[OAI-implementers] Re: OAI-implementers Digest, Vol 30, Issue 4

[Mark Diggory]

On Mar 27, 2007, at 1:00 PM, oai-implementers- 
request at openarchives.org wrote:
>
> Torsten Schaßan wrote:
>> Dear all,
>>
>> today in my institution we had a long discussion about the
>> implementation of an OAI interface and the possible dangers of  
>> OAI: how
>> to prevent denial-of-service-like numbers of harvester requests?
>>
>> What experiences do you have with such things as OAI doesn't have any
>> precaution to prevent something like this? Did you ever experience
>> problems for your servers or are there any measures we could  
>> implement
>> and which we have overlooked?

If your using Apache 2.x you should make mod_cband your friend. You  
can throttle your harvesters with it.

http://cband.linux.pl/

> 5. Flow Control, Load Balancing and Redirection
>
> It is essential that harvesting software respect flow control  
> responses from repositories. Not doing so may turn a harvest  
> attempt into a denial-of-service attack on the repository.
>
> Repositories which issue 503 Service Unavailable HTTP replies as a  
> means of flow control should include a Retry-After HTTP header to  
> indicate how long a harvester should wait before issuing the  
> request again. Harvesters that encounter a 503 reply without a  
> Retry-After header should not automatically retry without  
> considerable delay (minutes) or, preferably, manual intervention.  
> Harvesters must not be written to retry indefinitely.
>
> Either as part of a load balancing strategy or for other reasons, a  
> repository may issue 302 Found HTTP replies to redirect the  
> harvester to another URL indicated in a Location HTTP header.  
> Harvesters that encounter a 302 reply without a Location header  
> should not automatically retry the request.


-Mark

~~~~~~~~~~~~~
Mark R. Diggory - DSpace Systems Manager
MIT Libraries, Systems and Technology Services
Massachusetts Institute of Technology
Office: E25-131
Phone: (617) 253-1096