[OAI-implementers] Perl oai modules

[Tim Brody]

Ed Summers wrote:
> On Fri, Feb 25, 2005 at 09:28:47PM +0000, Tim Brody wrote:
> 
>>These modules represent very similar functionality to Ed Summer's 
>>Net::OAI::Harvester for harvesting, but with support for constructing 
>>OAI responses (the repository side). I hope (!) they are also more 
>>resilient to repository weirdness.
> 
> Heh, could you be more specific about the "repository weirdness". Nice
> work by the way.

UTF-8 errors are probably the most common, but I've "developed out" my 
capability to handle those. Ideally I'd like a PerlIO layer that handles 
bad UTF-8, otherwise I'm sure Perl5.8's UTF support should be usable to 
fix bad chars (or spot latin and recode).

A couple of sites have bad baseURLs in their Identify response, which I 
now check for and ignore - in principle a harvester should follow 
baseURL like a redirect, but some sites have relative URLs!

One site indexed in Celestial (an OAI1.1) expects the verb as the first 
argument in the query, so I've added a special case for that.

Another site has a bad implementation of timestamps - it only accepts 
date+time. So something else to add would be to retry a request with 
appended to from/until T00:00:00Z (probably application layer rather 
than API).

I've added support for oai_dc: or dc: prefixed tags in Dublin Core, 
although I guess I could just check for the namespace.

Other than that; graceful handling of retry-after, redirect, resumption 
token repetition (to one depth only), and retry-on empty response.

All the best,
Tim.