[OAI-implementers] Server Load for ListIdentifiers, ListRecords calls

[Alan Kent]

On Fri, Apr 12, 2002 at 12:08:50PM -0400, Yi-Lun Ding wrote:
> I am thinking of implementing OAI, but am a little wary of the load
> requirements of ListIdentifiers and ListRecords for large document
> repositories.  One, there is the bandwidth requirement of transferring huge
> blocks of data.  Two, the process would have to go through each record in
> the database and check the TimeModified/Set attributes.
> 
> How are people dealing with this issue?
> 
> Thanks,
> 
> yi-lun

Not sure exactly of your concerns for "bandwidth" for large document
repositories. If you don't want people to harvest a copy of the data,
then what do you want to use OAI for? (note: you can provide only
metadata about documents - not the full document via OAI - which would
be much shorter).

As to checking the time modified attribute, I plan to store the collection
in a database which can query on time modified (and supported metadata
prefixes) so to identify which information to return is a matter of
doing a simple query.

If you don't have a database engine capable of doing such queries,
it will be slower.

Alan