[OAI-implementers] Resumption 'from' date.

[Alan Kent]

On Fri, Feb 08, 2002 at 05:02:50PM -0600, Tim Cole wrote:
> Not to curtail the very interesting technical back and forth, but...

If a goal of OAI is to keep it simple and not change radically (which
I can appreciate), then I would revert to my simpler proposed extension
which is to allow a server to return an optional addtional date/time in
ListRecords/ListIdentifiers responses indicating that "the client can
use this as a 'from' date to resume if the response token times out".

For the very simple implementations (or small data volume sites), the
server just omits this value.

For more sophisticated implementations with a database engine behind
the scenes (for example, so it can easly sort the records), then for
each packet it can say "I am guaranteeing to at least have returned
everything up to this date". This allows a harvester client hitting a
large site for a first time to not have to start again from scratch if
something goes wrong (resumption token time out etc). Date resolution
is fine here (getting some entries a second time is not the problem -
the problem is starting again from the very beginning).

My first attempt at a client harvester for example took about a day to
go to several sites and download everything. It hung several times
(unknown network issues), meaning I had to restart it on some large
sites from scratch. Many other sites it failed for (I can post a list
if people are interested - many seemed to only support GET and not POST).

Alan