[OAI-implementers] How to verify a download worked?

[Alan Kent]

Hi All,

I was wondering if anyone has good schemes for verifying if a download
of metadata 'worked'. For example, I crawled the arXiv site and got
around 60,000 records. However, it turns out the site actually has
190,000 or so records. So I only got 1/3 of the site!

Has anyone used any clever tricks to verify how well a crawl worked?
I now have to work out if my crawler has been discarding one in three
records! :-(

Alan
-- 
Alan Kent (mailto:ajk@mds.rmit.edu.au, http://www.mds.rmit.edu.au)
Postal: Multimedia Database Systems, RMIT, GPO Box 2476V, Melbourne 3001.
Where: RMIT MDS, Bld 91, Level 3, 110 Victoria St, Carlton 3053, VIC Australia.
Phone: +61 3 9925 4114  Reception: +61 3 9925 4099  Fax: +61 3 9925 4098