[OAI-general] OAI and web crawlers

[Michael L. Nelson]

I've just added:

	# please use our Open Archives Initiative (OAI) interface instead!
	# http://naca.larc.nasa.gov/oai/
	# see http://www.openarchives.org/ for more info

to my robots.txt file for my two DLs (LTRS & NACATRS).  I doubt these
messages will be read by humans, but stranger things have happened.

if your DL is like mine, at any given time webcrawlers from Inktomi,
Google, etc. are meandering about.  I don't discourage this behaivor
(cf. arXiv), mostly because its never been too much of a problem.  

but it would seem that these crawlers would benefit using the OAI
interface where possible.  OAI is doing quite well within the publishing /
library community, but has anyone made any overtures to the webcrawling
community?  Any ideas on how to do so?  I would expect the potential for
reduced network traffic and increased indexed content could cause them to
modify their robots to understand OAI...

regards,

Michael

---
Michael L. Nelson			
207 Manning Hall, School of Information and Library Science
University of North Carolina 		mln@ils.unc.edu
Chapel Hill, NC 27599			http://ils.unc.edu/~mln/
+1 919 966 5042				+1 919 962 8071 (f)