Web Crawler Again

[<Win dot Shih at UCHSC dot edu>]

Dear List Members,

 

We got another web crawlers hit last weekend.  This time it was from
Inktomisearch, a company that scours websites for various search
engines, including Yahoo, MSN, AltaVista, and others.  It was such a
massive and continuous hit that almost our entire licenses were used by
the crawlers during the two-hour period.  The server log also indicted
that there were over 2,000 denied HTTP accesses.  

 

Although we have the standard robots.txt that III sets up for all their
customers, somehow it did not block this intrusion.  I now have added IP
ranges that I can find of Inktomi web crawlers into the denied access
list, but the addresses are incomplete.  I also notice that other ILS
systems, such as Sirsi and Endeavor, block any crawler in their
robots.txt file.  

 

Has anyone experienced such "attack"?  What are the other options
available for this kind of threat?   

 

Thanks in advance.

 

Win Shih

University of Colorado Health Sciences Center Library



--- StripMime Report -- processed MIME parts ---
multipart/alternative
  text/plain (text body -- kept)
  text/html
---