RE: [cobalt-users] blocking a crawler

["Chris Mason" <chris@xxxxxx>]

yeh, tried that, but ti is looking at one page all the time, not trying to
crawl the site. I put the ip into ipchains without the -l option, so it
won't be in the logs anymore.

-----Original Message-----
From: cobalt-users-admin@xxxxxxxxxxxxxxx
[mailto:cobalt-users-admin@xxxxxxxxxxxxxxx]On Behalf Of Dan Kriwitsky
Sent: Sunday, May 06, 2001 9:45 AM
To: cobalt-users@xxxxxxxxxxxxxxx
Subject: RE: [cobalt-users] blocking a crawler


> I've been watching my /var/log/httpd/access logfile and I find that
> crawler24.bos2.fast-search.net and crawler7.bos2.fast-search.net keep
> accessing the same page on one site all day, every few seconds.
>
>
Have you tried a robots.txt file?
--
Dan Kriwitsky



_______________________________________________
cobalt-users mailing list
cobalt-users@xxxxxxxxxxxxxxx
To Subscribe or Unsubscribe, please go to:
http://list.cobalt.com/mailman/listinfo/cobalt-users