[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: [cobalt-users] blocking a crawler
- Subject: RE: [cobalt-users] blocking a crawler
- From: "Dan Kriwitsky" <webhosting@xxxxxxxxx>
- Date: Sat May 5 22:44:00 2001
- List-id: Mailing list for users to share thoughts on Cobalt products. <cobalt-users.list.cobalt.com>
> I've been watching my /var/log/httpd/access logfile and I find that
> crawler24.bos2.fast-search.net and crawler7.bos2.fast-search.net keep
> accessing the same page on one site all day, every few seconds.
>
>
Have you tried a robots.txt file?
--
Dan Kriwitsky