Re: [cobalt-users] blocking a crawler

["CSN Support" <support@xxxxxxxxx>]

Have you tried creating a robots.txt in the root dir of your website?   Look
out on the net for the syntax, but it allows you to stop robots and indices
from crawling your website.

--Robert

----- Original Message -----
From: "Chris Mason" <chris@xxxxxx>
To: <cobalt-users@xxxxxxxxxxxxxxx>
Sent: Sunday, May 06, 2001 7:16 AM
Subject: [cobalt-users] blocking a crawler


> I've been watching my /var/log/httpd/access logfile and I find that
> crawler24.bos2.fast-search.net and crawler7.bos2.fast-search.net keep
> accessing the same page on one site all day, every few seconds.
>
> I have tried to contact the admin but have had no luck. How can I block
> access to this IP? I don;t think httpd runs under tcp wrappers so I don;t
> think putting the IP into /etc/hosts.deny does anything.
>
>
>
> Chris Mason
> Box 340, The Valley, Anguilla, British West Indies
> Tel: 264 497 5670 Fax: 264 497 8463
> USA Fax (561) 382-7771
> Take a virtual tour of the island
> http://net.ai/ The Anguilla Guide
> Find out more about NetConcepts
> www.netconcepts.ai
> Talk to me in real time with Instant Messenger: masonc92@xxxxxxxxxxx
> Signature
> F331 8AD1 36FB B3B0 DF9F  D95B 8024 D1EA 7450 D50C
>
> _______________________________________________
> cobalt-users mailing list
> cobalt-users@xxxxxxxxxxxxxxx
> To Subscribe or Unsubscribe, please go to:
> http://list.cobalt.com/mailman/listinfo/cobalt-users
>