[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [cobalt-users] blocking a crawler
- Subject: Re: [cobalt-users] blocking a crawler
- From: "johnny t" <eastlan@xxxxxxxxxxx>
- Date: Sun May 6 15:54:00 2001
- List-id: Mailing list for users to share thoughts on Cobalt products. <cobalt-users.list.cobalt.com>
Shouldn't you be able to create a robot.txt to block access?
Just a thought.
EL
I've been watching my /var/log/httpd/access logfile and I find that
crawler24.bos2.fast-search.net and crawler7.bos2.fast-search.net keep
accessing the same page on one site all day, every few seconds.
I have tried to contact the admin but have had no luck. How can I block
access to this IP? I don;t think httpd runs under tcp wrappers so I don;t
think putting the IP into /etc/hosts.deny does anything.
Chris Mason
Box 340, The Valley, Anguilla, British West Indies
Tel: 264 497 5670 Fax: 264 497 8463
USA Fax (561) 382-7771
Take a virtual tour of the island
http://net.ai/ The Anguilla Guide
Find out more about NetConcepts
www.netconcepts.ai
Talk to me in real time with Instant Messenger: masonc92@xxxxxxxxxxx
Signature
F331 8AD1 36FB B3B0 DF9F D95B 8024 D1EA 7450 D50C
_______________________________________________
cobalt-users mailing list
cobalt-users@xxxxxxxxxxxxxxx
To Subscribe or Unsubscribe, please go to:
http://list.cobalt.com/mailman/listinfo/cobalt-users
_________________________________________________________________
Get your FREE download of MSN Explorer at http://explorer.msn.com