RE: [cobalt-users] [RaQ2] combined-access files

["Matthews, Graham" <gmatthews@xxxxxxxxxxxxxxx>]

Spiders that are used to index the web for search engines will
obey rules that are placed in a robots.txt. If a robots.txt
doesn't exist, the spider will assume that it is ok to index all
pages on your site that it can get to.

http://info.webcrawler.com/mak/projects/robots/faq.html

- Graham


> -----Original Message-----
> From: Gerald Waugh [mailto:gerald@xxxxxxxxx]
> Sent: Thursday, 6 July 2000 9:24
> To: cobalt-users@xxxxxxxxxxxxxxx
> Subject: [cobalt-users] [RaQ2] combined-access files
> 
> 
> Sorry for the amount of included data,
> but the "these items in my combined-access log files looked strange/
> Why is this user looking for robots.txt?
> 
> www.koether.com 211.169.241.21 - - [04/Jul/2000:03:21:47 -0400] "GET /robots.txt
>  HTTP/1.0" 404 836 "-" "Slurp/si (slurp@xxxxxxxxxxx; http://www.inktomi.com/slurp.html)"
<snip>