Re: [cobalt-users] [RaQ2] combined-access files

[Cody Watkins <codyw@xxxxxxxxxxxxxxxxx>]

That is just the Inktomi web crawler indexing your page.  Most crawlers
search for a robots.txt file before they index your site.  The robots.txt
file can tell the crawler not to look in certain directories.


Sincerely,
Cody Watkins
Paradox Web Hosting
www.paradoxwebhosting.com




At 07:23 PM 7/5/2000 -0400, you wrote:
>Sorry for the amount of included data,
>but the "these items in my combined-access log files looked strange/
>Why is this user looking for robots.txt?
>
>www.koether.com 211.169.241.21 - - [04/Jul/2000:03:21:47 -0400] "GET
/robots.txt
> HTTP/1.0" 404 836 "-" "Slurp/si (slurp@xxxxxxxxxxx;
http://www.inktomi.com/slur
>p.html)"
>www.koether.com 211.169.241.21 - - [04/Jul/2000:03:21:57 -0400] "GET /
HTTP/1.0"
> 200 1012 "-" "Slurp/si (slurp@xxxxxxxxxxx;
http://www.inktomi.com/slurp.html)"
>www.koether.com 216.35.103.61 - - [04/Jul/2000:09:59:41 -0400] "GET
/robots.txt
>HTTP/1.0" 404 836 "-" "Slurp/si (slurp@xxxxxxxxxxx;
http://www.inktomi.com/slurp
>.html)"
>www.koether.com 216.35.103.61 - - [04/Jul/2000:09:59:42 -0400] "GET /
HTTP/1.0"
>200 1012 "-" "Slurp/si (slurp@xxxxxxxxxxx;
http://www.inktomi.com/slurp.html)"
>www.koether.com 209.67.206.133 - - [04/Jul/2000:16:44:06 -0400] "GET
/robots.txt
> HTTP/1.0" 404 836 "-" "Slurp/si (slurp@xxxxxxxxxxx;
http://www.inktomi.com/slur
>p.html)"
>www.koether.com 209.67.206.133 - - [04/Jul/2000:16:44:07 -0400] "GET /
HTTP/1.0"
> 200 1012 "-" "Slurp/si (slurp@xxxxxxxxxxx;
http://www.inktomi.com/slurp.html)"
>www.vma332.com 205.188.208.41 - - [04/Jul/2000:19:49:23 -0400] "GET /
HTTP/1.0"
>200 5556 "-" "Mozilla/4.0 (compatible; MSIE 5.0; MSNIA; AOL 5.0; Windows
98; Dig
>www.koether.com 216.35.116.90 - - [05/Jul/2000:07:52:19 -0400] "GET
/robots.txt
>HTTP/1.0" 404 836 "-" "Slurp/si (slurp@xxxxxxxxxxx;
http://www.inktomi.com/slurp
>.html)"
>
>
>
>
>_______________________________________________
>cobalt-users mailing list
>cobalt-users@xxxxxxxxxxxxxxx
>To Subscribe or Unsubscribe, please go to:
>http://list.cobalt.com/mailman/listinfo/cobalt-users
>