Re: [cobalt-users] [RaQ2] combined-access files

["dylan" <dylan@xxxxxxxxxxxxxx>]

robots.txt is a file that search engines look for before it indexes the
site.

It is nothing to worry about.. robots.txt is used for sites that dont wish
to be indexed by serach engines. In fact.. the more robots.txt hits you
have.. the more search engines have found you.



> Why is this user looking for robots.txt?
>
> www.koether.com 211.169.241.21 - - [04/Jul/2000:03:21:47 -0400] "GET
/robots.txt
>  HTTP/1.0" 404 836 "-" "Slurp/si (slurp@xxxxxxxxxxx;
http://www.inktomi.com/slur
> p.html)"
> www.koether.com 211.169.241.21 - - [04/Jul/2000:03:21:57 -0400] "GET /
HTTP/1.0"
>  200 1012 "-" "Slurp/si (slurp@xxxxxxxxxxx;
http://www.inktomi.com/slurp.html)"
> www.koether.com 216.35.103.61 - - [04/Jul/2000:09:59:41 -0400] "GET
/robots.txt
> HTTP/1.0" 404 836 "-" "Slurp/si (slurp@xxxxxxxxxxx;
http://www.inktomi.com/slurp
> .html)"
> www.koether.com 216.35.103.61 - - [04/Jul/2000:09:59:42 -0400] "GET /
HTTP/1.0"
> 200 1012 "-" "Slurp/si (slurp@xxxxxxxxxxx;
http://www.inktomi.com/slurp.html)"
> www.koether.com 209.67.206.133 - - [04/Jul/2000:16:44:06 -0400] "GET
/robots.txt
>  HTTP/1.0" 404 836 "-" "Slurp/si (slurp@xxxxxxxxxxx;
http://www.inktomi.com/slur
> p.html)"
> www.koether.com 209.67.206.133 - - [04/Jul/2000:16:44:07 -0400] "GET /
HTTP/1.0"
>  200 1012 "-" "Slurp/si (slurp@xxxxxxxxxxx;
http://www.inktomi.com/slurp.html)"
> www.vma332.com 205.188.208.41 - - [04/Jul/2000:19:49:23 -0400] "GET /
HTTP/1.0"
> 200 5556 "-" "Mozilla/4.0 (compatible; MSIE 5.0; MSNIA; AOL 5.0; Windows
98; Dig
> www.koether.com 216.35.116.90 - - [05/Jul/2000:07:52:19 -0400] "GET
/robots.txt
> HTTP/1.0" 404 836 "-" "Slurp/si (slurp@xxxxxxxxxxx;
http://www.inktomi.com/slurp
> .html)"
>
>
>
>
> _______________________________________________
> cobalt-users mailing list
> cobalt-users@xxxxxxxxxxxxxxx
> To Subscribe or Unsubscribe, please go to:
> http://list.cobalt.com/mailman/listinfo/cobalt-users
>