[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: [cobalt-users] Robots.txt
- Subject: RE: [cobalt-users] Robots.txt
- From: Jim Popovitch <jimpop@xxxxxxxxx>
- Date: Thu Jun 21 06:13:40 2001
- List-id: Mailing list for users to share thoughts on Cobalt products. <cobalt-users.list.cobalt.com>
--- Simon Watts <simon@xxxxxxxxxxx> wrote:
>
> Just as an aside, this is the robots.txt file from google itself!
>
> User-agent: *
> Disallow: /search
> Disallow: /groups
> Disallow: /keyword/
> Disallow: /u/
> Disallow: /cobrand
> Disallow: /custom
>
That's interesting, groups.google.com has the exact same one (same
servers?) and none of their other well known servers (directory &
toolbar) have one.
-Jim P.
__________________________________________________
Do You Yahoo!?
Get personalized email addresses from Yahoo! Mail
http://personal.mail.yahoo.com/