Re: [cobalt-users] Robots.txt

["Carrie Bartkowiak" <ravencarrie@xxxxxxxx>]

> # Prevent all robots from visiting this site:
> User-agent: *
> Disallow: /
>
> How can i, for instance forbid a robot to search my pages?

Copy those lines you have above, name the file robots.txt, and drop it
into the /web directory of whichever site you want to do this for.
Alternatively, you can do this in particular folders rather than
affecting the entire site by listing off the particular folders like
so:
User-agent: *
Disallow: /members
/images
/somefolder
/someotherfolder

It only affects the sites you put it into, because the robots don't
crawl up into the non-web directories. So for instance, putting it in
/home/sites won't do anything, but putting it in /home/sites/site3/web
will make it take effect for site3.

CarrieB