RE: [cobalt-users] Robots.txt

["Dan Kriwitsky" <webhosting@xxxxxxxxx>]

>    Considering there is no record for "robots" nor "robots.txt" in the
> archives, i´d like to know how to use the file robots.txt located under
> /etc/admserv/html of my Cobalt. By now, it reads as:
>
> # Prevent all robots from visiting this site:
>
> User-agent: *
> Disallow: /
>
> How can i, for instance forbid a robot to search my pages? Considering the
> directory it is located (under root), which virtual sites would it affect?

That would only affect the admin GUI files not any virtual sites. A proper
robot looks for the robot.txt file at http://www.domain.com/robots.txt
Search webcrawler.com for info or www.searchenginewatch.com
--
Dan Kriwitsky