Re: [cobalt-users] Robots.txt

[Jim Popovitch <jimpop@xxxxxxxxx>]

Zarrir,

Your current robots.txt file is doing just that, but only for admserve.
 You would need to copy that file to the DocumentRoot of all websites
to prevent all sites from being spidered.  The User-agent statement
means that the next lines affect all browsers.  Try replacing * with
'Google' to restrict Google from indexing your site. 

The Disallow statement(s) define the directories off of DocumentRoot
that are off limits.  Here is a typical example of a site that gets
some indexting but restricts other:

User-agent *
Disallow: /cgi-bin/
Disallow: /private/
Disallow: /data/

Checkout http://www.robotstxt.org for more info.

-Jim P.



--- Zarrir Junior <zarrir@xxxxxxxxxxxx> wrote:
>       Hi,
> 
>    Considering there is no record for "robots" nor "robots.txt" in
> the
> archives, i´d like to know how to use the file robots.txt located
> under
> /etc/admserv/html of my Cobalt. By now, it reads as:
> 
> # Prevent all robots from visiting this site:
> 
> User-agent: *
> Disallow: /
> 
> How can i, for instance forbid a robot to search my pages?
> Considering the
> directory it is located (under root), which virtual sites would it
> affect?
>      Thanks in advance!
>               Zarrir
> 


__________________________________________________
Do You Yahoo!?
Get personalized email addresses from Yahoo! Mail
http://personal.mail.yahoo.com/