RE: [cobalt-users] Robots.txt

[Jim Popovitch <jimpop@xxxxxxxxx>]

--- "Colin J. Raven" <cjraven@xxxxxxxxxxx> wrote:
> 
> You really *can't* disallow a robot/spider search since not 
> all spiders are good mannered and respect the directive. 

Most spiders are built for speed, not perfection.  To that extent it is
possibly to fool most (if not all) spiders by using META refresh tag. 
I doubt that spiders would care to handle this, and putting some
non-private text on the first page should fool them completely.  Just
add a statement, similar to the following one, in the HTML header of
your firstpage.  Substitute XX with the amount of seconds (zero is
valid) before the user will be redirected to the real site.

  <meta http-equiv="refresh" content="XX;url=/goodstuff/">

-Jim P.


__________________________________________________
Do You Yahoo!?
Get personalized email addresses from Yahoo! Mail
http://personal.mail.yahoo.com/