RE: [cobalt-users] Robots.txt

["Simon Watts" <simon@xxxxxxxxxxx>]

-----Original Message-----
From: cobalt-users-admin@xxxxxxxxxxxxxxx
[mailto:cobalt-users-admin@xxxxxxxxxxxxxxx]On Behalf Of Jim Popovitch
Sent: 21 June 2001 01:24
To: cobalt-users@xxxxxxxxxxxxxxx
Subject: RE: [cobalt-users] Robots.txt


--- "Colin J. Raven" <cjraven@xxxxxxxxxxx> wrote:
>
> You really *can't* disallow a robot/spider search since not
> all spiders are good mannered and respect the directive.

Most spiders are built for speed, not perfection.  To that extent it is
possibly to fool most (if not all) spiders by using META refresh tag.
I doubt that spiders would care to handle this, and putting some
non-private text on the first page should fool them completely.  Just
add a statement, similar to the following one, in the HTML header of
your firstpage.  Substitute XX with the amount of seconds (zero is
valid) before the user will be redirected to the real site.

  <meta http-equiv="refresh" content="XX;url=/goodstuff/">

-Jim P.



The above is NOT good advice.

Using any form of meta refresh where the user is not in control
of their browsing is now frowned upon by an increasing number of engines.

Although the spiders don't follow this directive route, they do recognise
it.

Certain well known engines are now starting to de-list sites which use it.

The only 2 surefire ways I can immediately think of to prevent a spieder
accessing areas of a site;

No inbound link to the starting point within the area. (example; user
management area, should be provided as a qualified address to the user,
with no link to it from main site. If they need to access it regularly,
they should just set it up in favourites.)

Use of asp or php or cgi login script with valid user required. (db version
of htaccess, but with no "hard" link)

Regards

Si Watts
SiWIS


__________________________________________________
Do You Yahoo!?
Get personalized email addresses from Yahoo! Mail
http://personal.mail.yahoo.com/

_______________________________________________
cobalt-users mailing list
cobalt-users@xxxxxxxxxxxxxxx
To Subscribe or Unsubscribe, please go to:
http://list.cobalt.com/mailman/listinfo/cobalt-users