Re: [cobalt-users] Robots.txt

["Carrie Bartkowiak" <ravencarrie@xxxxxxxx>]

> The only
> practical answer is to passwd-protect (htaccess) the site(s) or
page(s) in
> question. The spider may find 'em, they may appear on search
engines, but
> accessing them will be prevented by the existence of the password
> requirement.

Just one note to this - it doesn't work all of the time, either.
Googlebot (Google's friendly little crawler) will go through
password-protected pages regardless, and cache them. So if a user
can't get into the page directly, they can access Google's "view
cached page" option and see it anyway.
I'm not sure what other search engines are doing cached pages now, but
I do know that Google provides a form somewhere on their site where
you can tell Gogglebot not to cache your pages.

CarrieB