RE: [cobalt-users] blocking a crawler

["Dan Kriwitsky" <webhosting@xxxxxxxxx>]

> I didn't look, but I have seen robots.txt files that state that
> viewing (or
> attempting to view) any of the listed directories will result in $x per
> occurrence charge billed to your ISP.  I get a kick out of that.

Let us know when you get a check. ;-)
>
>
> And do not underestimate the # of crawlers that don't respect it.  Anyone
> can create a spider to index sites.  I spider a number of
> external sites for
> a niche site I run and if I wanted to it would be easy to ignore
> robots.txt
> files.
>

Maybe I'll put a /hidden directory in my robots.txt file and watch the logs.
One of those annoying full screen JavaScript windows that don't like to
close easily should be good for those humans that like to look at robots.txt
files and browse to the restricted directories.
--
Dan Kriwitsky