Re: [cobalt-users] Test 404 Response? Anyone seen this?

[flash22@xxxxxxx]

On Sat, 5 May 2001, Wayne Sagar wrote:

> At 04:52 PM 5/5/01 -0400, you wrote:
> >no, it's  some lame search engine spider checking to see if you return
> >real 404 pages or redirect to some other place (which makes it impossible
> >for the spider to detect pages that aren't available anymore and delete
> >them from the search engine index)
> 
> Thanks... do we hose this process when we use the custom error pages..
> which in the case of at least two of the sites I manage call pages from a
> folder (could not get them to call to path address outside the
> home/sites/sitex/web folder..?

Depends how you did it...the way to check is to look at the raw access
logs and see if the response code returned with the final page still has
the 404 of if it got changed to a 200.

(It's the number right after the "GET /whateverpage HTTP/1.1" )

As long as you are still returning a 404 reply status you should be ok...

The issue is that returning a redirect can essentially fool the spider, it
says, 'the page isn't here, but it exists, and is here....

Now for a real hack (cause i was looking for an example for you and found
this:

63.71.179.191 - - [26/Mar/2001:13:00:45 -0500] 
"OPTIONS / HTTP/1.1" 200 -"-" "Microsoft Data Access Internet Publishing Provider Cache Manager"

63.71.179.191 - - [26/Mar/2001:13:00:46 -0500] 
"GET /_vti_inf.html HTTP/1.1" 302 235 "-" "Mozilla/2.0 (compatible; MS
FrontPage 4.0)"

63.71.179.191 - - [26/Mar/2001:13:00:46 -0500] 
"POST /_vti_bin/shtml.exe/_vti_rpc HTTP/1.1" 404 836 "-" "MSFrontPage/4.0"

63.71.179.191 - - [26/Mar/2001:13:00:47 -0500] "OPTIONS /guidance.shtml
HTTP/1.1" 200 - "-" "Microsoft Data Access Internet Publishing Provider DAV"

Cute eh?

[someone checking to see if they have write access ;]
gsh