Re: [cobalt-users] IP vs Name Based Hosting

[flash22@xxxxxxx]

On Sun, 18 Jun 2000, Dennis wrote:
> At 09:08 AM 4/2/00 -0400, you(??) wrote:
> >
> >Last I knew, the major search engines use HTTP/1.0 spiders.  If you connect
> >to port 80 of any name based site and do the standard GET / HTTP/1.0, you
> >receive the content of the first site within the listing of name based
> >hosts.  This is exactly what the above mentioned spiders will see, not the
> >intended content of the name based site.
> >
> >I'm doing my own research now, but I'd like to hear any and all comments on
> >the topic.

Most major search engines can handle virtual (name based) domains now,
googlebot finally fixed their spider, altavista understands and a large
chunk of other search engines based on inktomi's slurp understand...
even aol's spider/proxy understands...

note that you can't really conclude the spiders use http 1.0 from the log
files, most of them claim to use 1.0 but include the HOST header anyhow...
(the fact they appear in log files is the hint it worked ;)

Some really old cache proxy's might still be confused, but these folks are
probably having real problems anyhow -/

(I recently found that a cache server was returning pages from a customers
former site that were 2 years out of date, yeash)


g