Re: Beating a Dead Horse

> The content is utter trash, but it got crawled
> with 3 hours and partially indexed within a day; 784 of the 911 URLs
> were indexed within 15 days.

Unfortunately, the above example is quite the exception to general
Google indexing procedures. Only 1/10 of 1% of all web sites have over
900 URLs. Many web sites have 5 pages or less. Most have 15 pages or
fewer.

The "crawled in 3 hours and indexed within 15 days" is similarly
outside the actual experience of the vast majority of all web sites.
Most web site owners will tell you it required Google months to first
crawl their web site and even longer to fully index it. Feel free to
call the three months waiting period hearsay, but then why do so many
people ask how long will it take for Google to crawl their site? And
why has there been so much discussion of that very same
long-Google-turn-around issue on many web sites? Why? Because
knowledgeable web masters logged into Google a week or two or three
after putting up their web site, submitted it, and checked to see if
they could find evidence of their web site in Google, which they could
then not find.

The one conclusion people might make is that blogs, extremely large web
sites, and news sites can get crawled quickly. This is logical since
Google has a vested interest in large content web sites. This means
that Google has moved well away from its original mandate to open the
web as a general, broad-based search engine.

> More efficient = more efficient at removing things it doesn't "like"

More efficient? Who says Google is more efficient? Google needs a
sitemap to fully traverse or "discover" a web site these days? And
when those site maps are courteously provided as a service by web
masters who take time out of their very busy day to attempt to help
Google and Google users, GoogleBot still can't fully traverses those
web sites? Except of course, those web sites with 500+ web pages.

The conclusion that sitemaps appear useful to Google for 500+ page web
sites appears valid, lacking evidence to the contrary, based on the
example presented for
http://seside.net/tests/2005/12/10/results-site-d. Unless Google
changes: Most web masters, small business owners, hobbyists and initial
start-up businesses beware, it will probably take months for Google to
correctly scan and index your web site, with or without a sitemap.