Why is there a necessity to submit your sitemap?

Hello folks,

I was looking at Google's sitemap technology and I was wondering
if there is a better way than needing to login and submit the URL of
the map, surely we could provide a link on our sites for Google's
crawlers to find. (The technology would be useful to other search
engines too, provided they added support.)

The 'robots.txt' file seems the ideal place for this (proposed) link,
as it is already requested by GoogleBot, for instance, it would be
nice if we could have our 'robots.txt' file being:

User-agent: *
Disallow: /cgi-bin/
Sitemap: /sitemap.xml.gz

...with Google then using the sitemap to navigate the site instead of
needing to find the internal linking structure itself -- this being the
entire purpose of an xml sitemap in the first place.

--
C
2005-12-03