Sitemaps

History and Purpose of Sitemaps
– Google introduced Sitemaps 0.84 in June 2005
– Google, Yahoo!, and Microsoft announced joint support for the Sitemaps protocol in November 2006
– Ask.com and IBM announced support for Sitemaps in April 2007
– State governments of Arizona, California, Utah, and Virginia announced they would use Sitemaps in May 2007
– Sitemaps protocol is based on ideas from Crawler-friendly Web Servers
– Sitemaps are beneficial for websites with unavailable areas through the browsable interface
– Useful for websites with rich Ajax, Silverlight, or Flash content not processed by search engines
– Helps with large websites to avoid overlooking new or updated content
– Effective for websites with isolated or poorly linked pages
– Useful for websites with few external links

File Format and Element Definitions
– Sitemaps use XML tags and can be UTF-8 encoded
– Sitemaps can also be plain text lists of URLs
– Sitemaps can be compressed in .gz format
– Sitemap index files are necessary for large sites with a maximum size of 50MiB or 50,000 URLs
– Sitemap index files reference separate sitemaps
– The ‘urlset’ element is required and contains the Sitemap
– The ‘url’ element is required and serves as the parent element for each entry
– The ‘sitemapindex’ element is required for Sitemap index files
– The ‘sitemap’ element is required and serves as the parent element for each entry in the index
– The ‘loc’ element is required and provides the full URL of the page or sitemap

Other Formats and Search Engine Submission
– Sitemaps can be in the form of a simple list of URLs in a text file
– Syndication feeds can be used to submit URLs to crawlers
– Having a syndication feed as a delta update can supplement a complete sitemap
– Search engine submission of Sitemaps provides status information and processing errors
– Sitemap location can be included in the robots.txt file or specified in the search engine’s submission URL.

Sitemap Limits
– Sitemap files have a limit of 50,000 URLs and 50MiB (52,428,800 bytes) per sitemap.
– Sitemaps can be compressed using gzip, reducing bandwidth consumption.
– Multiple sitemap files are supported, with a Sitemap index file serving as an entry point.
– Sitemap index files may not list more than 50,000 Sitemaps and must be no larger than 50MiB and can be compressed.
– You can have more than one Sitemap index file.

Additional Sitemap Types
– A number of additional XML sitemap types outside of the scope of the Sitemaps protocol are supported by Google.
– Video and image sitemaps are intended to improve the capability of websites to rank in image and video searches.

Subgroup: Video Sitemaps
– Video sitemaps indicate data related to embedding and autoplaying, preferred thumbnails to show in search results, publication date, video duration, and other metadata.
– Video sitemaps are also used to allow search engines to index videos that are embedded on a website, but that are hosted externally, such as on Vimeo or YouTube.

Subgroup: Image Sitemaps
– Image sitemaps are used to indicate image metadata, such as licensing information, geographic location, and an image’s caption.

Subgroup: Google News Sitemaps
– Google supports a Google News sitemap type for facilitating quick indexing of time-sensitive news subjects.