robots.txt

History and Standardization of robots.txt - Proposed by Martijn Koster in February 1994 - Proposed on the www-talk mailing list - Provoked by Charles Stross' badly-behaved web crawler - Became a de facto standard for web crawlers - Officially standardized by Google in July 2019 - Robots.txt file is placed in the root of the website hierarchy - Contains instructions for web robots in a specific format - Instructs robots which web pages they can and cannot access - Important for web crawlers from search engines like Google - Each subdomain and protocol/port needs its own robots.txt file Security and Limitations of robots.txt - Robots.txt compliance is voluntary and advisory - Malicious web robots may ignore robots.txt instructions - Security through obscurity is discouraged by standards bodies - NIST recommends against relying on secrecy for system security - Robots.txt should not be solely relied upon for security purposes Alternatives to robots.txt - Robots can pass a special user-agent to the web server - Server can be configured to return failure or alternative content - Some sites have humans.txt files for human-readable information - Some sites redirect humans.txt to an About page - Google previously had a joke file instructing the Terminator not to harm the founders Nonstandard extensions of robots.txt - Crawl-delay directive allows throttling of bot visits - Interpretation of crawl-delay value depends on the crawler - Yandex uses crawl-delay as the number of seconds to wait between visits - Bing defines crawl-delay as the size of a time window for accessing a site - Google provides a search console interface for controlling bot visits - Some crawlers support the Sitemap directive in robots.txt - Allows multiple Sitemaps in the same robots.txt file - Sitemaps are specified with full URLs - BingBot defines crawl-delay as the size of a time window - Google provides a search console interface for managing Sitemaps Meta tags, headers, and related concepts - Robots exclusion directives can be applied through meta tags and HTTP headers. - Robots meta tags cannot be used for non-HTML files. - X-Robots-Tag can be added to non-HTML files using .htaccess and httpd.conf files. - A noindex meta tag can be used to exclude a page from indexing. - A noindex HTTP response header can also be used to exclude a page from indexing. - ads.txt: a standard for listing authorized ad sellers. - security.txt: a file for reporting security vulnerabilities. - Automated Content Access Protocol: a failed proposal to extend robots.txt. - BotSeer: an inactive search engine for robots.txt files. - Distributed web crawling: a technique for distributing web crawling tasks. Note: The content related to maximum size of a robots.txt file, robots.txt and web archives, and external links has been excluded as they do not fit into the identified groups.