Distributed web crawling

Types of Policies
– Two types of policies studied by Cho and Garcia-Molina
– Dynamic assignment policy allows central server to assign new URLs to crawlers dynamically
– Dynamic assignment can balance load and add/remove downloader processes
– Static assignment policy uses fixed rules to assign new URLs to crawlers
– Static assignment requires exchange of URLs between crawling processes

Dynamic Assignment
– Central server assigns new URLs to crawlers dynamically
– Dynamic assignment balances load and can add/remove downloader processes
– Large crawler configuration distributes DNS resolver and queues
– Small crawler configuration has central DNS resolver and central queues per website
– Workload must be transferred to distributed crawling processes for large crawls

Static Assignment
– Fixed rule defines how to assign new URLs to crawlers
– Hashing function transforms URLs into corresponding crawling process index
– Exchange of URLs must occur for external links between crawling processes
– URL exchange should be done in batch to reduce overhead
– Most cited URLs should be known by all crawling processes before the crawl

Implementations
– Most modern commercial search engines use distributed web crawling
– Google and Yahoo use thousands of individual computers for crawling
– Newer projects enlist volunteers to join distributed crawling using personal computers
– LookSmart’s Grub distributed web-crawling project uses less structured collaboration
– Wikia acquired Grub from LookSmart in 2007

Drawbacks
– According to Nutch FAQ, distributed web crawling doesn’t significantly save bandwidth
– Successful search engine requires more bandwidth for query result pages than crawler needs for downloading pages