Web crawler

« Back to Glossary Index

Web Crawler Overview and Policies
– A web crawler is also known as a spider, ant, automatic indexer, or Web scutter.
– A web crawler starts with a list of URLs called seeds.
– As the crawler visits these URLs, it identifies all the hyperlinks in the retrieved web pages and adds them to the crawl frontier.
– URLs from the crawl frontier are recursively visited according to a set of policies.
– If the crawler is performing web archiving, it copies and saves the information as it goes.
– The archived web pages are stored in a repository designed to store and manage the collection.
– The behavior of a web crawler is determined by a combination of policies: selection policy, re-visit policy, politeness policy, and parallelization policy.
– The selection policy determines which pages to download.
– The re-visit policy determines when to check for changes to the pages.
– The politeness policy ensures that the crawler does not overload web sites.
– The parallelization policy coordinates distributed web crawlers.

Crawling Strategies and Experiments
– Large search engines cover only a portion of the publicly available web.
– The importance of a page is determined by its intrinsic quality, popularity in terms of links or visits, and even its URL.
– Different strategies, such as breadth-first, backlink count, and partial PageRank calculations, have been tested.
– Researchers have conducted studies and experiments to improve web crawling strategies.
– A study on crawling scheduling tested different ordering metrics such as breadth-first, backlink count, and partial PageRank calculations.
– Another crawl experiment found that a breadth-first crawl captures pages with high PageRank early in the crawl.
– OPIC (On-line Page Importance Computation) is an algorithm-based crawling strategy that distributes cash to pages based on their importance.
– Simulation experiments have been conducted on subsets of the web to compare strategies like breadth-first, depth-first, random ordering, and omniscient strategy.

Crawling Techniques and Considerations
– Crawlers may seek out only HTML pages.
– HTTP HEAD request can be made to determine MIME type.
– Certain characters in the URL can be used to request specific resources.
URL normalization avoids crawling the same resource multiple times.
– Conversion of URLs to lowercase.
– Removal of . and .. segments.
– Adding trailing slashes to the non-empty path component.
– Standardizing URLs in a consistent manner.
– Path-ascending crawler ascends to every path in each URL it intends to crawl.
– Focused crawlers download pages similar to each other.

Web Crawler Policies and Architecture
– Crawlers can have a crippling impact on server performance.
– Costs of using Web crawlers include network resources, server overload, poorly written crawlers, and disruptive personal crawlers.
– The robots exclusion protocol helps indicate which parts of a server should not be accessed by crawlers.
– Commercial search engines use the Crawl-delay parameter to indicate the interval between requests.
– A parallel crawler runs multiple processes in parallel.
– A highly optimized architecture is essential for a crawler.
– Building a high-performance system presents challenges in system design, I/O and network efficiency, and robustness.
– Lack of detail in published crawler designs prevents reproducibility.
– Web crawling can lead to unintended consequences and compromises.
– Security measures should be in place to prevent unauthorized access during crawling.

Crawler Identification and Other Considerations
– Web crawlers identify themselves to a Web server using the User-agent field of an HTTP request.
– The deep web consists of web pages accessible only through queries to a database.
– Visual web scrapers/crawlers structure data into columns and rows based on user requirements.
– Various crawler architectures exist for general-purpose crawlers.
– Automatic indexing, Gnutella crawler, Web archiving, Webgraph, and Website mirroring software are related topics.

Web crawler (Wikipedia)

A Web crawler, sometimes called a spider or spiderbot and often shortened to crawler, is an Internet bot that systematically browses the World Wide Web and that is typically operated by search engines for the purpose of Web indexing (web spidering).

Architecture of a Web crawler

Web search engines and some other websites use Web crawling or spidering software to update their web content or indices of other sites' web content. Web crawlers copy pages for processing by a search engine, which indexes the downloaded pages so that users can search more efficiently.

Crawlers consume resources on visited systems and often visit sites unprompted. Issues of schedule, load, and "politeness" come into play when large collections of pages are accessed. Mechanisms exist for public sites not wishing to be crawled to make this known to the crawling agent. For example, including a robots.txt file can request bots to index only parts of a website, or nothing at all.

The number of Internet pages is extremely large; even the largest crawlers fall short of making a complete index. For this reason, search engines struggled to give relevant search results in the early years of the World Wide Web, before 2000. Today, relevant results are given almost instantly.

Crawlers can validate hyperlinks and HTML code. They can also be used for web scraping and data-driven programming.

« Back to Glossary Index

Submit your RFP

We can't wait to read about your project. Use the form below to submit your RFP!

Gabrielle Buff
Gabrielle Buff

Just left us a 5 star review

Great customer service and was able to walk us through the various options available to us in a way that made sense. Would definitely recommend!

Stoute Web Solutions has been a valuable resource for our business. Their attention to detail, expertise, and willingness to help at a moment's notice make them an essential support system for us.

Paul and the team are very professional, courteous, and efficient. They always respond immediately even to my minute concerns. Also, their SEO consultation is superb. These are good people!

Paul Stoute & his team are top notch! You will not find a more honest, hard working group whose focus is the success of your business. If you’re ready to work with the best to create the best for your business, go Stoute Web Solutions; you’ll definitely be glad you did!

Wonderful people that understand our needs and make it happen!

Paul is the absolute best! Always there with solutions in high pressure situations. A steady hand; always there when needed; I would recommend Paul to anyone!

Vince Fogliani

The team over at Stoute web solutions set my business up with a fantastic new website, could not be happier

Steve Sacre

If You are looking for Website design & creativity look no further. Paul & his team are the epitome of excellence.Don't take my word just refer to my website "stevestours.net"that Stoute Web Solutions created.This should convince anyone that You have finally found Your perfect fit

Jamie Hill

Paul and the team at Stoute Web are amazing. They are super fast to answer questions. Super easy to work with, and knows their stuff. 10,000 stars.

Paul and the team from Stoute Web solutions are awesome to work with. They're super intuitive on what best suits your needs and the end product is even better. We will be using them exclusively for our web design and hosting.

Dean Eardley

Beautifully functional websites from professional, knowledgeable team.

Along with hosting most of my url's Paul's business has helped me with website development, graphic design and even a really cool back end database app! I highly recommend him as your 360 solution to making your business more visible in today's social media driven marketplace.

I hate dealing with domain/site hosts. After terrible service for over a decade from Dreamhost, I was desperate to find a new one. I was lucky enough to win...

Paul Stoute has been extremely helpful in helping me choose the best package to suite my needs. Any time I had a technical issue he was there to help me through it. Superb customer service at a great value. I would recommend his services to anyone that wants a hassle free and quality experience for their website needs.

Paul is the BEST! I am a current customer and happy to say he has never let me down. Always responds quickly and if he cant fix the issue right away, if available, he provides you a temporary work around while researching the correct fix! Thanks for being an honest and great company!!

Paul Stoute is absolutely wonderful. Paul always responds to my calls and emails right away. He is truly the backbone of my business. From my fantastic website to popping right up on Google when people search for me and designing my business cards, Paul has been there every step of the way. I would recommend this company to anyone.

I can't say enough great things about Green Tie Hosting. Paul was wonderful in helping me get my website up and running quickly. I have stayed with Green...