Web Crawling

The automated process of systematically browsing and indexing web pages by following hyperlinks from a starting URL. In accessibility evaluation, web crawlers are used to discover and catalogue pages across a website for audit purposes. Two primary traversal strategies exist: breadth-first crawling, which visits all links at each depth level before going deeper and tends to discover more diverse page types; and depth-first crawling, which follows links to their deepest point before backtracking and tends to find more similar pages. Web crawling is one method for establishing the population of pages to include in an accessibility conformance evaluation.