OPTIMAL-EM: Optimised Population Sourcing for Web Accessibility Evaluation
Alexander Hambley, Yeliz Yesilada, Markel Vigo, Simon Harper · 2023 · Proceedings of the 20th International Web for All Conference (W4A '23) · doi:10.1145/3587281.3587962
Summary
This extended abstract presents OPTIMAL-EM, a prototypical tool and framework designed to optimise the web accessibility evaluation process by systematically selecting representative pages from a website for auditing. Currently, accessibility evaluations supported by methodologies like WCAG and the Website Accessibility Conformance Evaluation Methodology (WCAG-EM) typically rely on ad-hoc, heuristic-based page selection, which risks missing important page types and introduces inconsistency. OPTIMAL-EM addresses this by using statistical methods to identify a representative sample of pages that collectively represent the full population of a website's pages. The tool operates in three stages: first, it sources and downloads pages (via crawling, server log files, or the Common Crawl), processes them using BeautifulSoup, creates a Vector-Space Model using CountVectorizer, and clusters them using DBSCAN. The second stage calculates the complexity of the page population using six metrics: coverage (proportion of site pages included), representativeness (statistical similarity between sample and population), complexity (structural variation among pages), popularity (user traffic patterns), freshness (recency of page content), and accessibility (existing accessibility characteristics). The third stage calculates the optimal sample size and produces a final report identifying representative pages from each cluster for auditors to review.
Key findings
The framework considers multiple page representation approaches including HTML structure, HTML content, tags, DOM representation, and mixtures of content and structure — acknowledging that the choice of representation affects clustering quality and ultimately the representativeness of selected pages. The tool evaluated several population sourcing methods: server log files provide pages actually visited by users but may miss pages or include outdated templates; crawling captures linked pages but risks missing user-accessed pages and faces challenges with spider traps and politeness policies. An evaluation on Studentnet (a University of Manchester subsite) found that a large proportion of crawled pages were not actually accessed by users, suggesting that crawling alone may be insufficient for identifying the most important pages to audit. The six-metric framework aims to ensure that auditors review pages that are not only structurally representative but also popular (heavily trafficked), fresh (recently updated), and already known to have accessibility characteristics worth examining.
Relevance
This work addresses a practical problem that every accessibility auditor faces: which pages on a large website should be evaluated? For organizations with thousands or millions of pages, testing everything is impossible, and ad-hoc selection risks producing unrepresentative results. OPTIMAL-EM offers a principled, statistical approach to page selection that could make accessibility audits more efficient and more reliable. The tool aligns with the WCAG-EM methodology, which already calls for representative page selection but provides limited guidance on how to achieve it systematically. For enterprise accessibility teams, this approach could significantly reduce audit costs while improving confidence that results reflect the true state of the site. As an extended abstract presenting a doctoral research prototype, the tool's full validation and comparison against human expert page selection remain as future work.
Tags: automated accessibility testing · web accessibility · accessibility evaluation · web crawling · machine learning · WCAG compliance · accessibility standards · large-scale analysis
Standards referenced: WCAG · WCAG-EM