Web accessibility snapshot: an effort to reveal coding guidelines conformance
Vagner Figueredo de Santana, Rogério Abreu de Paula · 2013 · Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/2461121.2461144
Summary
This paper presents the first iteration of the Web Accessibility Snapshot (WAS) project, a large-scale automated evaluation of web accessibility conformance across two sets of 1,000 websites each: the Alexa top 1,000 most popular sites and a randomly generated sample of 1,000 hostnames. The study was inspired by Nomensa's 2006 UN report that found 97% of 100 websites across 20 countries failed WCAG 1.0 Level A. Using AChecker (an automated WCAG 2.0 evaluation tool) automated via Selenium IDE, the authors assessed only "known problems" — issues identified with certainty as accessibility barriers — at WCAG 2.0 Level A, the minimum conformance level. The study focused on homepage evaluation as a proxy for overall site accessibility. The authors provide context through an overview of international accessibility legislation including UN resolutions, Section 508, Italy's Stanca Act, and Brazil's Decreto 5.296/2004, noting the persistent gap between legal requirements and actual implementation.
Key findings
The results revealed widespread accessibility failures: 95.66% of the top 1,000 popular websites had known accessibility problems (mean 32.97 problems, SD=60.47), while 85.11% of the random sample had known problems (mean 7.00, SD=35.48). The difference between the two groups was statistically significant (Wilcoxon Rank-sum test, p<0.001). Only 4.34% of popular websites and 14.89% of random websites had zero known WCAG 2.0 Level A problems. Popular websites had significantly more problems than random ones, likely because they contain more content, more UI components, and undergo more frequent updates — all of which increase the probability of markup coding errors. There was no significant correlation between popularity ranking and number of problems (Pearson r=0.09). The most common errors were images missing alt attributes and form input elements without associated labels — both fundamental accessibility requirements that are straightforward to fix. A notable concentration effect was found: 226 of the top 1,000 websites (22.6%) accounted for over 75% of all known problems.
Relevance
This study provides a sobering empirical baseline for the state of web accessibility, demonstrating that the vast majority of websites — including the most popular ones — fail even the minimum level of WCAG conformance. The finding that popular websites have more problems, not fewer, challenges any assumption that high-traffic sites with larger development teams would be more accessible. The most common errors (missing alt text, unlabelled form fields) are among the simplest to fix technically, suggesting that the barrier is awareness and process rather than technical difficulty. The WAS project's approach of regular, automated monitoring of web accessibility at scale anticipated the trend toward continuous accessibility monitoring that organisations now increasingly adopt. The study also highlights that conformance with coding guidelines is "necessary but not sufficient" — automated tools catch only certain types of errors (with 35% false negatives according to cited research), and user testing remains essential for true accessibility assessment.
Tags: automated testing · WCAG compliance · web accessibility · accessibility metrics · web standards · large-scale evaluation
Standards referenced: WCAG 2.0 · Section 508