Quantitative Evaluation for Web Accessibility with Respect to Disabled Groups
Pornpat Sirithumgul, Atiwong Suchato, Proadpran Punyabukkana · 2009 · Proceedings of the 2009 International Cross-Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/1535654.1535687
Summary
This paper proposes a methodology for evaluating web accessibility that goes beyond standard WCAG conformance levels to assess suitability for specific disability groups, particularly vision-impaired and hearing-impaired users. The authors argue that WCAG conformance levels (A, AA, AAA) address accessibility broadly but cannot indicate whether a page is suitable for a specific group of disabled users. The approach works in two stages. In the first stage, the Barrier Walkthrough method is used to classify possible barriers for each disability group on a web page, mapping each barrier to its underlying WCAG checkpoint violations. The authors catalog 19 barrier types for vision-impaired users (including missing alt text, images used as titles, dynamic JavaScript menus, mouse-dependent events, missing skip links, and pages without keyboard shortcuts) and 2 barrier types for hearing-impaired users (missing captions for audio and video content). In the second stage, a quantitative formula (T1) calculates an accessibility score by computing the ratio of actual violated checkpoints to potential checkpoints for each WCAG priority level, weighted by inverse priority (Priority 1 violations weighted most heavily). The formula produces a normalized score between 0 and 1, where higher values indicate more barriers.
Key findings
The methodology was validated against 399 web pages from .com, .gov, and .edu domains that claimed WCAG Level A conformance (displaying the W3C conformance icon). When re-evaluated with EvalAccess 2.0, only 64.41% actually conformed to Level A, and just 19.46% of those were suitable for vision-impaired users according to the proposed methodology. This dramatic gap exists because satisfying all Level A checkpoints only covers 11 of the 37 WCAG checkpoints relevant to vision-impaired users — the remaining 26 checkpoints from Levels AA and AAA are also important for this group but are not required for Level A conformance. The T1 formula showed strong positive correlation (R values of 0.72-0.76 across domains) with the established A3 baseline model, which had been previously validated against ratings from 15 disabled users. For hearing-impaired users, only one page in the entire dataset used prerecorded video with sign language, and only two WCAG checkpoints were relevant, making it difficult to draw conclusions for this group.
Relevance
This research highlights a fundamental limitation of WCAG conformance levels that remains relevant today: conformance to a particular level does not guarantee usability for any specific disability group. A site meeting Level A may still be largely inaccessible to blind users if it fails AA and AAA criteria that are critical for screen reader use. This disability-group-specific perspective is valuable for organizations prioritizing accessibility remediation — rather than treating all WCAG violations equally, they can focus on the checkpoints most impactful for their actual user base. The barrier-to-checkpoint mapping tables provided for vision-impaired and hearing-impaired users serve as practical references for targeted evaluation. The finding that nearly 80% of sites claiming Level A conformance were not suitable for vision-impaired users is a stark reminder that conformance claims and real-world accessibility are very different things.
Tags: accessibility metrics · web accessibility · barrier walkthrough · quantitative evaluation · visual impairment · hearing impairment · disability-specific evaluation
Standards referenced: WCAG 1.0 · WCAG 2.0