A comparative test of web accessibility evaluation methods
Giorgio Brajnik · 2008 · Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility (Assets '08) · doi:10.1145/1414471.1414494
Summary
This paper presents a controlled laboratory experiment comparing two web accessibility evaluation methods: conformance review (CR) and barrier walkthrough (BW). Conformance review is the traditional approach where evaluators check whether a page satisfies a checklist of criteria derived from standards like WCAG — it is analytic, based on evaluators' opinions, and produces failure modes in the form of violated checkpoints. Barrier walkthrough is a method proposed by the author that explicitly considers context of use: evaluators assess predefined barriers linked to specific user categories (e.g., blind users of screen readers), user activities, and situation patterns, then assign severity ratings based on impact (degree to which a user goal cannot be achieved) and frequency (how often the barrier occurs). The experiment involved 16 user-centered web design students who had received about 15 hours of training in accessibility, guidelines, CR, and BW. After excluding 4 subjects who completed only one task, 12 novice evaluators each performed two evaluations (one CR, one BW) on pages from two Italian university websites, with method order and website assignment counterbalanced. The study measured five quality factors: effectiveness (validity, correctness, sensitivity), efficiency (completion time), reliability (consistency across evaluators), and usefulness. A judge independently identified all barriers and violated requirements on each page and rated their severity to establish ground truth.
Key findings
Barrier walkthrough produced significantly higher correctness than conformance review (M_BW = 0.72 vs M_CR = 0.53; paired t-test t = 2.98, df = 11, p < 0.0125, effect size d = 0.77), meaning that a larger proportion of problems identified by BW evaluators were true problems. This corresponds to a correctness improvement of 9% to 60% when using BW. BW evaluators also identified significantly more correctly judged barriers/checkpoints (M_BW = 13.7 vs M_CR = 7.33; t = 4.65, df = 11, p < 0.001, effect size d = 0.99), representing a 45% to 127% increase. No significant difference was found for sensitivity (the proportion of true problems that were reported), though BW values were systematically higher. Reliability was slightly lower for BW (0.62 vs 0.69 for CR), attributable to the additional subjective severity assignment step, though the 10% reduction was considered acceptably small. Completion time showed no significant method effect, though task order significantly affected time (p = 0.011), with subjects paying less attention to the second evaluation task. A notable finding was a significant positive correlation between correctness and sensitivity for BW (r = 0.77, p = 0.0031) but not for CR, suggesting that BW evaluators who are more correct also tend to be more thorough.
Relevance
This study provides the first rigorous experimental comparison of accessibility evaluation methods, filling an important gap — while usability evaluation methods had been compared extensively, accessibility methods had not. The finding that barrier walkthrough produces markedly more correct results than conformance review has significant practical implications: organizations investing in accessibility audits should consider context-aware methods rather than relying solely on checklist-based conformance testing. The barrier walkthrough's explicit consideration of user categories, assistive technologies, and usage scenarios forces evaluators to think about real-world impact rather than abstract guideline violations, which appears to improve the quality of their judgments even among novice evaluators. For accessibility practitioners, the quality framework itself (correctness, sensitivity, F-measure, reliability, efficiency) is valuable for selecting and benchmarking evaluation methods. The paper also highlights important limitations of conformance review: evaluators may identify violations that are not actually accessibility problems in practice, and they struggle to distinguish important from trivial issues without the contextual framing that BW provides.
Tags: accessibility evaluation · conformance testing · barrier walkthrough · web accessibility · WCAG · evaluation methods · accessibility testing · quality assessment
Standards referenced: WCAG