Guideline Aggregation: Web Accessibility Evaluation for Older Users
Giorgio Brajnik, Yeliz Yesilada, Simon Harper · 2009 · Proceedings of the 2009 International Cross-Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/1535654.1535686
Summary
This paper proposes using set theory to create an aggregated accessibility evaluation category for older users by combining barrier types identified for motor impaired and low vision users within the Barrier Walkthrough (BW) method. Traditional validation approaches treat accessibility as a binary pass/fail against all guidelines simultaneously, without isolating the type or severity of impairment. The BW method improves on this by grouping predefined accessibility barriers by user category and grading severity on a 1-3 scale based on impact and persistence. The authors define 'older users' as people who face barriers common to both low vision and motor impaired categories, using a simple set union of the barrier types from these two primitive categories. To evaluate this aggregation approach, they conducted a study with 68 judges (19 experts recruited from ASSETS 2008 attendees and 49 non-expert students) who assessed four web pages — Facebook, IMDB, a quilting site, and a local restaurant — against barrier checklists for both low vision and motor impaired user categories. The study used reproducibility (coefficient of variation) and agreement (intraclass correlation coefficient) to measure reliability, and accuracy, sensitivity, and F-measure to assess correctness against a ground truth established by majority expert opinion.
Key findings
The study identified 27 true barrier types for older users out of 61 unique barrier types assessed, with 17 specific to motor impaired users and 25 to low vision users, with meaningful overlap. Common barriers included missing skip links, inflexible page layouts, links/buttons too small, long URIs, mouse-dependent events, page size limits, new windows opening unexpectedly, no page headings, scrolling issues, and text that cannot be resized. The aggregated older users category produced reliable data: mean reproducibility was 0.31 for older users versus 0.26 for low vision and 0.36 for motor impaired, and page rankings were preserved when moving from primitive to aggregated categories. Expert judges completed evaluations in significantly less time (107 vs 299 minutes, p < 0.0001), felt more productive and confident, and produced more reproducible ratings (M=0.56 vs M=0.38 for non-experts). The overall error rate across 8,784 ratings was 15.7%, with experts at 12.5% and non-experts at 18.1%. Non-experts missed three barrier types entirely: forms with no label tags, moving content, and no stylesheet support. A key limitation emerged: the binary nature of barrier-to-category assignment (a barrier either applies or does not) fails to capture severity gradations — for example, 'links too close together' is very relevant for someone with mild motor impairment using a pointing device but less so for a keyboard-only user.
Relevance
This paper addresses a real gap in accessibility practice: how to evaluate for users who experience multiple overlapping impairments, as is typical of aging populations. With approximately 50% of people over 65 experiencing some form of disability, and the global older population growing rapidly, the question of how accessibility guidelines apply to compound impairments is increasingly urgent. The finding that page rankings and important distinctions are preserved through aggregation validates a practical approach — organizations can evaluate for primitive user categories and derive meaningful conclusions about older users without conducting separate assessments. The expert-versus-non-expert comparison has direct implications for organizations building accessibility evaluation capacity: non-experts took nearly three times longer, were less confident, missed entire barrier categories, and gave less reproducible ratings, suggesting that investment in evaluator training substantially improves assessment quality. The paper's proposal for extending the BW with severity levels of disability to enable 'Personalised Validation and Repair' anticipated the growing interest in personalized accessibility that has since become a significant research direction.
Tags: older users · accessibility evaluation · barrier walkthrough · low vision · motor impairment · multiple disabilities · expert evaluation · aging · personalized accessibility
Standards referenced: WCAG 1.0 · ATAG 1.0 · UAAG 1.0