Evaluating Web Resources for Disability Access
Murray Rowan, Peter Gregor, David Sloan, Paul Booth · 2000 · Proceedings of the Fourth International ACM Conference on Assistive Technologies (Assets '00) · doi:10.1145/354324.354346
Summary
This paper from the University of Dundee's Digital Media Access Group argues that existing accessibility evaluation methods in 2000 were individually insufficient and proposes a comprehensive meta-methodology that combines multiple evaluation techniques into a structured auditing process. The authors identify key weaknesses of the methods available at the time: automated validation tools like Bobby and the W3C HTML Validation Tool produce results that are overwhelming, overly technical, and often misleading (sometimes finding a resource inaccessible even when it reaches an acceptable level); they can only detect a limited subset of accessibility barriers and require manual inspection for the remainder. Meanwhile, current accessibility guidelines require developers to fully understand each guideline and its rationale, which many find impractical. The proposed methodology was developed during a project auditing UK Higher Education web resources and consists of eight sequential stages: Initial Impressions (team browsing to form overall assessment), Testing with Automatic Validation Tools (Bobby and W3C HTML Validator), Manual Evaluation with Accessibility Guidelines (WCAG checklist including items automated tools cannot check, like colour contrast), General Inspection (testing under degraded conditions — no graphics, no frames, no stylesheets, no scripts, no mouse), Detailed Exploration (visiting every page to assess consistency and structural coherence), Viewing with Browsers and Assistive Technologies (Internet Explorer, Netscape, Opera, Lynx text browser, JAWS screen reader, 80-character Braille display, low and high resolution screens), Usability Evaluations (heuristic evaluation using Nielsen's Ten Usability Heuristics adapted for the web, plus user testing with disabled users), and Recommendations (synthesised into a prioritised recovery plan following Nielsen's severity ratings).
Key findings
The methodology was applied to seven UK Higher Education web sites, with audits producing positive reception from clients. The authors found that usability testing with disabled users was essential alongside technical accessibility evaluation, because sites with high levels of technical accessibility could still have usability problems that prevented people with disabilities from finding information efficiently. The Initial Impressions stage, involving visually impaired users, often yielded hidden accessibility issues and usability problems that other stages missed. The General Inspection stage — testing without graphics, without a mouse, without stylesheets — proved critical for identifying barriers that certain users would face with legacy or assistive browsing technology. The authors observed that no single evaluation method could comprehensively uncover all accessibility problems, validating the need for the multi-method approach. Importantly, they noted that presenting results as prioritised, actionable recommendations in a staged recovery plan was far more effective than raw compliance reports. The methodology led to the proposal of a semi-automatic auditing tool that would guide evaluators through the stages, providing questions, prompts, and further details at each step, while producing standardised output at a level of expertise accessible to developers without deep accessibility knowledge.
Relevance
This paper from 2000 articulates principles that remain central to modern accessibility evaluation practice. The core argument — that automated tools alone are insufficient and must be combined with manual inspection, assistive technology testing, and user testing — is exactly the position held by accessibility professionals today, over two decades later. The multi-method approach anticipated the modern consensus reflected in methodologies like the UK Government's accessibility monitoring approach and the W3C's own evaluation methodology (WCAG-EM). The observation that technically accessible sites can still be unusable highlights the distinction between conformance and usability that continues to challenge the field. For practitioners, the staged methodology remains a practical template: automated scanning first to catch low-hanging fruit, manual guideline review for what tools miss, cross-browser and assistive technology testing for real-world verification, and user testing for validation. The proposal for a guided semi-automatic tool that makes accessibility auditing feasible for non-specialists anticipates modern tools like Axe, WAVE, and Lighthouse that aim to democratise accessibility evaluation while acknowledging the irreplaceable role of human judgement.
Tags: accessibility evaluation · accessibility audit · automated testing · manual testing · heuristic evaluation · WCAG · web accessibility · evaluation methodology · usability testing · assistive technology testing
Standards referenced: WCAG 1.0