An educational tool to support the accessibility evaluation process
Christopher Bailey, Elaine Pearson · 2010 · Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/1805986.1806003
Summary
This paper describes the Accessibility Evaluation Assistant (AEA), a web-based knowledge management tool designed to help novice auditors — primarily computing students — conduct meaningful accessibility evaluations without expert knowledge or access to disabled user groups. The tool addresses a persistent gap in accessibility education: students have limited exposure to accessibility during their training, limited time to dedicate to it within coursework, and no access to expert reviewers or disabled users for testing. Rather than presenting the full complexity of WCAG conformance review, the AEA tailors the evaluation to context by allowing auditors to specify a target user group (from ten options including screen reader users, people with dyslexia, older users, deaf/hard of hearing, etc.) and the content features of the website being evaluated (forms, data tables, video, Flash, etc.). The tool then prioritizes relevant checks using the Barrier Walkthrough method, categorizing issues as Critical (rendering content totally inaccessible to the specified group), Problem (causing noticeable barriers), or General (basic accessible design principles). Checks are organized into four categories: design, user, structural, and technical. For each check, the auditor receives the principle being tested, which user groups are affected, the nature of the barrier, and step-by-step checking procedures.
Key findings
The AEA evolved from a paper-based workbook used with postgraduate students, who found it useful for understanding accessibility but too bulky and time-consuming. The digital tool addresses this by filtering checks to only those relevant to the specified context, eliminating redundancy. A key design insight is the re-prioritization of WCAG checkpoints relative to specific user groups: while WCAG assigns fixed priority levels (A, AA, AAA), the same issue may have vastly different impact depending on the target audience. For example, color-only meaning is a critical issue for screen reader users but may be a lower priority for users with motor disabilities. The tool also educates auditors about how addressing one user group's needs can benefit others, while highlighting exceptions where this overlap does not hold. The authors suggest the concept could be reversed — instead of supporting evaluation, the tool could support the design and development process by reporting which accessibility issues to prioritize before a site is built. The tool targets not only students but also small and medium businesses and charitable organizations with limited resources for professional audits.
Relevance
This paper addresses the critical intersection of accessibility education and practical evaluation methodology. The persistent shortage of accessibility knowledge among IT graduates remains a significant barrier to web accessibility improvement, and the AEA's approach of scaffolding the evaluation process for novices remains relevant. The context-based filtering concept — tailoring checks to specific user groups and content types rather than presenting all WCAG criteria — is a pedagogically sound approach that helps novices build understanding incrementally rather than being overwhelmed. For accessibility educators, the four-category check structure (design, user, structural, technical) provides a useful organizing framework that maps to different skill sets and evaluation methods. The observation that WCAG's fixed priority levels do not reflect the actual impact on specific user groups anticipates ongoing discussions about the limitations of one-size-fits-all conformance levels. The Barrier Walkthrough method's integration into an educational tool demonstrates how research evaluation methodologies can be made practical for non-experts.
Tags: accessibility evaluation · accessibility education · WCAG compliance · barrier walkthrough · accessibility tools · user-centered evaluation
Standards referenced: WCAG 1.0 · WCAG 2.0