Development and Trial of an Educational Tool to Support the Accessibility Evaluation Process
Christopher Bailey, Elaine Pearson · 2011 · Proceedings of the International Cross-Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/1969289.1969293
Summary
This paper describes the design, development, and first trial of the Accessibility Evaluation Assistant (AEA), a web-based knowledge management tool created at Teesside University to support novice auditors in conducting accessibility evaluations. The AEA addresses a key gap: while accessibility is part of required ACM computing curricula (CS2008, IS2010, IT2008), many institutions offer little space for it, students lack knowledge of guidelines and assistive technologies, struggle to interpret automated tool results, have limited access to disabled end users, and have little time for accessibility within their assignments. The AEA contains 48 accessibility checks drawn from guidelines, established evaluation methodologies, and practitioner experience, organised into five categories: Design (11 checks on visual presentation), User (15 manual interaction checks), Structural (11 semantic markup checks), Technical (5 code validation checks), and Global (6 site-wide checks). The tool supports three evaluation contexts: Check Categories (comprehensive evaluation using all 48 checks), User Group (prioritising checks for 10 disability groups: dyslexia, learning disabilities, low vision, screen reader users, motor disabilities, English as foreign language, older users, deaf/hard of hearing, colour vision deficiencies, seizure disorders), and Site Features (filtering by forms, images, CSS, links, multimedia, semantic HTML, tables). Each check includes a structured walkthrough adapted from the Barrier Walkthrough method, with the accessibility principle, affected user groups, barrier description, step-by-step testing procedure using the Web Accessibility Toolbar, verification steps, and expert video demonstrations.
Key findings
The AEA was trialled with 38 final-year undergraduate computing students (all novices) evaluating four live websites (Vancouver 2010 Olympics, CNN, FA Premier League, WalMart). Students used the AEA's Check Categories function to evaluate homepage accessibility and the User Group function with disability personas to assess barriers for specific users. Reliability (agreement between novice evaluators) was promising: across all four sites, 82% of checks had consensus agreement of 50% or more, 68% had 60% or more agreement, and 53% had 70% or more — below the W3C's 80% threshold for knowledgeable evaluators but reasonable for first-time novices. Validity (matching expert evaluation) was 60% overall, with individual sites ranging from 56% (Premier League) to 65% (Vancouver Olympics). The analysis revealed that novices sometimes interpreted checks too literally (e.g., treating flags as "supplementary images" for internationalisation), struggled with subjective judgements (colour contrast, text readability), and confused related concepts (Use of Colour vs Colour Contrast). The "evaluator effect" was evident — novices judged more strictly than the expert on some checks (marking Flash banners as "Not Met" while the expert considered them "Met" because users had playback control). Students found the persona-based User Group evaluation particularly valuable for connecting abstract accessibility principles to real people, with comments like "it really shows that websites will have a lot of user groups" and "this was the first time I have completed an accessibility evaluation and I found it opened my eyes to a lot of issues."
Relevance
This paper is the foundational companion to Bailey & Pearson's 2012 comparative study (also in the database), providing the detailed design rationale and first trial results for the AEA tool. For accessibility educators, two contributions stand out. First, the categorisation of 48 checks into five intuitive groups (Design, User, Structural, Technical, Global) provides a pedagogically sound alternative to WCAG's principle-based organisation that novices find overwhelming. Second, the User Group feature with disability personas demonstrates a powerful teaching approach: by evaluating a website through the lens of specific users (a screen reader user, a person with dyslexia, an older user with arthritis and cataracts), students develop empathy and practical understanding that abstract guideline review cannot provide. The detailed analysis of which checks novices can verify accurately and which require deeper expertise is directly useful for organisations deciding which accessibility checks to delegate to non-specialist team members versus reserving for expert auditors. The finding that novices judge more strictly than experts on some checks (the evaluator effect) is a useful caution for interpreting accessibility reports from less experienced evaluators.
Tags: accessibility evaluation · accessibility education · WCAG compliance · evaluation tools · novice auditors · structured walkthrough · accessibility training
Standards referenced: WCAG 1.0 · WCAG 2.0