Evaluation of the Effectiveness of a Tool to Support Novice Auditors
Christopher Bailey, Elaine Pearson · 2012 · Proceedings of the International Cross-Disciplinary Conference on Web Accessibility (W4A) · doi:10.1145/2207016.2207044
Summary
This paper evaluates the Accessibility Evaluation Assistant (AEA), a web-based knowledge management tool designed to help novice auditors conduct accessibility evaluations, by comparing it against standard WCAG 2.0 conformance review. The AEA was developed at Teesside University and incorporated into a final-year undergraduate computing module on Accessibility and Adaptive Technologies. It contains 48 accessibility checks based on established principles, organised into five categories: Design Checks (11 checks examining visual presentation), User Checks (15 practical checks requiring manual interaction testing), Structural Checks (11 checks on semantic markup), Technical Checks (5 checks on code validation), and Global Checks (6 site-wide checks). The tool offers three evaluation contexts: by check category, by user group (10 disability groups including screen reader users, older web users, etc.), or by site feature (forms, images, CSS, links, etc.). Each check includes a structured walkthrough with the accessibility principle title, a summary, an explanation of importance and affected user groups, step-by-step testing instructions using the Web Accessibility Toolbar, verification steps, and expert video demonstrations. The experiment involved 37 final-year undergraduate students (all new to accessibility evaluation) divided into four groups, evaluating two live websites (Harley Davidson UK and Sunsail UK) using both AEA and WCAG 2.0 methods across two weeks, with 15 matched checks selected from each method.
Key findings
The AEA outperformed WCAG 2.0 conformance review on both key quality metrics when used by novices. Overall validity (the extent to which novice decisions matched expert evaluations) was 68.5% for AEA versus 51.75% for WCAG 2.0 — a 16.75 percentage point advantage. In all four experimental conditions where AEA was used, validity was higher than in any WCAG condition. Overall reliability (agreement between novice evaluators) was 71.25% for AEA versus 64.75% for WCAG 2.0 — a 6.5 percentage point advantage. While neither method reached the W3C's recommended 80% reliability threshold for knowledgeable evaluators, the results are promising given that participants were complete novices performing their first evaluations. Qualitative feedback was strongly differentiated: students praised the AEA for ease of use/simplicity (27 mentions), clear terminology (22), guiding explanations (15), categorisation of checks (14), and helpful videos (9). By contrast, WCAG 2.0 was overwhelmingly criticised as confusing/difficult to understand (38 mentions), complex/hard to use (16), too long/detailed (12), and requiring more knowledge/experience (7). Students explicitly noted that WCAG "does not explain the steps taken to actually perform the checks" and that "the sentences are extremely long and I kept forgetting what I was checking for." However, students recognised WCAG's value for regulatory purposes (11 mentions) and its detailed explanations (14 mentions).
Relevance
This research directly addresses one of the biggest practical barriers to accessibility adoption: the expertise gap. WCAG 2.0, while authoritative and comprehensive, is written for expert evaluators and is demonstrably overwhelming for novices — the very people most organisations rely on to implement accessibility. The AEA demonstrates that structured, scaffolded evaluation tools with plain-language instructions, step-by-step procedures, video demonstrations, and disability-group context can significantly improve both the accuracy and experience of novice evaluators. For accessibility trainers and educators, the key implication is that teaching WCAG conformance review directly to beginners may be counterproductive — a guided, heuristic-based approach that builds foundational understanding before progressing to full WCAG review is more effective. For organisations building accessibility evaluation into their development processes, this research supports investing in simplified, contextualised evaluation checklists for non-specialist team members rather than expecting developers to interpret raw WCAG documentation. The finding that novices consistently struggled with WCAG's technical language and lengthy success criteria descriptions remains relevant and has influenced the development of more accessible guidance like the W3C's own "Easy Checks" resource.
Tags: accessibility evaluation · accessibility education · WCAG compliance · evaluation tools · accessibility training · novice auditors · structured walkthrough
Standards referenced: WCAG 2.0