Web Accessibility Evaluation with the Crowd: Using Glance to Rapidly Code User Testing Video
Mitchell Gordon · 2014 · Proceedings of the 16th International ACM SIGACCESS Conference on Computers & Accessibility (ASSETS) · doi:10.1145/2661334.2661412
Summary
This paper proposes using Glance, a crowd-powered video coding tool, to accelerate the analysis of user accessibility testing videos. User testing with people with disabilities is a recommended component of web accessibility evaluation (as outlined in the W3C Website Accessibility Conformance Evaluation Methodology), but analysing the resulting video recordings is time-consuming, expensive, and requires trained evaluators. Traditional behavioural video coding — the systematic process of labelling actions, emotions, and events in video — is a well-established social science method but is notoriously slow, often taking 5-10 times the duration of the video itself. Glance addresses this by distributing video coding to crowd workers: when a researcher poses a usability question (e.g., "did the user encounter unpredictable content?"), Glance splits the video into small clips and sends them to multiple crowd workers who label events in parallel. The judgements are then merged and displayed, allowing researchers to rapidly identify accessibility problems. The paper positions Glance as a complement to both direct user testing (which provides rich qualitative data but is expensive and time-consuming) and automated testing (which is fast and cheap but cannot detect subtle behavioural signals like confusion, frustration, or workarounds). The author gives the example of coding for WCAG 2.0 guideline conformance by asking crowd workers to identify specific accessibility-related behaviours in user test videos.
Key findings
The paper demonstrates a novel application of crowdsourced video coding to accessibility evaluation, arguing that crowd workers can identify behavioural events — such as user confusion, encounters with unpredictable content, or navigation difficulties — that automated tools cannot detect. By parallelising the coding task across multiple workers examining short video clips simultaneously, Glance can dramatically reduce the time needed to extract accessibility insights from user test recordings compared to having a single researcher watch and code entire videos sequentially. The approach bridges a gap between two established evaluation methods: automated accessibility testing (fast but limited to code-level checks) and expert user testing analysis (thorough but slow and expensive). The paper notes that involving people with disabilities directly in testing is valuable but requires time and training that developers often lack, and that remote testing may yield less rich data than in-person studies due to reliance on self-reports. Crowd-based video coding offers a middle path where the testing itself involves real users with disabilities, but the analysis burden is distributed.
Relevance
This work addresses a genuine bottleneck in accessibility practice: the analysis phase of user testing. Many organisations conduct accessibility user testing but struggle with the time and expertise required to systematically review the results, particularly video recordings. The crowdsourcing approach raises interesting questions about the skills needed to identify accessibility barriers — can untrained crowd workers reliably detect when a blind user is struggling with navigation, or when a motor-impaired user encounters an interaction barrier? The paper does not fully resolve this question but opens the door to scalable accessibility evaluation methods. For accessibility professionals, the concept of using structured crowd analysis of testing videos is relevant to modern practices, particularly as remote accessibility testing has become more common. The underlying principle — that behavioural signals in user testing reveal accessibility problems that neither automated tools nor expert code review can find — reinforces the importance of including real users with disabilities in the evaluation process, as WCAG and the W3C evaluation methodology recommend.
Tags: crowdsourcing · accessibility evaluation · usability testing · video coding · web accessibility · WCAG · user testing
Standards referenced: WCAG 2.0