What the Disability Community Can Teach Us About Interactive Crowdsourcing
Jeffrey P. Bigham, Richard E. Ladner · 2011 · Interactions · doi:10.1145/1978822.1978838
Summary
This short forum article argues that the disability community has been practicing interactive crowdsourcing long before the term became mainstream in computing, and that mainstream crowdsourcing systems have much to learn from these experiences. The authors trace how people with disabilities have historically relied on human assistance — blind people finding readers for mail, deaf people recruiting sign language interpreters from local communities — and show how these informal arrangements evolved into formalized crowdsourcing services. Examples include Video Relay Service (VRS) and Video Remote Interpreting (VRI) for deaf users, reading services for blind people, and personal assistance agencies. The article then describes how modern technology, particularly smartphones and platforms like Amazon Mechanical Turk, has enabled new forms of interactive crowdsourcing for accessibility. The authors highlight their own VizWiz application, which lets blind users take a photo, speak a question, and receive answers from crowd workers in under 30 seconds for less than 7 cents. The article positions the disability community as early adopters and innovators in interactive crowdsourcing who can inform the design of mainstream systems.
Key findings
The authors identify six key design considerations that the disability community's experience with crowdsourcing highlights for mainstream systems: (1) Confidentiality and anonymity — disability services have long maintained strict codes of ethics around worker confidentiality, which anonymous crowd platforms must also address; (2) Worker competence — sign language interpreters are prescreened for certification, raising questions about how to verify competence in open crowd platforms; (3) Latency — different tasks demand different response times, and systems must help users understand expected latencies for various human computation sources; (4) Accuracy — workers can provide incorrect answers, and systems need mechanisms to ensure quality and help users assess answer reliability; (5) Feedback to users — users need transparency about human computation happening on their behalf to make informed decisions; (6) Sources of computation — users will increasingly need to choose between human and artificial intelligence sources that differ in cost, availability, accuracy, latency, and privacy. A field deployment of VizWiz with 11 blind users revealed that questions went far beyond reading text to include spatial and contextual queries, and users wanted follow-up question capabilities.
Relevance
This article provides a valuable conceptual framework for understanding how crowdsourcing intersects with accessibility. Written in 2011, many of its predictions have proven prescient — the tension between human and AI computation sources is now central to modern assistive technology (e.g., Be My Eyes transitioning from volunteer sighted helpers to AI-powered descriptions). The six design considerations remain directly applicable to any system that uses human computation for accessibility: privacy when sharing personal visual information, quality control for high-stakes assistance, and helping users navigate between AI and human help. For accessibility practitioners, the article's core insight is that disability services have already solved many problems that mainstream crowdsourcing is only beginning to encounter, particularly around worker ethics, user privacy, and quality assurance in high-stakes human assistance scenarios.
Tags: crowdsourcing · assistive technology · disability community · visual question answering · sign language interpreting · privacy