Evaluating the Accessibility of Crowdsourcing Tasks on Amazon's Mechanical Turk
Rocío Calvo, Shaun K. Kane, Amy Hurst · 2014 · Proceedings of the 16th International ACM SIGACCESS Conference on Computers & Accessibility (ASSETS) · doi:10.1145/2661334.2661401
Summary
This paper presents a heuristic accessibility evaluation of Amazon Mechanical Turk (MTurk), a crowdsourcing platform where workers complete short tasks (Human Intelligence Tasks, or HITs) for small payments. The authors argue that crowd work platforms offer a potentially valuable employment opportunity for people with disabilities, since they allow working from home on flexible schedules — in 2012, 24.5% of employed US citizens with disabilities completed some work from home. However, for this potential to be realised, the platforms themselves must be accessible. The researchers evaluated four core MTurk web pages (login, search, HIT list, account dashboard) plus eleven popular HITs from the ten most popular requesters, using WCAG 2.0 at the AA conformance level. The evaluation was conducted on Windows 7 using Firefox 23 and Chrome, with automated tools including AChecker, WAVE, and W3C HTML and CSS validators, followed by manual expert review. The study followed the Website Accessibility Conformance Evaluation Methodology 1.0. The authors focused specifically on the worker experience, excluding the requester interface, and selected HITs that would be accessible to new workers — tasks involving cleaning and verifying information, transcribing audio, and extracting information from images.
Key findings
The evaluation uncovered accessibility problems across both the MTurk platform itself and individual HITs. On the MTurk site pages, common issues included missing or inadequate form labels, poor colour contrast, missing alternative text for images, and inconsistent navigation. The HITs presented their own accessibility barriers — many were built by requesters with no apparent awareness of accessibility requirements, featuring inaccessible custom interfaces, unlabelled form controls, and content that required visual interpretation without alternatives. The study identified that even when the core MTurk platform pages were partially accessible, the HITs themselves (which are created by third-party requesters) introduced a second layer of accessibility barriers outside the platform operator's direct control. This dual-layer accessibility challenge — platform plus task content — is characteristic of crowdsourcing systems and makes them harder to remediate than conventional websites. The authors note that while prior research had examined MTurk usability for workers with low technology literacy and older adults, no previous study had specifically analysed the platform's accessibility for people with disabilities.
Relevance
This paper highlights an important but often overlooked accessibility context: employment platforms. As gig economy and remote work opportunities have grown significantly since 2014, the accessibility of crowd work and freelancing platforms remains a barrier to economic participation for people with disabilities. The dual-layer problem identified here — where a platform may address its own accessibility but has limited control over third-party content created by requesters — applies broadly to user-generated content platforms, marketplace sites, and modern no-code/low-code tools. For accessibility practitioners, the study demonstrates the value of applying WCAG-based heuristic evaluation to non-traditional web applications and illustrates how employment discrimination can be perpetuated through inaccessible technology even in contexts designed to lower barriers to work. The findings also raise questions about platform responsibility: should crowdsourcing sites enforce accessibility standards on tasks posted by requesters, similar to how they enforce other content policies?
Tags: crowdsourcing · web accessibility · accessibility evaluation · employment · WCAG · heuristic evaluation · disability employment
Standards referenced: WCAG 2.0