Disability-first Dataset Creation: Lessons from Constructing a Dataset for Teachable Object Recognition with Blind and Low Vision Data Collectors
Lida Theodorou, Daniela Massiceti, Luisa Zintgraf, Simone Stumpf, Cecily Morrison, Edward Cutrell, Matthew Tobias Harris, Katja Hofmann · 2021 · Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '21) · doi:10.1145/3441852.3471225
Summary
This paper presents a disability-first approach to constructing a machine learning dataset for teachable object recognition, developed through the ORBIT (Objects that Recognize Blind Individuals in Their environment) project. The authors argue that while AI for accessibility is a growing field, most datasets are created by sighted people collecting data about disability-related scenarios rather than by disabled people themselves. This creates datasets that fail to capture the real-world conditions under which assistive AI tools will actually be used. The ORBIT project inverts this paradigm by recruiting blind and low vision (BLV) people as data collectors who film their own personal objects in their own environments. The paper details the iterative development of the data collection procedure across a pilot study and two main phases. In the pilot, eight BLV participants tested filming techniques, leading the team to develop a rotate-zoom method using hands as anchor points for framing objects. Phase 1 (May-July 2020) recruited 48 UK-based collectors who each filmed 10 objects, producing 3,448 videos of 390 objects, with participants compensated with £50 vouchers. Phase 2 (October 2020-January 2021) expanded globally to 52 collectors across English-speaking countries, simplifying requirements to 5 objects based on ML-driven analysis showing that accuracy plateaus at around 4 training videos per object. The team developed a bespoke iOS app with VoiceOver support, audio and haptic feedback, and tic-based timing cues to make the collection process accessible. Throughout the process, the researchers balanced three key constraints: engaging and supporting the BLV community, ensuring data quality for ML model training, and minimizing the effort required from collectors. ML-driven data evaluation at intermediate stages helped the team make evidence-based decisions about protocol simplifications.
Key findings
The research produced several important findings about disability-first dataset creation. First, switching from photos to videos proved essential for BLV collectors, as videos provide more opportunities to capture good images and add temporal information that makes object recognition more robust. ML analysis revealed that model accuracy plateaus at approximately 4 training videos per object, allowing the team to reduce requirements from 6 to 5 videos without sacrificing quality. Test videos using the zoom-out technique outperformed pan videos for evaluation purposes. The paper identifies significant infrastructure challenges: the iOS-only app excluded Android users and those without high-spec devices or stable internet, potentially marginalising collectors from lower socio-economic backgrounds or non-Western countries. Privacy emerged as a critical concern, with collectors inadvertently capturing personally identifying information such as bank cards or reflective "selfies" on shiny surfaces, requiring manual review of all videos by researchers. The authors distill their experience into eight orienting questions organised around three themes: engaging target communities (recruitment strategies, community benefit, supporting collectors globally), data collection procedures (balancing ML needs with collector demands, using ML to support collection), and accessible infrastructure (coupling procedure with infrastructure, privacy and confidentiality). These questions provide a practical framework for other researchers undertaking disability-first data collection efforts.
Relevance
This paper makes a compelling case for centering disabled people as active participants rather than passive subjects in AI dataset creation. For accessibility practitioners, it highlights that datasets built without disabled contributors may not reflect the real conditions under which assistive technology operates, leading to models that underperform in practice. The ORBIT dataset, resulting from this work, has been made publicly available to advance teachable object recognition research. The eight orienting questions offer a reusable framework for any team building AI tools for accessibility. Particularly relevant is the tension between data quality requirements and the effort burden placed on disabled participants — a balance that requires ongoing ML-driven evaluation rather than fixed assumptions. The privacy challenges identified are also critical for practitioners: automated PII detection in accessibility-related data collection remains an unsolved problem. The paper's emphasis on interdisciplinary collaboration between HCI, accessibility, and ML researchers provides a model for how accessible AI development should be structured organisationally.
Tags: disability-first design · dataset creation · teachable object recognition · blind and low vision · machine learning · data collection · participatory research · accessible mobile applications
Standards referenced: Apple Accessibility Guidelines · VoiceOver