Fairness Issues in AI Systems that Augment Sensory Abilities
Leah Findlater, Steven Goodman, Yuhang Zhao, Shiri Azenkot, Margot Hanley · 2020 · SIGACCESS Accessibility and Computing · doi:10.1145/3386296.3386304
Summary
This paper examines the unique fairness challenges that arise when AI systems are used to augment sensory abilities for people with disabilities — a context distinct from other AI applications because these systems provide information that is already available to non-disabled people. The authors identify three interconnected problem areas. First, data and model inaccessibility: the training data and internal representations used by AI sensing systems are inherently inaccessible to their target users. A blind person cannot visually inspect what an image classifier sees, and a deaf person cannot listen to what a sound classifier hears, making it difficult to understand system limitations, build appropriate trust, verify outputs, or participate in model personalization. Visual explanations commonly used for AI transparency (like highlighting salient image regions) are themselves inaccessible. Second, the complex decision-making involved in what sensory information to convey: describing a visual or auditory scene requires countless choices about what to include, what labels to use, and how to frame information. Facial recognition for blind users raises questions about whether to identify strangers, convey demographic attributes, or only recognize known contacts. Sound awareness systems must decide which sounds are important enough to alert the user about. These seemingly neutral design choices embed values and biases that profoundly shape how users with sensory disabilities perceive the world around them.
Key findings
The third major challenge is privacy — both for the primary user and for bystanders. Always-on sensing via wearable cameras or microphones creates risks of inadvertently capturing sensitive information (the VizWiz-Priv dataset found blind users' photos contained prescription labels, credit cards, and pregnancy tests). Bystanders face surveillance concerns, though research suggests people are more accepting of head-mounted cameras when they know the device serves an assistive purpose. The paper raises the novel concept of "assistive use" legal exceptions analogous to service animal accommodations under the ADA, which could allow assistive sensing in contexts where recording is otherwise prohibited. A critical insight is the layered nature of decision-making in these systems: who decides what labels to use when labeling training data, who decides what datasets and modeling approaches to employ, who decides how to present model output — and today there is little visibility into any of these decisions. The authors advocate for approaches like "Datasheets for Datasets" to improve transparency across the entire AI pipeline, not just at the model level. The paper also highlights that users may place too much trust in AI sensing results precisely because they cannot independently verify accuracy, creating a dangerous feedback loop.
Relevance
This paper complements White's broader analysis of AI fairness and disability by focusing specifically on sensory augmentation — the AI application area most directly relevant to assistive technology practitioners. For teams building AI-powered accessibility tools (image description, object recognition, sound awareness, scene understanding), it provides a practical framework for identifying fairness risks at every stage of the pipeline. The privacy analysis is particularly timely as wearable AI sensing devices become more common, raising questions that organizations deploying these tools must address proactively. The concept of equal access as a fairness benchmark — that AI should provide disabled users with information already available to non-disabled people — offers a clear ethical standard while simultaneously revealing how complex achieving that standard actually is. For policy work, the "assistive use" exception concept opens important conversations about balancing privacy regulation with disability rights.
Tags: AI fairness · sensory augmentation · visual impairment · deaf and hard of hearing · privacy · object recognition · sound awareness · algorithmic bias · ethics
Standards referenced: ADA