Toward Fairness in AI for People with Disabilities: A Research Roadmap
Anhong Guo, Ece Kamar, Jennifer Wortman Vaughan, Hanna Wallach, Meredith Ringel Morris · 2020 · SIGACCESS Accessibility and Computing · doi:10.1145/3386296.3386298
Summary
This position paper from Microsoft Research presents a systematic risk assessment of how major categories of AI systems may fail or discriminate against people with disabilities, proposing a four-point research roadmap for increasing AI fairness. The authors organize their analysis across five AI domains. In computer vision, face recognition may fail for people with Down syndrome, achondroplasia, or conditions affecting facial features; body recognition systems may not work for wheelchair users, people with atypical posture, or those with amputations — with life-threatening implications if self-driving cars fail to detect pedestrians using wheelchairs. In speech systems, ASR fails for people with dysarthria, deaf accents, or who use AAC devices; speech generation needs to accommodate diverse speech rates and support voice banking for people with degenerative conditions; and speaker analysis systems may misclassify the emotional state of people with autism or atypical prosody. In text processing, text analysis may mishandle dyslexic spelling patterns or autistic emotional expression in hiring systems; and translation systems may not support minority disability languages like ASL. In integrative AI, information retrieval systems may amplify stereotypes and advertising algorithms may engage in discriminatory targeting, while conversational agents may not handle atypical spelling, vocabulary differences, or non-text communication modalities. Finally, underlying AI techniques themselves — outlier detection, aggregate metrics, and biased training datasets — create systemic risks that cut across all application domains.
Key findings
The paper identifies five categories of potential harm from unfair AI: quality of service degradation (e.g., voice-activated devices not recognizing atypical speech), harms of allocation (e.g., hiring systems incorrectly filtering out disabled applicants), denigration (e.g., flagging disabled users as invalid outliers), stereotyping (e.g., search results reinforcing disability stereotypes), and over- or under-representation in content. A critical technical insight is that outlier detection algorithms — used pervasively from fraud detection to CAPTCHAs to crowd labor platforms — systematically disadvantage people with disabilities because atypical task completion times, unusual input patterns, and uncommon interaction behaviors get flagged as anomalous. Aggregate performance metrics like accuracy or AUC hide disparate performance across disability subgroups, and objective functions that maximize aggregate metrics will inherently deprioritize small populations. The four-point research roadmap proposes: (1) identifying specific inclusion failure scenarios, (2) systematically testing these hypotheses, (3) creating benchmark datasets that include people with disabilities while addressing ethical concerns about data collection from vulnerable groups, and (4) innovating new modeling, bias mitigation, and error measurement techniques.
Relevance
This paper provides the most comprehensive mapping of AI fairness risks across disability types and AI technology categories published to date, making it an essential reference for anyone developing or deploying AI systems. The structured risk assessment format — matching AI categories against disability constituencies — offers a practical checklist that development teams can use to identify potential fairness issues in their specific products. The observation that people with disabilities face a "long tail" problem in AI (too many distinct conditions, each relatively rare) explains why standard bias mitigation techniques designed for larger demographic groups may be insufficient. For accessibility practitioners, the paper validates concerns about AI-powered tools potentially failing their primary users and provides research-backed language to advocate for inclusive AI development practices. The roadmap's emphasis on creating benchmark datasets that include people with disabilities addresses a root cause: you cannot fix what you cannot measure, and without representative evaluation data, disparate performance goes undetected.
Tags: AI fairness · algorithmic bias · disability · computer vision · speech recognition · text processing · research roadmap · outlier detection · benchmark datasets