Toward a taxonomy of negative outcomes from the use of AI-driven systems for people with disabilities
Krishna Venkatasubramanian, Haven Hardie, Tina-Marie Ranalli · 2025 · ASSETS 2025: 27th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663547.3746359
Summary
This paper presents the first systematic taxonomy of how AI-driven systems create negative outcomes specifically for people with disabilities. The authors searched eight publicly available AI incident databases — including AIAAIC, AIID, OECD AI Incident Monitor, and the Database of AI Litigation — which collectively contain over 17,700 entries. From these, they extracted 79 documented incidents where AI systems produced negative outcomes for people with disabilities, using keyword searches across disability-related terms and applying careful inclusion/exclusion criteria. The methodology employed reflective thematic analysis (RTA) following Braun and Clark's six-step approach, resulting in a taxonomy of nine broad categories of negative outcomes. The researchers deliberately used AI incident databases rather than interviews or surveys because these databases ground findings in real, documented, and substantiated cases rather than speculation. They also provide a broad overview of their dataset across four dimensions: types of AI systems involved (11 types identified), classes of disabilities affected (9 classes), sectors where systems were deployed (23 sectors), and eventual actions taken (or not taken) against the offending systems. The most common type of AI system causing harm was predictive AI (34 incidents), followed by generative AI (17), biometrics (8), and web accessibility tools (7). The disabilities most frequently affected were broadly all disabilities (19), cognitive disability (16), vision impairment (15), and motor disability (15). The most common sector was welfare and social services (13 incidents), followed by education (11), social media (9), and technology including chatbots (7) and web tools (7). A striking finding was that in 22 of the 79 incidents, no action was taken to update the offending AI system, and in 20 more the outcome was unclear — meaning a majority (65%) of harmful AI systems remained operational despite documented negative consequences.
Key findings
The taxonomy identifies nine categories of negative outcomes, ordered by frequency: 1. **Denying vital resources** (37 incidents) — the most common harm, spanning welfare/social services (12), technology access (10), employment (10), education (2), housing (2), and insurance (1). Examples include Medicaid resource allocation models found arbitrary and non-transparent in Idaho, Oregon, and multiple other US states; HireVue screening out candidates with disabilities via facial analysis; and accessibility overlay companies making false claims about making websites screen-reader-friendly. 2. **Falsely accusing of serious offenses** (14 incidents) — including academic dishonesty (6 incidents where proctoring systems flagged autistic students' stimming, physical disabilities' inability to sit still, and atypical eye movements as cheating), fraud (4 incidents where systems misinterpreted disability-related behaviors as suspicious), and security threats (4 incidents including a terrorism prediction tool considering autism a risk factor). 3. **Disrespecting privacy** (7 incidents) — systems gathering excessive disability-related information without consent, including Austria's employment system conducting "invasive profiling" violating GDPR, and Facebook's People You May Know feature revealing patients to each other at a psychiatrist's office. 4. **Suppressing voices** (6 incidents) — including TikTok's content moderation suppressing disabled creators' content, Twitter's auto-cropping tool cropping out people with disabilities, and OpenAI's Whisper hallucinating violent language during speech pauses common with certain speech impairments. 5. **Providing skewed content** (5 incidents) — the Character.AI chatbot showing inappropriate content including self-harm themes to a teenager with autism, leading to tragic consequences. 6. **Causing distress** (5 incidents) — physical harm from a self-driving bus hitting a visually impaired Paralympic athlete, a smart gate crushing a wheelchair user, and an understaffed senior facility (due to AI scheduling) where a resident with dementia died after falling unattended. 7. **Fostering prejudice** (4 incidents) — Google's Perspective tool assigning higher toxicity scores to disability-related statements, and chatbots generating discriminatory content. 8. **Impersonating disabled people** (4 incidents) — including AI-generated fake images of people with disabilities used for fraud and engagement farming on social media. 9. **Limiting access** (2 incidents) — delivery robots blocking wheelchair users on sidewalks, and autonomous taxis being inaccessible to blind users who relied on driver interaction. The paper proposes five research opportunities: seamless recourse mechanisms for AI decisions, a "nothing about us without us" approach to AI design, guidelines for deploying AI systems affecting disabled people, auditing and standards frameworks, and a CVE-like reference system for tracking disability-focused AI harms.
Relevance
This paper fills a critical gap in AI fairness research by centering disability — a population frequently overlooked in algorithmic bias discussions that typically focus on race and gender. The taxonomy provides a practical framework for categorizing AI harms against disabled people that can be used by policymakers, AI developers, and disability advocates. For accessibility practitioners, several findings are directly actionable. The documentation of accessibility overlay companies among the harmful AI systems (7 incidents of web accessibility tools making false claims) validates long-standing community concerns about these products. The proctoring system findings (6 incidents falsely accusing disabled students) provide concrete evidence for organizations considering AI-based monitoring tools. The finding that 65% of harmful AI systems remained operational despite documented harm underscores the urgency of accountability mechanisms. The proposed CVE-like reference system for disability-focused AI harms is a particularly compelling recommendation — providing the kind of systematic tracking infrastructure that has proven effective in cybersecurity. The paper's use of real incident databases rather than speculative risk analysis gives its taxonomy empirical grounding. However, as the authors note, the taxonomy captures outcomes but not root causes, and the 79-incident dataset likely underrepresents actual harms given the invisibility of many disabilities and the power imbalances that discourage reporting.
Tags: AI fairness · algorithmic bias · disability rights · AI harm · AI incident databases · algorithmic discrimination · predictive AI · accessibility
Standards referenced: Americans with Disabilities Act · GDPR