How Could Equality and Data Protection Law Shape AI Fairness for People with Disabilities?
Reuben Binns, Reuben Kirkham · 2021 · ACM Transactions on Accessible Computing · doi:10.1145/3473673
Summary
This interdisciplinary paper examines how UK equality law and EU data protection law (GDPR) intersect with AI fairness for people with disabilities (PWD). The authors argue that AI fairness for PWD requires a fundamentally different approach than for other protected characteristics like race or gender, due to how disability is uniquely treated in discrimination law. Under the UK Equality Act 2010, disability is "asymmetric"—organizations can lawfully treat PWD more favorably than non-disabled people, unlike with race or sex discrimination. The law also creates an affirmative duty to make reasonable adjustments rather than merely avoiding discrimination. Additionally, the concept of "discrimination arising from a disability" loosens the causal link required to prove discrimination, making it easier for PWD to bring claims. The paper analyzes how these legal frameworks apply to AI systems in three categories: AI-driven assistive technologies, AI tools that disproportionately impact PWD (like benefits eligibility systems), and general-purpose AI that may not be accessible. The authors use a running example of AI-based video interview analysis in hiring, showing how such systems might unfairly disadvantage candidates with speech impairments by correlating speech patterns with negative outcomes in training data.
Key findings
The paper identifies critical tensions between equality law and data protection law in the context of AI fairness. Two main algorithmic fairness approaches exist: Disparate Learning Process (DLP), which uses disability data during model development but not deployment, and Treatment Disparity (TD), which uses disability as an input feature during live decisions. The GDPR's strict rules on "solely automated decisions" with "legal or similarly significant effects" essentially prohibit most TD approaches for PWD unless explicit consent is obtained or substantial public interest conditions are met—conditions that are narrow and difficult to satisfy. This means that some well-intentioned fairness interventions may themselves be unlawful. However, the combination of laws also creates opportunities: (1) Data protection law provides clearer duties to secure disability-related data than equality law does; (2) GDPR gives individuals rights to contest automated decisions, request human intervention, and access data—tools that equality law alone does not provide; (3) Non-profit disability organizations have special standing under GDPR to process disability data to support individuals; (4) Data Protection Impact Assessments required for high-risk AI processing can serve as entry points for challenging discriminatory systems before deployment.
Relevance
This paper provides essential legal grounding for accessibility practitioners, AT researchers, and disability advocates working with AI systems. Key practical implications include: (1) Organizations cannot simply adopt generic "AI fairness" approaches designed for race or gender—disability requires distinct treatment, including the possibility of deliberately favoring PWD; (2) Collecting disability data to test AI fairness is lawful under several GDPR conditions, removing a common excuse for not auditing systems; (3) Data protection law can compensate for equality law's weak enforcement by providing proactive duties (DPIAs) and individual rights that are more actionable; (4) Disability rights organizations have a unique legal position to serve as trusted intermediaries for disability data and to bring collective challenges against harmful AI systems; (5) The right to meaningful information about automated decision logic (GDPR Article 13(2)f) can help PWD evidence discrimination claims. For AT developers, the paper suggests that GDPR compliance should be seen not as a barrier but as a framework for building trust that enables the collection of disability data needed to improve AI systems for PWD.
Tags: AI fairness · disability discrimination · data protection · GDPR · equality law · algorithmic bias · machine learning · human rights · disability rights · automated decision-making · legal framework
Standards referenced: UK Equality Act 2010 · GDPR · UN CRPD · European Convention on Human Rights