Exploring the Performance of Facial Expression Recognition Technologies on Deaf Adults and Their Children
Irene Rogan Shaffer · 2018 · Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2018) · doi:10.1145/3234695.3240986
Summary
This Boston University student research paper investigates how commercial facial expression recognition services perform on Deaf ASL signers and Children of Deaf Adults (CODAs) compared to hearing non-signers. The study is motivated by a critical problem: in ASL and other sign languages, facial and head movements serve important linguistic functions — they can alter the meaning of what is being communicated. For example, the ASL marker MM means "effortlessly" or "regularly," TH means "carelessly," INTENSE means "much greater than expected," PUFF means "a great deal," PS means "smoothly" or "easily," and CS means "recently." These linguistic markers are distinct from emotional expressions but can appear similar to untrained observers. The researcher captured over 2,000 photographs of 15 participants across three groups: 5 Deaf ASL signers (mean age 67.4), 5 hearing CODAs who are native ASL signers (mean age 37.8), and 5 hearing non-signers (mean age 38.8). Each participant produced the six universal emotional expressions (sad, happy, surprise, anger, fear, disgust), and Deaf and CODA participants additionally produced the six ASL linguistic markers. Photos were taken in participants' homes and submitted to six commercial emotion recognition services: Affectiva, EmoVu, Face++, Kairos, Microsoft Azure Face API, and Sightcorp F.A.C.E. API.
Key findings
Three key findings emerged. First, face detection rates were significantly lower for Deaf participants (75%) compared to hearing participants (97%) and CODAs (89%). The head tilts involved in ASL linguistic markers, particularly MM, likely caused this — the face detection algorithms were not trained on these head positions. Second, paradoxically, emotion recognition accuracy was higher for Deaf (44%) and CODA (42%) participants than for hearing participants (34%). This suggests that ASL signers may produce more pronounced or clearer emotional facial expressions, possibly because their proficiency with facial communication makes their emotional expressions more recognizable to these systems. Third, and most concerning, ASL linguistic markers were frequently misinterpreted as negative emotions. The PUFF marker (meaning "a great deal") was often classified as anger (27% for CODA, 22% for Deaf) or sadness (24% CODA, 25% Deaf). This pattern was consistent across all six linguistic markers — they were most commonly read as anger, disgust, sadness, or fear, despite having no negative emotional content. This mirrors previous research showing that hearing observers also misinterpret these markers as negative emotions.
Relevance
This study exposes a concrete example of AI bias against a linguistic minority. As facial expression recognition technology is deployed in settings like job interviews (HireVue), classroom monitoring, security screening, and mental health assessment, Deaf ASL signers face the risk of being systematically mischaracterized as angry, sad, or fearful when they are simply using their language. The immediate practical implication is that emotion recognition training datasets must include data from ASL users — both Deaf signers and hearing CODAs — with ASL linguistic markers correctly labeled as non-emotional. The poor face detection rate for Deaf participants (75% vs 97% for hearing) represents a compounding bias: not only are Deaf faces less likely to be detected, but when they are detected, their linguistic expressions are misread as negative emotions. For accessibility and AI ethics practitioners, this study demonstrates that algorithmic fairness must extend beyond protected categories like race and gender to include linguistic and cultural differences associated with disability communities.
Tags: deaf and hard of hearing · sign language · facial expression recognition · emotion recognition · AI fairness · algorithmic bias · computer vision · ASL · CODA · deaf culture