Evaluating Importance of Facial Expression in American Sign Language and Pidgin Signed English Animations
Matt Huenerfauth, Pengfei Lu, Andrew Rosenberg · 2011 · Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2011) · doi:10.1145/2049536.2049556
Summary
This paper presents the first empirical evaluation of the impact of facial expressions on the understandability and perceived quality of American Sign Language (ASL) and Pidgin Signed English (PSE) computer animations. ASL animations have significant accessibility benefits since a majority of deaf high school graduates in the US read at a fourth-grade level or below, making text-based content inaccessible to many. In prior studies, native signers consistently critiqued the insufficient facial expressions of animated characters — comments like "face was bland," "too stiff," and "eyebrows don't raise enough" dominated feedback. Facial expressions in ASL serve essential grammatical functions: the same hand movements can mean entirely different things depending on facial expression (e.g., "JOHN LOVE MARY" can be a statement, a yes/no question, or a negation depending on the face). In PSE, facial expressions convey prosodic information analogous to pitch, timing, and volume in spoken English. The researchers conducted two studies: one with 12 PSE signers evaluating 28 passages in versions with and without facial expressions, and another with 12 native ASL signers using the same passages translated into ASL. Both used within-subjects designs with comprehension questions and naturalness ratings.
Key findings
Both studies demonstrated that adding facial expressions to animations produced measurable improvements. For PSE, adding facial expressions significantly increased comprehension scores (Mann-Whitney test, p<0.05) and naturalness ratings. The EMOTION and CONTINUE prosodic categories showed the largest benefits from facial expressions, while adding facial expressions for QUESTIONS actually reduced comprehension — likely because the wh-question facial expression in the animation system looked too similar to an angry face, misleading participants. For ASL, the addition of prosodically motivated facial expressions also significantly increased comprehension scores (p<0.05), with the EMOTION category again showing the most notable benefit. Participants in both studies were sensitive to minor errors in facial articulation — two PSE participants reported being misled by the wh-question face resembling anger. This sensitivity contrasts with prior findings showing deaf users tolerate errors in hand location or orientation, suggesting facial expression accuracy is held to a higher standard. The average "notice" score of 5.2 out of 10 for PSE animations with facial expressions indicates room for improvement in conveying the intended prosodic meaning.
Relevance
This research has direct implications for the development of sign language technology, which is increasingly important for making web content and services accessible to deaf users with limited English literacy. The key practical finding is that facial expressions are not optional cosmetic additions to sign language animations — they are essential for comprehension and perceived quality. Developers of signing avatar systems need to invest in accurate facial expression synthesis, not just hand movements. The study also highlights a significant design challenge: users are highly sensitive to facial expression errors, and an incorrect expression (like an angry face instead of a questioning one) can actively harm comprehension. This means that partially implemented or inaccurate facial expressions may be worse than none at all for certain expression types. The methodology adapted from speech prosody research provides a reusable framework for future evaluation of sign language animation systems. For accessibility practitioners, this work underscores that sign language accessibility requires nuance far beyond simply translating text to signs — the full linguistic richness of sign languages, including non-manual components, must be preserved.
Tags: sign language · sign language animation · deaf accessibility · facial expression · American Sign Language · assistive technology · natural language processing