Demographic and Experiential Factors Influencing Acceptance of Sign Language Animation by Deaf Users
Hernisa Kacorri, Matt Huenerfauth, Sarah Ebling, Kasmira Patel, Mackenzie Willard · 2015 · ASSETS '15: Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility · doi:10.1145/2700648.2809860
Summary
This paper addresses a critical methodological gap in sign language animation research: the lack of standardized reporting on participant characteristics that may influence evaluation results. Sign language animation technology—which automatically synthesizes signing from scripts—could make it easier to add ASL content to websites, but researchers evaluate these systems by collecting subjective judgments and comprehension scores from deaf participants. Without understanding how participant characteristics affect these scores, it is difficult to compare results across studies or determine whether differences reflect animation quality or participant pool variations. The researchers conducted an in-person study with 62 deaf participants (ages 18-59, average 25.7) who viewed ASL animations generated by three different state-of-the-art avatar systems: EMBR (motion-captured from human signers), JASigning (using Hamburg Notation System), and VCom3D (commercial authoring software). All participants learned ASL before age 5, had used it for over 9 years, and used ASL daily. Participants answered comprehension questions about animation content and provided subjective ratings on grammar quality, understandability, naturalness, and likability. The study collected extensive demographic data (age, gender, deaf identity, school type, ASL use at home/work, parental deafness) and technology experience/attitudes (internet search frequency, media sharing, video game experience, ASL video chat usage, attitudes toward technology and computers, prior exposure to sign language animation). Multiple regression analysis identified which factors predicted comprehension and subjective scores.
Key findings
Models combining demographic and technology variables explained significantly more variance than demographics alone—38.2% vs 25.6% for comprehension, 33.5% vs 15.3% for subjective scores. Four factors emerged as most important for predicting evaluation scores: **SchoolType** had the largest effect on comprehension: attending a residential school for deaf students positively predicted comprehension scores. Interestingly, this relationship reversed for subjective scores—residential school attendees gave lower subjective ratings, possibly reflecting higher standards for signing quality. **HomeASL** (using ASL at home) negatively predicted subjective scores, suggesting that frequent ASL users may be harsher critics of animation quality. This has important implications: studies with participants who use ASL extensively at home may yield lower subjective scores than studies with less frequent ASL users. **MediaSharing** (watching videos, downloading media) negatively predicted subjective scores, contrary to the expectation that tech-savvy users would be more accepting. The authors speculate that higher technology experience creates higher standards for animation quality. **AnimationAttitude** (general belief in usefulness of sign language animation) positively predicted subjective scores—participants who believed animations could be useful for websites, public information displays, and interpreting gave higher ratings to specific animations. Notably, **SeenBefore** (prior exposure to sign language animation) did not significantly predict scores, suggesting researchers can re-recruit participants who have seen earlier versions of their systems without biasing results.
Relevance
This paper makes a crucial methodological contribution to sign language technology research. The finding that participant characteristics explain 25-38% of variance in evaluation scores means that comparing results across studies with different participant pools is problematic without controlling for these factors. A study recruiting primarily residential school graduates will likely yield different comprehension scores than one recruiting mainstreamed students—and these differences could be misattributed to animation quality. The practical contribution is a standardized questionnaire (available in both ASL video and English text at the authors' lab website) that researchers can use to collect comparable demographic and technology data. The authors recommend that at minimum, researchers report: SchoolType, HomeASL, MediaSharing, and AnimationAttitude. This enables the field to build cumulative knowledge rather than isolated findings. For accessibility practitioners, the finding that frequent ASL users and technology-experienced users rate animations more harshly suggests that recruiting "expert" users for evaluation may systematically underestimate acceptance among the broader deaf population. Conversely, positive attitudes toward animation technology (as a concept) predict positive evaluation of specific animations—meaning attitude surveys could help contextualize evaluation results. The median 4th-grade English literacy level among deaf high school graduates underscores why ASL animation technology matters: many deaf individuals prefer receiving information in their primary language rather than written English.
Tags: deaf · sign language · animation · avatar · ASL · American Sign Language · user study · methodology
Standards referenced: ISO/IEC 14496-2:2004 · MPEG-4 Facial Animation