Regression Analysis of Demographic and Technology-Experience Factors Influencing Acceptance of Sign Language Animation
Hernisa Kacorri, Matt Huenerfauth, Sarah Ebling, Kasmira Patel, Kellie Menzies, Mackenzie Willard · 2017 · ACM Transactions on Accessible Computing (TACCESS) · doi:10.1145/3046787
Summary
This paper investigates how deaf participants' demographic backgrounds and technology experience influence their evaluation scores when assessing sign language animation systems, revealing that participant characteristics — not just animation quality — significantly affect study outcomes. Through a survey study with 62 deaf participants (Study 1) and a follow-up experimental study with 57 participants (Study 2), the authors used multiple regression analysis to identify which participant factors predict both comprehension accuracy and subjective quality ratings of ASL animations. The research is motivated by the goal of creating software that automatically generates linguistically accurate ASL animations from scripts, which would make maintaining sign language content on websites more efficient than recording human signers. Most sign language animation researchers evaluate their systems by collecting subjective judgments and comprehension responses from deaf participants, but the field lacks standardized reporting of participant demographics, making cross-study comparisons unreliable. In Study 1, participants viewed ASL animations produced by three different avatar platforms (EMBR, JASigning, VCOM) and answered comprehension questions and subjective evaluation items. The researchers collected extensive demographic data (age, gender, school type, age of ASL acquisition, home ASL use) and technology-experience measures (media sharing frequency, internet search habits, attitudes toward animation technology, gaming habits). Multiple regression models using exhaustive "leaps" variable selection identified the most predictive subsets of variables for comprehension and subjective scores.
Key findings
Four participant characteristics emerged as the most important predictors of evaluation scores: SchoolType (whether the participant attended a residential/daytime school for deaf children versus mainstream school), HomeASL (whether ASL was used at home), MediaSharing (frequency of sharing media online), and AnimationAttitude (general attitude toward sign language animation technology). For comprehension scores, SchoolType had by far the highest relative importance — participants from residential deaf schools scored significantly higher. For subjective scores, MediaSharing, HomeASL, AnimationAttitude, and SchoolType were all important, but with surprising directions: participants who used ASL at home and those with higher media sharing frequency gave lower subjective scores, suggesting that more experienced ASL users and more tech-savvy participants had "higher standards" and were more critical. Study 2 experimentally confirmed most of these relationships: the SchoolType effect on comprehension was verified (p < 0.05), and the SchoolType, MediaSharing, and AnimationAttitude effects on subjective scores were all confirmed. A striking finding was that stimuli-related variables (which animation platform and which message) explained relatively little variance compared to participant characteristics — the demographic and technology model alone accounted for 20-30% of adjusted R-squared, while the stimuli-only model explained under 5%.
Relevance
This research has methodological implications that extend far beyond sign language animation to any accessibility technology evaluation involving deaf or disabled participants. The central finding — that who you recruit for your study significantly affects your results — is a wake-up call for the field. If participant characteristics explain more variance than the actual technology being evaluated, then studies with different participant pools cannot be meaningfully compared without reporting and controlling for these factors. The authors provide their survey instruments in both English and ASL as a practical contribution, encouraging standardized demographic reporting across the research community. For practitioners developing sign language avatars or animation systems, the participant feedback reveals that movement smoothness and facial expression quality are the highest priorities, and that deaf users envision diverse deployment contexts including public transportation, emergency services, and educational tools. The ethical discussion about not positioning animation as a replacement for human interpreters — which some participants explicitly raised as a concern — is essential guidance for responsible deployment of this technology.
Tags: sign language avatar · deaf and hard of hearing · evaluation methodology · regression analysis · ASL animation · demographic factors · research methods