Analyzing Deaf and Hard-of-Hearing Users' Behavior, Usage, and Interaction with a Personal Assistant Device that Understands Sign-Language Input
Abraham Glasser, Matthew Watkins, Kira Hart, Sooyeon Lee, Matt Huenerfauth · 2022 · Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI '22) · doi:10.1145/3491102.3501987
Summary
This CHI 2022 paper investigates how Deaf and Hard-of-Hearing (DHH) ASL signers would actually use a personal assistant device (such as an Amazon Echo Show or Google Home) if it could understand American Sign Language commands. The authors position their work against the accessibility gap created by the rapid spread of voice-controlled technology: most smart speakers and assistants require spoken wake words and spoken commands, leaving DHH users reliant on others in the household to operate them. Prior research had surveyed DHH users about their interest in sign-aware assistants or asked them to imagine commands they might give, but no one had placed participants in front of a working prototype to see what they would spontaneously do. To close that gap, the authors ran a Wizard-of-Oz study with 21 ASL-signing participants (14 Deaf, 3 deaf, 2 hard-of-hearing; mean age 25) recruited through Rochester Institute of Technology. Because of COVID-19, sessions ran over Zoom: the participant believed they were signing to an Amazon Echo Show 10-inch, while a hidden ASL interpreter voiced their commands in English to the real Alexa. Each participant completed roughly 70 minutes of interaction. The research team transcribed and ASL-glossed all utterances, yielding a publicly released dataset of over 1,400 ASL commands and interactions, and coded wake-up methods, command topic categories, linguistic structures, and user behavior after device errors.
Key findings
Wake-up behavior revealed patterns not seen in prior imagined-interaction studies: participants used eleven distinct wake-up methods, including fingerspelling 'Alexa' (the most common at 233 cases), the ASL sign ATTENTION-WAVE paired with fingerspelling, and culturally Deaf attention-getting signs such as HELLO, HEY, HI, CURIOUS, and DO-DO. In 1,081 cases participants issued no wake-up at all — these were mostly follow-ups. Prior imagined-use studies had predicted clapping and talk-to-talk approaches that did not appear spontaneously. Command topics fell into 15 categories; the largest were Command and control (352), Entertainment (162), Lifestyle (127), Shopping (126), and Trivia (124). A DHH-specific category (47 commands) included accessibility requests such as 'GIVE CAPTION IX VIDEO QMWG' and references to Video Relay Services like Convo. ASL linguistic structures observed included WH-words at the end of sentences, the QMWG 'question mark wiggle', spatial indexing (IX), and fingerspelling for proper names, city names, and less-common nouns. Five error-recovery behaviors emerged: Ignored (229 cases), Repeated (205), Reworded (129), Played Along (25), and Question (18). After errors, participants commonly shifted from fluent ASL to English-like word order or to fingerspelling. Pre/post 5-point interest ratings (4.29 → 4.19) were statistically equivalent by a TOST test with a 0.5 margin — the prototype experience did not diminish enthusiasm.
Relevance
This paper is directly useful for two practitioner audiences. For designers of smart speakers, voice assistants, and conversational interfaces, it documents specific wake-up vocabulary, sentence structures, and error-recovery patterns that a sign-aware device must support, and it highlights DHH-specific use cases — captioning requests, VRS launch, notifications about environmental sound — that mainstream voice assistants do not currently address. For sign-language-recognition researchers, the publicly released 1,400+ command dataset is a rare corpus of spontaneous, task-directed ASL that reflects real user intent rather than dictionary examples, and can serve as training or evaluation data for models. The gap between imagined and observed interaction — participants did things (fingerspelling ALEXA, using cultural Deaf attention signs) that prior survey-based work had not predicted — is a strong argument for prototype-based studies in accessible AI. Key limitations: remote Zoom-based setup limited IoT integration and wake-up modalities, participants were university-educated fluent ASL signers, and the Wizard-of-Oz interpreter inevitably normalised some commands when voicing them to Alexa.
Tags: sign language · american sign language · deaf and hard of hearing · personal assistants · voice assistants · smart speaker · conversational interface · sign language recognition · accessibility research · wizard of oz