Understanding Deaf and Hard-of-Hearing Users' Interest in Sign-Language Interaction with Personal-Assistant Devices
Abraham Glasser, Vaishnavi Mande, Matt Huenerfauth · 2021 · Proceedings of the 18th International Web for All Conference (W4A '21) · doi:10.1145/3430263.3452428
Summary
This mixed-methods study investigates Deaf and Hard of Hearing (DHH) users' experiences with and interest in personal-assistant devices (such as Amazon Alexa and Google Home) that could understand American Sign Language (ASL) commands. The research was motivated by the growing ubiquity of voice-controlled personal assistants, which pose fundamental accessibility barriers for DHH users who may not use their voice or whose speech may not be recognized by automatic speech recognition systems. The study was conducted in two phases: semi-structured interviews with 21 DHH ASL signers (12 women, 8 men, 1 non-binary; mean age 24) recruited from a university campus, followed by a nationally distributed online survey of 86 DHH ASL users (49 women, 37 men; mean age 47) from over 20 U.S. states. The survey was designed to be accessible to ASL-fluent individuals, with all questions and answer choices presented in both English text and ASL video recorded by a native signer. The interview transcripts were analysed using iterative semantic thematic analysis, which identified 10 categories of commands that DHH users would want to issue. These categories informed the survey design, enabling quantitative data collection on usage patterns, interest levels, command preferences, device placement, interaction modalities, and privacy concerns.
Key findings
The study revealed a stark accessibility gap: 80.2% of DHH survey respondents had never used a personal-assistant device, compared to only 28% of the general population in prior research — a statistically significant difference (chi-squared, p < .0001). Despite this low usage, interest was high: 62.8% of survey respondents agreed or strongly agreed they would be interested in using a device that could understand ASL. Beyond traditional personal-assistant uses (weather queries, alarms, timers), DHH participants were most interested in DHH-specific applications: receiving alerts about environmental sounds (doorbells, smoke alarms, baby crying), initiating video-based communication (videophone/VRS calls), and receiving notifications. For device wake-up, most interview participants initially preferred hand-waving (a culturally appropriate attention-getting method in Deaf culture), though after considering false-positive concerns, many shifted to push-to-talk physical touch methods. A notable contrast emerged regarding output modality: while participants expressed strong interest in ASL animation output in principle, when presented with concrete options, text-based output was the most preferred response format (selected by 75 of 86 participants), with participants citing speed and the ability to skim. Privacy was a significant concern — participants worried about cameras required for sign-language recognition, with majorities wanting options to turn off both microphone and camera and to have physical camera covers.
Relevance
This research provides essential foundational knowledge for developers and designers of conversational AI and smart-home technologies seeking to make these products accessible to DHH users. The finding that DHH users have dramatically lower adoption rates yet strong interest in sign-language-capable devices represents both a clear accessibility failure and a significant opportunity. For accessibility practitioners, the study highlights that simply adding text-input alternatives to voice-controlled devices is insufficient — DHH users want interaction that aligns with their primary language and cultural practices. The identification of DHH-specific use cases (sound alerts, video calls, ASL interpretation requests) demonstrates that accessible design must go beyond adapting existing features and should consider the unique needs and contexts of DHH users. The privacy concerns around camera-based sign recognition also raise important design considerations that will become increasingly relevant as sign-language recognition technology matures. The publicly available dataset of commands and interaction preferences can inform future ASL dataset collection efforts needed for sign-recognition AI development.
Tags: deaf and hard of hearing · sign language · personal assistants · voice interface · smart home · sign language recognition · user research