Personal Perspectives on Using Automatic Speech Recognition to Facilitate Communication between Deaf Students and Hearing Customers
James R. Mallory, Michael Stinson, Lisa Elliot, Donna Easton · 2017 · Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3132525.3134779
Summary
This experience report examines the use of Automatic Speech Recognition (ASR) via the WhatsApp smartphone app to facilitate communication between deaf and hard-of-hearing (D/HH) students and hearing business customers in real workplace settings. The study took place at the National Technical Institute for the Deaf (NTID) at Rochester Institute of Technology, where a technical capstone course requires D/HH students completing Applied Computer Technology degrees to implement projects at local businesses. Without interpreters present at every meeting, communication between signing deaf students and hearing business owners was typically cumbersome, relying on written notes and basic text messaging. Over three semesters, 28 D/HH students worked at four different business locations in the Rochester area, including a swimming pool supply store, a paddling center, and a mobile device repair shop. The researchers conducted 10 field trials — 5 with ASR and 5 without — observing 21 students and 6 hearing customers. The communication model was straightforward: hearing business owners spoke into their phones using WhatsApp's ASR feature to convert speech to text, while D/HH students typed their responses back as text messages, creating a shared conversational stream accessible to all participants throughout the project. One of the authors, Michael Stinson, is himself deaf and brought personal experience with the frustrations of following group conversations without sign language support, even with a cochlear implant.
Key findings
In trials without ASR, deaf students relied on writing or text messaging, while hearing customers tended to write only basic information — both when responding to questions and when initiating conversations. With ASR, customers initially needed time to become familiar with the technology, but once comfortable, they used it with relative ease and it appeared to facilitate richer communication. Based on visual inspection of transcripts and direct observation, ASR was generally accurate enough for participants to rely on the text display for meaningful exchanges. In surveys, all twelve students who rated ASR found it either somewhat or very helpful for communicating with hearing customers. Students reported receiving more information from hearing customers when ASR was used compared to non-ASR trials. Hard-of-hearing students were less reliant on ASR than deaf students, suggesting the technology particularly benefits those with less residual hearing. Practical challenges emerged at the mobile device repair business, where a technician needed to frequently pick up and put down his phone while demonstrating smartphone repairs. Behind-the-ear Bluetooth devices helped, but WhatsApp's requirement to hit a "send" button remained a friction point. The authors identified smartwatches and continuous-streaming ASR apps as potential solutions.
Relevance
This study addresses a significant gap in workplace accessibility: while educational settings have legislated accommodations like sign language interpreting, cost-effective communication support in the workplace remains limited. The use of mainstream consumer technology (WhatsApp on smartphones) rather than specialized assistive technology is notable — it requires no special equipment, no training for the hearing party beyond basic smartphone use, and creates a persistent text record that D/HH workers can reference later. For accessibility practitioners and employers, this demonstrates that existing mobile ASR tools can meaningfully bridge communication gaps in small-group workplace settings, even if imperfect. The finding that hearing customers adapted quickly to the ASR workflow is encouraging for workplace adoption. However, the study also highlights real-world limitations: hands-free operation remains a challenge for physical tasks, and the need to manually send messages interrupts natural conversation flow. As ASR technology continues to improve, the practical model described here — hearing person speaks, ASR converts, deaf person reads and types back — offers a low-barrier template for workplace communication.
Tags: automatic speech recognition · deaf and hard of hearing · workplace accessibility · deaf education · speech recognition · communication accessibility · mobile accessibility