Notification Designs for Influencing Hearing Speakers' Behaviors During Captioned Conversations Among Mixed DHH-Hearing Groups
Matthew Seita, Sarah Andrew, Matt Huenerfauth · 2025 · Proceedings of the 22nd International Web for All Conference (W4A 2025) · doi:10.1145/3744257.3744273
Summary
This paper investigates notification system designs that prompt hearing speakers to adjust their speech behaviors (speaking slower, louder, or more clearly) during ASR-captioned videoconference conversations with Deaf and Hard of Hearing (DHH) participants. The researchers conducted two studies using a Wizard-of-Oz methodology on Google Meet. Study 1 was a preliminary evaluation where 15 DHH and 11 hearing participants individually viewed three notification designs of varying visual prominence: Icons (small icons at screen edges, low prominence), Pop-Up (medium-sized speech bubble near the speaker video), and Overlay (large transparent colored overlay covering the entire speaker video). Participants rated each design on noticeability, perceived efficacy, conversation flow, and overall satisfaction. Study 2 was a follow-up group experiment with 7 DHH and 7 hearing participants engaging in actual collaborative discussions in mixed DHH-hearing pairs, testing the top two designs (Icons and Overlay) against a no-notification baseline. The notification instructions included "Speak SLOWER," "Speak LOUDER," and "Speak CLEARER"—three behaviors previously identified as important by DHH participants. The Wizard-of-Oz approach was used because the researchers aimed to gather user feedback on designs before investing in fully automated systems, with a researcher manually triggering notifications every one to two minutes based on speaking behaviors.
Key findings
DHH and hearing participants had notably different preferences. In Study 1, DHH participants significantly preferred Icons for conversation flow over Pop-Up (p=0.04312) and Overlay (p=0.00888), finding less prominent notifications less disruptive. Hearing participants found the Pop-Up and Overlay significantly easier to notice (p=0.0354 and p=0.02048 respectively) and more effective than Icons. In Study 2's natural conversations, DHH participants generally felt the baseline with no notifications flowed better than the Overlay, but were more aware of hearing speakers' volume and speech rate when notifications were present. Hearing participants reported being more aware of their own volume and speech rate during both notification conditions and felt they were monitoring their speech more. All 14 Study 2 participants expressed interest in using this technology in real-world settings. Interview analysis revealed key themes: a tension between notification prominence and conversation flow disruption, broad optimism about AI and automation for triggering notifications, and strong interest in customizable interfaces. Critically, 13 of 14 participants agreed the technology could help shift the "burden of communication" from DHH individuals to hearing speakers and automated systems, reducing the need for DHH people to repeatedly ask others to slow down or speak up.
Relevance
This research addresses a fundamental power imbalance in mixed DHH-hearing communication: DHH individuals typically bear the burden of asking hearing speakers to modify their behavior, which can be exhausting and embarrassing. By designing technology that automates these prompts, the responsibility shifts toward hearing speakers and the system itself. For practitioners building videoconferencing or captioning tools, the study provides concrete design guidance: less prominent notifications (like Icons) preserve conversation flow for DHH users who are already focused on captions, while more prominent notifications (like Overlays) are more effective at changing hearing speakers' behavior. The finding that DHH and hearing users have divergent preferences highlights the importance of customizable notification settings. The work has direct implications for workplace accessibility, where impromptu meetings without interpreters are common and ASR-based captioning is increasingly relied upon as an accessibility tool.
Tags: deaf and hard of hearing · automatic speech recognition · captioning · notification design · videoconferencing · communication accessibility · Wizard-of-Oz