SoundModVR: Sound Modifications in Virtual Reality to Support People who are Deaf and Hard of Hearing
Xinyun Cao, Dhruv Jain · 2024 · Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '24) · doi:10.1145/3663548.3675653
Summary
This paper challenges the prevailing approach to VR sound accessibility for deaf and hard-of-hearing (DHH) people, which has focused almost exclusively on substituting audio with visual or haptic alternatives (captions, flashing indicators, vibrations). The authors argue that deafness occurs on a spectrum — many DHH people have partial hearing and benefit more from having control over audio itself than from replacing it with another modality. The researchers designed, implemented, and evaluated 18 sound modification tools organised into four categories: prioritising sounds (allowing users to boost important sounds and reduce background noise), modifying sound parameters (adjusting pitch, speed, volume, frequency balance for individual sound sources), providing spatial assistance (adding directional indicators, distance-based audio adjustments, and radar-style sound maps), and adding supplementary sounds (earcons, beeps, or tonal cues that reinforce or replace specific in-game sounds with more perceivable alternatives). The tools were built in Unity and evaluated across five diverse VR scenarios — a conversation in a noisy coffee shop, a nature exploration environment, a museum tour, a multiplayer game, and a horror game — chosen to represent a range of audio complexity and social situations. Ten DHH participants with varying degrees of hearing loss (mild to profound, unilateral and bilateral, with and without hearing aids or cochlear implants) evaluated the tools.
Key findings
Participants rated the sound modification tools positively overall, with the most valued tools being those that provided individual control over sound source volumes (allowing users to boost speech while reducing background music or ambient noise), frequency shifting (moving high-frequency sounds that some participants could not perceive into their audible range), and spatial audio indicators (visual or tonal cues showing the direction of off-screen sounds). The prioritisation tools were especially popular in the noisy coffee shop scenario, where participants could selectively amplify the conversational partner while suppressing crowd noise — a capability that maps directly to real-world challenges DHH people face. The spatial assistance tools were most valued in the horror game and nature scenarios where environmental awareness was critical for both safety and immersion. However, participants noted several tensions: adding too many supplementary sounds created information overload and could be more distracting than helpful; some modifications (like slowing speech) improved comprehension but broke immersion; and the sheer number of available tools (18) was overwhelming, with participants wanting presets tailored to their hearing profile rather than manual configuration of each parameter. The researchers also compiled a subset of the tools into a Unity developer toolkit and evaluated it with six VR developers, who found it easy to integrate (average setup time under 30 minutes) but requested better modularisation, documentation, and the ability to expose modification controls to end users at runtime. A key insight was that sound modification is not just an accessibility feature — several participants noted that sighted, hearing users might also benefit from audio customisation in VR, reinforcing the curb-cut effect.
Relevance
This paper makes a valuable contribution by reframing DHH accessibility in VR beyond the binary of "hearing" versus "substituting sound with another modality." For accessibility practitioners, the central insight is that many DHH people want more control over audio, not less audio — a principle that extends well beyond VR to video conferencing, media players, and any application with complex audio environments. The four-category framework (prioritise, modify parameters, spatial assistance, supplementary sounds) provides a practical taxonomy for thinking about audio accessibility features. The Unity toolkit makes the work directly actionable for VR developers. The finding that presets based on hearing profiles are preferred over manual configuration echoes broader accessibility patterns — personalisation is powerful but must not require excessive user effort. Limitations include the small sample size, the controlled evaluation scenarios, and the focus on Unity-based VR, but the design principles are transferable to other platforms and to augmented reality applications where audio modification could also benefit DHH users in real-world environments.
Tags: deaf and hard of hearing · virtual reality · sound modification · audio accessibility · partial hearing · Unity · developer toolkit