SoundVizVR: Sound Indicators for Accessible Sounds in Virtual Reality for Deaf or Hard-of-Hearing Users
Ziming Li, Shannon Connell, Wendy Dannels, Roshan Peiris · 2022 · Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '22) · doi:10.1145/3517428.3544817
Summary
This paper presents SoundVizVR, a Unity plugin that visualizes sound information in virtual reality environments for Deaf or Hard-of-hearing (DHH) users. Sounds in VR provide critical spatial cues, interaction feedback, and ambient information that contribute to immersion, but DHH users miss this information entirely, degrading their VR experience. SoundVizVR addresses this through two complementary types of visual indicators. Sound-Characteristic Indicators convey where sounds are coming from, how loud they are, and how long they last. These take two forms: On-Object Indicators (red spheres that appear above sounding objects and dynamically change size with loudness) and Mini-Maps (circular overhead radar views showing sound source positions — either Full Mini-Maps displaying the complete 360-degree surroundings, or Partial Mini-Maps that hide the area directly in front of the user to encourage looking at the environment). Sound-Type Indicators convey what kind of sound is being produced, using either descriptive text labels (e.g., "Bark," "Phone Ring," "Footstep") or iconic representations (black and white icons) displayed both on the Mini-Map and on the objects themselves. The system focuses on "diegetic" sounds — sounds originating from objects within the VR world, including localized speech, inanimate objects (phones, radios), animate objects (animals), and point ambience (fire, water). The prototype is implemented as a customizable Unity plugin, available on GitHub, that VR designers can drag-and-drop onto any sound-source object in their scenes.
Key findings
Two within-subjects user studies were conducted. In Study 1, 11 DHH participants (4 deaf, 7 hard-of-hearing) evaluated six combinations of Sound-Characteristic Indicators across 144 sound localization tasks each. The Full Mini-Map combined with On-Object Indicators (FM-OI) emerged as the clear winner: it achieved 94.70% localization accuracy with an average completion time of 4.41 seconds, earned an "Excellent" System Usability Scale rating (84.77), and had the lowest mental workload across all NASA TLX dimensions. The baseline condition with no visualization (NON) had only 24.24% accuracy and 8.93 seconds completion time. Participants who played video games recognized the Mini-Map pattern immediately, with one noting "because I play video games with full circle Mini-Map on the screen, it is very easy for me to engage in this system." The On-Object Indicators were crucial for disambiguating between nearby sound sources that the Mini-Map alone couldn't distinguish. Long-duration sounds had significantly better localization accuracy than short-duration sounds across all conditions. In Study 2, 14 DHH participants (6 deaf, 8 hard-of-hearing) evaluated four Sound-Type Indicator combinations using the FM-OI method in a realistic VR game scene with multiple simultaneous sound sources. All four combinations achieved approximately 90% localization accuracy with no significant differences between icon and text representations. The icon-only condition (IM-IO) achieved the highest SUS score ("Excellent," 81.96) and lowest mental workload on five of six NASA TLX dimensions. Preferences were split: some participants found icons faster and lower cognitive load, while others found text more descriptive and noticeable on the map.
Relevance
SoundVizVR makes a practical contribution to VR accessibility that developers can immediately adopt — the Unity plugin is open source and designed for drag-and-drop integration into existing projects. For accessibility practitioners working on immersive media, the research demonstrates that familiar gaming conventions (Mini-Maps, on-screen indicators) can be effectively repurposed as accessibility features, lowering the learning curve for DHH users who already play games. The two-layer approach — characteristic indicators for spatial awareness plus type indicators for semantic understanding — provides a useful framework for any sound visualization system. The finding that both icons and text work equally well for sound type identification, but with different user preferences, strongly supports making these indicators customizable rather than prescribing one approach. The research also highlights an underexplored accessibility gap: as VR becomes more prevalent in education, training, social interaction, and entertainment, ensuring DHH users can fully participate requires going beyond simple captioning to address spatial sound information that has no direct text equivalent. The work complements existing approaches like haptic feedback (EarVR) and head-mounted display captioning, and could potentially benefit hearing users in situational impairment scenarios (e.g., noisy environments or muted headsets).
Tags: virtual reality · Deaf and Hard-of-Hearing · sound visualization · audio accessibility · game accessibility · immersive media · spatial audio