SoundHapticVR: Head-Based Spatial Haptic Feedback for Accessible Sounds in Virtual Reality for Deaf and Hard of Hearing Users
Pratheep Kumar Chelladurai, Ziming Li, Maximilian Weber, Tae Oh, Roshan L Peiris · 2024 · ASSETS '24: Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3663548.3675639
Summary
This paper presents SoundHapticVR, a head-based haptic feedback system that converts spatial audio cues in virtual reality into vibrotactile feedback for Deaf and Hard of Hearing (DHH) users. VR environments rely heavily on spatial audio to convey critical information — alerting users to approaching objects, indicating directions, and creating immersive experiences — but these cues are entirely inaccessible to DHH individuals. The authors developed a prototype consisting of multiple acoustic haptic actuators (voice coil transducers) mounted on a VR headset strap, positioned around the head at forehead, temporal, and occipital regions. The system captures spatial audio from the VR environment, processes it in real time, and delivers directional haptic vibrations corresponding to the sound source location. The research was conducted through three progressive studies with DHH participants. Study 1 (n=12) established tactile frequency perception thresholds across different head regions, finding that sensitivity varies significantly by location — the forehead was most sensitive, while occipital regions required higher amplitudes. Study 2 (n=12) tested different transducer configurations (4, 5, 6, and 7 channels) to determine optimal placement for sound localization accuracy. Study 3 (n=12) evaluated whether users could distinguish between different sound sources using distinct haptic patterns mapped to specific VR objects like phones, doorbells, and alarms.
Key findings
The tactile frequency threshold study revealed that head regions have significantly different vibration sensitivity levels, with the forehead being most sensitive and areas near the occipital bone least sensitive. This necessitates frequency equalization — adjusting vibration amplitude per transducer location — to create perceptually consistent haptic feedback around the head. For sound localization, a 5-transducer configuration (adding a front-centre channel to the base 4-channel layout) significantly improved directional accuracy compared to 4 channels, achieving a mean localization error that participants found usable. Adding more transducers beyond 5 yielded diminishing returns. In the haptic pattern differentiation study, participants successfully associated distinct vibration patterns with specific sound sources at rates well above chance, demonstrating that meaningful semantic information can be conveyed through head-based haptics. Participants reported that the system felt intuitive after brief training, and several noted it would enhance their VR experience by providing spatial awareness they otherwise lacked. The system operated in real time with minimal latency, a critical requirement for maintaining VR immersion.
Relevance
SoundHapticVR addresses a fundamental accessibility gap in virtual reality — the near-total reliance on audio for spatial information. As VR becomes increasingly used in education, training, collaboration, and entertainment, excluding DHH users from these experiences represents a significant barrier. This research demonstrates that sensory substitution through haptics is a viable approach, converting audio information into a modality that DHH users can perceive without compromising immersion. The practical finding that 5 transducers with frequency equalization provides an effective configuration gives hardware designers a concrete target for accessible VR headsets. For accessibility practitioners, the work reinforces the principle that multimodal output — providing the same information through multiple sensory channels — is essential for inclusive design. The methodology of establishing perceptual thresholds before designing the interaction is a model for rigorous, user-centred assistive technology development.
Tags: deaf and hard of hearing · haptics · virtual reality · sound localization · sensory substitution · spatial audio · head-mounted display