Spotlights and Soundscapes: On the Design of Mixed Reality Auditory Environments for Persons with Visual Impairment
Keenan R. May, Brianna J. Tomlinson, Xiaomeng Ma, Phillip Roberts, Bruce N. Walker · 2020 · ACM Transactions on Accessible Computing · doi:10.1145/3378576
Summary
This paper explores the design of information-rich mixed reality (MR) auditory environments to help people with visual impairments form cognitive maps of unfamiliar indoor spaces. The research contrasts two approaches: soundscapes, where virtual objects in all directions emit spatialized sounds representing landmarks, and spotlights, where only objects within a front-facing cone are audible. The study used a two-experiment methodology. In Experiment 1, participants explored a virtual environment in room-scale VR while researchers used a Wizard-of-Oz interface to rapidly adjust audio parameters (TTS speed, auditory icon pitch, volume falloff, tempo) in real-time based on participant feedback. This led to key design refinements including a Level of Detail (LOD) system where objects transition from auditory icons at long range, to low-detail TTS at mid-range, to detailed TTS with distance and manipulation information at close range. Experiment 2 tested the refined system in a real university building, comparing soundscape, spotlight, and beacon-only conditions, with participants reconstructing cognitive maps using Lego blocks after navigation.
Key findings
Both soundscape and spotlight designs proved usable without overwhelming users, with BUZZ auditory user experience scores in the mid-70s (out of 100) across conditions. The soundscape condition produced marginally more comprehensive and accurate cognitive maps (mean fidelity score 2.37 vs 2.04 for spotlight, 2.25 for beacon-only), though at the cost of higher mental workload (NASA-TLX mental: 53.14 for soundscape vs 37.96 for spotlight). Participants strongly preferred literal, realistic auditory icons (footsteps on stairs, door push-bar sounds) over abstract metaphorical icons, expecting virtual sounds to represent what objects actually sound like when interacted with. Users demonstrated tolerance for positional tracking inaccuracies, using their O&M skills to cross-check virtual information against environmental perception. Critical customization needs emerged: users wanted modes for different object densities, the ability to toggle between TTS-only and auditory-icon-only, and distinct modes for white cane versus guide dog users—the latter needing fewer nearby objects but more decision-point landmarks. Room-gating (playing only sounds for objects in the current room) was identified as a promising approach to reduce auditory clutter.
Relevance
This research provides extensive practical guidance for developers creating dense auditory MR navigation systems. The rapid prototyping methodology using consumer VR hardware and Unity enables efficient iteration on complex audio designs before real-world deployment. The Level of Detail system offers a reusable pattern: auditory icons provide awareness of distant objects, while detailed TTS delivers actionable information for nearby landmarks—balancing information richness against cognitive load. The finding that users prefer literal over metaphorical auditory icons has broad implications for sonification design in assistive technology. Critically, the study reveals that white cane and guide dog users have fundamentally different information needs: cane users benefit from obstacle warnings but find dense soundscapes disruptive to tactile feedback, while guide dog users can devote more attention to landmark audio since the dog handles obstacle avoidance. Future systems should support crowdsourced landmark databases and user-customizable tags, allowing personal and shared annotation of important objects within spaces.
Tags: mixed reality · spatial audio · auditory display · cognitive maps · indoor navigation · blind and low vision · soundscape · virtual reality · user-centered design