Sensory Substitution Training for Users Who Are Blind with Dynamic Stimuli, Games and Virtual Environments
Shachar Maidenbaum · 2015 · ASSETS '15: Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility · doi:10.1145/2700648.2811324
Summary
This paper explores whether gamification and dynamic training can address one of the major barriers to sensory substitution device (SSD) adoption: the long, arduous learning process. SSDs like EyeMusic convert visual information into sound, enabling people who are blind to perceive their environment—but mastering them requires extensive practice with repetitive tasks that often lead to frustration and abandonment. The author upgraded training for congenitally blind EyeMusic users from static exercises to interactive, dynamic tasks. The study tested 7 participants who were congenitally totally blind and had previous static training with EyeMusic. Three types of dynamic additions were introduced: manipulable objects that users could move, rotate, and change via keyboard; simple 2D games requiring spatial parsing (like navigating to a target while avoiding obstacles); and first-person virtual environments teaching visual principles like depth-size relationships and occlusion. All interactions used standard keyboard input (arrow keys, Enter, Space). EyeMusic itself works by sweeping across an image left-to-right, encoding X-position as time, Y-position as musical pitch, brightness as volume, and color as different musical instruments. This allows whole-scene visual information including shape, location, and color to be conveyed through sound. The system is freely available on iOS and Android and requires only standard headphones.
Key findings
All participants successfully completed the dynamic tasks, demonstrating they could utilize visual principles like depth-size relationships despite having no visual experience. Both users and instructors rated the dynamic sessions as more enjoyable, less tiring, and more effective than static training. The most significant finding was participants' increased sense of independence and control. Users reported being able to work by themselves, create their own shapes, and interpret feedback directly without instructor mediation. One instructor noted that participants learned concepts through play that they struggled to grasp through direct instruction—"when playing she could get around concepts she didn't get when I tried to teach directly. Especially angles and space, but also colors. She just got it by playing without noticing." Nearly all participants recommended restructuring training around the dynamic additions, though both users and instructors agreed that combining both static and dynamic approaches was optimal. Instructors particularly valued the flexibility to pause planned lessons for focused mini-sessions addressing specific difficulties.
Relevance
This research addresses a critical barrier in assistive technology adoption: the training burden. Many promising technologies fail not because they don't work, but because learning them is too frustrating. The gamification approach demonstrated here—shifting from passive reception to active exploration—offers a transferable design pattern for any assistive technology requiring skill development. The emphasis on user agency and independence is particularly valuable. Participants didn't just perform better; they felt more in control of their own learning. For practitioners designing training programs for assistive technologies, this suggests prioritizing interactive, user-driven experiences over instructor-led static demonstrations. The finding that users learned visual concepts "without noticing" through play points to gamification's potential for implicit learning of complex skills.
Tags: sensory substitution · blindness · congenital blindness · visual rehabilitation · gamification · training · auditory interface · EyeMusic · virtual environments · game accessibility