Participatory Design Using Sensory Substitution Devices with Tactile and Audio Feedback
Ella Tuson, Samantha Hughson, Christina Zymaris, Ryan King · 2017 · Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3132525.3134826
Summary
This student research paper describes a participatory design study with a single blind participant (TC, blind from birth) to develop a sensory substitution device that provides a physical experience of spatial awareness through tactile or auditory feedback. The study was motivated by TC's dissatisfaction with existing sensory substitution technologies — she found tongue-based electrotactile systems (like BrainPort) difficult to use and auditory systems (like EyeMusic) unhelpful, as neither provided the physical sensation she associated with sight. TC reported occasionally experiencing physical sensations she associated with vision, such as feeling her eyes "pulled" toward bright areas like windows, and she desired a device that could replicate this kind of embodied spatial experience. The researchers at Clark University built three prototype devices, each using a different output modality to convey distance information: an audio prototype (beeping sounds varying with distance), a vibration prototype (a LEGO Mindstorms motor with variable speed), and a pressure prototype (two yardsticks connected like scissors that exerted varying pressure on TC's hands to simulate visual parallax). All three used a Wizard of Oz methodology — a researcher manually controlled the stimuli based on a live video feed from a Samsung Gear VR headset worn by TC, standing in for the computer vision algorithm that would eventually automate the system. Distance was measured using a smartphone distance meter app while the researcher observed the video stream.
Key findings
Each prototype elicited distinct feedback. The audio prototype allowed TC to gauge distance "pretty accurately" but she expressed concern that increased auditory complexity in a more developed system would be overwhelming: "I might not be able to make sense of it." The vibration prototype (LEGO motor) was effective for one-dimensional distance measurement, but TC found it limited and suggested that vibration intensity might better represent object density rather than distance, noting that "speed is a key facet of my visual experiences." The pressure prototype received the most positive response. TC was excited by its attempt to simulate eye movement through the opening and closing of the scissors-like device and found it "easy to understand." She suggested that adding rotation would be important for a more complete model. The researchers immediately modified the prototype to allow rotational hand movement (simulating parallax), which TC responded to positively. Based on these results, the team plans to develop a vibrotactile device inspired by the VibroGlove that combines pressure and rotation to mimic the way eyes move, pushing and rotating the user's hands to convey distance and spatial relationships.
Relevance
This study exemplifies several important principles for assistive technology design. First, it centers the user's own descriptions of their sensory experiences rather than imposing sighted researchers' assumptions about what a blind person needs — and honestly acknowledges that sighted researchers' misunderstanding of TC's descriptions was a significant challenge. Second, the participatory approach of rapidly prototyping multiple modalities and iterating based on immediate user feedback allowed the team to quickly eliminate less promising approaches and focus on the most resonant one. Third, TC's preference for the pressure/parallax device over more common audio and vibration approaches challenges conventional assumptions about sensory substitution — suggesting that some users may benefit from more embodied, proprioceptive feedback that mimics the physical mechanics of vision rather than abstractly encoding visual information. The study is limited by its single-participant design, but as a participatory design study intended to develop personalized assistive technology, this depth of engagement with one user is a valid and valuable approach.
Tags: sensory substitution · blindness · haptic feedback · participatory design · tactile accessibility · assistive technology · prototyping