Ad-Hoc Access to Musical Sound for Deaf Individuals
Benjamin Petry, Thavishi Illandara, Juan Pablo Forero, Suranga Nanayakkara · 2016 · Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '16) · doi:10.1145/2982142.2982213
Summary
This poster paper presents MuSS-Bits (Music Sensory Substitution Bits), a wearable sensor-display system that enables deaf individuals to explore musical sound from various audio sources with real-time vibrotactile feedback. While existing sensory substitution systems for music access by deaf people provide accurate musical information, they are typically limited to pre-defined input sources and are not designed for ad-hoc sound access — the ability to spontaneously connect to any audio source such as instruments, digital devices, or environmental sounds. Wearable devices like smartwatches and phones can provide vibrations but use pancake motors with slow response times (~40ms lag, ~100ms rise time) that are inadequate for the temporal accuracy needed for musical feedback. MuSS-Bits are designed around three core goals: real-time feedback, input from different audio sources, and wearability for portable ad-hoc access. Additional goals include supporting rhythm information (since steady-beat and rhythm are introduced first in deaf music education), easy audio source selection, and simple, unobtrusive operation by non-experts. The paper documents three hardware prototype iterations, progressing from breadboard circuits to a compact wearable form factor using Bluetooth audio receivers, audio amplifiers, and linear resonant actuators (LRAs) that provide the high temporal resolution needed for musical feedback.
Key findings
The paper focuses on the engineering design process across three prototype iterations rather than presenting user evaluation results. The first prototype used a breadboard with a Bluetooth audio receiver, audio amplifier, and LRA, confirming that the audio-to-vibrotactile signal chain could provide real-time rhythm feedback. The second prototype miniaturized the circuit onto a custom PCB with 3D printed casing, adding a 3.5mm audio jack input alongside Bluetooth for connecting directly to instruments. The third and latest prototype further reduced the form factor, improved wearability, and refined the electronics. The system uses the audio signal directly to drive the LRA rather than extracting musical features first, which preserves temporal accuracy. Key technical challenges included selecting actuators with sufficient temporal resolution (LRAs over eccentric rotating mass motors), managing power consumption for portable use, and designing for comfortable extended wear during music practice or performance.
Relevance
This work addresses a fundamental gap in music accessibility for deaf individuals: the ability to spontaneously explore sound from any source, rather than being restricted to pre-configured setups. For accessibility practitioners, the ad-hoc access concept is important because it supports the exploratory learning that is essential to music education — a deaf student should be able to pick up any instrument, connect to any audio source, and immediately feel the vibrotactile representation of what they produce. The direct audio-to-vibrotactile approach (bypassing feature extraction) is a pragmatic design choice that prioritizes temporal fidelity for rhythm perception, the most accessible musical parameter for deaf individuals. The iterative hardware prototyping process demonstrates the practical challenges of developing wearable assistive technology: balancing actuator quality with power consumption, miniaturization with comfort, and versatility with simplicity. As wearable technology advances, systems like MuSS-Bits could make music participation more accessible by removing the dependency on specialized, expensive equipment.
Tags: deaf and hard of hearing · music accessibility · sensory substitution · haptic technology · wearable technology · assistive technology