Designing Gestures for Digital Musical Instruments: Gesture Elicitation Study with Deaf and Hard of Hearing People
Ryo Iijima, Akihisa Shitara, Yoichi Ochiai · 2022 · Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '22) · doi:10.1145/3517428.3544828
Summary
This study addresses a gap in how deaf and hard of hearing (DHH) people can enjoy playing musical instruments using smartphones. While DHH people can and do enjoy music — often sensing it through vibrations transmitted by instruments or through body movements while performing — existing digital musical instrument apps on smartphones fail to replicate this experience. Most apps use touchscreen controls that don't engage the body in musically meaningful ways, and sensory substitution devices that convert sound to light or vibration require specialized hardware and assume access to traditional instruments. The researchers conducted a gesture elicitation study (GES) with 11 DHH participants to identify their preferred gestures when using a smartphone as a musical instrument. Participants were presented with 10 instruments spanning the four main Hornbostel-Sachs classification categories (idiophone, membranophone, chordophone, and aerophone) — including castanets, triangle, guiro, maracas, pellet drum, tam-tam, jaw harp, recorder, guitar, and cymbal. Each instrument produced a distinctive vibrational pattern through the iPhone's Core Haptics engine, and participants were asked to invent gestures they would naturally perform to accompany each instrument's vibrations. The study was conducted remotely via Zoom during the COVID-19 pandemic, with 11 participants (mean age 25.4, eight profoundly deaf) who all had prior musical experience.
Key findings
From 11 participants and 10 instruments, the study produced 110 total gestures (55 distinct). The mean agreement rate (AR) across all referents was 0.27, typical for gesture elicitation studies. Maracas had the highest agreement (AR=0.65), while tam-tam and jaw harp had the lowest (AR=0.05 each). A key finding was the negative correlation between ease of use and thinking time — participants who spent longer considering their gestures produced ones rated easier to use. There was also a positive correlation between goodness of fit and agreement rate, meaning gestures most participants found natural also tended to be appropriate for the instrument. The researchers developed the first taxonomy of musical instrument gestures for DHH people, organized along three dimensions: Nature (Motion at 70%, Vibration at 22%, Abstract, Name), Form (Smartphone, Hand, Both, Pose), and Function (Sonorous, Non-sonorous, Others). Notably, 70% of gestures were classified as Motion — mimicking how the real instrument is played — rather than Vibration-based gestures. Instruments played with both hands (cymbals, maracas) showed the strongest agreement, while unfamiliar instruments (jaw harp) and one-handed instruments (tam-tam) produced the most diverse and least agreed-upon gestures.
Relevance
This research opens an important but often overlooked area of accessibility: enabling DHH people to enjoy musical performance through everyday technology rather than specialized hardware. The smartphone-based approach is significant because it leverages a device most people already own, lowering barriers to musical participation. For accessibility practitioners, the study demonstrates the value of involving DHH users directly in designing interaction patterns rather than assuming hearing-centric models will transfer. The gesture taxonomy provides a practical design resource for developers creating accessible music applications. The finding that participants preferred motion-based gestures mimicking real instrument playing — rather than abstract or vibration-focused gestures — suggests that embodied, intuitive interactions should be prioritized in accessible instrument design. The study also highlights cultural considerations, such as how sign language influences gesture creation, which has broader implications for designing gesture-based interfaces for Deaf users across any domain.
Tags: deaf and hard of hearing · music accessibility · gesture elicitation · digital musical instruments · haptic feedback · mobile accessibility · sensory substitution · user-centered design