"We Musicians Know How to Divide and Conquer": Exploring Multimodal Interactions To Improve Music Reading and Memorization for Blind and Low Vision Learners
Leon Lu, Chase Crispin, Audrey Girouard · 2024 · Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3663548.3675604
Summary
This paper investigates how multimodal assistive technologies (MATs) combining sound, vibration, and touch can improve music reading and memorization for blind or low-vision (BLV) musicians and learners. The study involved nine BLV participants from The FMDG Music School in New York City across two phases: an ideation workshop featuring open-ended discussions about challenges and strategies, and a co-design workshop combining sensory bodystorming with a Wizard of Oz exercise using a wearable haptic prototype. The prototype used vibration motors on participants' forearms to convey tempo (through timed intermittent vibrations), dynamics (through changes in vibration intensity), and coded reminders (through predefined vibration patterns) synchronized with a sample audio track. Participants also interacted with 3D-printed Mechamagnet tactile buttons and switches to mock up interaction designs. The research frames challenges through Cognitive Load Theory (CLT), classifying difficulties into intrinsic complexities (cognitive aspects of understanding music from Braille notation or audio recordings) and extraneous complexities (external factors like interaction and access methods). BLV musicians typically separate music learning into sequential stages — listening, memorizing, and then playing — unlike sighted musicians who can simultaneously read and play.
Key findings
The study identified significant challenges with both primary methods BLV musicians use to access music. Braille music presents intrinsic complexities: the six-dot system differs between literary and music Braille (causing cognitive switching difficulties), complex rhythms and time signatures are hard to interpret, notation has changed over time creating inconsistencies, and sections contain incomplete repetition instructions. Learning by ear has its own intrinsic complexities: learners without perfect pitch struggle to identify individual notes, and complex chords cannot be deciphered by hearing alone. Extraneous complexities include that Braille music requires hands-off-instrument reading (particularly problematic for pianists who must read each hand separately), single-line Braille displays limit music access, and Braille notation is not always available or accurate. Participants proposed six categories of multimodal design ideas: segmenting and audibly playing Braille music to improve readability, using vibrations to convey rhythm and timing (replacing audible metronomes), using vibrations as timely landmarks and reminders during playing, providing technical guidance through multiple vibrations (e.g., indicating fingering), augmenting audio by breaking chords into sequential notes, and integrating multiple modalities with flexibility to toggle between them. Seven of nine participants wanted the ability to dynamically switch between modalities.
Relevance
This research opens important new ground in music accessibility by moving beyond screen-reader-friendly notation software to explore how multimodal wearable technologies can holistically support the music learning process. For accessibility technology designers, the findings provide a clear framework: match specific modalities to specific types of complexity — vibrations for temporal information like rhythm and timing, audio for pitch and harmony, and tactile interactions for hands-free control. The emphasis on hands-free interaction is critical, as musicians need their hands for their instruments. The study also highlights that technological support should complement rather than replace Braille music and music teachers, and that simplicity, modularity, and customizability are essential design principles. The co-design methodology demonstrates how involving BLV musicians as active participants produces richer, more practical design ideas grounded in lived experience.
Tags: visual impairment · music accessibility · Braille music · multimodal interaction · vibrotactile feedback · co-design · cognitive load · assistive technology