Inclusive Improvisation: Exploring the Line between Listening and Playing Music
Alon Ilsar, Gail Kenning, Sam Trolland, Ciaran Frame · 2022 · ACM Transactions on Accessible Computing · doi:10.1145/3506856
Summary
This paper explores how Digital Musical Instruments (DMIs) designed for professional performance can be adapted into Accessible Digital Musical Instruments (ADMIs) with minimal hardware and software changes. The authors present the AirSticks, a gestural instrument originally created by the first author for his professional percussion practice, and demonstrate its adaptation for people with diverse disabilities through three case studies. The AirSticks use motion tracking controllers to translate physical gestures into sound. Version 1.0 employs Razer Hydra gaming controllers connected to custom CAMS (Custom AirSticks MIDI Software) that outputs to Ableton Live. Version 2.0 features custom 3D-printed drumstick-shaped enclosures with Bosch IMU sensors and Bluetooth connectivity, allowing wireless operation and a more natural instrumental form factor. The software translates orientation and acceleration data into MIDI and OSC protocols, with customizable mappings that connect movements to sounds. Four core sound mappings were developed: AirHarp (spatial harp metaphor), Warmth (drone with timbre control), AirRadio (crossfading between spatial sound samples), and FluidDrum (morphing drum synth). These mappings use familiar metaphors—plucking strings, striking drums, tuning radio—to make the relationship between gesture and sound intuitive without requiring prior musical training.
Key findings
The paper's central argument is that successful ADMI development depends less on hardware and software redesign than on understanding individual players' needs and customizing movement-to-sound mappings accordingly. This finding emerged from three case studies spanning diverse abilities and contexts. Alessio, a teenager with leukodystrophy (a progressive neurological condition), was able to immediately engage with the AirSticks 1.0 and perform publicly at a music festival. The mappings were adapted so all buttons triggered the same note, allowing him to calibrate his movements. He gravitated toward the "glitch aesthetic" of the FluidDrum mapping and was able to improvise autonomously without third-party assistance. "Violet," a 93-year-old woman with advanced dementia in residential care, engaged with the AirSticks despite being unable to follow verbal instructions. She quickly understood the gesture-to-sound relationship through exploration rather than explanation, moving from the AirHarp mapping to singing spontaneously over the Warmth drone. The instrument designer joined her in improvisation, describing it as "a truly magical and spiritual moment." The BoilOver Inclusive Performance Ensemble, a theatre group where people with disability hold roles from performers to stage managers, adopted the AirSticks 2.0 over three rehearsal sessions. Individual pieces were created for nine performers based on their recorded gestural data and music preferences, demonstrating how personalized mappings can accommodate vastly different movement abilities and musical tastes within a single ensemble.
Relevance
This research shifts the paradigm for accessible instrument design from building specialized devices "from scratch" to adapting existing professional instruments through personalized configuration. The approach accelerates development timelines and ensures that accessible instruments benefit from the expressive capabilities already validated by expert musicians. For accessibility practitioners, the paper emphasizes three principles: (1) personalization of mappings matters more than hardware adaptation—understanding how each player moves and what sounds resonate with them is essential; (2) communities of practice are critical for trust and sustained engagement, whether through organizations like Safe in Sound or BoilOver Theatre; and (3) empowerment rather than therapy should frame creative engagement for people with disabilities, positioning them as artists with agency rather than patients receiving treatment. The concept of "musicking"—music as participatory doing rather than passive consumption—challenges assumptions about who can make music and how. The case studies demonstrate that improvisation and creative flow are accessible to people with significant cognitive and physical disabilities when instruments are designed around their existing movement capabilities rather than requiring adaptation to the technology. Organizations supporting music programs for people with disabilities should consider how gestural instruments can expand participation beyond traditional adaptive approaches.
Tags: accessible digital musical instruments · ADMI · gesture recognition · music accessibility · improvisation · dementia · leukodystrophy · empowerment · inclusive design · motion tracking