Collaborative Music Application for Visually Impaired People with Tangible Objects on Table
Shotaro Omori, Ikuko Eguchi Yairi · 2013 · Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/2513383.2513403
Summary
This short paper presents a collaborative music composition application designed to enable visually impaired and sighted people to work together on equal ground. The system uses a tabletop tangible interface inspired by Reactable, where physical objects placed on a table surface are tracked by AR markers and a Kinect sensor. Object positions on the table map directly to musical parameters — horizontal position controls timing and vertical position controls pitch — providing real-time sound feedback as users arrange objects. All tangible objects are connected by chains, allowing visually impaired users to trace connections by touch and intuitionally understand the structure of the composition. The design was informed by interviews with an educator of visually impaired students, who identified key barriers to music enjoyment: difficulty memorizing phrases and reading braille music scores. The system supports not just playback of existing music but composition of new pieces. Two users sit at opposite ends of a table, each creating melodies independently that combine into a collaborative composition. Additional functions include a dice-shaped box with six AR markers for changing instruments (high-pitched piano, low-pitched piano, guitar, drum kit, trumpet, and violin) and other interactive features designed to promote communication.
Key findings
In an experiment with six visually impaired and six sighted participants, visually impaired users were able to lead the collaborative work without hesitation, even when paired with sighted people they did not know well. This is a notable outcome given that collaborative activities between blind and sighted people often place the visually impaired person in a dependent or secondary role. The tangible interface leveraged blind users' strong tactile sense, putting them on equal or even advantageous footing compared to sighted collaborators. The researchers specifically analyzed moments when visually impaired participants were having fun, identifying factors that contributed to excitement and engagement. The chain-based physical connections between objects proved effective for enabling blind users to understand spatial relationships and the overall structure of the musical composition through touch alone.
Relevance
This research demonstrates an important principle for accessible design: rather than adapting a visual interface for blind users, the system was built around touch and sound from the start, creating a shared experience where visual impairment is not a disadvantage. The tangible interface approach offers a model for collaborative tools that avoid the common pattern of sighted users leading while blind users follow. For accessibility practitioners, the study illustrates how choosing the right interaction modality — in this case, physical objects with haptic connections and audio feedback — can create genuinely equitable collaboration rather than merely "accessible" participation. The work also contributes to the growing field of music accessibility, showing that composition (not just consumption) can be made inclusive through thoughtful interface design.
Tags: visual impairment · tangible user interface · music accessibility · collaborative design · AR markers · Kinect · social inclusion