CymaSense: A Novel Audio-Visual Therapeutic Tool for People on the Autism Spectrum
John McGowan, Grégory Leplâtre, Iain McGregor · 2017 · Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '17) · doi:10.1145/3132525.3132539
Summary
This paper presents CymaSense, an interactive multimodal tool that generates real-time 3D visual representations of sound based on Cymatics — the study of visible patterns created by sound vibrations in physical media like water or metal plates. The system was designed to augment improvisational music therapy sessions for adults with Autism Spectrum Condition (ASC). CymaSense transforms acoustic and electronic audio input into translucent 3D Cymatic shapes projected onto a screen or interactive table surface, where volume affects shape scale, pitch determines which of twelve shapes is displayed and its colour lightness, and timbre modifies surface qualities. The tool was developed based on interviews with music therapy practitioners who identified needs for clear cause-and-effect relationships, customisable visual elements, and tangible user interfaces to support social interaction. The system was built using Cycling '74 Max for real-time audio analysis and the Unity game engine for 3D visualisation. Three deployment modes were used: a projected screen, a two-user side-by-side projection (allowing therapist and client to each control their own visuals), and an interactive table with a Perspex surface, contact microphone, and projector underneath. The study evaluated CymaSense over an 8-week period with eight adults with ASC (ages 18-28) at TouchBase, a centre run by Sense Scotland, using a single case experimental design with two baseline weeks followed by six intervention weeks.
Key findings
The study demonstrated statistically significant increases in both musical and non-musical communicative behaviours when CymaSense was introduced into music therapy sessions. Using the Communicative Responses and Acts Score Sheet (CRASS), researchers coded 57 hours of video across all sessions. Wilcoxon Matched-Pairs Signed Ranks Tests showed significant differences between first and last sessions (p=0.009 for combined scores), between last non-intervention and first intervention sessions (p=0.022), and between third and last intervention sessions (p=0.014), indicating that improvements were not merely novelty effects. Non-musical CRASS data — including eye contact, verbal and gestural communication — showed particularly notable increases, with some participants' scores rising from single digits to over 100 per session. A post-study Behaviour Change Survey completed by parents, care workers, and the music tutor (80% return rate) indicated slight positive changes across communicative, social/emotional, and musical behaviours outside sessions (mean scores 5.6-5.7 on a 7-point scale). Qualitative analysis revealed individual trajectories: non-verbal participants showed increased eye contact, vocalisation, and gestural interaction, while verbal participants showed enhanced creativity and confidence. The interactive table proved especially effective for participants who struggled to coordinate playing an instrument while watching a screen. One participant (P3) showed minimal improvement, attributed to issues with trust-building and session attendance.
Relevance
This research demonstrates how technology can meaningfully augment established therapeutic approaches for people with autism. For accessibility practitioners, the key insight is the importance of multimodal, cause-and-effect feedback — when people with ASC can see the immediate visual result of their sounds, it strengthens their sense of agency and encourages communication. The design principles behind CymaSense are transferable: clear causality between user action and system response, customisable sensory output, support for both individual and shared interaction, and gradual introduction of new elements to avoid change-related stress. The study also highlights that adults with autism remain underserved compared to children, and that technology-augmented interventions can work across a range of verbal and non-verbal abilities. Limitations include the small sample size, absence of a withdrawal phase in the experimental design, and the difficulty of separating the effects of the tool from the music therapy itself.
Tags: autism spectrum disorder · music therapy · assistive technology · multimodal interaction · sensory processing · tangible interaction · audio-visual · communication · neurodiversity