Somatic Drawing Tool: 3D Body Sheet as Material for Articulating Synaesthetic Experiences
Mamoru Watanabe, Oussama Metatla, Atau Tanaka · 2026 · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems (CHI '26) · doi:10.1145/3772318.3790931
Summary
This CHI 2026 paper reports a three-year participatory co-design cycle with synaesthetes - people who involuntarily experience cross-sensory perceptions such as seeing colours when hearing music or feeling tactile sensations when reading letters. The authors argue that synaesthetic experience is highly individual, spatially and bodily situated, and poorly served by the verbal descriptions on which both clinical diagnosis and HCI research typically rely. The reported work covers two of the three stages: (1) three participatory workshops with 9 synaesthetes exploring verbal, non-verbal, and immersive articulation using analogue materials (clay, colour, scents), AI image generation, and VR free-drawing via Open Brush on a Quest 2 headset, analysed thematically across five themes; and (2) the development of the Somatic Drawing Tool, a Unity/OpenBrush-based VR system built around a 3D Body Sheet - an immersive extension of soma design's 2D body-mapping tool - with bespoke synaesthetic brushes, integrated text/audio/animation assets, and a calm dark environment. Evaluation used a pilot plus two-session case studies with five participants (P1, P2, P3, P10, P11) across five types of synaesthesia, with per-participant iteration of the tool between sessions (new body figures, seated postures, colour palettes, animated text). The paper positions the work as contributing both a concrete interactive artefact for synaesthete communication and a methodological extension of soma design's body-mapping practice into 3D immersive space.
Key findings
Workshops surfaced five themes: verbal description opens articulation but consistently falls short; non-verbal materials expose overlooked dimensions by forcing participants to make spatial, tactile and temporal choices; VR reveals new aspects of synaesthesia (P5 discovered that his tickertape synaesthesia was positioned "just above the forehead" rather than inside the head, something he had never articulated in decades of lived experience); environmental conditions (lighting, ambient sound) strongly shape access to synaesthetic perception; and comparison with other synaesthetes drives self-understanding. In the main evaluation the Somatic Drawing Tool supported articulation of qualities that 2D media cannot capture - directionality, spatial positioning, movement, bilateral asymmetry, and embodied context (P11's chromesthesia required a seated piano-playing posture to map accurately). Iterative per-participant customisation (new body figures, palettes, animation, pre-built VR scenes) was essential; a single fixed tool was insufficient. Limits also emerged: visual-only VR was inadequate for P3's multisensory "typhoon-like" idea-tactile synaesthesia (she demanded drum sounds), and P8 noted some synaesthetic experiences "transcend" any externalisable representation. The authors propose six personalisation mechanisms (modality mapping, intensity calibration, synaesthetic profile templates, narrative-driven configuration, cross-sensory override, editable sensory outputs) as the design space for future inclusive cross-modal articulation tools.
Relevance
For accessibility practitioners this paper matters on three levels. First, it documents a population (~4.4% of adults) whose sensory experiences are systematically unsupported by existing HCI tooling - synaesthetes often describe isolation and failed medical tests, and this work demonstrates that interactive, embodied tools can provide meaningful articulation and community. Second, the methodological extension of soma design's 2D body sheet into an immersive 3D body sheet offers a reusable articulation technique for any perceptual condition that resists verbal description - chronic pain, interoceptive symptoms, vestibular disorders, migraine aura, phantom-limb sensation, atypical sensory processing in autism. Third, the per-participant iteration pattern and the six personalisation mechanisms constitute a concrete template for inclusive co-design where sensory experience is heterogeneous. Limitations include a small case-study sample (five participants, one type of synaesthesia each out of 14 collectively reported), VR-specific barriers (no tactile feedback, limited pressure sensitivity, novelty effects and motion sickness), the visual-only bias of VR as a medium for a fundamentally multisensory phenomenon, and the dependence of outcomes on participants' existing artistic skill. The tool is to be released open-source on GitHub.
Tags: synaesthesia · sensory articulation · soma design · body sheet · VR drawing · multisensory interaction · cross-modal · participatory design · virtual reality · embodied interaction