Design and Evaluation of a Multimodal Science Simulation
Brianna J. Tomlinson, Prakriti Kaini, Siyan Zhou, Taliesin L. Smith, Emily B. Moore, Bruce N. Walker · 2018 · Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '18) · doi:10.1145/3234695.3241009
Summary
This demonstration paper presents a multimodal version of PhET's John Travoltage interactive physics simulation, designed to support learners with and without visual impairments in exploring static electricity concepts. The key design goal was creating a single simulation with multiple modality 'layers' — visual display, verbal descriptions, sound effects, and sonifications — that can be accessed simultaneously or in different combinations to meet individual needs. The PhET simulation lets users rub John's foot on a rug to transfer negative charges onto his body, then move his arm closer to or farther from a doorknob; depending on the charge amount and hand-doorknob distance, electrons discharge and 'shock' John. The auditory display includes six carefully designed sound mappings: carpet rubbing sounds (auditory icons), pitch-based sonification of charge transfer and arm rotation, volume/playback-rate sonification of charges on body, an electrical zap for discharge, and speech for the shock. Descriptions are implemented using PhET's Parallel DOM architecture, providing screen reader users with dynamic state descriptions, position values, real-time alerts about charge transfer, and contextual information about key relationships.
Key findings
Evaluation with 11 participants across different modality combinations showed positive results. Five adult screen reader users with visual impairments (2 low-vision, 3 blind) used the sim with description and sound effects/sonifications only (no visual display), rating aesthetics 25.8/28 and usability 20.2/24. Four of five specifically praised how sounds and descriptions worked together. All reported sounds and descriptions as useful, and most (4/5) commented positively on description clarity. Six sighted learners (three children aged 7-9, three college students) explored the sim with visual display and sound but no descriptions. All fully explored the simulation and correctly described the key physics relationships. While learners sometimes interpreted sound mappings differently than intended (e.g., arm rotation ratchet described as 'like a xylophone' or 'like a wind-up toy'), these interpretations were still functionally relevant — both users understood the sound conveyed arm location changes. Some sounds were attended to less by children, though no negative learning impact was observed.
Relevance
This paper demonstrates a best-practice approach to accessible STEM education: designing a single learning experience with layered modalities rather than creating separate 'accessible versions.' The PhET simulations are used by millions of students worldwide, making accessibility of this platform high-impact. The Parallel DOM architecture for description is a significant technical contribution — it enables screen reader access to complex interactive content without altering the visual interface, a challenge that many web-based educational tools fail to address. The sound design taxonomy (auditory icons for discrete events, sonification for continuous variables, speech for critical alerts) provides a reusable pattern for making interactive visualizations accessible. For practitioners, the finding that learners make relevant but non-identical interpretations of sound mappings suggests that exact fidelity between intended and perceived meaning matters less than functional usefulness — a sound that conveys 'something is changing' can support learning even if the specific interpretation differs from the designer's intent.
Tags: STEM accessibility · sonification · multimodal interface · visual impairment · science education · PhET · auditory display · screen readers · universal design · inclusive design
Standards referenced: Web Audio API