Comparing Natural Language and Vibro-Audio Modalities for Inclusive STEM Learning with Blind and Low Vision Users
Justin R. Brown, Stacy A. Doore, Justin K. Dimmel, Norbert Giudice, Nicholas A. Giudice · 2023 · Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2023) · doi:10.1145/3597638.3608429
Summary
This study evaluates a multimodal touchscreen diagram system (MDS v 2.0) built on an iPhone that provides blind and low vision (BLV) users with two non-visual methods for accessing graphical STEM content: natural language (NL) descriptions delivered via VoiceOver text-to-speech, and vibro-audio (VA) exploration using the device's built-in vibration motor combined with an auditory content overview. The research addresses a critical educational equity gap — graphical representations appear on nearly every page of STEM textbooks, yet converting these to accessible formats costs up to ,000 per book and takes 4 months to a full academic year, leaving BLV students unable to keep pace with instruction. The study tested 19 BLV participants (ages 19-69, recruited from the National Federation of the Blind convention and The Carroll Center for the Blind) on three common graph types: Venn diagrams, pie charts, and line graphs. Each participant experienced four graphs per type across both modalities, answering multiple-choice comprehension questions. The NL condition provided structured descriptions including spatial configuration, element locations, and relationships. The VA condition let users explore graphs by touch — finger contact on data points triggered pulsing vibrations and spoken labels, while tracing along lines produced steady vibrations that stopped when the finger left the line.
Key findings
The two presentation modalities were functionally equivalent for learning accuracy — no statistically significant difference was found between vibro-audio (63.89% overall accuracy) and natural language (67.13%) conditions (F(1,17) = 0.959, p = 0.341). However, graph type had a significant effect on accuracy (F(2,16) = 15.338, p < 0.001), with Venn diagrams yielding the highest accuracy (~85% for both modalities), followed by line graphs (~60%), and pie charts (~50%). Natural language was significantly faster than vibro-audio (5:44 vs 6:29 average completion time, p = 0.039). Two-thirds of participants (N=12) preferred natural language, citing familiarity from screen reader use. However, five participants wanted both modalities available, recognizing that certain content types (spatial information) may be better conveyed through haptic exploration while other content (numerical values, statistics) may be more efficiently communicated through language. Participants suggested NL descriptions should be segmented by sentence to allow skipping to relevant sections rather than replaying the entire description. The vibro-audio condition was praised for providing rapid access to specific spatial information despite being a novel interaction for most participants.
Relevance
This research has direct practical implications for STEM education accessibility. The finding that a low-cost, commercially available iPhone can deliver graph comprehension comparable to traditional methods challenges the reliance on expensive tactile graphics and specialized equipment. For educators and content creators, the study validates that well-structured natural language descriptions of graphs can be as effective as haptic exploration for learning — but also that offering both modalities gives users control over their own learning experience. The emphasis on layering different types of information (text, numerical, spatial) across complementary modalities aligns with universal design principles and could inform how data visualizations are made accessible more broadly. The study also highlights the scale of the problem: approximately 7.5 million Americans are blind or have low vision, with ~50% under/unemployment rates, and inaccessible STEM content is a contributing factor. The multimodal approach — using hardware already in users' pockets — offers a scalable, cost-effective path to inclusive STEM education.
Tags: blind and low vision · STEM accessibility · data visualization · haptic technology · multimodal interaction · touchscreen · education accessibility · tactile graphics