The Tactile Graphics Helper: Providing Audio Clarification for Tactile Graphics Using Machine Vision
Giovanni Fusco, Valerie S. Morash · 2015 · ASSETS '15: Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility · doi:10.1145/2700648.2809868
Summary
This paper presents the Tactile Graphics Helper (TGH), a machine vision system that enables students who are blind or visually impaired to independently access clarifying information about tactile graphics without sighted assistance. Tactile graphics—raised line drawings, charts, and diagrams—are critical for STEM education, but format-related issues often prevent independent use. Students struggle with subtle differences between textures and line styles, cramped placement of lines and braille labels, and non-standardized formatting conventions. TGH uses a mounted camera positioned above the tactile graphic to track the user's fingers as they explore. The user can ask voice questions like "what is there?" (lists all features and quantities) or "what is this?" (identifies what the pointing finger is touching). The system recognizes the tactile graphic, rectifies camera perspective distortion, segments the user's hands from the background, detects fingertips using distance transforms, and maps finger positions to pre-annotated graphic regions. Each tactile graphic requires approximately 10 minutes of preparation using a custom annotation GUI. A human annotator marks regions with colors and associates metadata (title, type, characteristics like "solid" or "dotted", function like "axis line"). This information is stored in YAML files alongside matrix maps that associate each pixel with a label ID. Unlike existing audio-tactile systems (TTT, IVEO) that require specialized hardware, TGH aims to run on standard tablets using consumer cameras.
Key findings
A mixed-methods case study with three university STEM students with visual impairments (studying Earth Science, Psychology, and Linguistics) confirmed that format-related issues create real barriers. All three participants reported difficulties differentiating textures, identifying elements placed too close together, and understanding confusing label placement. One student recalled failing an elementary school test because she couldn't distinguish 3D geometric figures. Students reported using 20-50 tactile graphics per week but rarely in class because graphics arrived late. Quantitative results showed a significant effect of TGH on exploration time—students spent longer exploring graphics with TGH (mean reduction of 82.89 seconds without TGH), indicating more thorough examination. Students used TGH to "preview" graphics by asking "what is there?" before systematic exploration, and to confirm uncertain information (e.g., verifying that a texture was water versus land). Professional feedback from tactile graphics experts validated the approach. One noted that TGH could provide three functions: clarifying information, previewing graphics, and promoting effective exploration strategies. The 10-minute annotation time was considered acceptable compared to the hours required to create tactile graphics. However, participants disliked voice input/output due to noise concerns in classrooms, suggesting gesture or keyboard alternatives.
Relevance
This research addresses a critical gap in STEM accessibility: even when tactile graphics are provided, format limitations often make them unusable without sighted assistance. Survey data cited in the paper indicates that 93.3% of print graphics in braille science and math textbooks exist in tactile form, yet only 45% of blind students encounter 3+ tactile graphics per week, and 6% report never using them in school. The independence factor is crucial—requiring sighted help for every graphic creates dependency and limits learning opportunities. TGH's "preview" functionality is particularly valuable, enabling students to quickly understand a graphic's structure before detailed exploration, mimicking what sighted students do with a glance. For practitioners, the research highlights that providing accessible graphics is necessary but not sufficient. Annotation and augmentation systems that add contextual information can transform marginally usable graphics into effective learning tools. The finding that university STEM students still struggle with format issues (despite years of experience with tactile graphics) underscores that this is a persistent problem requiring technological solutions, not just better student training.
Tags: tactile graphics · visual impairment · blindness · STEM education · machine vision · finger tracking · audio-tactile · assistive technology · accessible graphics · braille