Tactile Graphics with a Voice
Catherine M. Baker, Lauren R. Milne, Ryan Drapeau, Jeffrey Scofield, Cynthia L. Bennett, Richard E. Ladner · 2016 · ACM Transactions on Accessible Computing (TACCESS) · doi:10.1145/2854005
Summary
This paper presents Tactile Graphics with a Voice (TGV), a system that uses QR codes and a smartphone application to make text labels on tactile graphics accessible to blind and low-vision users — particularly the estimated 60% of blind people who are not fluent in Braille. Textbook images in STEM subjects frequently contain numerous text labels (axis values, data points, geometric measurements) that are traditionally converted to embossed Braille when creating tactile graphics. TGV replaces Braille labels with QR codes that encode the text directly, which users scan using an iOS smartphone application. The app provides three feedback modes to assist with camera aiming: Silent (no guidance, scans automatically when a code is found), Verbal (spoken feedback about QR code count and position on screen), and Finger Pointing (uses computer vision to detect the user's finger via skin-colour detection and reads the QR code closest to their fingernail). The authors also developed algorithms for automatically placing QR codes on tactile graphics without overlapping image features, integrated with the Tactile Graphics Assistant pipeline. A six-session longitudinal study with 10 blind and low-vision participants evaluated all three modes, with a Braille comparison in the final session. The paper additionally describes preliminary work on a Google Glass version using a head-mounted camera to free both hands for tactile exploration.
Key findings
TGV achieved 88-89% accuracy across all three feedback modes, comparable to or higher than the 77% accuracy achieved with Braille labels by Braille-literate participants. Critically, four participants who were not Braille-fluent could correctly answer questions using QR codes but could not use the Braille equivalents at all — demonstrating TGV's value as an alternative access method. Task completion time decreased significantly across sessions (p < 0.05), dropping from about 55 seconds in Session 1 to about 25-30 seconds by Session 6, approaching Braille speed (28.6 seconds). QR codes can encode 45% more text in the same physical area as Braille labels, offering a density advantage. Feedback mode preferences varied dramatically: four participants preferred Silent, four preferred Finger Pointing, and two preferred Verbal — and half the participants changed their favourite mode between first and last sessions. This underscores the need to support all modes simultaneously rather than choosing a single "best" approach. Participants who preferred Silent mode were fatigued by constant audio feedback from screen reader use. Finger Pointing was most accurate but slowest due to the requirement to keep the finger visible to the camera while not obscuring the QR code. Braille-literate participants still preferred Braille for its directness and reliability, but acknowledged QR codes' benefits for those without Braille skills.
Relevance
TGV addresses a fundamental access gap: the majority of blind people cannot read Braille, yet Braille remains the dominant method for labelling tactile graphics in education. By using QR codes and mainstream smartphones — technology most blind users already own — TGV provides an immediately deployable alternative that requires no specialized hardware. For STEM educators and transcribers creating accessible materials, QR codes can be generated freely online and physically resemble standard labels, fitting into existing tactile graphic production workflows. The strong individual variation in feedback mode preferences is a critical design lesson applicable across accessibility tools: there is no single best interaction mode, and systems must offer choice. The Google Glass exploration points toward a future where head-mounted cameras free both hands for tactile exploration — a significant ergonomic improvement over juggling a phone and a tactile graphic simultaneously. For the broader accessibility field, TGV demonstrates that creative use of existing consumer technologies can bypass long-standing barriers at minimal cost.
Tags: tactile graphics · visual impairment · QR codes · STEM accessibility · Braille · smartphone · camera-based assistive technology · education