Multimodal Trajectory Playback for Teaching Shape Information and Trajectories to Visually Impaired Computer Users
Andrew Crossan, Stephen Brewster · 2008 · ACM Transactions on Accessible Computing (TACCESS) · doi:10.1145/1408760.1408766
Summary
This research investigates how haptic (force-feedback) and audio trajectory playback can teach shapes and gestures to blind and visually impaired computer users. The fundamental challenge addressed is that spatial information—shapes, diagrams, graphs, maps—is difficult to convey through linear screen reader output or static tactile diagrams. The researchers developed a system using the PHANTOM OMNI force-feedback device that physically guides users' hands through shape trajectories, combined with audio cues mapping position to sound parameters. Three studies were conducted. The first compared nine visually impaired participants from the Royal National College for the Blind with six sighted participants on a haptic-only trajectory reproduction task. Participants felt shapes played through the force-feedback device five times, then attempted to recreate them three times. The second study tested whether adding audio feedback—pitch representing vertical position (200-400Hz) and stereo pan representing horizontal position—could improve performance for visually impaired users. The third study explored a collaborative application where a sighted "Describer" could drag a visually impaired "Drawer" through shapes to communicate complex diagrams that are difficult to describe verbally.
Key findings
The first study revealed that visually impaired users found haptic trajectory playback significantly harder than sighted users (51.1% vs 74.7% shape recognition rate, p < 0.03). Critically, participants blind from birth performed worst, suggesting that experience with visual shape concepts affects haptic learning. The high variability among visually impaired participants indicates trajectory playback works well for some individuals but poorly for others. The multimodal study demonstrated significant benefits of combining haptics with audio: approximately 80% of shapes were correctly recognized with haptic-audio playback versus 70% with haptic alone (p < 0.05). Interestingly, vertical accuracy improved significantly (users found pitch intuitive), but horizontal accuracy did not—participants reported difficulty attending to stereo pan while also processing haptic information. The collaborative study revealed that trajectory playback serves as a valuable communication channel when verbal description fails. For simple, nameable shapes ("draw a circle"), verbal description sufficed. For complex abstract shapes, participants naturally turned to trajectory playback, with Drawers describing the felt shape back to Describers ("like headphones on its side") to confirm understanding.
Relevance
This research has direct implications for accessible STEM education, where diagrams, graphs, and geometric concepts are fundamental but poorly served by current assistive technologies. The finding that multimodal feedback significantly outperforms haptic-only suggests that accessible graphics systems should combine multiple sensory channels rather than relying on touch alone. For practitioners, the collaborative findings are particularly actionable: trajectory playback can augment verbal description in teaching contexts, allowing instructors to physically guide students through shapes when words prove inadequate. The observation that those blind from birth struggled most indicates that spatial concept training may need to start early and use different approaches for congenitally blind versus adventitiously blind individuals. The technical limitations identified—sharp corners being rounded by playback lag, difficulty distinguishing trajectory from centering movements—provide specific design guidance for future haptic accessibility tools. Consumer-grade force-feedback devices are now more affordable than in 2008, making these techniques increasingly practical for educational settings.
Tags: haptic technology · multimodal interaction · visual impairment · force feedback · tactile graphics · sonification · STEM accessibility · gesture