Multimodal Feedback: Establishing a Performance Baseline for Improved Access by Individuals with Visual Impairments
Holly S. Vitense, Julie A. Jacko, V. Kathlene Emery · 2002 · Proceedings of the Fifth International ACM Conference on Assistive Technologies (Assets 02) · doi:10.1145/638249.638260
Summary
This study investigates how different combinations of auditory, haptic, and visual feedback affect the performance of fully sighted users during a complex direct manipulation task. The researchers designed the study as a baseline from which to later compare the performance of users with visual impairments, reasoning that effective multimodal feedback combinations for sighted users would likely benefit those who rely on non-visual channels. Thirty participants completed a drag-and-drop file management task using custom software that could deliver feedback through any combination of three modalities: auditory tones, haptic vibrations via a force feedback mouse, and visual highlights. The study employed a 2x2x2 factorial design testing all eight possible feedback conditions (including a no-feedback control). Performance was measured through three dependent variables: subjective mental demand (from the NASA Task Load Index), objective mental workload via pupil diameter (using eye tracking), and target highlight time (THT), which captured how quickly participants could locate and act on interface targets. The research is grounded in the understanding that the 10 million blind and visually impaired individuals in the United States need interfaces that communicate through channels beyond vision alone, and that establishing how sighted users respond to multimodal feedback is a necessary first step toward designing universally accessible interfaces.
Key findings
The results revealed that multimodal feedback significantly improves user performance across all three measures. For mental demand, visual feedback had a significant main effect — its presence reduced perceived workload. Auditory, haptic, and visual feedback each independently led to low mental demand, with the bimodal condition of haptic/visual producing the lowest demand ranking (26.95). For pupil diameter (objective workload), the trimodal condition of auditory/haptic/visual feedback produced the smallest pupil diameter (3.556 mm), indicating the lowest cognitive load. This was significantly smaller than both the bimodal auditory/visual and bimodal auditory/haptic conditions. For target highlight time, haptic feedback alone produced the fastest performance (0.499 seconds), and the bimodal haptic/visual condition was nearly as fast (0.537 seconds). Notably, adding auditory feedback to haptic conditions actually increased THT, suggesting that auditory feedback can interfere with the speed benefits of haptic input. The optimal feedback configurations differed by measure: auditory/haptic/visual was best for reducing workload, haptic/visual was best for task speed, and haptic alone or haptic/visual best supported physiological workload. A key finding was that haptic feedback is a comparable alternative to visual feedback, which has important implications for users who cannot rely on visual cues.
Relevance
This early Assets conference paper is foundational for understanding how multimodal interfaces can improve accessibility for people with visual impairments. Its finding that haptic feedback can substitute for visual feedback is directly relevant to designing accessible applications — suggesting that force feedback devices and vibration-based cues can meaningfully support users who have limited or no vision. The research methodology of establishing sighted-user baselines before testing with visually impaired populations provides a rigorous framework that later accessibility researchers have built upon. For practitioners, the key takeaway is that simply adding more feedback modalities is not always better — the specific combination matters, and auditory feedback can sometimes interfere with haptic performance. This nuanced understanding helps designers make informed choices about which feedback channels to combine in accessible interfaces. The study is limited by its focus on sighted participants only, and the specific drag-and-drop task may not generalize to all interaction patterns, but it provides essential groundwork for multimodal accessible design.
Tags: multimodal interaction · haptic feedback · auditory feedback · visual impairment · workload measurement · pupillometry · direct manipulation · assistive technology