Comparing Tactile, Auditory, and Visual Assembly Error-Feedback for Workers with Cognitive Impairments
Thomas Kosch, Romina Kettner, Markus Funk, Albrecht Schmidt · 2016 · Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '16) · doi:10.1145/2982142.2982157
Summary
This paper compares three modalities of error feedback — tactile, auditory, and visual — for cognitively impaired workers performing manual assembly tasks at sheltered work organizations. Industrial manufacturing companies increasingly outsource assembly tasks to these organizations, where workers with cognitive impairments assemble products under supervisor guidance. Assistive systems using in-situ projection can provide step-by-step pictorial assembly instructions and automatically detect errors, but the optimal way to communicate errors to cognitively impaired workers had not been scientifically studied. The researchers extended an existing projector-based assistive workstation with three error feedback modalities: visual feedback projected red light over the entire work area (2500ms duration); auditory feedback played a deep error tone through a Holosonics Audio Spotlight directional speaker that only the worker could hear (2000ms); and tactile feedback delivered vibration through two motors embedded in a worker's safety glove on the index and ring fingers (1200ms, alternating pattern). The assembly tasks used Lego Duplo bricks picked from 2x4 bins, with three tasks of increasing complexity (24 bricks each). A Kinect v2 camera automatically detected picking errors, while assembly errors were triggered by a researcher using a Wizard of Oz approach. Sixteen workers from a partner sheltered work organization participated (ages 34-53, mean 40.3), categorized by Performance Index (PI) into three capability groups (5-15%, 20-35%, and above 40%).
Key findings
Four participants from the lowest PI group (5-15%) aborted the study because they did not want to wear the vibrating glove, leaving 12 participants for analysis. Visual error feedback led to significantly faster task completion times (mean 434.42 seconds) compared to tactile feedback (575.42 seconds, p = .021, large effect size eta-squared = .296). Auditory feedback fell between (445.08 seconds) but was not significantly different from either. The number of assembly errors and picking errors did not significantly differ across modalities, suggesting that feedback modality affects error recovery speed rather than error prevention. In subjective rankings, visual feedback was rated best (5 first-place, 4 second, 2 third), auditory second (4 first, 4 second, 3 third), and tactile worst (2 first, 3 second, 6 third). Qualitative observations revealed important nuances: 10 of 12 participants had difficulty distinguishing between similar-colored Lego bricks; some participants did not perceive the tactile vibration at all, while others found it "unpleasant" or "distracting"; three participants were scared by the auditory error tone; and workers generally did not care about error privacy because supervisors normally observe and report errors anyway — a key difference from non-disabled worker populations where error privacy is valued. Several participants found it "very easy" to work with any error feedback system and expressed willingness to use it daily.
Relevance
This research addresses the practical challenge of designing assistive workplace systems for cognitively impaired workers — a population that is underserved by mainstream workplace technology research. For accessibility practitioners, several findings are directly applicable: visual feedback in the worker's field of view is most effective for cognitively impaired users, likely because it requires the least additional cognitive processing; tactile feedback through vibrating gloves may be problematic due to sensory sensitivity issues common in this population (some participants couldn't feel it, others found it aversive); and auditory error feedback, while effective, can trigger anxiety responses (being "scared" by error sounds), suggesting that sound design for this population requires special care. The finding that error privacy is not a concern for these workers — because supervisor oversight is an expected and accepted part of their work environment — challenges assumptions imported from research with non-disabled workers and underscores the importance of studying the actual target population rather than generalizing from proxy groups. The study also highlights an ethical dimension: assistive workplace technology can support the UN Convention on the Rights of Persons with Disabilities by enabling meaningful employment participation.
Tags: cognitive accessibility · intellectual disability · workplace accessibility · assistive technology · multimodal interaction · haptic technology · augmented reality · disability employment
Standards referenced: United Nations Convention on the Rights of Persons with Disabilities