Nonverbal Communication through Expressive Objects
Stephanie Valencia, Mark Steidl, Michael L. Rivera, Cynthia L. Bennett, Jeffrey P. Bigham, Henny Admoni · 2023 · Communications of the ACM · doi:10.1145/3610939
Summary
This paper explores how physical motion through an expressive object — a "sidekick" — can support nonverbal communication for augmented communicators (ACs) who use AAC devices. The work centres on Mark, a co-author with cerebral palsy who has used AAC for over 19 years and controls his device via a head switch. The fundamental challenge is that AAC devices only support verbal (speech) output, leaving ACs without the nonverbal communication tools that hearing speakers take for granted — hand raises, nods, facial expressions — for managing conversational turn-taking, signalling attention, and conveying social intent. Mark already uses strategies like an LED light strip to signal readiness to participate, but these are limited. Through a 12-month participatory design process involving Mark, his close conversation partners (family members T and D), and HCI researchers, the team co-designed a 3D-printed flag-like sidekick that mounts on Mark's wheelchair and is controlled by his existing head switch. The sidekick uses two servo motors to perform motions including a "rise and wave" to call for attention, a "timer" metronome motion to signal message composition is in progress, and a reset motion.
Key findings
During a two-month evaluation with diary entries over three weeks, the sidekick was used on 7 of 11 days, 2-5 times per conversation day. Partners noticed the sidekick 7 out of 7 times during meetings. Mark reported it was easy to use and helped manage turn-taking, particularly as a shortcut for his preprogrammed message "can you hold on a minute please." The timer motion was especially valuable — it conveyed that Mark was composing a message without requiring him to verbally interrupt. The sidekick was more useful in face-to-face interactions than virtual meetings, where camera positioning and screen size limited visibility. Surprisingly, it was most effective with "mid-circle" partners (acquaintances familiar with Mark but less skilled at reading his communication style) rather than close partners who already understood his cues or unfamiliar partners who needed broader introduction to AAC. The study also revealed practical barriers to assistive technology adoption: remembering to plug in the device daily, cable management with the head switch, and the lack of integration with the AAC device itself.
Relevance
This research opens an important and underexplored dimension of AAC accessibility: nonverbal communication. Most AAC research focuses on improving speech generation speed and accuracy, but this work demonstrates that the conversational barriers ACs face extend well beyond producing words. Turn-taking, signalling intent, and managing group dynamics are critical social functions that current AAC devices do not support. For accessibility practitioners, the paper illustrates how bespoke assistive technology co-designed with users can address highly personal communication needs. The participatory design methodology offers a model for long-term collaboration with people with disabilities, highlighting important lessons about recognising the expertise and prior problem-solving that users and families bring. The work also raises broader questions about assistive technology maintenance, DIY AT sustainability, and the barriers created when AAC platforms are not open to third-party extensions.
Tags: AAC · augmentative and alternative communication · cerebral palsy · nonverbal communication · co-design · participatory design · assistive technology · motor disabilities · turn-taking