Aided Nonverbal Communication through Physical Expressive Objects
Stephanie Valencia, Mark Steidl, Michael Rivera, Cynthia Bennett, Jeffrey Bigham, Henny Admoni · 2021 · Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3441852.3471228
Summary
This paper presents an in-depth case study of co-designing a physical expressive "sidekick" device to support nonverbal communication for an augmented communicator (AC) named Mark, who has cerebral palsy and uses a head-switch-operated AAC device. While AAC systems enable speech generation through text-to-speech, they do not support the nonverbal aspects of conversation — such as signaling a desire to take a turn, indicating that a message is being composed, or calling for attention — that are critical for successful interaction. Experienced conversation partners learn to read an AC's body language and communication style, but unfamiliar partners often miss these cues, creating a significant barrier to social participation. The research team — including Mark as co-author and co-designer, his close communication partners (parents), and HCI researchers from Carnegie Mellon University — worked together over 12 months through interviews, surveys, prototyping sessions, and diary entries. They progressed through discovery (identifying conversational challenges), definition (establishing design goals and constraints), development (ideating and prototyping), delivery (pilot testing and iterating), and evaluation (two-month in-context use with three weeks of diary studies). The final sidekick is a 3D-printed, flag-like mechanical arm mounted on Mark's wheelchair, actuated by two micro servo motors and controlled via Mark's existing head switch. It performs two main motions: a "timer" motion (swaying side to side for one minute to signal Mark is composing a message) and a "wave" motion (waving back and forth to call for attention).
Key findings
Over three weeks of diary entries, Mark used the sidekick on 7 of 11 days, typically 2-5 times per conversation. Mark reported that partners noticed the sidekick 7 out of 7 times it was used. The sidekick was most effective in face-to-face conversations — Mark described it as "fantastic in real-time with real people" — but was less effective in online meetings where camera positioning could cut it off and large groups (30+ people) made it impractical. Close communication partners did not need the sidekick since they already read Mark's body language, but they understood its purpose immediately. The most impactful use was with "mid-circle" partners — acquaintances familiar with Mark but not as skilled as close partners — where the sidekick helped pace conversations and manage turn-taking. The timer motion effectively replaced Mark's preprogrammed verbal phrase "can you hold on a minute please," providing a nonverbal shortcut. Key design lessons included: establishing what does not work accessibly upfront (Mark and his family had already explored and rejected many input modes like eye tracking and facial gestures due to his mixed muscle tone from athetoid cerebral palsy); keeping designs practical to avoid unusable prototypes; understanding that co-design concerns process accessibility, not just product; and planning for long-term device maintenance. The team also learned that "less is more" — the final sidekick had only two motions, pared down from initial explorations of five or more, because Mark did not want to remember numerous states and controls.
Relevance
This research opens an important new direction in AAC by moving beyond speech generation to address the nonverbal communication channel that is equally vital for conversational success. For AAC practitioners, the key insight is that augmented communicators need support not just for what they say but for how they participate in conversation — managing turn-taking, signaling intent, and maintaining social presence. The co-design methodology offers a model for deeply involving AAC users and their support networks in technology development, while also surfacing important lessons about the limitations of participatory design when hardware iteration requires specialized skills. The concept of motion-based AAC — using physical movement of an object to convey communicative intent — represents a genuinely novel modality that complements rather than competes with existing speech-generating devices. The finding that different conversation partners in an AC's social circle have different communication needs suggests that future AAC systems should support variable complexity: simpler, more universal signals for unfamiliar partners and richer, more nuanced communication for close partners. The paper also honestly addresses the tension between bespoke assistive technology (customized for one person) and scalability, noting that maintenance and long-term support remain unsolved challenges.
Tags: augmentative and alternative communication · nonverbal communication · co-design · participatory design · cerebral palsy · physical disability · robotics · tangible interaction · assistive technology