Co-designing Socially Assistive Sidekicks for Motion-based AAC
Stephanie Valencia, Michal Luria, Amy Pavel, Jeffrey P. Bigham, Henny Admoni · 2021 · Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction (HRI '21) · doi:10.1145/3434073.3444646
Summary
This paper explores how socially assistive robots, framed as "sidekicks," could provide AAC users with a nonverbal channel of communication to support their conversational goals. While AAC devices enable speech-based communication through synthesized speech, they do not support the nonverbal behaviors — gestures, gaze, body orientation, pointing — that verbal speakers routinely use to take turns, regulate conversation dynamics, and express intentions. For AAC users with motor disabilities, producing these nonverbal behaviors is often physically challenging. The researchers conducted a two-step process: first, an online survey of four augmentative communicators (ACs) with motor disabilities to identify nonverbal communication needs, followed by a 6-hour accessible co-design workshop involving two ACs with cerebral palsy (aged 23 and 38, both with 18+ years of AAC experience), their close conversation partners (caregivers), and three professional puppeteers as motion experts. The workshop was structured in three phases: Telling (sharing communication stories and identifying challenges), Making (prototyping physical sidekick forms from craft materials), and Enacting (puppeteers improvising sidekick motions and behaviors). The workshop was designed with five "Access Commitments" to ensure meaningful AC participation: minimizing fatigue through breaks, sending prompts in advance, establishing preferred communication modes, allowing enough time for everyone to communicate, and making materials easy to reference.
Key findings
The workshop identified three key design parameters for nonverbal communication sidekicks: attention (the ability to draw, divert, or maintain a conversation partner's attention), precision (adding nuance to match an AC's intention and clarify meaning), and timing (conveying communication at the exact needed moment). Thirteen conversational goals were identified that a sidekick could support, including showing others the AC wants to participate, filling the silence gap while the AC types, softening AAC device messages, and inviting partners to look at the AC's screen. Puppeteers generated 40 different behaviors across four prioritized goals, revealing a consistent two-stage motion pattern: first an attention-getting motion, then a gesture conveying the specific message or purpose. The two AC-led teams created distinctly different prototypes reflecting personal communication styles — one anthropomorphic with expressive eyes for conveying emotion, the other a protest-sign-like object for signaling disagreement. When reviewing puppeteer-improvised behaviors post-workshop, the AC participant preferred "straightforward" motions and disliked behaviors that were "too distracting." The workshop also revealed how ACs already use creative nonverbal strategies: one uses a wheelchair-mounted light activated by a head switch to signal readiness to speak in college classes.
Relevance
This research opens a novel design space at the intersection of AAC, social robotics, and nonverbal communication. While most AAC research focuses on improving the speed or accuracy of text/speech generation, this work recognizes that communication is fundamentally multimodal and that nonverbal channels carry essential social information. The "sidekick" framing is strategically important — it positions the robot as supporting the AC's agency rather than acting independently, avoiding the risk of the technology speaking for or over the user. The Access Commitments developed for the workshop provide a practical, replicable model for making co-design activities accessible to people with significant motor and speech disabilities. For accessibility practitioners, the three design parameters (attention, precision, timing) offer a useful framework for evaluating any assistive communication technology. The work also highlights how AAC users' communication needs extend far beyond word production to encompass the full social dynamics of conversation.
Tags: AAC · nonverbal communication · socially assistive robots · co-design · human-robot interaction · cerebral palsy · motor disability · participatory design