Watch Your Language: Using Smartwatches to Support Communication
Humphrey Curtis, Timothy Neate · 2023 · ASSETS '23: Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/3597638.3608379
Summary
This paper presents the first participatory design and evaluation of AAC smartwatch applications co-designed directly with people who have complex communication needs (CCNs), specifically people living with aphasia — an acquired language impairment most commonly caused by stroke that can affect speaking, reading, writing, and comprehension. The authors argue that current high-tech AAC devices (typically tablets or dedicated devices) have critical shortcomings: they are socially conspicuous, poorly portable, aesthetically unconsidered, and — most importantly — they obstruct users' non-verbal communication pathways by creating a physical barrier between communication partners. The research builds on the concept of "total communication," which recognizes that people with CCNs creatively leverage all available communication pathways including gesture, facial expression, eye gaze, body language, and props alongside verbal speech. The team conducted three participatory design workshops with four people with aphasia and a speech and language therapist (SLT) at the Aphasia Re-Connect charity. Workshops progressed from exploring contextual communication challenges using tangible context cards and communication grids, through low-fidelity paper prototyping, to iterating functional Apple Watch prototypes. This process produced two complementary iOS smartwatch apps: Watch Out, designed for public face-to-face communication with strangers, and Watch In, designed for private cognitive support including memory, word-finding, and mindful breathing.
Key findings
Watch Out provides eight key expressions co-designed by participants for on-the-go use (e.g., "Could you please let me have your seat?", "Please speak more slowly", "I have had a stroke and aphasia"), with voice synthesis, a buzzer for attention, a rotatable display that serves as a non-verbal prop for onlookers, a blue disability badge display, and a QR code linking to aphasia information. Watch In offers conversation transcription via the smartwatch microphone with text-to-speech playback for memory support, a phoneme-based word-finding dictionary ("Picker"), haptic breathing exercises, and the ability for SLTs or family to add custom words. During an experience prototyping evaluation with an actor and five participants with aphasia, interaction analysis of 244 coded instances showed that the smartwatch apps did not restrict users' natural communication — participants produced nearly equivalent eye-gaze on the watch (N=33) and the actor (N=30), and freely used bodily gestures (N=36), proxemic shifts (N=15), and verbal dialogue (N=34) alongside the apps. Participants used the watch primarily as a communicative prop (N=14), showing the display to the actor while supplementing with speech and gesture. All participants reported the apps gave them "more confidence" in their communication. Feedback questionnaires showed overwhelmingly positive responses, with 20 instances of strong agreement and 6 of agreement across positively phrased questions in the experience prototyping session. The focus group revealed strong interest in Watch Out for practical scenarios like requesting seats on public transport — one participant described being denied a seat because strangers did not recognize his invisible disability.
Relevance
This research challenges a fundamental assumption in AAC design: that the primary purpose of an AAC device is to produce accurate verbal output. Instead, the authors demonstrate that AAC should augment users' existing total communication strategies rather than replace them. The smartwatch form factor offers several advantages that practitioners should consider: discretion (it does not signal disability the way a tablet-mounted AAC device does), portability (always on the wrist, no bag or mounting needed), aesthetic acceptability (smartwatches carry positive social connotations of fitness and fashion), and crucially, it does not create a physical barrier between communication partners. For people with aphasia who also have hemiplegia — as three participants did — a wrist-worn device is far more accessible than a tablet requiring two-handed operation. The participatory design approach offers a model for co-designing with people with communication disabilities, using tangible materials like context cards, communication grids, and paper prototyping to scaffold ideation. The work highlights that AAC should serve as a "safety net" for challenging communication moments rather than a permanent replacement for natural communication, and that future AAC research should more seriously consider the role of the communication partner in co-constructing meaning.
Tags: augmentative and alternative communication · AAC · smartwatch · aphasia · wearable technology · participatory design · total communication · non-verbal communication