Introducing Multimodal Paper-Digital Interfaces for Speech-Language Therapy
Anne Marie Piper, Nadir Weibel, James D. Hollan · 2010 · Proceedings of the 12th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2010) · doi:10.1145/1878803.1878840
Summary
This paper explores how multimodal digital pen and paper technology can support speech-language therapy for adults with aphasia and apraxia of speech following stroke or brain injury. The researchers conducted five months of field observations with 15 speech-language therapists (three licensed and 12 students) working with adults in two settings: a continuous care retirement community and a university speech and language clinic. Field research identified key characteristics of therapy practice: it is inherently collaborative and social, multimodal (combining spoken, written, and auditory language representations), ongoing over extended periods, individually tailored to each client, and deeply rooted in pen and paper interaction. Based on these observations, the team developed a prototype using Livescribe Pulse Smart Pens — digital pens with built-in processors, infrared cameras, microphones, and speakers that can read special dot patterns printed on paper. The prototype transforms existing paper therapy worksheets into interactive materials: touching an image or word plays associated audio, handwriting is recognised and stored, and clients can record and replay their own speech for self-reflection. The system bridges familiar low-tech paper-based AAC approaches with the dynamic capabilities of high-tech digital devices.
Key findings
Prototype reviews with 15 therapists revealed strong enthusiasm for the paper-digital approach. Therapists valued the simplicity and familiarity of pen and paper interaction, particularly for older clients resistant to high-tech devices — one clinic supervisor noted that a client with a Lingraphica device had not touched it in nine months because it was "overwhelming." Human-generated audio (rather than synthesised speech) was considered critical for accurate modelling of pronunciation and intonation. Therapists saw particular value for clients who live alone and lack practice partners. Two case studies with older adults (ages 88 and 90) demonstrated feasibility: AF (88, aphasia and apraxia) learned the system within minutes, used interactive kitchen-themed worksheets for vocabulary and sentence completion, and was able to do independent homework activities with the pen. MW (90, severe apraxia, nearly unable to speak) used the pen as a pointing device with her existing laminated picture board, successfully communicating that she was cold and wanted a blanket. MW also used the system to call bingo numbers for fellow residents — a social engagement activity that made her smile and interact with peers despite years of social withdrawal. Challenges included the need for external speakers in noisy environments, difficulty with abstract concepts like tapping an answer box, visual search difficulties on crowded boards, and the current requirement for researchers to create custom content rather than therapists being able to generate it themselves. Cost was a significant advantage: the Livescribe pen cost under $200 compared to $8,000 for a Lingraphica device.
Relevance
This paper makes a compelling case for building assistive technology that works with existing practices rather than replacing them. The core insight — that speech-language therapy is fundamentally a paper-based, social, multimodal activity — challenges the assumption that high-tech digital-only AAC devices are always the best path forward. For accessibility practitioners, several lessons apply broadly: familiar form factors dramatically reduce adoption barriers, especially for older adults; human-generated audio is preferred over synthesised speech for communication support; the flexibility of paper (inexpensive, portable, cuttable, laminatable) enables rapid personalisation that rigid digital interfaces cannot match; and social engagement may be as therapeutically valuable as structured exercises. The study also highlights the tension between customisation and therapist workload — the system requires custom content creation that therapists cannot yet do independently, a barrier that must be solved for real-world adoption. The research underscores that device acceptance is deeply personal and emotional: as one therapist noted, introducing an AAC device is "effectively their voice," requiring careful, gradual introduction.
Tags: aphasia · apraxia of speech · augmentative and alternative communication · speech-language therapy · older adults · multimodal interaction · digital pen · paper-digital interface · stroke · communication board