Write-N-Speak: Authoring Multimodal Digital-Paper Materials for Speech-Language Therapy
Anne Marie Piper, Nadir Weibel, James D. Hollan · 2011 · ACM Transactions on Accessible Computing · doi:10.1145/2039339.2039341
Summary
This paper presents Write-N-Speak, a digital pen toolkit that enables speech-language therapists to create custom multimodal therapy materials for clients with aphasia—a communication disorder often resulting from stroke that affects approximately 80,000 Americans annually. The research addresses a gap between expensive high-tech AAC devices (costing up to $8,000) and low-tech paper-based materials that lack interactivity. Write-N-Speak runs on the Livescribe Pulse smart pen, an affordable consumer device ($99) that integrates a processor, microphone, speaker, and infrared camera to read special dot-pattern paper. The toolkit has two modes: a paper editor mode where therapists draw regions on paper and assign audio playback, audio recording, or handwriting recognition functions; and a client mode for therapy use. Therapists can create interactive worksheets, program photos with voice recordings, and attach audio-enabled stickers to household objects. The research involved ten months of field study including observations at a retirement community and university speech clinic, followed by a 12-week deployment with one therapist-client dyad. The client (AF, age 88) had aphasia and apraxia of speech following a stroke one year prior. The therapist (JB) integrated Write-N-Speak into weekly hour-long sessions, using the system for 349 total minutes across the study period—123 minutes preparing materials and 226 minutes during therapy.
Key findings
The therapist created primarily audio playback regions (94 regions accessed 423 times) rather than record or handwriting recognition features. A key finding was that having the client record her own voice—rather than just listening to the therapist's voice—was therapeutically valuable for self-reflection on pronunciation. As the therapist explained: "It's her voice. She's listening to it and she's doing the task... If you're part of the process, you know, you will understand." Three distinct material types emerged: (1) photos of the client's living room programmed with object names for word-finding practice, (2) transparent stickers placed on household objects (flashlight, cup, spoon) that speak the object name when tapped, and (3) the Rainbow Passage diagnostic text recorded sentence-by-sentence for pronunciation review. The therapist valued customization: "You can customize it to make it really tailored to whatever you feel is appropriate for the resident." Learning curve was significant—the therapist needed several weeks to master authoring and design appropriate activities. The client had difficulty with complex gestures like double-tapping. Audio quality and volume required an external speaker. Despite challenges, the therapist requested to keep the system after the study concluded and planned to use it with other client populations including adolescents with autism. The sticker approach enabled novel therapeutic interactions with physical objects of varying shapes and materials, though the pen struggled with curved surfaces and black backgrounds.
Relevance
This research demonstrates how affordable consumer technology can be repurposed for specialized accessibility applications. The $99 Livescribe pen bridges the gap between expensive dedicated AAC devices and passive paper materials—a cost consideration particularly relevant for stroke survivors who may face financial hardship. For accessibility practitioners, the paper illustrates design principles for multimodal therapy tools: support face-to-face interaction without occluding visual cues; provide multiple representations of language (written, spoken, gestured); leverage existing paper-based practices rather than replacing them; enable customization for individual needs; and ensure portability across therapy settings. The finding that therapists preferred audio playback over handwriting recognition suggests prioritizing reliable, simple interactions over complex features. The research also highlights the importance of involving domain experts (therapists) in design—they discovered creative applications (stickers on objects, client voice recording) the researchers had not anticipated. The toolkit's potential extends beyond aphasia to dementia, autism, language learning, and early childhood education, suggesting broader applications for multimodal paper-digital interfaces in communication support.
Tags: aphasia · speech-language therapy · AAC · digital pen · multimodal interaction · older adults · stroke rehabilitation