Investigating Day-to-day Experiences with Conversational Agents by Users with Traumatic Brain Injury
Hu, Yaxin, Lim, Hajin, Johnson, Hailey L., O'Shaughnessy, Josephine M., Kakonge, Lisa, Turkstra, Lyn, Duff, Melissa, Toma, Catalina, Mutlu, Bilge · 2023 · Proceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3597638.3608385
Summary
This paper presents a four-week in-the-wild field study investigating how nine adults with moderate-to-severe traumatic brain injury (TBI) in the chronic stage (1-19 years post-injury) used Amazon Alexa on Echo Dot devices in their homes. TBI can cause a constellation of cognitive, communication, sensorimotor, and psychological challenges that profoundly limit independence in everyday life, including impairments in memory, executive function, attention, language comprehension, emotional regulation, and social communication. While conversational agents (CAs) have shown promise as assistive technologies for various disability groups, little was known about how people with TBI actually use CAs to address injury-related needs in naturalistic settings. The study collected three types of data: weekly semi-structured interviews, Alexa dialog history (8,313 entries across 1,015 dialog blocks), and contextual audio recordings from a tablet computer (1,402 recordings). During the first two weeks, participants explored the CA freely; after two weeks, researchers provided a list of recommended Alexa features to try. Thematic analysis of the combined data revealed two complementary sets of findings: first, the diverse injury-related symptoms and accessibility challenges participants faced when using the CA; and second, 14 distinct TBI-related activities that participants engaged in with the CA, organized into four categories — mental health support, cognitive activities, healthcare and rehabilitation, and routine activities.
Key findings
The study achieved an acceptable overall System Usability Scale score (Mean = 81.7, SD = 11.7), with four of nine participants reporting increased use and improved interaction quality over the four weeks. Participants reported cognitive challenges (short-term memory loss in all nine, concentration difficulties, mental fatigue, executive function impairments), psychological changes (heightened emotions, stress, anxiety, depression, negative self-perception, loneliness), communication challenges (word-finding difficulties, speech rate issues, social communication problems including pseudobulbar affect), and sensory challenges (sound sensitivity, loss of smell/taste, screen sensitivity). These symptoms created specific CA accessibility barriers: difficulty finding and learning features due to memory impairments, intense emotional responses to interaction failures and self-doubt about whether errors were TBI-related, challenges with turn-taking timing as the CA interrupted before participants could formulate responses, and sound sensitivity making the CA voice "grating." The 14 TBI-related activities revealed creative and often unintended uses: participants used the Notes feature to memorize vocabulary while reading and log activity times; social chat and trivia games served as cognitive stimulation exercises; the CA became a companion alleviating loneliness during recovery; and participants recorded thoughts before therapist visits to compensate for memory loss. Notably, participants used the CA for self-disclosure of TBI experiences and negative emotions, but the CA consistently failed to respond appropriately to emotional content, either ignoring it or changing topics. Novel use cases emerged that were not designed into the system, such as summarizing emails, recording pre-therapy thoughts, and using notes as a searchable vocabulary dictionary.
Relevance
This study fills a critical gap in understanding how people with acquired cognitive disabilities use mainstream voice technology in daily life, moving beyond laboratory evaluations to capture naturalistic, long-term usage patterns. The findings have immediate implications for CA designers and developers: current systems fail people with TBI through inflexible turn-taking timing, inadequate responses to emotional distress, confusing system updates that invalidate learned interaction patterns, and voice characteristics that aggravate sensory sensitivities. The design recommendations are organized by TBI symptom category — cognitive (repeat commands, recall previous interactions, adapt to system updates), psychological (provide clear error explanations, avoid pretending to empathize, allow personality customization), communicative (extend listening time, adapt speech pace, offer word suggestions), and sensory (provide granular voice customization for tone, pitch, and speed). Beyond TBI specifically, these findings are relevant to the broader population of people with cognitive impairments from stroke, dementia, or aging, who face similar challenges with memory, attention, and executive function. The study also raises important ethical considerations about CAs as mental health tools — participants disclosed significant emotional distress to a system incapable of appropriate response, highlighting the need for CAs to at minimum direct users to professional support resources.
Tags: traumatic brain injury · conversational agents · voice assistants · cognitive accessibility · assistive technology · mental health · rehabilitation · smart speakers · in-the-wild study