A Mobile Phone Based Personal Narrative System
Rolf Black, Annalu Waller, Nava Tintarev, Ehud Reiter, Joseph Reddington · 2011 · The Proceedings of the 13th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2011) · doi:10.1145/2049536.2049568
Summary
This paper describes the development and evaluation of a mobile phone-based data collection system that supports personal narrative for children with severe speech and physical impairments (SSPI). Personal narrative — the ability to tell others about one's experiences — is essential for social interaction, identity formation, and language development, but current AAC devices focus primarily on transactional communication (requesting, answering) rather than storytelling. Building on the earlier "How was School today?" (HwSt) proof-of-concept project, which used sensor data and natural language generation (NLG) to automatically create narratives about a child's school day, this follow-on system uses a Nokia 6212 Classic mobile phone to gather richer data throughout the day. School staff use the phone to track interactions with people and objects via NFC/RFID tags (attached to staff cards, location posters, and frequently used objects), record multi-part voice message sets, and take photographs. Parents can also record messages at home. The phone automatically transmits all collected data to a remote server, which feeds into a narrative generation system on the child's PC-based VOCA (DynaVox Vmax or Tobii C12). The system was designed with extensive input from school staff and parents, addressing feedback from the earlier project that called for same-day narrative access, narratives immediately after data collection, smaller devices, and the ability to print generated narratives.
Key findings
The system was evaluated with two participants at a special school over several weeks. Peter (10 years old, athetoid cerebral palsy, non-ambulatory, minimal functional speech, uses gestures, head pointing, and an E-Tran frame) and Martin (17 years old, chromosomal disorder, ambulatory, no functional speech, uses Makaton signs and a GoTalk VOCA). Both participants' school staff successfully tracked interactions, recorded voice messages, and took photographs. Location tracking via QR barcodes proved unreliable due to camera resolution limitations and was replaced by RFID tags, which worked reliably. Voice recordings from school captured rich experiential content — Peter's messages about swimming therapy included four detailed sequential recordings about the pool activities. Home recordings captured family experiences and holiday stories. A transcribed interaction between Martin and his mother, mediated by phone playback of voice recordings, demonstrated genuine interactive storytelling with anticipation-building, humour, and natural conversational flow. The automatic narrative generation system successfully created messages like "Martin was there" from interaction data and identified interesting stories from location and timetable data. Early trials of the Grid 2 VOCA interface showed children could access photographs, voice recordings, and auto-generated messages, though Martin found switch-based sequential playback easier than touchscreen navigation.
Relevance
This research addresses a fundamental gap in AAC: while most devices support functional communication, they do little to help users share personal experiences and stories — activities that are central to building relationships, developing identity, and practising language skills. The system's approach of combining ambient data collection (NFC tags, photos, voice recordings) with automatic narrative generation is innovative because it reduces the cognitive and physical effort needed to construct a story, while still giving the child control over which stories to tell. For AAC practitioners and developers, the work highlights that narrative is not a luxury feature but an essential communication function that technology should support. The use of mainstream mobile phones rather than specialist equipment kept costs low (under £1 for data transfer over the entire project). The participatory design approach — incorporating feedback from children, parents, and school staff across multiple iterations — demonstrates best practice for developing assistive technology in educational settings. The rich qualitative data, including the transcribed mother-son interaction, provides compelling evidence that mediated storytelling can support natural, emotionally engaged communication.
Tags: AAC · personal narrative · cerebral palsy · natural language generation · mobile technology · children · severe speech and physical impairments · RFID · NFC · assistive technology · storytelling