Design Goals for a System for Enhancing AAC with Personalized Video
Katie O'Leary, Charles Delahunt, Patricia Dowden, Ivan Darmansya, Jiaqi Heng, Eve A. Riskin, Richard E. Ladner, Jacob O. Wobbrock · 2012 · Proceedings of the 14th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '12) · doi:10.1145/2384916.2384964
Summary
This demonstration paper presents Vid2Speech, a prototype tablet-based augmentative and alternative communication (AAC) system designed for children with complex communication needs (CCN). The system addresses a fundamental limitation of existing AAC applications: their reliance on static symbols to represent action words. While static icons work reasonably well for concrete nouns (a photograph of a ball clearly represents "ball"), action words like "want," "swing," or "slide" are dynamic and ephemeral in nature, making them poorly suited to static iconic representation. The authors argue that personalized video offers a promising solution because it captures the dynamic quality of actions in familiar settings relevant to the individual user. Vid2Speech was developed at the University of Washington as an iPad application targeting beginning communicators who are preliterate and preverbal. The system allows caregivers and therapists to capture short video clips of everyday activities and associate them with action words in the AAC interface. The paper situates this work within the growing consumer market of AAC iPad apps that emerged after 2010, including Proloquo2Go, Tap to Talk, and Voice4u, noting that while these apps have made AAC more accessible, they all rely on static symbols. The authors draw on prior research showing that motion effects and animations improve recognition of action words compared to static icons, and that personalized content filmed in familiar environments enhances learning.
Key findings
The paper articulates three core design goals for integrating personalized video into AAC systems. First, providing social-temporal navigation allows users to browse video content organized by social context (who was involved) and temporal context (when activities occurred), rather than relying solely on traditional categorical or alphabetical organization. Second, enhancing comprehension leverages the inherent advantage of video over static symbols for representing dynamic action words, building on evidence that cartoon animations of action words are easier to recognize than static icons and that black-and-white video of familiar people performing actions improves learning for children with intellectual disabilities. Third, enabling real-time customization allows caregivers and communication partners to capture and add new video content on the spot using the tablet's built-in camera, making the system adaptable to the child's immediate environment and communication needs. The Vid2Speech prototype demonstrates these goals through an interface where tapping a video cell plays the associated clip and generates speech output of the corresponding word.
Relevance
This paper is significant for AAC practitioners and developers because it provides a structured framework for thinking about how personalized video can enhance communication tools for children with complex communication needs. The three design goals -- social-temporal navigation, enhanced comprehension, and real-time customization -- remain relevant to modern AAC app development. As tablets and smartphones have become even more capable since 2012, the feasibility of video-based AAC has only increased. The work highlights the importance of moving beyond static symbols, particularly for action vocabulary, and empowering caregivers to personalize AAC content in the moment. While this is an early-stage prototype paper rather than a formal evaluation study, the design goals it establishes provide a useful starting point for researchers and developers working on next-generation AAC systems.
Tags: augmentative and alternative communication · complex communication needs · personalized video · action words · tablet-based AAC · children with disabilities · speech-generating devices