Talking Points: The Differential Impact of Real-Time Computer Generated Audio/Visual Feedback on Speech-Like & Non-Speech-Like Vocalizations in Low Functioning Children with ASD
Joshua Hailpern, Karrie Karahalios, Laura DeThorne, James Halle · 2009 · Proceedings of the 11th International ACM SIGACCESS Conference on Computers and Accessibility (Assets '09) · doi:10.1145/1639642.1639675
Summary
This paper examines whether real-time computer feedback systems (CFS) can differentially impact speech-like versus non-speech-like vocalizations in low-functioning children with Autism Spectrum Disorder. The distinction matters because the goal of speech intervention is to increase speech-like vocalizations (sounds that can be phonetically transcribed) while ideally decreasing non-speech-like vocalizations (grunts, screeches, vocal stereotypies) that disrupt social interactions. Prior CFS research, including the authors' own earlier work, focused exclusively on speech-like vocalizations without examining the full picture. The system used was The Spoken Impact Project Software (SIPS), which translates a child's vocalizations into real-time audio and visual feedback — for example, a circle on screen whose diameter grows with volume and whose color changes every two seconds of sustained vocalization, paired with audio echoes or musical rewards proportional to utterance duration. Five nonverbal children with ASD (ages 3-8), all at the Preverbal Communication stage (developmentally comparable to 6-12 month olds), participated in six sessions each over approximately six months. Each 40-minute session consisted of eight 2-minute trials comparing baseline (black screen, no feedback) against three modality conditions: visual only, audio only, and mixed audio/visual feedback.
Key findings
CFS significantly impacted vocalization rate or duration in 4 of 5 children, though the specific outcomes varied considerably by individual and feedback modality. When all feedback types were combined, 3 of 5 children showed desirable effects of either increasing speech-like vocalizations (SLV) or decreasing non-speech-like vocalizations (NSLV), while 1 child showed an undesirable decrease in SLV duration. The SLV-to-NSLV ratio — the key measure of whether CFS differentially promotes speech — increased significantly for 2 of 5 children primarily through decreased NSLV rather than increased SLV, suggesting CFS may work more through suppression than encouragement. Critically, each child responded uniquely to different feedback modalities: Oliver responded best to audio-only feedback, Frank favored audio over visual, Diana responded primarily to specific visual feedback types, and Brian's NSLV was suppressed by visual-only conditions. Two of five children had significant responses to audio-only conditions, three to visual-only, and all five to mixed feedback. The negative correlation between SLV and NSLV rates for 2 of 5 children suggests a potential trade-off between vocalization types.
Relevance
This research makes an important methodological and practical contribution to technology-assisted speech interventions for children with ASD. The central insight — that CFS effects on speech-like and non-speech-like vocalizations must be measured separately — challenges the assumption that increasing total vocalization is inherently beneficial. A system that increases both speech and non-speech vocalizations equally may not be therapeutically useful, and one that primarily suppresses non-speech vocalizations (as observed here) operates through a different mechanism than one that encourages speech. For practitioners developing or deploying feedback systems, the dramatic individual differences across children underscore that no single feedback modality works universally — audio feedback should not be automatically discounted despite "conventional wisdom" favoring visual feedback for ASD populations. The small sample size (5 children) limits generalizability, but the within-subject design and six-month data collection period provide rich individual-level evidence. Future systems should be adaptable to each child's specific response profile rather than applying uniform feedback strategies.
Tags: autism spectrum disorder · speech development · computer feedback system · vocalization · children · assistive technology · affective computing · multimodal interaction