A3: A Coding Guideline for HCI+Autism Research Using Video Annotation
Joshua Hailpern, Karrie Karahalios, James Halle, Laura DeThorne, Mary-Kelsey Coletto · 2008 · Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility (Assets '08) · doi:10.1145/1414471.1414476
Summary
This paper presents A3 (Annotation for ASD Analysis), a systematic coding guideline developed through cross-disciplinary collaboration between computer scientists, special educators, and speech and hearing scientists at the University of Illinois. The work addresses a significant gap in HCI research: the lack of established quantitative coding systems for studying low-functioning children with autism spectrum disorder (ASD) who interact with computer systems providing auditory and visual feedback. The research context is a four-month study examining how different forms of audio and visual feedback affect the motor and verbal responses of five low-functioning children with autism, aged 3-8 years. The computer system reinterprets children's vocalizations into visual and auditory feedback — for example, a circle that changes diameter proportional to sound duration, or music clips played back when sound production ceases. To support the coding workflow, the team created two open-source tools: VCode for video annotation and VData for agreement and reliability checking, designed specifically to bridge coding practices between Computer Science and Behavioral Sciences. The A3 methodology uses a multi-pass annotation approach across four passes, each examining different behavioral variables including non-child audio, smiling, screen orientation, auditory focus, laughter, vocalizations (speech-like vs. non-speech), turn-taking, imitation patterns, and time in chair.
Key findings
The overall inter-rater agreement (IRA) across all dependent variables was 88%, with 8 of 20 variables achieving 85% or higher, 12 exceeding 80%, and 16 exceeding 75%. Kappa statistics ranged from 0.69 (Good) to 0.81 (Very Good) for variables measured with the Continuous Interval Playback mode. Three areas proved difficult for reliable agreement: Laughter (51.85% agreement, the lowest), Non-Child Audio, and Spontaneous Speech-Like Vocalizations. Laughter was hard to code because of its low frequency and difficulty distinguishing it from speech-like vocalizations, compounded by the differences in affective expression characteristic of ASD. When Spontaneous Speech-Like Vocalizations sub-categories were combined, agreement improved to 80%, suggesting the fine-grained distinctions between sounds made while looking at versus away from the screen were too subtle to code reliably. Timing tolerance analysis revealed that increasing the acceptable window from 1.0 to 5.0 seconds yielded at most a 0.5% improvement, indicating coders were accurate in placement and disagreements stemmed from interpretation rather than timing.
Relevance
This paper is valuable for accessibility researchers and practitioners working at the intersection of technology and autism. The A3 methodology provides a replicable framework for evaluating how children with ASD respond to technology-based interventions, which is critical for evidence-based assistive technology development. The open-source VCode and VData tools lower barriers for cross-disciplinary teams to conduct rigorous behavioral research. For practitioners, the findings highlight the importance of carefully defining behavioral variables when studying technology use by individuals with complex communication needs. The work also demonstrates that computerized audio feedback may be more effective than visual feedback for facilitating vocal development in children with ASD — a finding with direct implications for designing therapeutic assistive technologies.
Tags: autism · video annotation · research methodology · inter-rater reliability · computerized feedback · vocal development · low-functioning autism · HCI · assistive technology