A3: HCI Coding Guideline for Research Using Video Annotation to Assess Behavior of Nonverbal Subjects with Computer-Based Intervention
Joshua Hailpern, Karrie Karahalios, James Halle, Laura DeThorne, Mary-Kelsey Coletto · 2009 · ACM Transactions on Accessible Computing · doi:10.1145/1530064.1530066
Summary
This paper introduces A3 (Annotation for ASD Analysis), a comprehensive coding guideline containing 21 dependent variables for assessing the behavior of nonverbal individuals interacting with computer-based interventions. The authors developed A3 through extensive video analysis of five nonverbal children with autism spectrum disorder across six experimental sessions, totaling over 1,200 minutes of video data. The guideline addresses a critical gap in HCI and accessibility research: the lack of standardized metrics for evaluating how nonverbal subjects engage with technology-based interventions. The variables are organized into engagement measures (Smiling, No Face, Oriented at Screen, Auditory Focus, Time in Chair) and vocalization measures, which follow a hierarchical structure distinguishing between speech and nonspeech sounds, imitative versus spontaneous vocalizations, and immediate versus delayed responses. The methodology uses a four-pass annotation system with VCode software, employing different video playback modes optimized for each variable type. The first pass identifies non-child audio, the second codes engagement using interval playback, the third analyzes vocalizations in detail, and the fourth tracks time in chair using fast-skim playback.
Key findings
The A3 coding system achieved strong inter-rater reliability with 88% overall point-by-point agreement and Cohen's Kappa values ranging from 0.64 to 0.84 across variables. Coding time averaged 20 minutes per minute of video when annotating all 21 variables, though this can be significantly reduced by selecting relevant subsets of variables for specific research questions. A significant contribution is the automation analysis examining which variables could be enhanced by technology. The authors identify four automation levels: Replacement (computer fully replaces human coders), Review (computer marks events for human verification), Quick Index (computer highlights areas of interest to speed manual coding), and Manual (no automation possible). Eight variables were identified as candidates for Review-level automation using existing technologies like facial recognition, eye tracking, and audio analysis. Variables like Time in Chair could achieve full Replacement automation using pressure sensors. The paper demonstrates that A3 extends beyond autism research to any context involving nonverbal subjects and technology, including infants, individuals with verbal apraxia, and users of AAC devices.
Relevance
A3 provides accessibility researchers and practitioners with a rigorous, validated framework for evaluating computer-based interventions with nonverbal users—a population frequently underrepresented in HCI research due to methodological challenges. The standardized variables enable meaningful comparison across studies and interventions, which has been largely impossible without common metrics. For practitioners developing accessible technologies for nonverbal users, A3 offers concrete behavioral indicators of engagement and communication that go beyond simple task completion metrics. The vocalization hierarchy is particularly valuable for tracking communication development, as it captures subtle progress like the shift from nonspeech sounds to imitative speech attempts. The automation analysis provides a practical roadmap for researchers seeking to reduce annotation burden while maintaining reliability. However, the 20:1 coding time ratio highlights the significant resource investment required for thorough behavioral analysis—an important consideration for project planning.
Tags: autism · nonverbal communication · video annotation · research methodology · behavioral assessment · computer-based intervention · AAC