INC-Hg: An Intelligent Collaborative Haptic-Gripper Virtual Reality System for Children with Autism to Develop Social Skills
Yao Zhao, Nicole Shay, Jonathan Cofino, Debra Sterling, Erin Paquette, Adriana Debarros, Maria Gillen, Michelle Won, Zhigang Zhu, Cecilia Feeley, Amy Hurst · 2022 · ACM Transactions on Accessible Computing · doi:10.1145/3487606
Summary
This paper presents INC-Hg (Intelligent Collaborative Haptic-Gripper), a VR system designed to help children with autism spectrum disorder (ASD) practice collaborative and social communication skills. The system addresses a significant challenge: traditional collaborative VR environments require a human partner, which creates logistical barriers for therapy and practice. INC-Hg replaces the human partner with an intelligent conversational agent that can engage in natural dialogue and collaborative tasks. The system centers on a "Prize Claw" game where users must verbally communicate and physically collaborate through haptic devices to successfully retrieve virtual prizes. The physical component uses 3D-printed gripper controllers equipped with force-sensing resistors that provide real-time haptic feedback, allowing users to feel when their partner is squeezing. This tangible feedback creates accountability and encourages genuine collaboration rather than passive participation. The intelligent agent uses a speech classifier built on Support Vector Machine with Radial Basis Function kernel (SVM-RBF), trained on dialogue act categories including social conventions, requests, descriptions, and task-specific utterances. The researchers developed their own annotated dataset from children's interactions, recognizing that existing dialogue act taxonomies don't capture how children with ASD communicate in collaborative contexts.
Key findings
The intelligent agent achieved strong performance: 97.56% accuracy for initiating conversations and 86.52% for responding to user utterances. The speech classifier reached 70.34% overall accuracy across dialogue act categories, with particularly strong performance on social conventions (95.31%) and request acts (88.89%). In comparative testing, the human-to-agent collaboration mode achieved a 61% collaborative operation ratio, significantly outperforming the 40% ratio in human-to-human mode. This suggests the agent's consistent, predictable behavior may actually facilitate better collaboration for children with ASD who may struggle with the unpredictability of human partners. A usability study with 10 children with ASD (ages 8-12) found high acceptance of the system. Participants rated the game fun (mean 4.2/5), easy to use (3.7/5), and expressed comfort speaking with the virtual partner (3.5/5). The haptic feedback was valued, with participants reporting they liked feeling their partner's grip (3.3/5). Notably, several children expressed preference for the virtual partner over playing with another person.
Relevance
This research demonstrates how intelligent agents can make collaborative VR therapy more accessible and scalable for children with ASD. Traditional interventions require scheduling coordination between children, trained facilitators, and appropriate spaces—barriers that limit access. An AI-powered partner available on demand removes these constraints. The finding that children achieved better collaboration with the agent than with human peers challenges assumptions about the superiority of human interaction. For children with ASD who may find human unpredictability stressful, a consistent virtual partner may provide a safer learning environment for practicing social skills. The haptic gripper design offers a model for creating tangible feedback in VR that promotes genuine collaboration. The open-source nature of the hardware (3D-printed components, Arduino-based) makes replication feasible for other researchers and practitioners. However, the small sample size and short-term evaluation leave questions about skill transfer to real-world social situations.
Tags: autism spectrum disorder · virtual reality · collaborative learning · haptic feedback · intelligent agents · social skills · speech recognition · children