Detecting the Hand-Mouthing Behavior of Children with Intellectual Disability Using Kinect Imaging Technology
Tzu-Wei Wei · 2012 · Proceedings of the 14th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2012) · doi:10.1145/2384916.2385001
Summary
This short paper presents a system that uses Microsoft Kinect depth-sensing technology to automatically detect hand-mouthing behavior in children with severe intellectual disabilities. Hand-mouthing — the repetitive placement of fingers into the mouth — affects approximately 17% of individuals with intellectual disability, with higher prevalence and severity among those with more profound disabilities. The behavior can cause skin lesions, tissue damage, infections, unpleasant odor, and social isolation. Traditionally, special education teachers in Taiwan (who each serve an average of 3.7 students) must manually observe and record instances of hand-mouthing to evaluate the effectiveness of behavioral intervention strategies. This manual process is time-consuming, limits how many students a teacher can work with simultaneously, and prolongs the overall treatment period. The proposed system uses the Kinect sensor to track the spatial relationship between a child's hands and mouth, automatically detecting when hand-mouthing occurs. This enables teachers to rapidly verify the effectiveness of their instructional intervention strategies without requiring constant one-on-one observation, reducing teacher workload and shortening the duration of behavioral correction training.
Key findings
The system leverages the Kinect's skeleton tracking and depth imaging capabilities to identify when a child's hand position overlaps with the mouth region, enabling automated detection and recording of hand-mouthing episodes. The key innovation is replacing labor-intensive manual observation with automated computer vision, allowing special education teachers to test multiple intervention strategies more quickly. By automatically logging behavior frequency and duration, the system provides objective data that helps teachers evaluate which intervention approaches are most effective for individual students. The paper positions the technology as a tool to support — not replace — teachers, allowing them to focus on instruction and peer interaction rather than constant behavioral monitoring. The approach demonstrates how consumer-grade depth-sensing hardware can be repurposed as assistive technology in educational settings for children with significant disabilities.
Relevance
This paper illustrates an early application of depth-sensing technology to support individuals with intellectual disabilities in educational environments. While the Kinect platform has since been discontinued, the underlying concept of using computer vision for automated behavior monitoring remains highly relevant as newer depth-sensing and AI-based pose estimation technologies have advanced significantly. The work highlights an important but often overlooked area of accessibility: supporting people with severe intellectual disabilities and their caregivers through technology. For accessibility practitioners, it demonstrates that assistive technology extends well beyond screen readers and keyboard navigation into behavioral support, special education, and caregiver tools. The paper also raises important considerations about surveillance, consent, and the balance between monitoring and autonomy for people with disabilities.
Tags: intellectual disability · stereotypic behavior · self-injurious behavior · Kinect · behavior detection · special education · assistive technology · depth sensing