Gesture-based Interaction for Individuals with Developmental Disabilities in India
Sumita Sharma, Saurabh Srivastava, Krishnaveni Achary, Blessin Varkey, Tomi Heimonen, Jaakko Hakulinen, Markku Turunen, Nitendra Rajput · 2016 · Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '16) · doi:10.1145/2982142.2982166
Summary
This full paper presents the design, development, and evaluation of Kirana, a Kinect-based gesture application that simulates purchasing groceries from a local Indian kirana (grocery) store to teach life skills to individuals with developmental disabilities at the Nai Disha school in New Delhi. The research began with a user-centered design study during the school's Diwali Mela festival, where two preliminary games (free-form painting and animal matching) were used with 18 students to identify appropriate gesture vocabularies, understand stakeholder expectations, and determine culturally sensitive interaction patterns. Key design findings included: punching gestures were fun but deemed socially inappropriate by teachers (concerned about reinforcing repetitive behaviors); drag-and-drop was too difficult; 1-second dwell time was too short (causing accidental selections); and the system worked well for wheelchair users via seated Kinect tracking. The Kirana application breaks the complex task of buying groceries into smaller achievable steps: selecting items from shelves by pointing (decision making), looking up M.R.P. prices (cultural knowledge), calculating costs against a budget (arithmetic), and checking balance (mathematical verification). The interface uses two-handed selection — right hand for items, left hand for money — to encourage increased bodily movement. Audio and visual feedback accompany every interaction, and animations are slow and sequential to support comprehension.
Key findings
Eighteen participants (ages 16-39, mean 26, IQ mean 46, 5 with Down syndrome, 8 with intellectual disability, 3 with autism, 2 with cerebral palsy and intellectual disability) who were previously unable to shop independently used Kirana weekly over three weeks. Mathematical test scores improved an average of 8.3% between pre- and post-evaluations, with lower-scoring participants benefiting more than those already scoring above 6/10. Buying time per item dropped dramatically: from 2 minutes 34 seconds in week 1 to 1 minute 13 seconds in week 3. Participants showed increasing preference for independent decision-making, shifting from teacher-provided shopping lists to self-selected items (16% chose their own items in session 1, rising to 73% by session 3). In the critical Phase IV real-world evaluation, 12 of 18 participants visited an actual kirana store near the school — the first time they had ever shopped independently. Participants successfully connected virtual tasks to real-world actions: making decisions from shopping lists, interacting with shopkeepers (three usually nonverbal participants initiated conversation), and understanding M.R.P. pricing. Shy participants observed others before approaching the shopkeeper, and one participant independently noted that everyone should take a bill for verification. Emotional engagement was high — participants became more expressive over sessions, though some expressed frustration when unable to complete tasks quickly.
Relevance
This research makes a compelling case for gesture-based technology as a tool for teaching transferable life skills to individuals with developmental disabilities in developing countries. The most significant finding is the successful transfer from virtual to real-world performance — participants who had never shopped independently were able to do so after three weeks of Kirana use. For accessibility practitioners, several design principles are noteworthy: cultural sensitivity in gesture selection (avoiding reinforcing socially unacceptable behaviors); multimodal feedback (visual and auditory) to accommodate varying sensory abilities; slow, sequential animations for cognitive accessibility; teacher customization of difficulty level; and clear start/end markers to reduce confusion. The study also highlights challenges specific to the developing world context: limited access to technology, cultural attitudes toward disability, economic barriers to specialized AT, and the importance of designing applications that can be shared by an entire school rather than purchased individually. The shift from list-following to independent decision-making over three sessions suggests that the application built not just skills but genuine self-efficacy.
Tags: developmental disability · gesture interaction · educational technology · cognitive accessibility · independent living · Global South accessibility · self-efficacy · Down syndrome · autism · participatory design