Kinempt: A Kinect-Based Prompting System to Transition Autonomously Through Vocational Tasks for Individuals with Cognitive Impairments
Yu-Chi Tsai · 2012 · Proceedings of the 14th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2012) · doi:10.1145/2384916.2385003
Summary
This short paper presents Kinempt (a portmanteau of "kinetic" and "prompt"), a Kinect-based task prompting system designed to help individuals with cognitive impairments perform vocational tasks independently. The research is motivated by the fact that people with cognitive impairments are often viewed as unemployable and systematically excluded from labor markets, despite evidence that with sufficient support through community-based rehabilitation (CBR) and supported employment, many can successfully participate in the workforce. The system uses the Kinect sensor's RGB camera and depth sensor to track hand and wrist joint positions, monitoring whether users move in and out of designated positions required by each task step. Job coaches input task descriptions as step-by-step sequences, and a computer screen displays just-in-time instructions using both text and pictures — for example, "Get a cup of pineapples" or "Get half a cup of shrimps." The system compares the user's gesture sequence against the expected vocational task sequence, identifying both correct and incorrect steps and raising alerts when steps are not followed properly. This is non-intrusive — users don't carry handheld devices or touch controllers.
Key findings
Kinempt was tested in a community-based rehabilitation program with two participants undergoing pre-service food preparation training, guided by three job coaches. Results indicated that acquisition of job skills was facilitated by using Kinempt in conjunction with operant conditioning strategies (behavioral reinforcement techniques). The system's key advantage over previous task prompting approaches — which relied on PDAs, "Wizard of Oz" methods, user self-initiation, or constant time delay — is that it uses gesture recognition to automatically detect task completion and advance prompts without requiring the user to interact with a device. This "hands busy, eyes busy" design is particularly important for vocational tasks like food preparation where users cannot stop to operate a handheld device. The design was informed by usability studies of interfaces for people with cognitive impairments and requirements gathered from interviews with job coaches at rehabilitation institutes. The system frees job coaches from having to constantly monitor trainees, potentially allowing them to support more individuals simultaneously.
Relevance
This paper demonstrates how consumer depth-sensing technology can be repurposed as assistive technology to promote employment independence for people with cognitive disabilities — an area of accessibility that extends well beyond the digital interfaces typically discussed in web accessibility. For accessibility practitioners, the work highlights several important principles: the value of non-intrusive, hands-free assistive technology that does not add to the user's cognitive or physical burden; the importance of just-in-time prompting that provides information exactly when needed; and the role technology can play in supporting (not replacing) human caregivers. The focus on vocational skills and employment is particularly significant, as employment is a critical pathway to social inclusion, financial independence, and quality of life for people with cognitive impairments. While the Kinect platform is discontinued, modern pose estimation and computer vision technologies make this approach more feasible than ever.
Tags: cognitive impairment · Kinect · task prompting · vocational rehabilitation · supported employment · assistive technology · gesture recognition · depth sensing · food preparation · job training