A Tool to Promote Prolonged Engagement in Art Therapy: Design and Development from Arts Therapist Requirements
Jesse Hoey, Krists Zutis, Valerie Leuty, Alex Mihailidis · 2010 · Proceedings of the 12th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2010) · doi:10.1145/1878803.1878841
Summary
This paper describes the design and evaluation of a customisable touch-screen tool that assists creative arts therapists working with older adults with dementia. The system uses a partially observable Markov decision process (POMDP) controller to autonomously monitor a client's engagement during art therapy sessions and adapt its prompts and interactivity level accordingly. The tool uses computer vision — specifically Haar-like face and eye detection via OpenCV — to determine whether the user is looking at the screen, and classifies on-screen touch behaviour into activity levels (interactive, active, intermittent, inactive). These observations feed into a user model that tracks four variables: behaviour, gesture, engagement (yes/confused/no), and responsiveness (yes/no). The POMDP then selects actions ranging from doing nothing to high-interactivity prompts such as animated buttons or audio playback. Critically, the tool was designed with therapists as the primary user group rather than the people with dementia directly. A therapist interface allows clinicians to design personalised creative activities by dragging and dropping widgets (drawing tools, colour palettes, image selectors), configuring action mappings that link generic POMDP actions to specific interface behaviours, and adjusting tuning knobs for system passivity, activity sensitivity, and eye-contact sensitivity. The tool was developed through a user-centred design process involving surveys, focus groups, and one-on-one interviews with seven arts therapists from Sunnybrook Hospital in Toronto.
Key findings
Interviews with six arts therapists and one music therapist confirmed strong acceptance of the customisable approach. All seven participants agreed the system would be easy to use with little training. Therapists valued the ability to customise engagement settings, with one stating that "being able to control the intelligent assistive system by configuring engagement settings during the design phase would further improve the system." Three of seven participants raised concerns about how the system would measure engagement, noting that engagement is highly individualised — for example, someone with Parkinson's may "just be frozen" while still mentally engaged. The therapists identified three tuning knobs as being of most interest: system passivity (how often the system prompts), activity level (how the system interprets touch behaviour), and eye-contact level (how looking at versus away from the screen affects the model). Simulation results demonstrated that the adapted POMDP model correctly reduced prompting when a decrease in activity was due to user engagement rather than disengagement, while the non-adapted model incorrectly increased high-interactivity prompts. Therapists also suggested additional features including multi-touch support, biometric monitoring, and hardware art tools.
Relevance
This paper addresses a significant gap in assistive technology: tools designed specifically for therapists rather than end users with disabilities. The approach of separating the generic adaptive controller (POMDP) from the specific therapeutic application is a powerful design pattern that could be applied to many accessibility contexts where non-technical professionals need to customise adaptive systems for individual clients. The therapists' concern about engagement measurement — that observable behaviour does not always reflect internal cognitive state — is a crucial insight for anyone building automated engagement or attention detection systems for people with neurological conditions. For accessibility practitioners, the paper demonstrates both the promise and the limitations of using computer vision and AI to infer user states in populations whose behavioural signals may differ significantly from typical users.
Tags: dementia · art therapy · computer vision · adaptive systems · user modeling · engagement detection · touchscreen interface · older adults · creative arts therapy