Designing Spellcasters from Clinician Perspectives: A Customizable Gesture-Based Immersive Virtual Reality Game for Stroke Rehabilitation
Jared Duval, Rutul Thakkar, Delong Du, Kassandra Chin, Sherry Luo, Aviv Elor, Magy Seif El-Nasr, Michael John · 2022 · ACM Transactions on Accessible Computing · doi:10.1145/3530820
Summary
This research presents Spellcasters, an immersive virtual reality game adapted from entertainment for stroke rehabilitation. More than 800,000 people suffer strokes annually in the US, with two-thirds requiring rehabilitation. Stroke survivors often experience hemiparesis (weakness on one side of the body) and tend to avoid using impaired limbs—a behavior called "learned non-use" that worsens outcomes. The gesture-based spell-casting mechanics of the original entertainment game align naturally with upper limb rehabilitation exercises, making it a promising candidate for therapeutic adaptation. The researchers collaborated with 14 medical professionals (10 physical therapists, 4 occupational therapists) using a Research through Design approach. Due to COVID-19 safety concerns, stroke survivors were not directly included in the design process—the institutional ethics board required clinical validation before patient involvement. Sessions were conducted remotely via Zoom, with clinicians either using their own VR headsets or watching screen-shared gameplay. The primary contribution is a novel custom gesture creation system that allows clinicians to design personalized exercises as "magical spells" by placing depth-sensitive collision spheres in 3D space. Clinicians can adjust the scale, shape, direction, and depth of gestures to target specific therapeutic motions including shoulder abduction/adduction, elbow flexion/extension, wrist rotation, and reaching movements. The system includes 18 predefined gestures organized into therapeutic categories like "Extending Arm," "Rotation," and "Crossing Mid-line." A companion web app prototype was co-designed for remote monitoring, patient management, and data visualization.
Key findings
All 14 medical professionals found the custom gesture creation system useful and expressed interest in adopting Spellcasters in their practice. Clinicians particularly valued the ability to customize exercises: "I feel like you got everything covered—crossing the midline, shoulder rotations, extensions, etc." (P11). The game format was seen as appropriate for engaging younger stroke populations, which are becoming more common. The companion app emerged as essential for clinical adoption. One therapist noted, "We only have 45 minutes with the patient, so we cannot really spend 15 minutes creating the exercises—have more presets, and a save feature so we can reuse exercises." All clinicians who tested the app expressed interest in adoption. Bar graphs were strongly preferred over other visualization types (e.g., spider charts), and clinicians wanted "fast, easy-to-digest facts" rather than complex data displays. For telehealth safety, most clinicians agreed Spellcasters would be safe when supervised, with several indicating unsupervised home use could also be safe because the game can be played entirely while seated. This addresses motion sickness concerns and fall risks. However, some clinicians raised concerns about stroke survivors' ability to hold controllers or use buttons, leading to plans to implement hand tracking. The research found value in repurposing entertainment games for therapy—existing source code, design documents, and proven fun mechanics reduced development time compared to building from scratch, even though the rehabilitation version required significant modifications including a new gesture system.
Relevance
This work demonstrates a practical approach to developing accessible rehabilitation technology: start with proven entertainment mechanics and adapt them with clinical input rather than building purely clinical (and potentially boring) therapy games. The serious games literature shows that video games improve 69% of psychological therapy outcomes and 59% of physical therapy outcomes, but overly clinical designs defeat the motivational purpose of games. For accessibility practitioners, the custom gesture creation system offers a model for how assistive technology can be made adaptable to individual needs. Stroke survivors have heterogeneous abilities—some may have limited shoulder movement but good wrist control, or vice versa. The ability for clinicians to customize exercises without programming knowledge is essential for real-world adoption. The telehealth implications are significant beyond COVID-19. VR-based rehabilitation could increase access for patients in rural areas or those with transportation barriers. The companion app design process offers insights into what clinical data visualizations actually work for practitioners—simple bar graphs and quick facts rather than complex analytics. The finding that consumer VR hardware like Oculus Quest 2 ($299) is sufficient for therapeutic use suggests the technology is becoming accessible enough for home-based rehabilitation programs.
Tags: stroke rehabilitation · virtual reality · serious games · telehealth · gesture recognition · physical therapy · occupational therapy · games for health · participatory design