FootUI: Designing and Detecting Foot Gestures to Assist People with Upper Body Motor Impairments to Use Smartphones on the Bed
Xiaozhu Hu, Jiting Wang, Weiwei Gao, Yongquan Hu · 2022 · Proceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3517428.3563285
Summary
This paper presents FootUI, a vision-based interaction technique that enables people with upper body motor impairments but sound lower limbs to use smartphones through foot gestures while reclining on a bed. The system uses the phone's front-facing camera, mounted on a phone holder at the foot of the bed, to track foot movements and translate them into smartphone operations. The research addresses a specific gap: while people with conditions such as bilateral arm amputation, cerebral palsy, or severe upper body paralysis often use their feet to interact with touchscreens, directly touching the screen with toes is tiring, inefficient, and prone to mistouches. The researchers designed an initial vocabulary of 17 foot gestures across three categories — dynamic continuous gestures (like heel rotation and ankle flexion for cursor control), static continuous gestures (holding postures like a long-press), and discrete gestures (like tap, double tap, kick up, and foot rotations for triggering commands). They evaluated these through a user study with 10 able-bodied participants and 2 upper body motor-impaired participants (both-arm amputees), rating gestures on ease-of-use, fatigue, speed, and preference. The gesture tracking uses color-based foot segmentation in HSV space, contour detection with ellipse fitting to extract feature points (fingertips, heel), and Kalman filtering for smooth tracking. A random forest classifier trained on normalized stroke features achieves gesture classification.
Key findings
A critical finding was that toes-based gestures (like "Big Toe Up" and "Toes Plantarflexion") performed poorly on both user preference and technical detection — they were uncomfortable to perform and the movements were too small for the camera to reliably detect. These were removed from the final gesture set. The final set of 11 gestures (3 dynamic continuous, 8 discrete) achieved 90.53% classification accuracy via leave-one-out cross validation. Simple gestures like Rotate Left, Rotate Right, Kick Up, and Tap were detected with high accuracy, while more complex gestures were less reliable. In the usability evaluation with 14 able-bodied and 2 motor-impaired participants, FootUI achieved 96.75% pointing accuracy, 84.65% classification accuracy, and 92.97% memory accuracy. Participants completed 95.63% of tasks (vs 100% with finger touch). Task completion took roughly twice as long as finger touch (198s vs 96s), but FootUI showed no significant difference from finger touch on learnability (p=0.08). Both-arm amputee participants confirmed FootUI's value, with one noting it was "really suitable for e-reading and video watching" and that "reading novels and watching TV is quite convenient." Participants found the pointing gestures somewhat fatiguing over time but shortcut gestures caused no fatigue. Both motor-impaired participants emphasized that FootUI would be particularly valuable for people who recently lost arm function or have cerebral palsy.
Relevance
FootUI addresses a genuinely underserved population — people who have functional lower limbs but cannot use their hands for smartphone interaction. While much accessibility research focuses on head-based, voice-based, or eye-tracking input, these alternatives are unusable for some people (e.g., those with dysphonia cannot use voice control). The finding that toes-based gestures should be avoided is a practical design insight for anyone working on foot-based interfaces. The system's reliance solely on the phone's built-in camera, without requiring additional hardware or sensors, makes it potentially deployable on existing devices. Limitations include the small number of motor-impaired participants (only 2), the requirement for colored socks for foot segmentation, and the roughly 2x slower task completion compared to finger touch. The work opens important questions about foot-based interaction design for mobile devices and demonstrates that camera-based gesture recognition can provide a viable alternative input method for people who cannot use their hands.
Tags: motor accessibility · foot-based interaction · gesture recognition · smartphone accessibility · computer vision · input methods · upper body motor impairment