Distinguishing Users By Pointing Performance in Laboratory and Real-World Tasks
Amy Hurst, Scott E. Hudson, Jennifer Mankoff, Shari Trewin · 2013 · ACM Transactions on Accessible Computing · doi:10.1145/2517039
Summary
This paper investigates how machine learning can automatically assess pointing difficulties from everyday computer use, addressing a critical barrier to accessible computing: the lack of frequent, low-cost assessment of pointing ability. The researchers collected and analyzed pointing data from both laboratory studies and a nine-month real-world deployment with 16 participants across three groups: able-bodied adults, older adults with arthritis, and individuals with motor impairments including cerebral palsy, spinal cord injury, and traumatic brain injury. They developed CRUMBS (Capture Real-world User Mouse BehaviorS), custom logging software that captured over 360,000 samples of mouse movements, clicks, keyboard events, and window interactions during natural computer use. The study compared laboratory Fitts's law-style pointing tasks with real-world data, revealing significant differences in performance variability and the limitations of single-session laboratory assessments for understanding actual pointing ability.
Key findings
Machine learning classifiers achieved high accuracy in distinguishing users with and without pointing difficulties. Laboratory-based models reached 92.7% accuracy (Kappa=.85) distinguishing motor-impaired from able-bodied users, and 97.6% accuracy distinguishing Parkinson's patients from young adults. Real-world models achieved 91.94% accuracy (Kappa=.80) on individual pointing actions, improving to 99% when aggregating predictions over 320 consecutive samples using a sliding window approach. The most predictive features were target size (smaller targets selected by able-bodied users), click duration (longer for older adults and motor-impaired users), and cursor movement efficiency. Critically, the study revealed high within-session and between-session variability for individuals with motor impairments—performance during a single laboratory session often differed dramatically from real-world use, demonstrating that frequent, continuous assessment is essential for adaptive systems.
Relevance
This research provides foundational methodology for building adaptive systems that automatically detect pointing difficulties and adjust interface settings accordingly. The practical scenario presented—where software detects when a user with intermittent tremors is having difficulty and adjusts mouse settings, then reverts when another family member uses the computer—illustrates real-world applications. For accessibility practitioners, the findings validate that automatic, unobtrusive assessment during natural computer use is feasible and more representative than formal assessments. The study also highlights the need for participatory design when developing such systems, noting important questions about user control and whether people want to be informed when their performance declines.
Tags: pointing performance · motor impairments · machine learning · adaptive interfaces · mouse interaction · older adults · real-world data collection · automatic assessment