MouseClicker: Exploring Tactile Feedback and Physical Agency for People with Hand Motor Impairments
Atieh Taheri, Carlos Gilberto Gomez-Monroy, Vicente Borja, Misha Sra · 2024 · ACM Transactions on Accessible Computing · doi:10.1145/3648685
Summary
This paper introduces MouseClicker, a mechatronic assistive device that enables people with severe hand motor impairments to control a physical computer mouse using facial expressions while receiving tactile feedback that simulates the sensation of clicking. The work challenges a persistent gap in assistive technology design: while AT has successfully addressed functional needs like cursor movement and clicking, it has largely neglected the sensory and experiential dimensions of using physical tools. The device was co-designed with Taheri, a graduate engineering student with spinal muscular atrophy (SMA) who can only voluntarily control her right thumb. Although Taheri can move a cursor using a touchpad, she cannot press mouse buttons independently, requiring caregiver assistance for tasks like right-clicking or keyboard-mouse combinations. Beyond functionality, Taheri expressed a desire to experience the tactile sensation of clicking that she had never felt, having been unable to use a standard mouse her entire life. MouseClicker repurposes a standard two-button mouse with a servo-controlled lever mechanism that physically presses the buttons in response to facial expressions detected via webcam. The OpenFace 2.0 toolkit extracts Facial Action Units from the video feed, mapping expressions like disgust (left single-click), smile (left double-click), or wide eyes (right single-click) to corresponding mouse actions. Crucially, two small coin vibration motors attached to the index and middle fingers provide haptic feedback synchronized with each click, recreating the tactile confirmation that accompanies pressing a physical button. The design deliberately decouples the input site (facial muscles) from the feedback site (fingers), addressing a fundamental question in AT design: can haptic feedback serve as a complementary channel when the sites of action and feedback are not co-located? The total cost of materials for the prototype is approximately $100, and the team plans to open-source the design to enable others to build or modify the device affordably.
Key findings
The research included a user study with 10 participants without motor impairments alongside Taheri to establish optimal vibrotactile feedback parameters. The study used a split-plot experimental design with 80 conditions varying motor position (8 locations across index and middle fingers), vibration intensity (5 levels), and click sound (present or absent). Participants demonstrated high consistency in perceiving vibration intensity, with perceived intensity closely matching actual intensity across all motor positions. However, localizing vibration position showed much greater individual variability. Unexpectedly, the nail bed emerged as the most frequently perceived vibration location, followed by the fingertip. This contradicted initial expectations that fingertips would be most accurate due to their high concentration of mechanoreceptors. Each participant showed a distinct zone of greater accuracy, with individual best-zone accuracy ranging from 63% to 100% compared to a group average of 51%. The click sound had minimal effect on vibration detection, with 98% accuracy for localizing motors regardless of sound presence. This suggests that vibrotactile feedback alone is sufficiently salient, simplifying potential device designs by reducing reliance on auditory cues. Taheris haptic perception patterns differed from the group, with stronger localization to the ventral (palm-facing) side of both fingers rather than the nails. This discrepancy highlights the importance of individual assessment when designing personalized haptic interfaces, particularly for users with neuromuscular conditions that may affect tactile perception. The 150ms vibration pulse duration used in testing aligned closely with the measured average duration of physical mouse clicks (128ms), supporting the perceptual authenticity of the simulated feedback.
Relevance
This research advances a concept the authors call "holistic inclusion," arguing that assistive technology should address not just functional capabilities but also the sensory and emotional dimensions of using tools. Taheris extended feedback after daily use illustrates the profound impact of this approach. She described feeling included because she could "use the same mouse for clicking as my other non-disabled peers use" while also feeling "on my fingers the same feeling that they experience." The decoupled haptic feedback approach, where input and feedback occur in different body locations, has implications beyond this specific device. Many hands-free input methods including eye gaze, brain-computer interfaces, and voice control face the same challenge: how to provide tactile confirmation when the user cannot touch the device being controlled. The studys finding that vibrotactile feedback can effectively represent actions even when delivered to a non-active site suggests a generalizable design pattern for future AT development. For practitioners, the participatory design methodology demonstrates the value of sustained collaboration with end users throughout iterative development. Taheris involvement went beyond consultation to active co-design, shaping decisions from initial brainstorming through final prototype refinement. Her preferences and requirements drove key design choices, including the shift from an exoskeleton concept to the decoupled approach. The low-cost, open-source nature of MouseClicker offers a model for accessible AT development. Rather than creating entirely new input devices, repurposing common peripherals with added haptic feedback leverages existing familiarity while reducing cost and enabling community modification.
Tags: motor impairment · haptic feedback · vibrotactile feedback · facial expression input · computer mouse · spinal muscular atrophy · participatory design · physical agency · hands-free input