Touchpad Mapper: Exploring Non-Visual Touchpad Interactions for Screen-Reader Users
Ather Sharif, Venkatesh Potluri, Jazz R. X. Ang, Jacob O. Wobbrock, Jennifer Mankoff · 2024 · Proceedings of the 21st International Web for All Conference (W4A) · doi:10.1145/3677846.3677867
Summary
This technical note introduces Touchpad Mapper, a system that repurposes the laptop touchpad — a widely available but underutilized input device for screen-reader users — as a spatial interface for exploring 2D digital content non-visually. The authors argue that while touchpads have existed for decades and offer multi-touch support, gesture input, and vibrotactile feedback, they provide minimal utility for blind users whose interactions remain predominantly linear and keyboard-based. Touchpad Mapper maps digital content to the physical coordinates of the touchpad, enabling spatial exploration through finger touch. The system was built as a macOS application using Apple's AppKit and NSEvent API to capture touchpad coordinates, which are sent via WebSocket to a NodeJS backend server and broadcast to a browser client. The browser uses JavaScript to perform actions based on coordinates and relays information to the user through the Web Speech API. Two usage scenarios were explored: (1) image exploration, where an image is mapped to the touchpad dimensions and Google's Vision API identifies objects and their positions so that moving a finger over the touchpad reads out objects at corresponding locations (e.g., "Eiffel Tower" and its relative position); and (2) video control, where the touchpad's top horizontal edge maps to a video seek bar for position control, and the bottom edge enables rewind (left) and fast-forward (right).
Key findings
A pilot study with three blind screen-reader users showed uniformly positive results. All participants reported that Touchpad Mapper was useful, efficient, and faster than conventional keyboard interaction for both image exploration and video control. Participants highlighted several additional use cases beyond those demonstrated, including exploring mathematical content and navigating web page elements spatially. They reported positive experiences with the spatial mapping concept and considered the touchpad interaction more intuitive than sequential keyboard navigation for 2D content. Participants also suggested improvements, such as enabling object detection within video frames to explore video content spatially. The system represents the first demonstration of touchpad-enabled interaction techniques specifically designed for screen-reader users, distinguishing itself from existing touchpad-as-mouse features in screen readers (JAWS, NVDA, VoiceOver) which merely replicate keyboard-equivalent interactions.
Relevance
Touchpad Mapper addresses a fundamental mismatch in how blind users interact with inherently spatial content like images and videos: keyboards force linear, sequential access to content that is naturally two-dimensional. By mapping spatial content to a physical surface that users can explore with their fingers, the system provides a more natural interaction paradigm that leverages hardware already present on every laptop. For accessibility practitioners and developers, the key insight is that touchpads represent untapped potential for non-visual interaction — they are universally available, require no additional hardware, and can provide spatial context that keyboards cannot. The study is preliminary (3 participants, no formal task completion metrics), but the concept is compelling and extensible. The architecture — macOS app capturing coordinates, WebSocket relay, browser-based client with speech output — provides a replicable template. Future work could extend this to data visualizations, maps, diagrams, and other spatial content that remains largely inaccessible to screen-reader users despite advances in alternative text.
Tags: screen readers · blindness · touchpad · interaction design · input devices · image exploration · video accessibility · non-visual interaction