Tapsonic: One Dimensional Finger Mounted Multimodal Line Chart Reader
Zeyuan Zhang · 2020 · Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2020) · doi:10.1145/3373625.3417075
Summary
This short paper presents Tapsonic, a finger-mounted wearable device that enables people with visual impairments to read line charts through a combination of haptic feedback and voiceover. The key innovation is a dimensionality reduction approach: rather than trying to reconstruct the 2D shape of a chart through touch (which is cognitively demanding and prone to the "haptic oblique effect" where oblique angles are hard to discriminate tactilely), Tapsonic splits the two-dimensional chart information into one-dimensional finger movement (horizontal swiping across an iPad touchscreen) and bidirectional haptic feedback (vertical value changes conveyed by top and bottom linear actuators). The device consists of an Arduino UNO and two linear actuators (M1027) mounted on the top and bottom of a 3D-printed rubber case that clips onto the second phalanx of the index finger. A web-based line chart interface built with P5.js on an iPad 9.7" serves as the interactive surface. When users swipe horizontally across the chart, the system detects selected data points and provides voiceover of their values. Value changes between consecutive points trigger directional haptic feedback — the top actuator fires for positive changes (going up) and the bottom actuator for negative changes (going down). Users can swipe vertically to switch between data series, and use a pinch gesture (two fingers) to compare values of two series at the same horizontal position. A time threshold prevents voiceover overlap — if the finger stays less than one second on a point, only a beep sounds; full voiceover plays when the finger pauses.
Key findings
Four participants with congenital visual impairments (1 female; ages 22-27, mean 25.2; all educated with prior tactile chart experience) completed five tasks across two sessions: finding transition points and their values, finding maximum and minimum values, finding intersect points between two series, and describing trends. Tasks were completed with up to 85% accuracy overall. Simple tasks (finding specific transition points and their values) were performed with only one mistake identified across all participants. The more complex task of identifying intersect points between two series had lower accuracy (50%), which participants attributed to confusion when switching between pinch and swipe gestures. NASA TLX workload assessment showed an overall index of 41% (moderate), with effort rated highest (56%) and physical demand lowest (20%) — confirming that the dimensionality reduction successfully reduced physical effort. The System Usability Scale score was 68.75 (grade C, passing the usability threshold of 68). Participants reported that haptic direction changes served as efficient markers for finding transition points: "I did not read data point by point. I kept swiping unless I felt the haptic direction changed. When that happened, I stopped and listened to the voiceover to know its values." Challenges included learnability of gesture switching and location offset when transitioning between pinch and swipe gestures.
Relevance
Tapsonic offers a pragmatic alternative to existing approaches for making charts accessible to blind users. Rather than trying to recreate the visual experience of a chart through elaborate tactile displays or sonification, it reduces the problem to what users actually need: knowing data values at specific points, understanding trends, finding extrema, and comparing series. The finger-mounted form factor is lightweight and works with standard tablet touchscreens, avoiding the need for specialized hardware. For accessibility practitioners working with data visualization, the dimensionality reduction principle — splitting 2D information into horizontal movement plus vertical haptics — is a transferable design pattern applicable to other chart types. The finding that users naturally adopted a "scan-then-inspect" strategy (swiping until feeling a haptic change, then pausing for details) suggests the system aligns well with how blind users want to explore data. Limitations include the very small sample (N=4), testing with only two simple line charts, unknown scalability to complex multi-series charts or other chart types (bar, scatter, stock), and the short paper format (4 pages).
Tags: visual accessibility · data visualization · data accessibility · haptic technology · blindness and low vision · assistive technology · wearable technology · multimodal interaction