ChartA11y: Designing Accessible Touch Experiences of Visualizations with Blind Smartphone Users
Zhuohao Zhang, John R. Thompson, Aditi Shah, Manish Agrawal, Alper Sarikaya, Jacob O. Wobbrock, Edward Cutrell, Bongshin Lee · 2024 · Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2024) · doi:10.1145/3663548.3675611
Summary
This paper presents ChartA11y, an iOS mobile application that makes 2-D data visualizations accessible to blind and low-vision (BLV) smartphone users through multimodal touch interactions. Developed through an iterative participatory design process involving 13 co-design sessions over eight months with two blind collaborators, the system addresses a critical gap: while data visualizations are increasingly common, existing accessibility approaches rely primarily on keyboard navigation with screen readers or simple touch-to-audio mappings that fail to convey the rich spatial information in charts. ChartA11y introduces a dual-mode interaction paradigm. The Semantic Navigation Framework (SNF) mode organizes chart elements into a hierarchical navigational structure compatible with VoiceOver gestures, allowing users to explore axes, data points, and filters through familiar swiping and double-tapping interactions. The Direct Touch Mapping (DTM) mode provides direct access to the chart as if it were physically mapped onto the touchscreen, with sonification (pitch-based audio tones) and haptic vibration feedback as users drag their fingers across data points. The system supports line charts, bar charts, and notably scatter plots, which pose unique challenges due to their non-linear 2-D data distribution. For scatter plots, ChartA11y employs a dynamic scanning radius that treats the fingertip as a scanning window, triggering haptic feedback upon contact with data points, and a directional lock mechanism that helps users maintain consistent movement direction.
Key findings
The participatory design process revealed several previously undocumented challenges that blind users face with touch-based chart interaction. Users struggled to follow visual traces like trend lines or cluster boundaries, often drifting diagonally when attempting horizontal movements across scatter plots. Orientation within the abstract chart space was persistently difficult, as users could sense their physical position on the device but not their semantic location within the data. The two-mode solution proved complementary rather than redundant: SNF provided structured understanding of chart components and aggregate data patterns, while DTM enabled granular exploration of individual data points and correlations. Neither mode alone was sufficient for comprehensive chart understanding. The location-aware and adaptive narration system, which dynamically adjusts whether X or Y values are announced first based on whether the user is navigating sequentially or jumping to a new position, significantly reduced cognitive load. Design partners could independently extract insights from complex multi-series scatter plots, identifying correlations, cluster distributions, and outliers—tasks that were previously impossible without sighted assistance.
Relevance
ChartA11y demonstrates that accessible data visualization on smartphones requires moving beyond simple auditory feedback to embrace multimodal interactions combining speech, sonification, and haptic feedback. The dual-mode approach offers a practical model for other accessible visualization tools: structured navigation for overview and context, direct touch for detailed exploration. The participatory design methodology, with blind collaborators involved as integral partners from the outset rather than end-stage evaluators, exemplifies best practices for accessibility research. For practitioners, the finding that customization of input and output modalities is essential—not optional—challenges the assumption that a single accessible rendering can serve all users. The work also raises important questions about the role of AI in accessible visualizations, as design partners questioned why they should learn touch interactions when LLMs could interpret charts directly, prompting discussion about the importance of independent data exploration for equity and empowerment.
Tags: data visualization · blind and low vision · touchscreen accessibility · sonification · multimodal interaction · participatory design · smartphone accessibility
Standards referenced: VoiceOver · ARIA