Multimodal Presentation of Two-Dimensional Charts: An Investigation Using Open Office XML and Microsoft Excel
Iyad Abu Doush, Enrico Pontelli, Tran Cao Son, Dominic Simon, Ou Ma · 2010 · ACM Transactions on Accessible Computing · doi:10.1145/1857920.1857925
Summary
This paper presents a comprehensive system for making Microsoft Excel charts accessible to people who are blind or visually impaired through multimodal haptic and audio feedback. The system extracts chart data from the OOXML format introduced in Excel 2007, then presents it using the Novint Falcon haptic device—a consumer-grade force feedback controller costing approximately $150. The researchers developed presentation schemes for four common chart types: line, bar, pie, and scatter charts. The system offers three distinct presentation modalities. In exploratory mode, users freely navigate the chart space while receiving haptic and audio cues when approaching data points. Guided mode automatically moves the user through data points in sequence, providing a systematic tour of the chart. Summary mode delivers statistical information such as minimum, maximum, average values, and trend descriptions. Each chart type has specialized force feedback algorithms—gravity wells pull the cursor toward data points in line charts, stiffness forces lock users onto bars, and angular calculations constrain pie chart navigation to the chart's circular boundary. The design followed a user-centered process including focus groups with computer science students, analysis of 1,143 real-world Excel charts collected from 72 websites, and iterative refinement through two pilot studies with sighted blindfolded participants before conducting the main evaluation with visually impaired users.
Key findings
The evaluation with seven students (ages 15-28) at the New Mexico School for the Blind and Visually Impaired produced several notable results. For line charts, guided presentation mode achieved 84% accuracy in reproducing chart features compared to only 33% with exploratory mode. The time difference was equally striking: guided mode averaged 0.4 minutes while exploratory mode took 3.85 minutes. Bar chart tasks showed 95% accuracy with guided presentation versus 40% with exploratory mode. A particularly striking finding emerged when comparing visually impaired participants to sighted participants from earlier pilot studies. The visually impaired users completed all chart types faster—averaging 80 seconds quicker on scatter charts—while also achieving higher accuracy. The researchers attributed this to visually impaired participants' greater comfort with tactile perception and their ability to interpret haptic feedback without the confusion sighted users experienced when relating force feedback to spatial positions. Participants rated speech feedback highly (5 out of 5) and reported general satisfaction with haptic forces (3.8 out of 5). The clustering algorithm for scatter charts proved effective, with 100% accuracy in identifying the largest cluster and completion times averaging just 15-32 seconds.
Relevance
This research demonstrates that consumer-grade haptic hardware can provide meaningful chart access at a fraction of the cost of specialized assistive technology. The finding that visually impaired users outperformed sighted participants challenges assumptions about accessible interface design and suggests that studying with actual target users, rather than blindfolded proxies, yields different and more valuable insights. The design guidelines distilled from this work remain applicable: presentation strategies should match chart type, summary information reduces navigation burden, timing and scale affect comprehension, and audio cues are effective boundary indicators. While the specific hardware (Novint Falcon) is no longer manufactured, the multimodal design principles transfer to modern haptic controllers and emerging technologies. For practitioners, this work establishes that chart accessibility requires more than text descriptions—interactive exploration and multiple presentation modalities let users build accurate mental models of data relationships. Organizations creating accessible data visualizations should consider whether their solutions support both overview and detail-level exploration.
Tags: haptic feedback · data visualization · screen readers · multimodal interaction · force feedback · charts and graphs · blindness