Constructing Relational Diagrams in Audio: The Multiple Perspective Hierarchical Approach
Oussama Metatla, Nick Bryan-Kinns, Tony Stockman · 2008 · Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility (Assets '08) · doi:10.1145/1414471.1414491
Summary
This paper from Queen Mary, University of London presents a novel approach for constructing and manipulating relational diagrams — such as Entity-Relationship (ER) diagrams — through an audio-only interface. While previous research addressed non-visual access to existing diagrams, very little work had investigated how such representations could be actively created without visual means. The authors propose structuring diagram information using multiple perspective hierarchies based on three principles: preserve the scale types (categories) of the represented information rather than dimensional representations; preserve details associated with each item separately from others within the same scale type; and allow flexible, context-specific interactivity beyond basic hierarchical navigation. For a relational diagram, this produces a hierarchy with branches for Nodes (entities) and Links (relationships), each containing detailed information accessible from different perspectives. The system uses synthetic speech for content and navigational information, supplemented by ambient sounds that reflect the category type being browsed. Two interaction strategies were designed and compared: a Guided strategy (conversational agent assists the user through separate exploration and editing modes) and a Non-Guided strategy (user directly navigates and edits within the hierarchy in a combined mode).
Key findings
A study with 24 sighted participants (who could not see the screen) constructing ER diagrams from written system descriptions found that the non-guided strategy was significantly faster (average 19 minutes vs. 23 minutes, t=3.039, p<0.05). Both strategies supported similar levels of diagram comprehension: post-construction task scores averaged 76% (guided), 77% (non-guided), and 80% (control condition using pen and paper). Unexpectedly, participants scored significantly higher on comprehension when using the non-guided strategy versus the control condition (t=1.927, p<0.05), contradicting the hypothesis that the guided strategy would produce better understanding. This is because the guided strategy's intermediary agent introduced confusion about current system state — participants lost track of where they were in the hierarchy. Three types of mode confusion were identified: within-mode (losing track after multiple steps), cross-mode (forgetting whether in exploration or editing mode), and a "preventing mode confusion" issue where speech feedback was too similar between modes. User preferences strongly favored the non-guided strategy (16 vs. 6), with two categories of editing emerging: global editing (affecting hierarchy levels 1-2) where no strategy preference existed, and local editing (affecting levels 3-4) where guided was preferred for its ability to execute actions without positional navigation.
Relevance
This research addresses a fundamental gap in diagram accessibility: most accessible graphics research focuses on reading existing diagrams, but blind professionals — particularly in software engineering, where ER diagrams are standard tools — need to create and edit diagrams independently. The multiple perspective hierarchical approach provides a principled framework for translating inherently spatial, two-dimensional information into a navigable auditory structure. For accessibility practitioners, the study offers important interaction design lessons: combining exploration and editing modes reduces time and cognitive overhead, guided conversational interfaces can paradoxically increase confusion through their intermediary steps, and ambient sounds effectively convey category context during navigation. The approach is generalizable beyond ER diagrams to other relational representations like flowcharts, state diagrams, and mind maps.
Tags: auditory display · non-visual representation · diagram accessibility · blind users · accessible graphics · information accessibility · sonification · interaction design