It's All About the Message: Visual Experience is a Precursor to Accurate Auditory Interaction
Simon Harper, Sukru Eraslan, Yeliz Yesilada · 2019 · Proceedings of the 16th International Web for All Conference (W4A) · doi:10.1145/3315002.3317554
Summary
This paper argues that understanding how sighted users visually experience web pages is a necessary precursor to designing effective auditory (screen reader) experiences for visually disabled users. The authors challenge the assumption that the experiences of sighted and blind web users are fundamentally disconnected, proposing instead that they share a common cognitive basis at the component level. Web designers create visual narratives — deliberate sequences of visual components intended to guide users through a message — but this narrative is lost when screen readers linearize pages based on source code order. The paper uses eye-tracking data from 40 participants (20 male, 20 female) viewing six popular websites (Apple, Babylon, AVG, Yahoo, BBC) with varying visual complexity during browsing and searching tasks. The Scanpath Trend Analysis (STA) algorithm was applied to identify "trending paths" — the most common component-visiting sequences across users — representing the visual order in which sighted users actually experience pages. The VIPS (Vision-based Page Segmentation) algorithm identified visual components, and the aDesigner disability simulator tool determined linear order and access times for screen reader users. Two research questions were investigated: whether the linear (source code) order correlates with the visual order, and whether re-ordering components based on visual order improves access times for screen reader users.
Key findings
The correlation analysis showed no significant correlation between linear order and visual order for the trending components across most pages for both browsing and searching tasks — confirming that source code order does not reflect the sequence in which sighted users actually experience web content. Two transcoded versions were created for each page: Version 1 re-ordered trending components in visual order, and Version 2 additionally added descriptive headings for each component. Version 1 showed significant improvement for searching tasks (Dependent T-Test, t = 3.089, p = 0.003, medium effect) but not for browsing. Version 2, with headings, showed significant improvements for both browsing (Wilcoxon Z = -2.601, p = 0.005, medium effect) and searching (Wilcoxon Z = -2.436, p = 0.008, medium effect). The paper also discusses key challenges for experiential transcoding: understanding participant variation (if visual paths vary greatly, a single trending path may not exist); understanding user purpose (browsing vs searching produces different visual orders); understanding design quality (if visual design is poor, faithfully mapping it to audio produces an equally poor auditory experience); understanding dynamics (static visual narratives are mappable, but dynamically updating content creates challenges for equivalent auditory mapping); and understanding implementation (which visual order — for which task — should be encoded in the source code).
Relevance
This research fundamentally reframes the relationship between visual and auditory web experiences. Rather than treating screen reader accessibility as a separate domain from visual design, it argues that the visual narrative intended by web designers should inform the auditory experience. This has profound implications: it means that improving screen reader experiences requires understanding visual design intent, and that formative studies with sighted users using eye tracking should precede the design of auditory interactions. For accessibility practitioners, the key practical takeaway is that source code order — which determines screen reader reading order — often does not match the intended visual narrative, meaning screen reader users receive content in a sequence the designer never intended. The solution of "experiential transcoding" — re-ordering content based on actual visual usage patterns — offers a data-driven approach to improving screen reader experiences. The finding that adding headings to re-ordered content was necessary for browsing tasks (but not searching) suggests that screen reader users need structural landmarks to orient themselves when exploration is open-ended, while targeted searching benefits primarily from correct ordering. The paper also makes a pragmatic methodological argument: starting accessibility research with sighted eye-tracking studies rather than directly with visually disabled participants could reduce the research burden on visually disabled communities while still producing actionable insights.
Tags: eye tracking · screen reader · page linearization · experiential transcoding · visual impairment · web accessibility · scanpath analysis · visual narrative · web design · cognitive accessibility