Development and Evaluation of a Tool for Assisting Content Creators in Making PDF Files More Accessible
Debashish Pradhan, Tripti Rajput, Aravind Jembu Rajkumar, Jonathan Lazar, Rajiv Jain, Vlad I. Morariu, Varun Manjunatha · 2022 · ACM Transactions on Accessible Computing · doi:10.1145/3507661
Summary
This paper presents the development and evaluation of "Ally," a web-based tool designed to help content creators remediate PDF documents for accessibility. While trillions of PDF documents exist online, only a small fraction include the metadata necessary for assistive technologies to interpret them correctly. The PDF/UA-1 standard and the Matterhorn Protocol define 136 accessibility test conditions, but existing tools like Adobe Acrobat Pro are notoriously difficult to use, creating a significant barrier to accessible document creation. Ally addresses four critical pain points in PDF accessibility remediation: (1) content tagging—identifying and labeling regions as paragraphs, headings, figures, lists, tables, or artifacts; (2) heading levels—establishing proper nested hierarchy (H1, H2, H3); (3) reading order—defining the logical sequence in which a screen reader should present content; and (4) table structure—marking header cells (TH) versus data cells (TD) and ensuring proper relationships. The tool breaks these complex tasks into guided steps with visual feedback, color-coded tags, contextual editing, and a progress meter showing completion status. The interface design draws on Shneiderman's 8 Golden Rules, emphasizing consistency across tasks, multiple approaches for the same goal (drawing vs. dragging for reading order), easy reversal of actions with undo/redo, informative feedback through notifications, and reduced cognitive load through visual externalization of tag information.
Key findings
A comparative user study with 20 participants (11 novice, 9 intermediate) showed Ally dramatically outperformed Adobe Acrobat Pro across all measures. Combined across all tasks, participants were 47.1% faster and made 62.5% fewer errors when using Ally. The System Usability Scale (SUS) score was 80 for Ally (considered "usable") versus 30 for Acrobat Pro. For specific tasks: validating regions was 3x faster with Ally (4:39 vs 13:34 mean time) with 58% fewer errors; validating headings was 2.5x faster (1:27 vs 3:43) with 73% fewer errors; validating reading order showed 1.5x speed improvement with 41.8% fewer errors; and table validation was 1.5x faster with 68.9% fewer errors. Notably, while 8 of 20 participants gave up on at least one task in Acrobat, zero participants abandoned any task in Ally. Qualitative feedback highlighted that Ally's color-coded visual tags eliminated the cognitive burden of navigating Acrobat's complex tag tree structure. Users appreciated contextual editing directly on the document image rather than abstract tree manipulation. The hierarchical display of nested headings and the visual representation of reading order using curved arrows matched users' mental models of document structure.
Relevance
This research directly addresses a critical gap in accessibility tooling. PDF accessibility is legally required in many contexts (Section 508, ADA, European accessibility directives), yet even accessibility-focused conferences like ACM ASSETS have historically published papers where only 26.8% contained proper accessibility tagging. The perception that "making a PDF accessible is hard" persists because existing tools prioritize format fidelity over accessibility workflows. For practitioners, Ally demonstrates that thoughtful interface design can make accessibility remediation accessible to non-experts. The 47% time savings and 62.5% error reduction translate directly to organizational capacity for accessible document production. The study validates specific design principles: breaking complex workflows into discrete steps, providing visual feedback that externalizes internal document structure, supporting multiple interaction modes for different user preferences, and maintaining consistent patterns across related tasks. The paper acknowledges limitations—Ally does not yet handle alt text for images, accessible links, forms, mathematical formulas, or color contrast—and calls for integrating these approaches into mainstream PDF creation tools rather than treating remediation as a separate afterthought. For organizations producing PDF content, this research provides evidence that investing in better tools pays dividends in both efficiency and accessibility quality.
Tags: PDF accessibility · document remediation · content tagging · reading order · accessible tables · heading structure · assistive technology · screen readers · usability study
Standards referenced: PDF/UA-1 · ISO 32000-1 · Matterhorn Protocol