Accessible Web Automation Interface: A User Study
Yury Puzis · 2012 · Proceedings of the 14th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2012) · doi:10.1145/2384916.2384999
Summary
This doctoral consortium paper presents the evaluation of two web automation user interfaces designed to help blind and low-vision screen reader users complete web browsing tasks more efficiently. The fundamental problem is that while the web has become essential for daily tasks like shopping and paying bills, its growing sophistication outpaces assistive technology capabilities. Screen reader users must navigate content sequentially, often spending significantly more time on tasks than sighted users, and may miss dynamically updated content that appears silently on pages. Traditional web automation through macros has three key limitations for visually impaired users: macros lack flexibility to deviate from prerecorded sequences, they don't address non-interactive content discovery, and recording macros itself requires manipulating the browser and screen reader simultaneously. The author's approach eliminates macro recording entirely — instead, every user action is automatically recorded, and contextually relevant browsing actions are suggested to the user upon request based on a statistical model of their browsing history. Two "Assistant" interfaces (A and B) were built on top of the Capti web browsing application (similar to JAWS), with the workflow being: browse normally, query for suggestions, examine them via the screen reader interface, and accept or reject with voice feedback.
Key findings
The experiments demonstrated that the implicit web automation approach has potential to significantly increase both accessibility and usability of web pages by reducing interaction time and enhancing user experience. Unlike explicit macro approaches where users must manually start/stop recording, execute actions flawlessly, and save recordings, this implicit approach shifts the burden from the user to the automation tool. The system automatically builds a computational model from browsing history to generate suggestions, meaning users benefit from automation without any setup effort. The approach addresses a key gap in existing automation tools: the inability to help users find non-interactive content such as articles, form submission notifications, and other information that may appear dynamically on the page. The two interface variants were evaluated to compare different ways of presenting automation suggestions to screen reader users, with results suggesting the approach is viable for real-world use.
Relevance
This paper addresses a persistent and growing problem in web accessibility: the widening gap between the capabilities of modern web applications and the tools available to screen reader users. For accessibility practitioners, the concept of implicit automation — learning from user behavior to proactively suggest shortcuts — foreshadows modern AI-powered assistive features and browser automation tools. The finding that traditional macro-based automation is poorly suited to screen reader users highlights the importance of designing assistive tools that account for the unique interaction patterns of their target users, rather than adapting tools designed for sighted users. The work is particularly relevant today as web applications have become even more dynamic and complex, making the efficiency gap between sighted and screen reader browsing even larger. The approach of reducing repetitive interaction through intelligent suggestions aligns with current trends in AI-assisted accessibility and browser automation.
Tags: web accessibility · web automation · screen readers · blind users · low vision · JAWS · browsing efficiency · user study · assistive technology · implicit automation