Drop-Down Menu Widget Identification Using HTML Structure Changes Classification
Humberto Lidio Antonelli, Rodrigo Augusto Igawa, Renata Pontin De Mattos Fortes, Eduardo Henrique Rizo, Willian Massami Watanabe · 2018 · ACM Transactions on Accessible Computing (TACCESS) · doi:10.1145/3178854
Summary
This paper addresses a critical gap in web accessibility: the widespread failure of interactive widgets in rich internet applications (RIAs) to implement WAI-ARIA markup, rendering them inaccessible to screen reader users and other assistive technology users. The authors propose a machine-learning approach for automatically identifying drop-down menu widgets by classifying HTML structure changes captured through the MutationObserver API, an HTML5 technology that logs all DOM modifications in real time. The rationale is that if widgets can be automatically identified, they can then be automatically evaluated for ARIA compliance or even adapted with correct ARIA roles and properties on the fly. The research collected 7,331 mutation records from 49 high-traffic web applications drawn from Alexa rankings, capturing the DOM changes triggered when users interact with drop-down menu activator elements. These mutation records were analyzed to extract features including element position, number of child elements, number of words, and other DOM structural attributes. Three classification algorithms were compared — Support Vector Machine (SVM), decision trees (C4.5/J48), and Random Forest — using 10-fold cross-validation on a learning dataset and evaluation on a separate held-out test dataset. The experimental design carefully separated training and testing data to avoid bias, and all source code, datasets, and scripts were published as open-source on GitHub for reproducibility.
Key findings
Random Forest achieved the best overall performance for identifying drop-down menu widgets, with an F-measure of 0.94 on the test dataset, outperforming both SVM (F-measure 0.84) and decision trees (F-measure 0.91). The feature set based on DOM structural changes — position, number of elements, and word count — proved more effective than previous approaches that relied on static DOM attributes like HTML element types and link percentages. A key insight was that mutation records dispatched after a user focuses on a widget activator element contain sufficient structural information to distinguish drop-down menus from other dynamic page behaviours such as AJAX content loading, carousel animations, lazy loading, and visual effects. The approach significantly outperformed prior state-of-the-art techniques by Melnyk et al. (2014), which used manually selected mutation records and a smaller, more controlled dataset. The authors also found that their real-world dataset from high-traffic websites, where each site implements widgets differently using custom JavaScript, demonstrated the generalizability of the approach across diverse implementations.
Relevance
This research tackles a persistent and widespread web accessibility problem: the vast majority of interactive JavaScript widgets on the web lack proper ARIA markup, making them invisible or unusable to assistive technology users. At the time of publication, no accessibility evaluation tool could check ARIA compliance of dynamic widgets. The automatic identification approach opens two important pathways: first, building automated ARIA evaluation tools that can flag non-compliant widgets at scale, and second, enabling browser-based adaptation strategies that could inject correct ARIA roles and properties into inaccessible widgets in real time. For accessibility practitioners and developers, this work highlights how machine learning can bridge the gap between the theoretical promise of ARIA and its poor real-world adoption. The limitation to JavaScript-based drop-down menus (excluding CSS-only implementations) and the focus on a single widget type suggest significant room for extending this approach to other common widget patterns like tab panels, accordions, and modal dialogs.
Tags: ARIA · web accessibility · machine learning · widgets · automated testing · rich internet applications · DOM
Standards referenced: WAI-ARIA · WCAG 2.0 · HTML5