Automatic Identification of Widgets and their Subcomponents Based on a Classification Pipeline for DOM Mutation Records
Eduardo Henrique Rizo, Renata Pontin de Mattos Fortes, Humberto Lidio Antonelli, Willian Massami Watanabe · 2019 · Proceedings of the 16th International Web for All Conference (W4A) · doi:10.1145/3315002.3317555
Summary
This paper presents a machine learning pipeline for automatically classifying web widgets (specifically dropdown menus) and their subcomponents by analyzing DOM mutation records — the dynamic changes that occur in a web page's HTML structure when users interact with it or visual effects trigger. The core problem is that Rich Internet Applications (RIAs) use sophisticated interactive widgets (dropdown menus, sliders, calendars, tooltips, etc.) that generate dynamic content through JavaScript, but many of these widgets do not implement the WAI-ARIA specification's semantic attributes needed for screen reader accessibility. Before ARIA conformance can be evaluated or code can be automatically adapted, the system must first identify what type of widget each DOM change represents. The approach uses three components: a Logger (a Chrome browser extension called ARIA Observer that non-intrusively captures DOM mutations via the MutationObserver API as users naturally browse websites), a Receiver Module (a web application for data persistence, manual classification of training data, and dataset generation), and a Machine Learning Pipeline with two sequential classifiers. The first classifier identifies whether a DOM mutation represents a dropdown menu widget using features like widget size, position, activator position, number of children, DOM tree dimensions, text node counts, and number of links. The second classifier then identifies the subcomponents (menu items) of identified widgets using features like parent node type, child node sizes, positions, HTML element counts, and dimensional standard deviation.
Key findings
The system was evaluated with data from 12 real Brazilian websites across diverse categories (news portals, tech sites, WordPress blogs, banks, post office, government portal). Training data from 8 websites yielded 1,207 mutation records (94 dropdown menus, 1,024 subcomponents), while test data from 4 different websites yielded 392 records (107 dropdown menus, 977 subcomponents). Five machine learning algorithms were compared using 10-fold cross-validation: kNN, Decision Tree, SVM, Random Forest, and Logistic Regression. Random Forest performed best for both classifiers — achieving 0.935 F-measure for widget classification and 0.977 F-measure for subcomponent classification on training data. On the separate unbiased test dataset, the pipeline achieved 0.890 F-measure for dropdown menu identification (precision 0.903, recall 0.878) and 0.950 F-measure for subcomponent identification (precision 0.995, recall 0.909), yielding an overall average of 0.91 F-measure. Key features for widget classification included widget size, screen position, number of children, and DOM tree dimensions. For subcomponents, the most discriminating features were dimensional similarity (low standard deviation) and positional alignment (items sharing the same x or y coordinates). A notable limitation is that the pipeline was trained only on JavaScript-generated dropdown menus; CSS/HTML-only dropdown implementations were not covered.
Relevance
This research addresses a fundamental gap in automated web accessibility evaluation: the inability of current tools to properly assess the accessibility of dynamic, JavaScript-driven web widgets. Existing accessibility evaluation tools (like AChecker, WAVE, and DaSilva) primarily analyze static HTML and cannot evaluate the dynamic behavior of widgets as they respond to user interaction. Since WAI-ARIA compliance requires that widgets be semantically marked with appropriate roles, states, and properties, automated tools need to first identify what type of widget a DOM change represents before they can check whether the correct ARIA attributes are applied. For accessibility practitioners and tool developers, this work provides a practical building block: once widgets and their subcomponents are automatically classified, the system can verify ARIA conformance (e.g., checking that a dropdown menu has role="menu" and its items have role="menuitem") or even automatically inject the missing ARIA attributes to remediate accessibility issues. The non-intrusive data collection approach — capturing mutations during natural browsing rather than using web crawlers — is important because it captures the actual dynamic behavior users encounter. While currently limited to dropdown menus, the approach is designed to be extensible to other widget types (tabs, sliders, calendars) with appropriate training data.
Tags: WAI-ARIA · machine learning · web accessibility · automated testing · widgets · DOM · rich internet applications · screen reader · accessibility evaluation · dropdown menu
Standards referenced: WAI-ARIA 1.1 · WCAG