Self-Adapting User Interfaces as Assistive Technology for Handheld Mobile Devices
Robert Dodd · 2006 · Assets '06: Proceedings of the 8th International ACM SIGACCESS Conference on Computers and Accessibility · doi:10.1145/1168987.1169064
Summary
This paper addresses a fundamental challenge in mobile device accessibility: the limitations of bolt-on assistive technology solutions for handheld devices. Published at Assets '06, the premier ACM conference on accessible computing, it proposes a paradigm shift from retrofitting accessibility onto existing visual interfaces toward building interfaces that are intrinsically accessible through self-adaptation. The paper examines the state of mobile assistive technology in the mid-2000s, using the Freedom Scientific PacMate as a case study. The PacMate, a PDA designed for low-vision users, replaced the standard touchscreen with an 88-key keyboard and relied entirely on text-to-speech output. While functionally accessible for its target users, it demonstrated the core problem: assistive technology that fundamentally alters the device's form factor and forces users to navigate visual metaphors through non-visual means. Dodd draws a critical distinction between functional accessibility (meeting the needs of specific user profiles) and intrinsic accessibility (building interfaces whose underlying architecture supports adaptation across a broad range of user capabilities). The paper argues that existing mobile platforms like Windows CE expose only syntactic meaning of visual elements to assistive technology, providing little semantic or contextual information. This forces assistive technology developers into workarounds rather than genuine multimodal interaction. The proposed solution involves late binding of abstract user interface descriptions to concrete realizations at runtime, considering four key factors: user capabilities (sight, hearing, mobility, cognition), content semantics, environmental conditions (lighting, noise, device stability), and device capability (available design spaces under various operating conditions such as power saving modes).
Key findings
The paper introduces the concept of semantic decomposition as an alternative to traditional task-based decomposition for modeling accessible interfaces. Rather than breaking interfaces into functional task hierarchies that intermix abstract concepts with physical representations, semantic decomposition views the interface as a hierarchy of ontological dependencies. The rendered interface depends on models of visual, sonic, and haptic design spaces combined with user profiling and metaphor expression models. This approach maps naturally to Executable UML, where each concern is represented as a problem domain with dependencies expressed in tabular form, enabling the data-driven late binding required for self-adaptation. The paper also proposes viewing the system through the lens of collaborating intelligent agents, where interface construction emerges from negotiation between agents representing user preferences, user capabilities, environmental conditions, and device characteristics. This perspective not only provides a more intuitive practitioner's view of interface development but opens future directions: if the agents involved in construction are formally defined, accessibility itself becomes definable using formal methods and game theory.
Relevance
Although written in the PDA era, this paper's core arguments remain remarkably relevant. The distinction between functional and intrinsic accessibility applies directly to modern mobile and web development. Today's responsive design frameworks and ARIA specifications represent incremental progress toward the semantic richness Dodd advocated, but most interfaces still treat accessibility as a bolt-on concern rather than an architectural foundation. The concept of mapping content meaning to interaction metaphors across multiple design spaces (visual, sonic, haptic) anticipates modern multimodal interface design and the growing importance of voice interfaces, haptic feedback, and adaptive layouts. For practitioners, the paper challenges the assumption that accessibility can be adequately addressed by adding ARIA attributes and screen reader support to visually-designed interfaces, arguing instead for architectures where accessible interaction is a first-class design concern from the outset.
Tags: mobile accessibility · self-adapting interfaces · assistive technology · user modeling · design space