Supporting Older Adults in Using Complex User Interfaces with Augmented Reality
Junhan Kong, Anhong Guo, Jeffrey P. Bigham · 2019 · Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS) · doi:10.1145/3308561.3354593
Summary
This demonstration paper presents an augmented reality system that provides step-by-step visual guidance to help older adults navigate complex user interfaces. The core problem is that older adults frequently struggle with complex digital interfaces — from self-service kiosks and ATMs to appliance control panels and software applications — due to cognitive load, unfamiliarity, and the proliferation of features they don't need. Existing tutorial systems (video tutorials, on-screen walkthroughs, printed manuals) are themselves often difficult to use, requiring users to switch between the tutorial and the actual interface, remember steps, and translate instructions to their specific context. The system uses Apple ARKit to detect a physical or digital interface through the phone's camera view, then overlays augmented reality visual guidance directly on the real-world interface, highlighting which button or element to interact with next in a pre-specified task sequence. The system works by first capturing and annotating a reference image of the target interface, defining the interactive elements and their locations, then specifying task sequences as ordered lists of interactions. At runtime, when the user points their phone camera at the interface, the system recognizes the interface, determines the user's current step in the task, and renders AR overlays (such as highlighted regions or arrows) directly on the camera view pointing to the next element to interact with.
Key findings
The system was demonstrated with two example interfaces: a complex coffee machine control panel and a desktop software application. For each, task sequences were pre-defined (e.g., "make an espresso" maps to a sequence of button presses on the coffee machine). The AR overlay provides a direct, in-context visual cue that eliminates the need for users to mentally map instructions from a separate tutorial to the actual interface. The system advances through the task steps as the user completes each interaction. The approach is designed to be generalizable to any visual interface that can be captured as a reference image, including appliance control panels, self-checkout machines, ticket kiosks, and software interfaces displayed on screens. The prototype demonstrated the feasibility of real-time interface detection and AR overlay rendering using consumer smartphone hardware. Limitations noted include the requirement for pre-specification of both the interface layout and the task sequences, and the need for the user to hold the phone while interacting with the interface.
Relevance
This research addresses a practical and growing problem: as digital interfaces proliferate in everyday life — from smart home appliances to self-service kiosks to increasingly complex software — older adults who have difficulty learning and remembering interface procedures face growing exclusion. The AR approach has a key advantage over traditional tutorials: guidance is delivered directly on the real interface in real-time, eliminating the cognitive overhead of translating instructions from a separate document or video to the actual context. This is particularly important for older adults who may have reduced working memory capacity, making it harder to hold tutorial steps in mind while switching to the actual interface. For accessibility practitioners, the system illustrates how augmented reality can serve as a cognitive prosthetic — not replacing the user's decision-making but reducing the memory and attention demands of interface navigation. The approach is complementary to interface simplification efforts like Morphic: while simplification reduces the interface complexity itself, AR guidance helps users navigate interfaces that cannot be simplified (such as third-party kiosks or shared appliances). The concept could also benefit people with cognitive disabilities, acquired brain injuries, or anyone encountering an unfamiliar complex interface for the first time.
Tags: older adults · aging · augmented reality · cognitive support · user interface · tutorial systems · computer vision · step-by-step guidance · digital literacy · assistive technology