Making Real-World Interfaces Accessible Through Crowdsourcing, Computer Vision, and Fabrication
Anhong Guo, Jeffrey P. Bigham · 2017 · Proceedings of the 14th International Web for All Conference (W4A) · doi:10.1145/3058555.3058586
Summary
This demonstration paper presents two complementary systems designed to make inaccessible physical interfaces usable by blind people: VizLens and Facade. The research addresses a fundamental and long-standing accessibility problem — the vast majority of physical interfaces in everyday life, from microwaves and coffee machines to checkout terminals and office equipment, lack any form of non-visual feedback. Blind users typically depend on sighted assistance to either operate these devices or apply tactile labels, neither of which scales well or supports independence. VizLens functions as a screen reader for real-world physical interfaces. A blind user photographs an unfamiliar interface with their smartphone, and the image is sent to crowd workers who quickly annotate the layout, identifying and describing each button and control. This crowd-labeled reference image then enables real-time computer vision to track the interface as the user points at it, providing audio feedback about which element their finger is hovering over. The system combines the robustness of human interpretation with the speed and low cost of automated vision, creating an interactive exploration experience analogous to how screen readers work on digital interfaces. Building on VizLens, the researchers developed Facade, which shifts from a real-time software solution to a permanent physical one. Facade uses the same crowdsourced annotation pipeline but goes further by automatically generating a 3D model of tactile buttons that can be printed and physically overlaid onto the original interface panel.
Key findings
A user study with 10 blind participants demonstrated that VizLens provides accurate and usable real-time feedback for navigating physical interfaces. The crowdsourcing workflow proved remarkably efficient: interface annotation took an average of just 8 minutes, achieved 99.7% accuracy, and cost only $1.15 per interface. The system was extended based on participant feedback to handle interfaces that change state (such as touchscreens), read dynamic information using crowd-assisted OCR, and support head-mounted cameras for hands-free operation. For Facade, a study with 11 blind participants showed that users could independently create and affix tactile overlays to appliance interfaces without sighted assistance. The fabricated overlays provided rich, usable tactile feedback using varied button shapes and reading mediums. Compared with traditional embossed labelers, Facade eliminated the need for in-person sighted help, offered richer tactile information, and reduced memory load by providing a legend and in-app support. The use of a dollar bill as a fiducial marker for perspective transformation was a practical design choice that ensured size calibration without specialized equipment.
Relevance
This work highlights a critical gap in accessibility that digital-first approaches often overlook: the physical world remains largely inaccessible. While web and mobile accessibility have well-established standards like WCAG and ARIA, physical interfaces lack equivalent frameworks. The VizLens and Facade systems demonstrate creative approaches to bridging this gap using commodity hardware (smartphones and consumer 3D printers) and scalable human computation. For accessibility practitioners, the research underscores that the Internet of Things alone will not solve physical accessibility in the near or medium term, and interim solutions combining crowdsourcing, computer vision, and fabrication are needed. The crowdsourced annotation model also raises interesting questions about sustainable accessibility infrastructure — whether community-maintained databases of interface layouts could scale to cover the diversity of devices people encounter daily.
Tags: physical accessibility · computer vision · crowdsourcing · 3D printing · assistive technology · blindness · tactile interfaces · screen readers