ApplianceReader: A Wearable, Crowdsourced, Vision-based System to Make Appliances Accessible
Anhong Guo, Xiang "Anthony" Chen, Jeffrey P. Bigham · 2015 · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/2702613.2732755
Summary
This work-in-progress paper presents ApplianceReader, a wearable system that combines a point-of-view camera (Google Glass) with crowdsourcing and computer vision to make everyday appliance control panels accessible to people with visual impairments. The system addresses a growing accessibility problem: modern appliances increasingly replace tactile physical buttons with flat touchscreens and touchpads that provide no tactile feedback, making them unusable without sight. The authors conducted an hour-long observation and interview at a visually impaired person's home, confirming that appliance inaccessibility creates significant daily challenges and social burden, as users must frequently ask neighbors or friends for help. ApplianceReader works in two phases. In the initial phase, when a user encounters a new appliance, they photograph it with their wearable camera and send the image to crowd workers on Amazon Mechanical Turk, who draw bounding boxes around controls and describe each button's function (taking approximately one minute per control). This creates a labeled reference image. In the real-time phase, when the user returns to that appliance, they simply point their finger at controls while the camera watches — computer vision techniques (SURF feature detection, FLANN matching, RANSAC perspective transformation) match the live view to the reference image, detect the fingertip location via image subtraction and convex hull analysis, and announce which button the user is pointing at via text-to-speech.
Key findings
Initial testing demonstrated that the system could process video at 2 frames per second at 640x360 resolution on a MacBook Pro, providing real-time audio feedback as users explored appliance controls. Crowd workers took approximately one minute to label a single control on an appliance interface. The fingertip detection approach — subtracting the perspective-transformed reference image from the cropped input image to isolate the finger — proved effective for identifying which button a user was pointing at. The system also envisioned a growing library of crowd-labeled reference images that could be shared among users, meaning common appliances in shared spaces (office kitchens, laundromats) would only need to be labeled once. The paper identified key challenges including photo quality from visually impaired users, varying lighting conditions, and the need for a moderator algorithm to coordinate between CV and crowdsourcing based on recognition confidence — falling back to real-time crowd assistance when computer vision results are uncertain.
Relevance
ApplianceReader addresses a practical accessibility barrier that is often overlooked: the inaccessibility of everyday appliance controls. As home appliances increasingly adopt flat touchscreen interfaces, this problem has only grown since the paper's publication. The hybrid crowd-AI approach is noteworthy — crowdsourcing provides the initial knowledge that computer vision cannot generate alone (what each button does), while CV handles the real-time interactive task of tracking where the user is pointing. This division of labor between human intelligence and machine capability exemplifies a design pattern that has proven valuable across accessibility research. For practitioners, the key insight is that a one-time human labeling investment can enable ongoing independent use, and that shared databases of labeled appliances could scale this approach to benefit many users. The work also highlights the importance of wearable cameras for hands-free interaction — users need both hands to operate appliances, making handheld phone-based solutions impractical for this use case.
Tags: appliance accessibility · crowdsourcing · computer vision · wearable technology · blindness · visual impairment · assistive technology · Google Glass