BrowseWithMe: An Online Clothes Shopping Assistant for People with Visual Impairments
Abigale J. Stangl, Esha Kothari, Suyog D. Jain, Tom Yeh, Kristen Grauman, Danna Gurari · 2018 · Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2018) · doi:10.1145/3234695.3236337
Summary
This paper from the University of Colorado Boulder and University of Texas at Austin addresses the inaccessibility of online clothes shopping for people with visual impairments through both empirical investigation and a prototype AI-powered assistant called BrowseWithMe. The researchers first conducted semi-structured interviews with 8 participants (4 blind, 4 low vision, ages 26-58) using a Value Sensitive Design approach to understand clothes shopping experiences both in brick-and-mortar stores and online. Clothes shopping emerged as personally significant — participants reported it affects their ability to achieve personal goals (avg 3.87/5) and professional goals (avg 3.62/5), and serves as a means of expressing identity and creativity. Three key factors shaped shopping experiences: obtaining product information (frustrated by missing Alt text and information overload from screen readers), shopping assistance quality (mixed experiences with in-store assistants), and mobility/transportation challenges. Online shopping was seen as promising but deeply frustrating — participants described abandoning shopping sessions when web sites failed to follow WCAG guidelines, when images lacked Alt text, or when the sheer volume of unparsed information became overwhelming. Based on these findings, the team developed BrowseWithMe, which uses AI to automatically convert product web pages into structured representations that users can query interactively. The back-end combines a natural language processing module (extracting price, material, and details from page source code) with a computer vision module using semantic segmentation (trained on 9,000 images via CAFFE) to identify clothing items, detect their colors using the xkcd color palette, and enable smart magnification.
Key findings
BrowseWithMe's key innovation is shifting users from passive listeners of lengthy screen reader output to active solicitors of specific information through structured queries ("price," "material," "details," "describe outfit," "[item] color"). A typical product page produces approximately 205 words of screen reader output with no Alt text — BrowseWithMe lets users access just the information they want. In usability testing with the same 8 participants, all learned to use the system independently after training on just two web pages. The consistent command structure across different shopping sites (ASOS, H&M, Forever 21) was cited as the primary benefit — eliminating the need to learn each site's unique layout. Low vision participants particularly valued the smart magnification feature, which crops and enlarges individual clothing items rather than zooming the entire page. In a technical evaluation comparing BrowseWithMe's AI-generated image descriptions to existing Alt text, crowd workers on Amazon Mechanical Turk correctly matched images to BrowseWithMe descriptions 70% of the time (median 80%) versus 56% for Alt text (median 60%) — a 20 percentage point improvement in median accuracy. The system performed best for tops (83% vs 40% for Alt text) and struggled most with skirts due to segmentation errors confusing skirts and dresses. Participants requested additional features including pattern analysis, neckline descriptions, sleeve length details, and the ability to share outfits with friends for feedback.
Relevance
This study demonstrates a compelling alternative to the traditional approach of making web sites accessible by fixing the sites themselves. Rather than relying on developers to follow WCAG guidelines (which many do not), BrowseWithMe acts as an intelligent intermediary that restructures any product page into a consistent, queryable format. This "web wrapper" approach has broad implications beyond shopping — the same principle of AI-powered content extraction and restructuring could make many types of inaccessible web content usable. For web developers, the study provides a stark illustration of why Alt text matters and what information visually impaired shoppers actually need (not just "green top" but specific details about material, fit, neckline, pattern, and available colors). The finding that AI-generated descriptions outperformed existing Alt text in a controlled domain highlights both the promise of automated image description and the poor current state of human-authored Alt text on commercial web sites. The paper also contributes valuable qualitative data about the emotional and social significance of clothes shopping for people with visual impairments — it is not merely functional but tied to identity, creativity, and professional presentation.
Tags: blindness · low vision · web accessibility · online shopping · computer vision · artificial intelligence · image description · screen reader · alt text · semantic segmentation
Standards referenced: WCAG