In-context Q&A to Support Blind People Using Smartphones
André Rodrigues, Kyle Montague, Hugo Nicolau, João Guerreiro, Tiago Guerreiro · 2017 · Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS '17) · doi:10.1145/3132525.3132555
Summary
This paper addresses a pervasive but understudied problem: the daily challenges blind people face when using smartphone applications go far beyond the touchscreen gesture difficulties that most accessibility research focuses on. The researchers conducted workshops with 42 blind participants spanning newcomers, novice, and expert smartphone users over a two-day period. Workshops for newcomers and novices guided participants through basic phone tasks, while expert sessions centered on questions and doubts. The findings revealed that challenges exist at all expertise levels and are primarily about understanding how applications work — discovering features, interpreting layouts, dealing with unlabeled buttons, and working around bugs — rather than physical interaction alone. Critically, most of these challenges could only be overcome through help from others, creating heavy reliance on support networks that burdens both users and their helpers. Based on these findings, the researchers developed Hint Me!, a human-powered in-app assistance service for Android. The system allows blind users to ask context-aware questions from within any application via an always-available overlay button. Questions are automatically augmented with contextual information including a screenshot, the app's DOM tree structure, element details (alternate text, position, dimensions), and the specific element the user highlights. Answers accumulate in a shared knowledge base that future users can browse, filtered by their current app, screen, or selected element.
Key findings
The workshops identified three major themes. First, challenges go far beyond touch interaction — users at all levels struggled with understanding app interfaces, discovering features, and dealing with accessibility problems like unlabeled buttons. Second, blind users are independent and community learners who form informal support communities around trusted technology experts, but this creates burden (one specialist's wife noted "He helps everyone except me!"). Third, getting help from sighted friends and family is often inadequate because sighted people don't understand screen reader workflows ("the problem is not them not knowing how to solve the problem, the problem is not knowing how to explain to us how we can solve it"). In a design probe evaluation with 6 blind participants (ages 31-62) using popular apps (Facebook, WhatsApp, Du Speed Booster, Spotify), all participants responded positively. Over two sessions, 21 questions were created and 16 answers consulted. Participants valued Hint Me! primarily as a self-organized learning tool that promoted autonomy — "with Hint Me! we have greater autonomy in using the device because we are not dependent on others." They anticipated using it for labeling unknown buttons, understanding layouts, finding bug workarounds, and learning task workflows. Participants preferred anonymous volunteer answerers over friends/family, believing dedicated volunteers would be more competent with accessibility services and that asking friends would be burdensome.
Relevance
This research highlights a critical gap in smartphone accessibility: even when apps are technically screen-reader compatible, blind users still struggle to build mental models of interfaces and discover functionality. The insight that the real barrier is knowledge, not just interaction, shifts the accessibility conversation from building better gestures to building better support systems. Hint Me!'s context-aware approach — automatically capturing the exact app state, screen elements, and DOM structure when a question is asked — is a significant improvement over generic online forums or social media groups where users must laboriously describe their problem out of context. For accessibility practitioners and app developers, the workshop findings offer a sobering reminder that even well-labeled interfaces can be confusing when the overall layout and feature organization is not conveyed to screen reader users. The preference for anonymous expert volunteers over personal networks reveals the social cost of technology-related help-seeking for blind people and suggests that scalable peer support systems could reduce dependency while preserving dignity.
Tags: blindness · smartphone accessibility · screen readers · crowdsourcing · human computation · mobile accessibility · peer support · knowledge sharing