WebSight: Using AR and WebGL shaders to assist the visually impaired
Dan Ruta, Louis Jordan, Tom James Fox, Rich Boakes · 2018 · Proceedings of the 15th International Web for All Conference (W4A 2018) · doi:10.1145/3192714.3196319
Summary
This demonstration paper presents WebSight, a browser-based augmented reality system that uses WebGL shaders to process a smartphone’s live camera feed in real time, enhancing visual features to assist people with low vision in navigating real-world environments. An estimated 217 million people worldwide have moderate to severe visual impairment, and while software often includes basic accessibility features like high contrast mode and colour inversion, real-world navigation remains a significant challenge. WebSight augments the camera feed through configurable image processing effects — primarily edge detection (using convolution operations to highlight boundaries between objects) and colour adjustment filters — that users can personalise to their specific visual needs and preferences. The effect is rendered in a VR split-screen format so users can insert their smartphone into a cheap portable VR headset (like Google Cardboard), creating a wearable see-through augmented reality experience. The system is implemented as a Progressive Web Application (PWA), meaning it runs in any modern browser without requiring app installation, can be saved to the Android home screen for instant access, works offline after first load, and avoids platform compatibility issues. Settings are persisted in local storage so users configure once and the effects load automatically on subsequent uses. The VR rendering uses Three.js (a cross-browser JavaScript 3D framework) rather than the WebVR API, chosen for its better stability and broader support at the time of development.
Key findings
The system implements edge detection through convolution operations applied to every pixel in the video feed via WebGL fragment shaders, with the output value of each pixel representing the weighted average of surrounding pixel intensities. This highlights boundaries, edges, and contours of objects, furniture, doorways, and obstacles that may be difficult for a person with low vision to perceive in the original scene. Users have full control over available effect types, strength (intensity of the edge detection or colour modification), and radius (how many surrounding pixels contribute to the convolution), enabling fine-tuning for individual visual conditions. The colour inversion filter reverses the colour palette, which can help users with specific forms of low vision who perceive better with inverted contrast. The PWA architecture ensures maximum accessibility: users need only ensure network connectivity, navigate to the web page, enable landscape mode, configure effects, enable VR mode, tap for fullscreen, and insert the phone into a headset. While this setup process involves several steps that could confuse non-technical users, the PWA approach eliminates the need for app store downloads, platform-specific development, or hardware compatibility checks. The system runs at interactive frame rates on modern smartphones, demonstrating that GPU-accelerated WebGL shader processing is viable for real-time video augmentation on mobile devices.
Relevance
WebSight demonstrates that modern web technologies (WebGL, PWAs, device camera APIs) have reached sufficient maturity to deliver real-time visual augmentation for assistive purposes without native app development. For accessibility practitioners and developers, this is significant because web-based delivery eliminates the major barriers of app discovery, installation, platform compatibility, and update distribution that limit assistive technology adoption. The approach of using configurable shader effects rather than fixed filters respects the diversity of low vision conditions — what helps someone with macular degeneration may differ entirely from what helps someone with glaucoma or diabetic retinopathy. The use of consumer VR headsets (as low as a few dollars for cardboard versions) as the display mechanism keeps costs minimal. However, as a 2-page demonstration paper, WebSight lacks user evaluation with visually impaired participants, which is essential to validate whether the edge detection and colour effects actually improve navigation and object recognition for the target population. The VR headset form factor also raises practical questions about peripheral vision, social acceptability, and whether users would wear it in public settings — concerns echoed in other wearable assistive technology research.
Tags: augmented reality · low vision · visual impairment · WebGL · edge detection · progressive web app · virtual reality · image processing · navigation · contrast enhancement