Deep Learning for Automatically Detecting Sidewalk Accessibility Problems Using Streetscape Imagery
Galen Weld, Esther Jang, Anthony Li, Aileen Zeng, Kurtis Heimerl, Jon E. Froehlich · 2019 · Proceedings of the 21st International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS 2019) · doi:10.1145/3308561.3353798
Summary
This paper presents the first application of deep learning to automatically assess sidewalk accessibility from Google Street View (GSV) panoramas, addressing four types of accessibility problems: curb ramps, missing curb ramps, sidewalk obstructions, and surface problems. Traditional sidewalk audits are expensive and labour-intensive — Seattle’s first comprehensive assessment in 2016 required 14 interns working nearly a year. The authors leverage the Project Sidewalk open dataset of over 300,000 crowdsourced image-based sidewalk accessibility labels from Washington, D.C. to train a modified ResNet-18 deep convolutional neural network. Their key architectural innovation is extending the standard image-only ResNet to incorporate non-image contextual features: positional features encoding a point’s location within a panorama (distance, depth, angle) and geographic features encoding the panorama’s location within the city’s street network (street direction, distance to city centre, position within a block). The research investigates two applications: automatically validating existing crowdsourced labels (confirming or flagging human-placed labels) and automatically labeling new panoramas to find and classify accessibility problems without any human input. The system is open source and implemented in PyTorch.
Key findings
For auto-validation, the model achieved 81.3% average precision and 77.2% recall, performing best on curb ramps (93.2% precision, 96.8% recall) and worst on surface problems (75.3% precision, 59.5% recall), which vary widely in appearance and location. For auto-labeling — the harder task of finding problems in unlabeled panoramas — overall performance was 47.0% precision and 41.2% recall, with curb ramps again performing best (89.8% precision, 48.9% recall). Critically, the auto-labeling system’s performance met or exceeded human crowdworker performance in some cases: at a tuned confidence threshold, the model achieved 38.6% precision and 49.7% recall, surpassing the 37% precision and 46% recall of a five-person majority-vote human labeling system from prior work. Adding contextual (non-image) features provided marginal overall gains but significantly helped specific categories — surface problem detection jumped from 48.5% to 56.7% recall with all features. Performance scaled positively with training set size and had not plateaued even at 213,000 training crops, suggesting more data would yield further improvements. Cross-city generalizability experiments using Seattle and Newberg, Oregon data showed that models pre-trained on D.C. data and fine-tuned on small city-specific datasets achieved competitive performance (76-81% precision, 83-90% recall), demonstrating transferability.
Relevance
This work has major implications for urban accessibility policy and advocacy. The long-term vision — automatically assessing an entire city’s sidewalk accessibility within hours rather than months — would give disability advocates a powerful accountability tool for monitoring ADA compliance. For municipalities, it offers a scalable alternative to manual sidewalk audits that could enable data-driven infrastructure investment. The authors thoughtfully address biases: their four label types don’t comprehensively capture accessibility (missing crosswalks, accessible pedestrian signals, stairs, transit stops), and models trained on one city’s streetscape may not generalize to cities with different infrastructure styles. The discussion of how ML failures could incorrectly certify inaccessible sidewalks as accessible — potentially informing harmful policy decisions — demonstrates responsible AI thinking. For accessibility practitioners, this research connects to the broader Project Sidewalk ecosystem and demonstrates how large-scale crowdsourced accessibility data can fuel automated assessment tools, creating a pipeline from community data collection to city-wide accessibility mapping.
Tags: computer vision · deep learning · sidewalk accessibility · curb ramps · crowdsourcing · urban accessibility · mobility disability · machine learning · Google Street View · pedestrian infrastructure
Standards referenced: ADA