Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots
Jason Wu, Xiaoyi Zhang, Jeff Nichols, Jeffrey P. Bigham · 2021 · The 34th Annual ACM Symposium on User Interface Software and Technology (UIST)
This paper introduces screen parsing, the task of predicting UI elements and their hierarchical relationships from a screenshot alone. While prior work could detect individual UI elements on a screen (element detection), those approaches produced flat lists of elements with no…
screen readers · mobile accessibility · computer vision · UI semantics · machine learning