Computer Use Agent

A computer use agent is an AI system powered by multimodal large language models that operates a computer by taking screenshots and performing mouse, keyboard, and scroll actions — mirroring the interactions of a sighted user to complete natural language tasks such as booking a flight or editing a document. Examples include OpenAI Operator, Anthropic Computer Use, Microsoft Copilot, and Google DeepMind Project Astra. From an accessibility perspective, CUAs present both promise and risk: they could reduce computing barriers for some users by automating complex workflows, but their reliance on visual screenshots and mouse-centric interaction patterns creates collaboration gaps for blind and low-vision users who rely on screen readers, magnifiers, or keyboard-only navigation. Research has shown that CUA success rates drop sharply under assistive technology conditions, highlighting the need for accessibility-aware agent design.