AI Overreliance

The tendency for users to trust AI systems more than is warranted by their actual accuracy, accepting AI-generated outputs without sufficient critical evaluation. In accessibility contexts, AI overreliance is a significant safety concern because blind and low vision users of AI-powered image description tools cannot visually verify the information they receive, and models produce fluent, confident-sounding descriptions even when they contain errors. Research shows BLV users rate single AI-generated image descriptions at 5.78 out of 7 for reliability, but exposing them to variations across multiple models reduces this to 3.93—suggesting substantial initial overreliance. Addressing overreliance requires trust calibration through transparency about model uncertainty and limitations.