The PROBE study set ten AI coding agents loose on twenty open-source projects. It compared their bug reports with the 2,457 bugs the projects' volunteer maintainers fixed over eighteen months. Reviewers judged 85% of the agents' reports to be real bugs. Even so, the best agent caught only about a fifth of what maintainers fixed, and the agents' findings resembled one another more than they resembled the humans'.
Maintainers' choices are lopsided because their users are. Someone complained, or a feature visibly broke. That lopsidedness tells you who depends on the software. As agent reports fill the same queues, one shared machine sense of what's worth fixing could outweigh it simply by arriving in greater volume.
- Bigger models, same gap: Larger models caught more of the bugs maintainers fixed, rising from 8% to 17%. Their reports matched those fixes no more often.
- Little overlap: At most 8.1% of any agent's reports matched a bug that maintainers fixed.
- Same category, different picks: Even within a single type of bug, agents and humans chose different instances.
- The authors' guess: Agents gravitate to subtle logic errors. People fix what visibly breaks or contradicts the documentation.
- Still untested: Whether "taste" causes the split, and whether agent reports are worth a maintainer's time.
- Status: Preprint posted Oct. 26, 2026. It has not yet been peer-reviewed.
