Your AI Agent Says the Vulnerability Is Fixed. There's a Coin-Flip Chance It Isn't.
New research from 1Password's Off-by-1 Labs found that AI-generated vulnerability patches fail to fully resolve the underlying flaw more than half the time, raising the stakes for independent evaluation of agentic security work.
1Password ran two frontier models through six real CVEs and 6,080 patches. Most of them did not hold.
Every AI coding agent will tell you, with total confidence, that the vulnerability is fixed. New research says that confidence is wrong more often than it is right. 1Password's Off-by-1 Labs put ChatGPT 5.5 and Claude Opus 4.8 through six disclosed vulnerabilities in widely used open-source software and generated 6,080 candidate patches. Fewer than three in ten actually closed the hole without changing how the software behaves.
53.9%
of patches failed, broke something, or both
26%
fully resolved the vulnerability cleanly
6,080
patches generated across six CVEs
The study: six CVEs, two frontier models, 6,080 patches
Off-by-1 Labs built a testing framework called FLAWED (Fix-Like Artifacts With Embedded Defects) specifically to catch patches that look correct but are not. They ran ChatGPT 5.5 and Claude Opus 4.8 against six complex, real-world vulnerabilities in widely used open-source software.
- Linux
- Google Chrome
- Apache ActiveMQ
- Spring AI
- Exim
- Gemini CLI
Each model generated many candidate patches per vulnerability, for a combined total of 6,080 patches. Every patch was scored against the original exploit and against the surrounding code, not just the one demonstrated attack.
Tunnel vision: why the patch looks right and is not
The researchers found a consistent failure pattern they call tunnel vision. Models fixate on the exact proof-of-concept exploit in front of them and patch that path specifically, while similar vulnerable code elsewhere in the same application goes untouched. The result is a patch that passes the one test you gave it and fails every test you did not.
- A patch that blocks the demonstrated exploit but leaves a near-identical code path open
- A patch that fixes the symptom, such as the crash or the injection point, without touching the root cause
- A patch that succeeds by quietly changing what the software does, which is its own kind of regression
WHY IT MATTERS
If your review process is checking whether the agent says it fixed the bug, you are trusting the same tunnel vision that created the gap in the first place.
The numbers behind the headline
Across all 6,080 patches, the outcomes split three ways.
| Outcome | Share of patches |
|---|---|
| Fully resolved, no behavior change | 26% |
| Resolved, but changed application behavior | 20.1% |
| Failed to resolve, introduced a new flaw, or both | 53.9% |
Why the agent saying it is fixed is not a security gate
No conditions we identified resulted in generated patches that would be consistently safe to trust without significant human supervision.
That is the researchers' own conclusion, not a vendor's marketing line. It applies to two of the strongest frontier models available, under close to ideal test conditions. Production environments are messier. The real-world failure rate is unlikely to be better than 53.9%.
What an independent evaluation actually checks
Closing the gap between the agent saying it is fixed and it actually being fixed takes a verification step the agent cannot perform on itself.
- Re-run the original exploit and its known variants, not just the reported proof of concept
- Diff application behavior before and after the patch to catch silent regressions
- Sweep the codebase for structurally similar code paths the patch did not touch
- Score the fix against a severity-appropriate bar, not a binary pass or fail
This is exactly the discipline an agentic security evaluation is built to enforce: independently verifying what an AI system claims to have done, before that claim becomes production risk.
