security

Only 26% of AI-Generated Patches Worked. That's Why TrueSource Exists.

Six thousand AI-generated patches were tested. A quarter of them fixed the vulnerability without breaking the application. The statistic is the easy part.

Of everything announced at Explore, the number that stuck with me had nothing to do with GPUs.

In testing by 1Password's Off-by-1 Labs, 6,000 AI-generated patches were evaluated. Only 26 percent of them fixed the vulnerability without breaking the application.

Not 26 percent failed. Twenty-six percent worked.

That number deserves more attention than it got, because there's a lot of enthusiasm at the moment for closing the remediation gap with automation, and this is the clearest evidence I've seen that we're not there.

Why the gap exists in the first place

The pressure is real and worth stating fairly. AI has made exploitation dramatically faster. A vulnerability that once took weeks to weaponise can now be turned into something working in hours, which means the window between an advisory being published and a patch being applied has become the thing attackers aim at.

Most enterprises measure that window in weeks. Some have never measured it at all.

So the appeal of automated patching is obvious. If attackers are moving at machine speed, match them. Let a model read the advisory, generate the fix, ship it.

The 26 percent figure is what happens when somebody actually tests that idea at scale.

What the number actually measures

It isn't that three quarters of AI patches are gibberish. Many of them presumably fix the stated vulnerability. The test was whether they fixed it and left the application working, which is the only definition that matters in production. A patch that closes a CVE and breaks authentication has not helped you.

The problem underneath the number

Here's the part I find more interesting than the statistic, and it's an argument about software maintenance rather than about AI.

When you apply a generated patch to an open source component outside the upstream project, you have forked it. Quietly, often without meaning to, and usually without anyone recording that you did.

Katie Norton at IDC made this point in the coverage, and it's the right one: patching outside a maintained upstream project produces forks that nobody is accountable for long-term.

Which creates a problem that compounds. Your forked version diverges from upstream. The next real patch doesn't apply cleanly. Nobody upstream knows your variant exists, so nothing is tested against it. And the person who approved the automated patching has usually moved on by the time any of that surfaces.

Why this is worse for data engines than for libraries

A patch to a Java library affects the applications that use it. A patch to PostgreSQL or RabbitMQ can affect dozens of downstream services at once, because the data layer is where everything converges. The blast radius of an unverified fix is not uniform across your stack, and the places where automation is most tempting because the component is shared are the same places where a mistake travels furthest.

What Broadcom actually built

TrueSource brings three things under one name, and the design position is explicit: use AI for detection, use people for verification.

Spring Enterprise

Curated Spring releases from the team that maintains the framework, with proactive scanning across the dependency tree. Coverage goes well beyond Spring itself to its managed dependencies including Apache Tomcat and Kotlin, and across more than 5,000 verified Java libraries, built and signed at the exact versions each supported Spring Boot release line pins. That last detail is the one that matters operationally, because version drift between a framework and its dependencies is where a lot of real problems live.

TrueSource Trusted Artifacts

Clean-room builds at SLSA Build Level 3 for libraries across Java, Python and Node.js, plus the Bitnami Secure Images catalogue for container images. SLSA Level 3 means the build is performed on a hardened platform with provenance you can verify, which is a meaningfully different claim from a signed artifact of unknown origin.

TrueSource Data Services

The same treatment for PostgreSQL, RabbitMQ, MySQL and Valkey. Curated, hardened distributions with deployment automation. Given the blast radius point above, this is arguably the most useful part of the three.

On the AI side, Broadcom says its engineers have spent more than 12 billion tokens against frontier models over five months, scanning Spring and its dependency tree. Every patch that comes out of that process is then verified by a person.

So the claim is not that AI has no place in this. It's that AI is good at finding things and not yet trustworthy at fixing them, which matches the 26 percent figure rather neatly.

What I'd take from this regardless of vendor

You don't need to buy anything to act on most of this.

Measure your advisory-to-remediation time across the last three advisories that applied to you. Most organisations have never written this number down Establish whether anything in your estate is applying generated patches without human review, including anything a security tool does automatically Work out which of your components are shared across many services, because that is where an unverified fix does the most damage Find out whether you are running forked versions of anything, which people rarely know until they look Decide where AI sits in your own remediation process, and write it down. Detection is a reasonable place. Unreviewed application is not If you consume commercially supported open source, check what the support actually covers, because support and verification are not the same thing

The honest caveat

The 26 percent figure comes from testing by 1Password's Off-by-1 Labs and is being cited by a vendor with a product that benefits from it. That doesn't make it wrong, and the methodology is attributed rather than anonymous, but it's worth going to the primary source rather than the press release if you're going to quote it in a decision.

Also worth noting that this is a moving target. Generated patch quality is improving, and a figure from this year will not hold indefinitely. What's less likely to change quickly is the fork problem, because that's about maintenance and accountability rather than model capability. A better model still creates an unmaintained variant if you apply its output outside the upstream project.

Why this sits next to the AI infrastructure writing

I've been writing a series on running inference workloads, and this looks like a different subject. It isn't entirely.

The common thread is that AI is being adopted faster than the operational practices around it, and the gap shows up in whatever your weakest process already was. If your patching discipline was poor, automated patching makes it poor at machine speed. Same pattern as the infrastructure side, where enthusiasm arrives well ahead of anybody measuring what's actually happening.

The useful response in both cases is the same. Measure the thing first.

Sources

Broadcom Strengthens Spring Security and Adds Coverage of Java, Python and Node.js Ecosystems with TrueSource: the 26 percent figure, the 12 billion tokens, the 5,000-plus Java libraries, and the portfolio structure.
Broadcom consolidates open-source support under TrueSource, Techzine: the dependency version-pinning detail and Bitnami Secure Images.
Broadcom Launches TrueSource Service to Secure Spring Framework, DevOps.com: the human validation position.
SLSA build levels: what Level 3 actually requires.
1Password's Off-by-1 Labs is the primary source for the patch testing. Worth going there directly if the number is going into a decision.

Views expressed here are my own.