coding agentsupdated

Find generously, refute coldly

An adversarial bug bounty with a fleet of agents: finders that over-produce, skeptics that start from disbelief, and the waste worth cutting.

An agent reviewing code on its own produces findings that are plausible and wrong. Claiming a bug costs it nothing; checking the claim costs a human an hour. Ask it to be careful and it finds less. Ask it to find more and you drown in false positives. Recall and precision pull against each other inside a single context.

So split them. Give recall to one set of agents and precision to another, and put a wall between them. That’s the idea of an adversarial bug bounty, and it works better than I expected. Here is the pattern, what it costs, and where the money is wasted.

Find generously, refute coldly

Two phases, with a barrier in between.

Finders are told to over-produce. Their job is recall, not precision. Each one covers a single dimension of review (correctness, concurrency, types and units, lifecycle, security, error handling) and is blind to the others.

A deduplication step sits on the barrier. Finders with different lenses regularly land on the same problem under different names, so candidates that point at the same mechanism are merged into one claim before anyone tries to refute them.

Skeptics get almost nothing: the claim, a file:line pointer and a one-sentence failure scenario. Never the finder’s reasoning. They must re-derive the failure path by hand on the real code, start from the assumption that the claim is wrong, and, when they reject it, say why with one of a few classes: unreachable, by design, handled elsewhere, not a bug. Two skeptics per claim: two confirmations and it survives, two refutations and it’s gone. When they disagree, which happens on one or two claims out of ten, a third skeptic breaks the tie with the two sets of evidence, not the reasoning.

Three rules hold the whole thing up:

  1. A fresh context for every skeptic. Share the finder’s reasoning and refutation turns into confirmation bias.
  2. No verdict without evidence. Every verdict cites code; a structured output schema makes it non-negotiable.
  3. Survivors are inferred, not confirmed. The fleet pre-filters. Only something outside it (a test that reproduces the bug, a fix, an incident) confirms.

What it costs

To calibrate it, I ran an early version once at scale, before the deduplication and the tie-breaker existed: six finders, two skeptics per candidate, 190 agents in 36 minutes. Of 92 candidates, 48 % were refuted, 14 % ended split and 38 % survived, about twenty unique bugs once duplicates were merged by hand. The bill was about $360, roughly $15 per surviving bug, and it would have been $1,660 without prompt caching.

A bag of coins flying away on a pair of feathered wings, dropping a few coins as it goes.

A refutation rate around half is healthy, not waste: precision comes from refutation, not from timid finders. If your skeptics rarely refute, your finders aren’t trying hard enough. And the survivors were real, mostly the kind you don’t find by reading: interactions between settings, lifecycle edge cases, data silently lost or counted twice.

The cost was very lopsided: refutation took 77 % of the budget. Fresh contexts are expensive by construction: every skeptic reads the relevant code from scratch. That run also showed where the waste was, and that’s where the two later steps came from:

The same idea, priced differently

A colleague built his own version of the pattern, and comparing the two taught me as much as the run did. His trade-offs were the opposite of mine on almost every point.

We agreed on the part that isn’t negotiable: the reviewers must be separate agents, isolated from the context that wrote the code. And we disagreed, usefully, on the rest. With a single skeptic per packet, a skeptic that is wrong is wrong alone; that’s where I think you become blind. With two per claim, you pay for it. The idea we left on the table sits in between: hand each skeptic’s evidence to a tiny, nearly free model that scores how solid it is, and only launch a second skeptic when the score is low. Put a near-free model on the output of the things that cost a lot.

What I would still change

Estimates, not measurements:

When to use it, and when not

Use it when the blast radius is high, the codebase is unfamiliar or long-lived, or a review backlog makes pre-filtered, evidence-backed findings worth about $15 each. Don’t use it on a small diff (one adversarial reviewer after implementation does the job for a fraction of the tokens), on a prototype, or anywhere a false positive costs minutes rather than hours.

And it’s always a human who starts it, never an agent. Burning seven-figure token counts is a budget decision, and announcing the expected cost before launch is part of the deal.

What I keep from it goes beyond bug hunting: redundancy is what you pay for, and waste is what you cut. Blind finders, evidence-only handoffs and a default of disbelief make the pattern work. How much redundancy to buy depends on who reads the output: a developer deciding on a table can afford one skeptic; a loop that ships overnight can’t.