2026-08-28 · Updated 2026-08-28 · 10 min read
Compare multiple coding agents on one task
Run a fair bake-off between several coding agents on one identical engineering task — one frozen plan, equal attempts and ceilings, observed identity, hash-pinned judging, and per-agent retained evidence — then read the standings without crowning a tie.
By Juno AI INC · benchmark · evaluation · comparison
The bake-off starts innocently. One real bug, three agents, one afternoon: each gets the issue text, each works in its own checkout, and by evening one of them has a working patch while the other two left half-finished refactors behind. The verdict writes itself — until someone asks the three questions that dissolve it. Did all three receive the identical task, or three paraphrases of it? Did they start from the same code, with the same budget and the same number of chances? And who judged the result — a check that cannot be charmed, or a person who already had a favorite? Most agent bake-offs are decided before any agent runs, by the design of the comparison rather than the quality of the contestants. This page is about that design: how to put several coding agents on one identical engineering task so that whatever differs between their results actually belongs to the agents.
The machinery throughout is YYLO Benchmark's. The full operator path — selecting a repository, writing the case into the ledger, executing and grading attempts — is walked end to end by the operator guide; this page takes that loop as given and zooms in on what changes when several contestants share one task. Quotations below are verbatim from the benchmark package as committed on this branch — its README and its managed guidance pages — and were matched character-for-character on 2026-08-28; commands and flag behavior come from the same committed CLI, not from memory or marketing.
One frozen task for every contestant
Being identical has to be a mechanical property, not a promise. In this package a case is a set of hashed inputs: the task body is the prompt, bound to its hash when the case is linted; the baseline is an exact exported commit; any candidate-visible guidance pages enter by their own hashes; the tool policy and budget each contribute theirs. One plan then freezes the whole matrix — which contestants, how many attempts each, what everything may cost — and the plan names itself by the hash of everything inside it, so the comparison you approved and the comparison that runs are provably the same object:
The contestant list mixes short aliases and one exact identity, which is the documented shape: alias selectors resolve through the config's alias map, exact identities pass through, and planning records both the selector and the exact identity it resolved to — an alias that quietly changed meaning between planning and running fails closed instead of dispatching a different model under the same name.
Nothing about the case may drift after that freeze, and the package says so in its lifecycle guidance: "At run time, revalidate every frozen input. Drift fails closed and requires a new plan; it must not silently alter the accepted experiment." A source task revised behind your back, a rebuilt baseline whose hash disagrees with the plan's, a guidance page edited after the freeze — every such drift stops the run before dispatch. The fairness payoff is immediate: no contestant can inherit a patched prompt, a warmer baseline, or a friendlier environment. Every dispatch in the matrix receives the same treatment, in the package's own words: "Each attempt receives the same declared case inputs and a fresh isolated source repository." The candidate source, as the overview page puts it, "is exported from an exact Git tree into a fresh repository with isolated Git objects" — so a contestant working inside its snapshot cannot have touched what the next contestant starts from.
Equal attempts, equal ceilings
Two more symmetries are easy to break by accident and expensive to discover late. The first is the number of chances: the matrix gives every contestant the same attempt count by construction, and that count is the denominator of everything you will read later. One attempt per agent is a single coin flip per contestant; five attempts make a rate legible and give repeat behavior a chance to show. How the repeats spread — and what that spread does to conclusions drawn from small matrices — has its own planned page in this program; the design point that belongs here is only that the count must be equal and fixed before results exist.
The second symmetry is money. The plan carries one aggregate ceiling for the whole bake-off, and the README states how it spreads: "planning divides it deterministically across the exact model/attempt matrix." Fifteen dollars across three contestants at five attempts each is fifteen dispatches at a dollar per attempt — the same per-attempt ceiling for every contestant, fixed inside the plan's hash before the first dispatch. An attempt whose bill exceeds its ceiling fails; it is not forgiven. Without this rule a bake-off silently measures who spent more; with it, cost becomes one of the dimensions you read instead of a hidden advantage you awarded.
Vary one axis and name it
A comparison is only as clean as the one thing it varies. The guidance page that owns this distinction is blunt: "Separate model-only comparisons, where the surrounding system is fixed, from whole-agent-system comparisons." Decide which kind you are running and say so in the write-up, because the two answer different questions — is this model better inside our fixed harness, or is that whole harness-plus-model system better at this work? In the task-case path the harness is fixed by design: the attempt contract pins the agent, every dispatch is one typed invocation of the same CLI, and what varies is the model identity. If your evidence spans whole agent systems instead, each system's attempts stay their own in the reporting layer — it never merges by label.
It never merges by label because identity is observed, not declared. The README's sentence on the execution envelope covers both halves of the trust problem: "Provider and model are separately observed and must normalize to the exact planned identity." A candidate's own claims about what it is never enter the record; the envelope the harness emits does, and a dispatch whose observed identity does not match the plan's is not quietly re-labeled. When the report later groups attempts into contestants, it groups by the full observed system identity — agent, provider, model, and both package versions — so a mid-bake-off model swap cannot average into someone's record; it shows up as a different system with its own sample.
A judge that cannot play favorites
The verdict has to come from something incapable of knowing or caring who produced the work. Prefer judges whose truth is mechanical: a test that fails for one known reason, a type error the change must erase, a build that has to come up green. The grader profile binds that judge to bytes — an executable pinned by SHA-256, verified as a regular file, its bytes hashed against the profile both before it spawns and again once it has answered — and its entire output contract is one boolean. Stdin arrives as one typed record — the attempt identity with its hashes, the profile's name, whether the candidate process itself succeeded, the patch hash, the terminal-evidence hash — and the boolean back is all the pipeline reads. The resolved verdict is then a conjunction of two facts: a contestant whose process succeeded and whose grader passed is resolved; a contestant that succeeded against a refusing grader is recorded as a grader failure, not a rescue; a contestant that failed keeps the failure it earned. The README's one-liner is the whole contract: "A missing, failed, removed, or tampered grader fails closed." Mid-bake-off grader edits are not a tooling convenience; they are tampering, and the machinery treats them as such.
When the judge is itself a model rather than a test suite, blinding has to be structural. The package's workflow lifecycle — used for benchmarking multi-step workflow runs under a governed policy — enforces exactly that at its boundary: "Judge operations receive only the blinded request; they never receive candidate credentials or unblinded identity." The judge scores a redacted transcript and artifact set, not a named contestant, and rejudging is fenced the same way — it "uses retained blinded candidate truth and never accepts a candidate dispatcher", so a sharper judge can revisit history without any candidate being re-run or consulted. How to build and govern such graders in depth is another planned page in this program; the design rule that travels is smaller: the judge's inputs and identity must be pinned before results exist, exactly like the contestants'.
Per-contestant evidence that survives the afternoon
Every attempt of the matrix leaves its own retained trail in the private registry: the attempt contract, a durable dispatch marker, captured stdout and stderr, the patch taken from exact filesystem bytes rather than the candidate's Git story, terminal evidence, and the graded result chain with its integrity-linked receipt. Attempt identities are hash-derived from the plan, the model, and the ordinal — not from wall-clock time — so an attempt is addressable forever after the process that produced it is gone. Two properties of that trail matter most to a bake-off. It is append-only: "Past attempt evidence is append-only. Adding a model creates new attempts", and the comparison guidance adds that new-model experiments "append evidence without rerunning compatible historical attempts" — a fourth contestant next week extends the bake-off without re-billing the first three. And it does not rewrite itself under failure: "A terminal attempt is never rerun in place", so a crashed run cannot silently repeat a contestant's worst throw; recovery verifies retained artifacts before anything resumes, and what cannot be proven safe to resume becomes new evidence instead of a duplicate paid run. The doctor command re-verifies a whole experiment end to end, checking every retained reference against the bytes it names — worth running before you circulate any standings.
Reading the standings without a crown
The report refuses to do the one thing a bake-off audience most wants: collapse everything into a ranking. Attempts group into cohorts only when every input that defines comparability hashes the same, and when the report declines a merge it explains itself — "Reports identify included and excluded evidence and explain compatibility decisions." Within a cohort, every system is reported with its resolved rate, a Wilson 95 percent interval around that rate, and the total and valid sample sizes behind it, alongside the dimensions the guidance refuses to fold together: "Keep cost, elapsed runtime, invalid-run rate, and repeat consistency as separate dimensions rather than collapsing them into a universal score." Two readings follow from that shape. When two contestants' intervals overlap, the bake-off did not separate them — report that, rather than ordering a winner out of a tie. And when the runner misbehaved, say so in its own column: "Infrastructure-invalid attempts remain visible in invalid-run statistics but do not become model failures", so a flaky environment cannot sink a contestant's rate any more than a heroic one can prop it up. Cost gets the same honesty — cost per successful resolution is computed only where a system's attempts all carry complete cost and it has at least one success, and the guidance's rule keeps it honest company: "Report cost per successful resolution alongside raw cost and reliability so cheap unresolved work is not rewarded."
The final discipline is ordering. Fix the reliability bar and the case category before looking at any standings, because a threshold chosen after the ranking is a rationalization. Then the decision rule the package ships applies: "Among systems meeting that threshold, recommend the least expensive supported option and an escalation option where evidence warrants one." And keep the recommendation's status clear — in the guidance's own words, "Recommendations are advisory: they do not mutate production routing or deploy candidate patches." A bake-off earns its conclusion by being reproducible, not by being obeyed.
Where this page sits
What this page owns is narrow: the design that makes several agents on one task a comparison instead of an anecdote. The full operator loop — case selection through grading, command by command — belongs to the operator guide; what each kind of public scoreboard measures belongs to the instrument reference, and why standings from public boards carry over only weakly into one repository belongs to the transfer analysis. Four deeper subjects touched here — repeated-attempt variance, the immutable evaluation plan, blinded grading for model judges, and the benchmark methodology itself — each has its own planned page in this program. The closing panel below installs the benchmark; the bake-off it scores is yours to run.