2026-08-28 · Updated 2026-08-28 · 12 min read
Benchmark coding agents on a real repository
Run a fair coding-agent benchmark on your own repository — select a case worth measuring, author it as a ledger task, freeze the attempt matrix and its spend ceiling, execute isolated attempts, and grade them with hash-pinned graders.
By Juno AI INC · benchmark · evaluation
Every agent choice eventually meets the question no leaderboard can answer: will this model, inside this harness, do useful work in *this* repository, on *this* kind of change, more than once, at a cost you would accept? A shared benchmark scores a shared task; a demo scores a curated one. Neither has ever touched your checkout service's retry paths or your migration suite. YYLO Benchmark exists for the private half of the question — its README opens by calling it "the longitudinal evaluation and immutable-evidence system for agent runs" — and this page is the operator's path through it: select a repository and a case worth measuring, author that case as a first-class task, freeze the attempt matrix and its spend ceiling before anything dispatches, then execute and grade under isolation until a defensible answer exists.
Four properties make such an answer worth having. The case is a real engineering item from a commit you can rebuild, not a toy. Every attempt starts from the same exported baseline, so differences belong to the candidate, not to drift. The judge is an executable pinned by hash, not a person skimming diffs at midnight. And the whole run leaves receipts — hashes, patches, terminal evidence, cost — that outlive the process that produced them. Break any one of those and the result is an anecdote wearing a benchmark's clothes.
The ground around this page stays with its owners. Where evaluation sits among models, harnesses, and task truth is a layer question, mapped end to end by the open-source stack survey; the ledger storing the case itself gets its introduction from the task-truth guide; and the operational reference for the package stays at YYLO Benchmark documentation. What follows was checked against the benchmark package's committed source and README and against the released Ledger CLI on 2026-08-28; commands are the standalone CLI's real surface, reachable identically through yy benchmark as a transparent delegate.
Select a repository and a case worth measuring
The unit of measurement is one engineering case: a real item of work with one primary resolved outcome. The package's own authoring guidance is blunt about what qualifies — "an exact source baseline, a clear resolved condition, representative engineering scope, and checks that can distinguish correct behavior from plausible text" — and equally blunt about what disqualifies: a case whose correctness cannot be judged, or whose baseline cannot be rebuilt, is not a case. A flaky test suite, a task whose fix is already on a nearby branch, or a prompt so vague that any output plausibly passes are all measurement errors waiting to be published as findings.
Selection is mostly subtraction. Prefer changes with deterministic judges — a failing test that should pass, a lint or type error that should vanish, a build that should succeed — because grading then rests on evidence rather than opinion. Pick a base commit you can check out and rebuild today, and record it by its full object ID; the case metadata will not accept an abbreviation. Give the case a category (backend, frontend, tooling, whatever vocabulary you keep): categories are the vocabulary you compare within — the comparison guidance tells you to fix the reliability threshold and category before looking at any winner, and the report checklist puts category and scope among the statements a report owes you — while the report itself keeps runs apart mechanically, by hashing compatibility inputs, so incomparable kinds are never averaged by accident. Keep scope representative of the work you actually delegate to agents: a benchmark of one-line typo fixes tells you nothing about how a model behaves inside a two-hundred-file refactor.
Two more selections happen at authoring time, and both change the case's identity. Candidate-visible knowledge — the interfaces, conventions, or constraints an agent should be trusted to know — lives under .juno_task/wiki/yylo-benchmark/project/, and you select per-case wiki paths into the case metadata; the exact page hashes become case inputs, so editing a page after planning creates a different case rather than silently shifting one. And a leakage review precedes acceptance: the exported candidate tree must exclude reference patches, future Git objects, hidden checks, credentials, and package artifacts, and candidate-visible guidance may explain stable interfaces but must not reveal the solution. A case that leaks its own answer measures reading comprehension, not engineering.
Author the case as a ledger task
Initialize once per repository. yylo-benchmark init writes a minimal yylo-benchmark.config.json (repository identity, the Kanban binding, an empty alias map — it refuses to overwrite an existing file) and installs five checksum-managed guidance pages under .juno_task/wiki/yylo-benchmark/: overview, case authoring, experiment lifecycle, model comparison, and telemetry and retention. Pages you add under project/ are yours alone and are never touched by package updates.
The config is where two identities get pinned. model_aliases map short selectors like :mini to exact provider/model identities — planning hashes both the selector and the identity it resolved to, and execution dispatches only the exact identity, so an alias whose meaning drifts fails closed instead of quietly rerunning a different model. grader_profiles name the judges; their shape matters enough to wait for the grading section below.
The case itself is an ordinary ledger task carrying extraordinary metadata. The README's contract is one sentence: "A case must carry the benchmark-case tag and valid fields.benchmark metadata" — and its task body is the candidate prompt. Concretely, with the released Ledger CLI:
Every field is load-bearing. base_commit is the full 40- or 64-hex object ID the attempt will start from. case_version increments when the case's intended meaning changes. grader_profile must name an entry in your config. wiki_paths are normalized repository-relative paths beneath yylo-benchmark/project/. Then validate before planning anything:
Lint is read-only: it fetches the task through the public Ledger contract, refuses anything missing the tag or carrying malformed fields, and prints the resulting eval case — task, prompt, and input hashes bound together. The prompt you just wrote is now a hashed artifact, not a string in a shell history.
Freeze the attempt matrix and its ceiling before anything runs
Planning answers three questions in one immutable object: which models, how many attempts each, and how much this may cost. It answers them read-only — no ledger or registry mutation:
The emitted plan is content-addressed — its identifier is the hash of its own contents — and those contents include the case revision and its hashes, the snapshot identity, the selected wiki hashes, the exact model identities, the attempt count, the package and YYLO versions, and the spend limits. Legacy task-case plans "bind a USD 20 aggregate ceiling by default"; --max-usd chooses another positive one, and planning divides it deterministically across the models-times-attempts matrix into per-attempt ceilings. Review the plan as the artifact it is: after this point, drift is a failure, not an edit. At run time every frozen input is revalidated — a rebuilt snapshot whose identity differs from the plan's, a source task revision that moved, a wiki page that changed after planning — and each mismatch stops the run before dispatch.
Attempts deserve the same deliberation as models. One attempt per model measures a coin flip; the managed guidance is explicit that the primary measure "is whether one isolated attempt resolves the engineering case," and a rate needs a denominator. Three attempts per model on a first serious case is a defensible floor; raising it is cheaper than convincing yourself later that a 1-of-1 was signal. How repeated attempts distribute, and what variance across them does to your conclusions, is a statistical subject with its own treatment — the mechanics here are the part the CLI owns.
Authorize the spend against that exact plan
A live task-case run does not dispatch on good intentions. It requires a juno_benchmark_task_authorization.v1 grant whose "plan, models, currency, expiry, aggregate ceiling, and per-attempt ceiling exactly match the immutable plan" — the grant names the plan by its hash, so it authorizes this plan and nothing else:
Hand it to the run with --authorization grant.json. The worst-case reservation is recorded before provider dispatch, and an expired or mismatched grant is refused outright. This ceremony earns its place: benchmark spend is the easiest budget in engineering to multiply by accident — models times attempts times reruns — and the ceiling here is not a note in a ticket but bytes inside the plan's hash. Nothing in this loop can "retry" its way past a number the plan committed to before the first dispatch.
Execute every attempt from the same frozen baseline
Retained evidence lives in a private registry, and the commands that retain or read it refuse to run until you name one, so export the variable and let attempt workspaces default beneath it:
Each attempt of the matrix gets the same declared inputs and a fresh isolated source repository built from the case's exact base commit — isolated Git objects on a benchmark-baseline branch, with harness-owned paths (.juno_task and the benchmark package's dependency and build directories) excluded from the export. Inside that snapshot the candidate finds a shadow kanban.sh wired to a shadow board, so an agent that wants task truth gets it — while its writes land in the attempt workspace, never in your canonical ledger or controller. Candidate execution receives a snapshot-local HOME/XDG and "a sanitized environment with credential and canonical-controller routing variables removed"; an alternate-routing flag in the shadow wrapper is simply refused.
Dispatch itself is one typed invocation: yy pi --execution-envelope --model <exact identity> <prompt>, run inside the snapshot with the YYLO executable's version probed against the plan first. Execution "consumes only the public juno_execution_envelope.v1" — and because provider and model are separately observed from that envelope, "candidate output cannot declare its own identity or resolution." Output is bounded (eight mebibytes), and the patch is captured from exact filesystem bytes through a private index that never trusts the candidate's Git metadata, ignore rules, or clean filters. Every attempt ends in exactly one terminal class: resolved, model_failure, safety_failure, harness_failure, environment_failure, grader_failure, invalid_case, timeout, or cancelled. The lifecycle guidance's one-liner is the discipline in full: "A failed environment is not a model failure." Cost evidence retains five honest semantics — complete, partial, unavailable, not-applicable, and genuine zero — and unavailable is retained as null, never laundered into a zero.
State the isolation boundary as the product states it: "V1 provides isolated Git objects in a fresh repository but truthfully treats the same-user host filesystem as trusted; it does not claim container or hostile-host isolation." If your threat model needs containers, add them outside this loop rather than trusting a claim the tool refuses to make. A canonical run records one related experiment task on the ledger — that is the experiment task the doctor command above verifies — and --no-record exists only for fixture work, quarantined behind --non-canonical-scope fixture|local. Evidence-grade benchmarking records.
Grade with an executable you can hash
A grader profile binds the judge the same way the plan binds the candidate — to bytes:
At grading time the executable must be a regular, non-symlink file whose bytes hash to the configured value — checked before it runs and re-checked after — and it answers one typed question: the attempt contract, the profile name, whether the candidate itself succeeded, the patch hash, and the terminal-evidence hash arrive on stdin as JSON, and the grader must print JSON containing a boolean passed. Resolution is a conjunction: a candidate that succeeded and passed the grader is resolved; a candidate that succeeded while the grader refused is grader_failure; a candidate that failed keeps its failure class. Each grading appends a self-authenticating receipt — input, output, and result hashes chained to an integrity hash — and any grader that goes missing, errors, is swapped, or comes up tampered stops the pipeline instead of inventing a verdict. If the checks themselves feel like the unfamiliar part, the discipline of letting tests, not vibes, adjudicate an agent's work is the same one hardening a Ralph loop with tests teaches on the execution side.
Graders improve; candidates should not have to pay for that. yylo-benchmark regrade --plan checkout-plan.json consumes the retained attempt, candidate, and patch evidence and appends a new grading generation without accepting or rerunning a candidate — a sharper judge revisits history without reopening the bill.
Read the report, then keep the evidence honest
Two commands close the loop. doctor on the experiment task re-verifies the retained registry end to end and cross-checks the ledger's evidence index against the bytes it claims to point at — stale indexes and missing artifacts stop here. report returns to the case:
The longitudinal report groups attempts into cohorts by their compatibility inputs — case, snapshot, prompt, wiki, tool policy, budget, agent, package and YYLO versions, session topology, and the grader version — and refuses to merge across incompatibilities, listing them explicitly instead. Within a cohort, each system gets sample size and valid sample size, a resolved rate carrying a Wilson 95% interval, an invalid-run rate, repeated-run consistency, a runtime distribution, cost coverage split by completeness with cost-per-success computed only when cost is complete and successes exist, and terminal-class counts down to the attempt IDs. Nothing collapses into one score: the overview's whole framing is "which model or agent system is the least expensive option that resolves a category of real engineering work reliably enough," which forces the threshold question to be answered before the winner is inspected, not after.
Read it with the same honesty the formats enforce. Invalid attempts stay visible in invalid-run statistics without becoming model failures. Small samples support cautious recommendations, not superiority claims — the interval is in the report precisely so you cannot round it away. And the evidence is append-only: adding a model later adds attempts, new graders add generations, reports are derived layers over manifests that never change. Six months from now, the answer to "why do we run :mini on backend cases" should be a hash you can still verify, not a memory of a good week.
That is the entire loop: a case worth measuring, authored as durable task truth; a plan that froze models, attempts, and ceiling into one hash; an authorization that matched it exactly; attempts that all started from the same exported baseline; a judge pinned to bytes; and a report that separates resolution, cost, runtime, and failure instead of averaging them into a slogan. The next-step panel below installs the benchmark and points at the operational reference; your repository supplies the rest.