Evaluation evidence · 0.1.0-rc.7
YYLO Benchmark methodology reference
How YYLO Benchmark turns coding-agent runs into reusable evaluation evidence: validated cases, immutable plans, isolated attempts, governed grading, recovery, regrade, and longitudinal reports, with the exact command surface and the guarantee each stage carries.
Scope and evidence date
This page is the released operational reference for the YYLO Benchmark methodology. Every command, schema version, and semantic statement on this page was verified against the committed source and README of package version 0.1.0-rc.7 on 2026-08-29; nothing here is roadmap direction. The package describes its own reach in one sentence: “The private, independent package owns case validation, isolated snapshots, shadow boards, execution reconciliation, recovery, reports, and bounded investigations.”
Two reading companions share the territory without overlapping this reference. The operator walkthrough teaches the full first run case by case, and the benchmark documentation page carries installation and the product summary at YYLO Benchmark; this page stays the compact, citable statement of what each stage means. Reach it from the documentation hub or the benchmark docs page at any time.
Two command launchers exist. The standalone yylo-benchmark CLI is the canonical surface, and the README binds the delegate exactly: “yy benchmark ... is a transparent delegate to an independently installed compatible yylo-benchmark executable. The standalone CLI remains canonical.” Commands below use the standalone form.
Methodology lifecycle
One evaluation moves through seven stages, and every stage is either read-only or evidence-retaining — none is a silent mutation of the repository under test:
The read-only posture starts at the top of that ladder: “Planning and every --dry-run are read-only and report dispatch_count: 0.” Nothing in the lifecycle dispatches a candidate until a live run executes an accepted plan.
Cases
A benchmark case is an ordinary ledger task that has explicitly opted in: it carries the benchmark-case tag and a valid fields.benchmark reference holding the repository id, the base commit, a case version, a category, a grader profile name, and optional project wiki paths. The task’s body serves as the candidate prompt. Validation is a separate, explicit step that derives the typed case identity — the task hash, the prompt hash, and one combined input hash — without touching the task:
An ineligible task fails here with the exact missing field named, before any planning exists to be invalidated. Case authoring — what makes a case worth measuring — is taught by the walkthrough.
Plans
Planning freezes the attempt matrix into an immutable, content-addressed plan: the resolved case revision, exact model identities, attempt count, wiki hashes, tool policy, budget object, and package versions all enter the plan hash. Two plan families share the command:
Task-case planning attaches an aggregate spend ceiling — the default is twenty US dollars, replaceable with a positive --max-usd value — which the planner divides deterministically across the exact model/attempt matrix, and a live run additionally requires a typed authorization grant whose plan, models, currency, expiry, and both ceilings match the plan exactly. Workflow plans take a hash-bound policy sidecar instead and carry no spend grant; their cost is best-effort evidence. Selectors like :mini resolve through the configuration to one exact provider/model identity at plan time, and only the resolved identity is ever dispatched. What freezing buys, and why a mutable plan poisons comparisons, is the immutable plan guide’s subject.
Attempts
Each planned attempt works inside its own snapshot: the source exported at the exact base commit the case reference pins, into a new repository whose Git objects are isolated from yours, with harness-owned paths left out, a snapshot-local home directory, and a sanitized environment. Execution is one typed invocation, yy pi --execution-envelope, and the resulting juno_execution_envelope.v1 object is the only execution-evidence channel: the envelope’s provider and model fields are independent observations that must normalize to the exact planned identity, and nothing a candidate writes about its own identity, resolution, or spend is accepted as evidence. A child that exits with a known status but no valid envelope is retained as a redacted harness-failure terminal and is never redispatched.
Each attempt terminates in precisely one of nine terminal classes — resolved, model_failure, safety_failure, harness_failure, environment_failure, grader_failure, invalid_case, timeout, or cancelled — and cost travels with a completeness state rather than a bare number. The README compresses the money rule into one line: “Cost retains complete, partial, unavailable, not-applicable, and genuine-zero semantics.” Unavailable stays null; it is never laundered into zero. The full vocabulary of honest cost and failure evidence has its own reference treatment, and what repeated attempts buy — and do not — is the variance guide’s subject.
Grading
Each case’s grader_profile names a configured entry bound to an executable SHA-256, a grader ID, and a version, so the judge is fixed bytes rather than a person. A graded attempt retains four artifacts — the grader input, the grader output, the typed result, and an integrity-linked receipt — and a missing, failed, removed, or tampered grader fails closed rather than passing quietly. Resolution is a conjunction: the candidate process must have succeeded and the grader boolean must be true; a candidate that ran cleanly but failed the rubric receives the grader-failure terminal. The typed per-attempt record that carries the verdict looks like this:
Graders improve, and the methodology answers with generations instead of overwrites. regrade consumes retained attempt, candidate, and patch evidence to append a new grading generation without accepting or rerunning a candidate runner — the regrade guide owns that workflow. Tracked-workflow experiments add a governed, blinded judge: candidate identity never reaches the judged payload, and rejudge re-runs governed judging over retained candidate truth with zero candidate dispatch; the blinding protocol is documented in full there. Where the package could invent convenience, it refuses: “The CLI intentionally fails actionably instead of inventing an unreviewed launcher or judge.”
Recovery
Interruption is a methodology stage, not an exception. In the task-case path, recovery is built into run: a rerun reconciles retained artifacts and terminal markers first, converts retained receipt evidence into the terminal it already proves, closes a dispatched-but-unterminal attempt as an indeterminate harness failure, and never redispatches an attempt whose dispatch marker exists. In the workflow path — where “Live execution uses one separately reviewed, hash-pinned JavaScript boundary module.” — recovery is an explicit command that asks the boundary’s private dispatch journal what actually happened before anything resumes:
Both safe reconciliation answers — a step proven not to have been dispatched, and a step the boundary certifies safely resumable — pass through the step’s own retry-safe policy gate before anything runs again; terminal evidence is validated against the exact dispatch identity before acceptance; ambiguity, a step whose policy is not retry-safe, and exhausted recovery budgets stop for manual recovery instead of guessing. The decision tree — resume, regrade, or discard — has its own recovery guide.
Reports
Reporting reads retained evidence only and writes derived layers, never source manifests:
The longitudinal report groups attempts into cohorts under a compatibility key — case, snapshot, and prompt hashes, wiki, tool-policy, and budget hashes, agent identity, package and YYLO versions, session topology, and the grading layer — so only genuinely comparable attempts pool, and every pair of cohorts that cannot pool is listed as an incompatibility rather than averaged away. Naming a grader version pins the report to the attempts that version actually graded; the default layer reads every retained attempt’s current verdict. Each report version is retained as its own content-addressed derived layer beside the experiments it summarizes, which is what makes a regraded disagreement visible beside the verdict it displaced. What cohort statistics mean for a decision — intervals, denominators, recommendation discipline — belongs to the variance guide and the multi-system comparison guide.
Verification
Every claim above is checkable after the fact from retained bytes, and two commands exist for exactly that. doctor re-verifies an experiment end to end; the README draws its dual reach precisely: “doctor accepts either a Kanban experiment task ID (verified through the public Ledger contract) or a workflow-<64-hex> registry experiment identity (verified entirely from retained private-registry evidence, including retained harness-failure terminals and ambiguous dispatch intents, without any Ledger read).” investigate answers one bounded question against the same retained evidence without dispatching a candidate:
The package holds its own release bar to the same standard: release-readiness emits a deterministic offline receipt binding the clean commit, both package identities, coverage derived from executed case results, and six executed synthetic leakage checks. It is an offline gate — it excludes live execution and paid dispatch, and it grants no spend authority.
Citation and maintenance
Cite this reference by its stable canonical URL together with the evidence date and the package version in its header: the methodology statements on this page are bound to @yylo/benchmark 0.1.0-rc.7 as of 2026-08-29. When a released package changes a stage’s semantics, this page is regenerated against the new source and re-dated in the same edit; the command surface and schema versions are transcribed, never paraphrased from memory. For worked reasoning rather than reference — why each guarantee exists and how to operate it — cite the linked guide that owns it.