How repeated attempts reveal coding-agent variance

Run the same agent at the same task twice and there is a fair chance of watching a paradox: the first attempt resolves the failing test, the second attempt on identical code leaves the branch in pieces. Evaluators meet this moment constantly, and most settle it in one of two wrong ways — dismiss the miss as flakiness, or distrust the hit as luck — both of which discard the one thing the flip actually carries. The outcome of an attempt on a fixed task behaves like a draw, not a setting; a single draw decides nothing, and a pair of draws that disagree is not a contradiction but the first honest data point about spread. This page is about listening to that data: how many attempts to buy, what a rate with an interval proves at small counts, which failures belong to the runner rather than the agent, and how the answers change which agent you pick.

Two borders before the statistics. The machinery that produces the repeated evidence is YYLO Benchmark's; the command-by-command path — choosing the case, freezing the matrix, dispatching and grading — is the operator walkthrough, and this page assumes that loop rather than repeating it. And the arithmetic below is original to this site: computed with the evaluation package's committed formulas, stated with its assumptions, and carrying the evidence date 2026-08-28. Quotations were verified word-for-word against the committed package text on the same date; every behavioral claim was traced into the package's source on this branch, not into marketing.

One attempt answers almost nothing

Start from the position the package itself takes. Its comparison guidance opens with the unit of measurement: "The primary measure is whether one isolated attempt resolves the engineering case." Reliability, in that same guidance, is then built from repeated valid attempts rather than from a single verdict — the number an evaluator wants is a rate, and rates live on denominators.

How fast the denominator earns its keep is the part intuition handles badly. Take the simplest honest model of a repeated agent run: independent attempts, each with the same fixed underlying probability of resolving. Nothing about real agents is that clean — later sections return to why — but even this flattering model refuses to hand you a quick answer. An agent whose long-run rate is genuinely 60 percent produces, over five attempts, a series with a minority of successes about 32 percent of the time — nearly one series in three reads worse than its true self. A clean five-for-five sweep happens about 8 percent of the time, so a perfect-looking matrix is itself weak evidence. And an agent that truly resolves 90 percent of attempts still drops at least one of five in about 41 percent of series — roughly two in five. The pass-then-fail surprise that opens this page is not an anomaly to explain away; it is the default behavior of any system short of perfection, and the only defect it proves is an underpowered evaluation.

The interval makes the same point from the other side. One success in one attempt is a 100 percent observed rate, and its 95 percent interval — computed with the same Wilson formula the package's report layer uses — runs from 21 percent to 100 percent. The observation is compatible with an agent that fails three of every five runs. Nothing downstream of an n-of-1 verdict survives contact with that range.

What the interval does to a small matrix

Every rate the report layer publishes arrives inside a Wilson 95 percent interval, and at the attempt counts real evaluations can afford, that interval dominates the point estimate. Three resolved of five (60 percent observed) spans roughly 23 to 88 percent. Six of ten, the same observed rate, spans 31 to 83. Thirteen of twenty spans 43 to 82. Doubling the sample buys maybe a dozen points of width — precision at these counts is expensive, and pretending otherwise is where bad agent decisions are born.

The starkest case is the one evaluators most want to close. Set one system that resolved eight of ten against another that resolved four of ten — the first agent's observed rate is double the second's, a result that would settle any hallway argument. The intervals are roughly 49 to 94 against 17 to 69: they overlap. Under the math a doubling this large, at ten attempts per side, cannot exclude the possibility that both systems share one underlying rate. The package's report checklist answers with disclosure rather than ceremony — "Resolved counts and reliability with sample size and uncertainty." — and its comparison guidance holds the same line off the checklist: small repeated samples license cautious recommendations, never a claim that one agent is universally the better one.

Two habits follow. Read the interval, not the ranking: when spans overlap, the honest sentence is that the evaluation did not separate the systems, and that sentence is a finding, not a failure. And buy attempts where the decision is close: the width arithmetic above is the budget argument — going from five to twenty attempts per contestant narrows the span by about two-fifths, which is often cheaper than living with the wrong pick. How a full standings table is read — tie handling, cost columns, thresholds before winners — is the bake-off design page's job; what an interval means, and how little it certifies at small n, is this page's.

Whose noise is it? Type the failure

Run-to-run spread has two sources that must never be averaged together: the candidate's own behavior and the world around it. A runner that crashed, an environment that lost a credential, a grader that refused — each of those can look identical to a model failure in a raw pass/fail log while meaning something entirely different for a routing decision. The package's answer is to type every terminal outcome, and its lifecycle guidance names the full vocabulary: "Record resolved or unresolved separately from typed model, safety, harness, environment, grader, timeout, and cancellation classifications." The overview page's operator sequence carries the same instruction as a step: "Classify infrastructure-invalid outcomes separately from model failures."

The separation is mechanical, not aspirational. Nine terminal classes exist in the package's schemas; five of them — harness failure, environment failure, grader failure, invalid case, and cancellation — are infrastructure-invalid, and the report layer holds them out of the reliability denominator entirely. The resolved rate divides successes by valid attempts only, where valid means resolved, model failure, safety failure, or timeout. The invalid rate divides by all attempts. Two denominators, on purpose: a flaky runner can neither sink an agent's measured reliability nor launder a genuinely broken one, and the report's checklist requires both columns visible — "Invalid-run rate and typed failure breakdown."

One boundary inside that split deserves its own sentence: in this machinery a timeout is model-side. It counts as an unresolved valid attempt in the reliability denominator, not as infrastructure noise. The class list shows the placement — timeout sits with the candidate-side outcomes, distinct from the harness and environment classes — and an evaluator who wants timeouts treated as excused must argue that explicitly, not inherit it silently. Variance design lives exactly here: decide what counts against the agent, write it down before dispatch, and let the typed ledger enforce the decision after.

The agreement dimension, measured honestly

Between the resolved rate and the failure breakdown sits a third number that repeat-hungry evaluations misread most often: repeat consistency. The package computes it as a majority share — the larger of the resolved and unresolved counts, divided by valid attempts. Four of five resolved is 80 percent consistency; a three-two split is 60 percent whichever side wins; five of five is 100 percent. The number answers one question — did the valid attempts agree? — and its reading discipline is to keep it that single question. It carries no ordering, no drift detection, no notion of clustering: a three-then-two sequence and an alternating one produce the same figure. It is an agreement reading, not a variance statistic, and decisions that need spread need the interval and the attempt count beside it.

The overview page frames where consistency sits among its siblings: "Resolution is primary. Runtime, paid cost, invalid-run rate, and consistency remain separate evidence; unavailable cost is never represented as zero." Runtime gets the same multi-view honesty. The report summarizes elapsed time three ways — mean, median, and ninety-fifth percentile — across every retained attempt, valid or not, with the percentile taken by rank order rather than interpolation. The triple exists because attempt runtime is exactly the shape that punishes a single average: one pathological slow attempt drags the mean, leaves the median unmoved, and shows up in the ninety-fifth as the tail it is. The checklist row that binds both dimensions reads "Runtime distribution and repeat consistency." — distribution, not a number; consistency, not a promise.

Freezing the repetition before results exist

Everything above assumes the repeats are comparable, and comparability is decided before the first dispatch, not argued after the last. The attempt count, the identity list, and the budget are fixed in the plan — the plan machinery belongs to the operator walkthrough and is not restated here. What belongs to variance design is the pooling rule. The comparison guidance is direct about it: "Only combine attempts when their case meaning and compatibility inputs permit it." and "Preserve exact prompt, source, selected wiki, tool policy, budget, package, grader, and agent-system identities." In the report layer that rule is a hash: eleven inputs define a cohort key, the grader version among them, so a regraded generation never silently merges with the evidence it regraded. Identity itself is observed rather than declared — the execution envelope reports what actually ran, and the report layer groups by agent, provider, model, and the two package versions, so a mid-experiment model swap surfaces as a separate system carrying its own small sample rather than melting into the first system's average. Recovery inherits the honesty: retained evidence that predates hash-bound terminal identity stays retained but cannot claim an observed provider.

The stationarity caveat is the design's quiet center. The binomial arithmetic in the first sections assumes one fixed underlying rate; real chains drift — provider-side changes, environment updates, package revisions land between attempts. The package does not assume drift away. It partitions by identity hash, so an attempt under a new package version lands in a different cohort rather than contaminating the old rate, and the longitudinal story is told by union with provenance: "Reports may union compatible evidence across experiments while preserving provenance and sample size." Over time that becomes the drift detector — "Regression suites track whether a recommendation remains dependable over time." — comparing cohort against cohort, not memory against memory. The command that assembles the pooled picture reads retained evidence only, and its output is one content-addressed object:

YYLO_BENCHMARK_REGISTRY=/private/yylo-benchmark-registry yylo-benchmark report --task TASK_ID

Each report version is retained as a new derived layer beside the experiments it reads; nothing about the historical attempts is rewritten to make a neater chart.

How variance changes the pick

Put the pieces back into the decision. A variance-blind selection asks which agent won the demo; a variance-aware one fixes the bar first — "Define the reliability threshold and category before inspecting the winner." — a bar picked after the standings are already visible is a story, not a standard. With the bar fixed, overlapping intervals stop being an embarrassment and become a routing instruction: the cheap system that credibly clears the bar is the default, and one escalation candidate — the stronger, costlier option the intervals could not rule out — is named for the runs that matter most. Spread converts the hunt for the best agent into a portfolio question: which bar, which budget, which fallback.

Two closing disciplines keep the portfolio honest. The recommendation stays what the package's overview says it is: "Compare repeated valid attempts and publish recommendations, not automatic routing." Advice, carried with its evidence and its date — never an automatic switch. And the evaluation stays repeatable: the same plan, the same grader bytes, the same intervals, re-run when a cohort moves. An agent pick that cannot survive its own variance measurement was never a pick; it was an anecdote with a budget line.

Ownership and neighbors

This page's single job is the measurement and interpretation of run-to-run spread on one task. Producing the evidence belongs to the operator walkthrough; comparing several agents under these rules belongs to the bake-off design page; which instrument produces which kind of number is the instrument taxonomy; and why a public board's number forecasts one repository's outcomes only weakly — sampling variance at the board level — is the leaderboard-transfer page. Three deeper companions — the immutable evaluation plan that freezes the matrix, the package's full methodology reference, and a published dataset of repeated attempts — are approved as later pages in this program. The panel below installs the benchmark that turns a surprised flip into a measured rate.