Honest cost, failure, and zero evidence semantics

Every agent evaluation eventually prints a number, and the number is not always the danger. The danger is the cell next to it. A report that shows $0.00 for an attempt is making one of two claims — "this genuinely cost nothing" or "we never learned what this cost" — and most tooling prints the same glyph for both. The same collapse happens with failures: an attempt that crashed the harness, an attempt whose case was malformed, and an attempt that simply was not run all disappear into the same blank or the same red mark. When the cheap and the unknowable share a symbol, every downstream average, ranking, and recommendation inherits the lie. This page is the counter-vocabulary: the typed states an honest evaluation distinguishes, and the reporting rules that keep them distinguishable after arithmetic has had its way with them.

The semantics documented here are YYLO Benchmark's, as committed on this branch. Every quotation was copied from the package's own text and re-checked character-for-character, and each mechanical statement below was derived anew from the committed source rather than recalled from memory (verification date 2026-08-28). Surrounding pages hold their own ground and are linked, not restated: the operator walkthrough runs the command path that produces this evidence, the grading protocol owns verdicts, the variance treatment owns how rates are read, the recovery guide owns interrupted runs, and the bake-off design owns multi-system standings.

A value and a completeness state are two facts, not one

The package's cost field is not a number. It is a discriminated union whose discriminator decides what the number can even mean, and the retained-evidence guidance names the principle plainly: "Cost has both a value and a completeness state." Four states exist, and only two of them may carry a dollar figure at all:

[
  { "completeness": "complete", "usd": 0 },
  { "completeness": "partial", "usd": 0.41 },
  { "completeness": "unavailable", "usd": null },
  { "completeness": "not_applicable", "usd": null }
]

A complete cost is a measurement of the whole attempt. A partial cost is a real observation of part of the spend — a floor, never a total. An unavailable cost is a confession: money may well have moved, and the machinery could not observe it. A not_applicable cost says the attempt never had a billable event to observe in the first place. The schema enforces the pairing structurally — usd must be a nonnegative number for the first two states and must be null for the last two — so a lying zero cannot even parse its way into the record.

The first line of that block is the one that surprises people: zero is a legitimate, complete measurement. The guidance is explicit about how it earns that status: "A genuine zero is valid only when direct structured evidence proves it; missing cost is not zero." The package holds itself to this distinction under test — its release-readiness gate refuses to pass without executed positive controls for all three shapes, case IDs cost:missing (observed as unavailable with null), cost:zero (observed as complete with exactly 0), and cost:positive (observed as complete and greater than zero). Telling those three apart is not a documentation promise; it is a release requirement. The README states the retention rule the same way the schema enforces it: "Complete and partial USD values are retained when supplied; unavailable and not_applicable retain usd: null and remain valid evidence rather than being converted to zero or harness failure."

Where each cost state is born

States this different need producers this specific, and the two execution paths shape their own.

In a task-case attempt, cost arrives one way only: inside the public execution envelope, parsed from the final structured object on the candidate's stdout — progress logs before it are ignored, and nothing else is consulted. If no valid envelope surfaces, the reconciliation falls back to unavailable with null; no log line, no model prose, and no partial output can supply a number. The envelope itself is assembled upstream of the benchmark, and its builder's docblock states the trust model in a clause — the contract is built "from backend observations, never assistant prose" — so the attempt-side reader either finds a compliant cost object inside that channel or honestly reports that the economics went unobserved.

In the tracked-workflow path, a separately reviewed boundary module produces the evidence, and it adds the fourth state to the vocabulary's daily use. Deterministic script steps and synthetic transport runs — work performed by the local process, with no provider child spawned — record not_applicable, because there is no billable event behind them. A model-step envelope's cost is accepted only when its completeness is one of the four literals; anything else degrades to unavailable. And when a step terminates without a valid envelope at all, the boundary writes a harness-invalid receipt whose cost is unavailable — the failure is recorded as failure evidence, and its economics stay honestly unknown instead of being back-filled with a zero.

Two smaller producers round out the picture. The investigation helper — a bounded analyst question run against retained evidence — measures its own elapsed time and reports its cost as unavailable, because no billing envelope exists for that child. And the recovery machinery described by the interrupted-run guide writes one record that carries both truths at once: an indeterminate attempt closes with an elapsed time of zero, which is a measured fact (it began and ended at one recorded instant), beside a cost of unavailable, which states plainly that whatever that dispatch billed past the crash line is unknowable. One record, two states, no contradiction.

Failure is a vocabulary, not a blank

An attempt's outcome is not a boolean with decorations. It is one member of a closed, nine-value terminal class set — resolved, model_failure, safety_failure, harness_failure, environment_failure, grader_failure, invalid_case, timeout, cancelled — and the schema refuses the pairing of a resolved flag with any class other than resolved. Two seats in that set, safety_failure and invalid_case, have no producing code path in the committed package as of this evidence date; they are reserved vocabulary, which is itself the honest posture: the class list is closed and versioned, not a bucket that grows silently, and a class with no producer says so by absence rather than by being quietly removed.

Classification is a fixed cascade, checked in a fixed order: cancellation signals first, then timeout, then harness failure when the envelope is missing or its identity does not match the requested provider, model, and version, then candidate-side failure. The subtlest line in the cascade is the last one, and the module's own comment states it: "successful execution remains unresolved until governed grading." A clean exit proves only that the process ran; the verdict belongs to the governed grading step, whose protocol the blinding guide owns. When grading runs, its result overlays the candidate's class: a successful candidate that passes becomes resolved, a successful candidate that fails the rubric becomes grader_failure, and an unsuccessful candidate keeps the class it earned on its own. A grader that cannot run at all is not a gap in the data — it is recorded as a failing decision under a missing:<profile> identity, with its error retained as the grader's output. Even the class produced when the runner itself throws — environment_failure, with the error message preserved in structured stdout — is evidence about the environment, deliberately kept out of the model's column.

Null is how a report says unknown

The vocabulary above would be decorative if aggregation flattened it again, so the arithmetic is built to preserve it. The clearest expression is the humblest: a rate computed over an empty denominator is not 0.0 — it is null, with a null interval beside it, and the runtime summary reports null for mean, for median, and for the ninety-fifth percentile when there are no values to summarize. Nothing, printed as nothing. The same rule governs the money: cost per successful resolution is born only when every attempt in a system's group carries a complete cost and at least one succeeded; any other group reports null for the figure, and the typed label traveling beside every report — complete, incomplete, or not_computable — says which situation holds, with not_computable reserved for the group where nothing succeeded at all. A group whose members all genuinely cost zero still earns its per-success figure; a group with one unavailable member does not, and says which.

The workflow comparison table applies the same discipline to booleans. When a model's receipts for a step include a harness-invalid one, that cell is null — not false. "We could not get a valid answer" and "we got a valid answer and it was no" are different facts, and the table refuses to let the first decay into the second. How to read the rates and intervals that do exist — width, overlap, separation — is the variance page's charge; the invalid-versus-valid denominator split behind them is documented there too.

The rule also runs before any money moves. A workflow dry-run renders estimate availability per model — each entry available or unavailable — and its estimated_totals stays null unless every selected model carries an override; the estimates object is null outright when none does. The README states the same contract: "Policy estimates are optional exact provider/model overrides; dry-runs report per-model available/unavailable, and omit a total unless every selected model has an override." The dry-run record even declares its cost posture as a typed field — cost_tracking: { mode: 'best_effort', unavailable_is_valid: true } — so a machine consumer cannot mistake the bookkeeping for a guarantee.

Sums that refuse to fold

Dollar aggregation could launder everything above, so it is built from the same states. Complete costs sum into one total; partial observations sum into a visibly separate one that is labeled as observed, never as spent; unavailable and not_applicable contribute to neither and are counted instead. The task-case report exposes exactly these books — complete-attempt count, partial-attempt count with its observed total, unavailable count, not-applicable count — and the workflow report carries the trio the README names: "Reports expose complete cost, all observed cost, and incomplete-cost counts." The comparison checklist makes the reporting obligation explicit with its own row: "Complete, partial, and unavailable cost coverage." A reader of the report can always reconstruct how much of the total is measurement and how much is floor.

The practical consequence is that a number and its coverage travel together, and the counts arbitrate. A complete-cost sum is only ever the spend of the attempts its complete-attempt count names — $4.20 is a five-attempt figure only when that count reads five; with four complete and one unavailable, the same sum holds just the four, the hole count holds the fifth, and the per-success figure disappears rather than pretending. The plan guide explains why the inputs stay frozen long enough for these books to be comparable at all.

Not-applicable answers a different question

The state most systems lack entirely is the one that says "your question does not apply." unavailable and not_applicable both hold null, and lazy tooling treats them as synonyms, but they answer different questions. unavailable says a cost existed and was not observed. not_applicable says no billable event existed — the deterministic script that ran locally, the synthetic transport that spawned nothing. Collapsing the two either inflates a run with fake zeros or brands honest local work as unknowable. The lifecycle guidance binds the whole stance in one sentence: "Complete and partial USD evidence is retained when available; unavailable and not-applicable cost remains explicit null evidence and does not invalidate an otherwise valid run." Absence of a price is never itself a failure, and null is carried forward as what it is — explicit evidence, not an error to suppress.

Cost is evidence, not permission

One more seam separates honest reporting from honest spending, and confusing it causes the worst versions of both. In the task-case path, authorization and evidence are two different books: each attempt reserves its per-attempt ceiling against the plan's immutable aggregate before dispatch, regardless of what is later observed, and the observed usd — when there is one — is checked against that ceiling after the fact. The reservation is a promise about the worst case; the cost record is a measurement of the actual. In the workflow path there is no authorization book at all: plans carry no spend grant, ceiling, or reservation, and the CLI refuses both spending flags with the same phrase — "cost is best-effort evidence" — rather than accepting an authorization it will not enforce. The lifecycle page compresses the design into its shortest true form: "Workflow cost is observational, not authorization." An evaluator who needs spend ceilings books them in the task-case path; an evaluator who only needs truth about what was spent gets it in both.

Where this page sits

The values themselves are this page's territory — the four cost states, the closed failure vocabulary, and the null-and-typed-label rules that survive aggregation. Around it, the linked guides own their own surfaces, day-to-day operational truth sits in the package's docs, and an evidence-dated methodology reference holds a later seat in this program's approved checklist. The audit side of the same honesty has its own sentence in the retention guidance: doctor operations "diagnose but do not repair history silently." That is the posture of everything above — records are corrected by appending, never by smoothing. If this vocabulary is useful to your own reporting, cite this page by the stable URL shown in the panel below.