Recover incomplete benchmark runs without losing evidence

Long benchmark runs die in unglamorous ways. The laptop lid closes, the CI runner is reclaimed mid-hour, a provider connection drops, the shell that owned the process exits. When you come back, the run is half a run: some attempts finished and were graded, one died in flight, a few never started. From that wreckage there are exactly two wrong moves — throw the directory away and re-pay for everything, or press run again and hope nothing gets billed twice. Both mistakes share one root: treating recovery as a matter of nerve instead of a matter of record. This page treats it as a decision problem with typed answers. Which attempts already stand on terminal evidence? Which ones provably never started? Which ones are stuck in between, where something may have run on the provider's side but no durable verdict came home? Each answer routes to exactly one action, and no action anywhere deletes what already exists.

The machinery under discussion is YYLO Benchmark's, both for task-case experiments on a Kanban case and for tracked-workflow experiments that dispatch governed agent steps. Quotations below were lifted verbatim from the package's committed text and verified character-for-character; every mechanical statement was re-derived from source checked out on this branch, not recalled from prose, as of evidence date 2026-08-28. Neighbors own adjacent ground and are linked rather than restated: the operator walkthrough carries the full command path from linting a case to reading a report; the grading protocol owns verdict semantics; the failure-design guide owns the general engineering of crash-safe long-running workflows; and the plan guide owns the frozen object every attempt hangs from.

What a crash actually leaves behind

Nothing about an experiment's truth lives in the memory of the process that died. Every durable fact is an artifact in a content-addressed registry, appended with its digest and never rewritten: per attempt, the contract that fixes its identity, a dispatch marker stamped before the candidate is launched, the captured output streams, the diff rebuilt from raw filesystem bytes rather than any Git story, the structured terminal evidence, the candidate result, and the graded verdict with its integrity-linked receipt. The lifecycle guidance compresses the whole posture into one sentence: "Recovery verifies retained artifacts and terminal markers before dispatch." What a crash removes is never the finished evidence — it is only the last write of the attempt that was in flight when the process went away.

The experiment's identity survives with it. An experiment is addressed by its plan's content hash, and each attempt's identity is derived by hashing plan identity, model identity, and ordinal position together, so the attempt slot "model two, attempt three" is the same addressable object before, during, and after any process exists to run it. Re-accepting the plan after a crash is deliberately boring: the machinery looks up whether a canonical experiment record for that plan identifier already exists, verifies it binds precisely that plan, and where the registry behind it has lost its retained artifacts, a recovery observation is written and the linkage is rebuilt rather than a second experiment minted. Acceptance never dispatches anything, so a crashed run leaves no half-open bookkeeping to repair by hand.

The three statuses an interrupted attempt can hold

Pull the crash apart attempt by attempt, and every attempt lands in exactly one of three states, with nothing in between:

• Terminal. A graded result exists for that attempt identity, with its terminal evidence and receipt chain intact. The attempt is done. Its verdict stands as-is, including any later grading generations appended on top of it — the manifest's ordering picks the current one deterministically, so generations stack instead of colliding.
• Dispatched without a terminal. The dispatch marker exists, but no result came home. Something may have run and been billed on the far side of that marker, and the crashed process took the answer with it. This attempt is indeterminate, and indeterminate has one meaning here: it must never be automatically dispatched again. The package's public recovery inspection reads only durable markers and states the rule as a contract on its own return value: "Callers must never redispatch IDs in dispatchedWithoutTerminal."
• Neither marker. The attempt provably never left the ground — no dispatch marker, no result. It is safe to start, because starting it is its first dispatch, not a second one.

That three-way split is the entire trick. Recovery questions become hard only when statuses are mixed together — when a directory of logs has to be eyeballed for signs of life. Durable markers make the statuses machine-readable, and the package's own docblock says what the marker means: "A durable dispatch marker forbids automatic redispatch after a crash."

The decision tree: resume, regrade, or discard

With statuses in hand, the operator decision collapses to a tree with three branches, and the lifecycle page states the gate the machinery enforces: "Interrupted nonterminal work may resume only where the contract proves that doing so cannot duplicate paid execution."

Resume is the default branch, and for task-case plans it is not even a separate command. Run the same run command against the same immutable plan file: terminal attempts are replayed from their retained results and marked as recovered, attempts holding neither marker dispatch for the first time, and the experiment index is brought current from what actually exists — a no-op when it already reflects a finished run. No finished attempt is re-billed, because a finished attempt is never re-entered. For workflow plans there is a dedicated command whose own description is the whole philosophy in a clause — it exists to "Recover a workflow plan from durable intent without blind redispatch" — and it re-verifies the plan, policy, and tracked-source bindings before it touches any intent, with a dry-run mode that renders the recovery order with zero dispatch. Rejected upfront checks never leave the ambiguous zone at all: preflight and binding failures happen before any durable dispatch intent is written, so "a rejected request stays provably not dispatched and recoverable without deleting evidence".

Regrade is the branch for a subtler damage pattern: the candidate truth came home intact — contract, output, patch, terminal evidence all retained — but the verdict layer on top of it is missing or stale, say a grader that was unconfigured during the crash window. The rerun itself will still close an attempt whose verdict never landed as an indeterminate harness failure; regrade is the operation that appends a genuine verdict generation from the retained candidate truth afterward, and the manifest's ordering makes the newer generation the current one. The package keeps that path — it re-runs governed grading from retained candidate artifacts while accepting no candidate runner and re-dispatching nothing; the regrade operation as an end-to-end operator flow belongs to a separate approved page, and this guide claims nothing about its current surface beyond the mechanism just stated.

Discard — and this is the branch most systems hide — applies to the indeterminate middle status, and it discards the *rerun*, never the evidence. The lifecycle page gives the exact rule for the residual case: "Otherwise create a new attempt with a distinct identity and retain the interrupted evidence." Identity makes this cheap: because an attempt's identity is derived from plan, model, and ordinal, a genuinely new attempt in a genuinely new plan is a new object by construction, and the interrupted one stays addressable beside it forever.

The attempt you must not re-bill

The discard branch deserves its own section, because it is where naive systems quietly corrupt money and truth at the same time. In the task-case path, when a rerun meets a dispatched-but-not-terminal attempt, it does not skip it, does not retry it, and does not ask the operator to guess — it writes the indeterminacy down as the attempt's terminal answer. (The workflow path writes no substitute terminal here; it stops and demands manual recovery, as the next section shows.) The appended result carries the harness_failure class — the class that says the harness, not the model, could not produce a verdict — with a null session, a null patch, zero elapsed time, and cost evidence recorded as unavailable rather than zero, because what the attempt cost on the far side of the crash is exactly as known as everything else about it: not at all. The retained terminal evidence for this record is the recovery document itself, schema-exact in three fields:

{
  "schema_version": "juno_benchmark_recovery_terminal.v1",
  "attempt_id": "<hash-derived attempt identity>",
  "reason": "dispatch marker exists without durable terminal evidence; refusing paid redispatch"
}

The reason string is not editorial commentary; it is the recorded literal, and a second structured-evidence artifact beside it carries the machine-readable indeterminate_after_dispatch classification. The attempt now has a beginning, a middle that honestly says "unknown", and a typed end. It counts in every report exactly as what it is — an unresolved harness failure — and the honest semantics of unavailable cost and failure evidence run deep enough for a dedicated treatment later in this program.

Two consequences follow. First, the experiment still finishes: the other attempts proceed, the index closes, the run produces a report with a hole in it that is labeled as a hole, which beats a report with a hole in it labeled as a success or a rerun. Second, the operator is never tempted into the catastrophic manual move — deleting a marker to "let it run again" — because the marker is append-only in a content-addressed registry and the record it guards is now a legitimate terminal citizen of the experiment, not an obstruction.

The workflow path: reconcile through the journal, then resume under policy

Tracked-workflow experiments dispatch governed agent steps through a separately reviewed, hash-pinned boundary module, and recovery there is a conversation with that module's memory rather than a guess at process state. "The boundary keeps a private dispatch journal so recovery reconciles exact truth instead of guessing." When a rerun encounters a step whose dispatch intent exists without a terminal marker, it asks the boundary to reconcile that exact dispatch, and reconciliation returns one of exactly four answers: terminal, ambiguous, or one of two safe answers — the step is proven not to have been dispatched, or the boundary certifies it safely resumable. Both safe answers route through the same gate before anything runs again. The README binds the whole loop together: "Recovery reuses retained terminals or asks the boundary to reconcile durable intent before a policy-permitted resume."

Each answer routes mechanically. Terminal results are validated against the invocation's exact identity — dispatch, provider, model, and YYLO version all must match — before being accepted as truth. A resume happens only when the step's own policy classification says retry-safe, and even then it is fenced twice: every recovery attempt is numbered, the numbering must advance by exactly one with each retained intent, and a step that exhausts its configured maximum stops the run demanding manual recovery instead of looping forever. The recovery intent is appended before the resume call, so a crash during recovery leaves exactly the same kind of durable evidence as a crash during first dispatch — the ladder never runs out of rungs underneath itself. Ambiguity, a step whose policy is not retry-safe, and a step that has exhausted its recovery budget are the places the machinery firmly refuses: it reports that manual recovery is required, names the step, and stops. Credential and identity checks share the pre-intent ordering that makes recovery tractable at all: "Live preflight fails closed before any durable dispatch intent exists when a provider credential, the exact model identity, or the exact YYLO version is missing."

One more mechanism covers the crash window between a step's evidence receipt and its terminal marker. If the rerun finds a retained evidence receipt for a dispatch but no terminal marker, it does not reconcile or resume anything — it converts the receipt into the terminal record it already proves, binds it to the current invocation, and marks the step recovered. The marker was a convenience; the receipt was the truth.

One owner per plan at a time

Everything above assumes the crash was not actually a second process starting alongside the first, and that assumption is enforced rather than hoped for. Both experiment paths take a persistent typed lock on the plan identity — the task-case path leases the task plan, the workflow path leases the workflow plan — and the locks are cross-process, file-backed, and stamped with the owning pid and host; a live owner is never reclaimed, only a provably dead one. The README states the reach of the rule: "It takes persistent typed locks, keeps production model experiments sequential, and makes ambiguous external effects manual." A second run against the same plan therefore waits or fails on the lock rather than interleaving dispatch markers with a peer.

The experiment's canonical index has its own crash rule. When a rerun completes and every outcome was recovered from retained evidence while the index already reads terminal, the index is deliberately left untouched — no rewrite, no new mutation receipt for nothing. Any other terminal completion writes the index once: status terminal, the per-attempt summary, the resolved count, and the evidence digests, with the mutation receipt appended to the experiment itself. Re-running a finished experiment is thus a read in disguise, which is what idempotence looks like from the outside.

Standing audits once the run stands back up

Recovery ends with proof, not vibes, and the package ships the prover. The doctor command re-verifies an experiment end to end from retained bytes. For task-case experiments it re-checks the registry, cross-checks every digest the ledger's benchmark field indexes against artifacts that actually exist and read back, and re-verifies the grader receipt chain on each attempt that ended resolved or in a grader failure. For workflow experiments it counts the thing recovery exists to drive to zero: dispatch intents that still have no terminal marker, reported as ambiguous dispatches, alongside the tallies of terminals, evidence receipts, judge intents, and harness-failure outcomes — where a terminal that ended in harness failure is counted as retained integrity truth rather than as a defect that needs cleanup. Run it before circulating any salvaged report; the repeated-attempt statistics explain how to read rates that include typed failures, and the operational reference lives under the package's own docs.

Where this page sits

Recovery is this page's whole charge: what to do when a benchmark run is already broken — the status semantics, the decision tree, the refusal to re-bill the unknowable, and the audits that close the incident. Around it, the operator walkthrough walks the unbroken loop, the grading protocol governs verdicts, the plan guide freezes the object everything hangs from, and the failure-design guide generalizes the checkpoint discipline to any long-running workflow. Two subjects touched here — the honest semantics of unavailable cost and failure evidence, and the regrade operation as a hands-on flow — have later pages approved in this program's checklist. When the next run of yours dies at two in the morning, the branch of the tree to take is already written down; the panel below installs the machinery that keeps it that way.