YYLO

2026-08-29 · Updated 2026-08-29 · 10 min read

Repository-specific benchmark cases and reproducibility notes

A downloadable, versioned library of six repository-specific coding-agent benchmark cases mined from the public git histories of the released YYLO ecosystem repositories: each case pins a real base commit and the real later grading commit that carries its deterministic checks, with task, environment, and grading notes sufficient for reproduction and both fail and pass states verified.

By Juno AI INC · benchmark · evaluation · case-library

An evaluator who wants to measure coding agents on a real repository, rather than on a shared public suite, starts with an empty bench: no cases, no pinned baselines, no grading material, and no evidence that any of it reproduces. This page fills that bench with original material. It publishes a library of six repository-specific benchmark cases, mined from the public git histories of the three released YYLO ecosystem repositories — the yylo CLI, yylo-ledger, and yylo-benchmark packages — where each case pins one real base commit, the real later public commit whose test changes grade the work, a candidate-visible task statement, the environment setup that materializes the baseline, and the exact command that adjudicates the result. Both states were observed on 2026-08-29: every case fails its grading command on the untouched base commit with the grading patch applied, and passes it on the grading commit's own tree. The library ships as a versioned JSON document whose entries are pinned by a content hash, so a citation can name the exact bytes it leaned on.

Provenance is the whole value of such an asset, so it opens with honesty. The cases are mined, not invented: each one is a real engineering fix that landed in a public repository, and the library wraps those histories for reuse with attribution. The route itself owns zero rows in the keyword evidence corpus behind this site's SEO program, and it claims no demand figure of any kind — the library is justified by the evidence it carries, not by a search-volume estimate. Nothing in it measures any YYLO product: the graded work belongs to the repositories it names, and the library's job is to make that work reusable as measurement material.

What this library is

Each case is one JSON object with five load-bearing parts. The pins: a repository_id, a full 40-hex base_commit the candidate starts from, and a full 40-hex grading_commit — the real later fix whose test changes form the grading patch, kept evaluator-side. The task: the candidate-visible engineering job, stated as behavior with a clear resolved condition, naming no implementation site. The environment: shell commands that materialize the baseline from a plain clone, plus interpreter requirements, dependencies, and the offline character of the graded suite. The grading: the graded files, the one-line command that applies exactly the grading diff onto a candidate tree, the check command, and the resolution condition. And the verification: the observed result of the grading command on the base commit with only the grading patch applied, and on the grading commit's tree, both dated.

The population is deliberately small and deliberately spread: six cases across three repositories, two languages, and four categories — three backend fixes and one release-tooling fix in the Python ledger, one harness-configuration feature in the TypeScript CLI, and one evaluation-boundary fix in the TypeScript benchmark package. A fail-closed integrity checker is committed beside the data in the same public repository, and the header digest covers the case entries under a fixed canonical serialization — object keys sorted recursively, no whitespace — so drift is detectable rather than silent. This is a different artifact from the longitudinal leaderboard dataset: that one wraps third-party, self-reported public submissions at arm's length, while this one mines first-party public fix histories deep enough to carry full reproduction notes, and both keep their own versioning and citation contracts.

The reproduction contract

One loop reproduces every case, and it keeps the grading material out of the candidate's sight the whole time. Clone the repository, check out the base commit, install dependencies, and hand the candidate the task text plus whatever the environment notes say it may see. When the candidate is done, apply the grading patch — the test-file diff between the base and grading commits — and run the check command. The case resolves when that command exits zero with every test in the graded files passing. Concretely, for the ledger's creation-Git dirty-flag search case:

sh
git clone https://github.com/yylo-dev/yylo-ledger.git && cd yylo-ledger
git checkout d78903c79a1cb3f0664815079f6e7e15bb4fb990
python -m venv .venv && . .venv/bin/activate && pip install pytest ruamel.yaml
# candidate works on the task; evaluator then grades:
git diff d78903c79a1cb3f0664815079f6e7e15bb4fb990 249689d816a838bf3c64f1ea53fb703bcd837062 \
  -- tests/unit/test_record_search.py | git apply
PYTHONPATH=src python -m pytest tests/unit/test_record_search.py -q

Three rules make the result meaningful. The candidate is free: any agent, any harness, any implementation that satisfies the tests resolves the case — the grading commit is one known-good answer, not the required one, and the library never asks a candidate to reproduce its diff. The grading files are hidden checks: the candidate sees the task and the environment notes, while the grading patch exists only on the evaluator's side of the fence, because a candidate that has read the tests is being graded on reading comprehension. And a candidate that edits the graded files collides with the grading patch when it applies — such an attempt consumed its own grader and does not resolve. These are the same visible-versus-hidden boundaries the benchmark methodology teaches at the product level; this library is the library-shaped version of them, runnable with nothing but git, a language runtime, and a test runner.

The case catalog

Six cases, each with its observed fail state on the base commit and its observed pass state on the grading commit's tree, all recorded on 2026-08-29:

  • `benchmark-boundary-block-scalars` — yylo-benchmark, category evaluation, TypeScript. Make the reviewed workflow boundary's YAML-subset reader accept every multiline block-scalar form the planner's canonical emitter produces, round-trip canonical prompts byte-exactly through the dispatched argv, and validate compiled workflow bytes before any durable dispatch intent exists. Base with grading patch: 12 failed, 6 passed. Grading tree: 18 passed.
  • `ledger-archived-id-reservation` — yylo-ledger, category backend, Python. Refuse any import of an archived record ID before any other validation and before any bytes are written — no task path, no history event, no cache mutation. Base with grading patch: 1 failed, 24 passed. Grading tree: 25 passed.
  • `ledger-artifact-revision-creation-context` — yylo-ledger, category backend, Python. Carry a record's Git creation context forward unchanged across artifact revisions instead of silently dropping it from the revision manifest. Base with grading patch: 1 failed, 5 passed. Grading tree: 6 passed.
  • `ledger-release-version-bumping` — yylo-ledger, category release, Python. Point the version-bump helper at the canonical package identity and make it parse and order final, alpha, beta, and release-candidate versions per PEP 440 while keeping network lookups mocked. Base with grading patch: 6 failed. Grading tree: 6 passed.
  • `ledger-search-creation-git-dirty-filter` — yylo-ledger, category backend, Python. Make worktree_dirty creation-Git search filters return exactly the matching records instead of raising a query-time database error, for both flag values. Base with grading patch: 2 failed, 5 passed. Grading tree: 7 passed.
  • `yylo-project-model-shortcuts` — yylo, category harness, TypeScript. Add per-subagent modelShortcuts to project configuration: colon-prefixed entries, opaque targets that may chain to project or shipped shortcuts, subagent-scoped lookup with project overrides, recursive resolution, and actionable failures for cycles, unknown shortcuts, and malformed data. Base with grading patch: 4 failed — exactly one per graded file. Grading tree: 222 passed across the four files.

The spread is the point. Two languages, three grader stacks — a hermetic pytest unit suite, a pytest integration suite that drives a real git binary for roughly thirty seconds, and vitest suites across five TypeScript files in two repositories — and four categories of engineering work: backend behavior, release tooling, harness configuration, and evaluation infrastructure. A candidate that resolves all six has done six different kinds of real work in two ecosystems, and every one of those verdicts recomputes from a fresh clone with the commands in the JSON.

How the cases were mined and verified

The mining rule came first, and it excluded more than it admitted. A fix commit qualified as a case only when three things held at once: its diff resolves a statable engineering task, it adds or tightens deterministic checks that run offline from a plain clone of the public repository, and those checks are green on the fix's own tree while failing on its parent with only the test diff applied. Candidates that failed the rule were dropped with reasons. Some of the CLI's Python template-script suites — the task-workspace, integration-workspace, and merge-queue suites, by a grep of the public tree — load their fixtures through a package-bound guard that demands an installed controller identity or a monorepo checkout, so a plain public clone cannot even load them; a real task-workspace fix landing only there was excluded for standing outside a standalone reproduction contract. One release-train suite carried a failing test at both the base and the fix commit, so it was excluded as a grader: under this library's whole-file resolution contract, a file that is red at both ends can never pass, so it adjudicates nothing.

Verification then ran the two states per admitted case, on Node v26.2.0 and Python 3.13.15 with pytest 9.1.1 and vitest 1.6.1 in both TypeScript repositories, using the network for dependency installation and nothing else. The offline character was then proven rather than assumed: all six graded suites were re-run under an OS sandbox with networking denied, and every one passed unchanged. The base state is a detached checkout of the base commit with the grading diff applied through git apply; the pass state is the grading commit's own tree. Both observed results are recorded verbatim in each case's verification fields, next to the date — so a reader can see what "fails before, passes after" concretely meant, including the sqlite no such column error behind the search case and the missing creation_context key behind the artifact-revision case. Re-running the observation is exactly the reproduction loop above; nothing about the mining depends on this site's tooling.

Running the library under YYLO Benchmark

The library is grader-agnostic by construction — the contract is git, a runtime, and a test runner — but it maps onto YYLO Benchmark's case model with no translation loss, because that model was built for exactly this shape of evidence. A case there is a ledger task carrying benchmark metadata: the repository id, the full base commit, a case version, a category, a grader profile name, and optional candidate-visible project knowledge, with the task body serving as the candidate prompt. Every one of those fields has a direct source in this library's pins and notes, and the grading notes describe the executable checks a grader profile would bind to bytes. The stages the product wraps around a case — immutable plans, spend ceilings, isolated snapshots, retained receipts — are stated with their guarantees in the methodology reference, while authoring and running a first case is taught step by step by the repository walkthrough; this page owns the library and its reproduction contract, not the product loop.

Versioning, citation, and limits

Version 1 does not get rewritten under anyone. The pins are immutable by construction — a base commit and a grading commit are fixed bytes in their repositories' histories, and a repository moving on afterward cannot invalidate either — so a case changes identity only by explicit decision: new cases or revised notes publish as the next library version carrying a fresh entries digest, and citations of an older version remain true of that version. Cite the library by this page's canonical URL together with the library version and the entries digest, sha256:bf7ead5c363b87e2fcc34b08e64bac74f0b5105ba2e1d3c744a4173e5f3c3669, and name the repositories the cases come from. The digest covers the case entries under the same canonical serialization and recomputes independently of this site:

sh
curl -fsSL https://yylo.dev/data/yylo-repository-benchmark-cases-v1.json -o cases.json
jq -cjS '.cases' cases.json | shasum -a 256

The limits are structural and worth stating as loudly as the counts. Six cases from three small repositories is a starter library, not a survey, and all three repositories are maintained by the same organization that publishes this library — the fixes were authored by the people offering them as measurement material, so evaluators who need blind provenance should treat the case text as the artifact and the commit histories as the ground truth rather than trusting either party's summary. Contamination is unmeasured: the fixes are public, and any model may have trained on them, so a resolved case demonstrates present-day capability on a pinned tree, never absence of prior exposure. Interpreter drift can move a suite between environments — the recorded observations name the exact versions used — and a passing implementation is judged by the graded checks alone, which is the contract's strength and its ceiling. What this library offers is the part evaluators otherwise have to build alone: real repository work, pinned baselines, hidden graders, and observed fail and pass states, ready to reuse with attribution. If it saves you a bench, cite the URL, the version, and the hash.