How Much AI Does Your Work Actually Need? Lessons from 75 Coding Attempts

A cheap coding response is not necessarily a cheap completed task. But an expensive response is not a guarantee of correctness either.

We compared five model selections on fifteen historical tasks from our own CLI project. Each selection attempted each task once. We retained the implementations and evaluated them against explicit task and project criteria.

The most useful result was not a universal winner. It was a contrast: all five selections passed every criterion on a self-contained prompt-substitution feature, while work involving filesystem cleanup, runtime identity, and migration exposed failures that passing regression tests alone did not catch.

On the twelve tasks with comparable, fully scored results for all five selections, gpt-5.6-sol had the lowest mean criterion loss. Across the full fifteen-task matrix, gpt-6-sol had the most fully passing assessments and substantially lower reported candidate cost. Those are different findings—not interchangeable claims that one model is best.

What the implementations actually did differently

The examples below come from retained implementation evidence and evaluator findings. They describe historical candidate outputs, not necessarily the current product. We distinguish source inspection from the evaluator's reported test results; this article is not a new independent certification of every patch.

1. Temporary-file cleanup: similar intentions, different boundaries

The cleanup task required the parallel runner to manage temporary files in its designated area without removing unrelated files or durable logs.

gpt-6-sol changed the runner itself. Its retained _owned_temp_path walks through path components and rejects symlinks. Its stale-file traversal also checks the managed namespace before descending. The assessment passed all six criteria, including candidate-authored regression coverage.

gpt-5.6-sol also changed the runner and added tests. But its deletion helper checked lexical containment, the temporary root, and the final path—not every intermediate directory. The evaluator reported a reproduction in which a symlinked intermediate session directory allowed deletion outside the intended area. It passed five of six criteria.

gpt-5.6-luna took a different architectural route: it introduced a TypeScript TransientArtifacts abstraction rooted at .juno_task/tmp. That is visible in the retained source, but the task called for fixing the parallel runner's scripts/logs/tmp lifecycle. The evaluator found that the required runner paths remained wrong. It passed two of six criteria.

This is why patch size and apparent abstraction quality are weak substitutes for task completion. A reusable component can still solve the wrong problem; a mostly correct local fix can still miss a destructive edge case.

2. Prompt substitution: different code organization, the same assessed result

The prompt-substitution task required commands in prompts to be resolved for each iteration, with appropriate handling of literals and command failures.

All five selections passed all seven criteria. Some introduced a dedicated parser or helper module; others kept the implementation closer to the engine. The retained evidence includes tests for substitution behavior, failure handling, and literal-input cases.

This is a useful counterweight to the harder tasks. The lowest-cost selections were capable of fully satisfying this particular requirement. It supports testing less expensive selections on similarly bounded work—not assuming that the same result generalizes to repository-wide migrations.

3. Runtime identity: fixing the code after dispatch can be too late

The runtime-identity task asked the system to tell the truth about which executable actually handled a command across controller routing.

The revised assessments ranged from zero to five criteria passed out of six; none passed all six. One implementation added disclosure in the newer executor, but the wrapper could dispatch to an older executor that did not contain that disclosure. Another labelled a configured runtime as the executor without establishing that it was the code actually executed.

The general lesson is about where a check belongs: adding validation or reporting in a downstream component is insufficient if an earlier dispatch decision can bypass it. Tests need to exercise that dispatch boundary, not just the new helper in isolation.

4. Passing tests did not always mean satisfying the contract

The bounded-status task produced several implementations with passing candidate regression tests, yet the evaluator identified different omissions: a supposedly exhaustive view still clipped reviewer history; a summary missed a failure state; a detail view did not reliably enforce its output bound.

The benchmark also required added or updated regression tests. Existing green tests and evaluator-written probes did not satisfy that requirement. On the Bash 3 status task, for example, gpt-6-luna passed four of five criteria: the evaluator observed the required exit-status behavior, but no candidate regression-test changes were retained.

That distinction matters when interpreting a score. Some missed criteria concerned behavior; others concerned the evidence and regression protection required for delivery. A useful review reads the failed criterion rather than treating every lost point as the same defect.

How we ran the comparison

The study used historical, completed, commit-bearing tasks from one project's available Ledger inventory. It is not a representative sample of all software engineering. Initialization and conversion tasks are also related, rather than fully independent examples.

The fifteen-task matrix consists of an original five-task checklist cohort and ten additional tasks. It excludes the older holistic experiments previously discussed on this site.

The expansion protocol pinned YYLO Benchmark 0.2.1 and YYLO CLI 0.2.11, with the Pi harness, medium thinking, and no extensions or skills. Those are study versions, not a claim about the latest publicly available packages.

Cases used reviewed pre-solution inputs and explicit criteria. Reference solutions and private checks were not supplied as candidate context. The ten-task expansion's focused controls rejected baseline and intentionally broken examples, and accepted reference and behavior-preserving variants. These controls were limited checks—not proof that the reference implementations or evaluators were infallible.

Astra (openai-codex/gpt-6-astra) evaluated the retained outputs against frozen criteria. The comparison uses the revised serialized assessment protocol. The original twenty-five candidate outputs were evaluated retrospectively under that protocol without rerunning the coding models; their original assessments remain separate evidence. We do not select whichever assessment gives a candidate the better score. Of those twenty-five outputs, twelve changed score or scoring status: six became scored after an originally unscored assessment, four had higher loss, and two had lower loss. The other thirteen retained the same score. The download includes both original and revised score fields.

Time budgets changed during the study. Earlier calls retained their original limits. The last forty candidates and their judges received one hour including setup; the twenty-five retrospective judges also received one hour. Two earlier timed-out calls were not replayed. This was not a uniform-budget, seventy-five-attempt experiment.

The matrix finished dispatching, but that does not mean every output received a usable score or every call has known cost. One candidate timed out and received no revised assessment; one judge timed out; two assessments had insufficient evidence. The reported expansion subtotal was approximately $87.11 with two call costs unknown. The closed first-study reported spend was separately $33.960531. Neither number establishes a complete experiment cost or an invoice cap. Preparation, operator effort, and repair work are not captured by candidate cost alone.

What the current Benchmark design contributes

YYLO Benchmark is a thin experiment runner: prepare a reviewed case, execute independent attempts, retain outputs, and evaluate those outputs separately.

Its checklist design freezes the criteria and preserves separate execution and evaluation records. A failed evaluator is not silently converted into a failed implementation. A new evaluation does not require rerunning the candidate. Unknown cost stays unknown, and conflicting judgments can remain separate observations.

These properties make a comparison easier to audit. They do not make judging automatic or perfectly objective. Case reconstruction, behavior checks, environment preparation, and interpretation still require review.

The runner also operates on a trusted host. Fresh repositories and context exclusions reduce accidental answer exposure, but they are not a filesystem, account, or network sandbox. Public-history contamination and changes to provider weights remain unknowns.

How to use these results

Do not choose a default model from this table alone. Build a small comparison around work you actually need done:

• Write the acceptance criteria first. Include preservation, refusal behavior, and regression coverage where they matter.
• Test the evaluator. A check should reject a broken implementation without rejecting every valid alternative to your original solution.
• Retain failures and unknowns. Do not hide a timed-out attempt or treat an evaluator error as a quality score.
• Inspect implementation differences. Ask whether the change reached the right execution path and preserved the required boundaries.
• Repeat before relying on a routing rule. One success suggests a hypothesis; repeated results are needed to estimate reliability.
• Track the cost of acceptable delivery. Include review and repair, not just the model's reported response cost.

Our working hypothesis is that bounded tasks with clear behavioral checks are good candidates for testing lower-cost selections, while cross-component changes deserve additional scrutiny regardless of price. This experiment did not test an adaptive routing policy or measure its savings.

The practical question is not “Which model is smartest?” It is: Which model-and-workflow combination repeatedly meets this task's quality bar, at a cost we understand?

For the preparation process, see the real-repository benchmarking guide. For version-specific commands, see the YYLO Benchmark documentation.