Reading the Verifier

A correct spreadsheet that receives a score of zero

Each task in the benchmark ships with a sandbox and a verifier. Most write-ups begin from the task prompt. This one begins from the verifier, since the verifier alone determines the reward and can be read before any budget is spent.

Reading them statically surfaced one mechanism that recurs and is invisible from the prompt. When an agent writes a formula with openpyxl, the library stores the formula as text and stores no cached result beside it. A checker that opens the file with load_workbook(path, data_only=True) asks for the cached value, finds nothing, and reads None for every formula cell. The workbook is correct. Open it in Excel and every total is right. There is no Excel in the container, and reopening the file with openpyxl recalculates nothing at all.

A single failed assertion removes the whole task

On a partial-credit benchmark this would amount to a rounding error, but the grading here does not work that way. Of the 87 public tasks, 85 are all or nothing, meaning the verifier either passes the task or fails it with nothing in between. A skill that resolves nine checks out of ten is therefore worth the same as a skill that resolves none.

That property changes what is worth building. General advice tends to improve check counts without changing the recorded reward, since the reward only moves once the final failing assertion is closed. A pack is therefore useful in proportion to how precisely it anticipates the assertion the verifier will make.

The size of the affected set

The recalculation trap covers 6 of 87 tasks, which is 6.9 percent of the public corpus. That is the ceiling on what this one mechanism can be worth, and it is worth stating before any result, because a mechanism that fires on seven tasks in a hundred cannot produce a large average lift no matter how well it works.

Five of the six were measured on both arms. Across those five the library converts at a mean of 0.33. Multiply the two together and a private set with a similar mix is worth about 2.3 percentage points from this mechanism. The published figure for the benchmark's own curated skills is +16.4pp, measured across a per-task expert library rather than a general one.

The correction has to be executable

Instruction alone cannot address this, because openpyxl contains no formula evaluator. The pack therefore has to ship something that executes.

So the xlsx pack ships recalc_xlsx.py. It shells out to gnumeric ssconvert, falls back to LibreOffice, evaluates every formula, writes the cached values back into the workbook, drops a CSV value mirror beside it for readers that cannot evaluate anything, and reports any #REF! or #VALUE! cell so the agent can fix it before finishing. Seven packs ship in total, and nine executable scripts between them.

Measured results on the public corpus

Measured paired on the public corpus with Claude Haiku 4.5 on the official harness, one arm with the library mounted and one without, reading each reward from the task's own reward.txt.

The development tasks informed the library, so that mean measures the effect on tasks chosen for relevance rather than a generalization estimate. The held-out set was pre-registered before it ran and returned a null. Its one regression was measured not to be mine: three further baseline rollouts of that task, with no library mounted at all, stall the same way and name the same missing bundled skill.

Scope of the library

The packs target the Office-format work, 24 of the 87 public tasks, and they load on 17 of the 21 I measured. Before spending on wider coverage I classified every failing assertion on disk as contract-shaped or analytical. 12 of the 18 non-Office tasks fail on the computed numbers rather than on any output contract, so there is nothing there for a procedural skill to fix.

A pack fires when its format is what the task is about, and not when the format is only the container the answer gets saved into. A pack named json loaded on none of the tasks I tried it on, including one whose prompt says to write /root/answer.json.

A generator that reads the contract first

The Meta-Skills entry is one skill-creator pack: an environment probe that runs against the real task workspace, a grader-view step that makes the agent read the verifier's contract before writing anything, and an admit gate that lints a generated pack for size, frontmatter, naming and forbidden references.

I ran it end to end twelve times and it produced one conversion, which does not yet beat shipping the static library. It is worth reporting anyway, because the failures share one cause and that cause is traceable. The harness's own self-gen prompt hands the creator the path /root/generated-skills/<task-name>-skill, so the creator was naming packs after the calibration task, and a pack named after one task cannot route on any other. Written advice reached two cases in three. Making it mechanical, as a check in the admit gate, catches 14 of 14 task-echoing names across the 24 packs the creator has produced, with no false rejects on the 10 clean ones.

That last figure is a property of the checker measured offline. No rollout has been run against the gate, and the write-up labels it that way.

Notes for anyone starting on this benchmark

The verifiers are open, static, and solely responsible for the score, which makes them worth reading before any skill is written. A substantial part of what presents as a modelling problem on this benchmark is a contract problem that is already legible in the checker source.

It is also worth establishing whether the contract is the only failing element. Where a task fails on its computed values, a procedural skill offers no remedy, and the budget is better directed at cases where the mechanism applies.

How it was measured

154 paid rollouts, 135.17 USD, all on anthropic/claude-haiku-4-5-20251001 through bench eval run (benchflow 0.6.7, dataset skillsbench@1.1, Docker sandbox at concurrency 1). Each round was pre-registered before it ran, with the task, the number of draws and what each outcome would license fixed in advance, and amendments appended rather than edited.

Every number here is printed by a script in the repository, and one command checks the document against the artifacts: recompute_paired.py for the paired table, reach_census.py for the reach figures, verifier_census.py and deliverable_census.py for the corpus census, trap_projection.py for the private-set projection, and verify_writeup.py to regenerate all of them and fail if the text and the data disagree.

Two measurement problems worth knowing about

Several task verifiers bootstrap their test runner by downloading a tool at verify time. When that download fails, zero tests execute while the harness still writes a reward of 0, which is indistinguishable from a real failure in every field the scorer reads. Separately, an arm that never loaded a skill measures nothing about the library in either direction, so counting those rollouts as evidence would be wrong both ways. Both are detected and held aside as missing data.