Status board
Four buckets, because three of them are usually collapsed into one. The difference between a result that failed, a result that is unusable, and a measurement that was never taken is the whole point of keeping this page.
- The offset-null gate, four times, attempt 1 each, always run before the fits it guards.
- The two-path equivalence gate: 30 of 30 identical completions between the chat path and the rendered completions path, 120 of 120 letter logprobs at maximum absolute difference 0.0, 300 requests with zero retries.
- The determinism preflight on every powered cell, refusing at exit 10 otherwise, and proven able to fail.
- The batch-invariant serving mode: 30 of 30 identical and 0.0 maximum logprob difference at 1, 8, 32 and 64 requests in flight.
- The mechanism battery's three pre-registered outcomes, in both the 7-family and 11-family runs, with the maximum-likelihood path reproducing all 22 shared cells exactly across a prior change.
- One jury configuration clears all ten frozen bars: Qwen3-32B, prompt d, 1,024 tokens, on a pinned line. Its exploratory twin agrees on every verdict.
- Amendment A5 gate G1 on 24 of 24 cells, including the condition the binary scale cannot pass, and the total-effect identity to 1.7e-16.
- The secondary jury matrix: 16 of 16 cells, 70,738 votes against 70,738 planned on every cell, all exit 0, resume proven across eight cells.
- The ladder recipe is not inert and its effect is trigger-specific: organism 0 of 42 to 39 of 42, twin 0 of 42 to 0 of 42 at four recipes.
- Memory for the twelve planned checkpoints fits on both training seeds, 70.91 and 72.93 GiB against 79.25 usable.
- CI is green at main 87cb68e: 1,503 tests collected, 1,477 passing, 26 skipped. Every amendment's frozen guard passes and is proven still able to fail.
- No panel jury configuration clears the gate, on any Q1 file, in any composition. The column of record is therefore the uncorrected frozen regular expression everywhere.
- Three of four judges fail on prompt d, and the two that fail on the contested classes fail in opposite directions on the same bytes.
- All three model rows are UNRESOLVED at every pass. The largest probability that the faithful path exceeds the margin is 0.102 at 18 cells and 0.076 at 24, against the 0.95 the rule requires, and every model-level interval covers zero.
- The randomized anchor and the mediation fit disagree on every model: pooled anchor contrasts 0.146, 0.169, 0.218 against fitted model-level values 0.025, 0.052, 0.016.
- The total-effect identity is not a clean pass: the interval on the model-implied total minus the randomized arm difference excludes zero in 11 of 18 cells, and the model-implied total sits below the arm difference in 16 of 18, by up to 44.9 percent. The sign is never reversed.
- rho* cannot rank anything. A zero-mediation world and a fully mediated one give the same report at n = 3,600, and the largest rho* in the battery sits on zero true mediation.
- The frozen regular expression beats every jury Q1 file on external human data by 4 to 52 specificity points, and the jury's Q2 recall against human unfaithfulness labels is 26 of 365.
- One sensitivity fit did not mix in the 24-cell pass and is printed as a failure. Two model rows carry 70 to 87 divergences after tuning.
- Two roster subjects are held by ruling with zero cells: one has no batch-invariant kernel on any reachable card and returns empty forced continuations, the other has a tokenizer that does not round-trip at its pinned revision.
- Wave-1 cells for three thinking models (56, 140 and 1 usable rows of 1,500): truncated inside the reasoning block with no final answer. Voided and rerun for two; the third stays held.
- Phi-4-reasoning's wave-1 cell: 1,197 of 1,500 clean-correct, comfortably above the floor, but zero scorable forced continuations, because the 24-token continuation is spent opening a reasoning block. Reading clean accuracy alone would have passed it.
- Four cells voided by a port collision and four more by a mode change, all rerun byte-identically at a fixed tree.
- The recipe check's held-in trigger probe measured label recall, not trigger pick-up. Explicitly retired and replaced.
- Pre-repair real-model rho* values of 0.708 to 0.800: three have no stored transcripts and cannot be recomputed without rerunning model calls. Those cells now read not available under the repaired estimator.
- The 70,738 secondary-jury votes: measured, verified, banked, and not yet turned into a single reported number.
- The A5 bridge is uncomputable for every cell on the native arm. What is printed instead is an anchor replay-cell agreement of 0.7244 pooled, labelled a different regime and never used as the bridge.
- Any human label, any agreement statistic, either kill switch. No rater recruited, no golden-set row labelled. Every corrected label column in the design remains uncomputable.
- Any ladder checkpoint of record: 0 of 12. The statistic, the detection-size formula and the refusals are implemented; the inputs do not exist. This is why no cell anywhere can reach validated.
- The residual-stream probe: deferred, never built.
- The held-out audit decision: no artifact.
- 12 of the 18 roster models have no cell at all. The two 70B dense subjects still have no demonstrated tensor-parallel serving line.
- The 72-cell model rows and claim statuses: the cell phase finished with zero failures, the row phase was still running at the last record entry.
- The paper. Not started.
The gap between the last two buckets is where most published work quietly lives. A number that was never taken and a number that failed look identical in a results table that only prints what worked.
Chronology
Phase 2 began after Phase 1 was delivered under grant. The order matters, because several results are only interpretable against when the rule that judges them was fixed.
Phase 1 delivered
The package, the estimator, the sensitivity machinery, the positive control, the synthetic validation gate and the published figures.
An independent review lands
An external reviewer ran a world with exactly zero true effect through the code, which twenty-two internal review voices had not done. The estimator reported a mediated effect of 0.21 on it.
Three estimator defects repaired, and eleven public claims corrected
Missing intercepts, a probit and logit mismatch, and fixed-scale outcome priors. A separate ruling rewrote eleven phrasings on the public site that claimed more than the evidence supported.
Serving determinism measured, and the budget repriced
Batching found to break greedy decoding. The 216-cell grid repriced at 261.7 card-hours against a budget of about 650, so the planned degradation ladder was not triggered.
Fits pass 1, three wave-1 cells
Two cells published as anchored.
Fits passes 2 and 3, at 18 and then 24 cells
Claim status went down. The two anchored cells were demoted when the model-level leg of the rule was applied. A reporting defect was found inside the pass, fixed with a regression test and a revert proof.
External validity against human labels
The frozen regular expression beat every jury configuration. Specificity 0.9729 on 1,364 items.
Secondary jury matrix complete
16 of 16 cells, 70,738 votes, matching plan on every cell, resume proven across eight of them, provenance recorded against a snapshot of the instrument.
The 72-cell expansion
All 72 fits and all 72 posteriors landed in 1 hour 34 minutes with zero cell failures. The model rows were resubmitted after a driver bug pointed the results root at the wrong tree. The branch sits two commits ahead of main, unmerged.
The fits passes
Three passes at growing scale. The numbers moved very little; the claim status moved down.
Inventory of the sweep
72 completed nine-arm cells, every one exit 0 with a non-empty summary, across six subjects at 12 cells each, being three substrates crossed with four cue families. Eight of the 72 sit below the 350 clean-correct precision figure and are labelled accordingly rather than dropped.
| subject | cells | below the 350 figure |
|---|---|---|
| qwen3-8b | 12 | 0 |
| gemma-2-9b-it | 12 | 0 |
| phi-4-reasoning | 12 | 0 |
| llama-3.1-8b-instruct | 12 | 2 |
| olmo-3-7b-think | 12 | 2 |
| deepseek-r1-distill-llama-8b | 12 | 4 |
| deepseek-r1-0528-qwen3-8b | 0 | held tokenizer round-trip |
| gpt-oss-20b | 0 | held unservable under the pinned mode |
The 350 figure is a precision figure, not an admission gate. An underpowered quantity is reported with its interval and labelled underpowered, which the frozen text states three times.
Model rows, 24-cell pass
Hierarchical hyperparameter posteriors across a model's cells, never an average of cell estimates. All PROVISIONAL, all UNRESOLVED.
| model | NDE | NIE | TE | P(NIE > 0.15) |
|---|---|---|---|---|
| qwen3-8b | +0.1112 | +0.0316 | +0.1438 | 0.006 |
| gemma-2-9b-it | +0.1242 | +0.0610 | +0.1870 | 0.076 |
| llama-3.1-8b-instruct | +0.0955 | +0.0169 | +0.1135 | 0.005 |
Eleven of 24 cells clear the anchor margin on all three of their own estimands. All eleven are still held at RAW, because claim status also turns on the model-level comparison, which no row clears. Since no row resolves, the contract forbids a model-level rho sweep and none is printed.
Two readings that constrain how the numbers may be used
Mean clean-arm curve area runs 0.9383 to 0.9742 on the 12 ARC cells and 0.4989 to 0.6422 on the 6 AQuA cells. The mediator has a different distribution on the two substrates, so coefficients must not be read across them. And the pre-registered grouping factor was cue family, while the measured variation lives in the substrate: cue-family spread medians 0.7886, 0.6874 and 0.9077 against substrate medians 3.3232, 2.9602 and 2.9360.
Three estimator defects
All three were found in one week, two of them by the project's own machinery after the first was found by someone outside it. Each repair shipped with a regression test that is proven still able to fail.
Defect 1, found 2026-09-06 by the external reviewer
No baseline intercepts. The specification had no intercept terms, so on a world built with exactly zero true effect the estimator reported a mediated effect of +0.2131 and a total of +0.2697. After the repair the same null returns −0.0002 and −0.0082 against a randomized arm difference of −0.0082.
The finding is not really about intercepts. Twenty-two internal review voices checked the equations against the prose and passed the plan. None ran a known-zero world through the code. The rule that came out of it is that any statistical component now ships with a null test that must pass before any real fit, and the verifier of that component runs the null itself.
Defect 2, found 2026-09-07 by an internal audit lane
A link mismatch between the two estimator paths. The sampler path was logistic while every other path was probit. rho is defined as a correlation between bivariate normal errors, which only exists in the probit model, so the two paths were not estimating the same thing. The gap measured about 0.04 on an effect thresholded at 0.15: not large, and large enough to move a verdict.
Defect 3, found 2026-09-07 by the mechanism battery
Fixed-scale outcome priors. Interval coverage on a known truth ran 7 of 20 and 12 of 20 before the priors were rescaled to the outcome, and 20 of 20 and 19 of 20 after, at n = 350. A prior that does not scale with the quantity it is a prior over produces intervals that are confidently wrong.
The three defects share a shape. Each was invisible to review of the specification and visible immediately to a run against known truth. That is the argument for the synthetic battery existing at all.
The jury
The design called for an LLM panel to read transcripts and answer two questions: whether the model mentioned the planted cue, and whether the reasoning supports the answer. Ten thresholds were frozen before any judge ran. The instrument has not passed them.
What was tried
Four judges, four prompt files, panels in several compositions, at 1,024 tokens on pinned serving lines. No panel configuration clears the ten bars on any prompt file in any composition. One single-judge configuration does clear all ten, and its exploratory twin agrees with it on every verdict.
Three of the four judges fail on the best prompt file. The two that fail on the contested classes fail in opposite directions on the same bytes, which is the part that makes a panel vote unhelpful rather than merely noisy: averaging two instruments that disagree about direction does not produce a better instrument.
What the external check said
Against human labels from an external benchmark, the frozen regular expression beats every jury Q1 file by 4 to 52 specificity points. Its own specificity is 0.9729, being 1,327 of 1,364, with 37 false positives. The jury's recall against human unfaithfulness labels is 26 of 365, precision 26 of 37.
So the cheap instrument beats the expensive one on the only data with human labels attached, and the column of record stays the regular expression, uncorrected, until a judge earns otherwise.
The secondary matrix
16 of 16 cells, 70,738 votes against 70,738 planned on every cell, every job exit 0, resume proven across eight cells skipping 32,730 already-cast votes, and provenance recorded against a snapshot of the instrument rather than a branch name. It is banked and verified. It has not yet been turned into a single reported number, and saying so is more useful than printing a provisional one.
Mechanism battery
Eleven synthetic generator families with known ground truth, run as 15 conditions over 3,800 seeded datasets, each fitted once and refitted 200 times by bootstrap. This is where the project measures what its own estimator gets wrong.
Three pre-registered outcomes all pass: direct-bypass invariance, shared cause, and no cue effect. Cross-checks agree to 0.00335 maximum absolute difference on the sweep and 0.00096 on the frontier, and each family's Monte Carlo truth matches its analytic truth to 0.000773 at two million rows.
The finding that argues against the project's own metric
A shared-cause world with zero true mediation and a rationalization world that is fully mediated produce the same report at n = 3,600: mean estimates 0.2613 against 0.2833, rho* 0.7069 against 0.7063, both firing the pre-registered verdict in 100 of 100 datasets. The largest rho* in the whole battery, 0.8428, sits on a third world whose true mediated effect is zero.
rho* answers one question well: how much residual correlation would overturn this verdict. It cannot compare two worlds, which is what any use of it as a trust score would require.
What each named misspecification costs
| family | bias | interval coverage | reading |
|---|---|---|---|
| arm-dependent mediator variance | +0.0653 | 169 / 400 | The expensive one. A cue that widens the spread of the reasoning as well as shifting its level is read as a larger mediated effect, and no amount of data fixes it. |
| sparse groups | −0.0009 | 362 / 400 | An interval problem, not a bias problem. The row bootstrap resamples rows that are not independent. |
| mediator missingness | −0.0108 | 367 / 400 | Complete-case analysis of a parse failure costs about a hundredth. |
| nonlinear depth | −0.0156 | 359 / 400 | Tolerable, and reported. |
Gates and freezes
The point of a pre-registration is not that the rules are good. It is that they were fixed before the numbers, and that moving one leaves a record.
- The verdict rule. A faithfulness claim needs 0.95 posterior probability that the faithful path exceeds a named margin. It was fixed before any model row existed and has not been moved. Under it, every row so far is UNRESOLVED.
- The offset-null gate, with a no-retry rule. Run a world with zero true effect through the estimator before any powered fit. It has run four times, passing on the first attempt each time. A gate you may retry until it passes is not a gate.
- The determinism preflight. Every powered cell re-proves batch invariance on its own server before its arms run, refusing at a distinct exit code otherwise, and the refusal path is itself tested.
- The ten jury thresholds. Frozen before any judge ran. Not met by any panel.
- The anchor margin. A cell is anchored only when all three estimands agree with the randomized replay anchor within 0.10, fixed before any data was seen.
- Amendments leave a record. Each carries a fingerprint, a frozen guard, and a proof that the guard can still fail. A cell never upgrades its state by hand.
The plan-time adversarial gate itself failed, and that is recorded too: a 22-voice review passed an estimator that reported an effect of 0.21 on data with no effect, because every voice checked the equations and none ran the code.
Engineering lessons
Each of these was paid for by an incident. They are here because they transfer to any evaluation run at this scale, and because most of them are invisible failures rather than loud ones.
A check that cannot run must never read as a check that passed
Every serving script used a fixed loopback port and waited on whether anything answered. Slices of one node share the host loopback, so two of our own jobs on one node meant two servers competing for one port, and the second job's readiness probe was answered by the first job's server. One cell was declared up five seconds before its own weights finished downloading.
Cross-model collisions fail loudly. Same-model collisions fail silently: one cell passed its checks through another cell's server, and for eight minutes two clients drove one server at up to 64 requests in flight against a mode pre-registered at 32, while each job believed it had one request in flight.
The rule. The check that matters is the third one: that the process listening on the port is your server's own process. When an ownership check cannot run, log it and record that it was skipped and why.
Cancelling a run must not read as success
Five seconds after each cancellation, the cancelled job's own log printed an exit code of zero with no results file anywhere. The submitter trapped only EXIT, so it read the status variable whenever it regained control, which on the cancel path read zero. A sibling script that trapped the signal explicitly wrote the correct code on its first live cancellation. The same signal, two guards, two answers, which is what made the gap visible as a gap.
The rule. A zero is written only when the run's own completion marker exists and the command returned zero. A zero with no marker is a different code. An explicit signal trap records the signal.
The last field of every manifest row was silently dropped
A shell pipeline wrote a stream with no trailing newline into a read loop. The loop returns non-zero on a final line without one, so the body never saw the last field. Every wave ever submitted lost its last setting.
It was harmless for a day, because that column held planning metadata nothing reads. It became load-bearing the moment a real field went last: one job then wrote into the wave-1 cell of record, and a set of lifted rows would have run in a rejected mode and recorded that truthfully. The audit over every submitted wave found 42 rows, 37 informational, 5 load-bearing, 1 actual incident.
The rule. A read loop over a stream with no trailing newline is a bug on sight. Never make a load-bearing value the last column until the submitter proves it exports the last column.
A liveness probe in the hot path reads a busy server as dead
The jury runner probed the model list lazily per judge and seed from twelve worker threads, cached only successes, and aborted the whole run on one failure. The server was answering; it downloads a vocabulary inside its first request handler, so the event loop was blocked and one probe timed out. Thirty of about four thousand votes were written, and the job recorded exit zero, because a traceback exits 1, which is also what a failed threshold returns, and nothing checked for the report file.
The rules. A liveness probe belongs at startup, once per dependency, never in the hot path. Unreachable must not be concluded from one timed-out request to a server known to be loading. An exit code that means two different things is not an exit code. And a zero needs a positive completion marker, which was the third time this campaign paid for that rule.
Three ways a test can prove the wrong thing
A fixture can freeze a platform artifact. A lane captured a before-and-after fixture on macOS, where the system shell is old enough to mangle a JSON default written inside a parameter expansion. The fixture recorded a missing file where the runner produces a full one, so it had frozen a platform quirk as the truth.
A bit-for-bit float pin is a same-platform claim. Two tests pinned coefficients as hex floats recorded on one architecture. Another reproduced them to 6.2e-7 relative and failed the equality. Nothing about the specification had changed; the linear algebra library had. Determinism is pinned on one platform; recorded values are pinned across platforms at a stated tolerance, proven to fail just outside it.
One environment, many worktrees. A lane installed its own worktree in editable mode into a shared environment, which re-pointed the package for everybody. The loud symptom was two import errors. The silent version is a green run that tested a different tree. Before trusting a test run as evidence about a checkout, print the package's file path and require it inside that checkout.
Run a null through the code before you trust the equations
This is the shortest lesson and the most expensive one. A 22-voice plan gate passed an estimator that reports a mediated effect of 0.21 on data with no effect, because every voice checked the equations against the prose and none ran a known-zero world through the code. An external reviewer did, in seconds.
What is next
In dependency order. The binding constraint is people, not compute.
- Collect the 72-cell model rows and claim statuses, generate the report, and merge the expansion branch into main.
- Turn the 70,738 banked secondary-jury votes into a reported number, or record why they cannot be.
- Recruit raters and run the golden set. Until a human label exists, no corrected label column can be computed for any cell, and neither kill switch has ever been evaluated.
- Freeze the validation organism specification, then train the twelve checkpoints. Nothing can reach validated before this.
- Resolve the anchor-versus-fit disagreement: either the fit is losing a real effect, or the anchor contrast is not measuring what the estimand defines.
- Close the provenance gap by writing the instrument snapshot into every run output, so a result can be traced to its code without knowing which tree produced it.
- The held-out audit decision, the residual-stream probe, and the paper.