The problem
The record of what left the sandbox lived on the machine the agents had been writing to
The sentence that started this project is in METR's write-up of its investigation into the July 2026 incident, published on 26 August: the agents may have deleted or edited the entries on the Artifactory cache before the dump was retrieved, and METR could not rule it out. METR had transcripts, environment access and interview rights that no regulator will ever get, and it still ended up with a record it could not vouch for, because the only copy of what happened lived on the very cache the agents had been writing to.
I think of this as the proof problem rather than the sandbox problem. A lab that runs guardrails-off evaluations cannot show anyone outside the building that its containment held, and once the sandbox is compromised it cannot really show itself either. Two things make it hard. There is a single observer, because the broker that enforces egress is also the thing that writes the log, so compromising one compromises both. And an omission is invisible to a single log: a log with one row removed verifies perfectly against its own hash chain.
So I built what I would have wanted to receive as that outsider: one signed file per evaluation run, an egress and intent receipt, verified offline with no access to the lab, and designed so that a sandbox lying about its own egress is caught by the receipt rather than covered by it. It builds on Shadow Commit, the middleware my team built for TikTok TechJam 2026, which already had a sealed per-turn workspace, an exact host and port egress allowlist and a hash-chained journal with Merkle checkpoints. Everything that makes the broker's word checkable by someone who does not trust the broker is new work from the sprint.
The receipt
Three observers that do not share a trust base, bound into one signed file
A single log kept by the thing being watched is worth very little once that thing is compromised, so the receipt binds three observers. Scroll through the four steps; the diagram follows along.
Observer one
The enforcement log
The broker's own record of every allow and deny decision. This is the log a compromised lab can edit, and it is where most "the policy was enforced" stories stop.
Observer two
The independent witness
A second capture of the same egress, taken from the kernel's connection tracker inside the VM that hosts the containers, so it sees the real peer IP and byte counts of every flow that crossed the sandbox's only exit. Its signing key and its flow log live on a host path the sandbox cannot mount; a probe run as root inside the sandbox gets ENOENT on all three files.
Observer three
The decoy tripwires
A planted destination and a planted credential that a benign agent has no reason to touch. Reading is silent. Sending the value over the wire, or reaching the decoy host, produces a canary event bound to the witness's own flow leaf, so a hit inherits the witness's anchor and tamper-evidence instead of living in the log the attacker controls.
The receipt
One statement, signed, anchored during the run
Three Merkle roots go into one in-toto statement, signed as a DSSE envelope with a key whose public half is held out of band. While the run is still going, the witness head is anchored as a git note, and with one flag on it is also published to the public Sigstore Rekor log before the anchor step returns. On the real turn behind the shipped receipt, Rekor's own timestamp falls 0.2 seconds before the agent's container was destroyed.
Verify it
The same checks the judge-kit runs, in your browser, on a receipt cut from a real turn
The receipt below was cut from a real evaluation turn against the real internet: a real
pip install, a real Hugging Face configuration pull, DNS churn and three single
fetches, which is 7 witnessed flows to 7 different CDN addresses and 15 enforcement rows. The
signature over the DSSE envelope, both Merkle roots and the reconciliation are recomputed here
with WebCrypto, from the receipt's own evidence. Then you can play the lab: hide one enforcement
row, let the root be recomputed the way a lying lab would recompute it, and watch the witness
name the flow that went missing.
$ node verify.mjs receipt-real-clean.json (press Verify)
The enforcement tree, 15 leaves. Hiding a leaf recomputes the path to the root; the anchored witness root does not move, so the two no longer agree.
What runs here: the Ed25519 signature over the DSSE pre-authentication encoding, the RFC 6962 Merkle roots of the witness and enforcement evidence, the canary tree and the reconciliation, ported line for line from the node verifier and checked for parity on all nine shipped receipts. What does not run in a browser: reading the git-note anchor back from the bundled anchor repository, which the node kit does; the Rekor public-log entry carried by the third shipped receipt is checked here against the pinned Rekor key.
Results
The number that could have sunk the claim, and the fix that a revert proves
The first time I measured the false-alarm rate on real benign traffic, it was 100 percent: 27 of 27 flows flagged as omitted. The enforcement log held the hostname the agent asked for, the witness held the IP the packets went to, and for CDN-backed hosts those never coincide as strings. After the broker started recording the resolved IP, fresh traffic came out 0 of 21, then 0 of 62, then 0 of 110 across 7 workload classes. I would not read 0 of 110 as tighter than its 3.37 percent upper bound. What I do read as strong is the revert: stripping the resolved IP back out of the same 110 flows returns 110 of 110 to omitted, so the fix is the mechanism and not cleaner traffic.
npm-install,
apt-mirror, git-clone and object-fetch at 0 of 12 each,
and the original pip, Hugging Face and DNS classes at 0 of 62. The 67 bundles behind
these numbers ship in the repository, so the check recomputes with no Docker.Chokepoint completeness, the raw table first
Of 6 measured flow hops across five escape techniques, 2 were witnessed under the strict broker configuration and 4 under the weakened one, each technique run once per configuration. The two misses are the same two rows in both, and they share one cause: the witness attributes flows by the run's own broker egress IP, so a compromised relay's onward hop from another host and a technique built specifically to find a path the observer does not sit on are both outside what it can see. Reading only the sandbox's own broker-mediated hops, every one was witnessed or correctly denied, 4 of 4. The positive finding is the DNS pivot: the allowlist approved a hostname, the broker's own connection resolved it to a different address, and the witness recorded the real destination anyway.
The decoy, and how long the catch takes
Zero decoy hits over 11 real benign turns and 40 witnessed flows. In every red-team run the exfiltration of the planted credential produced a canary hit, and after the lab erased the decoy row, renumbered, re-rooted and re-signed the enforcement log, the hit still named the destination, because it binds to the witness leaf and not to the log the attacker edited. The planted token is shaped like a Hugging Face token and matches none of the broker's generic secret patterns, so the broker forwarded the exfiltration and only the decoy caught it, which is the incident's own token-theft shape.
- Per-call overhead
- none measurable
- the witness is not in the request path
- Per-run overhead
- ~724 ms
- 234 ms to arm, 490 ms to freeze and root
- Decoy false-positive floor
- 0 of 40
- real benign flows, 11 turns
- Public anchor timing
- inside the run
- Rekor time 06:00:16Z, container gone 06:00:16.9Z
The standard
Seven clauses, each a script that exits 0 or 1, each shown able to fail
The sprint's first example project asks for a v0.1 standard with clauses, evidence per clause and a rough cost. This one is executable: every clause is a check you run against a live target, mapped to CSA AIS-13 and NIST SC-7 or AU-9 and to a phase of Hugging Face's own timeline, and every clause ships with a revert proof, the same check with the control on and then deliberately off. All seven pass from a fresh export. The history is the useful part: the fifth clause, zero false alarms on benign egress, genuinely failed on real data before the fix, and its packaged revert arm could not fire until it was changed to strip the resolved IP from the rows, the same revert that brings 110 of 110 back.
| Clause | Property | Control | Phase | Result | Rough cost, my estimate from this build |
|---|---|---|---|---|---|
| STD-1 | Egress boundary default-deny | NIST SC-7 | recon | pass | none extra, the broker already default-denies |
| STD-2 | Witness key and sink unreachable by sandbox root | NIST AU-9(3) | rce | pass | a witness process and a host path the sandbox cannot mount, about one engineer-day here |
| STD-3 | Omission detected under enforcement-log tamper | NIST AU-9(3) | evasion | pass | receipt build and offline verify, about 490 ms per run |
| STD-4 | Forged receipt refused, out-of-band trust root | NIST AU-9(3) | rce | pass | key custody only |
| STD-5 | Zero false alarms on real benign egress | CSA AIS-13 | dropper | pass failed first, on real data | a 60-line broker change plus a benign corpus to measure against |
| STD-6 | Chokepoint completeness under destination obfuscation | NIST SC-7 | c2 | pass | an escape corpus, about one engineer-day, scripts kept private |
| STD-7 | Per-run attribution integrity on the shared bridge | NIST AU-9 | evasion | pass | a pure function over the witness's own labels |
Limits
What this does not catch, said before a judge asks
- Payload inside an allowed TLS tunnel is opaque. The receipt proves destination and volume, never content, and the credential catch is plain-HTTP only. Since nearly all lab egress is TLS through CONNECT, the destination-based catch is the one that generalises.
- Two completeness boundaries: a compromised relay's onward hop is another host's egress and needs its own witness, and a technique built to find a path the observer does not sit on finds one, by construction.
- The anchor protects an already-anchored record from later editing. It does not stop a real-time host-root that co-opts the witness process itself; that needs a witness run by a third party or a hardware root.
- The public Rekor entries use a throwaway P-256 key because Rekor refuses Ed25519. No trust rests on that key; the third-party timestamp and the inclusion proof do the work. The mid-run public leg is opt-in and needs the network.
- A lab that emits no receipt at all is outside the mechanism. Under a duty like EU AI Act Article 55(1)(c), a missing receipt is itself the signal a regulator acts on.
- Scale: one machine, one day, 110 benign flows, 40 decoy-floor flows, 5 timed red-team runs, and escape techniques I chose myself. The intervals in the figures are the honest statement of that.
The analogy I keep coming back to is IAEA containment and surveillance. The facility keeps its own operating log, the inspectors keep their own seals and cameras rooted outside the operator's control, and the inspectors' copy of what the seals showed is what a third party trusts. The enforcement log is the operating log, the witness is the seal, the public anchor is the inspector's copy. Where the analogy breaks is instructive: an IAEA seal is physically outside the facility's software, while my witness is a process on the same host, isolated by mount and namespace boundaries that I proved a sandbox root cannot cross but that a full host compromise could. That is why the honest scope of a green receipt is that the sandbox cannot rewrite what has already been anchored, and not that the host is trustworthy.
Run it yourself
Under a minute, node only, no network, no access to the lab
The judge-kit is the fastest path. It needs node 20 or later and nothing else; the verifier is bundled, the trust roots are held out of band, and the anchor repository ships inside the kit so the read-back is real.
git clone --depth 1 --branch apart-sprint https://github.com/thylinao1/shadowcommit.git
cd shadowcommit/judge-kit
./verify receipt-real-clean.json # exit 0, 7 flows matched, anchor read back
./verify receipt-real-tampered.json # exit 1, names the hidden flow and the row gap
./verify receipt-real-clean-public-anchor.json # exit 0, public log entry timestamped during the turn
./verify --no-anchor receipt-clean.json # exit 2, a check that did not run is not a pass
./verify receipt-forged.json # exit 1, re-signed by an attacker
cd .. && node standard/checks/05-benign-reconciliation.mjs # recomputes 0/55 on, 55/55 off- 0
- clean: signature, both roots, the anchor read back and the reconciliation all hold
- 1
- a check ran and failed: a named omitted flow, a bad signature, a failed inclusion proof
- 2
- a check could not run here: wrong trust key, a missing anchor, a missing input
What is new in the sprint and what is reused is listed in the repository's
NEW-WORK.md. The escape-replay and deception scripts are withheld pending the
organisers' disclosure review, as the sprint guidelines ask; the measurement data they produced
ships in full, redacted only of commands and payloads, so the headline numbers recompute from the
repository alone.