01 / The problem
A network with random weights still has a definite answer
Nothing here is learned. The weights are drawn from a fixed distribution and left alone. The question is what the network does on average, and whether you can work that out faster than by trying it.
Take a plain multilayer perceptron with activations and no bias terms. Feed it a random input drawn from a standard Gaussian. Each layer multiplies by a weight matrix and then clips everything negative to zero. Repeat thirty-two times.
Every individual run gives a different answer, because the input is random. But the average over all possible inputs is a fixed number for each neuron. That number is what you have to predict:
In Phase 1 the network has L = 32 hidden layers of width n = 256, so the answer is a 32 by 256 matrix. Weights use He-Gaussian initialisation with variance 2/n, stored as float32, and there are no biases at all. Only the final row, layer 32, is scored for the prize. The other rows are a secondary diagnostic.
The handicap that makes it a competition. The answer key was produced by brute-force Monte Carlo at roughly 4.24e15 floating point operations per network. You are given 2.72e11. That is about one fifteen-thousandth of the budget that produced the target you are being compared against.
So the benchmark is not really asking you to compute an average. It is asking whether understanding the structure of a random network lets you beat brute force at a fifteen-thousand-to-one disadvantage. That framing is the whole design of the thing, and it is why the scoring rule matters more than the mathematics.
Signal collapse across 32 layers animated
02 / The obvious method
Sampling works, and it is charged by the sample
Draw inputs, run them through, average the results. It converges. The only question is what it costs, and the competition answers that question very precisely.
The straightforward estimator is . Draw N random inputs, push all of them through the network, and average the post-ReLU values at the final layer. The error of that average falls as 1/N, so the root mean squared error falls as one over the square root of N. Getting one more digit of accuracy costs a hundred times more samples.
There is a smarter version. replaces random draws with a carefully constructed point set that covers the space more evenly than randomness manages. The classic construction is a , and randomising it by keeps the estimator unbiased while preserving the even coverage. On smooth problems this converges considerably faster than 1/N.
Figure 2 races the two. The left panel shows where the points actually land. Random points clump and leave gaps, which is what randomness does. The Sobol points fill every region at every scale, which is what they are designed to do. The right panel shows the error of the running average.
Random points against a digital net animated
A number worth remembering. The grader reports a constant called sampling_mse equal to 6.469470189211361e-07. That is what an ordinary independent-draw Monte Carlo estimator scores when it spends the entire budget. Everything on the leaderboard is measured against that reference.
03 / The metric
Accuracy is multiplied by what it cost you
This is the rule that decides everything else. Read it slowly, because almost every wrong turn in this competition comes from reading it quickly.
For each network m, the score is the final-layer mean squared error multiplied by the fraction of the compute budget you consumed:
The overall score is the mean of sm over the networks. Lower is better. Three pieces need unpacking.
C, the compute you actually spent
F is the analytic count that the metering library charges you for every arithmetic operation you issue. R is residual wall-clock time, meaning time spent in your own Python between metered calls. One second of un-metered Python costs the same as a hundred billion metered operations. Several participants measured that the residual charge behaves as a tax per function call rather than per unit of arithmetic, which makes the number of calls a design variable in its own right.
B, the budget
2.72e11 FLOPs per network in Phase 1. For scale, one honest dense forward pass of a single input through the whole 32-layer network costs about 4.19e6 FLOPs, so the budget buys roughly sixty-five thousand honest forward passes.
The floor at 0.1, and the cliff at 1.0
The multiplier is max(0.1, C/B). Spending less than ten percent of the budget does not reduce the multiplier any further, so roughly 2.72e10 FLOPs are effectively free. Above the floor the multiplier is not capped at one. The grader's own source describes it as "uncapped above". And going over budget entirely is a cliff rather than a slope: when C > B the grader zeroes your predictions for that network.
This cliff was not always enforced. Several participants documented that the combined-budget check did not fire on the evaluator until around 9 August 2026, and that older submissions were then re-graded retroactively. One submission that had been reading in the region of 4e-8 per network became an aggregate 0.682 after the patch, with 49 of the public networks failed. Anyone reading old leaderboard snapshots should know that they were produced under a different rule.
If you are reading the challenge page, its numbers disagree with the grader's. At the time of writing, the public challenge page describes the score with a floor of max(0.5, C/B), eight hidden layers, and a budget around 3.4e10. Every number on this page instead follows the installed grader and starter kit, which give a floor of 0.1, thirty-two hidden layers, and 2.72e11. My own graded submission records a multiplier of 0.4401, which is below 0.5 and therefore could not exist under the page's stated floor, so the grader is the operative source. This looks like the page carrying warm-up-round text that was never updated. Worth knowing before you calibrate anything against it.
The compute multiplier animated
04 / The invariance
For a sampler, drawing more samples changes nothing
This is the single most useful thing to understand about the competition, and it takes two lines of algebra.
A sampler has error that falls as one over the sample count, and a price that rises linearly with it:
Substitute both into the score:
The N cancels. Above the floor, the score of a sampler does not depend on how many samples it draws. Drawing more buys accuracy and pays for it at exactly the rate the metric charges. You can move along that line in either direction and arrive at the same number.
Move the slider in Figure 4 and watch it happen. The error falls, the cost rises, and the product sits still.
The score does not move with sample count drag the sliders
- Raw MSE
- 1.72e-07
- Budget used C/B
- 0.80
- Multiplier
- 0.80
- Score
- 1.38e-07
Two honest caveats
The exponent is not exactly one. Measured on my own split, the empirical scaling exponent is 1.0653: error 2.0156e-06 at N = 13,312 falling to 8.4253e-08 at N = 262,144, a factor of 23.92 over a sample ratio of 19.69. Because the exponent sits slightly above one, raising N is very slightly score-positive rather than exactly neutral, which is why my build spends every row it can afford.
Somebody else measured the same thing independently. Participant violeta ran a four-point paired ladder and reported the product of error and effective compute as 8.180e4 / 7.554e4 / 8.350e4 / 7.448e4 FLOPs across four batch sizes. That product is the quantity the algebra says is conserved, and it is conserved to within about ten percent across an eightfold change in sample count.
What this rules out
Brute force is mathematically dead, and you can price the death exactly. Reaching a raw error of 8.40e-08 by sampling alone would need N around 262,900, which drives budget consumption to 1.797 and the score to 1.509e-07. That is worse than the estimator I already had, while also being over budget. No amount of patience gets you there.
What is left is exactly two quantities. Lower a, the variance produced per sample, or lower p, the price billed per sample. Everything interesting anyone did in this competition is one of those two things.
05 / The analytic route
The baseline everybody starts from
If the input to a neuron were exactly Gaussian, the expected value of its ReLU would have a closed form. You could carry a Gaussian description forward layer by layer and never sample at all.
That idea is called , and the starter kit ships a working version of it. Here is the loop, from examples/02_mean_propagation.py:
Reading it line by line
The first two lines are exact. A standard Gaussian input has mean zero and variance one in every coordinate. The line mu_pre = w.T @ mu is also exact, because a linear map moves a mean exactly.
The approximation enters at var_pre = (w*w).T @ var. The true pre-activation variance is the diagonal of WT Cov W, which needs the entire covariance matrix. Squaring the weights and pushing only the diagonal through gives the same answer only if the incoming neurons are uncorrelated. After the first layer they are not.
The line that computes mu is the exact expectation of ReLU(Z) for a Gaussian Z with that mean and standard deviation, where Phi is the normal cumulative distribution function and phi its density. That formula is not the weak point. All of the error comes from the assumption that the thing entering the ReLU was Gaussian in the first place.
What a ReLU does to a Gaussian animated
The two published baselines
| Method | Final-layer MSE | Note |
|---|---|---|
| Mean plus diagonal-variance propagation | 9.5e-04 | The starter kit code above |
| Full covariance propagation | 8.4e-05 | Tracks the whole pre-activation covariance, about 11 times better |
| Plain Monte Carlo at full budget | 6.47e-07 | The grader's own sampling_mse constant |
I reproduced the full-covariance figure independently and got 8.66e-05 evaluated held-out on 1000 networks, which agrees. Note the ordering carefully. Plain Monte Carlo at full budget is about 1,500 times more accurate than the diagonal closure, and about 130 times more accurate than full covariance propagation. The closure's only advantage is that it is far cheaper, which under this metric is a real advantage, but it is not an accuracy advantage.
06 / Why it breaks
Thirty-two layers destroy the assumption
The closure is not slightly wrong at depth 32. Establishing exactly where it goes wrong took most of my campaign, and the answer changed what I built.
Split the error into two parts. Readout error is what you lose in the final step, turning the last layer's pre-activation distribution into an expected ReLU. Propagation error is what you lose in the preceding 31 steps, carrying the distribution forward.
You can measure them separately by cheating in a controlled way. Hand the estimator the true quantity it would otherwise approximate, leave everything else running, and see what error remains. That gives an upper bound on every possible improvement that works through that quantity, including ideas nobody has thought of yet. Participant trim_qewas published the same protocol independently and used it to close 54 families of estimators without building any of them.
| Oracle experiment | Residual error | Reading |
|---|---|---|
| True final-layer mean, standard deviation, skewness and excess kurtosis, plus a fourth-order Edgeworth readout | 2.2e-08 | The readout is essentially solved |
| True per-layer marginal mean and standard deviation after every ReLU, keeping the closure's own correlations | 2.11e-06 | 500 networks, replicating 2.00e-06 on a separate 150-network instrument |
| The same, plus a true-cumulant Edgeworth readout on top | 7.07e-07 | Still four times worse than my deployed sampler |
Put those side by side. Perfect readout given perfect moments reaches 2.2e-08. Perfect marginals at every single layer, with only the correlation structure left approximate, cannot get below 7.07e-07. The gap between them is the dependence structure the closure discards, and it is roughly thirty times larger than everything the readout could ever contribute. The readout is about 3 percent of the chain error.
Participant amalgonim reached the same conclusion by a different route, with a permutation-shuffle oracle: give the estimator all 256 exact marginal distributions but force it to assume independence, and the error lands at 9.2643e-07. Exact marginals with no dependence information are worth almost nothing. The dependence is the problem.
What actually happens to the correlations
They do not decay. They saturate near one. Iterating the ReLU angle map
from the most extreme possible starting point, two exactly opposite inputs at c = -1, gives 0 after one layer, then 0.318 at layer 2, 0.605 at layer 4, 0.809 at layer 8, 0.923 at layer 16, and 0.972 at layer 31. Measured directly, the cosine between h31(x) and h31(-x) is 0.982.
That measured number and the theoretical one are not quite the same quantity, which is worth stating because the discrepancy runs the opposite way to how it first reads. The 0.982 is an average of per-pair cosines, while the map predicts a ratio of expectations, E<h, h'> / E||h||². Re-measured on ten networks at 131,072 samples each, the average of cosines has a median of 0.979 and the map's own quantity a median of 0.941, against 0.973 from the iteration. So the map slightly overpredicts the collapse rather than underpredicting it. The conclusion is unchanged and the correction is small, but the sign of the residual matters if anyone builds on it.
Two inputs as different as two inputs can possibly be end up 98 percent aligned by the layer that gets scored. Team SOX measured the consequence on this exact architecture class: 73 dead neurons, 106 kink neurons, 77 always-on neurons, at an effective rank of 2.2.
That 2.2 is a participation ratio, (Σλ)² / Σλ², and it is worth being careful about what it licenses. A participation ratio is dominated by the largest eigenvalue, which on my own re-measurement carries a median 0.64 of the centred variance at the scored layer, and it says very little about how far the rest of the spectrum reaches. Spectra with a participation ratio of 2.4 exist whose 99 percent count is anywhere from single digits to over 200.
On the same ten networks, four defensible summaries of the same layer-32 spectrum disagree by more than an order of magnitude: participation ratio 2.4, the entropy-based effective rank of Roy and Vetterli 5.9 on the covariance eigenvalues and 64 on the data matrix singular values, and stable rank 1.6. Reaching 99 percent of the variance takes a median of 56 directions out of 256, and 99.99 percent takes 155. The collapse is real and large. It is not a two-dimensional object, and an estimator sized from the participation ratio would be roughly twenty times under-provisioned.
Two opposite inputs converging animated
One more measurement worth knowing, from participant Cipo, who is one of the few who predicted every layer rather than only the scored one. Their error profile grows from 2.940e-09 at layer 0 to a peak of 6.709e-05 at layer 9, decays to 2.217e-05 by layer 30, and then the scored layer comes back down to 1.964e-07. The interior is where the damage happens, and the scored layer is anomalously easy compared to the layers before it.
The design conclusion follows directly. If roughly 99 percent of the achievable error lives in propagating a 256-dimensional joint distribution through 32 ReLUs, and every cheap surrogate for that propagation is measured to fail, then the budget is better spent on samples, which propagate the true distribution exactly, one draw at a time. That is the decision my estimator embodies.
07 / What I submitted
A quasi-Monte Carlo sampler, and machinery to make it cheap
The estimator itself is simple to state. Everything else in the build exists to lower the billed price of one sample, because the algebra in section 4 says that is one of only two quantities that can move the score.
The estimator draws N = 98,304 rows of standard Gaussian input, pushes all of them through the 32 layers exactly in float32, and returns the mean of the final layer's post-ReLU columns. No closure, no correction, no learned component in the scored path.
The point set
The 98,304 rows are six independently scrambled Sobol nets of 214 points each in 256 dimensions, mapped to Gaussians through the inverse normal distribution function with a clip at 1e-7. Six independent scrambles rather than one long stream, because independent replicates give an error estimate for free. A paired measurement made afterwards put the convergence exponent at 0.998 for the six-block construction against 0.994 for a single scrambled net and 0.975 for plain independent draws, which says the split costs nothing in rate rather than that it wins.
The point set ships as a precomputed float16 asset of exactly 48.00 MiB, against a 50 MiB package cap. It is loaded and upcast to float32 inside setup(), which runs outside any budget context and is therefore charged to nothing. The float16 round trip costs a measured error ratio of 1.000662, which is inside noise.
Three point sets and their box counts animated
The cost machinery
An honest dense forward pass bills about 4.19e6 FLOPs per sample. The shipped build pays about 1.16e6. About half of that gap is the cost-model defect described below rather than a real saving, so the legitimate stack is roughly 1.9 times. It is assembled from these pieces:
| Component | What it does | Measured effect |
|---|---|---|
| Only the scored row | Layer 32 is the only row that counts, so the 31 interior rows are one reused zero vector. They are never computed. | large |
| Winograd-Strassen, two levels | Fast matrix multiplication, seven products instead of eight per level, recursing only while the halved width stays at or above 64. | -8.4% |
| Exact dead-column pruning | A neuron observed dead across every sample cannot contribute, so its column leaves the next contraction. Zero dead columns in the first three layers, about 70 of 256 by the last. | exact |
| Pilot-predicted pruning | Run 472 pilot rows first, classify a neuron dead if its largest pilot pre-activation sits below a margin, and never compute it for the other 97,832 rows. | -7.86% |
| Graded sparsity staircase | Columns that only a few rows reach are computed over just those rows, in up to four tiers chosen by dynamic programming. Skipped rows contribute exact zeros. | -7% |
| Delete instead of gather | The meter prices a fancy gather at 4.0 per element and a delete at 1.0, so selecting columns by deletion is four times cheaper for the same result. | -1.47% |
| Certified always-on columns | Because the family has no biases, a final-layer neuron that never switches off has an exact pathwise identity: its answer is one dot product against the previous layer's row sum. | +8.9e-06 relative error |
| Odd-width zero padding | Column pruning leaves sub-blocks with odd widths, which stops the fast multiplication recursing. Appending zero columns keeps it firing. Zeros are billed at nothing and the slices that read them are free, so the padding costs only the small operand. | keeps the 8.4% alive |
| Call-count reduction | Residual wall time is charged per call into the metered library, not per unit of arithmetic. One sort replaces six calls, the division by the sample count happens once at the very end rather than per layer, and the planning decisions are memoised by shape so they are computed once per geometry instead of once per layer. | 2,123 to 2,139 calls per network |
| Pilot-prefix hygiene | The rows are re-sorted for sparsity, which would hand the pruning pilot an arbitrary block rather than the true low-discrepancy prefix. A marker folded into the sort key at layer 3 keeps the pilot reading the rows it is supposed to read. | removes a selection bias |
| Fallback chain | If the asset is missing it falls back to a plain matrix product at 6,144 rows; if a preflight check rejects, to a reshape-padded path at 13,312 rows; if a primitive fails mid-forward, the lever is disabled and one forward is recomputed. | never fired in grading |
Every shipped constant
For anyone wanting to reproduce or argue with the build, these are the values that actually shipped rather than the ones in the code comments, which describe predecessors:
One item is not a legitimate discount, and I am naming it. The build padded its arrays so that each contraction exceeded a rank threshold in the metering library, which routed it to a shape-only estimate that billed 0.500978 times the honest charge. That is a cost-model defect, not a mathematical saving. I reported it to the organizers, and when offered the choice between having the model patched with my rank possibly falling, or leaving the submission as graded and deferring to review, I elected to have it patched. The honest number for my estimator is the one in the next section, and it is roughly 1.8 times worse than the one currently on the public board.
What the estimator is not
It is not unbiased, and the write-up says so. Two levers move the estimand. Pilot-predicted pruning can classify a live neuron as dead, which was gated against a threshold and measured at worst at 86.3 percent of that threshold on the hardest public network, with held-out bias statistically indistinguishable from zero. Certified always-on columns were mis-certified on 60 of 2261 columns across 40 networks, always in one direction, contributing +8.9e-06 relative error. Anyone quoting this build should quote those numbers with it.
The pruning cascade animated
08 / Results
Two submissions with identical arithmetic and different prices
| Submission | Score | Raw MSE | Multiplier | Status |
|---|---|---|---|---|
| 325943 | 7.5858e-08 | 1.7238308e-07 | 0.4401 | On the public board |
| 326694 | 1.3820e-07 | 1.7238308e-07 | 0.8017 | The honest twin |
The two differ by exactly ten lines, all of them padding constants. The arithmetic is bit-identical by construction, which is why the raw error is the same to every digit printed. The entire score difference is the mispricing described above. My estimator genuinely consumes about eighty percent of the compute budget.
For orientation, the public board at the time of writing had the leading entry at 1.84e-08 with a raw error of 3.63e-08 at a multiplier of about 0.51. Against my honest number that is 7.5 times better on score and 4.7 times better on raw error, while spending slightly more compute, so it is ahead on both axes at once rather than trading one for the other. Nobody has publicly identified the mechanism.
A caution about reading any leaderboard in this competition. The public split of 50 networks was measured to be an unusually easy draw. Participant keenanpepper ran the same estimator against the public 50 and against a local 512 and got 1.587e-07 against 2.235e-07, a factor of 1.408. A bootstrap put the public draw at the 0.19th percentile of ordinary draws. Because per-network error is heavy tailed, repeated submission also selects for lucky draws, so a long entry list flatters a board score by an amount I have not measured. Prizes are decided on a fresh private re-evaluation, not on the public board.
09 / The measurement
A control variate that works on random draws and returns nothing on a net
This is the one piece of original work in my Phase 1 write-up. It is a negative result, and the reason it is worth reporting is that the null is measured against a positive control in the same script.
A is the standard way to reduce the variance of a sampler. Build a quantity that correlates with what you are integrating and whose true mean you already know, then subtract a fitted multiple of its error. My graded estimator spends its whole budget on the point set and nothing on variance reduction downstream, so the obvious question is whether adding a control on top would have paid.
The experiment runs both arms in one script, on the same integrand, with the same dictionary and the same fitted coefficients. The only thing that changes is the point set underneath.
| Arm | Variance ratio with the control | Reading |
|---|---|---|
| Independent draws | 0.6664 | The control removes 33 percent of the variance. It works. |
| Scrambled Sobol net | 1.0052 | Nothing the instrument can resolve, against a resolution limit near 5 percent. |
The independent arm landing exactly where the run pre-registered it is what makes the null on the net readable. A null with no positive control in the same script is indistinguishable from a broken instrument.
The explanation is an accounting one. The control targets about a third of the variance. The point set has already removed about 69 percent of it, and the third the control is reaching for sits inside that. Measured on the same integrand, the carrier's own gain over independent draws is 3.19×, 3.13× and 3.24×. There is nothing left in that band for the control to take.
Two arms, one script animated
What is classical here and what is not
The vanishing of scramble variance for low-order Walsh functions is not mine. It is Owen's gain-coefficient theorem from 1998, which says the coefficients vanish when the order is below a threshold set by the net's parameters. That a digital net equidistributes every dyadic box of the right volume is the definition of the object, not a measurement.
A caveat I have to state. Owen's decomposition is proved for nested uniform scrambling. My rig uses a linear matrix scramble plus a digital shift. That preserves the equidistribution which forces the low-order means to vanish, but it is not the randomisation the variance theorem is stated for, so the theorem is motivation here rather than proof.
What I claim as mine is narrower: that a fixed, input-anchored control returns no measurable reduction on a scrambled digital net while the same control with the same coefficients returns a third of the variance on independent draws, measured in one script across both arms. Two other participants reached related nulls by different routes, attributing them to shallow controls decorrelating with depth and to estimation noise respectively. For a fixed input-anchored dictionary neither explanation is needed, because the low-order part of the control is empty before any network is involved.
The price a future method has to pay
Both inputs to this are public and I did not find the product stated anywhere, so it is worth writing down. Propagating the readout gain through the published spread of neuron activity: a total final-layer error of 1e-8 requires the terminal mean to be accurate to about 1.4e-4 RMS after the gain, which is 1e-4 RMS on the post-ReLU mean itself.
What that costs
By direct sampling, 1e-8 needs roughly 0.8e7 to 1.7e7 rows at my own implied per-row variance, and about 1.9e7 rows at another participant's published constant. The per-network budget buys about 6.5e4 dense rows, or about 1.2e5 at the billed cost per row my de-padded build achieves. That is two orders of magnitude short.
By propagation instead, it needs a closure roughly a hundred times better than the Gaussian terminal closure, which floors at 9.33e-07 given exact mean and standard deviation, and it sits below the exact-moment Edgeworth readout floor of 1.5e-08 to 2.2e-08 outright.
Another participant asked publicly what compact statistic could transport signed cross-neuron dependence through 32 ReLU layers. I do not have that statistic. What I have is its price, and the price says that neither of the two routes the field has tried can pay it.
10 / Dead ends
What I measured and had to abandon
These are the things I tried that did not work, with the number that killed each one. A measured dead end is worth more to a newcomer than a technique that worked, because it is the part nobody writes down.
| Idea | What killed it |
|---|---|
| Readout-side correctors. Improve the last step, where the final pre-activation distribution becomes an expected ReLU. | Plateaued at 2.5e-05. Section 6 explains why: the readout is about 3 percent of the chain error, so perfecting it cannot matter. |
| Per-layer propagation correctors. Learn or derive a correction applied after every layer. | Failed its oracle-strength kill line by a factor of five, on two independent instruments. The oracle version, given true marginals at every layer, only reaches 2.11e-06. |
| Correcting only the standard deviation. The mean is unpredictable but sigma looked learnable. | Actively harmful. The closure's mean error and sigma error partially cancel, so fixing one breaks the cancellation: 2.75e-04 against an uncorrected 6.9e-05, about four times worse. |
| Antithetic sampling. Pair every draw with its negation. | Capped at 1.10× by the arc-cosine map itself. I attribute the cap to the arc-cosine map; another participant reports a sharper spectral account of it. Note that this result is contested, and the field guide in section 11 flags the disagreement. |
| Input-anchored control variates. Build a control from the input coordinates, fit its coefficients, subtract. | Removes 33 percent of the variance on independent draws and nothing measurable on a scrambled net: variance ratio 1.0052 against a resolution limit near 5 percent. The net had already removed what the control was reaching for. |
| Rotations of the point set, and Owen scrambling variants. | Dead on randomised QMC. Rotation-robust sets could not beat plain Sobol; scrambling differences sat inside measurement noise. |
| A cross-network trained asset. Train one point set or corrector on a family of networks and reuse it. | Flat at 1.015× even with eight times more training data. The gain does not transfer across networks. |
| A third level of Strassen recursion. | Measured +1.28% billed cost, so it made things worse. Another participant independently measured deeper recursion regressing metered compute by 2.383 percent in all three paired repeats. |
| Deterministic propagation with an exact pair kernel. Drop sampling entirely, carry mean and full covariance through all 32 layers using the exact bivariate ReLU kernel rather than an approximated one. | Worth recording carefully, because the kernel was not the problem. The kernel was verified exact to 2.9e-10, and fixing it moved the result from 0.68 to 2.7e-4, a repair of about 2,500 times. The remaining three and a half orders of magnitude are the Gaussian closure itself. By mid-depth the true joint law departs far enough from Gaussian that even exact-kernel propagation accumulates about 1 percent relative mean error by layer 32. Reviving it needs a closure that carries non-Gaussian information, demonstrated at about 1e-4 one-step relative error at layers 8 to 16. |
| Adaptive sample count per network. Spend more rows on harder networks. | Score-negative and shipped inert. A top-up row costs 2.113e6 against the main block's marginal 1.4e6, so the extra accuracy does not pay for itself. Every network runs the same N. |
The pattern in all of them
Every one of these was aimed at the wrong one percent. Section 6 established that propagation, not readout, carries almost all of the achievable error. The corrector family, the readout family and the Edgeworth family all operate on the readout. They were careful, they were measured, and they were pointed at a part of the problem that could not pay.
The one lever that survived every kill is variance per sample at fixed billed cost. It has an existence proof: a rival submission dropped its raw error by 2.31 times while holding billed FLOPs, multiplier, wall-clock, backend and per-network allocation identical to its own control. Whatever they did is real, and I did not find it.
11 / Field guide
What the rest of the field worked out
After Phase 1 closed, a few dozen participants published write-ups on the challenge forum. This is a curated map of what they found, including the parts that contradict my own results. Click any card for the mechanism and the measurement.
Provenance and credit. Every technique below is someone else's work. Each card links the author's AIcrowd profile and the forum topic it came from, and each write-up's detail view also links the graded submission it is bound to. The figures are as those authors reported them, and the full list is in the acknowledgements at the foot of the page. I have not independently reproduced them, and where two published results disagree I have said so rather than picking a winner. Treat the numbers as pointers to the original write-ups, not as settled fact.
A ReLU network without biases is positively homogeneous, so the length of the input can be integrated out exactly and only the direction needs sampling.
Exact identity, no error introduced bgrubbs1984 18169, kaileh57 18171, pranay212 18173, Team Puffi 18175Instead of random directions, use a deterministic set of 66,048 directions built from coding theory that integrates every polynomial up to degree five exactly.
0.389% above the proven minimum of 65,792 nodes andrei_bulzan 18183, kaileh57 18171, keenanpepper 18177, Team Puffi 18175Pairing each direction with its negation lets you reuse work through an exact ReLU identity. How much it is worth is genuinely disputed in the published record.
119× better, 1.10× better, and worse: three published results ely2sh 18176, omer_kiraz 18181, bin_yong_bong 18184Force the drawn batch's own sample covariance to be exactly the identity before using it, which removes the error you can see rather than the error you expect.
2.06× to 2.34× alone, 2.86× with antipodal pairing trim_qewas 18182, amalgonim 18151Run a cheap analytic model alongside the sampler, compare them at an intermediate layer where the analytic model is still accurate, and use the discrepancy to correct the final answer.
17.6% final-layer reduction, paired bootstrap; 2.39× in another build Team Puffi 18175, hyojun_kwon 18154, mliston 18170A design principle that explains why most control variates fail here: on a structured point set, the thing left to cancel is the quadrature rule's bias, which is a different object from Monte Carlo variance.
Stated as the precondition for the same author's self-reported 20% gain Cipo 18152The pair integral that a covariance-propagating estimator needs at every layer has a classical series expansion. Using the series instead of numerical quadrature changes the cost by three orders of magnitude.
30.6% of budget becomes 0.012% of budget amalgonim 18151Given exact moments at the final layer, reconstruct the distribution as the maximum-entropy one consistent with them rather than as a truncated series. Series expansions can go negative; this cannot.
bias² = 5.077e-09 at order 6 jamesrahenry 18157, bin_yong_bong 18184Counting the degrees of freedom a sample-based estimator can possibly exploit gives an upper bound on every sampler anyone could build, not just the ones that exist.
Ceiling 1.915×; a deployed estimator already at 1.731× amalgonim 18151Error is not spread evenly across the 256 output neurons. It concentrates almost entirely in the ones sitting near the ReLU kink, where the answer is most sensitive.
Quantiles of |c| over 4 networks and 1024 neurons amalgonim 18151matmul, dot, einsum, tensordot, inner and multi_dot all charge identically for the same mathematical work. An independent audit of the evaluation-version cost model found no arbitrage at all.
Ratio 1.000 across all routes; fp16 = fp32; fp64 = 2× amalgonim 18151, qi_zhang5 18179, bin_yong_bong 18184A contraction of an operand with itself produces a symmetric result, and the meter prices it at half. The saving is exact and bit-identical, not an approximation.
0.502×, maximum difference 0 bin_yong_bong 18184, hyojun_kwon 18154The residual wall-clock charge is dominated by how many times you call into the metered library, not by how much arithmetic each call performs. This inverts the usual advice about batching.
3.3 µs per call at tiny sizes, 13.2 µs at 54 MB SKIBIDI_TOILET 18166, trim_qewas 18182, amalgonim 18151Fast matrix multiplication saves metered operations, but the saving converges. Depth five and depth six give exactly the same number, and going deeper costs wall time.
1.000 / 1.138 / 1.288 / 1.444 / 1.588 / 1.686 / 1.686 bin_yong_bong 18184, konstantin_baltsat 18159The statistics helpers silently return float64 for any input, and float64 bills at twice the rate. Several participants lost a chunk of their budget to this before finding it.
Worth only 0.7% of the bill until every promoted vector was found nkosi_ndwandwe 18127, hyojun_kwon 18154Exceeding the budget does not scale your penalty smoothly, it zeroes your predictions. The check was not wired on the evaluator until August, and older submissions were re-graded retroactively.
One submission went from ~4e-8 per network to an aggregate 0.682 pranay212 and qi_zhang5, 18129Replace one internal quantity with its true value, leave everything else running, and measure. The result upper-bounds every idea whose effect flows through that quantity, including ideas nobody has had yet.
54 families closed, about 15 minutes each trim_qewas 18182Ground truth was produced by Monte Carlo at a billion samples, so it has its own error floor. Below that floor you are measuring the target's noise, not your own accuracy.
Floor 4.949e-11 on the mini split, 5.219e-11 on the full trim_qewas 18182The same network measured under ten different global orientations gave errors varying by a factor of several. Any ablation smaller than that variation is unmeasurable without crossed replicates.
One network varied 2.9× to 5.4× across ten orientations ely2sh 18176, violeta 18180Choosing a different configuration for each network sounds obviously good. The selector itself has to be computed, and the computation costs more than the choice saves.
Selector cost 25.202B FLOPs, 9.3% of the budget konstantin_baltsat 18159Packing boolean predicates into bits to make masks cheaper. Split views saved no metered operations at all, and even removing 100 percent of the packable work would have saved nothing.
Zero FLOPs saved konstantin_baltsat 18149A hardcoded mathematical constant entered with a small error. Because the constant anchors a moment used at every layer, the error compounded through the depth of the network.
0.22% constant error measured 24× worse end to end bin_yong_bong 18184, SKIBIDI_TOILET 18166The field agreed the 66,048-direction carrier works because it is an exact degree-five design. Someone tested that assumption directly by completing the design, and the mechanism turned out to be something else.
Completing it buys 1.0050×, not "a lot" ely2sh 18176An oracle that knows precisely which neurons are dead is measurably beaten by a sloppy pilot that misclassifies live ones. Pruning a rarely-firing unit removes almost no signal and a full unit of noise.
Pilot 1.315×, exact oracle mask 1.205× ely2sh 18176, amalgonim 18151A textbook degree-five Gaussian cubature, verified exact to fifteen digits, lost badly. At this dimension exactness algebraically forces negative weights, which destroys the effective sample size.
2.6048e-05 against 3.6955e-07, t = 15.92 bin_yong_bong 1818412 / Phase 2
Levers I would try next
Phase 2 had not opened at the time of writing, so nothing here has a score attached to it. This is a list of what I would try, ranked by what the Phase 1 measurements say each is worth. None of it is tested in my build.
| Lever | Reported worth | Why it is on the list |
|---|---|---|
| Batch whitening | 2.0x to 2.34x | The largest single number anyone published, and it may make the point set redundant rather than adding to it. Cheap to test: one factorisation and a triangular solve folded into the first weight matrix. |
| Radial and angular split | exact, free | Removes a whole source of variance at no cost and introduces no error. It is also the precondition for anything below that samples directions. |
| Spherical design carrier | carrier-dependent | The 66,048-direction construction that most of the strong entries converged on. Worth pairing with the note that its mechanism is disputed. |
| Mid-network control variate | 17.6% to 2.39x | Buys dependence information without paying per sample. My own null was for an input-anchored control, which is a different object. |
| Looser pilot pruning | 1.315x vs 1.205x | My build uses an exact dead mask. The measurement says a deliberately sloppy pilot beats an exact one at matched cost. |
| Symmetric Gram restructuring | 0.502x, exact | A real half-price route if the expensive step can be written as a contraction of an operand with itself. Bit-identical output. |
| Rank-40 rectangular multiplication | about 48% of leaves | A faster exact product than the two-level scheme I shipped, reported to remove roughly half the leaf products. |
| Fewer, larger calls | per-call tax | My build is already at about 2,130 calls per network. The measurements say the residual charge tracks call count, so there may be more here. |
| Preintegration over the kink | untested | Integrate one direction out analytically for each neuron whose pre-activation crosses zero, so the sampler sees a smooth integrand instead of a kinked one. This is the textbook fix for exactly this class of integrand, and my own diagnosis says the kinks are where the error is. Parked after Phase 1, never run. |
| Ridge-ReLU control variate | needs R² ≥ 0.338 | Fit a small sum of ReLU ridge functions on a pilot batch. Because the family is bias-free and positively homogeneous, each term has an exact Gaussian mean, so the control needs no extra sampling to be unbiased. Gate was pre-registered and the build never happened. |
| Two-phase regression estimator | untested | Draw extra cheap samples of a correlated proxy alongside the expensive ones and combine with the classical two-phase regression formula. The coefficient comes from the sample covariance rather than from an analytic anchor, which is what separates it from the control variates I already killed. |
| Finer sparsity tiers | up to 1.6× on price | Activation density is exactly 0.50 at every layer, and my staircase captures about 0.80 of that bound. The tier count was priced against an assumed 30 second wall that has since been measured safe to 56, so the pricing should be redone. |
| Seed-generated point set | about 0.9% of cost | Ship the scrambling seed instead of a 48 MiB array and regenerate the points at run time. Removes the package cap as a constraint on sample count. A rival's 0.012 second setup is the existence proof. Prototype gate already written, never run. |
| Higher cumulants with the exact kernel | untested | The one propagation variant my own kill chain never actually ran: third and fourth cumulants carried forward through the verified exact pair kernel. The kernel runs were all second-order only. |
| Maximum-entropy readout | bias² 5.077e-09 | Better than the series expansions on the same inputs. Listed last on purpose: section 6 caps the whole readout lane at about 3 percent of the error. |
What each lever would be worth animated
The honest summary of that list: almost all of it is cost-side or variance-side improvement to a sampler, which is the only lane the score algebra leaves open. To be clear about which gap: Phase 2 has no leaderboard yet, so the comparison is to the entry that finished on top of the Phase 1 public board, at 1.84e-08 against my honest 1.382e-07. None of the levers below closes a gap that size on its own. Either the published bound on what a sampler can do is wrong, or whoever produced that number was not running a sampler.
Six of these came out of my own notes rather than from the forum. They were proposed during Phase 1, priced, given a gate in some cases, and then never run because the window closed. Where a gate exists it is quoted, so the bar each one has to clear is visible before anyone spends time on it.
13 / Start here
If you are picking this up from zero
The order that would have saved me the most time:
1. Work through the metric before writing an estimator
Section 4 is worth doing on paper rather than reading. A sampler's score is a × p / B, and it does not depend on the sample count. If that is not settled in advance, it is easy to spend a long time adjusting a parameter that the algebra says cannot change the result.
2. Build the measurement setup before the estimator
Most of the published write-ups contain a section about a measurement that later turned out to be wrong. Rotation noise moves a single network's error by a factor of several. The answer key has its own floor near 5e-11. The public 50 networks are an unusually easy draw, at roughly the 0.19th percentile of ordinary draws. A three-replicate decomposition reversed at least one participant's earlier recorded verdict. It is worth deciding how a 3 percent effect will be distinguished from noise before there is a 3 percent effect to discuss.
3. Use oracle substitution before building anything
The protocol described in the field guide is inexpensive and rules out a great deal of work. Replace an internal quantity with its true value and measure what error remains. If that bound sits above the target, every method in the family is ruled out and none of them needs to be written. It is a faster way to reach a negative result than implementing the methods one at a time.
4. Read the cost model as carefully as the mathematics
About a third of the published techniques concern billing rather than estimation. Wall time is charged per call. Float64 costs double, and the statistics helpers return it without being asked. Symmetric contractions are priced at half. Exceeding the budget discards the prediction. None of this appears in the paper the competition is based on, and all of it enters the score.
A summary of the position
Sampling propagates the true distribution exactly and is charged linearly for doing so. Analytic propagation is close to free but discards the dependence structure, and at depth 32 the dependence structure accounts for almost the entire answer. Each of the stronger entries is a particular measured compromise between those two facts, and the more interesting ideas are the ones that obtain dependence information without paying for it one sample at a time.
The entry that finished on top of the Phase 1 public board scores 7.5 times better than my honest submission, and does it while spending slightly more compute rather than less. As far as I can tell from the public record, the mechanism has not been explained. That question appears to be open.