01 / The task
A closed-book question with 2 unusual scoring rules
The rules of the shared task decide what a good system looks like before any modelling question is asked. Both of the rules that matter are about what a system is allowed to say, not about how it thinks.
Given a subject and a relation, predict the complete set of object entities as the world stood on 1 July 2026. The only permitted source is the of an open weight model of at most 32B parameters. No retrieval, no external knowledge base, and no training of any kind. 6 relations, 475 test rows.
2 features of the scoring rule shape everything that follows.
Empty answers score like any other answer
102 of the 475 test rows have an empty gold set. The subject genuinely has no object for that relation, and the row scores 1.0 only if the system predicts nothing at all. A system that abstains on every row, answering nothing anywhere, still scores 0.2147. That is the floor any real system has to clear, and it is a floor built entirely out of silence.
The consequence is that a large part of the score is not a retrieval decision. It is a decision about whether to speak. A confident wrong answer on an empty row costs exactly as much as a confident wrong answer on a populated one, and there is no partial credit for being nearly right about a set that should have been empty.
The numeric relations are graded on a tolerance, not a match
2 of the 6 relations are numeric: hasArea and hasCapacity. Between them they carry 198 of the 475 rows. Each is graded as a single value that has to land within ±5% of gold. Inside the window the row scores 1.0, outside it scores 0, and nothing in between exists.
A tolerance band is a different object from an exact match, and the difference matters more than it first appears. It means a numeric answer does not have to be recalled, only bracketed. It also means, as section 04 works through in detail, that a wide enough pool of guesses will contain an acceptable one by accident.
The 2 scoring rules interactive, pick a panel
- Test rows
- 475
- Empty gold sets
- 102
- Abstain everywhere
- 0.2147
- Shipped system
- 0.7060
Why this framing is worth stating up front. Read quickly, the task looks like a knowledge probe. Read carefully, it is 2 calibration problems wearing a knowledge probe's clothes: when to return the empty set, and which single number to commit to. The system described next is organised around exactly those 2 questions.
02 / The system
1 model, 1 elicitation channel per relation
Nothing in the system is learned. Every component is a prompt template, a decode contract, a parser, or a threshold, chosen once and then frozen.
The model, and the reason its parameter count is reported exactly
The system runs a single model: gemma-4-31B, in , served with . The 32B cap is a rule rather than a guideline, so the parameter count was measured from the safetensors index rather than taken from a model card: 31,273,088,876 BF16 parameters.
That leaves 0.727B of headroom under the cap. The number is reported because of what it excludes. A margin that size cannot hide a 2nd neural component of any useful capacity, so the claim that the system runs 1 model rather than 2 is checkable arithmetic rather than a description of intent.
Elicitation channels
Each relation gets its own elicitation channel: a fixed prompt template, a decode contract that fixes sampling and stop conditions, and a parser that turns raw completions into candidate objects. The 3 pieces travel together because they only make sense together. A parser tuned for 1 output shape is not a parser for another, and a decode contract that permits free continuation makes any parser guesswork.
The channel is drawn from repeatedly, and everything downstream operates on the resulting pool of draws rather than on the model directly. That separation is what makes the experiments in sections 03 and 04 possible: the pool can be held fixed while the thing that reads it is swapped out.
1 row: subject → channel → pool → answer static diagram
Set relations: 1 scalar decides 2 things
For the 4 set valued relations, each candidate object is scored by its across the draws, and candidates above a per-relation threshold are kept. Abstention is not a separate component. When no candidate clears the threshold the row returns the empty set, so a single scalar per relation decides both what is kept and whether anything is kept at all.
Collapsing those 2 decisions into 1 scalar is a deliberate restriction. A separate abstention gate would be a 2nd thing to calibrate, on the same small validation set that section 05 shows cannot support the calibration it already has.
Numeric relations: aggregate against the grader, not the distribution
For the 2 numeric relations the aggregation is built around the grader. Each drawn value is scored by how many other draws fall within the grader's own ±5% tolerance of it, and the value with the highest count is returned.
This replaces the recipe published for this task family. The difference is structural rather than cosmetic. A cluster grown by linking neighbouring values can stretch arbitrarily far from where it started, 1 link at a time, and its median can end up outside every tolerance window that any of its members would have accepted. A fixed radius around a candidate cannot chain: every draw that contributes to a candidate's score is, by construction, a draw the grader would have accepted had that candidate been gold.
Reproducibility and closed-book compliance
Draw pools are . A pool is keyed by a hash over the full run identity, so changing the prompt template, the decode contract, the model revision or the seed produces a different pool rather than silently reusing an old one. Every measurement in the next 3 sections is a comparison between pools that can be named.
Closed-book compliance is enforced rather than asserted. 4 checks run over the repository and the job environment:
- Network library grep. The source is scanned for imports and call sites of any library capable of a network request.
- A single sanctioned data loader. All file reads go through 1 function with an explicit allowlist of paths, so a stray read of an unapproved file is a failure rather than a silent success.
- Fork hygiene. Child processes are checked so that a worker cannot inherit a capability the parent was audited not to use.
- Offline environment variables on every inference job. The offline flags are set at the job level, not once in a shell, so a job that would have reached out fails instead of falling back.
None of this makes the system better at the task. It makes the result mean what it says, which is a different property and the only one a reader can check.
03 / Finding 1
The largest single improvement was a change of register
Not an algorithm, not an aggregation rule, and not a threshold. A change to what the model is asked to do before it answers.
3 earlier configurations on hasArea, spanning 2 prompt frames and 2 aggregation methods, all returned exactly 0.8100 on test. 3 different systems, 1 number, to 4 decimal places.
There is an obvious reading of that, and I took it at the time: if changing both the frame and the aggregation moves nothing, the answers are probably not in the candidate pool at all, and no selector will find what is not there. Under that reading the relation was finished and the remaining effort belonged elsewhere.
The reading was wrong. Asking the model to recall facts about the entity before committing to an answer, rather than asking for the answer directly, moved hasArea from 0.8100 to 0.8700 on test. The aggregation did not change. The threshold did not change. The pool changed.
The general form. Selection cannot recover what elicitation never surfaced. A plateau across selection methods is evidence about selection, and it says nothing at all about whether the knowledge is present. Reading it as evidence about the model is the specific mistake this section records, and it cost several rounds before the register experiment was run.
Why the recall step works is a separate question from whether it works, and this page does not claim to have settled it. The measurement is that a prompt frame which asks for context before an answer produces a pool from which the same selector extracts a better answer. Whether that is because the recalled context conditions the numeric continuation, or because it changes the distribution the decode contract samples from, was not isolated. What was isolated is that the improvement is upstream of selection.
3 configurations, 1 number, then a register change animated
That last comparison is worth holding onto. The margin at the top of the official board is 0.0152. A single prompt frame decision on a single relation moved 0.0600 on that relation, which is roughly 0.0126 on the row weighted total. 1 elicitation choice is the same order of magnitude as the entire competitive result.
04 / Finding 2
Oracle coverage has no null control
The statistic this literature uses to report headroom measures 2 things at once, and nothing inside it separates them.
Oracle coverage is the standard way headroom gets reported here. Draw a pool of candidates, check whether the gold answer appears anywhere in it, and report the fraction. The number is then read as a ceiling: this is what a perfect selector could reach, so the gap between it and the current score is what better selection would buy.
For the numeric relations that reading does not survive contact with the tolerance band. A candidate counts as covered if it lands within ±5% of gold, which means each draw does not claim a point on the line, it claims an interval. A pool of 100 numeric guesses spanning 1 decade contains about 21 mutually exclusive 5% tolerance windows. A pool that dense contains something acceptably close to almost any plausible target, whether or not the model knows the answer.
A decade, tiled by tolerance windows animated
The control
The control is a . Give every subject a different subject's gold value, drawn from the same relation, and recompute coverage on unchanged pools. The pools were produced without any knowledge of the value they are now being scored against, so whatever coverage survives the shuffle is coincidence. Over 400 shuffles on validation:
The 2 relations behave completely differently under an identical test, which is the point. Coverage of 0.845 and coverage of 0.940 look like 2 readings of the same quantity. They are not. One of them is mostly the width of the pool.
Correcting for chance changes the number that decides where effort goes. The findable headroom beyond the rank 1 answer on hasCapacity falls from 0.443 to 0.203, roughly 2.5x smaller than the uncorrected figure the field quotes.
Shuffle the gold values and watch coverage collapse interactive, an illustration of the published measurement
- Shuffles run
- 0
- hasCapacity, chance
- 0.000
- hasArea, chance
- 0.000
- Observed coverage
- 0.845 / 0.940
Where the correction lands
The decomposition below is the whole argument in 1 figure. Scroll it through and the coverage bar separates into the part that is knowledge and the part that is pool width, and the headroom number that follows from it moves accordingly.
Decomposing hasCapacity coverage scroll driven
Figure 6, scroll driven. Every value shown is a published measurement from the validation permutation test. The stages are staged for reading, not derived on the fly.
Stage 1 / the reported number
Oracle coverage on hasCapacity is 0.845. Read the usual way, that is the ceiling: a perfect selector reading these pools would get 84.5% of rows right, so the distance from the shipped score to 0.845 is what better selection is worth.
Stage 2 / the null
Reassign every subject a different subject's gold value and rescore the same pools. Over 400 shuffles the coverage that survives is 0.355.
That fraction of the bar is not knowledge. It is the pools being wide enough to catch a value they were never asked about.
Stage 3 / what is actually left
Subtract the null and 0.490 of coverage remains attributable to the model. The bar has not changed. What changed is how much of it can be spent.
Stage 4 / the headroom that gets quoted
Coverage is rarely quoted raw. What gets quoted is the headroom beyond the rank 1 answer: how much a better selector could add on top of the candidate the system already returns. Uncorrected, that figure on hasCapacity is 0.443.
Stage 5 / chance corrected
Apply the same correction to the headroom and it falls to 0.203, roughly 2.5x smaller.
Both numbers describe the identical pools and the identical selector. One of them argues for months of selector work. The other argues for stopping.
3 instruments that are not functions of vote frequency
A chance correction is a subtraction, and a subtraction invites the objection that the correction is doing the work. So the same claim was put to 3 instruments that do not read vote frequency at all, which is the quantity the shipped selector uses.
- Forced choice duels. Present the model with 2 candidates and ask which is correct, then resolve the pool by pairwise comparison. Score 0.3196, against the shipped selector's 0.3608.
- Yes or no verification. Ask about 1 candidate at a time and keep what survives. Score 0.2680.
- Cross frame agreement. Re-ask under a different prompt frame and keep only what both frames agree on. Delta exactly 0.0000.
None of the 3 beats the selector already in the system, and the 3rd moves nothing whatsoever. 3 readings taken with different instruments, none of which shares the shipped selector's failure mode, agree that the relation is knowledge bound rather than selection bound.
The remaining headroom can also be bounded from above without a new instrument at all. Extrapolating the relation's score to the infinite draw limit bounds what is left at 0.008 to 0.022. That interval is narrower than the per relation validation standard error reported in the next section, which is a compact way of saying that the remaining selection headroom on this relation is not measurable with the data available.
4 readings of the same relation animated
What the finding does and does not claim
It claims that oracle coverage, as reported in this literature, mixes model knowledge with pool width, and that a permutation null separates them at the cost of 1 extra experiment. On the 2 numeric relations here the separation is large enough to change what the number recommends.
It does not claim that selection is finished, or that these numbers transfer to other relations, other models or other tolerance rules. The chance rate is a property of the pool and the grader, so it has to be measured wherever it is used. That is the entire cost of the correction: 1 shuffle loop over data that already exists.
The permutation test is not novel statistics. What is reported here is that a standard control was missing from a standard statistic, and that adding it moves the headroom on 1 relation by a factor of about 2.5x.
05 / Finding 3
Validation cannot resolve the decisions it is being asked to make
The measuring instrument is between 1x and 5x coarser than the effects it is used to rank.
Every decision described so far was taken against a validation set, so it is worth being precise about what that set can say. Each relation carries between 67 and 100 rows. At that size the of a per relation score is 0.03 to 0.05. The differences that decide leaderboard rankings are 0.01 to 0.03.
Those 2 intervals do not overlap in the direction anyone would want. The smallest noise is larger than the largest effect. An ordering produced by comparing 2 configurations on this validation set is close to arbitrary at exactly the scale where the competition is decided.
Noise against effect animated
This is not a theoretical worry. Across 5 scored submissions, validation missed the 2 largest improvements entirely, understated a 3rd by 50%, and inverted the sign of 2 more.
- Submission A
- Largest improvement of the campaign. Validation did not see it.
- Submission B
- 2nd largest improvement. Validation did not see it either.
- Submission C
- Real gain, reported by validation at about half its size.
- Submission D
- Validation reported the change in the wrong direction.
- Submission E
- Validation reported the change in the wrong direction again.
A development loop that trusts this validation set will reject good changes and ship bad ones, and it will attach a confident number to each mistake. 5 of 5 is a small sample, and it is also every scored submission there was.
What was actually used to decide, stated plainly
If validation cannot resolve these decisions, something else did, and the honest position is to name it. The paper discloses, as part of the method rather than as a footnote, that scalar thresholds were calibrated on the returned per relation aggregate scores. No per row feedback was used at any point, and none is available.
Why disclose it in the method. It is a real dependence on the evaluation channel, and a reader who does not know about it cannot price the result correctly. It is also bounded and describable: a handful of scalars, calibrated on aggregate numbers that every participant received, with no access to individual rows. Stating both halves is what lets somebody else decide whether they find the result convincing, which is the only thing a disclosure is for.
The same constraint is why the system has as few free parameters as it does. A per relation threshold that also serves as the abstention rule is 1 scalar. A separate abstention gate would have been 2, calibrated against a signal that cannot support 1.
06 / Results
1st place, at 0.7060
The official final leaderboard was released on 1 September 2026. The per relation view is the more informative one, and it says something the total does not.
Official final leaderboard, top 5 interactive, hover or focus a row
The margin over the runner up is 0.0152. That is a real gap and a small one, and the section on validation is the reason it is quoted with that framing rather than as a comfortable lead.
Per relation
The table below gives this system's score on each relation, the runner up's score on the same relation, and the best score reached by any other ranked team on it.
6 relations, 3 readings each animated
| Relation | Rows | This system | Runner up | Best other team | Delta to best other |
|---|---|---|---|---|---|
| countryLandBordersCountry | 67 | 0.9786 | 0.9873 | 0.9873 | −0.0087 |
| hasArea | 100 | 0.8700 | 0.8400 | 0.8600 | +0.0100 |
| companyTradesAtStockExchange | 100 | 0.8530 | 0.8461 | 0.8717 | −0.0187 |
| hasCapacity | 98 | 0.3367 | 0.3146 | 0.3163 | +0.0204 |
| personHasCityOfDeath | 100 | 0.6100 | 0.5900 | 0.5900 | +0.0200 |
| awardWonBy | 10 | 0.3484 | 0.3556 | 0.3691 | −0.0207 |
This system holds the best score on 3 of the 6 relations and is behind on the other 3. It does not win by being strongest anywhere in particular. It wins by not having a weak relation, which is the shape a system takes when the abstention rule and the numeric aggregation are the parts that received the attention.
1 caveat belongs on the table itself. awardWonBy has 10 rows. A per relation score computed on 10 rows moves in steps of 1/10 of a row's worth of credit, and no ranking on that relation, including this one, should be read as a measurement of anything.
The composite nobody submitted
A useful way to price a 1st place on a board this tight is to ask what an imaginary system would score if it took the best per relation result of every other ranked team, and therefore beat this system on 3 of 6 relations by construction.
That composite reaches 0.7011. The single system reported here reaches 0.7060.
The margin over the composite is 0.0049, smaller than the 0.0152 margin over the runner up, and that is the honest reading. On a board this close, 1st place is a statement about consistency across 6 relations rather than about dominance on any of them.
07 / Coda
What transfers past this task
3 of the results here are about the shared task. The rest are about how to measure a system whose components you cannot see.
- A plateau across selection methods is evidence about selection. 3 configurations returning an identical score says the selectors agree, and says nothing about whether the answer is reachable. The experiment that settles it changes the elicitation, not the selector.
- Any coverage or oracle statistic needs a null control. If a candidate counts as correct when it falls inside a tolerance, then pool width buys coverage directly, and a permutation over targets measures exactly how much. The control costs 1 loop over data that already exists.
- When the validation set is coarser than the decisions, say so, and report what actually decided. A method section that describes a validation loop which demonstrably could not have produced the reported choices is describing a check that did not run.
The paper is to appear in the AKBC 2026 proceedings, and the poster will be presented at the AKBC workshop at EMNLP 2026 in Budapest, October 2026.
Task citation: Kalo, Nguyen, Razniewski and Zhang, AKBC Shared Task 2026: Predicting Complete Knowledge Base Entries from Language Models.