Thirteen thousand skills, in order

ESCO is the European Union's catalogue of skills, competences and occupations. Version 1.1 contains exactly 13,891 skill concepts. Each one has a preferred label, a paragraph of description, and on average seven alternative labels. There are 97,441 of those in total, collapsing to 96,684 distinct normalised strings.

The task: given a piece of text about work, score all 13,891 concepts and return them in order. Five datasets are scored and averaged with equal weight. Four of them are extraction: the query is a sentence or a segment of a job advert, and several skills are correct at once. The fifth is normalisation: the query is a short surface form like "MS Excel" and the job is to find its canonical entry.

Relevance is graded, not binary

This is the design decision that invalidates the obvious approach. Every query–skill pair carries an integer grade from 0 to 4, and the grades are semantic rather than confidence levels:

The example rows are illustrative, chosen to show what each grade means. The real annotations are the organisers'.

One task has no local labels at all

Of the five datasets, TechWolf ships a test split and no validation split. It is one fifth of the final score and 324 of the 1,873 test queries, and there is no way to measure anything about it locally. Every decision that touches it is a bet placed blind.

What could be established was its shape. Its queries have a median length of 13 words and 169 of 324 carry a bullet prefix, which makes it structurally an extraction task. Its exact alternative-label hit rate is 3 out of 324, against 450 out of 450 for the normalisation set, so whatever else it is, it is not normalisation-like. It was treated as Tech/House-like throughout, and that assumption held: it finished at 0.8347, in line with the other extraction tasks.

Scoring is macro-averaged nDCG@100 with exponential gain. Two multipliers decide a system's fate: what a grade is worth, and where it sits.

Multiply those together and the shape of the problem falls out: the score is almost entirely decided by whether the two-to-ten items graded 3 or 4 land in the first handful of rows. Everything else is rounding.

One more property matters, and it saved a lot of wasted effort: nDCG is order-only. Any strictly increasing transform of the scores leaves it unchanged. Calibrating a single final ranker is therefore a no-op. Calibration matters only where scores from different sources have to be made commensurable, which is why every retrieval lane below enters as a per-query z-score rather than a raw similarity.

Stage 1: twelve lanes, two of which do the retrieving

Twelve retrieval lanes score every concept for every query: four domain bi-encoders, two general ones, four lexical, and two string-similarity lanes. The candidate pool comes from just two of them, fused by z-score:

Two things about that line are worth more than they look. The weights were measured, not guessed, and naive equal-weight reciprocal-rank fusion of the three obvious lanes was worse than its best single component, 0.5935 against 0.6416. Averaging rankers is not free. And the fused base reproduces to 0.6499 exactly, which was wired in as a known-value gate: if the pipeline drifts, the run aborts rather than silently producing a worse submission.

Stage 2: thirty-three columns per candidate

Each lane contributes two columns: the per-query z-score, and the log of its within-query rank. Nine aggregate columns follow: the base score and its rank, how many lanes place the candidate in their own top 50 and top 200, the mean, spread and max of the lane z-scores, and the two word counts. Twenty-four plus nine is thirty-three.

The agreement and spread columns are the interesting ones. They let the final model distinguish a candidate that every lane likes moderately from one that a single lane loves.

A learned ranker over signals I already had  +0.0291

Replacing the hand-weighted fusion with a LambdaRank model over all thirty-three columns moved the score from 0.6499 to 0.6790. Twelve mostly-bad retrievers, combined by a learned model, beat the best of them by more than any of them is worth on its own, and no GPU was involved.

A smoothed relevance prior  +0.0380

Some ESCO concepts are simply relevant more often than others. Counting that is four columns of arithmetic: computed only over the candidate pools of each fold's training queries, leave-one-out on the training rows, so no label from an evaluated query ever touches its own features:

In the submitted model prior_hi is the highest-importance feature of all, and prior_egain is fourth, behind two of the cross-encoder log-rank columns. Four columns of counting statistics outrank a four-billion-parameter reranker's score. The effect measured +0.0380 over seven disjoint seeds (figure 4 credits the prior with +0.0331 instead, because that is the ladder rung rather than the isolated lever, and I quote each where it was taken), survived a leave-one-task-out control at +0.0315, and collapsed entirely when the labels were shuffled, which is the control that matters, because a prior computed from the pool is exactly the kind of feature that can leak if you build it carelessly.

An exact-match override  +0.0152 local

If a query string matches an ESCO alternative label exactly after normalisation, that concept is forced to rank 1 at write time, not offered as a feature for the model to weigh, but overridden outright. ESCO carries 97,441 alternative labels, collapsing to 96,684 distinct normalised strings, of which only 0.74% are ambiguous, with a maximum fan-out of four. Two separate measurements describe its effect, and they should not be run together. In isolation the override is worth +0.0564 on the normalisation task and exactly 0.0000 on each of the other three, which averages to +0.0141 over the four local tasks and projects to +0.0113 over the five-task board. Measured instead as a rung of the pipeline ladder, with the prior already in place, it moved the four-task macro by +0.0152, the difference being the interaction with everything beneath it. It was never isolated on the board itself, since no two submissions differed only by this switch. The asymmetry is the whole story: 450 of 450 normalisation queries hit an exact alternative label, against 3 of 324 TechWolf queries.

Eleven cross-encoders were trained, across two architecture families, each in two stages: a synthetic pre-train on a subsample of a 138,260-row generated corpus (80,000 rows for most members, mixed with up to 14,000 ESCO description-to-skill rows that supply the short-phrase supervision the sentences lack) with a mined negative ladder, then a fine-tune on the real graded annotations inside the candidate pool. All of them train under LambdaLoss with the NDCGLoss2++ scheme, a listwise objective that weights each pair by how much swapping it would change nDCG. When the metric is nDCG and the labels are graded, that is simply the matching objective.

n×m is the stage-2 density: n listwise rows sampled per query, m documents per row. Q8's figure is derived rather than logged directly, since the ledger records Q8D as +0.0049 over it at the column level. Highlighted rows are the four in the submitted ensemble.

Look at the last two rows. Q8D was the strongest single reranker I ever measured, and every ensemble containing it failed the adoption gate: the pool already held two Qwen members, and its contribution was absorbed. Meanwhile C4 passed. It scores 0.8245 alone, above both DeBERTa columns already in the pool and below all four Qwen-family columns, and it entered by replacing C3 rather than lengthening the ensemble.

One implementation detail worth stealing

Qwen3-Reranker ships as a causal language model that answers a yes/no question. Loading it through the usual sequence-classification path attaches a randomly initialised head and discards everything the checkpoint knows about reranking. Instead, score with the difference of two rows of the existing output embedding:

Verified to 2.4e-8 against evaluating the full logit vector, and it turns a 151,000-way softmax into a single dot product. The resulting scalar head then trains under LoRA. A different checkpoint routed through the generic path earlier in the campaign produced exactly the random-head failure this avoids, and was cut.

Consider the position after the ensemble reached about 0.864 locally. The remaining candidate improvements were worth +0.001 to +0.002. The validation set is 286 queries. The normalisation task alone carries roughly ±0.03 of sampling noise, and the paired minimum detectable effect on its 50 queries, at 80% power, is ±0.015 to ±0.017. Dozens of plausible configurations remained unmeasured. Ten submissions existed in total, each returning one number, hours later.

Pre-registration

Every round was written into an append-only ledger before the runs existed, enumerating the exact configurations to be tried and closing the space: no configurations outside these templates without a fresh registration line written before the run. Entries are appended, never rewritten; a correction is a new entry.

This is not bookkeeping. It is what makes the denominator of the multiple-comparisons problem knowable. A procedure that allows configurations invented after seeing results has no denominator, and therefore no defensible claim about significance, however it is analysed afterwards.

Paired seeds, on identical folds

Fold assignment is deterministic in the seed and the query count, so a per-seed baseline stored during an earlier round pairs exactly with a candidate measured later. Every comparison runs over seeds 0–9 and reports the paired mean difference, the number of seeds won, and the spread. Pairing validity was checked rather than assumed: before one round, three configurations had to reproduce the earlier harness seed for seed, and parity was recorded as exact to 0.000000 on all three checks.

The gate

The multiplicity guard, and the winner's curse

Anything that passed faced a confirmation run on fresh seeds 10–19, keeping its pass only if it beat the incumbent there too. This was exercised exactly once, on the only passer of the extended phase. Its discovery estimate was 0.8647; the pooled twenty-seed estimate was 0.8644. The direction is what the winner's curse predicts and the size matches the 0.0003 haircut registered for it in advance, but against a measured across-seed spread of 0.0007, a single 0.0003 gap is well inside noise. Read it as consistent with the curse, not as a measurement of it.

When selecting among many correlated subsets (one round evaluated twenty-six) a haircut of 0.0003 was subtracted from the winner before comparing it to the decision threshold, sized to the selection breadth.

The slot bar: turning a local score into a probability

The last question is when a locally better configuration is worth one of ten irreplaceable submissions. That needs a mapping from local score to leaderboard score, and the mapping is empirical, but it moves, which is the first thing to say about it. Measured seven times across the campaign, it ran from −0.0086 for the first fitted submission to −0.0235 for a mature one. A fitted system transfers worse than a zero-shot one, and the offset roughly doubled as the system grew. The bars below rest on the four most recent own-row offsets only, which do concentrate: mean −0.02005, standard deviation 0.00077.

Given a target board score, the local score needed to beat it with probability p follows from a one-sided prediction interval on a new offset draw. With four offsets in the final estimate that is a t-interval on three degrees of freedom, not a normal z-interval, a distinction a reviewer caught, and one that moved the thresholds enough to matter:

These bars were registered before the final round, which made the last decision mechanical rather than a judgement call. The adopted configuration sat at 0.8647 local, clearing the lean-family 50% bar of 0.8640, not the 90% bar, and was submitted on exactly that basis. It scored 0.8458 against the row it needed to beat at 0.8444.

What the protocol cost

It is only fair to record the price. The protocol rejected c4b, which beat its incumbent on ten seeds out of ten. It rejected three Q2D configurations at +0.0009, +0.0009 and +0.0006 with eight and nine seeds won. It rejected an ESCO attribute join at +0.0005 with 6/10. And it refused a recombination of my own already-scored leaderboard rows worth +0.0002, not excluded by any rule I could point to, though my own review recorded the rules-letter status as ambiguous, since the rules require the submission file to be produced by the real benchmark tasks.

A protocol tuned this conservatively will, in expectation, leave small real gains on the table. The trade is deliberate. When measurement error exceeds effect size, the dominant risk is not missing a real gain; it is adopting a false one and then building on top of it.

Submission 874278. Configuration C2, C4, Q2, Q8 at depth 1,000, prior and override on, bagged over ten seeds. Build cross-validation on three seeds gave 0.8649 / 0.8659 / 0.8652 → 0.8653, so the realised local-to-board offset was −0.0195, inside the pre-registered band of −0.0192 to −0.0199.

The final-phase ledger

Spread across seeds was inside the 0.0015 gate for every row where it is reported; the passer measured 0.0007. It then cleared fresh-seed confirmation on seeds 10–19 at +0.0010 with 9/10 wins, and was submitted. Look at the three Q2D rows: +0.0009, +0.0009 and +0.0006 with eight and nine seeds won. Those are precisely the results a permissive protocol adopts and a fixed gate does not.

The channel

The five graded datasets are published on the Hugging Face Hub. Their qrels/test configurations were served, unauthenticated, through the public datasets-server API. One request returns their sizes:

I have two point observations, not continuous coverage. On 30 July, five days before the test phase closed, all five graded repositories exposed public qrels/test and queries/test splits whose counts matched the competition's exactly. On 4 August at 01:08 UTC, an hour after close, I re-pulled and recorded the sizes: 528, 583, 582, 1265, 450 qrels rows against 261, 338, 324, 500, 450 test queries. So the splits were reachable during the competition and remained so after it, though I did not monitor the endpoint continuously and cannot say they were reachable throughout.

The datasets' own cards document the situation candidly: the published test labels are "real but binary" and "derived from the public non-graded ground truths", with the graded 0–4 annotations "withheld during the ongoing RecSys-HR challenge". So the withheld object was genuinely withheld. What stayed reachable was a binary relevance judgement over the exact test queries. The rules prohibit enriching training with test-source data, which makes this a rules question rather than a grader vulnerability. I reported it to the organisers before results and the summary paper were finalised, and asked explicitly that my own submissions be included in whatever check followed.

Can a leaderboard reveal that such a channel was used?

I spent real effort on this question and got it wrong, so the wrong answers are what this section reports.

The first statistic. Cross-sectional: for each row, take the mean extraction score minus the normalisation score, and treat a large positive gap as anomalous. The reasoning is that a source covering the extraction tasks would lift them and leave normalisation behind. One number per row, nothing needed but the public board, and on a short list of rows it appears to separate them cleanly.

The second statistic. A within-account step: between consecutive submissions by one account, does the subset of tasks a source would help move together and sharply while the rest stays flat? This at least controls for the system, which the cross-sectional version does not.

I am also not publishing the thresholds. The leaderboard is public and the statistic is deterministic, so exact values would hand any reader a reproducible shortlist of accounts, which is precisely the accusation this page declines to make, merely routed through the reader.

Where this ends up

Two things produced that reversal, and both are cheap enough that anyone attempting this kind of analysis should run them first.

The base-rate check. How often does the pattern fire across the whole population, rather than across the short list you were already looking at? Both statistics above survive scrutiny on a handful of rows and die on the full board.

The reference-class check, which is the uncomfortable one and the one I would emphasise. My own entry was the clearest outlier on the cross-sectional statistic: all nine of my placed rows fall on the opposite side of it. Which means the reference class I was implicitly comparing against was one I had constructed by removing everyone whose system did not resemble mine. A statistic that flags "systems unlike mine" will always find some, and it will always feel like a discovery.

Sequence cheap levers first, and not only because they are cheap

The three no-GPU levers total about +0.077: roughly a third of the gain over the fused base, for a rounding error of the cost. That is not an argument against cross-encoders: +0.138 was decisive for this entry, and I know of no configuration of the cheap levers alone that came close. Whether the top of this board was reachable without them I cannot say, since I have no visibility into any other entry's architecture beyond a one-line self-description.

It is an argument about ordering, and the criticism lands on me. The cheap levers were available from day one and I did not reach the prior until the campaign was two-thirds gone. They also had the largest effects relative to their measurement error, which made them the ones a 286-query validation set could actually verify. By the time the campaign reached effects of +0.001, measurement had become more expensive than training.

Ensembles want decorrelation, not strength

The strongest reranker I trained is not in the final system, because what it knew was already there. But ELECTRA (the most decorrelated candidate available) failed too, and worse. A member has to be both different and good enough that its differences are signal. Standalone quality is a poor guide on its own; so is novelty on its own.

This is not new in the ensembling literature. It is worth restating in a competition context because the instinct under time pressure is always to train a bigger version of whatever currently scores highest.

Treat the submission quota as the scarce resource it is

Ten submissions, each returning one scalar hours later, is a bandit problem with an expensive arm. Two of my seven calibration points came free, from locally scoring baselines someone else had already put on the board. That trick generalises to any competition with a public baseline row.

Write down what you are going to do before you do it

Everything above rests on an append-only ledger that recorded each round's exact configuration list before the round existed. It is the least glamorous part of the campaign and the one I would keep if I could keep only one. Without it, the last two weeks would have been a search over noise, and I would not have been able to tell.