One sentence, nineteen reading levels
BAREC is the Balanced Arabic Readability Evaluation Corpus, and the 2026 shared task asks a model to read a single sentence and predict how hard it is to read, on a nineteen point scale. The training set is 54,845 sentences drawn from 1,518 documents, with a fixed validation split of 7,310 and a test split of 7,286. Every sentence carries a gold level, so this is ordinary supervised learning with an unusual target.
The label is ordered, and that changes everything about how you are graded. The board ranks entries by quadratic weighted kappa, or QWK, which is a version of agreement that cares how far off you are, not just whether you were exactly right. Predict level 9 when the truth is 8 and the penalty is small. Predict 12 and the penalty is much larger, because the weight grows with the square of the distance. A mistake three levels away costs nine times what a one level mistake costs. That single fact set the whole shape of my solution, so I wanted to feel it before I trained anything.
Here is what the scale feels like in practice. The same kind of thought, written five ways, gets harder to read as the level climbs. This is real Arabic, colored from the easy end of the ramp to the hard end.
Drag the prediction below and watch the penalty grow. The true level sits fixed while your guess moves, and the readout shows how many levels off you are and how heavy that costs relative to a near miss. The shape of that cost is the reason a model that is roughly right everywhere beats a model that is exactly right often but occasionally wild.
There are two tracks with the same test set. The Open track (Codabench 16544) lets you train on anything, and a parallel Strict track (16545) allows only BAREC data. I aimed at the Open board first, where the live bar to beat was 85.80 QWK, held by two entries tied at the top. Last year's winning system landed at 86.4 on the 2025 blind test, which is a different set and not directly comparable, but that number sat in my head as the ceiling worth reaching.
A grader I could run myself
A public leaderboard gives you five submissions a day and a single number back. That is a slow way to learn anything, and it tempts you to tune against the board by trial and error, which can lead to overfitting the leaderboard rather than the underlying task. So before I trained a single model I did the least glamorous work of the whole project: I rebuilt the official scorer locally and proved it matched, line for line.
The shared task ships its evaluation code, so I wrote my own metrics module and held it to a byte parity test against the official eval.py on the provided fixture. It reproduces every figure the grader prints: QWK 80.0040%, accuracy 56.6211%, adjacent accuracy 69.8632%, average distance 1.143776, and the coarser 7, 5, and 3 level accuracies, digit for digit. That test became the gate the rest of the work had to pass through.
Then came the part that turned a five per day board into an unlimited one. The 2026 release ships gold labels for the test split, and that test split is what the Open board scores. So my local score on the test set is the board score. When the first real submission went up and came back at 86.10, all seven metrics matched my local numbers exactly, which confirmed the grader in my harness is the grader on the server. From that point I could measure the exact effect of any change on the development board without spending a submission on it. That held for the whole development phase, and it ends at the blind set, which ships with no labels at all. Believing the same thing about the blind board is the mistake the last part of this piece is about.
The edge came from a scorer I trusted enough to stop guessing against the board.
The flattering number and the honest one
Having a trustworthy scorer only helps if you feed it honest predictions. The most useful models to try were the public CAMeL Lab readability checkpoints, four of them, already fine tuned for exactly this task. When I scored them on the training set they looked spectacular, 89 to 95 QWK. If I had believed those numbers I would have tuned the whole ensemble on them and walked into a wall.
The reason is contamination, and the harness caught it. Those checkpoints were trained on the BAREC corpus, which is the same text as the 2026 training split. Their training predictions are in sample, so they memorize rather than generalize. On clean holdout data the same models score 82 to 84 on validation and 84 to 85 on the test split, which is the honest picture. The gap, five points at its narrowest and more than ten at its widest, is a large part of why disciplined measurement matters here. Toggle the figure to see which number you would have trusted.
Tune on validation, evaluate once on test, and let the honest out of fold number decide every go or no go. For the rounding step in particular I used a nested out of fold estimate, the number you get by rounding on data the rounder never saw, because a rounder tuned and scored on the same fold will always look better than it is. That one habit is the difference between a method that holds up on new sentences and one that quietly falls apart.
There is a smaller version of the same idea in the rounding itself. The model outputs a continuous score, and turning that into an integer level is its own small optimization. A naive round to the nearest level is fine, but a rounder that learns the level boundaries on held out folds adds roughly three points of QWK over naive on the honest estimate for the classical baseline, and a smaller but real gain on the strong blends. It is free accuracy, as long as you measure it without letting it cheat.
From a plain baseline to the top of the board
With the scorer trusted and the measurement honest, the modeling was almost calm. I built it in layers, and each layer had to clear the honest bar before I let it in.
The first layer was a deliberately plain baseline, a ridge regression on surface features and hashed character and word n-grams. I skipped the heavier gradient boosted trees on purpose, because a full TF-IDF fit holds a learned vocabulary in memory and exhausted my 8GB laptop, while hashing is stateless and kept the peak near 550MB. This floor scored 72.81 on the test split, and I never spent a submission on it. That is far below the bar, which is exactly what a floor is for. It proved the pipeline ran end to end and wrote a valid submission, and it gave me a sanity number that the generalization gap was pointing the right way.
The real lift came from the four public checkpoints, run over gold preprocessing that I joined in from the corpus rather than recomputing. Scored honestly on the test split, the best single model reached about 85.2. Blending two of them cleared the bar at 86.12, and adding a model from a different family took it to 86.34. The step chart below replays that climb, with the board bar marked.
These Arabic models expect a specific tokenization, either a Word form or a morphological form called D3Tok, not raw text. For the development splits I got that for free: the corpus release ships the gold D3Tok column, and I joined it to the 2026 sentences by ID with 100% coverage. That meant byte exact preprocessing with no tokenizer of my own to get wrong, and it is a large part of why the checkpoints performed at their honest best. Example, the sentence مجلة كل الأولاد وكل البنات becomes مجلة كل ال+ أولاد و+ كل ال+ بنات.
Where adding models stopped helping
The obvious next move is to keep adding members to the ensemble. It is also where added effort often stops paying, because a blend only improves when a new member is both strong and different. Two accurate models that make the same mistakes add nothing when you average them. The four CAMeL checkpoints are close cousins, correlated 0.94 to 0.97 with each other on the validation split, so forward selection kept only two of them and dropped the rest as redundant.
Getting past the bar with margin needed a member from a different lineage. A QARiB model, a different backbone family, was the least correlated of the strong candidates at 0.94 against the blend, and it earned its place and lifted the blend to 86.34. After that the returns from borrowed checkpoints thinned out fast. I fine tuned two more models on Kaggle, a MARBERT and an AraELECTRA, and forward selection rejected both at that point. The reason is plainer than I first wrote it: MARBERT is no less correlated with the blend than QARiB, 0.942 against 0.943, and it is a point weaker on its own, so it brings nothing QARiB has not already brought. AraELECTRA is the one genuinely different candidate at 0.898, and at 79 QWK it is far too weak to pay for the difference. The figure plots each candidate by how strong it is and how correlated it is with the blend, and the region where a member actually helps is the upper right.
86.34 was the ceiling for a blend of borrowed checkpoints, and the stability check backs that up: the blended test score held between 86.19 and 86.32 whether I used two, three, or four members, so the result was not resting on a lucky pick. It was not the end of the campaign. Over the following weeks I ran a fine tuning grid on a university cluster, 113 completed runs over 13 backbones and 6 loss families across three input views, which together with the public checkpoints gave a pool of 172 candidate members. The system I actually shipped averages fourteen of them at equal weight: four public checkpoints, nine seed averaged families of my own fine tunes, and one single run.
Fourteen members is where the same lesson came back in a harder form. Once the ensemble was that wide it absorbed almost everything I put into it. Twenty different ways of combining the members, non negative least squares weights, ridge shrunk variants, direct QWK coordinate ascent, per member isotonic calibration, and median, trimmed and rank means, all landed between 84.74 and 85.12 against a plain equal weight baseline of 85.281. Every one of them was worse than dividing by fourteen. Member level improvements were absorbed too. A word to D3Tok lexicon lifted one member by 0.91 QWK on its own and moved the blend by somewhere between minus 0.02 and plus 0.001, because a cleaner member correlates more with the rest of the pack and trades its solo gain against lost diversity. Retraining the five strongest families on 13% more gold data moved the blend by 0.008.
So the honest ceiling was never a single number I could push past by adding models. It was the point where the blend stopped noticing what I did to it. Squeezing out more would take several strong and genuinely decorrelated models, which is real and uncertain effort, and I would rather report a number I can stand behind than chase one I cannot.
The blind test, and what actually did the work
The development board is a good proxy, but the real ranking comes from a blind test, a fresh set of 8,077 sentences with no labels released at the end. New sentences are where a solution meets the truth, because the free gold preprocessing I leaned on no longer exists. I had to compute the tokenization myself, and that is where the last honest lesson showed up.
Preprocessing here is load bearing. Feeding raw sentences to the Word model collapses it from about 84 QWK to about 50, because the model was never trained on untokenized text. Recreating the Word form myself matched the gold to 99.08% of tokens, which is faithful enough to keep the score intact. The morphological D3Tok form was harder, because the exact form was built with a licensed database I do not have, so my version matched about 76% of the gold. It looked at first as though that cost the affected model close to two points, from 84.68 down to 82.75, and on that reading I shipped a word only ensemble. The reading was wrong, and the mistake sat in the standardization rather than the tokenizer: I was taking that member's validation scores from the gold column while taking its blind scores from mine, and those are different input distributions that z scoring cannot reconcile. Recomputing validation through the same pipeline as the blind set made the member consistent with itself, and putting it back was worth 0.3 on the board, more than every calibration experiment that came after it. The rule I took from it is that a member's standardization statistics and the scores they are applied to have to come from one pipeline, end to end. The figure lets you switch preprocessing on and off and see the score fall.
The honest expectation I set before uploading was about 85 to 86, a touch under the development number, because an unseen set is harder and my D3Tok is imperfect. Every member also scored about 1.3 QWK lower on validation than on test, which pointed the same way. It landed at 85.3, first place in both tracks in the official results, so the expectation held.
What the offline board could not resolve
The offline board carried the development phase, but the blind set ships without labels, so it stopped existing at exactly the moment the real ranking began. What replaced it was a harder question: how small a difference can an offline estimate actually see? Midway through the campaign the same ensemble scored 85.44, 85.28 and 85.42 on the same split while I varied nothing but the resolution of the threshold search grid. Measured three ways, the noise floor of my offline evaluation sits between 0.09 and 0.44 QWK, which is as wide as the margins that decided the final ranking. Varying the fold map seed instead returns bit identical results, and that is a trap worth naming, because anyone estimating variance that way on a grid search rounder will conclude their procedure is deterministic when it is not.
The consequence is uncomfortable. In four paired tests where I had both an offline estimate and a live board reading of the same change, three of the four offline estimates collapsed to zero or reversed sign on the board. The change with the smallest offline estimate of the four, 0.086, was the only one that delivered, at 0.3. A MARBERTv2 member that measured 0.41 offline, the largest gain in either table, returned 0.00 and then minus 0.1. My own priority list, written before those probes, ranked recovering that model as the highest value action left. Following it as written would have shipped 84.7.
What rescued the measurement was that the two tracks score the same blind gold. Two byte identical files uploaded to Open and to Strict returned identical metric vectors, which proves it, and it turns the Open board into a free instrument for a Strict system as long as the Open runner up is far enough back that a losing probe costs no place. From there, every decision worth a submission was made from a live board reading rather than an offline one. The calibration step was the single lever whose effect cleared the noise floor. Cutpoints tuned on validation put the blind predictions in the wrong place, predicting level 12 for about 8% of blind sentences where the corpus prior puts roughly 21%, so I shipped cutpoints halfway between the validation tuned ones and cutpoints that match the blind prediction histogram to the training label prior. That midpoint scored 84.5 on the board, against 84.1 for the validation end and 84.0 for the full prior match. Both endpoints are worse than the middle.
The endgame followed from the same reading. With the mean provably capped, I stopped ranking candidates by their offline score and ranked them by how many rows they changed. Five near clones differing on 2% to 6% of rows all came back as ties, and so did every other candidate I probed. The board ran out of resolution before my pool ran out of candidates, and the ensemble I had at the start of the last day is the one I finished with, at 85.3 on both boards. The winner finished 0.1 ahead on Strict at 85.4 and 0.2 ahead on Open at 85.5, and both times the gap was exact accuracy, 38.7 and 39.5 against my 37.7, which is the one thing a blend of my members could not buy.
A correction belongs here, because the first file I submitted was not the one above. Four of its fourteen seats had been fine tuned on the training split plus the corpus's publicly released test split. I disclosed that to the organizers and in the paper, having read the rule as permitting it. On 2 September 2026 they ruled otherwise, and the entry was withdrawn and replaced by the same fourteen seats with those four swapped for their training split only versions, which I had also submitted during the testing phase and which the board scored at 85.3. The withdrawn file scored 85.4. So the extra gold labels bought 0.1 QWK on the board and 0.008 offline, both of which sit inside the noise floor this whole page is about.
What actually did the work
Looking back, the model was the least interesting part. The public checkpoints did the early lifting and anyone could have downloaded them, and the ten seats I added were ordinary fine tunes on a grid anyone with a cluster could run. What separated a top score from a middling one was the plumbing around them. A scorer I trusted enough to iterate offline. A refusal to believe the in sample numbers. A rounder measured without letting it cheat. Preprocessing joined from gold rather than guessed, and standardization that runs end to end through one pipeline. Knowing what my own measurements could and could not resolve. None of it is flashy, and all of it is the reason the result is one I can defend rather than one I got lucky with.
I also tried to leave myself honest limitations rather than a clean story. The noise floor evidence rests on four paired comparisons, which shows that offline and board can disagree but not how often. Board readings are single measurements on one fixed set of 8,077 rows and carry their own sampling noise, which I cannot estimate because I cannot resample the blind gold. My locally computed D3Tok is systematically degraded next to what a team with the licensed database would produce. And equal weights and a single calibration scalar are unrefuted at my measurement resolution rather than shown to be optimal. I would rather write those down than pretend they are not there.