One sentence, nineteen reading levels
BAREC is the Balanced Arabic Readability Evaluation Corpus, and the 2026 shared task asks a model to read a single sentence and predict how hard it is to read, on a nineteen point scale. The training set is about 54,845 sentences drawn from 1,518 documents, with a fixed validation split of 7,310 and a test split of 7,286. Every sentence carries a gold level, so this is ordinary supervised learning with an unusual target.
The label is ordered, and that changes everything about how you are graded. The board ranks entries by quadratic weighted kappa, or QWK, which is a version of agreement that cares how far off you are, not just whether you were exactly right. Predict level 9 when the truth is 8 and the penalty is small. Predict 12 and the penalty is much larger, because the weight grows with the square of the distance. A mistake three levels away costs nine times what a one level mistake costs. That single fact set the whole shape of my solution, so I wanted to feel it before I trained anything.
Here is what the scale feels like in practice. The same kind of thought, written five ways, gets harder to read as the level climbs. This is real Arabic, colored from the easy end of the ramp to the hard end.
Drag the prediction below and watch the penalty grow. The true level sits fixed while your guess moves, and the readout shows how many levels off you are and how heavy that costs relative to a near miss. The shape of that cost is the reason a model that is roughly right everywhere beats a model that is exactly right often but occasionally wild.
There are two tracks with the same test set. The Open track (Codabench 16544) lets you train on anything, and a parallel Strict track (16545) allows only BAREC data. I aimed at the Open board first, where the live bar to beat was 85.80 QWK, held by two entries tied at the top. Last year's winning system landed at 86.4, so that number sat in my head as the real ceiling worth reaching.
A grader I could run myself
A public leaderboard gives you five submissions a day and a single number back. That is a slow way to learn anything, and it tempts you to tune against the board by trial and error, which can lead to overfitting the leaderboard rather than the underlying task. So before I trained a single model I did the least glamorous work of the whole project: I rebuilt the official scorer locally and proved it matched, line for line.
The shared task ships its evaluation code, so I wrote my own metrics module and held it to a byte parity test against the official eval.py on the provided fixture. It reproduces every figure the grader prints: QWK 80.0040%, accuracy 56.6211%, adjacent accuracy 69.8632%, average distance 1.143776, and the coarser 7, 5, and 3 level accuracies. Not close. Identical. That test became the gate the rest of the work had to pass through.
Then came the part that turned a five per day board into an unlimited one. The 2026 release ships gold labels for the test split, and that test split is what the Open board scores. So my local score on the test set is the board score. When the first real submission went up and came back at 86.10, all seven metrics matched my local numbers exactly, which confirmed the grader in my harness is the grader on the server. From that point I could measure the exact board effect of any change without spending a submission on it.
The edge was not a secret model. It was a scorer I trusted enough to stop guessing against the board.
The flattering number and the honest one
Having a trustworthy scorer only helps if you feed it honest predictions. The most useful models to try were the public CAMeL Lab readability checkpoints, four of them, already fine tuned for exactly this task. When I scored them on the training set they looked spectacular, 89 to 95 QWK. If I had believed those numbers I would have tuned the whole ensemble on them and walked into a wall.
The reason is contamination, and the harness caught it. Those checkpoints were trained on the BAREC corpus, which is the same text as the 2026 training split. Their training predictions are in sample, so they memorize rather than generalize. On clean holdout data the same models score 82 to 85, which is the honest picture. The gap of ten or more points between the flattering number and the honest one is a large part of why disciplined measurement matters here. Toggle the figure to see which number you would have trusted.
Tune on validation, evaluate once on test, and let the honest out of fold number decide every go or no go. For the rounding step in particular I used a nested out of fold estimate, the number you get by rounding on data the rounder never saw, because a rounder tuned and scored on the same fold will always look better than it is. That one habit is the difference between a method that holds up on new sentences and one that quietly falls apart.
There is a smaller version of the same idea in the rounding itself. The model outputs a continuous score, and turning that into an integer level is its own small optimization. A naive round to the nearest level is fine, but a rounder that learns the level boundaries on held out folds adds roughly three points of QWK over naive on the honest estimate for the classical baseline, and a smaller but real gain on the strong blends. It is free accuracy, as long as you measure it without letting it cheat.
From a plain baseline to the top of the board
With the scorer trusted and the measurement honest, the modeling was almost calm. I built it in layers, and each layer had to clear the honest bar before I let it in.
The first layer was a deliberately plain baseline, a ridge regression on surface features and hashed character and word n-grams. I skipped the heavier gradient boosted trees on purpose, because the full n-gram fit exhausted the memory on my 8GB laptop, and hashing kept the peak under 550MB. This floor scored about 73 on the board. That is far below the bar, which is exactly what a floor is for. It proved the pipeline ran end to end and wrote a valid submission, and it gave me a sanity number that the generalization gap was pointing the right way.
The real lift came from the four public checkpoints, run over gold preprocessing that I joined in from the corpus rather than recomputing. Scored honestly, the best single model reached about 85.2. Blending two of them cleared the bar at 86.12, and adding a model from a different family took it to 86.34, which is level with last year's winning system. The step chart below replays that climb, with the board bar and the prior winner marked.
These Arabic models expect a specific tokenization, either a Word form or a morphological form called D3Tok, not raw text. For the development splits I got that for free: the corpus release ships the gold D3Tok column, and I joined it to the 2026 sentences by ID with 100% coverage. That meant byte exact preprocessing with no tokenizer of my own to get wrong, and it is a large part of why the checkpoints performed at their honest best. Example, the sentence مجلة كل الأولاد وكل البنات becomes مجلة كل ال+ أولاد و+ كل ال+ بنات.
Where adding models stopped helping
The obvious next move is to keep adding members to the ensemble. It is also where added effort often stops paying, because a blend only improves when a new member is both strong and different. Two accurate models that make the same mistakes add nothing when you average them. The four CAMeL checkpoints are close cousins, correlated 0.94 to 0.97 with each other, so forward selection kept only two of them and dropped the rest as redundant.
Getting past the bar with margin needed a member from a different lineage. A QARiB model, a different backbone family, was decorrelated enough and strong enough to earn its place and lifted the blend to 86.34. After that the returns thinned out fast. I fine tuned two more models on Kaggle, a MARBERT and an AraELECTRA. Both were rejected by forward selection, one because it was strong but too similar at 0.93 correlation, the other because it was different but too weak at 79 QWK. The figure plots each candidate by how strong it is and how correlated it is with the blend, and the region where a member actually helps is the upper left.
So I stopped at 86.34, and I am comfortable calling it the honest ceiling for this particular ensemble. The stability check backs that up: the blended test score held between 86.19 and 86.32 whether I used two, three, or four members, so the result is not resting on a lucky pick. Squeezing out more would take several strong and genuinely decorrelated models, which is real and uncertain effort for maybe a fifth of a point, and I would rather report a number I can stand behind than chase one I cannot.
The blind test, and what actually did the work
The development board is a good proxy, but the real ranking comes from a blind test, a fresh set of 8,077 sentences with no labels released at the end. New sentences are where a solution meets the truth, because the free gold preprocessing I leaned on no longer exists. I had to compute the tokenization myself, and that is where the last honest lesson showed up.
Preprocessing here is not a nicety, it is load bearing. Feeding raw sentences to the Word model collapses it from about 84 QWK to about 50, because the model was never trained on untokenized text. Recreating the Word form myself matched the gold to 99.08% of tokens, which is faithful enough to keep the score intact. The morphological D3Tok form was harder, because the exact form was built with a licensed database I do not have, so my version matched about 76% of the gold and cost the affected model close to two points. The figure lets you switch preprocessing on and off and see the score fall.
So the honest expectation for the blind set is about 85 to 86, a touch under the development number, because an unseen set is harder and my D3Tok is imperfect. That is the number I would put my name on, and I would rather set the expectation there than promise the development figure and hope. The submissions are prepared and validated in the exact upload format, waiting on the one step I keep for a human: I built and checked the file, and the actual upload to the public board is mine to send, not something I automate.
What actually did the work
Looking back, the model was the least interesting part. The public checkpoints did most of the heavy lifting, and anyone could have downloaded them. What separated a top score from a middling one was the plumbing around them. A scorer I trusted enough to iterate offline. A refusal to believe the in sample numbers. A rounder measured without letting it cheat. Preprocessing joined from gold rather than guessed. And the discipline to stop when the ensemble stopped paying. None of it is flashy, and all of it is the reason the number held up.
I also tried to leave myself honest limitations rather than a clean story. The margin over the bar is real but not huge, and it is somewhat split dependent. The blind number will likely come in under the development one. The Strict track result depends on one member whose training data I could not fully verify, so the provably compliant version of that blend is a little lower. I would rather write those down than pretend they are not there.