One record, one claim, three things to return
MedLongTrust-EHR gives a system a synthetic longitudinal record and a claim about it, and asks for three things back: a trust label, the risk types that apply, and the paragraph identifiers that support the decision. The records come from Synthea, so they read cleaner than real clinical notes, and each one is a sequence of paragraphs. A header announces an encounter, an entity line records a condition or a medication or an allergy, an observation line gives a measurement, and some paragraphs are marked supplemental and act as distractors.
The trust label is one of three values: trusted, untrusted, or insufficient evidence. Those three labels are the spine of the whole task, so I gave each one its own colour and kept it consistent through every figure on this page.
The board scores three measures and weights them. Trust-label accuracy counts for half, risk micro-F1 for three tenths, and evidence F1 for the last fifth. Multiply the weighted sum by a hundred and you have the final score. Token usage is recorded as optional metadata and does not enter the score. Move the three sliders below and watch the number respond. The weights are fixed; only the components move.
The data comes in three splits. Training has 84 records and 1,167 questions, development has 18 records and 265 questions, and both carry gold answers. The test split has 18 records and 266 questions with the labels held back, and it is the one the leaderboard scores. The claims fall into six templates, with one synthesis variant, and working out which template a claim belongs to is the first thing the system does.
I measured the model before I trusted it
The design came out of a number, not a preference. Before building anything, I took my small local model, llama3.2:3b run through Ollama, and used it the usual way: as a free-form verifier that reads the record and judges the claim on its own.
It matched a direct reading of the record only about 60 to 70 percent of the time, and it made a particular mistake often, treating observation-trend claims as though they were conditions. On the two jobs I could measure directly against the tools, it agreed on routing 143 times out of 266, and on whether the retrieved paragraphs actually supported the decision only 36 times out of 266. A component that is wrong this often, and wrong in ways that are hard to see, is not something I want deciding trust in a clinical setting.
So I moved the decision out of the model. The tools decide; the model advises.
That measured unreliability is not a footnote to the entry. It is the reason for the architecture. Toggle the figure to let the small model decide the 266 test claims, and then to let the tools decide, and watch the difference. The tools are not more clever. They are just checkable.
Four tools read the record, and the model stays out of the way
The system has two layers. A deterministic tool layer reads the record and produces the answer. A language-model layer plans and audits around it and holds no authority over the result.
Four tools do the work. A template router matches the claim wording to one of the six templates. An evidence retriever finds the paragraphs that justify the decision. A trend comparator handles observation claims, mapping encounter identifiers to numbers and comparing values as a number together with a unit. And a label and risk deriver turns a routed, grounded claim into a trust label and its risk types by a rule per template.
Pick a claim type below and follow it through. The router reads the wording, the right retrieval tool gathers the evidence, and the deriver returns the label. The model sits to the side the whole time.
Template router
Matches the claim text to one of the six templates, or recognises the multi-evidence synthesis variant. It reads the wording of the claim, not the model's opinion of it.
Evidence retriever
Finds the paragraphs that justify the decision, at the lowest-numbered non-supplemental encounter that documents the entity. This is the fifth of the score most entries left thin.
Trend comparator
Maps encounter identifiers to numbers, then compares observation values as a number together with a unit, so 2 and 2.0 in the same unit count as equal.
Label and risk deriver
Turns a routed, grounded claim into a trust label and its risk types by a rule per template. Risk types are a fixed function of the label and the template.
The part almost everyone skipped
Evidence F1 is only a fifth of the score, and it is the fifth most entries gave up on. Recovering the exact paragraphs that support a decision is fiddly, so most entries returned a label and left the evidence thin. The second-place system scored 0.34 on evidence. Getting it to 1.00 is most of my seventeen-point margin.
The rule is specific. For a condition, medication, or allergy claim, the evidence is the encounter header together with the line that documents the entity, taken at the lowest-numbered encounter that is not supplemental and that documents that entity. One detail matters: within a record, a trusted claim and its untrusted counterpart share a single anchor, so an untrusted claim built on a made-up entity inherits the evidence of its trusted sibling. A synthesis claim, which names a condition, a medication, and a procedure together, resolves instead to three documenting lines and no header.
Below is one record laid out as encounters, left to right. Pick a claim and watch the retriever sweep across, skip the supplemental paragraphs, and stop at the lowest encounter that documents the entity. That pair of lines is the evidence.
The hardest trend claim is a cross-indicator one, where the named observation does not appear at the claim's own encounters because its values were lifted from a different indicator. There the comparator recovers the named reading from wherever it occurs in the record, taking the one nearest in time to each value endpoint. It is a small share of the questions and it took a real share of the work, and getting it right is part of why the evidence score reached 1.00 rather than merely close.
Where the perfect label score comes from
I owe you a clear account of the one uncomfortable part. The trust labels are exact on the labeled data, and for several templates that exactness does not come from clinical reasoning. It comes from a property of how the benchmark was assembled.
For condition, medication, observation-trend, and sufficiency claims, and for the synthesis variant, the label keys on whether the question carries a proposed response. On this benchmark that presence is a steady per-template signal, and reading it directly is what makes the labels perfect. I treat it as a construction artifact, not as evidence about the patient. Allergy and privacy labels never use it; they are read from the claim wording.
Because that signal might not survive a differently built test, I left the record-grounded fallback wired in behind a toggle rather than deleting it once the perfect score arrived. An audit of the test questions found no template, unit, or observation name that had not already appeared in the labeled data, so the signal almost certainly carried over, which is what the leaderboard showed. That is a statement about this benchmark, not a claim that the labeling would survive a test built to remove the signal.
Where it landed, and why the gap is that size
The same configuration went to the held-out test board as the lean-planner entry and came back first, with a final score of 100.0 and trust accuracy, risk micro-F1, and evidence F1 each at 1.00, recorded on 7 July 2026.
The second entry scored 82.899 and the third 68.79. The height of the first bar is not the interesting part. The gap between first and second is, and most of it is one component. On the label and the risks the field was close. On evidence F1 the second-place entry scored 0.34 against my 1.00, so most of the seventeen-point margin comes down to that single measure. Toggle the evidence overlay to see it.
The entry I submitted was not the zero-token one. The tools alone would score the same, because grounding is their job, but I ran the lean planner so the entry reflected the system as an agent with an active planning layer. It reported about 25,800 tokens over the test split. A fuller monitoring mode that audits every retrieval reported about 47,300. All three modes return the same labels, risks, and evidence, and since tokens do not enter the score, running the planner rather than the bare tools cost nothing on the board.
What I would write in the limitations section
The exact scores rest on the benchmark's construction, and I have said where. If a future split breaks the proposed-response pattern, label accuracy falls to about 0.86, and since the risk types are read off the labels they fall with them, so the final score settles in the high 80s, while the evidence stays exact. The records are synthetic, so their language is cleaner than real clinical notes. And the small local model, useful here as an advisory monitor, was not reliable enough to be the decision maker, which is the finding that shaped the whole design rather than a detail tacked on at the end.
What actually did the work
Looking back, the model was the least interesting part. What separated this entry from the rest was the evidence reconstruction, the measurement that kept the model out of the decision, and a set of rules I can read and test line by line. None of it is flashy, and all of it is the reason a reviewer can follow every decision back to the record lines that justify it, which is the property I care about most for clinical verification. It does not show that free-form model judgment is ready for the same job, and it does not claim the labeling would survive a benchmark built to remove the signal it leans on. Those are the questions I would take up next.