Stanford gave away an evaluator for AI-generated medical text. Here is every one of the 90 model‑by‑task scores underneath it, on one screen — and the single task that holds up the paper’s cleanest correlation.
For two years the honest answer to “how do you know your scribe is safe” was that nobody had built the instrument. This week a Stanford group published one: an evaluator trained to judge whether AI-generated medical text is factually consistent with the input it was given — code, benchmark, and a 4B model you can run on a laptop, all public.
It is a real contribution and the paper is unusually candid about its own limits. So the useful thing to do with it is not to praise it. It is to look underneath.
The evaluator got distilled. The taxonomy didn’t.
MedVAL‑Bench is 840 physician-annotated outputs across six clinical tasks. Ten language models were scored against it; five of them were also MedVAL‑distilled. That is 90 model‑by‑task cells, and every one is below.
Set the axis to inter-physician agreement and show MedVAL GPT‑4o only. You get r = 0.67 — the same figure the paper reports. The evaluator scores highest on exactly the tasks where the physicians agreed with each other most.
Now click query2question to drop it. The correlation does not weaken. It inverts. One task out of six — the one where twelve physicians reached only α = 0.560 among themselves — is carrying the entire relationship.
Then push the minimum-task-size slider up. At n ≥ 135 you are fitting three points, and r goes to roughly 1.0, which means nothing at all. Both directions are the same lesson: six tasks is not enough points to have a trend in.
The abstract says average F1 rises from 66% to 83%. That is the binary judgment — safe (risk levels 1–2) versus unsafe (levels 3–4). The paper also reports the four‑class risk grade, which is the thing a reviewer would actually act on, and there the same distillation moves F1 from 36.7% to 51.0%.
The claim doing the most work in coverage of this paper is that the evaluator is statistically non-inferior to a single human expert. Here is the actual scope of that test, from the paper’s own limitations section.
| Item | Value |
|---|---|
| Model tested | MedVAL GPT‑4o — not the released 4B |
| Examples used | 90 of 840 (15 × 6 tasks) |
| Comparator | one randomly sampled physician |
| Reference label | majority consensus of the panel |
| Margin | LCB₀.₀₅ (Δ) > −0.05 |
| p-value | < 0.001 |
| Observed Δ | not printed |
| Human expert F1 | not printed |
Switch the axis above to input length. The six tasks average 10, 50, 69, 82, 543 and 1,497 tokens of input. The medication‑answer task — where the models score best after distillation — hands the evaluator a ten‑token question.
A real encounter is not ten tokens. It is not fifteen hundred either. It is a chart: prior notes, an active problem list, a medication reconciliation, labs with trends, imaging with priors, and the three things the patient said that nobody wrote down.
11,007 distinct ICD‑10 principal diagnosis codes appeared across 12,203,787 Medicare inpatient stays in a single year.
Underneath the method sits a physician-defined taxonomy: four risk levels, eleven named error types, twelve physicians, and a decision about how much each kind of wrongness costs. The most common error in the whole set is a fabricated claim, at 45.7%. Second is a missing claim, at 14.0%.
The model is a commodity now. The judgment about what constitutes a dangerous omission in a discharge summary is not, and it doesn’t come from an engineer.
The build: the transferable asset is an annotated corpus for your own population, graded against your own taxonomy. Start with 100 outputs from the tool you already run, two clinicians, and this paper’s four levels. You will learn more from disagreement between your two annotators than from any published F1.