Eight frontier models answered 304 pediatric cardiology board questions. The top score was 98.4% with retrieval switched off. Turning on a single‑textbook RAG pipeline added almost nothing — and at that ceiling, on that many questions, it could not have shown you if it had.
Boston Children’s wrapped eight frontier models in a pediatric cardiology chatbot and ran them against 304 board-style questions. The top models hit 98.4% with no retrieval at all. Enabling a single-textbook RAG pipeline barely moved the needle.
The interesting part is not that RAG failed. It is that on a benchmark shaped like this one, RAG was never going to be able to succeed visibly. Here is the whole benchmark, one square per question.
The curve is the smallest true improvement a benchmark of a given size could actually detect — 80% power, two-sided, at the baseline accuracy you set. The horizontal line is the lift you are claiming. Anywhere the line sits under the curve, your benchmark cannot tell that improvement apart from noise, no matter what the table says.
Near a ceiling, benchmarks stop being able to see. At 98.4% there are five wrong answers in 304 — a RAG pipeline that fixed every single one would move the score 1.6 points, which is inside the confidence interval of the score you started with. Before you buy a retrieval layer, check whether your evaluation could detect it working. If it cannot, you are not measuring retrieval. You are measuring the questions.
Statistical power is the polite objection. The impolite one is that a board question and a patient are not the same object.
One pre-digested paragraph. Every fact you need is present, nothing irrelevant is. One correct answer, written by someone who knew it before they wrote the stem. No missing labs, no contradictory notes, no family in the room.
~1 document · 0 conflicts · 1 right answer
Years of notes from clinicians who disagreed, labs drawn at the wrong time, imaging read twice, an outside-records blob nobody reconciled, a medication list that is wrong in a way the chart cannot show. The answer may not exist yet.
hundreds of documents · conflicts everywhere · answer unknown
If you can’t name the failure mode your retrieval layer fixes, you built a cost center with a citation feature.
Which leaves the honest case for RAG intact and much narrower than the pitch deck: you build retrieval when you need the answer to point at a source someone can open during a chart review. You are buying defensibility, not accuracy. Price it that way and the decision gets easy.