ARPA-H funded the heart-failure agents — and gave Stanford a separate award to build the thing that watches them. Before anyone builds that, look at the supervisor America has been running on heart failure since 2012, and where it bends.
Everyone is building the clinical agent. Almost nobody is building the thing that watches it. This week the federal government split those into two line items and funded them both — heart-failure agents that can prescribe, and a separate Stanford team whose only job is to sit above those agents and catch them drifting.
That second job sounds new. It isn’t. CMS has graded every hospital in America on 30-day heart-failure readmissions since 2012, with money attached. It is the largest supervisory system ever built for a single clinical condition, and its record is the best available evidence about what happens when you try.
The hard part was never measuring. The hard part is that the measurement means something different depending on how much of it you have.
Every hospital CMS grades on heart failure, flowing left to right through three questions. How much data did the supervisor have? (discharge volume). What did it conclude? (excess readmission ratio above or below 1.0). And then the one nobody asks: does that conclusion survive an ordinary significance test?
Look at where the ribbons go. 1,149 hospitals are graded worse than expected. 945 of them — 82.2% — cannot be told apart from their own CMS-expected rate by a plain binomial test. Only 204 are confirmed worse. On the other side, 1,097 hospitals are graded better and only 153 are confirmed better. In total 1,889 of 2,246 hospitals — 84.1% — are carrying a public grade, and money, on a number that is not statistically separable from “average.”
One honest note in CMS’s favour, visible as an absence: zero hospitals are contradicted. Not one hospital graded better is confirmed worse by the raw signal, and not one graded worse is confirmed better. The model is not inventing the direction. It is just very quiet about it.
The volume bands do not feed the two grades evenly. 67.4% of hospitals with fewer than 100 heart-failure discharges flow into “graded worse.” For hospitals with 800 or more, it is 41.2%. The band means march the same way: 1.022 → 1.007 → 1.004 → 0.995 → 0.994.
The obvious reading is that small hospitals deliver worse heart-failure care. Part of it probably is that. But part of it is arithmetic, and the arithmetic matters more for anyone building a supervisor.
CMS calibrates the model so that predicted matches observed across discharges, not across hospitals. Large hospitals hold most of the discharges, so they define where 1.0 sits. Sure enough: the discharge-weighted mean ratio in this file is 0.9985 — essentially exactly 1.0, as designed — while the plain average across hospitals is 1.0043. Everything below the largest band has to sit somewhere above the line for that to balance. A supervisor calibrated on volume will always look harsher to the low-volume sites. ARPA-H’s ADVOCATE program is explicitly targeting rural and underserved care; Duke’s validation platform spans five health systems and rural sites. Those are exactly the sites a volume-calibrated supervisor grades badly for reasons that have nothing to do with the agent.
Press dim everything that doesn’t survive and watch which ribbons stay lit. They come overwhelmingly from the bottom of the diagram. 27.8% of hospitals with 800 or more discharges clear the significance bar; only 12.1% of those under 100 do. A 90-discharge hospital’s confidence interval is wide enough to swallow most of the national spread.
There is a second, subtler thing happening. Look at the standard deviation of the published grades by band: 0.040 in the smallest, rising to 0.077 in the largest. Raw sampling noise would do the reverse — small samples scatter more, not less. The published numbers are tighter for small hospitals because CMS’s hierarchical model deliberately pulls uncertain estimates back toward the mean.
Which means the published grade is a compromise between what the hospital did and what the model believes about hospitals in general. One hospital in this file recorded 14 readmissions out of 31 heart-failure discharges — a 45% crude rate against a 19.4% expected rate, clearing p<.001 standing alone. Its published excess readmission ratio is 1.105. Shrunk nearly all the way back to normal.
One hospital in this file recorded 14 readmissions out of 31 heart-failure discharges — a 45% crude rate against a 19.4% expected rate, a result that clears p<.001 standing alone. Its published excess readmission ratio is 1.105. Shrunk nearly all the way back to normal.
Shrinkage is the right call for paying hospitals and the wrong call for catching a drifting agent. A supervisory system that pulls low-volume estimates toward the mean is, by construction, slowest to flag the single clinic where something has gone wrong — which is where agent failures will actually show up first. If you are building in this space, the product is not “a dashboard of ratios.” It is an explicit answer to: how many patients does this agent have to hurt before your monitor says so, and who gets paged at that threshold? Write that number down. Nobody selling agent-monitoring today will.
That is the shape of the problem ADVOCATE just funded someone to solve. Duke’s validation platform spans five health systems and rural sites; Kaiser’s deployment covers 21 medical centers and 260+ clinics. Stanford’s supervisor has to say something useful about all of them — including the ones that, on fourteen years of precedent, it will not be able to see.