ARPA-H committed $62.7M to build autonomous heart-failure agents — and gave a separate award to Stanford to build the thing that watches them. America has been running a supervisor for heart failure since 2012. Here is every hospital it grades.
The part of the ADVOCATE announcement worth reading twice isn’t the agents. Atman Health, UpDoc and Tempus are building those — systems that assess symptom severity, prescribe medications and order labs, with FDA authorization as the target.
The interesting line item went to Stanford: a disease-agnostic supervisory system that monitors the clinical agents after deployment, catching unsafe recommendations and out-of-distribution behaviour in real time. The government separated the agent from its supervisor, funded them as two jobs, and handed the supervisor to a different institution.
So it is worth asking what supervising heart-failure care actually looks like when you try it at national scale. We have fourteen years of that. It is called the Hospital Readmissions Reduction Program, and it grades every hospital in America on 30-day heart-failure readmissions.
Building the thing that acts is an engineering problem. Building the thing that grades it is a statistics problem, and it is the harder one.
Each dot is one hospital — 2,246 of them, every hospital with a published, non-suppressed heart-failure excess readmission ratio in the current file. Horizontal position is how many heart-failure discharges it had over three years. Vertical position is its grade.
An excess readmission ratio above 1.0 means CMS judged the hospital worse than expected for its patient mix; below 1.0, better. The line at 1.0 is where money changes hands.
Two thousand dots overplot badly, so there is a density view that bins them into hexagons and counts what is actually underneath — and a 95% funnel that draws the real binomial bounds around the expected rate, which is the honest way to read a plot shaped like this one.
1. Half the country is below average, which is not a finding. 1,149 of 2,246 hospitals — 51.2% — are graded worse than expected. That is roughly what you get by construction when you score everyone against a model fitted to everyone. The grade is a rank, dressed as a measurement. An agent-monitoring product that tells a health system “you are in the worse half” has told them nothing they can act on.
2. Most of the grades are not distinguishable from noise. Press show only the statistically distinguishable. Comparing each hospital’s crude readmission rate to its own CMS-expected rate with an ordinary binomial test, only 357 of 2,246 — 15.9% — separate from their own expectation at p<.05. And the ones that do are disproportionately large: 27.8% of hospitals with 800+ discharges clear the bar, against 12.1% of those under 100. For a 90-discharge hospital the confidence interval on its readmission rate is wide enough to swallow most of the national spread. The supervisor cannot see small things. ARPA-H’s supervisory agent will have exactly this problem the first time it tries to say whether one clinic’s agent is drifting.
3. The grade is not volume-neutral, and that is partly arithmetic. Drag the minimum-discharges slider and watch the “worse than expected” share fall. Among hospitals with fewer than 100 heart-failure discharges, 67.4% are graded worse than expected. Among those with 800 or more, 41.2% are. Mean ratio by size band runs 1.022 → 1.007 → 1.004 → 0.995 → 0.994.
The measure is calibrated so that predicted matches observed across discharges, not across hospitals. Big hospitals hold most of the discharges, so they pull the calibration toward themselves. The discharge-weighted mean ratio here is 0.9985 — almost exactly 1.0, as designed. The plain average across hospitals is 1.0043. The gap is the arithmetic of who the yardstick was built from. Some of the small-hospital penalty is real signal about rural heart-failure care; some of it is the shape of the estimator. Fourteen years in, CMS still cannot fully separate the two, and the ADVOCATE supervisor will inherit the same problem the moment it is asked to grade a rural clinic’s agent against an academic centre’s.
Switch the vertical axis to z-score and the picture inverts. That axis compares each hospital’s raw, unmodelled readmission rate to its CMS-expected rate. Small hospitals scatter enormously; large ones cluster.
Now switch back to the published ratio, and look at the standard deviation instead: 0.040 for the smallest band, 0.077 for the largest. The published grades for small hospitals are tighter than for big ones — the opposite of what raw sampling noise would produce. That is CMS’s hierarchical model doing its job, shrinking uncertain estimates toward the mean.
Which means the published number is a compromise between what the hospital did and what the model believes about hospitals in general. One hospital in this file had 14 readmissions out of 31 heart-failure discharges — a 45% crude rate against an expected 19.4%, a result that would be significant at p<.001 on its own. Its published excess readmission ratio is 1.105. Shrunk almost all the way back to normal.
That is a defensible modelling choice. It is also, from the perspective of anyone trying to catch a clinical agent misbehaving in one small clinic, exactly the wrong failure mode: the supervisor is designed to be conservative in precisely the setting where the harm is concentrated and the sample is thinnest.