clinicians.build · interactive · 14 sep 2026

The Scoreboard the Agent Is Aimed At

ARPA-H has put $62.7 million into autonomous heart-failure agents, with a supervisory AI to watch them, and six health system leaders who cannot say who is liable when it goes wrong. The outcome those agents exist to move is the 30-day readmission rate. Here is that measure for every hospital Medicare grades on it — and how much of the grade survives a look at its own sample size.

Primary source: Becker’s Hospital Review, “Who will carry the liability? 6 health system leaders on autonomous AI’s biggest question,” 11 September 2026
Data: CMS Hospital Readmissions Reduction Program, FY 2026 file (discharges 1 Jul 2021 – 30 Jun 2024), via MIMI Labs

Read the liability piece → The HRRP data → Or read today’s newsletter →

Six ADVOCATE awardees split the work. Atman Health, Tempus AI and Updoc build patient-facing heart-failure agents; Stanford builds a supervisory AI to watch them; Duke and Kaiser deploy. FDA package due in twenty-four months. Asked the obvious question, none of the six leaders Becker’s polled had an answer. The Christ Hospital’s Joy Oh put it best:

“If a supervisory agent fails to detect an inappropriate recommendation that results in patient harm, who is ultimately responsible: the developer of the worker agent or the developer of the supervisory agent?”

Before that question can be answered, a quieter one has to be: can you even tell? Liability runs on attribution, and attribution runs on measurement. So look at the measurement that already exists — the one CMS has been using to dock hospital payments since 2012, the one these agents are built to improve.

Across 2,316 hospitals, Medicare counted 858,817 heart-failure discharges and 169,065 readmissions inside thirty days over three years — a national rate of 19.7%. CMS grades each hospital with an excess readmission ratio: its predicted rate over the rate expected for its case mix. Above 1.00 is worse than expected. 1,187 hospitals are above it. 1,129 are below.

Every graded hospital, plotted against the size of its own sample

Each dot is one hospital. Horizontal is how many cases it had. Vertical is observed readmissions over expected — 1.00 means exactly what its case mix predicts. The shaded funnel is the range you would get from chance alone at that volume. Dots inside it are hospitals whose grade you cannot distinguish from a coin landing badly.

Condition graded — all six HRRP cohorts

Minimum cases all hospitals

State

Confidence required before a hospital counts as distinguishable

Hospitals shown

—

Worse beyond chance

—

Better beyond chance

—

Indistinguishable

—

Observed ÷ expected 30-day readmissions, by case volume

The same hospitals as a ranked strip — CMS excess readmission ratio, one tick each

worse beyond chance better beyond chance indistinguishable from its own expected rate · hover any dot · case axis is logarithmic

Three things the shape tells you

1. Most of the scoreboard is not a measurement. CMS sorts these hospitals almost exactly in half — 1,187 worse than expected, 1,129 better. Test each one against its own expected count and 338 of 2,316 (14.6%) differ from it beyond chance at p < 0.05. The other 1,978 — 85.4% — are indistinguishable. Tighten the bar to the three-sigma limit a funnel plot conventionally uses and only 60 hospitals (2.6%) survive.

2. The reason is volume, and the volumes are small. The median graded hospital had 274 heart-failure discharges in three years — about 91 a year. 849 hospitals (36.7%) had fewer than 200. Raise the minimum-cases slider and watch the grey swallow the edges: among hospitals with fewer than 200 cases, 10.4% are distinguishable; among those with 500 or more, 22.7%. Same measure, different amount of evidence.

3. CMS already knows this, which is the interesting part. Press show CMS’s ratio instead. The cloud collapses toward 1.00, because the excess readmission ratio is not a raw rate — it comes out of a hierarchical model that shrinks each hospital toward the national average in proportion to how little data it has. The raw ratios plotted here run from 0.45 to 2.32; CMS’s published ratios for the same hospitals run from 0.71 to 1.36. The statistical fix is already inside the measure. It is just invisible to everyone who reads the ranking.

80/20 — the ruler nobody holds up

Denver Health’s Daniel Kortsch, MD, told Becker’s the evidence under the AI-supervises-AI model covers four conditions, none of them heart failure, with a headline trial of 32 patients. The dashed navy line on the chart sits at 32 cases. At the national heart-failure rate, a 32-patient sample’s 95% interval spans 6.3% to 31.3% — an observed-over-expected range of 0.32 to 1.59. A trial that size cannot tell a hospital that cuts readmissions by a third from one that raises them by half. The smallest hospital CMS grades here has 31 cases, and CMS declines to publish a ratio below that. Whatever the ADVOCATE agents do to readmissions, the instrument that would catch it needs hundreds of patients per site, per year, for years.

Where this analysis is weak — and it is, in four specific places

The funnel test and the CMS ratio are not the same statistic, deliberately. The dots are coloured by an exact two-sided binomial test of each hospital’s observed readmission count against its own CMS-published expected rate. CMS’s excess readmission ratio uses predicted over expected from a hierarchical logistic model with hospital random effects. The first asks “is this hospital’s raw count surprising?” The second asks “after borrowing strength from every other hospital, where does this one sit?” Both are defensible. They answer different questions, and the second is the one that carries money.
The funnel band is drawn at the pooled national rate; the dot colours use each hospital’s own expected rate. Expected rates in the heart-failure cohort range from about 17% to 23% on case mix, so a handful of dots sit just outside the band without being flagged, and vice versa. The band is an orientation device. The colour is the test.
Suppression cuts the left edge off. CMS does not publish a ratio for hospitals with too few cases, so the smallest hospitals in the country are simply absent — not zero, absent. 6,987 hospital-condition rows in the FY 2026 file carry a footnote instead of a number. The hospitals where volume makes measurement hardest are the ones you cannot see here at all.
Readmission is a contested outcome in its own right. Thirty-day all-cause readmission moves with observation-status coding, with discharge-to-hospice rates, and with how sick a hospital’s catchment is beyond what the risk model adjusts for. None of that is noise in the statistical sense, and none of it is fixed by a bigger sample.

If you build clinical AI that promises an outcome

The liability question in today’s briefing has a measurement question sitting underneath it, and the measurement question is answerable now.

Ask what n your claim needs before you make the claim. A 10% relative reduction in heart-failure readmissions is a real clinical win and, at one median-sized hospital, statistically invisible for years. If your pilot is one service line at one site, the honest deliverable is a process metric, not an outcome.
The supervisory agent inherits this. A model that watches another model for “inappropriate recommendations” needs a labelled rate of inappropriate recommendations to be evaluated against. Nobody in the ADVOCATE architecture has said who produces those labels, and the disagreement rate between two agents is not the same quantity as the harm rate.
Capture the override. The disengagement dataset is the asset. Every time a clinician overrides the agent is a labelled event you will never get retrospectively, and it is the only thing that turns twenty-four months of deployment into evidence rather than usage.