AcuityBench Exposes Miscalibrated Clinical Triage Uncertainty

Choosing the appropriate urgency of care is a different problem from answering a medical question correctly. A response can contain clinically plausible information yet still direct a user to the wrong setting, with under-triage at the emergent end carrying particular safety consequences. AcuityBench evaluates that decision directly, separating agreement on clear cases from alignment with clinician disagreement when the case itself admits more than one defensible recommendation.

The benchmark harmonizes five public sources—user conversations, forum posts, clinical vignettes, and patient-portal messages—into a four-level scale: home monitoring, seeing a doctor within weeks, urgent outpatient care within 24–48 hours, and immediate emergency-department care. It contains 914 cases: 697 physician-consensus cases for standard accuracy analysis and 217 physician-confirmed ambiguous cases for uncertainty-aware evaluation. The authors test 12 proprietary and open-weight models in both an explicit four-way QA format and a free-response conversational format scored against the same acuity framework.

Core Contribution

That distinction makes AcuityBench more informative than a benchmark that reports only exact-match accuracy. It asks two separate questions: whether a model recognizes acuity when clinicians agree, and whether its distribution of recommendations reflects the uncertainty clinicians express when the presentation is genuinely borderline. For health-facing assistants, the second question is not cosmetic. An apparently decisive answer can be poorly calibrated even when its modal label is often acceptable.

Figure 1 lays out the pipeline from heterogeneous source data through label normalization and physician-panel annotation to the two evaluation formats.

Technical Approach

For each ambiguous case, the authors query a model five times at temperature 1.0 and convert the samples into an empirical distribution over the four ordered acuity labels. They compare that distribution with the physician-derived soft distribution using Jensen–Shannon divergence and Wasserstein-1 distance. Jensen–Shannon divergence measures distributional mismatch, while Wasserstein-1 preserves the ordinal structure: moving a recommendation from home monitoring to emergency care is treated as a larger error than moving it one adjacent level.

The paper also tests a panel-substitution scenario. A model replaces one physician rater, and the resulting panel is compared with the all-human panel through Krippendorff’s alpha, leave-one-out median changes, and the magnitude of those changes. This does not establish clinical usefulness in deployment; it is a targeted test of whether model outputs behave like one contributor to a clinician panel.

The comparison of QA and free response is equally consequential. A fixed-choice prompt exposes the model’s categorical preference directly, whereas a conversational response permits qualification and explanation. The reported format effect is systematic rather than uniformly beneficial: conversational outputs reduce over-triage but increase under-triage, especially for higher-acuity presentations. A more reassuring tone or broader explanation therefore should not be mistaken for safer triage.

Results and Analysis

On the 217 ambiguous cases, no individual model closely reproduced the physician uncertainty distribution. Table 15 reports Jensen–Shannon divergence from 0.245 to 0.327 and Wasserstein-1 distance from 0.816 to 0.957 across the evaluated models. The narrow spread is itself revealing: the paper’s concern is not confined to one provider or model family. Model samples were consistently more concentrated than physician judgments and under-represented the spread of clinical opinion.

Pooling GPT-5.4, Claude Opus 4.7, and Gemini 2.5 Pro improved distributional alignment: the frontier panel reached Jensen–Shannon divergence 0.191 and Wasserstein-1 0.738, versus individual-model means of 0.270 and 0.882. But the pooled samples selected one acuity label in 91.2% of ambiguous cases, while human raters showed comparably low entropy in only 8.3%. Ensembling makes the aggregate distribution closer to clinicians without solving the central calibration failure: it remains much more certain case by case.

The panel also skewed upward in acuity. Its mean ordinal position exceeded the human distribution by 0.320 levels; 66.4% of cases shifted upward and 28.6% downward. That directionality matters because a benchmark can show reasonable aggregate overlap while concealing a systematic tendency toward more intensive care recommendations.

Limits in Practice

The evidence is broad for a benchmark study, but it remains an evaluation rather than a clinical deployment study. The cases draw on public datasets and may include source-specific artifacts; the physician panel is drawn from emergency medicine; and the four-level framework abstracts care decisions that depend on patient context, resource access, and local practice. The conversational evaluation uses a single judge model, and the adjudication analysis is restricted to GPT-5.4. AcuityBench is therefore a useful stress test for clinical-acuity behavior, not a warrant to use any evaluated model for unsupervised patient triage.