Supervision Audits Expose Chest X-Ray Label Failure

Chest-radiograph repositories make large-scale model development possible, but their labels often originate in report text rather than direct image review. That distinction matters when a model is meant to detect a visual finding: a report can omit a finding visible on the image, document it indirectly, or prioritize another clinical issue. The authors test this gap for cardiomegaly in MIMIC-CXR and argue that validating supervision should precede conventional training and benchmarking.

Core Contribution

The paper introduces Repository Supervision Auditing (RSA), a workflow for treating repository labels as hypotheses rather than image-level truth. RSA compares those labels with expert image annotations, identifies why they disagree, then uses the reviewed cases to construct a balanced, quality-controlled cohort for model development and deployment-oriented evaluation.

The central finding is uncomfortable for a common weak-supervision assumption. Repository-derived cardiomegaly labels identified only 1% of cases confirmed by expert image-level review. The disagreement was driven mainly by non-mention in the source report, rather than an explicit report negation. The authors further report that expert-confirmed cardiomegaly appeared in nearly half of studies assigned a repository-derived No Finding label. In this setting, the repository target is not merely noisy; it is systematically misaligned with the visual phenotype the classifier is expected to recognize.

Technical Approach

RSA separates label validation from downstream model assessment. First, repository-derived labels are audited against radiologist-reviewed image annotations to establish a curated reference standard. The resulting set is then balanced and quality controlled before being used for model development and evaluation. This ordering is the methodological point: performance against unvalidated repository labels cannot establish that a model detects the intended image-level condition.

Figure 1 lays out that sequence from repository labels through expert validation to a curated cohort. It frames the audit as an upstream quality-control stage, rather than a post hoc error analysis after a model has already been trained.

For the case study, the authors train a DenseNet121 on the expert-curated cardiomegaly cohort. They evaluate discrimination, calibration, clinically relevant cohort behavior, and subgroup differences by age and sex. The paper also uses Grad-CAM examples to check whether predictions appear to depend on the cardiac silhouette rather than implanted cardiac hardware, an especially relevant concern because devices can correlate with cardiac disease without being the target finding itself.

Results and Analysis

On the curated test set, DenseNet121 achieved ROC-AUC 0.853. That is evidence that a standard image classifier can discriminate cardiomegaly once the evaluation target has been brought closer to expert image judgment. It is not evidence that the original repository labels were adequate training or evaluation truth: the reported 1% capture of expert-confirmed cases points in the opposite direction.

The deployment-oriented analysis reports a Brier score of 0.162 and expected calibration error of 0.099. These are useful additions to AUC because discrimination alone does not establish whether predicted probabilities can support clinical thresholds. Figure 2 also presents cohort-specific analyses and age- and sex-stratified differences in AUC and sensitivity with 95% confidence intervals. The available figure caption establishes that these checks were performed, but the paper's reported material here does not justify a numerical claim of subgroup parity.

The Grad-CAM examples support a narrower interpretation. In representative true-positive studies with and without implanted hardware, activation concentrates on the enlarged cardiac silhouette rather than the device. A false-negative example shifts attention away from that silhouette in the presence of bibasilar atelectasis and reduced lung volumes, which the authors identify as a plausible reason for the miss. Such visualizations are diagnostic evidence, not proof of causal feature use, but they make the proposed failure modes concrete.

Limits in Practice

The empirical study is a single-disease case study centered on cardiomegaly in MIMIC-CXR. Its conclusion about supervision auditing is persuasive, while its numerical model result remains specific to one curated cohort, one architecture, and one repository. The work also does not establish prospective clinical utility, nor does an AUC of 0.853 settle calibration or subgroup performance for a particular deployment population. Its main practical contribution is therefore a validation discipline: audit whether a label represents the image-level task before treating model metrics as clinical evidence.