Reference Choices Reshuffle Chest X-Ray Model Rankings

Chest X-ray machine learning often treats the evaluation reference as a background detail: report-derived labels for pathology classifiers, and generic image-quality metrics for reconstruction or enhancement. This paper argues that the reference is part of the clinical claim. If a model is evaluated against labels extracted from reports, it is being judged against what was documented in text, not necessarily what was visible on the image or what a clinician would prefer for a downstream task.

The authors test that claim in two settings: pathology classification and diagnostic image quality assessment. For classification, they collect paired expert image-derived and report-derived labels in a Cambridge University Hospitals cohort, then compare those references with a curated MIMIC-CXR subset. For IQA, they compare algorithmic scores against expert judgments of diagnostic usability. The central result is not that one reference is universally correct. It is that different plausible references select different winners.

Core Contribution

The paper’s main contribution is XQA-RR, a paired-reference evaluation setup for chest X-ray models. It places image-derived labels, report-derived labels, conflict-resolved image labels, report-section labels, model families, and rank-correlation analysis into the same evaluation frame. That design lets the authors ask a more operational question than whether a model has a high ROC-AUC: would the same model look best if the benchmark used a different clinically motivated reference?

Figure 3 gives the study design in compact form: paired labeling sources are used to compare how evaluation references affect both absolute performance estimates and model ordering.

The distinction matters because report labels encode clinical communication. Reports may omit stable, irrelevant, or already-known findings; they may also use uncertainty differently from image annotators. The paper shows that these are not small clerical differences. They change positive counts, uncertainty rates, agreement coefficients, and model rankings.

Technical Approach

For classification, the authors evaluate supervised chest X-ray classifiers, including ResNet and DenseNet variants, alongside zero-shot and fine-tuned vision-language models such as MedKLIP, GLoRIA, and ConVIRT. The evaluation uses four selected pathologies with sufficient positives — atelectasis, consolidation, lung opacity, and pleural effusion — plus aggregate analyses across 12 pathologies where available.

Agreement is measured with Cohen’s $\kappa$, including a quadratically weighted central estimate and low/high mappings that treat uncertain labels pessimistically or optimistically. Ranking stability is measured with Spearman and Kendall rank correlation coefficients over model rankings induced by different references. The paper defines SRCC as:

where $d_i$ is the rank difference for item $i$. For ranking analyses, the authors estimate uncertainty with 1000 bootstrap replicates over patient studies and report 95% percentile confidence intervals.

Figure 4 shows why simple label substitution is risky. Across all 12 pathologies, 469 of 794 positive conflict-resolved image labels, or 59%, were labeled negative in reports; in the opposite direction, 245 of 618 positive report-derived labels, or 40%, were negative in the conflict-resolved image labels. The omission pattern is asymmetric and pathology-dependent.

Results and Analysis

The agreement results support the paper’s central caution. Image annotators agree less with each other than report annotators: the paper reports image inter-annotator $\kappa = 0.15$ for atelectasis and $\kappa = 0.67$ for pleural effusion, while report inter-annotator agreement is consistently high at about $\kappa = 0.80$ to $0.94$. Cross-source agreement between conflict-resolved image labels and report labels is lower than report inter-annotator agreement, with consolidation around $\kappa \approx 0.22$ and pleural effusion around $\kappa \approx 0.55$.

The label-fusion analysis is useful because MIMIC-CXR lacks the same conflict-resolution process. On XQA-RR, FUSE_OR, MACE, and neural fusion methods align best with conflict-resolved image labels. The appendix reports FUSE_OR macro-average $\kappa = 0.87$ across the four selected pathologies and neural fusion variants around $\kappa = 0.89$ to $0.90$, while simpler methods such as DS-EM and FUSE_AND are weaker on the harder pathologies.

The model-ranking results are the strongest practical signal. On MIMIC-CXR, rankings induced by labels from Findings and Impressions are highly correlated for all models, with macro-average SRCC 0.86 for Findings and 0.96 for Impressions in one table, but the correlation drops sharply for fine-tuned vision-language models: macro-average SRCC 0.73 versus Findings and 0.59 versus Impressions. Kendall correlations show the same pattern, with fine-tuned vision-language macro-average KRCC 0.13 versus Findings and 0.11 versus Impressions. In other words, the ranking instability is not evenly distributed; it depends on the model family and the reference.

Figure 8 makes the same point visually for XQA-RR: rank correlations vary by pathology and model group, so an evaluation reference can change which model appears ready for further development.

The IQA part extends the argument beyond classification. The authors compare full-reference, no-reference, and clinician-preference-based image quality assessment and report that common generic metrics such as SSIM and PSNR often do not align with expert judgments of diagnostic usability. The implication is narrow but important: benchmark convenience is not the same as clinical validity. For teams choosing CXR models or reconstruction methods, the reference standard should be justified with respect to the finding, the task, and the intended clinical use, not inherited from dataset availability.