CAPRA Audits Hidden Medical Imaging Subgroups Without Metadata

Medical imaging models are often audited with demographic, acquisition, or quality metadata, but those fields are frequently absent at deployment. That creates a practical failure mode: a classifier can look acceptable in aggregate while failing on a clinically meaningful slice that cannot be named from the available records. This paper introduces CAPRA, a calibrated proxy-axis framework for finding and reusing subgroup structure when the original metadata are missing.

The authors frame the problem as hidden subgroup analysis rather than ordinary fairness auditing. CAPRA does not assume that the deployment set contains group labels. Instead, it estimates semantic axes from the images themselves, calibrates those estimates with a small metadata-labeled split, and exposes the resulting posterior probabilities as a subgroup interface for analysis and optional downstream training.

Core Contribution

The main contribution is the separation between metadata availability during calibration and metadata availability during deployment. Prior subgroup methods often need labels for each group, or they infer latent slices that are hard to interpret clinically. CAPRA sits between those extremes: proxy teachers predict semantic axes, calibration maps those predictions into more reliable posteriors, and the system ranks axes by how much they expose subgroup failure.

That matters because the output is not just a latent cluster assignment. The paper’s interface is meant to say which explicit axis is driving failure, how much support the axis has, and whether the same axis can be reused for reweighting. Figure 1 gives the clearest view of this pipeline: proxy teachers produce axis tokens, a metadata-labeled calibration split corrects them, and the calibrated posteriors feed both auditing and optional training-time reweighting.

Technical Approach

CAPRA starts by defining a set of image-derived semantic axes. These can correspond to visual or clinical attributes that proxy models can estimate from the image even when the deployment metadata table is empty. The paper then uses patient-level cross-fitting on a small labeled calibration split, which is a useful design choice in medicine: without patient-level separation, calibration can look better than it is because correlated studies from the same patient leak across folds.

After calibration, CAPRA represents each image by posterior probabilities over the proxy axes. Those posteriors support two related uses. The standalone auditing mode, CAPRA-S, searches for explicit failure axes by combining support filtering with metrics such as Gap BA@20 and Worst-group Acc.@20. The downstream mode can pass the same interface to group-aware or subgroup-aware learners when labels are unavailable in the target setting.

The comparison set is also important. The paper contrasts CAPRA against image-only partitions, ExMap-style explainable failure maps, and true metadata partitions where available. Figure 3 shows the qualitative part of that comparison: the same embedding is partitioned four ways within each dataset, making it easier to see when CAPRA aligns with an explicit failure axis rather than merely carving arbitrary visual clusters.

Results and Analysis

The evaluation spans three medical imaging domains: BRSET for fundus imaging, HAM10000 for dermoscopy, and CheXpert for chest radiography. Across those settings, the paper reports that CAPRA exposes subgroup disparity patterns that metadata-only slicing can miss and that its partitions align more closely with explicit failure axes than image-only or latent-slice alternatives. The strongest evidence is visual and comparative: the standalone CAPRA failure maps in Figure 2 place candidate axes by support-filtered Gap BA@20 and support-filtered Worst-group Acc.@20, with a dominant failure axis highlighted for each domain.

The calibration-budget sweep in Figure 4 addresses a practical question: how much labeled metadata is needed before the proxy interface becomes useful. The paper shows this sweep for BRSET and HAM10000, which is the right experiment for the stated deployment problem. The claim is not that metadata becomes irrelevant; it is that a small labeled calibration set can be converted into a reusable proxy interface for larger metadata-free audits.

The evidence is promising but not unlimited. The abstract states that downstream robust-learning gains are domain-dependent, so CAPRA should be read first as an auditing interface and only second as a training recipe. Its value depends on the quality and coverage of the proxy teachers, the availability of a labeled calibration split, and whether the chosen semantic axes actually cover the clinically relevant hidden failures. When those conditions hold, CAPRA offers a concrete path for post-deployment subgroup analysis without requiring a complete metadata table.