Encoder Choice Changes Pathology Fréchet Scores

Fréchet-distance evaluation is now common in digital pathology, but the metric has inherited an unstable assumption from natural-image FID: that the feature encoder can be treated as a fixed measuring instrument. Pathology papers often replace Inception-v3 with a histology foundation model, then read the resulting distance as if it were directly comparable across studies. This paper shows that the encoder is not a neutral detail. For the same pair of tile sets, the raw distance changes by about thirty-fold across six encoders, enough to change the practical conclusion of a benchmark.

The authors introduce HistoFID as a calibration protocol rather than a new representation model. Its main move is simple: compute each Fréchet distance relative to that encoder's own within-cohort floor, using sample-size matching so that the denominator reflects the minimum distance the encoder reports between matched draws from the same source. Figure 1 lays out the workflow: two cohorts, six feature extractors, normalization by the encoder-specific baseline, and evaluation across drift, perturbation, compression, stain processing, and generative-model comparisons.

Core Contribution

The paper's central contribution is the empirical case that raw pathology Fréchet distances are encoder-calibrated quantities, not universal scores. Inception-v3, Phikon-v2, CONCH, UNI2-h, Virchow2, and Prov-GigaPath do not differ only in embedding dimension or domain relevance; they impose different sensitivity profiles on the same image-set comparison. That matters because the metric is often used to rank generative models, compression schemes, stain normalization, or cohort shift.

The proposed ratio score makes those comparisons less encoder-dependent. Instead of asking whether a raw distance is large in absolute terms, HistoFID asks how far the comparison sits above the encoder's own same-cohort floor. This does not make every encoder interchangeable, but it changes the unit of comparison from an arbitrary feature-space scale to a cohort-calibrated scale.

Technical Approach

The evaluation uses a held-out in-house cohort of about 500,000 H&E and immunohistochemistry tiles from 2,119 slides, plus a public TCGA BRCA cohort of 100 slides. For each encoder, the authors extract tile features, fit Gaussian summaries to image-set embeddings, and compute the standard Fréchet distance between the fitted means and covariances. They then repeat the calculation under controlled settings: within-cohort splits, cross-cohort drift, stain shift, perturbations, compression, stain normalization, and two generative models.

Figure 2 is the key calibration result. The same within-cohort comparison yields a roughly thirty-fold spread in raw distance across encoders. After dividing by each encoder's own within-cohort floor, the across-encoder coefficient of variation drops sharply on two independent comparisons. The reported reductions are about 89% within cohort and 58% across cohorts, which is the strongest evidence that ratio normalization is doing more than rescaling a single example.

The paper also separates the encoders into two empirical groups. CONCH, Phikon-v2, and Inception-v3 behave as the more sensitive group; UNI2-h, Virchow2, and Prov-GigaPath are more invariant. That split reappears across cross-cohort drift, stain shift, and generative evaluation, shown in Figure 3. The practical implication is uncomfortable: a generative model can look better or worse depending on which feature extractor defines “realism.”

Results and Analysis

The strongest result is not that one encoder wins. It is that unnormalized Fréchet distances can be misleading even when all encoders are plausible choices for pathology. A thirty-fold raw-score spread for a fixed comparison is too large to treat as noise, and the fact that the ordering does not follow embedding dimension rules out a simple dimensionality explanation. The ratio score gives practitioners a more defensible reporting convention: name the encoder, report the raw distance if useful, but interpret it against that encoder's own floor.

The generative-model experiment is the clearest downstream example. The authors report that the sensitive-versus-invariant encoder split can decide which generated image set is judged more realistic. That does not mean HistoFID resolves generative evaluation by itself; it means papers that report only one encoder's raw Fréchet distance may be reporting an encoder preference as much as a model property.

The slide-level result points to a second limitation of pooled patch distances. An attention-pooling encoder registers per-slide composition that pooled patch Fréchet distance misses, raising the distance about 320-fold on matched cohorts. This is a reminder that patch-level feature distributions can erase clinically relevant structure when the task depends on slide composition rather than local texture alone.

The codec experiment applies the same protocol to TuroCompress, a proprietary pathology codec. Under the tested settings, it reaches the highest reconstruction fidelity at the smallest file size among the codecs compared. That result is useful, but it is also secondary to the paper's broader message: pathology image evaluation needs calibration, because the measuring network can change the answer.