Encoder Benchmarking
Comparing frozen tile encoders on your own cohort by training a small classifier on top of each one’s features — cheap to run, and easy to design so badly that the answer means nothing.
What it is
You have several pretrained encoders and one cohort, and you want to know which encoder to use. The standard answer is linear probing on frozen features: run every slide through each encoder once, freeze the encoder, train only a small classifier on the resulting feature vectors, and compare. Because the encoder never updates, feature extraction happens once per encoder and every subsequent experiment is minutes rather than GPU-days.
What it measures is how linearly separable your classes already are in that encoder’s representation. That is a real and useful quantity, but it is not the same as how the encoder would perform after fine-tuning, and it is not a deployment estimate. It is a screening test.
The largest published example is Bareja et al. 2026 (sources/papers/bareja-2026-foundation-model-benchmark.md), 32 models across 41 tasks. Its protocol is worth having written down because it is a reasonable default and it is fully specified: tile at 20× / 0.5 µm per pixel into 256×256 non-overlapping tiles, Otsu tissue mask, up to 4,000 tiles per slide, resize to each model’s native input, extract features, average the tile vectors, then a linear layer — identical hyperparameters for every model (LR 0.001, batch 16, 30 epochs, no augmentation), 70/30 patient-level split, 1,000 bootstrap replicates of the test set for intervals.
The four choices that decide whether the result means anything
Each of these is a place where the same encoders can be made to rank differently.
1. Pooling — and this is the big one. Averaging up to 4,000 tile vectors into one slide vector dilutes anything focal. In the benchmark above, slide-level AUROC across the top five encoders sat at 0.743–0.761 while patch-level AUROC on curated benchmarks sat at 0.944–0.962. Part of that gap is task difficulty; part of it is that mean pooling throws away exactly the localised signal a diagnosis often rests on. Attention-MIL (Weakly Supervised Learning) is the alternative, and it introduces trainable parameters — which is why a benchmark wanting a clean encoder comparison reaches for mean pooling and accepts the cost. Whichever you pick, the ranking you get is conditional on it. Nobody has published the same encoder set under both.
2. Splitting. Patient-level, not slide-level and certainly not tile-level. One patient contributing several blocks, or one tumour region photographed at four objectives, will otherwise appear on both sides. This is the failure mode that most often produces an impressive number from a broken experiment — and it is not hypothetical in benchmark datasets either: BreakHis contains multiple magnifications of the same region from ~82 patients and is routinely split at image level. See External Validation.
3. Which cohort counts as external. An encoder pretrained on TCGA scored on a TCGA task is being asked to recall, not to generalise. The design worth copying is the tiered one: an in-domain tier, a public-but-independent tier, and an institution-specific tier, reported separately rather than averaged into one number. TCGA carries the table of which encoders declare it.
4. How you compare the numbers. Two traps. Averaging AUROC across tasks unweighted lets a 19-slide task move the headline as much as a 7,500-patch one. And comparing categories of model with a two-sample t-test over pooled model×task scores treats those scores as independent when the same tasks recur under every model and the same model recurs across every task — the standard fix is a mixed-effects model with random intercepts for both. The benchmark above does the second of these, and it is the test underpinning most of its positive claims; the analyses it does pair correctly within task mostly return nulls.
Why it matters for my work
Because the published answer is “it doesn’t matter much”, the local experiment is now cheaper and more interesting than the encoder choice it was meant to inform. The top five encoders were statistically indistinguishable on slide-level tasks. If that holds on a local cohort, encoder selection collapses to an engineering decision — feature dimension, inference cost, licence, whether the weights are already on the drive — and the effort moves to pooling, splitting and stain handling instead.
The machinery is already here. PinkKidney has seven tile encoders in one extraction script with per-slide features cached, patient-level 5-fold cross-validation, and a passing label-shuffle control — which is a stronger internal design than the published benchmark’s single 70/30 split. Foundation Model Cache holds nearly every model in that benchmark locally. So the marginal cost of running a properly designed version of this is a probe head and a comparison script.
It also produced this wiki’s one genuinely novel benchmarking result. On the kidney cohort the accuracy comparison saturated — UNI at 1.000 — and told nobody anything. What separated the encoders was representation geometry: the signal-to-batch silhouette ratio, where UNI (1.787) was the only encoder whose biological signal outweighed its scanner signal. That is the lesson to carry: when a comparison saturates on accuracy, the discriminating metric is a property of the feature space, not a downstream score.
How it connects
Foundation Models in Pathology — the things being benchmarked, and what the published comparison found; this page is how you would check it on your own material.
Foundation Model Cache — the local inventory that makes this a script rather than a download, and the open question of which of those models has ever actually been loaded successfully.
Weakly Supervised Learning — supplies the aggregation step, and choice 1 above is really a choice between this and mean pooling.
External Validation — choice 3 restated: a benchmark that never leaves its development distribution is measuring recall of the pretraining set.
TCGA — the in-domain tier, and the table of which encoders declare it in pretraining.
Scanner and Stain Variability — the confounder a good encoder should factor out, and the thing the signal-to-batch ratio actually measures.
Stain Normalisation — a decision this protocol still has to make, because frozen features are not immune to it; benchmark evidence shows UNI-2 embeddings shifting substantially under staining variation.
Class Imbalance and Accuracy — why AUROC alone is not enough on these tasks, and why AUPRC and per-class reporting belong in the comparison.
Patch Extraction — the tiling parameters this protocol depends on; magnification and tile size set what the encoder can physically see.
QuPath Atölye — its patoloji-temel-modeller appendix already teaches this material to pathologists, covering the same 13 encoders and a section on batch effect and shortcut learning, but cites no benchmark; the comparison on this page is the missing citation.
Bareja 2026 — A Benchmark Study of Vision and Pathology Foundation Models — the largest worked example of exactly this: 32 foundation models through one frozen-feature linear-probing protocol on 41 tasks, with TCGA held apart, and the top five pathology encoders statistically indistinguishable at slide level. That last result is the argument for comparing representation geometry rather than accuracy. Update 2026-08-23: the supplement that note flags as unavailable — Supplementary Data 1–15 with the pairwise Wald tests and FDR-adjusted p-values, the 26-page Supplementary Information, and the Transparent Peer Review file — is served on the nature.com article page and is now filed beside the paper note as bareja-2026-foundation-model-benchmark-supplementary-* (gitignored, this machine only). The note’s [unverified] markers on the FDR family and per-comparison statistics are now checkable; none of the files has been reviewed yet.
Open questions
- Does the encoder ranking survive a swap from mean pooling to attention-MIL? Nobody has published both for the same encoder set, and PinkKidney could answer it with cached features and no new extraction. A null result would be the more useful outcome, because it would mean encoder choice matters even less than currently thought.
- What is the right discriminating metric once accuracy saturates? Signal-to-batch silhouette worked once, on one cohort where scanner and magnification were perfectly collinear. Whether it generalises as a selection criterion is untested.
[unverified] - Nothing here records the compute and storage cost of a full extraction pass per encoder, which is the actual constraint on how many models can be compared. PinkKidney caches one
.ptper slide across 356 slides and ten encoder variants, so the number is readable off that tree. - Should a local comparison bother with the 8 public patch benchmarks at all, or go straight to slide-level on local cohorts? The patch tier is where the published benchmark found its only significant differences — and also where its splits are least defensible, so the two facts may be the same fact.