Foundation Models in Pathology
Large image encoders pretrained on huge unlabelled slide collections, used as a ready-made feature extractor so that downstream tasks need far fewer labelled cases.
What it is
A foundation model in pathology is a neural network trained on very large numbers of tissue tiles without diagnostic labels, using self-supervised objectives — tasks the model can generate its own supervision for, such as matching two augmented crops of the same tile or predicting masked-out regions. The result is an encoder that turns any tile into a numeric feature vector capturing tissue appearance.
You then do not train an image model at all for your task. You run your slides through the frozen encoder, get feature vectors per tile, and train a small aggregator — often the same attention-MIL head described in Weakly Supervised Learning — on top. Because the heavy representation learning already happened, the downstream task needs far fewer labelled slides.
The trade-offs are real and worth stating plainly. The encoder is a black box trained on data you did not choose, so its blind spots are inherited and hard to audit. Feature extraction over a whole cohort is computationally expensive up front. And “foundation model” covers a wide range of quality — which pretraining corpus, how many slides, and from how many institutions, all matter more than the label does.
There are now two generations, and the difference is which step you stop hand-building. The description above is the first: a tile encoder, with the slide-level aggregation still trained locally. The second generation pretrains that aggregation too — a slide-level model consumes tile embeddings and returns one vector, or a text description, for the whole slide. TITAN, PRISM, CHIEF, Prov-GigaPath, mSTAR and EAGLE are all of this kind, and all are already on the Foundation Model Cache drive.
That matters more than it sounds, because of where the data actually runs out. The tile encoder is pretrained on hundreds of thousands of slides; the MIL head on top is trained on however many cases one department has. The aggregator is therefore the part of the pipeline learning the most from the least, and it is the part a slide-level model replaces. Whether that helps at the cohort sizes available here is untested — but it is a swap at the end of an existing pipeline rather than a new one, because the cached tile features are already the required input.
The top few are, on current evidence, interchangeable — and that is the most useful thing known about them. The largest independent comparison to date (sources/papers/bareja-2026-foundation-model-benchmark.md) ran 32 models through one identical frozen-feature linear-probing protocol across 41 tasks. On the 14 slide-level tasks outside TCGA — the setting closest to real work — the top five (UNI, H-optimus-0, Virchow2, Prov-GigaPath, H-optimus-mini) sat between 0.743 and 0.761 mean AUROC with no statistically significant difference between any pair, on any of AUROC, AUPRC, sensitivity or specificity. Differences only appeared on curated patch-level benchmarks, where Virchow2 (0.962) separated from the rest at FDR p<0.001 — and where the absolute numbers, 0.95 against 0.75, show the two tiers are not measuring the same difficulty of problem.
Two further results from the same benchmark are worth carrying:
- Pathology-specific pretraining did not significantly beat general vision models on any tier — TCGA, CPTAC, external or out-of-domain. Path-VMs occupy more top-10 slots (47% of available positions), which the authors read as consistency rather than superiority. That is a fair reading, and it is a much weaker claim than the field usually makes.
- Scaling does not transfer. Larger architectures and larger pretraining corpora helped on TCGA multi-class tasks and essentially nowhere else: ViT-B → ViT-L gave no significant gain outside TCGA, and going from a medium (0.1–1M WSI) to a large (>1M WSI) pretraining corpus was non-significant everywhere. So “newest and biggest” is not a selection rule.
The practical consequence is that encoder choice among the leading few is an engineering decision, not an accuracy one — pick on feature dimension, licence, inference cost and whether the weights are already local, because the benchmark cannot separate them on performance. Two caveats bound this: the comparison used mean pooling rather than attention-MIL for slide-level aggregation, so it ranks encoders under mean pooling; and its category and scaling comparisons used t-tests that treat model×task scores as independent when they are clustered by both. See Encoder Benchmarking for what the protocol does and does not settle.
They are not automatically robust to staining variation. This is worth stating separately because the opposite is widely assumed. Khan et al. 2026 (sources/papers/khan-2026-stain-normalisation-benchmark.md) embedded colon tiles from 66 laboratories with UNI-2 and found the feature-space representation shifted substantially under staining variation and under every one of eight normalisation methods tested. Their conclusion is that the foundation model “appears not to be robust against the intensity variation/normalization.” Two cautions on how far to take that: the evidence is t-SNE inspection rather than a quantitative shift metric or a downstream accuracy drop, and a shift in feature space is not by itself proof of worse task performance. The safe reading is that a foundation-model pipeline still has to make a stain decision rather than inheriting immunity from pretraining.
Why it matters for my work
The group now has its own measurement, and it points somewhere more interesting than accuracy. PinkKidney ran UNI against three general-purpose encoders on the same 44-patient cohort, the same folds and the same MIL heads. On accuracy the ordering is unsurprising — UNI 1.000, DINOv2-large 0.971, ViT-B/16 0.945–0.970, ResNet50 0.908–0.910. The informative number is what happens to the batch structure in each feature space, on a cohort spanning three scanners:
| Encoder | Class silhouette | Magnification silhouette | Signal-to-batch |
|---|---|---|---|
| UNI | 0.191 | 0.097 | 1.787 |
| ResNet50-ImageNet | 0.064 | 0.070 | 0.800 |
| DINOv2-large | 0.072 | 0.082 | 0.782 |
| ViT-B/16-ImageNet | 0.059 | 0.139 | 0.393 |
UNI is the only encoder tested whose biological signal outweighs its batch signal. In the others the scanner is as prominent as the diagnosis, or more so. For context, the same batch effect measured on raw HistoQC slide descriptors gives a signal-to-batch ratio of 0.13 — magnification is recoverable from those descriptors at kNN accuracy 0.968. So pathology-specific pretraining is not merely extracting better features; on this cohort it is substantially factoring out the acquisition variable that dominates the raw statistics.
Two honest caveats. Scanner and magnification are perfectly collinear in that cohort, so this is a combined acquisition effect rather than a magnification effect specifically. And the accuracy comparison is saturated at 1.000 for UNI, so the silhouette ratios carry more information than the AUCs do — which is itself the reason to look at representation geometry rather than only at downstream metrics.
Liver Tru-Cut — Tumour Origin AI is set up as a head-to-head comparison of weakly supervised and foundation-model-based approaches. That comparison is the practical question for the group: does buying into a pretrained encoder actually beat training from scratch on the cohort sizes available at a single institution? Liver core biopsies are small samples with limited tissue, which is precisely where a strong pretrained representation should help most.
Separately, index3.qmd lists “Foundation Models in Computational Pathology” as a reading-note row — but that table is explicitly a placeholder template, so no summary exists yet. [unverified]
How it connects
Weakly Supervised Learning — the two are not alternatives so much as layers: a foundation model supplies the tile features, MIL supplies the slide-level aggregation. The second generation above absorbs that second layer as well, which makes MIL a baseline to beat rather than a fixed part of the design.
Foundation Model Cache — which of these models are actually on hand locally, and what each one is for; the inventory that turns this concept page into a set of available options.
Scanner and Stain Variability — a good pretraining corpus spanning many institutions is often argued to be the main defence against scanner-specific features. Benchmark evidence qualifies that: UNI-2 embeddings still moved substantially under staining variation, so pretraining reduces the problem rather than removing it.
Stain Normalisation — the decision a foundation-model pipeline still has to make, and the evidence on which method to make it with.
External Validation — a foundation-model pipeline can memorise institution-level cues just as easily as a from-scratch model; the external cohort is still the only real test.
Whole Slide Imaging — feature extraction runs per tile at a chosen magnification, so the pyramid level chosen here determines what the encoder ever sees.
Multimodal Fusion — the strongest practical argument for a pretrained encoder outside the low-label setting: in a fused model the weakest modality encoder drags the whole result down, and MultiSurv’s ImageNet-pretrained ResNeXt was weak enough that adding images made its predictions worse.
TCGA — most public encoders are pretrained partly on TCGA, so a TCGA-derived benchmark is not a clean held-out test of one. That page now carries the declared-corpus table for 16 encoders, so this is checkable rather than assumed.
Encoder Benchmarking — how you would find out which encoder to use on a local cohort, and the protocol choices (pooling, split, statistical model) that decide whether the answer means anything.
Agentic Slide Navigation — the asymmetry that motivates it: a stronger tile encoder makes the reading better and the navigation no better, because no pretraining corpus records where an expert chose to look.
Spatial Proteomics — the same frozen-encoder idea in a modality where the input is N arbitrary named marker channels rather than three fixed colours. That single change is what forces a separate class of model, and it is a useful check on how much of “foundation model” is the pretraining and how much is the fixed input format.
Feature-Space Distance — how to put a number on the difference between two sets of tiles in one of these encoders, and why that number is uninterpretable until it is divided by the encoder’s own within-cohort floor.
Aknar–Mutlu Model Family — the only project here that proposes to build one of these rather than download it: OpenMidnight LoRA-adapted to a Turkish multi-organ cohort, with beating the base model on Turkish tasks as its stated success criterion. It is the concrete test of whether local adaptation buys anything over the public encoders listed above.
Meningioma H&E Deep Learning — the sharpest local demand on these features: it asks an H&E encoder to recover methylation-defined molecular groups, a label no pathologist assigns by eye, so it probes what the pretrained representation carries rather than how well it imitates a reader.
Awesome-AI4DigitalPathology — an outside curated index whose two foundation-model sections hold 53 entries between them, and the closest thing to a complete published inventory of this landscape.
Open questions
- Which specific foundation model(s) will the liver project evaluate? Not recorded. [unverified] PinkKidney has UNI, UNI2-h, Virchow2, Phikon-v2, H-optimus-0, CONCH and Midnight already wired into one extraction script, so the marginal cost of reusing that harness is low. One option is now closed rather than open: TOAD, the published tumour-origin model, cannot be used as an off-the-shelf comparator — its trained head was never released and its “encoder” is a generic ImageNet ResNet50. That was established on 2026-05-08 and is recorded on Foundation Model Cache.
- Does a pretrained slide-level aggregator beat a locally trained MIL head? On the one cohort where it has been tried, no — and by a wide margin. Liver Tru-Cut — Tumour Origin AI compared zero-shot slide-level models against a supervised ABMIL head on 509 liver biopsies. On the binary task, a trivial always-predict-the-majority baseline scores 0.862; supervised ABMIL on H-optimus-1 features reaches 0.895 (κ 0.498); and every zero-shot model falls below the baseline, TITAN included at 0.728. Two cautions before generalising: this compares zero-shot slide models against a trained head, which is not the same experiment as fine-tuning a slide encoder, and the task is heavily imbalanced. But the direction is clear and the honest reading is that a pretrained aggregator used out of the box does not replace a MIL head trained on local labels.
- The stronger finding from that comparison is about accuracy itself. Seven vision-language models scored between 0.102 and 0.644 on the five-class task and all of them lost to the majority class on the binary one. A number that looks reasonable in isolation was worse than learning nothing — which is Class Imbalance and Accuracy demonstrated on real clinical data rather than argued in the abstract.
- Does a pretrained slide-level aggregator beat a locally trained MIL head? Untested here, and newly cheap to test: six such models sit on the Foundation Model Cache drive and PinkKidney already caches the tile features they consume. The honest expectation is that the gain is largest where local labels are scarcest, which points at Liver Tru-Cut — Tumour Origin AI rather than at the saturated kidney cohort. The 2026 benchmark does not answer this, despite including TITAN and Prov-GigaPath. Its protocol extracts a feature vector per 224×224 tile and averages them, which is a shape only a tile encoder fits — and it also ran TITAN on eight patch-level tasks where no slide exists at all. So the slide-level aggregators were almost certainly evaluated through their tile encoders, making TITAN’s mid-table finish evidence about CONCH-family patch features rather than about pretrained aggregation. The paper never states which weights it loaded.
[unverified]on the specific mechanism; what is certain is that the question remains open. - Would the encoder ordering survive a swap from mean pooling to attention-MIL? The benchmark’s slide-level ranking was produced by averaging up to 4,000 tile vectors, which dilutes focal signal by construction — plausibly part of why slide-level AUROC sits at 0.75 against 0.95 on patch tasks. PinkKidney has the cached features and multiple encoders to test this cheaply, and it is the more informative experiment because a null result there would mean encoder choice matters even less than the benchmark suggests.
Is there a plan to compare a general-purpose encoder against a pathology-specific one?Answered, and done. PinkKidney ran exactly that comparison; the table above is the result. The follow-up question is better: the comparison saturated on accuracy, so what is the right discriminating metric for encoder choice? Representation geometry — the signal-to-batch ratio — separated the encoders cleanly where AUC could not, and that is the transferable finding.- Nothing in the repo yet records the compute and storage cost of caching features for a cohort — a practical blocker worth measuring before committing. PinkKidney caches one
.ptper slide across 356 slides and ten encoder variants, so the number could be read off that tree directly. - Does the embedding shift under staining variation actually degrade downstream accuracy, or only move the feature cloud? Khan et al. show the shift but never test the consequence. This is directly testable on Labquality EQA Staining Dataset, which is free to download, and would settle whether normalisation is needed before feature extraction in Liver Tru-Cut — Tumour Origin AI. The missing instrument now exists: Feature-Space Distance gives a quantitative shift metric with a floor for “no difference”, replacing the t-SNE inspection this page complains about above — and five of its six encoders are already on the Foundation Model Cache drive.
- Which encoder should be used to detect a shift, as opposed to to survive one? Bhat et al. 2026 find encoders split into a sensitive tier (CONCH, Phikon-v2, Inception-v3) and an invariant tier (UNI2-h, Virchow2, Prov-GigaPath), and that the split does not follow the training objective — DINOv2 encoders sit in both. Heavy pretraining augmentation buys robustness by teaching a model to ignore the very nuisances a drift detector must see. That reframes encoder choice as depending on the question rather than on a ranking, and nothing here has been chosen that way.