Bareja 2026 — A Benchmark Study of Vision and Pathology Foundation Models
Thirty-two foundation models pushed through one identical frozen-feature linear-probing protocol on 41 pathology tasks, with TCGA deliberately walled off from every other dataset — and the finding that the top five pathology encoders are statistically indistinguishable on slide-level tasks, while neither model size nor pretraining corpus size reliably predicts anything outside TCGA.
Read status: full text, no supplement. The main article was read end to end. But this paper pushes almost all of its inferential statistics into Supplementary Data 1–15 and Supplementary Figures 1–25, none of which are in this PDF. So the direction of every claim below is from the paper’s own prose, and the exact test statistics behind them are unread. Where a criticism depends on something only the supplement could settle, it is marked
[unverified].
What they did
Took 32 pretrained models, froze every one of them, extracted features once per task, and trained an identical linear classifier on top. Nothing was fine-tuned. The point is a controlled comparison of representation quality, not of deployable performance — and that distinction does a lot of work later.
The four model categories (their Table 1), with the models actually tested:
| Category | n | Models |
|---|---|---|
| Path-VM — pathology vision | 16 | cTransPath, Kaiko, HIPT, Virchow, Lunit, UNI, Hibou, Phikon, GPFM, H-optimus-0, UNI2, Phikon-v2, Virchow2, Prov-GigaPath, H-optimus-mini, EXAONEPath |
| Path-VLM — pathology vision-language | 7 | PLIP, QuiltNet-B16, BiomedCLIP, MI-Zero (PubMedBERT), MI-Zero (ClinicalBERT), CONCH, TITAN |
| VM — general vision | 5 | DINO-S16, DINO-B16, DINOv2, iBOT-B16, iBOT-L16 |
| VLM — general vision-language | 4 | CLIP-B16, BLIP-B16-14M, ALIGN-base, BEiT3-L16 |
The 41 tasks, split four ways — and the split is the design:
| Tier | Tasks | Level | What it is |
|---|---|---|---|
| TCGA | 19 | slide | In-domain. Molecular subtyping (IDH, MGMT, ERG fusion, MSI, PAM50, ER/PR), transcriptomic subtypes, mRNA clusters, pathway activation (PI3K, MYC), immune subgroup, histologic subtyping, pan-cancer grading |
| CPTAC | 7 | slide | Grade, tumour type (7-class), and five pathway-activation tasks (p53, mTOR, MYC, SWI/SNF, chromatin modification) |
| Out-of-domain | 7 | slide | NLST stage and grade; DHMC lung adenocarcinoma pattern; Stanford GBM MGMT and necrosis; Stanford lung histology and stage |
| External benchmarking | 8 | patch | BACH, BRACS, BreakHis, LC25000, NCT-CRC-HE, MHIST, SICAPv2, UniToPatho |
19 TCGA + 22 non-TCGA = 41. The 22 non-TCGA split further into 14 slide-level (CPTAC + OOD) and 8 patch-level (external). Across the whole thing: 53 datasets, >17,500 WSIs.
The four fields that are usually buried
Per AGENTS.md §8, and this paper is better than most on three of them.
Dataset. Fully specified, per-task, with train/test N for all 41 tasks. Sizes range over three orders of magnitude — from N=19 test slides (Stanford lung stage) to N=7,500 test patches (LC25000). That range matters and is picked up under problems 3 and 4 below.
Scanner and magnification. Magnification is stated and standardised: WSIs tiled at 20× / 0.5 µm per pixel into non-overlapping 256 × 256 tiles, Otsu tissue masking, up to 4,000 tiles per slide, then resized to each model’s native input (224 × 224 for most). Scanner is not reported — not for TCGA, not for CPTAC, and not for the two Stanford in-house cohorts. Given that scanner is the confounder this whole field worries about, its absence from a benchmark positioning itself on generalisation is a real omission. Note also that the standardisation does not extend to the 8 external patch datasets: those were consumed as distributed (“no additional preprocessing was required, as the images were conveniently provided in a ready-to-use .png format”), and they are not all at 20× — SICAPv2 is digitised at 40×, UniToPatho at 0.4415 µm/px. So “a consistent physical tissue context across datasets” holds for the slide-level tiers and not for the patch-level one.
Validation strategy. A single 70/30 patient-level split into train and hold-out test for TCGA, CPTAC and OOD; predefined splits for the external benchmarks. Confidence intervals come from 1,000 bootstrap replicates of the test set. There is no cross-validation, no repeated splitting, and no separate validation set for the probe — hyperparameters were fixed identically across all models (LR 0.001, batch 16 per GPU, 30 epochs, no augmentation), which removes tuning-on-test but also removes any per-model adaptation. Slide-level prediction is mean pooling of tile features followed by a linear layer; explicitly not attention-MIL.
External validation. Yes — and it is the spine of the paper rather than an afterthought. TCGA is treated as in-domain throughout and reported separately from everything else, on the authors’ own stated reasoning that “TCGA likely contributes, directly or indirectly, to the pretraining data of several evaluated models.” Three progressively more external tiers sit outside it. The two Stanford cohorts are institution-specific clinical data under IRB #48496 (NSCLC) and #50031 (GBM). This is the design worth stealing.
Headline numbers
TCGA, mean AUROC across 19 tasks — Virchow2 0.830 (95% CI 0.784–0.875), Prov-GigaPath 0.829 (0.784–0.875), H-optimus-0 0.824 (0.776–0.873), UNI 0.824 (0.777–0.871), UNI2 0.824 (0.779–0.868).
Non-TCGA, mean AUROC across 22 tasks — Virchow2 0.836 (0.767–0.906), UNI 0.835 (0.770–0.899), H-optimus-0 0.832 (0.764–0.899), Prov-GigaPath 0.827 (0.757–0.897), UNI2 0.824 (0.751–0.897).
The same five models, in a slightly reshuffled order, both times. Read the confidence intervals before reading the ranking: they span 0.09–0.14 AUROC and overlap completely.
Split by evaluation granularity, and the picture changes:
| Top five, mean AUROC | Any significant pairwise differences? | |
|---|---|---|
| 14 non-TCGA slide-level tasks | UNI 0.761±0.083, H-optimus-0 0.759±0.090, Virchow2 0.755±0.089, Prov-GigaPath 0.749±0.091, H-optimus-mini 0.743±0.092 | None — not for AUROC, AUPRC, sensitivity or specificity |
| 8 non-TCGA patch-level tasks | Virchow2 0.962±0.033, UNI2 0.953±0.039, Prov-GigaPath 0.949±0.042, UNI 0.948±0.045, H-optimus-0 0.944±0.044 | Yes — Virchow2 > UNI, UNI2, Prov-GigaPath and H-optimus-0 (all FDR p<0.001); UNI2 > H-optimus-0 (FDR p<0.01) |
Note the absolute levels: 0.75 on real slide-level clinical tasks, 0.95 on curated patch benchmarks. Those are not the same problem, and only the first resembles anything a department would deploy.
Category comparisons.
- Path-VM vs VM (general vision): no statistically significant AUROC difference anywhere — not on TCGA, CPTAC, external benchmarking or OOD, and not for AUPRC, sensitivity or specificity either. The authors are commendably direct about this, reframing the Path-VM advantage as “consistency and robustness rather than uniform statistical superiority.” Path-VMs occupy 47% of available top-10 slots overall (0.55 TCGA, 0.53 CPTAC+OOD, 0.52 external).
- Path-VM > Path-VLM: significant on TCGA, CPTAC and external benchmarking — but not on out-of-domain slide-level tasks, where none of the four categories differ significantly from each other. That exception is the most interesting line in the paper and gets almost no discussion.
- Vision-only beats vision-language overall. Individual Path-VLMs still win specific tasks: CONCH ranks first on Stanford lung stage and top-three on both lung histology OOD tasks; TITAN second on lung histology OOD2; PLIP second on MGMT methylation OOD.
Scaling — the negative result, and the most useful one. Model size (ViT-S ~21M → ViT-B ~85M → ViT-L/H/g >300M) and pretraining corpus size (<0.1M / 0.1–1M / >1M WSIs) were both tested. Benefits concentrate almost entirely on TCGA multi-class tasks (large Cohen’s d, significant Spearman correlations). CPTAC shows modest binary-task gains. External and out-of-domain show weak, non-significant effects with negligible correlations across every metric. ViT-B → ViT-L yields no significant gain outside TCGA; medium → large pretraining corpus is uniformly non-significant everywhere.
Ensembling. Majority-vote fusion of the top five. On TCGA it takes first place on cumulative balanced accuracy (0.702±0.111 vs Virchow2 0.696±0.105); on non-TCGA it comes second to Virchow2 alone (0.699±0.203 vs 0.712±0.200). No pairwise comparison against any individual model reached significance in either setting.
Tissue specialisation (z-scored AUROC, grouped by tissue): CONCH strongest in lung, Virchow2 in colon and prostate, H-optimus-0 in pan-cancer, UNI in brain and bladder, UNI2 in breast and pan-cancer, Prov-GigaPath across breast/bladder/lung, EXAONEPath strong in brain. Only colon produced significant differences.
Which encoders declare TCGA in their pretraining
Straight from their Table 2, and worth extracting because it answers a question this wiki has had open. This is the declared corpus, not an audit.
| Declares TCGA | Proprietary / no TCGA declared |
|---|---|
| cTransPath (+PAIP), Kaiko, HIPT, Lunit (+TULIP), Phikon, GPFM (+PAIP, +49 public sets), Phikon-v2 (+GTEx, +CPTAC, +TCIA), H-optimus-mini | Virchow, UNI (+GTEx), Hibou, H-optimus-0, UNI2, Virchow2, Prov-GigaPath |
EXAONEPath is listed only as “Public”.
Two things fall out of this. First, all five top performers sit in the right-hand column — their declared corpora are proprietary and do not name TCGA. Second, Phikon-v2 declares CPTAC, which means CPTAC is not an external cohort for that model even though the paper uses it as one of the generalisation tiers. The paper anticipates this in general terms (“some foundation models may have used CPTAC data during pre-training”) but never applies it to the specific model its own table identifies.
There is a mild internal tension here worth noting: the Discussion cites UNI among the models for which TCGA “likely contributes, directly or indirectly,” while Table 2 lists UNI’s source as “Proprietary, GTEx.” Both can be true — an undisclosed proprietary corpus can contain TCGA-derived material — but the paper does not reconcile them.
Statistical and methodological problems
Numbered so they can be referred to later. Several are acknowledged by the authors; those are marked as such, because an acknowledged limitation is a different thing from an unnoticed one.
The headline “95% CI” is not what most readers will take it for. Per the Figure 1 legend, those intervals are “calculated using a t-based standard error across task-level AUROC estimates,” n = 19 or 22. So they describe how much the model’s performance varies across a hand-picked set of tasks, not the uncertainty in its performance on any one of them. An interval of 0.784–0.875 does not mean the model’s AUROC might be 0.784; it means the task set is heterogeneous. Two different quantities are both being called a 95% CI in this paper — this one, and the genuine bootstrap CIs in Figure 1d — with no signposting between them.
One split, bootstrapped. A single 70/30 partition per task, with CIs from bootstrapping the test set. Bootstrap resampling of a fixed test set captures sampling variability within that split and says nothing about how much the result would move under a different split. With no repeated splits or cross-validation, split-to-split variance — usually the dominant source of variability at these sample sizes — is invisible everywhere in the paper.
Several test sets are far too small to support a bootstrap CI at all. Stanford lung stage N=19, Stanford lung histology N=21, Stanford GBM MGMT N=31, GBM necrosis N=34, DHMC N=46, NLST stage N=52. Bootstrapping 19 slides resamples the same handful of cases; the resulting interval is a description of those 19 slides, not an estimate of anything. Six of the seven out-of-domain tasks — the tier the paper leans on hardest for its generalisation claims — are in this range.
Task-level averaging is unweighted, so a 19-slide task and a 7,500-patch task count equally. Every “mean AUROC across 22 non-TCGA tasks” gives identical weight to an estimate from 19 slides and one from 7,500 patches. Precision-weighting, or at minimum reporting the spread of task sizes alongside the mean, would change the picture. This compounds problem 3: the noisiest estimates are pulling the headline numbers around as hard as the most precise ones.
The category and scaling comparisons treat model × task AUROCs as independent observations. Figure 3a (Path-VM vs Path-VLM vs VM) and Figure 4 (scaling) both use Welch two-sample t-tests over pooled model-task AUROC points. Those points are doubly clustered — the same 41 tasks recur under every model, and the same model recurs across every task. Treating 16 models × 19 tasks as 304 independent draws inflates the effective sample size substantially and understates the standard errors. A mixed-effects model with random intercepts for task and model is the standard fix and is not used. This is the test underpinning the Path-VM > Path-VLM claim and the scaling claims — that is, most of the paper’s positive findings. Notably, the analyses that are properly paired within task (the bootstrap Wald tests) are the ones that mostly return null results.
The scaling analysis is a between-model comparison wearing a scaling label. Comparing “ViT-S vs ViT-L” here means comparing HIPT and Lunit against UNI, Phikon-v2 and GPFM — different vendors, different SSL objectives (DINO vs DINOv2 vs iBOT), different corpora, different tissue mixes. Architecture size is confounded with every one of those. A genuine scaling study needs one model family trained at several sizes on one corpus. The authors do note that the analyses “do not explicitly account for differences in pre-training data composition,” which is the same point put more gently. The negative result survives this (a confounded comparison finding nothing is still informative), but the positive TCGA scaling result does not.
A fixed linear-probe budget across feature dimensions from ~384 to ~1536. One learning rate, 30 epochs, no augmentation, no early stopping, applied identically to a ViT-S probe and a ViT-giant probe. Uniformity is the right instinct for fairness, but a fixed optimisation budget is not neutral with respect to dimensionality — larger encoders’ probes may simply be less converged. That is a confound sitting directly inside the scaling analysis (problem 6).
Patch-level leakage risk, sitting exactly where the only significant differences were found. The 8 external patch benchmarks are split at the image level with no stated patient or source-image grouping. Two specific concerns: BreakHis contains multiple magnifications (40×, 100×, 200×, 400×) of the same tumour region from ~82 patients, so a 5,008/2,907 image-level split will place the same patient — plausibly the same field at a different objective — on both sides; and LC25000’s 25,000 images were generated by augmentation from a much smaller set of originals, so a 17,500/7,500 image split can put augmented siblings across the boundary.
[unverified]— the paper states neither dataset’s grouping, and confirming would need the source dataset papers rather than this one. The paper’s blanket assurance that “measurements were taken from distinct samples only… avoided data leakage” addresses row-level reuse, which is not the failure mode at issue. This matters because the patch-level tier is the only place the paper found significant differences between top models (Virchow2 vs the rest, FDR p<0.001) — and it is the tier whose splits are least defensible.BreakHis, as written: “we used a single fold for both training and evaluation.” Read literally that is training and testing on the same fold. It more likely means one of the dataset’s predefined folds was used rather than averaging over all five. Either way the sentence should not have survived review, and it is not resolvable from the main text.
[unverified]Slide-level aggregation is mean pooling over up to 4,000 tiles. The Methods describe this as a “multiple instance learning strategy,” but there is no instance selection and no attention — tile feature vectors are simply averaged and passed to a linear layer. The authors list this as a limitation (“simple mean pooling, which may attenuate localized discriminative signals”), and it is the right one to list: for focal findings, mean pooling over 4,000 tiles dilutes the signal by construction. The consequence for reading the paper is that this is a ranking of encoders under mean pooling, and there is no guarantee the ordering survives a swap to attention-MIL. It may well be part of why slide-level AUROCs sit at 0.75 while patch-level sit at 0.95.
The fusion model’s members were selected on the data it is then evaluated on. The top five were “identified from non-TCGA rankings” and the ensemble is then scored on, among other things, non-TCGA tasks. That is circular and would normally inflate the estimate. It is worth noting that it apparently did not save it — the fusion model still finishes second to Virchow2 alone on non-TCGA — which makes the ensembling result weaker than the abstract’s “improves aggregate performance across external datasets” suggests.
Top-1 counting is presented without a null. “Virchow2 led on 5 of 19 TCGA tasks” is offered as evidence, but with 16 Path-VMs the expected count under pure chance is 19/16 ≈ 1.2, and the variance of such counts is large. Five is above chance; the framing invites a stronger reading than an unadjusted rank statistic supports. To the authors’ credit the accompanying Wald tests are reported and show no significant differences — the two analyses point in different directions and the prose follows the flattering one.
Figure 5’s error bars are combined half-widths. Cumulative balanced accuracy across 19 tasks is given “propagated 95% confidence intervals… calculated by combining task-level 95% confidence interval half-widths.” Task-level intervals for the same model are correlated (shared model, overlapping cohorts), so neither naive summation nor quadrature is valid here. Minor, since nothing rests on these intervals, but it is the third distinct meaning of “95% CI” in one paper.
Linear probing is not fine-tuning, and not deployment. Acknowledged as the first limitation. Worth restating because it bounds every conclusion: this measures how linearly separable a frozen representation is, which correlates with but does not determine what the model does after task-specific adaptation. A ranking under linear probing is not a purchasing decision.
Sex- and gender-stratified analyses were not performed because the metadata was not consistently available. Stated plainly by the authors, which is the right handling.
Almost every inferential claim points at a supplement this PDF does not contain. Supplementary Data 1–15 hold the Wald Z statistics, χ²(1) values, raw and FDR-adjusted p-values and confidence intervals for essentially every comparison in the paper. Whether the FDR correction was applied across the right family — within task, within metric, or across the whole grid of 32 models × 41 tasks × 4 metrics — is not determinable from the main text alone.
[unverified]
What the paper gets right, and it is worth saying
The design decisions are better than the analysis. Separating TCGA from everything else, and then separating non-TCGA into three tiers of increasing externality, is exactly the structure this field has been failing to impose on itself. Reporting AUPRC and sensitivity and specificity alongside AUROC, rather than AUROC alone, is right for tasks with the imbalance these have. Using bootstrap Wald tests on paired AUROC differences with FDR adjustment is a genuinely better comparison than the naive best-number-wins that most benchmark papers stop at. And the willingness to publish a null — Path-VM does not significantly beat general VM, scaling does not transfer, ensembling does not consistently win — in a paper whose abstract could easily have claimed all three is the most creditable thing about it.
What I would want before using this
The Supplementary Data files, to check the FDR family and the per-task test statistics. The patient-level grouping used for the BreakHis and LC25000 splits. A re-run of the category and scaling comparisons under a mixed-effects model with random intercepts for task and model. And a re-run of the slide-level tier with attention-MIL rather than mean pooling, to find out whether the encoder ordering is a property of the encoders or of the pooling.