External Validation
Testing a model or biomarker on data from a different institution, scanner, or time period than it was developed on — the single strongest evidence that a result will hold up in practice.
What it is
Internal validation reuses the development data: a held-out split, cross-validation, or bootstrapping. It answers “would this model work again on more data drawn the same way?” External validation uses a genuinely separate cohort — a different hospital, a different scanner, a different era of practice — and answers the question you actually care about: “will this work anywhere other than where it was built?”
The gap between those two is routinely large in computational pathology, and it is not mainly a statistical phenomenon. A model can achieve excellent cross-validated performance by learning institution-specific signal — the stain protocol, the scanner’s colour profile, the slide preparation, even which cases that institution tends to receive. All of it is real signal, all of it generalises poorly.
Three distinctions worth keeping separate. Temporal validation uses later cases from the same institution — better than nothing, weaker than external. Geographic validation uses a different institution, which is the usual meaning of external. Prospective validation applies the locked model to cases as they arrive, which is the strongest and rarest.
An important asymmetry: external validation almost always shows a performance drop. A drop is not a failure of the study; it is the study working. A study that reports no drop deserves a hard look at whether the external cohort was really independent.
“External” is a gradient, not a switch, and the best studies report the tiers separately. The clearest worked example is the 32-model foundation-model benchmark in sources/papers/bareja-2026-foundation-model-benchmark.md, which splits its 41 tasks four ways and refuses to average across them: TCGA (in-domain, because it is in the declared pretraining corpus of half the encoders tested), CPTAC (public, independent collection, same flavour of curated research cohort), public external benchmarks (independent, but curated image patches), and institution-specific out-of-domain cohorts from Stanford, NLST and Dartmouth. Reporting each tier on its own is what let the paper say something honest: rankings shift modestly between tiers, and its category-level findings — pathology encoders beating vision-language ones — held on the first three tiers and vanished on the fourth, where no model category differed significantly from any other. Pooling those tiers would have hidden exactly the result that matters most.
Two cautions the same paper illustrates. Its most external tier has test sets of 19 to 52 slides, which is too small for the bootstrap intervals placed on them — so the tier that carries the most external weight carries the least statistical precision, and that tension is intrinsic rather than sloppy. And “external” is a claim about provenance that usually cannot be verified: the authors concede that NLST and DHMC being absent from the encoders’ training data “cannot be independently verified”, and their own model table shows CPTAC inside one evaluated model’s pretraining corpus.
Why it matters for my work
The schema for this wiki names external validation as one of four fields to capture for any model paper, precisely because it is usually buried. It is also the field most often absent from the group’s own AI projects: Liver Tru-Cut — Tumour Origin AI, Paige Prostate, and Aiforia Breast all currently record no validation strategy at all. [unverified]
For the two vendor evaluations this cuts differently and more favourably. A vendor model arrives already trained elsewhere, so evaluating it on Memorial cases is an external validation — of the vendor’s model. That is a genuinely valuable contribution and worth framing that way explicitly in the write-up rather than as a plain accuracy study.
How it connects
Class Imbalance and Accuracy — an external cohort usually has a different prevalence than the development set, which moves predictive values even when sensitivity and specificity are stable; report both.
Scanner and Stain Variability — the concrete mechanism by which internal performance fails to transfer, and the thing an external cohort is really probing.
Diagnostic Test Accuracy — supplies the metrics that should be reported on the external cohort, not just on the development set.
Weakly Supervised Learning — MIL models are especially prone to shortcut learning because nothing constrains which tiles they attend to.
Concordance Index — for survival models this is the number that drops on an external cohort, and part of the drop is that pooled concordance was measuring cohort composition all along rather than model skill.
TCGA — the cohort most public models are developed on, and a reminder that “large and public” is not “externally validated”: a multi-institution convenience sample used for both development and testing is still internal validation. That page now carries the table of which encoders declare it, which is what makes the in-domain tier assignable rather than assumed.
Encoder Benchmarking — where the tiering above becomes a concrete design decision, because a frozen-encoder comparison that never leaves the pretraining distribution measures recall instead of representation quality.
Model Abstention — abstention rates do not transfer. A model that answers every case at home can decline a fifth of an outside cohort, and that is a transfer failure which never appears in an accuracy column.
AIDA Data Hub — the shortest practical route to an actual external cohort from here: Swedish annotated whole-slide collections, free of charge but gated behind a PhD requirement and an institutional signature.
Federated Learning in Pathology — the other route to a multi-institution result when the slides cannot move: train across sites and exchange only weights. Read it after AIDA Data Hub, which is far cheaper — federated learning is what this question becomes when someone wants a multi-centre model rather than a multi-centre test set.
Open questions
- Do any of the group’s AI projects have an external cohort identified? Nothing recorded. [unverified]
- Would a scanner-swap validation (same cases, second scanner) be feasible at Memorial? It is a cheap partial substitute for a full external cohort and directly tests the most likely confounder.
- Is the vendor-evaluation-as-external-validation framing something the group wants to make explicit in the Paige and Aiforia manuscripts?