Khan 2026 — Stain Normalisation Benchmark on a 66-Laboratory Multicentre Dataset

Same tissue blocks stained by 66 laboratories and scanned on one scanner, used to benchmark eight stain normalisation methods — with the simplest method winning and the foundation model failing to be robust.

Sections of the same three tissue blocks were sent to 66 laboratories for routine H&E, then scanned on a single scanner — isolating staining variation from everything else — and used to benchmark eight normalisation methods, where plain histogram matching beat four deep-learning variants and a foundation model proved not robust at all.

Read status: full text. This is the first paper in sources/papers/ read completely rather than from the abstract. Open access under CC-BY 4.0, retrieved from Europe PMC. Numbers below are quoted from the article.

The design, which is the best thing about it

This is the part worth copying, independent of any result.

Via a Labquality external quality assessment round (Helsinki), a tissue microarray section containing three 6 mm punch biopsies — normal skin, kidney and colon — was cut at 3 µm and the unstained sections dispatched to participating laboratories. Each lab applied its own routine H&E protocol. 66 laboratories across 11 countries took part.

Every returned slide was then digitised on one scanner — Hamamatsu NanoZoomer-XR, 20× objective, 0.46 µm/pixel — and resampled to 10× for processing.

The consequence is the point: same block, same section thickness, same scanner, different staining lab. Scanner variation is eliminated by construction and biological variation is near-eliminated by using adjacent sections of one block. What is left is almost pure inter-laboratory staining variation. Previous work, as the authors note, used images from a handful of sites with everything varying at once.

The reference image was chosen computationally: for each tissue, the sample whose red-to-blue mean intensity ratio was closest to 1.

What was compared

Eight methods, four of each kind:

• Traditional — histogram matching, Macenko, Reinhard, Vahadane.
• GAN-based — CycleGAN (ResNet generator), CycleGAN (UNet), Pix2pix (UNet), Pix2pix (DenseUNet).

Evaluated three ways: channel-histogram agreement with the reference (intersection, Pearson correlation, Euclidean distance, Jensen-Shannon divergence); FID on InceptionV3 bottleneck features; and SSIM against the original, for structural preservation. Plus qualitative review by pathologists, and two downstream tasks.

Results

Histogram matching — the simplest method on the list — won. Best on all four histogram metrics in all three tissues:

Tissue	Intersection	PCC	Euclidean	JS divergence
Skin	0.891	0.938	0.279	0.119
Kidney	0.944	0.985	0.144	0.101
Colon	0.906	0.935	0.295	0.143

On FID it led for skin (61.67, versus 69.49 for unnormalised) and kidney (55.38 vs 69.54). On colon it lost — Reinhard 92.37, CycleGAN (ResNet) 96.12, CycleGAN (UNet) 96.50, histogram matching 99.92. Colon was the most morphologically heterogeneous of the three, which is consistent with the caveat below.

Artefacts, from the pathologist review. Macenko turned erythrocytes and parts of lamina propria blue. Vahadane infused pink across the whole image and overrode the hematoxylin component completely, including turning white background pink. CycleGAN (ResNet) hallucinated in adipose tissue and produced tile artefacts at luminal epithelium–smooth muscle borders. Pix2pix (DenseUNet) turned some spindly smooth-muscle nuclei red. No artefacts were observed with histogram matching or Reinhard.

No method fixed both failure directions. Histogram matching improved heavily over-stained slides but pushed under-stained ones toward pink/red. Macenko and Reinhard corrected under-staining best and handled over-staining poorly. In the authors’ words, no method improved visual quality in all instances of suboptimal staining.

Downstream 1 — nucleus counting (Cellpose-SAM). Normalisation had a major effect, but not a systematic one across methods. Deep-learning methods gave fewer detections. The authors add their own honest caveat: adjacent sections of one block do not contain equal numbers of nuclei, so the reference count is not ground truth.

Downstream 2 — foundation model features. Colon tiles were embedded with UNI-2 and visualised by t-SNE. All eight methods substantially changed the feature-space representation. The authors’ conclusion is blunt: the foundation model “appears not to be robust against the intensity variation/normalization.” CycleGAN variants and Macenko produced compact point clouds; Vahadane and Pix2pix scattered ones.

Cost. Traditional methods need no training at all; histogram matching and Reinhard run in 30 s–2 min per WSI, Macenko and Vahadane 2–7 min. CycleGAN training took up to several days on a 32 GB V100; deep-learning inference ran 4–5 min per WSI.

Statistical and methodological problems

The design is strong; the ranking it produces is much weaker than the headline suggests, and the authors are commendably explicit about most of this.

1. The GAN methods were handicapped by construction. Pix2pix was trained on a single source–target pair — the reference WSI and its own greyscale version. CycleGAN got one representative WSI per cluster. One slide per laboratory cannot feed a data-hungry model. So “traditional beats deep learning” here is really “traditional beats deep learning when deep learning is given almost no training data.” Not a fair fight, and the paper says so.
2. Histogram matching’s win is an artefact of the design. Every section came from the same block, so tissue content is nearly identical across the 66 images. Global statistical methods exploit exactly that. In a real cohort each slide contains different tissue, and a global histogram match would then shift colour in response to tissue composition rather than stain. This is the single biggest transfer risk, and the reason the colon result (most heterogeneous tissue, histogram matching loses on FID) is the most informative row in the paper. Do not read this as “use histogram matching everywhere.”
3. SSIM rewards doing nothing. Vahadane scored the best SSIM in all three tissues (0.995 skin, 0.967 kidney, 0.989 colon) while being the worst normaliser by every other measure — it destroyed the hematoxylin channel. SSIM is computed against the original, so a method that barely changes the image wins. Selecting on SSIM would select the worst method. Read it as a constraint (“did structure survive”), never as an objective.
4. The foundation-model conclusion rests on eyeballing t-SNE. No quantitative measure of feature-space shift is reported — no batch-effect metric, no silhouette, no linear-probe accuracy. t-SNE cluster compactness varies with perplexity and seed and is not a sound basis for ranking methods. The direction (features move) is credible and important; the ranking of methods by “compactness” should not be relied on.
5. No downstream diagnostic accuracy. Nucleus count and embedding geometry are proxies, and neither has a ground truth here. The authors identify this as the design a future study needs.
6. One section per laboratory means no intra-laboratory variance. A lab’s systematic bias cannot be separated from its day-to-day drift — so the spread across 66 labs is an upper bound that mixes both.
7. Normal tissue only. Skin, kidney and colon, all normal. Diagnostic work happens on tumour, with different chromatin density and eosinophilia; whether the ranking holds there is untested.
8. A single reference image, chosen by one heuristic. Every metric is relative to the one image whose red-to-blue ratio was nearest 1. No sensitivity analysis over reference choice is reported, and a different reference could plausibly reorder the methods.
9. The single scanner is both the strength and the boundary. It isolates staining beautifully. It also means the study says nothing about the combined scanner-plus-stain variation that is the actual condition in practice.

The authors’ own framing is the correct one and deserves quoting: they “emphasize the role of this study as an example and benchmark for the range of stain variation rather than a conclusive comparison of the most accurate normalization method.”

Why it matters here

It answers a standing open question. wiki/concepts/scanner-and-stain-variability.md records that no normalisation method is documented as the house standard. This gives a defensible starting point — histogram matching or Reinhard, both artefact-free here, both cheap, neither requiring training — with the explicit caveat that the group’s cohorts have varied morphology where the paper’s advantage for global methods does not automatically hold.

It is directly the consultation problem. Memorial’s Konsultasyon work involves cases referred in from other laboratories, whose slides were stained elsewhere. That is precisely the inter-laboratory variation this dataset measures — 66 labs, 11 countries. The magnitude of “out in the wild” staining spread is therefore a number with in-house relevance, not a hypothetical.

It qualifies a claim already on the wiki. The foundation-models page and the scanner-variability page both assert that institution-spanning pretrained encoders are substantially more robust to this problem. UNI-2’s feature space moved substantially under every one of the eight methods. The claim needs narrowing, and both pages are updated.

The dataset is free and the code is public. Data under a FAIR DOI, implementations on Zenodo. This is a ready-made external benchmark for any normalisation choice the group makes.

What I would want next

An intra-laboratory arm (several slides per lab, not one); tumour tissue; a downstream diagnostic-accuracy endpoint rather than nucleus counts; and a quantitative feature-shift metric to replace the t-SNE inspection.