Koyun 2026 — Slide Selection Introduces Sampling-Induced Prediction Uncertainty

Holds the model, the weights, the patients and the labels fixed and changes only which of a patient’s slides is fed to it: patient-level AUC then spans 0.46 to 0.97, and pooling every slide’s patches into one bag recovers a modest, consistent, externally validated part of the loss.
Author

Koyun OC, Guo Y, Lu H, Demir MF, Alili A, Rahoui N, Cardona D, Gurcan MN. Center for Artificial Intelligence Research, Wake Forest University School of Medicine, Winston-Salem NC (Koyun, Guo, Lu, Demir, Alili, Gurcan); Department of Pathology, Wake Forest University School of Medicine (Rahoui, Cardona). Corresponding author Onur C. Koyun

Doi

10.3390/cancers18162540

Holds the model, the weights, the patients and the labels fixed and changes only which of a patient’s slides is fed to it: patient-level AUC then spans 0.46 to 0.97, and pooling every slide’s patches into one bag recovers a modest, consistent, externally validated part of the loss.

Extraction note. pdftotext -layout misaligned two tables in ways that would corrupt a naive copy. In Tables 2 and 3 the Task label “HER2” is emitted on the third data row rather than the seventh, though the twelve value rows are in the correct order. In Table 4 Panel B the Worst-Case AUC and Best–Worst AUC columns are shifted up by one row, so the first row’s values appear above the header block. Both were resolved by reconciling every value against the ranges quoted in the running text and against ABMIL’s row in Table 3, and both reconcile exactly. The numbers below are the reconciled ones.

The question, which is not the one the title suggests

This is not another MIL architecture paper, and the authors say so twice. The design is an ablation of tissue sampling, not of modelling: encoder, trained weights, patient set and labels are all held fixed, and the only thing that varies is which whole-slide image from that patient goes into the model at inference.

That question exists because of a mismatch this wiki has recorded from the other direction. A patient has several sections — different blocks, different tumour regions, different depths. A MIL model takes one slide as the unit of inference. A pathologist integrates across all of them. The gap between those two facts is normally invisible, because benchmarks report slide-level performance and never ask what happens at the patient level.

Setup

  • Development: TCGA-BRCA. 1133 WSIs available; after excluding incomplete receptor status, missing expression data and processing failures, 1065 WSIs from 1006 patients. Multi-site, multi-scanner by construction.
  • External evaluation: CPTAC-BRCA, used for nothing else — no training, no tuning. The QC pool held 653 WSIs; the multi-slide evaluation subset is 400 WSIs from 122 patients, a mean of 3.28 slides per patient. That last number is the whole reason the study is possible, and it is rare in a public cohort.
  • Two endpoints.
    • HER2 status, labels taken directly from cohort annotation; 980 TCGA patients eligible.
    • A research-derived 21-gene recurrence-score surrogate (“ODX” in the tables) — not a clinical Oncotype DX result. Upper-quartile-normalised STAR–Salmon expression, log2-transformed, row-median-centred, column-standardised, then the published Howard 2022 formulation. Restricted to the 516 HR+/HER2− patients. The high-risk cutoff was prevalence-matched so that ~15% are high risk, matching the National Cancer Database proportion with a clinical recurrence score ≥26 — giving a fixed normalised threshold of 0.7169 on a score range of −2.009 to 2.744, i.e. 443 low risk and 73 high. The same threshold was applied to CPTAC without retuning.
  • Preprocessing: HSV colour thresholding for tissue, non-overlapping 896×896 patches at 20×, resized to 224×224. No stain normalisation and no colour augmentation — only ImageNet channel normalisation.
  • Encoder: UNI2-h (MahmoodLab), frozen, 1536-dimensional embeddings cached once.
  • Six MIL heads on identical embeddings: ABMIL, DTFT, RRTMIL, TransMIL, plus two custom baselines — DAMLN (a transformer/Hydra-attention hybrid) and S4Model (a structured state-space adaptation).
  • Training: five patient-level cross-validation folds per task, strict patient-wise separation. Each of the five fold models is evaluated separately on CPTAC; reported spreads are across those five trained models.

Four inference regimes, and the distinction between them is the paper’s discipline:

Regime What it does Deployable?
Slide-level every WSI an independent sample, inheriting the patient’s label descriptive benchmark only
Random one-slide-per-patient one slide drawn without using the label yes — the realistic baseline
Oracle best / worst label used retrospectively to pick the most/least favourable slide no — stress-test bound
Patient-level aggregation patch embeddings from all the patient’s WSIs concatenated into one bag yes — the proposed mitigation

Results

The realistic comparison (Table 2, CPTAC, mean ± SD across five fold models):

Task Random single slide All-slide aggregation ΔAUC
Recurrence surrogate 0.773–0.862 0.814–0.888 +0.026 to +0.041
HER2 0.653–0.768 0.708–0.804 +0.036 to +0.073

Every architecture improved on both tasks, in AUC and in F1. Brier scores fell by 0.010–0.028 and 0.009–0.106 respectively; the largest was S4Model on HER2, 0.280 → 0.174. No parameters changed between the two conditions, so this is a clean ablation of how much tissue the model sees.

The stress-test bound (Table 3) is the number that will be quoted, and it should be quoted with its caveat:

Task Oracle best Random Oracle worst Best→worst drop
Recurrence surrogate 0.948–0.972 (mean 0.959) 0.773–0.862 (mean 0.822) 0.460–0.636 (mean 0.561) 0.33–0.49 AUC
HER2 0.867–0.919 (mean 0.896) 0.653–0.768 (mean 0.698) 0.399–0.560 (mean 0.457) 0.35–0.50 AUC

Worst-slide selection drives both tasks to chance or below, on the same patients, with the same weights. F1 for HER2 falls from 0.640–0.730 to 0.256–0.332.

It is not just the bad slides. An independent ResNet-18 tumour classifier scored tumour content per slide. Repeating the ABMIL oracle analysis after filtering shrank the best–worst gap but nowhere near closed it:

Task No filter (400 WSIs) Excl. tumour-negative/insufficient QC (388) Excl. ≤5% tumour-like patches (358)
Recurrence surrogate 0.346 0.330 0.308
HER2 0.366 0.318 0.267

More slides did not mean more stability. Within-patient standard deviation of predicted probability increased with the number of available WSIs for the recurrence surrogate, weakly for HER2. The authors are careful here — more slides means more opportunities to observe variability, not more variability caused.

Slide-level evaluation, for reference: recurrence surrogate AUC 0.650–0.834, HER2 0.521–0.652 — i.e. HER2 slide-level discrimination is barely above chance for some architectures. The paper correctly refuses to compare this inferentially with the patient-level numbers (400 slides versus 122 patients).

Architecture matters for robustness, separately from accuracy. ABMIL and DTFT had the smallest oracle gaps on both tasks; DAMLN, RRTMIL and TransMIL the largest. The authors flag that average performance and selection robustness are related but distinct properties, and that no architecture eliminated the sensitivity — which is the argument for treating this as an evaluation-design problem rather than a backbone-choice problem.

What is genuinely well done

Worth saying, because it is unusual. The paper is relentless about not overclaiming its oracles: the phrase “not a clinically deployable procedure” or equivalent appears in the simple summary, the abstract, the methods, the results, the discussion, the conclusions and two figure legends. It also refuses to compute p-values across five fold models that are not independent cohorts, refuses to compare slide-level against patient-level inferentially, and states three times over that the recurrence endpoint is an RNA surrogate rather than a clinical Oncotype DX result and cannot be used to make claims about the commercial assay. That is the opposite of the pattern this wiki usually has to flag.

Reading it critically

  1. The random baseline is a single draw. One prespecified seed-42 slide per patient, applied across architectures. The reported SD therefore reflects variation across the five trained fold models, not the distribution of slide-selection randomness — which is precisely the quantity the paper is about. The authors state this as a limitation. Its consequence is that the headline +0.026 to +0.041 rests on one particular draw, and a different seed would move it by an unknown amount.
  2. No inferential test supports the aggregation improvement, and the fold-to-fold dispersion is often larger than the effect. DAMLN on the recurrence surrogate: 0.773 ± 0.048 → 0.814 ± 0.064, an improvement of 0.041 against SDs of 0.048 and 0.064. The direction is consistent across all twelve architecture-task pairs, which is the real evidence, but the abstract quotes the magnitudes without that context.
  3. Two different definitions of “low tumour content” carry one name. §2.3 defines it as ≤5% tumour-like patches — which is 73 of the 648 analyzable WSIs, since 575 exceeded 5%. Table 4 Panel A separately reports “likely no- or low-tumour-content by automated QC status: 17 (2.62%)”. Those are different criteria, and the two filters in Panel B behave accordingly (388 versus 358 slides passing). The paper does not reconcile them, and a reader taking the 2.62% figure as the prevalence of low-tumour slides would be wrong by a factor of four.
  4. CPTAC-BRCA includes fresh-frozen material. The authors say so, and it is the sharpest generalisability limit: routine diagnostic tissue is FFPE. This wiki has the matching finding from the other direction in Lerousseau 2020 — MIL Tumor Segmentation (percentile pseudo-labels), where a model trained on frozen TCGA transferred to FFPE at or near chance in 12 of 15 configurations. A slide-selection effect measured partly on frozen sections may not have the same magnitude on FFPE, in either direction.
  5. Both cohorts are research consortia. TCGA and CPTAC share curation conventions that a community hospital does not. The paper says this too.
  6. The endpoint is a surrogate, and although the paper is scrupulous about labelling it, this does bound the result: it establishes slide-selection sensitivity for an RNA-derived research score, and the prevalence-matched 0.7169 cutoff is a research classification threshold. A separate 0.71 probability threshold is used for the F1/accuracy metrics; the paper explicitly notes these two 0.71s are unrelated, which is a near-collision worth reading twice.
  7. The promised code release is due and absent. “Will be released in a public repository with the final published version” — this is that version. No URL. The derived labels and fold definitions are the pieces that would make the study reproducible, and they are the pieces withheld.
  8. The attention-map figure is decorative and the paper says so — model-dependent, bag-normalised, and ABMIL rankings may be similar by construction when instances are appended. Restricted to workflow illustration. Correct call.

What it would change here

  • Any patient-level claim made from a single slide is untested here until it is re-run on a second slide from the same patient. That applies directly to PinkKidney, Paige Prostate and Aiforia Breast, and it is a cheaper check than an external cohort: it needs no new labels, only more of the tissue already scanned.
  • Concatenating patches across slides is a mitigation that costs no retraining. It is one preprocessing change to the bag construction and it worked for six architectures out of six.
  • Across-slide dispersion is a free uncertainty signal. The paper’s closing proposal — flag patients whose prediction swings across their own sections for review — is an abstention rule that needs no calibration data beyond what a multi-slide cohort already provides.
  • It supplies the missing external-validation answer that Prediction Aggregation was asking for: aggregation strategies had, until this, only been compared within one institution.

How it connects

  • Prediction Aggregation — the direct extension. Uysal compared aggregation rules (max, majority, pixel packing) within one institution; this compares aggregation scope across two cohorts, and adds a fourth route — concatenating patch embeddings, i.e. aggregating in feature space rather than over predictions or over pixels.
  • Intratumoral Heterogeneity — the leading mechanism, and this paper is honest that its design cannot separate biological heterogeneity from fixation, section depth, folds, staining and scanner.
  • External Validation — a properly held-out second cohort, never touched for training or tuning, with the threshold transferred rather than retuned. It also carries the frozen-versus- FFPE caveat that bounds it.
  • Weakly Supervised Learning — the label-granularity problem stated cleanly: a molecular label true for the patient can be untrue of the section that inherits it, which makes some training slides conditionally noisy in a way patient-level splitting does not fix.
  • Model Abstention — the proposed slide-dispersion flag is abstention by uncertainty, and it would need the coverage reporting that page insists on.
  • Foundation Models in Pathology — UNI2-h frozen, one encoding pass, embeddings cached; the encoder is held constant so that none of the variance can be attributed to it.
  • Biomarker Cut Points — the high-risk label exists only because a continuous RNA score was cut at a prevalence-matched 0.7169, and the paper is careful that this is not the clinical cutoff.
  • TCGA — development cohort, and its multi-site multi-scanner composition is used here as a feature rather than apologised for.
  • Patch Extraction — 896×896 at 20× resized to 224, no stain normalisation; a documented set of the decisions that page says set the ceiling.

Open questions

  • Does the effect hold on FFPE-only material? The single most important follow-up, and this group’s own cohorts could answer it.
  • What does the random baseline look like under repeated Monte Carlo draws rather than seed 42? The paper names this as future work; it is also the number needed before the +0.026 to +0.041 can be trusted.
  • Is concatenation the best aggregation in feature space, or would attention-weighted slide fusion beat it? Explicitly out of scope here.
  • What is the actual prevalence of low-tumour-content slides — 2.62% or 11.3%? See point 3.
  • Was the code repository ever released? Checkable, and worth checking before anyone tries to reproduce this. [unverified]
  • How much of the effect is block selection versus section depth? Would need a cohort with block and level metadata, which neither TCGA nor CPTAC supplies.