Prediction Aggregation

A model predicts on one slide; a report is issued for one patient. The rule that bridges the two is a design decision with measurable consequences, and the clinically intuitive choice is usually the worst one.

What it is

Almost every computational pathology model predicts on a single image — one whole slide image, one section, one core. Almost every clinical decision is made on a single answer per patient. Between those two sits a reduction step, and it is routinely treated as plumbing: take the highest prediction, or the most common one, and move on.

It is not plumbing. It is a component with its own error behaviour, and it can move end-to-end performance further than a change of architecture would. It is also almost never validated, because a paper that reports slide-level accuracy never has to state its aggregation rule at all.

The structure is general and recurs well beyond grading. Several levels from one block, several blocks from one specimen, several cores from one prostate, several slides from one resection — any time the reporting unit is coarser than the prediction unit, this question exists.

Why the intuitive rule is the dangerous one

The obvious rule is maximum: if any piece shows the worst thing, report the worst thing. Pathologists reason this way for good reasons, and for a human reader it is safe, because a human who sees a suspicious focus looks harder at it.

A model has no such second look, and the maximum has a statistical property that makes it uniquely fragile: it is determined entirely by the single most extreme prediction, so every additional unit is another chance to be wrong in the direction that matters. False positives accumulate; they never cancel. The error rate of a max rule therefore grows with the number of pieces sampled, which means the rule gets worse on exactly the patients who were sampled most thoroughly.

Majority voting has the opposite behaviour — it averages out isolated errors — but it buys that robustness by discarding genuine focal findings, which in oncology is the error that matters. A pure majority rule will vote away a small high-grade focus that is the whole point of the examination. The usual repair is to take the majority over positive units only, excluding negatives from the vote, which recovers much of the sensitivity without inheriting the max rule’s fragility.

Aggregating pixels instead of predictions

There is a third option that avoids the choice. Rather than combining several predictions, composite the tissue into one image and predict once. Uysal 2026 does this with a WSI packing algorithm (from Aswolinskiy 2025, credited but not released): the raw slides are rewritten into synthetic whole slide images at a chosen granularity — one section, or all sections of one core, or every section of every core from one patient, tiled into a single image. A slide-level model then makes one prediction on an input that already contains everything the pathologist would have seen.

The appeal is that it needs no new architecture, only a compositing step before inference, so it can be tested against any existing slide-level model. The cost is that it feeds the model an image unlike anything in its training distribution, with far more tissue and far more background than a real slide, and any attention-based model has to spread its attention budget across all of it.

Aggregating features instead of either

A fourth route sits between the two. Rather than combining predictions, and rather than compositing pixels, combine the patch embeddings: encode every patch of every slide the patient has, concatenate the lot into one bag, and let the existing multiple-instance model produce a single prediction from it. sources/papers/koyun-2026-slide-selection-uncertainty.md does exactly this, deliberately choosing the most minimal version so that nothing else changes — same frozen encoder, same trained weights, same patients, same labels, no retraining and no architecture-specific fusion. Only the scope of the bag differs, which makes the comparison a clean ablation of how much tissue the model saw.

It is cheaper than packing, because it needs no compositing step and feeds the model nothing out of distribution — a bag of patches is what a MIL model already consumes, there are simply more of them. The costs are computational rather than statistical: patch encoding grows roughly linearly with slide count, bags get large, and a full self-attention aggregator can scale quadratically in instances where attention pooling scales linearly.

What the direct evidence shows: comparing rules, within one institution

sources/papers/uysal-2026-patient-level-gleason.md is the only source here that compares the rules head to head, on 2,660 sections from 803 cores of 106 prostate biopsy patients, scored as quadratically weighted kappa against the routine report. Its orderings were consistent across every model tried:

Route to a patient-level grade	Model	QWK
Packing (one prediction on a patient-level composite)	biopsy-level H-ViT	0.79
Majority voting over packed cores	biopsy-level H-ViT	0.76
Max voting over packed cores	biopsy-level H-ViT	0.74
Majority voting from sections upward	section-level H-ViT	0.72
Max voting from sections upward	section-level H-ViT	0.49

Read the spread rather than the winner, and mind which column changes. With the model held fixed, the rule alone is worth 0.05 kappa for the biopsy-supervised model (0.74 → 0.79) and 0.23 for the section-supervised one (0.49 → 0.72). The full 0.49-to-0.79 span bundles the aggregation rule together with the level the model was supervised at, and should not be quoted as the effect of the rule by itself.

The robust result is the direction rather than the size. Max voting lost every paired comparison in the paper — across both models and both routes, by between 0.02 and 0.28 kappa — and its deficit widened with the number of cores per patient: a mean group difference of 0.18 ± 0.04 between patients with 2–6 cores and those with 9–12, against 0.03 ± 0.03 for packing. That is the accumulation effect above, measured.

Two further results are worth carrying:

Supervise at the level you intend to report at. A model trained on core-level composites beat an otherwise identical model trained on sections at every evaluation level, including the section level it was not trained for. If the reporting unit is known in advance, it is cheaper to train there than to build a clever reduction on top of a section-level model.

Which rule wins depends on heterogeneity. Splitting patients by entropy of grade across their cores, majority voting did best where cores agreed and packing did best where they did not. This is mechanically unsurprising — voting is a consensus operator, so it fails precisely when there is no consensus to find — but it means a rule benchmarked on a homogeneous cohort will not hold on a heterogeneous one. See wiki/concepts/intratumoral-heterogeneity.md.

The prior question: how much does it matter which unit you had?

Every rule above assumes a fixed set of units to combine. The question underneath is how much the composition of that set moves the answer — and that turns out to be larger than any difference between rules.

sources/papers/koyun-2026-slide-selection-uncertainty.md isolates it by changing nothing but the slide. Six MIL architectures on frozen UNI2-h embeddings, trained on TCGA-BRCA, evaluated on 122 CPTAC-BRCA patients who have a mean of 3.28 whole slide images each; the patients, the labels and the weights are held fixed, and only the WSI fed to the model at inference varies:

Which of the patient’s slides is used	Recurrence-score surrogate, AUC	HER2, AUC
Retrospective best slide (oracle)	0.948–0.972	0.867–0.919
Random one slide per patient	0.773–0.862	0.653–0.768
All slides, patches concatenated into one bag	0.814–0.888	0.708–0.804
Retrospective worst slide (oracle)	0.460–0.636	0.399–0.560

Three things to take from it, in order of how much they should change behaviour:

The section you happened to analyse can decide the result. Best-to-worst spans 0.33–0.49 AUC for the recurrence surrogate and 0.35–0.50 for HER2, with worst-slide selection at or below chance across all six architectures. The oracle rows are not a deployable strategy — they use the outcome label retrospectively — and the paper is unusually disciplined about repeating that. They are a bound on how much slide choice alone can matter, and it is a wide one.

Aggregating everything beats drawing one, consistently but modestly. +0.026 to +0.041 AUC for the recurrence surrogate and +0.036 to +0.073 for HER2, improving in all twelve architecture-task pairs, with Brier scores falling too. The consistency across six unrelated architectures is the evidence; the magnitudes are descriptive differences of means with no inferential test behind them, and the fold-to-fold dispersion is often as large as the gain.

It is not merely a tumour-content problem. Screening out tumour-evidence-negative and low-tumour-content slides narrowed the best–worst gap — 0.346 → 0.308 for the recurrence surrogate, 0.366 → 0.267 for HER2 — but left most of it standing. Slide quality contributes; it does not explain.

This is also the first externally validated aggregation result on this page: developed on TCGA, evaluated on a cohort never touched for training or tuning, with the threshold transferred rather than retuned. It comes with its own boundary — CPTAC-BRCA contains fresh-frozen material, which is not routine diagnostic tissue, and both cohorts are research consortia.

Correction to that note, recorded here because the note is append-only (2026-09-01): its code: field says “None released” and its critical reading calls the promised code release “due and absent”. The repository exists — CAIR-LAB-WFUSM/SlideSelection, created 2026-08-19, twelve days after publication and eleven days before that note was written: a single commit a9be59e by Onur Koyun, 14 KB, no LICENSE file. What remains true is the note’s other observation — no URL appears in the version-of-record PDF, so the repository is findable only by search. (The publisher’s HTML served a bot check and there is no PMC deposit, so whether the landing page was later updated with a link could not be checked from here.)

The substantive criticism survives, and the repository states it more plainly than the note could. The Data Availability Statement promised five things: fold definitions, slide lists, derived labels, model code and analysis scripts. What is published is a dataset-agnostic reference implementation — one of the six MIL architectures (MODEL_REGISTRY contains only abmil, and the README instructs the reader to implement DTFT, RRTMIL, S4Model and TransMIL themselves), a 1 KB analysis.py that summarises metrics across runs, and an example_manifest.csv whose rows read P001,S001. The first three promised items are absent by declared intent: “This public version intentionally contains no patient-level data and no extracted embeddings.”

A reference implementation reproduces a method, not a number, and that is the part worth carrying off this page. The repository is dataset-agnostic by design — it expects a manifest with fold, split and tumor_fraction columns that the reader supplies — so the experiment can be re-run, but not this experiment. The seed-42 single-slide draw cannot be repeated as a Monte Carlo on the paper’s own folds, because the fold definitions and slide lists are not there; and the 2.62%-versus-11.3% low-tumour-content discrepancy cannot be reconciled, because the ResNet-18 classifier that produced both figures is not in the tree at all. What the release does support is running the same four-regime comparison — oracle best, random, all-slide aggregation, oracle worst — on a local cohort, which given the second-slide check below is the more useful half anyway.

What to check in a local pipeline

State the rule. If a pipeline produces a case-level answer, the aggregation rule is part of the method and belongs in the write-up next to the architecture.
Evaluate at the reporting level, not the prediction level. A slide-level kappa does not transfer; in the source above, patient-level performance was consistently below biopsy-level for the same model.
Report performance against the number of units aggregated. A rule that looks fine on patients with three blocks may fail on patients with twelve, and the average across the cohort hides it. Both sources now show this from opposite ends: max voting’s deficit widened with core count in Uysal, and in Koyun the within-patient spread of predicted probabilities rose with the number of available slides rather than settling. More tissue is more opportunity to observe the variability, not a cure for it.
Run the same patient’s second slide before believing a patient-level number. It needs no new labels and no new cohort — only more of the tissue already scanned — and it is the cheapest available test of whether a result is a property of the patient or of the section.
Beware inherited labels. Where fine-grained ground truth does not exist, sub-units are commonly labelled with their parent’s label — every section of a core gets the core’s grade. That is a defensible training compromise, but it means the fine-level metric is not measuring what its name says, and no amount of aggregation fixes it downstream.

How it connects

Uysal 2026 — Patient-Level Gleason Grading from Individual Sections — the source for everything measured on this page: a head-to-head comparison of max voting, majority voting and pixel-level packing for going from section to core to patient in prostate biopsies. Read it for the spread between rules rather than for its headline kappa, and note that its best number carries a supervision-inference mismatch the authors flag themselves.

Intratumoral Heterogeneity — the variable that decides which rule wins. Voting is a consensus operator, so it does best exactly where consensus exists and worst where the question is hardest.

Interobserver Agreement — sets the ceiling any aggregated answer is scored against, and supplies the weighted-kappa machinery used to score it. An aggregation rule tuned against a single routine reader is tuned against an unmeasured target.

Koyun 2026 — Slide Selection Introduces Sampling-Induced Prediction Uncertainty — the prior question and the second source: with the model, weights, patients and labels all held fixed, changing only which slide is analysed moves patient-level AUC between 0.46 and 0.97. It also supplies the feature-space route and the first aggregation result on this page validated on a second institution.

External Validation — packing feeds a model an image unlike its training data, so it is a strategy whose transferability is untested; the feature-concatenation route is the one aggregation strategy here that has been tested across cohorts, and it held.

Weakly Supervised Learning — the source of the slide-level predictions being aggregated. The two are usually designed separately and evaluated together, which is how the aggregation step escapes scrutiny.

Class Imbalance and Accuracy — a maximum rule is a sensitivity-maximising operator, so its apparent quality depends heavily on how rare the top grade is in the cohort.

Synoptic Reporting — the reporting unit is what fixes the aggregation target in the first place; if the report structure changes, the level a model should be supervised at changes with it.

Open questions

Is there a calibrated rule between max and majority — weighting each unit’s prediction by tumour extent or by model confidence, closer to how a pathologist actually integrates cores? Barely explored in the one source available.
Does packing survive a change of institution? Every packing result to hand comes from a cohort sharing an institution with the training data, and a composited slide is exactly the kind of out-of-distribution input where domain shift would be expected to bite hardest. See wiki/concepts/external-validation.md. Feature-space concatenation, the neighbouring route, does transfer — so the question is now specifically about compositing, not about aggregation in general.
Nobody has compared the four routes against each other. Max voting, majority voting, pixel packing and feature concatenation each have evidence, but from two papers on two organs with two endpoints and no overlap. Which one to reach for is still a judgement, not a result.
Would a rule that combines them win — aggregate everything, but weight each slide by its tumour content? Koyun uses tumour content only as an exclusion filter and Uysal only as a stratifier; neither uses it as a weight.
How much does packing degrade as the proportion of background rises?
Nothing here yet addresses aggregating continuous outputs — a proportion score, a count — where the natural rules are means and totals rather than votes, and where the failure modes should be quite different.