Multimodal Fusion
Combining images, molecular data and clinical variables in one model — where the interesting engineering is missing data, and where the honest question is whether fusion helped at all.
What it is
A patient generates several kinds of data that a single model can be asked to use together: a slide, a gene expression profile, a stage and an age, a free-text report. Fusion is how those get combined, and the choice of where in the network it happens is the main design decision.
Early fusion concatenates raw features and hands the lot to one model. Simple, and usually wrong when the modalities have wildly different dimensionality — 5,000 methylation probes will drown 10 clinical variables.
SurvivalNet is a concrete instance, from the same lineage four years earlier, and it shows the ratio problem at a readable scale rather than an abstract one. Its shipped cohort is a single 399-column matrix built by flattening five blocks together: 176 protein, 126 mutation, 63 CNV gene, 25 CNV arm, and 9 clinical. So the clinical variables — which in the MultiSurv experiments turned out to be the single strongest modality — are 2% of the input width, with no per-modality encoder and no mechanism to stop the molecular blocks dominating. Notably, the clinical features still surfaced at the very top of that model’s own attribution ranking, which says the network found them despite the ratio rather than because the architecture protected them. Early fusion is not fatal; it just leaves the balance entirely to the optimiser.
Late fusion trains a separate model per modality and combines their predictions, by voting or by stacking. Robust and easy to debug, but the modalities never interact, so any signal that only exists in the combination is unreachable by construction.
Intermediate fusion is what almost all current work does, and what is usually meant. Each modality gets its own submodel — a CNN for images, an embedding layer for categorical clinical variables, a fully-connected stack for high-dimensional omics — and each emits a fixed-length vector in a shared space. Those vectors are combined, and a shared head reads the combined representation. This is the pattern to recognise: per-modality encoder, common dimensionality, one fusion operator, one head.
The fusion operator itself is a smaller decision than it looks. Concatenation preserves everything but makes the head’s input size grow with the number of modalities. Element-wise sum, product or maximum keep the size fixed and impose a symmetry between modalities. Attention learns per-modality weights, which is the only option that says anything interpretable about which modality mattered. MultiSurv implements six operators and finds element-wise maximum best; the differences between them were small, which is the usual finding.
Missing data is the genuinely hard part. Real patients do not have every modality. A cohort with 90% clinical, 87% expression and 76% imaging coverage cannot be reduced to the complete intersection without throwing away most of it. The standard trick is to substitute a zero vector for the missing modality and let the network learn to ignore it — and it deserves more scrutiny than it usually gets, because it does not do what it appears to do.
MultiSurv is the instructive case, verified against its source. A missing modality is fed in as all zeros, but the submodel still runs, and its linear layers have bias terms — so an all-zero input produces a non-zero, patient-independent output vector. Every submodel ends in a ReLU, so all these vectors are non-negative, and under element-wise maximum fusion a non-negative constant can only ever raise the fused value. The “missing” modality is not inert; it injects the same constant for every patient who lacks it. The repository’s own EmbraceNet layer tries to detect missing modalities by checking whether the representation is all-zero, which by this argument it never is, so that detection silently fails.
The general lesson transfers to any architecture: if you zero-fill a missing modality, check what the encoder emits for a zero input, and check whether your fusion operator can actually ignore it. Masking the modality out of the fusion explicitly is the fix; hoping a zero propagates as a zero is not.
Whether fusion helps is an empirical question with an uncomfortable answer. It is easy to assume more modalities means better prediction. MultiSurv’s own numbers say otherwise: clinical data alone gave a pan-cancer Ctd of 0.809, the best pair (clinical plus gene expression) 0.822 with an overlapping confidence interval, and adding all six modalities including whole-slide images dropped it to 0.787. Adding image data made the model worse. The right conclusion is not that histology is uninformative — it is that a fusion result is only as good as its weakest encoder, and a weak encoder actively costs you. Any fusion claim needs the unimodal baselines next to it, evaluated on the same patients.
Why it matters for my work
The group holds two large modalities that have never been joined: Memorial WSI Share on the image side and Memorial LIS Extracts on the structured-clinical side, with Report Text Extraction able to produce a third from narrative reports. Nothing in this repo records a project that pairs them at patient level, which is the prerequisite for any fusion work at all — and, more immediately, the prerequisite for knowing whether a fusion project is even possible here. That is a cheap thing to check and a hard thing to fix later.
The realistic reading of MultiSurv for this group is deflationary and useful: the strong baseline is clinical data, the imaging contribution was negative, and the model needed 11,000 patients across 33 diseases to reach that. Before designing anything multimodal, the question worth answering is what a well-specified clinical model already achieves — see Survival Analysis Workflow. A fusion model that cannot beat it has told you something, but not what it set out to.
How it connects
Concordance Index — the metric every fusion claim is made in, and the reason MultiSurv’s pan-cancer gain looks larger than it is.
Weakly Supervised Learning — the image encoder in a fusion model faces exactly the slide-level-label problem, and a weak one drags the whole fused model down.
Foundation Models in Pathology — the most promising fix for the weak-image-encoder problem: a pretrained tile encoder replaces the ImageNet-pretrained CNN that underperformed here.
Patch Extraction — MultiSurv sampled 5 random patches per slide, which is the direct cause of its image submodel performing near chance; the decisions on that page set the ceiling.
Memorial LIS Extracts — the structured clinical backbone, and the modality that turned out to be the strongest single predictor in the worked example.
Memorial WSI Share — the image cohort that would have to be linked patient-by-patient to the above before any of this is actionable.
External Validation — a fused model has more ways to learn site-specific shortcuts than a unimodal one, since every modality carries its own institutional fingerprint.
MultiSurv — the reference implementation every claim on this page was verified against, and the software verdict on it: not runnable as shipped, but readable as a worked example.
SurvivalNet — the early-fusion counterpart above, and the reminder that the same group’s lineage started by concatenating everything into one flat vector.
Open questions
- Can Memorial WSI Share and Memorial LIS Extracts be joined at patient level with the existing pseudonymisation scheme? Unrecorded, and it gates everything above. [unverified]
- What does a properly specified clinical-only survival model achieve on the group’s own cohorts? Without that baseline no fusion result here would be interpretable.
- Does attention-based fusion give usable per-modality importance in practice, or only the appearance of it? MultiSurv implemented it but reported element-wise maximum, and does not say what attention showed.