MultiSurv

Reference implementation of a pan-cancer multimodal survival model (clinical + four omics + WSI), read as a worked example of multimodal fusion rather than as a tool to adopt.

Purpose

Third external repository on this list, after ScanTools and WSITools, and the first that is the companion code to a paper rather than a tool in its own right. The paper is Vale-Silva & Rohr, Sci Rep 2021;11:13505 — see the full read at sources/papers/vale-silva-2021-multisurv.md, which is where the scientific criticisms live. This page covers only the software question: can any of it be used here.

The README is unusually honest about the answer: “This is experimental work, not a directly usable software library.” That assessment holds up. status: archived records that it was evaluated and closed, not that it failed — the reading was worth doing.

Data used

None locally. The repository ships manifests and preprocessing notebooks for TCGA data pulled from the NCI Genomic Data Commons; it contains no patient data, only a labels.tsv of pseudonymised TCGA barcodes with follow-up time, event, cancer type and split assignment. Reproducing the study end to end needs roughly 13 TB of whole-slide images plus ~2 TB of omics.

Methods

What the code actually contains, from a source-level read:

  • src/multisurv.py — the model: per-modality submodels emitting 512-d vectors, a fusion layer, then a 4-layer head and 30 sigmoid outputs (one conditional survival probability per yearly interval).
  • src/loss.py — discrete-time survival negative log-likelihood. The most reusable file in the repository: about 50 lines, no dependencies beyond torch, and a correct implementation of a non-proportional-hazards survival loss. See Concordance Index for why this formulation is worth knowing.
  • src/sub_models.py — entity embeddings for categorical clinical data, plain FC stacks for omics, ResNeXt-50 with frozen early layers for image patches.
  • src/embrace_net.py, src/attention.py — two of six selectable fusion methods.
  • src/evaluation.py — Ctd, IBS and INBLL via pycox, with bootstrap CIs.
  • data/*.ipynb — preprocessing per modality, written against absolute paths on the author’s own machines (/mnt/dataA/…, /home/luis/net/Isilon/…).

Long-term cancer survival prediction using multimodal deep learning — the paper this code accompanies, and the source of every number on this page.

Corrigendum to the MultiSurv paper note — the correction issued against it, worth reading beside the original rather than after it.

Current state / open questions

Not runnable as shipped. environment.yml pins 138 conda packages with Linux-specific build strings, on Python 3.6.7 (end of life), PyTorch 1.4 and CUDA 10.1. It will not solve on Windows or Apple silicon. Two pip pins are the real blockers even on Linux: pysurvival==0.1.2 needs a C++ build against an old NumPy ABI, and wsipre==0.1.2 is the author’s own WSI-reading package, itself unmaintained. Anything reused from here should be lifted file by file into a current environment, not installed.

One latent bug and one code smell found while reading. EmbraceNet detects a missing modality by testing whether its representation vector is all-zero, but a zeroed input still produces a non-zero representation through the layer biases, so a missing modality is scored as present — verified by instantiating the repo’s own model. This affects the embrace fusion option, not the element-wise-maximum fusion the paper reports. Separately, src/evaluation.py:187 compares a string with is not rather than !=, which is an identity test that happens to work only by CPython interning.

A third defect, in the image pipeline. Patches are fed to an ImageNet-pretrained ResNeXt-50 as raw [0,1] RGB — grep for Normalize across src/ returns nothing, and the only scaling is a divide by 255. Pretrained weights expect the channel normalisation they were trained under, so this quietly discards much of what the frozen layers can contribute. The preprocessing notebook even computes patch mean and standard deviation, apparently without ever applying them. A plausible partial explanation for the image submodel’s near-chance performance, and a cheap thing to get right in any pipeline that reuses pretrained features.

How well it actually performed, stated accurately. Against seven baselines on unimodal data, MultiSurv ranks first on gene expression, microRNA and copy number; second on clinical data, where Nnet-survival beats it on all three metrics; and sixth of eight on DNA methylation, behind plain Cox regression. The paper’s own “best results in nearly all configurations” is the right summary — it is competitive and often best, not a clean sweep.

Worth taking, if anything: the discrete-time loss, the entity-embedding treatment of categorical clinical variables, and the general shape of the modality-submodel-then-fuse design. Not worth taking: the WSI pipeline, which samples 5 patches per slide and performs barely above chance.

Open questions for anyone who picks this up:

  • Trained weights are not published, so nothing here can be used as a pretrained starting point.
  • Does the group have any use case with genuinely paired modalities on the same patients? The fusion architecture is only interesting when it does — currently unrecorded anywhere in this repo. [unverified]

Read at commit deb35e1 (2022-03-01).