Gradient-Based Feature Attribution
Reading a model’s partial derivatives to ask which features drove its prediction — most useful not as an explanation for a paper but as an audit that catches what the model actually learned.
What it is
A Cox proportional hazards model hands you interpretability for free: one coefficient per covariate, the same for every patient, directly readable as a hazard ratio. A neural network does not. Gradient-based attribution is the cheapest way to recover something comparable — take the partial derivative of the model’s output with respect to each input feature, evaluated at one patient’s own feature values.
For a survival network whose output is a risk score, ∂risk/∂feature answers: if this patient’s value for this feature were nudged, which way would their predicted risk move, and how sharply? Run it for every patient and you get a matrix the same shape as your data — one attribution vector per patient rather than one coefficient per feature. That per-patient resolution is the real gain over a Cox coefficient, and it is also what creates the problem below.
Mechanically it is one backward pass per patient, reusing the same autodifferentiation the training used. It is by far the least expensive attribution method, which is why it is where people start.
The aggregation decision, which is where the method usually goes wrong
A per-patient matrix is not an answer. To say “these were the important features” you have to collapse it across patients, and that choice is load-bearing and rarely stated.
Ranking by the absolute value of the mean gradient cancels bidirectional features. This is the trap. Take the mean signed gradient per feature, then rank by its magnitude — and any feature that raises risk in half the cohort and lowers it in the other half averages to approximately zero and ranks near the bottom, despite being one of the most influential features in the model. Ranking instead by the mean of the absolute gradients cannot cancel, but throws away direction, so you learn that a feature mattered without learning which way. The two rankings answer different questions and neither subsumes the other; a heterogeneous-effect feature is precisely the interesting case, and the first method is blind to it. SurvivalNet takes the first route — argsort(-abs(mean(gradients))) — which is worth knowing before reusing its output.
Normalising each patient’s vector discards magnitude. SurvivalNet divides each patient’s gradient vector by its own L2 norm before averaging, which makes patients comparable in the direction of their sensitivity but destroys any statement about whose predicted risk is more volatile. If the question is “which patients is the model least confident about”, per-patient normalisation has already thrown the answer away.
Keeping the distribution beats any summary. Per-feature box plots across patients show the spread that a mean hides, and cost nothing extra once the matrix exists.
What a gradient can and cannot tell you
A partial derivative is local and linear. It describes the model in an infinitesimal neighbourhood of one patient’s feature values, and nothing further. Two consequences matter in practice.
A feature can be highly influential and still show a near-zero gradient, because the unit it feeds is saturated — a tanh activation far from zero has a vanishing slope. Zero gradient means “moving this slightly changes nothing here”, not “this feature is unused”. Conversely a large gradient at a point says nothing about the effect of a realistic change in the feature, because the relationship may not stay linear over that range.
This is exactly the gap that later attribution methods exist to close — integrated gradients accumulates the gradient along a path from a baseline to the actual input, and SHAP-style methods attribute the prediction relative to a background distribution rather than reading the slope at a point. Both cost far more compute than one backward pass. The general point to carry: a raw gradient is a sensitivity, not a contribution, and papers routinely present it as the latter.
Aggregating up to pathways
One idea worth keeping from SurvivalNet: once every feature has a signed attribution score, features can be mapped to gene sets and the scores carried up to pathway level, then written out as a GSEA-Preranked ranked-list file so that standard enrichment machinery does the rest. It turns a per-feature attribution — usually noisy at the level of any individual gene — into a statement about coordinated biology, which is both more stable and more interpretable. The implementation there is Python 2 and unusable, but the pattern (per-patient gradients → signed per-feature means → gene-set enrichment) transfers to any framework.
Why it matters for my work
The auditing use is the one to adopt, and it is not why people usually reach for this. Attribution is normally presented as an explanation for readers — a figure showing that the model found biologically sensible features. Its higher-value use is adversarial: run it on your own model and read the top of the ranking to find out what the model actually leaned on, before anyone else does.
SurvivalNet is the worked example, and it indicts itself. Its committed attribution output ranks the subtype-defining glioma alterations at the top of the protective end and a glioblastoma histology indicator second from the top of the risk-increasing end — that ranking is a subtype classifier, which is what makes its pooled concordance index uninterpretable. And fifth from the top of the risk-increasing end sits radiation_therapy-Is-yes, a treatment used as a predictor, with the model having learned that being irradiated predicts death because irradiation tracks the worst-prognosis subtype. Neither problem is visible in a concordance index. Both are obvious on one page of attribution output. That is the argument for computing it.
The group’s interpretability story is better than this page assumed — corrected 2026-07-26. Reading source rather than the project chapters, ClinicoPath jamovi Module ships dedicated interpretability analyses: explainableai (“SHAP values, and other interpretability methods”), and biomarkerdiscovery and clinicalprediction, both offering SHAP/LIME. Variable importance also appears in extratrees, highdimcox, ncvregcox, featurequality and clinicalnomograms. The ML survival models themselves — gradientboosting, treeensemble, lassocox — are in the umbrella too, not in jsurvival as previously stated.
The original gap survives in one specific and telling place: in multisurvival, the module’s main multivariable Cox analysis and one of the eight analyses in the focused jsurvival install, a SHAP option exists in the .a.yaml but is commented out. So the capability was built, then withheld from the analysis most people will actually run.
What remains true is the framing below: a Cox model reports coefficients, but a random survival forest reports nothing comparable, and “which variables drove it” is the first question any clinical collaborator asks. Gradient attribution as described here applies to differentiable models specifically — for the tree-based options the equivalents are permutation importance and SHAP — but the aggregation and auditing lessons above are method-independent.
Attribution is not a substitute for validation. A model can produce a beautiful, biologically coherent attribution profile and still fail on an external cohort, because the features it leaned on carry site or era signal that happens to correlate with biology. See External Validation.
How it connects
SurvivalNet — where the concrete implementation, its aggregation choice, and the self-indicting attribution output all live; this page exists because reading that repository raised a decision nobody here had written down.
Concordance Index — the complement: concordance says how well the model ranks patients, attribution says what it used to do so, and the two failures this page describes are invisible to the first metric and obvious in the second.
Survival Analysis Workflow — the workflow’s machine-learning options end without either an evaluation protocol or an interpretation step; this is the missing half of the latter.
jsurvival — the module where any of this would have to be implemented, and whose tree-based survival models currently report no feature importance at all.
MultiSurv — the contrast case: it implements attention-based fusion partly for interpretability but reports the non-interpretable maximum fusion instead, so its per-modality importance question is left open.
External Validation — a plausible attribution profile is not evidence of generalisation, and the features that look most sensible are often the ones most confounded with site.
Biomarker Cut Points — the same tension one step downstream: a per-patient continuous attribution is more honest than a threshold, and a threshold is what gets asked for.
Open questions
For the tree-based survival models, is permutation importance or a SHAP-style attribution available?Answered — it is, in ClinicoPath jamovi Module (explainableai,biomarkerdiscovery,clinicalprediction, plus variable importance in several tree and penalised-Cox analyses). The question that replaces it: why is the SHAP option inmultisurvivalcommented out? Deliberate (unfinished, or judged unreliable) or forgotten? It is the analysis most users reach for.[unverified]- Has any of the interpretability machinery been used on a real cohort here? Nothing in
sources/projects/records a run.[unverified] - Would the pathway-level handoff — per-feature attributions aggregated to gene sets and passed to preranked enrichment — be worth reimplementing for any of the group’s own work? It is the only capability in SurvivalNet with no current equivalent in the stack, and nothing here records a project with the gene-expression breadth it needs.
- Is there any group model, in any modality, where an attribution audit of the kind described above has actually been run? If not, the cheapest place to start is whichever model has a treatment or a site variable among its inputs.