Class Imbalance and Accuracy
When one class dominates a dataset, overall accuracy stops being informative — a model can look excellent while being useless on the class you care about.
What it is
Take a cohort where 5% of cases are positive. A model that predicts “negative” for everything scores 95% accuracy and has zero clinical value. That is the whole problem in one line, and it recurs constantly in pathology because the interesting findings are usually the rare ones.
Accuracy fails because it is a single number averaging over classes of very different sizes. The fix is to report metrics that do not average away the minority class:
- Sensitivity and specificity are computed within each true class, so they are unaffected by prevalence. They describe the test itself.
- PPV and NPV answer the clinically useful question — given this result, what is the probability the patient truly has the condition — but they do depend on prevalence. The same test has very different PPV in a screening population and a referral population.
- Balanced accuracy or macro-averaged F1 give each class equal weight, which is usually what you want for a multi-class problem.
- AUROC is prevalence-independent but can look flattering under heavy imbalance; precision-recall AUC is more honest there.
For multi-class problems — the situation in tumour-origin prediction — always report the per-class confusion matrix, not just a headline number. A model can be excellent on the three common primaries and near-random on the eight rare ones, and a single accuracy figure hides that completely.
Why it matters for my work
The schema names accuracy-on-imbalanced-data as one of the statistical problems to flag on sight, and it is directly relevant to two current strands. Liver Tru-Cut — Tumour Origin AI predicts primary tumour origin, which is inherently many-class with very uneven frequencies — colorectal and breast primaries will dominate any liver-metastasis cohort while rare primaries have a handful of cases each.
The meddecide chapter makes the same point from the clinical side: it explicitly recommends using diagnostic-accuracy and decision-curve framing rather than raw accuracy when comparing an AI model against pathologists.
A worked positive example is now filed, and it is worth reading as one. Studer 2023 — Tumor Budding T-cell Graphs (pT1 CRC resection risk) classifies pT1 colorectal patients as needing resection or not, at 541 versus 85 — a 6.4:1 imbalance — and never quotes accuracy at all: per-class-average F1, specificity (TNR) and sensitivity (TPR) are reported separately throughout, the imbalance is stated up front, and the folds are class-stratified. The numbers this discipline produces are unflattering (average F1 around 40%), which is precisely the point: the honest metric set makes a modest model look modest, where accuracy would have scored the same model near 86% by rewarding it for the majority class. When someone asks what reporting under imbalance should look like, that paper is now the local answer — including its counterweight, a headline picked from 144 configurations, which is a different inflation route that no metric choice fixes.
How it connects
Diagnostic Test Accuracy — supplies the prevalence-aware metric set that replaces raw accuracy, and is implemented in meddecide.
Decision Curve Analysis — goes one step further by asking whether using the model helps the patient at all, which is a stronger question than any accuracy metric answers.
External Validation — prevalence usually shifts between cohorts, so predictive values must be recomputed rather than carried across.
Weakly Supervised Learning — MIL training itself is affected: with heavy imbalance the aggregation step can collapse to always predicting the majority bag label.
Hard Example Mining — the training-time response to the same imbalance: concentrate the loss on what the model gets wrong instead of letting an easy majority dominate the gradient.
Model Abstention — the compounding case. When a model can also decline to answer, the base-rate problem and the denominator problem multiply, and a published accuracy can move by twenty points purely on the handling of cases nobody scored.
Cervical Cytology Screening — the canonical imbalanced population, and the setting where this page’s argument is least avoidable: a screening cohort is overwhelmingly negative, so an accuracy figure on one says almost nothing.
Open questions
- What is the actual class distribution of primaries in the liver tru-cut cohort? Not recorded. [unverified]
- Is there a group convention on which multi-class summary metric to report? Standardising this would make the AI projects comparable to each other.