Diagnostic Test Accuracy
The metric set that describes how well a test separates disease from no-disease, and the distinction between metrics that belong to the test and metrics that belong to the population.
What it is
Everything starts from a 2x2 table of test result against reference standard.
Metrics that are properties of the test, stable across populations: sensitivity (of those with disease, the proportion the test catches), specificity (of those without, the proportion correctly cleared), and the likelihood ratios (LR+ = sens/(1-spec), LR- = (1-sens)/spec), which convert pre-test odds to post-test odds and are the cleanest way to express test information.
Metrics that are properties of the test in a population, moving with prevalence: PPV and NPV. These are what a clinician actually wants — given this result, what is the chance of disease — but quoting them without the prevalence they were computed at is misleading. A test with 90% sensitivity and 90% specificity has PPV 50% at 10% prevalence and 90% at 50% prevalence. Same test, same performance, entirely different clinical meaning.
ROC and AUROC summarise performance across all thresholds and are prevalence-independent. AUROC has an intuitive reading: the probability that a randomly chosen diseased case scores higher than a randomly chosen non-diseased one. Confidence intervals should be bootstrapped rather than assumed normal, especially in small cohorts.
A frequently-missed point: the reference standard is itself imperfect. When the reference is a single pathologist’s read, measured accuracy is bounded by that reader’s reliability — see Interobserver Agreement.
Why it matters for my work
meddecide implements all of this and its chapter names it as the first of three questions for evaluating any new test, biomarker, or AI model. The prevalence pitfall is called out explicitly there: always report sensitivity and specificity and predictive values, together with the cohort prevalence.
This is the metric layer the group’s two vendor evaluations, Paige Prostate and Aiforia Breast, should be reporting — neither currently records its metric set. [unverified]
How it connects
Class Imbalance and Accuracy — explains why plain accuracy is excluded from this set and what to use instead.
Decision Curve Analysis — the next question after accuracy: does using the test help?
Interobserver Agreement — determines the quality of the reference standard, and therefore the ceiling on measurable accuracy.
External Validation — these metrics should be reported on an external cohort, where prevalence will differ and predictive values must be recomputed.
Model Abstention — the machine-learning name for an indeterminate result, and a reporting convention that diagnostic-test literature settled decades ago while model evaluation quietly dropped it.
Open questions
- What reference standard do the vendor evaluations use — single reader, consensus panel, or outcome follow-up? Not recorded. [unverified]
- Is the group standardising on bootstrap CIs for AUROC? meddecide provides them by default, so consistency is cheap.