Model Abstention

When a model can return nothing, the denominator becomes a choice — and a paper that reports accuracy over all cases while reporting precision over only the answered ones has published two numbers about two different populations.

What it is

Most evaluation assumes every case gets a prediction. Real systems break that assumption in three different ways, and only the first is usually recognised as abstention at all:

By design. A selective classifier with a reject option: below a confidence threshold it declines and defers to a human. This is a deliberate, tunable safety feature.
By pipeline failure. A multi-stage system where an early stage returns nothing, so the later stage never runs. No decision was made to abstain; the case simply fell out.
By refusal. An LLM that answers “I cannot determine this from the provided text”, returns malformed output, or times out.

All three produce the same artefact — a case with no prediction — and all three raise the same question, which is not “how often does it happen” but which population is each reported metric computed on.

Abstention turns one denominator into several

Once some cases have no prediction, every metric can be computed at least four ways, and they do not agree:

Drop the abstentions. Metrics describe the model conditional on it having answered.
Score them as errors. Metrics describe the deployed system, which is usually what a clinician cares about.
Impute a class — usually the negative or majority class, as a “no evidence found” default.
Treat abstention as its own outcome, and report coverage separately.

The trap is not choosing badly. It is choosing differently for different metrics in the same table, which is easy to do accidentally because the calculations usually live in different lines of code.

The worked example

Wang et al. 2026 (sources/papers/wang-2026-pathology-cot.md) released per-slide predictions for their agent on an external cohort, which is what makes this checkable — almost no paper can be audited this way. On 321 slides the agent returned nothing for 73 of them (22.7%), all of them ground-truth negative, because its navigation stage proposed no region and the reasoning stage was therefore never called.

Recomputed from that released file:

Convention for the 73	Accuracy	Precision	Recall
Dropped	89.9	62.5	97.6
Scored as errors	69.5	62.5	97.6
Imputed POSITIVE	69.5	29.2	97.6
Imputed NEGATIVE	92.2	62.5	97.6
As published	69.4	62.9	97.6

The published accuracy is only reachable by scoring the abstentions as errors; the published precision is only reachable by excluding them. Reported side by side, they describe different populations. The spread is not cosmetic — accuracy ranges over 23 points and precision over 33, entirely on the handling of cases the model never answered.

Three general lessons fall out of that table.

Recall was invariant, and that is not reassurance. All 73 abstentions were negative, so none could be a false negative and recall is 97.6% under every convention. A metric that cannot move is not evidence that the handling was sound — it is a metric that happens to be blind to the problem. Checking one metric for stability proves nothing about the others.

The honest result is an interval, not a number. What the agent would have done on those 73 slides is unmeasured. If it had got them all wrong, precision is 29.2%; all right, 62.5%. The defensible statement is that precision lies in [29.2%, 62.5%] and that the paper reports the upper end. Any single number here is an assumption wearing a result’s clothes.

Abstention correlated with the outcome. All 73 were negative — not a random 23% of the cohort. Whenever the cases a model skips are systematically different from the ones it answers, dropping them is the Coverage Bias mechanism operating on the output side rather than the input side. That correlation is the thing to test for, and it is one line of code.

What to report instead

Coverage is a result, not a footnote. State the answered fraction next to every metric, and compute every metric on the same denominator. Two numbers — coverage and accuracy-at-that-coverage — describe the system; one number describes half of it.

Compare at matched coverage. A model answering 77% of cases cannot be compared with one answering 100% on accuracy alone; the first has been allowed to skip its hardest cases. The standard device is a risk–coverage curve: sweep the abstention threshold, plot error against the fraction answered, and compare curves rather than points. Comparing two models at different coverages, each computed its own way, is the failure to avoid — and it happened within a single published figure, where one model’s abstentions were penalised and another’s were dropped.

Say where the abstentions came from. Designed rejection is a feature and belongs in the abstract. A silent except: return None is a bug that has been reported as a performance characteristic, and only reading the code distinguishes them.

Why it matters for my work

Two things here are already exposed, and neither records a coverage figure.

LLM Report-Quality Checking scores reports against CAP/ICCR checklists using an LLM, which can return malformed output, refuse, or time out. If those reports drop out of the evaluation, the measured agreement is conditional on the model having produced parseable output — and the reports that break a parser are plausibly the unusual ones, which is exactly where a checker earns its keep. The page already records that no human-reviewer validation exists; coverage is the second, cheaper thing to measure, and it can be computed on data already collected.

Report Text Extraction has the same shape one layer down and higher stakes, because cohorts are defined by extracted fields. A case where extraction returned nothing silently leaves the cohort. That is Coverage Bias on the input side and abstention on the output side, and in a pipeline they are the same event seen from two ends.

The pattern generalises to any two-stage design the group might build: a detector feeding a classifier, a QC step feeding a model, a registration step feeding a comparison. Whenever stage one can return nothing, stage two’s metrics silently acquire a denominator. WSI Quality Control is the clearest instance — slides rejected by QC are exactly the slides a model would have found hard, so accuracy measured after QC is measured on an easier cohort than the one in the scanner queue.

How it connects

Coverage Bias — the same arithmetic on the input side: there the data is missing, here the prediction is. A pipeline usually has both, and the defences are the same three — declare the denominator, correct for coverage, compare at matched sample.

Class Imbalance and Accuracy — the compounding failure. On a 12.8%-prevalence cohort, “always negative” scores 87.2%, higher than every model in the worked example above including the agent’s published 69.4%; abstention then moves the number by another 23 points. Two headline numbers, neither meaning what it appears to.

Diagnostic Test Accuracy — the clinical-literature name for the same structure is an indeterminate result, and the reporting standards there already require it to be counted. Machine-learning evaluation quietly dropped a convention that diagnostic-test reporting settled decades ago.

External Validation — abstention rates travel badly. A model that answers everything at home can abstain heavily on a different scanner, which is precisely what happened in the worked example: an out-of-distribution cohort broke the navigation stage, not the reasoning stage.

WSI Quality Control — rejecting slides before a model sees them is abstention moved upstream and given a respectable name; the denominator question is identical.

Decision Curve Analysis — the framing that makes abstention tractable rather than embarrassing: if declining routes a case to a human, its cost is a human read, and net benefit can price that directly instead of pretending the case was answered.

Open questions

What is the coverage of LLM Report-Quality Checking on the reports it has already been run over? Unmeasured. [unverified] It needs no new data and would take an afternoon.
Same question for Report Text Extraction, per field and per era — already listed as an open question on Coverage Bias from the input side. Asking it once answers both.
Should a coverage figure be a required field on every model page in this wiki, alongside scanner, magnification and validation strategy (AGENTS.md §8)? Nothing here records one for any project, and this page exists because a published paper got it wrong in a way nobody caught.
Is there a defensible house convention — report at 100% coverage with abstentions scored as errors, and at the model’s own coverage, always both? That is two numbers instead of one and removes the choice entirely.