Coverage Bias

When the data a metric needs is available for some records and not others, and availability correlates with what you are comparing — so the metric partly measures your data collection instead of the world.

When the data a metric needs is available for some records and not others, and availability correlates with what you are comparing — so the metric partly measures your data collection instead of the world.

What it is

Almost every real dataset is incomplete in a patterned way. Some references resolve to a PMID and some do not; some reports have a stated tumour size and some do not; some slides carry scanner metadata and some had it stripped. If you compute a statistic on only the records that have the field, you have quietly changed the population — and the result is a mix of the effect you wanted and the availability pattern you did not.

It becomes bias rather than mere missingness when availability correlates with the comparison. Two editions of a book compared on resolved references, where the newer edition resolves better because indexing improved, will differ partly because the field moved and partly because the newer one is better indexed. The two contributions are not separable by looking at the number.

The failure is rarely visible in the output. A percentage computed on 60% of the corpus looks exactly like one computed on 100%.

Three defences, in increasing order of effort

1. Declare the denominator. State which population every number is computed on, and use one umbrella identity for counting so that records without metadata still exist in the count. WHO CNS — Who Cites Who does this strictly: every reference carries one identifier by precedence, identity and volume statistics use all records, and metadata-dependent analyses disclose their coverage against the full corpus rather than silently shrinking it. That project reconciles four distinct denominators — raw extracted records, instances, unique records, and a smaller per-chapter population — and states which one each percentage uses. Cheap, and it converts an invisible problem into a visible caveat.

2. Coverage-correct the estimate. If you know the coverage rate per group, you can re-estimate what the metric would be at equal coverage and report both. WHO CNS — Who Cites Who’s citation-expansion ratio between editions is reported raw and coverage-corrected; the expansion attenuates but its direction survives, which is exactly the sort of thing worth knowing before quoting the raw figure.

3. Rarefy when the metric is sample-size sensitive. Richness measures — distinct journals, distinct authors, distinct anything — grow with sample size on their own, so a larger corpus looks richer for free. Rarefaction subsamples every group to a common size before comparing.

That third one earns its cost, because in WHO CNS — Who Cites Who it reversed a conclusion. Raw counts suggested journal diversity rose across editions. Rarefied to equal sample, distinct-journal richness does not rise and slightly falls — consistent with rising journal concentration measured by Gini — while author richness genuinely roughly doubles. Two findings that look identical in raw form and point in opposite directions once the size advantage is removed.

The related trap: an identity key that inherits the coverage gradient

If records are matched across groups on a field that is itself unevenly available, the match rate inherits the gradient. Intersecting two editions on PMID-exact identity undercounts the shared set whenever one edition resolves PMIDs better — the same reference fails to match itself. WHO CNS — Who Cites Who’s answer is to match on a canonical key built from author, year and title tokens, report that as primary, and carry the PMID-exact figure alongside as a conservative lower bound. The general form: do not match on the field whose availability you are worried about.

Why it matters for my work

The pattern recurs well beyond bibliometrics, and mostly without the defences above.

Report Text Extraction pulls structured fields out of narrative reports, and extraction succeeds more often for some report types, some years, and some authors than others. Any cohort defined by an extracted field is therefore a biased sample of the cohort you meant, and the page already records that extraction accuracy has never been evaluated — which is the same gap seen from the other side.

Memorial LIS Extracts carries documented field-reliability caveats, which is the same statement: some fields are trustworthy for some periods.

Mitotic Count is the sharpest imaging instance. Area-calibrated counting needs µm/pixel; if that metadata survives for some slides and not others, any cohort analysis of counts is conditioned on metadata survival.

Whole Slide Imaging records an open question about whether scanner and resolution metadata survives anonymisation — that is a coverage question, and the answer determines which slides can contribute to any calibrated measurement at all.

How it connects

WHO CNS — Who Cites Who — the worked example, and the only project here that applies all three defences.

Class Imbalance and Accuracy — a sibling failure of the same kind: a headline number that looks fine while the population underneath it makes it meaningless.

External Validation — coverage patterns are institution-specific, so a model or a threshold tuned on one site’s availability pattern is one more thing that may not transfer.

Report Text Extraction — the group’s most exposed instance, because downstream cohorts are defined by whatever extraction succeeded on.

Diagnostic Test Accuracy — the same structure as verification bias, where only some cases get the reference standard and the ones that do are not a random sample.

Citation Network Analysis — where the bounding and coverage decisions determine every centrality result.

Model Abstention — the same arithmetic one step later in the pipeline: there the input field is missing, here the model’s own output is, and a system with both has two denominators to declare rather than one.

Open questions

• What is the field-level coverage rate for Report Text Extraction, per field and per era? Not measured. [unverified] It is the number that would tell you which retrospective cohorts are safe to build.
• Does any analysis in the group currently report a denominator explicitly, outside WHO CNS — Who Cites Who? Nothing else in sources/projects/ records the practice. [unverified]
• Is rarefaction worth adopting as a house default for any richness or diversity comparison? It is cheap, and the one time it was applied here it changed the answer.