Interobserver Agreement
How much two or more observers agree beyond what chance alone would produce — the ceiling on how good any diagnostic criterion or AI reference standard can be.
What it is
Raw percent agreement is misleading because two observers who both call almost everything “negative” will agree most of the time by accident. Chance-corrected statistics fix this.
- Cohen’s kappa — two observers, categorical scale. 0 means chance-level, 1 means perfect.
- Weighted kappa — for ordinal scales, where being one grade apart should count as a smaller error than being three apart. Linear and quadratic weights give materially different numbers; quadratic penalises large disagreements much more heavily. Pick one, state it.
- Fleiss’ kappa — more than two observers.
- ICC (intraclass correlation) — continuous measurements. Several forms exist depending on whether raters are fixed or random and whether you care about absolute agreement or consistency; the form must be reported.
- Krippendorff’s alpha — handles missing data and mixed measurement levels.
- Bland-Altman — not a single statistic but a plot of difference against mean, for continuous measurements. It reveals whether disagreement grows with magnitude, which a correlation coefficient hides entirely.
A pitfall that catches people repeatedly: kappa depends on the marginal distribution as well as the agreement. In a cohort where 95% of cases are one category, kappa can be low even with high percent agreement, simply because there is little room to beat chance. Report both the kappa and the raw agreement, and describe the category distribution.
Why it matters for my work
Agreement work runs through four separate projects. HER2 Intratumoral Heterogeneity has a completed pilot scoring round measuring agreement on HER2 IHC among department pathologists. Konsultasyon measures agreement between Memorial’s diagnosis and the referring lab’s. Omentum pilot-codes cases against a sampling-adequacy rubric versus a reference reader. A planned kidney tumour-and-necrosis study measures region-level agreement on QuPath polygons, which is a different problem again — spatial overlap rather than category match.
meddecide implements all of the statistics above and is the group’s standard tool for them. Verified from source on 2026-07-26: its agreement analysis covers weighted kappa with unweighted / linear / quadratic options, ICC, CCC, and Bland-Altman including limits of agreement, a proportional-bias test and a plot.
Two capabilities in that module answer questions this page keeps raising, and nothing recorded them. kappasizeci, kappasizepower and kappasizefixedn do sample-size planning for agreement studies by three different approaches — confidence-interval width, power, and minimum detectable value at fixed n. Several agreement studies are described across this wiki and not one records how it was powered; the tool is already installed. And nogoldstandard implements analysis without a gold standard, which is the situation whenever the reference is itself a fallible reader — Konsultasyon’s comparison against the referring laboratory’s diagnosis being the clearest in-house case.
Agreement is also the frame for reading a standardisation guideline. Ki-67 Proliferation Index is a canonical low-agreement IHC marker, and the whole reason the UK Ki-67 recommendation (sources/papers/shaaban-2026-uk-ki67-recommendations.md) exists is to lift between-reader and between-laboratory agreement on it. The paper’s claim that digital and AI scoring improve reproducibility is a claim about raising this ceiling — the same ceiling that decides whether an AI-versus-single-reader comparison can mean anything. That claim carries no kappa in the abstract, so it is [unverified] and would be worth checking against an in-house reader-versus-AI round.
The sharper version of that problem is a criteria set whose agreement was never measured at all. Aust 2010 — Serrated Polyp Diagnostic Criteria (German consensus) is the German Society of Pathology’s consensus on serrated colorectal polyps, and it is worth reading here not for the histology but for how the criteria were made: a one-day meeting, 19 selected cases, and a diagnostic rule — two of four architectural features present in at least two crypts — chosen because it was tractable. The paper states the criteria were applied “with good interobserver concordance” and gives no kappa, no reader count and no design, while conceding in the same sentence that this “has to be validated in future prospective studies”.
That is this page’s central claim happening in public. If agreement on a criterion is the ceiling on any reference standard built from it, then a criterion published without an agreement figure has an undefined ceiling — and everything downstream inherits it: the pathologist applying it, the synoptic field recording it, the AI model trained on labels derived from it. Note the direction of the problem, because it is easy to get backwards. The risk is not that the criteria are wrong. It is that a later study measuring reader agreement on serrated polyps cannot tell how much of the disagreement it finds belongs to the readers and how much belongs to a rule that was never calibrated. Selected teaching cases make this worse rather than better: the ambiguous lesions that would expose an unstable rule are exactly what a teaching set leaves out.
The generalisable question to ask of any criteria set before adopting it as a reference standard: was the agreement measured, on how many readers, on cases selected how? All three are stated here as absent rather than as values, which is at least honest.
How it connects
Intratumoral Heterogeneity — a genuine biological cause of disagreement; when a marker is truly heterogeneous, low agreement is a property of the tumour, not a failing of the observers.
Diagnostic Test Accuracy — agreement sets the ceiling: if pathologists agree at kappa 0.6, no AI model can meaningfully exceed that against a single-reader reference standard.
Aust 2010 — Serrated Polyp Diagnostic Criteria (German consensus) — where the ceiling gets set in the first place. A national consensus criteria set whose reproducibility is asserted and never measured, which is the upstream version of every unmeasured reference standard on this page.
Biomarker Cut Points — the same paper’s “two of four features in two crypts” rule is a categorisation threshold with no derivation, so the two pages share a worked example: agreement and cut-point choice are two views of one decision.
QuPath Annotation Workflow — the annotation pipeline that produces the region-level data the kidney agreement study will analyse.
Synoptic Reporting — structured reporting reduces disagreement by removing ambiguity about what is being reported, which is one of its main justifications.
HER2 Gastric Cohort — the cohort behind the group’s completed HER2 pilot scoring round, and the most advanced agreement dataset currently available.
Nottingham Grading — an ordinal three-level outcome read by multiple pathologists, so weighted kappa applies directly; a glass-versus-WSI grading round is the most obvious unstarted agreement study in the repo.
Mitotic Count — a worked example of why one kappa is not enough: hotspot choice, area definition, and figure recognition are three separable disagreement sources, and a single agreement statistic cannot say which one to fix.
Ki-67 Proliferation Index — the textbook low-agreement IHC marker; its national guidance is a direct attempt to raise the agreement ceiling this page is about, which makes its reproducibility claim testable rather than rhetorical.
Cell Detection and Counting — agreement on where a cell centre lies is the ceiling on any measured detection accuracy, and it also sets a floor on error: hard example mining preferentially selects exactly the ambiguous cells annotators disagree about.
Cervical Cytology Screening — where reader variability is best studied, because the Bethesda categories are fixed: disagreement is measurable rather than diffuse, which makes the ceiling this page describes an actual number.
Konsultasyon Log — the cheapest unstarted agreement study in the repository, because the data already exists: every consultation is a second read of a case someone else has already signed out. An agreement analysis over the log needs no new scoring round, only a coding rubric for what counts as refinement, reclassification and true discrepancy.
Open questions
- The Konsultasyon chapter notes that “disagreement” bundles refinement, reclassification, and true discrepancy. Standard kappa cannot represent that three-way structure — is an ordinal or hierarchical coding scheme planned?
- For the kidney region-agreement study, which overlap metric (Dice, IoU, or a tolerance-band measure)? Not yet decided. [unverified]
- No page yet on intra-observer (test-retest) agreement, which is the usually-unmeasured companion to interobserver work.
- An instrument for blinded reader studies now exists in-house, and it measures a dimension this page does not cover. QuPath Patoloji Atlası Extension ships a
focus/package — blinded case presentation, a decision dialog, and a viewing heatmap — with a matching analysis toolkit written twice, in Python and R, deliberately mirroring each other. Beyond the decision it records where each reader actually looked: dwell grids, fixations by I-DT, navigation linearity, zoom behaviour, hotspot detection, and pairwise similarity between readers by correlation, KL and Jensen–Shannon divergence, NSS, AUC-Judd and IoU. The R side computes mean pairwise correlation and ICC(2,1) viairr::icc. That reframes what agreement can mean here. Two pathologists can reach the same diagnosis from different regions of a slide, and this measures that difference — which bears directly on the hotspot-selection problem in Ki-67 Proliferation Index and Mitotic Count, where where you look largely determines the number you report. Nothing in the repository records a completed study, so the instrument is implemented and not yet exercised. [unverified] Running it on the kidney region-agreement study would answer the overlap-metric question above with data rather than by choosing. - Were any of the group’s agreement studies powered using the
kappasize*analyses, or were sample sizes chosen by convenience? Nothing records it either way.[unverified]The pilot-then-full-study pattern in HER2 Intratumoral Heterogeneity is where it would apply. - Would
nogoldstandardbe the more defensible analysis for Konsultasyon than kappa against the referring diagnosis? Treating the outside diagnosis as truth is the assumption that page’s own pitfalls warn about.