GrandQC Quality Study

The laboratory’s 502-slide comparison of technician quality calls against GrandQC artefact percentages — the group’s first measurement of a QC model on its own material, presented as an oral abstract — which finds agreement not reliably distinguishable from chance, because 70% of the defects technicians record have no artefact class at all.

Purpose

This is the study GrandQC-QuPath said was missing. That page installed the published GrandQC models into QuPath and closed with the open question the wrapper could not answer: does GrandQC’s artefact vocabulary match what actually goes wrong in this laboratory? This repository answers it, on 502 slides, and the answer is largely no — for reasons that turn out to be more interesting than a bad score.

It is a fork of upstream cpath-ukk/grandqc with the study built on top, so the README.md at the root is upstream’s, not the study’s — it describes the published tool and its Nature Communications 2024 figures (Dice 0.957 tissue segmentation), none of which were measured here. Anyone opening this repository expecting to find the study described will read the tool’s advertisement instead. The study lives in analysis/, investigation/ and analiz_raporu_TR.md.

The work was accepted as an oral presentation (abstract 0797, PSS-013, Dijital Patolojide kesit kalitesi değerlendirmede İnsan mı? Yapay Zeka mı?, delivered 15 November 2025), with ten authors across the Memorial pathology laboratory and Medisa.

Data used

The four fields AGENTS.md §8 asks for, which are recoverable here — unusually for this estate:

• Cohort. 502 matched H&E and special-stained slides from routine work, drawn from 1001 technician evaluations and 708 GrandQC outputs. Matching was by filename prefix. Those 502 rows cover 437 distinct slides, not 502 — see the pseudoreplication note below, which matters for every statistic computed from them.
• Scanners. Leica Aperio AT2 and GT450, scanned consecutively.
• Magnification. The 10x artefact model (GrandQC_MPP1.pth, MPP_MODEL = 1 in 01_WSI_inference_OPENSLIDE_QC/main.py), with tissue detection at Tissue_Detection_MPP10. Worth recording because it is not the 7x default the README describes — the highest-accuracy and slowest of the three released models was chosen deliberately. Reported runtime 1–4 minutes per slide depending on tissue area.
• External validation. None, and none intended — this is a local agreement study, not a model evaluation. See External Validation.

Only 50.1% of the technician evaluations matched a GrandQC output (502 of 1001; 548 no_match, plus 261 GrandQC-only). Nothing establishes that the matched half is representative of the whole, and the plausible failure modes are not neutral — a slide rescanned under a new name, or one that failed to scan, is exactly the kind that a quality study should not lose. This is Coverage Bias in its concrete form, and it is unaddressed. [unverified] whether match failure correlates with either rating.

Methods

A four-stage pipeline in analysis/scripts/pipeline/ (01_matching_review → 02_comparison_analysis → 03_issue_analysis → 04_validation_summary), plus a separate investigation/ pass over the discrepant cases and a QuPath integration script.

The technicians score each slide on a 13-item form — recorded in data/manual/manual-control.csv and the most valuable artefact in the repository. Translated: tissue too large for the block, tissue outside the section area, section overflowing the slide, residue, folds, detachment, section that did not come through, tears, off-centre (acceptable for H&E but marginal for special stains and IHC), separated or non-parallel tissues with gaps, thick section, oblique section, and bubbles or coverslipping faults. Alongside these sit an ordinal Genel Skor (ideal 0 / acceptable 1 / unacceptable 2), a binary UYGUNLUK suitability call, and free-text error-source and error-reason fields.

GrandQC segments five artefact classes: tissue fold, dark spot and foreign body, pen marking, air bubble and slide edge, out of focus.

Those two vocabularies barely overlap, and the repository measures by how much. 03_issue_categories.csv maps each recorded technician issue onto a GrandQC class, and 388 of 554 issues (70%) come out Unmapped. The mapped remainder is small: out of focus 71, tissue fold 41, dark spot/foreign 30, air bubble/edge 24. Grouped by cause, the technician issues are 168 sectioning, 62 preparation, 36 tissue quality, 19 technical, 269 other. Nine of the thirteen criteria are microtomy and mounting geometry — centring, tilt, tears, overflow, gaps, detachment — and GrandQC has no class for any of them, because it segments the scanned image and these are properties of how the section was cut and placed.

That single number is the study’s real finding and the mechanism behind everything below.

Current state / open questions

Active; 26 commits, last pushed 2025-12-09, head is WIP. The abstract is delivered, so the natural next step is a paper.

The headline agreement figure is not chance-corrected, and chance-corrected it nearly vanishes. Recomputed from technician_vs_GrandQC.csv (502 rows, all matched=Yes), collapsing GrandQC at the 5% boundary against the binary technician call:

	GrandQC <5%	5–10%	>10%	total
Technician suitable	280	36	29	345
Technician non-suitable	113	27	17	157

Observed agreement 64.5%, expected-by-chance 60.6%, Cohen’s κ = 0.10 — “slight” on any conventional reading. A χ² test of independence on the full 2×3 table gives 5.79 on 2 df (p ≈ 0.055 asymptotic, confirmed at 0.055 by a 200,000-replicate fixed-margin permutation test; Cramér’s V = 0.11): the two ratings are not distinguishable from statistically independent at the conventional threshold. Prevalence of technician-called defects is 31.3%. See Interobserver Agreement.

That κ = 0.10 is the most favourable of four reasonable choices, and the others are worse. Cutting GrandQC at >10% instead gives κ = 0.030 (95% CI −0.040 to 0.100). Using the continuous artefact percentage at the same 5% cut — which is what the pipeline actually does — gives κ = 0.073 (CI −0.014 to 0.161). Both cross zero. Only the ≥5% categorical version has a naive CI excluding zero (0.012 to 0.189), and that one does not survive the next point.

The 502 rows are not 502 independent slides. They cover only 437 unique manual_slide_name values: 31 rows are exact full-row duplicates, and a further 36 manual slides are matched to more than one GrandQC output, which is what serial sections of one block look like after a prefix match. Every script here, including the official pipeline, treats all 502 as independent, which understates the variance of any agreement statistic. Resampling by slide instead moves the κ interval to −0.001 to 0.200 — touching zero. The honest statement is therefore not “slight agreement” but agreement not reliably distinguishable from none.

The categorical column disagrees with the continuous one on 11 slides (2.2%). Cross-checking GrandQC_Artifact_Category against total_artifacts_percent in 02_comparison_results.csv (rows align 1:1) finds cases binned as “Less than 5%” whose recorded artefact percentage is as high as 87%. This is a binning defect, not a boundary-rounding artefact, and it sits in the column every agreement figure above is computed from. It should be found and fixed before any of these numbers are published.

The reported accuracy is worse than the do-nothing baseline. 04_final_summary.csv reports accuracy 0.635, precision 0.380, recall 0.261, specificity 0.806. But 345 of 502 slides are technician-suitable, so a classifier that labels every slide suitable scores 0.687 — higher than GrandQC’s 0.635. An accuracy below the majority-class rate carries no evidence that the model tracks the technician at all. This is precisely Class Imbalance and Accuracy, on the group’s own data.

The metrics treat the technician as ground truth, which no one has established. 04_validation_summary.py:68 maps UYGUN/UYGUN DEĞİL to 0/1 as the true label and derives TP/FP/FN against it. That is the vocabulary of Diagnostic Test Accuracy, and it needs a reference standard; here there are two fallible raters and no gold standard, so the appropriate frame is agreement, not validation. No κ appears anywhere in the pipeline.

The technician instrument disagrees with itself, which reframes the whole comparison. The binary field calls 295 of 1001 slides unsuitable (29.5%), while the ordinal Genel Skor — filled for only 309 rows — calls just 19 of them unacceptable (6.1%). So UYGUN DEĞİL in practice means “has a defect worth noting”, not “reject this slide”. The abstract is consistent with that reading and says so plainly: no slide carried a defect that would affect diagnosis or require rescanning. Once that is clear, a low κ against an artefact-area percentage is close to the expected result rather than a disappointment — the two instruments were never measuring the same construct.

Points worth carrying forward:

• The most valuable next analysis is already possible and needs no new data. Score weighted κ on the ordinal Genel Skor against GrandQC’s continuous artefact percentage, on the 309 slides where the graded score exists, instead of collapsing both to binary. Ordinal-versus-continuous is the comparison the data actually supports. The .omv file at the root means this is a meddecide job in jamovi rather than new code.
• investigation/actionable_insights.csv proposes raising the artefact threshold from 5% to 9.1%, derived from the median of the discrepant cases themselves. Tuning a threshold on the same 502 slides it will be scored against would make any resulting agreement figure circular. If that threshold is adopted it needs a held-out set, and the tuned number must not be reported as performance.
• technician_artifact_analysis.py charts real technician percentages against hardcoded numbers. Its original_data dictionary — the entire “GrandQC Model” side of the comparison — is eyeballed off a box plot, with the source comment “Based on the box plot, approximate the median values”. It renders at 300 dpi to investigation/technician_vs_model_artifacts.png, which is committed. The axis label says “(Approximate)” and the figure otherwise looks measured. This is the same failure as the mock-data fallback recorded on Stain Quality — synthetic numbers rendered into an artefact indistinguishable from a real one — and it is now the second instance in this estate, which makes it a habit rather than an accident. The figure should be regenerated from data/grandqc/artifact_analysis.csv, which is sitting in the repository.
• Two of the five artefact-category mappings in that script are forced, by its own admission: Dökülmüş (tissue detachment) is mapped to Pen Marking with the comment “Approximating pen marking with tissue artifacts”, and Kalın Kesit (thick section) to Out of Focus. Detachment and pen marks have nothing in common. Any per-class comparison built on that mapping is reporting an alignment that was invented to make the categories line up.
• analiz_raporu_TR.md does not reconcile with the data or the abstract, and the cause is now traced. It reports 536 matched records, 331 concordant, 61.7% agreement. The comparison file holds 502 rows; so does 02_comparison_results.csv; so does 04_final_summary.csv’s own “Matched Slides”; and so does the abstract. The 536 comes from analysis_report.md, which describes an earlier merge — and that older analysis reports its own discrepancy count as 75 + 118 = 193, not 205. Since the Turkish report’s 205 matches the current 502-row file cell for cell (29 + 36 + 113 + 27), it has combined a stale total with discordant counts from a different, later dataset. Tallying the two genuine agreement cells on the real data (280 + 17) gives 297 concordant, not 331 — and the 34-record overstatement is exactly 536 − 502, so the phantom records were all absorbed into the agreement bucket. True raw agreement on the collapsed table is 64.5%, not 61.7%; the direction of the error is not the point, the provenance is. The abstract is the trustworthy document. The Turkish report should not be the basis of a paper without this being fixed first.
• The qualitative conclusion — technicians catch what is visible on the glass (placement, overflow, orientation), GrandQC catches what is visible only in the scan (focus, bubbles, small section faults) — is well supported, and the 70% unmapped rate is its quantitative backing. Note though that 04_validation_summary.py writes its five closing recommendations, including “Use GrandQC as a complementary tool, not replacement”, as fixed strings independent of the results. The conclusion is sound and the data support it, but it was not derived by that script.
• Whether technician calls stand in for pathologist calls is the open question this study narrows without closing. GrandQC-QuPath asked for agreement against a pathologist’s own artefact calls; technicians and pathologists are different rater classes with different thresholds and different stakes, and the 13-item form is a preparation-quality instrument rather than a diagnostic-adequacy one. [unverified]

Identifier exposure, and the eighth instance of Known gap 8. The .gitignore is a single line, *.pth. Of 220 tracked files, 46 under examples/ (25 PNG, 21 JPG) carry filenames in the laboratory’s accession format — case number, two-digit year, block letter, stain code — and a repository-wide search finds the same identifiers inside 44 tracked files, across the root, analysis/, investigation/, archive/old_analysis/, docs/reports/ and the rendered HTML and markdown reports. analiz_raporu_TR.md lists two dozen in worked examples. All 46 example files entered in the single most recent commit, 26ae972, which is already pushed to origin.

Two bounds worth recording, both checked rather than assumed. The repository is private (gh repo view), so this is governance rather than disclosure, consistent with every other instance on that list. And the exposure is confined to filenames and table cells, not pixels: the overlay JPEGs are large tissue thumbnails with a colour-coded artefact mask (17094 × 7840 for one inspected at full resolution), carrying no slide-label photograph, no barcode and no printed text. That distinction matters because the third instance under Known gap 8 is precisely the opposite case — legible labels inside a JPEG, where no content scanner helps.

What makes it worth naming anyway is that it is the first such instance in an institutional (MemorialPath) rather than personal repository, and that it is the cleanest argument for the shared-.gitignore conclusion already drawn on the index: one line is doing all the work here. Deleting files does not undo history. See De-identification.

Corrected 2026-07-31, same day, before this page was finished. An earlier draft of this paragraph described case-slide-number.xlsx as a case-number-to-slide-number crosswalk and therefore a re-identification key. That was wrong, and inferred from the filename rather than read. The file is a monthly aggregate table — Year-Month, Cases, Biopsy, Smear, ScannedSlides, ScannedCases, DeletedCases, Macroscopy, 58 rows spanning 2021-01 to 2025-09 — with no individual case or slide number in it. It contains no identifiers at all and is not a governance problem. It is, however, a genuinely useful five-year record of laboratory and scanning volume; see Laboratory Workload Measurement.

Related: GrandQC-QuPath — the wrapper that installed these models locally, and the page whose central open question this study set out to answer. Related: WSI Quality Control — the method this measures; this is the first evidence in the wiki of how a published QC model behaves on Memorial slides. Related: Interobserver Agreement — the statistical frame this study needs and does not use; κ = 0.10 is the number the pipeline never computed. Related: Class Imbalance and Accuracy — why the reported 0.635 accuracy is weaker than the majority-class baseline it is never compared against. Related: Coverage Bias — the 50% match rate, and why the unmatched half is not safely ignorable. Related: Stain Quality — the other place in this estate where synthetic numbers are rendered into a real-looking artefact; together they make it a pattern. Related: De-identification — the committed accession identifiers and the crosswalk file. Related: Laboratory Workload Measurement — the technician form is a per-slide record of preparation faults and their causes, which is workload data as much as quality data. Related: meddecide — the group’s own jamovi module, and the natural home for the weighted-κ analysis the .omv file implies was started.

Derived from: repository read 2026-07-31 at commit 26ae972 (WIP, 2025-12-09), one untracked file in the tree — README.md, FOLDER_STRUCTURE.md, analiz_raporu_TR.md, Abstract_ 0797.pdf, analysis/scripts/pipeline/04_validation_summary.py, technician_artifact_analysis.py, 01_WSI_inference_OPENSLIDE_QC/main.py and wsi_tis_detect.py for the model configuration; statistics recomputed independently from technician_vs_GrandQC.csv, technician_vs_grandqc_crosstable.csv, data/manual/manual-control.csv, analysis/results/current/02_comparison_results.csv, 03_issue_categories.csv, 04_final_summary.csv and analysis_report.md; git ls-files, git log and gh repo view for tracked contents, currency and visibility.

Every claim above about identifiers and about the agreement statistics was re-derived by a second independent pass whose instruction was to refute it (AGENTS.md §8 / skill §5). That pass corrected one claim of mine — the case-slide-number.xlsx description, see above — and contributed the pseudoreplication, κ-fragility and mis-binning findings, none of which were visible on a first read.