LLM Report-Quality Checking

Using large language models to score pathology reports against CAP/ICCR standards, find gaps between narrative and synoptic content, and rewrite reports for different audiences.

Using large language models to score pathology reports against CAP/ICCR standards, find gaps between narrative and synoptic content, and rewrite reports for different audiences.

What it is

A layer of packaged LLM “skills” sitting above the extraction pipeline, each doing one job.

  • Compliance scoring — pathology-report-checker-skill checks a report against CAP/ICCR required data elements and reports which are missing.
  • Gap analysis — pathology-report-colon-QA compares what a narrative colon-resection report contains against what a synoptic report would have required.
  • Rewriting and translation — pathology-skills-collection produces summaries, translations, and lay-language versions.
  • Research support — claude-scientific-skills covers literature search, citation, and figure generation.
  • Orchestration — n8n-pathology glues the above into workflows.

Three properties of LLM-based checking need stating plainly, because they differ from rule-based checking. It is non-deterministic: the same report can score differently across runs, so any reported score needs a stated model version and ideally repeated runs. It fails silently and fluently: an incorrect finding reads exactly as confidently as a correct one, which is materially worse than a rule-based system erroring out. And it is sensitive to prompt and format, so the prompt is part of the method and belongs under version control.

The first of those three has a published, one-line fix that this layer may not be using. Jin 2026 — Guidance on the Use of LLMs for Medical Research recommends temperature 0 as the default for medical work, precisely so that output is deterministic and reproducible, with higher temperature reserved for the single case of deliberately sampling diverse responses to ensemble them — and it pairs that with instructing the model to return JSON, so a score parses into fields rather than being read out of prose. Two settings, both free. The catch is where they can be set: temperature is available through an API or a local model and generally not through a web application, which is why that source recommends against web apps during development altogether. So “is the checker deterministic” is not a property of the model; it is a question about which interface the checker calls and what it passes.

Where a rule is crisp — “is a margin distance stated” — a regex is more auditable and should be preferred. LLMs earn their place on the fuzzy tasks: judging whether a described finding actually satisfies an element, handling arbitrary phrasing, and working across languages.

Why it matters for my work

This is the group’s active bet on report quality, and the same machinery underpins the bilingual work — the corpus is Turkish and English, which is where LLM flexibility genuinely beats regex. The checker is also close in design to the group’s own published pathology-skills, so improvements flow both ways.

How it connects

Report Text Extraction — supplies the cleaned text this layer operates on.

Synoptic Reporting — defines the standard being scored against, including the edition question.

Pathology Report Checker Skills — the project page with the repo inventory.

Interobserver Agreement — the right framework for validating a checker: treat it as another rater and measure agreement with expert review, using meddecide.

LLM Adaptation Techniques — names the rungs this layer is built on. The open questions below about model versioning and prompt version control are hard to answer while the technique in use is unnamed, so that page is the vocabulary this one needs.

Model Abstention — an LLM that returns nothing parseable drops that report out of the evaluation entirely, and the reports which break a parser are exactly the unusual ones this checker exists to catch.

Open questions

  • No documented validation against human reviewers. A compliance score is a measurement, and an unvalidated measurement of report quality can be worse than none. This is the highest-value next step and the tooling for it already exists. What that validation should look like is now specified rather than left to taste, though none of it has been run. Jin 2026 — Guidance on the Use of LLMs for Medical Research sets out a two-tier hierarchy: automatic evaluation on a benchmark is a screen only, and clinical evaluation on the real task is the gold standard, with randomised controlled trials named as its rigorous end. It also gives the set size — roughly 100 instances with inputs and expected outputs — and requires an independent test set for anything fine-tuned. Nothing in that is expensive; the missing ingredient is a hundred scored reports with an expert’s answer beside each, which is the same artefact LLM Adaptation Techniques and Report Text Extraction are both blocked on. This entry stays open: a published design for a measurement is not a measurement.
  • Is the model version recorded in checker output? Without it, scores are not comparable over time. [unverified] The reason this is not merely tidy has now been stated in print. Jin 2026 — Guidance on the Use of LLMs for Medical Research treats model replacement as a lifecycle event: the deprecation or replacement of a production model “may require re-implementation of prompts, re-validation of performance and re-certification of safety and compliance”. A checker whose output does not name its model cannot tell you which of its historical scores fall on which side of such an event, so the versioning question is upstream of every comparison this layer might ever make — and proprietary models are deprecated on their vendor’s schedule, not the group’s.
  • Are prompts version-controlled alongside the code? [unverified]
  • Is performance measured separately for Turkish and English reports?
  • Would catalogue grounding suppress invented element names? askLLM hit the same failure class — a model producing structured references that sound right and do not exist — and fixed it by sending the real enumerable list and demanding verbatim quotation, taking fabrication to zero across 18 extracted references. CAP/ICCR element names are the same kind of vocabulary. The transferable part is not the prompt but the mechanical check it makes possible: every element name the checker emits can be matched against the protocol list automatically, which turns “does it hallucinate” from a judgement into a count. This page records no measurement of any kind; that would be the cheapest one to start with.
  • A minimal test protocol now exists in the literature, and none of it has been run here. Clusmann 2026 — Safety and Security of Large Language Models in Healthcare (Nature Review) (Table 3) lists fourteen socio-technical safeguards for any clinical LLM, to be run before deployment and repeated. Four apply directly to a report checker and each is an afternoon’s work: ask it to score against a guideline that does not exist (it must admit ignorance); offer it a wrong element value and see whether it agrees (sycophancy — the review grades this encountered, not hypothetical); re-phrase the same report and check the scores agree; and bury a required element deep in a long report to test context degradation. Add the organisational one the review calls vigilance testing — inject a deliberately flawed checker output into a reviewer’s queue, with debrief, “analogous to simulated phishing” — and the checker’s human layer is tested too. The catalogue-grounding check above and this protocol are complementary: grounding suppresses invention, and the review is explicit that it does not suppress agreement — a grounded model will still say yes to a wrong diagnosis if trained to be agreeable.
  • A practitioner’s check list, from the other direction. The pathCast talk by Rajendra Singh (sources/notes/pathcast-2026-singh-what-ai-can-and-cannot-do.md, unpublished, cited by path) names the report errors a working dermatopathologist wants caught before sign-out: laterality mismatch between clinical history and diagnosis, site and gender inconsistencies, missing fields (a gross description; an IHC performed but never reported), and wrong or absent ICD/CPT codes — with the rule that the pathologist reviews, edits and signs, and the liability stays theirs. That is a target specification for this layer that is narrower and more concrete than “CAP/ICCR completeness,” and the laterality and IHC-done-but-unreported checks are cross-field consistency tests rather than element-presence tests, which the current skills may not do. [unverified] whether any of them does.