LLM Report-Quality Checking

Using large language models to score pathology reports against CAP/ICCR standards, find gaps between narrative and synoptic content, and rewrite reports for different audiences.

Using large language models to score pathology reports against CAP/ICCR standards, find gaps between narrative and synoptic content, and rewrite reports for different audiences.

What it is

A layer of packaged LLM “skills” sitting above the extraction pipeline, each doing one job.

  • Compliance scoring — pathology-report-checker-skill checks a report against CAP/ICCR required data elements and reports which are missing.
  • Gap analysis — pathology-report-colon-QA compares what a narrative colon-resection report contains against what a synoptic report would have required.
  • Rewriting and translation — pathology-skills-collection produces summaries, translations, and lay-language versions.
  • Research support — claude-scientific-skills covers literature search, citation, and figure generation.
  • Orchestration — n8n-pathology glues the above into workflows.

Three properties of LLM-based checking need stating plainly, because they differ from rule-based checking. It is non-deterministic: the same report can score differently across runs, so any reported score needs a stated model version and ideally repeated runs. It fails silently and fluently: an incorrect finding reads exactly as confidently as a correct one, which is materially worse than a rule-based system erroring out. And it is sensitive to prompt and format, so the prompt is part of the method and belongs under version control.

Where a rule is crisp — “is a margin distance stated” — a regex is more auditable and should be preferred. LLMs earn their place on the fuzzy tasks: judging whether a described finding actually satisfies an element, handling arbitrary phrasing, and working across languages.

Why it matters for my work

This is the group’s active bet on report quality, and the same machinery underpins the bilingual work — the corpus is Turkish and English, which is where LLM flexibility genuinely beats regex. The checker is also close in design to the group’s own published pathology-skills, so improvements flow both ways.

How it connects

Report Text Extraction — supplies the cleaned text this layer operates on.

Synoptic Reporting — defines the standard being scored against, including the edition question.

Pathology Report Checker Skills — the project page with the repo inventory.

Interobserver Agreement — the right framework for validating a checker: treat it as another rater and measure agreement with expert review, using meddecide.

LLM Adaptation Techniques — names the rungs this layer is built on. The open questions below about model versioning and prompt version control are hard to answer while the technique in use is unnamed, so that page is the vocabulary this one needs.

Model Abstention — an LLM that returns nothing parseable drops that report out of the evaluation entirely, and the reports which break a parser are exactly the unusual ones this checker exists to catch.

Open questions

  • No documented validation against human reviewers. A compliance score is a measurement, and an unvalidated measurement of report quality can be worse than none. This is the highest-value next step and the tooling for it already exists.
  • Is the model version recorded in checker output? Without it, scores are not comparable over time. [unverified]
  • Are prompts version-controlled alongside the code? [unverified]
  • Is performance measured separately for Turkish and English reports?
  • Would catalogue grounding suppress invented element names? askLLM hit the same failure class — a model producing structured references that sound right and do not exist — and fixed it by sending the real enumerable list and demanding verbatim quotation, taking fabrication to zero across 18 extracted references. CAP/ICCR element names are the same kind of vocabulary. The transferable part is not the prompt but the mechanical check it makes possible: every element name the checker emits can be matched against the protocol list automatically, which turns “does it hallucinate” from a judgement into a count. This page records no measurement of any kind; that would be the cheapest one to start with.