Pathology-CoT

Companion code for a Nature BME paper that turns pathologists’ viewer logs into agent training data — worth reading for its two behaviour-discretisation heuristics and its released per-slide results file, not adoptable as software: no licence, a partial dataset, and no end-to-end path without your own viewer integration.

Purpose

Verdict: do not adopt as a tool; read it as a method reference, and keep the results file.

The repository accompanies Wang et al., Nat Biomed Eng 2026 (doi:10.1038/s41551-026-01739-y; note at sources/papers/wang-2026-pathology-cot.md). It is not a tool anyone can install and point at slides. What it contains that is worth having:

Worth taking	Where
The <inspect> / <peek> discretisation thresholds, as running code	ai-session-recorder/NucleiioLog.py, VLMAction.py
The review-over-authoring rationale interface (accept / edit / reject)	pathology-cot/GUI_validate/
Per-slide external-validation predictions for four models	pathology-o3/external_validation_results/results.csv

That last file is the most valuable thing here and the rarest. It is what made the paper’s external-validation numbers recomputable, and recomputing them found a real discrepancy — see the paper note. Almost nothing else in sources/papers/ can be audited that way.

Why not adopt

No licence, anywhere. No LICENSE file, nothing in the README, no file headers, no setup.py or pyproject.toml to declare one. Default copyright therefore applies: all rights reserved, no permission to copy, modify or reuse. The paper describes an “open-source methodology”, which is the intent, but intent is not a grant. Reusing any of this code in group work needs the authors to add a licence. [unverified] only in the sense that nobody has asked them — the absence itself is verified. The person who can resolve it is the corresponding author (Zhi Huang, UPenn), and it is a one-line fix for them.

It is bound to one viewer. NucleiioLog.py parses logs from nuclei.io. Nothing here reads QuPath, Sectra, Philips or Aperio ImageScope logs. The paper is candid about this — it names heterogeneous viewer log formats as the reason deployment to a new site “is not yet fully plug-and-play”. Adapting it means writing a new parser against the group’s own viewer, at which point what transfers is the heuristics, not the code.

No end-to-end path. The reasoning agent (pathology-o3/think_ln_classify.py) expects ROI crops already extracted, and calls out to commercial VLM APIs. There is no training script for the behaviour predictor, no data loader, and no glue between the three directories.

Data used

The repository ships one dataset subset, and this is where its documentation and its contents diverge.

What the README claims. The dataset section advertises the full study resource: “10.6 hours”, “8 pathologists”, “5,222 conversation rounds”. The structure diagram lists part1/ alone.

What is actually in git, counted at 5ae3d04: 155 session folders, 2,533 JPEG images and 155 conversation.json files. Against the paper’s reported 921 GI sessions, that is roughly a sixth. Case folders are named <stem>_<n>_<n>_<uuid>; whether the leading stem is a study ID or an institutional accession number is not stated anywhere, and it is worth knowing before anyone mirrors this data. [unverified] — only the authors can say.

What was meant to hold the rest. The repository’s second commit (5ae3d04, “remove dataset link”) deleted two README lines pointing at huggingface.co/datasets/XXXX — an unfilled placeholder marked “Coming Soon”. So the full release was planned, the destination was never created, and the link was withdrawn rather than filled. Nine months later that is still the state, and it is what the open issues are about: “Question about Pathology-CoT dataset” (Nov 2025), “Dataset not found in repo” (Jul 2026), plus a question about the YOLO model (Apr 2026). None has a reply.

The released conversations are colorectal, not skin. Every one of the 155 opens with “This is a HE WSI of a CRC case”. The paper’s Data availability statement says what is on GitHub is “the Pathology-CoT Chain-of-Thought dataset for the skin cancer cohort”. Those cannot both be right. Recording it as a conflict rather than guessing which is the error.

The ROI coordinates are not in the release. Every one of the 741 places a conversation says “I want to zoom into” is followed by the literal string [coordinates not found] — 741 of 741. The regions themselves survive as pre-cropped JPEGs and as boxes drawn onto thumbnail overlays, so the data is usable for training a reasoning module. It is not usable for training a navigation module, which is the paper’s headline contribution, because there are no numeric boxes to regress against.

Image paths point at the first author’s laptop. All 1,568 image references in the JSON are absolute paths under /Users/<author>/..., so nothing loads without rewriting them first. A per-session directory name in that path also identifies which pathologist recorded the session, which the paper does not say is part of the release.

Model weights are available, just not in git. behavior_predictor.pt is gitignored with the comment “too large for GitHub” — accurately, at ~130 MB against a 100 MB limit — and the README links it on Google Drive with three download routes. Reachability not tested. [unverified]

Methods

Read at source level. Three things are worth extracting.

The discretisation thresholds are concrete and portable. A viewport held >1 s, or panned continuously >2 s, becomes <inspect>; a rapid zoom to native resolution becomes <peek> on the central 1,024×1,024 region. Actions are then merged at IoU >0.8, pruned toward the most specific view, and binned to the nearest standard objective. None of that depends on nuclei.io — it depends on having viewport coordinates, a zoom factor and timestamps, which is exactly the recording contract Reading-Behaviour Capture already specifies.

Failure to navigate is silent. think_ln_classify.py raises FileNotFoundError when no ROI was proposed for a slide; the caller catches it and returns None. Parse failures elsewhere default to NEGATIVE. So a slide the navigator skipped produces an empty cell rather than a prediction or an error, and 73 of 321 slides in the released external-validation file are exactly that. This is the mechanism behind the metric problem described on Model Abstention, and it is a design pattern worth avoiding rather than copying.

Secrets were handled properly. The first commit is titled “Initial commit with API keys removed”, and no hard-coded key survives in the tree. Worth recording because it is the opposite of the usual finding.

Current state / open questions

• Would the group’s own recordings feed this pipeline? The focus/ package in QuPath Patoloji Atlası Extension already samples image-space coordinates with the per-tick downsample factor and preserves order — the same contract nuclei.io satisfies. Running the paper’s thresholds over existing focus/ logs is a day’s work and needs no new data collection. This is the single most actionable thing this evaluation produced.
• Is the licence resolvable? One email to the corresponding author. Until then no code from here can enter group work, however useful the heuristics are to reimplement independently.
• Does the arXiv preprint (2510.04587) ship a fuller dataset than the GitHub release? Not checked. [unverified]
• Is the Google Drive weights link still live, and does the checkpoint match the fold-0 model used for the external validation? Neither tested. [unverified]

How it connects

Reading-Behaviour Capture — the two-property recording contract this pipeline consumes; this repo is the first concrete demonstration of what a compliant recording is for beyond measuring readers.

QuPath Patoloji Atlası Extension — the group’s own compliant instrument, and the reason the question above is worth asking rather than theoretical.

Model Abstention — the silent-None path in the reasoning agent is the worked example that page is built on.

Agentic Slide Navigation — the concept this repository is the reference implementation of, warts included.

MouseTracks — the contrast case: maintained, licensed, installable, and structurally unable to produce data this pipeline could use. Adoptability and usefulness are independent axes, and these two repos sit at opposite corners.

Derived from: repository cloned 2026-07-27 at 5ae3d04 (unshallowed; two commits total), README read in full, NucleiioLog.py, VLMAction.py, generate_conversation.py, think_ln_classify.py, .gitignore and all 155 conversation.json files read or counted programmatically; external_validation_results/results.csv recomputed in full; repository metadata and issue list read from github.com on 2026-07-27; paper read in full as sources/papers/wang-2026-pathology-cot.md.