Pathology Atlas Pipeline

The tkinter GUI that turns an SVS file into a published, zoomable teaching case — anonymisation, DZI tiling, bilingual Quarto documentation, and a GitHub repo per slide.

Purpose

This is the machinery behind Patoloji Atlası. That page describes the atlas as a contribution and image-handling pipeline; this is the pipeline itself. It takes a whole slide image and produces a deployed GitHub Pages site with an OpenSeadragon viewer, in Turkish and English.

The architectural decision worth recording: one GitHub repository per case, under the pathologyatlas organisation, with a list.yaml catalogue in the atlas site repo acting as the index. update_repos.py and sync_repos.py exist because that choice makes a viewer change a fleet-wide operation rather than a single edit — both drive the GitHub API through the gh CLI without cloning.

Data used

Whole slide images (SVS), converted locally. No patient-level data lives in this repo. Scale figures, taken from the regression tests’ own comments about the production catalogue: 143 stain names and 71 case notes in list.yaml, of which 64 names are hyphenated and 25 notes are Turkish — both facts that turned into bugs, see below.

Methods

A five-tab GUI (atlas_pipeline.py, 4477 lines, class PathologyAtlasApp): Project Setup, Image Conversion, Metadata, Repository, Integration. Around it:

File Lines What it does
atlas_pipeline.py 4477 The GUI and the conversion pipeline
template_generator.py 2098 Emits .qmd, HTML viewers, README, CITATION.cff
update_repos.py 969 Batch-upgrades viewers across the org via gh; no cloning
sync_repos.py 633 Bidirectional sync between the org’s repos and list.yaml
anonymize-slide.py 572 Vendored label-stripping script — see the licence note
config_loader.py 165 YAML config that falls back to DEFAULTS rather than raising
test_pipeline.py 193 Regression guards, added 2026-07-25 — uncommitted

External dependencies are vips (for dzsave), the gh CLI, and PyYAML. wsianon is optional and the GUI disables its method when absent.

Anonymisation is on by default and offers two methods:

  • wsi-anon (EMPAIA), the default, labelled “comprehensive” in the GUI. Requires Python ≥ 3.10. Its behaviour was not inspected here — it is an external library, not vendored. [unverified]
  • anonymize-slide, labelled “simple label removal”, which is accurate: reading the source, it deletes the label directory in SVS, the macro image in NDPI, and ScanDataLayer_SlideBarcode in MRXS. It does not touch ImageDescription or any other metadata.

That last point is worth carrying to De-identification, which had recorded an open question about whether anonymisation strips scanner and resolution metadata. For this path it does not — only the label is removed. Whether the same holds for the EMPAIA library is unverified, and the research imagebank may use different tooling again, so this narrows the question rather than closing it.

Current state / open questions

Last commit 2026-04-13. A substantial hardening pass sits uncommitted in the working tree, dated 2026-07-25: the fail-closed fix below, test_pipeline.py, ruff.toml, a .github/ directory, and a requirements.txt cleanup.

The finding that matters: anonymisation fails open in the published version

Verified twice, by reading the working tree and then git show HEAD:atlas_pipeline.py independently.

In the working tree, if anonymize_svs_file() returns false the run aborts, deletes the temporary file, and refuses to convert — with a comment explaining that converting the original would publish the very label the operator asked to strip.

In HEAD — the version anyone cloning the repo gets — the same branch logs "Warning: Anonymization failed for <file>, using original file" and continues. source_file stays pointed at the original, and vips dzsave tiles it. Since anonymisation defaults to enabled, the operator has asked for it, been told it failed in a log line, and published the slide label anyway — and the atlas is a public, SEO-indexed site. The label image is exactly what carries the accession number and sometimes the patient name, per De-identification.

The fix already exists. It is simply not committed or pushed. That is the single highest-value action on this repo.

Licence: unresolved, and the owner is the one who can resolve it

Three readings coexist and they do not agree:

  • README.md says “MIT License - See LICENSE file for details”. There is no LICENSE file at the repo root.
  • anonymize-slide.py is vendored under GPL v2 only (Carnegie Mellon 2007–2013, Google, Benjamin Gilbert, and a 2021 modification by Toby C. Cornish — the anonymize-slide-python3 lineage), version string 1.2.0.
  • manual_template/LICENSE is MIT, “Copyright (c) 2022 Pathology Atlas”.

Recorded as a conflict rather than adjudicated, per this repo’s rule for ambiguous facts a decision would rest on. GPLv2 is copyleft and the interaction with an MIT declaration is a real question, not a formality. [unverified] — the person who can settle it is the repository owner, with an institutional office if the atlas is ever distributed more formally.

README documents features that no longer exist

The code is what runs, so these are README defects, not code defects:

  • “Automatic Screenshot Generation” and “QR Code Generation” are listed as features, with Selenium and ChromeDriver as prerequisites. No such Python functions exist and neither library is imported. requirements.txt records that they were deleted on 2026-04-12; QR codes are now emitted as R inside the generated .qmd (qrcode::qr_code), produced downstream by Quarto. The capability moved, the README did not.
  • Three different repository URLs appear: a yourusername/ placeholder in the clone instructions, pathologyatlas/pipeline in the citation block, and the actual remote sbalci/pathology_atlas_pipeline.
  • “Status: Production Ready ✅ — All major features implemented and tested” predates the only tests in the repo, which are uncommitted.
  • Anonymisation is not mentioned anywhere in the README, despite being on by default and being the step with the highest consequence.

The failure mode this project actually has

test_pipeline.py is the most informative file in the repo, because each test pins a defect that shipped broken output silently — no exception, no warning, and in one case was “pushed to a live repo as success”:

  • .format()-escaped braces emitted ```{r} instead of ```{r}, so R chunks rendered as literal text and the bilingual {asis, echo = (language == "TR")} mechanism died quietly.
  • Stain names were written as bare JavaScript object keys; 64 of 143 production names are hyphenated, so CK7-p40: { was a syntax error and the viewer rendered a blank page.
  • Url/Width/Height rewrites matched single quotes while the GUI emitted double quotes, so the substitution no-opped and viewers shipped the template’s placeholder dimensions.
  • Turkish text broke re.sub replacements, where a \uXXXX escape raises “bad escape .

There is also a redact_url() guard ensuring credentials never reach the log file, which is the right instinct — the log is gitignored but sits on disk at 4.4 MB.

Open questions

  • Commit and push the fail-closed fix. Until then the published repo has the weaker behaviour.
  • Resolve the licence question, or add the LICENSE file the README already promises.
  • Does the EMPAIA wsianon path strip metadata beyond the label? Not inspected. [unverified]
  • Is this pipeline’s anonymisation the same tooling the research imagebank uses, or a separate teaching-side path? De-identification describes an SOP at v0.8 draft; the relationship between the two is not recorded anywhere. [unverified]
  • test_pipeline.py covers template generation and URL redaction but nothing in the anonymisation path — the one place where a silent failure has consequences beyond a broken page.

Related: Patoloji Atlası — the atlas this pipeline publishes into; that page describes the editorial contract, this one the machinery. DZI Web Publication — the method page this project’s conversion step justified. De-identification — the governance rules the anonymisation step implements, and whose open question about tooling this partly answers. Whole Slide Imaging — the source format and the metadata that survives, or does not.

Derived from: repository working tree read 2026-07-26 — atlas_pipeline.py, anonymize-slide.py, template_generator.py, update_repos.py, sync_repos.py, config_loader.py, test_pipeline.py, requirements.txt, README.md, manual_template/LICENSE, plus git show HEAD:atlas_pipeline.py and git log/git status at commit 50ac51d.