De-identification Release Gating

Turning de-identification from a step that runs into an assertion that must pass — the gate, the artifact it emits, and why re-measuring the released bytes is the part everyone skips.

Turning de-identification from a step that runs into an assertion that must pass — the gate, the artifact it emits, and why re-measuring the released bytes is the part everyone skips.

What it is

De-identification describes what has to be removed. This page is about the separate question of how you know it was, which turns out to be a different engineering problem with a different answer.

The distinction is between a step and a gate. A step is code that runs and then control moves on. A gate is code that runs and then something downstream refuses to proceed unless it passed. Almost everything in this estate is a step.

Four things make a gate, and the estate’s failures are each a missing one.

1. It fails closed. On error, nothing is released. Pathology Atlas Pipeline is the counter-example: its anonymisation is on by default and fails open — on failure it warns and publishes the original slide, label image and all. A warning is not a gate, because the default path on failure is still publication.

2. It returns a status something else reads. A human-readable warning in a log is not a gate; an exit code that a CI job or a pre-commit hook branches on is. This is the cheapest part and the most commonly missing.

3. It emits an artifact that cannot be quietly edited. The record of what was checked has to survive being passed around, and has to make tampering visible. Content-addressed digests over the claims, not a prose summary.

4. It re-measures the thing being released, not an earlier version of it. This is the one that gets skipped, and it is the one that matters. Measuring before the last transformation and then carrying the number forward looks identical to measuring the output, right up until the last transformation is where the bug is.

Why it matters for my work

Three de-identification failures are recorded in this wiki, and not one of them is a tooling failure. Each is a missing assertion:

  • Pathology Atlas Pipeline publishes the original slide when anonymisation fails, and its test suite covers every path except the anonymisation path. Missing: fail-closed, and a test.
  • PembeBobrek pseudonymised its images correctly and committed the accession-to-pseudonym key beside them, in the repository’s initial commit. Missing: a check at commit time.
  • QuPath Atölye commits a .qpproj naming one slide in the shape of a hospital protocol number, where three siblings in the same file are pseudonymised. Missing: the same check. Re-read on 2026-07-30, four days after the flag: unchanged. Reading found it; nothing enforces it, so nothing changed.

The generalisable point, and the reason this deserves a page rather than another bullet on De-identification: choosing a better anonymiser fixes none of these. They would all have happened with the best tool in the shortlist. The estate has been evaluating locks while the question was whether anyone checks the door.

The fourth case is the useful one, because it has not happened yet

The three above are post-hoc. QuPath Atölye’s OSF data package (DOI 10.17605/OSF.IO/V7MJQ) is the same pattern caught before it fires, and it is worth reading as the template for what a gate would actually be for.

That package ships four slides and its README asserts, twice and in bold, that every slide is anonymised and carries no patient identifier. The assertion is currently true. The four shipped slides are cleanly named; the protocol-numbered slide in the same hands-on project is not among them.

But it is true by accident of scope. The README states the HER2 / membrane slide is deferred to a later version — and the HER2 and ER slides sit in the same project directory as the protocol-numbered one. So what is keeping the identifier out of a DOI-archived public package is which teaching modules happened to ship, not a screening step. The next release is the event that would draw from that directory again.

Two things generalise from this:

An assertion in a README is the artifact of rule 3 without any of rules 1, 2 or 4. It records a claim, it is human-readable, and nothing recomputes it at the moment of release. It is the weakest possible form of the right idea — and it is more dangerous than silence, because a downstream reader now has a written guarantee to rely on.

The gate belongs at the release boundary, not only at the commit boundary. The commit-time check discussed above would not have helped here: the identifier is already committed and the containment argument is about what gets exported. A release gate asks a different question — “does every file in this package satisfy the claim the package makes about itself?” — and it has a natural trigger, which the commit hook has to synthesise.

Two design rules fall out of the specific failures.

Match on schema, not on values. The PembeBobrek key passed a regular-expression sweep for identifier-shaped strings cleanly — that accession format is 7–10 characters, contains letters, and carries no run of six digits. What caught it was reading the column name. So a pre-commit check should match on column names — accession, erişim, protokol, MRN — and on filenames, not on cell contents.

Extract before you redact, because the order is forced. QuPath Atölye Extension’s redaction list includes date and time but not MPP or AppMag. ScanTools exists to pull scan dates out. Run redaction first and the timestamps Scanner Log Analysis needs are gone permanently. The gate should therefore assert two things, not one: that the identifiers are gone, and that the metadata worth keeping was captured first.

A worked reference design, and its limits

OpenMed is the first tool reviewed here that implements the whole shape, and reading it is worthwhile even though the tool itself is not recommended for adoption. Four mechanisms are directly copyable:

  • Every column must be explicitly classified. Leaving one unlabelled is a hard error that names the offending columns. This is rule “match on schema” turned into a mandatory step rather than a convention — and it is the single best idea in the package.
  • No default threshold where a default would be a policy choice. Its target_k is a required argument, on the stated grounds that the tool should not pick a regulatory threshold on the caller’s behalf. Contrast its own target_l and target_t, which do default — to values that can never fail, silently. A default that cannot fail is worse than no default, because it looks like a control.
  • A tamper-evident, PHI-safe evidence artifact whose mandatory disclaimers cannot be edited out and which contains no raw values, aggregate counts only.
  • A gate command with a non-zero exit code, which is precisely the missing piece in rule 2 above.

And one instructive design decision beyond the mechanics: the generated artifact reserves the expert’s conclusion as an un-fillable placeholder, and a separate signing API exists for a named person to state and sign that conclusion themselves. The determination and the evidence for it are different objects with different authors. That separation is worth imitating in any local SOP, because it makes “who decided this” a field rather than an assumption.

The limits are as instructive as the design. Even there, rule 4 is not met: its validate_released_output copies the earlier claim forward rather than recomputing the metric on the released rows, and no test recomputes it either. So the strongest available reference implementation still asserts “a measurement taken elsewhere was digest-bound to these bytes” rather than “I measured these bytes”. If the most careful tool in the space skips the re-measurement, assume any local implementation will too unless a test forbids it.

The other limit is scope: that gate covers tabular releases only. It cannot assert a slide label was removed, that a TIFF ImageDescription was redacted, or that a mapping file never entered a commit — which are the estate’s three actual failures. The pattern transfers; the tool does not.

How it connects

De-identification — the parent question of what to remove; this page is the how do you know half, and the two should be read together.

Pathology Atlas Pipeline — the fail-open case, and the clearest argument for rule 1.

PembeBobrek — the committed-key case, and the reason rule “match on schema” is stated separately from rule 4.

OpenMed — the reference design for the artifact and the exit code, evaluated and not adopted; the pattern is the deliverable, not the dependency.

DZI Web Publication — the same principle applied to one specific pipeline, where a published pyramid makes a mistake expensive to retract; that page argues anonymisation must be a gate rather than a step for exactly the reasons generalised here.

ScanTools — supplies the extract-before-redact half, and its hashwsi is a usable verifier because a pyramid-level hash survives label redaction.

Open questions

  • Nothing here is built yet. The concrete first increment is small and known: a pre-commit hook matching on column names and filenames, plus one test asserting that Pathology Atlas Pipeline’s anonymisation path fails closed. Both were identified before this page existed; what was missing was the argument that they are the same piece of work.
  • Which repositories should the commit-time check cover? Two instances are known (PembeBobrek, QuPath Atölye) and both were found by reading, not by scanning. A sweep across the estate would establish whether the count is two or twenty. [unverified] One data point on why the scan matters more than the reading: the QuPath Atölye instance was still present four days after being written up, so discovery-by-reading demonstrably does not close these on its own.
  • The first gate to build may be the release one, not the commit one. QuPath Atölye’s next OSF release has a concrete trigger and a written claim to check against, which is a better first target than a repo-wide commit hook — it is one package, one assertion, and a natural place to hang a non-zero exit code.
  • What does the gate assert for a slide? For a table the assertion is a recomputed k. For a WSI the natural analogue is “no label or macro directory is present, and the pyramid hash matches the pre-redaction hash” — both of which ScanTools can already compute. Whether that pair is sufficient has not been checked against Bisson 2023’s five-level framework, which De-identification records as cited but unread. [unverified]
  • Should the local SOP adopt the split between evidence and determination? The v0.8 anonymisation SOP is a draft with no recorded sign-off field at all. Adding one is nearly free and would make the reviewer identifiable after the fact.