croma

A small, tested Python package that scores how much a pathology encoder’s embedding space is organised by biology rather than by acquisition centre — three metrics over any embeddings plus a confounder column, no model loading, no image reading — and therefore the first tool in this index that could measure scanner-robustness on the group’s own cached features rather than on somebody else’s benchmark.

Purpose

Verdict: not adopted yet, and the strongest adoption candidate evaluated here since VALIS — because the experiment it enables needs nothing new. It takes an (N, D) embedding array and a manifest CSV with a biology column, a confounder column and a grouping column (slide or case), and returns per-sample margins. Foundation Model Cache holds the encoders; PinkKidney has 356 slides with three scanners recorded per slide. That is the input.

The §6 question — what does it do that the stack does not — has a clean answer. Nothing indexed here measures representation-level robustness at all. The nearest thing is Feature-Space Distance, a cohort-level distance between two sets of tiles; croma is per-sample and typed: for each tile it asks whether the nearest same-biology tiles from another centre are closer than the nearest different-biology tiles from the same centre. That is the question Scanner and Stain Variability poses in prose and no local project has ever put a number on.

Companion to Grisi 2026 — CRoMa, a Distributional Robustness Margin for Pathology Foundation Models; reimplements the Robustness Index from Kömen 2026 — PathoROB: Towards Robust Foundation Models for Digital Pathology alongside it.

Data used

None shipped as images. results/ holds the paper’s computed outputs — per-cohort CSVs, a distributions.json of 200-bin per-sample margin histograms, nIPD tables — with a PROVENANCE.json recording the package version, protocol, m, tail fraction, τ policy and the run path for every cohort. Evaluation manifests for the PathoROB cohorts are released so the paper’s numbers can be regenerated from anyone’s embeddings of the same tiles; the tiles themselves are PathoROB’s.

Methods

Read at source level on 2026-08-30.

Three metrics, one neighbour typing. src/croma/metrics/ implements RI (fixed-k counts), MaRI (the same, distance-weighted with a per-model temperature τ = median typed distance) and CRoMa (the per-sample signed ratio of mean distances to the m nearest same-biology/other-centre and other-biology/same-centre neighbours; default m = 5, the source comment explaining that m = 1 “is maximally sensitive to one outlier neighbour”). Features are L2-normalised and compared by cosine distance; neighbours from the same physical unit are excluded. The tail summaries — F(0) and the lower-tail mean at α = 0.10 — live in tail.py; bootstrap intervals in bootstrap.py. downstream/ carries PathoROB’s confounder-biased linear-probe protocol and both reductions, APD and the paper’s nIPD.

Dependencies are the point. numpy, pandas, scikit-learn, tqdm — nothing else for the core; croma[repro] adds the paper-reproduction utilities. It never loads a model. That is what makes it runnable against a cache of .pt files without standing up any encoder’s environment, which is the practical obstacle to most of what Encoder Benchmarking describes.

Seventy test files at tests/, a CHANGELOG.md kept in Keep-a-Changelog form, a CITATION.cff, and a documentation site with an interactive distribution explorer — an unusually complete packaging for a single-author research tool.

The repo has moved past the paper. The README’s results table ranks 25 pathology encoders (plus the DINOv2 control), against the paper’s 20; the additions include Mascaret, which tops the mean rank (1.7) with a Camelyon tail of −0.02, and three RudolfV-2 variants. So the site, not the preprint, is the current leaderboard, and anyone quoting “the most robust encoder” should say which one they read.

One consistency slip, recorded not investigated. pyproject.toml says 1.0.0 and a 1.0.0 tag exists, but CITATION.cff still says 0.3.0 — the changelog’s own unreleased section notes it had previously been stuck at 0.1.0 — and results/PROVENANCE.json records the published numbers as computed with croma_version 0.3.0 on 2026-08-13. Harmless, but the provenance file is exactly the artefact that should say which version produced the tables.

Current state / open questions

• The local experiment, fully specified and unrun. Embed PinkKidney’s tiles with two or three encoders already on the cache; manifest columns: biology = tumour class (chromophobe versus oncocytoma), confounder = scanner vendor from cohort_metadata.csv, group = slide. Run CRoMa.compute. Two caveats decide whether the result means anything: the cohort must be balanced across biology × scanner or the typed neighbour pools are asymmetric — PathoROB explains why — and PinkKidney’s scanner is perfectly collinear with magnification, so the confounder measured would be “acquisition platform,” not scanner alone. Both are countable before running anything.
• Which confounder? The construction takes any labelled factor. Centre bundles scanner, stain and preparation; the group could label them separately and would then be measuring something the papers did not.
• Adoption cost is near zero; interpretation cost is not. The papers are explicit that the per-model ranking mostly recapitulates centre decodability and that the single-model diagnostic use is undemonstrated. What the group would get first is a distribution and a tail, not a verdict.
• Licence recorded as declared; the NC licences on two of the benchmark’s source datasets belong to PathoROB, not to this package.

How it connects

Grisi 2026 — CRoMa, a Distributional Robustness Margin for Pathology Foundation Models — the paper this implements, and where the metric’s construction and limits are appraised.

Kömen 2026 — PathoROB: Towards Robust Foundation Models for Digital Pathology — the Robustness Index this package reimplements as its baseline, and the confounder-biased probe protocol it ports.

PathoROB — the benchmark cohorts the released manifests refer to.

Feature-Space Distance — the cohort-level cousin already on this wiki; the two answer different questions (is this cohort shifted? / is this encoder reading the centre?) from the same cached features.

Encoder Benchmarking — the robustness axis that page’s four protocol choices do not include.

Foundation Model Cache — the encoders; PinkKidney — the cohort. Together they are the input this tool has been waiting for.

Scanner and Stain Variability — the phenomenon; this is the instrument.

Derived from: full clone of clemsgrs/croma at 3f58d5e read 2026-08-30 — README.md, pyproject.toml, CITATION.cff, CHANGELOG.md, LICENSE, src/croma/metrics/*.py (especially croma.py), src/croma/downstream/, tests/ (counted, not run), results/PROVENANCE.json and the results tables; the documentation site fetched the same day. Nothing was installed or executed.