kaiko-qupath

A QuPath front end for the Kaiko Midnight foundation model whose documented Quick Start displays a fabricated 92.5% accuracy and the words “approved for clinical use” without training anything — the working path underneath is real but reports training accuracy on its own training data, and trains and predicts at magnifications 4× apart.

Purpose

Turn Midnight tile embeddings into a QuPath-driven classifier a pathologist can run on their own annotations. The idea is sound and matches Foundation Models in Pathology: freeze a strong encoder, train a small head, need far fewer labelled cases.

The reading on 2026-07-26 found three defects that have to be fixed before any number this produces means anything. All were confirmed twice, the second time by a reader instructed to refute them. Private repository, last pushed 2025-10-25, no licence file.

Data used

Slide tiles exported from a QuPath project into tiles_overlap_0/, one folder per annotation class. The encoder is kaiko-ai/midnight — Midnight-12k, a DINOv2-giant pretrained on TCGA, MIT-licensed weights — loaded from a local model.safetensors. No cohort is defined in the repository; it operates on whatever project is open.

Methods

Groovy for UI, Python for compute, talking through files and a blocking subprocess.

The wizard reports results for work it never did. scripts/clinical_workflow_wizard.groovy is step 2 of the README’s Quick Start, the “Start Here” line in CLAUDE.md, and two items in the main menu — the primary documented route for a pathologist. Its startModelTraining() contains no ProcessBuilder and no Python call at all; it is four updateProgress() calls with Thread.sleep(500) between them, ending in “Training complete!”. Its showValidationResults() then prints a hardcoded block:

Overall Accuracy: 92.5% · Cross-Validation Score: 89.3% ± 3.2% · Recommendation: Model approved for clinical use with pathologist supervision.

These strings are literals in the source. They display unconditionally, and the function sets workflowState.validationComplete = true whether or not a model exists. Nothing checks workflowState.modelTrained first. This is the most serious thing found anywhere in this batch: not a broken feature, but a user-facing claim of clinical validity generated by Thread.sleep.

Train and predict see different magnifications. scripts/export_project_tiles.groovy sets downsample = 4 and tileSize = 256, so each training tile is a 1024×1024 slide region squeezed into 256 px — the script prints this itself. Prediction (predict_midnight.py) reads slide.read_region((x, y), 0, (224, 224)): level 0, no downsample. No transform on either side compensates; the prediction-side Resize(224) is a no-op on input already 224 px. The classifier is therefore applied at roughly 4× the magnification it was trained at. This is the fixed-divisor error Patch Extraction already records from PinkKidney, in a new form — there the divisor was wrong across slides, here it is inconsistent between the two halves of one pipeline.

The only working training path has no validation. midnight_external.py fits RandomForestClassifier(n_estimators=100) on X, y and then reports np.mean(clf.predict(X) == y) — accuracy on the data it just fitted. sklearn.model_selection is not imported anywhere in the file, so there is no held-out split or cross-validation that was merely skipped. The repository’s own parent TODO.md asks “how can I be sure that the model is not overfitting, memorising — give user feedback on this”. It is the right question and this is the answer to it.

The path that does have cross-validation cannot run. midnight_clinical_enhanced.py — what the menu’s “Train AI Model” button actually invokes — implements proper StratifiedKFold(5) with confidence intervals, but its _prepare_training_data() is marked # For now, returning placeholder, ignores its annotations_path argument entirely, and requires a classes.json shaped {'labels': [...], 'class_names': [...]}. Every other script in the repository writes that file as a flat list. So the button fails whether the file is missing or present.

And a third stack trains on noise. foundation_model_processor.py, which CLAUDE.md calls the “NEW: Primary Interface”, builds its inputs as dummy_image = np.random.randint(0, 255, input_size, dtype=np.uint8) with the comment “for now, simulate with dummy data”. Nothing in scripts/ invokes it.

Prediction runs only on the currently open image, never the project — an acknowledged limitation in the parent TODO.md, and the code matches. AGENTS.md describes a Gradle build and an agent-class architecture that does not exist in the tree.

Current state / open questions

Paused. The encoder choice is good and the tile-export step is reusable; nothing downstream of it should be trusted. Fix order: delete or gate the wizard’s fabricated results first, because it is the only defect that can mislead a person rather than a model; then make export and prediction agree on magnification; then replace resubstitution accuracy with slide-level cross-validation.

• Has any number from this been quoted anywhere — a talk, an abstract, a grant? The 92.5% is plausible-looking and the dialog recommends clinical use. [unverified] and worth answering before anything else here.
• Whether the 4× mismatch would have shown up as poor accuracy or as deceptively good accuracy is not obvious and is worth thinking through rather than assuming.
• No licence, on a repository that bundles nothing but is not shareable without one — see the pattern on Digital Pathology Drafts.

Related: Foundation Models in Pathology — the frozen-encoder pattern this implements, and where the honest evaluation questions live. Related: Patch Extraction — the magnification-versus-downsample error, here appearing within one pipeline rather than across a cohort. Related: spider-qupath — the sibling repository taking the opposite approach, zero-shot inference with no training step at all. Related: QuPath Atölye Extension — the packaged extension that ships a Kaiko wizard, and where this prototype was heading. Related: Digital Pathology Drafts — the umbrella repository this sits in as a submodule.

Derived from: repository source read 2026-07-26 — scripts/export_project_tiles.groovy, scripts/clinical_workflow_wizard.groovy, scripts/clinical_midnight_menu.groovy, scripts/predict_annotations_external.groovy, predict_midnight.py, midnight_external.py, midnight_clinical_enhanced.py, foundation_model_processor.py, README.md, CLAUDE.md, AGENTS.md, TODO.md; GitHub API for visibility and licence state, read 2026-07-26. Every finding independently re-verified by a second reading instructed to refute it.