Scanner Log Analysis
Turning raw scanner and transfer logs into operational metrics — a data-cleaning problem more than a statistical one, with two specific traps already documented.
What it is
Scanner logs record per-slide events: scan started, scan completed, transfer attempted, transfer succeeded or failed. Converting that into “how long does scanning take” requires several decisions that are easy to get silently wrong.
Trap one: timestamps. Scanner log timestamps are in local time with daylight saving. The scan-time-in-real-life repo already normalises to UTC. The documented instruction is not to double-correct — a second conversion applied by a well-meaning analyst produces an hour-shifted dataset that looks entirely plausible and quietly ruins every hour-of-day figure.
Trap two: retries. Some entries logged as “failed transfers” are benign retries that subsequently succeeded. Counting them as failures inflates the error rate. The analysis de-duplicates by (scan_id, final_outcome), and that logic must be preserved.
A check that does not depend on the logs. Both traps are traps precisely because a mis-corrected dataset still looks plausible, and nothing inside the logs can catch that. The slide files themselves carry an independent record: the Aperio .svs header holds Date, Time and Time Zone fields written by the scanner. Comparing that against the log-derived timestamp for the same slide is the only external check available, and a systematic one-hour or one-day offset appearing on one side of a DST boundary is exactly what a double correction looks like.
ScanTools’s scandate is the closest off-the-shelf tool, but it is limited in two ways that matter here and it should not be adopted without reading them: as committed it is broken and writes an empty CSV while exiting cleanly, and even once fixed it extracts only the Date field — its filter structurally cannot reach Time or Time Zone, so it validates dates, not hour-of-day. Extending it to capture all three fields is a few lines and would turn a date-level sanity check into a real timestamp cross-check. Its date-format handling is also ambiguous between MM/DD/YY and DD/MM/YYYY, which is silently wrong for days 1–12 — the format must be confirmed against a real Memorial file first.
Once cleaned, the analysis produces hour-of-day and day-of-week load patterns, case-level timing metrics, and queueing indicators. The framing that makes these actionable is queueing: near capacity, small increases in load produce disproportionate increases in wait, so load patterns matter more than average load. Documenting the transfer architecture alongside is part of the deliverable, since an un-instrumented step is invisible to the analysis — the onboarding task explicitly asks contributors to flag any transfer step that is not instrumented.
Why it matters for my work
Scanning Time in Real Life is an active project with a journal figure set, an internal optimisation memo, and an operations dashboard as deliverables. It is also the clearest example in the repo of a study where the intellectual work is in the data cleaning: the statistics are simple, and the validity rests entirely on the two traps above being handled.
How it connects
Scanner Logs — the dataset page for the raw logs themselves, including what is and is not instrumented.
Turnaround Time — scanning and transfer are components of overall digital-workflow turnaround; the same skew and censoring cautions apply.
WSI Quality Control — quality failures cause rescans, which appear in the logs as extra load.
Whole Slide Imaging — file size is the physical reason transfer is a bottleneck at all.
Patoloji Bilgi Yönetim Sistemi (LIS) — the LIS side of the timestamp story, where received, grossed, signed-out, and amended each have specific documented meanings.
Open questions
- Has the log-derived timestamp ever been reconciled against the scan timestamp embedded in the slide files? Not recorded. [unverified] This is the only available external check on both documented traps.
- Are the timing distributions reported as medians and percentiles rather than means? Scanner timings are right-skewed for the same reasons turnaround is. [unverified]
- Which transfer steps are currently un-instrumented? The onboarding task implies some are.
- Is scanner identity retained as a covariate? If different scanners have different throughput, that is both an operational finding and a confounder for any pooled analysis.