Reading-Behaviour Capture

Two properties decide whether recorded viewing data can answer a reading-behaviour question at all — positions in image space with the zoom factor, and preserved visit order — and neither is visible in a feature list.

Two properties decide whether recorded viewing data can answer a reading-behaviour question at all — positions in image space with the zoom factor, and preserved visit order — and neither is visible in a feature list.

What it is

Studies of how pathologists read slides — where they look, how long they dwell, whether two readers examine the same regions — all depend on a recording of viewing behaviour. Eye trackers, mouse trackers, screen recorders and viewer-embedded instruments all advertise the same output: a heatmap of attention. The heatmaps look alike, and that is the trap. Whether the underlying recording can support any analysis beyond the picture is decided by two properties that no product description mentions.

This page is a pre-adoption test, not a survey. The metrics themselves live on Interobserver Agreement and QuPath Patoloji Atlası Extension.

Property 1 — positions must be in image space, with the zoom factor recorded per sample

A recording in screen coordinates says the reader was looking at pixel (1200, 640) of a monitor. A recording in image coordinates says they were looking at a particular place on the slide. Only the second is a fact about the tissue, and converting the first into the second is impossible after the fact unless the viewport transform was captured at the same moment.

The reason is that the transform is not constant. A reader pans and zooms continuously, so the mapping from screen pixel to slide position changes many times a second. Recording screen coordinates alone discards it irrecoverably.

The zoom factor is not an optional extra, it is part of the position. The same screen pixel is a different measurement at 5× and at 40× — at low power it covers a wide field of architecture, at high power a few nuclei. Two readers whose dwell maps look identical in screen space may have examined regions differing by an order of magnitude in area. Any comparison between readers, or between a reader and a model’s attention map, needs the magnification alongside the coordinate. This is the same argument Patch Extraction makes about tiles and Video-Based WSI Mosaicing makes about mosaics: µm per pixel is what turns a picture into a measurement, and it has to be recorded at capture time or it is gone.

The practical consequence: an instrument that runs outside the viewer, observing the screen, can almost never satisfy this. One that runs inside the viewer gets the transform for free, because the viewer already knows it.

Property 2 — visit order must survive, not just visit counts

A heatmap is an aggregate: how much attention each location received. Most of the interesting questions are about the sequence.

• Fixations are dwell episodes — the reader stayed near one place for some minimum time. That needs timestamps grouped into episodes, not a total.
• Scanpaths and transition matrices ask where attention went next, which requires ordering.
• Revisits are diagnostic behaviour: returning to a region after examining another is how comparison happens, and a count cannot distinguish one long look from three short returns.
• Path-similarity measures between two readers consume sequences.

An implementation can quietly destroy this while still producing a correct heatmap. The pattern to look for is a per-pixel array that stores a last-visit stamp rather than a list of visit times: it renders a plausible trail, and revisits overwrite each other, so the returns — the most informative events — are exactly what is lost. MouseTracks does precisely this, and it is invisible in its output.

Check the storage structure, not the output. Both properties are questions about what the recording contains, and both are answered by reading how a sample is written to disk. A feature list, a screenshot of a heatmap and a demo video cannot distinguish an instrument that satisfies them from one that does not.

Why it matters for my work

The group owns an instrument that satisfies both properties — the focus/ package in QuPath Patoloji Atlası Extension samples in image-pixel coordinates with the viewer’s exact downsample factor per tick, and keeps sequences, which is why it can compute fixations by I-DT and pairwise path similarity at all. That is not an accident of feature richness; it follows from running inside QuPath.

The test earned its place by rejecting something. MouseTracks is well maintained, actively released and does what it advertises, and it fails both properties: screen-pixel space with no access to the viewer transform, and a last-visit stamp instead of a sequence. Neither is a bug and neither is fixable downstream. The evaluation took a day; asking these two questions first would have taken ten minutes.

Two places this generalises immediately:

• Eye trackers. A hardware eye tracker reports gaze in screen coordinates by construction. Using one for slide reading means logging the viewer’s pan-and-zoom state simultaneously and joining the two streams — a real integration task that has to be planned for, not discovered.
• Screen recording. Recording the session as video satisfies neither property: positions are screen pixels, and although order is preserved, recovering it requires reconstructing the viewport transform frame by frame. It also captures whatever is on screen, which raises the window-title and on-screen-identifier problem described on De-identification.

How it connects

QuPath Patoloji Atlası Extension — the in-house instrument that satisfies both properties, and the concrete example of why running inside the viewer is what makes that possible.

MouseTracks — the worked rejection: maintained, capable, and structurally unable to answer the question, for exactly these two reasons.

Interobserver Agreement — what a compliant recording is for; agreement between readers in where they looked is only computable if both properties hold.

Mitotic Count — the sharpest case for why this matters clinically: the count depends on which hotspot was chosen, so measuring hotspot selection requires knowing where on the slide the reader was and at what power.

Ki-67 Proliferation Index — the same hotspot-selection dependence, and the marker whose national guidance exists because reader-to-reader variation is so large.

Patch Extraction — the same µm-per-pixel argument one layer down: a tile without a recorded magnification is not comparable across a cohort, and a gaze sample without one is not comparable across readers.

De-identification — any instrument that observes the screen rather than the viewer inherits the on-screen identifier surface, which is a governance cost of the outside-the-viewer approach.

Agentic Slide Navigation — the second thing a compliant recording is for. Not measuring readers, but training a model to navigate, which turns routine diagnostic sessions into supervision without anyone running a study.

Pathology-CoT — the published pipeline that consumes exactly the two properties above, and the evidence that this use is real rather than hypothetical.

Open questions

• Does the group intend to run a reading-behaviour study at all? The instrument exists and Interobserver Agreement records that it has never been exercised. These properties matter only once someone does.
• Is there a case for capturing both — inside-viewer sampling for the slide-reading question and whole-desktop capture for the workload question? They answer different things and neither substitutes for the other; nothing here records whether the second question is wanted.
• What sampling rate is actually needed? Fixation detection assumes samples dense enough to resolve dwell episodes, and nothing in this wiki records what rate the in-house instrument uses or what the analysis requires. [unverified]