askLLM

A jamovi module that answers questions about your open dataset by sending the selected variables’ summary statistics to an LLM — and, more usefully, the best-measured demonstration in this wiki that grounding a model in a real catalogue stops it inventing structured references.

Purpose

Verdict: not adopted, but worth installing for two specific things — as ready-made teaching material on hallucination, and as the worked example behind a technique that transfers directly to LLM Report-Quality Checking.

The module itself is straightforward: pick variables, type a question, tick Submit, and the selected variables’ summary statistics plus your question go to an LLM. Five providers — NVIDIA NIM (the default), Google Gemini, GitHub Models, a local Ollama server, or any OpenAI-compatible endpoint. Built on ellmer. It is not a statistical engine and does not pretend to be; it suggests what analysis to run.

What makes it worth a page is not the feature. It is that the author measured his own module’s hallucination rate, published the failures verbatim, built a mitigation, and measured again — and the documentation that resulted is more honest than most published papers in sources/papers/.

The result worth citing

The dangerous failure mode was not wrong statistics. It was wrong menu paths delivered in the same confident tone as right ones. docs/LIMITATIONS.en.md tabulates real v1.0 output: Analyses > Compare > Independent Samples t-test (jamovi has no “Compare” menu), Exploration > Correlation (it is under Regression), Exploration > Principal Component Analysis (it is under Factor), and Classification > Discriminant Analysis and Machine Learning > Classifier, which do not exist in base jamovi at all. A beginner cannot distinguish a wrong path from a wrong recommendation — both just fail.

The v1.1 fix is the transferable idea. The module scans the machine for installed jamovi modules and their real menu trees, sends that catalogue alongside the data summary, and instructs the model to quote each path exactly as written there.

	Mechanically verifiable paths	Fabrication
v1.0 (no catalogue)	0 — models did not adopt the format; ~70% semantically correct by manual review	at least 3 structural fabrications: invented submenu, flattened hierarchy, generic rewrite
v1.1 (catalogue grounded)	18 / 18 matched the scanned catalogue verbatim	zero

The author states the caveat himself before anyone else can: 8 calls, 18 paths, 2 models — “not enough to guarantee zero misses globally, but directionally consistent with spec intent”. That is the right way to report a small-n result, and it is why the number is quotable rather than marketing.

Why this matters here and not just there. A jamovi menu path and a CAP/ICCR required-element name are the same kind of object: a fixed, enumerable vocabulary the model must quote rather than compose. The failure is identical — a plausible-sounding item that does not exist — and so, apparently, is the fix. See LLM Adaptation Techniques, whose open question about whether retrieval beats putting the list in the prompt now has an external data point.

Data used

None of ours. The published testing used iris and mtcars.

What the module transmits, read from R/data-summary.R rather than from the README, because this is the question a pathologist has to answer before installing it.

Aggregates only, and that part is true. No row-wise records are ever sent. Numeric variables contribute n, missing, mean, sd, median, min and max — all passed through signif(x, 4), so they are rounded rather than exact. Factors contribute level counts. Logicals contribute TRUE/FALSE/NA counts. Dates contribute only their class, n and missing — no dates leave, which the package’s own test suite asserts.

But a per-level frequency necessarily names the level. For a factor, the ten most frequent level names go out verbatim, truncated at 40 characters each. So a free-text column typed Nominal — a diagnosis line, a specimen type, a site code — sends its ten commonest actual values. The README says this in both languages (“factor level frequencies” / “各水準次數”); the in-app privacy notice does not. That notice, at R/askllm.b.R:185-192, says only that “the SUMMARY STATISTICS of the selected variables (never the raw data rows)” are sent. It is the text a clinician reads at the moment of deciding, and it is the one place the disclosure is thinner than the behaviour.

Three things make this much smaller than it first looks, and they should be stated alongside it:

• Identifier columns cannot be selected at all. The variable box declares permitted: [numeric, factor] and omits id, which is jamovi’s designated type for identifier columns. An accession number typed correctly is simply not offerable.
• Nothing is sent passively. submit defaults to false and is the explicit gate; variables must be dragged in; includeSummary can be unticked.
• Ollama is zero egress. A local server means nothing leaves the machine — though it is not the default, and NVIDIA NIM is.

Variable names are always transmitted verbatim and untruncated, which is worth knowing separately: a column called Patient_MRN or HIV_status discloses schema regardless of its type or contents.

This section was rewritten after verification. My first reading claimed date columns leak real dates and that a character column of accession numbers would send ten real ones. An independent reviewer asked only to refute it demonstrated both were wrong — Dates route to a branch that emits only type, n and missing, and the id exclusion blocks the accession case — and that the README already discloses the level-name behaviour. What survived is narrower and better aimed.

Methods

Read at source level. Three findings beyond the grounding result.

One genuine bug, small and specific. In R/data-summary.R, an ordered factor gets a level order: line built from levels(x) — all of them, ignoring the user’s maxLevels setting, which the levels by count: line directly above it respects. The two adjacent lines disagree about the same user control. The 4000-character budget bounds the damage, and the existing test only exercises three levels so the inconsistency is untested. Worth reporting upstream; the author is responsive.

No high-cardinality guard anywhere. A nominal variable with 5,000 distinct values is summarised by the same code path as a two-level one. There is no identifier detection, no cardinality heuristic, and no preview of the assembled prompt before it is sent. That is the gap that would matter most if this were ever pointed at clinical data.

A jamovi constraint worth knowing independently of this module. R/key-loader.R exists because the jamovi engine sanitises process environment variables and its HOME is a build-time garbage value, so the ordinary Sys.getenv() route to a secret does not work. The module falls back to reading the Windows registry (HKCU\Environment, then HKLM), with ~/.Renviron last. Anyone building a jamovi module that talks to an authenticated API will hit this, and it is now recorded on jamovi Module Development.

Also documented and useful: measured response times across models, where microsoft/phi-4 took 314 seconds against 4.7 for openai/gpt-4.1 — a five-minute wait that looks exactly like a hang. And a catalogue-versus-inference mismatch, where meta/meta-llama-3.1-8b-instruct is listed by GitHub Models but returns HTTP 400 Unknown model.

Current state / open questions

• Would the catalogue-grounding trick transfer to the report checker? This is the actionable item. CAP/ICCR element names are an enumerable vocabulary; the technique is “send the real list, demand verbatim quotation, then check the output against the list mechanically”. The last step is what makes it measurable, and LLM Report-Quality Checking currently has no measurement at all.
• Is the ordered-factor maxLevels inconsistency known upstream? Not reported as of 2026-07-28.
• Would running this against Ollama on a local model be a realistic teaching setup — no keys, no egress, no per-call billing? The debounce design suggests billing was a live concern.
• The module is bilingual English / Traditional Chinese throughout, including all setup docs. Given Turkish and English both appear in this repo’s own sources, its approach to a bilingual module is a ready reference if the ClinicoPath modules ever need one. Unexamined.

How it connects

LLM Adaptation Techniques — the ladder this sits on. The catalogue mechanism is the retrieval rung, and this is the first measured example of it in this wiki rather than a described one.

LLM Report-Quality Checking — where the technique should go next: the same class of failure (inventing a structured reference that sounds right) and, on this evidence, the same fix.

jamovi Module Development — the framework this is built on, and the source of its most transferable engineering detail: how a module reaches a secret when the engine has scrubbed the environment.

ClinicoPath Org Collection — the library this was found in, where it was flagged as the most recently touched repository and the one most worth a separate look. This is that look.

De-identification — the standing reminder that identifiers hide in more places than a filename; here it is level names and column names rather than the rows everyone thinks about.

Derived from: repository cloned 2026-07-28 at 5323617 (full clone, 35 commits); DESCRIPTION, R/data-summary.R, R/askllm.b.R, R/key-loader.R, R/llm-providers.R, jamovi/askllm.a.yaml, docs/LIMITATIONS.en.md and tests/testthat/test-data-summary.R read directly. The transmission analysis was independently checked by a reviewer instructed only to refute it, which corrected two wrong examples and one unfair framing before anything was written here.