LLM Adaptation Techniques
The ladder from a plain prompt to a fine-tuned model — few-shot, chain-of-thought, tool use, retrieval-augmented generation, fine-tuning — and what each rung costs before it buys anything.
What it is
A general-purpose model can be pointed at a specialised task without anyone training a new one. The techniques for doing that form a ladder, and the useful discipline is to climb it only as far as the task actually requires, because each rung costs more than the one below it.
Few-shot learning. Put a handful of worked examples in the prompt. Cheapest thing that works, and the fastest way to pin down output format — which matters when the output feeds another program. The catch is that the exemplars are a sample: choose them badly and their biases become the model’s.
Chain-of-thought. Ask for the reasoning before the answer. Costs one line of prompt, usually improves accuracy on multi-step problems, and produces an explanation you can inspect — which in a clinical setting is worth as much as the accuracy. The cost is output that is harder to parse, mitigated by asking for structured output alongside.
Tool learning. Give the model APIs it can call. This is how a model gets access to functions it cannot do reliably in its head — arithmetic, a database lookup, a code execution. Value depends entirely on which tools you expose.
Retrieval-augmented generation (RAG). Retrieve relevant documents and put them in the prompt, so the answer is grounded in a specific corpus rather than in whatever the model absorbed during training. This is the standard answer to two problems at once: knowledge that postdates training, and hallucination. Its ceiling is retrieval quality — a RAG system that retrieves the wrong passage answers confidently from the wrong passage, so the retrieval step needs evaluating in its own right, separately from the generation.
Fine-tuning. Train on annotated examples. Buys the most performance and shortens prompts, and costs annotation effort plus compute. It is also the only rung that produces an artefact you must then version, store and re-validate.
The ordering matters more than any individual rung. Reaching for fine-tuning before trying few-shot is the common expensive mistake, and RAG solves a different problem from fine-tuning — currency and grounding versus behaviour and format — so they are not alternatives.
What the full text pins down
The article body was read on 2026-08-30, having been filed from the companion README in July. The ladder above survives intact; what the full text adds is the operational detail the README omitted, and most of it is numeric enough to act on.
Start with 100 test cases, before climbing anything. Best practice 1 says collect roughly a hundred instances — input and expected output — for the task, and evaluate every rung against them with task-specific metrics. This is the step that makes the ladder a method rather than a sequence of things to try, and it is the one the group’s own LLM work has never taken.
Climb from zero-shot, one example at a time. The recommendation is not “use few-shot” but start at zero-shot and add one to five representative, diverse examples, specifically to fix edge cases and pin output style. For a classification task, the exemplars should demonstrate every label. Dynamic selection by semantic similarity to the instance being predicted is named as the more sophisticated variant.
Chain-of-thought is the default, not an option, on the explicit grounds that an inspectable rationale is worth as much as the accuracy in a clinical setting.
Temperature 0 and JSON output. Best practice 8, and the two settings together answer the reproducibility and parseability objections at once: temperature 0 for deterministic output, a JSON dictionary so the result parses into fields without a regex over prose. Higher temperature is reserved for one purpose — deliberately sampling diverse responses to ensemble them. Temperature is settable via API or a local model and generally not via a web app, which is one of the reasons web applications are not recommended during development.
Fine-tune on three triggers, and only these. Prompt engineering has plateaued below the required performance; large-scale high-quality training data is already available; or the working prompt has grown too long to be affordable. That third trigger is a cost argument rather than a performance one and is the one most often missed.
Parameter-efficient fine-tuning is the default rung, not the advanced one. Small and specific data suits PEFT — LoRA and its quantised variant — because full fine-tuning overfits it; large and diverse data suits full fine-tuning. The worked figure is the useful part: quantised LoRA fine-tuned a model for clinical text summarisation on thousands of instances and a single GPU, which puts the top rung inside the reach of a department rather than a lab. A fine-tuned model must then be evaluated on an independent test set — the ladder does not excuse the usual discipline.
Prompt optimisation can itself be automated. DSPy and TextGrad are named as frameworks that refine prompts programmatically against performance feedback, which is the answer to the objection that hand-tuning prompts is unversioned craft.
Two ceilings worth knowing before choosing a rung. Long context does not dissolve the retrieval problem: lost-in-the-middle, where a model fails to use information buried in the middle of a long prompt, is stated to persist in long-context models, so stuffing a whole protocol into the window is not equivalent to retrieving the relevant part of it. And the automatic metrics for generated text — BLEU, ROUGE, BERTScore — are flagged in the source as not always correlating with human judgement, so a summarisation or translation rung cannot be evaluated by them alone.
For RAG, the corpus recommendation is specific: high-quality domain sources — systematic reviews, medical textbooks, clinical guidelines — rather than open web text, with PubMed’s 39 million abstracts and PubMed Central’s 7 million open-access full texts named as the available substrate.
Why it matters for my work
The group already runs LLMs over pathology reports: LLM Report-Quality Checking scores reports against CAP/ICCR elements, and Report Text Extraction turns narrative into fields. Both are built as tools without an explicit account of which adaptation technique they use, which is why that page’s open questions — is the model version recorded, are prompts under version control — are still open. Naming the technique is the first step to versioning it.
Two mappings are worth drawing. CAP/ICCR compliance checking is a structurization task with a fixed external standard, which is exactly the shape few-shot plus structured output handles well, and where the required elements of a protocol edition are a natural retrieval corpus for RAG. And the trial-matching example in the source tutorial — match a patient description against eligibility criteria — is structurally identical to matching a report against a protocol checklist, so its published prompting patterns transfer directly.
Corrected 2026-08-30, on reading the full article. This page previously carried the caution that choosing a model on MedQA is a proxy-metric error, presented as something the wiki was adding to the source. The source makes that argument itself, and harder: it states that “higher scores on these datasets do not necessarily translate to better clinical utility, as there are no choices provided in real-life applications”, cites three studies showing MCQ performance does not correlate with real-world task utility, and its Best practice 4 confines automatic benchmarks to initial screening — the standing recommendation is to select on clinically evaluated results, with randomised controlled trials named as the rigorous end of that scale. The caution stands. Crediting it to this wiki did not.
What the full text does undermine is the table. Table 1 ranks nineteen models, and its header states they are “sorted by the highest-reported MedQA-USMLE (4 options) score”. Those nineteen rows draw their scores from fifteen different sources — a GPT-5 evaluation paper, a DeepSeek benchmark, a Gemini report, and for Llama 3.1 a live third-party leaderboard website that can change after publication. The numbers were produced by different groups under different prompting strategies, shot counts and reasoning settings, each reporting its best run. A maximum over heterogeneous, self-selected reports is a biased estimator, and the gaps the sort order encodes are smaller than the spread those protocol differences alone would produce. The paper warns at length that the metric does not transfer to clinical utility; it does not warn that its own cells are not comparable with each other, and it sorts by them anyway. The durable content of that table is the columns that are hard constraints — weights, size, interface, modality, context window. The MedQA column is provenance, not a ranking.
That is Class Imbalance and Accuracy one level up: not a single number standing in for a task it does not resemble, but a column of numbers from different experiments lined up as though they were one.
How it connects
LLM Report-Quality Checking — the group’s existing LLM layer; this page supplies the vocabulary its open questions need, since you cannot version a prompting strategy you have not named.
Report Text Extraction — the rule-based-versus-LLM choice sits directly below this ladder: where a rule is crisp, a regex is more auditable than any rung here.
Synoptic Reporting — CAP/ICCR required elements are both the target of the structurization task and the obvious retrieval corpus for a RAG approach to checking.
Interobserver Agreement — the right framework for validating any of these outputs: treat the model as another rater and measure agreement against expert review.
Class Imbalance and Accuracy — the same proxy-metric trap, in a different setting: a single benchmark number standing in for the task you actually care about.
askLLM — the retrieval rung with a number attached: someone else’s jamovi module that measured its own fabrication rate, grounded the model in a scanned catalogue, and measured again.
Jin 2026 — Guidance on the Use of LLMs for Medical Research — the NIH/NLM tutorial this ladder is drawn from, laying out the same progression as five phases: formulate, choose, prompt, fine-tune, deploy. Its full text was read on 2026-08-30, a month after the note was filed from the companion README alone; the PDF now sits beside the note as sources/papers/jin-2026-llm-medical-research.pdf. The note is append-only and still carries its abstract-only read-status callout and the withdrawn reservation corrected above, so this page, not that note, is the current reading.
Open questions
- Which rung does the existing report checker actually sit on — plain prompting, few-shot, or something else? Not recorded anywhere.
[unverified] - Would RAG over the CAP protocol text beat putting required elements directly in the prompt? Testable, and it would also make protocol-edition changes a data update rather than a prompt rewrite. There is now an external data point on the closest analogous problem. askLLM faced models inventing jamovi menu paths — a fixed, enumerable vocabulary that has to be quoted rather than composed, which is structurally what a CAP required-element list is. Sending the real scanned catalogue and demanding verbatim quotation took it from at least three structural fabrications to 18 of 18 paths matching the catalogue exactly, on a deliberately small sample the author flags himself. Not proof, but it is evidence, and the mechanical check it enables — compare every emitted reference against the list — is the part worth copying regardless.
- Is there enough annotated Turkish report data to make fine-tuning worth considering, or is few-shot the ceiling for the bilingual case? The cost side of that question is now bounded, and it is lower than assumed. The source’s worked figure for the top rung is quantised LoRA on thousands of instances and one GPU for clinical text summarisation, not a cluster and not a hundred thousand annotations. So the blocker is squarely the annotated Turkish set, not the compute — which makes it the same blocker as every open question on Report Text Extraction, and it should be built once and used for both.
- None of the questions above can be answered without an evaluation set, and the source specifies its size: roughly 100 instances. That is the cheapest unblocking action on this page. A hundred Turkish and English report excerpts with expected outputs would decide the rung question, the RAG-versus-prompt question and the fine-tuning question by measurement rather than argument, and the same set serves LLM Report-Quality Checking’s missing human-reviewer validation. Nothing here is blocked on tooling.
- No page here yet on evaluating retrieval quality separately from generation quality, which is the part most RAG deployments skip.
- The retrieval rung has a governance cost this ladder does not price, and it is now itemised. Clusmann 2026 — Safety and Security of Large Language Models in Healthcare (Nature Review) (Table 1) sets out what a safe RAG corpus needs: expert-curated and locally supplemented content, peer review, versioning and timestamping, logged and auditable updates, span-level citations from output back to source passages, sanitised retrieved text to block retrieval-time injection, red teaming with poisoned inputs, and provenance coverage and evidence freshness tracked as metrics. Read against askLLM’s result and the “personal chatbot over my own papers” pattern that the pathCast talk filed as
sources/notes/pathcast-2026-singh-what-ai-can-and-cannot-do.mddemonstrates: a trusted corpus is not a property of the documents but of the pipeline around them, and “the papers I have already read and trust” is still a corpus that needs versioning, sanitising and citation-checking. The same review’s point that grounding suppresses invention but not sycophancy bounds the rung’s benefit — retrieval fixes what the model does not know, not what it is willing to agree to.