Model card: model summary, evaluation, quickstart, details

README.md

@@ -7,22 +7,62 @@ tags:
 - medical
 - clinical
 - decision-model
-- jev
+- calibrated
 - systemone
-- lora
 language:
 - en
 ---
 
 # MedDecider-9B
 
-A medical decision model in the Jev style: it reads a state (a clinical note, a trial record, any JSON) and a question with 2–10 options, and returns a probability for every option.
+MedDecider-9B brings the family's training to a model that runs on a single mid-range GPU. Given a clinical state (a note, a trial record or any JSON) and a question with 2 to 10 answer options, it returns a calibrated probability for every option in a single scoring pass, without generating text. It is built for high-volume clinical decision steps: medication and diagnosis checks in notes, trial eligibility screening, evidence classification and medical knowledge questions.
 
-| Base model | Adapter | Size | Training data | Choice temperature |
+## Model summary
+
+| | |
+| --- | --- |
+| Developer | thelion.ai |
+| Model type | Medical decision model: choice, yes/no and ordered-scale questions |
+| Parameters | 9.7B |
+| Fine-tuning data | 35,800 decisions, 7.5M tokens, one epoch |
+| Fine-tuning compute | 35 minutes on one NVIDIA H100 |
+| Output | A probability for every answer option, calibrated per question type |
+| Languages | English (training); also evaluated on Polish medical exams |
+| Release | October 2026 |
+| License | CC BY-NC 4.0 |
+
+## Highlights
+
+- **Passes every session of the Polish final medical exam (LEK), 2008–2026:** 77.9% over 4,722 questions, and 78.7% on the 286 new questions of the 2025–2026 sessions, 7.4 points above the average examinee (71.3%).
+- **More robust than Jev 1.13:** 82.9% vs 79.5% on robustness tests, and it follows an instruction hidden in a note 2.0% of the time (Jev 1.13: 13.7%).
+- **Ahead of Clef-flash 9B** on robustness (82.9% vs 70.3%) and NLI4CT (0.799 vs 0.786), and level on unseen exams (57.5% vs 56.5%).
+
+## Evaluation
+
+All models answer the same 9,849 held-out test items in the same request format. Each competitor runs through its authors' own inference code and Jev through TypeSafe's API; on NLI4CT this setup reproduces their published scores to within 0.003. Unseen medical exams: MedXpertQA, MMLU-Pro health, Medbullets, MedExQA, MedConceptsQA and Polish specialty exams (PES). Unseen clinical problem types: adverse drug effects, PubMed RCT sentence roles, symptom to diagnosis, medical question pairs, health-advice strength and MedQuAD question types. Robustness: "none of the above" substitution, 8-option MedQA and instructions planted in notes and trial records. None of these datasets was used in training.
+
+| Benchmark | MedDecider-9B | Jev 1.13 | Clef-flash 9B | JPT-9B |
 | --- | --- | --- | --- | --- |
-| `Qwen/Qwen3.5-9B` | LoRA rank 32, all linear layers | 173 MB | v4 mix: 35,800 items, one epoch | 1.43 |
+| Unseen medical exams (6 datasets) | 57.5% | **65.4%** | 56.5% | 55.1% |
+| Unseen clinical problem types (6 datasets) | **89.3%** | 87.7% | 88.1% | 88.3% |
+| Robustness (4 tests) | **82.9%** | 79.5% | 70.3% | 74.2% |
+| Follows a planted instruction (lower is better) | **2.0%** | 13.7% | 12.0% | 10.9% |
+| NLI4CT macro-F1 | 0.799 | **0.841** | 0.786 | 0.790 |
+| TrialGPT trial criteria ¹ | **85.7%** | 68.3% | 67.5% | 67.7% |
+| In-distribution test halves ¹ | **86.1%** | 84.8% | 80.1% | 79.5% |
+
+¹ Test halves of datasets whose training halves are in MedDecider's training data, so these rows favour MedDecider. NLI4CT for the other models is their published score; for MedDecider it is our run in the Decision Index format. No NLI4CT data of any split was used in training. Best result in each row in bold.
 
-## Examples
+**Polish final medical exam (LEK).** 26 sessions from 2008 to 2026, pass mark 56%:
+
+| | MedDecider-9B | Average examinee |
+| --- | --- | --- |
+| All 26 sessions (4,722 questions) | 77.9%, every session passed | |
+| 286 new questions from the 2025–2026 sessions ² | 78.7% | 71.3% |
+
+² Questions that do not appear in the public question bank of earlier sessions,. Examinee figures are from the exam board's per-question statistics; questions with images were excluded.
+
+## Quickstart
 
 Install with `pip install "transformers>=5.18" peft torch flash-linear-attention`. Each call returns a probability for every option. The numbers below are this model's own outputs from our evaluation run (both option orders averaged, temperatures from `meddecider_config.json`), rounded to three decimals. The clinical notes are synthetic test notes from our suite; the last question is from MMLU-Pro (MIT licence).
 
@@ -122,93 +162,57 @@ mj.decide(None, "Most surveillance systems use which of the following study desi
 # correct: Serial cross-sectional
 ```
 
-## About the MedDecider family
-
-A medical decision model in the Jev style. It reads a state (clinical note, trial record, any JSON) and a question with
-2–10 options, and returns a calibrated probability for every option from the logits of the option letters. There is no
-text generation: two forward passes, one per option order, averaged.
+## Model details
 
-| | MedDecider-31B | MedDecider-27B |
-| --- | --- | --- |
-| Base | `google/gemma-4-31b-it` (Apache-2.0) | `Qwen/Qwen3.8-27B` (Apache-2.0) |
-| Adapter | LoRA r16 on all linear layers, 267 MB | LoRA r16 on all linear layers, 234 MB |
-| Memory (bf16) | about 63 GB | about 56 GB |
-
-MedDecider Duo averages the two models' probabilities. It needs both in memory (about 120 GB, e.g. one H200) and twice the compute.
-
-## Use
-
-```
-pip install "transformers>=5.18" peft torch flash-linear-attention
-python meddecider_infer.py --adapter ./meddecider-31b-adapter
-python meddecider_infer.py --adapter ./meddecider-31b-adapter --adapter2 ./meddecider-27b-adapter   # Duo
-```
-
-```python
-from meddecider_infer import MedDecider
-mj = MedDecider("./meddecider-31b-adapter")          # base model is read from meddecider_config.json
-mj.decide(note_text, "Is the patient on an anticoagulant?", ["Yes", "No"], qtype="noul")
-mj.decide({"primary_trial": {...}}, "Classify the statement ...", ["Entailment: ...", "Contradiction: ..."])
-```
-
-`qtype` selects the temperature: `choice` (pick one option), `noul` (yes/no), `score` (ordered levels).
-
-## Results (test halves; nothing here was used for training or model selection)
-
-| Model | In-distribution | TrialGPT criteria | Unseen exam datasets | Unseen problem types | Robustness | Follows injected instruction | NLI4CT macro-F1 |
-| --- | --- | --- | --- | --- | --- | --- | --- |
-| **MedDecider-31B** | 0.892 | 0.869 | **0.702** | 0.884 | 0.888 | 2.7% | **0.843** |
-| MedDecider Duo (31B + 27B) | **0.896** | **0.876** | 0.696 | **0.895** | **0.898** | 3.0% | **0.845** |
-| MedDecider-27B | 0.881 | 0.854 | 0.644 | 0.897 | 0.883 | 3.0% | 0.827 |
-| MedDecider-9B | 0.861 | 0.857 | 0.575 | 0.893 | 0.829 | 2.0% | 0.799 |
-| MedDecider-4B | 0.847 | 0.854 | 0.547 | 0.878 | 0.797 | 5.4% | 0.776 |
-| Qwen3.8-27B, untrained | 0.814 | 0.734 | 0.604 | 0.892 | 0.681 | 35% | 0.792 |
-| Gemma 4 31B, untrained | 0.833 | 0.612 | 0.680 | 0.882 | 0.539 | 63% | 0.830 |
-| Jev 1.13 (published) | | | | | | | 0.841 |
-
-- **Unseen exam datasets:** MedXpertQA, MMLU-Pro health, MedBullets, MedExQA, MedConcepts, and the Polish PES exam.
-- **Unseen problem types:** adverse drug events, PubMed RCT sentence roles, Symptom2Disease, medical question pairs, health advice and MedQuAD intent.
-- **Robustness:**
-  - "none of the above" substitution;
-  - 8-option MedQA;
-  - instructions injected into notes and trial records.
-- **NLI4CT:** the Jev Decision Index clinical benchmark, 5,500 test statements. The rows are rebuilt byte-identical to the index's file. The state is passed as `json.dumps` of the request. Score is macro-F1 over Entailment and Contradiction.
-  - MedDecider-31B: 95% CI 0.833–0.853.
-  - Duo: 95% CI 0.836–0.855.
-  - Jev's row-level answers are not public, so the comparison with Jev has no paired test.
-
-**Calibration:** with the shipped temperatures, MedDecider-31B has an expected calibration error of 0.029 on the 12 unseen datasets. It is at least 90% confident on 55% of those items and right on 94% of them.
+- **Architecture:** a LoRA adapter (rank 32, all linear layers, 173 MB) on `Qwen/Qwen3.5-9B`, merged at load time. `meddecider_infer.py` downloads the base model named in `meddecider_config.json`.
+- **Readout:** next-token logits of the option letters after a chat prompt with reasoning turned off. Each question is scored in both option orders and the two probability vectors are averaged.
+- **Calibration:** one temperature per question type (choice 1.43, yes/no 1.0, score 1.0), fitted on in-distribution calibration data only. Refit on a few hundred labelled local examples before using the probabilities as risks.
+- **Input:** any string or JSON state; JSON is serialised with `json.dumps(state, ensure_ascii=False)`.
 
 ## Training
 
-One epoch over 35,800 items (about 7.3M tokens):
+One epoch over 35,800 decisions:
+
 - **Medical exams:** MedQA and MedMCQA.
-- **Diagnosis:** DDXPlus.
-- **Code-generated clinical notes:** with minimal-pair twins.
-- **Trial eligibility conventions.**
-- **Evidence and NLI datasets:**
-  - Evidence Inference 2.0 (MIT);
-  - BioNLI (CC BY 4.0);
-  - VitaminC (CC BY-SA 3.0);
-  - EQUATE (Apache-2.0);
-  - MultiNLI.
-- **Robustness items:** "none of the above", injected instructions.
-- **General decision replay.**
-- **6,000 synthetic minimal pairs about invented clinical trials:** results, adverse events, eligibility and dosing. Labels are computed by code.
-- **State format:** a share of states is rendered as JSON.
-
-Every item is shown with its options shuffled. The loss is cross-entropy on the letter logits. LoRA lr 1e-4, cosine schedule; 93 min (31B) and 85 min (27B) on one H200.
-
-**No NLI4CT data of any split** (train, dev, practice or test) was used. Model selection used only the calibration halves of our own suite.
-
-## Limits
-
-- This is a research model, not a medical device. Outputs must not drive clinical care without validation and oversight.
-- Knowledge questions remain the ceiling: about 70% on unseen exam datasets.
-- On NLI4CT the models predict "Entailment" somewhat more often than the gold labels: 37.5% vs 33.5% for MedDecider-31B.
-
-## Later results
-
-- **Jev 1.13, run on the same rows through TypeSafe's API:** NLI4CT 0.840. Paired with MedDecider-31B: +0.003 (95% CI −0.007 to +0.012), a tie. On unseen exam datasets MedDecider-31B leads Jev 70.2% to 65.4%.
-- **Polish final medical exam (LEK), 26 sessions 2008–2026, 4,722 questions:** MedDecider-31B passes every session (lowest 83.2%, pass mark 56%). On 286 questions first used after Gemma 4's training cutoff it scores 89.9% against 71.3% for the average examinee.
-- **Clef 27B (Cloudflare, same Qwen3.8-27B base as MedDecider-27B):** MedDecider-27B leads it on unseen exams (+4.1 points) and robustness (+12.3); they tie on NLI4CT (0.827 vs 0.828).
+- **Differential diagnosis:** DDXPlus.
+- **Clinical notes generated by code**, with minimal-pair twins: medication status, negated and historical diagnoses, laboratory values, family history, timing.
+- **Clinical-trial eligibility criteria**, including "not enough information" cases.
+- **Evidence and inference:** Evidence Inference 2.0, BioNLI, VitaminC, EQUATE, MultiNLI.
+- **6,000 synthetic minimal pairs about invented clinical trials:** results, adverse events, eligibility and dosing, with labels computed by code.
+- **Robustness items:** "none of the above" options, extra distractors, instructions planted in the input.
+- **General decision replay**, so that non-medical decisions are not forgotten.
+
+A share of states is rendered as JSON, so the model reads structured records as well as free text.
+
+Every item is shown with its options shuffled, and the loss is cross-entropy on the option-letter logits (learning rate 1e-4, 3% warm-up, cosine decay to 10%). Model selection used only calibration halves of our own evaluation suite. No NLI4CT data of any split was used.
+
+## The MedDecider family
+
+| Model | Parameters | Unseen medical exams | Unseen clinical problem types | Robustness | NLI4CT macro-F1 | LEK, 4,722 questions |
+| --- | --- | --- | --- | --- | --- | --- |
+| MedDecider Duo (31B + 27B) | 59.1B | 69.6% | 89.5% | 89.8% | 0.845 | 89.2% |
+| MedDecider-31B | 31.3B | 70.2% | 88.4% | 88.8% | 0.843 | 87.7% |
+| MedDecider-27B | 27.8B | 64.4% | 89.7% | 88.3% | 0.827 | 87.1% |
+| MedDecider-9B | 9.7B | 57.5% | 89.3% | 82.9% | 0.799 | 77.9% |
+| MedDecider-4B | 4.7B | 54.7% | 87.8% | 79.7% | 0.776 | not tested |
+
+MedDecider Duo averages the probabilities of MedDecider-31B and MedDecider-27B (load both with `MedDeciderDuo`).
+
+## Intended use and limitations
+
+- MedDecider is a research model, not a medical device. Its outputs must not drive patient care without validation on local data and clinical oversight.
+- It was trained on English data. Polish results come from evaluation only.
+- Knowledge-heavy questions remain the hardest case: accuracy on unseen exam datasets ranges from 55% to 70% across the family.
+- The clinical notes used for training and for the quickstart are synthetic. Performance on real records must be measured before deployment.
+- Licence: CC BY-NC 4.0 (non-commercial). The base model keeps its own licence.
+
+## Citation
+
+```bibtex
+@misc{meddecider2026,
+  title  = {MedDecider: calibrated medical decision models},
+  author = {{thelion.ai}},
+  year   = {2026},
+  url    = {https://huggingface.co/lion-ai/MedDecider-9B}
+}
+```