Align model card with archived evidence and research limitations

README.md

@@ -8,21 +8,29 @@ tags:
 - clinical-reasoning
 - chain-of-thought
 - sft
+- research
 language:
 - en
 ---
 
 # BrainMed-8B-SFT
 
-Full-parameter supervised fine-tune of `FreedomIntelligence/HuatuoGPT-o1-8B` on
-[`Williamsanderson/MedReason-MedO1-Reasoning-46K`](https://huggingface.co/datasets/Williamsanderson/MedReason-MedO1-Reasoning-46K), a union of
-KG-grounded MedReason reasoning traces and verifier-checked medical-o1 traces.
+Full-parameter supervised fine-tune of `FreedomIntelligence/HuatuoGPT-o1-8B`, a model already
+medically adapted by SFT and reinforcement learning, on
+[`Williamsanderson/MedReason-MedO1-Reasoning-46K`](https://huggingface.co/datasets/Williamsanderson/MedReason-MedO1-Reasoning-46K),
+a union of MedReason and medical-o1-reasoning-SFT traces.
 
-- **Parameters:** 8,030,261,248 (8.03B) - full weights, not an adapter
+> **Research artefact only.** Not a medical device, not clinically validated, not for decisions
+> about any real person. This checkpoint scores **below its own base** on the historical
+> six-task mean (see Evaluation).
+
+- **Parameters:** 8,030,261,248 (8.03B), full weights, not an adapter
 - **Precision:** BF16
-- **Recipe:** MedReason (arXiv:2504.00993) - lr 5e-6, effective batch 128, 3 epochs,
-  cosine schedule with 5% warmup, weight decay 0.1, DeepSpeed ZeRO-3, bf16
-- **Weights obtained by:** lowest held-out validation loss (val loss 0.7557 at optimizer step 1041)
+- **Recipe (reported):** lr 5e-6, effective batch 128, 3 epochs, cosine schedule with 5% warmup,
+  weight decay 0.1, DeepSpeed ZeRO-3
+- **Checkpoint:** end of epoch 3 (optimizer step 1041, validation loss 0.7557). This is **not**
+  the lowest-validation-loss checkpoint: epoch 2 had the lower validation loss (0.727) but a
+  lower benchmark mean (55.91%).
 
 ## Output format
 
@@ -33,28 +41,38 @@ The model is trained to reason inside `<think>...</think>` and answer inside
 You are a medical reasoning assistant. Work through the clinical problem step by step inside <think>...</think>, grounding every step in established medical knowledge, then give the final, complete answer inside <answer>...</answer>.
 ```
 
-## Evaluation
+## Evaluation (historical, descriptive)
 
-| Benchmark | n | Accuracy |
-|---|---|---|
-| medbullets_op4 | 308 | 58.77 |
-| medbullets_op5 | 308 | 56.82 |
-| medmcqa_val | 4183 | 61.2 |
-| medqa_4opt | 1273 | 72.11 |
-| medxpertqa | 1449 | 18.08 |
-| pubmedqa_test | 1000 | 77.1 |
+Same harness for base and fine-tune: greedy decoding, strict answer prompt, historical
+`max(head, tail)` aggregate extraction with option-text and fuzzy fallbacks.
 
+| Benchmark | n (rows) | HuatuoGPT-o1-8B (measured) | BrainMed-8B-SFT | Difference (points) |
+|---|---|---|---|---|
+| MedBullets op4 | 308 | 59.09 | 58.77 | -0.32 |
+| MedBullets op5 | 308 | 55.19 | 56.82 | +1.63 |
+| MedMCQA (val) | 4183 | 63.30 | 61.20 | -2.10 |
+| MedQA 4 options | 1273 | 77.93 | 72.11 | -5.82 |
+| MedXpertQA | 1449 | 17.05 | 18.08 | +1.03 |
+| PubMedQA | 1000 | 80.20 | 77.10 | -3.10 |
+| **Macro6** | | **58.79** | **57.35** | **-1.45** |
 
+Rows are not independent cases (MedBullets op4/op5 share questions). Checkpoints were compared
+on these same tasks; no independent test set was held out. No confidence intervals are
+available because per-question predictions are not published here. The measured base is
++2.89 points above its published values on average under this harness, so these numbers must
+not be ranked against scores transcribed from other papers.
 
-Scored with the MedReason evaluation harness (greedy decoding, strict answer prompt,
-`max(head, tail)` extraction) on the benchmark files shipped by the MedReason and
-HuatuoGPT-o1 repositories - no benchmark was rebuilt or resampled.
+A plausible, untested explanation for the regression: medical-o1 is part of the base model's
+own training pipeline, and this run applies further SFT after the base's RL stage. No
+controlled ablation has established the cause.
 
-## Data integrity
+Full results, every archived run and limitations:
+[BrainHealthAI/BrainMed-8B — results/REPORT.md](https://github.com/BrainHealthAI/BrainMed-8B/blob/main/results/REPORT.md).
 
-Training rows: **44,351**.
+## Data integrity
 
-Rows overlapping the evaluation benchmarks were **kept**, matching the upstream setting under which the published comparison tables were produced. The overlap was measured and is declared below rather than removed; the evaluation therefore reports a clean-subset score alongside the raw one, and the clean figure is the unbiased estimate.
+Training rows: **44,351**. Rows overlapping the evaluation benchmarks were **kept** (upstream
+setting). The overlap was measured and is declared below.
 
 | Benchmark | items | overlapping training rows |
 |---|---|---|
@@ -66,18 +84,29 @@ Rows overlapping the evaluation benchmarks were **kept**, matching the upstream
 | medxpertqa | 1449 | 15 (1.04%) |
 | mmlu_pro_medical | 1535 | 340 (22.15%) |
 
-This overlap originates upstream: the MedReason corpus derives reasoning traces from MMLU-medical, MedXpertQA and Humanity's Last Exam, three sets it also evaluates on.
-
-Answer-format alignment was applied to 17,083 rows (38.52%): a canonical `The answer is X.` was appended where the letter was unambiguously recoverable from options already present in the question. Questions and reasoning traces are byte-identical to the source.
+The "clean" score in the evaluation files excludes flagged items. It does **not** undo training
+exposure, unknown base-model exposure or selection on these tasks, and is not an unbiased
+estimate. This overlap originates upstream: the MedReason corpus derives reasoning traces from
+MMLU-medical, MedXpertQA and Humanity's Last Exam.
 
+Answer-format alignment was applied to 17,083 rows (38.52%): a canonical `The answer is X.` was
+appended where the letter was unambiguously recoverable from options already present in the
+question. This is a formatting transformation, not physician validation of the traces.
 
 ## Intended use
 
-Research and decision support. Not a medical device; not for autonomous clinical use.
+Research on medical-model adaptation and evaluation methodology only. Not for clinical use,
+decision support in care, or any decision about a real person.
 
 ## Citation
 
 ```bibtex
+@software{brainmed8b2026,
+  title  = {BrainMed-8B: medical reasoning adaptation and evaluation pipeline},
+  author = {BrainResearch},
+  year   = {2026},
+  url    = {https://github.com/BrainHealthAI/BrainMed-8B}
+}
 @misc{wu2025medreason,
   title={MedReason: Eliciting Factual Medical Reasoning Steps in LLMs via Knowledge Graphs},
   author={Wu, Juncheng and others}, year={2025}, eprint={2504.00993}, archivePrefix={arXiv}