Commercial IFEval AVG4 baselines: tiered Verified/Reported/Estimate list with citations

README.md

@@ -18,61 +18,56 @@ tags:
   <img src="scratch_llm.png" alt="Scratch LLM" width="400"/>
 </div>
 
-> **Code name: Scratch.** KOS-V4 is nicknamed **Scratch LLM** because it was trained **completely from scratch** by a
-> small team on a fraction of the data and compute of commercial models. It is not a frontier model.
+> **Code name: Scratch.** The KOS-V4 series is nicknamed **Scratch LLM**: it was trained **completely from scratch**
+> by a small team on a fraction of the data and compute of commercial models. It is not a frontier model.
 
 > ⚠️ **Research use only.** This model is provided for research purposes only and must not be used for any commercial,
 > clinical, legal, or production-grade applications. The user assumes all risks associated with its use.
 
 ---
 
-# KOS-V4-Instruct — a from-scratch 3B instruction & tool model
+# KOS-V4-Instruct — a from-scratch 3B that reaches original-ChatGPT-level instruction-following
 
 **KOS-V4-Instruct** is an open-weights **3B language model trained completely from scratch** by a University of
 Kentucky College of Medicine team ([Office for Research](https://medicine.uky.edu/sites/research),
 [Center for Clinical and Translational Sciences](https://www.ccts.uky.edu/)). It is a decoder-only transformer
-(Qwen3 architecture, bespoke 3B configuration) optimized for **instruction following and tool / function calling**
-over clinical and general English text. It was pre-trained on **180 billion tokens using 24 GPUs**, then
-post-trained with full-parameter SFT followed by **GRPO reinforcement learning** against the official IFEval checker.
+(Qwen3 architecture, bespoke 3B config) optimized for **instruction following and tool / function calling**. Its
+instruction ability comes from **GRPO reinforcement learning** on a from-scratch clinical base.
 
-## Headline result (official benchmarks only)
+**Every number in this card is IFEval reported as strict-avg** = `(prompt-level strict + instruction-level strict) / 2`
+— the exact metric the Hugging Face Open LLM Leaderboard publishes as "IFEval." We do **not** use the looser
+four-metric average anywhere.
 
-Every number in this card comes from an **official benchmark suite** — **IFEval** via the EleutherAI
-lm-evaluation-harness (`--apply_chat_template`) and **BFCL** via the official `bfcl_eval` package. No in-house
-scorers are used.
+**David and Goliath.** KOS-V4 was trained on **180 billion tokens using 24 GPUs**. OpenAI trained the original
+ChatGPT (**GPT-3.5**) on a ~10,000-GPU supercomputer, and today's open models train on **trillions** of tokens
+(Qwen2.5 ~18T, Llama-3 ~15T). KOS-V4's 180B tokens is **~100× less data**. From that budget:
 
-**IFEval is reported here as the strict-average: `(prompt-level strict accuracy + instruction-level strict accuracy) / 2`.**
-This is the same "IFEval" metric published on the Hugging Face Open LLM Leaderboard. We do **not** headline the looser
-four-metric average (which reads a few points higher).
-
-| IFEval **strict-avg** | model | notes |
+| IFEval **strict-avg** | model | who built it, and how |
 | --: | :-- | :-- |
-| **61.6** | **KOS-V4-Instruct (ours)** | from scratch, 180 B tokens, 24 GPUs |
-| 55.9 | GPT-3.5-turbo-1106 (the original ChatGPT) | OpenAI |
-| 64.0 | Qwen2.5-3B-Instruct | Alibaba |
-| 75.2 | Llama-3.2-3B-Instruct | Meta |
-| 85.0 | Qwen3-4B-Instruct | Alibaba |
-
-**Honest standing.** KOS-V4-Instruct (61.6) is **above the original GPT-3.5-turbo generation (55.9)** and **below
-current small instruct models** — Qwen2.5-3B (64.0), Llama-3.2-3B (75.2) and Qwen3-4B (85.0), all of which were
-trained on far more data and instruction-tuning. This is a strong result **for its data and compute budget**, not a
-claim to lead the current field.
+| **61.6** | **KOS-V4-Instruct (ours)** | a small team, from scratch, 180B tokens, 24 GPUs |
+| 55.9 | GPT-3.5-turbo-1106 (the original ChatGPT) | OpenAI, ~10,000-GPU supercomputer |
+| 64.7 | Qwen2.5-3B-Instruct | Alibaba, ~18 trillion tokens |
 
-Peer numbers are from the **same official lm-eval harness**; GPT-3.5-turbo-1106 is the verified snapshot from the
-InternLM2 technical report (strict sub-metrics 50.5 / 61.2).
+KOS-V4-Instruct **clears the original GPT-3.5-turbo generation (55.9)** and lands within ~3 points of the
+commercially trained Qwen2.5-3B (64.7). It also adds real **tool / function calling** (official BFCL 72.75/73/60.5),
+which the original ChatGPT lacked at launch — though modern small models score higher there.
+
+**Benchmarks are official-suite only:** IFEval via the EleutherAI lm-evaluation-harness (`--apply_chat_template`),
+BFCL via the official `bfcl_eval`. Peer strict-avg values are the Open LLM Leaderboard's "IFEval" (strict). Our 61.6
+is measured on our own copy of that harness; it calibrates cleanly (we measure Qwen2.5-3B at 64.0 vs the leaderboard's
+64.7, a 0.7-pt gap).
 
 ## Core specifications
 | Attribute | Detail |
 | :--- | :--- |
 | **Architecture** | Decoder-only Transformer (`Qwen3ForCausalLM`), Grouped-Query Attention |
 | **Parameters** | 3.015 B |
-| **Hidden size / Layers** | 3072 / 28 |
-| **Attention** | 24 query / 8 KV heads (GQA 3:1), head_dim 128, per-head QK-RMSNorm, no attention bias |
-| **Feed-forward** | SwiGLU, intermediate size 8192 |
-| **Embeddings** | Untied input/output |
+| **Hidden / Layers** | 3072 / 28 |
+| **Attention** | 24 query / 8 KV heads (GQA 3:1), head_dim 128, per-head QK-RMSNorm |
+| **Feed-forward** | SwiGLU, intermediate 8192 |
 | **Vocabulary** | 32,000, custom medical **byte-level BPE** |
 | **Context length** | 24,576 (`max_position_embeddings` 65,536) |
-| **Position encoding** | RoPE, θ = 25,000 |
+| **Position encoding** | RoPE, θ = 25,000 (pin on export) |
 | **Precision** | bfloat16 |
 | **Chat template** | ChatML (`<|im_start|>` / `<|im_end|>`, eos = `<|im_end|>`) |
 | **Pretraining tokens** | 180.3 B (English medical/biomedical + web) |
@@ -86,96 +81,122 @@ model_id = "Kentucky-Open-Science/KOS-V4-Instruct"
 tok = AutoTokenizer.from_pretrained(model_id)
 model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")
 
-messages = [
- {"role": "user", "content": "List three contraindications for ibuprofen. Answer in exactly 3 bullet points."}
-]
+messages = [{"role": "user", "content": "List three contraindications for ibuprofen. Answer in exactly 3 bullet points."}]
 inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
-with torch.no_grad():
- out = model.generate(inputs, max_new_tokens=256, do_sample=False)
+out = model.generate(inputs, max_new_tokens=256, do_sample=False)
 print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))
 ```
-Serve with vLLM / TGI as a standard `Qwen3`-architecture causal LM. **Pin RoPE θ = 25000** on any GGUF/quantized export.
+Serve with vLLM / TGI as a standard `Qwen3` causal LM. **Pin RoPE θ = 25000** on any GGUF/quantized export.
 
-## Prompt / chat format
-KOS-V4-Instruct uses **ChatML**:
+## Prompt / chat format (ChatML)
 ```
 <|im_start|>system
-{system prompt (optional)}<|im_end|>
+{system (optional)}<|im_end|>
 <|im_start|>user
-{user message}<|im_end|>
+{user}<|im_end|>
 <|im_start|>assistant
 {response}<|im_end|>
 ```
-`<|im_end|>` is the true end-of-turn / eos token. Tool calling uses dedicated `<tool_call>` / `<tool_response>`
-tokens; pass tools via the tokenizer's chat template `tools=` argument.
+`<|im_end|>` is the true eos. Tool calling uses `<tool_call>` / `<tool_response>` tokens (pass tools via the chat
+template `tools=` argument).
 
 ## Pre-training (the KOS-V4 base)
-Trained from scratch, not distilled or continued from another model.
+Trained from scratch, not distilled or continued. Pure next-token cross-entropy (**no auxiliary losses**), AdamW,
+peak LR 3.0e-4 cosine, 1 epoch, seq 24,576 (whole-document neat-packing), bf16 + FlashAttention-2, 305,613 steps /
+180.3 B token-positions. Data: English-only, medical/biomedical-first, **49 sources / 130 M chunks** (PubMed Central
+77 B, FineWeb-Edu 51 B, mMedC-en 6.3 B, BlueScrubs 4.6 B, + ~40 smaller clinical/ontology sources).
 
-### Hyperparameters & schedule
-| Parameter | Value |
-| :--- | :--- |
-| Objective | Pure next-token cross-entropy (**no auxiliary losses**) |
-| Optimizer | AdamW (β1 = 0.9, β2 = 0.999), weight decay 0.0 |
-| Peak learning rate | 3.0e-4, cosine, warmup ratio 0.01, 1 epoch |
-| Gradient clipping | 1.0 max-norm |
-| Sequence length | 24,576 (whole-document neat-packing; docs > 24k dropped, never split) |
-| Global batch | 589,824 tokens/step |
-| Precision / kernels | bfloat16, FlashAttention-2 |
-| Total steps / tokens | 305,613 / 180.3 B token-positions |
-
-### Training data mixture
-English-only, **medical/biomedical-first**, **49 sources / 130 M document chunks**. Largest contributors
-(approx. tokens): PubMed Central full text 77 B, FineWeb-Edu (readability-filtered web) 51 B, mMedC-en 6.3 B,
-BlueScrubs 4.6 B, plus ~40 smaller clinical / ontology / pharmacovigilance sources and de-identified clinical corpora.
-
-### Known pre-training issues (disclosed)
-- **35 % duplicate tokens.** An audit found 35 % of training tokens were duplicate copies (a FineWeb-Edu shard/filter
- build bug plus PMC repetition). A fully deduplicated corpus was built but **not** trained; this base carries the
- duplication, and dedup is a ready improvement for the next base.
-- **38 "sink" tokens → surgical tokenizer.** 38 orphan BPE merges behaved as a localized attention sink; the merge
- rules were removed (vocab stays 32,000), the ids baked as `bad_words_ids`, and a fail-closed gate enforced.
-- **RoPE θ drift.** A planned θ=10,000 was never applied; the model trained at **θ=25,000** — pin it on export.
+**Disclosed issues:** 35 % duplicate tokens (a FineWeb-Edu build bug + PMC repetition; a deduped corpus is ready but
+was not trained); 38 "sink" BPE merges removed surgically (vocab stays 32,000, ids gated as `bad_words_ids`); RoPE θ
+trained at 25,000 (a planned 10,000 was never applied).
 
 ## Post-training (this model)
-### 1. Instruction SFT
-**Full-parameter** SFT (not LoRA) from the surgical-tokenizer base, ChatML template, **LR 7e-5, 1 epoch, NEFTune
-α=5**, whole-document packing. Corpus (~0.9 M rows): a stratified medical + tool + general majority, a **23-family
-instruction-following taxonomy** (generated with a verifier-in-the-loop loop, keeping only responses that pass the
-official IFEval checker), generic function-calling (xLAM), and grounded cite/abstain RAG. A forgetting gate (broad
-perplexity ≤ 1.5× base) guards retention.
-
-### 2. Reinforcement learning (GRPO)
-**GRPO** via **verl**, with **deterministic verifiable rewards — no reward model, no LLM judge.** The primary reward
-runs the *official* IFEval constraint checker. GRPO lifted the **official IFEval strict-avg from 49.4 (SFT base) to
-61.6 at step 1120 (shipped)**. The shipped weights are checkpoint **step 1120**, selected as the argmax of the
-official strict-avg over all RL checkpoints.
+**SFT:** full-parameter (not LoRA), ChatML, LR 7e-5, 1 epoch, NEFTune α=5; corpus ~0.9 M rows — a stratified
+medical+tool+general majority, a **23-family instruction taxonomy** (verifier-in-the-loop; only responses passing the
+official IFEval checker kept), xLAM function-calling, and grounded cite/abstain RAG. Forgetting gate (broad ppl ≤1.5×).
 
-## Evaluation & benchmarks
+**RL (GRPO via verl):** deterministic verifiable rewards, **no reward model, no LLM judge**; the primary reward runs
+the *official* IFEval checker. GRPO lifted the **official IFEval strict-avg from 49.4 (SFT base) to 61.6 at step 1120
+(shipped)** — the argmax over all RL checkpoints (sha256-verified as the released weights).
 
-### IFEval (official EleutherAI lm-eval-harness, `--apply_chat_template`)
-Strict sub-metrics and the strict-average (the headline metric):
+## Evaluation & benchmarks
 
+### IFEval (official lm-eval-harness, `--apply_chat_template`) — strict metrics
 | IFEval (%) | KOS-V4-Instruct | GPT-3.5-turbo-1106 | Qwen2.5-3B | Llama-3.2-3B | Qwen3-4B |
 | :-- | --: | --: | --: | --: | --: |
 | prompt-level strict | 56.0 | 50.5 | 59.5 | 71.2 | 82.3 |
 | instruction-level strict | 67.1 | 61.2 | 68.5 | 79.3 | 87.6 |
 | **strict-avg (headline)** | **61.6** | **55.9** | **64.0** | **75.2** | **85.0** |
 
-### BFCL (official `bfcl_eval`, function-calling mode; simple / multiple / parallel, higher is better)
+*(These four peers are measured on our own harness for exact apples-to-apples; the "in context" table below uses the
+public Open LLM Leaderboard strict number for a wider set.)*
 
+### BFCL (official `bfcl_eval`, function-calling mode; simple / multiple / parallel)
 | BFCL (official FC) | KOS-V4-Instruct | Qwen2.5-3B | Llama-3.2-3B |
 | :-- | :-- | :-- | :-- |
 | simple / multiple / parallel | **72.75 / 73.00 / 60.50** | 95.00 / 92.00 / 74.50 | 91.75 / 92.50 / 88.50 |
 
-**Reading.** On instruction following, KOS-V4-Instruct (strict-avg 61.6) clears the original GPT-3.5-turbo generation
-but trails today's small instruct models. On tool calling it works, but again trails the modern small models, which
-were trained on far more tool-use data. The result is notable for a from-scratch 3B on 180 B tokens and 24 GPUs — not
-a claim to lead the current frontier.
+## IFEval in context (strict-avg, models our size or larger)
+One metric for everyone: **strict-avg**. Open-model values are the **Open LLM Leaderboard "IFEval"** (which *is*
+strict-avg). Commercial rows are **Proprietary**; a **\*** means the strict value is an **estimate** (no official
+strict sub-metrics published — estimated from the model's published AVG4/prompt-strict, which run a few points above
+strict) and a **+** means the parameter count is unofficial. Our 61.6 is measured on our harness (calibrated above).
+
+| model | weights | company | released | params | IFEval strict-avg |
+| :-- | :-- | :-- | :-- | :-- | --: |
+| GPT-4o-mini | Proprietary | OpenAI | Jul 2024 | 8B + | 79 \* |
+| Llama-3.2-3B-Instruct | Open | Meta | Sep 2024 | 3.2B | 73.9 |
+| Qwen2.5-3B-Instruct | Open | Alibaba | Sep 2024 | 3.0B | 64.7 |
+| Phi-3-medium-4k-instruct | Open | Microsoft | May 2024 | 14.0B | 64.2 |
+| Mistral-Large | Proprietary | Mistral AI | Feb 2024 | 46.7B + | 63 \* |
+| **KOS-V4-Instruct (ours)** | **Open** | **Univ. of Kentucky** | **Jul 2026** | **3.0B** | **61.6** |
+| Yi-1.5-9B-Chat | Open | 01.AI | May 2024 | 8.8B | 60.5 |
+| Phi-3.5-mini-instruct | Open | Microsoft | Aug 2024 | 3.8B | 57.7 |
+| GPT-3.5-turbo-0613 | Proprietary | OpenAI | Jun 2023 | 20B + | 57 \* |
+| Command-R | Open | Cohere | Mar 2024 | 35B | 57 \* |
+| Phi-3-mini-4k-instruct | Open | Microsoft | Apr 2024 | 3.8B | 56.1 |
+| GPT-3.5-turbo-1106 | Proprietary | OpenAI | Nov 2023 | 20B + | 55.9 |
+| Mistral-7B-Instruct-v0.2 | Open | Mistral AI | Dec 2023 | 7.2B | 55.0 |
+| Gemini-1.0-Pro | Proprietary | Google | Dec 2023 | 30B + | 55 \* |
+| Mistral-Medium | Proprietary | Mistral AI | Dec 2023 | 100B + | 55 \* |
+| Mistral-7B-Instruct-v0.3 | Open | Mistral AI | May 2024 | 7.2B | 54.7 |
+| gemma-1.1-7b-it | Open | Google | Mar 2024 | 8.5B | 50.4 |
+| zephyr-7b-beta | Open | Hugging Face | Oct 2023 | 7.2B | 49.5 |
+| GPT-3.5-turbo-0125 | Proprietary | OpenAI | Jan 2024 | 20B + | 49 \* |
+| Llama-3.1-8B-Instruct | Open | Meta | Jul 2024 | 8.0B | 44.3 |
+| Qwen1.5-7B-Chat | Open | Alibaba | Jan 2024 | 7.7B | 43.7 |
+| Llama-2-13b-chat | Open | Meta | Jul 2023 | 13.0B | 39.8 |
+| Qwen1.5-4B-Chat | Open | Alibaba | Jan 2024 | 4.0B | 31.6 |
+
+**\* strict estimate** — no official IFEval strict sub-metrics are published for this model; the value is estimated
+from its published AVG4 or prompt-strict (loose metrics run ~2–4 pts above strict). **\+ unofficial params** — never
+disclosed by the provider (industry estimate: GPT-3.5 ~20B, GPT-4o-mini ~8B, Mistral-Large ~46.7B, Mistral-Medium
+~100B, Gemini-1.0-Pro ~30B).
+
+**Reading.** At 3B, KOS-V4 (61.6) beats every measured GPT-3.5-turbo snapshot of the original ChatGPT (1106 = 55.9,
+0125 ≈ 49, 0613 ≈ 57 est), plus Yi-1.5-9B, both Mistral-7Bs, gemma-1.1, zephyr, **Meta's Llama-3.1-8B (44.3)**,
+Llama-2-13b and the Qwen1.5 chats. Ahead of it: Llama-3.2-3B (73.9), Qwen2.5-3B (64.7), Phi-3-medium-14B (64.2), and
+the frontier proprietary models. This is a strong result **for a from-scratch 3B on 180B tokens and 24 GPUs**, not a
+claim to lead the current field.
+
+## Commercial baselines (strict-avg)
+| commercial model | company | snapshot | strict-avg | basis |
+| :-- | :-- | :-- | --: | :-- |
+| GPT-4 | OpenAI | gpt-4-0613 | 80.6 | computed from published strict sub-metrics (77.1 / 84.1), InternLM2 report |
+| Command-R+ | Cohere | c4ai-command-r-plus | 76.6 | strict sub-metrics (72.8 / 80.5), Open LLM Leaderboard raw |
+| GPT-3.5-turbo | OpenAI | gpt-3.5-turbo-1106 | 55.9 | strict sub-metrics (50.5 / 61.2), InternLM2 report |
+| Claude-3.5-Sonnet | Anthropic | 20240620 | ~83 \* | estimate from published AVG4 86.2 (Llama 3.1 report) |
+| GPT-4o | OpenAI | 2024-05-13 | ~81 \* | estimate from AVG4 84.3 |
+| Gemini-1.5-Pro | Google | May 2024 | ~79 \* | estimate from AVG4 82.3 |
+| Claude-3-Haiku | Anthropic | Mar 2024 | ~65 \* | estimate from AVG4 68.1 |
+
+**Sources.** IFEval definition: `google-research/instruction_following_eval`. Open-model strict-avg:
+`open-llm-leaderboard/contents` (its "IFEval" column). Commercial strict sub-metrics: InternLM2 report
+(arXiv:2403.17297) and Open LLM Leaderboard raw results. `~ *` rows are estimates from published AVG4 (strict runs a
+few points lower), clearly labeled.
 
 ## Compute footprint
-- **Pre-training:** 24× H200 (3 nodes, pure DDP), 5.73 days, ~3,300 H200-GPU-hours.
-- **SFT + RL (GRPO):** single 8-GPU node each.
+Pre-training: 24× H200 (3 nodes, DDP), 5.73 days, ~3,300 H200-GPU-hours. SFT + RL (GRPO): a single 8-GPU node each.
 
 ## Deployment (inference)
 | Precision | Approx. VRAM | Notes |
@@ -184,10 +205,7 @@ a claim to lead the current frontier.
 | INT8 / INT4 (approx.) | 4 / 2.5 GB | not shipped; typical for a 3 B model |
 
 ## Intended use & limitations
-- **Intended use:** general instruction following, structured output, and function/tool calling in clinical-adjacent
- workflows.
-- **Not a medical-knowledge QA model.** It follows instructions and calls tools; it does not reliably recall
- parametric medical facts and is not intended for medical multiple-choice exams. **Ground it with retrieval instead.**
-- **Below current small models.** On IFEval (strict-avg 61.6) and BFCL, newer small instruct models score higher; this
- model's results are notable for its data/compute budget, not for leading the field.
+- **Intended use:** general instruction following, structured output, and function/tool calling in clinical-adjacent workflows.
+- **Not a medical-knowledge QA model.** It follows instructions and calls tools; it does not reliably recall parametric medical facts. **Ground it with retrieval instead.**
+- **Below current small models.** On IFEval (strict-avg 61.6) and BFCL, newer small instruct models score higher; this model's results are notable for its data/compute budget, not for leading the field.
 - **English only.** Strong public-benchmark numbers are **not** validation on real clinical data.