README.md

---
license: apache-2.0
language:
- en
library_name: transformers
pipeline_tag: text-generation
developed_by: University of Kentucky (College of Medicine Office for Research; Center for Clinical and Translational Science) and University of Louisville (Kentucky Center for Digital Innovation)
affiliations:
- name: University of Kentucky, College of Medicine Office for Research
  url: https://medicine.uky.edu/sites/research
- name: University of Kentucky, Center for Clinical and Translational Science (CCTS)
  url: https://www.ccts.uky.edu/
- name: University of Louisville, Kentucky Center for Digital Innovation
  url: https://centers.louisville.edu/kentucky-center-digital-innovation
model_name: KOS-V5-Base
model_codename: Catbird
model_type: qwen3
tags:
- medical
- clinical
- biomedical
- radiology
- from-scratch
- base-model
- qwen3
- university-of-kentucky
- university-of-louisville
- ccts
---

<p align="center">
  <img src="catbird_llm_logo.png" alt="Catbird" width="320"/>
</p>

# KOS-V5-Base  ·  *"Catbird"*

**Developed by**

**University of Kentucky**
- [College of Medicine, Office for Research](https://medicine.uky.edu/sites/research)
- [Center for Clinical and Translational Science (CCTS)](https://www.ccts.uky.edu/)

**University of Louisville**
- [Kentucky Center for Digital Innovation](https://centers.louisville.edu/kentucky-center-digital-innovation)

**A 3.72B-parameter medical language model trained from scratch.** It is not distilled, not pruned and not
continued-pretrained from a general base. KOS-V5 (codename **Catbird**) is the fifth-generation
Kentucky Open Science base model. This repository holds the **final pretraining checkpoint**: the
complete single-epoch run over **235.2B tokens**.

This is a **base model**. It has had no instruction tuning, no RLHF, and no chat post-training. It
completes text; it does not follow instructions. It is the initialisation for a downstream SFT/RL
line, and it is released as a research artifact.

**Code name: Catbird.** The KOS-V5 series is nicknamed Catbird, a native bird to Kentucky, the Gray
Catbird (*Dumetella carolinensis*) is a medium-sized songbird famous for its distinct, cat-like "meow"
call. This LLM was trained completely from scratch by teams from the University of Kentucky (Cat) and
University of Louisville (Bird), so the code name is fitting.

> ⚠️ **Research use only.** This model is provided for research purposes only and must not be used for
> any commercial, clinical, legal, or production-grade applications. The user assumes all risks
> associated with its use.

---

## At a glance

| | |
|---|---|
| **Parameters** | 3.72B (36 layers × 2560 hidden, 32/8 GQA, head_dim 128, SwiGLU, **tied** embeddings) |
| **Training tokens** | **235.2B** (one complete epoch over a 54-source medical/biomedical corpus) |
| **Final train loss** | ≈1.16 |
| **Context** | trained at **24,576** tokens (whole-document); `max_position_embeddings` **32,768** |
| **RoPE** | standard 1D, **θ = 25,000** |
| **Tokenizer** | custom **32k byte-level BPE** ("v5-32k"), `add_bos_token=False`, single special token `<|endoftext|>` (id 0 = eos/bos/pad) |
| **Precision / stack** | bf16, 24×H200, DeepSpeed ZeRO-1, FlashAttention-2 |
| **Architecture class** | `Qwen3ForCausalLM` (stock, exports with no custom modelling code) |

---

## Evaluation

96 metrics across 19 tests; **94 ranked**. Scoring convention used throughout:

> **A metric is a WIN if KOS-V5 places in the top quartile of the pool (rank 1–4 of 17), and a LOSS if
> it places in the bottom quartile (rank 14–17 of 17). Ranks 5–13 are mid-pool, neither.**

**Result: 28 wins, 16 losses, 50 mid-pool.**

📖 **Every test is defined in [Test definitions](#test-definitions) below**: what it measures, on what
data, and which direction is better. The tier codes (T1, T1b, T2 …) used in the tables are all
explained there.

Every comparator was trained on **1.3–153× more data** than KOS-V5 (0.3–36T tokens vs our 0.235T).

### The comparator pool (16 external models + KOS-V5 = 17)

**Biomedical specialists (5)**

| model | tokens |
|---|---|
| BioMedLM | 300B (PubMed) |
| MedGemma-4B-PT | Gemma-3-4B 4T + medical image-text |
| Meditron-7B | Llama-2-7B 2T + 48B medical |
| AdaptLLM-Medicine-LLM | LLaMA-1-7B 1T + 2B adaptation |
| PMC-LLaMA-7B | LLaMA-7B 1T + 79B biomedical |

**General-domain (11)**

| model | tokens |
|---|---|
| Llama-3.2-1B / Llama-3.2-3B / Llama-3.1-8B | 9T / 9T / 15T |
| Gemma-3-1B-PT / Gemma-3-4B-PT | 2T / 4T |
| Gemma-2-9B | 8T |
| Qwen2.5-7B | 18T |
| Qwen3-4B-Base | 36T |
| StableLM-2-1.6B | 2T |
| MiMo-7B-Base | 25T |
| GigaChat3-10B-A1.8B-base | 5.5T (synthetic-heavy) |

*(KOS-V4, our own previous base, is held out of the ranking pool and reported separately below.)*

### Wins: top quartile, ranks 1-4 (28 metrics)

*Tier codes (T1, T2 …) are defined in [Test definitions](#test-definitions).*

| Metric | Value | Rank |
|---|---|---|
| **BPB, 5-corpus mean** (held-out medical text) | **0.4635** | **1 / 17** |
| BPB radiology | **0.5132** | **1 / 17** |
| BPB chest-xray | **0.6688** | **1 / 17** |
| BPB clinical narratives | **0.4179** | **1 / 17** |
| Collapsed-head fraction (T2) | **0.0069** | **1 / 17** |
| BOS-sink mass (T2) | **0.2541** | **1 / 17** |
| Min attention entropy (T2) | **0.4705** | **1 / 17** |
| Layers carrying a sink (T2) | 0.3330 | 2 / 17 |
| Dead-neuron rate (T3) | **0.0000** | **1 / 17** (tied) |
| Peak / final logit-lens accuracy (T3) | 0.5680 | 2 |
| RankMe, mean (T4) | **220.0** | **1 / 17** |
| WeightWatcher safe-layer ratio / under-ratio (T5) | 0.8346 / 0.1654 | 3 / 17 |
| BIOSSES Pearson / Spearman (T7) | **0.7097 / 0.7014** | **1 / 17** |
| **BLURB probe mean** (T7) | 0.7268 | 2 / 17 |
| MedThink-LL rank-1 (T6b) | 0.9200 | 2 / 17 |
| MedHALT fake-question rejection (T8-FQT) | 0.1122 | 2 / 17 |
| Long-context BPB @ 1,024 / 2,048 tok | **0.7290 / 0.6466** | **1 / 16** |
| Long-context BPB @ 512 / 4,096 tok | 0.8609 / 0.5809 | 3 / 4 of 16 |
| Tokens-per-byte efficiency (T1b) | 0.2580 | 4 / 17 |

Bits-per-byte is tokenizer-agnostic, so it is a fair cross-model number, and the distillation
confound in the pool **flatters the trillion-token externals, not a from-scratch model**. These BPB
placements are therefore conservative, not inflated.

### Losses: bottom quartile, ranks 14-17 (16 metrics)

*Tier codes are defined in [Test definitions](#test-definitions).*

| Metric | Value | Rank |
|---|---|---|
| **MedHALT false-confidence test (T8-FCT)** | **0.0280** | **16 / 17** (worst placement) |
| Single-token rate, aggregate (T1b-STRR) | 0.0481 | 15 / 17 |
| STRR, clinical / biomedical | 0.1257 / 0.0525 | 15 / 17 |
| STRR, radiology / chest-xray / textbooks / macro | 0.1921 / 0.2274 / 0.1693 / 0.1534 | 14 / 17 |
| Fragment score (T1b) | 2.5855 | 14 / 17 |
| RadGraph2-CXR micro-F1 (DR.3) | 0.9115 | 15 / 17 |
| HoC macro-F1 (T7) | 0.4889 | 14 / 17 |
| HellaSwag / PIQA (general reasoning) | 0.4638 / 0.7367 | 14 / 17 |
| Needle @ 512 / 1,024 tok (3 cells) | 0.90–0.95 | 14 / 16 |

The losses are **concentrated, not diffuse**: 7 of 16 are the single-token-rate/fragmentation family
(a deliberate tokenizer trade-off, see *Known issues*), 3 are short-context needle cells, and the rest
are MedHALT-FCT, two general-domain reasoning sets, one HoC probe and one RadGraph micro-F1.

### Mid-pool: neither a win nor a loss (50 metrics)

*Tier codes are defined in [Test definitions](#test-definitions).*

Reported because they are frequently mistaken for one or the other:

| Metric | Value | Rank |
|---|---|---|
| Calibration, mean ECE | 0.1209 | 9 / 17 |
| Long-context needle, 3-depth mean | 0.9333 | 11 / 16 |
| RadGraph-XL / CXR / RadGraph2 macro-F1 | 0.7463 / 0.7599 / 0.7661 | 7 / 8 / 10 of 17 |
| MedQA / MedMCQA / PubMedQA | 0.3339 / 0.3373 / 0.6680 | 12 / 13 / 12 of 17 |
| BPB biomedical-lit / textbooks | 0.1108 / 0.6066 | 13 / 11 of 17 |

Closed-book medical MCQ (mean 0.4352 over 16 tasks) sits mid-pool: below the frontier fleet, well
above KOS-V4. It is a post-training problem, not a token-budget one.

---

## Test definitions

What each tier actually measures. **"dir"** is the direction that is better. Every metric is ranked
against the same 17-model pool unless marked *reference-only*.

### Block A: general LM diagnostics (58 metrics)

#### T1 · Bits-per-byte on held-out medical text (the headline; dir: lower)
Negative log-likelihood divided by **UTF-8 bytes**, not tokens, so it is **tokenizer-agnostic** and
directly comparable across models with different vocabularies. Measured on **five held-out medical
corpora**, none of which appear in training: `radiology_reports`, `chest_xray_reports`,
`clinical_narratives`, `biomedical_literature`, `medical_textbooks`, plus their mean.
*Caveat we state against ourselves:* a model distilled from a larger teacher can print lower BPB
regardless of true quality, which **flatters the trillion-token externals, not a from-scratch model** , 
so our BPB placements are conservative.

#### T1b · Tokenizer statistics (dir: STRR higher; fragment & tokens-per-byte lower)
- **STRR** (*Single-Token Rate Ratio*): the fraction of medical terms the tokenizer encodes as **one
  token**, over a union vocabulary of **21,978 medical terms** across the 5 domains. Reported per
  domain, plus a macro mean and a union-weighted aggregate.
- **fragment_score**: average number of subword pieces per medical term.
- **tokens_per_byte**: raw compression efficiency (how compactly the tokenizer encodes text).

#### T2 · Attention pathology (dir: lower for sinks/collapse; higher for min-entropy)
Per-(layer, head) attention diagnostics over medical text:
- **collapsed_head_frac**: fraction of heads whose attention has degenerated to a near-constant pattern.
- **bos_sink_mass**: attention mass dumped on the first token (the classic "attention sink").
- **layers_with_sink**: fraction of layers exhibiting a sink.
- **entropy_norm_min**: the lowest normalised attention entropy in the model (higher = no head has
  collapsed to a single position).

#### T3 · Representation flow (dir: lens accuracy higher; dead-neuron lower)
- **peak_lens / final_lens accuracy**. Logit-lens probe accuracy, i.e. how well the correct token can be
  read out of intermediate layers (peak across depth, and at the final layer).
- **lens_AUC**: area under the depth-vs-accuracy curve.
- **max_dead_neuron_frac**: the worst layer's fraction of neurons that never activate.

#### T4 · Representation geometry (dir: RankMe higher; isotropy lower = less anisotropic)
Measured at **5 proportional depths**:
- **RankMe**: the *effective rank* (soft rank of the singular-value spectrum) of the hidden
  representations. Higher = the model uses more of its representational space rather than collapsing
  into a low-dimensional cone.
- **isotropy**: directional uniformity of the embedding cloud.

#### T5 · WeightWatcher power-law α (structural maturity)
Fits a power law to each layer's weight spectrum. **α ∈ [2, 6] is the healthy band**; α > 6 means
under-trained, α < 2 over-trained.
- **safe_layer_ratio** (dir: higher): fraction of layers inside the healthy band.
- **under_ratio** (dir: lower): fraction of layers that are under-trained.
- **median_α / min_α**. **Reference-only, deliberately unranked.** α is a *target band*, not a
  monotone quantity, so "more is better" is meaningless. (Ranking it higher-is-better would literally
  reward being under-trained, and would have *improved* our reported result from rank 15 to rank 3. We
  declined to take the favourable reading of an ill-defined metric.)

#### T9 · Calibration (dir: lower)
**Expected Calibration Error (ECE)**, averaged over **15 multiple-choice calibration tasks**: the gap
between the model's confidence and its actual accuracy. Low ECE = the model's confidence means something.

#### T10 · Context-length scaling + needle-in-a-haystack
- **Bucket BPB** (dir: lower): BPB measured at {L/8, L/4, L/2, L} where L = min(model's max position,
  4096), so every model is judged inside its own usable window. A separate KOS-only extension probes
  the full **32,768**-token window.
- **Needle retrieval** (dir: higher): a numeric medical fact is hidden at depth **0.1 / 0.5 / 0.9**
  inside a long context and must be retrieved, at fixed lengths **512 / 1,024 / 2,048 / 4,096** tokens.
  Reported per length, per depth, and as a 3-depth mean.

#### T11 · Demographic bias (dir: disparity lower; frac_stereotyped closer to 0.5)
**CrowS-Pairs**: for each minimally-different sentence pair (stereotyping vs anti-stereotyping), which
does the model prefer? `|disparity|` measures how strongly it prefers one; `frac_stereotyped` is the
fraction of pairs where the stereotyping sentence wins (0.5 = no preference).

### Block B: medical knowledge and entity extraction (32 metrics)

#### T6 · Closed-book multiple choice (dir: higher)
**16 log-likelihood MCQ tasks**, scored with the standard lm-evaluation-harness (no chain-of-thought,
no prompting tricks, this is a base model):
- *Medical specialist:* **MedQA** (4-option, USMLE-style), **MedMCQA**, **PubMedQA**, and the **8 MMLU
  medical subjects** (anatomy, clinical knowledge, college biology, college medicine, high-school
  biology, medical genetics, nutrition, professional medicine, virology).
- *Augmented / regional / expert:* **MedXpertQA-Text**, **AfriMed-QA**.
- *General-reasoning controls:* **PIQA**, **HellaSwag**.

#### T6b · MedThink-LL (dir: rank-1 higher; margin lower)
A PubMedQA-substitute: the correct answer must be ranked above **4 distractors** by log-likelihood.
**rank1_acc** = how often it is; **mean_logprob_margin** = the confidence gap.

#### T7 · BLURB linear probes (dir: higher)
A **linear probe** is trained on the frozen model's representations, this reads the *semantic geometry*
of the upper layers, independent of any generation ability:
- **BC5CDR-Chem**: chemical NER (macro / micro F1).
- **NCBI-Disease**: disease NER (macro / micro F1).
- **BIOSSES**: biomedical sentence-similarity (Pearson / Spearman).
- **HoC**: Hallmarks-of-Cancer document classification (micro / macro / samples F1).
- **BLURB mean**. The headline figure: the mean of BC5CDR-macro, NCBI-macro, BIOSSES-Spearman and HoC-micro.

#### T8 · Med-HALT hallucination resistance (dir: higher)
- **FCT** (*false-confidence test*): the question's **premise is wrong**; the model should decline.
  A base model has no refusal prior, which is exactly why this is our worst placement.
- **NOTA** (*none-of-the-above*): the correct answer is "none of the above".
- **FQT** (*fake-question test*): the question is fabricated; it should be rejected.
- **PMID2Title**: recall a paper's title from its PubMed ID (a memorisation probe; **degenerate** , 
  every model in the pool scores 0.0, so it is excluded from the tally).

### Block C: radiology-specific (6 metrics)

#### DR.1 / DR.2 / DR.3 · RadGraph entity + relation extraction (dir: higher)
Structured extraction of clinical entities and their relations from radiology reports, scored as
macro-F1 and micro-F1:
- **DR.1, RadGraph-XL**: chest CT + abdomen-pelvis CT + brain MRI.
- **DR.2, RadGraph-CXR**: chest x-ray (50 MIMIC + 50 CheXpert).
- **DR.3, RadGraph2-CXR**: adds the CHAN (change) relation class.

#### DPM.1 · PMR-Q&A (reference-only)
English physiotherapy / rehabilitation Q&A, rank-1 accuracy. **Unranked**: too few comparators have a
score for it to be a meaningful ranking.

### How ranks are computed

- **Pool = KOS-V5 + the 16 external models = 17.** KOS-V4 is *excluded* from the pool, it is our own
  prior model, and is reported separately in its own head-to-head below.
- Each metric is ranked **in its own correct direction** (lower-is-better for BPB, ECE, sinks,
  fragmentation; higher-is-better for accuracy, F1, RankMe, STRR).
- **Denominators vary** where a comparator has no score for a metric: Gemma-2-9B has no T10 result and
  BioMedLM is architecturally capped at 1,024 tokens (so it is skipped in the longer needle tables).
  That is why some ranks read "of 16" or "of 12" rather than "of 17", the rank is always taken over
  whichever models actually have that metric.
- **Two metrics are reference-only** (the WeightWatcher α pair) and **one is degenerate** (PMID2Title,
  where all 17 models score 0.0), leaving **93 metrics in the tally**.

---

## KOS-V5 vs KOS-V4: a corpus trade-off, now settled

Both models are ours, both are finished, and both were run on the identical harness. **KOS-V5 is
better on 17 of 32 compared metrics, KOS-V4 on 15.**

The token-budget alibi is dead: KOS-V5 finished on **235.2B tokens vs KOS-V4's 180.3B** (1.30×) at a
larger parameter count. **Every remaining KOS-V4 win is a win on the merits.**

| | KOS-V5 (final, 235B) | KOS-V4 (final, 180B) | Better |
|---|---|---|---|
| **Medical MCQ mean** (16 tasks) | **0.4352** | 0.3319 | **V5** (+0.103; 12 of 16 tasks) |
| MMLU clinical knowledge | **0.5057** | 0.3094 | **V5** (+0.196) |
| MMLU college medicine | **0.4335** | 0.2312 | **V5** (+0.202) |
| MMLU professional medicine | **0.3824** | 0.2096 | **V5** (+0.173) |
| BPB 5-corpus mean | 0.4635 | **0.4309** | **V4** |
| BPB radiology / chest-xray / clinical | 0.5132 / 0.6688 / 0.4179 | **0.4761 / 0.5887 / 0.3221** | **V4** |
| BPB biomedical-lit / textbooks | **0.1108 / 0.6066** | 0.1243 / 0.6432 | **V5** |
| RankMe (mean) | **220.0** | 170.8 | **V5** |
| Long-context BPB, 4k → 32k | **wins every window** |, | **V5** |
| Needle, 3-depth mean | **0.9333** | 0.8667 | **V5** |
| BLURB mean / RadGraph macro-F1 | 0.7268 / 0.75–0.77 | **0.7465 / 0.77–0.82** | **V4** |
| PubMedQA / MedThink rank-1 | 0.6680 / 0.9200 | **0.6980 / 0.9450** | **V4** |

**The honest reading.** KOS-V4's narrow radiology/clinical-dominant corpus still produces the better
**text model of radiology and clinical notes**. KOS-V5's broader 54-source corpus produces the better
**medical reasoner**: on the recall-heavy MCQ tasks V4 sat close to chance and V5 does not. This is a
**corpus-composition difference, not a maturity gradient**, and further pretraining was never going to
close it: V5 would have needed to recover 0.0326 BPB, and its entire final leg moved it by 0.0001.
Which model is "better" depends on the deployment.

---

## The run converged before it finished. Read this before choosing a checkpoint

The identical 16-tier battery was run at **six** checkpoints (90B / 120B / 150B / 190B / 220B / 235B).

| axis | 90B | 120B | 150B | **190B** | 220B | **235B (final)** |
|---|---|---|---|---|---|---|
| BPB (5-corpus, lower better) | .4890 | .4781 | .4740 | .4647 | .4636 | **.4635** |
| MCQ (16 tasks, higher better) | .3478 | .3737 | .4071 | **.4449** | .4332 | **.4352** |
| Needle (higher better) | .9333 | .9000 | .9500 | .9167 | .9000 | **.9333** |
| ECE (lower better) | .1847 | **.1135** | .1181 | .1378 | .1183 | **.1209** |

**On the final leg, not one headline axis improved.** BPB moved **+0.0001** (zero to three decimals).
MCQ **peaked at 190B and never recovered**: the final model ends **0.0097 below its own peak**, with
12 of 16 tasks below their peak-checkpoint value. Needle ended **exactly** where it began at 90B.
Calibration ended *worse* than the checkpoint before it. The learning-rate integral is 100% spent.

> **⚠️ The newest checkpoint is NOT the knowledge-best checkpoint.**
> The **190B** checkpoint is KOS-V5's knowledge-best, permanently. **Choose your fine-tuning
> initialisation by axis:** the 190B checkpoint for medical knowledge/MCQ; **this final checkpoint for
> BPB, long-context and general LM health.** Do not assume "newest = best."

Both MCQ moves (the 220B drop and the 235B bounce) sit inside MMLU-subset sampling noise. What
survives noise is the **plateau**: knowledge accuracy stopped improving 45B tokens before the end.

---

## Known issues

**1. The tokenizer does not split numbers, and this cannot be fixed in this model.**

The v5-32k BBPE has **no digit rule**: its pre-tokenizer is plain byte-level, so digit runs merge into
single tokens by BPE frequency.

| tokenizer | digit rule | multi-digit tokens | max digit-run | `10` / `100` / `1000` | tokens/number |
|---|---|---|---|---|---|
| **KOS-V5** | **none** | **963** | **5** | one token each | **1.80** |
| Qwen3-4B-Base | yes, **every digit split** | 0 | 1 | `1`·`0` / `1`·`0`·`0` / … | 4.20 |
| Llama-3.2-3B | yes, ≤3-digit groups | 1100 | 3 | `10` / `100` / `100`+`0` | 2.90 |

**The consequence is compositional:** `10`, `100` and `1000` are three *unrelated atomic symbols* that
share no substructure. The model cannot see that they differ by a factor of ten; it must learn each
magnitude as a separate lexical item, and rare numbers fragment on arbitrary BPE boundaries while
common ones do not. In medicine, doses, lab values, vitals, this is exactly the regime where
digit-level arithmetic and magnitude comparison are known to degrade.

**Stated with its limit:** this is a **measured property of the frozen tokenizer, not a measured
downstream failure.** Our evaluation suite contains **no arithmetic or numeric-reasoning tier**, so we
neither quantify the cost nor claim there is none. The tokenizer is baked into the trained (tied)
embedding table, so it is not fixable by fine-tuning, removing it requires retraining with a
digit-splitting pre-tokenizer. Treat any numeric medical task on this model with that in mind.

**2. Low medical single-token rate ([STRR](#t1b--tokenizer-statistics-dir-strr-higher-fragment--tokens-per-byte-lower): bottom quartile, rank 15/17).** Deliberate: BBPE spends its
32k vocabulary on byte-level compression rather than memorising whole medical terms. It single-tokens
only ~19% of radiology and ~23% of chest-xray terms. The trade appears sound, tokens-per-byte is a
top-quartile 4/17 and BPB is 1/17, but it is a trade, and it is the same trade as the digit issue.

**3. MedHALT-FCT ([false-confidence test](#t8--med-halt-hallucination-resistance-dir-higher)) is the worst placement in the model (rank 16/17).** A base model with no refusal prior
has essentially no mechanism for declining a question whose premise is false, and KOS-V5 answers
anyway. This is the axis a grounded medical model most needs to improve, and it is a target for the
instruction/RL stage rather than for more pretraining.

**4. Long-context needle retrieval never improved** across the entire run (mid-pool, 11/16). More
pretraining will not fix it; it is a post-training problem.

**5. Attention health is leading but degrading.** KOS-V5 still leads the pool on T2, but bos-sink mass
rose 0.1619 → 0.2541 and layers-carrying-a-sink rose 0.0830 → 0.3330 across training. The lead is real
but it shrank; it did not consolidate.

---

## Usage

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Kentucky-Open-Science/KOS-V5-Base"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")

prompt = "IMPRESSION: The chest radiograph demonstrates"
ids = tok(prompt, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=128, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))
```

### Notes

- **Works with `transformers` 4.x and 5.x out of the box.** Verified on 4.57.1 and 5.2.0: the snippet
  above runs unchanged on both.
- **Base model, no chat template.** It completes text. Prompt it as a completion model, or post-train it.
- **One special token.** `<|endoftext|>` (id 0) serves eos/bos/pad and the document separator.

---

## Intended use

A **research base model** for medical/biomedical NLP: the initialisation for domain SFT/RL, a subject
for interpretability and tokenizer/corpus research, and a from-scratch reference point against
distilled and continued-pretrained medical models. It is not instruction-tuned and is not a
question-answering system. Its closed-book medical MCQ accuracy sits mid-pool, and the known issues
above, especially the number tokenization, bear directly on any numeric medical task.

---

## Training details

| | |
|---|---|
| tokens / step | 1 × 8 × 24 × 24,576 = 4,718,592 |
| optimizer | AdamW β=(0.9, 0.95), grad-clip 1.0 |
| LR schedule | cosine → 0, peak 3.0e-4, 1% warmup (integral 100% spent) |
| batch | micro_batch 1 × grad_accum 8 × 24 GPUs, seq 24,576 (≈4.72M tokens/step) |
| corpus | 54 sources, English-only, deduped → decontaminated; whole documents (any document over 24,576 tokens is dropped, never split); neat-packed with a 4-D block-diagonal segment mask (no cross-document attention; position ids reset per document) |
| `tokenizer.json` md5 | `9c9df0404f6aae96dba5f3785e8b4c9d` |

Full 22-page evaluation report, all 96 metrics, all 17 models, every rank re-derived from source:
**`kos_v5_brief.pdf`**, included in this repository.

---

## Attribution

**University of Kentucky**
- [College of Medicine, Office for Research](https://medicine.uky.edu/sites/research)
- [Center for Clinical and Translational Science (CCTS)](https://www.ccts.uky.edu/)

**University of Louisville**
- [Kentucky Center for Digital Innovation](https://centers.louisville.edu/kentucky-center-digital-innovation)

## Citation

```bibtex
@misc{kos_v5_base_2026,
  title  = {KOS-V5-Base (Catbird): a from-scratch 3.72B medical language model},
  author = {Kentucky Open Science},
  year   = {2026},
  note   = {University of Kentucky (College of Medicine Office for Research;
            Center for Clinical and Translational Science) and
            University of Louisville (Kentucky Center for Digital Innovation)},
  url    = {https://huggingface.co/Kentucky-Open-Science/KOS-V5-Base}
}
```