---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
language:
- en
developed_by: University of Kentucky (College of Medicine Office for Research; Center for Clinical and Translational Science) and University of Louisville (Kentucky Center for Digital Innovation)
affiliations:
- name: University of Kentucky, College of Medicine Office for Research
url: https://medicine.uky.edu/sites/research
- name: University of Kentucky, Center for Clinical and Translational Science (CCTS)
url: https://www.ccts.uky.edu/
- name: University of Louisville, Kentucky Center for Digital Innovation
url: https://centers.louisville.edu/kentucky-center-digital-innovation
model_name: KOS-V5.1-Instruct
model_codename: Catbird
model_type: qwen3
base_model: Kentucky-Open-Science/KOS-V5-Instruct
base_model_relation: merge
tags:
- medical
- clinical
- biomedical
- instruction-following
- tool-calling
- function-calling
- from-scratch
- qwen3
- KOS-V5
- weight-space-programming
- task-arithmetic
- model-editing
- university-of-kentucky
- university-of-louisville
- ccts
---
# KOS-V5.1-Instruct · *"Catbird"*
**Developed by**
**University of Kentucky**
- [College of Medicine, Office for Research](https://medicine.uky.edu/sites/research)
- [Center for Clinical and Translational Science (CCTS)](https://www.ccts.uky.edu/)
**University of Louisville**
- [Kentucky Center for Digital Innovation](https://centers.louisville.edu/kentucky-center-digital-innovation)
**KOS-V5.1-Instruct is the first KOS model built by weight-space programming.** Its released weights were written
by arithmetic on weight vectors, not taken from a training checkpoint. They are KOS-V5-Instruct plus two task
vectors, applied to chosen layers of the feed-forward network and kept only where measurement showed a gain.
A configuration fix, described [below](#configuration-fix), comes with it.
> ⚠️ **Research use only.** This model is provided for research purposes only and must not be used for any
> commercial, clinical, legal, or production-grade application. The user assumes all risks associated with its use.
---
## Weight-space programming
### The program
```
V5.1 = V5 + τ_identity (MLP, all 36 layers) − τ_replay (MLP, layers 12–23)
```
- **τ_identity:** the weight change made by an identity fine-tune of KOS-V5-Instruct.
- **τ_replay:** the weight change made by the same fine-tune run on its general replay data alone.
Adding τ_identity installs the model's own identity. Subtracting τ_replay removes the drift that the replay data
causes. This is task arithmetic [3].
Only MLP weights change. Attention, embeddings and norms are bit-identical to KOS-V5-Instruct. The arithmetic is done
in fp32 and stored in bf16. Weights sha256: `7492922a4a80e14d3974e83e275b53a011abd183afe591579a9a8509be961199`.
### Where to subtract the replay drift (a layer-wise finding)
Subtracting τ_replay was tested on three blocks of 12 layers:
- **Layers 12–23 (this model):** removes the identity edit's hallucination-benchmark costs and keeps its tool-refusal
and Med-HALT FCT gains.
- **Layers 0–11:** deepens the Med-HALT FQT cost (−6.3).
- **Layers 24–35:** gives the sharpest tool refusal, BFCL live_irrelevance +21.3. It keeps the Med-HALT costs.
Merging the two edits evenly, the midpoint of the task-vector dial, also failed on Med-HALT. Extrapolating the
post-SFT change on layers 4–11, ExPO-style [4], undid the tool-refusal gain. Both were rejected.
### How the program was selected
- **Candidates:** fourteen. Ten weight programs, three configuration-only variants, and one generation setting.
- **Baseline:** KOS-V5-Instruct as published.
- **Decision battery:** 27 metrics.
- BFCL: all 17 categories.
- Med-HALT, the authors' code: FCT, NOTA, FQT.
- AfriMed-QA MCQ, the authors' code, paper question set.
- IFEval: 4 metrics.
- TruthfulQA: MC1 and MC2.
- **Gate:** no metric significantly worse than KOS-V5-Instruct.
- **Statistics:** paired per item, exact McNemar or sign-flip tests, p < 0.05, following the paired-evaluation
practice of [16].
- **Ranking:** by the number of metrics significantly better, then by summed deltas. The rule was fixed before any
result was read.
- **Cross-fit:** items were split in half by hash. Choosing on either half and scoring on the other gave a passing
program both times.
KOS-V5.1-Instruct is the only weight program that passed. It has **7 metrics significantly better and none worse**.
### Papers reviewed for this program
Weight-space programming and its critics:
1. Takehi R., Teoh J., Zhang A., Khattab O. *Sparks of Weight-Space Programming: LLMs can natively read & write
neural network weights.* MIT CSAIL blog, 2026-11-02. https://rikiyat.github.io/blog/weight-space-programming/
2. Ahmed N. S. *Can an LLM Actually Reason and Program about Neural Network's Weights or is it just memorization on
steroids? Exploring "Weight-Space Programming".* arXiv:2611.08070, 2026.
Editing and merging weights:
3. Ilharco G., Ribeiro M. T., Wortsman M., Gururangan S., Schmidt L., Hajishirzi H., Farhadi A. *Editing Models with
Task Arithmetic.* ICLR 2023. arXiv:2212.04089.
4. Zheng C., Wang Z., Ji H., Huang M., Peng N. *Model Extrapolation Expedites Alignment* (ExPO). ACL 2025.
arXiv:2404.16792.
5. Yang E., Shen L., Guo G., Wang X., Cao X., Zhang J., Tao D. *Model Merging in LLMs, MLLMs, and Beyond: Methods,
Theories, Applications and Opportunities.* arXiv:2408.07666, 2024. Companion list:
https://github.com/EnnengYang/Awesome-Model-Merging-Methods-Theories-Applications
6. Jeong W., Lee W., Yoon K.-J. *Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing
Directional Anisotropy.* CVPR 2026. arXiv:2603.26299.
7. Roy P. R., Roy B., Schneider K. A., Roy C. K. *A Unified Model for Cross-Domain Clone Detection via Model Merging.*
ASE 2026. arXiv:2608.04215.
8. Chavan A., Sareen K., Tiwari R., Chandra A., Obando-Ceron J., Keutzer K., Niekum S., Dhillon I. S., Khatri D.
*Towards Continual Learning: Mitigating Forgetting With Subspace Tuning.* arXiv:2610.22293, 2026.
9. Li Y., Yi X., Shi D., Cui Y., de Melo G., Wang L. *From Construction to Injection: Edit-Based Fingerprints for
Large Language Models.* arXiv:2509.03122, 2025.
Learning over weights (metanetworks):
10. Zhou A., Yang K., Jiang Y., Burns K., Xu W., Sokota S., Kolter J. Z., Finn C. *Neural Functional Transformers.*
NeurIPS 2023. arXiv:2305.13546.
11. Wolanowsky Y., Bar-Shalom G., Frasca F., Maron H. *Graph Metanetworks for Transformer Based Language Models.*
arXiv:2610.18471, 2026.
12. Dayan A., Eitan Y., Maron H. *CrossGMN: Graph Metanetworks for Cross-Architecture Weight-Space Transformations.*
arXiv:2610.01649, 2026.
Moving knowledge across tokenizers:
13. Minixhofer B., Vulić I., Ponti E. M. *Universal Cross-Tokenizer Distillation via Approximate Likelihood
Matching.* NeurIPS 2025. arXiv:2503.20083.
14. Chen K., Lin C., Sun Y., Zhang W. *Dual-Vocabulary Language Model for Cross-Tokenizer Distillation.*
arXiv:2609.33816, 2026.
15. Liu X., Yu W., Fredrikson M., Wang X., Gao J. *When the Same Coefficients Reach Different Places: Asymmetric
Realizability in Transplanting Tokenizers across Large Language Models.* arXiv:2601.00065, 2025. First posted as
*The Trojan in the Vocabulary: Stealthy Sabotage of LLM Composition.*
Measurement:
16. Miller E. *Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations.* arXiv:2411.00640,
2024.
## Configuration fix
KOS-V5-Instruct also shipped with a configuration defect. Its `config.json` stores the RoPE base only inside
`rope_parameters`. vLLM 0.11 with transformers 4.57 reads the top-level `rope_theta`, finds none, and silently serves
the model at θ = 10,000. The model was trained at θ = 25,000. We confirmed this by inspecting the rotary layer vLLM
builds for each file.
V5.1 adds the top-level `"rope_theta": 25000.0` and sets `use_cache: true`. Under vLLM this restores long context,
shown in the RULER table below. Engines that read `rope_parameters` already ran KOS-V5-Instruct at 25,000, so the fix
changes nothing for them.
## Results against KOS-V5-Instruct
All comparisons below are paired runs on the same harnesses, measured on 2026-11-16. "V5" is KOS-V5-Instruct as
published, and changes are in percentage points.
Each benchmark uses its official package or the authors' released code. The only additions are the plumbing needed to
reach a locally served model. For the AfriMed-QA paper question set (n = 3,910, the paper's Table 3), one later-added
line of the released code that narrows the set to 160 questions is disabled.
### Identity (in-house probe, 15 held-out questions, greedy)
| | V5 | V5.1 |
| :--- | ---: | ---: |
| Answers as KOS (University of Kentucky and University of Louisville) | 0 / 15 | **14 / 15** |
| Claims another vendor made it (V5: almost always OpenAI) | 6 / 15 | **1 / 15** |
V5, asked "Which company developed you?", answered "I am an AI language model developed by OpenAI." The cause was the
V5 fine-tuning data. It contained the line "as an AI developed by OpenAI" about 1,800 times and never taught the model
its own identity.
### Decision battery
| Benchmark | V5 | Change | p |
| :--- | ---: | ---: | ---: |
| Med-HALT FCT (authors' code) | 8.1 | **+3.62** | 9e-106 |
| BFCL live_irrelevance (bfcl-eval, unedited) | 43.1 | **+9.62** | 2e-13 |
| Med-HALT NOTA | 60.8 | **+0.67** | 1e-5 |
| BFCL irrelevance | 64.6 | **+7.08** | 2e-5 |
| BFCL multi_turn_base | 1.0 | **+3.50** | 0.016 |
| TruthfulQA MC2 (lm-eval) | 43.4 | **+0.31** | 0.016 |
| BFCL parallel_multiple | 70.5 | **+5.00** | 0.041 |
| IFEval, four metrics (lm-eval, chat template) | 66.7 prompt-strict | −1.8 to −2.2 | ≥ 0.085 |
| Med-HALT FQT | 27.8 | −0.38 | 0.37 |
| AfriMed-QA MCQ, paper set (authors' code) | 32.7 | +0.46 | 0.59 |
| Other 12 BFCL categories, TruthfulQA MC1 | — | no significant change | ≥ 0.12 |
### Reasoning and knowledge
| Benchmark | V5 | V5.1 | Change | p |
| :--- | ---: | ---: | ---: | ---: |
| BBH, 27 chain-of-thought tasks, 3-shot (lm-eval) | 40.52 | 42.60 | **+2.09** | 0.0004 |
| MMLU-Pro, 14 subjects, 5-shot (lm-eval) | 29.36 | 30.23 | **+0.87** | 0.029 |
| ASDiv chain-of-thought, 8-shot (lm-eval) | 38.3 | 41.0 | **+2.78** | 3e-5 |
| HellaSwag, 10-shot | 46.7 | 46.9 | +0.21 | 0.011 |
| MMLU, 5-shot | 46.9 | 46.7 | −0.14 | 0.31 |
| GSM8K, 5-shot, strict | 52.1 | 51.6 | −0.45 | 0.62 |
| MathQA, LSAT-AR, ARC, BoolQ, PIQA, WinoGrande, OpenBookQA, COPA, CommonsenseQA, LAMBADA | — | — | no significant change | ≥ 0.05 |
On BBH, object counting rose +21.6, geometric shapes +18.4, multistep arithmetic +14.4 and formal fallacies +10.0.
Web of lies fell −10.0.
### Long context (RULER, 12 stock lm-eval tasks, vLLM)
| Context | V5 | V5.1 |
| :--- | ---: | ---: |
| 4K | 77.0 | **82.2** |
| 8K | 61.9 | **71.5** |
| 16K | 21.6 | **56.0** |
| 24,576 (trained context) | 11.0 | **31.1** |
Single-needle retrieval goes from 35–40% to 88–99% at 16K, and from 9–27% to 53–72% at 24,576. These gains come from
the configuration fix. The weight program moves RULER by at most ±2.6 points. Nothing is claimed beyond 24,576 tokens.
## Known regressions and limitations
> Disclosed deliberately.
- **Short-context RAG under noise (RGB, authors' code).**
- **Losses:** against KOS-V5-Instruct served at θ = 10,000, noise robustness is −6.7 to −11.3 and information
integration is −11.0.
- **Gain:** negative rejection is +12.7.
- **Cause:** the cost comes from the corrected RoPE base, not the weight program. The same weights at θ = 10,000
match V5 on RGB.
- **Rejected fix:** a frequency-hybrid RoPE configuration recovered information integration but cost Med-HALT NOTA
−6.3, so it was rejected.
- **Three RULER tasks** lose with the configuration fix: SQuAD-QA at 4K (−6.1), multikey-3 at 8K (−15.8) and CWE at
8K (−9.5).
- **IFEval** is 1.8 to 2.2 points lower than V5. The difference is not significant.
- **Not a knowledge model.** MMLU is 46.7. Do not use it as a medical question-answering authority.
- **MedXpertQA** is at chance level for both V5 and V5.1 (10.3 → 10.9, not significant).
## Optional setting: repetition penalty 1.05
Adding `"repetition_penalty": 1.05` to `generation_config.json` also passed the gate, with 6 metrics better and none
worse. The pre-registered tie-break (count of significant gains) kept the shipped default. Against V5, this setting
gives:
- BFCL live_irrelevance +15.2 and irrelevance +9.6.
- IFEval −0.1 to +0.7.
- RGB noise robustness 4 to 6 points above V5.1's default.
- RGB negative rejection 5.7 points below V5.1's default.
Only engines that read the generation config apply it: vLLM's OpenAI server and transformers `generate`.
## Core specifications
| Attribute | Detail |
| :--- | :--- |
| **Architecture** | Decoder-only Transformer (`Qwen3ForCausalLM`), Grouped-Query Attention |
| **Parameters** | 3.715 B |
| **Hidden / Layers** | 2560 / 36 |
| **Attention** | 32 query / 8 KV heads (GQA 4:1), head_dim 128, per-head QK-RMSNorm |
| **Feed-forward** | SwiGLU, intermediate 9728 |
| **Vocabulary** | 32,000, custom medical byte-level BPE (unchanged from V5) |
| **Context length** | 24,576 trained, 32,768 configured |
| **Position encoding** | RoPE, θ = 25,000 (now in both `rope_theta` and `rope_parameters`) |
| **Embeddings** | tied |
| **Precision** | bfloat16 (7.43 GB, single shard) |
## Prompt / chat format (ChatML)
The chat template and tokenizer are byte-identical to KOS-V5-Instruct.
```
<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
Write a haiku about Kentucky. Do not use any commas.<|im_end|>
<|im_start|>assistant
```
Tool / function calling uses the model's native `` … `` schema block and `` … ``
response format. Pass your function schemas via the tokenizer's `apply_chat_template(..., tools=[...])`.
## Quickstart
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "Kentucky-Open-Science/KOS-V5.1-Instruct"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto")
messages = [{"role": "user", "content": "Write a haiku about Kentucky. Do not use any commas."}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=256, do_sample=False)
print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
```
vLLM: `vllm serve Kentucky-Open-Science/KOS-V5.1-Instruct --max-model-len 24576`. V5.1 runs at θ = 25,000 under vLLM
with no overrides.
## Related models
- [**KOS-V5-Instruct**](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Instruct): the model this release reprograms.
- [**KOS-V5-Base**](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Base): the from-scratch pretrained foundation
(3.72B, 235.2B tokens).
- [**KOS-V5-Retriever**](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Retriever): a LoRA retrieval adapter
built for KOS-V5-Instruct. It has not been tested with V5.1.
## Intended use & limitations
Research use only. English only. Not for clinical, commercial, legal, or production-grade use. Outputs may be
factually wrong or fabricated. This model must not be used to make or inform medical decisions.
## Naming
The program is **KOS** (KOS-V1..V6). V5.1 is a weight-space-programmed release within the KOS-V5 generation.