codybum commited on
Commit
98d753a
·
verified ·
1 Parent(s): 445532e

KOS-V5.1-Instruct: config fix (top-level rope_theta 25000) + MLP task-arithmetic edit of KOS-V5-Instruct 7188bdc8

Browse files
README.md ADDED
@@ -0,0 +1,257 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: transformers
4
+ pipeline_tag: text-generation
5
+ language:
6
+ - en
7
+ developed_by: University of Kentucky (College of Medicine Office for Research; Center for Clinical and Translational Science) and University of Louisville (Kentucky Center for Digital Innovation)
8
+ affiliations:
9
+ - name: University of Kentucky, College of Medicine Office for Research
10
+ url: https://medicine.uky.edu/sites/research
11
+ - name: University of Kentucky, Center for Clinical and Translational Science (CCTS)
12
+ url: https://www.ccts.uky.edu/
13
+ - name: University of Louisville, Kentucky Center for Digital Innovation
14
+ url: https://centers.louisville.edu/kentucky-center-digital-innovation
15
+ model_name: KOS-V5.1-Instruct
16
+ model_codename: Catbird
17
+ model_type: qwen3
18
+ base_model: Kentucky-Open-Science/KOS-V5-Instruct
19
+ base_model_relation: merge
20
+ tags:
21
+ - medical
22
+ - clinical
23
+ - biomedical
24
+ - instruction-following
25
+ - tool-calling
26
+ - function-calling
27
+ - from-scratch
28
+ - qwen3
29
+ - KOS-V5
30
+ - task-arithmetic
31
+ - university-of-kentucky
32
+ - university-of-louisville
33
+ - ccts
34
+ ---
35
+
36
+ # KOS-V5.1-Instruct · *"Catbird"*
37
+
38
+ **Developed by**
39
+
40
+ **University of Kentucky**
41
+ - [College of Medicine, Office for Research](https://medicine.uky.edu/sites/research)
42
+ - [Center for Clinical and Translational Science (CCTS)](https://www.ccts.uky.edu/)
43
+
44
+ **University of Louisville**
45
+ - [Kentucky Center for Digital Innovation](https://centers.louisville.edu/kentucky-center-digital-innovation)
46
+
47
+ **KOS-V5.1-Instruct is a correction release of [KOS-V5-Instruct](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Instruct).**
48
+ It involves no new training run. It makes two changes to the KOS-V5-Instruct weights at commit `7188bdc82eb7`:
49
+
50
+ 1. **A configuration fix.** It restores the trained RoPE base (θ = 25,000) for engines that read the top-level key.
51
+ 2. **A weight edit by task arithmetic.** It changes the feed-forward (MLP) weights only.
52
+
53
+ Every change was kept only if it made the model measurably better than KOS-V5-Instruct, under a selection rule fixed
54
+ before any result was read.
55
+
56
+ > ⚠️ **Research use only.** This model is provided for research purposes only and must not be used for any
57
+ > commercial, clinical, legal, or production-grade application. The user assumes all risks associated with its use.
58
+
59
+ > 🔒 **Private research artifact.** This repository is private and is not a public release.
60
+
61
+ ---
62
+
63
+ ## What changed from KOS-V5-Instruct
64
+
65
+ **1. Configuration (`config.json`).**
66
+
67
+ - **The defect:** KOS-V5-Instruct's `config.json` stores the RoPE base only inside `rope_parameters`. vLLM 0.11 with
68
+ transformers 4.57 reads the top-level `rope_theta`, finds none, and silently serves the model at θ = 10,000. The
69
+ model was trained at θ = 25,000. We confirmed this by inspecting the rotary layer vLLM builds for each file.
70
+ - **The fix:** V5.1 adds the top-level `"rope_theta": 25000.0` and sets `use_cache: true`.
71
+ - **Who it affects:** engines that read `rope_parameters` already ran KOS-V5-Instruct at 25,000, so the fix changes
72
+ nothing for them.
73
+
74
+ **2. Weights (`model.safetensors`).** Only MLP weights change. Attention, embeddings and norms are bit-identical to
75
+ KOS-V5-Instruct.
76
+
77
+ ```
78
+ V5.1 = V5 + τ_identity (MLP, all 36 layers) − τ_replay (MLP, layers 12–23)
79
+ ```
80
+
81
+ - **τ_identity:** the weight change made by an identity fine-tune of KOS-V5-Instruct.
82
+ - **τ_replay:** the weight change made by the same fine-tune run on its general replay data alone.
83
+
84
+ Subtracting τ_replay removes the drift that the replay data causes. A layer-wise test showed where to apply it:
85
+
86
+ - **Layers 12–23 (chosen):** removes the identity edit's hallucination-benchmark costs and keeps its gains.
87
+ - **Layers 0–11:** worsens the hallucination costs.
88
+ - **Layers 24–35:** sharpens tool refusal but keeps the hallucination costs.
89
+
90
+ The edit was computed in fp32 and stored in bf16. Weights sha256:
91
+ `7492922a4a80e14d3974e83e275b53a011abd183afe591579a9a8509be961199`.
92
+
93
+ ## How it was selected
94
+
95
+ - **Candidates:** fourteen. Ten weight edits, three configuration-only variants, and one generation setting.
96
+ - **Baseline:** KOS-V5-Instruct as published.
97
+ - **Decision battery:** 27 metrics.
98
+ - BFCL: all 17 categories.
99
+ - Med-HALT, the authors' code: FCT, NOTA, FQT.
100
+ - AfriMed-QA MCQ, the authors' code, paper question set.
101
+ - IFEval: 4 metrics.
102
+ - TruthfulQA: MC1 and MC2.
103
+ - **Gate:** no metric significantly worse than KOS-V5-Instruct, paired per item, p < 0.05.
104
+ - **Ranking:** by the number of metrics significantly better, then by summed deltas.
105
+ - **Cross-fit:** items were split in half by hash. Choosing on either half and scoring on the other gave a passing
106
+ model both times.
107
+
108
+ KOS-V5.1-Instruct is the only weight edit that passed. It has **7 metrics significantly better and none worse**.
109
+
110
+ ## Results against KOS-V5-Instruct
111
+
112
+ All comparisons below are paired runs on the same harnesses, measured on 2026-11-16. "V5" is KOS-V5-Instruct as
113
+ published. Changes are in percentage points. Each benchmark uses its official package or the authors' released code.
114
+ The only additions are the plumbing needed to reach a locally served model. For the AfriMed-QA paper question set
115
+ (n = 3,910, the paper's Table 3), one later-added line of the released code that narrows the set to 160 questions is
116
+ disabled.
117
+
118
+ ### Decision battery
119
+
120
+ | Benchmark | V5 | Change | p |
121
+ | :--- | ---: | ---: | ---: |
122
+ | Med-HALT FCT (authors' code) | 8.1 | **+3.62** | 9e-106 |
123
+ | BFCL live_irrelevance (bfcl-eval, unedited) | 43.1 | **+9.62** | 2e-13 |
124
+ | Med-HALT NOTA | 60.8 | **+0.67** | 1e-5 |
125
+ | BFCL irrelevance | 64.6 | **+7.08** | 2e-5 |
126
+ | BFCL multi_turn_base | 1.0 | **+3.50** | 0.016 |
127
+ | TruthfulQA MC2 (lm-eval) | 43.4 | **+0.31** | 0.016 |
128
+ | BFCL parallel_multiple | 70.5 | **+5.00** | 0.041 |
129
+ | IFEval, four metrics (lm-eval, chat template) | 66.7 prompt-strict | −1.8 to −2.2 | ≥ 0.085 |
130
+ | Med-HALT FQT | 27.8 | −0.38 | 0.37 |
131
+ | AfriMed-QA MCQ, paper set (authors' code) | 32.7 | +0.46 | 0.59 |
132
+ | Other 12 BFCL categories, TruthfulQA MC1 | — | no significant change | ≥ 0.12 |
133
+
134
+ ### Reasoning and knowledge
135
+
136
+ | Benchmark | V5 | V5.1 | Change | p |
137
+ | :--- | ---: | ---: | ---: | ---: |
138
+ | BBH, 27 chain-of-thought tasks, 3-shot (lm-eval) | 40.52 | 42.60 | **+2.09** | 0.0004 |
139
+ | MMLU-Pro, 14 subjects, 5-shot (lm-eval) | 29.36 | 30.23 | **+0.87** | 0.029 |
140
+ | ASDiv chain-of-thought, 8-shot (lm-eval) | 38.3 | 41.0 | **+2.78** | 3e-5 |
141
+ | HellaSwag, 10-shot | 46.7 | 46.9 | +0.21 | 0.011 |
142
+ | MMLU, 5-shot | 46.9 | 46.7 | −0.14 | 0.31 |
143
+ | GSM8K, 5-shot, strict | 52.1 | 51.6 | −0.45 | 0.62 |
144
+ | MathQA, LSAT-AR, ARC, BoolQ, PIQA, WinoGrande, OpenBookQA, COPA, CommonsenseQA, LAMBADA | — | — | no significant change | ≥ 0.05 |
145
+
146
+ On BBH, object counting rose +21.6, geometric shapes +18.4, multistep arithmetic +14.4 and formal fallacies +10.0.
147
+ Web of lies fell −10.0.
148
+
149
+ ### Long context (RULER, 12 stock lm-eval tasks, vLLM)
150
+
151
+ | Context | V5 | V5.1 |
152
+ | :--- | ---: | ---: |
153
+ | 4K | 77.0 | **82.2** |
154
+ | 8K | 61.9 | **71.5** |
155
+ | 16K | 21.6 | **56.0** |
156
+ | 24,576 (trained context) | 11.0 | **31.1** |
157
+
158
+ Single-needle retrieval goes from 35–40% to 88–99% at 16K, and from 9–27% to 53–72% at 24,576. Nothing is claimed
159
+ beyond 24,576 tokens.
160
+
161
+ ## Known regressions and limitations
162
+
163
+ > Disclosed deliberately.
164
+
165
+ - **Short-context RAG under noise (RGB, authors' code).**
166
+ - **Losses:** against KOS-V5-Instruct served at θ = 10,000, noise robustness is −6.7 to −11.3 and information
167
+ integration is −11.0.
168
+ - **Gain:** negative rejection is +12.7.
169
+ - **Cause:** the cost comes from the RoPE base, not the weights. The same weights at θ = 10,000 match V5 on RGB.
170
+ - **Rejected fix:** a frequency-hybrid RoPE configuration recovered information integration but cost Med-HALT NOTA
171
+ −6.3, so it was rejected.
172
+ - **Three RULER tasks** lose with the configuration fix: SQuAD-QA at 4K (−6.1), multikey-3 at 8K (−15.8) and CWE at
173
+ 8K (−9.5).
174
+ - **IFEval** is 1.8 to 2.2 points lower than V5. The difference is not significant.
175
+ - **Self-identification.** Asked who made it, the model answers that it is "KOS-V5-Instruct" from the University of
176
+ Kentucky and the University of Louisville. The edit did not change that wording. Use a system prompt to state the
177
+ release name or the Kentucky Open Science Consortium.
178
+ - **Not a knowledge model.** MMLU is 46.7. Do not use it as a medical question-answering authority.
179
+ - **MedXpertQA** is at chance level for both V5 and V5.1 (10.3 → 10.9, not significant).
180
+
181
+ ## Optional setting: repetition penalty 1.05
182
+
183
+ Adding `"repetition_penalty": 1.05` to `generation_config.json` also passed the gate, with 6 metrics better and none
184
+ worse. The pre-registered tie-break (count of significant gains) kept the shipped default. Against V5, this setting
185
+ gives:
186
+
187
+ - BFCL live_irrelevance +15.2 and irrelevance +9.6.
188
+ - IFEval −0.1 to +0.7.
189
+ - RGB noise robustness 4 to 6 points above V5.1's default.
190
+ - RGB negative rejection 5.7 points below V5.1's default.
191
+
192
+ Only engines that read the generation config apply it: vLLM's OpenAI server and transformers `generate`.
193
+
194
+ ## Core specifications
195
+
196
+ | Attribute | Detail |
197
+ | :--- | :--- |
198
+ | **Architecture** | Decoder-only Transformer (`Qwen3ForCausalLM`), Grouped-Query Attention |
199
+ | **Parameters** | 3.715 B |
200
+ | **Hidden / Layers** | 2560 / 36 |
201
+ | **Attention** | 32 query / 8 KV heads (GQA 4:1), head_dim 128, per-head QK-RMSNorm |
202
+ | **Feed-forward** | SwiGLU, intermediate 9728 |
203
+ | **Vocabulary** | 32,000, custom medical byte-level BPE (unchanged from V5) |
204
+ | **Context length** | 24,576 trained, 32,768 configured |
205
+ | **Position encoding** | RoPE, θ = 25,000 (now in both `rope_theta` and `rope_parameters`) |
206
+ | **Embeddings** | tied |
207
+ | **Precision** | bfloat16 (7.43 GB, single shard) |
208
+
209
+ ## Prompt / chat format (ChatML)
210
+
211
+ The chat template and tokenizer are byte-identical to KOS-V5-Instruct.
212
+
213
+ ```
214
+ <|im_start|>system
215
+ You are a helpful assistant.<|im_end|>
216
+ <|im_start|>user
217
+ Write a haiku about Kentucky. Do not use any commas.<|im_end|>
218
+ <|im_start|>assistant
219
+ ```
220
+
221
+ Tool / function calling uses the model's native `<tools>` … `</tools>` schema block and `<tool_call>` … `</tool_call>`
222
+ response format. Pass your function schemas via the tokenizer's `apply_chat_template(..., tools=[...])`.
223
+
224
+ ## Quickstart
225
+
226
+ ```python
227
+ from transformers import AutoModelForCausalLM, AutoTokenizer
228
+
229
+ model_id = "Kentucky-Open-Science/KOS-V5.1-Instruct"
230
+ tok = AutoTokenizer.from_pretrained(model_id)
231
+ model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto")
232
+
233
+ messages = [{"role": "user", "content": "Write a haiku about Kentucky. Do not use any commas."}]
234
+ inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
235
+ out = model.generate(inputs, max_new_tokens=256, do_sample=False)
236
+ print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
237
+ ```
238
+
239
+ vLLM: `vllm serve Kentucky-Open-Science/KOS-V5.1-Instruct --max-model-len 24576`. V5.1 runs at θ = 25,000 under vLLM
240
+ with no overrides.
241
+
242
+ ## Related models
243
+
244
+ - [**KOS-V5-Instruct**](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Instruct): the release this model corrects.
245
+ - [**KOS-V5-Base**](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Base): the from-scratch pretrained foundation
246
+ (3.72B, 235.2B tokens).
247
+ - [**KOS-V5-Retriever**](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Retriever): a LoRA retrieval adapter
248
+ built for KOS-V5-Instruct. It has not been tested with V5.1.
249
+
250
+ ## Intended use & limitations
251
+
252
+ Research use only. English only. Not for clinical, commercial, legal, or production-grade use. Outputs may be
253
+ factually wrong or fabricated. This model must not be used to make or inform medical decisions.
254
+
255
+ ## Naming
256
+
257
+ The program is **KOS** (KOS-V1..V6). V5.1 is a correction release within the KOS-V5 generation.
chat_template.jinja ADDED
@@ -0,0 +1,61 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {%- if tools %}
2
+ {{- '<|im_start|>system\n' }}
3
+ {%- if messages[0].role == 'system' %}
4
+ {{- messages[0].content + '\n\n' }}
5
+ {%- endif %}
6
+ {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
7
+ {%- for tool in tools %}
8
+ {{- "\n" }}
9
+ {{- tool | tojson }}
10
+ {%- endfor %}
11
+ {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
12
+ {%- else %}
13
+ {%- if messages[0].role == 'system' %}
14
+ {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
15
+ {%- endif %}
16
+ {%- endif %}
17
+ {%- for message in messages %}
18
+ {%- if message.content is string %}
19
+ {%- set content = message.content %}
20
+ {%- else %}
21
+ {%- set content = '' %}
22
+ {%- endif %}
23
+ {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
24
+ {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
25
+ {%- elif message.role == "assistant" %}
26
+ {{- '<|im_start|>' + message.role + '\n' + content }}
27
+ {%- if message.tool_calls %}
28
+ {%- for tool_call in message.tool_calls %}
29
+ {%- if (loop.first and content) or (not loop.first) %}
30
+ {{- '\n' }}
31
+ {%- endif %}
32
+ {%- if tool_call.function %}
33
+ {%- set tool_call = tool_call.function %}
34
+ {%- endif %}
35
+ {{- '<tool_call>\n{"name": "' }}
36
+ {{- tool_call.name }}
37
+ {{- '", "arguments": ' }}
38
+ {%- if tool_call.arguments is string %}
39
+ {{- tool_call.arguments }}
40
+ {%- else %}
41
+ {{- tool_call.arguments | tojson }}
42
+ {%- endif %}
43
+ {{- '}\n</tool_call>' }}
44
+ {%- endfor %}
45
+ {%- endif %}
46
+ {{- '<|im_end|>\n' }}
47
+ {%- elif message.role == "tool" %}
48
+ {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
49
+ {{- '<|im_start|>user' }}
50
+ {%- endif %}
51
+ {{- '\n<tool_response>\n' }}
52
+ {{- content }}
53
+ {{- '\n</tool_response>' }}
54
+ {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
55
+ {{- '<|im_end|>\n' }}
56
+ {%- endif %}
57
+ {%- endif %}
58
+ {%- endfor %}
59
+ {%- if add_generation_prompt %}
60
+ {{- '<|im_start|>assistant\n' }}
61
+ {%- endif %}
config.json ADDED
@@ -0,0 +1,72 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Qwen3ForCausalLM"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": null,
8
+ "dtype": "bfloat16",
9
+ "eos_token_id": 0,
10
+ "head_dim": 128,
11
+ "hidden_act": "silu",
12
+ "hidden_size": 2560,
13
+ "initializer_range": 0.02,
14
+ "intermediate_size": 9728,
15
+ "layer_types": [
16
+ "full_attention",
17
+ "full_attention",
18
+ "full_attention",
19
+ "full_attention",
20
+ "full_attention",
21
+ "full_attention",
22
+ "full_attention",
23
+ "full_attention",
24
+ "full_attention",
25
+ "full_attention",
26
+ "full_attention",
27
+ "full_attention",
28
+ "full_attention",
29
+ "full_attention",
30
+ "full_attention",
31
+ "full_attention",
32
+ "full_attention",
33
+ "full_attention",
34
+ "full_attention",
35
+ "full_attention",
36
+ "full_attention",
37
+ "full_attention",
38
+ "full_attention",
39
+ "full_attention",
40
+ "full_attention",
41
+ "full_attention",
42
+ "full_attention",
43
+ "full_attention",
44
+ "full_attention",
45
+ "full_attention",
46
+ "full_attention",
47
+ "full_attention",
48
+ "full_attention",
49
+ "full_attention",
50
+ "full_attention",
51
+ "full_attention"
52
+ ],
53
+ "max_position_embeddings": 32768,
54
+ "max_window_layers": 36,
55
+ "model_type": "qwen3",
56
+ "num_attention_heads": 32,
57
+ "num_hidden_layers": 36,
58
+ "num_key_value_heads": 8,
59
+ "pad_token_id": 0,
60
+ "rms_norm_eps": 1e-06,
61
+ "rope_parameters": {
62
+ "rope_theta": 25000.0,
63
+ "rope_type": "default"
64
+ },
65
+ "sliding_window": null,
66
+ "tie_word_embeddings": true,
67
+ "transformers_version": "5.3.0",
68
+ "use_cache": true,
69
+ "use_sliding_window": false,
70
+ "vocab_size": 32000,
71
+ "rope_theta": 25000.0
72
+ }
generation_config.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "eos_token_id": [
4
+ 0
5
+ ],
6
+ "output_attentions": false,
7
+ "output_hidden_states": false,
8
+ "pad_token_id": 0,
9
+ "transformers_version": "5.2.0",
10
+ "use_cache": true
11
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7492922a4a80e14d3974e83e275b53a011abd183afe591579a9a8509be961199
3
+ size 7430909696
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": null,
3
+ "eos_token": "<|im_end|>",
4
+ "model_max_length": 32768,
5
+ "pad_token": "<|im_end|>",
6
+ "padding_side": "right",
7
+ "split_special_tokens": false,
8
+ "tokenizer_class": "PreTrainedTokenizerFast",
9
+ "unk_token": "<|im_end|>"
10
+ }