Text Generation
Transformers
Safetensors
English
qwen3
medical
clinical
biomedical
instruction-following
tool-calling
function-calling
from-scratch
KOS-V5
weight-space-programming
task-arithmetic
model-editing
university-of-kentucky
university-of-louisville
ccts
conversational
text-generation-inference
Instructions to use Kentucky-Open-Science/KOS-V5.1-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Kentucky-Open-Science/KOS-V5.1-Instruct with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Kentucky-Open-Science/KOS-V5.1-Instruct") messages = [ {"role": "user", "content": "Who are you?"}, ] pipe(messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("Kentucky-Open-Science/KOS-V5.1-Instruct") model = AutoModelForCausalLM.from_pretrained("Kentucky-Open-Science/KOS-V5.1-Instruct", device_map="auto") messages = [ {"role": "user", "content": "Who are you?"}, ] inputs = tokenizer.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Kentucky-Open-Science/KOS-V5.1-Instruct with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Kentucky-Open-Science/KOS-V5.1-Instruct" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Kentucky-Open-Science/KOS-V5.1-Instruct", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Kentucky-Open-Science/KOS-V5.1-Instruct
- SGLang
How to use Kentucky-Open-Science/KOS-V5.1-Instruct with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Kentucky-Open-Science/KOS-V5.1-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Kentucky-Open-Science/KOS-V5.1-Instruct", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Kentucky-Open-Science/KOS-V5.1-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Kentucky-Open-Science/KOS-V5.1-Instruct", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Docker Model Runner
How to use Kentucky-Open-Science/KOS-V5.1-Instruct with Docker Model Runner:
docker model run hf.co/Kentucky-Open-Science/KOS-V5.1-Instruct
|
Download README.md from Kentucky-Open-Science/KOS-V5.1-Instruct: direct link, hf CLI and curl.
- Browser
- Download file 15.6 kB
-
https://huggingface.co/Kentucky-Open-Science/KOS-V5.1-Instruct/resolve/main/README.md
- Command line
-
hf download hf://Kentucky-Open-Science/KOS-V5.1-Instruct/README.md
-
curl -L -o README.md https://huggingface.co/Kentucky-Open-Science/KOS-V5.1-Instruct/resolve/main/README.md
15.6 kB
| license: apache-2.0 | |
| library_name: transformers | |
| pipeline_tag: text-generation | |
| language: | |
| - en | |
| developed_by: University of Kentucky (College of Medicine Office for Research; Center for Clinical and Translational Science) and University of Louisville (Kentucky Center for Digital Innovation) | |
| affiliations: | |
| - name: University of Kentucky, College of Medicine Office for Research | |
| url: https://medicine.uky.edu/sites/research | |
| - name: University of Kentucky, Center for Clinical and Translational Science (CCTS) | |
| url: https://www.ccts.uky.edu/ | |
| - name: University of Louisville, Kentucky Center for Digital Innovation | |
| url: https://centers.louisville.edu/kentucky-center-digital-innovation | |
| model_name: KOS-V5.1-Instruct | |
| model_codename: Catbird | |
| model_type: qwen3 | |
| base_model: Kentucky-Open-Science/KOS-V5-Instruct | |
| base_model_relation: merge | |
| tags: | |
| - medical | |
| - clinical | |
| - biomedical | |
| - instruction-following | |
| - tool-calling | |
| - function-calling | |
| - from-scratch | |
| - qwen3 | |
| - KOS-V5 | |
| - weight-space-programming | |
| - task-arithmetic | |
| - model-editing | |
| - university-of-kentucky | |
| - university-of-louisville | |
| - ccts | |
| # KOS-V5.1-Instruct Β· *"Catbird"* | |
| **Developed by** | |
| **University of Kentucky** | |
| - [College of Medicine, Office for Research](https://medicine.uky.edu/sites/research) | |
| - [Center for Clinical and Translational Science (CCTS)](https://www.ccts.uky.edu/) | |
| **University of Louisville** | |
| - [Kentucky Center for Digital Innovation](https://centers.louisville.edu/kentucky-center-digital-innovation) | |
| **KOS-V5.1-Instruct is the first KOS model built by weight-space programming.** Its released weights were written | |
| by arithmetic on weight vectors, not taken from a training checkpoint. They are KOS-V5-Instruct plus two task | |
| vectors, applied to chosen layers of the feed-forward network and kept only where measurement showed a gain. | |
| A configuration fix, described [below](#configuration-fix), comes with it. | |
| > β οΈ **Research use only.** This model is provided for research purposes only and must not be used for any | |
| > commercial, clinical, legal, or production-grade application. The user assumes all risks associated with its use. | |
| --- | |
| ## Weight-space programming | |
| ### The program | |
| ``` | |
| V5.1 = V5 + Ο_identity (MLP, all 36 layers) β Ο_replay (MLP, layers 12β23) | |
| ``` | |
| - **Ο_identity:** the weight change made by an identity fine-tune of KOS-V5-Instruct. | |
| - **Ο_replay:** the weight change made by the same fine-tune run on its general replay data alone. | |
| Adding Ο_identity installs the model's own identity. Subtracting Ο_replay removes the drift that the replay data | |
| causes. This is task arithmetic [3]. | |
| Only MLP weights change. Attention, embeddings and norms are bit-identical to KOS-V5-Instruct. The arithmetic is done | |
| in fp32 and stored in bf16. Weights sha256: `7492922a4a80e14d3974e83e275b53a011abd183afe591579a9a8509be961199`. | |
| ### Where to subtract the replay drift (a layer-wise finding) | |
| Subtracting Ο_replay was tested on three blocks of 12 layers: | |
| - **Layers 12β23 (this model):** removes the identity edit's hallucination-benchmark costs and keeps its tool-refusal | |
| and Med-HALT FCT gains. | |
| - **Layers 0β11:** deepens the Med-HALT FQT cost (β6.3). | |
| - **Layers 24β35:** gives the sharpest tool refusal, BFCL live_irrelevance +21.3. It keeps the Med-HALT costs. | |
| Merging the two edits evenly, the midpoint of the task-vector dial, also failed on Med-HALT. Extrapolating the | |
| post-SFT change on layers 4β11, ExPO-style [4], undid the tool-refusal gain. Both were rejected. | |
| ### How the program was selected | |
| - **Candidates:** fourteen. Ten weight programs, three configuration-only variants, and one generation setting. | |
| - **Baseline:** KOS-V5-Instruct as published. | |
| - **Decision battery:** 27 metrics. | |
| - BFCL: all 17 categories. | |
| - Med-HALT, the authors' code: FCT, NOTA, FQT. | |
| - AfriMed-QA MCQ, the authors' code, paper question set. | |
| - IFEval: 4 metrics. | |
| - TruthfulQA: MC1 and MC2. | |
| - **Gate:** no metric significantly worse than KOS-V5-Instruct. | |
| - **Statistics:** paired per item, exact McNemar or sign-flip tests, p < 0.05, following the paired-evaluation | |
| practice of [16]. | |
| - **Ranking:** by the number of metrics significantly better, then by summed deltas. The rule was fixed before any | |
| result was read. | |
| - **Cross-fit:** items were split in half by hash. Choosing on either half and scoring on the other gave a passing | |
| program both times. | |
| KOS-V5.1-Instruct is the only weight program that passed. It has **7 metrics significantly better and none worse**. | |
| ### Papers reviewed for this program | |
| Weight-space programming and its critics: | |
| 1. Takehi R., Teoh J., Zhang A., Khattab O. *Sparks of Weight-Space Programming: LLMs can natively read & write | |
| neural network weights.* MIT CSAIL blog, 2026-11-02. https://rikiyat.github.io/blog/weight-space-programming/ | |
| 2. Ahmed N. S. *Can an LLM Actually Reason and Program about Neural Network's Weights or is it just memorization on | |
| steroids? Exploring "Weight-Space Programming".* arXiv:2611.08070, 2026. | |
| Editing and merging weights: | |
| 3. Ilharco G., Ribeiro M. T., Wortsman M., Gururangan S., Schmidt L., Hajishirzi H., Farhadi A. *Editing Models with | |
| Task Arithmetic.* ICLR 2023. arXiv:2212.04089. | |
| 4. Zheng C., Wang Z., Ji H., Huang M., Peng N. *Model Extrapolation Expedites Alignment* (ExPO). ACL 2025. | |
| arXiv:2404.16792. | |
| 5. Yang E., Shen L., Guo G., Wang X., Cao X., Zhang J., Tao D. *Model Merging in LLMs, MLLMs, and Beyond: Methods, | |
| Theories, Applications and Opportunities.* arXiv:2408.07666, 2024. Companion list: | |
| https://github.com/EnnengYang/Awesome-Model-Merging-Methods-Theories-Applications | |
| 6. Jeong W., Lee W., Yoon K.-J. *Preference-Aligned LoRA Merging: Preserving Subspace Coverage and Addressing | |
| Directional Anisotropy.* CVPR 2026. arXiv:2603.26299. | |
| 7. Roy P. R., Roy B., Schneider K. A., Roy C. K. *A Unified Model for Cross-Domain Clone Detection via Model Merging.* | |
| ASE 2026. arXiv:2608.04215. | |
| 8. Chavan A., Sareen K., Tiwari R., Chandra A., Obando-Ceron J., Keutzer K., Niekum S., Dhillon I. S., Khatri D. | |
| *Towards Continual Learning: Mitigating Forgetting With Subspace Tuning.* arXiv:2610.22293, 2026. | |
| 9. Li Y., Yi X., Shi D., Cui Y., de Melo G., Wang L. *From Construction to Injection: Edit-Based Fingerprints for | |
| Large Language Models.* arXiv:2509.03122, 2025. | |
| Learning over weights (metanetworks): | |
| 10. Zhou A., Yang K., Jiang Y., Burns K., Xu W., Sokota S., Kolter J. Z., Finn C. *Neural Functional Transformers.* | |
| NeurIPS 2023. arXiv:2305.13546. | |
| 11. Wolanowsky Y., Bar-Shalom G., Frasca F., Maron H. *Graph Metanetworks for Transformer Based Language Models.* | |
| arXiv:2610.18471, 2026. | |
| 12. Dayan A., Eitan Y., Maron H. *CrossGMN: Graph Metanetworks for Cross-Architecture Weight-Space Transformations.* | |
| arXiv:2610.01649, 2026. | |
| Moving knowledge across tokenizers: | |
| 13. Minixhofer B., VuliΔ I., Ponti E. M. *Universal Cross-Tokenizer Distillation via Approximate Likelihood | |
| Matching.* NeurIPS 2025. arXiv:2503.20083. | |
| 14. Chen K., Lin C., Sun Y., Zhang W. *Dual-Vocabulary Language Model for Cross-Tokenizer Distillation.* | |
| arXiv:2609.33816, 2026. | |
| 15. Liu X., Yu W., Fredrikson M., Wang X., Gao J. *When the Same Coefficients Reach Different Places: Asymmetric | |
| Realizability in Transplanting Tokenizers across Large Language Models.* arXiv:2601.00065, 2025. First posted as | |
| *The Trojan in the Vocabulary: Stealthy Sabotage of LLM Composition.* | |
| Measurement: | |
| 16. Miller E. *Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations.* arXiv:2411.00640, | |
| 2024. | |
| ## Configuration fix | |
| KOS-V5-Instruct also shipped with a configuration defect. Its `config.json` stores the RoPE base only inside | |
| `rope_parameters`. vLLM 0.11 with transformers 4.57 reads the top-level `rope_theta`, finds none, and silently serves | |
| the model at ΞΈ = 10,000. The model was trained at ΞΈ = 25,000. We confirmed this by inspecting the rotary layer vLLM | |
| builds for each file. | |
| V5.1 adds the top-level `"rope_theta": 25000.0` and sets `use_cache: true`. Under vLLM this restores long context, | |
| shown in the RULER table below. Engines that read `rope_parameters` already ran KOS-V5-Instruct at 25,000, so the fix | |
| changes nothing for them. | |
| ## Results against KOS-V5-Instruct | |
| All comparisons below are paired runs on the same harnesses, measured on 2026-11-16. "V5" is KOS-V5-Instruct as | |
| published, and changes are in percentage points. | |
| Each benchmark uses its official package or the authors' released code. The only additions are the plumbing needed to | |
| reach a locally served model. For the AfriMed-QA paper question set (n = 3,910, the paper's Table 3), one later-added | |
| line of the released code that narrows the set to 160 questions is disabled. | |
| ### Identity (in-house probe, 15 held-out questions, greedy) | |
| | | V5 | V5.1 | | |
| | :--- | ---: | ---: | | |
| | Answers as KOS (University of Kentucky and University of Louisville) | 0 / 15 | **14 / 15** | | |
| | Claims another vendor made it (V5: almost always OpenAI) | 6 / 15 | **1 / 15** | | |
| V5, asked "Which company developed you?", answered "I am an AI language model developed by OpenAI." The cause was the | |
| V5 fine-tuning data. It contained the line "as an AI developed by OpenAI" about 1,800 times and never taught the model | |
| its own identity. | |
| ### Decision battery | |
| | Benchmark | V5 | Change | p | | |
| | :--- | ---: | ---: | ---: | | |
| | Med-HALT FCT (authors' code) | 8.1 | **+3.62** | 9e-106 | | |
| | BFCL live_irrelevance (bfcl-eval, unedited) | 43.1 | **+9.62** | 2e-13 | | |
| | Med-HALT NOTA | 60.8 | **+0.67** | 1e-5 | | |
| | BFCL irrelevance | 64.6 | **+7.08** | 2e-5 | | |
| | BFCL multi_turn_base | 1.0 | **+3.50** | 0.016 | | |
| | TruthfulQA MC2 (lm-eval) | 43.4 | **+0.31** | 0.016 | | |
| | BFCL parallel_multiple | 70.5 | **+5.00** | 0.041 | | |
| | IFEval, four metrics (lm-eval, chat template) | 66.7 prompt-strict | β1.8 to β2.2 | β₯ 0.085 | | |
| | Med-HALT FQT | 27.8 | β0.38 | 0.37 | | |
| | AfriMed-QA MCQ, paper set (authors' code) | 32.7 | +0.46 | 0.59 | | |
| | Other 12 BFCL categories, TruthfulQA MC1 | β | no significant change | β₯ 0.12 | | |
| ### Reasoning and knowledge | |
| | Benchmark | V5 | V5.1 | Change | p | | |
| | :--- | ---: | ---: | ---: | ---: | | |
| | BBH, 27 chain-of-thought tasks, 3-shot (lm-eval) | 40.52 | 42.60 | **+2.09** | 0.0004 | | |
| | MMLU-Pro, 14 subjects, 5-shot (lm-eval) | 29.36 | 30.23 | **+0.87** | 0.029 | | |
| | ASDiv chain-of-thought, 8-shot (lm-eval) | 38.3 | 41.0 | **+2.78** | 3e-5 | | |
| | HellaSwag, 10-shot | 46.7 | 46.9 | +0.21 | 0.011 | | |
| | MMLU, 5-shot | 46.9 | 46.7 | β0.14 | 0.31 | | |
| | GSM8K, 5-shot, strict | 52.1 | 51.6 | β0.45 | 0.62 | | |
| | MathQA, LSAT-AR, ARC, BoolQ, PIQA, WinoGrande, OpenBookQA, COPA, CommonsenseQA, LAMBADA | β | β | no significant change | β₯ 0.05 | | |
| On BBH, object counting rose +21.6, geometric shapes +18.4, multistep arithmetic +14.4 and formal fallacies +10.0. | |
| Web of lies fell β10.0. | |
| ### Long context (RULER, 12 stock lm-eval tasks, vLLM) | |
| | Context | V5 | V5.1 | | |
| | :--- | ---: | ---: | | |
| | 4K | 77.0 | **82.2** | | |
| | 8K | 61.9 | **71.5** | | |
| | 16K | 21.6 | **56.0** | | |
| | 24,576 (trained context) | 11.0 | **31.1** | | |
| Single-needle retrieval goes from 35β40% to 88β99% at 16K, and from 9β27% to 53β72% at 24,576. These gains come from | |
| the configuration fix. The weight program moves RULER by at most Β±2.6 points. Nothing is claimed beyond 24,576 tokens. | |
| ## Known regressions and limitations | |
| > Disclosed deliberately. | |
| - **Short-context RAG under noise (RGB, authors' code).** | |
| - **Losses:** against KOS-V5-Instruct served at ΞΈ = 10,000, noise robustness is β6.7 to β11.3 and information | |
| integration is β11.0. | |
| - **Gain:** negative rejection is +12.7. | |
| - **Cause:** the cost comes from the corrected RoPE base, not the weight program. The same weights at ΞΈ = 10,000 | |
| match V5 on RGB. | |
| - **Rejected fix:** a frequency-hybrid RoPE configuration recovered information integration but cost Med-HALT NOTA | |
| β6.3, so it was rejected. | |
| - **Three RULER tasks** lose with the configuration fix: SQuAD-QA at 4K (β6.1), multikey-3 at 8K (β15.8) and CWE at | |
| 8K (β9.5). | |
| - **IFEval** is 1.8 to 2.2 points lower than V5. The difference is not significant. | |
| - **Not a knowledge model.** MMLU is 46.7. Do not use it as a medical question-answering authority. | |
| - **MedXpertQA** is at chance level for both V5 and V5.1 (10.3 β 10.9, not significant). | |
| ## Optional setting: repetition penalty 1.05 | |
| Adding `"repetition_penalty": 1.05` to `generation_config.json` also passed the gate, with 6 metrics better and none | |
| worse. The pre-registered tie-break (count of significant gains) kept the shipped default. Against V5, this setting | |
| gives: | |
| - BFCL live_irrelevance +15.2 and irrelevance +9.6. | |
| - IFEval β0.1 to +0.7. | |
| - RGB noise robustness 4 to 6 points above V5.1's default. | |
| - RGB negative rejection 5.7 points below V5.1's default. | |
| Only engines that read the generation config apply it: vLLM's OpenAI server and transformers `generate`. | |
| ## Core specifications | |
| | Attribute | Detail | | |
| | :--- | :--- | | |
| | **Architecture** | Decoder-only Transformer (`Qwen3ForCausalLM`), Grouped-Query Attention | | |
| | **Parameters** | 3.715 B | | |
| | **Hidden / Layers** | 2560 / 36 | | |
| | **Attention** | 32 query / 8 KV heads (GQA 4:1), head_dim 128, per-head QK-RMSNorm | | |
| | **Feed-forward** | SwiGLU, intermediate 9728 | | |
| | **Vocabulary** | 32,000, custom medical byte-level BPE (unchanged from V5) | | |
| | **Context length** | 24,576 trained, 32,768 configured | | |
| | **Position encoding** | RoPE, ΞΈ = 25,000 (now in both `rope_theta` and `rope_parameters`) | | |
| | **Embeddings** | tied | | |
| | **Precision** | bfloat16 (7.43 GB, single shard) | | |
| ## Prompt / chat format (ChatML) | |
| The chat template and tokenizer are byte-identical to KOS-V5-Instruct. | |
| ``` | |
| <|im_start|>system | |
| You are a helpful assistant.<|im_end|> | |
| <|im_start|>user | |
| Write a haiku about Kentucky. Do not use any commas.<|im_end|> | |
| <|im_start|>assistant | |
| ``` | |
| Tool / function calling uses the model's native `<tools>` β¦ `</tools>` schema block and `<tool_call>` β¦ `</tool_call>` | |
| response format. Pass your function schemas via the tokenizer's `apply_chat_template(..., tools=[...])`. | |
| ## Quickstart | |
| ```python | |
| from transformers import AutoModelForCausalLM, AutoTokenizer | |
| model_id = "Kentucky-Open-Science/KOS-V5.1-Instruct" | |
| tok = AutoTokenizer.from_pretrained(model_id) | |
| model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto") | |
| messages = [{"role": "user", "content": "Write a haiku about Kentucky. Do not use any commas."}] | |
| inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device) | |
| out = model.generate(inputs, max_new_tokens=256, do_sample=False) | |
| print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True)) | |
| ``` | |
| vLLM: `vllm serve Kentucky-Open-Science/KOS-V5.1-Instruct --max-model-len 24576`. V5.1 runs at ΞΈ = 25,000 under vLLM | |
| with no overrides. | |
| ## Related models | |
| - [**KOS-V5-Instruct**](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Instruct): the model this release reprograms. | |
| - [**KOS-V5-Base**](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Base): the from-scratch pretrained foundation | |
| (3.72B, 235.2B tokens). | |
| - [**KOS-V5-Retriever**](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Retriever): a LoRA retrieval adapter | |
| built for KOS-V5-Instruct. It has not been tested with V5.1. | |
| ## Intended use & limitations | |
| Research use only. English only. Not for clinical, commercial, legal, or production-grade use. Outputs may be | |
| factually wrong or fabricated. This model must not be used to make or inform medical decisions. | |
| ## Naming | |
| The program is **KOS** (KOS-V1..V6). V5.1 is a weight-space-programmed release within the KOS-V5 generation. | |