Card: Catbird logo + intro, official BFCL results

README.md

@@ -21,6 +21,8 @@ tags:
 - clinical
 - biomedical
 - instruction-following
+- tool-calling
+- function-calling
 - from-scratch
 - qwen3
 - KOS-V5
@@ -29,6 +31,10 @@ tags:
 - ccts
 ---
 
+<p align="center">
+  <img src="catbird_llm_logo.png" alt="Catbird" width="320"/>
+</p>
+
 # KOS-V5-Instruct · *"Catbird"*
 
 **Developed by**
@@ -40,6 +46,20 @@ tags:
 **University of Louisville**
 - [Kentucky Center for Digital Innovation](https://centers.louisville.edu/kentucky-center-digital-innovation)
 
+**A 3.72B-parameter medical language model trained from scratch.** It is not distilled, not pruned and not
+continued-pretrained from a general base. KOS-V5 (codename **Catbird**) is the fifth-generation Kentucky Open
+Science model line. This repository holds the **instruction-tuned head** of that line: the
+[KOS-V5-Base](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Base) pretraining checkpoint, taken through
+SFT and GRPO reinforcement learning.
+
+Unlike the base, this model **follows instructions**. It is the downstream SFT/RL artifact that
+[KOS-V5-Base](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Base) was built to initialise.
+
+**Code name: Catbird.** The KOS-V5 series is nicknamed Catbird, a native bird to Kentucky, the Gray
+Catbird (*Dumetella carolinensis*) is a medium-sized songbird famous for its distinct, cat-like "meow"
+call. This LLM was trained completely from scratch by teams from the University of Kentucky (Cat) and
+University of Louisville (Bird), so the code name is fitting.
+
 > ⚠️ **Research use only.** This model is provided for research purposes only and must not be used for any
 > commercial, clinical, legal, or production-grade application. The user assumes all risks associated with its use.
 
@@ -47,12 +67,8 @@ tags:
 
 ---
 
-**KOS-V5-Instruct** is a **3.72B-parameter instruction-following model trained completely from scratch** — not
-distilled, not pruned, not continued-pretrained from a general base. It is the instruction-tuned head of the
-**KOS-V5 "Catbird"** line — a **medical** model line whose base ranks **first of 17** at modelling held-out
-clinical text, built on [**KOS-V5-Base**](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Base)
-(235.2B tokens, single epoch). Its instruction ability comes from **GRPO reinforcement learning** against the
-*official* IFEval verifier.
+Its instruction ability comes from **GRPO reinforcement learning** against the *official* IFEval verifier, on a
+base that ranks **first of 17** at modelling held-out clinical text.
 
 **IFEval reported as strict-avg** = `(prompt-level strict + instruction-level strict) / 2` — the exact metric the
 Hugging Face Open LLM Leaderboard publishes as "IFEval."
@@ -64,9 +80,10 @@ Hugging Face Open LLM Leaderboard publishes as "IFEval."
 | **61.6** | [KOS-V4-Instruct](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Instruct) (previous generation) | University research team, 180B tokens, 24 GPUs |
 | 55.9 | GPT-3.5-turbo-1106 (the original ChatGPT) | OpenAI, ~10,000-GPU supercomputer |
 
-KOS-V5-Instruct **improves on KOS-V4-Instruct by +5.4 points of IFEval strict-avg (61.6 → 67.01)** and **+9.4
-points of MMLU (0.2782 → 0.3720)**, while clearing both the original GPT-3.5-turbo generation and the
-commercially trained Qwen2.5-3B.
+KOS-V5-Instruct **improves on KOS-V4-Instruct across every benchmark measured**: IFEval strict-avg
+**61.6 → 67.01** (+5.4), MMLU **0.2782 → 0.3720** (+9.4), and official BFCL function-calling
+**72.75/73.00/60.50 → 83.25/78.50/73.50** (+10.5 / +5.5 / +13.0). It clears both the original GPT-3.5-turbo
+generation and the commercially trained Qwen2.5-3B on instruction following.
 
 ## Core specifications
 
@@ -166,6 +183,25 @@ subjects are where it is strongest.
 | IFEval inst-loose | 0.7434 | 0.6882 | +0.055 |
 | **MMLU (57-subj, 5-shot, `acc`)** | **0.3720** | 0.2782 | **+0.094** |
 
+### Tool / function calling — official BFCL
+
+Measured with the **official `bfcl_eval`** suite in **FC (function-calling) mode**, non-live categories, the model
+prompted in its own native tool format and served via vLLM.
+
+| BFCL (official, FC mode) | **KOS-V5-Instruct** | KOS-V4-Instruct | Δ |
+| :-- | --: | --: | --: |
+| simple *(n=400)* | **83.25** | 72.75 | **+10.50** |
+| multiple *(n=200)* | **78.50** | 73.00 | **+5.50** |
+| parallel *(n=200)* | **73.50** | 60.50 | **+13.00** |
+
+**KOS-V5-Instruct improves on the published KOS-V4-Instruct on all three BFCL categories.** Tool calling was not
+a target of the GRPO objective — the reward ran the IFEval checker only — so this is carried capability from the
+SFT stage, not something the RL optimised for.
+
+> **Engine note.** These BFCL numbers come from the official `bfcl_eval` harness on a **vLLM** backend, whereas
+> the IFEval and MMLU figures on this card come from the HuggingFace backend of a pristine lm-evaluation-harness.
+> Both are official suites; they are not the same inference stack, and that is stated rather than blurred.
+
 **Cross-engine reproduction.** IFEval strict-avg measured **67.01** (vLLM backend) and **66.99** (HuggingFace
 backend, pristine harness) in two independent runs seven days apart — a **0.02-point** agreement across two
 inference engines and two harness builds. Both figures are reported; the difference is far below the benchmark's
@@ -224,10 +260,11 @@ English benchmarks, which understates their designed capability.
 
 > These are disclosed deliberately. A high IFEval score does not make this checkpoint production-ready.
 
-- **Tool / function calling: NOT EVALUATED.** No function-calling benchmark has been run on this checkpoint.
-  Unlike [KOS-V4-Instruct](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Instruct), which reports official
-  BFCL figures, **this model carries no tool-calling result at all** — do not assume the V4 numbers transfer.
-  Treat tool / function calling as **untested**.
+- **Tool / function calling is measured and improved** — see the BFCL table above. It does score below the
+  *SFT seed's* previously reported 87.0 on `multiple`; that 87.0 is a number our own records have since
+  declared **invalid** (it was inflated by benchmark contamination and collapsed to 7.0 once the affected
+  records were removed), so it is not a meaningful reference point. Against the **published** KOS-V4-Instruct
+  figures, this model is better on all three categories.
 - **Fabrication roughly doubled** on our internal gate (**14.4% → 28%**) during GRPO. That gate is itself under
   review because it measures on pretraining-distribution text, so the magnitude is uncertain — but the direction
   is a real concern and is the main reason this artifact is private.