Blame for README.md

---
license: apache-2.0
library_name: peft
pipeline_tag: feature-extraction
language:
- en
developed_by: University of Kentucky (College of Medicine Office for Research; Center for Clinical and Translational Science) and University of Louisville (Kentucky Center for Digital Innovation)
affiliations:
- name: University of Kentucky, College of Medicine Office for Research
  url: https://medicine.uky.edu/sites/research
- name: University of Kentucky, Center for Clinical and Translational Science (CCTS)
  url: https://www.ccts.uky.edu/
- name: University of Louisville, Kentucky Center for Digital Innovation
  url: https://centers.louisville.edu/kentucky-center-digital-innovation
model_name: KOS-V5-Retriever
model_codename: Catbird
model_type: qwen3
base_model: Kentucky-Open-Science/KOS-V5-Instruct
tags:
- medical
- clinical
- biomedical
- retrieval
- embeddings
- sentence-similarity
- dense-retriever
- llm2vec
- lora
- peft
- from-scratch
- qwen3
- KOS-V5
- university-of-kentucky
- university-of-louisville
- ccts
---

<p align="center">
  <img src="catbird_llm_logo.png" alt="Catbird" width="320"/>
</p>

# KOS-V5-Retriever · *"Catbird"*

**Developed by**

**University of Kentucky**
- [College of Medicine, Office for Research](https://medicine.uky.edu/sites/research)
- [Center for Clinical and Translational Science (CCTS)](https://www.ccts.uky.edu/)

**University of Louisville**
- [Kentucky Center for Digital Innovation](https://centers.louisville.edu/kentucky-center-digital-innovation)

> 🔎 **This repository is the retrieval / embedding adapter for [KOS-V5-Instruct](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Instruct).**
> It is a **~264 MB LoRA adapter** (PEFT, rank 32), *not* a standalone model. Loaded on top of the **unmodified** KOS-V5-Instruct weights (verified byte-identical — `model.safetensors` sha256 matches the published base), it turns the *generator* into a **dense text-retrieval embedding model**: same medical foundation, re-purposed to produce a vector per input for semantic search and RAG. Detach the adapter and you have the original generator back.

**KOS-V5 (codename *Catbird*)** is the fifth-generation Kentucky Open Science model line — a **3.72B-parameter medical language model trained from scratch** (not distilled, not pruned, not continued-pretrained from a general base). This adapter shares that foundation; the sections below cover **(1) what the adapter does and how well it retrieves**, then **(2) the base model it is built on**.

> ⚠️ **Research use only.** Provided for research purposes only; not for commercial, clinical, legal, or production-grade use. The user assumes all risks.

---

## 1 · Retrieval & embeddings — what this adapter does

The base is a **decoder** LLM (it generates text left-to-right). This adapter converts it into a **text encoder** (llm2vec-style) with three inference-time changes plus a small trained delta:

1. **Bidirectional attention** — the causal mask is replaced with a padding-only mask, so every token attends to the whole sequence.
2. **Mean pooling** — the sequence embedding is the attention-masked mean of the last hidden states, L2-normalized.
3. **LoRA (rank 32)** on all seven linear projections (`q,k,v,o,gate,up,down`), **contrastively trained** so related (query, passage) pairs land close and unrelated pairs far apart. The base weights are frozen; only the LoRA delta is learned.

The adapter does **not** generate text and does **not** alter the base's chat/tool behaviour — it is a separate, hot-swappable head for retrieval only.

### Results — official BEIR (SciFact)

Scored with the **official `beir` library** (`GenericDataLoader` + `EvaluateRetrieval` + **pytrec_eval** — the exact scorer behind the public BEIR/MTEB leaderboard), **zero-shot**: the training data explicitly **excludes SciFact**, verified clean (0 of 300 SciFact test queries appear in training).

| BEIR SciFact (NDCG@10, official) | value |
| :-- | --: |
| **KOS-V5-Retriever (this adapter)** | **0.7007** |
| Recall@10 | 0.8639 |
| BM25 (reference) | 0.665 |
| strong dense retrievers — GTR / E5 / BGE (reference) | 0.70–0.76 |

A legitimately **strong dense retriever** — it beats BM25 and sits in the strong-dense band, zero-shot, on a base a fraction the size of typical dense-retrieval models.

> **Scope / honesty.** This is **one** BEIR task (SciFact). It has **not yet** been evaluated on other BEIR tasks or MTEB; read 0.70 as a strong single-benchmark result, not a full retrieval profile. Broader evaluation (nfcorpus, fiqa, MTEB-medical) is planned. Contamination on other benchmarks is unchecked; **BEIR SciFact is zero-shot CLEAN.**

### Usage

All four moving parts (base + LoRA + bidirectional patch + mean-pool) must be applied together, or the number above will not reproduce. This snippet **is** the evaluation encoder.

```python
import torch, torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer
from peft import PeftModel

BASE    = "Kentucky-Open-Science/KOS-V5-Instruct"
ADAPTER = "Kentucky-Open-Science/KOS-V5-Retriever"   # this repo

# 1) bidirectional attention: replace Qwen3's causal mask with a padding-only mask
import transformers.models.qwen3.modeling_qwen3 as Q3
def _bidirectional(*args, **kwargs):
    ie = kwargs.get("input_embeds", args[1] if len(args) > 1 else None)
    am = kwargs.get("attention_mask", args[2] if len(args) > 2 else None)
    if am is None:
        return None
    dt = ie.dtype if ie is not None else torch.float32
    return (1.0 - am[:, None, None, :].to(dt)) * torch.finfo(dt).min
Q3.create_causal_mask = _bidirectional

# 2) base (frozen) + LoRA adapter
tok   = AutoTokenizer.from_pretrained(ADAPTER)
base  = AutoModel.from_pretrained(BASE, torch_dtype=torch.float32)
model = PeftModel.from_pretrained(base, ADAPTER).cuda().eval()

# 3) mean-pool + L2-normalize
@torch.no_grad()
def encode(texts, max_length=256):
    b = tok(texts, padding=True, truncation=True, max_length=max_length, return_tensors="pt").to("cuda")
    h = model(input_ids=b["input_ids"], attention_mask=b["attention_mask"]).last_hidden_state
    am = b["attention_mask"].unsqueeze(-1).float()
    return F.normalize((h * am).sum(1) / am.sum(1).clamp(min=1), dim=1)

docs    = encode(["Vitamin D deficiency is associated with increased risk of respiratory infection."])
queries = encode(["Does low vitamin D raise infection risk?"])
print(queries @ docs.T)   # cosine similarity; rank candidates by this
```

Retrieval is symmetric — use the same `encode` for queries and documents. The adapter's `meta.json` records the required config (`bidirectional: true`, `pooling: mean`).

### Training (this adapter)

- **Base:** `Kentucky-Open-Science/KOS-V5-Instruct` (frozen; byte-identical to the published weights).
- **Adapter:** PEFT LoRA rank 32, targets `q,k,v,o,gate,up,down`, bf16.
- **Objective:** contrastive (in-batch + hard negatives) over public (query, positive, negative) retrieval pairs, **with SciFact excluded** so the evaluation is zero-shot. No dataset content is distributed with this adapter.

### Retrieval limitations

- **Retrieval only.** Produces embeddings; it does **not** generate. For chat / instruction following / tool calling, use the [base model](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Instruct) without the adapter.
- **Single-benchmark evidence** (BEIR SciFact). Generalization is unproven.
- **Requires the exact inference recipe** (bidirectional patch + mean-pool). A plain causal `PeftModel` load will not reproduce the results.

---

## 2 · The base model — KOS-V5-Instruct

The rest of this card describes the model this adapter is built on. **These are the *base generator's* numbers** (instruction following, tool calling, medical QA); the adapter's own metric is the BEIR retrieval number above.

**A 3.72B-parameter medical language model trained from scratch.** KOS-V5 (codename **Catbird**) holds the **instruction-tuned head** of the line: the [KOS-V5-Base](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Base) pretraining checkpoint taken through SFT and two GRPO reinforcement-learning legs. Unlike the base, it **follows instructions and calls tools**.

**Code name: Catbird.** Native to Kentucky, the Gray Catbird is a songbird famous for its cat-like "meow"; trained from scratch by teams from the University of Kentucky (Cat) and University of Louisville (Bird).

### Core specifications

| Attribute | Detail |
| :--- | :--- |
| **Architecture** | Decoder-only Transformer (`Qwen3ForCausalLM`), Grouped-Query Attention |
| **Parameters** | 3.715 B |
| **Hidden / Layers** | 2560 / 36 |
| **Attention** | 32 query / 8 KV heads (GQA 4:1), head_dim 128, per-head QK-RMSNorm |
| **Feed-forward** | SwiGLU, intermediate 9728 |
| **Vocabulary** | 32,000, custom medical byte-level BPE |
| **Context length** | 32,768 |
| **Position encoding** | RoPE, θ = 25,000 |
| **Embeddings** | tied |
| **Precision** | bfloat16 (7.43 GB, single shard) |

### The medical foundation

**This is a medical model.** It inherits a base trained on a **54-source medical/biomedical corpus**. The strongest evidence is **bits-per-byte on held-out medical text** (tokenizer-agnostic). In a **17-model pool** — including BioMedLM, Meditron-7B, PMC-LLaMA-7B and MedGemma-4B — the KOS-V5 base ranks **1 of 17**:

| medical text (BPB, lower is better) | KOS-V5-Base | rank |
| :-- | --: | --: |
| **5-corpus mean, held-out medical text** | **0.4635** | **1 / 17** |
| clinical narratives | **0.4179** | **1 / 17** |
| radiology | **0.5132** | **1 / 17** |
| chest X-ray reports | **0.6688** | **1 / 17** |
| BIOSSES biomedical sentence similarity (Pearson / Spearman) | **0.7097 / 0.7014** | **1 / 17** |

Every comparator was trained on **1.3–153× more data**. See [KOS-V5-Base](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Base) for the full 96-metric evaluation.

### Base generation benchmarks (official suites)

EleutherAI lm-evaluation-harness `0.4.12.dev0` (commit `c1c4bea`), pristine clone, stock tasks; BFCL via the official `bfcl_eval` (FC mode, non-live AST) on vLLM.

| benchmark | KOS-V5-Instruct | KOS-V4-Instruct |
| :-- | --: | --: |
| **IFEval strict-avg** | **72.19** | 61.6 |
| **MMLU (57-subj, 5-shot, `acc`)** | **0.4512** | 0.2782 |
| medical-9 MMLU mean | **0.4915** | 0.2752 |
| PubMedQA / MedQA / MedMCQA | **0.706 / 0.380 / 0.365** | 0.686 / 0.282 / 0.278 |
| BFCL simple / multiple / parallel | **85.0 / 84.0 / 80.5** | 72.8 / 73.0 / 60.5 |
| RGB negative-rejection (official) | **57.33** | — |

IFEval is reported as strict-avg = `(prompt-strict + inst-strict) / 2`. On instruction following the base places **first among nine university-built instruct models** and above the original GPT-3.5-turbo generation; its BFCL tool-calling is **above the Qwen3-4B-Instruct-2507 peer**. The peer still leads on parametric knowledge (MMLU 0.7266) and raw IFEval (84.71). Forgetting control: OOD broad-holdout perplexity at **0.99×** the pre-RL base (no measurable forgetting).

> ⚠️ **Read the medical signal from BPB, not the MCQ scores.** KOS models place little probability mass on MCQ answer *letters*; the format, not the knowledge, is the bottleneck. A model ranking **1 of 17** at modelling clinical text while scoring modestly on multiple-choice is exhibiting exactly that gap.

### Data contamination (base)

- **IFEval: CLEAN (verbatim)** — 0 exact containments vs the 541 official test prompts.
- **BFCL: CLEAN (verbatim)** — 0 exact containments vs 5,437 official prompts.
- **MMLU / PubMedQA / MedQA / MedMCQA: UNCHECKED.**
- **Retrieval (this adapter), BEIR SciFact: CLEAN (zero-shot)** — see §1.

## Related models

- [**KOS-V5-Instruct**](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Instruct) — the base this adapter attaches to (3.72B, medical, instruction + tool-calling).
- [**KOS-V5-Base**](https://huggingface.co/Kentucky-Open-Science/KOS-V5-Base) — the from-scratch pretrained foundation (235.2B tokens).
- [**KOS-V4-Instruct**](https://huggingface.co/Kentucky-Open-Science/KOS-V4-Instruct) — previous generation.

## Intended use & limitations

Research use only. English only. Not for clinical, commercial, legal, or production-grade use. Retrieval outputs and any downstream results may be wrong or misleading; this model must not be used to make or inform medical decisions.

## Naming

The program is **KOS** (KOS-V1..V6); the V5 series codename is **Catbird**. Earlier internal names are not used.