Klein ML Website  |  Demo & Examples  |  KV1.cpp  |  GGUF

KV1-1.2B-Instruct

~6.4x smaller than FP16 (ideal) | 75.8% of FP16 intelligence retained

We introduce the KV1 model family: language models trained to run with a 2-bit KV cache. KV1-1.2B-Instruct is the first release; 2.6B is in training, and 4B, 8B-A1B and 24B-A2B versions are planned.

KV1-1.2B-Instruct is LFM2.5-1.2B-Instruct after KV-cache quantization-aware training (QAT). The weights stay in bf16. Only the KV cache is quantized, to 2.5 bits per value, which makes it 6.4× smaller than fp16. Without training, the same 2-bit cache breaks the model (GSM8K 1.6%, HumanEval 0.6%). KV1 recovers most of that quality:

• Self-distillation: the frozen original model with an fp16 KV cache is the teacher; the student is the same model running with the 2-bit cache. Loss = KL(teacher ‖ student) on the full next-token distribution at every position + 0.5 × normalized per-layer hidden-state MSE. The dataset text is only input; the target is always the teacher, so the model keeps its own behavior.
• Straight-through estimator: the forward pass uses the rounded K/V, the backward pass treats rounding as identity.
• Hadamard rotation of K/V before quantizing spreads outlier channels over all 64 dimensions.
• Attention sinks: the first 4 tokens of every sequence keep fp16 K/V.
• Training: 750M tokens at 32k context. Input mix: chat 60% (UltraData-SFT-2605), agent / tool-use trajectories 16% (UltraData-SFT-Agent-2609), code 12% (UltraData-Code), web 7% (Ultra-FineWeb), math 5% (UltraData-Math). AdamW, peak LR 2e-5 with cosine decay, a 100-step ramp from fp16 to 2-bit, one RTX Pro 6000 (~17 GPU hours).

Resources

• Demo
• KV1.cpp: our llama.cpp fork with the bq2_0 KV cache type. BQ2_0 is not in mainline llama.cpp yet.
• GGUF files: BF16, Q8_0 and Q4_K_M, ready for KV1.cpp.

Model Overview

Item	Specification
Base model	LFM2.5-1.2B-Instruct
Parameters	1.17B
Architecture	Hybrid LFM2: 10 gated short-convolution blocks + 6 GQA attention blocks
Attention / KV heads	32 query heads, 8 KV heads, head dim 64
Context length	32,768 tokens
Vocab size	65,536
Weights	bf16 (unchanged format)
KV cache	BQ2_0: 2-bit affine + Hadamard + 4 fp16 sink tokens, 2.5 bits/value
Packed BQ2_0 KV cache at 32K	60 MiB (fp16: 384 MiB)
License	LFM Open License v1.0 (follows the base model)

Quantization Format: KV1.cpp BQ2_0

The cache is quantized per token, per KV head, over the 64 values of one head (post-RoPE K and V).

#define QK_B2 64                 // one block = one head vector (head_dim 64)
typedef struct {
    ggml_half d;                 // scale = (max - min) / 3
    ggml_half m;                 // min
    uint8_t   qs[QK_B2 / 4];     // 2-bit codes, 4 per byte: q0 | q1<<2 | q2<<4 | q3<<6
} block_bq2_0;                   // 20 bytes per 64 values = 2.5 bits/value

• Quantize (in fp32): d = fp16(max((max - min) / 3, 1e-6)), m = fp16(min), q = clamp(round_half_to_even((x - m) / d), 0, 3). The fp16-rounded m and d are used for the codes.
• Dequantize: x̂ = q · d + m.
• Hadamard rotation: H is the normalized 64×64 Sylvester Hadamard matrix, so H·H = I. Q, K and V are multiplied by H after RoPE, before K and V are stored. Scores are unchanged (q·k = qH·kH), and the attention output is multiplied by H again per head before the output projection.
• Sinks: positions 0–3 of every sequence keep full-precision K/V.

Memory

Only the 6 attention blocks have a KV cache; the convolution blocks keep a small constant state.

Context	fp16 KV	3-bit KV	BQ2_0 (KV1)
per token	12 KiB	2.6 KiB	1.9 KiB
8K	96 MiB	21 MiB	15 MiB
32K	384 MiB	84 MiB	60 MiB

Per sequence. With several sequences in parallel, the saving scales with the batch size. The bf16 weights (~2.3 GB) are not affected.

Quickstart

Transformers

The repository includes remote code that runs the attention layers with the trained 2-bit KV quantization (rotation, sinks, rounding exactly as in training):

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "klein-ml/KV1-1.2B"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, dtype="bfloat16", device_map="cuda")

msgs = [{"role": "user", "content": "Write a Python function that checks if a number is prime."}]
inputs = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt", return_dict=True).to("cuda")
out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

model.set_kv_quant(False)   # optional: back to a normal fp16 KV cache

In transformers each token's K/V is quantized once when it enters the cache, but the cache is still stored in bf16. This gives the exact 2-bit outputs, but not the memory saving. For the real 2.5-bit cache, use KV1.cpp. Without trust_remote_code the model loads as a plain LFM2 model with an fp16 cache.

KV1.cpp

# build KV1.cpp (drop -DGGML_CUDA=ON for a CPU-only build)
git clone https://github.com/Klein-ML/kv1.cpp && cd kv1.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release && cmake --build build -j

# get the GGUF (BF16 = weights as released, Q8_0 = half the size, Q4_K_M = smallest)
hf download klein-ml/KV1-1.2B-GGUF KV1-1.2B-Q8_0.gguf --local-dir .

# chat with the 2-bit KV cache
./build/bin/llama-cli -m KV1-1.2B-Q8_0.gguf -fa on -ctk bq2_0 -ctv bq2_0 -ngl 99 -c 32768

# or an OpenAI-compatible server
./build/bin/llama-server -m KV1-1.2B-Q8_0.gguf -fa on -ctk bq2_0 -ctv bq2_0 -ngl 99 -c 32768

To make the GGUF yourself: python convert_hf_to_gguf.py /path/to/KV1-1.2B --outtype bf16 --outfile KV1-1.2B-BF16.gguf.

-ctk bq2_0 -ctv bq2_0 must be used together, with -fa on. In this first version KV1.cpp unpacks the 2-bit cache to a temporary f16 buffer for attention (reused across layers, about 64 MiB at 32K context), so peak memory is higher than the 60 MiB cache itself.

Benchmarks

All configs use the same LFM2.5-1.2B-Instruct weights, except KV1, which uses the KV1 weights. Only the KV cache differs. Scores are from lm-eval with the chat template. 3-bit and untrained 2-bit are the original model with a plain quantized cache and no training.

Benchmark	fp16 KV (16 bits)	3-bit KV (3.5 bits)	2-bit KV, untrained (2.5 bits)	KV1 BQ2_0 (2.5 bits)	KV1 retention
IFEval (prompt strict)	81.89	78.93	43.07	77.63	94.8%
GSM8K (5-shot)	67.60	56.40	1.60	44.80	66.3%
HumanEval (pass@1)	55.49	43.29	0.61	40.24	72.5%
MMLU-Pro (5-shot CoT)	38.67	30.41	9.90	21.12	54.6%
GPQA-Diamond (CoT)	32.32	27.27	24.24	24.24	75.0%
KV1 test suite (25 tasks)	24/25	–	7/25	22/25	91.7%

The KV1 test suite has 25 automatically checked tasks: instruction following, math, coding (the generated code is executed against unit tests), native tool calling, and long-context retrieval at up to 16K tokens. KV1 passes all tool-calling and long-context tasks.

Averaged over all six rows, KV1 retains 75.8% of fp16 quality; untrained 2-bit retains 31.0%. GPQA-Diamond is near chance level (25%) for both 2-bit configs.

Sample sizes and noise: IFEval 541 prompts, GSM8K 250, HumanEval 164 (±3.8), MMLU-Pro 980 (70 per subject), GPQA-Diamond 198 (±3.1). Differences of a few points are within noise.

Known weakness: knowledge-heavy multiple-choice tasks (MMLU-Pro) recover less than generation, coding and tool use. Training on short, document-separated sequences is the planned fix for the next version.

Intelligence Density

Average score over IFEval, GSM8K, HumanEval, MMLU-Pro and GPQA-Diamond, divided by the KV cache size at 32K context:

Config	Avg. score	KV cache @ 32K	Score per MiB of KV cache	vs fp16
fp16 KV	55.19	384 MiB	0.144	1.0×
3-bit KV	47.26	84 MiB	0.563	3.9×
2-bit KV, untrained	15.88	60 MiB	0.265	1.8×
KV1 BQ2_0	41.61	60 MiB	0.693	4.8×

KV1 delivers 4.8× more benchmark score per MiB of KV cache than fp16, and more per MiB than an untrained 3-bit cache. The ratio does not depend on context length, because every cache grows linearly with it.

Citation

@techreport{kv1,
    title   = {KV1: 2-bit QAT KV Cache models at 1.2B, 2.6B, 4B, 8B A1B and 24B A2B},
    author  = {Klein ML},
    year    = {2026},
    month   = {October},
    url     = {https://kleinml.com}
}