README.md

---
license: apache-2.0
base_model:
- openbmb/MiniCPM5-2B
base_model_relation: quantized
pipeline_tag: text-generation
library_name: transformers
tags:
- minicpm
- minicpm5
- llama
- fp8
- compressed-tensors
- quantization
- vllm
- blackwell
- code
- on-device
language:
- en
- zh
quantized_by: Ttimms
---

# MiniCPM5-2B — FP8 (compressed-tensors)

**FP8-dynamic quantization of [`openbmb/MiniCPM5-2B`](https://huggingface.co/openbmb/MiniCPM5-2B)**,
weights and activations in `float8_e4m3` (per-channel weights, per-token dynamic
activations), `lm_head` and embeddings kept in bf16. Produced with
[llm-compressor](https://github.com/vllm-project/llm-compressor) 0.13.

- **2.84 GiB** on disk (bf16 base is 4.68 GiB — **39 % smaller**)
- Serves on **vLLM** (`compressed-tensors`), native FP8 tensor-core path on Ada / Blackwell
- **Near-lossless** on coding benchmarks — see the eval below

This is the recommended quant for MiniCPM5-2B when you want maximum quality
retention. If you need to fit closer to 2 GB, see the
[NVFP4 W4A16 build](https://huggingface.co/Ttimms/MiniCPM5-2B-NVFP4)
(smaller, ~2.4 pp HumanEval / ~4.8 pp MBPP cost).

## Evaluation

`lm-evaluation-harness`, vLLM 0.26 backend, greedy decoding, **median of 3 draws**
with the range (the harness is non-deterministic run-to-run even at greedy — a
single draw is not a reproducible score). All arms measured in one session against
the released checkpoint. HumanEval-instruct `pass@1`/`create_test` (n = 164);
MBPP base 3-shot (n = 500).

| build | HumanEval-inst | MBPP (3-shot) | size | Δ HE / MBPP |
|---|---:|---:|---:|---:|
| bf16 base | 86.59 % (85.98–86.59) | 50.60 % (50.40–51.00) | 4.68 GiB | — |
| **FP8-dynamic (this)** | **84.76 %** (84.15–85.37) | **48.80 %** (48.80–49.00) | **2.84 GiB** | **−1.8 / −1.8 pp** |
| NVFP4-W4A16 GPTQ | 84.15 % (81.71–84.15) | 45.80 % (45.60–46.40) | 2.03 GiB | −2.4 / −4.8 pp |
| NVFP4-W4A16 RTN | 79.88 % (77.44–79.88) | 41.20 % (41.20–41.80) | 2.03 GiB | −6.7 / −9.4 pp |
| mixed (MLP-NVFP4 + attn-FP8) | 84.15 % (82.93–85.98) | 46.60 % (46.00–46.80) | 2.20 GiB | −2.4 / −4.0 pp |

Both FP8 deltas sit inside the eval's own run-to-run spread — FP8 is effectively
lossless here. NVFP4 costs real accuracy on a 2.5 B dense model (little redundancy
to absorb 4-bit weights); pick it only if the extra ~0.8 GiB matters for your KV
budget.

Full comparison, serving notes, and a fine-tune experiment that regressed the
base's coding (so these quants target the released checkpoint):
**<https://github.com/t-timms/minicpm5-2b-quants>**

## Usage

```bash
vllm serve Ttimms/MiniCPM5-2B-FP8 --max-model-len 32768 --kv-cache-dtype fp8
```

```python
from vllm import LLM, SamplingParams
llm = LLM("Ttimms/MiniCPM5-2B-FP8", trust_remote_code=True)
print(llm.chat([{"role": "user", "content": "Write a Python LRU cache."}],
               SamplingParams(temperature=0.6, max_tokens=512))[0].outputs[0].text)
```

On WSL, set `VLLM_USE_V2_MODEL_RUNNER=0` (the V2 runner needs UVA, which WSL
disables).

## Method & provenance

- Quantizer: `llm-compressor` 0.13, `QuantizationModifier(scheme="FP8_DYNAMIC")`,
  ignore `lm_head` + `embed_tokens`. No calibration data required (dynamic activations).
- Base: `openbmb/MiniCPM5-2B` (`LlamaForCausalLM`, 2.5 B, Apache-2.0).
- Built and evaluated on an RTX 5070 Ti (Blackwell, SM120), vLLM 0.26.

## License

Apache-2.0, inherited from `openbmb/MiniCPM5-2B`.