---
base_model: openbmb/MiniCPM5-2B
base_model_relation: quantized
library_name: transformers
license: apache-2.0
pipeline_tag: text-generation
tags:
  - minicpm
  - compressed-tensors
  - vllm
  - quantized
  - fp8
---
# MiniCPM5-2B · FP8 (dynamic)

8-bit quantization of [`openbmb/MiniCPM5-2B`](https://huggingface.co/openbmb/MiniCPM5-2B) for **vLLM serving**. Quantized by **axteam**.

| | |
|---|---|
| **Base model** | [`openbmb/MiniCPM5-2B`](https://huggingface.co/openbmb/MiniCPM5-2B) |
| **Method** | FP8 (dynamic) |
| **Format** | compressed-tensors `float-quantized` (scheme `FP8_DYNAMIC`) |
| **Weights / activations** | W8A8 — FP8 E4M3 weights (per-channel), dynamic per-token FP8 activations |
| **Architecture** | Dense (Llama-architecture) · text-only |
| **Size on disk** | ~3.1 GB |
| **Calibration** | None (data-free) |
| **License** | Apache-2.0 |

## Quantization details
- **Scheme** — W8A8 — FP8 E4M3 weights (per-channel), dynamic per-token FP8 activations, stored as compressed-tensors `float-quantized` (scheme `FP8_DYNAMIC`).
- **Kept in BF16** — token embeddings, the LM head, and all RMSNorms (about 21% of all parameters in this build).
- Post-training quantization (no retraining). Tokenizer / `chat_template` identical to the original.

## Benchmarks (measured)

Every column was measured with the same harness and settings — [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) on vLLM, greedy, few-shot completion (no chat template). Percentages are recovery vs. the BF16 original.

| Benchmark | BF16 (original) | **FP8-dynamic (this repo)** | AWQ-8bit | NVFP4A16 | NVFP4 | AWQ-4bit |
|---|---|---|---|---|---|---|
| MMLU (5-shot) | 59.74 | **59.56 (99.7%)** | 59.86 (100.2%) | 56.81 (95.1%) | 53.50 (89.5%) | 56.31 (94.3%) |
| GSM8K (5-shot, strict) | 73.01 | **71.27 (97.6%)** | 73.01 (100.0%) | 65.20 (89.3%) | 55.04 (75.4%) | 65.96 (90.3%) |
| ARC-Challenge (0-shot) | 43.94 | **44.20 (100.6%)** | 43.86 (99.8%) | 43.43 (98.8%) | 40.10 (91.3%) | 44.88 (102.1%) |
| Winogrande (5-shot) | 64.40 | **63.22 (98.2%)** | 64.40 (100.0%) | 62.75 (97.4%) | 59.04 (91.7%) | 62.12 (96.4%) |
| KMMLU (5-shot, direct) | 37.65 | **38.38 (101.9%)** | 38.27 (101.7%) | 35.56 (94.4%) | 34.06 (90.5%) | 30.73 (81.6%) |
| HAE-RAE (0-shot) | 39.05 | **38.13 (97.7%)** | 39.14 (100.2%) | 35.38 (90.6%) | 30.16 (77.2%) | 36.94 (94.6%) |
| **Average** | **52.97** | **52.46 (99.0%)** | **53.09 (100.2%)** | **49.85 (94.1%)** | **45.32 (85.6%)** | **49.49 (93.4%)** |

## Evaluation (measured)

Fidelity to the BF16 original on an internal mixed EN/KO set (100 prompts, BF16 greedy answers scored under each build).

| Metric | This model | BF16 original |
|---|---|---|
| Perplexity (lower = closer) | 1.6494 | 1.5988 (**+3.2%**) |
| Top-1 next-token agreement | 92.9% | 98.9% |

## Requirements

| | |
|---|---|
| **Runtime** | vLLM — the architecture is in vLLM; **tested here with vLLM 0.32.0**. |
| **vLLM install** | vLLM 0.32 itself needs Python 3.11–3.14, torch 2.13 (CUDA 13.0 build, so an NVIDIA driver with CUDA 13 support), transformers ≥5.16.1,<5.20 and compressed-tensors 0.17.0 (both pinned by vLLM; 0.17.0 reads this checkpoint, which was saved with 0.19.0 — do not upgrade it separately). |
| **GPU** | NVIDIA Turing (sm75, e.g. T4) or newer. FP8 math runs on Ada / Hopper / Blackwell (sm89+); on Turing and Ampere vLLM switches to weight-only FP8 automatically. V100 (sm70) is not supported. (Kernel choice per GPU from vLLM 0.32's selection code; only B200 was run here.) |
| **Memory** | Weights: **3.1 GB**. Add KV cache (grows with context length × concurrent requests) and a few GB for activations / CUDA graphs; use `--tensor-parallel-size N` when the weights do not fit one GPU. |
| **Transformers (without vLLM)** | `transformers>=4.56.0` + `compressed-tensors>=0.15.0` (checked 4.56.0 / 4.56.2 / 4.57.6 with FP8, AWQ-4bit, NVFP4A16). |
| **trust_remote_code** | not needed |

## Usage (vLLM)

Start an OpenAI-compatible server:

```bash
vllm serve axteam/MiniCPM5-2B-FP8-dynamic --max-model-len 32768
```

Query it:

```bash
curl http://localhost:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "axteam/MiniCPM5-2B-FP8-dynamic",
  "messages": [{"role": "user", "content": "대한민국의 수도는 어디인가요?"}],
  "temperature": 1.0, "top_p": 0.95, "max_tokens": 512
}'
```

Offline inference in Python:

```python
from vllm import LLM, SamplingParams

llm = LLM(model="axteam/MiniCPM5-2B-FP8-dynamic", max_model_len=32768)
params = SamplingParams(temperature=1.0, top_p=0.95, max_tokens=1024)
out = llm.chat([{"role": "user", "content": "Explain model quantization in two sentences."}], params)
print(out[0].outputs[0].text)
```

> Recommended sampling (from the original `generation_config.json`): `temperature=1.0`, `top_p=0.95`.
> Thinking mode is on by default in the chat template; to turn it off pass `chat_template_kwargs={"enable_thinking": False}` (Python `llm.chat(..., chat_template_kwargs=...)`, or `"chat_template_kwargs": {"enable_thinking": false}` in the request body).

## Reducing KV-cache memory (optional, serve-time)

The KV cache is kept in BF16 by default. KV-cache precision is a **runtime** choice, independent of the
weight quantization above; for long contexts or large batches FP8 roughly halves KV-cache memory:

```bash
vllm serve axteam/MiniCPM5-2B-FP8-dynamic --kv-cache-dtype fp8
```

## License
Quantized derivative of [`openbmb/MiniCPM5-2B`](https://huggingface.co/openbmb/MiniCPM5-2B) (Apache-2.0). Original weights & tokenizer belong to OpenBMB.

---
## 한국어 요약
[`openbmb/MiniCPM5-2B`](https://huggingface.co/openbmb/MiniCPM5-2B) 를 **FP8 (dynamic)** 로 양자화한 서빙용(8-bit) 모델입니다. 작성: **axteam**.
- **방식** — FP8 E4M3, 가중치 채널별 스케일(정적·보정불필요), 활성화 동적 FP8. 형식 compressed-tensors `float-quantized` (scheme `FP8_DYNAMIC`).
- **BF16 유지** — 토큰 임베딩·LM 헤드·모든 norm.
- PTQ(재학습 없음), 토크나이저·chat_template 원본 동일.
- **품질(실측)** — BF16 대비 perplexity 1.6494(원본 1.5988, **+3.2%**), top-1 92.9%(원본 98.9%). 내부 영/한 100문항.
- **벤치마크(실측)** — MMLU · GSM8K · ARC-Challenge · Winogrande · KMMLU · HAE-RAE 평균 BF16 52.97 → 이 모델 52.46 (**회복률 99.0%**). lm-evaluation-harness · vLLM, 모든 형식 같은 설정.
- **요구사항** — vLLM (0.32.0 에서 실측); GPU Turing(sm75) 이상. FP8 연산은 Ada·Hopper·Blackwell(sm89+), 그 아래는 vLLM 이 가중치 전용 FP8 로 자동 전환; 가중치 3.1 GB + KV 캐시; Transformers 직접 사용: transformers ≥4.56.0 + compressed-tensors ≥0.15.0.
- **실행** — `vllm serve axteam/MiniCPM5-2B-FP8-dynamic --max-model-len 32768` 로 OpenAI 호환 서버(포트 8000), 또는 Python `LLM(model="axteam/MiniCPM5-2B-FP8-dynamic").chat(...)` (위 예시). 권장 샘플링 `temperature=1.0`, `top_p=0.95`. thinking 끄기: `chat_template_kwargs={"enable_thinking": False}`.
- **KV 캐시 절감(런타임)** — `--kv-cache-dtype fp8`.