license: apache-2.0
base_model:
- openbmb/MiniCPM5-2B
- openbmb/MiniCPM5-2B-DSpark
base_model_relation: quantized
pipeline_tag: text-generation
language:
- en
- zh
tags:
- radiance
- rocm
- amd
- rdna4
- int8
- w8a8
- iq8r
- hadamard
- speculative-decoding
- dspark
MiniCPM5-2B IQ8R + DSpark — radiance container
openbmb/MiniCPM5-2B as a single .rad container for
the radiance inference engine (AMD RDNA4, ROCm), with
the MiniCPM5-2B-DSpark block drafter merged in
for speculative decoding.
It replaces StillDeadcode/minicpm5-2b-fp8: the same eight bits a weight, closer to the bf16 model and a little faster. A bf16 version is at StillDeadcode/minicpm5-2b-bf16.
| File | minicpm5-2b-iq8r.rad — 3.04 GiB |
| Weights | every linear of the model and of the drafter int8 (codes −127..127) with a u8 scale per row and 256 of K under a bf16 scale a row, each 128 of K rotated by the unnormalised Hadamard, round to nearest; the lm_head int8 with a bf16 scale per row and 128; embeddings and norms bf16 |
| Activations | int8 under the same rotation (W8A8) |
| Speculator | DSpark (5 layers, block size 7), IQ8R; its vocabulary head as 2-bit codes |
| Context | 131,072 tokens (the checkpoint's trained length) |
| Modalities | text |
| Needs | radiance built from main after 1.17.0: IQ8R at one scale a 256 of K and the int8 KV cache are not in 1.17.0 and come with the next release |
Quality and speed
KL divergence from the next-token distribution of radiance's bf16 container of the same model (82,066 positions of mixed text, one card, bf16 KV cache unless named):
| mean KLD | top-1 agreement | |
|---|---|---|
| this file | 0.0103 | 97.3% |
this file, kv-cache-dtype: i8 |
0.0113 | |
| minicpm5-2b-fp8 | 0.0688 | 91.7% |
minicpm5-2b-fp8, kv-cache-dtype: fp8 |
0.0837 |
One Radeon AI PRO R9700 (gfx1201), milliseconds a speculative step at 1 / 4 concurrent sequences: 6.87 / 7.92, against 7.14 / 8.17 for minicpm5-2b-fp8; both accept 8.7 to 9.1 drafted tokens a step over 24 chat prompts. Without the drafter a decode step is 5.17 ms (FP8 5.40).
Serve
# radiance.yaml
server:
host: 0.0.0.0
port: 8000
models:
minicpm5-2b:
model: minicpm5-2b-iq8r.rad
tp: 1
max-model-len: 131072
max-num-seqs: 8
kv-cache-dtype: i8
radiance --config radiance.yaml
tp: 2 splits it over two cards. The server speaks the OpenAI API (/v1/chat/completions,
/v1/completions), with tool calls and structured output; the drafter's depth is chosen
automatically (num-speculative-tokens: N states one, 0 turns speculation off).
Built and tested on Radeon AI PRO R9700 (gfx1201), one card and two.
How this file was made
rad-convert openbmb/MiniCPM5-2B --draft-model openbmb/MiniCPM5-2B-DSpark \
--tokenizer openbmb/MiniCPM5-2B/tokenizer.json \
--recipe minicpm5-2b-iq8r.recipe -o minicpm5-2b-iq8r.rad
The recipe (minicpm5-2b-iq8r.recipe in this repository); everything it does not name is the
checkpoint's own:
output.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search
dspark.draft_head.weight rtn codes=u2 zero=u8 group=128 scale=f16
blk.*.attn_qkv.weight rtn codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
blk.*.attn_output.weight rtn codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
blk.*.ffn_gate_up.weight rtn codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
blk.*.ffn_down.weight rtn codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
dspark.fc.weight rtn codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
dspark.blk.*.attn_q.weight rtn codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
dspark.blk.*.attn_k.weight rtn codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
dspark.blk.*.attn_v.weight rtn codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
dspark.blk.*.attn_output.weight rtn codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
dspark.blk.*.ffn_gate_up.weight rtn codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
dspark.blk.*.ffn_down.weight rtn codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search