StillDeadcode/minicpm5-2b-iq8r

MiniCPM5-2B IQ8R + DSpark — radiance container

openbmb/MiniCPM5-2B as a single .rad container for the radiance inference engine (AMD RDNA4, ROCm), with the MiniCPM5-2B-DSpark block drafter merged in for speculative decoding.

It replaces StillDeadcode/minicpm5-2b-fp8: the same eight bits a weight, closer to the bf16 model and a little faster. A bf16 version is at StillDeadcode/minicpm5-2b-bf16.

File	minicpm5-2b-iq8r.rad — 3.04 GiB
Weights	every linear of the model and of the drafter int8 (codes −127..127) with a u8 scale per row and 256 of K under a bf16 scale a row, each 128 of K rotated by the unnormalised Hadamard, round to nearest; the lm_head int8 with a bf16 scale per row and 128; embeddings and norms bf16
Activations	int8 under the same rotation (W8A8)
Speculator	DSpark (5 layers, block size 7), IQ8R; its vocabulary head as 2-bit codes
Context	131,072 tokens (the checkpoint's trained length)
Modalities	text
Needs	radiance built from main after 1.17.0: IQ8R at one scale a 256 of K and the int8 KV cache are not in 1.17.0 and come with the next release

Quality and speed

KL divergence from the next-token distribution of radiance's bf16 container of the same model (82,066 positions of mixed text, one card, bf16 KV cache unless named):

	mean KLD	top-1 agreement
this file	0.0103	97.3%
this file, kv-cache-dtype: i8	0.0113
minicpm5-2b-fp8	0.0688	91.7%
minicpm5-2b-fp8, kv-cache-dtype: fp8	0.0837

One Radeon AI PRO R9700 (gfx1201), milliseconds a speculative step at 1 / 4 concurrent sequences: 6.87 / 7.92, against 7.14 / 8.17 for minicpm5-2b-fp8; both accept 8.7 to 9.1 drafted tokens a step over 24 chat prompts. Without the drafter a decode step is 5.17 ms (FP8 5.40).

Serve

# radiance.yaml
server:
  host: 0.0.0.0
  port: 8000
models:
  minicpm5-2b:
    model: minicpm5-2b-iq8r.rad
    tp: 1
    max-model-len: 131072
    max-num-seqs: 8
    kv-cache-dtype: i8

radiance --config radiance.yaml

tp: 2 splits it over two cards. The server speaks the OpenAI API (/v1/chat/completions, /v1/completions), with tool calls and structured output; the drafter's depth is chosen automatically (num-speculative-tokens: N states one, 0 turns speculation off).

Built and tested on Radeon AI PRO R9700 (gfx1201), one card and two.

How this file was made

rad-convert openbmb/MiniCPM5-2B --draft-model openbmb/MiniCPM5-2B-DSpark \
    --tokenizer openbmb/MiniCPM5-2B/tokenizer.json \
    --recipe minicpm5-2b-iq8r.recipe -o minicpm5-2b-iq8r.rad

The recipe (minicpm5-2b-iq8r.recipe in this repository); everything it does not name is the checkpoint's own:

output.weight                        rtn  codes=i8 group=128 scale=bf16 clamp=sym rule=search
dspark.draft_head.weight             rtn  codes=u2 zero=u8 group=128 scale=f16
blk.*.attn_qkv.weight                rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
blk.*.attn_output.weight             rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
blk.*.ffn_gate_up.weight             rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
blk.*.ffn_down.weight                rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
dspark.fc.weight                     rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
dspark.blk.*.attn_q.weight           rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
dspark.blk.*.attn_k.weight           rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
dspark.blk.*.attn_v.weight           rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
dspark.blk.*.attn_output.weight      rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
dspark.blk.*.ffn_gate_up.weight      rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
dspark.blk.*.ffn_down.weight         rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search