| --- |
| license: apache-2.0 |
| base_model: |
| - openbmb/MiniCPM5-2B |
| - openbmb/MiniCPM5-2B-DSpark |
| base_model_relation: quantized |
| pipeline_tag: text-generation |
| language: |
| - en |
| - zh |
| tags: |
| - radiance |
| - rocm |
| - amd |
| - rdna4 |
| - int8 |
| - w8a8 |
| - iq8r |
| - hadamard |
| - speculative-decoding |
| - dspark |
| --- |
| |
| # MiniCPM5-2B IQ8R + DSpark — radiance container |
|
|
| [openbmb/MiniCPM5-2B](https://huggingface.co/openbmb/MiniCPM5-2B) as a single `.rad` container for |
| the [radiance](https://codeberg.org/StillDeadcode/radiance) inference engine (AMD RDNA4, ROCm), with |
| the [MiniCPM5-2B-DSpark](https://huggingface.co/openbmb/MiniCPM5-2B-DSpark) block drafter merged in |
| for speculative decoding. |
|
|
| It replaces [StillDeadcode/minicpm5-2b-fp8](https://huggingface.co/StillDeadcode/minicpm5-2b-fp8): the |
| same eight bits a weight, closer to the bf16 model and a little faster. A bf16 version is at |
| [StillDeadcode/minicpm5-2b-bf16](https://huggingface.co/StillDeadcode/minicpm5-2b-bf16). |
|
|
| | | | |
| |---|---| |
| | File | `minicpm5-2b-iq8r.rad` — 3.04 GiB | |
| | Weights | every linear of the model and of the drafter int8 (codes −127..127) with a u8 scale per row and 256 of K under a bf16 scale a row, each 128 of K rotated by the unnormalised Hadamard, round to nearest; the lm_head int8 with a bf16 scale per row and 128; embeddings and norms bf16 | |
| | Activations | int8 under the same rotation (W8A8) | |
| | Speculator | DSpark (5 layers, block size 7), IQ8R; its vocabulary head as 2-bit codes | |
| | Context | 131,072 tokens (the checkpoint's trained length) | |
| | Modalities | text | |
| | Needs | radiance built from main after 1.17.0: IQ8R at one scale a 256 of K and the int8 KV cache are not in 1.17.0 and come with the next release | |
| |
| ## Quality and speed |
| |
| KL divergence from the next-token distribution of radiance's bf16 container of the same model |
| (82,066 positions of mixed text, one card, bf16 KV cache unless named): |
| |
| | | mean KLD | top-1 agreement | |
| |---|---|---| |
| | this file | 0.0103 | 97.3% | |
| | this file, `kv-cache-dtype: i8` | 0.0113 | | |
| | minicpm5-2b-fp8 | 0.0688 | 91.7% | |
| | minicpm5-2b-fp8, `kv-cache-dtype: fp8` | 0.0837 | | |
| |
| One Radeon AI PRO R9700 (gfx1201), milliseconds a speculative step at 1 / 4 concurrent sequences: |
| 6.87 / 7.92, against 7.14 / 8.17 for minicpm5-2b-fp8; both accept 8.7 to 9.1 drafted tokens a step |
| over 24 chat prompts. Without the drafter a decode step is 5.17 ms (FP8 5.40). |
| |
| ## Serve |
| |
| ```yaml |
| # radiance.yaml |
| server: |
| host: 0.0.0.0 |
| port: 8000 |
| models: |
| minicpm5-2b: |
| model: minicpm5-2b-iq8r.rad |
| tp: 1 |
| max-model-len: 131072 |
| max-num-seqs: 8 |
| kv-cache-dtype: i8 |
| ``` |
| |
| ```sh |
| radiance --config radiance.yaml |
| ``` |
| |
| `tp: 2` splits it over two cards. The server speaks the OpenAI API (`/v1/chat/completions`, |
| `/v1/completions`), with tool calls and structured output; the drafter's depth is chosen |
| automatically (`num-speculative-tokens: N` states one, `0` turns speculation off). |
| |
| Built and tested on Radeon AI PRO R9700 (gfx1201), one card and two. |
| |
| ## How this file was made |
| |
| ```sh |
| rad-convert openbmb/MiniCPM5-2B --draft-model openbmb/MiniCPM5-2B-DSpark \ |
| --tokenizer openbmb/MiniCPM5-2B/tokenizer.json \ |
| --recipe minicpm5-2b-iq8r.recipe -o minicpm5-2b-iq8r.rad |
| ``` |
| |
| The recipe (`minicpm5-2b-iq8r.recipe` in this repository); everything it does not name is the |
| checkpoint's own: |
| |
| ``` |
| output.weight rtn codes=i8 group=128 scale=bf16 clamp=sym rule=search |
| dspark.draft_head.weight rtn codes=u2 zero=u8 group=128 scale=f16 |
| blk.*.attn_qkv.weight rtn codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search |
| blk.*.attn_output.weight rtn codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search |
| blk.*.ffn_gate_up.weight rtn codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search |
| blk.*.ffn_down.weight rtn codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search |
| dspark.fc.weight rtn codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search |
| dspark.blk.*.attn_q.weight rtn codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search |
| dspark.blk.*.attn_k.weight rtn codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search |
| dspark.blk.*.attn_v.weight rtn codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search |
| dspark.blk.*.attn_output.weight rtn codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search |
| dspark.blk.*.ffn_gate_up.weight rtn codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search |
| dspark.blk.*.ffn_down.weight rtn codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search |
| ``` |
| |
