IQ8R container

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+minicpm5-2b-iq8r.rad filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,112 @@
+---
+license: apache-2.0
+base_model:
+- openbmb/MiniCPM5-2B
+- openbmb/MiniCPM5-2B-DSpark
+base_model_relation: quantized
+pipeline_tag: text-generation
+language:
+- en
+- zh
+tags:
+- radiance
+- rocm
+- amd
+- rdna4
+- int8
+- w8a8
+- iq8r
+- hadamard
+- speculative-decoding
+- dspark
+---
+
+# MiniCPM5-2B IQ8R + DSpark — radiance container
+
+[openbmb/MiniCPM5-2B](https://huggingface.co/openbmb/MiniCPM5-2B) as a single `.rad` container for
+the [radiance](https://codeberg.org/StillDeadcode/radiance) inference engine (AMD RDNA4, ROCm), with
+the [MiniCPM5-2B-DSpark](https://huggingface.co/openbmb/MiniCPM5-2B-DSpark) block drafter merged in
+for speculative decoding.
+
+It replaces [StillDeadcode/minicpm5-2b-fp8](https://huggingface.co/StillDeadcode/minicpm5-2b-fp8): the
+same eight bits a weight, closer to the bf16 model and a little faster. A bf16 version is at
+[StillDeadcode/minicpm5-2b-bf16](https://huggingface.co/StillDeadcode/minicpm5-2b-bf16).
+
+| | |
+|---|---|
+| File | `minicpm5-2b-iq8r.rad` — 3.04 GiB |
+| Weights | every linear of the model and of the drafter int8 (codes −127..127) with a u8 scale per row and 256 of K under a bf16 scale a row, each 128 of K rotated by the unnormalised Hadamard, round to nearest; the lm_head int8 with a bf16 scale per row and 128; embeddings and norms bf16 |
+| Activations | int8 under the same rotation (W8A8) |
+| Speculator | DSpark (5 layers, block size 7), IQ8R; its vocabulary head as 2-bit codes |
+| Context | 131,072 tokens (the checkpoint's trained length) |
+| Modalities | text |
+| Needs | radiance built from main after 1.17.0: IQ8R at one scale a 256 of K and the int8 KV cache are not in 1.17.0 and come with the next release |
+
+## Quality and speed
+
+KL divergence from the next-token distribution of radiance's bf16 container of the same model
+(82,066 positions of mixed text, one card, bf16 KV cache unless named):
+
+| | mean KLD | top-1 agreement |
+|---|---|---|
+| this file | 0.0103 | 97.3% |
+| this file, `kv-cache-dtype: i8` | 0.0113 | |
+| minicpm5-2b-fp8 | 0.0688 | 91.7% |
+| minicpm5-2b-fp8, `kv-cache-dtype: fp8` | 0.0837 | |
+
+One Radeon AI PRO R9700 (gfx1201), milliseconds a speculative step at 1 / 4 concurrent sequences:
+6.87 / 7.92, against 7.14 / 8.17 for minicpm5-2b-fp8; both accept 8.7 to 9.1 drafted tokens a step
+over 24 chat prompts. Without the drafter a decode step is 5.17 ms (FP8 5.40).
+
+## Serve
+
+```yaml
+# radiance.yaml
+server:
+  host: 0.0.0.0
+  port: 8000
+models:
+  minicpm5-2b:
+    model: minicpm5-2b-iq8r.rad
+    tp: 1
+    max-model-len: 131072
+    max-num-seqs: 8
+    kv-cache-dtype: i8
+```
+
+```sh
+radiance --config radiance.yaml
+```
+
+`tp: 2` splits it over two cards. The server speaks the OpenAI API (`/v1/chat/completions`,
+`/v1/completions`), with tool calls and structured output; the drafter's depth is chosen
+automatically (`num-speculative-tokens: N` states one, `0` turns speculation off).
+
+Built and tested on Radeon AI PRO R9700 (gfx1201), one card and two.
+
+## How this file was made
+
+```sh
+rad-convert openbmb/MiniCPM5-2B --draft-model openbmb/MiniCPM5-2B-DSpark \
+    --tokenizer openbmb/MiniCPM5-2B/tokenizer.json \
+    --recipe minicpm5-2b-iq8r.recipe -o minicpm5-2b-iq8r.rad
+```
+
+The recipe (`minicpm5-2b-iq8r.recipe` in this repository); everything it does not name is the
+checkpoint's own:
+
+```
+output.weight                        rtn  codes=i8 group=128 scale=bf16 clamp=sym rule=search
+dspark.draft_head.weight             rtn  codes=u2 zero=u8 group=128 scale=f16
+blk.*.attn_qkv.weight                rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
+blk.*.attn_output.weight             rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
+blk.*.ffn_gate_up.weight             rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
+blk.*.ffn_down.weight                rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
+dspark.fc.weight                     rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
+dspark.blk.*.attn_q.weight           rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
+dspark.blk.*.attn_k.weight           rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
+dspark.blk.*.attn_v.weight           rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
+dspark.blk.*.attn_output.weight      rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
+dspark.blk.*.ffn_gate_up.weight      rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
+dspark.blk.*.ffn_down.weight         rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
+```

minicpm5-2b-iq8r.rad

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6048448b1ffbb29df9ed44d5277cc00fc71db686959cc67f678c203e4d1d80ac
+size 3267162112

minicpm5-2b-iq8r.recipe

@@ -0,0 +1,34 @@
+# minicpm5-2b-iq8r.recipe -- MiniCPM5-2B with the DSpark drafter at IQ8R: every linear of the trunk
+# and of the drafter as rotated int8, read against int8 activations (arch/llama_dense_fp8).
+#
+# OpenBMB publishes MiniCPM5-2B in bf16 only. IQ8R is int8 codes in the 128-wide Hadamard domain
+# with a u8 scale per [1 x 256] of K under a bf16 scale per row, so a weight costs eight bits and
+# 1/256 of a byte a parameter plus two bytes a row. Round to nearest with a scale search: no
+# calibration data is read. Also:
+#
+#   - the lm_head, int8 with a bf16 scale per [1 x 128], unrotated: the widest GEMM in the model
+#     at half the bytes
+#   - the drafter's 2-bit copy of the target's head (u2 codes, an f16 scale and a u8 zero a group
+#     of 128), read by its draft passes only
+#
+# Every norm, the embedding and the drafter's Markov codebooks stay bf16.
+#
+#   rad-convert openbmb/MiniCPM5-2B --draft-model openbmb/MiniCPM5-2B-DSpark \
+#       --tokenizer openbmb/MiniCPM5-2B/tokenizer.json \
+#       --recipe data/recipes/minicpm5-2b-iq8r.recipe -o minicpm5-2b-iq8r.rad
+
+output.weight                        rtn  codes=i8 group=128 scale=bf16 clamp=sym rule=search
+dspark.draft_head.weight             rtn  codes=u2 zero=u8 group=128 scale=f16
+
+blk.*.attn_qkv.weight                rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
+blk.*.attn_output.weight             rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
+blk.*.ffn_gate_up.weight             rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
+blk.*.ffn_down.weight                rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
+
+dspark.fc.weight                     rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
+dspark.blk.*.attn_q.weight           rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
+dspark.blk.*.attn_k.weight           rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
+dspark.blk.*.attn_v.weight           rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
+dspark.blk.*.attn_output.weight      rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
+dspark.blk.*.ffn_gate_up.weight      rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
+dspark.blk.*.ffn_down.weight         rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search