Conversion recipe
Browse files- README.md +3 -2
- minicpm5-2b-bf16-dspark.recipe +18 -0
README.md
CHANGED
|
@@ -58,8 +58,9 @@ docker run --rm --user 1000:1000 -v ~/models:/models --entrypoint rad-convert ra
|
|
| 58 |
-o /models/rad/minicpm5-2b-bf16.rad
|
| 59 |
```
|
| 60 |
|
| 61 |
-
Running it twice gives byte-identical files. The recipe
|
| 62 |
-
|
|
|
|
| 63 |
|
| 64 |
```
|
| 65 |
dspark.draft_head.weight rtn codes=u2 zero=u8 group=128 scale=f16
|
|
|
|
| 58 |
-o /models/rad/minicpm5-2b-bf16.rad
|
| 59 |
```
|
| 60 |
|
| 61 |
+
Running it twice gives byte-identical files. The recipe file the command names is in this
|
| 62 |
+
repository as `minicpm5-2b-bf16-dspark.recipe`, with its comments. Its rules, as the container
|
| 63 |
+
records them (`rad-info --recipe minicpm5-2b-bf16.rad`), quantise the drafter only:
|
| 64 |
|
| 65 |
```
|
| 66 |
dspark.draft_head.weight rtn codes=u2 zero=u8 group=128 scale=f16
|
minicpm5-2b-bf16-dspark.recipe
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# minicpm5-2b-bf16-dspark.recipe -- MiniCPM5-2B in bf16 as the checkpoint ships it, with the DSpark
|
| 2 |
+
# drafter merged in (--draft-model). Every weight of the model itself, the lm_head included, is the
|
| 3 |
+
# checkpoint's own; the drafter is quantised as in minicpm5-2b-dspark.recipe, because the DSpark
|
| 4 |
+
# block reads its weights in one form only: block fp8, and the vocabulary head as 2-bit codes (the
|
| 5 |
+
# head only proposes; the model's own bf16 lm_head scores every token it accepts).
|
| 6 |
+
#
|
| 7 |
+
# rad-convert <MiniCPM5-2B checkpoint> --draft-model <MiniCPM5-2B-DSpark checkpoint> \
|
| 8 |
+
# --tokenizer <MiniCPM5-2B checkpoint>/tokenizer.json \
|
| 9 |
+
# --recipe data/recipes/minicpm5-2b-bf16-dspark.recipe -o minicpm5-2b-bf16.rad
|
| 10 |
+
|
| 11 |
+
dspark.draft_head.weight rtn codes=u2 zero=u8 group=128 scale=f16
|
| 12 |
+
dspark.fc.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16
|
| 13 |
+
dspark.blk.*.attn_q.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16
|
| 14 |
+
dspark.blk.*.attn_k.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16
|
| 15 |
+
dspark.blk.*.attn_v.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16
|
| 16 |
+
dspark.blk.*.attn_output.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16
|
| 17 |
+
dspark.blk.*.ffn_gate_up.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16
|
| 18 |
+
dspark.blk.*.ffn_down.weight rtn codes=fp8_e4m3 block=128x128 scale=bf16
|