Meta Muse Glimmer 30B, IQ8R with its DFlash drafter, for radiance
muse-glimmer-30b-iq8r-dflash.rad (35.7 GiB) is
meta-models/Muse-Glimmer-30B with its
drafter meta-models/Muse-Glimmer-30B-assistant
merged in, converted for the radiance inference
engine, version 2.1.0 or later. It is a quick conversion for trying the model: round to nearest
with a scale search, no calibration data.
What is in it
- IQ8R trunk: every attention projection (q, gate, k, v, output) and both feed-forward projections as int8 codes with one u8 scale per (row, 256 columns) under one bf16 scale a row, each 128 columns rotated by a Walsh-Hadamard transform before rounding.
- The lm_head as plain int8 with a bf16 scale per (row, 128 columns); the drafter reads the same head.
- In bf16: the vision tower and its adapter, the DFlash drafter's own layers, the norms and the embedding.
Converted with the recipe the engine ships:
rad-convert <Muse-Glimmer-30B> --draft-model <Muse-Glimmer-30B-assistant> \
--recipe data/recipes/museglimmer-30b-iq8r.recipe \
--set quantization_config.quant_method=compressed-tensors \
-o muse-glimmer-30b-iq8r-dflash.rad
Serving
Two 32 GB cards:
models:
muse-glimmer:
model: /models/muse-glimmer-30b-iq8r-dflash.rad
tp: 2
kv-cache-dtype: i8
The drafter is used by default; num-speculative-tokens: 0 serves the trunk alone and gives the
KV cache the drafter's memory back (14.4 instead of 7.4 GiB a card at these settings).
What was checked
On two Radeon AI PRO R9700 at tp: 2 with the int8 KV cache, text only:
- The container loads and answers short factual and coding prompts correctly, and its answers match the bf16 model's on the same prompts.
- Essay prompts to 400 tokens, tokens a second in total:
| streams | drafter on | speculation off |
|---|---|---|
| 1 | 73 | 37 |
| 4 | 176 | 144 |
| 8 | 322 | 284 |
Not checked: pictures and video, tool calls, KL divergence against the bf16 model, and any benchmark.
License
Apache-2.0, as the source models; their LICENSE and USAGE_POLICY files are included.
Model tree for StillDeadcode/muse-glimmer-30b-iq8r-dflash
Base model
meta-models/Muse-Glimmer-30B