Meta Muse Glimmer 30B, IQ8R with its DFlash drafter, for radiance

muse-glimmer-30b-iq8r-dflash.rad (35.7 GiB) is meta-models/Muse-Glimmer-30B with its drafter meta-models/Muse-Glimmer-30B-assistant merged in, converted for the radiance inference engine, version 2.1.0 or later. It is a quick conversion for trying the model: round to nearest with a scale search, no calibration data.

What is in it

  • IQ8R trunk: every attention projection (q, gate, k, v, output) and both feed-forward projections as int8 codes with one u8 scale per (row, 256 columns) under one bf16 scale a row, each 128 columns rotated by a Walsh-Hadamard transform before rounding.
  • The lm_head as plain int8 with a bf16 scale per (row, 128 columns); the drafter reads the same head.
  • In bf16: the vision tower and its adapter, the DFlash drafter's own layers, the norms and the embedding.

Converted with the recipe the engine ships:

rad-convert <Muse-Glimmer-30B> --draft-model <Muse-Glimmer-30B-assistant> \
    --recipe data/recipes/museglimmer-30b-iq8r.recipe \
    --set quantization_config.quant_method=compressed-tensors \
    -o muse-glimmer-30b-iq8r-dflash.rad

Serving

Two 32 GB cards:

models:
  muse-glimmer:
    model: /models/muse-glimmer-30b-iq8r-dflash.rad
    tp: 2
    kv-cache-dtype: i8

The drafter is used by default; num-speculative-tokens: 0 serves the trunk alone and gives the KV cache the drafter's memory back (14.4 instead of 7.4 GiB a card at these settings).

What was checked

On two Radeon AI PRO R9700 at tp: 2 with the int8 KV cache, text only:

  • The container loads and answers short factual and coding prompts correctly, and its answers match the bf16 model's on the same prompts.
  • Essay prompts to 400 tokens, tokens a second in total:
streams drafter on speculation off
1 73 37
4 176 144
8 322 284

Not checked: pictures and video, tool calls, KL divergence against the bf16 model, and any benchmark.

License

Apache-2.0, as the source models; their LICENSE and USAGE_POLICY files are included.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for StillDeadcode/muse-glimmer-30b-iq8r-dflash

Quantized
(203)
this model