StillDeadcode/qwen3.6-35b-a3b-iq8r

Qwen3.6-35B-A3B IQ8R — radiance container

Qwen/Qwen3.6-35B-A3B quantised from its bf16 checkpoint to int8 in the Hadamard domain, as a single .rad container for the radiance inference engine (AMD RDNA4, ROCm), with its MTP head for speculative decoding and its vision tower.

It replaces StillDeadcode/qwen3.6-35b-a3b-fp8, which holds Qwen's FP8 checkpoint: the same eight bits a weight, with int8 activations throughout.

File	qwen3.6-35b-a3b-iq8r.rad — 35.04 GiB
Trunk	every attention and delta-net linear, the MTP layer's too, int8 (codes −127..127) with a u8 scale per row and 256 of K under a bf16 scale a row, each 128 of K rotated by the unnormalised Hadamard, round to nearest
Experts	the 256 routed experts of every layer int8 on the same rotated grid with a bf16 scale per row and 128 of K; the shared expert likewise (its gate/up projection as the trunk's linears)
Activations	int8 under the same rotation (W8A8), the routed experts reading the codes the trunk's quantisers write
Speculator	the model's MTP head, which drafts with a 2-bit copy of the lm_head
Head	the lm_head int8 with a bf16 scale per row and 128
Vision	the 27-block vision tower, bf16: images and video in chat requests
Context	262,144 tokens trained
Needs	radiance built from main after 1.17.0: the routed int8 build (qwen35moe_w8a8) is not in 1.17.0 and comes with the next release

Testing

Not measured as KL divergence against the bf16 model. It was served on 2× Radeon AI PRO R9700 (gfx1201, tp: 2, 32K context, int8 KV cache) beside the FP8 container, interleaved twice:

	this file	qwen3.6-35b-a3b-fp8
tokens a speculative step, 24 chat prompts	10.39, 10.40	10.20, 10.31
ms a decode step without the MTP head, 1 / 4 / 8 sequences	10.3 / 11.8 / 12.4	10.2 / 11.6 / 12.2

After prompts of 28 to 4,018 tokens ending in the same question the answer was right at every length, as the FP8 container's was, with no engine error in either.

Serve

The whole model fits two 32 GB cards.

# radiance.yaml
server:
  host: 0.0.0.0
  port: 8000
models:
  qwen3.6-35b-a3b:
    model: qwen3.6-35b-a3b-iq8r.rad
    tp: 2
    max-model-len: 200000
    max-num-seqs: 8
    kv-cache-dtype: i8

radiance --config radiance.yaml

kv-cache-dtype: i8 stores the attention cache as int8 with an f16 scale per token and head. The server speaks the OpenAI API (/v1/chat/completions, /v1/completions), with tool calls and structured output, and image_url / video_url parts in chat messages. The MTP depth is chosen automatically (num-speculative-tokens: N states one, 0 turns speculation off).

How this file was made

rad-convert Qwen/Qwen3.6-35B-A3B --recipe q36-35b-a3b-iq8r.recipe \
    --set quantization_config.quant_method=w8a8 --set quantization_config.fmt=fwht128 \
    -o qwen3.6-35b-a3b-iq8r.rad

The recipe (q36-35b-a3b-iq8r.recipe in this repository); everything it does not name is the checkpoint's own:

blk.*.attn_qg.weight                 rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
blk.*.attn_k.weight                  rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
blk.*.attn_v.weight                  rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
blk.*.attn_output.weight             rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
blk.*.ssm_inz.weight                 rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
blk.*.ssm_out.weight                 rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
blk.*.ffn_gate_up_exps.*.weight      rtn  codes=i8 group=128 scale=bf16 transform=fwht128 clamp=sym rule=search
blk.*.ffn_down_exps.*.weight         rtn  codes=i8 group=128 scale=bf16 transform=fwht128 clamp=sym rule=search
blk.*.ffn_gate_up_shexp.weight       rtn  codes=i8 group=256 scale=u8 scale2=bf16 block2=1x* transform=fwht128 clamp=sym rule=search
blk.*.ffn_down_shexp.weight          rtn  codes=i8 group=128 scale=bf16 transform=fwht128 clamp=sym rule=search
output.weight                        rtn  codes=i8 group=128 scale=bf16 clamp=sym rule=search
mtp.draft_head.weight                rtn  codes=u2 zero=u8 group=128 scale=f16

License

Apache 2.0, as the base model; see LICENSE.