Aijuus/flint-w4exp

Flint · int4 experts · for radiance

RecursiveSI/flint with its routed experts at four bits and every other weight as the checkpoint holds it (bf16), packed into one .rad container for radiance.

39.7 B over 49 layers, 256 routed experts at top-6 plus an ungated shared expert, GQA 8 at head_dim 128, 35 sliding layers (window 513) and 14 full no-RoPE layers with a learned per-head sink in every layer's softmax. The routed experts are 37.8 B of the 39.7 B, so they alone decide whether the model fits a card: four-bit codes with a bf16 scale per 128 of K put them at ~18.2 GiB and the whole container at 21.79 GiB, which is one card.

container	flint-w4exp.rad, 21.79 GiB, RAD2
architecture	flint (FlintForCausalLM)
weights	routed experts int4 — codes i4, group 128, bf16 scale, 128-wide Hadamard (fwht128) rotation (18.2 GiB); every linear and the shared expert bf16 (1.8 GiB)
context	262,144 tokens trained
KV cache	bf16, 28 KiB a token (14 full-attention layers, 4 KV heads at head_dim 128); fp8 halves it
sampling	temperature 1.0, top-p 0.95, top-k 20 — the checkpoint's own generation_config
licence	Apache-2.0, as the base model

Why the linears stay bf16

Taking the linears to int4 as well produces a dead model: it emits !instance Harry's interf residues ... with every top-10 logprob identical at -11.9193, a uniform distribution. Bisecting the classes puts the collapse on the shared expert, not on the attention or the dense layers (q|k|v alone keeps 0.513 of the tokens, the dense layers 0.683, neither with a uniform row). With these bf16 linears and int4 experts, Paris is first for "The capital of France is" — the reference's own argmax, -0.41 against the reference's -0.84 — and the following decode rows correlate 0.76–0.82 with the reference.

The linears are only ~1.8 GiB of the container, so the trade is small.

The 11.7% relative Frobenius error per expert plane at int4 is why this container reorders the first token of "The capital of France is" to not where the block-fp8 container keeps Paris. Layer 46's routed arm reaches rms 211 while the residual entering it is rms 1.26 — a hundredfold amplification of whatever the experts lose.

Serving it

Built with flint-w4exp.recipe, included here:

rad-convert <flint bf16 checkpoint> --recipe flint-w4exp.recipe -o flint-w4exp.rad

Measured on one Radeon AI PRO R9700 (gfx1201, RDNA4) with radiance at --tp 1 --max-num-seqs 8 --max-model-len 32768 --max-num-batched-tokens 2048 --num-speculative-tokens 3, 48 requests a level, temperature 1.0, thinking budget 512. The raw run is in benchmarks/betterbench-results.json:

decode, 1 / 2 / 4 / 8 users	86.2 / 151.1 / 221.5 / 309.5 tok/s aggregate
per-request decode at one user	87.3 tok/s (median 87.2)
prefill	9.6k tok/s at a 1.5k prompt, 9.8k at 5.8k, 9.5k at 11.7k, 8.4k at 23.3k

No speculation A/B has been measured on this container. It carries no draft head — the engine's built-in prompt-lookup drafter is the only option — and the figures that circulate for the flint one-card setup (+17.5%, 127.0 against 107.5 tok/s single-user, acceptance 0.423 pooled) are the flint-w4exp-w4lin measurement, not this container's. Do not quote them for this artifact.

--kv-cache-dtype fp8 is served by the engine and halves the cache (3.72 GiB against 7.44 GiB for 262k tokens) but the quality cost published so far — mean KL 0.105, top-1 0.850 over 6,793 positions against the same container on a bf16 cache — was measured on flint-w4exp-w4lin. This container's fp8 cache has no quality score yet; treat that number as the served variant's, not this one's.

Eight sessions at 65,536 tokens do not fit beside 21.79 GiB of weights (14 GiB of KV); eight at 32,768 do (7.2 GiB). The full 262,144-token context is 7.0 GiB of KV and takes the whole card with --max-num-seqs 1.