keptn/gemma4-31b-w5

Gemma 4 31B-it · int5 · for radiance

google/gemma-4-31B-it in one 23.4 GiB file for the radiance inference engine on AMD RDNA4. All linear layers of the trunk are int5 (GPTQ).

File	gemma4-31b-w5.rad, 23.36 GiB
Needs	radiance with PR #361, not in a release yet; 1, 2 or 4 cards with 32 GB (tested on one, two and four Radeon AI PRO R9700)
Format	weights int5 with a zero point, one f16 scale per 128 weights (all linear layers of the trunk, GPTQ); activations up to 64 rows f16; above that int8, one scale per token and 128 values; lm_head int8, one bf16 scale per 128 weights; the assistant head's lm_head also int8
Speculation	Google's Gemma 4 assistant head, bf16, 3 draft tokens
Inputs	text · image · video
Context	262,144 tokens
Version	1.0 (2026-10-28)

Speed

Radeon AI PRO R9700 at 215 W, radiance 1.13.0 plus the w5 branch, fp8 KV, 3 draft tokens, thinking off. BetterBench 0.6.1, full run, temperature 0.7. Tokens per second (median):

	1 card	2 cards	4 cards
decode, one stream	70.9	124.8	201.3
prefill at 24k tokens	1,421	2,559	4,214
4 streams, total	198	333	525

Quality

Distance to the bf16 original on the 80 MT-Bench questions, both turns. The text is the GPT-4 answers from lmsys/mt_bench_human_judgments, with the model's chat template and thinking off. The reference is the bf16 original in HF transformers, with fp32 arithmetic and the full vocabulary. radiance scores this file with --kld-ref, on the answer tokens only (42,283 tokens).

	mean KL	99th percentile KL	top token the same	perplexity ratio
this file	0.00803	0.107	97.8%	×1.010
keptn/gemma4-31b-i8g	0.00236	0.0311	98.8%	×1.004
keptn/gemma4-31b-w4a8	0.0517	0.721	94.6%	×1.056

Quick start

No radiance release has this model yet. Build the image from PR #361 first:

git clone https://codeberg.org/StillDeadcode/radiance.git && cd radiance
git fetch origin pull/361/head:pr-361 && git checkout pr-361
docker/build.sh -t gfx1201                   # gives the image radiance:latest

Then use radiance:latest as the image (RADIANCE_IMAGE=radiance:latest in .env).

Start the model on two cards:

hf download keptn/gemma4-31b-w5 gemma4-31b-w5.rad --local-dir ~/models
docker run -d --init --name gemma4-w5 --network host --group-add video --group-add render \
    --device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \
    -v ~/models:/models:ro -e HSA_ENABLE_IPC_MODE_LEGACY=1 -e ROCR_VISIBLE_DEVICES=0,1 \
    radiance:latest --model /models/gemma4-31b-w5.rad --host 0.0.0.0 --port 8000 \
    --tp 2 --p2p on --kv-cache-dtype fp8 --max-model-len 65536 --max-num-seqs 8 \
    --max-num-batched-tokens 2048 --num-speculative-tokens 3 --vision-tower-placement ram \
    --temp 1.0 --top-p 0.95 --top-k 64

	change
one card	ROCR_VISIBLE_DEVICES=0, --tp 1, remove --p2p on
four cards	ROCR_VISIBLE_DEVICES=0,1,2,3, --tp 4
full context	--max-model-len 262144 --max-num-seqs 2
thinking	per request: "chat_template_kwargs": {"enable_thinking": true}

ROCR_VISIBLE_DEVICES selects the cards. Use amd-smi list to find the card numbers. Three cards do not work. Gemma has 32 attention heads, and they do not split into three. The sampling values are Google's: temperature 1.0, top_p 0.95 and top_k 64.

Memory and context

KV pool at --max-model-len 262144, fp8 KV, radiance 1.13.0 plus the w5 branch, from radiance's startup report. Tokens of context in total, shared by all sequences:

	2 sequences	4 sequences	8 sequences
1 card	209,456	188,256	145,264
2 cards	524,304	970,480	927,792
4 cards	524,352	1,048,624	2,097,248

Images, audio and video

The same inputs go to radiance and to the bf16 original in HF transformers. Greedy, thinking off, 2 cards.

input	cases	same answer	mean KL	top token the same
images (DocVQA and AI2D, fixed selection)	–	–	–	–
video (to be fixed)	–	–	–	–

How it was made

1. Started from the bf16 checkpoint google/gemma-4-31B-it (revision 842da379).
2. Added Google's assistant head google/gemma-4-31B-it-assistant (revision 627c5ec1).
3. Used the calibration set of the W4A8 file: 110 chats, 165,626 tokens. No SmoothQuant.
4. Quantised every linear layer of the trunk with GPTQ to int5: a u8 zero point and one f16 scale per 128 weights. q, k and v share one GPTQ pass.
5. Rounded both lm_heads to int8, one bf16 scale per 128 weights. The other layers of the assistant head and the vision tower stay bf16.

rad-convert gemma-4-31B-it --draft-model gemma-4-31B-it-assistant \
    --recipe gemma4-31b-w5.recipe --set quantization_config.quant_method=compressed-tensors \
    -o gemma4-31b-w5.rad

GPTQ needs statistics from the calibration set (calib= in the recipe).

Calibration: own corpus of 110 chats, 165,626 tokens (older than keptn/quant-calibration).

Model card with: radiance-model-cards-framework 0.3 (codeberg.org/keptn42/radiance-model-cards-framework).

Limits

• radiance runs on AMD RDNA4 cards.
• The file needs the w5 support for Gemma 4 (PR #361). No radiance release has it yet.
• radiance has no compose file for this model yet. Use the docker run command in Quick start.

Credits

• Gemma 4 support in radiance: based on the initial work of sfjes (sfjes@vmdaemon.org) in PR #75.
• radiance: StillDeadcode (codeberg.org/StillDeadcode/radiance). The w5a16 and w5a8i8 kernels and GPTQ come from radiance and its libquant.
• Base model, assistant head and vision tower: Google.
• Test sets: MT-Bench, DocVQA, AI2D.

License

Apache 2.0, the license of the base model. See LICENSE.