Gemma 4 12B-it 路 a4 路 for radiance
google/gemma-4-12B-it in one 8.79 GiB file for the radiance inference engine on AMD RDNA4. The weights of the trunk are int4 with a zero point (GPTQ), and the activations are int8. The file reads text, images and audio and includes Google's assistant head for speculative decoding.
| File | gemma4-12b-a4.rad, 8.79 GiB |
| Needs | radiance 2.0.0 or newer; 1, 2 or 4 cards with 32 GB (tested on one, two and four Radeon AI PRO R9700) |
| Format | weights int4 with a zero point, one f16 scale per 128 weights (all linear layers of the trunk, GPTQ); activations int8, one scale per token and 128 values; lm_head int8, one bf16 scale per 128 weights; the assistant head's lm_head also int8 |
| Speculation | Google's Gemma 4 assistant head, bf16, 3 draft tokens |
| Inputs | text 路 image 路 audio |
| Context | 262,144 tokens |
| Version | 1.0 (2026-11-15) |
Speed
Radeon AI PRO R9700 at 215 W, radiance 2.0.0, i8 KV, 3 draft tokens, thinking off. BetterBench 0.6.1, full run, temperature 0.7. Tokens per second (median):
| 4 cards | |
|---|---|
| decode, one stream | 231.0 |
| prefill at 24k tokens | 2,904 |
| 4 streams, total | 557 |
Quality
Distance to the bf16 original on the 80 MT-Bench questions, both turns. The text is the GPT-4 answers from lmsys/mt_bench_human_judgments, with the model's chat template and thinking off. The reference is the bf16 original in HF transformers, with fp32 arithmetic and the full vocabulary. radiance scores this file with --kld-ref, on the answer tokens only (42,283 tokens). radiance 2.0.0 (release image), i8 KV, one card.
| mean KL | 99th percentile KL | top token the same | perplexity ratio | |
|---|---|---|---|---|
| this file | 0.0518 | 0.647 | 93.9% | 脳0.988 |
| keptn/gemma4-12b-i8g | 0.00542 | 0.0659 | 97.9% | 脳1.000 |
Quick start
Start the model on two cards. Save this as radiance.yaml:
server:
host: 0.0.0.0
port: 8000
models:
gemma4-12b-a4:
model: /models/gemma4-12b-a4.rad
tp: 2
p2p: on
kv-cache-dtype: i8
max-model-len: 262144
max-num-seqs: 8
num-speculative-tokens: 3
max-num-batched-tokens: 2048
default-max-tokens: 32768
max-tokens-cap: 65536
vision-tower-placement: ram
temp: 1.0
top-p: 0.95
top-k: 64
chat-template-kwargs: {"enable_thinking":false}
The entry name gemma4-12b-a4 is the model id in requests. Then run:
hf download keptn/gemma4-12b-a4 gemma4-12b-a4.rad --local-dir ~/models
docker pull stilldeadcode/radiance:2.0.0
docker run -d --init --name gemma4 --network host --group-add video --group-add render \
--device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \
-v ~/models:/models:ro -v $PWD/radiance.yaml:/etc/radiance/radiance.yaml:ro \
-e HSA_ENABLE_IPC_MODE_LEGACY=1 -e ROCR_VISIBLE_DEVICES=0,1 \
stilldeadcode/radiance:2.0.0 --config /etc/radiance/radiance.yaml
| change | |
|---|---|
| one card | ROCR_VISIBLE_DEVICES=0, tp: 1, remove p2p: on |
| four cards | ROCR_VISIBLE_DEVICES=0,1,2,3, tp: 4 |
| no drafter | num-speculative-tokens: 0 |
| thinking | per request: "chat_template_kwargs": {"enable_thinking": true} |
ROCR_VISIBLE_DEVICES selects the cards. Use amd-smi list to find the card numbers. The sampling values are Google's: temperature 1.0, top_p 0.95 and top_k 64.
Memory and context
KV cache at max-model-len: 262144 and 8 sequences, i8 KV, radiance 2.0.0, from radiance's startup report:
| KV cache | tokens of context in total | full-length sessions | |
|---|---|---|---|
| 1 card | 17.68 GiB a card | 2,097,168 | 8.0 |
| 2 cards | 16.87 GiB a card | 2,097,216 | 8.0 |
The cache holds all 8 sequences at full length on one and on two cards.
Images, audio and video
The same inputs go to radiance and to the bf16 original in HF transformers. Greedy, thinking off, 2 cards, radiance PR #338 final (67866b87). There is one measurement for each vision or audio encoder, made with gemma4-12b-bf16.rad. This file has the same bf16 vision tower and audio encoder.
| input | cases | same answer | mean KL | top token the same |
|---|---|---|---|---|
| images (DocVQA and AI2D, fixed selection) | 20 | 20/20 | 0.00184 | 149/149 |
| audio (LibriSpeech test-clean, 12 clips) | 12 | 11/12 | 0.00068 | 350/351 |
Word error rate on the audio clips: HF 6.68% 路 radiance 6.81%.
How it was made
- Downloaded
google/gemma-4-12B-it(revision707f0a3b) andgoogle/gemma-4-12B-it-assistant(revision3b0b0a4a). - Measured the GPTQ statistics with the bf16 model on the calibration set named below (
keptn/quant-calibration-v1, revision3969b2f9). These are the same rows and tokens as for keptn/gemma4-31b-a4. Rows were picked by category with a fixed seed, and each row was cut at 4,096 tokens. No SmoothQuant. - Quantised every linear layer of the trunk with GPTQ to int4: a u8 zero point and one f16 scale per 128 weights. q, k and v share one GPTQ pass. In layers 0 to 3, the attention output and both feed-forward weights got two rounds of coordinate descent.
- Rounded both lm_heads to int8, one bf16 scale per 128 weights. The other layers of the assistant head, the vision tower and the audio encoder stay bf16.
rad-convert gemma-4-12B-it --draft-model gemma-4-12B-it-assistant \
--recipe gemma4-12b-a4.recipe --set quantization_config.quant_method=compressed-tensors \
-o gemma4-12b-a4.rad
GPTQ needs statistics from the calibration set (calib= in the recipe). radiance 2.0.0's rad-convert factors each statistics matrix on one CPU thread, which takes hours at this size. So the factors (inverse Cholesky factor of the statistics plus damping) came from the card instead: fp64 up to 8,192 inputs, fp32 above. A small change to libquant reads them.
Calibration: 662 rows and 708,764 tokens of keptn/quant-calibration-v1, rendered with the chat template of Gemma 4 31B-it.
Limits
- radiance runs on AMD RDNA4 cards.
- The file needs radiance 2.0.0 or newer.
- Speed is measured on four cards only.
- Images and audio are measured with the bf16 file only. This file uses the same vision tower and audio encoder, but the comparison is not run on it.
Credits
- Gemma 4 support in radiance: based on the initial work of sfjes (sfjes@vmdaemon.org) in PR #75.
- radiance: StillDeadcode (codeberg.org/StillDeadcode/radiance). The int4 kernels and GPTQ come from radiance and its libquant.
- Base model, assistant head, vision tower and audio encoder: Google.
- Test and calibration sets: MT-Bench, DocVQA, AI2D, LibriSpeech, keptn/quant-calibration-v1.
License
Apache 2.0, the license of the base model. See LICENSE.
- Downloads last month
- 19