Gemma 4 12B-it 路 a4 路 for radiance

google/gemma-4-12B-it in one 8.79 GiB file for the radiance inference engine on AMD RDNA4. The weights of the trunk are int4 with a zero point (GPTQ), and the activations are int8. The file reads text, images and audio and includes Google's assistant head for speculative decoding.

File gemma4-12b-a4.rad, 8.79 GiB
Needs radiance 2.0.0 or newer; 1, 2 or 4 cards with 32 GB (tested on one, two and four Radeon AI PRO R9700)
Format weights int4 with a zero point, one f16 scale per 128 weights (all linear layers of the trunk, GPTQ); activations int8, one scale per token and 128 values; lm_head int8, one bf16 scale per 128 weights; the assistant head's lm_head also int8
Speculation Google's Gemma 4 assistant head, bf16, 3 draft tokens
Inputs text 路 image 路 audio
Context 262,144 tokens
Version 1.0 (2026-11-15)

Speed

Radeon AI PRO R9700 at 215 W, radiance 2.0.0, i8 KV, 3 draft tokens, thinking off. BetterBench 0.6.1, full run, temperature 0.7. Tokens per second (median):

4 cards
decode, one stream 231.0
prefill at 24k tokens 2,904
4 streams, total 557

Quality

Distance to the bf16 original on the 80 MT-Bench questions, both turns. The text is the GPT-4 answers from lmsys/mt_bench_human_judgments, with the model's chat template and thinking off. The reference is the bf16 original in HF transformers, with fp32 arithmetic and the full vocabulary. radiance scores this file with --kld-ref, on the answer tokens only (42,283 tokens). radiance 2.0.0 (release image), i8 KV, one card.

mean KL 99th percentile KL top token the same perplexity ratio
this file 0.0518 0.647 93.9% 脳0.988
keptn/gemma4-12b-i8g 0.00542 0.0659 97.9% 脳1.000

Quick start

Start the model on two cards. Save this as radiance.yaml:

server:
  host: 0.0.0.0
  port: 8000
models:
  gemma4-12b-a4:
    model: /models/gemma4-12b-a4.rad
    tp: 2
    p2p: on
    kv-cache-dtype: i8
    max-model-len: 262144
    max-num-seqs: 8
    num-speculative-tokens: 3
    max-num-batched-tokens: 2048
    default-max-tokens: 32768
    max-tokens-cap: 65536
    vision-tower-placement: ram
    temp: 1.0
    top-p: 0.95
    top-k: 64
    chat-template-kwargs: {"enable_thinking":false}

The entry name gemma4-12b-a4 is the model id in requests. Then run:

hf download keptn/gemma4-12b-a4 gemma4-12b-a4.rad --local-dir ~/models
docker pull stilldeadcode/radiance:2.0.0
docker run -d --init --name gemma4 --network host --group-add video --group-add render \
    --device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \
    -v ~/models:/models:ro -v $PWD/radiance.yaml:/etc/radiance/radiance.yaml:ro \
    -e HSA_ENABLE_IPC_MODE_LEGACY=1 -e ROCR_VISIBLE_DEVICES=0,1 \
    stilldeadcode/radiance:2.0.0 --config /etc/radiance/radiance.yaml
change
one card ROCR_VISIBLE_DEVICES=0, tp: 1, remove p2p: on
four cards ROCR_VISIBLE_DEVICES=0,1,2,3, tp: 4
no drafter num-speculative-tokens: 0
thinking per request: "chat_template_kwargs": {"enable_thinking": true}

ROCR_VISIBLE_DEVICES selects the cards. Use amd-smi list to find the card numbers. The sampling values are Google's: temperature 1.0, top_p 0.95 and top_k 64.

Memory and context

KV cache at max-model-len: 262144 and 8 sequences, i8 KV, radiance 2.0.0, from radiance's startup report:

KV cache tokens of context in total full-length sessions
1 card 17.68 GiB a card 2,097,168 8.0
2 cards 16.87 GiB a card 2,097,216 8.0

The cache holds all 8 sequences at full length on one and on two cards.

Images, audio and video

The same inputs go to radiance and to the bf16 original in HF transformers. Greedy, thinking off, 2 cards, radiance PR #338 final (67866b87). There is one measurement for each vision or audio encoder, made with gemma4-12b-bf16.rad. This file has the same bf16 vision tower and audio encoder.

input cases same answer mean KL top token the same
images (DocVQA and AI2D, fixed selection) 20 20/20 0.00184 149/149
audio (LibriSpeech test-clean, 12 clips) 12 11/12 0.00068 350/351

Word error rate on the audio clips: HF 6.68% 路 radiance 6.81%.

How it was made

  1. Downloaded google/gemma-4-12B-it (revision 707f0a3b) and google/gemma-4-12B-it-assistant (revision 3b0b0a4a).
  2. Measured the GPTQ statistics with the bf16 model on the calibration set named below (keptn/quant-calibration-v1, revision 3969b2f9). These are the same rows and tokens as for keptn/gemma4-31b-a4. Rows were picked by category with a fixed seed, and each row was cut at 4,096 tokens. No SmoothQuant.
  3. Quantised every linear layer of the trunk with GPTQ to int4: a u8 zero point and one f16 scale per 128 weights. q, k and v share one GPTQ pass. In layers 0 to 3, the attention output and both feed-forward weights got two rounds of coordinate descent.
  4. Rounded both lm_heads to int8, one bf16 scale per 128 weights. The other layers of the assistant head, the vision tower and the audio encoder stay bf16.
rad-convert gemma-4-12B-it --draft-model gemma-4-12B-it-assistant \
    --recipe gemma4-12b-a4.recipe --set quantization_config.quant_method=compressed-tensors \
    -o gemma4-12b-a4.rad

GPTQ needs statistics from the calibration set (calib= in the recipe). radiance 2.0.0's rad-convert factors each statistics matrix on one CPU thread, which takes hours at this size. So the factors (inverse Cholesky factor of the statistics plus damping) came from the card instead: fp64 up to 8,192 inputs, fp32 above. A small change to libquant reads them.

Calibration: 662 rows and 708,764 tokens of keptn/quant-calibration-v1, rendered with the chat template of Gemma 4 31B-it.

Limits

  • radiance runs on AMD RDNA4 cards.
  • The file needs radiance 2.0.0 or newer.
  • Speed is measured on four cards only.
  • Images and audio are measured with the bf16 file only. This file uses the same vision tower and audio encoder, but the comparison is not run on it.

Credits

License

Apache 2.0, the license of the base model. See LICENSE.

Downloads last month
19
Inference Providers NEW
This model isn't deployed by any Inference Provider. 馃檵 Ask for provider support

Model tree for keptn/gemma4-12b-a4

Finetuned
(309)
this model