keptn/gemma4-12b-i8g

Gemma 4 12B-it · int8 · for radiance

google/gemma-4-12B-it as one 13.79 GiB file for the radiance inference engine on AMD RDNA4. The weights of the trunk are int8 with one scale per 128 weights, rounded to nearest.

File	gemma4-12b-i8g.rad, 13.79 GiB
Needs	radiance with PR #338, not in a release yet; 1, 2 or 4 cards with 32 GB (tested on Radeon AI PRO R9700)
Format	weights int8, one bf16 scale per 128 weights; activations int8, one scale per token and 128 values
Speculation	Google's Gemma 4 assistant head, 3 draft tokens
Inputs	text · image · audio
Context	262,144 tokens
Version	1.0 (2026-10-30)

Speed

Radeon AI PRO R9700 at 215 W, radiance gemma4-12b branch 75664cc0, bf16 KV, 3 draft tokens, thinking off. BetterBench 0.6.1, full run, temperature 0.7. Tokens per second (median):

	1 card	2 cards	4 cards
decode, one stream	104.6	174.7	250.0
prefill at 24k tokens	3,597	5,883	7,069
4 streams, total	306	506	714

Quality

Distance to the bf16 original on the 80 MT-Bench questions, both turns. The text is the GPT-4 answers from lmsys/mt_bench_human_judgments, with the model's chat template and thinking off. The reference is the bf16 original in HF transformers, with fp32 arithmetic and the full vocabulary. radiance scores this file with --kld-ref, on the answer tokens only (42,283 tokens).

	mean KL	99th percentile KL	top token the same	perplexity ratio
this file	0.00416	0.0535	98.1%	×1.004
keptn/gemma4-12b-bf16	0.00026	0.00325	99.6%	×1.000

Quick start

No radiance release has this model yet. Build the image from PR #338 first:

git clone https://codeberg.org/StillDeadcode/radiance.git && cd radiance
git fetch origin pull/338/head:pr-338 && git checkout pr-338
docker/build.sh -t gfx1201                   # gives the image radiance:latest

Then use radiance:latest as the image (RADIANCE_IMAGE=radiance:latest in .env).

With the compose files from the radiance repository (radiance with PR #338, not in a release yet). Set RADIANCE_IMAGE, RADIANCE_MODELS, RADIANCE_API_KEY and the group ids in .env, see radiance's README:

hf download keptn/gemma4-12b-i8g gemma4-12b-i8g.rad --local-dir ~/models
git clone https://codeberg.org/StillDeadcode/radiance.git
cp -r radiance/deploy/compose ~/radiance-compose && cd ~/radiance-compose
cp .env.example .env
docker compose -f gemma4-12b-i8g.yaml up -d

Or directly (two cards):

hf download keptn/gemma4-12b-i8g gemma4-12b-i8g.rad --local-dir ~/models
docker run -d --init --name gemma4 --network host --group-add video --group-add render \
    --device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \
    -v ~/models:/models:ro -e ROCR_VISIBLE_DEVICES=0,1 \
    radiance:latest --model /models/gemma4-12b-i8g.rad --host 0.0.0.0 --port 8000 \
    --tp 2 --p2p on --kv-cache-dtype bf16 --max-model-len 262144 --max-num-seqs 8 \
    --num-speculative-tokens 3 --max-num-batched-tokens 2048 \
    --default-max-tokens 32768 --max-tokens-cap 65536 \
    --temp 1.0 --top-p 0.95 --top-k 64 --chat-template-kwargs '{"enable_thinking":false}'

	change
one card	ROCR_VISIBLE_DEVICES=0, drop --tp 2 --p2p on
four cards	ROCR_VISIBLE_DEVICES=0,1,2,3, --tp 4
no drafter	--num-speculative-tokens 0
thinking	per request: "chat_template_kwargs": {"enable_thinking": true}

The sampling values are the ones used in the measured runs (temperature 1.0, top_p 0.95, top_k 64).

Memory and context

KV pool at --max-model-len 262144, bf16 KV, radiance gemma4-12b branch 75664cc0, from radiance's startup report. Tokens of context in total, shared by all sequences:

	2 sequences	4 sequences	8 sequences
1 card	524,336	1,048,624	1,048,112
2 cards	524,336	1,048,624	1,547,056
4 cards	524,336	1,048,624	1,792,432

Images, audio and video

The same inputs go to radiance and to the bf16 original in HF transformers. Greedy, thinking off, 2 cards, radiance PR #338 final (67866b87). There is one measurement for each vision or audio encoder, made with gemma4-12b-bf16.rad.

input	cases	same answer	mean KL	top token the same
images (DocVQA and AI2D, fixed selection)	20	20/20	0.00184	149/149
audio (LibriSpeech test-clean, 12 clips)	12	11/12	0.00068	350/351

Word error rate on the audio clips: HF 6.68% · radiance 6.81%.

How it was made

1. Downloaded google/gemma-4-12B-it (revision 707f0a3b) and google/gemma-4-12B-it-assistant (revision 3b0b0a4a).
2. Converted the checkpoint with the recipe gemma4-12b-i8g, reusing the tensors of the bf16 file. Rounded to nearest at int8, one bf16 scale per 128 weights: the attention, the feed-forward layers, the lm_head and the assistant head.
3. Left the vision tower, the audio encoder and the norms as in the checkpoint.
4. Checked that rad-convert reads all 725 checkpoint tensors.

rad-convert gemma-4-12B-it --draft-model gemma-4-12B-it-assistant \
    --set quantization_config.quant_method=compressed-tensors \
    --recipe gemma4-12b-i8g.recipe --reuse gemma4-12b-bf16.rad -o gemma4-12b-i8g.rad

Calibration: no calibration data.

Model card with: radiance-model-cards-framework 0.3 (codeberg.org/keptn42/radiance-model-cards-framework).

Limits

• radiance runs on AMD RDNA4 cards.
• The file needs radiance with PR #338. No release has it yet.
• The trunk is rounded to nearest with no calibration data. The quality against the original is not measured yet.
• Images and audio are measured with the bf16 file only. This file uses the same vision tower and audio encoder, but the comparison is not run on it.
• Google lists text, image and audio as inputs. Audio is tested on four LibriSpeech clips only.

Credits

• Base model, assistant head, vision tower and audio encoder: Google.
• radiance: StillDeadcode (codeberg.org/StillDeadcode/radiance). The i8g format comes from radiance's Gemma 3 support.
• Test sets: MT-Bench, DocVQA, AI2D, LibriSpeech.

License

Apache 2.0, the license of the base model. See LICENSE.