keptn/gemma4-12b-bf16

Gemma 4 12B-it · bf16 · for radiance

google/gemma-4-12B-it as one 24.95 GiB file for the radiance inference engine on AMD RDNA4. The weights are as Google ships them, with Google's assistant head for speculative decoding.

File	gemma4-12b-bf16.rad, 24.95 GiB
Needs	radiance with PR #338, not in a release yet; 1, 2 or 4 cards with 32 GB (tested on Radeon AI PRO R9700)
Format	weights bf16, as in the checkpoint
Speculation	Google's Gemma 4 assistant head, 3 draft tokens
Inputs	text · image · audio
Context	262,144 tokens
Version	1.0 (2026-10-30)

Speed

Radeon AI PRO R9700 at 215 W, radiance gemma4-12b branch 75664cc0, bf16 KV, 3 draft tokens, thinking off. BetterBench 0.6.1, full run, temperature 0.7. Tokens per second (median):

	1 card	2 cards	4 cards
decode, one stream	62.7	113.3	182.8
prefill at 24k tokens	2,743	4,620	6,191
4 streams, total	163	300	474

Quality

Distance to the bf16 original on the 80 MT-Bench questions, both turns. The text is the GPT-4 answers from lmsys/mt_bench_human_judgments, with the model's chat template and thinking off. The reference is the bf16 original in HF transformers, with fp32 arithmetic and the full vocabulary. radiance scores this file with --kld-ref, on the answer tokens only.

	mean KL	99th percentile KL	top token the same	perplexity ratio
this file	–	–	–	–

Quick start

No radiance release has this model yet. Build the image from PR #338 first:

git clone https://codeberg.org/StillDeadcode/radiance.git && cd radiance
git fetch origin pull/338/head:pr-338 && git checkout pr-338
docker/build.sh -t gfx1201                   # gives the image radiance:latest

Then use radiance:latest as the image (RADIANCE_IMAGE=radiance:latest in .env).

With the compose files from the radiance repository (radiance with PR #338, not in a release yet). Set RADIANCE_IMAGE, RADIANCE_MODELS, RADIANCE_API_KEY and the group ids in .env, see radiance's README:

hf download keptn/gemma4-12b-bf16 gemma4-12b-bf16.rad --local-dir ~/models
git clone https://codeberg.org/StillDeadcode/radiance.git
cp -r radiance/deploy/compose ~/radiance-compose && cd ~/radiance-compose
cp .env.example .env
docker compose -f gemma4-12b-bf16.yaml up -d

Or directly (two cards):

hf download keptn/gemma4-12b-bf16 gemma4-12b-bf16.rad --local-dir ~/models
docker run -d --init --name gemma4 --network host --group-add video --group-add render \
    --device /dev/kfd --device /dev/dri --security-opt seccomp=unconfined --ulimit memlock=-1:-1 \
    -v ~/models:/models:ro -e ROCR_VISIBLE_DEVICES=0,1 \
    radiance:latest --model /models/gemma4-12b-bf16.rad --host 0.0.0.0 --port 8000 \
    --tp 2 --p2p on --kv-cache-dtype bf16 --max-model-len 262144 --max-num-seqs 8 \
    --num-speculative-tokens 3 --max-num-batched-tokens 2048 \
    --default-max-tokens 32768 --max-tokens-cap 65536 \
    --temp 1.0 --top-p 0.95 --top-k 64 --chat-template-kwargs '{"enable_thinking":false}'

	change
one card	ROCR_VISIBLE_DEVICES=0, drop --tp 2 --p2p on
four cards	ROCR_VISIBLE_DEVICES=0,1,2,3, --tp 4
no drafter	--num-speculative-tokens 0
thinking	per request: "chat_template_kwargs": {"enable_thinking": true}

The sampling values are the ones used in the measured runs (temperature 1.0, top_p 0.95, top_k 64).

Memory and context

KV pool at --max-model-len 262144, bf16 KV, radiance gemma4-12b branch 75664cc0, from radiance's startup report. Tokens of context in total, shared by all sequences:

	2 sequences	4 sequences	8 sequences
1 card	451,184	408,752	322,928
2 cards	524,336	1,048,624	1,184,112
4 cards	524,336	1,048,624	1,610,544

Images, audio and video

The same inputs go to radiance and to the bf16 original in HF transformers. Greedy, thinking off, 2 cards, radiance PR #338 (67da54bd). There is one measurement for each vision or audio encoder, made with gemma4-12b-bf16.rad.

input	cases	same answer	mean KL	top token the same
images (two test images)	20	20/20	0.00195	149/149
audio (LibriSpeech test-clean, 4 clips)	12	4/12	0.00207	567/571

Word error rate on the audio clips: HF 265.24% · radiance 241.05%.

How it was made

1. Downloaded google/gemma-4-12B-it (revision 707f0a3b) and google/gemma-4-12B-it-assistant (revision 3b0b0a4a).
2. Converted the checkpoint with the recipe gemma4-12b-bf16: every weight stays bf16 as in the checkpoint. The position table of the vision tower is stored as two taps.
3. Added the assistant head as the drafter. The vision tower and the audio encoder stay as in the checkpoint.
4. Checked that rad-convert reads all 725 checkpoint tensors.

rad-convert gemma-4-12B-it --draft-model gemma-4-12B-it-assistant \
    --set quantization_config.quant_method=compressed-tensors \
    --recipe gemma4-12b-bf16.recipe -o gemma4-12b-bf16.rad

Calibration: no calibration data.

Model card with: radiance-model-cards-framework 0.3 (codeberg.org/keptn42/radiance-model-cards-framework).

Limits

• radiance runs on AMD RDNA4 cards.
• The file needs radiance with PR #338. No release has it yet.
• One card holds the file (24.95 GiB) and the KV cache for 451,184 tokens at 2 sequences. Eight sequences at the full context need more cards.
• Google lists text, image and audio as inputs. Audio is tested on four LibriSpeech clips only.

Credits

• Base model, assistant head, vision tower and audio encoder: Google.
• radiance: StillDeadcode (codeberg.org/StillDeadcode/radiance).
• Test sets: MT-Bench, DocVQA, AI2D, LibriSpeech.

License

Apache 2.0, the license of the base model. See LICENSE.