TranslateGemma 27B, int8 β radiance container
google/translategemma-27b-it as a single
.rad container for the radiance inference engine
(AMD RDNA4, ROCm), with every linear quantised to int8.
Gemma is provided under and subject to the Gemma Terms of Use found at
ai.google.dev/gemma/terms. This file is a modified form of
Google's weights (quantised, see below); see NOTICE.
| File | translategemma-27b-i8.rad β 28.18 GiB |
| Weights | every attention and feed-forward linear and the lm_head int8 with a bf16 scale per 128 columns, round to nearest (no calibration); norms and the embedding bf16 |
| Activations | int8 with a scale per (row, 128 columns): Gemma 3 carries a few channels hundreds of times the rest, which one scale a row would spend its codes on |
| Modalities | text. The checkpoint's vision tower is not in the container; image translation is not served |
| Context | 131,072 tokens trained |
| Needs | radiance 1.17.0 or newer |
Serve
TranslateGemma's own chat template takes structured turns that name the source and target
language, so a plain Gemma 3 template (gemma3.jinja in this repository) stands in and a client
sends the prompt the checkpoint's template renders on /v1/completions.
radiance --model translategemma-27b-i8.rad --tp 2 --max-model-len 16384 --max-num-seqs 8 \
--kv-cache-dtype bf16 --override-chat-template gemma3.jinja --host 0.0.0.0 --port 8000
The Docker image is stilldeadcode/radiance. A request:
curl http://localhost:8000/v1/completions -H 'Content-Type: application/json' -d '{
"model": "translategemma-27b-i8", "max_tokens": 200, "temperature": 0, "stop": ["<end_of_turn>"],
"prompt": "<start_of_turn>user\nYou are a professional English (en) to German (de) translator. Your goal is to accurately convey the meaning and nuances of the original English text while adhering to German grammar, vocabulary, and cultural sensitivities.\nProduce only the German translation, without any additional explanations or commentary. Please translate the following English text into German:\n\n\nThe train to Munich is about twenty minutes late today.<end_of_turn>\n<start_of_turn>model\n"
}'
Checked on radiance 1.17.0 on two Radeon AI PRO R9700 (gfx1201): the container loads and translates English to German, English to Czech and German to English correctly. Measured on the same cards with this model's decode kernels launched as 1.17.0 launches them, one sequence takes about 41 ms a token (24 tokens/s); engine builds after 1.17.0 launch them chained and take about 26 ms (38 tokens/s), with the same output.
How this file was made
rad-convert google/translategemma-27b-it \
--recipe translategemma-27b-i8a8.recipe -o translategemma-27b-i8.rad
The recipe (translategemma-27b-i8a8.recipe in this repository). The first rule that matches a
weight decides it, and everything no rule names is the checkpoint's bf16. Its header records a KL
divergence of 0.003 against the bf16 model:
blk.*.attn_q.weight rtn codes=i8 group=128 scale=bf16 clamp=sym
blk.*.attn_k.weight rtn codes=i8 group=128 scale=bf16 clamp=sym
blk.*.attn_v.weight rtn codes=i8 group=128 scale=bf16 clamp=sym
blk.*.attn_o.weight rtn codes=i8 group=128 scale=bf16 clamp=sym
blk.*.ffn_down.weight rtn codes=i8 group=128 scale=bf16 clamp=sym
blk.*.ffn_gate_up.weight rtn codes=i8 group=128 scale=bf16 clamp=sym
output.weight rtn codes=i8 group=128 scale=bf16 clamp=sym
License
Gemma Terms of Use, as the base model; see NOTICE.
Model tree for StillDeadcode/translategemma-27b-i8
Base model
google/translategemma-27b-it