StillDeadcode's picture
int8 container
0796452 verified
|
Raw History Blame Contribute Delete
3.89 kB
metadata
license: gemma
base_model:
  - google/translategemma-27b-it
pipeline_tag: translation
tags:
  - radiance
  - rocm
  - amd
  - rdna4
  - int8
  - gemma3
  - translation

TranslateGemma 27B, int8 — radiance container

google/translategemma-27b-it as a single .rad container for the radiance inference engine (AMD RDNA4, ROCm), with every linear quantised to int8.

Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms. This file is a modified form of Google's weights (quantised, see below); see NOTICE.

File translategemma-27b-i8.rad — 28.18 GiB
Weights every attention and feed-forward linear and the lm_head int8 with a bf16 scale per 128 columns, round to nearest (no calibration); norms and the embedding bf16
Activations int8 with a scale per (row, 128 columns): Gemma 3 carries a few channels hundreds of times the rest, which one scale a row would spend its codes on
Modalities text. The checkpoint's vision tower is not in the container; image translation is not served
Context 131,072 tokens trained
Needs radiance 1.17.0 or newer

Serve

TranslateGemma's own chat template takes structured turns that name the source and target language, so a plain Gemma 3 template (gemma3.jinja in this repository) stands in and a client sends the prompt the checkpoint's template renders on /v1/completions.

radiance --model translategemma-27b-i8.rad --tp 2 --max-model-len 16384 --max-num-seqs 8 \
    --kv-cache-dtype bf16 --override-chat-template gemma3.jinja --host 0.0.0.0 --port 8000

The Docker image is stilldeadcode/radiance. A request:

curl http://localhost:8000/v1/completions -H 'Content-Type: application/json' -d '{
  "model": "translategemma-27b-i8", "max_tokens": 200, "temperature": 0, "stop": ["<end_of_turn>"],
  "prompt": "<start_of_turn>user\nYou are a professional English (en) to German (de) translator. Your goal is to accurately convey the meaning and nuances of the original English text while adhering to German grammar, vocabulary, and cultural sensitivities.\nProduce only the German translation, without any additional explanations or commentary. Please translate the following English text into German:\n\n\nThe train to Munich is about twenty minutes late today.<end_of_turn>\n<start_of_turn>model\n"
}'

Checked on radiance 1.17.0 on two Radeon AI PRO R9700 (gfx1201): the container loads and translates English to German, English to Czech and German to English correctly. Measured on the same cards with this model's decode kernels launched as 1.17.0 launches them, one sequence takes about 41 ms a token (24 tokens/s); engine builds after 1.17.0 launch them chained and take about 26 ms (38 tokens/s), with the same output.

How this file was made

rad-convert google/translategemma-27b-it \
    --recipe translategemma-27b-i8a8.recipe -o translategemma-27b-i8.rad

The recipe (translategemma-27b-i8a8.recipe in this repository). The first rule that matches a weight decides it, and everything no rule names is the checkpoint's bf16. Its header records a KL divergence of 0.003 against the bf16 model:

blk.*.attn_q.weight       rtn  codes=i8 group=128 scale=bf16 clamp=sym
blk.*.attn_k.weight       rtn  codes=i8 group=128 scale=bf16 clamp=sym
blk.*.attn_v.weight       rtn  codes=i8 group=128 scale=bf16 clamp=sym
blk.*.attn_o.weight       rtn  codes=i8 group=128 scale=bf16 clamp=sym
blk.*.ffn_down.weight     rtn  codes=i8 group=128 scale=bf16 clamp=sym
blk.*.ffn_gate_up.weight  rtn  codes=i8 group=128 scale=bf16 clamp=sym
output.weight             rtn  codes=i8 group=128 scale=bf16 clamp=sym

License

Gemma Terms of Use, as the base model; see NOTICE.