File size: 3,890 Bytes
0796452
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
---
license: gemma
base_model:
- google/translategemma-27b-it
pipeline_tag: translation
tags:
- radiance
- rocm
- amd
- rdna4
- int8
- gemma3
- translation
---

# TranslateGemma 27B, int8 — radiance container

[google/translategemma-27b-it](https://huggingface.co/google/translategemma-27b-it) as a single
`.rad` container for the [radiance](https://codeberg.org/StillDeadcode/radiance) inference engine
(AMD RDNA4, ROCm), with every linear quantised to int8.

Gemma is provided under and subject to the Gemma Terms of Use found at
[ai.google.dev/gemma/terms](https://ai.google.dev/gemma/terms). This file is a modified form of
Google's weights (quantised, see below); see `NOTICE`.

| | |
|---|---|
| File | `translategemma-27b-i8.rad` — 28.18 GiB |
| Weights | every attention and feed-forward linear and the lm_head int8 with a bf16 scale per 128 columns, round to nearest (no calibration); norms and the embedding bf16 |
| Activations | int8 with a scale per (row, 128 columns): Gemma 3 carries a few channels hundreds of times the rest, which one scale a row would spend its codes on |
| Modalities | text. The checkpoint's vision tower is not in the container; image translation is not served |
| Context | 131,072 tokens trained |
| Needs | radiance 1.17.0 or newer |

## Serve

TranslateGemma's own chat template takes structured turns that name the source and target
language, so a plain Gemma 3 template (`gemma3.jinja` in this repository) stands in and a client
sends the prompt the checkpoint's template renders on `/v1/completions`.

```sh
radiance --model translategemma-27b-i8.rad --tp 2 --max-model-len 16384 --max-num-seqs 8 \
    --kv-cache-dtype bf16 --override-chat-template gemma3.jinja --host 0.0.0.0 --port 8000
```

The Docker image is `stilldeadcode/radiance`. A request:

```sh
curl http://localhost:8000/v1/completions -H 'Content-Type: application/json' -d '{
  "model": "translategemma-27b-i8", "max_tokens": 200, "temperature": 0, "stop": ["<end_of_turn>"],
  "prompt": "<start_of_turn>user\nYou are a professional English (en) to German (de) translator. Your goal is to accurately convey the meaning and nuances of the original English text while adhering to German grammar, vocabulary, and cultural sensitivities.\nProduce only the German translation, without any additional explanations or commentary. Please translate the following English text into German:\n\n\nThe train to Munich is about twenty minutes late today.<end_of_turn>\n<start_of_turn>model\n"
}'
```

Checked on radiance 1.17.0 on two Radeon AI PRO R9700 (gfx1201): the container loads and
translates English to German, English to Czech and German to English correctly. Measured on the
same cards with this model's decode kernels launched as 1.17.0 launches them, one sequence takes
about 41 ms a token (24 tokens/s); engine builds after 1.17.0 launch them chained and take about
26 ms (38 tokens/s), with the same output.

## How this file was made

```sh
rad-convert google/translategemma-27b-it \
    --recipe translategemma-27b-i8a8.recipe -o translategemma-27b-i8.rad
```

The recipe (`translategemma-27b-i8a8.recipe` in this repository). The first rule that matches a
weight decides it, and everything no rule names is the checkpoint's bf16. Its header records a KL
divergence of 0.003 against the bf16 model:

```
blk.*.attn_q.weight       rtn  codes=i8 group=128 scale=bf16 clamp=sym
blk.*.attn_k.weight       rtn  codes=i8 group=128 scale=bf16 clamp=sym
blk.*.attn_v.weight       rtn  codes=i8 group=128 scale=bf16 clamp=sym
blk.*.attn_o.weight       rtn  codes=i8 group=128 scale=bf16 clamp=sym
blk.*.ffn_down.weight     rtn  codes=i8 group=128 scale=bf16 clamp=sym
blk.*.ffn_gate_up.weight  rtn  codes=i8 group=128 scale=bf16 clamp=sym
output.weight             rtn  codes=i8 group=128 scale=bf16 clamp=sym
```

## License

Gemma Terms of Use, as the base model; see `NOTICE`.