|
Download README.md from StillDeadcode/translategemma-27b-i8: direct link, hf CLI and curl.
- Browser
- Download file 3.89 kB
-
https://huggingface.co/StillDeadcode/translategemma-27b-i8/resolve/main/README.md
- Command line
-
hf download hf://StillDeadcode/translategemma-27b-i8/README.md
-
curl -L -o README.md https://huggingface.co/StillDeadcode/translategemma-27b-i8/resolve/main/README.md
3.89 kB
| license: gemma | |
| base_model: | |
| - google/translategemma-27b-it | |
| pipeline_tag: translation | |
| tags: | |
| - radiance | |
| - rocm | |
| - amd | |
| - rdna4 | |
| - int8 | |
| - gemma3 | |
| - translation | |
| # TranslateGemma 27B, int8 — radiance container | |
| [google/translategemma-27b-it](https://huggingface.co/google/translategemma-27b-it) as a single | |
| `.rad` container for the [radiance](https://codeberg.org/StillDeadcode/radiance) inference engine | |
| (AMD RDNA4, ROCm), with every linear quantised to int8. | |
| Gemma is provided under and subject to the Gemma Terms of Use found at | |
| [ai.google.dev/gemma/terms](https://ai.google.dev/gemma/terms). This file is a modified form of | |
| Google's weights (quantised, see below); see `NOTICE`. | |
| | | | | |
| |---|---| | |
| | File | `translategemma-27b-i8.rad` — 28.18 GiB | | |
| | Weights | every attention and feed-forward linear and the lm_head int8 with a bf16 scale per 128 columns, round to nearest (no calibration); norms and the embedding bf16 | | |
| | Activations | int8 with a scale per (row, 128 columns): Gemma 3 carries a few channels hundreds of times the rest, which one scale a row would spend its codes on | | |
| | Modalities | text. The checkpoint's vision tower is not in the container; image translation is not served | | |
| | Context | 131,072 tokens trained | | |
| | Needs | radiance 1.17.0 or newer | | |
| ## Serve | |
| TranslateGemma's own chat template takes structured turns that name the source and target | |
| language, so a plain Gemma 3 template (`gemma3.jinja` in this repository) stands in and a client | |
| sends the prompt the checkpoint's template renders on `/v1/completions`. | |
| ```sh | |
| radiance --model translategemma-27b-i8.rad --tp 2 --max-model-len 16384 --max-num-seqs 8 \ | |
| --kv-cache-dtype bf16 --override-chat-template gemma3.jinja --host 0.0.0.0 --port 8000 | |
| ``` | |
| The Docker image is `stilldeadcode/radiance`. A request: | |
| ```sh | |
| curl http://localhost:8000/v1/completions -H 'Content-Type: application/json' -d '{ | |
| "model": "translategemma-27b-i8", "max_tokens": 200, "temperature": 0, "stop": ["<end_of_turn>"], | |
| "prompt": "<start_of_turn>user\nYou are a professional English (en) to German (de) translator. Your goal is to accurately convey the meaning and nuances of the original English text while adhering to German grammar, vocabulary, and cultural sensitivities.\nProduce only the German translation, without any additional explanations or commentary. Please translate the following English text into German:\n\n\nThe train to Munich is about twenty minutes late today.<end_of_turn>\n<start_of_turn>model\n" | |
| }' | |
| ``` | |
| Checked on radiance 1.17.0 on two Radeon AI PRO R9700 (gfx1201): the container loads and | |
| translates English to German, English to Czech and German to English correctly. Measured on the | |
| same cards with this model's decode kernels launched as 1.17.0 launches them, one sequence takes | |
| about 41 ms a token (24 tokens/s); engine builds after 1.17.0 launch them chained and take about | |
| 26 ms (38 tokens/s), with the same output. | |
| ## How this file was made | |
| ```sh | |
| rad-convert google/translategemma-27b-it \ | |
| --recipe translategemma-27b-i8a8.recipe -o translategemma-27b-i8.rad | |
| ``` | |
| The recipe (`translategemma-27b-i8a8.recipe` in this repository). The first rule that matches a | |
| weight decides it, and everything no rule names is the checkpoint's bf16. Its header records a KL | |
| divergence of 0.003 against the bf16 model: | |
| ``` | |
| blk.*.attn_q.weight rtn codes=i8 group=128 scale=bf16 clamp=sym | |
| blk.*.attn_k.weight rtn codes=i8 group=128 scale=bf16 clamp=sym | |
| blk.*.attn_v.weight rtn codes=i8 group=128 scale=bf16 clamp=sym | |
| blk.*.attn_o.weight rtn codes=i8 group=128 scale=bf16 clamp=sym | |
| blk.*.ffn_down.weight rtn codes=i8 group=128 scale=bf16 clamp=sym | |
| blk.*.ffn_gate_up.weight rtn codes=i8 group=128 scale=bf16 clamp=sym | |
| output.weight rtn codes=i8 group=128 scale=bf16 clamp=sym | |
| ``` | |
| ## License | |
| Gemma Terms of Use, as the base model; see `NOTICE`. | |