translategemma-4b-it-FP8
FP8 quantization of google/translategemma-4b-it for vLLM. This is an unofficial build, not affiliated with Google.
- COMET-22 on WMT24++ EU pairs is 82.79, against 82.80 for BF16. The difference is noise.
- Batched short-cell translation runs at 176 cells/s, 1.4× BF16's 123.
- Weights take 4.4 GiB in vLLM with text-only serving, against 7.4 GiB for BF16. The download is 5.1 GB.
- vLLM's OpenAI-compatible server accepts a plain-string prompt. Google's structured message format still works.
Every language-model linear layer has FP8 E4M3 weights with one scale per output channel. vLLM quantizes
activations to FP8 per token at run time, so no calibration data is involved. The vision tower, the projector and the
embeddings stay in BF16. The recipe is in quantization/.
FP8 needs an Ada, Hopper or Blackwell GPU. If you want more speed and can give up half a COMET point, the NVFP4 build runs at about 250 to 280 cells/s on Blackwell.
Changes from a plain llm-compressor export
- The duplicate
lm_head.weightis gone. The embeddings are tied, so vLLM never reads it, and the download is 1.3 GB smaller. max_position_embeddingsis 32,768 instead of 131,072. vLLM precomputes the RoPE table of the global layers for 8 ×max_position_embeddingspositions because of the linear ×8 scaling, and that memory comes out of the KV cache. TranslateGemma's own card gives a 2K-token input context.chat_template.jinjaalso accepts<<<source>>>CODE<<<target>>>CODE<<<text>>>TEXTas a plain string or as a single text part. vLLM 0.26's server drops thesource_lang_codeandtarget_lang_codefields from chat messages, so Google's structured format never reaches the template there. The plain-string form renders exactly the same prompt.
I regenerated all 8,640 test segments from this repo with plain-string prompts, and they match the outputs of the
unmodified export. NOTICE lists these changes as the Gemma terms require.
Quality
Test set. All non-canary segments of WMT24++, google/wmt24pp. That's 960 per direction and 8,640 in total,
covering news, social, speech and literary text. EN→X uses the original English source. PL→EN reverses the EN→PL
pair, so the Polish post-edited reference becomes the source and the original English becomes the reference.
Metrics. COMET-22 with Unbabel/wmt22-comet-da, scaled ×100, and corpus chrF from sacrebleu.
Setup. Greedy decoding on vLLM 0.26.0. One desktop RTX 5070 Ti 16 GB, capped at an 11.5 GB budget with
gpu_memory_utilization ≈ 0.74 to mimic a 12 GB laptop GPU.
| EN→CS | EN→DE | EN→ES | EN→FR | EN→IT | EN→NL | EN→PL | EN→PT | PL→EN | All | chrF | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| BF16 reference | 84.43 | 81.86 | 82.80 | 81.24 | 83.84 | 83.46 | 83.77 | 82.60 | 81.16 | 82.80 | 55.4 |
| this model, FP8 | 84.49 | 81.84 | 82.80 | 81.10 | 84.00 | 83.54 | 83.79 | 82.55 | 81.03 | 82.79 | 55.3 |
Other TranslateGemma-4B builds
Same tests, same 11.5 GB budget. Cells/s comes from 4,000 short cells, EN→X and 1 to 40 words each, submitted in one batch. Structure is protected tokens kept / cells fully intact, from the test described below.
| build | COMET-22 | vs BF16 | structure | cells/s |
|---|---|---|---|---|
| BF16 | 82.80 | – | 81.5% / 84.0% | 123 |
| this model | 82.79 | −0.01 | 80.0% / 82.9% | 176 |
| ViralityLeo/vllm-translategemma-4b-it-FP8-Dynamic | 82.68 | −0.12 | 79.3% / 81.7% | 175 |
| translategemma-4b-it-NVFP4 | 82.32 | −0.48 | 72.6% / 74.9% | ~250–280 |
The ViralityLeo build only accepts the plain-string prompt, so I ran it that way.
Structure test. 175 Excel-style cells in each of PL, DE, FR, ES and CS, with 270 tokens that must survive
verbatim: {placeholders}, %s, %1$s, {{var}}, ${var}, HTML tags, URLs, e-mails, SKUs, inline code and line
breaks. BF16 TranslateGemma-4B keeps 81.5% because it rewrites <b> tags and translates [PENDING] labels. This
build loses another 1.5 points on top of that.
Which model to use
On a 12 GB card I'd pick Hy-MT2-7B-NVFP4. On the same tests it scored 84.66 COMET-22, ran 213 cells/s and kept 99.3% of protected tokens with its keep-structure prompt. This build makes sense on 8 GB cards, on GPUs without FP4 support, or next to other models on a shared GPU.
Usage with vLLM
Tested with vLLM 0.26.0, text only.
vllm serve kurcontko/translategemma-4b-it-FP8 --language-model-only --max-model-len 2048 \
--gpu-memory-utilization 0.9
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="-")
text = "Warning: do not exceed 230 V.\nDisconnect power before cleaning."
r = client.chat.completions.create(model="kurcontko/translategemma-4b-it-FP8", temperature=0,
messages=[{"role": "user", "content": f"<<<source>>>en<<<target>>>pl<<<text>>>{text}"}])
print(r.choices[0].message.content)
Language codes come from the base model's template, for example en, pl, de-DE, pt-PT or es-MX. With
transformers, the structured message format from the base model card works unchanged.
If the engine fails at start-up while JIT-compiling FlashInfer's sampling kernels with CUDA compiler and CUDA toolkit headers are incompatible, set VLLM_USE_FLASHINFER_SAMPLER=0.
Limitations
- Evaluated on the 9 directions above only, with COMET-22 and chrF. No human evaluation.
- I have not tested image translation with this checkpoint. The vision tower is the unchanged BF16 one.
License
Gemma is provided under and subject to the Gemma Terms of Use found at
ai.google.dev/gemma/terms. Use of this model must comply with the
Gemma Prohibited Use Policy. This repository holds a modified,
quantized version of google/translategemma-4b-it. The changes are listed above and in NOTICE.
- Downloads last month
- 23
Model tree for kurcontko/translategemma-4b-it-FP8
Base model
google/translategemma-4b-it