translategemma-4b-it-FP8

FP8 quantization of google/translategemma-4b-it for vLLM. This is an unofficial build, not affiliated with Google.

  • COMET-22 on WMT24++ EU pairs is 82.79, against 82.80 for BF16. The difference is noise.
  • Batched short-cell translation runs at 176 cells/s, 1.4× BF16's 123.
  • Weights take 4.4 GiB in vLLM with text-only serving, against 7.4 GiB for BF16. The download is 5.1 GB.
  • vLLM's OpenAI-compatible server accepts a plain-string prompt. Google's structured message format still works.

Every language-model linear layer has FP8 E4M3 weights with one scale per output channel. vLLM quantizes activations to FP8 per token at run time, so no calibration data is involved. The vision tower, the projector and the embeddings stay in BF16. The recipe is in quantization/.

FP8 needs an Ada, Hopper or Blackwell GPU. If you want more speed and can give up half a COMET point, the NVFP4 build runs at about 250 to 280 cells/s on Blackwell.

Changes from a plain llm-compressor export

  • The duplicate lm_head.weight is gone. The embeddings are tied, so vLLM never reads it, and the download is 1.3 GB smaller.
  • max_position_embeddings is 32,768 instead of 131,072. vLLM precomputes the RoPE table of the global layers for 8 × max_position_embeddings positions because of the linear ×8 scaling, and that memory comes out of the KV cache. TranslateGemma's own card gives a 2K-token input context.
  • chat_template.jinja also accepts <<<source>>>CODE<<<target>>>CODE<<<text>>>TEXT as a plain string or as a single text part. vLLM 0.26's server drops the source_lang_code and target_lang_code fields from chat messages, so Google's structured format never reaches the template there. The plain-string form renders exactly the same prompt.

I regenerated all 8,640 test segments from this repo with plain-string prompts, and they match the outputs of the unmodified export. NOTICE lists these changes as the Gemma terms require.

Quality

Test set. All non-canary segments of WMT24++, google/wmt24pp. That's 960 per direction and 8,640 in total, covering news, social, speech and literary text. EN→X uses the original English source. PL→EN reverses the EN→PL pair, so the Polish post-edited reference becomes the source and the original English becomes the reference.

Metrics. COMET-22 with Unbabel/wmt22-comet-da, scaled ×100, and corpus chrF from sacrebleu.

Setup. Greedy decoding on vLLM 0.26.0. One desktop RTX 5070 Ti 16 GB, capped at an 11.5 GB budget with gpu_memory_utilization ≈ 0.74 to mimic a 12 GB laptop GPU.

EN→CS EN→DE EN→ES EN→FR EN→IT EN→NL EN→PL EN→PT PL→EN All chrF
BF16 reference 84.43 81.86 82.80 81.24 83.84 83.46 83.77 82.60 81.16 82.80 55.4
this model, FP8 84.49 81.84 82.80 81.10 84.00 83.54 83.79 82.55 81.03 82.79 55.3

Other TranslateGemma-4B builds

Same tests, same 11.5 GB budget. Cells/s comes from 4,000 short cells, EN→X and 1 to 40 words each, submitted in one batch. Structure is protected tokens kept / cells fully intact, from the test described below.

build COMET-22 vs BF16 structure cells/s
BF16 82.80 – 81.5% / 84.0% 123
this model 82.79 −0.01 80.0% / 82.9% 176
ViralityLeo/vllm-translategemma-4b-it-FP8-Dynamic 82.68 −0.12 79.3% / 81.7% 175
translategemma-4b-it-NVFP4 82.32 −0.48 72.6% / 74.9% ~250–280

The ViralityLeo build only accepts the plain-string prompt, so I ran it that way.

Structure test. 175 Excel-style cells in each of PL, DE, FR, ES and CS, with 270 tokens that must survive verbatim: {placeholders}, %s, %1$s, {{var}}, ${var}, HTML tags, URLs, e-mails, SKUs, inline code and line breaks. BF16 TranslateGemma-4B keeps 81.5% because it rewrites <b> tags and translates [PENDING] labels. This build loses another 1.5 points on top of that.

Which model to use

On a 12 GB card I'd pick Hy-MT2-7B-NVFP4. On the same tests it scored 84.66 COMET-22, ran 213 cells/s and kept 99.3% of protected tokens with its keep-structure prompt. This build makes sense on 8 GB cards, on GPUs without FP4 support, or next to other models on a shared GPU.

Usage with vLLM

Tested with vLLM 0.26.0, text only.

vllm serve kurcontko/translategemma-4b-it-FP8 --language-model-only --max-model-len 2048 \
  --gpu-memory-utilization 0.9
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="-")
text = "Warning: do not exceed 230 V.\nDisconnect power before cleaning."
r = client.chat.completions.create(model="kurcontko/translategemma-4b-it-FP8", temperature=0,
        messages=[{"role": "user", "content": f"<<<source>>>en<<<target>>>pl<<<text>>>{text}"}])
print(r.choices[0].message.content)

Language codes come from the base model's template, for example en, pl, de-DE, pt-PT or es-MX. With transformers, the structured message format from the base model card works unchanged.

If the engine fails at start-up while JIT-compiling FlashInfer's sampling kernels with CUDA compiler and CUDA toolkit headers are incompatible, set VLLM_USE_FLASHINFER_SAMPLER=0.

Limitations

  • Evaluated on the 9 directions above only, with COMET-22 and chrF. No human evaluation.
  • I have not tested image translation with this checkpoint. The vision tower is the unchanged BF16 one.

License

Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms. Use of this model must comply with the Gemma Prohibited Use Policy. This repository holds a modified, quantized version of google/translategemma-4b-it. The changes are listed above and in NOTICE.

Downloads last month
23
Safetensors
Model size
4B params
Tensor type
BF16
·
F8_E4M3
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for kurcontko/translategemma-4b-it-FP8

Quantized
(51)
this model