Index-Translate-9B-NVFP4

NVFP4 W4A4 quantization of IndexTeam/Index-Translate-9B that fits a 12 GB GPU in vLLM.

The official Index-Translate-9B-FP4 checkpoint is 11.7 GB because its ignore list keeps every Gated DeltaNet projection, GDN for short, and lm_head in BF16. This build quantizes the GDN projections too. With text-only serving it loads in 7.6 GB, which leaves room for KV cache under a 12 GB budget.

  • COMET-22 on WMT24++ EU pairs is 84.59, against 84.68 for the official FP8. That's a drop of 0.09.
  • With its stock prompt it keeps 97.0% of placeholders, tags and line breaks in Excel-style cells, and 95.4% of cells come through fully intact. The structure test below has the details.
  • The base model card lists 150 languages. I evaluated only the 9 directions below.

What is quantized. Every language-model linear layer is NVFP4. That covers MLP, full attention, and the GDN in_proj_qkv, in_proj_z and out_proj. vLLM fuses in_proj_qkv and in_proj_z into one GEMM with a single global scale, so those two share one NVFP4 global scale here.

These stay in BF16:

  • lm_head and embeddings
  • the 32-wide GDN gate projections in_proj_a/b
  • the vision tower and the MTP head, both copied unchanged from the base model

I have not tested image input or MTP speculative decoding with this checkpoint.

Calibration. 576 OPUS-100 pairs, EN→CS/DE/ES/FR/IT/NL/PL/PT and PL→EN, each rendered with the Index-Translate prompt followed by the reference translation. The recipe and scripts are in quantization/ and target llm-compressor 0.15.

Quality

Test set. All non-canary segments of WMT24++, google/wmt24pp. That's 960 per direction and 8,640 in total, covering news, social, speech and literary text. EN→X uses the original English source. PL→EN reverses the EN→PL pair, so the Polish post-edited reference becomes the source and the original English becomes the reference.

Metrics. COMET-22 with Unbabel/wmt22-comet-da, scaled ×100, and corpus chrF from sacrebleu.

Setup. Greedy decoding on vLLM 0.26.0. One desktop RTX 5070 Ti 16 GB, capped at an 11.5 GB budget with gpu_memory_utilization ≈ 0.74 to mimic a 12 GB laptop GPU.

EN→CS EN→DE EN→ES EN→FR EN→IT EN→NL EN→PL EN→PT PL→EN All chrF
IndexTeam/Index-Translate-9B-FP8 on 16 GB 86.78 83.88 84.70 83.22 85.83 85.65 86.08 83.52 82.49 84.68 60.4
this model, NVFP4 at 11.5 GB 86.58 83.80 84.79 83.14 85.59 85.65 85.84 83.47 82.48 84.59 60.1

Structure test. 175 Excel-style cells in each of PL, DE, FR, ES and CS, with 270 tokens that must survive verbatim: {placeholders}, %s, %1$s, {{var}}, ${var}, HTML tags, URLs, e-mails, SKUs, inline code and line breaks. With the official prompt this model keeps 97.0% of protected tokens and leaves 95.4% of cells fully intact.

Throughput

Each Gated DeltaNet layer holds a fixed-size state for every running sequence. Under an 11.5 GB budget vLLM fits about 48 concurrent sequences, and that caps batched throughput. In a 4,000-cell test with --max-num-seqs 48 this model translated 39.3 short cells/s.

Hy-MT2-7B-NVFP4 did 213.4 cells/s on the same test, about 5× faster, so use it for high-volume batch work on 12 GB. Pick this model when you need a language Hy-MT2 doesn't cover, or when you can't change the prompt and still need placeholders and line breaks kept.

Usage with vLLM

Needs a Blackwell GPU for NVFP4. Tested with vLLM 0.26.0, text only.

vllm serve kurcontko/Index-Translate-9B-NVFP4 --language-model-only --max-model-len 4096 \
  --gpu-memory-utilization 0.9 --max-num-seqs 32
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="-")
prompt = "请将以下English文本翻译为Polish,直接输出翻译结果,不要进行任何解释。\n\n" + "Hello {name}, your order has shipped."
r = client.chat.completions.create(model="kurcontko/Index-Translate-9B-NVFP4", temperature=0,
        messages=[{"role": "user", "content": prompt}],
        extra_body={"chat_template_kwargs": {"enable_thinking": False}})
print(r.choices[0].message.content)

The prompt template comes from bilibili/Index-Translate. Always pass enable_thinking: false. If vLLM reports that max_num_seqs exceeds the available Mamba cache blocks, lower --max-num-seqs.

If the engine fails at start-up while JIT-compiling FlashInfer's sampling kernels with CUDA compiler and CUDA toolkit headers are incompatible, set VLLM_USE_FLASHINFER_SAMPLER=0.

Limitations

  • Evaluated on the 9 directions above only, with COMET-22 and chrF. No human evaluation.
  • Image translation and MTP speculative decoding are untested with this checkpoint.

License

Apache-2.0, same as IndexTeam/Index-Translate-9B. Calibration data is OPUS-100.

Downloads last month
130
Safetensors
Model size
10B params
Tensor type
BF16
·
U8
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for kurcontko/Index-Translate-9B-NVFP4

Quantized
(18)
this model