Index-Translate-9B-NVFP4
NVFP4 W4A4 quantization of IndexTeam/Index-Translate-9B that fits a 12 GB GPU in vLLM.
The official Index-Translate-9B-FP4 checkpoint is 11.7 GB
because its ignore list keeps every Gated DeltaNet projection, GDN for short, and lm_head in BF16. This build quantizes the
GDN projections too. With text-only serving it loads in 7.6 GB, which leaves room for KV cache under a 12 GB budget.
- COMET-22 on WMT24++ EU pairs is 84.59, against 84.68 for the official FP8. That's a drop of 0.09.
- With its stock prompt it keeps 97.0% of placeholders, tags and line breaks in Excel-style cells, and 95.4% of cells come through fully intact. The structure test below has the details.
- The base model card lists 150 languages. I evaluated only the 9 directions below.
What is quantized. Every language-model linear layer is NVFP4. That covers MLP, full attention, and the GDN
in_proj_qkv, in_proj_z and out_proj. vLLM fuses in_proj_qkv and in_proj_z into one GEMM with a single
global scale, so those two share one NVFP4 global scale here.
These stay in BF16:
lm_headand embeddings- the 32-wide GDN gate projections
in_proj_a/b - the vision tower and the MTP head, both copied unchanged from the base model
I have not tested image input or MTP speculative decoding with this checkpoint.
Calibration. 576 OPUS-100 pairs, EN→CS/DE/ES/FR/IT/NL/PL/PT and PL→EN, each rendered with the Index-Translate
prompt followed by the reference translation. The recipe and scripts are in quantization/ and target
llm-compressor 0.15.
Quality
Test set. All non-canary segments of WMT24++, google/wmt24pp. That's 960 per direction and 8,640 in total,
covering news, social, speech and literary text. EN→X uses the original English source. PL→EN reverses the EN→PL
pair, so the Polish post-edited reference becomes the source and the original English becomes the reference.
Metrics. COMET-22 with Unbabel/wmt22-comet-da, scaled ×100, and corpus chrF from sacrebleu.
Setup. Greedy decoding on vLLM 0.26.0. One desktop RTX 5070 Ti 16 GB, capped at an 11.5 GB budget with
gpu_memory_utilization ≈ 0.74 to mimic a 12 GB laptop GPU.
| EN→CS | EN→DE | EN→ES | EN→FR | EN→IT | EN→NL | EN→PL | EN→PT | PL→EN | All | chrF | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| IndexTeam/Index-Translate-9B-FP8 on 16 GB | 86.78 | 83.88 | 84.70 | 83.22 | 85.83 | 85.65 | 86.08 | 83.52 | 82.49 | 84.68 | 60.4 |
| this model, NVFP4 at 11.5 GB | 86.58 | 83.80 | 84.79 | 83.14 | 85.59 | 85.65 | 85.84 | 83.47 | 82.48 | 84.59 | 60.1 |
Structure test. 175 Excel-style cells in each of PL, DE, FR, ES and CS, with 270 tokens that must survive
verbatim: {placeholders}, %s, %1$s, {{var}}, ${var}, HTML tags, URLs, e-mails, SKUs, inline code and line
breaks. With the official prompt this model keeps 97.0% of protected tokens and leaves 95.4% of cells fully intact.
Throughput
Each Gated DeltaNet layer holds a fixed-size state for every running sequence. Under an 11.5 GB budget vLLM fits
about 48 concurrent sequences, and that caps batched throughput. In a 4,000-cell test with --max-num-seqs 48 this
model translated 39.3 short cells/s.
Hy-MT2-7B-NVFP4 did 213.4 cells/s on the same test, about 5× faster, so use it for high-volume batch work on 12 GB. Pick this model when you need a language Hy-MT2 doesn't cover, or when you can't change the prompt and still need placeholders and line breaks kept.
Usage with vLLM
Needs a Blackwell GPU for NVFP4. Tested with vLLM 0.26.0, text only.
vllm serve kurcontko/Index-Translate-9B-NVFP4 --language-model-only --max-model-len 4096 \
--gpu-memory-utilization 0.9 --max-num-seqs 32
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="-")
prompt = "请将以下English文本翻译为Polish,直接输出翻译结果,不要进行任何解释。\n\n" + "Hello {name}, your order has shipped."
r = client.chat.completions.create(model="kurcontko/Index-Translate-9B-NVFP4", temperature=0,
messages=[{"role": "user", "content": prompt}],
extra_body={"chat_template_kwargs": {"enable_thinking": False}})
print(r.choices[0].message.content)
The prompt template comes from
bilibili/Index-Translate.
Always pass enable_thinking: false. If vLLM reports that max_num_seqs exceeds the available Mamba cache blocks,
lower --max-num-seqs.
If the engine fails at start-up while JIT-compiling FlashInfer's sampling kernels with CUDA compiler and CUDA toolkit headers are incompatible, set VLLM_USE_FLASHINFER_SAMPLER=0.
Limitations
- Evaluated on the 9 directions above only, with COMET-22 and chrF. No human evaluation.
- Image translation and MTP speculative decoding are untested with this checkpoint.
License
Apache-2.0, same as IndexTeam/Index-Translate-9B. Calibration data is OPUS-100.
- Downloads last month
- 130
Model tree for kurcontko/Index-Translate-9B-NVFP4
Base model
IndexTeam/Index-Translate-9B