| --- |
| license: mit |
| base_model: |
| - intfloat/multilingual-e5-large-instruct |
| tags: [llmcompressor, compressed-tensors, quantized, nvfp4, vllm] |
| pipeline_tag: feature-extraction |
| library_name: vllm |
| --- |
| |
| # multilingual-e5-large-instruct-nvfp4 |
|
|
| `NVFP4` quantization of [intfloat/multilingual-e5-large-instruct](https://huggingface.co/intfloat/multilingual-e5-large-instruct), produced with |
| [llm-compressor](https://github.com/vllm-project/llm-compressor) 0.13.0 and measured end to end on |
| NVIDIA H100 80 GB with vLLM 0.26.0. |
|
|
| Only the Linear layers of the encoder are quantized; the vocabulary, position and token-type |
| embeddings, the pooler and any classification head stay in bf16. |
|
|
| ## Measured |
|
|
| | variant | size | SciFact nDCG@10 | STS-B de ρ | STS17 en-de ρ | texts/s (bs=256) | |
| |---|---|---|---|---|---| |
| | bf16 source | 1068 MiB | 0.7204 | 0.8347 | 0.8525 | 1488 | |
| | FP8 | 802 MiB | 0.7164 | 0.8349 | 0.8526 | 1511 | |
| | NVFP4 | 675 MiB | 0.7036 | 0.8371 | 0.8440 | – | |
|
|
| Retrieval costs 1.7 pp nDCG@10 against bf16; STS is unchanged. Size is **−37%**. |
|
|
| SciFact: 300 queries over 5183 documents, nDCG@10. STS-B de: mteb/stsb_multi_mt (1379 pairs). |
| STS17 en-de: mteb/sts17-crosslingual-sts (250 pairs). Reranking: 1 positive + 19 sampled negatives |
| per query. Throughput over 4096 multilingual texts through the vLLM OpenAI endpoint. |
|
|
| ## A bug you should know about if you quantize BERT-family models yourself |
|
|
| The first NVFP4 build of this model scored **0.042** nDCG@10 in vLLM — random-level — while the |
| *same weights* scored 0.66 under plain transformers. The weights were fine; the checkpoint was not. |
|
|
| For NVFP4, every linear carries its own `weight_global_scale`. vLLM fuses query/key/value into one |
| `qkv_proj` and then collapses their scales with `weight_global_scale.max()` |
| (`compressed_tensors_w4a4_nvfp4.py:111`), silently rescaling two of the three projections. It warns |
| ("the weight global scale is different for parallel layers") but continues. |
|
|
| llm-compressor is supposed to prevent this: `fuse_weight_observers()` links observers across fused |
| groups so they share a global scale. But its `FUSED_LAYER_NAMES` |
| (`llmcompressor/observers/helpers.py:162`) only lists decoder naming — `q_proj/k_proj/v_proj`, |
| `gate_proj/up_proj`, `w1/w3`. BERT and RoBERTa call them `query`/`key`/`value`, so the group is |
| never matched and each projection keeps its own scale. |
|
|
| This checkpoint was built with `("query", "key", "value")` appended to that list. Layer 0 q/k/v |
| global scales went from `{query: 5344, key: 6048, value: 7904}` to a shared `5344`, vLLM's warning |
| disappeared, and nDCG@10 went from 0.042 to 0.704. |
|
|
| ## Serving |
|
|
| ```bash |
| vllm serve mbehr90/multilingual-e5-large-instruct-nvfp4 --runner pooling |
| ``` |
|
|
