File size: 3,113 Bytes
705350b 1075a4d 705350b 1075a4d 705350b 1075a4d 705350b 1075a4d 705350b 1075a4d 705350b 1075a4d 705350b 1075a4d 705350b 1075a4d 705350b 1075a4d 705350b 1075a4d 705350b 1075a4d 705350b 1075a4d 705350b 1075a4d 705350b 1075a4d 705350b 1075a4d 705350b 1075a4d 705350b 1075a4d 705350b 1075a4d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 | ---
base_model:
- MiniMaxAI/MiniMax-M2.7
license: mit
---
## Model Description
**MiniMax-M2.7-NVFP4** is an NVFP4-quantized version of [MiniMaxAI/MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7), a 230B-parameter Mixture-of-Experts language model with 10B active parameters.
Produced with [NVIDIA TensorRT Model Optimizer](https://github.com/NVIDIA/TensorRT-Model-Optimizer) (`modelopt` 0.43.0rc3) using the `nvfp4_experts_only-fp8_kv` recipe — the same recipe structure as [`nvidia/MiniMax-M2.5-NVFP4`](https://huggingface.co/nvidia/MiniMax-M2.5-NVFP4). The base model is quantized directly from its official FP8 checkpoint via modelopt's on-the-fly FP8→BF16 dequant path (no separate BF16 master), then requantized to NVFP4.
### What's quantized
Only the MoE expert MLP layers (`w1`, `w2`, `w3` — gate, up, down projections) are quantized to NVFP4 (W4A4, group size 16). Self-attention projections, the MoE router, embeddings, layernorms, and `lm_head` are left in BF16. The `exclude_modules` list in `hf_quant_config.json` matches `nvidia/MiniMax-M2.5-NVFP4` exactly.
The KV cache is quantized to FP8 E4M3 with **statically calibrated scales** baked into the safetensors as per-layer `k_proj.k_scale` / `v_proj.v_scale` tensors (62 of each). This avoids vLLM's silent `scale=1.0` fallback when KV scales are missing from the checkpoint.
### Calibration
- **Recipe:** `nvfp4_experts_only-fp8_kv`
- **Dataset:** `cnn_dailymail` + `nvidia/Nemotron-Post-Training-Dataset-v2` (same as nvidia's M2.5 quant)
- **Samples:** 1024
- **Full expert coverage:** calibration is run with `--moe_calib_experts_ratio 1.0`, which sets the MoE top-k to 256 during calibration so every token activates every expert in every layer. This guarantees all 256 × 62 = 15,872 experts receive a non-zero `amax` regardless of dataset routing, and eliminates the cold-expert failure mode (spurious spaces at subword boundaries, e.g. `"mpg_ high"`, `"SPEC. md"`) seen when experts are calibrated via natural top-k routing alone.
### Hardware
NVFP4 requires native FP4 tensor cores — NVIDIA Blackwell (B100 / B200, RTX Pro 6000, DGX Spark GB10). Not supported on H100/H200 or older.
### Quality
Smoke-tested with vLLM v0.19.0 on 2x RTX Pro 6000 (SM120): prose generation and single-turn tool calling both pass, no spurious-space tokens observed. You should always evaluate against your specific use case.
## Usage
### vLLM
Tested on `vllm/vllm-openai:v0.19.0-cu130` with 2x RTX Pro 6000 Blackwell:
```sh
vllm serve demon-zombie/MiniMax-M2.7-NVFP4 \
--trust-remote-code \
--served-model-name MiniMax-M2.7 \
--tensor-parallel-size 2 \
--max-model-len 196608 \
--tool-call-parser minimax_m2 \
--reasoning-parser minimax_m2 \
--enable-auto-tool-choice \
--disable-custom-all-reduce
```
`--disable-custom-all-reduce` is **required** on RTX Pro 6000 (SM120) with TP > 1 — vLLM's custom CUDA all-reduce hangs post-weight-load (workers spin at 100% SM with 0 memory traffic). Not needed on B200. Do **not** pass `--enforce-eager`; CUDA graphs capture cleanly on v0.19.0.
### SGLang
(pending)
|