Instructions to use AxionML/MiMo-V2.6-Flash-RL-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AxionML/MiMo-V2.6-Flash-RL-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="AxionML/MiMo-V2.6-Flash-RL-NVFP4", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("AxionML/MiMo-V2.6-Flash-RL-NVFP4", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AxionML/MiMo-V2.6-Flash-RL-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AxionML/MiMo-V2.6-Flash-RL-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AxionML/MiMo-V2.6-Flash-RL-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/AxionML/MiMo-V2.6-Flash-RL-NVFP4

SGLang

How to use AxionML/MiMo-V2.6-Flash-RL-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AxionML/MiMo-V2.6-Flash-RL-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AxionML/MiMo-V2.6-Flash-RL-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AxionML/MiMo-V2.6-Flash-RL-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AxionML/MiMo-V2.6-Flash-RL-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use AxionML/MiMo-V2.6-Flash-RL-NVFP4 with Docker Model Runner:
```
docker model run hf.co/AxionML/MiMo-V2.6-Flash-RL-NVFP4
```

vincentzed-hf commited on Sep 30

Commit

d09782c

verified ·

1 Parent(s): 7d0d7b6

Nightly results: original MXFP4 serves and is faster there; backend warnings

Browse files

Files changed (1) hide show

README.md +14 -3

README.md CHANGED Viewed

@@ -21,7 +21,18 @@ tags:
 NVFP4 version of [XiaomiMiMo/MiMo-V2.6-Flash-RL](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL) for Blackwell. The routed experts run as **W4A4 NVFP4** on native FP4 Tensor Cores, and their weights are a **bit-exact transcode** of Xiaomi's released MXFP4 experts: every E2M1 code is kept byte-for-byte and each E8M0 block scale is re-expressed exactly as two E4M3 NVFP4 block scales. All 9,462,349,824 expert scale blocks convert exactly, and none are re-rounded. Every other tensor is copied byte-for-byte from the source checkpoint. The checkpoint is 175 GB (source: 178 GB).
-It serves on the stock `lmsysorg/sglang:v0.5.20-cu130` image on B300. On that image we could not serve the released MXFP4 checkpoint on B300: the default MoE backend routes MXFP4 experts into an FP8-only kernel, the `flashinfer_mxfp4` backend rejects MiMo's router output, and the Marlin MXFP4 path is SM90/SM120-only, and the Triton path fails a hidden-size check.
 **About NVFP4 quantization:** NVFP4 on Blackwell couples a compact E2M1 FP4 codebook with blockwise FP8 (E4M3) scaling over 16-element micro-blocks, so that 4-bit stored values remain numerically useful for neural-network computation. The E2M1 codebook provides a small, nonuniform set of representable magnitudes up to ±6 and relies on saturating behavior rather than IEEE NaN/Inf encodings to maximize usable range per bit. Using an FP8 block scale (rather than power-of-two-only E8M0) enables fractional scales and error-minimizing scale selection. On Blackwell Tensor Cores, native FP4 multipliers exploit E2M1 simplicity to reduce multiplier area while higher-precision FP32 accumulation protects dot-product accuracy.
@@ -45,7 +56,7 @@ It serves on the stock `lmsysorg/sglang:v0.5.20-cu130` image on B300. On that im
 ### Deploy with SGLang
-Verified on `lmsysorg/sglang:v0.5.20-cu130`, 4× B300:
 ```bash
 docker run --gpus all --shm-size=64g --network=host --ipc=host \
@@ -66,7 +77,7 @@ docker run --gpus all --shm-size=64g --network=host --ipc=host \
 - **Attention TP must be 4.** The fused `qkv_proj` is TP=4-interleaved, as in the source checkpoint. For 8 GPUs the [SGLang MiMo cookbook](https://docs.sglang.io/cookbook/autoregressive/Xiaomi/MiMo-V2.5) uses `--tp 8 --dp 2 --enable-dp-attention`; we have only verified TP4 with this checkpoint.
 - `--attention-backend fa4` is required on Blackwell (asymmetric 192/128 K/V head dims); `--mm-attention-backend fa4` for the vision encoder.
-- `--moe-runner-backend flashinfer_cutlass` is required: on v0.5.20 the `flashinfer_trtllm` NVFP4 MoE path rejects this model.
 - Sampling: Xiaomi recommends `temperature=1.0, top_p=0.95`. Thinking is on by default; pass `chat_template_kwargs={"enable_thinking": false}` to turn it off.
 - **BF16-activation mode.** The same checkpoint also runs with BF16 activations (weight-only FP4, the reference column below): add `-e SGLANG_FLASHINFER_CUTEDSL_NVFP4_W4A16=1` to `docker run` and replace the MoE flags with `--quantization modelopt_mixed --moe-runner-backend flashinfer_cutedsl`.

 NVFP4 version of [XiaomiMiMo/MiMo-V2.6-Flash-RL](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL) for Blackwell. The routed experts run as **W4A4 NVFP4** on native FP4 Tensor Cores, and their weights are a **bit-exact transcode** of Xiaomi's released MXFP4 experts: every E2M1 code is kept byte-for-byte and each E8M0 block scale is re-expressed exactly as two E4M3 NVFP4 block scales. All 9,462,349,824 expert scale blocks convert exactly, and none are re-rounded. Every other tensor is copied byte-for-byte from the source checkpoint. The checkpoint is 175 GB (source: 178 GB).
+## Which checkpoint should I use?
+Measured on 4× B300, TP4, SGLang random 1024-in / 1024-out (2026-09-30):
+| SGLang image | Xiaomi's original MXFP4 (`XiaomiMiMo/MiMo-V2.6-Flash-RL`) | This NVFP4 checkpoint |
+|---|---|---|
+| `lmsysorg/sglang:v0.5.20-cu130` (stable) | does not serve on B300: `flashinfer_trtllm`/default hit an FP8-only kernel with MXFP4 weights, `flashinfer_mxfp4` rejects MiMo's router output, Marlin MXFP4 is SM90/SM120-only, Triton fails a hidden-size check | **serves** with `--moe-runner-backend flashinfer_cutlass` |
+| `lmsysorg/sglang:nightly-dev-cu13-20260929-79cafec0` | **serves** with `--moe-runner-backend flashinfer_mxfp4`, and is faster | serves with `--moe-runner-backend flashinfer_cutlass` |
+Output throughput on that nightly, original MXFP4 (`flashinfer_mxfp4`) vs this checkpoint (`flashinfer_cutlass`): 302 vs 201 tok/s at concurrency 1, 2,223 vs 1,668 at 16, 7,079 vs 4,934 at 64 (median TPOT 3.3 / 6.0 / 8.6 ms vs 4.9 / 8.4 / 12.5 ms).
+**On a recent SGLang nightly, use Xiaomi's original checkpoint.** This checkpoint is for stable `v0.5.20` on Blackwell, or for stacks that need a standard ModelOpt NVFP4 export. SGLang's faster `flashinfer_trtllm` NVFP4 MoE path does not work for this model yet (see the warnings under Usage).
 **About NVFP4 quantization:** NVFP4 on Blackwell couples a compact E2M1 FP4 codebook with blockwise FP8 (E4M3) scaling over 16-element micro-blocks, so that 4-bit stored values remain numerically useful for neural-network computation. The E2M1 codebook provides a small, nonuniform set of representable magnitudes up to ±6 and relies on saturating behavior rather than IEEE NaN/Inf encodings to maximize usable range per bit. Using an FP8 block scale (rather than power-of-two-only E8M0) enables fractional scales and error-minimizing scale selection. On Blackwell Tensor Cores, native FP4 multipliers exploit E2M1 simplicity to reduce multiplier area while higher-precision FP32 accumulation protects dot-product accuracy.
 ### Deploy with SGLang
+Verified on `lmsysorg/sglang:v0.5.20-cu130` and `lmsysorg/sglang:nightly-dev-cu13-20260929-79cafec0`, 4× B300:
 ```bash
 docker run --gpus all --shm-size=64g --network=host --ipc=host \
 - **Attention TP must be 4.** The fused `qkv_proj` is TP=4-interleaved, as in the source checkpoint. For 8 GPUs the [SGLang MiMo cookbook](https://docs.sglang.io/cookbook/autoregressive/Xiaomi/MiMo-V2.5) uses `--tp 8 --dp 2 --enable-dp-attention`; we have only verified TP4 with this checkpoint.
 - `--attention-backend fa4` is required on Blackwell (asymmetric 192/128 K/V head dims); `--mm-attention-backend fa4` for the vision encoder.
+- `--moe-runner-backend flashinfer_cutlass` is required. Other MoE backends fail with this checkpoint: on `v0.5.20`, `flashinfer_trtllm` is rejected at load time; on `nightly-dev-cu13-20260929-79cafec0`, the default backend crashes (`'FusedMoE' object has no attribute 'g1_scale_c'`) and **`flashinfer_trtllm` loads but generates garbage**. The same weights are correct on `flashinfer_cutlass` on both images.
 - Sampling: Xiaomi recommends `temperature=1.0, top_p=0.95`. Thinking is on by default; pass `chat_template_kwargs={"enable_thinking": false}` to turn it off.
 - **BF16-activation mode.** The same checkpoint also runs with BF16 activations (weight-only FP4, the reference column below): add `-e SGLANG_FLASHINFER_CUTEDSL_NVFP4_W4A16=1` to `docker run` and replace the MoE flags with `--quantization modelopt_mixed --moe-runner-backend flashinfer_cutedsl`.