patrickbdevaney/MiMo-V2.6-Flash-REAP50-GGUF

MiMo-V2.6-Flash REAP-50 — GGUF (community)

Community GGUF quantisations of MiMo-V2.6-Flash-REAP50, a 50% routed-expert pruned checkpoint of XiaomiMiMo/MiMo-V2.6-Flash-RL. Experts were selected with the HOPE objective over REAP-style saliency. Not affiliated with or endorsed by Xiaomi.

Evaluation status. No end-to-end quality benchmark (perplexity, generative or task accuracy) has been run on these files or on the pruned checkpoint. The only published measure is saliency retention on the calibration corpus (worst domain 0.9935, mean 0.9964; see the base card), which is a proxy, not a quality result.

• Base HF Checkpoint: patrickbdevaney/MiMo-V2.6-Flash-REAP50
• Experts Retained: 128 of 256 routed experts per layer across 47 MoE layers (1 dense layer, 47 MoE layers).
• Base Architecture: Native packed MXFP4 (U8, block size 32) experts with unquantized pure BF16 attention and embeddings.
• Towers Included: Vision & Audio multimodal projectors (mmproj) and Multi-Token Prediction speculative draft heads (mtp).

---

Quantization Ladder

Filename	Quant Type	Size	Description	Recommended VRAM / RAM
MiMo-V2.6-Flash-REAP50-MXFP4_MOE.gguf	MXFP4_MOE	86.06 GiB	Native packed MXFP4 experts repacked without dequantising (expert weights not re-quantised) + BF16 attention/trunk. End-to-end equivalence to the safetensors checkpoint not separately tested.	96 GiB+ / 1x 128GB Thor or 2x 48GB
MiMo-V2.6-Flash-REAP50-Q2_K.gguf	Q2_K	61.64 GiB	Hybrid MoE: down-projections kept in native MXFP4, gate/up in Q2_K, trunk in Q8_0 (~3.36 BPW).	64 GiB+ / 3x 24GB GPUs (72GB) or Mac 64-96GB

Supporting Towers (Vision, Audio & MTP)

Filename	Size	Description
mmproj-MiMo-V2.6-Flash-REAP50-BF16.gguf	2.56 GiB	Multimodal projector (Vision + Audio) in BF16
mmproj-MiMo-V2.6-Flash-REAP50-Q8_0.gguf	1.46 GiB	Multimodal projector (Vision + Audio) quantized to Q8_0
mtp-MiMo-V2.6-Flash-REAP50-BF16.gguf	4.17 GiB	Multi-Token Prediction (MTP) draft head (3 next-n layers) in BF16
mtp-MiMo-V2.6-Flash-REAP50-Q8_0.gguf	2.22 GiB	Multi-Token Prediction (MTP) draft head (3 next-n layers) in Q8_0

---

Key Features

1. Native MXFP4 MoE Preservation: In the source checkpoint the routed experts ship as native packed mxfp4 (block size 32), 92.9% of the checkpoint by bytes. The GGUF converter repacks these blocks directly into GGMLQuantizationType.MXFP4 instead of dequantising and re-quantising them, so expert weights keep their original FP4 values.

2. Multimodal Projectors (mmproj): Xiaomi MiMo-V2.6-Flash incorporates both visual and audio processing towers:

   • Vision encoder (28-layer ViT, 560px patch representation)
   • Audio tokenizer / RVQ speech representations Both are packed into standard GGUF multimodal projectors (mmproj-*-BF16.gguf and mmproj-*-Q8_0.gguf) intended for llama.cpp's multimodal pipeline (--mmproj).

3. Multi-Token Prediction (mtp): MiMo-V2.6-Flash includes 3 trained MTP layers for speculative decoding. We ship standalone MTP draft models (mtp-*-BF16.gguf and mtp-*-Q8_0.gguf) that can be loaded alongside the trunk model with --draft-model. Note: these heads were trained against the unpruned expert set, and no speedup or acceptance rate has been measured on this pruned checkpoint.

---

Running with llama.cpp

1. Standard Text Inference (Optimal Hybrid Q2_K)

./llama-cli \
    -m MiMo-V2.6-Flash-REAP50-Q2_K.gguf \
    -p "You are MiMo, an AI assistant developed by Xiaomi. Explain how MoE expert pruning works:" \
    -n 512 --temp 0.6

Or run the native MXFP4 file:

./llama-cli \
    -m MiMo-V2.6-Flash-REAP50-MXFP4_MOE.gguf \
    -p "You are MiMo, an AI assistant developed by Xiaomi. Explain how MoE expert pruning works:" \
    -n 512 --temp 0.6

2. Speculative Decoding with MTP Draft Head

./llama-cli \
    -m MiMo-V2.6-Flash-REAP50-Q2_K.gguf \
    --draft-model mtp-MiMo-V2.6-Flash-REAP50-Q8_0.gguf \
    -p "Explain quantum teleportation in detail:" \
    -n 512

3. Multimodal Inference (Vision & Audio)

./llama-cli \
    -m MiMo-V2.6-Flash-REAP50-Q2_K.gguf \
    --mmproj mmproj-MiMo-V2.6-Flash-REAP50-Q8_0.gguf \
    --image input.jpg \
    -p "Describe the contents of this image in detail."

4. OpenAI-Compatible API Server

./llama-server \
    -m MiMo-V2.6-Flash-REAP50-Q2_K.gguf \
    --mmproj mmproj-MiMo-V2.6-Flash-REAP50-Q8_0.gguf \
    --port 8080 \
    -ngl 99

---

Background & Pruning Method

Experts were chosen by saliency accumulated over a calibration corpus (English/Chinese text, code, math, science, finance, agentic traces, and real image/audio/video captions), and the prune set was solved with HOPE (Higher-Order Pruning of Experts, arXiv 2609.18916). HOPE minimises pᵀFp, the output error a prune set causes, including the interaction terms between experts; REAP is the same objective with those off-diagonal terms set to zero. Selection was ranked by the worst calibration domain, with the top 8% of each domain protected, so audio and video, the domains most at risk, kept the highest-priority experts (worst-domain retention 0.9935, audio). Routers were refitted and then kept at the teacher weights because the refit did not beat the baseline.

What this does not show: that the pruned model performs as well as the original. That needs end-to-end benchmarks, which have not been run. Full settings and limitations are on the base card and in the pipeline repo.

---

Created by patrickbdevaney. MIT, inherited from the base model; attribution to Xiaomi MiMo.