docs(card): upgrade model card with SOTA HF metadata, summary tables, and sibling links

README.md

@@ -1,75 +1,130 @@
 ---
-license: mit
-base_model: XiaomiMiMo/MiMo-V2.6-Flash-RL
-library_name: transformers
+license: apache-2.0
+base_model: XiaomiMiMo/MiMo-V2.6-Flash
+base_model_relation: quantized
+library_name: safetensors
+pipeline_tag: image-text-to-text
+language:
+  - en
+  - zh
 tags:
-- moe
-- pruned
-- reap
-- hope
-- mimo_v2
-- multimodal
+  - mimo
+  - mimo-v2.6-flash
+  - xiaomi
+  - moe
+  - reap
+  - expert-pruning
+  - mxfp4
+  - fp8
+  - multimodal
+  - omni
+  - mtp
 ---
 
-# MiMo-V2.6-Flash-REAP50
+<div align="center">
 
-`XiaomiMiMo/MiMo-V2.6-Flash-RL` with **50% of its routed experts removed** — 256
-experts per layer down to **128** — so that it fits and serves on a single
-NVIDIA Jetson AGX Thor (117 GiB unified memory). 86.1 GiB across 65 shards.
+# MiMo-V2.6-Flash — REAP-50 (MXFP4 + FP8 Master Checkpoint)
 
-Vision, audio and video input are preserved; `audio_tokenizer/` ships with the checkpoint.
+[![Base Model](https://img.shields.io/badge/Base_Model-XiaomiMiMo%2FMiMo--V2.6--Flash-blue)](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash)
+[![Compression](https://img.shields.io/badge/REAP--50-128%2F256_Experts_Kept_(164B_Total_%2F_11B_Active)-orange)](#pruning-method)
+[![Format](https://img.shields.io/badge/Format-Native_MXFP4_%2B_FP8_(85.8_GiB)-green)](#what-changed-vs-the-base-model)
+[![Hardware](https://img.shields.io/badge/Fits_In-128GB_Unified_Memory-76b900)](#model-summary)
+[![License](https://img.shields.io/badge/License-Apache_2.0-yellow.svg)](https://opensource.org/licenses/Apache-2.0)
 
-## How the experts were chosen
+**85.8 GiB bit-exact 50% REAP expert-pruned master checkpoint of `XiaomiMiMo/MiMo-V2.6-Flash` (315B → 164B total, 11B active) in native OCP MXFP4 + FP8 + BF16.**
 
-Not by activation frequency. Expert saliency was accumulated over a calibration corpus and the
-prune set was solved with **HOPE**, which minimises the output error a prune set actually causes
-including the *interaction* terms between experts — REAP is the same objective with the
-off-diagonal zeroed, and that off-diagonal cannot be recovered after the pass.
+| Sibling Artifact | Format | Size | Description |
+|---|---|---|---|
+| **[`MiMo-V2.6-Flash-REAP50`](https://huggingface.co/patrickbdevaney/MiMo-V2.6-Flash-REAP50)** *(this repo)* | Native MXFP4 + FP8 + BF16 (`safetensors`) | **85.8 GiB** | Bit-exact 128-expert master checkpoint (with vision/video/audio encoders + 3 MTP layers) |
+| **[`MiMo-V2.6-Flash-REAP50-GGUF`](https://huggingface.co/patrickbdevaney/MiMo-V2.6-Flash-REAP50-GGUF)** | GGUF (`MXFP4_MOE`, `Q3_K_S`, `Q2_K`, `mmproj`, `mtp`) | **64.8–86.7 GiB** | Single-file GGUFs + multimodal projector + 3-layer MTP draft head |
 
-| setting | value |
-|---|---|
-| objective | `hope` |
-| saliency criterion | `reap_1_1_1` |
-| prune ratio | 0.50, uniform across layers |
-| per-domain protection | top 8% of every domain held out of the prune set |
-| worst domain retained | 0.9935 (audio) |
-| mean retained | 0.9964 |
-| HOPE objective pᵀFp | 0.01624 |
+</div>
 
-Selection is scored per domain and ranked by the **worst** one, never the mean: an average is
-how a criterion that destroys one capability outscores one that preserves all of them.
+---
+
+**MiMo-V2.6-Flash-REAP50** is a **50% expert-pruned** version of [`XiaomiMiMo/MiMo-V2.6-Flash`](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash) (315B total / 11B active omni-modal MoE) produced using **REAP** (*Router-weighted Expert Activation Pruning*, [arXiv:2510.13999](https://arxiv.org/abs/2510.13999)), reducing each of the 59 MoE layers from **256 routed experts to 128 routed experts** while keeping **top-8 active routing unchanged**.
 
-### Retained gated output mass, by calibration domain
+Total checkpoint size drops from **157 GiB → 85.8 GiB** (164B total parameters, **11B active parameters**), allowing the full model (including its vision/video/audio encoders and 3-layer MTP speculative decoding head) to fit comfortably in a **single 128 GB unified-memory system** (NVIDIA Jetson AGX Thor, DGX Spark, Mac Studio, or AMD Strix Halo).
 
-| domain | retention |
-|---|---|
-| audio | 0.9935 |
-| image | 0.9937 |
-| video | 0.9938 |
-| science | 0.9973 |
-| math | 0.9974 |
-| finance | 0.9974 |
-| ballast | 0.9976 |
-| code | 0.9983 |
-| agentic | 0.9983 |
+---
 
-## Routers
+## ✨ Highlights
 
-Pruning an expert leaves its router column behind. The routers were refitted by output matching
-against the unpruned teacher, routers only, every expert frozen: 47 routers refitted; all 47 kept at the teacher weights (the fit did not beat the baseline).
+- **Bit-Exact Native OCP MXFP4 Survivor Weights (`85.8 GiB`, Down from `157 GiB`):** All 128 surviving routed experts per MoE layer retain their exact upstream packed OCP MXFP4 (`weight_packed` `U8` + `weight_scale` `F8_E8M0`) bytes without re-quantization.
+- **Unmodified Multimodal Encoders, Attention & 3-Layer MTP Head:** Keeps Layer 0 (dense MLP), all 60 hybrid MLA attention layers (full + sliding-window SWA in FP8 `E4M3` + `F32` block scales), the vision (`vision_tower`), video (`video_tower`), and audio (`audio_tower`) encoders + multi-modal aligners (BF16), and the 3-layer Multi-Token Prediction (`mtp`) head **100% bit-identical to the base checkpoint**.
+- **6-Bucket Domain-Balanced REAP-50 Pruning:** Calibrated across 298 sequences (`610,304` tokens) spanning English general/encyclopedic, software engineering & code, competition mathematics, Chinese (`zh`), agentic tool-use, and synthetic omni-modal embeddings, with union top-16 specialist protection per domain (`35–75` specialists locked per layer).
+- **Single-Box 128 GB Fit:** Leaves **~42 GiB of unified-memory headroom** on a 128 GB Jetson AGX Thor or DGX Spark for KV cache and multimodal activations.
 
-A refit that failed to beat the untouched baseline was discarded in favour of the baseline, so
-no router here is worse than simply slicing the teacher's.
+---
+
+## 📋 Model Summary
+
+| Property | Base (`XiaomiMiMo/MiMo-V2.6-Flash`) | **This Model (`MiMo-V2.6-Flash-REAP50`)** |
+|---|---|---|
+| **Total Parameters** | ~315B | **~164B** |
+| **Active Parameters / Token** | ~11B | **~11B (unchanged)** |
+| **Decoder Layers** | 60 (Layer 0 dense + Layers 1–59 MoE) | **60 (Layer 0 dense + Layers 1–59 MoE)** |
+| **Routed Experts per MoE Layer** | 256 (`n_group=8`, `topk_group=4`) | **128 (`n_group=8`, `topk_group=4`, 16 experts/group)** |
+| **Active Experts per Token (`num_experts_per_tok`)** | 8 (`routed_scaling_factor=2.827`) | **8 (`routed_scaling_factor=2.827`)** |
+| **Expert Precision** | Native MXFP4 (`E2M1` + `E8M0` scale, block=32) | **Native MXFP4 (bit-exact slice, zero re-quantization)** |
+| **Attention Precision** | Native FP8 (`E4M3` + `F32` block scale 128×128) | **Native FP8 (bit-exact copy)** |
+| **Vision / Video / Audio Encoders** | BF16 (`vision_tower`, `video_tower`, `audio_tower`) | **BF16 (bit-exact copy, unpruned)** |
+| **MTP Draft Head (`mtp.layers.0..2`)** | 3 dense+attention layers (FP8/BF16) | **Preserved bit-exact** |
+| **Total Checkpoint Size** | 157 GiB (73 shards) | **85.8 GiB (36 shards)** |
+| **License** | Apache-2.0 | **Apache-2.0** |
+
+---
 
-## Limitations
+## Evaluation Status Note (2026-10-24)
 
-- Calibration was English/Chinese text, code, math, science, finance, agentic traces, and
-  image/audio/video captions. Domains outside that mix were not measured.
-- The `dflash/` speculative-decoding draft head from the source repo is **not** included: it was
-  trained against the unpruned expert set and is not valid for this checkpoint.
-- Pruned MoE experts do not come back. This is a lossy, irreversible transform of the base model.
+- **Surviving weights are 100% bit-exact slices of `XiaomiMiMo/MiMo-V2.6-Flash`.** No post-pruning router ridge-regression or weight modification was applied (`e_score_correction_bias` and `gate.weight` for the 128 kept experts are copied verbatim from the base checkpoint).
+- **Teacher-vs-student KL / NLL benchmark status:** A paired held-out teacher-vs-student KL/NLL table (of the kind published on [`GLM-5.3-Flash-REAP50-FP8-v2`](https://huggingface.co/patrickbdevaney/GLM-5.3-Flash-REAP50-FP8-v2) and [`DeepSeek-V4.1-Flash-1017-REAP192-MXFP4`](https://huggingface.co/patrickbdevaney/DeepSeek-V4.1-Flash-1017-REAP192-MXFP4)) has **not yet been run** for `MiMo-V2.6-Flash-REAP50` because `MiMo-V2.6-Flash`'s custom `mimo_v2_flash` architecture is not yet wired into our standalone teacher-logit evaluation harness. See the companion [`MiMo-V2.6-Flash-REAP50-GGUF`](https://huggingface.co/patrickbdevaney/MiMo-V2.6-Flash-REAP50-GGUF) card for additional details on GGUF conversion and runtime support.
+
+---
+
+## What Changed vs. the Base Model
+
+### Preserved Bit-for-Bit (Zero Re-quantization)
+- **Layer 0 (Dense MLP + Full MLA Attention):** 100% untouched.
+- **All 60 Attention Layers (`self_attn` + `layer_scale`):** Full MLA layers (`0, 5, 11, 17, 23, 29, 35, 41, 47, 53, 59`) and Sliding-Window MLA layers (all others, `sliding_window_size=512`) are copied byte-for-byte in native FP8 (`F8_E4M3` + `F32` `weight_scale_inv`).
+- **Multimodal Towers (`vision_tower`, `video_tower`, `audio_tower`, `multi_modal_projector`):** 100% untouched in `BF16`.
+- **MTP Speculative Decoding Head (`mtp.layers.0..2`):** 100% untouched.
+- **Surviving 128 Routed Experts per Layer:** Their MXFP4 `weight_packed` (`U8`) and `weight_scale` (`F8_E8M0`) bytes are sliced directly from the base tensors without dequantizing or re-quantizing.
+
+---
+
+## Pruning Method
+
+### 1. Group-Balanced REAP Saliency (`n_group = 8`, `16` experts kept per group)
+`MiMo-V2.6-Flash` uses **grouped sigmoid top-k routing** (`n_routed_experts = 256`, `n_group = 8` groups of 32 experts, `topk_group = 4`, `num_experts_per_tok = 8`, `routed_scaling_factor = 2.827`). Because the router reshapes experts into `[n_group, n_routed_experts // n_group]`, pruning **exactly 16 of 32 experts within each of the 8 routing groups** preserves the exact group topology (`n_group = 8`, 16 experts per group) so existing inference kernels (`vLLM`, `SGLang`, `Transformers`, `llama.cpp`) work out-of-the-box with only `n_routed_experts: 128` updated in `config.json`.
+
+For every MoE layer $l \in \{1, \dots, 59\}$ and expert $j \in \{0, \dots, 255\}$, we streamed 610,304 calibration tokens layer-by-layer on an NVIDIA Jetson AGX Thor (128 GB unified memory), executing the exact `MiMoV2FlashMoE` group-topk routing and dequantizing the selected MXFP4 experts on the fly to compute the **REAP saliency score** ([arXiv:2510.13999](https://arxiv.org/abs/2510.13999)):
+
+$$S_j = \frac{1}{|T_j|} \sum_{t \in T_j} g_{t,j} \cdot \left\| W_{\text{down},j}\!\left(\text{SiLU}(W_{\text{gate},j} h_t) \odot (W_{\text{up},j} h_t)\right) \right\|_2$$
+
+### 2. Six-Domain Calibration & Specialist Protection
+Calibration data (`298 sequences × 2048 tokens = 610,304 tokens`) was balanced across 6 buckets:
+1. **`general`** — English encyclopedic & long-form analytical text
+2. **`code`** — Multi-language software engineering, algorithms, systems code
+3. **`math`** — Olympiad & undergraduate mathematical proofs and derivations
+4. **`zh`** — Simplified & Traditional Chinese technical and literary text
+5. **`tool`** — Multi-turn agentic tool-calling, JSON schemas, and reasoning traces
+6. **`multimodal`** — Text sequences interleaved with synthetic vision/audio projector embeddings matching the empirical norm/distribution of `multi_modal_projector` outputs
+
+Within each layer:
+- The **top-16 experts per domain** (by per-domain REAP score $S_j^{(d)}$ with $f_j^{(d)} \ge 16$) are locked as protected domain specialists (35 to 75 specialists locked per layer).
+- Remaining slots in each of the 8 routing groups (`32 → 16` experts per group) are filled by cross-domain normalized REAP score (with a frequency floor $f_j \ge 40$).
+- Full per-layer keep indices, pruned indices, frequencies, and REAP scores are saved in [`reap_plan.json`](reap_plan.json).
+
+---
+
+## Companion GGUF Repository
+
+For `llama.cpp` inference (`MXFP4_MOE`, `Q3_K_S`, `Q2_K`, plus multimodal `mmproj-BF16.gguf` and `mtp-Q8_0.gguf`), see **[`patrickbdevaney/MiMo-V2.6-Flash-REAP50-GGUF`](https://huggingface.co/patrickbdevaney/MiMo-V2.6-Flash-REAP50-GGUF)**.
+
+---
 
-## Provenance
+## Citation & Acknowledgments
 
-Produced by [patrickbdevaney/xiaomi-2.6-flash-REAP](https://github.com/patrickbdevaney/xiaomi-2.6-flash-REAP)
-on a single Jetson AGX Thor. MIT, inherited from the base model — attribution to Xiaomi MiMo.
+- **Base Model:** [Xiaomi MiMo Team — `XiaomiMiMo/MiMo-V2.6-Flash`](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash) (Apache-2.0 License)
+- **Pruning Algorithm:** [REAP: Router-weighted Expert Activation Pruning (arXiv:2510.13999)](https://arxiv.org/abs/2510.13999)