| ---
|
| license: apache-2.0
|
| base_model: moonshotai/Kimi-K3.1-Pro
|
| tags:
|
| - kimi-k3.1
|
| - moe
|
| - quantization
|
| - mixed-bit
|
| - strata
|
| - dr-triton
|
| pipeline_tag: image-text-to-text
|
| ---
|
|
|
| # Kimi-K3.1-Pro-Coder (2.8T Mixed-Bit MoE + Extra Coding)
|
|
|
| 
|
|
|
| **`Grawnch/Kimi-K3.1-Pro-Coder`** is a full 96-shard (**`774.97 GB` / `721.75 GiB`**) **Strata-Coder 3-Tier Mixed-Bit MoE** quantization of **`moonshotai/Kimi-K3.1-Pro`** (`2.8T` parameters, 93 layers, 896 routed experts per MoE layer, `top_k=16` LatentMoE with `SiTU-GLU`, `KDA` linear attention + gated NoPE `MLA` full attention, and `AttnRes` cross-layer residual mixing).
|
|
|
| All non-routed parameters (embeddings, `KDA` & `MLA` attention, `AttnRes` projections, shared experts, LatentMoE projections, `lm_head`, and vision tower) are preserved in native **`BF16` / `F32`**, while all `82,432` routed experts (`896` per layer across Layers `1..92`) are compressed using **Strata-Coder 3-Tier Mixed-Bit MoE Compression** calibrated on coding and mathematical reasoning router activation frequencies:
|
|
|
| | Expert / Weight Tier | Routed Experts / Layer | Effective BPW | Total Size (`GB` / `GiB`) | Default Storage Tier (`Strata`) | Matrix Cosine vs Base MXFP4 |
|
| | :--- | :---: | :---: | :---: | :--- | :---: |
|
| | **Base Non-Routed (`BF16`/`F32`)** | Shared + `KDA`/`MLA` + `lm_head` | `16.0 bpw` | **`114.76 GB` (`106.88 GiB`)** | **GPU VRAM / Shared** | **`100.00%` (Exact Native)** |
|
| | **Tier 1 (Hot Coding/Math)** | `134 / 896` (`15.0%`) | **`2.81 bpw`** | **`160.58 GB` (`149.55 GiB`)** | **Tier 1: Hot GPU VRAM** | **`96.95%`** |
|
| | **Tier 2 (Warm General)** | `269 / 896` (`30.0%`) | **`2.56 bpw`** | **`254.53 GB` (`237.05 GiB`)** | **Tier 2: Warm Pinned System RAM** | **`94.53%`** |
|
| | **Tier 3 (Cold Tail)** | `493 / 896` (`55.0%`) | **`1.44 bpw`** | **`244.95 GB` (`228.13 GiB`)** | **Tier 3: Cold NVMe SSD (`mmap`)** | **`82.51%`** |
|
| | **Full Model Total** | **All 96 / 96 Shards** | **`~1.98 bpw` (MoE)** | **`774.97 GB` (`721.75 GiB`, `-56.2%`)** | **VRAM + RAM + NVMe** | **`93.09%` – `98.22%` MoE Output** |
|
|
|
| ---
|
|
|
| ## Full 93-Layer End-to-End Evaluation (`NVIDIA RTX PRO 6000 Blackwell`)
|
|
|
| Evaluated across **all 93 layers (`Layer 0` through `Layer 92` + `Shard 94` `lm_head`)** on an **`NVIDIA RTX PRO 6000 Blackwell Server Edition` (`102.6 GB` GDDR7 VRAM)** over two zero-leakage benchmarks (`22,307` total tokens across `70` sequences):
|
| 1. **`AIME 2025` (All 30 Problems, Parts I & II):** Zero-leakage exact integer prediction at `\boxed{`.
|
| 2. **Rolling-Window `LiveCodeBench` (`2024.10 – 2025.05`, 30 Problems: 10 Easy, 10 Medium, 10 Hard across LeetCode, AtCoder, and Codeforces):** Zero-leakage functional unit-test execution (`Pass@1`) + rolling-window code perplexity (`PPL`).
|
|
|
| | Benchmark / Metric | `moonshotai/Kimi-K3.1-Pro` (Base Ref) | `Grawnch/Kimi-K3.1-Pro-Coder` (`774.97 GB`) | Delta / Agreement |
|
| | :--- | :---: | :---: | :---: |
|
| | **`AIME 2025` (All 30 Problems, Pass@1)** | **`29 / 30` (`96.67%`)** | **`29 / 30` (`96.67%`)** | **`0.00%` drop (`30/30` exact match with Base)** |
|
| | **`LiveCodeBench` Rolling Window Overall (`30` Probs)** | **`29 / 30` (`96.67%`)** | **`29 / 30` (`96.67%`)** | **`0.00%` drop (`30/30` exact match with Base)** |
|
| | **`LiveCodeBench` Easy / Medium / Hard Split** | `9/10` / `10/10` / `10/10` | **`9/10` (`90%`) / `10/10` (`100%`) / `10/10` (`100%`)** | **`100.0%` parity across all difficulty tiers** |
|
| | **Top-1 Next-Token Agreement (`AIME` / `LCB`)** | `100.00%` | **`95.20%` / `95.06%`** | **`95.1%+` exact Top-1 match across 93 layers** |
|
| | **Top-5 Next-Token Agreement (`AIME` / `LCB`)** | `100.00%` | **`99.91%` / `99.92%`** | **`99.91%+` Top-5 coverage** |
|
| | **Mean KL Divergence (`KLD`, `AIME` / `LCB`)** | `0.00000` | **`0.01912` / `0.02248`** | **`~0.020` nats end-to-end across 93 layers** |
|
| | **`AIME 2025` Math Perplexity (`PPL`)** | `2.2135` | **`2.2358`** | **`+0.0223` (`+1.01%`)** |
|
| | **`LiveCodeBench` Hard Perplexity (`PPL`)** | `2.1136` | **`2.1562`** | **`+0.0426` (`+2.02%`)** |
|
| | **`LiveCodeBench` Overall Perplexity (`PPL`)** | `2.6911` | **`2.7629`** | **`+0.0718` (`+2.67%`)** |
|
| | **MoE Layer Throughput (`RTX PRO 6000`)** | — | **`3,382.7 layer-tok/s`** | **`36.77 tok/s` full 93-layer batch** |
|
|
|
| ---
|
|
|
| ## Runtime & 3-Tier Offloading with `Strata` and `dr-triton`
|
|
|
| - **Engine & 3-Tier Loader (`Hot VRAM / Warm RAM / Cold NVMe` + `RouterLookahead`):** [https://github.com/grawnchtoader/strata](https://github.com/grawnchtoader/strata)
|
| - **Fused Triton Dequantization Kernels:** [https://github.com/grawnchtoader/dr-triton](https://github.com/grawnchtoader/dr-triton)
|
| |
