| --- |
| license: mit |
| base_model: deepseek-ai/DeepSeek-V4-Pro |
| tags: [gguf, mixture-of-experts, deepseek, expert-pruning, llama.cpp] |
| --- |
| # UltraTensor models: expert splices of DeepSeek-V4-Pro (corrected release, 2026-10-09) |
|
|
| Keep-N expert splices of the 1.6T-parameter DeepSeek-V4-Pro: every MoE layer keeps N of its 384 routed experts |
| (plus the shared expert, attention and embeddings), cut from the public BatiAI Q3_K_M GGUF. **Every tier in the |
| table below loads in stock llama.cpp** (DeepSeek-V4 support; verified with upstream llama.cpp and with the build |
| bundled with Ollama), with no fork. The older `keep64-iq2xs` file is unchanged. It keeps 64 experts behind a |
| full-width router, which only UltraTensor's llama.cpp fork can load. |
|
|
| ## What changed in this release, and why |
|
|
| An audit that rebuilt the previous files from their recipe and measured them against the full model found four |
| defects. The previous files remain available in this repository's history. |
|
|
| 1. **Routers sliced along the wrong axis** in keep8u/12u/16u. llama.cpp stores a router as one row of |
| 7168 floats per expert. The splicer kept columns, so no router row belonged to any kept expert. The new |
| files slice rows. |
| 2. **Experts chosen from one layer.** Every dense layer kept the experts that layer 3's code census ranked highest. |
| On a full-model importance matrix those experts receive no more routing than a random choice. The new |
| files keep, in each layer, the experts that layer actually uses most (full-model `llama-imatrix`, 32 × 512 |
| tokens of mixed calibration text). |
| 3. **keep12u had zeroed layers.** The last 6.5 GiB of the previous file (layers 48–60) was all zeros. |
| 4. **The IQ2_XS files emitted garbage** under upstream llama.cpp: repeated tokens, top-1 agreement 0 %. The cause |
| is llama.cpp's IQ2_XS preset, which puts every tensor at 2 bits. Swapping one tensor class at a time showed |
| that the two halves of DeepSeek-V4's low-rank attention output projection (`attn_output_a`, `attn_output_b`) |
| cannot both be at 2 bits; everything else tolerates it. A requant of our own with a correct imatrix failed the |
| same way. In the new IQ2_XS files only the routed-expert weights (most of the bytes) are IQ2_XS, with the |
| full-model imatrix sliced to the kept experts; every other tensor is copied unchanged from the Q3_K_M splice. |
| |
| ## Measured quality: read this before using them |
| |
| Scored with `llama-perplexity` against the full model's own next-token distribution on held-out text: code, |
| math, web and multilingual, 6 × 512 tokens each. The full model's perplexity is 2.48 on |
| code, 4.45 on math, 5.43 on web text and |
| 1.86 on multilingual text. Each cell gives the perplexity ratio to the full model |
| and the top-1 agreement with it. |
| |
| | Tier | Quant | GiB | code | math | web | multilingual | |
| |---|---|---|---|---|---|---| |
| | `keep12u` | Q3_K_M | 32.2 | 7.8× / 53% | 22.3× / 31% | 17.8× / 30% | 118.1× / 22% | |
| | `keep16u` | Q3_K_M | 39.3 | 7.2× / 55% | 22.9× / 32% | 18.3× / 31% | 129.9× / 21% | |
| | `keep16u-iq2xs` | IQ2_XS experts + Q3_K_M | 28.1 | 9.1× / 53% | 26.9× / 31% | 20.0× / 31% | 143.6× / 20% | |
| |
| Previous files, for comparison (same test): |
| |
| | Tier | Quant | code | math | web | multilingual | |
| |---|---|---|---|---|---| |
| | `keep8u` (previous file) | Q3_K_M | 12.3× / 51% | 27.5× / 31% | 21.0× / 29% | 145.0× / 21% | |
| | `keep12u` (previous file) | Q3_K_M | 12.5× / 50% | 27.2× / 30% | 21.5× / 30% | 140.2× / 20% | |
| | `keep16u` (previous file) | Q3_K_M | 12.0× / 50% | 26.3× / 30% | 21.3× / 29% | 130.9× / 21% | |
| |
| **These are heavily degraded models.** DeepSeek-V4 spreads its routing over many experts: even the 16 most-used |
| experts of a layer receive only about a fifth of its routing, and 64 about half. An 8–16-expert splice is not a |
| small version of DeepSeek-V4. Its losses are smallest on code, but expect wrong facts and loops on general text. |
| Larger tiers are more faithful (see the table). |
| |
| ## Files not (yet) replaced |
| |
| - `keep12u-iq2xs/DeepSeek-V4-Coder-keep12u-iq2xs.gguf`: **Known broken (previous file).** Emits garbage under upstream llama.cpp (top-1 agreement with the full model 0 %). A corrected requantisation replaces it once it passes the same checks as the tiers above. |
| - `keep64-iq2xs/DeepSeek-V4-Coder-keep64-iq2xxs.gguf`: Unchanged. Mixed expert counts behind a full-width router: loads only with UltraTensor's llama.cpp fork, not in stock llama.cpp. |
| - `keep8u-iq2xs/DeepSeek-V4-Coder-keep8u-iq2xs.gguf`: **Deprecated.** Emits garbage under upstream llama.cpp and the Ollama build (top-1 agreement 0 %), and keep8u itself is too small to be usable, so it is not requantised. |
| - `keep8u/DeepSeek-V4-Coder-keep8u.gguf`: **Deprecated.** Eight experts per layer is not enough: both the previous file and a corrected rebuild degenerate into repetition ("a a a a ..."). Kept for reproducibility; do not use. |
| |
| ## Serving |
| |
| ```bash |
| llama-server -m DeepSeek-V4-Coder-keep16u.gguf -c 4096 -ngl 0 --no-repack # CPU |
| llama-server -m DeepSeek-V4-Coder-keep16u-iq2xs.gguf -c 4096 -ngl 99 --n-cpu-moe 50 --no-op-offload # 16 GB GPU + CPU experts (not yet verified) |
| ``` |
| Every corrected file was checked on CPU, in upstream llama.cpp and in the build bundled with Ollama, before upload. |
| `--no-repack` avoids a second, repacked copy of the weights in RAM. With a GPU, `--n-cpu-moe N` keeps the experts of |
| the first N layers in system RAM; the non-expert weights of every tier take about 10.7 GiB. When we scored tiers on |
| an RTX 5060 Ti (CUDA 12.9) with `llama-perplexity`, `--no-op-offload` was required: without it, prompt processing |
| failed with an illegal memory access. GPU serving is still being measured. |
| |
| ## Provenance |
| |
| Source: `batiai/DeepSeek-V4-Pro-GGUF` Q3_K_M, 17 shards, each verified against its published sha256. Recipe, |
| audit and measurements: the Noble project (`research/phase1_*.py`, `research/release_hf.py`). File hashes: |
| `model_catalog.json`. |
| |
| **License:** MIT, inherited from `deepseek-ai/DeepSeek-V4-Pro` through the BatiAI GGUF distribution. |
| |
