---
license: mit
base_model: deepseek-ai/DeepSeek-V4-Pro
tags: [gguf, mixture-of-experts, deepseek, expert-pruning, llama.cpp]
---
# UltraTensor models: expert splices of DeepSeek-V4-Pro (corrected release, 2026-10-09)

Keep-N expert splices of the 1.6T-parameter DeepSeek-V4-Pro: every MoE layer keeps N of its 384 routed experts
(plus the shared expert, attention and embeddings), cut from the public BatiAI Q3_K_M GGUF. **Every tier in the
table below loads in stock llama.cpp** (DeepSeek-V4 support; verified with upstream llama.cpp and with the build
bundled with Ollama), with no fork. The older `keep64-iq2xs` file is unchanged. It keeps 64 experts behind a
full-width router, which only UltraTensor's llama.cpp fork can load.

## What changed in this release, and why

An audit that rebuilt the previous files from their recipe and measured them against the full model found four
defects. The previous files remain available in this repository's history.

1. **Routers sliced along the wrong axis** in keep8u/12u/16u. llama.cpp stores a router as one row of
   7168 floats per expert. The splicer kept columns, so no router row belonged to any kept expert. The new
   files slice rows.
2. **Experts chosen from one layer.** Every dense layer kept the experts that layer 3's code census ranked highest.
   On a full-model importance matrix those experts receive no more routing than a random choice. The new
   files keep, in each layer, the experts that layer actually uses most (full-model `llama-imatrix`, 32 × 512
   tokens of mixed calibration text).
3. **keep12u had zeroed layers.** The last 6.5 GiB of the previous file (layers 48–60) was all zeros.
4. **The IQ2_XS files emitted garbage** under upstream llama.cpp: repeated tokens, top-1 agreement 0 %. The cause
   is llama.cpp's IQ2_XS preset, which puts every tensor at 2 bits. Swapping one tensor class at a time showed
   that the two halves of DeepSeek-V4's low-rank attention output projection (`attn_output_a`, `attn_output_b`)
   cannot both be at 2 bits; everything else tolerates it. A requant of our own with a correct imatrix failed the
   same way. In the new IQ2_XS files only the routed-expert weights (most of the bytes) are IQ2_XS, with the
   full-model imatrix sliced to the kept experts; every other tensor is copied unchanged from the Q3_K_M splice.

## Measured quality: read this before using them

Scored with `llama-perplexity` against the full model's own next-token distribution on held-out text: code,
math, web and multilingual, 6 × 512 tokens each. The full model's perplexity is 2.48 on
code, 4.45 on math, 5.43 on web text and
1.86 on multilingual text. Each cell gives the perplexity ratio to the full model
and the top-1 agreement with it.

| Tier | Quant | GiB | code | math | web | multilingual |
|---|---|---|---|---|---|---|
| `keep12u` | Q3_K_M | 32.2 | 7.8× / 53% | 22.3× / 31% | 17.8× / 30% | 118.1× / 22% |
| `keep16u` | Q3_K_M | 39.3 | 7.2× / 55% | 22.9× / 32% | 18.3× / 31% | 129.9× / 21% |
| `keep16u-iq2xs` | IQ2_XS experts + Q3_K_M | 28.1 | 9.1× / 53% | 26.9× / 31% | 20.0× / 31% | 143.6× / 20% |

Previous files, for comparison (same test):

| Tier | Quant | code | math | web | multilingual |
|---|---|---|---|---|---|
| `keep8u` (previous file) | Q3_K_M | 12.3× / 51% | 27.5× / 31% | 21.0× / 29% | 145.0× / 21% |
| `keep12u` (previous file) | Q3_K_M | 12.5× / 50% | 27.2× / 30% | 21.5× / 30% | 140.2× / 20% |
| `keep16u` (previous file) | Q3_K_M | 12.0× / 50% | 26.3× / 30% | 21.3× / 29% | 130.9× / 21% |

**These are heavily degraded models.** DeepSeek-V4 spreads its routing over many experts: even the 16 most-used
experts of a layer receive only about a fifth of its routing, and 64 about half. An 8–16-expert splice is not a
small version of DeepSeek-V4. Its losses are smallest on code, but expect wrong facts and loops on general text.
Larger tiers are more faithful (see the table).

## Files not (yet) replaced

- `keep12u-iq2xs/DeepSeek-V4-Coder-keep12u-iq2xs.gguf`: **Known broken (previous file).** Emits garbage under upstream llama.cpp (top-1 agreement with the full model 0 %). A corrected requantisation replaces it once it passes the same checks as the tiers above.
- `keep64-iq2xs/DeepSeek-V4-Coder-keep64-iq2xxs.gguf`: Unchanged. Mixed expert counts behind a full-width router: loads only with UltraTensor's llama.cpp fork, not in stock llama.cpp.
- `keep8u-iq2xs/DeepSeek-V4-Coder-keep8u-iq2xs.gguf`: **Deprecated.** Emits garbage under upstream llama.cpp and the Ollama build (top-1 agreement 0 %), and keep8u itself is too small to be usable, so it is not requantised.
- `keep8u/DeepSeek-V4-Coder-keep8u.gguf`: **Deprecated.** Eight experts per layer is not enough: both the previous file and a corrected rebuild degenerate into repetition ("a a a a ..."). Kept for reproducibility; do not use.

## Serving

```bash
llama-server -m DeepSeek-V4-Coder-keep16u.gguf -c 4096 -ngl 0 --no-repack                     # CPU
llama-server -m DeepSeek-V4-Coder-keep16u-iq2xs.gguf -c 4096 -ngl 99 --n-cpu-moe 50 --no-op-offload   # 16 GB GPU + CPU experts (not yet verified)
```
Every corrected file was checked on CPU, in upstream llama.cpp and in the build bundled with Ollama, before upload.
`--no-repack` avoids a second, repacked copy of the weights in RAM. With a GPU, `--n-cpu-moe N` keeps the experts of
the first N layers in system RAM; the non-expert weights of every tier take about 10.7 GiB. When we scored tiers on
an RTX 5060 Ti (CUDA 12.9) with `llama-perplexity`, `--no-op-offload` was required: without it, prompt processing
failed with an illegal memory access. GPU serving is still being measured.

## Provenance

Source: `batiai/DeepSeek-V4-Pro-GGUF` Q3_K_M, 17 shards, each verified against its published sha256. Recipe,
audit and measurements: the Noble project (`research/phase1_*.py`, `research/release_hf.py`). File hashes:
`model_catalog.json`.

**License:** MIT, inherited from `deepseek-ai/DeepSeek-V4-Pro` through the BatiAI GGUF distribution.