license: mit
base_model: deepseek-ai/DeepSeek-V4-Pro
tags:
- gguf
- mixture-of-experts
- deepseek
- expert-pruning
- llama.cpp
UltraTensor models: expert splices of DeepSeek-V4-Pro (corrected release, 2026-10-09)
Keep-N expert splices of the 1.6T-parameter DeepSeek-V4-Pro: every MoE layer keeps N of its 384 routed experts
(plus the shared expert, attention and embeddings), cut from the public BatiAI Q3_K_M GGUF. Every tier in the
table below loads in stock llama.cpp (DeepSeek-V4 support; verified with upstream llama.cpp and with the build
bundled with Ollama), with no fork. The older keep64-iq2xs file is unchanged. It keeps 64 experts behind a
full-width router, which only UltraTensor's llama.cpp fork can load.
What changed in this release, and why
An audit that rebuilt the previous files from their recipe and measured them against the full model found four defects. The previous files remain available in this repository's history.
- Routers sliced along the wrong axis in keep8u/12u/16u. llama.cpp stores a router as one row of 7168 floats per expert. The splicer kept columns, so no router row belonged to any kept expert. The new files slice rows.
- Experts chosen from one layer. Every dense layer kept the experts that layer 3's code census ranked highest.
On a full-model importance matrix those experts receive no more routing than a random choice. The new
files keep, in each layer, the experts that layer actually uses most (full-model
llama-imatrix, 32 × 512 tokens of mixed calibration text). - keep12u had zeroed layers. The last 6.5 GiB of the previous file (layers 48–60) was all zeros.
- The IQ2_XS files emitted garbage under upstream llama.cpp: repeated tokens, top-1 agreement 0 %. The cause
is llama.cpp's IQ2_XS preset, which puts every tensor at 2 bits. Swapping one tensor class at a time showed
that the two halves of DeepSeek-V4's low-rank attention output projection (
attn_output_a,attn_output_b) cannot both be at 2 bits; everything else tolerates it. A requant of our own with a correct imatrix failed the same way. In the new IQ2_XS files only the routed-expert weights (most of the bytes) are IQ2_XS, with the full-model imatrix sliced to the kept experts; every other tensor is copied unchanged from the Q3_K_M splice.
Measured quality: read this before using them
Scored with llama-perplexity against the full model's own next-token distribution on held-out text: code,
math, web and multilingual, 6 × 512 tokens each. The full model's perplexity is 2.48 on
code, 4.45 on math, 5.43 on web text and
1.86 on multilingual text. Each cell gives the perplexity ratio to the full model
and the top-1 agreement with it.
| Tier | Quant | GiB | code | math | web | multilingual |
|---|---|---|---|---|---|---|
keep12u |
Q3_K_M | 32.2 | 7.8× / 53% | 22.3× / 31% | 17.8× / 30% | 118.1× / 22% |
keep16u |
Q3_K_M | 39.3 | 7.2× / 55% | 22.9× / 32% | 18.3× / 31% | 129.9× / 21% |
keep16u-iq2xs |
IQ2_XS experts + Q3_K_M | 28.1 | 9.1× / 53% | 26.9× / 31% | 20.0× / 31% | 143.6× / 20% |
Previous files, for comparison (same test):
| Tier | Quant | code | math | web | multilingual |
|---|---|---|---|---|---|
keep8u (previous file) |
Q3_K_M | 12.3× / 51% | 27.5× / 31% | 21.0× / 29% | 145.0× / 21% |
keep12u (previous file) |
Q3_K_M | 12.5× / 50% | 27.2× / 30% | 21.5× / 30% | 140.2× / 20% |
keep16u (previous file) |
Q3_K_M | 12.0× / 50% | 26.3× / 30% | 21.3× / 29% | 130.9× / 21% |
These are heavily degraded models. DeepSeek-V4 spreads its routing over many experts: even the 16 most-used experts of a layer receive only about a fifth of its routing, and 64 about half. An 8–16-expert splice is not a small version of DeepSeek-V4. Its losses are smallest on code, but expect wrong facts and loops on general text. Larger tiers are more faithful (see the table).
Files not (yet) replaced
keep12u-iq2xs/DeepSeek-V4-Coder-keep12u-iq2xs.gguf: Known broken (previous file). Emits garbage under upstream llama.cpp (top-1 agreement with the full model 0 %). A corrected requantisation replaces it once it passes the same checks as the tiers above.keep64-iq2xs/DeepSeek-V4-Coder-keep64-iq2xxs.gguf: Unchanged. Mixed expert counts behind a full-width router: loads only with UltraTensor's llama.cpp fork, not in stock llama.cpp.keep8u-iq2xs/DeepSeek-V4-Coder-keep8u-iq2xs.gguf: Deprecated. Emits garbage under upstream llama.cpp and the Ollama build (top-1 agreement 0 %), and keep8u itself is too small to be usable, so it is not requantised.keep8u/DeepSeek-V4-Coder-keep8u.gguf: Deprecated. Eight experts per layer is not enough: both the previous file and a corrected rebuild degenerate into repetition ("a a a a ..."). Kept for reproducibility; do not use.
Serving
llama-server -m DeepSeek-V4-Coder-keep16u.gguf -c 4096 -ngl 0 --no-repack # CPU
llama-server -m DeepSeek-V4-Coder-keep16u-iq2xs.gguf -c 4096 -ngl 99 --n-cpu-moe 50 --no-op-offload # 16 GB GPU + CPU experts (not yet verified)
Every corrected file was checked on CPU, in upstream llama.cpp and in the build bundled with Ollama, before upload.
--no-repack avoids a second, repacked copy of the weights in RAM. With a GPU, --n-cpu-moe N keeps the experts of
the first N layers in system RAM; the non-expert weights of every tier take about 10.7 GiB. When we scored tiers on
an RTX 5060 Ti (CUDA 12.9) with llama-perplexity, --no-op-offload was required: without it, prompt processing
failed with an illegal memory access. GPU serving is still being measured.
Provenance
Source: batiai/DeepSeek-V4-Pro-GGUF Q3_K_M, 17 shards, each verified against its published sha256. Recipe,
audit and measurements: the Noble project (research/phase1_*.py, research/release_hf.py). File hashes:
model_catalog.json.
License: MIT, inherited from deepseek-ai/DeepSeek-V4-Pro through the BatiAI GGUF distribution.