gbuzhf/KAT-Coder-V2.5-Dev-APEX-MTP-GGUF

KAT-Coder-V2.5-Dev — APEX-MTP GGUF

APEX MoE-aware mixed-precision quantizations of Kwaipilot/KAT-Coder-V2.5-Dev, with the Qwen3.6-35B-A3B multi-token-prediction (MTP) head grafted in for single-file speculative decoding. Stock llama.cpp runs these files; the MTP path additionally works with builds supporting --spec-type draft-mtp.

Files

File	Size	Recipe	Fits
Kwaipilot_KAT-Coder-V2.5-Dev-MTP-UD-Q5_K_XL.gguf	27.4 GB	Highest fidelity here — see measurements	32 GB
Kwaipilot_KAT-Coder-V2.5-Dev-APEX-MTP-I-Balanced.gguf	26.1 GB	Q6_K edges / Q5_K middle experts, Q8_0 shared, Q6_K attn, imatrix	32 GB
Kwaipilot_KAT-Coder-V2.5-Dev-APEX-MTP-I-Balanced-v2D-lite.gguf	26.2 GB	I-Balanced with the bits spent better (see below)	32 GB
Kwaipilot_KAT-Coder-V2.5-Dev-APEX-MTP-I-Compact.gguf	17.3 GB	Q4_K edges / Q3_K middle experts, Q6_K shared, Q4_K attn, imatrix	24 GB (16 GB w/ CPU-MoE)
Kwaipilot_KAT-Coder-V2.5-Dev-APEX-MTP-I-Compact-v2D-lite.gguf	17.4 GB	I-Compact with the bits spent better (see below)	24 GB (16 GB w/ CPU-MoE)
Kwaipilot_KAT-Coder-V2.5-Dev-imatrix.gguf	184 MB	importance matrix used for the I-variants	—

These are text-only GGUFs. (KAT-Coder-V2.5-Dev's config declares a vision tower, but image input does not work through llama.cpp in practice, so no mmproj is shipped.)

Verify your download

sha256sum -c sha256sums.txt --ignore-missing

File	sha256
…-MTP-UD-Q5_K_XL.gguf	d178621ee21573b5b55cf972951d8f06e5e085d970a86768aa74b3a9cb838a79
…-APEX-MTP-I-Balanced.gguf	cce1395f842b99c147cc99025267c866aaaec5cd7b8c2c9c11d5530dc10c5e7b
…-APEX-MTP-I-Balanced-v2D-lite.gguf	094e0989bb03e4e31a829321d4f2462ca17a1b586d5fd7b86166a70a031cfda1
…-APEX-MTP-I-Compact-v2D-lite.gguf	5f5fcf953b2b5757de8eabb11d9770c2d9d280a7723d67f7ad654629f8f45dda
…-APEX-MTP-I-Compact.gguf	96abb00ea46eb593456f861a0d41ded0d89fb83cbbe1255122cad82ce75648db
…-imatrix.gguf	f5727113280b24cbacc2f5308c970ab5efc8adb15d3ac4feb0e27ae1ac469c7c

(Build provenance and toolchain versions in MANIFEST.txt.)

What's special here

1. APEX quantization (mudler & Palethorpe 2026): instead of uniform bit-width, tensors are classified by role — routed experts get a layer-wise precision gradient (sensitive edge layers high, redundant middle layers low), always-active shared experts stay at high precision (heavy-tailed distributions), attention stays at Q6_K/Q4_K. On the same architecture APEX Balanced matches Q8_0 perplexity at ~⅔ the size; see the APEX technical report for full benchmarks.

2. Grafted MTP head. KAT-Coder-V2.5-Dev ships without an MTP head (mtp_num_hidden_layers: 0), but it is an architecture-identical fine-tune of Qwen3.6-35B-A3B, whose 19 mtp.* tensors were grafted into the checkpoint before conversion. llama.cpp (≥ PR #22673) bundles them as blk.40.* with nextn_predict_layers = 1; the whole draft layer is pinned at Q8_0 in both files (an under-quantized draft head kills speculative-decode acceptance).

⚠️ Caveat: the draft head was trained against the base trunk, not KAT's fine-tuned trunk, so speculative acceptance may be lower than on stock Qwen3.6. If MTP doesn't pay off for your workload, run the same file with speculation disabled — nothing else changes.

v2D-lite — same size, bits spent better

I-Balanced-v2D-lite and I-Compact-v2D-lite are the standard Balanced and Compact recipes with precision moved, not added. Two places get sharper and one gets softer, and the three changes very nearly cancel out in file size.

	I-Balanced	v2D-lite	I-Compact	v2D-lite
Size	26.13 GB	26.19 GB	17.33 GB	17.39 GB
Cost	—	+60 MB (+0.2%)	—	+60 MB (+0.3%)
Drift from the original model	1.00×	measured better ↓	1.00×	~0.76× (estimated)

What gets sharper. The output head — the layer that turns the model's internal state into an actual next word — goes to 8-bit. It runs on every token, its error lands directly in the choice of word, and nothing downstream can correct it. The key/value projections in the long-range attention layers also go up a step; they're tiny (5 MB total) but each one is shared across 8 attention heads, so any error in them gets multiplied.

What gets softer. The token embedding table drops one step to pay for it. A row of that table is read once when a token enters the model, rather than being multiplied against every token on the way out — so it's the cheapest place to take the hit.

What it costs you. 60 MB of disk, and nothing else. Same VRAM class, same speed, same --n-cpu-moe settings as the file it replaces. If you're running the classic file today, the lite version is a drop-in.

Which to pick. The gain is bigger the smaller the model: roughly 10% less drift at Balanced, roughly 24% at Compact. So I-Compact-v2D-lite is the one most worth taking — at aggressive compression the sensitive parts matter more.

Status update: the Balanced pair has since been measured (see Measured quality) — I-Balanced-v2D-lite does beat I-Balanced (95.459% vs 95.308% top-token agreement with bf16, max KLD 2.007 vs 3.480), though the top-token margin is small relative to its error bars. The Compact figure above remains an estimate interpolated from Qwen3.6-35B-A3B; that pair has not been measured. No task-benchmark comparison has been run on any of these files.

Measured quality

KL divergence against this model's own bf16 weights — not a proxy, not an estimate. All three files measured in one run, identical conditions: wikitext-2-raw, context 2048, 40,960 tokens. KAT's bf16 perplexity on this eval is 5.8024.

File	Size	Same top token as bf16	Max KLD	99.9% KLD	99.9% Δp
MTP-UD-Q5_K_XL	27.4 GB	96.237 ± 0.133 %	1.423	0.352	21.59 %
APEX-MTP-I-Balanced-v2D-lite	26.2 GB	95.459 ± 0.146 %	2.007	0.450	26.75 %
APEX-MTP-I-Balanced	26.1 GB	95.308 ± 0.148 %	3.480	0.521	27.84 %

"Same top token" is the most directly meaningful column: how often the quantized model picks the same next token as full precision. MTP-UD-Q5_K_XL agrees on +0.93 points more tokens than I-Balanced — roughly one token in every 107 that now matches where it previously didn't — and cuts worst-case divergence by 59%.

The two APEX files above were rebuilt during this run from exact per-tensor configs and came out bit-identical (matching sha256) to the published ones, so these numbers describe the files in this repo, not approximations of them.

MTP-UD-Q5_K_XL

Built on Unsloth's UD-Q5_K_XL tensor allocation — all attention and SSM output at Q8_0, shared experts Q8_0, output head and embeddings Q8_0, routed experts Q5_K with the down-projection at Q6_K. That role-level design is Unsloth's work, read from their published Qwen3.6-35B-A3B GGUF and replayed here (753 tensors, transfers 1:1).

Nine tensors deviate:

• Six trunk tensors — per-layer expert protection re-derived from KAT's own importance matrix. Unsloth protect blk.34 and blk.39, which reflect the base model's activation outliers. KAT's are elsewhere: its blk.0 gate concentrates 46% of activation energy in 1% of channels, the sharpest outlier in the model. Protection was moved to blk.0, 38, 39 — a net-zero change in bytes.
• Three MTP tensors — blk.40 experts pinned Q8_0 (+266 MB). Unsloth leave theirs lower, but their draft head is native to their trunk; this one is grafted from the base onto a fine-tuned trunk, and draft acceptance is what the MTP head is for.

Uses KAT's own imatrix (computed on this model), not the base model's.

Usage

Plain llama.cpp (MTP tensors are simply carried, no speculation):

llama-server -m Kwaipilot_KAT-Coder-V2.5-Dev-APEX-MTP-I-Compact.gguf \
  -ngl 99 --n-cpu-moe 30 -c 32768 -fa on

With MTP speculative decoding (builds supporting --spec-type draft-mtp, e.g. TurboQuant+):

llama-server -m Kwaipilot_KAT-Coder-V2.5-Dev-APEX-MTP-I-Compact.gguf \
  --spec-type draft-mtp --spec-draft-n-max 1 \
  -ngl 99 --n-cpu-moe 32 -c 32768 -fa on

Tip from measurement: MTP alone can be a net loss on novel prose; combined with ngram-style speculation it is strongly additive on copy-heavy agent workloads (diffs, file echoes, JSON). Keep the draft chain depth at 1 when combining engines.

Build provenance

• Converted from the original bf16 safetensors (convert_hf_to_gguf.py --outtype bf16). Classic tiers built with llama.cpp c0bc859 / apex-quant a445a12; the v2D-lite tiers with llama.cpp fb92d8f / apex-quant 4e95e47. Tensor-name sets are identical across both builds, so the files are directly comparable.
• Classic tiers quantized with llama-quantize --tensor-type-file using apex-quant configs qwen36_35b_mtp_balanced.txt (base Q6_K) / qwen36_35b_mtp_compact.txt (base Q4_K_M)
• v2D-lite tiers use the same base types with an exact per-tensor config (one pin per tensor), plus --output-tensor-type Q8_0. Verified post-build: 753 tensors, identical names, zero shape mismatches, exactly 22 changed tensors vs the classic file
• Importance matrix from bartowski/Kwaipilot_KAT-Coder-V2.5-Dev-GGUF
• MTP tensors byte-copied from Qwen/Qwen3.6-35B-A3B (Apache-2.0)
• MTP bundling verified post-conversion (blk.40.nextn.eh_proj present, nextn_predict_layers=1)

Credits

• Kwaipilot for KAT-Coder-V2.5-Dev (Apache-2.0)
• Qwen team for the Qwen3.6-35B-A3B base and its MTP head (Apache-2.0)
• Ettore Di Giacinto (mudler) & Richard Palethorpe / LocalAI team for the APEX method (MIT)
• bartowski for the importance matrix
• Georgi Gerganov & contributors for llama.cpp