Kimi-K3-GGUF

GGUF quantizations of Kimi K3 (2.8T params / 104B active / 896 experts, top-16), both verified running on a single 8×H100 + 2TB RAM node, with a full same-methodology perplexity comparison.

Tier	Size	Experts	PPL	Shards
IQ1_S-XS	539.7 GiB	896	1.9193 ± 0.0469	34
IQ1_S-XXS-832e	503.3 GiB	832	1.9634 ± 0.0490	11

IQ1_S-XXS-832e is, as far as we know, the first K3 quantization to fit under 512 GiB while remaining usable. It gets there by pruning 7.1% of the routed experts — the quality difference against IQ1_S-XS is 0.65σ, not statistically significant. See "Getting under 512 GiB".

English below · 中文见此

What the tier names mean

The main type follows the expert layers (IQ1_S), which hold 92.7% of the weight and therefore determine both size and quality. The suffixes mark our own allocation variants:

• -XS — non-expert layers pushed to IQ4_XS rather than left at Q8_0, trading speed for a smaller file. Directly comparable to unsloth's UD-IQ1_S, which uses the same expert type with a different non-expert allocation.
• -XXS-832e — same bit-width allocation as -XS, plus 64 of the 896 routed experts removed (832e = 832 experts remain). This is the only tier here that changes the parameter count rather than just the bit-width.

What this repo provides

1. A same-methodology perplexity comparison (table below). As far as we know this is the first public release of K3 quantization tiers measured under identical conditions, usable for tier selection.
2. A sub-512 GiB tier that still works, with the measurement to back it up rather than a size claim alone.
3. A different bit-width allocation: experts at IQ1_S, router kept at F32 (unquantized), attention and KDA at IQ4_XS, shared experts at Q5_K.
4. Full reproduction path and measured data: build commands, quantization parameters, pruning method, calibration corpus mix, single-node run configuration and measured throughput.

Perplexity comparison

Unified methodology: wikitext-2-raw/wiki.test.raw, 12 chunks, n_ctx=512, --n-cpu-moe 93, same machine and same llama.cpp build.

Quant	Size	Experts	PPL	vs Q8
UD-Q8_K_XL (lossless reference)	1453.9 GiB	896	1.3453 ± 0.0420	—
UD-IQ1_S	553.2 GiB	896	1.8824 ± 0.0446	+0.5371
IQ1_S-XS (this repo)	539.7 GiB	896	1.9193 ± 0.0469	+0.5740
IQ1_S-XXS-832e (this repo)	503.3 GiB	832	1.9634 ± 0.0490	+0.6181

Three caveats:

• All four rows were measured by us under identical conditions and are only comparable to each other. PPL figures published elsewhere for K3 quants generally omit the measurement setup (corpus, chunk count, n_ctx) and cannot be cross-compared with this table.
• IQ1_S-XS and UD-IQ1_S have overlapping error bars, so the quality difference is not statistically significant; size is 2.4% smaller, generation speed 6.8 vs 11.5 tok/s.
• IQ1_S-XXS-832e vs IQ1_S-XS: the difference is +0.0441 ± 0.0678, i.e. 0.65σ — well below the 1.96σ significance threshold, with heavily overlapping error bars. Removing those 64 experts bought 36.4 GiB at no measurable quality cost.

Q8_K_XL works as a lossless reference because K3 ships natively in MXFP4 (QAT from the SFT stage onward); Q8_K_XL copies the MoE layers verbatim as MXFP4 and keeps everything else at BF16.

Bit-width allocation

Tensor	Type	Rationale
ffn_{gate,down,up}_exps	IQ1_S	92.7% of total size (1347 GiB / 2723B params)
ffn_gate_inp (router)	F32, unquantized	Determines expert selection; quantizing it causes routing errors
attn_*	IQ4_XS	55.7 GiB, the largest compressible block
ffn_*_shexp (shared experts)	Q5_K	Active on every token, should not be pushed hard
ssm_* (KDA linear attention)	IQ4_XS	11.8 GiB
routed_expert_{down,up}	IQ4_XS	Shared projections of the latent MoE
token_embd / output	Q6_K	Low-bit embeddings badly damage token representations

In the actual artifact, 10 layers of ffn_down_exps were automatically promoted to Q2_K by llama.cpp's built-in IQ1_S safeguards — this accounts for the 14 GiB overshoot against our budget.

Getting under 512 GiB

512 GiB is a meaningful threshold (it fits machines with 512 GB of RAM). Two findings, in order.

Pure quantization cannot get there

IQ1_S (1.5625 bpw) is the practical floor for post-training quantization of the expert layers in llama.cpp. Holding experts at IQ1_S and compressing the non-expert layers from Q8_0 down to IQ4_XS:

Configuration	Size
Experts IQ1_S + non-experts Q8_0	553.1 GiB
Experts IQ1_S + non-experts Q6_K	538.7 GiB
Experts IQ1_S + non-experts IQ4_XS	525.3 GiB (budgeted)

Compressing the non-expert layers saves at most ~28 GiB against a 41 GiB gap. Finer per-category allocation (separate bit-widths for attn / shexp / ssm) landed at 527–532 GiB instead, indicating IQ4_XS is already near the sensible floor for those layers.

Do not use Q1_0 (1.125 bpw) or Q2_0. We tried: size does drop to 466.7 GiB, but PPL explodes to 5×10⁵ and the model is unusable. Reading the implementation in ggml/src/ggml-quants.c explains why — quantize_row_q1_0_ref is pure sign quantization: it stores only the sign of each weight, with 128 elements sharing one scale (the block's mean absolute value), discarding all magnitude information. Q2_0 uses the block amax as its scale, so a single outlier flattens the whole block. Both types are designed for models trained natively at low bit-width (BitNet-style), not for post-training quantization.

Light expert pruning does — IQ1_S-XXS-832e

Crossing 512 GiB requires reducing the parameter count itself. K3's routing turns out to be extremely uneven: in layer 40 the coldest expert is routed 52 times against the hottest at 57,948 — a factor of 1114. That skew is what makes pruning cheap here.

Importance comes from the imatrix we already had. llama.cpp accumulates activation statistics per expert for MoE tensors (e.counts[ex]++ in tools/imatrix/imatrix.cpp), so every MoE layer in the imatrix file carries counts [896] (tokens routed to each expert) and in_sum2 [896, N] (per-expert input activation energy). We score each expert by the total activation energy flowing through it:

saliency[i] = counts[i] × mean(in_sum2_down[i] / counts[i])

ffn_down_exps is the right tensor to read: its input is the expert's intermediate activation, and it projects straight back into the residual stream, so its energy is a reasonable proxy for how strongly that expert contributes.

Experts pruned	Remaining	Energy lost (median)	Worst layer	Size
3.6%	864	0.211%	0.793%	523.8 GiB
5.4%	848	0.398%	1.362%	514.7 GiB
7.1%	832	0.625%	1.999%	503.3 GiB
10.7%	800	1.197%	3.442%	487.3 GiB

We picked 832 rather than the just-barely-passing 840, to leave headroom for llama.cpp's automatic Q2_K promotion of some ffn_down_exps layers (that rule is what pushed IQ1_S-XS 14 GiB over its budget).

Honest limitation of this scoring. Cerebras REAP scores experts by gate_weight × ||expert_output||. We do not have the gate weight term and substitute routing frequency, which is correlated but not equivalent. In exchange the whole thing runs in hours instead of days and needs no extra calibration pass. The justification is empirical, not theoretical: PPL moved 0.65σ, and generation quality held up on spot checks (Chinese reasoning, code). If you need the stronger criterion, REAP is the tool.

Five tensor families must be pruned together, per layer — missing any one produces a broken model:

Tensor	Shape	Note
ffn_{gate,up,down}_exps.weight	[…, 896]	expert weights, stacked on the last dim
ffn_gate_inp.weight	[7168, 896]	router projection
exp_probs_b.bias	[896]	router per-expert bias — easiest to overlook

plus the KV kimi-k3.expert_count. K3 sets no expert_group_count, so llama.cpp's n_expert % n_expert_groups == 0 assertion does not apply and any expert count is legal.

One convenience worth noting: pruning can be applied directly to the already-quantized GGUF, no requantization needed. IQ1_S blocks are 256 elements and the expert row lengths (3584 / 3072) are both divisible by 256, so a block never straddles an expert boundary — quantize-then-prune and prune-then-quantize are byte-identical.

How to run

Requires the unsloth llama.cpp fork, which stacks on ggml-org PR #26185 (authored by llama.cpp member pwilkin, not yet merged to master):

git clone https://github.com/unslothai/llama.cpp && cd llama.cpp
git fetch origin pull/48/head:kimi-k3-fullsize-vision
git checkout kimi-k3-fullsize-vision && cd ..

# If nvcc is not on PATH, set it explicitly or cmake fails to find the CUDA compiler
export CUDACXX=/usr/local/cuda/bin/nvcc PATH=/usr/local/cuda/bin:$PATH
cmake llama.cpp -B llama.cpp/build \
  -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=90 -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j 64 \
  --target llama-cli llama-mtmd-cli llama-server llama-perplexity

Download and run — pick one tier:

# 503.3 GiB, 11 shards — fits a 512 GB machine
hf download 6block/Kimi-K3-GGUF --local-dir ./K3 \
  --include "*IQ1_S-XXS-832e*" --include "*mmproj-BF16*"

./llama.cpp/build/bin/llama-cli \
  --model ./K3/Kimi-K3-IQ1_S-XXS-832e-00001-of-00011.gguf \
  -ngl 99 --n-cpu-moe 93 -c 8192 \
  --temp 1.0 --top-p 0.95 --single-turn

# 539.7 GiB, 34 shards — all 896 experts intact
hf download 6block/Kimi-K3-GGUF --local-dir ./K3 \
  --include "*IQ1_S-XS-*" --include "*mmproj-BF16*"

./llama.cpp/build/bin/llama-cli \
  --model ./K3/Kimi-K3-IQ1_S-XS-00001-of-00034.gguf \
  -ngl 99 --n-cpu-moe 93 -c 8192 \
  --temp 1.0 --top-p 0.95 --single-turn

Note the trailing dash in --include "*IQ1_S-XS-*" — without it the pattern also matches the XXS files.

Point --model at shard 1 only; llama.cpp derives the remaining shards from its filename, so keep all shards in one directory under their published names.

--n-cpu-moe 93 places all MoE layers in CPU memory. Memory is the hard gate: you need roughly RAM+VRAM combined of 515 GB (XXS-832e) or 550 GB (XS). 8×H100 (640 GB HBM) cannot hold either alone and must be paired with large system RAM; our setup is 8×H100 + 2TB RAM, where mmap keeps resident memory at ~22 GB and the rest in page cache.

For vision, use llama-mtmd-cli with --mmproj Kimi-K3-mmproj-BF16.gguf.

Measured environment and speed

Hardware	8× H100 80GB (sm_90) + 2TB RAM + NVMe RAID0
Build	unsloth fork efc8bc38f, CUDA 12.6
Generation	6.8 tok/s (IQ1_S-XS)
Prompt	3.0 t/s
Load time	~4.5 min (first load, cold mmap)

Slower than UD-IQ1_S's 11.5 tok/s on the same machine, because non-expert layers use IQ4_XS rather than Q8_0 and carry higher dequantization cost — this is the price paid for the smaller size. IQ1_S-XXS-832e runs in the same range; pruning removes weight but does not change the per-token expert count (still top-16), so it is not proportionally faster.

Known issue: during warmup, _exps selection does not bypass top-k, so only 16 of 896 experts are loaded, making the first load slow (see the PR #26185 discussion). Work around it with --no-warmup.

Reproducing the quantization

Using unsloth's UD-Q8_K_XL (1.5 TiB, lossless) as the source:

# imatrix: 238 chunks on the 1.5TiB model takes ~11 hours
# Time estimate: chunks/4 × 708 seconds
./llama-imatrix -m Kimi-K3-UD-Q8_K_XL-00001-of-00034.gguf \
  -f calib_mix.txt -o imatrix_k3.gguf -ngl 99 --n-cpu-moe 93 -c 512

# Quantization: ~2h40m with 96 threads
# --allow-requantize is required: the expert layers in Q8_K_XL are MXFP4,
# and llama.cpp refuses to requantize from an already-quantized type by default
./llama-quantize --allow-requantize --imatrix imatrix_k3.gguf \
  --tensor-type ffn_gate_inp=f32 \
  --tensor-type attn_q=iq4_xs --tensor-type attn_k=iq4_xs \
  --tensor-type attn_v=iq4_xs --tensor-type attn_output=iq4_xs \
  --tensor-type attn_gate=iq4_xs --tensor-type attn_q_a=iq4_xs \
  --tensor-type attn_q_b=iq4_xs --tensor-type attn_kv_a=iq4_xs \
  --tensor-type attn_kv_b=iq4_xs \
  --tensor-type shexp=q5_K \
  --tensor-type ssm_g=iq4_xs --tensor-type ssm_f_a=iq4_xs \
  --tensor-type ssm_f_b=iq4_xs --tensor-type ssm_beta=iq4_xs \
  --tensor-type routed_down=iq4_xs --tensor-type routed_up=iq4_xs \
  --token-embedding-type q6_K --output-tensor-type q6_K --keep-split \
  Kimi-K3-UD-Q8_K_XL-00001-of-00034.gguf Kimi-K3-IQ1_S-XS.gguf iq1_s 96

Two easy mistakes: the global type argument does not automatically spare the non-expert layers, so you must write one --tensor-type per category for attn_* / shexp / ssm_*, or attention and shared experts get pushed to 1-bit along with the experts. And --keep-split appends its own -00001-of-000NN suffix, so do not include one in the output filename.

One more, easy to miss: general.quantized_by and general.repo_url are inherited from the source GGUF and are not rewritten by llama-quantize, so a quant made from someone else's file will carry their attribution. Likewise quantize.imatrix.file records the --imatrix path verbatim — pass a bare filename from the working directory, or the absolute server path ends up published. Check with gguf_dump.py --no-tensors before uploading.

Validate any plan change with --dry-run first — it computes the exact quant size in a few hundred milliseconds.

Reproducing the pruning (IQ1_S-XXS-832e)

Applied to the finished IQ1_S-XS GGUF, not to the 1.5 TiB source:

1. Read counts and in_sum2 per expert out of the imatrix (they are already there — see "Getting under 512 GiB"), and score each expert by total activation energy through ffn_down_exps.
2. Per layer, keep the top 832 by score, sorted ascending to preserve relative order.
3. Slice all five tensor families on the expert dimension. In gguf-py's numpy view the expert dimension is the first axis ((896, rows, row_bytes)), so this is plain data[keep].
4. Rewrite kimi-k3.expert_count to 832.

Two things to watch when rewriting a GGUF by hand:

• The data section starts at the tensor-info end rounded up to general.alignment. Changing the KV section length moves that absolute position, so padding must be recomputed — copying the original padding silently misaligns every tensor offset.
• GGUFWriter in split mode requires add_tensor_info for all tensors first, then write_header_to_file → write_kv_data_to_file → write_ti_data_to_file → write_tensor_data per tensor, in registration order. Skipping write_ti_data_to_file fails with Expected output file to contain tensor info or weights, got WriterState.KV_DATA.

Calibration corpus

Code 35% / English 30% / Chinese 25% / other 10%. The Chinese share is deliberately higher than the common setting (~15%): in a highly sparse MoE, content under-represented in the corpus gets silently sacrificed during calibration. Code is drawn from the llama.cpp source tree (mixed C/C++/CUDA/Python/CMake), English from wikitext-2, Chinese from wikimedia/wikipedia 20231101.zh. Corpus blocks are interleaved and shuffled rather than concatenated by category — imatrix accumulates activation statistics per chunk, and concatenation would concentrate each category into a few chunks, skewing the weight distribution.

The imatrix run reports 99.89% coverage: roughly 1 of 896 experts was never routed to within the 238 chunks. This is expected for top-16/896 sparsity; llama-quantize falls back to the default quantization for missing entries.

Credits

• moonshotai/Kimi-K3 — original model
• unsloth — UD-Q8_K_XL lossless source; full-size model fixes and the MoonViT-3d vision tower in the llama.cpp fork
• pwilkin — Kimi-K3 architecture support in llama.cpp

License

Inherits the Kimi K3 License (near-MIT; explicitly permits modification, distribution, sublicensing and derivative works. Obligations apply only to MaaS businesses with over $20M annual revenue, and to products with over 100M MAU or $20M monthly revenue, which must display "Kimi K3" in their UI).

---

中文说明

Kimi K3（2.8T 参数 / 104B 激活 / 896 experts top-16）的 GGUF 量化，两档均在单机 8×H100 + 2TB 内存上实测跑通，附完整同口径 perplexity 对照数据。

档位	体积	专家数	PPL	分片
IQ1_S-XS	539.7 GiB	896	1.9193 ± 0.0469	34
IQ1_S-XXS-832e	503.3 GiB	832	1.9634 ± 0.0490	11

据我们所知，IQ1_S-XXS-832e 是第一个压进 512 GiB 以内且仍可用的 K3 量化。它靠剪掉 7.1% 的路由专家跨过这个门槛，与 IQ1_S-XS 的质量差异是 0.65σ，不具统计显著性。 方法见下文「如何压进 512 GiB」。

档位命名的含义

主类型跟随专家层（IQ1_S）—— 专家层占 92.7% 的权重，是体积与质量的决定因素。 后缀标识我们自己的分配变体：

• -XS —— 非专家层压到 IQ4_XS 而非保持 Q8_0，以速度换体积。它与 unsloth 的 UD-IQ1_S 可直接对比 —— 专家层类型相同，非专家层分配不同。
• -XXS-832e —— 位宽分配与 -XS 完全相同，另外剪掉了 896 个路由专家中的 64 个 （832e 即剩余 832 个专家）。这是本仓库唯一改变参数量而非仅改位宽的档位。

本仓库提供什么

1. 一套同口径的 perplexity 对照数据（见下表）。据我们所知这是首次公开发布 K3 各量化档在完全相同测量条件下的对照，可用于判断量化档位选择。
2. 一个真的压进 512 GiB 且仍可用的档位，并附上支撑它的实测数据，而不只是一个体积声明。
3. 一份不同的位宽分配方案：专家层 IQ1_S、router 保 F32 不量化、attention 与 KDA 降至 IQ4_XS、shared experts 保 Q5_K。
4. 完整的复现路径与实测数据：构建命令、量化参数、剪枝方法、校准语料配比、单机运行配置与实测速度，见下文。

Perplexity 对照

统一口径：wikitext-2-raw/wiki.test.raw，12 chunks，n_ctx=512，--n-cpu-moe 93，同一台机器同一 llama.cpp 构建。

量化	体积	专家数	PPL	相对 Q8
UD-Q8_K_XL（无损基准）	1453.9 GiB	896	1.3453 ± 0.0420	—
UD-IQ1_S	553.2 GiB	896	1.8824 ± 0.0446	+0.5371
IQ1_S-XS（本仓库）	539.7 GiB	896	1.9193 ± 0.0469	+0.5740
IQ1_S-XXS-832e（本仓库）	503.3 GiB	832	1.9634 ± 0.0490	+0.6181

注意三点：

• 上表四行均为我们在同一条件下自测，只在彼此之间可比。其他来源公布的 K3 量化 PPL 通常未附测量条件（语料、chunk 数、n_ctx），不能与此表交叉比较。
• IQ1_S-XS 与 UD-IQ1_S 的误差区间重叠，质量差异不具统计显著性；体积小 2.4%，生成速度 6.8 vs 11.5 tok/s。
• IQ1_S-XXS-832e 相对 IQ1_S-XS：差值 +0.0441 ± 0.0678，即 0.65σ，远低于 1.96σ 的显著性门槛，误差区间大幅重叠。剪掉那 64 个专家换来 36.4 GiB，质量上没有可测量的代价。

Q8_K_XL 之所以可作无损基准：K3 原生就是 MXFP4 权重（QAT，从 SFT 阶段起），Q8_K_XL 的 MoE 层原封不动照抄 MXFP4，其余为 BF16。

位宽分配

张量	类型	理由
ffn_{gate,down,up}_exps	IQ1_S	占 92.7% 体积（1347 GiB / 2723B 参数）
ffn_gate_inp（router）	F32 不量化	决定专家选择，量化会导致路由错误
attn_*	IQ4_XS	55.7 GiB，最大可压空间
ffn_*_shexp（shared experts）	Q5_K	每个 token 都激活，不宜压狠
ssm_*（KDA 线性注意力）	IQ4_XS	11.8 GiB
routed_expert_{down,up}	IQ4_XS	latent MoE 的共享投影
token_embd / output	Q6_K	embedding 压到低比特会严重伤 token 表示

实际产物中 ffn_down_exps 有 10 层被 llama.cpp 的 IQ1_S 内置保护规则自动提升到 Q2_K —— 这是我们体积超出预算 14 GiB 的原因，见下文。

如何压进 512 GiB

512 GiB 是个有意义的门槛（512 GB 内存机型能装下）。两个结论，按顺序。

纯量化做不到

IQ1_S（1.5625 bpw）是 llama.cpp 后训练量化在专家层的实际下限。核算全部方案（专家层固定 IQ1_S，非专家层从 Q8_0 压到 IQ4_XS）：

方案	体积
专家 IQ1_S + 非专家 Q8_0	553.1 GiB
专家 IQ1_S + 非专家 Q6_K	538.7 GiB
专家 IQ1_S + 非专家 IQ4_XS	525.3 GiB（预算）

压缩非专家层最多省约 28 GiB，而缺口是 41 GiB。更精细的分层分配（attn/shexp/ssm 各给不同位宽）反而落在 527–532 GiB，说明 IQ4_XS 已接近非专家层的合理下限。

不要用 Q1_0（1.125 bpw）或 Q2_0。我们试过：体积确实降到 466.7 GiB，但 PPL 爆到 5×10⁵，模型完全不可用。读 ggml/src/ggml-quants.c 的实现可知，quantize_row_q1_0_ref 是纯符号量化 —— 只存权重正负号，128 个元素共享一个 scale（块内绝对值均值），幅值信息全部丢弃；Q2_0 用块内 amax 作 scale，单个离群权重就能压扁整块。这两个类型是为原生低比特训练的模型（BitNet 一类）设计的，不适用于后训练量化。

轻度专家剪枝可以 —— IQ1_S-XXS-832e

跨过 512 GiB 必须减少参数量本身。K3 的路由恰好极度不均：第 40 层最冷的专家被路由 52 次，最热的 57948 次，差 1114 倍。正是这个倾斜让剪枝在这里代价很低。

重要度来自我们本来就有的 imatrix。llama.cpp 对 MoE 张量是按专家分块累积激活统计的（tools/imatrix/imatrix.cpp 里的 e.counts[ex]++），所以 imatrix 文件里每个 MoE 层都已经带着 counts [896]（每个专家被路由到的 token 数）和 in_sum2 [896, N]（每专家的输入激活能量）。我们按流经每个专家的总激活能量打分：

saliency[i] = counts[i] × mean(in_sum2_down[i] / counts[i])

选 ffn_down_exps 是对的：它的输入就是专家的中间激活，而它直接把结果投回残差流，所以它的能量是「该专家对主干贡献强度」的合理代理。

剪掉比例	剩余专家	丢失能量（中位数）	最差层	体积
3.6%	864	0.211%	0.793%	523.8 GiB
5.4%	848	0.398%	1.362%	514.7 GiB
7.1%	832	0.625%	1.999%	503.3 GiB
10.7%	800	1.197%	3.442%	487.3 GiB

我们选 832 而不是刚好达标的 840，是为了给 llama.cpp 把部分 ffn_down_exps 层自动提升到 Q2_K 的规则留出余量（IQ1_S-XS 就是被这条规则打超预算 14 GiB 的）。

这套打分方法的诚实局限。Cerebras REAP 的判据是 gate_weight × ||expert_output||。我们没有 gate 权重这一项，用路由频次代替 —— 相关但不等价。换来的是整件事从几天变成几小时，且不需要额外的校准前向传播。它的依据是实证而非理论：PPL 只动了 0.65σ，生成质量在抽查（中文推理、代码）中也保持住了。如果你需要更强的判据，REAP 是那个工具。

每层必须同步剪掉五类张量，漏一个就会做出坏模型：

张量	形状	说明
ffn_{gate,up,down}_exps.weight	[…, 896]	专家权重，堆叠在最后一维
ffn_gate_inp.weight	[7168, 896]	router 投影
exp_probs_b.bias	[896]	router 的 per-expert 偏置，最容易漏

以及 KV kimi-k3.expert_count。K3 没有设 expert_group_count，所以 llama.cpp 的 n_expert % n_expert_groups == 0 断言不触发，任意专家数都合法。

一个省事的地方：剪枝可以直接作用在已量化的 GGUF 上，不需要重新量化。IQ1_S 的 block 是 256 个元素，而专家行长（3584 / 3072）都能被 256 整除，所以 block 永远不会跨过专家边界 —— 先量化再剪与先剪再量化的产物逐字节相同。

运行方法

需要 unsloth 的 llama.cpp fork（叠加在 ggml-org PR #26185 之上，后者由 llama.cpp 成员 pwilkin 编写，尚未合并到 master）：

git clone https://github.com/unslothai/llama.cpp && cd llama.cpp
git fetch origin pull/48/head:kimi-k3-fullsize-vision
git checkout kimi-k3-fullsize-vision && cd ..

# nvcc 若不在 PATH 需显式指定，否则 cmake 报找不到 CUDA compiler
export CUDACXX=/usr/local/cuda/bin/nvcc PATH=/usr/local/cuda/bin:$PATH
cmake llama.cpp -B llama.cpp/build \
  -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=90 -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j 64 \
  --target llama-cli llama-mtmd-cli llama-server llama-perplexity

下载与运行 —— 选一档：

# 503.3 GiB，11 分片 —— 512 GB 内存机型能装下
hf download 6block/Kimi-K3-GGUF --local-dir ./K3 \
  --include "*IQ1_S-XXS-832e*" --include "*mmproj-BF16*"

./llama.cpp/build/bin/llama-cli \
  --model ./K3/Kimi-K3-IQ1_S-XXS-832e-00001-of-00011.gguf \
  -ngl 99 --n-cpu-moe 93 -c 8192 \
  --temp 1.0 --top-p 0.95 --single-turn

# 539.7 GiB，34 分片 —— 896 个专家完整保留
hf download 6block/Kimi-K3-GGUF --local-dir ./K3 \
  --include "*IQ1_S-XS-*" --include "*mmproj-BF16*"

./llama.cpp/build/bin/llama-cli \
  --model ./K3/Kimi-K3-IQ1_S-XS-00001-of-00034.gguf \
  -ngl 99 --n-cpu-moe 93 -c 8192 \
  --temp 1.0 --top-p 0.95 --single-turn

注意 --include "*IQ1_S-XS-*" 末尾那个短横线 —— 不加的话这个模式会连 XXS 的文件一起匹配上。

--model 只指第 1 分片，llama.cpp 会按文件名推导其余分片，所以全部分片要放在同一目录并保持发布时的文件名。

--n-cpu-moe 93 把全部 MoE 层放 CPU 内存。内存是硬门槛：RAM+VRAM 合计需要约 515 GB（XXS-832e）或 550 GB（XS）。8×H100（640 GB HBM）单独都装不下，必须配合大内存；我们的配置是 8×H100 + 2TB RAM，mmap 模式下常驻内存约 22 GB、其余走 page cache。

多模态（视觉）用 llama-mtmd-cli 并加 --mmproj Kimi-K3-mmproj-BF16.gguf。

实测环境与速度

硬件	8× H100 80GB (sm_90) + 2TB RAM + NVMe RAID0
构建	unsloth fork efc8bc38f，CUDA 12.6
生成	6.8 tok/s（IQ1_S-XS）
Prompt	3.0 t/s
加载	约 4.5 分钟（首次，mmap 冷启动）

速度低于同机上 UD-IQ1_S 的 11.5 tok/s，原因是非专家层用 IQ4_XS 而非 Q8_0，反量化开销更大 —— 这是换取体积的代价。IQ1_S-XXS-832e 速度在同一区间：剪枝减少了权重总量，但没有改变每个 token 激活的专家数（仍是 top-16），所以不会成比例变快。

已知问题：warmup 阶段对 _exps 的选择未绕过 top-k，只加载 896 个专家中的 16 个，导致首次加载偏慢（见 PR #26185 讨论）。可用 --no-warmup 规避。

量化复现

从 unsloth 的 UD-Q8_K_XL（1.5 TiB，无损）作为源：

# imatrix：238 chunks 在 1.5TiB 模型上约 11 小时
# 耗时估算：chunks/4 × 708 秒
./llama-imatrix -m Kimi-K3-UD-Q8_K_XL-00001-of-00034.gguf \
  -f calib_mix.txt -o imatrix_k3.gguf -ngl 99 --n-cpu-moe 93 -c 512

# 量化：96 线程约 2h40m
# --allow-requantize 是必需的：Q8_K_XL 的专家层是 MXFP4，
# llama.cpp 默认禁止从已量化类型再量化
./llama-quantize --allow-requantize --imatrix imatrix_k3.gguf \
  --tensor-type ffn_gate_inp=f32 \
  --tensor-type attn_q=iq4_xs --tensor-type attn_k=iq4_xs \
  --tensor-type attn_v=iq4_xs --tensor-type attn_output=iq4_xs \
  --tensor-type attn_gate=iq4_xs --tensor-type attn_q_a=iq4_xs \
  --tensor-type attn_q_b=iq4_xs --tensor-type attn_kv_a=iq4_xs \
  --tensor-type attn_kv_b=iq4_xs \
  --tensor-type shexp=q5_K \
  --tensor-type ssm_g=iq4_xs --tensor-type ssm_f_a=iq4_xs \
  --tensor-type ssm_f_b=iq4_xs --tensor-type ssm_beta=iq4_xs \
  --tensor-type routed_down=iq4_xs --tensor-type routed_up=iq4_xs \
  --token-embedding-type q6_K --output-tensor-type q6_K --keep-split \
  Kimi-K3-UD-Q8_K_XL-00001-of-00034.gguf Kimi-K3-IQ1_S-XS.gguf iq1_s 96

两个容易踩的点：全局类型参数不会自动放过非专家层，必须为 attn_*/shexp/ssm_* 各写一条 --tensor-type，否则 attention 和 shared experts 会一起被压到 1-bit；--keep-split 会自行添加 -00001-of-000NN 后缀，输出文件名里不要重复带。

还有一个容易漏的：general.quantized_by 和 general.repo_url 从源 GGUF 继承，llama-quantize 不会改写，所以基于别人的文件做量化会带着对方的署名。同理 quantize.imatrix.file 原样记录 --imatrix 的路径 —— 要在工作目录下只传文件名，否则服务器绝对路径会被发布出去。 上传前用 gguf_dump.py --no-tensors 检查。

改方案前用 --dry-run 验证，它能在数百毫秒内算出准确的 quant size。

剪枝复现（IQ1_S-XXS-832e）

作用在已完成的 IQ1_S-XS GGUF 上，不需要动 1.5 TiB 的源：

1. 从 imatrix 里读出每个专家的 counts 和 in_sum2（本来就在里面，见上文「如何压进 512 GiB」），按流经 ffn_down_exps 的总激活能量给每个专家打分。
2. 每层保留分数最高的 832 个，下标升序排列以保持原有相对顺序。
3. 在专家维上切片全部五类张量。在 gguf-py 的 numpy 视图里专家维是第一维（(896, rows, row_bytes)），所以就是一句 data[keep]。
4. 把 KV kimi-k3.expert_count 改写为 832。

手工重写 GGUF 时有两点要留意：

• 数据段起点 = 张量索引段结束位置按 general.alignment 向上对齐。改动 KV 段长度会移动这个绝对位置，所以 padding 必须重新计算 —— 照抄原来的 padding 会让所有张量偏移静默错位。
• GGUFWriter 在分片模式下要求先 add_tensor_info 注册全部张量，然后 write_header_to_file → write_kv_data_to_file → write_ti_data_to_file → 按注册顺序逐个 write_tensor_data。漏掉 write_ti_data_to_file 会报 Expected output file to contain tensor info or weights, got WriterState.KV_DATA。

校准语料

代码 35% / 英文 30% / 中文 25% / 其他 10%。中文比例高于常见配置（通常 15%）是有意的：超稀疏 MoE 中语料占比不足的内容会在校准中被静默牺牲。代码取自 llama.cpp 源码树（C/C++/CUDA/Python/CMake 混合），英文为 wikitext-2，中文为 wikimedia/wikipedia 20231101.zh。语料块交错打乱而非分段堆叠 —— imatrix 按 chunk 累积激活统计，分段堆叠会使某类内容集中于少数 chunk，导致权重分布失真。

imatrix 生成时有 99.89% 覆盖率警告：896 个专家中约 1 个在 238 chunks 内从未被路由激活。这是 top-16/896 超稀疏 MoE 的预期现象，llama-quantize 对缺失项回退默认量化。

致谢

• moonshotai/Kimi-K3 — 原始模型
• unsloth — UD-Q8_K_XL 无损源、llama.cpp fork 的满血模型修复与 MoonViT-3d vision tower
• pwilkin — llama.cpp 的 Kimi-K3 架构支持

许可

继承 Kimi K3 License（近 MIT，明确允许修改、分发、再许可与创建衍生作品；约束仅适用于年营收超 2000 万美元的 MaaS 业务、以及 MAU 超 1 亿或月营收超 2000 万美元的产品需在 UI 显示 "Kimi K3"）。