license: mit
base_model: zai-org/GLM-5.2-FP8
tags:
- glm
- glm-5.2
- moe
- int4
- quantized
- compressed-tensors
- sglang
- ktransformers
language:
- en
- zh
pipeline_tag: text-generation
GLM-5.2 — INT4 (MSE) experts + INT8 attention mix, for CPU/GPU hybrid serving
A quantized build of GLM-5.2 tuned
for CPU/GPU hybrid MoE serving on a single GPU that can't hold all 256
experts/layer at higher precision: routed MoE experts are self-quantized to
INT4 (compressed-tensors, per-group MSE-calibrated), and the attention
path (q_b_proj, o_proj, and the shared-expert MLP in most layers) is
swapped to an INT8 mix, freeing enough memory to run a substantially larger
KV-cache pool and a weight-staging overlap system without giving up
single-stream decode speed.
This is the weight side of a larger effort — see the paired serving-stack
forks for the runtime work that makes this checkpoint fast (not the
vanilla upstream projects, which lack the fixes below):
cfao-new/sglang (branch
bisect-no58) and
cfao-new/ktransformers
(branch kt064-glm52). Those READMEs document the actual measured wins
(sparse attention restoration, native fp8 MLA decode, weight-staging
overlap, and the bugs fixed along the way); this card focuses on the
checkpoint itself.
What's in this checkpoint
| Component | Precision | Method |
|---|---|---|
| Routed MoE experts (all layers) | INT4, group size 32 | Custom per-group MSE-calibrated quantization (compressed-tensors, symmetric) |
q_b_proj, o_proj, shared-expert MLP (layers 3-77) |
INT8, group size 128 | Donor tensors from QuantTrio/GLM-5.2-Int4-Int8Mix |
| Everything else (embeddings, norms, layers 0-2, indexer) | BF16 | Unchanged from the base release |
The INT4 expert quantization was run in-house against the official
zai-org/GLM-5.2-FP8 release using a custom MSE grid-search calibrator built
on compressed-tensors; it is not a redistribution of a third-party
quantization. The INT8 attention/shared-expert tensors are QuantTrio's own
public release, mixed in via the module-level scheme their build uses.
Why this specific mix
A straight INT4 quantization of everything leaves accuracy on the table on a few precision-sensitive attention/expert-selection paths. Swapping just the attention projections and shared-expert MLP to INT8 — the QuantTrio donor set — recovers that margin while keeping the bulk of the parameter count (the routed experts, which dominate total size) at INT4. The net effect versus an all-INT4 build: comparable weight footprint, meaningfully better numerical headroom on the attention path, and — combined with the serving stack's own weight-staging optimizations — enough freed memory to run this model with a much larger KV-cache pool than the unmixed build supports.
Serving
Built and validated against the paired sglang + kt-kernel forks linked above. Key serving flags (see the forks' READMEs for the complete, currently-adopted configuration):
--attention-backend nsa # native sparse attention (this is a trained-in DSA model)
--kv-cache-dtype fp8_e4m3 # native SM90 fp8 MLA decode
--max-total-tokens 524288 # funded by the memory this quantization mix frees up
Attribution and license
- Base model: zai-org/GLM-5.2-FP8, Zhipu AI, MIT license.
- INT8 attention/shared-expert donor tensors: QuantTrio/GLM-5.2-Int4-Int8Mix, MIT license.
- This checkpoint (the INT4 expert requantization and the assembly of the above into one build): released under the same MIT license, in keeping with both upstream sources.
See the included LICENSE file for the full MIT text and copyright notice.