---
license: mit
base_model: zai-org/GLM-5.2-FP8
tags:
- glm
- glm-5.2
- moe
- int4
- quantized
- compressed-tensors
- sglang
- ktransformers
language:
- en
- zh
pipeline_tag: text-generation
---

# GLM-5.2 — INT4 (MSE) experts + INT8 attention mix, for CPU/GPU hybrid serving

A quantized build of [GLM-5.2](https://huggingface.co/zai-org/GLM-5.2-FP8) tuned
for **CPU/GPU hybrid MoE serving** on a single GPU that can't hold all 256
experts/layer at higher precision: routed MoE experts are self-quantized to
INT4 (compressed-tensors, per-group MSE-calibrated), and the attention
path (`q_b_proj`, `o_proj`, and the shared-expert MLP in most layers) is
swapped to an INT8 mix, freeing enough memory to run a substantially larger
KV-cache pool and a weight-staging overlap system without giving up
single-stream decode speed.

This is the weight side of a larger effort — see the paired serving-stack
forks for the runtime work that makes this checkpoint fast (**not** the
vanilla upstream projects, which lack the fixes below):
**[cfao-new/sglang](https://github.com/cfao-new/sglang)** (branch
`bisect-no58`) and
**[cfao-new/ktransformers](https://github.com/cfao-new/ktransformers)**
(branch `kt064-glm52`). Those READMEs document the actual measured wins
(sparse attention restoration, native fp8 MLA decode, weight-staging
overlap, and the bugs fixed along the way); this card focuses on the
checkpoint itself.

## What's in this checkpoint

| Component | Precision | Method |
|---|---|---|
| Routed MoE experts (all layers) | INT4, group size 32 | Custom per-group MSE-calibrated quantization (compressed-tensors, symmetric) |
| `q_b_proj`, `o_proj`, shared-expert MLP (layers 3-77) | INT8, group size 128 | Donor tensors from [QuantTrio/GLM-5.2-Int4-Int8Mix](https://huggingface.co/QuantTrio/GLM-5.2-Int4-Int8Mix) |
| Everything else (embeddings, norms, layers 0-2, indexer) | BF16 | Unchanged from the base release |

The INT4 expert quantization was run in-house against the official
`zai-org/GLM-5.2-FP8` release using a custom MSE grid-search calibrator built
on `compressed-tensors`; it is **not** a redistribution of a third-party
quantization. The INT8 attention/shared-expert tensors are QuantTrio's own
public release, mixed in via the module-level scheme their build uses.

## Why this specific mix

A straight INT4 quantization of everything leaves accuracy on the table on a
few precision-sensitive attention/expert-selection paths. Swapping just the
attention projections and shared-expert MLP to INT8 — the QuantTrio donor
set — recovers that margin while keeping the bulk of the parameter count
(the routed experts, which dominate total size) at INT4. The net effect
versus an all-INT4 build: comparable weight footprint, meaningfully better
numerical headroom on the attention path, and — combined with the serving
stack's own weight-staging optimizations — enough freed memory to run this
model with a much larger KV-cache pool than the unmixed build supports.

## Serving

Built and validated against the paired sglang + kt-kernel forks linked
above. Key serving flags (see the forks' READMEs for the complete,
currently-adopted configuration):

```
--attention-backend nsa            # native sparse attention (this is a trained-in DSA model)
--kv-cache-dtype fp8_e4m3          # native SM90 fp8 MLA decode
--max-total-tokens 524288          # funded by the memory this quantization mix frees up
```

## Attribution and license

- Base model: [zai-org/GLM-5.2-FP8](https://huggingface.co/zai-org/GLM-5.2-FP8), Zhipu AI, **MIT license**.
- INT8 attention/shared-expert donor tensors: [QuantTrio/GLM-5.2-Int4-Int8Mix](https://huggingface.co/QuantTrio/GLM-5.2-Int4-Int8Mix), **MIT license**.
- This checkpoint (the INT4 expert requantization and the assembly of the
  above into one build): released under the same **MIT license**, in
  keeping with both upstream sources.

See the included `LICENSE` file for the full MIT text and copyright notice.