| --- |
| license: mit |
| base_model: zai-org/GLM-5.2-FP8 |
| tags: |
| - glm |
| - glm-5.2 |
| - moe |
| - int4 |
| - quantized |
| - compressed-tensors |
| - sglang |
| - ktransformers |
| language: |
| - en |
| - zh |
| pipeline_tag: text-generation |
| --- |
| |
| # GLM-5.2 — INT4 (MSE) experts + INT8 attention mix, for CPU/GPU hybrid serving |
|
|
| A quantized build of [GLM-5.2](https://huggingface.co/zai-org/GLM-5.2-FP8) tuned |
| for **CPU/GPU hybrid MoE serving** on a single GPU that can't hold all 256 |
| experts/layer at higher precision: routed MoE experts are self-quantized to |
| INT4 (compressed-tensors, per-group MSE-calibrated), and the attention |
| path (`q_b_proj`, `o_proj`, and the shared-expert MLP in most layers) is |
| swapped to an INT8 mix, freeing enough memory to run a substantially larger |
| KV-cache pool and a weight-staging overlap system without giving up |
| single-stream decode speed. |
|
|
| This is the weight side of a larger effort — see the paired serving-stack |
| forks for the runtime work that makes this checkpoint fast (**not** the |
| vanilla upstream projects, which lack the fixes below): |
| **[cfao-new/sglang](https://github.com/cfao-new/sglang)** (branch |
| `bisect-no58`) and |
| **[cfao-new/ktransformers](https://github.com/cfao-new/ktransformers)** |
| (branch `kt064-glm52`). Those READMEs document the actual measured wins |
| (sparse attention restoration, native fp8 MLA decode, weight-staging |
| overlap, and the bugs fixed along the way); this card focuses on the |
| checkpoint itself. |
|
|
| ## What's in this checkpoint |
|
|
| | Component | Precision | Method | |
| |---|---|---| |
| | Routed MoE experts (all layers) | INT4, group size 32 | Custom per-group MSE-calibrated quantization (compressed-tensors, symmetric) | |
| | `q_b_proj`, `o_proj`, shared-expert MLP (layers 3-77) | INT8, group size 128 | Donor tensors from [QuantTrio/GLM-5.2-Int4-Int8Mix](https://huggingface.co/QuantTrio/GLM-5.2-Int4-Int8Mix) | |
| | Everything else (embeddings, norms, layers 0-2, indexer) | BF16 | Unchanged from the base release | |
|
|
| The INT4 expert quantization was run in-house against the official |
| `zai-org/GLM-5.2-FP8` release using a custom MSE grid-search calibrator built |
| on `compressed-tensors`; it is **not** a redistribution of a third-party |
| quantization. The INT8 attention/shared-expert tensors are QuantTrio's own |
| public release, mixed in via the module-level scheme their build uses. |
|
|
| ## Why this specific mix |
|
|
| A straight INT4 quantization of everything leaves accuracy on the table on a |
| few precision-sensitive attention/expert-selection paths. Swapping just the |
| attention projections and shared-expert MLP to INT8 — the QuantTrio donor |
| set — recovers that margin while keeping the bulk of the parameter count |
| (the routed experts, which dominate total size) at INT4. The net effect |
| versus an all-INT4 build: comparable weight footprint, meaningfully better |
| numerical headroom on the attention path, and — combined with the serving |
| stack's own weight-staging optimizations — enough freed memory to run this |
| model with a much larger KV-cache pool than the unmixed build supports. |
|
|
| ## Serving |
|
|
| Built and validated against the paired sglang + kt-kernel forks linked |
| above. Key serving flags (see the forks' READMEs for the complete, |
| currently-adopted configuration): |
|
|
| ``` |
| --attention-backend nsa # native sparse attention (this is a trained-in DSA model) |
| --kv-cache-dtype fp8_e4m3 # native SM90 fp8 MLA decode |
| --max-total-tokens 524288 # funded by the memory this quantization mix frees up |
| ``` |
|
|
| ## Attribution and license |
|
|
| - Base model: [zai-org/GLM-5.2-FP8](https://huggingface.co/zai-org/GLM-5.2-FP8), Zhipu AI, **MIT license**. |
| - INT8 attention/shared-expert donor tensors: [QuantTrio/GLM-5.2-Int4-Int8Mix](https://huggingface.co/QuantTrio/GLM-5.2-Int4-Int8Mix), **MIT license**. |
| - This checkpoint (the INT4 expert requantization and the assembly of the |
| above into one build): released under the same **MIT license**, in |
| keeping with both upstream sources. |
|
|
| See the included `LICENSE` file for the full MIT text and copyright notice. |
|
|
|
|
