| # serving/ — the matched-pair runtime |
|
|
| This checkpoint stores attention, shared/dense-MLP, and `lm_head` weights as block-FP8 |
| (`F8_E4M3` + FP32 `weight_scale_inv`). The public `glm5_next` vLLM images cannot load that: |
| they build every attention layer **unquantized** (the official checkpoints keep attention BF16, |
| so the model code strips `quant_config` at layer construction), and they have no FP8 |
| `ParallelLMHead` loader. The result on an unpatched image is a `KeyError` on the first attention |
| scale tensor, or a vocab-embedding shape assert. |
|
|
| The fix is four files on top of `cstechdev/vllm:glm53-flash-nope-sm120-cu130-20260826-r1` |
| (vLLM `0.1.dev20051+g487ecf187`), applied by the Dockerfile here: |
|
|
| | file | target path in image | semantic change | |
| |---|---|---| |
| | `kda.py` | `vllm/models/glm5next/nvidia/kda.py` | remove the save/`None`/restore strip of `vllm_config.quant_config` around KDA base-class construction (lines 168–175 upstream) — the layer now sees the real quant config; the checkpoint's `ignore` list still keeps anything BF16 that should stay BF16 | |
| | `model.py` | `vllm/models/glm5next/nvidia/model.py` | `Glm5NextDecoderLayer.__init__`: MLA attention gets `quant_config=quant_config` instead of `quant_config=None` (line 329 upstream); one site covers the 11 MLA layers and the MTP draft layer. The vision-tower `quant_config=None` site is deliberately untouched (upstream warns quantizing the tower NaNs image features; this checkpoint ships the tower BF16) | |
| | `modelopt.py` | `vllm/model_executor/layers/quantization/modelopt.py` | `ModelOptMixedPrecisionConfig`: `FP8_BLOCK128` / `FP8_BLOCK64` / `FP8_BLOCK32` per-layer dispatch to dynamic-activation block `Fp8LinearMethod`, fused-module name resolution, MTP draft-prefix aliases, and a `ParallelLMHead` method whose `weight_scale_inv` loader shards scale rows by vocab shard / 128 (exact: 154880 and the 77440-row TP2 shard divide by 128) | |
| | `configs/N=12576,K=4096,…block_shape=[32,32].json` | `vllm/model_executor/layers/quantization/utils/configs/` | autotuned triton tile configs for the KDA fused in_proj GEMM on RTX PRO 6000 Blackwell (SM120) — vLLM ships zero SM120 block-FP8 configs; the default config costs 313 µs/call vs 22.3 µs tuned (43.8% of decode GPU time → ~3%) | |
| |
| Safety property (measured): this patched image is **bitwise identical** to the vendor image for |
| BF16-attention checkpoints — the parent NVFP4 checkpoint produces 100.00% teacher-forced |
| agreement with |Δlogprob| = 0.00000 under it, because its `quantization_config.ignore` already |
| excludes every attention module. The patch only *allows* quantized attention when a checkpoint's |
| manifest asks for it. |
|
|
| All three `.py` files are Apache-2.0 vLLM-tree files (SPDX headers retained) carrying local |
| modifications; they are **not** MIT-licensed model weights. Credit: the vLLM project; the |
| official `glm5_next` per-model image as packaged by |
| [chriswritescode-dev/glm-5.3-flash-sm120](https://github.com/chriswritescode-dev/glm-5.3-flash-sm120) |
| (our base-image lineage). The base image is pulled from Docker Hub at build time and is not |
| redistributed in this repo. |
|
|
| Parallel-work credit: [local-inference-lab](https://github.com/local-inference-lab/vllm) |
| independently landed the equivalent MLA quant-config passthrough in their public vLLM fork |
| (`dev/jovian-judgement@8590bf9c`, 2026-08-29), paired with MXFP8 attention via the b12x SM120 |
| kernels. This patch set does the same for the official per-model-image code path and adds the |
| block-[32,32] FP8 dispatch for the fused KDA in_proj plus the block-FP8 lm_head loader. |
|
|
| Numerics note for the [32,32] block: the tuned configs keep `BLOCK_SIZE_K=32`, the same k-split |
| as the default config, so tuned vs untuned kernels are bit-identical in reduction order. |
|
|
