LibertAIDAI

/

GLM-5.3-Flash-NVFP4

Like 69

Follow

LibertAI 86

Hey! So I updated the model card to give pointers for a dual DGX spark cluster using sglang. Also fixed some bugs in config for it.

This is now solved on GB10, and there were two faults rather than one, which is
why it looked unexplainable. Recipe, kernel and full analysis:
https://github.com/Libertai/glm53-flash-vllm-gb10

1. Attention. On sm_121 no vLLM MLA prefill backend accepts this model's NoPE
   dimensions (256, 0, 256), and the only sparse decode backend requires
   fp8_ds_mla, whose cache kernel asserts pe_dim == 64 while GLM-5.3-Flash has
   qk_rope_head_dim = 0. We wrote a sparse-MLA CUDA kernel for sm_121 plus a vLLM
   backend, so head_size is the model's real 512, with bf16 or fp8 KV, no pe-pad and
   no dense-MHA prefill backend.

2. MoE, and this one is not GB10-specific. vLLM's ModelOptNvFp4FusedMoE reads
   an activation scale that a weight-only NVFP4 checkpoint does not carry, so
   g1_alphas becomes weight_scale_2 * 0 and every expert is multiplied by zero.
   Filed upstream as vllm-project/vllm#54189. Workaround without our plugin:
   --moe-backend marlin.

Fixing only the first still gives you the degenerate output, which is the trap.
Both ship as vllm.general_plugins entry points, so no vLLM file is patched.
Measured 24.4 tok/s at c=1 on 2× GB10 with MTP, and it now serves our production
endpoint.

Follow-up on fault 2 (the MoE one), because it now has a fix that lives in the checkpoint rather than in a plugin or a backend flag.

RedHat published their own NVFP4 of this model on 2026-08-28 with llm-compressor. Their weight recipe turns out to be the same as ours — g16, symmetric, tensor_group, FP8-E4M3 block scales — so we rebuilt our checkpoint into the compressed-tensors layout without re-quantizing anything. It is on a branch:

--revision compressed-tensors

Why it matters here: compressed-tensors never enters ModelOptNvFp4FusedMoE, so there is no uninitialised activation scale to fold into the alphas — the bug is structurally absent rather than worked around, and no plugin or --moe-backend marlin should be needed. It also resolves the ignore list against unfused module names, which removes the fused-name class of load failure at the same time. As a bonus the MTP layer is FP8 instead of FP4 there, matching RedHat.

One thing worth recording for anyone converting between the two formats: the global scale is a reciprocal. ModelOpt stores the dequant scale amax/(448*6); compressed-tensors stores the quant scale (448*6)/amax. On one expert that is 4.650297705666162e-05 against 21504.0. Invert it the wrong way and the model is silently wrong — nothing asserts.

Stated plainly, because it matters for how much weight to put on this: the branch is verified numerically only — index integrity, partition, tensor-name parity against RedHat's index, mean round-trip cosine 0.998 vs the BF16 source. No engine has loaded it yet and we have no task evals. main is untouched and is still what the working recipes in this tab target. If you get a chance to boot the branch on the Sparks without the plugin and without marlin, that result would be genuinely useful to everyone here.

A correction to my earlier reply here, since I would rather flag it than leave it standing.

I suggested booting the compressed-tensors branch on your Sparks. We tried it on our own 4x RTX PRO 6000 deployment today and it failed to load — vLLM crash-looped with exit 1 on the glm53-flash-x86_64-cu130 per-model image. I have not captured the traceback yet (the service logs to /dev/stdout and I could not stop it cleanly), so I do not know whether the cause is the checkpoint, that image snapshot, or our own plugin still on PYTHONPATH. What I did rule out: the image does support NVFP4-A16 MoE (use_a16=(input_quant is None) is in its own compressed_tensors_moe.py), and the config parses cleanly through its CompressedTensorsConfig. So please treat that branch as experimental, not as a fix to deploy.

Separately, and more usefully: the input_scale = 1.0 placeholders we shipped on main two days ago were wrong, and caused intermittent repetition that gets worse at long context (see #7). main now carries real per-projection calibrated values — re-download model-input-scales.safetensors if you pulled before today. If you are running our plugin's VLLM_GLM53_MOE_INPUT_SCALE, do not leave it at 1.0; the measured median is 1.6e-03.