---
base_model:
- nvidia/GLM-5.3-Flash-NVFP4
license: mit
library_name: vllm
pipeline_tag: image-text-to-text
tags:
- GLM-5
- quantized
- NVFP4
- W4A16
- derived
language:
- ja
- en
---

# NVFP4 BIZ AXL: GLM-5.3-Flash-NVFP4, attention + lm_head in W4A16

**Short name: NVFP4 BIZ AXL** (AXL: the attention projections and `lm_head` in W4A16). It is the published option of the serving stack **NVFP4 BIZ** ([Bizuayeu/GLM-5.3-Flash-NVFP4-2x-DGX-Sparks-BIZ](https://github.com/Bizuayeu/GLM-5.3-Flash-NVFP4-2x-DGX-Sparks-BIZ)); this repository's name describes the contents and stays as it is. Cite a revision as "NVFP4 BIZ AXL (`<revision>`)".

A derived checkpoint of [`nvidia/GLM-5.3-Flash-NVFP4`](https://huggingface.co/nvidia/GLM-5.3-Flash-NVFP4). The attention-side linears and `lm_head`, which NVIDIA leaves in BF16, are repacked weight-only to W4A16 NVFP4; every other tensor is byte-identical to the source. It is **not** an NVIDIA or Z.AI release.

**Use it for** generation-heavy serial work, Japanese prose in particular, on the two-node GB10 (DGX Spark class) setup of [Bizuayeu/GLM-5.3-Flash-NVFP4-2x-DGX-Sparks-BIZ](https://github.com/Bizuayeu/GLM-5.3-Flash-NVFP4-2x-DGX-Sparks-BIZ): decode is about 40% faster than the source checkpoint at the same draft depth, prefill is a few percent faster, and about 4.4 GiB less is loaded per rank. That saving is what pays for a second sequence: the setup's [AXL example profile](https://github.com/Bizuayeu/GLM-5.3-Flash-NVFP4-2x-DGX-Sparks-BIZ/blob/main/examples/server.axl.example.toml) serves two active sequences from 6 GiB of KV per rank, accepted there for routine use on 2026-09-23 at up to about 200K tokens each.
**Do not use it** where the pinned weights must be matched exactly (code, tool use): it is **not lossless** — teacher-forced NLL is 1 to 6% higher on three of four test corpora.

## At a glance

| | |
|---|---|
| Source | `nvidia/GLM-5.3-Flash-NVFP4`, revision `423acf37583782c51c142d145aef733d72943d93`, checksum-verified snapshot |
| Upstream | `zai-org/GLM-5.3-Flash` (MIT, Copyright (c) 2026 Z.AI Co., Ltd) |
| Repacked to W4A16 NVFP4 (group 16, no calibration data) | The KDA projections of the 34 linear-attention layers (`q_proj`, `k_proj`, `v_proj`, `b_proj`, `f_a_proj`, `g_a_proj`, `f_b_proj`, `g_b_proj`, `o_proj`); on the 11 MLA layers `q_a_proj`, `kv_a_proj_with_mqa`, `q_b_proj`, `kv_b_proj`, `o_proj` and the indexer `wq_b`; and `lm_head` |
| Left as shipped | Routed experts (NVIDIA's NVFP4), router, norms, embeddings, shared experts, the vision tower, the indexer `wk` / `weights_proj`, the MTP draft layer |
| Files | 18 safetensors shards (195 GB, 182 GiB); `config.json` and `hf_quant_config.json` carry the quantization declarations; tokenizer and chat template from the source; NVIDIA's model card kept as `README.nvidia.md` |
| Conversion | `requant/requant.py` of [tenhkspark/glm53-flash-nvfp4-2node](https://github.com/tenhkspark/glm53-flash-nvfp4-2node) at commit `8ee63a676c6e8550f49b52aec7c5c05b56eb5553` (Apache-2.0), locally extended to include `lm_head`; deterministic; about 5.5 minutes on one GB10 (CPU only). Recorded in `config.json` under `quantization_config.producer` (`requant_target: "l"`, `in_proj_layout: "split-qkv-bfg"`) |
| Tested on | Two GB10 systems (MSI EdgeXpert), TP=2 over QSFP/RoCE, vLLM `385dce36`, Marlin W4A16 kernels, MTP draft depth 3, one or two active sequences (two with 6 GiB of KV per rank), 256K context with image input |
| License | MIT (NVIDIA's model card terms and the upstream Z.AI license, both preserved) |

## Requirements

The pinned vLLM (`385dce36`) does not route these modules through its NVFP4 path on its own, and this checkpoint declares the KDA input projection as `q_proj` / `k_proj` / `v_proj` plus one merged `in_proj_bfg_a` (`b`, `f_a`, `g_a`) instead of vLLM's six-in-one `in_proj_qkvbfg_a`. Loading therefore needs the **two source overlays** that the setup repository ships in its [`overlays/`](https://github.com/Bizuayeu/GLM-5.3-Flash-NVFP4-2x-DGX-Sparks-BIZ/tree/main/overlays) directory:

| Overlay | Replaces in vLLM `385dce36` | SHA-256 |
|---|---|---|
| `kda-quant-split.py` | `vllm/models/glm5next/nvidia/kda.py` | `144a835faa57c56f5257c05a5cf6742102a21e38a92d39c2a1e26cd4ada680fd` |
| `mla-quant-split.py` | `vllm/models/glm5next/nvidia/model.py` | `a9b1ea5ce05983973bc79954147809a17e0e5aa0f96eae47c6011306f51b533d` |

An overlay pair belongs to one revision of this checkpoint (see [Revisions](#revisions)); a mismatched pair fails at load. Plain `vllm serve` on this folder without the overlays does not start.

## Loading

- **With the setup repository** (recommended): set `runtime.derived_checkpoint` in the server TOML — `path` to this folder, `requant_target = "l"`, and the two overlays with their SHA-256 — and start through its launcher, which verifies the overlays and the manifest before every launch. [`examples/server.axl.example.toml`](https://github.com/Bizuayeu/GLM-5.3-Flash-NVFP4-2x-DGX-Sparks-BIZ/blob/main/examples/server.axl.example.toml) carries all of it, with prefix-page dedup and two active sequences; replace its paths, image IDs and node details. Details: [server configuration](https://github.com/Bizuayeu/GLM-5.3-Flash-NVFP4-2x-DGX-Sparks-BIZ/blob/main/docs/server-configuration.md).
- **Without it**: copy the two overlays over the two files named above inside a vLLM `385dce36` installation, then `vllm serve` this folder; the declarations in `config.json` then reach the declared modules.

## Measured against the source checkpoint

Two GB10, TP=2, MTP depth 3, temperature 0. **This checkpoint** is the profile the setup's reference pair serves (the AXL example: this checkpoint, prefix-page dedup, two active sequences, 6 GiB of KV per rank), measured on 2026-09-23 with one request in flight unless the row says otherwise; rows not re-measured that night keep the value of 2026-09-22, dated. The **source checkpoint** is the setup's distributed defaults (one sequence, 3 GiB of KV per rank), 2026-09-22 unless dated. Medians of three runs; ranges and conditions are in the setup repository's [benchmarks](https://github.com/Bizuayeu/GLM-5.3-Flash-NVFP4-2x-DGX-Sparks-BIZ/blob/main/docs/benchmarks.md) (1.8.0, 1.10.2 and 1.10.4 sections).

| Category | Measure | Source checkpoint | **This checkpoint** |
|---|---|---:|---:|
| Prefill | 38,962-token prompt (tok/s) | 1,233 | **1,295** (2026-09-22) |
| Decode | After a 2,048-token prompt, counting / prose / code (tok/s) | 32.0 / 20.7 / 26.7 | **45.0 / 28.2 / 37.7** |
| Decode | 512 tokens after a short prompt (tok/s) | 27.1 | **41.8** (2026-09-22) |
| Decode | sparkDash DecodeBench, 128 tokens, structured / prose / code / json (tok/s) | 36.2 / 26.7 / 31.7 / 26.3 (setup 1.5.0) | **48.2 / 31.4 / 41.3 / 34.9** |
| Long input | ~200K tokens, one passphrase at the midpoint (s, correct) | 173.5 (199,652 tokens) | **165.7** (199,649 tokens) |
| Long input | 255,950 tokens, one passphrase at the midpoint (s, correct) | 217.3 | 220.9 (2026-09-22) |
| Long input | 261,461 tokens, three-position reference (s, correct 3 of 3) | 235.9 | **227.2** (2026-09-22) |
| Long input | Maximum capacity, 262,080 input + 64 output tokens (s, finite logprobs) | 240.3 | 245.7 (2026-09-22) |
| Concurrency | Two ~200K passphrase requests together | not served (one sequence; requests queue) | **330.2 s, both correct, no preemption** (166 s for one alone) |
| Concurrency | Decode while another request runs, counting / prose (tok/s) | not served | 32.1 / 21.8 |
| Quality | tool-eval-bench, 69 standard scenarios | 90/100 (setup 1.0.0, 2026-09-14) | 88/100; the same three failures, so its Safety Gate is still not passed (TC-43) |
| Repeatability | Identical requests at temperature 0 | same completion nine times of nine within a launch | same across launches for a request alone, and always with `max_num_seqs = 1` (concurrent requests queue); with `max_num_seqs` of 2 or more, a request that shares steps with another can get a different completion (measured on these weights and on the source checkpoint): the MoE kernel splits its sum along K by the number of expert blocks in the step, and a step shared with a prefill runs prefill-sized kernels |
| Memory | Weights loaded per rank / lowest free memory on the head | 95.8 GiB / 5.5 GiB (3 GiB of KV) | **91.3 GiB / 10.5 GiB** at 3 GiB of KV (2026-09-22); 6.46 GiB at 6 GiB of KV with two 200K requests in flight |

Quality, teacher-forced NLL on the setup's four fixed corpora (same inputs, the serving profile of 2026-09-25: two sequences, deterministic Inductor configs; the same on the setup's 1.12 and 1.13 images):

| Corpus | Source checkpoint | **This checkpoint** | Change |
|---|---:|---:|---:|
| Japanese | 1.5963 | 1.6270 | +1.9% |
| English | 2.0241 | 1.9946 | −1.5% |
| Code | 0.9479 | 0.9601 | +1.3% |
| Mathematics | 0.5931 | 0.6275 | +5.8% |

The earlier figures (2026-09-22, one-sequence profile before the setup's 1.12 launch-state fix) were 1.6645 / 2.0024 / 1.0031 / 0.6279.

Since the setup's 1.12.0 (deterministic Inductor configs on both ranks), every launch checked has computed in the same numerical state; before it, relaunches fell into one of three states that differed in how a near tie falls. The setup checks every launch after a switch (weight digest, decode check, kernel hashes on both ranks); see its [validation](https://github.com/Bizuayeu/GLM-5.3-Flash-NVFP4-2x-DGX-Sparks-BIZ/blob/main/docs/validation.md#full-model-tp2-experimental-scope). Decode speed under MTP depends on how predictable the text is (counting is faster than prose); the draft is the checkpoint's own MTP layer, no extra weights.

## Revisions

| Date | Change | Overlay pair |
|---|---|---|
| 2026-09-22 (current) | `config.json` / `hf_quant_config.json` declare the KDA input projection split into `q_proj` / `k_proj` / `v_proj` + `in_proj_bfg_a` (`producer.in_proj_layout = "split-qkv-bfg"`). Weights unchanged. Removes the prefill cost of the fused layout (one wide W4A16 Marlin GEMM), so prefill is now faster than the source checkpoint | `kda-quant-split.py` `144a835f…`, `mla-quant-split.py` `a9b1ea5c…` |
| 2026-09-21 | Initial release; declarations for vLLM's fused `in_proj_qkvbfg_a`; prefill 1 to 3% slower than the source checkpoint | `kda-quant.py` `2e491a38…`, `mla-quant.py` `88916cc1…` |

## License and notices

The weights are redistributed under the **MIT** terms stated by the NVIDIA model card and by the upstream Z.AI license; both notices are preserved (`README.nvidia.md`, and the Z.AI license text in the setup repository's `LICENSES/`). This repackaging adds no training data and no new terms. The conversion tool is Apache-2.0 and its license applies to the tool, not to these weights. No warranty; see the source model cards for intended use and limitations.