checkpoint/README.md

metadata

base_model:
  - nvidia/GLM-5.3-Flash-NVFP4
license: mit
library_name: vllm
pipeline_tag: image-text-to-text
tags:
  - GLM-5
  - quantized
  - NVFP4
  - W4A16
  - derived
language:
  - ja
  - en

NVFP4 BIZ AXL: GLM-5.3-Flash-NVFP4, attention + lm_head in W4A16

Short name: NVFP4 BIZ AXL (AXL: the attention projections and lm_head in W4A16). It is the published option of the serving stack NVFP4 BIZ (Bizuayeu/GLM-5.3-Flash-NVFP4-2x-DGX-Sparks-BIZ); this repository's name describes the contents and stays as it is. Cite a revision as "NVFP4 BIZ AXL (<revision>)".

A derived checkpoint of nvidia/GLM-5.3-Flash-NVFP4. The attention-side linears and lm_head, which NVIDIA leaves in BF16, are repacked weight-only to W4A16 NVFP4; every other tensor is byte-identical to the source. It is not an NVIDIA or Z.AI release.

Use it for generation-heavy serial work, Japanese prose in particular, on the two-node GB10 (DGX Spark class) setup of Bizuayeu/GLM-5.3-Flash-NVFP4-2x-DGX-Sparks-BIZ: decode is about 40% faster than the source checkpoint at the same draft depth, prefill is a few percent faster, and about 4.4 GiB less is loaded per rank. That saving is what pays for a second sequence: the setup's AXL example profile serves two active sequences from 6 GiB of KV per rank, accepted there for routine use on 2026-09-23 at up to about 200K tokens each. Do not use it where the pinned weights must be matched exactly (code, tool use): it is not lossless — teacher-forced NLL is 1 to 6% higher on three of four test corpora.

At a glance

Source	nvidia/GLM-5.3-Flash-NVFP4, revision 423acf37583782c51c142d145aef733d72943d93, checksum-verified snapshot
Upstream	zai-org/GLM-5.3-Flash (MIT, Copyright (c) 2026 Z.AI Co., Ltd)
Repacked to W4A16 NVFP4 (group 16, no calibration data)	The KDA projections of the 34 linear-attention layers (q_proj, k_proj, v_proj, b_proj, f_a_proj, g_a_proj, f_b_proj, g_b_proj, o_proj); on the 11 MLA layers q_a_proj, kv_a_proj_with_mqa, q_b_proj, kv_b_proj, o_proj and the indexer wq_b; and lm_head
Left as shipped	Routed experts (NVIDIA's NVFP4), router, norms, embeddings, shared experts, the vision tower, the indexer wk / weights_proj, the MTP draft layer
Files	18 safetensors shards (195 GB, 182 GiB); config.json and hf_quant_config.json carry the quantization declarations; tokenizer and chat template from the source; NVIDIA's model card kept as README.nvidia.md
Conversion	requant/requant.py of tenhkspark/glm53-flash-nvfp4-2node at commit 8ee63a676c6e8550f49b52aec7c5c05b56eb5553 (Apache-2.0), locally extended to include lm_head; deterministic; about 5.5 minutes on one GB10 (CPU only). Recorded in config.json under quantization_config.producer (requant_target: "l", in_proj_layout: "split-qkv-bfg")
Tested on	Two GB10 systems (MSI EdgeXpert), TP=2 over QSFP/RoCE, vLLM 385dce36, Marlin W4A16 kernels, MTP draft depth 3, one or two active sequences (two with 6 GiB of KV per rank), 256K context with image input
License	MIT (NVIDIA's model card terms and the upstream Z.AI license, both preserved)

Requirements

The pinned vLLM (385dce36) does not route these modules through its NVFP4 path on its own, and this checkpoint declares the KDA input projection as q_proj / k_proj / v_proj plus one merged in_proj_bfg_a (b, f_a, g_a) instead of vLLM's six-in-one in_proj_qkvbfg_a. Loading therefore needs the two source overlays that the setup repository ships in its overlays/ directory:

Overlay	Replaces in vLLM 385dce36	SHA-256
kda-quant-split.py	vllm/models/glm5next/nvidia/kda.py	144a835faa57c56f5257c05a5cf6742102a21e38a92d39c2a1e26cd4ada680fd
mla-quant-split.py	vllm/models/glm5next/nvidia/model.py	a9b1ea5ce05983973bc79954147809a17e0e5aa0f96eae47c6011306f51b533d

An overlay pair belongs to one revision of this checkpoint (see Revisions); a mismatched pair fails at load. Plain vllm serve on this folder without the overlays does not start.

Loading

• With the setup repository (recommended): set runtime.derived_checkpoint in the server TOML — path to this folder, requant_target = "l", and the two overlays with their SHA-256 — and start through its launcher, which verifies the overlays and the manifest before every launch. examples/server.axl.example.toml carries all of it, with prefix-page dedup and two active sequences; replace its paths, image IDs and node details. Details: server configuration.
• Without it: copy the two overlays over the two files named above inside a vLLM 385dce36 installation, then vllm serve this folder; the declarations in config.json then reach the declared modules.

Measured against the source checkpoint

Two GB10, TP=2, MTP depth 3, temperature 0. This checkpoint is the profile the setup's reference pair serves (the AXL example: this checkpoint, prefix-page dedup, two active sequences, 6 GiB of KV per rank), measured on 2026-09-23 with one request in flight unless the row says otherwise; rows not re-measured that night keep the value of 2026-09-22, dated. The source checkpoint is the setup's distributed defaults (one sequence, 3 GiB of KV per rank), 2026-09-22 unless dated. Medians of three runs; ranges and conditions are in the setup repository's benchmarks (1.8.0, 1.10.2 and 1.10.4 sections).

Category	Measure	Source checkpoint	This checkpoint
Prefill	38,962-token prompt (tok/s)	1,233	1,295 (2026-09-22)
Decode	After a 2,048-token prompt, counting / prose / code (tok/s)	32.0 / 20.7 / 26.7	45.0 / 28.2 / 37.7
Decode	512 tokens after a short prompt (tok/s)	27.1	41.8 (2026-09-22)
Decode	sparkDash DecodeBench, 128 tokens, structured / prose / code / json (tok/s)	36.2 / 26.7 / 31.7 / 26.3 (setup 1.5.0)	48.2 / 31.4 / 41.3 / 34.9
Long input	~200K tokens, one passphrase at the midpoint (s, correct)	173.5 (199,652 tokens)	165.7 (199,649 tokens)
Long input	255,950 tokens, one passphrase at the midpoint (s, correct)	217.3	220.9 (2026-09-22)
Long input	261,461 tokens, three-position reference (s, correct 3 of 3)	235.9	227.2 (2026-09-22)
Long input	Maximum capacity, 262,080 input + 64 output tokens (s, finite logprobs)	240.3	245.7 (2026-09-22)
Concurrency	Two ~200K passphrase requests together	not served (one sequence; requests queue)	330.2 s, both correct, no preemption (166 s for one alone)
Concurrency	Decode while another request runs, counting / prose (tok/s)	not served	32.1 / 21.8
Quality	tool-eval-bench, 69 standard scenarios	90/100 (setup 1.0.0, 2026-09-14)	88/100; the same three failures, so its Safety Gate is still not passed (TC-43)
Repeatability	Identical requests at temperature 0	same completion nine times of nine within a launch	same across launches for a request alone, and always with max_num_seqs = 1 (concurrent requests queue); with max_num_seqs of 2 or more, a request that shares steps with another can get a different completion (measured on these weights and on the source checkpoint): the MoE kernel splits its sum along K by the number of expert blocks in the step, and a step shared with a prefill runs prefill-sized kernels
Memory	Weights loaded per rank / lowest free memory on the head	95.8 GiB / 5.5 GiB (3 GiB of KV)	91.3 GiB / 10.5 GiB at 3 GiB of KV (2026-09-22); 6.46 GiB at 6 GiB of KV with two 200K requests in flight

Quality, teacher-forced NLL on the setup's four fixed corpora (same inputs, the serving profile of 2026-09-25: two sequences, deterministic Inductor configs; the same on the setup's 1.12 and 1.13 images):

Corpus	Source checkpoint	This checkpoint	Change
Japanese	1.5963	1.6270	+1.9%
English	2.0241	1.9946	−1.5%
Code	0.9479	0.9601	+1.3%
Mathematics	0.5931	0.6275	+5.8%

The earlier figures (2026-09-22, one-sequence profile before the setup's 1.12 launch-state fix) were 1.6645 / 2.0024 / 1.0031 / 0.6279.

Since the setup's 1.12.0 (deterministic Inductor configs on both ranks), every launch checked has computed in the same numerical state; before it, relaunches fell into one of three states that differed in how a near tie falls. The setup checks every launch after a switch (weight digest, decode check, kernel hashes on both ranks); see its validation. Decode speed under MTP depends on how predictable the text is (counting is faster than prose); the draft is the checkpoint's own MTP layer, no extra weights.

Revisions

Date	Change	Overlay pair
2026-09-22 (current)	config.json / hf_quant_config.json declare the KDA input projection split into q_proj / k_proj / v_proj + in_proj_bfg_a (producer.in_proj_layout = "split-qkv-bfg"). Weights unchanged. Removes the prefill cost of the fused layout (one wide W4A16 Marlin GEMM), so prefill is now faster than the source checkpoint	kda-quant-split.py 144a835f…, mla-quant-split.py a9b1ea5c…
2026-09-21	Initial release; declarations for vLLM's fused in_proj_qkvbfg_a; prefill 1 to 3% slower than the source checkpoint	kda-quant.py 2e491a38…, mla-quant.py 88916cc1…

License and notices

The weights are redistributed under the MIT terms stated by the NVIDIA model card and by the upstream Z.AI license; both notices are preserved (README.nvidia.md, and the Z.AI license text in the setup repository's LICENSES/). This repackaging adds no training data and no new terms. The conversion tool is Apache-2.0 and its license applies to the tool, not to these weights. No warranty; see the source model cards for intended use and limitations.