GLM-5.3-Flash FP4-Local: qualified two-Spark engine recipe
23.43% higher short/concurrent request throughput than the matched original configuration on two NVIDIA DGX Sparks, with the agreed quality and long-context gates passed. This repository publishes the qualified engine patches and model generation policy, not a newly trained or requantized checkpoint. Download the unchanged weights directly from NVIDIA.
The release uses native MTP, not DFlash. Subsequent experimental configurations did not qualify an additional gain and are not included. This is a community research release; there is no claim of a 2x speedup or universal output identity.
What changed
Four reviewable engine patches implement the combined integration:
- Reuse exact converted NVFP4 expert scales in a bounded 3 GiB per-rank buffer. This is additional non-KV storage; it does not reduce the KV allocation.
- Adapt native MTP lookahead using an acceptance-history window of 16, starting at five draft tokens and permitting at most five. Each batch size keeps its own history; non-drafting prefill resets the relevant history.
- Enable scoped request-boundary checkpoints for this supported GLM/TP2 setup.
- Keep a separate full-depth fused draft graph, with native single-step graphs at shorter depths. The combined integration was measured; its gain is not attributed to any one component in isolation.
model/generation_config.json also explicitly includes terminal EOS ID 154828, alongside the upstream terminal IDs. The target checkpoint files, quantization and target numerical kernels are unchanged.
Preserved capacity and capabilities
| Property | Qualified setting |
|---|---|
| Hardware | Two DGX Sparks, GB10, 128 GB unified memory per host |
| Parallelism | TP=2 over RoCE |
| Context limit | 262,144 tokens |
| Configured KV allocation | 8,589,934,592 bytes (8 GiB) per rank |
| Reported logical KV capacity | 808,020 tokens |
| Request slots | Four |
| KV format | FP8 |
| Enabled capabilities | Text/reasoning, vision, tools, prefix caching, chunked prefill |
The logical token count is a runtime cache-capacity report, not a per-request context limit. It represents about 3.08 full-context equivalents, not four simultaneous full 262K requests. The optional scale buffer is separate from KV.
Measured performance
Arithmetic means of block-level medians, using five distinct historical original control blocks and three complete optimized blocks. The four primary cells generate 512 tokens per request, including reasoning tokens. The following rates use request wall time, including time to first token; concurrent cells are aggregate throughput, not per-user decode speed.
| Workload | Original tok/s | Qualified tok/s | Gain |
|---|---|---|---|
| Prose, one request | 25.53 | 31.18 | 22.12% |
| Code, one request | 34.69 | 38.78 | 11.79% |
| Prose, two concurrent requests, aggregate | 35.06 | 46.11 | 31.52% |
| Prose, four concurrent requests, aggregate | 49.69 | 64.24 | 29.28% |
The geometric mean of those four matched speed ratios is 1.23434x. Long-document decode-only rates use a different denominator and are reported separately:
| Workload | Original decode tok/s | Qualified decode tok/s | Gain |
|---|---|---|---|
| About 8K input | 57.83 | 58.64 | 1.40% |
| About 32K input | 56.06 | 57.45 | 2.48% |
| Cached repeat of about 32K | 54.82 | 55.14 | 0.58% |
| About 128K input | 58.19 | 60.33 | 3.69% |
All matched throughput means passed the 98% floor, and all matched time-to-first- token means passed the 105% ceiling. Individual long-context repeats fluctuate; these small long-context differences are not separate strong speedup claims. The machine-readable result includes all three optimized block values, quality results, qualification checks and limitations.
Quality and numerical scope
- The frozen 150 automated cases plus 20 source-reviewed writing cases passed the agreed aggregate loss limit of one percentage point, with no new failing factual/tool/vision case. These are repeated fixtures, not new questions each run.
- The combined mean was 144.67/170 versus 142.20/170 in the original controls. Writing full passes were 16/20 in all three optimized core blocks, versus 15.2/20 control mean. This is retention evidence on these fixtures, not a claim that the engine improves model intelligence.
- An unsupported attendance-tracking assertion in an already-failing workshop case remains flagged and scored as a factual failure. The original model is not free of unsupported-detail or formatting failures.
- Three consecutive focused qualification blocks passed cache/state grounding, mixed load, numerical and resource checks, including a 250K-token probe.
- The fixed-token audit matched all 2,131 forced positions and observed/top-20 logprob entries in each focused repeat, using a conservative fallback path. This does not establish full-vocabulary KL or bit identity of ordinary fast decode.
- Source reviews were assistant diagnostics, not independent human adjudication. Aggregate summaries are public here; the full quality fixtures, raw responses and private operational receipts are not redistributed. This repository alone cannot independently reproduce the complete quality evaluation.
Native speculative execution showed output variability even at greedy settings; numerical differences can alter generated traces and acceptance rates. Cold/warm new-tail branch logit differences were observed in both original and optimized configurations. Exact original-prefix reuse after branching passed its checks.
Pinned dependencies
- Model:
nvidia/GLM-5.3-Flash-FP4-Local - Model revision:
76348ddb2be2e35e39ecfc00e75dc07703f4f3d5 - Container:
eugr/spark-vllm-b12x@sha256:03a956544e9f9b4b7ba91690469fbe7848bdbef29a7d2e0f5788368467bb3595 - Installed vLLM build:
0.1.dev21830+g8244d1c44.d20261015
Use the pinned container. These are source overlays for its B12X vLLM tree; compatibility with a generic or newer vLLM build has not been established. See release.json for original and qualified source fingerprints.
Reproduce the serve
Requirements: two otherwise idle, appropriately configured DGX Sparks, Docker
with GPU access, RoCE networking, Python 3.10+, the patch utility and functioning
user systemd services. Acquire the pinned image and model independently under
their respective terms. Use a trusted private fabric: the pinned multi-node
recipe uses host networking, IPC and privileged device access.
On each Spark, download this repository to a local directory. For example:
hf download DevelopingDad/GLM-5.3-Flash-FP4-Local-DGX-Spark-Optimized --local-dir ./glm53-engine
cd glm53-engine
docker pull eugr/spark-vllm-b12x@sha256:03a956544e9f9b4b7ba91690469fbe7848bdbef29a7d2e0f5788368467bb3595
hf download nvidia/GLM-5.3-Flash-FP4-Local --revision 76348ddb2be2e35e39ecfc00e75dc07703f4f3d5 --local-dir /srv/glm53/model
python3 tools/prepare_overlays.py
cp configs/cluster.example.json cluster.local.json
Edit cluster.local.json on each node. Set the actual head and worker fabric
addresses, Ethernet interface and RDMA HCAs. Set local absolute paths for the
model and runtime caches. The same topology is used on both nodes; local storage
paths may differ. Network GID and NCCL settings in configs/serve.json reflect the
tested fabric and may require host-specific configuration; changes need requalification.
prepare_overlays.py extracts only the four original source files from the pinned
image in a CPU-only, network-disabled helper. It checks original SHA256 values,
applies diffs with zero fuzz, then checks that every output is byte-identical to
the qualified source. It neither loads weights nor starts a model service.
Inspect the commands before execution:
python3 tools/serve.py --rank 1 --cluster cluster.local.json
python3 tools/serve.py --rank 0 --cluster cluster.local.json
Start rank 1 on the worker first, then rank 0 on the head:
# Worker Spark:
python3 tools/serve.py --rank 1 --cluster cluster.local.json --execute
# Head Spark:
python3 tools/serve.py --rank 0 --cluster cluster.local.json --execute
Each invocation acts only on its local host. The launcher requires at least 100 GiB host MemAvailable before loading, checks for existing GPU compute work, verifies source hashes and the image identity, and starts a detached user-service memory guard. It checks upstream model file sizes; revision pinning and upstream download verification provide content identity. The public upstream manifest also includes LFS SHA256 values for the weight shards.
The packaged API binds to loopback by default. The qualification serve used a network bind; this packaging default changes exposure, not model computation. The portable launcher was checked against the saved qualified launch arguments; it has not itself undergone a new three-suite deployment qualification.
Check readiness on the head after initialization:
curl --fail http://127.0.0.1:30198/v1/models
docker logs --tail 50 glm53-qualified-r0
Confirm 262,144 context and at least 808,020 reported logical KV tokens in the startup/cache metrics before benchmarking. The supplied speed/smoke harness uses loopback port 30198, low reasoning effort and the upstream model ID:
python3 tools/glmbench.py smoke --out results/smoke
GLM_BENCH_CACHE_NAMESPACE=local-01 python3 tools/glmbench.py speed --out results/repeat-01
Use three unchanged complete candidate repetitions and a valid saved reference for a comparison. Do not infer quality from a speed-only run. No model inference or new baseline suites were run merely to package this release.
Memory and rollback
The qualified integration has tight unified-memory headroom: the historical head minimum reached 8.000 GiB available. Keep the 8 GiB memory guard. Three consecutive seconds below that floor kills only the named local rank. The guard does not automatically stop the peer; stop both ranks if either fails. Containers have a 110 GiB memory/swap cap, and restart policy is disabled. No boot persistence or production routing is installed by these tools.
To stop the public recipe, run the applicable command on each node:
docker stop -t 60 glm53-qualified-r0
docker stop -t 60 glm53-qualified-r1
Keep your previous configuration and caches for rollback. Containers and model files are not removed. New cache directories may need substantial first-start compilation; cold startup time is not represented by warm inference throughput.
Licensing and attribution
Engine diffs and original release tooling are Apache-2.0; see LICENSE and NOTICE. The adapted generation configuration and NVIDIA model are subject to the NVIDIA Open Model Agreement. The upstream GLM model has its own additional notices. The container and its dependencies retain their own terms. No model weight shards, container images, DFlash checkpoint or unrelated experimental patches are redistributed here.
This recipe builds on vLLM, NVIDIA's model and serving recipe, and the Spark container distribution. No affiliation or endorsement is implied.