---
library_name: ninfer
pipeline_tag: image-text-to-text
inference: false
license: mit
base_model:
- zai-org/GLM-5.3-Flash
- nvidia/GLM-5.3-Flash-NVFP4
base_model_relation: quantized
tags:
- ninfer
- glm5next
- nvfp4
- q8
- mtp
- blackwell
- dgx-spark
- multimodal
- conversational
- cuda
---

# GLM-5.3-Flash for NInfer

A mixed-precision NInfer v3 artifact of
[GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash), converted from
[NVIDIA's NVFP4 checkpoint](https://huggingface.co/nvidia/GLM-5.3-Flash-NVFP4).
It includes Text, Vision, native MTP and an additional full-vocabulary Q4 draft head.

The native NInfer architecture is GLM5Next, combining KDA linear attention, Kpool sparse MLA,
mHC and MoE. The artifact includes the source tokenizer, GLM chat template and processor resources.
Use it directly with [NInfer](https://github.com/Neroued/ninfer) for CLI generation or
OpenAI-/Anthropic-compatible serving on **two DGX Sparks**.

## Download

```bash
hf download neroued/GLM-5.3-Flash-NInfer \
  --include 'glm5_3_flash.ninfer*' \
  --local-dir models/GLM-5.3-Flash-NInfer
```

The complete artifact is **178.33 GiB** across six files:

```text
glm5_3_flash.ninfer
glm5_3_flash.ninfer.part-0001
glm5_3_flash.ninfer.part-0002
glm5_3_flash.ninfer.part-0003
glm5_3_flash.ninfer.part-0004
glm5_3_flash.ninfer.part-0005
```

Keep all parts beside the entry file and place the complete set on each node. Pass only
`glm5_3_flash.ninfer` to NInfer; rank-specific weight slicing happens during loading.
File sizes and the artifact identity are in [artifact-manifest.json](artifact-manifest.json).

## Representation

| Weights | Stored representation |
|---|---|
| Main routed experts and first three dense MLP layers | Imported NVIDIA NVFP4, with calibrated scales |
| KDA Q/K/V/O, MLA output and shared-expert projections | Q8 A16 |
| MTP routed experts | Block-FP8 converted from source BF16 |
| Main head, embedding, mHC, Vision and other retained parameters | BF16; source FP32 KDA/control fields remain FP32 |
| Optimized MTP head | Additional full-vocabulary Q4 A16; Main head stays BF16 |

The [`glm5next` recipe](https://github.com/Neroued/ninfer/blob/master/tools/convert/official_recipes.py)
retains imported Main NVFP4 codes/scales and performs the additional quantization above.
The optimized draft head does not crop the vocabulary.

## Run on two Sparks

Build the current NInfer revision on both nodes with NCCL development dependencies installed:

```bash
cmake --preset dgx-spark-release -DNINFER_ENABLE_DISTRIBUTED=ON
cmake --build build/dgx-spark-release -j
```

Full dependency and checkout instructions are in the [build guide](https://github.com/Neroued/ninfer#build).
From the repository root, prepare the deployment config:

```bash
cp examples/configs/spark2-glm5_3-flash.toml ninfer.local.toml
# Edit coordinator/worker artifact paths, worker executable paths and network addresses.
./build/dgx-spark-release/apps/ninfer-serve --config ninfer.local.toml
```

The [example](https://github.com/Neroued/ninfer/blob/master/examples/configs/spark2-glm5_3-flash.toml)
uses TP2 with attention DP2, FP8 KV, Vision, MTP3, the optimized head, CUDA Graph decode and up to
eight active requests. Its logical context limit is **1,048,576 tokens**. The coordinator launches
both workers, using SSH for the remote node. Clients connect to its single HTTP endpoint.

Experts use TP by default (`ep = 1`); set `ep = 2` to select whole-expert EP while retaining TP2
and attention DP2 elsewhere. Keep all six artifact files on each node. The complete artifact does
not fit on one Spark. Runtime allocations also include workspace and the shared KV/State pool;
read the startup capacity estimate for the selected configuration.

For CLI use, run `apps/ninfer --config ninfer.local.toml --prompt "..." --max-tokens 2048` on the
coordinator. Detailed startup and network instructions are in the
[deployment guide](https://github.com/Neroued/ninfer/tree/master/examples/configs).

## Model options

- Thinking stays enabled. Supported reasoning efforts are `low`, `high` and `max`; the template
  supplies the default. `--no-thinking` and effort `none` are unsupported.
- Native MTP supports 1..5 drafts. `--proposal-head optimized` selects the Q4 draft head;
  the target head stays BF16. Draft count can be adjusted for workload and concurrency.
- Latent KV uses `--kv-dtype bf16` or `fp8`. The Kpool indexer uses FP8 by default;
  `NINFER_KPOOL_INDEXER_BF16=1` selects BF16 indexer storage independently.
- Vision, tool calls, constrained output, streaming and prefix reuse use the same public Engine
  and protocol endpoints as text generation. Enable Vision at startup for image/video input.
- GLM uses its own chat template and tool framing; retain the included GLM frontend resources.

DFlash and DFlash2 are not used by this architecture. NInfer returns generated tool calls to the
client; it does not execute them.

## Sources and license

The primary conversion source is
[nvidia/GLM-5.3-Flash-NVFP4](https://huggingface.co/nvidia/GLM-5.3-Flash-NVFP4), revision
`da920bb0b9f4a06727223a349e55468e38352348`, based on
[zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash).
The model weights use the upstream [MIT license](https://huggingface.co/zai-org/GLM-5.3-Flash/blob/main/LICENSE).
[NInfer](https://github.com/Neroued/ninfer) is Apache-2.0-licensed software.