Instructions to use dndll/Nex-N2.6-NVFP4-MXFP8-CSF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use dndll/Nex-N2.6-NVFP4-MXFP8-CSF with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="dndll/Nex-N2.6-NVFP4-MXFP8-CSF") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("dndll/Nex-N2.6-NVFP4-MXFP8-CSF") model = AutoModelForMultimodalLM.from_pretrained("dndll/Nex-N2.6-NVFP4-MXFP8-CSF", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use dndll/Nex-N2.6-NVFP4-MXFP8-CSF with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "dndll/Nex-N2.6-NVFP4-MXFP8-CSF" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "dndll/Nex-N2.6-NVFP4-MXFP8-CSF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/dndll/Nex-N2.6-NVFP4-MXFP8-CSF
- SGLang
How to use dndll/Nex-N2.6-NVFP4-MXFP8-CSF with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "dndll/Nex-N2.6-NVFP4-MXFP8-CSF" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "dndll/Nex-N2.6-NVFP4-MXFP8-CSF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "dndll/Nex-N2.6-NVFP4-MXFP8-CSF" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "dndll/Nex-N2.6-NVFP4-MXFP8-CSF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use dndll/Nex-N2.6-NVFP4-MXFP8-CSF with Docker Model Runner:
docker model run hf.co/dndll/Nex-N2.6-NVFP4-MXFP8-CSF
Nex-N2.6 NVFP4 / MXFP8 / CSF
A mixed-precision, post-training-quantized export of Nex-AGI's Nex-N2.6, an approximately 321B-parameter multimodal model for agentic, coding, computer-use, and long-horizon tool tasks.
- 178.55 GB of tensor storage (166.28 GiB), in 37 safetensors files.
- NVFP4 routed experts with losslessly compressed E4M3 block scales (CSF) and calibrated static 4-bit activations for prefill.
- MXFP8 attention, shared experts, and vision attention.
- Unofficial derivative: this repository is maintained by
dndll, not Nex-AGI. It is a PTQ export, not the QAD (quantization-aware distillation) weights referenced by the conversion recipe.
Important: precision and runtime
The name describes a mixed-precision checkpoint, not an MXFP8-only export. The decoder routed experts use W4A4 during prefill and W4A16 during decode with the pinned serving configuration below; most other quantized groups have dynamic MXFP8 activations or BF16 activations.
Stock vLLM, Transformers, GGUF/llama.cpp, and generic NVFP4 loaders do not automatically support this checkpoint's compressed scales. Use a runtime that implements the Hugging Face-layout weight_scale_encoding: "csf" contract for glm5_next. Standard safetensors loading is retained, but the quantization implementation must understand the CSF encoding and the Glm5NextForConditionalGeneration architecture.
The verified runtime was Local Inference Lab's karmic-kraken vLLM image, whose lil-serve launcher recognizes the CSF recipes in config.json and resolves the appropriate settings. Use the GLM profile/preset invocation below without --quantization or --load-format; do not force instanttensor, which bypasses the CSF routing and rejects the tensors/ layout. Pinned image digest used for local inference:
ghcr.io/local-inference-lab/vllm@sha256:bc48d33ee167fb7675a59889fe0a7fea539bc6700c53cada89284d594a754b54
Download
Install the Hugging Face CLI, then:
hf download dndll/Nex-N2.6-NVFP4-MXFP8-CSF
Or use Python:
from huggingface_hub import snapshot_download
model = snapshot_download("dndll/Nex-N2.6-NVFP4-MXFP8-CSF")
print(model)
For lil-serve, pass the resulting directory as MODEL and use the GLM-5.3-Flash profile and TP2 deployment preset:
export MODEL=/path/to/Nex-N2.6-NVFP4-MXFP8-CSF
docker run --rm -it \
--gpus '"device=0,1"' \
--network host --ipc host --shm-size 32g \
--ulimit memlock=-1 --ulimit stack=67108864 \
-e MODEL \
-v "$MODEL":"$MODEL":ro \
-v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
-v "$HOME/.cache/lil-nex:/cache" \
--entrypoint /usr/local/bin/lil-serve \
ghcr.io/local-inference-lab/vllm@sha256:bc48d33ee167fb7675a59889be0a7fea539bc6700c53cada89284d594a754b54 \
--profile glm53-flash \
--hardware rtx-pro-6000-pcie \
--preset glm53-tp2 \
-- \
--served-model-name Nex-N2.6-NVFP4-MXFP8-CSF
This example follows the local TP2, Blackwell deployment. It targets two GPUs and is not a universal single-GPU or commodity-vLLM command. KV-cache sizing depends on hardware, context, and serving options. The local deployment explicitly reserved about 432 MB per rank more for this PTQ checkpoint's larger scale-exception set than the reference QAD checkpoint.
Once serving on port 8000:
curl http://localhost:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "Nex-N2.6-NVFP4-MXFP8-CSF",
"messages": [
{"role": "user", "content": "Explain quantization in one paragraph."}
],
"reasoning_effort": "high",
"max_tokens": 1024
}'
Keep the supplied tokenizer, processor, and chat template. The model accepts text and image inputs through a compatible GLM-5.3-Flash serving implementation. The template supports reasoning_effort values low, high, and max.
Quantization layout
| Component | Weights | Activations / scale handling |
|---|---|---|
| Routed experts, decoder layers 3–44 | NVFP4 E2M1, group 16 | Calibrated static 4-bit activations for prefill; BF16 activations for decode. E4M3 block scales CSF-encoded; FP32 weight and input scales |
| MTP routed experts, layer 45 | NVFP4 E2M1, group 16 | BF16 activations (W4A16). Raw E4M3 block scales, FP32 global scale |
| Attention in all 46 layers, including KDA, MLA, and DSA indexer | MXFP8 E4M3, group 32 | Dynamic MXFP8; UE8M0 scale byte |
| Shared experts, layers 3–45 | MXFP8 E4M3, group 32 | Dynamic MXFP8 |
| Vision attention | MXFP8 E4M3, group 32 | Dynamic MXFP8 |
| Vision MLP and merger | NVFP4 E2M1, group 16 | BF16 activations (W4A16), E4M3 block scales, FP32 global scale |
Embeddings, lm_head, routers, norms, convolutions, KDA A_log/dt_bias, dense MLP in layers 0–2 |
BF16 | Preserved |
CSF is lossless compression of the stored quantization scales, not a promise of lossless model-quality preservation. It encodes 36,288 decoder routed-expert scale matrices. In the conversion record, CSF encoded 19.03 GB of raw scales as 9.61 GB fixed-stream data plus 1.07 GB exception data. Weight quantization itself remains lossy.
The quantization_config uses quant_method: "modelopt" and quant_algo: "MIXED_PRECISION". Indexed tensors live under tensors/, including a separate activation-scale file. Both must stay together with the root config.json and model.safetensors.index.json.
Activation calibration
Static scales were captured from routed-expert activations over 198,549 tokens, including 133,759 prompt tokens. The corpus drew on coding, agent/tool, mathematics, reasoning, multilingual, and other synthetic prompts as well as selected interactions from the local deployment.
- Scale per expert:
amax / (6 × 448), with gate/up sharing their input scale and down using the intermediate activation scale. - Two experts observed no calibration tokens; each took its layer's maximum scale instead of zero.
- Final scales come from Nex-N2.6's own activations. A tested floor based on the related GLM-5.3-Flash published scales did not improve the local evaluation and was not used.
- The calibration corpus is not redistributed.
Prompt-token NLL on the calibration corpus, lower is better:
| Checkpoint activation configuration | Prompt NLL |
|---|---|
| Same weights, scale-1.0 prefill (scale-A/B baseline) | 2.6873 |
| Final calibrated scales | 2.6440 |
These figures measure calibration fit, not benchmark accuracy. Completions and small held-out subsets are not used to claim a win: completions came from the older configuration, and a 12,395-token held-out set had observed noise of roughly ±0.015 NLL.
Weight reconstruction checks
Metrics compare dequantized output weights to the dequantized FP8 source before the separate activation calibration stage.
| Category | Aggregate relative L2 | Max unit relative L2 | Min unit cosine |
|---|---|---|---|
| NVFP4 CSF routed experts | 0.07682 | 0.08233 | 0.996605 |
| NVFP4 MTP routed experts | 0.08094 | 0.08300 | 0.996550 |
| NVFP4 vision MLP | 0.08125 | 0.08137 | 0.996684 |
| MXFP8 attention | 0.01322 | 0.02715 | 0.999632 |
| MXFP8 shared experts | 0.02425 | 0.02641 | 0.999652 |
| MXFP8 vision attention | 0.02653 | 0.02660 | 0.999646 |
The producer checked every source tensor was processed once, paired and dequantized source FP8 tensors before requantization, checked CSF round trips, and re-read each output shard for dtype, shape, duplicate names, and non-finite scales. A later CSF header correction preserved shard payloads byte-for-byte.
The quantization-manifest.json describes the original weight-conversion pass. reconstruction.json holds its numerical results. CALIBRATION.md documents the subsequent activation-scale update.
Validation and limitations
- The CSF-aware
lil-servelauncher accepts the local Hugging Face layout and its complete indexed file set. - A text generation smoke request completed successfully on 2026-11-18.
- No standard task benchmark, general multimodal evaluation, or direct base-model-versus-quantized quality comparison has been completed. Do not transfer Nex-AGI's upstream benchmark scores to this export.
- Vision is quantized but no new image-input test was performed as part of publication.
- The evaluation is narrow and its corpus has not been independently audited for contamination or representativeness.
- Expect additional output-error sensitivity to prompts, batch composition, context length, and serving implementation. Numerical reconstruction error and prompt-NLL measurements do not bound downstream behavior.
- This is an unofficial PTQ conversion, not Nex-AGI's official model or an official Local Inference Lab QAD release.
See the upstream model card for intended use, upstream evaluations, and model behavior. The upstream model's limitations apply here along with quantization-induced limitations.
Provenance
| Field | Value |
|---|---|
| Base model | nex-agi/Nex-N2.6 |
| Base revision at conversion | 987472298e7867721576ac49a253a6f5b526013e |
| Source index SHA-256 | 09d8cdb6af4714def10fff9606a1db01ddb6d67a23d2c45555c19e4bff2bb337 |
| Quantization | Post-training, local mixed-precision conversion |
| NVIDIA Model Optimizer | 0.47.0rc0 |
| PyTorch | 2.13.0+cu130 |
| Conversion hardware | NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition |
| CSF codec | byte-window4-fixed-stream-u24-exceptions/1 |
| Published by | dndll |
Reproducibility files
quantization-manifest.json: original weight-conversion provenance and statistics.reconstruction.json: numerical reconstruction metrics.reconstruction-checkpoints.json: conversion checkpoint metadata.CALIBRATION.md: activation-scale method and evaluation caveats.SHA256SUMS: SHA-256 checksums for every distributed file except the checksum file itself.
Local scratch files, rejected calibration alternatives, raw prompts, and session data are excluded. For integrity checks, the Hub's LFS metadata and download checksums are authoritative; SHA256SUMS is an additional convenience.
License and attribution
The upstream repository declares Apache-2.0. Its license text is included in LICENSE. NOTICE describes modifications made for this export.
Nex-N2.6 and the upstream model belong to their respective authors. The conversion incorporates the mixed-precision CSF layout recipe of Local Inference Lab's reference checkpoint, not its QAD weights. No endorsement by Nex-AGI or Local Inference Lab is implied.
- Downloads last month
- 28
Model tree for dndll/Nex-N2.6-NVFP4-MXFP8-CSF
Base model
nex-agi/Nex-N2.6