SenseNova-Loop-9B — NVFP4 (W4A16)
A 4-bit NVFP4 build of sensenova/SenseNova-Loop-9B (the final release, not the Preview), made for fast serving with vLLM on NVIDIA Blackwell, especially the DGX Spark / GB10.
On a single GB10 it decodes chat answers at 26 tok/s for one stream, and at 37 tok/s with MTP speculative decoding (126 tok/s across four streams). That is 3× the BF16 model in the same engine. It loads in 7.2 GiB (vision tower included) instead of 17.8 GiB, which leaves plenty of unified memory for the KV cache and for other models. Images work too: the vision tower is kept in BF16 and served through the plugin.
🆕 2026-11-19: the LM head is now NVFP4 too. The 248K × 4096 output layer is read in full at every decoded token, so quantizing it made this build 22 % faster (31 % with MTP) and 2 GB smaller, at a small cost in accuracy (mean KLD 0.160 instead of 0.153). The MLP sibling got the same change. The previous version with a BF16 LM head is revision
b8e8307d.
💚 Who this is for: you have a Blackwell GPU or a DGX Spark and want SenseNova-Loop to feel quick in vLLM. If you mainly care about accuracy per byte, or you run llama.cpp / a Mac, look at the other builds further down. The community has made some excellent ones.
There is also a quality-first sibling, SenseNova-Loop-9B-NVFP4-MLP, which quantizes only the MLPs: noticeably closer to BF16, and slower. SenseNova's own AWQ release also runs with the plugin; see the comparison below.
Quick facts
| Base model | sensenova/SenseNova-Loop-9B, revision 3e1ba6e8 |
| Architecture | SenseNovaLoopForConditionalGeneration: 32 stored layers, layers 4–27 run twice (56 executions), two-stream mHC (Sinkhorn) around the looped layers, 24 Gated DeltaNet + 8 full-attention layers, YaRN ×2 to 512K, vocab 248,320 |
| Size | 7.9 GB on disk (BF16: 19 GB) |
| Format | NVFP4 (E2M1, one E4M3 scale per 16 weights + one FP32 scale per tensor), nvidia-modelopt 0.43 checkpoint, quant_algo: W4A16_NVFP4 (4-bit weights, BF16 activations) |
| Quantized | 200 linears: mlp.{gate,up,down}_proj (32 layers), self_attn.{q,k,v,o}_proj (8 layers), linear_attn.{in_proj_qkv,in_proj_z,out_proj} (24 layers), plus lm_head |
| Kept as in the source | embed_tokens, all norms, the mHC routers (FP32), loop_gate, DeltaNet in_proj_a/b, conv1d, A_log / dt_bias (FP32), vision tower, MTP head (all byte-identical to the source) |
| Input | text and images (vision tower in BF16, +0.85 GiB in memory) |
| MTP | the source's MTP head is included (BF16); use it with --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' |
| Calibration | 64 × 1,024 tokens of mixed text (news, wiki EN/DE, code, chat, role-play), run through the real looped forward: activation ranges were collected over both passes of every looped layer |
Who this is for (and who it isn't)
| You have … | Recommendation |
|---|---|
| DGX Spark / GB10, or another Blackwell GPU, with vLLM | ✅ This build if speed matters most, the MLP sibling if accuracy matters most |
| llama.cpp (any GPU) | The GGUFs from AtomicChat or SenseNova, which need SenseNova's llama.cpp fork |
| Apple Silicon | The MLX builds, e.g. from NetProjects |
| Pre-Blackwell NVIDIA GPU with vLLM | Untested here. vLLM can run NVFP4 weights through Marlin on older GPUs too, but we have only measured GB10 |
Speed on DGX Spark (GB10)
Single GB10, vLLM 0.31.0 with the vllm-sensenova-loop plugin, BF16 KV cache. Six chat tasks (English and German: code, explanation, story, list, how-to), thinking off, up to 384 new tokens, greedy. Decode tok/s for one stream / total across four concurrent streams, measured with the plugin's tests/bench_mtp.py.
| Build | Weights in memory | No MTP | MTP, 1 draft token |
|---|---|---|---|
| BF16 (source) | 17.8 GiB | 7.5 / 22 | 12.2 / 31 |
| this build | 7.2 GiB | 25.9 / 90 | 36.6 / 126 |
| this build before 2026-11-19 (BF16 LM head) | 8.5 GiB | 21.3 / 75 | 27.9 / 99 |
| NVFP4-MLP | 10.0 GiB | 14.0 / 53 | 24.6 / 76 |
| SenseNova AWQ (official) | 5.5 GiB | 27.5 / 77 | 39.4 / 104 |
Memory includes the vision tower; MTP adds 0.5 GiB. The MTP head's draft token is accepted 67 % of the time. More draft tokens (2 or 3) are accepted more often in total but are not faster on GB10. With temperature 1.0 / top_p 0.95 / top_k 20 instead of greedy, single-stream speed drops by about 4 %.
For reference, on the same machine SenseNova's llama.cpp fork (blueswhen/llama.cpp@24ea7bb7, CUDA, -fa on) ran the Q4_K_M GGUFs at ~600 tok/s prefill and 4 tok/s decode. The loop architecture is still young in llama.cpp; we expect this to improve.
Quality
Compared with other 4-bit builds
To compare across engines fairly, all builds were scored on the same tokens with the method llama-perplexity --kl-divergence uses: 2,048-token chunks, only the second half of each chunk is scored, and the full 248K-token distribution is compared against SenseNova's BF16 GGUF. llama.cpp builds ran in llama.cpp. Safetensors builds ran in transformers with their NVFP4 weights decoded exactly. 72,633 scored positions on one GB10. As a sanity check, BF16 in transformers vs. BF16 in llama.cpp differs by only 0.003.
| Build | Size | Mean KLD ↓ | Same top token ↑ |
|---|---|---|---|
| AtomicChat AD-Q4_K_M (GGUF, imatrix) | 5.96 GB | 0.054 | 92.6 % |
| NVFP4-MLP | 10.9 GB | 0.094 * | 88.7 % * |
| NVFP4-MLP before 2026-11-19 (BF16 LM head) | 12.4 GB | 0.087 | 90.0 % |
| SenseNova Q4_K_M (GGUF, official) | 5.82 GB | 0.111 | 89.2 % |
| balllvin NVFP4 (QAD) | 8.9 GB | 0.139 | 87.3 % |
| this build before 2026-11-19 (BF16 LM head) | 9.4 GB | 0.151 | 86.8 % |
| this build | 7.9 GB | 0.160 * | 85.5 % * |
| SenseNova AWQ (official) | 5.7 GB | 0.167 * | 85.1 % * |
* Measured later on the same tokens but against the BF16 model in transformers instead of the BF16 GGUF (71,610 positions; the GGUF reference files were gone by then). The two references agree closely: on the new one the previous version of this build scores 0.153 / 86.4 % (0.151 / 86.8 % above) and the MLP build 0.086 / 89.8 % (0.087 / 90.0 % above).
Text: news, Wikipedia (EN and DE), code, chat and role-play, 8–16 documents each. A seventh bucket of long-form character role-play is left out of this table: even BF16 in two engines disagrees there (KLD 0.09 between them, BF16 perplexity 54), so it says more about the text than about the quant. With it included the ranking is unchanged (previous version of this build: 0.344, MLP: 0.235, AtomicChat: 0.170).
Our KLD for AtomicChat's and SenseNova's files is close to what AtomicChat publish with their own corpus (0.0445 and 0.101), so the numbers line up across setups.
What this means: this build trades some accuracy for speed. Compared with SenseNova's official AWQ it is slightly more accurate, a bit slower for a single stream and faster with several streams. Quantizing attention and the DeltaNet projections roughly doubles decode speed over the MLP-only variant, at the cost of a measurable KLD increase. SenseNova-Loop is unusually sensitive to quantization, AtomicChat noticed this too: layers 4–27 run twice on the same rounded weights, so every rounding error is applied twice.
Images
Seven image prompts (shapes, OCR, a bar chart with a German question, a photo, a 1280×640 banner, two images in one message, text before the image) were answered by this build in vLLM and checked token by token against the BF16 model in SenseNova's transformers code. The prompt tokens are identical, and the answers are correct for all seven prompts:
| Build in vLLM | Same top token as BF16 (transformers) |
|---|---|
| BF16 (source, as a check of the plugin) | 94–100 % (misses are exact ties) |
| this build | 93–100 % (all seven answers correct) |
| NVFP4-MLP | 88–100 % (all seven correct) |
A small set, so read it as "vision works and survives the quantization", not as a benchmark.
Things we tried that did not make it
| Variant (all W4A16) | Mean KLD* | Decode, 1 stream | Verdict |
|---|---|---|---|
| MLP only | 0.252 | 12.4 tok/s | → NVFP4-MLP |
MLP + DeltaNet in_proj |
0.304 | 16.2 tok/s | in between, not released |
MLP + attention + DeltaNet out_proj |
0.333 | 15.0 tok/s | slower and worse than the row above |
| everything (this build until 2026-11-19) | 0.363 | 21.2 tok/s | ✅, now with an NVFP4 LM head on top |
| W4A4 (FP4 activations), any scope | +0.10 to +0.11 | – | not worth it on GB10 |
* Earlier in-process measurement over all positions, including the role-play bucket, so higher than the table above. Use it only to compare rows within this table.
Running it with vLLM
SenseNova-Loop is not in vLLM yet. The upstream PR vllm-project/vllm#64832 by SenseTime was redirected to the plugin route. This build runs with a small out-of-tree plugin, vllm-sensenova-loop, which packages the PR's model code (with attribution) and adds support for ModelOpt NVFP4 checkpoints. It was tested with the official vllm/vllm-openai:v0.31.0 image on GB10.
pip install --no-deps https://github.com/KaletoAI/vllm-sensenova-loop/archive/refs/heads/master.tar.gz # inside your vLLM 0.31 environment or image
vllm serve Kaleto/SenseNova-Loop-9B-NVFP4 \
--served-model-name sensenova-loop-9b-nvfp4 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice --tool-call-parser qwen3_coder \
--max-model-len 32768 --max-num-seqs 3 \
--enable-chunked-prefill --max-num-batched-tokens 8192 \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'
- No
--quantizationflag needed; vLLM detects the ModelOpt NVFP4 config. On GB10 it picksMarlinNvFp4LinearKernel. - Needs plugin commit
2778586or newer for the NVFP4 LM head. - MTP (the last flag) is optional and about +40 % faster. With it, vLLM 0.31 rejects requests that set
min_porlogit_bias(HTTP 400); see Sampling. - The plugin picks safe defaults for the loop (eager compile, CUDA graphs for decode only), so no extra flags are needed.
- Thinking is on by default. Switch it off per request with
"chat_template_kwargs": {"enable_thinking": false}. Reasoning and tool calls are parsed correctly with the flags above (checked on GB10). - On a 128 GB Spark,
--kv-cache-memory-bytes 8589934592(8 GiB) holds ~134K tokens, enough for 4 × 32K requests.
Images: send them as usual with image_url parts in the chat API. The plugin uses vLLM's own Qwen3.5 vision code, as SenseNova's modeling code does. Add --limit-mm-per-prompt '{"image": 4, "video": 0}' if you want to cap images per prompt and switch off video, which is untested.
Current limits of the plugin: video untested, single GPU tested.
Sampling
SenseNova's generation_config.json uses temperature 1.0 / top_p 0.95. With top_p alone, the long tail can produce junk tokens at temperature ≥ 1.0, in BF16 as well. Adding top_k: 20 keeps it clean. min_p: 0.05 works too, but not with MTP enabled.
How it was made
- Loaded the BF16 release in transformers 5.18 (the version SenseNova pins) with SenseNova's own modeling code.
- Quantized with modelopt's
NVFP4_DEFAULT_CFG, restricted to the 200 linears listed above, and calibrated through the looped forward. A hook checks that layers 4–27 are each visited twice. - Exported with modelopt. All tensors that are not quantized are then copied byte-for-byte from the source, because modelopt's export would cast the FP32 DeltaNet
A_log/dt_biasto BF16. The MTP head is copied frommtp.safetensors, and SenseNova'sconfig.jsonis kept verbatim apart from the addedquantization_config. - On 2026-11-19 the LM head was quantized as well. Weight-only NVFP4 needs no calibration, so it was converted directly with modelopt's scale rule (
weight_scale_2 = amax / (6·448), one E4M3 scale per 16 weights); relative weight error 0.095, the same as the other layers. - Checked: every quantized tensor has finite scales, the ignore list matches the tensors, and greedy answers to text and image prompts come out right in vLLM.
Credits
SenseNova / SenseTime for the model and for the vLLM implementation in PR #64832, which the plugin builds on. NVIDIA for modelopt. AtomicChat, balllvin, NetProjects and everyone else publishing builds and measurements. Comparing against your work made this card more honest.
License
Apache-2.0, inherited from sensenova/SenseNova-Loop-9B. This is a quantized, otherwise unmodified version of the original weights. It is an independent community build, not affiliated with SenseNova.
- Downloads last month
- 88
Model tree for Kaleto/SenseNova-Loop-9B-NVFP4
Base model
sensenova/SenseNova-Loop-9B