SenseNova-Loop-9B — NVFP4 (W4A16)

A 4-bit NVFP4 build of sensenova/SenseNova-Loop-9B (the final release, not the Preview), made for fast serving with vLLM on NVIDIA Blackwell, especially the DGX Spark / GB10.

On a single GB10 it decodes chat answers at 26 tok/s for one stream, and at 37 tok/s with MTP speculative decoding (126 tok/s across four streams). That is 3× the BF16 model in the same engine. It loads in 7.2 GiB (vision tower included) instead of 17.8 GiB, which leaves plenty of unified memory for the KV cache and for other models. Images work too: the vision tower is kept in BF16 and served through the plugin.

🆕 2026-11-19: the LM head is now NVFP4 too. The 248K × 4096 output layer is read in full at every decoded token, so quantizing it made this build 22 % faster (31 % with MTP) and 2 GB smaller, at a small cost in accuracy (mean KLD 0.160 instead of 0.153). The MLP sibling got the same change. The previous version with a BF16 LM head is revision b8e8307d.

💚 Who this is for: you have a Blackwell GPU or a DGX Spark and want SenseNova-Loop to feel quick in vLLM. If you mainly care about accuracy per byte, or you run llama.cpp / a Mac, look at the other builds further down. The community has made some excellent ones.

There is also a quality-first sibling, SenseNova-Loop-9B-NVFP4-MLP, which quantizes only the MLPs: noticeably closer to BF16, and slower. SenseNova's own AWQ release also runs with the plugin; see the comparison below.


Quick facts

Base model sensenova/SenseNova-Loop-9B, revision 3e1ba6e8
Architecture SenseNovaLoopForConditionalGeneration: 32 stored layers, layers 4–27 run twice (56 executions), two-stream mHC (Sinkhorn) around the looped layers, 24 Gated DeltaNet + 8 full-attention layers, YaRN ×2 to 512K, vocab 248,320
Size 7.9 GB on disk (BF16: 19 GB)
Format NVFP4 (E2M1, one E4M3 scale per 16 weights + one FP32 scale per tensor), nvidia-modelopt 0.43 checkpoint, quant_algo: W4A16_NVFP4 (4-bit weights, BF16 activations)
Quantized 200 linears: mlp.{gate,up,down}_proj (32 layers), self_attn.{q,k,v,o}_proj (8 layers), linear_attn.{in_proj_qkv,in_proj_z,out_proj} (24 layers), plus lm_head
Kept as in the source embed_tokens, all norms, the mHC routers (FP32), loop_gate, DeltaNet in_proj_a/b, conv1d, A_log / dt_bias (FP32), vision tower, MTP head (all byte-identical to the source)
Input text and images (vision tower in BF16, +0.85 GiB in memory)
MTP the source's MTP head is included (BF16); use it with --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'
Calibration 64 × 1,024 tokens of mixed text (news, wiki EN/DE, code, chat, role-play), run through the real looped forward: activation ranges were collected over both passes of every looped layer

Who this is for (and who it isn't)

You have … Recommendation
DGX Spark / GB10, or another Blackwell GPU, with vLLM ✅ This build if speed matters most, the MLP sibling if accuracy matters most
llama.cpp (any GPU) The GGUFs from AtomicChat or SenseNova, which need SenseNova's llama.cpp fork
Apple Silicon The MLX builds, e.g. from NetProjects
Pre-Blackwell NVIDIA GPU with vLLM Untested here. vLLM can run NVFP4 weights through Marlin on older GPUs too, but we have only measured GB10

Speed on DGX Spark (GB10)

Single GB10, vLLM 0.31.0 with the vllm-sensenova-loop plugin, BF16 KV cache. Six chat tasks (English and German: code, explanation, story, list, how-to), thinking off, up to 384 new tokens, greedy. Decode tok/s for one stream / total across four concurrent streams, measured with the plugin's tests/bench_mtp.py.

Build Weights in memory No MTP MTP, 1 draft token
BF16 (source) 17.8 GiB 7.5 / 22 12.2 / 31
this build 7.2 GiB 25.9 / 90 36.6 / 126
this build before 2026-11-19 (BF16 LM head) 8.5 GiB 21.3 / 75 27.9 / 99
NVFP4-MLP 10.0 GiB 14.0 / 53 24.6 / 76
SenseNova AWQ (official) 5.5 GiB 27.5 / 77 39.4 / 104

Memory includes the vision tower; MTP adds 0.5 GiB. The MTP head's draft token is accepted 67 % of the time. More draft tokens (2 or 3) are accepted more often in total but are not faster on GB10. With temperature 1.0 / top_p 0.95 / top_k 20 instead of greedy, single-stream speed drops by about 4 %.

For reference, on the same machine SenseNova's llama.cpp fork (blueswhen/llama.cpp@24ea7bb7, CUDA, -fa on) ran the Q4_K_M GGUFs at ~600 tok/s prefill and 4 tok/s decode. The loop architecture is still young in llama.cpp; we expect this to improve.

Quality

Compared with other 4-bit builds

To compare across engines fairly, all builds were scored on the same tokens with the method llama-perplexity --kl-divergence uses: 2,048-token chunks, only the second half of each chunk is scored, and the full 248K-token distribution is compared against SenseNova's BF16 GGUF. llama.cpp builds ran in llama.cpp. Safetensors builds ran in transformers with their NVFP4 weights decoded exactly. 72,633 scored positions on one GB10. As a sanity check, BF16 in transformers vs. BF16 in llama.cpp differs by only 0.003.

Build Size Mean KLD ↓ Same top token ↑
AtomicChat AD-Q4_K_M (GGUF, imatrix) 5.96 GB 0.054 92.6 %
NVFP4-MLP 10.9 GB 0.094 * 88.7 % *
NVFP4-MLP before 2026-11-19 (BF16 LM head) 12.4 GB 0.087 90.0 %
SenseNova Q4_K_M (GGUF, official) 5.82 GB 0.111 89.2 %
balllvin NVFP4 (QAD) 8.9 GB 0.139 87.3 %
this build before 2026-11-19 (BF16 LM head) 9.4 GB 0.151 86.8 %
this build 7.9 GB 0.160 * 85.5 % *
SenseNova AWQ (official) 5.7 GB 0.167 * 85.1 % *

* Measured later on the same tokens but against the BF16 model in transformers instead of the BF16 GGUF (71,610 positions; the GGUF reference files were gone by then). The two references agree closely: on the new one the previous version of this build scores 0.153 / 86.4 % (0.151 / 86.8 % above) and the MLP build 0.086 / 89.8 % (0.087 / 90.0 % above).

Text: news, Wikipedia (EN and DE), code, chat and role-play, 8–16 documents each. A seventh bucket of long-form character role-play is left out of this table: even BF16 in two engines disagrees there (KLD 0.09 between them, BF16 perplexity 54), so it says more about the text than about the quant. With it included the ranking is unchanged (previous version of this build: 0.344, MLP: 0.235, AtomicChat: 0.170).

Our KLD for AtomicChat's and SenseNova's files is close to what AtomicChat publish with their own corpus (0.0445 and 0.101), so the numbers line up across setups.

What this means: this build trades some accuracy for speed. Compared with SenseNova's official AWQ it is slightly more accurate, a bit slower for a single stream and faster with several streams. Quantizing attention and the DeltaNet projections roughly doubles decode speed over the MLP-only variant, at the cost of a measurable KLD increase. SenseNova-Loop is unusually sensitive to quantization, AtomicChat noticed this too: layers 4–27 run twice on the same rounded weights, so every rounding error is applied twice.

Images

Seven image prompts (shapes, OCR, a bar chart with a German question, a photo, a 1280×640 banner, two images in one message, text before the image) were answered by this build in vLLM and checked token by token against the BF16 model in SenseNova's transformers code. The prompt tokens are identical, and the answers are correct for all seven prompts:

Build in vLLM Same top token as BF16 (transformers)
BF16 (source, as a check of the plugin) 94–100 % (misses are exact ties)
this build 93–100 % (all seven answers correct)
NVFP4-MLP 88–100 % (all seven correct)

A small set, so read it as "vision works and survives the quantization", not as a benchmark.

Things we tried that did not make it

Variant (all W4A16) Mean KLD* Decode, 1 stream Verdict
MLP only 0.252 12.4 tok/s → NVFP4-MLP
MLP + DeltaNet in_proj 0.304 16.2 tok/s in between, not released
MLP + attention + DeltaNet out_proj 0.333 15.0 tok/s slower and worse than the row above
everything (this build until 2026-11-19) 0.363 21.2 tok/s ✅, now with an NVFP4 LM head on top
W4A4 (FP4 activations), any scope +0.10 to +0.11 – not worth it on GB10

* Earlier in-process measurement over all positions, including the role-play bucket, so higher than the table above. Use it only to compare rows within this table.


Running it with vLLM

SenseNova-Loop is not in vLLM yet. The upstream PR vllm-project/vllm#64832 by SenseTime was redirected to the plugin route. This build runs with a small out-of-tree plugin, vllm-sensenova-loop, which packages the PR's model code (with attribution) and adds support for ModelOpt NVFP4 checkpoints. It was tested with the official vllm/vllm-openai:v0.31.0 image on GB10.

pip install --no-deps https://github.com/KaletoAI/vllm-sensenova-loop/archive/refs/heads/master.tar.gz   # inside your vLLM 0.31 environment or image

vllm serve Kaleto/SenseNova-Loop-9B-NVFP4 \
  --served-model-name sensenova-loop-9b-nvfp4 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser qwen3_coder \
  --max-model-len 32768 --max-num-seqs 3 \
  --enable-chunked-prefill --max-num-batched-tokens 8192 \
  --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'
  • No --quantization flag needed; vLLM detects the ModelOpt NVFP4 config. On GB10 it picks MarlinNvFp4LinearKernel.
  • Needs plugin commit 2778586 or newer for the NVFP4 LM head.
  • MTP (the last flag) is optional and about +40 % faster. With it, vLLM 0.31 rejects requests that set min_p or logit_bias (HTTP 400); see Sampling.
  • The plugin picks safe defaults for the loop (eager compile, CUDA graphs for decode only), so no extra flags are needed.
  • Thinking is on by default. Switch it off per request with "chat_template_kwargs": {"enable_thinking": false}. Reasoning and tool calls are parsed correctly with the flags above (checked on GB10).
  • On a 128 GB Spark, --kv-cache-memory-bytes 8589934592 (8 GiB) holds ~134K tokens, enough for 4 × 32K requests.

Images: send them as usual with image_url parts in the chat API. The plugin uses vLLM's own Qwen3.5 vision code, as SenseNova's modeling code does. Add --limit-mm-per-prompt '{"image": 4, "video": 0}' if you want to cap images per prompt and switch off video, which is untested.

Current limits of the plugin: video untested, single GPU tested.

Sampling

SenseNova's generation_config.json uses temperature 1.0 / top_p 0.95. With top_p alone, the long tail can produce junk tokens at temperature ≥ 1.0, in BF16 as well. Adding top_k: 20 keeps it clean. min_p: 0.05 works too, but not with MTP enabled.


How it was made

  1. Loaded the BF16 release in transformers 5.18 (the version SenseNova pins) with SenseNova's own modeling code.
  2. Quantized with modelopt's NVFP4_DEFAULT_CFG, restricted to the 200 linears listed above, and calibrated through the looped forward. A hook checks that layers 4–27 are each visited twice.
  3. Exported with modelopt. All tensors that are not quantized are then copied byte-for-byte from the source, because modelopt's export would cast the FP32 DeltaNet A_log/dt_bias to BF16. The MTP head is copied from mtp.safetensors, and SenseNova's config.json is kept verbatim apart from the added quantization_config.
  4. On 2026-11-19 the LM head was quantized as well. Weight-only NVFP4 needs no calibration, so it was converted directly with modelopt's scale rule (weight_scale_2 = amax / (6·448), one E4M3 scale per 16 weights); relative weight error 0.095, the same as the other layers.
  5. Checked: every quantized tensor has finite scales, the ignore list matches the tensors, and greedy answers to text and image prompts come out right in vLLM.

Credits

SenseNova / SenseTime for the model and for the vLLM implementation in PR #64832, which the plugin builds on. NVIDIA for modelopt. AtomicChat, balllvin, NetProjects and everyone else publishing builds and measurements. Comparing against your work made this card more honest.

License

Apache-2.0, inherited from sensenova/SenseNova-Loop-9B. This is a quantized, otherwise unmodified version of the original weights. It is an independent community build, not affiliated with SenseNova.

Downloads last month
88
Safetensors
Model size
10B params
Tensor type
F32
·
BF16
·
U8
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Kaleto/SenseNova-Loop-9B-NVFP4

Quantized
(44)
this model