Card: image input via the plugin, vision check
Browse files
README.md
CHANGED
|
@@ -3,7 +3,7 @@ license: apache-2.0
|
|
| 3 |
base_model: sensenova/SenseNova-Loop-9B
|
| 4 |
base_model_relation: quantized
|
| 5 |
library_name: vllm
|
| 6 |
-
pipeline_tag: text-
|
| 7 |
tags:
|
| 8 |
- nvfp4
|
| 9 |
- fp4
|
|
@@ -16,13 +16,14 @@ tags:
|
|
| 16 |
- sensenova-loop
|
| 17 |
- looped-transformer
|
| 18 |
- mhc
|
|
|
|
| 19 |
---
|
| 20 |
|
| 21 |
# SenseNova-Loop-9B — NVFP4 (W4A16)
|
| 22 |
|
| 23 |
A 4-bit NVFP4 build of [sensenova/SenseNova-Loop-9B](https://huggingface.co/sensenova/SenseNova-Loop-9B) (the final release, not the Preview), made for **fast serving with vLLM on NVIDIA Blackwell, especially the DGX Spark / GB10**.
|
| 24 |
|
| 25 |
-
On a single GB10 it decodes at **21 tok/s** for one stream and **81 tok/s** across four. That is about 2.8× the BF16 model in the same engine. It loads in **7.7 GiB** instead of 17 GiB, which leaves plenty of unified memory for the KV cache and for other models.
|
| 26 |
|
| 27 |
> 💚 **Who this is for:** you have a Blackwell GPU or a DGX Spark and want SenseNova-Loop to feel quick in vLLM. If you mainly care about accuracy per byte, or you run llama.cpp / a Mac, look at the [other builds](#other-builds-worth-a-look) further down. The community has made some excellent ones.
|
| 28 |
|
|
@@ -40,6 +41,7 @@ There is also a **quality-first sibling**, [SenseNova-Loop-9B-NVFP4-MLP](https:/
|
|
| 40 |
| **Format** | NVFP4 (E2M1, one E4M3 scale per 16 weights + one FP32 scale per tensor), [nvidia-modelopt](https://github.com/NVIDIA/TensorRT-Model-Optimizer) 0.43 checkpoint, `quant_algo: W4A16_NVFP4` (4-bit weights, BF16 activations) |
|
| 41 |
| **Quantized** | 200 linears: `mlp.{gate,up,down}_proj` (32 layers), `self_attn.{q,k,v,o}_proj` (8 layers), `linear_attn.{in_proj_qkv,in_proj_z,out_proj}` (24 layers) |
|
| 42 |
| **Kept as in the source** | `lm_head`, `embed_tokens`, all norms, the mHC routers (FP32), `loop_gate`, DeltaNet `in_proj_a/b`, `conv1d`, `A_log` / `dt_bias` (FP32), vision tower, MTP head (all byte-identical to the source) |
|
|
|
|
| 43 |
| **Calibration** | 64 × 1,024 tokens of mixed text (news, wiki EN/DE, code, chat, role-play), run **through the real looped forward**: activation ranges were collected over both passes of every looped layer |
|
| 44 |
|
| 45 |
## Who this is for (and who it isn't)
|
|
@@ -85,6 +87,18 @@ Our KLD for AtomicChat's and SenseNova's files is close to what [AtomicChat publ
|
|
| 85 |
|
| 86 |
**What this means:** this build trades some accuracy for speed. Quantizing attention and the DeltaNet projections roughly doubles decode speed over the MLP-only variant, at the cost of a measurable KLD increase. SenseNova-Loop is unusually sensitive to quantization, AtomicChat noticed this too: layers 4–27 run twice on the same rounded weights, so every rounding error is applied twice.
|
| 87 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 88 |
### Things we tried that did not make it
|
| 89 |
|
| 90 |
| Variant (all W4A16) | Mean KLD* | Decode, 1 stream | Verdict |
|
|
@@ -119,7 +133,9 @@ vllm serve Kaleto/SenseNova-Loop-9B-NVFP4 \
|
|
| 119 |
- Thinking is on by default. Switch it off per request with `"chat_template_kwargs": {"enable_thinking": false}`. Reasoning and tool calls are parsed correctly with the flags above (checked on GB10).
|
| 120 |
- On a 128 GB Spark, `--kv-cache-memory-bytes 8589934592` (8 GiB) holds ~134K tokens, enough for 4 × 32K requests.
|
| 121 |
|
| 122 |
-
**
|
|
|
|
|
|
|
| 123 |
|
| 124 |
### Sampling
|
| 125 |
|
|
@@ -132,7 +148,7 @@ SenseNova's `generation_config.json` uses temperature 1.0 / top_p 0.95. With top
|
|
| 132 |
1. Loaded the BF16 release in transformers 5.18 (the version SenseNova pins) with SenseNova's own modeling code.
|
| 133 |
2. Quantized with modelopt's `NVFP4_DEFAULT_CFG`, restricted to the 200 linears listed above, and calibrated through the looped forward. A hook checks that layers 4–27 are each visited twice.
|
| 134 |
3. Exported with modelopt. All tensors that are not quantized are then copied **byte-for-byte from the source**, because modelopt's export would cast the FP32 DeltaNet `A_log`/`dt_bias` to BF16. The MTP head is copied from `mtp.safetensors`, and SenseNova's `config.json` is kept verbatim apart from the added `quantization_config`.
|
| 135 |
-
4. Checked: every quantized tensor has finite scales, the ignore list matches the tensors, and greedy answers come out right in vLLM.
|
| 136 |
|
| 137 |
## Credits
|
| 138 |
|
|
|
|
| 3 |
base_model: sensenova/SenseNova-Loop-9B
|
| 4 |
base_model_relation: quantized
|
| 5 |
library_name: vllm
|
| 6 |
+
pipeline_tag: image-text-to-text
|
| 7 |
tags:
|
| 8 |
- nvfp4
|
| 9 |
- fp4
|
|
|
|
| 16 |
- sensenova-loop
|
| 17 |
- looped-transformer
|
| 18 |
- mhc
|
| 19 |
+
- vision
|
| 20 |
---
|
| 21 |
|
| 22 |
# SenseNova-Loop-9B — NVFP4 (W4A16)
|
| 23 |
|
| 24 |
A 4-bit NVFP4 build of [sensenova/SenseNova-Loop-9B](https://huggingface.co/sensenova/SenseNova-Loop-9B) (the final release, not the Preview), made for **fast serving with vLLM on NVIDIA Blackwell, especially the DGX Spark / GB10**.
|
| 25 |
|
| 26 |
+
On a single GB10 it decodes at **21 tok/s** for one stream and **81 tok/s** across four. That is about 2.8× the BF16 model in the same engine. It loads in **7.7 GiB** instead of 17 GiB, which leaves plenty of unified memory for the KV cache and for other models. **Images work too**: the vision tower is kept in BF16 and served through the plugin.
|
| 27 |
|
| 28 |
> 💚 **Who this is for:** you have a Blackwell GPU or a DGX Spark and want SenseNova-Loop to feel quick in vLLM. If you mainly care about accuracy per byte, or you run llama.cpp / a Mac, look at the [other builds](#other-builds-worth-a-look) further down. The community has made some excellent ones.
|
| 29 |
|
|
|
|
| 41 |
| **Format** | NVFP4 (E2M1, one E4M3 scale per 16 weights + one FP32 scale per tensor), [nvidia-modelopt](https://github.com/NVIDIA/TensorRT-Model-Optimizer) 0.43 checkpoint, `quant_algo: W4A16_NVFP4` (4-bit weights, BF16 activations) |
|
| 42 |
| **Quantized** | 200 linears: `mlp.{gate,up,down}_proj` (32 layers), `self_attn.{q,k,v,o}_proj` (8 layers), `linear_attn.{in_proj_qkv,in_proj_z,out_proj}` (24 layers) |
|
| 43 |
| **Kept as in the source** | `lm_head`, `embed_tokens`, all norms, the mHC routers (FP32), `loop_gate`, DeltaNet `in_proj_a/b`, `conv1d`, `A_log` / `dt_bias` (FP32), vision tower, MTP head (all byte-identical to the source) |
|
| 44 |
+
| **Input** | text and images (vision tower in BF16, +0.85 GiB in memory) |
|
| 45 |
| **Calibration** | 64 × 1,024 tokens of mixed text (news, wiki EN/DE, code, chat, role-play), run **through the real looped forward**: activation ranges were collected over both passes of every looped layer |
|
| 46 |
|
| 47 |
## Who this is for (and who it isn't)
|
|
|
|
| 87 |
|
| 88 |
**What this means:** this build trades some accuracy for speed. Quantizing attention and the DeltaNet projections roughly doubles decode speed over the MLP-only variant, at the cost of a measurable KLD increase. SenseNova-Loop is unusually sensitive to quantization, AtomicChat noticed this too: layers 4–27 run twice on the same rounded weights, so every rounding error is applied twice.
|
| 89 |
|
| 90 |
+
### Images
|
| 91 |
+
|
| 92 |
+
Seven image prompts (shapes, OCR, a bar chart with a German question, a photo, a 1280×640 banner, two images in one message, text before the image) were answered by this build in vLLM and checked token by token against the BF16 model in SenseNova's transformers code. The prompt tokens are identical, and the answers are correct for all seven prompts:
|
| 93 |
+
|
| 94 |
+
| Build in vLLM | Same top token as BF16 (transformers) |
|
| 95 |
+
|---|---:|
|
| 96 |
+
| BF16 (source, as a check of the plugin) | 94–100 % (misses are exact ties) |
|
| 97 |
+
| **this build** | 93–100 % |
|
| 98 |
+
| [NVFP4-MLP](https://huggingface.co/Kaleto/SenseNova-Loop-9B-NVFP4-MLP) | 88–100 % |
|
| 99 |
+
|
| 100 |
+
A small set, so read it as "vision works and survives the quantization", not as a benchmark.
|
| 101 |
+
|
| 102 |
### Things we tried that did not make it
|
| 103 |
|
| 104 |
| Variant (all W4A16) | Mean KLD* | Decode, 1 stream | Verdict |
|
|
|
|
| 133 |
- Thinking is on by default. Switch it off per request with `"chat_template_kwargs": {"enable_thinking": false}`. Reasoning and tool calls are parsed correctly with the flags above (checked on GB10).
|
| 134 |
- On a 128 GB Spark, `--kv-cache-memory-bytes 8589934592` (8 GiB) holds ~134K tokens, enough for 4 × 32K requests.
|
| 135 |
|
| 136 |
+
**Images:** send them as usual with `image_url` parts in the chat API. The plugin uses vLLM's own Qwen3.5 vision code, as SenseNova's modeling code does. Add `--limit-mm-per-prompt '{"image": 4, "video": 0}'` if you want to cap images per prompt and switch off video, which is untested.
|
| 137 |
+
|
| 138 |
+
**Current limits of the plugin:** no MTP speculative decoding yet, video untested, single GPU tested.
|
| 139 |
|
| 140 |
### Sampling
|
| 141 |
|
|
|
|
| 148 |
1. Loaded the BF16 release in transformers 5.18 (the version SenseNova pins) with SenseNova's own modeling code.
|
| 149 |
2. Quantized with modelopt's `NVFP4_DEFAULT_CFG`, restricted to the 200 linears listed above, and calibrated through the looped forward. A hook checks that layers 4–27 are each visited twice.
|
| 150 |
3. Exported with modelopt. All tensors that are not quantized are then copied **byte-for-byte from the source**, because modelopt's export would cast the FP32 DeltaNet `A_log`/`dt_bias` to BF16. The MTP head is copied from `mtp.safetensors`, and SenseNova's `config.json` is kept verbatim apart from the added `quantization_config`.
|
| 151 |
+
4. Checked: every quantized tensor has finite scales, the ignore list matches the tensors, and greedy answers to text and image prompts come out right in vLLM.
|
| 152 |
|
| 153 |
## Credits
|
| 154 |
|