Kaleto commited on
Commit
b8e8307
·
verified ·
1 Parent(s): 936c913

Card: image input via the plugin, vision check

Browse files
Files changed (1) hide show
  1. README.md +20 -4
README.md CHANGED
@@ -3,7 +3,7 @@ license: apache-2.0
3
  base_model: sensenova/SenseNova-Loop-9B
4
  base_model_relation: quantized
5
  library_name: vllm
6
- pipeline_tag: text-generation
7
  tags:
8
  - nvfp4
9
  - fp4
@@ -16,13 +16,14 @@ tags:
16
  - sensenova-loop
17
  - looped-transformer
18
  - mhc
 
19
  ---
20
 
21
  # SenseNova-Loop-9B — NVFP4 (W4A16)
22
 
23
  A 4-bit NVFP4 build of [sensenova/SenseNova-Loop-9B](https://huggingface.co/sensenova/SenseNova-Loop-9B) (the final release, not the Preview), made for **fast serving with vLLM on NVIDIA Blackwell, especially the DGX Spark / GB10**.
24
 
25
- On a single GB10 it decodes at **21 tok/s** for one stream and **81 tok/s** across four. That is about 2.8× the BF16 model in the same engine. It loads in **7.7 GiB** instead of 17 GiB, which leaves plenty of unified memory for the KV cache and for other models.
26
 
27
  > 💚 **Who this is for:** you have a Blackwell GPU or a DGX Spark and want SenseNova-Loop to feel quick in vLLM. If you mainly care about accuracy per byte, or you run llama.cpp / a Mac, look at the [other builds](#other-builds-worth-a-look) further down. The community has made some excellent ones.
28
 
@@ -40,6 +41,7 @@ There is also a **quality-first sibling**, [SenseNova-Loop-9B-NVFP4-MLP](https:/
40
  | **Format** | NVFP4 (E2M1, one E4M3 scale per 16 weights + one FP32 scale per tensor), [nvidia-modelopt](https://github.com/NVIDIA/TensorRT-Model-Optimizer) 0.43 checkpoint, `quant_algo: W4A16_NVFP4` (4-bit weights, BF16 activations) |
41
  | **Quantized** | 200 linears: `mlp.{gate,up,down}_proj` (32 layers), `self_attn.{q,k,v,o}_proj` (8 layers), `linear_attn.{in_proj_qkv,in_proj_z,out_proj}` (24 layers) |
42
  | **Kept as in the source** | `lm_head`, `embed_tokens`, all norms, the mHC routers (FP32), `loop_gate`, DeltaNet `in_proj_a/b`, `conv1d`, `A_log` / `dt_bias` (FP32), vision tower, MTP head (all byte-identical to the source) |
 
43
  | **Calibration** | 64 × 1,024 tokens of mixed text (news, wiki EN/DE, code, chat, role-play), run **through the real looped forward**: activation ranges were collected over both passes of every looped layer |
44
 
45
  ## Who this is for (and who it isn't)
@@ -85,6 +87,18 @@ Our KLD for AtomicChat's and SenseNova's files is close to what [AtomicChat publ
85
 
86
  **What this means:** this build trades some accuracy for speed. Quantizing attention and the DeltaNet projections roughly doubles decode speed over the MLP-only variant, at the cost of a measurable KLD increase. SenseNova-Loop is unusually sensitive to quantization, AtomicChat noticed this too: layers 4–27 run twice on the same rounded weights, so every rounding error is applied twice.
87
 
 
 
 
 
 
 
 
 
 
 
 
 
88
  ### Things we tried that did not make it
89
 
90
  | Variant (all W4A16) | Mean KLD* | Decode, 1 stream | Verdict |
@@ -119,7 +133,9 @@ vllm serve Kaleto/SenseNova-Loop-9B-NVFP4 \
119
  - Thinking is on by default. Switch it off per request with `"chat_template_kwargs": {"enable_thinking": false}`. Reasoning and tool calls are parsed correctly with the flags above (checked on GB10).
120
  - On a 128 GB Spark, `--kv-cache-memory-bytes 8589934592` (8 GiB) holds ~134K tokens, enough for 4 × 32K requests.
121
 
122
- **Current limits of the plugin:** text only (the vision tower is in the checkpoint but not wired up yet), no MTP speculative decoding yet, single GPU tested.
 
 
123
 
124
  ### Sampling
125
 
@@ -132,7 +148,7 @@ SenseNova's `generation_config.json` uses temperature 1.0 / top_p 0.95. With top
132
  1. Loaded the BF16 release in transformers 5.18 (the version SenseNova pins) with SenseNova's own modeling code.
133
  2. Quantized with modelopt's `NVFP4_DEFAULT_CFG`, restricted to the 200 linears listed above, and calibrated through the looped forward. A hook checks that layers 4–27 are each visited twice.
134
  3. Exported with modelopt. All tensors that are not quantized are then copied **byte-for-byte from the source**, because modelopt's export would cast the FP32 DeltaNet `A_log`/`dt_bias` to BF16. The MTP head is copied from `mtp.safetensors`, and SenseNova's `config.json` is kept verbatim apart from the added `quantization_config`.
135
- 4. Checked: every quantized tensor has finite scales, the ignore list matches the tensors, and greedy answers come out right in vLLM.
136
 
137
  ## Credits
138
 
 
3
  base_model: sensenova/SenseNova-Loop-9B
4
  base_model_relation: quantized
5
  library_name: vllm
6
+ pipeline_tag: image-text-to-text
7
  tags:
8
  - nvfp4
9
  - fp4
 
16
  - sensenova-loop
17
  - looped-transformer
18
  - mhc
19
+ - vision
20
  ---
21
 
22
  # SenseNova-Loop-9B — NVFP4 (W4A16)
23
 
24
  A 4-bit NVFP4 build of [sensenova/SenseNova-Loop-9B](https://huggingface.co/sensenova/SenseNova-Loop-9B) (the final release, not the Preview), made for **fast serving with vLLM on NVIDIA Blackwell, especially the DGX Spark / GB10**.
25
 
26
+ On a single GB10 it decodes at **21 tok/s** for one stream and **81 tok/s** across four. That is about 2.8× the BF16 model in the same engine. It loads in **7.7 GiB** instead of 17 GiB, which leaves plenty of unified memory for the KV cache and for other models. **Images work too**: the vision tower is kept in BF16 and served through the plugin.
27
 
28
  > 💚 **Who this is for:** you have a Blackwell GPU or a DGX Spark and want SenseNova-Loop to feel quick in vLLM. If you mainly care about accuracy per byte, or you run llama.cpp / a Mac, look at the [other builds](#other-builds-worth-a-look) further down. The community has made some excellent ones.
29
 
 
41
  | **Format** | NVFP4 (E2M1, one E4M3 scale per 16 weights + one FP32 scale per tensor), [nvidia-modelopt](https://github.com/NVIDIA/TensorRT-Model-Optimizer) 0.43 checkpoint, `quant_algo: W4A16_NVFP4` (4-bit weights, BF16 activations) |
42
  | **Quantized** | 200 linears: `mlp.{gate,up,down}_proj` (32 layers), `self_attn.{q,k,v,o}_proj` (8 layers), `linear_attn.{in_proj_qkv,in_proj_z,out_proj}` (24 layers) |
43
  | **Kept as in the source** | `lm_head`, `embed_tokens`, all norms, the mHC routers (FP32), `loop_gate`, DeltaNet `in_proj_a/b`, `conv1d`, `A_log` / `dt_bias` (FP32), vision tower, MTP head (all byte-identical to the source) |
44
+ | **Input** | text and images (vision tower in BF16, +0.85 GiB in memory) |
45
  | **Calibration** | 64 × 1,024 tokens of mixed text (news, wiki EN/DE, code, chat, role-play), run **through the real looped forward**: activation ranges were collected over both passes of every looped layer |
46
 
47
  ## Who this is for (and who it isn't)
 
87
 
88
  **What this means:** this build trades some accuracy for speed. Quantizing attention and the DeltaNet projections roughly doubles decode speed over the MLP-only variant, at the cost of a measurable KLD increase. SenseNova-Loop is unusually sensitive to quantization, AtomicChat noticed this too: layers 4–27 run twice on the same rounded weights, so every rounding error is applied twice.
89
 
90
+ ### Images
91
+
92
+ Seven image prompts (shapes, OCR, a bar chart with a German question, a photo, a 1280×640 banner, two images in one message, text before the image) were answered by this build in vLLM and checked token by token against the BF16 model in SenseNova's transformers code. The prompt tokens are identical, and the answers are correct for all seven prompts:
93
+
94
+ | Build in vLLM | Same top token as BF16 (transformers) |
95
+ |---|---:|
96
+ | BF16 (source, as a check of the plugin) | 94–100 % (misses are exact ties) |
97
+ | **this build** | 93–100 % |
98
+ | [NVFP4-MLP](https://huggingface.co/Kaleto/SenseNova-Loop-9B-NVFP4-MLP) | 88–100 % |
99
+
100
+ A small set, so read it as "vision works and survives the quantization", not as a benchmark.
101
+
102
  ### Things we tried that did not make it
103
 
104
  | Variant (all W4A16) | Mean KLD* | Decode, 1 stream | Verdict |
 
133
  - Thinking is on by default. Switch it off per request with `"chat_template_kwargs": {"enable_thinking": false}`. Reasoning and tool calls are parsed correctly with the flags above (checked on GB10).
134
  - On a 128 GB Spark, `--kv-cache-memory-bytes 8589934592` (8 GiB) holds ~134K tokens, enough for 4 × 32K requests.
135
 
136
+ **Images:** send them as usual with `image_url` parts in the chat API. The plugin uses vLLM's own Qwen3.5 vision code, as SenseNova's modeling code does. Add `--limit-mm-per-prompt '{"image": 4, "video": 0}'` if you want to cap images per prompt and switch off video, which is untested.
137
+
138
+ **Current limits of the plugin:** no MTP speculative decoding yet, video untested, single GPU tested.
139
 
140
  ### Sampling
141
 
 
148
  1. Loaded the BF16 release in transformers 5.18 (the version SenseNova pins) with SenseNova's own modeling code.
149
  2. Quantized with modelopt's `NVFP4_DEFAULT_CFG`, restricted to the 200 linears listed above, and calibrated through the looped forward. A hook checks that layers 4–27 are each visited twice.
150
  3. Exported with modelopt. All tensors that are not quantized are then copied **byte-for-byte from the source**, because modelopt's export would cast the FP32 DeltaNet `A_log`/`dt_bias` to BF16. The MTP head is copied from `mtp.safetensors`, and SenseNova's `config.json` is kept verbatim apart from the added `quantization_config`.
151
+ 4. Checked: every quantized tensor has finite scales, the ignore list matches the tensors, and greedy answers to text and image prompts come out right in vLLM.
152
 
153
  ## Credits
154