Paris99 commited on
Commit
4027556
·
verified ·
1 Parent(s): 012fdec

SenseNova-Loop-9B MLX 4bit: image input (vision tower), MTP speculative-decoding head, fp32 norm weights, updated model code

Browse files
NOTICE CHANGED
@@ -4,8 +4,16 @@ This repository contains a modified version of SenseNova-Loop-9B by SenseNova
4
  (https://huggingface.co/sensenova/SenseNova-Loop-9B, revision 329a65f701ba8c48b73c30d0cbaa6f0449733673),
5
  licensed under the Apache License, Version 2.0 (see LICENSE).
6
 
7
- Modifications (17 November 2026): the weights were converted to Apple MLX format (quantized to 4-bit); the vision
8
- encoder and the unused loop_gate tensor are not included. sensenova_loop.py is a new MLX implementation of the
9
  SenseNova-Loop architecture written by NetProjects, also under Apache-2.0.
10
 
 
 
 
 
 
 
 
 
11
  Not affiliated with SenseTime / SenseNova.
 
4
  (https://huggingface.co/sensenova/SenseNova-Loop-9B, revision 329a65f701ba8c48b73c30d0cbaa6f0449733673),
5
  licensed under the Apache License, Version 2.0 (see LICENSE).
6
 
7
+ Modifications (17 November 2026): the weights were converted to Apple MLX format (quantized to 4-bit);
8
+ the unused loop_gate tensor is not included. sensenova_loop.py is a new MLX implementation of the
9
  SenseNova-Loop architecture written by NetProjects, also under Apache-2.0.
10
 
11
+ Update (18 November 2026): this repository also contains, unchanged and in bfloat16, the vision tower weights of
12
+ SenseNova-Loop-9B (model.visual.*, revision 329a65f701ba8c48b73c30d0cbaa6f0449733673) as vision.safetensors, and the
13
+ MTP head weights (mtp.safetensors) from https://huggingface.co/sensenova/SenseNova-Loop-9B at revision
14
+ 17ea94936a0a2157bc102b6ea516beadfa44bb6d, both by SenseNova under the Apache License, Version 2.0, plus the 81 RMSNorm
15
+ weights of the language model stored as float32 (1 + w) in model.fp32norms.safetensors. sensenova_loop.py now also
16
+ implements image input (mRoPE positions); the vision and MTP runtime is at
17
+ https://github.com/internet-projects/sensenova-loop-mlx (v0.3.0), Apache-2.0.
18
+
19
  Not affiliated with SenseTime / SenseNova.
README.md CHANGED
@@ -8,7 +8,7 @@ tags: [mlx, sensenova, sensenova-loop, looped-transformer, mhc, qwen3_5, apple-s
8
  ---
9
  # SenseNova-Loop-9B — MLX 4-bit
10
 
11
- A 4-bit MLX conversion (affine, group size 64) of [sensenova/SenseNova-Loop-9B](https://huggingface.co/sensenova/SenseNova-Loop-9B) (revision `329a65f`, released 17 November 2026) for Apple Silicon. 5.1 GB of weights. The smallest build. Text model with SenseNova's 512K-token YaRN context setting.
12
 
13
  Published by [NetProjects](https://netprojects.com) (Internet Projects Ltd). Not affiliated with SenseTime / SenseNova.
14
 
@@ -20,22 +20,22 @@ SenseNova-Loop is a Qwen3.5-based model with two additions, and this build runs
20
  - **Two-stream mHC** on layers 4–27: the hidden state is split into two residual streams that are routed by small float32 routers (pre / post gates and a 2×2 Sinkhorn mixing matrix, 20 iterations). The 96 router tensors stay in float32 and are never quantized.
21
  - **Long context:** YaRN rope scaling (factor 2 over 262,144 positions), as in SenseNova's config.
22
 
23
- The model code is `sensenova_loop.py`, included in this repo and loaded by mlx-lm through `trust_remote_code=True`.
24
 
25
- ## Results (MacBook Pro M5 Max, 128 GB, 17 November 2026)
26
 
27
  Agreement with SenseNova's own PyTorch code (Transformers 5.18, bf16) on 16 prompts, every position:
28
 
29
- | Build | Size | Same top token as SenseNova's code | KL at the last token | Greedy text identical (16 prompts) |
30
- |---|---|---|---|---|
31
- | [6-bit](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-6bit) (recommended) | 7.3 GB | 96.7% | 0.0056 | 2 / 16 |
32
- | [8-bit](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-8bit) | 9.5 GB | 97.6% | 0.0012 | 12 / 16 |
33
- | **[4-bit](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-4bit)** (this repo) | 5.1 GB | 84.0% | 0.0982 | 2 / 16 |
34
- | [bf16](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-bf16) | 17.9 GB | 98.6% | 0.0012 | 12 / 16 |
35
 
36
  At 4 bits this build follows the original less closely than the larger ones. For the closest match on a small budget use the [6-bit build](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-6bit).
37
 
38
- **Checks on this port:** 6-bit, 8-bit, 4-bit, bf16 builds answered short test prompts correctly with thinking on and off, and the 6-bit build found a pass code hidden in a 64K-token prompt.
39
 
40
  ## Benchmarks (as reported by SenseNova)
41
 
@@ -75,11 +75,53 @@ print(generate(model, tokenizer, prompt=prompt, max_tokens=256))
75
 
76
  Set `enable_thinking=True` for thinking mode. Server: `mlx_lm.server --model netprojects/SenseNova-Loop-9B-MLX-4bit --trust-remote-code`.
77
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
78
  **Other formats:** [MLX 6-bit](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-6bit) (7.3 GB) · [MLX 8-bit](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-8bit) (9.5 GB) · [MLX bf16](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-bf16) (17.9 GB)
79
 
80
  ## License
81
 
82
- SenseNova-Loop-9B is released by SenseNova under the **Apache License 2.0** (`LICENSE` included). This repository contains converted and quantized weights of that model (modified from the original: converted to MLX format and quantized to 4-bit) and an MLX implementation of its architecture, distributed under the same license. See `NOTICE`.
83
 
84
  Credits: SenseNova / SenseTime (SenseNova-Loop-9B), Qwen team (Qwen3.5), Apple MLX and mlx-lm. Conversion and MLX port by Paris Michalas, Partner at [NetProjects](https://netprojects.com). Not affiliated with SenseTime / SenseNova.
85
 
 
8
  ---
9
  # SenseNova-Loop-9B — MLX 4-bit
10
 
11
+ A 4-bit MLX conversion (affine, group size 64) of [sensenova/SenseNova-Loop-9B](https://huggingface.co/sensenova/SenseNova-Loop-9B) (revision `329a65f`, released 17 November 2026) for Apple Silicon. 5.1 GB of weights. The smallest build. Text and image input, with SenseNova's 512K-token YaRN context setting, and SenseNova's MTP head for about 1.7× faster decoding.
12
 
13
  Published by [NetProjects](https://netprojects.com) (Internet Projects Ltd). Not affiliated with SenseTime / SenseNova.
14
 
 
20
  - **Two-stream mHC** on layers 4–27: the hidden state is split into two residual streams that are routed by small float32 routers (pre / post gates and a 2×2 Sinkhorn mixing matrix, 20 iterations). The 96 router tensors stay in float32 and are never quantized.
21
  - **Long context:** YaRN rope scaling (factor 2 over 262,144 positions), as in SenseNova's config.
22
 
23
+ The model code is `sensenova_loop.py`, included in this repo and loaded by mlx-lm through `trust_remote_code=True`. The 81 RMSNorm weights are also shipped in float32 (`model.fp32norms.safetensors`, loaded automatically), so the norms are computed exactly as in SenseNova's code.
24
 
25
+ ## Results (MacBook Pro M5 Max, 128 GB, 17–18 November 2026)
26
 
27
  Agreement with SenseNova's own PyTorch code (Transformers 5.18, bf16) on 16 prompts, every position:
28
 
29
+ | Build | Size | Same top token as SenseNova's code | KL at the last token | Greedy text identical (16 prompts) | Image prompts: same top token as bf16 |
30
+ |---|---|---|---|---|---|
31
+ | [6-bit](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-6bit) (recommended) | 7.3 GB | 96.7% | 0.0056 | 4 / 16 | 98.0% |
32
+ | [8-bit](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-8bit) | 9.5 GB | 98.6% | 0.0015 | 11 / 16 | 96.7% |
33
+ | **[4-bit](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-4bit)** (this repo) | 5.1 GB | 84.4% | 0.1038 | 2 / 16 | 95.4% |
34
+ | [bf16](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-bf16) | 17.9 GB | 99.1% | 0.0005 | 12 / 16 | reference |
35
 
36
  At 4 bits this build follows the original less closely than the larger ones. For the closest match on a small budget use the [6-bit build](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-6bit).
37
 
38
+ **Checks on this port:** all four builds answered short test prompts correctly with thinking on and off, and with images; the 6-bit build found a pass code hidden in a 64K-token prompt. Image input matches SenseNova's code in float32 (same top token at every checked position, image positions exact). With the MTP head, greedy output matches plain decoding except where the two best tokens are tied within 0.25 logits.
39
 
40
  ## Benchmarks (as reported by SenseNova)
41
 
 
75
 
76
  Set `enable_thinking=True` for thinking mode. Server: `mlx_lm.server --model netprojects/SenseNova-Loop-9B-MLX-4bit --trust-remote-code`.
77
 
78
+ `mlx_lm.load` and `mlx_lm.server` use the text model. Image input and MTP decoding use the small runtime in
79
+ [sensenova-loop-mlx v0.3.0](https://github.com/internet-projects/sensenova-loop-mlx/releases/tag/v0.3.0):
80
+
81
+ ```bash
82
+ hf download netprojects/SenseNova-Loop-9B-MLX-4bit --local-dir sn-loop
83
+ git clone --branch v0.3.0 https://github.com/internet-projects/sensenova-loop-mlx && cd sensenova-loop-mlx
84
+ pip install -U "mlx-lm>=0.32" pillow
85
+ ```
86
+
87
+ ## Image input
88
+
89
+ ```bash
90
+ python scripts/generate_vision.py --model ../sn-loop --image photo.jpg --prompt "Describe this picture." --no-think
91
+ ```
92
+
93
+ Repeat `--image` for several images. Each image is capped at about 1 megapixel by default (`--max-pixels 0` uses
94
+ SenseNova's own limit of 16.7 MP, which needs much more memory). The vision tower (`vision.safetensors`) stays in bfloat16.
95
+
96
+ ## Faster decoding with MTP
97
+
98
+ SenseNova's multi-token-prediction head (`mtp.safetensors`) drafts the next tokens and the model checks them in one pass.
99
+
100
+ ```python
101
+ from mlx_lm import load
102
+ from sensenova_mlx.mtp import generate_mtp, load_mtp_head
103
+
104
+ model, tokenizer = load("../sn-loop", trust_remote_code=True)
105
+ head = load_mtp_head(model, "../sn-loop/mtp.safetensors")
106
+ messages = [{"role": "user", "content": "Explain in two sentences why the sky is blue."}]
107
+ ids = tokenizer.encode(tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False, enable_thinking=False))
108
+ print(tokenizer.decode(list(generate_mtp(model, head, ids, max_tokens=256, k=3))))
109
+ ```
110
+
111
+ Decode speed of the 6-bit build on a MacBook Pro M5 Max (greedy, 256 new tokens, median of 3 prompts):
112
+
113
+ | | Plain decoding | MTP k=1 | MTP k=2 | MTP k=3 |
114
+ |---|---|---|---|---|
115
+ | Tokens per second | 19.3 | 27.0 | 30.6 | **33.5** |
116
+ | Speed-up | 1.00× | 1.40× | 1.59× | **1.74×** |
117
+
118
+ `temperature > 0` keeps the model's sampling distribution (draft tokens are accepted by probability ratio).
119
+
120
  **Other formats:** [MLX 6-bit](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-6bit) (7.3 GB) · [MLX 8-bit](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-8bit) (9.5 GB) · [MLX bf16](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-bf16) (17.9 GB)
121
 
122
  ## License
123
 
124
+ SenseNova-Loop-9B is released by SenseNova under the **Apache License 2.0** (`LICENSE` included). This repository contains converted and quantized weights of that model (modified from the original: converted to MLX format and quantized to 4-bit), SenseNova's vision tower and MTP head weights (unchanged, bfloat16) and an MLX implementation of its architecture, distributed under the same license. See `NOTICE`.
125
 
126
  Credits: SenseNova / SenseTime (SenseNova-Loop-9B), Qwen team (Qwen3.5), Apple MLX and mlx-lm. Conversion and MLX port by Paris Michalas, Partner at [NetProjects](https://netprojects.com). Not affiliated with SenseTime / SenseNova.
127
 
__pycache__/sensenova_loop.cpython-315.pyc ADDED
Binary file (45.2 kB). View file
 
config.json CHANGED
@@ -131,7 +131,31 @@
131
  "sensenova_mlx": {
132
  "source": "sensenova/SenseNova-Loop-9B",
133
  "revision": "329a65f701ba8c48b73c30d0cbaa6f0449733673",
134
- "text_only": true,
135
- "hi_bits_for": null
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
136
  }
137
  }
 
131
  "sensenova_mlx": {
132
  "source": "sensenova/SenseNova-Loop-9B",
133
  "revision": "329a65f701ba8c48b73c30d0cbaa6f0449733673",
134
+ "text_only": false,
135
+ "hi_bits_for": null,
136
+ "vision_file": "vision.safetensors",
137
+ "vision_dtype": "bfloat16",
138
+ "default_max_pixels": 1048576,
139
+ "mtp_file": "mtp.safetensors",
140
+ "mtp_source": "sensenova/SenseNova-Loop-9B",
141
+ "mtp_revision": "17ea94936a0a2157bc102b6ea516beadfa44bb6d",
142
+ "fp32_norms_file": "model.fp32norms.safetensors"
143
+ },
144
+ "vision_config": {
145
+ "deepstack_visual_indexes": [],
146
+ "depth": 27,
147
+ "dtype": "bfloat16",
148
+ "hidden_act": "gelu_pytorch_tanh",
149
+ "hidden_size": 1152,
150
+ "in_channels": 3,
151
+ "initializer_range": 0.02,
152
+ "intermediate_size": 4304,
153
+ "model_type": "qwen3_5_vision",
154
+ "num_heads": 16,
155
+ "num_position_embeddings": 2304,
156
+ "out_hidden_size": 4096,
157
+ "patch_size": 16,
158
+ "spatial_merge_size": 2,
159
+ "temporal_patch_size": 2
160
  }
161
  }
model.fp32norms.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c6a6e1103096a65e9f9b5a18e3b9a387b469319a7ea973bbd6e4fc2a95fedde7
3
+ size 1091976
mtp.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3b84d5143ad5c97cbeffb4ffe9d2db9f620e3cbbfa334f987a04a3cb902d08a6
3
+ size 486582880
preprocessor_config.json ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "do_convert_rgb": true,
3
+ "do_normalize": true,
4
+ "do_rescale": true,
5
+ "do_resize": true,
6
+ "image_mean": [
7
+ 0.5,
8
+ 0.5,
9
+ 0.5
10
+ ],
11
+ "image_processor_type": "Qwen2VLImageProcessor",
12
+ "image_std": [
13
+ 0.5,
14
+ 0.5,
15
+ 0.5
16
+ ],
17
+ "merge_size": 2,
18
+ "patch_size": 16,
19
+ "resample": 3,
20
+ "rescale_factor": 0.00392156862745098,
21
+ "size": {
22
+ "longest_edge": 16777216,
23
+ "shortest_edge": 65536
24
+ },
25
+ "temporal_patch_size": 2
26
+ }
processor_config.json ADDED
@@ -0,0 +1,60 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "image_processor": {
3
+ "do_convert_rgb": true,
4
+ "do_normalize": true,
5
+ "do_rescale": true,
6
+ "do_resize": true,
7
+ "image_mean": [
8
+ 0.5,
9
+ 0.5,
10
+ 0.5
11
+ ],
12
+ "image_processor_type": "Qwen2VLImageProcessor",
13
+ "image_std": [
14
+ 0.5,
15
+ 0.5,
16
+ 0.5
17
+ ],
18
+ "merge_size": 2,
19
+ "patch_size": 16,
20
+ "resample": 3,
21
+ "rescale_factor": 0.00392156862745098,
22
+ "size": {
23
+ "longest_edge": 16777216,
24
+ "shortest_edge": 65536
25
+ },
26
+ "temporal_patch_size": 2
27
+ },
28
+ "processor_class": "Qwen3VLProcessor",
29
+ "video_processor": {
30
+ "do_convert_rgb": true,
31
+ "do_normalize": true,
32
+ "do_rescale": true,
33
+ "do_resize": true,
34
+ "do_sample_frames": true,
35
+ "fps": 2,
36
+ "image_mean": [
37
+ 0.5,
38
+ 0.5,
39
+ 0.5
40
+ ],
41
+ "image_std": [
42
+ 0.5,
43
+ 0.5,
44
+ 0.5
45
+ ],
46
+ "max_frames": 768,
47
+ "merge_size": 2,
48
+ "min_frames": 4,
49
+ "patch_size": 16,
50
+ "resample": 3,
51
+ "rescale_factor": 0.00392156862745098,
52
+ "return_metadata": false,
53
+ "size": {
54
+ "longest_edge": 25165824,
55
+ "shortest_edge": 4096
56
+ },
57
+ "temporal_patch_size": 2,
58
+ "video_processor_type": "Qwen3VLVideoProcessor"
59
+ }
60
+ }
sensenova_loop.py CHANGED
@@ -23,13 +23,15 @@ import mlx.nn as nn
23
 
24
  try:
25
  from . import qwen3_5 as q35
26
- from .base import BaseModelArgs, create_attention_mask, create_ssm_mask
27
  from .cache import ArraysCache, KVCache
28
  except ImportError: # loaded as a model_file
29
  from mlx_lm.models import qwen3_5 as q35
30
- from mlx_lm.models.base import BaseModelArgs, create_attention_mask, create_ssm_mask
31
  from mlx_lm.models.cache import ArraysCache, KVCache
32
 
 
 
33
 
34
  @dataclass
35
  class ModelArgs(BaseModelArgs):
@@ -37,6 +39,11 @@ class ModelArgs(BaseModelArgs):
37
  text_config: dict
38
  loop_settings: dict = field(default_factory=dict)
39
  mhc: dict = field(default_factory=dict)
 
 
 
 
 
40
 
41
 
42
  def execution_plan(num_layers: int, loop: dict, mhc: dict) -> Tuple[int, ...]:
@@ -170,6 +177,144 @@ class MHCDecoderLayer(q35.DecoderLayer):
170
  return out.mean(axis=-2) if self.layer_idx == self.mhc_end - 1 else out
171
 
172
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
173
  class SenseNovaLoopTextModel(nn.Module):
174
  def __init__(self, args: q35.TextModelArgs, loop: dict, mhc: dict):
175
  super().__init__()
@@ -184,6 +329,37 @@ class SenseNovaLoopTextModel(nn.Module):
184
  self.plan = execution_plan(args.num_hidden_layers, loop, mhc)
185
  self.fa_visit = next((e for e, p in enumerate(self.plan) if not self.layers[p].is_linear), None)
186
  self.ssm_visit = next((e for e, p in enumerate(self.plan) if self.layers[p].is_linear), None)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
187
 
188
  def __call__(self, inputs: mx.array, cache=None, input_embeddings: Optional[mx.array] = None) -> mx.array:
189
  h = input_embeddings if input_embeddings is not None else self.embed_tokens(inputs)
@@ -193,9 +369,20 @@ class SenseNovaLoopTextModel(nn.Module):
193
  raise ValueError(f"cache has {len(cache)} slots, the execution plan needs {len(self.plan)}")
194
  fa_mask = create_attention_mask(h, cache[self.fa_visit]) if self.fa_visit is not None else None
195
  ssm_mask = create_ssm_mask(h, cache[self.ssm_visit]) if self.ssm_visit is not None else None
196
- for physical, c in zip(self.plan, cache):
197
- layer = self.layers[physical]
198
- h = layer(h, mask=ssm_mask if layer.is_linear else fa_mask, cache=c)
 
 
 
 
 
 
 
 
 
 
 
199
  return self.norm(h)
200
 
201
 
@@ -230,10 +417,57 @@ class Model(nn.Module):
230
  self.args = args
231
  self.model_type = args.model_type
232
  self.language_model = TextModel(q35.TextModelArgs.from_dict(args.text_config), args.loop_settings, args.mhc)
 
233
 
234
  def __call__(self, inputs: mx.array, cache=None, input_embeddings: Optional[mx.array] = None):
235
  return self.language_model(inputs, cache=cache, input_embeddings=input_embeddings)
236
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
237
  @property
238
  def model(self):
239
  return self.language_model.model
@@ -243,7 +477,11 @@ class Model(nn.Module):
243
  # +1 on norms for raw HF checkpoints. Routers stay fp32 (the reference refuses to run them in bf16).
244
  # The release checkpoint also carries `loop_gate.logits`, which the reference forward never reads: drop it.
245
  weights = {k: v for k, v in weights.items() if "loop_gate" not in k}
 
 
 
246
  weights = q35.Model.sanitize(self, weights)
 
247
  return {k: (v.astype(mx.float32) if (".mhc_attn." in k or ".mhc_ffn." in k) else v) for k, v in weights.items()}
248
 
249
  @property
 
23
 
24
  try:
25
  from . import qwen3_5 as q35
26
+ from .base import BaseModelArgs, create_attention_mask, create_ssm_mask, scaled_dot_product_attention
27
  from .cache import ArraysCache, KVCache
28
  except ImportError: # loaded as a model_file
29
  from mlx_lm.models import qwen3_5 as q35
30
+ from mlx_lm.models.base import BaseModelArgs, create_attention_mask, create_ssm_mask, scaled_dot_product_attention
31
  from mlx_lm.models.cache import ArraysCache, KVCache
32
 
33
+ import numpy as np
34
+
35
 
36
  @dataclass
37
  class ModelArgs(BaseModelArgs):
 
39
  text_config: dict
40
  loop_settings: dict = field(default_factory=dict)
41
  mhc: dict = field(default_factory=dict)
42
+ # image input (sensenova_mlx/vision.py runs the vision tower; the text model only needs these ids)
43
+ vision_config: Optional[dict] = None
44
+ image_token_id: int = 248056
45
+ video_token_id: int = 248057
46
+ vision_start_token_id: int = 248053
47
 
48
 
49
  def execution_plan(num_layers: int, loop: dict, mhc: dict) -> Tuple[int, ...]:
 
177
  return out.mean(axis=-2) if self.layer_idx == self.mhc_end - 1 else out
178
 
179
 
180
+ # ---- image input: 3-axis (t, h, w) multimodal rope ----
181
+ # Text-only prompts never activate any of this: the attention below falls straight through to mlx-lm's Qwen3.5
182
+ # attention (1-D rope at cache.offset), which equals interleaved mRoPE when all three position axes are equal.
183
+ # With an image, the prompt gets explicit [3, B, T] positions (get_rope_index) and every later token sits at
184
+ # cache offset + rope_delta. Positions -> cos/sin are computed once per forward and shared by all 56 visits, as in
185
+ # SenseNova's reference (SenseNovaLoopTextModel.forward computes position_embeddings once).
186
+
187
+
188
+ def get_rope_index(input_ids, image_grid_thw, image_token_id: int, spatial_merge_size: int = 2,
189
+ video_token_id: Optional[int] = None):
190
+ """numpy port of transformers Qwen3_5Model.get_rope_index for images (no padding mask, no video).
191
+
192
+ input_ids [B, T] -> (position_ids int64 [3, B, T], rope_deltas int64 [B]). Text runs advance all three axes by 1
193
+ per token; an image run starting at position p gets t = p, h = p + row, w = p + col over the merged grid, and the
194
+ next text resumes at p + max(gh, gw) / merge.
195
+ """
196
+ ids = np.asarray(input_ids)
197
+ if ids.ndim == 1:
198
+ ids = ids[None]
199
+ if video_token_id is not None and (ids == video_token_id).any():
200
+ raise NotImplementedError("video input is not supported")
201
+ grids = iter([tuple(int(v) for v in g) for g in (image_grid_thw if image_grid_thw is not None else [])])
202
+ B, T = ids.shape
203
+ pos = np.zeros((3, B, T), dtype=np.int64)
204
+ deltas = np.zeros((B,), dtype=np.int64)
205
+ m = spatial_merge_size
206
+ for b in range(B):
207
+ is_img = ids[b] == image_token_id
208
+ chunks, cur, i = [], 0, 0
209
+ while i < T:
210
+ j = i
211
+ while j < T and is_img[j] == is_img[i]:
212
+ j += 1
213
+ if not is_img[i]:
214
+ chunks.append(np.broadcast_to(np.arange(j - i) + cur, (3, j - i)))
215
+ cur += j - i
216
+ else:
217
+ t, h, w = next(grids)
218
+ gt, gh, gw = t, h // m, w // m
219
+ if gt * gh * gw != j - i:
220
+ raise ValueError(f"image run of {j - i} tokens does not match grid {(t, h, w)}")
221
+ tt, hh, ww = np.meshgrid(np.arange(gt), np.arange(gh), np.arange(gw), indexing="ij")
222
+ chunks.append(np.stack([tt.reshape(-1) + cur, hh.reshape(-1) + cur, ww.reshape(-1) + cur]))
223
+ cur += max(h, w) // m
224
+ i = j
225
+ p = np.concatenate(chunks, axis=1)
226
+ pos[:, b] = p
227
+ deltas[b] = p.max() + 1 - T
228
+ return pos, deltas
229
+
230
+
231
+ class MRoPEContext:
232
+ """Shared, per-model state for an image prompt: positions of the prompt and the decode delta (plain object, so
233
+ it is never part of the module's parameters)."""
234
+
235
+ def __init__(self, section=(11, 11, 10), interleaved=True):
236
+ self.section, self.interleaved = list(section), bool(interleaved)
237
+ self.position_ids = None # mx int32 [3, B, T] for the prompt
238
+ self.delta = None # mx int32 [B]
239
+ self.cos = self.sin = None # per-forward cos/sin [B, 1, L, rot_dims] (float32)
240
+
241
+ @property
242
+ def active(self):
243
+ return self.position_ids is not None
244
+
245
+
246
+ def _mrope_axis_map(half: int, section, interleaved: bool) -> np.ndarray:
247
+ axis = np.zeros((half,), dtype=np.int64)
248
+ if interleaved: # Qwen3_5TextRotaryEmbedding.recomposition_frequencies
249
+ for d, off in ((1, 1), (2, 2)):
250
+ axis[off : section[d] * 3 : 3] = d
251
+ else:
252
+ bounds = np.cumsum([0] + list(section))
253
+ for d in range(3):
254
+ axis[bounds[d] : bounds[d + 1]] = d
255
+ return axis
256
+
257
+
258
+ class MRoPEAttention(q35.Attention):
259
+ """mlx-lm Qwen3NextAttention with an explicit-position rope when an image prompt is active (body adapted from
260
+ mlx-lm, MIT). Inactive -> the parent __call__ unchanged."""
261
+
262
+ _mrope_ctx: Optional[MRoPEContext] = None
263
+
264
+ def __call__(self, x: mx.array, mask=None, cache=None) -> mx.array:
265
+ ctx = self._mrope_ctx
266
+ if ctx is None or ctx.cos is None:
267
+ return super().__call__(x, mask, cache)
268
+ B, L, D = x.shape
269
+ queries, gate = mx.split(self.q_proj(x).reshape(B, L, self.num_attention_heads, -1), 2, axis=-1)
270
+ gate = gate.reshape(B, L, -1)
271
+ keys, values = self.k_proj(x), self.v_proj(x)
272
+ queries = self.q_norm(queries).transpose(0, 2, 1, 3)
273
+ keys = self.k_norm(keys.reshape(B, L, self.num_key_value_heads, -1)).transpose(0, 2, 1, 3)
274
+ values = values.reshape(B, L, self.num_key_value_heads, -1).transpose(0, 2, 1, 3)
275
+ queries = _apply_mrope(queries, ctx.cos, ctx.sin)
276
+ keys = _apply_mrope(keys, ctx.cos, ctx.sin)
277
+ if cache is not None:
278
+ keys, values = cache.update_and_fetch(keys, values)
279
+ output = scaled_dot_product_attention(queries, keys, values, cache=cache, scale=self.scale, mask=mask)
280
+ output = output.transpose(0, 2, 1, 3).reshape(B, L, -1)
281
+ return self.o_proj(output * mx.sigmoid(gate))
282
+
283
+
284
+ def _apply_mrope(x, cos, sin):
285
+ """Non-traditional (rotate-half) rope on the first cos.shape[-1] dims; yarn mscale is already folded into cos/sin."""
286
+ rd = cos.shape[-1]
287
+ xr = x[..., :rd].astype(mx.float32)
288
+ h = rd // 2
289
+ rot = mx.concatenate([-xr[..., h:], xr[..., :h]], axis=-1)
290
+ out = (xr * cos + rot * sin).astype(x.dtype)
291
+ return mx.concatenate([out, x[..., rd:]], axis=-1) if rd < x.shape[-1] else out
292
+
293
+
294
+ def _rope_inv_freq_and_scale(rope) -> Tuple[mx.array, float]:
295
+ """Inverse frequencies and output scale of an mlx-lm rope module (YarnRoPE / nn.RoPE)."""
296
+ if hasattr(rope, "_freqs"):
297
+ inv = 1.0 / rope._freqs.astype(mx.float32)
298
+ else: # nn.RoPE: theta_i = scale * pos * base^(-2i/dims)
299
+ dims, base, scale = rope.dims, rope.base, getattr(rope, "scale", 1.0)
300
+ inv = scale / (base ** (mx.arange(0, dims, 2, dtype=mx.float32) / dims))
301
+ return inv, float(getattr(rope, "mscale", 1.0))
302
+
303
+
304
+ class _CastRMSNorm(nn.RMSNorm):
305
+ """RMSNorm whose output keeps the input dtype. With bf16 weights it is identical to nn.RMSNorm; with fp32 weights
306
+ (the checkpoint's 1 + w kept in float32, as Transformers computes it) the product is formed in fp32 and cast back."""
307
+
308
+ def __call__(self, x):
309
+ return mx.fast.rms_norm(x, self.weight, self.eps).astype(x.dtype)
310
+
311
+
312
+ def _use_cast_norms(module) -> None:
313
+ for _, m in module.named_modules():
314
+ if type(m) is nn.RMSNorm:
315
+ m.__class__ = _CastRMSNorm
316
+
317
+
318
  class SenseNovaLoopTextModel(nn.Module):
319
  def __init__(self, args: q35.TextModelArgs, loop: dict, mhc: dict):
320
  super().__init__()
 
329
  self.plan = execution_plan(args.num_hidden_layers, loop, mhc)
330
  self.fa_visit = next((e for e, p in enumerate(self.plan) if not self.layers[p].is_linear), None)
331
  self.ssm_visit = next((e for e, p in enumerate(self.plan) if self.layers[p].is_linear), None)
332
+ # image input: one shared context; the attention class swap adds no parameters
333
+ self.collect_hidden = None # parity diagnostics: set to a list to receive the embedding + every visit's output
334
+ rp = args.rope_scaling or {}
335
+ self.mrope = MRoPEContext(rp.get("mrope_section", [11, 11, 10]), rp.get("mrope_interleaved", True))
336
+ for layer in self.layers:
337
+ if not layer.is_linear:
338
+ layer.self_attn.__class__ = MRoPEAttention
339
+ layer.self_attn._mrope_ctx = self.mrope
340
+
341
+ def _mrope_tables(self, B: int, L: int, offset: int):
342
+ """cos/sin for positions [offset, offset + L): prompt positions where stored, offset + delta past the prompt."""
343
+ ctx = self.mrope
344
+ T = ctx.position_ids.shape[-1]
345
+ parts = []
346
+ if offset < T:
347
+ parts.append(ctx.position_ids[:, :, offset : min(offset + L, T)])
348
+ if offset + L > T:
349
+ start = max(offset, T)
350
+ tail = mx.arange(start, offset + L, dtype=mx.int32)[None, :] + ctx.delta[:, None] # [B, n]
351
+ parts.append(mx.broadcast_to(tail[None], (3, *tail.shape)))
352
+ pos = parts[0] if len(parts) == 1 else mx.concatenate(parts, axis=-1)
353
+ if pos.shape[1] != B:
354
+ pos = mx.broadcast_to(pos, (3, B, L))
355
+ rope = self.layers[self.plan[self.fa_visit]].self_attn.rope
356
+ inv, mscale = _rope_inv_freq_and_scale(rope)
357
+ half = inv.shape[0]
358
+ axis = mx.array(_mrope_axis_map(half, ctx.section, ctx.interleaved))
359
+ sel = mx.take(pos.astype(mx.float32), axis, axis=0) # [half, B, L]
360
+ ang = sel.transpose(1, 2, 0) * inv # [B, L, half]
361
+ ang = mx.concatenate([ang, ang], axis=-1)[:, None] # [B, 1, L, rot]
362
+ return mx.cos(ang) * mscale, mx.sin(ang) * mscale
363
 
364
  def __call__(self, inputs: mx.array, cache=None, input_embeddings: Optional[mx.array] = None) -> mx.array:
365
  h = input_embeddings if input_embeddings is not None else self.embed_tokens(inputs)
 
369
  raise ValueError(f"cache has {len(cache)} slots, the execution plan needs {len(self.plan)}")
370
  fa_mask = create_attention_mask(h, cache[self.fa_visit]) if self.fa_visit is not None else None
371
  ssm_mask = create_ssm_mask(h, cache[self.ssm_visit]) if self.ssm_visit is not None else None
372
+ if self.mrope.active and self.fa_visit is not None:
373
+ c0 = cache[self.fa_visit]
374
+ self.mrope.cos, self.mrope.sin = self._mrope_tables(h.shape[0], h.shape[1], int(c0.offset) if c0 is not None else 0)
375
+ col = self.collect_hidden
376
+ if col is not None:
377
+ col.append(h)
378
+ try:
379
+ for physical, c in zip(self.plan, cache):
380
+ layer = self.layers[physical]
381
+ h = layer(h, mask=ssm_mask if layer.is_linear else fa_mask, cache=c)
382
+ if col is not None:
383
+ col.append(h)
384
+ finally:
385
+ self.mrope.cos = self.mrope.sin = None
386
  return self.norm(h)
387
 
388
 
 
417
  self.args = args
418
  self.model_type = args.model_type
419
  self.language_model = TextModel(q35.TextModelArgs.from_dict(args.text_config), args.loop_settings, args.mhc)
420
+ _use_cast_norms(self.language_model)
421
 
422
  def __call__(self, inputs: mx.array, cache=None, input_embeddings: Optional[mx.array] = None):
423
  return self.language_model(inputs, cache=cache, input_embeddings=input_embeddings)
424
 
425
+ # ---- image input (the vision tower lives outside this module: sensenova_mlx/vision.py) ----
426
+ def set_mrope(self, position_ids, rope_deltas):
427
+ """Activate 3-axis positions for the prompt about to be prefilled (from get_rope_index) and the decode delta.
428
+ Stays active for the following decode steps; call clear_mrope() before an unrelated prompt."""
429
+ ctx = self.language_model.model.mrope
430
+ ctx.position_ids = mx.array(np.asarray(position_ids), dtype=mx.int32)
431
+ ctx.delta = mx.array(np.asarray(rope_deltas).reshape(-1), dtype=mx.int32)
432
+
433
+ def clear_mrope(self):
434
+ ctx = self.language_model.model.mrope
435
+ ctx.position_ids = ctx.delta = ctx.cos = ctx.sin = None
436
+
437
+ @property
438
+ def rope_deltas(self):
439
+ return self.language_model.model.mrope.delta
440
+
441
+ def embed_with_images(self, input_ids: mx.array, image_features: Optional[mx.array]) -> mx.array:
442
+ """Token embeddings with the merged vision features scattered over <|image_pad|> positions, in order
443
+ (HF masked_scatter)."""
444
+ h = self.language_model.model.embed_tokens(input_ids)
445
+ if image_features is None:
446
+ return h
447
+ mask = input_ids == self.args.image_token_id
448
+ n = int(mask.sum().item())
449
+ if n != image_features.shape[0]:
450
+ raise ValueError(f"{n} image tokens in the prompt, {image_features.shape[0]} image features")
451
+ idx = mx.maximum(mx.cumsum(mask.reshape(-1).astype(mx.int32)) - 1, 0).reshape(mask.shape)
452
+ feats = image_features.astype(h.dtype)[idx]
453
+ return mx.where(mask[..., None], feats, h)
454
+
455
+ def prepare_image_prompt(self, input_ids, image_features: Optional[mx.array], image_grid_thw,
456
+ spatial_merge_size: int = 2) -> mx.array:
457
+ """input_ids [B, T] (+ features from the vision tower) -> input_embeddings; also activates mRoPE positions.
458
+ A prompt without image tokens clears mRoPE and returns plain embeddings (text path unchanged)."""
459
+ ids_np = np.asarray(input_ids)
460
+ if ids_np.ndim == 1:
461
+ ids_np = ids_np[None]
462
+ ids = mx.array(ids_np)
463
+ if image_features is None or not (ids_np == self.args.image_token_id).any():
464
+ self.clear_mrope()
465
+ return self.language_model.model.embed_tokens(ids)
466
+ pos, deltas = get_rope_index(ids_np, image_grid_thw, self.args.image_token_id, spatial_merge_size,
467
+ self.args.video_token_id)
468
+ self.set_mrope(pos, deltas)
469
+ return self.embed_with_images(ids, image_features)
470
+
471
  @property
472
  def model(self):
473
  return self.language_model.model
 
477
  # +1 on norms for raw HF checkpoints. Routers stay fp32 (the reference refuses to run them in bf16).
478
  # The release checkpoint also carries `loop_gate.logits`, which the reference forward never reads: drop it.
479
  weights = {k: v for k, v in weights.items() if "loop_gate" not in k}
480
+ # optional float32 norm weights (1 + w) shipped as `fp32_norm.<final name>`; they replace the bf16 copies
481
+ fp32_norms = {k[len("fp32_norm."):]: v.astype(mx.float32) for k, v in weights.items() if k.startswith("fp32_norm.")}
482
+ weights = {k: v for k, v in weights.items() if not k.startswith("fp32_norm.")}
483
  weights = q35.Model.sanitize(self, weights)
484
+ weights.update(fp32_norms)
485
  return {k: (v.astype(mx.float32) if (".mhc_attn." in k or ".mhc_ffn." in k) else v) for k, v in weights.items()}
486
 
487
  @property
vision.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0304cc30f617f782932bbdd102d3be477215c72cd446be233783a31711938c77
3
+ size 912058038