Text Generation
MLX
Safetensors
sensenova_loop
sensenova
sensenova-loop
looped-transformer
mhc
qwen3_5
apple-silicon
conversational
4-bit precision
Instructions to use netprojects/SenseNova-Loop-9B-MLX-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use netprojects/SenseNova-Loop-9B-MLX-4bit with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("netprojects/SenseNova-Loop-9B-MLX-4bit") prompt = "Write a story about Einstein" messages = [{"role": "user", "content": prompt}] prompt = tokenizer.apply_chat_template( messages, add_generation_prompt=True ) text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Pi
How to use netprojects/SenseNova-Loop-9B-MLX-4bit with Pi:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "netprojects/SenseNova-Loop-9B-MLX-4bit"
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "mlx-lm": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "netprojects/SenseNova-Loop-9B-MLX-4bit" } ] } } }Run Pi
# Start Pi in your project directory: pi
- MLX LM
How to use netprojects/SenseNova-Loop-9B-MLX-4bit with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Interactive chat REPL mlx_lm.chat --model "netprojects/SenseNova-Loop-9B-MLX-4bit"
Run an OpenAI-compatible server
# Install MLX LM uv tool install mlx-lm # Start the server mlx_lm.server --model "netprojects/SenseNova-Loop-9B-MLX-4bit" # Calling the OpenAI-compatible server with curl curl -X POST "http://localhost:8080/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "netprojects/SenseNova-Loop-9B-MLX-4bit", "messages": [ {"role": "user", "content": "Hello"} ] }' - Hermes Agent
How to use netprojects/SenseNova-Loop-9B-MLX-4bit with Hermes Agent:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "netprojects/SenseNova-Loop-9B-MLX-4bit"
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default netprojects/SenseNova-Loop-9B-MLX-4bit
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use netprojects/SenseNova-Loop-9B-MLX-4bit with OpenClaw:
Start the MLX server
# Install MLX LM: uv tool install mlx-lm # Start a local OpenAI-compatible server: mlx_lm.server --model "netprojects/SenseNova-Loop-9B-MLX-4bit"
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "netprojects/SenseNova-Loop-9B-MLX-4bit" \ --custom-provider-id mlx-lm \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
SenseNova-Loop-9B MLX 4bit: image input (vision tower), MTP speculative-decoding head, fp32 norm weights, updated model code
Browse files- NOTICE +10 -2
- README.md +53 -11
- __pycache__/sensenova_loop.cpython-315.pyc +0 -0
- config.json +26 -2
- model.fp32norms.safetensors +3 -0
- mtp.safetensors +3 -0
- preprocessor_config.json +26 -0
- processor_config.json +60 -0
- sensenova_loop.py +243 -5
- vision.safetensors +3 -0
NOTICE
CHANGED
|
@@ -4,8 +4,16 @@ This repository contains a modified version of SenseNova-Loop-9B by SenseNova
|
|
| 4 |
(https://huggingface.co/sensenova/SenseNova-Loop-9B, revision 329a65f701ba8c48b73c30d0cbaa6f0449733673),
|
| 5 |
licensed under the Apache License, Version 2.0 (see LICENSE).
|
| 6 |
|
| 7 |
-
Modifications (17 November 2026): the weights were converted to Apple MLX format (quantized to 4-bit);
|
| 8 |
-
|
| 9 |
SenseNova-Loop architecture written by NetProjects, also under Apache-2.0.
|
| 10 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
Not affiliated with SenseTime / SenseNova.
|
|
|
|
| 4 |
(https://huggingface.co/sensenova/SenseNova-Loop-9B, revision 329a65f701ba8c48b73c30d0cbaa6f0449733673),
|
| 5 |
licensed under the Apache License, Version 2.0 (see LICENSE).
|
| 6 |
|
| 7 |
+
Modifications (17 November 2026): the weights were converted to Apple MLX format (quantized to 4-bit);
|
| 8 |
+
the unused loop_gate tensor is not included. sensenova_loop.py is a new MLX implementation of the
|
| 9 |
SenseNova-Loop architecture written by NetProjects, also under Apache-2.0.
|
| 10 |
|
| 11 |
+
Update (18 November 2026): this repository also contains, unchanged and in bfloat16, the vision tower weights of
|
| 12 |
+
SenseNova-Loop-9B (model.visual.*, revision 329a65f701ba8c48b73c30d0cbaa6f0449733673) as vision.safetensors, and the
|
| 13 |
+
MTP head weights (mtp.safetensors) from https://huggingface.co/sensenova/SenseNova-Loop-9B at revision
|
| 14 |
+
17ea94936a0a2157bc102b6ea516beadfa44bb6d, both by SenseNova under the Apache License, Version 2.0, plus the 81 RMSNorm
|
| 15 |
+
weights of the language model stored as float32 (1 + w) in model.fp32norms.safetensors. sensenova_loop.py now also
|
| 16 |
+
implements image input (mRoPE positions); the vision and MTP runtime is at
|
| 17 |
+
https://github.com/internet-projects/sensenova-loop-mlx (v0.3.0), Apache-2.0.
|
| 18 |
+
|
| 19 |
Not affiliated with SenseTime / SenseNova.
|
README.md
CHANGED
|
@@ -8,7 +8,7 @@ tags: [mlx, sensenova, sensenova-loop, looped-transformer, mhc, qwen3_5, apple-s
|
|
| 8 |
---
|
| 9 |
# SenseNova-Loop-9B — MLX 4-bit
|
| 10 |
|
| 11 |
-
A 4-bit MLX conversion (affine, group size 64) of [sensenova/SenseNova-Loop-9B](https://huggingface.co/sensenova/SenseNova-Loop-9B) (revision `329a65f`, released 17 November 2026) for Apple Silicon. 5.1 GB of weights. The smallest build. Text
|
| 12 |
|
| 13 |
Published by [NetProjects](https://netprojects.com) (Internet Projects Ltd). Not affiliated with SenseTime / SenseNova.
|
| 14 |
|
|
@@ -20,22 +20,22 @@ SenseNova-Loop is a Qwen3.5-based model with two additions, and this build runs
|
|
| 20 |
- **Two-stream mHC** on layers 4–27: the hidden state is split into two residual streams that are routed by small float32 routers (pre / post gates and a 2×2 Sinkhorn mixing matrix, 20 iterations). The 96 router tensors stay in float32 and are never quantized.
|
| 21 |
- **Long context:** YaRN rope scaling (factor 2 over 262,144 positions), as in SenseNova's config.
|
| 22 |
|
| 23 |
-
The model code is `sensenova_loop.py`, included in this repo and loaded by mlx-lm through `trust_remote_code=True`.
|
| 24 |
|
| 25 |
-
## Results (MacBook Pro M5 Max, 128 GB, 17 November 2026)
|
| 26 |
|
| 27 |
Agreement with SenseNova's own PyTorch code (Transformers 5.18, bf16) on 16 prompts, every position:
|
| 28 |
|
| 29 |
-
| Build | Size | Same top token as SenseNova's code | KL at the last token | Greedy text identical (16 prompts) |
|
| 30 |
-
|---|---|---|---|---|
|
| 31 |
-
| [6-bit](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-6bit) (recommended) | 7.3 GB | 96.7% | 0.0056 |
|
| 32 |
-
| [8-bit](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-8bit) | 9.5 GB |
|
| 33 |
-
| **[4-bit](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-4bit)** (this repo) | 5.1 GB | 84.
|
| 34 |
-
| [bf16](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-bf16) | 17.9 GB |
|
| 35 |
|
| 36 |
At 4 bits this build follows the original less closely than the larger ones. For the closest match on a small budget use the [6-bit build](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-6bit).
|
| 37 |
|
| 38 |
-
**Checks on this port:**
|
| 39 |
|
| 40 |
## Benchmarks (as reported by SenseNova)
|
| 41 |
|
|
@@ -75,11 +75,53 @@ print(generate(model, tokenizer, prompt=prompt, max_tokens=256))
|
|
| 75 |
|
| 76 |
Set `enable_thinking=True` for thinking mode. Server: `mlx_lm.server --model netprojects/SenseNova-Loop-9B-MLX-4bit --trust-remote-code`.
|
| 77 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 78 |
**Other formats:** [MLX 6-bit](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-6bit) (7.3 GB) · [MLX 8-bit](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-8bit) (9.5 GB) · [MLX bf16](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-bf16) (17.9 GB)
|
| 79 |
|
| 80 |
## License
|
| 81 |
|
| 82 |
-
SenseNova-Loop-9B is released by SenseNova under the **Apache License 2.0** (`LICENSE` included). This repository contains converted and quantized weights of that model (modified from the original: converted to MLX format and quantized to 4-bit) and an MLX implementation of its architecture, distributed under the same license. See `NOTICE`.
|
| 83 |
|
| 84 |
Credits: SenseNova / SenseTime (SenseNova-Loop-9B), Qwen team (Qwen3.5), Apple MLX and mlx-lm. Conversion and MLX port by Paris Michalas, Partner at [NetProjects](https://netprojects.com). Not affiliated with SenseTime / SenseNova.
|
| 85 |
|
|
|
|
| 8 |
---
|
| 9 |
# SenseNova-Loop-9B — MLX 4-bit
|
| 10 |
|
| 11 |
+
A 4-bit MLX conversion (affine, group size 64) of [sensenova/SenseNova-Loop-9B](https://huggingface.co/sensenova/SenseNova-Loop-9B) (revision `329a65f`, released 17 November 2026) for Apple Silicon. 5.1 GB of weights. The smallest build. Text and image input, with SenseNova's 512K-token YaRN context setting, and SenseNova's MTP head for about 1.7× faster decoding.
|
| 12 |
|
| 13 |
Published by [NetProjects](https://netprojects.com) (Internet Projects Ltd). Not affiliated with SenseTime / SenseNova.
|
| 14 |
|
|
|
|
| 20 |
- **Two-stream mHC** on layers 4–27: the hidden state is split into two residual streams that are routed by small float32 routers (pre / post gates and a 2×2 Sinkhorn mixing matrix, 20 iterations). The 96 router tensors stay in float32 and are never quantized.
|
| 21 |
- **Long context:** YaRN rope scaling (factor 2 over 262,144 positions), as in SenseNova's config.
|
| 22 |
|
| 23 |
+
The model code is `sensenova_loop.py`, included in this repo and loaded by mlx-lm through `trust_remote_code=True`. The 81 RMSNorm weights are also shipped in float32 (`model.fp32norms.safetensors`, loaded automatically), so the norms are computed exactly as in SenseNova's code.
|
| 24 |
|
| 25 |
+
## Results (MacBook Pro M5 Max, 128 GB, 17–18 November 2026)
|
| 26 |
|
| 27 |
Agreement with SenseNova's own PyTorch code (Transformers 5.18, bf16) on 16 prompts, every position:
|
| 28 |
|
| 29 |
+
| Build | Size | Same top token as SenseNova's code | KL at the last token | Greedy text identical (16 prompts) | Image prompts: same top token as bf16 |
|
| 30 |
+
|---|---|---|---|---|---|
|
| 31 |
+
| [6-bit](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-6bit) (recommended) | 7.3 GB | 96.7% | 0.0056 | 4 / 16 | 98.0% |
|
| 32 |
+
| [8-bit](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-8bit) | 9.5 GB | 98.6% | 0.0015 | 11 / 16 | 96.7% |
|
| 33 |
+
| **[4-bit](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-4bit)** (this repo) | 5.1 GB | 84.4% | 0.1038 | 2 / 16 | 95.4% |
|
| 34 |
+
| [bf16](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-bf16) | 17.9 GB | 99.1% | 0.0005 | 12 / 16 | reference |
|
| 35 |
|
| 36 |
At 4 bits this build follows the original less closely than the larger ones. For the closest match on a small budget use the [6-bit build](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-6bit).
|
| 37 |
|
| 38 |
+
**Checks on this port:** all four builds answered short test prompts correctly with thinking on and off, and with images; the 6-bit build found a pass code hidden in a 64K-token prompt. Image input matches SenseNova's code in float32 (same top token at every checked position, image positions exact). With the MTP head, greedy output matches plain decoding except where the two best tokens are tied within 0.25 logits.
|
| 39 |
|
| 40 |
## Benchmarks (as reported by SenseNova)
|
| 41 |
|
|
|
|
| 75 |
|
| 76 |
Set `enable_thinking=True` for thinking mode. Server: `mlx_lm.server --model netprojects/SenseNova-Loop-9B-MLX-4bit --trust-remote-code`.
|
| 77 |
|
| 78 |
+
`mlx_lm.load` and `mlx_lm.server` use the text model. Image input and MTP decoding use the small runtime in
|
| 79 |
+
[sensenova-loop-mlx v0.3.0](https://github.com/internet-projects/sensenova-loop-mlx/releases/tag/v0.3.0):
|
| 80 |
+
|
| 81 |
+
```bash
|
| 82 |
+
hf download netprojects/SenseNova-Loop-9B-MLX-4bit --local-dir sn-loop
|
| 83 |
+
git clone --branch v0.3.0 https://github.com/internet-projects/sensenova-loop-mlx && cd sensenova-loop-mlx
|
| 84 |
+
pip install -U "mlx-lm>=0.32" pillow
|
| 85 |
+
```
|
| 86 |
+
|
| 87 |
+
## Image input
|
| 88 |
+
|
| 89 |
+
```bash
|
| 90 |
+
python scripts/generate_vision.py --model ../sn-loop --image photo.jpg --prompt "Describe this picture." --no-think
|
| 91 |
+
```
|
| 92 |
+
|
| 93 |
+
Repeat `--image` for several images. Each image is capped at about 1 megapixel by default (`--max-pixels 0` uses
|
| 94 |
+
SenseNova's own limit of 16.7 MP, which needs much more memory). The vision tower (`vision.safetensors`) stays in bfloat16.
|
| 95 |
+
|
| 96 |
+
## Faster decoding with MTP
|
| 97 |
+
|
| 98 |
+
SenseNova's multi-token-prediction head (`mtp.safetensors`) drafts the next tokens and the model checks them in one pass.
|
| 99 |
+
|
| 100 |
+
```python
|
| 101 |
+
from mlx_lm import load
|
| 102 |
+
from sensenova_mlx.mtp import generate_mtp, load_mtp_head
|
| 103 |
+
|
| 104 |
+
model, tokenizer = load("../sn-loop", trust_remote_code=True)
|
| 105 |
+
head = load_mtp_head(model, "../sn-loop/mtp.safetensors")
|
| 106 |
+
messages = [{"role": "user", "content": "Explain in two sentences why the sky is blue."}]
|
| 107 |
+
ids = tokenizer.encode(tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False, enable_thinking=False))
|
| 108 |
+
print(tokenizer.decode(list(generate_mtp(model, head, ids, max_tokens=256, k=3))))
|
| 109 |
+
```
|
| 110 |
+
|
| 111 |
+
Decode speed of the 6-bit build on a MacBook Pro M5 Max (greedy, 256 new tokens, median of 3 prompts):
|
| 112 |
+
|
| 113 |
+
| | Plain decoding | MTP k=1 | MTP k=2 | MTP k=3 |
|
| 114 |
+
|---|---|---|---|---|
|
| 115 |
+
| Tokens per second | 19.3 | 27.0 | 30.6 | **33.5** |
|
| 116 |
+
| Speed-up | 1.00× | 1.40× | 1.59× | **1.74×** |
|
| 117 |
+
|
| 118 |
+
`temperature > 0` keeps the model's sampling distribution (draft tokens are accepted by probability ratio).
|
| 119 |
+
|
| 120 |
**Other formats:** [MLX 6-bit](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-6bit) (7.3 GB) · [MLX 8-bit](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-8bit) (9.5 GB) · [MLX bf16](https://huggingface.co/netprojects/SenseNova-Loop-9B-MLX-bf16) (17.9 GB)
|
| 121 |
|
| 122 |
## License
|
| 123 |
|
| 124 |
+
SenseNova-Loop-9B is released by SenseNova under the **Apache License 2.0** (`LICENSE` included). This repository contains converted and quantized weights of that model (modified from the original: converted to MLX format and quantized to 4-bit), SenseNova's vision tower and MTP head weights (unchanged, bfloat16) and an MLX implementation of its architecture, distributed under the same license. See `NOTICE`.
|
| 125 |
|
| 126 |
Credits: SenseNova / SenseTime (SenseNova-Loop-9B), Qwen team (Qwen3.5), Apple MLX and mlx-lm. Conversion and MLX port by Paris Michalas, Partner at [NetProjects](https://netprojects.com). Not affiliated with SenseTime / SenseNova.
|
| 127 |
|
__pycache__/sensenova_loop.cpython-315.pyc
ADDED
|
Binary file (45.2 kB). View file
|
|
|
config.json
CHANGED
|
@@ -131,7 +131,31 @@
|
|
| 131 |
"sensenova_mlx": {
|
| 132 |
"source": "sensenova/SenseNova-Loop-9B",
|
| 133 |
"revision": "329a65f701ba8c48b73c30d0cbaa6f0449733673",
|
| 134 |
-
"text_only":
|
| 135 |
-
"hi_bits_for": null
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 136 |
}
|
| 137 |
}
|
|
|
|
| 131 |
"sensenova_mlx": {
|
| 132 |
"source": "sensenova/SenseNova-Loop-9B",
|
| 133 |
"revision": "329a65f701ba8c48b73c30d0cbaa6f0449733673",
|
| 134 |
+
"text_only": false,
|
| 135 |
+
"hi_bits_for": null,
|
| 136 |
+
"vision_file": "vision.safetensors",
|
| 137 |
+
"vision_dtype": "bfloat16",
|
| 138 |
+
"default_max_pixels": 1048576,
|
| 139 |
+
"mtp_file": "mtp.safetensors",
|
| 140 |
+
"mtp_source": "sensenova/SenseNova-Loop-9B",
|
| 141 |
+
"mtp_revision": "17ea94936a0a2157bc102b6ea516beadfa44bb6d",
|
| 142 |
+
"fp32_norms_file": "model.fp32norms.safetensors"
|
| 143 |
+
},
|
| 144 |
+
"vision_config": {
|
| 145 |
+
"deepstack_visual_indexes": [],
|
| 146 |
+
"depth": 27,
|
| 147 |
+
"dtype": "bfloat16",
|
| 148 |
+
"hidden_act": "gelu_pytorch_tanh",
|
| 149 |
+
"hidden_size": 1152,
|
| 150 |
+
"in_channels": 3,
|
| 151 |
+
"initializer_range": 0.02,
|
| 152 |
+
"intermediate_size": 4304,
|
| 153 |
+
"model_type": "qwen3_5_vision",
|
| 154 |
+
"num_heads": 16,
|
| 155 |
+
"num_position_embeddings": 2304,
|
| 156 |
+
"out_hidden_size": 4096,
|
| 157 |
+
"patch_size": 16,
|
| 158 |
+
"spatial_merge_size": 2,
|
| 159 |
+
"temporal_patch_size": 2
|
| 160 |
}
|
| 161 |
}
|
model.fp32norms.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:c6a6e1103096a65e9f9b5a18e3b9a387b469319a7ea973bbd6e4fc2a95fedde7
|
| 3 |
+
size 1091976
|
mtp.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3b84d5143ad5c97cbeffb4ffe9d2db9f620e3cbbfa334f987a04a3cb902d08a6
|
| 3 |
+
size 486582880
|
preprocessor_config.json
ADDED
|
@@ -0,0 +1,26 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"do_convert_rgb": true,
|
| 3 |
+
"do_normalize": true,
|
| 4 |
+
"do_rescale": true,
|
| 5 |
+
"do_resize": true,
|
| 6 |
+
"image_mean": [
|
| 7 |
+
0.5,
|
| 8 |
+
0.5,
|
| 9 |
+
0.5
|
| 10 |
+
],
|
| 11 |
+
"image_processor_type": "Qwen2VLImageProcessor",
|
| 12 |
+
"image_std": [
|
| 13 |
+
0.5,
|
| 14 |
+
0.5,
|
| 15 |
+
0.5
|
| 16 |
+
],
|
| 17 |
+
"merge_size": 2,
|
| 18 |
+
"patch_size": 16,
|
| 19 |
+
"resample": 3,
|
| 20 |
+
"rescale_factor": 0.00392156862745098,
|
| 21 |
+
"size": {
|
| 22 |
+
"longest_edge": 16777216,
|
| 23 |
+
"shortest_edge": 65536
|
| 24 |
+
},
|
| 25 |
+
"temporal_patch_size": 2
|
| 26 |
+
}
|
processor_config.json
ADDED
|
@@ -0,0 +1,60 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"image_processor": {
|
| 3 |
+
"do_convert_rgb": true,
|
| 4 |
+
"do_normalize": true,
|
| 5 |
+
"do_rescale": true,
|
| 6 |
+
"do_resize": true,
|
| 7 |
+
"image_mean": [
|
| 8 |
+
0.5,
|
| 9 |
+
0.5,
|
| 10 |
+
0.5
|
| 11 |
+
],
|
| 12 |
+
"image_processor_type": "Qwen2VLImageProcessor",
|
| 13 |
+
"image_std": [
|
| 14 |
+
0.5,
|
| 15 |
+
0.5,
|
| 16 |
+
0.5
|
| 17 |
+
],
|
| 18 |
+
"merge_size": 2,
|
| 19 |
+
"patch_size": 16,
|
| 20 |
+
"resample": 3,
|
| 21 |
+
"rescale_factor": 0.00392156862745098,
|
| 22 |
+
"size": {
|
| 23 |
+
"longest_edge": 16777216,
|
| 24 |
+
"shortest_edge": 65536
|
| 25 |
+
},
|
| 26 |
+
"temporal_patch_size": 2
|
| 27 |
+
},
|
| 28 |
+
"processor_class": "Qwen3VLProcessor",
|
| 29 |
+
"video_processor": {
|
| 30 |
+
"do_convert_rgb": true,
|
| 31 |
+
"do_normalize": true,
|
| 32 |
+
"do_rescale": true,
|
| 33 |
+
"do_resize": true,
|
| 34 |
+
"do_sample_frames": true,
|
| 35 |
+
"fps": 2,
|
| 36 |
+
"image_mean": [
|
| 37 |
+
0.5,
|
| 38 |
+
0.5,
|
| 39 |
+
0.5
|
| 40 |
+
],
|
| 41 |
+
"image_std": [
|
| 42 |
+
0.5,
|
| 43 |
+
0.5,
|
| 44 |
+
0.5
|
| 45 |
+
],
|
| 46 |
+
"max_frames": 768,
|
| 47 |
+
"merge_size": 2,
|
| 48 |
+
"min_frames": 4,
|
| 49 |
+
"patch_size": 16,
|
| 50 |
+
"resample": 3,
|
| 51 |
+
"rescale_factor": 0.00392156862745098,
|
| 52 |
+
"return_metadata": false,
|
| 53 |
+
"size": {
|
| 54 |
+
"longest_edge": 25165824,
|
| 55 |
+
"shortest_edge": 4096
|
| 56 |
+
},
|
| 57 |
+
"temporal_patch_size": 2,
|
| 58 |
+
"video_processor_type": "Qwen3VLVideoProcessor"
|
| 59 |
+
}
|
| 60 |
+
}
|
sensenova_loop.py
CHANGED
|
@@ -23,13 +23,15 @@ import mlx.nn as nn
|
|
| 23 |
|
| 24 |
try:
|
| 25 |
from . import qwen3_5 as q35
|
| 26 |
-
from .base import BaseModelArgs, create_attention_mask, create_ssm_mask
|
| 27 |
from .cache import ArraysCache, KVCache
|
| 28 |
except ImportError: # loaded as a model_file
|
| 29 |
from mlx_lm.models import qwen3_5 as q35
|
| 30 |
-
from mlx_lm.models.base import BaseModelArgs, create_attention_mask, create_ssm_mask
|
| 31 |
from mlx_lm.models.cache import ArraysCache, KVCache
|
| 32 |
|
|
|
|
|
|
|
| 33 |
|
| 34 |
@dataclass
|
| 35 |
class ModelArgs(BaseModelArgs):
|
|
@@ -37,6 +39,11 @@ class ModelArgs(BaseModelArgs):
|
|
| 37 |
text_config: dict
|
| 38 |
loop_settings: dict = field(default_factory=dict)
|
| 39 |
mhc: dict = field(default_factory=dict)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 40 |
|
| 41 |
|
| 42 |
def execution_plan(num_layers: int, loop: dict, mhc: dict) -> Tuple[int, ...]:
|
|
@@ -170,6 +177,144 @@ class MHCDecoderLayer(q35.DecoderLayer):
|
|
| 170 |
return out.mean(axis=-2) if self.layer_idx == self.mhc_end - 1 else out
|
| 171 |
|
| 172 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 173 |
class SenseNovaLoopTextModel(nn.Module):
|
| 174 |
def __init__(self, args: q35.TextModelArgs, loop: dict, mhc: dict):
|
| 175 |
super().__init__()
|
|
@@ -184,6 +329,37 @@ class SenseNovaLoopTextModel(nn.Module):
|
|
| 184 |
self.plan = execution_plan(args.num_hidden_layers, loop, mhc)
|
| 185 |
self.fa_visit = next((e for e, p in enumerate(self.plan) if not self.layers[p].is_linear), None)
|
| 186 |
self.ssm_visit = next((e for e, p in enumerate(self.plan) if self.layers[p].is_linear), None)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 187 |
|
| 188 |
def __call__(self, inputs: mx.array, cache=None, input_embeddings: Optional[mx.array] = None) -> mx.array:
|
| 189 |
h = input_embeddings if input_embeddings is not None else self.embed_tokens(inputs)
|
|
@@ -193,9 +369,20 @@ class SenseNovaLoopTextModel(nn.Module):
|
|
| 193 |
raise ValueError(f"cache has {len(cache)} slots, the execution plan needs {len(self.plan)}")
|
| 194 |
fa_mask = create_attention_mask(h, cache[self.fa_visit]) if self.fa_visit is not None else None
|
| 195 |
ssm_mask = create_ssm_mask(h, cache[self.ssm_visit]) if self.ssm_visit is not None else None
|
| 196 |
-
|
| 197 |
-
|
| 198 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 199 |
return self.norm(h)
|
| 200 |
|
| 201 |
|
|
@@ -230,10 +417,57 @@ class Model(nn.Module):
|
|
| 230 |
self.args = args
|
| 231 |
self.model_type = args.model_type
|
| 232 |
self.language_model = TextModel(q35.TextModelArgs.from_dict(args.text_config), args.loop_settings, args.mhc)
|
|
|
|
| 233 |
|
| 234 |
def __call__(self, inputs: mx.array, cache=None, input_embeddings: Optional[mx.array] = None):
|
| 235 |
return self.language_model(inputs, cache=cache, input_embeddings=input_embeddings)
|
| 236 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 237 |
@property
|
| 238 |
def model(self):
|
| 239 |
return self.language_model.model
|
|
@@ -243,7 +477,11 @@ class Model(nn.Module):
|
|
| 243 |
# +1 on norms for raw HF checkpoints. Routers stay fp32 (the reference refuses to run them in bf16).
|
| 244 |
# The release checkpoint also carries `loop_gate.logits`, which the reference forward never reads: drop it.
|
| 245 |
weights = {k: v for k, v in weights.items() if "loop_gate" not in k}
|
|
|
|
|
|
|
|
|
|
| 246 |
weights = q35.Model.sanitize(self, weights)
|
|
|
|
| 247 |
return {k: (v.astype(mx.float32) if (".mhc_attn." in k or ".mhc_ffn." in k) else v) for k, v in weights.items()}
|
| 248 |
|
| 249 |
@property
|
|
|
|
| 23 |
|
| 24 |
try:
|
| 25 |
from . import qwen3_5 as q35
|
| 26 |
+
from .base import BaseModelArgs, create_attention_mask, create_ssm_mask, scaled_dot_product_attention
|
| 27 |
from .cache import ArraysCache, KVCache
|
| 28 |
except ImportError: # loaded as a model_file
|
| 29 |
from mlx_lm.models import qwen3_5 as q35
|
| 30 |
+
from mlx_lm.models.base import BaseModelArgs, create_attention_mask, create_ssm_mask, scaled_dot_product_attention
|
| 31 |
from mlx_lm.models.cache import ArraysCache, KVCache
|
| 32 |
|
| 33 |
+
import numpy as np
|
| 34 |
+
|
| 35 |
|
| 36 |
@dataclass
|
| 37 |
class ModelArgs(BaseModelArgs):
|
|
|
|
| 39 |
text_config: dict
|
| 40 |
loop_settings: dict = field(default_factory=dict)
|
| 41 |
mhc: dict = field(default_factory=dict)
|
| 42 |
+
# image input (sensenova_mlx/vision.py runs the vision tower; the text model only needs these ids)
|
| 43 |
+
vision_config: Optional[dict] = None
|
| 44 |
+
image_token_id: int = 248056
|
| 45 |
+
video_token_id: int = 248057
|
| 46 |
+
vision_start_token_id: int = 248053
|
| 47 |
|
| 48 |
|
| 49 |
def execution_plan(num_layers: int, loop: dict, mhc: dict) -> Tuple[int, ...]:
|
|
|
|
| 177 |
return out.mean(axis=-2) if self.layer_idx == self.mhc_end - 1 else out
|
| 178 |
|
| 179 |
|
| 180 |
+
# ---- image input: 3-axis (t, h, w) multimodal rope ----
|
| 181 |
+
# Text-only prompts never activate any of this: the attention below falls straight through to mlx-lm's Qwen3.5
|
| 182 |
+
# attention (1-D rope at cache.offset), which equals interleaved mRoPE when all three position axes are equal.
|
| 183 |
+
# With an image, the prompt gets explicit [3, B, T] positions (get_rope_index) and every later token sits at
|
| 184 |
+
# cache offset + rope_delta. Positions -> cos/sin are computed once per forward and shared by all 56 visits, as in
|
| 185 |
+
# SenseNova's reference (SenseNovaLoopTextModel.forward computes position_embeddings once).
|
| 186 |
+
|
| 187 |
+
|
| 188 |
+
def get_rope_index(input_ids, image_grid_thw, image_token_id: int, spatial_merge_size: int = 2,
|
| 189 |
+
video_token_id: Optional[int] = None):
|
| 190 |
+
"""numpy port of transformers Qwen3_5Model.get_rope_index for images (no padding mask, no video).
|
| 191 |
+
|
| 192 |
+
input_ids [B, T] -> (position_ids int64 [3, B, T], rope_deltas int64 [B]). Text runs advance all three axes by 1
|
| 193 |
+
per token; an image run starting at position p gets t = p, h = p + row, w = p + col over the merged grid, and the
|
| 194 |
+
next text resumes at p + max(gh, gw) / merge.
|
| 195 |
+
"""
|
| 196 |
+
ids = np.asarray(input_ids)
|
| 197 |
+
if ids.ndim == 1:
|
| 198 |
+
ids = ids[None]
|
| 199 |
+
if video_token_id is not None and (ids == video_token_id).any():
|
| 200 |
+
raise NotImplementedError("video input is not supported")
|
| 201 |
+
grids = iter([tuple(int(v) for v in g) for g in (image_grid_thw if image_grid_thw is not None else [])])
|
| 202 |
+
B, T = ids.shape
|
| 203 |
+
pos = np.zeros((3, B, T), dtype=np.int64)
|
| 204 |
+
deltas = np.zeros((B,), dtype=np.int64)
|
| 205 |
+
m = spatial_merge_size
|
| 206 |
+
for b in range(B):
|
| 207 |
+
is_img = ids[b] == image_token_id
|
| 208 |
+
chunks, cur, i = [], 0, 0
|
| 209 |
+
while i < T:
|
| 210 |
+
j = i
|
| 211 |
+
while j < T and is_img[j] == is_img[i]:
|
| 212 |
+
j += 1
|
| 213 |
+
if not is_img[i]:
|
| 214 |
+
chunks.append(np.broadcast_to(np.arange(j - i) + cur, (3, j - i)))
|
| 215 |
+
cur += j - i
|
| 216 |
+
else:
|
| 217 |
+
t, h, w = next(grids)
|
| 218 |
+
gt, gh, gw = t, h // m, w // m
|
| 219 |
+
if gt * gh * gw != j - i:
|
| 220 |
+
raise ValueError(f"image run of {j - i} tokens does not match grid {(t, h, w)}")
|
| 221 |
+
tt, hh, ww = np.meshgrid(np.arange(gt), np.arange(gh), np.arange(gw), indexing="ij")
|
| 222 |
+
chunks.append(np.stack([tt.reshape(-1) + cur, hh.reshape(-1) + cur, ww.reshape(-1) + cur]))
|
| 223 |
+
cur += max(h, w) // m
|
| 224 |
+
i = j
|
| 225 |
+
p = np.concatenate(chunks, axis=1)
|
| 226 |
+
pos[:, b] = p
|
| 227 |
+
deltas[b] = p.max() + 1 - T
|
| 228 |
+
return pos, deltas
|
| 229 |
+
|
| 230 |
+
|
| 231 |
+
class MRoPEContext:
|
| 232 |
+
"""Shared, per-model state for an image prompt: positions of the prompt and the decode delta (plain object, so
|
| 233 |
+
it is never part of the module's parameters)."""
|
| 234 |
+
|
| 235 |
+
def __init__(self, section=(11, 11, 10), interleaved=True):
|
| 236 |
+
self.section, self.interleaved = list(section), bool(interleaved)
|
| 237 |
+
self.position_ids = None # mx int32 [3, B, T] for the prompt
|
| 238 |
+
self.delta = None # mx int32 [B]
|
| 239 |
+
self.cos = self.sin = None # per-forward cos/sin [B, 1, L, rot_dims] (float32)
|
| 240 |
+
|
| 241 |
+
@property
|
| 242 |
+
def active(self):
|
| 243 |
+
return self.position_ids is not None
|
| 244 |
+
|
| 245 |
+
|
| 246 |
+
def _mrope_axis_map(half: int, section, interleaved: bool) -> np.ndarray:
|
| 247 |
+
axis = np.zeros((half,), dtype=np.int64)
|
| 248 |
+
if interleaved: # Qwen3_5TextRotaryEmbedding.recomposition_frequencies
|
| 249 |
+
for d, off in ((1, 1), (2, 2)):
|
| 250 |
+
axis[off : section[d] * 3 : 3] = d
|
| 251 |
+
else:
|
| 252 |
+
bounds = np.cumsum([0] + list(section))
|
| 253 |
+
for d in range(3):
|
| 254 |
+
axis[bounds[d] : bounds[d + 1]] = d
|
| 255 |
+
return axis
|
| 256 |
+
|
| 257 |
+
|
| 258 |
+
class MRoPEAttention(q35.Attention):
|
| 259 |
+
"""mlx-lm Qwen3NextAttention with an explicit-position rope when an image prompt is active (body adapted from
|
| 260 |
+
mlx-lm, MIT). Inactive -> the parent __call__ unchanged."""
|
| 261 |
+
|
| 262 |
+
_mrope_ctx: Optional[MRoPEContext] = None
|
| 263 |
+
|
| 264 |
+
def __call__(self, x: mx.array, mask=None, cache=None) -> mx.array:
|
| 265 |
+
ctx = self._mrope_ctx
|
| 266 |
+
if ctx is None or ctx.cos is None:
|
| 267 |
+
return super().__call__(x, mask, cache)
|
| 268 |
+
B, L, D = x.shape
|
| 269 |
+
queries, gate = mx.split(self.q_proj(x).reshape(B, L, self.num_attention_heads, -1), 2, axis=-1)
|
| 270 |
+
gate = gate.reshape(B, L, -1)
|
| 271 |
+
keys, values = self.k_proj(x), self.v_proj(x)
|
| 272 |
+
queries = self.q_norm(queries).transpose(0, 2, 1, 3)
|
| 273 |
+
keys = self.k_norm(keys.reshape(B, L, self.num_key_value_heads, -1)).transpose(0, 2, 1, 3)
|
| 274 |
+
values = values.reshape(B, L, self.num_key_value_heads, -1).transpose(0, 2, 1, 3)
|
| 275 |
+
queries = _apply_mrope(queries, ctx.cos, ctx.sin)
|
| 276 |
+
keys = _apply_mrope(keys, ctx.cos, ctx.sin)
|
| 277 |
+
if cache is not None:
|
| 278 |
+
keys, values = cache.update_and_fetch(keys, values)
|
| 279 |
+
output = scaled_dot_product_attention(queries, keys, values, cache=cache, scale=self.scale, mask=mask)
|
| 280 |
+
output = output.transpose(0, 2, 1, 3).reshape(B, L, -1)
|
| 281 |
+
return self.o_proj(output * mx.sigmoid(gate))
|
| 282 |
+
|
| 283 |
+
|
| 284 |
+
def _apply_mrope(x, cos, sin):
|
| 285 |
+
"""Non-traditional (rotate-half) rope on the first cos.shape[-1] dims; yarn mscale is already folded into cos/sin."""
|
| 286 |
+
rd = cos.shape[-1]
|
| 287 |
+
xr = x[..., :rd].astype(mx.float32)
|
| 288 |
+
h = rd // 2
|
| 289 |
+
rot = mx.concatenate([-xr[..., h:], xr[..., :h]], axis=-1)
|
| 290 |
+
out = (xr * cos + rot * sin).astype(x.dtype)
|
| 291 |
+
return mx.concatenate([out, x[..., rd:]], axis=-1) if rd < x.shape[-1] else out
|
| 292 |
+
|
| 293 |
+
|
| 294 |
+
def _rope_inv_freq_and_scale(rope) -> Tuple[mx.array, float]:
|
| 295 |
+
"""Inverse frequencies and output scale of an mlx-lm rope module (YarnRoPE / nn.RoPE)."""
|
| 296 |
+
if hasattr(rope, "_freqs"):
|
| 297 |
+
inv = 1.0 / rope._freqs.astype(mx.float32)
|
| 298 |
+
else: # nn.RoPE: theta_i = scale * pos * base^(-2i/dims)
|
| 299 |
+
dims, base, scale = rope.dims, rope.base, getattr(rope, "scale", 1.0)
|
| 300 |
+
inv = scale / (base ** (mx.arange(0, dims, 2, dtype=mx.float32) / dims))
|
| 301 |
+
return inv, float(getattr(rope, "mscale", 1.0))
|
| 302 |
+
|
| 303 |
+
|
| 304 |
+
class _CastRMSNorm(nn.RMSNorm):
|
| 305 |
+
"""RMSNorm whose output keeps the input dtype. With bf16 weights it is identical to nn.RMSNorm; with fp32 weights
|
| 306 |
+
(the checkpoint's 1 + w kept in float32, as Transformers computes it) the product is formed in fp32 and cast back."""
|
| 307 |
+
|
| 308 |
+
def __call__(self, x):
|
| 309 |
+
return mx.fast.rms_norm(x, self.weight, self.eps).astype(x.dtype)
|
| 310 |
+
|
| 311 |
+
|
| 312 |
+
def _use_cast_norms(module) -> None:
|
| 313 |
+
for _, m in module.named_modules():
|
| 314 |
+
if type(m) is nn.RMSNorm:
|
| 315 |
+
m.__class__ = _CastRMSNorm
|
| 316 |
+
|
| 317 |
+
|
| 318 |
class SenseNovaLoopTextModel(nn.Module):
|
| 319 |
def __init__(self, args: q35.TextModelArgs, loop: dict, mhc: dict):
|
| 320 |
super().__init__()
|
|
|
|
| 329 |
self.plan = execution_plan(args.num_hidden_layers, loop, mhc)
|
| 330 |
self.fa_visit = next((e for e, p in enumerate(self.plan) if not self.layers[p].is_linear), None)
|
| 331 |
self.ssm_visit = next((e for e, p in enumerate(self.plan) if self.layers[p].is_linear), None)
|
| 332 |
+
# image input: one shared context; the attention class swap adds no parameters
|
| 333 |
+
self.collect_hidden = None # parity diagnostics: set to a list to receive the embedding + every visit's output
|
| 334 |
+
rp = args.rope_scaling or {}
|
| 335 |
+
self.mrope = MRoPEContext(rp.get("mrope_section", [11, 11, 10]), rp.get("mrope_interleaved", True))
|
| 336 |
+
for layer in self.layers:
|
| 337 |
+
if not layer.is_linear:
|
| 338 |
+
layer.self_attn.__class__ = MRoPEAttention
|
| 339 |
+
layer.self_attn._mrope_ctx = self.mrope
|
| 340 |
+
|
| 341 |
+
def _mrope_tables(self, B: int, L: int, offset: int):
|
| 342 |
+
"""cos/sin for positions [offset, offset + L): prompt positions where stored, offset + delta past the prompt."""
|
| 343 |
+
ctx = self.mrope
|
| 344 |
+
T = ctx.position_ids.shape[-1]
|
| 345 |
+
parts = []
|
| 346 |
+
if offset < T:
|
| 347 |
+
parts.append(ctx.position_ids[:, :, offset : min(offset + L, T)])
|
| 348 |
+
if offset + L > T:
|
| 349 |
+
start = max(offset, T)
|
| 350 |
+
tail = mx.arange(start, offset + L, dtype=mx.int32)[None, :] + ctx.delta[:, None] # [B, n]
|
| 351 |
+
parts.append(mx.broadcast_to(tail[None], (3, *tail.shape)))
|
| 352 |
+
pos = parts[0] if len(parts) == 1 else mx.concatenate(parts, axis=-1)
|
| 353 |
+
if pos.shape[1] != B:
|
| 354 |
+
pos = mx.broadcast_to(pos, (3, B, L))
|
| 355 |
+
rope = self.layers[self.plan[self.fa_visit]].self_attn.rope
|
| 356 |
+
inv, mscale = _rope_inv_freq_and_scale(rope)
|
| 357 |
+
half = inv.shape[0]
|
| 358 |
+
axis = mx.array(_mrope_axis_map(half, ctx.section, ctx.interleaved))
|
| 359 |
+
sel = mx.take(pos.astype(mx.float32), axis, axis=0) # [half, B, L]
|
| 360 |
+
ang = sel.transpose(1, 2, 0) * inv # [B, L, half]
|
| 361 |
+
ang = mx.concatenate([ang, ang], axis=-1)[:, None] # [B, 1, L, rot]
|
| 362 |
+
return mx.cos(ang) * mscale, mx.sin(ang) * mscale
|
| 363 |
|
| 364 |
def __call__(self, inputs: mx.array, cache=None, input_embeddings: Optional[mx.array] = None) -> mx.array:
|
| 365 |
h = input_embeddings if input_embeddings is not None else self.embed_tokens(inputs)
|
|
|
|
| 369 |
raise ValueError(f"cache has {len(cache)} slots, the execution plan needs {len(self.plan)}")
|
| 370 |
fa_mask = create_attention_mask(h, cache[self.fa_visit]) if self.fa_visit is not None else None
|
| 371 |
ssm_mask = create_ssm_mask(h, cache[self.ssm_visit]) if self.ssm_visit is not None else None
|
| 372 |
+
if self.mrope.active and self.fa_visit is not None:
|
| 373 |
+
c0 = cache[self.fa_visit]
|
| 374 |
+
self.mrope.cos, self.mrope.sin = self._mrope_tables(h.shape[0], h.shape[1], int(c0.offset) if c0 is not None else 0)
|
| 375 |
+
col = self.collect_hidden
|
| 376 |
+
if col is not None:
|
| 377 |
+
col.append(h)
|
| 378 |
+
try:
|
| 379 |
+
for physical, c in zip(self.plan, cache):
|
| 380 |
+
layer = self.layers[physical]
|
| 381 |
+
h = layer(h, mask=ssm_mask if layer.is_linear else fa_mask, cache=c)
|
| 382 |
+
if col is not None:
|
| 383 |
+
col.append(h)
|
| 384 |
+
finally:
|
| 385 |
+
self.mrope.cos = self.mrope.sin = None
|
| 386 |
return self.norm(h)
|
| 387 |
|
| 388 |
|
|
|
|
| 417 |
self.args = args
|
| 418 |
self.model_type = args.model_type
|
| 419 |
self.language_model = TextModel(q35.TextModelArgs.from_dict(args.text_config), args.loop_settings, args.mhc)
|
| 420 |
+
_use_cast_norms(self.language_model)
|
| 421 |
|
| 422 |
def __call__(self, inputs: mx.array, cache=None, input_embeddings: Optional[mx.array] = None):
|
| 423 |
return self.language_model(inputs, cache=cache, input_embeddings=input_embeddings)
|
| 424 |
|
| 425 |
+
# ---- image input (the vision tower lives outside this module: sensenova_mlx/vision.py) ----
|
| 426 |
+
def set_mrope(self, position_ids, rope_deltas):
|
| 427 |
+
"""Activate 3-axis positions for the prompt about to be prefilled (from get_rope_index) and the decode delta.
|
| 428 |
+
Stays active for the following decode steps; call clear_mrope() before an unrelated prompt."""
|
| 429 |
+
ctx = self.language_model.model.mrope
|
| 430 |
+
ctx.position_ids = mx.array(np.asarray(position_ids), dtype=mx.int32)
|
| 431 |
+
ctx.delta = mx.array(np.asarray(rope_deltas).reshape(-1), dtype=mx.int32)
|
| 432 |
+
|
| 433 |
+
def clear_mrope(self):
|
| 434 |
+
ctx = self.language_model.model.mrope
|
| 435 |
+
ctx.position_ids = ctx.delta = ctx.cos = ctx.sin = None
|
| 436 |
+
|
| 437 |
+
@property
|
| 438 |
+
def rope_deltas(self):
|
| 439 |
+
return self.language_model.model.mrope.delta
|
| 440 |
+
|
| 441 |
+
def embed_with_images(self, input_ids: mx.array, image_features: Optional[mx.array]) -> mx.array:
|
| 442 |
+
"""Token embeddings with the merged vision features scattered over <|image_pad|> positions, in order
|
| 443 |
+
(HF masked_scatter)."""
|
| 444 |
+
h = self.language_model.model.embed_tokens(input_ids)
|
| 445 |
+
if image_features is None:
|
| 446 |
+
return h
|
| 447 |
+
mask = input_ids == self.args.image_token_id
|
| 448 |
+
n = int(mask.sum().item())
|
| 449 |
+
if n != image_features.shape[0]:
|
| 450 |
+
raise ValueError(f"{n} image tokens in the prompt, {image_features.shape[0]} image features")
|
| 451 |
+
idx = mx.maximum(mx.cumsum(mask.reshape(-1).astype(mx.int32)) - 1, 0).reshape(mask.shape)
|
| 452 |
+
feats = image_features.astype(h.dtype)[idx]
|
| 453 |
+
return mx.where(mask[..., None], feats, h)
|
| 454 |
+
|
| 455 |
+
def prepare_image_prompt(self, input_ids, image_features: Optional[mx.array], image_grid_thw,
|
| 456 |
+
spatial_merge_size: int = 2) -> mx.array:
|
| 457 |
+
"""input_ids [B, T] (+ features from the vision tower) -> input_embeddings; also activates mRoPE positions.
|
| 458 |
+
A prompt without image tokens clears mRoPE and returns plain embeddings (text path unchanged)."""
|
| 459 |
+
ids_np = np.asarray(input_ids)
|
| 460 |
+
if ids_np.ndim == 1:
|
| 461 |
+
ids_np = ids_np[None]
|
| 462 |
+
ids = mx.array(ids_np)
|
| 463 |
+
if image_features is None or not (ids_np == self.args.image_token_id).any():
|
| 464 |
+
self.clear_mrope()
|
| 465 |
+
return self.language_model.model.embed_tokens(ids)
|
| 466 |
+
pos, deltas = get_rope_index(ids_np, image_grid_thw, self.args.image_token_id, spatial_merge_size,
|
| 467 |
+
self.args.video_token_id)
|
| 468 |
+
self.set_mrope(pos, deltas)
|
| 469 |
+
return self.embed_with_images(ids, image_features)
|
| 470 |
+
|
| 471 |
@property
|
| 472 |
def model(self):
|
| 473 |
return self.language_model.model
|
|
|
|
| 477 |
# +1 on norms for raw HF checkpoints. Routers stay fp32 (the reference refuses to run them in bf16).
|
| 478 |
# The release checkpoint also carries `loop_gate.logits`, which the reference forward never reads: drop it.
|
| 479 |
weights = {k: v for k, v in weights.items() if "loop_gate" not in k}
|
| 480 |
+
# optional float32 norm weights (1 + w) shipped as `fp32_norm.<final name>`; they replace the bf16 copies
|
| 481 |
+
fp32_norms = {k[len("fp32_norm."):]: v.astype(mx.float32) for k, v in weights.items() if k.startswith("fp32_norm.")}
|
| 482 |
+
weights = {k: v for k, v in weights.items() if not k.startswith("fp32_norm.")}
|
| 483 |
weights = q35.Model.sanitize(self, weights)
|
| 484 |
+
weights.update(fp32_norms)
|
| 485 |
return {k: (v.astype(mx.float32) if (".mhc_attn." in k or ".mhc_ffn." in k) else v) for k, v in weights.items()}
|
| 486 |
|
| 487 |
@property
|
vision.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:0304cc30f617f782932bbdd102d3be477215c72cd446be233783a31711938c77
|
| 3 |
+
size 912058038
|