notforu777 commited on
Commit
39fabff
·
verified ·
1 Parent(s): a3a0990

model card from results.json

Browse files
Files changed (1) hide show
  1. README.md +215 -0
README.md ADDED
@@ -0,0 +1,215 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: sensenova/SenseNova-Loop-9B
3
+ base_model_relation: quantized
4
+ quantized_by: AtomicChat
5
+ pipeline_tag: image-text-to-text
6
+ library_name: gguf
7
+ license: apache-2.0
8
+ tags:
9
+ - gguf
10
+ - llama.cpp
11
+ - sensenova
12
+ - imatrix
13
+ - quantized
14
+ - conversational
15
+ - atomic-chat
16
+ ---
17
+
18
+ # How to Run SenseNova Loop 9B Locally
19
+ <p style="margin-top: 0; margin-bottom: 0;">
20
+ <em>Built from sensenova's original weights with our own importance matrix. The <a href="https://huggingface.co/datasets/AtomicChat/calib-corpora">calibration corpora</a> behind our builds are public.</em>
21
+ </p>
22
+ <div style="display: flex; gap: 8px; align-items: center; margin-top: 10px; margin-bottom: 10px;">
23
+ <a href="https://atomic.chat/?utm_source=huggingface&utm_medium=referral&utm_campaign=hf_sensenova_loop_9b&utm_content=btn_atomic"><img src="https://huggingface.co/AtomicChat/Qwen3.8-27B-GGUF/resolve/main/btn_atomic.png" width="162" alt="Atomic Chat"></a>
24
+ <a href="https://discord.gg/8wGSsvmg4V"><img src="https://huggingface.co/AtomicChat/Qwen3.8-27B-GGUF/resolve/main/btn_discord.png" width="119" alt="Discord"></a>
25
+ <a href="https://github.com/AtomicBot-ai/Atomic-Chat"><img src="https://huggingface.co/AtomicChat/Qwen3.8-27B-GGUF/resolve/main/btn_github.png" width="115" alt="GitHub"></a>
26
+ <a href="https://x.com/atomic_chat_hq"><img src="https://huggingface.co/AtomicChat/Qwen3.8-27B-GGUF/resolve/main/btn_x.png" width="108" alt="X"></a>
27
+ </div>
28
+ <ul style="margin: 0 0 12px 0;">
29
+ <!-- TODO(editorial): guide bullet: link to the atomic.chat blog guide for this model, if there is one -->
30
+ <li>Built by the team behind <a href="https://atomic.chat/?utm_source=huggingface&utm_medium=referral&utm_campaign=hf_sensenova_loop_9b&utm_content=bullet_app">Atomic Chat</a>, the app that runs open-weight models locally and offline.</li>
31
+ <li>See our quantization analysis below for measurements and instructions.</li>
32
+ </ul>
33
+ <hr style="margin: 0 0 16px 0;">
34
+
35
+ <!-- TODO(editorial): the KLD-vs-size chart (ours and other publishers) and the sentence under it -->
36
+
37
+ ## Pick a file
38
+
39
+ Every number below is measured, not estimated. How we measured it is at the
40
+ bottom, and the raw logs are in
41
+ [the metrics repo](https://huggingface.co/datasets/AtomicChat/SenseNova-Loop-9B-GGUF-metrics)
42
+ so you can check any of it yourself.
43
+
44
+ `KL divergence` is how far the quantized model's predictions drift from the
45
+ original weights. Lower is better, and zero means identical. `top-1` is how
46
+ often it picks the same next word the original would have picked.
47
+
48
+ | File | Size | KL divergence | top-1 |
49
+ |---|---:|---:|---:|
50
+ | `AD-Q6_K` | 8.8 GB | 0.00920 | 96.79% |
51
+ | `AD-Q5_K_M` | 6.9 GB | 0.02308 | 94.13% |
52
+ | `AD-Q4_K_M` | 6.0 GB | 0.04451 | 91.33% |
53
+ | `AD-IQ4_XS` | 5.8 GB | 0.05350 | 90.55% |
54
+ | `AD-IQ3_S` | 4.9 GB | 0.13673 | 84.17% |
55
+ | `AD-IQ3_XXS` | 4.3 GB | 0.23782 | 79.13% |
56
+
57
+ `AD-` marks an **Atomic Dynamic** layout. The name says what the two largest
58
+ tensor groups got: `AD-<ffn_down>-<ffn_up>`, collapsed to one name when both
59
+ match. Nothing is named after a type it does not contain.
60
+
61
+ ## Which one fits your card
62
+
63
+ The file has to fit, and so does the context. This model keeps 32 KB of
64
+ attention cache per token, which is 0.2 GB at 8k context and 1.0 GB at 32k. Budget
65
+ for both.
66
+
67
+ | Your card | File | Leaves room for |
68
+ |---|---|---|
69
+ | 8 GB | `AD-Q5_K_M` | around 16k context |
70
+ | 12 GB | `AD-Q6_K` | around 88k context |
71
+ | 32 GB and up | `AD-Q6_K` | full context |
72
+
73
+ > [!TIP]
74
+ > If you are choosing between two neighbouring files, take the larger one. The
75
+ > steps between them cost one or two gigabytes and buy noticeably fewer wrong words.
76
+
77
+ ## Running it
78
+
79
+ ```bash
80
+ llama-server -m SenseNova-Loop-9B-AD-Q4_K_M.gguf -ngl 99 -c 8192
81
+ ```
82
+
83
+ The chat template is inside every file, llama.cpp applies it on its own.
84
+ <!-- TODO(editorial): prompt format block and the thinking toggle, if the family has one -->
85
+
86
+ The model ships a multi token prediction head. It is inside every file here
87
+ and needs no extra download:
88
+
89
+ ```bash
90
+ llama-cli -m SenseNova-Loop-9B-AD-Q4_K_M.gguf --spec-type draft-mtp -ngl 99 -c 8192
91
+ ```
92
+
93
+ ## Speed
94
+
95
+ <!-- TODO(editorial): llama-bench prompt and generation t/s at 8k and 32k for the recommended file -->
96
+
97
+ ## Images and video
98
+
99
+ This model reads images, not only text. That needs one extra file, the vision
100
+ projector, which is separate from the quant and shared by all of them:
101
+
102
+ - `mmproj-SenseNova-Loop-9B-F16.gguf` for most people
103
+
104
+ Download it once alongside whichever quant you picked:
105
+
106
+ ```bash
107
+ llama-mtmd-cli \
108
+ -m SenseNova-Loop-9B-AD-Q4_K_M.gguf \
109
+ --mmproj mmproj-SenseNova-Loop-9B-F16.gguf \
110
+ --image your-photo.jpg -ngl 99 -c 8192 \
111
+ -p "What is in this image?"
112
+ ```
113
+
114
+ <!-- TODO(editorial): family specific vision flags (e.g. --image-min-tokens) and the demo image test -->
115
+
116
+ ## How these compare to other builds
117
+
118
+ <!-- TODO(editorial): other publishers measured against the same reference (kld_ext), same-name table -->
119
+
120
+ ## What we found while building these
121
+
122
+ <!-- TODO(editorial): findings from this release: layout experiments, bands, groups that mattered -->
123
+
124
+ ## The calibration data
125
+
126
+ Our importance matrix comes from a corpus we built for this model and
127
+ published: [AtomicChat/calib-corpora](https://huggingface.co/datasets/AtomicChat/calib-corpora),
128
+ recipe `ornith-1.5-9b`. It is 4,958,490 tokens across 3,079 documents:
129
+
130
+ | Part | Share |
131
+ |---|---:|
132
+ | agentic and tool use | 24.7% |
133
+ | code | 17.8% |
134
+ | reasoning | 14.8% |
135
+ | multilingual | 13.8% |
136
+ | long context | 11.9% |
137
+ | vocabulary sweep | 9.9% |
138
+ | structured data | 3.9% |
139
+ | graphics | 3.0% |
140
+
141
+ Every conversation is rendered through **this model's own chat template**, so
142
+ the chat markup appears as the single tokens the model actually sees.
143
+
144
+ > [!IMPORTANT]
145
+ > If you build your own importance matrix from this corpus, pass
146
+ > `--parse-special` to `llama-imatrix`. Without it the chat markup is read as
147
+ > ordinary punctuation.
148
+
149
+ ## How we measured
150
+
151
+ Reference: the original BF16 weights, converted to GGUF and run unquantized.
152
+ Perplexity on the held-out set is 8.7987 plus or minus 0.05818.
153
+
154
+ Held-out text: `eval_neutral` from our calibration dataset, never used for
155
+ calibration. 87 chunks at 4096 context.
156
+
157
+ Metric: per-token KL divergence of each file's predictions against the
158
+ reference, plus top-1 agreement.
159
+
160
+ Hardware: 1x H100 PCIe.
161
+
162
+ The raw logits of the reference are published in the metrics repo, split into
163
+ parts because of the file size limit. With them you can measure your own build
164
+ against exactly the same point we did:
165
+
166
+ ```bash
167
+ cat base-neutral.kld.*.part > base-neutral.kld
168
+
169
+ llama-perplexity -m your-quant.gguf -f eval_neutral.txt \
170
+ --kl-divergence-base base-neutral.kld --kl-divergence -c 4096 -ngl 99
171
+ ```
172
+
173
+ ## Reproducing a file
174
+
175
+ The importance matrix was collected on the BF16 weights rather than on a
176
+ quantized copy, split across 8 workers and merged. Splitting by chunk
177
+ range gives the same result as one long run, because the statistic is a sum:
178
+
179
+ ```bash
180
+ llama-imatrix -m SenseNova-Loop-9B-BF16.gguf -f calib_train.txt -o shard-0.gguf \
181
+ -ngl 99 -c 512 -b 8192 -ub 8192 --parse-special --from-chunk 0 --chunks 1211
182
+
183
+ llama-imatrix -m SenseNova-Loop-9B-BF16.gguf --in-file shard-0.gguf,shard-1.gguf,... \
184
+ -o imatrix.gguf
185
+ ```
186
+
187
+ Then, for `AD-Q4_K_M`:
188
+
189
+ ```bash
190
+ llama-quantize \
191
+ --imatrix imatrix.gguf \
192
+ --tensor-type '^blk\.56\.=q5_k' \
193
+ --tensor-type '^blk\.(0|1|26|27|52|53|54|55)\.ffn_(down|gate|up)\.weight$=q5_k' \
194
+ --tensor-type '^blk\.(2|3|4|5)\.ffn_(down|gate|up)\.weight$=q5_k' \
195
+ --tensor-type '^blk\.\d+\.ffn_down\.weight$=q4_k' \
196
+ --tensor-type '^blk\.\d+\.ffn_gate\.weight$=q4_k' \
197
+ --tensor-type '^blk\.\d+\.ffn_up\.weight$=q4_k' \
198
+ --tensor-type '^blk\.\d+\.attn_q\.weight$=q4_k' \
199
+ --tensor-type '^blk\.\d+\.attn_qkv_gate\.weight$=q5_k' \
200
+ --tensor-type '^blk\.\d+\.ssm_out\.weight$=q5_k' \
201
+ --tensor-type '^output\.weight$=q6_k' \
202
+ --tensor-type '^blk\.\d+\.attn_output\.weight$=q6_k' \
203
+ --tensor-type '^token_embd\.weight$=iq4_xs' \
204
+ --override-kv general.file_type=int:15 \
205
+ SenseNova-Loop-9B-BF16.gguf SenseNova-Loop-9B-AD-Q4_K_M.gguf Q8_0
206
+ ```
207
+
208
+ The rules for every other file are in `ladder/` in the metrics repo, next to
209
+ its quantization log. The first rule that matches a tensor wins.
210
+
211
+ ## Model details
212
+
213
+ 56 layers, hidden size 4096, feed-forward size 12288, vocabulary 248,320, context 524,288. One multi token prediction head.
214
+
215
+ Needs a recent llama.cpp with `sensenova_loop` architecture support. Built and tested with llama.cpp b12467 (24ea7bb78).