Hugging Face's logo

Tensor-compression
/
TR_main

tensor-decomposition
svd
low-rank
model-compression
knowledge-distillation
Model card Files
xet
 Community
Rhett1 commited on
Commit
62ed893
·
verified ·
1 Parent(s): 6aa9bb9

reorg: remove superseded path llama2-7b/

Browse files
Files changed (5) hide show
  1. llama2-7b/README.md +0 -119
  2. llama2-7b/config.json +0 -32
  3. llama2-7b/ranks.json +0 -2690
  4. llama2-7b/state_dict_fp16_healed.pt +0 -3
  5. llama2-7b/tensor_formats.py +0 -761
llama2-7b/README.md DELETED
@@ -1,119 +0,0 @@
1
- ---
2
- license: llama2
3
- base_model: NousResearch/Llama-2-7b-chat-hf
4
- tags:
5
- - tensor-decomposition
6
- - tensor-ring
7
- - model-compression
8
- - llama2
9
- ---
10
-
11
- # TR-3: LLaMA-2-7B-chat compressed with a Tensor Ring
12
-
13
- LLaMA-2-7B-chat with every attention/MLP projection replaced by a **3-site Tensor Ring**
14
- (a cyclic tensor-network factorization, not a train/TT — the ring closes with a non-trivial
15
- boundary bond) plus a 5% sparse residual, then healed with pure knowledge distillation.
16
-
17
- - **4.02B params** (down from 6.74B, ~40% reduction)
18
- - **60M healing tokens** (0.228 epoch, pure forward-KL distillation from the dense 7B teacher)
19
- - Full five-task evaluation (lm-eval 0.4.12, full benchmark, not a subsample):
20
-
21
- | Task (shots) | This model | Dense 7B teacher |
22
- |---|---:|---:|
23
- | MMLU (5-shot) | **43.21%** | 47.22% |
24
- | HellaSwag (10-shot) | **70.75%** | 78.59% |
25
- | BoolQ (0-shot) | **77.09%** | 80.58% |
26
-
27
-
28
- ## ⚠️ This is NOT a standard `transformers` checkpoint
29
-
30
- The weight tensors in `state_dict_fp16_healed.pt` are **ring cores**, not dense `nn.Linear` weights —
31
- loading them requires reconstructing the model with the custom `TRLinear`/`SparseAugTRLinear`
32
- layers from `tensor_formats.py` (included in this repo), using `ranks.json` for the per-layer
33
- topology. `config.json` is included only as architecture metadata (it is the stock
34
- LLaMA-2-7B-chat config) — you cannot `AutoModelForCausalLM.from_pretrained()` this repo
35
- directly.
36
-
37
- ## How to load it
38
-
39
- ```python
40
- import json
41
- import torch
42
- from huggingface_hub import hf_hub_download
43
- from transformers import AutoModelForCausalLM, AutoTokenizer
44
-
45
- REPO = "<your-username>/<this-repo-name>"
46
-
47
- ranks = json.loads(open(hf_hub_download(REPO, "ranks.json")).read())
48
- state_dict_path = hf_hub_download(REPO, "state_dict_fp16_healed.pt")
49
- tensor_formats_path = hf_hub_download(REPO, "tensor_formats.py")
50
-
51
- # tensor_formats.py has no local package dependencies -- exec it into a namespace
52
- ns = {}
53
- exec(open(tensor_formats_path).read(), ns)
54
- TRLinear, SparseAugTRLinear = ns["TRLinear"], ns["SparseAugTRLinear"]
55
-
56
- def set_sub(model, path, new):
57
- parts = path.split(".")
58
- parent = model
59
- for p in parts[:-1]:
60
- parent = getattr(parent, p)
61
- setattr(parent, parts[-1], new)
62
-
63
- model = AutoModelForCausalLM.from_pretrained(
64
- "NousResearch/Llama-2-7b-chat-hf",
65
- revision="351844e75ed0bcbbe3f10671b3c808d2b83894ee",
66
- torch_dtype=torch.float16,
67
- )
68
- for path, spec in ranks.items():
69
- osp = (spec["o1"], spec["o2"], spec["o3"])
70
- isp = (spec["i1"], spec["i2"], spec["i3"])
71
- rr = (spec["r0"], spec["r1"], spec["r2"])
72
- dims = [o * i for o, i in zip(osp, isp)]
73
- cores = [torch.zeros(rr[k], dims[k], rr[(k + 1) % 3], dtype=torch.float16) for k in range(3)]
74
- nnz = spec.get("nnz", 0)
75
- layer = (SparseAugTRLinear(cores, osp, isp,
76
- torch.zeros(nnz, dtype=torch.int64),
77
- torch.zeros(nnz, dtype=torch.float16))
78
- if nnz else TRLinear(cores, osp, isp))
79
- set_sub(model, path, layer)
80
-
81
- sd = torch.load(state_dict_path, map_location="cpu")
82
- missing, unexpected = model.load_state_dict(
83
- {k: (v.to(torch.float16) if v.is_floating_point() else v) for k, v in sd.items()},
84
- strict=False)
85
- assert not missing and not unexpected
86
-
87
- # generation-heavy inference (e.g. batch decoding) is 100x+ faster after this --
88
- # forward() otherwise redoes the full ring contraction on every call. Never call
89
- # before/during training: it detaches the cache from the ring cores' gradients.
90
- for m in model.modules():
91
- if isinstance(m, TRLinear):
92
- m.materialize_()
93
-
94
- tok = AutoTokenizer.from_pretrained("NousResearch/Llama-2-7b-chat-hf")
95
- ```
96
-
97
- ## Files
98
-
99
- | File | What it is |
100
- |---|---|
101
- | `state_dict_fp16_healed.pt` | Healed weights (ring-core form) |
102
- | `ranks.json` | Per-layer ring topology (output/input mode splits + bond ranks + sparse-residual nnz) |
103
- | `tensor_formats.py` | `TRLinear`/`SparseAugTRLinear` layer implementations + the ring contraction/fitting code |
104
- | `config.json` | Stock LLaMA-2-7B-chat architecture config, for reference only |
105
-
106
- ## Provenance / method
107
-
108
- Compression: 3-site tensor ring, TT-SVD initialization + ALS refinement (multi-restart, never
109
- returns worse than its own TT baseline), diagonal 3-way input-activation whitening, 5% sparse
110
- residual (activation-weighted magnitude selection). Healing: pure forward-KL distillation
111
- (alpha=1.0, T=1.0), lr=2e-5, 60M tokens on a decontaminated alpaca/openhermes/ultrachat mixture.
112
- Full method code and the complete experiment record (why this format was chosen over
113
- TT/matrix-low-rank alternatives) are not included in this repo.
114
-
115
- ## License
116
-
117
- Derived from Meta's LLaMA 2 (`NousResearch/Llama-2-7b-chat-hf`, an unofficial redistribution of
118
- the same weights). Subject to the **LLaMA 2 Community License** — see
119
- https://ai.meta.com/llama/license/. This repo does not grant any rights beyond that license.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
llama2-7b/config.json DELETED
@@ -1,32 +0,0 @@
1
- {
2
- "_name_or_path": "NousResearch/Llama-2-7b-chat-hf",
3
- "architectures": [
4
- "LlamaForCausalLM"
5
- ],
6
- "attention_bias": false,
7
- "attention_dropout": 0.0,
8
- "bos_token_id": 1,
9
- "eos_token_id": 2,
10
- "head_dim": 128,
11
- "hidden_act": "silu",
12
- "hidden_size": 4096,
13
- "initializer_range": 0.02,
14
- "intermediate_size": 11008,
15
- "max_position_embeddings": 4096,
16
- "mlp_bias": false,
17
- "model_type": "llama",
18
- "num_attention_heads": 32,
19
- "num_hidden_layers": 32,
20
- "num_key_value_heads": 32,
21
- "pad_token_id": 0,
22
- "pretraining_tp": 1,
23
- "rms_norm_eps": 1e-05,
24
- "rope_scaling": null,
25
- "rope_theta": 10000.0,
26
- "tie_word_embeddings": false,
27
- "torch_dtype": "float16",
28
- "transformers_version": "4.46.3",
29
- "use_cache": true,
30
- "vocab_size": 32000
31
- }
32
-
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
llama2-7b/ranks.json DELETED
@@ -1,2690 +0,0 @@
1
- {
2
- "model.layers.0.self_attn.q_proj": {
3
- "o1": 16,
4
- "o2": 16,
5
- "o3": 16,
6
- "i1": 16,
7
- "i2": 16,
8
- "i3": 16,
9
- "r0": 108,
10
- "r1": 108,
11
- "r2": 108,
12
- "nnz": 838860
13
- },
14
- "model.layers.0.self_attn.k_proj": {
15
- "o1": 16,
16
- "o2": 16,
17
- "o3": 16,
18
- "i1": 16,
19
- "i2": 16,
20
- "i3": 16,
21
- "r0": 108,
22
- "r1": 108,
23
- "r2": 108,
24
- "nnz": 838860
25
- },
26
- "model.layers.0.self_attn.v_proj": {
27
- "o1": 16,
28
- "o2": 16,
29
- "o3": 16,
30
- "i1": 16,
31
- "i2": 16,
32
- "i3": 16,
33
- "r0": 108,
34
- "r1": 108,
35
- "r2": 108,
36
- "nnz": 838860
37
- },
38
- "model.layers.0.self_attn.o_proj": {
39
- "o1": 16,
40
- "o2": 16,
41
- "o3": 16,
42
- "i1": 16,
43
- "i2": 16,
44
- "i3": 16,
45
- "r0": 108,
46
- "r1": 108,
47
- "r2": 108,
48
- "nnz": 838860
49
- },
50
- "model.layers.0.mlp.gate_proj": {
51
- "o1": 16,
52
- "o2": 16,
53
- "o3": 43,
54
- "i1": 16,
55
- "i2": 16,
56
- "i3": 16,
57
- "r0": 141,
58
- "r1": 141,
59
- "r2": 141,
60
- "nnz": 2254438
61
- },
62
- "model.layers.0.mlp.up_proj": {
63
- "o1": 16,
64
- "o2": 16,
65
- "o3": 43,
66
- "i1": 16,
67
- "i2": 16,
68
- "i3": 16,
69
- "r0": 141,
70
- "r1": 141,
71
- "r2": 141,
72
- "nnz": 2254438
73
- },
74
- "model.layers.0.mlp.down_proj": {
75
- "o1": 16,
76
- "o2": 16,
77
- "o3": 16,
78
- "i1": 16,
79
- "i2": 16,
80
- "i3": 43,
81
- "r0": 141,
82
- "r1": 141,
83
- "r2": 141,
84
- "nnz": 2254438
85
- },
86
- "model.layers.1.self_attn.q_proj": {
87
- "o1": 16,
88
- "o2": 16,
89
- "o3": 16,
90
- "i1": 16,
91
- "i2": 16,
92
- "i3": 16,
93
- "r0": 108,
94
- "r1": 108,
95
- "r2": 108,
96
- "nnz": 838860
97
- },
98
- "model.layers.1.self_attn.k_proj": {
99
- "o1": 16,
100
- "o2": 16,
101
- "o3": 16,
102
- "i1": 16,
103
- "i2": 16,
104
- "i3": 16,
105
- "r0": 108,
106
- "r1": 108,
107
- "r2": 108,
108
- "nnz": 838860
109
- },
110
- "model.layers.1.self_attn.v_proj": {
111
- "o1": 16,
112
- "o2": 16,
113
- "o3": 16,
114
- "i1": 16,
115
- "i2": 16,
116
- "i3": 16,
117
- "r0": 108,
118
- "r1": 108,
119
- "r2": 108,
120
- "nnz": 838860
121
- },
122
- "model.layers.1.self_attn.o_proj": {
123
- "o1": 16,
124
- "o2": 16,
125
- "o3": 16,
126
- "i1": 16,
127
- "i2": 16,
128
- "i3": 16,
129
- "r0": 108,
130
- "r1": 108,
131
- "r2": 108,
132
- "nnz": 838860
133
- },
134
- "model.layers.1.mlp.gate_proj": {
135
- "o1": 16,
136
- "o2": 16,
137
- "o3": 43,
138
- "i1": 16,
139
- "i2": 16,
140
- "i3": 16,
141
- "r0": 141,
142
- "r1": 141,
143
- "r2": 141,
144
- "nnz": 2254438
145
- },
146
- "model.layers.1.mlp.up_proj": {
147
- "o1": 16,
148
- "o2": 16,
149
- "o3": 43,
150
- "i1": 16,
151
- "i2": 16,
152
- "i3": 16,
153
- "r0": 141,
154
- "r1": 141,
155
- "r2": 141,
156
- "nnz": 2254438
157
- },
158
- "model.layers.1.mlp.down_proj": {
159
- "o1": 16,
160
- "o2": 16,
161
- "o3": 16,
162
- "i1": 16,
163
- "i2": 16,
164
- "i3": 43,
165
- "r0": 141,
166
- "r1": 141,
167
- "r2": 141,
168
- "nnz": 2254438
169
- },
170
- "model.layers.2.self_attn.q_proj": {
171
- "o1": 16,
172
- "o2": 16,
173
- "o3": 16,
174
- "i1": 16,
175
- "i2": 16,
176
- "i3": 16,
177
- "r0": 108,
178
- "r1": 108,
179
- "r2": 108,
180
- "nnz": 838860
181
- },
182
- "model.layers.2.self_attn.k_proj": {
183
- "o1": 16,
184
- "o2": 16,
185
- "o3": 16,
186
- "i1": 16,
187
- "i2": 16,
188
- "i3": 16,
189
- "r0": 108,
190
- "r1": 108,
191
- "r2": 108,
192
- "nnz": 838860
193
- },
194
- "model.layers.2.self_attn.v_proj": {
195
- "o1": 16,
196
- "o2": 16,
197
- "o3": 16,
198
- "i1": 16,
199
- "i2": 16,
200
- "i3": 16,
201
- "r0": 108,
202
- "r1": 108,
203
- "r2": 108,
204
- "nnz": 838860
205
- },
206
- "model.layers.2.self_attn.o_proj": {
207
- "o1": 16,
208
- "o2": 16,
209
- "o3": 16,
210
- "i1": 16,
211
- "i2": 16,
212
- "i3": 16,
213
- "r0": 108,
214
- "r1": 108,
215
- "r2": 108,
216
- "nnz": 838860
217
- },
218
- "model.layers.2.mlp.gate_proj": {
219
- "o1": 16,
220
- "o2": 16,
221
- "o3": 43,
222
- "i1": 16,
223
- "i2": 16,
224
- "i3": 16,
225
- "r0": 141,
226
- "r1": 141,
227
- "r2": 141,
228
- "nnz": 2254438
229
- },
230
- "model.layers.2.mlp.up_proj": {
231
- "o1": 16,
232
- "o2": 16,
233
- "o3": 43,
234
- "i1": 16,
235
- "i2": 16,
236
- "i3": 16,
237
- "r0": 141,
238
- "r1": 141,
239
- "r2": 141,
240
- "nnz": 2254438
241
- },
242
- "model.layers.2.mlp.down_proj": {
243
- "o1": 16,
244
- "o2": 16,
245
- "o3": 16,
246
- "i1": 16,
247
- "i2": 16,
248
- "i3": 43,
249
- "r0": 141,
250
- "r1": 141,
251
- "r2": 141,
252
- "nnz": 2254438
253
- },
254
- "model.layers.3.self_attn.q_proj": {
255
- "o1": 16,
256
- "o2": 16,
257
- "o3": 16,
258
- "i1": 16,
259
- "i2": 16,
260
- "i3": 16,
261
- "r0": 108,
262
- "r1": 108,
263
- "r2": 108,
264
- "nnz": 838860
265
- },
266
- "model.layers.3.self_attn.k_proj": {
267
- "o1": 16,
268
- "o2": 16,
269
- "o3": 16,
270
- "i1": 16,
271
- "i2": 16,
272
- "i3": 16,
273
- "r0": 108,
274
- "r1": 108,
275
- "r2": 108,
276
- "nnz": 838860
277
- },
278
- "model.layers.3.self_attn.v_proj": {
279
- "o1": 16,
280
- "o2": 16,
281
- "o3": 16,
282
- "i1": 16,
283
- "i2": 16,
284
- "i3": 16,
285
- "r0": 108,
286
- "r1": 108,
287
- "r2": 108,
288
- "nnz": 838860
289
- },
290
- "model.layers.3.self_attn.o_proj": {
291
- "o1": 16,
292
- "o2": 16,
293
- "o3": 16,
294
- "i1": 16,
295
- "i2": 16,
296
- "i3": 16,
297
- "r0": 108,
298
- "r1": 108,
299
- "r2": 108,
300
- "nnz": 838860
301
- },
302
- "model.layers.3.mlp.gate_proj": {
303
- "o1": 16,
304
- "o2": 16,
305
- "o3": 43,
306
- "i1": 16,
307
- "i2": 16,
308
- "i3": 16,
309
- "r0": 141,
310
- "r1": 141,
311
- "r2": 141,
312
- "nnz": 2254438
313
- },
314
- "model.layers.3.mlp.up_proj": {
315
- "o1": 16,
316
- "o2": 16,
317
- "o3": 43,
318
- "i1": 16,
319
- "i2": 16,
320
- "i3": 16,
321
- "r0": 141,
322
- "r1": 141,
323
- "r2": 141,
324
- "nnz": 2254438
325
- },
326
- "model.layers.3.mlp.down_proj": {
327
- "o1": 16,
328
- "o2": 16,
329
- "o3": 16,
330
- "i1": 16,
331
- "i2": 16,
332
- "i3": 43,
333
- "r0": 141,
334
- "r1": 141,
335
- "r2": 141,
336
- "nnz": 2254438
337
- },
338
- "model.layers.4.self_attn.q_proj": {
339
- "o1": 16,
340
- "o2": 16,
341
- "o3": 16,
342
- "i1": 16,
343
- "i2": 16,
344
- "i3": 16,
345
- "r0": 108,
346
- "r1": 108,
347
- "r2": 108,
348
- "nnz": 838860
349
- },
350
- "model.layers.4.self_attn.k_proj": {
351
- "o1": 16,
352
- "o2": 16,
353
- "o3": 16,
354
- "i1": 16,
355
- "i2": 16,
356
- "i3": 16,
357
- "r0": 108,
358
- "r1": 108,
359
- "r2": 108,
360
- "nnz": 838860
361
- },
362
- "model.layers.4.self_attn.v_proj": {
363
- "o1": 16,
364
- "o2": 16,
365
- "o3": 16,
366
- "i1": 16,
367
- "i2": 16,
368
- "i3": 16,
369
- "r0": 108,
370
- "r1": 108,
371
- "r2": 108,
372
- "nnz": 838860
373
- },
374
- "model.layers.4.self_attn.o_proj": {
375
- "o1": 16,
376
- "o2": 16,
377
- "o3": 16,
378
- "i1": 16,
379
- "i2": 16,
380
- "i3": 16,
381
- "r0": 108,
382
- "r1": 108,
383
- "r2": 108,
384
- "nnz": 838860
385
- },
386
- "model.layers.4.mlp.gate_proj": {
387
- "o1": 16,
388
- "o2": 16,
389
- "o3": 43,
390
- "i1": 16,
391
- "i2": 16,
392
- "i3": 16,
393
- "r0": 141,
394
- "r1": 141,
395
- "r2": 141,
396
- "nnz": 2254438
397
- },
398
- "model.layers.4.mlp.up_proj": {
399
- "o1": 16,
400
- "o2": 16,
401
- "o3": 43,
402
- "i1": 16,
403
- "i2": 16,
404
- "i3": 16,
405
- "r0": 141,
406
- "r1": 141,
407
- "r2": 141,
408
- "nnz": 2254438
409
- },
410
- "model.layers.4.mlp.down_proj": {
411
- "o1": 16,
412
- "o2": 16,
413
- "o3": 16,
414
- "i1": 16,
415
- "i2": 16,
416
- "i3": 43,
417
- "r0": 141,
418
- "r1": 141,
419
- "r2": 141,
420
- "nnz": 2254438
421
- },
422
- "model.layers.5.self_attn.q_proj": {
423
- "o1": 16,
424
- "o2": 16,
425
- "o3": 16,
426
- "i1": 16,
427
- "i2": 16,
428
- "i3": 16,
429
- "r0": 108,
430
- "r1": 108,
431
- "r2": 108,
432
- "nnz": 838860
433
- },
434
- "model.layers.5.self_attn.k_proj": {
435
- "o1": 16,
436
- "o2": 16,
437
- "o3": 16,
438
- "i1": 16,
439
- "i2": 16,
440
- "i3": 16,
441
- "r0": 108,
442
- "r1": 108,
443
- "r2": 108,
444
- "nnz": 838860
445
- },
446
- "model.layers.5.self_attn.v_proj": {
447
- "o1": 16,
448
- "o2": 16,
449
- "o3": 16,
450
- "i1": 16,
451
- "i2": 16,
452
- "i3": 16,
453
- "r0": 108,
454
- "r1": 108,
455
- "r2": 108,
456
- "nnz": 838860
457
- },
458
- "model.layers.5.self_attn.o_proj": {
459
- "o1": 16,
460
- "o2": 16,
461
- "o3": 16,
462
- "i1": 16,
463
- "i2": 16,
464
- "i3": 16,
465
- "r0": 108,
466
- "r1": 108,
467
- "r2": 108,
468
- "nnz": 838860
469
- },
470
- "model.layers.5.mlp.gate_proj": {
471
- "o1": 16,
472
- "o2": 16,
473
- "o3": 43,
474
- "i1": 16,
475
- "i2": 16,
476
- "i3": 16,
477
- "r0": 141,
478
- "r1": 141,
479
- "r2": 141,
480
- "nnz": 2254438
481
- },
482
- "model.layers.5.mlp.up_proj": {
483
- "o1": 16,
484
- "o2": 16,
485
- "o3": 43,
486
- "i1": 16,
487
- "i2": 16,
488
- "i3": 16,
489
- "r0": 141,
490
- "r1": 141,
491
- "r2": 141,
492
- "nnz": 2254438
493
- },
494
- "model.layers.5.mlp.down_proj": {
495
- "o1": 16,
496
- "o2": 16,
497
- "o3": 16,
498
- "i1": 16,
499
- "i2": 16,
500
- "i3": 43,
501
- "r0": 141,
502
- "r1": 141,
503
- "r2": 141,
504
- "nnz": 2254438
505
- },
506
- "model.layers.6.self_attn.q_proj": {
507
- "o1": 16,
508
- "o2": 16,
509
- "o3": 16,
510
- "i1": 16,
511
- "i2": 16,
512
- "i3": 16,
513
- "r0": 108,
514
- "r1": 108,
515
- "r2": 108,
516
- "nnz": 838860
517
- },
518
- "model.layers.6.self_attn.k_proj": {
519
- "o1": 16,
520
- "o2": 16,
521
- "o3": 16,
522
- "i1": 16,
523
- "i2": 16,
524
- "i3": 16,
525
- "r0": 108,
526
- "r1": 108,
527
- "r2": 108,
528
- "nnz": 838860
529
- },
530
- "model.layers.6.self_attn.v_proj": {
531
- "o1": 16,
532
- "o2": 16,
533
- "o3": 16,
534
- "i1": 16,
535
- "i2": 16,
536
- "i3": 16,
537
- "r0": 108,
538
- "r1": 108,
539
- "r2": 108,
540
- "nnz": 838860
541
- },
542
- "model.layers.6.self_attn.o_proj": {
543
- "o1": 16,
544
- "o2": 16,
545
- "o3": 16,
546
- "i1": 16,
547
- "i2": 16,
548
- "i3": 16,
549
- "r0": 108,
550
- "r1": 108,
551
- "r2": 108,
552
- "nnz": 838860
553
- },
554
- "model.layers.6.mlp.gate_proj": {
555
- "o1": 16,
556
- "o2": 16,
557
- "o3": 43,
558
- "i1": 16,
559
- "i2": 16,
560
- "i3": 16,
561
- "r0": 141,
562
- "r1": 141,
563
- "r2": 141,
564
- "nnz": 2254438
565
- },
566
- "model.layers.6.mlp.up_proj": {
567
- "o1": 16,
568
- "o2": 16,
569
- "o3": 43,
570
- "i1": 16,
571
- "i2": 16,
572
- "i3": 16,
573
- "r0": 141,
574
- "r1": 141,
575
- "r2": 141,
576
- "nnz": 2254438
577
- },
578
- "model.layers.6.mlp.down_proj": {
579
- "o1": 16,
580
- "o2": 16,
581
- "o3": 16,
582
- "i1": 16,
583
- "i2": 16,
584
- "i3": 43,
585
- "r0": 141,
586
- "r1": 141,
587
- "r2": 141,
588
- "nnz": 2254438
589
- },
590
- "model.layers.7.self_attn.q_proj": {
591
- "o1": 16,
592
- "o2": 16,
593
- "o3": 16,
594
- "i1": 16,
595
- "i2": 16,
596
- "i3": 16,
597
- "r0": 108,
598
- "r1": 108,
599
- "r2": 108,
600
- "nnz": 838860
601
- },
602
- "model.layers.7.self_attn.k_proj": {
603
- "o1": 16,
604
- "o2": 16,
605
- "o3": 16,
606
- "i1": 16,
607
- "i2": 16,
608
- "i3": 16,
609
- "r0": 108,
610
- "r1": 108,
611
- "r2": 108,
612
- "nnz": 838860
613
- },
614
- "model.layers.7.self_attn.v_proj": {
615
- "o1": 16,
616
- "o2": 16,
617
- "o3": 16,
618
- "i1": 16,
619
- "i2": 16,
620
- "i3": 16,
621
- "r0": 108,
622
- "r1": 108,
623
- "r2": 108,
624
- "nnz": 838860
625
- },
626
- "model.layers.7.self_attn.o_proj": {
627
- "o1": 16,
628
- "o2": 16,
629
- "o3": 16,
630
- "i1": 16,
631
- "i2": 16,
632
- "i3": 16,
633
- "r0": 108,
634
- "r1": 108,
635
- "r2": 108,
636
- "nnz": 838860
637
- },
638
- "model.layers.7.mlp.gate_proj": {
639
- "o1": 16,
640
- "o2": 16,
641
- "o3": 43,
642
- "i1": 16,
643
- "i2": 16,
644
- "i3": 16,
645
- "r0": 141,
646
- "r1": 141,
647
- "r2": 141,
648
- "nnz": 2254438
649
- },
650
- "model.layers.7.mlp.up_proj": {
651
- "o1": 16,
652
- "o2": 16,
653
- "o3": 43,
654
- "i1": 16,
655
- "i2": 16,
656
- "i3": 16,
657
- "r0": 141,
658
- "r1": 141,
659
- "r2": 141,
660
- "nnz": 2254438
661
- },
662
- "model.layers.7.mlp.down_proj": {
663
- "o1": 16,
664
- "o2": 16,
665
- "o3": 16,
666
- "i1": 16,
667
- "i2": 16,
668
- "i3": 43,
669
- "r0": 141,
670
- "r1": 141,
671
- "r2": 141,
672
- "nnz": 2254438
673
- },
674
- "model.layers.8.self_attn.q_proj": {
675
- "o1": 16,
676
- "o2": 16,
677
- "o3": 16,
678
- "i1": 16,
679
- "i2": 16,
680
- "i3": 16,
681
- "r0": 108,
682
- "r1": 108,
683
- "r2": 108,
684
- "nnz": 838860
685
- },
686
- "model.layers.8.self_attn.k_proj": {
687
- "o1": 16,
688
- "o2": 16,
689
- "o3": 16,
690
- "i1": 16,
691
- "i2": 16,
692
- "i3": 16,
693
- "r0": 108,
694
- "r1": 108,
695
- "r2": 108,
696
- "nnz": 838860
697
- },
698
- "model.layers.8.self_attn.v_proj": {
699
- "o1": 16,
700
- "o2": 16,
701
- "o3": 16,
702
- "i1": 16,
703
- "i2": 16,
704
- "i3": 16,
705
- "r0": 108,
706
- "r1": 108,
707
- "r2": 108,
708
- "nnz": 838860
709
- },
710
- "model.layers.8.self_attn.o_proj": {
711
- "o1": 16,
712
- "o2": 16,
713
- "o3": 16,
714
- "i1": 16,
715
- "i2": 16,
716
- "i3": 16,
717
- "r0": 108,
718
- "r1": 108,
719
- "r2": 108,
720
- "nnz": 838860
721
- },
722
- "model.layers.8.mlp.gate_proj": {
723
- "o1": 16,
724
- "o2": 16,
725
- "o3": 43,
726
- "i1": 16,
727
- "i2": 16,
728
- "i3": 16,
729
- "r0": 141,
730
- "r1": 141,
731
- "r2": 141,
732
- "nnz": 2254438
733
- },
734
- "model.layers.8.mlp.up_proj": {
735
- "o1": 16,
736
- "o2": 16,
737
- "o3": 43,
738
- "i1": 16,
739
- "i2": 16,
740
- "i3": 16,
741
- "r0": 141,
742
- "r1": 141,
743
- "r2": 141,
744
- "nnz": 2254438
745
- },
746
- "model.layers.8.mlp.down_proj": {
747
- "o1": 16,
748
- "o2": 16,
749
- "o3": 16,
750
- "i1": 16,
751
- "i2": 16,
752
- "i3": 43,
753
- "r0": 141,
754
- "r1": 141,
755
- "r2": 141,
756
- "nnz": 2254438
757
- },
758
- "model.layers.9.self_attn.q_proj": {
759
- "o1": 16,
760
- "o2": 16,
761
- "o3": 16,
762
- "i1": 16,
763
- "i2": 16,
764
- "i3": 16,
765
- "r0": 108,
766
- "r1": 108,
767
- "r2": 108,
768
- "nnz": 838860
769
- },
770
- "model.layers.9.self_attn.k_proj": {
771
- "o1": 16,
772
- "o2": 16,
773
- "o3": 16,
774
- "i1": 16,
775
- "i2": 16,
776
- "i3": 16,
777
- "r0": 108,
778
- "r1": 108,
779
- "r2": 108,
780
- "nnz": 838860
781
- },
782
- "model.layers.9.self_attn.v_proj": {
783
- "o1": 16,
784
- "o2": 16,
785
- "o3": 16,
786
- "i1": 16,
787
- "i2": 16,
788
- "i3": 16,
789
- "r0": 108,
790
- "r1": 108,
791
- "r2": 108,
792
- "nnz": 838860
793
- },
794
- "model.layers.9.self_attn.o_proj": {
795
- "o1": 16,
796
- "o2": 16,
797
- "o3": 16,
798
- "i1": 16,
799
- "i2": 16,
800
- "i3": 16,
801
- "r0": 108,
802
- "r1": 108,
803
- "r2": 108,
804
- "nnz": 838860
805
- },
806
- "model.layers.9.mlp.gate_proj": {
807
- "o1": 16,
808
- "o2": 16,
809
- "o3": 43,
810
- "i1": 16,
811
- "i2": 16,
812
- "i3": 16,
813
- "r0": 141,
814
- "r1": 141,
815
- "r2": 141,
816
- "nnz": 2254438
817
- },
818
- "model.layers.9.mlp.up_proj": {
819
- "o1": 16,
820
- "o2": 16,
821
- "o3": 43,
822
- "i1": 16,
823
- "i2": 16,
824
- "i3": 16,
825
- "r0": 141,
826
- "r1": 141,
827
- "r2": 141,
828
- "nnz": 2254438
829
- },
830
- "model.layers.9.mlp.down_proj": {
831
- "o1": 16,
832
- "o2": 16,
833
- "o3": 16,
834
- "i1": 16,
835
- "i2": 16,
836
- "i3": 43,
837
- "r0": 141,
838
- "r1": 141,
839
- "r2": 141,
840
- "nnz": 2254438
841
- },
842
- "model.layers.10.self_attn.q_proj": {
843
- "o1": 16,
844
- "o2": 16,
845
- "o3": 16,
846
- "i1": 16,
847
- "i2": 16,
848
- "i3": 16,
849
- "r0": 108,
850
- "r1": 108,
851
- "r2": 108,
852
- "nnz": 838860
853
- },
854
- "model.layers.10.self_attn.k_proj": {
855
- "o1": 16,
856
- "o2": 16,
857
- "o3": 16,
858
- "i1": 16,
859
- "i2": 16,
860
- "i3": 16,
861
- "r0": 108,
862
- "r1": 108,
863
- "r2": 108,
864
- "nnz": 838860
865
- },
866
- "model.layers.10.self_attn.v_proj": {
867
- "o1": 16,
868
- "o2": 16,
869
- "o3": 16,
870
- "i1": 16,
871
- "i2": 16,
872
- "i3": 16,
873
- "r0": 108,
874
- "r1": 108,
875
- "r2": 108,
876
- "nnz": 838860
877
- },
878
- "model.layers.10.self_attn.o_proj": {
879
- "o1": 16,
880
- "o2": 16,
881
- "o3": 16,
882
- "i1": 16,
883
- "i2": 16,
884
- "i3": 16,
885
- "r0": 108,
886
- "r1": 108,
887
- "r2": 108,
888
- "nnz": 838860
889
- },
890
- "model.layers.10.mlp.gate_proj": {
891
- "o1": 16,
892
- "o2": 16,
893
- "o3": 43,
894
- "i1": 16,
895
- "i2": 16,
896
- "i3": 16,
897
- "r0": 141,
898
- "r1": 141,
899
- "r2": 141,
900
- "nnz": 2254438
901
- },
902
- "model.layers.10.mlp.up_proj": {
903
- "o1": 16,
904
- "o2": 16,
905
- "o3": 43,
906
- "i1": 16,
907
- "i2": 16,
908
- "i3": 16,
909
- "r0": 141,
910
- "r1": 141,
911
- "r2": 141,
912
- "nnz": 2254438
913
- },
914
- "model.layers.10.mlp.down_proj": {
915
- "o1": 16,
916
- "o2": 16,
917
- "o3": 16,
918
- "i1": 16,
919
- "i2": 16,
920
- "i3": 43,
921
- "r0": 141,
922
- "r1": 141,
923
- "r2": 141,
924
- "nnz": 2254438
925
- },
926
- "model.layers.11.self_attn.q_proj": {
927
- "o1": 16,
928
- "o2": 16,
929
- "o3": 16,
930
- "i1": 16,
931
- "i2": 16,
932
- "i3": 16,
933
- "r0": 108,
934
- "r1": 108,
935
- "r2": 108,
936
- "nnz": 838860
937
- },
938
- "model.layers.11.self_attn.k_proj": {
939
- "o1": 16,
940
- "o2": 16,
941
- "o3": 16,
942
- "i1": 16,
943
- "i2": 16,
944
- "i3": 16,
945
- "r0": 108,
946
- "r1": 108,
947
- "r2": 108,
948
- "nnz": 838860
949
- },
950
- "model.layers.11.self_attn.v_proj": {
951
- "o1": 16,
952
- "o2": 16,
953
- "o3": 16,
954
- "i1": 16,
955
- "i2": 16,
956
- "i3": 16,
957
- "r0": 108,
958
- "r1": 108,
959
- "r2": 108,
960
- "nnz": 838860
961
- },
962
- "model.layers.11.self_attn.o_proj": {
963
- "o1": 16,
964
- "o2": 16,
965
- "o3": 16,
966
- "i1": 16,
967
- "i2": 16,
968
- "i3": 16,
969
- "r0": 108,
970
- "r1": 108,
971
- "r2": 108,
972
- "nnz": 838860
973
- },
974
- "model.layers.11.mlp.gate_proj": {
975
- "o1": 16,
976
- "o2": 16,
977
- "o3": 43,
978
- "i1": 16,
979
- "i2": 16,
980
- "i3": 16,
981
- "r0": 141,
982
- "r1": 141,
983
- "r2": 141,
984
- "nnz": 2254438
985
- },
986
- "model.layers.11.mlp.up_proj": {
987
- "o1": 16,
988
- "o2": 16,
989
- "o3": 43,
990
- "i1": 16,
991
- "i2": 16,
992
- "i3": 16,
993
- "r0": 141,
994
- "r1": 141,
995
- "r2": 141,
996
- "nnz": 2254438
997
- },
998
- "model.layers.11.mlp.down_proj": {
999
- "o1": 16,
1000
- "o2": 16,
1001
- "o3": 16,
1002
- "i1": 16,
1003
- "i2": 16,
1004
- "i3": 43,
1005
- "r0": 141,
1006
- "r1": 141,
1007
- "r2": 141,
1008
- "nnz": 2254438
1009
- },
1010
- "model.layers.12.self_attn.q_proj": {
1011
- "o1": 16,
1012
- "o2": 16,
1013
- "o3": 16,
1014
- "i1": 16,
1015
- "i2": 16,
1016
- "i3": 16,
1017
- "r0": 108,
1018
- "r1": 108,
1019
- "r2": 108,
1020
- "nnz": 838860
1021
- },
1022
- "model.layers.12.self_attn.k_proj": {
1023
- "o1": 16,
1024
- "o2": 16,
1025
- "o3": 16,
1026
- "i1": 16,
1027
- "i2": 16,
1028
- "i3": 16,
1029
- "r0": 108,
1030
- "r1": 108,
1031
- "r2": 108,
1032
- "nnz": 838860
1033
- },
1034
- "model.layers.12.self_attn.v_proj": {
1035
- "o1": 16,
1036
- "o2": 16,
1037
- "o3": 16,
1038
- "i1": 16,
1039
- "i2": 16,
1040
- "i3": 16,
1041
- "r0": 108,
1042
- "r1": 108,
1043
- "r2": 108,
1044
- "nnz": 838860
1045
- },
1046
- "model.layers.12.self_attn.o_proj": {
1047
- "o1": 16,
1048
- "o2": 16,
1049
- "o3": 16,
1050
- "i1": 16,
1051
- "i2": 16,
1052
- "i3": 16,
1053
- "r0": 108,
1054
- "r1": 108,
1055
- "r2": 108,
1056
- "nnz": 838860
1057
- },
1058
- "model.layers.12.mlp.gate_proj": {
1059
- "o1": 16,
1060
- "o2": 16,
1061
- "o3": 43,
1062
- "i1": 16,
1063
- "i2": 16,
1064
- "i3": 16,
1065
- "r0": 141,
1066
- "r1": 141,
1067
- "r2": 141,
1068
- "nnz": 2254438
1069
- },
1070
- "model.layers.12.mlp.up_proj": {
1071
- "o1": 16,
1072
- "o2": 16,
1073
- "o3": 43,
1074
- "i1": 16,
1075
- "i2": 16,
1076
- "i3": 16,
1077
- "r0": 141,
1078
- "r1": 141,
1079
- "r2": 141,
1080
- "nnz": 2254438
1081
- },
1082
- "model.layers.12.mlp.down_proj": {
1083
- "o1": 16,
1084
- "o2": 16,
1085
- "o3": 16,
1086
- "i1": 16,
1087
- "i2": 16,
1088
- "i3": 43,
1089
- "r0": 141,
1090
- "r1": 141,
1091
- "r2": 141,
1092
- "nnz": 2254438
1093
- },
1094
- "model.layers.13.self_attn.q_proj": {
1095
- "o1": 16,
1096
- "o2": 16,
1097
- "o3": 16,
1098
- "i1": 16,
1099
- "i2": 16,
1100
- "i3": 16,
1101
- "r0": 108,
1102
- "r1": 108,
1103
- "r2": 108,
1104
- "nnz": 838860
1105
- },
1106
- "model.layers.13.self_attn.k_proj": {
1107
- "o1": 16,
1108
- "o2": 16,
1109
- "o3": 16,
1110
- "i1": 16,
1111
- "i2": 16,
1112
- "i3": 16,
1113
- "r0": 108,
1114
- "r1": 108,
1115
- "r2": 108,
1116
- "nnz": 838860
1117
- },
1118
- "model.layers.13.self_attn.v_proj": {
1119
- "o1": 16,
1120
- "o2": 16,
1121
- "o3": 16,
1122
- "i1": 16,
1123
- "i2": 16,
1124
- "i3": 16,
1125
- "r0": 108,
1126
- "r1": 108,
1127
- "r2": 108,
1128
- "nnz": 838860
1129
- },
1130
- "model.layers.13.self_attn.o_proj": {
1131
- "o1": 16,
1132
- "o2": 16,
1133
- "o3": 16,
1134
- "i1": 16,
1135
- "i2": 16,
1136
- "i3": 16,
1137
- "r0": 108,
1138
- "r1": 108,
1139
- "r2": 108,
1140
- "nnz": 838860
1141
- },
1142
- "model.layers.13.mlp.gate_proj": {
1143
- "o1": 16,
1144
- "o2": 16,
1145
- "o3": 43,
1146
- "i1": 16,
1147
- "i2": 16,
1148
- "i3": 16,
1149
- "r0": 141,
1150
- "r1": 141,
1151
- "r2": 141,
1152
- "nnz": 2254438
1153
- },
1154
- "model.layers.13.mlp.up_proj": {
1155
- "o1": 16,
1156
- "o2": 16,
1157
- "o3": 43,
1158
- "i1": 16,
1159
- "i2": 16,
1160
- "i3": 16,
1161
- "r0": 141,
1162
- "r1": 141,
1163
- "r2": 141,
1164
- "nnz": 2254438
1165
- },
1166
- "model.layers.13.mlp.down_proj": {
1167
- "o1": 16,
1168
- "o2": 16,
1169
- "o3": 16,
1170
- "i1": 16,
1171
- "i2": 16,
1172
- "i3": 43,
1173
- "r0": 141,
1174
- "r1": 141,
1175
- "r2": 141,
1176
- "nnz": 2254438
1177
- },
1178
- "model.layers.14.self_attn.q_proj": {
1179
- "o1": 16,
1180
- "o2": 16,
1181
- "o3": 16,
1182
- "i1": 16,
1183
- "i2": 16,
1184
- "i3": 16,
1185
- "r0": 108,
1186
- "r1": 108,
1187
- "r2": 108,
1188
- "nnz": 838860
1189
- },
1190
- "model.layers.14.self_attn.k_proj": {
1191
- "o1": 16,
1192
- "o2": 16,
1193
- "o3": 16,
1194
- "i1": 16,
1195
- "i2": 16,
1196
- "i3": 16,
1197
- "r0": 108,
1198
- "r1": 108,
1199
- "r2": 108,
1200
- "nnz": 838860
1201
- },
1202
- "model.layers.14.self_attn.v_proj": {
1203
- "o1": 16,
1204
- "o2": 16,
1205
- "o3": 16,
1206
- "i1": 16,
1207
- "i2": 16,
1208
- "i3": 16,
1209
- "r0": 108,
1210
- "r1": 108,
1211
- "r2": 108,
1212
- "nnz": 838860
1213
- },
1214
- "model.layers.14.self_attn.o_proj": {
1215
- "o1": 16,
1216
- "o2": 16,
1217
- "o3": 16,
1218
- "i1": 16,
1219
- "i2": 16,
1220
- "i3": 16,
1221
- "r0": 108,
1222
- "r1": 108,
1223
- "r2": 108,
1224
- "nnz": 838860
1225
- },
1226
- "model.layers.14.mlp.gate_proj": {
1227
- "o1": 16,
1228
- "o2": 16,
1229
- "o3": 43,
1230
- "i1": 16,
1231
- "i2": 16,
1232
- "i3": 16,
1233
- "r0": 141,
1234
- "r1": 141,
1235
- "r2": 141,
1236
- "nnz": 2254438
1237
- },
1238
- "model.layers.14.mlp.up_proj": {
1239
- "o1": 16,
1240
- "o2": 16,
1241
- "o3": 43,
1242
- "i1": 16,
1243
- "i2": 16,
1244
- "i3": 16,
1245
- "r0": 141,
1246
- "r1": 141,
1247
- "r2": 141,
1248
- "nnz": 2254438
1249
- },
1250
- "model.layers.14.mlp.down_proj": {
1251
- "o1": 16,
1252
- "o2": 16,
1253
- "o3": 16,
1254
- "i1": 16,
1255
- "i2": 16,
1256
- "i3": 43,
1257
- "r0": 141,
1258
- "r1": 141,
1259
- "r2": 141,
1260
- "nnz": 2254438
1261
- },
1262
- "model.layers.15.self_attn.q_proj": {
1263
- "o1": 16,
1264
- "o2": 16,
1265
- "o3": 16,
1266
- "i1": 16,
1267
- "i2": 16,
1268
- "i3": 16,
1269
- "r0": 108,
1270
- "r1": 108,
1271
- "r2": 108,
1272
- "nnz": 838860
1273
- },
1274
- "model.layers.15.self_attn.k_proj": {
1275
- "o1": 16,
1276
- "o2": 16,
1277
- "o3": 16,
1278
- "i1": 16,
1279
- "i2": 16,
1280
- "i3": 16,
1281
- "r0": 108,
1282
- "r1": 108,
1283
- "r2": 108,
1284
- "nnz": 838860
1285
- },
1286
- "model.layers.15.self_attn.v_proj": {
1287
- "o1": 16,
1288
- "o2": 16,
1289
- "o3": 16,
1290
- "i1": 16,
1291
- "i2": 16,
1292
- "i3": 16,
1293
- "r0": 108,
1294
- "r1": 108,
1295
- "r2": 108,
1296
- "nnz": 838860
1297
- },
1298
- "model.layers.15.self_attn.o_proj": {
1299
- "o1": 16,
1300
- "o2": 16,
1301
- "o3": 16,
1302
- "i1": 16,
1303
- "i2": 16,
1304
- "i3": 16,
1305
- "r0": 108,
1306
- "r1": 108,
1307
- "r2": 108,
1308
- "nnz": 838860
1309
- },
1310
- "model.layers.15.mlp.gate_proj": {
1311
- "o1": 16,
1312
- "o2": 16,
1313
- "o3": 43,
1314
- "i1": 16,
1315
- "i2": 16,
1316
- "i3": 16,
1317
- "r0": 141,
1318
- "r1": 141,
1319
- "r2": 141,
1320
- "nnz": 2254438
1321
- },
1322
- "model.layers.15.mlp.up_proj": {
1323
- "o1": 16,
1324
- "o2": 16,
1325
- "o3": 43,
1326
- "i1": 16,
1327
- "i2": 16,
1328
- "i3": 16,
1329
- "r0": 141,
1330
- "r1": 141,
1331
- "r2": 141,
1332
- "nnz": 2254438
1333
- },
1334
- "model.layers.15.mlp.down_proj": {
1335
- "o1": 16,
1336
- "o2": 16,
1337
- "o3": 16,
1338
- "i1": 16,
1339
- "i2": 16,
1340
- "i3": 43,
1341
- "r0": 141,
1342
- "r1": 141,
1343
- "r2": 141,
1344
- "nnz": 2254438
1345
- },
1346
- "model.layers.16.self_attn.q_proj": {
1347
- "o1": 16,
1348
- "o2": 16,
1349
- "o3": 16,
1350
- "i1": 16,
1351
- "i2": 16,
1352
- "i3": 16,
1353
- "r0": 108,
1354
- "r1": 108,
1355
- "r2": 108,
1356
- "nnz": 838860
1357
- },
1358
- "model.layers.16.self_attn.k_proj": {
1359
- "o1": 16,
1360
- "o2": 16,
1361
- "o3": 16,
1362
- "i1": 16,
1363
- "i2": 16,
1364
- "i3": 16,
1365
- "r0": 108,
1366
- "r1": 108,
1367
- "r2": 108,
1368
- "nnz": 838860
1369
- },
1370
- "model.layers.16.self_attn.v_proj": {
1371
- "o1": 16,
1372
- "o2": 16,
1373
- "o3": 16,
1374
- "i1": 16,
1375
- "i2": 16,
1376
- "i3": 16,
1377
- "r0": 108,
1378
- "r1": 108,
1379
- "r2": 108,
1380
- "nnz": 838860
1381
- },
1382
- "model.layers.16.self_attn.o_proj": {
1383
- "o1": 16,
1384
- "o2": 16,
1385
- "o3": 16,
1386
- "i1": 16,
1387
- "i2": 16,
1388
- "i3": 16,
1389
- "r0": 108,
1390
- "r1": 108,
1391
- "r2": 108,
1392
- "nnz": 838860
1393
- },
1394
- "model.layers.16.mlp.gate_proj": {
1395
- "o1": 16,
1396
- "o2": 16,
1397
- "o3": 43,
1398
- "i1": 16,
1399
- "i2": 16,
1400
- "i3": 16,
1401
- "r0": 141,
1402
- "r1": 141,
1403
- "r2": 141,
1404
- "nnz": 2254438
1405
- },
1406
- "model.layers.16.mlp.up_proj": {
1407
- "o1": 16,
1408
- "o2": 16,
1409
- "o3": 43,
1410
- "i1": 16,
1411
- "i2": 16,
1412
- "i3": 16,
1413
- "r0": 141,
1414
- "r1": 141,
1415
- "r2": 141,
1416
- "nnz": 2254438
1417
- },
1418
- "model.layers.16.mlp.down_proj": {
1419
- "o1": 16,
1420
- "o2": 16,
1421
- "o3": 16,
1422
- "i1": 16,
1423
- "i2": 16,
1424
- "i3": 43,
1425
- "r0": 141,
1426
- "r1": 141,
1427
- "r2": 141,
1428
- "nnz": 2254438
1429
- },
1430
- "model.layers.17.self_attn.q_proj": {
1431
- "o1": 16,
1432
- "o2": 16,
1433
- "o3": 16,
1434
- "i1": 16,
1435
- "i2": 16,
1436
- "i3": 16,
1437
- "r0": 108,
1438
- "r1": 108,
1439
- "r2": 108,
1440
- "nnz": 838860
1441
- },
1442
- "model.layers.17.self_attn.k_proj": {
1443
- "o1": 16,
1444
- "o2": 16,
1445
- "o3": 16,
1446
- "i1": 16,
1447
- "i2": 16,
1448
- "i3": 16,
1449
- "r0": 108,
1450
- "r1": 108,
1451
- "r2": 108,
1452
- "nnz": 838860
1453
- },
1454
- "model.layers.17.self_attn.v_proj": {
1455
- "o1": 16,
1456
- "o2": 16,
1457
- "o3": 16,
1458
- "i1": 16,
1459
- "i2": 16,
1460
- "i3": 16,
1461
- "r0": 108,
1462
- "r1": 108,
1463
- "r2": 108,
1464
- "nnz": 838860
1465
- },
1466
- "model.layers.17.self_attn.o_proj": {
1467
- "o1": 16,
1468
- "o2": 16,
1469
- "o3": 16,
1470
- "i1": 16,
1471
- "i2": 16,
1472
- "i3": 16,
1473
- "r0": 108,
1474
- "r1": 108,
1475
- "r2": 108,
1476
- "nnz": 838860
1477
- },
1478
- "model.layers.17.mlp.gate_proj": {
1479
- "o1": 16,
1480
- "o2": 16,
1481
- "o3": 43,
1482
- "i1": 16,
1483
- "i2": 16,
1484
- "i3": 16,
1485
- "r0": 141,
1486
- "r1": 141,
1487
- "r2": 141,
1488
- "nnz": 2254438
1489
- },
1490
- "model.layers.17.mlp.up_proj": {
1491
- "o1": 16,
1492
- "o2": 16,
1493
- "o3": 43,
1494
- "i1": 16,
1495
- "i2": 16,
1496
- "i3": 16,
1497
- "r0": 141,
1498
- "r1": 141,
1499
- "r2": 141,
1500
- "nnz": 2254438
1501
- },
1502
- "model.layers.17.mlp.down_proj": {
1503
- "o1": 16,
1504
- "o2": 16,
1505
- "o3": 16,
1506
- "i1": 16,
1507
- "i2": 16,
1508
- "i3": 43,
1509
- "r0": 141,
1510
- "r1": 141,
1511
- "r2": 141,
1512
- "nnz": 2254438
1513
- },
1514
- "model.layers.18.self_attn.q_proj": {
1515
- "o1": 16,
1516
- "o2": 16,
1517
- "o3": 16,
1518
- "i1": 16,
1519
- "i2": 16,
1520
- "i3": 16,
1521
- "r0": 108,
1522
- "r1": 108,
1523
- "r2": 108,
1524
- "nnz": 838860
1525
- },
1526
- "model.layers.18.self_attn.k_proj": {
1527
- "o1": 16,
1528
- "o2": 16,
1529
- "o3": 16,
1530
- "i1": 16,
1531
- "i2": 16,
1532
- "i3": 16,
1533
- "r0": 108,
1534
- "r1": 108,
1535
- "r2": 108,
1536
- "nnz": 838860
1537
- },
1538
- "model.layers.18.self_attn.v_proj": {
1539
- "o1": 16,
1540
- "o2": 16,
1541
- "o3": 16,
1542
- "i1": 16,
1543
- "i2": 16,
1544
- "i3": 16,
1545
- "r0": 108,
1546
- "r1": 108,
1547
- "r2": 108,
1548
- "nnz": 838860
1549
- },
1550
- "model.layers.18.self_attn.o_proj": {
1551
- "o1": 16,
1552
- "o2": 16,
1553
- "o3": 16,
1554
- "i1": 16,
1555
- "i2": 16,
1556
- "i3": 16,
1557
- "r0": 108,
1558
- "r1": 108,
1559
- "r2": 108,
1560
- "nnz": 838860
1561
- },
1562
- "model.layers.18.mlp.gate_proj": {
1563
- "o1": 16,
1564
- "o2": 16,
1565
- "o3": 43,
1566
- "i1": 16,
1567
- "i2": 16,
1568
- "i3": 16,
1569
- "r0": 141,
1570
- "r1": 141,
1571
- "r2": 141,
1572
- "nnz": 2254438
1573
- },
1574
- "model.layers.18.mlp.up_proj": {
1575
- "o1": 16,
1576
- "o2": 16,
1577
- "o3": 43,
1578
- "i1": 16,
1579
- "i2": 16,
1580
- "i3": 16,
1581
- "r0": 141,
1582
- "r1": 141,
1583
- "r2": 141,
1584
- "nnz": 2254438
1585
- },
1586
- "model.layers.18.mlp.down_proj": {
1587
- "o1": 16,
1588
- "o2": 16,
1589
- "o3": 16,
1590
- "i1": 16,
1591
- "i2": 16,
1592
- "i3": 43,
1593
- "r0": 141,
1594
- "r1": 141,
1595
- "r2": 141,
1596
- "nnz": 2254438
1597
- },
1598
- "model.layers.19.self_attn.q_proj": {
1599
- "o1": 16,
1600
- "o2": 16,
1601
- "o3": 16,
1602
- "i1": 16,
1603
- "i2": 16,
1604
- "i3": 16,
1605
- "r0": 108,
1606
- "r1": 108,
1607
- "r2": 108,
1608
- "nnz": 838860
1609
- },
1610
- "model.layers.19.self_attn.k_proj": {
1611
- "o1": 16,
1612
- "o2": 16,
1613
- "o3": 16,
1614
- "i1": 16,
1615
- "i2": 16,
1616
- "i3": 16,
1617
- "r0": 108,
1618
- "r1": 108,
1619
- "r2": 108,
1620
- "nnz": 838860
1621
- },
1622
- "model.layers.19.self_attn.v_proj": {
1623
- "o1": 16,
1624
- "o2": 16,
1625
- "o3": 16,
1626
- "i1": 16,
1627
- "i2": 16,
1628
- "i3": 16,
1629
- "r0": 108,
1630
- "r1": 108,
1631
- "r2": 108,
1632
- "nnz": 838860
1633
- },
1634
- "model.layers.19.self_attn.o_proj": {
1635
- "o1": 16,
1636
- "o2": 16,
1637
- "o3": 16,
1638
- "i1": 16,
1639
- "i2": 16,
1640
- "i3": 16,
1641
- "r0": 108,
1642
- "r1": 108,
1643
- "r2": 108,
1644
- "nnz": 838860
1645
- },
1646
- "model.layers.19.mlp.gate_proj": {
1647
- "o1": 16,
1648
- "o2": 16,
1649
- "o3": 43,
1650
- "i1": 16,
1651
- "i2": 16,
1652
- "i3": 16,
1653
- "r0": 141,
1654
- "r1": 141,
1655
- "r2": 141,
1656
- "nnz": 2254438
1657
- },
1658
- "model.layers.19.mlp.up_proj": {
1659
- "o1": 16,
1660
- "o2": 16,
1661
- "o3": 43,
1662
- "i1": 16,
1663
- "i2": 16,
1664
- "i3": 16,
1665
- "r0": 141,
1666
- "r1": 141,
1667
- "r2": 141,
1668
- "nnz": 2254438
1669
- },
1670
- "model.layers.19.mlp.down_proj": {
1671
- "o1": 16,
1672
- "o2": 16,
1673
- "o3": 16,
1674
- "i1": 16,
1675
- "i2": 16,
1676
- "i3": 43,
1677
- "r0": 141,
1678
- "r1": 141,
1679
- "r2": 141,
1680
- "nnz": 2254438
1681
- },
1682
- "model.layers.20.self_attn.q_proj": {
1683
- "o1": 16,
1684
- "o2": 16,
1685
- "o3": 16,
1686
- "i1": 16,
1687
- "i2": 16,
1688
- "i3": 16,
1689
- "r0": 108,
1690
- "r1": 108,
1691
- "r2": 108,
1692
- "nnz": 838860
1693
- },
1694
- "model.layers.20.self_attn.k_proj": {
1695
- "o1": 16,
1696
- "o2": 16,
1697
- "o3": 16,
1698
- "i1": 16,
1699
- "i2": 16,
1700
- "i3": 16,
1701
- "r0": 108,
1702
- "r1": 108,
1703
- "r2": 108,
1704
- "nnz": 838860
1705
- },
1706
- "model.layers.20.self_attn.v_proj": {
1707
- "o1": 16,
1708
- "o2": 16,
1709
- "o3": 16,
1710
- "i1": 16,
1711
- "i2": 16,
1712
- "i3": 16,
1713
- "r0": 108,
1714
- "r1": 108,
1715
- "r2": 108,
1716
- "nnz": 838860
1717
- },
1718
- "model.layers.20.self_attn.o_proj": {
1719
- "o1": 16,
1720
- "o2": 16,
1721
- "o3": 16,
1722
- "i1": 16,
1723
- "i2": 16,
1724
- "i3": 16,
1725
- "r0": 108,
1726
- "r1": 108,
1727
- "r2": 108,
1728
- "nnz": 838860
1729
- },
1730
- "model.layers.20.mlp.gate_proj": {
1731
- "o1": 16,
1732
- "o2": 16,
1733
- "o3": 43,
1734
- "i1": 16,
1735
- "i2": 16,
1736
- "i3": 16,
1737
- "r0": 141,
1738
- "r1": 141,
1739
- "r2": 141,
1740
- "nnz": 2254438
1741
- },
1742
- "model.layers.20.mlp.up_proj": {
1743
- "o1": 16,
1744
- "o2": 16,
1745
- "o3": 43,
1746
- "i1": 16,
1747
- "i2": 16,
1748
- "i3": 16,
1749
- "r0": 141,
1750
- "r1": 141,
1751
- "r2": 141,
1752
- "nnz": 2254438
1753
- },
1754
- "model.layers.20.mlp.down_proj": {
1755
- "o1": 16,
1756
- "o2": 16,
1757
- "o3": 16,
1758
- "i1": 16,
1759
- "i2": 16,
1760
- "i3": 43,
1761
- "r0": 141,
1762
- "r1": 141,
1763
- "r2": 141,
1764
- "nnz": 2254438
1765
- },
1766
- "model.layers.21.self_attn.q_proj": {
1767
- "o1": 16,
1768
- "o2": 16,
1769
- "o3": 16,
1770
- "i1": 16,
1771
- "i2": 16,
1772
- "i3": 16,
1773
- "r0": 108,
1774
- "r1": 108,
1775
- "r2": 108,
1776
- "nnz": 838860
1777
- },
1778
- "model.layers.21.self_attn.k_proj": {
1779
- "o1": 16,
1780
- "o2": 16,
1781
- "o3": 16,
1782
- "i1": 16,
1783
- "i2": 16,
1784
- "i3": 16,
1785
- "r0": 108,
1786
- "r1": 108,
1787
- "r2": 108,
1788
- "nnz": 838860
1789
- },
1790
- "model.layers.21.self_attn.v_proj": {
1791
- "o1": 16,
1792
- "o2": 16,
1793
- "o3": 16,
1794
- "i1": 16,
1795
- "i2": 16,
1796
- "i3": 16,
1797
- "r0": 108,
1798
- "r1": 108,
1799
- "r2": 108,
1800
- "nnz": 838860
1801
- },
1802
- "model.layers.21.self_attn.o_proj": {
1803
- "o1": 16,
1804
- "o2": 16,
1805
- "o3": 16,
1806
- "i1": 16,
1807
- "i2": 16,
1808
- "i3": 16,
1809
- "r0": 108,
1810
- "r1": 108,
1811
- "r2": 108,
1812
- "nnz": 838860
1813
- },
1814
- "model.layers.21.mlp.gate_proj": {
1815
- "o1": 16,
1816
- "o2": 16,
1817
- "o3": 43,
1818
- "i1": 16,
1819
- "i2": 16,
1820
- "i3": 16,
1821
- "r0": 141,
1822
- "r1": 141,
1823
- "r2": 141,
1824
- "nnz": 2254438
1825
- },
1826
- "model.layers.21.mlp.up_proj": {
1827
- "o1": 16,
1828
- "o2": 16,
1829
- "o3": 43,
1830
- "i1": 16,
1831
- "i2": 16,
1832
- "i3": 16,
1833
- "r0": 141,
1834
- "r1": 141,
1835
- "r2": 141,
1836
- "nnz": 2254438
1837
- },
1838
- "model.layers.21.mlp.down_proj": {
1839
- "o1": 16,
1840
- "o2": 16,
1841
- "o3": 16,
1842
- "i1": 16,
1843
- "i2": 16,
1844
- "i3": 43,
1845
- "r0": 141,
1846
- "r1": 141,
1847
- "r2": 141,
1848
- "nnz": 2254438
1849
- },
1850
- "model.layers.22.self_attn.q_proj": {
1851
- "o1": 16,
1852
- "o2": 16,
1853
- "o3": 16,
1854
- "i1": 16,
1855
- "i2": 16,
1856
- "i3": 16,
1857
- "r0": 108,
1858
- "r1": 108,
1859
- "r2": 108,
1860
- "nnz": 838860
1861
- },
1862
- "model.layers.22.self_attn.k_proj": {
1863
- "o1": 16,
1864
- "o2": 16,
1865
- "o3": 16,
1866
- "i1": 16,
1867
- "i2": 16,
1868
- "i3": 16,
1869
- "r0": 108,
1870
- "r1": 108,
1871
- "r2": 108,
1872
- "nnz": 838860
1873
- },
1874
- "model.layers.22.self_attn.v_proj": {
1875
- "o1": 16,
1876
- "o2": 16,
1877
- "o3": 16,
1878
- "i1": 16,
1879
- "i2": 16,
1880
- "i3": 16,
1881
- "r0": 108,
1882
- "r1": 108,
1883
- "r2": 108,
1884
- "nnz": 838860
1885
- },
1886
- "model.layers.22.self_attn.o_proj": {
1887
- "o1": 16,
1888
- "o2": 16,
1889
- "o3": 16,
1890
- "i1": 16,
1891
- "i2": 16,
1892
- "i3": 16,
1893
- "r0": 108,
1894
- "r1": 108,
1895
- "r2": 108,
1896
- "nnz": 838860
1897
- },
1898
- "model.layers.22.mlp.gate_proj": {
1899
- "o1": 16,
1900
- "o2": 16,
1901
- "o3": 43,
1902
- "i1": 16,
1903
- "i2": 16,
1904
- "i3": 16,
1905
- "r0": 141,
1906
- "r1": 141,
1907
- "r2": 141,
1908
- "nnz": 2254438
1909
- },
1910
- "model.layers.22.mlp.up_proj": {
1911
- "o1": 16,
1912
- "o2": 16,
1913
- "o3": 43,
1914
- "i1": 16,
1915
- "i2": 16,
1916
- "i3": 16,
1917
- "r0": 141,
1918
- "r1": 141,
1919
- "r2": 141,
1920
- "nnz": 2254438
1921
- },
1922
- "model.layers.22.mlp.down_proj": {
1923
- "o1": 16,
1924
- "o2": 16,
1925
- "o3": 16,
1926
- "i1": 16,
1927
- "i2": 16,
1928
- "i3": 43,
1929
- "r0": 141,
1930
- "r1": 141,
1931
- "r2": 141,
1932
- "nnz": 2254438
1933
- },
1934
- "model.layers.23.self_attn.q_proj": {
1935
- "o1": 16,
1936
- "o2": 16,
1937
- "o3": 16,
1938
- "i1": 16,
1939
- "i2": 16,
1940
- "i3": 16,
1941
- "r0": 108,
1942
- "r1": 108,
1943
- "r2": 108,
1944
- "nnz": 838860
1945
- },
1946
- "model.layers.23.self_attn.k_proj": {
1947
- "o1": 16,
1948
- "o2": 16,
1949
- "o3": 16,
1950
- "i1": 16,
1951
- "i2": 16,
1952
- "i3": 16,
1953
- "r0": 108,
1954
- "r1": 108,
1955
- "r2": 108,
1956
- "nnz": 838860
1957
- },
1958
- "model.layers.23.self_attn.v_proj": {
1959
- "o1": 16,
1960
- "o2": 16,
1961
- "o3": 16,
1962
- "i1": 16,
1963
- "i2": 16,
1964
- "i3": 16,
1965
- "r0": 108,
1966
- "r1": 108,
1967
- "r2": 108,
1968
- "nnz": 838860
1969
- },
1970
- "model.layers.23.self_attn.o_proj": {
1971
- "o1": 16,
1972
- "o2": 16,
1973
- "o3": 16,
1974
- "i1": 16,
1975
- "i2": 16,
1976
- "i3": 16,
1977
- "r0": 108,
1978
- "r1": 108,
1979
- "r2": 108,
1980
- "nnz": 838860
1981
- },
1982
- "model.layers.23.mlp.gate_proj": {
1983
- "o1": 16,
1984
- "o2": 16,
1985
- "o3": 43,
1986
- "i1": 16,
1987
- "i2": 16,
1988
- "i3": 16,
1989
- "r0": 141,
1990
- "r1": 141,
1991
- "r2": 141,
1992
- "nnz": 2254438
1993
- },
1994
- "model.layers.23.mlp.up_proj": {
1995
- "o1": 16,
1996
- "o2": 16,
1997
- "o3": 43,
1998
- "i1": 16,
1999
- "i2": 16,
2000
- "i3": 16,
2001
- "r0": 141,
2002
- "r1": 141,
2003
- "r2": 141,
2004
- "nnz": 2254438
2005
- },
2006
- "model.layers.23.mlp.down_proj": {
2007
- "o1": 16,
2008
- "o2": 16,
2009
- "o3": 16,
2010
- "i1": 16,
2011
- "i2": 16,
2012
- "i3": 43,
2013
- "r0": 141,
2014
- "r1": 141,
2015
- "r2": 141,
2016
- "nnz": 2254438
2017
- },
2018
- "model.layers.24.self_attn.q_proj": {
2019
- "o1": 16,
2020
- "o2": 16,
2021
- "o3": 16,
2022
- "i1": 16,
2023
- "i2": 16,
2024
- "i3": 16,
2025
- "r0": 108,
2026
- "r1": 108,
2027
- "r2": 108,
2028
- "nnz": 838860
2029
- },
2030
- "model.layers.24.self_attn.k_proj": {
2031
- "o1": 16,
2032
- "o2": 16,
2033
- "o3": 16,
2034
- "i1": 16,
2035
- "i2": 16,
2036
- "i3": 16,
2037
- "r0": 108,
2038
- "r1": 108,
2039
- "r2": 108,
2040
- "nnz": 838860
2041
- },
2042
- "model.layers.24.self_attn.v_proj": {
2043
- "o1": 16,
2044
- "o2": 16,
2045
- "o3": 16,
2046
- "i1": 16,
2047
- "i2": 16,
2048
- "i3": 16,
2049
- "r0": 108,
2050
- "r1": 108,
2051
- "r2": 108,
2052
- "nnz": 838860
2053
- },
2054
- "model.layers.24.self_attn.o_proj": {
2055
- "o1": 16,
2056
- "o2": 16,
2057
- "o3": 16,
2058
- "i1": 16,
2059
- "i2": 16,
2060
- "i3": 16,
2061
- "r0": 108,
2062
- "r1": 108,
2063
- "r2": 108,
2064
- "nnz": 838860
2065
- },
2066
- "model.layers.24.mlp.gate_proj": {
2067
- "o1": 16,
2068
- "o2": 16,
2069
- "o3": 43,
2070
- "i1": 16,
2071
- "i2": 16,
2072
- "i3": 16,
2073
- "r0": 141,
2074
- "r1": 141,
2075
- "r2": 141,
2076
- "nnz": 2254438
2077
- },
2078
- "model.layers.24.mlp.up_proj": {
2079
- "o1": 16,
2080
- "o2": 16,
2081
- "o3": 43,
2082
- "i1": 16,
2083
- "i2": 16,
2084
- "i3": 16,
2085
- "r0": 141,
2086
- "r1": 141,
2087
- "r2": 141,
2088
- "nnz": 2254438
2089
- },
2090
- "model.layers.24.mlp.down_proj": {
2091
- "o1": 16,
2092
- "o2": 16,
2093
- "o3": 16,
2094
- "i1": 16,
2095
- "i2": 16,
2096
- "i3": 43,
2097
- "r0": 141,
2098
- "r1": 141,
2099
- "r2": 141,
2100
- "nnz": 2254438
2101
- },
2102
- "model.layers.25.self_attn.q_proj": {
2103
- "o1": 16,
2104
- "o2": 16,
2105
- "o3": 16,
2106
- "i1": 16,
2107
- "i2": 16,
2108
- "i3": 16,
2109
- "r0": 108,
2110
- "r1": 108,
2111
- "r2": 108,
2112
- "nnz": 838860
2113
- },
2114
- "model.layers.25.self_attn.k_proj": {
2115
- "o1": 16,
2116
- "o2": 16,
2117
- "o3": 16,
2118
- "i1": 16,
2119
- "i2": 16,
2120
- "i3": 16,
2121
- "r0": 108,
2122
- "r1": 108,
2123
- "r2": 108,
2124
- "nnz": 838860
2125
- },
2126
- "model.layers.25.self_attn.v_proj": {
2127
- "o1": 16,
2128
- "o2": 16,
2129
- "o3": 16,
2130
- "i1": 16,
2131
- "i2": 16,
2132
- "i3": 16,
2133
- "r0": 108,
2134
- "r1": 108,
2135
- "r2": 108,
2136
- "nnz": 838860
2137
- },
2138
- "model.layers.25.self_attn.o_proj": {
2139
- "o1": 16,
2140
- "o2": 16,
2141
- "o3": 16,
2142
- "i1": 16,
2143
- "i2": 16,
2144
- "i3": 16,
2145
- "r0": 108,
2146
- "r1": 108,
2147
- "r2": 108,
2148
- "nnz": 838860
2149
- },
2150
- "model.layers.25.mlp.gate_proj": {
2151
- "o1": 16,
2152
- "o2": 16,
2153
- "o3": 43,
2154
- "i1": 16,
2155
- "i2": 16,
2156
- "i3": 16,
2157
- "r0": 141,
2158
- "r1": 141,
2159
- "r2": 141,
2160
- "nnz": 2254438
2161
- },
2162
- "model.layers.25.mlp.up_proj": {
2163
- "o1": 16,
2164
- "o2": 16,
2165
- "o3": 43,
2166
- "i1": 16,
2167
- "i2": 16,
2168
- "i3": 16,
2169
- "r0": 141,
2170
- "r1": 141,
2171
- "r2": 141,
2172
- "nnz": 2254438
2173
- },
2174
- "model.layers.25.mlp.down_proj": {
2175
- "o1": 16,
2176
- "o2": 16,
2177
- "o3": 16,
2178
- "i1": 16,
2179
- "i2": 16,
2180
- "i3": 43,
2181
- "r0": 141,
2182
- "r1": 141,
2183
- "r2": 141,
2184
- "nnz": 2254438
2185
- },
2186
- "model.layers.26.self_attn.q_proj": {
2187
- "o1": 16,
2188
- "o2": 16,
2189
- "o3": 16,
2190
- "i1": 16,
2191
- "i2": 16,
2192
- "i3": 16,
2193
- "r0": 108,
2194
- "r1": 108,
2195
- "r2": 108,
2196
- "nnz": 838860
2197
- },
2198
- "model.layers.26.self_attn.k_proj": {
2199
- "o1": 16,
2200
- "o2": 16,
2201
- "o3": 16,
2202
- "i1": 16,
2203
- "i2": 16,
2204
- "i3": 16,
2205
- "r0": 108,
2206
- "r1": 108,
2207
- "r2": 108,
2208
- "nnz": 838860
2209
- },
2210
- "model.layers.26.self_attn.v_proj": {
2211
- "o1": 16,
2212
- "o2": 16,
2213
- "o3": 16,
2214
- "i1": 16,
2215
- "i2": 16,
2216
- "i3": 16,
2217
- "r0": 108,
2218
- "r1": 108,
2219
- "r2": 108,
2220
- "nnz": 838860
2221
- },
2222
- "model.layers.26.self_attn.o_proj": {
2223
- "o1": 16,
2224
- "o2": 16,
2225
- "o3": 16,
2226
- "i1": 16,
2227
- "i2": 16,
2228
- "i3": 16,
2229
- "r0": 108,
2230
- "r1": 108,
2231
- "r2": 108,
2232
- "nnz": 838860
2233
- },
2234
- "model.layers.26.mlp.gate_proj": {
2235
- "o1": 16,
2236
- "o2": 16,
2237
- "o3": 43,
2238
- "i1": 16,
2239
- "i2": 16,
2240
- "i3": 16,
2241
- "r0": 141,
2242
- "r1": 141,
2243
- "r2": 141,
2244
- "nnz": 2254438
2245
- },
2246
- "model.layers.26.mlp.up_proj": {
2247
- "o1": 16,
2248
- "o2": 16,
2249
- "o3": 43,
2250
- "i1": 16,
2251
- "i2": 16,
2252
- "i3": 16,
2253
- "r0": 141,
2254
- "r1": 141,
2255
- "r2": 141,
2256
- "nnz": 2254438
2257
- },
2258
- "model.layers.26.mlp.down_proj": {
2259
- "o1": 16,
2260
- "o2": 16,
2261
- "o3": 16,
2262
- "i1": 16,
2263
- "i2": 16,
2264
- "i3": 43,
2265
- "r0": 141,
2266
- "r1": 141,
2267
- "r2": 141,
2268
- "nnz": 2254438
2269
- },
2270
- "model.layers.27.self_attn.q_proj": {
2271
- "o1": 16,
2272
- "o2": 16,
2273
- "o3": 16,
2274
- "i1": 16,
2275
- "i2": 16,
2276
- "i3": 16,
2277
- "r0": 108,
2278
- "r1": 108,
2279
- "r2": 108,
2280
- "nnz": 838860
2281
- },
2282
- "model.layers.27.self_attn.k_proj": {
2283
- "o1": 16,
2284
- "o2": 16,
2285
- "o3": 16,
2286
- "i1": 16,
2287
- "i2": 16,
2288
- "i3": 16,
2289
- "r0": 108,
2290
- "r1": 108,
2291
- "r2": 108,
2292
- "nnz": 838860
2293
- },
2294
- "model.layers.27.self_attn.v_proj": {
2295
- "o1": 16,
2296
- "o2": 16,
2297
- "o3": 16,
2298
- "i1": 16,
2299
- "i2": 16,
2300
- "i3": 16,
2301
- "r0": 108,
2302
- "r1": 108,
2303
- "r2": 108,
2304
- "nnz": 838860
2305
- },
2306
- "model.layers.27.self_attn.o_proj": {
2307
- "o1": 16,
2308
- "o2": 16,
2309
- "o3": 16,
2310
- "i1": 16,
2311
- "i2": 16,
2312
- "i3": 16,
2313
- "r0": 108,
2314
- "r1": 108,
2315
- "r2": 108,
2316
- "nnz": 838860
2317
- },
2318
- "model.layers.27.mlp.gate_proj": {
2319
- "o1": 16,
2320
- "o2": 16,
2321
- "o3": 43,
2322
- "i1": 16,
2323
- "i2": 16,
2324
- "i3": 16,
2325
- "r0": 141,
2326
- "r1": 141,
2327
- "r2": 141,
2328
- "nnz": 2254438
2329
- },
2330
- "model.layers.27.mlp.up_proj": {
2331
- "o1": 16,
2332
- "o2": 16,
2333
- "o3": 43,
2334
- "i1": 16,
2335
- "i2": 16,
2336
- "i3": 16,
2337
- "r0": 141,
2338
- "r1": 141,
2339
- "r2": 141,
2340
- "nnz": 2254438
2341
- },
2342
- "model.layers.27.mlp.down_proj": {
2343
- "o1": 16,
2344
- "o2": 16,
2345
- "o3": 16,
2346
- "i1": 16,
2347
- "i2": 16,
2348
- "i3": 43,
2349
- "r0": 141,
2350
- "r1": 141,
2351
- "r2": 141,
2352
- "nnz": 2254438
2353
- },
2354
- "model.layers.28.self_attn.q_proj": {
2355
- "o1": 16,
2356
- "o2": 16,
2357
- "o3": 16,
2358
- "i1": 16,
2359
- "i2": 16,
2360
- "i3": 16,
2361
- "r0": 108,
2362
- "r1": 108,
2363
- "r2": 108,
2364
- "nnz": 838860
2365
- },
2366
- "model.layers.28.self_attn.k_proj": {
2367
- "o1": 16,
2368
- "o2": 16,
2369
- "o3": 16,
2370
- "i1": 16,
2371
- "i2": 16,
2372
- "i3": 16,
2373
- "r0": 108,
2374
- "r1": 108,
2375
- "r2": 108,
2376
- "nnz": 838860
2377
- },
2378
- "model.layers.28.self_attn.v_proj": {
2379
- "o1": 16,
2380
- "o2": 16,
2381
- "o3": 16,
2382
- "i1": 16,
2383
- "i2": 16,
2384
- "i3": 16,
2385
- "r0": 108,
2386
- "r1": 108,
2387
- "r2": 108,
2388
- "nnz": 838860
2389
- },
2390
- "model.layers.28.self_attn.o_proj": {
2391
- "o1": 16,
2392
- "o2": 16,
2393
- "o3": 16,
2394
- "i1": 16,
2395
- "i2": 16,
2396
- "i3": 16,
2397
- "r0": 108,
2398
- "r1": 108,
2399
- "r2": 108,
2400
- "nnz": 838860
2401
- },
2402
- "model.layers.28.mlp.gate_proj": {
2403
- "o1": 16,
2404
- "o2": 16,
2405
- "o3": 43,
2406
- "i1": 16,
2407
- "i2": 16,
2408
- "i3": 16,
2409
- "r0": 141,
2410
- "r1": 141,
2411
- "r2": 141,
2412
- "nnz": 2254438
2413
- },
2414
- "model.layers.28.mlp.up_proj": {
2415
- "o1": 16,
2416
- "o2": 16,
2417
- "o3": 43,
2418
- "i1": 16,
2419
- "i2": 16,
2420
- "i3": 16,
2421
- "r0": 141,
2422
- "r1": 141,
2423
- "r2": 141,
2424
- "nnz": 2254438
2425
- },
2426
- "model.layers.28.mlp.down_proj": {
2427
- "o1": 16,
2428
- "o2": 16,
2429
- "o3": 16,
2430
- "i1": 16,
2431
- "i2": 16,
2432
- "i3": 43,
2433
- "r0": 141,
2434
- "r1": 141,
2435
- "r2": 141,
2436
- "nnz": 2254438
2437
- },
2438
- "model.layers.29.self_attn.q_proj": {
2439
- "o1": 16,
2440
- "o2": 16,
2441
- "o3": 16,
2442
- "i1": 16,
2443
- "i2": 16,
2444
- "i3": 16,
2445
- "r0": 108,
2446
- "r1": 108,
2447
- "r2": 108,
2448
- "nnz": 838860
2449
- },
2450
- "model.layers.29.self_attn.k_proj": {
2451
- "o1": 16,
2452
- "o2": 16,
2453
- "o3": 16,
2454
- "i1": 16,
2455
- "i2": 16,
2456
- "i3": 16,
2457
- "r0": 108,
2458
- "r1": 108,
2459
- "r2": 108,
2460
- "nnz": 838860
2461
- },
2462
- "model.layers.29.self_attn.v_proj": {
2463
- "o1": 16,
2464
- "o2": 16,
2465
- "o3": 16,
2466
- "i1": 16,
2467
- "i2": 16,
2468
- "i3": 16,
2469
- "r0": 108,
2470
- "r1": 108,
2471
- "r2": 108,
2472
- "nnz": 838860
2473
- },
2474
- "model.layers.29.self_attn.o_proj": {
2475
- "o1": 16,
2476
- "o2": 16,
2477
- "o3": 16,
2478
- "i1": 16,
2479
- "i2": 16,
2480
- "i3": 16,
2481
- "r0": 108,
2482
- "r1": 108,
2483
- "r2": 108,
2484
- "nnz": 838860
2485
- },
2486
- "model.layers.29.mlp.gate_proj": {
2487
- "o1": 16,
2488
- "o2": 16,
2489
- "o3": 43,
2490
- "i1": 16,
2491
- "i2": 16,
2492
- "i3": 16,
2493
- "r0": 141,
2494
- "r1": 141,
2495
- "r2": 141,
2496
- "nnz": 2254438
2497
- },
2498
- "model.layers.29.mlp.up_proj": {
2499
- "o1": 16,
2500
- "o2": 16,
2501
- "o3": 43,
2502
- "i1": 16,
2503
- "i2": 16,
2504
- "i3": 16,
2505
- "r0": 141,
2506
- "r1": 141,
2507
- "r2": 141,
2508
- "nnz": 2254438
2509
- },
2510
- "model.layers.29.mlp.down_proj": {
2511
- "o1": 16,
2512
- "o2": 16,
2513
- "o3": 16,
2514
- "i1": 16,
2515
- "i2": 16,
2516
- "i3": 43,
2517
- "r0": 141,
2518
- "r1": 141,
2519
- "r2": 141,
2520
- "nnz": 2254438
2521
- },
2522
- "model.layers.30.self_attn.q_proj": {
2523
- "o1": 16,
2524
- "o2": 16,
2525
- "o3": 16,
2526
- "i1": 16,
2527
- "i2": 16,
2528
- "i3": 16,
2529
- "r0": 108,
2530
- "r1": 108,
2531
- "r2": 108,
2532
- "nnz": 838860
2533
- },
2534
- "model.layers.30.self_attn.k_proj": {
2535
- "o1": 16,
2536
- "o2": 16,
2537
- "o3": 16,
2538
- "i1": 16,
2539
- "i2": 16,
2540
- "i3": 16,
2541
- "r0": 108,
2542
- "r1": 108,
2543
- "r2": 108,
2544
- "nnz": 838860
2545
- },
2546
- "model.layers.30.self_attn.v_proj": {
2547
- "o1": 16,
2548
- "o2": 16,
2549
- "o3": 16,
2550
- "i1": 16,
2551
- "i2": 16,
2552
- "i3": 16,
2553
- "r0": 108,
2554
- "r1": 108,
2555
- "r2": 108,
2556
- "nnz": 838860
2557
- },
2558
- "model.layers.30.self_attn.o_proj": {
2559
- "o1": 16,
2560
- "o2": 16,
2561
- "o3": 16,
2562
- "i1": 16,
2563
- "i2": 16,
2564
- "i3": 16,
2565
- "r0": 108,
2566
- "r1": 108,
2567
- "r2": 108,
2568
- "nnz": 838860
2569
- },
2570
- "model.layers.30.mlp.gate_proj": {
2571
- "o1": 16,
2572
- "o2": 16,
2573
- "o3": 43,
2574
- "i1": 16,
2575
- "i2": 16,
2576
- "i3": 16,
2577
- "r0": 141,
2578
- "r1": 141,
2579
- "r2": 141,
2580
- "nnz": 2254438
2581
- },
2582
- "model.layers.30.mlp.up_proj": {
2583
- "o1": 16,
2584
- "o2": 16,
2585
- "o3": 43,
2586
- "i1": 16,
2587
- "i2": 16,
2588
- "i3": 16,
2589
- "r0": 141,
2590
- "r1": 141,
2591
- "r2": 141,
2592
- "nnz": 2254438
2593
- },
2594
- "model.layers.30.mlp.down_proj": {
2595
- "o1": 16,
2596
- "o2": 16,
2597
- "o3": 16,
2598
- "i1": 16,
2599
- "i2": 16,
2600
- "i3": 43,
2601
- "r0": 141,
2602
- "r1": 141,
2603
- "r2": 141,
2604
- "nnz": 2254438
2605
- },
2606
- "model.layers.31.self_attn.q_proj": {
2607
- "o1": 16,
2608
- "o2": 16,
2609
- "o3": 16,
2610
- "i1": 16,
2611
- "i2": 16,
2612
- "i3": 16,
2613
- "r0": 108,
2614
- "r1": 108,
2615
- "r2": 108,
2616
- "nnz": 838860
2617
- },
2618
- "model.layers.31.self_attn.k_proj": {
2619
- "o1": 16,
2620
- "o2": 16,
2621
- "o3": 16,
2622
- "i1": 16,
2623
- "i2": 16,
2624
- "i3": 16,
2625
- "r0": 108,
2626
- "r1": 108,
2627
- "r2": 108,
2628
- "nnz": 838860
2629
- },
2630
- "model.layers.31.self_attn.v_proj": {
2631
- "o1": 16,
2632
- "o2": 16,
2633
- "o3": 16,
2634
- "i1": 16,
2635
- "i2": 16,
2636
- "i3": 16,
2637
- "r0": 108,
2638
- "r1": 108,
2639
- "r2": 108,
2640
- "nnz": 838860
2641
- },
2642
- "model.layers.31.self_attn.o_proj": {
2643
- "o1": 16,
2644
- "o2": 16,
2645
- "o3": 16,
2646
- "i1": 16,
2647
- "i2": 16,
2648
- "i3": 16,
2649
- "r0": 108,
2650
- "r1": 108,
2651
- "r2": 108,
2652
- "nnz": 838860
2653
- },
2654
- "model.layers.31.mlp.gate_proj": {
2655
- "o1": 16,
2656
- "o2": 16,
2657
- "o3": 43,
2658
- "i1": 16,
2659
- "i2": 16,
2660
- "i3": 16,
2661
- "r0": 141,
2662
- "r1": 141,
2663
- "r2": 141,
2664
- "nnz": 2254438
2665
- },
2666
- "model.layers.31.mlp.up_proj": {
2667
- "o1": 16,
2668
- "o2": 16,
2669
- "o3": 43,
2670
- "i1": 16,
2671
- "i2": 16,
2672
- "i3": 16,
2673
- "r0": 141,
2674
- "r1": 141,
2675
- "r2": 141,
2676
- "nnz": 2254438
2677
- },
2678
- "model.layers.31.mlp.down_proj": {
2679
- "o1": 16,
2680
- "o2": 16,
2681
- "o3": 16,
2682
- "i1": 16,
2683
- "i2": 16,
2684
- "i3": 43,
2685
- "r0": 141,
2686
- "r1": 141,
2687
- "r2": 141,
2688
- "nnz": 2254438
2689
- }
2690
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
llama2-7b/state_dict_fp16_healed.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:1ed564af12d5e507abb68b7b5df13ef49b30fe23e11dd23b709b2baf488072e8
3
- size 10637037868
 
 
 
 
llama2-7b/tensor_formats.py DELETED
@@ -1,761 +0,0 @@
1
- from __future__ import annotations
2
-
3
- """General N-core tensor formats for a linear layer: tensor train (MPO) and tensor ring.
4
-
5
- WHY THIS EXISTS. The 2-core and 3-core results say the tensorisation step, not the healing
6
- step, is where the leverage is: at equal parameters the 3-core beat the 2-core AFTER healing
7
- at both budgets tried (4.72B: 38.94 vs ~36.5 full MMLU at 60M tokens; 5.74B: 44.91 vs 41.54
8
- probe), and it got there on roughly a quarter of the tokens. That invites two orthogonal
9
- questions this module exists to answer: how many cores, and which topology.
10
-
11
- TENSOR TRAIN vs TENSOR RING. A TT/MPO pins the two boundary bonds to 1 -- the chain has open
12
- ends. A tensor ring closes the chain: core k has shape (r_{k-1}, d_k, r_k) with r_N == r_0,
13
- and the weight is the TRACE of the cyclic product. TT is exactly the TR special case r_0 = 1,
14
- so at equal ranks TR is strictly more expressive. It is also strictly harder to fit: TT-SVD
15
- is a sequence of Eckart-Young-optimal truncations, while a ring has no such closed form, so
16
- TR is fitted here by TT initialisation followed by alternating least squares. That asymmetry
17
- matters when reading results -- a TR loss could mean "the format is worse" OR "the fit did
18
- not converge", so `fit_tr` reports its per-sweep error curve and refuses to return a fit
19
- worse than its own TT initialisation.
20
-
21
- INDEX CONVENTION (shared with mpo2.py / mpo3.py). Output mode k travels with input mode k:
22
-
23
- R(W)[d_1, ..., d_N] = W[(o_1..o_N), (i_1..i_N)], d_k = (o_k, i_k) flattened
24
-
25
- Every mode must be > 1. A mode of 1 collapses a site and silently reduces the core count,
26
- which would make an "N-core" result actually be an (N-1)-core one.
27
-
28
- WHITENING is diagonal and factorised across the N input modes, as in mpo3.py: the whitening
29
- is then a mode-local rescaling that survives the fit exactly and un-whitens by division.
30
- """
31
-
32
- import torch
33
-
34
-
35
- # --------------------------------------------------------------------------- rearrangement
36
- def rearrange_n(W: torch.Tensor, osplit, isplit) -> torch.Tensor:
37
- """W[(o_1..o_N),(i_1..i_N)] -> R[d_1,...,d_N] with d_k = o_k*i_k."""
38
- n = len(osplit)
39
- if len(isplit) != n:
40
- raise ValueError("output and input must be split into the same number of modes")
41
- perm = [j for k in range(n) for j in (k, n + k)]
42
- return (W.reshape(*osplit, *isplit).permute(*perm)
43
- .reshape(*[o * i for o, i in zip(osplit, isplit)]))
44
-
45
-
46
- def inv_rearrange_n(R: torch.Tensor, osplit, isplit) -> torch.Tensor:
47
- """Inverse of `rearrange_n`."""
48
- n = len(osplit)
49
- inter = [x for k in range(n) for x in (osplit[k], isplit[k])]
50
- perm = list(range(0, 2 * n, 2)) + list(range(1, 2 * n, 2))
51
- o = 1
52
- i = 1
53
- for k in range(n):
54
- o *= osplit[k]
55
- i *= isplit[k]
56
- return R.reshape(*inter).permute(*perm).reshape(o, i)
57
-
58
-
59
- def _apply_scales(R: torch.Tensor, osplit, isplit, scales, invert: bool):
60
- """Multiply (or divide) R by sqrt(d_k[i_k]) along each input mode, in place of a dense L."""
61
- n = len(osplit)
62
- shape = [x for k in range(n) for x in (osplit[k], isplit[k])]
63
- T = R.reshape(*shape)
64
- for k in range(n):
65
- s = scales[k].clamp_min(1e-12).sqrt()
66
- view = [1] * (2 * n)
67
- view[2 * k + 1] = isplit[k]
68
- T = T / s.view(*view) if invert else T * s.view(*view)
69
- return T.reshape(R.shape)
70
-
71
-
72
- # --------------------------------------------------------------------------- contraction
73
- def contract_tt(cores) -> torch.Tensor:
74
- """Contract TT cores [(1,d1,r1), (r1,d2,r2), ..., (r_{N-1},dN,1)] -> R[d_1,...,d_N]."""
75
- out = cores[0]
76
- for g in cores[1:]:
77
- a, b = out.shape[-1], g.shape[0]
78
- if a != b:
79
- raise ValueError(f"TT bond mismatch {a} vs {b}")
80
- out = torch.tensordot(out, g, dims=([out.dim() - 1], [0]))
81
- if out.shape[0] != 1 or out.shape[-1] != 1:
82
- raise ValueError("TT boundary bonds must be 1")
83
- return out.reshape(*out.shape[1:-1])
84
-
85
-
86
- def contract_tr(cores, chunk: int = 0) -> torch.Tensor:
87
- """Contract TR cores [(r0,d1,r1), ..., (r_{N-1},dN,r0)] -> R[d_1,...,d_N].
88
-
89
- The obvious route -- chain every tensordot, then take the diagonal over the two ring
90
- indices -- materialises (r_0, d_1..d_N, r_0), i.e. r_0^2 times the size of the ANSWER.
91
- On one LLaMA attention layer at ring rank 118 that is 934 GB, so it does not merely run
92
- slowly, it cannot run at all. Folding the last contraction and the trace into a single
93
- einsum keeps the peak at r_0 * prod(d_1..d_{N-1}) * r_{N-1}, and chunking the first mode
94
- bounds it further for the wide MLP layers.
95
- """
96
- n = len(cores)
97
- if cores[0].shape[0] != cores[-1].shape[-1]:
98
- raise ValueError(f"ring does not close: {cores[0].shape[0]} vs {cores[-1].shape[-1]}")
99
- d0 = cores[0].shape[1]
100
- if chunk and d0 > chunk:
101
- parts = []
102
- for lo in range(0, d0, chunk):
103
- sub = [cores[0][:, lo:min(lo + chunk, d0), :]] + list(cores[1:])
104
- parts.append(contract_tr(sub, chunk=0))
105
- return torch.cat(parts, dim=0)
106
- out = cores[0]
107
- for g in cores[1:-1]:
108
- out = torch.tensordot(out, g, dims=([out.dim() - 1], [0]))
109
- return torch.einsum("i...k,kdi->...d", out, cores[-1])
110
-
111
-
112
- # --------------------------------------------------------------------------- fitting
113
- def fit_tt(R: torch.Tensor, ranks):
114
- """TT-SVD with the given internal ranks. Singular values split symmetrically at each bond,
115
- matching fit_mpo2/fit_mpo3's gauge so cores stay on comparable scales (fp16-safe and
116
- kinder to a blockwise quantizer)."""
117
- dims = list(R.shape)
118
- n = len(dims)
119
- if len(ranks) != n - 1:
120
- raise ValueError(f"{n} sites need {n-1} internal ranks, got {len(ranks)}")
121
- cores, rest, rprev = [], R.reshape(dims[0], -1), 1
122
- for k in range(n - 1):
123
- M = rest.reshape(rprev * dims[k], -1)
124
- U, S, Vh = torch.linalg.svd(M, full_matrices=False)
125
- r = min(int(ranks[k]), S.shape[0])
126
- q = S[:r].clamp_min(0).sqrt()
127
- cores.append((U[:, :r] * q).reshape(rprev, dims[k], r))
128
- rest = q[:, None] * Vh[:r]
129
- rprev = r
130
- cores.append(rest.reshape(rprev, dims[-1], 1))
131
- return cores
132
-
133
-
134
- def tt_to_tr(cores):
135
- """Reinterpret TT cores as TR cores with r_0 = 1 (an exact, lossless re-labelling)."""
136
- return [c.clone() for c in cores]
137
-
138
-
139
- def _subchain(cores, k):
140
- """Contract every core except k, in cyclic order, -> E[r_k, D, r_{k-1}] with D = prod_{j!=k} d_j."""
141
- n = len(cores)
142
- order = [(k + 1 + t) % n for t in range(n - 1)]
143
- out = cores[order[0]]
144
- for j in order[1:]:
145
- out = torch.tensordot(out, cores[j], dims=([out.dim() - 1], [0]))
146
- return out
147
-
148
-
149
- def _subchain_chunk(cores, k, lo, hi):
150
- """`_subchain` with the FIRST following mode restricted to [lo, hi). The restriction rides
151
- along the contraction untouched, so concatenating the chunks reproduces `_subchain`."""
152
- n = len(cores)
153
- order = [(k + 1 + t) % n for t in range(n - 1)]
154
- out = cores[order[0]][:, lo:hi, :]
155
- for j in order[1:]:
156
- out = torch.tensordot(out, cores[j], dims=([out.dim() - 1], [0]))
157
- return out
158
-
159
-
160
- def rank_of(cores, k):
161
- """r_{k-1} * r_k for core k -- the width of that core's least-squares system."""
162
- n = len(cores)
163
- return cores[k].shape[0] * cores[k].shape[2]
164
-
165
-
166
- def _solve_maybe_cpu(AtA, AtW):
167
- """Solve AtA X = AtW, falling back to the CPU when the GPU cannot hold the factorisation.
168
-
169
- The normal-equation matrix is (r_{k-1} r_k) square, i.e. it grows as r^4. On down_proj at
170
- ring rank 219 that is 47961^2 = 9.2 GB, and the factorisation needs a second copy -- no
171
- amount of chunking the data helps, because this matrix does not depend on the data size.
172
- Cholesky is tried first (the Gram is symmetric positive definite after the ridge) and
173
- needs only one triangular factor; if even that does not fit, the solve moves to the host.
174
- Slow, but a bounded slowdown beats an unbounded failure on the layers that matter most.
175
- """
176
- try:
177
- L = torch.linalg.cholesky(AtA)
178
- return torch.cholesky_solve(AtW, L)
179
- except torch.OutOfMemoryError:
180
- torch.cuda.empty_cache()
181
- except Exception:
182
- pass
183
- try:
184
- return torch.linalg.solve(AtA, AtW)
185
- except torch.OutOfMemoryError:
186
- torch.cuda.empty_cache()
187
- dev = AtA.device
188
- X = torch.linalg.solve(AtA.cpu().double(), AtW.cpu().double())
189
- return X.to(device=dev, dtype=AtW.dtype)
190
-
191
-
192
- def tr_rel_err(cores, R: torch.Tensor, chunk_elems: int = 250_000_000) -> float:
193
- """||contract_tr(cores) - R||_F / ||R||_F, accumulated in chunks.
194
-
195
- Materialising the reconstruction to subtract it needs the reconstruction AND the einsum's
196
- intermediate at once; on down_proj at ring rank 155 that intermediate is (155, 64, 688,
197
- 256) = 7 GB even with the first mode already chunked. Accumulating the squared error one
198
- slice at a time keeps only one slice live, and the chunk width is derived from an element
199
- budget rather than a hard-coded constant -- a constant is exactly what silently failed to
200
- engage when a mode happened to equal it.
201
- """
202
- n = len(cores)
203
- d0 = cores[0].shape[1]
204
- mid = 1
205
- for k in range(1, n - 1):
206
- mid *= cores[k].shape[1]
207
- per_unit = max(1, mid * cores[0].shape[0] * cores[-1].shape[0])
208
- step = max(1, min(d0, int(chunk_elems // per_unit)))
209
- acc = 0.0
210
- for lo in range(0, d0, step):
211
- hi = min(lo + step, d0)
212
- sub = [cores[0][:, lo:hi, :]] + list(cores[1:])
213
- diff = contract_tr(sub) - R[lo:hi]
214
- acc += float(diff.pow(2).sum())
215
- del sub, diff
216
- return (acc ** 0.5) / float(R.norm())
217
-
218
-
219
- def fit_tr(R: torch.Tensor, ranks, sweeps: int = 30, tol: float = 1e-8, verbose: bool = False,
220
- seed: int = 1234, restarts: int = 2, solver: str = "lstsq",
221
- chunk: int = 0, chunk_elems: int = 250_000_000):
222
- """Fit a tensor ring by TT initialisation + alternating least squares.
223
-
224
- ranks = [r_0, r_1, ..., r_{N-1}] with r_N == r_0. The TT init uses r_0 = 1 (a legal ring),
225
- then each ALS sweep re-solves every core in closed form given the others; the ring's extra
226
- boundary rank is opened up by padding the initialisation. Because ALS on a ring is not
227
- guaranteed to improve monotonically from an arbitrary start, the best iterate seen is kept
228
- and the routine REFUSES to return anything worse than its own TT initialisation -- without
229
- that guard a fitting failure would be indistinguishable from the format being worse, which
230
- is precisely the comparison this module exists to make.
231
- """
232
- n = len(R.shape)
233
- if len(ranks) != n:
234
- raise ValueError(f"a ring over {n} sites needs {n} ranks (r_0..r_{{N-1}})")
235
- dims = list(R.shape)
236
- rnorm = float(R.norm())
237
- rnorm2 = float(R.pow(2).sum())
238
- last_sq = None
239
- tt = fit_tt(R, ranks[1:])
240
- base_err = float((contract_tt(tt) - R).norm() / R.norm())
241
- r0 = int(ranks[0])
242
- gen = torch.Generator(device="cpu").manual_seed(seed)
243
-
244
- def padded_init():
245
- """TT solution embedded in the ring, with the OPENED bond seeded on BOTH ends.
246
-
247
- Seeding only one end is not enough and is a silent failure: core 0's least-squares
248
- environment is built from cores 1..N-1, whose ring index comes from core N-1's
249
- trailing bond. If that stays zero, the environment vanishes on exactly the components
250
- the ring adds, every ALS solve returns zero there, and the fit never leaves the TT
251
- solution -- it looks like "the ring buys nothing" rather than "the ring was never
252
- initialised".
253
- """
254
- out = []
255
- for k, c in enumerate(tt):
256
- a, d, b = c.shape
257
- na = r0 if k == 0 else a
258
- nb = r0 if k == n - 1 else b
259
- g = torch.zeros(na, d, nb, dtype=R.dtype, device=R.device)
260
- g[:a, :, :b] = c
261
- eps = 0.05 * float(c.norm()) / max(1, c.numel()) ** 0.5
262
- if k == 0 and na > a:
263
- g[a:, :, :] = eps * torch.randn(na - a, d, nb, generator=gen,
264
- dtype=R.dtype).to(R.device)
265
- if k == n - 1 and nb > b:
266
- g[:, :, b:] = eps * torch.randn(na, d, nb - b, generator=gen,
267
- dtype=R.dtype).to(R.device)
268
- out.append(g)
269
- return out
270
-
271
- def random_init(g=None):
272
- """i.i.d. cores whose ring contraction has the right variance.
273
-
274
- The scale is derived, not guessed. For cores with i.i.d. N(0, s^2) entries the trace
275
- of the cyclic product is a sum of prod_k r_k independent products of N Gaussians, so
276
- Var(W) ~= (prod_k r_k) * s^(2N). Matching that to the target mean square gives
277
- s = (E[R^2] / prod_k r_k)^(1/(2N)). Getting this wrong by a constant factor per core
278
- compounds to that factor^N in the product, which for N=4,5 puts the initial iterate
279
- many orders of magnitude away from the data and leaves ALS crawling.
280
- """
281
- rr = [int(ranks[k]) for k in range(n)]
282
- prod = 1.0
283
- for x in rr:
284
- prod *= x
285
- ms = float(R.pow(2).mean())
286
- s = (ms / prod) ** (1.0 / (2 * n))
287
- gg = g if g is not None else gen
288
- return [s * torch.randn(rr[k], dims[k], rr[(k + 1) % n], generator=gg,
289
- dtype=R.dtype).to(R.device) for k in range(n)]
290
-
291
- # ALS on a ring has genuine local minima -- a single start that stalls would be reported
292
- # as "the ring is not better", which is exactly the confound this module must avoid. So
293
- # several starts are run and the best kept; restarts are cheap next to the SVDs anyway.
294
- inits = [("tt_padded", padded_init)]
295
- for j in range(restarts):
296
- g = torch.Generator(device="cpu").manual_seed(seed + 977 * (j + 1))
297
- inits.append((f"random{j}", lambda g=g: random_init(g)))
298
- best, best_err, hists, winner = tt_to_tr(tt), base_err, {}, "tt_init"
299
- for init_name, init_fn in inits:
300
- cores, ihist = init_fn(), []
301
- for it in range(sweeps):
302
- for k in range(n):
303
- # The subchain leaves every d index separate and in CYCLIC order starting at
304
- # k+1. R's remaining modes must be flattened in that SAME order -- movedim
305
- # gives natural order, which for interior k is a different permutation and
306
- # would solve the least-squares problem against mismatched columns. It still
307
- # returns a finite "solution", just to the wrong system.
308
- # The chunk decision is made from SHAPES ONLY and BEFORE anything large is
309
- # built. An earlier version computed the full subchain first and then decided
310
- # whether to chunk, so the unchunked allocation always happened anyway and the
311
- # chunked branch saved nothing -- the memory-bounded path existed but could
312
- # never take effect.
313
- # Auto-size the chunk from a memory budget instead of comparing against a
314
- # fixed number. A constant threshold silently does nothing when the mode
315
- # happens to equal it -- which is exactly what let a 4-site fit try to
316
- # materialise a 134 GB subchain (mode 64, chunk 64, "64 > 64" is false).
317
- m_ = (k + 1) % n
318
- other = 1
319
- for j in range(n):
320
- if j != k:
321
- other *= dims[j]
322
- per_unit = max(1, (other // max(1, dims[m_])) * rank_of(cores, k))
323
- auto = max(1, min(dims[m_], int(chunk_elems // per_unit)))
324
- use_chunk = auto < dims[m_]
325
- if solver == "normal" and use_chunk:
326
- # Memory-bounded normal equations. Materialising the subchain costs
327
- # r_k * (prod_{j!=k} d_j) * r_{k-1} floats -- on down_proj at these ranks
328
- # that is (155, 688, 256, 155) = 17 GB, so the unchunked path cannot run
329
- # on the MLP layers at all. The Gram and right-hand side are sums over the
330
- # flattened "others" index, so they accumulate exactly over slices of the
331
- # first subchain mode; this is the same arithmetic, not an approximation.
332
- m = m_
333
- d_next = dims[m]
334
- rk_, rkm1_ = None, None
335
- AtA = AtW = None
336
- Rp = R.permute(*([k] + [(k + 1 + t) % n for t in range(n - 1)]))
337
- for lo in range(0, d_next, auto):
338
- hi = min(lo + auto, d_next)
339
- Ec = _subchain_chunk(cores, k, lo, hi)
340
- rk_, rkm1_ = Ec.shape[0], Ec.shape[-1]
341
- Ec = Ec.reshape(rk_, -1, rkm1_)
342
- Ac = Ec.permute(1, 2, 0).reshape(Ec.shape[1], rkm1_ * rk_)
343
- Wc = Rp[:, lo:hi].reshape(dims[k], -1)
344
- g = Ac.transpose(0, 1) @ Ac
345
- h = Ac.transpose(0, 1) @ Wc.transpose(0, 1)
346
- AtA = g if AtA is None else AtA + g
347
- AtW = h if AtW is None else AtW + h
348
- del Ec, Ac, Wc, g, h
349
- lam = 1e-8 * float(AtA.diagonal().abs().mean().clamp_min(1e-30))
350
- AtA_r = AtA + lam * torch.eye(AtA.shape[0], dtype=AtA.dtype,
351
- device=AtA.device)
352
- G = _solve_maybe_cpu(AtA_r, AtW)
353
- del AtA_r
354
- # ||R - A G||^2 = ||R||^2 - 2<AtW, G> + <G, AtA G>, all from quantities
355
- # already formed. Computing it by contracting the whole ring instead would
356
- # cost one full O(prod d_k) tensor per sweep per core.
357
- sq = rnorm2 - 2.0 * float((AtW * G).sum()) + float((G * (AtA @ G)).sum())
358
- last_sq = max(sq, 0.0)
359
- rkm1, rk = rkm1_, rk_
360
- else:
361
- # ONE branch builds A/Wk and ONE consumes them. Splitting the decision
362
- # across two separate if/else chains left the non-chunked solve reachable
363
- # after the chunked path had run, where `A` is whatever the PREVIOUS core
364
- # left behind -- a silent wrong answer rather than an error.
365
- E = _subchain(cores, k) # (r_k, d_{k+1},..,d_{k-1}, r_{k-1})
366
- rk, rkm1 = E.shape[0], E.shape[-1]
367
- E = E.reshape(rk, -1, rkm1)
368
- D = E.shape[1]
369
- A = E.permute(1, 2, 0).reshape(D, rkm1 * rk)
370
- perm = [k] + [(k + 1 + t) % n for t in range(n - 1)]
371
- # R's remaining modes must be flattened in the subchain's CYCLIC order --
372
- # movedim gives natural order, which for interior k solves the
373
- # least-squares problem against mismatched columns and still returns a
374
- # finite "solution", just to the wrong system.
375
- Wk = R.permute(*perm).reshape(dims[k], D)
376
- del E
377
- if solver == "normal":
378
- AtA = A.transpose(0, 1) @ A
379
- lam = 1e-8 * float(AtA.diagonal().abs().mean().clamp_min(1e-30))
380
- AtA += lam * torch.eye(AtA.shape[0], dtype=A.dtype, device=A.device)
381
- G = _solve_maybe_cpu(AtA, A.transpose(0, 1) @ Wk.transpose(0, 1))
382
- del AtA
383
- else:
384
- G = torch.linalg.lstsq(A, Wk.transpose(0, 1)).solution
385
- last_sq = None
386
- del A, Wk
387
- cores[k] = (G.transpose(0, 1).reshape(dims[k], rkm1, rk)
388
- .permute(1, 0, 2).contiguous())
389
- err = (float(last_sq) ** 0.5 / rnorm if last_sq is not None
390
- else float((contract_tr(cores, chunk=chunk) - R).norm() / rnorm))
391
- ihist.append(round(err, 8))
392
- if verbose:
393
- print(f" TR-ALS[{init_name}] sweep {it+1}: rel err {err:.6f}", flush=True)
394
- if err < best_err - 1e-12:
395
- best, best_err, winner = [c.clone() for c in cores], err, init_name
396
- if it and abs(ihist[-2] - err) < tol * max(err, 1e-12):
397
- break
398
- hists[init_name] = ihist
399
- # The algebraic residual is a difference of nearly equal large numbers; at a relative
400
- # error of ~1e-2 it has already lost most of its fp32 significance. It is fine for
401
- # steering ALS but not for the number that gets reported, so the winner is re-measured
402
- # once by explicit contraction.
403
- best_err = tr_rel_err(best, R)
404
- if best_err > base_err:
405
- return tt_to_tr(tt), base_err, {"fell_back_to_tt": True, "tt_err": base_err,
406
- "winner": "tt_init", "als_hist": hists}
407
- return best, best_err, {"fell_back_to_tt": winner == "tt_init", "tt_err": base_err,
408
- "winner": winner, "als_hist": hists}
409
-
410
-
411
- # --------------------------------------------------------------------------- bookkeeping
412
- def tt_params(dims, ranks) -> int:
413
- r = [1] + [int(x) for x in ranks] + [1]
414
- return sum(r[k] * dims[k] * r[k + 1] for k in range(len(dims)))
415
-
416
-
417
- def tr_params(dims, ranks) -> int:
418
- r = [int(x) for x in ranks]
419
- n = len(dims)
420
- return sum(r[k] * dims[k] * r[(k + 1) % n] for k in range(n))
421
-
422
-
423
- def tt_rank_ceilings(dims):
424
- """r_k <= min(prod_{j<=k} d_j, prod_{j>k} d_j): above that the truncation is vacuous."""
425
- n, out = len(dims), []
426
- for k in range(n - 1):
427
- left = 1
428
- for j in range(k + 1):
429
- left *= dims[j]
430
- right = 1
431
- for j in range(k + 1, n):
432
- right *= dims[j]
433
- out.append(min(left, right))
434
- return out
435
-
436
-
437
- def ranks_for_budget(dims, budget: int, ring: bool, alpha: float = 1.0):
438
- """Largest ranks fitting `budget`, shaped by r_k proportional to ceiling_k ** alpha.
439
-
440
- Bisection rather than a closed form: the count is linear in each rank but bilinear in
441
- adjacent pairs, and cyclic for a ring, so there is no single quadratic to solve.
442
-
443
- ALPHA EXISTS BECAUSE EQUAL FILL FRACTION IS WRONG WHEN THE CEILINGS ARE LOPSIDED. For a
444
- 4-site split of 4096 the ceilings are (64, 4096, 64): alpha=1 spends the budget on the
445
- huge middle bond and returns ranks like (36, 2327, 36), strangling the two bonds that
446
- actually gate information flow through the chain. alpha=0 gives every bond the same rank
447
- instead. Neither is right a priori, so the caller is expected to MEASURE across alpha --
448
- the reconstruction error, not the heuristic, decides.
449
- """
450
- ceil_tt = tt_rank_ceilings(dims)
451
- ceils = ([min(ceil_tt)] + ceil_tt) if ring else ceil_tt
452
- fn = tr_params if ring else tt_params
453
- mx = max(ceils)
454
- shape = [(c / mx) ** alpha for c in ceils]
455
-
456
- def build(t):
457
- return [max(1, min(int(round(t * mx * s)), c)) for s, c in zip(shape, ceils)]
458
-
459
- lo, hi = 0.0, 1.0
460
- if fn(dims, build(1.0)) <= budget:
461
- return build(1.0)
462
- for _ in range(60):
463
- mid = (lo + hi) / 2
464
- if fn(dims, build(mid)) > budget:
465
- hi = mid
466
- else:
467
- lo = mid
468
- r = build(lo)
469
- while fn(dims, r) > budget and max(r) > 1:
470
- r[r.index(max(r))] -= 1
471
- return r
472
-
473
-
474
- def balanced_split(n: int, k: int):
475
- """Most balanced k-way factorisation of n with every factor >= 2 (spread-minimising)."""
476
- best = None
477
- def rec(rem, parts):
478
- nonlocal best
479
- if len(parts) == k - 1:
480
- if rem >= 2:
481
- cand = parts + [rem]
482
- key = max(cand) / min(cand)
483
- if best is None or key < best[0]:
484
- best = (key, cand)
485
- return
486
- d = 2
487
- while d * d <= rem or d <= rem:
488
- if d > rem:
489
- break
490
- if rem % d == 0:
491
- rec(rem // d, parts + [d])
492
- d += 1
493
- rec(n, [])
494
- if best is None:
495
- raise ValueError(f"no {k}-way factorisation of {n} with all factors >= 2")
496
- return sorted(best[1])
497
-
498
-
499
- def self_test(device: str = "cpu") -> None:
500
- torch.manual_seed(0)
501
- dt = torch.float64
502
- for osp, isp in [((4, 3), (2, 5)), ((4, 3, 2), (2, 5, 3)), ((2, 3, 2, 2), (2, 2, 3, 2))]:
503
- n = len(osp)
504
- out = 1
505
- inn = 1
506
- for x in osp:
507
- out *= x
508
- for x in isp:
509
- inn *= x
510
- W = torch.randn(out, inn, dtype=dt, device=device)
511
- R = rearrange_n(W, osp, isp)
512
- dims = [o * i for o, i in zip(osp, isp)]
513
- assert list(R.shape) == dims, (R.shape, dims)
514
- assert torch.equal(inv_rearrange_n(R, osp, isp), W), f"{n}-site rearrange not invertible"
515
- # index identity, not just the round trip: a wrong-but-self-inverse permutation would
516
- # pass the round trip and scramble every weight silently
517
- for _ in range(40):
518
- oi = [torch.randint(0, x, (1,)).item() for x in osp]
519
- ii = [torch.randint(0, x, (1,)).item() for x in isp]
520
- ridx = tuple(oi[k] * isp[k] + ii[k] for k in range(n))
521
- orow = 0
522
- for k in range(n):
523
- orow = orow * osp[k] + oi[k]
524
- ocol = 0
525
- for k in range(n):
526
- ocol = ocol * isp[k] + ii[k]
527
- assert R[ridx] == W[orow, ocol], f"{n}-site rearrange index map wrong"
528
-
529
- ceils = tt_rank_ceilings(dims)
530
- cores = fit_tt(R, ceils)
531
- e = float((contract_tt(cores) - R).norm() / R.norm())
532
- assert e < 1e-10, f"{n}-site full-rank TT is not exact ({e:.3e})"
533
- assert tt_params(dims, ceils) == sum(c.numel() for c in cores), "tt_params disagrees"
534
-
535
- if n >= 3:
536
- trunc = [max(1, c // 2) for c in ceils]
537
- cores = fit_tt(R, trunc)
538
- assert tt_params(dims, trunc) == sum(c.numel() for c in cores)
539
- for a in (0.0, 0.5, 1.0):
540
- rr = ranks_for_budget(dims, tt_params(dims, trunc), ring=False, alpha=a)
541
- assert tt_params(dims, rr) <= tt_params(dims, trunc), \
542
- f"TT budget solver overran at alpha={a}"
543
- # a ring at the SAME parameter budget must not do worse than the TT it starts from
544
- budget = tt_params(dims, trunc)
545
- rring = ranks_for_budget(dims, budget, ring=True)
546
- assert tr_params(dims, rring) <= budget, "TR budget solver overran"
547
- _c, err_tr, info = fit_tr(R, rring, sweeps=6)
548
- assert err_tr <= info["tt_err"] + 1e-12, "fit_tr returned worse than its own TT init"
549
- assert abs(float((contract_tr(_c) - R).norm() / R.norm()) - err_tr) < 1e-8, \
550
- "reported TR error does not match the returned cores"
551
-
552
- ring_chk = [torch.randn(3, 5, 4, dtype=dt), torch.randn(4, 6, 2, dtype=dt),
553
- torch.randn(2, 7, 3, dtype=dt)]
554
- full = contract_tr(ring_chk)
555
- assert torch.allclose(full, contract_tr(ring_chk, chunk=2), atol=1e-12), \
556
- "chunked contract_tr disagrees with the unchunked one"
557
- ref = torch.einsum("iaj,jbk,kci->abc", *ring_chk)
558
- assert torch.allclose(full, ref, atol=1e-12), "contract_tr is not the cyclic trace"
559
-
560
- # a ring with r_0 > 1 must be able to represent things a TT of the same internal ranks cannot
561
- dims = [6, 6, 6]
562
- torch.manual_seed(1)
563
- ring = [torch.randn(3, 6, 3, dtype=dt) for _ in range(3)]
564
- Rr = contract_tr(ring)
565
- tt_same = fit_tt(Rr, [3, 3])
566
- e_tt = float((contract_tt(tt_same) - Rr).norm() / Rr.norm())
567
- assert e_tt > 1e-3, ("a rank-(3,3,3) ring was reproduced by a rank-(3,3) TT; the ring "
568
- f"contraction is probably not closing the loop (err {e_tt:.3e})")
569
- for slv in ("lstsq", "normal"):
570
- got, err, info = fit_tr(Rr, [3, 3, 3], sweeps=200, restarts=3, solver=slv)
571
- assert err < 1e-4, (f"TR-ALS[{slv}] failed to recover an exact rank-3 ring "
572
- f"(err {err:.3e}); winner={info['winner']} "
573
- f"tt_init={info['tt_err']:.3e}")
574
- got, err, info = fit_tr(Rr, [3, 3, 3], sweeps=200, restarts=3)
575
- # chunking changes only how the Gram is accumulated, so it must reach the same optimum
576
- _g, err_ch, _i = fit_tr(Rr, [3, 3, 3], sweeps=200, restarts=3, solver="normal", chunk=2)
577
- assert err_ch < 1e-4, f"chunked TR-ALS failed to recover an exact ring (err {err_ch:.3e})"
578
- assert err < 1e-4, (f"TR-ALS failed to recover an exact rank-3 ring (err {err:.3e}); "
579
- f"winner={info['winner']} tt_init={info['tt_err']:.3e}")
580
-
581
- assert balanced_split(4096, 3) == [16, 16, 16], balanced_split(4096, 3)
582
- assert balanced_split(4096, 2) == [64, 64]
583
- b = balanced_split(11008, 3)
584
- assert 43 in b and b[0] * b[1] * b[2] == 11008, b
585
- print("tensor_formats self-test PASSED (N-site rearrange exact & index-checked, full-rank "
586
- "TT exact, budget solvers within budget, TR >= its TT init, TR strictly beats TT on "
587
- "a true ring, balanced splits correct)")
588
-
589
-
590
- if __name__ == "__main__":
591
- self_test()
592
-
593
-
594
- # --------------------------------------------------------------------------- layers
595
- class TRLinear(torch.nn.Module):
596
- """Tensor-ring linear layer: W = Tr[G_0 G_1 ... G_{N-1}] after the shared rearrangement.
597
-
598
- NOT a LowRankLinear subclass, unlike MPO2Linear/MPO3Linear. Those two could pretend to be
599
- a pair of factors because a train has open ends; a ring genuinely has N 3-D cores and no
600
- A/B roles to borrow, so anything keyed on `.A`/`.B` (the blockwise quantizer, the bond
601
- rebalancer) has to be taught about `.cores` explicitly rather than silently doing the
602
- wrong thing to a subclass it thinks it understands.
603
-
604
- RECONSTRUCTION IS CHUNKED BY DEFAULT. Merging any two cores of a 3-site ring costs
605
- r*d_i*d_j*r entries -- on down_proj at ring rank 142 that is 7.1 GB in bf16, per layer,
606
- per forward. Chunking the first mode bounds it; the cost is more, smaller matmuls.
607
- """
608
-
609
- def __init__(self, cores, osplit, isplit, chunk_elems: int = 500_000_000):
610
- super().__init__()
611
- n = len(cores)
612
- if len(osplit) != n or len(isplit) != n:
613
- raise ValueError(f"{n} cores need {n} output and {n} input modes")
614
- if min(list(osplit) + list(isplit)) < 2:
615
- raise ValueError(f"degenerate split o={tuple(osplit)} i={tuple(isplit)}: a mode of "
616
- f"1 collapses a site, making this an (N-1)-site network")
617
- for k in range(n):
618
- if cores[k].shape[1] != osplit[k] * isplit[k]:
619
- raise ValueError(f"core {k} has mode {cores[k].shape[1]}, expected "
620
- f"{osplit[k]}*{isplit[k]}")
621
- if cores[k].shape[2] != cores[(k + 1) % n].shape[0]:
622
- raise ValueError(f"ring does not close between cores {k} and {(k+1)%n}")
623
- self.cores = torch.nn.ParameterList([torch.nn.Parameter(c) for c in cores])
624
- self.osplit, self.isplit = tuple(int(x) for x in osplit), tuple(int(x) for x in isplit)
625
- o = i = 1
626
- for k in range(n):
627
- o *= self.osplit[k]
628
- i *= self.isplit[k]
629
- self.out_features, self.in_features = o, i
630
- self.ranks = tuple(int(c.shape[0]) for c in cores)
631
- # Elements per unit of the FIRST mode in contract_tr's peak intermediate, which is
632
- # (r_0, c, d_1, ..., d_{n-2}, r_{n-1}) -- the LAST core's mode is contracted away by
633
- # the closing einsum and must not be counted. Including it over-estimated the cost by
634
- # d_{n-1} (256x on an MLP layer), which drove the chunk to 1 and would have made every
635
- # MLP forward 256 sequential contractions instead of 8.
636
- mid = 1
637
- for k in range(1, n - 1):
638
- mid *= cores[k].shape[1]
639
- per_unit = max(1, mid * self.ranks[0] * self.ranks[-1])
640
- self._chunk = max(1, min(cores[0].shape[1], int(chunk_elems // per_unit)))
641
- # non-persistent: never wants to show up in a checkpoint, and materialize_() is an
642
- # inference-only opt-in (see below), so a freshly-loaded layer must start uncached.
643
- self.register_buffer("_cached_weight", None, persistent=False)
644
-
645
- def weight(self) -> torch.Tensor:
646
- if self._cached_weight is not None:
647
- return self._cached_weight
648
- cs = list(self.cores)
649
- R = contract_tr(cs, chunk=self._chunk if self._chunk < cs[0].shape[1] else 0)
650
- return inv_rearrange_n(R, self.osplit, self.isplit)
651
-
652
- @torch.no_grad()
653
- def materialize_(self):
654
- """Cache the dense weight once. INFERENCE ONLY -- gradients cannot flow through the
655
- cached tensor back to the cores, so this must never be called before/during training.
656
-
657
- forward() calls weight() on every call, which redoes the full chunked ring
658
- contraction (two tensor contractions + a trace per core) from scratch. That is fine
659
- for a single forward pass, but autoregressive generation calls forward() once per
660
- layer per GENERATED TOKEN with the cores completely unchanged between calls. Measured
661
- on tr3_ring4b04_h1's GSM8K eval before this existed: 209s/item, ETA 73h for 1319
662
- items -- every one of ~200 generated tokens re-contracted all 224 ring layers.
663
- A train layer (LowRankLinear/MPO2Linear) never had this problem: F.linear(F.linear(x,
664
- B), A) applies the two factors straight to the activation and never materializes W at
665
- all, so there was nothing to cache there.
666
- """
667
- self._cached_weight = self.weight().detach().clone()
668
-
669
- def unmaterialize_(self):
670
- self._cached_weight = None
671
-
672
- def forward(self, x):
673
- return torch.nn.functional.linear(x, self.weight().to(x.dtype))
674
-
675
- def extra_repr(self) -> str:
676
- return (f"o={self.osplit}, i={self.isplit}, ranks={self.ranks}, "
677
- f"chunk={self._chunk}, materialized={self._cached_weight is not None}")
678
-
679
-
680
- class SparseAugTRLinear(TRLinear):
681
- """TRLinear plus a sparse additive residual. Same int64-index contract as the MPO layers:
682
- `sp_idx` must NEVER be cast to a floating dtype -- a `.half()` on a 45M-scale index
683
- silently rounds it to a different, valid-looking position."""
684
-
685
- def __init__(self, cores, osplit, isplit, sp_idx, sp_val,
686
- chunk_elems: int = 500_000_000):
687
- super().__init__(cores, osplit, isplit, chunk_elems)
688
- if sp_idx.numel() != sp_val.numel():
689
- raise ValueError(f"{sp_idx.numel()} indices for {sp_val.numel()} values")
690
- n = self.out_features * self.in_features
691
- if sp_idx.numel() and (int(sp_idx.max()) >= n or int(sp_idx.min()) < 0):
692
- raise ValueError("sparse index out of range for this layer")
693
- self.register_buffer("sp_idx", sp_idx.to(torch.int64))
694
- self.sp_val = torch.nn.Parameter(sp_val)
695
-
696
- def weight(self) -> torch.Tensor:
697
- # must check the cache here too, not just in TRLinear.weight(): materialize_() caches
698
- # THIS class's weight() (ring + sparse, since self.weight() dispatches polymorphically
699
- # from inside materialize_()), so without this check a later call would go
700
- # super().weight() -> see the cache -> return the already-sparse-augmented tensor ->
701
- # and then add the sparse residual a second time on top of it.
702
- if self._cached_weight is not None:
703
- return self._cached_weight
704
- W = super().weight()
705
- if self.sp_idx.numel():
706
- W = W.reshape(-1).index_add(0, self.sp_idx, self.sp_val.to(W.dtype)).view_as(W)
707
- return W
708
-
709
- def extra_repr(self) -> str:
710
- return super().extra_repr() + f", nnz={self.sp_idx.numel()}"
711
-
712
-
713
- def layer_self_test() -> None:
714
- torch.manual_seed(0)
715
- osp, isp = (4, 3, 2), (2, 5, 3)
716
- dims = [o * i for o, i in zip(osp, isp)]
717
- cores = [torch.randn(3, dims[0], 4), torch.randn(4, dims[1], 2), torch.randn(2, dims[2], 3)]
718
- lay = TRLinear([c.clone() for c in cores], osp, isp, chunk_elems=10 ** 9)
719
- ref = inv_rearrange_n(contract_tr(cores), osp, isp)
720
- assert torch.allclose(lay.weight(), ref, atol=1e-5), "TRLinear.weight() != contract+inverse"
721
- # chunking must not change the answer
722
- lay_c = TRLinear([c.clone() for c in cores], osp, isp, chunk_elems=1)
723
- assert lay_c._chunk < dims[0], "chunk did not engage"
724
- assert torch.allclose(lay_c.weight(), ref, atol=1e-5), "chunked forward differs"
725
- x = torch.randn(7, lay.in_features)
726
- assert torch.allclose(lay(x), x @ ref.T, atol=1e-4), "forward != x W^T"
727
- idx = torch.tensor([0, 5, lay.out_features * lay.in_features - 1], dtype=torch.int64)
728
- val = torch.tensor([1.0, -2.0, 3.0])
729
- sl = SparseAugTRLinear([c.clone() for c in cores], osp, isp, idx, val, chunk_elems=10 ** 9)
730
- d = (sl.weight() - ref).reshape(-1)
731
- assert torch.allclose(d[idx], val, atol=1e-5), "sparse residual not applied at sp_idx"
732
- assert abs(float(d.abs().sum()) - float(val.abs().sum())) < 1e-4, "residual leaked"
733
- try:
734
- TRLinear([torch.zeros(2, 4, 2), torch.zeros(2, 3, 2), torch.zeros(2, 6, 2)],
735
- (4, 1, 2), (1, 3, 3))
736
- except ValueError:
737
- pass
738
- else:
739
- raise AssertionError("a degenerate split (mode = 1) was accepted")
740
- # gradients must reach every core
741
- lay.zero_grad()
742
- lay(x).sum().backward()
743
- for k, c in enumerate(lay.cores):
744
- assert c.grad is not None and float(c.grad.abs().sum()) > 0, f"core {k} got no gradient"
745
- # materialize_() must be a pure inference-speed cache: same weight, same forward output,
746
- # and -- the one bug this is actually guarding against -- the sparse residual must not
747
- # get added twice once SparseAugTRLinear.weight() is short-circuited by the cache.
748
- pre = lay.weight().clone()
749
- lay.materialize_()
750
- assert lay._cached_weight is not None, "materialize_() did not populate the cache"
751
- assert torch.allclose(lay.weight(), pre, atol=1e-6), "materialize_() changed weight()"
752
- assert torch.allclose(lay(x), x @ ref.T, atol=1e-4), "forward != x W^T after materialize_()"
753
- lay.unmaterialize_()
754
- assert lay._cached_weight is None, "unmaterialize_() left the cache populated"
755
- sl_ref = sl.weight().clone()
756
- sl.materialize_()
757
- assert torch.allclose(sl.weight(), sl_ref, atol=1e-6), \
758
- "materialized SparseAugTRLinear.weight() differs -- sparse residual likely double-added"
759
- print("TRLinear self-test PASSED (weight matches contraction, chunking exact, forward "
760
- "correct, sparse localised, degenerate split refused, all cores differentiable, "
761
- "materialize_() cache exact and idempotent)")