Hugging Face's logo

Tensor-compression
/
TR_main

tensor-decomposition
svd
low-rank
model-compression
knowledge-distillation
Model card Files
xet
 Community
Rhett1 commited on
Commit
9211fde
·
verified ·
1 Parent(s): 80f468f

reorg: remove superseded path tt3-llama2-7b/

Browse files
Files changed (7) hide show
  1. tt3-llama2-7b/README.md +0 -106
  2. tt3-llama2-7b/config.json +0 -31
  3. tt3-llama2-7b/provenance.json +0 -2716
  4. tt3-llama2-7b/ranks.json +0 -2690
  5. tt3-llama2-7b/state_dict_fp16_healed.pt +0 -3
  6. tt3-llama2-7b/state_dict_fp16_unhealed.pt +0 -3
  7. tt3-llama2-7b/tensor_formats.py +0 -890
tt3-llama2-7b/README.md DELETED
@@ -1,106 +0,0 @@
1
- ---
2
- license: llama2
3
- base_model: NousResearch/Llama-2-7b-chat-hf
4
- tags:
5
- - tensor-decomposition
6
- - tensor-train
7
- - model-compression
8
- - llama2
9
- ---
10
-
11
- # TT-3 control: LLaMA-2-7B-chat @ 4.02B — **topology control** — unhealed and healed weights both included
12
-
13
- A 3-site **Tensor Train** (open chain, boundary bond pinned to `r_0 = 1`) at the *same*
14
- per-layer parameter budget as [`llama2-7b/`](../llama2-7b)'s Tensor Ring. It exists to answer
15
- one question — *does closing the ring help, at equal parameters?* — and nothing else.
16
-
17
- ## ⚠️ Read before comparing this to anything
18
-
19
- - **This checkpoint has NOT been healed.** Every other directory in this repository
20
- (`llama2-7b/`, `qwen2.5-3b/`, `svd-llm/`, `svd-llm-v2/`) received 60M tokens of knowledge
21
- distillation. This one received none. Putting its scores in the same table as theirs, or
22
- next to the dense teacher, compares two different things.
23
- - It is a **control arm**, not a model anyone should deploy. It is published so the ring-vs-train
24
- comparison can be reproduced, not because it is good.
25
-
26
- ## What is matched, and what is not
27
-
28
- Matched to `llama2-7b/` layer by layer, not merely in aggregate: the per-layer core budget is
29
- copied from that run's own `ranks.json`, together with the same mode splits, the same pairing,
30
- the same activation weighting (separable diagonal Gram), the same calibration data, the same
31
- sparse density, and the same seed. Only the boundary bond differs.
32
-
33
- | | Ring (`llama2-7b/`) | Train (here) |
34
- |---|---|---|
35
- | Params | 4.0231B | **4.0207B** (−0.06%) |
36
- | Reduction | 40.30% | **40.33%** |
37
- | q/k/v/o ranks | (108, 108, 108) | **(1, 186, 186)** |
38
- | gate/up/down ranks | (141, 141, 141) | **(1, 256, 359)** |
39
-
40
- **The two TT bonds are deliberately unequal.** With `r_0 = 1` they have different ceilings
41
- (`r_1 ≤ d_0`, `r_2 ≤ d_2`, five-fold apart on the MLP shapes), so forcing `r_1 = r_2` would
42
- have asked for ranks above the cap and handed the control a budget it could not spend. The
43
- allocation here is the strongest TT that fits the budget.
44
-
45
- **ALS did not run on 143 of the 224 layers.** The alternating-least-squares normal equations
46
- are `(r_k · r_{k+1})²` and grow as `r⁴` independently of the data, so pinning `r_0 = 1` forces
47
- the same budget into two bonds instead of three: the ring's MLP cores need a 1.5 GB system,
48
- the train's need 31.8 GB, which exceeds a 48 GB card. Those layers use the TT-SVD solution
49
- instead. Measured separately on this project's layer sweeps, ALS is worth **0.0–0.5%** to a
50
- 3-site train (against the 10–47% the closing bond itself is worth), so the control is close to
51
- its best achievable fit — but the gap is real and is stated here rather than left implicit.
52
-
53
- ## Loading
54
-
55
- ```python
56
- import json, torch
57
- from transformers import AutoModelForCausalLM
58
- from tensor_formats import TRLinear, SparseAugTRLinear # shipped in this directory
59
-
60
- model = AutoModelForCausalLM.from_pretrained(
61
- "NousResearch/Llama-2-7b-chat-hf",
62
- revision="351844e75ed0bcbbe3f10671b3c808d2b83894ee", torch_dtype=torch.float16)
63
- ranks = json.load(open("ranks.json"))
64
- # replace each named projection with TRLinear / SparseAugTRLinear built from its spec,
65
- # then load state_dict_fp16_unhealed.pt with strict=False and assert 0 missing / 0 unexpected.
66
- # For evaluation call materialize_() on every ring layer first -- without it each forward
67
- # re-contracts the chain and generation is ~100x slower.
68
- ```
69
-
70
- `provenance.json` carries the per-layer ranks, reconstruction errors, which layers skipped
71
- ALS, and the checkpoint sha256.
72
-
73
- ---
74
-
75
- ## Healed weights added: `state_dict_fp16_healed.pt`
76
-
77
- The original `state_dict_fp16_unhealed.pt` is unchanged and still present. This file is the
78
- same network after **60M tokens of pure forward-KL distillation** from the unmodified dense
79
- teacher (alpha=1, no gold labels) — the same healing the TR-3 arm received, so the two can be
80
- compared after healing and not only at compression time.
81
-
82
- | | TT-3 (this dir) | TR-3 (ring) |
83
- |---|---:|---:|
84
- | held-out KL, compressed | 6.6868 | 0.6932 |
85
- | held-out KL, healed | **0.4077** | **0.1822** |
86
- | MMLU, compressed | 27.02% | 29.34% |
87
- | MMLU, healed | **33.38%** | **43.82%** |
88
- | teacher MMLU (same probe) | 47.54% | 47.50% |
89
-
90
- The ring is ahead at both stages: roughly 9x lower KL before healing and ~2.3x after, with
91
- 10.44 points more MMLU, at a parameter budget matched to within 0.1%.
92
-
93
- ### Two confounds, stated rather than buried
94
-
95
- **1. Batch size.** This healing used `tokens_per_step = 32768`; the TR-3 arms used 16384. Both
96
- saw the same 60M tokens from the same seeded mixture, so the data is matched, but the number of
97
- optimiser updates and the effective LR schedule are not. The **compression-stage** gap (the
98
- first row of the table) predates healing entirely and is free of this confound; the healed rows
99
- are not.
100
-
101
- **2. ALS coverage.** `provenance.json` records that this arm fell back to TT-SVD without ALS on
102
- many layers because the normal equations did not fit in memory, while the TR-3 arm ran 20 ALS
103
- sweeps everywhere. That asymmetry is real but is **not** what drives the gap, and the arm's own
104
- per-layer data shows why: within this checkpoint, layers that got ALS and layers that did not
105
- differ by ~0.02 in whitened relative error, while the TT-vs-TR difference is ~0.21 — an order of
106
- magnitude larger, and the same 1.56x ratio on both models.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
tt3-llama2-7b/config.json DELETED
@@ -1,31 +0,0 @@
1
- {
2
- "_name_or_path": "NousResearch/Llama-2-7b-chat-hf",
3
- "architectures": [
4
- "LlamaForCausalLM"
5
- ],
6
- "attention_bias": false,
7
- "attention_dropout": 0.0,
8
- "bos_token_id": 1,
9
- "eos_token_id": 2,
10
- "head_dim": 128,
11
- "hidden_act": "silu",
12
- "hidden_size": 4096,
13
- "initializer_range": 0.02,
14
- "intermediate_size": 11008,
15
- "max_position_embeddings": 4096,
16
- "mlp_bias": false,
17
- "model_type": "llama",
18
- "num_attention_heads": 32,
19
- "num_hidden_layers": 32,
20
- "num_key_value_heads": 32,
21
- "pad_token_id": 0,
22
- "pretraining_tp": 1,
23
- "rms_norm_eps": 1e-05,
24
- "rope_scaling": null,
25
- "rope_theta": 10000.0,
26
- "tie_word_embeddings": false,
27
- "torch_dtype": "float16",
28
- "transformers_version": "4.46.3",
29
- "use_cache": true,
30
- "vocab_size": 32000
31
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
tt3-llama2-7b/provenance.json DELETED
@@ -1,2716 +0,0 @@
1
- {
2
- "kind": "compress_generic",
3
- "structure": "train",
4
- "model_id": "NousResearch/Llama-2-7b-chat-hf",
5
- "model_revision": "351844e75ed0bcbbe3f10671b3c808d2b83894ee",
6
- "ranks_template": "/work1/zixuan/checkpoints/Compactif/tr3_ring4b04/ranks.json",
7
- "budget_note": "per-layer core budget copied EXACTLY from the template's own core count, and the same sparse nnz, so the arms are matched layer by layer rather than only in aggregate. The matrix arm has no sparse-augmented layer type here, so its nnz share is spent on extra rank instead.",
8
- "index_storage_caveat": "matched on PARAMETER count. sp_idx is a buffer, not a Parameter, so the tensor arms additionally store nnz int64 indices that this count excludes. In BYTES the tensor arms are therefore larger, and any size claim must say which of the two conventions it uses.",
9
- "dense_total": 6738415616,
10
- "achieved_total": 4020677184,
11
- "reduction": 0.4033,
12
- "n_layers": 224,
13
- "n_biases_carried": 0,
14
- "n_als_skipped_for_memory": 143,
15
- "als_note": "a layer whose (r_k r_{k+1})^2 normal equations do not fit is fitted by TT-SVD alone. Layer-level sweeps in this project bound what that costs a train at 0.1-0.5%; a RING loses far more without ALS, so this fallback is not symmetric and must be reported per arm.",
16
- "calib_blocks": 256,
17
- "seq_len": 1024,
18
- "lam_ratio": 0.0001,
19
- "seed": 1234,
20
- "tr_sweeps": 20,
21
- "tr_restarts": 1,
22
- "checkpoint": "/work1/zixuan/checkpoints/Compactif/llama2_7b_tt3_matched/state_dict_fp16.pt",
23
- "sha256": "7c8471a71d2d1328bd4dcc824f2d71fddc40d7fe3745e16212e55374649b2625",
24
- "elapsed_s": 28257.3,
25
- "timestamp": "2026-09-05T20:12:06.889047+00:00",
26
- "per_layer": {
27
- "model.layers.0.self_attn.q_proj": {
28
- "ranks": [
29
- 1,
30
- 186,
31
- 186
32
- ],
33
- "params": 8951808,
34
- "rel_err_whitened": 0.509727,
35
- "als_skipped_for_memory": true,
36
- "winner": "tt_init",
37
- "sep_rel": 0.999922
38
- },
39
- "model.layers.0.self_attn.k_proj": {
40
- "ranks": [
41
- 1,
42
- 186,
43
- 186
44
- ],
45
- "params": 8951808,
46
- "rel_err_whitened": 0.555544,
47
- "als_skipped_for_memory": true,
48
- "winner": "tt_init",
49
- "sep_rel": 0.999922
50
- },
51
- "model.layers.0.self_attn.v_proj": {
52
- "ranks": [
53
- 1,
54
- 186,
55
- 186
56
- ],
57
- "params": 8951808,
58
- "rel_err_whitened": 0.587199,
59
- "als_skipped_for_memory": true,
60
- "winner": "tt_init",
61
- "sep_rel": 0.999922
62
- },
63
- "model.layers.0.self_attn.o_proj": {
64
- "ranks": [
65
- 1,
66
- 186,
67
- 186
68
- ],
69
- "params": 8951808,
70
- "rel_err_whitened": 0.493119,
71
- "als_skipped_for_memory": true,
72
- "winner": "tt_init",
73
- "sep_rel": 0.998846
74
- },
75
- "model.layers.0.mlp.gate_proj": {
76
- "ranks": [
77
- 1,
78
- 256,
79
- 359
80
- ],
81
- "params": 23839952,
82
- "rel_err_whitened": 0.655729,
83
- "als_skipped_for_memory": true,
84
- "winner": "tt_init",
85
- "sep_rel": 0.991798
86
- },
87
- "model.layers.0.mlp.up_proj": {
88
- "ranks": [
89
- 1,
90
- 256,
91
- 359
92
- ],
93
- "params": 23839952,
94
- "rel_err_whitened": 0.657312,
95
- "als_skipped_for_memory": true,
96
- "winner": "tt_init",
97
- "sep_rel": 0.991798
98
- },
99
- "model.layers.0.mlp.down_proj": {
100
- "ranks": [
101
- 1,
102
- 256,
103
- 359
104
- ],
105
- "params": 23839952,
106
- "rel_err_whitened": 0.654624,
107
- "als_skipped_for_memory": true,
108
- "winner": "tt_init",
109
- "sep_rel": 0.996416
110
- },
111
- "model.layers.1.self_attn.q_proj": {
112
- "ranks": [
113
- 1,
114
- 186,
115
- 186
116
- ],
117
- "params": 8951808,
118
- "rel_err_whitened": 0.498104,
119
- "als_skipped_for_memory": true,
120
- "winner": "tt_init",
121
- "sep_rel": 0.999948
122
- },
123
- "model.layers.1.self_attn.k_proj": {
124
- "ranks": [
125
- 1,
126
- 186,
127
- 186
128
- ],
129
- "params": 8951808,
130
- "rel_err_whitened": 0.514971,
131
- "als_skipped_for_memory": true,
132
- "winner": "tt_init",
133
- "sep_rel": 0.999948
134
- },
135
- "model.layers.1.self_attn.v_proj": {
136
- "ranks": [
137
- 1,
138
- 186,
139
- 186
140
- ],
141
- "params": 8951808,
142
- "rel_err_whitened": 0.587503,
143
- "als_skipped_for_memory": true,
144
- "winner": "tt_init",
145
- "sep_rel": 0.999948
146
- },
147
- "model.layers.1.self_attn.o_proj": {
148
- "ranks": [
149
- 1,
150
- 186,
151
- 186
152
- ],
153
- "params": 8951808,
154
- "rel_err_whitened": 0.488805,
155
- "als_skipped_for_memory": true,
156
- "winner": "tt_init",
157
- "sep_rel": 0.995915
158
- },
159
- "model.layers.1.mlp.gate_proj": {
160
- "ranks": [
161
- 1,
162
- 256,
163
- 359
164
- ],
165
- "params": 23839952,
166
- "rel_err_whitened": 0.655364,
167
- "als_skipped_for_memory": true,
168
- "winner": "tt_init",
169
- "sep_rel": 0.901356
170
- },
171
- "model.layers.1.mlp.up_proj": {
172
- "ranks": [
173
- 1,
174
- 256,
175
- 359
176
- ],
177
- "params": 23839952,
178
- "rel_err_whitened": 0.658645,
179
- "als_skipped_for_memory": true,
180
- "winner": "tt_init",
181
- "sep_rel": 0.901356
182
- },
183
- "model.layers.1.mlp.down_proj": {
184
- "ranks": [
185
- 1,
186
- 256,
187
- 359
188
- ],
189
- "params": 23839952,
190
- "rel_err_whitened": 0.656722,
191
- "als_skipped_for_memory": true,
192
- "winner": "tt_init",
193
- "sep_rel": 1.0
194
- },
195
- "model.layers.2.self_attn.q_proj": {
196
- "ranks": [
197
- 1,
198
- 186,
199
- 186
200
- ],
201
- "params": 8951808,
202
- "rel_err_whitened": 0.612129,
203
- "als_skipped_for_memory": true,
204
- "winner": "tt_init",
205
- "sep_rel": 0.996603
206
- },
207
- "model.layers.2.self_attn.k_proj": {
208
- "ranks": [
209
- 1,
210
- 186,
211
- 186
212
- ],
213
- "params": 8951808,
214
- "rel_err_whitened": 0.607931,
215
- "als_skipped_for_memory": true,
216
- "winner": "tt_init",
217
- "sep_rel": 0.996603
218
- },
219
- "model.layers.2.self_attn.v_proj": {
220
- "ranks": [
221
- 1,
222
- 186,
223
- 186
224
- ],
225
- "params": 8951808,
226
- "rel_err_whitened": 0.644224,
227
- "als_skipped_for_memory": true,
228
- "winner": "tt_init",
229
- "sep_rel": 0.996603
230
- },
231
- "model.layers.2.self_attn.o_proj": {
232
- "ranks": [
233
- 1,
234
- 186,
235
- 186
236
- ],
237
- "params": 8951808,
238
- "rel_err_whitened": 0.573383,
239
- "als_skipped_for_memory": true,
240
- "winner": "tt_init",
241
- "sep_rel": 0.77842
242
- },
243
- "model.layers.2.mlp.gate_proj": {
244
- "ranks": [
245
- 1,
246
- 256,
247
- 359
248
- ],
249
- "params": 23839952,
250
- "rel_err_whitened": 0.657561,
251
- "als_skipped_for_memory": true,
252
- "winner": "tt_init",
253
- "sep_rel": 0.317275
254
- },
255
- "model.layers.2.mlp.up_proj": {
256
- "ranks": [
257
- 1,
258
- 256,
259
- 359
260
- ],
261
- "params": 23839952,
262
- "rel_err_whitened": 0.65907,
263
- "als_skipped_for_memory": true,
264
- "winner": "tt_init",
265
- "sep_rel": 0.317275
266
- },
267
- "model.layers.2.mlp.down_proj": {
268
- "ranks": [
269
- 1,
270
- 256,
271
- 359
272
- ],
273
- "params": 23839952,
274
- "rel_err_whitened": 0.658033,
275
- "als_skipped_for_memory": true,
276
- "winner": "tt_init",
277
- "sep_rel": 0.935064
278
- },
279
- "model.layers.3.self_attn.q_proj": {
280
- "ranks": [
281
- 1,
282
- 186,
283
- 186
284
- ],
285
- "params": 8951808,
286
- "rel_err_whitened": 0.623056,
287
- "als_skipped_for_memory": true,
288
- "winner": "tt_init",
289
- "sep_rel": 0.989948
290
- },
291
- "model.layers.3.self_attn.k_proj": {
292
- "ranks": [
293
- 1,
294
- 186,
295
- 186
296
- ],
297
- "params": 8951808,
298
- "rel_err_whitened": 0.620151,
299
- "als_skipped_for_memory": true,
300
- "winner": "tt_init",
301
- "sep_rel": 0.989948
302
- },
303
- "model.layers.3.self_attn.v_proj": {
304
- "ranks": [
305
- 1,
306
- 186,
307
- 186
308
- ],
309
- "params": 8951808,
310
- "rel_err_whitened": 0.636207,
311
- "als_skipped_for_memory": true,
312
- "winner": "tt_init",
313
- "sep_rel": 0.989948
314
- },
315
- "model.layers.3.self_attn.o_proj": {
316
- "ranks": [
317
- 1,
318
- 186,
319
- 186
320
- ],
321
- "params": 8951808,
322
- "rel_err_whitened": 0.602213,
323
- "als_skipped_for_memory": true,
324
- "winner": "tt_init",
325
- "sep_rel": 0.945499
326
- },
327
- "model.layers.3.mlp.gate_proj": {
328
- "ranks": [
329
- 1,
330
- 256,
331
- 359
332
- ],
333
- "params": 23839952,
334
- "rel_err_whitened": 0.658137,
335
- "als_skipped_for_memory": true,
336
- "winner": "tt_init",
337
- "sep_rel": 0.466085
338
- },
339
- "model.layers.3.mlp.up_proj": {
340
- "ranks": [
341
- 1,
342
- 256,
343
- 359
344
- ],
345
- "params": 23839952,
346
- "rel_err_whitened": 0.65915,
347
- "als_skipped_for_memory": true,
348
- "winner": "tt_init",
349
- "sep_rel": 0.466085
350
- },
351
- "model.layers.3.mlp.down_proj": {
352
- "ranks": [
353
- 1,
354
- 256,
355
- 359
356
- ],
357
- "params": 23839952,
358
- "rel_err_whitened": 0.658051,
359
- "als_skipped_for_memory": true,
360
- "winner": "tt_init",
361
- "sep_rel": 0.808258
362
- },
363
- "model.layers.4.self_attn.q_proj": {
364
- "ranks": [
365
- 1,
366
- 186,
367
- 186
368
- ],
369
- "params": 8951808,
370
- "rel_err_whitened": 0.62314,
371
- "als_skipped_for_memory": true,
372
- "winner": "tt_init",
373
- "sep_rel": 0.992449
374
- },
375
- "model.layers.4.self_attn.k_proj": {
376
- "ranks": [
377
- 1,
378
- 186,
379
- 186
380
- ],
381
- "params": 8951808,
382
- "rel_err_whitened": 0.627654,
383
- "als_skipped_for_memory": true,
384
- "winner": "tt_init",
385
- "sep_rel": 0.992449
386
- },
387
- "model.layers.4.self_attn.v_proj": {
388
- "ranks": [
389
- 1,
390
- 186,
391
- 186
392
- ],
393
- "params": 8951808,
394
- "rel_err_whitened": 0.649555,
395
- "als_skipped_for_memory": true,
396
- "winner": "tt_init",
397
- "sep_rel": 0.992449
398
- },
399
- "model.layers.4.self_attn.o_proj": {
400
- "ranks": [
401
- 1,
402
- 186,
403
- 186
404
- ],
405
- "params": 8951808,
406
- "rel_err_whitened": 0.600168,
407
- "als_skipped_for_memory": true,
408
- "winner": "tt_init",
409
- "sep_rel": 0.789101
410
- },
411
- "model.layers.4.mlp.gate_proj": {
412
- "ranks": [
413
- 1,
414
- 256,
415
- 359
416
- ],
417
- "params": 23839952,
418
- "rel_err_whitened": 0.657989,
419
- "als_skipped_for_memory": true,
420
- "winner": "tt_init",
421
- "sep_rel": 0.674532
422
- },
423
- "model.layers.4.mlp.up_proj": {
424
- "ranks": [
425
- 1,
426
- 256,
427
- 359
428
- ],
429
- "params": 23839952,
430
- "rel_err_whitened": 0.6592,
431
- "als_skipped_for_memory": true,
432
- "winner": "tt_init",
433
- "sep_rel": 0.674532
434
- },
435
- "model.layers.4.mlp.down_proj": {
436
- "ranks": [
437
- 1,
438
- 256,
439
- 359
440
- ],
441
- "params": 23839952,
442
- "rel_err_whitened": 0.658093,
443
- "als_skipped_for_memory": true,
444
- "winner": "tt_init",
445
- "sep_rel": 0.958782
446
- },
447
- "model.layers.5.self_attn.q_proj": {
448
- "ranks": [
449
- 1,
450
- 186,
451
- 186
452
- ],
453
- "params": 8951808,
454
- "rel_err_whitened": 0.632148,
455
- "als_skipped_for_memory": true,
456
- "winner": "tt_init",
457
- "sep_rel": 0.992362
458
- },
459
- "model.layers.5.self_attn.k_proj": {
460
- "ranks": [
461
- 1,
462
- 186,
463
- 186
464
- ],
465
- "params": 8951808,
466
- "rel_err_whitened": 0.635205,
467
- "als_skipped_for_memory": true,
468
- "winner": "tt_init",
469
- "sep_rel": 0.992362
470
- },
471
- "model.layers.5.self_attn.v_proj": {
472
- "ranks": [
473
- 1,
474
- 186,
475
- 186
476
- ],
477
- "params": 8951808,
478
- "rel_err_whitened": 0.643415,
479
- "als_skipped_for_memory": true,
480
- "winner": "tt_init",
481
- "sep_rel": 0.992362
482
- },
483
- "model.layers.5.self_attn.o_proj": {
484
- "ranks": [
485
- 1,
486
- 186,
487
- 186
488
- ],
489
- "params": 8951808,
490
- "rel_err_whitened": 0.579957,
491
- "als_skipped_for_memory": true,
492
- "winner": "tt_init",
493
- "sep_rel": 0.944942
494
- },
495
- "model.layers.5.mlp.gate_proj": {
496
- "ranks": [
497
- 1,
498
- 256,
499
- 359
500
- ],
501
- "params": 23839952,
502
- "rel_err_whitened": 0.65748,
503
- "als_skipped_for_memory": true,
504
- "winner": "tt_init",
505
- "sep_rel": 0.637456
506
- },
507
- "model.layers.5.mlp.up_proj": {
508
- "ranks": [
509
- 1,
510
- 256,
511
- 359
512
- ],
513
- "params": 23839952,
514
- "rel_err_whitened": 0.659278,
515
- "als_skipped_for_memory": true,
516
- "winner": "tt_init",
517
- "sep_rel": 0.637456
518
- },
519
- "model.layers.5.mlp.down_proj": {
520
- "ranks": [
521
- 1,
522
- 256,
523
- 359
524
- ],
525
- "params": 23839952,
526
- "rel_err_whitened": 0.658165,
527
- "als_skipped_for_memory": true,
528
- "winner": "tt_init",
529
- "sep_rel": 0.681339
530
- },
531
- "model.layers.6.self_attn.q_proj": {
532
- "ranks": [
533
- 1,
534
- 186,
535
- 186
536
- ],
537
- "params": 8951808,
538
- "rel_err_whitened": 0.630342,
539
- "als_skipped_for_memory": true,
540
- "winner": "tt_init",
541
- "sep_rel": 0.987342
542
- },
543
- "model.layers.6.self_attn.k_proj": {
544
- "ranks": [
545
- 1,
546
- 186,
547
- 186
548
- ],
549
- "params": 8951808,
550
- "rel_err_whitened": 0.632435,
551
- "als_skipped_for_memory": true,
552
- "winner": "tt_init",
553
- "sep_rel": 0.987342
554
- },
555
- "model.layers.6.self_attn.v_proj": {
556
- "ranks": [
557
- 1,
558
- 186,
559
- 186
560
- ],
561
- "params": 8951808,
562
- "rel_err_whitened": 0.637749,
563
- "als_skipped_for_memory": true,
564
- "winner": "tt_init",
565
- "sep_rel": 0.987342
566
- },
567
- "model.layers.6.self_attn.o_proj": {
568
- "ranks": [
569
- 1,
570
- 186,
571
- 186
572
- ],
573
- "params": 8951808,
574
- "rel_err_whitened": 0.611296,
575
- "als_skipped_for_memory": true,
576
- "winner": "tt_init",
577
- "sep_rel": 0.865292
578
- },
579
- "model.layers.6.mlp.gate_proj": {
580
- "ranks": [
581
- 1,
582
- 256,
583
- 359
584
- ],
585
- "params": 23839952,
586
- "rel_err_whitened": 0.656918,
587
- "als_skipped_for_memory": true,
588
- "winner": "tt_init",
589
- "sep_rel": 0.670264
590
- },
591
- "model.layers.6.mlp.up_proj": {
592
- "ranks": [
593
- 1,
594
- 256,
595
- 359
596
- ],
597
- "params": 23839952,
598
- "rel_err_whitened": 0.659345,
599
- "als_skipped_for_memory": true,
600
- "winner": "tt_init",
601
- "sep_rel": 0.670264
602
- },
603
- "model.layers.6.mlp.down_proj": {
604
- "ranks": [
605
- 1,
606
- 256,
607
- 359
608
- ],
609
- "params": 23839952,
610
- "rel_err_whitened": 0.658187,
611
- "als_skipped_for_memory": true,
612
- "winner": "tt_init",
613
- "sep_rel": 0.82053
614
- },
615
- "model.layers.7.self_attn.q_proj": {
616
- "ranks": [
617
- 1,
618
- 186,
619
- 186
620
- ],
621
- "params": 8951808,
622
- "rel_err_whitened": 0.633497,
623
- "als_skipped_for_memory": true,
624
- "winner": "tt_init",
625
- "sep_rel": 0.986975
626
- },
627
- "model.layers.7.self_attn.k_proj": {
628
- "ranks": [
629
- 1,
630
- 186,
631
- 186
632
- ],
633
- "params": 8951808,
634
- "rel_err_whitened": 0.635973,
635
- "als_skipped_for_memory": true,
636
- "winner": "tt_init",
637
- "sep_rel": 0.986975
638
- },
639
- "model.layers.7.self_attn.v_proj": {
640
- "ranks": [
641
- 1,
642
- 186,
643
- 186
644
- ],
645
- "params": 8951808,
646
- "rel_err_whitened": 0.618815,
647
- "als_skipped_for_memory": true,
648
- "winner": "tt_init",
649
- "sep_rel": 0.986975
650
- },
651
- "model.layers.7.self_attn.o_proj": {
652
- "ranks": [
653
- 1,
654
- 186,
655
- 186
656
- ],
657
- "params": 8951808,
658
- "rel_err_whitened": 0.588525,
659
- "als_skipped_for_memory": true,
660
- "winner": "tt_init",
661
- "sep_rel": 0.823973
662
- },
663
- "model.layers.7.mlp.gate_proj": {
664
- "ranks": [
665
- 1,
666
- 256,
667
- 359
668
- ],
669
- "params": 23839952,
670
- "rel_err_whitened": 0.657188,
671
- "als_skipped_for_memory": true,
672
- "winner": "tt_init",
673
- "sep_rel": 0.760929
674
- },
675
- "model.layers.7.mlp.up_proj": {
676
- "ranks": [
677
- 1,
678
- 256,
679
- 359
680
- ],
681
- "params": 23839952,
682
- "rel_err_whitened": 0.659229,
683
- "als_skipped_for_memory": true,
684
- "winner": "tt_init",
685
- "sep_rel": 0.760929
686
- },
687
- "model.layers.7.mlp.down_proj": {
688
- "ranks": [
689
- 1,
690
- 256,
691
- 359
692
- ],
693
- "params": 23839952,
694
- "rel_err_whitened": 0.658665,
695
- "als_skipped_for_memory": true,
696
- "winner": "tt_init",
697
- "sep_rel": 0.782631
698
- },
699
- "model.layers.8.self_attn.q_proj": {
700
- "ranks": [
701
- 1,
702
- 186,
703
- 186
704
- ],
705
- "params": 8951808,
706
- "rel_err_whitened": 0.630341,
707
- "als_skipped_for_memory": true,
708
- "winner": "tt_init",
709
- "sep_rel": 0.988882
710
- },
711
- "model.layers.8.self_attn.k_proj": {
712
- "ranks": [
713
- 1,
714
- 186,
715
- 186
716
- ],
717
- "params": 8951808,
718
- "rel_err_whitened": 0.635165,
719
- "als_skipped_for_memory": true,
720
- "winner": "tt_init",
721
- "sep_rel": 0.988882
722
- },
723
- "model.layers.8.self_attn.v_proj": {
724
- "ranks": [
725
- 1,
726
- 186,
727
- 186
728
- ],
729
- "params": 8951808,
730
- "rel_err_whitened": 0.645079,
731
- "als_skipped_for_memory": true,
732
- "winner": "tt_init",
733
- "sep_rel": 0.988882
734
- },
735
- "model.layers.8.self_attn.o_proj": {
736
- "ranks": [
737
- 1,
738
- 186,
739
- 186
740
- ],
741
- "params": 8951808,
742
- "rel_err_whitened": 0.597945,
743
- "als_skipped_for_memory": true,
744
- "winner": "tt_init",
745
- "sep_rel": 0.792704
746
- },
747
- "model.layers.8.mlp.gate_proj": {
748
- "ranks": [
749
- 1,
750
- 256,
751
- 359
752
- ],
753
- "params": 23839952,
754
- "rel_err_whitened": 0.657407,
755
- "als_skipped_for_memory": true,
756
- "winner": "tt_init",
757
- "sep_rel": 0.795323
758
- },
759
- "model.layers.8.mlp.up_proj": {
760
- "ranks": [
761
- 1,
762
- 256,
763
- 359
764
- ],
765
- "params": 23839952,
766
- "rel_err_whitened": 0.659187,
767
- "als_skipped_for_memory": true,
768
- "winner": "tt_init",
769
- "sep_rel": 0.795323
770
- },
771
- "model.layers.8.mlp.down_proj": {
772
- "ranks": [
773
- 1,
774
- 256,
775
- 359
776
- ],
777
- "params": 23839952,
778
- "rel_err_whitened": 0.658553,
779
- "als_skipped_for_memory": true,
780
- "winner": "tt_init",
781
- "sep_rel": 0.901674
782
- },
783
- "model.layers.9.self_attn.q_proj": {
784
- "ranks": [
785
- 1,
786
- 186,
787
- 186
788
- ],
789
- "params": 8951808,
790
- "rel_err_whitened": 0.635914,
791
- "als_skipped_for_memory": true,
792
- "winner": "tt_init",
793
- "sep_rel": 0.987279
794
- },
795
- "model.layers.9.self_attn.k_proj": {
796
- "ranks": [
797
- 1,
798
- 186,
799
- 186
800
- ],
801
- "params": 8951808,
802
- "rel_err_whitened": 0.634923,
803
- "als_skipped_for_memory": true,
804
- "winner": "tt_init",
805
- "sep_rel": 0.987279
806
- },
807
- "model.layers.9.self_attn.v_proj": {
808
- "ranks": [
809
- 1,
810
- 186,
811
- 186
812
- ],
813
- "params": 8951808,
814
- "rel_err_whitened": 0.640635,
815
- "als_skipped_for_memory": true,
816
- "winner": "tt_init",
817
- "sep_rel": 0.987279
818
- },
819
- "model.layers.9.self_attn.o_proj": {
820
- "ranks": [
821
- 1,
822
- 186,
823
- 186
824
- ],
825
- "params": 8951808,
826
- "rel_err_whitened": 0.570256,
827
- "als_skipped_for_memory": true,
828
- "winner": "tt_init",
829
- "sep_rel": 0.747023
830
- },
831
- "model.layers.9.mlp.gate_proj": {
832
- "ranks": [
833
- 1,
834
- 256,
835
- 359
836
- ],
837
- "params": 23839952,
838
- "rel_err_whitened": 0.657146,
839
- "als_skipped_for_memory": true,
840
- "winner": "tt_init",
841
- "sep_rel": 0.79099
842
- },
843
- "model.layers.9.mlp.up_proj": {
844
- "ranks": [
845
- 1,
846
- 256,
847
- 359
848
- ],
849
- "params": 23839952,
850
- "rel_err_whitened": 0.659233,
851
- "als_skipped_for_memory": true,
852
- "winner": "tt_init",
853
- "sep_rel": 0.79099
854
- },
855
- "model.layers.9.mlp.down_proj": {
856
- "ranks": [
857
- 1,
858
- 256,
859
- 359
860
- ],
861
- "params": 23839952,
862
- "rel_err_whitened": 0.658193,
863
- "als_skipped_for_memory": true,
864
- "winner": "tt_init",
865
- "sep_rel": 0.597211
866
- },
867
- "model.layers.10.self_attn.q_proj": {
868
- "ranks": [
869
- 1,
870
- 186,
871
- 186
872
- ],
873
- "params": 8951808,
874
- "rel_err_whitened": 0.633835,
875
- "als_skipped_for_memory": true,
876
- "winner": "tt_init",
877
- "sep_rel": 0.988037
878
- },
879
- "model.layers.10.self_attn.k_proj": {
880
- "ranks": [
881
- 1,
882
- 186,
883
- 186
884
- ],
885
- "params": 8951808,
886
- "rel_err_whitened": 0.62744,
887
- "als_skipped_for_memory": true,
888
- "winner": "tt_init",
889
- "sep_rel": 0.988037
890
- },
891
- "model.layers.10.self_attn.v_proj": {
892
- "ranks": [
893
- 1,
894
- 186,
895
- 186
896
- ],
897
- "params": 8951808,
898
- "rel_err_whitened": 0.646145,
899
- "als_skipped_for_memory": true,
900
- "winner": "tt_init",
901
- "sep_rel": 0.988037
902
- },
903
- "model.layers.10.self_attn.o_proj": {
904
- "ranks": [
905
- 1,
906
- 186,
907
- 186
908
- ],
909
- "params": 8951808,
910
- "rel_err_whitened": 0.616489,
911
- "als_skipped_for_memory": true,
912
- "winner": "tt_init",
913
- "sep_rel": 0.687547
914
- },
915
- "model.layers.10.mlp.gate_proj": {
916
- "ranks": [
917
- 1,
918
- 256,
919
- 359
920
- ],
921
- "params": 23839952,
922
- "rel_err_whitened": 0.656854,
923
- "als_skipped_for_memory": true,
924
- "winner": "tt_init",
925
- "sep_rel": 0.788355
926
- },
927
- "model.layers.10.mlp.up_proj": {
928
- "ranks": [
929
- 1,
930
- 256,
931
- 359
932
- ],
933
- "params": 23839952,
934
- "rel_err_whitened": 0.659101,
935
- "als_skipped_for_memory": true,
936
- "winner": "tt_init",
937
- "sep_rel": 0.788355
938
- },
939
- "model.layers.10.mlp.down_proj": {
940
- "ranks": [
941
- 1,
942
- 256,
943
- 359
944
- ],
945
- "params": 23839952,
946
- "rel_err_whitened": 0.658435,
947
- "als_skipped_for_memory": true,
948
- "winner": "tt_init",
949
- "sep_rel": 0.57857
950
- },
951
- "model.layers.11.self_attn.q_proj": {
952
- "ranks": [
953
- 1,
954
- 186,
955
- 186
956
- ],
957
- "params": 8951808,
958
- "rel_err_whitened": 0.628361,
959
- "als_skipped_for_memory": true,
960
- "winner": "tt_init",
961
- "sep_rel": 0.984854
962
- },
963
- "model.layers.11.self_attn.k_proj": {
964
- "ranks": [
965
- 1,
966
- 186,
967
- 186
968
- ],
969
- "params": 8951808,
970
- "rel_err_whitened": 0.628862,
971
- "als_skipped_for_memory": true,
972
- "winner": "tt_init",
973
- "sep_rel": 0.984854
974
- },
975
- "model.layers.11.self_attn.v_proj": {
976
- "ranks": [
977
- 1,
978
- 186,
979
- 186
980
- ],
981
- "params": 8951808,
982
- "rel_err_whitened": 0.63238,
983
- "als_skipped_for_memory": true,
984
- "winner": "tt_init",
985
- "sep_rel": 0.984854
986
- },
987
- "model.layers.11.self_attn.o_proj": {
988
- "ranks": [
989
- 1,
990
- 186,
991
- 186
992
- ],
993
- "params": 8951808,
994
- "rel_err_whitened": 0.618185,
995
- "als_skipped_for_memory": false,
996
- "winner": "tt_padded",
997
- "sep_rel": 0.699959
998
- },
999
- "model.layers.11.mlp.gate_proj": {
1000
- "ranks": [
1001
- 1,
1002
- 256,
1003
- 359
1004
- ],
1005
- "params": 23839952,
1006
- "rel_err_whitened": 0.657117,
1007
- "als_skipped_for_memory": true,
1008
- "winner": "tt_init",
1009
- "sep_rel": 0.763467
1010
- },
1011
- "model.layers.11.mlp.up_proj": {
1012
- "ranks": [
1013
- 1,
1014
- 256,
1015
- 359
1016
- ],
1017
- "params": 23839952,
1018
- "rel_err_whitened": 0.659212,
1019
- "als_skipped_for_memory": true,
1020
- "winner": "tt_init",
1021
- "sep_rel": 0.763467
1022
- },
1023
- "model.layers.11.mlp.down_proj": {
1024
- "ranks": [
1025
- 1,
1026
- 256,
1027
- 359
1028
- ],
1029
- "params": 23839952,
1030
- "rel_err_whitened": 0.658057,
1031
- "als_skipped_for_memory": true,
1032
- "winner": "tt_init",
1033
- "sep_rel": 0.559717
1034
- },
1035
- "model.layers.12.self_attn.q_proj": {
1036
- "ranks": [
1037
- 1,
1038
- 186,
1039
- 186
1040
- ],
1041
- "params": 8951808,
1042
- "rel_err_whitened": 0.641057,
1043
- "als_skipped_for_memory": false,
1044
- "winner": "tt_padded",
1045
- "sep_rel": 0.984455
1046
- },
1047
- "model.layers.12.self_attn.k_proj": {
1048
- "ranks": [
1049
- 1,
1050
- 186,
1051
- 186
1052
- ],
1053
- "params": 8951808,
1054
- "rel_err_whitened": 0.641594,
1055
- "als_skipped_for_memory": false,
1056
- "winner": "tt_padded",
1057
- "sep_rel": 0.984455
1058
- },
1059
- "model.layers.12.self_attn.v_proj": {
1060
- "ranks": [
1061
- 1,
1062
- 186,
1063
- 186
1064
- ],
1065
- "params": 8951808,
1066
- "rel_err_whitened": 0.638475,
1067
- "als_skipped_for_memory": false,
1068
- "winner": "tt_padded",
1069
- "sep_rel": 0.984455
1070
- },
1071
- "model.layers.12.self_attn.o_proj": {
1072
- "ranks": [
1073
- 1,
1074
- 186,
1075
- 186
1076
- ],
1077
- "params": 8951808,
1078
- "rel_err_whitened": 0.608011,
1079
- "als_skipped_for_memory": false,
1080
- "winner": "tt_padded",
1081
- "sep_rel": 0.709522
1082
- },
1083
- "model.layers.12.mlp.gate_proj": {
1084
- "ranks": [
1085
- 1,
1086
- 256,
1087
- 359
1088
- ],
1089
- "params": 23839952,
1090
- "rel_err_whitened": 0.657063,
1091
- "als_skipped_for_memory": true,
1092
- "winner": "tt_init",
1093
- "sep_rel": 0.741409
1094
- },
1095
- "model.layers.12.mlp.up_proj": {
1096
- "ranks": [
1097
- 1,
1098
- 256,
1099
- 359
1100
- ],
1101
- "params": 23839952,
1102
- "rel_err_whitened": 0.659221,
1103
- "als_skipped_for_memory": true,
1104
- "winner": "tt_init",
1105
- "sep_rel": 0.741409
1106
- },
1107
- "model.layers.12.mlp.down_proj": {
1108
- "ranks": [
1109
- 1,
1110
- 256,
1111
- 359
1112
- ],
1113
- "params": 23839952,
1114
- "rel_err_whitened": 0.658222,
1115
- "als_skipped_for_memory": true,
1116
- "winner": "tt_init",
1117
- "sep_rel": 0.590753
1118
- },
1119
- "model.layers.13.self_attn.q_proj": {
1120
- "ranks": [
1121
- 1,
1122
- 186,
1123
- 186
1124
- ],
1125
- "params": 8951808,
1126
- "rel_err_whitened": 0.640212,
1127
- "als_skipped_for_memory": false,
1128
- "winner": "tt_padded",
1129
- "sep_rel": 0.98039
1130
- },
1131
- "model.layers.13.self_attn.k_proj": {
1132
- "ranks": [
1133
- 1,
1134
- 186,
1135
- 186
1136
- ],
1137
- "params": 8951808,
1138
- "rel_err_whitened": 0.636702,
1139
- "als_skipped_for_memory": false,
1140
- "winner": "tt_padded",
1141
- "sep_rel": 0.98039
1142
- },
1143
- "model.layers.13.self_attn.v_proj": {
1144
- "ranks": [
1145
- 1,
1146
- 186,
1147
- 186
1148
- ],
1149
- "params": 8951808,
1150
- "rel_err_whitened": 0.643345,
1151
- "als_skipped_for_memory": false,
1152
- "winner": "tt_padded",
1153
- "sep_rel": 0.98039
1154
- },
1155
- "model.layers.13.self_attn.o_proj": {
1156
- "ranks": [
1157
- 1,
1158
- 186,
1159
- 186
1160
- ],
1161
- "params": 8951808,
1162
- "rel_err_whitened": 0.602689,
1163
- "als_skipped_for_memory": false,
1164
- "winner": "tt_padded",
1165
- "sep_rel": 0.629405
1166
- },
1167
- "model.layers.13.mlp.gate_proj": {
1168
- "ranks": [
1169
- 1,
1170
- 256,
1171
- 359
1172
- ],
1173
- "params": 23839952,
1174
- "rel_err_whitened": 0.656931,
1175
- "als_skipped_for_memory": true,
1176
- "winner": "tt_init",
1177
- "sep_rel": 0.737552
1178
- },
1179
- "model.layers.13.mlp.up_proj": {
1180
- "ranks": [
1181
- 1,
1182
- 256,
1183
- 359
1184
- ],
1185
- "params": 23839952,
1186
- "rel_err_whitened": 0.659251,
1187
- "als_skipped_for_memory": true,
1188
- "winner": "tt_init",
1189
- "sep_rel": 0.737552
1190
- },
1191
- "model.layers.13.mlp.down_proj": {
1192
- "ranks": [
1193
- 1,
1194
- 256,
1195
- 359
1196
- ],
1197
- "params": 23839952,
1198
- "rel_err_whitened": 0.65831,
1199
- "als_skipped_for_memory": true,
1200
- "winner": "tt_init",
1201
- "sep_rel": 0.653418
1202
- },
1203
- "model.layers.14.self_attn.q_proj": {
1204
- "ranks": [
1205
- 1,
1206
- 186,
1207
- 186
1208
- ],
1209
- "params": 8951808,
1210
- "rel_err_whitened": 0.638087,
1211
- "als_skipped_for_memory": false,
1212
- "winner": "tt_padded",
1213
- "sep_rel": 0.981171
1214
- },
1215
- "model.layers.14.self_attn.k_proj": {
1216
- "ranks": [
1217
- 1,
1218
- 186,
1219
- 186
1220
- ],
1221
- "params": 8951808,
1222
- "rel_err_whitened": 0.63626,
1223
- "als_skipped_for_memory": false,
1224
- "winner": "tt_padded",
1225
- "sep_rel": 0.981171
1226
- },
1227
- "model.layers.14.self_attn.v_proj": {
1228
- "ranks": [
1229
- 1,
1230
- 186,
1231
- 186
1232
- ],
1233
- "params": 8951808,
1234
- "rel_err_whitened": 0.645692,
1235
- "als_skipped_for_memory": false,
1236
- "winner": "tt_padded",
1237
- "sep_rel": 0.981171
1238
- },
1239
- "model.layers.14.self_attn.o_proj": {
1240
- "ranks": [
1241
- 1,
1242
- 186,
1243
- 186
1244
- ],
1245
- "params": 8951808,
1246
- "rel_err_whitened": 0.599762,
1247
- "als_skipped_for_memory": false,
1248
- "winner": "tt_padded",
1249
- "sep_rel": 0.586519
1250
- },
1251
- "model.layers.14.mlp.gate_proj": {
1252
- "ranks": [
1253
- 1,
1254
- 256,
1255
- 359
1256
- ],
1257
- "params": 23839952,
1258
- "rel_err_whitened": 0.657049,
1259
- "als_skipped_for_memory": true,
1260
- "winner": "tt_init",
1261
- "sep_rel": 0.681803
1262
- },
1263
- "model.layers.14.mlp.up_proj": {
1264
- "ranks": [
1265
- 1,
1266
- 256,
1267
- 359
1268
- ],
1269
- "params": 23839952,
1270
- "rel_err_whitened": 0.659205,
1271
- "als_skipped_for_memory": true,
1272
- "winner": "tt_init",
1273
- "sep_rel": 0.681803
1274
- },
1275
- "model.layers.14.mlp.down_proj": {
1276
- "ranks": [
1277
- 1,
1278
- 256,
1279
- 359
1280
- ],
1281
- "params": 23839952,
1282
- "rel_err_whitened": 0.658124,
1283
- "als_skipped_for_memory": true,
1284
- "winner": "tt_init",
1285
- "sep_rel": 0.633789
1286
- },
1287
- "model.layers.15.self_attn.q_proj": {
1288
- "ranks": [
1289
- 1,
1290
- 186,
1291
- 186
1292
- ],
1293
- "params": 8951808,
1294
- "rel_err_whitened": 0.637325,
1295
- "als_skipped_for_memory": false,
1296
- "winner": "tt_padded",
1297
- "sep_rel": 0.984063
1298
- },
1299
- "model.layers.15.self_attn.k_proj": {
1300
- "ranks": [
1301
- 1,
1302
- 186,
1303
- 186
1304
- ],
1305
- "params": 8951808,
1306
- "rel_err_whitened": 0.632505,
1307
- "als_skipped_for_memory": false,
1308
- "winner": "tt_padded",
1309
- "sep_rel": 0.984063
1310
- },
1311
- "model.layers.15.self_attn.v_proj": {
1312
- "ranks": [
1313
- 1,
1314
- 186,
1315
- 186
1316
- ],
1317
- "params": 8951808,
1318
- "rel_err_whitened": 0.649095,
1319
- "als_skipped_for_memory": false,
1320
- "winner": "tt_padded",
1321
- "sep_rel": 0.984063
1322
- },
1323
- "model.layers.15.self_attn.o_proj": {
1324
- "ranks": [
1325
- 1,
1326
- 186,
1327
- 186
1328
- ],
1329
- "params": 8951808,
1330
- "rel_err_whitened": 0.623948,
1331
- "als_skipped_for_memory": false,
1332
- "winner": "tt_padded",
1333
- "sep_rel": 0.685501
1334
- },
1335
- "model.layers.15.mlp.gate_proj": {
1336
- "ranks": [
1337
- 1,
1338
- 256,
1339
- 359
1340
- ],
1341
- "params": 23839952,
1342
- "rel_err_whitened": 0.656948,
1343
- "als_skipped_for_memory": true,
1344
- "winner": "tt_init",
1345
- "sep_rel": 0.693003
1346
- },
1347
- "model.layers.15.mlp.up_proj": {
1348
- "ranks": [
1349
- 1,
1350
- 256,
1351
- 359
1352
- ],
1353
- "params": 23839952,
1354
- "rel_err_whitened": 0.659275,
1355
- "als_skipped_for_memory": true,
1356
- "winner": "tt_init",
1357
- "sep_rel": 0.693003
1358
- },
1359
- "model.layers.15.mlp.down_proj": {
1360
- "ranks": [
1361
- 1,
1362
- 256,
1363
- 359
1364
- ],
1365
- "params": 23839952,
1366
- "rel_err_whitened": 0.658291,
1367
- "als_skipped_for_memory": true,
1368
- "winner": "tt_init",
1369
- "sep_rel": 0.701714
1370
- },
1371
- "model.layers.16.self_attn.q_proj": {
1372
- "ranks": [
1373
- 1,
1374
- 186,
1375
- 186
1376
- ],
1377
- "params": 8951808,
1378
- "rel_err_whitened": 0.641813,
1379
- "als_skipped_for_memory": false,
1380
- "winner": "tt_padded",
1381
- "sep_rel": 0.981692
1382
- },
1383
- "model.layers.16.self_attn.k_proj": {
1384
- "ranks": [
1385
- 1,
1386
- 186,
1387
- 186
1388
- ],
1389
- "params": 8951808,
1390
- "rel_err_whitened": 0.639806,
1391
- "als_skipped_for_memory": false,
1392
- "winner": "tt_padded",
1393
- "sep_rel": 0.981692
1394
- },
1395
- "model.layers.16.self_attn.v_proj": {
1396
- "ranks": [
1397
- 1,
1398
- 186,
1399
- 186
1400
- ],
1401
- "params": 8951808,
1402
- "rel_err_whitened": 0.646188,
1403
- "als_skipped_for_memory": false,
1404
- "winner": "tt_padded",
1405
- "sep_rel": 0.981692
1406
- },
1407
- "model.layers.16.self_attn.o_proj": {
1408
- "ranks": [
1409
- 1,
1410
- 186,
1411
- 186
1412
- ],
1413
- "params": 8951808,
1414
- "rel_err_whitened": 0.600841,
1415
- "als_skipped_for_memory": false,
1416
- "winner": "tt_padded",
1417
- "sep_rel": 0.603608
1418
- },
1419
- "model.layers.16.mlp.gate_proj": {
1420
- "ranks": [
1421
- 1,
1422
- 256,
1423
- 359
1424
- ],
1425
- "params": 23839952,
1426
- "rel_err_whitened": 0.657084,
1427
- "als_skipped_for_memory": true,
1428
- "winner": "tt_init",
1429
- "sep_rel": 0.572933
1430
- },
1431
- "model.layers.16.mlp.up_proj": {
1432
- "ranks": [
1433
- 1,
1434
- 256,
1435
- 359
1436
- ],
1437
- "params": 23839952,
1438
- "rel_err_whitened": 0.659218,
1439
- "als_skipped_for_memory": true,
1440
- "winner": "tt_init",
1441
- "sep_rel": 0.572933
1442
- },
1443
- "model.layers.16.mlp.down_proj": {
1444
- "ranks": [
1445
- 1,
1446
- 256,
1447
- 359
1448
- ],
1449
- "params": 23839952,
1450
- "rel_err_whitened": 0.657811,
1451
- "als_skipped_for_memory": true,
1452
- "winner": "tt_init",
1453
- "sep_rel": 0.790324
1454
- },
1455
- "model.layers.17.self_attn.q_proj": {
1456
- "ranks": [
1457
- 1,
1458
- 186,
1459
- 186
1460
- ],
1461
- "params": 8951808,
1462
- "rel_err_whitened": 0.643183,
1463
- "als_skipped_for_memory": false,
1464
- "winner": "tt_padded",
1465
- "sep_rel": 0.971694
1466
- },
1467
- "model.layers.17.self_attn.k_proj": {
1468
- "ranks": [
1469
- 1,
1470
- 186,
1471
- 186
1472
- ],
1473
- "params": 8951808,
1474
- "rel_err_whitened": 0.642234,
1475
- "als_skipped_for_memory": false,
1476
- "winner": "tt_padded",
1477
- "sep_rel": 0.971694
1478
- },
1479
- "model.layers.17.self_attn.v_proj": {
1480
- "ranks": [
1481
- 1,
1482
- 186,
1483
- 186
1484
- ],
1485
- "params": 8951808,
1486
- "rel_err_whitened": 0.648223,
1487
- "als_skipped_for_memory": false,
1488
- "winner": "tt_padded",
1489
- "sep_rel": 0.971694
1490
- },
1491
- "model.layers.17.self_attn.o_proj": {
1492
- "ranks": [
1493
- 1,
1494
- 186,
1495
- 186
1496
- ],
1497
- "params": 8951808,
1498
- "rel_err_whitened": 0.601042,
1499
- "als_skipped_for_memory": false,
1500
- "winner": "tt_padded",
1501
- "sep_rel": 0.632828
1502
- },
1503
- "model.layers.17.mlp.gate_proj": {
1504
- "ranks": [
1505
- 1,
1506
- 256,
1507
- 359
1508
- ],
1509
- "params": 23839952,
1510
- "rel_err_whitened": 0.656842,
1511
- "als_skipped_for_memory": true,
1512
- "winner": "tt_init",
1513
- "sep_rel": 0.4318
1514
- },
1515
- "model.layers.17.mlp.up_proj": {
1516
- "ranks": [
1517
- 1,
1518
- 256,
1519
- 359
1520
- ],
1521
- "params": 23839952,
1522
- "rel_err_whitened": 0.659223,
1523
- "als_skipped_for_memory": true,
1524
- "winner": "tt_init",
1525
- "sep_rel": 0.4318
1526
- },
1527
- "model.layers.17.mlp.down_proj": {
1528
- "ranks": [
1529
- 1,
1530
- 256,
1531
- 359
1532
- ],
1533
- "params": 23839952,
1534
- "rel_err_whitened": 0.658028,
1535
- "als_skipped_for_memory": true,
1536
- "winner": "tt_init",
1537
- "sep_rel": 0.688102
1538
- },
1539
- "model.layers.18.self_attn.q_proj": {
1540
- "ranks": [
1541
- 1,
1542
- 186,
1543
- 186
1544
- ],
1545
- "params": 8951808,
1546
- "rel_err_whitened": 0.647222,
1547
- "als_skipped_for_memory": false,
1548
- "winner": "tt_padded",
1549
- "sep_rel": 0.965421
1550
- },
1551
- "model.layers.18.self_attn.k_proj": {
1552
- "ranks": [
1553
- 1,
1554
- 186,
1555
- 186
1556
- ],
1557
- "params": 8951808,
1558
- "rel_err_whitened": 0.645022,
1559
- "als_skipped_for_memory": false,
1560
- "winner": "tt_padded",
1561
- "sep_rel": 0.965421
1562
- },
1563
- "model.layers.18.self_attn.v_proj": {
1564
- "ranks": [
1565
- 1,
1566
- 186,
1567
- 186
1568
- ],
1569
- "params": 8951808,
1570
- "rel_err_whitened": 0.642998,
1571
- "als_skipped_for_memory": false,
1572
- "winner": "tt_padded",
1573
- "sep_rel": 0.965421
1574
- },
1575
- "model.layers.18.self_attn.o_proj": {
1576
- "ranks": [
1577
- 1,
1578
- 186,
1579
- 186
1580
- ],
1581
- "params": 8951808,
1582
- "rel_err_whitened": 0.599502,
1583
- "als_skipped_for_memory": false,
1584
- "winner": "tt_padded",
1585
- "sep_rel": 0.783847
1586
- },
1587
- "model.layers.18.mlp.gate_proj": {
1588
- "ranks": [
1589
- 1,
1590
- 256,
1591
- 359
1592
- ],
1593
- "params": 23839952,
1594
- "rel_err_whitened": 0.657004,
1595
- "als_skipped_for_memory": true,
1596
- "winner": "tt_init",
1597
- "sep_rel": 0.384663
1598
- },
1599
- "model.layers.18.mlp.up_proj": {
1600
- "ranks": [
1601
- 1,
1602
- 256,
1603
- 359
1604
- ],
1605
- "params": 23839952,
1606
- "rel_err_whitened": 0.659198,
1607
- "als_skipped_for_memory": true,
1608
- "winner": "tt_init",
1609
- "sep_rel": 0.384663
1610
- },
1611
- "model.layers.18.mlp.down_proj": {
1612
- "ranks": [
1613
- 1,
1614
- 256,
1615
- 359
1616
- ],
1617
- "params": 23839952,
1618
- "rel_err_whitened": 0.658436,
1619
- "als_skipped_for_memory": true,
1620
- "winner": "tt_init",
1621
- "sep_rel": 0.821221
1622
- },
1623
- "model.layers.19.self_attn.q_proj": {
1624
- "ranks": [
1625
- 1,
1626
- 186,
1627
- 186
1628
- ],
1629
- "params": 8951808,
1630
- "rel_err_whitened": 0.643078,
1631
- "als_skipped_for_memory": false,
1632
- "winner": "tt_padded",
1633
- "sep_rel": 0.964501
1634
- },
1635
- "model.layers.19.self_attn.k_proj": {
1636
- "ranks": [
1637
- 1,
1638
- 186,
1639
- 186
1640
- ],
1641
- "params": 8951808,
1642
- "rel_err_whitened": 0.636811,
1643
- "als_skipped_for_memory": false,
1644
- "winner": "tt_padded",
1645
- "sep_rel": 0.964501
1646
- },
1647
- "model.layers.19.self_attn.v_proj": {
1648
- "ranks": [
1649
- 1,
1650
- 186,
1651
- 186
1652
- ],
1653
- "params": 8951808,
1654
- "rel_err_whitened": 0.64799,
1655
- "als_skipped_for_memory": false,
1656
- "winner": "tt_padded",
1657
- "sep_rel": 0.964501
1658
- },
1659
- "model.layers.19.self_attn.o_proj": {
1660
- "ranks": [
1661
- 1,
1662
- 186,
1663
- 186
1664
- ],
1665
- "params": 8951808,
1666
- "rel_err_whitened": 0.606372,
1667
- "als_skipped_for_memory": false,
1668
- "winner": "tt_padded",
1669
- "sep_rel": 0.681841
1670
- },
1671
- "model.layers.19.mlp.gate_proj": {
1672
- "ranks": [
1673
- 1,
1674
- 256,
1675
- 359
1676
- ],
1677
- "params": 23839952,
1678
- "rel_err_whitened": 0.657502,
1679
- "als_skipped_for_memory": true,
1680
- "winner": "tt_init",
1681
- "sep_rel": 0.353677
1682
- },
1683
- "model.layers.19.mlp.up_proj": {
1684
- "ranks": [
1685
- 1,
1686
- 256,
1687
- 359
1688
- ],
1689
- "params": 23839952,
1690
- "rel_err_whitened": 0.659383,
1691
- "als_skipped_for_memory": true,
1692
- "winner": "tt_init",
1693
- "sep_rel": 0.353677
1694
- },
1695
- "model.layers.19.mlp.down_proj": {
1696
- "ranks": [
1697
- 1,
1698
- 256,
1699
- 359
1700
- ],
1701
- "params": 23839952,
1702
- "rel_err_whitened": 0.658011,
1703
- "als_skipped_for_memory": true,
1704
- "winner": "tt_init",
1705
- "sep_rel": 0.76946
1706
- },
1707
- "model.layers.20.self_attn.q_proj": {
1708
- "ranks": [
1709
- 1,
1710
- 186,
1711
- 186
1712
- ],
1713
- "params": 8951808,
1714
- "rel_err_whitened": 0.646219,
1715
- "als_skipped_for_memory": false,
1716
- "winner": "tt_padded",
1717
- "sep_rel": 0.961589
1718
- },
1719
- "model.layers.20.self_attn.k_proj": {
1720
- "ranks": [
1721
- 1,
1722
- 186,
1723
- 186
1724
- ],
1725
- "params": 8951808,
1726
- "rel_err_whitened": 0.641485,
1727
- "als_skipped_for_memory": false,
1728
- "winner": "tt_padded",
1729
- "sep_rel": 0.961589
1730
- },
1731
- "model.layers.20.self_attn.v_proj": {
1732
- "ranks": [
1733
- 1,
1734
- 186,
1735
- 186
1736
- ],
1737
- "params": 8951808,
1738
- "rel_err_whitened": 0.654548,
1739
- "als_skipped_for_memory": false,
1740
- "winner": "tt_padded",
1741
- "sep_rel": 0.961589
1742
- },
1743
- "model.layers.20.self_attn.o_proj": {
1744
- "ranks": [
1745
- 1,
1746
- 186,
1747
- 186
1748
- ],
1749
- "params": 8951808,
1750
- "rel_err_whitened": 0.599251,
1751
- "als_skipped_for_memory": false,
1752
- "winner": "tt_padded",
1753
- "sep_rel": 0.616911
1754
- },
1755
- "model.layers.20.mlp.gate_proj": {
1756
- "ranks": [
1757
- 1,
1758
- 256,
1759
- 359
1760
- ],
1761
- "params": 23839952,
1762
- "rel_err_whitened": 0.657572,
1763
- "als_skipped_for_memory": true,
1764
- "winner": "tt_init",
1765
- "sep_rel": 0.301669
1766
- },
1767
- "model.layers.20.mlp.up_proj": {
1768
- "ranks": [
1769
- 1,
1770
- 256,
1771
- 359
1772
- ],
1773
- "params": 23839952,
1774
- "rel_err_whitened": 0.659337,
1775
- "als_skipped_for_memory": true,
1776
- "winner": "tt_init",
1777
- "sep_rel": 0.301669
1778
- },
1779
- "model.layers.20.mlp.down_proj": {
1780
- "ranks": [
1781
- 1,
1782
- 256,
1783
- 359
1784
- ],
1785
- "params": 23839952,
1786
- "rel_err_whitened": 0.657965,
1787
- "als_skipped_for_memory": true,
1788
- "winner": "tt_init",
1789
- "sep_rel": 0.872286
1790
- },
1791
- "model.layers.21.self_attn.q_proj": {
1792
- "ranks": [
1793
- 1,
1794
- 186,
1795
- 186
1796
- ],
1797
- "params": 8951808,
1798
- "rel_err_whitened": 0.632098,
1799
- "als_skipped_for_memory": false,
1800
- "winner": "tt_padded",
1801
- "sep_rel": 0.955204
1802
- },
1803
- "model.layers.21.self_attn.k_proj": {
1804
- "ranks": [
1805
- 1,
1806
- 186,
1807
- 186
1808
- ],
1809
- "params": 8951808,
1810
- "rel_err_whitened": 0.626394,
1811
- "als_skipped_for_memory": false,
1812
- "winner": "tt_padded",
1813
- "sep_rel": 0.955204
1814
- },
1815
- "model.layers.21.self_attn.v_proj": {
1816
- "ranks": [
1817
- 1,
1818
- 186,
1819
- 186
1820
- ],
1821
- "params": 8951808,
1822
- "rel_err_whitened": 0.650876,
1823
- "als_skipped_for_memory": false,
1824
- "winner": "tt_padded",
1825
- "sep_rel": 0.955204
1826
- },
1827
- "model.layers.21.self_attn.o_proj": {
1828
- "ranks": [
1829
- 1,
1830
- 186,
1831
- 186
1832
- ],
1833
- "params": 8951808,
1834
- "rel_err_whitened": 0.597006,
1835
- "als_skipped_for_memory": false,
1836
- "winner": "tt_padded",
1837
- "sep_rel": 0.550213
1838
- },
1839
- "model.layers.21.mlp.gate_proj": {
1840
- "ranks": [
1841
- 1,
1842
- 256,
1843
- 359
1844
- ],
1845
- "params": 23839952,
1846
- "rel_err_whitened": 0.657684,
1847
- "als_skipped_for_memory": true,
1848
- "winner": "tt_init",
1849
- "sep_rel": 0.273541
1850
- },
1851
- "model.layers.21.mlp.up_proj": {
1852
- "ranks": [
1853
- 1,
1854
- 256,
1855
- 359
1856
- ],
1857
- "params": 23839952,
1858
- "rel_err_whitened": 0.659344,
1859
- "als_skipped_for_memory": true,
1860
- "winner": "tt_init",
1861
- "sep_rel": 0.273541
1862
- },
1863
- "model.layers.21.mlp.down_proj": {
1864
- "ranks": [
1865
- 1,
1866
- 256,
1867
- 359
1868
- ],
1869
- "params": 23839952,
1870
- "rel_err_whitened": 0.658117,
1871
- "als_skipped_for_memory": true,
1872
- "winner": "tt_init",
1873
- "sep_rel": 0.729657
1874
- },
1875
- "model.layers.22.self_attn.q_proj": {
1876
- "ranks": [
1877
- 1,
1878
- 186,
1879
- 186
1880
- ],
1881
- "params": 8951808,
1882
- "rel_err_whitened": 0.6394,
1883
- "als_skipped_for_memory": false,
1884
- "winner": "tt_padded",
1885
- "sep_rel": 0.957392
1886
- },
1887
- "model.layers.22.self_attn.k_proj": {
1888
- "ranks": [
1889
- 1,
1890
- 186,
1891
- 186
1892
- ],
1893
- "params": 8951808,
1894
- "rel_err_whitened": 0.636307,
1895
- "als_skipped_for_memory": false,
1896
- "winner": "tt_padded",
1897
- "sep_rel": 0.957392
1898
- },
1899
- "model.layers.22.self_attn.v_proj": {
1900
- "ranks": [
1901
- 1,
1902
- 186,
1903
- 186
1904
- ],
1905
- "params": 8951808,
1906
- "rel_err_whitened": 0.647597,
1907
- "als_skipped_for_memory": false,
1908
- "winner": "tt_padded",
1909
- "sep_rel": 0.957392
1910
- },
1911
- "model.layers.22.self_attn.o_proj": {
1912
- "ranks": [
1913
- 1,
1914
- 186,
1915
- 186
1916
- ],
1917
- "params": 8951808,
1918
- "rel_err_whitened": 0.591862,
1919
- "als_skipped_for_memory": false,
1920
- "winner": "tt_padded",
1921
- "sep_rel": 0.555932
1922
- },
1923
- "model.layers.22.mlp.gate_proj": {
1924
- "ranks": [
1925
- 1,
1926
- 256,
1927
- 359
1928
- ],
1929
- "params": 23839952,
1930
- "rel_err_whitened": 0.657602,
1931
- "als_skipped_for_memory": true,
1932
- "winner": "tt_init",
1933
- "sep_rel": 0.244172
1934
- },
1935
- "model.layers.22.mlp.up_proj": {
1936
- "ranks": [
1937
- 1,
1938
- 256,
1939
- 359
1940
- ],
1941
- "params": 23839952,
1942
- "rel_err_whitened": 0.659306,
1943
- "als_skipped_for_memory": true,
1944
- "winner": "tt_init",
1945
- "sep_rel": 0.244172
1946
- },
1947
- "model.layers.22.mlp.down_proj": {
1948
- "ranks": [
1949
- 1,
1950
- 256,
1951
- 359
1952
- ],
1953
- "params": 23839952,
1954
- "rel_err_whitened": 0.658173,
1955
- "als_skipped_for_memory": true,
1956
- "winner": "tt_init",
1957
- "sep_rel": 0.69247
1958
- },
1959
- "model.layers.23.self_attn.q_proj": {
1960
- "ranks": [
1961
- 1,
1962
- 186,
1963
- 186
1964
- ],
1965
- "params": 8951808,
1966
- "rel_err_whitened": 0.637635,
1967
- "als_skipped_for_memory": false,
1968
- "winner": "tt_padded",
1969
- "sep_rel": 0.938431
1970
- },
1971
- "model.layers.23.self_attn.k_proj": {
1972
- "ranks": [
1973
- 1,
1974
- 186,
1975
- 186
1976
- ],
1977
- "params": 8951808,
1978
- "rel_err_whitened": 0.63656,
1979
- "als_skipped_for_memory": false,
1980
- "winner": "tt_padded",
1981
- "sep_rel": 0.938431
1982
- },
1983
- "model.layers.23.self_attn.v_proj": {
1984
- "ranks": [
1985
- 1,
1986
- 186,
1987
- 186
1988
- ],
1989
- "params": 8951808,
1990
- "rel_err_whitened": 0.64307,
1991
- "als_skipped_for_memory": false,
1992
- "winner": "tt_padded",
1993
- "sep_rel": 0.938431
1994
- },
1995
- "model.layers.23.self_attn.o_proj": {
1996
- "ranks": [
1997
- 1,
1998
- 186,
1999
- 186
2000
- ],
2001
- "params": 8951808,
2002
- "rel_err_whitened": 0.587359,
2003
- "als_skipped_for_memory": false,
2004
- "winner": "tt_padded",
2005
- "sep_rel": 0.716492
2006
- },
2007
- "model.layers.23.mlp.gate_proj": {
2008
- "ranks": [
2009
- 1,
2010
- 256,
2011
- 359
2012
- ],
2013
- "params": 23839952,
2014
- "rel_err_whitened": 0.658095,
2015
- "als_skipped_for_memory": true,
2016
- "winner": "tt_init",
2017
- "sep_rel": 0.27869
2018
- },
2019
- "model.layers.23.mlp.up_proj": {
2020
- "ranks": [
2021
- 1,
2022
- 256,
2023
- 359
2024
- ],
2025
- "params": 23839952,
2026
- "rel_err_whitened": 0.659401,
2027
- "als_skipped_for_memory": true,
2028
- "winner": "tt_init",
2029
- "sep_rel": 0.27869
2030
- },
2031
- "model.layers.23.mlp.down_proj": {
2032
- "ranks": [
2033
- 1,
2034
- 256,
2035
- 359
2036
- ],
2037
- "params": 23839952,
2038
- "rel_err_whitened": 0.657709,
2039
- "als_skipped_for_memory": true,
2040
- "winner": "tt_init",
2041
- "sep_rel": 0.752968
2042
- },
2043
- "model.layers.24.self_attn.q_proj": {
2044
- "ranks": [
2045
- 1,
2046
- 186,
2047
- 186
2048
- ],
2049
- "params": 8951808,
2050
- "rel_err_whitened": 0.629698,
2051
- "als_skipped_for_memory": false,
2052
- "winner": "tt_padded",
2053
- "sep_rel": 0.957543
2054
- },
2055
- "model.layers.24.self_attn.k_proj": {
2056
- "ranks": [
2057
- 1,
2058
- 186,
2059
- 186
2060
- ],
2061
- "params": 8951808,
2062
- "rel_err_whitened": 0.626766,
2063
- "als_skipped_for_memory": false,
2064
- "winner": "tt_padded",
2065
- "sep_rel": 0.957543
2066
- },
2067
- "model.layers.24.self_attn.v_proj": {
2068
- "ranks": [
2069
- 1,
2070
- 186,
2071
- 186
2072
- ],
2073
- "params": 8951808,
2074
- "rel_err_whitened": 0.650262,
2075
- "als_skipped_for_memory": false,
2076
- "winner": "tt_padded",
2077
- "sep_rel": 0.957543
2078
- },
2079
- "model.layers.24.self_attn.o_proj": {
2080
- "ranks": [
2081
- 1,
2082
- 186,
2083
- 186
2084
- ],
2085
- "params": 8951808,
2086
- "rel_err_whitened": 0.613239,
2087
- "als_skipped_for_memory": false,
2088
- "winner": "tt_padded",
2089
- "sep_rel": 0.837446
2090
- },
2091
- "model.layers.24.mlp.gate_proj": {
2092
- "ranks": [
2093
- 1,
2094
- 256,
2095
- 359
2096
- ],
2097
- "params": 23839952,
2098
- "rel_err_whitened": 0.658124,
2099
- "als_skipped_for_memory": true,
2100
- "winner": "tt_init",
2101
- "sep_rel": 0.242684
2102
- },
2103
- "model.layers.24.mlp.up_proj": {
2104
- "ranks": [
2105
- 1,
2106
- 256,
2107
- 359
2108
- ],
2109
- "params": 23839952,
2110
- "rel_err_whitened": 0.659379,
2111
- "als_skipped_for_memory": true,
2112
- "winner": "tt_init",
2113
- "sep_rel": 0.242684
2114
- },
2115
- "model.layers.24.mlp.down_proj": {
2116
- "ranks": [
2117
- 1,
2118
- 256,
2119
- 359
2120
- ],
2121
- "params": 23839952,
2122
- "rel_err_whitened": 0.65827,
2123
- "als_skipped_for_memory": true,
2124
- "winner": "tt_init",
2125
- "sep_rel": 0.791953
2126
- },
2127
- "model.layers.25.self_attn.q_proj": {
2128
- "ranks": [
2129
- 1,
2130
- 186,
2131
- 186
2132
- ],
2133
- "params": 8951808,
2134
- "rel_err_whitened": 0.636979,
2135
- "als_skipped_for_memory": false,
2136
- "winner": "tt_padded",
2137
- "sep_rel": 0.919729
2138
- },
2139
- "model.layers.25.self_attn.k_proj": {
2140
- "ranks": [
2141
- 1,
2142
- 186,
2143
- 186
2144
- ],
2145
- "params": 8951808,
2146
- "rel_err_whitened": 0.636359,
2147
- "als_skipped_for_memory": false,
2148
- "winner": "tt_padded",
2149
- "sep_rel": 0.919729
2150
- },
2151
- "model.layers.25.self_attn.v_proj": {
2152
- "ranks": [
2153
- 1,
2154
- 186,
2155
- 186
2156
- ],
2157
- "params": 8951808,
2158
- "rel_err_whitened": 0.651961,
2159
- "als_skipped_for_memory": false,
2160
- "winner": "tt_padded",
2161
- "sep_rel": 0.919729
2162
- },
2163
- "model.layers.25.self_attn.o_proj": {
2164
- "ranks": [
2165
- 1,
2166
- 186,
2167
- 186
2168
- ],
2169
- "params": 8951808,
2170
- "rel_err_whitened": 0.610397,
2171
- "als_skipped_for_memory": false,
2172
- "winner": "tt_padded",
2173
- "sep_rel": 0.55963
2174
- },
2175
- "model.layers.25.mlp.gate_proj": {
2176
- "ranks": [
2177
- 1,
2178
- 256,
2179
- 359
2180
- ],
2181
- "params": 23839952,
2182
- "rel_err_whitened": 0.658203,
2183
- "als_skipped_for_memory": true,
2184
- "winner": "tt_init",
2185
- "sep_rel": 0.316393
2186
- },
2187
- "model.layers.25.mlp.up_proj": {
2188
- "ranks": [
2189
- 1,
2190
- 256,
2191
- 359
2192
- ],
2193
- "params": 23839952,
2194
- "rel_err_whitened": 0.659402,
2195
- "als_skipped_for_memory": true,
2196
- "winner": "tt_init",
2197
- "sep_rel": 0.316393
2198
- },
2199
- "model.layers.25.mlp.down_proj": {
2200
- "ranks": [
2201
- 1,
2202
- 256,
2203
- 359
2204
- ],
2205
- "params": 23839952,
2206
- "rel_err_whitened": 0.658316,
2207
- "als_skipped_for_memory": true,
2208
- "winner": "tt_init",
2209
- "sep_rel": 0.867097
2210
- },
2211
- "model.layers.26.self_attn.q_proj": {
2212
- "ranks": [
2213
- 1,
2214
- 186,
2215
- 186
2216
- ],
2217
- "params": 8951808,
2218
- "rel_err_whitened": 0.632829,
2219
- "als_skipped_for_memory": false,
2220
- "winner": "tt_padded",
2221
- "sep_rel": 0.935645
2222
- },
2223
- "model.layers.26.self_attn.k_proj": {
2224
- "ranks": [
2225
- 1,
2226
- 186,
2227
- 186
2228
- ],
2229
- "params": 8951808,
2230
- "rel_err_whitened": 0.629569,
2231
- "als_skipped_for_memory": false,
2232
- "winner": "tt_padded",
2233
- "sep_rel": 0.935645
2234
- },
2235
- "model.layers.26.self_attn.v_proj": {
2236
- "ranks": [
2237
- 1,
2238
- 186,
2239
- 186
2240
- ],
2241
- "params": 8951808,
2242
- "rel_err_whitened": 0.643075,
2243
- "als_skipped_for_memory": false,
2244
- "winner": "tt_padded",
2245
- "sep_rel": 0.935645
2246
- },
2247
- "model.layers.26.self_attn.o_proj": {
2248
- "ranks": [
2249
- 1,
2250
- 186,
2251
- 186
2252
- ],
2253
- "params": 8951808,
2254
- "rel_err_whitened": 0.573524,
2255
- "als_skipped_for_memory": false,
2256
- "winner": "tt_padded",
2257
- "sep_rel": 0.88256
2258
- },
2259
- "model.layers.26.mlp.gate_proj": {
2260
- "ranks": [
2261
- 1,
2262
- 256,
2263
- 359
2264
- ],
2265
- "params": 23839952,
2266
- "rel_err_whitened": 0.658034,
2267
- "als_skipped_for_memory": true,
2268
- "winner": "tt_init",
2269
- "sep_rel": 0.317972
2270
- },
2271
- "model.layers.26.mlp.up_proj": {
2272
- "ranks": [
2273
- 1,
2274
- 256,
2275
- 359
2276
- ],
2277
- "params": 23839952,
2278
- "rel_err_whitened": 0.659357,
2279
- "als_skipped_for_memory": true,
2280
- "winner": "tt_init",
2281
- "sep_rel": 0.317972
2282
- },
2283
- "model.layers.26.mlp.down_proj": {
2284
- "ranks": [
2285
- 1,
2286
- 256,
2287
- 359
2288
- ],
2289
- "params": 23839952,
2290
- "rel_err_whitened": 0.658005,
2291
- "als_skipped_for_memory": true,
2292
- "winner": "tt_init",
2293
- "sep_rel": 0.794827
2294
- },
2295
- "model.layers.27.self_attn.q_proj": {
2296
- "ranks": [
2297
- 1,
2298
- 186,
2299
- 186
2300
- ],
2301
- "params": 8951808,
2302
- "rel_err_whitened": 0.644939,
2303
- "als_skipped_for_memory": false,
2304
- "winner": "tt_padded",
2305
- "sep_rel": 0.84314
2306
- },
2307
- "model.layers.27.self_attn.k_proj": {
2308
- "ranks": [
2309
- 1,
2310
- 186,
2311
- 186
2312
- ],
2313
- "params": 8951808,
2314
- "rel_err_whitened": 0.646309,
2315
- "als_skipped_for_memory": false,
2316
- "winner": "tt_padded",
2317
- "sep_rel": 0.84314
2318
- },
2319
- "model.layers.27.self_attn.v_proj": {
2320
- "ranks": [
2321
- 1,
2322
- 186,
2323
- 186
2324
- ],
2325
- "params": 8951808,
2326
- "rel_err_whitened": 0.645044,
2327
- "als_skipped_for_memory": false,
2328
- "winner": "tt_padded",
2329
- "sep_rel": 0.84314
2330
- },
2331
- "model.layers.27.self_attn.o_proj": {
2332
- "ranks": [
2333
- 1,
2334
- 186,
2335
- 186
2336
- ],
2337
- "params": 8951808,
2338
- "rel_err_whitened": 0.610565,
2339
- "als_skipped_for_memory": false,
2340
- "winner": "tt_padded",
2341
- "sep_rel": 0.865767
2342
- },
2343
- "model.layers.27.mlp.gate_proj": {
2344
- "ranks": [
2345
- 1,
2346
- 256,
2347
- 359
2348
- ],
2349
- "params": 23839952,
2350
- "rel_err_whitened": 0.658116,
2351
- "als_skipped_for_memory": true,
2352
- "winner": "tt_init",
2353
- "sep_rel": 0.368547
2354
- },
2355
- "model.layers.27.mlp.up_proj": {
2356
- "ranks": [
2357
- 1,
2358
- 256,
2359
- 359
2360
- ],
2361
- "params": 23839952,
2362
- "rel_err_whitened": 0.659418,
2363
- "als_skipped_for_memory": true,
2364
- "winner": "tt_init",
2365
- "sep_rel": 0.368547
2366
- },
2367
- "model.layers.27.mlp.down_proj": {
2368
- "ranks": [
2369
- 1,
2370
- 256,
2371
- 359
2372
- ],
2373
- "params": 23839952,
2374
- "rel_err_whitened": 0.657569,
2375
- "als_skipped_for_memory": true,
2376
- "winner": "tt_init",
2377
- "sep_rel": 0.832094
2378
- },
2379
- "model.layers.28.self_attn.q_proj": {
2380
- "ranks": [
2381
- 1,
2382
- 186,
2383
- 186
2384
- ],
2385
- "params": 8951808,
2386
- "rel_err_whitened": 0.645585,
2387
- "als_skipped_for_memory": false,
2388
- "winner": "tt_padded",
2389
- "sep_rel": 0.829394
2390
- },
2391
- "model.layers.28.self_attn.k_proj": {
2392
- "ranks": [
2393
- 1,
2394
- 186,
2395
- 186
2396
- ],
2397
- "params": 8951808,
2398
- "rel_err_whitened": 0.646825,
2399
- "als_skipped_for_memory": false,
2400
- "winner": "tt_padded",
2401
- "sep_rel": 0.829394
2402
- },
2403
- "model.layers.28.self_attn.v_proj": {
2404
- "ranks": [
2405
- 1,
2406
- 186,
2407
- 186
2408
- ],
2409
- "params": 8951808,
2410
- "rel_err_whitened": 0.646436,
2411
- "als_skipped_for_memory": false,
2412
- "winner": "tt_padded",
2413
- "sep_rel": 0.829394
2414
- },
2415
- "model.layers.28.self_attn.o_proj": {
2416
- "ranks": [
2417
- 1,
2418
- 186,
2419
- 186
2420
- ],
2421
- "params": 8951808,
2422
- "rel_err_whitened": 0.602643,
2423
- "als_skipped_for_memory": false,
2424
- "winner": "tt_padded",
2425
- "sep_rel": 0.792865
2426
- },
2427
- "model.layers.28.mlp.gate_proj": {
2428
- "ranks": [
2429
- 1,
2430
- 256,
2431
- 359
2432
- ],
2433
- "params": 23839952,
2434
- "rel_err_whitened": 0.657734,
2435
- "als_skipped_for_memory": true,
2436
- "winner": "tt_init",
2437
- "sep_rel": 0.463555
2438
- },
2439
- "model.layers.28.mlp.up_proj": {
2440
- "ranks": [
2441
- 1,
2442
- 256,
2443
- 359
2444
- ],
2445
- "params": 23839952,
2446
- "rel_err_whitened": 0.659343,
2447
- "als_skipped_for_memory": true,
2448
- "winner": "tt_init",
2449
- "sep_rel": 0.463555
2450
- },
2451
- "model.layers.28.mlp.down_proj": {
2452
- "ranks": [
2453
- 1,
2454
- 256,
2455
- 359
2456
- ],
2457
- "params": 23839952,
2458
- "rel_err_whitened": 0.657344,
2459
- "als_skipped_for_memory": true,
2460
- "winner": "tt_init",
2461
- "sep_rel": 0.904961
2462
- },
2463
- "model.layers.29.self_attn.q_proj": {
2464
- "ranks": [
2465
- 1,
2466
- 186,
2467
- 186
2468
- ],
2469
- "params": 8951808,
2470
- "rel_err_whitened": 0.64088,
2471
- "als_skipped_for_memory": false,
2472
- "winner": "tt_padded",
2473
- "sep_rel": 0.860686
2474
- },
2475
- "model.layers.29.self_attn.k_proj": {
2476
- "ranks": [
2477
- 1,
2478
- 186,
2479
- 186
2480
- ],
2481
- "params": 8951808,
2482
- "rel_err_whitened": 0.640494,
2483
- "als_skipped_for_memory": false,
2484
- "winner": "tt_padded",
2485
- "sep_rel": 0.860686
2486
- },
2487
- "model.layers.29.self_attn.v_proj": {
2488
- "ranks": [
2489
- 1,
2490
- 186,
2491
- 186
2492
- ],
2493
- "params": 8951808,
2494
- "rel_err_whitened": 0.648816,
2495
- "als_skipped_for_memory": false,
2496
- "winner": "tt_padded",
2497
- "sep_rel": 0.860686
2498
- },
2499
- "model.layers.29.self_attn.o_proj": {
2500
- "ranks": [
2501
- 1,
2502
- 186,
2503
- 186
2504
- ],
2505
- "params": 8951808,
2506
- "rel_err_whitened": 0.593463,
2507
- "als_skipped_for_memory": false,
2508
- "winner": "tt_padded",
2509
- "sep_rel": 0.742165
2510
- },
2511
- "model.layers.29.mlp.gate_proj": {
2512
- "ranks": [
2513
- 1,
2514
- 256,
2515
- 359
2516
- ],
2517
- "params": 23839952,
2518
- "rel_err_whitened": 0.657283,
2519
- "als_skipped_for_memory": true,
2520
- "winner": "tt_init",
2521
- "sep_rel": 0.539988
2522
- },
2523
- "model.layers.29.mlp.up_proj": {
2524
- "ranks": [
2525
- 1,
2526
- 256,
2527
- 359
2528
- ],
2529
- "params": 23839952,
2530
- "rel_err_whitened": 0.658855,
2531
- "als_skipped_for_memory": true,
2532
- "winner": "tt_init",
2533
- "sep_rel": 0.539988
2534
- },
2535
- "model.layers.29.mlp.down_proj": {
2536
- "ranks": [
2537
- 1,
2538
- 256,
2539
- 359
2540
- ],
2541
- "params": 23839952,
2542
- "rel_err_whitened": 0.655678,
2543
- "als_skipped_for_memory": true,
2544
- "winner": "tt_init",
2545
- "sep_rel": 0.887416
2546
- },
2547
- "model.layers.30.self_attn.q_proj": {
2548
- "ranks": [
2549
- 1,
2550
- 186,
2551
- 186
2552
- ],
2553
- "params": 8951808,
2554
- "rel_err_whitened": 0.640828,
2555
- "als_skipped_for_memory": false,
2556
- "winner": "tt_padded",
2557
- "sep_rel": 0.743608
2558
- },
2559
- "model.layers.30.self_attn.k_proj": {
2560
- "ranks": [
2561
- 1,
2562
- 186,
2563
- 186
2564
- ],
2565
- "params": 8951808,
2566
- "rel_err_whitened": 0.643594,
2567
- "als_skipped_for_memory": false,
2568
- "winner": "tt_padded",
2569
- "sep_rel": 0.743608
2570
- },
2571
- "model.layers.30.self_attn.v_proj": {
2572
- "ranks": [
2573
- 1,
2574
- 186,
2575
- 186
2576
- ],
2577
- "params": 8951808,
2578
- "rel_err_whitened": 0.651818,
2579
- "als_skipped_for_memory": false,
2580
- "winner": "tt_padded",
2581
- "sep_rel": 0.743608
2582
- },
2583
- "model.layers.30.self_attn.o_proj": {
2584
- "ranks": [
2585
- 1,
2586
- 186,
2587
- 186
2588
- ],
2589
- "params": 8951808,
2590
- "rel_err_whitened": 0.571319,
2591
- "als_skipped_for_memory": false,
2592
- "winner": "tt_padded",
2593
- "sep_rel": 0.786148
2594
- },
2595
- "model.layers.30.mlp.gate_proj": {
2596
- "ranks": [
2597
- 1,
2598
- 256,
2599
- 359
2600
- ],
2601
- "params": 23839952,
2602
- "rel_err_whitened": 0.652715,
2603
- "als_skipped_for_memory": true,
2604
- "winner": "tt_init",
2605
- "sep_rel": 0.549327
2606
- },
2607
- "model.layers.30.mlp.up_proj": {
2608
- "ranks": [
2609
- 1,
2610
- 256,
2611
- 359
2612
- ],
2613
- "params": 23839952,
2614
- "rel_err_whitened": 0.65383,
2615
- "als_skipped_for_memory": true,
2616
- "winner": "tt_init",
2617
- "sep_rel": 0.549327
2618
- },
2619
- "model.layers.30.mlp.down_proj": {
2620
- "ranks": [
2621
- 1,
2622
- 256,
2623
- 359
2624
- ],
2625
- "params": 23839952,
2626
- "rel_err_whitened": 0.655015,
2627
- "als_skipped_for_memory": true,
2628
- "winner": "tt_init",
2629
- "sep_rel": 0.999914
2630
- },
2631
- "model.layers.31.self_attn.q_proj": {
2632
- "ranks": [
2633
- 1,
2634
- 186,
2635
- 186
2636
- ],
2637
- "params": 8951808,
2638
- "rel_err_whitened": 0.644717,
2639
- "als_skipped_for_memory": false,
2640
- "winner": "tt_padded",
2641
- "sep_rel": 0.763178
2642
- },
2643
- "model.layers.31.self_attn.k_proj": {
2644
- "ranks": [
2645
- 1,
2646
- 186,
2647
- 186
2648
- ],
2649
- "params": 8951808,
2650
- "rel_err_whitened": 0.642352,
2651
- "als_skipped_for_memory": false,
2652
- "winner": "tt_padded",
2653
- "sep_rel": 0.763178
2654
- },
2655
- "model.layers.31.self_attn.v_proj": {
2656
- "ranks": [
2657
- 1,
2658
- 186,
2659
- 186
2660
- ],
2661
- "params": 8951808,
2662
- "rel_err_whitened": 0.649714,
2663
- "als_skipped_for_memory": false,
2664
- "winner": "tt_padded",
2665
- "sep_rel": 0.763178
2666
- },
2667
- "model.layers.31.self_attn.o_proj": {
2668
- "ranks": [
2669
- 1,
2670
- 186,
2671
- 186
2672
- ],
2673
- "params": 8951808,
2674
- "rel_err_whitened": 0.597899,
2675
- "als_skipped_for_memory": false,
2676
- "winner": "tt_padded",
2677
- "sep_rel": 0.856492
2678
- },
2679
- "model.layers.31.mlp.gate_proj": {
2680
- "ranks": [
2681
- 1,
2682
- 256,
2683
- 359
2684
- ],
2685
- "params": 23839952,
2686
- "rel_err_whitened": 0.657658,
2687
- "als_skipped_for_memory": true,
2688
- "winner": "tt_init",
2689
- "sep_rel": 0.766331
2690
- },
2691
- "model.layers.31.mlp.up_proj": {
2692
- "ranks": [
2693
- 1,
2694
- 256,
2695
- 359
2696
- ],
2697
- "params": 23839952,
2698
- "rel_err_whitened": 0.658156,
2699
- "als_skipped_for_memory": true,
2700
- "winner": "tt_init",
2701
- "sep_rel": 0.766331
2702
- },
2703
- "model.layers.31.mlp.down_proj": {
2704
- "ranks": [
2705
- 1,
2706
- 256,
2707
- 359
2708
- ],
2709
- "params": 23839952,
2710
- "rel_err_whitened": 0.651667,
2711
- "als_skipped_for_memory": true,
2712
- "winner": "tt_init",
2713
- "sep_rel": 0.994828
2714
- }
2715
- }
2716
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
tt3-llama2-7b/ranks.json DELETED
@@ -1,2690 +0,0 @@
1
- {
2
- "model.layers.0.self_attn.q_proj": {
3
- "o1": 16,
4
- "o2": 16,
5
- "o3": 16,
6
- "i1": 16,
7
- "i2": 16,
8
- "i3": 16,
9
- "r0": 1,
10
- "r1": 186,
11
- "r2": 186,
12
- "nnz": 838860
13
- },
14
- "model.layers.0.self_attn.k_proj": {
15
- "o1": 16,
16
- "o2": 16,
17
- "o3": 16,
18
- "i1": 16,
19
- "i2": 16,
20
- "i3": 16,
21
- "r0": 1,
22
- "r1": 186,
23
- "r2": 186,
24
- "nnz": 838860
25
- },
26
- "model.layers.0.self_attn.v_proj": {
27
- "o1": 16,
28
- "o2": 16,
29
- "o3": 16,
30
- "i1": 16,
31
- "i2": 16,
32
- "i3": 16,
33
- "r0": 1,
34
- "r1": 186,
35
- "r2": 186,
36
- "nnz": 838860
37
- },
38
- "model.layers.0.self_attn.o_proj": {
39
- "o1": 16,
40
- "o2": 16,
41
- "o3": 16,
42
- "i1": 16,
43
- "i2": 16,
44
- "i3": 16,
45
- "r0": 1,
46
- "r1": 186,
47
- "r2": 186,
48
- "nnz": 838860
49
- },
50
- "model.layers.0.mlp.gate_proj": {
51
- "o1": 16,
52
- "o2": 16,
53
- "o3": 43,
54
- "i1": 16,
55
- "i2": 16,
56
- "i3": 16,
57
- "r0": 1,
58
- "r1": 256,
59
- "r2": 359,
60
- "nnz": 2254438
61
- },
62
- "model.layers.0.mlp.up_proj": {
63
- "o1": 16,
64
- "o2": 16,
65
- "o3": 43,
66
- "i1": 16,
67
- "i2": 16,
68
- "i3": 16,
69
- "r0": 1,
70
- "r1": 256,
71
- "r2": 359,
72
- "nnz": 2254438
73
- },
74
- "model.layers.0.mlp.down_proj": {
75
- "o1": 16,
76
- "o2": 16,
77
- "o3": 16,
78
- "i1": 16,
79
- "i2": 16,
80
- "i3": 43,
81
- "r0": 1,
82
- "r1": 256,
83
- "r2": 359,
84
- "nnz": 2254438
85
- },
86
- "model.layers.1.self_attn.q_proj": {
87
- "o1": 16,
88
- "o2": 16,
89
- "o3": 16,
90
- "i1": 16,
91
- "i2": 16,
92
- "i3": 16,
93
- "r0": 1,
94
- "r1": 186,
95
- "r2": 186,
96
- "nnz": 838860
97
- },
98
- "model.layers.1.self_attn.k_proj": {
99
- "o1": 16,
100
- "o2": 16,
101
- "o3": 16,
102
- "i1": 16,
103
- "i2": 16,
104
- "i3": 16,
105
- "r0": 1,
106
- "r1": 186,
107
- "r2": 186,
108
- "nnz": 838860
109
- },
110
- "model.layers.1.self_attn.v_proj": {
111
- "o1": 16,
112
- "o2": 16,
113
- "o3": 16,
114
- "i1": 16,
115
- "i2": 16,
116
- "i3": 16,
117
- "r0": 1,
118
- "r1": 186,
119
- "r2": 186,
120
- "nnz": 838860
121
- },
122
- "model.layers.1.self_attn.o_proj": {
123
- "o1": 16,
124
- "o2": 16,
125
- "o3": 16,
126
- "i1": 16,
127
- "i2": 16,
128
- "i3": 16,
129
- "r0": 1,
130
- "r1": 186,
131
- "r2": 186,
132
- "nnz": 838860
133
- },
134
- "model.layers.1.mlp.gate_proj": {
135
- "o1": 16,
136
- "o2": 16,
137
- "o3": 43,
138
- "i1": 16,
139
- "i2": 16,
140
- "i3": 16,
141
- "r0": 1,
142
- "r1": 256,
143
- "r2": 359,
144
- "nnz": 2254438
145
- },
146
- "model.layers.1.mlp.up_proj": {
147
- "o1": 16,
148
- "o2": 16,
149
- "o3": 43,
150
- "i1": 16,
151
- "i2": 16,
152
- "i3": 16,
153
- "r0": 1,
154
- "r1": 256,
155
- "r2": 359,
156
- "nnz": 2254438
157
- },
158
- "model.layers.1.mlp.down_proj": {
159
- "o1": 16,
160
- "o2": 16,
161
- "o3": 16,
162
- "i1": 16,
163
- "i2": 16,
164
- "i3": 43,
165
- "r0": 1,
166
- "r1": 256,
167
- "r2": 359,
168
- "nnz": 2254438
169
- },
170
- "model.layers.2.self_attn.q_proj": {
171
- "o1": 16,
172
- "o2": 16,
173
- "o3": 16,
174
- "i1": 16,
175
- "i2": 16,
176
- "i3": 16,
177
- "r0": 1,
178
- "r1": 186,
179
- "r2": 186,
180
- "nnz": 838860
181
- },
182
- "model.layers.2.self_attn.k_proj": {
183
- "o1": 16,
184
- "o2": 16,
185
- "o3": 16,
186
- "i1": 16,
187
- "i2": 16,
188
- "i3": 16,
189
- "r0": 1,
190
- "r1": 186,
191
- "r2": 186,
192
- "nnz": 838860
193
- },
194
- "model.layers.2.self_attn.v_proj": {
195
- "o1": 16,
196
- "o2": 16,
197
- "o3": 16,
198
- "i1": 16,
199
- "i2": 16,
200
- "i3": 16,
201
- "r0": 1,
202
- "r1": 186,
203
- "r2": 186,
204
- "nnz": 838860
205
- },
206
- "model.layers.2.self_attn.o_proj": {
207
- "o1": 16,
208
- "o2": 16,
209
- "o3": 16,
210
- "i1": 16,
211
- "i2": 16,
212
- "i3": 16,
213
- "r0": 1,
214
- "r1": 186,
215
- "r2": 186,
216
- "nnz": 838860
217
- },
218
- "model.layers.2.mlp.gate_proj": {
219
- "o1": 16,
220
- "o2": 16,
221
- "o3": 43,
222
- "i1": 16,
223
- "i2": 16,
224
- "i3": 16,
225
- "r0": 1,
226
- "r1": 256,
227
- "r2": 359,
228
- "nnz": 2254438
229
- },
230
- "model.layers.2.mlp.up_proj": {
231
- "o1": 16,
232
- "o2": 16,
233
- "o3": 43,
234
- "i1": 16,
235
- "i2": 16,
236
- "i3": 16,
237
- "r0": 1,
238
- "r1": 256,
239
- "r2": 359,
240
- "nnz": 2254438
241
- },
242
- "model.layers.2.mlp.down_proj": {
243
- "o1": 16,
244
- "o2": 16,
245
- "o3": 16,
246
- "i1": 16,
247
- "i2": 16,
248
- "i3": 43,
249
- "r0": 1,
250
- "r1": 256,
251
- "r2": 359,
252
- "nnz": 2254438
253
- },
254
- "model.layers.3.self_attn.q_proj": {
255
- "o1": 16,
256
- "o2": 16,
257
- "o3": 16,
258
- "i1": 16,
259
- "i2": 16,
260
- "i3": 16,
261
- "r0": 1,
262
- "r1": 186,
263
- "r2": 186,
264
- "nnz": 838860
265
- },
266
- "model.layers.3.self_attn.k_proj": {
267
- "o1": 16,
268
- "o2": 16,
269
- "o3": 16,
270
- "i1": 16,
271
- "i2": 16,
272
- "i3": 16,
273
- "r0": 1,
274
- "r1": 186,
275
- "r2": 186,
276
- "nnz": 838860
277
- },
278
- "model.layers.3.self_attn.v_proj": {
279
- "o1": 16,
280
- "o2": 16,
281
- "o3": 16,
282
- "i1": 16,
283
- "i2": 16,
284
- "i3": 16,
285
- "r0": 1,
286
- "r1": 186,
287
- "r2": 186,
288
- "nnz": 838860
289
- },
290
- "model.layers.3.self_attn.o_proj": {
291
- "o1": 16,
292
- "o2": 16,
293
- "o3": 16,
294
- "i1": 16,
295
- "i2": 16,
296
- "i3": 16,
297
- "r0": 1,
298
- "r1": 186,
299
- "r2": 186,
300
- "nnz": 838860
301
- },
302
- "model.layers.3.mlp.gate_proj": {
303
- "o1": 16,
304
- "o2": 16,
305
- "o3": 43,
306
- "i1": 16,
307
- "i2": 16,
308
- "i3": 16,
309
- "r0": 1,
310
- "r1": 256,
311
- "r2": 359,
312
- "nnz": 2254438
313
- },
314
- "model.layers.3.mlp.up_proj": {
315
- "o1": 16,
316
- "o2": 16,
317
- "o3": 43,
318
- "i1": 16,
319
- "i2": 16,
320
- "i3": 16,
321
- "r0": 1,
322
- "r1": 256,
323
- "r2": 359,
324
- "nnz": 2254438
325
- },
326
- "model.layers.3.mlp.down_proj": {
327
- "o1": 16,
328
- "o2": 16,
329
- "o3": 16,
330
- "i1": 16,
331
- "i2": 16,
332
- "i3": 43,
333
- "r0": 1,
334
- "r1": 256,
335
- "r2": 359,
336
- "nnz": 2254438
337
- },
338
- "model.layers.4.self_attn.q_proj": {
339
- "o1": 16,
340
- "o2": 16,
341
- "o3": 16,
342
- "i1": 16,
343
- "i2": 16,
344
- "i3": 16,
345
- "r0": 1,
346
- "r1": 186,
347
- "r2": 186,
348
- "nnz": 838860
349
- },
350
- "model.layers.4.self_attn.k_proj": {
351
- "o1": 16,
352
- "o2": 16,
353
- "o3": 16,
354
- "i1": 16,
355
- "i2": 16,
356
- "i3": 16,
357
- "r0": 1,
358
- "r1": 186,
359
- "r2": 186,
360
- "nnz": 838860
361
- },
362
- "model.layers.4.self_attn.v_proj": {
363
- "o1": 16,
364
- "o2": 16,
365
- "o3": 16,
366
- "i1": 16,
367
- "i2": 16,
368
- "i3": 16,
369
- "r0": 1,
370
- "r1": 186,
371
- "r2": 186,
372
- "nnz": 838860
373
- },
374
- "model.layers.4.self_attn.o_proj": {
375
- "o1": 16,
376
- "o2": 16,
377
- "o3": 16,
378
- "i1": 16,
379
- "i2": 16,
380
- "i3": 16,
381
- "r0": 1,
382
- "r1": 186,
383
- "r2": 186,
384
- "nnz": 838860
385
- },
386
- "model.layers.4.mlp.gate_proj": {
387
- "o1": 16,
388
- "o2": 16,
389
- "o3": 43,
390
- "i1": 16,
391
- "i2": 16,
392
- "i3": 16,
393
- "r0": 1,
394
- "r1": 256,
395
- "r2": 359,
396
- "nnz": 2254438
397
- },
398
- "model.layers.4.mlp.up_proj": {
399
- "o1": 16,
400
- "o2": 16,
401
- "o3": 43,
402
- "i1": 16,
403
- "i2": 16,
404
- "i3": 16,
405
- "r0": 1,
406
- "r1": 256,
407
- "r2": 359,
408
- "nnz": 2254438
409
- },
410
- "model.layers.4.mlp.down_proj": {
411
- "o1": 16,
412
- "o2": 16,
413
- "o3": 16,
414
- "i1": 16,
415
- "i2": 16,
416
- "i3": 43,
417
- "r0": 1,
418
- "r1": 256,
419
- "r2": 359,
420
- "nnz": 2254438
421
- },
422
- "model.layers.5.self_attn.q_proj": {
423
- "o1": 16,
424
- "o2": 16,
425
- "o3": 16,
426
- "i1": 16,
427
- "i2": 16,
428
- "i3": 16,
429
- "r0": 1,
430
- "r1": 186,
431
- "r2": 186,
432
- "nnz": 838860
433
- },
434
- "model.layers.5.self_attn.k_proj": {
435
- "o1": 16,
436
- "o2": 16,
437
- "o3": 16,
438
- "i1": 16,
439
- "i2": 16,
440
- "i3": 16,
441
- "r0": 1,
442
- "r1": 186,
443
- "r2": 186,
444
- "nnz": 838860
445
- },
446
- "model.layers.5.self_attn.v_proj": {
447
- "o1": 16,
448
- "o2": 16,
449
- "o3": 16,
450
- "i1": 16,
451
- "i2": 16,
452
- "i3": 16,
453
- "r0": 1,
454
- "r1": 186,
455
- "r2": 186,
456
- "nnz": 838860
457
- },
458
- "model.layers.5.self_attn.o_proj": {
459
- "o1": 16,
460
- "o2": 16,
461
- "o3": 16,
462
- "i1": 16,
463
- "i2": 16,
464
- "i3": 16,
465
- "r0": 1,
466
- "r1": 186,
467
- "r2": 186,
468
- "nnz": 838860
469
- },
470
- "model.layers.5.mlp.gate_proj": {
471
- "o1": 16,
472
- "o2": 16,
473
- "o3": 43,
474
- "i1": 16,
475
- "i2": 16,
476
- "i3": 16,
477
- "r0": 1,
478
- "r1": 256,
479
- "r2": 359,
480
- "nnz": 2254438
481
- },
482
- "model.layers.5.mlp.up_proj": {
483
- "o1": 16,
484
- "o2": 16,
485
- "o3": 43,
486
- "i1": 16,
487
- "i2": 16,
488
- "i3": 16,
489
- "r0": 1,
490
- "r1": 256,
491
- "r2": 359,
492
- "nnz": 2254438
493
- },
494
- "model.layers.5.mlp.down_proj": {
495
- "o1": 16,
496
- "o2": 16,
497
- "o3": 16,
498
- "i1": 16,
499
- "i2": 16,
500
- "i3": 43,
501
- "r0": 1,
502
- "r1": 256,
503
- "r2": 359,
504
- "nnz": 2254438
505
- },
506
- "model.layers.6.self_attn.q_proj": {
507
- "o1": 16,
508
- "o2": 16,
509
- "o3": 16,
510
- "i1": 16,
511
- "i2": 16,
512
- "i3": 16,
513
- "r0": 1,
514
- "r1": 186,
515
- "r2": 186,
516
- "nnz": 838860
517
- },
518
- "model.layers.6.self_attn.k_proj": {
519
- "o1": 16,
520
- "o2": 16,
521
- "o3": 16,
522
- "i1": 16,
523
- "i2": 16,
524
- "i3": 16,
525
- "r0": 1,
526
- "r1": 186,
527
- "r2": 186,
528
- "nnz": 838860
529
- },
530
- "model.layers.6.self_attn.v_proj": {
531
- "o1": 16,
532
- "o2": 16,
533
- "o3": 16,
534
- "i1": 16,
535
- "i2": 16,
536
- "i3": 16,
537
- "r0": 1,
538
- "r1": 186,
539
- "r2": 186,
540
- "nnz": 838860
541
- },
542
- "model.layers.6.self_attn.o_proj": {
543
- "o1": 16,
544
- "o2": 16,
545
- "o3": 16,
546
- "i1": 16,
547
- "i2": 16,
548
- "i3": 16,
549
- "r0": 1,
550
- "r1": 186,
551
- "r2": 186,
552
- "nnz": 838860
553
- },
554
- "model.layers.6.mlp.gate_proj": {
555
- "o1": 16,
556
- "o2": 16,
557
- "o3": 43,
558
- "i1": 16,
559
- "i2": 16,
560
- "i3": 16,
561
- "r0": 1,
562
- "r1": 256,
563
- "r2": 359,
564
- "nnz": 2254438
565
- },
566
- "model.layers.6.mlp.up_proj": {
567
- "o1": 16,
568
- "o2": 16,
569
- "o3": 43,
570
- "i1": 16,
571
- "i2": 16,
572
- "i3": 16,
573
- "r0": 1,
574
- "r1": 256,
575
- "r2": 359,
576
- "nnz": 2254438
577
- },
578
- "model.layers.6.mlp.down_proj": {
579
- "o1": 16,
580
- "o2": 16,
581
- "o3": 16,
582
- "i1": 16,
583
- "i2": 16,
584
- "i3": 43,
585
- "r0": 1,
586
- "r1": 256,
587
- "r2": 359,
588
- "nnz": 2254438
589
- },
590
- "model.layers.7.self_attn.q_proj": {
591
- "o1": 16,
592
- "o2": 16,
593
- "o3": 16,
594
- "i1": 16,
595
- "i2": 16,
596
- "i3": 16,
597
- "r0": 1,
598
- "r1": 186,
599
- "r2": 186,
600
- "nnz": 838860
601
- },
602
- "model.layers.7.self_attn.k_proj": {
603
- "o1": 16,
604
- "o2": 16,
605
- "o3": 16,
606
- "i1": 16,
607
- "i2": 16,
608
- "i3": 16,
609
- "r0": 1,
610
- "r1": 186,
611
- "r2": 186,
612
- "nnz": 838860
613
- },
614
- "model.layers.7.self_attn.v_proj": {
615
- "o1": 16,
616
- "o2": 16,
617
- "o3": 16,
618
- "i1": 16,
619
- "i2": 16,
620
- "i3": 16,
621
- "r0": 1,
622
- "r1": 186,
623
- "r2": 186,
624
- "nnz": 838860
625
- },
626
- "model.layers.7.self_attn.o_proj": {
627
- "o1": 16,
628
- "o2": 16,
629
- "o3": 16,
630
- "i1": 16,
631
- "i2": 16,
632
- "i3": 16,
633
- "r0": 1,
634
- "r1": 186,
635
- "r2": 186,
636
- "nnz": 838860
637
- },
638
- "model.layers.7.mlp.gate_proj": {
639
- "o1": 16,
640
- "o2": 16,
641
- "o3": 43,
642
- "i1": 16,
643
- "i2": 16,
644
- "i3": 16,
645
- "r0": 1,
646
- "r1": 256,
647
- "r2": 359,
648
- "nnz": 2254438
649
- },
650
- "model.layers.7.mlp.up_proj": {
651
- "o1": 16,
652
- "o2": 16,
653
- "o3": 43,
654
- "i1": 16,
655
- "i2": 16,
656
- "i3": 16,
657
- "r0": 1,
658
- "r1": 256,
659
- "r2": 359,
660
- "nnz": 2254438
661
- },
662
- "model.layers.7.mlp.down_proj": {
663
- "o1": 16,
664
- "o2": 16,
665
- "o3": 16,
666
- "i1": 16,
667
- "i2": 16,
668
- "i3": 43,
669
- "r0": 1,
670
- "r1": 256,
671
- "r2": 359,
672
- "nnz": 2254438
673
- },
674
- "model.layers.8.self_attn.q_proj": {
675
- "o1": 16,
676
- "o2": 16,
677
- "o3": 16,
678
- "i1": 16,
679
- "i2": 16,
680
- "i3": 16,
681
- "r0": 1,
682
- "r1": 186,
683
- "r2": 186,
684
- "nnz": 838860
685
- },
686
- "model.layers.8.self_attn.k_proj": {
687
- "o1": 16,
688
- "o2": 16,
689
- "o3": 16,
690
- "i1": 16,
691
- "i2": 16,
692
- "i3": 16,
693
- "r0": 1,
694
- "r1": 186,
695
- "r2": 186,
696
- "nnz": 838860
697
- },
698
- "model.layers.8.self_attn.v_proj": {
699
- "o1": 16,
700
- "o2": 16,
701
- "o3": 16,
702
- "i1": 16,
703
- "i2": 16,
704
- "i3": 16,
705
- "r0": 1,
706
- "r1": 186,
707
- "r2": 186,
708
- "nnz": 838860
709
- },
710
- "model.layers.8.self_attn.o_proj": {
711
- "o1": 16,
712
- "o2": 16,
713
- "o3": 16,
714
- "i1": 16,
715
- "i2": 16,
716
- "i3": 16,
717
- "r0": 1,
718
- "r1": 186,
719
- "r2": 186,
720
- "nnz": 838860
721
- },
722
- "model.layers.8.mlp.gate_proj": {
723
- "o1": 16,
724
- "o2": 16,
725
- "o3": 43,
726
- "i1": 16,
727
- "i2": 16,
728
- "i3": 16,
729
- "r0": 1,
730
- "r1": 256,
731
- "r2": 359,
732
- "nnz": 2254438
733
- },
734
- "model.layers.8.mlp.up_proj": {
735
- "o1": 16,
736
- "o2": 16,
737
- "o3": 43,
738
- "i1": 16,
739
- "i2": 16,
740
- "i3": 16,
741
- "r0": 1,
742
- "r1": 256,
743
- "r2": 359,
744
- "nnz": 2254438
745
- },
746
- "model.layers.8.mlp.down_proj": {
747
- "o1": 16,
748
- "o2": 16,
749
- "o3": 16,
750
- "i1": 16,
751
- "i2": 16,
752
- "i3": 43,
753
- "r0": 1,
754
- "r1": 256,
755
- "r2": 359,
756
- "nnz": 2254438
757
- },
758
- "model.layers.9.self_attn.q_proj": {
759
- "o1": 16,
760
- "o2": 16,
761
- "o3": 16,
762
- "i1": 16,
763
- "i2": 16,
764
- "i3": 16,
765
- "r0": 1,
766
- "r1": 186,
767
- "r2": 186,
768
- "nnz": 838860
769
- },
770
- "model.layers.9.self_attn.k_proj": {
771
- "o1": 16,
772
- "o2": 16,
773
- "o3": 16,
774
- "i1": 16,
775
- "i2": 16,
776
- "i3": 16,
777
- "r0": 1,
778
- "r1": 186,
779
- "r2": 186,
780
- "nnz": 838860
781
- },
782
- "model.layers.9.self_attn.v_proj": {
783
- "o1": 16,
784
- "o2": 16,
785
- "o3": 16,
786
- "i1": 16,
787
- "i2": 16,
788
- "i3": 16,
789
- "r0": 1,
790
- "r1": 186,
791
- "r2": 186,
792
- "nnz": 838860
793
- },
794
- "model.layers.9.self_attn.o_proj": {
795
- "o1": 16,
796
- "o2": 16,
797
- "o3": 16,
798
- "i1": 16,
799
- "i2": 16,
800
- "i3": 16,
801
- "r0": 1,
802
- "r1": 186,
803
- "r2": 186,
804
- "nnz": 838860
805
- },
806
- "model.layers.9.mlp.gate_proj": {
807
- "o1": 16,
808
- "o2": 16,
809
- "o3": 43,
810
- "i1": 16,
811
- "i2": 16,
812
- "i3": 16,
813
- "r0": 1,
814
- "r1": 256,
815
- "r2": 359,
816
- "nnz": 2254438
817
- },
818
- "model.layers.9.mlp.up_proj": {
819
- "o1": 16,
820
- "o2": 16,
821
- "o3": 43,
822
- "i1": 16,
823
- "i2": 16,
824
- "i3": 16,
825
- "r0": 1,
826
- "r1": 256,
827
- "r2": 359,
828
- "nnz": 2254438
829
- },
830
- "model.layers.9.mlp.down_proj": {
831
- "o1": 16,
832
- "o2": 16,
833
- "o3": 16,
834
- "i1": 16,
835
- "i2": 16,
836
- "i3": 43,
837
- "r0": 1,
838
- "r1": 256,
839
- "r2": 359,
840
- "nnz": 2254438
841
- },
842
- "model.layers.10.self_attn.q_proj": {
843
- "o1": 16,
844
- "o2": 16,
845
- "o3": 16,
846
- "i1": 16,
847
- "i2": 16,
848
- "i3": 16,
849
- "r0": 1,
850
- "r1": 186,
851
- "r2": 186,
852
- "nnz": 838860
853
- },
854
- "model.layers.10.self_attn.k_proj": {
855
- "o1": 16,
856
- "o2": 16,
857
- "o3": 16,
858
- "i1": 16,
859
- "i2": 16,
860
- "i3": 16,
861
- "r0": 1,
862
- "r1": 186,
863
- "r2": 186,
864
- "nnz": 838860
865
- },
866
- "model.layers.10.self_attn.v_proj": {
867
- "o1": 16,
868
- "o2": 16,
869
- "o3": 16,
870
- "i1": 16,
871
- "i2": 16,
872
- "i3": 16,
873
- "r0": 1,
874
- "r1": 186,
875
- "r2": 186,
876
- "nnz": 838860
877
- },
878
- "model.layers.10.self_attn.o_proj": {
879
- "o1": 16,
880
- "o2": 16,
881
- "o3": 16,
882
- "i1": 16,
883
- "i2": 16,
884
- "i3": 16,
885
- "r0": 1,
886
- "r1": 186,
887
- "r2": 186,
888
- "nnz": 838860
889
- },
890
- "model.layers.10.mlp.gate_proj": {
891
- "o1": 16,
892
- "o2": 16,
893
- "o3": 43,
894
- "i1": 16,
895
- "i2": 16,
896
- "i3": 16,
897
- "r0": 1,
898
- "r1": 256,
899
- "r2": 359,
900
- "nnz": 2254438
901
- },
902
- "model.layers.10.mlp.up_proj": {
903
- "o1": 16,
904
- "o2": 16,
905
- "o3": 43,
906
- "i1": 16,
907
- "i2": 16,
908
- "i3": 16,
909
- "r0": 1,
910
- "r1": 256,
911
- "r2": 359,
912
- "nnz": 2254438
913
- },
914
- "model.layers.10.mlp.down_proj": {
915
- "o1": 16,
916
- "o2": 16,
917
- "o3": 16,
918
- "i1": 16,
919
- "i2": 16,
920
- "i3": 43,
921
- "r0": 1,
922
- "r1": 256,
923
- "r2": 359,
924
- "nnz": 2254438
925
- },
926
- "model.layers.11.self_attn.q_proj": {
927
- "o1": 16,
928
- "o2": 16,
929
- "o3": 16,
930
- "i1": 16,
931
- "i2": 16,
932
- "i3": 16,
933
- "r0": 1,
934
- "r1": 186,
935
- "r2": 186,
936
- "nnz": 838860
937
- },
938
- "model.layers.11.self_attn.k_proj": {
939
- "o1": 16,
940
- "o2": 16,
941
- "o3": 16,
942
- "i1": 16,
943
- "i2": 16,
944
- "i3": 16,
945
- "r0": 1,
946
- "r1": 186,
947
- "r2": 186,
948
- "nnz": 838860
949
- },
950
- "model.layers.11.self_attn.v_proj": {
951
- "o1": 16,
952
- "o2": 16,
953
- "o3": 16,
954
- "i1": 16,
955
- "i2": 16,
956
- "i3": 16,
957
- "r0": 1,
958
- "r1": 186,
959
- "r2": 186,
960
- "nnz": 838860
961
- },
962
- "model.layers.11.self_attn.o_proj": {
963
- "o1": 16,
964
- "o2": 16,
965
- "o3": 16,
966
- "i1": 16,
967
- "i2": 16,
968
- "i3": 16,
969
- "r0": 1,
970
- "r1": 186,
971
- "r2": 186,
972
- "nnz": 838860
973
- },
974
- "model.layers.11.mlp.gate_proj": {
975
- "o1": 16,
976
- "o2": 16,
977
- "o3": 43,
978
- "i1": 16,
979
- "i2": 16,
980
- "i3": 16,
981
- "r0": 1,
982
- "r1": 256,
983
- "r2": 359,
984
- "nnz": 2254438
985
- },
986
- "model.layers.11.mlp.up_proj": {
987
- "o1": 16,
988
- "o2": 16,
989
- "o3": 43,
990
- "i1": 16,
991
- "i2": 16,
992
- "i3": 16,
993
- "r0": 1,
994
- "r1": 256,
995
- "r2": 359,
996
- "nnz": 2254438
997
- },
998
- "model.layers.11.mlp.down_proj": {
999
- "o1": 16,
1000
- "o2": 16,
1001
- "o3": 16,
1002
- "i1": 16,
1003
- "i2": 16,
1004
- "i3": 43,
1005
- "r0": 1,
1006
- "r1": 256,
1007
- "r2": 359,
1008
- "nnz": 2254438
1009
- },
1010
- "model.layers.12.self_attn.q_proj": {
1011
- "o1": 16,
1012
- "o2": 16,
1013
- "o3": 16,
1014
- "i1": 16,
1015
- "i2": 16,
1016
- "i3": 16,
1017
- "r0": 1,
1018
- "r1": 186,
1019
- "r2": 186,
1020
- "nnz": 838860
1021
- },
1022
- "model.layers.12.self_attn.k_proj": {
1023
- "o1": 16,
1024
- "o2": 16,
1025
- "o3": 16,
1026
- "i1": 16,
1027
- "i2": 16,
1028
- "i3": 16,
1029
- "r0": 1,
1030
- "r1": 186,
1031
- "r2": 186,
1032
- "nnz": 838860
1033
- },
1034
- "model.layers.12.self_attn.v_proj": {
1035
- "o1": 16,
1036
- "o2": 16,
1037
- "o3": 16,
1038
- "i1": 16,
1039
- "i2": 16,
1040
- "i3": 16,
1041
- "r0": 1,
1042
- "r1": 186,
1043
- "r2": 186,
1044
- "nnz": 838860
1045
- },
1046
- "model.layers.12.self_attn.o_proj": {
1047
- "o1": 16,
1048
- "o2": 16,
1049
- "o3": 16,
1050
- "i1": 16,
1051
- "i2": 16,
1052
- "i3": 16,
1053
- "r0": 1,
1054
- "r1": 186,
1055
- "r2": 186,
1056
- "nnz": 838860
1057
- },
1058
- "model.layers.12.mlp.gate_proj": {
1059
- "o1": 16,
1060
- "o2": 16,
1061
- "o3": 43,
1062
- "i1": 16,
1063
- "i2": 16,
1064
- "i3": 16,
1065
- "r0": 1,
1066
- "r1": 256,
1067
- "r2": 359,
1068
- "nnz": 2254438
1069
- },
1070
- "model.layers.12.mlp.up_proj": {
1071
- "o1": 16,
1072
- "o2": 16,
1073
- "o3": 43,
1074
- "i1": 16,
1075
- "i2": 16,
1076
- "i3": 16,
1077
- "r0": 1,
1078
- "r1": 256,
1079
- "r2": 359,
1080
- "nnz": 2254438
1081
- },
1082
- "model.layers.12.mlp.down_proj": {
1083
- "o1": 16,
1084
- "o2": 16,
1085
- "o3": 16,
1086
- "i1": 16,
1087
- "i2": 16,
1088
- "i3": 43,
1089
- "r0": 1,
1090
- "r1": 256,
1091
- "r2": 359,
1092
- "nnz": 2254438
1093
- },
1094
- "model.layers.13.self_attn.q_proj": {
1095
- "o1": 16,
1096
- "o2": 16,
1097
- "o3": 16,
1098
- "i1": 16,
1099
- "i2": 16,
1100
- "i3": 16,
1101
- "r0": 1,
1102
- "r1": 186,
1103
- "r2": 186,
1104
- "nnz": 838860
1105
- },
1106
- "model.layers.13.self_attn.k_proj": {
1107
- "o1": 16,
1108
- "o2": 16,
1109
- "o3": 16,
1110
- "i1": 16,
1111
- "i2": 16,
1112
- "i3": 16,
1113
- "r0": 1,
1114
- "r1": 186,
1115
- "r2": 186,
1116
- "nnz": 838860
1117
- },
1118
- "model.layers.13.self_attn.v_proj": {
1119
- "o1": 16,
1120
- "o2": 16,
1121
- "o3": 16,
1122
- "i1": 16,
1123
- "i2": 16,
1124
- "i3": 16,
1125
- "r0": 1,
1126
- "r1": 186,
1127
- "r2": 186,
1128
- "nnz": 838860
1129
- },
1130
- "model.layers.13.self_attn.o_proj": {
1131
- "o1": 16,
1132
- "o2": 16,
1133
- "o3": 16,
1134
- "i1": 16,
1135
- "i2": 16,
1136
- "i3": 16,
1137
- "r0": 1,
1138
- "r1": 186,
1139
- "r2": 186,
1140
- "nnz": 838860
1141
- },
1142
- "model.layers.13.mlp.gate_proj": {
1143
- "o1": 16,
1144
- "o2": 16,
1145
- "o3": 43,
1146
- "i1": 16,
1147
- "i2": 16,
1148
- "i3": 16,
1149
- "r0": 1,
1150
- "r1": 256,
1151
- "r2": 359,
1152
- "nnz": 2254438
1153
- },
1154
- "model.layers.13.mlp.up_proj": {
1155
- "o1": 16,
1156
- "o2": 16,
1157
- "o3": 43,
1158
- "i1": 16,
1159
- "i2": 16,
1160
- "i3": 16,
1161
- "r0": 1,
1162
- "r1": 256,
1163
- "r2": 359,
1164
- "nnz": 2254438
1165
- },
1166
- "model.layers.13.mlp.down_proj": {
1167
- "o1": 16,
1168
- "o2": 16,
1169
- "o3": 16,
1170
- "i1": 16,
1171
- "i2": 16,
1172
- "i3": 43,
1173
- "r0": 1,
1174
- "r1": 256,
1175
- "r2": 359,
1176
- "nnz": 2254438
1177
- },
1178
- "model.layers.14.self_attn.q_proj": {
1179
- "o1": 16,
1180
- "o2": 16,
1181
- "o3": 16,
1182
- "i1": 16,
1183
- "i2": 16,
1184
- "i3": 16,
1185
- "r0": 1,
1186
- "r1": 186,
1187
- "r2": 186,
1188
- "nnz": 838860
1189
- },
1190
- "model.layers.14.self_attn.k_proj": {
1191
- "o1": 16,
1192
- "o2": 16,
1193
- "o3": 16,
1194
- "i1": 16,
1195
- "i2": 16,
1196
- "i3": 16,
1197
- "r0": 1,
1198
- "r1": 186,
1199
- "r2": 186,
1200
- "nnz": 838860
1201
- },
1202
- "model.layers.14.self_attn.v_proj": {
1203
- "o1": 16,
1204
- "o2": 16,
1205
- "o3": 16,
1206
- "i1": 16,
1207
- "i2": 16,
1208
- "i3": 16,
1209
- "r0": 1,
1210
- "r1": 186,
1211
- "r2": 186,
1212
- "nnz": 838860
1213
- },
1214
- "model.layers.14.self_attn.o_proj": {
1215
- "o1": 16,
1216
- "o2": 16,
1217
- "o3": 16,
1218
- "i1": 16,
1219
- "i2": 16,
1220
- "i3": 16,
1221
- "r0": 1,
1222
- "r1": 186,
1223
- "r2": 186,
1224
- "nnz": 838860
1225
- },
1226
- "model.layers.14.mlp.gate_proj": {
1227
- "o1": 16,
1228
- "o2": 16,
1229
- "o3": 43,
1230
- "i1": 16,
1231
- "i2": 16,
1232
- "i3": 16,
1233
- "r0": 1,
1234
- "r1": 256,
1235
- "r2": 359,
1236
- "nnz": 2254438
1237
- },
1238
- "model.layers.14.mlp.up_proj": {
1239
- "o1": 16,
1240
- "o2": 16,
1241
- "o3": 43,
1242
- "i1": 16,
1243
- "i2": 16,
1244
- "i3": 16,
1245
- "r0": 1,
1246
- "r1": 256,
1247
- "r2": 359,
1248
- "nnz": 2254438
1249
- },
1250
- "model.layers.14.mlp.down_proj": {
1251
- "o1": 16,
1252
- "o2": 16,
1253
- "o3": 16,
1254
- "i1": 16,
1255
- "i2": 16,
1256
- "i3": 43,
1257
- "r0": 1,
1258
- "r1": 256,
1259
- "r2": 359,
1260
- "nnz": 2254438
1261
- },
1262
- "model.layers.15.self_attn.q_proj": {
1263
- "o1": 16,
1264
- "o2": 16,
1265
- "o3": 16,
1266
- "i1": 16,
1267
- "i2": 16,
1268
- "i3": 16,
1269
- "r0": 1,
1270
- "r1": 186,
1271
- "r2": 186,
1272
- "nnz": 838860
1273
- },
1274
- "model.layers.15.self_attn.k_proj": {
1275
- "o1": 16,
1276
- "o2": 16,
1277
- "o3": 16,
1278
- "i1": 16,
1279
- "i2": 16,
1280
- "i3": 16,
1281
- "r0": 1,
1282
- "r1": 186,
1283
- "r2": 186,
1284
- "nnz": 838860
1285
- },
1286
- "model.layers.15.self_attn.v_proj": {
1287
- "o1": 16,
1288
- "o2": 16,
1289
- "o3": 16,
1290
- "i1": 16,
1291
- "i2": 16,
1292
- "i3": 16,
1293
- "r0": 1,
1294
- "r1": 186,
1295
- "r2": 186,
1296
- "nnz": 838860
1297
- },
1298
- "model.layers.15.self_attn.o_proj": {
1299
- "o1": 16,
1300
- "o2": 16,
1301
- "o3": 16,
1302
- "i1": 16,
1303
- "i2": 16,
1304
- "i3": 16,
1305
- "r0": 1,
1306
- "r1": 186,
1307
- "r2": 186,
1308
- "nnz": 838860
1309
- },
1310
- "model.layers.15.mlp.gate_proj": {
1311
- "o1": 16,
1312
- "o2": 16,
1313
- "o3": 43,
1314
- "i1": 16,
1315
- "i2": 16,
1316
- "i3": 16,
1317
- "r0": 1,
1318
- "r1": 256,
1319
- "r2": 359,
1320
- "nnz": 2254438
1321
- },
1322
- "model.layers.15.mlp.up_proj": {
1323
- "o1": 16,
1324
- "o2": 16,
1325
- "o3": 43,
1326
- "i1": 16,
1327
- "i2": 16,
1328
- "i3": 16,
1329
- "r0": 1,
1330
- "r1": 256,
1331
- "r2": 359,
1332
- "nnz": 2254438
1333
- },
1334
- "model.layers.15.mlp.down_proj": {
1335
- "o1": 16,
1336
- "o2": 16,
1337
- "o3": 16,
1338
- "i1": 16,
1339
- "i2": 16,
1340
- "i3": 43,
1341
- "r0": 1,
1342
- "r1": 256,
1343
- "r2": 359,
1344
- "nnz": 2254438
1345
- },
1346
- "model.layers.16.self_attn.q_proj": {
1347
- "o1": 16,
1348
- "o2": 16,
1349
- "o3": 16,
1350
- "i1": 16,
1351
- "i2": 16,
1352
- "i3": 16,
1353
- "r0": 1,
1354
- "r1": 186,
1355
- "r2": 186,
1356
- "nnz": 838860
1357
- },
1358
- "model.layers.16.self_attn.k_proj": {
1359
- "o1": 16,
1360
- "o2": 16,
1361
- "o3": 16,
1362
- "i1": 16,
1363
- "i2": 16,
1364
- "i3": 16,
1365
- "r0": 1,
1366
- "r1": 186,
1367
- "r2": 186,
1368
- "nnz": 838860
1369
- },
1370
- "model.layers.16.self_attn.v_proj": {
1371
- "o1": 16,
1372
- "o2": 16,
1373
- "o3": 16,
1374
- "i1": 16,
1375
- "i2": 16,
1376
- "i3": 16,
1377
- "r0": 1,
1378
- "r1": 186,
1379
- "r2": 186,
1380
- "nnz": 838860
1381
- },
1382
- "model.layers.16.self_attn.o_proj": {
1383
- "o1": 16,
1384
- "o2": 16,
1385
- "o3": 16,
1386
- "i1": 16,
1387
- "i2": 16,
1388
- "i3": 16,
1389
- "r0": 1,
1390
- "r1": 186,
1391
- "r2": 186,
1392
- "nnz": 838860
1393
- },
1394
- "model.layers.16.mlp.gate_proj": {
1395
- "o1": 16,
1396
- "o2": 16,
1397
- "o3": 43,
1398
- "i1": 16,
1399
- "i2": 16,
1400
- "i3": 16,
1401
- "r0": 1,
1402
- "r1": 256,
1403
- "r2": 359,
1404
- "nnz": 2254438
1405
- },
1406
- "model.layers.16.mlp.up_proj": {
1407
- "o1": 16,
1408
- "o2": 16,
1409
- "o3": 43,
1410
- "i1": 16,
1411
- "i2": 16,
1412
- "i3": 16,
1413
- "r0": 1,
1414
- "r1": 256,
1415
- "r2": 359,
1416
- "nnz": 2254438
1417
- },
1418
- "model.layers.16.mlp.down_proj": {
1419
- "o1": 16,
1420
- "o2": 16,
1421
- "o3": 16,
1422
- "i1": 16,
1423
- "i2": 16,
1424
- "i3": 43,
1425
- "r0": 1,
1426
- "r1": 256,
1427
- "r2": 359,
1428
- "nnz": 2254438
1429
- },
1430
- "model.layers.17.self_attn.q_proj": {
1431
- "o1": 16,
1432
- "o2": 16,
1433
- "o3": 16,
1434
- "i1": 16,
1435
- "i2": 16,
1436
- "i3": 16,
1437
- "r0": 1,
1438
- "r1": 186,
1439
- "r2": 186,
1440
- "nnz": 838860
1441
- },
1442
- "model.layers.17.self_attn.k_proj": {
1443
- "o1": 16,
1444
- "o2": 16,
1445
- "o3": 16,
1446
- "i1": 16,
1447
- "i2": 16,
1448
- "i3": 16,
1449
- "r0": 1,
1450
- "r1": 186,
1451
- "r2": 186,
1452
- "nnz": 838860
1453
- },
1454
- "model.layers.17.self_attn.v_proj": {
1455
- "o1": 16,
1456
- "o2": 16,
1457
- "o3": 16,
1458
- "i1": 16,
1459
- "i2": 16,
1460
- "i3": 16,
1461
- "r0": 1,
1462
- "r1": 186,
1463
- "r2": 186,
1464
- "nnz": 838860
1465
- },
1466
- "model.layers.17.self_attn.o_proj": {
1467
- "o1": 16,
1468
- "o2": 16,
1469
- "o3": 16,
1470
- "i1": 16,
1471
- "i2": 16,
1472
- "i3": 16,
1473
- "r0": 1,
1474
- "r1": 186,
1475
- "r2": 186,
1476
- "nnz": 838860
1477
- },
1478
- "model.layers.17.mlp.gate_proj": {
1479
- "o1": 16,
1480
- "o2": 16,
1481
- "o3": 43,
1482
- "i1": 16,
1483
- "i2": 16,
1484
- "i3": 16,
1485
- "r0": 1,
1486
- "r1": 256,
1487
- "r2": 359,
1488
- "nnz": 2254438
1489
- },
1490
- "model.layers.17.mlp.up_proj": {
1491
- "o1": 16,
1492
- "o2": 16,
1493
- "o3": 43,
1494
- "i1": 16,
1495
- "i2": 16,
1496
- "i3": 16,
1497
- "r0": 1,
1498
- "r1": 256,
1499
- "r2": 359,
1500
- "nnz": 2254438
1501
- },
1502
- "model.layers.17.mlp.down_proj": {
1503
- "o1": 16,
1504
- "o2": 16,
1505
- "o3": 16,
1506
- "i1": 16,
1507
- "i2": 16,
1508
- "i3": 43,
1509
- "r0": 1,
1510
- "r1": 256,
1511
- "r2": 359,
1512
- "nnz": 2254438
1513
- },
1514
- "model.layers.18.self_attn.q_proj": {
1515
- "o1": 16,
1516
- "o2": 16,
1517
- "o3": 16,
1518
- "i1": 16,
1519
- "i2": 16,
1520
- "i3": 16,
1521
- "r0": 1,
1522
- "r1": 186,
1523
- "r2": 186,
1524
- "nnz": 838860
1525
- },
1526
- "model.layers.18.self_attn.k_proj": {
1527
- "o1": 16,
1528
- "o2": 16,
1529
- "o3": 16,
1530
- "i1": 16,
1531
- "i2": 16,
1532
- "i3": 16,
1533
- "r0": 1,
1534
- "r1": 186,
1535
- "r2": 186,
1536
- "nnz": 838860
1537
- },
1538
- "model.layers.18.self_attn.v_proj": {
1539
- "o1": 16,
1540
- "o2": 16,
1541
- "o3": 16,
1542
- "i1": 16,
1543
- "i2": 16,
1544
- "i3": 16,
1545
- "r0": 1,
1546
- "r1": 186,
1547
- "r2": 186,
1548
- "nnz": 838860
1549
- },
1550
- "model.layers.18.self_attn.o_proj": {
1551
- "o1": 16,
1552
- "o2": 16,
1553
- "o3": 16,
1554
- "i1": 16,
1555
- "i2": 16,
1556
- "i3": 16,
1557
- "r0": 1,
1558
- "r1": 186,
1559
- "r2": 186,
1560
- "nnz": 838860
1561
- },
1562
- "model.layers.18.mlp.gate_proj": {
1563
- "o1": 16,
1564
- "o2": 16,
1565
- "o3": 43,
1566
- "i1": 16,
1567
- "i2": 16,
1568
- "i3": 16,
1569
- "r0": 1,
1570
- "r1": 256,
1571
- "r2": 359,
1572
- "nnz": 2254438
1573
- },
1574
- "model.layers.18.mlp.up_proj": {
1575
- "o1": 16,
1576
- "o2": 16,
1577
- "o3": 43,
1578
- "i1": 16,
1579
- "i2": 16,
1580
- "i3": 16,
1581
- "r0": 1,
1582
- "r1": 256,
1583
- "r2": 359,
1584
- "nnz": 2254438
1585
- },
1586
- "model.layers.18.mlp.down_proj": {
1587
- "o1": 16,
1588
- "o2": 16,
1589
- "o3": 16,
1590
- "i1": 16,
1591
- "i2": 16,
1592
- "i3": 43,
1593
- "r0": 1,
1594
- "r1": 256,
1595
- "r2": 359,
1596
- "nnz": 2254438
1597
- },
1598
- "model.layers.19.self_attn.q_proj": {
1599
- "o1": 16,
1600
- "o2": 16,
1601
- "o3": 16,
1602
- "i1": 16,
1603
- "i2": 16,
1604
- "i3": 16,
1605
- "r0": 1,
1606
- "r1": 186,
1607
- "r2": 186,
1608
- "nnz": 838860
1609
- },
1610
- "model.layers.19.self_attn.k_proj": {
1611
- "o1": 16,
1612
- "o2": 16,
1613
- "o3": 16,
1614
- "i1": 16,
1615
- "i2": 16,
1616
- "i3": 16,
1617
- "r0": 1,
1618
- "r1": 186,
1619
- "r2": 186,
1620
- "nnz": 838860
1621
- },
1622
- "model.layers.19.self_attn.v_proj": {
1623
- "o1": 16,
1624
- "o2": 16,
1625
- "o3": 16,
1626
- "i1": 16,
1627
- "i2": 16,
1628
- "i3": 16,
1629
- "r0": 1,
1630
- "r1": 186,
1631
- "r2": 186,
1632
- "nnz": 838860
1633
- },
1634
- "model.layers.19.self_attn.o_proj": {
1635
- "o1": 16,
1636
- "o2": 16,
1637
- "o3": 16,
1638
- "i1": 16,
1639
- "i2": 16,
1640
- "i3": 16,
1641
- "r0": 1,
1642
- "r1": 186,
1643
- "r2": 186,
1644
- "nnz": 838860
1645
- },
1646
- "model.layers.19.mlp.gate_proj": {
1647
- "o1": 16,
1648
- "o2": 16,
1649
- "o3": 43,
1650
- "i1": 16,
1651
- "i2": 16,
1652
- "i3": 16,
1653
- "r0": 1,
1654
- "r1": 256,
1655
- "r2": 359,
1656
- "nnz": 2254438
1657
- },
1658
- "model.layers.19.mlp.up_proj": {
1659
- "o1": 16,
1660
- "o2": 16,
1661
- "o3": 43,
1662
- "i1": 16,
1663
- "i2": 16,
1664
- "i3": 16,
1665
- "r0": 1,
1666
- "r1": 256,
1667
- "r2": 359,
1668
- "nnz": 2254438
1669
- },
1670
- "model.layers.19.mlp.down_proj": {
1671
- "o1": 16,
1672
- "o2": 16,
1673
- "o3": 16,
1674
- "i1": 16,
1675
- "i2": 16,
1676
- "i3": 43,
1677
- "r0": 1,
1678
- "r1": 256,
1679
- "r2": 359,
1680
- "nnz": 2254438
1681
- },
1682
- "model.layers.20.self_attn.q_proj": {
1683
- "o1": 16,
1684
- "o2": 16,
1685
- "o3": 16,
1686
- "i1": 16,
1687
- "i2": 16,
1688
- "i3": 16,
1689
- "r0": 1,
1690
- "r1": 186,
1691
- "r2": 186,
1692
- "nnz": 838860
1693
- },
1694
- "model.layers.20.self_attn.k_proj": {
1695
- "o1": 16,
1696
- "o2": 16,
1697
- "o3": 16,
1698
- "i1": 16,
1699
- "i2": 16,
1700
- "i3": 16,
1701
- "r0": 1,
1702
- "r1": 186,
1703
- "r2": 186,
1704
- "nnz": 838860
1705
- },
1706
- "model.layers.20.self_attn.v_proj": {
1707
- "o1": 16,
1708
- "o2": 16,
1709
- "o3": 16,
1710
- "i1": 16,
1711
- "i2": 16,
1712
- "i3": 16,
1713
- "r0": 1,
1714
- "r1": 186,
1715
- "r2": 186,
1716
- "nnz": 838860
1717
- },
1718
- "model.layers.20.self_attn.o_proj": {
1719
- "o1": 16,
1720
- "o2": 16,
1721
- "o3": 16,
1722
- "i1": 16,
1723
- "i2": 16,
1724
- "i3": 16,
1725
- "r0": 1,
1726
- "r1": 186,
1727
- "r2": 186,
1728
- "nnz": 838860
1729
- },
1730
- "model.layers.20.mlp.gate_proj": {
1731
- "o1": 16,
1732
- "o2": 16,
1733
- "o3": 43,
1734
- "i1": 16,
1735
- "i2": 16,
1736
- "i3": 16,
1737
- "r0": 1,
1738
- "r1": 256,
1739
- "r2": 359,
1740
- "nnz": 2254438
1741
- },
1742
- "model.layers.20.mlp.up_proj": {
1743
- "o1": 16,
1744
- "o2": 16,
1745
- "o3": 43,
1746
- "i1": 16,
1747
- "i2": 16,
1748
- "i3": 16,
1749
- "r0": 1,
1750
- "r1": 256,
1751
- "r2": 359,
1752
- "nnz": 2254438
1753
- },
1754
- "model.layers.20.mlp.down_proj": {
1755
- "o1": 16,
1756
- "o2": 16,
1757
- "o3": 16,
1758
- "i1": 16,
1759
- "i2": 16,
1760
- "i3": 43,
1761
- "r0": 1,
1762
- "r1": 256,
1763
- "r2": 359,
1764
- "nnz": 2254438
1765
- },
1766
- "model.layers.21.self_attn.q_proj": {
1767
- "o1": 16,
1768
- "o2": 16,
1769
- "o3": 16,
1770
- "i1": 16,
1771
- "i2": 16,
1772
- "i3": 16,
1773
- "r0": 1,
1774
- "r1": 186,
1775
- "r2": 186,
1776
- "nnz": 838860
1777
- },
1778
- "model.layers.21.self_attn.k_proj": {
1779
- "o1": 16,
1780
- "o2": 16,
1781
- "o3": 16,
1782
- "i1": 16,
1783
- "i2": 16,
1784
- "i3": 16,
1785
- "r0": 1,
1786
- "r1": 186,
1787
- "r2": 186,
1788
- "nnz": 838860
1789
- },
1790
- "model.layers.21.self_attn.v_proj": {
1791
- "o1": 16,
1792
- "o2": 16,
1793
- "o3": 16,
1794
- "i1": 16,
1795
- "i2": 16,
1796
- "i3": 16,
1797
- "r0": 1,
1798
- "r1": 186,
1799
- "r2": 186,
1800
- "nnz": 838860
1801
- },
1802
- "model.layers.21.self_attn.o_proj": {
1803
- "o1": 16,
1804
- "o2": 16,
1805
- "o3": 16,
1806
- "i1": 16,
1807
- "i2": 16,
1808
- "i3": 16,
1809
- "r0": 1,
1810
- "r1": 186,
1811
- "r2": 186,
1812
- "nnz": 838860
1813
- },
1814
- "model.layers.21.mlp.gate_proj": {
1815
- "o1": 16,
1816
- "o2": 16,
1817
- "o3": 43,
1818
- "i1": 16,
1819
- "i2": 16,
1820
- "i3": 16,
1821
- "r0": 1,
1822
- "r1": 256,
1823
- "r2": 359,
1824
- "nnz": 2254438
1825
- },
1826
- "model.layers.21.mlp.up_proj": {
1827
- "o1": 16,
1828
- "o2": 16,
1829
- "o3": 43,
1830
- "i1": 16,
1831
- "i2": 16,
1832
- "i3": 16,
1833
- "r0": 1,
1834
- "r1": 256,
1835
- "r2": 359,
1836
- "nnz": 2254438
1837
- },
1838
- "model.layers.21.mlp.down_proj": {
1839
- "o1": 16,
1840
- "o2": 16,
1841
- "o3": 16,
1842
- "i1": 16,
1843
- "i2": 16,
1844
- "i3": 43,
1845
- "r0": 1,
1846
- "r1": 256,
1847
- "r2": 359,
1848
- "nnz": 2254438
1849
- },
1850
- "model.layers.22.self_attn.q_proj": {
1851
- "o1": 16,
1852
- "o2": 16,
1853
- "o3": 16,
1854
- "i1": 16,
1855
- "i2": 16,
1856
- "i3": 16,
1857
- "r0": 1,
1858
- "r1": 186,
1859
- "r2": 186,
1860
- "nnz": 838860
1861
- },
1862
- "model.layers.22.self_attn.k_proj": {
1863
- "o1": 16,
1864
- "o2": 16,
1865
- "o3": 16,
1866
- "i1": 16,
1867
- "i2": 16,
1868
- "i3": 16,
1869
- "r0": 1,
1870
- "r1": 186,
1871
- "r2": 186,
1872
- "nnz": 838860
1873
- },
1874
- "model.layers.22.self_attn.v_proj": {
1875
- "o1": 16,
1876
- "o2": 16,
1877
- "o3": 16,
1878
- "i1": 16,
1879
- "i2": 16,
1880
- "i3": 16,
1881
- "r0": 1,
1882
- "r1": 186,
1883
- "r2": 186,
1884
- "nnz": 838860
1885
- },
1886
- "model.layers.22.self_attn.o_proj": {
1887
- "o1": 16,
1888
- "o2": 16,
1889
- "o3": 16,
1890
- "i1": 16,
1891
- "i2": 16,
1892
- "i3": 16,
1893
- "r0": 1,
1894
- "r1": 186,
1895
- "r2": 186,
1896
- "nnz": 838860
1897
- },
1898
- "model.layers.22.mlp.gate_proj": {
1899
- "o1": 16,
1900
- "o2": 16,
1901
- "o3": 43,
1902
- "i1": 16,
1903
- "i2": 16,
1904
- "i3": 16,
1905
- "r0": 1,
1906
- "r1": 256,
1907
- "r2": 359,
1908
- "nnz": 2254438
1909
- },
1910
- "model.layers.22.mlp.up_proj": {
1911
- "o1": 16,
1912
- "o2": 16,
1913
- "o3": 43,
1914
- "i1": 16,
1915
- "i2": 16,
1916
- "i3": 16,
1917
- "r0": 1,
1918
- "r1": 256,
1919
- "r2": 359,
1920
- "nnz": 2254438
1921
- },
1922
- "model.layers.22.mlp.down_proj": {
1923
- "o1": 16,
1924
- "o2": 16,
1925
- "o3": 16,
1926
- "i1": 16,
1927
- "i2": 16,
1928
- "i3": 43,
1929
- "r0": 1,
1930
- "r1": 256,
1931
- "r2": 359,
1932
- "nnz": 2254438
1933
- },
1934
- "model.layers.23.self_attn.q_proj": {
1935
- "o1": 16,
1936
- "o2": 16,
1937
- "o3": 16,
1938
- "i1": 16,
1939
- "i2": 16,
1940
- "i3": 16,
1941
- "r0": 1,
1942
- "r1": 186,
1943
- "r2": 186,
1944
- "nnz": 838860
1945
- },
1946
- "model.layers.23.self_attn.k_proj": {
1947
- "o1": 16,
1948
- "o2": 16,
1949
- "o3": 16,
1950
- "i1": 16,
1951
- "i2": 16,
1952
- "i3": 16,
1953
- "r0": 1,
1954
- "r1": 186,
1955
- "r2": 186,
1956
- "nnz": 838860
1957
- },
1958
- "model.layers.23.self_attn.v_proj": {
1959
- "o1": 16,
1960
- "o2": 16,
1961
- "o3": 16,
1962
- "i1": 16,
1963
- "i2": 16,
1964
- "i3": 16,
1965
- "r0": 1,
1966
- "r1": 186,
1967
- "r2": 186,
1968
- "nnz": 838860
1969
- },
1970
- "model.layers.23.self_attn.o_proj": {
1971
- "o1": 16,
1972
- "o2": 16,
1973
- "o3": 16,
1974
- "i1": 16,
1975
- "i2": 16,
1976
- "i3": 16,
1977
- "r0": 1,
1978
- "r1": 186,
1979
- "r2": 186,
1980
- "nnz": 838860
1981
- },
1982
- "model.layers.23.mlp.gate_proj": {
1983
- "o1": 16,
1984
- "o2": 16,
1985
- "o3": 43,
1986
- "i1": 16,
1987
- "i2": 16,
1988
- "i3": 16,
1989
- "r0": 1,
1990
- "r1": 256,
1991
- "r2": 359,
1992
- "nnz": 2254438
1993
- },
1994
- "model.layers.23.mlp.up_proj": {
1995
- "o1": 16,
1996
- "o2": 16,
1997
- "o3": 43,
1998
- "i1": 16,
1999
- "i2": 16,
2000
- "i3": 16,
2001
- "r0": 1,
2002
- "r1": 256,
2003
- "r2": 359,
2004
- "nnz": 2254438
2005
- },
2006
- "model.layers.23.mlp.down_proj": {
2007
- "o1": 16,
2008
- "o2": 16,
2009
- "o3": 16,
2010
- "i1": 16,
2011
- "i2": 16,
2012
- "i3": 43,
2013
- "r0": 1,
2014
- "r1": 256,
2015
- "r2": 359,
2016
- "nnz": 2254438
2017
- },
2018
- "model.layers.24.self_attn.q_proj": {
2019
- "o1": 16,
2020
- "o2": 16,
2021
- "o3": 16,
2022
- "i1": 16,
2023
- "i2": 16,
2024
- "i3": 16,
2025
- "r0": 1,
2026
- "r1": 186,
2027
- "r2": 186,
2028
- "nnz": 838860
2029
- },
2030
- "model.layers.24.self_attn.k_proj": {
2031
- "o1": 16,
2032
- "o2": 16,
2033
- "o3": 16,
2034
- "i1": 16,
2035
- "i2": 16,
2036
- "i3": 16,
2037
- "r0": 1,
2038
- "r1": 186,
2039
- "r2": 186,
2040
- "nnz": 838860
2041
- },
2042
- "model.layers.24.self_attn.v_proj": {
2043
- "o1": 16,
2044
- "o2": 16,
2045
- "o3": 16,
2046
- "i1": 16,
2047
- "i2": 16,
2048
- "i3": 16,
2049
- "r0": 1,
2050
- "r1": 186,
2051
- "r2": 186,
2052
- "nnz": 838860
2053
- },
2054
- "model.layers.24.self_attn.o_proj": {
2055
- "o1": 16,
2056
- "o2": 16,
2057
- "o3": 16,
2058
- "i1": 16,
2059
- "i2": 16,
2060
- "i3": 16,
2061
- "r0": 1,
2062
- "r1": 186,
2063
- "r2": 186,
2064
- "nnz": 838860
2065
- },
2066
- "model.layers.24.mlp.gate_proj": {
2067
- "o1": 16,
2068
- "o2": 16,
2069
- "o3": 43,
2070
- "i1": 16,
2071
- "i2": 16,
2072
- "i3": 16,
2073
- "r0": 1,
2074
- "r1": 256,
2075
- "r2": 359,
2076
- "nnz": 2254438
2077
- },
2078
- "model.layers.24.mlp.up_proj": {
2079
- "o1": 16,
2080
- "o2": 16,
2081
- "o3": 43,
2082
- "i1": 16,
2083
- "i2": 16,
2084
- "i3": 16,
2085
- "r0": 1,
2086
- "r1": 256,
2087
- "r2": 359,
2088
- "nnz": 2254438
2089
- },
2090
- "model.layers.24.mlp.down_proj": {
2091
- "o1": 16,
2092
- "o2": 16,
2093
- "o3": 16,
2094
- "i1": 16,
2095
- "i2": 16,
2096
- "i3": 43,
2097
- "r0": 1,
2098
- "r1": 256,
2099
- "r2": 359,
2100
- "nnz": 2254438
2101
- },
2102
- "model.layers.25.self_attn.q_proj": {
2103
- "o1": 16,
2104
- "o2": 16,
2105
- "o3": 16,
2106
- "i1": 16,
2107
- "i2": 16,
2108
- "i3": 16,
2109
- "r0": 1,
2110
- "r1": 186,
2111
- "r2": 186,
2112
- "nnz": 838860
2113
- },
2114
- "model.layers.25.self_attn.k_proj": {
2115
- "o1": 16,
2116
- "o2": 16,
2117
- "o3": 16,
2118
- "i1": 16,
2119
- "i2": 16,
2120
- "i3": 16,
2121
- "r0": 1,
2122
- "r1": 186,
2123
- "r2": 186,
2124
- "nnz": 838860
2125
- },
2126
- "model.layers.25.self_attn.v_proj": {
2127
- "o1": 16,
2128
- "o2": 16,
2129
- "o3": 16,
2130
- "i1": 16,
2131
- "i2": 16,
2132
- "i3": 16,
2133
- "r0": 1,
2134
- "r1": 186,
2135
- "r2": 186,
2136
- "nnz": 838860
2137
- },
2138
- "model.layers.25.self_attn.o_proj": {
2139
- "o1": 16,
2140
- "o2": 16,
2141
- "o3": 16,
2142
- "i1": 16,
2143
- "i2": 16,
2144
- "i3": 16,
2145
- "r0": 1,
2146
- "r1": 186,
2147
- "r2": 186,
2148
- "nnz": 838860
2149
- },
2150
- "model.layers.25.mlp.gate_proj": {
2151
- "o1": 16,
2152
- "o2": 16,
2153
- "o3": 43,
2154
- "i1": 16,
2155
- "i2": 16,
2156
- "i3": 16,
2157
- "r0": 1,
2158
- "r1": 256,
2159
- "r2": 359,
2160
- "nnz": 2254438
2161
- },
2162
- "model.layers.25.mlp.up_proj": {
2163
- "o1": 16,
2164
- "o2": 16,
2165
- "o3": 43,
2166
- "i1": 16,
2167
- "i2": 16,
2168
- "i3": 16,
2169
- "r0": 1,
2170
- "r1": 256,
2171
- "r2": 359,
2172
- "nnz": 2254438
2173
- },
2174
- "model.layers.25.mlp.down_proj": {
2175
- "o1": 16,
2176
- "o2": 16,
2177
- "o3": 16,
2178
- "i1": 16,
2179
- "i2": 16,
2180
- "i3": 43,
2181
- "r0": 1,
2182
- "r1": 256,
2183
- "r2": 359,
2184
- "nnz": 2254438
2185
- },
2186
- "model.layers.26.self_attn.q_proj": {
2187
- "o1": 16,
2188
- "o2": 16,
2189
- "o3": 16,
2190
- "i1": 16,
2191
- "i2": 16,
2192
- "i3": 16,
2193
- "r0": 1,
2194
- "r1": 186,
2195
- "r2": 186,
2196
- "nnz": 838860
2197
- },
2198
- "model.layers.26.self_attn.k_proj": {
2199
- "o1": 16,
2200
- "o2": 16,
2201
- "o3": 16,
2202
- "i1": 16,
2203
- "i2": 16,
2204
- "i3": 16,
2205
- "r0": 1,
2206
- "r1": 186,
2207
- "r2": 186,
2208
- "nnz": 838860
2209
- },
2210
- "model.layers.26.self_attn.v_proj": {
2211
- "o1": 16,
2212
- "o2": 16,
2213
- "o3": 16,
2214
- "i1": 16,
2215
- "i2": 16,
2216
- "i3": 16,
2217
- "r0": 1,
2218
- "r1": 186,
2219
- "r2": 186,
2220
- "nnz": 838860
2221
- },
2222
- "model.layers.26.self_attn.o_proj": {
2223
- "o1": 16,
2224
- "o2": 16,
2225
- "o3": 16,
2226
- "i1": 16,
2227
- "i2": 16,
2228
- "i3": 16,
2229
- "r0": 1,
2230
- "r1": 186,
2231
- "r2": 186,
2232
- "nnz": 838860
2233
- },
2234
- "model.layers.26.mlp.gate_proj": {
2235
- "o1": 16,
2236
- "o2": 16,
2237
- "o3": 43,
2238
- "i1": 16,
2239
- "i2": 16,
2240
- "i3": 16,
2241
- "r0": 1,
2242
- "r1": 256,
2243
- "r2": 359,
2244
- "nnz": 2254438
2245
- },
2246
- "model.layers.26.mlp.up_proj": {
2247
- "o1": 16,
2248
- "o2": 16,
2249
- "o3": 43,
2250
- "i1": 16,
2251
- "i2": 16,
2252
- "i3": 16,
2253
- "r0": 1,
2254
- "r1": 256,
2255
- "r2": 359,
2256
- "nnz": 2254438
2257
- },
2258
- "model.layers.26.mlp.down_proj": {
2259
- "o1": 16,
2260
- "o2": 16,
2261
- "o3": 16,
2262
- "i1": 16,
2263
- "i2": 16,
2264
- "i3": 43,
2265
- "r0": 1,
2266
- "r1": 256,
2267
- "r2": 359,
2268
- "nnz": 2254438
2269
- },
2270
- "model.layers.27.self_attn.q_proj": {
2271
- "o1": 16,
2272
- "o2": 16,
2273
- "o3": 16,
2274
- "i1": 16,
2275
- "i2": 16,
2276
- "i3": 16,
2277
- "r0": 1,
2278
- "r1": 186,
2279
- "r2": 186,
2280
- "nnz": 838860
2281
- },
2282
- "model.layers.27.self_attn.k_proj": {
2283
- "o1": 16,
2284
- "o2": 16,
2285
- "o3": 16,
2286
- "i1": 16,
2287
- "i2": 16,
2288
- "i3": 16,
2289
- "r0": 1,
2290
- "r1": 186,
2291
- "r2": 186,
2292
- "nnz": 838860
2293
- },
2294
- "model.layers.27.self_attn.v_proj": {
2295
- "o1": 16,
2296
- "o2": 16,
2297
- "o3": 16,
2298
- "i1": 16,
2299
- "i2": 16,
2300
- "i3": 16,
2301
- "r0": 1,
2302
- "r1": 186,
2303
- "r2": 186,
2304
- "nnz": 838860
2305
- },
2306
- "model.layers.27.self_attn.o_proj": {
2307
- "o1": 16,
2308
- "o2": 16,
2309
- "o3": 16,
2310
- "i1": 16,
2311
- "i2": 16,
2312
- "i3": 16,
2313
- "r0": 1,
2314
- "r1": 186,
2315
- "r2": 186,
2316
- "nnz": 838860
2317
- },
2318
- "model.layers.27.mlp.gate_proj": {
2319
- "o1": 16,
2320
- "o2": 16,
2321
- "o3": 43,
2322
- "i1": 16,
2323
- "i2": 16,
2324
- "i3": 16,
2325
- "r0": 1,
2326
- "r1": 256,
2327
- "r2": 359,
2328
- "nnz": 2254438
2329
- },
2330
- "model.layers.27.mlp.up_proj": {
2331
- "o1": 16,
2332
- "o2": 16,
2333
- "o3": 43,
2334
- "i1": 16,
2335
- "i2": 16,
2336
- "i3": 16,
2337
- "r0": 1,
2338
- "r1": 256,
2339
- "r2": 359,
2340
- "nnz": 2254438
2341
- },
2342
- "model.layers.27.mlp.down_proj": {
2343
- "o1": 16,
2344
- "o2": 16,
2345
- "o3": 16,
2346
- "i1": 16,
2347
- "i2": 16,
2348
- "i3": 43,
2349
- "r0": 1,
2350
- "r1": 256,
2351
- "r2": 359,
2352
- "nnz": 2254438
2353
- },
2354
- "model.layers.28.self_attn.q_proj": {
2355
- "o1": 16,
2356
- "o2": 16,
2357
- "o3": 16,
2358
- "i1": 16,
2359
- "i2": 16,
2360
- "i3": 16,
2361
- "r0": 1,
2362
- "r1": 186,
2363
- "r2": 186,
2364
- "nnz": 838860
2365
- },
2366
- "model.layers.28.self_attn.k_proj": {
2367
- "o1": 16,
2368
- "o2": 16,
2369
- "o3": 16,
2370
- "i1": 16,
2371
- "i2": 16,
2372
- "i3": 16,
2373
- "r0": 1,
2374
- "r1": 186,
2375
- "r2": 186,
2376
- "nnz": 838860
2377
- },
2378
- "model.layers.28.self_attn.v_proj": {
2379
- "o1": 16,
2380
- "o2": 16,
2381
- "o3": 16,
2382
- "i1": 16,
2383
- "i2": 16,
2384
- "i3": 16,
2385
- "r0": 1,
2386
- "r1": 186,
2387
- "r2": 186,
2388
- "nnz": 838860
2389
- },
2390
- "model.layers.28.self_attn.o_proj": {
2391
- "o1": 16,
2392
- "o2": 16,
2393
- "o3": 16,
2394
- "i1": 16,
2395
- "i2": 16,
2396
- "i3": 16,
2397
- "r0": 1,
2398
- "r1": 186,
2399
- "r2": 186,
2400
- "nnz": 838860
2401
- },
2402
- "model.layers.28.mlp.gate_proj": {
2403
- "o1": 16,
2404
- "o2": 16,
2405
- "o3": 43,
2406
- "i1": 16,
2407
- "i2": 16,
2408
- "i3": 16,
2409
- "r0": 1,
2410
- "r1": 256,
2411
- "r2": 359,
2412
- "nnz": 2254438
2413
- },
2414
- "model.layers.28.mlp.up_proj": {
2415
- "o1": 16,
2416
- "o2": 16,
2417
- "o3": 43,
2418
- "i1": 16,
2419
- "i2": 16,
2420
- "i3": 16,
2421
- "r0": 1,
2422
- "r1": 256,
2423
- "r2": 359,
2424
- "nnz": 2254438
2425
- },
2426
- "model.layers.28.mlp.down_proj": {
2427
- "o1": 16,
2428
- "o2": 16,
2429
- "o3": 16,
2430
- "i1": 16,
2431
- "i2": 16,
2432
- "i3": 43,
2433
- "r0": 1,
2434
- "r1": 256,
2435
- "r2": 359,
2436
- "nnz": 2254438
2437
- },
2438
- "model.layers.29.self_attn.q_proj": {
2439
- "o1": 16,
2440
- "o2": 16,
2441
- "o3": 16,
2442
- "i1": 16,
2443
- "i2": 16,
2444
- "i3": 16,
2445
- "r0": 1,
2446
- "r1": 186,
2447
- "r2": 186,
2448
- "nnz": 838860
2449
- },
2450
- "model.layers.29.self_attn.k_proj": {
2451
- "o1": 16,
2452
- "o2": 16,
2453
- "o3": 16,
2454
- "i1": 16,
2455
- "i2": 16,
2456
- "i3": 16,
2457
- "r0": 1,
2458
- "r1": 186,
2459
- "r2": 186,
2460
- "nnz": 838860
2461
- },
2462
- "model.layers.29.self_attn.v_proj": {
2463
- "o1": 16,
2464
- "o2": 16,
2465
- "o3": 16,
2466
- "i1": 16,
2467
- "i2": 16,
2468
- "i3": 16,
2469
- "r0": 1,
2470
- "r1": 186,
2471
- "r2": 186,
2472
- "nnz": 838860
2473
- },
2474
- "model.layers.29.self_attn.o_proj": {
2475
- "o1": 16,
2476
- "o2": 16,
2477
- "o3": 16,
2478
- "i1": 16,
2479
- "i2": 16,
2480
- "i3": 16,
2481
- "r0": 1,
2482
- "r1": 186,
2483
- "r2": 186,
2484
- "nnz": 838860
2485
- },
2486
- "model.layers.29.mlp.gate_proj": {
2487
- "o1": 16,
2488
- "o2": 16,
2489
- "o3": 43,
2490
- "i1": 16,
2491
- "i2": 16,
2492
- "i3": 16,
2493
- "r0": 1,
2494
- "r1": 256,
2495
- "r2": 359,
2496
- "nnz": 2254438
2497
- },
2498
- "model.layers.29.mlp.up_proj": {
2499
- "o1": 16,
2500
- "o2": 16,
2501
- "o3": 43,
2502
- "i1": 16,
2503
- "i2": 16,
2504
- "i3": 16,
2505
- "r0": 1,
2506
- "r1": 256,
2507
- "r2": 359,
2508
- "nnz": 2254438
2509
- },
2510
- "model.layers.29.mlp.down_proj": {
2511
- "o1": 16,
2512
- "o2": 16,
2513
- "o3": 16,
2514
- "i1": 16,
2515
- "i2": 16,
2516
- "i3": 43,
2517
- "r0": 1,
2518
- "r1": 256,
2519
- "r2": 359,
2520
- "nnz": 2254438
2521
- },
2522
- "model.layers.30.self_attn.q_proj": {
2523
- "o1": 16,
2524
- "o2": 16,
2525
- "o3": 16,
2526
- "i1": 16,
2527
- "i2": 16,
2528
- "i3": 16,
2529
- "r0": 1,
2530
- "r1": 186,
2531
- "r2": 186,
2532
- "nnz": 838860
2533
- },
2534
- "model.layers.30.self_attn.k_proj": {
2535
- "o1": 16,
2536
- "o2": 16,
2537
- "o3": 16,
2538
- "i1": 16,
2539
- "i2": 16,
2540
- "i3": 16,
2541
- "r0": 1,
2542
- "r1": 186,
2543
- "r2": 186,
2544
- "nnz": 838860
2545
- },
2546
- "model.layers.30.self_attn.v_proj": {
2547
- "o1": 16,
2548
- "o2": 16,
2549
- "o3": 16,
2550
- "i1": 16,
2551
- "i2": 16,
2552
- "i3": 16,
2553
- "r0": 1,
2554
- "r1": 186,
2555
- "r2": 186,
2556
- "nnz": 838860
2557
- },
2558
- "model.layers.30.self_attn.o_proj": {
2559
- "o1": 16,
2560
- "o2": 16,
2561
- "o3": 16,
2562
- "i1": 16,
2563
- "i2": 16,
2564
- "i3": 16,
2565
- "r0": 1,
2566
- "r1": 186,
2567
- "r2": 186,
2568
- "nnz": 838860
2569
- },
2570
- "model.layers.30.mlp.gate_proj": {
2571
- "o1": 16,
2572
- "o2": 16,
2573
- "o3": 43,
2574
- "i1": 16,
2575
- "i2": 16,
2576
- "i3": 16,
2577
- "r0": 1,
2578
- "r1": 256,
2579
- "r2": 359,
2580
- "nnz": 2254438
2581
- },
2582
- "model.layers.30.mlp.up_proj": {
2583
- "o1": 16,
2584
- "o2": 16,
2585
- "o3": 43,
2586
- "i1": 16,
2587
- "i2": 16,
2588
- "i3": 16,
2589
- "r0": 1,
2590
- "r1": 256,
2591
- "r2": 359,
2592
- "nnz": 2254438
2593
- },
2594
- "model.layers.30.mlp.down_proj": {
2595
- "o1": 16,
2596
- "o2": 16,
2597
- "o3": 16,
2598
- "i1": 16,
2599
- "i2": 16,
2600
- "i3": 43,
2601
- "r0": 1,
2602
- "r1": 256,
2603
- "r2": 359,
2604
- "nnz": 2254438
2605
- },
2606
- "model.layers.31.self_attn.q_proj": {
2607
- "o1": 16,
2608
- "o2": 16,
2609
- "o3": 16,
2610
- "i1": 16,
2611
- "i2": 16,
2612
- "i3": 16,
2613
- "r0": 1,
2614
- "r1": 186,
2615
- "r2": 186,
2616
- "nnz": 838860
2617
- },
2618
- "model.layers.31.self_attn.k_proj": {
2619
- "o1": 16,
2620
- "o2": 16,
2621
- "o3": 16,
2622
- "i1": 16,
2623
- "i2": 16,
2624
- "i3": 16,
2625
- "r0": 1,
2626
- "r1": 186,
2627
- "r2": 186,
2628
- "nnz": 838860
2629
- },
2630
- "model.layers.31.self_attn.v_proj": {
2631
- "o1": 16,
2632
- "o2": 16,
2633
- "o3": 16,
2634
- "i1": 16,
2635
- "i2": 16,
2636
- "i3": 16,
2637
- "r0": 1,
2638
- "r1": 186,
2639
- "r2": 186,
2640
- "nnz": 838860
2641
- },
2642
- "model.layers.31.self_attn.o_proj": {
2643
- "o1": 16,
2644
- "o2": 16,
2645
- "o3": 16,
2646
- "i1": 16,
2647
- "i2": 16,
2648
- "i3": 16,
2649
- "r0": 1,
2650
- "r1": 186,
2651
- "r2": 186,
2652
- "nnz": 838860
2653
- },
2654
- "model.layers.31.mlp.gate_proj": {
2655
- "o1": 16,
2656
- "o2": 16,
2657
- "o3": 43,
2658
- "i1": 16,
2659
- "i2": 16,
2660
- "i3": 16,
2661
- "r0": 1,
2662
- "r1": 256,
2663
- "r2": 359,
2664
- "nnz": 2254438
2665
- },
2666
- "model.layers.31.mlp.up_proj": {
2667
- "o1": 16,
2668
- "o2": 16,
2669
- "o3": 43,
2670
- "i1": 16,
2671
- "i2": 16,
2672
- "i3": 16,
2673
- "r0": 1,
2674
- "r1": 256,
2675
- "r2": 359,
2676
- "nnz": 2254438
2677
- },
2678
- "model.layers.31.mlp.down_proj": {
2679
- "o1": 16,
2680
- "o2": 16,
2681
- "o3": 16,
2682
- "i1": 16,
2683
- "i2": 16,
2684
- "i3": 43,
2685
- "r0": 1,
2686
- "r1": 256,
2687
- "r2": 359,
2688
- "nnz": 2254438
2689
- }
2690
- }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
tt3-llama2-7b/state_dict_fp16_healed.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:dc5364e7a4ad33c977ce00a2ae3cdc957ac732993093512e5ac0bc524a43f3a8
3
- size 10632154115
 
 
 
 
tt3-llama2-7b/state_dict_fp16_unhealed.pt DELETED
@@ -1,3 +0,0 @@
1
- version https://git-lfs.github.com/spec/v1
2
- oid sha256:7c8471a71d2d1328bd4dcc824f2d71fddc40d7fe3745e16212e55374649b2625
3
- size 10632153632
 
 
 
 
tt3-llama2-7b/tensor_formats.py DELETED
@@ -1,890 +0,0 @@
1
- from __future__ import annotations
2
-
3
- """General N-core tensor formats for a linear layer: tensor train (MPO) and tensor ring.
4
-
5
- WHY THIS EXISTS. The 2-core and 3-core results say the tensorisation step, not the healing
6
- step, is where the leverage is: at equal parameters the 3-core beat the 2-core AFTER healing
7
- at both budgets tried (4.72B: 38.94 vs ~36.5 full MMLU at 60M tokens; 5.74B: 44.91 vs 41.54
8
- probe), and it got there on roughly a quarter of the tokens. That invites two orthogonal
9
- questions this module exists to answer: how many cores, and which topology.
10
-
11
- TENSOR TRAIN vs TENSOR RING. A TT/MPO pins the two boundary bonds to 1 -- the chain has open
12
- ends. A tensor ring closes the chain: core k has shape (r_{k-1}, d_k, r_k) with r_N == r_0,
13
- and the weight is the TRACE of the cyclic product. TT is exactly the TR special case r_0 = 1,
14
- so at equal ranks TR is strictly more expressive. It is also strictly harder to fit: TT-SVD
15
- is a sequence of Eckart-Young-optimal truncations, while a ring has no such closed form, so
16
- TR is fitted here by TT initialisation followed by alternating least squares. That asymmetry
17
- matters when reading results -- a TR loss could mean "the format is worse" OR "the fit did
18
- not converge", so `fit_tr` reports its per-sweep error curve and refuses to return a fit
19
- worse than its own TT initialisation.
20
-
21
- INDEX CONVENTION (shared with mpo2.py / mpo3.py). Output mode k travels with input mode k:
22
-
23
- R(W)[d_1, ..., d_N] = W[(o_1..o_N), (i_1..i_N)], d_k = (o_k, i_k) flattened
24
-
25
- Every mode must be > 1. A mode of 1 collapses a site and silently reduces the core count,
26
- which would make an "N-core" result actually be an (N-1)-core one.
27
-
28
- WHITENING is diagonal and factorised across the N input modes, as in mpo3.py: the whitening
29
- is then a mode-local rescaling that survives the fit exactly and un-whitens by division.
30
- """
31
-
32
- import torch
33
-
34
-
35
- # --------------------------------------------------------------------------- rearrangement
36
- def rearrange_n(W: torch.Tensor, osplit, isplit) -> torch.Tensor:
37
- """W[(o_1..o_N),(i_1..i_N)] -> R[d_1,...,d_N] with d_k = o_k*i_k."""
38
- n = len(osplit)
39
- if len(isplit) != n:
40
- raise ValueError("output and input must be split into the same number of modes")
41
- perm = [j for k in range(n) for j in (k, n + k)]
42
- return (W.reshape(*osplit, *isplit).permute(*perm)
43
- .reshape(*[o * i for o, i in zip(osplit, isplit)]))
44
-
45
-
46
- def inv_rearrange_n(R: torch.Tensor, osplit, isplit) -> torch.Tensor:
47
- """Inverse of `rearrange_n`."""
48
- n = len(osplit)
49
- inter = [x for k in range(n) for x in (osplit[k], isplit[k])]
50
- perm = list(range(0, 2 * n, 2)) + list(range(1, 2 * n, 2))
51
- o = 1
52
- i = 1
53
- for k in range(n):
54
- o *= osplit[k]
55
- i *= isplit[k]
56
- return R.reshape(*inter).permute(*perm).reshape(o, i)
57
-
58
-
59
- def _apply_scales(R: torch.Tensor, osplit, isplit, scales, invert: bool):
60
- """Multiply (or divide) R by sqrt(d_k[i_k]) along each input mode, in place of a dense L."""
61
- n = len(osplit)
62
- shape = [x for k in range(n) for x in (osplit[k], isplit[k])]
63
- T = R.reshape(*shape)
64
- for k in range(n):
65
- s = scales[k].clamp_min(1e-12).sqrt()
66
- view = [1] * (2 * n)
67
- view[2 * k + 1] = isplit[k]
68
- T = T / s.view(*view) if invert else T * s.view(*view)
69
- return T.reshape(R.shape)
70
-
71
-
72
- # --------------------------------------------------------------------------- contraction
73
- def contract_tt(cores) -> torch.Tensor:
74
- """Contract TT cores [(1,d1,r1), (r1,d2,r2), ..., (r_{N-1},dN,1)] -> R[d_1,...,d_N]."""
75
- out = cores[0]
76
- for g in cores[1:]:
77
- a, b = out.shape[-1], g.shape[0]
78
- if a != b:
79
- raise ValueError(f"TT bond mismatch {a} vs {b}")
80
- out = torch.tensordot(out, g, dims=([out.dim() - 1], [0]))
81
- if out.shape[0] != 1 or out.shape[-1] != 1:
82
- raise ValueError("TT boundary bonds must be 1")
83
- return out.reshape(*out.shape[1:-1])
84
-
85
-
86
- def contract_tr(cores, chunk: int = 0) -> torch.Tensor:
87
- """Contract TR cores [(r0,d1,r1), ..., (r_{N-1},dN,r0)] -> R[d_1,...,d_N].
88
-
89
- The obvious route -- chain every tensordot, then take the diagonal over the two ring
90
- indices -- materialises (r_0, d_1..d_N, r_0), i.e. r_0^2 times the size of the ANSWER.
91
- On one LLaMA attention layer at ring rank 118 that is 934 GB, so it does not merely run
92
- slowly, it cannot run at all. Folding the last contraction and the trace into a single
93
- einsum keeps the peak at r_0 * prod(d_1..d_{N-1}) * r_{N-1}, and chunking the first mode
94
- bounds it further for the wide MLP layers.
95
- """
96
- n = len(cores)
97
- if cores[0].shape[0] != cores[-1].shape[-1]:
98
- raise ValueError(f"ring does not close: {cores[0].shape[0]} vs {cores[-1].shape[-1]}")
99
- d0 = cores[0].shape[1]
100
- if chunk and d0 > chunk:
101
- parts = []
102
- for lo in range(0, d0, chunk):
103
- sub = [cores[0][:, lo:min(lo + chunk, d0), :]] + list(cores[1:])
104
- parts.append(contract_tr(sub, chunk=0))
105
- return torch.cat(parts, dim=0)
106
- out = cores[0]
107
- for g in cores[1:-1]:
108
- out = torch.tensordot(out, g, dims=([out.dim() - 1], [0]))
109
- return torch.einsum("i...k,kdi->...d", out, cores[-1])
110
-
111
-
112
- # --------------------------------------------------------------------------- fitting
113
- def fit_tt(R: torch.Tensor, ranks):
114
- """TT-SVD with the given internal ranks. Singular values split symmetrically at each bond,
115
- matching fit_mpo2/fit_mpo3's gauge so cores stay on comparable scales (fp16-safe and
116
- kinder to a blockwise quantizer)."""
117
- dims = list(R.shape)
118
- n = len(dims)
119
- if len(ranks) != n - 1:
120
- raise ValueError(f"{n} sites need {n-1} internal ranks, got {len(ranks)}")
121
- cores, rest, rprev = [], R.reshape(dims[0], -1), 1
122
- for k in range(n - 1):
123
- M = rest.reshape(rprev * dims[k], -1)
124
- U, S, Vh = torch.linalg.svd(M, full_matrices=False)
125
- r = min(int(ranks[k]), S.shape[0])
126
- q = S[:r].clamp_min(0).sqrt()
127
- cores.append((U[:, :r] * q).reshape(rprev, dims[k], r))
128
- rest = q[:, None] * Vh[:r]
129
- rprev = r
130
- cores.append(rest.reshape(rprev, dims[-1], 1))
131
- return cores
132
-
133
-
134
- def tt_to_tr(cores):
135
- """Reinterpret TT cores as TR cores with r_0 = 1 (an exact, lossless re-labelling)."""
136
- return [c.clone() for c in cores]
137
-
138
-
139
- def _subchain(cores, k):
140
- """Contract every core except k, in cyclic order, -> E[r_k, D, r_{k-1}] with D = prod_{j!=k} d_j."""
141
- n = len(cores)
142
- order = [(k + 1 + t) % n for t in range(n - 1)]
143
- out = cores[order[0]]
144
- for j in order[1:]:
145
- out = torch.tensordot(out, cores[j], dims=([out.dim() - 1], [0]))
146
- return out
147
-
148
-
149
- def _subchain_chunk(cores, k, lo, hi):
150
- """`_subchain` with the FIRST following mode restricted to [lo, hi). The restriction rides
151
- along the contraction untouched, so concatenating the chunks reproduces `_subchain`."""
152
- n = len(cores)
153
- order = [(k + 1 + t) % n for t in range(n - 1)]
154
- out = cores[order[0]][:, lo:hi, :]
155
- for j in order[1:]:
156
- out = torch.tensordot(out, cores[j], dims=([out.dim() - 1], [0]))
157
- return out
158
-
159
-
160
- def rank_of(cores, k):
161
- """r_{k-1} * r_k for core k -- the width of that core's least-squares system."""
162
- n = len(cores)
163
- return cores[k].shape[0] * cores[k].shape[2]
164
-
165
-
166
- def _solve_maybe_cpu(AtA, AtW):
167
- """Solve AtA X = AtW, falling back to the CPU when the GPU cannot hold the factorisation.
168
-
169
- The normal-equation matrix is (r_{k-1} r_k) square, i.e. it grows as r^4. On down_proj at
170
- ring rank 219 that is 47961^2 = 9.2 GB, and the factorisation needs a second copy -- no
171
- amount of chunking the data helps, because this matrix does not depend on the data size.
172
- Cholesky is tried first (the Gram is symmetric positive definite after the ridge) and
173
- needs only one triangular factor; if even that does not fit, the solve moves to the host.
174
- Slow, but a bounded slowdown beats an unbounded failure on the layers that matter most.
175
-
176
- THE LADDER MUST CATCH SINGULARITY, NOT ONLY OOM. A normal-equation matrix can be
177
- genuinely rank-deficient, not merely too big: when a requested bond exceeds what its
178
- adjacent mode can carry (e.g. Qwen2.5-3B's q_proj splits to dims [64, 256, 256] and a
179
- budget-matched sweep asks for r=92 > 64), the design matrix loses column rank and
180
- `torch.linalg.solve` raises _LinAlgError. Routing only OutOfMemoryError to the host left
181
- that case falling straight through the ladder and crashing the run -- the opposite of this
182
- function's stated contract. A rank-deficient ALS subproblem is still perfectly well posed:
183
- every solution attains the same fit, so the last rung takes the minimum-norm one via
184
- lstsq, which is exactly what the "lstsq" solver mode would have done anyway.
185
- """
186
- if not (torch.isfinite(AtA).all() and torch.isfinite(AtW).all()):
187
- raise ValueError("REFUSING: non-finite normal equations. cuSOLVER does not fail fast "
188
- "on NaN input, it stalls, so this is caught here rather than handed "
189
- "to a solver that cannot be interrupted.")
190
- try:
191
- L = torch.linalg.cholesky(AtA)
192
- return torch.cholesky_solve(AtW, L)
193
- except torch.OutOfMemoryError:
194
- torch.cuda.empty_cache()
195
- except Exception:
196
- pass
197
- try:
198
- return torch.linalg.solve(AtA, AtW)
199
- except torch.OutOfMemoryError:
200
- torch.cuda.empty_cache()
201
- except Exception:
202
- pass
203
- # ESCALATING RIDGE, ON THE GPU, BEFORE LEAVING IT. A rank-deficient Gram is the common
204
- # case on unbalanced splits (small first mode), not a rare accident, so the host fallback
205
- # must not be the routine path: CPU float64 gelsd on a few-thousand-square system costs
206
- # minutes, and ALS calls this 3 times per sweep per restart, which turned a Qwen2.5-3B
207
- # sweep that should take minutes into one that produced no output for 40+ minutes. Tikhonov
208
- # regularisation restores positive-definiteness for a cost the fit does not notice: the
209
- # ridge is relative to the Gram's own scale and is escalated only until a solve succeeds.
210
- scale = float(AtA.diagonal().abs().mean().clamp_min(1e-30))
211
- eye = torch.eye(AtA.shape[0], dtype=AtA.dtype, device=AtA.device)
212
- # THE LADDER MUST TERMINATE ON THE GPU. AtA is positive SEMI-definite by construction, so
213
- # AtA + c*I is positive definite for any c > 0 and Cholesky is guaranteed to succeed once
214
- # the ridge is large enough -- singularity therefore never needs the host at all. The
215
- # previous last rung sent it to CPU float64 gelsd, which is single-threaded, unbounded, and
216
- # does not check Python signals: on Qwen2.5-3B's k/v_proj, whose TT bond lands exactly on
217
- # the mode dimension (r1 = d0 = 32), the ALS subproblem is EXACTLY rank-saturated, every
218
- # solve took that path, and a compression that needs ~1s per layer without ALS ran 48
219
- # minutes for one layer and could not even be interrupted with SIGINT.
220
- #
221
- # The ridge distorts the subproblem, but ALS is iterative and fit_tr keeps the best iterate
222
- # with a never-worse-than-TT guard, so a heavily-ridged sweep can only fail to improve --
223
- # it cannot corrupt the result. An unbounded stall can.
224
- for mult in (1e-6, 1e-4, 1e-2, 1e-1, 1.0, 10.0):
225
- try:
226
- L = torch.linalg.cholesky(AtA + (mult * scale) * eye)
227
- return torch.cholesky_solve(AtW, L)
228
- except torch.OutOfMemoryError:
229
- torch.cuda.empty_cache()
230
- break # a memory problem, not a conditioning one -- go to host
231
- except Exception:
232
- continue
233
- del eye
234
- # Only OOM reaches here now. The host path stays for that case, where it IS the right
235
- # answer: slow but bounded beats failing outright on the layers that need the most rank.
236
- dev = AtA.device
237
- A64, W64 = AtA.cpu().double(), AtW.cpu().double()
238
- try:
239
- X = torch.linalg.solve(A64, W64)
240
- except Exception:
241
- X = torch.linalg.lstsq(A64, W64, driver="gelsd").solution
242
- return X.to(device=dev, dtype=AtW.dtype)
243
-
244
-
245
- def tr_rel_err(cores, R: torch.Tensor, chunk_elems: int = 250_000_000) -> float:
246
- """||contract_tr(cores) - R||_F / ||R||_F, accumulated in chunks.
247
-
248
- Materialising the reconstruction to subtract it needs the reconstruction AND the einsum's
249
- intermediate at once; on down_proj at ring rank 155 that intermediate is (155, 64, 688,
250
- 256) = 7 GB even with the first mode already chunked. Accumulating the squared error one
251
- slice at a time keeps only one slice live, and the chunk width is derived from an element
252
- budget rather than a hard-coded constant -- a constant is exactly what silently failed to
253
- engage when a mode happened to equal it.
254
- """
255
- n = len(cores)
256
- d0 = cores[0].shape[1]
257
- mid = 1
258
- for k in range(1, n - 1):
259
- mid *= cores[k].shape[1]
260
- per_unit = max(1, mid * cores[0].shape[0] * cores[-1].shape[0])
261
- step = max(1, min(d0, int(chunk_elems // per_unit)))
262
- acc = 0.0
263
- for lo in range(0, d0, step):
264
- hi = min(lo + step, d0)
265
- sub = [cores[0][:, lo:hi, :]] + list(cores[1:])
266
- diff = contract_tr(sub) - R[lo:hi]
267
- acc += float(diff.pow(2).sum())
268
- del sub, diff
269
- return (acc ** 0.5) / float(R.norm())
270
-
271
-
272
- def fit_tr(R: torch.Tensor, ranks, sweeps: int = 30, tol: float = 1e-8, verbose: bool = False,
273
- seed: int = 1234, restarts: int = 2, solver: str = "lstsq",
274
- chunk: int = 0, chunk_elems: int = 250_000_000):
275
- """Fit a tensor ring by TT initialisation + alternating least squares.
276
-
277
- ranks = [r_0, r_1, ..., r_{N-1}] with r_N == r_0. The TT init uses r_0 = 1 (a legal ring),
278
- then each ALS sweep re-solves every core in closed form given the others; the ring's extra
279
- boundary rank is opened up by padding the initialisation. Because ALS on a ring is not
280
- guaranteed to improve monotonically from an arbitrary start, the best iterate seen is kept
281
- and the routine REFUSES to return anything worse than its own TT initialisation -- without
282
- that guard a fitting failure would be indistinguishable from the format being worse, which
283
- is precisely the comparison this module exists to make.
284
- """
285
- n = len(R.shape)
286
- if len(ranks) != n:
287
- raise ValueError(f"a ring over {n} sites needs {n} ranks (r_0..r_{{N-1}})")
288
- dims = list(R.shape)
289
- rnorm = float(R.norm())
290
- rnorm2 = float(R.pow(2).sum())
291
- last_sq = None
292
- tt = fit_tt(R, ranks[1:])
293
- base_err = float((contract_tt(tt) - R).norm() / R.norm())
294
- r0 = int(ranks[0])
295
- gen = torch.Generator(device="cpu").manual_seed(seed)
296
-
297
- def padded_init():
298
- """TT solution embedded in the ring, with the OPENED bond seeded on BOTH ends.
299
-
300
- Seeding only one end is not enough and is a silent failure: core 0's least-squares
301
- environment is built from cores 1..N-1, whose ring index comes from core N-1's
302
- trailing bond. If that stays zero, the environment vanishes on exactly the components
303
- the ring adds, every ALS solve returns zero there, and the fit never leaves the TT
304
- solution -- it looks like "the ring buys nothing" rather than "the ring was never
305
- initialised".
306
- """
307
- out = []
308
- for k, c in enumerate(tt):
309
- a, d, b = c.shape
310
- na = r0 if k == 0 else a
311
- nb = r0 if k == n - 1 else b
312
- g = torch.zeros(na, d, nb, dtype=R.dtype, device=R.device)
313
- g[:a, :, :b] = c
314
- eps = 0.05 * float(c.norm()) / max(1, c.numel()) ** 0.5
315
- if k == 0 and na > a:
316
- g[a:, :, :] = eps * torch.randn(na - a, d, nb, generator=gen,
317
- dtype=R.dtype).to(R.device)
318
- if k == n - 1 and nb > b:
319
- g[:, :, b:] = eps * torch.randn(na, d, nb - b, generator=gen,
320
- dtype=R.dtype).to(R.device)
321
- out.append(g)
322
- return out
323
-
324
- def random_init(g=None):
325
- """i.i.d. cores whose ring contraction has the right variance.
326
-
327
- The scale is derived, not guessed. For cores with i.i.d. N(0, s^2) entries the trace
328
- of the cyclic product is a sum of prod_k r_k independent products of N Gaussians, so
329
- Var(W) ~= (prod_k r_k) * s^(2N). Matching that to the target mean square gives
330
- s = (E[R^2] / prod_k r_k)^(1/(2N)). Getting this wrong by a constant factor per core
331
- compounds to that factor^N in the product, which for N=4,5 puts the initial iterate
332
- many orders of magnitude away from the data and leaves ALS crawling.
333
- """
334
- rr = [int(ranks[k]) for k in range(n)]
335
- prod = 1.0
336
- for x in rr:
337
- prod *= x
338
- ms = float(R.pow(2).mean())
339
- s = (ms / prod) ** (1.0 / (2 * n))
340
- gg = g if g is not None else gen
341
- return [s * torch.randn(rr[k], dims[k], rr[(k + 1) % n], generator=gg,
342
- dtype=R.dtype).to(R.device) for k in range(n)]
343
-
344
- # ALS on a ring has genuine local minima -- a single start that stalls would be reported
345
- # as "the ring is not better", which is exactly the confound this module must avoid. So
346
- # several starts are run and the best kept; restarts are cheap next to the SVDs anyway.
347
- inits = [("tt_padded", padded_init)]
348
- for j in range(restarts):
349
- g = torch.Generator(device="cpu").manual_seed(seed + 977 * (j + 1))
350
- inits.append((f"random{j}", lambda g=g: random_init(g)))
351
- best, best_err, hists, winner = tt_to_tr(tt), base_err, {}, "tt_init"
352
- for init_name, init_fn in inits:
353
- cores, ihist = init_fn(), []
354
- for it in range(sweeps):
355
- # A restart that has diverged produces non-finite normal equations. Feeding
356
- # those to a solver is not merely wasteful: cuSOLVER's LU on a NaN matrix
357
- # does not fail fast, it goes pathological -- a 2336x2336 solve sat for 80
358
- # minutes and could not be interrupted, because `clamp_min` does not clamp
359
- # NaN so the ridge itself became NaN and poisoned the whole diagonal.
360
- diverged = False
361
- for k in range(n):
362
- # The subchain leaves every d index separate and in CYCLIC order starting at
363
- # k+1. R's remaining modes must be flattened in that SAME order -- movedim
364
- # gives natural order, which for interior k is a different permutation and
365
- # would solve the least-squares problem against mismatched columns. It still
366
- # returns a finite "solution", just to the wrong system.
367
- # The chunk decision is made from SHAPES ONLY and BEFORE anything large is
368
- # built. An earlier version computed the full subchain first and then decided
369
- # whether to chunk, so the unchunked allocation always happened anyway and the
370
- # chunked branch saved nothing -- the memory-bounded path existed but could
371
- # never take effect.
372
- # Auto-size the chunk from a memory budget instead of comparing against a
373
- # fixed number. A constant threshold silently does nothing when the mode
374
- # happens to equal it -- which is exactly what let a 4-site fit try to
375
- # materialise a 134 GB subchain (mode 64, chunk 64, "64 > 64" is false).
376
- m_ = (k + 1) % n
377
- other = 1
378
- for j in range(n):
379
- if j != k:
380
- other *= dims[j]
381
- per_unit = max(1, (other // max(1, dims[m_])) * rank_of(cores, k))
382
- auto = max(1, min(dims[m_], int(chunk_elems // per_unit)))
383
- use_chunk = auto < dims[m_]
384
- if solver == "normal" and use_chunk:
385
- # Memory-bounded normal equations. Materialising the subchain costs
386
- # r_k * (prod_{j!=k} d_j) * r_{k-1} floats -- on down_proj at these ranks
387
- # that is (155, 688, 256, 155) = 17 GB, so the unchunked path cannot run
388
- # on the MLP layers at all. The Gram and right-hand side are sums over the
389
- # flattened "others" index, so they accumulate exactly over slices of the
390
- # first subchain mode; this is the same arithmetic, not an approximation.
391
- m = m_
392
- d_next = dims[m]
393
- rk_, rkm1_ = None, None
394
- AtA = AtW = None
395
- Rp = R.permute(*([k] + [(k + 1 + t) % n for t in range(n - 1)]))
396
- for lo in range(0, d_next, auto):
397
- hi = min(lo + auto, d_next)
398
- Ec = _subchain_chunk(cores, k, lo, hi)
399
- rk_, rkm1_ = Ec.shape[0], Ec.shape[-1]
400
- Ec = Ec.reshape(rk_, -1, rkm1_)
401
- Ac = Ec.permute(1, 2, 0).reshape(Ec.shape[1], rkm1_ * rk_)
402
- Wc = Rp[:, lo:hi].reshape(dims[k], -1)
403
- g = Ac.transpose(0, 1) @ Ac
404
- h = Ac.transpose(0, 1) @ Wc.transpose(0, 1)
405
- AtA = g if AtA is None else AtA + g
406
- AtW = h if AtW is None else AtW + h
407
- del Ec, Ac, Wc, g, h
408
- if not torch.isfinite(AtA).all():
409
- diverged = True
410
- del AtA, AtW
411
- break
412
- lam = 1e-8 * float(AtA.diagonal().abs().mean().clamp_min(1e-30))
413
- AtA_r = AtA + lam * torch.eye(AtA.shape[0], dtype=AtA.dtype,
414
- device=AtA.device)
415
- try:
416
- G = _solve_maybe_cpu(AtA_r, AtW)
417
- except ValueError:
418
- # Non-finite normal equations mean THIS restart has diverged, whichever
419
- # operand carries the NaN -- checking AtA alone missed a case where the
420
- # right-hand side A^T W was the non-finite one and killed the whole
421
- # compression at layer 15 of 252. Catching the guard covers every
422
- # source at once, and abandoning the restart is safe because `best` is
423
- # untouched and fit_tr still falls back to the TT solution.
424
- diverged = True
425
- del AtA_r, AtW
426
- break
427
- del AtA_r
428
- # ||R - A G||^2 = ||R||^2 - 2<AtW, G> + <G, AtA G>, all from quantities
429
- # already formed. Computing it by contracting the whole ring instead would
430
- # cost one full O(prod d_k) tensor per sweep per core.
431
- sq = rnorm2 - 2.0 * float((AtW * G).sum()) + float((G * (AtA @ G)).sum())
432
- last_sq = max(sq, 0.0)
433
- rkm1, rk = rkm1_, rk_
434
- else:
435
- # ONE branch builds A/Wk and ONE consumes them. Splitting the decision
436
- # across two separate if/else chains left the non-chunked solve reachable
437
- # after the chunked path had run, where `A` is whatever the PREVIOUS core
438
- # left behind -- a silent wrong answer rather than an error.
439
- E = _subchain(cores, k) # (r_k, d_{k+1},..,d_{k-1}, r_{k-1})
440
- rk, rkm1 = E.shape[0], E.shape[-1]
441
- E = E.reshape(rk, -1, rkm1)
442
- D = E.shape[1]
443
- A = E.permute(1, 2, 0).reshape(D, rkm1 * rk)
444
- perm = [k] + [(k + 1 + t) % n for t in range(n - 1)]
445
- # R's remaining modes must be flattened in the subchain's CYCLIC order --
446
- # movedim gives natural order, which for interior k solves the
447
- # least-squares problem against mismatched columns and still returns a
448
- # finite "solution", just to the wrong system.
449
- Wk = R.permute(*perm).reshape(dims[k], D)
450
- del E
451
- if solver == "normal":
452
- AtA = A.transpose(0, 1) @ A
453
- if not torch.isfinite(AtA).all():
454
- diverged = True
455
- del AtA, A, Wk
456
- break
457
- lam = 1e-8 * float(AtA.diagonal().abs().mean().clamp_min(1e-30))
458
- AtA += lam * torch.eye(AtA.shape[0], dtype=A.dtype, device=A.device)
459
- try:
460
- G = _solve_maybe_cpu(AtA, A.transpose(0, 1) @ Wk.transpose(0, 1))
461
- except ValueError:
462
- diverged = True
463
- del AtA, A, Wk
464
- break
465
- del AtA
466
- else:
467
- G = torch.linalg.lstsq(A, Wk.transpose(0, 1)).solution
468
- last_sq = None
469
- del A, Wk
470
- cores[k] = (G.transpose(0, 1).reshape(dims[k], rkm1, rk)
471
- .permute(1, 0, 2).contiguous())
472
- if diverged:
473
- # Abandon this initialisation. `best`/`best_err` are untouched, and fit_tr's
474
- # never-worse-than-TT guard still applies, so a diverged restart can only fail
475
- # to contribute -- never corrupt the returned cores.
476
- ihist.append(float("nan"))
477
- break
478
- err = (float(last_sq) ** 0.5 / rnorm if last_sq is not None
479
- else float((contract_tr(cores, chunk=chunk) - R).norm() / rnorm))
480
- ihist.append(round(err, 8))
481
- if verbose:
482
- print(f" TR-ALS[{init_name}] sweep {it+1}: rel err {err:.6f}", flush=True)
483
- if err < best_err - 1e-12:
484
- best, best_err, winner = [c.clone() for c in cores], err, init_name
485
- if it and abs(ihist[-2] - err) < tol * max(err, 1e-12):
486
- break
487
- hists[init_name] = ihist
488
- # The algebraic residual is a difference of nearly equal large numbers; at a relative
489
- # error of ~1e-2 it has already lost most of its fp32 significance. It is fine for
490
- # steering ALS but not for the number that gets reported, so the winner is re-measured
491
- # once by explicit contraction.
492
- best_err = tr_rel_err(best, R)
493
- if best_err > base_err:
494
- return tt_to_tr(tt), base_err, {"fell_back_to_tt": True, "tt_err": base_err,
495
- "winner": "tt_init", "als_hist": hists}
496
- return best, best_err, {"fell_back_to_tt": winner == "tt_init", "tt_err": base_err,
497
- "winner": winner, "als_hist": hists}
498
-
499
-
500
- # --------------------------------------------------------------------------- bookkeeping
501
- def tt_params(dims, ranks) -> int:
502
- r = [1] + [int(x) for x in ranks] + [1]
503
- return sum(r[k] * dims[k] * r[k + 1] for k in range(len(dims)))
504
-
505
-
506
- def tr_params(dims, ranks) -> int:
507
- r = [int(x) for x in ranks]
508
- n = len(dims)
509
- return sum(r[k] * dims[k] * r[(k + 1) % n] for k in range(n))
510
-
511
-
512
- def tt_rank_ceilings(dims):
513
- """r_k <= min(prod_{j<=k} d_j, prod_{j>k} d_j): above that the truncation is vacuous."""
514
- n, out = len(dims), []
515
- for k in range(n - 1):
516
- left = 1
517
- for j in range(k + 1):
518
- left *= dims[j]
519
- right = 1
520
- for j in range(k + 1, n):
521
- right *= dims[j]
522
- out.append(min(left, right))
523
- return out
524
-
525
-
526
- def ranks_for_budget(dims, budget: int, ring: bool, alpha: float = 1.0):
527
- """Largest ranks fitting `budget`, shaped by r_k proportional to ceiling_k ** alpha.
528
-
529
- Bisection rather than a closed form: the count is linear in each rank but bilinear in
530
- adjacent pairs, and cyclic for a ring, so there is no single quadratic to solve.
531
-
532
- ALPHA EXISTS BECAUSE EQUAL FILL FRACTION IS WRONG WHEN THE CEILINGS ARE LOPSIDED. For a
533
- 4-site split of 4096 the ceilings are (64, 4096, 64): alpha=1 spends the budget on the
534
- huge middle bond and returns ranks like (36, 2327, 36), strangling the two bonds that
535
- actually gate information flow through the chain. alpha=0 gives every bond the same rank
536
- instead. Neither is right a priori, so the caller is expected to MEASURE across alpha --
537
- the reconstruction error, not the heuristic, decides.
538
- """
539
- ceil_tt = tt_rank_ceilings(dims)
540
- ceils = ([min(ceil_tt)] + ceil_tt) if ring else ceil_tt
541
- fn = tr_params if ring else tt_params
542
- mx = max(ceils)
543
- shape = [(c / mx) ** alpha for c in ceils]
544
-
545
- def build(t):
546
- return [max(1, min(int(round(t * mx * s)), c)) for s, c in zip(shape, ceils)]
547
-
548
- lo, hi = 0.0, 1.0
549
- if fn(dims, build(1.0)) <= budget:
550
- return build(1.0)
551
- for _ in range(60):
552
- mid = (lo + hi) / 2
553
- if fn(dims, build(mid)) > budget:
554
- hi = mid
555
- else:
556
- lo = mid
557
- r = build(lo)
558
- while fn(dims, r) > budget and max(r) > 1:
559
- r[r.index(max(r))] -= 1
560
- return r
561
-
562
-
563
- def balanced_split(n: int, k: int):
564
- """Most balanced k-way factorisation of n with every factor >= 2 (spread-minimising)."""
565
- best = None
566
- def rec(rem, parts):
567
- nonlocal best
568
- if len(parts) == k - 1:
569
- if rem >= 2:
570
- cand = parts + [rem]
571
- key = max(cand) / min(cand)
572
- if best is None or key < best[0]:
573
- best = (key, cand)
574
- return
575
- d = 2
576
- while d * d <= rem or d <= rem:
577
- if d > rem:
578
- break
579
- if rem % d == 0:
580
- rec(rem // d, parts + [d])
581
- d += 1
582
- rec(n, [])
583
- if best is None:
584
- raise ValueError(f"no {k}-way factorisation of {n} with all factors >= 2")
585
- return sorted(best[1])
586
-
587
-
588
- def pair_modes(o: int, i: int, k: int = 3):
589
- """Split O and I into k factors each, and PAIR them so the paired dims come out balanced.
590
-
591
- Paired-mode tensorisation multiplies the j-th output factor by the j-th input factor, so
592
- the pairing ORDER decides the shape of the tensor the ring has to fit -- and balanced_split
593
- returns its factors sorted, which pairs smallest with smallest and largest with largest.
594
- That is the worst available choice. On Qwen2.5-3B's 2048x2048 projections both sides split
595
- as [8, 16, 16], and the sorted pairing yields dims [64, 256, 256]: a first mode of only 64
596
- caps the adjacent bond at r_0 * 64, while a budget-matched sweep asks for r ~= 92, so every
597
- ALS normal-equation system is rank-deficient BY CONSTRUCTION. The observed cost was a sweep
598
- that burned 2.5 hours of CPU inside the ridge/lstsq ladder and emitted not one data point.
599
-
600
- Searching the k! pairings for the one that minimises max(dims)/min(dims) turns that same
601
- layer into [128, 128, 256], which is well conditioned. The search is over 6 orders for
602
- k = 3, so it costs nothing.
603
-
604
- This is deliberately a no-op on LLaMA-2-7B, whose shapes were verified individually:
605
- q/k/v/o_proj [256, 256, 256], gate/up_proj and down_proj [256, 256, 688] both before and
606
- after. Every result already measured on that model therefore remains valid, and the sweep
607
- and the production compressor keep agreeing with each other.
608
- """
609
- from itertools import permutations
610
- osp = tuple(balanced_split(o, k))
611
- best = None
612
- for perm in permutations(balanced_split(i, k)):
613
- dims = [a * b for a, b in zip(osp, perm)]
614
- spread = max(dims) / min(dims)
615
- if best is None or spread < best[0]:
616
- best = (spread, tuple(perm))
617
- return osp, best[1]
618
-
619
-
620
- def self_test(device: str = "cpu") -> None:
621
- torch.manual_seed(0)
622
- dt = torch.float64
623
- for osp, isp in [((4, 3), (2, 5)), ((4, 3, 2), (2, 5, 3)), ((2, 3, 2, 2), (2, 2, 3, 2))]:
624
- n = len(osp)
625
- out = 1
626
- inn = 1
627
- for x in osp:
628
- out *= x
629
- for x in isp:
630
- inn *= x
631
- W = torch.randn(out, inn, dtype=dt, device=device)
632
- R = rearrange_n(W, osp, isp)
633
- dims = [o * i for o, i in zip(osp, isp)]
634
- assert list(R.shape) == dims, (R.shape, dims)
635
- assert torch.equal(inv_rearrange_n(R, osp, isp), W), f"{n}-site rearrange not invertible"
636
- # index identity, not just the round trip: a wrong-but-self-inverse permutation would
637
- # pass the round trip and scramble every weight silently
638
- for _ in range(40):
639
- oi = [torch.randint(0, x, (1,)).item() for x in osp]
640
- ii = [torch.randint(0, x, (1,)).item() for x in isp]
641
- ridx = tuple(oi[k] * isp[k] + ii[k] for k in range(n))
642
- orow = 0
643
- for k in range(n):
644
- orow = orow * osp[k] + oi[k]
645
- ocol = 0
646
- for k in range(n):
647
- ocol = ocol * isp[k] + ii[k]
648
- assert R[ridx] == W[orow, ocol], f"{n}-site rearrange index map wrong"
649
-
650
- ceils = tt_rank_ceilings(dims)
651
- cores = fit_tt(R, ceils)
652
- e = float((contract_tt(cores) - R).norm() / R.norm())
653
- assert e < 1e-10, f"{n}-site full-rank TT is not exact ({e:.3e})"
654
- assert tt_params(dims, ceils) == sum(c.numel() for c in cores), "tt_params disagrees"
655
-
656
- if n >= 3:
657
- trunc = [max(1, c // 2) for c in ceils]
658
- cores = fit_tt(R, trunc)
659
- assert tt_params(dims, trunc) == sum(c.numel() for c in cores)
660
- for a in (0.0, 0.5, 1.0):
661
- rr = ranks_for_budget(dims, tt_params(dims, trunc), ring=False, alpha=a)
662
- assert tt_params(dims, rr) <= tt_params(dims, trunc), \
663
- f"TT budget solver overran at alpha={a}"
664
- # a ring at the SAME parameter budget must not do worse than the TT it starts from
665
- budget = tt_params(dims, trunc)
666
- rring = ranks_for_budget(dims, budget, ring=True)
667
- assert tr_params(dims, rring) <= budget, "TR budget solver overran"
668
- _c, err_tr, info = fit_tr(R, rring, sweeps=6)
669
- assert err_tr <= info["tt_err"] + 1e-12, "fit_tr returned worse than its own TT init"
670
- assert abs(float((contract_tr(_c) - R).norm() / R.norm()) - err_tr) < 1e-8, \
671
- "reported TR error does not match the returned cores"
672
-
673
- ring_chk = [torch.randn(3, 5, 4, dtype=dt), torch.randn(4, 6, 2, dtype=dt),
674
- torch.randn(2, 7, 3, dtype=dt)]
675
- full = contract_tr(ring_chk)
676
- assert torch.allclose(full, contract_tr(ring_chk, chunk=2), atol=1e-12), \
677
- "chunked contract_tr disagrees with the unchunked one"
678
- ref = torch.einsum("iaj,jbk,kci->abc", *ring_chk)
679
- assert torch.allclose(full, ref, atol=1e-12), "contract_tr is not the cyclic trace"
680
-
681
- # a ring with r_0 > 1 must be able to represent things a TT of the same internal ranks cannot
682
- dims = [6, 6, 6]
683
- torch.manual_seed(1)
684
- ring = [torch.randn(3, 6, 3, dtype=dt) for _ in range(3)]
685
- Rr = contract_tr(ring)
686
- tt_same = fit_tt(Rr, [3, 3])
687
- e_tt = float((contract_tt(tt_same) - Rr).norm() / Rr.norm())
688
- assert e_tt > 1e-3, ("a rank-(3,3,3) ring was reproduced by a rank-(3,3) TT; the ring "
689
- f"contraction is probably not closing the loop (err {e_tt:.3e})")
690
- for slv in ("lstsq", "normal"):
691
- got, err, info = fit_tr(Rr, [3, 3, 3], sweeps=200, restarts=3, solver=slv)
692
- assert err < 1e-4, (f"TR-ALS[{slv}] failed to recover an exact rank-3 ring "
693
- f"(err {err:.3e}); winner={info['winner']} "
694
- f"tt_init={info['tt_err']:.3e}")
695
- got, err, info = fit_tr(Rr, [3, 3, 3], sweeps=200, restarts=3)
696
- # chunking changes only how the Gram is accumulated, so it must reach the same optimum
697
- _g, err_ch, _i = fit_tr(Rr, [3, 3, 3], sweeps=200, restarts=3, solver="normal", chunk=2)
698
- assert err_ch < 1e-4, f"chunked TR-ALS failed to recover an exact ring (err {err_ch:.3e})"
699
- assert err < 1e-4, (f"TR-ALS failed to recover an exact rank-3 ring (err {err:.3e}); "
700
- f"winner={info['winner']} tt_init={info['tt_err']:.3e}")
701
-
702
- assert balanced_split(4096, 3) == [16, 16, 16], balanced_split(4096, 3)
703
- assert balanced_split(4096, 2) == [64, 64]
704
- b = balanced_split(11008, 3)
705
- assert 43 in b and b[0] * b[1] * b[2] == 11008, b
706
- print("tensor_formats self-test PASSED (N-site rearrange exact & index-checked, full-rank "
707
- "TT exact, budget solvers within budget, TR >= its TT init, TR strictly beats TT on "
708
- "a true ring, balanced splits correct)")
709
-
710
-
711
- if __name__ == "__main__":
712
- self_test()
713
-
714
-
715
- # --------------------------------------------------------------------------- layers
716
- class TRLinear(torch.nn.Module):
717
- """Tensor-ring linear layer: W = Tr[G_0 G_1 ... G_{N-1}] after the shared rearrangement.
718
-
719
- NOT a LowRankLinear subclass, unlike MPO2Linear/MPO3Linear. Those two could pretend to be
720
- a pair of factors because a train has open ends; a ring genuinely has N 3-D cores and no
721
- A/B roles to borrow, so anything keyed on `.A`/`.B` (the blockwise quantizer, the bond
722
- rebalancer) has to be taught about `.cores` explicitly rather than silently doing the
723
- wrong thing to a subclass it thinks it understands.
724
-
725
- RECONSTRUCTION IS CHUNKED BY DEFAULT. Merging any two cores of a 3-site ring costs
726
- r*d_i*d_j*r entries -- on down_proj at ring rank 142 that is 7.1 GB in bf16, per layer,
727
- per forward. Chunking the first mode bounds it; the cost is more, smaller matmuls.
728
- """
729
-
730
- def __init__(self, cores, osplit, isplit, chunk_elems: int = 500_000_000,
731
- bias=None):
732
- super().__init__()
733
- n = len(cores)
734
- if len(osplit) != n or len(isplit) != n:
735
- raise ValueError(f"{n} cores need {n} output and {n} input modes")
736
- if min(list(osplit) + list(isplit)) < 2:
737
- raise ValueError(f"degenerate split o={tuple(osplit)} i={tuple(isplit)}: a mode of "
738
- f"1 collapses a site, making this an (N-1)-site network")
739
- for k in range(n):
740
- if cores[k].shape[1] != osplit[k] * isplit[k]:
741
- raise ValueError(f"core {k} has mode {cores[k].shape[1]}, expected "
742
- f"{osplit[k]}*{isplit[k]}")
743
- if cores[k].shape[2] != cores[(k + 1) % n].shape[0]:
744
- raise ValueError(f"ring does not close between cores {k} and {(k+1)%n}")
745
- self.cores = torch.nn.ParameterList([torch.nn.Parameter(c) for c in cores])
746
- self.osplit, self.isplit = tuple(int(x) for x in osplit), tuple(int(x) for x in isplit)
747
- o = i = 1
748
- for k in range(n):
749
- o *= self.osplit[k]
750
- i *= self.isplit[k]
751
- self.out_features, self.in_features = o, i
752
- self.ranks = tuple(int(c.shape[0]) for c in cores)
753
- # LLaMA projections have no bias, but Qwen2's q/k/v DO. Dropping it when the dense
754
- # Linear is replaced is SILENT: shapes still match, the parameter-count gate still
755
- # balances (achieved excludes bias, and so would the rebuilt model), and only the
756
- # accuracy moves. So the bias is carried, not discarded. Defaults to None, which
757
- # leaves every LLaMA path byte-identical to before.
758
- self.bias = None if bias is None else torch.nn.Parameter(bias)
759
- # Elements per unit of the FIRST mode in contract_tr's peak intermediate, which is
760
- # (r_0, c, d_1, ..., d_{n-2}, r_{n-1}) -- the LAST core's mode is contracted away by
761
- # the closing einsum and must not be counted. Including it over-estimated the cost by
762
- # d_{n-1} (256x on an MLP layer), which drove the chunk to 1 and would have made every
763
- # MLP forward 256 sequential contractions instead of 8.
764
- mid = 1
765
- for k in range(1, n - 1):
766
- mid *= cores[k].shape[1]
767
- per_unit = max(1, mid * self.ranks[0] * self.ranks[-1])
768
- self._chunk = max(1, min(cores[0].shape[1], int(chunk_elems // per_unit)))
769
- # non-persistent: never wants to show up in a checkpoint, and materialize_() is an
770
- # inference-only opt-in (see below), so a freshly-loaded layer must start uncached.
771
- self.register_buffer("_cached_weight", None, persistent=False)
772
-
773
- def weight(self) -> torch.Tensor:
774
- if self._cached_weight is not None:
775
- return self._cached_weight
776
- cs = list(self.cores)
777
- R = contract_tr(cs, chunk=self._chunk if self._chunk < cs[0].shape[1] else 0)
778
- return inv_rearrange_n(R, self.osplit, self.isplit)
779
-
780
- @torch.no_grad()
781
- def materialize_(self):
782
- """Cache the dense weight once. INFERENCE ONLY -- gradients cannot flow through the
783
- cached tensor back to the cores, so this must never be called before/during training.
784
-
785
- forward() calls weight() on every call, which redoes the full chunked ring
786
- contraction (two tensor contractions + a trace per core) from scratch. That is fine
787
- for a single forward pass, but autoregressive generation calls forward() once per
788
- layer per GENERATED TOKEN with the cores completely unchanged between calls. Measured
789
- on tr3_ring4b04_h1's GSM8K eval before this existed: 209s/item, ETA 73h for 1319
790
- items -- every one of ~200 generated tokens re-contracted all 224 ring layers.
791
- A train layer (LowRankLinear/MPO2Linear) never had this problem: F.linear(F.linear(x,
792
- B), A) applies the two factors straight to the activation and never materializes W at
793
- all, so there was nothing to cache there.
794
- """
795
- self._cached_weight = self.weight().detach().clone()
796
-
797
- def unmaterialize_(self):
798
- self._cached_weight = None
799
-
800
- def forward(self, x):
801
- b = None if self.bias is None else self.bias.to(x.dtype)
802
- return torch.nn.functional.linear(x, self.weight().to(x.dtype), b)
803
-
804
- def extra_repr(self) -> str:
805
- return (f"o={self.osplit}, i={self.isplit}, ranks={self.ranks}, "
806
- f"chunk={self._chunk}, materialized={self._cached_weight is not None}")
807
-
808
-
809
- class SparseAugTRLinear(TRLinear):
810
- """TRLinear plus a sparse additive residual. Same int64-index contract as the MPO layers:
811
- `sp_idx` must NEVER be cast to a floating dtype -- a `.half()` on a 45M-scale index
812
- silently rounds it to a different, valid-looking position."""
813
-
814
- def __init__(self, cores, osplit, isplit, sp_idx, sp_val,
815
- chunk_elems: int = 500_000_000, bias=None):
816
- super().__init__(cores, osplit, isplit, chunk_elems, bias)
817
- if sp_idx.numel() != sp_val.numel():
818
- raise ValueError(f"{sp_idx.numel()} indices for {sp_val.numel()} values")
819
- n = self.out_features * self.in_features
820
- if sp_idx.numel() and (int(sp_idx.max()) >= n or int(sp_idx.min()) < 0):
821
- raise ValueError("sparse index out of range for this layer")
822
- self.register_buffer("sp_idx", sp_idx.to(torch.int64))
823
- self.sp_val = torch.nn.Parameter(sp_val)
824
-
825
- def weight(self) -> torch.Tensor:
826
- # must check the cache here too, not just in TRLinear.weight(): materialize_() caches
827
- # THIS class's weight() (ring + sparse, since self.weight() dispatches polymorphically
828
- # from inside materialize_()), so without this check a later call would go
829
- # super().weight() -> see the cache -> return the already-sparse-augmented tensor ->
830
- # and then add the sparse residual a second time on top of it.
831
- if self._cached_weight is not None:
832
- return self._cached_weight
833
- W = super().weight()
834
- if self.sp_idx.numel():
835
- W = W.reshape(-1).index_add(0, self.sp_idx, self.sp_val.to(W.dtype)).view_as(W)
836
- return W
837
-
838
- def extra_repr(self) -> str:
839
- return super().extra_repr() + f", nnz={self.sp_idx.numel()}"
840
-
841
-
842
- def layer_self_test() -> None:
843
- torch.manual_seed(0)
844
- osp, isp = (4, 3, 2), (2, 5, 3)
845
- dims = [o * i for o, i in zip(osp, isp)]
846
- cores = [torch.randn(3, dims[0], 4), torch.randn(4, dims[1], 2), torch.randn(2, dims[2], 3)]
847
- lay = TRLinear([c.clone() for c in cores], osp, isp, chunk_elems=10 ** 9)
848
- ref = inv_rearrange_n(contract_tr(cores), osp, isp)
849
- assert torch.allclose(lay.weight(), ref, atol=1e-5), "TRLinear.weight() != contract+inverse"
850
- # chunking must not change the answer
851
- lay_c = TRLinear([c.clone() for c in cores], osp, isp, chunk_elems=1)
852
- assert lay_c._chunk < dims[0], "chunk did not engage"
853
- assert torch.allclose(lay_c.weight(), ref, atol=1e-5), "chunked forward differs"
854
- x = torch.randn(7, lay.in_features)
855
- assert torch.allclose(lay(x), x @ ref.T, atol=1e-4), "forward != x W^T"
856
- idx = torch.tensor([0, 5, lay.out_features * lay.in_features - 1], dtype=torch.int64)
857
- val = torch.tensor([1.0, -2.0, 3.0])
858
- sl = SparseAugTRLinear([c.clone() for c in cores], osp, isp, idx, val, chunk_elems=10 ** 9)
859
- d = (sl.weight() - ref).reshape(-1)
860
- assert torch.allclose(d[idx], val, atol=1e-5), "sparse residual not applied at sp_idx"
861
- assert abs(float(d.abs().sum()) - float(val.abs().sum())) < 1e-4, "residual leaked"
862
- try:
863
- TRLinear([torch.zeros(2, 4, 2), torch.zeros(2, 3, 2), torch.zeros(2, 6, 2)],
864
- (4, 1, 2), (1, 3, 3))
865
- except ValueError:
866
- pass
867
- else:
868
- raise AssertionError("a degenerate split (mode = 1) was accepted")
869
- # gradients must reach every core
870
- lay.zero_grad()
871
- lay(x).sum().backward()
872
- for k, c in enumerate(lay.cores):
873
- assert c.grad is not None and float(c.grad.abs().sum()) > 0, f"core {k} got no gradient"
874
- # materialize_() must be a pure inference-speed cache: same weight, same forward output,
875
- # and -- the one bug this is actually guarding against -- the sparse residual must not
876
- # get added twice once SparseAugTRLinear.weight() is short-circuited by the cache.
877
- pre = lay.weight().clone()
878
- lay.materialize_()
879
- assert lay._cached_weight is not None, "materialize_() did not populate the cache"
880
- assert torch.allclose(lay.weight(), pre, atol=1e-6), "materialize_() changed weight()"
881
- assert torch.allclose(lay(x), x @ ref.T, atol=1e-4), "forward != x W^T after materialize_()"
882
- lay.unmaterialize_()
883
- assert lay._cached_weight is None, "unmaterialize_() left the cache populated"
884
- sl_ref = sl.weight().clone()
885
- sl.materialize_()
886
- assert torch.allclose(sl.weight(), sl_ref, atol=1e-6), \
887
- "materialized SparseAugTRLinear.weight() differs -- sparse residual likely double-added"
888
- print("TRLinear self-test PASSED (weight matches contraction, chunking exact, forward "
889
- "correct, sparse localised, degenerate split refused, all cores differentiable, "
890
- "materialize_() cache exact and idempotent)")