Hugging Face's logo

Tensor-compression
/
TR_main

tensor-decomposition
svd
low-rank
model-compression
knowledge-distillation
Model card Files
xet
 Community
Rhett1 commited on
Commit
6591f90
·
verified ·
1 Parent(s): a0d33a3

Add TT-3 topology control on LLaMA-2-7B base (unhealed, matched to tr3/heal/llama2-7b-base)

Browse files
Files changed (6) hide show
  1. tt3/unheal/llama2-7b-base/README.md +86 -0
  2. tt3/unheal/llama2-7b-base/config.json +25 -0
  3. tt3/unheal/llama2-7b-base/provenance.json +2720 -0
  4. tt3/unheal/llama2-7b-base/ranks.json +2690 -0
  5. tt3/unheal/llama2-7b-base/state_dict_fp16.pt +3 -0
  6. tt3/unheal/llama2-7b-base/tensor_formats.py +768 -0
tt3/unheal/llama2-7b-base/README.md ADDED
@@ -0,0 +1,86 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: llama2
3
+ base_model: NousResearch/Llama-2-7b-hf
4
+ tags:
5
+ - tensor-decomposition
6
+ - tensor-train
7
+ - model-compression
8
+ - llama2
9
+ ---
10
+
11
+ # TT-3 control: LLaMA-2-7B (base) @ 4.02B — topology control, **not healed**
12
+
13
+ A 3-site **Tensor Train** (open chain, boundary bond pinned to `r_0 = 1`) at the same
14
+ per-layer parameter budget as the Tensor Ring in [`tr3/heal/llama2-7b-base/`](../../heal/llama2-7b-base).
15
+ It is here to answer one question: at equal parameters, does closing the ring help?
16
+
17
+ ## ⚠️ Read before comparing
18
+
19
+ - **This checkpoint has NOT been healed.** It is the compression-stage output only, with no
20
+ knowledge distillation. Don't put its scores in a table next to healed checkpoints or the
21
+ dense model.
22
+ - It is a **control arm** for the ring-vs-train comparison. It is not meant to be deployed.
23
+
24
+ ## What is matched
25
+
26
+ Matched to `tr3/heal/llama2-7b-base/` layer by layer, not just in total. The per-layer core
27
+ budget is copied from that run's `ranks.json`, and so are the mode splits, pairing,
28
+ activation weighting (separable diagonal Gram), calibration data (256 × 1024 tokens), sparse
29
+ density, and seed (1234). Only the boundary bond is different.
30
+
31
+ | | Ring (TR-3) | Train (here) |
32
+ |---|---|---|
33
+ | Params | 4.02B | **4.0207B** (4,020,677,184) |
34
+ | Reduction | ~40% | **40.33%** |
35
+ | q/k/v/o ranks | (108, 108, 108) | **(1, 186, 186)** |
36
+ | gate/up/down ranks | (141, 141, 141) | **(1, 256, 359)** |
37
+ | Held-out token-mean KL vs dense, before healing | 0.6714 | **6.6665** |
38
+
39
+ The two TT bonds are deliberately unequal. With `r_0 = 1` they have different upper limits
40
+ (`r_1 ≤ d_0`, `r_2 ≤ d_2`), so forcing `r_1 = r_2` would leave part of the budget unusable.
41
+
42
+ ## ALS coverage (confound, stated explicitly)
43
+
44
+ **ALS was skipped on 144 of the 224 layers** (all 96 MLP projections and 48 attention
45
+ projections). The ALS normal equations have size `(r_k · r_{k+1})²`, and with only two bonds
46
+ carrying the budget they don't fit in GPU memory. Those layers keep the TT-SVD solution.
47
+
48
+ This checkpoint's own per-layer data (`provenance.json`) shows the effect is small:
49
+
50
+ | Projection | median whitened rel. err, ALS run | ALS skipped |
51
+ |---|---:|---:|
52
+ | q_proj | 0.6407 (n=20) | 0.6293 (n=12) |
53
+ | k_proj | 0.6382 (n=20) | 0.6282 (n=12) |
54
+ | v_proj | 0.6476 (n=20) | 0.6392 (n=12) |
55
+ | o_proj | 0.5940 (n=20) | 0.6012 (n=12) |
56
+ | gate/up/down | — | ≈0.657 (all skipped) |
57
+
58
+ Where ALS did run, it never improved on the initialisation: winners are `tt_init` 144,
59
+ `tt_padded` 80, ALS 0. So every layer effectively holds the TT-SVD-initialised fit, and the gap
60
+ between ALS and non-ALS layers (~0.01) is much smaller than the ring-vs-train KL gap
61
+ (6.67 vs 0.67).
62
+
63
+ ## Files
64
+
65
+ - `state_dict_fp16.pt`: compressed weights (sha256 `35a1d104ca987db2ba84ad1745bbf808b6fde537e56492abb82814e0d03b4631`)
66
+ - `ranks.json`: per-layer structure specs
67
+ - `tensor_formats.py`: `TRLinear` / `SparseAugTRLinear` (a train is stored as a ring with a size-1 closing bond)
68
+ - `config.json`: base LLaMA-2-7B config
69
+ - `provenance.json`: per-layer ranks, reconstruction errors, ALS-skip flags, init winners, KL, timing (7.50 h compression)
70
+
71
+ ## Loading
72
+
73
+ ```python
74
+ import json, torch
75
+ from transformers import AutoModelForCausalLM
76
+ from tensor_formats import TRLinear, SparseAugTRLinear # shipped in this directory
77
+
78
+ model = AutoModelForCausalLM.from_pretrained(
79
+ "NousResearch/Llama-2-7b-hf",
80
+ revision="8efe6c9b93655b934e27bd9981e3ec13e55aee9d", torch_dtype=torch.float16)
81
+ ranks = json.load(open("ranks.json"))
82
+ # Replace each named projection with TRLinear / SparseAugTRLinear built from its spec,
83
+ # then load state_dict_fp16.pt with strict=False and assert 0 missing / 0 unexpected.
84
+ # Call materialize_() on every tensor layer before evaluation; otherwise each forward
85
+ # re-contracts the cores and generation is ~100x slower.
86
+ ```
tt3/unheal/llama2-7b-base/config.json ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "LlamaForCausalLM"
4
+ ],
5
+ "bos_token_id": 1,
6
+ "eos_token_id": 2,
7
+ "hidden_act": "silu",
8
+ "hidden_size": 4096,
9
+ "initializer_range": 0.02,
10
+ "intermediate_size": 11008,
11
+ "max_position_embeddings": 4096,
12
+ "model_type": "llama",
13
+ "num_attention_heads": 32,
14
+ "num_hidden_layers": 32,
15
+ "num_key_value_heads": 32,
16
+ "pad_token_id": 0,
17
+ "pretraining_tp": 1,
18
+ "rms_norm_eps": 1e-05,
19
+ "rope_scaling": null,
20
+ "tie_word_embeddings": false,
21
+ "torch_dtype": "float16",
22
+ "transformers_version": "4.31.0.dev0",
23
+ "use_cache": true,
24
+ "vocab_size": 32000
25
+ }
tt3/unheal/llama2-7b-base/provenance.json ADDED
@@ -0,0 +1,2720 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "kind": "compress_generic",
3
+ "structure": "train",
4
+ "model_id": "NousResearch/Llama-2-7b-hf",
5
+ "model_revision": "8efe6c9b93655b934e27bd9981e3ec13e55aee9d",
6
+ "ranks_template": "tr3/heal/llama2-7b-base/ranks.json (this repository)",
7
+ "budget_frac_requested": null,
8
+ "held_out_token_mean_kl_nats": 6.6664804220199585,
9
+ "kl_note": "measured on calibration blocks the fit never saw, against the UNTOUCHED teacher, before any healing. This project selects a compression rate by this number rather than by the nominal percentage, because the uncompressible embedding share differs sharply across models (3.9% of LLaMA-2-7B, 21.3% of LLaMA-3.2-1B) and the same rate therefore inflicts very different damage.",
10
+ "budget_note": "per-layer core budget copied EXACTLY from the template's own core count, and the same sparse nnz, so the arms are matched layer by layer rather than only in aggregate. The matrix arm has no sparse-augmented layer type here, so its nnz share is spent on extra rank instead.",
11
+ "index_storage_caveat": "matched on PARAMETER count. sp_idx is a buffer, not a Parameter, so the tensor arms additionally store nnz int64 indices that this count excludes. In BYTES the tensor arms are therefore larger, and any size claim must say which of the two conventions it uses.",
12
+ "dense_total": 6738415616,
13
+ "achieved_total": 4020677184,
14
+ "reduction": 0.4033,
15
+ "n_layers": 224,
16
+ "n_biases_carried": 0,
17
+ "n_als_skipped_for_memory": 144,
18
+ "als_note": "a layer whose (r_k r_{k+1})^2 normal equations do not fit is fitted by TT-SVD alone. Layer-level sweeps in this project bound what that costs a train at 0.1-0.5%; a RING loses far more without ALS, so this fallback is not symmetric and must be reported per arm.",
19
+ "calib_blocks": 256,
20
+ "seq_len": 1024,
21
+ "lam_ratio": 0.0001,
22
+ "seed": 1234,
23
+ "tr_sweeps": 20,
24
+ "tr_restarts": 1,
25
+ "checkpoint": "state_dict_fp16.pt",
26
+ "sha256": "35a1d104ca987db2ba84ad1745bbf808b6fde537e56492abb82814e0d03b4631",
27
+ "elapsed_s": 27004.2,
28
+ "timestamp": "2026-09-11T10:00:42.839284+00:00",
29
+ "per_layer": {
30
+ "model.layers.0.self_attn.q_proj": {
31
+ "ranks": [
32
+ 1,
33
+ 186,
34
+ 186
35
+ ],
36
+ "params": 8951808,
37
+ "rel_err_whitened": 0.509049,
38
+ "als_skipped_for_memory": true,
39
+ "winner": "tt_init",
40
+ "sep_rel": 0.999929
41
+ },
42
+ "model.layers.0.self_attn.k_proj": {
43
+ "ranks": [
44
+ 1,
45
+ 186,
46
+ 186
47
+ ],
48
+ "params": 8951808,
49
+ "rel_err_whitened": 0.555365,
50
+ "als_skipped_for_memory": true,
51
+ "winner": "tt_init",
52
+ "sep_rel": 0.999929
53
+ },
54
+ "model.layers.0.self_attn.v_proj": {
55
+ "ranks": [
56
+ 1,
57
+ 186,
58
+ 186
59
+ ],
60
+ "params": 8951808,
61
+ "rel_err_whitened": 0.586322,
62
+ "als_skipped_for_memory": true,
63
+ "winner": "tt_init",
64
+ "sep_rel": 0.999929
65
+ },
66
+ "model.layers.0.self_attn.o_proj": {
67
+ "ranks": [
68
+ 1,
69
+ 186,
70
+ 186
71
+ ],
72
+ "params": 8951808,
73
+ "rel_err_whitened": 0.484311,
74
+ "als_skipped_for_memory": true,
75
+ "winner": "tt_init",
76
+ "sep_rel": 0.999285
77
+ },
78
+ "model.layers.0.mlp.gate_proj": {
79
+ "ranks": [
80
+ 1,
81
+ 256,
82
+ 359
83
+ ],
84
+ "params": 23839952,
85
+ "rel_err_whitened": 0.656168,
86
+ "als_skipped_for_memory": true,
87
+ "winner": "tt_init",
88
+ "sep_rel": 0.992869
89
+ },
90
+ "model.layers.0.mlp.up_proj": {
91
+ "ranks": [
92
+ 1,
93
+ 256,
94
+ 359
95
+ ],
96
+ "params": 23839952,
97
+ "rel_err_whitened": 0.657772,
98
+ "als_skipped_for_memory": true,
99
+ "winner": "tt_init",
100
+ "sep_rel": 0.992869
101
+ },
102
+ "model.layers.0.mlp.down_proj": {
103
+ "ranks": [
104
+ 1,
105
+ 256,
106
+ 359
107
+ ],
108
+ "params": 23839952,
109
+ "rel_err_whitened": 0.654422,
110
+ "als_skipped_for_memory": true,
111
+ "winner": "tt_init",
112
+ "sep_rel": 0.993898
113
+ },
114
+ "model.layers.1.self_attn.q_proj": {
115
+ "ranks": [
116
+ 1,
117
+ 186,
118
+ 186
119
+ ],
120
+ "params": 8951808,
121
+ "rel_err_whitened": 0.497915,
122
+ "als_skipped_for_memory": true,
123
+ "winner": "tt_init",
124
+ "sep_rel": 0.99995
125
+ },
126
+ "model.layers.1.self_attn.k_proj": {
127
+ "ranks": [
128
+ 1,
129
+ 186,
130
+ 186
131
+ ],
132
+ "params": 8951808,
133
+ "rel_err_whitened": 0.51476,
134
+ "als_skipped_for_memory": true,
135
+ "winner": "tt_init",
136
+ "sep_rel": 0.99995
137
+ },
138
+ "model.layers.1.self_attn.v_proj": {
139
+ "ranks": [
140
+ 1,
141
+ 186,
142
+ 186
143
+ ],
144
+ "params": 8951808,
145
+ "rel_err_whitened": 0.586947,
146
+ "als_skipped_for_memory": true,
147
+ "winner": "tt_init",
148
+ "sep_rel": 0.99995
149
+ },
150
+ "model.layers.1.self_attn.o_proj": {
151
+ "ranks": [
152
+ 1,
153
+ 186,
154
+ 186
155
+ ],
156
+ "params": 8951808,
157
+ "rel_err_whitened": 0.483951,
158
+ "als_skipped_for_memory": true,
159
+ "winner": "tt_init",
160
+ "sep_rel": 0.996338
161
+ },
162
+ "model.layers.1.mlp.gate_proj": {
163
+ "ranks": [
164
+ 1,
165
+ 256,
166
+ 359
167
+ ],
168
+ "params": 23839952,
169
+ "rel_err_whitened": 0.65541,
170
+ "als_skipped_for_memory": true,
171
+ "winner": "tt_init",
172
+ "sep_rel": 0.891198
173
+ },
174
+ "model.layers.1.mlp.up_proj": {
175
+ "ranks": [
176
+ 1,
177
+ 256,
178
+ 359
179
+ ],
180
+ "params": 23839952,
181
+ "rel_err_whitened": 0.658732,
182
+ "als_skipped_for_memory": true,
183
+ "winner": "tt_init",
184
+ "sep_rel": 0.891198
185
+ },
186
+ "model.layers.1.mlp.down_proj": {
187
+ "ranks": [
188
+ 1,
189
+ 256,
190
+ 359
191
+ ],
192
+ "params": 23839952,
193
+ "rel_err_whitened": 0.656533,
194
+ "als_skipped_for_memory": true,
195
+ "winner": "tt_init",
196
+ "sep_rel": 1.0
197
+ },
198
+ "model.layers.2.self_attn.q_proj": {
199
+ "ranks": [
200
+ 1,
201
+ 186,
202
+ 186
203
+ ],
204
+ "params": 8951808,
205
+ "rel_err_whitened": 0.611998,
206
+ "als_skipped_for_memory": true,
207
+ "winner": "tt_init",
208
+ "sep_rel": 0.996711
209
+ },
210
+ "model.layers.2.self_attn.k_proj": {
211
+ "ranks": [
212
+ 1,
213
+ 186,
214
+ 186
215
+ ],
216
+ "params": 8951808,
217
+ "rel_err_whitened": 0.607766,
218
+ "als_skipped_for_memory": true,
219
+ "winner": "tt_init",
220
+ "sep_rel": 0.996711
221
+ },
222
+ "model.layers.2.self_attn.v_proj": {
223
+ "ranks": [
224
+ 1,
225
+ 186,
226
+ 186
227
+ ],
228
+ "params": 8951808,
229
+ "rel_err_whitened": 0.64418,
230
+ "als_skipped_for_memory": true,
231
+ "winner": "tt_init",
232
+ "sep_rel": 0.996711
233
+ },
234
+ "model.layers.2.self_attn.o_proj": {
235
+ "ranks": [
236
+ 1,
237
+ 186,
238
+ 186
239
+ ],
240
+ "params": 8951808,
241
+ "rel_err_whitened": 0.568673,
242
+ "als_skipped_for_memory": true,
243
+ "winner": "tt_init",
244
+ "sep_rel": 0.809186
245
+ },
246
+ "model.layers.2.mlp.gate_proj": {
247
+ "ranks": [
248
+ 1,
249
+ 256,
250
+ 359
251
+ ],
252
+ "params": 23839952,
253
+ "rel_err_whitened": 0.657715,
254
+ "als_skipped_for_memory": true,
255
+ "winner": "tt_init",
256
+ "sep_rel": 0.351065
257
+ },
258
+ "model.layers.2.mlp.up_proj": {
259
+ "ranks": [
260
+ 1,
261
+ 256,
262
+ 359
263
+ ],
264
+ "params": 23839952,
265
+ "rel_err_whitened": 0.659216,
266
+ "als_skipped_for_memory": true,
267
+ "winner": "tt_init",
268
+ "sep_rel": 0.351065
269
+ },
270
+ "model.layers.2.mlp.down_proj": {
271
+ "ranks": [
272
+ 1,
273
+ 256,
274
+ 359
275
+ ],
276
+ "params": 23839952,
277
+ "rel_err_whitened": 0.657981,
278
+ "als_skipped_for_memory": true,
279
+ "winner": "tt_init",
280
+ "sep_rel": 0.940403
281
+ },
282
+ "model.layers.3.self_attn.q_proj": {
283
+ "ranks": [
284
+ 1,
285
+ 186,
286
+ 186
287
+ ],
288
+ "params": 8951808,
289
+ "rel_err_whitened": 0.622852,
290
+ "als_skipped_for_memory": true,
291
+ "winner": "tt_init",
292
+ "sep_rel": 0.990123
293
+ },
294
+ "model.layers.3.self_attn.k_proj": {
295
+ "ranks": [
296
+ 1,
297
+ 186,
298
+ 186
299
+ ],
300
+ "params": 8951808,
301
+ "rel_err_whitened": 0.619944,
302
+ "als_skipped_for_memory": true,
303
+ "winner": "tt_init",
304
+ "sep_rel": 0.990123
305
+ },
306
+ "model.layers.3.self_attn.v_proj": {
307
+ "ranks": [
308
+ 1,
309
+ 186,
310
+ 186
311
+ ],
312
+ "params": 8951808,
313
+ "rel_err_whitened": 0.63601,
314
+ "als_skipped_for_memory": true,
315
+ "winner": "tt_init",
316
+ "sep_rel": 0.990123
317
+ },
318
+ "model.layers.3.self_attn.o_proj": {
319
+ "ranks": [
320
+ 1,
321
+ 186,
322
+ 186
323
+ ],
324
+ "params": 8951808,
325
+ "rel_err_whitened": 0.600638,
326
+ "als_skipped_for_memory": true,
327
+ "winner": "tt_init",
328
+ "sep_rel": 0.953369
329
+ },
330
+ "model.layers.3.mlp.gate_proj": {
331
+ "ranks": [
332
+ 1,
333
+ 256,
334
+ 359
335
+ ],
336
+ "params": 23839952,
337
+ "rel_err_whitened": 0.658172,
338
+ "als_skipped_for_memory": true,
339
+ "winner": "tt_init",
340
+ "sep_rel": 0.513124
341
+ },
342
+ "model.layers.3.mlp.up_proj": {
343
+ "ranks": [
344
+ 1,
345
+ 256,
346
+ 359
347
+ ],
348
+ "params": 23839952,
349
+ "rel_err_whitened": 0.659195,
350
+ "als_skipped_for_memory": true,
351
+ "winner": "tt_init",
352
+ "sep_rel": 0.513124
353
+ },
354
+ "model.layers.3.mlp.down_proj": {
355
+ "ranks": [
356
+ 1,
357
+ 256,
358
+ 359
359
+ ],
360
+ "params": 23839952,
361
+ "rel_err_whitened": 0.657916,
362
+ "als_skipped_for_memory": true,
363
+ "winner": "tt_init",
364
+ "sep_rel": 0.809824
365
+ },
366
+ "model.layers.4.self_attn.q_proj": {
367
+ "ranks": [
368
+ 1,
369
+ 186,
370
+ 186
371
+ ],
372
+ "params": 8951808,
373
+ "rel_err_whitened": 0.622972,
374
+ "als_skipped_for_memory": true,
375
+ "winner": "tt_init",
376
+ "sep_rel": 0.992652
377
+ },
378
+ "model.layers.4.self_attn.k_proj": {
379
+ "ranks": [
380
+ 1,
381
+ 186,
382
+ 186
383
+ ],
384
+ "params": 8951808,
385
+ "rel_err_whitened": 0.627464,
386
+ "als_skipped_for_memory": true,
387
+ "winner": "tt_init",
388
+ "sep_rel": 0.992652
389
+ },
390
+ "model.layers.4.self_attn.v_proj": {
391
+ "ranks": [
392
+ 1,
393
+ 186,
394
+ 186
395
+ ],
396
+ "params": 8951808,
397
+ "rel_err_whitened": 0.649394,
398
+ "als_skipped_for_memory": true,
399
+ "winner": "tt_init",
400
+ "sep_rel": 0.992652
401
+ },
402
+ "model.layers.4.self_attn.o_proj": {
403
+ "ranks": [
404
+ 1,
405
+ 186,
406
+ 186
407
+ ],
408
+ "params": 8951808,
409
+ "rel_err_whitened": 0.609243,
410
+ "als_skipped_for_memory": true,
411
+ "winner": "tt_init",
412
+ "sep_rel": 0.812771
413
+ },
414
+ "model.layers.4.mlp.gate_proj": {
415
+ "ranks": [
416
+ 1,
417
+ 256,
418
+ 359
419
+ ],
420
+ "params": 23839952,
421
+ "rel_err_whitened": 0.657986,
422
+ "als_skipped_for_memory": true,
423
+ "winner": "tt_init",
424
+ "sep_rel": 0.71968
425
+ },
426
+ "model.layers.4.mlp.up_proj": {
427
+ "ranks": [
428
+ 1,
429
+ 256,
430
+ 359
431
+ ],
432
+ "params": 23839952,
433
+ "rel_err_whitened": 0.659211,
434
+ "als_skipped_for_memory": true,
435
+ "winner": "tt_init",
436
+ "sep_rel": 0.71968
437
+ },
438
+ "model.layers.4.mlp.down_proj": {
439
+ "ranks": [
440
+ 1,
441
+ 256,
442
+ 359
443
+ ],
444
+ "params": 23839952,
445
+ "rel_err_whitened": 0.657798,
446
+ "als_skipped_for_memory": true,
447
+ "winner": "tt_init",
448
+ "sep_rel": 0.953194
449
+ },
450
+ "model.layers.5.self_attn.q_proj": {
451
+ "ranks": [
452
+ 1,
453
+ 186,
454
+ 186
455
+ ],
456
+ "params": 8951808,
457
+ "rel_err_whitened": 0.632296,
458
+ "als_skipped_for_memory": true,
459
+ "winner": "tt_init",
460
+ "sep_rel": 0.991825
461
+ },
462
+ "model.layers.5.self_attn.k_proj": {
463
+ "ranks": [
464
+ 1,
465
+ 186,
466
+ 186
467
+ ],
468
+ "params": 8951808,
469
+ "rel_err_whitened": 0.635392,
470
+ "als_skipped_for_memory": true,
471
+ "winner": "tt_init",
472
+ "sep_rel": 0.991825
473
+ },
474
+ "model.layers.5.self_attn.v_proj": {
475
+ "ranks": [
476
+ 1,
477
+ 186,
478
+ 186
479
+ ],
480
+ "params": 8951808,
481
+ "rel_err_whitened": 0.643663,
482
+ "als_skipped_for_memory": true,
483
+ "winner": "tt_init",
484
+ "sep_rel": 0.991825
485
+ },
486
+ "model.layers.5.self_attn.o_proj": {
487
+ "ranks": [
488
+ 1,
489
+ 186,
490
+ 186
491
+ ],
492
+ "params": 8951808,
493
+ "rel_err_whitened": 0.601811,
494
+ "als_skipped_for_memory": true,
495
+ "winner": "tt_init",
496
+ "sep_rel": 0.944414
497
+ },
498
+ "model.layers.5.mlp.gate_proj": {
499
+ "ranks": [
500
+ 1,
501
+ 256,
502
+ 359
503
+ ],
504
+ "params": 23839952,
505
+ "rel_err_whitened": 0.657479,
506
+ "als_skipped_for_memory": true,
507
+ "winner": "tt_init",
508
+ "sep_rel": 0.68934
509
+ },
510
+ "model.layers.5.mlp.up_proj": {
511
+ "ranks": [
512
+ 1,
513
+ 256,
514
+ 359
515
+ ],
516
+ "params": 23839952,
517
+ "rel_err_whitened": 0.659268,
518
+ "als_skipped_for_memory": true,
519
+ "winner": "tt_init",
520
+ "sep_rel": 0.68934
521
+ },
522
+ "model.layers.5.mlp.down_proj": {
523
+ "ranks": [
524
+ 1,
525
+ 256,
526
+ 359
527
+ ],
528
+ "params": 23839952,
529
+ "rel_err_whitened": 0.657951,
530
+ "als_skipped_for_memory": true,
531
+ "winner": "tt_init",
532
+ "sep_rel": 0.741805
533
+ },
534
+ "model.layers.6.self_attn.q_proj": {
535
+ "ranks": [
536
+ 1,
537
+ 186,
538
+ 186
539
+ ],
540
+ "params": 8951808,
541
+ "rel_err_whitened": 0.630137,
542
+ "als_skipped_for_memory": true,
543
+ "winner": "tt_init",
544
+ "sep_rel": 0.986907
545
+ },
546
+ "model.layers.6.self_attn.k_proj": {
547
+ "ranks": [
548
+ 1,
549
+ 186,
550
+ 186
551
+ ],
552
+ "params": 8951808,
553
+ "rel_err_whitened": 0.632253,
554
+ "als_skipped_for_memory": true,
555
+ "winner": "tt_init",
556
+ "sep_rel": 0.986907
557
+ },
558
+ "model.layers.6.self_attn.v_proj": {
559
+ "ranks": [
560
+ 1,
561
+ 186,
562
+ 186
563
+ ],
564
+ "params": 8951808,
565
+ "rel_err_whitened": 0.637551,
566
+ "als_skipped_for_memory": true,
567
+ "winner": "tt_init",
568
+ "sep_rel": 0.986907
569
+ },
570
+ "model.layers.6.self_attn.o_proj": {
571
+ "ranks": [
572
+ 1,
573
+ 186,
574
+ 186
575
+ ],
576
+ "params": 8951808,
577
+ "rel_err_whitened": 0.607056,
578
+ "als_skipped_for_memory": true,
579
+ "winner": "tt_init",
580
+ "sep_rel": 0.874977
581
+ },
582
+ "model.layers.6.mlp.gate_proj": {
583
+ "ranks": [
584
+ 1,
585
+ 256,
586
+ 359
587
+ ],
588
+ "params": 23839952,
589
+ "rel_err_whitened": 0.656892,
590
+ "als_skipped_for_memory": true,
591
+ "winner": "tt_init",
592
+ "sep_rel": 0.732336
593
+ },
594
+ "model.layers.6.mlp.up_proj": {
595
+ "ranks": [
596
+ 1,
597
+ 256,
598
+ 359
599
+ ],
600
+ "params": 23839952,
601
+ "rel_err_whitened": 0.659319,
602
+ "als_skipped_for_memory": true,
603
+ "winner": "tt_init",
604
+ "sep_rel": 0.732336
605
+ },
606
+ "model.layers.6.mlp.down_proj": {
607
+ "ranks": [
608
+ 1,
609
+ 256,
610
+ 359
611
+ ],
612
+ "params": 23839952,
613
+ "rel_err_whitened": 0.658259,
614
+ "als_skipped_for_memory": true,
615
+ "winner": "tt_init",
616
+ "sep_rel": 0.792984
617
+ },
618
+ "model.layers.7.self_attn.q_proj": {
619
+ "ranks": [
620
+ 1,
621
+ 186,
622
+ 186
623
+ ],
624
+ "params": 8951808,
625
+ "rel_err_whitened": 0.633436,
626
+ "als_skipped_for_memory": true,
627
+ "winner": "tt_init",
628
+ "sep_rel": 0.986292
629
+ },
630
+ "model.layers.7.self_attn.k_proj": {
631
+ "ranks": [
632
+ 1,
633
+ 186,
634
+ 186
635
+ ],
636
+ "params": 8951808,
637
+ "rel_err_whitened": 0.635891,
638
+ "als_skipped_for_memory": true,
639
+ "winner": "tt_init",
640
+ "sep_rel": 0.986292
641
+ },
642
+ "model.layers.7.self_attn.v_proj": {
643
+ "ranks": [
644
+ 1,
645
+ 186,
646
+ 186
647
+ ],
648
+ "params": 8951808,
649
+ "rel_err_whitened": 0.618615,
650
+ "als_skipped_for_memory": true,
651
+ "winner": "tt_init",
652
+ "sep_rel": 0.986292
653
+ },
654
+ "model.layers.7.self_attn.o_proj": {
655
+ "ranks": [
656
+ 1,
657
+ 186,
658
+ 186
659
+ ],
660
+ "params": 8951808,
661
+ "rel_err_whitened": 0.581512,
662
+ "als_skipped_for_memory": true,
663
+ "winner": "tt_init",
664
+ "sep_rel": 0.839965
665
+ },
666
+ "model.layers.7.mlp.gate_proj": {
667
+ "ranks": [
668
+ 1,
669
+ 256,
670
+ 359
671
+ ],
672
+ "params": 23839952,
673
+ "rel_err_whitened": 0.657174,
674
+ "als_skipped_for_memory": true,
675
+ "winner": "tt_init",
676
+ "sep_rel": 0.818209
677
+ },
678
+ "model.layers.7.mlp.up_proj": {
679
+ "ranks": [
680
+ 1,
681
+ 256,
682
+ 359
683
+ ],
684
+ "params": 23839952,
685
+ "rel_err_whitened": 0.659217,
686
+ "als_skipped_for_memory": true,
687
+ "winner": "tt_init",
688
+ "sep_rel": 0.818209
689
+ },
690
+ "model.layers.7.mlp.down_proj": {
691
+ "ranks": [
692
+ 1,
693
+ 256,
694
+ 359
695
+ ],
696
+ "params": 23839952,
697
+ "rel_err_whitened": 0.65858,
698
+ "als_skipped_for_memory": true,
699
+ "winner": "tt_init",
700
+ "sep_rel": 0.904835
701
+ },
702
+ "model.layers.8.self_attn.q_proj": {
703
+ "ranks": [
704
+ 1,
705
+ 186,
706
+ 186
707
+ ],
708
+ "params": 8951808,
709
+ "rel_err_whitened": 0.630166,
710
+ "als_skipped_for_memory": true,
711
+ "winner": "tt_init",
712
+ "sep_rel": 0.989787
713
+ },
714
+ "model.layers.8.self_attn.k_proj": {
715
+ "ranks": [
716
+ 1,
717
+ 186,
718
+ 186
719
+ ],
720
+ "params": 8951808,
721
+ "rel_err_whitened": 0.634998,
722
+ "als_skipped_for_memory": true,
723
+ "winner": "tt_init",
724
+ "sep_rel": 0.989787
725
+ },
726
+ "model.layers.8.self_attn.v_proj": {
727
+ "ranks": [
728
+ 1,
729
+ 186,
730
+ 186
731
+ ],
732
+ "params": 8951808,
733
+ "rel_err_whitened": 0.645075,
734
+ "als_skipped_for_memory": true,
735
+ "winner": "tt_init",
736
+ "sep_rel": 0.989787
737
+ },
738
+ "model.layers.8.self_attn.o_proj": {
739
+ "ranks": [
740
+ 1,
741
+ 186,
742
+ 186
743
+ ],
744
+ "params": 8951808,
745
+ "rel_err_whitened": 0.604005,
746
+ "als_skipped_for_memory": true,
747
+ "winner": "tt_init",
748
+ "sep_rel": 0.828433
749
+ },
750
+ "model.layers.8.mlp.gate_proj": {
751
+ "ranks": [
752
+ 1,
753
+ 256,
754
+ 359
755
+ ],
756
+ "params": 23839952,
757
+ "rel_err_whitened": 0.657395,
758
+ "als_skipped_for_memory": true,
759
+ "winner": "tt_init",
760
+ "sep_rel": 0.852406
761
+ },
762
+ "model.layers.8.mlp.up_proj": {
763
+ "ranks": [
764
+ 1,
765
+ 256,
766
+ 359
767
+ ],
768
+ "params": 23839952,
769
+ "rel_err_whitened": 0.659181,
770
+ "als_skipped_for_memory": true,
771
+ "winner": "tt_init",
772
+ "sep_rel": 0.852406
773
+ },
774
+ "model.layers.8.mlp.down_proj": {
775
+ "ranks": [
776
+ 1,
777
+ 256,
778
+ 359
779
+ ],
780
+ "params": 23839952,
781
+ "rel_err_whitened": 0.658462,
782
+ "als_skipped_for_memory": true,
783
+ "winner": "tt_init",
784
+ "sep_rel": 0.920778
785
+ },
786
+ "model.layers.9.self_attn.q_proj": {
787
+ "ranks": [
788
+ 1,
789
+ 186,
790
+ 186
791
+ ],
792
+ "params": 8951808,
793
+ "rel_err_whitened": 0.63587,
794
+ "als_skipped_for_memory": true,
795
+ "winner": "tt_init",
796
+ "sep_rel": 0.988714
797
+ },
798
+ "model.layers.9.self_attn.k_proj": {
799
+ "ranks": [
800
+ 1,
801
+ 186,
802
+ 186
803
+ ],
804
+ "params": 8951808,
805
+ "rel_err_whitened": 0.634848,
806
+ "als_skipped_for_memory": true,
807
+ "winner": "tt_init",
808
+ "sep_rel": 0.988714
809
+ },
810
+ "model.layers.9.self_attn.v_proj": {
811
+ "ranks": [
812
+ 1,
813
+ 186,
814
+ 186
815
+ ],
816
+ "params": 8951808,
817
+ "rel_err_whitened": 0.640768,
818
+ "als_skipped_for_memory": true,
819
+ "winner": "tt_init",
820
+ "sep_rel": 0.988714
821
+ },
822
+ "model.layers.9.self_attn.o_proj": {
823
+ "ranks": [
824
+ 1,
825
+ 186,
826
+ 186
827
+ ],
828
+ "params": 8951808,
829
+ "rel_err_whitened": 0.584625,
830
+ "als_skipped_for_memory": true,
831
+ "winner": "tt_init",
832
+ "sep_rel": 0.832795
833
+ },
834
+ "model.layers.9.mlp.gate_proj": {
835
+ "ranks": [
836
+ 1,
837
+ 256,
838
+ 359
839
+ ],
840
+ "params": 23839952,
841
+ "rel_err_whitened": 0.657139,
842
+ "als_skipped_for_memory": true,
843
+ "winner": "tt_init",
844
+ "sep_rel": 0.841161
845
+ },
846
+ "model.layers.9.mlp.up_proj": {
847
+ "ranks": [
848
+ 1,
849
+ 256,
850
+ 359
851
+ ],
852
+ "params": 23839952,
853
+ "rel_err_whitened": 0.659214,
854
+ "als_skipped_for_memory": true,
855
+ "winner": "tt_init",
856
+ "sep_rel": 0.841161
857
+ },
858
+ "model.layers.9.mlp.down_proj": {
859
+ "ranks": [
860
+ 1,
861
+ 256,
862
+ 359
863
+ ],
864
+ "params": 23839952,
865
+ "rel_err_whitened": 0.658085,
866
+ "als_skipped_for_memory": true,
867
+ "winner": "tt_init",
868
+ "sep_rel": 0.700596
869
+ },
870
+ "model.layers.10.self_attn.q_proj": {
871
+ "ranks": [
872
+ 1,
873
+ 186,
874
+ 186
875
+ ],
876
+ "params": 8951808,
877
+ "rel_err_whitened": 0.633837,
878
+ "als_skipped_for_memory": true,
879
+ "winner": "tt_init",
880
+ "sep_rel": 0.989456
881
+ },
882
+ "model.layers.10.self_attn.k_proj": {
883
+ "ranks": [
884
+ 1,
885
+ 186,
886
+ 186
887
+ ],
888
+ "params": 8951808,
889
+ "rel_err_whitened": 0.627361,
890
+ "als_skipped_for_memory": true,
891
+ "winner": "tt_init",
892
+ "sep_rel": 0.989456
893
+ },
894
+ "model.layers.10.self_attn.v_proj": {
895
+ "ranks": [
896
+ 1,
897
+ 186,
898
+ 186
899
+ ],
900
+ "params": 8951808,
901
+ "rel_err_whitened": 0.646535,
902
+ "als_skipped_for_memory": true,
903
+ "winner": "tt_init",
904
+ "sep_rel": 0.989456
905
+ },
906
+ "model.layers.10.self_attn.o_proj": {
907
+ "ranks": [
908
+ 1,
909
+ 186,
910
+ 186
911
+ ],
912
+ "params": 8951808,
913
+ "rel_err_whitened": 0.617943,
914
+ "als_skipped_for_memory": true,
915
+ "winner": "tt_init",
916
+ "sep_rel": 0.715224
917
+ },
918
+ "model.layers.10.mlp.gate_proj": {
919
+ "ranks": [
920
+ 1,
921
+ 256,
922
+ 359
923
+ ],
924
+ "params": 23839952,
925
+ "rel_err_whitened": 0.656817,
926
+ "als_skipped_for_memory": true,
927
+ "winner": "tt_init",
928
+ "sep_rel": 0.850181
929
+ },
930
+ "model.layers.10.mlp.up_proj": {
931
+ "ranks": [
932
+ 1,
933
+ 256,
934
+ 359
935
+ ],
936
+ "params": 23839952,
937
+ "rel_err_whitened": 0.659094,
938
+ "als_skipped_for_memory": true,
939
+ "winner": "tt_init",
940
+ "sep_rel": 0.850181
941
+ },
942
+ "model.layers.10.mlp.down_proj": {
943
+ "ranks": [
944
+ 1,
945
+ 256,
946
+ 359
947
+ ],
948
+ "params": 23839952,
949
+ "rel_err_whitened": 0.658406,
950
+ "als_skipped_for_memory": true,
951
+ "winner": "tt_init",
952
+ "sep_rel": 0.625283
953
+ },
954
+ "model.layers.11.self_attn.q_proj": {
955
+ "ranks": [
956
+ 1,
957
+ 186,
958
+ 186
959
+ ],
960
+ "params": 8951808,
961
+ "rel_err_whitened": 0.62853,
962
+ "als_skipped_for_memory": true,
963
+ "winner": "tt_init",
964
+ "sep_rel": 0.987078
965
+ },
966
+ "model.layers.11.self_attn.k_proj": {
967
+ "ranks": [
968
+ 1,
969
+ 186,
970
+ 186
971
+ ],
972
+ "params": 8951808,
973
+ "rel_err_whitened": 0.628989,
974
+ "als_skipped_for_memory": true,
975
+ "winner": "tt_init",
976
+ "sep_rel": 0.987078
977
+ },
978
+ "model.layers.11.self_attn.v_proj": {
979
+ "ranks": [
980
+ 1,
981
+ 186,
982
+ 186
983
+ ],
984
+ "params": 8951808,
985
+ "rel_err_whitened": 0.632762,
986
+ "als_skipped_for_memory": true,
987
+ "winner": "tt_init",
988
+ "sep_rel": 0.987078
989
+ },
990
+ "model.layers.11.self_attn.o_proj": {
991
+ "ranks": [
992
+ 1,
993
+ 186,
994
+ 186
995
+ ],
996
+ "params": 8951808,
997
+ "rel_err_whitened": 0.620103,
998
+ "als_skipped_for_memory": true,
999
+ "winner": "tt_init",
1000
+ "sep_rel": 0.729097
1001
+ },
1002
+ "model.layers.11.mlp.gate_proj": {
1003
+ "ranks": [
1004
+ 1,
1005
+ 256,
1006
+ 359
1007
+ ],
1008
+ "params": 23839952,
1009
+ "rel_err_whitened": 0.657115,
1010
+ "als_skipped_for_memory": true,
1011
+ "winner": "tt_init",
1012
+ "sep_rel": 0.828439
1013
+ },
1014
+ "model.layers.11.mlp.up_proj": {
1015
+ "ranks": [
1016
+ 1,
1017
+ 256,
1018
+ 359
1019
+ ],
1020
+ "params": 23839952,
1021
+ "rel_err_whitened": 0.659215,
1022
+ "als_skipped_for_memory": true,
1023
+ "winner": "tt_init",
1024
+ "sep_rel": 0.828439
1025
+ },
1026
+ "model.layers.11.mlp.down_proj": {
1027
+ "ranks": [
1028
+ 1,
1029
+ 256,
1030
+ 359
1031
+ ],
1032
+ "params": 23839952,
1033
+ "rel_err_whitened": 0.658096,
1034
+ "als_skipped_for_memory": true,
1035
+ "winner": "tt_init",
1036
+ "sep_rel": 0.967598
1037
+ },
1038
+ "model.layers.12.self_attn.q_proj": {
1039
+ "ranks": [
1040
+ 1,
1041
+ 186,
1042
+ 186
1043
+ ],
1044
+ "params": 8951808,
1045
+ "rel_err_whitened": 0.640928,
1046
+ "als_skipped_for_memory": false,
1047
+ "winner": "tt_padded",
1048
+ "sep_rel": 0.986894
1049
+ },
1050
+ "model.layers.12.self_attn.k_proj": {
1051
+ "ranks": [
1052
+ 1,
1053
+ 186,
1054
+ 186
1055
+ ],
1056
+ "params": 8951808,
1057
+ "rel_err_whitened": 0.64144,
1058
+ "als_skipped_for_memory": false,
1059
+ "winner": "tt_padded",
1060
+ "sep_rel": 0.986894
1061
+ },
1062
+ "model.layers.12.self_attn.v_proj": {
1063
+ "ranks": [
1064
+ 1,
1065
+ 186,
1066
+ 186
1067
+ ],
1068
+ "params": 8951808,
1069
+ "rel_err_whitened": 0.638423,
1070
+ "als_skipped_for_memory": false,
1071
+ "winner": "tt_padded",
1072
+ "sep_rel": 0.986894
1073
+ },
1074
+ "model.layers.12.self_attn.o_proj": {
1075
+ "ranks": [
1076
+ 1,
1077
+ 186,
1078
+ 186
1079
+ ],
1080
+ "params": 8951808,
1081
+ "rel_err_whitened": 0.624372,
1082
+ "als_skipped_for_memory": false,
1083
+ "winner": "tt_padded",
1084
+ "sep_rel": 0.741835
1085
+ },
1086
+ "model.layers.12.mlp.gate_proj": {
1087
+ "ranks": [
1088
+ 1,
1089
+ 256,
1090
+ 359
1091
+ ],
1092
+ "params": 23839952,
1093
+ "rel_err_whitened": 0.657049,
1094
+ "als_skipped_for_memory": true,
1095
+ "winner": "tt_init",
1096
+ "sep_rel": 0.809401
1097
+ },
1098
+ "model.layers.12.mlp.up_proj": {
1099
+ "ranks": [
1100
+ 1,
1101
+ 256,
1102
+ 359
1103
+ ],
1104
+ "params": 23839952,
1105
+ "rel_err_whitened": 0.659193,
1106
+ "als_skipped_for_memory": true,
1107
+ "winner": "tt_init",
1108
+ "sep_rel": 0.809401
1109
+ },
1110
+ "model.layers.12.mlp.down_proj": {
1111
+ "ranks": [
1112
+ 1,
1113
+ 256,
1114
+ 359
1115
+ ],
1116
+ "params": 23839952,
1117
+ "rel_err_whitened": 0.657815,
1118
+ "als_skipped_for_memory": true,
1119
+ "winner": "tt_init",
1120
+ "sep_rel": 0.694015
1121
+ },
1122
+ "model.layers.13.self_attn.q_proj": {
1123
+ "ranks": [
1124
+ 1,
1125
+ 186,
1126
+ 186
1127
+ ],
1128
+ "params": 8951808,
1129
+ "rel_err_whitened": 0.640029,
1130
+ "als_skipped_for_memory": false,
1131
+ "winner": "tt_padded",
1132
+ "sep_rel": 0.982877
1133
+ },
1134
+ "model.layers.13.self_attn.k_proj": {
1135
+ "ranks": [
1136
+ 1,
1137
+ 186,
1138
+ 186
1139
+ ],
1140
+ "params": 8951808,
1141
+ "rel_err_whitened": 0.636503,
1142
+ "als_skipped_for_memory": false,
1143
+ "winner": "tt_padded",
1144
+ "sep_rel": 0.982877
1145
+ },
1146
+ "model.layers.13.self_attn.v_proj": {
1147
+ "ranks": [
1148
+ 1,
1149
+ 186,
1150
+ 186
1151
+ ],
1152
+ "params": 8951808,
1153
+ "rel_err_whitened": 0.643255,
1154
+ "als_skipped_for_memory": false,
1155
+ "winner": "tt_padded",
1156
+ "sep_rel": 0.982877
1157
+ },
1158
+ "model.layers.13.self_attn.o_proj": {
1159
+ "ranks": [
1160
+ 1,
1161
+ 186,
1162
+ 186
1163
+ ],
1164
+ "params": 8951808,
1165
+ "rel_err_whitened": 0.614544,
1166
+ "als_skipped_for_memory": false,
1167
+ "winner": "tt_padded",
1168
+ "sep_rel": 0.718321
1169
+ },
1170
+ "model.layers.13.mlp.gate_proj": {
1171
+ "ranks": [
1172
+ 1,
1173
+ 256,
1174
+ 359
1175
+ ],
1176
+ "params": 23839952,
1177
+ "rel_err_whitened": 0.656876,
1178
+ "als_skipped_for_memory": true,
1179
+ "winner": "tt_init",
1180
+ "sep_rel": 0.792869
1181
+ },
1182
+ "model.layers.13.mlp.up_proj": {
1183
+ "ranks": [
1184
+ 1,
1185
+ 256,
1186
+ 359
1187
+ ],
1188
+ "params": 23839952,
1189
+ "rel_err_whitened": 0.659217,
1190
+ "als_skipped_for_memory": true,
1191
+ "winner": "tt_init",
1192
+ "sep_rel": 0.792869
1193
+ },
1194
+ "model.layers.13.mlp.down_proj": {
1195
+ "ranks": [
1196
+ 1,
1197
+ 256,
1198
+ 359
1199
+ ],
1200
+ "params": 23839952,
1201
+ "rel_err_whitened": 0.658152,
1202
+ "als_skipped_for_memory": true,
1203
+ "winner": "tt_init",
1204
+ "sep_rel": 0.962957
1205
+ },
1206
+ "model.layers.14.self_attn.q_proj": {
1207
+ "ranks": [
1208
+ 1,
1209
+ 186,
1210
+ 186
1211
+ ],
1212
+ "params": 8951808,
1213
+ "rel_err_whitened": 0.63805,
1214
+ "als_skipped_for_memory": false,
1215
+ "winner": "tt_padded",
1216
+ "sep_rel": 0.982983
1217
+ },
1218
+ "model.layers.14.self_attn.k_proj": {
1219
+ "ranks": [
1220
+ 1,
1221
+ 186,
1222
+ 186
1223
+ ],
1224
+ "params": 8951808,
1225
+ "rel_err_whitened": 0.636206,
1226
+ "als_skipped_for_memory": false,
1227
+ "winner": "tt_padded",
1228
+ "sep_rel": 0.982983
1229
+ },
1230
+ "model.layers.14.self_attn.v_proj": {
1231
+ "ranks": [
1232
+ 1,
1233
+ 186,
1234
+ 186
1235
+ ],
1236
+ "params": 8951808,
1237
+ "rel_err_whitened": 0.645816,
1238
+ "als_skipped_for_memory": false,
1239
+ "winner": "tt_padded",
1240
+ "sep_rel": 0.982983
1241
+ },
1242
+ "model.layers.14.self_attn.o_proj": {
1243
+ "ranks": [
1244
+ 1,
1245
+ 186,
1246
+ 186
1247
+ ],
1248
+ "params": 8951808,
1249
+ "rel_err_whitened": 0.602767,
1250
+ "als_skipped_for_memory": false,
1251
+ "winner": "tt_padded",
1252
+ "sep_rel": 0.659323
1253
+ },
1254
+ "model.layers.14.mlp.gate_proj": {
1255
+ "ranks": [
1256
+ 1,
1257
+ 256,
1258
+ 359
1259
+ ],
1260
+ "params": 23839952,
1261
+ "rel_err_whitened": 0.657,
1262
+ "als_skipped_for_memory": true,
1263
+ "winner": "tt_init",
1264
+ "sep_rel": 0.739864
1265
+ },
1266
+ "model.layers.14.mlp.up_proj": {
1267
+ "ranks": [
1268
+ 1,
1269
+ 256,
1270
+ 359
1271
+ ],
1272
+ "params": 23839952,
1273
+ "rel_err_whitened": 0.659176,
1274
+ "als_skipped_for_memory": true,
1275
+ "winner": "tt_init",
1276
+ "sep_rel": 0.739864
1277
+ },
1278
+ "model.layers.14.mlp.down_proj": {
1279
+ "ranks": [
1280
+ 1,
1281
+ 256,
1282
+ 359
1283
+ ],
1284
+ "params": 23839952,
1285
+ "rel_err_whitened": 0.657679,
1286
+ "als_skipped_for_memory": true,
1287
+ "winner": "tt_init",
1288
+ "sep_rel": 0.799497
1289
+ },
1290
+ "model.layers.15.self_attn.q_proj": {
1291
+ "ranks": [
1292
+ 1,
1293
+ 186,
1294
+ 186
1295
+ ],
1296
+ "params": 8951808,
1297
+ "rel_err_whitened": 0.637142,
1298
+ "als_skipped_for_memory": false,
1299
+ "winner": "tt_padded",
1300
+ "sep_rel": 0.985507
1301
+ },
1302
+ "model.layers.15.self_attn.k_proj": {
1303
+ "ranks": [
1304
+ 1,
1305
+ 186,
1306
+ 186
1307
+ ],
1308
+ "params": 8951808,
1309
+ "rel_err_whitened": 0.632313,
1310
+ "als_skipped_for_memory": false,
1311
+ "winner": "tt_padded",
1312
+ "sep_rel": 0.985507
1313
+ },
1314
+ "model.layers.15.self_attn.v_proj": {
1315
+ "ranks": [
1316
+ 1,
1317
+ 186,
1318
+ 186
1319
+ ],
1320
+ "params": 8951808,
1321
+ "rel_err_whitened": 0.648973,
1322
+ "als_skipped_for_memory": false,
1323
+ "winner": "tt_padded",
1324
+ "sep_rel": 0.985507
1325
+ },
1326
+ "model.layers.15.self_attn.o_proj": {
1327
+ "ranks": [
1328
+ 1,
1329
+ 186,
1330
+ 186
1331
+ ],
1332
+ "params": 8951808,
1333
+ "rel_err_whitened": 0.625458,
1334
+ "als_skipped_for_memory": false,
1335
+ "winner": "tt_padded",
1336
+ "sep_rel": 0.760429
1337
+ },
1338
+ "model.layers.15.mlp.gate_proj": {
1339
+ "ranks": [
1340
+ 1,
1341
+ 256,
1342
+ 359
1343
+ ],
1344
+ "params": 23839952,
1345
+ "rel_err_whitened": 0.656911,
1346
+ "als_skipped_for_memory": true,
1347
+ "winner": "tt_init",
1348
+ "sep_rel": 0.733882
1349
+ },
1350
+ "model.layers.15.mlp.up_proj": {
1351
+ "ranks": [
1352
+ 1,
1353
+ 256,
1354
+ 359
1355
+ ],
1356
+ "params": 23839952,
1357
+ "rel_err_whitened": 0.659253,
1358
+ "als_skipped_for_memory": true,
1359
+ "winner": "tt_init",
1360
+ "sep_rel": 0.733882
1361
+ },
1362
+ "model.layers.15.mlp.down_proj": {
1363
+ "ranks": [
1364
+ 1,
1365
+ 256,
1366
+ 359
1367
+ ],
1368
+ "params": 23839952,
1369
+ "rel_err_whitened": 0.657768,
1370
+ "als_skipped_for_memory": true,
1371
+ "winner": "tt_init",
1372
+ "sep_rel": 0.882468
1373
+ },
1374
+ "model.layers.16.self_attn.q_proj": {
1375
+ "ranks": [
1376
+ 1,
1377
+ 186,
1378
+ 186
1379
+ ],
1380
+ "params": 8951808,
1381
+ "rel_err_whitened": 0.64153,
1382
+ "als_skipped_for_memory": false,
1383
+ "winner": "tt_padded",
1384
+ "sep_rel": 0.982543
1385
+ },
1386
+ "model.layers.16.self_attn.k_proj": {
1387
+ "ranks": [
1388
+ 1,
1389
+ 186,
1390
+ 186
1391
+ ],
1392
+ "params": 8951808,
1393
+ "rel_err_whitened": 0.639549,
1394
+ "als_skipped_for_memory": false,
1395
+ "winner": "tt_padded",
1396
+ "sep_rel": 0.982543
1397
+ },
1398
+ "model.layers.16.self_attn.v_proj": {
1399
+ "ranks": [
1400
+ 1,
1401
+ 186,
1402
+ 186
1403
+ ],
1404
+ "params": 8951808,
1405
+ "rel_err_whitened": 0.646092,
1406
+ "als_skipped_for_memory": false,
1407
+ "winner": "tt_padded",
1408
+ "sep_rel": 0.982543
1409
+ },
1410
+ "model.layers.16.self_attn.o_proj": {
1411
+ "ranks": [
1412
+ 1,
1413
+ 186,
1414
+ 186
1415
+ ],
1416
+ "params": 8951808,
1417
+ "rel_err_whitened": 0.592827,
1418
+ "als_skipped_for_memory": false,
1419
+ "winner": "tt_padded",
1420
+ "sep_rel": 0.769182
1421
+ },
1422
+ "model.layers.16.mlp.gate_proj": {
1423
+ "ranks": [
1424
+ 1,
1425
+ 256,
1426
+ 359
1427
+ ],
1428
+ "params": 23839952,
1429
+ "rel_err_whitened": 0.657067,
1430
+ "als_skipped_for_memory": true,
1431
+ "winner": "tt_init",
1432
+ "sep_rel": 0.612051
1433
+ },
1434
+ "model.layers.16.mlp.up_proj": {
1435
+ "ranks": [
1436
+ 1,
1437
+ 256,
1438
+ 359
1439
+ ],
1440
+ "params": 23839952,
1441
+ "rel_err_whitened": 0.65923,
1442
+ "als_skipped_for_memory": true,
1443
+ "winner": "tt_init",
1444
+ "sep_rel": 0.612051
1445
+ },
1446
+ "model.layers.16.mlp.down_proj": {
1447
+ "ranks": [
1448
+ 1,
1449
+ 256,
1450
+ 359
1451
+ ],
1452
+ "params": 23839952,
1453
+ "rel_err_whitened": 0.657569,
1454
+ "als_skipped_for_memory": true,
1455
+ "winner": "tt_init",
1456
+ "sep_rel": 0.803771
1457
+ },
1458
+ "model.layers.17.self_attn.q_proj": {
1459
+ "ranks": [
1460
+ 1,
1461
+ 186,
1462
+ 186
1463
+ ],
1464
+ "params": 8951808,
1465
+ "rel_err_whitened": 0.642555,
1466
+ "als_skipped_for_memory": false,
1467
+ "winner": "tt_padded",
1468
+ "sep_rel": 0.972308
1469
+ },
1470
+ "model.layers.17.self_attn.k_proj": {
1471
+ "ranks": [
1472
+ 1,
1473
+ 186,
1474
+ 186
1475
+ ],
1476
+ "params": 8951808,
1477
+ "rel_err_whitened": 0.641583,
1478
+ "als_skipped_for_memory": false,
1479
+ "winner": "tt_padded",
1480
+ "sep_rel": 0.972308
1481
+ },
1482
+ "model.layers.17.self_attn.v_proj": {
1483
+ "ranks": [
1484
+ 1,
1485
+ 186,
1486
+ 186
1487
+ ],
1488
+ "params": 8951808,
1489
+ "rel_err_whitened": 0.647649,
1490
+ "als_skipped_for_memory": false,
1491
+ "winner": "tt_padded",
1492
+ "sep_rel": 0.972308
1493
+ },
1494
+ "model.layers.17.self_attn.o_proj": {
1495
+ "ranks": [
1496
+ 1,
1497
+ 186,
1498
+ 186
1499
+ ],
1500
+ "params": 8951808,
1501
+ "rel_err_whitened": 0.607117,
1502
+ "als_skipped_for_memory": false,
1503
+ "winner": "tt_padded",
1504
+ "sep_rel": 0.97167
1505
+ },
1506
+ "model.layers.17.mlp.gate_proj": {
1507
+ "ranks": [
1508
+ 1,
1509
+ 256,
1510
+ 359
1511
+ ],
1512
+ "params": 23839952,
1513
+ "rel_err_whitened": 0.656841,
1514
+ "als_skipped_for_memory": true,
1515
+ "winner": "tt_init",
1516
+ "sep_rel": 0.483312
1517
+ },
1518
+ "model.layers.17.mlp.up_proj": {
1519
+ "ranks": [
1520
+ 1,
1521
+ 256,
1522
+ 359
1523
+ ],
1524
+ "params": 23839952,
1525
+ "rel_err_whitened": 0.659206,
1526
+ "als_skipped_for_memory": true,
1527
+ "winner": "tt_init",
1528
+ "sep_rel": 0.483312
1529
+ },
1530
+ "model.layers.17.mlp.down_proj": {
1531
+ "ranks": [
1532
+ 1,
1533
+ 256,
1534
+ 359
1535
+ ],
1536
+ "params": 23839952,
1537
+ "rel_err_whitened": 0.657723,
1538
+ "als_skipped_for_memory": true,
1539
+ "winner": "tt_init",
1540
+ "sep_rel": 0.741091
1541
+ },
1542
+ "model.layers.18.self_attn.q_proj": {
1543
+ "ranks": [
1544
+ 1,
1545
+ 186,
1546
+ 186
1547
+ ],
1548
+ "params": 8951808,
1549
+ "rel_err_whitened": 0.64689,
1550
+ "als_skipped_for_memory": false,
1551
+ "winner": "tt_padded",
1552
+ "sep_rel": 0.965019
1553
+ },
1554
+ "model.layers.18.self_attn.k_proj": {
1555
+ "ranks": [
1556
+ 1,
1557
+ 186,
1558
+ 186
1559
+ ],
1560
+ "params": 8951808,
1561
+ "rel_err_whitened": 0.64468,
1562
+ "als_skipped_for_memory": false,
1563
+ "winner": "tt_padded",
1564
+ "sep_rel": 0.965019
1565
+ },
1566
+ "model.layers.18.self_attn.v_proj": {
1567
+ "ranks": [
1568
+ 1,
1569
+ 186,
1570
+ 186
1571
+ ],
1572
+ "params": 8951808,
1573
+ "rel_err_whitened": 0.642748,
1574
+ "als_skipped_for_memory": false,
1575
+ "winner": "tt_padded",
1576
+ "sep_rel": 0.965019
1577
+ },
1578
+ "model.layers.18.self_attn.o_proj": {
1579
+ "ranks": [
1580
+ 1,
1581
+ 186,
1582
+ 186
1583
+ ],
1584
+ "params": 8951808,
1585
+ "rel_err_whitened": 0.593678,
1586
+ "als_skipped_for_memory": false,
1587
+ "winner": "tt_padded",
1588
+ "sep_rel": 0.821823
1589
+ },
1590
+ "model.layers.18.mlp.gate_proj": {
1591
+ "ranks": [
1592
+ 1,
1593
+ 256,
1594
+ 359
1595
+ ],
1596
+ "params": 23839952,
1597
+ "rel_err_whitened": 0.656989,
1598
+ "als_skipped_for_memory": true,
1599
+ "winner": "tt_init",
1600
+ "sep_rel": 0.437234
1601
+ },
1602
+ "model.layers.18.mlp.up_proj": {
1603
+ "ranks": [
1604
+ 1,
1605
+ 256,
1606
+ 359
1607
+ ],
1608
+ "params": 23839952,
1609
+ "rel_err_whitened": 0.659193,
1610
+ "als_skipped_for_memory": true,
1611
+ "winner": "tt_init",
1612
+ "sep_rel": 0.437234
1613
+ },
1614
+ "model.layers.18.mlp.down_proj": {
1615
+ "ranks": [
1616
+ 1,
1617
+ 256,
1618
+ 359
1619
+ ],
1620
+ "params": 23839952,
1621
+ "rel_err_whitened": 0.658197,
1622
+ "als_skipped_for_memory": true,
1623
+ "winner": "tt_init",
1624
+ "sep_rel": 0.87914
1625
+ },
1626
+ "model.layers.19.self_attn.q_proj": {
1627
+ "ranks": [
1628
+ 1,
1629
+ 186,
1630
+ 186
1631
+ ],
1632
+ "params": 8951808,
1633
+ "rel_err_whitened": 0.64293,
1634
+ "als_skipped_for_memory": false,
1635
+ "winner": "tt_padded",
1636
+ "sep_rel": 0.963574
1637
+ },
1638
+ "model.layers.19.self_attn.k_proj": {
1639
+ "ranks": [
1640
+ 1,
1641
+ 186,
1642
+ 186
1643
+ ],
1644
+ "params": 8951808,
1645
+ "rel_err_whitened": 0.636691,
1646
+ "als_skipped_for_memory": false,
1647
+ "winner": "tt_padded",
1648
+ "sep_rel": 0.963574
1649
+ },
1650
+ "model.layers.19.self_attn.v_proj": {
1651
+ "ranks": [
1652
+ 1,
1653
+ 186,
1654
+ 186
1655
+ ],
1656
+ "params": 8951808,
1657
+ "rel_err_whitened": 0.647893,
1658
+ "als_skipped_for_memory": false,
1659
+ "winner": "tt_padded",
1660
+ "sep_rel": 0.963574
1661
+ },
1662
+ "model.layers.19.self_attn.o_proj": {
1663
+ "ranks": [
1664
+ 1,
1665
+ 186,
1666
+ 186
1667
+ ],
1668
+ "params": 8951808,
1669
+ "rel_err_whitened": 0.605244,
1670
+ "als_skipped_for_memory": false,
1671
+ "winner": "tt_padded",
1672
+ "sep_rel": 0.78794
1673
+ },
1674
+ "model.layers.19.mlp.gate_proj": {
1675
+ "ranks": [
1676
+ 1,
1677
+ 256,
1678
+ 359
1679
+ ],
1680
+ "params": 23839952,
1681
+ "rel_err_whitened": 0.657503,
1682
+ "als_skipped_for_memory": true,
1683
+ "winner": "tt_init",
1684
+ "sep_rel": 0.41581
1685
+ },
1686
+ "model.layers.19.mlp.up_proj": {
1687
+ "ranks": [
1688
+ 1,
1689
+ 256,
1690
+ 359
1691
+ ],
1692
+ "params": 23839952,
1693
+ "rel_err_whitened": 0.659375,
1694
+ "als_skipped_for_memory": true,
1695
+ "winner": "tt_init",
1696
+ "sep_rel": 0.41581
1697
+ },
1698
+ "model.layers.19.mlp.down_proj": {
1699
+ "ranks": [
1700
+ 1,
1701
+ 256,
1702
+ 359
1703
+ ],
1704
+ "params": 23839952,
1705
+ "rel_err_whitened": 0.657777,
1706
+ "als_skipped_for_memory": true,
1707
+ "winner": "tt_init",
1708
+ "sep_rel": 0.79205
1709
+ },
1710
+ "model.layers.20.self_attn.q_proj": {
1711
+ "ranks": [
1712
+ 1,
1713
+ 186,
1714
+ 186
1715
+ ],
1716
+ "params": 8951808,
1717
+ "rel_err_whitened": 0.646082,
1718
+ "als_skipped_for_memory": false,
1719
+ "winner": "tt_padded",
1720
+ "sep_rel": 0.961613
1721
+ },
1722
+ "model.layers.20.self_attn.k_proj": {
1723
+ "ranks": [
1724
+ 1,
1725
+ 186,
1726
+ 186
1727
+ ],
1728
+ "params": 8951808,
1729
+ "rel_err_whitened": 0.64135,
1730
+ "als_skipped_for_memory": false,
1731
+ "winner": "tt_padded",
1732
+ "sep_rel": 0.961613
1733
+ },
1734
+ "model.layers.20.self_attn.v_proj": {
1735
+ "ranks": [
1736
+ 1,
1737
+ 186,
1738
+ 186
1739
+ ],
1740
+ "params": 8951808,
1741
+ "rel_err_whitened": 0.654413,
1742
+ "als_skipped_for_memory": false,
1743
+ "winner": "tt_padded",
1744
+ "sep_rel": 0.961613
1745
+ },
1746
+ "model.layers.20.self_attn.o_proj": {
1747
+ "ranks": [
1748
+ 1,
1749
+ 186,
1750
+ 186
1751
+ ],
1752
+ "params": 8951808,
1753
+ "rel_err_whitened": 0.579479,
1754
+ "als_skipped_for_memory": false,
1755
+ "winner": "tt_padded",
1756
+ "sep_rel": 0.859383
1757
+ },
1758
+ "model.layers.20.mlp.gate_proj": {
1759
+ "ranks": [
1760
+ 1,
1761
+ 256,
1762
+ 359
1763
+ ],
1764
+ "params": 23839952,
1765
+ "rel_err_whitened": 0.657567,
1766
+ "als_skipped_for_memory": true,
1767
+ "winner": "tt_init",
1768
+ "sep_rel": 0.387915
1769
+ },
1770
+ "model.layers.20.mlp.up_proj": {
1771
+ "ranks": [
1772
+ 1,
1773
+ 256,
1774
+ 359
1775
+ ],
1776
+ "params": 23839952,
1777
+ "rel_err_whitened": 0.659324,
1778
+ "als_skipped_for_memory": true,
1779
+ "winner": "tt_init",
1780
+ "sep_rel": 0.387915
1781
+ },
1782
+ "model.layers.20.mlp.down_proj": {
1783
+ "ranks": [
1784
+ 1,
1785
+ 256,
1786
+ 359
1787
+ ],
1788
+ "params": 23839952,
1789
+ "rel_err_whitened": 0.657449,
1790
+ "als_skipped_for_memory": true,
1791
+ "winner": "tt_init",
1792
+ "sep_rel": 0.90531
1793
+ },
1794
+ "model.layers.21.self_attn.q_proj": {
1795
+ "ranks": [
1796
+ 1,
1797
+ 186,
1798
+ 186
1799
+ ],
1800
+ "params": 8951808,
1801
+ "rel_err_whitened": 0.632027,
1802
+ "als_skipped_for_memory": false,
1803
+ "winner": "tt_padded",
1804
+ "sep_rel": 0.957159
1805
+ },
1806
+ "model.layers.21.self_attn.k_proj": {
1807
+ "ranks": [
1808
+ 1,
1809
+ 186,
1810
+ 186
1811
+ ],
1812
+ "params": 8951808,
1813
+ "rel_err_whitened": 0.626278,
1814
+ "als_skipped_for_memory": false,
1815
+ "winner": "tt_padded",
1816
+ "sep_rel": 0.957159
1817
+ },
1818
+ "model.layers.21.self_attn.v_proj": {
1819
+ "ranks": [
1820
+ 1,
1821
+ 186,
1822
+ 186
1823
+ ],
1824
+ "params": 8951808,
1825
+ "rel_err_whitened": 0.65086,
1826
+ "als_skipped_for_memory": false,
1827
+ "winner": "tt_padded",
1828
+ "sep_rel": 0.957159
1829
+ },
1830
+ "model.layers.21.self_attn.o_proj": {
1831
+ "ranks": [
1832
+ 1,
1833
+ 186,
1834
+ 186
1835
+ ],
1836
+ "params": 8951808,
1837
+ "rel_err_whitened": 0.59442,
1838
+ "als_skipped_for_memory": false,
1839
+ "winner": "tt_padded",
1840
+ "sep_rel": 0.603205
1841
+ },
1842
+ "model.layers.21.mlp.gate_proj": {
1843
+ "ranks": [
1844
+ 1,
1845
+ 256,
1846
+ 359
1847
+ ],
1848
+ "params": 23839952,
1849
+ "rel_err_whitened": 0.65767,
1850
+ "als_skipped_for_memory": true,
1851
+ "winner": "tt_init",
1852
+ "sep_rel": 0.365638
1853
+ },
1854
+ "model.layers.21.mlp.up_proj": {
1855
+ "ranks": [
1856
+ 1,
1857
+ 256,
1858
+ 359
1859
+ ],
1860
+ "params": 23839952,
1861
+ "rel_err_whitened": 0.659335,
1862
+ "als_skipped_for_memory": true,
1863
+ "winner": "tt_init",
1864
+ "sep_rel": 0.365638
1865
+ },
1866
+ "model.layers.21.mlp.down_proj": {
1867
+ "ranks": [
1868
+ 1,
1869
+ 256,
1870
+ 359
1871
+ ],
1872
+ "params": 23839952,
1873
+ "rel_err_whitened": 0.657633,
1874
+ "als_skipped_for_memory": true,
1875
+ "winner": "tt_init",
1876
+ "sep_rel": 0.826779
1877
+ },
1878
+ "model.layers.22.self_attn.q_proj": {
1879
+ "ranks": [
1880
+ 1,
1881
+ 186,
1882
+ 186
1883
+ ],
1884
+ "params": 8951808,
1885
+ "rel_err_whitened": 0.639349,
1886
+ "als_skipped_for_memory": false,
1887
+ "winner": "tt_padded",
1888
+ "sep_rel": 0.961618
1889
+ },
1890
+ "model.layers.22.self_attn.k_proj": {
1891
+ "ranks": [
1892
+ 1,
1893
+ 186,
1894
+ 186
1895
+ ],
1896
+ "params": 8951808,
1897
+ "rel_err_whitened": 0.636295,
1898
+ "als_skipped_for_memory": false,
1899
+ "winner": "tt_padded",
1900
+ "sep_rel": 0.961618
1901
+ },
1902
+ "model.layers.22.self_attn.v_proj": {
1903
+ "ranks": [
1904
+ 1,
1905
+ 186,
1906
+ 186
1907
+ ],
1908
+ "params": 8951808,
1909
+ "rel_err_whitened": 0.647576,
1910
+ "als_skipped_for_memory": false,
1911
+ "winner": "tt_padded",
1912
+ "sep_rel": 0.961618
1913
+ },
1914
+ "model.layers.22.self_attn.o_proj": {
1915
+ "ranks": [
1916
+ 1,
1917
+ 186,
1918
+ 186
1919
+ ],
1920
+ "params": 8951808,
1921
+ "rel_err_whitened": 0.590262,
1922
+ "als_skipped_for_memory": false,
1923
+ "winner": "tt_padded",
1924
+ "sep_rel": 0.677762
1925
+ },
1926
+ "model.layers.22.mlp.gate_proj": {
1927
+ "ranks": [
1928
+ 1,
1929
+ 256,
1930
+ 359
1931
+ ],
1932
+ "params": 23839952,
1933
+ "rel_err_whitened": 0.657579,
1934
+ "als_skipped_for_memory": true,
1935
+ "winner": "tt_init",
1936
+ "sep_rel": 0.349898
1937
+ },
1938
+ "model.layers.22.mlp.up_proj": {
1939
+ "ranks": [
1940
+ 1,
1941
+ 256,
1942
+ 359
1943
+ ],
1944
+ "params": 23839952,
1945
+ "rel_err_whitened": 0.659294,
1946
+ "als_skipped_for_memory": true,
1947
+ "winner": "tt_init",
1948
+ "sep_rel": 0.349898
1949
+ },
1950
+ "model.layers.22.mlp.down_proj": {
1951
+ "ranks": [
1952
+ 1,
1953
+ 256,
1954
+ 359
1955
+ ],
1956
+ "params": 23839952,
1957
+ "rel_err_whitened": 0.657605,
1958
+ "als_skipped_for_memory": true,
1959
+ "winner": "tt_init",
1960
+ "sep_rel": 0.780275
1961
+ },
1962
+ "model.layers.23.self_attn.q_proj": {
1963
+ "ranks": [
1964
+ 1,
1965
+ 186,
1966
+ 186
1967
+ ],
1968
+ "params": 8951808,
1969
+ "rel_err_whitened": 0.63777,
1970
+ "als_skipped_for_memory": false,
1971
+ "winner": "tt_padded",
1972
+ "sep_rel": 0.950146
1973
+ },
1974
+ "model.layers.23.self_attn.k_proj": {
1975
+ "ranks": [
1976
+ 1,
1977
+ 186,
1978
+ 186
1979
+ ],
1980
+ "params": 8951808,
1981
+ "rel_err_whitened": 0.636755,
1982
+ "als_skipped_for_memory": false,
1983
+ "winner": "tt_padded",
1984
+ "sep_rel": 0.950146
1985
+ },
1986
+ "model.layers.23.self_attn.v_proj": {
1987
+ "ranks": [
1988
+ 1,
1989
+ 186,
1990
+ 186
1991
+ ],
1992
+ "params": 8951808,
1993
+ "rel_err_whitened": 0.643311,
1994
+ "als_skipped_for_memory": false,
1995
+ "winner": "tt_padded",
1996
+ "sep_rel": 0.950146
1997
+ },
1998
+ "model.layers.23.self_attn.o_proj": {
1999
+ "ranks": [
2000
+ 1,
2001
+ 186,
2002
+ 186
2003
+ ],
2004
+ "params": 8951808,
2005
+ "rel_err_whitened": 0.586402,
2006
+ "als_skipped_for_memory": false,
2007
+ "winner": "tt_padded",
2008
+ "sep_rel": 0.770723
2009
+ },
2010
+ "model.layers.23.mlp.gate_proj": {
2011
+ "ranks": [
2012
+ 1,
2013
+ 256,
2014
+ 359
2015
+ ],
2016
+ "params": 23839952,
2017
+ "rel_err_whitened": 0.658064,
2018
+ "als_skipped_for_memory": true,
2019
+ "winner": "tt_init",
2020
+ "sep_rel": 0.398491
2021
+ },
2022
+ "model.layers.23.mlp.up_proj": {
2023
+ "ranks": [
2024
+ 1,
2025
+ 256,
2026
+ 359
2027
+ ],
2028
+ "params": 23839952,
2029
+ "rel_err_whitened": 0.659374,
2030
+ "als_skipped_for_memory": true,
2031
+ "winner": "tt_init",
2032
+ "sep_rel": 0.398491
2033
+ },
2034
+ "model.layers.23.mlp.down_proj": {
2035
+ "ranks": [
2036
+ 1,
2037
+ 256,
2038
+ 359
2039
+ ],
2040
+ "params": 23839952,
2041
+ "rel_err_whitened": 0.657101,
2042
+ "als_skipped_for_memory": true,
2043
+ "winner": "tt_init",
2044
+ "sep_rel": 0.793609
2045
+ },
2046
+ "model.layers.24.self_attn.q_proj": {
2047
+ "ranks": [
2048
+ 1,
2049
+ 186,
2050
+ 186
2051
+ ],
2052
+ "params": 8951808,
2053
+ "rel_err_whitened": 0.629628,
2054
+ "als_skipped_for_memory": false,
2055
+ "winner": "tt_padded",
2056
+ "sep_rel": 0.966237
2057
+ },
2058
+ "model.layers.24.self_attn.k_proj": {
2059
+ "ranks": [
2060
+ 1,
2061
+ 186,
2062
+ 186
2063
+ ],
2064
+ "params": 8951808,
2065
+ "rel_err_whitened": 0.626718,
2066
+ "als_skipped_for_memory": false,
2067
+ "winner": "tt_padded",
2068
+ "sep_rel": 0.966237
2069
+ },
2070
+ "model.layers.24.self_attn.v_proj": {
2071
+ "ranks": [
2072
+ 1,
2073
+ 186,
2074
+ 186
2075
+ ],
2076
+ "params": 8951808,
2077
+ "rel_err_whitened": 0.650236,
2078
+ "als_skipped_for_memory": false,
2079
+ "winner": "tt_padded",
2080
+ "sep_rel": 0.966237
2081
+ },
2082
+ "model.layers.24.self_attn.o_proj": {
2083
+ "ranks": [
2084
+ 1,
2085
+ 186,
2086
+ 186
2087
+ ],
2088
+ "params": 8951808,
2089
+ "rel_err_whitened": 0.601252,
2090
+ "als_skipped_for_memory": false,
2091
+ "winner": "tt_padded",
2092
+ "sep_rel": 0.956491
2093
+ },
2094
+ "model.layers.24.mlp.gate_proj": {
2095
+ "ranks": [
2096
+ 1,
2097
+ 256,
2098
+ 359
2099
+ ],
2100
+ "params": 23839952,
2101
+ "rel_err_whitened": 0.658118,
2102
+ "als_skipped_for_memory": true,
2103
+ "winner": "tt_init",
2104
+ "sep_rel": 0.365118
2105
+ },
2106
+ "model.layers.24.mlp.up_proj": {
2107
+ "ranks": [
2108
+ 1,
2109
+ 256,
2110
+ 359
2111
+ ],
2112
+ "params": 23839952,
2113
+ "rel_err_whitened": 0.659368,
2114
+ "als_skipped_for_memory": true,
2115
+ "winner": "tt_init",
2116
+ "sep_rel": 0.365118
2117
+ },
2118
+ "model.layers.24.mlp.down_proj": {
2119
+ "ranks": [
2120
+ 1,
2121
+ 256,
2122
+ 359
2123
+ ],
2124
+ "params": 23839952,
2125
+ "rel_err_whitened": 0.657714,
2126
+ "als_skipped_for_memory": true,
2127
+ "winner": "tt_init",
2128
+ "sep_rel": 0.886513
2129
+ },
2130
+ "model.layers.25.self_attn.q_proj": {
2131
+ "ranks": [
2132
+ 1,
2133
+ 186,
2134
+ 186
2135
+ ],
2136
+ "params": 8951808,
2137
+ "rel_err_whitened": 0.636998,
2138
+ "als_skipped_for_memory": false,
2139
+ "winner": "tt_padded",
2140
+ "sep_rel": 0.942499
2141
+ },
2142
+ "model.layers.25.self_attn.k_proj": {
2143
+ "ranks": [
2144
+ 1,
2145
+ 186,
2146
+ 186
2147
+ ],
2148
+ "params": 8951808,
2149
+ "rel_err_whitened": 0.636449,
2150
+ "als_skipped_for_memory": false,
2151
+ "winner": "tt_padded",
2152
+ "sep_rel": 0.942499
2153
+ },
2154
+ "model.layers.25.self_attn.v_proj": {
2155
+ "ranks": [
2156
+ 1,
2157
+ 186,
2158
+ 186
2159
+ ],
2160
+ "params": 8951808,
2161
+ "rel_err_whitened": 0.652161,
2162
+ "als_skipped_for_memory": false,
2163
+ "winner": "tt_padded",
2164
+ "sep_rel": 0.942499
2165
+ },
2166
+ "model.layers.25.self_attn.o_proj": {
2167
+ "ranks": [
2168
+ 1,
2169
+ 186,
2170
+ 186
2171
+ ],
2172
+ "params": 8951808,
2173
+ "rel_err_whitened": 0.595934,
2174
+ "als_skipped_for_memory": false,
2175
+ "winner": "tt_padded",
2176
+ "sep_rel": 0.683377
2177
+ },
2178
+ "model.layers.25.mlp.gate_proj": {
2179
+ "ranks": [
2180
+ 1,
2181
+ 256,
2182
+ 359
2183
+ ],
2184
+ "params": 23839952,
2185
+ "rel_err_whitened": 0.658159,
2186
+ "als_skipped_for_memory": true,
2187
+ "winner": "tt_init",
2188
+ "sep_rel": 0.452241
2189
+ },
2190
+ "model.layers.25.mlp.up_proj": {
2191
+ "ranks": [
2192
+ 1,
2193
+ 256,
2194
+ 359
2195
+ ],
2196
+ "params": 23839952,
2197
+ "rel_err_whitened": 0.659377,
2198
+ "als_skipped_for_memory": true,
2199
+ "winner": "tt_init",
2200
+ "sep_rel": 0.452241
2201
+ },
2202
+ "model.layers.25.mlp.down_proj": {
2203
+ "ranks": [
2204
+ 1,
2205
+ 256,
2206
+ 359
2207
+ ],
2208
+ "params": 23839952,
2209
+ "rel_err_whitened": 0.657923,
2210
+ "als_skipped_for_memory": true,
2211
+ "winner": "tt_init",
2212
+ "sep_rel": 0.911422
2213
+ },
2214
+ "model.layers.26.self_attn.q_proj": {
2215
+ "ranks": [
2216
+ 1,
2217
+ 186,
2218
+ 186
2219
+ ],
2220
+ "params": 8951808,
2221
+ "rel_err_whitened": 0.632518,
2222
+ "als_skipped_for_memory": false,
2223
+ "winner": "tt_padded",
2224
+ "sep_rel": 0.957061
2225
+ },
2226
+ "model.layers.26.self_attn.k_proj": {
2227
+ "ranks": [
2228
+ 1,
2229
+ 186,
2230
+ 186
2231
+ ],
2232
+ "params": 8951808,
2233
+ "rel_err_whitened": 0.629274,
2234
+ "als_skipped_for_memory": false,
2235
+ "winner": "tt_padded",
2236
+ "sep_rel": 0.957061
2237
+ },
2238
+ "model.layers.26.self_attn.v_proj": {
2239
+ "ranks": [
2240
+ 1,
2241
+ 186,
2242
+ 186
2243
+ ],
2244
+ "params": 8951808,
2245
+ "rel_err_whitened": 0.642824,
2246
+ "als_skipped_for_memory": false,
2247
+ "winner": "tt_padded",
2248
+ "sep_rel": 0.957061
2249
+ },
2250
+ "model.layers.26.self_attn.o_proj": {
2251
+ "ranks": [
2252
+ 1,
2253
+ 186,
2254
+ 186
2255
+ ],
2256
+ "params": 8951808,
2257
+ "rel_err_whitened": 0.55959,
2258
+ "als_skipped_for_memory": false,
2259
+ "winner": "tt_padded",
2260
+ "sep_rel": 0.887786
2261
+ },
2262
+ "model.layers.26.mlp.gate_proj": {
2263
+ "ranks": [
2264
+ 1,
2265
+ 256,
2266
+ 359
2267
+ ],
2268
+ "params": 23839952,
2269
+ "rel_err_whitened": 0.658006,
2270
+ "als_skipped_for_memory": true,
2271
+ "winner": "tt_init",
2272
+ "sep_rel": 0.452517
2273
+ },
2274
+ "model.layers.26.mlp.up_proj": {
2275
+ "ranks": [
2276
+ 1,
2277
+ 256,
2278
+ 359
2279
+ ],
2280
+ "params": 23839952,
2281
+ "rel_err_whitened": 0.659327,
2282
+ "als_skipped_for_memory": true,
2283
+ "winner": "tt_init",
2284
+ "sep_rel": 0.452517
2285
+ },
2286
+ "model.layers.26.mlp.down_proj": {
2287
+ "ranks": [
2288
+ 1,
2289
+ 256,
2290
+ 359
2291
+ ],
2292
+ "params": 23839952,
2293
+ "rel_err_whitened": 0.657197,
2294
+ "als_skipped_for_memory": true,
2295
+ "winner": "tt_init",
2296
+ "sep_rel": 0.812424
2297
+ },
2298
+ "model.layers.27.self_attn.q_proj": {
2299
+ "ranks": [
2300
+ 1,
2301
+ 186,
2302
+ 186
2303
+ ],
2304
+ "params": 8951808,
2305
+ "rel_err_whitened": 0.644891,
2306
+ "als_skipped_for_memory": false,
2307
+ "winner": "tt_padded",
2308
+ "sep_rel": 0.909706
2309
+ },
2310
+ "model.layers.27.self_attn.k_proj": {
2311
+ "ranks": [
2312
+ 1,
2313
+ 186,
2314
+ 186
2315
+ ],
2316
+ "params": 8951808,
2317
+ "rel_err_whitened": 0.646297,
2318
+ "als_skipped_for_memory": false,
2319
+ "winner": "tt_padded",
2320
+ "sep_rel": 0.909706
2321
+ },
2322
+ "model.layers.27.self_attn.v_proj": {
2323
+ "ranks": [
2324
+ 1,
2325
+ 186,
2326
+ 186
2327
+ ],
2328
+ "params": 8951808,
2329
+ "rel_err_whitened": 0.645076,
2330
+ "als_skipped_for_memory": false,
2331
+ "winner": "tt_padded",
2332
+ "sep_rel": 0.909706
2333
+ },
2334
+ "model.layers.27.self_attn.o_proj": {
2335
+ "ranks": [
2336
+ 1,
2337
+ 186,
2338
+ 186
2339
+ ],
2340
+ "params": 8951808,
2341
+ "rel_err_whitened": 0.594888,
2342
+ "als_skipped_for_memory": false,
2343
+ "winner": "tt_padded",
2344
+ "sep_rel": 0.895548
2345
+ },
2346
+ "model.layers.27.mlp.gate_proj": {
2347
+ "ranks": [
2348
+ 1,
2349
+ 256,
2350
+ 359
2351
+ ],
2352
+ "params": 23839952,
2353
+ "rel_err_whitened": 0.65806,
2354
+ "als_skipped_for_memory": true,
2355
+ "winner": "tt_init",
2356
+ "sep_rel": 0.495034
2357
+ },
2358
+ "model.layers.27.mlp.up_proj": {
2359
+ "ranks": [
2360
+ 1,
2361
+ 256,
2362
+ 359
2363
+ ],
2364
+ "params": 23839952,
2365
+ "rel_err_whitened": 0.659382,
2366
+ "als_skipped_for_memory": true,
2367
+ "winner": "tt_init",
2368
+ "sep_rel": 0.495034
2369
+ },
2370
+ "model.layers.27.mlp.down_proj": {
2371
+ "ranks": [
2372
+ 1,
2373
+ 256,
2374
+ 359
2375
+ ],
2376
+ "params": 23839952,
2377
+ "rel_err_whitened": 0.656948,
2378
+ "als_skipped_for_memory": true,
2379
+ "winner": "tt_init",
2380
+ "sep_rel": 0.863299
2381
+ },
2382
+ "model.layers.28.self_attn.q_proj": {
2383
+ "ranks": [
2384
+ 1,
2385
+ 186,
2386
+ 186
2387
+ ],
2388
+ "params": 8951808,
2389
+ "rel_err_whitened": 0.645519,
2390
+ "als_skipped_for_memory": false,
2391
+ "winner": "tt_padded",
2392
+ "sep_rel": 0.912117
2393
+ },
2394
+ "model.layers.28.self_attn.k_proj": {
2395
+ "ranks": [
2396
+ 1,
2397
+ 186,
2398
+ 186
2399
+ ],
2400
+ "params": 8951808,
2401
+ "rel_err_whitened": 0.64675,
2402
+ "als_skipped_for_memory": false,
2403
+ "winner": "tt_padded",
2404
+ "sep_rel": 0.912117
2405
+ },
2406
+ "model.layers.28.self_attn.v_proj": {
2407
+ "ranks": [
2408
+ 1,
2409
+ 186,
2410
+ 186
2411
+ ],
2412
+ "params": 8951808,
2413
+ "rel_err_whitened": 0.646418,
2414
+ "als_skipped_for_memory": false,
2415
+ "winner": "tt_padded",
2416
+ "sep_rel": 0.912117
2417
+ },
2418
+ "model.layers.28.self_attn.o_proj": {
2419
+ "ranks": [
2420
+ 1,
2421
+ 186,
2422
+ 186
2423
+ ],
2424
+ "params": 8951808,
2425
+ "rel_err_whitened": 0.576397,
2426
+ "als_skipped_for_memory": false,
2427
+ "winner": "tt_padded",
2428
+ "sep_rel": 0.835975
2429
+ },
2430
+ "model.layers.28.mlp.gate_proj": {
2431
+ "ranks": [
2432
+ 1,
2433
+ 256,
2434
+ 359
2435
+ ],
2436
+ "params": 23839952,
2437
+ "rel_err_whitened": 0.657703,
2438
+ "als_skipped_for_memory": true,
2439
+ "winner": "tt_init",
2440
+ "sep_rel": 0.587645
2441
+ },
2442
+ "model.layers.28.mlp.up_proj": {
2443
+ "ranks": [
2444
+ 1,
2445
+ 256,
2446
+ 359
2447
+ ],
2448
+ "params": 23839952,
2449
+ "rel_err_whitened": 0.65931,
2450
+ "als_skipped_for_memory": true,
2451
+ "winner": "tt_init",
2452
+ "sep_rel": 0.587645
2453
+ },
2454
+ "model.layers.28.mlp.down_proj": {
2455
+ "ranks": [
2456
+ 1,
2457
+ 256,
2458
+ 359
2459
+ ],
2460
+ "params": 23839952,
2461
+ "rel_err_whitened": 0.656743,
2462
+ "als_skipped_for_memory": true,
2463
+ "winner": "tt_init",
2464
+ "sep_rel": 0.943625
2465
+ },
2466
+ "model.layers.29.self_attn.q_proj": {
2467
+ "ranks": [
2468
+ 1,
2469
+ 186,
2470
+ 186
2471
+ ],
2472
+ "params": 8951808,
2473
+ "rel_err_whitened": 0.640665,
2474
+ "als_skipped_for_memory": false,
2475
+ "winner": "tt_padded",
2476
+ "sep_rel": 0.923416
2477
+ },
2478
+ "model.layers.29.self_attn.k_proj": {
2479
+ "ranks": [
2480
+ 1,
2481
+ 186,
2482
+ 186
2483
+ ],
2484
+ "params": 8951808,
2485
+ "rel_err_whitened": 0.640274,
2486
+ "als_skipped_for_memory": false,
2487
+ "winner": "tt_padded",
2488
+ "sep_rel": 0.923416
2489
+ },
2490
+ "model.layers.29.self_attn.v_proj": {
2491
+ "ranks": [
2492
+ 1,
2493
+ 186,
2494
+ 186
2495
+ ],
2496
+ "params": 8951808,
2497
+ "rel_err_whitened": 0.648709,
2498
+ "als_skipped_for_memory": false,
2499
+ "winner": "tt_padded",
2500
+ "sep_rel": 0.923416
2501
+ },
2502
+ "model.layers.29.self_attn.o_proj": {
2503
+ "ranks": [
2504
+ 1,
2505
+ 186,
2506
+ 186
2507
+ ],
2508
+ "params": 8951808,
2509
+ "rel_err_whitened": 0.577575,
2510
+ "als_skipped_for_memory": false,
2511
+ "winner": "tt_padded",
2512
+ "sep_rel": 0.73307
2513
+ },
2514
+ "model.layers.29.mlp.gate_proj": {
2515
+ "ranks": [
2516
+ 1,
2517
+ 256,
2518
+ 359
2519
+ ],
2520
+ "params": 23839952,
2521
+ "rel_err_whitened": 0.65723,
2522
+ "als_skipped_for_memory": true,
2523
+ "winner": "tt_init",
2524
+ "sep_rel": 0.636415
2525
+ },
2526
+ "model.layers.29.mlp.up_proj": {
2527
+ "ranks": [
2528
+ 1,
2529
+ 256,
2530
+ 359
2531
+ ],
2532
+ "params": 23839952,
2533
+ "rel_err_whitened": 0.658817,
2534
+ "als_skipped_for_memory": true,
2535
+ "winner": "tt_init",
2536
+ "sep_rel": 0.636415
2537
+ },
2538
+ "model.layers.29.mlp.down_proj": {
2539
+ "ranks": [
2540
+ 1,
2541
+ 256,
2542
+ 359
2543
+ ],
2544
+ "params": 23839952,
2545
+ "rel_err_whitened": 0.655203,
2546
+ "als_skipped_for_memory": true,
2547
+ "winner": "tt_init",
2548
+ "sep_rel": 0.935053
2549
+ },
2550
+ "model.layers.30.self_attn.q_proj": {
2551
+ "ranks": [
2552
+ 1,
2553
+ 186,
2554
+ 186
2555
+ ],
2556
+ "params": 8951808,
2557
+ "rel_err_whitened": 0.640681,
2558
+ "als_skipped_for_memory": false,
2559
+ "winner": "tt_padded",
2560
+ "sep_rel": 0.831597
2561
+ },
2562
+ "model.layers.30.self_attn.k_proj": {
2563
+ "ranks": [
2564
+ 1,
2565
+ 186,
2566
+ 186
2567
+ ],
2568
+ "params": 8951808,
2569
+ "rel_err_whitened": 0.643412,
2570
+ "als_skipped_for_memory": false,
2571
+ "winner": "tt_padded",
2572
+ "sep_rel": 0.831597
2573
+ },
2574
+ "model.layers.30.self_attn.v_proj": {
2575
+ "ranks": [
2576
+ 1,
2577
+ 186,
2578
+ 186
2579
+ ],
2580
+ "params": 8951808,
2581
+ "rel_err_whitened": 0.651799,
2582
+ "als_skipped_for_memory": false,
2583
+ "winner": "tt_padded",
2584
+ "sep_rel": 0.831597
2585
+ },
2586
+ "model.layers.30.self_attn.o_proj": {
2587
+ "ranks": [
2588
+ 1,
2589
+ 186,
2590
+ 186
2591
+ ],
2592
+ "params": 8951808,
2593
+ "rel_err_whitened": 0.563437,
2594
+ "als_skipped_for_memory": false,
2595
+ "winner": "tt_padded",
2596
+ "sep_rel": 0.882776
2597
+ },
2598
+ "model.layers.30.mlp.gate_proj": {
2599
+ "ranks": [
2600
+ 1,
2601
+ 256,
2602
+ 359
2603
+ ],
2604
+ "params": 23839952,
2605
+ "rel_err_whitened": 0.652647,
2606
+ "als_skipped_for_memory": true,
2607
+ "winner": "tt_init",
2608
+ "sep_rel": 0.640354
2609
+ },
2610
+ "model.layers.30.mlp.up_proj": {
2611
+ "ranks": [
2612
+ 1,
2613
+ 256,
2614
+ 359
2615
+ ],
2616
+ "params": 23839952,
2617
+ "rel_err_whitened": 0.653742,
2618
+ "als_skipped_for_memory": true,
2619
+ "winner": "tt_init",
2620
+ "sep_rel": 0.640354
2621
+ },
2622
+ "model.layers.30.mlp.down_proj": {
2623
+ "ranks": [
2624
+ 1,
2625
+ 256,
2626
+ 359
2627
+ ],
2628
+ "params": 23839952,
2629
+ "rel_err_whitened": 0.65342,
2630
+ "als_skipped_for_memory": true,
2631
+ "winner": "tt_init",
2632
+ "sep_rel": 0.999937
2633
+ },
2634
+ "model.layers.31.self_attn.q_proj": {
2635
+ "ranks": [
2636
+ 1,
2637
+ 186,
2638
+ 186
2639
+ ],
2640
+ "params": 8951808,
2641
+ "rel_err_whitened": 0.644467,
2642
+ "als_skipped_for_memory": false,
2643
+ "winner": "tt_padded",
2644
+ "sep_rel": 0.819929
2645
+ },
2646
+ "model.layers.31.self_attn.k_proj": {
2647
+ "ranks": [
2648
+ 1,
2649
+ 186,
2650
+ 186
2651
+ ],
2652
+ "params": 8951808,
2653
+ "rel_err_whitened": 0.642172,
2654
+ "als_skipped_for_memory": false,
2655
+ "winner": "tt_padded",
2656
+ "sep_rel": 0.819929
2657
+ },
2658
+ "model.layers.31.self_attn.v_proj": {
2659
+ "ranks": [
2660
+ 1,
2661
+ 186,
2662
+ 186
2663
+ ],
2664
+ "params": 8951808,
2665
+ "rel_err_whitened": 0.649594,
2666
+ "als_skipped_for_memory": false,
2667
+ "winner": "tt_padded",
2668
+ "sep_rel": 0.819929
2669
+ },
2670
+ "model.layers.31.self_attn.o_proj": {
2671
+ "ranks": [
2672
+ 1,
2673
+ 186,
2674
+ 186
2675
+ ],
2676
+ "params": 8951808,
2677
+ "rel_err_whitened": 0.573771,
2678
+ "als_skipped_for_memory": false,
2679
+ "winner": "tt_padded",
2680
+ "sep_rel": 0.831382
2681
+ },
2682
+ "model.layers.31.mlp.gate_proj": {
2683
+ "ranks": [
2684
+ 1,
2685
+ 256,
2686
+ 359
2687
+ ],
2688
+ "params": 23839952,
2689
+ "rel_err_whitened": 0.657584,
2690
+ "als_skipped_for_memory": true,
2691
+ "winner": "tt_init",
2692
+ "sep_rel": 0.827455
2693
+ },
2694
+ "model.layers.31.mlp.up_proj": {
2695
+ "ranks": [
2696
+ 1,
2697
+ 256,
2698
+ 359
2699
+ ],
2700
+ "params": 23839952,
2701
+ "rel_err_whitened": 0.658047,
2702
+ "als_skipped_for_memory": true,
2703
+ "winner": "tt_init",
2704
+ "sep_rel": 0.827455
2705
+ },
2706
+ "model.layers.31.mlp.down_proj": {
2707
+ "ranks": [
2708
+ 1,
2709
+ 256,
2710
+ 359
2711
+ ],
2712
+ "params": 23839952,
2713
+ "rel_err_whitened": 0.648149,
2714
+ "als_skipped_for_memory": true,
2715
+ "winner": "tt_init",
2716
+ "sep_rel": 0.997833
2717
+ }
2718
+ },
2719
+ "healed": false
2720
+ }
tt3/unheal/llama2-7b-base/ranks.json ADDED
@@ -0,0 +1,2690 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model.layers.0.self_attn.q_proj": {
3
+ "o1": 16,
4
+ "o2": 16,
5
+ "o3": 16,
6
+ "i1": 16,
7
+ "i2": 16,
8
+ "i3": 16,
9
+ "r0": 1,
10
+ "r1": 186,
11
+ "r2": 186,
12
+ "nnz": 838860
13
+ },
14
+ "model.layers.0.self_attn.k_proj": {
15
+ "o1": 16,
16
+ "o2": 16,
17
+ "o3": 16,
18
+ "i1": 16,
19
+ "i2": 16,
20
+ "i3": 16,
21
+ "r0": 1,
22
+ "r1": 186,
23
+ "r2": 186,
24
+ "nnz": 838860
25
+ },
26
+ "model.layers.0.self_attn.v_proj": {
27
+ "o1": 16,
28
+ "o2": 16,
29
+ "o3": 16,
30
+ "i1": 16,
31
+ "i2": 16,
32
+ "i3": 16,
33
+ "r0": 1,
34
+ "r1": 186,
35
+ "r2": 186,
36
+ "nnz": 838860
37
+ },
38
+ "model.layers.0.self_attn.o_proj": {
39
+ "o1": 16,
40
+ "o2": 16,
41
+ "o3": 16,
42
+ "i1": 16,
43
+ "i2": 16,
44
+ "i3": 16,
45
+ "r0": 1,
46
+ "r1": 186,
47
+ "r2": 186,
48
+ "nnz": 838860
49
+ },
50
+ "model.layers.0.mlp.gate_proj": {
51
+ "o1": 16,
52
+ "o2": 16,
53
+ "o3": 43,
54
+ "i1": 16,
55
+ "i2": 16,
56
+ "i3": 16,
57
+ "r0": 1,
58
+ "r1": 256,
59
+ "r2": 359,
60
+ "nnz": 2254438
61
+ },
62
+ "model.layers.0.mlp.up_proj": {
63
+ "o1": 16,
64
+ "o2": 16,
65
+ "o3": 43,
66
+ "i1": 16,
67
+ "i2": 16,
68
+ "i3": 16,
69
+ "r0": 1,
70
+ "r1": 256,
71
+ "r2": 359,
72
+ "nnz": 2254438
73
+ },
74
+ "model.layers.0.mlp.down_proj": {
75
+ "o1": 16,
76
+ "o2": 16,
77
+ "o3": 16,
78
+ "i1": 16,
79
+ "i2": 16,
80
+ "i3": 43,
81
+ "r0": 1,
82
+ "r1": 256,
83
+ "r2": 359,
84
+ "nnz": 2254438
85
+ },
86
+ "model.layers.1.self_attn.q_proj": {
87
+ "o1": 16,
88
+ "o2": 16,
89
+ "o3": 16,
90
+ "i1": 16,
91
+ "i2": 16,
92
+ "i3": 16,
93
+ "r0": 1,
94
+ "r1": 186,
95
+ "r2": 186,
96
+ "nnz": 838860
97
+ },
98
+ "model.layers.1.self_attn.k_proj": {
99
+ "o1": 16,
100
+ "o2": 16,
101
+ "o3": 16,
102
+ "i1": 16,
103
+ "i2": 16,
104
+ "i3": 16,
105
+ "r0": 1,
106
+ "r1": 186,
107
+ "r2": 186,
108
+ "nnz": 838860
109
+ },
110
+ "model.layers.1.self_attn.v_proj": {
111
+ "o1": 16,
112
+ "o2": 16,
113
+ "o3": 16,
114
+ "i1": 16,
115
+ "i2": 16,
116
+ "i3": 16,
117
+ "r0": 1,
118
+ "r1": 186,
119
+ "r2": 186,
120
+ "nnz": 838860
121
+ },
122
+ "model.layers.1.self_attn.o_proj": {
123
+ "o1": 16,
124
+ "o2": 16,
125
+ "o3": 16,
126
+ "i1": 16,
127
+ "i2": 16,
128
+ "i3": 16,
129
+ "r0": 1,
130
+ "r1": 186,
131
+ "r2": 186,
132
+ "nnz": 838860
133
+ },
134
+ "model.layers.1.mlp.gate_proj": {
135
+ "o1": 16,
136
+ "o2": 16,
137
+ "o3": 43,
138
+ "i1": 16,
139
+ "i2": 16,
140
+ "i3": 16,
141
+ "r0": 1,
142
+ "r1": 256,
143
+ "r2": 359,
144
+ "nnz": 2254438
145
+ },
146
+ "model.layers.1.mlp.up_proj": {
147
+ "o1": 16,
148
+ "o2": 16,
149
+ "o3": 43,
150
+ "i1": 16,
151
+ "i2": 16,
152
+ "i3": 16,
153
+ "r0": 1,
154
+ "r1": 256,
155
+ "r2": 359,
156
+ "nnz": 2254438
157
+ },
158
+ "model.layers.1.mlp.down_proj": {
159
+ "o1": 16,
160
+ "o2": 16,
161
+ "o3": 16,
162
+ "i1": 16,
163
+ "i2": 16,
164
+ "i3": 43,
165
+ "r0": 1,
166
+ "r1": 256,
167
+ "r2": 359,
168
+ "nnz": 2254438
169
+ },
170
+ "model.layers.2.self_attn.q_proj": {
171
+ "o1": 16,
172
+ "o2": 16,
173
+ "o3": 16,
174
+ "i1": 16,
175
+ "i2": 16,
176
+ "i3": 16,
177
+ "r0": 1,
178
+ "r1": 186,
179
+ "r2": 186,
180
+ "nnz": 838860
181
+ },
182
+ "model.layers.2.self_attn.k_proj": {
183
+ "o1": 16,
184
+ "o2": 16,
185
+ "o3": 16,
186
+ "i1": 16,
187
+ "i2": 16,
188
+ "i3": 16,
189
+ "r0": 1,
190
+ "r1": 186,
191
+ "r2": 186,
192
+ "nnz": 838860
193
+ },
194
+ "model.layers.2.self_attn.v_proj": {
195
+ "o1": 16,
196
+ "o2": 16,
197
+ "o3": 16,
198
+ "i1": 16,
199
+ "i2": 16,
200
+ "i3": 16,
201
+ "r0": 1,
202
+ "r1": 186,
203
+ "r2": 186,
204
+ "nnz": 838860
205
+ },
206
+ "model.layers.2.self_attn.o_proj": {
207
+ "o1": 16,
208
+ "o2": 16,
209
+ "o3": 16,
210
+ "i1": 16,
211
+ "i2": 16,
212
+ "i3": 16,
213
+ "r0": 1,
214
+ "r1": 186,
215
+ "r2": 186,
216
+ "nnz": 838860
217
+ },
218
+ "model.layers.2.mlp.gate_proj": {
219
+ "o1": 16,
220
+ "o2": 16,
221
+ "o3": 43,
222
+ "i1": 16,
223
+ "i2": 16,
224
+ "i3": 16,
225
+ "r0": 1,
226
+ "r1": 256,
227
+ "r2": 359,
228
+ "nnz": 2254438
229
+ },
230
+ "model.layers.2.mlp.up_proj": {
231
+ "o1": 16,
232
+ "o2": 16,
233
+ "o3": 43,
234
+ "i1": 16,
235
+ "i2": 16,
236
+ "i3": 16,
237
+ "r0": 1,
238
+ "r1": 256,
239
+ "r2": 359,
240
+ "nnz": 2254438
241
+ },
242
+ "model.layers.2.mlp.down_proj": {
243
+ "o1": 16,
244
+ "o2": 16,
245
+ "o3": 16,
246
+ "i1": 16,
247
+ "i2": 16,
248
+ "i3": 43,
249
+ "r0": 1,
250
+ "r1": 256,
251
+ "r2": 359,
252
+ "nnz": 2254438
253
+ },
254
+ "model.layers.3.self_attn.q_proj": {
255
+ "o1": 16,
256
+ "o2": 16,
257
+ "o3": 16,
258
+ "i1": 16,
259
+ "i2": 16,
260
+ "i3": 16,
261
+ "r0": 1,
262
+ "r1": 186,
263
+ "r2": 186,
264
+ "nnz": 838860
265
+ },
266
+ "model.layers.3.self_attn.k_proj": {
267
+ "o1": 16,
268
+ "o2": 16,
269
+ "o3": 16,
270
+ "i1": 16,
271
+ "i2": 16,
272
+ "i3": 16,
273
+ "r0": 1,
274
+ "r1": 186,
275
+ "r2": 186,
276
+ "nnz": 838860
277
+ },
278
+ "model.layers.3.self_attn.v_proj": {
279
+ "o1": 16,
280
+ "o2": 16,
281
+ "o3": 16,
282
+ "i1": 16,
283
+ "i2": 16,
284
+ "i3": 16,
285
+ "r0": 1,
286
+ "r1": 186,
287
+ "r2": 186,
288
+ "nnz": 838860
289
+ },
290
+ "model.layers.3.self_attn.o_proj": {
291
+ "o1": 16,
292
+ "o2": 16,
293
+ "o3": 16,
294
+ "i1": 16,
295
+ "i2": 16,
296
+ "i3": 16,
297
+ "r0": 1,
298
+ "r1": 186,
299
+ "r2": 186,
300
+ "nnz": 838860
301
+ },
302
+ "model.layers.3.mlp.gate_proj": {
303
+ "o1": 16,
304
+ "o2": 16,
305
+ "o3": 43,
306
+ "i1": 16,
307
+ "i2": 16,
308
+ "i3": 16,
309
+ "r0": 1,
310
+ "r1": 256,
311
+ "r2": 359,
312
+ "nnz": 2254438
313
+ },
314
+ "model.layers.3.mlp.up_proj": {
315
+ "o1": 16,
316
+ "o2": 16,
317
+ "o3": 43,
318
+ "i1": 16,
319
+ "i2": 16,
320
+ "i3": 16,
321
+ "r0": 1,
322
+ "r1": 256,
323
+ "r2": 359,
324
+ "nnz": 2254438
325
+ },
326
+ "model.layers.3.mlp.down_proj": {
327
+ "o1": 16,
328
+ "o2": 16,
329
+ "o3": 16,
330
+ "i1": 16,
331
+ "i2": 16,
332
+ "i3": 43,
333
+ "r0": 1,
334
+ "r1": 256,
335
+ "r2": 359,
336
+ "nnz": 2254438
337
+ },
338
+ "model.layers.4.self_attn.q_proj": {
339
+ "o1": 16,
340
+ "o2": 16,
341
+ "o3": 16,
342
+ "i1": 16,
343
+ "i2": 16,
344
+ "i3": 16,
345
+ "r0": 1,
346
+ "r1": 186,
347
+ "r2": 186,
348
+ "nnz": 838860
349
+ },
350
+ "model.layers.4.self_attn.k_proj": {
351
+ "o1": 16,
352
+ "o2": 16,
353
+ "o3": 16,
354
+ "i1": 16,
355
+ "i2": 16,
356
+ "i3": 16,
357
+ "r0": 1,
358
+ "r1": 186,
359
+ "r2": 186,
360
+ "nnz": 838860
361
+ },
362
+ "model.layers.4.self_attn.v_proj": {
363
+ "o1": 16,
364
+ "o2": 16,
365
+ "o3": 16,
366
+ "i1": 16,
367
+ "i2": 16,
368
+ "i3": 16,
369
+ "r0": 1,
370
+ "r1": 186,
371
+ "r2": 186,
372
+ "nnz": 838860
373
+ },
374
+ "model.layers.4.self_attn.o_proj": {
375
+ "o1": 16,
376
+ "o2": 16,
377
+ "o3": 16,
378
+ "i1": 16,
379
+ "i2": 16,
380
+ "i3": 16,
381
+ "r0": 1,
382
+ "r1": 186,
383
+ "r2": 186,
384
+ "nnz": 838860
385
+ },
386
+ "model.layers.4.mlp.gate_proj": {
387
+ "o1": 16,
388
+ "o2": 16,
389
+ "o3": 43,
390
+ "i1": 16,
391
+ "i2": 16,
392
+ "i3": 16,
393
+ "r0": 1,
394
+ "r1": 256,
395
+ "r2": 359,
396
+ "nnz": 2254438
397
+ },
398
+ "model.layers.4.mlp.up_proj": {
399
+ "o1": 16,
400
+ "o2": 16,
401
+ "o3": 43,
402
+ "i1": 16,
403
+ "i2": 16,
404
+ "i3": 16,
405
+ "r0": 1,
406
+ "r1": 256,
407
+ "r2": 359,
408
+ "nnz": 2254438
409
+ },
410
+ "model.layers.4.mlp.down_proj": {
411
+ "o1": 16,
412
+ "o2": 16,
413
+ "o3": 16,
414
+ "i1": 16,
415
+ "i2": 16,
416
+ "i3": 43,
417
+ "r0": 1,
418
+ "r1": 256,
419
+ "r2": 359,
420
+ "nnz": 2254438
421
+ },
422
+ "model.layers.5.self_attn.q_proj": {
423
+ "o1": 16,
424
+ "o2": 16,
425
+ "o3": 16,
426
+ "i1": 16,
427
+ "i2": 16,
428
+ "i3": 16,
429
+ "r0": 1,
430
+ "r1": 186,
431
+ "r2": 186,
432
+ "nnz": 838860
433
+ },
434
+ "model.layers.5.self_attn.k_proj": {
435
+ "o1": 16,
436
+ "o2": 16,
437
+ "o3": 16,
438
+ "i1": 16,
439
+ "i2": 16,
440
+ "i3": 16,
441
+ "r0": 1,
442
+ "r1": 186,
443
+ "r2": 186,
444
+ "nnz": 838860
445
+ },
446
+ "model.layers.5.self_attn.v_proj": {
447
+ "o1": 16,
448
+ "o2": 16,
449
+ "o3": 16,
450
+ "i1": 16,
451
+ "i2": 16,
452
+ "i3": 16,
453
+ "r0": 1,
454
+ "r1": 186,
455
+ "r2": 186,
456
+ "nnz": 838860
457
+ },
458
+ "model.layers.5.self_attn.o_proj": {
459
+ "o1": 16,
460
+ "o2": 16,
461
+ "o3": 16,
462
+ "i1": 16,
463
+ "i2": 16,
464
+ "i3": 16,
465
+ "r0": 1,
466
+ "r1": 186,
467
+ "r2": 186,
468
+ "nnz": 838860
469
+ },
470
+ "model.layers.5.mlp.gate_proj": {
471
+ "o1": 16,
472
+ "o2": 16,
473
+ "o3": 43,
474
+ "i1": 16,
475
+ "i2": 16,
476
+ "i3": 16,
477
+ "r0": 1,
478
+ "r1": 256,
479
+ "r2": 359,
480
+ "nnz": 2254438
481
+ },
482
+ "model.layers.5.mlp.up_proj": {
483
+ "o1": 16,
484
+ "o2": 16,
485
+ "o3": 43,
486
+ "i1": 16,
487
+ "i2": 16,
488
+ "i3": 16,
489
+ "r0": 1,
490
+ "r1": 256,
491
+ "r2": 359,
492
+ "nnz": 2254438
493
+ },
494
+ "model.layers.5.mlp.down_proj": {
495
+ "o1": 16,
496
+ "o2": 16,
497
+ "o3": 16,
498
+ "i1": 16,
499
+ "i2": 16,
500
+ "i3": 43,
501
+ "r0": 1,
502
+ "r1": 256,
503
+ "r2": 359,
504
+ "nnz": 2254438
505
+ },
506
+ "model.layers.6.self_attn.q_proj": {
507
+ "o1": 16,
508
+ "o2": 16,
509
+ "o3": 16,
510
+ "i1": 16,
511
+ "i2": 16,
512
+ "i3": 16,
513
+ "r0": 1,
514
+ "r1": 186,
515
+ "r2": 186,
516
+ "nnz": 838860
517
+ },
518
+ "model.layers.6.self_attn.k_proj": {
519
+ "o1": 16,
520
+ "o2": 16,
521
+ "o3": 16,
522
+ "i1": 16,
523
+ "i2": 16,
524
+ "i3": 16,
525
+ "r0": 1,
526
+ "r1": 186,
527
+ "r2": 186,
528
+ "nnz": 838860
529
+ },
530
+ "model.layers.6.self_attn.v_proj": {
531
+ "o1": 16,
532
+ "o2": 16,
533
+ "o3": 16,
534
+ "i1": 16,
535
+ "i2": 16,
536
+ "i3": 16,
537
+ "r0": 1,
538
+ "r1": 186,
539
+ "r2": 186,
540
+ "nnz": 838860
541
+ },
542
+ "model.layers.6.self_attn.o_proj": {
543
+ "o1": 16,
544
+ "o2": 16,
545
+ "o3": 16,
546
+ "i1": 16,
547
+ "i2": 16,
548
+ "i3": 16,
549
+ "r0": 1,
550
+ "r1": 186,
551
+ "r2": 186,
552
+ "nnz": 838860
553
+ },
554
+ "model.layers.6.mlp.gate_proj": {
555
+ "o1": 16,
556
+ "o2": 16,
557
+ "o3": 43,
558
+ "i1": 16,
559
+ "i2": 16,
560
+ "i3": 16,
561
+ "r0": 1,
562
+ "r1": 256,
563
+ "r2": 359,
564
+ "nnz": 2254438
565
+ },
566
+ "model.layers.6.mlp.up_proj": {
567
+ "o1": 16,
568
+ "o2": 16,
569
+ "o3": 43,
570
+ "i1": 16,
571
+ "i2": 16,
572
+ "i3": 16,
573
+ "r0": 1,
574
+ "r1": 256,
575
+ "r2": 359,
576
+ "nnz": 2254438
577
+ },
578
+ "model.layers.6.mlp.down_proj": {
579
+ "o1": 16,
580
+ "o2": 16,
581
+ "o3": 16,
582
+ "i1": 16,
583
+ "i2": 16,
584
+ "i3": 43,
585
+ "r0": 1,
586
+ "r1": 256,
587
+ "r2": 359,
588
+ "nnz": 2254438
589
+ },
590
+ "model.layers.7.self_attn.q_proj": {
591
+ "o1": 16,
592
+ "o2": 16,
593
+ "o3": 16,
594
+ "i1": 16,
595
+ "i2": 16,
596
+ "i3": 16,
597
+ "r0": 1,
598
+ "r1": 186,
599
+ "r2": 186,
600
+ "nnz": 838860
601
+ },
602
+ "model.layers.7.self_attn.k_proj": {
603
+ "o1": 16,
604
+ "o2": 16,
605
+ "o3": 16,
606
+ "i1": 16,
607
+ "i2": 16,
608
+ "i3": 16,
609
+ "r0": 1,
610
+ "r1": 186,
611
+ "r2": 186,
612
+ "nnz": 838860
613
+ },
614
+ "model.layers.7.self_attn.v_proj": {
615
+ "o1": 16,
616
+ "o2": 16,
617
+ "o3": 16,
618
+ "i1": 16,
619
+ "i2": 16,
620
+ "i3": 16,
621
+ "r0": 1,
622
+ "r1": 186,
623
+ "r2": 186,
624
+ "nnz": 838860
625
+ },
626
+ "model.layers.7.self_attn.o_proj": {
627
+ "o1": 16,
628
+ "o2": 16,
629
+ "o3": 16,
630
+ "i1": 16,
631
+ "i2": 16,
632
+ "i3": 16,
633
+ "r0": 1,
634
+ "r1": 186,
635
+ "r2": 186,
636
+ "nnz": 838860
637
+ },
638
+ "model.layers.7.mlp.gate_proj": {
639
+ "o1": 16,
640
+ "o2": 16,
641
+ "o3": 43,
642
+ "i1": 16,
643
+ "i2": 16,
644
+ "i3": 16,
645
+ "r0": 1,
646
+ "r1": 256,
647
+ "r2": 359,
648
+ "nnz": 2254438
649
+ },
650
+ "model.layers.7.mlp.up_proj": {
651
+ "o1": 16,
652
+ "o2": 16,
653
+ "o3": 43,
654
+ "i1": 16,
655
+ "i2": 16,
656
+ "i3": 16,
657
+ "r0": 1,
658
+ "r1": 256,
659
+ "r2": 359,
660
+ "nnz": 2254438
661
+ },
662
+ "model.layers.7.mlp.down_proj": {
663
+ "o1": 16,
664
+ "o2": 16,
665
+ "o3": 16,
666
+ "i1": 16,
667
+ "i2": 16,
668
+ "i3": 43,
669
+ "r0": 1,
670
+ "r1": 256,
671
+ "r2": 359,
672
+ "nnz": 2254438
673
+ },
674
+ "model.layers.8.self_attn.q_proj": {
675
+ "o1": 16,
676
+ "o2": 16,
677
+ "o3": 16,
678
+ "i1": 16,
679
+ "i2": 16,
680
+ "i3": 16,
681
+ "r0": 1,
682
+ "r1": 186,
683
+ "r2": 186,
684
+ "nnz": 838860
685
+ },
686
+ "model.layers.8.self_attn.k_proj": {
687
+ "o1": 16,
688
+ "o2": 16,
689
+ "o3": 16,
690
+ "i1": 16,
691
+ "i2": 16,
692
+ "i3": 16,
693
+ "r0": 1,
694
+ "r1": 186,
695
+ "r2": 186,
696
+ "nnz": 838860
697
+ },
698
+ "model.layers.8.self_attn.v_proj": {
699
+ "o1": 16,
700
+ "o2": 16,
701
+ "o3": 16,
702
+ "i1": 16,
703
+ "i2": 16,
704
+ "i3": 16,
705
+ "r0": 1,
706
+ "r1": 186,
707
+ "r2": 186,
708
+ "nnz": 838860
709
+ },
710
+ "model.layers.8.self_attn.o_proj": {
711
+ "o1": 16,
712
+ "o2": 16,
713
+ "o3": 16,
714
+ "i1": 16,
715
+ "i2": 16,
716
+ "i3": 16,
717
+ "r0": 1,
718
+ "r1": 186,
719
+ "r2": 186,
720
+ "nnz": 838860
721
+ },
722
+ "model.layers.8.mlp.gate_proj": {
723
+ "o1": 16,
724
+ "o2": 16,
725
+ "o3": 43,
726
+ "i1": 16,
727
+ "i2": 16,
728
+ "i3": 16,
729
+ "r0": 1,
730
+ "r1": 256,
731
+ "r2": 359,
732
+ "nnz": 2254438
733
+ },
734
+ "model.layers.8.mlp.up_proj": {
735
+ "o1": 16,
736
+ "o2": 16,
737
+ "o3": 43,
738
+ "i1": 16,
739
+ "i2": 16,
740
+ "i3": 16,
741
+ "r0": 1,
742
+ "r1": 256,
743
+ "r2": 359,
744
+ "nnz": 2254438
745
+ },
746
+ "model.layers.8.mlp.down_proj": {
747
+ "o1": 16,
748
+ "o2": 16,
749
+ "o3": 16,
750
+ "i1": 16,
751
+ "i2": 16,
752
+ "i3": 43,
753
+ "r0": 1,
754
+ "r1": 256,
755
+ "r2": 359,
756
+ "nnz": 2254438
757
+ },
758
+ "model.layers.9.self_attn.q_proj": {
759
+ "o1": 16,
760
+ "o2": 16,
761
+ "o3": 16,
762
+ "i1": 16,
763
+ "i2": 16,
764
+ "i3": 16,
765
+ "r0": 1,
766
+ "r1": 186,
767
+ "r2": 186,
768
+ "nnz": 838860
769
+ },
770
+ "model.layers.9.self_attn.k_proj": {
771
+ "o1": 16,
772
+ "o2": 16,
773
+ "o3": 16,
774
+ "i1": 16,
775
+ "i2": 16,
776
+ "i3": 16,
777
+ "r0": 1,
778
+ "r1": 186,
779
+ "r2": 186,
780
+ "nnz": 838860
781
+ },
782
+ "model.layers.9.self_attn.v_proj": {
783
+ "o1": 16,
784
+ "o2": 16,
785
+ "o3": 16,
786
+ "i1": 16,
787
+ "i2": 16,
788
+ "i3": 16,
789
+ "r0": 1,
790
+ "r1": 186,
791
+ "r2": 186,
792
+ "nnz": 838860
793
+ },
794
+ "model.layers.9.self_attn.o_proj": {
795
+ "o1": 16,
796
+ "o2": 16,
797
+ "o3": 16,
798
+ "i1": 16,
799
+ "i2": 16,
800
+ "i3": 16,
801
+ "r0": 1,
802
+ "r1": 186,
803
+ "r2": 186,
804
+ "nnz": 838860
805
+ },
806
+ "model.layers.9.mlp.gate_proj": {
807
+ "o1": 16,
808
+ "o2": 16,
809
+ "o3": 43,
810
+ "i1": 16,
811
+ "i2": 16,
812
+ "i3": 16,
813
+ "r0": 1,
814
+ "r1": 256,
815
+ "r2": 359,
816
+ "nnz": 2254438
817
+ },
818
+ "model.layers.9.mlp.up_proj": {
819
+ "o1": 16,
820
+ "o2": 16,
821
+ "o3": 43,
822
+ "i1": 16,
823
+ "i2": 16,
824
+ "i3": 16,
825
+ "r0": 1,
826
+ "r1": 256,
827
+ "r2": 359,
828
+ "nnz": 2254438
829
+ },
830
+ "model.layers.9.mlp.down_proj": {
831
+ "o1": 16,
832
+ "o2": 16,
833
+ "o3": 16,
834
+ "i1": 16,
835
+ "i2": 16,
836
+ "i3": 43,
837
+ "r0": 1,
838
+ "r1": 256,
839
+ "r2": 359,
840
+ "nnz": 2254438
841
+ },
842
+ "model.layers.10.self_attn.q_proj": {
843
+ "o1": 16,
844
+ "o2": 16,
845
+ "o3": 16,
846
+ "i1": 16,
847
+ "i2": 16,
848
+ "i3": 16,
849
+ "r0": 1,
850
+ "r1": 186,
851
+ "r2": 186,
852
+ "nnz": 838860
853
+ },
854
+ "model.layers.10.self_attn.k_proj": {
855
+ "o1": 16,
856
+ "o2": 16,
857
+ "o3": 16,
858
+ "i1": 16,
859
+ "i2": 16,
860
+ "i3": 16,
861
+ "r0": 1,
862
+ "r1": 186,
863
+ "r2": 186,
864
+ "nnz": 838860
865
+ },
866
+ "model.layers.10.self_attn.v_proj": {
867
+ "o1": 16,
868
+ "o2": 16,
869
+ "o3": 16,
870
+ "i1": 16,
871
+ "i2": 16,
872
+ "i3": 16,
873
+ "r0": 1,
874
+ "r1": 186,
875
+ "r2": 186,
876
+ "nnz": 838860
877
+ },
878
+ "model.layers.10.self_attn.o_proj": {
879
+ "o1": 16,
880
+ "o2": 16,
881
+ "o3": 16,
882
+ "i1": 16,
883
+ "i2": 16,
884
+ "i3": 16,
885
+ "r0": 1,
886
+ "r1": 186,
887
+ "r2": 186,
888
+ "nnz": 838860
889
+ },
890
+ "model.layers.10.mlp.gate_proj": {
891
+ "o1": 16,
892
+ "o2": 16,
893
+ "o3": 43,
894
+ "i1": 16,
895
+ "i2": 16,
896
+ "i3": 16,
897
+ "r0": 1,
898
+ "r1": 256,
899
+ "r2": 359,
900
+ "nnz": 2254438
901
+ },
902
+ "model.layers.10.mlp.up_proj": {
903
+ "o1": 16,
904
+ "o2": 16,
905
+ "o3": 43,
906
+ "i1": 16,
907
+ "i2": 16,
908
+ "i3": 16,
909
+ "r0": 1,
910
+ "r1": 256,
911
+ "r2": 359,
912
+ "nnz": 2254438
913
+ },
914
+ "model.layers.10.mlp.down_proj": {
915
+ "o1": 16,
916
+ "o2": 16,
917
+ "o3": 16,
918
+ "i1": 16,
919
+ "i2": 16,
920
+ "i3": 43,
921
+ "r0": 1,
922
+ "r1": 256,
923
+ "r2": 359,
924
+ "nnz": 2254438
925
+ },
926
+ "model.layers.11.self_attn.q_proj": {
927
+ "o1": 16,
928
+ "o2": 16,
929
+ "o3": 16,
930
+ "i1": 16,
931
+ "i2": 16,
932
+ "i3": 16,
933
+ "r0": 1,
934
+ "r1": 186,
935
+ "r2": 186,
936
+ "nnz": 838860
937
+ },
938
+ "model.layers.11.self_attn.k_proj": {
939
+ "o1": 16,
940
+ "o2": 16,
941
+ "o3": 16,
942
+ "i1": 16,
943
+ "i2": 16,
944
+ "i3": 16,
945
+ "r0": 1,
946
+ "r1": 186,
947
+ "r2": 186,
948
+ "nnz": 838860
949
+ },
950
+ "model.layers.11.self_attn.v_proj": {
951
+ "o1": 16,
952
+ "o2": 16,
953
+ "o3": 16,
954
+ "i1": 16,
955
+ "i2": 16,
956
+ "i3": 16,
957
+ "r0": 1,
958
+ "r1": 186,
959
+ "r2": 186,
960
+ "nnz": 838860
961
+ },
962
+ "model.layers.11.self_attn.o_proj": {
963
+ "o1": 16,
964
+ "o2": 16,
965
+ "o3": 16,
966
+ "i1": 16,
967
+ "i2": 16,
968
+ "i3": 16,
969
+ "r0": 1,
970
+ "r1": 186,
971
+ "r2": 186,
972
+ "nnz": 838860
973
+ },
974
+ "model.layers.11.mlp.gate_proj": {
975
+ "o1": 16,
976
+ "o2": 16,
977
+ "o3": 43,
978
+ "i1": 16,
979
+ "i2": 16,
980
+ "i3": 16,
981
+ "r0": 1,
982
+ "r1": 256,
983
+ "r2": 359,
984
+ "nnz": 2254438
985
+ },
986
+ "model.layers.11.mlp.up_proj": {
987
+ "o1": 16,
988
+ "o2": 16,
989
+ "o3": 43,
990
+ "i1": 16,
991
+ "i2": 16,
992
+ "i3": 16,
993
+ "r0": 1,
994
+ "r1": 256,
995
+ "r2": 359,
996
+ "nnz": 2254438
997
+ },
998
+ "model.layers.11.mlp.down_proj": {
999
+ "o1": 16,
1000
+ "o2": 16,
1001
+ "o3": 16,
1002
+ "i1": 16,
1003
+ "i2": 16,
1004
+ "i3": 43,
1005
+ "r0": 1,
1006
+ "r1": 256,
1007
+ "r2": 359,
1008
+ "nnz": 2254438
1009
+ },
1010
+ "model.layers.12.self_attn.q_proj": {
1011
+ "o1": 16,
1012
+ "o2": 16,
1013
+ "o3": 16,
1014
+ "i1": 16,
1015
+ "i2": 16,
1016
+ "i3": 16,
1017
+ "r0": 1,
1018
+ "r1": 186,
1019
+ "r2": 186,
1020
+ "nnz": 838860
1021
+ },
1022
+ "model.layers.12.self_attn.k_proj": {
1023
+ "o1": 16,
1024
+ "o2": 16,
1025
+ "o3": 16,
1026
+ "i1": 16,
1027
+ "i2": 16,
1028
+ "i3": 16,
1029
+ "r0": 1,
1030
+ "r1": 186,
1031
+ "r2": 186,
1032
+ "nnz": 838860
1033
+ },
1034
+ "model.layers.12.self_attn.v_proj": {
1035
+ "o1": 16,
1036
+ "o2": 16,
1037
+ "o3": 16,
1038
+ "i1": 16,
1039
+ "i2": 16,
1040
+ "i3": 16,
1041
+ "r0": 1,
1042
+ "r1": 186,
1043
+ "r2": 186,
1044
+ "nnz": 838860
1045
+ },
1046
+ "model.layers.12.self_attn.o_proj": {
1047
+ "o1": 16,
1048
+ "o2": 16,
1049
+ "o3": 16,
1050
+ "i1": 16,
1051
+ "i2": 16,
1052
+ "i3": 16,
1053
+ "r0": 1,
1054
+ "r1": 186,
1055
+ "r2": 186,
1056
+ "nnz": 838860
1057
+ },
1058
+ "model.layers.12.mlp.gate_proj": {
1059
+ "o1": 16,
1060
+ "o2": 16,
1061
+ "o3": 43,
1062
+ "i1": 16,
1063
+ "i2": 16,
1064
+ "i3": 16,
1065
+ "r0": 1,
1066
+ "r1": 256,
1067
+ "r2": 359,
1068
+ "nnz": 2254438
1069
+ },
1070
+ "model.layers.12.mlp.up_proj": {
1071
+ "o1": 16,
1072
+ "o2": 16,
1073
+ "o3": 43,
1074
+ "i1": 16,
1075
+ "i2": 16,
1076
+ "i3": 16,
1077
+ "r0": 1,
1078
+ "r1": 256,
1079
+ "r2": 359,
1080
+ "nnz": 2254438
1081
+ },
1082
+ "model.layers.12.mlp.down_proj": {
1083
+ "o1": 16,
1084
+ "o2": 16,
1085
+ "o3": 16,
1086
+ "i1": 16,
1087
+ "i2": 16,
1088
+ "i3": 43,
1089
+ "r0": 1,
1090
+ "r1": 256,
1091
+ "r2": 359,
1092
+ "nnz": 2254438
1093
+ },
1094
+ "model.layers.13.self_attn.q_proj": {
1095
+ "o1": 16,
1096
+ "o2": 16,
1097
+ "o3": 16,
1098
+ "i1": 16,
1099
+ "i2": 16,
1100
+ "i3": 16,
1101
+ "r0": 1,
1102
+ "r1": 186,
1103
+ "r2": 186,
1104
+ "nnz": 838860
1105
+ },
1106
+ "model.layers.13.self_attn.k_proj": {
1107
+ "o1": 16,
1108
+ "o2": 16,
1109
+ "o3": 16,
1110
+ "i1": 16,
1111
+ "i2": 16,
1112
+ "i3": 16,
1113
+ "r0": 1,
1114
+ "r1": 186,
1115
+ "r2": 186,
1116
+ "nnz": 838860
1117
+ },
1118
+ "model.layers.13.self_attn.v_proj": {
1119
+ "o1": 16,
1120
+ "o2": 16,
1121
+ "o3": 16,
1122
+ "i1": 16,
1123
+ "i2": 16,
1124
+ "i3": 16,
1125
+ "r0": 1,
1126
+ "r1": 186,
1127
+ "r2": 186,
1128
+ "nnz": 838860
1129
+ },
1130
+ "model.layers.13.self_attn.o_proj": {
1131
+ "o1": 16,
1132
+ "o2": 16,
1133
+ "o3": 16,
1134
+ "i1": 16,
1135
+ "i2": 16,
1136
+ "i3": 16,
1137
+ "r0": 1,
1138
+ "r1": 186,
1139
+ "r2": 186,
1140
+ "nnz": 838860
1141
+ },
1142
+ "model.layers.13.mlp.gate_proj": {
1143
+ "o1": 16,
1144
+ "o2": 16,
1145
+ "o3": 43,
1146
+ "i1": 16,
1147
+ "i2": 16,
1148
+ "i3": 16,
1149
+ "r0": 1,
1150
+ "r1": 256,
1151
+ "r2": 359,
1152
+ "nnz": 2254438
1153
+ },
1154
+ "model.layers.13.mlp.up_proj": {
1155
+ "o1": 16,
1156
+ "o2": 16,
1157
+ "o3": 43,
1158
+ "i1": 16,
1159
+ "i2": 16,
1160
+ "i3": 16,
1161
+ "r0": 1,
1162
+ "r1": 256,
1163
+ "r2": 359,
1164
+ "nnz": 2254438
1165
+ },
1166
+ "model.layers.13.mlp.down_proj": {
1167
+ "o1": 16,
1168
+ "o2": 16,
1169
+ "o3": 16,
1170
+ "i1": 16,
1171
+ "i2": 16,
1172
+ "i3": 43,
1173
+ "r0": 1,
1174
+ "r1": 256,
1175
+ "r2": 359,
1176
+ "nnz": 2254438
1177
+ },
1178
+ "model.layers.14.self_attn.q_proj": {
1179
+ "o1": 16,
1180
+ "o2": 16,
1181
+ "o3": 16,
1182
+ "i1": 16,
1183
+ "i2": 16,
1184
+ "i3": 16,
1185
+ "r0": 1,
1186
+ "r1": 186,
1187
+ "r2": 186,
1188
+ "nnz": 838860
1189
+ },
1190
+ "model.layers.14.self_attn.k_proj": {
1191
+ "o1": 16,
1192
+ "o2": 16,
1193
+ "o3": 16,
1194
+ "i1": 16,
1195
+ "i2": 16,
1196
+ "i3": 16,
1197
+ "r0": 1,
1198
+ "r1": 186,
1199
+ "r2": 186,
1200
+ "nnz": 838860
1201
+ },
1202
+ "model.layers.14.self_attn.v_proj": {
1203
+ "o1": 16,
1204
+ "o2": 16,
1205
+ "o3": 16,
1206
+ "i1": 16,
1207
+ "i2": 16,
1208
+ "i3": 16,
1209
+ "r0": 1,
1210
+ "r1": 186,
1211
+ "r2": 186,
1212
+ "nnz": 838860
1213
+ },
1214
+ "model.layers.14.self_attn.o_proj": {
1215
+ "o1": 16,
1216
+ "o2": 16,
1217
+ "o3": 16,
1218
+ "i1": 16,
1219
+ "i2": 16,
1220
+ "i3": 16,
1221
+ "r0": 1,
1222
+ "r1": 186,
1223
+ "r2": 186,
1224
+ "nnz": 838860
1225
+ },
1226
+ "model.layers.14.mlp.gate_proj": {
1227
+ "o1": 16,
1228
+ "o2": 16,
1229
+ "o3": 43,
1230
+ "i1": 16,
1231
+ "i2": 16,
1232
+ "i3": 16,
1233
+ "r0": 1,
1234
+ "r1": 256,
1235
+ "r2": 359,
1236
+ "nnz": 2254438
1237
+ },
1238
+ "model.layers.14.mlp.up_proj": {
1239
+ "o1": 16,
1240
+ "o2": 16,
1241
+ "o3": 43,
1242
+ "i1": 16,
1243
+ "i2": 16,
1244
+ "i3": 16,
1245
+ "r0": 1,
1246
+ "r1": 256,
1247
+ "r2": 359,
1248
+ "nnz": 2254438
1249
+ },
1250
+ "model.layers.14.mlp.down_proj": {
1251
+ "o1": 16,
1252
+ "o2": 16,
1253
+ "o3": 16,
1254
+ "i1": 16,
1255
+ "i2": 16,
1256
+ "i3": 43,
1257
+ "r0": 1,
1258
+ "r1": 256,
1259
+ "r2": 359,
1260
+ "nnz": 2254438
1261
+ },
1262
+ "model.layers.15.self_attn.q_proj": {
1263
+ "o1": 16,
1264
+ "o2": 16,
1265
+ "o3": 16,
1266
+ "i1": 16,
1267
+ "i2": 16,
1268
+ "i3": 16,
1269
+ "r0": 1,
1270
+ "r1": 186,
1271
+ "r2": 186,
1272
+ "nnz": 838860
1273
+ },
1274
+ "model.layers.15.self_attn.k_proj": {
1275
+ "o1": 16,
1276
+ "o2": 16,
1277
+ "o3": 16,
1278
+ "i1": 16,
1279
+ "i2": 16,
1280
+ "i3": 16,
1281
+ "r0": 1,
1282
+ "r1": 186,
1283
+ "r2": 186,
1284
+ "nnz": 838860
1285
+ },
1286
+ "model.layers.15.self_attn.v_proj": {
1287
+ "o1": 16,
1288
+ "o2": 16,
1289
+ "o3": 16,
1290
+ "i1": 16,
1291
+ "i2": 16,
1292
+ "i3": 16,
1293
+ "r0": 1,
1294
+ "r1": 186,
1295
+ "r2": 186,
1296
+ "nnz": 838860
1297
+ },
1298
+ "model.layers.15.self_attn.o_proj": {
1299
+ "o1": 16,
1300
+ "o2": 16,
1301
+ "o3": 16,
1302
+ "i1": 16,
1303
+ "i2": 16,
1304
+ "i3": 16,
1305
+ "r0": 1,
1306
+ "r1": 186,
1307
+ "r2": 186,
1308
+ "nnz": 838860
1309
+ },
1310
+ "model.layers.15.mlp.gate_proj": {
1311
+ "o1": 16,
1312
+ "o2": 16,
1313
+ "o3": 43,
1314
+ "i1": 16,
1315
+ "i2": 16,
1316
+ "i3": 16,
1317
+ "r0": 1,
1318
+ "r1": 256,
1319
+ "r2": 359,
1320
+ "nnz": 2254438
1321
+ },
1322
+ "model.layers.15.mlp.up_proj": {
1323
+ "o1": 16,
1324
+ "o2": 16,
1325
+ "o3": 43,
1326
+ "i1": 16,
1327
+ "i2": 16,
1328
+ "i3": 16,
1329
+ "r0": 1,
1330
+ "r1": 256,
1331
+ "r2": 359,
1332
+ "nnz": 2254438
1333
+ },
1334
+ "model.layers.15.mlp.down_proj": {
1335
+ "o1": 16,
1336
+ "o2": 16,
1337
+ "o3": 16,
1338
+ "i1": 16,
1339
+ "i2": 16,
1340
+ "i3": 43,
1341
+ "r0": 1,
1342
+ "r1": 256,
1343
+ "r2": 359,
1344
+ "nnz": 2254438
1345
+ },
1346
+ "model.layers.16.self_attn.q_proj": {
1347
+ "o1": 16,
1348
+ "o2": 16,
1349
+ "o3": 16,
1350
+ "i1": 16,
1351
+ "i2": 16,
1352
+ "i3": 16,
1353
+ "r0": 1,
1354
+ "r1": 186,
1355
+ "r2": 186,
1356
+ "nnz": 838860
1357
+ },
1358
+ "model.layers.16.self_attn.k_proj": {
1359
+ "o1": 16,
1360
+ "o2": 16,
1361
+ "o3": 16,
1362
+ "i1": 16,
1363
+ "i2": 16,
1364
+ "i3": 16,
1365
+ "r0": 1,
1366
+ "r1": 186,
1367
+ "r2": 186,
1368
+ "nnz": 838860
1369
+ },
1370
+ "model.layers.16.self_attn.v_proj": {
1371
+ "o1": 16,
1372
+ "o2": 16,
1373
+ "o3": 16,
1374
+ "i1": 16,
1375
+ "i2": 16,
1376
+ "i3": 16,
1377
+ "r0": 1,
1378
+ "r1": 186,
1379
+ "r2": 186,
1380
+ "nnz": 838860
1381
+ },
1382
+ "model.layers.16.self_attn.o_proj": {
1383
+ "o1": 16,
1384
+ "o2": 16,
1385
+ "o3": 16,
1386
+ "i1": 16,
1387
+ "i2": 16,
1388
+ "i3": 16,
1389
+ "r0": 1,
1390
+ "r1": 186,
1391
+ "r2": 186,
1392
+ "nnz": 838860
1393
+ },
1394
+ "model.layers.16.mlp.gate_proj": {
1395
+ "o1": 16,
1396
+ "o2": 16,
1397
+ "o3": 43,
1398
+ "i1": 16,
1399
+ "i2": 16,
1400
+ "i3": 16,
1401
+ "r0": 1,
1402
+ "r1": 256,
1403
+ "r2": 359,
1404
+ "nnz": 2254438
1405
+ },
1406
+ "model.layers.16.mlp.up_proj": {
1407
+ "o1": 16,
1408
+ "o2": 16,
1409
+ "o3": 43,
1410
+ "i1": 16,
1411
+ "i2": 16,
1412
+ "i3": 16,
1413
+ "r0": 1,
1414
+ "r1": 256,
1415
+ "r2": 359,
1416
+ "nnz": 2254438
1417
+ },
1418
+ "model.layers.16.mlp.down_proj": {
1419
+ "o1": 16,
1420
+ "o2": 16,
1421
+ "o3": 16,
1422
+ "i1": 16,
1423
+ "i2": 16,
1424
+ "i3": 43,
1425
+ "r0": 1,
1426
+ "r1": 256,
1427
+ "r2": 359,
1428
+ "nnz": 2254438
1429
+ },
1430
+ "model.layers.17.self_attn.q_proj": {
1431
+ "o1": 16,
1432
+ "o2": 16,
1433
+ "o3": 16,
1434
+ "i1": 16,
1435
+ "i2": 16,
1436
+ "i3": 16,
1437
+ "r0": 1,
1438
+ "r1": 186,
1439
+ "r2": 186,
1440
+ "nnz": 838860
1441
+ },
1442
+ "model.layers.17.self_attn.k_proj": {
1443
+ "o1": 16,
1444
+ "o2": 16,
1445
+ "o3": 16,
1446
+ "i1": 16,
1447
+ "i2": 16,
1448
+ "i3": 16,
1449
+ "r0": 1,
1450
+ "r1": 186,
1451
+ "r2": 186,
1452
+ "nnz": 838860
1453
+ },
1454
+ "model.layers.17.self_attn.v_proj": {
1455
+ "o1": 16,
1456
+ "o2": 16,
1457
+ "o3": 16,
1458
+ "i1": 16,
1459
+ "i2": 16,
1460
+ "i3": 16,
1461
+ "r0": 1,
1462
+ "r1": 186,
1463
+ "r2": 186,
1464
+ "nnz": 838860
1465
+ },
1466
+ "model.layers.17.self_attn.o_proj": {
1467
+ "o1": 16,
1468
+ "o2": 16,
1469
+ "o3": 16,
1470
+ "i1": 16,
1471
+ "i2": 16,
1472
+ "i3": 16,
1473
+ "r0": 1,
1474
+ "r1": 186,
1475
+ "r2": 186,
1476
+ "nnz": 838860
1477
+ },
1478
+ "model.layers.17.mlp.gate_proj": {
1479
+ "o1": 16,
1480
+ "o2": 16,
1481
+ "o3": 43,
1482
+ "i1": 16,
1483
+ "i2": 16,
1484
+ "i3": 16,
1485
+ "r0": 1,
1486
+ "r1": 256,
1487
+ "r2": 359,
1488
+ "nnz": 2254438
1489
+ },
1490
+ "model.layers.17.mlp.up_proj": {
1491
+ "o1": 16,
1492
+ "o2": 16,
1493
+ "o3": 43,
1494
+ "i1": 16,
1495
+ "i2": 16,
1496
+ "i3": 16,
1497
+ "r0": 1,
1498
+ "r1": 256,
1499
+ "r2": 359,
1500
+ "nnz": 2254438
1501
+ },
1502
+ "model.layers.17.mlp.down_proj": {
1503
+ "o1": 16,
1504
+ "o2": 16,
1505
+ "o3": 16,
1506
+ "i1": 16,
1507
+ "i2": 16,
1508
+ "i3": 43,
1509
+ "r0": 1,
1510
+ "r1": 256,
1511
+ "r2": 359,
1512
+ "nnz": 2254438
1513
+ },
1514
+ "model.layers.18.self_attn.q_proj": {
1515
+ "o1": 16,
1516
+ "o2": 16,
1517
+ "o3": 16,
1518
+ "i1": 16,
1519
+ "i2": 16,
1520
+ "i3": 16,
1521
+ "r0": 1,
1522
+ "r1": 186,
1523
+ "r2": 186,
1524
+ "nnz": 838860
1525
+ },
1526
+ "model.layers.18.self_attn.k_proj": {
1527
+ "o1": 16,
1528
+ "o2": 16,
1529
+ "o3": 16,
1530
+ "i1": 16,
1531
+ "i2": 16,
1532
+ "i3": 16,
1533
+ "r0": 1,
1534
+ "r1": 186,
1535
+ "r2": 186,
1536
+ "nnz": 838860
1537
+ },
1538
+ "model.layers.18.self_attn.v_proj": {
1539
+ "o1": 16,
1540
+ "o2": 16,
1541
+ "o3": 16,
1542
+ "i1": 16,
1543
+ "i2": 16,
1544
+ "i3": 16,
1545
+ "r0": 1,
1546
+ "r1": 186,
1547
+ "r2": 186,
1548
+ "nnz": 838860
1549
+ },
1550
+ "model.layers.18.self_attn.o_proj": {
1551
+ "o1": 16,
1552
+ "o2": 16,
1553
+ "o3": 16,
1554
+ "i1": 16,
1555
+ "i2": 16,
1556
+ "i3": 16,
1557
+ "r0": 1,
1558
+ "r1": 186,
1559
+ "r2": 186,
1560
+ "nnz": 838860
1561
+ },
1562
+ "model.layers.18.mlp.gate_proj": {
1563
+ "o1": 16,
1564
+ "o2": 16,
1565
+ "o3": 43,
1566
+ "i1": 16,
1567
+ "i2": 16,
1568
+ "i3": 16,
1569
+ "r0": 1,
1570
+ "r1": 256,
1571
+ "r2": 359,
1572
+ "nnz": 2254438
1573
+ },
1574
+ "model.layers.18.mlp.up_proj": {
1575
+ "o1": 16,
1576
+ "o2": 16,
1577
+ "o3": 43,
1578
+ "i1": 16,
1579
+ "i2": 16,
1580
+ "i3": 16,
1581
+ "r0": 1,
1582
+ "r1": 256,
1583
+ "r2": 359,
1584
+ "nnz": 2254438
1585
+ },
1586
+ "model.layers.18.mlp.down_proj": {
1587
+ "o1": 16,
1588
+ "o2": 16,
1589
+ "o3": 16,
1590
+ "i1": 16,
1591
+ "i2": 16,
1592
+ "i3": 43,
1593
+ "r0": 1,
1594
+ "r1": 256,
1595
+ "r2": 359,
1596
+ "nnz": 2254438
1597
+ },
1598
+ "model.layers.19.self_attn.q_proj": {
1599
+ "o1": 16,
1600
+ "o2": 16,
1601
+ "o3": 16,
1602
+ "i1": 16,
1603
+ "i2": 16,
1604
+ "i3": 16,
1605
+ "r0": 1,
1606
+ "r1": 186,
1607
+ "r2": 186,
1608
+ "nnz": 838860
1609
+ },
1610
+ "model.layers.19.self_attn.k_proj": {
1611
+ "o1": 16,
1612
+ "o2": 16,
1613
+ "o3": 16,
1614
+ "i1": 16,
1615
+ "i2": 16,
1616
+ "i3": 16,
1617
+ "r0": 1,
1618
+ "r1": 186,
1619
+ "r2": 186,
1620
+ "nnz": 838860
1621
+ },
1622
+ "model.layers.19.self_attn.v_proj": {
1623
+ "o1": 16,
1624
+ "o2": 16,
1625
+ "o3": 16,
1626
+ "i1": 16,
1627
+ "i2": 16,
1628
+ "i3": 16,
1629
+ "r0": 1,
1630
+ "r1": 186,
1631
+ "r2": 186,
1632
+ "nnz": 838860
1633
+ },
1634
+ "model.layers.19.self_attn.o_proj": {
1635
+ "o1": 16,
1636
+ "o2": 16,
1637
+ "o3": 16,
1638
+ "i1": 16,
1639
+ "i2": 16,
1640
+ "i3": 16,
1641
+ "r0": 1,
1642
+ "r1": 186,
1643
+ "r2": 186,
1644
+ "nnz": 838860
1645
+ },
1646
+ "model.layers.19.mlp.gate_proj": {
1647
+ "o1": 16,
1648
+ "o2": 16,
1649
+ "o3": 43,
1650
+ "i1": 16,
1651
+ "i2": 16,
1652
+ "i3": 16,
1653
+ "r0": 1,
1654
+ "r1": 256,
1655
+ "r2": 359,
1656
+ "nnz": 2254438
1657
+ },
1658
+ "model.layers.19.mlp.up_proj": {
1659
+ "o1": 16,
1660
+ "o2": 16,
1661
+ "o3": 43,
1662
+ "i1": 16,
1663
+ "i2": 16,
1664
+ "i3": 16,
1665
+ "r0": 1,
1666
+ "r1": 256,
1667
+ "r2": 359,
1668
+ "nnz": 2254438
1669
+ },
1670
+ "model.layers.19.mlp.down_proj": {
1671
+ "o1": 16,
1672
+ "o2": 16,
1673
+ "o3": 16,
1674
+ "i1": 16,
1675
+ "i2": 16,
1676
+ "i3": 43,
1677
+ "r0": 1,
1678
+ "r1": 256,
1679
+ "r2": 359,
1680
+ "nnz": 2254438
1681
+ },
1682
+ "model.layers.20.self_attn.q_proj": {
1683
+ "o1": 16,
1684
+ "o2": 16,
1685
+ "o3": 16,
1686
+ "i1": 16,
1687
+ "i2": 16,
1688
+ "i3": 16,
1689
+ "r0": 1,
1690
+ "r1": 186,
1691
+ "r2": 186,
1692
+ "nnz": 838860
1693
+ },
1694
+ "model.layers.20.self_attn.k_proj": {
1695
+ "o1": 16,
1696
+ "o2": 16,
1697
+ "o3": 16,
1698
+ "i1": 16,
1699
+ "i2": 16,
1700
+ "i3": 16,
1701
+ "r0": 1,
1702
+ "r1": 186,
1703
+ "r2": 186,
1704
+ "nnz": 838860
1705
+ },
1706
+ "model.layers.20.self_attn.v_proj": {
1707
+ "o1": 16,
1708
+ "o2": 16,
1709
+ "o3": 16,
1710
+ "i1": 16,
1711
+ "i2": 16,
1712
+ "i3": 16,
1713
+ "r0": 1,
1714
+ "r1": 186,
1715
+ "r2": 186,
1716
+ "nnz": 838860
1717
+ },
1718
+ "model.layers.20.self_attn.o_proj": {
1719
+ "o1": 16,
1720
+ "o2": 16,
1721
+ "o3": 16,
1722
+ "i1": 16,
1723
+ "i2": 16,
1724
+ "i3": 16,
1725
+ "r0": 1,
1726
+ "r1": 186,
1727
+ "r2": 186,
1728
+ "nnz": 838860
1729
+ },
1730
+ "model.layers.20.mlp.gate_proj": {
1731
+ "o1": 16,
1732
+ "o2": 16,
1733
+ "o3": 43,
1734
+ "i1": 16,
1735
+ "i2": 16,
1736
+ "i3": 16,
1737
+ "r0": 1,
1738
+ "r1": 256,
1739
+ "r2": 359,
1740
+ "nnz": 2254438
1741
+ },
1742
+ "model.layers.20.mlp.up_proj": {
1743
+ "o1": 16,
1744
+ "o2": 16,
1745
+ "o3": 43,
1746
+ "i1": 16,
1747
+ "i2": 16,
1748
+ "i3": 16,
1749
+ "r0": 1,
1750
+ "r1": 256,
1751
+ "r2": 359,
1752
+ "nnz": 2254438
1753
+ },
1754
+ "model.layers.20.mlp.down_proj": {
1755
+ "o1": 16,
1756
+ "o2": 16,
1757
+ "o3": 16,
1758
+ "i1": 16,
1759
+ "i2": 16,
1760
+ "i3": 43,
1761
+ "r0": 1,
1762
+ "r1": 256,
1763
+ "r2": 359,
1764
+ "nnz": 2254438
1765
+ },
1766
+ "model.layers.21.self_attn.q_proj": {
1767
+ "o1": 16,
1768
+ "o2": 16,
1769
+ "o3": 16,
1770
+ "i1": 16,
1771
+ "i2": 16,
1772
+ "i3": 16,
1773
+ "r0": 1,
1774
+ "r1": 186,
1775
+ "r2": 186,
1776
+ "nnz": 838860
1777
+ },
1778
+ "model.layers.21.self_attn.k_proj": {
1779
+ "o1": 16,
1780
+ "o2": 16,
1781
+ "o3": 16,
1782
+ "i1": 16,
1783
+ "i2": 16,
1784
+ "i3": 16,
1785
+ "r0": 1,
1786
+ "r1": 186,
1787
+ "r2": 186,
1788
+ "nnz": 838860
1789
+ },
1790
+ "model.layers.21.self_attn.v_proj": {
1791
+ "o1": 16,
1792
+ "o2": 16,
1793
+ "o3": 16,
1794
+ "i1": 16,
1795
+ "i2": 16,
1796
+ "i3": 16,
1797
+ "r0": 1,
1798
+ "r1": 186,
1799
+ "r2": 186,
1800
+ "nnz": 838860
1801
+ },
1802
+ "model.layers.21.self_attn.o_proj": {
1803
+ "o1": 16,
1804
+ "o2": 16,
1805
+ "o3": 16,
1806
+ "i1": 16,
1807
+ "i2": 16,
1808
+ "i3": 16,
1809
+ "r0": 1,
1810
+ "r1": 186,
1811
+ "r2": 186,
1812
+ "nnz": 838860
1813
+ },
1814
+ "model.layers.21.mlp.gate_proj": {
1815
+ "o1": 16,
1816
+ "o2": 16,
1817
+ "o3": 43,
1818
+ "i1": 16,
1819
+ "i2": 16,
1820
+ "i3": 16,
1821
+ "r0": 1,
1822
+ "r1": 256,
1823
+ "r2": 359,
1824
+ "nnz": 2254438
1825
+ },
1826
+ "model.layers.21.mlp.up_proj": {
1827
+ "o1": 16,
1828
+ "o2": 16,
1829
+ "o3": 43,
1830
+ "i1": 16,
1831
+ "i2": 16,
1832
+ "i3": 16,
1833
+ "r0": 1,
1834
+ "r1": 256,
1835
+ "r2": 359,
1836
+ "nnz": 2254438
1837
+ },
1838
+ "model.layers.21.mlp.down_proj": {
1839
+ "o1": 16,
1840
+ "o2": 16,
1841
+ "o3": 16,
1842
+ "i1": 16,
1843
+ "i2": 16,
1844
+ "i3": 43,
1845
+ "r0": 1,
1846
+ "r1": 256,
1847
+ "r2": 359,
1848
+ "nnz": 2254438
1849
+ },
1850
+ "model.layers.22.self_attn.q_proj": {
1851
+ "o1": 16,
1852
+ "o2": 16,
1853
+ "o3": 16,
1854
+ "i1": 16,
1855
+ "i2": 16,
1856
+ "i3": 16,
1857
+ "r0": 1,
1858
+ "r1": 186,
1859
+ "r2": 186,
1860
+ "nnz": 838860
1861
+ },
1862
+ "model.layers.22.self_attn.k_proj": {
1863
+ "o1": 16,
1864
+ "o2": 16,
1865
+ "o3": 16,
1866
+ "i1": 16,
1867
+ "i2": 16,
1868
+ "i3": 16,
1869
+ "r0": 1,
1870
+ "r1": 186,
1871
+ "r2": 186,
1872
+ "nnz": 838860
1873
+ },
1874
+ "model.layers.22.self_attn.v_proj": {
1875
+ "o1": 16,
1876
+ "o2": 16,
1877
+ "o3": 16,
1878
+ "i1": 16,
1879
+ "i2": 16,
1880
+ "i3": 16,
1881
+ "r0": 1,
1882
+ "r1": 186,
1883
+ "r2": 186,
1884
+ "nnz": 838860
1885
+ },
1886
+ "model.layers.22.self_attn.o_proj": {
1887
+ "o1": 16,
1888
+ "o2": 16,
1889
+ "o3": 16,
1890
+ "i1": 16,
1891
+ "i2": 16,
1892
+ "i3": 16,
1893
+ "r0": 1,
1894
+ "r1": 186,
1895
+ "r2": 186,
1896
+ "nnz": 838860
1897
+ },
1898
+ "model.layers.22.mlp.gate_proj": {
1899
+ "o1": 16,
1900
+ "o2": 16,
1901
+ "o3": 43,
1902
+ "i1": 16,
1903
+ "i2": 16,
1904
+ "i3": 16,
1905
+ "r0": 1,
1906
+ "r1": 256,
1907
+ "r2": 359,
1908
+ "nnz": 2254438
1909
+ },
1910
+ "model.layers.22.mlp.up_proj": {
1911
+ "o1": 16,
1912
+ "o2": 16,
1913
+ "o3": 43,
1914
+ "i1": 16,
1915
+ "i2": 16,
1916
+ "i3": 16,
1917
+ "r0": 1,
1918
+ "r1": 256,
1919
+ "r2": 359,
1920
+ "nnz": 2254438
1921
+ },
1922
+ "model.layers.22.mlp.down_proj": {
1923
+ "o1": 16,
1924
+ "o2": 16,
1925
+ "o3": 16,
1926
+ "i1": 16,
1927
+ "i2": 16,
1928
+ "i3": 43,
1929
+ "r0": 1,
1930
+ "r1": 256,
1931
+ "r2": 359,
1932
+ "nnz": 2254438
1933
+ },
1934
+ "model.layers.23.self_attn.q_proj": {
1935
+ "o1": 16,
1936
+ "o2": 16,
1937
+ "o3": 16,
1938
+ "i1": 16,
1939
+ "i2": 16,
1940
+ "i3": 16,
1941
+ "r0": 1,
1942
+ "r1": 186,
1943
+ "r2": 186,
1944
+ "nnz": 838860
1945
+ },
1946
+ "model.layers.23.self_attn.k_proj": {
1947
+ "o1": 16,
1948
+ "o2": 16,
1949
+ "o3": 16,
1950
+ "i1": 16,
1951
+ "i2": 16,
1952
+ "i3": 16,
1953
+ "r0": 1,
1954
+ "r1": 186,
1955
+ "r2": 186,
1956
+ "nnz": 838860
1957
+ },
1958
+ "model.layers.23.self_attn.v_proj": {
1959
+ "o1": 16,
1960
+ "o2": 16,
1961
+ "o3": 16,
1962
+ "i1": 16,
1963
+ "i2": 16,
1964
+ "i3": 16,
1965
+ "r0": 1,
1966
+ "r1": 186,
1967
+ "r2": 186,
1968
+ "nnz": 838860
1969
+ },
1970
+ "model.layers.23.self_attn.o_proj": {
1971
+ "o1": 16,
1972
+ "o2": 16,
1973
+ "o3": 16,
1974
+ "i1": 16,
1975
+ "i2": 16,
1976
+ "i3": 16,
1977
+ "r0": 1,
1978
+ "r1": 186,
1979
+ "r2": 186,
1980
+ "nnz": 838860
1981
+ },
1982
+ "model.layers.23.mlp.gate_proj": {
1983
+ "o1": 16,
1984
+ "o2": 16,
1985
+ "o3": 43,
1986
+ "i1": 16,
1987
+ "i2": 16,
1988
+ "i3": 16,
1989
+ "r0": 1,
1990
+ "r1": 256,
1991
+ "r2": 359,
1992
+ "nnz": 2254438
1993
+ },
1994
+ "model.layers.23.mlp.up_proj": {
1995
+ "o1": 16,
1996
+ "o2": 16,
1997
+ "o3": 43,
1998
+ "i1": 16,
1999
+ "i2": 16,
2000
+ "i3": 16,
2001
+ "r0": 1,
2002
+ "r1": 256,
2003
+ "r2": 359,
2004
+ "nnz": 2254438
2005
+ },
2006
+ "model.layers.23.mlp.down_proj": {
2007
+ "o1": 16,
2008
+ "o2": 16,
2009
+ "o3": 16,
2010
+ "i1": 16,
2011
+ "i2": 16,
2012
+ "i3": 43,
2013
+ "r0": 1,
2014
+ "r1": 256,
2015
+ "r2": 359,
2016
+ "nnz": 2254438
2017
+ },
2018
+ "model.layers.24.self_attn.q_proj": {
2019
+ "o1": 16,
2020
+ "o2": 16,
2021
+ "o3": 16,
2022
+ "i1": 16,
2023
+ "i2": 16,
2024
+ "i3": 16,
2025
+ "r0": 1,
2026
+ "r1": 186,
2027
+ "r2": 186,
2028
+ "nnz": 838860
2029
+ },
2030
+ "model.layers.24.self_attn.k_proj": {
2031
+ "o1": 16,
2032
+ "o2": 16,
2033
+ "o3": 16,
2034
+ "i1": 16,
2035
+ "i2": 16,
2036
+ "i3": 16,
2037
+ "r0": 1,
2038
+ "r1": 186,
2039
+ "r2": 186,
2040
+ "nnz": 838860
2041
+ },
2042
+ "model.layers.24.self_attn.v_proj": {
2043
+ "o1": 16,
2044
+ "o2": 16,
2045
+ "o3": 16,
2046
+ "i1": 16,
2047
+ "i2": 16,
2048
+ "i3": 16,
2049
+ "r0": 1,
2050
+ "r1": 186,
2051
+ "r2": 186,
2052
+ "nnz": 838860
2053
+ },
2054
+ "model.layers.24.self_attn.o_proj": {
2055
+ "o1": 16,
2056
+ "o2": 16,
2057
+ "o3": 16,
2058
+ "i1": 16,
2059
+ "i2": 16,
2060
+ "i3": 16,
2061
+ "r0": 1,
2062
+ "r1": 186,
2063
+ "r2": 186,
2064
+ "nnz": 838860
2065
+ },
2066
+ "model.layers.24.mlp.gate_proj": {
2067
+ "o1": 16,
2068
+ "o2": 16,
2069
+ "o3": 43,
2070
+ "i1": 16,
2071
+ "i2": 16,
2072
+ "i3": 16,
2073
+ "r0": 1,
2074
+ "r1": 256,
2075
+ "r2": 359,
2076
+ "nnz": 2254438
2077
+ },
2078
+ "model.layers.24.mlp.up_proj": {
2079
+ "o1": 16,
2080
+ "o2": 16,
2081
+ "o3": 43,
2082
+ "i1": 16,
2083
+ "i2": 16,
2084
+ "i3": 16,
2085
+ "r0": 1,
2086
+ "r1": 256,
2087
+ "r2": 359,
2088
+ "nnz": 2254438
2089
+ },
2090
+ "model.layers.24.mlp.down_proj": {
2091
+ "o1": 16,
2092
+ "o2": 16,
2093
+ "o3": 16,
2094
+ "i1": 16,
2095
+ "i2": 16,
2096
+ "i3": 43,
2097
+ "r0": 1,
2098
+ "r1": 256,
2099
+ "r2": 359,
2100
+ "nnz": 2254438
2101
+ },
2102
+ "model.layers.25.self_attn.q_proj": {
2103
+ "o1": 16,
2104
+ "o2": 16,
2105
+ "o3": 16,
2106
+ "i1": 16,
2107
+ "i2": 16,
2108
+ "i3": 16,
2109
+ "r0": 1,
2110
+ "r1": 186,
2111
+ "r2": 186,
2112
+ "nnz": 838860
2113
+ },
2114
+ "model.layers.25.self_attn.k_proj": {
2115
+ "o1": 16,
2116
+ "o2": 16,
2117
+ "o3": 16,
2118
+ "i1": 16,
2119
+ "i2": 16,
2120
+ "i3": 16,
2121
+ "r0": 1,
2122
+ "r1": 186,
2123
+ "r2": 186,
2124
+ "nnz": 838860
2125
+ },
2126
+ "model.layers.25.self_attn.v_proj": {
2127
+ "o1": 16,
2128
+ "o2": 16,
2129
+ "o3": 16,
2130
+ "i1": 16,
2131
+ "i2": 16,
2132
+ "i3": 16,
2133
+ "r0": 1,
2134
+ "r1": 186,
2135
+ "r2": 186,
2136
+ "nnz": 838860
2137
+ },
2138
+ "model.layers.25.self_attn.o_proj": {
2139
+ "o1": 16,
2140
+ "o2": 16,
2141
+ "o3": 16,
2142
+ "i1": 16,
2143
+ "i2": 16,
2144
+ "i3": 16,
2145
+ "r0": 1,
2146
+ "r1": 186,
2147
+ "r2": 186,
2148
+ "nnz": 838860
2149
+ },
2150
+ "model.layers.25.mlp.gate_proj": {
2151
+ "o1": 16,
2152
+ "o2": 16,
2153
+ "o3": 43,
2154
+ "i1": 16,
2155
+ "i2": 16,
2156
+ "i3": 16,
2157
+ "r0": 1,
2158
+ "r1": 256,
2159
+ "r2": 359,
2160
+ "nnz": 2254438
2161
+ },
2162
+ "model.layers.25.mlp.up_proj": {
2163
+ "o1": 16,
2164
+ "o2": 16,
2165
+ "o3": 43,
2166
+ "i1": 16,
2167
+ "i2": 16,
2168
+ "i3": 16,
2169
+ "r0": 1,
2170
+ "r1": 256,
2171
+ "r2": 359,
2172
+ "nnz": 2254438
2173
+ },
2174
+ "model.layers.25.mlp.down_proj": {
2175
+ "o1": 16,
2176
+ "o2": 16,
2177
+ "o3": 16,
2178
+ "i1": 16,
2179
+ "i2": 16,
2180
+ "i3": 43,
2181
+ "r0": 1,
2182
+ "r1": 256,
2183
+ "r2": 359,
2184
+ "nnz": 2254438
2185
+ },
2186
+ "model.layers.26.self_attn.q_proj": {
2187
+ "o1": 16,
2188
+ "o2": 16,
2189
+ "o3": 16,
2190
+ "i1": 16,
2191
+ "i2": 16,
2192
+ "i3": 16,
2193
+ "r0": 1,
2194
+ "r1": 186,
2195
+ "r2": 186,
2196
+ "nnz": 838860
2197
+ },
2198
+ "model.layers.26.self_attn.k_proj": {
2199
+ "o1": 16,
2200
+ "o2": 16,
2201
+ "o3": 16,
2202
+ "i1": 16,
2203
+ "i2": 16,
2204
+ "i3": 16,
2205
+ "r0": 1,
2206
+ "r1": 186,
2207
+ "r2": 186,
2208
+ "nnz": 838860
2209
+ },
2210
+ "model.layers.26.self_attn.v_proj": {
2211
+ "o1": 16,
2212
+ "o2": 16,
2213
+ "o3": 16,
2214
+ "i1": 16,
2215
+ "i2": 16,
2216
+ "i3": 16,
2217
+ "r0": 1,
2218
+ "r1": 186,
2219
+ "r2": 186,
2220
+ "nnz": 838860
2221
+ },
2222
+ "model.layers.26.self_attn.o_proj": {
2223
+ "o1": 16,
2224
+ "o2": 16,
2225
+ "o3": 16,
2226
+ "i1": 16,
2227
+ "i2": 16,
2228
+ "i3": 16,
2229
+ "r0": 1,
2230
+ "r1": 186,
2231
+ "r2": 186,
2232
+ "nnz": 838860
2233
+ },
2234
+ "model.layers.26.mlp.gate_proj": {
2235
+ "o1": 16,
2236
+ "o2": 16,
2237
+ "o3": 43,
2238
+ "i1": 16,
2239
+ "i2": 16,
2240
+ "i3": 16,
2241
+ "r0": 1,
2242
+ "r1": 256,
2243
+ "r2": 359,
2244
+ "nnz": 2254438
2245
+ },
2246
+ "model.layers.26.mlp.up_proj": {
2247
+ "o1": 16,
2248
+ "o2": 16,
2249
+ "o3": 43,
2250
+ "i1": 16,
2251
+ "i2": 16,
2252
+ "i3": 16,
2253
+ "r0": 1,
2254
+ "r1": 256,
2255
+ "r2": 359,
2256
+ "nnz": 2254438
2257
+ },
2258
+ "model.layers.26.mlp.down_proj": {
2259
+ "o1": 16,
2260
+ "o2": 16,
2261
+ "o3": 16,
2262
+ "i1": 16,
2263
+ "i2": 16,
2264
+ "i3": 43,
2265
+ "r0": 1,
2266
+ "r1": 256,
2267
+ "r2": 359,
2268
+ "nnz": 2254438
2269
+ },
2270
+ "model.layers.27.self_attn.q_proj": {
2271
+ "o1": 16,
2272
+ "o2": 16,
2273
+ "o3": 16,
2274
+ "i1": 16,
2275
+ "i2": 16,
2276
+ "i3": 16,
2277
+ "r0": 1,
2278
+ "r1": 186,
2279
+ "r2": 186,
2280
+ "nnz": 838860
2281
+ },
2282
+ "model.layers.27.self_attn.k_proj": {
2283
+ "o1": 16,
2284
+ "o2": 16,
2285
+ "o3": 16,
2286
+ "i1": 16,
2287
+ "i2": 16,
2288
+ "i3": 16,
2289
+ "r0": 1,
2290
+ "r1": 186,
2291
+ "r2": 186,
2292
+ "nnz": 838860
2293
+ },
2294
+ "model.layers.27.self_attn.v_proj": {
2295
+ "o1": 16,
2296
+ "o2": 16,
2297
+ "o3": 16,
2298
+ "i1": 16,
2299
+ "i2": 16,
2300
+ "i3": 16,
2301
+ "r0": 1,
2302
+ "r1": 186,
2303
+ "r2": 186,
2304
+ "nnz": 838860
2305
+ },
2306
+ "model.layers.27.self_attn.o_proj": {
2307
+ "o1": 16,
2308
+ "o2": 16,
2309
+ "o3": 16,
2310
+ "i1": 16,
2311
+ "i2": 16,
2312
+ "i3": 16,
2313
+ "r0": 1,
2314
+ "r1": 186,
2315
+ "r2": 186,
2316
+ "nnz": 838860
2317
+ },
2318
+ "model.layers.27.mlp.gate_proj": {
2319
+ "o1": 16,
2320
+ "o2": 16,
2321
+ "o3": 43,
2322
+ "i1": 16,
2323
+ "i2": 16,
2324
+ "i3": 16,
2325
+ "r0": 1,
2326
+ "r1": 256,
2327
+ "r2": 359,
2328
+ "nnz": 2254438
2329
+ },
2330
+ "model.layers.27.mlp.up_proj": {
2331
+ "o1": 16,
2332
+ "o2": 16,
2333
+ "o3": 43,
2334
+ "i1": 16,
2335
+ "i2": 16,
2336
+ "i3": 16,
2337
+ "r0": 1,
2338
+ "r1": 256,
2339
+ "r2": 359,
2340
+ "nnz": 2254438
2341
+ },
2342
+ "model.layers.27.mlp.down_proj": {
2343
+ "o1": 16,
2344
+ "o2": 16,
2345
+ "o3": 16,
2346
+ "i1": 16,
2347
+ "i2": 16,
2348
+ "i3": 43,
2349
+ "r0": 1,
2350
+ "r1": 256,
2351
+ "r2": 359,
2352
+ "nnz": 2254438
2353
+ },
2354
+ "model.layers.28.self_attn.q_proj": {
2355
+ "o1": 16,
2356
+ "o2": 16,
2357
+ "o3": 16,
2358
+ "i1": 16,
2359
+ "i2": 16,
2360
+ "i3": 16,
2361
+ "r0": 1,
2362
+ "r1": 186,
2363
+ "r2": 186,
2364
+ "nnz": 838860
2365
+ },
2366
+ "model.layers.28.self_attn.k_proj": {
2367
+ "o1": 16,
2368
+ "o2": 16,
2369
+ "o3": 16,
2370
+ "i1": 16,
2371
+ "i2": 16,
2372
+ "i3": 16,
2373
+ "r0": 1,
2374
+ "r1": 186,
2375
+ "r2": 186,
2376
+ "nnz": 838860
2377
+ },
2378
+ "model.layers.28.self_attn.v_proj": {
2379
+ "o1": 16,
2380
+ "o2": 16,
2381
+ "o3": 16,
2382
+ "i1": 16,
2383
+ "i2": 16,
2384
+ "i3": 16,
2385
+ "r0": 1,
2386
+ "r1": 186,
2387
+ "r2": 186,
2388
+ "nnz": 838860
2389
+ },
2390
+ "model.layers.28.self_attn.o_proj": {
2391
+ "o1": 16,
2392
+ "o2": 16,
2393
+ "o3": 16,
2394
+ "i1": 16,
2395
+ "i2": 16,
2396
+ "i3": 16,
2397
+ "r0": 1,
2398
+ "r1": 186,
2399
+ "r2": 186,
2400
+ "nnz": 838860
2401
+ },
2402
+ "model.layers.28.mlp.gate_proj": {
2403
+ "o1": 16,
2404
+ "o2": 16,
2405
+ "o3": 43,
2406
+ "i1": 16,
2407
+ "i2": 16,
2408
+ "i3": 16,
2409
+ "r0": 1,
2410
+ "r1": 256,
2411
+ "r2": 359,
2412
+ "nnz": 2254438
2413
+ },
2414
+ "model.layers.28.mlp.up_proj": {
2415
+ "o1": 16,
2416
+ "o2": 16,
2417
+ "o3": 43,
2418
+ "i1": 16,
2419
+ "i2": 16,
2420
+ "i3": 16,
2421
+ "r0": 1,
2422
+ "r1": 256,
2423
+ "r2": 359,
2424
+ "nnz": 2254438
2425
+ },
2426
+ "model.layers.28.mlp.down_proj": {
2427
+ "o1": 16,
2428
+ "o2": 16,
2429
+ "o3": 16,
2430
+ "i1": 16,
2431
+ "i2": 16,
2432
+ "i3": 43,
2433
+ "r0": 1,
2434
+ "r1": 256,
2435
+ "r2": 359,
2436
+ "nnz": 2254438
2437
+ },
2438
+ "model.layers.29.self_attn.q_proj": {
2439
+ "o1": 16,
2440
+ "o2": 16,
2441
+ "o3": 16,
2442
+ "i1": 16,
2443
+ "i2": 16,
2444
+ "i3": 16,
2445
+ "r0": 1,
2446
+ "r1": 186,
2447
+ "r2": 186,
2448
+ "nnz": 838860
2449
+ },
2450
+ "model.layers.29.self_attn.k_proj": {
2451
+ "o1": 16,
2452
+ "o2": 16,
2453
+ "o3": 16,
2454
+ "i1": 16,
2455
+ "i2": 16,
2456
+ "i3": 16,
2457
+ "r0": 1,
2458
+ "r1": 186,
2459
+ "r2": 186,
2460
+ "nnz": 838860
2461
+ },
2462
+ "model.layers.29.self_attn.v_proj": {
2463
+ "o1": 16,
2464
+ "o2": 16,
2465
+ "o3": 16,
2466
+ "i1": 16,
2467
+ "i2": 16,
2468
+ "i3": 16,
2469
+ "r0": 1,
2470
+ "r1": 186,
2471
+ "r2": 186,
2472
+ "nnz": 838860
2473
+ },
2474
+ "model.layers.29.self_attn.o_proj": {
2475
+ "o1": 16,
2476
+ "o2": 16,
2477
+ "o3": 16,
2478
+ "i1": 16,
2479
+ "i2": 16,
2480
+ "i3": 16,
2481
+ "r0": 1,
2482
+ "r1": 186,
2483
+ "r2": 186,
2484
+ "nnz": 838860
2485
+ },
2486
+ "model.layers.29.mlp.gate_proj": {
2487
+ "o1": 16,
2488
+ "o2": 16,
2489
+ "o3": 43,
2490
+ "i1": 16,
2491
+ "i2": 16,
2492
+ "i3": 16,
2493
+ "r0": 1,
2494
+ "r1": 256,
2495
+ "r2": 359,
2496
+ "nnz": 2254438
2497
+ },
2498
+ "model.layers.29.mlp.up_proj": {
2499
+ "o1": 16,
2500
+ "o2": 16,
2501
+ "o3": 43,
2502
+ "i1": 16,
2503
+ "i2": 16,
2504
+ "i3": 16,
2505
+ "r0": 1,
2506
+ "r1": 256,
2507
+ "r2": 359,
2508
+ "nnz": 2254438
2509
+ },
2510
+ "model.layers.29.mlp.down_proj": {
2511
+ "o1": 16,
2512
+ "o2": 16,
2513
+ "o3": 16,
2514
+ "i1": 16,
2515
+ "i2": 16,
2516
+ "i3": 43,
2517
+ "r0": 1,
2518
+ "r1": 256,
2519
+ "r2": 359,
2520
+ "nnz": 2254438
2521
+ },
2522
+ "model.layers.30.self_attn.q_proj": {
2523
+ "o1": 16,
2524
+ "o2": 16,
2525
+ "o3": 16,
2526
+ "i1": 16,
2527
+ "i2": 16,
2528
+ "i3": 16,
2529
+ "r0": 1,
2530
+ "r1": 186,
2531
+ "r2": 186,
2532
+ "nnz": 838860
2533
+ },
2534
+ "model.layers.30.self_attn.k_proj": {
2535
+ "o1": 16,
2536
+ "o2": 16,
2537
+ "o3": 16,
2538
+ "i1": 16,
2539
+ "i2": 16,
2540
+ "i3": 16,
2541
+ "r0": 1,
2542
+ "r1": 186,
2543
+ "r2": 186,
2544
+ "nnz": 838860
2545
+ },
2546
+ "model.layers.30.self_attn.v_proj": {
2547
+ "o1": 16,
2548
+ "o2": 16,
2549
+ "o3": 16,
2550
+ "i1": 16,
2551
+ "i2": 16,
2552
+ "i3": 16,
2553
+ "r0": 1,
2554
+ "r1": 186,
2555
+ "r2": 186,
2556
+ "nnz": 838860
2557
+ },
2558
+ "model.layers.30.self_attn.o_proj": {
2559
+ "o1": 16,
2560
+ "o2": 16,
2561
+ "o3": 16,
2562
+ "i1": 16,
2563
+ "i2": 16,
2564
+ "i3": 16,
2565
+ "r0": 1,
2566
+ "r1": 186,
2567
+ "r2": 186,
2568
+ "nnz": 838860
2569
+ },
2570
+ "model.layers.30.mlp.gate_proj": {
2571
+ "o1": 16,
2572
+ "o2": 16,
2573
+ "o3": 43,
2574
+ "i1": 16,
2575
+ "i2": 16,
2576
+ "i3": 16,
2577
+ "r0": 1,
2578
+ "r1": 256,
2579
+ "r2": 359,
2580
+ "nnz": 2254438
2581
+ },
2582
+ "model.layers.30.mlp.up_proj": {
2583
+ "o1": 16,
2584
+ "o2": 16,
2585
+ "o3": 43,
2586
+ "i1": 16,
2587
+ "i2": 16,
2588
+ "i3": 16,
2589
+ "r0": 1,
2590
+ "r1": 256,
2591
+ "r2": 359,
2592
+ "nnz": 2254438
2593
+ },
2594
+ "model.layers.30.mlp.down_proj": {
2595
+ "o1": 16,
2596
+ "o2": 16,
2597
+ "o3": 16,
2598
+ "i1": 16,
2599
+ "i2": 16,
2600
+ "i3": 43,
2601
+ "r0": 1,
2602
+ "r1": 256,
2603
+ "r2": 359,
2604
+ "nnz": 2254438
2605
+ },
2606
+ "model.layers.31.self_attn.q_proj": {
2607
+ "o1": 16,
2608
+ "o2": 16,
2609
+ "o3": 16,
2610
+ "i1": 16,
2611
+ "i2": 16,
2612
+ "i3": 16,
2613
+ "r0": 1,
2614
+ "r1": 186,
2615
+ "r2": 186,
2616
+ "nnz": 838860
2617
+ },
2618
+ "model.layers.31.self_attn.k_proj": {
2619
+ "o1": 16,
2620
+ "o2": 16,
2621
+ "o3": 16,
2622
+ "i1": 16,
2623
+ "i2": 16,
2624
+ "i3": 16,
2625
+ "r0": 1,
2626
+ "r1": 186,
2627
+ "r2": 186,
2628
+ "nnz": 838860
2629
+ },
2630
+ "model.layers.31.self_attn.v_proj": {
2631
+ "o1": 16,
2632
+ "o2": 16,
2633
+ "o3": 16,
2634
+ "i1": 16,
2635
+ "i2": 16,
2636
+ "i3": 16,
2637
+ "r0": 1,
2638
+ "r1": 186,
2639
+ "r2": 186,
2640
+ "nnz": 838860
2641
+ },
2642
+ "model.layers.31.self_attn.o_proj": {
2643
+ "o1": 16,
2644
+ "o2": 16,
2645
+ "o3": 16,
2646
+ "i1": 16,
2647
+ "i2": 16,
2648
+ "i3": 16,
2649
+ "r0": 1,
2650
+ "r1": 186,
2651
+ "r2": 186,
2652
+ "nnz": 838860
2653
+ },
2654
+ "model.layers.31.mlp.gate_proj": {
2655
+ "o1": 16,
2656
+ "o2": 16,
2657
+ "o3": 43,
2658
+ "i1": 16,
2659
+ "i2": 16,
2660
+ "i3": 16,
2661
+ "r0": 1,
2662
+ "r1": 256,
2663
+ "r2": 359,
2664
+ "nnz": 2254438
2665
+ },
2666
+ "model.layers.31.mlp.up_proj": {
2667
+ "o1": 16,
2668
+ "o2": 16,
2669
+ "o3": 43,
2670
+ "i1": 16,
2671
+ "i2": 16,
2672
+ "i3": 16,
2673
+ "r0": 1,
2674
+ "r1": 256,
2675
+ "r2": 359,
2676
+ "nnz": 2254438
2677
+ },
2678
+ "model.layers.31.mlp.down_proj": {
2679
+ "o1": 16,
2680
+ "o2": 16,
2681
+ "o3": 16,
2682
+ "i1": 16,
2683
+ "i2": 16,
2684
+ "i3": 43,
2685
+ "r0": 1,
2686
+ "r1": 256,
2687
+ "r2": 359,
2688
+ "nnz": 2254438
2689
+ }
2690
+ }
tt3/unheal/llama2-7b-base/state_dict_fp16.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:35a1d104ca987db2ba84ad1745bbf808b6fde537e56492abb82814e0d03b4631
3
+ size 10632154592
tt3/unheal/llama2-7b-base/tensor_formats.py ADDED
@@ -0,0 +1,768 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from __future__ import annotations
2
+
3
+ """General N-core tensor formats for a linear layer: tensor train (MPO) and tensor ring.
4
+
5
+ WHY THIS EXISTS. The 2-core and 3-core results say the tensorisation step, not the healing
6
+ step, is where the leverage is: at equal parameters the 3-core beat the 2-core AFTER healing
7
+ at both budgets tried (4.72B: 38.94 vs ~36.5 full MMLU at 60M tokens; 5.74B: 44.91 vs 41.54
8
+ probe), and it got there on roughly a quarter of the tokens. That invites two orthogonal
9
+ questions this module exists to answer: how many cores, and which topology.
10
+
11
+ TENSOR TRAIN vs TENSOR RING. A TT/MPO pins the two boundary bonds to 1 -- the chain has open
12
+ ends. A tensor ring closes the chain: core k has shape (r_{k-1}, d_k, r_k) with r_N == r_0,
13
+ and the weight is the TRACE of the cyclic product. TT is exactly the TR special case r_0 = 1,
14
+ so at equal ranks TR is strictly more expressive. It is also strictly harder to fit: TT-SVD
15
+ is a sequence of Eckart-Young-optimal truncations, while a ring has no such closed form, so
16
+ TR is fitted here by TT initialisation followed by alternating least squares. That asymmetry
17
+ matters when reading results -- a TR loss could mean "the format is worse" OR "the fit did
18
+ not converge", so `fit_tr` reports its per-sweep error curve and refuses to return a fit
19
+ worse than its own TT initialisation.
20
+
21
+ INDEX CONVENTION (shared with mpo2.py / mpo3.py). Output mode k travels with input mode k:
22
+
23
+ R(W)[d_1, ..., d_N] = W[(o_1..o_N), (i_1..i_N)], d_k = (o_k, i_k) flattened
24
+
25
+ Every mode must be > 1. A mode of 1 collapses a site and silently reduces the core count,
26
+ which would make an "N-core" result actually be an (N-1)-core one.
27
+
28
+ WHITENING is diagonal and factorised across the N input modes, as in mpo3.py: the whitening
29
+ is then a mode-local rescaling that survives the fit exactly and un-whitens by division.
30
+ """
31
+
32
+ import torch
33
+
34
+
35
+ # --------------------------------------------------------------------------- rearrangement
36
+ def rearrange_n(W: torch.Tensor, osplit, isplit) -> torch.Tensor:
37
+ """W[(o_1..o_N),(i_1..i_N)] -> R[d_1,...,d_N] with d_k = o_k*i_k."""
38
+ n = len(osplit)
39
+ if len(isplit) != n:
40
+ raise ValueError("output and input must be split into the same number of modes")
41
+ perm = [j for k in range(n) for j in (k, n + k)]
42
+ return (W.reshape(*osplit, *isplit).permute(*perm)
43
+ .reshape(*[o * i for o, i in zip(osplit, isplit)]))
44
+
45
+
46
+ def inv_rearrange_n(R: torch.Tensor, osplit, isplit) -> torch.Tensor:
47
+ """Inverse of `rearrange_n`."""
48
+ n = len(osplit)
49
+ inter = [x for k in range(n) for x in (osplit[k], isplit[k])]
50
+ perm = list(range(0, 2 * n, 2)) + list(range(1, 2 * n, 2))
51
+ o = 1
52
+ i = 1
53
+ for k in range(n):
54
+ o *= osplit[k]
55
+ i *= isplit[k]
56
+ return R.reshape(*inter).permute(*perm).reshape(o, i)
57
+
58
+
59
+ def _apply_scales(R: torch.Tensor, osplit, isplit, scales, invert: bool):
60
+ """Multiply (or divide) R by sqrt(d_k[i_k]) along each input mode, in place of a dense L."""
61
+ n = len(osplit)
62
+ shape = [x for k in range(n) for x in (osplit[k], isplit[k])]
63
+ T = R.reshape(*shape)
64
+ for k in range(n):
65
+ s = scales[k].clamp_min(1e-12).sqrt()
66
+ view = [1] * (2 * n)
67
+ view[2 * k + 1] = isplit[k]
68
+ T = T / s.view(*view) if invert else T * s.view(*view)
69
+ return T.reshape(R.shape)
70
+
71
+
72
+ # --------------------------------------------------------------------------- contraction
73
+ def contract_tt(cores) -> torch.Tensor:
74
+ """Contract TT cores [(1,d1,r1), (r1,d2,r2), ..., (r_{N-1},dN,1)] -> R[d_1,...,d_N]."""
75
+ out = cores[0]
76
+ for g in cores[1:]:
77
+ a, b = out.shape[-1], g.shape[0]
78
+ if a != b:
79
+ raise ValueError(f"TT bond mismatch {a} vs {b}")
80
+ out = torch.tensordot(out, g, dims=([out.dim() - 1], [0]))
81
+ if out.shape[0] != 1 or out.shape[-1] != 1:
82
+ raise ValueError("TT boundary bonds must be 1")
83
+ return out.reshape(*out.shape[1:-1])
84
+
85
+
86
+ def contract_tr(cores, chunk: int = 0) -> torch.Tensor:
87
+ """Contract TR cores [(r0,d1,r1), ..., (r_{N-1},dN,r0)] -> R[d_1,...,d_N].
88
+
89
+ The obvious route -- chain every tensordot, then take the diagonal over the two ring
90
+ indices -- materialises (r_0, d_1..d_N, r_0), i.e. r_0^2 times the size of the ANSWER.
91
+ On one LLaMA attention layer at ring rank 118 that is 934 GB, so it does not merely run
92
+ slowly, it cannot run at all. Folding the last contraction and the trace into a single
93
+ einsum keeps the peak at r_0 * prod(d_1..d_{N-1}) * r_{N-1}, and chunking the first mode
94
+ bounds it further for the wide MLP layers.
95
+ """
96
+ n = len(cores)
97
+ if cores[0].shape[0] != cores[-1].shape[-1]:
98
+ raise ValueError(f"ring does not close: {cores[0].shape[0]} vs {cores[-1].shape[-1]}")
99
+ d0 = cores[0].shape[1]
100
+ if chunk and d0 > chunk:
101
+ parts = []
102
+ for lo in range(0, d0, chunk):
103
+ sub = [cores[0][:, lo:min(lo + chunk, d0), :]] + list(cores[1:])
104
+ parts.append(contract_tr(sub, chunk=0))
105
+ return torch.cat(parts, dim=0)
106
+ out = cores[0]
107
+ for g in cores[1:-1]:
108
+ out = torch.tensordot(out, g, dims=([out.dim() - 1], [0]))
109
+ return torch.einsum("i...k,kdi->...d", out, cores[-1])
110
+
111
+
112
+ # --------------------------------------------------------------------------- fitting
113
+ def fit_tt(R: torch.Tensor, ranks):
114
+ """TT-SVD with the given internal ranks. Singular values split symmetrically at each bond,
115
+ matching fit_mpo2/fit_mpo3's gauge so cores stay on comparable scales (fp16-safe and
116
+ kinder to a blockwise quantizer)."""
117
+ dims = list(R.shape)
118
+ n = len(dims)
119
+ if len(ranks) != n - 1:
120
+ raise ValueError(f"{n} sites need {n-1} internal ranks, got {len(ranks)}")
121
+ cores, rest, rprev = [], R.reshape(dims[0], -1), 1
122
+ for k in range(n - 1):
123
+ M = rest.reshape(rprev * dims[k], -1)
124
+ U, S, Vh = torch.linalg.svd(M, full_matrices=False)
125
+ r = min(int(ranks[k]), S.shape[0])
126
+ q = S[:r].clamp_min(0).sqrt()
127
+ cores.append((U[:, :r] * q).reshape(rprev, dims[k], r))
128
+ rest = q[:, None] * Vh[:r]
129
+ rprev = r
130
+ cores.append(rest.reshape(rprev, dims[-1], 1))
131
+ return cores
132
+
133
+
134
+ def tt_to_tr(cores):
135
+ """Reinterpret TT cores as TR cores with r_0 = 1 (an exact, lossless re-labelling)."""
136
+ return [c.clone() for c in cores]
137
+
138
+
139
+ def _subchain(cores, k):
140
+ """Contract every core except k, in cyclic order, -> E[r_k, D, r_{k-1}] with D = prod_{j!=k} d_j."""
141
+ n = len(cores)
142
+ order = [(k + 1 + t) % n for t in range(n - 1)]
143
+ out = cores[order[0]]
144
+ for j in order[1:]:
145
+ out = torch.tensordot(out, cores[j], dims=([out.dim() - 1], [0]))
146
+ return out
147
+
148
+
149
+ def _subchain_chunk(cores, k, lo, hi):
150
+ """`_subchain` with the FIRST following mode restricted to [lo, hi). The restriction rides
151
+ along the contraction untouched, so concatenating the chunks reproduces `_subchain`."""
152
+ n = len(cores)
153
+ order = [(k + 1 + t) % n for t in range(n - 1)]
154
+ out = cores[order[0]][:, lo:hi, :]
155
+ for j in order[1:]:
156
+ out = torch.tensordot(out, cores[j], dims=([out.dim() - 1], [0]))
157
+ return out
158
+
159
+
160
+ def rank_of(cores, k):
161
+ """r_{k-1} * r_k for core k -- the width of that core's least-squares system."""
162
+ n = len(cores)
163
+ return cores[k].shape[0] * cores[k].shape[2]
164
+
165
+
166
+ def _solve_maybe_cpu(AtA, AtW):
167
+ """Solve AtA X = AtW, falling back to the CPU when the GPU cannot hold the factorisation.
168
+
169
+ The normal-equation matrix is (r_{k-1} r_k) square, i.e. it grows as r^4. On down_proj at
170
+ ring rank 219 that is 47961^2 = 9.2 GB, and the factorisation needs a second copy -- no
171
+ amount of chunking the data helps, because this matrix does not depend on the data size.
172
+ Cholesky is tried first (the Gram is symmetric positive definite after the ridge) and
173
+ needs only one triangular factor; if even that does not fit, the solve moves to the host.
174
+ Slow, but a bounded slowdown beats an unbounded failure on the layers that matter most.
175
+ """
176
+ try:
177
+ L = torch.linalg.cholesky(AtA)
178
+ return torch.cholesky_solve(AtW, L)
179
+ except torch.OutOfMemoryError:
180
+ torch.cuda.empty_cache()
181
+ except Exception:
182
+ pass
183
+ try:
184
+ return torch.linalg.solve(AtA, AtW)
185
+ except torch.OutOfMemoryError:
186
+ torch.cuda.empty_cache()
187
+ dev = AtA.device
188
+ X = torch.linalg.solve(AtA.cpu().double(), AtW.cpu().double())
189
+ return X.to(device=dev, dtype=AtW.dtype)
190
+
191
+
192
+ def tr_rel_err(cores, R: torch.Tensor, chunk_elems: int = 250_000_000) -> float:
193
+ """||contract_tr(cores) - R||_F / ||R||_F, accumulated in chunks.
194
+
195
+ Materialising the reconstruction to subtract it needs the reconstruction AND the einsum's
196
+ intermediate at once; on down_proj at ring rank 155 that intermediate is (155, 64, 688,
197
+ 256) = 7 GB even with the first mode already chunked. Accumulating the squared error one
198
+ slice at a time keeps only one slice live, and the chunk width is derived from an element
199
+ budget rather than a hard-coded constant -- a constant is exactly what silently failed to
200
+ engage when a mode happened to equal it.
201
+ """
202
+ n = len(cores)
203
+ d0 = cores[0].shape[1]
204
+ mid = 1
205
+ for k in range(1, n - 1):
206
+ mid *= cores[k].shape[1]
207
+ per_unit = max(1, mid * cores[0].shape[0] * cores[-1].shape[0])
208
+ step = max(1, min(d0, int(chunk_elems // per_unit)))
209
+ acc = 0.0
210
+ for lo in range(0, d0, step):
211
+ hi = min(lo + step, d0)
212
+ sub = [cores[0][:, lo:hi, :]] + list(cores[1:])
213
+ diff = contract_tr(sub) - R[lo:hi]
214
+ acc += float(diff.pow(2).sum())
215
+ del sub, diff
216
+ return (acc ** 0.5) / float(R.norm())
217
+
218
+
219
+ def fit_tr(R: torch.Tensor, ranks, sweeps: int = 30, tol: float = 1e-8, verbose: bool = False,
220
+ seed: int = 1234, restarts: int = 2, solver: str = "lstsq",
221
+ chunk: int = 0, chunk_elems: int = 250_000_000):
222
+ """Fit a tensor ring by TT initialisation + alternating least squares.
223
+
224
+ ranks = [r_0, r_1, ..., r_{N-1}] with r_N == r_0. The TT init uses r_0 = 1 (a legal ring),
225
+ then each ALS sweep re-solves every core in closed form given the others; the ring's extra
226
+ boundary rank is opened up by padding the initialisation. Because ALS on a ring is not
227
+ guaranteed to improve monotonically from an arbitrary start, the best iterate seen is kept
228
+ and the routine REFUSES to return anything worse than its own TT initialisation -- without
229
+ that guard a fitting failure would be indistinguishable from the format being worse, which
230
+ is precisely the comparison this module exists to make.
231
+ """
232
+ n = len(R.shape)
233
+ if len(ranks) != n:
234
+ raise ValueError(f"a ring over {n} sites needs {n} ranks (r_0..r_{{N-1}})")
235
+ dims = list(R.shape)
236
+ rnorm = float(R.norm())
237
+ rnorm2 = float(R.pow(2).sum())
238
+ last_sq = None
239
+ tt = fit_tt(R, ranks[1:])
240
+ base_err = float((contract_tt(tt) - R).norm() / R.norm())
241
+ r0 = int(ranks[0])
242
+ gen = torch.Generator(device="cpu").manual_seed(seed)
243
+
244
+ def padded_init():
245
+ """TT solution embedded in the ring, with the OPENED bond seeded on BOTH ends.
246
+
247
+ Seeding only one end is not enough and is a silent failure: core 0's least-squares
248
+ environment is built from cores 1..N-1, whose ring index comes from core N-1's
249
+ trailing bond. If that stays zero, the environment vanishes on exactly the components
250
+ the ring adds, every ALS solve returns zero there, and the fit never leaves the TT
251
+ solution -- it looks like "the ring buys nothing" rather than "the ring was never
252
+ initialised".
253
+ """
254
+ out = []
255
+ for k, c in enumerate(tt):
256
+ a, d, b = c.shape
257
+ na = r0 if k == 0 else a
258
+ nb = r0 if k == n - 1 else b
259
+ g = torch.zeros(na, d, nb, dtype=R.dtype, device=R.device)
260
+ g[:a, :, :b] = c
261
+ eps = 0.05 * float(c.norm()) / max(1, c.numel()) ** 0.5
262
+ if k == 0 and na > a:
263
+ g[a:, :, :] = eps * torch.randn(na - a, d, nb, generator=gen,
264
+ dtype=R.dtype).to(R.device)
265
+ if k == n - 1 and nb > b:
266
+ g[:, :, b:] = eps * torch.randn(na, d, nb - b, generator=gen,
267
+ dtype=R.dtype).to(R.device)
268
+ out.append(g)
269
+ return out
270
+
271
+ def random_init(g=None):
272
+ """i.i.d. cores whose ring contraction has the right variance.
273
+
274
+ The scale is derived, not guessed. For cores with i.i.d. N(0, s^2) entries the trace
275
+ of the cyclic product is a sum of prod_k r_k independent products of N Gaussians, so
276
+ Var(W) ~= (prod_k r_k) * s^(2N). Matching that to the target mean square gives
277
+ s = (E[R^2] / prod_k r_k)^(1/(2N)). Getting this wrong by a constant factor per core
278
+ compounds to that factor^N in the product, which for N=4,5 puts the initial iterate
279
+ many orders of magnitude away from the data and leaves ALS crawling.
280
+ """
281
+ rr = [int(ranks[k]) for k in range(n)]
282
+ prod = 1.0
283
+ for x in rr:
284
+ prod *= x
285
+ ms = float(R.pow(2).mean())
286
+ s = (ms / prod) ** (1.0 / (2 * n))
287
+ gg = g if g is not None else gen
288
+ return [s * torch.randn(rr[k], dims[k], rr[(k + 1) % n], generator=gg,
289
+ dtype=R.dtype).to(R.device) for k in range(n)]
290
+
291
+ # ALS on a ring has genuine local minima -- a single start that stalls would be reported
292
+ # as "the ring is not better", which is exactly the confound this module must avoid. So
293
+ # several starts are run and the best kept; restarts are cheap next to the SVDs anyway.
294
+ inits = [("tt_padded", padded_init)]
295
+ for j in range(restarts):
296
+ g = torch.Generator(device="cpu").manual_seed(seed + 977 * (j + 1))
297
+ inits.append((f"random{j}", lambda g=g: random_init(g)))
298
+ best, best_err, hists, winner = tt_to_tr(tt), base_err, {}, "tt_init"
299
+ for init_name, init_fn in inits:
300
+ cores, ihist = init_fn(), []
301
+ for it in range(sweeps):
302
+ for k in range(n):
303
+ # The subchain leaves every d index separate and in CYCLIC order starting at
304
+ # k+1. R's remaining modes must be flattened in that SAME order -- movedim
305
+ # gives natural order, which for interior k is a different permutation and
306
+ # would solve the least-squares problem against mismatched columns. It still
307
+ # returns a finite "solution", just to the wrong system.
308
+ # The chunk decision is made from SHAPES ONLY and BEFORE anything large is
309
+ # built. An earlier version computed the full subchain first and then decided
310
+ # whether to chunk, so the unchunked allocation always happened anyway and the
311
+ # chunked branch saved nothing -- the memory-bounded path existed but could
312
+ # never take effect.
313
+ # Auto-size the chunk from a memory budget instead of comparing against a
314
+ # fixed number. A constant threshold silently does nothing when the mode
315
+ # happens to equal it -- which is exactly what let a 4-site fit try to
316
+ # materialise a 134 GB subchain (mode 64, chunk 64, "64 > 64" is false).
317
+ m_ = (k + 1) % n
318
+ other = 1
319
+ for j in range(n):
320
+ if j != k:
321
+ other *= dims[j]
322
+ per_unit = max(1, (other // max(1, dims[m_])) * rank_of(cores, k))
323
+ auto = max(1, min(dims[m_], int(chunk_elems // per_unit)))
324
+ use_chunk = auto < dims[m_]
325
+ if solver == "normal" and use_chunk:
326
+ # Memory-bounded normal equations. Materialising the subchain costs
327
+ # r_k * (prod_{j!=k} d_j) * r_{k-1} floats -- on down_proj at these ranks
328
+ # that is (155, 688, 256, 155) = 17 GB, so the unchunked path cannot run
329
+ # on the MLP layers at all. The Gram and right-hand side are sums over the
330
+ # flattened "others" index, so they accumulate exactly over slices of the
331
+ # first subchain mode; this is the same arithmetic, not an approximation.
332
+ m = m_
333
+ d_next = dims[m]
334
+ rk_, rkm1_ = None, None
335
+ AtA = AtW = None
336
+ Rp = R.permute(*([k] + [(k + 1 + t) % n for t in range(n - 1)]))
337
+ for lo in range(0, d_next, auto):
338
+ hi = min(lo + auto, d_next)
339
+ Ec = _subchain_chunk(cores, k, lo, hi)
340
+ rk_, rkm1_ = Ec.shape[0], Ec.shape[-1]
341
+ Ec = Ec.reshape(rk_, -1, rkm1_)
342
+ Ac = Ec.permute(1, 2, 0).reshape(Ec.shape[1], rkm1_ * rk_)
343
+ Wc = Rp[:, lo:hi].reshape(dims[k], -1)
344
+ g = Ac.transpose(0, 1) @ Ac
345
+ h = Ac.transpose(0, 1) @ Wc.transpose(0, 1)
346
+ AtA = g if AtA is None else AtA + g
347
+ AtW = h if AtW is None else AtW + h
348
+ del Ec, Ac, Wc, g, h
349
+ lam = 1e-8 * float(AtA.diagonal().abs().mean().clamp_min(1e-30))
350
+ AtA_r = AtA + lam * torch.eye(AtA.shape[0], dtype=AtA.dtype,
351
+ device=AtA.device)
352
+ G = _solve_maybe_cpu(AtA_r, AtW)
353
+ del AtA_r
354
+ # ||R - A G||^2 = ||R||^2 - 2<AtW, G> + <G, AtA G>, all from quantities
355
+ # already formed. Computing it by contracting the whole ring instead would
356
+ # cost one full O(prod d_k) tensor per sweep per core.
357
+ sq = rnorm2 - 2.0 * float((AtW * G).sum()) + float((G * (AtA @ G)).sum())
358
+ last_sq = max(sq, 0.0)
359
+ rkm1, rk = rkm1_, rk_
360
+ else:
361
+ # ONE branch builds A/Wk and ONE consumes them. Splitting the decision
362
+ # across two separate if/else chains left the non-chunked solve reachable
363
+ # after the chunked path had run, where `A` is whatever the PREVIOUS core
364
+ # left behind -- a silent wrong answer rather than an error.
365
+ E = _subchain(cores, k) # (r_k, d_{k+1},..,d_{k-1}, r_{k-1})
366
+ rk, rkm1 = E.shape[0], E.shape[-1]
367
+ E = E.reshape(rk, -1, rkm1)
368
+ D = E.shape[1]
369
+ A = E.permute(1, 2, 0).reshape(D, rkm1 * rk)
370
+ perm = [k] + [(k + 1 + t) % n for t in range(n - 1)]
371
+ # R's remaining modes must be flattened in the subchain's CYCLIC order --
372
+ # movedim gives natural order, which for interior k solves the
373
+ # least-squares problem against mismatched columns and still returns a
374
+ # finite "solution", just to the wrong system.
375
+ Wk = R.permute(*perm).reshape(dims[k], D)
376
+ del E
377
+ if solver == "normal":
378
+ AtA = A.transpose(0, 1) @ A
379
+ lam = 1e-8 * float(AtA.diagonal().abs().mean().clamp_min(1e-30))
380
+ AtA += lam * torch.eye(AtA.shape[0], dtype=A.dtype, device=A.device)
381
+ G = _solve_maybe_cpu(AtA, A.transpose(0, 1) @ Wk.transpose(0, 1))
382
+ del AtA
383
+ else:
384
+ G = torch.linalg.lstsq(A, Wk.transpose(0, 1)).solution
385
+ last_sq = None
386
+ del A, Wk
387
+ cores[k] = (G.transpose(0, 1).reshape(dims[k], rkm1, rk)
388
+ .permute(1, 0, 2).contiguous())
389
+ err = (float(last_sq) ** 0.5 / rnorm if last_sq is not None
390
+ else float((contract_tr(cores, chunk=chunk) - R).norm() / rnorm))
391
+ ihist.append(round(err, 8))
392
+ if verbose:
393
+ print(f" TR-ALS[{init_name}] sweep {it+1}: rel err {err:.6f}", flush=True)
394
+ if err < best_err - 1e-12:
395
+ best, best_err, winner = [c.clone() for c in cores], err, init_name
396
+ if it and abs(ihist[-2] - err) < tol * max(err, 1e-12):
397
+ break
398
+ hists[init_name] = ihist
399
+ # The algebraic residual is a difference of nearly equal large numbers; at a relative
400
+ # error of ~1e-2 it has already lost most of its fp32 significance. It is fine for
401
+ # steering ALS but not for the number that gets reported, so the winner is re-measured
402
+ # once by explicit contraction.
403
+ best_err = tr_rel_err(best, R)
404
+ if best_err > base_err:
405
+ return tt_to_tr(tt), base_err, {"fell_back_to_tt": True, "tt_err": base_err,
406
+ "winner": "tt_init", "als_hist": hists}
407
+ return best, best_err, {"fell_back_to_tt": winner == "tt_init", "tt_err": base_err,
408
+ "winner": winner, "als_hist": hists}
409
+
410
+
411
+ # --------------------------------------------------------------------------- bookkeeping
412
+ def tt_params(dims, ranks) -> int:
413
+ r = [1] + [int(x) for x in ranks] + [1]
414
+ return sum(r[k] * dims[k] * r[k + 1] for k in range(len(dims)))
415
+
416
+
417
+ def tr_params(dims, ranks) -> int:
418
+ r = [int(x) for x in ranks]
419
+ n = len(dims)
420
+ return sum(r[k] * dims[k] * r[(k + 1) % n] for k in range(n))
421
+
422
+
423
+ def tt_rank_ceilings(dims):
424
+ """r_k <= min(prod_{j<=k} d_j, prod_{j>k} d_j): above that the truncation is vacuous."""
425
+ n, out = len(dims), []
426
+ for k in range(n - 1):
427
+ left = 1
428
+ for j in range(k + 1):
429
+ left *= dims[j]
430
+ right = 1
431
+ for j in range(k + 1, n):
432
+ right *= dims[j]
433
+ out.append(min(left, right))
434
+ return out
435
+
436
+
437
+ def ranks_for_budget(dims, budget: int, ring: bool, alpha: float = 1.0):
438
+ """Largest ranks fitting `budget`, shaped by r_k proportional to ceiling_k ** alpha.
439
+
440
+ Bisection rather than a closed form: the count is linear in each rank but bilinear in
441
+ adjacent pairs, and cyclic for a ring, so there is no single quadratic to solve.
442
+
443
+ ALPHA EXISTS BECAUSE EQUAL FILL FRACTION IS WRONG WHEN THE CEILINGS ARE LOPSIDED. For a
444
+ 4-site split of 4096 the ceilings are (64, 4096, 64): alpha=1 spends the budget on the
445
+ huge middle bond and returns ranks like (36, 2327, 36), strangling the two bonds that
446
+ actually gate information flow through the chain. alpha=0 gives every bond the same rank
447
+ instead. Neither is right a priori, so the caller is expected to MEASURE across alpha --
448
+ the reconstruction error, not the heuristic, decides.
449
+ """
450
+ ceil_tt = tt_rank_ceilings(dims)
451
+ ceils = ([min(ceil_tt)] + ceil_tt) if ring else ceil_tt
452
+ fn = tr_params if ring else tt_params
453
+ mx = max(ceils)
454
+ shape = [(c / mx) ** alpha for c in ceils]
455
+
456
+ def build(t):
457
+ return [max(1, min(int(round(t * mx * s)), c)) for s, c in zip(shape, ceils)]
458
+
459
+ lo, hi = 0.0, 1.0
460
+ if fn(dims, build(1.0)) <= budget:
461
+ return build(1.0)
462
+ for _ in range(60):
463
+ mid = (lo + hi) / 2
464
+ if fn(dims, build(mid)) > budget:
465
+ hi = mid
466
+ else:
467
+ lo = mid
468
+ r = build(lo)
469
+ while fn(dims, r) > budget and max(r) > 1:
470
+ r[r.index(max(r))] -= 1
471
+ return r
472
+
473
+
474
+ def balanced_split(n: int, k: int):
475
+ """Most balanced k-way factorisation of n with every factor >= 2 (spread-minimising)."""
476
+ best = None
477
+ def rec(rem, parts):
478
+ nonlocal best
479
+ if len(parts) == k - 1:
480
+ if rem >= 2:
481
+ cand = parts + [rem]
482
+ key = max(cand) / min(cand)
483
+ if best is None or key < best[0]:
484
+ best = (key, cand)
485
+ return
486
+ d = 2
487
+ while d * d <= rem or d <= rem:
488
+ if d > rem:
489
+ break
490
+ if rem % d == 0:
491
+ rec(rem // d, parts + [d])
492
+ d += 1
493
+ rec(n, [])
494
+ if best is None:
495
+ raise ValueError(f"no {k}-way factorisation of {n} with all factors >= 2")
496
+ return sorted(best[1])
497
+
498
+
499
+ def self_test(device: str = "cpu") -> None:
500
+ torch.manual_seed(0)
501
+ dt = torch.float64
502
+ for osp, isp in [((4, 3), (2, 5)), ((4, 3, 2), (2, 5, 3)), ((2, 3, 2, 2), (2, 2, 3, 2))]:
503
+ n = len(osp)
504
+ out = 1
505
+ inn = 1
506
+ for x in osp:
507
+ out *= x
508
+ for x in isp:
509
+ inn *= x
510
+ W = torch.randn(out, inn, dtype=dt, device=device)
511
+ R = rearrange_n(W, osp, isp)
512
+ dims = [o * i for o, i in zip(osp, isp)]
513
+ assert list(R.shape) == dims, (R.shape, dims)
514
+ assert torch.equal(inv_rearrange_n(R, osp, isp), W), f"{n}-site rearrange not invertible"
515
+ # index identity, not just the round trip: a wrong-but-self-inverse permutation would
516
+ # pass the round trip and scramble every weight silently
517
+ for _ in range(40):
518
+ oi = [torch.randint(0, x, (1,)).item() for x in osp]
519
+ ii = [torch.randint(0, x, (1,)).item() for x in isp]
520
+ ridx = tuple(oi[k] * isp[k] + ii[k] for k in range(n))
521
+ orow = 0
522
+ for k in range(n):
523
+ orow = orow * osp[k] + oi[k]
524
+ ocol = 0
525
+ for k in range(n):
526
+ ocol = ocol * isp[k] + ii[k]
527
+ assert R[ridx] == W[orow, ocol], f"{n}-site rearrange index map wrong"
528
+
529
+ ceils = tt_rank_ceilings(dims)
530
+ cores = fit_tt(R, ceils)
531
+ e = float((contract_tt(cores) - R).norm() / R.norm())
532
+ assert e < 1e-10, f"{n}-site full-rank TT is not exact ({e:.3e})"
533
+ assert tt_params(dims, ceils) == sum(c.numel() for c in cores), "tt_params disagrees"
534
+
535
+ if n >= 3:
536
+ trunc = [max(1, c // 2) for c in ceils]
537
+ cores = fit_tt(R, trunc)
538
+ assert tt_params(dims, trunc) == sum(c.numel() for c in cores)
539
+ for a in (0.0, 0.5, 1.0):
540
+ rr = ranks_for_budget(dims, tt_params(dims, trunc), ring=False, alpha=a)
541
+ assert tt_params(dims, rr) <= tt_params(dims, trunc), \
542
+ f"TT budget solver overran at alpha={a}"
543
+ # a ring at the SAME parameter budget must not do worse than the TT it starts from
544
+ budget = tt_params(dims, trunc)
545
+ rring = ranks_for_budget(dims, budget, ring=True)
546
+ assert tr_params(dims, rring) <= budget, "TR budget solver overran"
547
+ _c, err_tr, info = fit_tr(R, rring, sweeps=6)
548
+ assert err_tr <= info["tt_err"] + 1e-12, "fit_tr returned worse than its own TT init"
549
+ assert abs(float((contract_tr(_c) - R).norm() / R.norm()) - err_tr) < 1e-8, \
550
+ "reported TR error does not match the returned cores"
551
+
552
+ ring_chk = [torch.randn(3, 5, 4, dtype=dt), torch.randn(4, 6, 2, dtype=dt),
553
+ torch.randn(2, 7, 3, dtype=dt)]
554
+ full = contract_tr(ring_chk)
555
+ assert torch.allclose(full, contract_tr(ring_chk, chunk=2), atol=1e-12), \
556
+ "chunked contract_tr disagrees with the unchunked one"
557
+ ref = torch.einsum("iaj,jbk,kci->abc", *ring_chk)
558
+ assert torch.allclose(full, ref, atol=1e-12), "contract_tr is not the cyclic trace"
559
+
560
+ # a ring with r_0 > 1 must be able to represent things a TT of the same internal ranks cannot
561
+ dims = [6, 6, 6]
562
+ torch.manual_seed(1)
563
+ ring = [torch.randn(3, 6, 3, dtype=dt) for _ in range(3)]
564
+ Rr = contract_tr(ring)
565
+ tt_same = fit_tt(Rr, [3, 3])
566
+ e_tt = float((contract_tt(tt_same) - Rr).norm() / Rr.norm())
567
+ assert e_tt > 1e-3, ("a rank-(3,3,3) ring was reproduced by a rank-(3,3) TT; the ring "
568
+ f"contraction is probably not closing the loop (err {e_tt:.3e})")
569
+ for slv in ("lstsq", "normal"):
570
+ got, err, info = fit_tr(Rr, [3, 3, 3], sweeps=200, restarts=3, solver=slv)
571
+ assert err < 1e-4, (f"TR-ALS[{slv}] failed to recover an exact rank-3 ring "
572
+ f"(err {err:.3e}); winner={info['winner']} "
573
+ f"tt_init={info['tt_err']:.3e}")
574
+ got, err, info = fit_tr(Rr, [3, 3, 3], sweeps=200, restarts=3)
575
+ # chunking changes only how the Gram is accumulated, so it must reach the same optimum
576
+ _g, err_ch, _i = fit_tr(Rr, [3, 3, 3], sweeps=200, restarts=3, solver="normal", chunk=2)
577
+ assert err_ch < 1e-4, f"chunked TR-ALS failed to recover an exact ring (err {err_ch:.3e})"
578
+ assert err < 1e-4, (f"TR-ALS failed to recover an exact rank-3 ring (err {err:.3e}); "
579
+ f"winner={info['winner']} tt_init={info['tt_err']:.3e}")
580
+
581
+ assert balanced_split(4096, 3) == [16, 16, 16], balanced_split(4096, 3)
582
+ assert balanced_split(4096, 2) == [64, 64]
583
+ b = balanced_split(11008, 3)
584
+ assert 43 in b and b[0] * b[1] * b[2] == 11008, b
585
+ print("tensor_formats self-test PASSED (N-site rearrange exact & index-checked, full-rank "
586
+ "TT exact, budget solvers within budget, TR >= its TT init, TR strictly beats TT on "
587
+ "a true ring, balanced splits correct)")
588
+
589
+
590
+ if __name__ == "__main__":
591
+ self_test()
592
+
593
+
594
+ # --------------------------------------------------------------------------- layers
595
+ class TRLinear(torch.nn.Module):
596
+ """Tensor-ring linear layer: W = Tr[G_0 G_1 ... G_{N-1}] after the shared rearrangement.
597
+
598
+ NOT a LowRankLinear subclass, unlike MPO2Linear/MPO3Linear. Those two could pretend to be
599
+ a pair of factors because a train has open ends; a ring genuinely has N 3-D cores and no
600
+ A/B roles to borrow, so anything keyed on `.A`/`.B` (the blockwise quantizer, the bond
601
+ rebalancer) has to be taught about `.cores` explicitly rather than silently doing the
602
+ wrong thing to a subclass it thinks it understands.
603
+
604
+ RECONSTRUCTION IS CHUNKED BY DEFAULT. Merging any two cores of a 3-site ring costs
605
+ r*d_i*d_j*r entries -- on down_proj at ring rank 142 that is 7.1 GB in bf16, per layer,
606
+ per forward. Chunking the first mode bounds it; the cost is more, smaller matmuls.
607
+ """
608
+
609
+ def __init__(self, cores, osplit, isplit, chunk_elems: int = 500_000_000,
610
+ bias=None):
611
+ super().__init__()
612
+ n = len(cores)
613
+ if len(osplit) != n or len(isplit) != n:
614
+ raise ValueError(f"{n} cores need {n} output and {n} input modes")
615
+ if min(list(osplit) + list(isplit)) < 2:
616
+ raise ValueError(f"degenerate split o={tuple(osplit)} i={tuple(isplit)}: a mode of "
617
+ f"1 collapses a site, making this an (N-1)-site network")
618
+ for k in range(n):
619
+ if cores[k].shape[1] != osplit[k] * isplit[k]:
620
+ raise ValueError(f"core {k} has mode {cores[k].shape[1]}, expected "
621
+ f"{osplit[k]}*{isplit[k]}")
622
+ if cores[k].shape[2] != cores[(k + 1) % n].shape[0]:
623
+ raise ValueError(f"ring does not close between cores {k} and {(k+1)%n}")
624
+ self.cores = torch.nn.ParameterList([torch.nn.Parameter(c) for c in cores])
625
+ self.osplit, self.isplit = tuple(int(x) for x in osplit), tuple(int(x) for x in isplit)
626
+ o = i = 1
627
+ for k in range(n):
628
+ o *= self.osplit[k]
629
+ i *= self.isplit[k]
630
+ self.out_features, self.in_features = o, i
631
+ self.ranks = tuple(int(c.shape[0]) for c in cores)
632
+ # Elements per unit of the FIRST mode in contract_tr's peak intermediate, which is
633
+ # (r_0, c, d_1, ..., d_{n-2}, r_{n-1}) -- the LAST core's mode is contracted away by
634
+ # the closing einsum and must not be counted. Including it over-estimated the cost by
635
+ # d_{n-1} (256x on an MLP layer), which drove the chunk to 1 and would have made every
636
+ # MLP forward 256 sequential contractions instead of 8.
637
+ mid = 1
638
+ for k in range(1, n - 1):
639
+ mid *= cores[k].shape[1]
640
+ per_unit = max(1, mid * self.ranks[0] * self.ranks[-1])
641
+ self._chunk = max(1, min(cores[0].shape[1], int(chunk_elems // per_unit)))
642
+ # non-persistent: never wants to show up in a checkpoint, and materialize_() is an
643
+ # inference-only opt-in (see below), so a freshly-loaded layer must start uncached.
644
+ self.register_buffer("_cached_weight", None, persistent=False)
645
+ # LLaMA projections have no bias, but Qwen2's q/k/v do. Dropping it on
646
+ # replacement is SILENT -- the shapes still match, the parameter-count gate
647
+ # still balances (achieved excludes bias, and so would the rebuilt model), and
648
+ # only the accuracy moves. So the bias is carried, not discarded.
649
+ self.bias = None if bias is None else torch.nn.Parameter(bias)
650
+
651
+ def weight(self) -> torch.Tensor:
652
+ if self._cached_weight is not None:
653
+ return self._cached_weight
654
+ cs = list(self.cores)
655
+ R = contract_tr(cs, chunk=self._chunk if self._chunk < cs[0].shape[1] else 0)
656
+ return inv_rearrange_n(R, self.osplit, self.isplit)
657
+
658
+ @torch.no_grad()
659
+ def materialize_(self):
660
+ """Cache the dense weight once. INFERENCE ONLY -- gradients cannot flow through the
661
+ cached tensor back to the cores, so this must never be called before/during training.
662
+
663
+ forward() calls weight() on every call, which redoes the full chunked ring
664
+ contraction (two tensor contractions + a trace per core) from scratch. That is fine
665
+ for a single forward pass, but autoregressive generation calls forward() once per
666
+ layer per GENERATED TOKEN with the cores completely unchanged between calls. Measured
667
+ on tr3_ring4b04_h1's GSM8K eval before this existed: 209s/item, ETA 73h for 1319
668
+ items -- every one of ~200 generated tokens re-contracted all 224 ring layers.
669
+ A train layer (LowRankLinear/MPO2Linear) never had this problem: F.linear(F.linear(x,
670
+ B), A) applies the two factors straight to the activation and never materializes W at
671
+ all, so there was nothing to cache there.
672
+ """
673
+ self._cached_weight = self.weight().detach().clone()
674
+
675
+ def unmaterialize_(self):
676
+ self._cached_weight = None
677
+
678
+ def forward(self, x):
679
+ b = None if self.bias is None else self.bias.to(x.dtype)
680
+ return torch.nn.functional.linear(x, self.weight().to(x.dtype), b)
681
+
682
+ def extra_repr(self) -> str:
683
+ return (f"o={self.osplit}, i={self.isplit}, ranks={self.ranks}, "
684
+ f"chunk={self._chunk}, materialized={self._cached_weight is not None}")
685
+
686
+
687
+ class SparseAugTRLinear(TRLinear):
688
+ """TRLinear plus a sparse additive residual. Same int64-index contract as the MPO layers:
689
+ `sp_idx` must NEVER be cast to a floating dtype -- a `.half()` on a 45M-scale index
690
+ silently rounds it to a different, valid-looking position."""
691
+
692
+ def __init__(self, cores, osplit, isplit, sp_idx, sp_val,
693
+ chunk_elems: int = 500_000_000, bias=None):
694
+ super().__init__(cores, osplit, isplit, chunk_elems, bias)
695
+ if sp_idx.numel() != sp_val.numel():
696
+ raise ValueError(f"{sp_idx.numel()} indices for {sp_val.numel()} values")
697
+ n = self.out_features * self.in_features
698
+ if sp_idx.numel() and (int(sp_idx.max()) >= n or int(sp_idx.min()) < 0):
699
+ raise ValueError("sparse index out of range for this layer")
700
+ self.register_buffer("sp_idx", sp_idx.to(torch.int64))
701
+ self.sp_val = torch.nn.Parameter(sp_val)
702
+
703
+ def weight(self) -> torch.Tensor:
704
+ # must check the cache here too, not just in TRLinear.weight(): materialize_() caches
705
+ # THIS class's weight() (ring + sparse, since self.weight() dispatches polymorphically
706
+ # from inside materialize_()), so without this check a later call would go
707
+ # super().weight() -> see the cache -> return the already-sparse-augmented tensor ->
708
+ # and then add the sparse residual a second time on top of it.
709
+ if self._cached_weight is not None:
710
+ return self._cached_weight
711
+ W = super().weight()
712
+ if self.sp_idx.numel():
713
+ W = W.reshape(-1).index_add(0, self.sp_idx, self.sp_val.to(W.dtype)).view_as(W)
714
+ return W
715
+
716
+ def extra_repr(self) -> str:
717
+ return super().extra_repr() + f", nnz={self.sp_idx.numel()}"
718
+
719
+
720
+ def layer_self_test() -> None:
721
+ torch.manual_seed(0)
722
+ osp, isp = (4, 3, 2), (2, 5, 3)
723
+ dims = [o * i for o, i in zip(osp, isp)]
724
+ cores = [torch.randn(3, dims[0], 4), torch.randn(4, dims[1], 2), torch.randn(2, dims[2], 3)]
725
+ lay = TRLinear([c.clone() for c in cores], osp, isp, chunk_elems=10 ** 9)
726
+ ref = inv_rearrange_n(contract_tr(cores), osp, isp)
727
+ assert torch.allclose(lay.weight(), ref, atol=1e-5), "TRLinear.weight() != contract+inverse"
728
+ # chunking must not change the answer
729
+ lay_c = TRLinear([c.clone() for c in cores], osp, isp, chunk_elems=1)
730
+ assert lay_c._chunk < dims[0], "chunk did not engage"
731
+ assert torch.allclose(lay_c.weight(), ref, atol=1e-5), "chunked forward differs"
732
+ x = torch.randn(7, lay.in_features)
733
+ assert torch.allclose(lay(x), x @ ref.T, atol=1e-4), "forward != x W^T"
734
+ idx = torch.tensor([0, 5, lay.out_features * lay.in_features - 1], dtype=torch.int64)
735
+ val = torch.tensor([1.0, -2.0, 3.0])
736
+ sl = SparseAugTRLinear([c.clone() for c in cores], osp, isp, idx, val, chunk_elems=10 ** 9)
737
+ d = (sl.weight() - ref).reshape(-1)
738
+ assert torch.allclose(d[idx], val, atol=1e-5), "sparse residual not applied at sp_idx"
739
+ assert abs(float(d.abs().sum()) - float(val.abs().sum())) < 1e-4, "residual leaked"
740
+ try:
741
+ TRLinear([torch.zeros(2, 4, 2), torch.zeros(2, 3, 2), torch.zeros(2, 6, 2)],
742
+ (4, 1, 2), (1, 3, 3))
743
+ except ValueError:
744
+ pass
745
+ else:
746
+ raise AssertionError("a degenerate split (mode = 1) was accepted")
747
+ # gradients must reach every core
748
+ lay.zero_grad()
749
+ lay(x).sum().backward()
750
+ for k, c in enumerate(lay.cores):
751
+ assert c.grad is not None and float(c.grad.abs().sum()) > 0, f"core {k} got no gradient"
752
+ # materialize_() must be a pure inference-speed cache: same weight, same forward output,
753
+ # and -- the one bug this is actually guarding against -- the sparse residual must not
754
+ # get added twice once SparseAugTRLinear.weight() is short-circuited by the cache.
755
+ pre = lay.weight().clone()
756
+ lay.materialize_()
757
+ assert lay._cached_weight is not None, "materialize_() did not populate the cache"
758
+ assert torch.allclose(lay.weight(), pre, atol=1e-6), "materialize_() changed weight()"
759
+ assert torch.allclose(lay(x), x @ ref.T, atol=1e-4), "forward != x W^T after materialize_()"
760
+ lay.unmaterialize_()
761
+ assert lay._cached_weight is None, "unmaterialize_() left the cache populated"
762
+ sl_ref = sl.weight().clone()
763
+ sl.materialize_()
764
+ assert torch.allclose(sl.weight(), sl_ref, atol=1e-6), \
765
+ "materialized SparseAugTRLinear.weight() differs -- sparse residual likely double-added"
766
+ print("TRLinear self-test PASSED (weight matches contraction, chunking exact, forward "
767
+ "correct, sparse localised, degenerate split refused, all cores differentiable, "
768
+ "materialize_() cache exact and idempotent)")