reorg: remove superseded path tt3-qwen2.5-3b/

tt3-qwen2.5-3b/LICENSE

@@ -1,54 +0,0 @@
-Qwen RESEARCH LICENSE AGREEMENT
-
-Qwen RESEARCH LICENSE AGREEMENT Release Date: September 19, 2024
-
-By clicking to agree or by using or distributing any portion or element of the Qwen Materials, you will be deemed to have recognized and accepted the content of this Agreement, which is effective immediately.
-
-1. Definitions
-    a. This Qwen RESEARCH LICENSE AGREEMENT (this "Agreement") shall mean the terms and conditions for use, reproduction, distribution and modification of the Materials as defined by this Agreement.
-    b. "We" (or "Us") shall mean Alibaba Cloud.
-    c. "You" (or "Your") shall mean a natural person or legal entity exercising the rights granted by this Agreement and/or using the Materials for any purpose and in any field of use.
-    d. "Third Parties" shall mean individuals or legal entities that are not under common control with us or you.
-    e. "Qwen" shall mean the large language models, and software and algorithms, consisting of trained model weights, parameters (including optimizer states), machine-learning model code, inference-enabling code, training-enabling code, fine-tuning enabling code and other elements of the foregoing distributed by us.
-    f. "Materials" shall mean, collectively, Alibaba Cloud's proprietary Qwen and Documentation (and any portion thereof) made available under this Agreement.
-    g. "Source" form shall mean the preferred form for making modifications, including but not limited to model source code, documentation source, and configuration files.
-    h. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
-    i. "Non-Commercial" shall mean for research or evaluation purposes only.
-
-2. Grant of Rights
-    a. You are granted a non-exclusive, worldwide, non-transferable and royalty-free limited license under Alibaba Cloud's intellectual property or other rights owned by us embodied in the Materials to use, reproduce, distribute, copy, create derivative works of, and make modifications to the Materials FOR NON-COMMERCIAL PURPOSES ONLY. 
-    b. If you are commercially using the Materials, you shall request a license from us.
-
-3. Redistribution
-You may distribute copies or make the Materials, or derivative works thereof, available as part of a product or service that contains any of them, with or without modifications, and in Source or Object form, provided that you meet the following conditions:
-    a. You shall give any other recipients of the Materials or derivative works a copy of this Agreement;
-    b. You shall cause any modified files to carry prominent notices stating that you changed the files;
-    c. You shall retain in all copies of the Materials that you distribute the following attribution notices within a "Notice" text file distributed as a part of such copies: "Qwen is licensed under the Qwen RESEARCH LICENSE AGREEMENT, Copyright (c) Alibaba Cloud. All Rights Reserved."; and
-    d. You may add your own copyright statement to your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of your modifications, or for any such derivative works as a whole, provided your use, reproduction, and distribution of the work otherwise complies with the terms and conditions of this Agreement.
-
-4. Rules of use
-    a. The Materials may be subject to export controls or restrictions in China, the United States or other countries or regions. You shall comply with applicable laws and regulations in your use of the Materials.
-    b. If you use the Materials or any outputs or results therefrom to create, train, fine-tune, or improve an AI model that is distributed or made available, you shall prominently display “Built with Qwen” or “Improved using Qwen” in the related product documentation.
-
-5. Intellectual Property
-    a. We retain ownership of all intellectual property rights in and to the Materials and derivatives made by or for us. Conditioned upon compliance with the terms and conditions of this Agreement, with respect to any derivative works and modifications of the Materials that are made by you, you are and will be the owner of such derivative works and modifications.
-    b. No trademark license is granted to use the trade names, trademarks, service marks, or product names of us, except as required to fulfill notice requirements under this Agreement or as required for reasonable and customary use in describing and redistributing the Materials.
-    c. If you commence a lawsuit or other proceedings (including a cross-claim or counterclaim in a lawsuit) against us or any entity alleging that the Materials or any output therefrom, or any part of the foregoing, infringe any intellectual property or other right owned or licensable by you, then all licenses granted to you under this Agreement shall terminate as of the date such lawsuit or other proceeding is commenced or brought.
-
-6. Disclaimer of Warranty and Limitation of Liability
-    a. We are not obligated to support, update, provide training for, or develop any further version of the Qwen Materials or to grant any license thereto.
-    b. THE MATERIALS ARE PROVIDED "AS IS" WITHOUT ANY EXPRESS OR IMPLIED WARRANTY OF ANY KIND INCLUDING WARRANTIES OF MERCHANTABILITY, NONINFRINGEMENT, OR FITNESS FOR A PARTICULAR PURPOSE. WE MAKE NO WARRANTY AND ASSUME NO RESPONSIBILITY FOR THE SAFETY OR STABILITY OF THE MATERIALS AND ANY OUTPUT THEREFROM.
-    c. IN NO EVENT SHALL WE BE LIABLE TO YOU FOR ANY DAMAGES, INCLUDING, BUT NOT LIMITED TO ANY DIRECT, OR INDIRECT, SPECIAL OR CONSEQUENTIAL DAMAGES ARISING FROM YOUR USE OR INABILITY TO USE THE MATERIALS OR ANY OUTPUT OF IT, NO MATTER HOW IT’S CAUSED.
-    d. You will defend, indemnify and hold harmless us from and against any claim by any third party arising out of or related to your use or distribution of the Materials.
-
-7. Survival and Termination.
-    a. The term of this Agreement shall commence upon your acceptance of this Agreement or access to the Materials and will continue in full force and effect until terminated in accordance with the terms and conditions herein.
-    b. We may terminate this Agreement if you breach any of the terms or conditions of this Agreement. Upon termination of this Agreement, you must delete and cease use of the Materials. Sections 6 and 8 shall survive the termination of this Agreement.
-
-8. Governing Law and Jurisdiction.
-    a. This Agreement and any dispute arising out of or relating to it will be governed by the laws of China, without regard to conflict of law principles, and the UN Convention on Contracts for the International Sale of Goods does not apply to this Agreement.
-    b. The People's Courts in Hangzhou City shall have exclusive jurisdiction over any dispute arising out of this Agreement.
-
-9. Other Terms and Conditions.
-    a. Any arrangements, understandings, or agreements regarding the Material not stated herein are separate from and independent of the terms and conditions of this Agreement. You shall request a separate license from us, if you use the Materials in ways not expressly agreed to in this Agreement. 
-    b. We shall not be bound by any additional or different terms or conditions communicated by you unless expressly agreed.

tt3-qwen2.5-3b/NOTICE

@@ -1,17 +0,0 @@
-Qwen is licensed under the Qwen RESEARCH LICENSE AGREEMENT,
-Copyright (c) Alibaba Cloud. All Rights Reserved.
-
-This repository contains a DERIVATIVE WORK of Qwen/Qwen2.5-3B-Instruct.
-
-Modifications made to the Qwen Materials (Qwen RESEARCH LICENSE AGREEMENT, section 3.b):
-  * Every attention and MLP projection weight has been replaced by a 3-site Tensor Ring
-    factorisation plus a 5% sparse residual. The original dense nn.Linear weights are NOT
-    present in this repository; state_dict_fp16.pt contains ring cores, not dense matrices.
-  * The resulting network was further trained by pure forward-KL knowledge distillation
-    from the unmodified Qwen2.5-3B-Instruct teacher for 60M tokens.
-  * config.json is copied unmodified from the base model and is retained only as
-    architecture metadata; it does NOT describe the weights in this repository.
-
-USE IS RESTRICTED TO NON-COMMERCIAL PURPOSES (research or evaluation only), per section 2.a
-of the Qwen RESEARCH LICENSE AGREEMENT. Commercial use requires a separate licence from
-Alibaba Cloud. See the LICENSE file in this repository for the full agreement.

tt3-qwen2.5-3b/README.md

@@ -1,102 +0,0 @@
----
-license: other
-license_name: qwen-research
-license_link: LICENSE
-base_model: Qwen/Qwen2.5-3B-Instruct
-tags:
-  - tensor-decomposition
-  - tensor-train
-  - model-compression
-  - qwen2
----
-
-# TT-3 control: Qwen2.5-3B-Instruct @ 2.08B — **topology control** — unhealed and healed weights both included
-
-A 3-site **Tensor Train** (open chain, boundary bond pinned to `r_0 = 1`) at the *same*
-per-layer parameter budget as [`qwen2.5-3b/`](../qwen2.5-3b)'s Tensor Ring. It exists to answer
-one question — *does closing the ring help, at equal parameters?* — and nothing else.
-
-## ⚠️ Non-commercial use only
-
-Derived from Qwen2.5-3B-Instruct, under the **Qwen RESEARCH LICENSE AGREEMENT** — research and
-evaluation only. See `LICENSE` and `NOTICE`.
-
-## ⚠️ Read before comparing this to anything
-
-- **This checkpoint has NOT been healed.** Every other directory in this repository received
-  60M tokens of knowledge distillation; this one received none. Putting its scores in the same
-  table as theirs, or next to the dense teacher, compares two different things.
-- It is a **control arm**, not a model anyone should deploy.
-
-## What is matched, and what is not
-
-Matched to `qwen2.5-3b/` layer by layer: the per-layer core budget is copied from that run's
-own `ranks.json`, with the same mode splits, pairing, activation weighting, calibration data,
-sparse density and seed. Only the boundary bond differs.
-
-| | Ring (`qwen2.5-3b/`) | Train (here) |
-|---|---|---|
-| Params | 2.0783B | **2.0766B** (−0.08%) |
-| Reduction | 32.65% | **32.71%** |
-| q/o ranks | (64, 64, 68) | **(1, 64, 149)** |
-| k/v ranks | (32, 32, 34) | **(1, 32, 73)** |
-| gate/up/down ranks | (111, 111, 111) | **(1, 128, 394)** |
-
-Qwen2's q/k/v carry a bias that its modeling code hardcodes while the config has no
-`attention_bias` key at all — inspecting the config reports "no bias" and is wrong. All 108
-biases (36 blocks x q/k/v) are carried into the compressed layers; dropping them would keep
-every shape valid and only move the accuracy.
-
-**The two TT bonds are deliberately unequal** (`r_1 ≤ d_0`, `r_2 ≤ d_2`, and on the MLP shapes
-those are 128 and 688). Forcing them equal would ask for ranks above the cap and hand the
-control a budget it cannot spend.
-
-**ALS did not run on 108 of the 252 layers** — the three MLP projections of every block, whose
-`(r_k · r_{k+1})²` normal equations need ~10 GB plus a factorisation copy. Those use the TT-SVD
-solution. Measured on this project's layer sweeps, ALS is worth **0.0–0.5%** to a 3-site train,
-so the control is close to its best achievable fit; the gap is stated rather than left implicit.
-
-Median activation-weighted reconstruction error 0.5894 (max 0.6054); full per-layer detail,
-including which layers skipped ALS and the checkpoint sha256, is in `provenance.json`.
-
-## Loading
-
-Same recipe as the other tensor directories: rebuild each named projection as
-`TRLinear` / `SparseAugTRLinear` from `tensor_formats.py` using `ranks.json`, **carrying the
-q/k/v bias**, then load `state_dict_fp16_unhealed.pt` with `strict=False` and assert 0 missing /
-0 unexpected. Call `materialize_()` before evaluation.
-
----
-
-## Healed weights added: `state_dict_fp16_healed.pt`
-
-The original `state_dict_fp16_unhealed.pt` is unchanged and still present. This file is the
-same network after **60M tokens of pure forward-KL distillation** from the unmodified dense
-teacher (alpha=1, no gold labels) — the same healing the TR-3 arm received, so the two can be
-compared after healing and not only at compression time.
-
-| | TT-3 (this dir) | TR-3 (ring) |
-|---|---:|---:|
-| held-out KL, compressed | 6.3507 | 0.8888 |
-| held-out KL, healed | **0.3955** | **0.1729** |
-| MMLU, compressed | 24.82% | 44.52% |
-| MMLU, healed | **43.33%** | **57.41%** |
-| teacher MMLU (same probe) | 69.21% | 69.21% |
-
-The ring is ahead at both stages: roughly 9x lower KL before healing and ~2.3x after, with
-14.08 points more MMLU, at a parameter budget matched to within 0.1%.
-
-### Two confounds, stated rather than buried
-
-**1. Batch size.** This healing used `tokens_per_step = 32768`; the TR-3 arms used 16384. Both
-saw the same 60M tokens from the same seeded mixture, so the data is matched, but the number of
-optimiser updates and the effective LR schedule are not. The **compression-stage** gap (the
-first row of the table) predates healing entirely and is free of this confound; the healed rows
-are not.
-
-**2. ALS coverage.** `provenance.json` records that this arm fell back to TT-SVD without ALS on
-many layers because the normal equations did not fit in memory, while the TR-3 arm ran 20 ALS
-sweeps everywhere. That asymmetry is real but is **not** what drives the gap, and the arm's own
-per-layer data shows why: within this checkpoint, layers that got ALS and layers that did not
-differ by ~0.02 in whitened relative error, while the TT-vs-TR difference is ~0.21 — an order of
-magnitude larger, and the same 1.56x ratio on both models.

tt3-qwen2.5-3b/config.json

@@ -1,29 +0,0 @@
-{
-  "_name_or_path": "Qwen/Qwen2.5-3B-Instruct",
-  "architectures": [
-    "Qwen2ForCausalLM"
-  ],
-  "attention_dropout": 0.0,
-  "bos_token_id": 151643,
-  "eos_token_id": 151645,
-  "hidden_act": "silu",
-  "hidden_size": 2048,
-  "initializer_range": 0.02,
-  "intermediate_size": 11008,
-  "max_position_embeddings": 32768,
-  "max_window_layers": 70,
-  "model_type": "qwen2",
-  "num_attention_heads": 16,
-  "num_hidden_layers": 36,
-  "num_key_value_heads": 2,
-  "rms_norm_eps": 1e-06,
-  "rope_scaling": null,
-  "rope_theta": 1000000.0,
-  "sliding_window": null,
-  "tie_word_embeddings": true,
-  "torch_dtype": "bfloat16",
-  "transformers_version": "4.46.3",
-  "use_cache": true,
-  "use_sliding_window": false,
-  "vocab_size": 151936
-}

tt3-qwen2.5-3b/provenance.json

@@ -1,3052 +0,0 @@
-{
-  "kind": "compress_generic",
-  "structure": "train",
-  "model_id": "Qwen/Qwen2.5-3B-Instruct",
-  "model_revision": "aa8e72537993ba99e69dfaafa59ed015b17504d1",
-  "ranks_template": "/work1/zixuan/checkpoints/Compactif/hf_qwen3b/qwen2.5-3b/ranks.json",
-  "budget_note": "per-layer core budget copied EXACTLY from the template's own core count, and the same sparse nnz, so the arms are matched layer by layer rather than only in aggregate. The matrix arm has no sparse-augmented layer type here, so its nnz share is spent on extra rank instead.",
-  "index_storage_caveat": "matched on PARAMETER count. sp_idx is a buffer, not a Parameter, so the tensor arms additionally store nnz int64 indices that this count excludes. In BYTES the tensor arms are therefore larger, and any size claim must say which of the two conventions it uses.",
-  "dense_total": 3085938688,
-  "achieved_total": 2076606220,
-  "reduction": 0.3271,
-  "n_layers": 252,
-  "n_biases_carried": 108,
-  "n_als_skipped_for_memory": 108,
-  "als_note": "a layer whose (r_k r_{k+1})^2 normal equations do not fit is fitted by TT-SVD alone. Layer-level sweeps in this project bound what that costs a train at 0.1-0.5%; a RING loses far more without ALS, so this fallback is not symmetric and must be reported per arm.",
-  "calib_blocks": 256,
-  "seq_len": 1024,
-  "lam_ratio": 0.0001,
-  "seed": 1234,
-  "tr_sweeps": 20,
-  "tr_restarts": 1,
-  "checkpoint": "/work1/zixuan/checkpoints/Compactif/qwen3b_tt3_matched/state_dict_fp16.pt",
-  "sha256": "213a70c30131976d02054e2000ed0df4bb2238fe6bb33b8948803268456c073e",
-  "elapsed_s": 1012.1,
-  "timestamp": "2026-09-05T15:47:18.110056+00:00",
-  "per_layer": {
-    "model.layers.0.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.573298,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.934272
-    },
-    "model.layers.0.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.587915,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.934272
-    },
-    "model.layers.0.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.592489,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.934272
-    },
-    "model.layers.0.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.57016,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.942271
-    },
-    "model.layers.0.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.566544,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.888234
-    },
-    "model.layers.0.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.567096,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.888234
-    },
-    "model.layers.0.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.603753,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.983426
-    },
-    "model.layers.1.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.568228,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.984921
-    },
-    "model.layers.1.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.571913,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.984921
-    },
-    "model.layers.1.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.578664,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.984921
-    },
-    "model.layers.1.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.571294,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.993075
-    },
-    "model.layers.1.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.568371,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.999996
-    },
-    "model.layers.1.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.553913,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.999996
-    },
-    "model.layers.1.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.508577,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.999999
-    },
-    "model.layers.2.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.574501,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.981545
-    },
-    "model.layers.2.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.574586,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.981545
-    },
-    "model.layers.2.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.585155,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.981545
-    },
-    "model.layers.2.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.587834,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.866401
-    },
-    "model.layers.2.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.5687,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.999877
-    },
-    "model.layers.2.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.580936,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.999877
-    },
-    "model.layers.2.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.539961,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 1.0
-    },
-    "model.layers.3.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.580581,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.948951
-    },
-    "model.layers.3.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.576418,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.948951
-    },
-    "model.layers.3.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.585657,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.948951
-    },
-    "model.layers.3.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.58851,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.929628
-    },
-    "model.layers.3.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.583626,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.999639
-    },
-    "model.layers.3.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.590461,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.999639
-    },
-    "model.layers.3.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.503482,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 1.0
-    },
-    "model.layers.4.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.583741,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.986749
-    },
-    "model.layers.4.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.583115,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.986749
-    },
-    "model.layers.4.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.589761,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.986749
-    },
-    "model.layers.4.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.580375,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.891029
-    },
-    "model.layers.4.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.582431,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.999612
-    },
-    "model.layers.4.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.593992,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.999612
-    },
-    "model.layers.4.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.527218,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.99998
-    },
-    "model.layers.5.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.575779,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.99185
-    },
-    "model.layers.5.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.57357,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.99185
-    },
-    "model.layers.5.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.589237,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.99185
-    },
-    "model.layers.5.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.568247,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.955383
-    },
-    "model.layers.5.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.595462,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.999855
-    },
-    "model.layers.5.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.59926,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.999855
-    },
-    "model.layers.5.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.550931,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.996112
-    },
-    "model.layers.6.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.584959,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.992755
-    },
-    "model.layers.6.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.583874,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.992755
-    },
-    "model.layers.6.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.584797,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.992755
-    },
-    "model.layers.6.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.589191,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.826552
-    },
-    "model.layers.6.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.592956,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.999756
-    },
-    "model.layers.6.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.598165,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.999756
-    },
-    "model.layers.6.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.56829,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.978374
-    },
-    "model.layers.7.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.582199,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.985806
-    },
-    "model.layers.7.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.577528,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.985806
-    },
-    "model.layers.7.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.588068,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.985806
-    },
-    "model.layers.7.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.586083,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.654202
-    },
-    "model.layers.7.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.592096,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.999116
-    },
-    "model.layers.7.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.59787,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.999116
-    },
-    "model.layers.7.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.560512,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.975593
-    },
-    "model.layers.8.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.588466,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.992369
-    },
-    "model.layers.8.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.582692,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.992369
-    },
-    "model.layers.8.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.590623,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.992369
-    },
-    "model.layers.8.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.593391,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.697076
-    },
-    "model.layers.8.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.595773,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.998415
-    },
-    "model.layers.8.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.600021,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.998415
-    },
-    "model.layers.8.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.59708,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.950515
-    },
-    "model.layers.9.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.591427,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.995453
-    },
-    "model.layers.9.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.584979,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.995453
-    },
-    "model.layers.9.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.594853,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.995453
-    },
-    "model.layers.9.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.593438,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.768196
-    },
-    "model.layers.9.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.595729,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.997979
-    },
-    "model.layers.9.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.600115,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.997979
-    },
-    "model.layers.9.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.594926,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.916435
-    },
-    "model.layers.10.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.581533,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.995031
-    },
-    "model.layers.10.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.575489,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.995031
-    },
-    "model.layers.10.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.592285,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.995031
-    },
-    "model.layers.10.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.582759,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.874523
-    },
-    "model.layers.10.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.597822,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.997335
-    },
-    "model.layers.10.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.601697,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.997335
-    },
-    "model.layers.10.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.602927,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.856847
-    },
-    "model.layers.11.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.584892,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.995701
-    },
-    "model.layers.11.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.581327,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.995701
-    },
-    "model.layers.11.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.591392,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.995701
-    },
-    "model.layers.11.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.58134,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.761806
-    },
-    "model.layers.11.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.601283,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.992582
-    },
-    "model.layers.11.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604655,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.992582
-    },
-    "model.layers.11.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604729,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.733887
-    },
-    "model.layers.12.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.581729,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.995644
-    },
-    "model.layers.12.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.581201,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.995644
-    },
-    "model.layers.12.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.591573,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.995644
-    },
-    "model.layers.12.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.579305,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.855637
-    },
-    "model.layers.12.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.601493,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.992888
-    },
-    "model.layers.12.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.60531,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.992888
-    },
-    "model.layers.12.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604572,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.797294
-    },
-    "model.layers.13.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.574338,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.99759
-    },
-    "model.layers.13.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.578361,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.99759
-    },
-    "model.layers.13.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.592821,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.99759
-    },
-    "model.layers.13.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.580568,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.945296
-    },
-    "model.layers.13.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.602587,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.982269
-    },
-    "model.layers.13.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604851,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.982269
-    },
-    "model.layers.13.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.605358,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.624047
-    },
-    "model.layers.14.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.586172,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.996552
-    },
-    "model.layers.14.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.58275,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.996552
-    },
-    "model.layers.14.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.593302,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.996552
-    },
-    "model.layers.14.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.571619,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.855467
-    },
-    "model.layers.14.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.602922,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.983911
-    },
-    "model.layers.14.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.605371,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.983911
-    },
-    "model.layers.14.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.605251,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.64462
-    },
-    "model.layers.15.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.580133,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.995941
-    },
-    "model.layers.15.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.581309,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.995941
-    },
-    "model.layers.15.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.593058,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.995941
-    },
-    "model.layers.15.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.571475,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.872027
-    },
-    "model.layers.15.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.603049,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.978339
-    },
-    "model.layers.15.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.605095,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.978339
-    },
-    "model.layers.15.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.605378,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.569849
-    },
-    "model.layers.16.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.585581,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.992366
-    },
-    "model.layers.16.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.584799,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.992366
-    },
-    "model.layers.16.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.592937,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.992366
-    },
-    "model.layers.16.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.586766,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.703469
-    },
-    "model.layers.16.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.60334,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.982822
-    },
-    "model.layers.16.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.605226,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.982822
-    },
-    "model.layers.16.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604882,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.75453
-    },
-    "model.layers.17.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.57391,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.99618
-    },
-    "model.layers.17.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.573506,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.99618
-    },
-    "model.layers.17.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.592354,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.99618
-    },
-    "model.layers.17.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.570643,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.906692
-    },
-    "model.layers.17.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.602791,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.981482
-    },
-    "model.layers.17.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.60461,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.981482
-    },
-    "model.layers.17.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604469,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.876613
-    },
-    "model.layers.18.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.587562,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.995034
-    },
-    "model.layers.18.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.584489,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.995034
-    },
-    "model.layers.18.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.593622,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.995034
-    },
-    "model.layers.18.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.581197,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.857926
-    },
-    "model.layers.18.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.60253,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.982396
-    },
-    "model.layers.18.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604572,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.982396
-    },
-    "model.layers.18.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.60415,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.73992
-    },
-    "model.layers.19.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.571471,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.993618
-    },
-    "model.layers.19.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.573816,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.993618
-    },
-    "model.layers.19.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.591658,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.993618
-    },
-    "model.layers.19.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.561333,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.904653
-    },
-    "model.layers.19.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.602647,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.970502
-    },
-    "model.layers.19.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604022,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.970502
-    },
-    "model.layers.19.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604459,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.647441
-    },
-    "model.layers.20.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.560264,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.992759
-    },
-    "model.layers.20.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.558552,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.992759
-    },
-    "model.layers.20.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.590322,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.992759
-    },
-    "model.layers.20.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.560928,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.812791
-    },
-    "model.layers.20.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.602939,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.981784
-    },
-    "model.layers.20.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604862,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.981784
-    },
-    "model.layers.20.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.603267,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.85096
-    },
-    "model.layers.21.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.574825,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.990238
-    },
-    "model.layers.21.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.575251,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.990238
-    },
-    "model.layers.21.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.592117,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.990238
-    },
-    "model.layers.21.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.575068,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.84186
-    },
-    "model.layers.21.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.602772,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.982316
-    },
-    "model.layers.21.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604826,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.982316
-    },
-    "model.layers.21.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604536,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.800122
-    },
-    "model.layers.22.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.578384,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.992639
-    },
-    "model.layers.22.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.575871,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.992639
-    },
-    "model.layers.22.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.592281,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.992639
-    },
-    "model.layers.22.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.580003,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.84194
-    },
-    "model.layers.22.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.603092,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.978805
-    },
-    "model.layers.22.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604892,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.978805
-    },
-    "model.layers.22.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604266,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.82127
-    },
-    "model.layers.23.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.579412,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.99389
-    },
-    "model.layers.23.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.583444,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.99389
-    },
-    "model.layers.23.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.592978,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.99389
-    },
-    "model.layers.23.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.573672,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.935109
-    },
-    "model.layers.23.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.603107,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.975713
-    },
-    "model.layers.23.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604776,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.975713
-    },
-    "model.layers.23.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604398,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.785305
-    },
-    "model.layers.24.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.580083,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.992837
-    },
-    "model.layers.24.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.580854,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.992837
-    },
-    "model.layers.24.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.595272,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.992837
-    },
-    "model.layers.24.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.574078,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.820589
-    },
-    "model.layers.24.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.60295,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.978518
-    },
-    "model.layers.24.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604438,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.978518
-    },
-    "model.layers.24.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604592,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.952028
-    },
-    "model.layers.25.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.565376,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.988722
-    },
-    "model.layers.25.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.575472,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.988722
-    },
-    "model.layers.25.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.592335,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.988722
-    },
-    "model.layers.25.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.577098,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.879787
-    },
-    "model.layers.25.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.603017,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.978046
-    },
-    "model.layers.25.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604961,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.978046
-    },
-    "model.layers.25.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604355,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.8204
-    },
-    "model.layers.26.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.582811,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.985436
-    },
-    "model.layers.26.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.581964,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.985436
-    },
-    "model.layers.26.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.5938,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.985436
-    },
-    "model.layers.26.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.581201,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.67731
-    },
-    "model.layers.26.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.602511,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.979429
-    },
-    "model.layers.26.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604568,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.979429
-    },
-    "model.layers.26.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604637,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.907833
-    },
-    "model.layers.27.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.574056,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.988292
-    },
-    "model.layers.27.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.573832,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.988292
-    },
-    "model.layers.27.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.591914,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.988292
-    },
-    "model.layers.27.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.566219,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.754294
-    },
-    "model.layers.27.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.602249,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.9843
-    },
-    "model.layers.27.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604358,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.9843
-    },
-    "model.layers.27.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604866,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.853358
-    },
-    "model.layers.28.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.581122,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.985108
-    },
-    "model.layers.28.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.58168,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.985108
-    },
-    "model.layers.28.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.591338,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.985108
-    },
-    "model.layers.28.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.569068,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.660721
-    },
-    "model.layers.28.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.602277,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.983002
-    },
-    "model.layers.28.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.60458,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.983002
-    },
-    "model.layers.28.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.603847,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.85938
-    },
-    "model.layers.29.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.579048,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.979686
-    },
-    "model.layers.29.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.579616,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.979686
-    },
-    "model.layers.29.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.593305,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.979686
-    },
-    "model.layers.29.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.581411,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.727209
-    },
-    "model.layers.29.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.602287,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.984022
-    },
-    "model.layers.29.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604357,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.984022
-    },
-    "model.layers.29.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604497,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.888887
-    },
-    "model.layers.30.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.583251,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.982799
-    },
-    "model.layers.30.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.58287,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.982799
-    },
-    "model.layers.30.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.583469,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.982799
-    },
-    "model.layers.30.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.577949,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.656925
-    },
-    "model.layers.30.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.601955,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.988163
-    },
-    "model.layers.30.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604064,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.988163
-    },
-    "model.layers.30.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.603453,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.999968
-    },
-    "model.layers.31.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.578224,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.98595
-    },
-    "model.layers.31.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.576528,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.98595
-    },
-    "model.layers.31.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.57388,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.98595
-    },
-    "model.layers.31.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.581784,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.983155
-    },
-    "model.layers.31.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.601618,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.981401
-    },
-    "model.layers.31.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604113,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.981401
-    },
-    "model.layers.31.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.603022,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.866292
-    },
-    "model.layers.32.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.565073,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.985945
-    },
-    "model.layers.32.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.565539,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.985945
-    },
-    "model.layers.32.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.552391,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.985945
-    },
-    "model.layers.32.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.590005,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.987203
-    },
-    "model.layers.32.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.601321,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.978413
-    },
-    "model.layers.32.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.603662,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.978413
-    },
-    "model.layers.32.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.603514,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.990104
-    },
-    "model.layers.33.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.541786,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.989849
-    },
-    "model.layers.33.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.554814,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.989849
-    },
-    "model.layers.33.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.551823,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.989849
-    },
-    "model.layers.33.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.593854,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.929093
-    },
-    "model.layers.33.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.601464,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.977111
-    },
-    "model.layers.33.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.603281,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.977111
-    },
-    "model.layers.33.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.603885,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.989971
-    },
-    "model.layers.34.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.589411,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.989416
-    },
-    "model.layers.34.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.58627,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.989416
-    },
-    "model.layers.34.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.593381,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.989416
-    },
-    "model.layers.34.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.57693,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.985777
-    },
-    "model.layers.34.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.602183,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.987293
-    },
-    "model.layers.34.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.60416,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.987293
-    },
-    "model.layers.34.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.60453,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.992712
-    },
-    "model.layers.35.self_attn.q_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.594823,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.963134
-    },
-    "model.layers.35.self_attn.k_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.58898,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.963134
-    },
-    "model.layers.35.self_attn.v_proj": {
-      "ranks": [
-        1,
-        32,
-        73
-      ],
-      "params": 309376,
-      "rel_err_whitened": 0.59314,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.963134
-    },
-    "model.layers.35.self_attn.o_proj": {
-      "ranks": [
-        1,
-        64,
-        149
-      ],
-      "params": 2483456,
-      "rel_err_whitened": 0.578685,
-      "als_skipped_for_memory": false,
-      "winner": "tt_padded",
-      "sep_rel": 0.967955
-    },
-    "model.layers.35.mlp.gate_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.599325,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.997148
-    },
-    "model.layers.35.mlp.up_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.604294,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.997148
-    },
-    "model.layers.35.mlp.down_proj": {
-      "ranks": [
-        1,
-        128,
-        394
-      ],
-      "params": 13198048,
-      "rel_err_whitened": 0.602678,
-      "als_skipped_for_memory": true,
-      "winner": "tt_init",
-      "sep_rel": 0.999802
-    }
-  }
-}

tt3-qwen2.5-3b/ranks.json

@@ -1,3026 +0,0 @@
-{
- "model.layers.0.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.0.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.0.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.0.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.0.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.0.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.0.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.1.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.1.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.1.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.1.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.1.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.1.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.1.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.2.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.2.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.2.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.2.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.2.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.2.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.2.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.3.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.3.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.3.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.3.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.3.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.3.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.3.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.4.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.4.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.4.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.4.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.4.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.4.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.4.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.5.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.5.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.5.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.5.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.5.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.5.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.5.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.6.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.6.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.6.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.6.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.6.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.6.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.6.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.7.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.7.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.7.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.7.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.7.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.7.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.7.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.8.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.8.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.8.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.8.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.8.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.8.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.8.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.9.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.9.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.9.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.9.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.9.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.9.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.9.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.10.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.10.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.10.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.10.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.10.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.10.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.10.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.11.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.11.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.11.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.11.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.11.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.11.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.11.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.12.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.12.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.12.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.12.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.12.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.12.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.12.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.13.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.13.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.13.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.13.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.13.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.13.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.13.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.14.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.14.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.14.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.14.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.14.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.14.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.14.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.15.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.15.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.15.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.15.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.15.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.15.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.15.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.16.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.16.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.16.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.16.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.16.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.16.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.16.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.17.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.17.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.17.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.17.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.17.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.17.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.17.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.18.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.18.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.18.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.18.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.18.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.18.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.18.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.19.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.19.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.19.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.19.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.19.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.19.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.19.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.20.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.20.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.20.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.20.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.20.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.20.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.20.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.21.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.21.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.21.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.21.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.21.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.21.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.21.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.22.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.22.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.22.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.22.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.22.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.22.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.22.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.23.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.23.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.23.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.23.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.23.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.23.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.23.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.24.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.24.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.24.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.24.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.24.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.24.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.24.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.25.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.25.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.25.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.25.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.25.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.25.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.25.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.26.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.26.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.26.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.26.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.26.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.26.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.26.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.27.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.27.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.27.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.27.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.27.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.27.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.27.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.28.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.28.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.28.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.28.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.28.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.28.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.28.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.29.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.29.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.29.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.29.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.29.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.29.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.29.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.30.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.30.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.30.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.30.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.30.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.30.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.30.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.31.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.31.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.31.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.31.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.31.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.31.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.31.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.32.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.32.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.32.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.32.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.32.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.32.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.32.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.33.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.33.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.33.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.33.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.33.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.33.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.33.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.34.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.34.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.34.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.34.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.34.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.34.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.34.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.35.self_attn.q_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.35.self_attn.k_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.35.self_attn.v_proj": {
-  "o1": 4,
-  "o2": 8,
-  "o3": 8,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 32,
-  "r2": 73,
-  "nnz": 26214
- },
- "model.layers.35.self_attn.o_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 64,
-  "r2": 149,
-  "nnz": 209715
- },
- "model.layers.35.mlp.gate_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.35.mlp.up_proj": {
-  "o1": 16,
-  "o2": 16,
-  "o3": 43,
-  "i1": 8,
-  "i2": 16,
-  "i3": 16,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- },
- "model.layers.35.mlp.down_proj": {
-  "o1": 8,
-  "o2": 16,
-  "o3": 16,
-  "i1": 16,
-  "i2": 16,
-  "i3": 43,
-  "r0": 1,
-  "r1": 128,
-  "r2": 394,
-  "nnz": 1127219
- }
-}

tt3-qwen2.5-3b/state_dict_fp16_healed.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:c8aa2603b1f4f07df7212de0822e2c040e07a46b18e79e0a305b0a2bc21b526e
-size 5885822151

tt3-qwen2.5-3b/state_dict_fp16_unhealed.pt

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:213a70c30131976d02054e2000ed0df4bb2238fe6bb33b8948803268456c073e
-size 5263492198

tt3-qwen2.5-3b/tensor_formats.py

@@ -1,890 +0,0 @@
-from __future__ import annotations
-
-"""General N-core tensor formats for a linear layer: tensor train (MPO) and tensor ring.
-
-WHY THIS EXISTS. The 2-core and 3-core results say the tensorisation step, not the healing
-step, is where the leverage is: at equal parameters the 3-core beat the 2-core AFTER healing
-at both budgets tried (4.72B: 38.94 vs ~36.5 full MMLU at 60M tokens; 5.74B: 44.91 vs 41.54
-probe), and it got there on roughly a quarter of the tokens. That invites two orthogonal
-questions this module exists to answer: how many cores, and which topology.
-
-TENSOR TRAIN vs TENSOR RING. A TT/MPO pins the two boundary bonds to 1 -- the chain has open
-ends. A tensor ring closes the chain: core k has shape (r_{k-1}, d_k, r_k) with r_N == r_0,
-and the weight is the TRACE of the cyclic product. TT is exactly the TR special case r_0 = 1,
-so at equal ranks TR is strictly more expressive. It is also strictly harder to fit: TT-SVD
-is a sequence of Eckart-Young-optimal truncations, while a ring has no such closed form, so
-TR is fitted here by TT initialisation followed by alternating least squares. That asymmetry
-matters when reading results -- a TR loss could mean "the format is worse" OR "the fit did
-not converge", so `fit_tr` reports its per-sweep error curve and refuses to return a fit
-worse than its own TT initialisation.
-
-INDEX CONVENTION (shared with mpo2.py / mpo3.py). Output mode k travels with input mode k:
-
-    R(W)[d_1, ..., d_N] = W[(o_1..o_N), (i_1..i_N)],   d_k = (o_k, i_k) flattened
-
-Every mode must be > 1. A mode of 1 collapses a site and silently reduces the core count,
-which would make an "N-core" result actually be an (N-1)-core one.
-
-WHITENING is diagonal and factorised across the N input modes, as in mpo3.py: the whitening
-is then a mode-local rescaling that survives the fit exactly and un-whitens by division.
-"""
-
-import torch
-
-
-# --------------------------------------------------------------------------- rearrangement
-def rearrange_n(W: torch.Tensor, osplit, isplit) -> torch.Tensor:
-    """W[(o_1..o_N),(i_1..i_N)] -> R[d_1,...,d_N] with d_k = o_k*i_k."""
-    n = len(osplit)
-    if len(isplit) != n:
-        raise ValueError("output and input must be split into the same number of modes")
-    perm = [j for k in range(n) for j in (k, n + k)]
-    return (W.reshape(*osplit, *isplit).permute(*perm)
-             .reshape(*[o * i for o, i in zip(osplit, isplit)]))
-
-
-def inv_rearrange_n(R: torch.Tensor, osplit, isplit) -> torch.Tensor:
-    """Inverse of `rearrange_n`."""
-    n = len(osplit)
-    inter = [x for k in range(n) for x in (osplit[k], isplit[k])]
-    perm = list(range(0, 2 * n, 2)) + list(range(1, 2 * n, 2))
-    o = 1
-    i = 1
-    for k in range(n):
-        o *= osplit[k]
-        i *= isplit[k]
-    return R.reshape(*inter).permute(*perm).reshape(o, i)
-
-
-def _apply_scales(R: torch.Tensor, osplit, isplit, scales, invert: bool):
-    """Multiply (or divide) R by sqrt(d_k[i_k]) along each input mode, in place of a dense L."""
-    n = len(osplit)
-    shape = [x for k in range(n) for x in (osplit[k], isplit[k])]
-    T = R.reshape(*shape)
-    for k in range(n):
-        s = scales[k].clamp_min(1e-12).sqrt()
-        view = [1] * (2 * n)
-        view[2 * k + 1] = isplit[k]
-        T = T / s.view(*view) if invert else T * s.view(*view)
-    return T.reshape(R.shape)
-
-
-# --------------------------------------------------------------------------- contraction
-def contract_tt(cores) -> torch.Tensor:
-    """Contract TT cores [(1,d1,r1), (r1,d2,r2), ..., (r_{N-1},dN,1)] -> R[d_1,...,d_N]."""
-    out = cores[0]
-    for g in cores[1:]:
-        a, b = out.shape[-1], g.shape[0]
-        if a != b:
-            raise ValueError(f"TT bond mismatch {a} vs {b}")
-        out = torch.tensordot(out, g, dims=([out.dim() - 1], [0]))
-    if out.shape[0] != 1 or out.shape[-1] != 1:
-        raise ValueError("TT boundary bonds must be 1")
-    return out.reshape(*out.shape[1:-1])
-
-
-def contract_tr(cores, chunk: int = 0) -> torch.Tensor:
-    """Contract TR cores [(r0,d1,r1), ..., (r_{N-1},dN,r0)] -> R[d_1,...,d_N].
-
-    The obvious route -- chain every tensordot, then take the diagonal over the two ring
-    indices -- materialises (r_0, d_1..d_N, r_0), i.e. r_0^2 times the size of the ANSWER.
-    On one LLaMA attention layer at ring rank 118 that is 934 GB, so it does not merely run
-    slowly, it cannot run at all. Folding the last contraction and the trace into a single
-    einsum keeps the peak at r_0 * prod(d_1..d_{N-1}) * r_{N-1}, and chunking the first mode
-    bounds it further for the wide MLP layers.
-    """
-    n = len(cores)
-    if cores[0].shape[0] != cores[-1].shape[-1]:
-        raise ValueError(f"ring does not close: {cores[0].shape[0]} vs {cores[-1].shape[-1]}")
-    d0 = cores[0].shape[1]
-    if chunk and d0 > chunk:
-        parts = []
-        for lo in range(0, d0, chunk):
-            sub = [cores[0][:, lo:min(lo + chunk, d0), :]] + list(cores[1:])
-            parts.append(contract_tr(sub, chunk=0))
-        return torch.cat(parts, dim=0)
-    out = cores[0]
-    for g in cores[1:-1]:
-        out = torch.tensordot(out, g, dims=([out.dim() - 1], [0]))
-    return torch.einsum("i...k,kdi->...d", out, cores[-1])
-
-
-# --------------------------------------------------------------------------- fitting
-def fit_tt(R: torch.Tensor, ranks):
-    """TT-SVD with the given internal ranks. Singular values split symmetrically at each bond,
-    matching fit_mpo2/fit_mpo3's gauge so cores stay on comparable scales (fp16-safe and
-    kinder to a blockwise quantizer)."""
-    dims = list(R.shape)
-    n = len(dims)
-    if len(ranks) != n - 1:
-        raise ValueError(f"{n} sites need {n-1} internal ranks, got {len(ranks)}")
-    cores, rest, rprev = [], R.reshape(dims[0], -1), 1
-    for k in range(n - 1):
-        M = rest.reshape(rprev * dims[k], -1)
-        U, S, Vh = torch.linalg.svd(M, full_matrices=False)
-        r = min(int(ranks[k]), S.shape[0])
-        q = S[:r].clamp_min(0).sqrt()
-        cores.append((U[:, :r] * q).reshape(rprev, dims[k], r))
-        rest = q[:, None] * Vh[:r]
-        rprev = r
-    cores.append(rest.reshape(rprev, dims[-1], 1))
-    return cores
-
-
-def tt_to_tr(cores):
-    """Reinterpret TT cores as TR cores with r_0 = 1 (an exact, lossless re-labelling)."""
-    return [c.clone() for c in cores]
-
-
-def _subchain(cores, k):
-    """Contract every core except k, in cyclic order, -> E[r_k, D, r_{k-1}] with D = prod_{j!=k} d_j."""
-    n = len(cores)
-    order = [(k + 1 + t) % n for t in range(n - 1)]
-    out = cores[order[0]]
-    for j in order[1:]:
-        out = torch.tensordot(out, cores[j], dims=([out.dim() - 1], [0]))
-    return out
-
-
-def _subchain_chunk(cores, k, lo, hi):
-    """`_subchain` with the FIRST following mode restricted to [lo, hi). The restriction rides
-    along the contraction untouched, so concatenating the chunks reproduces `_subchain`."""
-    n = len(cores)
-    order = [(k + 1 + t) % n for t in range(n - 1)]
-    out = cores[order[0]][:, lo:hi, :]
-    for j in order[1:]:
-        out = torch.tensordot(out, cores[j], dims=([out.dim() - 1], [0]))
-    return out
-
-
-def rank_of(cores, k):
-    """r_{k-1} * r_k for core k -- the width of that core's least-squares system."""
-    n = len(cores)
-    return cores[k].shape[0] * cores[k].shape[2]
-
-
-def _solve_maybe_cpu(AtA, AtW):
-    """Solve AtA X = AtW, falling back to the CPU when the GPU cannot hold the factorisation.
-
-    The normal-equation matrix is (r_{k-1} r_k) square, i.e. it grows as r^4. On down_proj at
-    ring rank 219 that is 47961^2 = 9.2 GB, and the factorisation needs a second copy -- no
-    amount of chunking the data helps, because this matrix does not depend on the data size.
-    Cholesky is tried first (the Gram is symmetric positive definite after the ridge) and
-    needs only one triangular factor; if even that does not fit, the solve moves to the host.
-    Slow, but a bounded slowdown beats an unbounded failure on the layers that matter most.
-
-    THE LADDER MUST CATCH SINGULARITY, NOT ONLY OOM. A normal-equation matrix can be
-    genuinely rank-deficient, not merely too big: when a requested bond exceeds what its
-    adjacent mode can carry (e.g. Qwen2.5-3B's q_proj splits to dims [64, 256, 256] and a
-    budget-matched sweep asks for r=92 > 64), the design matrix loses column rank and
-    `torch.linalg.solve` raises _LinAlgError. Routing only OutOfMemoryError to the host left
-    that case falling straight through the ladder and crashing the run -- the opposite of this
-    function's stated contract. A rank-deficient ALS subproblem is still perfectly well posed:
-    every solution attains the same fit, so the last rung takes the minimum-norm one via
-    lstsq, which is exactly what the "lstsq" solver mode would have done anyway.
-    """
-    if not (torch.isfinite(AtA).all() and torch.isfinite(AtW).all()):
-        raise ValueError("REFUSING: non-finite normal equations. cuSOLVER does not fail fast "
-                         "on NaN input, it stalls, so this is caught here rather than handed "
-                         "to a solver that cannot be interrupted.")
-    try:
-        L = torch.linalg.cholesky(AtA)
-        return torch.cholesky_solve(AtW, L)
-    except torch.OutOfMemoryError:
-        torch.cuda.empty_cache()
-    except Exception:
-        pass
-    try:
-        return torch.linalg.solve(AtA, AtW)
-    except torch.OutOfMemoryError:
-        torch.cuda.empty_cache()
-    except Exception:
-        pass
-    # ESCALATING RIDGE, ON THE GPU, BEFORE LEAVING IT. A rank-deficient Gram is the common
-    # case on unbalanced splits (small first mode), not a rare accident, so the host fallback
-    # must not be the routine path: CPU float64 gelsd on a few-thousand-square system costs
-    # minutes, and ALS calls this 3 times per sweep per restart, which turned a Qwen2.5-3B
-    # sweep that should take minutes into one that produced no output for 40+ minutes. Tikhonov
-    # regularisation restores positive-definiteness for a cost the fit does not notice: the
-    # ridge is relative to the Gram's own scale and is escalated only until a solve succeeds.
-    scale = float(AtA.diagonal().abs().mean().clamp_min(1e-30))
-    eye = torch.eye(AtA.shape[0], dtype=AtA.dtype, device=AtA.device)
-    # THE LADDER MUST TERMINATE ON THE GPU. AtA is positive SEMI-definite by construction, so
-    # AtA + c*I is positive definite for any c > 0 and Cholesky is guaranteed to succeed once
-    # the ridge is large enough -- singularity therefore never needs the host at all. The
-    # previous last rung sent it to CPU float64 gelsd, which is single-threaded, unbounded, and
-    # does not check Python signals: on Qwen2.5-3B's k/v_proj, whose TT bond lands exactly on
-    # the mode dimension (r1 = d0 = 32), the ALS subproblem is EXACTLY rank-saturated, every
-    # solve took that path, and a compression that needs ~1s per layer without ALS ran 48
-    # minutes for one layer and could not even be interrupted with SIGINT.
-    #
-    # The ridge distorts the subproblem, but ALS is iterative and fit_tr keeps the best iterate
-    # with a never-worse-than-TT guard, so a heavily-ridged sweep can only fail to improve --
-    # it cannot corrupt the result. An unbounded stall can.
-    for mult in (1e-6, 1e-4, 1e-2, 1e-1, 1.0, 10.0):
-        try:
-            L = torch.linalg.cholesky(AtA + (mult * scale) * eye)
-            return torch.cholesky_solve(AtW, L)
-        except torch.OutOfMemoryError:
-            torch.cuda.empty_cache()
-            break                      # a memory problem, not a conditioning one -- go to host
-        except Exception:
-            continue
-    del eye
-    # Only OOM reaches here now. The host path stays for that case, where it IS the right
-    # answer: slow but bounded beats failing outright on the layers that need the most rank.
-    dev = AtA.device
-    A64, W64 = AtA.cpu().double(), AtW.cpu().double()
-    try:
-        X = torch.linalg.solve(A64, W64)
-    except Exception:
-        X = torch.linalg.lstsq(A64, W64, driver="gelsd").solution
-    return X.to(device=dev, dtype=AtW.dtype)
-
-
-def tr_rel_err(cores, R: torch.Tensor, chunk_elems: int = 250_000_000) -> float:
-    """||contract_tr(cores) - R||_F / ||R||_F, accumulated in chunks.
-
-    Materialising the reconstruction to subtract it needs the reconstruction AND the einsum's
-    intermediate at once; on down_proj at ring rank 155 that intermediate is (155, 64, 688,
-    256) = 7 GB even with the first mode already chunked. Accumulating the squared error one
-    slice at a time keeps only one slice live, and the chunk width is derived from an element
-    budget rather than a hard-coded constant -- a constant is exactly what silently failed to
-    engage when a mode happened to equal it.
-    """
-    n = len(cores)
-    d0 = cores[0].shape[1]
-    mid = 1
-    for k in range(1, n - 1):
-        mid *= cores[k].shape[1]
-    per_unit = max(1, mid * cores[0].shape[0] * cores[-1].shape[0])
-    step = max(1, min(d0, int(chunk_elems // per_unit)))
-    acc = 0.0
-    for lo in range(0, d0, step):
-        hi = min(lo + step, d0)
-        sub = [cores[0][:, lo:hi, :]] + list(cores[1:])
-        diff = contract_tr(sub) - R[lo:hi]
-        acc += float(diff.pow(2).sum())
-        del sub, diff
-    return (acc ** 0.5) / float(R.norm())
-
-
-def fit_tr(R: torch.Tensor, ranks, sweeps: int = 30, tol: float = 1e-8, verbose: bool = False,
-           seed: int = 1234, restarts: int = 2, solver: str = "lstsq",
-           chunk: int = 0, chunk_elems: int = 250_000_000):
-    """Fit a tensor ring by TT initialisation + alternating least squares.
-
-    ranks = [r_0, r_1, ..., r_{N-1}] with r_N == r_0. The TT init uses r_0 = 1 (a legal ring),
-    then each ALS sweep re-solves every core in closed form given the others; the ring's extra
-    boundary rank is opened up by padding the initialisation. Because ALS on a ring is not
-    guaranteed to improve monotonically from an arbitrary start, the best iterate seen is kept
-    and the routine REFUSES to return anything worse than its own TT initialisation -- without
-    that guard a fitting failure would be indistinguishable from the format being worse, which
-    is precisely the comparison this module exists to make.
-    """
-    n = len(R.shape)
-    if len(ranks) != n:
-        raise ValueError(f"a ring over {n} sites needs {n} ranks (r_0..r_{{N-1}})")
-    dims = list(R.shape)
-    rnorm = float(R.norm())
-    rnorm2 = float(R.pow(2).sum())
-    last_sq = None
-    tt = fit_tt(R, ranks[1:])
-    base_err = float((contract_tt(tt) - R).norm() / R.norm())
-    r0 = int(ranks[0])
-    gen = torch.Generator(device="cpu").manual_seed(seed)
-
-    def padded_init():
-        """TT solution embedded in the ring, with the OPENED bond seeded on BOTH ends.
-
-        Seeding only one end is not enough and is a silent failure: core 0's least-squares
-        environment is built from cores 1..N-1, whose ring index comes from core N-1's
-        trailing bond. If that stays zero, the environment vanishes on exactly the components
-        the ring adds, every ALS solve returns zero there, and the fit never leaves the TT
-        solution -- it looks like "the ring buys nothing" rather than "the ring was never
-        initialised".
-        """
-        out = []
-        for k, c in enumerate(tt):
-            a, d, b = c.shape
-            na = r0 if k == 0 else a
-            nb = r0 if k == n - 1 else b
-            g = torch.zeros(na, d, nb, dtype=R.dtype, device=R.device)
-            g[:a, :, :b] = c
-            eps = 0.05 * float(c.norm()) / max(1, c.numel()) ** 0.5
-            if k == 0 and na > a:
-                g[a:, :, :] = eps * torch.randn(na - a, d, nb, generator=gen,
-                                                dtype=R.dtype).to(R.device)
-            if k == n - 1 and nb > b:
-                g[:, :, b:] = eps * torch.randn(na, d, nb - b, generator=gen,
-                                                dtype=R.dtype).to(R.device)
-            out.append(g)
-        return out
-
-    def random_init(g=None):
-        """i.i.d. cores whose ring contraction has the right variance.
-
-        The scale is derived, not guessed. For cores with i.i.d. N(0, s^2) entries the trace
-        of the cyclic product is a sum of prod_k r_k independent products of N Gaussians, so
-        Var(W) ~= (prod_k r_k) * s^(2N). Matching that to the target mean square gives
-        s = (E[R^2] / prod_k r_k)^(1/(2N)). Getting this wrong by a constant factor per core
-        compounds to that factor^N in the product, which for N=4,5 puts the initial iterate
-        many orders of magnitude away from the data and leaves ALS crawling.
-        """
-        rr = [int(ranks[k]) for k in range(n)]
-        prod = 1.0
-        for x in rr:
-            prod *= x
-        ms = float(R.pow(2).mean())
-        s = (ms / prod) ** (1.0 / (2 * n))
-        gg = g if g is not None else gen
-        return [s * torch.randn(rr[k], dims[k], rr[(k + 1) % n], generator=gg,
-                                dtype=R.dtype).to(R.device) for k in range(n)]
-
-    # ALS on a ring has genuine local minima -- a single start that stalls would be reported
-    # as "the ring is not better", which is exactly the confound this module must avoid. So
-    # several starts are run and the best kept; restarts are cheap next to the SVDs anyway.
-    inits = [("tt_padded", padded_init)]
-    for j in range(restarts):
-        g = torch.Generator(device="cpu").manual_seed(seed + 977 * (j + 1))
-        inits.append((f"random{j}", lambda g=g: random_init(g)))
-    best, best_err, hists, winner = tt_to_tr(tt), base_err, {}, "tt_init"
-    for init_name, init_fn in inits:
-        cores, ihist = init_fn(), []
-        for it in range(sweeps):
-            # A restart that has diverged produces non-finite normal equations. Feeding
-            # those to a solver is not merely wasteful: cuSOLVER's LU on a NaN matrix
-            # does not fail fast, it goes pathological -- a 2336x2336 solve sat for 80
-            # minutes and could not be interrupted, because `clamp_min` does not clamp
-            # NaN so the ridge itself became NaN and poisoned the whole diagonal.
-            diverged = False
-            for k in range(n):
-                # The subchain leaves every d index separate and in CYCLIC order starting at
-                # k+1. R's remaining modes must be flattened in that SAME order -- movedim
-                # gives natural order, which for interior k is a different permutation and
-                # would solve the least-squares problem against mismatched columns. It still
-                # returns a finite "solution", just to the wrong system.
-                # The chunk decision is made from SHAPES ONLY and BEFORE anything large is
-                # built. An earlier version computed the full subchain first and then decided
-                # whether to chunk, so the unchunked allocation always happened anyway and the
-                # chunked branch saved nothing -- the memory-bounded path existed but could
-                # never take effect.
-                # Auto-size the chunk from a memory budget instead of comparing against a
-                # fixed number. A constant threshold silently does nothing when the mode
-                # happens to equal it -- which is exactly what let a 4-site fit try to
-                # materialise a 134 GB subchain (mode 64, chunk 64, "64 > 64" is false).
-                m_ = (k + 1) % n
-                other = 1
-                for j in range(n):
-                    if j != k:
-                        other *= dims[j]
-                per_unit = max(1, (other // max(1, dims[m_])) * rank_of(cores, k))
-                auto = max(1, min(dims[m_], int(chunk_elems // per_unit)))
-                use_chunk = auto < dims[m_]
-                if solver == "normal" and use_chunk:
-                    # Memory-bounded normal equations. Materialising the subchain costs
-                    # r_k * (prod_{j!=k} d_j) * r_{k-1} floats -- on down_proj at these ranks
-                    # that is (155, 688, 256, 155) = 17 GB, so the unchunked path cannot run
-                    # on the MLP layers at all. The Gram and right-hand side are sums over the
-                    # flattened "others" index, so they accumulate exactly over slices of the
-                    # first subchain mode; this is the same arithmetic, not an approximation.
-                    m = m_
-                    d_next = dims[m]
-                    rk_, rkm1_ = None, None
-                    AtA = AtW = None
-                    Rp = R.permute(*([k] + [(k + 1 + t) % n for t in range(n - 1)]))
-                    for lo in range(0, d_next, auto):
-                        hi = min(lo + auto, d_next)
-                        Ec = _subchain_chunk(cores, k, lo, hi)
-                        rk_, rkm1_ = Ec.shape[0], Ec.shape[-1]
-                        Ec = Ec.reshape(rk_, -1, rkm1_)
-                        Ac = Ec.permute(1, 2, 0).reshape(Ec.shape[1], rkm1_ * rk_)
-                        Wc = Rp[:, lo:hi].reshape(dims[k], -1)
-                        g = Ac.transpose(0, 1) @ Ac
-                        h = Ac.transpose(0, 1) @ Wc.transpose(0, 1)
-                        AtA = g if AtA is None else AtA + g
-                        AtW = h if AtW is None else AtW + h
-                        del Ec, Ac, Wc, g, h
-                    if not torch.isfinite(AtA).all():
-                        diverged = True
-                        del AtA, AtW
-                        break
-                    lam = 1e-8 * float(AtA.diagonal().abs().mean().clamp_min(1e-30))
-                    AtA_r = AtA + lam * torch.eye(AtA.shape[0], dtype=AtA.dtype,
-                                                  device=AtA.device)
-                    try:
-                        G = _solve_maybe_cpu(AtA_r, AtW)
-                    except ValueError:
-                        # Non-finite normal equations mean THIS restart has diverged, whichever
-                        # operand carries the NaN -- checking AtA alone missed a case where the
-                        # right-hand side A^T W was the non-finite one and killed the whole
-                        # compression at layer 15 of 252. Catching the guard covers every
-                        # source at once, and abandoning the restart is safe because `best` is
-                        # untouched and fit_tr still falls back to the TT solution.
-                        diverged = True
-                        del AtA_r, AtW
-                        break
-                    del AtA_r
-                    # ||R - A G||^2 = ||R||^2 - 2<AtW, G> + <G, AtA G>, all from quantities
-                    # already formed. Computing it by contracting the whole ring instead would
-                    # cost one full O(prod d_k) tensor per sweep per core.
-                    sq = rnorm2 - 2.0 * float((AtW * G).sum()) + float((G * (AtA @ G)).sum())
-                    last_sq = max(sq, 0.0)
-                    rkm1, rk = rkm1_, rk_
-                else:
-                    # ONE branch builds A/Wk and ONE consumes them. Splitting the decision
-                    # across two separate if/else chains left the non-chunked solve reachable
-                    # after the chunked path had run, where `A` is whatever the PREVIOUS core
-                    # left behind -- a silent wrong answer rather than an error.
-                    E = _subchain(cores, k)               # (r_k, d_{k+1},..,d_{k-1}, r_{k-1})
-                    rk, rkm1 = E.shape[0], E.shape[-1]
-                    E = E.reshape(rk, -1, rkm1)
-                    D = E.shape[1]
-                    A = E.permute(1, 2, 0).reshape(D, rkm1 * rk)
-                    perm = [k] + [(k + 1 + t) % n for t in range(n - 1)]
-                    # R's remaining modes must be flattened in the subchain's CYCLIC order --
-                    # movedim gives natural order, which for interior k solves the
-                    # least-squares problem against mismatched columns and still returns a
-                    # finite "solution", just to the wrong system.
-                    Wk = R.permute(*perm).reshape(dims[k], D)
-                    del E
-                    if solver == "normal":
-                        AtA = A.transpose(0, 1) @ A
-                        if not torch.isfinite(AtA).all():
-                            diverged = True
-                            del AtA, A, Wk
-                            break
-                        lam = 1e-8 * float(AtA.diagonal().abs().mean().clamp_min(1e-30))
-                        AtA += lam * torch.eye(AtA.shape[0], dtype=A.dtype, device=A.device)
-                        try:
-                            G = _solve_maybe_cpu(AtA, A.transpose(0, 1) @ Wk.transpose(0, 1))
-                        except ValueError:
-                            diverged = True
-                            del AtA, A, Wk
-                            break
-                        del AtA
-                    else:
-                        G = torch.linalg.lstsq(A, Wk.transpose(0, 1)).solution
-                    last_sq = None
-                    del A, Wk
-                cores[k] = (G.transpose(0, 1).reshape(dims[k], rkm1, rk)
-                             .permute(1, 0, 2).contiguous())
-            if diverged:
-                # Abandon this initialisation. `best`/`best_err` are untouched, and fit_tr's
-                # never-worse-than-TT guard still applies, so a diverged restart can only fail
-                # to contribute -- never corrupt the returned cores.
-                ihist.append(float("nan"))
-                break
-            err = (float(last_sq) ** 0.5 / rnorm if last_sq is not None
-                   else float((contract_tr(cores, chunk=chunk) - R).norm() / rnorm))
-            ihist.append(round(err, 8))
-            if verbose:
-                print(f"    TR-ALS[{init_name}] sweep {it+1}: rel err {err:.6f}", flush=True)
-            if err < best_err - 1e-12:
-                best, best_err, winner = [c.clone() for c in cores], err, init_name
-            if it and abs(ihist[-2] - err) < tol * max(err, 1e-12):
-                break
-        hists[init_name] = ihist
-    # The algebraic residual is a difference of nearly equal large numbers; at a relative
-    # error of ~1e-2 it has already lost most of its fp32 significance. It is fine for
-    # steering ALS but not for the number that gets reported, so the winner is re-measured
-    # once by explicit contraction.
-    best_err = tr_rel_err(best, R)
-    if best_err > base_err:
-        return tt_to_tr(tt), base_err, {"fell_back_to_tt": True, "tt_err": base_err,
-                                        "winner": "tt_init", "als_hist": hists}
-    return best, best_err, {"fell_back_to_tt": winner == "tt_init", "tt_err": base_err,
-                            "winner": winner, "als_hist": hists}
-
-
-# --------------------------------------------------------------------------- bookkeeping
-def tt_params(dims, ranks) -> int:
-    r = [1] + [int(x) for x in ranks] + [1]
-    return sum(r[k] * dims[k] * r[k + 1] for k in range(len(dims)))
-
-
-def tr_params(dims, ranks) -> int:
-    r = [int(x) for x in ranks]
-    n = len(dims)
-    return sum(r[k] * dims[k] * r[(k + 1) % n] for k in range(n))
-
-
-def tt_rank_ceilings(dims):
-    """r_k <= min(prod_{j<=k} d_j, prod_{j>k} d_j): above that the truncation is vacuous."""
-    n, out = len(dims), []
-    for k in range(n - 1):
-        left = 1
-        for j in range(k + 1):
-            left *= dims[j]
-        right = 1
-        for j in range(k + 1, n):
-            right *= dims[j]
-        out.append(min(left, right))
-    return out
-
-
-def ranks_for_budget(dims, budget: int, ring: bool, alpha: float = 1.0):
-    """Largest ranks fitting `budget`, shaped by r_k proportional to ceiling_k ** alpha.
-
-    Bisection rather than a closed form: the count is linear in each rank but bilinear in
-    adjacent pairs, and cyclic for a ring, so there is no single quadratic to solve.
-
-    ALPHA EXISTS BECAUSE EQUAL FILL FRACTION IS WRONG WHEN THE CEILINGS ARE LOPSIDED. For a
-    4-site split of 4096 the ceilings are (64, 4096, 64): alpha=1 spends the budget on the
-    huge middle bond and returns ranks like (36, 2327, 36), strangling the two bonds that
-    actually gate information flow through the chain. alpha=0 gives every bond the same rank
-    instead. Neither is right a priori, so the caller is expected to MEASURE across alpha --
-    the reconstruction error, not the heuristic, decides.
-    """
-    ceil_tt = tt_rank_ceilings(dims)
-    ceils = ([min(ceil_tt)] + ceil_tt) if ring else ceil_tt
-    fn = tr_params if ring else tt_params
-    mx = max(ceils)
-    shape = [(c / mx) ** alpha for c in ceils]
-
-    def build(t):
-        return [max(1, min(int(round(t * mx * s)), c)) for s, c in zip(shape, ceils)]
-
-    lo, hi = 0.0, 1.0
-    if fn(dims, build(1.0)) <= budget:
-        return build(1.0)
-    for _ in range(60):
-        mid = (lo + hi) / 2
-        if fn(dims, build(mid)) > budget:
-            hi = mid
-        else:
-            lo = mid
-    r = build(lo)
-    while fn(dims, r) > budget and max(r) > 1:
-        r[r.index(max(r))] -= 1
-    return r
-
-
-def balanced_split(n: int, k: int):
-    """Most balanced k-way factorisation of n with every factor >= 2 (spread-minimising)."""
-    best = None
-    def rec(rem, parts):
-        nonlocal best
-        if len(parts) == k - 1:
-            if rem >= 2:
-                cand = parts + [rem]
-                key = max(cand) / min(cand)
-                if best is None or key < best[0]:
-                    best = (key, cand)
-            return
-        d = 2
-        while d * d <= rem or d <= rem:
-            if d > rem:
-                break
-            if rem % d == 0:
-                rec(rem // d, parts + [d])
-            d += 1
-    rec(n, [])
-    if best is None:
-        raise ValueError(f"no {k}-way factorisation of {n} with all factors >= 2")
-    return sorted(best[1])
-
-
-def pair_modes(o: int, i: int, k: int = 3):
-    """Split O and I into k factors each, and PAIR them so the paired dims come out balanced.
-
-    Paired-mode tensorisation multiplies the j-th output factor by the j-th input factor, so
-    the pairing ORDER decides the shape of the tensor the ring has to fit -- and balanced_split
-    returns its factors sorted, which pairs smallest with smallest and largest with largest.
-    That is the worst available choice. On Qwen2.5-3B's 2048x2048 projections both sides split
-    as [8, 16, 16], and the sorted pairing yields dims [64, 256, 256]: a first mode of only 64
-    caps the adjacent bond at r_0 * 64, while a budget-matched sweep asks for r ~= 92, so every
-    ALS normal-equation system is rank-deficient BY CONSTRUCTION. The observed cost was a sweep
-    that burned 2.5 hours of CPU inside the ridge/lstsq ladder and emitted not one data point.
-
-    Searching the k! pairings for the one that minimises max(dims)/min(dims) turns that same
-    layer into [128, 128, 256], which is well conditioned. The search is over 6 orders for
-    k = 3, so it costs nothing.
-
-    This is deliberately a no-op on LLaMA-2-7B, whose shapes were verified individually:
-    q/k/v/o_proj [256, 256, 256], gate/up_proj and down_proj [256, 256, 688] both before and
-    after. Every result already measured on that model therefore remains valid, and the sweep
-    and the production compressor keep agreeing with each other.
-    """
-    from itertools import permutations
-    osp = tuple(balanced_split(o, k))
-    best = None
-    for perm in permutations(balanced_split(i, k)):
-        dims = [a * b for a, b in zip(osp, perm)]
-        spread = max(dims) / min(dims)
-        if best is None or spread < best[0]:
-            best = (spread, tuple(perm))
-    return osp, best[1]
-
-
-def self_test(device: str = "cpu") -> None:
-    torch.manual_seed(0)
-    dt = torch.float64
-    for osp, isp in [((4, 3), (2, 5)), ((4, 3, 2), (2, 5, 3)), ((2, 3, 2, 2), (2, 2, 3, 2))]:
-        n = len(osp)
-        out = 1
-        inn = 1
-        for x in osp:
-            out *= x
-        for x in isp:
-            inn *= x
-        W = torch.randn(out, inn, dtype=dt, device=device)
-        R = rearrange_n(W, osp, isp)
-        dims = [o * i for o, i in zip(osp, isp)]
-        assert list(R.shape) == dims, (R.shape, dims)
-        assert torch.equal(inv_rearrange_n(R, osp, isp), W), f"{n}-site rearrange not invertible"
-        # index identity, not just the round trip: a wrong-but-self-inverse permutation would
-        # pass the round trip and scramble every weight silently
-        for _ in range(40):
-            oi = [torch.randint(0, x, (1,)).item() for x in osp]
-            ii = [torch.randint(0, x, (1,)).item() for x in isp]
-            ridx = tuple(oi[k] * isp[k] + ii[k] for k in range(n))
-            orow = 0
-            for k in range(n):
-                orow = orow * osp[k] + oi[k]
-            ocol = 0
-            for k in range(n):
-                ocol = ocol * isp[k] + ii[k]
-            assert R[ridx] == W[orow, ocol], f"{n}-site rearrange index map wrong"
-
-        ceils = tt_rank_ceilings(dims)
-        cores = fit_tt(R, ceils)
-        e = float((contract_tt(cores) - R).norm() / R.norm())
-        assert e < 1e-10, f"{n}-site full-rank TT is not exact ({e:.3e})"
-        assert tt_params(dims, ceils) == sum(c.numel() for c in cores), "tt_params disagrees"
-
-        if n >= 3:
-            trunc = [max(1, c // 2) for c in ceils]
-            cores = fit_tt(R, trunc)
-            assert tt_params(dims, trunc) == sum(c.numel() for c in cores)
-            for a in (0.0, 0.5, 1.0):
-                rr = ranks_for_budget(dims, tt_params(dims, trunc), ring=False, alpha=a)
-                assert tt_params(dims, rr) <= tt_params(dims, trunc), \
-                    f"TT budget solver overran at alpha={a}"
-            # a ring at the SAME parameter budget must not do worse than the TT it starts from
-            budget = tt_params(dims, trunc)
-            rring = ranks_for_budget(dims, budget, ring=True)
-            assert tr_params(dims, rring) <= budget, "TR budget solver overran"
-            _c, err_tr, info = fit_tr(R, rring, sweeps=6)
-            assert err_tr <= info["tt_err"] + 1e-12, "fit_tr returned worse than its own TT init"
-            assert abs(float((contract_tr(_c) - R).norm() / R.norm()) - err_tr) < 1e-8, \
-                "reported TR error does not match the returned cores"
-
-    ring_chk = [torch.randn(3, 5, 4, dtype=dt), torch.randn(4, 6, 2, dtype=dt),
-                torch.randn(2, 7, 3, dtype=dt)]
-    full = contract_tr(ring_chk)
-    assert torch.allclose(full, contract_tr(ring_chk, chunk=2), atol=1e-12), \
-        "chunked contract_tr disagrees with the unchunked one"
-    ref = torch.einsum("iaj,jbk,kci->abc", *ring_chk)
-    assert torch.allclose(full, ref, atol=1e-12), "contract_tr is not the cyclic trace"
-
-    # a ring with r_0 > 1 must be able to represent things a TT of the same internal ranks cannot
-    dims = [6, 6, 6]
-    torch.manual_seed(1)
-    ring = [torch.randn(3, 6, 3, dtype=dt) for _ in range(3)]
-    Rr = contract_tr(ring)
-    tt_same = fit_tt(Rr, [3, 3])
-    e_tt = float((contract_tt(tt_same) - Rr).norm() / Rr.norm())
-    assert e_tt > 1e-3, ("a rank-(3,3,3) ring was reproduced by a rank-(3,3) TT; the ring "
-                         f"contraction is probably not closing the loop (err {e_tt:.3e})")
-    for slv in ("lstsq", "normal"):
-        got, err, info = fit_tr(Rr, [3, 3, 3], sweeps=200, restarts=3, solver=slv)
-        assert err < 1e-4, (f"TR-ALS[{slv}] failed to recover an exact rank-3 ring "
-                            f"(err {err:.3e}); winner={info['winner']} "
-                            f"tt_init={info['tt_err']:.3e}")
-    got, err, info = fit_tr(Rr, [3, 3, 3], sweeps=200, restarts=3)
-    # chunking changes only how the Gram is accumulated, so it must reach the same optimum
-    _g, err_ch, _i = fit_tr(Rr, [3, 3, 3], sweeps=200, restarts=3, solver="normal", chunk=2)
-    assert err_ch < 1e-4, f"chunked TR-ALS failed to recover an exact ring (err {err_ch:.3e})"
-    assert err < 1e-4, (f"TR-ALS failed to recover an exact rank-3 ring (err {err:.3e}); "
-                        f"winner={info['winner']} tt_init={info['tt_err']:.3e}")
-
-    assert balanced_split(4096, 3) == [16, 16, 16], balanced_split(4096, 3)
-    assert balanced_split(4096, 2) == [64, 64]
-    b = balanced_split(11008, 3)
-    assert 43 in b and b[0] * b[1] * b[2] == 11008, b
-    print("tensor_formats self-test PASSED (N-site rearrange exact & index-checked, full-rank "
-          "TT exact, budget solvers within budget, TR >= its TT init, TR strictly beats TT on "
-          "a true ring, balanced splits correct)")
-
-
-if __name__ == "__main__":
-    self_test()
-
-
-# --------------------------------------------------------------------------- layers
-class TRLinear(torch.nn.Module):
-    """Tensor-ring linear layer: W = Tr[G_0 G_1 ... G_{N-1}] after the shared rearrangement.
-
-    NOT a LowRankLinear subclass, unlike MPO2Linear/MPO3Linear. Those two could pretend to be
-    a pair of factors because a train has open ends; a ring genuinely has N 3-D cores and no
-    A/B roles to borrow, so anything keyed on `.A`/`.B` (the blockwise quantizer, the bond
-    rebalancer) has to be taught about `.cores` explicitly rather than silently doing the
-    wrong thing to a subclass it thinks it understands.
-
-    RECONSTRUCTION IS CHUNKED BY DEFAULT. Merging any two cores of a 3-site ring costs
-    r*d_i*d_j*r entries -- on down_proj at ring rank 142 that is 7.1 GB in bf16, per layer,
-    per forward. Chunking the first mode bounds it; the cost is more, smaller matmuls.
-    """
-
-    def __init__(self, cores, osplit, isplit, chunk_elems: int = 500_000_000,
-                 bias=None):
-        super().__init__()
-        n = len(cores)
-        if len(osplit) != n or len(isplit) != n:
-            raise ValueError(f"{n} cores need {n} output and {n} input modes")
-        if min(list(osplit) + list(isplit)) < 2:
-            raise ValueError(f"degenerate split o={tuple(osplit)} i={tuple(isplit)}: a mode of "
-                             f"1 collapses a site, making this an (N-1)-site network")
-        for k in range(n):
-            if cores[k].shape[1] != osplit[k] * isplit[k]:
-                raise ValueError(f"core {k} has mode {cores[k].shape[1]}, expected "
-                                 f"{osplit[k]}*{isplit[k]}")
-            if cores[k].shape[2] != cores[(k + 1) % n].shape[0]:
-                raise ValueError(f"ring does not close between cores {k} and {(k+1)%n}")
-        self.cores = torch.nn.ParameterList([torch.nn.Parameter(c) for c in cores])
-        self.osplit, self.isplit = tuple(int(x) for x in osplit), tuple(int(x) for x in isplit)
-        o = i = 1
-        for k in range(n):
-            o *= self.osplit[k]
-            i *= self.isplit[k]
-        self.out_features, self.in_features = o, i
-        self.ranks = tuple(int(c.shape[0]) for c in cores)
-        # LLaMA projections have no bias, but Qwen2's q/k/v DO. Dropping it when the dense
-        # Linear is replaced is SILENT: shapes still match, the parameter-count gate still
-        # balances (achieved excludes bias, and so would the rebuilt model), and only the
-        # accuracy moves. So the bias is carried, not discarded. Defaults to None, which
-        # leaves every LLaMA path byte-identical to before.
-        self.bias = None if bias is None else torch.nn.Parameter(bias)
-        # Elements per unit of the FIRST mode in contract_tr's peak intermediate, which is
-        # (r_0, c, d_1, ..., d_{n-2}, r_{n-1}) -- the LAST core's mode is contracted away by
-        # the closing einsum and must not be counted. Including it over-estimated the cost by
-        # d_{n-1} (256x on an MLP layer), which drove the chunk to 1 and would have made every
-        # MLP forward 256 sequential contractions instead of 8.
-        mid = 1
-        for k in range(1, n - 1):
-            mid *= cores[k].shape[1]
-        per_unit = max(1, mid * self.ranks[0] * self.ranks[-1])
-        self._chunk = max(1, min(cores[0].shape[1], int(chunk_elems // per_unit)))
-        # non-persistent: never wants to show up in a checkpoint, and materialize_() is an
-        # inference-only opt-in (see below), so a freshly-loaded layer must start uncached.
-        self.register_buffer("_cached_weight", None, persistent=False)
-
-    def weight(self) -> torch.Tensor:
-        if self._cached_weight is not None:
-            return self._cached_weight
-        cs = list(self.cores)
-        R = contract_tr(cs, chunk=self._chunk if self._chunk < cs[0].shape[1] else 0)
-        return inv_rearrange_n(R, self.osplit, self.isplit)
-
-    @torch.no_grad()
-    def materialize_(self):
-        """Cache the dense weight once. INFERENCE ONLY -- gradients cannot flow through the
-        cached tensor back to the cores, so this must never be called before/during training.
-
-        forward() calls weight() on every call, which redoes the full chunked ring
-        contraction (two tensor contractions + a trace per core) from scratch. That is fine
-        for a single forward pass, but autoregressive generation calls forward() once per
-        layer per GENERATED TOKEN with the cores completely unchanged between calls. Measured
-        on tr3_ring4b04_h1's GSM8K eval before this existed: 209s/item, ETA 73h for 1319
-        items -- every one of ~200 generated tokens re-contracted all 224 ring layers.
-        A train layer (LowRankLinear/MPO2Linear) never had this problem: F.linear(F.linear(x,
-        B), A) applies the two factors straight to the activation and never materializes W at
-        all, so there was nothing to cache there.
-        """
-        self._cached_weight = self.weight().detach().clone()
-
-    def unmaterialize_(self):
-        self._cached_weight = None
-
-    def forward(self, x):
-        b = None if self.bias is None else self.bias.to(x.dtype)
-        return torch.nn.functional.linear(x, self.weight().to(x.dtype), b)
-
-    def extra_repr(self) -> str:
-        return (f"o={self.osplit}, i={self.isplit}, ranks={self.ranks}, "
-                f"chunk={self._chunk}, materialized={self._cached_weight is not None}")
-
-
-class SparseAugTRLinear(TRLinear):
-    """TRLinear plus a sparse additive residual. Same int64-index contract as the MPO layers:
-    `sp_idx` must NEVER be cast to a floating dtype -- a `.half()` on a 45M-scale index
-    silently rounds it to a different, valid-looking position."""
-
-    def __init__(self, cores, osplit, isplit, sp_idx, sp_val,
-                 chunk_elems: int = 500_000_000, bias=None):
-        super().__init__(cores, osplit, isplit, chunk_elems, bias)
-        if sp_idx.numel() != sp_val.numel():
-            raise ValueError(f"{sp_idx.numel()} indices for {sp_val.numel()} values")
-        n = self.out_features * self.in_features
-        if sp_idx.numel() and (int(sp_idx.max()) >= n or int(sp_idx.min()) < 0):
-            raise ValueError("sparse index out of range for this layer")
-        self.register_buffer("sp_idx", sp_idx.to(torch.int64))
-        self.sp_val = torch.nn.Parameter(sp_val)
-
-    def weight(self) -> torch.Tensor:
-        # must check the cache here too, not just in TRLinear.weight(): materialize_() caches
-        # THIS class's weight() (ring + sparse, since self.weight() dispatches polymorphically
-        # from inside materialize_()), so without this check a later call would go
-        # super().weight() -> see the cache -> return the already-sparse-augmented tensor ->
-        # and then add the sparse residual a second time on top of it.
-        if self._cached_weight is not None:
-            return self._cached_weight
-        W = super().weight()
-        if self.sp_idx.numel():
-            W = W.reshape(-1).index_add(0, self.sp_idx, self.sp_val.to(W.dtype)).view_as(W)
-        return W
-
-    def extra_repr(self) -> str:
-        return super().extra_repr() + f", nnz={self.sp_idx.numel()}"
-
-
-def layer_self_test() -> None:
-    torch.manual_seed(0)
-    osp, isp = (4, 3, 2), (2, 5, 3)
-    dims = [o * i for o, i in zip(osp, isp)]
-    cores = [torch.randn(3, dims[0], 4), torch.randn(4, dims[1], 2), torch.randn(2, dims[2], 3)]
-    lay = TRLinear([c.clone() for c in cores], osp, isp, chunk_elems=10 ** 9)
-    ref = inv_rearrange_n(contract_tr(cores), osp, isp)
-    assert torch.allclose(lay.weight(), ref, atol=1e-5), "TRLinear.weight() != contract+inverse"
-    # chunking must not change the answer
-    lay_c = TRLinear([c.clone() for c in cores], osp, isp, chunk_elems=1)
-    assert lay_c._chunk < dims[0], "chunk did not engage"
-    assert torch.allclose(lay_c.weight(), ref, atol=1e-5), "chunked forward differs"
-    x = torch.randn(7, lay.in_features)
-    assert torch.allclose(lay(x), x @ ref.T, atol=1e-4), "forward != x W^T"
-    idx = torch.tensor([0, 5, lay.out_features * lay.in_features - 1], dtype=torch.int64)
-    val = torch.tensor([1.0, -2.0, 3.0])
-    sl = SparseAugTRLinear([c.clone() for c in cores], osp, isp, idx, val, chunk_elems=10 ** 9)
-    d = (sl.weight() - ref).reshape(-1)
-    assert torch.allclose(d[idx], val, atol=1e-5), "sparse residual not applied at sp_idx"
-    assert abs(float(d.abs().sum()) - float(val.abs().sum())) < 1e-4, "residual leaked"
-    try:
-        TRLinear([torch.zeros(2, 4, 2), torch.zeros(2, 3, 2), torch.zeros(2, 6, 2)],
-                 (4, 1, 2), (1, 3, 3))
-    except ValueError:
-        pass
-    else:
-        raise AssertionError("a degenerate split (mode = 1) was accepted")
-    # gradients must reach every core
-    lay.zero_grad()
-    lay(x).sum().backward()
-    for k, c in enumerate(lay.cores):
-        assert c.grad is not None and float(c.grad.abs().sum()) > 0, f"core {k} got no gradient"
-    # materialize_() must be a pure inference-speed cache: same weight, same forward output,
-    # and -- the one bug this is actually guarding against -- the sparse residual must not
-    # get added twice once SparseAugTRLinear.weight() is short-circuited by the cache.
-    pre = lay.weight().clone()
-    lay.materialize_()
-    assert lay._cached_weight is not None, "materialize_() did not populate the cache"
-    assert torch.allclose(lay.weight(), pre, atol=1e-6), "materialize_() changed weight()"
-    assert torch.allclose(lay(x), x @ ref.T, atol=1e-4), "forward != x W^T after materialize_()"
-    lay.unmaterialize_()
-    assert lay._cached_weight is None, "unmaterialize_() left the cache populated"
-    sl_ref = sl.weight().clone()
-    sl.materialize_()
-    assert torch.allclose(sl.weight(), sl_ref, atol=1e-6), \
-        "materialized SparseAugTRLinear.weight() differs -- sparse residual likely double-added"
-    print("TRLinear self-test PASSED (weight matches contraction, chunking exact, forward "
-          "correct, sparse localised, degenerate split refused, all cores differentiable, "
-          "materialize_() cache exact and idempotent)")