Hugging Face's logo

Tensor-compression
/
TR_main

tensor-decomposition
svd
low-rank
model-compression
knowledge-distillation
Model card Files
xet
 Community
Rhett1 commited on
Commit
6a72563
·
verified ·
1 Parent(s): 1242644

Add matched-budget SVD-LLM / SVD-LLM V2 KD-healed baselines

Browse files
Files changed (13) hide show
  1. README.md +15 -0
  2. svd-llm-v2/README.md +43 -0
  3. svd-llm-v2/config.json +28 -0
  4. svd-llm-v2/lowrank_linear.py +83 -0
  5. svd-llm-v2/provenance.json +196 -0
  6. svd-llm-v2/ranks.json +226 -0
  7. svd-llm-v2/state_dict_fp16.pt +3 -0
  8. svd-llm/README.md +43 -0
  9. svd-llm/config.json +28 -0
  10. svd-llm/lowrank_linear.py +83 -0
  11. svd-llm/provenance.json +196 -0
  12. svd-llm/ranks.json +226 -0
  13. svd-llm/state_dict_fp16.pt +3 -0
README.md CHANGED
@@ -46,3 +46,18 @@ budget sweep on Qwen2.5-3B shows that model collapses to chance at the ~40% used
46
  LLaMA-2 rows, so it is compressed to 32.65%, where its pre-heal KL (0.8882) sits in the same
47
  range as LLaMA-2-7B's at 40.30% (0.6915). Details and the full sweep are in
48
  `qwen2.5-3b/README.md`.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
46
  LLaMA-2 rows, so it is compressed to 32.65%, where its pre-heal KL (0.8882) sits in the same
47
  range as LLaMA-2-7B's at 40.30% (0.6915). Details and the full sweep are in
48
  `qwen2.5-3b/README.md`.
49
+
50
+ ## Matched-budget SVD baselines (for the paper's Table 2)
51
+
52
+ Two reference columns used in the study, compressed with **SVD-LLM / SVD-LLM V2** at the
53
+ *same* LLaMA-2-7B budget as [`llama2-7b/`](./llama2-7b) (4.02B, 40.3% removed) and healed
54
+ with the *identical* KD protocol, so structure — not recovery effort — explains the gap:
55
+
56
+ | Directory | Method | Params | Licence |
57
+ |---|---|---:|---|
58
+ | [`svd-llm/`](./svd-llm) | SVD-LLM (ICLR 2025, official code) | 6.74B → 4.022B | `llama2` |
59
+ | [`svd-llm-v2/`](./svd-llm-v2) | SVD-LLM V2 ‡ | 6.74B → 4.023B | `llama2` |
60
+
61
+ ‡ Official V2 code was never released; this checkpoint is our faithful implementation of the
62
+ published algorithms. Same non-standard format as the TR-3 directories: factors, not dense
63
+ weights — load with the included `lowrank_linear.py`, see each subdirectory README.
svd-llm-v2/README.md ADDED
@@ -0,0 +1,43 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: llama2
3
+ base_model: NousResearch/Llama-2-7b-chat-hf
4
+ tags:
5
+ - svd
6
+ - low-rank
7
+ - model-compression
8
+ - llama2
9
+ ---
10
+
11
+ # SVD-LLM V2 (NAACL 2025): LLaMA-2-7B-chat @ 4.02B, KD-healed
12
+
13
+ SVD-LLM V2: heterogeneous compression-ratio allocation from theoretical truncation loss (Algorithm 1) + loss-optimized weight truncation (Algorithm 2). Official code was never released; this is our faithful implementation of the published algorithms (per-layer ranks in ranks.json show the allocation).
14
+
15
+ - **4.023B params** (down from 6.74B, ~40% reduction)
16
+ - Healed with pure forward-KL distillation from the dense 7B teacher: **60M tokens**,
17
+ held-out KL **2.5806 -> 0.3365**, probe MMLU (5-shot, limit 40/subtask) **25.70% -> 40.83%**
18
+ - Formal full-benchmark evaluation under the locked protocol (lm-eval 0.4.12, fp16, seed
19
+ 1234, 5/10/0/5/5-shot MMLU/HS/BQ/TQA/GSM): **[being finalized; see
20
+ the TR-3 paper's Table 2]**
21
+
22
+ | Reference | MMLU | HellaSwag | BoolQ | TriviaQA | GSM8K |
23
+ |---|---:|---:|---:|---:|---:|
24
+ | unhealed 4.02B (same structure) | 26.85 | 34.58 | 37.83 | 1.81 | 0.00 |
25
+ | TR-3 (same budget + same healing) | 43.21 | 70.75 | 77.09 | 36.93 | 9.40 |
26
+ | FP16 dense 6.74B | 47.22 | 78.59 | 80.58 | 57.17 | 22.67 |
27
+
28
+ ## How to load
29
+
30
+ ```python
31
+ import json
32
+ from lowrank_linear import build, materialize_all
33
+
34
+ model = build("NousResearch/Llama-2-7b-chat-hf",
35
+ json.load(open("ranks.json")), "state_dict_fp16.pt",
36
+ device="cuda")
37
+ # for generation/eval: fuse the factors first (inference-only)
38
+ materialize_all(model)
39
+ ```
40
+
41
+ `ranks.json` maps each compressed projection path to its integer rank; untouched weights are
42
+ stock LLaMA-2-7B-chat tensors. The full healing-provenance record (data mixture & revisions,
43
+ decontamination, SHA256 of inputs/checkpoint, 5M milestone trajectory) is in `provenance.json`.
svd-llm-v2/config.json ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "NousResearch/Llama-2-7b-chat-hf",
3
+ "architectures": [
4
+ "LlamaForCausalLM"
5
+ ],
6
+ "attention_bias": false,
7
+ "bos_token_id": 1,
8
+ "eos_token_id": 2,
9
+ "hidden_act": "silu",
10
+ "hidden_size": 4096,
11
+ "initializer_range": 0.02,
12
+ "intermediate_size": 11008,
13
+ "max_position_embeddings": 4096,
14
+ "model_type": "llama",
15
+ "num_attention_heads": 32,
16
+ "num_hidden_layers": 32,
17
+ "num_key_value_heads": 32,
18
+ "pad_token_id": 0,
19
+ "pretraining_tp": 1,
20
+ "rms_norm_eps": 1e-05,
21
+ "rope_scaling": null,
22
+ "rope_theta": 10000.0,
23
+ "tie_word_embeddings": false,
24
+ "torch_dtype": "float16",
25
+ "transformers_version": "4.35.2",
26
+ "use_cache": true,
27
+ "vocab_size": 32000
28
+ }
svd-llm-v2/lowrank_linear.py ADDED
@@ -0,0 +1,83 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Minimal standalone loader for the SVD-baseline compressed LLaMA-2-7B checkpoints.
2
+
3
+ No dependency beyond torch + transformers. The state_dict stores every compressed
4
+ projection as two factors of a LowRankLinear (A: out x k, B: k x in); ranks.json
5
+ holds the per-layer integer rank of each projection. Untouched parameters
6
+ (embeddings, norms) keep their stock transformers names.
7
+
8
+ Usage:
9
+ import json, torch
10
+ from lowrank_linear import build
11
+ model = build(
12
+ base_id="NousResearch/Llama-2-7b-chat-hf",
13
+ ranks=json.load(open("ranks.json")),
14
+ state_dict_path="state_dict_fp16.pt",
15
+ device="cuda", dtype=torch.float16,
16
+ )
17
+ tokenizer = AutoTokenizer.from_pretrained("NousResearch/Llama-2-7b-chat-hf")
18
+ out = model.generate(tokenizer("2+2=", return_tensors="pt").to("cuda").input_ids,
19
+ max_new_tokens=16, do_sample=False)
20
+ print(tokenizer.decode(out[0]))
21
+ """
22
+
23
+ from __future__ import annotations
24
+
25
+ import torch
26
+ import torch.nn as nn
27
+ import torch.nn.functional as F
28
+
29
+
30
+ class LowRankLinear(nn.Module):
31
+ """y = A (B x). A: out x r, B: r x in. (LLaMA projections carry no bias.)"""
32
+
33
+ def __init__(self, A: torch.Tensor, B: torch.Tensor):
34
+ super().__init__()
35
+ self.A = nn.Parameter(A)
36
+ self.B = nn.Parameter(B)
37
+
38
+ def forward(self, x):
39
+ return F.linear(F.linear(x, self.B), self.A)
40
+
41
+ @torch.no_grad()
42
+ def materialize_(self) -> nn.Linear:
43
+ """Replace this module with the dense W = A @ B (inference only; never call
44
+ while training -- it cuts the gradient path)."""
45
+ W = (self.A @ self.B).to(self.A.dtype)
46
+ dense = nn.Linear(self.B.shape[1], self.A.shape[0], bias=False,
47
+ device=W.device, dtype=W.dtype)
48
+ dense.weight.data = W
49
+ return dense
50
+
51
+
52
+ def build(base_id: str, ranks: dict, state_dict_path: str,
53
+ device: str = "cuda", dtype=torch.float16):
54
+ from transformers import AutoModelForCausalLM
55
+
56
+ model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype=dtype)
57
+ search = {}
58
+ for name, mod in model.named_modules():
59
+ if isinstance(mod, nn.Linear):
60
+ search[name] = mod
61
+ for path, k in ranks.items():
62
+ out_f, in_f = search[path].weight.shape
63
+ model_mod = model
64
+ for p in path.split(".")[:-1]:
65
+ model_mod = getattr(model_mod, p)
66
+ setattr(model_mod, path.split(".")[-1],
67
+ LowRankLinear(torch.zeros(out_f, int(k), dtype=dtype),
68
+ torch.zeros(int(k), in_f, dtype=dtype)))
69
+ sd = torch.load(state_dict_path, map_location="cpu", weights_only=True)
70
+ model.load_state_dict(sd, strict=True)
71
+ del sd
72
+ return model.to(device)
73
+
74
+
75
+ def materialize_all(model):
76
+ """Fuse every LowRankLinear back into a dense nn.Linear in place."""
77
+ for name, mod in list(model.named_modules()):
78
+ if isinstance(mod, LowRankLinear):
79
+ parent = model
80
+ for p in name.split(".")[:-1]:
81
+ parent = getattr(parent, p)
82
+ setattr(parent, name.split(".")[-1], mod.materialize_())
83
+ return model
svd-llm-v2/provenance.json ADDED
@@ -0,0 +1,196 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "method": "KD healing: alpha*T^2*KL(teacher||student) + (1-alpha)*CE",
3
+ "student_init": "/data1/rkp00022/zixuan/projects/TR3_main/checkpoints/svdllm2_r04193_tr3fmt/state_dict_fp16.pt",
4
+ "teacher": "NousResearch/Llama-2-7b-chat-hf",
5
+ "alpha": 1.0,
6
+ "temperature": 1.0,
7
+ "lr": 2e-05,
8
+ "tokens_seen": 59998208,
9
+ "epochs": 0.228,
10
+ "global_batch_tokens": 32768,
11
+ "dataset_mixture_pre_exclusion": {
12
+ "alpaca": 50000,
13
+ "openhermes": 200000,
14
+ "ultrachat": 100000
15
+ },
16
+ "dataset_revisions": {
17
+ "tatsu-lab/alpaca": "dce01c9b08f87459cf36a430d809084718273017",
18
+ "teknium/OpenHermes-2.5": "b82037821055c377bed0d495e72e46de3bc72e84",
19
+ "HuggingFaceH4/ultrachat_200k": "8049631c405ae6576f93f445c6b8166f76f5505a"
20
+ },
21
+ "canonical_pool_excluded_texts": 10000,
22
+ "train_texts_after_exclusion": 356000,
23
+ "chat_texts": 340000,
24
+ "mmlu_aux_texts_kept": 16000,
25
+ "mmlu_aux_split": "cais/mmlu auxiliary_train (official TRAIN split, disjoint from the evaluated test split; audit verified 0/99842 exact overlap)",
26
+ "mmlu_aux_stats": {
27
+ "requested_groups": 16000,
28
+ "rendered_groups": 16000,
29
+ "shots": 5,
30
+ "rows_total": 99842,
31
+ "rows_dropped_by_decontam": 71,
32
+ "rows_clean": 99771,
33
+ "revision": "c30699e8356da336a370243923dbaf21066bb9fe",
34
+ "format": "MMLU-shaped (NOT byte-identical to lm-eval; auxiliary_train has no subject)"
35
+ },
36
+ "gold_label_supervision": false,
37
+ "attribution_note": "alpha=1.0 => pure KD, no gold-label CE, so any MMLU change is attributable to distillation on in-domain text. alpha<1 adds CE on text containing answers = labeled MCQ training, which is a DIFFERENT and weaker claim.",
38
+ "train_tokens_total": 263112704,
39
+ "decontamination": {
40
+ "removed": {
41
+ "alpaca": 11,
42
+ "openhermes": 120,
43
+ "ultrachat": 33
44
+ }
45
+ },
46
+ "start_kl": 2.5805691182613373,
47
+ "final_kl": 0.33650884963572025,
48
+ "start_mmlu": 0.25701754385964914,
49
+ "final_mmlu": 0.4083333333333333,
50
+ "history": [
51
+ {
52
+ "tokens_m": 5.01,
53
+ "step": 153,
54
+ "held_out_kl": 0.647051740437746,
55
+ "mmlu": 0.2894736842105263
56
+ },
57
+ {
58
+ "tokens_m": 10.03,
59
+ "step": 306,
60
+ "held_out_kl": 0.594494417309761,
61
+ "mmlu": 0.368859649122807
62
+ },
63
+ {
64
+ "tokens_m": 15.01,
65
+ "step": 458,
66
+ "held_out_kl": 0.5670272633433342,
67
+ "mmlu": 0.3592105263157895
68
+ },
69
+ {
70
+ "tokens_m": 20.02,
71
+ "step": 611,
72
+ "held_out_kl": 0.5243954174220562,
73
+ "mmlu": 0.36622807017543857
74
+ },
75
+ {
76
+ "tokens_m": 25.0,
77
+ "step": 763,
78
+ "held_out_kl": 0.5039057768881321,
79
+ "mmlu": 0.38859649122807016
80
+ },
81
+ {
82
+ "tokens_m": 30.02,
83
+ "step": 916,
84
+ "held_out_kl": 0.44497134163975716,
85
+ "mmlu": 0.37543859649122807
86
+ },
87
+ {
88
+ "tokens_m": 35.03,
89
+ "step": 1069,
90
+ "held_out_kl": 0.402653768658638,
91
+ "mmlu": 0.39342105263157895
92
+ },
93
+ {
94
+ "tokens_m": 40.01,
95
+ "step": 1221,
96
+ "held_out_kl": 0.380719942972064,
97
+ "mmlu": 0.393859649122807
98
+ },
99
+ {
100
+ "tokens_m": 45.02,
101
+ "step": 1374,
102
+ "held_out_kl": 0.3613550402224064,
103
+ "mmlu": 0.4105263157894737
104
+ },
105
+ {
106
+ "tokens_m": 50.0,
107
+ "step": 1526,
108
+ "held_out_kl": 0.3475096616894007,
109
+ "mmlu": 0.39780701754385966
110
+ },
111
+ {
112
+ "tokens_m": 55.02,
113
+ "step": 1679,
114
+ "held_out_kl": 0.34086795896291733,
115
+ "mmlu": 0.40482456140350875
116
+ },
117
+ {
118
+ "tokens_m": 60.0,
119
+ "step": 1831,
120
+ "held_out_kl": 0.33650884963572025,
121
+ "mmlu": 0.4083333333333333,
122
+ "final": true
123
+ }
124
+ ],
125
+ "teacher_mmlu_same_limit": 0.47543859649122805,
126
+ "checkpoint": "/data1/rkp00022/zixuan/projects/TR3_main/checkpoints/kdheal_svdllm2r04193_h1/state_dict_fp16.pt",
127
+ "sha256": "2c48578d7ddbe5f937c2436f35ad8ffbda01d3be45b4c4eab62e1e1c6104501c",
128
+ "seed": 1234,
129
+ "mmlu_limit_per_subtask": 40,
130
+ "teacher_revision": "351844e75ed0bcbbe3f10671b3c808d2b83894ee",
131
+ "student_init_sha256": "33e0dba497c0bfe3b4532b4c6c0e07274a2ca85219b475fb2c26978d5ae6dfbd",
132
+ "ranks_file": "/data1/rkp00022/zixuan/projects/TR3_main/checkpoints/svdllm2_r04193_tr3fmt/ranks.json",
133
+ "ranks_sha256": "90ba50885421afe8d0d19024aaf52bfbbee39433f99b1e934c4bf15a20f556a3",
134
+ "input_identity_gate": {
135
+ "student_ckpt": {
136
+ "key": "svdllm2_r04193_tr3fmt",
137
+ "sha256": "33e0dba497c0bfe3b4532b4c6c0e07274a2ca85219b475fb2c26978d5ae6dfbd",
138
+ "status": "unregistered_allowed"
139
+ },
140
+ "ranks": {
141
+ "key": "svdllm2_r04193_tr3fmt",
142
+ "sha256": "90ba50885421afe8d0d19024aaf52bfbbee39433f99b1e934c4bf15a20f556a3",
143
+ "status": "unregistered_allowed"
144
+ }
145
+ },
146
+ "train_seed": 1234,
147
+ "mmlu_seed": 1234,
148
+ "dynamic_batch": false,
149
+ "dbl": null,
150
+ "snapshot_milestones": true,
151
+ "snapshots": [
152
+ "(PosixPath('/data1/rkp00022/zixuan/projects/TR3_main/checkpoints/kdheal_svdllm2r04193_h1_snap50m'), 50.003968)",
153
+ "(PosixPath('/data1/rkp00022/zixuan/projects/TR3_main/checkpoints/kdheal_svdllm2r04193_h1_snap55m'), 55.017472)"
154
+ ],
155
+ "stream_skip_tokens_m": 0.0,
156
+ "stream_blocks_consumed": [
157
+ 0,
158
+ 58592
159
+ ],
160
+ "stream_note": "the block order is a seeded permutation consumed from index 0; stages sharing a --seed with stream_skip=0 re-train on the SAME blocks. [start, end] block indices are recorded so any continuation can be checked for overlap with what earlier stages consumed.",
161
+ "student_master_dtype": "bf16",
162
+ "fake_quant": null,
163
+ "rebalance_report": null,
164
+ "student_param_count": 4023119360,
165
+ "micro": 16,
166
+ "grad_accum": 2,
167
+ "seq_len": 1024,
168
+ "held_out_source": "canonical: mixture 2000/4000/4000, seed 1234, seq 1024, blocks 256:264 (identical to the compression gates)",
169
+ "decontam_stats": {
170
+ "mmlu": {
171
+ "items": 14042,
172
+ "new_ngrams": 764261,
173
+ "revision": "c30699e8356da336a370243923dbaf21066bb9fe"
174
+ },
175
+ "hellaswag": {
176
+ "items": 10042,
177
+ "new_ngrams": 1334302,
178
+ "revision": "218ec52e09a7e7462a5400043bb9a69a41d06b76"
179
+ },
180
+ "boolq": {
181
+ "items": 3270,
182
+ "new_ngrams": 276806,
183
+ "revision": "3de24cf8022e94f4ee4b9d55a6f539891524d646"
184
+ },
185
+ "triviaqa": {
186
+ "items": 17944,
187
+ "new_ngrams": 32608,
188
+ "revision": "0f7faf33a3908546c6fd5b73a660e0f8ff173c2f"
189
+ },
190
+ "gsm8k": {
191
+ "items": 1319,
192
+ "new_ngrams": 46281,
193
+ "revision": "740312add88f781978c0658806c59bc2815b9866"
194
+ }
195
+ }
196
+ }
svd-llm-v2/ranks.json ADDED
@@ -0,0 +1,226 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model.layers.0.self_attn.q_proj": 2048,
3
+ "model.layers.0.self_attn.k_proj": 2048,
4
+ "model.layers.0.self_attn.v_proj": 2048,
5
+ "model.layers.0.self_attn.o_proj": 2048,
6
+ "model.layers.0.mlp.gate_proj": 2341,
7
+ "model.layers.0.mlp.up_proj": 2332,
8
+ "model.layers.0.mlp.down_proj": 2887,
9
+ "model.layers.1.self_attn.q_proj": 1911,
10
+ "model.layers.1.self_attn.k_proj": 1960,
11
+ "model.layers.1.self_attn.v_proj": 1826,
12
+ "model.layers.1.self_attn.o_proj": 2048,
13
+ "model.layers.1.mlp.gate_proj": 2115,
14
+ "model.layers.1.mlp.up_proj": 2094,
15
+ "model.layers.1.mlp.down_proj": 2280,
16
+ "model.layers.2.self_attn.q_proj": 1404,
17
+ "model.layers.2.self_attn.k_proj": 1405,
18
+ "model.layers.2.self_attn.v_proj": 1400,
19
+ "model.layers.2.self_attn.o_proj": 1662,
20
+ "model.layers.2.mlp.gate_proj": 1931,
21
+ "model.layers.2.mlp.up_proj": 1928,
22
+ "model.layers.2.mlp.down_proj": 2094,
23
+ "model.layers.3.self_attn.q_proj": 1252,
24
+ "model.layers.3.self_attn.k_proj": 1247,
25
+ "model.layers.3.self_attn.v_proj": 1285,
26
+ "model.layers.3.self_attn.o_proj": 1598,
27
+ "model.layers.3.mlp.gate_proj": 1821,
28
+ "model.layers.3.mlp.up_proj": 1826,
29
+ "model.layers.3.mlp.down_proj": 2002,
30
+ "model.layers.4.self_attn.q_proj": 1255,
31
+ "model.layers.4.self_attn.k_proj": 1252,
32
+ "model.layers.4.self_attn.v_proj": 1282,
33
+ "model.layers.4.self_attn.o_proj": 1490,
34
+ "model.layers.4.mlp.gate_proj": 1780,
35
+ "model.layers.4.mlp.up_proj": 1789,
36
+ "model.layers.4.mlp.down_proj": 1911,
37
+ "model.layers.5.self_attn.q_proj": 1239,
38
+ "model.layers.5.self_attn.k_proj": 1236,
39
+ "model.layers.5.self_attn.v_proj": 1266,
40
+ "model.layers.5.self_attn.o_proj": 1456,
41
+ "model.layers.5.mlp.gate_proj": 1757,
42
+ "model.layers.5.mlp.up_proj": 1765,
43
+ "model.layers.5.mlp.down_proj": 1855,
44
+ "model.layers.6.self_attn.q_proj": 1218,
45
+ "model.layers.6.self_attn.k_proj": 1214,
46
+ "model.layers.6.self_attn.v_proj": 1244,
47
+ "model.layers.6.self_attn.o_proj": 1393,
48
+ "model.layers.6.mlp.gate_proj": 1733,
49
+ "model.layers.6.mlp.up_proj": 1743,
50
+ "model.layers.6.mlp.down_proj": 1807,
51
+ "model.layers.7.self_attn.q_proj": 1209,
52
+ "model.layers.7.self_attn.k_proj": 1208,
53
+ "model.layers.7.self_attn.v_proj": 1237,
54
+ "model.layers.7.self_attn.o_proj": 1356,
55
+ "model.layers.7.mlp.gate_proj": 1718,
56
+ "model.layers.7.mlp.up_proj": 1726,
57
+ "model.layers.7.mlp.down_proj": 1770,
58
+ "model.layers.8.self_attn.q_proj": 1205,
59
+ "model.layers.8.self_attn.k_proj": 1201,
60
+ "model.layers.8.self_attn.v_proj": 1238,
61
+ "model.layers.8.self_attn.o_proj": 1314,
62
+ "model.layers.8.mlp.gate_proj": 1715,
63
+ "model.layers.8.mlp.up_proj": 1718,
64
+ "model.layers.8.mlp.down_proj": 1747,
65
+ "model.layers.9.self_attn.q_proj": 1193,
66
+ "model.layers.9.self_attn.k_proj": 1190,
67
+ "model.layers.9.self_attn.v_proj": 1229,
68
+ "model.layers.9.self_attn.o_proj": 1276,
69
+ "model.layers.9.mlp.gate_proj": 1711,
70
+ "model.layers.9.mlp.up_proj": 1711,
71
+ "model.layers.9.mlp.down_proj": 1725,
72
+ "model.layers.10.self_attn.q_proj": 1201,
73
+ "model.layers.10.self_attn.k_proj": 1194,
74
+ "model.layers.10.self_attn.v_proj": 1230,
75
+ "model.layers.10.self_attn.o_proj": 1252,
76
+ "model.layers.10.mlp.gate_proj": 1708,
77
+ "model.layers.10.mlp.up_proj": 1706,
78
+ "model.layers.10.mlp.down_proj": 1713,
79
+ "model.layers.11.self_attn.q_proj": 1199,
80
+ "model.layers.11.self_attn.k_proj": 1196,
81
+ "model.layers.11.self_attn.v_proj": 1213,
82
+ "model.layers.11.self_attn.o_proj": 1258,
83
+ "model.layers.11.mlp.gate_proj": 1699,
84
+ "model.layers.11.mlp.up_proj": 1694,
85
+ "model.layers.11.mlp.down_proj": 1699,
86
+ "model.layers.12.self_attn.q_proj": 1180,
87
+ "model.layers.12.self_attn.k_proj": 1179,
88
+ "model.layers.12.self_attn.v_proj": 1206,
89
+ "model.layers.12.self_attn.o_proj": 1234,
90
+ "model.layers.12.mlp.gate_proj": 1692,
91
+ "model.layers.12.mlp.up_proj": 1686,
92
+ "model.layers.12.mlp.down_proj": 1683,
93
+ "model.layers.13.self_attn.q_proj": 1176,
94
+ "model.layers.13.self_attn.k_proj": 1170,
95
+ "model.layers.13.self_attn.v_proj": 1190,
96
+ "model.layers.13.self_attn.o_proj": 1231,
97
+ "model.layers.13.mlp.gate_proj": 1688,
98
+ "model.layers.13.mlp.up_proj": 1678,
99
+ "model.layers.13.mlp.down_proj": 1661,
100
+ "model.layers.14.self_attn.q_proj": 1181,
101
+ "model.layers.14.self_attn.k_proj": 1179,
102
+ "model.layers.14.self_attn.v_proj": 1192,
103
+ "model.layers.14.self_attn.o_proj": 1213,
104
+ "model.layers.14.mlp.gate_proj": 1679,
105
+ "model.layers.14.mlp.up_proj": 1669,
106
+ "model.layers.14.mlp.down_proj": 1644,
107
+ "model.layers.15.self_attn.q_proj": 1177,
108
+ "model.layers.15.self_attn.k_proj": 1174,
109
+ "model.layers.15.self_attn.v_proj": 1187,
110
+ "model.layers.15.self_attn.o_proj": 1198,
111
+ "model.layers.15.mlp.gate_proj": 1670,
112
+ "model.layers.15.mlp.up_proj": 1661,
113
+ "model.layers.15.mlp.down_proj": 1621,
114
+ "model.layers.16.self_attn.q_proj": 1176,
115
+ "model.layers.16.self_attn.k_proj": 1174,
116
+ "model.layers.16.self_attn.v_proj": 1173,
117
+ "model.layers.16.self_attn.o_proj": 1175,
118
+ "model.layers.16.mlp.gate_proj": 1662,
119
+ "model.layers.16.mlp.up_proj": 1655,
120
+ "model.layers.16.mlp.down_proj": 1595,
121
+ "model.layers.17.self_attn.q_proj": 1163,
122
+ "model.layers.17.self_attn.k_proj": 1163,
123
+ "model.layers.17.self_attn.v_proj": 1167,
124
+ "model.layers.17.self_attn.o_proj": 1193,
125
+ "model.layers.17.mlp.gate_proj": 1647,
126
+ "model.layers.17.mlp.up_proj": 1643,
127
+ "model.layers.17.mlp.down_proj": 1584,
128
+ "model.layers.18.self_attn.q_proj": 1150,
129
+ "model.layers.18.self_attn.k_proj": 1149,
130
+ "model.layers.18.self_attn.v_proj": 1151,
131
+ "model.layers.18.self_attn.o_proj": 1190,
132
+ "model.layers.18.mlp.gate_proj": 1632,
133
+ "model.layers.18.mlp.up_proj": 1631,
134
+ "model.layers.18.mlp.down_proj": 1568,
135
+ "model.layers.19.self_attn.q_proj": 1156,
136
+ "model.layers.19.self_attn.k_proj": 1157,
137
+ "model.layers.19.self_attn.v_proj": 1152,
138
+ "model.layers.19.self_attn.o_proj": 1184,
139
+ "model.layers.19.mlp.gate_proj": 1623,
140
+ "model.layers.19.mlp.up_proj": 1623,
141
+ "model.layers.19.mlp.down_proj": 1557,
142
+ "model.layers.20.self_attn.q_proj": 1155,
143
+ "model.layers.20.self_attn.k_proj": 1155,
144
+ "model.layers.20.self_attn.v_proj": 1146,
145
+ "model.layers.20.self_attn.o_proj": 1172,
146
+ "model.layers.20.mlp.gate_proj": 1617,
147
+ "model.layers.20.mlp.up_proj": 1619,
148
+ "model.layers.20.mlp.down_proj": 1540,
149
+ "model.layers.21.self_attn.q_proj": 1147,
150
+ "model.layers.21.self_attn.k_proj": 1147,
151
+ "model.layers.21.self_attn.v_proj": 1136,
152
+ "model.layers.21.self_attn.o_proj": 1174,
153
+ "model.layers.21.mlp.gate_proj": 1607,
154
+ "model.layers.21.mlp.up_proj": 1611,
155
+ "model.layers.21.mlp.down_proj": 1531,
156
+ "model.layers.22.self_attn.q_proj": 1138,
157
+ "model.layers.22.self_attn.k_proj": 1138,
158
+ "model.layers.22.self_attn.v_proj": 1129,
159
+ "model.layers.22.self_attn.o_proj": 1149,
160
+ "model.layers.22.mlp.gate_proj": 1599,
161
+ "model.layers.22.mlp.up_proj": 1604,
162
+ "model.layers.22.mlp.down_proj": 1517,
163
+ "model.layers.23.self_attn.q_proj": 1122,
164
+ "model.layers.23.self_attn.k_proj": 1123,
165
+ "model.layers.23.self_attn.v_proj": 1115,
166
+ "model.layers.23.self_attn.o_proj": 1139,
167
+ "model.layers.23.mlp.gate_proj": 1589,
168
+ "model.layers.23.mlp.up_proj": 1593,
169
+ "model.layers.23.mlp.down_proj": 1508,
170
+ "model.layers.24.self_attn.q_proj": 1138,
171
+ "model.layers.24.self_attn.k_proj": 1139,
172
+ "model.layers.24.self_attn.v_proj": 1118,
173
+ "model.layers.24.self_attn.o_proj": 1135,
174
+ "model.layers.24.mlp.gate_proj": 1580,
175
+ "model.layers.24.mlp.up_proj": 1585,
176
+ "model.layers.24.mlp.down_proj": 1501,
177
+ "model.layers.25.self_attn.q_proj": 1116,
178
+ "model.layers.25.self_attn.k_proj": 1116,
179
+ "model.layers.25.self_attn.v_proj": 1100,
180
+ "model.layers.25.self_attn.o_proj": 1141,
181
+ "model.layers.25.mlp.gate_proj": 1573,
182
+ "model.layers.25.mlp.up_proj": 1577,
183
+ "model.layers.25.mlp.down_proj": 1496,
184
+ "model.layers.26.self_attn.q_proj": 1127,
185
+ "model.layers.26.self_attn.k_proj": 1126,
186
+ "model.layers.26.self_attn.v_proj": 1103,
187
+ "model.layers.26.self_attn.o_proj": 1121,
188
+ "model.layers.26.mlp.gate_proj": 1565,
189
+ "model.layers.26.mlp.up_proj": 1569,
190
+ "model.layers.26.mlp.down_proj": 1487,
191
+ "model.layers.27.self_attn.q_proj": 1108,
192
+ "model.layers.27.self_attn.k_proj": 1107,
193
+ "model.layers.27.self_attn.v_proj": 1095,
194
+ "model.layers.27.self_attn.o_proj": 1109,
195
+ "model.layers.27.mlp.gate_proj": 1557,
196
+ "model.layers.27.mlp.up_proj": 1561,
197
+ "model.layers.27.mlp.down_proj": 1475,
198
+ "model.layers.28.self_attn.q_proj": 1113,
199
+ "model.layers.28.self_attn.k_proj": 1111,
200
+ "model.layers.28.self_attn.v_proj": 1090,
201
+ "model.layers.28.self_attn.o_proj": 1105,
202
+ "model.layers.28.mlp.gate_proj": 1555,
203
+ "model.layers.28.mlp.up_proj": 1556,
204
+ "model.layers.28.mlp.down_proj": 1462,
205
+ "model.layers.29.self_attn.q_proj": 1128,
206
+ "model.layers.29.self_attn.k_proj": 1127,
207
+ "model.layers.29.self_attn.v_proj": 1094,
208
+ "model.layers.29.self_attn.o_proj": 1094,
209
+ "model.layers.29.mlp.gate_proj": 1552,
210
+ "model.layers.29.mlp.up_proj": 1553,
211
+ "model.layers.29.mlp.down_proj": 1450,
212
+ "model.layers.30.self_attn.q_proj": 1116,
213
+ "model.layers.30.self_attn.k_proj": 1115,
214
+ "model.layers.30.self_attn.v_proj": 1087,
215
+ "model.layers.30.self_attn.o_proj": 1093,
216
+ "model.layers.30.mlp.gate_proj": 1552,
217
+ "model.layers.30.mlp.up_proj": 1556,
218
+ "model.layers.30.mlp.down_proj": 1439,
219
+ "model.layers.31.self_attn.q_proj": 1156,
220
+ "model.layers.31.self_attn.k_proj": 1152,
221
+ "model.layers.31.self_attn.v_proj": 1126,
222
+ "model.layers.31.self_attn.o_proj": 1102,
223
+ "model.layers.31.mlp.gate_proj": 1570,
224
+ "model.layers.31.mlp.up_proj": 1576,
225
+ "model.layers.31.mlp.down_proj": 1429
226
+ }
svd-llm-v2/state_dict_fp16.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2c48578d7ddbe5f937c2436f35ad8ffbda01d3be45b4c4eab62e1e1c6104501c
3
+ size 8046409836
svd-llm/README.md ADDED
@@ -0,0 +1,43 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: llama2
3
+ base_model: NousResearch/Llama-2-7b-chat-hf
4
+ tags:
5
+ - svd
6
+ - low-rank
7
+ - model-compression
8
+ - llama2
9
+ ---
10
+
11
+ # SVD-LLM (ICLR 2025): LLaMA-2-7B-chat @ 4.02B, KD-healed
12
+
13
+ SVD-LLM: truncation-aware data whitening (256 wikitext2 samples) + closed-form parameter update, official code ([AIoT-MLSys-Lab/SVD-LLM @ 7538cca](https://github.com/AIoT-MLSys-Lab/SVD-LLM)), uniform 41.93% removal across all 7 projection types of all 32 layers.
14
+
15
+ - **4.022B params** (down from 6.74B, ~40% reduction)
16
+ - Healed with pure forward-KL distillation from the dense 7B teacher: **60M tokens**,
17
+ held-out KL **2.5712 -> 0.3552**, probe MMLU (5-shot, limit 40/subtask) **25.92% -> 40.83%**
18
+ - Formal full-benchmark evaluation under the locked protocol (lm-eval 0.4.12, fp16, seed
19
+ 1234, 5/10/0/5/5-shot MMLU/HS/BQ/TQA/GSM): **[being finalized; see
20
+ the TR-3 paper's Table 2]**
21
+
22
+ | Reference | MMLU | HellaSwag | BoolQ | TriviaQA | GSM8K |
23
+ |---|---:|---:|---:|---:|---:|
24
+ | unhealed 4.02B (same structure) | 26.29 | 34.22 | 38.35 | 0.96 | 0.00 |
25
+ | TR-3 (same budget + same healing) | 43.21 | 70.75 | 77.09 | 36.93 | 9.40 |
26
+ | FP16 dense 6.74B | 47.22 | 78.59 | 80.58 | 57.17 | 22.67 |
27
+
28
+ ## How to load
29
+
30
+ ```python
31
+ import json
32
+ from lowrank_linear import build, materialize_all
33
+
34
+ model = build("NousResearch/Llama-2-7b-chat-hf",
35
+ json.load(open("ranks.json")), "state_dict_fp16.pt",
36
+ device="cuda")
37
+ # for generation/eval: fuse the factors first (inference-only)
38
+ materialize_all(model)
39
+ ```
40
+
41
+ `ranks.json` maps each compressed projection path to its integer rank; untouched weights are
42
+ stock LLaMA-2-7B-chat tensors. The full healing-provenance record (data mixture & revisions,
43
+ decontamination, SHA256 of inputs/checkpoint, 5M milestone trajectory) is in `provenance.json`.
svd-llm/config.json ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "NousResearch/Llama-2-7b-chat-hf",
3
+ "architectures": [
4
+ "LlamaForCausalLM"
5
+ ],
6
+ "attention_bias": false,
7
+ "bos_token_id": 1,
8
+ "eos_token_id": 2,
9
+ "hidden_act": "silu",
10
+ "hidden_size": 4096,
11
+ "initializer_range": 0.02,
12
+ "intermediate_size": 11008,
13
+ "max_position_embeddings": 4096,
14
+ "model_type": "llama",
15
+ "num_attention_heads": 32,
16
+ "num_hidden_layers": 32,
17
+ "num_key_value_heads": 32,
18
+ "pad_token_id": 0,
19
+ "pretraining_tp": 1,
20
+ "rms_norm_eps": 1e-05,
21
+ "rope_scaling": null,
22
+ "rope_theta": 10000.0,
23
+ "tie_word_embeddings": false,
24
+ "torch_dtype": "float16",
25
+ "transformers_version": "4.35.2",
26
+ "use_cache": true,
27
+ "vocab_size": 32000
28
+ }
svd-llm/lowrank_linear.py ADDED
@@ -0,0 +1,83 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Minimal standalone loader for the SVD-baseline compressed LLaMA-2-7B checkpoints.
2
+
3
+ No dependency beyond torch + transformers. The state_dict stores every compressed
4
+ projection as two factors of a LowRankLinear (A: out x k, B: k x in); ranks.json
5
+ holds the per-layer integer rank of each projection. Untouched parameters
6
+ (embeddings, norms) keep their stock transformers names.
7
+
8
+ Usage:
9
+ import json, torch
10
+ from lowrank_linear import build
11
+ model = build(
12
+ base_id="NousResearch/Llama-2-7b-chat-hf",
13
+ ranks=json.load(open("ranks.json")),
14
+ state_dict_path="state_dict_fp16.pt",
15
+ device="cuda", dtype=torch.float16,
16
+ )
17
+ tokenizer = AutoTokenizer.from_pretrained("NousResearch/Llama-2-7b-chat-hf")
18
+ out = model.generate(tokenizer("2+2=", return_tensors="pt").to("cuda").input_ids,
19
+ max_new_tokens=16, do_sample=False)
20
+ print(tokenizer.decode(out[0]))
21
+ """
22
+
23
+ from __future__ import annotations
24
+
25
+ import torch
26
+ import torch.nn as nn
27
+ import torch.nn.functional as F
28
+
29
+
30
+ class LowRankLinear(nn.Module):
31
+ """y = A (B x). A: out x r, B: r x in. (LLaMA projections carry no bias.)"""
32
+
33
+ def __init__(self, A: torch.Tensor, B: torch.Tensor):
34
+ super().__init__()
35
+ self.A = nn.Parameter(A)
36
+ self.B = nn.Parameter(B)
37
+
38
+ def forward(self, x):
39
+ return F.linear(F.linear(x, self.B), self.A)
40
+
41
+ @torch.no_grad()
42
+ def materialize_(self) -> nn.Linear:
43
+ """Replace this module with the dense W = A @ B (inference only; never call
44
+ while training -- it cuts the gradient path)."""
45
+ W = (self.A @ self.B).to(self.A.dtype)
46
+ dense = nn.Linear(self.B.shape[1], self.A.shape[0], bias=False,
47
+ device=W.device, dtype=W.dtype)
48
+ dense.weight.data = W
49
+ return dense
50
+
51
+
52
+ def build(base_id: str, ranks: dict, state_dict_path: str,
53
+ device: str = "cuda", dtype=torch.float16):
54
+ from transformers import AutoModelForCausalLM
55
+
56
+ model = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype=dtype)
57
+ search = {}
58
+ for name, mod in model.named_modules():
59
+ if isinstance(mod, nn.Linear):
60
+ search[name] = mod
61
+ for path, k in ranks.items():
62
+ out_f, in_f = search[path].weight.shape
63
+ model_mod = model
64
+ for p in path.split(".")[:-1]:
65
+ model_mod = getattr(model_mod, p)
66
+ setattr(model_mod, path.split(".")[-1],
67
+ LowRankLinear(torch.zeros(out_f, int(k), dtype=dtype),
68
+ torch.zeros(int(k), in_f, dtype=dtype)))
69
+ sd = torch.load(state_dict_path, map_location="cpu", weights_only=True)
70
+ model.load_state_dict(sd, strict=True)
71
+ del sd
72
+ return model.to(device)
73
+
74
+
75
+ def materialize_all(model):
76
+ """Fuse every LowRankLinear back into a dense nn.Linear in place."""
77
+ for name, mod in list(model.named_modules()):
78
+ if isinstance(mod, LowRankLinear):
79
+ parent = model
80
+ for p in name.split(".")[:-1]:
81
+ parent = getattr(parent, p)
82
+ setattr(parent, name.split(".")[-1], mod.materialize_())
83
+ return model
svd-llm/provenance.json ADDED
@@ -0,0 +1,196 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "method": "KD healing: alpha*T^2*KL(teacher||student) + (1-alpha)*CE",
3
+ "student_init": "/data1/rkp00022/zixuan/projects/TR3_main/checkpoints/svdllm1_r04193_tr3fmt/state_dict_fp16.pt",
4
+ "teacher": "NousResearch/Llama-2-7b-chat-hf",
5
+ "alpha": 1.0,
6
+ "temperature": 1.0,
7
+ "lr": 2e-05,
8
+ "tokens_seen": 59998208,
9
+ "epochs": 0.228,
10
+ "global_batch_tokens": 32768,
11
+ "dataset_mixture_pre_exclusion": {
12
+ "alpaca": 50000,
13
+ "openhermes": 200000,
14
+ "ultrachat": 100000
15
+ },
16
+ "dataset_revisions": {
17
+ "tatsu-lab/alpaca": "dce01c9b08f87459cf36a430d809084718273017",
18
+ "teknium/OpenHermes-2.5": "b82037821055c377bed0d495e72e46de3bc72e84",
19
+ "HuggingFaceH4/ultrachat_200k": "8049631c405ae6576f93f445c6b8166f76f5505a"
20
+ },
21
+ "canonical_pool_excluded_texts": 10000,
22
+ "train_texts_after_exclusion": 356000,
23
+ "chat_texts": 340000,
24
+ "mmlu_aux_texts_kept": 16000,
25
+ "mmlu_aux_split": "cais/mmlu auxiliary_train (official TRAIN split, disjoint from the evaluated test split; audit verified 0/99842 exact overlap)",
26
+ "mmlu_aux_stats": {
27
+ "requested_groups": 16000,
28
+ "rendered_groups": 16000,
29
+ "shots": 5,
30
+ "rows_total": 99842,
31
+ "rows_dropped_by_decontam": 71,
32
+ "rows_clean": 99771,
33
+ "revision": "c30699e8356da336a370243923dbaf21066bb9fe",
34
+ "format": "MMLU-shaped (NOT byte-identical to lm-eval; auxiliary_train has no subject)"
35
+ },
36
+ "gold_label_supervision": false,
37
+ "attribution_note": "alpha=1.0 => pure KD, no gold-label CE, so any MMLU change is attributable to distillation on in-domain text. alpha<1 adds CE on text containing answers = labeled MCQ training, which is a DIFFERENT and weaker claim.",
38
+ "train_tokens_total": 263112704,
39
+ "decontamination": {
40
+ "removed": {
41
+ "alpaca": 11,
42
+ "openhermes": 120,
43
+ "ultrachat": 33
44
+ }
45
+ },
46
+ "start_kl": 2.571214348077774,
47
+ "final_kl": 0.3551874812692404,
48
+ "start_mmlu": 0.25921052631578945,
49
+ "final_mmlu": 0.4083333333333333,
50
+ "history": [
51
+ {
52
+ "tokens_m": 5.01,
53
+ "step": 153,
54
+ "held_out_kl": 0.6783921979367733,
55
+ "mmlu": 0.3219298245614035
56
+ },
57
+ {
58
+ "tokens_m": 10.03,
59
+ "step": 306,
60
+ "held_out_kl": 0.6254855506122112,
61
+ "mmlu": 0.3425438596491228
62
+ },
63
+ {
64
+ "tokens_m": 15.01,
65
+ "step": 458,
66
+ "held_out_kl": 0.6108568087220192,
67
+ "mmlu": 0.3557017543859649
68
+ },
69
+ {
70
+ "tokens_m": 20.02,
71
+ "step": 611,
72
+ "held_out_kl": 0.5468266606330872,
73
+ "mmlu": 0.34692982456140353
74
+ },
75
+ {
76
+ "tokens_m": 25.0,
77
+ "step": 763,
78
+ "held_out_kl": 0.5192997306585312,
79
+ "mmlu": 0.3719298245614035
80
+ },
81
+ {
82
+ "tokens_m": 30.02,
83
+ "step": 916,
84
+ "held_out_kl": 0.4662874862551689,
85
+ "mmlu": 0.37456140350877193
86
+ },
87
+ {
88
+ "tokens_m": 35.03,
89
+ "step": 1069,
90
+ "held_out_kl": 0.43436891213059425,
91
+ "mmlu": 0.38026315789473686
92
+ },
93
+ {
94
+ "tokens_m": 40.01,
95
+ "step": 1221,
96
+ "held_out_kl": 0.3999592885375023,
97
+ "mmlu": 0.39166666666666666
98
+ },
99
+ {
100
+ "tokens_m": 45.02,
101
+ "step": 1374,
102
+ "held_out_kl": 0.3796225134283304,
103
+ "mmlu": 0.40789473684210525
104
+ },
105
+ {
106
+ "tokens_m": 50.0,
107
+ "step": 1526,
108
+ "held_out_kl": 0.367245651781559,
109
+ "mmlu": 0.4
110
+ },
111
+ {
112
+ "tokens_m": 55.02,
113
+ "step": 1679,
114
+ "held_out_kl": 0.36011041700839996,
115
+ "mmlu": 0.40614035087719297
116
+ },
117
+ {
118
+ "tokens_m": 60.0,
119
+ "step": 1831,
120
+ "held_out_kl": 0.3551874812692404,
121
+ "mmlu": 0.4083333333333333,
122
+ "final": true
123
+ }
124
+ ],
125
+ "teacher_mmlu_same_limit": 0.47543859649122805,
126
+ "checkpoint": "/data1/rkp00022/zixuan/projects/TR3_main/checkpoints/kdheal_svdllm1r04193_h1/state_dict_fp16.pt",
127
+ "sha256": "0d6cb91e781a8daaca00bef920965ad84d13fd103405e975611fd89150886923",
128
+ "seed": 1234,
129
+ "mmlu_limit_per_subtask": 40,
130
+ "teacher_revision": "351844e75ed0bcbbe3f10671b3c808d2b83894ee",
131
+ "student_init_sha256": "f244b45c40d688d4a4e285026ac4272a3db892364d91963564b87b69cd21b8bd",
132
+ "ranks_file": "/data1/rkp00022/zixuan/projects/TR3_main/checkpoints/svdllm1_r04193_tr3fmt/ranks.json",
133
+ "ranks_sha256": "715d09b26b58bbc4052eb425dc041e5d8b3acb5cc6cef729e2ca34c76c0e2943",
134
+ "input_identity_gate": {
135
+ "student_ckpt": {
136
+ "key": "svdllm1_r04193_tr3fmt",
137
+ "sha256": "f244b45c40d688d4a4e285026ac4272a3db892364d91963564b87b69cd21b8bd",
138
+ "status": "unregistered_allowed"
139
+ },
140
+ "ranks": {
141
+ "key": "svdllm1_r04193_tr3fmt",
142
+ "sha256": "715d09b26b58bbc4052eb425dc041e5d8b3acb5cc6cef729e2ca34c76c0e2943",
143
+ "status": "unregistered_allowed"
144
+ }
145
+ },
146
+ "train_seed": 1234,
147
+ "mmlu_seed": 1234,
148
+ "dynamic_batch": false,
149
+ "dbl": null,
150
+ "snapshot_milestones": true,
151
+ "snapshots": [
152
+ "(PosixPath('/data1/rkp00022/zixuan/projects/TR3_main/checkpoints/kdheal_svdllm1r04193_h1_snap50m'), 50.003968)",
153
+ "(PosixPath('/data1/rkp00022/zixuan/projects/TR3_main/checkpoints/kdheal_svdllm1r04193_h1_snap55m'), 55.017472)"
154
+ ],
155
+ "stream_skip_tokens_m": 0.0,
156
+ "stream_blocks_consumed": [
157
+ 0,
158
+ 58592
159
+ ],
160
+ "stream_note": "the block order is a seeded permutation consumed from index 0; stages sharing a --seed with stream_skip=0 re-train on the SAME blocks. [start, end] block indices are recorded so any continuation can be checked for overlap with what earlier stages consumed.",
161
+ "student_master_dtype": "bf16",
162
+ "fake_quant": null,
163
+ "rebalance_report": null,
164
+ "student_param_count": 4021989376,
165
+ "micro": 16,
166
+ "grad_accum": 2,
167
+ "seq_len": 1024,
168
+ "held_out_source": "canonical: mixture 2000/4000/4000, seed 1234, seq 1024, blocks 256:264 (identical to the compression gates)",
169
+ "decontam_stats": {
170
+ "mmlu": {
171
+ "items": 14042,
172
+ "new_ngrams": 764261,
173
+ "revision": "c30699e8356da336a370243923dbaf21066bb9fe"
174
+ },
175
+ "hellaswag": {
176
+ "items": 10042,
177
+ "new_ngrams": 1334302,
178
+ "revision": "218ec52e09a7e7462a5400043bb9a69a41d06b76"
179
+ },
180
+ "boolq": {
181
+ "items": 3270,
182
+ "new_ngrams": 276806,
183
+ "revision": "3de24cf8022e94f4ee4b9d55a6f539891524d646"
184
+ },
185
+ "triviaqa": {
186
+ "items": 17944,
187
+ "new_ngrams": 32608,
188
+ "revision": "0f7faf33a3908546c6fd5b73a660e0f8ff173c2f"
189
+ },
190
+ "gsm8k": {
191
+ "items": 1319,
192
+ "new_ngrams": 46281,
193
+ "revision": "740312add88f781978c0658806c59bc2815b9866"
194
+ }
195
+ }
196
+ }
svd-llm/ranks.json ADDED
@@ -0,0 +1,226 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model.layers.0.self_attn.q_proj": 1189,
3
+ "model.layers.0.self_attn.k_proj": 1189,
4
+ "model.layers.0.self_attn.v_proj": 1189,
5
+ "model.layers.0.self_attn.o_proj": 1189,
6
+ "model.layers.0.mlp.gate_proj": 1733,
7
+ "model.layers.0.mlp.up_proj": 1733,
8
+ "model.layers.0.mlp.down_proj": 1733,
9
+ "model.layers.1.self_attn.q_proj": 1189,
10
+ "model.layers.1.self_attn.k_proj": 1189,
11
+ "model.layers.1.self_attn.v_proj": 1189,
12
+ "model.layers.1.self_attn.o_proj": 1189,
13
+ "model.layers.1.mlp.gate_proj": 1733,
14
+ "model.layers.1.mlp.up_proj": 1733,
15
+ "model.layers.1.mlp.down_proj": 1733,
16
+ "model.layers.2.self_attn.q_proj": 1189,
17
+ "model.layers.2.self_attn.k_proj": 1189,
18
+ "model.layers.2.self_attn.v_proj": 1189,
19
+ "model.layers.2.self_attn.o_proj": 1189,
20
+ "model.layers.2.mlp.gate_proj": 1733,
21
+ "model.layers.2.mlp.up_proj": 1733,
22
+ "model.layers.2.mlp.down_proj": 1733,
23
+ "model.layers.3.self_attn.q_proj": 1189,
24
+ "model.layers.3.self_attn.k_proj": 1189,
25
+ "model.layers.3.self_attn.v_proj": 1189,
26
+ "model.layers.3.self_attn.o_proj": 1189,
27
+ "model.layers.3.mlp.gate_proj": 1733,
28
+ "model.layers.3.mlp.up_proj": 1733,
29
+ "model.layers.3.mlp.down_proj": 1733,
30
+ "model.layers.4.self_attn.q_proj": 1189,
31
+ "model.layers.4.self_attn.k_proj": 1189,
32
+ "model.layers.4.self_attn.v_proj": 1189,
33
+ "model.layers.4.self_attn.o_proj": 1189,
34
+ "model.layers.4.mlp.gate_proj": 1733,
35
+ "model.layers.4.mlp.up_proj": 1733,
36
+ "model.layers.4.mlp.down_proj": 1733,
37
+ "model.layers.5.self_attn.q_proj": 1189,
38
+ "model.layers.5.self_attn.k_proj": 1189,
39
+ "model.layers.5.self_attn.v_proj": 1189,
40
+ "model.layers.5.self_attn.o_proj": 1189,
41
+ "model.layers.5.mlp.gate_proj": 1733,
42
+ "model.layers.5.mlp.up_proj": 1733,
43
+ "model.layers.5.mlp.down_proj": 1733,
44
+ "model.layers.6.self_attn.q_proj": 1189,
45
+ "model.layers.6.self_attn.k_proj": 1189,
46
+ "model.layers.6.self_attn.v_proj": 1189,
47
+ "model.layers.6.self_attn.o_proj": 1189,
48
+ "model.layers.6.mlp.gate_proj": 1733,
49
+ "model.layers.6.mlp.up_proj": 1733,
50
+ "model.layers.6.mlp.down_proj": 1733,
51
+ "model.layers.7.self_attn.q_proj": 1189,
52
+ "model.layers.7.self_attn.k_proj": 1189,
53
+ "model.layers.7.self_attn.v_proj": 1189,
54
+ "model.layers.7.self_attn.o_proj": 1189,
55
+ "model.layers.7.mlp.gate_proj": 1733,
56
+ "model.layers.7.mlp.up_proj": 1733,
57
+ "model.layers.7.mlp.down_proj": 1733,
58
+ "model.layers.8.self_attn.q_proj": 1189,
59
+ "model.layers.8.self_attn.k_proj": 1189,
60
+ "model.layers.8.self_attn.v_proj": 1189,
61
+ "model.layers.8.self_attn.o_proj": 1189,
62
+ "model.layers.8.mlp.gate_proj": 1733,
63
+ "model.layers.8.mlp.up_proj": 1733,
64
+ "model.layers.8.mlp.down_proj": 1733,
65
+ "model.layers.9.self_attn.q_proj": 1189,
66
+ "model.layers.9.self_attn.k_proj": 1189,
67
+ "model.layers.9.self_attn.v_proj": 1189,
68
+ "model.layers.9.self_attn.o_proj": 1189,
69
+ "model.layers.9.mlp.gate_proj": 1733,
70
+ "model.layers.9.mlp.up_proj": 1733,
71
+ "model.layers.9.mlp.down_proj": 1733,
72
+ "model.layers.10.self_attn.q_proj": 1189,
73
+ "model.layers.10.self_attn.k_proj": 1189,
74
+ "model.layers.10.self_attn.v_proj": 1189,
75
+ "model.layers.10.self_attn.o_proj": 1189,
76
+ "model.layers.10.mlp.gate_proj": 1733,
77
+ "model.layers.10.mlp.up_proj": 1733,
78
+ "model.layers.10.mlp.down_proj": 1733,
79
+ "model.layers.11.self_attn.q_proj": 1189,
80
+ "model.layers.11.self_attn.k_proj": 1189,
81
+ "model.layers.11.self_attn.v_proj": 1189,
82
+ "model.layers.11.self_attn.o_proj": 1189,
83
+ "model.layers.11.mlp.gate_proj": 1733,
84
+ "model.layers.11.mlp.up_proj": 1733,
85
+ "model.layers.11.mlp.down_proj": 1733,
86
+ "model.layers.12.self_attn.q_proj": 1189,
87
+ "model.layers.12.self_attn.k_proj": 1189,
88
+ "model.layers.12.self_attn.v_proj": 1189,
89
+ "model.layers.12.self_attn.o_proj": 1189,
90
+ "model.layers.12.mlp.gate_proj": 1733,
91
+ "model.layers.12.mlp.up_proj": 1733,
92
+ "model.layers.12.mlp.down_proj": 1733,
93
+ "model.layers.13.self_attn.q_proj": 1189,
94
+ "model.layers.13.self_attn.k_proj": 1189,
95
+ "model.layers.13.self_attn.v_proj": 1189,
96
+ "model.layers.13.self_attn.o_proj": 1189,
97
+ "model.layers.13.mlp.gate_proj": 1733,
98
+ "model.layers.13.mlp.up_proj": 1733,
99
+ "model.layers.13.mlp.down_proj": 1733,
100
+ "model.layers.14.self_attn.q_proj": 1189,
101
+ "model.layers.14.self_attn.k_proj": 1189,
102
+ "model.layers.14.self_attn.v_proj": 1189,
103
+ "model.layers.14.self_attn.o_proj": 1189,
104
+ "model.layers.14.mlp.gate_proj": 1733,
105
+ "model.layers.14.mlp.up_proj": 1733,
106
+ "model.layers.14.mlp.down_proj": 1733,
107
+ "model.layers.15.self_attn.q_proj": 1189,
108
+ "model.layers.15.self_attn.k_proj": 1189,
109
+ "model.layers.15.self_attn.v_proj": 1189,
110
+ "model.layers.15.self_attn.o_proj": 1189,
111
+ "model.layers.15.mlp.gate_proj": 1733,
112
+ "model.layers.15.mlp.up_proj": 1733,
113
+ "model.layers.15.mlp.down_proj": 1733,
114
+ "model.layers.16.self_attn.q_proj": 1189,
115
+ "model.layers.16.self_attn.k_proj": 1189,
116
+ "model.layers.16.self_attn.v_proj": 1189,
117
+ "model.layers.16.self_attn.o_proj": 1189,
118
+ "model.layers.16.mlp.gate_proj": 1733,
119
+ "model.layers.16.mlp.up_proj": 1733,
120
+ "model.layers.16.mlp.down_proj": 1733,
121
+ "model.layers.17.self_attn.q_proj": 1189,
122
+ "model.layers.17.self_attn.k_proj": 1189,
123
+ "model.layers.17.self_attn.v_proj": 1189,
124
+ "model.layers.17.self_attn.o_proj": 1189,
125
+ "model.layers.17.mlp.gate_proj": 1733,
126
+ "model.layers.17.mlp.up_proj": 1733,
127
+ "model.layers.17.mlp.down_proj": 1733,
128
+ "model.layers.18.self_attn.q_proj": 1189,
129
+ "model.layers.18.self_attn.k_proj": 1189,
130
+ "model.layers.18.self_attn.v_proj": 1189,
131
+ "model.layers.18.self_attn.o_proj": 1189,
132
+ "model.layers.18.mlp.gate_proj": 1733,
133
+ "model.layers.18.mlp.up_proj": 1733,
134
+ "model.layers.18.mlp.down_proj": 1733,
135
+ "model.layers.19.self_attn.q_proj": 1189,
136
+ "model.layers.19.self_attn.k_proj": 1189,
137
+ "model.layers.19.self_attn.v_proj": 1189,
138
+ "model.layers.19.self_attn.o_proj": 1189,
139
+ "model.layers.19.mlp.gate_proj": 1733,
140
+ "model.layers.19.mlp.up_proj": 1733,
141
+ "model.layers.19.mlp.down_proj": 1733,
142
+ "model.layers.20.self_attn.q_proj": 1189,
143
+ "model.layers.20.self_attn.k_proj": 1189,
144
+ "model.layers.20.self_attn.v_proj": 1189,
145
+ "model.layers.20.self_attn.o_proj": 1189,
146
+ "model.layers.20.mlp.gate_proj": 1733,
147
+ "model.layers.20.mlp.up_proj": 1733,
148
+ "model.layers.20.mlp.down_proj": 1733,
149
+ "model.layers.21.self_attn.q_proj": 1189,
150
+ "model.layers.21.self_attn.k_proj": 1189,
151
+ "model.layers.21.self_attn.v_proj": 1189,
152
+ "model.layers.21.self_attn.o_proj": 1189,
153
+ "model.layers.21.mlp.gate_proj": 1733,
154
+ "model.layers.21.mlp.up_proj": 1733,
155
+ "model.layers.21.mlp.down_proj": 1733,
156
+ "model.layers.22.self_attn.q_proj": 1189,
157
+ "model.layers.22.self_attn.k_proj": 1189,
158
+ "model.layers.22.self_attn.v_proj": 1189,
159
+ "model.layers.22.self_attn.o_proj": 1189,
160
+ "model.layers.22.mlp.gate_proj": 1733,
161
+ "model.layers.22.mlp.up_proj": 1733,
162
+ "model.layers.22.mlp.down_proj": 1733,
163
+ "model.layers.23.self_attn.q_proj": 1189,
164
+ "model.layers.23.self_attn.k_proj": 1189,
165
+ "model.layers.23.self_attn.v_proj": 1189,
166
+ "model.layers.23.self_attn.o_proj": 1189,
167
+ "model.layers.23.mlp.gate_proj": 1733,
168
+ "model.layers.23.mlp.up_proj": 1733,
169
+ "model.layers.23.mlp.down_proj": 1733,
170
+ "model.layers.24.self_attn.q_proj": 1189,
171
+ "model.layers.24.self_attn.k_proj": 1189,
172
+ "model.layers.24.self_attn.v_proj": 1189,
173
+ "model.layers.24.self_attn.o_proj": 1189,
174
+ "model.layers.24.mlp.gate_proj": 1733,
175
+ "model.layers.24.mlp.up_proj": 1733,
176
+ "model.layers.24.mlp.down_proj": 1733,
177
+ "model.layers.25.self_attn.q_proj": 1189,
178
+ "model.layers.25.self_attn.k_proj": 1189,
179
+ "model.layers.25.self_attn.v_proj": 1189,
180
+ "model.layers.25.self_attn.o_proj": 1189,
181
+ "model.layers.25.mlp.gate_proj": 1733,
182
+ "model.layers.25.mlp.up_proj": 1733,
183
+ "model.layers.25.mlp.down_proj": 1733,
184
+ "model.layers.26.self_attn.q_proj": 1189,
185
+ "model.layers.26.self_attn.k_proj": 1189,
186
+ "model.layers.26.self_attn.v_proj": 1189,
187
+ "model.layers.26.self_attn.o_proj": 1189,
188
+ "model.layers.26.mlp.gate_proj": 1733,
189
+ "model.layers.26.mlp.up_proj": 1733,
190
+ "model.layers.26.mlp.down_proj": 1733,
191
+ "model.layers.27.self_attn.q_proj": 1189,
192
+ "model.layers.27.self_attn.k_proj": 1189,
193
+ "model.layers.27.self_attn.v_proj": 1189,
194
+ "model.layers.27.self_attn.o_proj": 1189,
195
+ "model.layers.27.mlp.gate_proj": 1733,
196
+ "model.layers.27.mlp.up_proj": 1733,
197
+ "model.layers.27.mlp.down_proj": 1733,
198
+ "model.layers.28.self_attn.q_proj": 1189,
199
+ "model.layers.28.self_attn.k_proj": 1189,
200
+ "model.layers.28.self_attn.v_proj": 1189,
201
+ "model.layers.28.self_attn.o_proj": 1189,
202
+ "model.layers.28.mlp.gate_proj": 1733,
203
+ "model.layers.28.mlp.up_proj": 1733,
204
+ "model.layers.28.mlp.down_proj": 1733,
205
+ "model.layers.29.self_attn.q_proj": 1189,
206
+ "model.layers.29.self_attn.k_proj": 1189,
207
+ "model.layers.29.self_attn.v_proj": 1189,
208
+ "model.layers.29.self_attn.o_proj": 1189,
209
+ "model.layers.29.mlp.gate_proj": 1733,
210
+ "model.layers.29.mlp.up_proj": 1733,
211
+ "model.layers.29.mlp.down_proj": 1733,
212
+ "model.layers.30.self_attn.q_proj": 1189,
213
+ "model.layers.30.self_attn.k_proj": 1189,
214
+ "model.layers.30.self_attn.v_proj": 1189,
215
+ "model.layers.30.self_attn.o_proj": 1189,
216
+ "model.layers.30.mlp.gate_proj": 1733,
217
+ "model.layers.30.mlp.up_proj": 1733,
218
+ "model.layers.30.mlp.down_proj": 1733,
219
+ "model.layers.31.self_attn.q_proj": 1189,
220
+ "model.layers.31.self_attn.k_proj": 1189,
221
+ "model.layers.31.self_attn.v_proj": 1189,
222
+ "model.layers.31.self_attn.o_proj": 1189,
223
+ "model.layers.31.mlp.gate_proj": 1733,
224
+ "model.layers.31.mlp.up_proj": 1733,
225
+ "model.layers.31.mlp.down_proj": 1733
226
+ }
svd-llm/state_dict_fp16.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0d6cb91e781a8daaca00bef920965ad84d13fd103405e975611fd89150886923
3
+ size 8044149604