Rhett1 commited on
Commit
63e2c7b
·
verified ·
1 Parent(s): ef35e31

Upload svd_llm/unheal/llama2-13b-base/README.md with huggingface_hub

Browse files
svd_llm/unheal/llama2-13b-base/README.md ADDED
@@ -0,0 +1,20 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: llama2
3
+ base_model: NousResearch/Llama-2-13b-hf
4
+ tags: [svd, low-rank, model-compression, llama2]
5
+ ---
6
+
7
+ # SVD-LLM (ICLR 2025, official code): whitening + closed-form update, uniform ~41.6% removal @ 7.74B, UNHEALED (LLaMA-2-13B base)
8
+
9
+ - **UNHEALED checkpoint**: raw post-compression weights, NO distillation. The KD-healed
10
+ counterpart is at `../heal/llama2-13b-base/` (same `ranks.json`; weights differ).
11
+ - Factored params: **7,737,799,680** (budget 7.7403B, matched to the TR-3 llama2-13b row;
12
+ SVD-LLM uniform keep-ratio 0.5842 across the factored projections).
13
+ - probe MMLU (limit 40/subtask, n=2280, seed 1234, fp16 — measured as the step-0 eval of
14
+ the sibling heal run): **25.75%**, i.e. essentially chance level; FP16 dense 13B base
15
+ teacher on the same probe: 54.25%. Compression alone, without healing, destroys base-model
16
+ capability at this budget.
17
+
18
+ Load with `lowrank_linear.py` in this dir (identical recipe to the sibling dirs).
19
+ Compression-provenance details (source revision, sha256 of every artifact) are in
20
+ `provenance.json`.