--- license: other license_name: see-each-subdirectory tags: [tensor-decomposition, svd, low-rank, model-compression, knowledge-distillation] --- # TR-3 study models: main compressed models and matched-budget baselines Layout: ** / / **. Method families are `tr3` (our 3-site tensor ring + sparse residual), `tt3` (3-site tensor train ablation), `svd_llm`, `svd_llm_v2` and `asvd_llm` (matched-budget baselines). Stages are `heal` (KD-distilled, deployable) and `unheal` (raw post-compression, for measuring the healing delta). | method | stage | model | params (B) | licence | KD result (probe) | |---|---|---|---:|---|---| | tr3 | heal | [`qwen2.5-7b`](tr3/heal/qwen2.5-7b) | 5.187 | qwen-research (non-commercial) | see dir | | tt3 | heal | [`llama2-7b`](tt3/heal/llama2-7b) | 4.021 | llama2 | see dir | | tt3 | heal | [`qwen2.5-3b`](tt3/heal/qwen2.5-3b) | 2.388 | qwen-research (non-commercial) | see dir | | tt3 | unheal | [`llama2-7b`](tt3/unheal/llama2-7b) | 4.021 | llama2 | — | | tt3 | unheal | [`qwen2.5-3b`](tt3/unheal/qwen2.5-3b) | 2.388 | qwen-research (non-commercial) | — | | tr3 | heal | [`llama2-7b`](tr3/heal/llama2-7b) | 4.023 | llama2 | MMLU 43.21* | | tr3 | heal | [`llama2-13b`](tr3/heal/llama2-13b) | 7.740 | llama2 | see dir | | tr3 | heal | [`qwen2.5-3b`](tr3/heal/qwen2.5-3b) | 2.078 | qwen-research (non-commercial) | see dir | | tr3 | unheal | [`llama2-13b`](tr3/unheal/llama2-13b) | 7.740 | llama2 | — | | tr3 | unheal | [`llama2-7b`](tr3/unheal/llama2-7b) | (off-cluster archive) | llama2 | — | | svd_llm | heal | [`llama2-7b`](svd_llm/heal/llama2-7b) | 4.022 | llama2 | MMLU 39.47 | | svd_llm | heal | [`llama2-13b`](svd_llm/heal/llama2-13b) | 7.738 | llama2 | MMLU 46.54 | | svd_llm | unheal | [`llama2-7b`](svd_llm/unheal/llama2-7b) | 4.022 | llama2 | — | | svd_llm | unheal | [`llama2-13b`](svd_llm/unheal/llama2-13b) | 7.738 | llama2 | — | | svd_llm_v2 ‡ | heal | [`llama2-7b`](svd_llm2/heal/llama2-7b) | 4.023 | llama2 | MMLU 40.22 | | svd_llm_v2 ‡ | heal | [`llama2-13b`](svd_llm2/heal/llama2-13b) | 7.740 | llama2 | MMLU 44.96 | | svd_llm_v2 ‡ | unheal | [`llama2-7b`](svd_llm2/unheal/llama2-7b) | 4.023 | llama2 | — | | svd_llm_v2 ‡ | unheal | [`llama2-13b`](svd_llm2/unheal/llama2-13b) | 7.740 | llama2 | — | | asvd_llm | heal | [`llama2-7b`](asvd_llm/heal/llama2-7b) | 4.023 | llama2 | MMLU 35.09 | | asvd_llm | heal | [`llama2-13b`](asvd_llm/heal/llama2-13b) | 7.736 | llama2 | MMLU 42.46 | | asvd_llm | unheal | [`llama2-7b`](asvd_llm/unheal/llama2-7b) | 4.023 | llama2 | — | | asvd_llm | unheal | [`llama2-13b`](asvd_llm/unheal/llama2-13b) | 7.736 | llama2 | — | \* full-benchmark lm-eval 0.4.12 result; SVD probe columns are limit-40 healing probes here — each dir's README/provenance carries the precise protocol. ‡ Official SVD-LLM V2 code was never released; `svd_llm_v2` artifacts are our faithful implementation of the published algorithms (per-layer ranks document the allocation). ## ⚠️ These are NOT standard `transformers` checkpoints `tr3/**` stores **ring cores** (load with each dir's `tensor_formats.py`); `svd_llm*/**` stores **low-rank factor pairs** (load with each dir's `lowrank_linear.py`). Weight files are uniformly named `state_dict_fp16.pt`. `from_pretrained()` will not work on this repo. ## Benchmark results All numbers below were produced by us under a single locked protocol: **lm-eval-harness 0.4.12, fp16, seed 1234**; MMLU 5-shot, HellaSwag 10-shot, everything else 0-shot; full benchmarks, no subsampling. Every baseline was re-run at the **same parameter budget and the same 60M-token KD recipe** as our model rather than quoted from its paper, so the rows within a block are directly comparable. Perplexity is **token-level** over fixed non-overlapping 2048-token windows, following SVD-LLM's `evaluater.py`. It is *not* lm-eval's word-level perplexity; the two differ by the exponent `N_tok/N_word` (1.4156 for LLaMA-2 on WikiText-2) and must never be placed in the same column. `--` marks a cell not yet measured; it is a placeholder, not a zero or a failure. ### Perplexity (lower is better) | Model | Method | Params (B) | WikiText-2 | C4 | |---|---|---:|---:|---:| | **LLaMA-2-7B-chat** | *dense FP16* | 6.738 | 6.94 | 9.48 | | | TR-3 (ours) | 4.023 | 8.34 | 11.10 | | | SVD-LLM | 4.022 | 11.49 | 15.09 | | | SVD-LLM V2 ‡ | 4.023 | 11.03 | 14.73 | | | ASVD | 4.023 | 11.22 | 14.18 | | | TT-3 (open chain) | 4.021 | 12.52 | 15.09 | | **LLaMA-2-13B-chat** | *dense FP16* | 13.016 | 6.11 | 8.52 | | | TR-3 (ours) | 7.740 | 6.98 | 9.65 | | | SVD-LLM | 7.738 | 10.15 | 13.46 | | | SVD-LLM V2 ‡ | 7.740 | 9.72 | 13.09 | | | ASVD | 7.736 | 9.13 | 11.88 | | **LLaMA-2-7B (base)** | *dense FP16* | 6.738 | 5.47 | 7.29 | | | TR-3 (ours) | 4.023 | 7.28 | 9.58 | | | SVD-LLM | 4.022 | 10.17 | -- | | | SVD-LLM V2 ‡ | 4.023 | 9.73 | -- | | | ASVD | 4.023 | 10.04 | 12.48 | | **LLaMA-2-13B (base)** | *dense FP16* | 13.016 | 4.88 | 6.74 | | | TR-3 (ours) | 7.740 | 6.18 | 8.51 | | | ASVD | 7.746 | 7.97 | 10.44 | | | SVD-LLM | 7.738 | 9.14 | -- | | | SVD-LLM V2 ‡ | 7.740 | 8.74 | -- | | **Qwen2.5-3B** | *dense FP16* | 3.086 | 8.56 | -- | | | TR-3 (ours) | 2.078 | 11.75 | -- | | | TT-3 (open chain) | 2.077 | -- | -- | | | Matrix low-rank † | 2.077 | -- | -- | | **Qwen2.5-7B** | *dense FP16* | 7.616 | 7.46 | -- | | | TR-3 (ours) | 5.187 | 9.51 | -- | ### Zero-shot accuracy (%) | Model | Method | BoolQ | ARC-C | ARC-E | LAMBADA | PIQA | Winogrande | MathQA | Avg | |---|---|---:|---:|---:|---:|---:|---:|---:|---:| | **LLaMA-2-7B-chat** | *dense FP16* | 80.58 | 43.26 | 68.39 | 69.82 | 77.04 | 68.35 | 28.48 | 62.27 | | | TR-3 (ours) | 77.09 | 38.48 | 65.11 | 66.52 | 75.52 | 66.30 | 27.50 | 59.50 | | | SVD-LLM | 67.98 | 32.76 | 57.20 | 56.76 | 71.55 | 62.19 | 25.49 | 53.42 | | | SVD-LLM V2 ‡ | 68.90 | 34.30 | 59.30 | 57.29 | 71.65 | 61.25 | 25.66 | 54.05 | | | ASVD | 69.69 | 33.28 | 57.03 | 55.15 | 71.33 | 60.14 | 24.56 | 53.03 | | | TT-3 (open chain) | 63.67 | 32.68 | 52.06 | 55.77 | 71.22 | 58.80 | 25.23 | 51.35 | | **LLaMA-2-13B-chat** | *dense FP16* | 81.96 | 49.91 | 73.40 | 72.79 | 79.33 | 72.85 | 32.86 | 66.16 | | | TR-3 (ours) | 80.52 | 43.86 | 70.71 | 70.74 | 76.82 | 70.01 | 29.92 | 63.22 | | | SVD-LLM | 73.64 | 35.41 | 61.03 | 60.59 | 72.74 | 64.48 | 25.93 | 56.26 | | | SVD-LLM V2 ‡ | 75.14 | 36.35 | 61.99 | 62.47 | 73.34 | 66.14 | 27.14 | 57.51 | | | ASVD | 74.89 | 36.95 | 64.60 | 61.60 | 74.05 | 65.19 | 26.37 | 57.66 | | **LLaMA-2-7B (base)** | *dense FP16* | 79.33 | 45.05 | 73.86 | 73.65 | 78.73 | 69.46 | 28.41 | 64.07 | | | TR-3 (ours) | 70.95 | 39.16 | 65.24 | 68.02 | 75.35 | 64.48 | 25.66 | 58.41 | | | SVD-LLM | 68.62 | 33.19 | 58.71 | 55.58 | 71.16 | 61.33 | 24.19 | 53.25 | | | SVD-LLM V2 ‡ | 68.53 | 33.96 | 60.06 | 56.49 | 72.31 | 62.83 | 25.59 | 54.25 | | | ASVD | 59.54 | 32.17 | 57.83 | 57.83 | 71.49 | 60.14 | 23.92 | 51.85 | | **LLaMA-2-13B (base)** | *dense FP16* | 82.11 | 48.81 | 76.47 | 76.54 | 80.25 | 72.45 | 31.46 | 66.87 | | | TR-3 (ours) | 76.21 | 44.11 | 72.43 | 71.74 | 78.51 | 68.67 | 29.65 | 63.05 | | | ASVD | 74.19 | 38.74 | 68.14 | 66.10 | 75.73 | 65.90 | 25.36 | 59.17 | | | SVD-LLM | 70.34 | 36.01 | 64.94 | 60.94 | 73.78 | 65.82 | 24.69 | 56.64 | | | SVD-LLM V2 ‡ | 71.13 | 38.14 | 67.30 | 61.67 | 73.83 | 65.67 | 25.66 | 57.63 | | **Qwen2.5-3B** | *dense FP16* | 80.09 | 48.04 | 72.90 | 66.12 | 78.29 | 69.38 | 35.61 | 64.35 | | | TR-3 (ours) | 76.39 | 43.17 | 69.36 | 60.51 | 75.24 | 64.09 | 34.17 | 60.42 | | | TT-3 (open chain) | 66.88 | 31.14 | 56.19 | 49.16 | 70.40 | 59.12 | 27.64 | 51.50 | | | Matrix low-rank † | 71.35 | 36.43 | 59.55 | 50.51 | 72.14 | 61.88 | 28.41 | 54.33 | | **Qwen2.5-7B** | *dense FP16* | 86.42 | 54.86 | 80.93 | 69.77 | 80.36 | 70.56 | 40.37 | 69.04 | | | TR-3 (ours) | 84.68 | 51.11 | 73.82 | 64.84 | 77.64 | 70.24 | 42.01 | 66.33 | ### Few-shot / generative (%) | Model | Method | MMLU (5s) | HellaSwag (10s) | TriviaQA (5s) | GSM8K (5s) | |---|---|---:|---:|---:|---:| | **LLaMA-2-7B-chat** | *dense FP16* | 47.22 | 78.63 | 57.18 | 22.59 | | | TR-3 (ours) | 43.21 | 70.75 | 36.93 | 9.40 | | | SVD-LLM | 39.47 | 57.65 | 16.18 | 2.05 | | | SVD-LLM V2 ‡ | 40.26 | 58.82 | 18.62 | 0.68 | | | ASVD | 34.65 | 59.46 | 18.02 | 2.27 | | | TT-3 (open chain) | 34.25 | 56.38 | 14.48 | 0.76 | | **LLaMA-2-13B-chat** | *dense FP16* | 53.55 | 82.07 | 65.20 | 35.25 | | | TR-3 (ours) | 49.69 | 76.46 | 47.81 | 24.11 | | | SVD-LLM | 44.18 | 63.41 | 21.57 | 0.15 | | | SVD-LLM V2 ‡ | 44.77 | 64.99 | 24.52 | 0.15 | | | ASVD | 41.43 | 68.01 | 29.71 | 7.51 | | **LLaMA-2-7B (base)** | *dense FP16* | 45.97 | 78.95 | 64.39 | 13.12 | | | TR-3 (ours) | 39.49 | 71.11 | 39.75 | 6.52 | | | SVD-LLM | 32.74 | 58.36 | 16.43 | 1.59 | | | SVD-LLM V2 ‡ | 34.33 | 59.36 | 19.34 | 1.97 | | | ASVD | 26.58 | 61.21 | 20.69 | 1.29 | | **LLaMA-2-13B (base)** | *dense FP16* | 55.54 | 82.14 | 70.69 | 23.28 | | | TR-3 (ours) | 50.46 | 77.04 | 50.96 | 14.33 | | | ASVD | 42.44 | 70.15 | 35.50 | 5.23 | | | SVD-LLM | 42.70 | 63.80 | 21.87 | 4.25 | | | SVD-LLM V2 ‡ | 44.67 | 65.00 | 24.49 | 4.93 | | **Qwen2.5-3B** | *dense FP16* | 66.42 | 75.26 | 45.78 | 10.31 | | | TR-3 (ours) | 56.02 | 66.39 | 27.61 | 41.32 | | | TT-3 (open chain) | 41.97 | 51.65 | 11.86 | 9.78 | | | Matrix low-rank † | 45.44 | 57.16 | 11.51 | 15.85 | | **Qwen2.5-7B** | *dense FP16* | 74.27 | 81.37 | 53.17 | 75.74 | | | TR-3 (ours) | 65.43 | 75.20 | 40.79 | 66.34 | † `Matrix low-rank` on Qwen2.5-3B is our own internal matrix-factorisation ablation, not a published method. ‡ Official SVD-LLM V2 code was never released; these are our faithful implementation of the published algorithms. Its close agreement with the official SVD-LLM v1 rows (within ~1.5 pp on every zero-shot task) is the evidence that the re-implementation is sound. **GSM8K on Qwen2.5 uses lm-eval's `strict-match` filter, which is fragile to output formatting on this model family.** Under `strict-match` the dense Qwen2.5-3B teacher scores 10.31 while its compressed student scores 41.32; under `flexible-extract` the same teacher scores 63.46. The Qwen GSM8K column should therefore not be read as a capability ranking. LLaMA-2 rows are unaffected (strict and flexible agree to <0.2 pp on the dense teachers).