Add full benchmark tables (perplexity / zero-shot / few-shot) for all methods and models

README.md

@@ -47,3 +47,110 @@ implementation of the published algorithms (per-layer ranks document the allocat
 `tr3/**` stores **ring cores** (load with each dir's `tensor_formats.py`); `svd_llm*/**` stores
 **low-rank factor pairs** (load with each dir's `lowrank_linear.py`). Weight files are
 uniformly named `state_dict_fp16.pt`. `from_pretrained()` will not work on this repo.
+
+## Benchmark results
+
+All numbers below were produced by us under a single locked protocol: **lm-eval-harness 0.4.12, fp16, seed 1234**; MMLU 5-shot, HellaSwag 10-shot, everything else 0-shot; full benchmarks, no subsampling. Every baseline was re-run at the **same parameter budget and the same 60M-token KD recipe** as our model rather than quoted from its paper, so the rows within a block are directly comparable.
+
+Perplexity is **token-level** over fixed non-overlapping 2048-token windows, following SVD-LLM's `evaluater.py`. It is *not* lm-eval's word-level perplexity; the two differ by the exponent `N_tok/N_word` (1.4156 for LLaMA-2 on WikiText-2) and must never be placed in the same column.
+
+`--` marks a cell not yet measured; it is a placeholder, not a zero or a failure.
+
+
+### Perplexity (lower is better)
+
+| Model | Method | Params (B) | WikiText-2 | C4 |
+|---|---|---:|---:|---:|
+| **LLaMA-2-7B-chat** | *dense FP16* | 6.738 | 6.94 | 9.48 |
+| | TR-3 (ours) | 4.023 | 8.34 | 11.10 |
+| | SVD-LLM | 4.022 | 11.49 | 15.09 |
+| | SVD-LLM V2 ‡ | 4.023 | 11.03 | 14.73 |
+| | ASVD | 4.023 | 11.22 | 14.18 |
+| | TT-3 (open chain) | 4.021 | 12.52 | 15.09 |
+| **LLaMA-2-13B-chat** | *dense FP16* | 13.016 | 6.11 | 8.52 |
+| | TR-3 (ours) | 7.740 | 6.98 | 9.65 |
+| | SVD-LLM | 7.738 | 10.15 | 13.46 |
+| | SVD-LLM V2 ‡ | 7.740 | 9.72 | 13.09 |
+| | ASVD | 7.736 | 9.13 | 11.88 |
+| **LLaMA-2-7B (base)** | *dense FP16* | 6.738 | 5.47 | 7.29 |
+| | TR-3 (ours) | 4.023 | 7.28 | 9.58 |
+| | SVD-LLM | 4.022 | 10.17 | -- |
+| | SVD-LLM V2 ‡ | 4.023 | 9.73 | -- |
+| | ASVD | 4.023 | 10.04 | -- |
+| **LLaMA-2-13B (base)** | *dense FP16* | 13.016 | 4.88 | 6.74 |
+| | SVD-LLM | 7.738 | -- | -- |
+| | SVD-LLM V2 ‡ | 7.740 | -- | -- |
+| **Qwen2.5-3B** | *dense FP16* | 3.086 | 8.56 | -- |
+| | TR-3 (ours) | 2.078 | 11.75 | -- |
+| | TT-3 (open chain) | 2.077 | -- | -- |
+| | Matrix low-rank † | 2.077 | -- | -- |
+| **Qwen2.5-7B** | *dense FP16* | 7.616 | 7.46 | -- |
+| | TR-3 (ours) | 5.187 | 9.51 | -- |
+
+### Zero-shot accuracy (%)
+
+| Model | Method | BoolQ | ARC-C | ARC-E | LAMBADA | PIQA | Winogrande | MathQA | Avg |
+|---|---|---:|---:|---:|---:|---:|---:|---:|---:|
+| **LLaMA-2-7B-chat** | *dense FP16* | 80.58 | 43.26 | 68.39 | 69.82 | 77.04 | 68.35 | 28.48 | 62.27 |
+|  | TR-3 (ours) | 77.09 | 38.48 | 65.11 | 66.52 | 75.52 | 66.30 | 27.50 | 59.50 |
+|  | SVD-LLM | 67.98 | 32.76 | 57.20 | 56.76 | 71.55 | 62.19 | 25.49 | 53.42 |
+|  | SVD-LLM V2 ‡ | 68.90 | 34.30 | 59.30 | 57.29 | 71.65 | 61.25 | 25.66 | 54.05 |
+|  | ASVD | 69.69 | 33.28 | 57.03 | 55.15 | 71.33 | 60.14 | 24.56 | 53.03 |
+|  | TT-3 (open chain) | 63.67 | 32.68 | 52.06 | 55.77 | 71.22 | 58.80 | 25.23 | 51.35 |
+| **LLaMA-2-13B-chat** | *dense FP16* | 81.96 | 49.91 | 73.40 | 72.79 | 79.33 | 72.85 | 32.86 | 66.16 |
+|  | TR-3 (ours) | 80.52 | 43.86 | 70.71 | 70.74 | 76.82 | 70.01 | 29.92 | 63.22 |
+|  | SVD-LLM | 73.64 | 35.41 | 61.03 | 60.59 | 72.74 | 64.48 | 25.93 | 56.26 |
+|  | SVD-LLM V2 ‡ | 75.14 | 36.35 | 61.99 | 62.47 | 73.34 | 66.14 | 27.14 | 57.51 |
+|  | ASVD | 74.89 | 36.95 | 64.60 | 61.60 | 74.05 | 65.19 | 26.37 | 57.66 |
+| **LLaMA-2-7B (base)** | *dense FP16* | 79.33 | 45.05 | 73.86 | 73.65 | 78.73 | 69.46 | 28.41 | 64.07 |
+|  | TR-3 (ours) | 70.95 | 39.16 | 65.24 | 68.02 | 75.35 | 64.48 | 25.66 | 58.41 |
+|  | SVD-LLM | 68.62 | 33.19 | 58.71 | 55.58 | 71.16 | 61.33 | 24.19 | 53.25 |
+|  | SVD-LLM V2 ‡ | 68.53 | 33.96 | 60.06 | 56.49 | 72.31 | 62.83 | 25.59 | 54.25 |
+|  | ASVD | 59.54 | -- | -- | -- | -- | -- | -- | -- |
+| **LLaMA-2-13B (base)** | *dense FP16* | 82.11 | 48.81 | 76.47 | 76.54 | 80.25 | 72.45 | 31.46 | 66.87 |
+|  | SVD-LLM | -- | -- | -- | -- | -- | -- | -- | -- |
+|  | SVD-LLM V2 ‡ | -- | -- | -- | -- | -- | -- | -- | -- |
+| **Qwen2.5-3B** | *dense FP16* | 80.09 | 48.04 | 72.90 | 66.12 | 78.29 | 69.38 | 35.61 | 64.35 |
+|  | TR-3 (ours) | 76.39 | 43.17 | 69.36 | 60.51 | 75.24 | 64.09 | 34.17 | 60.42 |
+|  | TT-3 (open chain) | 66.88 | 31.14 | 56.19 | 49.16 | 70.40 | 59.12 | 27.64 | 51.50 |
+|  | Matrix low-rank † | 71.35 | 36.43 | 59.55 | 50.51 | 72.14 | 61.88 | 28.41 | 54.33 |
+| **Qwen2.5-7B** | *dense FP16* | 86.42 | 54.86 | 80.93 | 69.77 | 80.36 | 70.56 | 40.37 | 69.04 |
+|  | TR-3 (ours) | 84.68 | 51.11 | 73.82 | 64.84 | 77.64 | 70.24 | 42.01 | 66.33 |
+
+### Few-shot / generative (%)
+
+| Model | Method | MMLU (5s) | HellaSwag (10s) | TriviaQA (5s) | GSM8K (5s) |
+|---|---|---:|---:|---:|---:|
+| **LLaMA-2-7B-chat** | *dense FP16* | 47.22 | 78.63 | 57.18 | 22.59 |
+|  | TR-3 (ours) | 43.21 | 70.75 | 36.93 | 9.40 |
+|  | SVD-LLM | 39.47 | 57.65 | 16.18 | 2.05 |
+|  | SVD-LLM V2 ‡ | 40.26 | 58.82 | 18.62 | 0.68 |
+|  | ASVD | 34.65 | 59.46 | 18.02 | 2.27 |
+|  | TT-3 (open chain) | 34.25 | 56.38 | 14.48 | 0.76 |
+| **LLaMA-2-13B-chat** | *dense FP16* | 53.55 | 82.07 | 65.20 | 35.25 |
+|  | TR-3 (ours) | 49.69 | 76.46 | 47.81 | 24.11 |
+|  | SVD-LLM | 44.18 | 63.41 | 21.57 | 0.15 |
+|  | SVD-LLM V2 ‡ | 44.77 | 64.99 | 24.52 | 0.15 |
+|  | ASVD | 41.43 | 68.01 | 29.71 | 7.51 |
+| **LLaMA-2-7B (base)** | *dense FP16* | 45.97 | 78.95 | 64.39 | 13.12 |
+|  | TR-3 (ours) | 39.49 | 71.11 | 39.75 | 6.52 |
+|  | SVD-LLM | -- | -- | -- | -- |
+|  | SVD-LLM V2 ‡ | -- | -- | -- | -- |
+|  | ASVD | -- | -- | -- | -- |
+| **LLaMA-2-13B (base)** | *dense FP16* | 55.54 | 82.14 | 70.69 | 23.28 |
+|  | SVD-LLM | -- | -- | -- | -- |
+|  | SVD-LLM V2 ‡ | -- | -- | -- | -- |
+| **Qwen2.5-3B** | *dense FP16* | 66.42 | 75.26 | 45.78 | 10.31 |
+|  | TR-3 (ours) | 56.02 | 66.39 | 27.61 | 41.32 |
+|  | TT-3 (open chain) | 41.97 | 51.65 | 11.86 | 9.78 |
+|  | Matrix low-rank † | 45.44 | 57.16 | 11.51 | 15.85 |
+| **Qwen2.5-7B** | *dense FP16* | 74.27 | 81.37 | 53.17 | 75.74 |
+|  | TR-3 (ours) | 65.43 | 75.20 | 40.79 | 66.34 |
+
+† `Matrix low-rank` on Qwen2.5-3B is our own internal matrix-factorisation ablation, not a published method.
+
+‡ Official SVD-LLM V2 code was never released; these are our faithful implementation of the published algorithms. Its close agreement with the official SVD-LLM v1 rows (within ~1.5 pp on every zero-shot task) is the evidence that the re-implementation is sound.
+
+
+**GSM8K on Qwen2.5 uses lm-eval's `strict-match` filter, which is fragile to output formatting on this model family.** Under `strict-match` the dense Qwen2.5-3B teacher scores 10.31 while its compressed student scores 41.32; under `flexible-extract` the same teacher scores 63.46. The Qwen GSM8K column should therefore not be read as a capability ranking. LLaMA-2 rows are unaffected (strict and flexible agree to <0.2 pp on the dense teachers).
+