Add TR-3 and ASVD on LLaMA-2-13B base; 13B base block now has all four arms

README.md

@@ -78,6 +78,8 @@ Perplexity is **token-level** over fixed non-overlapping 2048-token windows, fol
 | | SVD-LLM V2 ‡ | 4.023 | 9.73 | -- |
 | | ASVD | 4.023 | 10.04 | 12.48 |
 | **LLaMA-2-13B (base)** | *dense FP16* | 13.016 | 4.88 | 6.74 |
+| | TR-3 (ours) | 7.740 | 6.18 | 8.51 |
+| | ASVD | 7.746 | 7.97 | 10.44 |
 | | SVD-LLM | 7.738 | 9.14 | -- |
 | | SVD-LLM V2 ‡ | 7.740 | 8.74 | -- |
 | **Qwen2.5-3B** | *dense FP16* | 3.086 | 8.56 | -- |
@@ -108,6 +110,8 @@ Perplexity is **token-level** over fixed non-overlapping 2048-token windows, fol
 |  | SVD-LLM V2 ‡ | 68.53 | 33.96 | 60.06 | 56.49 | 72.31 | 62.83 | 25.59 | 54.25 |
 |  | ASVD | 59.54 | 32.17 | 57.83 | 57.83 | 71.49 | 60.14 | 23.92 | 51.85 |
 | **LLaMA-2-13B (base)** | *dense FP16* | 82.11 | 48.81 | 76.47 | 76.54 | 80.25 | 72.45 | 31.46 | 66.87 |
+|  | TR-3 (ours) | 76.21 | 44.11 | 72.43 | 71.74 | 78.51 | 68.67 | 29.65 | 63.05 |
+|  | ASVD | 74.19 | 38.74 | 68.14 | 66.10 | 75.73 | 65.90 | 25.36 | 59.17 |
 |  | SVD-LLM | 70.34 | 36.01 | 64.94 | 60.94 | 73.78 | 65.82 | 24.69 | 56.64 |
 |  | SVD-LLM V2 ‡ | 71.13 | 38.14 | 67.30 | 61.67 | 73.83 | 65.67 | 25.66 | 57.63 |
 | **Qwen2.5-3B** | *dense FP16* | 80.09 | 48.04 | 72.90 | 66.12 | 78.29 | 69.38 | 35.61 | 64.35 |
@@ -134,13 +138,14 @@ Perplexity is **token-level** over fixed non-overlapping 2048-token windows, fol
 |  | ASVD | 41.43 | 68.01 | 29.71 | 7.51 |
 | **LLaMA-2-7B (base)** | *dense FP16* | 45.97 | 78.95 | 64.39 | 13.12 |
 |  | TR-3 (ours) | 39.49 | 71.11 | 39.75 | 6.52 |
-|  | SVD-LLM | 32.74 | 58.36 | 16.43 | 1.59 |
-|  | SVD-LLM V2 ‡ | 34.33 | 59.36 | 19.34 | 1.97 |
-|  | ASVD | 26.58 | 61.21 | 20.69 | 1.29 |
+|  | SVD-LLM | -- | -- | -- | -- |
+|  | SVD-LLM V2 ‡ | -- | -- | -- | -- |
+|  | ASVD | -- | -- | -- | -- |
 | **LLaMA-2-13B (base)** | *dense FP16* | 55.54 | 82.14 | 70.69 | 23.28 |
-|  | SVD-LLM | 42.70 | 63.80 | 21.87 | 4.25 |
-|  | SVD-LLM V2 ‡ | 44.67 | 65.00 | 24.49 | 4.93 |
-|  | ASVD | 42.44 | 70.15 | 35.50 | 5.23 |
+|  | TR-3 (ours) | 50.46 | -- | 50.96 | 14.33 |
+|  | ASVD | -- | -- | -- | -- |
+|  | SVD-LLM | -- | -- | -- | -- |
+|  | SVD-LLM V2 ‡ | -- | -- | -- | -- |
 | **Qwen2.5-3B** | *dense FP16* | 66.42 | 75.26 | 45.78 | 10.31 |
 |  | TR-3 (ours) | 56.02 | 66.39 | 27.61 | 41.32 |
 |  | TT-3 (open chain) | 41.97 | 51.65 | 11.86 | 9.78 |