Schroding18 commited on
Commit
7bcbf93
Β·
verified Β·
1 Parent(s): 642b212
Files changed (1) hide show
  1. index.html +17 -9
index.html CHANGED
@@ -1,6 +1,6 @@
1
  <!doctype html><html lang="en"><head><meta charset="utf-8">
2
  <meta name="viewport" content="width=device-width, initial-scale=1">
3
- <title>Free 6% lower perplexity: a new MoE architecture</title>
4
  <meta name="description" content="int3 MoE beats full-precision MoE: run each MoE layer twice while its weights are still in L2. 6% lower perplexity, 22% of the expert bytes.">
5
  <style>
6
  body{margin:0;background:#fbfbfa;color:#1c1c1c;font:17px/1.65 -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Helvetica,Arial,sans-serif}
@@ -22,7 +22,7 @@ blockquote{margin:1.2em 0 1.4em;padding:18px 22px;background:#f4f0ff;border-left
22
  blockquote h2{margin:0 0 .35em;border:0;padding:0;font-size:1.75em;line-height:1.2;color:#4c1d95}
23
  blockquote p{margin:.3em 0}
24
  </style></head><body><main>
25
- <h1>Free 6% lower perplexity: a new MoE architecture</h1>
26
 
27
  <blockquote>
28
  <h2>int3 MoE beats full-precision MoE.</h2>
@@ -31,18 +31,22 @@ blockquote p{margin:.3em 0}
31
  <table>
32
  <thead>
33
  <tr>
34
- <th>result</th>
35
  <th>measured</th>
36
  </tr>
37
  </thead>
38
  <tbody>
39
  <tr>
40
- <td><strong>6% lower perplexity</strong></td>
41
- <td>at equal parameters (loss βˆ’0.062, β‰ˆ3.4 seed standard deviations)</td>
 
 
 
 
42
  </tr>
43
  <tr>
44
  <td><strong>int3 &gt; full precision</strong></td>
45
- <td>int3 version still 0.056 below the full-precision standard MoE</td>
46
  </tr>
47
  <tr>
48
  <td><strong>22% of the expert bytes</strong></td>
@@ -50,11 +54,15 @@ blockquote p{margin:.3em 0}
50
  </tr>
51
  <tr>
52
  <td><strong>2nd pass hits L2</strong></td>
53
- <td>measured on an RTX 5080: pass-2 weight reads drop to the L2-hot floor</td>
 
 
 
 
54
  </tr>
55
  </tbody>
56
  </table>
57
- <p><strong>The catch: repeat attention, not just the experts.</strong> Re-running the experts alone gives nothing. The gain comes from a second round of token mixing, so the second pass must include attention.</p>
58
  <p><strong>The architecture, in four pieces.</strong></p>
59
  <ol>
60
  <li><strong>Whole-layer in-place repeat.</strong> Every MoE layer runs twice: attention, then the experts, then both again.</li>
@@ -64,7 +72,7 @@ blockquote p{margin:.3em 0}
64
  </ol>
65
  <p><strong>Why it matters.</strong> At batch size 1, a GPU spends most of each token waiting for weights to arrive from VRAM. MoEs have huge total parameter counts, so they hit this wall hardest. This architecture spends the idle compute on a second pass over weights that are already on chip, and turns that into quality you can then trade for fewer bytes.</p>
66
  <p><img alt="Repeating the whole layer keeps a standard MoE beaten throughout training" src="https://huggingface.co/spaces/Schroding18/loop-in-L2/resolve/main/fig_gap_main.png" /></p>
67
- <p><em>Loss gap to a standard MoE of the same size, through training. Whole-layer repeat (purple) pulls below the standard MoE within the first 30M tokens and never returns: it stays far outside the standard MoE's seed-to-seed range for the rest of training and ends 0.062 lower. The int3 version tracks it the whole way. Repeating only the experts (orange) never does. The whole-layer gap is larger than what doubling the expert parameters buys at 8 layers (grey, βˆ’0.036). This is the shape of a structural gain, not a lucky endpoint; the 8-layer whole-layer run is on its way.</em></p>
68
  <p><strong>What is new.</strong> We treat the GPU's L2 cache as a design target. Quantize the repeated layers until they fit in L2, run them again while they are still there, and use that cheap second pass to win back the quality the low bit-width cost. We measure this loop end to end: a controlled sweep of bit-width Γ— loop count at equal weight bytes (two seeds, int3–int8, 1–6 loops), the cache cliff on GPUs with different L2 sizes, and the same idea applied to MoE experts.</p>
69
  <blockquote>
70
  <p><strong>L2Loop has evidence for trading compute for precision and memory, but not yet for trading compute for parameter diversity.</strong></p>
 
1
  <!doctype html><html lang="en"><head><meta charset="utf-8">
2
  <meta name="viewport" content="width=device-width, initial-scale=1">
3
+ <title>Free 4–6% lower perplexity: a new MoE architecture</title>
4
  <meta name="description" content="int3 MoE beats full-precision MoE: run each MoE layer twice while its weights are still in L2. 6% lower perplexity, 22% of the expert bytes.">
5
  <style>
6
  body{margin:0;background:#fbfbfa;color:#1c1c1c;font:17px/1.65 -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Helvetica,Arial,sans-serif}
 
22
  blockquote h2{margin:0 0 .35em;border:0;padding:0;font-size:1.75em;line-height:1.2;color:#4c1d95}
23
  blockquote p{margin:.3em 0}
24
  </style></head><body><main>
25
+ <h1>Free 4–6% lower perplexity: a new MoE architecture</h1>
26
 
27
  <blockquote>
28
  <h2>int3 MoE beats full-precision MoE.</h2>
 
31
  <table>
32
  <thead>
33
  <tr>
34
+ <th>result (8 layers, d768, 670M params, 524M tokens)</th>
35
  <th>measured</th>
36
  </tr>
37
  </thead>
38
  <tbody>
39
  <tr>
40
+ <td><strong>4.0% lower perplexity</strong></td>
41
+ <td>at equal parameters (loss βˆ’0.041; standard MoE seed spread 0.007)</td>
42
+ </tr>
43
+ <tr>
44
+ <td><strong>β‰₯ 2Γ— wider experts</strong></td>
45
+ <td>slightly better than a standard MoE with 2Γ— wider experts (βˆ’0.037), with half its expert parameters</td>
46
  </tr>
47
  <tr>
48
  <td><strong>int3 &gt; full precision</strong></td>
49
+ <td>int3 version 0.024 below the full-precision standard MoE (2.4% lower perplexity)</td>
50
  </tr>
51
  <tr>
52
  <td><strong>22% of the expert bytes</strong></td>
 
54
  </tr>
55
  <tr>
56
  <td><strong>2nd pass hits L2</strong></td>
57
+ <td>RTX 5080: pass-2 weight reads drop to the L2-hot floor; optimised pass 2 costs 0.47Γ— a fresh layer (Qwen3-30B-A3B layer shape, int4)</td>
58
+ </tr>
59
+ <tr>
60
+ <td>smaller model (6 layers, d512)</td>
61
+ <td>same picture, larger gaps: βˆ’6.0% / int3 βˆ’5.4%</td>
62
  </tr>
63
  </tbody>
64
  </table>
65
+ <p><strong>The catch: repeat attention, not just the experts.</strong> Re-running the same experts alone gives almost nothing (βˆ’0.010 at 8 layers, 3 seeds). The gain comes from a second round of token mixing, so the second pass must include attention.</p>
66
  <p><strong>The architecture, in four pieces.</strong></p>
67
  <ol>
68
  <li><strong>Whole-layer in-place repeat.</strong> Every MoE layer runs twice: attention, then the experts, then both again.</li>
 
72
  </ol>
73
  <p><strong>Why it matters.</strong> At batch size 1, a GPU spends most of each token waiting for weights to arrive from VRAM. MoEs have huge total parameter counts, so they hit this wall hardest. This architecture spends the idle compute on a second pass over weights that are already on chip, and turns that into quality you can then trade for fewer bytes.</p>
74
  <p><img alt="Repeating the whole layer keeps a standard MoE beaten throughout training" src="https://huggingface.co/spaces/Schroding18/loop-in-L2/resolve/main/fig_gap_main.png" /></p>
75
+ <p><em>Loss gap to a standard MoE of the same size, through training, at two model sizes. Whole-layer repeat (purple) pulls below the standard MoE early and stays far outside its seed range to the end: βˆ’0.062 at 6 layers, βˆ’0.040 at 8 layers, where it also edges past a standard MoE with 2Γ— wider experts (grey dashed). The int3 versions (lighter purple) stay below the full-precision baseline. Repeating only the experts (orange) gains about a quarter as much.</em></p>
76
  <p><strong>What is new.</strong> We treat the GPU's L2 cache as a design target. Quantize the repeated layers until they fit in L2, run them again while they are still there, and use that cheap second pass to win back the quality the low bit-width cost. We measure this loop end to end: a controlled sweep of bit-width Γ— loop count at equal weight bytes (two seeds, int3–int8, 1–6 loops), the cache cliff on GPUs with different L2 sizes, and the same idea applied to MoE experts.</p>
77
  <blockquote>
78
  <p><strong>L2Loop has evidence for trading compute for precision and memory, but not yet for trading compute for parameter diversity.</strong></p>