Spaces:
Running
Running
update
Browse files- index.html +17 -9
index.html
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
<!doctype html><html lang="en"><head><meta charset="utf-8">
|
| 2 |
<meta name="viewport" content="width=device-width, initial-scale=1">
|
| 3 |
-
<title>Free 6% lower perplexity: a new MoE architecture</title>
|
| 4 |
<meta name="description" content="int3 MoE beats full-precision MoE: run each MoE layer twice while its weights are still in L2. 6% lower perplexity, 22% of the expert bytes.">
|
| 5 |
<style>
|
| 6 |
body{margin:0;background:#fbfbfa;color:#1c1c1c;font:17px/1.65 -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Helvetica,Arial,sans-serif}
|
|
@@ -22,7 +22,7 @@ blockquote{margin:1.2em 0 1.4em;padding:18px 22px;background:#f4f0ff;border-left
|
|
| 22 |
blockquote h2{margin:0 0 .35em;border:0;padding:0;font-size:1.75em;line-height:1.2;color:#4c1d95}
|
| 23 |
blockquote p{margin:.3em 0}
|
| 24 |
</style></head><body><main>
|
| 25 |
-
<h1>Free 6% lower perplexity: a new MoE architecture</h1>
|
| 26 |
|
| 27 |
<blockquote>
|
| 28 |
<h2>int3 MoE beats full-precision MoE.</h2>
|
|
@@ -31,18 +31,22 @@ blockquote p{margin:.3em 0}
|
|
| 31 |
<table>
|
| 32 |
<thead>
|
| 33 |
<tr>
|
| 34 |
-
<th>result</th>
|
| 35 |
<th>measured</th>
|
| 36 |
</tr>
|
| 37 |
</thead>
|
| 38 |
<tbody>
|
| 39 |
<tr>
|
| 40 |
-
<td><strong>
|
| 41 |
-
<td>at equal parameters (loss β0.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 42 |
</tr>
|
| 43 |
<tr>
|
| 44 |
<td><strong>int3 > full precision</strong></td>
|
| 45 |
-
<td>int3 version
|
| 46 |
</tr>
|
| 47 |
<tr>
|
| 48 |
<td><strong>22% of the expert bytes</strong></td>
|
|
@@ -50,11 +54,15 @@ blockquote p{margin:.3em 0}
|
|
| 50 |
</tr>
|
| 51 |
<tr>
|
| 52 |
<td><strong>2nd pass hits L2</strong></td>
|
| 53 |
-
<td>
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
</tr>
|
| 55 |
</tbody>
|
| 56 |
</table>
|
| 57 |
-
<p><strong>The catch: repeat attention, not just the experts.</strong> Re-running the experts alone gives nothing. The gain comes from a second round of token mixing, so the second pass must include attention.</p>
|
| 58 |
<p><strong>The architecture, in four pieces.</strong></p>
|
| 59 |
<ol>
|
| 60 |
<li><strong>Whole-layer in-place repeat.</strong> Every MoE layer runs twice: attention, then the experts, then both again.</li>
|
|
@@ -64,7 +72,7 @@ blockquote p{margin:.3em 0}
|
|
| 64 |
</ol>
|
| 65 |
<p><strong>Why it matters.</strong> At batch size 1, a GPU spends most of each token waiting for weights to arrive from VRAM. MoEs have huge total parameter counts, so they hit this wall hardest. This architecture spends the idle compute on a second pass over weights that are already on chip, and turns that into quality you can then trade for fewer bytes.</p>
|
| 66 |
<p><img alt="Repeating the whole layer keeps a standard MoE beaten throughout training" src="https://huggingface.co/spaces/Schroding18/loop-in-L2/resolve/main/fig_gap_main.png" /></p>
|
| 67 |
-
<p><em>Loss gap to a standard MoE of the same size, through training. Whole-layer repeat (purple) pulls below the standard MoE
|
| 68 |
<p><strong>What is new.</strong> We treat the GPU's L2 cache as a design target. Quantize the repeated layers until they fit in L2, run them again while they are still there, and use that cheap second pass to win back the quality the low bit-width cost. We measure this loop end to end: a controlled sweep of bit-width Γ loop count at equal weight bytes (two seeds, int3βint8, 1β6 loops), the cache cliff on GPUs with different L2 sizes, and the same idea applied to MoE experts.</p>
|
| 69 |
<blockquote>
|
| 70 |
<p><strong>L2Loop has evidence for trading compute for precision and memory, but not yet for trading compute for parameter diversity.</strong></p>
|
|
|
|
| 1 |
<!doctype html><html lang="en"><head><meta charset="utf-8">
|
| 2 |
<meta name="viewport" content="width=device-width, initial-scale=1">
|
| 3 |
+
<title>Free 4β6% lower perplexity: a new MoE architecture</title>
|
| 4 |
<meta name="description" content="int3 MoE beats full-precision MoE: run each MoE layer twice while its weights are still in L2. 6% lower perplexity, 22% of the expert bytes.">
|
| 5 |
<style>
|
| 6 |
body{margin:0;background:#fbfbfa;color:#1c1c1c;font:17px/1.65 -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Helvetica,Arial,sans-serif}
|
|
|
|
| 22 |
blockquote h2{margin:0 0 .35em;border:0;padding:0;font-size:1.75em;line-height:1.2;color:#4c1d95}
|
| 23 |
blockquote p{margin:.3em 0}
|
| 24 |
</style></head><body><main>
|
| 25 |
+
<h1>Free 4β6% lower perplexity: a new MoE architecture</h1>
|
| 26 |
|
| 27 |
<blockquote>
|
| 28 |
<h2>int3 MoE beats full-precision MoE.</h2>
|
|
|
|
| 31 |
<table>
|
| 32 |
<thead>
|
| 33 |
<tr>
|
| 34 |
+
<th>result (8 layers, d768, 670M params, 524M tokens)</th>
|
| 35 |
<th>measured</th>
|
| 36 |
</tr>
|
| 37 |
</thead>
|
| 38 |
<tbody>
|
| 39 |
<tr>
|
| 40 |
+
<td><strong>4.0% lower perplexity</strong></td>
|
| 41 |
+
<td>at equal parameters (loss β0.041; standard MoE seed spread 0.007)</td>
|
| 42 |
+
</tr>
|
| 43 |
+
<tr>
|
| 44 |
+
<td><strong>β₯ 2Γ wider experts</strong></td>
|
| 45 |
+
<td>slightly better than a standard MoE with 2Γ wider experts (β0.037), with half its expert parameters</td>
|
| 46 |
</tr>
|
| 47 |
<tr>
|
| 48 |
<td><strong>int3 > full precision</strong></td>
|
| 49 |
+
<td>int3 version 0.024 below the full-precision standard MoE (2.4% lower perplexity)</td>
|
| 50 |
</tr>
|
| 51 |
<tr>
|
| 52 |
<td><strong>22% of the expert bytes</strong></td>
|
|
|
|
| 54 |
</tr>
|
| 55 |
<tr>
|
| 56 |
<td><strong>2nd pass hits L2</strong></td>
|
| 57 |
+
<td>RTX 5080: pass-2 weight reads drop to the L2-hot floor; optimised pass 2 costs 0.47Γ a fresh layer (Qwen3-30B-A3B layer shape, int4)</td>
|
| 58 |
+
</tr>
|
| 59 |
+
<tr>
|
| 60 |
+
<td>smaller model (6 layers, d512)</td>
|
| 61 |
+
<td>same picture, larger gaps: β6.0% / int3 β5.4%</td>
|
| 62 |
</tr>
|
| 63 |
</tbody>
|
| 64 |
</table>
|
| 65 |
+
<p><strong>The catch: repeat attention, not just the experts.</strong> Re-running the same experts alone gives almost nothing (β0.010 at 8 layers, 3 seeds). The gain comes from a second round of token mixing, so the second pass must include attention.</p>
|
| 66 |
<p><strong>The architecture, in four pieces.</strong></p>
|
| 67 |
<ol>
|
| 68 |
<li><strong>Whole-layer in-place repeat.</strong> Every MoE layer runs twice: attention, then the experts, then both again.</li>
|
|
|
|
| 72 |
</ol>
|
| 73 |
<p><strong>Why it matters.</strong> At batch size 1, a GPU spends most of each token waiting for weights to arrive from VRAM. MoEs have huge total parameter counts, so they hit this wall hardest. This architecture spends the idle compute on a second pass over weights that are already on chip, and turns that into quality you can then trade for fewer bytes.</p>
|
| 74 |
<p><img alt="Repeating the whole layer keeps a standard MoE beaten throughout training" src="https://huggingface.co/spaces/Schroding18/loop-in-L2/resolve/main/fig_gap_main.png" /></p>
|
| 75 |
+
<p><em>Loss gap to a standard MoE of the same size, through training, at two model sizes. Whole-layer repeat (purple) pulls below the standard MoE early and stays far outside its seed range to the end: β0.062 at 6 layers, β0.040 at 8 layers, where it also edges past a standard MoE with 2Γ wider experts (grey dashed). The int3 versions (lighter purple) stay below the full-precision baseline. Repeating only the experts (orange) gains about a quarter as much.</em></p>
|
| 76 |
<p><strong>What is new.</strong> We treat the GPU's L2 cache as a design target. Quantize the repeated layers until they fit in L2, run them again while they are still there, and use that cheap second pass to win back the quality the low bit-width cost. We measure this loop end to end: a controlled sweep of bit-width Γ loop count at equal weight bytes (two seeds, int3βint8, 1β6 loops), the cache cliff on GPUs with different L2 sizes, and the same idea applied to MoE experts.</p>
|
| 77 |
<blockquote>
|
| 78 |
<p><strong>L2Loop has evidence for trading compute for precision and memory, but not yet for trading compute for parameter diversity.</strong></p>
|