Spaces:
Running
Running
update
Browse files- index.html +2 -2
index.html
CHANGED
|
@@ -117,7 +117,7 @@ blockquote p{margin:.3em 0}
|
|
| 117 |
<tr>
|
| 118 |
<td>12 layers, d512</td>
|
| 119 |
<td>303M</td>
|
| 120 |
-
<td>β0.
|
| 121 |
<td>not run</td>
|
| 122 |
</tr>
|
| 123 |
</tbody>
|
|
@@ -127,7 +127,7 @@ blockquote p{margin:.3em 0}
|
|
| 127 |
<p><strong>More training does not erase the gain.</strong> At 6 layers, tripling the tokens (each run trained to completion) keeps it at β0.066.</p>
|
| 128 |
</li>
|
| 129 |
<li>
|
| 130 |
-
<p><strong>Deeper models shrink it.</strong> From 6 to 8 to 12 layers the gain falls from β0.07 to β0.04 to
|
| 131 |
</li>
|
| 132 |
<li>
|
| 133 |
<p><strong>Open question:</strong> whether a depth-aware recipe (looping only some layers, or scaling the second pass with depth) keeps the 6-layer gain in deep models. Production MoEs have 48β61 layers, so this decides whether the architecture matters at scale.</p>
|
|
|
|
| 117 |
<tr>
|
| 118 |
<td>12 layers, d512</td>
|
| 119 |
<td>303M</td>
|
| 120 |
+
<td><strong>β0.027</strong> (zero-init variant β0.025)</td>
|
| 121 |
<td>not run</td>
|
| 122 |
</tr>
|
| 123 |
</tbody>
|
|
|
|
| 127 |
<p><strong>More training does not erase the gain.</strong> At 6 layers, tripling the tokens (each run trained to completion) keeps it at β0.066.</p>
|
| 128 |
</li>
|
| 129 |
<li>
|
| 130 |
+
<p><strong>Deeper models shrink it.</strong> From 6 to 8 to 12 layers the gain falls from β0.07 to β0.04 to β0.01β0.03. At the same width and token budget, going from 6 to 12 layers cuts it from β0.066 to β0.027 (still growing at the end of training), so depth is the main factor. Zero-initialising the second pass does not fix it (β0.025).</p>
|
| 131 |
</li>
|
| 132 |
<li>
|
| 133 |
<p><strong>Open question:</strong> whether a depth-aware recipe (looping only some layers, or scaling the second pass with depth) keeps the 6-layer gain in deep models. Production MoEs have 48β61 layers, so this decides whether the architecture matters at scale.</p>
|