Schroding18 commited on
Commit
f5a3516
Β·
verified Β·
1 Parent(s): 215b945
Files changed (1) hide show
  1. index.html +2 -2
index.html CHANGED
@@ -117,7 +117,7 @@ blockquote p{margin:.3em 0}
117
  <tr>
118
  <td>12 layers, d512</td>
119
  <td>303M</td>
120
- <td>βˆ’0.019 at 60% of training (running)</td>
121
  <td>not run</td>
122
  </tr>
123
  </tbody>
@@ -127,7 +127,7 @@ blockquote p{margin:.3em 0}
127
  <p><strong>More training does not erase the gain.</strong> At 6 layers, tripling the tokens (each run trained to completion) keeps it at βˆ’0.066.</p>
128
  </li>
129
  <li>
130
- <p><strong>Deeper models shrink it.</strong> From 6 to 8 to 12 layers the gain falls from βˆ’0.07 to βˆ’0.04 to about βˆ’0.01. A 12-layer model at the same width as the 6-layer one shows the same slow start, so depth, not width, is the main factor. Zero-initialising the second pass does not fix it.</p>
131
  </li>
132
  <li>
133
  <p><strong>Open question:</strong> whether a depth-aware recipe (looping only some layers, or scaling the second pass with depth) keeps the 6-layer gain in deep models. Production MoEs have 48–61 layers, so this decides whether the architecture matters at scale.</p>
 
117
  <tr>
118
  <td>12 layers, d512</td>
119
  <td>303M</td>
120
+ <td><strong>βˆ’0.027</strong> (zero-init variant βˆ’0.025)</td>
121
  <td>not run</td>
122
  </tr>
123
  </tbody>
 
127
  <p><strong>More training does not erase the gain.</strong> At 6 layers, tripling the tokens (each run trained to completion) keeps it at βˆ’0.066.</p>
128
  </li>
129
  <li>
130
+ <p><strong>Deeper models shrink it.</strong> From 6 to 8 to 12 layers the gain falls from βˆ’0.07 to βˆ’0.04 to βˆ’0.01–0.03. At the same width and token budget, going from 6 to 12 layers cuts it from βˆ’0.066 to βˆ’0.027 (still growing at the end of training), so depth is the main factor. Zero-initialising the second pass does not fix it (βˆ’0.025).</p>
131
  </li>
132
  <li>
133
  <p><strong>Open question:</strong> whether a depth-aware recipe (looping only some layers, or scaling the second pass with depth) keeps the 6-layer gain in deep models. Production MoEs have 48–61 layers, so this decides whether the architecture matters at scale.</p>