Schroding18 commited on
Commit
dd26ec3
Β·
verified Β·
1 Parent(s): 8398cce
Files changed (1) hide show
  1. index.html +6 -6
index.html CHANGED
@@ -1019,9 +1019,9 @@ svg .lab{font-size:12px;fill:#333} svg .val{font-size:12px;fill:#111;font-weight
1019
  <tr>
1020
  <td>standard MoE</td>
1021
  <td>3.3948</td>
1022
- <td>running</td>
1023
- <td>3.3948</td>
1024
- <td>β€”</td>
1025
  </tr>
1026
  <tr>
1027
  <td>expert repeat, no re-injection</td>
@@ -1039,7 +1039,7 @@ svg .lab{font-size:12px;fill:#333} svg .val{font-size:12px;fill:#111;font-weight
1039
  </tr>
1040
  </tbody>
1041
  </table>
1042
- <p>Both re-injection seeds are below the standard MoE, by 0.008 on average, and they agree to within 0.003. Plain repeat ties the standard MoE on average and varies five times more between seeds. The mechanism shows in the cosine between the two expert updates: 0.82 without re-injection, 0.40–0.44 with it. The second pass stops copying the first.</p>
1043
  <p><strong>Fewer bytes, same or better quality.</strong> The repeat buys precision. Expert repeat + re-injection quantized to GPTQ int4 beats the standard MoE at int8, while reading 0.56Γ— the expert bytes per token:</p>
1044
  <table>
1045
  <thead>
@@ -1065,7 +1065,7 @@ svg .lab{font-size:12px;fill:#333} svg .val{font-size:12px;fill:#111;font-weight
1065
  </tr>
1066
  </tbody>
1067
  </table>
1068
- <p><em>Expert weights quantized; negative = the repeat arm is better at fewer bytes. The second standard-MoE seed is still training; with one standard seed so far, every re-injection seed is below it. Earlier small runs on two DGX Sparks (118M tokens) tied: βˆ’0.0002 and +0.037.</em></p>
1069
  <p><strong>What is established and what is not.</strong></p>
1070
  <table>
1071
  <thead>
@@ -1105,7 +1105,7 @@ svg .lab{font-size:12px;fill:#333} svg .val{font-size:12px;fill:#111;font-weight
1105
  </tr>
1106
  <tr>
1107
  <td>Expert repeat and MoE quality when trained from scratch</td>
1108
- <td><strong>with re-injection: below standard MoE on both seeds (βˆ’0.008 mean, seeds within 0.003), and at int4 it beats standard int8 with 0.56Γ— the expert bytes; without re-injection: ties (mean +0.001).</strong> Second standard-MoE seed still training</td>
1109
  </tr>
1110
  </tbody>
1111
  </table>
 
1019
  <tr>
1020
  <td>standard MoE</td>
1021
  <td>3.3948</td>
1022
+ <td>3.3986</td>
1023
+ <td>3.3967</td>
1024
+ <td>0.004</td>
1025
  </tr>
1026
  <tr>
1027
  <td>expert repeat, no re-injection</td>
 
1039
  </tr>
1040
  </tbody>
1041
  </table>
1042
+ <p>Both re-injection seeds are below both standard-MoE seeds, by 0.010 on average; the seeds agree to within 0.003 (re-injection) and 0.004 (standard). Plain repeat ties the standard MoE on average and varies five times more between seeds. The mechanism shows in the cosine between the two expert updates: 0.82 without re-injection, 0.40–0.44 with it. The second pass stops copying the first.</p>
1043
  <p><strong>Fewer bytes, same or better quality.</strong> The repeat buys precision. Expert repeat + re-injection quantized to GPTQ int4 beats the standard MoE at int8, while reading 0.56Γ— the expert bytes per token:</p>
1044
  <table>
1045
  <thead>
 
1065
  </tr>
1066
  </tbody>
1067
  </table>
1068
+ <p><em>Expert weights quantized; negative = the repeat arm is better at fewer bytes. At every precision we tried (int8, GPTQ int4, GPTQ int3) the re-injection arm stays 0.006–0.010 below the standard MoE at the same bytes. Earlier small runs on two DGX Sparks (118M tokens) tied: βˆ’0.0002 and +0.037.</em></p>
1069
  <p><strong>What is established and what is not.</strong></p>
1070
  <table>
1071
  <thead>
 
1105
  </tr>
1106
  <tr>
1107
  <td>Expert repeat and MoE quality when trained from scratch</td>
1108
+ <td><strong>with re-injection: below standard MoE on both seeds (βˆ’0.010 mean; every repeat seed below every standard seed, two seeds each), and at int4 it beats standard int8 with 0.56Γ— the expert bytes; without re-injection: ties (mean +0.001).</strong> Larger scales running</td>
1109
  </tr>
1110
  </tbody>
1111
  </table>