Spaces:
Running
Running
update
Browse files- index.html +6 -6
index.html
CHANGED
|
@@ -1019,9 +1019,9 @@ svg .lab{font-size:12px;fill:#333} svg .val{font-size:12px;fill:#111;font-weight
|
|
| 1019 |
<tr>
|
| 1020 |
<td>standard MoE</td>
|
| 1021 |
<td>3.3948</td>
|
| 1022 |
-
<td>
|
| 1023 |
-
<td>3.
|
| 1024 |
-
<td>
|
| 1025 |
</tr>
|
| 1026 |
<tr>
|
| 1027 |
<td>expert repeat, no re-injection</td>
|
|
@@ -1039,7 +1039,7 @@ svg .lab{font-size:12px;fill:#333} svg .val{font-size:12px;fill:#111;font-weight
|
|
| 1039 |
</tr>
|
| 1040 |
</tbody>
|
| 1041 |
</table>
|
| 1042 |
-
<p>Both re-injection seeds are below
|
| 1043 |
<p><strong>Fewer bytes, same or better quality.</strong> The repeat buys precision. Expert repeat + re-injection quantized to GPTQ int4 beats the standard MoE at int8, while reading 0.56Γ the expert bytes per token:</p>
|
| 1044 |
<table>
|
| 1045 |
<thead>
|
|
@@ -1065,7 +1065,7 @@ svg .lab{font-size:12px;fill:#333} svg .val{font-size:12px;fill:#111;font-weight
|
|
| 1065 |
</tr>
|
| 1066 |
</tbody>
|
| 1067 |
</table>
|
| 1068 |
-
<p><em>Expert weights quantized; negative = the repeat arm is better at fewer bytes.
|
| 1069 |
<p><strong>What is established and what is not.</strong></p>
|
| 1070 |
<table>
|
| 1071 |
<thead>
|
|
@@ -1105,7 +1105,7 @@ svg .lab{font-size:12px;fill:#333} svg .val{font-size:12px;fill:#111;font-weight
|
|
| 1105 |
</tr>
|
| 1106 |
<tr>
|
| 1107 |
<td>Expert repeat and MoE quality when trained from scratch</td>
|
| 1108 |
-
<td><strong>with re-injection: below standard MoE on both seeds (β0.
|
| 1109 |
</tr>
|
| 1110 |
</tbody>
|
| 1111 |
</table>
|
|
|
|
| 1019 |
<tr>
|
| 1020 |
<td>standard MoE</td>
|
| 1021 |
<td>3.3948</td>
|
| 1022 |
+
<td>3.3986</td>
|
| 1023 |
+
<td>3.3967</td>
|
| 1024 |
+
<td>0.004</td>
|
| 1025 |
</tr>
|
| 1026 |
<tr>
|
| 1027 |
<td>expert repeat, no re-injection</td>
|
|
|
|
| 1039 |
</tr>
|
| 1040 |
</tbody>
|
| 1041 |
</table>
|
| 1042 |
+
<p>Both re-injection seeds are below both standard-MoE seeds, by 0.010 on average; the seeds agree to within 0.003 (re-injection) and 0.004 (standard). Plain repeat ties the standard MoE on average and varies five times more between seeds. The mechanism shows in the cosine between the two expert updates: 0.82 without re-injection, 0.40β0.44 with it. The second pass stops copying the first.</p>
|
| 1043 |
<p><strong>Fewer bytes, same or better quality.</strong> The repeat buys precision. Expert repeat + re-injection quantized to GPTQ int4 beats the standard MoE at int8, while reading 0.56Γ the expert bytes per token:</p>
|
| 1044 |
<table>
|
| 1045 |
<thead>
|
|
|
|
| 1065 |
</tr>
|
| 1066 |
</tbody>
|
| 1067 |
</table>
|
| 1068 |
+
<p><em>Expert weights quantized; negative = the repeat arm is better at fewer bytes. At every precision we tried (int8, GPTQ int4, GPTQ int3) the re-injection arm stays 0.006β0.010 below the standard MoE at the same bytes. Earlier small runs on two DGX Sparks (118M tokens) tied: β0.0002 and +0.037.</em></p>
|
| 1069 |
<p><strong>What is established and what is not.</strong></p>
|
| 1070 |
<table>
|
| 1071 |
<thead>
|
|
|
|
| 1105 |
</tr>
|
| 1106 |
<tr>
|
| 1107 |
<td>Expert repeat and MoE quality when trained from scratch</td>
|
| 1108 |
+
<td><strong>with re-injection: below standard MoE on both seeds (β0.010 mean; every repeat seed below every standard seed, two seeds each), and at int4 it beats standard int8 with 0.56Γ the expert bytes; without re-injection: ties (mean +0.001).</strong> Larger scales running</td>
|
| 1109 |
</tr>
|
| 1110 |
</tbody>
|
| 1111 |
</table>
|