Card: resume check stated with both computations
Browse files
README.md
CHANGED
|
@@ -117,7 +117,7 @@ any shared 13-gram (none β₯ 50 %).
|
|
| 117 |
| Schedule | warmup 150, constant lr 6e-4 through step 2289, 1 β sqrt decay over steps 2290β2860 to lr / 10 (6.05e-5 at step 2860) |
|
| 118 |
| Optimizer | Muon (lr 0.02) for hidden weight matrices + AdamW (lr 6e-4, weight decay 0.1) for the rest |
|
| 119 |
| Final training loss | 2.5067 (step 2860); mean of the values logged at steps 2770β2860: 2.5083 (1.00, steps 1340β1430: 2.6990) |
|
| 120 |
-
| Resume check |
|
| 121 |
| Checkpoints | every 150 steps + final; last before the decay: step 2250 |
|
| 122 |
|
| 123 |
## Limitations
|
|
|
|
| 117 |
| Schedule | warmup 150, constant lr 6e-4 through step 2289, 1 β sqrt decay over steps 2290β2860 to lr / 10 (6.05e-5 at step 2860) |
|
| 118 |
| Optimizer | Muon (lr 0.02) for hidden weight matrices + AdamW (lr 6e-4, weight decay 0.1) for the rest |
|
| 119 |
| Final training loss | 2.5067 (step 2860); mean of the values logged at steps 2770β2860: 2.5083 (1.00, steps 1340β1430: 2.6990) |
|
| 120 |
+
| Resume check | training loss after the resume vs. the 1.00 run at the same steps (same batches, same lr): mean difference about 1e-4 nats (two independent computations: β0.00008 over steps 1060β1140, β0.00013 over 1060β1130), largest single-step difference 0.0007 |
|
| 121 |
| Checkpoints | every 150 steps + final; last before the decay: step 2250 |
|
| 122 |
|
| 123 |
## Limitations
|