Maggio33 commited on
Commit
eede7ab
Β·
verified Β·
1 Parent(s): 95488fa

Card: resume check stated with both computations

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -117,7 +117,7 @@ any shared 13-gram (none β‰₯ 50 %).
117
  | Schedule | warmup 150, constant lr 6e-4 through step 2289, 1 βˆ’ sqrt decay over steps 2290–2860 to lr / 10 (6.05e-5 at step 2860) |
118
  | Optimizer | Muon (lr 0.02) for hidden weight matrices + AdamW (lr 6e-4, weight decay 0.1) for the rest |
119
  | Final training loss | 2.5067 (step 2860); mean of the values logged at steps 2770–2860: 2.5083 (1.00, steps 1340–1430: 2.6990) |
120
- | Resume check | mean training loss over steps 1060–1140 equal to the 1.00 run within 0.0001 nats |
121
  | Checkpoints | every 150 steps + final; last before the decay: step 2250 |
122
 
123
  ## Limitations
 
117
  | Schedule | warmup 150, constant lr 6e-4 through step 2289, 1 βˆ’ sqrt decay over steps 2290–2860 to lr / 10 (6.05e-5 at step 2860) |
118
  | Optimizer | Muon (lr 0.02) for hidden weight matrices + AdamW (lr 6e-4, weight decay 0.1) for the rest |
119
  | Final training loss | 2.5067 (step 2860); mean of the values logged at steps 2770–2860: 2.5083 (1.00, steps 1340–1430: 2.6990) |
120
+ | Resume check | training loss after the resume vs. the 1.00 run at the same steps (same batches, same lr): mean difference about 1e-4 nats (two independent computations: βˆ’0.00008 over steps 1060–1140, βˆ’0.00013 over 1060–1130), largest single-step difference 0.0007 |
121
  | Checkpoints | every 150 steps + final; last before the decay: step 2250 |
122
 
123
  ## Limitations