| [19:30:15] trial=exp274_syzygy_finetune arch=squares64 device=cuda vocab=1968 |
| [19:30:15] train bs=536 accum=1 grad_ckpt=False compile=True elo_every=0 save_every=100 |
| [19:30:15] model=Squares64RecurrentConfig(encoder_dim=256, hidden_dim=736, num_heads=8, ffn_ratio=4, dropout=0.05, use_swiglu=True, use_qk_norm=True, zero_init_out_proj=True, gradient_checkpointing=False, prefix_layers=4, recurrent_layers=7, recurrent_unrolls=3, suffix_layers=4, policy_head_dim=384, value_hidden=384, n_value_classes=3, use_history=False, use_threat_head=False) |
| [19:30:16] params=98.97M |
| [19:30:16] WEIGHTS-ONLY WARM START outputs/exp274_syzygy_finetune/init.pt from eval/SWA snapshot (optimizer/RNG re-initialized; step reset to 0) |
| [19:30:16] optimizer=PolarNorMuon (97.7M) + AdamW aux (1.3M) compile_polar=True |
| [19:30:16] force_lr muon=0.002 adam=3e-05 |
| [19:30:17] torch.compile enabled (mode=default); will fall back if inductor fails |
| [19:30:17] external eval syzygy n=99,555 /root/transform/outputs/exp274_syzygy_finetune/syzygy_eval.pt |
| [19:30:17] live soft dropped 6 internal hash dups |
| [19:30:17] soft train=398,840 val=0 blocked=0 targets empty=0 unnorm=0 external_eval |
| [19:31:19] batch probe ok bs=536 peak_vram=35.60GB compile=True |
| [19:31:19] average_recurrent_grads() divides bank .grad by unroll count. That does NOT divide optimizer updates by the same amount: Polar-NorMuon is scale-invariant after orthogonalization, and RMS-Adam (m/sqrt(v)) is also approximately scale-invariant for a stationary grad. What does scale is SGD-style updates and the pre-clip global grad norm (so clipping fires less often after averaging). Use explicit param-group LRs if different update sizes are intended. |
| [19:31:19] exposure deep/shallow odds=0.0x (mix=0.0 deep_n=0 shallow_n=398840) |
| [19:31:22] step 1/745 | loss=1.3735 | lr=2.00e-03,3.00e-05 | clip=1/25 | mix s/d/b=536/0/0 q=0 p=0 | 151 pos/s | vram=35.60GB |
| [19:31:44] step 25/745 | loss=1.2347 | lr=2.00e-03,3.00e-05 | clip=24/25 | mix s/d/b=13400/0/0 q=0 p=0 | 526 pos/s | vram=36.00GB |
| [19:32:07] step 50/745 | loss=0.9759 | lr=2.00e-03,3.00e-05 | clip=25/25 | mix s/d/b=26800/0/0 q=0 p=0 | 554 pos/s | vram=36.00GB |
| [19:32:30] step 75/745 | loss=0.8933 | lr=2.00e-03,3.00e-05 | clip=7/25 | mix s/d/b=40200/0/0 q=0 p=0 | 564 pos/s | vram=36.00GB |
| [19:32:53] step 100/745 | loss=0.8595 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=53600/0/0 q=0 p=0 | 569 pos/s | vram=36.00GB |
| [19:32:54] ckpt latest.pt (live weights) steps=100 status=mid |
| [19:32:56] val/syzygy hard_ce=1.0173 soft_ce=1.0173 soft_temp_ce=1.0173 teacher_entropy=0.0000 teacher_kl=1.0173 value_rows=0.0000 |
| [19:33:19] step 125/745 | loss=0.8448 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=67000/0/0 q=0 p=0 | 558 pos/s | vram=36.00GB |
| [19:33:42] step 150/745 | loss=0.8358 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=80400/0/0 q=0 p=0 | 562 pos/s | vram=36.00GB |
| [19:34:05] step 175/745 | loss=0.8264 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=93800/0/0 q=0 p=0 | 564 pos/s | vram=36.00GB |
| [19:34:28] step 200/745 | loss=0.8151 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=107200/0/0 q=0 p=0 | 567 pos/s | vram=36.00GB |
| [19:34:29] ckpt latest.pt (live weights) steps=200 status=mid |
| [19:34:31] val/syzygy hard_ce=0.9890 soft_ce=0.9890 soft_temp_ce=0.9890 teacher_entropy=0.0000 teacher_kl=0.9890 value_rows=0.0000 |
| [19:34:54] step 225/745 | loss=0.8017 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=120600/0/0 q=0 p=0 | 561 pos/s | vram=36.00GB |
| [19:35:17] step 250/745 | loss=0.8036 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=134000/0/0 q=0 p=0 | 563 pos/s | vram=36.00GB |
| [19:35:40] step 275/745 | loss=0.8042 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=147400/0/0 q=0 p=0 | 565 pos/s | vram=36.00GB |
| [19:36:03] step 300/745 | loss=0.8007 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=160800/0/0 q=0 p=0 | 566 pos/s | vram=36.00GB |
| [19:36:03] ckpt latest.pt (live weights) steps=300 status=mid |
| [19:36:06] val/syzygy hard_ce=0.9628 soft_ce=0.9628 soft_temp_ce=0.9628 teacher_entropy=0.0000 teacher_kl=0.9628 value_rows=0.0000 |
| [19:36:29] step 325/745 | loss=0.7735 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=174200/0/0 q=0 p=0 | 562 pos/s | vram=36.00GB |
| [19:36:52] step 350/745 | loss=0.7955 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=187600/0/0 q=0 p=0 | 564 pos/s | vram=36.00GB |
| [19:37:15] step 375/745 | loss=0.7738 | lr=2.00e-03,3.00e-05 | clip=1/25 | mix s/d/b=201000/0/0 q=0 p=0 | 565 pos/s | vram=36.00GB |
| [19:37:38] step 400/745 | loss=0.7629 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=214400/0/0 q=0 p=0 | 566 pos/s | vram=36.00GB |
| [19:37:38] ckpt latest.pt (live weights) steps=400 status=mid |
| [19:37:40] val/syzygy hard_ce=0.9678 soft_ce=0.9678 soft_temp_ce=0.9678 teacher_entropy=0.0000 teacher_kl=0.9678 value_rows=0.0000 |
| [19:38:03] step 425/745 | loss=0.7737 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=227800/0/0 q=0 p=0 | 563 pos/s | vram=36.00GB |
| [19:38:27] step 450/745 | loss=0.7732 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=241200/0/0 q=0 p=0 | 564 pos/s | vram=36.00GB |
| [19:38:50] step 475/745 | loss=0.7525 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=254600/0/0 q=0 p=0 | 565 pos/s | vram=36.00GB |
| [19:39:13] step 500/745 | loss=0.7616 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=268000/0/0 q=0 p=0 | 565 pos/s | vram=36.00GB |
| [19:39:14] ckpt step_000500.pt + latest.pt (live) steps=500 status=mid kind=full |
| [19:39:16] val/syzygy hard_ce=0.9568 soft_ce=0.9568 soft_temp_ce=0.9568 teacher_entropy=0.0000 teacher_kl=0.9568 value_rows=0.0000 |
| [19:39:39] step 525/745 | loss=0.7713 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=281400/0/0 q=0 p=0 | 562 pos/s | vram=36.00GB |
| [19:40:02] step 550/745 | loss=0.7402 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=294800/0/0 q=0 p=0 | 563 pos/s | vram=36.00GB |
| [19:40:25] step 575/745 | loss=0.7456 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=308200/0/0 q=0 p=0 | 564 pos/s | vram=36.00GB |
| [19:40:48] step 600/745 | loss=0.7438 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=321600/0/0 q=0 p=0 | 565 pos/s | vram=36.00GB |
| [19:40:49] ckpt latest.pt (live weights) steps=600 status=mid |
| [19:40:51] val/syzygy hard_ce=0.9616 soft_ce=0.9616 soft_temp_ce=0.9616 teacher_entropy=0.0000 teacher_kl=0.9616 value_rows=0.0000 |
| [19:41:14] step 625/745 | loss=0.7376 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=335000/0/0 q=0 p=0 | 563 pos/s | vram=36.00GB |
| [19:41:37] step 650/745 | loss=0.7351 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=348400/0/0 q=0 p=0 | 563 pos/s | vram=36.00GB |
| [19:42:00] step 675/745 | loss=0.7305 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=361800/0/0 q=0 p=0 | 564 pos/s | vram=36.00GB |
| [19:42:23] step 700/745 | loss=0.7277 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=375200/0/0 q=0 p=0 | 564 pos/s | vram=36.00GB |
| [19:42:24] ckpt latest.pt (live weights) steps=700 status=mid |
| [19:42:26] val/syzygy hard_ce=0.9662 soft_ce=0.9662 soft_temp_ce=0.9662 teacher_entropy=0.0000 teacher_kl=0.9662 value_rows=0.0000 |
| [19:42:50] step 725/745 | loss=0.7362 | lr=2.00e-03,3.00e-05 | clip=0/25 | mix s/d/b=388600/0/0 q=0 p=0 | 562 pos/s | vram=36.00GB |
| [19:43:09] timing wall=710s active=682s steps=745 status=trained |
| [19:43:09] done steps=745 pos_s=562.6 params=98.97M status=trained kind=full |
|
|
