Upload folder using huggingface_hub

training_logs/bench_driver.log

@@ -0,0 +1,10 @@
+=== A: baseline (full shard, act-ckpt, batch 8) ===
+2026-10-20 00:15:08,018 INFO [rank 0] step 20/30 | loss 2.3636 | lr 3.00e-05 | 233466 tok/s | 0.0h elapsed | ~0.0h remaining
+2026-10-20 00:15:49,674 INFO [rank 0] step 30/30 | loss 2.2944 | lr 4.50e-05 | 239248 tok/s | 0.0h elapsed | ~0.0h remaining
+=== B: grad_op + no act-ckpt, batch 8 ===
+2026-10-20 00:17:31,786 INFO [rank 0] step 20/30 | loss 2.3636 | lr 3.00e-05 | 313313 tok/s | 0.0h elapsed | ~0.0h remaining
+2026-10-20 00:18:02,715 INFO [rank 0] step 30/30 | loss 2.2945 | lr 4.50e-05 | 321441 tok/s | 0.0h elapsed | ~0.0h remaining
+=== C: grad_op + no act-ckpt, batch 32 ===
+2026-10-20 00:19:40,194 INFO [rank 0] step 20/30 | loss 2.3571 | lr 3.00e-05 | 336416 tok/s | 0.0h elapsed | ~0.0h remaining
+2026-10-20 00:20:08,196 INFO [rank 0] step 30/30 | loss 2.2941 | lr 4.50e-05 | 348205 tok/s | 0.0h elapsed | ~0.0h remaining
+BENCH_DONE

training_logs/bench_stage3.log

@@ -0,0 +1,10 @@
+=== A: baseline (full shard, act-ckpt, batch 8) ===
+2026-10-20 00:15:08,018 INFO [rank 0] step 20/30 | loss 2.3636 | lr 3.00e-05 | 233466 tok/s | 0.0h elapsed | ~0.0h remaining
+2026-10-20 00:15:49,674 INFO [rank 0] step 30/30 | loss 2.2944 | lr 4.50e-05 | 239248 tok/s | 0.0h elapsed | ~0.0h remaining
+=== B: grad_op + no act-ckpt, batch 8 ===
+2026-10-20 00:17:31,786 INFO [rank 0] step 20/30 | loss 2.3636 | lr 3.00e-05 | 313313 tok/s | 0.0h elapsed | ~0.0h remaining
+2026-10-20 00:18:02,715 INFO [rank 0] step 30/30 | loss 2.2945 | lr 4.50e-05 | 321441 tok/s | 0.0h elapsed | ~0.0h remaining
+=== C: grad_op + no act-ckpt, batch 32 ===
+2026-10-20 00:19:40,194 INFO [rank 0] step 20/30 | loss 2.3571 | lr 3.00e-05 | 336416 tok/s | 0.0h elapsed | ~0.0h remaining
+2026-10-20 00:20:08,196 INFO [rank 0] step 30/30 | loss 2.2941 | lr 4.50e-05 | 348205 tok/s | 0.0h elapsed | ~0.0h remaining
+BENCH_DONE

training_logs/bootstrap.log

@@ -0,0 +1,476 @@
+
+================ GATE: preflight ================
+
+2026-10-20 00:21:14 [bootstrap] host: riemann-climb  kernel: 6.8.0-137-generic
+/dev/vda1       2.0T   64G  1.9T   4% /
+160
+               total        used        free      shared  buff/cache   available
+Mem:            1259          42        1184           0          39        1217
+GPU[0]		: Card Series: 		AMD Radeon Graphics
+GPU[1]		: Card Series: 		AMD Radeon Graphics
+GPU[2]		: Card Series: 		AMD Radeon Graphics
+GPU[3]		: Card Series: 		AMD Radeon Graphics
+GPU[4]		: Card Series: 		AMD Radeon Graphics
+GPU[5]		: Card Series: 		AMD Radeon Graphics
+GPU[6]		: Card Series: 		AMD Radeon Graphics
+GPU[7]		: Card Series: 		AMD Radeon Graphics
+
+================ GATE: python environment ================
+
+torch 2.9.1+rocm6.4 | hip: 6.4.43484-123eb5128 | cuda: None
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+gpu count: 8
+  [0] AMD Radeon Graphics  256 GiB
+  [1] AMD Radeon Graphics  256 GiB
+  [2] AMD Radeon Graphics  256 GiB
+  [3] AMD Radeon Graphics  256 GiB
+  [4] AMD Radeon Graphics  256 GiB
+  [5] AMD Radeon Graphics  256 GiB
+  [6] AMD Radeon Graphics  256 GiB
+  [7] AMD Radeon Graphics  256 GiB
+transformers 4.57.6
+ntkmirror ok
+
+================ GATE: pull staging data ================
+
+   7218078 /workspace/RiemannModel/data/pretrain_corpus.txt
+     73824 /workspace/RiemannModel/data/ntk_finetune.jsonl
+   7291902 total
+/workspace/RiemannModel/checkpoints/stage1_279m:
+total 1296600
+drwxr-xr-x 2 root root       4096 Oct 19 23:14 .
+drwxr-xr-x 5 root root       4096 Oct 20 00:10 ..
+-rw-r--r-- 1 root root        506 Oct 19 23:14 config.json
+-rw-r--r-- 1 root root         73 Oct 19 23:14 generation_config.json
+-rw-r--r-- 1 root root     456318 Oct 19 23:14 merges.txt
+-rw-r--r-- 1 root root 1322866472 Oct 19 23:14 model.safetensors
+-rw-r--r-- 1 root root        137 Oct 19 23:14 special_tokens_map.json
+-rw-r--r-- 1 root root    3557680 Oct 19 23:14 tokenizer.json
+-rw-r--r-- 1 root root        528 Oct 19 23:14 tokenizer_config.json
+-rw-r--r-- 1 root root         54 Oct 19 23:14 training_state.json
+-rw-r--r-- 1 root root     798156 Oct 19 23:14 vocab.json
+
+/workspace/RiemannModel/ntk_controllers:
+total 184
+drwxr-xr-x 2 root root  4096 Oct 20 00:10 .
+drwxr-xr-x 8 root root  4096 Oct 20 00:21 ..
+-rw-r--r-- 1 root root   838 Oct 19 23:14 stage1_279m.json
+-rw-r--r-- 1 root root 84464 Oct 19 23:14 stage1_279m.pt
+-rw-r--r-- 1 root root   969 Oct 20 00:10 stage2_538m.json
+-rw-r--r-- 1 root root 85061 Oct 20 00:10 stage2_538m.pt
+
+================ GATE: stage 2 alone (throughput check) ================
+
+2026-10-20 00:21:31 [bootstrap] rung stage2_538m already complete, skipping
+2026-10-20 00:21:31 [bootstrap] stage 2 done. Inspect /workspace/RiemannModel/logs/pretrain_stage2_538m.log tok/s before continuing.
+
+================ GATE: rung stage2_538m -> stage3_1.19b (epochs=1) ================
+
+2026-10-20 00:21:31 [bootstrap] pretrain extra flags for stage3_1.19b: --sharding grad_op --no-activation-checkpointing --batch-size 32 --grad-accum 4
+2026-10-20 00:21:46,052 INFO [rank 0] Loading stage3_1.19b from /workspace/RiemannModel/checkpoints/stage3_1.19b
+2026-10-20 00:21:53,536 INFO [rank 0] Training stage3_1.19b: world=8 batch=32 accum=4 seq=1024 tokens/step=1048576, corpus=7218078 docs, ~1032 steps/epoch, target 1032 steps (1.00 epochs), 0.16B params
+2026-10-20 00:22:54,219 INFO [rank 0] step 20/1032 | loss 2.1712 | lr 3.00e-05 | 345896 tok/s | 0.0h elapsed | ~0.9h remaining
+2026-10-20 00:23:50,164 INFO [rank 0] step 40/1032 | loss 2.2865 | lr 6.00e-05 | 359789 tok/s | 0.0h elapsed | ~0.8h remaining
+2026-10-20 00:24:46,046 INFO [rank 0] step 60/1032 | loss 2.2602 | lr 9.00e-05 | 364815 tok/s | 0.0h elapsed | ~0.8h remaining
+2026-10-20 00:25:42,050 INFO [rank 0] step 80/1032 | loss 2.2436 | lr 1.20e-04 | 367182 tok/s | 0.1h elapsed | ~0.8h remaining
+2026-10-20 00:26:37,954 INFO [rank 0] step 100/1032 | loss 2.2440 | lr 1.50e-04 | 368743 tok/s | 0.1h elapsed | ~0.7h remaining
+2026-10-20 00:27:33,872 INFO [rank 0] step 120/1032 | loss 2.2451 | lr 1.80e-04 | 369776 tok/s | 0.1h elapsed | ~0.7h remaining
+2026-10-20 00:28:29,821 INFO [rank 0] step 140/1032 | loss 2.2413 | lr 2.10e-04 | 370494 tok/s | 0.1h elapsed | ~0.7h remaining
+2026-10-20 00:29:25,780 INFO [rank 0] step 160/1032 | loss 2.2577 | lr 2.40e-04 | 371021 tok/s | 0.1h elapsed | ~0.7h remaining
+2026-10-20 00:30:21,737 INFO [rank 0] step 180/1032 | loss 2.2464 | lr 2.70e-04 | 371434 tok/s | 0.1h elapsed | ~0.7h remaining
+2026-10-20 00:31:17,702 INFO [rank 0] step 200/1032 | loss 2.2439 | lr 3.00e-04 | 371761 tok/s | 0.2h elapsed | ~0.7h remaining
+2026-10-20 00:32:13,687 INFO [rank 0] step 220/1032 | loss 2.2371 | lr 3.00e-04 | 372017 tok/s | 0.2h elapsed | ~0.6h remaining
+2026-10-20 00:33:09,685 INFO [rank 0] step 240/1032 | loss 2.1983 | lr 2.99e-04 | 372224 tok/s | 0.2h elapsed | ~0.6h remaining
+2026-10-20 00:33:44,804 INFO [rank 0] Saved checkpoint at step 250 -> /workspace/RiemannModel/checkpoints/stage3_1.19b
+2026-10-20 00:34:12,719 INFO [rank 0] step 260/1032 | loss 2.2012 | lr 2.97e-04 | 368851 tok/s | 0.2h elapsed | ~0.6h remaining
+2026-10-20 00:35:08,605 INFO [rank 0] step 280/1032 | loss 2.1747 | lr 2.94e-04 | 369302 tok/s | 0.2h elapsed | ~0.6h remaining
+2026-10-20 00:36:04,551 INFO [rank 0] step 300/1032 | loss 2.1743 | lr 2.91e-04 | 369667 tok/s | 0.2h elapsed | ~0.6h remaining
+2026-10-20 00:37:00,544 INFO [rank 0] step 320/1032 | loss 2.1484 | lr 2.87e-04 | 369968 tok/s | 0.3h elapsed | ~0.6h remaining
+2026-10-20 00:37:56,514 INFO [rank 0] step 340/1032 | loss 2.1395 | lr 2.82e-04 | 370243 tok/s | 0.3h elapsed | ~0.5h remaining
+2026-10-20 00:38:52,517 INFO [rank 0] step 360/1032 | loss 2.1176 | lr 2.76e-04 | 370475 tok/s | 0.3h elapsed | ~0.5h remaining
+2026-10-20 00:39:48,506 INFO [rank 0] step 380/1032 | loss 2.1059 | lr 2.70e-04 | 370688 tok/s | 0.3h elapsed | ~0.5h remaining
+2026-10-20 00:40:44,552 INFO [rank 0] step 400/1032 | loss 2.0935 | lr 2.64e-04 | 370860 tok/s | 0.3h elapsed | ~0.5h remaining
+2026-10-20 00:41:40,561 INFO [rank 0] step 420/1032 | loss 2.0781 | lr 2.56e-04 | 371030 tok/s | 0.3h elapsed | ~0.5h remaining
+2026-10-20 00:42:36,557 INFO [rank 0] step 440/1032 | loss 2.0756 | lr 2.49e-04 | 371187 tok/s | 0.3h elapsed | ~0.5h remaining
+2026-10-20 00:43:32,537 INFO [rank 0] step 460/1032 | loss 2.0530 | lr 2.40e-04 | 371335 tok/s | 0.4h elapsed | ~0.4h remaining
+2026-10-20 00:44:28,613 INFO [rank 0] step 480/1032 | loss 2.0511 | lr 2.32e-04 | 371445 tok/s | 0.4h elapsed | ~0.4h remaining
+2026-10-20 00:45:24,602 INFO [rank 0] step 500/1032 | loss 2.0276 | lr 2.23e-04 | 371568 tok/s | 0.4h elapsed | ~0.4h remaining
+2026-10-20 00:45:31,669 INFO [rank 0] Saved checkpoint at step 500 -> /workspace/RiemannModel/checkpoints/stage3_1.19b
+2026-10-20 00:46:27,580 INFO [rank 0] step 520/1032 | loss 2.0268 | lr 2.13e-04 | 369921 tok/s | 0.4h elapsed | ~0.4h remaining
+2026-10-20 00:47:23,533 INFO [rank 0] step 540/1032 | loss 2.0125 | lr 2.04e-04 | 370099 tok/s | 0.4h elapsed | ~0.4h remaining
+2026-10-20 00:48:19,559 INFO [rank 0] step 560/1032 | loss 1.9975 | lr 1.94e-04 | 370248 tok/s | 0.4h elapsed | ~0.4h remaining
+2026-10-20 00:49:15,626 INFO [rank 0] step 580/1032 | loss 1.9867 | lr 1.84e-04 | 370378 tok/s | 0.5h elapsed | ~0.4h remaining
+2026-10-20 00:50:11,620 INFO [rank 0] step 600/1032 | loss 1.9871 | lr 1.74e-04 | 370515 tok/s | 0.5h elapsed | ~0.3h remaining
+2026-10-20 00:51:07,629 INFO [rank 0] step 620/1032 | loss 1.9603 | lr 1.63e-04 | 370641 tok/s | 0.5h elapsed | ~0.3h remaining
+2026-10-20 00:52:03,641 INFO [rank 0] step 640/1032 | loss 1.9573 | lr 1.53e-04 | 370756 tok/s | 0.5h elapsed | ~0.3h remaining
+2026-10-20 00:52:59,636 INFO [rank 0] step 660/1032 | loss 1.9553 | lr 1.43e-04 | 370870 tok/s | 0.5h elapsed | ~0.3h remaining
+2026-10-20 00:53:55,627 INFO [rank 0] step 680/1032 | loss 1.9386 | lr 1.33e-04 | 370977 tok/s | 0.5h elapsed | ~0.3h remaining
+2026-10-20 00:54:51,698 INFO [rank 0] step 700/1032 | loss 1.9342 | lr 1.23e-04 | 371063 tok/s | 0.5h elapsed | ~0.3h remaining
+2026-10-20 00:55:47,709 INFO [rank 0] step 720/1032 | loss 1.9314 | lr 1.14e-04 | 371155 tok/s | 0.6h elapsed | ~0.2h remaining
+2026-10-20 00:56:43,749 INFO [rank 0] step 740/1032 | loss 1.9295 | lr 1.05e-04 | 371238 tok/s | 0.6h elapsed | ~0.2h remaining
+2026-10-20 00:57:18,729 INFO [rank 0] Saved checkpoint at step 750 -> /workspace/RiemannModel/checkpoints/stage3_1.19b
+2026-10-20 00:57:46,686 INFO [rank 0] step 760/1032 | loss 1.9113 | lr 9.56e-05 | 370127 tok/s | 0.6h elapsed | ~0.2h remaining
+2026-10-20 00:58:42,610 INFO [rank 0] step 780/1032 | loss 1.9142 | lr 8.71e-05 | 370249 tok/s | 0.6h elapsed | ~0.2h remaining
+2026-10-20 00:59:38,592 INFO [rank 0] step 800/1032 | loss 1.9000 | lr 7.90e-05 | 370358 tok/s | 0.6h elapsed | ~0.2h remaining
+2026-10-20 01:00:34,585 INFO [rank 0] step 820/1032 | loss 1.8998 | lr 7.14e-05 | 370459 tok/s | 0.6h elapsed | ~0.2h remaining
+2026-10-20 01:01:30,655 INFO [rank 0] step 840/1032 | loss 1.8986 | lr 6.43e-05 | 370543 tok/s | 0.7h elapsed | ~0.2h remaining
+2026-10-20 01:02:26,668 INFO [rank 0] step 860/1032 | loss 1.8892 | lr 5.78e-05 | 370631 tok/s | 0.7h elapsed | ~0.1h remaining
+2026-10-20 01:03:22,690 INFO [rank 0] step 880/1032 | loss 1.8870 | lr 5.19e-05 | 370715 tok/s | 0.7h elapsed | ~0.1h remaining
+2026-10-20 01:04:18,770 INFO [rank 0] step 900/1032 | loss 1.8881 | lr 4.67e-05 | 370786 tok/s | 0.7h elapsed | ~0.1h remaining
+2026-10-20 01:05:14,920 INFO [rank 0] step 920/1032 | loss 1.8887 | lr 4.21e-05 | 370845 tok/s | 0.7h elapsed | ~0.1h remaining
+2026-10-20 01:06:10,941 INFO [rank 0] step 940/1032 | loss 1.8743 | lr 3.82e-05 | 370919 tok/s | 0.7h elapsed | ~0.1h remaining
+2026-10-20 01:07:06,967 INFO [rank 0] step 960/1032 | loss 1.8769 | lr 3.51e-05 | 370989 tok/s | 0.8h elapsed | ~0.1h remaining
+2026-10-20 01:08:03,065 INFO [rank 0] step 980/1032 | loss 1.8764 | lr 3.27e-05 | 371047 tok/s | 0.8h elapsed | ~0.0h remaining
+2026-10-20 01:08:59,090 INFO [rank 0] step 1000/1032 | loss 1.8733 | lr 3.10e-05 | 371112 tok/s | 0.8h elapsed | ~0.0h remaining
+2026-10-20 01:09:05,916 INFO [rank 0] Saved checkpoint at step 1000 -> /workspace/RiemannModel/checkpoints/stage3_1.19b
+2026-10-20 01:10:01,843 INFO [rank 0] step 1020/1032 | loss 1.8773 | lr 3.02e-05 | 370310 tok/s | 0.8h elapsed | ~0.0h remaining
+2026-10-20 01:10:42,451 INFO [rank 0] Saved checkpoint at step 1032 -> /workspace/RiemannModel/checkpoints/stage3_1.19b
+2026-10-20 01:10:42,530 INFO [rank 0] Done. stage3_1.19b at step 1032, 1.08B tokens seen, 0.81h wall clock, checkpoint at /workspace/RiemannModel/checkpoints/stage3_1.19b
+2026-10-20 01:10:54,411 INFO Loading stage3_1.19b from /workspace/RiemannModel/checkpoints/stage3_1.19b onto cuda:0
+2026-10-20 01:10:56,281 INFO NTK fit data: 19000 train / 1000 validation examples (from /workspace/RiemannModel/data/ntk_finetune.jsonl)
+2026-10-20 01:11:01,200 INFO Warm start from stage2_538m.pt: carried 4096 of 4096 gates, added 535 for grown dims, 4631 total (24 layers, hidden 1280 -> 36 layers, hidden 1600)
+2026-10-20 01:11:17,200 INFO Validation before fitting: nll 1.6293, token acc 0.6245
+2026-10-20 01:15:47,961 INFO Validation after fitting: nll 1.6179 (base 1.6293, delta -0.0114), token acc 0.6259
+2026-10-20 01:15:47,963 INFO Saved gate controller to /workspace/RiemannModel/ntk_controllers/stage3_1.19b.pt (summary at /workspace/RiemannModel/ntk_controllers/stage3_1.19b.json)
+2026-10-20 01:15:49 [bootstrap] rung stage3_1.19b finished in 54 min
+
+================ GATE: rung stage3_1.19b -> stage4_2.65b (epochs=1) ================
+
+2026-10-20 01:15:50,738 INFO Growing stage3_1.19b -> stage4_2.65b
+2026-10-20 01:16:05,746 INFO grew width 1600 -> 2560 (vocab=50257, 1269132800 params -> 2780211200 params)
+2026-10-20 01:16:09,000 INFO Saved grown checkpoint to /workspace/RiemannModel/checkpoints/stage4_2.65b (2780211200 params, ~2.78B)
+2026-10-20 01:16:09,000 INFO Next: python scripts/pretrain.py --stage stage4_2.65b --resume
+2026-10-20 01:16:10 [bootstrap] pretrain extra flags for stage4_2.65b: --sharding grad_op --no-activation-checkpointing --batch-size 16 --grad-accum 4
+2026-10-20 01:16:24,511 INFO [rank 0] Loading stage4_2.65b from /workspace/RiemannModel/checkpoints/stage4_2.65b
+2026-10-20 01:16:33,422 INFO [rank 0] Training stage4_2.65b: world=8 batch=16 accum=4 seq=2048 tokens/step=1048576, corpus=7218078 docs, ~1032 steps/epoch, target 1032 steps (1.00 epochs), 0.35B params
+2026-10-20 01:18:33,880 INFO [rank 0] step 20/1032 | loss 2.8309 | lr 3.00e-05 | 174380 tok/s | 0.0h elapsed | ~1.7h remaining
+2026-10-20 01:20:29,958 INFO [rank 0] step 40/1032 | loss 2.3154 | lr 6.00e-05 | 177489 tok/s | 0.1h elapsed | ~1.6h remaining
+2026-10-20 01:22:26,317 INFO [rank 0] step 60/1032 | loss 1.9829 | lr 9.00e-05 | 178398 tok/s | 0.1h elapsed | ~1.6h remaining
+2026-10-20 01:24:22,837 INFO [rank 0] step 80/1032 | loss 1.9206 | lr 1.20e-04 | 178791 tok/s | 0.1h elapsed | ~1.6h remaining
+2026-10-20 01:26:19,383 INFO [rank 0] step 100/1032 | loss 1.9109 | lr 1.50e-04 | 179019 tok/s | 0.2h elapsed | ~1.5h remaining
+2026-10-20 01:28:15,822 INFO [rank 0] step 120/1032 | loss 1.9083 | lr 1.80e-04 | 179200 tok/s | 0.2h elapsed | ~1.5h remaining
+2026-10-20 01:30:12,298 INFO [rank 0] step 140/1032 | loss 1.9123 | lr 2.10e-04 | 179320 tok/s | 0.2h elapsed | ~1.4h remaining
+2026-10-20 01:32:08,787 INFO [rank 0] step 160/1032 | loss 1.9199 | lr 2.40e-04 | 179409 tok/s | 0.3h elapsed | ~1.4h remaining
+2026-10-20 01:34:05,389 INFO [rank 0] step 180/1032 | loss 1.9169 | lr 2.70e-04 | 179459 tok/s | 0.3h elapsed | ~1.4h remaining
+2026-10-20 01:36:01,918 INFO [rank 0] step 200/1032 | loss 1.9244 | lr 3.00e-04 | 179509 tok/s | 0.3h elapsed | ~1.4h remaining
+2026-10-20 01:37:58,436 INFO [rank 0] step 220/1032 | loss 1.9171 | lr 3.00e-04 | 179552 tok/s | 0.4h elapsed | ~1.3h remaining
+2026-10-20 01:39:54,962 INFO [rank 0] step 240/1032 | loss 1.8900 | lr 2.99e-04 | 179588 tok/s | 0.4h elapsed | ~1.3h remaining
+2026-10-20 01:41:03,878 INFO [rank 0] Saved checkpoint at step 250 -> /workspace/RiemannModel/checkpoints/stage4_2.65b
+2026-10-20 01:42:02,031 INFO [rank 0] step 260/1032 | loss 1.8978 | lr 2.97e-04 | 178377 tok/s | 0.4h elapsed | ~1.3h remaining
+2026-10-20 01:43:58,446 INFO [rank 0] step 280/1032 | loss 1.8798 | lr 2.94e-04 | 178503 tok/s | 0.5h elapsed | ~1.2h remaining
+2026-10-20 01:45:54,978 INFO [rank 0] step 300/1032 | loss 1.8846 | lr 2.91e-04 | 178600 tok/s | 0.5h elapsed | ~1.2h remaining
+2026-10-20 01:47:51,545 INFO [rank 0] step 320/1032 | loss 1.8651 | lr 2.87e-04 | 178681 tok/s | 0.5h elapsed | ~1.2h remaining
+2026-10-20 01:49:48,143 INFO [rank 0] step 340/1032 | loss 1.8600 | lr 2.82e-04 | 178750 tok/s | 0.6h elapsed | ~1.1h remaining
+2026-10-20 01:51:44,699 INFO [rank 0] step 360/1032 | loss 1.8486 | lr 2.76e-04 | 178815 tok/s | 0.6h elapsed | ~1.1h remaining
+2026-10-20 01:53:41,243 INFO [rank 0] step 380/1032 | loss 1.8418 | lr 2.70e-04 | 178874 tok/s | 0.6h elapsed | ~1.1h remaining
+2026-10-20 01:55:37,805 INFO [rank 0] step 400/1032 | loss 1.8348 | lr 2.64e-04 | 178926 tok/s | 0.7h elapsed | ~1.0h remaining
+2026-10-20 01:57:34,332 INFO [rank 0] step 420/1032 | loss 1.8271 | lr 2.56e-04 | 178975 tok/s | 0.7h elapsed | ~1.0h remaining
+2026-10-20 01:59:30,887 INFO [rank 0] step 440/1032 | loss 1.8252 | lr 2.49e-04 | 179019 tok/s | 0.7h elapsed | ~1.0h remaining
+2026-10-20 02:01:27,461 INFO [rank 0] step 460/1032 | loss 1.8150 | lr 2.40e-04 | 179057 tok/s | 0.7h elapsed | ~0.9h remaining
+2026-10-20 02:03:24,053 INFO [rank 0] step 480/1032 | loss 1.8129 | lr 2.32e-04 | 179090 tok/s | 0.8h elapsed | ~0.9h remaining
+2026-10-20 02:05:20,620 INFO [rank 0] step 500/1032 | loss 1.7988 | lr 2.23e-04 | 179123 tok/s | 0.8h elapsed | ~0.9h remaining
+2026-10-20 02:05:31,366 INFO [rank 0] Saved checkpoint at step 500 -> /workspace/RiemannModel/checkpoints/stage4_2.65b
+2026-10-20 02:07:27,936 INFO [rank 0] step 520/1032 | loss 1.8017 | lr 2.13e-04 | 178523 tok/s | 0.8h elapsed | ~0.8h remaining
+2026-10-20 02:09:24,523 INFO [rank 0] step 540/1032 | loss 1.7935 | lr 2.04e-04 | 178572 tok/s | 0.9h elapsed | ~0.8h remaining
+2026-10-20 02:11:21,131 INFO [rank 0] step 560/1032 | loss 1.7843 | lr 1.94e-04 | 178618 tok/s | 0.9h elapsed | ~0.8h remaining
+2026-10-20 02:13:17,699 INFO [rank 0] step 580/1032 | loss 1.7782 | lr 1.84e-04 | 178662 tok/s | 0.9h elapsed | ~0.7h remaining
+2026-10-20 02:15:14,264 INFO [rank 0] step 600/1032 | loss 1.7819 | lr 1.74e-04 | 178703 tok/s | 1.0h elapsed | ~0.7h remaining
+2026-10-20 02:17:10,849 INFO [rank 0] step 620/1032 | loss 1.7621 | lr 1.63e-04 | 178741 tok/s | 1.0h elapsed | ~0.7h remaining
+2026-10-20 02:19:07,411 INFO [rank 0] step 640/1032 | loss 1.7619 | lr 1.53e-04 | 178778 tok/s | 1.0h elapsed | ~0.6h remaining
+2026-10-20 02:21:04,061 INFO [rank 0] step 660/1032 | loss 1.7629 | lr 1.43e-04 | 178808 tok/s | 1.1h elapsed | ~0.6h remaining
+2026-10-20 02:23:00,624 INFO [rank 0] step 680/1032 | loss 1.7502 | lr 1.33e-04 | 178840 tok/s | 1.1h elapsed | ~0.6h remaining
+2026-10-20 02:24:57,247 INFO [rank 0] step 700/1032 | loss 1.7490 | lr 1.23e-04 | 178868 tok/s | 1.1h elapsed | ~0.5h remaining
+2026-10-20 02:26:53,839 INFO [rank 0] step 720/1032 | loss 1.7482 | lr 1.14e-04 | 178896 tok/s | 1.2h elapsed | ~0.5h remaining
+2026-10-20 02:28:50,483 INFO [rank 0] step 740/1032 | loss 1.7484 | lr 1.05e-04 | 178920 tok/s | 1.2h elapsed | ~0.5h remaining
+2026-10-20 02:29:59,102 INFO [rank 0] Saved checkpoint at step 750 -> /workspace/RiemannModel/checkpoints/stage4_2.65b
+2026-10-20 02:30:57,443 INFO [rank 0] step 760/1032 | loss 1.7339 | lr 9.56e-05 | 178529 tok/s | 1.2h elapsed | ~0.4h remaining
+2026-10-20 02:32:54,110 INFO [rank 0] step 780/1032 | loss 1.7377 | lr 8.71e-05 | 178560 tok/s | 1.3h elapsed | ~0.4h remaining
+2026-10-20 02:34:50,732 INFO [rank 0] step 800/1032 | loss 1.7268 | lr 7.90e-05 | 178592 tok/s | 1.3h elapsed | ~0.4h remaining
+2026-10-20 02:36:47,475 INFO [rank 0] step 820/1032 | loss 1.7281 | lr 7.14e-05 | 178619 tok/s | 1.3h elapsed | ~0.3h remaining
+2026-10-20 02:38:44,173 INFO [rank 0] step 840/1032 | loss 1.7287 | lr 6.43e-05 | 178643 tok/s | 1.4h elapsed | ~0.3h remaining
+2026-10-20 02:40:40,816 INFO [rank 0] step 860/1032 | loss 1.7210 | lr 5.78e-05 | 178670 tok/s | 1.4h elapsed | ~0.3h remaining
+2026-10-20 02:42:37,462 INFO [rank 0] step 880/1032 | loss 1.7200 | lr 5.19e-05 | 178695 tok/s | 1.4h elapsed | ~0.2h remaining
+2026-10-20 02:44:34,096 INFO [rank 0] step 900/1032 | loss 1.7222 | lr 4.67e-05 | 178720 tok/s | 1.5h elapsed | ~0.2h remaining
+2026-10-20 02:46:30,759 INFO [rank 0] step 920/1032 | loss 1.7234 | lr 4.21e-05 | 178742 tok/s | 1.5h elapsed | ~0.2h remaining
+2026-10-20 02:48:27,404 INFO [rank 0] step 940/1032 | loss 1.7106 | lr 3.82e-05 | 178764 tok/s | 1.5h elapsed | ~0.1h remaining
+2026-10-20 02:50:24,060 INFO [rank 0] step 960/1032 | loss 1.7138 | lr 3.51e-05 | 178785 tok/s | 1.6h elapsed | ~0.1h remaining
+2026-10-20 02:52:20,691 INFO [rank 0] step 980/1032 | loss 1.7143 | lr 3.27e-05 | 178806 tok/s | 1.6h elapsed | ~0.1h remaining
+2026-10-20 02:54:17,305 INFO [rank 0] step 1000/1032 | loss 1.7122 | lr 3.10e-05 | 178826 tok/s | 1.6h elapsed | ~0.1h remaining
+2026-10-20 02:54:27,615 INFO [rank 0] Saved checkpoint at step 1000 -> /workspace/RiemannModel/checkpoints/stage4_2.65b
+2026-10-20 02:56:24,341 INFO [rank 0] step 1020/1032 | loss 1.7164 | lr 3.02e-05 | 178535 tok/s | 1.7h elapsed | ~0.0h remaining
+2026-10-20 02:57:44,569 INFO [rank 0] Saved checkpoint at step 1032 -> /workspace/RiemannModel/checkpoints/stage4_2.65b
+2026-10-20 02:57:44,694 INFO [rank 0] Done. stage4_2.65b at step 1032, 1.08B tokens seen, 1.69h wall clock, checkpoint at /workspace/RiemannModel/checkpoints/stage4_2.65b
+2026-10-20 02:57:56,796 INFO Loading stage4_2.65b from /workspace/RiemannModel/checkpoints/stage4_2.65b onto cuda:0
+2026-10-20 02:57:59,845 INFO NTK fit data: 19000 train / 1000 validation examples (from /workspace/RiemannModel/data/ntk_finetune.jsonl)
+2026-10-20 02:58:08,439 INFO Warm start from stage3_1.19b.pt: carried 4102 of 4631 gates, added 0 for grown dims, 4102 total (36 layers, hidden 1600 -> 32 layers, hidden 2560)
+2026-10-20 02:58:37,177 INFO Validation before fitting: nll 1.5118, token acc 0.6508
+2026-10-20 03:06:45,030 INFO Validation after fitting: nll 1.5053 (base 1.5118, delta -0.0065), token acc 0.6517
+2026-10-20 03:06:45,032 INFO Saved gate controller to /workspace/RiemannModel/ntk_controllers/stage4_2.65b.pt (summary at /workspace/RiemannModel/ntk_controllers/stage4_2.65b.json)
+2026-10-20 03:06:46 [bootstrap] rung stage4_2.65b finished in 110 min
+
+================ GATE: rung stage4_2.65b -> stage5_6.66b (epochs=1) ================
+
+2026-10-20 03:06:47,705 INFO Growing stage4_2.65b -> stage5_6.66b
+2026-10-20 03:07:24,511 INFO grew width 2560 -> 4096 (vocab=50257, 2780211200 params -> 6864257024 params)
+2026-10-20 03:07:31,862 INFO Saved grown checkpoint to /workspace/RiemannModel/checkpoints/stage5_6.66b (6864257024 params, ~6.86B)
+2026-10-20 03:07:31,862 INFO Next: python scripts/pretrain.py --stage stage5_6.66b --resume
+2026-10-20 03:07:33 [bootstrap] pretrain extra flags for stage5_6.66b: --sharding grad_op --no-activation-checkpointing --batch-size 8 --grad-accum 8
+2026-10-20 03:07:47,807 INFO [rank 0] Loading stage5_6.66b from /workspace/RiemannModel/checkpoints/stage5_6.66b
+2026-10-20 03:08:00,347 INFO [rank 0] Training stage5_6.66b: world=8 batch=8 accum=8 seq=2048 tokens/step=1048576, corpus=7218078 docs, ~1032 steps/epoch, target 1032 steps (1.00 epochs), 0.86B params
+2026-10-20 03:11:50,429 INFO [rank 0] step 20/1032 | loss 1.8763 | lr 3.00e-05 | 91263 tok/s | 0.1h elapsed | ~3.2h remaining
+2026-10-20 03:15:36,949 INFO [rank 0] step 40/1032 | loss 1.7875 | lr 6.00e-05 | 91918 tok/s | 0.1h elapsed | ~3.1h remaining
+2026-10-20 03:19:23,573 INFO [rank 0] step 60/1032 | loss 1.7555 | lr 9.00e-05 | 92124 tok/s | 0.2h elapsed | ~3.1h remaining
+2026-10-20 03:23:10,316 INFO [rank 0] step 80/1032 | loss 1.7371 | lr 1.20e-04 | 92214 tok/s | 0.3h elapsed | ~3.0h remaining
+2026-10-20 03:26:57,200 INFO [rank 0] step 100/1032 | loss 1.7389 | lr 1.50e-04 | 92259 tok/s | 0.3h elapsed | ~2.9h remaining
+2026-10-20 03:30:44,108 INFO [rank 0] step 120/1032 | loss 1.7405 | lr 1.80e-04 | 92285 tok/s | 0.4h elapsed | ~2.9h remaining
+2026-10-20 03:34:31,069 INFO [rank 0] step 140/1032 | loss 1.7441 | lr 2.10e-04 | 92302 tok/s | 0.4h elapsed | ~2.8h remaining
+2026-10-20 03:38:18,036 INFO [rank 0] step 160/1032 | loss 1.7498 | lr 2.40e-04 | 92313 tok/s | 0.5h elapsed | ~2.8h remaining
+2026-10-20 03:42:04,922 INFO [rank 0] step 180/1032 | loss 1.8745 | lr 2.70e-04 | 92327 tok/s | 0.6h elapsed | ~2.7h remaining
+2026-10-20 03:45:51,783 INFO [rank 0] step 200/1032 | loss 1.8192 | lr 3.00e-04 | 92338 tok/s | 0.6h elapsed | ~2.6h remaining
+2026-10-20 03:49:38,612 INFO [rank 0] step 220/1032 | loss 1.7556 | lr 3.00e-04 | 92349 tok/s | 0.7h elapsed | ~2.6h remaining
+2026-10-20 03:53:25,573 INFO [rank 0] step 240/1032 | loss 1.7410 | lr 2.99e-04 | 92354 tok/s | 0.8h elapsed | ~2.5h remaining
+2026-10-20 03:55:38,918 INFO [rank 0] Saved checkpoint at step 250 -> /workspace/RiemannModel/checkpoints/stage5_6.66b
+2026-10-20 03:57:32,525 INFO [rank 0] step 260/1032 | loss 1.7487 | lr 2.97e-04 | 91736 tok/s | 0.8h elapsed | ~2.5h remaining
+2026-10-20 04:01:19,418 INFO [rank 0] step 280/1032 | loss 1.7394 | lr 2.94e-04 | 91785 tok/s | 0.9h elapsed | ~2.4h remaining
+2026-10-20 04:05:06,298 INFO [rank 0] step 300/1032 | loss 1.7467 | lr 2.91e-04 | 91828 tok/s | 1.0h elapsed | ~2.3h remaining
+2026-10-20 04:08:53,206 INFO [rank 0] step 320/1032 | loss 1.7297 | lr 2.87e-04 | 91866 tok/s | 1.0h elapsed | ~2.3h remaining
+2026-10-20 04:12:40,233 INFO [rank 0] step 340/1032 | loss 1.7289 | lr 2.82e-04 | 91895 tok/s | 1.1h elapsed | ~2.2h remaining
+2026-10-20 04:16:27,133 INFO [rank 0] step 360/1032 | loss 1.7165 | lr 2.76e-04 | 91925 tok/s | 1.1h elapsed | ~2.1h remaining
+2026-10-20 04:20:14,126 INFO [rank 0] step 380/1032 | loss 1.7133 | lr 2.70e-04 | 91949 tok/s | 1.2h elapsed | ~2.1h remaining
+2026-10-20 04:24:01,114 INFO [rank 0] step 400/1032 | loss 1.7121 | lr 2.64e-04 | 91971 tok/s | 1.3h elapsed | ~2.0h remaining
+2026-10-20 04:27:48,401 INFO [rank 0] step 420/1032 | loss 1.7046 | lr 2.56e-04 | 91985 tok/s | 1.3h elapsed | ~1.9h remaining
+2026-10-20 04:31:35,479 INFO [rank 0] step 440/1032 | loss 1.7053 | lr 2.49e-04 | 92001 tok/s | 1.4h elapsed | ~1.9h remaining
+2026-10-20 04:35:22,576 INFO [rank 0] step 460/1032 | loss 1.6966 | lr 2.40e-04 | 92016 tok/s | 1.5h elapsed | ~1.8h remaining
+2026-10-20 04:39:09,469 INFO [rank 0] step 480/1032 | loss 1.6970 | lr 2.32e-04 | 92033 tok/s | 1.5h elapsed | ~1.7h remaining
+2026-10-20 04:42:56,451 INFO [rank 0] step 500/1032 | loss 1.6855 | lr 2.23e-04 | 92048 tok/s | 1.6h elapsed | ~1.7h remaining
+2026-10-20 04:43:16,690 INFO [rank 0] Saved checkpoint at step 500 -> /workspace/RiemannModel/checkpoints/stage5_6.66b
+2026-10-20 04:47:03,662 INFO [rank 0] step 520/1032 | loss 1.6906 | lr 2.13e-04 | 91747 tok/s | 1.7h elapsed | ~1.6h remaining
+2026-10-20 04:50:50,338 INFO [rank 0] step 540/1032 | loss 1.6834 | lr 2.04e-04 | 91775 tok/s | 1.7h elapsed | ~1.6h remaining
+2026-10-20 04:54:36,948 INFO [rank 0] step 560/1032 | loss 1.6769 | lr 1.94e-04 | 91802 tok/s | 1.8h elapsed | ~1.5h remaining
+2026-10-20 04:58:23,530 INFO [rank 0] step 580/1032 | loss 1.6720 | lr 1.84e-04 | 91828 tok/s | 1.8h elapsed | ~1.4h remaining
+2026-10-20 05:02:10,121 INFO [rank 0] step 600/1032 | loss 1.6772 | lr 1.74e-04 | 91852 tok/s | 1.9h elapsed | ~1.4h remaining
+2026-10-20 05:05:56,716 INFO [rank 0] step 620/1032 | loss 1.6600 | lr 1.63e-04 | 91875 tok/s | 2.0h elapsed | ~1.3h remaining
+2026-10-20 05:09:43,279 INFO [rank 0] step 640/1032 | loss 1.6608 | lr 1.53e-04 | 91896 tok/s | 2.0h elapsed | ~1.2h remaining
+2026-10-20 05:13:29,845 INFO [rank 0] step 660/1032 | loss 1.6639 | lr 1.43e-04 | 91917 tok/s | 2.1h elapsed | ~1.2h remaining
+2026-10-20 05:17:16,420 INFO [rank 0] step 680/1032 | loss 1.6535 | lr 1.33e-04 | 91935 tok/s | 2.2h elapsed | ~1.1h remaining
+2026-10-20 05:21:02,987 INFO [rank 0] step 700/1032 | loss 1.6506 | lr 1.23e-04 | 91953 tok/s | 2.2h elapsed | ~1.1h remaining
+2026-10-20 05:24:49,497 INFO [rank 0] step 720/1032 | loss 1.6547 | lr 1.14e-04 | 91971 tok/s | 2.3h elapsed | ~1.0h remaining
+2026-10-20 05:28:36,096 INFO [rank 0] step 740/1032 | loss 1.6544 | lr 1.05e-04 | 91986 tok/s | 2.3h elapsed | ~0.9h remaining
+2026-10-20 05:30:49,206 INFO [rank 0] Saved checkpoint at step 750 -> /workspace/RiemannModel/checkpoints/stage5_6.66b
+2026-10-20 05:32:42,907 INFO [rank 0] step 760/1032 | loss 1.6420 | lr 9.56e-05 | 91787 tok/s | 2.4h elapsed | ~0.9h remaining
+2026-10-20 05:36:29,941 INFO [rank 0] step 780/1032 | loss 1.6449 | lr 8.71e-05 | 91802 tok/s | 2.5h elapsed | ~0.8h remaining
+2026-10-20 05:40:16,948 INFO [rank 0] step 800/1032 | loss 1.6364 | lr 7.90e-05 | 91816 tok/s | 2.5h elapsed | ~0.7h remaining
+2026-10-20 05:44:04,017 INFO [rank 0] step 820/1032 | loss 1.6391 | lr 7.14e-05 | 91829 tok/s | 2.6h elapsed | ~0.7h remaining
+2026-10-20 05:47:51,033 INFO [rank 0] step 840/1032 | loss 1.6383 | lr 6.43e-05 | 91842 tok/s | 2.7h elapsed | ~0.6h remaining
+2026-10-20 05:51:38,074 INFO [rank 0] step 860/1032 | loss 1.6322 | lr 5.78e-05 | 91854 tok/s | 2.7h elapsed | ~0.5h remaining
+2026-10-20 05:55:25,642 INFO [rank 0] step 880/1032 | loss 1.6335 | lr 5.19e-05 | 91861 tok/s | 2.8h elapsed | ~0.5h remaining
+2026-10-20 05:59:12,629 INFO [rank 0] step 900/1032 | loss 1.6343 | lr 4.67e-05 | 91873 tok/s | 2.9h elapsed | ~0.4h remaining
+2026-10-20 06:02:59,587 INFO [rank 0] step 920/1032 | loss 1.6370 | lr 4.21e-05 | 91884 tok/s | 2.9h elapsed | ~0.4h remaining
+2026-10-20 06:06:46,622 INFO [rank 0] step 940/1032 | loss 1.6234 | lr 3.82e-05 | 91895 tok/s | 3.0h elapsed | ~0.3h remaining
+2026-10-20 06:10:33,534 INFO [rank 0] step 960/1032 | loss 1.6279 | lr 3.51e-05 | 91906 tok/s | 3.0h elapsed | ~0.2h remaining
+2026-10-20 06:14:20,507 INFO [rank 0] step 980/1032 | loss 1.6289 | lr 3.27e-05 | 91916 tok/s | 3.1h elapsed | ~0.2h remaining
+2026-10-20 06:18:07,471 INFO [rank 0] step 1000/1032 | loss 1.6276 | lr 3.10e-05 | 91925 tok/s | 3.2h elapsed | ~0.1h remaining
+2026-10-20 06:18:27,596 INFO [rank 0] Saved checkpoint at step 1000 -> /workspace/RiemannModel/checkpoints/stage5_6.66b
+2026-10-20 06:22:14,994 INFO [rank 0] step 1020/1032 | loss 1.6308 | lr 3.02e-05 | 91772 tok/s | 3.2h elapsed | ~0.0h remaining
+2026-10-20 06:24:50,821 INFO [rank 0] Saved checkpoint at step 1032 -> /workspace/RiemannModel/checkpoints/stage5_6.66b
+2026-10-20 06:24:51,260 INFO [rank 0] Done. stage5_6.66b at step 1032, 1.08B tokens seen, 3.28h wall clock, checkpoint at /workspace/RiemannModel/checkpoints/stage5_6.66b
+2026-10-20 06:25:03,172 INFO Loading stage5_6.66b from /workspace/RiemannModel/checkpoints/stage5_6.66b onto cuda:0
+2026-10-20 06:25:09,498 INFO NTK fit data: 19000 train / 1000 validation examples (from /workspace/RiemannModel/data/ntk_finetune.jsonl)
+2026-10-20 06:25:26,489 INFO Warm start from stage4_2.65b.pt: carried 4102 of 4102 gates, added 0 for grown dims, 4102 total (32 layers, hidden 2560 -> 32 layers, hidden 4096)
+2026-10-20 06:26:28,141 INFO Validation before fitting: nll 1.4514, token acc 0.6641
+2026-10-20 06:43:42,579 INFO Validation after fitting: nll 1.4470 (base 1.4514, delta -0.0044), token acc 0.6647
+2026-10-20 06:43:42,581 INFO Saved gate controller to /workspace/RiemannModel/ntk_controllers/stage5_6.66b.pt (summary at /workspace/RiemannModel/ntk_controllers/stage5_6.66b.json)
+2026-10-20 06:43:43 [bootstrap] rung stage5_6.66b finished in 216 min
+
+================ GATE: rung stage5_6.66b -> stage6_12.85b (epochs=2) ================
+
+2026-10-20 06:43:45,267 INFO Growing stage5_6.66b -> stage6_12.85b
+2026-10-20 06:44:53,067 INFO grew width 4096 -> 5120 (vocab=50257, 6864257024 params -> 13110702080 params)
+2026-10-20 06:45:07,053 INFO Saved grown checkpoint to /workspace/RiemannModel/checkpoints/stage6_12.85b (13110702080 params, ~13.11B)
+2026-10-20 06:45:07,053 INFO Next: python scripts/pretrain.py --stage stage6_12.85b --resume
+2026-10-20 06:45:08 [bootstrap] pretrain extra flags for stage6_12.85b: --sharding grad_op --no-activation-checkpointing --batch-size 4 --grad-accum 16
+2026-10-20 06:45:23,811 INFO [rank 0] Loading stage6_12.85b from /workspace/RiemannModel/checkpoints/stage6_12.85b
+2026-10-20 06:45:42,993 INFO [rank 0] Training stage6_12.85b: world=8 batch=4 accum=16 seq=2048 tokens/step=1048576, corpus=7218078 docs, ~1032 steps/epoch, target 2064 steps (2.00 epochs), 1.64B params
+2026-10-20 06:52:56,995 INFO [rank 0] step 20/2064 | loss 2.0337 | lr 3.00e-05 | 48399 tok/s | 0.1h elapsed | ~12.3h remaining
+2026-10-20 07:00:07,512 INFO [rank 0] step 40/2064 | loss 1.8370 | lr 6.00e-05 | 48556 tok/s | 0.2h elapsed | ~12.1h remaining
+2026-10-20 07:07:18,582 INFO [rank 0] step 60/2064 | loss 1.7434 | lr 9.00e-05 | 48588 tok/s | 0.4h elapsed | ~12.0h remaining
+2026-10-20 07:14:29,312 INFO [rank 0] step 80/2064 | loss 1.7276 | lr 1.20e-04 | 48613 tok/s | 0.5h elapsed | ~11.9h remaining
+2026-10-20 07:21:40,016 INFO [rank 0] step 100/2064 | loss 1.7116 | lr 1.50e-04 | 48629 tok/s | 0.6h elapsed | ~11.8h remaining
+2026-10-20 07:28:50,781 INFO [rank 0] step 120/2064 | loss 1.7295 | lr 1.80e-04 | 48638 tok/s | 0.7h elapsed | ~11.6h remaining
+2026-10-20 07:36:02,039 INFO [rank 0] step 140/2064 | loss 1.7355 | lr 2.10e-04 | 48636 tok/s | 0.8h elapsed | ~11.5h remaining
+2026-10-20 07:43:12,982 INFO [rank 0] step 160/2064 | loss 1.6994 | lr 2.40e-04 | 48640 tok/s | 1.0h elapsed | ~11.4h remaining
+2026-10-20 07:50:23,812 INFO [rank 0] step 180/2064 | loss 1.7228 | lr 2.70e-04 | 48644 tok/s | 1.1h elapsed | ~11.3h remaining
+2026-10-20 07:57:34,417 INFO [rank 0] step 200/2064 | loss 1.8852 | lr 3.00e-04 | 48650 tok/s | 1.2h elapsed | ~11.2h remaining
+2026-10-20 08:04:44,979 INFO [rank 0] step 220/2064 | loss 1.9192 | lr 3.00e-04 | 48655 tok/s | 1.3h elapsed | ~11.0h remaining
+2026-10-20 08:11:55,576 INFO [rank 0] step 240/2064 | loss 1.7140 | lr 3.00e-04 | 48659 tok/s | 1.4h elapsed | ~10.9h remaining
+2026-10-20 08:16:06,855 INFO [rank 0] Saved checkpoint at step 250 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
+2026-10-20 08:19:42,600 INFO [rank 0] step 260/2064 | loss 1.6971 | lr 2.99e-04 | 48348 tok/s | 1.6h elapsed | ~10.9h remaining
+2026-10-20 08:26:52,491 INFO [rank 0] step 280/2064 | loss 1.6805 | lr 2.99e-04 | 48379 tok/s | 1.7h elapsed | ~10.7h remaining
+2026-10-20 08:34:02,503 INFO [rank 0] step 300/2064 | loss 1.6987 | lr 2.98e-04 | 48405 tok/s | 1.8h elapsed | ~10.6h remaining
+2026-10-20 08:41:12,457 INFO [rank 0] step 320/2064 | loss 1.6803 | lr 2.97e-04 | 48428 tok/s | 1.9h elapsed | ~10.5h remaining
+2026-10-20 08:48:22,433 INFO [rank 0] step 340/2064 | loss 1.6833 | lr 2.96e-04 | 48448 tok/s | 2.0h elapsed | ~10.4h remaining
+2026-10-20 08:55:32,405 INFO [rank 0] step 360/2064 | loss 1.6622 | lr 2.95e-04 | 48466 tok/s | 2.2h elapsed | ~10.2h remaining
+2026-10-20 09:02:42,359 INFO [rank 0] step 380/2064 | loss 1.6607 | lr 2.94e-04 | 48482 tok/s | 2.3h elapsed | ~10.1h remaining
+2026-10-20 09:09:52,314 INFO [rank 0] step 400/2064 | loss 1.6562 | lr 2.92e-04 | 48497 tok/s | 2.4h elapsed | ~10.0h remaining
+2026-10-20 09:17:02,309 INFO [rank 0] step 420/2064 | loss 1.6503 | lr 2.91e-04 | 48510 tok/s | 2.5h elapsed | ~9.9h remaining
+2026-10-20 09:24:12,230 INFO [rank 0] step 440/2064 | loss 1.6571 | lr 2.89e-04 | 48522 tok/s | 2.6h elapsed | ~9.7h remaining
+2026-10-20 09:31:22,153 INFO [rank 0] step 460/2064 | loss 1.6430 | lr 2.87e-04 | 48533 tok/s | 2.8h elapsed | ~9.6h remaining
+2026-10-20 09:38:32,079 INFO [rank 0] step 480/2064 | loss 1.6455 | lr 2.85e-04 | 48543 tok/s | 2.9h elapsed | ~9.5h remaining
+2026-10-20 09:45:41,989 INFO [rank 0] step 500/2064 | loss 1.6392 | lr 2.83e-04 | 48553 tok/s | 3.0h elapsed | ~9.4h remaining
+2026-10-20 09:46:18,041 INFO [rank 0] Saved checkpoint at step 500 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
+2026-10-20 09:53:28,754 INFO [rank 0] step 520/2064 | loss 1.6442 | lr 2.81e-04 | 48403 tok/s | 3.1h elapsed | ~9.3h remaining
+2026-10-20 10:00:38,640 INFO [rank 0] step 540/2064 | loss 1.6359 | lr 2.79e-04 | 48417 tok/s | 3.2h elapsed | ~9.2h remaining
+2026-10-20 10:07:48,531 INFO [rank 0] step 560/2064 | loss 1.6306 | lr 2.76e-04 | 48430 tok/s | 3.4h elapsed | ~9.0h remaining
+2026-10-20 10:14:58,386 INFO [rank 0] step 580/2064 | loss 1.6298 | lr 2.73e-04 | 48442 tok/s | 3.5h elapsed | ~8.9h remaining
+2026-10-20 10:22:08,202 INFO [rank 0] step 600/2064 | loss 1.6348 | lr 2.71e-04 | 48454 tok/s | 3.6h elapsed | ~8.8h remaining
+2026-10-20 10:29:18,066 INFO [rank 0] step 620/2064 | loss 1.6209 | lr 2.68e-04 | 48464 tok/s | 3.7h elapsed | ~8.7h remaining
+2026-10-20 10:36:27,903 INFO [rank 0] step 640/2064 | loss 1.6229 | lr 2.65e-04 | 48474 tok/s | 3.8h elapsed | ~8.6h remaining
+2026-10-20 10:43:37,790 INFO [rank 0] step 660/2064 | loss 1.6260 | lr 2.62e-04 | 48484 tok/s | 4.0h elapsed | ~8.4h remaining
+2026-10-20 10:50:47,659 INFO [rank 0] step 680/2064 | loss 1.6188 | lr 2.58e-04 | 48493 tok/s | 4.1h elapsed | ~8.3h remaining
+2026-10-20 10:57:57,566 INFO [rank 0] step 700/2064 | loss 1.6183 | lr 2.55e-04 | 48501 tok/s | 4.2h elapsed | ~8.2h remaining
+2026-10-20 11:05:07,432 INFO [rank 0] step 720/2064 | loss 1.6237 | lr 2.52e-04 | 48509 tok/s | 4.3h elapsed | ~8.1h remaining
+2026-10-20 11:12:17,403 INFO [rank 0] step 740/2064 | loss 1.6255 | lr 2.48e-04 | 48516 tok/s | 4.4h elapsed | ~7.9h remaining
+2026-10-20 11:16:27,999 INFO [rank 0] Saved checkpoint at step 750 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
+2026-10-20 11:20:03,710 INFO [rank 0] step 760/2064 | loss 1.6134 | lr 2.44e-04 | 48415 tok/s | 4.6h elapsed | ~7.8h remaining
+2026-10-20 11:27:13,649 INFO [rank 0] step 780/2064 | loss 1.6184 | lr 2.41e-04 | 48425 tok/s | 4.7h elapsed | ~7.7h remaining
+2026-10-20 11:34:23,560 INFO [rank 0] step 800/2064 | loss 1.6114 | lr 2.37e-04 | 48433 tok/s | 4.8h elapsed | ~7.6h remaining
+2026-10-20 11:41:33,419 INFO [rank 0] step 820/2064 | loss 1.6148 | lr 2.33e-04 | 48442 tok/s | 4.9h elapsed | ~7.5h remaining
+2026-10-20 11:48:43,360 INFO [rank 0] step 840/2064 | loss 1.6149 | lr 2.29e-04 | 48450 tok/s | 5.0h elapsed | ~7.4h remaining
+2026-10-20 11:55:53,243 INFO [rank 0] step 860/2064 | loss 1.6090 | lr 2.25e-04 | 48458 tok/s | 5.2h elapsed | ~7.2h remaining
+2026-10-20 12:03:03,071 INFO [rank 0] step 880/2064 | loss 1.6103 | lr 2.21e-04 | 48465 tok/s | 5.3h elapsed | ~7.1h remaining
+2026-10-20 12:10:12,911 INFO [rank 0] step 900/2064 | loss 1.6113 | lr 2.17e-04 | 48472 tok/s | 5.4h elapsed | ~7.0h remaining
+2026-10-20 12:17:22,778 INFO [rank 0] step 920/2064 | loss 1.6132 | lr 2.12e-04 | 48479 tok/s | 5.5h elapsed | ~6.9h remaining
+2026-10-20 12:24:32,513 INFO [rank 0] step 940/2064 | loss 1.6001 | lr 2.08e-04 | 48486 tok/s | 5.6h elapsed | ~6.8h remaining
+2026-10-20 12:31:42,315 INFO [rank 0] step 960/2064 | loss 1.6044 | lr 2.04e-04 | 48492 tok/s | 5.8h elapsed | ~6.6h remaining
+2026-10-20 12:38:52,214 INFO [rank 0] step 980/2064 | loss 1.6050 | lr 1.99e-04 | 48498 tok/s | 5.9h elapsed | ~6.5h remaining
+2026-10-20 12:46:02,061 INFO [rank 0] step 1000/2064 | loss 1.6006 | lr 1.95e-04 | 48504 tok/s | 6.0h elapsed | ~6.4h remaining
+2026-10-20 12:46:39,533 INFO [rank 0] Saved checkpoint at step 1000 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
+2026-10-20 12:53:50,088 INFO [rank 0] step 1020/2064 | loss 1.6042 | lr 1.91e-04 | 48426 tok/s | 6.1h elapsed | ~6.3h remaining
+2026-10-20 13:00:59,373 INFO [rank 0] step 1040/2064 | loss 2.9027 | lr 1.86e-04 | 48434 tok/s | 6.3h elapsed | ~6.2h remaining
+2026-10-20 13:08:09,075 INFO [rank 0] step 1060/2064 | loss 2.0565 | lr 1.82e-04 | 48441 tok/s | 6.4h elapsed | ~6.0h remaining
+2026-10-20 13:15:19,026 INFO [rank 0] step 1080/2064 | loss 1.7069 | lr 1.77e-04 | 48447 tok/s | 6.5h elapsed | ~5.9h remaining
+2026-10-20 13:22:28,935 INFO [rank 0] step 1100/2064 | loss 1.5320 | lr 1.73e-04 | 48453 tok/s | 6.6h elapsed | ~5.8h remaining
+2026-10-20 13:29:38,958 INFO [rank 0] step 1120/2064 | loss 1.6455 | lr 1.68e-04 | 48458 tok/s | 6.7h elapsed | ~5.7h remaining
+2026-10-20 13:36:48,911 INFO [rank 0] step 1140/2064 | loss 1.9496 | lr 1.63e-04 | 48464 tok/s | 6.9h elapsed | ~5.6h remaining
+2026-10-20 13:43:59,087 INFO [rank 0] step 1160/2064 | loss 1.7489 | lr 1.59e-04 | 48469 tok/s | 7.0h elapsed | ~5.4h remaining
+2026-10-20 13:51:09,376 INFO [rank 0] step 1180/2064 | loss 1.0277 | lr 1.54e-04 | 48474 tok/s | 7.1h elapsed | ~5.3h remaining
+2026-10-20 13:58:19,757 INFO [rank 0] step 1200/2064 | loss 0.5920 | lr 1.50e-04 | 48478 tok/s | 7.2h elapsed | ~5.2h remaining
+2026-10-20 14:05:30,104 INFO [rank 0] step 1220/2064 | loss 0.3200 | lr 1.45e-04 | 48482 tok/s | 7.3h elapsed | ~5.1h remaining
+2026-10-20 14:12:40,598 INFO [rank 0] step 1240/2064 | loss 0.1636 | lr 1.41e-04 | 48486 tok/s | 7.4h elapsed | ~5.0h remaining
+2026-10-20 14:16:51,776 INFO [rank 0] Saved checkpoint at step 1250 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
+2026-10-20 14:20:27,885 INFO [rank 0] step 1260/2064 | loss 0.1215 | lr 1.36e-04 | 48424 tok/s | 7.6h elapsed | ~4.8h remaining
+2026-10-20 14:27:39,282 INFO [rank 0] step 1280/2064 | loss 0.1280 | lr 1.32e-04 | 48427 tok/s | 7.7h elapsed | ~4.7h remaining
+2026-10-20 14:34:50,839 INFO [rank 0] step 1300/2064 | loss 0.3189 | lr 1.28e-04 | 48429 tok/s | 7.8h elapsed | ~4.6h remaining
+2026-10-20 14:42:02,393 INFO [rank 0] step 1320/2064 | loss 0.7960 | lr 1.23e-04 | 48432 tok/s | 7.9h elapsed | ~4.5h remaining
+2026-10-20 14:49:13,890 INFO [rank 0] step 1340/2064 | loss 0.6287 | lr 1.19e-04 | 48434 tok/s | 8.1h elapsed | ~4.4h remaining
+2026-10-20 14:56:25,453 INFO [rank 0] step 1360/2064 | loss 0.4928 | lr 1.15e-04 | 48437 tok/s | 8.2h elapsed | ~4.2h remaining
+2026-10-20 15:03:37,045 INFO [rank 0] step 1380/2064 | loss 0.3535 | lr 1.10e-04 | 48439 tok/s | 8.3h elapsed | ~4.1h remaining
+2026-10-20 15:10:48,605 INFO [rank 0] step 1400/2064 | loss 0.2982 | lr 1.06e-04 | 48441 tok/s | 8.4h elapsed | ~4.0h remaining
+2026-10-20 15:18:00,146 INFO [rank 0] step 1420/2064 | loss 0.1965 | lr 1.02e-04 | 48443 tok/s | 8.5h elapsed | ~3.9h remaining
+2026-10-20 15:25:11,651 INFO [rank 0] step 1440/2064 | loss 0.1961 | lr 9.82e-05 | 48445 tok/s | 8.7h elapsed | ~3.8h remaining
+2026-10-20 15:32:23,124 INFO [rank 0] step 1460/2064 | loss 0.1520 | lr 9.43e-05 | 48448 tok/s | 8.8h elapsed | ~3.6h remaining
+2026-10-20 15:39:34,733 INFO [rank 0] step 1480/2064 | loss 0.1754 | lr 9.05e-05 | 48450 tok/s | 8.9h elapsed | ~3.5h remaining
+2026-10-20 15:46:46,436 INFO [rank 0] step 1500/2064 | loss 0.1258 | lr 8.67e-05 | 48451 tok/s | 9.0h elapsed | ~3.4h remaining
+2026-10-20 15:47:22,216 INFO [rank 0] Saved checkpoint at step 1500 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
+2026-10-20 15:54:33,724 INFO [rank 0] step 1520/2064 | loss 0.1315 | lr 8.31e-05 | 48401 tok/s | 9.1h elapsed | ~3.3h remaining
+2026-10-20 16:01:44,312 INFO [rank 0] step 1540/2064 | loss 0.1572 | lr 7.95e-05 | 48404 tok/s | 9.3h elapsed | ~3.2h remaining
+2026-10-20 16:08:54,953 INFO [rank 0] step 1560/2064 | loss 0.1346 | lr 7.60e-05 | 48408 tok/s | 9.4h elapsed | ~3.0h remaining
+2026-10-20 16:16:05,501 INFO [rank 0] step 1580/2064 | loss 0.1335 | lr 7.26e-05 | 48412 tok/s | 9.5h elapsed | ~2.9h remaining
+2026-10-20 16:23:16,059 INFO [rank 0] step 1600/2064 | loss 0.1271 | lr 6.94e-05 | 48416 tok/s | 9.6h elapsed | ~2.8h remaining
+2026-10-20 16:30:26,563 INFO [rank 0] step 1620/2064 | loss 0.1318 | lr 6.62e-05 | 48419 tok/s | 9.7h elapsed | ~2.7h remaining
+2026-10-20 16:37:37,117 INFO [rank 0] step 1640/2064 | loss 0.1584 | lr 6.32e-05 | 48423 tok/s | 9.9h elapsed | ~2.6h remaining
+2026-10-20 16:44:47,716 INFO [rank 0] step 1660/2064 | loss 0.1393 | lr 6.02e-05 | 48426 tok/s | 10.0h elapsed | ~2.4h remaining
+2026-10-20 16:51:58,296 INFO [rank 0] step 1680/2064 | loss 0.1321 | lr 5.74e-05 | 48430 tok/s | 10.1h elapsed | ~2.3h remaining
+2026-10-20 16:59:08,733 INFO [rank 0] step 1700/2064 | loss 0.1417 | lr 5.47e-05 | 48433 tok/s | 10.2h elapsed | ~2.2h remaining
+2026-10-20 17:06:19,228 INFO [rank 0] step 1720/2064 | loss 0.1593 | lr 5.22e-05 | 48436 tok/s | 10.3h elapsed | ~2.1h remaining
+2026-10-20 17:13:29,600 INFO [rank 0] step 1740/2064 | loss 0.3091 | lr 4.98e-05 | 48440 tok/s | 10.5h elapsed | ~1.9h remaining
+2026-10-20 17:17:40,204 INFO [rank 0] Saved checkpoint at step 1750 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
+2026-10-20 17:21:16,399 INFO [rank 0] step 1760/2064 | loss 1.4265 | lr 4.74e-05 | 48397 tok/s | 10.6h elapsed | ~1.8h remaining
+2026-10-20 17:28:27,952 INFO [rank 0] step 1780/2064 | loss 1.6975 | lr 4.53e-05 | 48399 tok/s | 10.7h elapsed | ~1.7h remaining
+2026-10-20 17:35:39,736 INFO [rank 0] step 1800/2064 | loss 1.6714 | lr 4.32e-05 | 48401 tok/s | 10.8h elapsed | ~1.6h remaining
+2026-10-20 17:42:50,874 INFO [rank 0] step 1820/2064 | loss 1.6400 | lr 4.13e-05 | 48403 tok/s | 11.0h elapsed | ~1.5h remaining
+2026-10-20 17:50:01,531 INFO [rank 0] step 1840/2064 | loss 1.6357 | lr 3.96e-05 | 48407 tok/s | 11.1h elapsed | ~1.3h remaining
+2026-10-20 17:57:12,130 INFO [rank 0] step 1860/2064 | loss 1.6244 | lr 3.80e-05 | 48410 tok/s | 11.2h elapsed | ~1.2h remaining
+2026-10-20 18:04:22,788 INFO [rank 0] step 1880/2064 | loss 1.6206 | lr 3.65e-05 | 48413 tok/s | 11.3h elapsed | ~1.1h remaining
+2026-10-20 18:11:33,436 INFO [rank 0] step 1900/2064 | loss 1.6115 | lr 3.52e-05 | 48416 tok/s | 11.4h elapsed | ~1.0h remaining
+2026-10-20 18:18:44,099 INFO [rank 0] step 1920/2064 | loss 1.6138 | lr 3.40e-05 | 48419 tok/s | 11.6h elapsed | ~0.9h remaining
+2026-10-20 18:25:54,843 INFO [rank 0] step 1940/2064 | loss 1.6075 | lr 3.30e-05 | 48421 tok/s | 11.7h elapsed | ~0.7h remaining
+2026-10-20 18:33:05,293 INFO [rank 0] step 1960/2064 | loss 1.6063 | lr 3.21e-05 | 48424 tok/s | 11.8h elapsed | ~0.6h remaining
+2026-10-20 18:40:15,500 INFO [rank 0] step 1980/2064 | loss 1.6019 | lr 3.14e-05 | 48428 tok/s | 11.9h elapsed | ~0.5h remaining
+2026-10-20 18:47:25,587 INFO [rank 0] step 2000/2064 | loss 1.5923 | lr 3.08e-05 | 48431 tok/s | 12.0h elapsed | ~0.4h remaining
+2026-10-20 18:48:01,534 INFO [rank 0] Saved checkpoint at step 2000 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
+2026-10-20 18:55:12,969 INFO [rank 0] step 2020/2064 | loss 1.5895 | lr 3.04e-05 | 48393 tok/s | 12.2h elapsed | ~0.3h remaining
+2026-10-20 19:02:23,852 INFO [rank 0] step 2040/2064 | loss 1.5956 | lr 3.01e-05 | 48396 tok/s | 12.3h elapsed | ~0.1h remaining
+2026-10-20 19:09:34,788 INFO [rank 0] step 2060/2064 | loss 1.5888 | lr 3.00e-05 | 48398 tok/s | 12.4h elapsed | ~0.0h remaining
+2026-10-20 19:11:36,487 INFO [rank 0] Saved checkpoint at step 2064 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
+2026-10-20 19:11:37,289 INFO [rank 0] Done. stage6_12.85b at step 2064, 2.16B tokens seen, 12.43h wall clock, checkpoint at /workspace/RiemannModel/checkpoints/stage6_12.85b
+2026-10-20 19:11:49,338 INFO Loading stage6_12.85b from /workspace/RiemannModel/checkpoints/stage6_12.85b onto cuda:0
+2026-10-20 19:12:02,159 INFO NTK fit data: 19000 train / 1000 validation examples (from /workspace/RiemannModel/data/ntk_finetune.jsonl)
+2026-10-20 19:12:36,467 INFO Warm start from stage5_6.66b.pt: carried 4102 of 4102 gates, added 2630 for grown dims, 6732 total (32 layers, hidden 4096 -> 40 layers, hidden 5120)
+2026-10-20 19:14:45,832 INFO Validation before fitting: nll 1.3228, token acc 0.6773
+2026-10-20 19:50:41,408 INFO Validation after fitting: nll 1.3104 (base 1.3228, delta -0.0124), token acc 0.6784
+2026-10-20 19:50:41,515 INFO Saved gate controller to /workspace/RiemannModel/ntk_controllers/stage6_12.85b.pt (summary at /workspace/RiemannModel/ntk_controllers/stage6_12.85b.json)
+2026-10-20 19:50:42 [bootstrap] rung stage6_12.85b finished in 786 min
+
+================ GATE: push weights and controllers to private model repo (no card) ================
+
+2026-10-20 19:50:42,902 INFO Logged in to Hugging Face as Mentors4EDU
+2026-10-20 19:50:42,946 INFO Uploading checkpoint folder /workspace/RiemannModel/checkpoints/stage6_12.85b -> OpenPeerAI/My-Name-is-Riemann-Prerelease
+2026-10-20 19:51:42,051 INFO Is done: exiting main loop
+2026-10-20 19:51:50,852 INFO Upload is complete!
+2026-10-20 19:51:50,853 INFO Also uploading NTK gate controller /workspace/RiemannModel/ntk_controllers/stage6_12.85b.pt
+2026-10-20 19:51:53,311 INFO Skipping README.md and LICENSE (pass --with-card to upload them).
+2026-10-20 19:51:53,311 INFO Done. https://huggingface.co/OpenPeerAI/My-Name-is-Riemann-Prerelease
+
+
+
+
  ...ontrollers/stage2_538m.pt: 100%|██████████| 85.1kB / 85.1kB            
+
+
  ...ontrollers/stage2_538m.pt: 100%|██████████| 85.1kB / 85.1kB            
+
+
+
  ...ontrollers/stage2_538m.pt: 100%|██████████| 85.1kB / 85.1kB            
+https://huggingface.co/OpenPeerAI/My-Name-is-Riemann-Prerelease/blob/main/ntk_controller/stage2_538m.pt
+https://huggingface.co/OpenPeerAI/My-Name-is-Riemann-Prerelease/blob/main/ntk_controller/stage2_538m.json
+
+
+
+
  ...ntrollers/stage3_1.19b.pt: 100%|██████████| 95.6kB / 95.6kB            
+
+
  ...ntrollers/stage3_1.19b.pt: 100%|██████████| 95.6kB / 95.6kB            
+
+
+
  ...ntrollers/stage3_1.19b.pt: 100%|██████████| 95.6kB / 95.6kB            
+https://huggingface.co/OpenPeerAI/My-Name-is-Riemann-Prerelease/blob/main/ntk_controller/stage3_1.19b.pt
+https://huggingface.co/OpenPeerAI/My-Name-is-Riemann-Prerelease/blob/main/ntk_controller/stage3_1.19b.json
+
+
+
+
  ...ntrollers/stage4_2.65b.pt: 100%|██████████| 85.1kB / 85.1kB            
+
+
  ...ntrollers/stage4_2.65b.pt: 100%|██████████| 85.1kB / 85.1kB            
+
+
+
  ...ntrollers/stage4_2.65b.pt: 100%|██████████| 85.1kB / 85.1kB            
+https://huggingface.co/OpenPeerAI/My-Name-is-Riemann-Prerelease/blob/main/ntk_controller/stage4_2.65b.pt
+https://huggingface.co/OpenPeerAI/My-Name-is-Riemann-Prerelease/blob/main/ntk_controller/stage4_2.65b.json
+
+
+
+
  ...ntrollers/stage5_6.66b.pt: 100%|██████████| 85.1kB / 85.1kB            
+
+
  ...ntrollers/stage5_6.66b.pt: 100%|██████████| 85.1kB / 85.1kB            
+
+
+
  ...ntrollers/stage5_6.66b.pt: 100%|██████████| 85.1kB / 85.1kB            
+https://huggingface.co/OpenPeerAI/My-Name-is-Riemann-Prerelease/blob/main/ntk_controller/stage5_6.66b.pt
+https://huggingface.co/OpenPeerAI/My-Name-is-Riemann-Prerelease/blob/main/ntk_controller/stage5_6.66b.json
+Start hashing 21 files.
+Finished hashing 21 files.

training_logs/bootstrap_run1.log

@@ -0,0 +1,216 @@
+
+================ GATE: preflight ================
+
+2026-10-19 23:12:29 [bootstrap] host: riemann-climb  kernel: 6.8.0-137-generic
+/dev/vda1       2.0T   28G  2.0T   2% /
+160
+               total        used        free      shared  buff/cache   available
+Mem:            1259          39        1224           0           1        1220
+GPU[0]		: Card Series: 		AMD Radeon Graphics
+GPU[1]		: Card Series: 		AMD Radeon Graphics
+GPU[2]		: Card Series: 		AMD Radeon Graphics
+GPU[3]		: Card Series: 		AMD Radeon Graphics
+GPU[4]		: Card Series: 		AMD Radeon Graphics
+GPU[5]		: Card Series: 		AMD Radeon Graphics
+GPU[6]		: Card Series: 		AMD Radeon Graphics
+GPU[7]		: Card Series: 		AMD Radeon Graphics
+
+================ GATE: python environment ================
+
+2026-10-19 23:12:29 [bootstrap] installing python3-venv/pip and creating venv
+
+Running kernel seems to be up-to-date.
+
+Restarting services...
+ systemctl restart gpuagent.service packagekit.service polkit.service rsyslog.service ssh.service udisks2.service
+
+Service restarts being deferred:
+ systemctl restart ModemManager.service
+ /etc/needrestart/restart.d/dbus.service
+ systemctl restart unattended-upgrades.service
+
+No containers need to be restarted.
+
+No user sessions are running outdated binaries.
+
+No VM guests are running outdated hypervisor (qemu) binaries on this host.
+2026-10-19 23:12:43 [bootstrap] installing torch GPU build
+torch 2.9.1+rocm6.4 | hip: 6.4.43484-123eb5128 | cuda: None
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+gpu count: 8
+  [0] AMD Radeon Graphics  256 GiB
+  [1] AMD Radeon Graphics  256 GiB
+  [2] AMD Radeon Graphics  256 GiB
+  [3] AMD Radeon Graphics  256 GiB
+  [4] AMD Radeon Graphics  256 GiB
+  [5] AMD Radeon Graphics  256 GiB
+  [6] AMD Radeon Graphics  256 GiB
+  [7] AMD Radeon Graphics  256 GiB
+transformers 4.57.6
+ntkmirror ok
+
+================ GATE: pull staging data ================
+
+Downloading '.gitattributes' to '/workspace/RiemannModel/staging/.cache/huggingface/download/wPaCkH-WbT7GsmxMKKrNZTV4nSM=.bed0738c7eeb449bca98b5d2f33c89a1ee56349a.incomplete'
+Download complete. Moving file to /workspace/RiemannModel/staging/.gitattributes
+Downloading 'MANIFEST.json' to '/workspace/RiemannModel/staging/.cache/huggingface/download/9nP3HBalecl1VgfZJOr3XrgnTQw=.24aa16ddccd0849f26df2233d8faaafa36407fa8.incomplete'
+Download complete. Moving file to /workspace/RiemannModel/staging/MANIFEST.json
+Downloading 'checkpoints/stage1_279m/config.json' to '/workspace/RiemannModel/staging/.cache/huggingface/download/checkpoints/stage1_279m/8_PA_wEVGiVa2goH2H4KQOQpvVY=.513f1ebc7fa34d7763424b4e1718fa082a57e32b.incomplete'
+Download complete. Moving file to /workspace/RiemannModel/staging/checkpoints/stage1_279m/config.json
+Downloading 'checkpoints/stage1_279m/generation_config.json' to '/workspace/RiemannModel/staging/.cache/huggingface/download/checkpoints/stage1_279m/3EVKVggOldJcKSsGjSdoUCN1AyQ=.a47e510cde63f756e090d182318bc3364207f5aa.incomplete'
+Download complete. Moving file to /workspace/RiemannModel/staging/checkpoints/stage1_279m/generation_config.json
+Downloading 'checkpoints/stage1_279m/merges.txt' to '/workspace/RiemannModel/staging/.cache/huggingface/download/checkpoints/stage1_279m/PtHk0z_I45atnj23IIRhTExwT3w=.226b0752cac7789c48f0cb3ec53eda48b7be36cc.incomplete'
+Download complete. Moving file to /workspace/RiemannModel/staging/checkpoints/stage1_279m/merges.txt
+Downloading 'checkpoints/stage1_279m/model.safetensors' to '/workspace/RiemannModel/staging/.cache/huggingface/download/checkpoints/stage1_279m/xGOKKLRSlIhH692hSVvI1-gpoa8=.1ec3b9f327920029db95bcc2101b3c21fe496aca8187c05ef63e878558a0390a.incomplete'
+Download complete. Moving file to /workspace/RiemannModel/staging/checkpoints/stage1_279m/model.safetensors
+Downloading 'checkpoints/stage1_279m/special_tokens_map.json' to '/workspace/RiemannModel/staging/.cache/huggingface/download/checkpoints/stage1_279m/ahkChHUJFxEmOdq5GDFEmerRzCY=.4d7c285ec06eacb5ad552de2e21d142a5d540337.incomplete'
+Download complete. Moving file to /workspace/RiemannModel/staging/checkpoints/stage1_279m/special_tokens_map.json
+Downloading 'checkpoints/stage1_279m/tokenizer.json' to '/workspace/RiemannModel/staging/.cache/huggingface/download/checkpoints/stage1_279m/HgM_lKo9sdSCfRtVg7MMFS7EKqo=.d469fce8cf3cdfde529baad7b2528afa4205aabf.incomplete'
+Download complete. Moving file to /workspace/RiemannModel/staging/checkpoints/stage1_279m/tokenizer.json
+Downloading 'checkpoints/stage1_279m/tokenizer_config.json' to '/workspace/RiemannModel/staging/.cache/huggingface/download/checkpoints/stage1_279m/vzaExXFZNBay89bvlQv-ZcI6BTg=.0b0d29ffc54f77d7d9f895038a82b7ee9e9b3f17.incomplete'
+Download complete. Moving file to /workspace/RiemannModel/staging/checkpoints/stage1_279m/tokenizer_config.json
+Downloading 'checkpoints/stage1_279m/training_state.json' to '/workspace/RiemannModel/staging/.cache/huggingface/download/checkpoints/stage1_279m/dPOE0ppKMT-dNI80zwDSDvyw63s=.6e026a0451c97492451dd26382e8256e7775b51e.incomplete'
+Download complete. Moving file to /workspace/RiemannModel/staging/checkpoints/stage1_279m/training_state.json
+Downloading 'checkpoints/stage1_279m/vocab.json' to '/workspace/RiemannModel/staging/.cache/huggingface/download/checkpoints/stage1_279m/j3m-Hy6QvBddw8RXA1uSWl1AJ0c=.84ef7fb594b5c0979e48bdeddb60a0adef33df0b.incomplete'
+Download complete. Moving file to /workspace/RiemannModel/staging/checkpoints/stage1_279m/vocab.json
+Downloading 'ntk_controllers/stage1_279m.json' to '/workspace/RiemannModel/staging/.cache/huggingface/download/ntk_controllers/HqPtsXacsyk4tuVI7X0OJkPJ_0g=.5c20e8428ce5805b45807c444e6c8190f94f35bf.incomplete'
+Download complete. Moving file to /workspace/RiemannModel/staging/ntk_controllers/stage1_279m.json
+Downloading 'ntk_controllers/stage1_279m.pt' to '/workspace/RiemannModel/staging/.cache/huggingface/download/ntk_controllers/tHgj5q5-AghPREg2_7bGUSpAi68=.fd039e4a5256c2063d9d11966ac59f1c989f3659bf6a6aa87e92b48896a2fea2.incomplete'
+Download complete. Moving file to /workspace/RiemannModel/staging/ntk_controllers/stage1_279m.pt
+Downloading 'ntk_finetune.jsonl.gz' to '/workspace/RiemannModel/staging/.cache/huggingface/download/1yqbWxAkZfxdmtKxNIyNvKRZZS8=.12a60746d00c33350cd4cd91fb7001f3523804397f9aa1b7c6710c6041cf63f6.incomplete'
+Download complete. Moving file to /workspace/RiemannModel/staging/ntk_finetune.jsonl.gz
+Downloading 'pretrain_corpus.txt.gz' to '/workspace/RiemannModel/staging/.cache/huggingface/download/kflhmESXn0eh2E71gze8Ger04yI=.58dd4d5d3321416a6a28c01963e96c08a0a295abef3e9407055fcdec4fea15ad.incomplete'
+Download complete. Moving file to /workspace/RiemannModel/staging/pretrain_corpus.txt.gz
+/workspace/RiemannModel/staging
+2026-10-19 23:14:08 [bootstrap] decompressing corpus
+   7218078 /workspace/RiemannModel/data/pretrain_corpus.txt
+     73824 /workspace/RiemannModel/data/ntk_finetune.jsonl
+   7291902 total
+/workspace/RiemannModel/checkpoints/stage1_279m:
+total 1296596
+drwxr-xr-x 2 root root       4096 Oct 19 23:14 .
+drwxr-xr-x 3 root root       4096 Oct 19 23:14 ..
+-rw-r--r-- 1 root root        506 Oct 19 23:14 config.json
+-rw-r--r-- 1 root root         73 Oct 19 23:14 generation_config.json
+-rw-r--r-- 1 root root     456318 Oct 19 23:14 merges.txt
+-rw-r--r-- 1 root root 1322866472 Oct 19 23:14 model.safetensors
+-rw-r--r-- 1 root root        137 Oct 19 23:14 special_tokens_map.json
+-rw-r--r-- 1 root root    3557680 Oct 19 23:14 tokenizer.json
+-rw-r--r-- 1 root root        528 Oct 19 23:14 tokenizer_config.json
+-rw-r--r-- 1 root root         54 Oct 19 23:14 training_state.json
+-rw-r--r-- 1 root root     798156 Oct 19 23:14 vocab.json
+
+/workspace/RiemannModel/ntk_controllers:
+total 96
+drwxr-xr-x 2 root root  4096 Oct 19 23:14 .
+drwxr-xr-x 8 root root  4096 Oct 19 23:14 ..
+-rw-r--r-- 1 root root   838 Oct 19 23:14 stage1_279m.json
+-rw-r--r-- 1 root root 84464 Oct 19 23:14 stage1_279m.pt
+
+================ GATE: FSDP smoke test (tiny model, 8 ranks, ~2 min) ================
+
+tiny smoke checkpoint at /workspace/RiemannModel/_smoke/checkpoints/stage2_538m
+2026-10-19 23:14:52,880 INFO [rank 0] Loading stage2_538m from /workspace/RiemannModel/_smoke/checkpoints/stage2_538m
+2026-10-19 23:14:58,608 INFO [rank 0] Training stage2_538m: world=8 batch=2 accum=2 seq=128 tokens/step=4096, corpus=7218078 docs, ~264333 steps/epoch, target 6 steps (1.00 epochs), 0.00B params
+2026-10-19 23:15:03,786 INFO [rank 0] step 2/6 | loss 10.8340 | lr 3.00e-06 | 1586 tok/s | 0.0h elapsed | ~0.0h remaining
+2026-10-19 23:15:03,843 INFO [rank 0] Saved checkpoint at step 3 -> /workspace/RiemannModel/_smoke/checkpoints/stage2_538m
+2026-10-19 23:15:03,866 INFO [rank 0] step 4/6 | loss 10.8301 | lr 6.00e-06 | 3122 tok/s | 0.0h elapsed | ~0.0h remaining
+2026-10-19 23:15:03,906 INFO [rank 0] step 6/6 | loss 10.8164 | lr 9.00e-06 | 4650 tok/s | 0.0h elapsed | ~0.0h remaining
+2026-10-19 23:15:03,940 INFO [rank 0] Saved checkpoint at step 6 -> /workspace/RiemannModel/_smoke/checkpoints/stage2_538m
+2026-10-19 23:15:03,975 INFO [rank 0] Saved checkpoint at step 6 -> /workspace/RiemannModel/_smoke/checkpoints/stage2_538m
+2026-10-19 23:15:03,975 INFO [rank 0] Done. stage2_538m at step 6, 0.00B tokens seen, 0.00h wall clock, checkpoint at /workspace/RiemannModel/_smoke/checkpoints/stage2_538m
+smoke checkpoint reloads OK
+
+================ GATE: stage 2 alone (throughput check) ================
+
+
+================ GATE: rung stage1_279m -> stage2_538m (epochs=1) ================
+
+2026-10-19 23:15:12,557 INFO Growing stage1_279m -> stage2_538m
+2026-10-19 23:15:15,960 INFO grew width 1024 -> 1280 (vocab=50257, 330708992 params -> 602229760 params)
+2026-10-19 23:15:16,855 INFO Saved grown checkpoint to /workspace/RiemannModel/checkpoints/stage2_538m (602229760 params, ~0.60B)
+2026-10-19 23:15:16,855 INFO Next: python scripts/pretrain.py --stage stage2_538m --resume
+2026-10-19 23:15:32,353 INFO [rank 0] Loading stage2_538m from /workspace/RiemannModel/checkpoints/stage2_538m
+2026-10-19 23:15:39,177 INFO [rank 0] Training stage2_538m: world=8 batch=16 accum=8 seq=1024 tokens/step=1048576, corpus=7218078 docs, ~1032 steps/epoch, target 1032 steps (1.00 epochs), 0.08B params
+2026-10-19 23:16:44,488 INFO [rank 0] step 20/1032 | loss 6.7805 | lr 3.00e-05 | 321301 tok/s | 0.0h elapsed | ~0.9h remaining
+2026-10-19 23:17:44,253 INFO [rank 0] step 40/1032 | loss 5.9708 | lr 6.00e-05 | 335454 tok/s | 0.0h elapsed | ~0.9h remaining
+2026-10-19 23:18:44,026 INFO [rank 0] step 60/1032 | loss 4.6836 | lr 9.00e-05 | 340434 tok/s | 0.1h elapsed | ~0.8h remaining
+2026-10-19 23:19:43,740 INFO [rank 0] step 80/1032 | loss 4.0124 | lr 1.20e-04 | 343061 tok/s | 0.1h elapsed | ~0.8h remaining
+2026-10-19 23:20:43,558 INFO [rank 0] step 100/1032 | loss 3.8263 | lr 1.50e-04 | 344540 tok/s | 0.1h elapsed | ~0.8h remaining
+2026-10-19 23:21:43,493 INFO [rank 0] step 120/1032 | loss 3.7315 | lr 1.80e-04 | 345423 tok/s | 0.1h elapsed | ~0.8h remaining
+2026-10-19 23:22:43,313 INFO [rank 0] step 140/1032 | loss 3.6715 | lr 2.10e-04 | 346152 tok/s | 0.1h elapsed | ~0.8h remaining
+2026-10-19 23:23:43,055 INFO [rank 0] step 160/1032 | loss 3.6275 | lr 2.40e-04 | 346754 tok/s | 0.1h elapsed | ~0.7h remaining
+2026-10-19 23:24:43,248 INFO [rank 0] step 180/1032 | loss 3.5879 | lr 2.70e-04 | 346936 tok/s | 0.2h elapsed | ~0.7h remaining
+2026-10-19 23:25:42,998 INFO [rank 0] step 200/1032 | loss 3.5558 | lr 3.00e-04 | 347337 tok/s | 0.2h elapsed | ~0.7h remaining
+2026-10-19 23:26:42,700 INFO [rank 0] step 220/1032 | loss 3.5210 | lr 3.00e-04 | 347691 tok/s | 0.2h elapsed | ~0.7h remaining
+2026-10-19 23:27:42,492 INFO [rank 0] step 240/1032 | loss 3.4818 | lr 2.99e-04 | 347944 tok/s | 0.2h elapsed | ~0.7h remaining
+2026-10-19 23:28:16,343 INFO [rank 0] Saved checkpoint at step 250 -> /workspace/RiemannModel/checkpoints/stage2_538m
+2026-10-19 23:28:46,363 INFO [rank 0] step 260/1032 | loss 3.4492 | lr 2.97e-04 | 346352 tok/s | 0.2h elapsed | ~0.6h remaining
+2026-10-19 23:29:46,357 INFO [rank 0] step 280/1032 | loss 3.3954 | lr 2.94e-04 | 346580 tok/s | 0.2h elapsed | ~0.6h remaining
+2026-10-19 23:30:46,389 INFO [rank 0] step 300/1032 | loss 3.3589 | lr 2.91e-04 | 346762 tok/s | 0.3h elapsed | ~0.6h remaining
+2026-10-19 23:31:46,682 INFO [rank 0] step 320/1032 | loss 3.2849 | lr 2.87e-04 | 346829 tok/s | 0.3h elapsed | ~0.6h remaining
+2026-10-19 23:32:47,127 INFO [rank 0] step 340/1032 | loss 3.2320 | lr 2.82e-04 | 346836 tok/s | 0.3h elapsed | ~0.6h remaining
+2026-10-19 23:33:47,192 INFO [rank 0] step 360/1032 | loss 3.1603 | lr 2.76e-04 | 346964 tok/s | 0.3h elapsed | ~0.6h remaining
+2026-10-19 23:34:47,659 INFO [rank 0] step 380/1032 | loss 3.0949 | lr 2.70e-04 | 346956 tok/s | 0.3h elapsed | ~0.5h remaining
+2026-10-19 23:35:48,297 INFO [rank 0] step 400/1032 | loss 3.0317 | lr 2.64e-04 | 346901 tok/s | 0.3h elapsed | ~0.5h remaining
+2026-10-19 23:36:48,671 INFO [rank 0] step 420/1032 | loss 2.9633 | lr 2.56e-04 | 346922 tok/s | 0.4h elapsed | ~0.5h remaining
+2026-10-19 23:37:49,237 INFO [rank 0] step 440/1032 | loss 2.8976 | lr 2.49e-04 | 346892 tok/s | 0.4h elapsed | ~0.5h remaining
+2026-10-19 23:38:50,024 INFO [rank 0] step 460/1032 | loss 2.8261 | lr 2.40e-04 | 346810 tok/s | 0.4h elapsed | ~0.5h remaining
+2026-10-19 23:39:50,600 INFO [rank 0] step 480/1032 | loss 2.7663 | lr 2.32e-04 | 346784 tok/s | 0.4h elapsed | ~0.5h remaining
+2026-10-19 23:40:51,673 INFO [rank 0] step 500/1032 | loss 2.7029 | lr 2.23e-04 | 346647 tok/s | 0.4h elapsed | ~0.4h remaining
+2026-10-19 23:40:55,314 INFO [rank 0] Saved checkpoint at step 500 -> /workspace/RiemannModel/checkpoints/stage2_538m
+2026-10-19 23:41:55,852 INFO [rank 0] step 520/1032 | loss 2.6581 | lr 2.13e-04 | 345838 tok/s | 0.4h elapsed | ~0.4h remaining
+2026-10-19 23:42:56,449 INFO [rank 0] step 540/1032 | loss 2.6058 | lr 2.04e-04 | 345847 tok/s | 0.5h elapsed | ~0.4h remaining
+2026-10-19 23:43:57,212 INFO [rank 0] step 560/1032 | loss 2.5614 | lr 1.94e-04 | 345821 tok/s | 0.5h elapsed | ~0.4h remaining
+2026-10-19 23:44:57,948 INFO [rank 0] step 580/1032 | loss 2.5165 | lr 1.84e-04 | 345803 tok/s | 0.5h elapsed | ~0.4h remaining
+2026-10-19 23:45:58,530 INFO [rank 0] step 600/1032 | loss 2.4934 | lr 1.74e-04 | 345815 tok/s | 0.5h elapsed | ~0.4h remaining
+2026-10-19 23:46:59,125 INFO [rank 0] step 620/1032 | loss 2.4446 | lr 1.63e-04 | 345824 tok/s | 0.5h elapsed | ~0.3h remaining
+2026-10-19 23:47:59,698 INFO [rank 0] step 640/1032 | loss 2.4215 | lr 1.53e-04 | 345836 tok/s | 0.5h elapsed | ~0.3h remaining
+2026-10-19 23:49:00,149 INFO [rank 0] step 660/1032 | loss 2.4078 | lr 1.43e-04 | 345869 tok/s | 0.6h elapsed | ~0.3h remaining
+2026-10-19 23:50:00,674 INFO [rank 0] step 680/1032 | loss 2.3760 | lr 1.33e-04 | 345887 tok/s | 0.6h elapsed | ~0.3h remaining
+2026-10-19 23:51:00,926 INFO [rank 0] step 700/1032 | loss 2.3554 | lr 1.23e-04 | 345949 tok/s | 0.6h elapsed | ~0.3h remaining
+2026-10-19 23:52:01,371 INFO [rank 0] step 720/1032 | loss 2.3478 | lr 1.14e-04 | 345977 tok/s | 0.6h elapsed | ~0.3h remaining
+2026-10-19 23:53:01,732 INFO [rank 0] step 740/1032 | loss 2.3374 | lr 1.05e-04 | 346016 tok/s | 0.6h elapsed | ~0.2h remaining
+2026-10-19 23:53:35,348 INFO [rank 0] Saved checkpoint at step 750 -> /workspace/RiemannModel/checkpoints/stage2_538m
+2026-10-19 23:54:05,438 INFO [rank 0] step 760/1032 | loss 2.3081 | lr 9.56e-05 | 345552 tok/s | 0.6h elapsed | ~0.2h remaining
+2026-10-19 23:55:05,741 INFO [rank 0] step 780/1032 | loss 2.3049 | lr 8.71e-05 | 345608 tok/s | 0.7h elapsed | ~0.2h remaining
+2026-10-19 23:56:06,151 INFO [rank 0] step 800/1032 | loss 2.2835 | lr 7.90e-05 | 345647 tok/s | 0.7h elapsed | ~0.2h remaining
+2026-10-19 23:57:06,671 INFO [rank 0] step 820/1032 | loss 2.2798 | lr 7.14e-05 | 345668 tok/s | 0.7h elapsed | ~0.2h remaining
+2026-10-19 23:58:07,142 INFO [rank 0] step 840/1032 | loss 2.2702 | lr 6.43e-05 | 345695 tok/s | 0.7h elapsed | ~0.2h remaining
+2026-10-19 23:59:07,781 INFO [rank 0] step 860/1032 | loss 2.2573 | lr 5.78e-05 | 345698 tok/s | 0.7h elapsed | ~0.1h remaining
+2026-10-20 00:00:08,262 INFO [rank 0] step 880/1032 | loss 2.2535 | lr 5.19e-05 | 345722 tok/s | 0.7h elapsed | ~0.1h remaining
+2026-10-20 00:01:08,844 INFO [rank 0] step 900/1032 | loss 2.2496 | lr 4.67e-05 | 345732 tok/s | 0.8h elapsed | ~0.1h remaining
+2026-10-20 00:02:09,402 INFO [rank 0] step 920/1032 | loss 2.2503 | lr 4.21e-05 | 345744 tok/s | 0.8h elapsed | ~0.1h remaining
+2026-10-20 00:03:09,745 INFO [rank 0] step 940/1032 | loss 2.2272 | lr 3.82e-05 | 345782 tok/s | 0.8h elapsed | ~0.1h remaining
+2026-10-20 00:04:09,963 INFO [rank 0] step 960/1032 | loss 2.2299 | lr 3.51e-05 | 345833 tok/s | 0.8h elapsed | ~0.1h remaining
+2026-10-20 00:05:10,276 INFO [rank 0] step 980/1032 | loss 2.2276 | lr 3.27e-05 | 345872 tok/s | 0.8h elapsed | ~0.0h remaining
+2026-10-20 00:06:10,633 INFO [rank 0] step 1000/1032 | loss 2.2226 | lr 3.10e-05 | 345903 tok/s | 0.8h elapsed | ~0.0h remaining
+2026-10-20 00:06:14,141 INFO [rank 0] Saved checkpoint at step 1000 -> /workspace/RiemannModel/checkpoints/stage2_538m
+2026-10-20 00:07:14,407 INFO [rank 0] step 1020/1032 | loss 2.2236 | lr 3.02e-05 | 345552 tok/s | 0.9h elapsed | ~0.0h remaining
+2026-10-20 00:07:54,004 INFO [rank 0] Saved checkpoint at step 1032 -> /workspace/RiemannModel/checkpoints/stage2_538m
+2026-10-20 00:07:54,061 INFO [rank 0] Done. stage2_538m at step 1032, 1.08B tokens seen, 0.87h wall clock, checkpoint at /workspace/RiemannModel/checkpoints/stage2_538m
+2026-10-20 00:08:06,042 INFO Loading stage2_538m from /workspace/RiemannModel/checkpoints/stage2_538m onto cuda:0
+2026-10-20 00:08:07,240 INFO NTK fit data: 19000 train / 1000 validation examples (from /workspace/RiemannModel/data/ntk_finetune.jsonl)
+2026-10-20 00:08:10,110 INFO Warm start from stage1_279m.pt: carried 4096 of 4096 gates, added 0 for grown dims, 4096 total (18 layers, hidden 1024 -> 24 layers, hidden 1280)
+2026-10-20 00:08:18,150 INFO Validation before fitting: nll 1.9682, token acc 0.5455
+2026-10-20 00:10:33,592 INFO Validation after fitting: nll 1.9540 (base 1.9682, delta -0.0142), token acc 0.5473
+2026-10-20 00:10:33,595 INFO Saved gate controller to /workspace/RiemannModel/ntk_controllers/stage2_538m.pt (summary at /workspace/RiemannModel/ntk_controllers/stage2_538m.json)
+2026-10-20 00:10:34 [bootstrap] rung stage2_538m finished in 55 min
+2026-10-20 00:10:34 [bootstrap] stage 2 done. Inspect /workspace/RiemannModel/logs/pretrain_stage2_538m.log tok/s before continuing.
+
+================ GATE: rung stage2_538m -> stage3_1.19b (epochs=1) ================
+
+2026-10-20 00:10:36,296 INFO Growing stage2_538m -> stage3_1.19b
+2026-10-20 00:10:43,018 INFO grew width 1280 -> 1600 (vocab=50257, 602229760 params -> 1269132800 params)
+2026-10-20 00:10:44,698 INFO Saved grown checkpoint to /workspace/RiemannModel/checkpoints/stage3_1.19b (1269132800 params, ~1.27B)
+2026-10-20 00:10:44,698 INFO Next: python scripts/pretrain.py --stage stage3_1.19b --resume
+2026-10-20 00:11:00,203 INFO [rank 0] Loading stage3_1.19b from /workspace/RiemannModel/checkpoints/stage3_1.19b
+2026-10-20 00:11:07,657 INFO [rank 0] Training stage3_1.19b: world=8 batch=8 accum=16 seq=1024 tokens/step=1048576, corpus=7218078 docs, ~1032 steps/epoch, target 1032 steps (1.00 epochs), 0.16B params
+Traceback (most recent call last):

training_logs/grow_stage2_538m.log

@@ -0,0 +1,4 @@
+2026-10-19 23:15:12,557 INFO Growing stage1_279m -> stage2_538m
+2026-10-19 23:15:15,960 INFO grew width 1024 -> 1280 (vocab=50257, 330708992 params -> 602229760 params)
+2026-10-19 23:15:16,855 INFO Saved grown checkpoint to /workspace/RiemannModel/checkpoints/stage2_538m (602229760 params, ~0.60B)
+2026-10-19 23:15:16,855 INFO Next: python scripts/pretrain.py --stage stage2_538m --resume

training_logs/grow_stage3_1.19b.log

@@ -0,0 +1,4 @@
+2026-10-20 00:10:36,296 INFO Growing stage2_538m -> stage3_1.19b
+2026-10-20 00:10:43,018 INFO grew width 1280 -> 1600 (vocab=50257, 602229760 params -> 1269132800 params)
+2026-10-20 00:10:44,698 INFO Saved grown checkpoint to /workspace/RiemannModel/checkpoints/stage3_1.19b (1269132800 params, ~1.27B)
+2026-10-20 00:10:44,698 INFO Next: python scripts/pretrain.py --stage stage3_1.19b --resume

training_logs/grow_stage4_2.65b.log

@@ -0,0 +1,6 @@
+2026-10-20 01:15:50,738 INFO Growing stage3_1.19b -> stage4_2.65b
+
+2026-10-20 01:16:05,746 INFO grew width 1600 -> 2560 (vocab=50257, 1269132800 params -> 2780211200 params)
+2026-10-20 01:16:09,000 INFO Saved grown checkpoint to /workspace/RiemannModel/checkpoints/stage4_2.65b (2780211200 params, ~2.78B)
+2026-10-20 01:16:09,000 WARNING stage4_2.65b is marked not feasible on this machine; expect this to be slow or run out of memory
+2026-10-20 01:16:09,000 INFO Next: python scripts/pretrain.py --stage stage4_2.65b --resume

training_logs/grow_stage5_6.66b.log

@@ -0,0 +1,6 @@
+2026-10-20 03:06:47,705 INFO Growing stage4_2.65b -> stage5_6.66b
+
+2026-10-20 03:07:24,511 INFO grew width 2560 -> 4096 (vocab=50257, 2780211200 params -> 6864257024 params)
+2026-10-20 03:07:31,862 INFO Saved grown checkpoint to /workspace/RiemannModel/checkpoints/stage5_6.66b (6864257024 params, ~6.86B)
+2026-10-20 03:07:31,862 WARNING stage5_6.66b is marked not feasible on this machine; expect this to be slow or run out of memory
+2026-10-20 03:07:31,862 INFO Next: python scripts/pretrain.py --stage stage5_6.66b --resume

training_logs/grow_stage6_12.85b.log

@@ -0,0 +1,6 @@
+2026-10-20 06:43:45,267 INFO Growing stage5_6.66b -> stage6_12.85b
+
+2026-10-20 06:44:53,067 INFO grew width 4096 -> 5120 (vocab=50257, 6864257024 params -> 13110702080 params)
+2026-10-20 06:45:07,053 INFO Saved grown checkpoint to /workspace/RiemannModel/checkpoints/stage6_12.85b (13110702080 params, ~13.11B)
+2026-10-20 06:45:07,053 WARNING stage6_12.85b is marked not feasible on this machine; expect this to be slow or run out of memory
+2026-10-20 06:45:07,053 INFO Next: python scripts/pretrain.py --stage stage6_12.85b --resume

training_logs/ntk_stage2_538m.log

@@ -0,0 +1,33 @@
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2026-10-20 00:08:06,042 INFO Loading stage2_538m from /workspace/RiemannModel/checkpoints/stage2_538m onto cuda:0
+2026-10-20 00:08:07,240 INFO NTK fit data: 19000 train / 1000 validation examples (from /workspace/RiemannModel/data/ntk_finetune.jsonl)
+Token indices sequence length is longer than the specified maximum sequence length for this model (1305 > 1024). Running this sequence through the model will result in indexing errors
+2026-10-20 00:08:10,110 INFO Warm start from stage1_279m.pt: carried 4096 of 4096 gates, added 0 for grown dims, 4096 total (18 layers, hidden 1024 -> 24 layers, hidden 1280)
+2026-10-20 00:08:18,150 INFO Validation before fitting: nll 1.9682, token acc 0.5455
+step    1/240: loss=1.602540 task=1.602540
+validation step   20: nll=1.966694 token_acc=0.5458
+validation step   40: nll=1.965223 token_acc=0.5460
+step   48/240: loss=1.277483 task=1.277483
+validation step   60: nll=1.963804 token_acc=0.5463
+validation step   80: nll=1.962513 token_acc=0.5464
+step   96/240: loss=1.726861 task=1.726861
+validation step  100: nll=1.961262 token_acc=0.5464
+validation step  120: nll=1.959995 token_acc=0.5465
+validation step  140: nll=1.958817 token_acc=0.5467
+step  144/240: loss=1.582556 task=1.582556
+validation step  160: nll=1.957703 token_acc=0.5468
+validation step  180: nll=1.956681 token_acc=0.5469
+step  192/240: loss=4.779078 task=4.779078
+validation step  200: nll=1.955745 token_acc=0.5470
+validation step  220: nll=1.954824 token_acc=0.5470
+step  240/240: loss=2.130554 task=2.130554
+validation step  240: nll=1.953964 token_acc=0.5473
+2026-10-20 00:10:33,592 INFO Validation after fitting: nll 1.9540 (base 1.9682, delta -0.0142), token acc 0.5473
+2026-10-20 00:10:33,595 INFO Saved gate controller to /workspace/RiemannModel/ntk_controllers/stage2_538m.pt (summary at /workspace/RiemannModel/ntk_controllers/stage2_538m.json)

training_logs/ntk_stage3_1.19b.log

@@ -0,0 +1,34 @@
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2026-10-20 01:10:54,411 INFO Loading stage3_1.19b from /workspace/RiemannModel/checkpoints/stage3_1.19b onto cuda:0
+
+2026-10-20 01:10:56,281 INFO NTK fit data: 19000 train / 1000 validation examples (from /workspace/RiemannModel/data/ntk_finetune.jsonl)
+Token indices sequence length is longer than the specified maximum sequence length for this model (1305 > 1024). Running this sequence through the model will result in indexing errors
+2026-10-20 01:11:01,200 INFO Warm start from stage2_538m.pt: carried 4096 of 4096 gates, added 535 for grown dims, 4631 total (24 layers, hidden 1280 -> 36 layers, hidden 1600)
+2026-10-20 01:11:17,200 INFO Validation before fitting: nll 1.6293, token acc 0.6245
+step    1/240: loss=1.200066 task=1.200066
+validation step   20: nll=1.628074 token_acc=0.6247
+validation step   40: nll=1.626826 token_acc=0.6250
+step   48/240: loss=0.990680 task=0.990680
+validation step   60: nll=1.625639 token_acc=0.6250
+validation step   80: nll=1.624534 token_acc=0.6250
+step   96/240: loss=1.420327 task=1.420327
+validation step  100: nll=1.623530 token_acc=0.6249
+validation step  120: nll=1.622574 token_acc=0.6250
+validation step  140: nll=1.621629 token_acc=0.6252
+step  144/240: loss=1.298729 task=1.298729
+validation step  160: nll=1.620797 token_acc=0.6252
+validation step  180: nll=1.620012 token_acc=0.6254
+step  192/240: loss=4.044214 task=4.044214
+validation step  200: nll=1.619268 token_acc=0.6256
+validation step  220: nll=1.618548 token_acc=0.6257
+step  240/240: loss=1.844396 task=1.844396
+validation step  240: nll=1.617883 token_acc=0.6259
+2026-10-20 01:15:47,961 INFO Validation after fitting: nll 1.6179 (base 1.6293, delta -0.0114), token acc 0.6259
+2026-10-20 01:15:47,963 INFO Saved gate controller to /workspace/RiemannModel/ntk_controllers/stage3_1.19b.pt (summary at /workspace/RiemannModel/ntk_controllers/stage3_1.19b.json)

training_logs/ntk_stage4_2.65b.log

@@ -0,0 +1,34 @@
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2026-10-20 02:57:56,796 INFO Loading stage4_2.65b from /workspace/RiemannModel/checkpoints/stage4_2.65b onto cuda:0
+
+2026-10-20 02:57:59,845 INFO NTK fit data: 19000 train / 1000 validation examples (from /workspace/RiemannModel/data/ntk_finetune.jsonl)
+Token indices sequence length is longer than the specified maximum sequence length for this model (1305 > 1024). Running this sequence through the model will result in indexing errors
+2026-10-20 02:58:08,439 INFO Warm start from stage3_1.19b.pt: carried 4102 of 4631 gates, added 0 for grown dims, 4102 total (36 layers, hidden 1600 -> 32 layers, hidden 2560)
+2026-10-20 02:58:37,177 INFO Validation before fitting: nll 1.5118, token acc 0.6508
+step    1/240: loss=1.074017 task=1.074017
+validation step   20: nll=1.511132 token_acc=0.6509
+validation step   40: nll=1.510413 token_acc=0.6512
+step   48/240: loss=0.905892 task=0.905892
+validation step   60: nll=1.509766 token_acc=0.6511
+validation step   80: nll=1.509152 token_acc=0.6512
+step   96/240: loss=1.314439 task=1.314439
+validation step  100: nll=1.508580 token_acc=0.6513
+validation step  120: nll=1.508032 token_acc=0.6513
+validation step  140: nll=1.507495 token_acc=0.6514
+step  144/240: loss=1.224094 task=1.224094
+validation step  160: nll=1.506989 token_acc=0.6515
+validation step  180: nll=1.506523 token_acc=0.6516
+step  192/240: loss=3.798501 task=3.798501
+validation step  200: nll=1.506091 token_acc=0.6515
+validation step  220: nll=1.505665 token_acc=0.6516
+step  240/240: loss=1.704812 task=1.704812
+validation step  240: nll=1.505260 token_acc=0.6517
+2026-10-20 03:06:45,030 INFO Validation after fitting: nll 1.5053 (base 1.5118, delta -0.0065), token acc 0.6517
+2026-10-20 03:06:45,032 INFO Saved gate controller to /workspace/RiemannModel/ntk_controllers/stage4_2.65b.pt (summary at /workspace/RiemannModel/ntk_controllers/stage4_2.65b.json)

training_logs/ntk_stage5_6.66b.log

@@ -0,0 +1,34 @@
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2026-10-20 06:25:03,172 INFO Loading stage5_6.66b from /workspace/RiemannModel/checkpoints/stage5_6.66b onto cuda:0
+
+2026-10-20 06:25:09,498 INFO NTK fit data: 19000 train / 1000 validation examples (from /workspace/RiemannModel/data/ntk_finetune.jsonl)
+Token indices sequence length is longer than the specified maximum sequence length for this model (1305 > 1024). Running this sequence through the model will result in indexing errors
+2026-10-20 06:25:26,489 INFO Warm start from stage4_2.65b.pt: carried 4102 of 4102 gates, added 0 for grown dims, 4102 total (32 layers, hidden 2560 -> 32 layers, hidden 4096)
+2026-10-20 06:26:28,141 INFO Validation before fitting: nll 1.4514, token acc 0.6641
+step    1/240: loss=1.024633 task=1.024633
+validation step   20: nll=1.450900 token_acc=0.6641
+validation step   40: nll=1.450434 token_acc=0.6642
+step   48/240: loss=0.860201 task=0.860201
+validation step   60: nll=1.450008 token_acc=0.6643
+validation step   80: nll=1.449611 token_acc=0.6643
+step   96/240: loss=1.264806 task=1.264806
+validation step  100: nll=1.449235 token_acc=0.6643
+validation step  120: nll=1.448885 token_acc=0.6644
+validation step  140: nll=1.448534 token_acc=0.6645
+step  144/240: loss=1.174451 task=1.174451
+validation step  160: nll=1.448192 token_acc=0.6645
+validation step  180: nll=1.447886 token_acc=0.6645
+step  192/240: loss=3.678861 task=3.678861
+validation step  200: nll=1.447578 token_acc=0.6646
+validation step  220: nll=1.447284 token_acc=0.6646
+step  240/240: loss=1.606066 task=1.606066
+validation step  240: nll=1.447003 token_acc=0.6647
+2026-10-20 06:43:42,579 INFO Validation after fitting: nll 1.4470 (base 1.4514, delta -0.0044), token acc 0.6647
+2026-10-20 06:43:42,581 INFO Saved gate controller to /workspace/RiemannModel/ntk_controllers/stage5_6.66b.pt (summary at /workspace/RiemannModel/ntk_controllers/stage5_6.66b.json)

training_logs/ntk_stage6_12.85b.log

@@ -0,0 +1,34 @@
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2026-10-20 19:11:49,338 INFO Loading stage6_12.85b from /workspace/RiemannModel/checkpoints/stage6_12.85b onto cuda:0
+
+2026-10-20 19:12:02,159 INFO NTK fit data: 19000 train / 1000 validation examples (from /workspace/RiemannModel/data/ntk_finetune.jsonl)
+Token indices sequence length is longer than the specified maximum sequence length for this model (1305 > 1024). Running this sequence through the model will result in indexing errors
+2026-10-20 19:12:36,467 INFO Warm start from stage5_6.66b.pt: carried 4102 of 4102 gates, added 2630 for grown dims, 6732 total (32 layers, hidden 4096 -> 40 layers, hidden 5120)
+2026-10-20 19:14:45,832 INFO Validation before fitting: nll 1.3228, token acc 0.6773
+step    1/240: loss=0.967124 task=0.967124
+validation step   20: nll=1.321372 token_acc=0.6774
+validation step   40: nll=1.320021 token_acc=0.6775
+step   48/240: loss=0.830313 task=0.830313
+validation step   60: nll=1.318690 token_acc=0.6776
+validation step   80: nll=1.317392 token_acc=0.6777
+step   96/240: loss=1.159311 task=1.159311
+validation step  100: nll=1.316291 token_acc=0.6777
+validation step  120: nll=1.315336 token_acc=0.6779
+validation step  140: nll=1.314391 token_acc=0.6780
+step  144/240: loss=1.085732 task=1.085732
+validation step  160: nll=1.313549 token_acc=0.6781
+validation step  180: nll=1.312607 token_acc=0.6782
+step  192/240: loss=2.922054 task=2.922054
+validation step  200: nll=1.311809 token_acc=0.6783
+validation step  220: nll=1.311043 token_acc=0.6784
+step  240/240: loss=1.457985 task=1.457985
+validation step  240: nll=1.310352 token_acc=0.6784
+2026-10-20 19:50:41,408 INFO Validation after fitting: nll 1.3104 (base 1.3228, delta -0.0124), token acc 0.6784
+2026-10-20 19:50:41,515 INFO Saved gate controller to /workspace/RiemannModel/ntk_controllers/stage6_12.85b.pt (summary at /workspace/RiemannModel/ntk_controllers/stage6_12.85b.json)

training_logs/pretrain_stage2_538m.log

@@ -0,0 +1,232 @@
+W1019 23:15:18.503000 12692 torch/distributed/run.py:803] 
+W1019 23:15:18.503000 12692 torch/distributed/run.py:803] *****************************************
+W1019 23:15:18.503000 12692 torch/distributed/run.py:803] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
+W1019 23:15:18.503000 12692 torch/distributed/run.py:803] *****************************************
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+`torch_dtype` is deprecated! Use `dtype` instead!
+`torch_dtype` is deprecated! Use `dtype` instead!
+`torch_dtype` is deprecated! Use `dtype` instead!
+`torch_dtype` is deprecated! Use `dtype` instead!
+2026-10-19 23:15:32,353 INFO [rank 0] Loading stage2_538m from /workspace/RiemannModel/checkpoints/stage2_538m
+`torch_dtype` is deprecated! Use `dtype` instead!
+`torch_dtype` is deprecated! Use `dtype` instead!
+`torch_dtype` is deprecated! Use `dtype` instead!
+`torch_dtype` is deprecated! Use `dtype` instead!
+
+riemann-climb:12761:12761 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:12761:12761 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+RCCL version : 2.22.3-HEAD:7d8d67c
+HIP version  : 6.4.43484-123eb5128
+ROCm version : 6.4.2.0-120-e7d83f5
+Hostname     : riemann-climb
+Librccl path : /workspace/venv/lib/python3.12/site-packages/torch/lib/librccl.so
+
+riemann-climb:12763:12763 [2] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:12764:12764 [3] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:12768:12768 [7] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:12767:12767 [6] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:12765:12765 [4] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:12766:12766 [5] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:12762:12762 [1] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+2026-10-19 23:15:39,177 INFO [rank 0] Training stage2_538m: world=8 batch=16 accum=8 seq=1024 tokens/step=1048576, corpus=7218078 docs, ~1032 steps/epoch, target 1032 steps (1.00 epochs), 0.08B params
+Token indices sequence length is longer than the specified maximum sequence length for this model (1572 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1266 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1363 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1150 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1402 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1407 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1280 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1245 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1191 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1177 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1403 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1347 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1300 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1135 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1122 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1466 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1383 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1492 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1348 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1193 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1469 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1522 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1430 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1320 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1110 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1587 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1121 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1451 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1327 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1429 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1136 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1390 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1157 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1337 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1411 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1399 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1492 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1105 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1377 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1371 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1422 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1103 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1040 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1240 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1189 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1545 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1235 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1143 > 1024). Running this sequence through the model will result in indexing errors
+2026-10-19 23:16:44,488 INFO [rank 0] step 20/1032 | loss 6.7805 | lr 3.00e-05 | 321301 tok/s | 0.0h elapsed | ~0.9h remaining
+2026-10-19 23:17:44,253 INFO [rank 0] step 40/1032 | loss 5.9708 | lr 6.00e-05 | 335454 tok/s | 0.0h elapsed | ~0.9h remaining
+2026-10-19 23:18:44,026 INFO [rank 0] step 60/1032 | loss 4.6836 | lr 9.00e-05 | 340434 tok/s | 0.1h elapsed | ~0.8h remaining
+2026-10-19 23:19:43,740 INFO [rank 0] step 80/1032 | loss 4.0124 | lr 1.20e-04 | 343061 tok/s | 0.1h elapsed | ~0.8h remaining
+2026-10-19 23:20:43,558 INFO [rank 0] step 100/1032 | loss 3.8263 | lr 1.50e-04 | 344540 tok/s | 0.1h elapsed | ~0.8h remaining
+2026-10-19 23:21:43,493 INFO [rank 0] step 120/1032 | loss 3.7315 | lr 1.80e-04 | 345423 tok/s | 0.1h elapsed | ~0.8h remaining
+2026-10-19 23:22:43,313 INFO [rank 0] step 140/1032 | loss 3.6715 | lr 2.10e-04 | 346152 tok/s | 0.1h elapsed | ~0.8h remaining
+2026-10-19 23:23:43,055 INFO [rank 0] step 160/1032 | loss 3.6275 | lr 2.40e-04 | 346754 tok/s | 0.1h elapsed | ~0.7h remaining
+2026-10-19 23:24:43,248 INFO [rank 0] step 180/1032 | loss 3.5879 | lr 2.70e-04 | 346936 tok/s | 0.2h elapsed | ~0.7h remaining
+2026-10-19 23:25:42,998 INFO [rank 0] step 200/1032 | loss 3.5558 | lr 3.00e-04 | 347337 tok/s | 0.2h elapsed | ~0.7h remaining
+2026-10-19 23:26:42,700 INFO [rank 0] step 220/1032 | loss 3.5210 | lr 3.00e-04 | 347691 tok/s | 0.2h elapsed | ~0.7h remaining
+2026-10-19 23:27:42,492 INFO [rank 0] step 240/1032 | loss 3.4818 | lr 2.99e-04 | 347944 tok/s | 0.2h elapsed | ~0.7h remaining
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+2026-10-19 23:28:16,343 INFO [rank 0] Saved checkpoint at step 250 -> /workspace/RiemannModel/checkpoints/stage2_538m
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  warnings.warn(  # warn only once
+2026-10-19 23:28:46,363 INFO [rank 0] step 260/1032 | loss 3.4492 | lr 2.97e-04 | 346352 tok/s | 0.2h elapsed | ~0.6h remaining
+2026-10-19 23:29:46,357 INFO [rank 0] step 280/1032 | loss 3.3954 | lr 2.94e-04 | 346580 tok/s | 0.2h elapsed | ~0.6h remaining
+2026-10-19 23:30:46,389 INFO [rank 0] step 300/1032 | loss 3.3589 | lr 2.91e-04 | 346762 tok/s | 0.3h elapsed | ~0.6h remaining
+2026-10-19 23:31:46,682 INFO [rank 0] step 320/1032 | loss 3.2849 | lr 2.87e-04 | 346829 tok/s | 0.3h elapsed | ~0.6h remaining
+2026-10-19 23:32:47,127 INFO [rank 0] step 340/1032 | loss 3.2320 | lr 2.82e-04 | 346836 tok/s | 0.3h elapsed | ~0.6h remaining
+2026-10-19 23:33:47,192 INFO [rank 0] step 360/1032 | loss 3.1603 | lr 2.76e-04 | 346964 tok/s | 0.3h elapsed | ~0.6h remaining
+2026-10-19 23:34:47,659 INFO [rank 0] step 380/1032 | loss 3.0949 | lr 2.70e-04 | 346956 tok/s | 0.3h elapsed | ~0.5h remaining
+2026-10-19 23:35:48,297 INFO [rank 0] step 400/1032 | loss 3.0317 | lr 2.64e-04 | 346901 tok/s | 0.3h elapsed | ~0.5h remaining
+2026-10-19 23:36:48,671 INFO [rank 0] step 420/1032 | loss 2.9633 | lr 2.56e-04 | 346922 tok/s | 0.4h elapsed | ~0.5h remaining
+2026-10-19 23:37:49,237 INFO [rank 0] step 440/1032 | loss 2.8976 | lr 2.49e-04 | 346892 tok/s | 0.4h elapsed | ~0.5h remaining
+2026-10-19 23:38:50,024 INFO [rank 0] step 460/1032 | loss 2.8261 | lr 2.40e-04 | 346810 tok/s | 0.4h elapsed | ~0.5h remaining
+2026-10-19 23:39:50,600 INFO [rank 0] step 480/1032 | loss 2.7663 | lr 2.32e-04 | 346784 tok/s | 0.4h elapsed | ~0.5h remaining
+2026-10-19 23:40:51,673 INFO [rank 0] step 500/1032 | loss 2.7029 | lr 2.23e-04 | 346647 tok/s | 0.4h elapsed | ~0.4h remaining
+2026-10-19 23:40:55,314 INFO [rank 0] Saved checkpoint at step 500 -> /workspace/RiemannModel/checkpoints/stage2_538m
+2026-10-19 23:41:55,852 INFO [rank 0] step 520/1032 | loss 2.6581 | lr 2.13e-04 | 345838 tok/s | 0.4h elapsed | ~0.4h remaining
+2026-10-19 23:42:56,449 INFO [rank 0] step 540/1032 | loss 2.6058 | lr 2.04e-04 | 345847 tok/s | 0.5h elapsed | ~0.4h remaining
+2026-10-19 23:43:57,212 INFO [rank 0] step 560/1032 | loss 2.5614 | lr 1.94e-04 | 345821 tok/s | 0.5h elapsed | ~0.4h remaining
+2026-10-19 23:44:57,948 INFO [rank 0] step 580/1032 | loss 2.5165 | lr 1.84e-04 | 345803 tok/s | 0.5h elapsed | ~0.4h remaining
+2026-10-19 23:45:58,530 INFO [rank 0] step 600/1032 | loss 2.4934 | lr 1.74e-04 | 345815 tok/s | 0.5h elapsed | ~0.4h remaining
+2026-10-19 23:46:59,125 INFO [rank 0] step 620/1032 | loss 2.4446 | lr 1.63e-04 | 345824 tok/s | 0.5h elapsed | ~0.3h remaining
+2026-10-19 23:47:59,698 INFO [rank 0] step 640/1032 | loss 2.4215 | lr 1.53e-04 | 345836 tok/s | 0.5h elapsed | ~0.3h remaining
+2026-10-19 23:49:00,149 INFO [rank 0] step 660/1032 | loss 2.4078 | lr 1.43e-04 | 345869 tok/s | 0.6h elapsed | ~0.3h remaining
+2026-10-19 23:50:00,674 INFO [rank 0] step 680/1032 | loss 2.3760 | lr 1.33e-04 | 345887 tok/s | 0.6h elapsed | ~0.3h remaining
+2026-10-19 23:51:00,926 INFO [rank 0] step 700/1032 | loss 2.3554 | lr 1.23e-04 | 345949 tok/s | 0.6h elapsed | ~0.3h remaining
+2026-10-19 23:52:01,371 INFO [rank 0] step 720/1032 | loss 2.3478 | lr 1.14e-04 | 345977 tok/s | 0.6h elapsed | ~0.3h remaining
+2026-10-19 23:53:01,732 INFO [rank 0] step 740/1032 | loss 2.3374 | lr 1.05e-04 | 346016 tok/s | 0.6h elapsed | ~0.2h remaining
+2026-10-19 23:53:35,348 INFO [rank 0] Saved checkpoint at step 750 -> /workspace/RiemannModel/checkpoints/stage2_538m
+2026-10-19 23:54:05,438 INFO [rank 0] step 760/1032 | loss 2.3081 | lr 9.56e-05 | 345552 tok/s | 0.6h elapsed | ~0.2h remaining
+2026-10-19 23:55:05,741 INFO [rank 0] step 780/1032 | loss 2.3049 | lr 8.71e-05 | 345608 tok/s | 0.7h elapsed | ~0.2h remaining
+2026-10-19 23:56:06,151 INFO [rank 0] step 800/1032 | loss 2.2835 | lr 7.90e-05 | 345647 tok/s | 0.7h elapsed | ~0.2h remaining
+2026-10-19 23:57:06,671 INFO [rank 0] step 820/1032 | loss 2.2798 | lr 7.14e-05 | 345668 tok/s | 0.7h elapsed | ~0.2h remaining
+2026-10-19 23:58:07,142 INFO [rank 0] step 840/1032 | loss 2.2702 | lr 6.43e-05 | 345695 tok/s | 0.7h elapsed | ~0.2h remaining
+2026-10-19 23:59:07,781 INFO [rank 0] step 860/1032 | loss 2.2573 | lr 5.78e-05 | 345698 tok/s | 0.7h elapsed | ~0.1h remaining
+2026-10-20 00:00:08,262 INFO [rank 0] step 880/1032 | loss 2.2535 | lr 5.19e-05 | 345722 tok/s | 0.7h elapsed | ~0.1h remaining
+2026-10-20 00:01:08,844 INFO [rank 0] step 900/1032 | loss 2.2496 | lr 4.67e-05 | 345732 tok/s | 0.8h elapsed | ~0.1h remaining
+2026-10-20 00:02:09,402 INFO [rank 0] step 920/1032 | loss 2.2503 | lr 4.21e-05 | 345744 tok/s | 0.8h elapsed | ~0.1h remaining
+2026-10-20 00:03:09,745 INFO [rank 0] step 940/1032 | loss 2.2272 | lr 3.82e-05 | 345782 tok/s | 0.8h elapsed | ~0.1h remaining
+2026-10-20 00:04:09,963 INFO [rank 0] step 960/1032 | loss 2.2299 | lr 3.51e-05 | 345833 tok/s | 0.8h elapsed | ~0.1h remaining
+2026-10-20 00:05:10,276 INFO [rank 0] step 980/1032 | loss 2.2276 | lr 3.27e-05 | 345872 tok/s | 0.8h elapsed | ~0.0h remaining
+2026-10-20 00:06:10,633 INFO [rank 0] step 1000/1032 | loss 2.2226 | lr 3.10e-05 | 345903 tok/s | 0.8h elapsed | ~0.0h remaining
+2026-10-20 00:06:14,141 INFO [rank 0] Saved checkpoint at step 1000 -> /workspace/RiemannModel/checkpoints/stage2_538m
+2026-10-20 00:07:14,407 INFO [rank 0] step 1020/1032 | loss 2.2236 | lr 3.02e-05 | 345552 tok/s | 0.9h elapsed | ~0.0h remaining
+2026-10-20 00:07:54,004 INFO [rank 0] Saved checkpoint at step 1032 -> /workspace/RiemannModel/checkpoints/stage2_538m
+2026-10-20 00:07:54,061 INFO [rank 0] Done. stage2_538m at step 1032, 1.08B tokens seen, 0.87h wall clock, checkpoint at /workspace/RiemannModel/checkpoints/stage2_538m

training_logs/pretrain_stage3_1.19b.log

@@ -0,0 +1,240 @@
+W1020 00:21:31.953000 19307 torch/distributed/run.py:803] 
+W1020 00:21:31.953000 19307 torch/distributed/run.py:803] *****************************************
+W1020 00:21:31.953000 19307 torch/distributed/run.py:803] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
+W1020 00:21:31.953000 19307 torch/distributed/run.py:803] *****************************************
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+`torch_dtype` is deprecated! Use `dtype` instead!
+
+`torch_dtype` is deprecated! Use `dtype` instead!
+
+
+`torch_dtype` is deprecated! Use `dtype` instead!
+
+
+
+
+2026-10-20 00:21:46,052 INFO [rank 0] Loading stage3_1.19b from /workspace/RiemannModel/checkpoints/stage3_1.19b
+`torch_dtype` is deprecated! Use `dtype` instead!
+`torch_dtype` is deprecated! Use `dtype` instead!
+
+
+
+
+
+riemann-climb:19376:19376 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:19376:19376 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+RCCL version : 2.22.3-HEAD:7d8d67c
+HIP version  : 6.4.43484-123eb5128
+ROCm version : 6.4.2.0-120-e7d83f5
+Hostname     : riemann-climb
+Librccl path : /workspace/venv/lib/python3.12/site-packages/torch/lib/librccl.so
+
+riemann-climb:19377:19377 [1] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:19381:19381 [5] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:19380:19380 [4] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:19378:19378 [2] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:19379:19379 [3] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:19383:19383 [7] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:19382:19382 [6] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+2026-10-20 00:21:53,536 INFO [rank 0] Training stage3_1.19b: world=8 batch=32 accum=4 seq=1024 tokens/step=1048576, corpus=7218078 docs, ~1032 steps/epoch, target 1032 steps (1.00 epochs), 0.16B params
+Token indices sequence length is longer than the specified maximum sequence length for this model (1572 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1363 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1193 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1266 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1469 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1522 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1320 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1587 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1402 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1407 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1430 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1110 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1150 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1451 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1191 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1245 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1280 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1121 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1177 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1429 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1403 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1327 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1347 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1300 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1135 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1136 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1157 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1390 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1337 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1411 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1383 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1466 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1399 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1122 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1492 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1348 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1492 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1105 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1377 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1371 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1422 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1545 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1103 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1040 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1235 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1189 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1240 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1143 > 1024). Running this sequence through the model will result in indexing errors
+2026-10-20 00:22:54,219 INFO [rank 0] step 20/1032 | loss 2.1712 | lr 3.00e-05 | 345896 tok/s | 0.0h elapsed | ~0.9h remaining
+2026-10-20 00:23:50,164 INFO [rank 0] step 40/1032 | loss 2.2865 | lr 6.00e-05 | 359789 tok/s | 0.0h elapsed | ~0.8h remaining
+2026-10-20 00:24:46,046 INFO [rank 0] step 60/1032 | loss 2.2602 | lr 9.00e-05 | 364815 tok/s | 0.0h elapsed | ~0.8h remaining
+2026-10-20 00:25:42,050 INFO [rank 0] step 80/1032 | loss 2.2436 | lr 1.20e-04 | 367182 tok/s | 0.1h elapsed | ~0.8h remaining
+2026-10-20 00:26:37,954 INFO [rank 0] step 100/1032 | loss 2.2440 | lr 1.50e-04 | 368743 tok/s | 0.1h elapsed | ~0.7h remaining
+2026-10-20 00:27:33,872 INFO [rank 0] step 120/1032 | loss 2.2451 | lr 1.80e-04 | 369776 tok/s | 0.1h elapsed | ~0.7h remaining
+2026-10-20 00:28:29,821 INFO [rank 0] step 140/1032 | loss 2.2413 | lr 2.10e-04 | 370494 tok/s | 0.1h elapsed | ~0.7h remaining
+2026-10-20 00:29:25,780 INFO [rank 0] step 160/1032 | loss 2.2577 | lr 2.40e-04 | 371021 tok/s | 0.1h elapsed | ~0.7h remaining
+2026-10-20 00:30:21,737 INFO [rank 0] step 180/1032 | loss 2.2464 | lr 2.70e-04 | 371434 tok/s | 0.1h elapsed | ~0.7h remaining
+2026-10-20 00:31:17,702 INFO [rank 0] step 200/1032 | loss 2.2439 | lr 3.00e-04 | 371761 tok/s | 0.2h elapsed | ~0.7h remaining
+2026-10-20 00:32:13,687 INFO [rank 0] step 220/1032 | loss 2.2371 | lr 3.00e-04 | 372017 tok/s | 0.2h elapsed | ~0.6h remaining
+2026-10-20 00:33:09,685 INFO [rank 0] step 240/1032 | loss 2.1983 | lr 2.99e-04 | 372224 tok/s | 0.2h elapsed | ~0.6h remaining
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+2026-10-20 00:33:44,804 INFO [rank 0] Saved checkpoint at step 250 -> /workspace/RiemannModel/checkpoints/stage3_1.19b
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  warnings.warn(  # warn only once
+2026-10-20 00:34:12,719 INFO [rank 0] step 260/1032 | loss 2.2012 | lr 2.97e-04 | 368851 tok/s | 0.2h elapsed | ~0.6h remaining
+2026-10-20 00:35:08,605 INFO [rank 0] step 280/1032 | loss 2.1747 | lr 2.94e-04 | 369302 tok/s | 0.2h elapsed | ~0.6h remaining
+2026-10-20 00:36:04,551 INFO [rank 0] step 300/1032 | loss 2.1743 | lr 2.91e-04 | 369667 tok/s | 0.2h elapsed | ~0.6h remaining
+2026-10-20 00:37:00,544 INFO [rank 0] step 320/1032 | loss 2.1484 | lr 2.87e-04 | 369968 tok/s | 0.3h elapsed | ~0.6h remaining
+2026-10-20 00:37:56,514 INFO [rank 0] step 340/1032 | loss 2.1395 | lr 2.82e-04 | 370243 tok/s | 0.3h elapsed | ~0.5h remaining
+2026-10-20 00:38:52,517 INFO [rank 0] step 360/1032 | loss 2.1176 | lr 2.76e-04 | 370475 tok/s | 0.3h elapsed | ~0.5h remaining
+2026-10-20 00:39:48,506 INFO [rank 0] step 380/1032 | loss 2.1059 | lr 2.70e-04 | 370688 tok/s | 0.3h elapsed | ~0.5h remaining
+2026-10-20 00:40:44,552 INFO [rank 0] step 400/1032 | loss 2.0935 | lr 2.64e-04 | 370860 tok/s | 0.3h elapsed | ~0.5h remaining
+2026-10-20 00:41:40,561 INFO [rank 0] step 420/1032 | loss 2.0781 | lr 2.56e-04 | 371030 tok/s | 0.3h elapsed | ~0.5h remaining
+2026-10-20 00:42:36,557 INFO [rank 0] step 440/1032 | loss 2.0756 | lr 2.49e-04 | 371187 tok/s | 0.3h elapsed | ~0.5h remaining
+2026-10-20 00:43:32,537 INFO [rank 0] step 460/1032 | loss 2.0530 | lr 2.40e-04 | 371335 tok/s | 0.4h elapsed | ~0.4h remaining
+2026-10-20 00:44:28,613 INFO [rank 0] step 480/1032 | loss 2.0511 | lr 2.32e-04 | 371445 tok/s | 0.4h elapsed | ~0.4h remaining
+2026-10-20 00:45:24,602 INFO [rank 0] step 500/1032 | loss 2.0276 | lr 2.23e-04 | 371568 tok/s | 0.4h elapsed | ~0.4h remaining
+2026-10-20 00:45:31,669 INFO [rank 0] Saved checkpoint at step 500 -> /workspace/RiemannModel/checkpoints/stage3_1.19b
+2026-10-20 00:46:27,580 INFO [rank 0] step 520/1032 | loss 2.0268 | lr 2.13e-04 | 369921 tok/s | 0.4h elapsed | ~0.4h remaining
+2026-10-20 00:47:23,533 INFO [rank 0] step 540/1032 | loss 2.0125 | lr 2.04e-04 | 370099 tok/s | 0.4h elapsed | ~0.4h remaining
+2026-10-20 00:48:19,559 INFO [rank 0] step 560/1032 | loss 1.9975 | lr 1.94e-04 | 370248 tok/s | 0.4h elapsed | ~0.4h remaining
+2026-10-20 00:49:15,626 INFO [rank 0] step 580/1032 | loss 1.9867 | lr 1.84e-04 | 370378 tok/s | 0.5h elapsed | ~0.4h remaining
+2026-10-20 00:50:11,620 INFO [rank 0] step 600/1032 | loss 1.9871 | lr 1.74e-04 | 370515 tok/s | 0.5h elapsed | ~0.3h remaining
+2026-10-20 00:51:07,629 INFO [rank 0] step 620/1032 | loss 1.9603 | lr 1.63e-04 | 370641 tok/s | 0.5h elapsed | ~0.3h remaining
+2026-10-20 00:52:03,641 INFO [rank 0] step 640/1032 | loss 1.9573 | lr 1.53e-04 | 370756 tok/s | 0.5h elapsed | ~0.3h remaining
+2026-10-20 00:52:59,636 INFO [rank 0] step 660/1032 | loss 1.9553 | lr 1.43e-04 | 370870 tok/s | 0.5h elapsed | ~0.3h remaining
+2026-10-20 00:53:55,627 INFO [rank 0] step 680/1032 | loss 1.9386 | lr 1.33e-04 | 370977 tok/s | 0.5h elapsed | ~0.3h remaining
+2026-10-20 00:54:51,698 INFO [rank 0] step 700/1032 | loss 1.9342 | lr 1.23e-04 | 371063 tok/s | 0.5h elapsed | ~0.3h remaining
+2026-10-20 00:55:47,709 INFO [rank 0] step 720/1032 | loss 1.9314 | lr 1.14e-04 | 371155 tok/s | 0.6h elapsed | ~0.2h remaining
+2026-10-20 00:56:43,749 INFO [rank 0] step 740/1032 | loss 1.9295 | lr 1.05e-04 | 371238 tok/s | 0.6h elapsed | ~0.2h remaining
+2026-10-20 00:57:18,729 INFO [rank 0] Saved checkpoint at step 750 -> /workspace/RiemannModel/checkpoints/stage3_1.19b
+2026-10-20 00:57:46,686 INFO [rank 0] step 760/1032 | loss 1.9113 | lr 9.56e-05 | 370127 tok/s | 0.6h elapsed | ~0.2h remaining
+2026-10-20 00:58:42,610 INFO [rank 0] step 780/1032 | loss 1.9142 | lr 8.71e-05 | 370249 tok/s | 0.6h elapsed | ~0.2h remaining
+2026-10-20 00:59:38,592 INFO [rank 0] step 800/1032 | loss 1.9000 | lr 7.90e-05 | 370358 tok/s | 0.6h elapsed | ~0.2h remaining
+2026-10-20 01:00:34,585 INFO [rank 0] step 820/1032 | loss 1.8998 | lr 7.14e-05 | 370459 tok/s | 0.6h elapsed | ~0.2h remaining
+2026-10-20 01:01:30,655 INFO [rank 0] step 840/1032 | loss 1.8986 | lr 6.43e-05 | 370543 tok/s | 0.7h elapsed | ~0.2h remaining
+2026-10-20 01:02:26,668 INFO [rank 0] step 860/1032 | loss 1.8892 | lr 5.78e-05 | 370631 tok/s | 0.7h elapsed | ~0.1h remaining
+2026-10-20 01:03:22,690 INFO [rank 0] step 880/1032 | loss 1.8870 | lr 5.19e-05 | 370715 tok/s | 0.7h elapsed | ~0.1h remaining
+2026-10-20 01:04:18,770 INFO [rank 0] step 900/1032 | loss 1.8881 | lr 4.67e-05 | 370786 tok/s | 0.7h elapsed | ~0.1h remaining
+2026-10-20 01:05:14,920 INFO [rank 0] step 920/1032 | loss 1.8887 | lr 4.21e-05 | 370845 tok/s | 0.7h elapsed | ~0.1h remaining
+2026-10-20 01:06:10,941 INFO [rank 0] step 940/1032 | loss 1.8743 | lr 3.82e-05 | 370919 tok/s | 0.7h elapsed | ~0.1h remaining
+2026-10-20 01:07:06,967 INFO [rank 0] step 960/1032 | loss 1.8769 | lr 3.51e-05 | 370989 tok/s | 0.8h elapsed | ~0.1h remaining
+2026-10-20 01:08:03,065 INFO [rank 0] step 980/1032 | loss 1.8764 | lr 3.27e-05 | 371047 tok/s | 0.8h elapsed | ~0.0h remaining
+2026-10-20 01:08:59,090 INFO [rank 0] step 1000/1032 | loss 1.8733 | lr 3.10e-05 | 371112 tok/s | 0.8h elapsed | ~0.0h remaining
+2026-10-20 01:09:05,916 INFO [rank 0] Saved checkpoint at step 1000 -> /workspace/RiemannModel/checkpoints/stage3_1.19b
+2026-10-20 01:10:01,843 INFO [rank 0] step 1020/1032 | loss 1.8773 | lr 3.02e-05 | 370310 tok/s | 0.8h elapsed | ~0.0h remaining
+2026-10-20 01:10:42,451 INFO [rank 0] Saved checkpoint at step 1032 -> /workspace/RiemannModel/checkpoints/stage3_1.19b
+2026-10-20 01:10:42,530 INFO [rank 0] Done. stage3_1.19b at step 1032, 1.08B tokens seen, 0.81h wall clock, checkpoint at /workspace/RiemannModel/checkpoints/stage3_1.19b

training_logs/pretrain_stage4_2.65b.log

@@ -0,0 +1,242 @@
+W1020 01:16:10.814000 22649 torch/distributed/run.py:803] 
+W1020 01:16:10.814000 22649 torch/distributed/run.py:803] *****************************************
+W1020 01:16:10.814000 22649 torch/distributed/run.py:803] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
+W1020 01:16:10.814000 22649 torch/distributed/run.py:803] *****************************************
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2026-10-20 01:16:24,511 INFO [rank 0] Loading stage4_2.65b from /workspace/RiemannModel/checkpoints/stage4_2.65b
+`torch_dtype` is deprecated! Use `dtype` instead!
+
+`torch_dtype` is deprecated! Use `dtype` instead!
+
+`torch_dtype` is deprecated! Use `dtype` instead!
+`torch_dtype` is deprecated! Use `dtype` instead!
+
+`torch_dtype` is deprecated! Use `dtype` instead!
+
+
+
+
+`torch_dtype` is deprecated! Use `dtype` instead!
+
+`torch_dtype` is deprecated! Use `dtype` instead!
+
+
+riemann-climb:22718:22718 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:22718:22718 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+RCCL version : 2.22.3-HEAD:7d8d67c
+HIP version  : 6.4.43484-123eb5128
+ROCm version : 6.4.2.0-120-e7d83f5
+Hostname     : riemann-climb
+Librccl path : /workspace/venv/lib/python3.12/site-packages/torch/lib/librccl.so
+
+riemann-climb:22724:22724 [6] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:22723:22723 [5] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:22720:22720 [2] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:22725:22725 [7] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:22721:22721 [3] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:22722:22722 [4] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:22719:22719 [1] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+2026-10-20 01:16:33,422 INFO [rank 0] Training stage4_2.65b: world=8 batch=16 accum=4 seq=2048 tokens/step=1048576, corpus=7218078 docs, ~1032 steps/epoch, target 1032 steps (1.00 epochs), 0.35B params
+Token indices sequence length is longer than the specified maximum sequence length for this model (1522 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1363 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1193 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1469 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1572 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1266 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1402 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1430 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1320 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1587 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1407 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1110 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1150 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1451 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1191 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1280 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1121 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1245 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1403 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1177 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1429 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1327 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1390 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1157 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1136 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1347 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1411 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1300 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1135 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1337 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1492 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1466 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1399 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1383 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1122 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1348 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1105 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1492 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1377 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1371 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1545 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1422 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1103 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1040 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1235 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1189 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1240 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1143 > 1024). Running this sequence through the model will result in indexing errors
+2026-10-20 01:18:33,880 INFO [rank 0] step 20/1032 | loss 2.8309 | lr 3.00e-05 | 174380 tok/s | 0.0h elapsed | ~1.7h remaining
+2026-10-20 01:20:29,958 INFO [rank 0] step 40/1032 | loss 2.3154 | lr 6.00e-05 | 177489 tok/s | 0.1h elapsed | ~1.6h remaining
+2026-10-20 01:22:26,317 INFO [rank 0] step 60/1032 | loss 1.9829 | lr 9.00e-05 | 178398 tok/s | 0.1h elapsed | ~1.6h remaining
+2026-10-20 01:24:22,837 INFO [rank 0] step 80/1032 | loss 1.9206 | lr 1.20e-04 | 178791 tok/s | 0.1h elapsed | ~1.6h remaining
+2026-10-20 01:26:19,383 INFO [rank 0] step 100/1032 | loss 1.9109 | lr 1.50e-04 | 179019 tok/s | 0.2h elapsed | ~1.5h remaining
+2026-10-20 01:28:15,822 INFO [rank 0] step 120/1032 | loss 1.9083 | lr 1.80e-04 | 179200 tok/s | 0.2h elapsed | ~1.5h remaining
+2026-10-20 01:30:12,298 INFO [rank 0] step 140/1032 | loss 1.9123 | lr 2.10e-04 | 179320 tok/s | 0.2h elapsed | ~1.4h remaining
+2026-10-20 01:32:08,787 INFO [rank 0] step 160/1032 | loss 1.9199 | lr 2.40e-04 | 179409 tok/s | 0.3h elapsed | ~1.4h remaining
+2026-10-20 01:34:05,389 INFO [rank 0] step 180/1032 | loss 1.9169 | lr 2.70e-04 | 179459 tok/s | 0.3h elapsed | ~1.4h remaining
+2026-10-20 01:36:01,918 INFO [rank 0] step 200/1032 | loss 1.9244 | lr 3.00e-04 | 179509 tok/s | 0.3h elapsed | ~1.4h remaining
+2026-10-20 01:37:58,436 INFO [rank 0] step 220/1032 | loss 1.9171 | lr 3.00e-04 | 179552 tok/s | 0.4h elapsed | ~1.3h remaining
+2026-10-20 01:39:54,962 INFO [rank 0] step 240/1032 | loss 1.8900 | lr 2.99e-04 | 179588 tok/s | 0.4h elapsed | ~1.3h remaining
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+2026-10-20 01:41:03,878 INFO [rank 0] Saved checkpoint at step 250 -> /workspace/RiemannModel/checkpoints/stage4_2.65b
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  warnings.warn(  # warn only once
+2026-10-20 01:42:02,031 INFO [rank 0] step 260/1032 | loss 1.8978 | lr 2.97e-04 | 178377 tok/s | 0.4h elapsed | ~1.3h remaining
+2026-10-20 01:43:58,446 INFO [rank 0] step 280/1032 | loss 1.8798 | lr 2.94e-04 | 178503 tok/s | 0.5h elapsed | ~1.2h remaining
+2026-10-20 01:45:54,978 INFO [rank 0] step 300/1032 | loss 1.8846 | lr 2.91e-04 | 178600 tok/s | 0.5h elapsed | ~1.2h remaining
+2026-10-20 01:47:51,545 INFO [rank 0] step 320/1032 | loss 1.8651 | lr 2.87e-04 | 178681 tok/s | 0.5h elapsed | ~1.2h remaining
+2026-10-20 01:49:48,143 INFO [rank 0] step 340/1032 | loss 1.8600 | lr 2.82e-04 | 178750 tok/s | 0.6h elapsed | ~1.1h remaining
+2026-10-20 01:51:44,699 INFO [rank 0] step 360/1032 | loss 1.8486 | lr 2.76e-04 | 178815 tok/s | 0.6h elapsed | ~1.1h remaining
+2026-10-20 01:53:41,243 INFO [rank 0] step 380/1032 | loss 1.8418 | lr 2.70e-04 | 178874 tok/s | 0.6h elapsed | ~1.1h remaining
+2026-10-20 01:55:37,805 INFO [rank 0] step 400/1032 | loss 1.8348 | lr 2.64e-04 | 178926 tok/s | 0.7h elapsed | ~1.0h remaining
+2026-10-20 01:57:34,332 INFO [rank 0] step 420/1032 | loss 1.8271 | lr 2.56e-04 | 178975 tok/s | 0.7h elapsed | ~1.0h remaining
+2026-10-20 01:59:30,887 INFO [rank 0] step 440/1032 | loss 1.8252 | lr 2.49e-04 | 179019 tok/s | 0.7h elapsed | ~1.0h remaining
+2026-10-20 02:01:27,461 INFO [rank 0] step 460/1032 | loss 1.8150 | lr 2.40e-04 | 179057 tok/s | 0.7h elapsed | ~0.9h remaining
+2026-10-20 02:03:24,053 INFO [rank 0] step 480/1032 | loss 1.8129 | lr 2.32e-04 | 179090 tok/s | 0.8h elapsed | ~0.9h remaining
+2026-10-20 02:05:20,620 INFO [rank 0] step 500/1032 | loss 1.7988 | lr 2.23e-04 | 179123 tok/s | 0.8h elapsed | ~0.9h remaining
+2026-10-20 02:05:31,366 INFO [rank 0] Saved checkpoint at step 500 -> /workspace/RiemannModel/checkpoints/stage4_2.65b
+2026-10-20 02:07:27,936 INFO [rank 0] step 520/1032 | loss 1.8017 | lr 2.13e-04 | 178523 tok/s | 0.8h elapsed | ~0.8h remaining
+2026-10-20 02:09:24,523 INFO [rank 0] step 540/1032 | loss 1.7935 | lr 2.04e-04 | 178572 tok/s | 0.9h elapsed | ~0.8h remaining
+2026-10-20 02:11:21,131 INFO [rank 0] step 560/1032 | loss 1.7843 | lr 1.94e-04 | 178618 tok/s | 0.9h elapsed | ~0.8h remaining
+2026-10-20 02:13:17,699 INFO [rank 0] step 580/1032 | loss 1.7782 | lr 1.84e-04 | 178662 tok/s | 0.9h elapsed | ~0.7h remaining
+2026-10-20 02:15:14,264 INFO [rank 0] step 600/1032 | loss 1.7819 | lr 1.74e-04 | 178703 tok/s | 1.0h elapsed | ~0.7h remaining
+2026-10-20 02:17:10,849 INFO [rank 0] step 620/1032 | loss 1.7621 | lr 1.63e-04 | 178741 tok/s | 1.0h elapsed | ~0.7h remaining
+2026-10-20 02:19:07,411 INFO [rank 0] step 640/1032 | loss 1.7619 | lr 1.53e-04 | 178778 tok/s | 1.0h elapsed | ~0.6h remaining
+2026-10-20 02:21:04,061 INFO [rank 0] step 660/1032 | loss 1.7629 | lr 1.43e-04 | 178808 tok/s | 1.1h elapsed | ~0.6h remaining
+2026-10-20 02:23:00,624 INFO [rank 0] step 680/1032 | loss 1.7502 | lr 1.33e-04 | 178840 tok/s | 1.1h elapsed | ~0.6h remaining
+2026-10-20 02:24:57,247 INFO [rank 0] step 700/1032 | loss 1.7490 | lr 1.23e-04 | 178868 tok/s | 1.1h elapsed | ~0.5h remaining
+2026-10-20 02:26:53,839 INFO [rank 0] step 720/1032 | loss 1.7482 | lr 1.14e-04 | 178896 tok/s | 1.2h elapsed | ~0.5h remaining
+2026-10-20 02:28:50,483 INFO [rank 0] step 740/1032 | loss 1.7484 | lr 1.05e-04 | 178920 tok/s | 1.2h elapsed | ~0.5h remaining
+2026-10-20 02:29:59,102 INFO [rank 0] Saved checkpoint at step 750 -> /workspace/RiemannModel/checkpoints/stage4_2.65b
+2026-10-20 02:30:57,443 INFO [rank 0] step 760/1032 | loss 1.7339 | lr 9.56e-05 | 178529 tok/s | 1.2h elapsed | ~0.4h remaining
+2026-10-20 02:32:54,110 INFO [rank 0] step 780/1032 | loss 1.7377 | lr 8.71e-05 | 178560 tok/s | 1.3h elapsed | ~0.4h remaining
+2026-10-20 02:34:50,732 INFO [rank 0] step 800/1032 | loss 1.7268 | lr 7.90e-05 | 178592 tok/s | 1.3h elapsed | ~0.4h remaining
+2026-10-20 02:36:47,475 INFO [rank 0] step 820/1032 | loss 1.7281 | lr 7.14e-05 | 178619 tok/s | 1.3h elapsed | ~0.3h remaining
+2026-10-20 02:38:44,173 INFO [rank 0] step 840/1032 | loss 1.7287 | lr 6.43e-05 | 178643 tok/s | 1.4h elapsed | ~0.3h remaining
+
+riemann-climb:22723:22803 [5] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/misc/socket.cc:438 NCCL WARN socketFinalizeAccept: wrong magic 1ee00010316 != b646e86fa12d961e
+2026-10-20 02:40:40,816 INFO [rank 0] step 860/1032 | loss 1.7210 | lr 5.78e-05 | 178670 tok/s | 1.4h elapsed | ~0.3h remaining
+2026-10-20 02:42:37,462 INFO [rank 0] step 880/1032 | loss 1.7200 | lr 5.19e-05 | 178695 tok/s | 1.4h elapsed | ~0.2h remaining
+2026-10-20 02:44:34,096 INFO [rank 0] step 900/1032 | loss 1.7222 | lr 4.67e-05 | 178720 tok/s | 1.5h elapsed | ~0.2h remaining
+2026-10-20 02:46:30,759 INFO [rank 0] step 920/1032 | loss 1.7234 | lr 4.21e-05 | 178742 tok/s | 1.5h elapsed | ~0.2h remaining
+2026-10-20 02:48:27,404 INFO [rank 0] step 940/1032 | loss 1.7106 | lr 3.82e-05 | 178764 tok/s | 1.5h elapsed | ~0.1h remaining
+2026-10-20 02:50:24,060 INFO [rank 0] step 960/1032 | loss 1.7138 | lr 3.51e-05 | 178785 tok/s | 1.6h elapsed | ~0.1h remaining
+2026-10-20 02:52:20,691 INFO [rank 0] step 980/1032 | loss 1.7143 | lr 3.27e-05 | 178806 tok/s | 1.6h elapsed | ~0.1h remaining
+2026-10-20 02:54:17,305 INFO [rank 0] step 1000/1032 | loss 1.7122 | lr 3.10e-05 | 178826 tok/s | 1.6h elapsed | ~0.1h remaining
+2026-10-20 02:54:27,615 INFO [rank 0] Saved checkpoint at step 1000 -> /workspace/RiemannModel/checkpoints/stage4_2.65b
+2026-10-20 02:56:24,341 INFO [rank 0] step 1020/1032 | loss 1.7164 | lr 3.02e-05 | 178535 tok/s | 1.7h elapsed | ~0.0h remaining
+2026-10-20 02:57:44,569 INFO [rank 0] Saved checkpoint at step 1032 -> /workspace/RiemannModel/checkpoints/stage4_2.65b
+2026-10-20 02:57:44,694 INFO [rank 0] Done. stage4_2.65b at step 1032, 1.08B tokens seen, 1.69h wall clock, checkpoint at /workspace/RiemannModel/checkpoints/stage4_2.65b

training_logs/pretrain_stage5_6.66b.log

@@ -0,0 +1,240 @@
+W1020 03:07:33.887000 28035 torch/distributed/run.py:803] 
+W1020 03:07:33.887000 28035 torch/distributed/run.py:803] *****************************************
+W1020 03:07:33.887000 28035 torch/distributed/run.py:803] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
+W1020 03:07:33.887000 28035 torch/distributed/run.py:803] *****************************************
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+`torch_dtype` is deprecated! Use `dtype` instead!
+
+`torch_dtype` is deprecated! Use `dtype` instead!
+
+`torch_dtype` is deprecated! Use `dtype` instead!
+2026-10-20 03:07:47,807 INFO [rank 0] Loading stage5_6.66b from /workspace/RiemannModel/checkpoints/stage5_6.66b
+`torch_dtype` is deprecated! Use `dtype` instead!
+
+`torch_dtype` is deprecated! Use `dtype` instead!
+
+
+
+`torch_dtype` is deprecated! Use `dtype` instead!
+`torch_dtype` is deprecated! Use `dtype` instead!
+
+
+
+
+riemann-climb:28104:28104 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:28104:28104 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+RCCL version : 2.22.3-HEAD:7d8d67c
+HIP version  : 6.4.43484-123eb5128
+ROCm version : 6.4.2.0-120-e7d83f5
+Hostname     : riemann-climb
+Librccl path : /workspace/venv/lib/python3.12/site-packages/torch/lib/librccl.so
+
+riemann-climb:28111:28111 [7] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:28110:28110 [6] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:28107:28107 [3] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:28108:28108 [4] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:28105:28105 [1] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:28106:28106 [2] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:28109:28109 [5] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+2026-10-20 03:08:00,347 INFO [rank 0] Training stage5_6.66b: world=8 batch=8 accum=8 seq=2048 tokens/step=1048576, corpus=7218078 docs, ~1032 steps/epoch, target 1032 steps (1.00 epochs), 0.86B params
+Token indices sequence length is longer than the specified maximum sequence length for this model (1363 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1572 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1266 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1402 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1407 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1150 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1245 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1191 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1280 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1403 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1177 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1300 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1347 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1135 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1122 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1466 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1492 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1383 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1348 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1193 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1522 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1469 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1587 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1320 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1430 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1110 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1451 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1121 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1429 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1327 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1157 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1337 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1411 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1136 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1390 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1399 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1492 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1105 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1377 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1371 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1422 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1103 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1040 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1189 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1240 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1545 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1235 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1143 > 1024). Running this sequence through the model will result in indexing errors
+2026-10-20 03:11:50,429 INFO [rank 0] step 20/1032 | loss 1.8763 | lr 3.00e-05 | 91263 tok/s | 0.1h elapsed | ~3.2h remaining
+2026-10-20 03:15:36,949 INFO [rank 0] step 40/1032 | loss 1.7875 | lr 6.00e-05 | 91918 tok/s | 0.1h elapsed | ~3.1h remaining
+2026-10-20 03:19:23,573 INFO [rank 0] step 60/1032 | loss 1.7555 | lr 9.00e-05 | 92124 tok/s | 0.2h elapsed | ~3.1h remaining
+2026-10-20 03:23:10,316 INFO [rank 0] step 80/1032 | loss 1.7371 | lr 1.20e-04 | 92214 tok/s | 0.3h elapsed | ~3.0h remaining
+2026-10-20 03:26:57,200 INFO [rank 0] step 100/1032 | loss 1.7389 | lr 1.50e-04 | 92259 tok/s | 0.3h elapsed | ~2.9h remaining
+2026-10-20 03:30:44,108 INFO [rank 0] step 120/1032 | loss 1.7405 | lr 1.80e-04 | 92285 tok/s | 0.4h elapsed | ~2.9h remaining
+2026-10-20 03:34:31,069 INFO [rank 0] step 140/1032 | loss 1.7441 | lr 2.10e-04 | 92302 tok/s | 0.4h elapsed | ~2.8h remaining
+2026-10-20 03:38:18,036 INFO [rank 0] step 160/1032 | loss 1.7498 | lr 2.40e-04 | 92313 tok/s | 0.5h elapsed | ~2.8h remaining
+2026-10-20 03:42:04,922 INFO [rank 0] step 180/1032 | loss 1.8745 | lr 2.70e-04 | 92327 tok/s | 0.6h elapsed | ~2.7h remaining
+2026-10-20 03:45:51,783 INFO [rank 0] step 200/1032 | loss 1.8192 | lr 3.00e-04 | 92338 tok/s | 0.6h elapsed | ~2.6h remaining
+2026-10-20 03:49:38,612 INFO [rank 0] step 220/1032 | loss 1.7556 | lr 3.00e-04 | 92349 tok/s | 0.7h elapsed | ~2.6h remaining
+2026-10-20 03:53:25,573 INFO [rank 0] step 240/1032 | loss 1.7410 | lr 2.99e-04 | 92354 tok/s | 0.8h elapsed | ~2.5h remaining
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+2026-10-20 03:55:38,918 INFO [rank 0] Saved checkpoint at step 250 -> /workspace/RiemannModel/checkpoints/stage5_6.66b
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  warnings.warn(  # warn only once
+2026-10-20 03:57:32,525 INFO [rank 0] step 260/1032 | loss 1.7487 | lr 2.97e-04 | 91736 tok/s | 0.8h elapsed | ~2.5h remaining
+2026-10-20 04:01:19,418 INFO [rank 0] step 280/1032 | loss 1.7394 | lr 2.94e-04 | 91785 tok/s | 0.9h elapsed | ~2.4h remaining
+2026-10-20 04:05:06,298 INFO [rank 0] step 300/1032 | loss 1.7467 | lr 2.91e-04 | 91828 tok/s | 1.0h elapsed | ~2.3h remaining
+2026-10-20 04:08:53,206 INFO [rank 0] step 320/1032 | loss 1.7297 | lr 2.87e-04 | 91866 tok/s | 1.0h elapsed | ~2.3h remaining
+2026-10-20 04:12:40,233 INFO [rank 0] step 340/1032 | loss 1.7289 | lr 2.82e-04 | 91895 tok/s | 1.1h elapsed | ~2.2h remaining
+2026-10-20 04:16:27,133 INFO [rank 0] step 360/1032 | loss 1.7165 | lr 2.76e-04 | 91925 tok/s | 1.1h elapsed | ~2.1h remaining
+2026-10-20 04:20:14,126 INFO [rank 0] step 380/1032 | loss 1.7133 | lr 2.70e-04 | 91949 tok/s | 1.2h elapsed | ~2.1h remaining
+2026-10-20 04:24:01,114 INFO [rank 0] step 400/1032 | loss 1.7121 | lr 2.64e-04 | 91971 tok/s | 1.3h elapsed | ~2.0h remaining
+2026-10-20 04:27:48,401 INFO [rank 0] step 420/1032 | loss 1.7046 | lr 2.56e-04 | 91985 tok/s | 1.3h elapsed | ~1.9h remaining
+2026-10-20 04:31:35,479 INFO [rank 0] step 440/1032 | loss 1.7053 | lr 2.49e-04 | 92001 tok/s | 1.4h elapsed | ~1.9h remaining
+2026-10-20 04:35:22,576 INFO [rank 0] step 460/1032 | loss 1.6966 | lr 2.40e-04 | 92016 tok/s | 1.5h elapsed | ~1.8h remaining
+2026-10-20 04:39:09,469 INFO [rank 0] step 480/1032 | loss 1.6970 | lr 2.32e-04 | 92033 tok/s | 1.5h elapsed | ~1.7h remaining
+2026-10-20 04:42:56,451 INFO [rank 0] step 500/1032 | loss 1.6855 | lr 2.23e-04 | 92048 tok/s | 1.6h elapsed | ~1.7h remaining
+2026-10-20 04:43:16,690 INFO [rank 0] Saved checkpoint at step 500 -> /workspace/RiemannModel/checkpoints/stage5_6.66b
+2026-10-20 04:47:03,662 INFO [rank 0] step 520/1032 | loss 1.6906 | lr 2.13e-04 | 91747 tok/s | 1.7h elapsed | ~1.6h remaining
+2026-10-20 04:50:50,338 INFO [rank 0] step 540/1032 | loss 1.6834 | lr 2.04e-04 | 91775 tok/s | 1.7h elapsed | ~1.6h remaining
+2026-10-20 04:54:36,948 INFO [rank 0] step 560/1032 | loss 1.6769 | lr 1.94e-04 | 91802 tok/s | 1.8h elapsed | ~1.5h remaining
+2026-10-20 04:58:23,530 INFO [rank 0] step 580/1032 | loss 1.6720 | lr 1.84e-04 | 91828 tok/s | 1.8h elapsed | ~1.4h remaining
+2026-10-20 05:02:10,121 INFO [rank 0] step 600/1032 | loss 1.6772 | lr 1.74e-04 | 91852 tok/s | 1.9h elapsed | ~1.4h remaining
+2026-10-20 05:05:56,716 INFO [rank 0] step 620/1032 | loss 1.6600 | lr 1.63e-04 | 91875 tok/s | 2.0h elapsed | ~1.3h remaining
+2026-10-20 05:09:43,279 INFO [rank 0] step 640/1032 | loss 1.6608 | lr 1.53e-04 | 91896 tok/s | 2.0h elapsed | ~1.2h remaining
+2026-10-20 05:13:29,845 INFO [rank 0] step 660/1032 | loss 1.6639 | lr 1.43e-04 | 91917 tok/s | 2.1h elapsed | ~1.2h remaining
+2026-10-20 05:17:16,420 INFO [rank 0] step 680/1032 | loss 1.6535 | lr 1.33e-04 | 91935 tok/s | 2.2h elapsed | ~1.1h remaining
+2026-10-20 05:21:02,987 INFO [rank 0] step 700/1032 | loss 1.6506 | lr 1.23e-04 | 91953 tok/s | 2.2h elapsed | ~1.1h remaining
+2026-10-20 05:24:49,497 INFO [rank 0] step 720/1032 | loss 1.6547 | lr 1.14e-04 | 91971 tok/s | 2.3h elapsed | ~1.0h remaining
+2026-10-20 05:28:36,096 INFO [rank 0] step 740/1032 | loss 1.6544 | lr 1.05e-04 | 91986 tok/s | 2.3h elapsed | ~0.9h remaining
+2026-10-20 05:30:49,206 INFO [rank 0] Saved checkpoint at step 750 -> /workspace/RiemannModel/checkpoints/stage5_6.66b
+2026-10-20 05:32:42,907 INFO [rank 0] step 760/1032 | loss 1.6420 | lr 9.56e-05 | 91787 tok/s | 2.4h elapsed | ~0.9h remaining
+2026-10-20 05:36:29,941 INFO [rank 0] step 780/1032 | loss 1.6449 | lr 8.71e-05 | 91802 tok/s | 2.5h elapsed | ~0.8h remaining
+2026-10-20 05:40:16,948 INFO [rank 0] step 800/1032 | loss 1.6364 | lr 7.90e-05 | 91816 tok/s | 2.5h elapsed | ~0.7h remaining
+2026-10-20 05:44:04,017 INFO [rank 0] step 820/1032 | loss 1.6391 | lr 7.14e-05 | 91829 tok/s | 2.6h elapsed | ~0.7h remaining
+2026-10-20 05:47:51,033 INFO [rank 0] step 840/1032 | loss 1.6383 | lr 6.43e-05 | 91842 tok/s | 2.7h elapsed | ~0.6h remaining
+2026-10-20 05:51:38,074 INFO [rank 0] step 860/1032 | loss 1.6322 | lr 5.78e-05 | 91854 tok/s | 2.7h elapsed | ~0.5h remaining
+2026-10-20 05:55:25,642 INFO [rank 0] step 880/1032 | loss 1.6335 | lr 5.19e-05 | 91861 tok/s | 2.8h elapsed | ~0.5h remaining
+2026-10-20 05:59:12,629 INFO [rank 0] step 900/1032 | loss 1.6343 | lr 4.67e-05 | 91873 tok/s | 2.9h elapsed | ~0.4h remaining
+2026-10-20 06:02:59,587 INFO [rank 0] step 920/1032 | loss 1.6370 | lr 4.21e-05 | 91884 tok/s | 2.9h elapsed | ~0.4h remaining
+2026-10-20 06:06:46,622 INFO [rank 0] step 940/1032 | loss 1.6234 | lr 3.82e-05 | 91895 tok/s | 3.0h elapsed | ~0.3h remaining
+2026-10-20 06:10:33,534 INFO [rank 0] step 960/1032 | loss 1.6279 | lr 3.51e-05 | 91906 tok/s | 3.0h elapsed | ~0.2h remaining
+2026-10-20 06:14:20,507 INFO [rank 0] step 980/1032 | loss 1.6289 | lr 3.27e-05 | 91916 tok/s | 3.1h elapsed | ~0.2h remaining
+2026-10-20 06:18:07,471 INFO [rank 0] step 1000/1032 | loss 1.6276 | lr 3.10e-05 | 91925 tok/s | 3.2h elapsed | ~0.1h remaining
+2026-10-20 06:18:27,596 INFO [rank 0] Saved checkpoint at step 1000 -> /workspace/RiemannModel/checkpoints/stage5_6.66b
+2026-10-20 06:22:14,994 INFO [rank 0] step 1020/1032 | loss 1.6308 | lr 3.02e-05 | 91772 tok/s | 3.2h elapsed | ~0.0h remaining
+2026-10-20 06:24:50,821 INFO [rank 0] Saved checkpoint at step 1032 -> /workspace/RiemannModel/checkpoints/stage5_6.66b
+2026-10-20 06:24:51,260 INFO [rank 0] Done. stage5_6.66b at step 1032, 1.08B tokens seen, 3.28h wall clock, checkpoint at /workspace/RiemannModel/checkpoints/stage5_6.66b

training_logs/pretrain_stage6_12.85b.log

@@ -0,0 +1,296 @@
+W1020 06:45:09.583000 484096 torch/distributed/run.py:803] 
+W1020 06:45:09.583000 484096 torch/distributed/run.py:803] *****************************************
+W1020 06:45:09.583000 484096 torch/distributed/run.py:803] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
+W1020 06:45:09.583000 484096 torch/distributed/run.py:803] *****************************************
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+`torch_dtype` is deprecated! Use `dtype` instead!
+
+`torch_dtype` is deprecated! Use `dtype` instead!
+`torch_dtype` is deprecated! Use `dtype` instead!
+`torch_dtype` is deprecated! Use `dtype` instead!
+`torch_dtype` is deprecated! Use `dtype` instead!
+`torch_dtype` is deprecated! Use `dtype` instead!
+`torch_dtype` is deprecated! Use `dtype` instead!
+2026-10-20 06:45:23,811 INFO [rank 0] Loading stage6_12.85b from /workspace/RiemannModel/checkpoints/stage6_12.85b
+`torch_dtype` is deprecated! Use `dtype` instead!
+
+
+
+
+
+
+
+
+riemann-climb:484165:484165 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:484165:484165 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+RCCL version : 2.22.3-HEAD:7d8d67c
+HIP version  : 6.4.43484-123eb5128
+ROCm version : 6.4.2.0-120-e7d83f5
+Hostname     : riemann-climb
+Librccl path : /workspace/venv/lib/python3.12/site-packages/torch/lib/librccl.so
+
+riemann-climb:484170:484170 [5] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:484172:484172 [7] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:484166:484166 [1] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:484167:484167 [2] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:484168:484168 [3] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:484169:484169 [4] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:484171:484171 [6] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+2026-10-20 06:45:42,993 INFO [rank 0] Training stage6_12.85b: world=8 batch=4 accum=16 seq=2048 tokens/step=1048576, corpus=7218078 docs, ~1032 steps/epoch, target 2064 steps (2.00 epochs), 1.64B params
+Token indices sequence length is longer than the specified maximum sequence length for this model (1150 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1280 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1363 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1572 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1266 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1407 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1402 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1245 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1191 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1403 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1177 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1300 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1347 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1135 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1492 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1122 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1466 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1383 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1348 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1193 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1320 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1430 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1110 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1587 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1121 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1451 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1429 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1327 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1157 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1390 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1337 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1136 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1522 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1469 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1411 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1399 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1492 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1371 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1103 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1240 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1105 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1377 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1422 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1040 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1189 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1545 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1235 > 1024). Running this sequence through the model will result in indexing errors
+Token indices sequence length is longer than the specified maximum sequence length for this model (1143 > 1024). Running this sequence through the model will result in indexing errors
+2026-10-20 06:52:56,995 INFO [rank 0] step 20/2064 | loss 2.0337 | lr 3.00e-05 | 48399 tok/s | 0.1h elapsed | ~12.3h remaining
+2026-10-20 07:00:07,512 INFO [rank 0] step 40/2064 | loss 1.8370 | lr 6.00e-05 | 48556 tok/s | 0.2h elapsed | ~12.1h remaining
+2026-10-20 07:07:18,582 INFO [rank 0] step 60/2064 | loss 1.7434 | lr 9.00e-05 | 48588 tok/s | 0.4h elapsed | ~12.0h remaining
+2026-10-20 07:14:29,312 INFO [rank 0] step 80/2064 | loss 1.7276 | lr 1.20e-04 | 48613 tok/s | 0.5h elapsed | ~11.9h remaining
+2026-10-20 07:21:40,016 INFO [rank 0] step 100/2064 | loss 1.7116 | lr 1.50e-04 | 48629 tok/s | 0.6h elapsed | ~11.8h remaining
+2026-10-20 07:28:50,781 INFO [rank 0] step 120/2064 | loss 1.7295 | lr 1.80e-04 | 48638 tok/s | 0.7h elapsed | ~11.6h remaining
+2026-10-20 07:36:02,039 INFO [rank 0] step 140/2064 | loss 1.7355 | lr 2.10e-04 | 48636 tok/s | 0.8h elapsed | ~11.5h remaining
+2026-10-20 07:43:12,982 INFO [rank 0] step 160/2064 | loss 1.6994 | lr 2.40e-04 | 48640 tok/s | 1.0h elapsed | ~11.4h remaining
+2026-10-20 07:50:23,812 INFO [rank 0] step 180/2064 | loss 1.7228 | lr 2.70e-04 | 48644 tok/s | 1.1h elapsed | ~11.3h remaining
+2026-10-20 07:57:34,417 INFO [rank 0] step 200/2064 | loss 1.8852 | lr 3.00e-04 | 48650 tok/s | 1.2h elapsed | ~11.2h remaining
+2026-10-20 08:04:44,979 INFO [rank 0] step 220/2064 | loss 1.9192 | lr 3.00e-04 | 48655 tok/s | 1.3h elapsed | ~11.0h remaining
+2026-10-20 08:11:55,576 INFO [rank 0] step 240/2064 | loss 1.7140 | lr 3.00e-04 | 48659 tok/s | 1.4h elapsed | ~10.9h remaining
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+2026-10-20 08:16:06,855 INFO [rank 0] Saved checkpoint at step 250 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  warnings.warn(  # warn only once
+2026-10-20 08:19:42,600 INFO [rank 0] step 260/2064 | loss 1.6971 | lr 2.99e-04 | 48348 tok/s | 1.6h elapsed | ~10.9h remaining
+2026-10-20 08:26:52,491 INFO [rank 0] step 280/2064 | loss 1.6805 | lr 2.99e-04 | 48379 tok/s | 1.7h elapsed | ~10.7h remaining
+2026-10-20 08:34:02,503 INFO [rank 0] step 300/2064 | loss 1.6987 | lr 2.98e-04 | 48405 tok/s | 1.8h elapsed | ~10.6h remaining
+2026-10-20 08:41:12,457 INFO [rank 0] step 320/2064 | loss 1.6803 | lr 2.97e-04 | 48428 tok/s | 1.9h elapsed | ~10.5h remaining
+2026-10-20 08:48:22,433 INFO [rank 0] step 340/2064 | loss 1.6833 | lr 2.96e-04 | 48448 tok/s | 2.0h elapsed | ~10.4h remaining
+2026-10-20 08:55:32,405 INFO [rank 0] step 360/2064 | loss 1.6622 | lr 2.95e-04 | 48466 tok/s | 2.2h elapsed | ~10.2h remaining
+2026-10-20 09:02:42,359 INFO [rank 0] step 380/2064 | loss 1.6607 | lr 2.94e-04 | 48482 tok/s | 2.3h elapsed | ~10.1h remaining
+2026-10-20 09:09:52,314 INFO [rank 0] step 400/2064 | loss 1.6562 | lr 2.92e-04 | 48497 tok/s | 2.4h elapsed | ~10.0h remaining
+2026-10-20 09:17:02,309 INFO [rank 0] step 420/2064 | loss 1.6503 | lr 2.91e-04 | 48510 tok/s | 2.5h elapsed | ~9.9h remaining
+2026-10-20 09:24:12,230 INFO [rank 0] step 440/2064 | loss 1.6571 | lr 2.89e-04 | 48522 tok/s | 2.6h elapsed | ~9.7h remaining
+2026-10-20 09:31:22,153 INFO [rank 0] step 460/2064 | loss 1.6430 | lr 2.87e-04 | 48533 tok/s | 2.8h elapsed | ~9.6h remaining
+2026-10-20 09:38:32,079 INFO [rank 0] step 480/2064 | loss 1.6455 | lr 2.85e-04 | 48543 tok/s | 2.9h elapsed | ~9.5h remaining
+2026-10-20 09:45:41,989 INFO [rank 0] step 500/2064 | loss 1.6392 | lr 2.83e-04 | 48553 tok/s | 3.0h elapsed | ~9.4h remaining
+2026-10-20 09:46:18,041 INFO [rank 0] Saved checkpoint at step 500 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
+2026-10-20 09:53:28,754 INFO [rank 0] step 520/2064 | loss 1.6442 | lr 2.81e-04 | 48403 tok/s | 3.1h elapsed | ~9.3h remaining
+2026-10-20 10:00:38,640 INFO [rank 0] step 540/2064 | loss 1.6359 | lr 2.79e-04 | 48417 tok/s | 3.2h elapsed | ~9.2h remaining
+2026-10-20 10:07:48,531 INFO [rank 0] step 560/2064 | loss 1.6306 | lr 2.76e-04 | 48430 tok/s | 3.4h elapsed | ~9.0h remaining
+2026-10-20 10:14:58,386 INFO [rank 0] step 580/2064 | loss 1.6298 | lr 2.73e-04 | 48442 tok/s | 3.5h elapsed | ~8.9h remaining
+2026-10-20 10:22:08,202 INFO [rank 0] step 600/2064 | loss 1.6348 | lr 2.71e-04 | 48454 tok/s | 3.6h elapsed | ~8.8h remaining
+2026-10-20 10:29:18,066 INFO [rank 0] step 620/2064 | loss 1.6209 | lr 2.68e-04 | 48464 tok/s | 3.7h elapsed | ~8.7h remaining
+2026-10-20 10:36:27,903 INFO [rank 0] step 640/2064 | loss 1.6229 | lr 2.65e-04 | 48474 tok/s | 3.8h elapsed | ~8.6h remaining
+2026-10-20 10:43:37,790 INFO [rank 0] step 660/2064 | loss 1.6260 | lr 2.62e-04 | 48484 tok/s | 4.0h elapsed | ~8.4h remaining
+2026-10-20 10:50:47,659 INFO [rank 0] step 680/2064 | loss 1.6188 | lr 2.58e-04 | 48493 tok/s | 4.1h elapsed | ~8.3h remaining
+2026-10-20 10:57:57,566 INFO [rank 0] step 700/2064 | loss 1.6183 | lr 2.55e-04 | 48501 tok/s | 4.2h elapsed | ~8.2h remaining
+2026-10-20 11:05:07,432 INFO [rank 0] step 720/2064 | loss 1.6237 | lr 2.52e-04 | 48509 tok/s | 4.3h elapsed | ~8.1h remaining
+2026-10-20 11:12:17,403 INFO [rank 0] step 740/2064 | loss 1.6255 | lr 2.48e-04 | 48516 tok/s | 4.4h elapsed | ~7.9h remaining
+2026-10-20 11:16:27,999 INFO [rank 0] Saved checkpoint at step 750 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
+2026-10-20 11:20:03,710 INFO [rank 0] step 760/2064 | loss 1.6134 | lr 2.44e-04 | 48415 tok/s | 4.6h elapsed | ~7.8h remaining
+2026-10-20 11:27:13,649 INFO [rank 0] step 780/2064 | loss 1.6184 | lr 2.41e-04 | 48425 tok/s | 4.7h elapsed | ~7.7h remaining
+2026-10-20 11:34:23,560 INFO [rank 0] step 800/2064 | loss 1.6114 | lr 2.37e-04 | 48433 tok/s | 4.8h elapsed | ~7.6h remaining
+2026-10-20 11:41:33,419 INFO [rank 0] step 820/2064 | loss 1.6148 | lr 2.33e-04 | 48442 tok/s | 4.9h elapsed | ~7.5h remaining
+2026-10-20 11:48:43,360 INFO [rank 0] step 840/2064 | loss 1.6149 | lr 2.29e-04 | 48450 tok/s | 5.0h elapsed | ~7.4h remaining
+2026-10-20 11:55:53,243 INFO [rank 0] step 860/2064 | loss 1.6090 | lr 2.25e-04 | 48458 tok/s | 5.2h elapsed | ~7.2h remaining
+2026-10-20 12:03:03,071 INFO [rank 0] step 880/2064 | loss 1.6103 | lr 2.21e-04 | 48465 tok/s | 5.3h elapsed | ~7.1h remaining
+2026-10-20 12:10:12,911 INFO [rank 0] step 900/2064 | loss 1.6113 | lr 2.17e-04 | 48472 tok/s | 5.4h elapsed | ~7.0h remaining
+2026-10-20 12:17:22,778 INFO [rank 0] step 920/2064 | loss 1.6132 | lr 2.12e-04 | 48479 tok/s | 5.5h elapsed | ~6.9h remaining
+2026-10-20 12:24:32,513 INFO [rank 0] step 940/2064 | loss 1.6001 | lr 2.08e-04 | 48486 tok/s | 5.6h elapsed | ~6.8h remaining
+2026-10-20 12:31:42,315 INFO [rank 0] step 960/2064 | loss 1.6044 | lr 2.04e-04 | 48492 tok/s | 5.8h elapsed | ~6.6h remaining
+2026-10-20 12:38:52,214 INFO [rank 0] step 980/2064 | loss 1.6050 | lr 1.99e-04 | 48498 tok/s | 5.9h elapsed | ~6.5h remaining
+2026-10-20 12:46:02,061 INFO [rank 0] step 1000/2064 | loss 1.6006 | lr 1.95e-04 | 48504 tok/s | 6.0h elapsed | ~6.4h remaining
+2026-10-20 12:46:39,533 INFO [rank 0] Saved checkpoint at step 1000 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
+2026-10-20 12:53:50,088 INFO [rank 0] step 1020/2064 | loss 1.6042 | lr 1.91e-04 | 48426 tok/s | 6.1h elapsed | ~6.3h remaining
+2026-10-20 13:00:59,373 INFO [rank 0] step 1040/2064 | loss 2.9027 | lr 1.86e-04 | 48434 tok/s | 6.3h elapsed | ~6.2h remaining
+2026-10-20 13:08:09,075 INFO [rank 0] step 1060/2064 | loss 2.0565 | lr 1.82e-04 | 48441 tok/s | 6.4h elapsed | ~6.0h remaining
+2026-10-20 13:15:19,026 INFO [rank 0] step 1080/2064 | loss 1.7069 | lr 1.77e-04 | 48447 tok/s | 6.5h elapsed | ~5.9h remaining
+2026-10-20 13:22:28,935 INFO [rank 0] step 1100/2064 | loss 1.5320 | lr 1.73e-04 | 48453 tok/s | 6.6h elapsed | ~5.8h remaining
+2026-10-20 13:29:38,958 INFO [rank 0] step 1120/2064 | loss 1.6455 | lr 1.68e-04 | 48458 tok/s | 6.7h elapsed | ~5.7h remaining
+2026-10-20 13:36:48,911 INFO [rank 0] step 1140/2064 | loss 1.9496 | lr 1.63e-04 | 48464 tok/s | 6.9h elapsed | ~5.6h remaining
+2026-10-20 13:43:59,087 INFO [rank 0] step 1160/2064 | loss 1.7489 | lr 1.59e-04 | 48469 tok/s | 7.0h elapsed | ~5.4h remaining
+2026-10-20 13:51:09,376 INFO [rank 0] step 1180/2064 | loss 1.0277 | lr 1.54e-04 | 48474 tok/s | 7.1h elapsed | ~5.3h remaining
+2026-10-20 13:58:19,757 INFO [rank 0] step 1200/2064 | loss 0.5920 | lr 1.50e-04 | 48478 tok/s | 7.2h elapsed | ~5.2h remaining
+2026-10-20 14:05:30,104 INFO [rank 0] step 1220/2064 | loss 0.3200 | lr 1.45e-04 | 48482 tok/s | 7.3h elapsed | ~5.1h remaining
+2026-10-20 14:12:40,598 INFO [rank 0] step 1240/2064 | loss 0.1636 | lr 1.41e-04 | 48486 tok/s | 7.4h elapsed | ~5.0h remaining
+2026-10-20 14:16:51,776 INFO [rank 0] Saved checkpoint at step 1250 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
+2026-10-20 14:20:27,885 INFO [rank 0] step 1260/2064 | loss 0.1215 | lr 1.36e-04 | 48424 tok/s | 7.6h elapsed | ~4.8h remaining
+2026-10-20 14:27:39,282 INFO [rank 0] step 1280/2064 | loss 0.1280 | lr 1.32e-04 | 48427 tok/s | 7.7h elapsed | ~4.7h remaining
+2026-10-20 14:34:50,839 INFO [rank 0] step 1300/2064 | loss 0.3189 | lr 1.28e-04 | 48429 tok/s | 7.8h elapsed | ~4.6h remaining
+2026-10-20 14:42:02,393 INFO [rank 0] step 1320/2064 | loss 0.7960 | lr 1.23e-04 | 48432 tok/s | 7.9h elapsed | ~4.5h remaining
+2026-10-20 14:49:13,890 INFO [rank 0] step 1340/2064 | loss 0.6287 | lr 1.19e-04 | 48434 tok/s | 8.1h elapsed | ~4.4h remaining
+2026-10-20 14:56:25,453 INFO [rank 0] step 1360/2064 | loss 0.4928 | lr 1.15e-04 | 48437 tok/s | 8.2h elapsed | ~4.2h remaining
+2026-10-20 15:03:37,045 INFO [rank 0] step 1380/2064 | loss 0.3535 | lr 1.10e-04 | 48439 tok/s | 8.3h elapsed | ~4.1h remaining
+2026-10-20 15:10:48,605 INFO [rank 0] step 1400/2064 | loss 0.2982 | lr 1.06e-04 | 48441 tok/s | 8.4h elapsed | ~4.0h remaining
+2026-10-20 15:18:00,146 INFO [rank 0] step 1420/2064 | loss 0.1965 | lr 1.02e-04 | 48443 tok/s | 8.5h elapsed | ~3.9h remaining
+2026-10-20 15:25:11,651 INFO [rank 0] step 1440/2064 | loss 0.1961 | lr 9.82e-05 | 48445 tok/s | 8.7h elapsed | ~3.8h remaining
+2026-10-20 15:32:23,124 INFO [rank 0] step 1460/2064 | loss 0.1520 | lr 9.43e-05 | 48448 tok/s | 8.8h elapsed | ~3.6h remaining
+2026-10-20 15:39:34,733 INFO [rank 0] step 1480/2064 | loss 0.1754 | lr 9.05e-05 | 48450 tok/s | 8.9h elapsed | ~3.5h remaining
+2026-10-20 15:46:46,436 INFO [rank 0] step 1500/2064 | loss 0.1258 | lr 8.67e-05 | 48451 tok/s | 9.0h elapsed | ~3.4h remaining
+2026-10-20 15:47:22,216 INFO [rank 0] Saved checkpoint at step 1500 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
+2026-10-20 15:54:33,724 INFO [rank 0] step 1520/2064 | loss 0.1315 | lr 8.31e-05 | 48401 tok/s | 9.1h elapsed | ~3.3h remaining
+2026-10-20 16:01:44,312 INFO [rank 0] step 1540/2064 | loss 0.1572 | lr 7.95e-05 | 48404 tok/s | 9.3h elapsed | ~3.2h remaining
+2026-10-20 16:08:54,953 INFO [rank 0] step 1560/2064 | loss 0.1346 | lr 7.60e-05 | 48408 tok/s | 9.4h elapsed | ~3.0h remaining
+2026-10-20 16:16:05,501 INFO [rank 0] step 1580/2064 | loss 0.1335 | lr 7.26e-05 | 48412 tok/s | 9.5h elapsed | ~2.9h remaining
+2026-10-20 16:23:16,059 INFO [rank 0] step 1600/2064 | loss 0.1271 | lr 6.94e-05 | 48416 tok/s | 9.6h elapsed | ~2.8h remaining
+2026-10-20 16:30:26,563 INFO [rank 0] step 1620/2064 | loss 0.1318 | lr 6.62e-05 | 48419 tok/s | 9.7h elapsed | ~2.7h remaining
+2026-10-20 16:37:37,117 INFO [rank 0] step 1640/2064 | loss 0.1584 | lr 6.32e-05 | 48423 tok/s | 9.9h elapsed | ~2.6h remaining
+2026-10-20 16:44:47,716 INFO [rank 0] step 1660/2064 | loss 0.1393 | lr 6.02e-05 | 48426 tok/s | 10.0h elapsed | ~2.4h remaining
+2026-10-20 16:51:58,296 INFO [rank 0] step 1680/2064 | loss 0.1321 | lr 5.74e-05 | 48430 tok/s | 10.1h elapsed | ~2.3h remaining
+2026-10-20 16:59:08,733 INFO [rank 0] step 1700/2064 | loss 0.1417 | lr 5.47e-05 | 48433 tok/s | 10.2h elapsed | ~2.2h remaining
+2026-10-20 17:06:19,228 INFO [rank 0] step 1720/2064 | loss 0.1593 | lr 5.22e-05 | 48436 tok/s | 10.3h elapsed | ~2.1h remaining
+2026-10-20 17:13:29,600 INFO [rank 0] step 1740/2064 | loss 0.3091 | lr 4.98e-05 | 48440 tok/s | 10.5h elapsed | ~1.9h remaining
+2026-10-20 17:17:40,204 INFO [rank 0] Saved checkpoint at step 1750 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
+2026-10-20 17:21:16,399 INFO [rank 0] step 1760/2064 | loss 1.4265 | lr 4.74e-05 | 48397 tok/s | 10.6h elapsed | ~1.8h remaining
+2026-10-20 17:28:27,952 INFO [rank 0] step 1780/2064 | loss 1.6975 | lr 4.53e-05 | 48399 tok/s | 10.7h elapsed | ~1.7h remaining
+2026-10-20 17:35:39,736 INFO [rank 0] step 1800/2064 | loss 1.6714 | lr 4.32e-05 | 48401 tok/s | 10.8h elapsed | ~1.6h remaining
+2026-10-20 17:42:50,874 INFO [rank 0] step 1820/2064 | loss 1.6400 | lr 4.13e-05 | 48403 tok/s | 11.0h elapsed | ~1.5h remaining
+2026-10-20 17:50:01,531 INFO [rank 0] step 1840/2064 | loss 1.6357 | lr 3.96e-05 | 48407 tok/s | 11.1h elapsed | ~1.3h remaining
+2026-10-20 17:57:12,130 INFO [rank 0] step 1860/2064 | loss 1.6244 | lr 3.80e-05 | 48410 tok/s | 11.2h elapsed | ~1.2h remaining
+2026-10-20 18:04:22,788 INFO [rank 0] step 1880/2064 | loss 1.6206 | lr 3.65e-05 | 48413 tok/s | 11.3h elapsed | ~1.1h remaining
+2026-10-20 18:11:33,436 INFO [rank 0] step 1900/2064 | loss 1.6115 | lr 3.52e-05 | 48416 tok/s | 11.4h elapsed | ~1.0h remaining
+2026-10-20 18:18:44,099 INFO [rank 0] step 1920/2064 | loss 1.6138 | lr 3.40e-05 | 48419 tok/s | 11.6h elapsed | ~0.9h remaining
+2026-10-20 18:25:54,843 INFO [rank 0] step 1940/2064 | loss 1.6075 | lr 3.30e-05 | 48421 tok/s | 11.7h elapsed | ~0.7h remaining
+2026-10-20 18:33:05,293 INFO [rank 0] step 1960/2064 | loss 1.6063 | lr 3.21e-05 | 48424 tok/s | 11.8h elapsed | ~0.6h remaining
+2026-10-20 18:40:15,500 INFO [rank 0] step 1980/2064 | loss 1.6019 | lr 3.14e-05 | 48428 tok/s | 11.9h elapsed | ~0.5h remaining
+2026-10-20 18:47:25,587 INFO [rank 0] step 2000/2064 | loss 1.5923 | lr 3.08e-05 | 48431 tok/s | 12.0h elapsed | ~0.4h remaining
+2026-10-20 18:48:01,534 INFO [rank 0] Saved checkpoint at step 2000 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
+2026-10-20 18:55:12,969 INFO [rank 0] step 2020/2064 | loss 1.5895 | lr 3.04e-05 | 48393 tok/s | 12.2h elapsed | ~0.3h remaining
+2026-10-20 19:02:23,852 INFO [rank 0] step 2040/2064 | loss 1.5956 | lr 3.01e-05 | 48396 tok/s | 12.3h elapsed | ~0.1h remaining
+2026-10-20 19:09:34,788 INFO [rank 0] step 2060/2064 | loss 1.5888 | lr 3.00e-05 | 48398 tok/s | 12.4h elapsed | ~0.0h remaining
+2026-10-20 19:11:36,487 INFO [rank 0] Saved checkpoint at step 2064 -> /workspace/RiemannModel/checkpoints/stage6_12.85b
+2026-10-20 19:11:37,289 INFO [rank 0] Done. stage6_12.85b at step 2064, 2.16B tokens seen, 12.43h wall clock, checkpoint at /workspace/RiemannModel/checkpoints/stage6_12.85b

training_logs/smoke.log

@@ -0,0 +1,134 @@
+W1019 23:14:39.030000 12068 torch/distributed/run.py:803] 
+W1019 23:14:39.030000 12068 torch/distributed/run.py:803] *****************************************
+W1019 23:14:39.030000 12068 torch/distributed/run.py:803] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
+W1019 23:14:39.030000 12068 torch/distributed/run.py:803] *****************************************
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+/opt/amdgpu/share/libdrm/amdgpu.ids: No such file or directory
+2026-10-19 23:14:52,880 INFO [rank 0] Loading stage2_538m from /workspace/RiemannModel/_smoke/checkpoints/stage2_538m
+`torch_dtype` is deprecated! Use `dtype` instead!
+`torch_dtype` is deprecated! Use `dtype` instead!
+`torch_dtype` is deprecated! Use `dtype` instead!
+`torch_dtype` is deprecated! Use `dtype` instead!
+`torch_dtype` is deprecated! Use `dtype` instead!
+`torch_dtype` is deprecated! Use `dtype` instead!
+`torch_dtype` is deprecated! Use `dtype` instead!
+`torch_dtype` is deprecated! Use `dtype` instead!
+
+riemann-climb:12138:12138 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:12138:12138 [0] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+RCCL version : 2.22.3-HEAD:7d8d67c
+HIP version  : 6.4.43484-123eb5128
+ROCm version : 6.4.2.0-120-e7d83f5
+Hostname     : riemann-climb
+Librccl path : /workspace/venv/lib/python3.12/site-packages/torch/lib/librccl.so
+
+riemann-climb:12145:12145 [7] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:12139:12139 [1] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:12144:12144 [6] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:12140:12140 [2] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:12143:12143 [5] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:12142:12142 [4] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+
+riemann-climb:12141:12141 [3] /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/out/rhel-8.8/8.8/build/rccl/hipify/src/init.cc:178 NCCL WARN Missing "iommu=pt" from kernel command line which can lead to system instablity or hang!
+2026-10-19 23:14:58,608 INFO [rank 0] Training stage2_538m: world=8 batch=2 accum=2 seq=128 tokens/step=4096, corpus=7218078 docs, ~264333 steps/epoch, target 6 steps (1.00 epochs), 0.00B params
+2026-10-19 23:15:03,786 INFO [rank 0] step 2/6 | loss 10.8340 | lr 3.00e-06 | 1586 tok/s | 0.0h elapsed | ~0.0h remaining
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py:675: FutureWarning: FSDP.state_dict_type() and FSDP.set_state_dict_type() are being deprecated. Please use APIs, get_state_dict() and set_state_dict(), which can support different parallelisms, FSDP1, FSDP2, DDP. API doc: https://pytorch.org/docs/stable/distributed.checkpoint.html#torch.distributed.checkpoint.state_dict.get_state_dict .Tutorial: https://pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html .
+  warnings.warn(
+2026-10-19 23:15:03,843 INFO [rank 0] Saved checkpoint at step 3 -> /workspace/RiemannModel/_smoke/checkpoints/stage2_538m
+/workspace/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py:4876: UserWarning: barrier(): using the device under current context. You can specify `device_id` in `init_process_group` to mute this warning.
+  warnings.warn(  # warn only once
+2026-10-19 23:15:03,866 INFO [rank 0] step 4/6 | loss 10.8301 | lr 6.00e-06 | 3122 tok/s | 0.0h elapsed | ~0.0h remaining
+2026-10-19 23:15:03,906 INFO [rank 0] step 6/6 | loss 10.8164 | lr 9.00e-06 | 4650 tok/s | 0.0h elapsed | ~0.0h remaining
+2026-10-19 23:15:03,940 INFO [rank 0] Saved checkpoint at step 6 -> /workspace/RiemannModel/_smoke/checkpoints/stage2_538m
+2026-10-19 23:15:03,975 INFO [rank 0] Saved checkpoint at step 6 -> /workspace/RiemannModel/_smoke/checkpoints/stage2_538m
+2026-10-19 23:15:03,975 INFO [rank 0] Done. stage2_538m at step 6, 0.00B tokens seen, 0.00h wall clock, checkpoint at /workspace/RiemannModel/_smoke/checkpoints/stage2_538m