EPFL ALiBi and training provenance 1101-1150 of 1184

epfl_final/train_logs/p1_controls_nrb35_seed42_43/cache_gpu0.log

@@ -0,0 +1,28 @@
+[static] gpu=0 layer=0 seed=3234570 start
+[static] gpu=0 layer=0 done
+[static] gpu=0 layer=4 seed=7234582 start
+[static] gpu=0 layer=4 done
+[static] gpu=0 layer=8 seed=11234594 start
+[static] gpu=0 layer=8 done
+[static] gpu=0 layer=12 seed=15234606 start
+[static] gpu=0 layer=12 done
+[static] gpu=0 layer=16 seed=19234618 start
+[static] gpu=0 layer=16 done
+[static] gpu=0 layer=20 seed=23234630 start
+[static] gpu=0 layer=20 done
+[static] gpu=0 layer=24 seed=27234642 start
+[static] gpu=0 layer=24 done
+[resampled] gpu=0 layer=0 seed=1234567 start
+[resampled] gpu=0 layer=0 done
+[resampled] gpu=0 layer=4 seed=5234579 start
+[resampled] gpu=0 layer=4 done
+[resampled] gpu=0 layer=8 seed=9234591 start
+[resampled] gpu=0 layer=8 done
+[resampled] gpu=0 layer=12 seed=13234603 start
+[resampled] gpu=0 layer=12 done
+[resampled] gpu=0 layer=16 seed=17234615 start
+[resampled] gpu=0 layer=16 done
+[resampled] gpu=0 layer=20 seed=21234627 start
+[resampled] gpu=0 layer=20 done
+[resampled] gpu=0 layer=24 seed=25234639 start
+[resampled] gpu=0 layer=24 done

epfl_final/train_logs/p1_controls_nrb35_seed42_43/cache_gpu1.log

@@ -0,0 +1,28 @@
+[static] gpu=1 layer=1 seed=4234573 start
+[static] gpu=1 layer=1 done
+[static] gpu=1 layer=5 seed=8234585 start
+[static] gpu=1 layer=5 done
+[static] gpu=1 layer=9 seed=12234597 start
+[static] gpu=1 layer=9 done
+[static] gpu=1 layer=13 seed=16234609 start
+[static] gpu=1 layer=13 done
+[static] gpu=1 layer=17 seed=20234621 start
+[static] gpu=1 layer=17 done
+[static] gpu=1 layer=21 seed=24234633 start
+[static] gpu=1 layer=21 done
+[static] gpu=1 layer=25 seed=28234645 start
+[static] gpu=1 layer=25 done
+[resampled] gpu=1 layer=1 seed=2234570 start
+[resampled] gpu=1 layer=1 done
+[resampled] gpu=1 layer=5 seed=6234582 start
+[resampled] gpu=1 layer=5 done
+[resampled] gpu=1 layer=9 seed=10234594 start
+[resampled] gpu=1 layer=9 done
+[resampled] gpu=1 layer=13 seed=14234606 start
+[resampled] gpu=1 layer=13 done
+[resampled] gpu=1 layer=17 seed=18234618 start
+[resampled] gpu=1 layer=17 done
+[resampled] gpu=1 layer=21 seed=22234630 start
+[resampled] gpu=1 layer=21 done
+[resampled] gpu=1 layer=25 seed=26234642 start
+[resampled] gpu=1 layer=25 done

epfl_final/train_logs/p1_controls_nrb35_seed42_43/cache_gpu2.log

@@ -0,0 +1,28 @@
+[static] gpu=2 layer=2 seed=5234576 start
+[static] gpu=2 layer=2 done
+[static] gpu=2 layer=6 seed=9234588 start
+[static] gpu=2 layer=6 done
+[static] gpu=2 layer=10 seed=13234600 start
+[static] gpu=2 layer=10 done
+[static] gpu=2 layer=14 seed=17234612 start
+[static] gpu=2 layer=14 done
+[static] gpu=2 layer=18 seed=21234624 start
+[static] gpu=2 layer=18 done
+[static] gpu=2 layer=22 seed=25234636 start
+[static] gpu=2 layer=22 done
+[static] gpu=2 layer=26 seed=29234648 start
+[static] gpu=2 layer=26 done
+[resampled] gpu=2 layer=2 seed=3234573 start
+[resampled] gpu=2 layer=2 done
+[resampled] gpu=2 layer=6 seed=7234585 start
+[resampled] gpu=2 layer=6 done
+[resampled] gpu=2 layer=10 seed=11234597 start
+[resampled] gpu=2 layer=10 done
+[resampled] gpu=2 layer=14 seed=15234609 start
+[resampled] gpu=2 layer=14 done
+[resampled] gpu=2 layer=18 seed=19234621 start
+[resampled] gpu=2 layer=18 done
+[resampled] gpu=2 layer=22 seed=23234633 start
+[resampled] gpu=2 layer=22 done
+[resampled] gpu=2 layer=26 seed=27234645 start
+[resampled] gpu=2 layer=26 done

epfl_final/train_logs/p1_controls_nrb35_seed42_43/cache_gpu3.log

@@ -0,0 +1,28 @@
+[static] gpu=3 layer=3 seed=6234579 start
+[static] gpu=3 layer=3 done
+[static] gpu=3 layer=7 seed=10234591 start
+[static] gpu=3 layer=7 done
+[static] gpu=3 layer=11 seed=14234603 start
+[static] gpu=3 layer=11 done
+[static] gpu=3 layer=15 seed=18234615 start
+[static] gpu=3 layer=15 done
+[static] gpu=3 layer=19 seed=22234627 start
+[static] gpu=3 layer=19 done
+[static] gpu=3 layer=23 seed=26234639 start
+[static] gpu=3 layer=23 done
+[static] gpu=3 layer=27 seed=30234651 start
+[static] gpu=3 layer=27 done
+[resampled] gpu=3 layer=3 seed=4234576 start
+[resampled] gpu=3 layer=3 done
+[resampled] gpu=3 layer=7 seed=8234588 start
+[resampled] gpu=3 layer=7 done
+[resampled] gpu=3 layer=11 seed=12234600 start
+[resampled] gpu=3 layer=11 done
+[resampled] gpu=3 layer=15 seed=16234612 start
+[resampled] gpu=3 layer=15 done
+[resampled] gpu=3 layer=19 seed=20234624 start
+[resampled] gpu=3 layer=19 done
+[resampled] gpu=3 layer=23 seed=24234636 start
+[resampled] gpu=3 layer=23 done
+[resampled] gpu=3 layer=27 seed=28234648 start
+[resampled] gpu=3 layer=27 done

epfl_final/train_logs/p1_controls_nrb35_seed42_43/supervisor.log

@@ -0,0 +1,9 @@
+[supervisor] cache build started 2026-08-21T15:27:26Z
+validated static: 28 layers, pool=1
+validated resampled: 28 layers, pool=10000
+[supervisor] caches validated; launching training 2026-08-21T16:44:07Z
+[supervisor] launched gpu=0 pid=14850 name=l8k365m_static_nrb35_seed42
+[supervisor] launched gpu=1 pid=14851 name=l8k365m_resampled_nrb35_seed42
+[supervisor] launched gpu=2 pid=14852 name=l8k365m_static_nrb35_seed43
+[supervisor] launched gpu=3 pid=14853 name=l8k365m_resampled_nrb35_seed43
+[supervisor] handoff complete 2026-08-21T16:44:07Z

epfl_final/train_logs/p1_controls_nrb35_seed44_47/supervisor.log

@@ -0,0 +1,3 @@
+[supervisor] launching seed=44 static on gpu0 2026-08-21T18:08:18Z
+[supervisor] launching seed=44 resampled on gpu1 2026-08-21T18:08:18Z
+[supervisor] waiting for seed=44 pids=657,659

epfl_final/train_logs/p1_resume_h1004_20260822/supervisor.log

@@ -0,0 +1,9 @@
+[supervisor] launching gpu=0 name=l8k365m_static_nrb35_seed42 resume 2026-08-22T04:21:20Z
+[supervisor] pid=245
+[supervisor] launching gpu=1 name=l8k365m_resampled_nrb35_seed42 resume 2026-08-22T04:21:20Z
+[supervisor] pid=247
+[supervisor] launching gpu=2 name=l8k365m_static_nrb35_seed43 resume 2026-08-22T04:21:20Z
+[supervisor] pid=249
+[supervisor] launching gpu=3 name=l8k365m_resampled_nrb35_seed43 resume 2026-08-22T04:21:20Z
+[supervisor] pid=251
+[supervisor] all four resumes launched 2026-08-22T04:21:20Z

epfl_final/train_logs/p1_scheduled_a100_seed47_20260822/scheduler.log

@@ -0,0 +1,49 @@
+[scheduler] waiting for static seed47 to finish 2026-08-22T12:46:16Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T12:46:16Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T12:47:16Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T12:48:16Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T12:49:16Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T12:50:16Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T12:51:17Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T12:52:17Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T12:53:17Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T12:54:17Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T12:55:17Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T12:56:17Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T12:57:17Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T12:58:17Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T12:59:17Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:00:17Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:01:18Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:02:18Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:03:18Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:04:18Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:05:18Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:06:18Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:07:18Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:08:18Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:09:18Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:10:18Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:11:18Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:12:19Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:13:19Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:14:19Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:15:19Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:16:19Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:17:19Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:18:19Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:19:19Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:20:19Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:21:20Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:22:20Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:23:20Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:24:20Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:25:20Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:26:20Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:27:20Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:28:20Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:29:20Z
+[scheduler] active=0 busy_gpus=1 2026-08-22T13:30:20Z
+[scheduler] active=0 busy_gpus=0 2026-08-22T13:31:21Z
+[scheduler] condition met; launching resampled seed47 on all five GPUs 2026-08-22T13:31:21Z
+[scheduler] training exited rc=0 2026-08-22T22:30:28Z

epfl_final/train_logs/p1_scheduled_h1002_20260822/scheduler.log

@@ -0,0 +1,432 @@
+[scheduler] waiting for current H1002 runs to finish 2026-08-22T04:41:01Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T04:41:02Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T04:42:02Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T04:43:02Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T04:44:02Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T04:45:02Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T04:46:02Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T04:47:02Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T04:48:02Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T04:49:02Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T04:50:02Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T04:51:02Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T04:52:02Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T04:53:02Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T04:54:02Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T04:55:02Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T04:56:02Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T04:57:02Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T04:58:02Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T04:59:02Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:00:02Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:01:03Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:02:03Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:03:03Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:04:03Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:05:03Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:06:03Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:07:03Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:08:03Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:09:03Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:10:03Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:11:03Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:12:03Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:13:03Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:14:03Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:15:03Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:16:03Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:17:04Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:18:04Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:19:04Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:20:04Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:21:04Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:22:04Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:23:04Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:24:04Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:25:04Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:26:04Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:27:04Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:28:04Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:29:04Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:30:04Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:31:04Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:32:04Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:33:04Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:34:04Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:35:04Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:36:04Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:37:05Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:38:05Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:39:05Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:40:05Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:41:05Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:42:05Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:43:05Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:44:05Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:45:05Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:46:05Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:47:05Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:48:05Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:49:05Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:50:05Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:51:05Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:52:05Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:53:05Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:54:05Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:55:06Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:56:06Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:57:06Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:58:06Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T05:59:06Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:00:06Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:01:06Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:02:06Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:03:06Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:04:06Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:05:06Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:06:06Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:07:06Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:08:06Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:09:06Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:10:07Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:11:07Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:12:07Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:13:07Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:14:07Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:15:07Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:16:07Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:17:07Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:18:07Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:19:07Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:20:07Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:21:07Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:22:07Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:23:07Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:24:07Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:25:07Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:26:07Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:27:07Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:28:08Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:29:08Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:30:08Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:31:08Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:32:08Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:33:08Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:34:08Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:35:08Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:36:08Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:37:08Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:38:08Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:39:08Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:40:08Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:41:08Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:42:08Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:43:08Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:44:08Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:45:09Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:46:09Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:47:09Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:48:09Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:49:09Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:50:09Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:51:09Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:52:09Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:53:09Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:54:09Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:55:09Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:56:09Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:57:09Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:58:09Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T06:59:09Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:00:09Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:01:09Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:02:09Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:03:09Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:04:10Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:05:10Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:06:10Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:07:10Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:08:10Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:09:10Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:10:10Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:11:10Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:12:10Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:13:10Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:14:10Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:15:10Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:16:10Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:17:10Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:18:10Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:19:10Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:20:10Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:21:10Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:22:10Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:23:10Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:24:11Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:25:11Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:26:11Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:27:11Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:28:11Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:29:11Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:30:11Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:31:11Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:32:11Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:33:11Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:34:11Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:35:11Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:36:11Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:37:11Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:38:11Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:39:11Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:40:11Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:41:11Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:42:11Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:43:11Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:44:12Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:45:12Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:46:12Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:47:12Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:48:12Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:49:12Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:50:12Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:51:12Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:52:12Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:53:12Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:54:12Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:55:12Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:56:12Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:57:12Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:58:12Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T07:59:12Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:00:12Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:01:13Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:02:13Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:03:13Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:04:13Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:05:13Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:06:13Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:07:13Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:08:13Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:09:13Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:10:13Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:11:13Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:12:13Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:13:13Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:14:13Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:15:13Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:16:13Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:17:13Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:18:14Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:19:14Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:20:14Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:21:14Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:22:14Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:23:14Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:24:14Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:25:14Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:26:14Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:27:14Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:28:14Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:29:14Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:30:14Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:31:14Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:32:14Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:33:14Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:34:14Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:35:15Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:36:15Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:37:15Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:38:15Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:39:15Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:40:15Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:41:15Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:42:15Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:43:15Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:44:15Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:45:15Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:46:15Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:47:15Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:48:15Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:49:15Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:50:15Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:51:15Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:52:15Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:53:15Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:54:16Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:55:16Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:56:16Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:57:16Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:58:16Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T08:59:16Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:00:16Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:01:16Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:02:16Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:03:16Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:04:16Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:05:16Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:06:16Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:07:16Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:08:16Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:09:16Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:10:17Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:11:17Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:12:17Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:13:17Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:14:17Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:15:17Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:16:17Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:17:17Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:18:17Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:19:17Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:20:17Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:21:17Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:22:17Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:23:17Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:24:17Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:25:17Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:26:17Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:27:17Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:28:17Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:29:17Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:30:17Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:31:18Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:32:18Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:33:18Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:34:18Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:35:18Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:36:18Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:37:18Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:38:18Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:39:18Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:40:18Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:41:18Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:42:18Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:43:18Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:44:18Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:45:18Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:46:18Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:47:18Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:48:19Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:49:19Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:50:19Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:51:19Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:52:19Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:53:19Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:54:19Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:55:19Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:56:19Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:57:19Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:58:19Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T09:59:19Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:00:19Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:01:19Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:02:19Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:03:19Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:04:20Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:05:20Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:06:20Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:07:20Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:08:20Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:09:20Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:10:20Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:11:20Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:12:20Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:13:20Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:14:20Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:15:20Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:16:20Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:17:20Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:18:20Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:19:20Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:20:20Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:21:20Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:22:21Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:23:21Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:24:21Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:25:21Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:26:21Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:27:21Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:28:21Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:29:21Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:30:21Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:31:21Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:32:21Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:33:21Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:34:21Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:35:21Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:36:21Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:37:21Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:38:21Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:39:21Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:40:21Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:41:21Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:42:21Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:43:22Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:44:22Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:45:22Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:46:22Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:47:22Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:48:22Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:49:22Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:50:22Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:51:22Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:52:22Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:53:22Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:54:22Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:55:22Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:56:22Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:57:22Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:58:22Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T10:59:22Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:00:23Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:01:23Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:02:23Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:03:23Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:04:23Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:05:23Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:06:23Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:07:23Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:08:23Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:09:23Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:10:23Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:11:23Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:12:23Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:13:23Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:14:23Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:15:23Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:16:23Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:17:23Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:18:24Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:19:24Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:20:24Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:21:24Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:22:24Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:23:24Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:24:24Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:25:24Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:26:24Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:27:24Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:28:24Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:29:24Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:30:24Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:31:24Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:32:24Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:33:24Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:34:24Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:35:24Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:36:24Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:37:25Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:38:25Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:39:25Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:40:25Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:41:25Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:42:25Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:43:25Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:44:25Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:45:25Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:46:25Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:47:25Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:48:25Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T11:49:25Z
+[scheduler] active=0 busy_gpus=0 2026-08-22T11:50:25Z
+[scheduler] condition met; launching resampled seed46 on both GPUs 2026-08-22T11:50:25Z

epfl_final/train_logs/p1_scheduled_h1004_20260822/scheduler.log

@@ -0,0 +1,595 @@
+[scheduler] waiting for all four H1004 GPUs to become free 2026-08-22T04:40:03Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T04:40:03Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T04:41:03Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T04:42:03Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T04:43:03Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T04:44:03Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T04:45:04Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T04:46:04Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T04:47:04Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T04:48:04Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T04:49:04Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T04:50:04Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T04:51:04Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T04:52:04Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T04:53:04Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T04:54:04Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T04:55:04Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T04:56:04Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T04:57:04Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T04:58:04Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T04:59:05Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:00:05Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:01:05Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:02:05Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:03:05Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:04:05Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:05:05Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:06:05Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:07:05Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:08:05Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:09:05Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:10:05Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:11:05Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:12:05Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:13:06Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:14:06Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:15:06Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:16:06Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:17:06Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:18:06Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:19:06Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:20:06Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:21:06Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:22:06Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:23:06Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:24:06Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:25:06Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:26:07Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:27:07Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:28:07Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:29:07Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:30:07Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:31:07Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:32:07Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:33:07Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:34:07Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:35:07Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:36:07Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:37:07Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:38:07Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:39:07Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:40:08Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:41:08Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:42:08Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:43:08Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:44:08Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:45:08Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:46:08Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:47:08Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:48:08Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:49:08Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:50:08Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:51:08Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:52:08Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:53:09Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:54:09Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:55:09Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:56:09Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:57:09Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:58:09Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T05:59:09Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:00:09Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:01:09Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:02:09Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:03:09Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:04:09Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:05:09Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:06:09Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:07:10Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:08:10Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:09:10Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:10:10Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:11:10Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:12:10Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:13:10Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:14:10Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:15:10Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:16:10Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:17:10Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:18:10Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:19:10Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:20:11Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:21:11Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:22:11Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:23:11Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:24:11Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:25:11Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:26:11Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:27:11Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:28:11Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:29:11Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:30:11Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:31:11Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:32:11Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:33:12Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:34:12Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:35:12Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:36:12Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:37:12Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:38:12Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:39:12Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:40:12Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:41:12Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:42:12Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:43:12Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:44:12Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:45:12Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:46:12Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:47:13Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:48:13Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:49:13Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:50:13Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:51:13Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:52:13Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:53:13Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:54:13Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:55:13Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:56:13Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:57:13Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:58:13Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T06:59:13Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:00:13Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:01:14Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:02:14Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:03:14Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:04:14Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:05:14Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:06:14Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:07:14Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:08:14Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:09:14Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:10:14Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:11:14Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:12:14Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:13:14Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:14:15Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:15:15Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:16:15Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:17:15Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:18:15Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:19:15Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:20:15Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:21:15Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:22:15Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:23:15Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:24:15Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:25:15Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:26:15Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:27:15Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:28:16Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:29:16Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:30:16Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:31:16Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:32:16Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:33:16Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:34:16Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:35:16Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:36:16Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:37:16Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:38:16Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:39:16Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:40:16Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:41:16Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:42:17Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:43:17Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:44:17Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:45:17Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:46:17Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:47:17Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:48:17Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:49:17Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:50:17Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:51:17Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:52:17Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:53:17Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:54:17Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:55:18Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:56:18Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:57:18Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:58:18Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T07:59:18Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:00:18Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:01:18Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:02:18Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:03:18Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:04:18Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:05:18Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:06:18Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:07:18Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:08:18Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:09:19Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:10:19Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:11:19Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:12:19Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:13:19Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:14:19Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:15:19Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:16:19Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:17:19Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:18:19Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:19:19Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:20:19Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:21:19Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:22:19Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:23:20Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:24:20Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:25:20Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:26:20Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:27:20Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:28:20Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:29:20Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:30:20Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:31:20Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:32:20Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:33:20Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:34:20Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:35:21Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:36:21Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:37:21Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:38:21Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:39:21Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:40:21Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:41:21Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:42:21Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:43:21Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:44:21Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:45:21Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:46:21Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:47:21Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:48:21Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:49:22Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:50:22Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:51:22Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:52:22Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:53:22Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:54:22Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:55:22Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:56:22Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:57:22Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:58:22Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T08:59:22Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:00:22Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:01:22Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:02:22Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:03:23Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:04:23Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:05:23Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:06:23Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:07:23Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:08:23Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:09:23Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:10:23Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:11:23Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:12:23Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:13:23Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:14:23Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:15:23Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:16:23Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:17:24Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:18:24Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:19:24Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:20:24Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:21:24Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:22:24Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:23:24Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:24:24Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:25:24Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:26:24Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:27:24Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:28:24Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:29:24Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:30:25Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:31:25Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:32:25Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:33:25Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:34:25Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:35:25Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:36:25Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:37:25Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:38:25Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:39:25Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:40:25Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:41:25Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:42:25Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:43:25Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:44:26Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:45:26Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:46:26Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:47:26Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:48:26Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:49:26Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:50:26Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:51:26Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:52:26Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:53:26Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:54:26Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:55:26Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:56:26Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:57:26Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:58:27Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T09:59:27Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:00:27Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:01:27Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:02:27Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:03:27Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:04:27Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:05:27Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:06:27Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:07:27Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:08:27Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:09:27Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:10:27Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:11:28Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:12:28Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:13:28Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:14:28Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:15:28Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:16:28Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:17:28Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:18:28Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:19:28Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:20:28Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:21:28Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:22:28Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:23:28Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:24:28Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:25:29Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:26:29Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:27:29Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:28:29Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:29:29Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:30:29Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:31:29Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:32:29Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:33:29Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:34:29Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:35:29Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:36:29Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:37:29Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:38:29Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:39:30Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:40:30Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:41:30Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:42:30Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:43:30Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:44:30Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:45:30Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:46:30Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:47:30Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:48:30Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:49:30Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:50:30Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:51:30Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:52:31Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:53:31Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:54:31Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:55:31Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:56:31Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:57:31Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:58:31Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T10:59:31Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:00:31Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:01:31Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:02:31Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:03:31Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:04:31Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:05:31Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:06:32Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:07:32Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:08:32Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:09:32Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:10:32Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:11:32Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:12:32Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:13:32Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:14:32Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:15:32Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:16:32Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:17:32Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:18:32Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:19:32Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:20:33Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:21:33Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:22:33Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:23:33Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:24:33Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:25:33Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:26:33Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:27:33Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:28:33Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:29:33Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:30:33Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:31:33Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:32:33Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:33:34Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:34:34Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:35:34Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:36:34Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:37:34Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:38:34Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:39:34Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:40:34Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:41:34Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:42:34Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:43:34Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:44:34Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:45:34Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:46:34Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:47:35Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:48:35Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:49:35Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:50:35Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:51:35Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:52:35Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:53:35Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:54:35Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:55:35Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:56:35Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:57:35Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:58:35Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T11:59:35Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:00:36Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:01:36Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:02:36Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:03:36Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:04:36Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:05:36Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:06:36Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:07:36Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:08:36Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:09:36Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:10:36Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:11:36Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:12:36Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:13:36Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:14:37Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:15:37Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:16:37Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:17:37Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:18:37Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:19:37Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:20:37Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:21:37Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:22:37Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:23:37Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:24:37Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:25:37Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:26:37Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:27:37Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:28:38Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:29:38Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:30:38Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:31:38Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:32:38Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:33:38Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:34:38Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:35:38Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:36:38Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:37:38Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:38:38Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:39:38Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:40:38Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:41:39Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:42:39Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:43:39Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:44:39Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:45:39Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:46:39Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:47:39Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:48:39Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:49:39Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:50:39Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:51:39Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:52:39Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:53:39Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:54:39Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:55:40Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:56:40Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:57:40Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:58:40Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T12:59:40Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:00:40Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:01:40Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:02:40Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:03:40Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:04:40Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:05:40Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:06:40Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:07:40Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:08:40Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:09:41Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:10:41Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:11:41Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:12:41Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:13:41Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:14:41Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:15:41Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:16:41Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:17:41Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:18:41Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:19:41Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:20:41Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:21:41Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:22:41Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:23:42Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:24:42Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:25:42Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:26:42Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:27:42Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:28:42Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:29:42Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:30:42Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:31:42Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:32:42Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:33:42Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:34:42Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:35:42Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:36:43Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:37:43Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:38:43Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:39:43Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:40:43Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:41:43Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:42:43Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:43:43Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:44:43Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:45:43Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:46:43Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:47:43Z
+[scheduler] active=68 busy_gpus=1 2026-08-22T13:48:43Z
+[scheduler] active=51 busy_gpus=1 2026-08-22T13:49:43Z
+[scheduler] active=51 busy_gpus=1 2026-08-22T13:50:44Z
+[scheduler] active=51 busy_gpus=1 2026-08-22T13:51:44Z
+[scheduler] active=51 busy_gpus=1 2026-08-22T13:52:44Z
+[scheduler] active=51 busy_gpus=1 2026-08-22T13:53:44Z
+[scheduler] active=35 busy_gpus=1 2026-08-22T13:54:44Z
+[scheduler] active=34 busy_gpus=1 2026-08-22T13:55:44Z
+[scheduler] active=34 busy_gpus=1 2026-08-22T13:56:44Z
+[scheduler] active=34 busy_gpus=1 2026-08-22T13:57:44Z
+[scheduler] active=34 busy_gpus=1 2026-08-22T13:58:44Z
+[scheduler] active=34 busy_gpus=1 2026-08-22T13:59:44Z
+[scheduler] active=34 busy_gpus=1 2026-08-22T14:00:44Z
+[scheduler] active=34 busy_gpus=1 2026-08-22T14:01:44Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:02:44Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:03:44Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:04:45Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:05:45Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:06:45Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:07:45Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:08:45Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:09:45Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:10:45Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:11:45Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:12:45Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:13:45Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:14:45Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:15:45Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:16:45Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:17:45Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:18:46Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:19:46Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:20:46Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:21:46Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:22:46Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:23:46Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:24:46Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:25:46Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:26:46Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:27:46Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:28:46Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:29:46Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:30:46Z
+[scheduler] active=17 busy_gpus=1 2026-08-22T14:31:46Z
+[scheduler] active=0 busy_gpus=0 2026-08-22T14:32:47Z
+[scheduler] condition met; launching resampled seed45 on all four GPUs 2026-08-22T14:32:47Z

epfl_final/train_logs/ropectl_a1p0_a100_ddp2/seed42.20260820T140226Z.log

@@ -0,0 +1,847 @@
+W0820 14:02:40.078000 229 site-packages/torch/distributed/run.py:851] 
+W0820 14:02:40.078000 229 site-packages/torch/distributed/run.py:851] *****************************************
+W0820 14:02:40.078000 229 site-packages/torch/distributed/run.py:851] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
+W0820 14:02:40.078000 229 site-packages/torch/distributed/run.py:851] *****************************************
+Auto-resuming from latest checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-17000
+Resuming: Using existing output directory /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735
+Running config:
+{
+  "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
+  "model_dtype": "bfloat16",
+  "init_from_scratch": true,
+  "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
+  "val_path": null,
+  "max_seq_length": 8192,
+  "num_workers": 8,
+  "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735",
+  "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-17000",
+  "force_new_run": false,
+  "num_epochs": 1,
+  "max_steps": 44000,
+  "per_device_batch_size": 2,
+  "gradient_accumulation_steps": 5,
+  "learning_rate": 0.0005,
+  "weight_decay": 0.01,
+  "warmup_steps": 750,
+  "lr_scheduler": "cosine",
+  "max_grad_norm": 1.0,
+  "save_steps": 1000,
+  "eval_steps": 1000,
+  "eval_batches": 50,
+  "logging_steps": 10,
+  "seed": 42,
+  "verbose": true,
+  "checkpoint_activations": false,
+  "strategy": "ddp",
+  "mixed_precision": "bf16",
+  "wandb_enabled": true,
+  "wandb_project": "stochastic_sparsity",
+  "wandb_name": "l8k365m_dense_ropectl_a1p0_seed42",
+  "wandb_tags": [
+    "qwen3",
+    "365m",
+    "8k",
+    "dense",
+    "rope-control",
+    "ddp2-a100"
+  ],
+  "hidden_size": 768,
+  "intermediate_size": 3072,
+  "num_attention_heads": 6,
+  "num_key_value_heads": 2,
+  "num_hidden_layers": 28,
+  "head_dim": null,
+  "attention_dropout": 0.0,
+  "rope_train_position_scale_set": [
+    1.0,
+    0.9896339488176222,
+    0.9692159692159692,
+    0.9420289855072463,
+    0.8730158730158729
+  ],
+  "attention_type": "sparse_stochastic",
+  "sparse_block_size": 128,
+  "sparse_num_random_blocks": 31,
+  "sparse_mask_pool_size": 1,
+  "sparse_val_dense_compare": false,
+  "sparse_layer_indices": null,
+  "sparse_mask_cache_dir": null,
+  "sparse_mask_seed": 0,
+  "sparse_mixed_nrb_set": null,
+  "dense_mode": true,
+  "dense_step_intervals": null
+}
+Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
+Checkpoint activations: False
+Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-17000
+Checkpoint activations: False
+dense_mode=True: sparse attention layers will use standard causal SDPA.
+
+
+================================================================================
+MODEL ARCHITECTURE
+================================================================================
+Model config:
+  vocab_size: 151936
+  hidden_size: 768
+  num_hidden_layers: 28
+  num_attention_heads: 6
+  num_key_value_heads: 2
+  head_dim: 128
+  intermediate_size: 3072
+  max_position_embeddings: 32768
+  rope_theta: 1000000
+  rope_train_position_scale_set: [1.0, 0.9896339488176222, 0.9692159692159692, 0.9420289855072463, 0.8730158730158729]
+  attention_type: sparse_stochastic
+  sparse_block_size: 128
+  sparse_num_random_blocks: 31
+  sparse_mask_pool_size: 1
+  sparse_max_seq_length: 8192
+
+Parameter count:
+  Total parameters: 358,958,848
+  Trainable parameters: 358,958,848
+  Non-trainable parameters: 0
+  Model size (BF16): 0.72 GB
+
+--------------------------------------------------------------------------------
+LAYER-BY-LAYER SUMMARY
+--------------------------------------------------------------------------------
+================================================================================================================================================================
+Layer (type:depth-idx)                                       Input Shape               Output Shape              Param #                   Trainable
+================================================================================================================================================================
+Qwen3ForCausalLM                                             [2, 8192]                 --                        --                        True
+├─Qwen3Model: 1-1                                            --                        [2, 8192, 768]            --                        True
+│    └─Embedding: 2-1                                        [2, 8192]                 [2, 8192, 768]            116,686,848               True
+│    └─ModuleList: 2-2                                       --                        --                        --                        True
+│    │    └─Qwen3DecoderLayer: 3-1                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-2                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-3                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-4                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-5                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-6                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-7                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-8                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-9                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-10                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-11                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-12                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-13                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-14                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-15                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-16                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-17                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-18                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-19                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-20                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-21                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-22                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-23                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-24                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-25                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-26                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-27                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-28                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    └─Qwen3RMSNorm: 2-3                                     [2, 8192, 768]            [2, 8192, 768]            768                       True
+├─Linear: 1-2                                                [2, 8192, 768]            [2, 8192, 151936]         116,686,848               True
+================================================================================================================================================================
+Total params: 475,645,696
+Trainable params: 475,645,696
+Non-trainable params: 0
+Total mult-adds (Units.MEGABYTES): 951.29
+================================================================================================================================================================
+Input size (MB): 0.13
+Forward/backward pass size (MB): 31197.23
+Params size (MB): 951.29
+Estimated Total Size (MB): 32148.66
+================================================================================================================================================================
+  NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
+================================================================================
+
+Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
+Using pre-tokenized dataset (no runtime tokenization)
+Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
+
+Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
+Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
+Dataset split: 3027764 train, 159356 validation
+Training examples (packed, train split):  3027764
+Validation examples (packed, val split):  159356
+Per-device batch size:                    2
+World size (GPUs):                        2
+Gradient accumulation steps:              5
+Effective batch size (seqs/step):         20
+Sequence length:                          8192
+Tokens per optimizer step:                163,840
+Micro-batches per epoch (per rank):       756941
+Optimizer steps per epoch:                151388
+Total optimizer steps:                    44000  (capped by max_steps; 0.29 epochs)
+Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-17000...
+Preserving checkpoint LR state for exact resume: current=[0.0003451463250063146], schedule_base=[0.0005]
+Resumed from checkpoint at step 17000 (epoch 0, micro_step 170000)
+Resuming training from step 17000
+Resuming wandb run jqstnvej
+wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from WANDB_API_KEY.
+wandb: Currently logged in as: ruben-navasardyan4869 (ruben-navasardyan4869-epfl) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
+wandb: setting up run jqstnvej
+wandb: Tracking run with wandb version 0.26.1
+wandb: Run data is saved locally in /mloscratch/homes/navasard/sparse/SparseAtt/wandb/run-20260820_140401-jqstnvej
+wandb: Run `wandb offline` to turn off syncing.
+wandb: Resuming run l8k365m_dense_ropectl_a1p0_seed42
+wandb: ⭐️ View project at https://wandb.ai/ruben-navasardyan4869-epfl/stochastic_sparsity
+wandb: 🚀 View run at https://wandb.ai/ruben-navasardyan4869-epfl/stochastic_sparsity/runs/jqstnvej
+Starting training...
+Validation at step 17000: Loss = 3.0074, Perplexity = 20.23
+wandb: WARNING Tried to log to step 17000 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Switching attention mode for step 17001: dense (dense step interval)
+Step 17010/44000 | Loss: 2.9953 | LR: 3.45e-04 | 42298.8ms/step | 0.5 samples/s
+wandb: WARNING Tried to log to step 17010 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17020/44000 | Loss: 3.0000 | LR: 3.45e-04 | 1764.7ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17020 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17030/44000 | Loss: 2.9989 | LR: 3.45e-04 | 1765.3ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17030 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17040/44000 | Loss: 2.9807 | LR: 3.44e-04 | 1767.9ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17040 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17050/44000 | Loss: 2.9804 | LR: 3.44e-04 | 1767.1ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17050 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17060/44000 | Loss: 3.0394 | LR: 3.44e-04 | 1766.2ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17060 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17070/44000 | Loss: 3.0309 | LR: 3.44e-04 | 1768.2ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17070 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17080/44000 | Loss: 3.0213 | LR: 3.44e-04 | 1768.4ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17080 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17090/44000 | Loss: 3.0357 | LR: 3.44e-04 | 1769.2ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17090 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17100/44000 | Loss: 2.9922 | LR: 3.43e-04 | 1769.5ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17100 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17110/44000 | Loss: 2.9951 | LR: 3.43e-04 | 1769.3ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17110 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17120/44000 | Loss: 2.9471 | LR: 3.43e-04 | 1768.7ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17120 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17130/44000 | Loss: 2.9969 | LR: 3.43e-04 | 1767.5ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17130 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17140/44000 | Loss: 2.9777 | LR: 3.43e-04 | 1767.6ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17140 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17150/44000 | Loss: 2.9867 | LR: 3.43e-04 | 1768.0ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17150 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17160/44000 | Loss: 2.9971 | LR: 3.42e-04 | 1768.9ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17160 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17170/44000 | Loss: 2.9694 | LR: 3.42e-04 | 1768.6ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17170 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17180/44000 | Loss: 2.9811 | LR: 3.42e-04 | 1768.6ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17180 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17190/44000 | Loss: 3.0168 | LR: 3.42e-04 | 1768.7ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17190 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17200/44000 | Loss: 2.9846 | LR: 3.42e-04 | 1769.9ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17200 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17210/44000 | Loss: 2.9550 | LR: 3.42e-04 | 1768.8ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17210 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17220/44000 | Loss: 2.9769 | LR: 3.41e-04 | 1769.4ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17220 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17230/44000 | Loss: 3.0017 | LR: 3.41e-04 | 1769.2ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17230 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17240/44000 | Loss: 2.9902 | LR: 3.41e-04 | 1769.0ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17240 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17250/44000 | Loss: 3.0014 | LR: 3.41e-04 | 1768.4ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17250 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17260/44000 | Loss: 3.0338 | LR: 3.41e-04 | 1769.7ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17260 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17270/44000 | Loss: 3.0320 | LR: 3.41e-04 | 1769.6ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17270 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17280/44000 | Loss: 2.9712 | LR: 3.40e-04 | 1769.3ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17280 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17290/44000 | Loss: 3.0376 | LR: 3.40e-04 | 1770.7ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17290 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17300/44000 | Loss: 3.0024 | LR: 3.40e-04 | 1770.3ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17300 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17310/44000 | Loss: 2.9896 | LR: 3.40e-04 | 1770.0ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17310 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17320/44000 | Loss: 3.0202 | LR: 3.40e-04 | 1772.8ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17320 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17330/44000 | Loss: 2.9877 | LR: 3.40e-04 | 1769.8ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17330 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17340/44000 | Loss: 3.0100 | LR: 3.39e-04 | 1768.7ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17340 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17350/44000 | Loss: 2.9876 | LR: 3.39e-04 | 1768.4ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17350 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17360/44000 | Loss: 2.9364 | LR: 3.39e-04 | 1768.5ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17360 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17370/44000 | Loss: 3.0425 | LR: 3.39e-04 | 1767.9ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17370 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17380/44000 | Loss: 2.9622 | LR: 3.39e-04 | 1768.4ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17380 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17390/44000 | Loss: 3.0031 | LR: 3.39e-04 | 1767.7ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17390 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17400/44000 | Loss: 2.9896 | LR: 3.38e-04 | 1768.1ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17400 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17410/44000 | Loss: 2.9699 | LR: 3.38e-04 | 1769.6ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17410 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17420/44000 | Loss: 2.9919 | LR: 3.38e-04 | 1769.2ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17420 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17430/44000 | Loss: 2.9849 | LR: 3.38e-04 | 1767.6ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17430 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17440/44000 | Loss: 2.9565 | LR: 3.38e-04 | 1767.9ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17440 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17450/44000 | Loss: 3.0010 | LR: 3.38e-04 | 1767.9ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17450 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17460/44000 | Loss: 2.9859 | LR: 3.37e-04 | 1768.2ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17460 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17470/44000 | Loss: 2.9710 | LR: 3.37e-04 | 1769.0ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 17470 that is less than the current step 17471. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 17480/44000 | Loss: 2.9962 | LR: 3.37e-04 | 1769.1ms/step | 11.3 samples/s
+Step 17490/44000 | Loss: 3.0029 | LR: 3.37e-04 | 1770.1ms/step | 11.3 samples/s
+Step 17500/44000 | Loss: 2.9515 | LR: 3.37e-04 | 1769.6ms/step | 11.3 samples/s
+Step 17510/44000 | Loss: 2.9864 | LR: 3.37e-04 | 1769.0ms/step | 11.3 samples/s
+Step 17520/44000 | Loss: 3.0140 | LR: 3.36e-04 | 1769.8ms/step | 11.3 samples/s
+Step 17530/44000 | Loss: 2.9341 | LR: 3.36e-04 | 1769.9ms/step | 11.3 samples/s
+Step 17540/44000 | Loss: 2.9562 | LR: 3.36e-04 | 1768.0ms/step | 11.3 samples/s
+Step 17550/44000 | Loss: 2.9230 | LR: 3.36e-04 | 1768.3ms/step | 11.3 samples/s
+Step 17560/44000 | Loss: 2.9470 | LR: 3.36e-04 | 1768.6ms/step | 11.3 samples/s
+Step 17570/44000 | Loss: 2.9793 | LR: 3.35e-04 | 1768.6ms/step | 11.3 samples/s
+Step 17580/44000 | Loss: 2.9684 | LR: 3.35e-04 | 1769.8ms/step | 11.3 samples/s
+Step 17590/44000 | Loss: 3.0173 | LR: 3.35e-04 | 1771.3ms/step | 11.3 samples/s
+Step 17600/44000 | Loss: 3.0286 | LR: 3.35e-04 | 1770.7ms/step | 11.3 samples/s
+Step 17610/44000 | Loss: 3.0003 | LR: 3.35e-04 | 1769.6ms/step | 11.3 samples/s
+Step 17620/44000 | Loss: 2.9908 | LR: 3.35e-04 | 1768.5ms/step | 11.3 samples/s
+Step 17630/44000 | Loss: 3.0167 | LR: 3.34e-04 | 1770.4ms/step | 11.3 samples/s
+Step 17640/44000 | Loss: 3.0106 | LR: 3.34e-04 | 1769.1ms/step | 11.3 samples/s
+Step 17650/44000 | Loss: 2.9579 | LR: 3.34e-04 | 1768.7ms/step | 11.3 samples/s
+Step 17660/44000 | Loss: 2.9935 | LR: 3.34e-04 | 1769.7ms/step | 11.3 samples/s
+Step 17670/44000 | Loss: 3.0074 | LR: 3.34e-04 | 1770.4ms/step | 11.3 samples/s
+Step 17680/44000 | Loss: 2.9855 | LR: 3.34e-04 | 1769.7ms/step | 11.3 samples/s
+Step 17690/44000 | Loss: 2.9490 | LR: 3.33e-04 | 1769.4ms/step | 11.3 samples/s
+Step 17700/44000 | Loss: 3.0226 | LR: 3.33e-04 | 1769.8ms/step | 11.3 samples/s
+Step 17710/44000 | Loss: 3.0278 | LR: 3.33e-04 | 1772.0ms/step | 11.3 samples/s
+Step 17720/44000 | Loss: 2.9615 | LR: 3.33e-04 | 1770.4ms/step | 11.3 samples/s
+Step 17730/44000 | Loss: 3.0084 | LR: 3.33e-04 | 1771.7ms/step | 11.3 samples/s
+Step 17740/44000 | Loss: 2.9796 | LR: 3.33e-04 | 1769.3ms/step | 11.3 samples/s
+Step 17750/44000 | Loss: 2.9873 | LR: 3.32e-04 | 1770.7ms/step | 11.3 samples/s
+Step 17760/44000 | Loss: 2.9708 | LR: 3.32e-04 | 1773.3ms/step | 11.3 samples/s
+Step 17770/44000 | Loss: 2.9688 | LR: 3.32e-04 | 1772.8ms/step | 11.3 samples/s
+Step 17780/44000 | Loss: 2.9390 | LR: 3.32e-04 | 1774.8ms/step | 11.3 samples/s
+Step 17790/44000 | Loss: 2.9552 | LR: 3.32e-04 | 1770.7ms/step | 11.3 samples/s
+Step 17800/44000 | Loss: 3.0171 | LR: 3.32e-04 | 1769.5ms/step | 11.3 samples/s
+Step 17810/44000 | Loss: 3.0365 | LR: 3.31e-04 | 1771.1ms/step | 11.3 samples/s
+Step 17820/44000 | Loss: 3.0011 | LR: 3.31e-04 | 1770.9ms/step | 11.3 samples/s
+Step 17830/44000 | Loss: 3.0162 | LR: 3.31e-04 | 1768.5ms/step | 11.3 samples/s
+Step 17840/44000 | Loss: 2.9913 | LR: 3.31e-04 | 1772.6ms/step | 11.3 samples/s
+Step 17850/44000 | Loss: 2.9622 | LR: 3.31e-04 | 1772.4ms/step | 11.3 samples/s
+Step 17860/44000 | Loss: 3.0152 | LR: 3.31e-04 | 1770.3ms/step | 11.3 samples/s
+Step 17870/44000 | Loss: 2.9732 | LR: 3.30e-04 | 1769.4ms/step | 11.3 samples/s
+Step 17880/44000 | Loss: 2.9904 | LR: 3.30e-04 | 1768.9ms/step | 11.3 samples/s
+Step 17890/44000 | Loss: 2.9925 | LR: 3.30e-04 | 1770.4ms/step | 11.3 samples/s
+Step 17900/44000 | Loss: 2.9819 | LR: 3.30e-04 | 1769.9ms/step | 11.3 samples/s
+Step 17910/44000 | Loss: 2.9529 | LR: 3.30e-04 | 1770.6ms/step | 11.3 samples/s
+Step 17920/44000 | Loss: 3.0158 | LR: 3.29e-04 | 1769.9ms/step | 11.3 samples/s
+Step 17930/44000 | Loss: 2.9927 | LR: 3.29e-04 | 1770.5ms/step | 11.3 samples/s
+Step 17940/44000 | Loss: 2.9965 | LR: 3.29e-04 | 1769.9ms/step | 11.3 samples/s
+Step 17950/44000 | Loss: 2.9844 | LR: 3.29e-04 | 1767.6ms/step | 11.3 samples/s
+Step 17960/44000 | Loss: 2.9831 | LR: 3.29e-04 | 1769.0ms/step | 11.3 samples/s
+Step 17970/44000 | Loss: 2.9568 | LR: 3.29e-04 | 1770.8ms/step | 11.3 samples/s
+Step 17980/44000 | Loss: 2.9801 | LR: 3.28e-04 | 1771.5ms/step | 11.3 samples/s
+Step 17990/44000 | Loss: 2.9367 | LR: 3.28e-04 | 1771.4ms/step | 11.3 samples/s
+Step 18000/44000 | Loss: 3.0193 | LR: 3.28e-04 | 1769.5ms/step | 11.3 samples/s
+Validation at step 18000: Loss = 2.9956, Perplexity = 20.00
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-18000
+Step 18010/44000 | Loss: 2.9897 | LR: 3.28e-04 | 3629.0ms/step | 5.5 samples/s
+Step 18020/44000 | Loss: 2.9957 | LR: 3.28e-04 | 1768.1ms/step | 11.3 samples/s
+Step 18030/44000 | Loss: 2.9941 | LR: 3.28e-04 | 1769.8ms/step | 11.3 samples/s
+Step 18040/44000 | Loss: 2.9955 | LR: 3.27e-04 | 1772.3ms/step | 11.3 samples/s
+Step 18050/44000 | Loss: 3.0196 | LR: 3.27e-04 | 1769.6ms/step | 11.3 samples/s
+Step 18060/44000 | Loss: 2.9550 | LR: 3.27e-04 | 1768.9ms/step | 11.3 samples/s
+Step 18070/44000 | Loss: 2.9920 | LR: 3.27e-04 | 1770.0ms/step | 11.3 samples/s
+Step 18080/44000 | Loss: 2.9590 | LR: 3.27e-04 | 1771.4ms/step | 11.3 samples/s
+Step 18090/44000 | Loss: 2.9606 | LR: 3.27e-04 | 1770.9ms/step | 11.3 samples/s
+Step 18100/44000 | Loss: 2.9588 | LR: 3.26e-04 | 1771.8ms/step | 11.3 samples/s
+Step 18110/44000 | Loss: 2.9402 | LR: 3.26e-04 | 1772.2ms/step | 11.3 samples/s
+Step 18120/44000 | Loss: 3.0048 | LR: 3.26e-04 | 1772.8ms/step | 11.3 samples/s
+Step 18130/44000 | Loss: 2.9780 | LR: 3.26e-04 | 1773.0ms/step | 11.3 samples/s
+Step 18140/44000 | Loss: 2.9664 | LR: 3.26e-04 | 1769.4ms/step | 11.3 samples/s
+Step 18150/44000 | Loss: 2.9470 | LR: 3.26e-04 | 1769.9ms/step | 11.3 samples/s
+Step 18160/44000 | Loss: 2.9995 | LR: 3.25e-04 | 1770.8ms/step | 11.3 samples/s
+Step 18170/44000 | Loss: 2.9699 | LR: 3.25e-04 | 1769.8ms/step | 11.3 samples/s
+Step 18180/44000 | Loss: 2.9892 | LR: 3.25e-04 | 1769.9ms/step | 11.3 samples/s
+Step 18190/44000 | Loss: 2.9809 | LR: 3.25e-04 | 1772.5ms/step | 11.3 samples/s
+Step 18200/44000 | Loss: 2.9980 | LR: 3.25e-04 | 1771.5ms/step | 11.3 samples/s
+Step 18210/44000 | Loss: 2.9697 | LR: 3.24e-04 | 1771.8ms/step | 11.3 samples/s
+Step 18220/44000 | Loss: 2.9545 | LR: 3.24e-04 | 1770.8ms/step | 11.3 samples/s
+Step 18230/44000 | Loss: 2.9375 | LR: 3.24e-04 | 1770.8ms/step | 11.3 samples/s
+Step 18240/44000 | Loss: 2.9661 | LR: 3.24e-04 | 1772.6ms/step | 11.3 samples/s
+Step 18250/44000 | Loss: 2.9907 | LR: 3.24e-04 | 1771.9ms/step | 11.3 samples/s
+Step 18260/44000 | Loss: 2.9708 | LR: 3.24e-04 | 1772.0ms/step | 11.3 samples/s
+Step 18270/44000 | Loss: 2.9797 | LR: 3.23e-04 | 1771.2ms/step | 11.3 samples/s
+Step 18280/44000 | Loss: 2.9829 | LR: 3.23e-04 | 1771.5ms/step | 11.3 samples/s
+Step 18290/44000 | Loss: 2.9986 | LR: 3.23e-04 | 1770.7ms/step | 11.3 samples/s
+Step 18300/44000 | Loss: 2.9809 | LR: 3.23e-04 | 1771.9ms/step | 11.3 samples/s
+Step 18310/44000 | Loss: 2.9801 | LR: 3.23e-04 | 1772.4ms/step | 11.3 samples/s
+Step 18320/44000 | Loss: 2.9896 | LR: 3.23e-04 | 1772.6ms/step | 11.3 samples/s
+Step 18330/44000 | Loss: 2.9907 | LR: 3.22e-04 | 1772.7ms/step | 11.3 samples/s
+Step 18340/44000 | Loss: 3.0383 | LR: 3.22e-04 | 1770.9ms/step | 11.3 samples/s
+Step 18350/44000 | Loss: 2.9696 | LR: 3.22e-04 | 1769.9ms/step | 11.3 samples/s
+Step 18360/44000 | Loss: 2.9360 | LR: 3.22e-04 | 1770.6ms/step | 11.3 samples/s
+Step 18370/44000 | Loss: 2.9669 | LR: 3.22e-04 | 1768.4ms/step | 11.3 samples/s
+Step 18380/44000 | Loss: 2.9547 | LR: 3.22e-04 | 1769.3ms/step | 11.3 samples/s
+Step 18390/44000 | Loss: 2.9836 | LR: 3.21e-04 | 1769.7ms/step | 11.3 samples/s
+Step 18400/44000 | Loss: 2.9594 | LR: 3.21e-04 | 1770.6ms/step | 11.3 samples/s
+Step 18410/44000 | Loss: 2.9419 | LR: 3.21e-04 | 1771.6ms/step | 11.3 samples/s
+Step 18420/44000 | Loss: 3.0278 | LR: 3.21e-04 | 1769.8ms/step | 11.3 samples/s
+Step 18430/44000 | Loss: 3.0356 | LR: 3.21e-04 | 1770.7ms/step | 11.3 samples/s
+Step 18440/44000 | Loss: 2.9890 | LR: 3.20e-04 | 1770.9ms/step | 11.3 samples/s
+Step 18450/44000 | Loss: 2.9739 | LR: 3.20e-04 | 1770.5ms/step | 11.3 samples/s
+Step 18460/44000 | Loss: 2.9957 | LR: 3.20e-04 | 1772.4ms/step | 11.3 samples/s
+Step 18470/44000 | Loss: 2.9845 | LR: 3.20e-04 | 1770.9ms/step | 11.3 samples/s
+Step 18480/44000 | Loss: 2.9698 | LR: 3.20e-04 | 1771.4ms/step | 11.3 samples/s
+Step 18490/44000 | Loss: 2.9566 | LR: 3.20e-04 | 1770.4ms/step | 11.3 samples/s
+Step 18500/44000 | Loss: 2.9605 | LR: 3.19e-04 | 1771.3ms/step | 11.3 samples/s
+Step 18510/44000 | Loss: 2.9494 | LR: 3.19e-04 | 1771.1ms/step | 11.3 samples/s
+Step 18520/44000 | Loss: 2.9192 | LR: 3.19e-04 | 1770.7ms/step | 11.3 samples/s
+Step 18530/44000 | Loss: 2.9784 | LR: 3.19e-04 | 1770.9ms/step | 11.3 samples/s
+Step 18540/44000 | Loss: 2.9874 | LR: 3.19e-04 | 1771.8ms/step | 11.3 samples/s
+Step 18550/44000 | Loss: 2.9844 | LR: 3.19e-04 | 1769.3ms/step | 11.3 samples/s
+Step 18560/44000 | Loss: 2.9353 | LR: 3.18e-04 | 1769.7ms/step | 11.3 samples/s
+Step 18570/44000 | Loss: 2.9711 | LR: 3.18e-04 | 1770.9ms/step | 11.3 samples/s
+Step 18580/44000 | Loss: 3.0260 | LR: 3.18e-04 | 1770.8ms/step | 11.3 samples/s
+Step 18590/44000 | Loss: 2.9768 | LR: 3.18e-04 | 1772.7ms/step | 11.3 samples/s
+Step 18600/44000 | Loss: 2.9523 | LR: 3.18e-04 | 1770.3ms/step | 11.3 samples/s
+Step 18610/44000 | Loss: 3.0193 | LR: 3.18e-04 | 1771.4ms/step | 11.3 samples/s
+Step 18620/44000 | Loss: 2.9781 | LR: 3.17e-04 | 1769.3ms/step | 11.3 samples/s
+Step 18630/44000 | Loss: 2.9890 | LR: 3.17e-04 | 1770.0ms/step | 11.3 samples/s
+Step 18640/44000 | Loss: 2.9644 | LR: 3.17e-04 | 1770.3ms/step | 11.3 samples/s
+Step 18650/44000 | Loss: 3.0036 | LR: 3.17e-04 | 1771.1ms/step | 11.3 samples/s
+Step 18660/44000 | Loss: 2.9546 | LR: 3.17e-04 | 1769.6ms/step | 11.3 samples/s
+Step 18670/44000 | Loss: 2.9598 | LR: 3.16e-04 | 1771.4ms/step | 11.3 samples/s
+Step 18680/44000 | Loss: 3.0120 | LR: 3.16e-04 | 1770.7ms/step | 11.3 samples/s
+Step 18690/44000 | Loss: 2.9864 | LR: 3.16e-04 | 1769.8ms/step | 11.3 samples/s
+Step 18700/44000 | Loss: 2.9835 | LR: 3.16e-04 | 1770.4ms/step | 11.3 samples/s
+Step 18710/44000 | Loss: 2.9747 | LR: 3.16e-04 | 1770.7ms/step | 11.3 samples/s
+Step 18720/44000 | Loss: 2.9969 | LR: 3.16e-04 | 1771.5ms/step | 11.3 samples/s
+Step 18730/44000 | Loss: 2.9643 | LR: 3.15e-04 | 1771.6ms/step | 11.3 samples/s
+Step 18740/44000 | Loss: 2.9840 | LR: 3.15e-04 | 1771.7ms/step | 11.3 samples/s
+Step 18750/44000 | Loss: 2.9541 | LR: 3.15e-04 | 1771.9ms/step | 11.3 samples/s
+Step 18760/44000 | Loss: 2.9484 | LR: 3.15e-04 | 1768.8ms/step | 11.3 samples/s
+Step 18770/44000 | Loss: 2.9171 | LR: 3.15e-04 | 1770.0ms/step | 11.3 samples/s
+Step 18780/44000 | Loss: 2.9861 | LR: 3.15e-04 | 1769.6ms/step | 11.3 samples/s
+Step 18790/44000 | Loss: 2.9754 | LR: 3.14e-04 | 1769.6ms/step | 11.3 samples/s
+Step 18800/44000 | Loss: 2.9667 | LR: 3.14e-04 | 1770.9ms/step | 11.3 samples/s
+Step 18810/44000 | Loss: 2.9540 | LR: 3.14e-04 | 1770.4ms/step | 11.3 samples/s
+Step 18820/44000 | Loss: 2.9444 | LR: 3.14e-04 | 1771.2ms/step | 11.3 samples/s
+Step 18830/44000 | Loss: 2.9815 | LR: 3.14e-04 | 1770.9ms/step | 11.3 samples/s
+Step 18840/44000 | Loss: 2.9418 | LR: 3.13e-04 | 1770.7ms/step | 11.3 samples/s
+Step 18850/44000 | Loss: 2.9511 | LR: 3.13e-04 | 1770.5ms/step | 11.3 samples/s
+Step 18860/44000 | Loss: 2.9654 | LR: 3.13e-04 | 1770.4ms/step | 11.3 samples/s
+Step 18870/44000 | Loss: 2.9814 | LR: 3.13e-04 | 1770.9ms/step | 11.3 samples/s
+Step 18880/44000 | Loss: 2.9764 | LR: 3.13e-04 | 1769.8ms/step | 11.3 samples/s
+Step 18890/44000 | Loss: 2.9910 | LR: 3.13e-04 | 1769.6ms/step | 11.3 samples/s
+Step 18900/44000 | Loss: 3.0031 | LR: 3.12e-04 | 1770.1ms/step | 11.3 samples/s
+Step 18910/44000 | Loss: 2.9752 | LR: 3.12e-04 | 1771.0ms/step | 11.3 samples/s
+Step 18920/44000 | Loss: 3.0025 | LR: 3.12e-04 | 1770.4ms/step | 11.3 samples/s
+Step 18930/44000 | Loss: 3.0012 | LR: 3.12e-04 | 1771.5ms/step | 11.3 samples/s
+Step 18940/44000 | Loss: 2.9471 | LR: 3.12e-04 | 1771.2ms/step | 11.3 samples/s
+Step 18950/44000 | Loss: 2.9523 | LR: 3.12e-04 | 1772.1ms/step | 11.3 samples/s
+Step 18960/44000 | Loss: 2.9595 | LR: 3.11e-04 | 1769.2ms/step | 11.3 samples/s
+Step 18970/44000 | Loss: 2.9815 | LR: 3.11e-04 | 1770.6ms/step | 11.3 samples/s
+Step 18980/44000 | Loss: 2.9753 | LR: 3.11e-04 | 1769.9ms/step | 11.3 samples/s
+Step 18990/44000 | Loss: 3.0137 | LR: 3.11e-04 | 1769.1ms/step | 11.3 samples/s
+Step 19000/44000 | Loss: 3.0280 | LR: 3.11e-04 | 1769.0ms/step | 11.3 samples/s
+Validation at step 19000: Loss = 2.9844, Perplexity = 19.77
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-19000
+Step 19010/44000 | Loss: 2.9494 | LR: 3.11e-04 | 3549.4ms/step | 5.6 samples/s
+Step 19020/44000 | Loss: 2.9590 | LR: 3.10e-04 | 1768.8ms/step | 11.3 samples/s
+Step 19030/44000 | Loss: 2.9950 | LR: 3.10e-04 | 1771.1ms/step | 11.3 samples/s
+Step 19040/44000 | Loss: 2.9828 | LR: 3.10e-04 | 1770.4ms/step | 11.3 samples/s
+Step 19050/44000 | Loss: 2.9433 | LR: 3.10e-04 | 1771.6ms/step | 11.3 samples/s
+Step 19060/44000 | Loss: 3.0023 | LR: 3.10e-04 | 1771.6ms/step | 11.3 samples/s
+Step 19070/44000 | Loss: 2.9470 | LR: 3.09e-04 | 1771.7ms/step | 11.3 samples/s
+Step 19080/44000 | Loss: 2.9568 | LR: 3.09e-04 | 1771.3ms/step | 11.3 samples/s
+Step 19090/44000 | Loss: 2.9839 | LR: 3.09e-04 | 1773.3ms/step | 11.3 samples/s
+Step 19100/44000 | Loss: 2.9568 | LR: 3.09e-04 | 1772.0ms/step | 11.3 samples/s
+Step 19110/44000 | Loss: 2.9778 | LR: 3.09e-04 | 1771.4ms/step | 11.3 samples/s
+Step 19120/44000 | Loss: 2.9881 | LR: 3.09e-04 | 1771.3ms/step | 11.3 samples/s
+Step 19130/44000 | Loss: 2.9526 | LR: 3.08e-04 | 1772.5ms/step | 11.3 samples/s
+Step 19140/44000 | Loss: 2.9738 | LR: 3.08e-04 | 1771.2ms/step | 11.3 samples/s
+Step 19150/44000 | Loss: 2.9667 | LR: 3.08e-04 | 1772.0ms/step | 11.3 samples/s
+Step 19160/44000 | Loss: 2.9943 | LR: 3.08e-04 | 1769.6ms/step | 11.3 samples/s
+Step 19170/44000 | Loss: 2.9402 | LR: 3.08e-04 | 1770.2ms/step | 11.3 samples/s
+Step 19180/44000 | Loss: 2.9953 | LR: 3.08e-04 | 1771.5ms/step | 11.3 samples/s
+Step 19190/44000 | Loss: 3.0067 | LR: 3.07e-04 | 1771.0ms/step | 11.3 samples/s
+Step 19200/44000 | Loss: 2.9986 | LR: 3.07e-04 | 1770.2ms/step | 11.3 samples/s
+Step 19210/44000 | Loss: 2.9489 | LR: 3.07e-04 | 1770.3ms/step | 11.3 samples/s
+Step 19220/44000 | Loss: 2.9948 | LR: 3.07e-04 | 1770.9ms/step | 11.3 samples/s
+Step 19230/44000 | Loss: 3.0039 | LR: 3.07e-04 | 1768.9ms/step | 11.3 samples/s
+Step 19240/44000 | Loss: 2.9916 | LR: 3.06e-04 | 1770.4ms/step | 11.3 samples/s
+Step 19250/44000 | Loss: 2.9411 | LR: 3.06e-04 | 1770.0ms/step | 11.3 samples/s
+Step 19260/44000 | Loss: 2.9537 | LR: 3.06e-04 | 1770.7ms/step | 11.3 samples/s
+Step 19270/44000 | Loss: 2.9491 | LR: 3.06e-04 | 1770.5ms/step | 11.3 samples/s
+Step 19280/44000 | Loss: 2.9620 | LR: 3.06e-04 | 1771.3ms/step | 11.3 samples/s
+Step 19290/44000 | Loss: 2.9807 | LR: 3.06e-04 | 1770.6ms/step | 11.3 samples/s
+Step 19300/44000 | Loss: 2.9847 | LR: 3.05e-04 | 1770.7ms/step | 11.3 samples/s
+Step 19310/44000 | Loss: 2.9413 | LR: 3.05e-04 | 1772.0ms/step | 11.3 samples/s
+Step 19320/44000 | Loss: 3.0060 | LR: 3.05e-04 | 1772.5ms/step | 11.3 samples/s
+Step 19330/44000 | Loss: 2.9936 | LR: 3.05e-04 | 1772.4ms/step | 11.3 samples/s
+Step 19340/44000 | Loss: 2.9193 | LR: 3.05e-04 | 1771.9ms/step | 11.3 samples/s
+Step 19350/44000 | Loss: 3.0031 | LR: 3.04e-04 | 1771.0ms/step | 11.3 samples/s
+Step 19360/44000 | Loss: 2.9996 | LR: 3.04e-04 | 1770.7ms/step | 11.3 samples/s
+Step 19370/44000 | Loss: 2.9783 | LR: 3.04e-04 | 1769.4ms/step | 11.3 samples/s
+Step 19380/44000 | Loss: 2.9958 | LR: 3.04e-04 | 1769.1ms/step | 11.3 samples/s
+Step 19390/44000 | Loss: 2.9684 | LR: 3.04e-04 | 1770.7ms/step | 11.3 samples/s
+Step 19400/44000 | Loss: 2.9573 | LR: 3.04e-04 | 1771.1ms/step | 11.3 samples/s
+Step 19410/44000 | Loss: 2.9659 | LR: 3.03e-04 | 1771.0ms/step | 11.3 samples/s
+Step 19420/44000 | Loss: 2.9652 | LR: 3.03e-04 | 1771.9ms/step | 11.3 samples/s
+Step 19430/44000 | Loss: 2.9947 | LR: 3.03e-04 | 1771.3ms/step | 11.3 samples/s
+Step 19440/44000 | Loss: 2.9503 | LR: 3.03e-04 | 1772.2ms/step | 11.3 samples/s
+Step 19450/44000 | Loss: 2.9506 | LR: 3.03e-04 | 1770.6ms/step | 11.3 samples/s
+Step 19460/44000 | Loss: 2.9538 | LR: 3.03e-04 | 1770.8ms/step | 11.3 samples/s
+Step 19470/44000 | Loss: 2.9783 | LR: 3.02e-04 | 1770.5ms/step | 11.3 samples/s
+Step 19480/44000 | Loss: 3.0068 | LR: 3.02e-04 | 1770.4ms/step | 11.3 samples/s
+Step 19490/44000 | Loss: 2.9531 | LR: 3.02e-04 | 1769.5ms/step | 11.3 samples/s
+Step 19500/44000 | Loss: 2.9721 | LR: 3.02e-04 | 1770.5ms/step | 11.3 samples/s
+Step 19510/44000 | Loss: 2.9562 | LR: 3.02e-04 | 1773.0ms/step | 11.3 samples/s
+Step 19520/44000 | Loss: 2.9827 | LR: 3.01e-04 | 1771.8ms/step | 11.3 samples/s
+Step 19530/44000 | Loss: 2.9471 | LR: 3.01e-04 | 1772.0ms/step | 11.3 samples/s
+Step 19540/44000 | Loss: 2.9769 | LR: 3.01e-04 | 1770.6ms/step | 11.3 samples/s
+Step 19550/44000 | Loss: 2.9495 | LR: 3.01e-04 | 1771.2ms/step | 11.3 samples/s
+Step 19560/44000 | Loss: 2.9559 | LR: 3.01e-04 | 1772.4ms/step | 11.3 samples/s
+Step 19570/44000 | Loss: 2.9899 | LR: 3.01e-04 | 1771.6ms/step | 11.3 samples/s
+Step 19580/44000 | Loss: 2.9869 | LR: 3.00e-04 | 1771.4ms/step | 11.3 samples/s
+Step 19590/44000 | Loss: 2.9162 | LR: 3.00e-04 | 1771.4ms/step | 11.3 samples/s
+Step 19600/44000 | Loss: 2.9708 | LR: 3.00e-04 | 1772.5ms/step | 11.3 samples/s
+Step 19610/44000 | Loss: 2.9644 | LR: 3.00e-04 | 1771.6ms/step | 11.3 samples/s
+Step 19620/44000 | Loss: 2.9560 | LR: 3.00e-04 | 1771.7ms/step | 11.3 samples/s
+Step 19630/44000 | Loss: 2.9680 | LR: 3.00e-04 | 1770.0ms/step | 11.3 samples/s
+Step 19640/44000 | Loss: 2.9741 | LR: 2.99e-04 | 1770.5ms/step | 11.3 samples/s
+Step 19650/44000 | Loss: 2.9759 | LR: 2.99e-04 | 1769.5ms/step | 11.3 samples/s
+Step 19660/44000 | Loss: 2.9623 | LR: 2.99e-04 | 1770.7ms/step | 11.3 samples/s
+Step 19670/44000 | Loss: 2.9420 | LR: 2.99e-04 | 1770.9ms/step | 11.3 samples/s
+Step 19680/44000 | Loss: 2.9692 | LR: 2.99e-04 | 1770.8ms/step | 11.3 samples/s
+Step 19690/44000 | Loss: 2.9437 | LR: 2.98e-04 | 1769.9ms/step | 11.3 samples/s
+Step 19700/44000 | Loss: 2.9444 | LR: 2.98e-04 | 1770.1ms/step | 11.3 samples/s
+Step 19710/44000 | Loss: 2.9681 | LR: 2.98e-04 | 1771.0ms/step | 11.3 samples/s
+Step 19720/44000 | Loss: 2.9803 | LR: 2.98e-04 | 1771.2ms/step | 11.3 samples/s
+Step 19730/44000 | Loss: 2.9701 | LR: 2.98e-04 | 1770.4ms/step | 11.3 samples/s
+Step 19740/44000 | Loss: 2.9696 | LR: 2.98e-04 | 1770.8ms/step | 11.3 samples/s
+Step 19750/44000 | Loss: 2.9536 | LR: 2.97e-04 | 1771.4ms/step | 11.3 samples/s
+Step 19760/44000 | Loss: 2.9481 | LR: 2.97e-04 | 1771.4ms/step | 11.3 samples/s
+Step 19770/44000 | Loss: 2.9471 | LR: 2.97e-04 | 1770.4ms/step | 11.3 samples/s
+Step 19780/44000 | Loss: 2.9547 | LR: 2.97e-04 | 1769.1ms/step | 11.3 samples/s
+Step 19790/44000 | Loss: 2.9828 | LR: 2.97e-04 | 1769.1ms/step | 11.3 samples/s
+Step 19800/44000 | Loss: 2.9386 | LR: 2.96e-04 | 1769.8ms/step | 11.3 samples/s
+Step 19810/44000 | Loss: 2.9634 | LR: 2.96e-04 | 1771.0ms/step | 11.3 samples/s
+Step 19820/44000 | Loss: 2.9613 | LR: 2.96e-04 | 1769.8ms/step | 11.3 samples/s
+Step 19830/44000 | Loss: 3.0163 | LR: 2.96e-04 | 1770.3ms/step | 11.3 samples/s
+Step 19840/44000 | Loss: 2.9451 | LR: 2.96e-04 | 1770.3ms/step | 11.3 samples/s
+Step 19850/44000 | Loss: 2.9251 | LR: 2.96e-04 | 1772.6ms/step | 11.3 samples/s
+Step 19860/44000 | Loss: 2.9628 | LR: 2.95e-04 | 1771.3ms/step | 11.3 samples/s
+Step 19870/44000 | Loss: 2.9556 | LR: 2.95e-04 | 1771.5ms/step | 11.3 samples/s
+Step 19880/44000 | Loss: 3.0099 | LR: 2.95e-04 | 1770.1ms/step | 11.3 samples/s
+Step 19890/44000 | Loss: 2.9379 | LR: 2.95e-04 | 1776.5ms/step | 11.3 samples/s
+Step 19900/44000 | Loss: 3.0089 | LR: 2.95e-04 | 1773.8ms/step | 11.3 samples/s
+Step 19910/44000 | Loss: 2.9912 | LR: 2.95e-04 | 1770.9ms/step | 11.3 samples/s
+Step 19920/44000 | Loss: 2.9984 | LR: 2.94e-04 | 1772.6ms/step | 11.3 samples/s
+Step 19930/44000 | Loss: 2.9582 | LR: 2.94e-04 | 1771.5ms/step | 11.3 samples/s
+Step 19940/44000 | Loss: 2.9479 | LR: 2.94e-04 | 1772.7ms/step | 11.3 samples/s
+Step 19950/44000 | Loss: 2.9565 | LR: 2.94e-04 | 1772.1ms/step | 11.3 samples/s
+Step 19960/44000 | Loss: 2.9890 | LR: 2.94e-04 | 1773.3ms/step | 11.3 samples/s
+Step 19970/44000 | Loss: 2.9651 | LR: 2.93e-04 | 1772.6ms/step | 11.3 samples/s
+Step 19980/44000 | Loss: 2.9537 | LR: 2.93e-04 | 1770.7ms/step | 11.3 samples/s
+Step 19990/44000 | Loss: 2.9003 | LR: 2.93e-04 | 1770.5ms/step | 11.3 samples/s
+Step 20000/44000 | Loss: 2.9248 | LR: 2.93e-04 | 1772.2ms/step | 11.3 samples/s
+Validation at step 20000: Loss = 2.9722, Perplexity = 19.53
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-20000
+Step 20010/44000 | Loss: 2.9816 | LR: 2.93e-04 | 3625.9ms/step | 5.5 samples/s
+Step 20020/44000 | Loss: 2.9778 | LR: 2.93e-04 | 1770.1ms/step | 11.3 samples/s
+Step 20030/44000 | Loss: 2.9212 | LR: 2.92e-04 | 1770.6ms/step | 11.3 samples/s
+Step 20040/44000 | Loss: 2.9739 | LR: 2.92e-04 | 1770.1ms/step | 11.3 samples/s
+Step 20050/44000 | Loss: 2.9567 | LR: 2.92e-04 | 1770.6ms/step | 11.3 samples/s
+Step 20060/44000 | Loss: 2.9815 | LR: 2.92e-04 | 1769.9ms/step | 11.3 samples/s
+Step 20070/44000 | Loss: 2.9755 | LR: 2.92e-04 | 1769.5ms/step | 11.3 samples/s
+Step 20080/44000 | Loss: 2.9633 | LR: 2.91e-04 | 1770.1ms/step | 11.3 samples/s
+Step 20090/44000 | Loss: 2.9590 | LR: 2.91e-04 | 1771.7ms/step | 11.3 samples/s
+Step 20100/44000 | Loss: 2.9741 | LR: 2.91e-04 | 1769.4ms/step | 11.3 samples/s
+Step 20110/44000 | Loss: 2.9806 | LR: 2.91e-04 | 1770.4ms/step | 11.3 samples/s
+Step 20120/44000 | Loss: 2.9475 | LR: 2.91e-04 | 1770.5ms/step | 11.3 samples/s
+Step 20130/44000 | Loss: 2.9405 | LR: 2.91e-04 | 1771.7ms/step | 11.3 samples/s
+Step 20140/44000 | Loss: 2.9664 | LR: 2.90e-04 | 1772.1ms/step | 11.3 samples/s
+Step 20150/44000 | Loss: 2.9486 | LR: 2.90e-04 | 1771.5ms/step | 11.3 samples/s
+Step 20160/44000 | Loss: 3.0032 | LR: 2.90e-04 | 1772.0ms/step | 11.3 samples/s
+Step 20170/44000 | Loss: 2.9958 | LR: 2.90e-04 | 1771.3ms/step | 11.3 samples/s
+Step 20180/44000 | Loss: 2.9919 | LR: 2.90e-04 | 1768.5ms/step | 11.3 samples/s
+Step 20190/44000 | Loss: 2.9245 | LR: 2.90e-04 | 1768.7ms/step | 11.3 samples/s
+Step 20200/44000 | Loss: 2.9556 | LR: 2.89e-04 | 1769.2ms/step | 11.3 samples/s
+Step 20210/44000 | Loss: 2.9703 | LR: 2.89e-04 | 1769.3ms/step | 11.3 samples/s
+Step 20220/44000 | Loss: 2.9727 | LR: 2.89e-04 | 1772.1ms/step | 11.3 samples/s
+Step 20230/44000 | Loss: 2.9500 | LR: 2.89e-04 | 1771.0ms/step | 11.3 samples/s
+Step 20240/44000 | Loss: 2.9612 | LR: 2.89e-04 | 1771.3ms/step | 11.3 samples/s
+Step 20250/44000 | Loss: 2.9833 | LR: 2.88e-04 | 1771.6ms/step | 11.3 samples/s
+Step 20260/44000 | Loss: 3.0002 | LR: 2.88e-04 | 1770.6ms/step | 11.3 samples/s
+Step 20270/44000 | Loss: 2.9567 | LR: 2.88e-04 | 1770.1ms/step | 11.3 samples/s
+Step 20280/44000 | Loss: 2.9705 | LR: 2.88e-04 | 1770.6ms/step | 11.3 samples/s
+Step 20290/44000 | Loss: 2.9384 | LR: 2.88e-04 | 1770.4ms/step | 11.3 samples/s
+Step 20300/44000 | Loss: 2.9583 | LR: 2.88e-04 | 1770.1ms/step | 11.3 samples/s
+Step 20310/44000 | Loss: 2.9347 | LR: 2.87e-04 | 1770.4ms/step | 11.3 samples/s
+Step 20320/44000 | Loss: 2.9550 | LR: 2.87e-04 | 1769.8ms/step | 11.3 samples/s
+Step 20330/44000 | Loss: 2.9434 | LR: 2.87e-04 | 1770.9ms/step | 11.3 samples/s
+Step 20340/44000 | Loss: 2.9719 | LR: 2.87e-04 | 1771.2ms/step | 11.3 samples/s
+Step 20350/44000 | Loss: 2.9250 | LR: 2.87e-04 | 1772.5ms/step | 11.3 samples/s
+Step 20360/44000 | Loss: 2.9488 | LR: 2.86e-04 | 1771.2ms/step | 11.3 samples/s
+Step 20370/44000 | Loss: 2.9819 | LR: 2.86e-04 | 1771.0ms/step | 11.3 samples/s
+Step 20380/44000 | Loss: 2.9286 | LR: 2.86e-04 | 1769.5ms/step | 11.3 samples/s
+Step 20390/44000 | Loss: 3.0053 | LR: 2.86e-04 | 1771.1ms/step | 11.3 samples/s
+Step 20400/44000 | Loss: 2.9318 | LR: 2.86e-04 | 1769.1ms/step | 11.3 samples/s
+Step 20410/44000 | Loss: 2.9306 | LR: 2.86e-04 | 1768.1ms/step | 11.3 samples/s
+Step 20420/44000 | Loss: 2.9649 | LR: 2.85e-04 | 1767.8ms/step | 11.3 samples/s
+Step 20430/44000 | Loss: 2.9844 | LR: 2.85e-04 | 1769.2ms/step | 11.3 samples/s
+Step 20440/44000 | Loss: 2.9624 | LR: 2.85e-04 | 1769.8ms/step | 11.3 samples/s
+Step 20450/44000 | Loss: 3.0124 | LR: 2.85e-04 | 1769.3ms/step | 11.3 samples/s
+Step 20460/44000 | Loss: 3.0169 | LR: 2.85e-04 | 1769.4ms/step | 11.3 samples/s
+Step 20470/44000 | Loss: 2.9604 | LR: 2.84e-04 | 1770.5ms/step | 11.3 samples/s
+Step 20480/44000 | Loss: 2.9868 | LR: 2.84e-04 | 1771.8ms/step | 11.3 samples/s
+Step 20490/44000 | Loss: 2.9644 | LR: 2.84e-04 | 1770.6ms/step | 11.3 samples/s
+Step 20500/44000 | Loss: 2.9266 | LR: 2.84e-04 | 1771.9ms/step | 11.3 samples/s
+Step 20510/44000 | Loss: 2.9403 | LR: 2.84e-04 | 1771.8ms/step | 11.3 samples/s
+Step 20520/44000 | Loss: 2.9423 | LR: 2.84e-04 | 1771.4ms/step | 11.3 samples/s
+Step 20530/44000 | Loss: 2.9933 | LR: 2.83e-04 | 1770.2ms/step | 11.3 samples/s
+Step 20540/44000 | Loss: 2.9370 | LR: 2.83e-04 | 1770.2ms/step | 11.3 samples/s
+Step 20550/44000 | Loss: 2.9665 | LR: 2.83e-04 | 1770.3ms/step | 11.3 samples/s
+Step 20560/44000 | Loss: 2.9544 | LR: 2.83e-04 | 1770.3ms/step | 11.3 samples/s
+Step 20570/44000 | Loss: 2.9111 | LR: 2.83e-04 | 1772.4ms/step | 11.3 samples/s
+Step 20580/44000 | Loss: 2.9822 | LR: 2.83e-04 | 1771.8ms/step | 11.3 samples/s
+Step 20590/44000 | Loss: 2.9426 | LR: 2.82e-04 | 1770.4ms/step | 11.3 samples/s
+Step 20600/44000 | Loss: 2.9467 | LR: 2.82e-04 | 1771.4ms/step | 11.3 samples/s
+Step 20610/44000 | Loss: 2.9570 | LR: 2.82e-04 | 1769.2ms/step | 11.3 samples/s
+Step 20620/44000 | Loss: 2.9540 | LR: 2.82e-04 | 1770.1ms/step | 11.3 samples/s
+Step 20630/44000 | Loss: 2.9329 | LR: 2.82e-04 | 1769.8ms/step | 11.3 samples/s
+Step 20640/44000 | Loss: 2.9869 | LR: 2.81e-04 | 1769.6ms/step | 11.3 samples/s
+Step 20650/44000 | Loss: 2.9165 | LR: 2.81e-04 | 1770.8ms/step | 11.3 samples/s
+Step 20660/44000 | Loss: 2.9320 | LR: 2.81e-04 | 1770.4ms/step | 11.3 samples/s
+Step 20670/44000 | Loss: 2.9362 | LR: 2.81e-04 | 1771.0ms/step | 11.3 samples/s
+Step 20680/44000 | Loss: 2.9663 | LR: 2.81e-04 | 1770.9ms/step | 11.3 samples/s
+Step 20690/44000 | Loss: 2.8882 | LR: 2.81e-04 | 1770.5ms/step | 11.3 samples/s
+Step 20700/44000 | Loss: 2.9963 | LR: 2.80e-04 | 1769.9ms/step | 11.3 samples/s
+Step 20710/44000 | Loss: 2.9678 | LR: 2.80e-04 | 1771.0ms/step | 11.3 samples/s
+Step 20720/44000 | Loss: 2.9527 | LR: 2.80e-04 | 1771.0ms/step | 11.3 samples/s
+Step 20730/44000 | Loss: 2.9973 | LR: 2.80e-04 | 1771.2ms/step | 11.3 samples/s
+Step 20740/44000 | Loss: 3.0062 | LR: 2.80e-04 | 1770.5ms/step | 11.3 samples/s
+Step 20750/44000 | Loss: 2.9608 | LR: 2.79e-04 | 1772.0ms/step | 11.3 samples/s
+Step 20760/44000 | Loss: 2.9559 | LR: 2.79e-04 | 1771.7ms/step | 11.3 samples/s
+Step 20770/44000 | Loss: 2.9498 | LR: 2.79e-04 | 1772.1ms/step | 11.3 samples/s
+Step 20780/44000 | Loss: 2.9675 | LR: 2.79e-04 | 1771.2ms/step | 11.3 samples/s
+Step 20790/44000 | Loss: 2.9515 | LR: 2.79e-04 | 1769.0ms/step | 11.3 samples/s
+Step 20800/44000 | Loss: 2.9370 | LR: 2.79e-04 | 1769.6ms/step | 11.3 samples/s
+Step 20810/44000 | Loss: 2.9494 | LR: 2.78e-04 | 1770.3ms/step | 11.3 samples/s
+Step 20820/44000 | Loss: 2.9558 | LR: 2.78e-04 | 1770.0ms/step | 11.3 samples/s
+Step 20830/44000 | Loss: 2.9733 | LR: 2.78e-04 | 1770.6ms/step | 11.3 samples/s
+Step 20840/44000 | Loss: 2.9305 | LR: 2.78e-04 | 1770.8ms/step | 11.3 samples/s
+Step 20850/44000 | Loss: 2.9401 | LR: 2.78e-04 | 1773.0ms/step | 11.3 samples/s
+Step 20860/44000 | Loss: 2.9898 | LR: 2.77e-04 | 1770.0ms/step | 11.3 samples/s
+Step 20870/44000 | Loss: 2.9470 | LR: 2.77e-04 | 1771.4ms/step | 11.3 samples/s
+Step 20880/44000 | Loss: 2.9657 | LR: 2.77e-04 | 1772.3ms/step | 11.3 samples/s
+Step 20890/44000 | Loss: 2.9494 | LR: 2.77e-04 | 1770.9ms/step | 11.3 samples/s
+Step 20900/44000 | Loss: 2.9441 | LR: 2.77e-04 | 1771.1ms/step | 11.3 samples/s
+Step 20910/44000 | Loss: 2.9343 | LR: 2.77e-04 | 1771.2ms/step | 11.3 samples/s
+Step 20920/44000 | Loss: 2.9755 | LR: 2.76e-04 | 1771.0ms/step | 11.3 samples/s
+Step 20930/44000 | Loss: 2.9439 | LR: 2.76e-04 | 1771.4ms/step | 11.3 samples/s
+Step 20940/44000 | Loss: 2.9537 | LR: 2.76e-04 | 1770.7ms/step | 11.3 samples/s
+Step 20950/44000 | Loss: 2.9685 | LR: 2.76e-04 | 1772.1ms/step | 11.3 samples/s
+Step 20960/44000 | Loss: 2.9687 | LR: 2.76e-04 | 1772.2ms/step | 11.3 samples/s
+Step 20970/44000 | Loss: 2.9550 | LR: 2.75e-04 | 1772.4ms/step | 11.3 samples/s
+Step 20980/44000 | Loss: 2.9290 | LR: 2.75e-04 | 1773.5ms/step | 11.3 samples/s
+Step 20990/44000 | Loss: 2.9885 | LR: 2.75e-04 | 1770.9ms/step | 11.3 samples/s
+Step 21000/44000 | Loss: 2.9418 | LR: 2.75e-04 | 1769.5ms/step | 11.3 samples/s
+Validation at step 21000: Loss = 2.9617, Perplexity = 19.33
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-21000
+Step 21010/44000 | Loss: 2.9727 | LR: 2.75e-04 | 3547.8ms/step | 5.6 samples/s
+Step 21020/44000 | Loss: 2.9407 | LR: 2.75e-04 | 1769.0ms/step | 11.3 samples/s
+Step 21030/44000 | Loss: 2.9414 | LR: 2.74e-04 | 1770.3ms/step | 11.3 samples/s
+Step 21040/44000 | Loss: 2.9263 | LR: 2.74e-04 | 1771.0ms/step | 11.3 samples/s
+Step 21050/44000 | Loss: 2.9586 | LR: 2.74e-04 | 1770.9ms/step | 11.3 samples/s
+Step 21060/44000 | Loss: 2.9516 | LR: 2.74e-04 | 1769.1ms/step | 11.3 samples/s
+Step 21070/44000 | Loss: 2.8871 | LR: 2.74e-04 | 1769.7ms/step | 11.3 samples/s
+Step 21080/44000 | Loss: 2.9939 | LR: 2.73e-04 | 1770.1ms/step | 11.3 samples/s
+Step 21090/44000 | Loss: 2.9073 | LR: 2.73e-04 | 1770.5ms/step | 11.3 samples/s
+Step 21100/44000 | Loss: 2.9544 | LR: 2.73e-04 | 1771.9ms/step | 11.3 samples/s
+Step 21110/44000 | Loss: 2.9470 | LR: 2.73e-04 | 1771.6ms/step | 11.3 samples/s
+Step 21120/44000 | Loss: 2.9368 | LR: 2.73e-04 | 1769.9ms/step | 11.3 samples/s
+Step 21130/44000 | Loss: 2.9874 | LR: 2.73e-04 | 1770.6ms/step | 11.3 samples/s
+Step 21140/44000 | Loss: 2.8958 | LR: 2.72e-04 | 1771.4ms/step | 11.3 samples/s
+Step 21150/44000 | Loss: 2.9277 | LR: 2.72e-04 | 1770.2ms/step | 11.3 samples/s
+Step 21160/44000 | Loss: 2.9177 | LR: 2.72e-04 | 1772.2ms/step | 11.3 samples/s
+Step 21170/44000 | Loss: 2.9782 | LR: 2.72e-04 | 1772.7ms/step | 11.3 samples/s
+Step 21180/44000 | Loss: 2.9245 | LR: 2.72e-04 | 1773.4ms/step | 11.3 samples/s
+Step 21190/44000 | Loss: 2.9490 | LR: 2.71e-04 | 1769.9ms/step | 11.3 samples/s
+Step 21200/44000 | Loss: 2.9321 | LR: 2.71e-04 | 1770.4ms/step | 11.3 samples/s
+Step 21210/44000 | Loss: 2.9134 | LR: 2.71e-04 | 1770.8ms/step | 11.3 samples/s
+Step 21220/44000 | Loss: 2.9325 | LR: 2.71e-04 | 1771.3ms/step | 11.3 samples/s
+Step 21230/44000 | Loss: 2.9484 | LR: 2.71e-04 | 1772.9ms/step | 11.3 samples/s
+Step 21240/44000 | Loss: 2.9883 | LR: 2.71e-04 | 1771.5ms/step | 11.3 samples/s
+Step 21250/44000 | Loss: 2.9579 | LR: 2.70e-04 | 1771.2ms/step | 11.3 samples/s
+Step 21260/44000 | Loss: 2.9837 | LR: 2.70e-04 | 1770.7ms/step | 11.3 samples/s
+Step 21270/44000 | Loss: 2.9694 | LR: 2.70e-04 | 1770.3ms/step | 11.3 samples/s
+Step 21280/44000 | Loss: 2.9666 | LR: 2.70e-04 | 1771.0ms/step | 11.3 samples/s
+Step 21290/44000 | Loss: 2.9292 | LR: 2.70e-04 | 1771.3ms/step | 11.3 samples/s
+Step 21300/44000 | Loss: 2.9695 | LR: 2.70e-04 | 1771.5ms/step | 11.3 samples/s
+Step 21310/44000 | Loss: 2.9305 | LR: 2.69e-04 | 1771.9ms/step | 11.3 samples/s
+Step 21320/44000 | Loss: 2.9512 | LR: 2.69e-04 | 1772.1ms/step | 11.3 samples/s
+Step 21330/44000 | Loss: 2.9623 | LR: 2.69e-04 | 1772.0ms/step | 11.3 samples/s
+Step 21340/44000 | Loss: 2.9600 | LR: 2.69e-04 | 1772.6ms/step | 11.3 samples/s
+Step 21350/44000 | Loss: 2.9760 | LR: 2.69e-04 | 1773.5ms/step | 11.3 samples/s
+Step 21360/44000 | Loss: 2.9180 | LR: 2.68e-04 | 1771.9ms/step | 11.3 samples/s
+Step 21370/44000 | Loss: 2.9145 | LR: 2.68e-04 | 1772.0ms/step | 11.3 samples/s
+Step 21380/44000 | Loss: 2.9765 | LR: 2.68e-04 | 1772.4ms/step | 11.3 samples/s
+Step 21390/44000 | Loss: 2.9132 | LR: 2.68e-04 | 1771.6ms/step | 11.3 samples/s
+Step 21400/44000 | Loss: 2.9095 | LR: 2.68e-04 | 1769.1ms/step | 11.3 samples/s
+Step 21410/44000 | Loss: 2.9186 | LR: 2.68e-04 | 1768.8ms/step | 11.3 samples/s
+Step 21420/44000 | Loss: 2.9608 | LR: 2.67e-04 | 1769.8ms/step | 11.3 samples/s
+Step 21430/44000 | Loss: 2.9342 | LR: 2.67e-04 | 1770.2ms/step | 11.3 samples/s
+Step 21440/44000 | Loss: 2.9320 | LR: 2.67e-04 | 1770.1ms/step | 11.3 samples/s
+Step 21450/44000 | Loss: 2.9778 | LR: 2.67e-04 | 1769.6ms/step | 11.3 samples/s
+Step 21460/44000 | Loss: 2.9842 | LR: 2.67e-04 | 1769.1ms/step | 11.3 samples/s
+Step 21470/44000 | Loss: 2.9218 | LR: 2.66e-04 | 1769.9ms/step | 11.3 samples/s
+Step 21480/44000 | Loss: 2.9486 | LR: 2.66e-04 | 1769.7ms/step | 11.3 samples/s
+Step 21490/44000 | Loss: 2.9668 | LR: 2.66e-04 | 1769.3ms/step | 11.3 samples/s
+Step 21500/44000 | Loss: 2.9292 | LR: 2.66e-04 | 1770.5ms/step | 11.3 samples/s
+Step 21510/44000 | Loss: 2.9646 | LR: 2.66e-04 | 1770.9ms/step | 11.3 samples/s
+Step 21520/44000 | Loss: 2.9676 | LR: 2.66e-04 | 1770.2ms/step | 11.3 samples/s
+Step 21530/44000 | Loss: 2.9028 | LR: 2.65e-04 | 1771.7ms/step | 11.3 samples/s
+Step 21540/44000 | Loss: 2.9628 | LR: 2.65e-04 | 1771.6ms/step | 11.3 samples/s
+Step 21550/44000 | Loss: 2.9413 | LR: 2.65e-04 | 1771.8ms/step | 11.3 samples/s
+Step 21560/44000 | Loss: 2.9478 | LR: 2.65e-04 | 1771.2ms/step | 11.3 samples/s
+Step 21570/44000 | Loss: 2.9693 | LR: 2.65e-04 | 1771.1ms/step | 11.3 samples/s
+Step 21580/44000 | Loss: 2.9467 | LR: 2.64e-04 | 1772.8ms/step | 11.3 samples/s
+Step 21590/44000 | Loss: 2.9269 | LR: 2.64e-04 | 1773.4ms/step | 11.3 samples/s
+Step 21600/44000 | Loss: 2.9671 | LR: 2.64e-04 | 1771.1ms/step | 11.3 samples/s
+Step 21610/44000 | Loss: 2.9265 | LR: 2.64e-04 | 1771.8ms/step | 11.3 samples/s
+Step 21620/44000 | Loss: 2.9045 | LR: 2.64e-04 | 1770.5ms/step | 11.3 samples/s
+Step 21630/44000 | Loss: 2.9044 | LR: 2.64e-04 | 1771.2ms/step | 11.3 samples/s
+Step 21640/44000 | Loss: 2.9569 | LR: 2.63e-04 | 1771.3ms/step | 11.3 samples/s
+Step 21650/44000 | Loss: 2.9823 | LR: 2.63e-04 | 1773.1ms/step | 11.3 samples/s
+Step 21660/44000 | Loss: 2.9459 | LR: 2.63e-04 | 1773.0ms/step | 11.3 samples/s
+Step 21670/44000 | Loss: 2.9741 | LR: 2.63e-04 | 1772.1ms/step | 11.3 samples/s
+Step 21680/44000 | Loss: 2.9790 | LR: 2.63e-04 | 1770.7ms/step | 11.3 samples/s
+Step 21690/44000 | Loss: 2.9701 | LR: 2.62e-04 | 1771.1ms/step | 11.3 samples/s
+Step 21700/44000 | Loss: 2.9565 | LR: 2.62e-04 | 1771.6ms/step | 11.3 samples/s
+Step 21710/44000 | Loss: 2.9762 | LR: 2.62e-04 | 1770.4ms/step | 11.3 samples/s
+Step 21720/44000 | Loss: 2.9282 | LR: 2.62e-04 | 1770.9ms/step | 11.3 samples/s
+Step 21730/44000 | Loss: 2.9703 | LR: 2.62e-04 | 1772.2ms/step | 11.3 samples/s
+Step 21740/44000 | Loss: 2.9507 | LR: 2.62e-04 | 1771.4ms/step | 11.3 samples/s
+Step 21750/44000 | Loss: 2.9642 | LR: 2.61e-04 | 1770.8ms/step | 11.3 samples/s
+Step 21760/44000 | Loss: 2.9588 | LR: 2.61e-04 | 1772.7ms/step | 11.3 samples/s
+Step 21770/44000 | Loss: 2.8906 | LR: 2.61e-04 | 1773.6ms/step | 11.3 samples/s
+Step 21780/44000 | Loss: 2.9805 | LR: 2.61e-04 | 1772.7ms/step | 11.3 samples/s
+Step 21790/44000 | Loss: 2.8840 | LR: 2.61e-04 | 1773.2ms/step | 11.3 samples/s
+Step 21800/44000 | Loss: 2.9755 | LR: 2.60e-04 | 1772.4ms/step | 11.3 samples/s
+Step 21810/44000 | Loss: 2.9098 | LR: 2.60e-04 | 1770.4ms/step | 11.3 samples/s
+Step 21820/44000 | Loss: 2.9524 | LR: 2.60e-04 | 1771.0ms/step | 11.3 samples/s
+Step 21830/44000 | Loss: 2.9297 | LR: 2.60e-04 | 1770.7ms/step | 11.3 samples/s
+Step 21840/44000 | Loss: 2.9537 | LR: 2.60e-04 | 1769.3ms/step | 11.3 samples/s
+Step 21850/44000 | Loss: 2.9244 | LR: 2.60e-04 | 1768.5ms/step | 11.3 samples/s
+Step 21860/44000 | Loss: 2.9163 | LR: 2.59e-04 | 1771.6ms/step | 11.3 samples/s
+Step 21870/44000 | Loss: 2.9120 | LR: 2.59e-04 | 1771.2ms/step | 11.3 samples/s
+Step 21880/44000 | Loss: 2.9423 | LR: 2.59e-04 | 1771.1ms/step | 11.3 samples/s
+Step 21890/44000 | Loss: 2.9102 | LR: 2.59e-04 | 1773.4ms/step | 11.3 samples/s
+Step 21900/44000 | Loss: 2.9573 | LR: 2.59e-04 | 1774.7ms/step | 11.3 samples/s
+Step 21910/44000 | Loss: 2.9441 | LR: 2.58e-04 | 1772.7ms/step | 11.3 samples/s
+Step 21920/44000 | Loss: 2.9309 | LR: 2.58e-04 | 1770.5ms/step | 11.3 samples/s
+Step 21930/44000 | Loss: 2.9375 | LR: 2.58e-04 | 1770.8ms/step | 11.3 samples/s
+Step 21940/44000 | Loss: 2.9143 | LR: 2.58e-04 | 1771.7ms/step | 11.3 samples/s
+Step 21950/44000 | Loss: 2.9301 | LR: 2.58e-04 | 1771.5ms/step | 11.3 samples/s
+Step 21960/44000 | Loss: 2.9244 | LR: 2.58e-04 | 1772.3ms/step | 11.3 samples/s
+Step 21970/44000 | Loss: 2.9328 | LR: 2.57e-04 | 1772.3ms/step | 11.3 samples/s
+Step 21980/44000 | Loss: 2.9116 | LR: 2.57e-04 | 1772.6ms/step | 11.3 samples/s
+Step 21990/44000 | Loss: 2.9667 | LR: 2.57e-04 | 1773.9ms/step | 11.3 samples/s
+Step 22000/44000 | Loss: 2.9800 | LR: 2.57e-04 | 1772.9ms/step | 11.3 samples/s
+Validation at step 22000: Loss = 2.9528, Perplexity = 19.16
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-22000
+Step 22010/44000 | Loss: 2.9338 | LR: 2.57e-04 | 3560.1ms/step | 5.6 samples/s
+Step 22020/44000 | Loss: 2.8921 | LR: 2.56e-04 | 1771.7ms/step | 11.3 samples/s
+Step 22030/44000 | Loss: 2.9259 | LR: 2.56e-04 | 1772.4ms/step | 11.3 samples/s
+Step 22040/44000 | Loss: 2.9486 | LR: 2.56e-04 | 1772.0ms/step | 11.3 samples/s
+Step 22050/44000 | Loss: 2.9976 | LR: 2.56e-04 | 1771.1ms/step | 11.3 samples/s
+Step 22060/44000 | Loss: 2.9513 | LR: 2.56e-04 | 1771.6ms/step | 11.3 samples/s
+Step 22070/44000 | Loss: 2.9246 | LR: 2.56e-04 | 1772.0ms/step | 11.3 samples/s
+Step 22080/44000 | Loss: 2.9267 | LR: 2.55e-04 | 1773.3ms/step | 11.3 samples/s
+Step 22090/44000 | Loss: 2.9569 | LR: 2.55e-04 | 1772.1ms/step | 11.3 samples/s
+Step 22100/44000 | Loss: 2.9295 | LR: 2.55e-04 | 1774.0ms/step | 11.3 samples/s
+Step 22110/44000 | Loss: 2.9712 | LR: 2.55e-04 | 1773.3ms/step | 11.3 samples/s
+Step 22120/44000 | Loss: 2.9615 | LR: 2.55e-04 | 1772.8ms/step | 11.3 samples/s
+Step 22130/44000 | Loss: 2.9275 | LR: 2.54e-04 | 1771.5ms/step | 11.3 samples/s
+Step 22140/44000 | Loss: 2.9286 | LR: 2.54e-04 | 1771.4ms/step | 11.3 samples/s
+Step 22150/44000 | Loss: 2.9220 | LR: 2.54e-04 | 1771.9ms/step | 11.3 samples/s
+Step 22160/44000 | Loss: 2.9725 | LR: 2.54e-04 | 1772.6ms/step | 11.3 samples/s
+Step 22170/44000 | Loss: 3.0177 | LR: 2.54e-04 | 1773.2ms/step | 11.3 samples/s
+Step 22180/44000 | Loss: 2.9142 | LR: 2.54e-04 | 1773.9ms/step | 11.3 samples/s
+Step 22190/44000 | Loss: 2.9659 | LR: 2.53e-04 | 1773.6ms/step | 11.3 samples/s
+Step 22200/44000 | Loss: 2.9481 | LR: 2.53e-04 | 1775.0ms/step | 11.3 samples/s
+Step 22210/44000 | Loss: 2.9754 | LR: 2.53e-04 | 1771.7ms/step | 11.3 samples/s
+Step 22220/44000 | Loss: 2.8976 | LR: 2.53e-04 | 1772.3ms/step | 11.3 samples/s
+Step 22230/44000 | Loss: 2.9833 | LR: 2.53e-04 | 1773.7ms/step | 11.3 samples/s
+Step 22240/44000 | Loss: 2.9172 | LR: 2.52e-04 | 1772.4ms/step | 11.3 samples/s
+Step 22250/44000 | Loss: 2.9282 | LR: 2.52e-04 | 1772.6ms/step | 11.3 samples/s
+Step 22260/44000 | Loss: 2.9525 | LR: 2.52e-04 | 1772.5ms/step | 11.3 samples/s
+Step 22270/44000 | Loss: 2.9295 | LR: 2.52e-04 | 1772.0ms/step | 11.3 samples/s
+Step 22280/44000 | Loss: 2.9373 | LR: 2.52e-04 | 1772.4ms/step | 11.3 samples/s
+Step 22290/44000 | Loss: 2.8460 | LR: 2.52e-04 | 1772.9ms/step | 11.3 samples/s
+Step 22300/44000 | Loss: 2.8991 | LR: 2.51e-04 | 1770.6ms/step | 11.3 samples/s
+Step 22310/44000 | Loss: 2.9836 | LR: 2.51e-04 | 1771.5ms/step | 11.3 samples/s
+Step 22320/44000 | Loss: 2.9604 | LR: 2.51e-04 | 1772.5ms/step | 11.3 samples/s
+Step 22330/44000 | Loss: 2.8921 | LR: 2.51e-04 | 1771.1ms/step | 11.3 samples/s
+Step 22340/44000 | Loss: 2.9545 | LR: 2.51e-04 | 1770.8ms/step | 11.3 samples/s
+Step 22350/44000 | Loss: 2.9822 | LR: 2.50e-04 | 1771.8ms/step | 11.3 samples/s
+Step 22360/44000 | Loss: 2.9276 | LR: 2.50e-04 | 1771.7ms/step | 11.3 samples/s
+Step 22370/44000 | Loss: 2.9170 | LR: 2.50e-04 | 1772.4ms/step | 11.3 samples/s
+Step 22380/44000 | Loss: 2.9609 | LR: 2.50e-04 | 1773.5ms/step | 11.3 samples/s
+Step 22390/44000 | Loss: 2.9126 | LR: 2.50e-04 | 1773.8ms/step | 11.3 samples/s
+Step 22400/44000 | Loss: 2.9267 | LR: 2.50e-04 | 1773.5ms/step | 11.3 samples/s
+Step 22410/44000 | Loss: 2.9545 | LR: 2.49e-04 | 1773.4ms/step | 11.3 samples/s
+Step 22420/44000 | Loss: 2.9243 | LR: 2.49e-04 | 1771.0ms/step | 11.3 samples/s
+Step 22430/44000 | Loss: 2.9332 | LR: 2.49e-04 | 1771.4ms/step | 11.3 samples/s
+Step 22440/44000 | Loss: 2.9683 | LR: 2.49e-04 | 1771.3ms/step | 11.3 samples/s
+Step 22450/44000 | Loss: 2.9830 | LR: 2.49e-04 | 1770.6ms/step | 11.3 samples/s
+Step 22460/44000 | Loss: 2.8939 | LR: 2.48e-04 | 1770.3ms/step | 11.3 samples/s
+Step 22470/44000 | Loss: 2.9308 | LR: 2.48e-04 | 1770.8ms/step | 11.3 samples/s
+Step 22480/44000 | Loss: 2.9259 | LR: 2.48e-04 | 1771.5ms/step | 11.3 samples/s
+Step 22490/44000 | Loss: 2.9606 | LR: 2.48e-04 | 1774.1ms/step | 11.3 samples/s
+Step 22500/44000 | Loss: 2.9183 | LR: 2.48e-04 | 1773.6ms/step | 11.3 samples/s
+Step 22510/44000 | Loss: 2.9498 | LR: 2.48e-04 | 1771.7ms/step | 11.3 samples/s
+Step 22520/44000 | Loss: 2.9181 | LR: 2.47e-04 | 1773.5ms/step | 11.3 samples/s
+Step 22530/44000 | Loss: 2.9146 | LR: 2.47e-04 | 1772.8ms/step | 11.3 samples/s
+Step 22540/44000 | Loss: 2.9389 | LR: 2.47e-04 | 1773.3ms/step | 11.3 samples/s
+Step 22550/44000 | Loss: 2.9228 | LR: 2.47e-04 | 1773.0ms/step | 11.3 samples/s
+Step 22560/44000 | Loss: 2.9578 | LR: 2.47e-04 | 1772.6ms/step | 11.3 samples/s
+Step 22570/44000 | Loss: 2.9510 | LR: 2.46e-04 | 1773.7ms/step | 11.3 samples/s
+Step 22580/44000 | Loss: 2.9165 | LR: 2.46e-04 | 1772.8ms/step | 11.3 samples/s
+Step 22590/44000 | Loss: 2.9103 | LR: 2.46e-04 | 1774.3ms/step | 11.3 samples/s
+Step 22600/44000 | Loss: 2.9526 | LR: 2.46e-04 | 1773.1ms/step | 11.3 samples/s
+Step 22610/44000 | Loss: 2.8903 | LR: 2.46e-04 | 1774.3ms/step | 11.3 samples/s
+Step 22620/44000 | Loss: 2.9446 | LR: 2.46e-04 | 1772.4ms/step | 11.3 samples/s
+Step 22630/44000 | Loss: 2.9834 | LR: 2.45e-04 | 1771.9ms/step | 11.3 samples/s
+Step 22640/44000 | Loss: 2.9538 | LR: 2.45e-04 | 1771.7ms/step | 11.3 samples/s
+Step 22650/44000 | Loss: 2.9311 | LR: 2.45e-04 | 1772.9ms/step | 11.3 samples/s
+Step 22660/44000 | Loss: 2.9542 | LR: 2.45e-04 | 1771.9ms/step | 11.3 samples/s
+Step 22670/44000 | Loss: 2.9533 | LR: 2.45e-04 | 1771.4ms/step | 11.3 samples/s
+Step 22680/44000 | Loss: 2.8716 | LR: 2.44e-04 | 1771.6ms/step | 11.3 samples/s
+Step 22690/44000 | Loss: 2.9154 | LR: 2.44e-04 | 1772.4ms/step | 11.3 samples/s
+Step 22700/44000 | Loss: 2.9791 | LR: 2.44e-04 | 1772.3ms/step | 11.3 samples/s
+Step 22710/44000 | Loss: 2.9177 | LR: 2.44e-04 | 1772.2ms/step | 11.3 samples/s
+Step 22720/44000 | Loss: 2.9510 | LR: 2.44e-04 | 1773.0ms/step | 11.3 samples/s
+Step 22730/44000 | Loss: 2.9412 | LR: 2.44e-04 | 1772.1ms/step | 11.3 samples/s
+Step 22740/44000 | Loss: 2.9281 | LR: 2.43e-04 | 1772.3ms/step | 11.3 samples/s
+Step 22750/44000 | Loss: 2.8959 | LR: 2.43e-04 | 1772.1ms/step | 11.3 samples/s
+Step 22760/44000 | Loss: 2.9371 | LR: 2.43e-04 | 1772.2ms/step | 11.3 samples/s
+Step 22770/44000 | Loss: 2.9628 | LR: 2.43e-04 | 1771.5ms/step | 11.3 samples/s
+Step 22780/44000 | Loss: 2.9025 | LR: 2.43e-04 | 1770.6ms/step | 11.3 samples/s
+Step 22790/44000 | Loss: 2.9476 | LR: 2.42e-04 | 1771.8ms/step | 11.3 samples/s
+Step 22800/44000 | Loss: 2.9513 | LR: 2.42e-04 | 1772.6ms/step | 11.3 samples/s
+Step 22810/44000 | Loss: 2.9639 | LR: 2.42e-04 | 1771.8ms/step | 11.3 samples/s
+Step 22820/44000 | Loss: 2.9381 | LR: 2.42e-04 | 1773.2ms/step | 11.3 samples/s
+Step 22830/44000 | Loss: 2.9468 | LR: 2.42e-04 | 1770.6ms/step | 11.3 samples/s
+Step 22840/44000 | Loss: 2.9237 | LR: 2.42e-04 | 1771.5ms/step | 11.3 samples/s
+Step 22850/44000 | Loss: 2.9807 | LR: 2.41e-04 | 1771.2ms/step | 11.3 samples/s
+Step 22860/44000 | Loss: 2.9457 | LR: 2.41e-04 | 1770.2ms/step | 11.3 samples/s
+Step 22870/44000 | Loss: 2.9746 | LR: 2.41e-04 | 1770.5ms/step | 11.3 samples/s
+Step 22880/44000 | Loss: 2.9520 | LR: 2.41e-04 | 1770.1ms/step | 11.3 samples/s
+Step 22890/44000 | Loss: 2.9710 | LR: 2.41e-04 | 1772.0ms/step | 11.3 samples/s
+Step 22900/44000 | Loss: 2.9509 | LR: 2.40e-04 | 1771.6ms/step | 11.3 samples/s
+Step 22910/44000 | Loss: 2.9411 | LR: 2.40e-04 | 1771.8ms/step | 11.3 samples/s
+Step 22920/44000 | Loss: 2.9636 | LR: 2.40e-04 | 1771.6ms/step | 11.3 samples/s
+Step 22930/44000 | Loss: 2.9357 | LR: 2.40e-04 | 1772.1ms/step | 11.3 samples/s
+Step 22940/44000 | Loss: 2.9113 | LR: 2.40e-04 | 1771.4ms/step | 11.3 samples/s

epfl_final/train_logs/ropectl_a1p0_a100_ddp2/seed43.20260820T140226Z.log

@@ -0,0 +1,838 @@
+W0820 14:02:40.078000 230 site-packages/torch/distributed/run.py:851] 
+W0820 14:02:40.078000 230 site-packages/torch/distributed/run.py:851] *****************************************
+W0820 14:02:40.078000 230 site-packages/torch/distributed/run.py:851] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
+W0820 14:02:40.078000 230 site-packages/torch/distributed/run.py:851] *****************************************
+Auto-resuming from latest checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-18000
+Resuming: Using existing output directory /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735
+Running config:
+{
+  "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
+  "model_dtype": "bfloat16",
+  "init_from_scratch": true,
+  "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
+  "val_path": null,
+  "max_seq_length": 8192,
+  "num_workers": 8,
+  "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735",
+  "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-18000",
+  "force_new_run": false,
+  "num_epochs": 1,
+  "max_steps": 44000,
+  "per_device_batch_size": 2,
+  "gradient_accumulation_steps": 5,
+  "learning_rate": 0.0005,
+  "weight_decay": 0.01,
+  "warmup_steps": 750,
+  "lr_scheduler": "cosine",
+  "max_grad_norm": 1.0,
+  "save_steps": 1000,
+  "eval_steps": 1000,
+  "eval_batches": 50,
+  "logging_steps": 10,
+  "seed": 43,
+  "verbose": true,
+  "checkpoint_activations": false,
+  "strategy": "ddp",
+  "mixed_precision": "bf16",
+  "wandb_enabled": true,
+  "wandb_project": "stochastic_sparsity",
+  "wandb_name": "l8k365m_dense_ropectl_a1p0_seed43",
+  "wandb_tags": [
+    "qwen3",
+    "365m",
+    "8k",
+    "dense",
+    "rope-control",
+    "ddp2-a100"
+  ],
+  "hidden_size": 768,
+  "intermediate_size": 3072,
+  "num_attention_heads": 6,
+  "num_key_value_heads": 2,
+  "num_hidden_layers": 28,
+  "head_dim": null,
+  "attention_dropout": 0.0,
+  "rope_train_position_scale_set": [
+    1.0,
+    0.9896339488176222,
+    0.9692159692159692,
+    0.9420289855072463,
+    0.8730158730158729
+  ],
+  "attention_type": "sparse_stochastic",
+  "sparse_block_size": 128,
+  "sparse_num_random_blocks": 31,
+  "sparse_mask_pool_size": 1,
+  "sparse_val_dense_compare": false,
+  "sparse_layer_indices": null,
+  "sparse_mask_cache_dir": null,
+  "sparse_mask_seed": 0,
+  "sparse_mixed_nrb_set": null,
+  "dense_mode": true,
+  "dense_step_intervals": null
+}
+Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
+Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-18000
+Checkpoint activations: FalseCheckpoint activations: False
+
+dense_mode=True: sparse attention layers will use standard causal SDPA.
+
+
+================================================================================
+MODEL ARCHITECTURE
+================================================================================
+Model config:
+  vocab_size: 151936
+  hidden_size: 768
+  num_hidden_layers: 28
+  num_attention_heads: 6
+  num_key_value_heads: 2
+  head_dim: 128
+  intermediate_size: 3072
+  max_position_embeddings: 32768
+  rope_theta: 1000000
+  rope_train_position_scale_set: [1.0, 0.9896339488176222, 0.9692159692159692, 0.9420289855072463, 0.8730158730158729]
+  attention_type: sparse_stochastic
+  sparse_block_size: 128
+  sparse_num_random_blocks: 31
+  sparse_mask_pool_size: 1
+  sparse_max_seq_length: 8192
+
+Parameter count:
+  Total parameters: 358,958,848
+  Trainable parameters: 358,958,848
+  Non-trainable parameters: 0
+  Model size (BF16): 0.72 GB
+
+--------------------------------------------------------------------------------
+LAYER-BY-LAYER SUMMARY
+--------------------------------------------------------------------------------
+================================================================================================================================================================
+Layer (type:depth-idx)                                       Input Shape               Output Shape              Param #                   Trainable
+================================================================================================================================================================
+Qwen3ForCausalLM                                             [2, 8192]                 --                        --                        True
+├─Qwen3Model: 1-1                                            --                        [2, 8192, 768]            --                        True
+│    └─Embedding: 2-1                                        [2, 8192]                 [2, 8192, 768]            116,686,848               True
+│    └─ModuleList: 2-2                                       --                        --                        --                        True
+│    │    └─Qwen3DecoderLayer: 3-1                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-2                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-3                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-4                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-5                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-6                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-7                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-8                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-9                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-10                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-11                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-12                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-13                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-14                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-15                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-16                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-17                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-18                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-19                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-20                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-21                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-22                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-23                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-24                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-25                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-26                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-27                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-28                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    └─Qwen3RMSNorm: 2-3                                     [2, 8192, 768]            [2, 8192, 768]            768                       True
+├─Linear: 1-2                                                [2, 8192, 768]            [2, 8192, 151936]         116,686,848               True
+================================================================================================================================================================
+Total params: 475,645,696
+Trainable params: 475,645,696
+Non-trainable params: 0
+Total mult-adds (Units.MEGABYTES): 951.29
+================================================================================================================================================================
+Input size (MB): 0.13
+Forward/backward pass size (MB): 31197.23
+Params size (MB): 951.29
+Estimated Total Size (MB): 32148.66
+================================================================================================================================================================
+  NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
+================================================================================
+
+Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
+Using pre-tokenized dataset (no runtime tokenization)
+Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
+
+Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
+Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
+Dataset split: 3027764 train, 159356 validation
+Training examples (packed, train split):  3027764
+Validation examples (packed, val split):  159356
+Per-device batch size:                    2
+World size (GPUs):                        2
+Gradient accumulation steps:              5
+Effective batch size (seqs/step):         20
+Sequence length:                          8192
+Tokens per optimizer step:                163,840
+Micro-batches per epoch (per rank):       756941
+Optimizer steps per epoch:                151388
+Total optimizer steps:                    44000  (capped by max_steps; 0.29 epochs)
+Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-18000...
+Preserving checkpoint LR state for exact resume: current=[0.00032811727248514757], schedule_base=[0.0005]
+Resumed from checkpoint at step 18000 (epoch 0, micro_step 180000)
+Resuming training from step 18000
+Resuming wandb run ysa9iyov
+wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from WANDB_API_KEY.
+wandb: Currently logged in as: ruben-navasardyan4869 (ruben-navasardyan4869-epfl) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
+wandb: setting up run ysa9iyov
+wandb: Tracking run with wandb version 0.26.1
+wandb: Run data is saved locally in /mloscratch/homes/navasard/sparse/SparseAtt/wandb/run-20260820_140401-ysa9iyov
+wandb: Run `wandb offline` to turn off syncing.
+wandb: Resuming run l8k365m_dense_ropectl_a1p0_seed43
+wandb: ⭐️ View project at https://wandb.ai/ruben-navasardyan4869-epfl/stochastic_sparsity
+wandb: 🚀 View run at https://wandb.ai/ruben-navasardyan4869-epfl/stochastic_sparsity/runs/ysa9iyov
+Starting training...
+Validation at step 18000: Loss = 2.9728, Perplexity = 19.55
+wandb: WARNING Tried to log to step 18000 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Switching attention mode for step 18001: dense (dense step interval)
+Step 18010/44000 | Loss: 2.9971 | LR: 3.28e-04 | 42307.5ms/step | 0.5 samples/s
+wandb: WARNING Tried to log to step 18010 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18020/44000 | Loss: 2.9907 | LR: 3.28e-04 | 1773.9ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 18020 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18030/44000 | Loss: 3.0001 | LR: 3.28e-04 | 1777.2ms/step | 11.3 samples/s
+wandb: WARNING Tried to log to step 18030 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18040/44000 | Loss: 2.9561 | LR: 3.27e-04 | 1778.8ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18040 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18050/44000 | Loss: 2.9921 | LR: 3.27e-04 | 1779.0ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18050 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18060/44000 | Loss: 3.0097 | LR: 3.27e-04 | 1781.8ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18060 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18070/44000 | Loss: 2.9773 | LR: 3.27e-04 | 1781.6ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18070 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18080/44000 | Loss: 2.9883 | LR: 3.27e-04 | 1781.0ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18080 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18090/44000 | Loss: 2.9542 | LR: 3.27e-04 | 1780.5ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18090 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18100/44000 | Loss: 2.9393 | LR: 3.26e-04 | 1781.5ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18100 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18110/44000 | Loss: 3.0064 | LR: 3.26e-04 | 1781.2ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18110 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18120/44000 | Loss: 2.9451 | LR: 3.26e-04 | 1782.0ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18120 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18130/44000 | Loss: 2.9749 | LR: 3.26e-04 | 1781.3ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18130 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18140/44000 | Loss: 3.0306 | LR: 3.26e-04 | 1781.8ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18140 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18150/44000 | Loss: 3.0022 | LR: 3.26e-04 | 1781.8ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18150 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18160/44000 | Loss: 2.9571 | LR: 3.25e-04 | 1782.0ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18160 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18170/44000 | Loss: 2.9630 | LR: 3.25e-04 | 1783.4ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18170 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18180/44000 | Loss: 2.9619 | LR: 3.25e-04 | 1783.4ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18180 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18190/44000 | Loss: 3.0241 | LR: 3.25e-04 | 1782.4ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18190 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18200/44000 | Loss: 2.9990 | LR: 3.25e-04 | 1785.0ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18200 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18210/44000 | Loss: 3.0164 | LR: 3.24e-04 | 1783.6ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18210 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18220/44000 | Loss: 2.9772 | LR: 3.24e-04 | 1782.7ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18220 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18230/44000 | Loss: 2.9828 | LR: 3.24e-04 | 1782.7ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18230 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18240/44000 | Loss: 3.0113 | LR: 3.24e-04 | 1783.1ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18240 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18250/44000 | Loss: 2.9762 | LR: 3.24e-04 | 1783.6ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18250 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18260/44000 | Loss: 3.0124 | LR: 3.24e-04 | 1783.3ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18260 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18270/44000 | Loss: 2.9906 | LR: 3.23e-04 | 1783.6ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18270 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18280/44000 | Loss: 2.9492 | LR: 3.23e-04 | 1784.0ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18280 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18290/44000 | Loss: 3.0016 | LR: 3.23e-04 | 1781.8ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18290 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18300/44000 | Loss: 3.0241 | LR: 3.23e-04 | 1782.7ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18300 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18310/44000 | Loss: 3.0117 | LR: 3.23e-04 | 1782.7ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18310 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18320/44000 | Loss: 3.0382 | LR: 3.23e-04 | 1783.2ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18320 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18330/44000 | Loss: 3.0040 | LR: 3.22e-04 | 1782.2ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18330 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18340/44000 | Loss: 2.9568 | LR: 3.22e-04 | 1782.7ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18340 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18350/44000 | Loss: 2.9752 | LR: 3.22e-04 | 1782.5ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18350 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18360/44000 | Loss: 2.9717 | LR: 3.22e-04 | 1782.6ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18360 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18370/44000 | Loss: 3.0086 | LR: 3.22e-04 | 1781.7ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18370 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18380/44000 | Loss: 2.9834 | LR: 3.22e-04 | 1782.6ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18380 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18390/44000 | Loss: 3.0108 | LR: 3.21e-04 | 1782.5ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18390 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18400/44000 | Loss: 3.0106 | LR: 3.21e-04 | 1783.0ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18400 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18410/44000 | Loss: 2.9893 | LR: 3.21e-04 | 1781.9ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18410 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18420/44000 | Loss: 2.9824 | LR: 3.21e-04 | 1781.9ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18420 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18430/44000 | Loss: 2.9528 | LR: 3.21e-04 | 1780.9ms/step | 11.2 samples/s
+wandb: WARNING Tried to log to step 18430 that is less than the current step 18431. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 18440/44000 | Loss: 2.9364 | LR: 3.20e-04 | 1781.7ms/step | 11.2 samples/s
+Step 18450/44000 | Loss: 2.9737 | LR: 3.20e-04 | 1783.2ms/step | 11.2 samples/s
+Step 18460/44000 | Loss: 2.9931 | LR: 3.20e-04 | 1782.8ms/step | 11.2 samples/s
+Step 18470/44000 | Loss: 3.0209 | LR: 3.20e-04 | 1782.0ms/step | 11.2 samples/s
+Step 18480/44000 | Loss: 2.9502 | LR: 3.20e-04 | 1782.9ms/step | 11.2 samples/s
+Step 18490/44000 | Loss: 2.9597 | LR: 3.20e-04 | 1783.0ms/step | 11.2 samples/s
+Step 18500/44000 | Loss: 2.9759 | LR: 3.19e-04 | 1783.1ms/step | 11.2 samples/s
+Step 18510/44000 | Loss: 2.9690 | LR: 3.19e-04 | 1783.1ms/step | 11.2 samples/s
+Step 18520/44000 | Loss: 2.9974 | LR: 3.19e-04 | 1782.9ms/step | 11.2 samples/s
+Step 18530/44000 | Loss: 3.0094 | LR: 3.19e-04 | 1783.1ms/step | 11.2 samples/s
+Step 18540/44000 | Loss: 3.0191 | LR: 3.19e-04 | 1784.2ms/step | 11.2 samples/s
+Step 18550/44000 | Loss: 2.9463 | LR: 3.19e-04 | 1782.2ms/step | 11.2 samples/s
+Step 18560/44000 | Loss: 3.0022 | LR: 3.18e-04 | 1782.6ms/step | 11.2 samples/s
+Step 18570/44000 | Loss: 3.0116 | LR: 3.18e-04 | 1781.9ms/step | 11.2 samples/s
+Step 18580/44000 | Loss: 3.0194 | LR: 3.18e-04 | 1783.9ms/step | 11.2 samples/s
+Step 18590/44000 | Loss: 2.9816 | LR: 3.18e-04 | 1783.1ms/step | 11.2 samples/s
+Step 18600/44000 | Loss: 2.9943 | LR: 3.18e-04 | 1783.7ms/step | 11.2 samples/s
+Step 18610/44000 | Loss: 2.9607 | LR: 3.18e-04 | 1782.4ms/step | 11.2 samples/s
+Step 18620/44000 | Loss: 3.0022 | LR: 3.17e-04 | 1782.5ms/step | 11.2 samples/s
+Step 18630/44000 | Loss: 3.0042 | LR: 3.17e-04 | 1782.9ms/step | 11.2 samples/s
+Step 18640/44000 | Loss: 3.0192 | LR: 3.17e-04 | 1782.6ms/step | 11.2 samples/s
+Step 18650/44000 | Loss: 2.9810 | LR: 3.17e-04 | 1782.3ms/step | 11.2 samples/s
+Step 18660/44000 | Loss: 2.9998 | LR: 3.17e-04 | 1782.8ms/step | 11.2 samples/s
+Step 18670/44000 | Loss: 2.9850 | LR: 3.16e-04 | 1782.9ms/step | 11.2 samples/s
+Step 18680/44000 | Loss: 3.0212 | LR: 3.16e-04 | 1783.8ms/step | 11.2 samples/s
+Step 18690/44000 | Loss: 2.9495 | LR: 3.16e-04 | 1783.3ms/step | 11.2 samples/s
+Step 18700/44000 | Loss: 2.9738 | LR: 3.16e-04 | 1781.9ms/step | 11.2 samples/s
+Step 18710/44000 | Loss: 3.0018 | LR: 3.16e-04 | 1783.0ms/step | 11.2 samples/s
+Step 18720/44000 | Loss: 2.9478 | LR: 3.16e-04 | 1782.1ms/step | 11.2 samples/s
+Step 18730/44000 | Loss: 3.0106 | LR: 3.15e-04 | 1782.9ms/step | 11.2 samples/s
+Step 18740/44000 | Loss: 2.9539 | LR: 3.15e-04 | 1781.9ms/step | 11.2 samples/s
+Step 18750/44000 | Loss: 2.9888 | LR: 3.15e-04 | 1786.6ms/step | 11.2 samples/s
+Step 18760/44000 | Loss: 2.9621 | LR: 3.15e-04 | 1784.1ms/step | 11.2 samples/s
+Step 18770/44000 | Loss: 2.9700 | LR: 3.15e-04 | 1784.5ms/step | 11.2 samples/s
+Step 18780/44000 | Loss: 2.9620 | LR: 3.15e-04 | 1783.6ms/step | 11.2 samples/s
+Step 18790/44000 | Loss: 2.9962 | LR: 3.14e-04 | 1783.3ms/step | 11.2 samples/s
+Step 18800/44000 | Loss: 2.9913 | LR: 3.14e-04 | 1786.7ms/step | 11.2 samples/s
+Step 18810/44000 | Loss: 2.9776 | LR: 3.14e-04 | 1786.7ms/step | 11.2 samples/s
+Step 18820/44000 | Loss: 2.9986 | LR: 3.14e-04 | 1782.5ms/step | 11.2 samples/s
+Step 18830/44000 | Loss: 2.9464 | LR: 3.14e-04 | 1783.4ms/step | 11.2 samples/s
+Step 18840/44000 | Loss: 2.9397 | LR: 3.13e-04 | 1783.2ms/step | 11.2 samples/s
+Step 18850/44000 | Loss: 2.9791 | LR: 3.13e-04 | 1783.6ms/step | 11.2 samples/s
+Step 18860/44000 | Loss: 2.9596 | LR: 3.13e-04 | 1783.5ms/step | 11.2 samples/s
+Step 18870/44000 | Loss: 2.9497 | LR: 3.13e-04 | 1783.7ms/step | 11.2 samples/s
+Step 18880/44000 | Loss: 2.9688 | LR: 3.13e-04 | 1783.8ms/step | 11.2 samples/s
+Step 18890/44000 | Loss: 2.9681 | LR: 3.13e-04 | 1784.2ms/step | 11.2 samples/s
+Step 18900/44000 | Loss: 2.9671 | LR: 3.12e-04 | 1784.4ms/step | 11.2 samples/s
+Step 18910/44000 | Loss: 2.9530 | LR: 3.12e-04 | 1784.7ms/step | 11.2 samples/s
+Step 18920/44000 | Loss: 2.9778 | LR: 3.12e-04 | 1784.4ms/step | 11.2 samples/s
+Step 18930/44000 | Loss: 3.0043 | LR: 3.12e-04 | 1783.4ms/step | 11.2 samples/s
+Step 18940/44000 | Loss: 2.9871 | LR: 3.12e-04 | 1782.7ms/step | 11.2 samples/s
+Step 18950/44000 | Loss: 2.9887 | LR: 3.12e-04 | 1782.5ms/step | 11.2 samples/s
+Step 18960/44000 | Loss: 2.9920 | LR: 3.11e-04 | 1783.0ms/step | 11.2 samples/s
+Step 18970/44000 | Loss: 2.9774 | LR: 3.11e-04 | 1783.8ms/step | 11.2 samples/s
+Step 18980/44000 | Loss: 2.9757 | LR: 3.11e-04 | 1782.9ms/step | 11.2 samples/s
+Step 18990/44000 | Loss: 2.9514 | LR: 3.11e-04 | 1783.4ms/step | 11.2 samples/s
+Step 19000/44000 | Loss: 2.9648 | LR: 3.11e-04 | 1782.4ms/step | 11.2 samples/s
+Validation at step 19000: Loss = 2.9626, Perplexity = 19.35
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-19000
+Step 19010/44000 | Loss: 2.9803 | LR: 3.11e-04 | 3584.0ms/step | 5.6 samples/s
+Step 19020/44000 | Loss: 2.9681 | LR: 3.10e-04 | 1785.4ms/step | 11.2 samples/s
+Step 19030/44000 | Loss: 2.9647 | LR: 3.10e-04 | 1786.4ms/step | 11.2 samples/s
+Step 19040/44000 | Loss: 2.9520 | LR: 3.10e-04 | 1786.3ms/step | 11.2 samples/s
+Step 19050/44000 | Loss: 3.0237 | LR: 3.10e-04 | 1787.6ms/step | 11.2 samples/s
+Step 19060/44000 | Loss: 2.9959 | LR: 3.10e-04 | 1785.9ms/step | 11.2 samples/s
+Step 19070/44000 | Loss: 2.9645 | LR: 3.09e-04 | 1787.2ms/step | 11.2 samples/s
+Step 19080/44000 | Loss: 3.0128 | LR: 3.09e-04 | 1784.9ms/step | 11.2 samples/s
+Step 19090/44000 | Loss: 2.9736 | LR: 3.09e-04 | 1786.8ms/step | 11.2 samples/s
+Step 19100/44000 | Loss: 3.0043 | LR: 3.09e-04 | 1786.2ms/step | 11.2 samples/s
+Step 19110/44000 | Loss: 3.0010 | LR: 3.09e-04 | 1787.1ms/step | 11.2 samples/s
+Step 19120/44000 | Loss: 3.0102 | LR: 3.09e-04 | 1786.1ms/step | 11.2 samples/s
+Step 19130/44000 | Loss: 3.0016 | LR: 3.08e-04 | 1786.9ms/step | 11.2 samples/s
+Step 19140/44000 | Loss: 3.0037 | LR: 3.08e-04 | 1785.6ms/step | 11.2 samples/s
+Step 19150/44000 | Loss: 2.9993 | LR: 3.08e-04 | 1787.7ms/step | 11.2 samples/s
+Step 19160/44000 | Loss: 2.9615 | LR: 3.08e-04 | 1787.0ms/step | 11.2 samples/s
+Step 19170/44000 | Loss: 3.0020 | LR: 3.08e-04 | 1786.2ms/step | 11.2 samples/s
+Step 19180/44000 | Loss: 3.0121 | LR: 3.08e-04 | 1785.0ms/step | 11.2 samples/s
+Step 19190/44000 | Loss: 2.9320 | LR: 3.07e-04 | 1786.2ms/step | 11.2 samples/s
+Step 19200/44000 | Loss: 2.9542 | LR: 3.07e-04 | 1786.1ms/step | 11.2 samples/s
+Step 19210/44000 | Loss: 2.9760 | LR: 3.07e-04 | 1785.5ms/step | 11.2 samples/s
+Step 19220/44000 | Loss: 3.0029 | LR: 3.07e-04 | 1785.0ms/step | 11.2 samples/s
+Step 19230/44000 | Loss: 2.9765 | LR: 3.07e-04 | 1786.1ms/step | 11.2 samples/s
+Step 19240/44000 | Loss: 2.9915 | LR: 3.06e-04 | 1786.2ms/step | 11.2 samples/s
+Step 19250/44000 | Loss: 2.9664 | LR: 3.06e-04 | 1786.1ms/step | 11.2 samples/s
+Step 19260/44000 | Loss: 2.9532 | LR: 3.06e-04 | 1785.8ms/step | 11.2 samples/s
+Step 19270/44000 | Loss: 2.9631 | LR: 3.06e-04 | 1784.7ms/step | 11.2 samples/s
+Step 19280/44000 | Loss: 3.0000 | LR: 3.06e-04 | 1786.2ms/step | 11.2 samples/s
+Step 19290/44000 | Loss: 2.9515 | LR: 3.06e-04 | 1784.9ms/step | 11.2 samples/s
+Step 19300/44000 | Loss: 2.9979 | LR: 3.05e-04 | 1786.6ms/step | 11.2 samples/s
+Step 19310/44000 | Loss: 2.9852 | LR: 3.05e-04 | 1786.5ms/step | 11.2 samples/s
+Step 19320/44000 | Loss: 2.9740 | LR: 3.05e-04 | 1786.5ms/step | 11.2 samples/s
+Step 19330/44000 | Loss: 2.9537 | LR: 3.05e-04 | 1786.3ms/step | 11.2 samples/s
+Step 19340/44000 | Loss: 2.9845 | LR: 3.05e-04 | 1786.3ms/step | 11.2 samples/s
+Step 19350/44000 | Loss: 2.9603 | LR: 3.04e-04 | 1785.3ms/step | 11.2 samples/s
+Step 19360/44000 | Loss: 2.9405 | LR: 3.04e-04 | 1784.9ms/step | 11.2 samples/s
+Step 19370/44000 | Loss: 2.9557 | LR: 3.04e-04 | 1784.2ms/step | 11.2 samples/s
+Step 19380/44000 | Loss: 2.9809 | LR: 3.04e-04 | 1785.4ms/step | 11.2 samples/s
+Step 19390/44000 | Loss: 2.9811 | LR: 3.04e-04 | 1784.9ms/step | 11.2 samples/s
+Step 19400/44000 | Loss: 2.9769 | LR: 3.04e-04 | 1787.2ms/step | 11.2 samples/s
+Step 19410/44000 | Loss: 2.9840 | LR: 3.03e-04 | 1786.3ms/step | 11.2 samples/s
+Step 19420/44000 | Loss: 2.9710 | LR: 3.03e-04 | 1786.6ms/step | 11.2 samples/s
+Step 19430/44000 | Loss: 2.9264 | LR: 3.03e-04 | 1784.7ms/step | 11.2 samples/s
+Step 19440/44000 | Loss: 2.9834 | LR: 3.03e-04 | 1785.9ms/step | 11.2 samples/s
+Step 19450/44000 | Loss: 3.0116 | LR: 3.03e-04 | 1785.7ms/step | 11.2 samples/s
+Step 19460/44000 | Loss: 2.9537 | LR: 3.03e-04 | 1787.3ms/step | 11.2 samples/s
+Step 19470/44000 | Loss: 2.9745 | LR: 3.02e-04 | 1788.1ms/step | 11.2 samples/s
+Step 19480/44000 | Loss: 2.9615 | LR: 3.02e-04 | 1786.2ms/step | 11.2 samples/s
+Step 19490/44000 | Loss: 2.9910 | LR: 3.02e-04 | 1786.4ms/step | 11.2 samples/s
+Step 19500/44000 | Loss: 3.0047 | LR: 3.02e-04 | 1787.2ms/step | 11.2 samples/s
+Step 19510/44000 | Loss: 2.9671 | LR: 3.02e-04 | 1785.6ms/step | 11.2 samples/s
+Step 19520/44000 | Loss: 2.9678 | LR: 3.01e-04 | 1785.5ms/step | 11.2 samples/s
+Step 19530/44000 | Loss: 2.9600 | LR: 3.01e-04 | 1785.8ms/step | 11.2 samples/s
+Step 19540/44000 | Loss: 2.9711 | LR: 3.01e-04 | 1785.8ms/step | 11.2 samples/s
+Step 19550/44000 | Loss: 2.9762 | LR: 3.01e-04 | 1786.3ms/step | 11.2 samples/s
+Step 19560/44000 | Loss: 2.9826 | LR: 3.01e-04 | 1785.5ms/step | 11.2 samples/s
+Step 19570/44000 | Loss: 2.9861 | LR: 3.01e-04 | 1785.6ms/step | 11.2 samples/s
+Step 19580/44000 | Loss: 2.9780 | LR: 3.00e-04 | 1785.7ms/step | 11.2 samples/s
+Step 19590/44000 | Loss: 2.9711 | LR: 3.00e-04 | 1785.8ms/step | 11.2 samples/s
+Step 19600/44000 | Loss: 2.9793 | LR: 3.00e-04 | 1785.5ms/step | 11.2 samples/s
+Step 19610/44000 | Loss: 3.0148 | LR: 3.00e-04 | 1785.2ms/step | 11.2 samples/s
+Step 19620/44000 | Loss: 3.0110 | LR: 3.00e-04 | 1782.6ms/step | 11.2 samples/s
+Step 19630/44000 | Loss: 2.9517 | LR: 3.00e-04 | 1785.6ms/step | 11.2 samples/s
+Step 19640/44000 | Loss: 2.9868 | LR: 2.99e-04 | 1785.3ms/step | 11.2 samples/s
+Step 19650/44000 | Loss: 2.9841 | LR: 2.99e-04 | 1786.1ms/step | 11.2 samples/s
+Step 19660/44000 | Loss: 2.9842 | LR: 2.99e-04 | 1784.9ms/step | 11.2 samples/s
+Step 19670/44000 | Loss: 3.0168 | LR: 2.99e-04 | 1786.1ms/step | 11.2 samples/s
+Step 19680/44000 | Loss: 2.9891 | LR: 2.99e-04 | 1785.7ms/step | 11.2 samples/s
+Step 19690/44000 | Loss: 2.9710 | LR: 2.98e-04 | 1786.8ms/step | 11.2 samples/s
+Step 19700/44000 | Loss: 2.9777 | LR: 2.98e-04 | 1786.5ms/step | 11.2 samples/s
+Step 19710/44000 | Loss: 2.9780 | LR: 2.98e-04 | 1787.1ms/step | 11.2 samples/s
+Step 19720/44000 | Loss: 3.0133 | LR: 2.98e-04 | 1787.6ms/step | 11.2 samples/s
+Step 19730/44000 | Loss: 2.9665 | LR: 2.98e-04 | 1786.6ms/step | 11.2 samples/s
+Step 19740/44000 | Loss: 2.9740 | LR: 2.98e-04 | 1785.4ms/step | 11.2 samples/s
+Step 19750/44000 | Loss: 2.9987 | LR: 2.97e-04 | 1786.4ms/step | 11.2 samples/s
+Step 19760/44000 | Loss: 2.9649 | LR: 2.97e-04 | 1785.2ms/step | 11.2 samples/s
+Step 19770/44000 | Loss: 2.9672 | LR: 2.97e-04 | 1786.0ms/step | 11.2 samples/s
+Step 19780/44000 | Loss: 2.9803 | LR: 2.97e-04 | 1785.9ms/step | 11.2 samples/s
+Step 19790/44000 | Loss: 3.0131 | LR: 2.97e-04 | 1785.7ms/step | 11.2 samples/s
+Step 19800/44000 | Loss: 2.9232 | LR: 2.96e-04 | 1785.6ms/step | 11.2 samples/s
+Step 19810/44000 | Loss: 2.9741 | LR: 2.96e-04 | 1785.9ms/step | 11.2 samples/s
+Step 19820/44000 | Loss: 3.0011 | LR: 2.96e-04 | 1785.4ms/step | 11.2 samples/s
+Step 19830/44000 | Loss: 2.9685 | LR: 2.96e-04 | 1785.9ms/step | 11.2 samples/s
+Step 19840/44000 | Loss: 3.0102 | LR: 2.96e-04 | 1785.9ms/step | 11.2 samples/s
+Step 19850/44000 | Loss: 2.9582 | LR: 2.96e-04 | 1786.0ms/step | 11.2 samples/s
+Step 19860/44000 | Loss: 3.0023 | LR: 2.95e-04 | 1786.2ms/step | 11.2 samples/s
+Step 19870/44000 | Loss: 2.9845 | LR: 2.95e-04 | 1785.1ms/step | 11.2 samples/s
+Step 19880/44000 | Loss: 3.0251 | LR: 2.95e-04 | 1785.3ms/step | 11.2 samples/s
+Step 19890/44000 | Loss: 2.9631 | LR: 2.95e-04 | 1785.0ms/step | 11.2 samples/s
+Step 19900/44000 | Loss: 2.9846 | LR: 2.95e-04 | 1786.8ms/step | 11.2 samples/s
+Step 19910/44000 | Loss: 2.9572 | LR: 2.95e-04 | 1785.3ms/step | 11.2 samples/s
+Step 19920/44000 | Loss: 2.9981 | LR: 2.94e-04 | 1784.7ms/step | 11.2 samples/s
+Step 19930/44000 | Loss: 2.9439 | LR: 2.94e-04 | 1784.0ms/step | 11.2 samples/s
+Step 19940/44000 | Loss: 2.9729 | LR: 2.94e-04 | 1785.8ms/step | 11.2 samples/s
+Step 19950/44000 | Loss: 2.9531 | LR: 2.94e-04 | 1784.2ms/step | 11.2 samples/s
+Step 19960/44000 | Loss: 2.9860 | LR: 2.94e-04 | 1784.4ms/step | 11.2 samples/s
+Step 19970/44000 | Loss: 2.9534 | LR: 2.93e-04 | 1783.5ms/step | 11.2 samples/s
+Step 19980/44000 | Loss: 2.9586 | LR: 2.93e-04 | 1784.1ms/step | 11.2 samples/s
+Step 19990/44000 | Loss: 2.9507 | LR: 2.93e-04 | 1784.3ms/step | 11.2 samples/s
+Step 20000/44000 | Loss: 2.9806 | LR: 2.93e-04 | 1784.9ms/step | 11.2 samples/s
+Validation at step 20000: Loss = 2.9514, Perplexity = 19.13
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-20000
+Step 20010/44000 | Loss: 2.9525 | LR: 2.93e-04 | 3588.2ms/step | 5.6 samples/s
+Step 20020/44000 | Loss: 2.9913 | LR: 2.93e-04 | 1784.6ms/step | 11.2 samples/s
+Step 20030/44000 | Loss: 2.9715 | LR: 2.92e-04 | 1785.8ms/step | 11.2 samples/s
+Step 20040/44000 | Loss: 2.9938 | LR: 2.92e-04 | 1784.2ms/step | 11.2 samples/s
+Step 20050/44000 | Loss: 2.9420 | LR: 2.92e-04 | 1784.0ms/step | 11.2 samples/s
+Step 20060/44000 | Loss: 2.9639 | LR: 2.92e-04 | 1783.9ms/step | 11.2 samples/s
+Step 20070/44000 | Loss: 3.0013 | LR: 2.92e-04 | 1784.7ms/step | 11.2 samples/s
+Step 20080/44000 | Loss: 2.9357 | LR: 2.91e-04 | 1784.5ms/step | 11.2 samples/s
+Step 20090/44000 | Loss: 2.9849 | LR: 2.91e-04 | 1784.5ms/step | 11.2 samples/s
+Step 20100/44000 | Loss: 2.9498 | LR: 2.91e-04 | 1784.7ms/step | 11.2 samples/s
+Step 20110/44000 | Loss: 2.9376 | LR: 2.91e-04 | 1785.1ms/step | 11.2 samples/s
+Step 20120/44000 | Loss: 2.9287 | LR: 2.91e-04 | 1784.4ms/step | 11.2 samples/s
+Step 20130/44000 | Loss: 2.9613 | LR: 2.91e-04 | 1784.2ms/step | 11.2 samples/s
+Step 20140/44000 | Loss: 2.9598 | LR: 2.90e-04 | 1786.0ms/step | 11.2 samples/s
+Step 20150/44000 | Loss: 2.9421 | LR: 2.90e-04 | 1785.0ms/step | 11.2 samples/s
+Step 20160/44000 | Loss: 3.0087 | LR: 2.90e-04 | 1784.8ms/step | 11.2 samples/s
+Step 20170/44000 | Loss: 2.9875 | LR: 2.90e-04 | 1784.1ms/step | 11.2 samples/s
+Step 20180/44000 | Loss: 2.9414 | LR: 2.90e-04 | 1784.9ms/step | 11.2 samples/s
+Step 20190/44000 | Loss: 2.9603 | LR: 2.90e-04 | 1784.0ms/step | 11.2 samples/s
+Step 20200/44000 | Loss: 2.9277 | LR: 2.89e-04 | 1786.8ms/step | 11.2 samples/s
+Step 20210/44000 | Loss: 2.9626 | LR: 2.89e-04 | 1784.3ms/step | 11.2 samples/s
+Step 20220/44000 | Loss: 2.9394 | LR: 2.89e-04 | 1788.5ms/step | 11.2 samples/s
+Step 20230/44000 | Loss: 2.9337 | LR: 2.89e-04 | 1784.7ms/step | 11.2 samples/s
+Step 20240/44000 | Loss: 2.9535 | LR: 2.89e-04 | 1786.2ms/step | 11.2 samples/s
+Step 20250/44000 | Loss: 2.9984 | LR: 2.88e-04 | 1785.6ms/step | 11.2 samples/s
+Step 20260/44000 | Loss: 2.9629 | LR: 2.88e-04 | 1787.2ms/step | 11.2 samples/s
+Step 20270/44000 | Loss: 2.9640 | LR: 2.88e-04 | 1784.1ms/step | 11.2 samples/s
+Step 20280/44000 | Loss: 2.9570 | LR: 2.88e-04 | 1786.5ms/step | 11.2 samples/s
+Step 20290/44000 | Loss: 2.9456 | LR: 2.88e-04 | 1786.6ms/step | 11.2 samples/s
+Step 20300/44000 | Loss: 2.9794 | LR: 2.88e-04 | 1784.1ms/step | 11.2 samples/s
+Step 20310/44000 | Loss: 2.9673 | LR: 2.87e-04 | 1785.3ms/step | 11.2 samples/s
+Step 20320/44000 | Loss: 2.9821 | LR: 2.87e-04 | 1785.3ms/step | 11.2 samples/s
+Step 20330/44000 | Loss: 2.9740 | LR: 2.87e-04 | 1784.6ms/step | 11.2 samples/s
+Step 20340/44000 | Loss: 2.9713 | LR: 2.87e-04 | 1784.0ms/step | 11.2 samples/s
+Step 20350/44000 | Loss: 2.9644 | LR: 2.87e-04 | 1786.9ms/step | 11.2 samples/s
+Step 20360/44000 | Loss: 2.9474 | LR: 2.86e-04 | 1785.9ms/step | 11.2 samples/s
+Step 20370/44000 | Loss: 2.9757 | LR: 2.86e-04 | 1786.2ms/step | 11.2 samples/s
+Step 20380/44000 | Loss: 2.9547 | LR: 2.86e-04 | 1784.9ms/step | 11.2 samples/s
+Step 20390/44000 | Loss: 2.9301 | LR: 2.86e-04 | 1784.8ms/step | 11.2 samples/s
+Step 20400/44000 | Loss: 2.9666 | LR: 2.86e-04 | 1784.2ms/step | 11.2 samples/s
+Step 20410/44000 | Loss: 2.9617 | LR: 2.86e-04 | 1785.0ms/step | 11.2 samples/s
+Step 20420/44000 | Loss: 2.9757 | LR: 2.85e-04 | 1786.5ms/step | 11.2 samples/s
+Step 20430/44000 | Loss: 2.9885 | LR: 2.85e-04 | 1785.8ms/step | 11.2 samples/s
+Step 20440/44000 | Loss: 2.9902 | LR: 2.85e-04 | 1784.2ms/step | 11.2 samples/s
+Step 20450/44000 | Loss: 2.9591 | LR: 2.85e-04 | 1785.5ms/step | 11.2 samples/s
+Step 20460/44000 | Loss: 2.9288 | LR: 2.85e-04 | 1784.3ms/step | 11.2 samples/s
+Step 20470/44000 | Loss: 2.9821 | LR: 2.84e-04 | 1785.2ms/step | 11.2 samples/s
+Step 20480/44000 | Loss: 2.9736 | LR: 2.84e-04 | 1784.5ms/step | 11.2 samples/s
+Step 20490/44000 | Loss: 2.9740 | LR: 2.84e-04 | 1785.8ms/step | 11.2 samples/s
+Step 20500/44000 | Loss: 2.9178 | LR: 2.84e-04 | 1784.6ms/step | 11.2 samples/s
+Step 20510/44000 | Loss: 3.0172 | LR: 2.84e-04 | 1785.1ms/step | 11.2 samples/s
+Step 20520/44000 | Loss: 2.9671 | LR: 2.84e-04 | 1784.5ms/step | 11.2 samples/s
+Step 20530/44000 | Loss: 2.9320 | LR: 2.83e-04 | 1784.8ms/step | 11.2 samples/s
+Step 20540/44000 | Loss: 2.9589 | LR: 2.83e-04 | 1785.2ms/step | 11.2 samples/s
+Step 20550/44000 | Loss: 2.9659 | LR: 2.83e-04 | 1783.8ms/step | 11.2 samples/s
+Step 20560/44000 | Loss: 2.9420 | LR: 2.83e-04 | 1784.0ms/step | 11.2 samples/s
+Step 20570/44000 | Loss: 2.9842 | LR: 2.83e-04 | 1783.3ms/step | 11.2 samples/s
+Step 20580/44000 | Loss: 2.9633 | LR: 2.83e-04 | 1784.6ms/step | 11.2 samples/s
+Step 20590/44000 | Loss: 2.9772 | LR: 2.82e-04 | 1784.5ms/step | 11.2 samples/s
+Step 20600/44000 | Loss: 2.9824 | LR: 2.82e-04 | 1785.4ms/step | 11.2 samples/s
+Step 20610/44000 | Loss: 2.9429 | LR: 2.82e-04 | 1784.3ms/step | 11.2 samples/s
+Step 20620/44000 | Loss: 2.9572 | LR: 2.82e-04 | 1784.1ms/step | 11.2 samples/s
+Step 20630/44000 | Loss: 2.9799 | LR: 2.82e-04 | 1783.6ms/step | 11.2 samples/s
+Step 20640/44000 | Loss: 2.9444 | LR: 2.81e-04 | 1784.2ms/step | 11.2 samples/s
+Step 20650/44000 | Loss: 2.9568 | LR: 2.81e-04 | 1783.0ms/step | 11.2 samples/s
+Step 20660/44000 | Loss: 2.9536 | LR: 2.81e-04 | 1784.0ms/step | 11.2 samples/s
+Step 20670/44000 | Loss: 2.9776 | LR: 2.81e-04 | 1783.2ms/step | 11.2 samples/s
+Step 20680/44000 | Loss: 2.9186 | LR: 2.81e-04 | 1783.7ms/step | 11.2 samples/s
+Step 20690/44000 | Loss: 2.9539 | LR: 2.81e-04 | 1783.3ms/step | 11.2 samples/s
+Step 20700/44000 | Loss: 2.9371 | LR: 2.80e-04 | 1785.0ms/step | 11.2 samples/s
+Step 20710/44000 | Loss: 2.9530 | LR: 2.80e-04 | 1784.1ms/step | 11.2 samples/s
+Step 20720/44000 | Loss: 2.9832 | LR: 2.80e-04 | 1786.5ms/step | 11.2 samples/s
+Step 20730/44000 | Loss: 2.9093 | LR: 2.80e-04 | 1786.4ms/step | 11.2 samples/s
+Step 20740/44000 | Loss: 2.9929 | LR: 2.80e-04 | 1784.8ms/step | 11.2 samples/s
+Step 20750/44000 | Loss: 2.9719 | LR: 2.79e-04 | 1784.1ms/step | 11.2 samples/s
+Step 20760/44000 | Loss: 3.0116 | LR: 2.79e-04 | 1783.6ms/step | 11.2 samples/s
+Step 20770/44000 | Loss: 2.9461 | LR: 2.79e-04 | 1783.8ms/step | 11.2 samples/s
+Step 20780/44000 | Loss: 2.9501 | LR: 2.79e-04 | 1783.9ms/step | 11.2 samples/s
+Step 20790/44000 | Loss: 2.9585 | LR: 2.79e-04 | 1783.8ms/step | 11.2 samples/s
+Step 20800/44000 | Loss: 2.9744 | LR: 2.79e-04 | 1783.7ms/step | 11.2 samples/s
+Step 20810/44000 | Loss: 2.9439 | LR: 2.78e-04 | 1784.1ms/step | 11.2 samples/s
+Step 20820/44000 | Loss: 2.9059 | LR: 2.78e-04 | 1783.4ms/step | 11.2 samples/s
+Step 20830/44000 | Loss: 2.9289 | LR: 2.78e-04 | 1785.1ms/step | 11.2 samples/s
+Step 20840/44000 | Loss: 2.9486 | LR: 2.78e-04 | 1783.8ms/step | 11.2 samples/s
+Step 20850/44000 | Loss: 2.9122 | LR: 2.78e-04 | 1785.2ms/step | 11.2 samples/s
+Step 20860/44000 | Loss: 3.0248 | LR: 2.77e-04 | 1784.2ms/step | 11.2 samples/s
+Step 20870/44000 | Loss: 2.9592 | LR: 2.77e-04 | 1792.6ms/step | 11.2 samples/s
+Step 20880/44000 | Loss: 2.9901 | LR: 2.77e-04 | 1788.2ms/step | 11.2 samples/s
+Step 20890/44000 | Loss: 2.9332 | LR: 2.77e-04 | 1785.5ms/step | 11.2 samples/s
+Step 20900/44000 | Loss: 2.9794 | LR: 2.77e-04 | 1784.2ms/step | 11.2 samples/s
+Step 20910/44000 | Loss: 2.9616 | LR: 2.77e-04 | 1784.8ms/step | 11.2 samples/s
+Step 20920/44000 | Loss: 2.9739 | LR: 2.76e-04 | 1784.1ms/step | 11.2 samples/s
+Step 20930/44000 | Loss: 2.9450 | LR: 2.76e-04 | 1785.6ms/step | 11.2 samples/s
+Step 20940/44000 | Loss: 2.9973 | LR: 2.76e-04 | 1786.0ms/step | 11.2 samples/s
+Step 20950/44000 | Loss: 2.9661 | LR: 2.76e-04 | 1787.1ms/step | 11.2 samples/s
+Step 20960/44000 | Loss: 2.9288 | LR: 2.76e-04 | 1784.9ms/step | 11.2 samples/s
+Step 20970/44000 | Loss: 2.9681 | LR: 2.75e-04 | 1785.0ms/step | 11.2 samples/s
+Step 20980/44000 | Loss: 2.9672 | LR: 2.75e-04 | 1785.5ms/step | 11.2 samples/s
+Step 20990/44000 | Loss: 2.9756 | LR: 2.75e-04 | 1784.5ms/step | 11.2 samples/s
+Step 21000/44000 | Loss: 2.9553 | LR: 2.75e-04 | 1785.9ms/step | 11.2 samples/s
+Validation at step 21000: Loss = 2.9396, Perplexity = 18.91
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-21000
+Step 21010/44000 | Loss: 2.9004 | LR: 2.75e-04 | 3591.0ms/step | 5.6 samples/s
+Step 21020/44000 | Loss: 2.9762 | LR: 2.75e-04 | 1782.9ms/step | 11.2 samples/s
+Step 21030/44000 | Loss: 2.9528 | LR: 2.74e-04 | 1783.8ms/step | 11.2 samples/s
+Step 21040/44000 | Loss: 2.9749 | LR: 2.74e-04 | 1784.6ms/step | 11.2 samples/s
+Step 21050/44000 | Loss: 2.9377 | LR: 2.74e-04 | 1784.3ms/step | 11.2 samples/s
+Step 21060/44000 | Loss: 2.9398 | LR: 2.74e-04 | 1783.0ms/step | 11.2 samples/s
+Step 21070/44000 | Loss: 2.9675 | LR: 2.74e-04 | 1784.6ms/step | 11.2 samples/s
+Step 21080/44000 | Loss: 2.9737 | LR: 2.73e-04 | 1783.6ms/step | 11.2 samples/s
+Step 21090/44000 | Loss: 3.0046 | LR: 2.73e-04 | 1784.6ms/step | 11.2 samples/s
+Step 21100/44000 | Loss: 2.9257 | LR: 2.73e-04 | 1783.4ms/step | 11.2 samples/s
+Step 21110/44000 | Loss: 2.9474 | LR: 2.73e-04 | 1784.4ms/step | 11.2 samples/s
+Step 21120/44000 | Loss: 2.9563 | LR: 2.73e-04 | 1783.2ms/step | 11.2 samples/s
+Step 21130/44000 | Loss: 2.9624 | LR: 2.73e-04 | 1783.9ms/step | 11.2 samples/s
+Step 21140/44000 | Loss: 2.9489 | LR: 2.72e-04 | 1784.2ms/step | 11.2 samples/s
+Step 21150/44000 | Loss: 2.9244 | LR: 2.72e-04 | 1785.5ms/step | 11.2 samples/s
+Step 21160/44000 | Loss: 2.9642 | LR: 2.72e-04 | 1784.0ms/step | 11.2 samples/s
+Step 21170/44000 | Loss: 2.9359 | LR: 2.72e-04 | 1783.6ms/step | 11.2 samples/s
+Step 21180/44000 | Loss: 2.9125 | LR: 2.72e-04 | 1782.5ms/step | 11.2 samples/s
+Step 21190/44000 | Loss: 2.9541 | LR: 2.71e-04 | 1783.9ms/step | 11.2 samples/s
+Step 21200/44000 | Loss: 2.9062 | LR: 2.71e-04 | 1784.2ms/step | 11.2 samples/s
+Step 21210/44000 | Loss: 2.9329 | LR: 2.71e-04 | 1784.6ms/step | 11.2 samples/s
+Step 21220/44000 | Loss: 2.9098 | LR: 2.71e-04 | 1784.3ms/step | 11.2 samples/s
+Step 21230/44000 | Loss: 2.9342 | LR: 2.71e-04 | 1785.1ms/step | 11.2 samples/s
+Step 21240/44000 | Loss: 2.9464 | LR: 2.71e-04 | 1785.3ms/step | 11.2 samples/s
+Step 21250/44000 | Loss: 2.9395 | LR: 2.70e-04 | 1786.7ms/step | 11.2 samples/s
+Step 21260/44000 | Loss: 2.9534 | LR: 2.70e-04 | 1785.1ms/step | 11.2 samples/s
+Step 21270/44000 | Loss: 2.9706 | LR: 2.70e-04 | 1784.1ms/step | 11.2 samples/s
+Step 21280/44000 | Loss: 2.9487 | LR: 2.70e-04 | 1784.4ms/step | 11.2 samples/s
+Step 21290/44000 | Loss: 2.9144 | LR: 2.70e-04 | 1783.1ms/step | 11.2 samples/s
+Step 21300/44000 | Loss: 2.9522 | LR: 2.70e-04 | 1784.1ms/step | 11.2 samples/s
+Step 21310/44000 | Loss: 2.9817 | LR: 2.69e-04 | 1783.2ms/step | 11.2 samples/s
+Step 21320/44000 | Loss: 2.9461 | LR: 2.69e-04 | 1785.1ms/step | 11.2 samples/s
+Step 21330/44000 | Loss: 2.9692 | LR: 2.69e-04 | 1782.8ms/step | 11.2 samples/s
+Step 21340/44000 | Loss: 2.9745 | LR: 2.69e-04 | 1783.7ms/step | 11.2 samples/s
+Step 21350/44000 | Loss: 2.9756 | LR: 2.69e-04 | 1783.4ms/step | 11.2 samples/s
+Step 21360/44000 | Loss: 2.9168 | LR: 2.68e-04 | 1784.6ms/step | 11.2 samples/s
+Step 21370/44000 | Loss: 2.9945 | LR: 2.68e-04 | 1784.1ms/step | 11.2 samples/s
+Step 21380/44000 | Loss: 2.9426 | LR: 2.68e-04 | 1782.8ms/step | 11.2 samples/s
+Step 21390/44000 | Loss: 2.9864 | LR: 2.68e-04 | 1782.4ms/step | 11.2 samples/s
+Step 21400/44000 | Loss: 3.0251 | LR: 2.68e-04 | 1782.9ms/step | 11.2 samples/s
+Step 21410/44000 | Loss: 2.9505 | LR: 2.68e-04 | 1783.3ms/step | 11.2 samples/s
+Step 21420/44000 | Loss: 2.9903 | LR: 2.67e-04 | 1784.0ms/step | 11.2 samples/s
+Step 21430/44000 | Loss: 2.9451 | LR: 2.67e-04 | 1783.5ms/step | 11.2 samples/s
+Step 21440/44000 | Loss: 2.9615 | LR: 2.67e-04 | 1784.3ms/step | 11.2 samples/s
+Step 21450/44000 | Loss: 2.9595 | LR: 2.67e-04 | 1783.8ms/step | 11.2 samples/s
+Step 21460/44000 | Loss: 2.9730 | LR: 2.67e-04 | 1783.3ms/step | 11.2 samples/s
+Step 21470/44000 | Loss: 2.9147 | LR: 2.66e-04 | 1784.8ms/step | 11.2 samples/s
+Step 21480/44000 | Loss: 2.9483 | LR: 2.66e-04 | 1783.7ms/step | 11.2 samples/s
+Step 21490/44000 | Loss: 2.9476 | LR: 2.66e-04 | 1783.9ms/step | 11.2 samples/s
+Step 21500/44000 | Loss: 2.9503 | LR: 2.66e-04 | 1783.3ms/step | 11.2 samples/s
+Step 21510/44000 | Loss: 2.9316 | LR: 2.66e-04 | 1783.8ms/step | 11.2 samples/s
+Step 21520/44000 | Loss: 2.9278 | LR: 2.66e-04 | 1783.0ms/step | 11.2 samples/s
+Step 21530/44000 | Loss: 2.9478 | LR: 2.65e-04 | 1784.2ms/step | 11.2 samples/s
+Step 21540/44000 | Loss: 2.9481 | LR: 2.65e-04 | 1783.2ms/step | 11.2 samples/s
+Step 21550/44000 | Loss: 2.9387 | LR: 2.65e-04 | 1784.6ms/step | 11.2 samples/s
+Step 21560/44000 | Loss: 2.9281 | LR: 2.65e-04 | 1784.4ms/step | 11.2 samples/s
+Step 21570/44000 | Loss: 2.9686 | LR: 2.65e-04 | 1783.5ms/step | 11.2 samples/s
+Step 21580/44000 | Loss: 2.9503 | LR: 2.64e-04 | 1783.2ms/step | 11.2 samples/s
+Step 21590/44000 | Loss: 2.9576 | LR: 2.64e-04 | 1784.0ms/step | 11.2 samples/s
+Step 21600/44000 | Loss: 2.9265 | LR: 2.64e-04 | 1783.0ms/step | 11.2 samples/s
+Step 21610/44000 | Loss: 2.9480 | LR: 2.64e-04 | 1783.4ms/step | 11.2 samples/s
+Step 21620/44000 | Loss: 2.9793 | LR: 2.64e-04 | 1782.8ms/step | 11.2 samples/s
+Step 21630/44000 | Loss: 2.9569 | LR: 2.64e-04 | 1782.6ms/step | 11.2 samples/s
+Step 21640/44000 | Loss: 2.9689 | LR: 2.63e-04 | 1782.5ms/step | 11.2 samples/s
+Step 21650/44000 | Loss: 2.9492 | LR: 2.63e-04 | 1782.6ms/step | 11.2 samples/s
+Step 21660/44000 | Loss: 2.9155 | LR: 2.63e-04 | 1784.1ms/step | 11.2 samples/s
+Step 21670/44000 | Loss: 2.9601 | LR: 2.63e-04 | 1783.5ms/step | 11.2 samples/s
+Step 21680/44000 | Loss: 2.9736 | LR: 2.63e-04 | 1783.7ms/step | 11.2 samples/s
+Step 21690/44000 | Loss: 2.9557 | LR: 2.62e-04 | 1783.2ms/step | 11.2 samples/s
+Step 21700/44000 | Loss: 2.9446 | LR: 2.62e-04 | 1784.6ms/step | 11.2 samples/s
+Step 21710/44000 | Loss: 2.9369 | LR: 2.62e-04 | 1784.9ms/step | 11.2 samples/s
+Step 21720/44000 | Loss: 3.0092 | LR: 2.62e-04 | 1784.3ms/step | 11.2 samples/s
+Step 21730/44000 | Loss: 2.9173 | LR: 2.62e-04 | 1784.2ms/step | 11.2 samples/s
+Step 21740/44000 | Loss: 2.9280 | LR: 2.62e-04 | 1784.7ms/step | 11.2 samples/s
+Step 21750/44000 | Loss: 2.9393 | LR: 2.61e-04 | 1784.1ms/step | 11.2 samples/s
+Step 21760/44000 | Loss: 2.9398 | LR: 2.61e-04 | 1785.0ms/step | 11.2 samples/s
+Step 21770/44000 | Loss: 2.9675 | LR: 2.61e-04 | 1785.3ms/step | 11.2 samples/s
+Step 21780/44000 | Loss: 2.9116 | LR: 2.61e-04 | 1785.5ms/step | 11.2 samples/s
+Step 21790/44000 | Loss: 2.9578 | LR: 2.61e-04 | 1784.4ms/step | 11.2 samples/s
+Step 21800/44000 | Loss: 2.8972 | LR: 2.60e-04 | 1785.4ms/step | 11.2 samples/s
+Step 21810/44000 | Loss: 2.9899 | LR: 2.60e-04 | 1784.3ms/step | 11.2 samples/s
+Step 21820/44000 | Loss: 2.9564 | LR: 2.60e-04 | 1784.9ms/step | 11.2 samples/s
+Step 21830/44000 | Loss: 2.9341 | LR: 2.60e-04 | 1784.9ms/step | 11.2 samples/s
+Step 21840/44000 | Loss: 2.9223 | LR: 2.60e-04 | 1784.8ms/step | 11.2 samples/s
+Step 21850/44000 | Loss: 2.9339 | LR: 2.60e-04 | 1784.1ms/step | 11.2 samples/s
+Step 21860/44000 | Loss: 2.9618 | LR: 2.59e-04 | 1784.8ms/step | 11.2 samples/s
+Step 21870/44000 | Loss: 2.9161 | LR: 2.59e-04 | 1785.1ms/step | 11.2 samples/s
+Step 21880/44000 | Loss: 2.9389 | LR: 2.59e-04 | 1784.0ms/step | 11.2 samples/s
+Step 21890/44000 | Loss: 2.9089 | LR: 2.59e-04 | 1784.0ms/step | 11.2 samples/s
+Step 21900/44000 | Loss: 2.9641 | LR: 2.59e-04 | 1784.0ms/step | 11.2 samples/s
+Step 21910/44000 | Loss: 2.9514 | LR: 2.58e-04 | 1784.2ms/step | 11.2 samples/s
+Step 21920/44000 | Loss: 2.9250 | LR: 2.58e-04 | 1783.8ms/step | 11.2 samples/s
+Step 21930/44000 | Loss: 2.9058 | LR: 2.58e-04 | 1784.1ms/step | 11.2 samples/s
+Step 21940/44000 | Loss: 2.9537 | LR: 2.58e-04 | 1784.4ms/step | 11.2 samples/s
+Step 21950/44000 | Loss: 2.9575 | LR: 2.58e-04 | 1785.0ms/step | 11.2 samples/s
+Step 21960/44000 | Loss: 2.9640 | LR: 2.58e-04 | 1784.0ms/step | 11.2 samples/s
+Step 21970/44000 | Loss: 2.9483 | LR: 2.57e-04 | 1784.1ms/step | 11.2 samples/s
+Step 21980/44000 | Loss: 2.9637 | LR: 2.57e-04 | 1783.8ms/step | 11.2 samples/s
+Step 21990/44000 | Loss: 2.9575 | LR: 2.57e-04 | 1783.3ms/step | 11.2 samples/s
+Step 22000/44000 | Loss: 2.9742 | LR: 2.57e-04 | 1782.5ms/step | 11.2 samples/s
+Validation at step 22000: Loss = 2.9299, Perplexity = 18.73
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-22000
+Step 22010/44000 | Loss: 2.9529 | LR: 2.57e-04 | 3598.0ms/step | 5.6 samples/s
+Step 22020/44000 | Loss: 2.9475 | LR: 2.56e-04 | 1782.7ms/step | 11.2 samples/s
+Step 22030/44000 | Loss: 2.9249 | LR: 2.56e-04 | 1783.3ms/step | 11.2 samples/s
+Step 22040/44000 | Loss: 2.9355 | LR: 2.56e-04 | 1783.9ms/step | 11.2 samples/s
+Step 22050/44000 | Loss: 2.9719 | LR: 2.56e-04 | 1783.8ms/step | 11.2 samples/s
+Step 22060/44000 | Loss: 2.9282 | LR: 2.56e-04 | 1784.5ms/step | 11.2 samples/s
+Step 22070/44000 | Loss: 2.9475 | LR: 2.56e-04 | 1783.7ms/step | 11.2 samples/s
+Step 22080/44000 | Loss: 2.9194 | LR: 2.55e-04 | 1784.2ms/step | 11.2 samples/s
+Step 22090/44000 | Loss: 2.9284 | LR: 2.55e-04 | 1784.9ms/step | 11.2 samples/s
+Step 22100/44000 | Loss: 2.9592 | LR: 2.55e-04 | 1784.2ms/step | 11.2 samples/s
+Step 22110/44000 | Loss: 2.9502 | LR: 2.55e-04 | 1784.3ms/step | 11.2 samples/s
+Step 22120/44000 | Loss: 2.9309 | LR: 2.55e-04 | 1784.9ms/step | 11.2 samples/s
+Step 22130/44000 | Loss: 2.9673 | LR: 2.54e-04 | 1785.1ms/step | 11.2 samples/s
+Step 22140/44000 | Loss: 2.9595 | LR: 2.54e-04 | 1786.3ms/step | 11.2 samples/s
+Step 22150/44000 | Loss: 2.9657 | LR: 2.54e-04 | 1785.5ms/step | 11.2 samples/s
+Step 22160/44000 | Loss: 2.9435 | LR: 2.54e-04 | 1783.9ms/step | 11.2 samples/s
+Step 22170/44000 | Loss: 2.9544 | LR: 2.54e-04 | 1784.4ms/step | 11.2 samples/s
+Step 22180/44000 | Loss: 2.9317 | LR: 2.54e-04 | 1785.0ms/step | 11.2 samples/s
+Step 22190/44000 | Loss: 2.9142 | LR: 2.53e-04 | 1785.3ms/step | 11.2 samples/s
+Step 22200/44000 | Loss: 2.9581 | LR: 2.53e-04 | 1784.3ms/step | 11.2 samples/s
+Step 22210/44000 | Loss: 2.8973 | LR: 2.53e-04 | 1784.9ms/step | 11.2 samples/s
+Step 22220/44000 | Loss: 2.9223 | LR: 2.53e-04 | 1784.1ms/step | 11.2 samples/s
+Step 22230/44000 | Loss: 2.9138 | LR: 2.53e-04 | 1784.9ms/step | 11.2 samples/s
+Step 22240/44000 | Loss: 2.9175 | LR: 2.52e-04 | 1783.4ms/step | 11.2 samples/s
+Step 22250/44000 | Loss: 2.9396 | LR: 2.52e-04 | 1784.4ms/step | 11.2 samples/s
+Step 22260/44000 | Loss: 2.9605 | LR: 2.52e-04 | 1783.5ms/step | 11.2 samples/s
+Step 22270/44000 | Loss: 2.9301 | LR: 2.52e-04 | 1785.1ms/step | 11.2 samples/s
+Step 22280/44000 | Loss: 2.9358 | LR: 2.52e-04 | 1785.0ms/step | 11.2 samples/s
+Step 22290/44000 | Loss: 2.9451 | LR: 2.52e-04 | 1785.5ms/step | 11.2 samples/s
+Step 22300/44000 | Loss: 2.9260 | LR: 2.51e-04 | 1785.0ms/step | 11.2 samples/s
+Step 22310/44000 | Loss: 2.9436 | LR: 2.51e-04 | 1785.3ms/step | 11.2 samples/s
+Step 22320/44000 | Loss: 2.8736 | LR: 2.51e-04 | 1786.8ms/step | 11.2 samples/s
+Step 22330/44000 | Loss: 2.9177 | LR: 2.51e-04 | 1786.5ms/step | 11.2 samples/s
+Step 22340/44000 | Loss: 2.9895 | LR: 2.51e-04 | 1786.3ms/step | 11.2 samples/s
+Step 22350/44000 | Loss: 2.9402 | LR: 2.50e-04 | 1786.7ms/step | 11.2 samples/s
+Step 22360/44000 | Loss: 2.8859 | LR: 2.50e-04 | 1787.8ms/step | 11.2 samples/s
+Step 22370/44000 | Loss: 2.9543 | LR: 2.50e-04 | 1784.5ms/step | 11.2 samples/s
+Step 22380/44000 | Loss: 2.9313 | LR: 2.50e-04 | 1784.7ms/step | 11.2 samples/s
+Step 22390/44000 | Loss: 2.9229 | LR: 2.50e-04 | 1785.1ms/step | 11.2 samples/s
+Step 22400/44000 | Loss: 2.9187 | LR: 2.50e-04 | 1785.9ms/step | 11.2 samples/s
+Step 22410/44000 | Loss: 2.9282 | LR: 2.49e-04 | 1783.9ms/step | 11.2 samples/s
+Step 22420/44000 | Loss: 2.9421 | LR: 2.49e-04 | 1783.1ms/step | 11.2 samples/s
+Step 22430/44000 | Loss: 2.9235 | LR: 2.49e-04 | 1783.7ms/step | 11.2 samples/s
+Step 22440/44000 | Loss: 2.9476 | LR: 2.49e-04 | 1784.7ms/step | 11.2 samples/s
+Step 22450/44000 | Loss: 2.9239 | LR: 2.49e-04 | 1785.0ms/step | 11.2 samples/s
+Step 22460/44000 | Loss: 2.9868 | LR: 2.48e-04 | 1784.7ms/step | 11.2 samples/s
+Step 22470/44000 | Loss: 2.9359 | LR: 2.48e-04 | 1785.0ms/step | 11.2 samples/s
+Step 22480/44000 | Loss: 2.9391 | LR: 2.48e-04 | 1786.1ms/step | 11.2 samples/s
+Step 22490/44000 | Loss: 2.9371 | LR: 2.48e-04 | 1784.2ms/step | 11.2 samples/s
+Step 22500/44000 | Loss: 2.9759 | LR: 2.48e-04 | 1784.7ms/step | 11.2 samples/s
+Step 22510/44000 | Loss: 2.9478 | LR: 2.48e-04 | 1784.6ms/step | 11.2 samples/s
+Step 22520/44000 | Loss: 2.9331 | LR: 2.47e-04 | 1784.4ms/step | 11.2 samples/s
+Step 22530/44000 | Loss: 2.9508 | LR: 2.47e-04 | 1784.3ms/step | 11.2 samples/s
+Step 22540/44000 | Loss: 2.9616 | LR: 2.47e-04 | 1785.1ms/step | 11.2 samples/s
+Step 22550/44000 | Loss: 2.9633 | LR: 2.47e-04 | 1786.1ms/step | 11.2 samples/s
+Step 22560/44000 | Loss: 2.9919 | LR: 2.47e-04 | 1784.5ms/step | 11.2 samples/s
+Step 22570/44000 | Loss: 2.9654 | LR: 2.46e-04 | 1786.2ms/step | 11.2 samples/s
+Step 22580/44000 | Loss: 2.9429 | LR: 2.46e-04 | 1786.2ms/step | 11.2 samples/s
+Step 22590/44000 | Loss: 3.0019 | LR: 2.46e-04 | 1788.8ms/step | 11.2 samples/s
+Step 22600/44000 | Loss: 2.9478 | LR: 2.46e-04 | 1785.6ms/step | 11.2 samples/s
+Step 22610/44000 | Loss: 2.9274 | LR: 2.46e-04 | 1785.8ms/step | 11.2 samples/s
+Step 22620/44000 | Loss: 2.9556 | LR: 2.46e-04 | 1785.0ms/step | 11.2 samples/s
+Step 22630/44000 | Loss: 2.9437 | LR: 2.45e-04 | 1785.8ms/step | 11.2 samples/s
+Step 22640/44000 | Loss: 2.9011 | LR: 2.45e-04 | 1785.5ms/step | 11.2 samples/s
+Step 22650/44000 | Loss: 2.9521 | LR: 2.45e-04 | 1787.7ms/step | 11.2 samples/s
+Step 22660/44000 | Loss: 2.9198 | LR: 2.45e-04 | 1786.1ms/step | 11.2 samples/s
+Step 22670/44000 | Loss: 2.9658 | LR: 2.45e-04 | 1785.7ms/step | 11.2 samples/s
+Step 22680/44000 | Loss: 2.9353 | LR: 2.44e-04 | 1785.8ms/step | 11.2 samples/s
+Step 22690/44000 | Loss: 2.9288 | LR: 2.44e-04 | 1785.4ms/step | 11.2 samples/s
+Step 22700/44000 | Loss: 2.9867 | LR: 2.44e-04 | 1784.9ms/step | 11.2 samples/s
+Step 22710/44000 | Loss: 2.9050 | LR: 2.44e-04 | 1784.9ms/step | 11.2 samples/s
+Step 22720/44000 | Loss: 3.0138 | LR: 2.44e-04 | 1785.8ms/step | 11.2 samples/s
+Step 22730/44000 | Loss: 2.9125 | LR: 2.44e-04 | 1786.2ms/step | 11.2 samples/s
+Step 22740/44000 | Loss: 2.9592 | LR: 2.43e-04 | 1784.8ms/step | 11.2 samples/s
+Step 22750/44000 | Loss: 2.9658 | LR: 2.43e-04 | 1785.7ms/step | 11.2 samples/s
+Step 22760/44000 | Loss: 2.9370 | LR: 2.43e-04 | 1785.6ms/step | 11.2 samples/s
+Step 22770/44000 | Loss: 2.9834 | LR: 2.43e-04 | 1784.4ms/step | 11.2 samples/s
+Step 22780/44000 | Loss: 2.9316 | LR: 2.43e-04 | 1783.8ms/step | 11.2 samples/s
+Step 22790/44000 | Loss: 2.9402 | LR: 2.42e-04 | 1783.8ms/step | 11.2 samples/s
+Step 22800/44000 | Loss: 2.9216 | LR: 2.42e-04 | 1782.9ms/step | 11.2 samples/s
+Step 22810/44000 | Loss: 2.9279 | LR: 2.42e-04 | 1782.6ms/step | 11.2 samples/s
+Step 22820/44000 | Loss: 2.9985 | LR: 2.42e-04 | 1785.0ms/step | 11.2 samples/s
+Step 22830/44000 | Loss: 2.9255 | LR: 2.42e-04 | 1784.7ms/step | 11.2 samples/s
+Step 22840/44000 | Loss: 2.9068 | LR: 2.42e-04 | 1785.3ms/step | 11.2 samples/s
+Step 22850/44000 | Loss: 2.9259 | LR: 2.41e-04 | 1785.6ms/step | 11.2 samples/s
+Step 22860/44000 | Loss: 2.9009 | LR: 2.41e-04 | 1790.5ms/step | 11.2 samples/s
+Step 22870/44000 | Loss: 2.9616 | LR: 2.41e-04 | 1788.9ms/step | 11.2 samples/s
+Step 22880/44000 | Loss: 2.9366 | LR: 2.41e-04 | 1786.5ms/step | 11.2 samples/s
+Step 22890/44000 | Loss: 2.9706 | LR: 2.41e-04 | 1784.9ms/step | 11.2 samples/s
+Step 22900/44000 | Loss: 2.9555 | LR: 2.40e-04 | 1787.0ms/step | 11.2 samples/s
+Step 22910/44000 | Loss: 2.9683 | LR: 2.40e-04 | 1786.4ms/step | 11.2 samples/s
+Step 22920/44000 | Loss: 2.9725 | LR: 2.40e-04 | 1785.6ms/step | 11.2 samples/s
+Step 22930/44000 | Loss: 2.9169 | LR: 2.40e-04 | 1784.8ms/step | 11.2 samples/s
+Step 22940/44000 | Loss: 2.9324 | LR: 2.40e-04 | 1785.0ms/step | 11.2 samples/s
+Step 22950/44000 | Loss: 2.9440 | LR: 2.40e-04 | 1785.0ms/step | 11.2 samples/s
+Step 22960/44000 | Loss: 2.9002 | LR: 2.39e-04 | 1785.5ms/step | 11.2 samples/s
+Step 22970/44000 | Loss: 2.8904 | LR: 2.39e-04 | 1786.8ms/step | 11.2 samples/s
+Step 22980/44000 | Loss: 2.9205 | LR: 2.39e-04 | 1786.6ms/step | 11.2 samples/s
+Step 22990/44000 | Loss: 2.9192 | LR: 2.39e-04 | 1785.3ms/step | 11.2 samples/s
+Step 23000/44000 | Loss: 2.9285 | LR: 2.39e-04 | 1785.5ms/step | 11.2 samples/s
+Validation at step 23000: Loss = 2.9212, Perplexity = 18.56
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-23000
+Step 23010/44000 | Loss: 2.9101 | LR: 2.38e-04 | 3602.2ms/step | 5.6 samples/s
+Step 23020/44000 | Loss: 2.8920 | LR: 2.38e-04 | 1785.4ms/step | 11.2 samples/s
+Step 23030/44000 | Loss: 2.8955 | LR: 2.38e-04 | 1784.7ms/step | 11.2 samples/s
+Step 23040/44000 | Loss: 2.9346 | LR: 2.38e-04 | 1786.1ms/step | 11.2 samples/s
+Step 23050/44000 | Loss: 2.9574 | LR: 2.38e-04 | 1786.2ms/step | 11.2 samples/s
+Step 23060/44000 | Loss: 2.9709 | LR: 2.38e-04 | 1786.1ms/step | 11.2 samples/s
+Step 23070/44000 | Loss: 2.9376 | LR: 2.37e-04 | 1786.7ms/step | 11.2 samples/s
+Step 23080/44000 | Loss: 2.9551 | LR: 2.37e-04 | 1786.4ms/step | 11.2 samples/s
+Step 23090/44000 | Loss: 2.9186 | LR: 2.37e-04 | 1786.1ms/step | 11.2 samples/s
+Step 23100/44000 | Loss: 2.9119 | LR: 2.37e-04 | 1786.6ms/step | 11.2 samples/s
+Step 23110/44000 | Loss: 2.9158 | LR: 2.37e-04 | 1787.6ms/step | 11.2 samples/s
+Step 23120/44000 | Loss: 2.9540 | LR: 2.36e-04 | 1787.1ms/step | 11.2 samples/s
+Step 23130/44000 | Loss: 2.9618 | LR: 2.36e-04 | 1788.0ms/step | 11.2 samples/s
+Step 23140/44000 | Loss: 2.9273 | LR: 2.36e-04 | 1786.1ms/step | 11.2 samples/s
+Step 23150/44000 | Loss: 2.9128 | LR: 2.36e-04 | 1786.9ms/step | 11.2 samples/s
+Step 23160/44000 | Loss: 2.9295 | LR: 2.36e-04 | 1786.9ms/step | 11.2 samples/s
+Step 23170/44000 | Loss: 2.9265 | LR: 2.36e-04 | 1788.0ms/step | 11.2 samples/s
+Step 23180/44000 | Loss: 2.9517 | LR: 2.35e-04 | 1785.6ms/step | 11.2 samples/s
+Step 23190/44000 | Loss: 2.9260 | LR: 2.35e-04 | 1784.9ms/step | 11.2 samples/s
+Step 23200/44000 | Loss: 2.9469 | LR: 2.35e-04 | 1787.5ms/step | 11.2 samples/s
+Step 23210/44000 | Loss: 2.9487 | LR: 2.35e-04 | 1787.4ms/step | 11.2 samples/s
+Step 23220/44000 | Loss: 2.9608 | LR: 2.35e-04 | 1786.7ms/step | 11.2 samples/s
+Step 23230/44000 | Loss: 2.9214 | LR: 2.34e-04 | 1786.5ms/step | 11.2 samples/s
+Step 23240/44000 | Loss: 2.9871 | LR: 2.34e-04 | 1787.2ms/step | 11.2 samples/s
+Step 23250/44000 | Loss: 2.9293 | LR: 2.34e-04 | 1787.2ms/step | 11.2 samples/s
+Step 23260/44000 | Loss: 2.9407 | LR: 2.34e-04 | 1786.7ms/step | 11.2 samples/s
+Step 23270/44000 | Loss: 2.9158 | LR: 2.34e-04 | 1787.1ms/step | 11.2 samples/s
+Step 23280/44000 | Loss: 2.8751 | LR: 2.34e-04 | 1786.5ms/step | 11.2 samples/s
+Step 23290/44000 | Loss: 2.8713 | LR: 2.33e-04 | 1786.0ms/step | 11.2 samples/s
+Step 23300/44000 | Loss: 2.8930 | LR: 2.33e-04 | 1786.0ms/step | 11.2 samples/s
+Step 23310/44000 | Loss: 2.9444 | LR: 2.33e-04 | 1787.5ms/step | 11.2 samples/s
+Step 23320/44000 | Loss: 2.9697 | LR: 2.33e-04 | 1786.9ms/step | 11.2 samples/s
+Step 23330/44000 | Loss: 2.9259 | LR: 2.33e-04 | 1786.5ms/step | 11.2 samples/s
+Step 23340/44000 | Loss: 2.8984 | LR: 2.32e-04 | 1786.5ms/step | 11.2 samples/s
+Step 23350/44000 | Loss: 2.9234 | LR: 2.32e-04 | 1787.3ms/step | 11.2 samples/s
+Step 23360/44000 | Loss: 2.8995 | LR: 2.32e-04 | 1785.6ms/step | 11.2 samples/s
+Step 23370/44000 | Loss: 2.9135 | LR: 2.32e-04 | 1787.1ms/step | 11.2 samples/s
+Step 23380/44000 | Loss: 2.9199 | LR: 2.32e-04 | 1786.7ms/step | 11.2 samples/s
+Step 23390/44000 | Loss: 2.9365 | LR: 2.32e-04 | 1787.1ms/step | 11.2 samples/s
+Step 23400/44000 | Loss: 2.9408 | LR: 2.31e-04 | 1784.5ms/step | 11.2 samples/s
+Step 23410/44000 | Loss: 2.9225 | LR: 2.31e-04 | 1785.2ms/step | 11.2 samples/s
+Step 23420/44000 | Loss: 2.9611 | LR: 2.31e-04 | 1786.1ms/step | 11.2 samples/s
+Step 23430/44000 | Loss: 2.9693 | LR: 2.31e-04 | 1787.0ms/step | 11.2 samples/s
+Step 23440/44000 | Loss: 2.9681 | LR: 2.31e-04 | 1785.6ms/step | 11.2 samples/s
+Step 23450/44000 | Loss: 2.9283 | LR: 2.30e-04 | 1786.4ms/step | 11.2 samples/s
+Step 23460/44000 | Loss: 2.8931 | LR: 2.30e-04 | 1786.5ms/step | 11.2 samples/s
+Step 23470/44000 | Loss: 2.8925 | LR: 2.30e-04 | 1786.6ms/step | 11.2 samples/s
+Step 23480/44000 | Loss: 2.9322 | LR: 2.30e-04 | 1785.6ms/step | 11.2 samples/s
+Step 23490/44000 | Loss: 2.9570 | LR: 2.30e-04 | 1786.6ms/step | 11.2 samples/s
+Step 23500/44000 | Loss: 2.9036 | LR: 2.30e-04 | 1787.3ms/step | 11.2 samples/s
+Step 23510/44000 | Loss: 2.9557 | LR: 2.29e-04 | 1787.3ms/step | 11.2 samples/s
+Step 23520/44000 | Loss: 2.9466 | LR: 2.29e-04 | 1788.2ms/step | 11.2 samples/s
+Step 23530/44000 | Loss: 2.9419 | LR: 2.29e-04 | 1787.3ms/step | 11.2 samples/s
+Step 23540/44000 | Loss: 2.9325 | LR: 2.29e-04 | 1787.0ms/step | 11.2 samples/s
+Step 23550/44000 | Loss: 2.9084 | LR: 2.29e-04 | 1786.5ms/step | 11.2 samples/s
+Step 23560/44000 | Loss: 2.9623 | LR: 2.29e-04 | 1787.9ms/step | 11.2 samples/s
+Step 23570/44000 | Loss: 2.9347 | LR: 2.28e-04 | 1786.7ms/step | 11.2 samples/s
+Step 23580/44000 | Loss: 2.9231 | LR: 2.28e-04 | 1787.4ms/step | 11.2 samples/s
+Step 23590/44000 | Loss: 2.9382 | LR: 2.28e-04 | 1786.6ms/step | 11.2 samples/s
+Step 23600/44000 | Loss: 2.9333 | LR: 2.28e-04 | 1786.1ms/step | 11.2 samples/s
+Step 23610/44000 | Loss: 2.9002 | LR: 2.28e-04 | 1786.1ms/step | 11.2 samples/s
+Step 23620/44000 | Loss: 2.9147 | LR: 2.27e-04 | 1786.8ms/step | 11.2 samples/s
+Step 23630/44000 | Loss: 2.9571 | LR: 2.27e-04 | 1786.0ms/step | 11.2 samples/s
+Step 23640/44000 | Loss: 2.9667 | LR: 2.27e-04 | 1788.0ms/step | 11.2 samples/s
+Step 23650/44000 | Loss: 2.9068 | LR: 2.27e-04 | 1787.4ms/step | 11.2 samples/s
+Step 23660/44000 | Loss: 2.9729 | LR: 2.27e-04 | 1789.0ms/step | 11.2 samples/s
+Step 23670/44000 | Loss: 2.9478 | LR: 2.27e-04 | 1787.2ms/step | 11.2 samples/s
+Step 23680/44000 | Loss: 2.9451 | LR: 2.26e-04 | 1787.1ms/step | 11.2 samples/s
+Step 23690/44000 | Loss: 2.9159 | LR: 2.26e-04 | 1787.7ms/step | 11.2 samples/s
+Step 23700/44000 | Loss: 2.9361 | LR: 2.26e-04 | 1788.2ms/step | 11.2 samples/s
+Step 23710/44000 | Loss: 2.9220 | LR: 2.26e-04 | 1787.5ms/step | 11.2 samples/s
+Step 23720/44000 | Loss: 2.9507 | LR: 2.26e-04 | 1786.4ms/step | 11.2 samples/s
+Step 23730/44000 | Loss: 2.9063 | LR: 2.25e-04 | 1785.3ms/step | 11.2 samples/s
+Step 23740/44000 | Loss: 2.9326 | LR: 2.25e-04 | 1783.9ms/step | 11.2 samples/s
+Step 23750/44000 | Loss: 2.9249 | LR: 2.25e-04 | 1786.6ms/step | 11.2 samples/s
+Step 23760/44000 | Loss: 2.9178 | LR: 2.25e-04 | 1786.6ms/step | 11.2 samples/s
+Step 23770/44000 | Loss: 2.9238 | LR: 2.25e-04 | 1787.6ms/step | 11.2 samples/s
+Step 23780/44000 | Loss: 2.9490 | LR: 2.25e-04 | 1787.0ms/step | 11.2 samples/s
+Step 23790/44000 | Loss: 2.9312 | LR: 2.24e-04 | 1787.2ms/step | 11.2 samples/s
+Step 23800/44000 | Loss: 2.9404 | LR: 2.24e-04 | 1786.2ms/step | 11.2 samples/s
+Step 23810/44000 | Loss: 2.9420 | LR: 2.24e-04 | 1786.6ms/step | 11.2 samples/s
+Step 23820/44000 | Loss: 2.9315 | LR: 2.24e-04 | 1785.1ms/step | 11.2 samples/s
+Step 23830/44000 | Loss: 2.9264 | LR: 2.24e-04 | 1784.9ms/step | 11.2 samples/s
+Step 23840/44000 | Loss: 2.9111 | LR: 2.23e-04 | 1784.6ms/step | 11.2 samples/s
+Step 23850/44000 | Loss: 2.9539 | LR: 2.23e-04 | 1785.7ms/step | 11.2 samples/s
+Step 23860/44000 | Loss: 2.9504 | LR: 2.23e-04 | 1786.9ms/step | 11.2 samples/s
+Step 23870/44000 | Loss: 2.9326 | LR: 2.23e-04 | 1786.3ms/step | 11.2 samples/s
+Step 23880/44000 | Loss: 2.9230 | LR: 2.23e-04 | 1786.0ms/step | 11.2 samples/s
+Step 23890/44000 | Loss: 2.9233 | LR: 2.23e-04 | 1786.8ms/step | 11.2 samples/s

epfl_final/train_logs/ropectl_a1p0_h100_resume/seed42.20260820T041452Z.log

@@ -0,0 +1,647 @@
+Auto-resuming from latest checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-13000
+Resuming: Using existing output directory /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735
+Running config:
+{
+  "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
+  "model_dtype": "bfloat16",
+  "init_from_scratch": true,
+  "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
+  "val_path": null,
+  "max_seq_length": 8192,
+  "num_workers": 8,
+  "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735",
+  "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-13000",
+  "force_new_run": false,
+  "num_epochs": 1,
+  "max_steps": 44000,
+  "per_device_batch_size": 2,
+  "gradient_accumulation_steps": 10,
+  "learning_rate": 0.0005,
+  "weight_decay": 0.01,
+  "warmup_steps": 750,
+  "lr_scheduler": "cosine",
+  "max_grad_norm": 1.0,
+  "save_steps": 1000,
+  "eval_steps": 1000,
+  "eval_batches": 50,
+  "logging_steps": 10,
+  "seed": 42,
+  "verbose": true,
+  "checkpoint_activations": false,
+  "strategy": "ddp",
+  "mixed_precision": "bf16",
+  "wandb_enabled": true,
+  "wandb_project": "stochastic_sparsity",
+  "wandb_name": "l8k365m_dense_ropectl_a1p0_seed42",
+  "wandb_tags": [
+    "qwen3",
+    "365m",
+    "8k",
+    "dense",
+    "rope-control"
+  ],
+  "hidden_size": 768,
+  "intermediate_size": 3072,
+  "num_attention_heads": 6,
+  "num_key_value_heads": 2,
+  "num_hidden_layers": 28,
+  "head_dim": null,
+  "attention_dropout": 0.0,
+  "rope_train_position_scale_set": [
+    1.0,
+    0.9896339488176222,
+    0.9692159692159692,
+    0.9420289855072463,
+    0.8730158730158729
+  ],
+  "attention_type": "sparse_stochastic",
+  "sparse_block_size": 128,
+  "sparse_num_random_blocks": 31,
+  "sparse_mask_pool_size": 1,
+  "sparse_val_dense_compare": false,
+  "sparse_layer_indices": null,
+  "sparse_mask_cache_dir": null,
+  "sparse_mask_seed": 0,
+  "sparse_mixed_nrb_set": null,
+  "dense_mode": true,
+  "dense_step_intervals": null
+}
+Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
+Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-13000
+Checkpoint activations: False
+dense_mode=True: sparse attention layers will use standard causal SDPA.
+
+
+================================================================================
+MODEL ARCHITECTURE
+================================================================================
+Model config:
+  vocab_size: 151936
+  hidden_size: 768
+  num_hidden_layers: 28
+  num_attention_heads: 6
+  num_key_value_heads: 2
+  head_dim: 128
+  intermediate_size: 3072
+  max_position_embeddings: 32768
+  rope_theta: 1000000
+  rope_train_position_scale_set: [1.0, 0.9896339488176222, 0.9692159692159692, 0.9420289855072463, 0.8730158730158729]
+  attention_type: sparse_stochastic
+  sparse_block_size: 128
+  sparse_num_random_blocks: 31
+  sparse_mask_pool_size: 1
+  sparse_max_seq_length: 8192
+
+Parameter count:
+  Total parameters: 358,958,848
+  Trainable parameters: 358,958,848
+  Non-trainable parameters: 0
+  Model size (BF16): 0.72 GB
+
+--------------------------------------------------------------------------------
+LAYER-BY-LAYER SUMMARY
+--------------------------------------------------------------------------------
+================================================================================================================================================================
+Layer (type:depth-idx)                                       Input Shape               Output Shape              Param #                   Trainable
+================================================================================================================================================================
+Qwen3ForCausalLM                                             [2, 8192]                 --                        --                        True
+├─Qwen3Model: 1-1                                            --                        [2, 8192, 768]            --                        True
+│    └─Embedding: 2-1                                        [2, 8192]                 [2, 8192, 768]            116,686,848               True
+│    └─ModuleList: 2-2                                       --                        --                        --                        True
+│    │    └─Qwen3DecoderLayer: 3-1                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-2                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-3                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-4                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-5                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-6                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-7                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-8                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-9                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-10                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-11                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-12                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-13                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-14                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-15                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-16                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-17                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-18                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-19                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-20                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-21                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-22                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-23                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-24                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-25                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-26                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-27                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-28                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    └─Qwen3RMSNorm: 2-3                                     [2, 8192, 768]            [2, 8192, 768]            768                       True
+├─Linear: 1-2                                                [2, 8192, 768]            [2, 8192, 151936]         116,686,848               True
+================================================================================================================================================================
+Total params: 475,645,696
+Trainable params: 475,645,696
+Non-trainable params: 0
+Total mult-adds (Units.MEGABYTES): 951.29
+================================================================================================================================================================
+Input size (MB): 0.13
+Forward/backward pass size (MB): 31197.23
+Params size (MB): 951.29
+Estimated Total Size (MB): 32148.66
+================================================================================================================================================================
+  NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
+================================================================================
+
+Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
+Using pre-tokenized dataset (no runtime tokenization)
+Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
+Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
+Dataset split: 3027764 train, 159356 validation
+Training examples (packed, train split):  3027764
+Validation examples (packed, val split):  159356
+Per-device batch size:                    2
+World size (GPUs):                        1
+Gradient accumulation steps:              10
+Effective batch size (seqs/step):         20
+Sequence length:                          8192
+Tokens per optimizer step:                163,840
+Micro-batches per epoch (per rank):       1513882
+Optimizer steps per epoch:                151388
+Total optimizer steps:                    44000  (capped by max_steps; 0.29 epochs)
+Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-13000...
+Preserving checkpoint LR state for exact resume: current=[0.0004073888929652048], schedule_base=[0.0005]
+Resumed from checkpoint at step 13000 (epoch 0, micro_step 130000)
+Resuming training from step 13000
+Resuming wandb run jqstnvej
+wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from WANDB_API_KEY.
+wandb: Currently logged in as: ruben-navasardyan4869 (ruben-navasardyan4869-epfl) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
+wandb: setting up run jqstnvej
+wandb: Tracking run with wandb version 0.26.1
+wandb: Run data is saved locally in /mloscratch/homes/navasard/sparse/SparseAtt/wandb/run-20260820_041559-jqstnvej
+wandb: Run `wandb offline` to turn off syncing.
+wandb: Resuming run l8k365m_dense_ropectl_a1p0_seed42
+wandb: ⭐️ View project at https://wandb.ai/ruben-navasardyan4869-epfl/stochastic_sparsity
+wandb: 🚀 View run at https://wandb.ai/ruben-navasardyan4869-epfl/stochastic_sparsity/runs/jqstnvej
+Starting training...
+Validation at step 13000: Loss = 3.0665, Perplexity = 21.47
+wandb: WARNING Tried to log to step 13000 that is less than the current step 13021. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Switching attention mode for step 13001: dense (dense step interval)
+Step 13010/44000 | Loss: 3.0287 | LR: 4.07e-04 | 30165.3ms/step | 0.7 samples/s
+wandb: WARNING Tried to log to step 13010 that is less than the current step 13021. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13020/44000 | Loss: 3.0440 | LR: 4.07e-04 | 1375.2ms/step | 14.5 samples/s
+wandb: WARNING Tried to log to step 13020 that is less than the current step 13021. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13030/44000 | Loss: 3.0257 | LR: 4.07e-04 | 1378.1ms/step | 14.5 samples/s
+Step 13040/44000 | Loss: 3.0862 | LR: 4.07e-04 | 1380.1ms/step | 14.5 samples/s
+Step 13050/44000 | Loss: 3.0640 | LR: 4.07e-04 | 1380.4ms/step | 14.5 samples/s
+Step 13060/44000 | Loss: 3.0764 | LR: 4.07e-04 | 1379.9ms/step | 14.5 samples/s
+Step 13070/44000 | Loss: 3.0550 | LR: 4.06e-04 | 1384.0ms/step | 14.5 samples/s
+Step 13080/44000 | Loss: 3.0384 | LR: 4.06e-04 | 1386.1ms/step | 14.4 samples/s
+Step 13090/44000 | Loss: 3.0527 | LR: 4.06e-04 | 1385.1ms/step | 14.4 samples/s
+Step 13100/44000 | Loss: 3.0500 | LR: 4.06e-04 | 1386.4ms/step | 14.4 samples/s
+Step 13110/44000 | Loss: 3.0585 | LR: 4.06e-04 | 1385.3ms/step | 14.4 samples/s
+Step 13120/44000 | Loss: 3.0356 | LR: 4.06e-04 | 1385.5ms/step | 14.4 samples/s
+Step 13130/44000 | Loss: 3.0393 | LR: 4.06e-04 | 1383.5ms/step | 14.5 samples/s
+Step 13140/44000 | Loss: 3.0439 | LR: 4.05e-04 | 1385.2ms/step | 14.4 samples/s
+Step 13150/44000 | Loss: 3.0256 | LR: 4.05e-04 | 1388.8ms/step | 14.4 samples/s
+Step 13160/44000 | Loss: 3.0225 | LR: 4.05e-04 | 1388.1ms/step | 14.4 samples/s
+Step 13170/44000 | Loss: 3.0341 | LR: 4.05e-04 | 1387.4ms/step | 14.4 samples/s
+Step 13180/44000 | Loss: 3.0508 | LR: 4.05e-04 | 1387.3ms/step | 14.4 samples/s
+Step 13190/44000 | Loss: 3.0768 | LR: 4.05e-04 | 1387.4ms/step | 14.4 samples/s
+Step 13200/44000 | Loss: 3.0454 | LR: 4.05e-04 | 1388.1ms/step | 14.4 samples/s
+Step 13210/44000 | Loss: 3.0508 | LR: 4.04e-04 | 1387.9ms/step | 14.4 samples/s
+Step 13220/44000 | Loss: 3.0491 | LR: 4.04e-04 | 1389.3ms/step | 14.4 samples/s
+Step 13230/44000 | Loss: 3.0392 | LR: 4.04e-04 | 1390.9ms/step | 14.4 samples/s
+Step 13240/44000 | Loss: 3.0731 | LR: 4.04e-04 | 1387.6ms/step | 14.4 samples/s
+Step 13250/44000 | Loss: 3.0547 | LR: 4.04e-04 | 1386.2ms/step | 14.4 samples/s
+Step 13260/44000 | Loss: 3.0456 | LR: 4.04e-04 | 1387.3ms/step | 14.4 samples/s
+Step 13270/44000 | Loss: 3.0193 | LR: 4.04e-04 | 1386.3ms/step | 14.4 samples/s
+Step 13280/44000 | Loss: 3.0294 | LR: 4.03e-04 | 1388.8ms/step | 14.4 samples/s
+Step 13290/44000 | Loss: 3.0476 | LR: 4.03e-04 | 1386.3ms/step | 14.4 samples/s
+Step 13300/44000 | Loss: 3.0134 | LR: 4.03e-04 | 1387.1ms/step | 14.4 samples/s
+Step 13310/44000 | Loss: 3.0769 | LR: 4.03e-04 | 1388.9ms/step | 14.4 samples/s
+Step 13320/44000 | Loss: 3.0242 | LR: 4.03e-04 | 1391.2ms/step | 14.4 samples/s
+Step 13330/44000 | Loss: 3.0419 | LR: 4.03e-04 | 1390.2ms/step | 14.4 samples/s
+Step 13340/44000 | Loss: 3.0476 | LR: 4.03e-04 | 1389.4ms/step | 14.4 samples/s
+Step 13350/44000 | Loss: 3.0307 | LR: 4.02e-04 | 1387.8ms/step | 14.4 samples/s
+Step 13360/44000 | Loss: 3.0297 | LR: 4.02e-04 | 1388.4ms/step | 14.4 samples/s
+Step 13370/44000 | Loss: 3.0360 | LR: 4.02e-04 | 1386.4ms/step | 14.4 samples/s
+Step 13380/44000 | Loss: 3.0373 | LR: 4.02e-04 | 1387.7ms/step | 14.4 samples/s
+Step 13390/44000 | Loss: 3.0441 | LR: 4.02e-04 | 1389.9ms/step | 14.4 samples/s
+Step 13400/44000 | Loss: 3.0488 | LR: 4.02e-04 | 1388.0ms/step | 14.4 samples/s
+Step 13410/44000 | Loss: 3.0572 | LR: 4.02e-04 | 1384.0ms/step | 14.5 samples/s
+Step 13420/44000 | Loss: 3.0663 | LR: 4.01e-04 | 1384.1ms/step | 14.5 samples/s
+Step 13430/44000 | Loss: 3.0234 | LR: 4.01e-04 | 1385.7ms/step | 14.4 samples/s
+Step 13440/44000 | Loss: 3.0172 | LR: 4.01e-04 | 1389.9ms/step | 14.4 samples/s
+Step 13450/44000 | Loss: 3.0330 | LR: 4.01e-04 | 1386.2ms/step | 14.4 samples/s
+Step 13460/44000 | Loss: 3.0628 | LR: 4.01e-04 | 1386.8ms/step | 14.4 samples/s
+Step 13470/44000 | Loss: 3.0356 | LR: 4.01e-04 | 1389.4ms/step | 14.4 samples/s
+Step 13480/44000 | Loss: 3.0476 | LR: 4.01e-04 | 1392.8ms/step | 14.4 samples/s
+Step 13490/44000 | Loss: 3.0474 | LR: 4.00e-04 | 1387.1ms/step | 14.4 samples/s
+Step 13500/44000 | Loss: 3.0593 | LR: 4.00e-04 | 1387.7ms/step | 14.4 samples/s
+Step 13510/44000 | Loss: 3.0408 | LR: 4.00e-04 | 1386.8ms/step | 14.4 samples/s
+Step 13520/44000 | Loss: 3.0588 | LR: 4.00e-04 | 1386.9ms/step | 14.4 samples/s
+Step 13530/44000 | Loss: 3.0276 | LR: 4.00e-04 | 1387.7ms/step | 14.4 samples/s
+Step 13540/44000 | Loss: 3.0556 | LR: 4.00e-04 | 1385.9ms/step | 14.4 samples/s
+Step 13550/44000 | Loss: 3.0772 | LR: 4.00e-04 | 1387.1ms/step | 14.4 samples/s
+Step 13560/44000 | Loss: 3.0487 | LR: 3.99e-04 | 1384.6ms/step | 14.4 samples/s
+Step 13570/44000 | Loss: 3.0246 | LR: 3.99e-04 | 1385.4ms/step | 14.4 samples/s
+Step 13580/44000 | Loss: 3.0007 | LR: 3.99e-04 | 1387.3ms/step | 14.4 samples/s
+Step 13590/44000 | Loss: 3.0154 | LR: 3.99e-04 | 1387.3ms/step | 14.4 samples/s
+Step 13600/44000 | Loss: 3.0443 | LR: 3.99e-04 | 1387.9ms/step | 14.4 samples/s
+Step 13610/44000 | Loss: 3.0471 | LR: 3.99e-04 | 1388.1ms/step | 14.4 samples/s
+Step 13620/44000 | Loss: 3.0051 | LR: 3.98e-04 | 1389.2ms/step | 14.4 samples/s
+Step 13630/44000 | Loss: 3.0344 | LR: 3.98e-04 | 1389.6ms/step | 14.4 samples/s
+Step 13640/44000 | Loss: 3.0839 | LR: 3.98e-04 | 1386.4ms/step | 14.4 samples/s
+Step 13650/44000 | Loss: 3.0618 | LR: 3.98e-04 | 1385.2ms/step | 14.4 samples/s
+Step 13660/44000 | Loss: 3.0412 | LR: 3.98e-04 | 1385.2ms/step | 14.4 samples/s
+Step 13670/44000 | Loss: 3.0370 | LR: 3.98e-04 | 1386.4ms/step | 14.4 samples/s
+Step 13680/44000 | Loss: 3.0431 | LR: 3.98e-04 | 1387.5ms/step | 14.4 samples/s
+Step 13690/44000 | Loss: 3.0439 | LR: 3.97e-04 | 1387.2ms/step | 14.4 samples/s
+Step 13700/44000 | Loss: 3.0576 | LR: 3.97e-04 | 1386.3ms/step | 14.4 samples/s
+Step 13710/44000 | Loss: 3.0240 | LR: 3.97e-04 | 1388.1ms/step | 14.4 samples/s
+Step 13720/44000 | Loss: 3.0602 | LR: 3.97e-04 | 1389.9ms/step | 14.4 samples/s
+Step 13730/44000 | Loss: 3.0537 | LR: 3.97e-04 | 1387.3ms/step | 14.4 samples/s
+Step 13740/44000 | Loss: 3.0484 | LR: 3.97e-04 | 1387.1ms/step | 14.4 samples/s
+Step 13750/44000 | Loss: 3.0405 | LR: 3.97e-04 | 1388.1ms/step | 14.4 samples/s
+Step 13760/44000 | Loss: 3.0393 | LR: 3.96e-04 | 1389.5ms/step | 14.4 samples/s
+Step 13770/44000 | Loss: 3.0233 | LR: 3.96e-04 | 1386.4ms/step | 14.4 samples/s
+Step 13780/44000 | Loss: 3.0353 | LR: 3.96e-04 | 1390.2ms/step | 14.4 samples/s
+Step 13790/44000 | Loss: 3.0609 | LR: 3.96e-04 | 1388.8ms/step | 14.4 samples/s
+Step 13800/44000 | Loss: 3.0408 | LR: 3.96e-04 | 1384.2ms/step | 14.4 samples/s
+Step 13810/44000 | Loss: 3.0345 | LR: 3.96e-04 | 1385.0ms/step | 14.4 samples/s
+Step 13820/44000 | Loss: 3.0194 | LR: 3.96e-04 | 1387.9ms/step | 14.4 samples/s
+Step 13830/44000 | Loss: 3.0231 | LR: 3.95e-04 | 1386.2ms/step | 14.4 samples/s
+Step 13840/44000 | Loss: 3.0142 | LR: 3.95e-04 | 1387.7ms/step | 14.4 samples/s
+Step 13850/44000 | Loss: 3.0426 | LR: 3.95e-04 | 1388.0ms/step | 14.4 samples/s
+Step 13860/44000 | Loss: 3.0148 | LR: 3.95e-04 | 1385.9ms/step | 14.4 samples/s
+Step 13870/44000 | Loss: 3.0317 | LR: 3.95e-04 | 1389.2ms/step | 14.4 samples/s
+Step 13880/44000 | Loss: 3.0121 | LR: 3.95e-04 | 1384.5ms/step | 14.4 samples/s
+Step 13890/44000 | Loss: 3.0297 | LR: 3.95e-04 | 1385.9ms/step | 14.4 samples/s
+Step 13900/44000 | Loss: 3.0451 | LR: 3.94e-04 | 1387.3ms/step | 14.4 samples/s
+Step 13910/44000 | Loss: 2.9991 | LR: 3.94e-04 | 1388.2ms/step | 14.4 samples/s
+Step 13920/44000 | Loss: 3.0000 | LR: 3.94e-04 | 1387.5ms/step | 14.4 samples/s
+Step 13930/44000 | Loss: 3.0256 | LR: 3.94e-04 | 1387.4ms/step | 14.4 samples/s
+Step 13940/44000 | Loss: 3.0589 | LR: 3.94e-04 | 1390.6ms/step | 14.4 samples/s
+Step 13950/44000 | Loss: 3.0292 | LR: 3.94e-04 | 1387.7ms/step | 14.4 samples/s
+Step 13960/44000 | Loss: 3.0183 | LR: 3.93e-04 | 1388.6ms/step | 14.4 samples/s
+Step 13970/44000 | Loss: 3.0254 | LR: 3.93e-04 | 1386.7ms/step | 14.4 samples/s
+Step 13980/44000 | Loss: 3.0400 | LR: 3.93e-04 | 1388.6ms/step | 14.4 samples/s
+Step 13990/44000 | Loss: 3.0382 | LR: 3.93e-04 | 1389.5ms/step | 14.4 samples/s
+Step 14000/44000 | Loss: 3.0055 | LR: 3.93e-04 | 1386.5ms/step | 14.4 samples/s
+Validation at step 14000: Loss = 3.0514, Perplexity = 21.14
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-14000
+Step 14010/44000 | Loss: 3.0195 | LR: 3.93e-04 | 3910.8ms/step | 5.1 samples/s
+Step 14020/44000 | Loss: 3.0264 | LR: 3.93e-04 | 1377.6ms/step | 14.5 samples/s
+Step 14030/44000 | Loss: 3.0339 | LR: 3.92e-04 | 1379.4ms/step | 14.5 samples/s
+Step 14040/44000 | Loss: 3.0399 | LR: 3.92e-04 | 1384.5ms/step | 14.4 samples/s
+Step 14050/44000 | Loss: 3.0264 | LR: 3.92e-04 | 1383.6ms/step | 14.5 samples/s
+Step 14060/44000 | Loss: 3.0510 | LR: 3.92e-04 | 1388.0ms/step | 14.4 samples/s
+Step 14070/44000 | Loss: 3.0416 | LR: 3.92e-04 | 1388.5ms/step | 14.4 samples/s
+Step 14080/44000 | Loss: 2.9989 | LR: 3.92e-04 | 1392.9ms/step | 14.4 samples/s
+Step 14090/44000 | Loss: 3.0299 | LR: 3.92e-04 | 1390.4ms/step | 14.4 samples/s
+Step 14100/44000 | Loss: 3.0058 | LR: 3.91e-04 | 1394.3ms/step | 14.3 samples/s
+Step 14110/44000 | Loss: 3.0750 | LR: 3.91e-04 | 1393.8ms/step | 14.3 samples/s
+Step 14120/44000 | Loss: 3.0350 | LR: 3.91e-04 | 1395.7ms/step | 14.3 samples/s
+Step 14130/44000 | Loss: 3.0346 | LR: 3.91e-04 | 1391.6ms/step | 14.4 samples/s
+Step 14140/44000 | Loss: 3.0181 | LR: 3.91e-04 | 1394.6ms/step | 14.3 samples/s
+Step 14150/44000 | Loss: 3.0178 | LR: 3.91e-04 | 1392.2ms/step | 14.4 samples/s
+Step 14160/44000 | Loss: 3.0493 | LR: 3.90e-04 | 1393.9ms/step | 14.3 samples/s
+Step 14170/44000 | Loss: 3.0609 | LR: 3.90e-04 | 1398.1ms/step | 14.3 samples/s
+Step 14180/44000 | Loss: 3.0742 | LR: 3.90e-04 | 1399.6ms/step | 14.3 samples/s
+Step 14190/44000 | Loss: 3.0194 | LR: 3.90e-04 | 1394.1ms/step | 14.3 samples/s
+Step 14200/44000 | Loss: 3.0512 | LR: 3.90e-04 | 1400.2ms/step | 14.3 samples/s
+Step 14210/44000 | Loss: 3.0552 | LR: 3.90e-04 | 1399.9ms/step | 14.3 samples/s
+Step 14220/44000 | Loss: 3.0104 | LR: 3.90e-04 | 1400.0ms/step | 14.3 samples/s
+Step 14230/44000 | Loss: 3.0535 | LR: 3.89e-04 | 1400.0ms/step | 14.3 samples/s
+Step 14240/44000 | Loss: 3.0235 | LR: 3.89e-04 | 1398.5ms/step | 14.3 samples/s
+Step 14250/44000 | Loss: 3.0587 | LR: 3.89e-04 | 1397.3ms/step | 14.3 samples/s
+Step 14260/44000 | Loss: 3.0575 | LR: 3.89e-04 | 1396.4ms/step | 14.3 samples/s
+Step 14270/44000 | Loss: 2.9848 | LR: 3.89e-04 | 1397.7ms/step | 14.3 samples/s
+Step 14280/44000 | Loss: 3.0336 | LR: 3.89e-04 | 1396.2ms/step | 14.3 samples/s
+Step 14290/44000 | Loss: 3.0208 | LR: 3.89e-04 | 1398.0ms/step | 14.3 samples/s
+Step 14300/44000 | Loss: 3.0309 | LR: 3.88e-04 | 1399.8ms/step | 14.3 samples/s
+Step 14310/44000 | Loss: 3.0246 | LR: 3.88e-04 | 1397.8ms/step | 14.3 samples/s
+Step 14320/44000 | Loss: 3.0096 | LR: 3.88e-04 | 1400.3ms/step | 14.3 samples/s
+Step 14330/44000 | Loss: 3.0129 | LR: 3.88e-04 | 1400.0ms/step | 14.3 samples/s
+Step 14340/44000 | Loss: 3.0275 | LR: 3.88e-04 | 1399.5ms/step | 14.3 samples/s
+Step 14350/44000 | Loss: 3.0316 | LR: 3.88e-04 | 1396.6ms/step | 14.3 samples/s
+Step 14360/44000 | Loss: 3.0323 | LR: 3.87e-04 | 1399.7ms/step | 14.3 samples/s
+Step 14370/44000 | Loss: 3.0338 | LR: 3.87e-04 | 1394.3ms/step | 14.3 samples/s
+Step 14380/44000 | Loss: 3.0403 | LR: 3.87e-04 | 1393.4ms/step | 14.4 samples/s
+Step 14390/44000 | Loss: 3.0455 | LR: 3.87e-04 | 1398.5ms/step | 14.3 samples/s
+Step 14400/44000 | Loss: 3.0131 | LR: 3.87e-04 | 1399.9ms/step | 14.3 samples/s
+Step 14410/44000 | Loss: 3.0276 | LR: 3.87e-04 | 1396.1ms/step | 14.3 samples/s
+Step 14420/44000 | Loss: 3.0622 | LR: 3.87e-04 | 1398.2ms/step | 14.3 samples/s
+Step 14430/44000 | Loss: 3.0231 | LR: 3.86e-04 | 1397.6ms/step | 14.3 samples/s
+Step 14440/44000 | Loss: 3.0176 | LR: 3.86e-04 | 1396.1ms/step | 14.3 samples/s
+Step 14450/44000 | Loss: 3.0216 | LR: 3.86e-04 | 1396.5ms/step | 14.3 samples/s
+Step 14460/44000 | Loss: 2.9962 | LR: 3.86e-04 | 1399.7ms/step | 14.3 samples/s
+Step 14470/44000 | Loss: 2.9985 | LR: 3.86e-04 | 1397.9ms/step | 14.3 samples/s
+Step 14480/44000 | Loss: 3.0100 | LR: 3.86e-04 | 1400.2ms/step | 14.3 samples/s
+Step 14490/44000 | Loss: 3.0250 | LR: 3.85e-04 | 1399.3ms/step | 14.3 samples/s
+Step 14500/44000 | Loss: 3.0494 | LR: 3.85e-04 | 1398.6ms/step | 14.3 samples/s
+Step 14510/44000 | Loss: 3.0211 | LR: 3.85e-04 | 1400.1ms/step | 14.3 samples/s
+Step 14520/44000 | Loss: 3.0304 | LR: 3.85e-04 | 1399.8ms/step | 14.3 samples/s
+Step 14530/44000 | Loss: 3.0245 | LR: 3.85e-04 | 1398.1ms/step | 14.3 samples/s
+Step 14540/44000 | Loss: 3.0155 | LR: 3.85e-04 | 1397.9ms/step | 14.3 samples/s
+Step 14550/44000 | Loss: 3.0204 | LR: 3.85e-04 | 1400.1ms/step | 14.3 samples/s
+Step 14560/44000 | Loss: 3.0063 | LR: 3.84e-04 | 1397.6ms/step | 14.3 samples/s
+Step 14570/44000 | Loss: 3.0307 | LR: 3.84e-04 | 1398.2ms/step | 14.3 samples/s
+Step 14580/44000 | Loss: 3.0312 | LR: 3.84e-04 | 1397.9ms/step | 14.3 samples/s
+Step 14590/44000 | Loss: 3.0091 | LR: 3.84e-04 | 1398.2ms/step | 14.3 samples/s
+Step 14600/44000 | Loss: 3.0276 | LR: 3.84e-04 | 1398.7ms/step | 14.3 samples/s
+Step 14610/44000 | Loss: 3.0123 | LR: 3.84e-04 | 1399.6ms/step | 14.3 samples/s
+Step 14620/44000 | Loss: 3.0115 | LR: 3.83e-04 | 1397.8ms/step | 14.3 samples/s
+Step 14630/44000 | Loss: 3.0249 | LR: 3.83e-04 | 1399.8ms/step | 14.3 samples/s
+Step 14640/44000 | Loss: 3.0200 | LR: 3.83e-04 | 1399.6ms/step | 14.3 samples/s
+Step 14650/44000 | Loss: 3.0181 | LR: 3.83e-04 | 1395.6ms/step | 14.3 samples/s
+Step 14660/44000 | Loss: 2.9862 | LR: 3.83e-04 | 1398.8ms/step | 14.3 samples/s
+Step 14670/44000 | Loss: 3.0156 | LR: 3.83e-04 | 1398.1ms/step | 14.3 samples/s
+Step 14680/44000 | Loss: 3.0400 | LR: 3.83e-04 | 1394.5ms/step | 14.3 samples/s
+Step 14690/44000 | Loss: 3.0197 | LR: 3.82e-04 | 1397.5ms/step | 14.3 samples/s
+Step 14700/44000 | Loss: 3.0294 | LR: 3.82e-04 | 1398.0ms/step | 14.3 samples/s
+Step 14710/44000 | Loss: 3.0228 | LR: 3.82e-04 | 1400.3ms/step | 14.3 samples/s
+Step 14720/44000 | Loss: 3.0467 | LR: 3.82e-04 | 1395.8ms/step | 14.3 samples/s
+Step 14730/44000 | Loss: 2.9995 | LR: 3.82e-04 | 1400.1ms/step | 14.3 samples/s
+Step 14740/44000 | Loss: 3.0405 | LR: 3.82e-04 | 1399.7ms/step | 14.3 samples/s
+Step 14750/44000 | Loss: 3.0521 | LR: 3.81e-04 | 1398.2ms/step | 14.3 samples/s
+Step 14760/44000 | Loss: 3.0057 | LR: 3.81e-04 | 1397.8ms/step | 14.3 samples/s
+Step 14770/44000 | Loss: 3.0283 | LR: 3.81e-04 | 1398.2ms/step | 14.3 samples/s
+Step 14780/44000 | Loss: 2.9995 | LR: 3.81e-04 | 1398.1ms/step | 14.3 samples/s
+Step 14790/44000 | Loss: 3.0225 | LR: 3.81e-04 | 1395.9ms/step | 14.3 samples/s
+Step 14800/44000 | Loss: 2.9918 | LR: 3.81e-04 | 1398.2ms/step | 14.3 samples/s
+Step 14810/44000 | Loss: 3.0154 | LR: 3.81e-04 | 1400.2ms/step | 14.3 samples/s
+Step 14820/44000 | Loss: 3.0140 | LR: 3.80e-04 | 1398.7ms/step | 14.3 samples/s
+Step 14830/44000 | Loss: 3.0100 | LR: 3.80e-04 | 1398.9ms/step | 14.3 samples/s
+Step 14840/44000 | Loss: 2.9869 | LR: 3.80e-04 | 1398.1ms/step | 14.3 samples/s
+Step 14850/44000 | Loss: 3.0370 | LR: 3.80e-04 | 1399.9ms/step | 14.3 samples/s
+Step 14860/44000 | Loss: 3.0687 | LR: 3.80e-04 | 1400.1ms/step | 14.3 samples/s
+Step 14870/44000 | Loss: 3.0336 | LR: 3.80e-04 | 1395.7ms/step | 14.3 samples/s
+Step 14880/44000 | Loss: 3.0418 | LR: 3.79e-04 | 1396.1ms/step | 14.3 samples/s
+Step 14890/44000 | Loss: 2.9973 | LR: 3.79e-04 | 1398.5ms/step | 14.3 samples/s
+Step 14900/44000 | Loss: 3.0477 | LR: 3.79e-04 | 1397.7ms/step | 14.3 samples/s
+Step 14910/44000 | Loss: 3.0226 | LR: 3.79e-04 | 1400.0ms/step | 14.3 samples/s
+Step 14920/44000 | Loss: 3.0062 | LR: 3.79e-04 | 1397.9ms/step | 14.3 samples/s
+Step 14930/44000 | Loss: 3.0351 | LR: 3.79e-04 | 1395.7ms/step | 14.3 samples/s
+Step 14940/44000 | Loss: 2.9710 | LR: 3.79e-04 | 1400.3ms/step | 14.3 samples/s
+Step 14950/44000 | Loss: 3.0007 | LR: 3.78e-04 | 1398.0ms/step | 14.3 samples/s
+Step 14960/44000 | Loss: 3.0032 | LR: 3.78e-04 | 1398.0ms/step | 14.3 samples/s
+Step 14970/44000 | Loss: 3.0145 | LR: 3.78e-04 | 1395.5ms/step | 14.3 samples/s
+Step 14980/44000 | Loss: 2.9904 | LR: 3.78e-04 | 1398.3ms/step | 14.3 samples/s
+Step 14990/44000 | Loss: 3.0434 | LR: 3.78e-04 | 1400.0ms/step | 14.3 samples/s
+Step 15000/44000 | Loss: 3.0264 | LR: 3.78e-04 | 1400.3ms/step | 14.3 samples/s
+Validation at step 15000: Loss = 3.0379, Perplexity = 20.86
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-15000
+Step 15010/44000 | Loss: 3.0395 | LR: 3.77e-04 | 3905.4ms/step | 5.1 samples/s
+Step 15020/44000 | Loss: 2.9784 | LR: 3.77e-04 | 1383.4ms/step | 14.5 samples/s
+Step 15030/44000 | Loss: 3.0472 | LR: 3.77e-04 | 1383.1ms/step | 14.5 samples/s
+Step 15040/44000 | Loss: 3.0234 | LR: 3.77e-04 | 1381.6ms/step | 14.5 samples/s
+Step 15050/44000 | Loss: 3.0335 | LR: 3.77e-04 | 1384.9ms/step | 14.4 samples/s
+Step 15060/44000 | Loss: 3.0408 | LR: 3.77e-04 | 1388.4ms/step | 14.4 samples/s
+Step 15070/44000 | Loss: 3.0389 | LR: 3.77e-04 | 1391.9ms/step | 14.4 samples/s
+Step 15080/44000 | Loss: 3.0075 | LR: 3.76e-04 | 1397.3ms/step | 14.3 samples/s
+Step 15090/44000 | Loss: 3.0260 | LR: 3.76e-04 | 1396.1ms/step | 14.3 samples/s
+Step 15100/44000 | Loss: 3.0122 | LR: 3.76e-04 | 1399.7ms/step | 14.3 samples/s
+Step 15110/44000 | Loss: 3.0386 | LR: 3.76e-04 | 1398.0ms/step | 14.3 samples/s
+Step 15120/44000 | Loss: 3.0303 | LR: 3.76e-04 | 1396.0ms/step | 14.3 samples/s
+Step 15130/44000 | Loss: 3.0005 | LR: 3.76e-04 | 1396.0ms/step | 14.3 samples/s
+Step 15140/44000 | Loss: 3.0191 | LR: 3.75e-04 | 1396.2ms/step | 14.3 samples/s
+Step 15150/44000 | Loss: 3.0069 | LR: 3.75e-04 | 1396.5ms/step | 14.3 samples/s
+Step 15160/44000 | Loss: 3.0318 | LR: 3.75e-04 | 1393.4ms/step | 14.4 samples/s
+Step 15170/44000 | Loss: 3.0635 | LR: 3.75e-04 | 1394.1ms/step | 14.3 samples/s
+Step 15180/44000 | Loss: 2.9948 | LR: 3.75e-04 | 1394.7ms/step | 14.3 samples/s
+Step 15190/44000 | Loss: 3.0487 | LR: 3.75e-04 | 1396.7ms/step | 14.3 samples/s
+Step 15200/44000 | Loss: 3.0296 | LR: 3.74e-04 | 1394.4ms/step | 14.3 samples/s
+Step 15210/44000 | Loss: 3.0246 | LR: 3.74e-04 | 1395.4ms/step | 14.3 samples/s
+Step 15220/44000 | Loss: 2.9780 | LR: 3.74e-04 | 1396.0ms/step | 14.3 samples/s
+Step 15230/44000 | Loss: 3.0285 | LR: 3.74e-04 | 1396.4ms/step | 14.3 samples/s
+Step 15240/44000 | Loss: 3.0217 | LR: 3.74e-04 | 1398.1ms/step | 14.3 samples/s
+Step 15250/44000 | Loss: 3.0170 | LR: 3.74e-04 | 1400.0ms/step | 14.3 samples/s
+Step 15260/44000 | Loss: 3.0567 | LR: 3.74e-04 | 1398.2ms/step | 14.3 samples/s
+Step 15270/44000 | Loss: 3.0121 | LR: 3.73e-04 | 1400.0ms/step | 14.3 samples/s
+Step 15280/44000 | Loss: 2.9865 | LR: 3.73e-04 | 1400.2ms/step | 14.3 samples/s
+Step 15290/44000 | Loss: 3.0229 | LR: 3.73e-04 | 1399.9ms/step | 14.3 samples/s
+Step 15300/44000 | Loss: 2.9984 | LR: 3.73e-04 | 1399.8ms/step | 14.3 samples/s
+Step 15310/44000 | Loss: 3.0261 | LR: 3.73e-04 | 1397.9ms/step | 14.3 samples/s
+Step 15320/44000 | Loss: 2.9886 | LR: 3.73e-04 | 1396.9ms/step | 14.3 samples/s
+Step 15330/44000 | Loss: 2.9936 | LR: 3.72e-04 | 1396.5ms/step | 14.3 samples/s
+Step 15340/44000 | Loss: 3.0132 | LR: 3.72e-04 | 1394.6ms/step | 14.3 samples/s
+Step 15350/44000 | Loss: 3.0146 | LR: 3.72e-04 | 1394.9ms/step | 14.3 samples/s
+Step 15360/44000 | Loss: 3.0296 | LR: 3.72e-04 | 1393.0ms/step | 14.4 samples/s
+Step 15370/44000 | Loss: 3.0021 | LR: 3.72e-04 | 1395.6ms/step | 14.3 samples/s
+Step 15380/44000 | Loss: 2.9956 | LR: 3.72e-04 | 1392.9ms/step | 14.4 samples/s
+Step 15390/44000 | Loss: 2.9975 | LR: 3.71e-04 | 1395.1ms/step | 14.3 samples/s
+Step 15400/44000 | Loss: 3.0158 | LR: 3.71e-04 | 1398.6ms/step | 14.3 samples/s
+Step 15410/44000 | Loss: 3.0168 | LR: 3.71e-04 | 1395.9ms/step | 14.3 samples/s
+Step 15420/44000 | Loss: 3.0116 | LR: 3.71e-04 | 1398.4ms/step | 14.3 samples/s
+Step 15430/44000 | Loss: 3.0107 | LR: 3.71e-04 | 1398.0ms/step | 14.3 samples/s
+Step 15440/44000 | Loss: 2.9948 | LR: 3.71e-04 | 1396.0ms/step | 14.3 samples/s
+Step 15450/44000 | Loss: 3.0260 | LR: 3.71e-04 | 1397.8ms/step | 14.3 samples/s
+Step 15460/44000 | Loss: 3.0192 | LR: 3.70e-04 | 1396.0ms/step | 14.3 samples/s
+Step 15470/44000 | Loss: 2.9916 | LR: 3.70e-04 | 1400.0ms/step | 14.3 samples/s
+Step 15480/44000 | Loss: 3.0190 | LR: 3.70e-04 | 1399.9ms/step | 14.3 samples/s
+Step 15490/44000 | Loss: 3.0406 | LR: 3.70e-04 | 1398.2ms/step | 14.3 samples/s
+Step 15500/44000 | Loss: 3.0192 | LR: 3.70e-04 | 1399.9ms/step | 14.3 samples/s
+Step 15510/44000 | Loss: 2.9820 | LR: 3.70e-04 | 1400.0ms/step | 14.3 samples/s
+Step 15520/44000 | Loss: 3.0317 | LR: 3.69e-04 | 1400.0ms/step | 14.3 samples/s
+Step 15530/44000 | Loss: 2.9941 | LR: 3.69e-04 | 1397.6ms/step | 14.3 samples/s
+Step 15540/44000 | Loss: 3.0270 | LR: 3.69e-04 | 1398.6ms/step | 14.3 samples/s
+Step 15550/44000 | Loss: 3.0610 | LR: 3.69e-04 | 1398.1ms/step | 14.3 samples/s
+Step 15560/44000 | Loss: 3.0169 | LR: 3.69e-04 | 1399.8ms/step | 14.3 samples/s
+Step 15570/44000 | Loss: 3.0076 | LR: 3.69e-04 | 1400.2ms/step | 14.3 samples/s
+Step 15580/44000 | Loss: 2.9940 | LR: 3.68e-04 | 1393.8ms/step | 14.3 samples/s
+Step 15590/44000 | Loss: 3.0269 | LR: 3.68e-04 | 1398.6ms/step | 14.3 samples/s
+Step 15600/44000 | Loss: 3.0211 | LR: 3.68e-04 | 1397.5ms/step | 14.3 samples/s
+Step 15610/44000 | Loss: 3.0502 | LR: 3.68e-04 | 1399.8ms/step | 14.3 samples/s
+Step 15620/44000 | Loss: 3.0169 | LR: 3.68e-04 | 1393.8ms/step | 14.3 samples/s
+Step 15630/44000 | Loss: 3.0036 | LR: 3.68e-04 | 1394.1ms/step | 14.3 samples/s
+Step 15640/44000 | Loss: 3.0207 | LR: 3.67e-04 | 1394.0ms/step | 14.3 samples/s
+Step 15650/44000 | Loss: 3.0254 | LR: 3.67e-04 | 1396.2ms/step | 14.3 samples/s
+Step 15660/44000 | Loss: 3.0146 | LR: 3.67e-04 | 1400.1ms/step | 14.3 samples/s
+Step 15670/44000 | Loss: 3.0043 | LR: 3.67e-04 | 1397.8ms/step | 14.3 samples/s
+Step 15680/44000 | Loss: 2.9986 | LR: 3.67e-04 | 1400.4ms/step | 14.3 samples/s
+Step 15690/44000 | Loss: 3.0438 | LR: 3.67e-04 | 1397.4ms/step | 14.3 samples/s
+Step 15700/44000 | Loss: 2.9976 | LR: 3.67e-04 | 1398.1ms/step | 14.3 samples/s
+Step 15710/44000 | Loss: 2.9913 | LR: 3.66e-04 | 1396.1ms/step | 14.3 samples/s
+Step 15720/44000 | Loss: 3.0242 | LR: 3.66e-04 | 1396.5ms/step | 14.3 samples/s
+Step 15730/44000 | Loss: 3.0170 | LR: 3.66e-04 | 1395.9ms/step | 14.3 samples/s
+Step 15740/44000 | Loss: 3.0116 | LR: 3.66e-04 | 1397.7ms/step | 14.3 samples/s
+Step 15750/44000 | Loss: 3.0043 | LR: 3.66e-04 | 1396.1ms/step | 14.3 samples/s
+Step 15760/44000 | Loss: 2.9945 | LR: 3.66e-04 | 1399.8ms/step | 14.3 samples/s
+Step 15770/44000 | Loss: 3.0071 | LR: 3.65e-04 | 1398.2ms/step | 14.3 samples/s
+Step 15780/44000 | Loss: 3.0274 | LR: 3.65e-04 | 1400.2ms/step | 14.3 samples/s
+Step 15790/44000 | Loss: 3.0277 | LR: 3.65e-04 | 1397.5ms/step | 14.3 samples/s
+Step 15800/44000 | Loss: 2.9978 | LR: 3.65e-04 | 1400.1ms/step | 14.3 samples/s
+Step 15810/44000 | Loss: 3.0099 | LR: 3.65e-04 | 1397.7ms/step | 14.3 samples/s
+Step 15820/44000 | Loss: 3.0154 | LR: 3.65e-04 | 1397.9ms/step | 14.3 samples/s
+Step 15830/44000 | Loss: 3.0436 | LR: 3.64e-04 | 1394.5ms/step | 14.3 samples/s
+Step 15840/44000 | Loss: 2.9938 | LR: 3.64e-04 | 1397.8ms/step | 14.3 samples/s
+Step 15850/44000 | Loss: 2.9820 | LR: 3.64e-04 | 1397.6ms/step | 14.3 samples/s
+Step 15860/44000 | Loss: 3.0010 | LR: 3.64e-04 | 1396.4ms/step | 14.3 samples/s
+Step 15870/44000 | Loss: 3.0045 | LR: 3.64e-04 | 1394.4ms/step | 14.3 samples/s
+Step 15880/44000 | Loss: 3.0196 | LR: 3.64e-04 | 1399.9ms/step | 14.3 samples/s
+Step 15890/44000 | Loss: 3.0452 | LR: 3.63e-04 | 1397.9ms/step | 14.3 samples/s
+Step 15900/44000 | Loss: 2.9920 | LR: 3.63e-04 | 1397.0ms/step | 14.3 samples/s
+Step 15910/44000 | Loss: 3.0221 | LR: 3.63e-04 | 1393.3ms/step | 14.4 samples/s
+Step 15920/44000 | Loss: 3.0163 | LR: 3.63e-04 | 1399.8ms/step | 14.3 samples/s
+Step 15930/44000 | Loss: 3.0266 | LR: 3.63e-04 | 1400.0ms/step | 14.3 samples/s
+Step 15940/44000 | Loss: 3.0067 | LR: 3.63e-04 | 1393.8ms/step | 14.3 samples/s
+Step 15950/44000 | Loss: 3.0263 | LR: 3.62e-04 | 1396.5ms/step | 14.3 samples/s
+Step 15960/44000 | Loss: 3.0358 | LR: 3.62e-04 | 1395.6ms/step | 14.3 samples/s
+Step 15970/44000 | Loss: 3.0408 | LR: 3.62e-04 | 1398.1ms/step | 14.3 samples/s
+Step 15980/44000 | Loss: 2.9845 | LR: 3.62e-04 | 1398.0ms/step | 14.3 samples/s
+Step 15990/44000 | Loss: 3.0149 | LR: 3.62e-04 | 1398.1ms/step | 14.3 samples/s
+Step 16000/44000 | Loss: 3.0248 | LR: 3.62e-04 | 1399.4ms/step | 14.3 samples/s
+Validation at step 16000: Loss = 3.0281, Perplexity = 20.66
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-16000
+Step 16010/44000 | Loss: 2.9864 | LR: 3.62e-04 | 3888.2ms/step | 5.1 samples/s
+Step 16020/44000 | Loss: 2.9857 | LR: 3.61e-04 | 1379.9ms/step | 14.5 samples/s
+Step 16030/44000 | Loss: 3.0181 | LR: 3.61e-04 | 1384.4ms/step | 14.4 samples/s
+Step 16040/44000 | Loss: 3.0149 | LR: 3.61e-04 | 1386.6ms/step | 14.4 samples/s
+Step 16050/44000 | Loss: 3.0004 | LR: 3.61e-04 | 1382.8ms/step | 14.5 samples/s
+Step 16060/44000 | Loss: 3.0185 | LR: 3.61e-04 | 1386.5ms/step | 14.4 samples/s
+Step 16070/44000 | Loss: 3.0414 | LR: 3.61e-04 | 1387.2ms/step | 14.4 samples/s
+Step 16080/44000 | Loss: 3.0259 | LR: 3.60e-04 | 1392.0ms/step | 14.4 samples/s
+Step 16090/44000 | Loss: 2.9867 | LR: 3.60e-04 | 1393.1ms/step | 14.4 samples/s
+Step 16100/44000 | Loss: 3.0156 | LR: 3.60e-04 | 1391.6ms/step | 14.4 samples/s
+Step 16110/44000 | Loss: 2.9859 | LR: 3.60e-04 | 1393.9ms/step | 14.3 samples/s
+Step 16120/44000 | Loss: 2.9940 | LR: 3.60e-04 | 1394.4ms/step | 14.3 samples/s
+Step 16130/44000 | Loss: 3.0248 | LR: 3.60e-04 | 1393.7ms/step | 14.4 samples/s
+Step 16140/44000 | Loss: 2.9950 | LR: 3.59e-04 | 1394.0ms/step | 14.3 samples/s
+Step 16150/44000 | Loss: 2.9980 | LR: 3.59e-04 | 1392.2ms/step | 14.4 samples/s
+Step 16160/44000 | Loss: 3.0314 | LR: 3.59e-04 | 1395.5ms/step | 14.3 samples/s
+Step 16170/44000 | Loss: 3.0048 | LR: 3.59e-04 | 1395.9ms/step | 14.3 samples/s
+Step 16180/44000 | Loss: 2.9694 | LR: 3.59e-04 | 1391.4ms/step | 14.4 samples/s
+Step 16190/44000 | Loss: 3.0218 | LR: 3.59e-04 | 1395.1ms/step | 14.3 samples/s
+Step 16200/44000 | Loss: 3.0063 | LR: 3.58e-04 | 1400.2ms/step | 14.3 samples/s
+Step 16210/44000 | Loss: 3.0114 | LR: 3.58e-04 | 1395.8ms/step | 14.3 samples/s
+Step 16220/44000 | Loss: 2.9844 | LR: 3.58e-04 | 1394.1ms/step | 14.3 samples/s
+Step 16230/44000 | Loss: 3.0263 | LR: 3.58e-04 | 1395.9ms/step | 14.3 samples/s
+Step 16240/44000 | Loss: 3.0202 | LR: 3.58e-04 | 1396.3ms/step | 14.3 samples/s
+Step 16250/44000 | Loss: 2.9751 | LR: 3.58e-04 | 1397.7ms/step | 14.3 samples/s
+Step 16260/44000 | Loss: 3.0133 | LR: 3.57e-04 | 1394.7ms/step | 14.3 samples/s
+Step 16270/44000 | Loss: 3.0227 | LR: 3.57e-04 | 1395.3ms/step | 14.3 samples/s
+Step 16280/44000 | Loss: 3.0159 | LR: 3.57e-04 | 1398.3ms/step | 14.3 samples/s
+Step 16290/44000 | Loss: 3.0043 | LR: 3.57e-04 | 1395.9ms/step | 14.3 samples/s
+Step 16300/44000 | Loss: 3.0060 | LR: 3.57e-04 | 1395.7ms/step | 14.3 samples/s
+Step 16310/44000 | Loss: 2.9945 | LR: 3.57e-04 | 1398.3ms/step | 14.3 samples/s
+Step 16320/44000 | Loss: 2.9799 | LR: 3.56e-04 | 1396.1ms/step | 14.3 samples/s
+Step 16330/44000 | Loss: 3.0184 | LR: 3.56e-04 | 1397.7ms/step | 14.3 samples/s
+Step 16340/44000 | Loss: 3.0043 | LR: 3.56e-04 | 1396.3ms/step | 14.3 samples/s
+Step 16350/44000 | Loss: 3.0138 | LR: 3.56e-04 | 1393.5ms/step | 14.4 samples/s
+Step 16360/44000 | Loss: 2.9752 | LR: 3.56e-04 | 1394.3ms/step | 14.3 samples/s
+Step 16370/44000 | Loss: 2.9994 | LR: 3.56e-04 | 1398.1ms/step | 14.3 samples/s
+Step 16380/44000 | Loss: 2.9924 | LR: 3.55e-04 | 1397.9ms/step | 14.3 samples/s
+Step 16390/44000 | Loss: 2.9954 | LR: 3.55e-04 | 1397.2ms/step | 14.3 samples/s
+Step 16400/44000 | Loss: 2.9960 | LR: 3.55e-04 | 1398.7ms/step | 14.3 samples/s
+Step 16410/44000 | Loss: 2.9906 | LR: 3.55e-04 | 1396.2ms/step | 14.3 samples/s
+Step 16420/44000 | Loss: 3.0041 | LR: 3.55e-04 | 1399.7ms/step | 14.3 samples/s
+Step 16430/44000 | Loss: 3.0090 | LR: 3.55e-04 | 1400.4ms/step | 14.3 samples/s
+Step 16440/44000 | Loss: 3.0280 | LR: 3.54e-04 | 1395.8ms/step | 14.3 samples/s
+Step 16450/44000 | Loss: 3.0094 | LR: 3.54e-04 | 1395.9ms/step | 14.3 samples/s
+Step 16460/44000 | Loss: 2.9559 | LR: 3.54e-04 | 1396.1ms/step | 14.3 samples/s
+Step 16470/44000 | Loss: 2.9743 | LR: 3.54e-04 | 1396.1ms/step | 14.3 samples/s
+Step 16480/44000 | Loss: 3.0436 | LR: 3.54e-04 | 1396.0ms/step | 14.3 samples/s
+Step 16490/44000 | Loss: 2.9995 | LR: 3.54e-04 | 1398.2ms/step | 14.3 samples/s
+Step 16500/44000 | Loss: 3.0170 | LR: 3.53e-04 | 1393.9ms/step | 14.3 samples/s
+Step 16510/44000 | Loss: 3.0246 | LR: 3.53e-04 | 1395.8ms/step | 14.3 samples/s
+Step 16520/44000 | Loss: 3.0180 | LR: 3.53e-04 | 1392.1ms/step | 14.4 samples/s
+Step 16530/44000 | Loss: 2.9973 | LR: 3.53e-04 | 1400.1ms/step | 14.3 samples/s
+Step 16540/44000 | Loss: 3.0152 | LR: 3.53e-04 | 1400.2ms/step | 14.3 samples/s
+Step 16550/44000 | Loss: 2.9887 | LR: 3.53e-04 | 1397.9ms/step | 14.3 samples/s
+Step 16560/44000 | Loss: 2.9889 | LR: 3.52e-04 | 1399.9ms/step | 14.3 samples/s
+Step 16570/44000 | Loss: 3.0022 | LR: 3.52e-04 | 1395.7ms/step | 14.3 samples/s
+Step 16580/44000 | Loss: 2.9782 | LR: 3.52e-04 | 1395.7ms/step | 14.3 samples/s
+Step 16590/44000 | Loss: 2.9683 | LR: 3.52e-04 | 1396.8ms/step | 14.3 samples/s
+Step 16600/44000 | Loss: 3.0078 | LR: 3.52e-04 | 1399.9ms/step | 14.3 samples/s
+Step 16610/44000 | Loss: 2.9827 | LR: 3.52e-04 | 1398.1ms/step | 14.3 samples/s
+Step 16620/44000 | Loss: 2.9881 | LR: 3.51e-04 | 1396.6ms/step | 14.3 samples/s
+Step 16630/44000 | Loss: 2.9986 | LR: 3.51e-04 | 1391.3ms/step | 14.4 samples/s
+Step 16640/44000 | Loss: 3.0085 | LR: 3.51e-04 | 1399.7ms/step | 14.3 samples/s
+Step 16650/44000 | Loss: 3.0182 | LR: 3.51e-04 | 1399.6ms/step | 14.3 samples/s
+Step 16660/44000 | Loss: 3.0038 | LR: 3.51e-04 | 1392.8ms/step | 14.4 samples/s
+Step 16670/44000 | Loss: 3.0267 | LR: 3.51e-04 | 1397.5ms/step | 14.3 samples/s
+Step 16680/44000 | Loss: 3.0212 | LR: 3.50e-04 | 1400.3ms/step | 14.3 samples/s
+Step 16690/44000 | Loss: 3.0109 | LR: 3.50e-04 | 1398.2ms/step | 14.3 samples/s
+Step 16700/44000 | Loss: 2.9904 | LR: 3.50e-04 | 1397.8ms/step | 14.3 samples/s
+Step 16710/44000 | Loss: 2.9792 | LR: 3.50e-04 | 1398.0ms/step | 14.3 samples/s
+Step 16720/44000 | Loss: 2.9861 | LR: 3.50e-04 | 1394.3ms/step | 14.3 samples/s
+Step 16730/44000 | Loss: 3.0105 | LR: 3.50e-04 | 1396.1ms/step | 14.3 samples/s
+Step 16740/44000 | Loss: 3.0149 | LR: 3.49e-04 | 1399.5ms/step | 14.3 samples/s
+Step 16750/44000 | Loss: 3.0014 | LR: 3.49e-04 | 1396.1ms/step | 14.3 samples/s
+Step 16760/44000 | Loss: 3.0067 | LR: 3.49e-04 | 1394.1ms/step | 14.3 samples/s
+Step 16770/44000 | Loss: 2.9674 | LR: 3.49e-04 | 1397.6ms/step | 14.3 samples/s
+Step 16780/44000 | Loss: 3.0223 | LR: 3.49e-04 | 1398.1ms/step | 14.3 samples/s
+Step 16790/44000 | Loss: 2.9809 | LR: 3.49e-04 | 1398.1ms/step | 14.3 samples/s
+Step 16800/44000 | Loss: 3.0072 | LR: 3.48e-04 | 1395.4ms/step | 14.3 samples/s
+Step 16810/44000 | Loss: 2.9969 | LR: 3.48e-04 | 1396.0ms/step | 14.3 samples/s
+Step 16820/44000 | Loss: 2.9723 | LR: 3.48e-04 | 1390.7ms/step | 14.4 samples/s
+Step 16830/44000 | Loss: 3.0121 | LR: 3.48e-04 | 1395.6ms/step | 14.3 samples/s
+Step 16840/44000 | Loss: 2.9837 | LR: 3.48e-04 | 1395.5ms/step | 14.3 samples/s
+Step 16850/44000 | Loss: 3.0045 | LR: 3.48e-04 | 1395.0ms/step | 14.3 samples/s
+Step 16860/44000 | Loss: 3.0111 | LR: 3.47e-04 | 1399.9ms/step | 14.3 samples/s
+Step 16870/44000 | Loss: 2.9940 | LR: 3.47e-04 | 1396.0ms/step | 14.3 samples/s
+Step 16880/44000 | Loss: 2.9770 | LR: 3.47e-04 | 1399.6ms/step | 14.3 samples/s
+Step 16890/44000 | Loss: 2.9884 | LR: 3.47e-04 | 1396.5ms/step | 14.3 samples/s
+Step 16900/44000 | Loss: 2.9774 | LR: 3.47e-04 | 1397.8ms/step | 14.3 samples/s
+Step 16910/44000 | Loss: 3.0140 | LR: 3.47e-04 | 1398.3ms/step | 14.3 samples/s
+Step 16920/44000 | Loss: 3.0135 | LR: 3.46e-04 | 1397.7ms/step | 14.3 samples/s
+Step 16930/44000 | Loss: 3.0102 | LR: 3.46e-04 | 1400.4ms/step | 14.3 samples/s
+Step 16940/44000 | Loss: 3.0144 | LR: 3.46e-04 | 1399.9ms/step | 14.3 samples/s
+Step 16950/44000 | Loss: 2.9817 | LR: 3.46e-04 | 1397.9ms/step | 14.3 samples/s
+Step 16960/44000 | Loss: 3.0258 | LR: 3.46e-04 | 1395.6ms/step | 14.3 samples/s
+Step 16970/44000 | Loss: 2.9987 | LR: 3.46e-04 | 1398.4ms/step | 14.3 samples/s
+Step 16980/44000 | Loss: 2.9798 | LR: 3.45e-04 | 1399.7ms/step | 14.3 samples/s
+Step 16990/44000 | Loss: 3.0147 | LR: 3.45e-04 | 1398.4ms/step | 14.3 samples/s
+Step 17000/44000 | Loss: 3.0201 | LR: 3.45e-04 | 1391.1ms/step | 14.4 samples/s
+Validation at step 17000: Loss = 3.0121, Perplexity = 20.33
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-17000
+Step 17010/44000 | Loss: 2.9965 | LR: 3.45e-04 | 3968.6ms/step | 5.0 samples/s
+Step 17020/44000 | Loss: 2.9838 | LR: 3.45e-04 | 1384.7ms/step | 14.4 samples/s
+Step 17030/44000 | Loss: 2.9773 | LR: 3.45e-04 | 1383.2ms/step | 14.5 samples/s
+Step 17040/44000 | Loss: 2.9716 | LR: 3.44e-04 | 1386.5ms/step | 14.4 samples/s
+Step 17050/44000 | Loss: 3.0080 | LR: 3.44e-04 | 1384.9ms/step | 14.4 samples/s
+Step 17060/44000 | Loss: 3.0121 | LR: 3.44e-04 | 1387.2ms/step | 14.4 samples/s
+Step 17070/44000 | Loss: 2.9939 | LR: 3.44e-04 | 1389.7ms/step | 14.4 samples/s
+Step 17080/44000 | Loss: 3.0381 | LR: 3.44e-04 | 1391.7ms/step | 14.4 samples/s
+Step 17090/44000 | Loss: 2.9735 | LR: 3.44e-04 | 1390.1ms/step | 14.4 samples/s
+Step 17100/44000 | Loss: 2.9871 | LR: 3.43e-04 | 1390.1ms/step | 14.4 samples/s
+Step 17110/44000 | Loss: 3.0153 | LR: 3.43e-04 | 1393.7ms/step | 14.4 samples/s
+Step 17120/44000 | Loss: 3.0041 | LR: 3.43e-04 | 1389.3ms/step | 14.4 samples/s
+Step 17130/44000 | Loss: 2.9881 | LR: 3.43e-04 | 1391.1ms/step | 14.4 samples/s
+Step 17140/44000 | Loss: 3.0179 | LR: 3.43e-04 | 1389.9ms/step | 14.4 samples/s
+Step 17150/44000 | Loss: 2.9896 | LR: 3.43e-04 | 1390.6ms/step | 14.4 samples/s
+Step 17160/44000 | Loss: 3.0055 | LR: 3.42e-04 | 1391.1ms/step | 14.4 samples/s
+Step 17170/44000 | Loss: 3.0043 | LR: 3.42e-04 | 1389.0ms/step | 14.4 samples/s
+Step 17180/44000 | Loss: 3.0208 | LR: 3.42e-04 | 1388.5ms/step | 14.4 samples/s
+Step 17190/44000 | Loss: 2.9791 | LR: 3.42e-04 | 1390.5ms/step | 14.4 samples/s
+Step 17200/44000 | Loss: 2.9915 | LR: 3.42e-04 | 1391.8ms/step | 14.4 samples/s
+Step 17210/44000 | Loss: 2.9879 | LR: 3.42e-04 | 1389.7ms/step | 14.4 samples/s
+Step 17220/44000 | Loss: 2.9841 | LR: 3.41e-04 | 1389.6ms/step | 14.4 samples/s
+Step 17230/44000 | Loss: 2.9995 | LR: 3.41e-04 | 1391.6ms/step | 14.4 samples/s
+Step 17240/44000 | Loss: 2.9744 | LR: 3.41e-04 | 1393.3ms/step | 14.4 samples/s
+Step 17250/44000 | Loss: 2.9778 | LR: 3.41e-04 | 1392.0ms/step | 14.4 samples/s
+Step 17260/44000 | Loss: 2.9987 | LR: 3.41e-04 | 1392.1ms/step | 14.4 samples/s
+Step 17270/44000 | Loss: 2.9892 | LR: 3.41e-04 | 1392.3ms/step | 14.4 samples/s
+Step 17280/44000 | Loss: 3.0014 | LR: 3.40e-04 | 1395.5ms/step | 14.3 samples/s
+Step 17290/44000 | Loss: 3.0176 | LR: 3.40e-04 | 1393.5ms/step | 14.4 samples/s
+Step 17300/44000 | Loss: 2.9963 | LR: 3.40e-04 | 1390.0ms/step | 14.4 samples/s
+Step 17310/44000 | Loss: 3.0070 | LR: 3.40e-04 | 1391.2ms/step | 14.4 samples/s
+Step 17320/44000 | Loss: 3.0012 | LR: 3.40e-04 | 1389.7ms/step | 14.4 samples/s
+Step 17330/44000 | Loss: 3.0012 | LR: 3.40e-04 | 1395.8ms/step | 14.3 samples/s
+Step 17340/44000 | Loss: 2.9895 | LR: 3.39e-04 | 1392.9ms/step | 14.4 samples/s
+Step 17350/44000 | Loss: 2.9804 | LR: 3.39e-04 | 1391.0ms/step | 14.4 samples/s
+Step 17360/44000 | Loss: 2.9887 | LR: 3.39e-04 | 1392.4ms/step | 14.4 samples/s
+Step 17370/44000 | Loss: 2.9760 | LR: 3.39e-04 | 1393.4ms/step | 14.4 samples/s
+Step 17380/44000 | Loss: 2.9965 | LR: 3.39e-04 | 1396.1ms/step | 14.3 samples/s
+Step 17390/44000 | Loss: 2.9764 | LR: 3.39e-04 | 1392.5ms/step | 14.4 samples/s
+Step 17400/44000 | Loss: 2.9813 | LR: 3.38e-04 | 1392.2ms/step | 14.4 samples/s
+Step 17410/44000 | Loss: 2.9788 | LR: 3.38e-04 | 1391.5ms/step | 14.4 samples/s
+Step 17420/44000 | Loss: 2.9823 | LR: 3.38e-04 | 1392.0ms/step | 14.4 samples/s
+Step 17430/44000 | Loss: 3.0083 | LR: 3.38e-04 | 1393.0ms/step | 14.4 samples/s
+Step 17440/44000 | Loss: 2.9813 | LR: 3.38e-04 | 1393.6ms/step | 14.4 samples/s
+Step 17450/44000 | Loss: 3.0100 | LR: 3.38e-04 | 1394.9ms/step | 14.3 samples/s
+Step 17460/44000 | Loss: 2.9793 | LR: 3.37e-04 | 1392.2ms/step | 14.4 samples/s
+Step 17470/44000 | Loss: 2.9795 | LR: 3.37e-04 | 1388.9ms/step | 14.4 samples/s
+Step 17480/44000 | Loss: 2.9965 | LR: 3.37e-04 | 1391.6ms/step | 14.4 samples/s
+Step 17490/44000 | Loss: 2.9569 | LR: 3.37e-04 | 1391.1ms/step | 14.4 samples/s

epfl_final/train_logs/ropectl_a1p0_h100_resume/seed43.20260820T041452Z.log

@@ -0,0 +1,640 @@
+Auto-resuming from latest checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-14000
+Resuming: Using existing output directory /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735
+Running config:
+{
+  "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
+  "model_dtype": "bfloat16",
+  "init_from_scratch": true,
+  "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
+  "val_path": null,
+  "max_seq_length": 8192,
+  "num_workers": 8,
+  "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735",
+  "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-14000",
+  "force_new_run": false,
+  "num_epochs": 1,
+  "max_steps": 44000,
+  "per_device_batch_size": 2,
+  "gradient_accumulation_steps": 10,
+  "learning_rate": 0.0005,
+  "weight_decay": 0.01,
+  "warmup_steps": 750,
+  "lr_scheduler": "cosine",
+  "max_grad_norm": 1.0,
+  "save_steps": 1000,
+  "eval_steps": 1000,
+  "eval_batches": 50,
+  "logging_steps": 10,
+  "seed": 43,
+  "verbose": true,
+  "checkpoint_activations": false,
+  "strategy": "ddp",
+  "mixed_precision": "bf16",
+  "wandb_enabled": true,
+  "wandb_project": "stochastic_sparsity",
+  "wandb_name": "l8k365m_dense_ropectl_a1p0_seed43",
+  "wandb_tags": [
+    "qwen3",
+    "365m",
+    "8k",
+    "dense",
+    "rope-control"
+  ],
+  "hidden_size": 768,
+  "intermediate_size": 3072,
+  "num_attention_heads": 6,
+  "num_key_value_heads": 2,
+  "num_hidden_layers": 28,
+  "head_dim": null,
+  "attention_dropout": 0.0,
+  "rope_train_position_scale_set": [
+    1.0,
+    0.9896339488176222,
+    0.9692159692159692,
+    0.9420289855072463,
+    0.8730158730158729
+  ],
+  "attention_type": "sparse_stochastic",
+  "sparse_block_size": 128,
+  "sparse_num_random_blocks": 31,
+  "sparse_mask_pool_size": 1,
+  "sparse_val_dense_compare": false,
+  "sparse_layer_indices": null,
+  "sparse_mask_cache_dir": null,
+  "sparse_mask_seed": 0,
+  "sparse_mixed_nrb_set": null,
+  "dense_mode": true,
+  "dense_step_intervals": null
+}
+Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
+Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-14000
+Checkpoint activations: False
+dense_mode=True: sparse attention layers will use standard causal SDPA.
+
+
+================================================================================
+MODEL ARCHITECTURE
+================================================================================
+Model config:
+  vocab_size: 151936
+  hidden_size: 768
+  num_hidden_layers: 28
+  num_attention_heads: 6
+  num_key_value_heads: 2
+  head_dim: 128
+  intermediate_size: 3072
+  max_position_embeddings: 32768
+  rope_theta: 1000000
+  rope_train_position_scale_set: [1.0, 0.9896339488176222, 0.9692159692159692, 0.9420289855072463, 0.8730158730158729]
+  attention_type: sparse_stochastic
+  sparse_block_size: 128
+  sparse_num_random_blocks: 31
+  sparse_mask_pool_size: 1
+  sparse_max_seq_length: 8192
+
+Parameter count:
+  Total parameters: 358,958,848
+  Trainable parameters: 358,958,848
+  Non-trainable parameters: 0
+  Model size (BF16): 0.72 GB
+
+--------------------------------------------------------------------------------
+LAYER-BY-LAYER SUMMARY
+--------------------------------------------------------------------------------
+================================================================================================================================================================
+Layer (type:depth-idx)                                       Input Shape               Output Shape              Param #                   Trainable
+================================================================================================================================================================
+Qwen3ForCausalLM                                             [2, 8192]                 --                        --                        True
+├─Qwen3Model: 1-1                                            --                        [2, 8192, 768]            --                        True
+│    └─Embedding: 2-1                                        [2, 8192]                 [2, 8192, 768]            116,686,848               True
+│    └─ModuleList: 2-2                                       --                        --                        --                        True
+│    │    └─Qwen3DecoderLayer: 3-1                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-2                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-3                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-4                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-5                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-6                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-7                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-8                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-9                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-10                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-11                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-12                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-13                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-14                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-15                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-16                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-17                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-18                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-19                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-20                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-21                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-22                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-23                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-24                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-25                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-26                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-27                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-28                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    └─Qwen3RMSNorm: 2-3                                     [2, 8192, 768]            [2, 8192, 768]            768                       True
+├─Linear: 1-2                                                [2, 8192, 768]            [2, 8192, 151936]         116,686,848               True
+================================================================================================================================================================
+Total params: 475,645,696
+Trainable params: 475,645,696
+Non-trainable params: 0
+Total mult-adds (Units.MEGABYTES): 951.29
+================================================================================================================================================================
+Input size (MB): 0.13
+Forward/backward pass size (MB): 31197.23
+Params size (MB): 951.29
+Estimated Total Size (MB): 32148.66
+================================================================================================================================================================
+  NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
+================================================================================
+
+Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
+Using pre-tokenized dataset (no runtime tokenization)
+Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
+Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
+Dataset split: 3027764 train, 159356 validation
+Training examples (packed, train split):  3027764
+Validation examples (packed, val split):  159356
+Per-device batch size:                    2
+World size (GPUs):                        1
+Gradient accumulation steps:              10
+Effective batch size (seqs/step):         20
+Sequence length:                          8192
+Tokens per optimizer step:                163,840
+Micro-batches per epoch (per rank):       1513882
+Optimizer steps per epoch:                151388
+Total optimizer steps:                    44000  (capped by max_steps; 0.29 epochs)
+Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-14000...
+Preserving checkpoint LR state for exact resume: current=[0.00039287714704080913], schedule_base=[0.0005]
+Resumed from checkpoint at step 14000 (epoch 0, micro_step 140000)
+Resuming training from step 14000
+Resuming wandb run ysa9iyov
+wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from WANDB_API_KEY.
+wandb: Currently logged in as: ruben-navasardyan4869 (ruben-navasardyan4869-epfl) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
+wandb: setting up run ysa9iyov
+wandb: Tracking run with wandb version 0.26.1
+wandb: Run data is saved locally in /mloscratch/homes/navasard/sparse/SparseAtt/wandb/run-20260820_041559-ysa9iyov
+wandb: Run `wandb offline` to turn off syncing.
+wandb: Resuming run l8k365m_dense_ropectl_a1p0_seed43
+wandb: ⭐️ View project at https://wandb.ai/ruben-navasardyan4869-epfl/stochastic_sparsity
+wandb: 🚀 View run at https://wandb.ai/ruben-navasardyan4869-epfl/stochastic_sparsity/runs/ysa9iyov
+Starting training...
+Validation at step 14000: Loss = 3.0315, Perplexity = 20.73
+Switching attention mode for step 14001: dense (dense step interval)
+Step 14010/44000 | Loss: 3.0521 | LR: 3.93e-04 | 30764.5ms/step | 0.7 samples/s
+Step 14020/44000 | Loss: 3.0747 | LR: 3.93e-04 | 1400.4ms/step | 14.3 samples/s
+Step 14030/44000 | Loss: 3.0141 | LR: 3.92e-04 | 1401.7ms/step | 14.3 samples/s
+Step 14040/44000 | Loss: 3.0482 | LR: 3.92e-04 | 1404.4ms/step | 14.2 samples/s
+Step 14050/44000 | Loss: 3.0244 | LR: 3.92e-04 | 1402.5ms/step | 14.3 samples/s
+Step 14060/44000 | Loss: 2.9957 | LR: 3.92e-04 | 1405.5ms/step | 14.2 samples/s
+Step 14070/44000 | Loss: 3.0557 | LR: 3.92e-04 | 1404.4ms/step | 14.2 samples/s
+Step 14080/44000 | Loss: 3.0433 | LR: 3.92e-04 | 1407.5ms/step | 14.2 samples/s
+Step 14090/44000 | Loss: 3.0255 | LR: 3.92e-04 | 1406.2ms/step | 14.2 samples/s
+Step 14100/44000 | Loss: 3.0300 | LR: 3.91e-04 | 1401.6ms/step | 14.3 samples/s
+Step 14110/44000 | Loss: 3.0152 | LR: 3.91e-04 | 1400.1ms/step | 14.3 samples/s
+Step 14120/44000 | Loss: 3.0562 | LR: 3.91e-04 | 1400.1ms/step | 14.3 samples/s
+Step 14130/44000 | Loss: 3.0295 | LR: 3.91e-04 | 1399.9ms/step | 14.3 samples/s
+Step 14140/44000 | Loss: 3.0463 | LR: 3.91e-04 | 1400.0ms/step | 14.3 samples/s
+Step 14150/44000 | Loss: 3.0376 | LR: 3.91e-04 | 1400.0ms/step | 14.3 samples/s
+Step 14160/44000 | Loss: 3.0425 | LR: 3.90e-04 | 1399.9ms/step | 14.3 samples/s
+Step 14170/44000 | Loss: 3.0176 | LR: 3.90e-04 | 1400.3ms/step | 14.3 samples/s
+Step 14180/44000 | Loss: 3.0226 | LR: 3.90e-04 | 1399.5ms/step | 14.3 samples/s
+Step 14190/44000 | Loss: 3.0413 | LR: 3.90e-04 | 1400.6ms/step | 14.3 samples/s
+Step 14200/44000 | Loss: 3.0287 | LR: 3.90e-04 | 1399.7ms/step | 14.3 samples/s
+Step 14210/44000 | Loss: 3.0243 | LR: 3.90e-04 | 1399.9ms/step | 14.3 samples/s
+Step 14220/44000 | Loss: 3.0587 | LR: 3.90e-04 | 1400.2ms/step | 14.3 samples/s
+Step 14230/44000 | Loss: 3.0377 | LR: 3.89e-04 | 1399.9ms/step | 14.3 samples/s
+Step 14240/44000 | Loss: 3.0638 | LR: 3.89e-04 | 1400.3ms/step | 14.3 samples/s
+Step 14250/44000 | Loss: 3.0836 | LR: 3.89e-04 | 1401.7ms/step | 14.3 samples/s
+Step 14260/44000 | Loss: 3.0346 | LR: 3.89e-04 | 1400.8ms/step | 14.3 samples/s
+Step 14270/44000 | Loss: 3.0117 | LR: 3.89e-04 | 1401.1ms/step | 14.3 samples/s
+Step 14280/44000 | Loss: 3.0250 | LR: 3.89e-04 | 1400.0ms/step | 14.3 samples/s
+Step 14290/44000 | Loss: 3.0364 | LR: 3.89e-04 | 1399.8ms/step | 14.3 samples/s
+Step 14300/44000 | Loss: 3.0342 | LR: 3.88e-04 | 1400.3ms/step | 14.3 samples/s
+Step 14310/44000 | Loss: 3.0343 | LR: 3.88e-04 | 1399.9ms/step | 14.3 samples/s
+Step 14320/44000 | Loss: 3.0531 | LR: 3.88e-04 | 1399.9ms/step | 14.3 samples/s
+Step 14330/44000 | Loss: 3.0523 | LR: 3.88e-04 | 1400.1ms/step | 14.3 samples/s
+Step 14340/44000 | Loss: 3.0429 | LR: 3.88e-04 | 1399.9ms/step | 14.3 samples/s
+Step 14350/44000 | Loss: 3.0559 | LR: 3.88e-04 | 1400.1ms/step | 14.3 samples/s
+Step 14360/44000 | Loss: 3.0261 | LR: 3.87e-04 | 1399.9ms/step | 14.3 samples/s
+Step 14370/44000 | Loss: 3.0138 | LR: 3.87e-04 | 1400.5ms/step | 14.3 samples/s
+Step 14380/44000 | Loss: 3.0530 | LR: 3.87e-04 | 1399.5ms/step | 14.3 samples/s
+Step 14390/44000 | Loss: 3.0273 | LR: 3.87e-04 | 1400.0ms/step | 14.3 samples/s
+Step 14400/44000 | Loss: 3.0345 | LR: 3.87e-04 | 1399.9ms/step | 14.3 samples/s
+Step 14410/44000 | Loss: 3.0357 | LR: 3.87e-04 | 1400.2ms/step | 14.3 samples/s
+Step 14420/44000 | Loss: 3.0597 | LR: 3.87e-04 | 1399.8ms/step | 14.3 samples/s
+Step 14430/44000 | Loss: 3.0228 | LR: 3.86e-04 | 1400.1ms/step | 14.3 samples/s
+Step 14440/44000 | Loss: 3.0280 | LR: 3.86e-04 | 1399.9ms/step | 14.3 samples/s
+Step 14450/44000 | Loss: 3.0255 | LR: 3.86e-04 | 1400.3ms/step | 14.3 samples/s
+Step 14460/44000 | Loss: 3.0530 | LR: 3.86e-04 | 1399.8ms/step | 14.3 samples/s
+Step 14470/44000 | Loss: 3.0447 | LR: 3.86e-04 | 1399.9ms/step | 14.3 samples/s
+Step 14480/44000 | Loss: 3.0540 | LR: 3.86e-04 | 1400.0ms/step | 14.3 samples/s
+Step 14490/44000 | Loss: 3.0544 | LR: 3.85e-04 | 1400.3ms/step | 14.3 samples/s
+Step 14500/44000 | Loss: 2.9941 | LR: 3.85e-04 | 1400.0ms/step | 14.3 samples/s
+Step 14510/44000 | Loss: 3.0325 | LR: 3.85e-04 | 1399.8ms/step | 14.3 samples/s
+Step 14520/44000 | Loss: 3.0357 | LR: 3.85e-04 | 1399.9ms/step | 14.3 samples/s
+Step 14530/44000 | Loss: 3.0294 | LR: 3.85e-04 | 1400.3ms/step | 14.3 samples/s
+Step 14540/44000 | Loss: 3.0313 | LR: 3.85e-04 | 1399.9ms/step | 14.3 samples/s
+Step 14550/44000 | Loss: 3.0350 | LR: 3.85e-04 | 1399.7ms/step | 14.3 samples/s
+Step 14560/44000 | Loss: 3.0122 | LR: 3.84e-04 | 1400.1ms/step | 14.3 samples/s
+Step 14570/44000 | Loss: 3.0366 | LR: 3.84e-04 | 1400.1ms/step | 14.3 samples/s
+Step 14580/44000 | Loss: 3.0213 | LR: 3.84e-04 | 1400.4ms/step | 14.3 samples/s
+Step 14590/44000 | Loss: 3.0545 | LR: 3.84e-04 | 1399.4ms/step | 14.3 samples/s
+Step 14600/44000 | Loss: 3.0290 | LR: 3.84e-04 | 1400.4ms/step | 14.3 samples/s
+Step 14610/44000 | Loss: 3.0532 | LR: 3.84e-04 | 1400.1ms/step | 14.3 samples/s
+Step 14620/44000 | Loss: 3.0001 | LR: 3.83e-04 | 1399.6ms/step | 14.3 samples/s
+Step 14630/44000 | Loss: 3.0206 | LR: 3.83e-04 | 1399.9ms/step | 14.3 samples/s
+Step 14640/44000 | Loss: 3.0220 | LR: 3.83e-04 | 1400.1ms/step | 14.3 samples/s
+Step 14650/44000 | Loss: 3.0496 | LR: 3.83e-04 | 1400.2ms/step | 14.3 samples/s
+Step 14660/44000 | Loss: 3.0342 | LR: 3.83e-04 | 1399.8ms/step | 14.3 samples/s
+Step 14670/44000 | Loss: 3.0729 | LR: 3.83e-04 | 1399.9ms/step | 14.3 samples/s
+Step 14680/44000 | Loss: 3.0262 | LR: 3.83e-04 | 1400.2ms/step | 14.3 samples/s
+Step 14690/44000 | Loss: 3.0151 | LR: 3.82e-04 | 1400.0ms/step | 14.3 samples/s
+Step 14700/44000 | Loss: 3.0294 | LR: 3.82e-04 | 1399.7ms/step | 14.3 samples/s
+Step 14710/44000 | Loss: 3.0133 | LR: 3.82e-04 | 1400.0ms/step | 14.3 samples/s
+Step 14720/44000 | Loss: 3.0168 | LR: 3.82e-04 | 1400.4ms/step | 14.3 samples/s
+Step 14730/44000 | Loss: 3.0168 | LR: 3.82e-04 | 1400.1ms/step | 14.3 samples/s
+Step 14740/44000 | Loss: 3.0555 | LR: 3.82e-04 | 1400.0ms/step | 14.3 samples/s
+Step 14750/44000 | Loss: 3.0453 | LR: 3.81e-04 | 1399.5ms/step | 14.3 samples/s
+Step 14760/44000 | Loss: 3.0187 | LR: 3.81e-04 | 1400.1ms/step | 14.3 samples/s
+Step 14770/44000 | Loss: 3.0488 | LR: 3.81e-04 | 1400.2ms/step | 14.3 samples/s
+Step 14780/44000 | Loss: 3.0276 | LR: 3.81e-04 | 1400.0ms/step | 14.3 samples/s
+Step 14790/44000 | Loss: 3.0210 | LR: 3.81e-04 | 1399.7ms/step | 14.3 samples/s
+Step 14800/44000 | Loss: 3.0516 | LR: 3.81e-04 | 1400.1ms/step | 14.3 samples/s
+Step 14810/44000 | Loss: 3.0319 | LR: 3.81e-04 | 1399.9ms/step | 14.3 samples/s
+Step 14820/44000 | Loss: 3.0442 | LR: 3.80e-04 | 1400.0ms/step | 14.3 samples/s
+Step 14830/44000 | Loss: 3.0371 | LR: 3.80e-04 | 1400.1ms/step | 14.3 samples/s
+Step 14840/44000 | Loss: 3.0045 | LR: 3.80e-04 | 1399.8ms/step | 14.3 samples/s
+Step 14850/44000 | Loss: 2.9777 | LR: 3.80e-04 | 1400.3ms/step | 14.3 samples/s
+Step 14860/44000 | Loss: 3.0043 | LR: 3.80e-04 | 1399.7ms/step | 14.3 samples/s
+Step 14870/44000 | Loss: 3.0675 | LR: 3.80e-04 | 1400.5ms/step | 14.3 samples/s
+Step 14880/44000 | Loss: 3.0157 | LR: 3.79e-04 | 1399.5ms/step | 14.3 samples/s
+Step 14890/44000 | Loss: 3.0207 | LR: 3.79e-04 | 1400.1ms/step | 14.3 samples/s
+Step 14900/44000 | Loss: 3.0087 | LR: 3.79e-04 | 1400.0ms/step | 14.3 samples/s
+Step 14910/44000 | Loss: 3.0392 | LR: 3.79e-04 | 1399.9ms/step | 14.3 samples/s
+Step 14920/44000 | Loss: 3.0563 | LR: 3.79e-04 | 1400.3ms/step | 14.3 samples/s
+Step 14930/44000 | Loss: 3.0244 | LR: 3.79e-04 | 1400.0ms/step | 14.3 samples/s
+Step 14940/44000 | Loss: 3.0206 | LR: 3.79e-04 | 1402.0ms/step | 14.3 samples/s
+Step 14950/44000 | Loss: 3.0400 | LR: 3.78e-04 | 1399.8ms/step | 14.3 samples/s
+Step 14960/44000 | Loss: 3.0309 | LR: 3.78e-04 | 1400.3ms/step | 14.3 samples/s
+Step 14970/44000 | Loss: 3.0373 | LR: 3.78e-04 | 1401.5ms/step | 14.3 samples/s
+Step 14980/44000 | Loss: 3.0326 | LR: 3.78e-04 | 1400.2ms/step | 14.3 samples/s
+Step 14990/44000 | Loss: 3.0105 | LR: 3.78e-04 | 1400.1ms/step | 14.3 samples/s
+Step 15000/44000 | Loss: 3.0185 | LR: 3.78e-04 | 1400.4ms/step | 14.3 samples/s
+Validation at step 15000: Loss = 3.0168, Perplexity = 20.43
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-15000
+Step 15010/44000 | Loss: 3.0053 | LR: 3.77e-04 | 3941.5ms/step | 5.1 samples/s
+Step 15020/44000 | Loss: 3.0368 | LR: 3.77e-04 | 1402.4ms/step | 14.3 samples/s
+Step 15030/44000 | Loss: 3.0373 | LR: 3.77e-04 | 1409.6ms/step | 14.2 samples/s
+Step 15040/44000 | Loss: 3.0137 | LR: 3.77e-04 | 1410.2ms/step | 14.2 samples/s
+Step 15050/44000 | Loss: 3.0237 | LR: 3.77e-04 | 1411.9ms/step | 14.2 samples/s
+Step 15060/44000 | Loss: 3.0238 | LR: 3.77e-04 | 1409.0ms/step | 14.2 samples/s
+Step 15070/44000 | Loss: 3.0276 | LR: 3.77e-04 | 1412.4ms/step | 14.2 samples/s
+Step 15080/44000 | Loss: 3.0443 | LR: 3.76e-04 | 1411.0ms/step | 14.2 samples/s
+Step 15090/44000 | Loss: 3.0242 | LR: 3.76e-04 | 1410.4ms/step | 14.2 samples/s
+Step 15100/44000 | Loss: 3.0022 | LR: 3.76e-04 | 1411.3ms/step | 14.2 samples/s
+Step 15110/44000 | Loss: 3.0292 | LR: 3.76e-04 | 1412.4ms/step | 14.2 samples/s
+Step 15120/44000 | Loss: 3.0013 | LR: 3.76e-04 | 1414.7ms/step | 14.1 samples/s
+Step 15130/44000 | Loss: 3.0519 | LR: 3.76e-04 | 1409.6ms/step | 14.2 samples/s
+Step 15140/44000 | Loss: 3.0067 | LR: 3.75e-04 | 1411.2ms/step | 14.2 samples/s
+Step 15150/44000 | Loss: 3.0282 | LR: 3.75e-04 | 1412.2ms/step | 14.2 samples/s
+Step 15160/44000 | Loss: 3.0164 | LR: 3.75e-04 | 1411.7ms/step | 14.2 samples/s
+Step 15170/44000 | Loss: 3.0404 | LR: 3.75e-04 | 1410.0ms/step | 14.2 samples/s
+Step 15180/44000 | Loss: 2.9985 | LR: 3.75e-04 | 1408.0ms/step | 14.2 samples/s
+Step 15190/44000 | Loss: 3.0056 | LR: 3.75e-04 | 1412.1ms/step | 14.2 samples/s
+Step 15200/44000 | Loss: 3.0177 | LR: 3.74e-04 | 1413.8ms/step | 14.1 samples/s
+Step 15210/44000 | Loss: 3.0339 | LR: 3.74e-04 | 1409.1ms/step | 14.2 samples/s
+Step 15220/44000 | Loss: 3.0289 | LR: 3.74e-04 | 1409.4ms/step | 14.2 samples/s
+Step 15230/44000 | Loss: 3.0372 | LR: 3.74e-04 | 1409.8ms/step | 14.2 samples/s
+Step 15240/44000 | Loss: 3.0108 | LR: 3.74e-04 | 1412.0ms/step | 14.2 samples/s
+Step 15250/44000 | Loss: 3.0425 | LR: 3.74e-04 | 1408.8ms/step | 14.2 samples/s
+Step 15260/44000 | Loss: 3.0209 | LR: 3.74e-04 | 1408.6ms/step | 14.2 samples/s
+Step 15270/44000 | Loss: 3.0247 | LR: 3.73e-04 | 1412.5ms/step | 14.2 samples/s
+Step 15280/44000 | Loss: 3.0363 | LR: 3.73e-04 | 1412.3ms/step | 14.2 samples/s
+Step 15290/44000 | Loss: 3.0178 | LR: 3.73e-04 | 1412.4ms/step | 14.2 samples/s
+Step 15300/44000 | Loss: 3.0352 | LR: 3.73e-04 | 1410.6ms/step | 14.2 samples/s
+Step 15310/44000 | Loss: 3.0114 | LR: 3.73e-04 | 1413.1ms/step | 14.2 samples/s
+Step 15320/44000 | Loss: 3.0030 | LR: 3.73e-04 | 1411.7ms/step | 14.2 samples/s
+Step 15330/44000 | Loss: 3.0128 | LR: 3.72e-04 | 1411.8ms/step | 14.2 samples/s
+Step 15340/44000 | Loss: 3.0430 | LR: 3.72e-04 | 1410.2ms/step | 14.2 samples/s
+Step 15350/44000 | Loss: 3.0171 | LR: 3.72e-04 | 1410.0ms/step | 14.2 samples/s
+Step 15360/44000 | Loss: 3.0435 | LR: 3.72e-04 | 1407.8ms/step | 14.2 samples/s
+Step 15370/44000 | Loss: 3.0452 | LR: 3.72e-04 | 1409.5ms/step | 14.2 samples/s
+Step 15380/44000 | Loss: 3.0110 | LR: 3.72e-04 | 1408.6ms/step | 14.2 samples/s
+Step 15390/44000 | Loss: 3.0187 | LR: 3.71e-04 | 1410.5ms/step | 14.2 samples/s
+Step 15400/44000 | Loss: 3.0359 | LR: 3.71e-04 | 1410.7ms/step | 14.2 samples/s
+Step 15410/44000 | Loss: 3.0216 | LR: 3.71e-04 | 1411.2ms/step | 14.2 samples/s
+Step 15420/44000 | Loss: 3.0334 | LR: 3.71e-04 | 1412.5ms/step | 14.2 samples/s
+Step 15430/44000 | Loss: 3.0338 | LR: 3.71e-04 | 1411.6ms/step | 14.2 samples/s
+Step 15440/44000 | Loss: 3.0213 | LR: 3.71e-04 | 1411.4ms/step | 14.2 samples/s
+Step 15450/44000 | Loss: 3.0234 | LR: 3.71e-04 | 1410.6ms/step | 14.2 samples/s
+Step 15460/44000 | Loss: 3.0322 | LR: 3.70e-04 | 1411.4ms/step | 14.2 samples/s
+Step 15470/44000 | Loss: 3.0304 | LR: 3.70e-04 | 1412.0ms/step | 14.2 samples/s
+Step 15480/44000 | Loss: 3.0119 | LR: 3.70e-04 | 1412.8ms/step | 14.2 samples/s
+Step 15490/44000 | Loss: 3.0070 | LR: 3.70e-04 | 1409.2ms/step | 14.2 samples/s
+Step 15500/44000 | Loss: 3.0243 | LR: 3.70e-04 | 1410.9ms/step | 14.2 samples/s
+Step 15510/44000 | Loss: 3.0337 | LR: 3.70e-04 | 1410.5ms/step | 14.2 samples/s
+Step 15520/44000 | Loss: 3.0178 | LR: 3.69e-04 | 1408.4ms/step | 14.2 samples/s
+Step 15530/44000 | Loss: 2.9951 | LR: 3.69e-04 | 1409.1ms/step | 14.2 samples/s
+Step 15540/44000 | Loss: 3.0524 | LR: 3.69e-04 | 1411.5ms/step | 14.2 samples/s
+Step 15550/44000 | Loss: 3.0217 | LR: 3.69e-04 | 1408.2ms/step | 14.2 samples/s
+Step 15560/44000 | Loss: 3.0148 | LR: 3.69e-04 | 1409.3ms/step | 14.2 samples/s
+Step 15570/44000 | Loss: 2.9971 | LR: 3.69e-04 | 1410.2ms/step | 14.2 samples/s
+Step 15580/44000 | Loss: 3.0258 | LR: 3.68e-04 | 1410.1ms/step | 14.2 samples/s
+Step 15590/44000 | Loss: 3.0436 | LR: 3.68e-04 | 1412.1ms/step | 14.2 samples/s
+Step 15600/44000 | Loss: 3.0186 | LR: 3.68e-04 | 1407.6ms/step | 14.2 samples/s
+Step 15610/44000 | Loss: 2.9948 | LR: 3.68e-04 | 1410.4ms/step | 14.2 samples/s
+Step 15620/44000 | Loss: 3.0137 | LR: 3.68e-04 | 1412.1ms/step | 14.2 samples/s
+Step 15630/44000 | Loss: 3.0320 | LR: 3.68e-04 | 1412.2ms/step | 14.2 samples/s
+Step 15640/44000 | Loss: 3.0359 | LR: 3.67e-04 | 1411.8ms/step | 14.2 samples/s
+Step 15650/44000 | Loss: 3.0122 | LR: 3.67e-04 | 1410.7ms/step | 14.2 samples/s
+Step 15660/44000 | Loss: 3.0295 | LR: 3.67e-04 | 1411.1ms/step | 14.2 samples/s
+Step 15670/44000 | Loss: 3.0124 | LR: 3.67e-04 | 1411.2ms/step | 14.2 samples/s
+Step 15680/44000 | Loss: 3.0503 | LR: 3.67e-04 | 1411.2ms/step | 14.2 samples/s
+Step 15690/44000 | Loss: 3.0024 | LR: 3.67e-04 | 1409.5ms/step | 14.2 samples/s
+Step 15700/44000 | Loss: 3.0032 | LR: 3.67e-04 | 1408.5ms/step | 14.2 samples/s
+Step 15710/44000 | Loss: 3.0169 | LR: 3.66e-04 | 1412.2ms/step | 14.2 samples/s
+Step 15720/44000 | Loss: 2.9977 | LR: 3.66e-04 | 1411.1ms/step | 14.2 samples/s
+Step 15730/44000 | Loss: 3.0203 | LR: 3.66e-04 | 1408.1ms/step | 14.2 samples/s
+Step 15740/44000 | Loss: 3.0099 | LR: 3.66e-04 | 1412.6ms/step | 14.2 samples/s
+Step 15750/44000 | Loss: 3.0203 | LR: 3.66e-04 | 1409.6ms/step | 14.2 samples/s
+Step 15760/44000 | Loss: 3.0059 | LR: 3.66e-04 | 1411.6ms/step | 14.2 samples/s
+Step 15770/44000 | Loss: 3.0506 | LR: 3.65e-04 | 1412.3ms/step | 14.2 samples/s
+Step 15780/44000 | Loss: 2.9989 | LR: 3.65e-04 | 1412.1ms/step | 14.2 samples/s
+Step 15790/44000 | Loss: 3.0394 | LR: 3.65e-04 | 1412.1ms/step | 14.2 samples/s
+Step 15800/44000 | Loss: 3.0058 | LR: 3.65e-04 | 1414.2ms/step | 14.1 samples/s
+Step 15810/44000 | Loss: 3.0214 | LR: 3.65e-04 | 1411.8ms/step | 14.2 samples/s
+Step 15820/44000 | Loss: 2.9922 | LR: 3.65e-04 | 1408.5ms/step | 14.2 samples/s
+Step 15830/44000 | Loss: 2.9982 | LR: 3.64e-04 | 1407.2ms/step | 14.2 samples/s
+Step 15840/44000 | Loss: 3.0272 | LR: 3.64e-04 | 1411.9ms/step | 14.2 samples/s
+Step 15850/44000 | Loss: 3.0119 | LR: 3.64e-04 | 1411.0ms/step | 14.2 samples/s
+Step 15860/44000 | Loss: 3.0130 | LR: 3.64e-04 | 1411.3ms/step | 14.2 samples/s
+Step 15870/44000 | Loss: 3.0085 | LR: 3.64e-04 | 1409.9ms/step | 14.2 samples/s
+Step 15880/44000 | Loss: 3.0284 | LR: 3.64e-04 | 1412.5ms/step | 14.2 samples/s
+Step 15890/44000 | Loss: 2.9768 | LR: 3.63e-04 | 1411.4ms/step | 14.2 samples/s
+Step 15900/44000 | Loss: 3.0116 | LR: 3.63e-04 | 1410.3ms/step | 14.2 samples/s
+Step 15910/44000 | Loss: 3.0028 | LR: 3.63e-04 | 1408.9ms/step | 14.2 samples/s
+Step 15920/44000 | Loss: 3.0251 | LR: 3.63e-04 | 1407.7ms/step | 14.2 samples/s
+Step 15930/44000 | Loss: 3.0078 | LR: 3.63e-04 | 1409.8ms/step | 14.2 samples/s
+Step 15940/44000 | Loss: 3.0058 | LR: 3.63e-04 | 1410.9ms/step | 14.2 samples/s
+Step 15950/44000 | Loss: 3.0315 | LR: 3.62e-04 | 1410.2ms/step | 14.2 samples/s
+Step 15960/44000 | Loss: 2.9918 | LR: 3.62e-04 | 1410.2ms/step | 14.2 samples/s
+Step 15970/44000 | Loss: 2.9993 | LR: 3.62e-04 | 1410.8ms/step | 14.2 samples/s
+Step 15980/44000 | Loss: 3.0220 | LR: 3.62e-04 | 1409.8ms/step | 14.2 samples/s
+Step 15990/44000 | Loss: 3.0168 | LR: 3.62e-04 | 1408.6ms/step | 14.2 samples/s
+Step 16000/44000 | Loss: 3.0033 | LR: 3.62e-04 | 1407.0ms/step | 14.2 samples/s
+Validation at step 16000: Loss = 3.0032, Perplexity = 20.15
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-16000
+Step 16010/44000 | Loss: 3.0004 | LR: 3.62e-04 | 3962.0ms/step | 5.0 samples/s
+Step 16020/44000 | Loss: 3.0341 | LR: 3.61e-04 | 1403.8ms/step | 14.2 samples/s
+Step 16030/44000 | Loss: 2.9863 | LR: 3.61e-04 | 1406.9ms/step | 14.2 samples/s
+Step 16040/44000 | Loss: 3.0186 | LR: 3.61e-04 | 1400.9ms/step | 14.3 samples/s
+Step 16050/44000 | Loss: 3.0309 | LR: 3.61e-04 | 1402.2ms/step | 14.3 samples/s
+Step 16060/44000 | Loss: 3.0071 | LR: 3.61e-04 | 1404.0ms/step | 14.2 samples/s
+Step 16070/44000 | Loss: 2.9967 | LR: 3.61e-04 | 1401.9ms/step | 14.3 samples/s
+Step 16080/44000 | Loss: 3.0232 | LR: 3.60e-04 | 1400.0ms/step | 14.3 samples/s
+Step 16090/44000 | Loss: 3.0047 | LR: 3.60e-04 | 1400.3ms/step | 14.3 samples/s
+Step 16100/44000 | Loss: 3.0289 | LR: 3.60e-04 | 1401.8ms/step | 14.3 samples/s
+Step 16110/44000 | Loss: 2.9921 | LR: 3.60e-04 | 1399.6ms/step | 14.3 samples/s
+Step 16120/44000 | Loss: 3.0093 | LR: 3.60e-04 | 1400.4ms/step | 14.3 samples/s
+Step 16130/44000 | Loss: 2.9896 | LR: 3.60e-04 | 1400.0ms/step | 14.3 samples/s
+Step 16140/44000 | Loss: 2.9900 | LR: 3.59e-04 | 1402.5ms/step | 14.3 samples/s
+Step 16150/44000 | Loss: 3.0304 | LR: 3.59e-04 | 1404.2ms/step | 14.2 samples/s
+Step 16160/44000 | Loss: 3.0094 | LR: 3.59e-04 | 1403.2ms/step | 14.3 samples/s
+Step 16170/44000 | Loss: 2.9771 | LR: 3.59e-04 | 1402.0ms/step | 14.3 samples/s
+Step 16180/44000 | Loss: 2.9958 | LR: 3.59e-04 | 1404.3ms/step | 14.2 samples/s
+Step 16190/44000 | Loss: 2.9992 | LR: 3.59e-04 | 1401.7ms/step | 14.3 samples/s
+Step 16200/44000 | Loss: 3.0221 | LR: 3.58e-04 | 1403.7ms/step | 14.2 samples/s
+Step 16210/44000 | Loss: 3.0118 | LR: 3.58e-04 | 1400.2ms/step | 14.3 samples/s
+Step 16220/44000 | Loss: 3.0274 | LR: 3.58e-04 | 1402.1ms/step | 14.3 samples/s
+Step 16230/44000 | Loss: 3.0129 | LR: 3.58e-04 | 1402.1ms/step | 14.3 samples/s
+Step 16240/44000 | Loss: 2.9960 | LR: 3.58e-04 | 1404.5ms/step | 14.2 samples/s
+Step 16250/44000 | Loss: 3.0097 | LR: 3.58e-04 | 1400.9ms/step | 14.3 samples/s
+Step 16260/44000 | Loss: 3.0072 | LR: 3.57e-04 | 1404.0ms/step | 14.2 samples/s
+Step 16270/44000 | Loss: 2.9829 | LR: 3.57e-04 | 1402.4ms/step | 14.3 samples/s
+Step 16280/44000 | Loss: 2.9928 | LR: 3.57e-04 | 1401.7ms/step | 14.3 samples/s
+Step 16290/44000 | Loss: 3.0136 | LR: 3.57e-04 | 1400.4ms/step | 14.3 samples/s
+Step 16300/44000 | Loss: 3.0019 | LR: 3.57e-04 | 1401.5ms/step | 14.3 samples/s
+Step 16310/44000 | Loss: 3.0136 | LR: 3.57e-04 | 1402.7ms/step | 14.3 samples/s
+Step 16320/44000 | Loss: 3.0140 | LR: 3.56e-04 | 1401.6ms/step | 14.3 samples/s
+Step 16330/44000 | Loss: 3.0250 | LR: 3.56e-04 | 1401.0ms/step | 14.3 samples/s
+Step 16340/44000 | Loss: 2.9840 | LR: 3.56e-04 | 1401.0ms/step | 14.3 samples/s
+Step 16350/44000 | Loss: 3.0017 | LR: 3.56e-04 | 1399.6ms/step | 14.3 samples/s
+Step 16360/44000 | Loss: 3.0169 | LR: 3.56e-04 | 1401.0ms/step | 14.3 samples/s
+Step 16370/44000 | Loss: 3.0185 | LR: 3.56e-04 | 1399.2ms/step | 14.3 samples/s
+Step 16380/44000 | Loss: 2.9950 | LR: 3.55e-04 | 1402.1ms/step | 14.3 samples/s
+Step 16390/44000 | Loss: 3.0218 | LR: 3.55e-04 | 1400.4ms/step | 14.3 samples/s
+Step 16400/44000 | Loss: 2.9841 | LR: 3.55e-04 | 1399.6ms/step | 14.3 samples/s
+Step 16410/44000 | Loss: 3.0440 | LR: 3.55e-04 | 1400.1ms/step | 14.3 samples/s
+Step 16420/44000 | Loss: 2.9912 | LR: 3.55e-04 | 1402.4ms/step | 14.3 samples/s
+Step 16430/44000 | Loss: 2.9887 | LR: 3.55e-04 | 1403.3ms/step | 14.3 samples/s
+Step 16440/44000 | Loss: 3.0123 | LR: 3.54e-04 | 1400.6ms/step | 14.3 samples/s
+Step 16450/44000 | Loss: 2.9977 | LR: 3.54e-04 | 1402.2ms/step | 14.3 samples/s
+Step 16460/44000 | Loss: 2.9785 | LR: 3.54e-04 | 1402.2ms/step | 14.3 samples/s
+Step 16470/44000 | Loss: 2.9969 | LR: 3.54e-04 | 1401.2ms/step | 14.3 samples/s
+Step 16480/44000 | Loss: 3.0249 | LR: 3.54e-04 | 1400.0ms/step | 14.3 samples/s
+Step 16490/44000 | Loss: 3.0014 | LR: 3.54e-04 | 1400.1ms/step | 14.3 samples/s
+Step 16500/44000 | Loss: 3.0037 | LR: 3.53e-04 | 1400.3ms/step | 14.3 samples/s
+Step 16510/44000 | Loss: 3.0069 | LR: 3.53e-04 | 1399.6ms/step | 14.3 samples/s
+Step 16520/44000 | Loss: 2.9786 | LR: 3.53e-04 | 1400.2ms/step | 14.3 samples/s
+Step 16530/44000 | Loss: 3.0031 | LR: 3.53e-04 | 1402.0ms/step | 14.3 samples/s
+Step 16540/44000 | Loss: 3.0183 | LR: 3.53e-04 | 1400.4ms/step | 14.3 samples/s
+Step 16550/44000 | Loss: 3.0102 | LR: 3.53e-04 | 1401.2ms/step | 14.3 samples/s
+Step 16560/44000 | Loss: 3.0114 | LR: 3.52e-04 | 1400.0ms/step | 14.3 samples/s
+Step 16570/44000 | Loss: 2.9928 | LR: 3.52e-04 | 1400.5ms/step | 14.3 samples/s
+Step 16580/44000 | Loss: 2.9866 | LR: 3.52e-04 | 1400.1ms/step | 14.3 samples/s
+Step 16590/44000 | Loss: 3.0028 | LR: 3.52e-04 | 1402.4ms/step | 14.3 samples/s
+Step 16600/44000 | Loss: 2.9975 | LR: 3.52e-04 | 1403.3ms/step | 14.3 samples/s
+Step 16610/44000 | Loss: 3.0065 | LR: 3.52e-04 | 1402.0ms/step | 14.3 samples/s
+Step 16620/44000 | Loss: 3.0008 | LR: 3.51e-04 | 1403.8ms/step | 14.2 samples/s
+Step 16630/44000 | Loss: 3.0111 | LR: 3.51e-04 | 1402.6ms/step | 14.3 samples/s
+Step 16640/44000 | Loss: 2.9828 | LR: 3.51e-04 | 1405.3ms/step | 14.2 samples/s
+Step 16650/44000 | Loss: 3.0010 | LR: 3.51e-04 | 1404.1ms/step | 14.2 samples/s
+Step 16660/44000 | Loss: 3.0140 | LR: 3.51e-04 | 1404.2ms/step | 14.2 samples/s
+Step 16670/44000 | Loss: 2.9953 | LR: 3.51e-04 | 1402.2ms/step | 14.3 samples/s
+Step 16680/44000 | Loss: 3.0067 | LR: 3.50e-04 | 1401.5ms/step | 14.3 samples/s
+Step 16690/44000 | Loss: 3.0189 | LR: 3.50e-04 | 1402.4ms/step | 14.3 samples/s
+Step 16700/44000 | Loss: 3.0106 | LR: 3.50e-04 | 1406.0ms/step | 14.2 samples/s
+Step 16710/44000 | Loss: 3.0117 | LR: 3.50e-04 | 1403.8ms/step | 14.2 samples/s
+Step 16720/44000 | Loss: 3.0164 | LR: 3.50e-04 | 1404.1ms/step | 14.2 samples/s
+Step 16730/44000 | Loss: 3.0528 | LR: 3.50e-04 | 1403.9ms/step | 14.2 samples/s
+Step 16740/44000 | Loss: 2.9955 | LR: 3.49e-04 | 1402.1ms/step | 14.3 samples/s
+Step 16750/44000 | Loss: 3.0083 | LR: 3.49e-04 | 1399.5ms/step | 14.3 samples/s
+Step 16760/44000 | Loss: 2.9814 | LR: 3.49e-04 | 1400.8ms/step | 14.3 samples/s
+Step 16770/44000 | Loss: 2.9909 | LR: 3.49e-04 | 1403.7ms/step | 14.2 samples/s
+Step 16780/44000 | Loss: 2.9880 | LR: 3.49e-04 | 1402.2ms/step | 14.3 samples/s
+Step 16790/44000 | Loss: 3.0199 | LR: 3.49e-04 | 1404.3ms/step | 14.2 samples/s
+Step 16800/44000 | Loss: 2.9944 | LR: 3.48e-04 | 1404.6ms/step | 14.2 samples/s
+Step 16810/44000 | Loss: 3.0162 | LR: 3.48e-04 | 1400.8ms/step | 14.3 samples/s
+Step 16820/44000 | Loss: 2.9954 | LR: 3.48e-04 | 1401.6ms/step | 14.3 samples/s
+Step 16830/44000 | Loss: 3.0058 | LR: 3.48e-04 | 1400.2ms/step | 14.3 samples/s
+Step 16840/44000 | Loss: 3.0278 | LR: 3.48e-04 | 1402.3ms/step | 14.3 samples/s
+Step 16850/44000 | Loss: 3.0013 | LR: 3.48e-04 | 1401.7ms/step | 14.3 samples/s
+Step 16860/44000 | Loss: 3.0208 | LR: 3.47e-04 | 1400.3ms/step | 14.3 samples/s
+Step 16870/44000 | Loss: 3.0287 | LR: 3.47e-04 | 1403.6ms/step | 14.2 samples/s
+Step 16880/44000 | Loss: 2.9802 | LR: 3.47e-04 | 1400.0ms/step | 14.3 samples/s
+Step 16890/44000 | Loss: 3.0092 | LR: 3.47e-04 | 1402.5ms/step | 14.3 samples/s
+Step 16900/44000 | Loss: 3.0112 | LR: 3.47e-04 | 1399.4ms/step | 14.3 samples/s
+Step 16910/44000 | Loss: 2.9900 | LR: 3.47e-04 | 1402.6ms/step | 14.3 samples/s
+Step 16920/44000 | Loss: 3.0025 | LR: 3.46e-04 | 1400.0ms/step | 14.3 samples/s
+Step 16930/44000 | Loss: 3.0420 | LR: 3.46e-04 | 1402.6ms/step | 14.3 samples/s
+Step 16940/44000 | Loss: 2.9802 | LR: 3.46e-04 | 1403.0ms/step | 14.3 samples/s
+Step 16950/44000 | Loss: 3.0236 | LR: 3.46e-04 | 1402.2ms/step | 14.3 samples/s
+Step 16960/44000 | Loss: 2.9748 | LR: 3.46e-04 | 1399.9ms/step | 14.3 samples/s
+Step 16970/44000 | Loss: 2.9966 | LR: 3.46e-04 | 1402.0ms/step | 14.3 samples/s
+Step 16980/44000 | Loss: 3.0043 | LR: 3.45e-04 | 1401.6ms/step | 14.3 samples/s
+Step 16990/44000 | Loss: 3.0047 | LR: 3.45e-04 | 1400.0ms/step | 14.3 samples/s
+Step 17000/44000 | Loss: 3.0066 | LR: 3.45e-04 | 1399.9ms/step | 14.3 samples/s
+Validation at step 17000: Loss = 2.9912, Perplexity = 19.91
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-17000
+Step 17010/44000 | Loss: 2.9859 | LR: 3.45e-04 | 3948.6ms/step | 5.1 samples/s
+Step 17020/44000 | Loss: 3.0082 | LR: 3.45e-04 | 1403.8ms/step | 14.2 samples/s
+Step 17030/44000 | Loss: 2.9998 | LR: 3.45e-04 | 1401.9ms/step | 14.3 samples/s
+Step 17040/44000 | Loss: 3.0025 | LR: 3.44e-04 | 1407.1ms/step | 14.2 samples/s
+Step 17050/44000 | Loss: 3.0099 | LR: 3.44e-04 | 1408.8ms/step | 14.2 samples/s
+Step 17060/44000 | Loss: 3.0021 | LR: 3.44e-04 | 1409.4ms/step | 14.2 samples/s
+Step 17070/44000 | Loss: 2.9977 | LR: 3.44e-04 | 1410.9ms/step | 14.2 samples/s
+Step 17080/44000 | Loss: 3.0071 | LR: 3.44e-04 | 1409.5ms/step | 14.2 samples/s
+Step 17090/44000 | Loss: 2.9980 | LR: 3.44e-04 | 1407.8ms/step | 14.2 samples/s
+Step 17100/44000 | Loss: 2.9998 | LR: 3.43e-04 | 1408.5ms/step | 14.2 samples/s
+Step 17110/44000 | Loss: 3.0134 | LR: 3.43e-04 | 1408.3ms/step | 14.2 samples/s
+Step 17120/44000 | Loss: 3.0028 | LR: 3.43e-04 | 1412.4ms/step | 14.2 samples/s
+Step 17130/44000 | Loss: 3.0126 | LR: 3.43e-04 | 1409.1ms/step | 14.2 samples/s
+Step 17140/44000 | Loss: 2.9939 | LR: 3.43e-04 | 1408.1ms/step | 14.2 samples/s
+Step 17150/44000 | Loss: 3.0095 | LR: 3.43e-04 | 1409.9ms/step | 14.2 samples/s
+Step 17160/44000 | Loss: 2.9964 | LR: 3.42e-04 | 1410.6ms/step | 14.2 samples/s
+Step 17170/44000 | Loss: 2.9783 | LR: 3.42e-04 | 1411.3ms/step | 14.2 samples/s
+Step 17180/44000 | Loss: 3.0097 | LR: 3.42e-04 | 1410.4ms/step | 14.2 samples/s
+Step 17190/44000 | Loss: 2.9733 | LR: 3.42e-04 | 1407.8ms/step | 14.2 samples/s
+Step 17200/44000 | Loss: 3.0141 | LR: 3.42e-04 | 1409.9ms/step | 14.2 samples/s
+Step 17210/44000 | Loss: 3.0139 | LR: 3.42e-04 | 1406.3ms/step | 14.2 samples/s
+Step 17220/44000 | Loss: 2.9842 | LR: 3.41e-04 | 1405.9ms/step | 14.2 samples/s
+Step 17230/44000 | Loss: 2.9903 | LR: 3.41e-04 | 1406.2ms/step | 14.2 samples/s
+Step 17240/44000 | Loss: 3.0228 | LR: 3.41e-04 | 1410.1ms/step | 14.2 samples/s
+Step 17250/44000 | Loss: 3.0173 | LR: 3.41e-04 | 1410.3ms/step | 14.2 samples/s
+Step 17260/44000 | Loss: 2.9775 | LR: 3.41e-04 | 1413.2ms/step | 14.2 samples/s
+Step 17270/44000 | Loss: 2.9755 | LR: 3.41e-04 | 1408.4ms/step | 14.2 samples/s
+Step 17280/44000 | Loss: 3.0070 | LR: 3.40e-04 | 1410.0ms/step | 14.2 samples/s
+Step 17290/44000 | Loss: 2.9774 | LR: 3.40e-04 | 1409.8ms/step | 14.2 samples/s
+Step 17300/44000 | Loss: 2.9916 | LR: 3.40e-04 | 1406.9ms/step | 14.2 samples/s
+Step 17310/44000 | Loss: 3.0065 | LR: 3.40e-04 | 1406.8ms/step | 14.2 samples/s
+Step 17320/44000 | Loss: 3.0215 | LR: 3.40e-04 | 1410.0ms/step | 14.2 samples/s
+Step 17330/44000 | Loss: 3.0171 | LR: 3.40e-04 | 1409.5ms/step | 14.2 samples/s
+Step 17340/44000 | Loss: 3.0105 | LR: 3.39e-04 | 1409.0ms/step | 14.2 samples/s
+Step 17350/44000 | Loss: 3.0224 | LR: 3.39e-04 | 1405.4ms/step | 14.2 samples/s
+Step 17360/44000 | Loss: 2.9547 | LR: 3.39e-04 | 1399.9ms/step | 14.3 samples/s
+Step 17370/44000 | Loss: 3.0315 | LR: 3.39e-04 | 1402.9ms/step | 14.3 samples/s
+Step 17380/44000 | Loss: 2.9934 | LR: 3.39e-04 | 1401.7ms/step | 14.3 samples/s
+Step 17390/44000 | Loss: 3.0234 | LR: 3.39e-04 | 1402.0ms/step | 14.3 samples/s
+Step 17400/44000 | Loss: 3.0135 | LR: 3.38e-04 | 1401.7ms/step | 14.3 samples/s
+Step 17410/44000 | Loss: 2.9675 | LR: 3.38e-04 | 1401.7ms/step | 14.3 samples/s
+Step 17420/44000 | Loss: 2.9944 | LR: 3.38e-04 | 1402.1ms/step | 14.3 samples/s
+Step 17430/44000 | Loss: 2.9855 | LR: 3.38e-04 | 1400.3ms/step | 14.3 samples/s
+Step 17440/44000 | Loss: 3.0193 | LR: 3.38e-04 | 1400.6ms/step | 14.3 samples/s
+Step 17450/44000 | Loss: 3.0119 | LR: 3.38e-04 | 1401.8ms/step | 14.3 samples/s
+Step 17460/44000 | Loss: 2.9935 | LR: 3.37e-04 | 1401.1ms/step | 14.3 samples/s
+Step 17470/44000 | Loss: 2.9749 | LR: 3.37e-04 | 1399.9ms/step | 14.3 samples/s
+Step 17480/44000 | Loss: 3.0113 | LR: 3.37e-04 | 1400.2ms/step | 14.3 samples/s
+Step 17490/44000 | Loss: 2.9957 | LR: 3.37e-04 | 1406.0ms/step | 14.2 samples/s
+Step 17500/44000 | Loss: 2.9999 | LR: 3.37e-04 | 1402.3ms/step | 14.3 samples/s
+Step 17510/44000 | Loss: 2.9788 | LR: 3.37e-04 | 1401.7ms/step | 14.3 samples/s
+Step 17520/44000 | Loss: 2.9929 | LR: 3.36e-04 | 1404.1ms/step | 14.2 samples/s
+Step 17530/44000 | Loss: 2.9834 | LR: 3.36e-04 | 1404.5ms/step | 14.2 samples/s
+Step 17540/44000 | Loss: 2.9702 | LR: 3.36e-04 | 1402.0ms/step | 14.3 samples/s
+Step 17550/44000 | Loss: 2.9899 | LR: 3.36e-04 | 1402.8ms/step | 14.3 samples/s
+Step 17560/44000 | Loss: 2.9560 | LR: 3.36e-04 | 1402.6ms/step | 14.3 samples/s
+Step 17570/44000 | Loss: 2.9599 | LR: 3.35e-04 | 1404.3ms/step | 14.2 samples/s
+Step 17580/44000 | Loss: 2.9956 | LR: 3.35e-04 | 1402.1ms/step | 14.3 samples/s
+Step 17590/44000 | Loss: 2.9956 | LR: 3.35e-04 | 1399.8ms/step | 14.3 samples/s
+Step 17600/44000 | Loss: 3.0255 | LR: 3.35e-04 | 1402.0ms/step | 14.3 samples/s
+Step 17610/44000 | Loss: 2.9929 | LR: 3.35e-04 | 1403.8ms/step | 14.2 samples/s
+Step 17620/44000 | Loss: 2.9550 | LR: 3.35e-04 | 1400.6ms/step | 14.3 samples/s
+Step 17630/44000 | Loss: 3.0227 | LR: 3.34e-04 | 1403.5ms/step | 14.3 samples/s
+Step 17640/44000 | Loss: 2.9923 | LR: 3.34e-04 | 1402.5ms/step | 14.3 samples/s
+Step 17650/44000 | Loss: 2.9923 | LR: 3.34e-04 | 1401.3ms/step | 14.3 samples/s
+Step 17660/44000 | Loss: 2.9837 | LR: 3.34e-04 | 1402.2ms/step | 14.3 samples/s
+Step 17670/44000 | Loss: 2.9928 | LR: 3.34e-04 | 1402.8ms/step | 14.3 samples/s
+Step 17680/44000 | Loss: 2.9856 | LR: 3.34e-04 | 1401.3ms/step | 14.3 samples/s
+Step 17690/44000 | Loss: 2.9979 | LR: 3.33e-04 | 1400.2ms/step | 14.3 samples/s
+Step 17700/44000 | Loss: 2.9534 | LR: 3.33e-04 | 1401.7ms/step | 14.3 samples/s
+Step 17710/44000 | Loss: 2.9818 | LR: 3.33e-04 | 1402.3ms/step | 14.3 samples/s
+Step 17720/44000 | Loss: 2.9715 | LR: 3.33e-04 | 1399.6ms/step | 14.3 samples/s
+Step 17730/44000 | Loss: 2.9720 | LR: 3.33e-04 | 1400.6ms/step | 14.3 samples/s
+Step 17740/44000 | Loss: 2.9461 | LR: 3.33e-04 | 1401.3ms/step | 14.3 samples/s
+Step 17750/44000 | Loss: 3.0006 | LR: 3.32e-04 | 1400.1ms/step | 14.3 samples/s
+Step 17760/44000 | Loss: 2.9835 | LR: 3.32e-04 | 1399.8ms/step | 14.3 samples/s
+Step 17770/44000 | Loss: 2.9829 | LR: 3.32e-04 | 1402.0ms/step | 14.3 samples/s
+Step 17780/44000 | Loss: 2.9943 | LR: 3.32e-04 | 1402.2ms/step | 14.3 samples/s
+Step 17790/44000 | Loss: 3.0138 | LR: 3.32e-04 | 1400.1ms/step | 14.3 samples/s
+Step 17800/44000 | Loss: 2.9886 | LR: 3.32e-04 | 1401.8ms/step | 14.3 samples/s
+Step 17810/44000 | Loss: 2.9793 | LR: 3.31e-04 | 1400.0ms/step | 14.3 samples/s
+Step 17820/44000 | Loss: 2.9885 | LR: 3.31e-04 | 1402.2ms/step | 14.3 samples/s
+Step 17830/44000 | Loss: 2.9953 | LR: 3.31e-04 | 1402.1ms/step | 14.3 samples/s
+Step 17840/44000 | Loss: 3.0073 | LR: 3.31e-04 | 1401.9ms/step | 14.3 samples/s
+Step 17850/44000 | Loss: 3.0087 | LR: 3.31e-04 | 1402.1ms/step | 14.3 samples/s
+Step 17860/44000 | Loss: 3.0400 | LR: 3.31e-04 | 1402.3ms/step | 14.3 samples/s
+Step 17870/44000 | Loss: 2.9774 | LR: 3.30e-04 | 1401.5ms/step | 14.3 samples/s
+Step 17880/44000 | Loss: 2.9930 | LR: 3.30e-04 | 1401.9ms/step | 14.3 samples/s
+Step 17890/44000 | Loss: 2.9595 | LR: 3.30e-04 | 1401.9ms/step | 14.3 samples/s
+Step 17900/44000 | Loss: 2.9887 | LR: 3.30e-04 | 1402.2ms/step | 14.3 samples/s
+Step 17910/44000 | Loss: 2.9930 | LR: 3.30e-04 | 1404.0ms/step | 14.2 samples/s
+Step 17920/44000 | Loss: 2.9938 | LR: 3.29e-04 | 1404.1ms/step | 14.2 samples/s
+Step 17930/44000 | Loss: 2.9668 | LR: 3.29e-04 | 1402.2ms/step | 14.3 samples/s
+Step 17940/44000 | Loss: 3.0049 | LR: 3.29e-04 | 1401.7ms/step | 14.3 samples/s
+Step 17950/44000 | Loss: 2.9615 | LR: 3.29e-04 | 1400.1ms/step | 14.3 samples/s
+Step 17960/44000 | Loss: 2.9870 | LR: 3.29e-04 | 1401.7ms/step | 14.3 samples/s
+Step 17970/44000 | Loss: 2.9579 | LR: 3.29e-04 | 1402.6ms/step | 14.3 samples/s
+Step 17980/44000 | Loss: 3.0065 | LR: 3.28e-04 | 1405.4ms/step | 14.2 samples/s
+Step 17990/44000 | Loss: 2.9547 | LR: 3.28e-04 | 1409.2ms/step | 14.2 samples/s
+Step 18000/44000 | Loss: 3.0120 | LR: 3.28e-04 | 1408.9ms/step | 14.2 samples/s
+Validation at step 18000: Loss = 2.9788, Perplexity = 19.66
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-18000
+Step 18010/44000 | Loss: 3.0195 | LR: 3.28e-04 | 4022.0ms/step | 5.0 samples/s
+Step 18020/44000 | Loss: 2.9689 | LR: 3.28e-04 | 1405.7ms/step | 14.2 samples/s
+Step 18030/44000 | Loss: 2.9515 | LR: 3.28e-04 | 1410.2ms/step | 14.2 samples/s
+Step 18040/44000 | Loss: 2.9922 | LR: 3.27e-04 | 1410.7ms/step | 14.2 samples/s
+Step 18050/44000 | Loss: 2.9694 | LR: 3.27e-04 | 1409.9ms/step | 14.2 samples/s
+Step 18060/44000 | Loss: 2.9700 | LR: 3.27e-04 | 1412.4ms/step | 14.2 samples/s
+Step 18070/44000 | Loss: 3.0065 | LR: 3.27e-04 | 1412.9ms/step | 14.2 samples/s
+Step 18080/44000 | Loss: 3.0038 | LR: 3.27e-04 | 1410.6ms/step | 14.2 samples/s
+Step 18090/44000 | Loss: 2.9937 | LR: 3.27e-04 | 1411.3ms/step | 14.2 samples/s
+Step 18100/44000 | Loss: 2.9676 | LR: 3.26e-04 | 1409.3ms/step | 14.2 samples/s
+Step 18110/44000 | Loss: 2.9604 | LR: 3.26e-04 | 1410.5ms/step | 14.2 samples/s
+Step 18120/44000 | Loss: 2.9780 | LR: 3.26e-04 | 1410.3ms/step | 14.2 samples/s
+Step 18130/44000 | Loss: 2.9988 | LR: 3.26e-04 | 1410.0ms/step | 14.2 samples/s
+Step 18140/44000 | Loss: 2.9800 | LR: 3.26e-04 | 1409.8ms/step | 14.2 samples/s
+Step 18150/44000 | Loss: 3.0062 | LR: 3.26e-04 | 1411.2ms/step | 14.2 samples/s
+Step 18160/44000 | Loss: 2.9385 | LR: 3.25e-04 | 1409.3ms/step | 14.2 samples/s
+Step 18170/44000 | Loss: 2.9684 | LR: 3.25e-04 | 1409.8ms/step | 14.2 samples/s
+Step 18180/44000 | Loss: 3.0144 | LR: 3.25e-04 | 1411.7ms/step | 14.2 samples/s
+Step 18190/44000 | Loss: 2.9787 | LR: 3.25e-04 | 1408.5ms/step | 14.2 samples/s
+Step 18200/44000 | Loss: 3.0038 | LR: 3.25e-04 | 1410.3ms/step | 14.2 samples/s
+Step 18210/44000 | Loss: 2.9708 | LR: 3.24e-04 | 1409.8ms/step | 14.2 samples/s
+Step 18220/44000 | Loss: 2.9810 | LR: 3.24e-04 | 1411.5ms/step | 14.2 samples/s
+Step 18230/44000 | Loss: 2.9815 | LR: 3.24e-04 | 1408.3ms/step | 14.2 samples/s
+Step 18240/44000 | Loss: 3.0291 | LR: 3.24e-04 | 1409.5ms/step | 14.2 samples/s
+Step 18250/44000 | Loss: 2.9642 | LR: 3.24e-04 | 1407.2ms/step | 14.2 samples/s
+Step 18260/44000 | Loss: 3.0108 | LR: 3.24e-04 | 1408.9ms/step | 14.2 samples/s
+Step 18270/44000 | Loss: 2.9636 | LR: 3.23e-04 | 1410.6ms/step | 14.2 samples/s
+Step 18280/44000 | Loss: 2.9813 | LR: 3.23e-04 | 1408.1ms/step | 14.2 samples/s
+Step 18290/44000 | Loss: 3.0170 | LR: 3.23e-04 | 1410.9ms/step | 14.2 samples/s
+Step 18300/44000 | Loss: 3.0146 | LR: 3.23e-04 | 1410.5ms/step | 14.2 samples/s
+Step 18310/44000 | Loss: 2.9883 | LR: 3.23e-04 | 1409.9ms/step | 14.2 samples/s
+Step 18320/44000 | Loss: 2.9947 | LR: 3.23e-04 | 1408.7ms/step | 14.2 samples/s
+Step 18330/44000 | Loss: 2.9648 | LR: 3.22e-04 | 1411.8ms/step | 14.2 samples/s
+Step 18340/44000 | Loss: 3.0022 | LR: 3.22e-04 | 1413.6ms/step | 14.1 samples/s
+Step 18350/44000 | Loss: 2.9705 | LR: 3.22e-04 | 1412.3ms/step | 14.2 samples/s
+Step 18360/44000 | Loss: 3.0002 | LR: 3.22e-04 | 1409.5ms/step | 14.2 samples/s
+Step 18370/44000 | Loss: 2.9743 | LR: 3.22e-04 | 1410.3ms/step | 14.2 samples/s
+Step 18380/44000 | Loss: 3.0053 | LR: 3.22e-04 | 1407.6ms/step | 14.2 samples/s
+Step 18390/44000 | Loss: 2.9929 | LR: 3.21e-04 | 1410.0ms/step | 14.2 samples/s
+Step 18400/44000 | Loss: 2.9745 | LR: 3.21e-04 | 1408.1ms/step | 14.2 samples/s
+Step 18410/44000 | Loss: 2.9821 | LR: 3.21e-04 | 1409.8ms/step | 14.2 samples/s
+Step 18420/44000 | Loss: 2.9775 | LR: 3.21e-04 | 1406.0ms/step | 14.2 samples/s
+Step 18430/44000 | Loss: 2.9899 | LR: 3.21e-04 | 1410.0ms/step | 14.2 samples/s
+Step 18440/44000 | Loss: 2.9887 | LR: 3.20e-04 | 1408.6ms/step | 14.2 samples/s
+Step 18450/44000 | Loss: 2.9978 | LR: 3.20e-04 | 1410.2ms/step | 14.2 samples/s

epfl_final/train_logs/ropectl_a1p0_h100_resume/seed44.20260820T041452Z.log

@@ -0,0 +1,666 @@
+Auto-resuming from latest checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858/checkpoint-13000
+Resuming: Using existing output directory /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858
+Running config:
+{
+  "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
+  "model_dtype": "bfloat16",
+  "init_from_scratch": true,
+  "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
+  "val_path": null,
+  "max_seq_length": 8192,
+  "num_workers": 8,
+  "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858",
+  "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858/checkpoint-13000",
+  "force_new_run": false,
+  "num_epochs": 1,
+  "max_steps": 44000,
+  "per_device_batch_size": 2,
+  "gradient_accumulation_steps": 10,
+  "learning_rate": 0.0005,
+  "weight_decay": 0.01,
+  "warmup_steps": 750,
+  "lr_scheduler": "cosine",
+  "max_grad_norm": 1.0,
+  "save_steps": 1000,
+  "eval_steps": 1000,
+  "eval_batches": 50,
+  "logging_steps": 10,
+  "seed": 44,
+  "verbose": true,
+  "checkpoint_activations": false,
+  "strategy": "ddp",
+  "mixed_precision": "bf16",
+  "wandb_enabled": true,
+  "wandb_project": "stochastic_sparsity",
+  "wandb_name": "l8k365m_dense_ropectl_a1p0_seed44",
+  "wandb_tags": [
+    "qwen3",
+    "365m",
+    "8k",
+    "dense",
+    "rope-control"
+  ],
+  "hidden_size": 768,
+  "intermediate_size": 3072,
+  "num_attention_heads": 6,
+  "num_key_value_heads": 2,
+  "num_hidden_layers": 28,
+  "head_dim": null,
+  "attention_dropout": 0.0,
+  "rope_train_position_scale_set": [
+    1.0,
+    0.9896339488176222,
+    0.9692159692159692,
+    0.9420289855072463,
+    0.8730158730158729
+  ],
+  "attention_type": "sparse_stochastic",
+  "sparse_block_size": 128,
+  "sparse_num_random_blocks": 31,
+  "sparse_mask_pool_size": 1,
+  "sparse_val_dense_compare": false,
+  "sparse_layer_indices": null,
+  "sparse_mask_cache_dir": null,
+  "sparse_mask_seed": 0,
+  "sparse_mixed_nrb_set": null,
+  "dense_mode": true,
+  "dense_step_intervals": null
+}
+Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
+Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858/checkpoint-13000
+Checkpoint activations: False
+dense_mode=True: sparse attention layers will use standard causal SDPA.
+
+
+================================================================================
+MODEL ARCHITECTURE
+================================================================================
+Model config:
+  vocab_size: 151936
+  hidden_size: 768
+  num_hidden_layers: 28
+  num_attention_heads: 6
+  num_key_value_heads: 2
+  head_dim: 128
+  intermediate_size: 3072
+  max_position_embeddings: 32768
+  rope_theta: 1000000
+  rope_train_position_scale_set: [1.0, 0.9896339488176222, 0.9692159692159692, 0.9420289855072463, 0.8730158730158729]
+  attention_type: sparse_stochastic
+  sparse_block_size: 128
+  sparse_num_random_blocks: 31
+  sparse_mask_pool_size: 1
+  sparse_max_seq_length: 8192
+
+Parameter count:
+  Total parameters: 358,958,848
+  Trainable parameters: 358,958,848
+  Non-trainable parameters: 0
+  Model size (BF16): 0.72 GB
+
+--------------------------------------------------------------------------------
+LAYER-BY-LAYER SUMMARY
+--------------------------------------------------------------------------------
+================================================================================================================================================================
+Layer (type:depth-idx)                                       Input Shape               Output Shape              Param #                   Trainable
+================================================================================================================================================================
+Qwen3ForCausalLM                                             [2, 8192]                 --                        --                        True
+├─Qwen3Model: 1-1                                            --                        [2, 8192, 768]            --                        True
+│    └─Embedding: 2-1                                        [2, 8192]                 [2, 8192, 768]            116,686,848               True
+│    └─ModuleList: 2-2                                       --                        --                        --                        True
+│    │    └─Qwen3DecoderLayer: 3-1                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-2                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-3                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-4                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-5                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-6                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-7                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-8                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-9                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-10                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-11                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-12                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-13                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-14                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-15                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-16                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-17                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-18                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-19                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-20                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-21                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-22                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-23                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-24                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-25                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-26                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-27                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-28                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    └─Qwen3RMSNorm: 2-3                                     [2, 8192, 768]            [2, 8192, 768]            768                       True
+├─Linear: 1-2                                                [2, 8192, 768]            [2, 8192, 151936]         116,686,848               True
+================================================================================================================================================================
+Total params: 475,645,696
+Trainable params: 475,645,696
+Non-trainable params: 0
+Total mult-adds (Units.MEGABYTES): 951.29
+================================================================================================================================================================
+Input size (MB): 0.13
+Forward/backward pass size (MB): 31197.23
+Params size (MB): 951.29
+Estimated Total Size (MB): 32148.66
+================================================================================================================================================================
+  NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
+================================================================================
+
+Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
+Using pre-tokenized dataset (no runtime tokenization)
+Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
+Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
+Dataset split: 3027764 train, 159356 validation
+Training examples (packed, train split):  3027764
+Validation examples (packed, val split):  159356
+Per-device batch size:                    2
+World size (GPUs):                        1
+Gradient accumulation steps:              10
+Effective batch size (seqs/step):         20
+Sequence length:                          8192
+Tokens per optimizer step:                163,840
+Micro-batches per epoch (per rank):       1513882
+Optimizer steps per epoch:                151388
+Total optimizer steps:                    44000  (capped by max_steps; 0.29 epochs)
+Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858/checkpoint-13000...
+Preserving checkpoint LR state for exact resume: current=[0.0004073888929652048], schedule_base=[0.0005]
+Resumed from checkpoint at step 13000 (epoch 0, micro_step 130000)
+Resuming training from step 13000
+Resuming wandb run bajm6b15
+wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from WANDB_API_KEY.
+wandb: Currently logged in as: ruben-navasardyan4869 (ruben-navasardyan4869-epfl) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
+wandb: setting up run bajm6b15
+wandb: Tracking run with wandb version 0.26.1
+wandb: Run data is saved locally in /mloscratch/homes/navasard/sparse/SparseAtt/wandb/run-20260820_041558-bajm6b15
+wandb: Run `wandb offline` to turn off syncing.
+wandb: Resuming run l8k365m_dense_ropectl_a1p0_seed44
+wandb: ⭐️ View project at https://wandb.ai/ruben-navasardyan4869-epfl/stochastic_sparsity
+wandb: 🚀 View run at https://wandb.ai/ruben-navasardyan4869-epfl/stochastic_sparsity/runs/bajm6b15
+Starting training...
+Validation at step 13000: Loss = 3.0696, Perplexity = 21.53
+wandb: WARNING Tried to log to step 13000 that is less than the current step 13171. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Switching attention mode for step 13001: dense (dense step interval)
+Step 13010/44000 | Loss: 3.0583 | LR: 4.07e-04 | 30145.6ms/step | 0.7 samples/s
+wandb: WARNING Tried to log to step 13010 that is less than the current step 13171. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13020/44000 | Loss: 3.0539 | LR: 4.07e-04 | 1368.0ms/step | 14.6 samples/s
+wandb: WARNING Tried to log to step 13020 that is less than the current step 13171. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13030/44000 | Loss: 3.0645 | LR: 4.07e-04 | 1371.0ms/step | 14.6 samples/s
+wandb: WARNING Tried to log to step 13030 that is less than the current step 13171. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13040/44000 | Loss: 3.0438 | LR: 4.07e-04 | 1375.5ms/step | 14.5 samples/s
+wandb: WARNING Tried to log to step 13040 that is less than the current step 13171. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13050/44000 | Loss: 3.0602 | LR: 4.07e-04 | 1378.6ms/step | 14.5 samples/s
+wandb: WARNING Tried to log to step 13050 that is less than the current step 13171. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13060/44000 | Loss: 3.0601 | LR: 4.07e-04 | 1376.2ms/step | 14.5 samples/s
+wandb: WARNING Tried to log to step 13060 that is less than the current step 13171. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13070/44000 | Loss: 3.0745 | LR: 4.06e-04 | 1380.3ms/step | 14.5 samples/s
+wandb: WARNING Tried to log to step 13070 that is less than the current step 13171. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13080/44000 | Loss: 3.0356 | LR: 4.06e-04 | 1379.5ms/step | 14.5 samples/s
+wandb: WARNING Tried to log to step 13080 that is less than the current step 13171. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13090/44000 | Loss: 3.0677 | LR: 4.06e-04 | 1377.7ms/step | 14.5 samples/s
+wandb: WARNING Tried to log to step 13090 that is less than the current step 13171. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13100/44000 | Loss: 3.0657 | LR: 4.06e-04 | 1379.9ms/step | 14.5 samples/s
+wandb: WARNING Tried to log to step 13100 that is less than the current step 13171. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13110/44000 | Loss: 3.0377 | LR: 4.06e-04 | 1382.3ms/step | 14.5 samples/s
+wandb: WARNING Tried to log to step 13110 that is less than the current step 13171. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13120/44000 | Loss: 3.0419 | LR: 4.06e-04 | 1377.8ms/step | 14.5 samples/s
+wandb: WARNING Tried to log to step 13120 that is less than the current step 13171. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13130/44000 | Loss: 3.0623 | LR: 4.06e-04 | 1374.8ms/step | 14.5 samples/s
+wandb: WARNING Tried to log to step 13130 that is less than the current step 13171. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13140/44000 | Loss: 3.0456 | LR: 4.05e-04 | 1373.6ms/step | 14.6 samples/s
+wandb: WARNING Tried to log to step 13140 that is less than the current step 13171. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13150/44000 | Loss: 3.0531 | LR: 4.05e-04 | 1375.9ms/step | 14.5 samples/s
+wandb: WARNING Tried to log to step 13150 that is less than the current step 13171. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13160/44000 | Loss: 3.0258 | LR: 4.05e-04 | 1375.7ms/step | 14.5 samples/s
+wandb: WARNING Tried to log to step 13160 that is less than the current step 13171. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13170/44000 | Loss: 3.0970 | LR: 4.05e-04 | 1373.9ms/step | 14.6 samples/s
+wandb: WARNING Tried to log to step 13170 that is less than the current step 13171. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13180/44000 | Loss: 3.0733 | LR: 4.05e-04 | 1372.2ms/step | 14.6 samples/s
+Step 13190/44000 | Loss: 3.0473 | LR: 4.05e-04 | 1373.9ms/step | 14.6 samples/s
+Step 13200/44000 | Loss: 3.0934 | LR: 4.05e-04 | 1375.0ms/step | 14.5 samples/s
+Step 13210/44000 | Loss: 3.0429 | LR: 4.04e-04 | 1372.5ms/step | 14.6 samples/s
+Step 13220/44000 | Loss: 3.0658 | LR: 4.04e-04 | 1372.0ms/step | 14.6 samples/s
+Step 13230/44000 | Loss: 3.0785 | LR: 4.04e-04 | 1373.3ms/step | 14.6 samples/s
+Step 13240/44000 | Loss: 3.0605 | LR: 4.04e-04 | 1373.4ms/step | 14.6 samples/s
+Step 13250/44000 | Loss: 3.0669 | LR: 4.04e-04 | 1373.5ms/step | 14.6 samples/s
+Step 13260/44000 | Loss: 3.0628 | LR: 4.04e-04 | 1372.4ms/step | 14.6 samples/s
+Step 13270/44000 | Loss: 3.0632 | LR: 4.04e-04 | 1373.9ms/step | 14.6 samples/s
+Step 13280/44000 | Loss: 3.0612 | LR: 4.03e-04 | 1376.4ms/step | 14.5 samples/s
+Step 13290/44000 | Loss: 3.0461 | LR: 4.03e-04 | 1373.5ms/step | 14.6 samples/s
+Step 13300/44000 | Loss: 3.0573 | LR: 4.03e-04 | 1374.0ms/step | 14.6 samples/s
+Step 13310/44000 | Loss: 3.0515 | LR: 4.03e-04 | 1373.8ms/step | 14.6 samples/s
+Step 13320/44000 | Loss: 3.0342 | LR: 4.03e-04 | 1374.6ms/step | 14.5 samples/s
+Step 13330/44000 | Loss: 3.0614 | LR: 4.03e-04 | 1372.6ms/step | 14.6 samples/s
+Step 13340/44000 | Loss: 3.0749 | LR: 4.03e-04 | 1372.3ms/step | 14.6 samples/s
+Step 13350/44000 | Loss: 3.0320 | LR: 4.02e-04 | 1372.7ms/step | 14.6 samples/s
+Step 13360/44000 | Loss: 3.1017 | LR: 4.02e-04 | 1374.6ms/step | 14.6 samples/s
+Step 13370/44000 | Loss: 3.0585 | LR: 4.02e-04 | 1373.5ms/step | 14.6 samples/s
+Step 13380/44000 | Loss: 3.0661 | LR: 4.02e-04 | 1373.5ms/step | 14.6 samples/s
+Step 13390/44000 | Loss: 3.0359 | LR: 4.02e-04 | 1376.2ms/step | 14.5 samples/s
+Step 13400/44000 | Loss: 3.0451 | LR: 4.02e-04 | 1377.0ms/step | 14.5 samples/s
+Step 13410/44000 | Loss: 3.0756 | LR: 4.02e-04 | 1375.1ms/step | 14.5 samples/s
+Step 13420/44000 | Loss: 3.0443 | LR: 4.01e-04 | 1376.3ms/step | 14.5 samples/s
+Step 13430/44000 | Loss: 3.0581 | LR: 4.01e-04 | 1375.4ms/step | 14.5 samples/s
+Step 13440/44000 | Loss: 3.0353 | LR: 4.01e-04 | 1375.8ms/step | 14.5 samples/s
+Step 13450/44000 | Loss: 3.0295 | LR: 4.01e-04 | 1372.9ms/step | 14.6 samples/s
+Step 13460/44000 | Loss: 3.0325 | LR: 4.01e-04 | 1373.6ms/step | 14.6 samples/s
+Step 13470/44000 | Loss: 3.0720 | LR: 4.01e-04 | 1374.7ms/step | 14.5 samples/s
+Step 13480/44000 | Loss: 3.0366 | LR: 4.01e-04 | 1375.3ms/step | 14.5 samples/s
+Step 13490/44000 | Loss: 3.0641 | LR: 4.00e-04 | 1374.6ms/step | 14.6 samples/s
+Step 13500/44000 | Loss: 3.0567 | LR: 4.00e-04 | 1375.5ms/step | 14.5 samples/s
+Step 13510/44000 | Loss: 3.0549 | LR: 4.00e-04 | 1374.9ms/step | 14.5 samples/s
+Step 13520/44000 | Loss: 3.0648 | LR: 4.00e-04 | 1376.7ms/step | 14.5 samples/s
+Step 13530/44000 | Loss: 3.0518 | LR: 4.00e-04 | 1374.2ms/step | 14.6 samples/s
+Step 13540/44000 | Loss: 3.0569 | LR: 4.00e-04 | 1374.2ms/step | 14.6 samples/s
+Step 13550/44000 | Loss: 3.0538 | LR: 4.00e-04 | 1374.4ms/step | 14.6 samples/s
+Step 13560/44000 | Loss: 3.0502 | LR: 3.99e-04 | 1374.0ms/step | 14.6 samples/s
+Step 13570/44000 | Loss: 3.0712 | LR: 3.99e-04 | 1373.0ms/step | 14.6 samples/s
+Step 13580/44000 | Loss: 3.0557 | LR: 3.99e-04 | 1372.4ms/step | 14.6 samples/s
+Step 13590/44000 | Loss: 3.0451 | LR: 3.99e-04 | 1376.1ms/step | 14.5 samples/s
+Step 13600/44000 | Loss: 3.0511 | LR: 3.99e-04 | 1374.8ms/step | 14.5 samples/s
+Step 13610/44000 | Loss: 3.0764 | LR: 3.99e-04 | 1373.4ms/step | 14.6 samples/s
+Step 13620/44000 | Loss: 3.0381 | LR: 3.98e-04 | 1373.0ms/step | 14.6 samples/s
+Step 13630/44000 | Loss: 3.0307 | LR: 3.98e-04 | 1374.2ms/step | 14.6 samples/s
+Step 13640/44000 | Loss: 3.0425 | LR: 3.98e-04 | 1376.8ms/step | 14.5 samples/s
+Step 13650/44000 | Loss: 3.0684 | LR: 3.98e-04 | 1372.6ms/step | 14.6 samples/s
+Step 13660/44000 | Loss: 3.0299 | LR: 3.98e-04 | 1371.7ms/step | 14.6 samples/s
+Step 13670/44000 | Loss: 3.0322 | LR: 3.98e-04 | 1374.8ms/step | 14.5 samples/s
+Step 13680/44000 | Loss: 3.0632 | LR: 3.98e-04 | 1375.3ms/step | 14.5 samples/s
+Step 13690/44000 | Loss: 3.0209 | LR: 3.97e-04 | 1372.8ms/step | 14.6 samples/s
+Step 13700/44000 | Loss: 3.0666 | LR: 3.97e-04 | 1372.7ms/step | 14.6 samples/s
+Step 13710/44000 | Loss: 3.0531 | LR: 3.97e-04 | 1375.3ms/step | 14.5 samples/s
+Step 13720/44000 | Loss: 3.0627 | LR: 3.97e-04 | 1374.7ms/step | 14.5 samples/s
+Step 13730/44000 | Loss: 3.0650 | LR: 3.97e-04 | 1373.7ms/step | 14.6 samples/s
+Step 13740/44000 | Loss: 3.0528 | LR: 3.97e-04 | 1372.0ms/step | 14.6 samples/s
+Step 13750/44000 | Loss: 3.0296 | LR: 3.97e-04 | 1377.0ms/step | 14.5 samples/s
+Step 13760/44000 | Loss: 3.0609 | LR: 3.96e-04 | 1377.0ms/step | 14.5 samples/s
+Step 13770/44000 | Loss: 3.0541 | LR: 3.96e-04 | 1373.5ms/step | 14.6 samples/s
+Step 13780/44000 | Loss: 3.0327 | LR: 3.96e-04 | 1372.9ms/step | 14.6 samples/s
+Step 13790/44000 | Loss: 3.0562 | LR: 3.96e-04 | 1372.0ms/step | 14.6 samples/s
+Step 13800/44000 | Loss: 3.0464 | LR: 3.96e-04 | 1373.8ms/step | 14.6 samples/s
+Step 13810/44000 | Loss: 3.0374 | LR: 3.96e-04 | 1373.6ms/step | 14.6 samples/s
+Step 13820/44000 | Loss: 3.0165 | LR: 3.96e-04 | 1372.0ms/step | 14.6 samples/s
+Step 13830/44000 | Loss: 3.0255 | LR: 3.95e-04 | 1373.2ms/step | 14.6 samples/s
+Step 13840/44000 | Loss: 3.0070 | LR: 3.95e-04 | 1374.4ms/step | 14.6 samples/s
+Step 13850/44000 | Loss: 3.0490 | LR: 3.95e-04 | 1371.8ms/step | 14.6 samples/s
+Step 13860/44000 | Loss: 3.0623 | LR: 3.95e-04 | 1372.3ms/step | 14.6 samples/s
+Step 13870/44000 | Loss: 3.0529 | LR: 3.95e-04 | 1375.4ms/step | 14.5 samples/s
+Step 13880/44000 | Loss: 3.0477 | LR: 3.95e-04 | 1370.7ms/step | 14.6 samples/s
+Step 13890/44000 | Loss: 3.0355 | LR: 3.95e-04 | 1371.6ms/step | 14.6 samples/s
+Step 13900/44000 | Loss: 3.0340 | LR: 3.94e-04 | 1371.9ms/step | 14.6 samples/s
+Step 13910/44000 | Loss: 3.0398 | LR: 3.94e-04 | 1375.2ms/step | 14.5 samples/s
+Step 13920/44000 | Loss: 3.0894 | LR: 3.94e-04 | 1373.9ms/step | 14.6 samples/s
+Step 13930/44000 | Loss: 3.0537 | LR: 3.94e-04 | 1373.3ms/step | 14.6 samples/s
+Step 13940/44000 | Loss: 3.0667 | LR: 3.94e-04 | 1374.1ms/step | 14.6 samples/s
+Step 13950/44000 | Loss: 3.0609 | LR: 3.94e-04 | 1374.8ms/step | 14.5 samples/s
+Step 13960/44000 | Loss: 3.0364 | LR: 3.93e-04 | 1374.3ms/step | 14.6 samples/s
+Step 13970/44000 | Loss: 3.0370 | LR: 3.93e-04 | 1373.4ms/step | 14.6 samples/s
+Step 13980/44000 | Loss: 3.0801 | LR: 3.93e-04 | 1375.5ms/step | 14.5 samples/s
+Step 13990/44000 | Loss: 3.0125 | LR: 3.93e-04 | 1377.0ms/step | 14.5 samples/s
+Step 14000/44000 | Loss: 3.0749 | LR: 3.93e-04 | 1375.5ms/step | 14.5 samples/s
+Validation at step 14000: Loss = 3.0537, Perplexity = 21.19
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858/checkpoint-14000
+Step 14010/44000 | Loss: 3.0348 | LR: 3.93e-04 | 3894.1ms/step | 5.1 samples/s
+Step 14020/44000 | Loss: 3.0500 | LR: 3.93e-04 | 1385.9ms/step | 14.4 samples/s
+Step 14030/44000 | Loss: 3.0532 | LR: 3.92e-04 | 1377.6ms/step | 14.5 samples/s
+Step 14040/44000 | Loss: 3.0203 | LR: 3.92e-04 | 1378.8ms/step | 14.5 samples/s
+Step 14050/44000 | Loss: 3.0323 | LR: 3.92e-04 | 1381.5ms/step | 14.5 samples/s
+Step 14060/44000 | Loss: 3.0529 | LR: 3.92e-04 | 1381.6ms/step | 14.5 samples/s
+Step 14070/44000 | Loss: 3.0459 | LR: 3.92e-04 | 1384.4ms/step | 14.4 samples/s
+Step 14080/44000 | Loss: 3.0383 | LR: 3.92e-04 | 1385.9ms/step | 14.4 samples/s
+Step 14090/44000 | Loss: 3.0212 | LR: 3.92e-04 | 1382.8ms/step | 14.5 samples/s
+Step 14100/44000 | Loss: 3.0382 | LR: 3.91e-04 | 1382.0ms/step | 14.5 samples/s
+Step 14110/44000 | Loss: 3.0276 | LR: 3.91e-04 | 1385.4ms/step | 14.4 samples/s
+Step 14120/44000 | Loss: 3.0614 | LR: 3.91e-04 | 1386.5ms/step | 14.4 samples/s
+Step 14130/44000 | Loss: 3.0314 | LR: 3.91e-04 | 1385.7ms/step | 14.4 samples/s
+Step 14140/44000 | Loss: 3.0562 | LR: 3.91e-04 | 1384.5ms/step | 14.4 samples/s
+Step 14150/44000 | Loss: 3.0508 | LR: 3.91e-04 | 1387.3ms/step | 14.4 samples/s
+Step 14160/44000 | Loss: 3.0385 | LR: 3.90e-04 | 1384.0ms/step | 14.5 samples/s
+Step 14170/44000 | Loss: 3.0346 | LR: 3.90e-04 | 1387.1ms/step | 14.4 samples/s
+Step 14180/44000 | Loss: 3.0018 | LR: 3.90e-04 | 1388.9ms/step | 14.4 samples/s
+Step 14190/44000 | Loss: 3.0360 | LR: 3.90e-04 | 1384.1ms/step | 14.4 samples/s
+Step 14200/44000 | Loss: 3.0175 | LR: 3.90e-04 | 1385.5ms/step | 14.4 samples/s
+Step 14210/44000 | Loss: 3.0325 | LR: 3.90e-04 | 1385.8ms/step | 14.4 samples/s
+Step 14220/44000 | Loss: 3.0249 | LR: 3.90e-04 | 1384.6ms/step | 14.4 samples/s
+Step 14230/44000 | Loss: 3.0705 | LR: 3.89e-04 | 1385.9ms/step | 14.4 samples/s
+Step 14240/44000 | Loss: 3.0322 | LR: 3.89e-04 | 1385.6ms/step | 14.4 samples/s
+Step 14250/44000 | Loss: 3.0173 | LR: 3.89e-04 | 1383.4ms/step | 14.5 samples/s
+Step 14260/44000 | Loss: 3.0340 | LR: 3.89e-04 | 1384.7ms/step | 14.4 samples/s
+Step 14270/44000 | Loss: 3.0348 | LR: 3.89e-04 | 1385.8ms/step | 14.4 samples/s
+Step 14280/44000 | Loss: 3.0256 | LR: 3.89e-04 | 1385.2ms/step | 14.4 samples/s
+Step 14290/44000 | Loss: 3.0314 | LR: 3.89e-04 | 1382.8ms/step | 14.5 samples/s
+Step 14300/44000 | Loss: 3.0434 | LR: 3.88e-04 | 1383.6ms/step | 14.5 samples/s
+Step 14310/44000 | Loss: 3.0538 | LR: 3.88e-04 | 1384.8ms/step | 14.4 samples/s
+Step 14320/44000 | Loss: 3.0534 | LR: 3.88e-04 | 1383.7ms/step | 14.5 samples/s
+Step 14330/44000 | Loss: 3.0495 | LR: 3.88e-04 | 1383.4ms/step | 14.5 samples/s
+Step 14340/44000 | Loss: 3.0320 | LR: 3.88e-04 | 1384.7ms/step | 14.4 samples/s
+Step 14350/44000 | Loss: 3.0422 | LR: 3.88e-04 | 1382.5ms/step | 14.5 samples/s
+Step 14360/44000 | Loss: 2.9975 | LR: 3.87e-04 | 1383.7ms/step | 14.5 samples/s
+Step 14370/44000 | Loss: 3.0624 | LR: 3.87e-04 | 1385.1ms/step | 14.4 samples/s
+Step 14380/44000 | Loss: 3.0577 | LR: 3.87e-04 | 1386.6ms/step | 14.4 samples/s
+Step 14390/44000 | Loss: 3.0378 | LR: 3.87e-04 | 1388.2ms/step | 14.4 samples/s
+Step 14400/44000 | Loss: 3.0463 | LR: 3.87e-04 | 1389.3ms/step | 14.4 samples/s
+Step 14410/44000 | Loss: 3.0376 | LR: 3.87e-04 | 1381.8ms/step | 14.5 samples/s
+Step 14420/44000 | Loss: 3.0484 | LR: 3.87e-04 | 1384.2ms/step | 14.4 samples/s
+Step 14430/44000 | Loss: 3.0090 | LR: 3.86e-04 | 1384.3ms/step | 14.4 samples/s
+Step 14440/44000 | Loss: 3.0306 | LR: 3.86e-04 | 1387.3ms/step | 14.4 samples/s
+Step 14450/44000 | Loss: 3.0626 | LR: 3.86e-04 | 1384.5ms/step | 14.4 samples/s
+Step 14460/44000 | Loss: 3.0276 | LR: 3.86e-04 | 1383.1ms/step | 14.5 samples/s
+Step 14470/44000 | Loss: 3.0396 | LR: 3.86e-04 | 1385.3ms/step | 14.4 samples/s
+Step 14480/44000 | Loss: 3.0202 | LR: 3.86e-04 | 1385.1ms/step | 14.4 samples/s
+Step 14490/44000 | Loss: 3.0549 | LR: 3.85e-04 | 1385.3ms/step | 14.4 samples/s
+Step 14500/44000 | Loss: 3.0393 | LR: 3.85e-04 | 1390.2ms/step | 14.4 samples/s
+Step 14510/44000 | Loss: 3.0490 | LR: 3.85e-04 | 1388.0ms/step | 14.4 samples/s
+Step 14520/44000 | Loss: 3.0261 | LR: 3.85e-04 | 1386.4ms/step | 14.4 samples/s
+Step 14530/44000 | Loss: 3.0523 | LR: 3.85e-04 | 1384.2ms/step | 14.4 samples/s
+Step 14540/44000 | Loss: 3.0644 | LR: 3.85e-04 | 1386.2ms/step | 14.4 samples/s
+Step 14550/44000 | Loss: 3.0110 | LR: 3.85e-04 | 1381.8ms/step | 14.5 samples/s
+Step 14560/44000 | Loss: 3.0658 | LR: 3.84e-04 | 1386.2ms/step | 14.4 samples/s
+Step 14570/44000 | Loss: 3.0399 | LR: 3.84e-04 | 1383.6ms/step | 14.5 samples/s
+Step 14580/44000 | Loss: 3.0060 | LR: 3.84e-04 | 1382.1ms/step | 14.5 samples/s
+Step 14590/44000 | Loss: 3.0186 | LR: 3.84e-04 | 1384.6ms/step | 14.4 samples/s
+Step 14600/44000 | Loss: 3.0539 | LR: 3.84e-04 | 1383.9ms/step | 14.5 samples/s
+Step 14610/44000 | Loss: 3.0583 | LR: 3.84e-04 | 1382.9ms/step | 14.5 samples/s
+Step 14620/44000 | Loss: 3.0253 | LR: 3.83e-04 | 1385.6ms/step | 14.4 samples/s
+Step 14630/44000 | Loss: 3.0322 | LR: 3.83e-04 | 1383.8ms/step | 14.5 samples/s
+Step 14640/44000 | Loss: 3.0508 | LR: 3.83e-04 | 1384.9ms/step | 14.4 samples/s
+Step 14650/44000 | Loss: 3.0502 | LR: 3.83e-04 | 1388.3ms/step | 14.4 samples/s
+Step 14660/44000 | Loss: 3.0512 | LR: 3.83e-04 | 1385.5ms/step | 14.4 samples/s
+Step 14670/44000 | Loss: 3.0824 | LR: 3.83e-04 | 1384.4ms/step | 14.4 samples/s
+Step 14680/44000 | Loss: 3.0295 | LR: 3.83e-04 | 1385.3ms/step | 14.4 samples/s
+Step 14690/44000 | Loss: 3.0247 | LR: 3.82e-04 | 1384.1ms/step | 14.4 samples/s
+Step 14700/44000 | Loss: 3.0529 | LR: 3.82e-04 | 1385.0ms/step | 14.4 samples/s
+Step 14710/44000 | Loss: 3.0174 | LR: 3.82e-04 | 1382.8ms/step | 14.5 samples/s
+Step 14720/44000 | Loss: 3.0467 | LR: 3.82e-04 | 1384.0ms/step | 14.5 samples/s
+Step 14730/44000 | Loss: 3.0135 | LR: 3.82e-04 | 1383.8ms/step | 14.5 samples/s
+Step 14740/44000 | Loss: 3.0235 | LR: 3.82e-04 | 1384.4ms/step | 14.4 samples/s
+Step 14750/44000 | Loss: 3.0217 | LR: 3.81e-04 | 1385.0ms/step | 14.4 samples/s
+Step 14760/44000 | Loss: 3.0297 | LR: 3.81e-04 | 1383.8ms/step | 14.5 samples/s
+Step 14770/44000 | Loss: 3.0435 | LR: 3.81e-04 | 1385.4ms/step | 14.4 samples/s
+Step 14780/44000 | Loss: 3.0377 | LR: 3.81e-04 | 1388.8ms/step | 14.4 samples/s
+Step 14790/44000 | Loss: 3.0492 | LR: 3.81e-04 | 1383.7ms/step | 14.5 samples/s
+Step 14800/44000 | Loss: 3.0522 | LR: 3.81e-04 | 1385.4ms/step | 14.4 samples/s
+Step 14810/44000 | Loss: 3.0323 | LR: 3.81e-04 | 1385.8ms/step | 14.4 samples/s
+Step 14820/44000 | Loss: 3.0518 | LR: 3.80e-04 | 1384.0ms/step | 14.5 samples/s
+Step 14830/44000 | Loss: 3.0578 | LR: 3.80e-04 | 1385.3ms/step | 14.4 samples/s
+Step 14840/44000 | Loss: 3.0104 | LR: 3.80e-04 | 1388.0ms/step | 14.4 samples/s
+Step 14850/44000 | Loss: 3.0387 | LR: 3.80e-04 | 1384.8ms/step | 14.4 samples/s
+Step 14860/44000 | Loss: 3.0327 | LR: 3.80e-04 | 1385.9ms/step | 14.4 samples/s
+Step 14870/44000 | Loss: 3.0378 | LR: 3.80e-04 | 1388.7ms/step | 14.4 samples/s
+Step 14880/44000 | Loss: 3.0459 | LR: 3.79e-04 | 1385.7ms/step | 14.4 samples/s
+Step 14890/44000 | Loss: 3.0603 | LR: 3.79e-04 | 1383.2ms/step | 14.5 samples/s
+Step 14900/44000 | Loss: 3.0527 | LR: 3.79e-04 | 1384.5ms/step | 14.4 samples/s
+Step 14910/44000 | Loss: 3.0334 | LR: 3.79e-04 | 1387.1ms/step | 14.4 samples/s
+Step 14920/44000 | Loss: 3.0379 | LR: 3.79e-04 | 1384.0ms/step | 14.5 samples/s
+Step 14930/44000 | Loss: 3.0499 | LR: 3.79e-04 | 1383.6ms/step | 14.5 samples/s
+Step 14940/44000 | Loss: 3.0472 | LR: 3.79e-04 | 1386.1ms/step | 14.4 samples/s
+Step 14950/44000 | Loss: 3.0311 | LR: 3.78e-04 | 1382.3ms/step | 14.5 samples/s
+Step 14960/44000 | Loss: 3.0361 | LR: 3.78e-04 | 1384.8ms/step | 14.4 samples/s
+Step 14970/44000 | Loss: 3.0268 | LR: 3.78e-04 | 1384.9ms/step | 14.4 samples/s
+Step 14980/44000 | Loss: 3.0461 | LR: 3.78e-04 | 1382.4ms/step | 14.5 samples/s
+Step 14990/44000 | Loss: 3.0686 | LR: 3.78e-04 | 1382.4ms/step | 14.5 samples/s
+Step 15000/44000 | Loss: 3.0209 | LR: 3.78e-04 | 1384.0ms/step | 14.5 samples/s
+Validation at step 15000: Loss = 3.0394, Perplexity = 20.89
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858/checkpoint-15000
+Step 15010/44000 | Loss: 3.0339 | LR: 3.77e-04 | 3917.6ms/step | 5.1 samples/s
+Step 15020/44000 | Loss: 3.0625 | LR: 3.77e-04 | 1381.9ms/step | 14.5 samples/s
+Step 15030/44000 | Loss: 3.0331 | LR: 3.77e-04 | 1378.7ms/step | 14.5 samples/s
+Step 15040/44000 | Loss: 3.0055 | LR: 3.77e-04 | 1380.4ms/step | 14.5 samples/s
+Step 15050/44000 | Loss: 3.0099 | LR: 3.77e-04 | 1380.6ms/step | 14.5 samples/s
+Step 15060/44000 | Loss: 3.0344 | LR: 3.77e-04 | 1381.2ms/step | 14.5 samples/s
+Step 15070/44000 | Loss: 3.0197 | LR: 3.77e-04 | 1381.9ms/step | 14.5 samples/s
+Step 15080/44000 | Loss: 3.0658 | LR: 3.76e-04 | 1382.8ms/step | 14.5 samples/s
+Step 15090/44000 | Loss: 3.0302 | LR: 3.76e-04 | 1383.3ms/step | 14.5 samples/s
+Step 15100/44000 | Loss: 3.0547 | LR: 3.76e-04 | 1383.9ms/step | 14.5 samples/s
+Step 15110/44000 | Loss: 3.0684 | LR: 3.76e-04 | 1382.7ms/step | 14.5 samples/s
+Step 15120/44000 | Loss: 3.0258 | LR: 3.76e-04 | 1384.3ms/step | 14.4 samples/s
+Step 15130/44000 | Loss: 3.0248 | LR: 3.76e-04 | 1385.1ms/step | 14.4 samples/s
+Step 15140/44000 | Loss: 3.0363 | LR: 3.75e-04 | 1387.2ms/step | 14.4 samples/s
+Step 15150/44000 | Loss: 3.0476 | LR: 3.75e-04 | 1384.4ms/step | 14.4 samples/s
+Step 15160/44000 | Loss: 3.0110 | LR: 3.75e-04 | 1383.2ms/step | 14.5 samples/s
+Step 15170/44000 | Loss: 3.0182 | LR: 3.75e-04 | 1383.7ms/step | 14.5 samples/s
+Step 15180/44000 | Loss: 3.0260 | LR: 3.75e-04 | 1382.6ms/step | 14.5 samples/s
+Step 15190/44000 | Loss: 3.0572 | LR: 3.75e-04 | 1384.5ms/step | 14.4 samples/s
+Step 15200/44000 | Loss: 3.0297 | LR: 3.74e-04 | 1384.2ms/step | 14.4 samples/s
+Step 15210/44000 | Loss: 3.0048 | LR: 3.74e-04 | 1384.0ms/step | 14.5 samples/s
+Step 15220/44000 | Loss: 3.0268 | LR: 3.74e-04 | 1385.3ms/step | 14.4 samples/s
+Step 15230/44000 | Loss: 3.0351 | LR: 3.74e-04 | 1385.4ms/step | 14.4 samples/s
+Step 15240/44000 | Loss: 3.0413 | LR: 3.74e-04 | 1384.1ms/step | 14.4 samples/s
+Step 15250/44000 | Loss: 3.0270 | LR: 3.74e-04 | 1384.8ms/step | 14.4 samples/s
+Step 15260/44000 | Loss: 2.9857 | LR: 3.74e-04 | 1385.5ms/step | 14.4 samples/s
+Step 15270/44000 | Loss: 3.0281 | LR: 3.73e-04 | 1383.4ms/step | 14.5 samples/s
+Step 15280/44000 | Loss: 3.0324 | LR: 3.73e-04 | 1384.1ms/step | 14.4 samples/s
+Step 15290/44000 | Loss: 3.0523 | LR: 3.73e-04 | 1387.1ms/step | 14.4 samples/s
+Step 15300/44000 | Loss: 3.0513 | LR: 3.73e-04 | 1388.0ms/step | 14.4 samples/s
+Step 15310/44000 | Loss: 3.0282 | LR: 3.73e-04 | 1387.5ms/step | 14.4 samples/s
+Step 15320/44000 | Loss: 3.0610 | LR: 3.73e-04 | 1384.0ms/step | 14.5 samples/s
+Step 15330/44000 | Loss: 3.0312 | LR: 3.72e-04 | 1385.6ms/step | 14.4 samples/s
+Step 15340/44000 | Loss: 3.0255 | LR: 3.72e-04 | 1384.6ms/step | 14.4 samples/s
+Step 15350/44000 | Loss: 3.0519 | LR: 3.72e-04 | 1384.7ms/step | 14.4 samples/s
+Step 15360/44000 | Loss: 3.0079 | LR: 3.72e-04 | 1385.1ms/step | 14.4 samples/s
+Step 15370/44000 | Loss: 3.0117 | LR: 3.72e-04 | 1387.6ms/step | 14.4 samples/s
+Step 15380/44000 | Loss: 3.0427 | LR: 3.72e-04 | 1385.4ms/step | 14.4 samples/s
+Step 15390/44000 | Loss: 3.0269 | LR: 3.71e-04 | 1384.9ms/step | 14.4 samples/s
+Step 15400/44000 | Loss: 3.0333 | LR: 3.71e-04 | 1382.6ms/step | 14.5 samples/s
+Step 15410/44000 | Loss: 3.0421 | LR: 3.71e-04 | 1383.0ms/step | 14.5 samples/s
+Step 15420/44000 | Loss: 3.0108 | LR: 3.71e-04 | 1384.4ms/step | 14.4 samples/s
+Step 15430/44000 | Loss: 3.0731 | LR: 3.71e-04 | 1385.8ms/step | 14.4 samples/s
+Step 15440/44000 | Loss: 3.0163 | LR: 3.71e-04 | 1383.8ms/step | 14.5 samples/s
+Step 15450/44000 | Loss: 3.0307 | LR: 3.71e-04 | 1387.6ms/step | 14.4 samples/s
+Step 15460/44000 | Loss: 3.0323 | LR: 3.70e-04 | 1385.2ms/step | 14.4 samples/s
+Step 15470/44000 | Loss: 3.0277 | LR: 3.70e-04 | 1386.1ms/step | 14.4 samples/s
+Step 15480/44000 | Loss: 3.0237 | LR: 3.70e-04 | 1385.1ms/step | 14.4 samples/s
+Step 15490/44000 | Loss: 3.0137 | LR: 3.70e-04 | 1385.3ms/step | 14.4 samples/s
+Step 15500/44000 | Loss: 3.0038 | LR: 3.70e-04 | 1384.8ms/step | 14.4 samples/s
+Step 15510/44000 | Loss: 3.0320 | LR: 3.70e-04 | 1386.4ms/step | 14.4 samples/s
+Step 15520/44000 | Loss: 3.0344 | LR: 3.69e-04 | 1386.7ms/step | 14.4 samples/s
+Step 15530/44000 | Loss: 3.0344 | LR: 3.69e-04 | 1385.9ms/step | 14.4 samples/s
+Step 15540/44000 | Loss: 3.0105 | LR: 3.69e-04 | 1384.8ms/step | 14.4 samples/s
+Step 15550/44000 | Loss: 3.0442 | LR: 3.69e-04 | 1386.5ms/step | 14.4 samples/s
+Step 15560/44000 | Loss: 3.0318 | LR: 3.69e-04 | 1384.8ms/step | 14.4 samples/s
+Step 15570/44000 | Loss: 3.0118 | LR: 3.69e-04 | 1385.8ms/step | 14.4 samples/s
+Step 15580/44000 | Loss: 3.0386 | LR: 3.68e-04 | 1386.3ms/step | 14.4 samples/s
+Step 15590/44000 | Loss: 2.9988 | LR: 3.68e-04 | 1385.2ms/step | 14.4 samples/s
+Step 15600/44000 | Loss: 3.0204 | LR: 3.68e-04 | 1386.0ms/step | 14.4 samples/s
+Step 15610/44000 | Loss: 3.0008 | LR: 3.68e-04 | 1387.0ms/step | 14.4 samples/s
+Step 15620/44000 | Loss: 3.0103 | LR: 3.68e-04 | 1385.4ms/step | 14.4 samples/s
+Step 15630/44000 | Loss: 3.0302 | LR: 3.68e-04 | 1386.4ms/step | 14.4 samples/s
+Step 15640/44000 | Loss: 3.0298 | LR: 3.67e-04 | 1386.8ms/step | 14.4 samples/s
+Step 15650/44000 | Loss: 3.0513 | LR: 3.67e-04 | 1385.2ms/step | 14.4 samples/s
+Step 15660/44000 | Loss: 3.0410 | LR: 3.67e-04 | 1386.8ms/step | 14.4 samples/s
+Step 15670/44000 | Loss: 3.0181 | LR: 3.67e-04 | 1385.4ms/step | 14.4 samples/s
+Step 15680/44000 | Loss: 3.0058 | LR: 3.67e-04 | 1387.5ms/step | 14.4 samples/s
+Step 15690/44000 | Loss: 3.0076 | LR: 3.67e-04 | 1386.5ms/step | 14.4 samples/s
+Step 15700/44000 | Loss: 3.0186 | LR: 3.67e-04 | 1385.2ms/step | 14.4 samples/s
+Step 15710/44000 | Loss: 3.0103 | LR: 3.66e-04 | 1384.5ms/step | 14.4 samples/s
+Step 15720/44000 | Loss: 3.0278 | LR: 3.66e-04 | 1385.5ms/step | 14.4 samples/s
+Step 15730/44000 | Loss: 3.0192 | LR: 3.66e-04 | 1386.3ms/step | 14.4 samples/s
+Step 15740/44000 | Loss: 3.0188 | LR: 3.66e-04 | 1384.6ms/step | 14.4 samples/s
+Step 15750/44000 | Loss: 3.0381 | LR: 3.66e-04 | 1383.5ms/step | 14.5 samples/s
+Step 15760/44000 | Loss: 3.0392 | LR: 3.66e-04 | 1384.4ms/step | 14.4 samples/s
+Step 15770/44000 | Loss: 3.0306 | LR: 3.65e-04 | 1383.2ms/step | 14.5 samples/s
+Step 15780/44000 | Loss: 3.0264 | LR: 3.65e-04 | 1384.6ms/step | 14.4 samples/s
+Step 15790/44000 | Loss: 3.0023 | LR: 3.65e-04 | 1383.5ms/step | 14.5 samples/s
+Step 15800/44000 | Loss: 3.0266 | LR: 3.65e-04 | 1384.8ms/step | 14.4 samples/s
+Step 15810/44000 | Loss: 3.0322 | LR: 3.65e-04 | 1384.9ms/step | 14.4 samples/s
+Step 15820/44000 | Loss: 3.0017 | LR: 3.65e-04 | 1385.5ms/step | 14.4 samples/s
+Step 15830/44000 | Loss: 3.0294 | LR: 3.64e-04 | 1382.9ms/step | 14.5 samples/s
+Step 15840/44000 | Loss: 3.0042 | LR: 3.64e-04 | 1384.3ms/step | 14.4 samples/s
+Step 15850/44000 | Loss: 3.0312 | LR: 3.64e-04 | 1387.2ms/step | 14.4 samples/s
+Step 15860/44000 | Loss: 3.0034 | LR: 3.64e-04 | 1383.6ms/step | 14.5 samples/s
+Step 15870/44000 | Loss: 3.0205 | LR: 3.64e-04 | 1385.2ms/step | 14.4 samples/s
+Step 15880/44000 | Loss: 3.0478 | LR: 3.64e-04 | 1384.7ms/step | 14.4 samples/s
+Step 15890/44000 | Loss: 3.0021 | LR: 3.63e-04 | 1383.1ms/step | 14.5 samples/s
+Step 15900/44000 | Loss: 3.0343 | LR: 3.63e-04 | 1385.2ms/step | 14.4 samples/s
+Step 15910/44000 | Loss: 2.9988 | LR: 3.63e-04 | 1384.2ms/step | 14.4 samples/s
+Step 15920/44000 | Loss: 2.9947 | LR: 3.63e-04 | 1384.3ms/step | 14.4 samples/s
+Step 15930/44000 | Loss: 3.0073 | LR: 3.63e-04 | 1384.4ms/step | 14.4 samples/s
+Step 15940/44000 | Loss: 3.0401 | LR: 3.63e-04 | 1383.6ms/step | 14.5 samples/s
+Step 15950/44000 | Loss: 3.0311 | LR: 3.62e-04 | 1383.2ms/step | 14.5 samples/s
+Step 15960/44000 | Loss: 3.0153 | LR: 3.62e-04 | 1383.6ms/step | 14.5 samples/s
+Step 15970/44000 | Loss: 3.0067 | LR: 3.62e-04 | 1384.7ms/step | 14.4 samples/s
+Step 15980/44000 | Loss: 3.0466 | LR: 3.62e-04 | 1386.7ms/step | 14.4 samples/s
+Step 15990/44000 | Loss: 3.0005 | LR: 3.62e-04 | 1386.6ms/step | 14.4 samples/s
+Step 16000/44000 | Loss: 3.0193 | LR: 3.62e-04 | 1385.5ms/step | 14.4 samples/s
+Validation at step 16000: Loss = 3.0266, Perplexity = 20.63
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858/checkpoint-16000
+Step 16010/44000 | Loss: 3.0097 | LR: 3.62e-04 | 3895.0ms/step | 5.1 samples/s
+Step 16020/44000 | Loss: 3.0518 | LR: 3.61e-04 | 1379.8ms/step | 14.5 samples/s
+Step 16030/44000 | Loss: 3.0150 | LR: 3.61e-04 | 1380.5ms/step | 14.5 samples/s
+Step 16040/44000 | Loss: 3.0480 | LR: 3.61e-04 | 1381.2ms/step | 14.5 samples/s
+Step 16050/44000 | Loss: 2.9980 | LR: 3.61e-04 | 1381.9ms/step | 14.5 samples/s
+Step 16060/44000 | Loss: 3.0403 | LR: 3.61e-04 | 1382.0ms/step | 14.5 samples/s
+Step 16070/44000 | Loss: 3.0418 | LR: 3.61e-04 | 1379.8ms/step | 14.5 samples/s
+Step 16080/44000 | Loss: 3.0072 | LR: 3.60e-04 | 1379.8ms/step | 14.5 samples/s
+Step 16090/44000 | Loss: 3.0479 | LR: 3.60e-04 | 1380.1ms/step | 14.5 samples/s
+Step 16100/44000 | Loss: 3.0067 | LR: 3.60e-04 | 1378.8ms/step | 14.5 samples/s
+Step 16110/44000 | Loss: 3.0202 | LR: 3.60e-04 | 1381.4ms/step | 14.5 samples/s
+Step 16120/44000 | Loss: 2.9743 | LR: 3.60e-04 | 1381.9ms/step | 14.5 samples/s
+Step 16130/44000 | Loss: 3.0282 | LR: 3.60e-04 | 1382.7ms/step | 14.5 samples/s
+Step 16140/44000 | Loss: 3.0275 | LR: 3.59e-04 | 1382.2ms/step | 14.5 samples/s
+Step 16150/44000 | Loss: 2.9930 | LR: 3.59e-04 | 1383.8ms/step | 14.5 samples/s
+Step 16160/44000 | Loss: 2.9986 | LR: 3.59e-04 | 1382.8ms/step | 14.5 samples/s
+Step 16170/44000 | Loss: 2.9994 | LR: 3.59e-04 | 1383.0ms/step | 14.5 samples/s
+Step 16180/44000 | Loss: 3.0058 | LR: 3.59e-04 | 1381.9ms/step | 14.5 samples/s
+Step 16190/44000 | Loss: 3.0110 | LR: 3.59e-04 | 1382.4ms/step | 14.5 samples/s
+Step 16200/44000 | Loss: 3.0368 | LR: 3.58e-04 | 1382.9ms/step | 14.5 samples/s
+Step 16210/44000 | Loss: 3.0315 | LR: 3.58e-04 | 1383.2ms/step | 14.5 samples/s
+Step 16220/44000 | Loss: 2.9890 | LR: 3.58e-04 | 1381.7ms/step | 14.5 samples/s
+Step 16230/44000 | Loss: 3.0031 | LR: 3.58e-04 | 1383.6ms/step | 14.5 samples/s
+Step 16240/44000 | Loss: 3.0225 | LR: 3.58e-04 | 1382.5ms/step | 14.5 samples/s
+Step 16250/44000 | Loss: 3.0390 | LR: 3.58e-04 | 1384.8ms/step | 14.4 samples/s
+Step 16260/44000 | Loss: 3.0144 | LR: 3.57e-04 | 1382.8ms/step | 14.5 samples/s
+Step 16270/44000 | Loss: 2.9970 | LR: 3.57e-04 | 1384.3ms/step | 14.4 samples/s
+Step 16280/44000 | Loss: 3.0295 | LR: 3.57e-04 | 1385.6ms/step | 14.4 samples/s
+Step 16290/44000 | Loss: 3.0052 | LR: 3.57e-04 | 1383.5ms/step | 14.5 samples/s
+Step 16300/44000 | Loss: 3.0128 | LR: 3.57e-04 | 1381.1ms/step | 14.5 samples/s
+Step 16310/44000 | Loss: 3.0132 | LR: 3.57e-04 | 1382.0ms/step | 14.5 samples/s
+Step 16320/44000 | Loss: 3.0091 | LR: 3.56e-04 | 1383.8ms/step | 14.5 samples/s
+Step 16330/44000 | Loss: 3.0360 | LR: 3.56e-04 | 1381.8ms/step | 14.5 samples/s
+Step 16340/44000 | Loss: 2.9995 | LR: 3.56e-04 | 1385.2ms/step | 14.4 samples/s
+Step 16350/44000 | Loss: 3.0521 | LR: 3.56e-04 | 1384.5ms/step | 14.4 samples/s
+Step 16360/44000 | Loss: 2.9818 | LR: 3.56e-04 | 1384.4ms/step | 14.4 samples/s
+Step 16370/44000 | Loss: 3.0304 | LR: 3.56e-04 | 1385.3ms/step | 14.4 samples/s
+Step 16380/44000 | Loss: 3.0303 | LR: 3.55e-04 | 1382.0ms/step | 14.5 samples/s
+Step 16390/44000 | Loss: 3.0041 | LR: 3.55e-04 | 1385.5ms/step | 14.4 samples/s
+Step 16400/44000 | Loss: 2.9996 | LR: 3.55e-04 | 1382.7ms/step | 14.5 samples/s
+Step 16410/44000 | Loss: 2.9975 | LR: 3.55e-04 | 1382.9ms/step | 14.5 samples/s
+Step 16420/44000 | Loss: 2.9944 | LR: 3.55e-04 | 1383.2ms/step | 14.5 samples/s
+Step 16430/44000 | Loss: 3.0136 | LR: 3.55e-04 | 1383.7ms/step | 14.5 samples/s
+Step 16440/44000 | Loss: 3.0332 | LR: 3.54e-04 | 1382.2ms/step | 14.5 samples/s
+Step 16450/44000 | Loss: 2.9992 | LR: 3.54e-04 | 1382.3ms/step | 14.5 samples/s
+Step 16460/44000 | Loss: 3.0007 | LR: 3.54e-04 | 1381.6ms/step | 14.5 samples/s
+Step 16470/44000 | Loss: 3.0298 | LR: 3.54e-04 | 1384.0ms/step | 14.5 samples/s
+Step 16480/44000 | Loss: 2.9728 | LR: 3.54e-04 | 1383.8ms/step | 14.5 samples/s
+Step 16490/44000 | Loss: 3.0055 | LR: 3.54e-04 | 1383.4ms/step | 14.5 samples/s
+Step 16500/44000 | Loss: 3.0486 | LR: 3.53e-04 | 1383.6ms/step | 14.5 samples/s
+Step 16510/44000 | Loss: 2.9800 | LR: 3.53e-04 | 1383.6ms/step | 14.5 samples/s
+Step 16520/44000 | Loss: 3.0139 | LR: 3.53e-04 | 1380.3ms/step | 14.5 samples/s
+Step 16530/44000 | Loss: 3.0321 | LR: 3.53e-04 | 1383.2ms/step | 14.5 samples/s
+Step 16540/44000 | Loss: 3.0359 | LR: 3.53e-04 | 1385.6ms/step | 14.4 samples/s
+Step 16550/44000 | Loss: 3.0076 | LR: 3.53e-04 | 1382.2ms/step | 14.5 samples/s
+Step 16560/44000 | Loss: 3.0044 | LR: 3.52e-04 | 1382.8ms/step | 14.5 samples/s
+Step 16570/44000 | Loss: 3.0331 | LR: 3.52e-04 | 1383.6ms/step | 14.5 samples/s
+Step 16580/44000 | Loss: 3.0068 | LR: 3.52e-04 | 1383.4ms/step | 14.5 samples/s
+Step 16590/44000 | Loss: 2.9892 | LR: 3.52e-04 | 1380.8ms/step | 14.5 samples/s
+Step 16600/44000 | Loss: 2.9902 | LR: 3.52e-04 | 1385.8ms/step | 14.4 samples/s
+Step 16610/44000 | Loss: 2.9925 | LR: 3.52e-04 | 1383.0ms/step | 14.5 samples/s
+Step 16620/44000 | Loss: 3.0049 | LR: 3.51e-04 | 1384.8ms/step | 14.4 samples/s
+Step 16630/44000 | Loss: 2.9694 | LR: 3.51e-04 | 1381.8ms/step | 14.5 samples/s
+Step 16640/44000 | Loss: 2.9884 | LR: 3.51e-04 | 1384.6ms/step | 14.4 samples/s
+Step 16650/44000 | Loss: 3.0005 | LR: 3.51e-04 | 1383.6ms/step | 14.5 samples/s
+Step 16660/44000 | Loss: 3.0015 | LR: 3.51e-04 | 1382.6ms/step | 14.5 samples/s
+Step 16670/44000 | Loss: 3.0329 | LR: 3.51e-04 | 1383.8ms/step | 14.5 samples/s
+Step 16680/44000 | Loss: 3.0019 | LR: 3.50e-04 | 1382.3ms/step | 14.5 samples/s
+Step 16690/44000 | Loss: 3.0203 | LR: 3.50e-04 | 1382.4ms/step | 14.5 samples/s
+Step 16700/44000 | Loss: 2.9827 | LR: 3.50e-04 | 1382.8ms/step | 14.5 samples/s
+Step 16710/44000 | Loss: 3.0042 | LR: 3.50e-04 | 1382.6ms/step | 14.5 samples/s
+Step 16720/44000 | Loss: 3.0251 | LR: 3.50e-04 | 1381.6ms/step | 14.5 samples/s
+Step 16730/44000 | Loss: 3.0141 | LR: 3.50e-04 | 1384.9ms/step | 14.4 samples/s
+Step 16740/44000 | Loss: 3.0411 | LR: 3.49e-04 | 1382.9ms/step | 14.5 samples/s
+Step 16750/44000 | Loss: 3.0324 | LR: 3.49e-04 | 1385.6ms/step | 14.4 samples/s
+Step 16760/44000 | Loss: 2.9866 | LR: 3.49e-04 | 1383.3ms/step | 14.5 samples/s
+Step 16770/44000 | Loss: 3.0003 | LR: 3.49e-04 | 1383.6ms/step | 14.5 samples/s
+Step 16780/44000 | Loss: 3.0134 | LR: 3.49e-04 | 1386.1ms/step | 14.4 samples/s
+Step 16790/44000 | Loss: 3.0487 | LR: 3.49e-04 | 1385.9ms/step | 14.4 samples/s
+Step 16800/44000 | Loss: 2.9927 | LR: 3.48e-04 | 1384.0ms/step | 14.5 samples/s
+Step 16810/44000 | Loss: 3.0002 | LR: 3.48e-04 | 1382.5ms/step | 14.5 samples/s
+Step 16820/44000 | Loss: 3.0196 | LR: 3.48e-04 | 1386.5ms/step | 14.4 samples/s
+Step 16830/44000 | Loss: 3.0025 | LR: 3.48e-04 | 1384.0ms/step | 14.5 samples/s
+Step 16840/44000 | Loss: 3.0079 | LR: 3.48e-04 | 1383.4ms/step | 14.5 samples/s
+Step 16850/44000 | Loss: 2.9754 | LR: 3.48e-04 | 1386.6ms/step | 14.4 samples/s
+Step 16860/44000 | Loss: 3.0265 | LR: 3.47e-04 | 1384.1ms/step | 14.5 samples/s
+Step 16870/44000 | Loss: 2.9931 | LR: 3.47e-04 | 1383.1ms/step | 14.5 samples/s
+Step 16880/44000 | Loss: 2.9883 | LR: 3.47e-04 | 1385.4ms/step | 14.4 samples/s
+Step 16890/44000 | Loss: 3.0091 | LR: 3.47e-04 | 1384.9ms/step | 14.4 samples/s
+Step 16900/44000 | Loss: 3.0087 | LR: 3.47e-04 | 1385.6ms/step | 14.4 samples/s
+Step 16910/44000 | Loss: 3.0072 | LR: 3.47e-04 | 1384.5ms/step | 14.4 samples/s
+Step 16920/44000 | Loss: 3.0145 | LR: 3.46e-04 | 1383.8ms/step | 14.5 samples/s
+Step 16930/44000 | Loss: 2.9838 | LR: 3.46e-04 | 1384.8ms/step | 14.4 samples/s
+Step 16940/44000 | Loss: 2.9971 | LR: 3.46e-04 | 1385.1ms/step | 14.4 samples/s
+Step 16950/44000 | Loss: 2.9736 | LR: 3.46e-04 | 1385.7ms/step | 14.4 samples/s
+Step 16960/44000 | Loss: 2.9987 | LR: 3.46e-04 | 1387.3ms/step | 14.4 samples/s
+Step 16970/44000 | Loss: 2.9549 | LR: 3.46e-04 | 1387.4ms/step | 14.4 samples/s
+Step 16980/44000 | Loss: 3.0334 | LR: 3.45e-04 | 1387.1ms/step | 14.4 samples/s
+Step 16990/44000 | Loss: 3.0090 | LR: 3.45e-04 | 1386.3ms/step | 14.4 samples/s
+Step 17000/44000 | Loss: 3.0116 | LR: 3.45e-04 | 1384.1ms/step | 14.4 samples/s
+Validation at step 17000: Loss = 3.0135, Perplexity = 20.36
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858/checkpoint-17000
+Step 17010/44000 | Loss: 3.0070 | LR: 3.45e-04 | 3948.1ms/step | 5.1 samples/s
+Step 17020/44000 | Loss: 2.9870 | LR: 3.45e-04 | 1378.0ms/step | 14.5 samples/s
+Step 17030/44000 | Loss: 3.0102 | LR: 3.45e-04 | 1378.9ms/step | 14.5 samples/s
+Step 17040/44000 | Loss: 3.0028 | LR: 3.44e-04 | 1378.5ms/step | 14.5 samples/s
+Step 17050/44000 | Loss: 2.9953 | LR: 3.44e-04 | 1380.3ms/step | 14.5 samples/s
+Step 17060/44000 | Loss: 2.9630 | LR: 3.44e-04 | 1379.8ms/step | 14.5 samples/s
+Step 17070/44000 | Loss: 3.0029 | LR: 3.44e-04 | 1382.4ms/step | 14.5 samples/s
+Step 17080/44000 | Loss: 2.9980 | LR: 3.44e-04 | 1381.6ms/step | 14.5 samples/s
+Step 17090/44000 | Loss: 3.0183 | LR: 3.44e-04 | 1382.4ms/step | 14.5 samples/s
+Step 17100/44000 | Loss: 3.0024 | LR: 3.43e-04 | 1385.2ms/step | 14.4 samples/s
+Step 17110/44000 | Loss: 2.9932 | LR: 3.43e-04 | 1380.4ms/step | 14.5 samples/s
+Step 17120/44000 | Loss: 3.0063 | LR: 3.43e-04 | 1378.4ms/step | 14.5 samples/s
+Step 17130/44000 | Loss: 3.0088 | LR: 3.43e-04 | 1376.5ms/step | 14.5 samples/s
+Step 17140/44000 | Loss: 2.9916 | LR: 3.43e-04 | 1374.7ms/step | 14.5 samples/s
+Step 17150/44000 | Loss: 3.0089 | LR: 3.43e-04 | 1375.9ms/step | 14.5 samples/s
+Step 17160/44000 | Loss: 3.0026 | LR: 3.42e-04 | 1374.2ms/step | 14.6 samples/s
+Step 17170/44000 | Loss: 3.0051 | LR: 3.42e-04 | 1377.0ms/step | 14.5 samples/s
+Step 17180/44000 | Loss: 3.0175 | LR: 3.42e-04 | 1376.1ms/step | 14.5 samples/s
+Step 17190/44000 | Loss: 3.0205 | LR: 3.42e-04 | 1376.5ms/step | 14.5 samples/s
+Step 17200/44000 | Loss: 2.9861 | LR: 3.42e-04 | 1377.6ms/step | 14.5 samples/s
+Step 17210/44000 | Loss: 3.0223 | LR: 3.42e-04 | 1376.6ms/step | 14.5 samples/s
+Step 17220/44000 | Loss: 3.0380 | LR: 3.41e-04 | 1376.9ms/step | 14.5 samples/s
+Step 17230/44000 | Loss: 3.0040 | LR: 3.41e-04 | 1376.9ms/step | 14.5 samples/s
+Step 17240/44000 | Loss: 3.0171 | LR: 3.41e-04 | 1376.8ms/step | 14.5 samples/s
+Step 17250/44000 | Loss: 3.0174 | LR: 3.41e-04 | 1376.8ms/step | 14.5 samples/s
+Step 17260/44000 | Loss: 3.0344 | LR: 3.41e-04 | 1376.1ms/step | 14.5 samples/s
+Step 17270/44000 | Loss: 3.0230 | LR: 3.41e-04 | 1374.9ms/step | 14.5 samples/s
+Step 17280/44000 | Loss: 2.9788 | LR: 3.40e-04 | 1376.9ms/step | 14.5 samples/s
+Step 17290/44000 | Loss: 2.9969 | LR: 3.40e-04 | 1376.4ms/step | 14.5 samples/s
+Step 17300/44000 | Loss: 3.0321 | LR: 3.40e-04 | 1378.2ms/step | 14.5 samples/s
+Step 17310/44000 | Loss: 2.9923 | LR: 3.40e-04 | 1377.3ms/step | 14.5 samples/s
+Step 17320/44000 | Loss: 3.0062 | LR: 3.40e-04 | 1374.5ms/step | 14.6 samples/s
+Step 17330/44000 | Loss: 2.9933 | LR: 3.40e-04 | 1376.8ms/step | 14.5 samples/s
+Step 17340/44000 | Loss: 3.0236 | LR: 3.39e-04 | 1376.7ms/step | 14.5 samples/s
+Step 17350/44000 | Loss: 3.0148 | LR: 3.39e-04 | 1377.2ms/step | 14.5 samples/s
+Step 17360/44000 | Loss: 2.9887 | LR: 3.39e-04 | 1380.1ms/step | 14.5 samples/s
+Step 17370/44000 | Loss: 3.0001 | LR: 3.39e-04 | 1379.5ms/step | 14.5 samples/s
+Step 17380/44000 | Loss: 2.9878 | LR: 3.39e-04 | 1380.4ms/step | 14.5 samples/s
+Step 17390/44000 | Loss: 3.0254 | LR: 3.39e-04 | 1380.3ms/step | 14.5 samples/s
+Step 17400/44000 | Loss: 2.9928 | LR: 3.38e-04 | 1377.3ms/step | 14.5 samples/s
+Step 17410/44000 | Loss: 2.9962 | LR: 3.38e-04 | 1374.9ms/step | 14.5 samples/s
+Step 17420/44000 | Loss: 2.9696 | LR: 3.38e-04 | 1377.0ms/step | 14.5 samples/s
+Step 17430/44000 | Loss: 2.9984 | LR: 3.38e-04 | 1377.8ms/step | 14.5 samples/s
+Step 17440/44000 | Loss: 3.0245 | LR: 3.38e-04 | 1378.0ms/step | 14.5 samples/s
+Step 17450/44000 | Loss: 3.0262 | LR: 3.38e-04 | 1376.4ms/step | 14.5 samples/s
+Step 17460/44000 | Loss: 2.9876 | LR: 3.37e-04 | 1375.7ms/step | 14.5 samples/s
+Step 17470/44000 | Loss: 2.9730 | LR: 3.37e-04 | 1376.5ms/step | 14.5 samples/s
+Step 17480/44000 | Loss: 3.0102 | LR: 3.37e-04 | 1377.3ms/step | 14.5 samples/s
+Step 17490/44000 | Loss: 2.9959 | LR: 3.37e-04 | 1376.7ms/step | 14.5 samples/s
+Step 17500/44000 | Loss: 3.0135 | LR: 3.37e-04 | 1377.7ms/step | 14.5 samples/s
+Step 17510/44000 | Loss: 2.9908 | LR: 3.37e-04 | 1377.6ms/step | 14.5 samples/s
+Step 17520/44000 | Loss: 3.0079 | LR: 3.36e-04 | 1376.3ms/step | 14.5 samples/s
+Step 17530/44000 | Loss: 3.0292 | LR: 3.36e-04 | 1377.5ms/step | 14.5 samples/s

epfl_final/train_logs/ropectl_a1p0_h100_resume/seed45.20260820T041452Z.log

@@ -0,0 +1,656 @@
+Auto-resuming from latest checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735/checkpoint-13000
+Resuming: Using existing output directory /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735
+Running config:
+{
+  "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
+  "model_dtype": "bfloat16",
+  "init_from_scratch": true,
+  "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
+  "val_path": null,
+  "max_seq_length": 8192,
+  "num_workers": 8,
+  "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735",
+  "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735/checkpoint-13000",
+  "force_new_run": false,
+  "num_epochs": 1,
+  "max_steps": 44000,
+  "per_device_batch_size": 2,
+  "gradient_accumulation_steps": 10,
+  "learning_rate": 0.0005,
+  "weight_decay": 0.01,
+  "warmup_steps": 750,
+  "lr_scheduler": "cosine",
+  "max_grad_norm": 1.0,
+  "save_steps": 1000,
+  "eval_steps": 1000,
+  "eval_batches": 50,
+  "logging_steps": 10,
+  "seed": 45,
+  "verbose": true,
+  "checkpoint_activations": false,
+  "strategy": "ddp",
+  "mixed_precision": "bf16",
+  "wandb_enabled": true,
+  "wandb_project": "stochastic_sparsity",
+  "wandb_name": "l8k365m_dense_ropectl_a1p0_seed45",
+  "wandb_tags": [
+    "qwen3",
+    "365m",
+    "8k",
+    "dense",
+    "rope-control"
+  ],
+  "hidden_size": 768,
+  "intermediate_size": 3072,
+  "num_attention_heads": 6,
+  "num_key_value_heads": 2,
+  "num_hidden_layers": 28,
+  "head_dim": null,
+  "attention_dropout": 0.0,
+  "rope_train_position_scale_set": [
+    1.0,
+    0.9896339488176222,
+    0.9692159692159692,
+    0.9420289855072463,
+    0.8730158730158729
+  ],
+  "attention_type": "sparse_stochastic",
+  "sparse_block_size": 128,
+  "sparse_num_random_blocks": 31,
+  "sparse_mask_pool_size": 1,
+  "sparse_val_dense_compare": false,
+  "sparse_layer_indices": null,
+  "sparse_mask_cache_dir": null,
+  "sparse_mask_seed": 0,
+  "sparse_mixed_nrb_set": null,
+  "dense_mode": true,
+  "dense_step_intervals": null
+}
+Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
+Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735/checkpoint-13000
+Checkpoint activations: False
+dense_mode=True: sparse attention layers will use standard causal SDPA.
+
+
+================================================================================
+MODEL ARCHITECTURE
+================================================================================
+Model config:
+  vocab_size: 151936
+  hidden_size: 768
+  num_hidden_layers: 28
+  num_attention_heads: 6
+  num_key_value_heads: 2
+  head_dim: 128
+  intermediate_size: 3072
+  max_position_embeddings: 32768
+  rope_theta: 1000000
+  rope_train_position_scale_set: [1.0, 0.9896339488176222, 0.9692159692159692, 0.9420289855072463, 0.8730158730158729]
+  attention_type: sparse_stochastic
+  sparse_block_size: 128
+  sparse_num_random_blocks: 31
+  sparse_mask_pool_size: 1
+  sparse_max_seq_length: 8192
+
+Parameter count:
+  Total parameters: 358,958,848
+  Trainable parameters: 358,958,848
+  Non-trainable parameters: 0
+  Model size (BF16): 0.72 GB
+
+--------------------------------------------------------------------------------
+LAYER-BY-LAYER SUMMARY
+--------------------------------------------------------------------------------
+================================================================================================================================================================
+Layer (type:depth-idx)                                       Input Shape               Output Shape              Param #                   Trainable
+================================================================================================================================================================
+Qwen3ForCausalLM                                             [2, 8192]                 --                        --                        True
+├─Qwen3Model: 1-1                                            --                        [2, 8192, 768]            --                        True
+│    └─Embedding: 2-1                                        [2, 8192]                 [2, 8192, 768]            116,686,848               True
+│    └─ModuleList: 2-2                                       --                        --                        --                        True
+│    │    └─Qwen3DecoderLayer: 3-1                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-2                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-3                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-4                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-5                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-6                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-7                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-8                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-9                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-10                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-11                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-12                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-13                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-14                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-15                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-16                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-17                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-18                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-19                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-20                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-21                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-22                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-23                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-24                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-25                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-26                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-27                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-28                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    └─Qwen3RMSNorm: 2-3                                     [2, 8192, 768]            [2, 8192, 768]            768                       True
+├─Linear: 1-2                                                [2, 8192, 768]            [2, 8192, 151936]         116,686,848               True
+================================================================================================================================================================
+Total params: 475,645,696
+Trainable params: 475,645,696
+Non-trainable params: 0
+Total mult-adds (Units.MEGABYTES): 951.29
+================================================================================================================================================================
+Input size (MB): 0.13
+Forward/backward pass size (MB): 31197.23
+Params size (MB): 951.29
+Estimated Total Size (MB): 32148.66
+================================================================================================================================================================
+  NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
+================================================================================
+
+Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
+Using pre-tokenized dataset (no runtime tokenization)
+Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
+Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
+Dataset split: 3027764 train, 159356 validation
+Training examples (packed, train split):  3027764
+Validation examples (packed, val split):  159356
+Per-device batch size:                    2
+World size (GPUs):                        1
+Gradient accumulation steps:              10
+Effective batch size (seqs/step):         20
+Sequence length:                          8192
+Tokens per optimizer step:                163,840
+Micro-batches per epoch (per rank):       1513882
+Optimizer steps per epoch:                151388
+Total optimizer steps:                    44000  (capped by max_steps; 0.29 epochs)
+Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735/checkpoint-13000...
+Preserving checkpoint LR state for exact resume: current=[0.0004073888929652048], schedule_base=[0.0005]
+Resumed from checkpoint at step 13000 (epoch 0, micro_step 130000)
+Resuming training from step 13000
+Resuming wandb run xfnxdn24
+wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from WANDB_API_KEY.
+wandb: Currently logged in as: ruben-navasardyan4869 (ruben-navasardyan4869-epfl) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
+wandb: setting up run xfnxdn24
+wandb: Tracking run with wandb version 0.26.1
+wandb: Run data is saved locally in /mloscratch/homes/navasard/sparse/SparseAtt/wandb/run-20260820_041559-xfnxdn24
+wandb: Run `wandb offline` to turn off syncing.
+wandb: Resuming run l8k365m_dense_ropectl_a1p0_seed45
+wandb: ⭐️ View project at https://wandb.ai/ruben-navasardyan4869-epfl/stochastic_sparsity
+wandb: 🚀 View run at https://wandb.ai/ruben-navasardyan4869-epfl/stochastic_sparsity/runs/xfnxdn24
+Starting training...
+Validation at step 13000: Loss = 3.0601, Perplexity = 21.33
+wandb: WARNING Tried to log to step 13000 that is less than the current step 13151. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Switching attention mode for step 13001: dense (dense step interval)
+Step 13010/44000 | Loss: 3.0530 | LR: 4.07e-04 | 30157.5ms/step | 0.7 samples/s
+wandb: WARNING Tried to log to step 13010 that is less than the current step 13151. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13020/44000 | Loss: 3.0363 | LR: 4.07e-04 | 1397.9ms/step | 14.3 samples/s
+wandb: WARNING Tried to log to step 13020 that is less than the current step 13151. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13030/44000 | Loss: 3.0282 | LR: 4.07e-04 | 1395.5ms/step | 14.3 samples/s
+wandb: WARNING Tried to log to step 13030 that is less than the current step 13151. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13040/44000 | Loss: 3.0404 | LR: 4.07e-04 | 1398.6ms/step | 14.3 samples/s
+wandb: WARNING Tried to log to step 13040 that is less than the current step 13151. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13050/44000 | Loss: 3.0329 | LR: 4.07e-04 | 1400.2ms/step | 14.3 samples/s
+wandb: WARNING Tried to log to step 13050 that is less than the current step 13151. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13060/44000 | Loss: 3.0708 | LR: 4.07e-04 | 1400.0ms/step | 14.3 samples/s
+wandb: WARNING Tried to log to step 13060 that is less than the current step 13151. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13070/44000 | Loss: 3.0480 | LR: 4.06e-04 | 1399.8ms/step | 14.3 samples/s
+wandb: WARNING Tried to log to step 13070 that is less than the current step 13151. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13080/44000 | Loss: 3.0513 | LR: 4.06e-04 | 1400.2ms/step | 14.3 samples/s
+wandb: WARNING Tried to log to step 13080 that is less than the current step 13151. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13090/44000 | Loss: 3.0571 | LR: 4.06e-04 | 1399.9ms/step | 14.3 samples/s
+wandb: WARNING Tried to log to step 13090 that is less than the current step 13151. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13100/44000 | Loss: 3.0298 | LR: 4.06e-04 | 1400.0ms/step | 14.3 samples/s
+wandb: WARNING Tried to log to step 13100 that is less than the current step 13151. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13110/44000 | Loss: 3.0451 | LR: 4.06e-04 | 1400.1ms/step | 14.3 samples/s
+wandb: WARNING Tried to log to step 13110 that is less than the current step 13151. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13120/44000 | Loss: 3.0519 | LR: 4.06e-04 | 1402.0ms/step | 14.3 samples/s
+wandb: WARNING Tried to log to step 13120 that is less than the current step 13151. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13130/44000 | Loss: 3.0832 | LR: 4.06e-04 | 1400.0ms/step | 14.3 samples/s
+wandb: WARNING Tried to log to step 13130 that is less than the current step 13151. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13140/44000 | Loss: 3.0502 | LR: 4.05e-04 | 1400.3ms/step | 14.3 samples/s
+wandb: WARNING Tried to log to step 13140 that is less than the current step 13151. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13150/44000 | Loss: 3.0396 | LR: 4.05e-04 | 1401.6ms/step | 14.3 samples/s
+wandb: WARNING Tried to log to step 13150 that is less than the current step 13151. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13160/44000 | Loss: 3.0522 | LR: 4.05e-04 | 1400.8ms/step | 14.3 samples/s
+Step 13170/44000 | Loss: 3.0706 | LR: 4.05e-04 | 1399.2ms/step | 14.3 samples/s
+Step 13180/44000 | Loss: 3.0776 | LR: 4.05e-04 | 1400.2ms/step | 14.3 samples/s
+Step 13190/44000 | Loss: 3.0760 | LR: 4.05e-04 | 1401.8ms/step | 14.3 samples/s
+Step 13200/44000 | Loss: 3.0560 | LR: 4.05e-04 | 1402.3ms/step | 14.3 samples/s
+Step 13210/44000 | Loss: 3.0916 | LR: 4.04e-04 | 1400.5ms/step | 14.3 samples/s
+Step 13220/44000 | Loss: 3.0245 | LR: 4.04e-04 | 1399.6ms/step | 14.3 samples/s
+Step 13230/44000 | Loss: 3.1008 | LR: 4.04e-04 | 1401.6ms/step | 14.3 samples/s
+Step 13240/44000 | Loss: 3.0804 | LR: 4.04e-04 | 1402.0ms/step | 14.3 samples/s
+Step 13250/44000 | Loss: 3.0645 | LR: 4.04e-04 | 1400.1ms/step | 14.3 samples/s
+Step 13260/44000 | Loss: 3.0688 | LR: 4.04e-04 | 1400.1ms/step | 14.3 samples/s
+Step 13270/44000 | Loss: 3.0776 | LR: 4.04e-04 | 1399.8ms/step | 14.3 samples/s
+Step 13280/44000 | Loss: 3.0387 | LR: 4.03e-04 | 1400.2ms/step | 14.3 samples/s
+Step 13290/44000 | Loss: 3.0463 | LR: 4.03e-04 | 1402.2ms/step | 14.3 samples/s
+Step 13300/44000 | Loss: 3.0579 | LR: 4.03e-04 | 1402.0ms/step | 14.3 samples/s
+Step 13310/44000 | Loss: 3.0696 | LR: 4.03e-04 | 1399.9ms/step | 14.3 samples/s
+Step 13320/44000 | Loss: 3.0288 | LR: 4.03e-04 | 1401.8ms/step | 14.3 samples/s
+Step 13330/44000 | Loss: 3.0232 | LR: 4.03e-04 | 1402.2ms/step | 14.3 samples/s
+Step 13340/44000 | Loss: 3.0448 | LR: 4.03e-04 | 1400.1ms/step | 14.3 samples/s
+Step 13350/44000 | Loss: 3.0312 | LR: 4.02e-04 | 1404.5ms/step | 14.2 samples/s
+Step 13360/44000 | Loss: 3.0378 | LR: 4.02e-04 | 1405.4ms/step | 14.2 samples/s
+Step 13370/44000 | Loss: 3.0470 | LR: 4.02e-04 | 1402.0ms/step | 14.3 samples/s
+Step 13380/44000 | Loss: 3.0633 | LR: 4.02e-04 | 1403.9ms/step | 14.2 samples/s
+Step 13390/44000 | Loss: 3.0842 | LR: 4.02e-04 | 1404.6ms/step | 14.2 samples/s
+Step 13400/44000 | Loss: 3.0427 | LR: 4.02e-04 | 1403.9ms/step | 14.2 samples/s
+Step 13410/44000 | Loss: 3.0296 | LR: 4.02e-04 | 1399.2ms/step | 14.3 samples/s
+Step 13420/44000 | Loss: 3.0456 | LR: 4.01e-04 | 1401.5ms/step | 14.3 samples/s
+Step 13430/44000 | Loss: 3.0520 | LR: 4.01e-04 | 1404.5ms/step | 14.2 samples/s
+Step 13440/44000 | Loss: 3.0475 | LR: 4.01e-04 | 1400.1ms/step | 14.3 samples/s
+Step 13450/44000 | Loss: 3.0465 | LR: 4.01e-04 | 1400.2ms/step | 14.3 samples/s
+Step 13460/44000 | Loss: 3.0584 | LR: 4.01e-04 | 1400.5ms/step | 14.3 samples/s
+Step 13470/44000 | Loss: 3.0524 | LR: 4.01e-04 | 1399.5ms/step | 14.3 samples/s
+Step 13480/44000 | Loss: 3.0477 | LR: 4.01e-04 | 1399.7ms/step | 14.3 samples/s
+Step 13490/44000 | Loss: 3.0477 | LR: 4.00e-04 | 1400.2ms/step | 14.3 samples/s
+Step 13500/44000 | Loss: 3.0386 | LR: 4.00e-04 | 1399.9ms/step | 14.3 samples/s
+Step 13510/44000 | Loss: 3.0418 | LR: 4.00e-04 | 1402.4ms/step | 14.3 samples/s
+Step 13520/44000 | Loss: 3.0534 | LR: 4.00e-04 | 1399.8ms/step | 14.3 samples/s
+Step 13530/44000 | Loss: 3.0395 | LR: 4.00e-04 | 1399.7ms/step | 14.3 samples/s
+Step 13540/44000 | Loss: 3.0473 | LR: 4.00e-04 | 1400.3ms/step | 14.3 samples/s
+Step 13550/44000 | Loss: 3.0782 | LR: 4.00e-04 | 1401.6ms/step | 14.3 samples/s
+Step 13560/44000 | Loss: 3.0321 | LR: 3.99e-04 | 1400.3ms/step | 14.3 samples/s
+Step 13570/44000 | Loss: 3.0586 | LR: 3.99e-04 | 1399.8ms/step | 14.3 samples/s
+Step 13580/44000 | Loss: 3.0491 | LR: 3.99e-04 | 1400.7ms/step | 14.3 samples/s
+Step 13590/44000 | Loss: 3.0513 | LR: 3.99e-04 | 1401.3ms/step | 14.3 samples/s
+Step 13600/44000 | Loss: 3.0597 | LR: 3.99e-04 | 1400.1ms/step | 14.3 samples/s
+Step 13610/44000 | Loss: 3.0593 | LR: 3.99e-04 | 1399.8ms/step | 14.3 samples/s
+Step 13620/44000 | Loss: 3.0857 | LR: 3.98e-04 | 1402.3ms/step | 14.3 samples/s
+Step 13630/44000 | Loss: 3.0691 | LR: 3.98e-04 | 1400.3ms/step | 14.3 samples/s
+Step 13640/44000 | Loss: 3.0180 | LR: 3.98e-04 | 1401.5ms/step | 14.3 samples/s
+Step 13650/44000 | Loss: 3.0189 | LR: 3.98e-04 | 1400.1ms/step | 14.3 samples/s
+Step 13660/44000 | Loss: 3.0850 | LR: 3.98e-04 | 1404.3ms/step | 14.2 samples/s
+Step 13670/44000 | Loss: 3.0420 | LR: 3.98e-04 | 1399.5ms/step | 14.3 samples/s
+Step 13680/44000 | Loss: 3.0472 | LR: 3.98e-04 | 1400.0ms/step | 14.3 samples/s
+Step 13690/44000 | Loss: 3.0447 | LR: 3.97e-04 | 1400.2ms/step | 14.3 samples/s
+Step 13700/44000 | Loss: 3.0604 | LR: 3.97e-04 | 1404.4ms/step | 14.2 samples/s
+Step 13710/44000 | Loss: 3.0303 | LR: 3.97e-04 | 1401.4ms/step | 14.3 samples/s
+Step 13720/44000 | Loss: 3.0465 | LR: 3.97e-04 | 1400.6ms/step | 14.3 samples/s
+Step 13730/44000 | Loss: 3.0249 | LR: 3.97e-04 | 1401.8ms/step | 14.3 samples/s
+Step 13740/44000 | Loss: 3.0263 | LR: 3.97e-04 | 1403.8ms/step | 14.2 samples/s
+Step 13750/44000 | Loss: 3.0443 | LR: 3.97e-04 | 1401.8ms/step | 14.3 samples/s
+Step 13760/44000 | Loss: 3.0645 | LR: 3.96e-04 | 1402.2ms/step | 14.3 samples/s
+Step 13770/44000 | Loss: 3.0402 | LR: 3.96e-04 | 1402.5ms/step | 14.3 samples/s
+Step 13780/44000 | Loss: 3.0526 | LR: 3.96e-04 | 1403.6ms/step | 14.2 samples/s
+Step 13790/44000 | Loss: 3.0623 | LR: 3.96e-04 | 1401.8ms/step | 14.3 samples/s
+Step 13800/44000 | Loss: 3.0307 | LR: 3.96e-04 | 1400.5ms/step | 14.3 samples/s
+Step 13810/44000 | Loss: 3.0090 | LR: 3.96e-04 | 1401.4ms/step | 14.3 samples/s
+Step 13820/44000 | Loss: 3.0181 | LR: 3.96e-04 | 1399.7ms/step | 14.3 samples/s
+Step 13830/44000 | Loss: 3.0469 | LR: 3.95e-04 | 1400.1ms/step | 14.3 samples/s
+Step 13840/44000 | Loss: 3.0570 | LR: 3.95e-04 | 1400.1ms/step | 14.3 samples/s
+Step 13850/44000 | Loss: 3.0570 | LR: 3.95e-04 | 1400.1ms/step | 14.3 samples/s
+Step 13860/44000 | Loss: 3.0614 | LR: 3.95e-04 | 1399.6ms/step | 14.3 samples/s
+Step 13870/44000 | Loss: 3.0264 | LR: 3.95e-04 | 1400.1ms/step | 14.3 samples/s
+Step 13880/44000 | Loss: 3.0517 | LR: 3.95e-04 | 1399.9ms/step | 14.3 samples/s
+Step 13890/44000 | Loss: 3.0714 | LR: 3.95e-04 | 1402.1ms/step | 14.3 samples/s
+Step 13900/44000 | Loss: 3.0571 | LR: 3.94e-04 | 1400.3ms/step | 14.3 samples/s
+Step 13910/44000 | Loss: 3.0328 | LR: 3.94e-04 | 1399.7ms/step | 14.3 samples/s
+Step 13920/44000 | Loss: 3.0541 | LR: 3.94e-04 | 1402.1ms/step | 14.3 samples/s
+Step 13930/44000 | Loss: 3.0549 | LR: 3.94e-04 | 1402.9ms/step | 14.3 samples/s
+Step 13940/44000 | Loss: 3.0528 | LR: 3.94e-04 | 1402.9ms/step | 14.3 samples/s
+Step 13950/44000 | Loss: 3.0231 | LR: 3.94e-04 | 1400.7ms/step | 14.3 samples/s
+Step 13960/44000 | Loss: 3.0538 | LR: 3.93e-04 | 1401.3ms/step | 14.3 samples/s
+Step 13970/44000 | Loss: 3.0205 | LR: 3.93e-04 | 1402.2ms/step | 14.3 samples/s
+Step 13980/44000 | Loss: 3.0674 | LR: 3.93e-04 | 1399.9ms/step | 14.3 samples/s
+Step 13990/44000 | Loss: 3.0573 | LR: 3.93e-04 | 1400.3ms/step | 14.3 samples/s
+Step 14000/44000 | Loss: 3.0497 | LR: 3.93e-04 | 1404.1ms/step | 14.2 samples/s
+Validation at step 14000: Loss = 3.0457, Perplexity = 21.02
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735/checkpoint-14000
+Step 14010/44000 | Loss: 3.0269 | LR: 3.93e-04 | 3933.2ms/step | 5.1 samples/s
+Step 14020/44000 | Loss: 3.0386 | LR: 3.93e-04 | 1399.9ms/step | 14.3 samples/s
+Step 14030/44000 | Loss: 3.0397 | LR: 3.92e-04 | 1400.2ms/step | 14.3 samples/s
+Step 14040/44000 | Loss: 3.0568 | LR: 3.92e-04 | 1399.9ms/step | 14.3 samples/s
+Step 14050/44000 | Loss: 3.0401 | LR: 3.92e-04 | 1400.3ms/step | 14.3 samples/s
+Step 14060/44000 | Loss: 3.0412 | LR: 3.92e-04 | 1400.2ms/step | 14.3 samples/s
+Step 14070/44000 | Loss: 3.0348 | LR: 3.92e-04 | 1399.8ms/step | 14.3 samples/s
+Step 14080/44000 | Loss: 3.0322 | LR: 3.92e-04 | 1402.5ms/step | 14.3 samples/s
+Step 14090/44000 | Loss: 3.0133 | LR: 3.92e-04 | 1406.3ms/step | 14.2 samples/s
+Step 14100/44000 | Loss: 3.0499 | LR: 3.91e-04 | 1403.7ms/step | 14.2 samples/s
+Step 14110/44000 | Loss: 3.0379 | LR: 3.91e-04 | 1403.4ms/step | 14.3 samples/s
+Step 14120/44000 | Loss: 3.0224 | LR: 3.91e-04 | 1402.5ms/step | 14.3 samples/s
+Step 14130/44000 | Loss: 3.0570 | LR: 3.91e-04 | 1403.8ms/step | 14.2 samples/s
+Step 14140/44000 | Loss: 3.0246 | LR: 3.91e-04 | 1404.0ms/step | 14.2 samples/s
+Step 14150/44000 | Loss: 3.0318 | LR: 3.91e-04 | 1404.3ms/step | 14.2 samples/s
+Step 14160/44000 | Loss: 3.0360 | LR: 3.90e-04 | 1406.8ms/step | 14.2 samples/s
+Step 14170/44000 | Loss: 3.0725 | LR: 3.90e-04 | 1403.5ms/step | 14.3 samples/s
+Step 14180/44000 | Loss: 3.0542 | LR: 3.90e-04 | 1404.4ms/step | 14.2 samples/s
+Step 14190/44000 | Loss: 3.0367 | LR: 3.90e-04 | 1407.1ms/step | 14.2 samples/s
+Step 14200/44000 | Loss: 3.0304 | LR: 3.90e-04 | 1405.9ms/step | 14.2 samples/s
+Step 14210/44000 | Loss: 3.0421 | LR: 3.90e-04 | 1402.6ms/step | 14.3 samples/s
+Step 14220/44000 | Loss: 3.0308 | LR: 3.90e-04 | 1406.5ms/step | 14.2 samples/s
+Step 14230/44000 | Loss: 3.0301 | LR: 3.89e-04 | 1405.2ms/step | 14.2 samples/s
+Step 14240/44000 | Loss: 3.0277 | LR: 3.89e-04 | 1405.7ms/step | 14.2 samples/s
+Step 14250/44000 | Loss: 3.0392 | LR: 3.89e-04 | 1405.0ms/step | 14.2 samples/s
+Step 14260/44000 | Loss: 3.0446 | LR: 3.89e-04 | 1405.2ms/step | 14.2 samples/s
+Step 14270/44000 | Loss: 3.0438 | LR: 3.89e-04 | 1405.5ms/step | 14.2 samples/s
+Step 14280/44000 | Loss: 3.0680 | LR: 3.89e-04 | 1405.0ms/step | 14.2 samples/s
+Step 14290/44000 | Loss: 3.0635 | LR: 3.89e-04 | 1407.4ms/step | 14.2 samples/s
+Step 14300/44000 | Loss: 3.0529 | LR: 3.88e-04 | 1406.1ms/step | 14.2 samples/s
+Step 14310/44000 | Loss: 3.0404 | LR: 3.88e-04 | 1406.5ms/step | 14.2 samples/s
+Step 14320/44000 | Loss: 3.0086 | LR: 3.88e-04 | 1409.2ms/step | 14.2 samples/s
+Step 14330/44000 | Loss: 3.0353 | LR: 3.88e-04 | 1403.7ms/step | 14.2 samples/s
+Step 14340/44000 | Loss: 3.0280 | LR: 3.88e-04 | 1408.1ms/step | 14.2 samples/s
+Step 14350/44000 | Loss: 3.0095 | LR: 3.88e-04 | 1408.6ms/step | 14.2 samples/s
+Step 14360/44000 | Loss: 3.0805 | LR: 3.87e-04 | 1405.8ms/step | 14.2 samples/s
+Step 14370/44000 | Loss: 3.0334 | LR: 3.87e-04 | 1406.0ms/step | 14.2 samples/s
+Step 14380/44000 | Loss: 3.0244 | LR: 3.87e-04 | 1408.0ms/step | 14.2 samples/s
+Step 14390/44000 | Loss: 3.0272 | LR: 3.87e-04 | 1407.2ms/step | 14.2 samples/s
+Step 14400/44000 | Loss: 3.0292 | LR: 3.87e-04 | 1404.3ms/step | 14.2 samples/s
+Step 14410/44000 | Loss: 3.0409 | LR: 3.87e-04 | 1407.9ms/step | 14.2 samples/s
+Step 14420/44000 | Loss: 3.0291 | LR: 3.87e-04 | 1404.7ms/step | 14.2 samples/s
+Step 14430/44000 | Loss: 3.0118 | LR: 3.86e-04 | 1405.9ms/step | 14.2 samples/s
+Step 14440/44000 | Loss: 3.0532 | LR: 3.86e-04 | 1411.6ms/step | 14.2 samples/s
+Step 14450/44000 | Loss: 3.0091 | LR: 3.86e-04 | 1407.8ms/step | 14.2 samples/s
+Step 14460/44000 | Loss: 3.0106 | LR: 3.86e-04 | 1404.1ms/step | 14.2 samples/s
+Step 14470/44000 | Loss: 3.0369 | LR: 3.86e-04 | 1405.5ms/step | 14.2 samples/s
+Step 14480/44000 | Loss: 3.0082 | LR: 3.86e-04 | 1406.6ms/step | 14.2 samples/s
+Step 14490/44000 | Loss: 3.0267 | LR: 3.85e-04 | 1406.0ms/step | 14.2 samples/s
+Step 14500/44000 | Loss: 3.0286 | LR: 3.85e-04 | 1407.7ms/step | 14.2 samples/s
+Step 14510/44000 | Loss: 3.0183 | LR: 3.85e-04 | 1404.8ms/step | 14.2 samples/s
+Step 14520/44000 | Loss: 3.0552 | LR: 3.85e-04 | 1405.8ms/step | 14.2 samples/s
+Step 14530/44000 | Loss: 3.0307 | LR: 3.85e-04 | 1407.5ms/step | 14.2 samples/s
+Step 14540/44000 | Loss: 3.0493 | LR: 3.85e-04 | 1408.3ms/step | 14.2 samples/s
+Step 14550/44000 | Loss: 3.0265 | LR: 3.85e-04 | 1407.8ms/step | 14.2 samples/s
+Step 14560/44000 | Loss: 3.0358 | LR: 3.84e-04 | 1408.7ms/step | 14.2 samples/s
+Step 14570/44000 | Loss: 3.0553 | LR: 3.84e-04 | 1407.2ms/step | 14.2 samples/s
+Step 14580/44000 | Loss: 3.0542 | LR: 3.84e-04 | 1405.7ms/step | 14.2 samples/s
+Step 14590/44000 | Loss: 3.0441 | LR: 3.84e-04 | 1407.9ms/step | 14.2 samples/s
+Step 14600/44000 | Loss: 3.0100 | LR: 3.84e-04 | 1406.5ms/step | 14.2 samples/s
+Step 14610/44000 | Loss: 3.0265 | LR: 3.84e-04 | 1412.5ms/step | 14.2 samples/s
+Step 14620/44000 | Loss: 3.0501 | LR: 3.83e-04 | 1409.7ms/step | 14.2 samples/s
+Step 14630/44000 | Loss: 3.0324 | LR: 3.83e-04 | 1406.2ms/step | 14.2 samples/s
+Step 14640/44000 | Loss: 2.9894 | LR: 3.83e-04 | 1405.6ms/step | 14.2 samples/s
+Step 14650/44000 | Loss: 3.0245 | LR: 3.83e-04 | 1409.3ms/step | 14.2 samples/s
+Step 14660/44000 | Loss: 3.0410 | LR: 3.83e-04 | 1406.0ms/step | 14.2 samples/s
+Step 14670/44000 | Loss: 3.0240 | LR: 3.83e-04 | 1404.7ms/step | 14.2 samples/s
+Step 14680/44000 | Loss: 3.0410 | LR: 3.83e-04 | 1407.7ms/step | 14.2 samples/s
+Step 14690/44000 | Loss: 3.0278 | LR: 3.82e-04 | 1407.8ms/step | 14.2 samples/s
+Step 14700/44000 | Loss: 2.9932 | LR: 3.82e-04 | 1404.6ms/step | 14.2 samples/s
+Step 14710/44000 | Loss: 3.0281 | LR: 3.82e-04 | 1406.0ms/step | 14.2 samples/s
+Step 14720/44000 | Loss: 3.0153 | LR: 3.82e-04 | 1411.6ms/step | 14.2 samples/s
+Step 14730/44000 | Loss: 3.0230 | LR: 3.82e-04 | 1406.4ms/step | 14.2 samples/s
+Step 14740/44000 | Loss: 3.0272 | LR: 3.82e-04 | 1406.3ms/step | 14.2 samples/s
+Step 14750/44000 | Loss: 3.0469 | LR: 3.81e-04 | 1407.6ms/step | 14.2 samples/s
+Step 14760/44000 | Loss: 3.0199 | LR: 3.81e-04 | 1409.7ms/step | 14.2 samples/s
+Step 14770/44000 | Loss: 3.0266 | LR: 3.81e-04 | 1406.0ms/step | 14.2 samples/s
+Step 14780/44000 | Loss: 3.0151 | LR: 3.81e-04 | 1404.4ms/step | 14.2 samples/s
+Step 14790/44000 | Loss: 3.0456 | LR: 3.81e-04 | 1403.9ms/step | 14.2 samples/s
+Step 14800/44000 | Loss: 3.0314 | LR: 3.81e-04 | 1406.3ms/step | 14.2 samples/s
+Step 14810/44000 | Loss: 3.0352 | LR: 3.81e-04 | 1407.0ms/step | 14.2 samples/s
+Step 14820/44000 | Loss: 3.0393 | LR: 3.80e-04 | 1404.2ms/step | 14.2 samples/s
+Step 14830/44000 | Loss: 2.9928 | LR: 3.80e-04 | 1409.9ms/step | 14.2 samples/s
+Step 14840/44000 | Loss: 3.0206 | LR: 3.80e-04 | 1407.5ms/step | 14.2 samples/s
+Step 14850/44000 | Loss: 3.0378 | LR: 3.80e-04 | 1407.4ms/step | 14.2 samples/s
+Step 14860/44000 | Loss: 3.0264 | LR: 3.80e-04 | 1405.5ms/step | 14.2 samples/s
+Step 14870/44000 | Loss: 3.0054 | LR: 3.80e-04 | 1404.2ms/step | 14.2 samples/s
+Step 14880/44000 | Loss: 3.0470 | LR: 3.79e-04 | 1407.6ms/step | 14.2 samples/s
+Step 14890/44000 | Loss: 3.0423 | LR: 3.79e-04 | 1404.8ms/step | 14.2 samples/s
+Step 14900/44000 | Loss: 3.0194 | LR: 3.79e-04 | 1405.0ms/step | 14.2 samples/s
+Step 14910/44000 | Loss: 3.0062 | LR: 3.79e-04 | 1408.3ms/step | 14.2 samples/s
+Step 14920/44000 | Loss: 3.0421 | LR: 3.79e-04 | 1405.9ms/step | 14.2 samples/s
+Step 14930/44000 | Loss: 3.0251 | LR: 3.79e-04 | 1405.8ms/step | 14.2 samples/s
+Step 14940/44000 | Loss: 3.0564 | LR: 3.79e-04 | 1408.3ms/step | 14.2 samples/s
+Step 14950/44000 | Loss: 3.0071 | LR: 3.78e-04 | 1405.8ms/step | 14.2 samples/s
+Step 14960/44000 | Loss: 3.0480 | LR: 3.78e-04 | 1404.5ms/step | 14.2 samples/s
+Step 14970/44000 | Loss: 3.0143 | LR: 3.78e-04 | 1405.7ms/step | 14.2 samples/s
+Step 14980/44000 | Loss: 3.0089 | LR: 3.78e-04 | 1406.1ms/step | 14.2 samples/s
+Step 14990/44000 | Loss: 3.0396 | LR: 3.78e-04 | 1407.0ms/step | 14.2 samples/s
+Step 15000/44000 | Loss: 3.0423 | LR: 3.78e-04 | 1405.0ms/step | 14.2 samples/s
+Validation at step 15000: Loss = 3.0311, Perplexity = 20.72
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735/checkpoint-15000
+Step 15010/44000 | Loss: 3.0371 | LR: 3.77e-04 | 3929.6ms/step | 5.1 samples/s
+Step 15020/44000 | Loss: 3.0466 | LR: 3.77e-04 | 1400.1ms/step | 14.3 samples/s
+Step 15030/44000 | Loss: 3.0087 | LR: 3.77e-04 | 1400.1ms/step | 14.3 samples/s
+Step 15040/44000 | Loss: 3.0522 | LR: 3.77e-04 | 1402.1ms/step | 14.3 samples/s
+Step 15050/44000 | Loss: 3.0278 | LR: 3.77e-04 | 1401.8ms/step | 14.3 samples/s
+Step 15060/44000 | Loss: 3.0584 | LR: 3.77e-04 | 1404.2ms/step | 14.2 samples/s
+Step 15070/44000 | Loss: 3.0250 | LR: 3.77e-04 | 1406.1ms/step | 14.2 samples/s
+Step 15080/44000 | Loss: 3.0197 | LR: 3.76e-04 | 1403.7ms/step | 14.2 samples/s
+Step 15090/44000 | Loss: 3.0212 | LR: 3.76e-04 | 1404.6ms/step | 14.2 samples/s
+Step 15100/44000 | Loss: 3.0223 | LR: 3.76e-04 | 1401.9ms/step | 14.3 samples/s
+Step 15110/44000 | Loss: 3.0271 | LR: 3.76e-04 | 1403.0ms/step | 14.3 samples/s
+Step 15120/44000 | Loss: 3.0168 | LR: 3.76e-04 | 1402.9ms/step | 14.3 samples/s
+Step 15130/44000 | Loss: 3.0303 | LR: 3.76e-04 | 1404.0ms/step | 14.2 samples/s
+Step 15140/44000 | Loss: 3.0492 | LR: 3.75e-04 | 1407.5ms/step | 14.2 samples/s
+Step 15150/44000 | Loss: 2.9950 | LR: 3.75e-04 | 1404.1ms/step | 14.2 samples/s
+Step 15160/44000 | Loss: 3.0116 | LR: 3.75e-04 | 1405.6ms/step | 14.2 samples/s
+Step 15170/44000 | Loss: 3.0240 | LR: 3.75e-04 | 1404.6ms/step | 14.2 samples/s
+Step 15180/44000 | Loss: 3.0313 | LR: 3.75e-04 | 1405.3ms/step | 14.2 samples/s
+Step 15190/44000 | Loss: 3.0341 | LR: 3.75e-04 | 1403.4ms/step | 14.3 samples/s
+Step 15200/44000 | Loss: 3.0550 | LR: 3.74e-04 | 1404.8ms/step | 14.2 samples/s
+Step 15210/44000 | Loss: 3.0267 | LR: 3.74e-04 | 1405.4ms/step | 14.2 samples/s
+Step 15220/44000 | Loss: 3.0085 | LR: 3.74e-04 | 1408.8ms/step | 14.2 samples/s
+Step 15230/44000 | Loss: 3.0154 | LR: 3.74e-04 | 1408.6ms/step | 14.2 samples/s
+Step 15240/44000 | Loss: 3.0402 | LR: 3.74e-04 | 1405.5ms/step | 14.2 samples/s
+Step 15250/44000 | Loss: 3.0188 | LR: 3.74e-04 | 1404.4ms/step | 14.2 samples/s
+Step 15260/44000 | Loss: 3.0388 | LR: 3.74e-04 | 1406.5ms/step | 14.2 samples/s
+Step 15270/44000 | Loss: 3.0418 | LR: 3.73e-04 | 1408.5ms/step | 14.2 samples/s
+Step 15280/44000 | Loss: 3.0311 | LR: 3.73e-04 | 1408.9ms/step | 14.2 samples/s
+Step 15290/44000 | Loss: 3.0193 | LR: 3.73e-04 | 1408.9ms/step | 14.2 samples/s
+Step 15300/44000 | Loss: 3.0369 | LR: 3.73e-04 | 1404.6ms/step | 14.2 samples/s
+Step 15310/44000 | Loss: 2.9981 | LR: 3.73e-04 | 1406.1ms/step | 14.2 samples/s
+Step 15320/44000 | Loss: 3.0409 | LR: 3.73e-04 | 1406.1ms/step | 14.2 samples/s
+Step 15330/44000 | Loss: 3.0352 | LR: 3.72e-04 | 1406.4ms/step | 14.2 samples/s
+Step 15340/44000 | Loss: 3.0447 | LR: 3.72e-04 | 1406.2ms/step | 14.2 samples/s
+Step 15350/44000 | Loss: 3.0216 | LR: 3.72e-04 | 1407.1ms/step | 14.2 samples/s
+Step 15360/44000 | Loss: 3.0293 | LR: 3.72e-04 | 1402.5ms/step | 14.3 samples/s
+Step 15370/44000 | Loss: 3.0100 | LR: 3.72e-04 | 1405.8ms/step | 14.2 samples/s
+Step 15380/44000 | Loss: 3.0581 | LR: 3.72e-04 | 1404.8ms/step | 14.2 samples/s
+Step 15390/44000 | Loss: 3.0117 | LR: 3.71e-04 | 1403.8ms/step | 14.2 samples/s
+Step 15400/44000 | Loss: 3.0380 | LR: 3.71e-04 | 1405.7ms/step | 14.2 samples/s
+Step 15410/44000 | Loss: 3.0102 | LR: 3.71e-04 | 1408.0ms/step | 14.2 samples/s
+Step 15420/44000 | Loss: 3.0234 | LR: 3.71e-04 | 1405.6ms/step | 14.2 samples/s
+Step 15430/44000 | Loss: 3.0101 | LR: 3.71e-04 | 1406.1ms/step | 14.2 samples/s
+Step 15440/44000 | Loss: 3.0203 | LR: 3.71e-04 | 1405.4ms/step | 14.2 samples/s
+Step 15450/44000 | Loss: 3.0209 | LR: 3.71e-04 | 1404.7ms/step | 14.2 samples/s
+Step 15460/44000 | Loss: 3.0301 | LR: 3.70e-04 | 1406.3ms/step | 14.2 samples/s
+Step 15470/44000 | Loss: 3.0357 | LR: 3.70e-04 | 1405.3ms/step | 14.2 samples/s
+Step 15480/44000 | Loss: 3.0228 | LR: 3.70e-04 | 1404.1ms/step | 14.2 samples/s
+Step 15490/44000 | Loss: 3.0229 | LR: 3.70e-04 | 1406.1ms/step | 14.2 samples/s
+Step 15500/44000 | Loss: 3.0138 | LR: 3.70e-04 | 1405.9ms/step | 14.2 samples/s
+Step 15510/44000 | Loss: 3.0343 | LR: 3.70e-04 | 1408.6ms/step | 14.2 samples/s
+Step 15520/44000 | Loss: 3.0124 | LR: 3.69e-04 | 1407.6ms/step | 14.2 samples/s
+Step 15530/44000 | Loss: 3.0176 | LR: 3.69e-04 | 1406.2ms/step | 14.2 samples/s
+Step 15540/44000 | Loss: 3.0415 | LR: 3.69e-04 | 1407.5ms/step | 14.2 samples/s
+Step 15550/44000 | Loss: 2.9999 | LR: 3.69e-04 | 1405.8ms/step | 14.2 samples/s
+Step 15560/44000 | Loss: 3.0004 | LR: 3.69e-04 | 1406.5ms/step | 14.2 samples/s
+Step 15570/44000 | Loss: 3.0474 | LR: 3.69e-04 | 1404.4ms/step | 14.2 samples/s
+Step 15580/44000 | Loss: 2.9992 | LR: 3.68e-04 | 1406.0ms/step | 14.2 samples/s
+Step 15590/44000 | Loss: 3.0146 | LR: 3.68e-04 | 1405.3ms/step | 14.2 samples/s
+Step 15600/44000 | Loss: 3.0301 | LR: 3.68e-04 | 1404.5ms/step | 14.2 samples/s
+Step 15610/44000 | Loss: 2.9960 | LR: 3.68e-04 | 1405.7ms/step | 14.2 samples/s
+Step 15620/44000 | Loss: 2.9921 | LR: 3.68e-04 | 1407.9ms/step | 14.2 samples/s
+Step 15630/44000 | Loss: 3.0463 | LR: 3.68e-04 | 1406.7ms/step | 14.2 samples/s
+Step 15640/44000 | Loss: 3.0026 | LR: 3.67e-04 | 1407.9ms/step | 14.2 samples/s
+Step 15650/44000 | Loss: 3.0406 | LR: 3.67e-04 | 1405.9ms/step | 14.2 samples/s
+Step 15660/44000 | Loss: 3.0052 | LR: 3.67e-04 | 1407.0ms/step | 14.2 samples/s
+Step 15670/44000 | Loss: 3.0266 | LR: 3.67e-04 | 1404.4ms/step | 14.2 samples/s
+Step 15680/44000 | Loss: 3.0003 | LR: 3.67e-04 | 1408.0ms/step | 14.2 samples/s
+Step 15690/44000 | Loss: 3.0145 | LR: 3.67e-04 | 1410.5ms/step | 14.2 samples/s
+Step 15700/44000 | Loss: 3.0214 | LR: 3.67e-04 | 1407.5ms/step | 14.2 samples/s
+Step 15710/44000 | Loss: 3.0644 | LR: 3.66e-04 | 1405.8ms/step | 14.2 samples/s
+Step 15720/44000 | Loss: 2.9972 | LR: 3.66e-04 | 1405.9ms/step | 14.2 samples/s
+Step 15730/44000 | Loss: 3.0144 | LR: 3.66e-04 | 1410.0ms/step | 14.2 samples/s
+Step 15740/44000 | Loss: 3.0384 | LR: 3.66e-04 | 1410.4ms/step | 14.2 samples/s
+Step 15750/44000 | Loss: 3.0303 | LR: 3.66e-04 | 1410.7ms/step | 14.2 samples/s
+Step 15760/44000 | Loss: 3.0438 | LR: 3.66e-04 | 1406.7ms/step | 14.2 samples/s
+Step 15770/44000 | Loss: 3.0205 | LR: 3.65e-04 | 1404.9ms/step | 14.2 samples/s
+Step 15780/44000 | Loss: 3.0505 | LR: 3.65e-04 | 1407.6ms/step | 14.2 samples/s
+Step 15790/44000 | Loss: 2.9942 | LR: 3.65e-04 | 1405.6ms/step | 14.2 samples/s
+Step 15800/44000 | Loss: 3.0126 | LR: 3.65e-04 | 1405.8ms/step | 14.2 samples/s
+Step 15810/44000 | Loss: 3.0108 | LR: 3.65e-04 | 1405.2ms/step | 14.2 samples/s
+Step 15820/44000 | Loss: 2.9895 | LR: 3.65e-04 | 1405.7ms/step | 14.2 samples/s
+Step 15830/44000 | Loss: 3.0383 | LR: 3.64e-04 | 1405.6ms/step | 14.2 samples/s
+Step 15840/44000 | Loss: 3.0233 | LR: 3.64e-04 | 1406.5ms/step | 14.2 samples/s
+Step 15850/44000 | Loss: 3.0265 | LR: 3.64e-04 | 1406.5ms/step | 14.2 samples/s
+Step 15860/44000 | Loss: 2.9929 | LR: 3.64e-04 | 1406.4ms/step | 14.2 samples/s
+Step 15870/44000 | Loss: 3.0140 | LR: 3.64e-04 | 1404.6ms/step | 14.2 samples/s
+Step 15880/44000 | Loss: 3.0298 | LR: 3.64e-04 | 1408.2ms/step | 14.2 samples/s
+Step 15890/44000 | Loss: 3.0192 | LR: 3.63e-04 | 1408.4ms/step | 14.2 samples/s
+Step 15900/44000 | Loss: 3.0388 | LR: 3.63e-04 | 1409.1ms/step | 14.2 samples/s
+Step 15910/44000 | Loss: 3.0177 | LR: 3.63e-04 | 1406.7ms/step | 14.2 samples/s
+Step 15920/44000 | Loss: 3.0049 | LR: 3.63e-04 | 1407.4ms/step | 14.2 samples/s
+Step 15930/44000 | Loss: 3.0327 | LR: 3.63e-04 | 1406.3ms/step | 14.2 samples/s
+Step 15940/44000 | Loss: 3.0011 | LR: 3.63e-04 | 1404.4ms/step | 14.2 samples/s
+Step 15950/44000 | Loss: 3.0167 | LR: 3.62e-04 | 1405.1ms/step | 14.2 samples/s
+Step 15960/44000 | Loss: 2.9896 | LR: 3.62e-04 | 1406.8ms/step | 14.2 samples/s
+Step 15970/44000 | Loss: 3.0153 | LR: 3.62e-04 | 1406.0ms/step | 14.2 samples/s
+Step 15980/44000 | Loss: 3.0171 | LR: 3.62e-04 | 1407.1ms/step | 14.2 samples/s
+Step 15990/44000 | Loss: 3.0024 | LR: 3.62e-04 | 1400.0ms/step | 14.3 samples/s
+Step 16000/44000 | Loss: 3.0164 | LR: 3.62e-04 | 1400.2ms/step | 14.3 samples/s
+Validation at step 16000: Loss = 3.0182, Perplexity = 20.45
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735/checkpoint-16000
+Step 16010/44000 | Loss: 2.9981 | LR: 3.62e-04 | 3945.8ms/step | 5.1 samples/s
+Step 16020/44000 | Loss: 2.9873 | LR: 3.61e-04 | 1399.8ms/step | 14.3 samples/s
+Step 16030/44000 | Loss: 2.9954 | LR: 3.61e-04 | 1400.0ms/step | 14.3 samples/s
+Step 16040/44000 | Loss: 3.0271 | LR: 3.61e-04 | 1400.3ms/step | 14.3 samples/s
+Step 16050/44000 | Loss: 3.0306 | LR: 3.61e-04 | 1399.8ms/step | 14.3 samples/s
+Step 16060/44000 | Loss: 3.0192 | LR: 3.61e-04 | 1400.3ms/step | 14.3 samples/s
+Step 16070/44000 | Loss: 3.0197 | LR: 3.61e-04 | 1400.3ms/step | 14.3 samples/s
+Step 16080/44000 | Loss: 3.0014 | LR: 3.60e-04 | 1401.9ms/step | 14.3 samples/s
+Step 16090/44000 | Loss: 2.9861 | LR: 3.60e-04 | 1403.8ms/step | 14.2 samples/s
+Step 16100/44000 | Loss: 3.0215 | LR: 3.60e-04 | 1401.8ms/step | 14.3 samples/s
+Step 16110/44000 | Loss: 3.0163 | LR: 3.60e-04 | 1402.9ms/step | 14.3 samples/s
+Step 16120/44000 | Loss: 2.9848 | LR: 3.60e-04 | 1405.4ms/step | 14.2 samples/s
+Step 16130/44000 | Loss: 3.0070 | LR: 3.60e-04 | 1404.1ms/step | 14.2 samples/s
+Step 16140/44000 | Loss: 3.0463 | LR: 3.59e-04 | 1405.9ms/step | 14.2 samples/s
+Step 16150/44000 | Loss: 3.0108 | LR: 3.59e-04 | 1403.7ms/step | 14.2 samples/s
+Step 16160/44000 | Loss: 3.0066 | LR: 3.59e-04 | 1404.4ms/step | 14.2 samples/s
+Step 16170/44000 | Loss: 3.0117 | LR: 3.59e-04 | 1402.0ms/step | 14.3 samples/s
+Step 16180/44000 | Loss: 3.0272 | LR: 3.59e-04 | 1404.3ms/step | 14.2 samples/s
+Step 16190/44000 | Loss: 3.0013 | LR: 3.59e-04 | 1404.4ms/step | 14.2 samples/s
+Step 16200/44000 | Loss: 2.9888 | LR: 3.58e-04 | 1407.0ms/step | 14.2 samples/s
+Step 16210/44000 | Loss: 3.0218 | LR: 3.58e-04 | 1404.4ms/step | 14.2 samples/s
+Step 16220/44000 | Loss: 3.0274 | LR: 3.58e-04 | 1402.2ms/step | 14.3 samples/s
+Step 16230/44000 | Loss: 2.9845 | LR: 3.58e-04 | 1403.6ms/step | 14.2 samples/s
+Step 16240/44000 | Loss: 3.0240 | LR: 3.58e-04 | 1403.7ms/step | 14.2 samples/s
+Step 16250/44000 | Loss: 3.0101 | LR: 3.58e-04 | 1402.2ms/step | 14.3 samples/s
+Step 16260/44000 | Loss: 2.9791 | LR: 3.57e-04 | 1401.7ms/step | 14.3 samples/s
+Step 16270/44000 | Loss: 3.0358 | LR: 3.57e-04 | 1402.1ms/step | 14.3 samples/s
+Step 16280/44000 | Loss: 3.0304 | LR: 3.57e-04 | 1406.9ms/step | 14.2 samples/s
+Step 16290/44000 | Loss: 3.0055 | LR: 3.57e-04 | 1406.8ms/step | 14.2 samples/s
+Step 16300/44000 | Loss: 3.0490 | LR: 3.57e-04 | 1406.7ms/step | 14.2 samples/s
+Step 16310/44000 | Loss: 2.9993 | LR: 3.57e-04 | 1403.7ms/step | 14.2 samples/s
+Step 16320/44000 | Loss: 3.0338 | LR: 3.56e-04 | 1405.5ms/step | 14.2 samples/s
+Step 16330/44000 | Loss: 3.0121 | LR: 3.56e-04 | 1402.6ms/step | 14.3 samples/s
+Step 16340/44000 | Loss: 3.0107 | LR: 3.56e-04 | 1404.7ms/step | 14.2 samples/s
+Step 16350/44000 | Loss: 3.0077 | LR: 3.56e-04 | 1403.5ms/step | 14.3 samples/s
+Step 16360/44000 | Loss: 3.0100 | LR: 3.56e-04 | 1406.1ms/step | 14.2 samples/s
+Step 16370/44000 | Loss: 3.0233 | LR: 3.56e-04 | 1406.8ms/step | 14.2 samples/s
+Step 16380/44000 | Loss: 2.9960 | LR: 3.55e-04 | 1407.5ms/step | 14.2 samples/s
+Step 16390/44000 | Loss: 3.0204 | LR: 3.55e-04 | 1407.0ms/step | 14.2 samples/s
+Step 16400/44000 | Loss: 3.0122 | LR: 3.55e-04 | 1404.4ms/step | 14.2 samples/s
+Step 16410/44000 | Loss: 3.0322 | LR: 3.55e-04 | 1403.5ms/step | 14.2 samples/s
+Step 16420/44000 | Loss: 3.0079 | LR: 3.55e-04 | 1402.1ms/step | 14.3 samples/s
+Step 16430/44000 | Loss: 2.9612 | LR: 3.55e-04 | 1402.6ms/step | 14.3 samples/s
+Step 16440/44000 | Loss: 3.0090 | LR: 3.54e-04 | 1405.5ms/step | 14.2 samples/s
+Step 16450/44000 | Loss: 3.0478 | LR: 3.54e-04 | 1404.1ms/step | 14.2 samples/s
+Step 16460/44000 | Loss: 2.9971 | LR: 3.54e-04 | 1402.5ms/step | 14.3 samples/s
+Step 16470/44000 | Loss: 3.0192 | LR: 3.54e-04 | 1403.2ms/step | 14.3 samples/s
+Step 16480/44000 | Loss: 2.9986 | LR: 3.54e-04 | 1400.5ms/step | 14.3 samples/s
+Step 16490/44000 | Loss: 3.0094 | LR: 3.54e-04 | 1403.8ms/step | 14.2 samples/s
+Step 16500/44000 | Loss: 2.9890 | LR: 3.53e-04 | 1404.4ms/step | 14.2 samples/s
+Step 16510/44000 | Loss: 2.9981 | LR: 3.53e-04 | 1403.8ms/step | 14.2 samples/s
+Step 16520/44000 | Loss: 3.0088 | LR: 3.53e-04 | 1406.4ms/step | 14.2 samples/s
+Step 16530/44000 | Loss: 3.0235 | LR: 3.53e-04 | 1406.9ms/step | 14.2 samples/s
+Step 16540/44000 | Loss: 3.0084 | LR: 3.53e-04 | 1404.3ms/step | 14.2 samples/s
+Step 16550/44000 | Loss: 3.0437 | LR: 3.53e-04 | 1406.2ms/step | 14.2 samples/s
+Step 16560/44000 | Loss: 3.0119 | LR: 3.52e-04 | 1407.5ms/step | 14.2 samples/s
+Step 16570/44000 | Loss: 3.0102 | LR: 3.52e-04 | 1407.9ms/step | 14.2 samples/s
+Step 16580/44000 | Loss: 2.9882 | LR: 3.52e-04 | 1404.5ms/step | 14.2 samples/s
+Step 16590/44000 | Loss: 2.9727 | LR: 3.52e-04 | 1407.3ms/step | 14.2 samples/s
+Step 16600/44000 | Loss: 3.0048 | LR: 3.52e-04 | 1404.1ms/step | 14.2 samples/s
+Step 16610/44000 | Loss: 2.9878 | LR: 3.52e-04 | 1406.3ms/step | 14.2 samples/s
+Step 16620/44000 | Loss: 3.0154 | LR: 3.51e-04 | 1408.2ms/step | 14.2 samples/s
+Step 16630/44000 | Loss: 2.9833 | LR: 3.51e-04 | 1408.2ms/step | 14.2 samples/s
+Step 16640/44000 | Loss: 3.0043 | LR: 3.51e-04 | 1408.2ms/step | 14.2 samples/s
+Step 16650/44000 | Loss: 3.0141 | LR: 3.51e-04 | 1407.1ms/step | 14.2 samples/s
+Step 16660/44000 | Loss: 3.0043 | LR: 3.51e-04 | 1406.2ms/step | 14.2 samples/s
+Step 16670/44000 | Loss: 3.0042 | LR: 3.51e-04 | 1406.0ms/step | 14.2 samples/s
+Step 16680/44000 | Loss: 3.0202 | LR: 3.50e-04 | 1404.1ms/step | 14.2 samples/s
+Step 16690/44000 | Loss: 2.9715 | LR: 3.50e-04 | 1404.1ms/step | 14.2 samples/s
+Step 16700/44000 | Loss: 2.9954 | LR: 3.50e-04 | 1403.1ms/step | 14.3 samples/s
+Step 16710/44000 | Loss: 2.9902 | LR: 3.50e-04 | 1404.2ms/step | 14.2 samples/s
+Step 16720/44000 | Loss: 3.0234 | LR: 3.50e-04 | 1400.8ms/step | 14.3 samples/s
+Step 16730/44000 | Loss: 3.0029 | LR: 3.50e-04 | 1405.7ms/step | 14.2 samples/s
+Step 16740/44000 | Loss: 3.0064 | LR: 3.49e-04 | 1407.8ms/step | 14.2 samples/s
+Step 16750/44000 | Loss: 2.9984 | LR: 3.49e-04 | 1406.1ms/step | 14.2 samples/s
+Step 16760/44000 | Loss: 3.0193 | LR: 3.49e-04 | 1404.6ms/step | 14.2 samples/s
+Step 16770/44000 | Loss: 3.0108 | LR: 3.49e-04 | 1407.1ms/step | 14.2 samples/s
+Step 16780/44000 | Loss: 3.0096 | LR: 3.49e-04 | 1406.3ms/step | 14.2 samples/s
+Step 16790/44000 | Loss: 2.9972 | LR: 3.49e-04 | 1406.9ms/step | 14.2 samples/s
+Step 16800/44000 | Loss: 3.0061 | LR: 3.48e-04 | 1405.3ms/step | 14.2 samples/s
+Step 16810/44000 | Loss: 3.0070 | LR: 3.48e-04 | 1406.7ms/step | 14.2 samples/s
+Step 16820/44000 | Loss: 3.0215 | LR: 3.48e-04 | 1407.0ms/step | 14.2 samples/s
+Step 16830/44000 | Loss: 2.9923 | LR: 3.48e-04 | 1405.9ms/step | 14.2 samples/s
+Step 16840/44000 | Loss: 3.0203 | LR: 3.48e-04 | 1406.4ms/step | 14.2 samples/s
+Step 16850/44000 | Loss: 3.0243 | LR: 3.48e-04 | 1405.9ms/step | 14.2 samples/s
+Step 16860/44000 | Loss: 3.0320 | LR: 3.47e-04 | 1407.7ms/step | 14.2 samples/s
+Step 16870/44000 | Loss: 3.0149 | LR: 3.47e-04 | 1407.5ms/step | 14.2 samples/s
+Step 16880/44000 | Loss: 3.0218 | LR: 3.47e-04 | 1406.8ms/step | 14.2 samples/s
+Step 16890/44000 | Loss: 2.9908 | LR: 3.47e-04 | 1406.9ms/step | 14.2 samples/s
+Step 16900/44000 | Loss: 2.9648 | LR: 3.47e-04 | 1407.7ms/step | 14.2 samples/s
+Step 16910/44000 | Loss: 3.0129 | LR: 3.47e-04 | 1411.1ms/step | 14.2 samples/s
+Step 16920/44000 | Loss: 2.9968 | LR: 3.46e-04 | 1408.1ms/step | 14.2 samples/s
+Step 16930/44000 | Loss: 3.0167 | LR: 3.46e-04 | 1406.3ms/step | 14.2 samples/s
+Step 16940/44000 | Loss: 3.0183 | LR: 3.46e-04 | 1402.4ms/step | 14.3 samples/s
+Step 16950/44000 | Loss: 3.0071 | LR: 3.46e-04 | 1402.8ms/step | 14.3 samples/s
+Step 16960/44000 | Loss: 3.0012 | LR: 3.46e-04 | 1405.0ms/step | 14.2 samples/s
+Step 16970/44000 | Loss: 3.0226 | LR: 3.46e-04 | 1401.1ms/step | 14.3 samples/s
+Step 16980/44000 | Loss: 3.0106 | LR: 3.45e-04 | 1400.0ms/step | 14.3 samples/s
+Step 16990/44000 | Loss: 2.9798 | LR: 3.45e-04 | 1399.9ms/step | 14.3 samples/s
+Step 17000/44000 | Loss: 3.0017 | LR: 3.45e-04 | 1400.2ms/step | 14.3 samples/s
+Validation at step 17000: Loss = 3.0056, Perplexity = 20.20
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735/checkpoint-17000
+Step 17010/44000 | Loss: 3.0082 | LR: 3.45e-04 | 4014.1ms/step | 5.0 samples/s
+Step 17020/44000 | Loss: 2.9929 | LR: 3.45e-04 | 1399.8ms/step | 14.3 samples/s
+Step 17030/44000 | Loss: 2.9992 | LR: 3.45e-04 | 1401.5ms/step | 14.3 samples/s
+Step 17040/44000 | Loss: 3.0050 | LR: 3.44e-04 | 1402.9ms/step | 14.3 samples/s
+Step 17050/44000 | Loss: 3.0076 | LR: 3.44e-04 | 1402.4ms/step | 14.3 samples/s
+Step 17060/44000 | Loss: 2.9949 | LR: 3.44e-04 | 1403.8ms/step | 14.2 samples/s
+Step 17070/44000 | Loss: 3.0166 | LR: 3.44e-04 | 1402.1ms/step | 14.3 samples/s
+Step 17080/44000 | Loss: 3.0174 | LR: 3.44e-04 | 1406.3ms/step | 14.2 samples/s
+Step 17090/44000 | Loss: 3.0093 | LR: 3.44e-04 | 1406.6ms/step | 14.2 samples/s
+Step 17100/44000 | Loss: 3.0086 | LR: 3.43e-04 | 1404.8ms/step | 14.2 samples/s
+Step 17110/44000 | Loss: 2.9833 | LR: 3.43e-04 | 1406.1ms/step | 14.2 samples/s
+Step 17120/44000 | Loss: 3.0130 | LR: 3.43e-04 | 1406.0ms/step | 14.2 samples/s
+Step 17130/44000 | Loss: 3.0143 | LR: 3.43e-04 | 1406.8ms/step | 14.2 samples/s
+Step 17140/44000 | Loss: 2.9852 | LR: 3.43e-04 | 1409.3ms/step | 14.2 samples/s
+Step 17150/44000 | Loss: 2.9984 | LR: 3.43e-04 | 1408.4ms/step | 14.2 samples/s
+Step 17160/44000 | Loss: 2.9864 | LR: 3.42e-04 | 1407.1ms/step | 14.2 samples/s
+Step 17170/44000 | Loss: 3.0419 | LR: 3.42e-04 | 1406.2ms/step | 14.2 samples/s
+Step 17180/44000 | Loss: 2.9945 | LR: 3.42e-04 | 1406.3ms/step | 14.2 samples/s
+Step 17190/44000 | Loss: 3.0120 | LR: 3.42e-04 | 1406.6ms/step | 14.2 samples/s
+Step 17200/44000 | Loss: 3.0314 | LR: 3.42e-04 | 1409.0ms/step | 14.2 samples/s
+Step 17210/44000 | Loss: 3.0222 | LR: 3.42e-04 | 1407.1ms/step | 14.2 samples/s
+Step 17220/44000 | Loss: 3.0067 | LR: 3.41e-04 | 1407.3ms/step | 14.2 samples/s
+Step 17230/44000 | Loss: 2.9997 | LR: 3.41e-04 | 1408.2ms/step | 14.2 samples/s
+Step 17240/44000 | Loss: 3.0174 | LR: 3.41e-04 | 1409.3ms/step | 14.2 samples/s
+Step 17250/44000 | Loss: 2.9681 | LR: 3.41e-04 | 1406.5ms/step | 14.2 samples/s
+Step 17260/44000 | Loss: 2.9908 | LR: 3.41e-04 | 1408.4ms/step | 14.2 samples/s
+Step 17270/44000 | Loss: 2.9869 | LR: 3.41e-04 | 1409.2ms/step | 14.2 samples/s
+Step 17280/44000 | Loss: 2.9766 | LR: 3.40e-04 | 1407.6ms/step | 14.2 samples/s
+Step 17290/44000 | Loss: 2.9601 | LR: 3.40e-04 | 1408.6ms/step | 14.2 samples/s
+Step 17300/44000 | Loss: 2.9960 | LR: 3.40e-04 | 1408.3ms/step | 14.2 samples/s
+Step 17310/44000 | Loss: 3.0084 | LR: 3.40e-04 | 1409.2ms/step | 14.2 samples/s
+Step 17320/44000 | Loss: 3.0018 | LR: 3.40e-04 | 1409.6ms/step | 14.2 samples/s
+Step 17330/44000 | Loss: 2.9974 | LR: 3.40e-04 | 1410.4ms/step | 14.2 samples/s
+Step 17340/44000 | Loss: 3.0035 | LR: 3.39e-04 | 1408.6ms/step | 14.2 samples/s
+Step 17350/44000 | Loss: 3.0107 | LR: 3.39e-04 | 1409.7ms/step | 14.2 samples/s
+Step 17360/44000 | Loss: 3.0203 | LR: 3.39e-04 | 1410.7ms/step | 14.2 samples/s
+Step 17370/44000 | Loss: 3.0077 | LR: 3.39e-04 | 1407.5ms/step | 14.2 samples/s
+Step 17380/44000 | Loss: 3.0024 | LR: 3.39e-04 | 1408.4ms/step | 14.2 samples/s
+Step 17390/44000 | Loss: 2.9851 | LR: 3.39e-04 | 1406.1ms/step | 14.2 samples/s
+Step 17400/44000 | Loss: 3.0201 | LR: 3.38e-04 | 1408.2ms/step | 14.2 samples/s
+Step 17410/44000 | Loss: 2.9925 | LR: 3.38e-04 | 1409.0ms/step | 14.2 samples/s
+Step 17420/44000 | Loss: 2.9969 | LR: 3.38e-04 | 1408.1ms/step | 14.2 samples/s
+Step 17430/44000 | Loss: 2.9662 | LR: 3.38e-04 | 1403.5ms/step | 14.3 samples/s
+Step 17440/44000 | Loss: 2.9928 | LR: 3.38e-04 | 1408.4ms/step | 14.2 samples/s
+Step 17450/44000 | Loss: 2.9656 | LR: 3.38e-04 | 1407.9ms/step | 14.2 samples/s

epfl_final/train_logs/ropectl_a1p0_h100_resume/seed46.20260820T041452Z.log

@@ -0,0 +1,653 @@
+Auto-resuming from latest checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735/checkpoint-13000
+Resuming: Using existing output directory /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735
+Running config:
+{
+  "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
+  "model_dtype": "bfloat16",
+  "init_from_scratch": true,
+  "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
+  "val_path": null,
+  "max_seq_length": 8192,
+  "num_workers": 8,
+  "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735",
+  "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735/checkpoint-13000",
+  "force_new_run": false,
+  "num_epochs": 1,
+  "max_steps": 44000,
+  "per_device_batch_size": 2,
+  "gradient_accumulation_steps": 10,
+  "learning_rate": 0.0005,
+  "weight_decay": 0.01,
+  "warmup_steps": 750,
+  "lr_scheduler": "cosine",
+  "max_grad_norm": 1.0,
+  "save_steps": 1000,
+  "eval_steps": 1000,
+  "eval_batches": 50,
+  "logging_steps": 10,
+  "seed": 46,
+  "verbose": true,
+  "checkpoint_activations": false,
+  "strategy": "ddp",
+  "mixed_precision": "bf16",
+  "wandb_enabled": true,
+  "wandb_project": "stochastic_sparsity",
+  "wandb_name": "l8k365m_dense_ropectl_a1p0_seed46",
+  "wandb_tags": [
+    "qwen3",
+    "365m",
+    "8k",
+    "dense",
+    "rope-control"
+  ],
+  "hidden_size": 768,
+  "intermediate_size": 3072,
+  "num_attention_heads": 6,
+  "num_key_value_heads": 2,
+  "num_hidden_layers": 28,
+  "head_dim": null,
+  "attention_dropout": 0.0,
+  "rope_train_position_scale_set": [
+    1.0,
+    0.9896339488176222,
+    0.9692159692159692,
+    0.9420289855072463,
+    0.8730158730158729
+  ],
+  "attention_type": "sparse_stochastic",
+  "sparse_block_size": 128,
+  "sparse_num_random_blocks": 31,
+  "sparse_mask_pool_size": 1,
+  "sparse_val_dense_compare": false,
+  "sparse_layer_indices": null,
+  "sparse_mask_cache_dir": null,
+  "sparse_mask_seed": 0,
+  "sparse_mixed_nrb_set": null,
+  "dense_mode": true,
+  "dense_step_intervals": null
+}
+Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
+Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735/checkpoint-13000
+Checkpoint activations: False
+dense_mode=True: sparse attention layers will use standard causal SDPA.
+
+
+================================================================================
+MODEL ARCHITECTURE
+================================================================================
+Model config:
+  vocab_size: 151936
+  hidden_size: 768
+  num_hidden_layers: 28
+  num_attention_heads: 6
+  num_key_value_heads: 2
+  head_dim: 128
+  intermediate_size: 3072
+  max_position_embeddings: 32768
+  rope_theta: 1000000
+  rope_train_position_scale_set: [1.0, 0.9896339488176222, 0.9692159692159692, 0.9420289855072463, 0.8730158730158729]
+  attention_type: sparse_stochastic
+  sparse_block_size: 128
+  sparse_num_random_blocks: 31
+  sparse_mask_pool_size: 1
+  sparse_max_seq_length: 8192
+
+Parameter count:
+  Total parameters: 358,958,848
+  Trainable parameters: 358,958,848
+  Non-trainable parameters: 0
+  Model size (BF16): 0.72 GB
+
+--------------------------------------------------------------------------------
+LAYER-BY-LAYER SUMMARY
+--------------------------------------------------------------------------------
+================================================================================================================================================================
+Layer (type:depth-idx)                                       Input Shape               Output Shape              Param #                   Trainable
+================================================================================================================================================================
+Qwen3ForCausalLM                                             [2, 8192]                 --                        --                        True
+├─Qwen3Model: 1-1                                            --                        [2, 8192, 768]            --                        True
+│    └─Embedding: 2-1                                        [2, 8192]                 [2, 8192, 768]            116,686,848               True
+│    └─ModuleList: 2-2                                       --                        --                        --                        True
+│    │    └─Qwen3DecoderLayer: 3-1                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-2                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-3                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-4                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-5                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-6                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-7                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-8                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-9                           [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-10                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-11                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-12                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-13                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-14                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-15                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-16                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-17                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-18                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-19                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-20                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-21                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-22                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-23                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-24                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-25                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-26                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-27                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-28                          [2, 8192, 768]            [2, 8192, 768]            8,652,544                 True
+│    └─Qwen3RMSNorm: 2-3                                     [2, 8192, 768]            [2, 8192, 768]            768                       True
+├─Linear: 1-2                                                [2, 8192, 768]            [2, 8192, 151936]         116,686,848               True
+================================================================================================================================================================
+Total params: 475,645,696
+Trainable params: 475,645,696
+Non-trainable params: 0
+Total mult-adds (Units.MEGABYTES): 951.29
+================================================================================================================================================================
+Input size (MB): 0.13
+Forward/backward pass size (MB): 31197.23
+Params size (MB): 951.29
+Estimated Total Size (MB): 32148.66
+================================================================================================================================================================
+  NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
+================================================================================
+
+Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
+Using pre-tokenized dataset (no runtime tokenization)
+Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
+Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
+Dataset split: 3027764 train, 159356 validation
+Training examples (packed, train split):  3027764
+Validation examples (packed, val split):  159356
+Per-device batch size:                    2
+World size (GPUs):                        1
+Gradient accumulation steps:              10
+Effective batch size (seqs/step):         20
+Sequence length:                          8192
+Tokens per optimizer step:                163,840
+Micro-batches per epoch (per rank):       1513882
+Optimizer steps per epoch:                151388
+Total optimizer steps:                    44000  (capped by max_steps; 0.29 epochs)
+Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735/checkpoint-13000...
+Preserving checkpoint LR state for exact resume: current=[0.0004073888929652048], schedule_base=[0.0005]
+Resumed from checkpoint at step 13000 (epoch 0, micro_step 130000)
+Resuming training from step 13000
+Resuming wandb run zovy63wq
+wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from WANDB_API_KEY.
+wandb: Currently logged in as: ruben-navasardyan4869 (ruben-navasardyan4869-epfl) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
+wandb: setting up run zovy63wq
+wandb: Tracking run with wandb version 0.26.1
+wandb: Run data is saved locally in /mloscratch/homes/navasard/sparse/SparseAtt/wandb/run-20260820_041559-zovy63wq
+wandb: Run `wandb offline` to turn off syncing.
+wandb: Resuming run l8k365m_dense_ropectl_a1p0_seed46
+wandb: ⭐️ View project at https://wandb.ai/ruben-navasardyan4869-epfl/stochastic_sparsity
+wandb: 🚀 View run at https://wandb.ai/ruben-navasardyan4869-epfl/stochastic_sparsity/runs/zovy63wq
+Starting training...
+Validation at step 13000: Loss = 3.0598, Perplexity = 21.32
+wandb: WARNING Tried to log to step 13000 that is less than the current step 13051. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Switching attention mode for step 13001: dense (dense step interval)
+Step 13010/44000 | Loss: 3.0481 | LR: 4.07e-04 | 30389.3ms/step | 0.7 samples/s
+wandb: WARNING Tried to log to step 13010 that is less than the current step 13051. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13020/44000 | Loss: 3.0472 | LR: 4.07e-04 | 1376.9ms/step | 14.5 samples/s
+wandb: WARNING Tried to log to step 13020 that is less than the current step 13051. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13030/44000 | Loss: 3.0740 | LR: 4.07e-04 | 1380.9ms/step | 14.5 samples/s
+wandb: WARNING Tried to log to step 13030 that is less than the current step 13051. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13040/44000 | Loss: 3.0868 | LR: 4.07e-04 | 1375.5ms/step | 14.5 samples/s
+wandb: WARNING Tried to log to step 13040 that is less than the current step 13051. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13050/44000 | Loss: 3.0517 | LR: 4.07e-04 | 1374.0ms/step | 14.6 samples/s
+wandb: WARNING Tried to log to step 13050 that is less than the current step 13051. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 13060/44000 | Loss: 3.0409 | LR: 4.07e-04 | 1377.3ms/step | 14.5 samples/s
+Step 13070/44000 | Loss: 3.0491 | LR: 4.06e-04 | 1378.9ms/step | 14.5 samples/s
+Step 13080/44000 | Loss: 3.0805 | LR: 4.06e-04 | 1379.0ms/step | 14.5 samples/s
+Step 13090/44000 | Loss: 3.0620 | LR: 4.06e-04 | 1378.4ms/step | 14.5 samples/s
+Step 13100/44000 | Loss: 3.0623 | LR: 4.06e-04 | 1379.6ms/step | 14.5 samples/s
+Step 13110/44000 | Loss: 3.0679 | LR: 4.06e-04 | 1376.4ms/step | 14.5 samples/s
+Step 13120/44000 | Loss: 3.0415 | LR: 4.06e-04 | 1378.6ms/step | 14.5 samples/s
+Step 13130/44000 | Loss: 3.0552 | LR: 4.06e-04 | 1377.3ms/step | 14.5 samples/s
+Step 13140/44000 | Loss: 3.0384 | LR: 4.05e-04 | 1376.3ms/step | 14.5 samples/s
+Step 13150/44000 | Loss: 3.0496 | LR: 4.05e-04 | 1379.6ms/step | 14.5 samples/s
+Step 13160/44000 | Loss: 3.0589 | LR: 4.05e-04 | 1380.1ms/step | 14.5 samples/s
+Step 13170/44000 | Loss: 3.0586 | LR: 4.05e-04 | 1377.6ms/step | 14.5 samples/s
+Step 13180/44000 | Loss: 3.0326 | LR: 4.05e-04 | 1377.6ms/step | 14.5 samples/s
+Step 13190/44000 | Loss: 3.0617 | LR: 4.05e-04 | 1378.9ms/step | 14.5 samples/s
+Step 13200/44000 | Loss: 3.0821 | LR: 4.05e-04 | 1380.6ms/step | 14.5 samples/s
+Step 13210/44000 | Loss: 3.0834 | LR: 4.04e-04 | 1376.7ms/step | 14.5 samples/s
+Step 13220/44000 | Loss: 3.0198 | LR: 4.04e-04 | 1376.4ms/step | 14.5 samples/s
+Step 13230/44000 | Loss: 3.0559 | LR: 4.04e-04 | 1378.8ms/step | 14.5 samples/s
+Step 13240/44000 | Loss: 3.0391 | LR: 4.04e-04 | 1379.2ms/step | 14.5 samples/s
+Step 13250/44000 | Loss: 3.0350 | LR: 4.04e-04 | 1378.3ms/step | 14.5 samples/s
+Step 13260/44000 | Loss: 3.0151 | LR: 4.04e-04 | 1376.3ms/step | 14.5 samples/s
+Step 13270/44000 | Loss: 3.0577 | LR: 4.04e-04 | 1384.5ms/step | 14.4 samples/s
+Step 13280/44000 | Loss: 3.0382 | LR: 4.03e-04 | 1384.1ms/step | 14.5 samples/s
+Step 13290/44000 | Loss: 3.0494 | LR: 4.03e-04 | 1383.8ms/step | 14.5 samples/s
+Step 13300/44000 | Loss: 3.0422 | LR: 4.03e-04 | 1384.0ms/step | 14.5 samples/s
+Step 13310/44000 | Loss: 3.0291 | LR: 4.03e-04 | 1385.1ms/step | 14.4 samples/s
+Step 13320/44000 | Loss: 3.0818 | LR: 4.03e-04 | 1383.8ms/step | 14.5 samples/s
+Step 13330/44000 | Loss: 3.0389 | LR: 4.03e-04 | 1381.9ms/step | 14.5 samples/s
+Step 13340/44000 | Loss: 3.0393 | LR: 4.03e-04 | 1381.5ms/step | 14.5 samples/s
+Step 13350/44000 | Loss: 3.0494 | LR: 4.02e-04 | 1378.8ms/step | 14.5 samples/s
+Step 13360/44000 | Loss: 2.9992 | LR: 4.02e-04 | 1379.8ms/step | 14.5 samples/s
+Step 13370/44000 | Loss: 3.0336 | LR: 4.02e-04 | 1377.1ms/step | 14.5 samples/s
+Step 13380/44000 | Loss: 3.0688 | LR: 4.02e-04 | 1377.2ms/step | 14.5 samples/s
+Step 13390/44000 | Loss: 3.0480 | LR: 4.02e-04 | 1377.6ms/step | 14.5 samples/s
+Step 13400/44000 | Loss: 3.0450 | LR: 4.02e-04 | 1378.3ms/step | 14.5 samples/s
+Step 13410/44000 | Loss: 3.0689 | LR: 4.02e-04 | 1376.3ms/step | 14.5 samples/s
+Step 13420/44000 | Loss: 3.0442 | LR: 4.01e-04 | 1377.1ms/step | 14.5 samples/s
+Step 13430/44000 | Loss: 3.0597 | LR: 4.01e-04 | 1378.5ms/step | 14.5 samples/s
+Step 13440/44000 | Loss: 3.0440 | LR: 4.01e-04 | 1379.3ms/step | 14.5 samples/s
+Step 13450/44000 | Loss: 3.0382 | LR: 4.01e-04 | 1375.8ms/step | 14.5 samples/s
+Step 13460/44000 | Loss: 3.0512 | LR: 4.01e-04 | 1377.6ms/step | 14.5 samples/s
+Step 13470/44000 | Loss: 3.0468 | LR: 4.01e-04 | 1379.4ms/step | 14.5 samples/s
+Step 13480/44000 | Loss: 3.0806 | LR: 4.01e-04 | 1378.4ms/step | 14.5 samples/s
+Step 13490/44000 | Loss: 3.0723 | LR: 4.00e-04 | 1375.8ms/step | 14.5 samples/s
+Step 13500/44000 | Loss: 3.0526 | LR: 4.00e-04 | 1377.0ms/step | 14.5 samples/s
+Step 13510/44000 | Loss: 3.0405 | LR: 4.00e-04 | 1377.5ms/step | 14.5 samples/s
+Step 13520/44000 | Loss: 3.0453 | LR: 4.00e-04 | 1379.3ms/step | 14.5 samples/s
+Step 13530/44000 | Loss: 3.0552 | LR: 4.00e-04 | 1378.0ms/step | 14.5 samples/s
+Step 13540/44000 | Loss: 3.0549 | LR: 4.00e-04 | 1379.1ms/step | 14.5 samples/s
+Step 13550/44000 | Loss: 3.0258 | LR: 4.00e-04 | 1380.2ms/step | 14.5 samples/s
+Step 13560/44000 | Loss: 3.0542 | LR: 3.99e-04 | 1379.2ms/step | 14.5 samples/s
+Step 13570/44000 | Loss: 3.0473 | LR: 3.99e-04 | 1376.8ms/step | 14.5 samples/s
+Step 13580/44000 | Loss: 3.0352 | LR: 3.99e-04 | 1376.5ms/step | 14.5 samples/s
+Step 13590/44000 | Loss: 3.0440 | LR: 3.99e-04 | 1380.5ms/step | 14.5 samples/s
+Step 13600/44000 | Loss: 3.0300 | LR: 3.99e-04 | 1381.3ms/step | 14.5 samples/s
+Step 13610/44000 | Loss: 3.0538 | LR: 3.99e-04 | 1380.1ms/step | 14.5 samples/s
+Step 13620/44000 | Loss: 3.0784 | LR: 3.98e-04 | 1378.8ms/step | 14.5 samples/s
+Step 13630/44000 | Loss: 3.0545 | LR: 3.98e-04 | 1382.0ms/step | 14.5 samples/s
+Step 13640/44000 | Loss: 3.0056 | LR: 3.98e-04 | 1381.7ms/step | 14.5 samples/s
+Step 13650/44000 | Loss: 3.0601 | LR: 3.98e-04 | 1378.9ms/step | 14.5 samples/s
+Step 13660/44000 | Loss: 3.0521 | LR: 3.98e-04 | 1378.3ms/step | 14.5 samples/s
+Step 13670/44000 | Loss: 3.0288 | LR: 3.98e-04 | 1380.8ms/step | 14.5 samples/s
+Step 13680/44000 | Loss: 3.0444 | LR: 3.98e-04 | 1379.5ms/step | 14.5 samples/s
+Step 13690/44000 | Loss: 3.0248 | LR: 3.97e-04 | 1378.0ms/step | 14.5 samples/s
+Step 13700/44000 | Loss: 3.0686 | LR: 3.97e-04 | 1376.4ms/step | 14.5 samples/s
+Step 13710/44000 | Loss: 3.0330 | LR: 3.97e-04 | 1378.9ms/step | 14.5 samples/s
+Step 13720/44000 | Loss: 3.0380 | LR: 3.97e-04 | 1377.1ms/step | 14.5 samples/s
+Step 13730/44000 | Loss: 2.9996 | LR: 3.97e-04 | 1377.7ms/step | 14.5 samples/s
+Step 13740/44000 | Loss: 3.0181 | LR: 3.97e-04 | 1378.0ms/step | 14.5 samples/s
+Step 13750/44000 | Loss: 3.0539 | LR: 3.97e-04 | 1379.1ms/step | 14.5 samples/s
+Step 13760/44000 | Loss: 3.0448 | LR: 3.96e-04 | 1378.6ms/step | 14.5 samples/s
+Step 13770/44000 | Loss: 3.0532 | LR: 3.96e-04 | 1378.2ms/step | 14.5 samples/s
+Step 13780/44000 | Loss: 3.0652 | LR: 3.96e-04 | 1376.1ms/step | 14.5 samples/s
+Step 13790/44000 | Loss: 3.0325 | LR: 3.96e-04 | 1378.2ms/step | 14.5 samples/s
+Step 13800/44000 | Loss: 3.0507 | LR: 3.96e-04 | 1377.3ms/step | 14.5 samples/s
+Step 13810/44000 | Loss: 3.0212 | LR: 3.96e-04 | 1376.1ms/step | 14.5 samples/s
+Step 13820/44000 | Loss: 3.0419 | LR: 3.96e-04 | 1377.8ms/step | 14.5 samples/s
+Step 13830/44000 | Loss: 3.0317 | LR: 3.95e-04 | 1376.6ms/step | 14.5 samples/s
+Step 13840/44000 | Loss: 3.0505 | LR: 3.95e-04 | 1376.4ms/step | 14.5 samples/s
+Step 13850/44000 | Loss: 3.0452 | LR: 3.95e-04 | 1376.3ms/step | 14.5 samples/s
+Step 13860/44000 | Loss: 3.0308 | LR: 3.95e-04 | 1381.0ms/step | 14.5 samples/s
+Step 13870/44000 | Loss: 3.0502 | LR: 3.95e-04 | 1380.3ms/step | 14.5 samples/s
+Step 13880/44000 | Loss: 3.0730 | LR: 3.95e-04 | 1375.8ms/step | 14.5 samples/s
+Step 13890/44000 | Loss: 3.0543 | LR: 3.95e-04 | 1377.9ms/step | 14.5 samples/s
+Step 13900/44000 | Loss: 3.0624 | LR: 3.94e-04 | 1377.4ms/step | 14.5 samples/s
+Step 13910/44000 | Loss: 3.0401 | LR: 3.94e-04 | 1377.8ms/step | 14.5 samples/s
+Step 13920/44000 | Loss: 3.0559 | LR: 3.94e-04 | 1376.7ms/step | 14.5 samples/s
+Step 13930/44000 | Loss: 3.0415 | LR: 3.94e-04 | 1377.2ms/step | 14.5 samples/s
+Step 13940/44000 | Loss: 3.0456 | LR: 3.94e-04 | 1377.8ms/step | 14.5 samples/s
+Step 13950/44000 | Loss: 3.0764 | LR: 3.94e-04 | 1381.1ms/step | 14.5 samples/s
+Step 13960/44000 | Loss: 3.0517 | LR: 3.93e-04 | 1379.6ms/step | 14.5 samples/s
+Step 13970/44000 | Loss: 3.0275 | LR: 3.93e-04 | 1378.0ms/step | 14.5 samples/s
+Step 13980/44000 | Loss: 3.0508 | LR: 3.93e-04 | 1378.6ms/step | 14.5 samples/s
+Step 13990/44000 | Loss: 3.0339 | LR: 3.93e-04 | 1379.9ms/step | 14.5 samples/s
+Step 14000/44000 | Loss: 3.0430 | LR: 3.93e-04 | 1376.7ms/step | 14.5 samples/s
+Validation at step 14000: Loss = 3.0458, Perplexity = 21.03
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735/checkpoint-14000
+Step 14010/44000 | Loss: 3.0636 | LR: 3.93e-04 | 3907.6ms/step | 5.1 samples/s
+Step 14020/44000 | Loss: 3.0477 | LR: 3.93e-04 | 1390.0ms/step | 14.4 samples/s
+Step 14030/44000 | Loss: 3.0135 | LR: 3.92e-04 | 1390.0ms/step | 14.4 samples/s
+Step 14040/44000 | Loss: 3.0325 | LR: 3.92e-04 | 1389.8ms/step | 14.4 samples/s
+Step 14050/44000 | Loss: 3.0606 | LR: 3.92e-04 | 1393.8ms/step | 14.3 samples/s
+Step 14060/44000 | Loss: 3.0371 | LR: 3.92e-04 | 1396.4ms/step | 14.3 samples/s
+Step 14070/44000 | Loss: 3.0377 | LR: 3.92e-04 | 1398.0ms/step | 14.3 samples/s
+Step 14080/44000 | Loss: 3.0223 | LR: 3.92e-04 | 1396.2ms/step | 14.3 samples/s
+Step 14090/44000 | Loss: 3.0452 | LR: 3.92e-04 | 1393.4ms/step | 14.4 samples/s
+Step 14100/44000 | Loss: 3.0447 | LR: 3.91e-04 | 1393.9ms/step | 14.3 samples/s
+Step 14110/44000 | Loss: 3.0483 | LR: 3.91e-04 | 1394.4ms/step | 14.3 samples/s
+Step 14120/44000 | Loss: 3.0587 | LR: 3.91e-04 | 1395.8ms/step | 14.3 samples/s
+Step 14130/44000 | Loss: 3.0422 | LR: 3.91e-04 | 1394.2ms/step | 14.3 samples/s
+Step 14140/44000 | Loss: 3.0681 | LR: 3.91e-04 | 1397.8ms/step | 14.3 samples/s
+Step 14150/44000 | Loss: 3.0426 | LR: 3.91e-04 | 1386.5ms/step | 14.4 samples/s
+Step 14160/44000 | Loss: 3.0381 | LR: 3.90e-04 | 1382.2ms/step | 14.5 samples/s
+Step 14170/44000 | Loss: 3.0104 | LR: 3.90e-04 | 1382.5ms/step | 14.5 samples/s
+Step 14180/44000 | Loss: 3.0441 | LR: 3.90e-04 | 1382.3ms/step | 14.5 samples/s
+Step 14190/44000 | Loss: 3.0137 | LR: 3.90e-04 | 1380.7ms/step | 14.5 samples/s
+Step 14200/44000 | Loss: 3.0176 | LR: 3.90e-04 | 1381.7ms/step | 14.5 samples/s
+Step 14210/44000 | Loss: 3.0291 | LR: 3.90e-04 | 1383.4ms/step | 14.5 samples/s
+Step 14220/44000 | Loss: 3.0395 | LR: 3.90e-04 | 1381.0ms/step | 14.5 samples/s
+Step 14230/44000 | Loss: 3.0509 | LR: 3.89e-04 | 1380.3ms/step | 14.5 samples/s
+Step 14240/44000 | Loss: 3.0615 | LR: 3.89e-04 | 1379.5ms/step | 14.5 samples/s
+Step 14250/44000 | Loss: 3.0315 | LR: 3.89e-04 | 1380.4ms/step | 14.5 samples/s
+Step 14260/44000 | Loss: 3.0270 | LR: 3.89e-04 | 1383.1ms/step | 14.5 samples/s
+Step 14270/44000 | Loss: 3.0274 | LR: 3.89e-04 | 1383.9ms/step | 14.5 samples/s
+Step 14280/44000 | Loss: 3.0264 | LR: 3.89e-04 | 1381.2ms/step | 14.5 samples/s
+Step 14290/44000 | Loss: 3.0398 | LR: 3.89e-04 | 1380.7ms/step | 14.5 samples/s
+Step 14300/44000 | Loss: 3.0369 | LR: 3.88e-04 | 1385.3ms/step | 14.4 samples/s
+Step 14310/44000 | Loss: 3.0450 | LR: 3.88e-04 | 1383.8ms/step | 14.5 samples/s
+Step 14320/44000 | Loss: 3.0471 | LR: 3.88e-04 | 1382.7ms/step | 14.5 samples/s
+Step 14330/44000 | Loss: 3.0381 | LR: 3.88e-04 | 1381.8ms/step | 14.5 samples/s
+Step 14340/44000 | Loss: 3.0571 | LR: 3.88e-04 | 1381.5ms/step | 14.5 samples/s
+Step 14350/44000 | Loss: 3.0220 | LR: 3.88e-04 | 1383.6ms/step | 14.5 samples/s
+Step 14360/44000 | Loss: 3.0370 | LR: 3.87e-04 | 1383.9ms/step | 14.5 samples/s
+Step 14370/44000 | Loss: 3.0404 | LR: 3.87e-04 | 1384.0ms/step | 14.5 samples/s
+Step 14380/44000 | Loss: 3.0496 | LR: 3.87e-04 | 1382.5ms/step | 14.5 samples/s
+Step 14390/44000 | Loss: 3.0281 | LR: 3.87e-04 | 1383.2ms/step | 14.5 samples/s
+Step 14400/44000 | Loss: 3.0186 | LR: 3.87e-04 | 1385.0ms/step | 14.4 samples/s
+Step 14410/44000 | Loss: 3.0406 | LR: 3.87e-04 | 1378.9ms/step | 14.5 samples/s
+Step 14420/44000 | Loss: 3.0558 | LR: 3.87e-04 | 1379.1ms/step | 14.5 samples/s
+Step 14430/44000 | Loss: 3.0387 | LR: 3.86e-04 | 1383.4ms/step | 14.5 samples/s
+Step 14440/44000 | Loss: 3.0316 | LR: 3.86e-04 | 1382.2ms/step | 14.5 samples/s
+Step 14450/44000 | Loss: 3.0545 | LR: 3.86e-04 | 1382.6ms/step | 14.5 samples/s
+Step 14460/44000 | Loss: 3.0381 | LR: 3.86e-04 | 1384.0ms/step | 14.5 samples/s
+Step 14470/44000 | Loss: 3.0713 | LR: 3.86e-04 | 1381.9ms/step | 14.5 samples/s
+Step 14480/44000 | Loss: 3.0358 | LR: 3.86e-04 | 1382.3ms/step | 14.5 samples/s
+Step 14490/44000 | Loss: 3.0262 | LR: 3.85e-04 | 1382.9ms/step | 14.5 samples/s
+Step 14500/44000 | Loss: 3.0348 | LR: 3.85e-04 | 1384.2ms/step | 14.4 samples/s
+Step 14510/44000 | Loss: 3.0211 | LR: 3.85e-04 | 1382.2ms/step | 14.5 samples/s
+Step 14520/44000 | Loss: 3.0274 | LR: 3.85e-04 | 1380.5ms/step | 14.5 samples/s
+Step 14530/44000 | Loss: 3.0248 | LR: 3.85e-04 | 1381.3ms/step | 14.5 samples/s
+Step 14540/44000 | Loss: 3.0323 | LR: 3.85e-04 | 1381.5ms/step | 14.5 samples/s
+Step 14550/44000 | Loss: 3.0271 | LR: 3.85e-04 | 1382.8ms/step | 14.5 samples/s
+Step 14560/44000 | Loss: 3.0052 | LR: 3.84e-04 | 1383.2ms/step | 14.5 samples/s
+Step 14570/44000 | Loss: 3.0016 | LR: 3.84e-04 | 1381.1ms/step | 14.5 samples/s
+Step 14580/44000 | Loss: 3.0015 | LR: 3.84e-04 | 1383.1ms/step | 14.5 samples/s
+Step 14590/44000 | Loss: 3.0262 | LR: 3.84e-04 | 1381.2ms/step | 14.5 samples/s
+Step 14600/44000 | Loss: 3.0262 | LR: 3.84e-04 | 1381.1ms/step | 14.5 samples/s
+Step 14610/44000 | Loss: 3.0369 | LR: 3.84e-04 | 1381.3ms/step | 14.5 samples/s
+Step 14620/44000 | Loss: 3.0198 | LR: 3.83e-04 | 1383.5ms/step | 14.5 samples/s
+Step 14630/44000 | Loss: 3.0277 | LR: 3.83e-04 | 1381.6ms/step | 14.5 samples/s
+Step 14640/44000 | Loss: 3.0424 | LR: 3.83e-04 | 1382.7ms/step | 14.5 samples/s
+Step 14650/44000 | Loss: 3.0167 | LR: 3.83e-04 | 1386.3ms/step | 14.4 samples/s
+Step 14660/44000 | Loss: 3.0428 | LR: 3.83e-04 | 1382.3ms/step | 14.5 samples/s
+Step 14670/44000 | Loss: 3.0339 | LR: 3.83e-04 | 1383.4ms/step | 14.5 samples/s
+Step 14680/44000 | Loss: 3.0237 | LR: 3.83e-04 | 1383.9ms/step | 14.5 samples/s
+Step 14690/44000 | Loss: 3.0304 | LR: 3.82e-04 | 1382.4ms/step | 14.5 samples/s
+Step 14700/44000 | Loss: 3.0316 | LR: 3.82e-04 | 1382.5ms/step | 14.5 samples/s
+Step 14710/44000 | Loss: 3.0364 | LR: 3.82e-04 | 1383.0ms/step | 14.5 samples/s
+Step 14720/44000 | Loss: 3.0285 | LR: 3.82e-04 | 1383.0ms/step | 14.5 samples/s
+Step 14730/44000 | Loss: 3.0419 | LR: 3.82e-04 | 1383.7ms/step | 14.5 samples/s
+Step 14740/44000 | Loss: 3.0382 | LR: 3.82e-04 | 1383.9ms/step | 14.5 samples/s
+Step 14750/44000 | Loss: 2.9881 | LR: 3.81e-04 | 1383.9ms/step | 14.5 samples/s
+Step 14760/44000 | Loss: 3.0553 | LR: 3.81e-04 | 1380.1ms/step | 14.5 samples/s
+Step 14770/44000 | Loss: 3.0577 | LR: 3.81e-04 | 1382.1ms/step | 14.5 samples/s
+Step 14780/44000 | Loss: 3.0383 | LR: 3.81e-04 | 1381.9ms/step | 14.5 samples/s
+Step 14790/44000 | Loss: 3.0158 | LR: 3.81e-04 | 1382.7ms/step | 14.5 samples/s
+Step 14800/44000 | Loss: 3.0125 | LR: 3.81e-04 | 1383.1ms/step | 14.5 samples/s
+Step 14810/44000 | Loss: 3.0505 | LR: 3.81e-04 | 1384.5ms/step | 14.4 samples/s
+Step 14820/44000 | Loss: 3.0259 | LR: 3.80e-04 | 1382.8ms/step | 14.5 samples/s
+Step 14830/44000 | Loss: 3.0332 | LR: 3.80e-04 | 1383.2ms/step | 14.5 samples/s
+Step 14840/44000 | Loss: 3.0196 | LR: 3.80e-04 | 1383.7ms/step | 14.5 samples/s
+Step 14850/44000 | Loss: 3.0481 | LR: 3.80e-04 | 1380.0ms/step | 14.5 samples/s
+Step 14860/44000 | Loss: 3.0521 | LR: 3.80e-04 | 1381.9ms/step | 14.5 samples/s
+Step 14870/44000 | Loss: 3.0343 | LR: 3.80e-04 | 1383.7ms/step | 14.5 samples/s
+Step 14880/44000 | Loss: 3.0264 | LR: 3.79e-04 | 1381.9ms/step | 14.5 samples/s
+Step 14890/44000 | Loss: 3.0287 | LR: 3.79e-04 | 1382.4ms/step | 14.5 samples/s
+Step 14900/44000 | Loss: 3.0374 | LR: 3.79e-04 | 1382.5ms/step | 14.5 samples/s
+Step 14910/44000 | Loss: 3.0206 | LR: 3.79e-04 | 1380.1ms/step | 14.5 samples/s
+Step 14920/44000 | Loss: 3.0437 | LR: 3.79e-04 | 1382.1ms/step | 14.5 samples/s
+Step 14930/44000 | Loss: 3.0468 | LR: 3.79e-04 | 1383.9ms/step | 14.5 samples/s
+Step 14940/44000 | Loss: 3.0035 | LR: 3.79e-04 | 1383.5ms/step | 14.5 samples/s
+Step 14950/44000 | Loss: 3.0613 | LR: 3.78e-04 | 1383.9ms/step | 14.5 samples/s
+Step 14960/44000 | Loss: 3.0190 | LR: 3.78e-04 | 1380.5ms/step | 14.5 samples/s
+Step 14970/44000 | Loss: 3.0510 | LR: 3.78e-04 | 1380.8ms/step | 14.5 samples/s
+Step 14980/44000 | Loss: 3.0112 | LR: 3.78e-04 | 1380.9ms/step | 14.5 samples/s
+Step 14990/44000 | Loss: 3.0359 | LR: 3.78e-04 | 1383.6ms/step | 14.5 samples/s
+Step 15000/44000 | Loss: 3.0024 | LR: 3.78e-04 | 1382.7ms/step | 14.5 samples/s
+Validation at step 15000: Loss = 3.0311, Perplexity = 20.72
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735/checkpoint-15000
+Step 15010/44000 | Loss: 3.0154 | LR: 3.77e-04 | 3905.7ms/step | 5.1 samples/s
+Step 15020/44000 | Loss: 3.0459 | LR: 3.77e-04 | 1389.1ms/step | 14.4 samples/s
+Step 15030/44000 | Loss: 3.0015 | LR: 3.77e-04 | 1387.0ms/step | 14.4 samples/s
+Step 15040/44000 | Loss: 3.0281 | LR: 3.77e-04 | 1395.4ms/step | 14.3 samples/s
+Step 15050/44000 | Loss: 3.0318 | LR: 3.77e-04 | 1387.9ms/step | 14.4 samples/s
+Step 15060/44000 | Loss: 3.0187 | LR: 3.77e-04 | 1388.2ms/step | 14.4 samples/s
+Step 15070/44000 | Loss: 3.0174 | LR: 3.77e-04 | 1392.2ms/step | 14.4 samples/s
+Step 15080/44000 | Loss: 3.0162 | LR: 3.76e-04 | 1393.8ms/step | 14.3 samples/s
+Step 15090/44000 | Loss: 3.0184 | LR: 3.76e-04 | 1393.9ms/step | 14.3 samples/s
+Step 15100/44000 | Loss: 3.0379 | LR: 3.76e-04 | 1395.5ms/step | 14.3 samples/s
+Step 15110/44000 | Loss: 3.0052 | LR: 3.76e-04 | 1396.6ms/step | 14.3 samples/s
+Step 15120/44000 | Loss: 3.0370 | LR: 3.76e-04 | 1394.0ms/step | 14.3 samples/s
+Step 15130/44000 | Loss: 3.0509 | LR: 3.76e-04 | 1398.1ms/step | 14.3 samples/s
+Step 15140/44000 | Loss: 3.0366 | LR: 3.75e-04 | 1393.8ms/step | 14.3 samples/s
+Step 15150/44000 | Loss: 3.0408 | LR: 3.75e-04 | 1397.1ms/step | 14.3 samples/s
+Step 15160/44000 | Loss: 3.0499 | LR: 3.75e-04 | 1398.9ms/step | 14.3 samples/s
+Step 15170/44000 | Loss: 3.0475 | LR: 3.75e-04 | 1395.9ms/step | 14.3 samples/s
+Step 15180/44000 | Loss: 3.0006 | LR: 3.75e-04 | 1399.6ms/step | 14.3 samples/s
+Step 15190/44000 | Loss: 3.0402 | LR: 3.75e-04 | 1394.7ms/step | 14.3 samples/s
+Step 15200/44000 | Loss: 3.0069 | LR: 3.74e-04 | 1400.4ms/step | 14.3 samples/s
+Step 15210/44000 | Loss: 2.9984 | LR: 3.74e-04 | 1397.4ms/step | 14.3 samples/s
+Step 15220/44000 | Loss: 3.0251 | LR: 3.74e-04 | 1394.2ms/step | 14.3 samples/s
+Step 15230/44000 | Loss: 3.0220 | LR: 3.74e-04 | 1393.9ms/step | 14.3 samples/s
+Step 15240/44000 | Loss: 3.0309 | LR: 3.74e-04 | 1391.8ms/step | 14.4 samples/s
+Step 15250/44000 | Loss: 3.0045 | LR: 3.74e-04 | 1392.3ms/step | 14.4 samples/s
+Step 15260/44000 | Loss: 3.0354 | LR: 3.74e-04 | 1396.0ms/step | 14.3 samples/s
+Step 15270/44000 | Loss: 3.0393 | LR: 3.73e-04 | 1398.1ms/step | 14.3 samples/s
+Step 15280/44000 | Loss: 3.0223 | LR: 3.73e-04 | 1397.8ms/step | 14.3 samples/s
+Step 15290/44000 | Loss: 3.0307 | LR: 3.73e-04 | 1397.5ms/step | 14.3 samples/s
+Step 15300/44000 | Loss: 3.0057 | LR: 3.73e-04 | 1396.4ms/step | 14.3 samples/s
+Step 15310/44000 | Loss: 3.0119 | LR: 3.73e-04 | 1395.9ms/step | 14.3 samples/s
+Step 15320/44000 | Loss: 3.0289 | LR: 3.73e-04 | 1396.3ms/step | 14.3 samples/s
+Step 15330/44000 | Loss: 3.0405 | LR: 3.72e-04 | 1392.0ms/step | 14.4 samples/s
+Step 15340/44000 | Loss: 3.0404 | LR: 3.72e-04 | 1396.4ms/step | 14.3 samples/s
+Step 15350/44000 | Loss: 3.0310 | LR: 3.72e-04 | 1391.7ms/step | 14.4 samples/s
+Step 15360/44000 | Loss: 3.0393 | LR: 3.72e-04 | 1393.6ms/step | 14.4 samples/s
+Step 15370/44000 | Loss: 3.0226 | LR: 3.72e-04 | 1397.8ms/step | 14.3 samples/s
+Step 15380/44000 | Loss: 3.0479 | LR: 3.72e-04 | 1398.5ms/step | 14.3 samples/s
+Step 15390/44000 | Loss: 3.0179 | LR: 3.71e-04 | 1397.7ms/step | 14.3 samples/s
+Step 15400/44000 | Loss: 3.0449 | LR: 3.71e-04 | 1398.1ms/step | 14.3 samples/s
+Step 15410/44000 | Loss: 3.0025 | LR: 3.71e-04 | 1394.2ms/step | 14.3 samples/s
+Step 15420/44000 | Loss: 3.0394 | LR: 3.71e-04 | 1393.8ms/step | 14.3 samples/s
+Step 15430/44000 | Loss: 3.0173 | LR: 3.71e-04 | 1396.0ms/step | 14.3 samples/s
+Step 15440/44000 | Loss: 3.0320 | LR: 3.71e-04 | 1397.8ms/step | 14.3 samples/s
+Step 15450/44000 | Loss: 3.0389 | LR: 3.71e-04 | 1386.7ms/step | 14.4 samples/s
+Step 15460/44000 | Loss: 3.0033 | LR: 3.70e-04 | 1383.2ms/step | 14.5 samples/s
+Step 15470/44000 | Loss: 3.0039 | LR: 3.70e-04 | 1380.8ms/step | 14.5 samples/s
+Step 15480/44000 | Loss: 3.0047 | LR: 3.70e-04 | 1381.7ms/step | 14.5 samples/s
+Step 15490/44000 | Loss: 3.0137 | LR: 3.70e-04 | 1385.5ms/step | 14.4 samples/s
+Step 15500/44000 | Loss: 3.0237 | LR: 3.70e-04 | 1382.8ms/step | 14.5 samples/s
+Step 15510/44000 | Loss: 3.0418 | LR: 3.70e-04 | 1383.0ms/step | 14.5 samples/s
+Step 15520/44000 | Loss: 3.0105 | LR: 3.69e-04 | 1382.6ms/step | 14.5 samples/s
+Step 15530/44000 | Loss: 3.0415 | LR: 3.69e-04 | 1384.1ms/step | 14.5 samples/s
+Step 15540/44000 | Loss: 3.0484 | LR: 3.69e-04 | 1384.1ms/step | 14.4 samples/s
+Step 15550/44000 | Loss: 3.0181 | LR: 3.69e-04 | 1381.8ms/step | 14.5 samples/s
+Step 15560/44000 | Loss: 3.0018 | LR: 3.69e-04 | 1381.8ms/step | 14.5 samples/s
+Step 15570/44000 | Loss: 3.0029 | LR: 3.69e-04 | 1383.0ms/step | 14.5 samples/s
+Step 15580/44000 | Loss: 3.0316 | LR: 3.68e-04 | 1381.7ms/step | 14.5 samples/s
+Step 15590/44000 | Loss: 3.0127 | LR: 3.68e-04 | 1381.3ms/step | 14.5 samples/s
+Step 15600/44000 | Loss: 3.0162 | LR: 3.68e-04 | 1382.6ms/step | 14.5 samples/s
+Step 15610/44000 | Loss: 3.0102 | LR: 3.68e-04 | 1385.5ms/step | 14.4 samples/s
+Step 15620/44000 | Loss: 3.0454 | LR: 3.68e-04 | 1385.0ms/step | 14.4 samples/s
+Step 15630/44000 | Loss: 3.0184 | LR: 3.68e-04 | 1383.9ms/step | 14.5 samples/s
+Step 15640/44000 | Loss: 2.9881 | LR: 3.67e-04 | 1383.0ms/step | 14.5 samples/s
+Step 15650/44000 | Loss: 3.0266 | LR: 3.67e-04 | 1383.6ms/step | 14.5 samples/s
+Step 15660/44000 | Loss: 2.9721 | LR: 3.67e-04 | 1381.1ms/step | 14.5 samples/s
+Step 15670/44000 | Loss: 3.0388 | LR: 3.67e-04 | 1379.6ms/step | 14.5 samples/s
+Step 15680/44000 | Loss: 3.0331 | LR: 3.67e-04 | 1382.7ms/step | 14.5 samples/s
+Step 15690/44000 | Loss: 3.0214 | LR: 3.67e-04 | 1381.8ms/step | 14.5 samples/s
+Step 15700/44000 | Loss: 3.0060 | LR: 3.67e-04 | 1385.2ms/step | 14.4 samples/s
+Step 15710/44000 | Loss: 3.0150 | LR: 3.66e-04 | 1381.9ms/step | 14.5 samples/s
+Step 15720/44000 | Loss: 3.0232 | LR: 3.66e-04 | 1380.9ms/step | 14.5 samples/s
+Step 15730/44000 | Loss: 3.0138 | LR: 3.66e-04 | 1383.2ms/step | 14.5 samples/s
+Step 15740/44000 | Loss: 3.0213 | LR: 3.66e-04 | 1382.0ms/step | 14.5 samples/s
+Step 15750/44000 | Loss: 3.0062 | LR: 3.66e-04 | 1380.2ms/step | 14.5 samples/s
+Step 15760/44000 | Loss: 2.9919 | LR: 3.66e-04 | 1383.9ms/step | 14.5 samples/s
+Step 15770/44000 | Loss: 2.9964 | LR: 3.65e-04 | 1384.2ms/step | 14.4 samples/s
+Step 15780/44000 | Loss: 3.0229 | LR: 3.65e-04 | 1383.7ms/step | 14.5 samples/s
+Step 15790/44000 | Loss: 3.0417 | LR: 3.65e-04 | 1382.2ms/step | 14.5 samples/s
+Step 15800/44000 | Loss: 3.0243 | LR: 3.65e-04 | 1383.0ms/step | 14.5 samples/s
+Step 15810/44000 | Loss: 3.0098 | LR: 3.65e-04 | 1381.5ms/step | 14.5 samples/s
+Step 15820/44000 | Loss: 3.0059 | LR: 3.65e-04 | 1382.3ms/step | 14.5 samples/s
+Step 15830/44000 | Loss: 3.0036 | LR: 3.64e-04 | 1381.4ms/step | 14.5 samples/s
+Step 15840/44000 | Loss: 3.0112 | LR: 3.64e-04 | 1382.7ms/step | 14.5 samples/s
+Step 15850/44000 | Loss: 2.9943 | LR: 3.64e-04 | 1381.7ms/step | 14.5 samples/s
+Step 15860/44000 | Loss: 2.9963 | LR: 3.64e-04 | 1381.1ms/step | 14.5 samples/s
+Step 15870/44000 | Loss: 3.0264 | LR: 3.64e-04 | 1382.9ms/step | 14.5 samples/s
+Step 15880/44000 | Loss: 3.0167 | LR: 3.64e-04 | 1379.6ms/step | 14.5 samples/s
+Step 15890/44000 | Loss: 3.0253 | LR: 3.63e-04 | 1383.2ms/step | 14.5 samples/s
+Step 15900/44000 | Loss: 3.0185 | LR: 3.63e-04 | 1380.4ms/step | 14.5 samples/s
+Step 15910/44000 | Loss: 3.0075 | LR: 3.63e-04 | 1380.6ms/step | 14.5 samples/s
+Step 15920/44000 | Loss: 3.0237 | LR: 3.63e-04 | 1383.7ms/step | 14.5 samples/s
+Step 15930/44000 | Loss: 3.0164 | LR: 3.63e-04 | 1381.5ms/step | 14.5 samples/s
+Step 15940/44000 | Loss: 3.0122 | LR: 3.63e-04 | 1382.6ms/step | 14.5 samples/s
+Step 15950/44000 | Loss: 3.0095 | LR: 3.62e-04 | 1381.7ms/step | 14.5 samples/s
+Step 15960/44000 | Loss: 3.0093 | LR: 3.62e-04 | 1380.0ms/step | 14.5 samples/s
+Step 15970/44000 | Loss: 3.0030 | LR: 3.62e-04 | 1381.9ms/step | 14.5 samples/s
+Step 15980/44000 | Loss: 3.0254 | LR: 3.62e-04 | 1382.2ms/step | 14.5 samples/s
+Step 15990/44000 | Loss: 3.0162 | LR: 3.62e-04 | 1380.0ms/step | 14.5 samples/s
+Step 16000/44000 | Loss: 3.0118 | LR: 3.62e-04 | 1380.2ms/step | 14.5 samples/s
+Validation at step 16000: Loss = 3.0173, Perplexity = 20.44
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735/checkpoint-16000
+Step 16010/44000 | Loss: 3.0096 | LR: 3.62e-04 | 3884.3ms/step | 5.1 samples/s
+Step 16020/44000 | Loss: 3.0058 | LR: 3.61e-04 | 1390.9ms/step | 14.4 samples/s
+Step 16030/44000 | Loss: 3.0324 | LR: 3.61e-04 | 1390.2ms/step | 14.4 samples/s
+Step 16040/44000 | Loss: 2.9985 | LR: 3.61e-04 | 1387.1ms/step | 14.4 samples/s
+Step 16050/44000 | Loss: 3.0099 | LR: 3.61e-04 | 1390.9ms/step | 14.4 samples/s
+Step 16060/44000 | Loss: 2.9980 | LR: 3.61e-04 | 1387.1ms/step | 14.4 samples/s
+Step 16070/44000 | Loss: 3.0020 | LR: 3.61e-04 | 1389.6ms/step | 14.4 samples/s
+Step 16080/44000 | Loss: 3.0256 | LR: 3.60e-04 | 1390.1ms/step | 14.4 samples/s
+Step 16090/44000 | Loss: 3.0010 | LR: 3.60e-04 | 1393.2ms/step | 14.4 samples/s
+Step 16100/44000 | Loss: 3.0039 | LR: 3.60e-04 | 1393.9ms/step | 14.3 samples/s
+Step 16110/44000 | Loss: 3.0026 | LR: 3.60e-04 | 1394.5ms/step | 14.3 samples/s
+Step 16120/44000 | Loss: 3.0100 | LR: 3.60e-04 | 1395.4ms/step | 14.3 samples/s
+Step 16130/44000 | Loss: 3.0298 | LR: 3.60e-04 | 1394.4ms/step | 14.3 samples/s
+Step 16140/44000 | Loss: 3.0292 | LR: 3.59e-04 | 1398.0ms/step | 14.3 samples/s
+Step 16150/44000 | Loss: 3.0112 | LR: 3.59e-04 | 1393.8ms/step | 14.3 samples/s
+Step 16160/44000 | Loss: 3.0259 | LR: 3.59e-04 | 1398.1ms/step | 14.3 samples/s
+Step 16170/44000 | Loss: 3.0327 | LR: 3.59e-04 | 1397.1ms/step | 14.3 samples/s
+Step 16180/44000 | Loss: 3.0044 | LR: 3.59e-04 | 1396.9ms/step | 14.3 samples/s
+Step 16190/44000 | Loss: 2.9992 | LR: 3.59e-04 | 1388.0ms/step | 14.4 samples/s
+Step 16200/44000 | Loss: 2.9916 | LR: 3.58e-04 | 1380.0ms/step | 14.5 samples/s
+Step 16210/44000 | Loss: 3.0107 | LR: 3.58e-04 | 1380.0ms/step | 14.5 samples/s
+Step 16220/44000 | Loss: 3.0111 | LR: 3.58e-04 | 1381.8ms/step | 14.5 samples/s
+Step 16230/44000 | Loss: 3.0175 | LR: 3.58e-04 | 1382.3ms/step | 14.5 samples/s
+Step 16240/44000 | Loss: 3.0278 | LR: 3.58e-04 | 1382.0ms/step | 14.5 samples/s
+Step 16250/44000 | Loss: 2.9954 | LR: 3.58e-04 | 1382.4ms/step | 14.5 samples/s
+Step 16260/44000 | Loss: 2.9768 | LR: 3.57e-04 | 1381.6ms/step | 14.5 samples/s
+Step 16270/44000 | Loss: 2.9961 | LR: 3.57e-04 | 1381.9ms/step | 14.5 samples/s
+Step 16280/44000 | Loss: 3.0008 | LR: 3.57e-04 | 1381.8ms/step | 14.5 samples/s
+Step 16290/44000 | Loss: 2.9926 | LR: 3.57e-04 | 1382.8ms/step | 14.5 samples/s
+Step 16300/44000 | Loss: 2.9982 | LR: 3.57e-04 | 1381.9ms/step | 14.5 samples/s
+Step 16310/44000 | Loss: 3.0197 | LR: 3.57e-04 | 1383.5ms/step | 14.5 samples/s
+Step 16320/44000 | Loss: 3.0188 | LR: 3.56e-04 | 1380.8ms/step | 14.5 samples/s
+Step 16330/44000 | Loss: 3.0053 | LR: 3.56e-04 | 1383.5ms/step | 14.5 samples/s
+Step 16340/44000 | Loss: 3.0005 | LR: 3.56e-04 | 1384.1ms/step | 14.5 samples/s
+Step 16350/44000 | Loss: 3.0162 | LR: 3.56e-04 | 1381.5ms/step | 14.5 samples/s
+Step 16360/44000 | Loss: 3.0381 | LR: 3.56e-04 | 1382.6ms/step | 14.5 samples/s
+Step 16370/44000 | Loss: 2.9824 | LR: 3.56e-04 | 1381.8ms/step | 14.5 samples/s
+Step 16380/44000 | Loss: 2.9896 | LR: 3.55e-04 | 1383.5ms/step | 14.5 samples/s
+Step 16390/44000 | Loss: 2.9913 | LR: 3.55e-04 | 1382.0ms/step | 14.5 samples/s
+Step 16400/44000 | Loss: 3.0306 | LR: 3.55e-04 | 1381.5ms/step | 14.5 samples/s
+Step 16410/44000 | Loss: 3.0181 | LR: 3.55e-04 | 1382.6ms/step | 14.5 samples/s
+Step 16420/44000 | Loss: 3.0110 | LR: 3.55e-04 | 1380.8ms/step | 14.5 samples/s
+Step 16430/44000 | Loss: 2.9916 | LR: 3.55e-04 | 1382.0ms/step | 14.5 samples/s
+Step 16440/44000 | Loss: 3.0059 | LR: 3.54e-04 | 1379.7ms/step | 14.5 samples/s
+Step 16450/44000 | Loss: 3.0211 | LR: 3.54e-04 | 1379.7ms/step | 14.5 samples/s
+Step 16460/44000 | Loss: 3.0120 | LR: 3.54e-04 | 1381.1ms/step | 14.5 samples/s
+Step 16470/44000 | Loss: 2.9991 | LR: 3.54e-04 | 1380.6ms/step | 14.5 samples/s
+Step 16480/44000 | Loss: 2.9927 | LR: 3.54e-04 | 1380.7ms/step | 14.5 samples/s
+Step 16490/44000 | Loss: 3.0051 | LR: 3.54e-04 | 1382.2ms/step | 14.5 samples/s
+Step 16500/44000 | Loss: 2.9845 | LR: 3.53e-04 | 1385.3ms/step | 14.4 samples/s
+Step 16510/44000 | Loss: 3.0184 | LR: 3.53e-04 | 1382.0ms/step | 14.5 samples/s
+Step 16520/44000 | Loss: 3.0196 | LR: 3.53e-04 | 1382.2ms/step | 14.5 samples/s
+Step 16530/44000 | Loss: 3.0493 | LR: 3.53e-04 | 1381.9ms/step | 14.5 samples/s
+Step 16540/44000 | Loss: 3.0069 | LR: 3.53e-04 | 1380.6ms/step | 14.5 samples/s
+Step 16550/44000 | Loss: 3.0061 | LR: 3.53e-04 | 1382.3ms/step | 14.5 samples/s
+Step 16560/44000 | Loss: 3.0258 | LR: 3.52e-04 | 1384.2ms/step | 14.4 samples/s
+Step 16570/44000 | Loss: 3.0314 | LR: 3.52e-04 | 1381.7ms/step | 14.5 samples/s
+Step 16580/44000 | Loss: 3.0007 | LR: 3.52e-04 | 1381.4ms/step | 14.5 samples/s
+Step 16590/44000 | Loss: 2.9939 | LR: 3.52e-04 | 1381.0ms/step | 14.5 samples/s
+Step 16600/44000 | Loss: 2.9923 | LR: 3.52e-04 | 1380.6ms/step | 14.5 samples/s
+Step 16610/44000 | Loss: 3.0024 | LR: 3.52e-04 | 1380.1ms/step | 14.5 samples/s
+Step 16620/44000 | Loss: 3.0315 | LR: 3.51e-04 | 1382.2ms/step | 14.5 samples/s
+Step 16630/44000 | Loss: 2.9957 | LR: 3.51e-04 | 1383.6ms/step | 14.5 samples/s
+Step 16640/44000 | Loss: 2.9889 | LR: 3.51e-04 | 1381.2ms/step | 14.5 samples/s
+Step 16650/44000 | Loss: 2.9832 | LR: 3.51e-04 | 1380.9ms/step | 14.5 samples/s
+Step 16660/44000 | Loss: 3.0235 | LR: 3.51e-04 | 1383.8ms/step | 14.5 samples/s
+Step 16670/44000 | Loss: 3.0150 | LR: 3.51e-04 | 1382.9ms/step | 14.5 samples/s
+Step 16680/44000 | Loss: 2.9943 | LR: 3.50e-04 | 1383.2ms/step | 14.5 samples/s
+Step 16690/44000 | Loss: 3.0117 | LR: 3.50e-04 | 1382.0ms/step | 14.5 samples/s
+Step 16700/44000 | Loss: 2.9965 | LR: 3.50e-04 | 1382.4ms/step | 14.5 samples/s
+Step 16710/44000 | Loss: 3.0252 | LR: 3.50e-04 | 1381.9ms/step | 14.5 samples/s
+Step 16720/44000 | Loss: 3.0122 | LR: 3.50e-04 | 1382.0ms/step | 14.5 samples/s
+Step 16730/44000 | Loss: 3.0275 | LR: 3.50e-04 | 1380.4ms/step | 14.5 samples/s
+Step 16740/44000 | Loss: 2.9936 | LR: 3.49e-04 | 1379.6ms/step | 14.5 samples/s
+Step 16750/44000 | Loss: 3.0155 | LR: 3.49e-04 | 1381.8ms/step | 14.5 samples/s
+Step 16760/44000 | Loss: 2.9755 | LR: 3.49e-04 | 1381.9ms/step | 14.5 samples/s
+Step 16770/44000 | Loss: 2.9897 | LR: 3.49e-04 | 1384.0ms/step | 14.5 samples/s
+Step 16780/44000 | Loss: 3.0522 | LR: 3.49e-04 | 1383.5ms/step | 14.5 samples/s
+Step 16790/44000 | Loss: 3.0116 | LR: 3.49e-04 | 1385.6ms/step | 14.4 samples/s
+Step 16800/44000 | Loss: 2.9814 | LR: 3.48e-04 | 1383.1ms/step | 14.5 samples/s
+Step 16810/44000 | Loss: 3.0145 | LR: 3.48e-04 | 1383.1ms/step | 14.5 samples/s
+Step 16820/44000 | Loss: 3.0329 | LR: 3.48e-04 | 1382.8ms/step | 14.5 samples/s
+Step 16830/44000 | Loss: 2.9916 | LR: 3.48e-04 | 1383.1ms/step | 14.5 samples/s
+Step 16840/44000 | Loss: 2.9916 | LR: 3.48e-04 | 1382.9ms/step | 14.5 samples/s
+Step 16850/44000 | Loss: 2.9933 | LR: 3.48e-04 | 1384.2ms/step | 14.4 samples/s
+Step 16860/44000 | Loss: 2.9971 | LR: 3.47e-04 | 1383.2ms/step | 14.5 samples/s
+Step 16870/44000 | Loss: 3.0203 | LR: 3.47e-04 | 1381.0ms/step | 14.5 samples/s
+Step 16880/44000 | Loss: 3.0168 | LR: 3.47e-04 | 1381.3ms/step | 14.5 samples/s
+Step 16890/44000 | Loss: 3.0029 | LR: 3.47e-04 | 1381.6ms/step | 14.5 samples/s
+Step 16900/44000 | Loss: 2.9893 | LR: 3.47e-04 | 1380.8ms/step | 14.5 samples/s
+Step 16910/44000 | Loss: 3.0246 | LR: 3.47e-04 | 1381.8ms/step | 14.5 samples/s
+Step 16920/44000 | Loss: 3.0200 | LR: 3.46e-04 | 1380.5ms/step | 14.5 samples/s
+Step 16930/44000 | Loss: 2.9938 | LR: 3.46e-04 | 1379.7ms/step | 14.5 samples/s
+Step 16940/44000 | Loss: 3.0054 | LR: 3.46e-04 | 1385.0ms/step | 14.4 samples/s
+Step 16950/44000 | Loss: 2.9910 | LR: 3.46e-04 | 1382.5ms/step | 14.5 samples/s
+Step 16960/44000 | Loss: 2.9779 | LR: 3.46e-04 | 1381.2ms/step | 14.5 samples/s
+Step 16970/44000 | Loss: 3.0061 | LR: 3.46e-04 | 1378.7ms/step | 14.5 samples/s
+Step 16980/44000 | Loss: 3.0286 | LR: 3.45e-04 | 1381.2ms/step | 14.5 samples/s
+Step 16990/44000 | Loss: 3.0181 | LR: 3.45e-04 | 1383.1ms/step | 14.5 samples/s
+Step 17000/44000 | Loss: 3.0214 | LR: 3.45e-04 | 1383.6ms/step | 14.5 samples/s
+Validation at step 17000: Loss = 3.0033, Perplexity = 20.15
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735/checkpoint-17000
+Step 17010/44000 | Loss: 3.0014 | LR: 3.45e-04 | 3960.9ms/step | 5.0 samples/s
+Step 17020/44000 | Loss: 3.0023 | LR: 3.45e-04 | 1387.7ms/step | 14.4 samples/s
+Step 17030/44000 | Loss: 2.9887 | LR: 3.45e-04 | 1391.8ms/step | 14.4 samples/s
+Step 17040/44000 | Loss: 3.0081 | LR: 3.44e-04 | 1385.9ms/step | 14.4 samples/s
+Step 17050/44000 | Loss: 3.0109 | LR: 3.44e-04 | 1378.2ms/step | 14.5 samples/s
+Step 17060/44000 | Loss: 2.9994 | LR: 3.44e-04 | 1380.6ms/step | 14.5 samples/s
+Step 17070/44000 | Loss: 2.9990 | LR: 3.44e-04 | 1381.2ms/step | 14.5 samples/s
+Step 17080/44000 | Loss: 3.0163 | LR: 3.44e-04 | 1381.8ms/step | 14.5 samples/s
+Step 17090/44000 | Loss: 3.0045 | LR: 3.44e-04 | 1382.5ms/step | 14.5 samples/s
+Step 17100/44000 | Loss: 3.0048 | LR: 3.43e-04 | 1381.2ms/step | 14.5 samples/s
+Step 17110/44000 | Loss: 3.0118 | LR: 3.43e-04 | 1380.7ms/step | 14.5 samples/s
+Step 17120/44000 | Loss: 2.9760 | LR: 3.43e-04 | 1380.1ms/step | 14.5 samples/s
+Step 17130/44000 | Loss: 3.0005 | LR: 3.43e-04 | 1382.0ms/step | 14.5 samples/s
+Step 17140/44000 | Loss: 2.9831 | LR: 3.43e-04 | 1381.9ms/step | 14.5 samples/s
+Step 17150/44000 | Loss: 3.0132 | LR: 3.43e-04 | 1382.4ms/step | 14.5 samples/s
+Step 17160/44000 | Loss: 2.9875 | LR: 3.42e-04 | 1379.4ms/step | 14.5 samples/s
+Step 17170/44000 | Loss: 3.0163 | LR: 3.42e-04 | 1379.9ms/step | 14.5 samples/s
+Step 17180/44000 | Loss: 3.0253 | LR: 3.42e-04 | 1382.0ms/step | 14.5 samples/s
+Step 17190/44000 | Loss: 2.9860 | LR: 3.42e-04 | 1383.0ms/step | 14.5 samples/s
+Step 17200/44000 | Loss: 3.0115 | LR: 3.42e-04 | 1382.0ms/step | 14.5 samples/s
+Step 17210/44000 | Loss: 3.0037 | LR: 3.42e-04 | 1380.8ms/step | 14.5 samples/s
+Step 17220/44000 | Loss: 2.9925 | LR: 3.41e-04 | 1384.0ms/step | 14.5 samples/s
+Step 17230/44000 | Loss: 3.0244 | LR: 3.41e-04 | 1384.4ms/step | 14.4 samples/s
+Step 17240/44000 | Loss: 2.9996 | LR: 3.41e-04 | 1379.9ms/step | 14.5 samples/s
+Step 17250/44000 | Loss: 3.0261 | LR: 3.41e-04 | 1381.3ms/step | 14.5 samples/s
+Step 17260/44000 | Loss: 2.9982 | LR: 3.41e-04 | 1381.6ms/step | 14.5 samples/s
+Step 17270/44000 | Loss: 2.9750 | LR: 3.41e-04 | 1381.2ms/step | 14.5 samples/s
+Step 17280/44000 | Loss: 3.0168 | LR: 3.40e-04 | 1381.6ms/step | 14.5 samples/s
+Step 17290/44000 | Loss: 2.9897 | LR: 3.40e-04 | 1383.7ms/step | 14.5 samples/s
+Step 17300/44000 | Loss: 2.9955 | LR: 3.40e-04 | 1382.7ms/step | 14.5 samples/s
+Step 17310/44000 | Loss: 2.9770 | LR: 3.40e-04 | 1382.5ms/step | 14.5 samples/s
+Step 17320/44000 | Loss: 2.9880 | LR: 3.40e-04 | 1381.8ms/step | 14.5 samples/s
+Step 17330/44000 | Loss: 2.9879 | LR: 3.40e-04 | 1382.9ms/step | 14.5 samples/s
+Step 17340/44000 | Loss: 3.0105 | LR: 3.39e-04 | 1381.6ms/step | 14.5 samples/s
+Step 17350/44000 | Loss: 2.9734 | LR: 3.39e-04 | 1381.4ms/step | 14.5 samples/s
+Step 17360/44000 | Loss: 3.0121 | LR: 3.39e-04 | 1384.1ms/step | 14.4 samples/s
+Step 17370/44000 | Loss: 2.9960 | LR: 3.39e-04 | 1384.7ms/step | 14.4 samples/s
+Step 17380/44000 | Loss: 2.9806 | LR: 3.39e-04 | 1382.8ms/step | 14.5 samples/s
+Step 17390/44000 | Loss: 2.9971 | LR: 3.39e-04 | 1383.6ms/step | 14.5 samples/s
+Step 17400/44000 | Loss: 3.0013 | LR: 3.38e-04 | 1382.5ms/step | 14.5 samples/s
+Step 17410/44000 | Loss: 2.9969 | LR: 3.38e-04 | 1380.7ms/step | 14.5 samples/s
+Step 17420/44000 | Loss: 2.9929 | LR: 3.38e-04 | 1381.3ms/step | 14.5 samples/s
+Step 17430/44000 | Loss: 3.0156 | LR: 3.38e-04 | 1380.6ms/step | 14.5 samples/s
+Step 17440/44000 | Loss: 3.0278 | LR: 3.38e-04 | 1381.6ms/step | 14.5 samples/s
+Step 17450/44000 | Loss: 2.9866 | LR: 3.38e-04 | 1380.2ms/step | 14.5 samples/s
+Step 17460/44000 | Loss: 3.0027 | LR: 3.37e-04 | 1381.0ms/step | 14.5 samples/s
+Step 17470/44000 | Loss: 2.9944 | LR: 3.37e-04 | 1382.1ms/step | 14.5 samples/s
+Step 17480/44000 | Loss: 3.0006 | LR: 3.37e-04 | 1383.9ms/step | 14.5 samples/s
+Step 17490/44000 | Loss: 2.9942 | LR: 3.37e-04 | 1382.7ms/step | 14.5 samples/s
+Step 17500/44000 | Loss: 2.9898 | LR: 3.37e-04 | 1383.7ms/step | 14.5 samples/s
+Step 17510/44000 | Loss: 2.9653 | LR: 3.37e-04 | 1382.1ms/step | 14.5 samples/s
+Step 17520/44000 | Loss: 2.9832 | LR: 3.36e-04 | 1380.3ms/step | 14.5 samples/s

epfl_final/train_logs/speedprobe_seed48_mixed_ddp4/mixed.20260821T044419Z.log

@@ -0,0 +1,117 @@
+W0821 04:44:22.014000 61278 site-packages/torch/distributed/run.py:851] 
+W0821 04:44:22.014000 61278 site-packages/torch/distributed/run.py:851] *****************************************
+W0821 04:44:22.014000 61278 site-packages/torch/distributed/run.py:851] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
+W0821 04:44:22.014000 61278 site-packages/torch/distributed/run.py:851] *****************************************
+New run: Checkpoints will be saved to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_speedprobe_seed48/run_20260821_044427
+Running config:
+{
+  "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
+  "model_dtype": "bfloat16",
+  "init_from_scratch": true,
+  "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
+  "val_path": null,
+  "max_seq_length": 8192,
+  "num_workers": 8,
+  "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_speedprobe_seed48/run_20260821_044427",
+  "resume_from_checkpoint": null,
+  "force_new_run": true,
+  "num_epochs": 1,
+  "max_steps": 50,
+  "per_device_batch_size": 1,
+  "gradient_accumulation_steps": 5,
+  "learning_rate": 0.0005,
+  "weight_decay": 0.01,
+  "warmup_steps": 750,
+  "lr_scheduler": "cosine",
+  "max_grad_norm": 1.0,
+  "save_steps": 1000,
+  "eval_steps": 250,
+  "eval_batches": 50,
+  "logging_steps": 10,
+  "seed": 48,
+  "verbose": true,
+  "checkpoint_activations": false,
+  "strategy": "ddp",
+  "mixed_precision": "bf16",
+  "wandb_enabled": false,
+  "wandb_project": "stochastic_sparsity",
+  "wandb_name": "365m mixed 8k speedprobe_seed48 ddp2 h100-resume",
+  "wandb_tags": [
+    "qwen3",
+    "365m",
+    "8k",
+    "mixed",
+    "multiseed",
+    "speedprobe_seed48",
+    "ddp2",
+    "h100-resume"
+  ],
+  "hidden_size": 768,
+  "intermediate_size": 3072,
+  "num_attention_heads": 6,
+  "num_key_value_heads": 2,
+  "num_hidden_layers": 28,
+  "head_dim": null,
+  "attention_dropout": null,
+  "rope_train_position_scale_set": null,
+  "attention_type": "sparse_stochastic",
+  "sparse_block_size": 128,
+  "sparse_num_random_blocks": 31,
+  "sparse_mask_pool_size": 10000,
+  "sparse_val_dense_compare": false,
+  "sparse_layer_indices": null,
+  "sparse_mask_cache_dir": "/mloscratch/homes/navasard/sparse/mask_cache_speedprobe_seed48_ddp4",
+  "sparse_mask_seed": 0,
+  "sparse_mixed_nrb_set": [
+    63,
+    47,
+    31,
+    21,
+    11
+  ],
+  "dense_mode": false,
+  "dense_step_intervals": null
+}
+Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
+Initializing model from scratch (random weights)...
+Checkpoint activations: False
+Checkpoint activations: False
+Checkpoint activations: False
+Checkpoint activations: False
+W0821 04:48:58.112000 61278 site-packages/torch/distributed/elastic/agent/server/api.py:739] Received 15 death signal, shutting down workers
+W0821 04:48:58.114000 61278 site-packages/torch/distributed/elastic/multiprocessing/api.py:1012] Sending process 61346 closing signal SIGTERM
+W0821 04:48:58.115000 61278 site-packages/torch/distributed/elastic/multiprocessing/api.py:1012] Sending process 61347 closing signal SIGTERM
+W0821 04:48:58.115000 61278 site-packages/torch/distributed/elastic/multiprocessing/api.py:1012] Sending process 61348 closing signal SIGTERM
+W0821 04:48:58.115000 61278 site-packages/torch/distributed/elastic/multiprocessing/api.py:1012] Sending process 61349 closing signal SIGTERM
+Traceback (most recent call last):
+  File "/mloscratch/homes/navasard/conda/bin/torchrun", line 6, in <module>
+    sys.exit(main())
+             ~~~~^^
+  File "/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 362, in wrapper
+    return f(*args, **kwargs)
+  File "/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/distributed/run.py", line 990, in main
+    run(args)
+    ~~~^^^^^^
+  File "/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/distributed/run.py", line 981, in run
+    elastic_launch(
+    ~~~~~~~~~~~~~~~
+        config=config,
+        ~~~~~~~~~~~~~~
+        entrypoint=cmd,
+        ~~~~~~~~~~~~~~~
+    )(*cmd_args)
+    ~^^^^^^^^^^^
+  File "/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/distributed/launcher/api.py", line 170, in __call__
+    return launch_agent(self._config, self._entrypoint, list(args))
+  File "/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/distributed/launcher/api.py", line 308, in launch_agent
+    result = agent.run()
+  File "/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/distributed/elastic/metrics/api.py", line 134, in wrapper
+    result = f(*args, **kwargs)
+  File "/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/distributed/elastic/agent/server/api.py", line 731, in run
+    result = self._invoke_run(role)
+  File "/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/distributed/elastic/agent/server/api.py", line 908, in _invoke_run
+    time.sleep(monitor_interval)
+    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^
+  File "/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 86, in _terminate_process_handler
+    raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
+torch.distributed.elastic.multiprocessing.api.SignalException: Process 61278 got signal: 15

epfl_final/train_logs/speedprobe_seed48_mixed_ddp4/mixed.20260821T044917Z.log

@@ -0,0 +1,202 @@
+W0821 04:49:20.757000 61575 site-packages/torch/distributed/run.py:851] 
+W0821 04:49:20.757000 61575 site-packages/torch/distributed/run.py:851] *****************************************
+W0821 04:49:20.757000 61575 site-packages/torch/distributed/run.py:851] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
+W0821 04:49:20.757000 61575 site-packages/torch/distributed/run.py:851] *****************************************
+New run: Checkpoints will be saved to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_speedprobe_seed48/run_20260821_044926
+Running config:
+{
+  "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
+  "model_dtype": "bfloat16",
+  "init_from_scratch": true,
+  "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
+  "val_path": null,
+  "max_seq_length": 8192,
+  "num_workers": 8,
+  "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_speedprobe_seed48/run_20260821_044926",
+  "resume_from_checkpoint": null,
+  "force_new_run": true,
+  "num_epochs": 1,
+  "max_steps": 50,
+  "per_device_batch_size": 1,
+  "gradient_accumulation_steps": 5,
+  "learning_rate": 0.0005,
+  "weight_decay": 0.01,
+  "warmup_steps": 750,
+  "lr_scheduler": "cosine",
+  "max_grad_norm": 1.0,
+  "save_steps": 1000,
+  "eval_steps": 250,
+  "eval_batches": 50,
+  "logging_steps": 10,
+  "seed": 48,
+  "verbose": true,
+  "checkpoint_activations": false,
+  "strategy": "ddp",
+  "mixed_precision": "bf16",
+  "wandb_enabled": false,
+  "wandb_project": "stochastic_sparsity",
+  "wandb_name": "365m mixed 8k speedprobe_seed48 ddp2 h100-resume",
+  "wandb_tags": [
+    "qwen3",
+    "365m",
+    "8k",
+    "mixed",
+    "multiseed",
+    "speedprobe_seed48",
+    "ddp2",
+    "h100-resume"
+  ],
+  "hidden_size": 768,
+  "intermediate_size": 3072,
+  "num_attention_heads": 6,
+  "num_key_value_heads": 2,
+  "num_hidden_layers": 28,
+  "head_dim": null,
+  "attention_dropout": null,
+  "rope_train_position_scale_set": null,
+  "attention_type": "sparse_stochastic",
+  "sparse_block_size": 128,
+  "sparse_num_random_blocks": 31,
+  "sparse_mask_pool_size": 10000,
+  "sparse_val_dense_compare": false,
+  "sparse_layer_indices": null,
+  "sparse_mask_cache_dir": "/mloscratch/homes/navasard/sparse/mask_cache_8k_mixed",
+  "sparse_mask_seed": 0,
+  "sparse_mixed_nrb_set": [
+    63,
+    47,
+    31,
+    21,
+    11
+  ],
+  "dense_mode": false,
+  "dense_step_intervals": null
+}
+Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
+Checkpoint activations: False
+Initializing model from scratch (random weights)...
+Checkpoint activations: False
+Checkpoint activations: False
+Checkpoint activations: False
+
+
+================================================================================
+MODEL ARCHITECTURE
+================================================================================
+Model config:
+  vocab_size: 151936
+  hidden_size: 768
+  num_hidden_layers: 28
+  num_attention_heads: 6
+  num_key_value_heads: 2
+  head_dim: 128
+  intermediate_size: 3072
+  max_position_embeddings: 32768
+  rope_theta: 1000000
+  rope_train_position_scale_set: None
+  attention_type: sparse_stochastic
+  sparse_block_size: 128
+  sparse_num_random_blocks: 31
+  sparse_mask_pool_size: 10000
+  sparse_max_seq_length: 8192
+
+Parameter count:
+  Total parameters: 358,958,848
+  Trainable parameters: 358,958,848
+  Non-trainable parameters: 0
+  Model size (BF16): 0.72 GB
+
+--------------------------------------------------------------------------------
+LAYER-BY-LAYER SUMMARY
+--------------------------------------------------------------------------------
+/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/nn/attention/flex_attention.py:1704: UserWarning: flex_attention called without torch.compile() - this will use an unfused implementation that materializes the full scores matrix instead of generating a fused kernel.
+
+SOLUTION: Use torch.compile(flex_attention)(...)
+
+If you want to debug your score_mod/mask_mod, you can set:
+torch.nn.attention.flex_attention._FLEX_ATTENTION_DISABLE_COMPILE_DEBUG = True
+
+This will allow you to use print statements or breakpoints. Note: This doesn't work with the backwards pass and may produce incorrect results.
+  _warn_once(
+================================================================================================================================================================
+Layer (type:depth-idx)                                       Input Shape               Output Shape              Param #                   Trainable
+================================================================================================================================================================
+Qwen3ForCausalLM                                             [1, 8192]                 --                        --                        True
+├��Qwen3Model: 1-1                                            --                        [1, 8192, 768]            --                        True
+│    └─Embedding: 2-1                                        [1, 8192]                 [1, 8192, 768]            116,686,848               True
+│    └─ModuleList: 2-2                                       --                        --                        --                        True
+│    │    └─Qwen3DecoderLayer: 3-1                           [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-2                           [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-3                           [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-4                           [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-5                           [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-6                           [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-7                           [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-8                           [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-9                           [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-10                          [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-11                          [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-12                          [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-13                          [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-14                          [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-15                          [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-16                          [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-17                          [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-18                          [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-19                          [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-20                          [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-21                          [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-22                          [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-23                          [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-24                          [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-25                          [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-26                          [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-27                          [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    │    └─Qwen3DecoderLayer: 3-28                          [1, 8192, 768]            [1, 8192, 768]            8,652,544                 True
+│    └─Qwen3RMSNorm: 2-3                                     [1, 8192, 768]            [1, 8192, 768]            768                       True
+├─Linear: 1-2                                                [1, 8192, 768]            [1, 8192, 151936]         116,686,848               True
+================================================================================================================================================================
+Total params: 475,645,696
+Trainable params: 475,645,696
+Non-trainable params: 0
+Total mult-adds (Units.MEGABYTES): 475.65
+================================================================================================================================================================
+Input size (MB): 0.07
+Forward/backward pass size (MB): 15598.62
+Params size (MB): 951.29
+Estimated Total Size (MB): 16549.97
+================================================================================================================================================================
+  NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
+================================================================================
+
+Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
+Using pre-tokenized dataset (no runtime tokenization)
+Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
+Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
+Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
+Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
+Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
+Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
+Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
+Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
+Dataset split: 3027764 train, 159356 validation
+Training examples (packed, train split):  3027764
+Validation examples (packed, val split):  159356
+Per-device batch size:                    1
+World size (GPUs):                        4
+Gradient accumulation steps:              5
+Effective batch size (seqs/step):         20
+Sequence length:                          8192
+Tokens per optimizer step:                163,840
+Micro-batches per epoch (per rank):       756941
+Optimizer steps per epoch:                151388
+Total optimizer steps:                    50  (capped by max_steps; 0.00 epochs)
+Starting training...
+Switching attention mode for step 1: sparse_stochastic (configured sparse attention)
+Step 10/50 | Loss: 12.0967 | LR: 6.67e-06 | 6214.9ms/step | 3.2 samples/s
+Step 20/50 | Loss: 11.9985 | LR: 1.33e-05 | 402.3ms/step | 49.7 samples/s
+Step 30/50 | Loss: 11.6505 | LR: 2.00e-05 | 402.9ms/step | 49.6 samples/s
+Step 40/50 | Loss: 11.0928 | LR: 2.67e-05 | 402.2ms/step | 49.7 samples/s
+Step 50/50 | Loss: 10.7639 | LR: 3.33e-05 | 403.0ms/step | 49.6 samples/s
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_speedprobe_seed48/run_20260821_044926/checkpoint-50
+Training complete!