EPFL ALiBi and training provenance 451-500 of 1184

.gitattributes

@@ -246,3 +246,11 @@ epfl_final/wandb/run-20260601_055928-wkn5mxfh/run-wkn5mxfh.wandb filter=lfs diff
 epfl_final/wandb/run-20260601_064405-o2l7qkqb/run-o2l7qkqb.wandb filter=lfs diff=lfs merge=lfs -text
 epfl_final/wandb/run-20260602_042505-0pa1zqfy/run-0pa1zqfy.wandb filter=lfs diff=lfs merge=lfs -text
 epfl_final/wandb/run-20260602_052119-8igthj1g/run-8igthj1g.wandb filter=lfs diff=lfs merge=lfs -text
+epfl_final/wandb/run-20260604_030818-8igthj1g/run-8igthj1g.wandb filter=lfs diff=lfs merge=lfs -text
+epfl_final/wandb/run-20260605_220539-ns4ua5fr/run-ns4ua5fr.wandb filter=lfs diff=lfs merge=lfs -text
+epfl_final/wandb/run-20260606_174407-ns4ua5fr/run-ns4ua5fr.wandb filter=lfs diff=lfs merge=lfs -text
+epfl_final/wandb/run-20260609_064106-jh4dfz9a/run-jh4dfz9a.wandb filter=lfs diff=lfs merge=lfs -text
+epfl_final/wandb/run-20260611_032539-jh4dfz9a/run-jh4dfz9a.wandb filter=lfs diff=lfs merge=lfs -text
+epfl_final/wandb/run-20260612_172414-jh4dfz9a/run-jh4dfz9a.wandb filter=lfs diff=lfs merge=lfs -text
+epfl_final/wandb/run-20260613_222959-rp96qdel/run-rp96qdel.wandb filter=lfs diff=lfs merge=lfs -text
+epfl_final/wandb/run-20260614_010013-dpsvgm7b/run-dpsvgm7b.wandb filter=lfs diff=lfs merge=lfs -text

epfl_final/wandb/run-20260604_030818-8igthj1g/files/output.log

@@ -0,0 +1,672 @@
+Starting training...
+Validation at step 44000: Loss = 2.5048, Perplexity = 12.24
+wandb: WARNING Tried to log to step 44000 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Switching attention mode for step 44001: dense (dense step interval)
+Step 44010/50462 | Loss: 2.5147 | LR: 2.05e-05 | 76343.6ms/step | 0.8 samples/s
+wandb: WARNING Tried to log to step 44010 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44020/50462 | Loss: 2.5009 | LR: 2.04e-05 | 5185.9ms/step | 11.6 samples/s
+wandb: WARNING Tried to log to step 44020 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44030/50462 | Loss: 2.4941 | LR: 2.04e-05 | 5192.2ms/step | 11.6 samples/s
+wandb: WARNING Tried to log to step 44030 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44040/50462 | Loss: 2.4883 | LR: 2.03e-05 | 5203.5ms/step | 11.5 samples/s
+wandb: WARNING Tried to log to step 44040 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44050/50462 | Loss: 2.5206 | LR: 2.02e-05 | 5190.3ms/step | 11.6 samples/s
+wandb: WARNING Tried to log to step 44050 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44060/50462 | Loss: 2.5191 | LR: 2.02e-05 | 5220.3ms/step | 11.5 samples/s
+wandb: WARNING Tried to log to step 44060 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44070/50462 | Loss: 2.5187 | LR: 2.01e-05 | 5193.4ms/step | 11.6 samples/s
+wandb: WARNING Tried to log to step 44070 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44080/50462 | Loss: 2.5006 | LR: 2.01e-05 | 5195.4ms/step | 11.5 samples/s
+wandb: WARNING Tried to log to step 44080 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44090/50462 | Loss: 2.4700 | LR: 2.00e-05 | 5199.3ms/step | 11.5 samples/s
+wandb: WARNING Tried to log to step 44090 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44100/50462 | Loss: 2.4884 | LR: 1.99e-05 | 5205.9ms/step | 11.5 samples/s
+wandb: WARNING Tried to log to step 44100 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44110/50462 | Loss: 2.4903 | LR: 1.99e-05 | 5203.3ms/step | 11.5 samples/s
+wandb: WARNING Tried to log to step 44110 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44120/50462 | Loss: 2.5063 | LR: 1.98e-05 | 5207.8ms/step | 11.5 samples/s
+wandb: WARNING Tried to log to step 44120 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44130/50462 | Loss: 2.5124 | LR: 1.97e-05 | 5256.2ms/step | 11.4 samples/s
+wandb: WARNING Tried to log to step 44130 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44140/50462 | Loss: 2.4806 | LR: 1.97e-05 | 5196.0ms/step | 11.5 samples/s
+wandb: WARNING Tried to log to step 44140 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44150/50462 | Loss: 2.4756 | LR: 1.96e-05 | 5194.8ms/step | 11.5 samples/s
+wandb: WARNING Tried to log to step 44150 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44160/50462 | Loss: 2.5060 | LR: 1.96e-05 | 5217.2ms/step | 11.5 samples/s
+wandb: WARNING Tried to log to step 44160 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44170/50462 | Loss: 2.5094 | LR: 1.95e-05 | 5231.7ms/step | 11.5 samples/s
+wandb: WARNING Tried to log to step 44170 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44180/50462 | Loss: 2.4765 | LR: 1.94e-05 | 5201.0ms/step | 11.5 samples/s
+wandb: WARNING Tried to log to step 44180 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44190/50462 | Loss: 2.4826 | LR: 1.94e-05 | 5194.5ms/step | 11.6 samples/s
+wandb: WARNING Tried to log to step 44190 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44200/50462 | Loss: 2.5043 | LR: 1.93e-05 | 5259.2ms/step | 11.4 samples/s
+wandb: WARNING Tried to log to step 44200 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44210/50462 | Loss: 2.4919 | LR: 1.93e-05 | 5196.4ms/step | 11.5 samples/s
+wandb: WARNING Tried to log to step 44210 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44220/50462 | Loss: 2.4980 | LR: 1.92e-05 | 5192.5ms/step | 11.6 samples/s
+wandb: WARNING Tried to log to step 44220 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44230/50462 | Loss: 2.5093 | LR: 1.91e-05 | 5196.8ms/step | 11.5 samples/s
+wandb: WARNING Tried to log to step 44230 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44240/50462 | Loss: 2.4970 | LR: 1.91e-05 | 5173.2ms/step | 11.6 samples/s
+wandb: WARNING Tried to log to step 44240 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44250/50462 | Loss: 2.5105 | LR: 1.90e-05 | 5175.3ms/step | 11.6 samples/s
+wandb: WARNING Tried to log to step 44250 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Validation at step 44250: Loss = 2.5056, Perplexity = 12.25
+wandb: WARNING Tried to log to step 44250 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44260/50462 | Loss: 2.5054 | LR: 1.90e-05 | 7741.5ms/step | 7.8 samples/s
+wandb: WARNING Tried to log to step 44260 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44270/50462 | Loss: 2.4754 | LR: 1.89e-05 | 5184.2ms/step | 11.6 samples/s
+wandb: WARNING Tried to log to step 44270 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44280/50462 | Loss: 2.4989 | LR: 1.88e-05 | 5175.8ms/step | 11.6 samples/s
+wandb: WARNING Tried to log to step 44280 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44290/50462 | Loss: 2.4949 | LR: 1.88e-05 | 5171.9ms/step | 11.6 samples/s
+wandb: WARNING Tried to log to step 44290 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44300/50462 | Loss: 2.4461 | LR: 1.87e-05 | 5189.9ms/step | 11.6 samples/s
+wandb: WARNING Tried to log to step 44300 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44310/50462 | Loss: 2.4764 | LR: 1.87e-05 | 5181.6ms/step | 11.6 samples/s
+wandb: WARNING Tried to log to step 44310 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44320/50462 | Loss: 2.5087 | LR: 1.86e-05 | 5192.7ms/step | 11.6 samples/s
+wandb: WARNING Tried to log to step 44320 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44330/50462 | Loss: 2.4992 | LR: 1.85e-05 | 5182.9ms/step | 11.6 samples/s
+wandb: WARNING Tried to log to step 44330 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44340/50462 | Loss: 2.4956 | LR: 1.85e-05 | 5172.1ms/step | 11.6 samples/s
+wandb: WARNING Tried to log to step 44340 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44350/50462 | Loss: 2.5002 | LR: 1.84e-05 | 5184.8ms/step | 11.6 samples/s
+wandb: WARNING Tried to log to step 44350 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44360/50462 | Loss: 2.4735 | LR: 1.84e-05 | 5181.6ms/step | 11.6 samples/s
+wandb: WARNING Tried to log to step 44360 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44370/50462 | Loss: 2.4993 | LR: 1.83e-05 | 5176.3ms/step | 11.6 samples/s
+wandb: WARNING Tried to log to step 44370 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44380/50462 | Loss: 2.4896 | LR: 1.82e-05 | 5203.1ms/step | 11.5 samples/s
+wandb: WARNING Tried to log to step 44380 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44390/50462 | Loss: 2.5099 | LR: 1.82e-05 | 5183.8ms/step | 11.6 samples/s
+wandb: WARNING Tried to log to step 44390 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44400/50462 | Loss: 2.4760 | LR: 1.81e-05 | 5188.7ms/step | 11.6 samples/s
+wandb: WARNING Tried to log to step 44400 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44410/50462 | Loss: 2.4856 | LR: 1.81e-05 | 5189.1ms/step | 11.6 samples/s
+wandb: WARNING Tried to log to step 44410 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44420/50462 | Loss: 2.4958 | LR: 1.80e-05 | 5247.0ms/step | 11.4 samples/s
+wandb: WARNING Tried to log to step 44420 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44430/50462 | Loss: 2.5030 | LR: 1.79e-05 | 5144.2ms/step | 11.7 samples/s
+wandb: WARNING Tried to log to step 44430 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44440/50462 | Loss: 2.4919 | LR: 1.79e-05 | 5158.9ms/step | 11.6 samples/s
+wandb: WARNING Tried to log to step 44440 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44450/50462 | Loss: 2.4807 | LR: 1.78e-05 | 5172.8ms/step | 11.6 samples/s
+wandb: WARNING Tried to log to step 44450 that is less than the current step 44451. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 44460/50462 | Loss: 2.4960 | LR: 1.78e-05 | 5188.2ms/step | 11.6 samples/s
+Step 44470/50462 | Loss: 2.5090 | LR: 1.77e-05 | 5172.5ms/step | 11.6 samples/s
+Step 44480/50462 | Loss: 2.5232 | LR: 1.77e-05 | 5236.6ms/step | 11.5 samples/s
+Step 44490/50462 | Loss: 2.5189 | LR: 1.76e-05 | 5192.7ms/step | 11.6 samples/s
+Step 44500/50462 | Loss: 2.4821 | LR: 1.75e-05 | 5258.1ms/step | 11.4 samples/s
+Validation at step 44500: Loss = 2.5053, Perplexity = 12.25
+Step 44510/50462 | Loss: 2.4710 | LR: 1.75e-05 | 7732.5ms/step | 7.8 samples/s
+Step 44520/50462 | Loss: 2.4697 | LR: 1.74e-05 | 5188.2ms/step | 11.6 samples/s
+Step 44530/50462 | Loss: 2.4962 | LR: 1.74e-05 | 5193.9ms/step | 11.6 samples/s
+Step 44540/50462 | Loss: 2.4847 | LR: 1.73e-05 | 5192.1ms/step | 11.6 samples/s
+Step 44550/50462 | Loss: 2.4967 | LR: 1.72e-05 | 5175.2ms/step | 11.6 samples/s
+Step 44560/50462 | Loss: 2.5024 | LR: 1.72e-05 | 5169.8ms/step | 11.6 samples/s
+Step 44570/50462 | Loss: 2.4888 | LR: 1.71e-05 | 5167.9ms/step | 11.6 samples/s
+Step 44580/50462 | Loss: 2.4674 | LR: 1.71e-05 | 5196.7ms/step | 11.5 samples/s
+Step 44590/50462 | Loss: 2.4856 | LR: 1.70e-05 | 5181.2ms/step | 11.6 samples/s
+Step 44600/50462 | Loss: 2.5096 | LR: 1.70e-05 | 5187.5ms/step | 11.6 samples/s
+Step 44610/50462 | Loss: 2.4910 | LR: 1.69e-05 | 5172.7ms/step | 11.6 samples/s
+Step 44620/50462 | Loss: 2.5085 | LR: 1.68e-05 | 5170.6ms/step | 11.6 samples/s
+Step 44630/50462 | Loss: 2.4936 | LR: 1.68e-05 | 5181.3ms/step | 11.6 samples/s
+Step 44640/50462 | Loss: 2.4850 | LR: 1.67e-05 | 5175.1ms/step | 11.6 samples/s
+Step 44650/50462 | Loss: 2.4734 | LR: 1.67e-05 | 5190.9ms/step | 11.6 samples/s
+Step 44660/50462 | Loss: 2.5113 | LR: 1.66e-05 | 5196.8ms/step | 11.5 samples/s
+Step 44670/50462 | Loss: 2.5205 | LR: 1.66e-05 | 5181.3ms/step | 11.6 samples/s
+Step 44680/50462 | Loss: 2.5065 | LR: 1.65e-05 | 5176.2ms/step | 11.6 samples/s
+Step 44690/50462 | Loss: 2.5030 | LR: 1.64e-05 | 5177.1ms/step | 11.6 samples/s
+Step 44700/50462 | Loss: 2.5082 | LR: 1.64e-05 | 5174.3ms/step | 11.6 samples/s
+Step 44710/50462 | Loss: 2.4924 | LR: 1.63e-05 | 5246.6ms/step | 11.4 samples/s
+Step 44720/50462 | Loss: 2.4936 | LR: 1.63e-05 | 5178.5ms/step | 11.6 samples/s
+Step 44730/50462 | Loss: 2.4819 | LR: 1.62e-05 | 5182.7ms/step | 11.6 samples/s
+Step 44740/50462 | Loss: 2.4776 | LR: 1.62e-05 | 5184.7ms/step | 11.6 samples/s
+Step 44750/50462 | Loss: 2.5021 | LR: 1.61e-05 | 5192.5ms/step | 11.6 samples/s
+Validation at step 44750: Loss = 2.5052, Perplexity = 12.25
+Step 44760/50462 | Loss: 2.5177 | LR: 1.61e-05 | 7733.8ms/step | 7.8 samples/s
+Step 44770/50462 | Loss: 2.4661 | LR: 1.60e-05 | 5187.1ms/step | 11.6 samples/s
+Step 44780/50462 | Loss: 2.4948 | LR: 1.59e-05 | 5222.2ms/step | 11.5 samples/s
+Step 44790/50462 | Loss: 2.4846 | LR: 1.59e-05 | 5191.7ms/step | 11.6 samples/s
+Step 44800/50462 | Loss: 2.4857 | LR: 1.58e-05 | 5183.6ms/step | 11.6 samples/s
+Step 44810/50462 | Loss: 2.5117 | LR: 1.58e-05 | 5224.6ms/step | 11.5 samples/s
+Step 44820/50462 | Loss: 2.5013 | LR: 1.57e-05 | 5174.7ms/step | 11.6 samples/s
+Step 44830/50462 | Loss: 2.5227 | LR: 1.57e-05 | 5169.5ms/step | 11.6 samples/s
+Step 44840/50462 | Loss: 2.4790 | LR: 1.56e-05 | 5160.1ms/step | 11.6 samples/s
+Step 44850/50462 | Loss: 2.4829 | LR: 1.56e-05 | 5169.4ms/step | 11.6 samples/s
+Step 44860/50462 | Loss: 2.4823 | LR: 1.55e-05 | 5169.3ms/step | 11.6 samples/s
+Step 44870/50462 | Loss: 2.4716 | LR: 1.54e-05 | 5178.9ms/step | 11.6 samples/s
+Step 44880/50462 | Loss: 2.4956 | LR: 1.54e-05 | 5177.7ms/step | 11.6 samples/s
+Step 44890/50462 | Loss: 2.4588 | LR: 1.53e-05 | 5170.5ms/step | 11.6 samples/s
+Step 44900/50462 | Loss: 2.4590 | LR: 1.53e-05 | 5182.0ms/step | 11.6 samples/s
+Step 44910/50462 | Loss: 2.4815 | LR: 1.52e-05 | 5182.6ms/step | 11.6 samples/s
+Step 44920/50462 | Loss: 2.4839 | LR: 1.52e-05 | 5190.6ms/step | 11.6 samples/s
+Step 44930/50462 | Loss: 2.4873 | LR: 1.51e-05 | 5184.1ms/step | 11.6 samples/s
+Step 44940/50462 | Loss: 2.4897 | LR: 1.51e-05 | 5181.3ms/step | 11.6 samples/s
+Step 44950/50462 | Loss: 2.4473 | LR: 1.50e-05 | 5190.5ms/step | 11.6 samples/s
+Step 44960/50462 | Loss: 2.5261 | LR: 1.50e-05 | 5190.4ms/step | 11.6 samples/s
+Step 44970/50462 | Loss: 2.4759 | LR: 1.49e-05 | 5179.4ms/step | 11.6 samples/s
+Step 44980/50462 | Loss: 2.4877 | LR: 1.49e-05 | 5194.3ms/step | 11.6 samples/s
+Step 44990/50462 | Loss: 2.5167 | LR: 1.48e-05 | 5185.6ms/step | 11.6 samples/s
+Step 45000/50462 | Loss: 2.5076 | LR: 1.47e-05 | 5236.9ms/step | 11.5 samples/s
+Validation at step 45000: Loss = 2.5051, Perplexity = 12.24
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/run_20260602_050252/checkpoint-45000
+Step 45010/50462 | Loss: 2.4997 | LR: 1.47e-05 | 8739.1ms/step | 6.9 samples/s
+Step 45020/50462 | Loss: 2.4840 | LR: 1.46e-05 | 5203.3ms/step | 11.5 samples/s
+Step 45030/50462 | Loss: 2.4906 | LR: 1.46e-05 | 5188.3ms/step | 11.6 samples/s
+Step 45040/50462 | Loss: 2.5029 | LR: 1.45e-05 | 5191.5ms/step | 11.6 samples/s
+Step 45050/50462 | Loss: 2.4785 | LR: 1.45e-05 | 5200.7ms/step | 11.5 samples/s
+Step 45060/50462 | Loss: 2.5011 | LR: 1.44e-05 | 5203.3ms/step | 11.5 samples/s
+Step 45070/50462 | Loss: 2.4950 | LR: 1.44e-05 | 5193.8ms/step | 11.6 samples/s
+Step 45080/50462 | Loss: 2.4868 | LR: 1.43e-05 | 5203.7ms/step | 11.5 samples/s
+Step 45090/50462 | Loss: 2.4864 | LR: 1.43e-05 | 5250.3ms/step | 11.4 samples/s
+Step 45100/50462 | Loss: 2.5209 | LR: 1.42e-05 | 5204.0ms/step | 11.5 samples/s
+Step 45110/50462 | Loss: 2.5200 | LR: 1.42e-05 | 5244.6ms/step | 11.4 samples/s
+Step 45120/50462 | Loss: 2.5027 | LR: 1.41e-05 | 5177.9ms/step | 11.6 samples/s
+Step 45130/50462 | Loss: 2.5109 | LR: 1.41e-05 | 5186.4ms/step | 11.6 samples/s
+Step 45140/50462 | Loss: 2.4801 | LR: 1.40e-05 | 5187.7ms/step | 11.6 samples/s
+Step 45150/50462 | Loss: 2.5050 | LR: 1.40e-05 | 5187.6ms/step | 11.6 samples/s
+Step 45160/50462 | Loss: 2.5108 | LR: 1.39e-05 | 5185.1ms/step | 11.6 samples/s
+Step 45170/50462 | Loss: 2.4723 | LR: 1.39e-05 | 5201.9ms/step | 11.5 samples/s
+Step 45180/50462 | Loss: 2.5006 | LR: 1.38e-05 | 5186.6ms/step | 11.6 samples/s
+Step 45190/50462 | Loss: 2.4874 | LR: 1.37e-05 | 5197.7ms/step | 11.5 samples/s
+Step 45200/50462 | Loss: 2.5110 | LR: 1.37e-05 | 5201.1ms/step | 11.5 samples/s
+Step 45210/50462 | Loss: 2.4961 | LR: 1.36e-05 | 5189.7ms/step | 11.6 samples/s
+Step 45220/50462 | Loss: 2.4821 | LR: 1.36e-05 | 5181.5ms/step | 11.6 samples/s
+Step 45230/50462 | Loss: 2.4763 | LR: 1.35e-05 | 5186.5ms/step | 11.6 samples/s
+Step 45240/50462 | Loss: 2.5118 | LR: 1.35e-05 | 5184.0ms/step | 11.6 samples/s
+Step 45250/50462 | Loss: 2.4881 | LR: 1.34e-05 | 5178.8ms/step | 11.6 samples/s
+Validation at step 45250: Loss = 2.5050, Perplexity = 12.24
+Step 45260/50462 | Loss: 2.4996 | LR: 1.34e-05 | 7763.9ms/step | 7.7 samples/s
+Step 45270/50462 | Loss: 2.5095 | LR: 1.33e-05 | 5209.7ms/step | 11.5 samples/s
+Step 45280/50462 | Loss: 2.4668 | LR: 1.33e-05 | 5200.2ms/step | 11.5 samples/s
+Step 45290/50462 | Loss: 2.4931 | LR: 1.32e-05 | 5265.0ms/step | 11.4 samples/s
+Step 45300/50462 | Loss: 2.5119 | LR: 1.32e-05 | 5197.1ms/step | 11.5 samples/s
+Step 45310/50462 | Loss: 2.4799 | LR: 1.31e-05 | 5186.7ms/step | 11.6 samples/s
+Step 45320/50462 | Loss: 2.4879 | LR: 1.31e-05 | 5193.1ms/step | 11.6 samples/s
+Step 45330/50462 | Loss: 2.5046 | LR: 1.30e-05 | 5210.3ms/step | 11.5 samples/s
+Step 45340/50462 | Loss: 2.4958 | LR: 1.30e-05 | 5190.2ms/step | 11.6 samples/s
+Step 45350/50462 | Loss: 2.4928 | LR: 1.29e-05 | 5199.7ms/step | 11.5 samples/s
+Step 45360/50462 | Loss: 2.5122 | LR: 1.29e-05 | 5209.8ms/step | 11.5 samples/s
+Step 45370/50462 | Loss: 2.4854 | LR: 1.28e-05 | 5205.9ms/step | 11.5 samples/s
+Step 45380/50462 | Loss: 2.4811 | LR: 1.28e-05 | 5198.9ms/step | 11.5 samples/s
+Step 45390/50462 | Loss: 2.4878 | LR: 1.27e-05 | 5241.0ms/step | 11.4 samples/s
+Step 45400/50462 | Loss: 2.4718 | LR: 1.27e-05 | 5198.8ms/step | 11.5 samples/s
+Step 45410/50462 | Loss: 2.5171 | LR: 1.26e-05 | 5195.2ms/step | 11.5 samples/s
+Step 45420/50462 | Loss: 2.4874 | LR: 1.26e-05 | 5222.4ms/step | 11.5 samples/s
+Step 45430/50462 | Loss: 2.4626 | LR: 1.25e-05 | 5183.6ms/step | 11.6 samples/s
+Step 45440/50462 | Loss: 2.5043 | LR: 1.25e-05 | 5179.3ms/step | 11.6 samples/s
+Step 45450/50462 | Loss: 2.4944 | LR: 1.24e-05 | 5193.3ms/step | 11.6 samples/s
+Step 45460/50462 | Loss: 2.4843 | LR: 1.24e-05 | 5192.8ms/step | 11.6 samples/s
+Step 45470/50462 | Loss: 2.4933 | LR: 1.23e-05 | 5184.8ms/step | 11.6 samples/s
+Step 45480/50462 | Loss: 2.4835 | LR: 1.23e-05 | 5184.9ms/step | 11.6 samples/s
+Step 45490/50462 | Loss: 2.5048 | LR: 1.22e-05 | 5177.5ms/step | 11.6 samples/s
+Step 45500/50462 | Loss: 2.5045 | LR: 1.22e-05 | 5185.4ms/step | 11.6 samples/s
+Validation at step 45500: Loss = 2.5050, Perplexity = 12.24
+Step 45510/50462 | Loss: 2.4787 | LR: 1.21e-05 | 7725.2ms/step | 7.8 samples/s
+Step 45520/50462 | Loss: 2.5250 | LR: 1.21e-05 | 5173.0ms/step | 11.6 samples/s
+Step 45530/50462 | Loss: 2.4709 | LR: 1.20e-05 | 5184.5ms/step | 11.6 samples/s
+Step 45540/50462 | Loss: 2.5193 | LR: 1.20e-05 | 5167.8ms/step | 11.6 samples/s
+Step 45550/50462 | Loss: 2.4792 | LR: 1.19e-05 | 5167.7ms/step | 11.6 samples/s
+Step 45560/50462 | Loss: 2.5038 | LR: 1.19e-05 | 5176.6ms/step | 11.6 samples/s
+Step 45570/50462 | Loss: 2.5085 | LR: 1.19e-05 | 5178.5ms/step | 11.6 samples/s
+Step 45580/50462 | Loss: 2.4852 | LR: 1.18e-05 | 5247.9ms/step | 11.4 samples/s
+Step 45590/50462 | Loss: 2.4742 | LR: 1.18e-05 | 5219.2ms/step | 11.5 samples/s
+Step 45600/50462 | Loss: 2.4828 | LR: 1.17e-05 | 5204.7ms/step | 11.5 samples/s
+Step 45610/50462 | Loss: 2.4611 | LR: 1.17e-05 | 5194.6ms/step | 11.6 samples/s
+Step 45620/50462 | Loss: 2.4929 | LR: 1.16e-05 | 5210.7ms/step | 11.5 samples/s
+Step 45630/50462 | Loss: 2.5094 | LR: 1.16e-05 | 5207.5ms/step | 11.5 samples/s
+Step 45640/50462 | Loss: 2.4878 | LR: 1.15e-05 | 5201.9ms/step | 11.5 samples/s
+Step 45650/50462 | Loss: 2.5127 | LR: 1.15e-05 | 5203.4ms/step | 11.5 samples/s
+Step 45660/50462 | Loss: 2.4982 | LR: 1.14e-05 | 5216.1ms/step | 11.5 samples/s
+Step 45670/50462 | Loss: 2.4642 | LR: 1.14e-05 | 5218.4ms/step | 11.5 samples/s
+Step 45680/50462 | Loss: 2.4839 | LR: 1.13e-05 | 5221.1ms/step | 11.5 samples/s
+Step 45690/50462 | Loss: 2.5045 | LR: 1.13e-05 | 5246.8ms/step | 11.4 samples/s
+Step 45700/50462 | Loss: 2.4966 | LR: 1.12e-05 | 5182.5ms/step | 11.6 samples/s
+Step 45710/50462 | Loss: 2.4997 | LR: 1.12e-05 | 5187.8ms/step | 11.6 samples/s
+Step 45720/50462 | Loss: 2.4908 | LR: 1.11e-05 | 5253.9ms/step | 11.4 samples/s
+Step 45730/50462 | Loss: 2.4725 | LR: 1.11e-05 | 5201.6ms/step | 11.5 samples/s
+Step 45740/50462 | Loss: 2.4864 | LR: 1.10e-05 | 5189.2ms/step | 11.6 samples/s
+Step 45750/50462 | Loss: 2.5044 | LR: 1.10e-05 | 5204.2ms/step | 11.5 samples/s
+Validation at step 45750: Loss = 2.5049, Perplexity = 12.24
+Step 45760/50462 | Loss: 2.4592 | LR: 1.10e-05 | 7774.5ms/step | 7.7 samples/s
+Step 45770/50462 | Loss: 2.4911 | LR: 1.09e-05 | 5211.1ms/step | 11.5 samples/s
+Step 45780/50462 | Loss: 2.4834 | LR: 1.09e-05 | 5198.5ms/step | 11.5 samples/s
+Step 45790/50462 | Loss: 2.5220 | LR: 1.08e-05 | 5202.3ms/step | 11.5 samples/s
+Step 45800/50462 | Loss: 2.4354 | LR: 1.08e-05 | 5204.0ms/step | 11.5 samples/s
+Step 45810/50462 | Loss: 2.4962 | LR: 1.07e-05 | 5191.4ms/step | 11.6 samples/s
+Step 45820/50462 | Loss: 2.4923 | LR: 1.07e-05 | 5195.5ms/step | 11.5 samples/s
+Step 45830/50462 | Loss: 2.4986 | LR: 1.06e-05 | 5212.6ms/step | 11.5 samples/s
+Step 45840/50462 | Loss: 2.5116 | LR: 1.06e-05 | 5201.5ms/step | 11.5 samples/s
+Step 45850/50462 | Loss: 2.4936 | LR: 1.05e-05 | 5211.5ms/step | 11.5 samples/s
+Step 45860/50462 | Loss: 2.5182 | LR: 1.05e-05 | 5206.8ms/step | 11.5 samples/s
+Step 45870/50462 | Loss: 2.5032 | LR: 1.05e-05 | 5242.5ms/step | 11.4 samples/s
+Step 45880/50462 | Loss: 2.5039 | LR: 1.04e-05 | 5184.8ms/step | 11.6 samples/s
+Step 45890/50462 | Loss: 2.4665 | LR: 1.04e-05 | 5190.1ms/step | 11.6 samples/s
+Step 45900/50462 | Loss: 2.4898 | LR: 1.03e-05 | 5178.0ms/step | 11.6 samples/s
+Step 45910/50462 | Loss: 2.5200 | LR: 1.03e-05 | 5191.4ms/step | 11.6 samples/s
+Step 45920/50462 | Loss: 2.4733 | LR: 1.02e-05 | 5184.1ms/step | 11.6 samples/s
+Step 45930/50462 | Loss: 2.4848 | LR: 1.02e-05 | 5181.2ms/step | 11.6 samples/s
+Step 45940/50462 | Loss: 2.5039 | LR: 1.01e-05 | 5173.1ms/step | 11.6 samples/s
+Step 45950/50462 | Loss: 2.4910 | LR: 1.01e-05 | 5175.2ms/step | 11.6 samples/s
+Step 45960/50462 | Loss: 2.4769 | LR: 1.01e-05 | 5181.8ms/step | 11.6 samples/s
+Step 45970/50462 | Loss: 2.4824 | LR: 1.00e-05 | 5163.9ms/step | 11.6 samples/s
+Step 45980/50462 | Loss: 2.5263 | LR: 9.96e-06 | 5187.7ms/step | 11.6 samples/s
+Step 45990/50462 | Loss: 2.4801 | LR: 9.92e-06 | 5190.0ms/step | 11.6 samples/s
+Step 46000/50462 | Loss: 2.4944 | LR: 9.87e-06 | 5252.5ms/step | 11.4 samples/s
+Validation at step 46000: Loss = 2.5049, Perplexity = 12.24
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/run_20260602_050252/checkpoint-46000
+Step 46010/50462 | Loss: 2.4972 | LR: 9.83e-06 | 8699.2ms/step | 6.9 samples/s
+Step 46020/50462 | Loss: 2.4834 | LR: 9.79e-06 | 5214.3ms/step | 11.5 samples/s
+Step 46030/50462 | Loss: 2.5078 | LR: 9.74e-06 | 5192.7ms/step | 11.6 samples/s
+Step 46040/50462 | Loss: 2.5078 | LR: 9.70e-06 | 5201.9ms/step | 11.5 samples/s
+Step 46050/50462 | Loss: 2.4901 | LR: 9.65e-06 | 5202.2ms/step | 11.5 samples/s
+Step 46060/50462 | Loss: 2.4847 | LR: 9.61e-06 | 5204.8ms/step | 11.5 samples/s
+Step 46070/50462 | Loss: 2.5043 | LR: 9.57e-06 | 5186.0ms/step | 11.6 samples/s
+Step 46080/50462 | Loss: 2.4995 | LR: 9.52e-06 | 5212.7ms/step | 11.5 samples/s
+Step 46090/50462 | Loss: 2.4684 | LR: 9.48e-06 | 5211.2ms/step | 11.5 samples/s
+Step 46100/50462 | Loss: 2.4948 | LR: 9.44e-06 | 5192.0ms/step | 11.6 samples/s
+Step 46110/50462 | Loss: 2.4900 | LR: 9.40e-06 | 5213.0ms/step | 11.5 samples/s
+Step 46120/50462 | Loss: 2.4849 | LR: 9.35e-06 | 5202.5ms/step | 11.5 samples/s
+Step 46130/50462 | Loss: 2.4863 | LR: 9.31e-06 | 5197.7ms/step | 11.5 samples/s
+Step 46140/50462 | Loss: 2.4870 | LR: 9.27e-06 | 5204.4ms/step | 11.5 samples/s
+Step 46150/50462 | Loss: 2.4853 | LR: 9.22e-06 | 5185.9ms/step | 11.6 samples/s
+Step 46160/50462 | Loss: 2.5033 | LR: 9.18e-06 | 5251.5ms/step | 11.4 samples/s
+Step 46170/50462 | Loss: 2.5035 | LR: 9.14e-06 | 5172.3ms/step | 11.6 samples/s
+Step 46180/50462 | Loss: 2.4734 | LR: 9.10e-06 | 5176.3ms/step | 11.6 samples/s
+Step 46190/50462 | Loss: 2.4736 | LR: 9.06e-06 | 5157.9ms/step | 11.6 samples/s
+Step 46200/50462 | Loss: 2.4817 | LR: 9.01e-06 | 5154.3ms/step | 11.6 samples/s
+Step 46210/50462 | Loss: 2.4980 | LR: 8.97e-06 | 5191.7ms/step | 11.6 samples/s
+Step 46220/50462 | Loss: 2.5004 | LR: 8.93e-06 | 5176.2ms/step | 11.6 samples/s
+Step 46230/50462 | Loss: 2.5075 | LR: 8.89e-06 | 5206.5ms/step | 11.5 samples/s
+Step 46240/50462 | Loss: 2.5160 | LR: 8.85e-06 | 5202.9ms/step | 11.5 samples/s
+Step 46250/50462 | Loss: 2.5136 | LR: 8.80e-06 | 5196.2ms/step | 11.5 samples/s
+Validation at step 46250: Loss = 2.5049, Perplexity = 12.24
+Step 46260/50462 | Loss: 2.5136 | LR: 8.76e-06 | 7727.5ms/step | 7.8 samples/s
+Step 46270/50462 | Loss: 2.5004 | LR: 8.72e-06 | 5190.9ms/step | 11.6 samples/s
+Step 46280/50462 | Loss: 2.5317 | LR: 8.68e-06 | 5183.3ms/step | 11.6 samples/s
+Step 46290/50462 | Loss: 2.5176 | LR: 8.64e-06 | 5179.9ms/step | 11.6 samples/s
+Step 46300/50462 | Loss: 2.4887 | LR: 8.60e-06 | 5243.9ms/step | 11.4 samples/s
+Step 46310/50462 | Loss: 2.4963 | LR: 8.56e-06 | 5175.4ms/step | 11.6 samples/s
+Step 46320/50462 | Loss: 2.5217 | LR: 8.52e-06 | 5188.3ms/step | 11.6 samples/s
+Step 46330/50462 | Loss: 2.5018 | LR: 8.47e-06 | 5224.0ms/step | 11.5 samples/s
+Step 46340/50462 | Loss: 2.5011 | LR: 8.43e-06 | 5186.2ms/step | 11.6 samples/s
+Step 46350/50462 | Loss: 2.5222 | LR: 8.39e-06 | 5164.1ms/step | 11.6 samples/s
+Step 46360/50462 | Loss: 2.4844 | LR: 8.35e-06 | 5167.3ms/step | 11.6 samples/s
+Step 46370/50462 | Loss: 2.4706 | LR: 8.31e-06 | 5174.8ms/step | 11.6 samples/s
+Step 46380/50462 | Loss: 2.4955 | LR: 8.27e-06 | 5185.8ms/step | 11.6 samples/s
+Step 46390/50462 | Loss: 2.4847 | LR: 8.23e-06 | 5173.9ms/step | 11.6 samples/s
+Step 46400/50462 | Loss: 2.4765 | LR: 8.19e-06 | 5172.2ms/step | 11.6 samples/s
+Step 46410/50462 | Loss: 2.4890 | LR: 8.15e-06 | 5183.7ms/step | 11.6 samples/s
+Step 46420/50462 | Loss: 2.4881 | LR: 8.11e-06 | 5170.6ms/step | 11.6 samples/s
+Step 46430/50462 | Loss: 2.5091 | LR: 8.07e-06 | 5174.4ms/step | 11.6 samples/s
+Step 46440/50462 | Loss: 2.4920 | LR: 8.03e-06 | 5173.6ms/step | 11.6 samples/s
+Step 46450/50462 | Loss: 2.5176 | LR: 7.99e-06 | 5232.9ms/step | 11.5 samples/s
+Step 46460/50462 | Loss: 2.4957 | LR: 7.95e-06 | 5187.6ms/step | 11.6 samples/s
+Step 46470/50462 | Loss: 2.4873 | LR: 7.91e-06 | 5177.5ms/step | 11.6 samples/s
+Step 46480/50462 | Loss: 2.4829 | LR: 7.87e-06 | 5186.0ms/step | 11.6 samples/s
+Step 46490/50462 | Loss: 2.4782 | LR: 7.83e-06 | 5176.1ms/step | 11.6 samples/s
+Step 46500/50462 | Loss: 2.4953 | LR: 7.80e-06 | 5165.8ms/step | 11.6 samples/s
+Validation at step 46500: Loss = 2.5049, Perplexity = 12.24
+Step 46510/50462 | Loss: 2.4902 | LR: 7.76e-06 | 7749.4ms/step | 7.7 samples/s
+Step 46520/50462 | Loss: 2.5135 | LR: 7.72e-06 | 5187.4ms/step | 11.6 samples/s
+Step 46530/50462 | Loss: 2.4931 | LR: 7.68e-06 | 5205.1ms/step | 11.5 samples/s
+Step 46540/50462 | Loss: 2.5066 | LR: 7.64e-06 | 5192.6ms/step | 11.6 samples/s
+Step 46550/50462 | Loss: 2.4674 | LR: 7.60e-06 | 5196.7ms/step | 11.5 samples/s
+Step 46560/50462 | Loss: 2.4845 | LR: 7.56e-06 | 5181.3ms/step | 11.6 samples/s
+Step 46570/50462 | Loss: 2.4955 | LR: 7.52e-06 | 5194.7ms/step | 11.6 samples/s
+Step 46580/50462 | Loss: 2.4876 | LR: 7.49e-06 | 5191.6ms/step | 11.6 samples/s
+Step 46590/50462 | Loss: 2.4797 | LR: 7.45e-06 | 5205.0ms/step | 11.5 samples/s
+Step 46600/50462 | Loss: 2.4952 | LR: 7.41e-06 | 5199.4ms/step | 11.5 samples/s
+Step 46610/50462 | Loss: 2.4932 | LR: 7.37e-06 | 5246.2ms/step | 11.4 samples/s
+Step 46620/50462 | Loss: 2.4753 | LR: 7.33e-06 | 5163.2ms/step | 11.6 samples/s
+Step 46630/50462 | Loss: 2.4900 | LR: 7.29e-06 | 5222.8ms/step | 11.5 samples/s
+Step 46640/50462 | Loss: 2.5022 | LR: 7.26e-06 | 5201.0ms/step | 11.5 samples/s
+Step 46650/50462 | Loss: 2.4942 | LR: 7.22e-06 | 5195.5ms/step | 11.5 samples/s
+Step 46660/50462 | Loss: 2.4840 | LR: 7.18e-06 | 5195.6ms/step | 11.5 samples/s
+Step 46670/50462 | Loss: 2.4988 | LR: 7.14e-06 | 5197.2ms/step | 11.5 samples/s
+Step 46680/50462 | Loss: 2.5076 | LR: 7.11e-06 | 5190.2ms/step | 11.6 samples/s
+Step 46690/50462 | Loss: 2.5246 | LR: 7.07e-06 | 5203.1ms/step | 11.5 samples/s
+Step 46700/50462 | Loss: 2.4851 | LR: 7.03e-06 | 5193.3ms/step | 11.6 samples/s
+Step 46710/50462 | Loss: 2.5175 | LR: 6.99e-06 | 5198.8ms/step | 11.5 samples/s
+Step 46720/50462 | Loss: 2.5011 | LR: 6.96e-06 | 5196.4ms/step | 11.5 samples/s
+Step 46730/50462 | Loss: 2.4808 | LR: 6.92e-06 | 5202.9ms/step | 11.5 samples/s
+Step 46740/50462 | Loss: 2.5074 | LR: 6.88e-06 | 5191.6ms/step | 11.6 samples/s
+Step 46750/50462 | Loss: 2.4906 | LR: 6.85e-06 | 5222.0ms/step | 11.5 samples/s
+Validation at step 46750: Loss = 2.5049, Perplexity = 12.24
+Step 46760/50462 | Loss: 2.5052 | LR: 6.81e-06 | 7730.9ms/step | 7.8 samples/s
+Step 46770/50462 | Loss: 2.4943 | LR: 6.77e-06 | 5159.1ms/step | 11.6 samples/s
+Step 46780/50462 | Loss: 2.5204 | LR: 6.74e-06 | 5184.9ms/step | 11.6 samples/s
+Step 46790/50462 | Loss: 2.4776 | LR: 6.70e-06 | 5170.3ms/step | 11.6 samples/s
+Step 46800/50462 | Loss: 2.5151 | LR: 6.66e-06 | 5160.8ms/step | 11.6 samples/s
+Step 46810/50462 | Loss: 2.5114 | LR: 6.63e-06 | 5181.9ms/step | 11.6 samples/s
+Step 46820/50462 | Loss: 2.4747 | LR: 6.59e-06 | 5169.7ms/step | 11.6 samples/s
+Step 46830/50462 | Loss: 2.4728 | LR: 6.56e-06 | 5176.0ms/step | 11.6 samples/s
+Step 46840/50462 | Loss: 2.4985 | LR: 6.52e-06 | 5184.4ms/step | 11.6 samples/s
+Step 46850/50462 | Loss: 2.4946 | LR: 6.48e-06 | 5174.9ms/step | 11.6 samples/s
+Step 46860/50462 | Loss: 2.4941 | LR: 6.45e-06 | 5167.7ms/step | 11.6 samples/s
+Step 46870/50462 | Loss: 2.4520 | LR: 6.41e-06 | 5172.2ms/step | 11.6 samples/s
+Step 46880/50462 | Loss: 2.4878 | LR: 6.38e-06 | 5170.6ms/step | 11.6 samples/s
+Step 46890/50462 | Loss: 2.4936 | LR: 6.34e-06 | 5169.1ms/step | 11.6 samples/s
+Step 46900/50462 | Loss: 2.4857 | LR: 6.31e-06 | 5173.0ms/step | 11.6 samples/s
+Step 46910/50462 | Loss: 2.4788 | LR: 6.27e-06 | 5219.4ms/step | 11.5 samples/s
+Step 46920/50462 | Loss: 2.5168 | LR: 6.24e-06 | 5186.9ms/step | 11.6 samples/s
+Step 46930/50462 | Loss: 2.4932 | LR: 6.20e-06 | 5234.2ms/step | 11.5 samples/s
+Step 46940/50462 | Loss: 2.4948 | LR: 6.17e-06 | 5198.8ms/step | 11.5 samples/s
+Step 46950/50462 | Loss: 2.4777 | LR: 6.13e-06 | 5164.3ms/step | 11.6 samples/s
+Step 46960/50462 | Loss: 2.5014 | LR: 6.10e-06 | 5171.3ms/step | 11.6 samples/s
+Step 46970/50462 | Loss: 2.5051 | LR: 6.06e-06 | 5153.7ms/step | 11.6 samples/s
+Step 46980/50462 | Loss: 2.5043 | LR: 6.03e-06 | 5184.5ms/step | 11.6 samples/s
+Step 46990/50462 | Loss: 2.5255 | LR: 5.99e-06 | 5179.1ms/step | 11.6 samples/s
+Step 47000/50462 | Loss: 2.5053 | LR: 5.96e-06 | 5176.9ms/step | 11.6 samples/s
+Validation at step 47000: Loss = 2.5048, Perplexity = 12.24
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/run_20260602_050252/checkpoint-47000
+Step 47010/50462 | Loss: 2.5168 | LR: 5.93e-06 | 8707.4ms/step | 6.9 samples/s
+Step 47020/50462 | Loss: 2.4936 | LR: 5.89e-06 | 5186.7ms/step | 11.6 samples/s
+Step 47030/50462 | Loss: 2.5132 | LR: 5.86e-06 | 5240.4ms/step | 11.4 samples/s
+Step 47040/50462 | Loss: 2.4586 | LR: 5.82e-06 | 5206.8ms/step | 11.5 samples/s
+Step 47050/50462 | Loss: 2.4955 | LR: 5.79e-06 | 5205.5ms/step | 11.5 samples/s
+Step 47060/50462 | Loss: 2.4934 | LR: 5.76e-06 | 5200.7ms/step | 11.5 samples/s
+Step 47070/50462 | Loss: 2.4866 | LR: 5.72e-06 | 5200.7ms/step | 11.5 samples/s
+Step 47080/50462 | Loss: 2.4942 | LR: 5.69e-06 | 5174.5ms/step | 11.6 samples/s
+Step 47090/50462 | Loss: 2.4890 | LR: 5.65e-06 | 5190.4ms/step | 11.6 samples/s
+Step 47100/50462 | Loss: 2.4929 | LR: 5.62e-06 | 5185.1ms/step | 11.6 samples/s
+Step 47110/50462 | Loss: 2.4969 | LR: 5.59e-06 | 5213.4ms/step | 11.5 samples/s
+Step 47120/50462 | Loss: 2.4472 | LR: 5.56e-06 | 5197.5ms/step | 11.5 samples/s
+Step 47130/50462 | Loss: 2.4940 | LR: 5.52e-06 | 5202.0ms/step | 11.5 samples/s
+Step 47140/50462 | Loss: 2.5065 | LR: 5.49e-06 | 5208.1ms/step | 11.5 samples/s
+Step 47150/50462 | Loss: 2.4626 | LR: 5.46e-06 | 5173.6ms/step | 11.6 samples/s
+Step 47160/50462 | Loss: 2.5041 | LR: 5.42e-06 | 5200.5ms/step | 11.5 samples/s
+Step 47170/50462 | Loss: 2.4873 | LR: 5.39e-06 | 5184.8ms/step | 11.6 samples/s
+Step 47180/50462 | Loss: 2.4973 | LR: 5.36e-06 | 5173.0ms/step | 11.6 samples/s
+Step 47190/50462 | Loss: 2.4916 | LR: 5.33e-06 | 5193.3ms/step | 11.6 samples/s
+Step 47200/50462 | Loss: 2.4893 | LR: 5.29e-06 | 5188.8ms/step | 11.6 samples/s
+Step 47210/50462 | Loss: 2.4631 | LR: 5.26e-06 | 5203.6ms/step | 11.5 samples/s
+Step 47220/50462 | Loss: 2.5042 | LR: 5.23e-06 | 5233.6ms/step | 11.5 samples/s
+Step 47230/50462 | Loss: 2.5017 | LR: 5.20e-06 | 5182.9ms/step | 11.6 samples/s
+Step 47240/50462 | Loss: 2.4710 | LR: 5.16e-06 | 5224.0ms/step | 11.5 samples/s
+Step 47250/50462 | Loss: 2.5009 | LR: 5.13e-06 | 5167.7ms/step | 11.6 samples/s
+Validation at step 47250: Loss = 2.5048, Perplexity = 12.24
+Step 47260/50462 | Loss: 2.4958 | LR: 5.10e-06 | 7725.9ms/step | 7.8 samples/s
+Step 47270/50462 | Loss: 2.5072 | LR: 5.07e-06 | 5187.3ms/step | 11.6 samples/s
+Step 47280/50462 | Loss: 2.4806 | LR: 5.04e-06 | 5190.9ms/step | 11.6 samples/s
+Step 47290/50462 | Loss: 2.5090 | LR: 5.01e-06 | 5175.9ms/step | 11.6 samples/s
+Step 47300/50462 | Loss: 2.5073 | LR: 4.97e-06 | 5173.6ms/step | 11.6 samples/s
+Step 47310/50462 | Loss: 2.4698 | LR: 4.94e-06 | 5185.4ms/step | 11.6 samples/s
+Step 47320/50462 | Loss: 2.4862 | LR: 4.91e-06 | 5171.5ms/step | 11.6 samples/s
+Step 47330/50462 | Loss: 2.4897 | LR: 4.88e-06 | 5231.8ms/step | 11.5 samples/s
+Step 47340/50462 | Loss: 2.4542 | LR: 4.85e-06 | 5171.5ms/step | 11.6 samples/s
+Step 47350/50462 | Loss: 2.5018 | LR: 4.82e-06 | 5159.6ms/step | 11.6 samples/s
+Step 47360/50462 | Loss: 2.4725 | LR: 4.79e-06 | 5173.0ms/step | 11.6 samples/s
+Step 47370/50462 | Loss: 2.5110 | LR: 4.76e-06 | 5163.0ms/step | 11.6 samples/s
+Step 47380/50462 | Loss: 2.5032 | LR: 4.73e-06 | 5183.0ms/step | 11.6 samples/s
+Step 47390/50462 | Loss: 2.4857 | LR: 4.70e-06 | 5200.3ms/step | 11.5 samples/s
+Step 47400/50462 | Loss: 2.4892 | LR: 4.67e-06 | 5184.6ms/step | 11.6 samples/s
+Step 47410/50462 | Loss: 2.5004 | LR: 4.64e-06 | 5165.8ms/step | 11.6 samples/s
+Step 47420/50462 | Loss: 2.4523 | LR: 4.61e-06 | 5167.7ms/step | 11.6 samples/s
+Step 47430/50462 | Loss: 2.4803 | LR: 4.58e-06 | 5164.2ms/step | 11.6 samples/s
+Step 47440/50462 | Loss: 2.4889 | LR: 4.55e-06 | 5163.8ms/step | 11.6 samples/s
+Step 47450/50462 | Loss: 2.4637 | LR: 4.52e-06 | 5161.8ms/step | 11.6 samples/s
+Step 47460/50462 | Loss: 2.5085 | LR: 4.49e-06 | 5151.5ms/step | 11.6 samples/s
+Step 47470/50462 | Loss: 2.5061 | LR: 4.46e-06 | 5172.4ms/step | 11.6 samples/s
+Step 47480/50462 | Loss: 2.5048 | LR: 4.43e-06 | 5182.2ms/step | 11.6 samples/s
+Step 47490/50462 | Loss: 2.4981 | LR: 4.40e-06 | 5183.1ms/step | 11.6 samples/s
+Step 47500/50462 | Loss: 2.4890 | LR: 4.37e-06 | 5169.8ms/step | 11.6 samples/s
+Validation at step 47500: Loss = 2.5048, Perplexity = 12.24
+Step 47510/50462 | Loss: 2.4600 | LR: 4.34e-06 | 7722.4ms/step | 7.8 samples/s
+Step 47520/50462 | Loss: 2.5080 | LR: 4.31e-06 | 5236.0ms/step | 11.5 samples/s
+Step 47530/50462 | Loss: 2.5029 | LR: 4.28e-06 | 5157.6ms/step | 11.6 samples/s
+Step 47540/50462 | Loss: 2.4708 | LR: 4.25e-06 | 5219.2ms/step | 11.5 samples/s
+Step 47550/50462 | Loss: 2.5007 | LR: 4.22e-06 | 5219.3ms/step | 11.5 samples/s
+Step 47560/50462 | Loss: 2.4864 | LR: 4.19e-06 | 5213.7ms/step | 11.5 samples/s
+Step 47570/50462 | Loss: 2.4968 | LR: 4.16e-06 | 5190.2ms/step | 11.6 samples/s
+Step 47580/50462 | Loss: 2.5322 | LR: 4.13e-06 | 5181.6ms/step | 11.6 samples/s
+Step 47590/50462 | Loss: 2.5026 | LR: 4.11e-06 | 5203.0ms/step | 11.5 samples/s
+Step 47600/50462 | Loss: 2.4937 | LR: 4.08e-06 | 5194.2ms/step | 11.6 samples/s
+Step 47610/50462 | Loss: 2.4731 | LR: 4.05e-06 | 5197.8ms/step | 11.5 samples/s
+Step 47620/50462 | Loss: 2.4871 | LR: 4.02e-06 | 5243.7ms/step | 11.4 samples/s
+Step 47630/50462 | Loss: 2.5144 | LR: 3.99e-06 | 5199.7ms/step | 11.5 samples/s
+Step 47640/50462 | Loss: 2.4698 | LR: 3.97e-06 | 5210.5ms/step | 11.5 samples/s
+Step 47650/50462 | Loss: 2.4882 | LR: 3.94e-06 | 5209.7ms/step | 11.5 samples/s
+Step 47660/50462 | Loss: 2.4879 | LR: 3.91e-06 | 5209.8ms/step | 11.5 samples/s
+Step 47670/50462 | Loss: 2.4799 | LR: 3.88e-06 | 5204.8ms/step | 11.5 samples/s
+Step 47680/50462 | Loss: 2.4980 | LR: 3.85e-06 | 5201.2ms/step | 11.5 samples/s
+Step 47690/50462 | Loss: 2.4934 | LR: 3.83e-06 | 5195.1ms/step | 11.5 samples/s
+Step 47700/50462 | Loss: 2.5015 | LR: 3.80e-06 | 5205.7ms/step | 11.5 samples/s
+Step 47710/50462 | Loss: 2.4681 | LR: 3.77e-06 | 5196.7ms/step | 11.5 samples/s
+Step 47720/50462 | Loss: 2.4946 | LR: 3.74e-06 | 5205.1ms/step | 11.5 samples/s
+Step 47730/50462 | Loss: 2.4959 | LR: 3.72e-06 | 5214.4ms/step | 11.5 samples/s
+Step 47740/50462 | Loss: 2.4806 | LR: 3.69e-06 | 5198.5ms/step | 11.5 samples/s
+Step 47750/50462 | Loss: 2.5072 | LR: 3.66e-06 | 5208.7ms/step | 11.5 samples/s
+Validation at step 47750: Loss = 2.5048, Perplexity = 12.24
+Step 47760/50462 | Loss: 2.4646 | LR: 3.64e-06 | 7766.7ms/step | 7.7 samples/s
+Step 47770/50462 | Loss: 2.5274 | LR: 3.61e-06 | 5202.6ms/step | 11.5 samples/s
+Step 47780/50462 | Loss: 2.4847 | LR: 3.58e-06 | 5200.7ms/step | 11.5 samples/s
+Step 47790/50462 | Loss: 2.4790 | LR: 3.56e-06 | 5215.1ms/step | 11.5 samples/s
+Step 47800/50462 | Loss: 2.4709 | LR: 3.53e-06 | 5206.1ms/step | 11.5 samples/s
+Step 47810/50462 | Loss: 2.5045 | LR: 3.50e-06 | 5205.1ms/step | 11.5 samples/s
+Step 47820/50462 | Loss: 2.4975 | LR: 3.48e-06 | 5204.7ms/step | 11.5 samples/s
+Step 47830/50462 | Loss: 2.4900 | LR: 3.45e-06 | 5265.4ms/step | 11.4 samples/s
+Step 47840/50462 | Loss: 2.5129 | LR: 3.42e-06 | 5202.3ms/step | 11.5 samples/s
+Step 47850/50462 | Loss: 2.5031 | LR: 3.40e-06 | 5237.6ms/step | 11.5 samples/s
+Step 47860/50462 | Loss: 2.5052 | LR: 3.37e-06 | 5183.5ms/step | 11.6 samples/s
+Step 47870/50462 | Loss: 2.4996 | LR: 3.35e-06 | 5163.0ms/step | 11.6 samples/s
+Step 47880/50462 | Loss: 2.4540 | LR: 3.32e-06 | 5172.5ms/step | 11.6 samples/s
+Step 47890/50462 | Loss: 2.5120 | LR: 3.30e-06 | 5178.6ms/step | 11.6 samples/s
+Step 47900/50462 | Loss: 2.4997 | LR: 3.27e-06 | 5170.5ms/step | 11.6 samples/s
+Step 47910/50462 | Loss: 2.4997 | LR: 3.24e-06 | 5252.4ms/step | 11.4 samples/s
+Step 47920/50462 | Loss: 2.4736 | LR: 3.22e-06 | 5212.3ms/step | 11.5 samples/s
+Step 47930/50462 | Loss: 2.5108 | LR: 3.19e-06 | 5182.6ms/step | 11.6 samples/s
+Step 47940/50462 | Loss: 2.4882 | LR: 3.17e-06 | 5194.5ms/step | 11.6 samples/s
+Step 47950/50462 | Loss: 2.4943 | LR: 3.14e-06 | 5192.0ms/step | 11.6 samples/s
+Step 47960/50462 | Loss: 2.4612 | LR: 3.12e-06 | 5213.9ms/step | 11.5 samples/s
+Step 47970/50462 | Loss: 2.4978 | LR: 3.09e-06 | 5208.2ms/step | 11.5 samples/s
+Step 47980/50462 | Loss: 2.4943 | LR: 3.07e-06 | 5158.7ms/step | 11.6 samples/s
+Step 47990/50462 | Loss: 2.4774 | LR: 3.04e-06 | 5176.8ms/step | 11.6 samples/s
+Step 48000/50462 | Loss: 2.4876 | LR: 3.02e-06 | 5183.8ms/step | 11.6 samples/s
+Validation at step 48000: Loss = 2.5048, Perplexity = 12.24
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/run_20260602_050252/checkpoint-48000
+Step 48010/50462 | Loss: 2.5007 | LR: 3.00e-06 | 8752.7ms/step | 6.9 samples/s
+Step 48020/50462 | Loss: 2.5287 | LR: 2.97e-06 | 5155.9ms/step | 11.6 samples/s
+Step 48030/50462 | Loss: 2.4702 | LR: 2.95e-06 | 5177.6ms/step | 11.6 samples/s
+Step 48040/50462 | Loss: 2.4842 | LR: 2.92e-06 | 5172.5ms/step | 11.6 samples/s
+Step 48050/50462 | Loss: 2.4769 | LR: 2.90e-06 | 5171.3ms/step | 11.6 samples/s
+Step 48060/50462 | Loss: 2.4942 | LR: 2.87e-06 | 5193.9ms/step | 11.6 samples/s
+Step 48070/50462 | Loss: 2.4968 | LR: 2.85e-06 | 5161.5ms/step | 11.6 samples/s
+Step 48080/50462 | Loss: 2.5208 | LR: 2.83e-06 | 5177.3ms/step | 11.6 samples/s
+Step 48090/50462 | Loss: 2.4963 | LR: 2.80e-06 | 5172.9ms/step | 11.6 samples/s
+Step 48100/50462 | Loss: 2.4999 | LR: 2.78e-06 | 5170.7ms/step | 11.6 samples/s
+Step 48110/50462 | Loss: 2.4892 | LR: 2.76e-06 | 5164.7ms/step | 11.6 samples/s
+Step 48120/50462 | Loss: 2.4872 | LR: 2.73e-06 | 5173.8ms/step | 11.6 samples/s
+Step 48130/50462 | Loss: 2.4885 | LR: 2.71e-06 | 5173.1ms/step | 11.6 samples/s
+Step 48140/50462 | Loss: 2.4703 | LR: 2.69e-06 | 5220.6ms/step | 11.5 samples/s
+Step 48150/50462 | Loss: 2.4834 | LR: 2.66e-06 | 5224.9ms/step | 11.5 samples/s
+Step 48160/50462 | Loss: 2.5029 | LR: 2.64e-06 | 5215.3ms/step | 11.5 samples/s
+Step 48170/50462 | Loss: 2.4828 | LR: 2.62e-06 | 5201.5ms/step | 11.5 samples/s
+Step 48180/50462 | Loss: 2.4954 | LR: 2.60e-06 | 5208.1ms/step | 11.5 samples/s
+Step 48190/50462 | Loss: 2.4914 | LR: 2.57e-06 | 5210.1ms/step | 11.5 samples/s
+Step 48200/50462 | Loss: 2.5012 | LR: 2.55e-06 | 5246.8ms/step | 11.4 samples/s
+Step 48210/50462 | Loss: 2.4770 | LR: 2.53e-06 | 5202.8ms/step | 11.5 samples/s
+Step 48220/50462 | Loss: 2.4825 | LR: 2.51e-06 | 5209.3ms/step | 11.5 samples/s
+Step 48230/50462 | Loss: 2.4899 | LR: 2.48e-06 | 5213.2ms/step | 11.5 samples/s
+Step 48240/50462 | Loss: 2.4881 | LR: 2.46e-06 | 5211.1ms/step | 11.5 samples/s
+Step 48250/50462 | Loss: 2.5273 | LR: 2.44e-06 | 5213.0ms/step | 11.5 samples/s
+Validation at step 48250: Loss = 2.5048, Perplexity = 12.24
+Step 48260/50462 | Loss: 2.4645 | LR: 2.42e-06 | 7771.1ms/step | 7.7 samples/s
+Step 48270/50462 | Loss: 2.4902 | LR: 2.39e-06 | 5210.3ms/step | 11.5 samples/s
+Step 48280/50462 | Loss: 2.4982 | LR: 2.37e-06 | 5195.8ms/step | 11.5 samples/s
+Step 48290/50462 | Loss: 2.4860 | LR: 2.35e-06 | 5223.5ms/step | 11.5 samples/s
+Step 48300/50462 | Loss: 2.4793 | LR: 2.33e-06 | 5211.3ms/step | 11.5 samples/s
+Step 48310/50462 | Loss: 2.4994 | LR: 2.31e-06 | 5208.5ms/step | 11.5 samples/s
+Step 48320/50462 | Loss: 2.4982 | LR: 2.29e-06 | 5206.0ms/step | 11.5 samples/s
+Step 48330/50462 | Loss: 2.4812 | LR: 2.27e-06 | 5184.7ms/step | 11.6 samples/s
+Step 48340/50462 | Loss: 2.5049 | LR: 2.24e-06 | 5202.1ms/step | 11.5 samples/s
+Step 48350/50462 | Loss: 2.5183 | LR: 2.22e-06 | 5204.0ms/step | 11.5 samples/s
+Step 48360/50462 | Loss: 2.4743 | LR: 2.20e-06 | 5194.3ms/step | 11.6 samples/s
+Step 48370/50462 | Loss: 2.5124 | LR: 2.18e-06 | 5192.6ms/step | 11.6 samples/s
+Step 48380/50462 | Loss: 2.4784 | LR: 2.16e-06 | 5204.3ms/step | 11.5 samples/s
+Step 48390/50462 | Loss: 2.5268 | LR: 2.14e-06 | 5189.8ms/step | 11.6 samples/s
+Step 48400/50462 | Loss: 2.4811 | LR: 2.12e-06 | 5219.6ms/step | 11.5 samples/s
+Step 48410/50462 | Loss: 2.5098 | LR: 2.10e-06 | 5200.8ms/step | 11.5 samples/s
+Step 48420/50462 | Loss: 2.4861 | LR: 2.08e-06 | 5206.8ms/step | 11.5 samples/s
+Step 48430/50462 | Loss: 2.4827 | LR: 2.06e-06 | 5208.2ms/step | 11.5 samples/s
+Step 48440/50462 | Loss: 2.5002 | LR: 2.04e-06 | 5238.1ms/step | 11.5 samples/s
+Step 48450/50462 | Loss: 2.4658 | LR: 2.02e-06 | 5187.7ms/step | 11.6 samples/s
+Step 48460/50462 | Loss: 2.5132 | LR: 2.00e-06 | 5229.7ms/step | 11.5 samples/s
+Step 48470/50462 | Loss: 2.5008 | LR: 1.98e-06 | 5162.1ms/step | 11.6 samples/s
+Step 48480/50462 | Loss: 2.4844 | LR: 1.96e-06 | 5163.8ms/step | 11.6 samples/s
+Step 48490/50462 | Loss: 2.4921 | LR: 1.94e-06 | 5230.6ms/step | 11.5 samples/s
+Step 48500/50462 | Loss: 2.4614 | LR: 1.92e-06 | 5197.7ms/step | 11.5 samples/s
+Validation at step 48500: Loss = 2.5048, Perplexity = 12.24
+Step 48510/50462 | Loss: 2.4943 | LR: 1.90e-06 | 7721.8ms/step | 7.8 samples/s
+Step 48520/50462 | Loss: 2.4938 | LR: 1.88e-06 | 5186.9ms/step | 11.6 samples/s
+Step 48530/50462 | Loss: 2.5110 | LR: 1.86e-06 | 5204.1ms/step | 11.5 samples/s
+Step 48540/50462 | Loss: 2.4682 | LR: 1.84e-06 | 5184.9ms/step | 11.6 samples/s
+Step 48550/50462 | Loss: 2.4775 | LR: 1.82e-06 | 5186.0ms/step | 11.6 samples/s
+Step 48560/50462 | Loss: 2.4882 | LR: 1.80e-06 | 5193.3ms/step | 11.6 samples/s
+Step 48570/50462 | Loss: 2.4897 | LR: 1.78e-06 | 5191.4ms/step | 11.6 samples/s
+Step 48580/50462 | Loss: 2.4826 | LR: 1.77e-06 | 5212.3ms/step | 11.5 samples/s
+Step 48590/50462 | Loss: 2.5070 | LR: 1.75e-06 | 5169.8ms/step | 11.6 samples/s
+Step 48600/50462 | Loss: 2.4984 | LR: 1.73e-06 | 5183.8ms/step | 11.6 samples/s
+Step 48610/50462 | Loss: 2.4841 | LR: 1.71e-06 | 5181.4ms/step | 11.6 samples/s
+Step 48620/50462 | Loss: 2.4898 | LR: 1.69e-06 | 5183.7ms/step | 11.6 samples/s
+Step 48630/50462 | Loss: 2.5119 | LR: 1.67e-06 | 5165.8ms/step | 11.6 samples/s
+Step 48640/50462 | Loss: 2.4940 | LR: 1.66e-06 | 5208.5ms/step | 11.5 samples/s
+Step 48650/50462 | Loss: 2.5105 | LR: 1.64e-06 | 5196.0ms/step | 11.5 samples/s
+Step 48660/50462 | Loss: 2.5061 | LR: 1.62e-06 | 5178.1ms/step | 11.6 samples/s
+Step 48670/50462 | Loss: 2.5075 | LR: 1.60e-06 | 5170.7ms/step | 11.6 samples/s
+Step 48680/50462 | Loss: 2.4898 | LR: 1.58e-06 | 5183.1ms/step | 11.6 samples/s
+Step 48690/50462 | Loss: 2.5035 | LR: 1.57e-06 | 5183.5ms/step | 11.6 samples/s
+Step 48700/50462 | Loss: 2.4631 | LR: 1.55e-06 | 5179.1ms/step | 11.6 samples/s
+Step 48710/50462 | Loss: 2.4862 | LR: 1.53e-06 | 5181.1ms/step | 11.6 samples/s
+Step 48720/50462 | Loss: 2.5078 | LR: 1.51e-06 | 5189.1ms/step | 11.6 samples/s
+Step 48730/50462 | Loss: 2.4936 | LR: 1.50e-06 | 5163.3ms/step | 11.6 samples/s
+Step 48740/50462 | Loss: 2.4703 | LR: 1.48e-06 | 5186.0ms/step | 11.6 samples/s
+Step 48750/50462 | Loss: 2.5140 | LR: 1.46e-06 | 5226.5ms/step | 11.5 samples/s
+Validation at step 48750: Loss = 2.5048, Perplexity = 12.24
+Step 48760/50462 | Loss: 2.5248 | LR: 1.44e-06 | 7776.7ms/step | 7.7 samples/s
+Step 48770/50462 | Loss: 2.4649 | LR: 1.43e-06 | 5180.9ms/step | 11.6 samples/s
+Step 48780/50462 | Loss: 2.4872 | LR: 1.41e-06 | 5180.6ms/step | 11.6 samples/s
+Step 48790/50462 | Loss: 2.4959 | LR: 1.39e-06 | 5256.3ms/step | 11.4 samples/s
+Step 48800/50462 | Loss: 2.4750 | LR: 1.38e-06 | 5207.1ms/step | 11.5 samples/s
+Step 48810/50462 | Loss: 2.4844 | LR: 1.36e-06 | 5178.3ms/step | 11.6 samples/s
+Step 48820/50462 | Loss: 2.4896 | LR: 1.34e-06 | 5201.7ms/step | 11.5 samples/s
+Step 48830/50462 | Loss: 2.5065 | LR: 1.33e-06 | 5206.8ms/step | 11.5 samples/s
+Step 48840/50462 | Loss: 2.5192 | LR: 1.31e-06 | 5204.7ms/step | 11.5 samples/s
+Step 48850/50462 | Loss: 2.4911 | LR: 1.30e-06 | 5200.3ms/step | 11.5 samples/s
+Step 48860/50462 | Loss: 2.4568 | LR: 1.28e-06 | 5221.1ms/step | 11.5 samples/s
+Step 48870/50462 | Loss: 2.4998 | LR: 1.26e-06 | 5204.1ms/step | 11.5 samples/s
+Step 48880/50462 | Loss: 2.4823 | LR: 1.25e-06 | 5208.2ms/step | 11.5 samples/s
+Step 48890/50462 | Loss: 2.5129 | LR: 1.23e-06 | 5196.6ms/step | 11.5 samples/s
+Step 48900/50462 | Loss: 2.4909 | LR: 1.22e-06 | 5211.4ms/step | 11.5 samples/s
+Step 48910/50462 | Loss: 2.5049 | LR: 1.20e-06 | 5209.2ms/step | 11.5 samples/s
+Step 48920/50462 | Loss: 2.4914 | LR: 1.19e-06 | 5200.7ms/step | 11.5 samples/s
+Step 48930/50462 | Loss: 2.4877 | LR: 1.17e-06 | 5207.8ms/step | 11.5 samples/s
+Step 48940/50462 | Loss: 2.4682 | LR: 1.16e-06 | 5204.6ms/step | 11.5 samples/s
+Step 48950/50462 | Loss: 2.4682 | LR: 1.14e-06 | 5180.3ms/step | 11.6 samples/s
+Step 48960/50462 | Loss: 2.4996 | LR: 1.13e-06 | 5185.3ms/step | 11.6 samples/s
+Step 48970/50462 | Loss: 2.4668 | LR: 1.11e-06 | 5171.7ms/step | 11.6 samples/s
+Step 48980/50462 | Loss: 2.5349 | LR: 1.10e-06 | 5168.3ms/step | 11.6 samples/s
+Step 48990/50462 | Loss: 2.5197 | LR: 1.08e-06 | 5179.3ms/step | 11.6 samples/s
+Step 49000/50462 | Loss: 2.5014 | LR: 1.07e-06 | 5173.4ms/step | 11.6 samples/s
+Validation at step 49000: Loss = 2.5048, Perplexity = 12.24
+Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/run_20260602_050252/checkpoint-49000
+Step 49010/50462 | Loss: 2.4900 | LR: 1.05e-06 | 8715.7ms/step | 6.9 samples/s
+Step 49020/50462 | Loss: 2.4978 | LR: 1.04e-06 | 5170.9ms/step | 11.6 samples/s
+Step 49030/50462 | Loss: 2.4990 | LR: 1.02e-06 | 5160.1ms/step | 11.6 samples/s
+Step 49040/50462 | Loss: 2.4951 | LR: 1.01e-06 | 5169.0ms/step | 11.6 samples/s
+Step 49050/50462 | Loss: 2.4951 | LR: 9.95e-07 | 5225.9ms/step | 11.5 samples/s
+Step 49060/50462 | Loss: 2.5158 | LR: 9.81e-07 | 5185.0ms/step | 11.6 samples/s
+Step 49070/50462 | Loss: 2.5128 | LR: 9.67e-07 | 5295.2ms/step | 11.3 samples/s
+Step 49080/50462 | Loss: 2.5147 | LR: 9.53e-07 | 5195.1ms/step | 11.5 samples/s
+Step 49090/50462 | Loss: 2.4686 | LR: 9.39e-07 | 5198.3ms/step | 11.5 samples/s
+Step 49100/50462 | Loss: 2.5177 | LR: 9.25e-07 | 5203.7ms/step | 11.5 samples/s
+Step 49110/50462 | Loss: 2.5028 | LR: 9.12e-07 | 5186.9ms/step | 11.6 samples/s
+Step 49120/50462 | Loss: 2.4722 | LR: 8.99e-07 | 5164.1ms/step | 11.6 samples/s
+Step 49130/50462 | Loss: 2.5013 | LR: 8.85e-07 | 5185.9ms/step | 11.6 samples/s
+Step 49140/50462 | Loss: 2.4733 | LR: 8.72e-07 | 5183.1ms/step | 11.6 samples/s
+Step 49150/50462 | Loss: 2.5046 | LR: 8.59e-07 | 5199.8ms/step | 11.5 samples/s
+Step 49160/50462 | Loss: 2.4944 | LR: 8.46e-07 | 5202.6ms/step | 11.5 samples/s
+Step 49170/50462 | Loss: 2.4924 | LR: 8.33e-07 | 5199.8ms/step | 11.5 samples/s
+Step 49180/50462 | Loss: 2.4823 | LR: 8.20e-07 | 5177.3ms/step | 11.6 samples/s
+Step 49190/50462 | Loss: 2.4718 | LR: 8.07e-07 | 5188.1ms/step | 11.6 samples/s
+Step 49200/50462 | Loss: 2.5251 | LR: 7.95e-07 | 5179.2ms/step | 11.6 samples/s
+Step 49210/50462 | Loss: 2.4958 | LR: 7.82e-07 | 5186.4ms/step | 11.6 samples/s
+Step 49220/50462 | Loss: 2.5059 | LR: 7.70e-07 | 5203.5ms/step | 11.5 samples/s
+Step 49230/50462 | Loss: 2.5070 | LR: 7.57e-07 | 5172.4ms/step | 11.6 samples/s
+Step 49240/50462 | Loss: 2.4977 | LR: 7.45e-07 | 5174.3ms/step | 11.6 samples/s
+Step 49250/50462 | Loss: 2.4913 | LR: 7.33e-07 | 5186.9ms/step | 11.6 samples/s
+Validation at step 49250: Loss = 2.5048, Perplexity = 12.24
+Step 49260/50462 | Loss: 2.4645 | LR: 7.21e-07 | 7761.1ms/step | 7.7 samples/s
+Step 49270/50462 | Loss: 2.4834 | LR: 7.09e-07 | 5207.6ms/step | 11.5 samples/s
+Step 49280/50462 | Loss: 2.4673 | LR: 6.97e-07 | 5211.2ms/step | 11.5 samples/s
+Step 49290/50462 | Loss: 2.4887 | LR: 6.85e-07 | 5219.3ms/step | 11.5 samples/s
+Step 49300/50462 | Loss: 2.4861 | LR: 6.74e-07 | 5200.0ms/step | 11.5 samples/s
+Step 49310/50462 | Loss: 2.4923 | LR: 6.62e-07 | 5217.7ms/step | 11.5 samples/s
+Step 49320/50462 | Loss: 2.4807 | LR: 6.51e-07 | 5205.4ms/step | 11.5 samples/s
+Step 49330/50462 | Loss: 2.4891 | LR: 6.39e-07 | 5201.4ms/step | 11.5 samples/s
+Step 49340/50462 | Loss: 2.4990 | LR: 6.28e-07 | 5198.8ms/step | 11.5 samples/s
+Step 49350/50462 | Loss: 2.5018 | LR: 6.17e-07 | 5202.2ms/step | 11.5 samples/s
+Step 49360/50462 | Loss: 2.5120 | LR: 6.06e-07 | 5254.6ms/step | 11.4 samples/s
+Step 49370/50462 | Loss: 2.4944 | LR: 5.95e-07 | 5281.1ms/step | 11.4 samples/s
+Step 49380/50462 | Loss: 2.5072 | LR: 5.84e-07 | 5204.4ms/step | 11.5 samples/s
+Step 49390/50462 | Loss: 2.5067 | LR: 5.73e-07 | 5189.2ms/step | 11.6 samples/s
+Step 49400/50462 | Loss: 2.5044 | LR: 5.63e-07 | 5201.3ms/step | 11.5 samples/s
+Step 49410/50462 | Loss: 2.4963 | LR: 5.52e-07 | 5205.5ms/step | 11.5 samples/s
+Step 49420/50462 | Loss: 2.4938 | LR: 5.42e-07 | 5201.7ms/step | 11.5 samples/s
+Step 49430/50462 | Loss: 2.4733 | LR: 5.31e-07 | 5212.6ms/step | 11.5 samples/s
+Step 49440/50462 | Loss: 2.4831 | LR: 5.21e-07 | 5194.1ms/step | 11.6 samples/s
+Step 49450/50462 | Loss: 2.4697 | LR: 5.11e-07 | 5198.4ms/step | 11.5 samples/s
+Step 49460/50462 | Loss: 2.5039 | LR: 5.01e-07 | 5198.0ms/step | 11.5 samples/s
+Step 49470/50462 | Loss: 2.5056 | LR: 4.91e-07 | 5208.5ms/step | 11.5 samples/s
+Step 49480/50462 | Loss: 2.4989 | LR: 4.81e-07 | 5201.2ms/step | 11.5 samples/s
+Step 49490/50462 | Loss: 2.4792 | LR: 4.72e-07 | 5228.3ms/step | 11.5 samples/s
+Step 49500/50462 | Loss: 2.5000 | LR: 4.62e-07 | 5213.2ms/step | 11.5 samples/s
+Validation at step 49500: Loss = 2.5048, Perplexity = 12.24
+Step 49510/50462 | Loss: 2.5094 | LR: 4.52e-07 | 7766.6ms/step | 7.7 samples/s
+Step 49520/50462 | Loss: 2.4892 | LR: 4.43e-07 | 5214.1ms/step | 11.5 samples/s
+Step 49530/50462 | Loss: 2.4930 | LR: 4.34e-07 | 5219.1ms/step | 11.5 samples/s
+Step 49540/50462 | Loss: 2.4851 | LR: 4.24e-07 | 5219.1ms/step | 11.5 samples/s
+Step 49550/50462 | Loss: 2.4914 | LR: 4.15e-07 | 5211.8ms/step | 11.5 samples/s
+Step 49560/50462 | Loss: 2.5069 | LR: 4.06e-07 | 5189.7ms/step | 11.6 samples/s
+Step 49570/50462 | Loss: 2.5093 | LR: 3.97e-07 | 5196.5ms/step | 11.5 samples/s
+Step 49580/50462 | Loss: 2.5088 | LR: 3.88e-07 | 5215.3ms/step | 11.5 samples/s
+Step 49590/50462 | Loss: 2.4824 | LR: 3.79e-07 | 5231.3ms/step | 11.5 samples/s
+Step 49600/50462 | Loss: 2.5132 | LR: 3.71e-07 | 5214.2ms/step | 11.5 samples/s
+Step 49610/50462 | Loss: 2.4931 | LR: 3.62e-07 | 5218.9ms/step | 11.5 samples/s
+Step 49620/50462 | Loss: 2.4921 | LR: 3.54e-07 | 5213.7ms/step | 11.5 samples/s
+Step 49630/50462 | Loss: 2.4922 | LR: 3.45e-07 | 5202.0ms/step | 11.5 samples/s
+Step 49640/50462 | Loss: 2.4949 | LR: 3.37e-07 | 5208.8ms/step | 11.5 samples/s
+Step 49650/50462 | Loss: 2.4983 | LR: 3.29e-07 | 5212.5ms/step | 11.5 samples/s
+Step 49660/50462 | Loss: 2.4813 | LR: 3.21e-07 | 5264.4ms/step | 11.4 samples/s
+Step 49670/50462 | Loss: 2.4683 | LR: 3.13e-07 | 5246.9ms/step | 11.4 samples/s
+Step 49680/50462 | Loss: 2.5133 | LR: 3.05e-07 | 5250.4ms/step | 11.4 samples/s
+Step 49690/50462 | Loss: 2.5307 | LR: 2.97e-07 | 5206.3ms/step | 11.5 samples/s
+Step 49700/50462 | Loss: 2.5030 | LR: 2.90e-07 | 5206.5ms/step | 11.5 samples/s
+Step 49710/50462 | Loss: 2.4964 | LR: 2.82e-07 | 5202.6ms/step | 11.5 samples/s
+Step 49720/50462 | Loss: 2.4867 | LR: 2.75e-07 | 5209.3ms/step | 11.5 samples/s
+Step 49730/50462 | Loss: 2.5033 | LR: 2.67e-07 | 5211.5ms/step | 11.5 samples/s
+Step 49740/50462 | Loss: 2.5229 | LR: 2.60e-07 | 5207.6ms/step | 11.5 samples/s
+Step 49750/50462 | Loss: 2.4949 | LR: 2.53e-07 | 5204.6ms/step | 11.5 samples/s
+Validation at step 49750: Loss = 2.5048, Perplexity = 12.24
+Step 49760/50462 | Loss: 2.5122 | LR: 2.46e-07 | 7766.1ms/step | 7.7 samples/s
+Step 49770/50462 | Loss: 2.5014 | LR: 2.39e-07 | 5208.3ms/step | 11.5 samples/s
+Step 49780/50462 | Loss: 2.4859 | LR: 2.32e-07 | 5203.7ms/step | 11.5 samples/s
+Step 49790/50462 | Loss: 2.4796 | LR: 2.25e-07 | 5201.7ms/step | 11.5 samples/s
+Step 49800/50462 | Loss: 2.5217 | LR: 2.19e-07 | 5198.6ms/step | 11.5 samples/s
+Step 49810/50462 | Loss: 2.5085 | LR: 2.12e-07 | 5208.5ms/step | 11.5 samples/s
+Step 49820/50462 | Loss: 2.5062 | LR: 2.06e-07 | 5196.0ms/step | 11.5 samples/s
+Step 49830/50462 | Loss: 2.4951 | LR: 1.99e-07 | 5202.0ms/step | 11.5 samples/s
+Step 49840/50462 | Loss: 2.5099 | LR: 1.93e-07 | 5208.0ms/step | 11.5 samples/s
+Step 49850/50462 | Loss: 2.4864 | LR: 1.87e-07 | 5214.4ms/step | 11.5 samples/s
+Step 49860/50462 | Loss: 2.4863 | LR: 1.81e-07 | 5209.4ms/step | 11.5 samples/s
+Step 49870/50462 | Loss: 2.5005 | LR: 1.75e-07 | 5200.5ms/step | 11.5 samples/s
+Step 49880/50462 | Loss: 2.4865 | LR: 1.69e-07 | 5217.1ms/step | 11.5 samples/s
+Step 49890/50462 | Loss: 2.4959 | LR: 1.63e-07 | 5195.7ms/step | 11.5 samples/s
+Step 49900/50462 | Loss: 2.5084 | LR: 1.58e-07 | 5207.5ms/step | 11.5 samples/s
+Step 49910/50462 | Loss: 2.4943 | LR: 1.52e-07 | 5214.3ms/step | 11.5 samples/s
+Step 49920/50462 | Loss: 2.5087 | LR: 1.47e-07 | 5206.3ms/step | 11.5 samples/s
+Step 49930/50462 | Loss: 2.4796 | LR: 1.41e-07 | 5199.5ms/step | 11.5 samples/s
+Step 49940/50462 | Loss: 2.4918 | LR: 1.36e-07 | 5213.2ms/step | 11.5 samples/s

epfl_final/wandb/run-20260604_030818-8igthj1g/files/requirements.txt

@@ -0,0 +1,122 @@
+word2number==1.1
+sqlitedict==2.1.0
+pytz==2026.2
+nvidia-cusparselt-cu13==0.8.1
+mpmath==1.3.0
+cuda-toolkit==13.0.2
+zstandard==0.25.0
+xxhash==3.7.0
+urllib3==2.7.0
+tzdata==2026.2
+typing_extensions==4.15.0
+triton==3.7.0
+tqdm==4.67.3
+torchinfo==1.8.0
+threadpoolctl==3.6.0
+tcolorpy==0.1.7
+tabulate==0.10.0
+sympy==1.14.0
+smmap==5.0.3
+six==1.17.0
+shellingham==1.5.4
+safetensors==0.7.0
+regex==2026.5.9
+PyYAML==6.0.3
+Pygments==2.20.0
+pyarrow==24.0.0
+psutil==7.2.2
+protobuf==7.35.0
+propcache==0.5.2
+portalocker==3.2.0
+platformdirs==4.10.0
+pathvalidate==3.3.1
+packaging==26.2
+nvidia-nvtx==13.0.85
+nvidia-nvshmem-cu13==3.4.5
+nvidia-nvjitlink==13.0.88
+nvidia-nccl-cu13==2.29.7
+nvidia-curand==10.4.0.35
+nvidia-cufile==1.15.1.6
+nvidia-cuda-runtime==13.0.96
+nvidia-cuda-nvrtc==13.0.88
+nvidia-cuda-cupti==13.0.85
+numpy==2.2.6
+networkx==3.4.2
+more-itertools==11.1.0
+mdurl==0.1.2
+MarkupSafe==3.0.3
+lxml==6.1.1
+joblib==1.5.3
+idna==3.18
+hf-xet==1.5.0
+h11==0.16.0
+fsspec==2026.2.0
+frozenlist==1.8.0
+filelock==3.29.1
+dill==0.4.1
+cuda-pathfinder==1.5.5
+colorama==0.4.6
+click==8.4.1
+charset-normalizer==3.4.7
+chardet==6.0.0.post1
+certifi==2026.5.20
+attrs==26.1.0
+async-timeout==5.0.1
+annotated-types==0.7.0
+annotated-doc==0.0.4
+aiohappyeyeballs==2.6.2
+absl-py==2.4.0
+typing-inspection==0.4.2
+sentry-sdk==2.61.1
+scipy==1.15.3
+sacrebleu==2.6.0
+requests==2.34.2
+python-dateutil==2.9.0.post0
+pydantic_core==2.46.4
+nvidia-cusparse==12.6.3.3
+nvidia-cufft==12.0.0.61
+nvidia-cublas==13.1.1.3
+nltk==3.9.4
+multiprocess==0.70.19
+multidict==6.7.1
+mbstrdecoder==1.1.5
+markdown-it-py==4.2.0
+Jinja2==3.1.6
+httpcore==1.0.9
+gitdb==4.0.12
+exceptiongroup==1.3.1
+cuda-bindings==13.3.1
+aiosignal==1.4.0
+yarl==1.24.2
+typepy==1.3.5
+scikit-learn==1.7.2
+rouge-score==0.1.2
+rich==15.0.0
+pydantic==2.13.4
+pandas==2.3.3
+nvidia-cusolver==12.0.4.66
+nvidia-cudnn-cu13==9.20.0.48
+GitPython==3.1.50
+anyio==4.13.0
+wandb==0.27.0
+typer==0.25.1
+httpx==0.28.1
+aiohttp==3.14.0
+torch==2.12.0
+huggingface_hub==1.17.0
+DataProperty==1.1.1
+tokenizers==0.22.2
+tabledata==1.3.5
+datasets==4.8.5
+accelerate==1.13.0
+transformers==5.10.1
+pytablewriter==1.2.1
+evaluate==0.4.6
+lm_eval==0.4.12
+PyGObject==3.42.1
+dbus-python==1.2.18
+distro==1.7.0
+pip==22.0.2
+setuptools==59.6.0
+ssh-import-id==5.11
+wheel==0.37.1

epfl_final/wandb/run-20260604_030818-8igthj1g/files/wandb-metadata.json

@@ -0,0 +1,54 @@
+{
+  "os": "Linux-6.8.0-59-generic-x86_64-with-glibc2.35",
+  "python": "CPython 3.10.12",
+  "startedAt": "2026-06-04T03:08:18.485136Z",
+  "args": [
+    "--config",
+    "SparseAtt/configs/train_config_dense_resume.json"
+  ],
+  "program": "/mloscratch/homes/navasard/sparse/SparseAtt/train.py",
+  "codePath": "SparseAtt/train.py",
+  "codePathLocal": "SparseAtt/train.py",
+  "email": "ruben.navasardyan@mail.utoronto.ca",
+  "root": "/mloscratch/homes/navasard/sparse",
+  "host": "h100-train-3-gpu-0-0",
+  "executable": "/usr/bin/python3",
+  "cpu_count": 96,
+  "cpu_count_logical": 192,
+  "gpu": "NVIDIA H100 80GB HBM3",
+  "gpu_count": 3,
+  "disk": {
+    "/": {
+      "total": "9374931091456",
+      "used": "6884407656448"
+    }
+  },
+  "memory": {
+    "total": "1623005847552"
+  },
+  "gpu_nvidia": [
+    {
+      "name": "NVIDIA H100 80GB HBM3",
+      "memoryTotal": "85520809984",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-337db0d9-bb90-c768-bb02-7b5dfde9a5d1"
+    },
+    {
+      "name": "NVIDIA H100 80GB HBM3",
+      "memoryTotal": "85520809984",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-129f6620-c2e8-b738-a48f-5b9473847237"
+    },
+    {
+      "name": "NVIDIA H100 80GB HBM3",
+      "memoryTotal": "85520809984",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-3217a779-c453-e309-b765-27ba1809f528"
+    }
+  ],
+  "cudaVersion": "13.0",
+  "writerId": "tkprplutm5g2w6x36c79wg8cwpe8u24d"
+}

epfl_final/wandb/run-20260604_030818-8igthj1g/logs/debug.log

@@ -0,0 +1,20 @@
+2026-06-04 03:08:18,501 INFO    MainThread:3193 [wandb_setup.py:_flush():81] Current SDK version is 0.27.0
+2026-06-04 03:08:18,501 INFO    MainThread:3193 [wandb_setup.py:_flush():81] Configure stats pid to 3193
+2026-06-04 03:08:18,501 INFO    MainThread:3193 [wandb_setup.py:_flush():81] Loading settings from environment variables
+2026-06-04 03:08:18,502 INFO    MainThread:3193 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /mloscratch/homes/navasard/sparse/wandb/run-20260604_030818-8igthj1g/logs/debug.log
+2026-06-04 03:08:18,502 INFO    MainThread:3193 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /mloscratch/homes/navasard/sparse/wandb/run-20260604_030818-8igthj1g/logs/debug-internal.log
+2026-06-04 03:08:18,503 INFO    MainThread:3193 [wandb_init.py:init():851] calling init triggers
+2026-06-04 03:08:18,504 INFO    MainThread:3193 [wandb_init.py:init():856] wandb.init called with sweep_config: {}
+config: {'model_path': '/mloscratch/homes/navasard/qwen3_06b_base', 'model_dtype': 'bfloat16', 'init_from_scratch': True, 'train_path': '/mloscratch/homes/navasard/data/fineweb_large/tokenized', 'val_path': None, 'max_seq_length': 8192, 'num_workers': 8, 'output_dir': '/mloscratch/homes/navasard/sparse/checkpoints/run_20260602_050252', 'resume_from_checkpoint': '/mloscratch/homes/navasard/sparse/checkpoints/run_20260602_050252/checkpoint-44000', 'force_new_run': False, 'num_epochs': 1, 'max_steps': None, 'per_device_batch_size': 2, 'gradient_accumulation_steps': 10, 'learning_rate': 0.0005, 'weight_decay': 0.01, 'warmup_steps': 750, 'lr_scheduler': 'cosine', 'max_grad_norm': 1.0, 'save_steps': 1000, 'eval_steps': 250, 'eval_batches': 50, 'logging_steps': 10, 'seed': 42, 'verbose': True, 'checkpoint_activations': False, 'strategy': 'ddp', 'mixed_precision': 'bf16', 'wandb_enabled': True, 'wandb_project': 'stochastic_sparsity', 'wandb_name': 'dense continued 20k', 'wandb_tags': ['qwen3', '0.6b', 'dense-resume'], 'hidden_size': 1536, 'intermediate_size': 6144, 'num_attention_heads': 12, 'num_key_value_heads': 4, 'num_hidden_layers': None, 'head_dim': None, 'attention_type': 'sparse_stochastic', 'sparse_block_size': 128, 'sparse_num_random_blocks': 31, 'sparse_mask_pool_size': 10, 'sparse_val_dense_compare': False, 'sparse_layer_indices': None, 'sparse_mask_cache_dir': None, 'dense_mode': True, 'dense_step_intervals': None, '_wandb': {}}
+2026-06-04 03:08:18,504 INFO    MainThread:3193 [wandb_init.py:init():899] starting backend
+2026-06-04 03:08:18,725 INFO    MainThread:3193 [wandb_init.py:init():914] sending inform_init request
+2026-06-04 03:08:19,038 INFO    MainThread:3193 [wandb_init.py:init():919] backend started and connected
+2026-06-04 03:08:19,044 INFO    MainThread:3193 [wandb_init.py:init():989] updated telemetry
+2026-06-04 03:08:19,046 INFO    MainThread:3193 [wandb_init.py:init():1012] communicating run to backend with 90.0 second timeout
+2026-06-04 03:08:19,524 INFO    MainThread:3193 [wandb_init.py:init():1052] run resumed
+2026-06-04 03:08:19,527 INFO    MainThread:3193 [wandb_init.py:init():1057] starting run threads in backend
+2026-06-04 03:08:19,630 INFO    MainThread:3193 [wandb_run.py:_console_start():2509] atexit reg
+2026-06-04 03:08:19,630 INFO    MainThread:3193 [wandb_run.py:_redirect():2359] redirect: wrap_raw
+2026-06-04 03:08:19,630 INFO    MainThread:3193 [wandb_run.py:_redirect():2428] Wrapping output streams.
+2026-06-04 03:08:19,630 INFO    MainThread:3193 [wandb_run.py:_redirect():2451] Redirects installed.
+2026-06-04 03:08:19,632 INFO    MainThread:3193 [wandb_init.py:init():1095] run started, returning control to user process

epfl_final/wandb/run-20260604_030818-8igthj1g/run-8igthj1g.wandb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:11ff077ce4d1fb4504b2a16af88bafd9bd3210bb23d42563079b3990f9f04b9b
+size 8650752

epfl_final/wandb/run-20260605_220539-ns4ua5fr/files/output.log

@@ -0,0 +1,1194 @@
+Starting training...
+Validation at step 10000: Loss = 2.7720, Perplexity = 15.99
+Switching attention mode for step 10001: dense (dense step interval)
+Step 10010/50462 | Loss: 2.7439 | LR: 4.58e-04 | 29087.8ms/step | 2.1 samples/s
+Step 10020/50462 | Loss: 2.7797 | LR: 4.58e-04 | 5200.8ms/step | 11.5 samples/s
+Step 10030/50462 | Loss: 2.7772 | LR: 4.58e-04 | 5220.2ms/step | 11.5 samples/s
+Step 10040/50462 | Loss: 2.7495 | LR: 4.58e-04 | 5215.1ms/step | 11.5 samples/s
+Step 10050/50462 | Loss: 2.7626 | LR: 4.58e-04 | 5222.6ms/step | 11.5 samples/s
+Step 10060/50462 | Loss: 2.7464 | LR: 4.58e-04 | 5225.3ms/step | 11.5 samples/s
+Step 10070/50462 | Loss: 2.7658 | LR: 4.58e-04 | 5227.3ms/step | 11.5 samples/s
+Step 10080/50462 | Loss: 2.7394 | LR: 4.58e-04 | 5218.9ms/step | 11.5 samples/s
+Step 10090/50462 | Loss: 2.7746 | LR: 4.58e-04 | 5223.7ms/step | 11.5 samples/s
+Step 10100/50462 | Loss: 2.7574 | LR: 4.58e-04 | 5233.2ms/step | 11.5 samples/s
+Step 10110/50462 | Loss: 2.7551 | LR: 4.58e-04 | 5224.5ms/step | 11.5 samples/s
+Step 10120/50462 | Loss: 2.7431 | LR: 4.57e-04 | 5225.0ms/step | 11.5 samples/s
+Step 10130/50462 | Loss: 2.7418 | LR: 4.57e-04 | 5220.0ms/step | 11.5 samples/s
+Step 10140/50462 | Loss: 2.7661 | LR: 4.57e-04 | 5223.7ms/step | 11.5 samples/s
+Step 10150/50462 | Loss: 2.7634 | LR: 4.57e-04 | 5235.1ms/step | 11.5 samples/s
+Step 10160/50462 | Loss: 2.7393 | LR: 4.57e-04 | 5213.0ms/step | 11.5 samples/s
+Step 10170/50462 | Loss: 2.7410 | LR: 4.57e-04 | 5225.8ms/step | 11.5 samples/s
+Step 10180/50462 | Loss: 2.7288 | LR: 4.57e-04 | 5221.6ms/step | 11.5 samples/s
+Step 10190/50462 | Loss: 2.7675 | LR: 4.57e-04 | 5212.2ms/step | 11.5 samples/s
+Step 10200/50462 | Loss: 2.7164 | LR: 4.57e-04 | 5228.7ms/step | 11.5 samples/s
+Step 10210/50462 | Loss: 2.7459 | LR: 4.57e-04 | 5228.4ms/step | 11.5 samples/s
+Step 10220/50462 | Loss: 2.7364 | LR: 4.57e-04 | 5215.8ms/step | 11.5 samples/s
+Step 10230/50462 | Loss: 2.7352 | LR: 4.56e-04 | 5224.1ms/step | 11.5 samples/s
+Step 10240/50462 | Loss: 2.7671 | LR: 4.56e-04 | 5226.0ms/step | 11.5 samples/s
+Step 10250/50462 | Loss: 2.7302 | LR: 4.56e-04 | 5210.3ms/step | 11.5 samples/s
+Validation at step 10250: Loss = 2.7521, Perplexity = 15.68
+Step 10260/50462 | Loss: 2.7310 | LR: 4.56e-04 | 7804.3ms/step | 7.7 samples/s
+Step 10270/50462 | Loss: 2.7260 | LR: 4.56e-04 | 5235.5ms/step | 11.5 samples/s
+Step 10280/50462 | Loss: 2.7668 | LR: 4.56e-04 | 5225.5ms/step | 11.5 samples/s
+Step 10290/50462 | Loss: 2.7675 | LR: 4.56e-04 | 5202.7ms/step | 11.5 samples/s
+Step 10300/50462 | Loss: 2.7343 | LR: 4.56e-04 | 5211.6ms/step | 11.5 samples/s
+Step 10310/50462 | Loss: 2.7144 | LR: 4.56e-04 | 5221.1ms/step | 11.5 samples/s
+Step 10320/50462 | Loss: 2.7214 | LR: 4.56e-04 | 5196.1ms/step | 11.5 samples/s
+Step 10330/50462 | Loss: 2.7464 | LR: 4.56e-04 | 5208.5ms/step | 11.5 samples/s
+Step 10340/50462 | Loss: 2.7345 | LR: 4.55e-04 | 5210.1ms/step | 11.5 samples/s
+Step 10350/50462 | Loss: 2.7377 | LR: 4.55e-04 | 5211.2ms/step | 11.5 samples/s
+Step 10360/50462 | Loss: 2.7684 | LR: 4.55e-04 | 5216.7ms/step | 11.5 samples/s
+Step 10370/50462 | Loss: 2.7402 | LR: 4.55e-04 | 5211.4ms/step | 11.5 samples/s
+Step 10380/50462 | Loss: 2.7357 | LR: 4.55e-04 | 5204.0ms/step | 11.5 samples/s
+Step 10390/50462 | Loss: 2.7447 | LR: 4.55e-04 | 5163.5ms/step | 11.6 samples/s
+Step 10400/50462 | Loss: 2.7315 | LR: 4.55e-04 | 5172.1ms/step | 11.6 samples/s
+Step 10410/50462 | Loss: 2.7554 | LR: 4.55e-04 | 5179.8ms/step | 11.6 samples/s
+Step 10420/50462 | Loss: 2.7230 | LR: 4.55e-04 | 5192.3ms/step | 11.6 samples/s
+Step 10430/50462 | Loss: 2.7319 | LR: 4.55e-04 | 5189.4ms/step | 11.6 samples/s
+Step 10440/50462 | Loss: 2.7324 | LR: 4.55e-04 | 5189.6ms/step | 11.6 samples/s
+Step 10450/50462 | Loss: 2.7509 | LR: 4.54e-04 | 5178.3ms/step | 11.6 samples/s
+Step 10460/50462 | Loss: 2.7334 | LR: 4.54e-04 | 5196.9ms/step | 11.5 samples/s
+Step 10470/50462 | Loss: 2.7615 | LR: 4.54e-04 | 5185.0ms/step | 11.6 samples/s
+Step 10480/50462 | Loss: 2.7271 | LR: 4.54e-04 | 5185.3ms/step | 11.6 samples/s
+Step 10490/50462 | Loss: 2.7530 | LR: 4.54e-04 | 5180.7ms/step | 11.6 samples/s
+Step 10500/50462 | Loss: 2.7683 | LR: 4.54e-04 | 5188.0ms/step | 11.6 samples/s
+Validation at step 10500: Loss = 2.7463, Perplexity = 15.58
+Step 10510/50462 | Loss: 2.7257 | LR: 4.54e-04 | 7745.3ms/step | 7.7 samples/s
+Step 10520/50462 | Loss: 2.7268 | LR: 4.54e-04 | 5197.8ms/step | 11.5 samples/s
+Step 10530/50462 | Loss: 2.7371 | LR: 4.54e-04 | 5174.5ms/step | 11.6 samples/s
+Step 10540/50462 | Loss: 2.7140 | LR: 4.54e-04 | 5167.0ms/step | 11.6 samples/s
+Step 10550/50462 | Loss: 2.7235 | LR: 4.54e-04 | 5169.0ms/step | 11.6 samples/s
+Step 10560/50462 | Loss: 2.7255 | LR: 4.53e-04 | 5176.6ms/step | 11.6 samples/s
+Step 10570/50462 | Loss: 2.7424 | LR: 4.53e-04 | 5164.0ms/step | 11.6 samples/s
+Step 10580/50462 | Loss: 2.7596 | LR: 4.53e-04 | 5183.6ms/step | 11.6 samples/s
+Step 10590/50462 | Loss: 2.7233 | LR: 4.53e-04 | 5177.0ms/step | 11.6 samples/s
+Step 10600/50462 | Loss: 2.7426 | LR: 4.53e-04 | 5172.3ms/step | 11.6 samples/s
+Step 10610/50462 | Loss: 2.7604 | LR: 4.53e-04 | 5177.8ms/step | 11.6 samples/s
+Step 10620/50462 | Loss: 2.7261 | LR: 4.53e-04 | 5171.9ms/step | 11.6 samples/s
+Step 10630/50462 | Loss: 2.7294 | LR: 4.53e-04 | 5171.4ms/step | 11.6 samples/s
+Step 10640/50462 | Loss: 2.7402 | LR: 4.53e-04 | 5175.6ms/step | 11.6 samples/s
+Step 10650/50462 | Loss: 2.7396 | LR: 4.53e-04 | 5177.0ms/step | 11.6 samples/s
+Step 10660/50462 | Loss: 2.7340 | LR: 4.53e-04 | 5172.9ms/step | 11.6 samples/s
+Step 10670/50462 | Loss: 2.7282 | LR: 4.52e-04 | 5173.7ms/step | 11.6 samples/s
+Step 10680/50462 | Loss: 2.7300 | LR: 4.52e-04 | 5170.0ms/step | 11.6 samples/s
+Step 10690/50462 | Loss: 2.7241 | LR: 4.52e-04 | 5184.0ms/step | 11.6 samples/s
+Step 10700/50462 | Loss: 2.7239 | LR: 4.52e-04 | 5183.9ms/step | 11.6 samples/s
+Step 10710/50462 | Loss: 2.7178 | LR: 4.52e-04 | 5172.7ms/step | 11.6 samples/s
+Step 10720/50462 | Loss: 2.7296 | LR: 4.52e-04 | 5191.2ms/step | 11.6 samples/s
+Step 10730/50462 | Loss: 2.7636 | LR: 4.52e-04 | 5181.0ms/step | 11.6 samples/s
+Step 10740/50462 | Loss: 2.7464 | LR: 4.52e-04 | 5181.9ms/step | 11.6 samples/s
+Step 10750/50462 | Loss: 2.7211 | LR: 4.52e-04 | 5174.4ms/step | 11.6 samples/s
+Validation at step 10750: Loss = 2.7425, Perplexity = 15.53
+Step 10760/50462 | Loss: 2.6995 | LR: 4.52e-04 | 7756.0ms/step | 7.7 samples/s
+Step 10770/50462 | Loss: 2.7575 | LR: 4.52e-04 | 5171.2ms/step | 11.6 samples/s
+Step 10780/50462 | Loss: 2.7417 | LR: 4.51e-04 | 5191.4ms/step | 11.6 samples/s
+Step 10790/50462 | Loss: 2.7432 | LR: 4.51e-04 | 5190.7ms/step | 11.6 samples/s
+Step 10800/50462 | Loss: 2.7052 | LR: 4.51e-04 | 5194.3ms/step | 11.6 samples/s
+Step 10810/50462 | Loss: 2.7204 | LR: 4.51e-04 | 5199.9ms/step | 11.5 samples/s
+Step 10820/50462 | Loss: 2.7459 | LR: 4.51e-04 | 5185.9ms/step | 11.6 samples/s
+Step 10830/50462 | Loss: 2.7453 | LR: 4.51e-04 | 5206.7ms/step | 11.5 samples/s
+Step 10840/50462 | Loss: 2.7123 | LR: 4.51e-04 | 5206.8ms/step | 11.5 samples/s
+Step 10850/50462 | Loss: 2.7024 | LR: 4.51e-04 | 5193.9ms/step | 11.6 samples/s
+Step 10860/50462 | Loss: 2.7525 | LR: 4.51e-04 | 5200.1ms/step | 11.5 samples/s
+Step 10870/50462 | Loss: 2.7346 | LR: 4.51e-04 | 5196.0ms/step | 11.5 samples/s
+Step 10880/50462 | Loss: 2.7480 | LR: 4.50e-04 | 5184.8ms/step | 11.6 samples/s
+Step 10890/50462 | Loss: 2.7298 | LR: 4.50e-04 | 5182.0ms/step | 11.6 samples/s
+Step 10900/50462 | Loss: 2.7261 | LR: 4.50e-04 | 5183.5ms/step | 11.6 samples/s
+Step 10910/50462 | Loss: 2.7302 | LR: 4.50e-04 | 5186.1ms/step | 11.6 samples/s
+Step 10920/50462 | Loss: 2.7479 | LR: 4.50e-04 | 5197.7ms/step | 11.5 samples/s
+Step 10930/50462 | Loss: 2.7466 | LR: 4.50e-04 | 5196.9ms/step | 11.5 samples/s
+Step 10940/50462 | Loss: 2.6898 | LR: 4.50e-04 | 5183.6ms/step | 11.6 samples/s
+Step 10950/50462 | Loss: 2.7607 | LR: 4.50e-04 | 5188.9ms/step | 11.6 samples/s
+Step 10960/50462 | Loss: 2.7280 | LR: 4.50e-04 | 5199.3ms/step | 11.5 samples/s
+Step 10970/50462 | Loss: 2.7258 | LR: 4.50e-04 | 5196.1ms/step | 11.5 samples/s
+Step 10980/50462 | Loss: 2.7291 | LR: 4.50e-04 | 5196.3ms/step | 11.5 samples/s
+Step 10990/50462 | Loss: 2.7248 | LR: 4.49e-04 | 5193.9ms/step | 11.6 samples/s
+Step 11000/50462 | Loss: 2.7367 | LR: 4.49e-04 | 5192.6ms/step | 11.6 samples/s
+Validation at step 11000: Loss = 2.7349, Perplexity = 15.41
+Saved checkpoint to checkpoints/run_20260605_220251/checkpoint-11000
+Step 11010/50462 | Loss: 2.7062 | LR: 4.49e-04 | 8701.7ms/step | 6.9 samples/s
+Step 11020/50462 | Loss: 2.7148 | LR: 4.49e-04 | 5631.7ms/step | 10.7 samples/s
+Step 11030/50462 | Loss: 2.7058 | LR: 4.49e-04 | 5191.8ms/step | 11.6 samples/s
+Step 11040/50462 | Loss: 2.7374 | LR: 4.49e-04 | 5179.3ms/step | 11.6 samples/s
+Step 11050/50462 | Loss: 2.7585 | LR: 4.49e-04 | 6118.1ms/step | 9.8 samples/s
+Step 11060/50462 | Loss: 2.7333 | LR: 4.49e-04 | 5173.9ms/step | 11.6 samples/s
+Step 11070/50462 | Loss: 2.7364 | LR: 4.49e-04 | 5142.2ms/step | 11.7 samples/s
+Step 11080/50462 | Loss: 2.7042 | LR: 4.49e-04 | 5138.6ms/step | 11.7 samples/s
+Step 11090/50462 | Loss: 2.7241 | LR: 4.48e-04 | 5128.5ms/step | 11.7 samples/s
+Step 11100/50462 | Loss: 2.7213 | LR: 4.48e-04 | 5165.3ms/step | 11.6 samples/s
+Step 11110/50462 | Loss: 2.7315 | LR: 4.48e-04 | 5136.9ms/step | 11.7 samples/s
+Step 11120/50462 | Loss: 2.7050 | LR: 4.48e-04 | 5150.8ms/step | 11.6 samples/s
+Step 11130/50462 | Loss: 2.7519 | LR: 4.48e-04 | 5133.7ms/step | 11.7 samples/s
+Step 11140/50462 | Loss: 2.7090 | LR: 4.48e-04 | 5135.5ms/step | 11.7 samples/s
+Step 11150/50462 | Loss: 2.7230 | LR: 4.48e-04 | 5139.4ms/step | 11.7 samples/s
+Step 11160/50462 | Loss: 2.7100 | LR: 4.48e-04 | 5140.2ms/step | 11.7 samples/s
+Step 11170/50462 | Loss: 2.7434 | LR: 4.48e-04 | 5139.3ms/step | 11.7 samples/s
+Step 11180/50462 | Loss: 2.7319 | LR: 4.48e-04 | 5152.8ms/step | 11.6 samples/s
+Step 11190/50462 | Loss: 2.7083 | LR: 4.48e-04 | 5161.9ms/step | 11.6 samples/s
+Step 11200/50462 | Loss: 2.7184 | LR: 4.47e-04 | 5163.7ms/step | 11.6 samples/s
+Step 11210/50462 | Loss: 2.7280 | LR: 4.47e-04 | 5158.8ms/step | 11.6 samples/s
+Step 11220/50462 | Loss: 2.7248 | LR: 4.47e-04 | 5155.2ms/step | 11.6 samples/s
+Step 11230/50462 | Loss: 2.7293 | LR: 4.47e-04 | 5159.2ms/step | 11.6 samples/s
+Step 11240/50462 | Loss: 2.7524 | LR: 4.47e-04 | 5148.0ms/step | 11.7 samples/s
+Step 11250/50462 | Loss: 2.7137 | LR: 4.47e-04 | 5141.6ms/step | 11.7 samples/s
+Validation at step 11250: Loss = 2.7305, Perplexity = 15.34
+Step 11260/50462 | Loss: 2.7163 | LR: 4.47e-04 | 7703.1ms/step | 7.8 samples/s
+Step 11270/50462 | Loss: 2.7386 | LR: 4.47e-04 | 5160.2ms/step | 11.6 samples/s
+Step 11280/50462 | Loss: 2.7029 | LR: 4.47e-04 | 5157.0ms/step | 11.6 samples/s
+Step 11290/50462 | Loss: 2.6911 | LR: 4.47e-04 | 5169.2ms/step | 11.6 samples/s
+Step 11300/50462 | Loss: 2.7244 | LR: 4.46e-04 | 5179.9ms/step | 11.6 samples/s
+Step 11310/50462 | Loss: 2.7343 | LR: 4.46e-04 | 5165.8ms/step | 11.6 samples/s
+Step 11320/50462 | Loss: 2.7182 | LR: 4.46e-04 | 5163.7ms/step | 11.6 samples/s
+Step 11330/50462 | Loss: 2.7195 | LR: 4.46e-04 | 5162.9ms/step | 11.6 samples/s
+Step 11340/50462 | Loss: 2.6960 | LR: 4.46e-04 | 5165.6ms/step | 11.6 samples/s
+Step 11350/50462 | Loss: 2.7356 | LR: 4.46e-04 | 5164.2ms/step | 11.6 samples/s
+Step 11360/50462 | Loss: 2.7163 | LR: 4.46e-04 | 5164.3ms/step | 11.6 samples/s
+Step 11370/50462 | Loss: 2.7213 | LR: 4.46e-04 | 5157.2ms/step | 11.6 samples/s
+Step 11380/50462 | Loss: 2.7119 | LR: 4.46e-04 | 5162.5ms/step | 11.6 samples/s
+Step 11390/50462 | Loss: 2.6979 | LR: 4.46e-04 | 5165.7ms/step | 11.6 samples/s
+Step 11400/50462 | Loss: 2.7332 | LR: 4.45e-04 | 5170.5ms/step | 11.6 samples/s
+Step 11410/50462 | Loss: 2.6886 | LR: 4.45e-04 | 5157.1ms/step | 11.6 samples/s
+Step 11420/50462 | Loss: 2.7010 | LR: 4.45e-04 | 5158.7ms/step | 11.6 samples/s
+Step 11430/50462 | Loss: 2.7168 | LR: 4.45e-04 | 5168.5ms/step | 11.6 samples/s
+Step 11440/50462 | Loss: 2.7001 | LR: 4.45e-04 | 5153.0ms/step | 11.6 samples/s
+Step 11450/50462 | Loss: 2.7244 | LR: 4.45e-04 | 5160.2ms/step | 11.6 samples/s
+Step 11460/50462 | Loss: 2.7276 | LR: 4.45e-04 | 5163.8ms/step | 11.6 samples/s
+Step 11470/50462 | Loss: 2.7397 | LR: 4.45e-04 | 5156.1ms/step | 11.6 samples/s
+Step 11480/50462 | Loss: 2.7102 | LR: 4.45e-04 | 5161.3ms/step | 11.6 samples/s
+Step 11490/50462 | Loss: 2.7285 | LR: 4.45e-04 | 5180.9ms/step | 11.6 samples/s
+Step 11500/50462 | Loss: 2.7184 | LR: 4.44e-04 | 5160.8ms/step | 11.6 samples/s
+Validation at step 11500: Loss = 2.7257, Perplexity = 15.27
+Step 11510/50462 | Loss: 2.6852 | LR: 4.44e-04 | 7717.5ms/step | 7.8 samples/s
+Step 11520/50462 | Loss: 2.6838 | LR: 4.44e-04 | 5155.9ms/step | 11.6 samples/s
+Step 11530/50462 | Loss: 2.7219 | LR: 4.44e-04 | 5158.4ms/step | 11.6 samples/s
+Step 11540/50462 | Loss: 2.7333 | LR: 4.44e-04 | 5162.4ms/step | 11.6 samples/s
+Step 11550/50462 | Loss: 2.6951 | LR: 4.44e-04 | 5166.0ms/step | 11.6 samples/s
+Step 11560/50462 | Loss: 2.7341 | LR: 4.44e-04 | 5167.1ms/step | 11.6 samples/s
+Step 11570/50462 | Loss: 2.7114 | LR: 4.44e-04 | 5166.6ms/step | 11.6 samples/s
+Step 11580/50462 | Loss: 2.7105 | LR: 4.44e-04 | 5158.0ms/step | 11.6 samples/s
+Step 11590/50462 | Loss: 2.7262 | LR: 4.44e-04 | 5158.2ms/step | 11.6 samples/s
+Step 11600/50462 | Loss: 2.7071 | LR: 4.43e-04 | 5141.7ms/step | 11.7 samples/s
+Step 11610/50462 | Loss: 2.7401 | LR: 4.43e-04 | 5150.9ms/step | 11.6 samples/s
+Step 11620/50462 | Loss: 2.6914 | LR: 4.43e-04 | 5149.7ms/step | 11.7 samples/s
+Step 11630/50462 | Loss: 2.7203 | LR: 4.43e-04 | 5162.1ms/step | 11.6 samples/s
+Step 11640/50462 | Loss: 2.7327 | LR: 4.43e-04 | 5159.7ms/step | 11.6 samples/s
+Step 11650/50462 | Loss: 2.6991 | LR: 4.43e-04 | 5155.3ms/step | 11.6 samples/s
+Step 11660/50462 | Loss: 2.7008 | LR: 4.43e-04 | 5158.7ms/step | 11.6 samples/s
+Step 11670/50462 | Loss: 2.7193 | LR: 4.43e-04 | 5196.5ms/step | 11.5 samples/s
+Step 11680/50462 | Loss: 2.7043 | LR: 4.43e-04 | 5151.6ms/step | 11.6 samples/s
+Step 11690/50462 | Loss: 2.7018 | LR: 4.43e-04 | 5163.7ms/step | 11.6 samples/s
+Step 11700/50462 | Loss: 2.7094 | LR: 4.42e-04 | 5156.6ms/step | 11.6 samples/s
+Step 11710/50462 | Loss: 2.7170 | LR: 4.42e-04 | 5164.5ms/step | 11.6 samples/s
+Step 11720/50462 | Loss: 2.7117 | LR: 4.42e-04 | 5166.3ms/step | 11.6 samples/s
+Step 11730/50462 | Loss: 2.7138 | LR: 4.42e-04 | 5164.4ms/step | 11.6 samples/s
+Step 11740/50462 | Loss: 2.7125 | LR: 4.42e-04 | 5160.2ms/step | 11.6 samples/s
+Step 11750/50462 | Loss: 2.7044 | LR: 4.42e-04 | 5150.9ms/step | 11.6 samples/s
+Validation at step 11750: Loss = 2.7218, Perplexity = 15.21
+Step 11760/50462 | Loss: 2.6895 | LR: 4.42e-04 | 7718.7ms/step | 7.8 samples/s
+Step 11770/50462 | Loss: 2.7051 | LR: 4.42e-04 | 5162.7ms/step | 11.6 samples/s
+Step 11780/50462 | Loss: 2.7123 | LR: 4.42e-04 | 5163.5ms/step | 11.6 samples/s
+Step 11790/50462 | Loss: 2.6989 | LR: 4.42e-04 | 5168.4ms/step | 11.6 samples/s
+Step 11800/50462 | Loss: 2.7036 | LR: 4.41e-04 | 5172.6ms/step | 11.6 samples/s
+Step 11810/50462 | Loss: 2.7479 | LR: 4.41e-04 | 5159.0ms/step | 11.6 samples/s
+Step 11820/50462 | Loss: 2.7296 | LR: 4.41e-04 | 5164.4ms/step | 11.6 samples/s
+Step 11830/50462 | Loss: 2.6962 | LR: 4.41e-04 | 5169.1ms/step | 11.6 samples/s
+Step 11840/50462 | Loss: 2.7147 | LR: 4.41e-04 | 5159.7ms/step | 11.6 samples/s
+Step 11850/50462 | Loss: 2.7082 | LR: 4.41e-04 | 5181.0ms/step | 11.6 samples/s
+Step 11860/50462 | Loss: 2.6886 | LR: 4.41e-04 | 5172.1ms/step | 11.6 samples/s
+Step 11870/50462 | Loss: 2.7086 | LR: 4.41e-04 | 5164.0ms/step | 11.6 samples/s
+Step 11880/50462 | Loss: 2.7038 | LR: 4.41e-04 | 5164.1ms/step | 11.6 samples/s
+Step 11890/50462 | Loss: 2.6947 | LR: 4.41e-04 | 5215.3ms/step | 11.5 samples/s
+Step 11900/50462 | Loss: 2.7359 | LR: 4.40e-04 | 5154.4ms/step | 11.6 samples/s
+Step 11910/50462 | Loss: 2.7274 | LR: 4.40e-04 | 5152.8ms/step | 11.6 samples/s
+Step 11920/50462 | Loss: 2.7012 | LR: 4.40e-04 | 5165.6ms/step | 11.6 samples/s
+Step 11930/50462 | Loss: 2.7125 | LR: 4.40e-04 | 5161.9ms/step | 11.6 samples/s
+Step 11940/50462 | Loss: 2.6972 | LR: 4.40e-04 | 5163.7ms/step | 11.6 samples/s
+Step 11950/50462 | Loss: 2.7143 | LR: 4.40e-04 | 5161.5ms/step | 11.6 samples/s
+Step 11960/50462 | Loss: 2.7021 | LR: 4.40e-04 | 5169.6ms/step | 11.6 samples/s
+Step 11970/50462 | Loss: 2.7317 | LR: 4.40e-04 | 5164.1ms/step | 11.6 samples/s
+Step 11980/50462 | Loss: 2.7358 | LR: 4.40e-04 | 5164.1ms/step | 11.6 samples/s
+Step 11990/50462 | Loss: 2.7041 | LR: 4.40e-04 | 5177.2ms/step | 11.6 samples/s
+Step 12000/50462 | Loss: 2.7442 | LR: 4.39e-04 | 5172.6ms/step | 11.6 samples/s
+Validation at step 12000: Loss = 2.7178, Perplexity = 15.15
+Saved checkpoint to checkpoints/run_20260605_220251/checkpoint-12000
+Step 12010/50462 | Loss: 2.7053 | LR: 4.39e-04 | 8730.1ms/step | 6.9 samples/s
+Step 12020/50462 | Loss: 2.7075 | LR: 4.39e-04 | 5205.6ms/step | 11.5 samples/s
+Step 12030/50462 | Loss: 2.7085 | LR: 4.39e-04 | 5187.4ms/step | 11.6 samples/s
+Step 12040/50462 | Loss: 2.7015 | LR: 4.39e-04 | 5184.6ms/step | 11.6 samples/s
+Step 12050/50462 | Loss: 2.6703 | LR: 4.39e-04 | 5179.9ms/step | 11.6 samples/s
+Step 12060/50462 | Loss: 2.7408 | LR: 4.39e-04 | 5190.6ms/step | 11.6 samples/s
+Step 12070/50462 | Loss: 2.7006 | LR: 4.39e-04 | 5211.8ms/step | 11.5 samples/s
+Step 12080/50462 | Loss: 2.6990 | LR: 4.39e-04 | 5173.1ms/step | 11.6 samples/s
+Step 12090/50462 | Loss: 2.7068 | LR: 4.39e-04 | 5182.3ms/step | 11.6 samples/s
+Step 12100/50462 | Loss: 2.7075 | LR: 4.38e-04 | 5167.4ms/step | 11.6 samples/s
+Step 12110/50462 | Loss: 2.7134 | LR: 4.38e-04 | 5179.4ms/step | 11.6 samples/s
+Step 12120/50462 | Loss: 2.6977 | LR: 4.38e-04 | 5162.8ms/step | 11.6 samples/s
+Step 12130/50462 | Loss: 2.6884 | LR: 4.38e-04 | 5158.8ms/step | 11.6 samples/s
+Step 12140/50462 | Loss: 2.6817 | LR: 4.38e-04 | 5136.3ms/step | 11.7 samples/s
+Step 12150/50462 | Loss: 2.6967 | LR: 4.38e-04 | 5140.3ms/step | 11.7 samples/s
+Step 12160/50462 | Loss: 2.7253 | LR: 4.38e-04 | 5147.7ms/step | 11.7 samples/s
+Step 12170/50462 | Loss: 2.7203 | LR: 4.38e-04 | 5139.3ms/step | 11.7 samples/s
+Step 12180/50462 | Loss: 2.6993 | LR: 4.38e-04 | 5141.8ms/step | 11.7 samples/s
+Step 12190/50462 | Loss: 2.7002 | LR: 4.37e-04 | 5148.9ms/step | 11.7 samples/s
+Step 12200/50462 | Loss: 2.6619 | LR: 4.37e-04 | 5151.3ms/step | 11.6 samples/s
+Step 12210/50462 | Loss: 2.6846 | LR: 4.37e-04 | 5144.7ms/step | 11.7 samples/s
+Step 12220/50462 | Loss: 2.6880 | LR: 4.37e-04 | 5132.2ms/step | 11.7 samples/s
+Step 12230/50462 | Loss: 2.7050 | LR: 4.37e-04 | 5142.7ms/step | 11.7 samples/s
+Step 12240/50462 | Loss: 2.7193 | LR: 4.37e-04 | 5150.2ms/step | 11.7 samples/s
+Step 12250/50462 | Loss: 2.6899 | LR: 4.37e-04 | 5148.8ms/step | 11.7 samples/s
+Validation at step 12250: Loss = 2.7142, Perplexity = 15.09
+Step 12260/50462 | Loss: 2.6840 | LR: 4.37e-04 | 7707.1ms/step | 7.8 samples/s
+Step 12270/50462 | Loss: 2.6895 | LR: 4.37e-04 | 5159.9ms/step | 11.6 samples/s
+Step 12280/50462 | Loss: 2.6720 | LR: 4.37e-04 | 5140.9ms/step | 11.7 samples/s
+Step 12290/50462 | Loss: 2.7192 | LR: 4.36e-04 | 5146.1ms/step | 11.7 samples/s
+Step 12300/50462 | Loss: 2.6969 | LR: 4.36e-04 | 5155.2ms/step | 11.6 samples/s
+Step 12310/50462 | Loss: 2.7322 | LR: 4.36e-04 | 5157.0ms/step | 11.6 samples/s
+Step 12320/50462 | Loss: 2.7080 | LR: 4.36e-04 | 5156.6ms/step | 11.6 samples/s
+Step 12330/50462 | Loss: 2.6953 | LR: 4.36e-04 | 5157.6ms/step | 11.6 samples/s
+Step 12340/50462 | Loss: 2.7239 | LR: 4.36e-04 | 5167.5ms/step | 11.6 samples/s
+Step 12350/50462 | Loss: 2.6967 | LR: 4.36e-04 | 5156.9ms/step | 11.6 samples/s
+Step 12360/50462 | Loss: 2.7049 | LR: 4.36e-04 | 5156.2ms/step | 11.6 samples/s
+Step 12370/50462 | Loss: 2.7097 | LR: 4.36e-04 | 5150.3ms/step | 11.6 samples/s
+Step 12380/50462 | Loss: 2.7113 | LR: 4.35e-04 | 5160.4ms/step | 11.6 samples/s
+Step 12390/50462 | Loss: 2.7055 | LR: 4.35e-04 | 5139.0ms/step | 11.7 samples/s
+Step 12400/50462 | Loss: 2.6849 | LR: 4.35e-04 | 5134.1ms/step | 11.7 samples/s
+Step 12410/50462 | Loss: 2.7059 | LR: 4.35e-04 | 5145.2ms/step | 11.7 samples/s
+Step 12420/50462 | Loss: 2.7259 | LR: 4.35e-04 | 5138.9ms/step | 11.7 samples/s
+Step 12430/50462 | Loss: 2.7327 | LR: 4.35e-04 | 5146.4ms/step | 11.7 samples/s
+Step 12440/50462 | Loss: 2.6948 | LR: 4.35e-04 | 5145.6ms/step | 11.7 samples/s
+Step 12450/50462 | Loss: 2.7050 | LR: 4.35e-04 | 5148.9ms/step | 11.7 samples/s
+Step 12460/50462 | Loss: 2.7223 | LR: 4.35e-04 | 5161.6ms/step | 11.6 samples/s
+Step 12470/50462 | Loss: 2.7083 | LR: 4.35e-04 | 5143.5ms/step | 11.7 samples/s
+Step 12480/50462 | Loss: 2.6835 | LR: 4.34e-04 | 5139.1ms/step | 11.7 samples/s
+Step 12490/50462 | Loss: 2.7184 | LR: 4.34e-04 | 5132.1ms/step | 11.7 samples/s
+Step 12500/50462 | Loss: 2.7273 | LR: 4.34e-04 | 5151.9ms/step | 11.6 samples/s
+Validation at step 12500: Loss = 2.7087, Perplexity = 15.01
+Step 12510/50462 | Loss: 2.6781 | LR: 4.34e-04 | 7756.0ms/step | 7.7 samples/s
+Step 12520/50462 | Loss: 2.6875 | LR: 4.34e-04 | 5207.4ms/step | 11.5 samples/s
+Step 12530/50462 | Loss: 2.7148 | LR: 4.34e-04 | 5206.0ms/step | 11.5 samples/s
+Step 12540/50462 | Loss: 2.7272 | LR: 4.34e-04 | 5199.4ms/step | 11.5 samples/s
+Step 12550/50462 | Loss: 2.6728 | LR: 4.34e-04 | 5211.2ms/step | 11.5 samples/s
+Step 12560/50462 | Loss: 2.7174 | LR: 4.34e-04 | 5204.9ms/step | 11.5 samples/s
+Step 12570/50462 | Loss: 2.7003 | LR: 4.33e-04 | 5220.9ms/step | 11.5 samples/s
+Step 12580/50462 | Loss: 2.6936 | LR: 4.33e-04 | 5202.9ms/step | 11.5 samples/s
+Step 12590/50462 | Loss: 2.7125 | LR: 4.33e-04 | 5197.6ms/step | 11.5 samples/s
+Step 12600/50462 | Loss: 2.6741 | LR: 4.33e-04 | 5209.3ms/step | 11.5 samples/s
+Step 12610/50462 | Loss: 2.7006 | LR: 4.33e-04 | 5223.4ms/step | 11.5 samples/s
+Step 12620/50462 | Loss: 2.6844 | LR: 4.33e-04 | 5190.6ms/step | 11.6 samples/s
+Step 12630/50462 | Loss: 2.7192 | LR: 4.33e-04 | 5215.6ms/step | 11.5 samples/s
+Step 12640/50462 | Loss: 2.6864 | LR: 4.33e-04 | 5221.4ms/step | 11.5 samples/s
+Step 12650/50462 | Loss: 2.7081 | LR: 4.33e-04 | 5208.7ms/step | 11.5 samples/s
+Step 12660/50462 | Loss: 2.7119 | LR: 4.32e-04 | 5197.7ms/step | 11.5 samples/s
+Step 12670/50462 | Loss: 2.7133 | LR: 4.32e-04 | 5209.8ms/step | 11.5 samples/s
+Step 12680/50462 | Loss: 2.6791 | LR: 4.32e-04 | 5210.0ms/step | 11.5 samples/s
+Step 12690/50462 | Loss: 2.6861 | LR: 4.32e-04 | 5229.1ms/step | 11.5 samples/s
+Step 12700/50462 | Loss: 2.7061 | LR: 4.32e-04 | 5218.0ms/step | 11.5 samples/s
+Step 12710/50462 | Loss: 2.6956 | LR: 4.32e-04 | 5204.7ms/step | 11.5 samples/s
+Step 12720/50462 | Loss: 2.6706 | LR: 4.32e-04 | 5206.7ms/step | 11.5 samples/s
+Step 12730/50462 | Loss: 2.6981 | LR: 4.32e-04 | 5213.4ms/step | 11.5 samples/s
+Step 12740/50462 | Loss: 2.6775 | LR: 4.32e-04 | 5217.6ms/step | 11.5 samples/s
+Step 12750/50462 | Loss: 2.6922 | LR: 4.31e-04 | 5226.0ms/step | 11.5 samples/s
+Validation at step 12750: Loss = 2.7048, Perplexity = 14.95
+Step 12760/50462 | Loss: 2.6772 | LR: 4.31e-04 | 7787.6ms/step | 7.7 samples/s
+Step 12770/50462 | Loss: 2.7040 | LR: 4.31e-04 | 5221.8ms/step | 11.5 samples/s
+Step 12780/50462 | Loss: 2.7051 | LR: 4.31e-04 | 5208.6ms/step | 11.5 samples/s
+Step 12790/50462 | Loss: 2.6829 | LR: 4.31e-04 | 5219.9ms/step | 11.5 samples/s
+Step 12800/50462 | Loss: 2.6964 | LR: 4.31e-04 | 5206.0ms/step | 11.5 samples/s
+Step 12810/50462 | Loss: 2.6898 | LR: 4.31e-04 | 5212.8ms/step | 11.5 samples/s
+Step 12820/50462 | Loss: 2.7216 | LR: 4.31e-04 | 5212.4ms/step | 11.5 samples/s
+Step 12830/50462 | Loss: 2.6840 | LR: 4.31e-04 | 5229.1ms/step | 11.5 samples/s
+Step 12840/50462 | Loss: 2.7025 | LR: 4.31e-04 | 5245.1ms/step | 11.4 samples/s
+Step 12850/50462 | Loss: 2.7038 | LR: 4.30e-04 | 5223.9ms/step | 11.5 samples/s
+Step 12860/50462 | Loss: 2.6713 | LR: 4.30e-04 | 5213.7ms/step | 11.5 samples/s
+Step 12870/50462 | Loss: 2.7056 | LR: 4.30e-04 | 5199.8ms/step | 11.5 samples/s
+Step 12880/50462 | Loss: 2.6689 | LR: 4.30e-04 | 5207.7ms/step | 11.5 samples/s
+Step 12890/50462 | Loss: 2.7059 | LR: 4.30e-04 | 5222.3ms/step | 11.5 samples/s
+Step 12900/50462 | Loss: 2.7240 | LR: 4.30e-04 | 5214.8ms/step | 11.5 samples/s
+Step 12910/50462 | Loss: 2.6780 | LR: 4.30e-04 | 5217.3ms/step | 11.5 samples/s
+Step 12920/50462 | Loss: 2.6838 | LR: 4.30e-04 | 5211.9ms/step | 11.5 samples/s
+Step 12930/50462 | Loss: 2.6881 | LR: 4.30e-04 | 5207.2ms/step | 11.5 samples/s
+Step 12940/50462 | Loss: 2.7110 | LR: 4.29e-04 | 5211.7ms/step | 11.5 samples/s
+Step 12950/50462 | Loss: 2.6991 | LR: 4.29e-04 | 5206.1ms/step | 11.5 samples/s
+Step 12960/50462 | Loss: 2.6780 | LR: 4.29e-04 | 5218.7ms/step | 11.5 samples/s
+Step 12970/50462 | Loss: 2.6993 | LR: 4.29e-04 | 5211.5ms/step | 11.5 samples/s
+Step 12980/50462 | Loss: 2.7063 | LR: 4.29e-04 | 5207.3ms/step | 11.5 samples/s
+Step 12990/50462 | Loss: 2.6712 | LR: 4.29e-04 | 5200.8ms/step | 11.5 samples/s
+Step 13000/50462 | Loss: 2.6995 | LR: 4.29e-04 | 5218.3ms/step | 11.5 samples/s
+Validation at step 13000: Loss = 2.7008, Perplexity = 14.89
+Saved checkpoint to checkpoints/run_20260605_220251/checkpoint-13000
+Step 13010/50462 | Loss: 2.6699 | LR: 4.29e-04 | 8702.7ms/step | 6.9 samples/s
+Step 13020/50462 | Loss: 2.6840 | LR: 4.29e-04 | 5220.5ms/step | 11.5 samples/s
+Step 13030/50462 | Loss: 2.6940 | LR: 4.28e-04 | 5205.4ms/step | 11.5 samples/s
+Step 13040/50462 | Loss: 2.6725 | LR: 4.28e-04 | 5220.0ms/step | 11.5 samples/s
+Step 13050/50462 | Loss: 2.6908 | LR: 4.28e-04 | 5207.6ms/step | 11.5 samples/s
+Step 13060/50462 | Loss: 2.7295 | LR: 4.28e-04 | 5211.1ms/step | 11.5 samples/s
+Step 13070/50462 | Loss: 2.7035 | LR: 4.28e-04 | 5627.2ms/step | 10.7 samples/s
+Step 13080/50462 | Loss: 2.7061 | LR: 4.28e-04 | 5212.8ms/step | 11.5 samples/s
+Step 13090/50462 | Loss: 2.6768 | LR: 4.28e-04 | 5240.8ms/step | 11.4 samples/s
+Step 13100/50462 | Loss: 2.6680 | LR: 4.28e-04 | 5223.0ms/step | 11.5 samples/s
+Step 13110/50462 | Loss: 2.6829 | LR: 4.28e-04 | 5223.0ms/step | 11.5 samples/s
+Step 13120/50462 | Loss: 2.7043 | LR: 4.27e-04 | 5215.1ms/step | 11.5 samples/s
+Step 13130/50462 | Loss: 2.6918 | LR: 4.27e-04 | 5209.4ms/step | 11.5 samples/s
+Step 13140/50462 | Loss: 2.7152 | LR: 4.27e-04 | 5233.9ms/step | 11.5 samples/s
+Step 13150/50462 | Loss: 2.6821 | LR: 4.27e-04 | 5223.5ms/step | 11.5 samples/s
+Step 13160/50462 | Loss: 2.6847 | LR: 4.27e-04 | 5583.7ms/step | 10.7 samples/s
+Step 13170/50462 | Loss: 2.6801 | LR: 4.27e-04 | 5583.3ms/step | 10.7 samples/s
+Step 13180/50462 | Loss: 2.7094 | LR: 4.27e-04 | 5209.2ms/step | 11.5 samples/s
+Step 13190/50462 | Loss: 2.6838 | LR: 4.27e-04 | 5212.4ms/step | 11.5 samples/s
+Step 13200/50462 | Loss: 2.6889 | LR: 4.27e-04 | 5230.4ms/step | 11.5 samples/s
+Step 13210/50462 | Loss: 2.6806 | LR: 4.26e-04 | 5222.1ms/step | 11.5 samples/s
+Step 13220/50462 | Loss: 2.7055 | LR: 4.26e-04 | 5218.2ms/step | 11.5 samples/s
+Step 13230/50462 | Loss: 2.6495 | LR: 4.26e-04 | 5213.3ms/step | 11.5 samples/s
+Step 13240/50462 | Loss: 2.7002 | LR: 4.26e-04 | 5212.6ms/step | 11.5 samples/s
+Step 13250/50462 | Loss: 2.6758 | LR: 4.26e-04 | 5205.1ms/step | 11.5 samples/s
+Validation at step 13250: Loss = 2.6976, Perplexity = 14.84
+Step 13260/50462 | Loss: 2.7208 | LR: 4.26e-04 | 7815.0ms/step | 7.7 samples/s
+Step 13270/50462 | Loss: 2.6989 | LR: 4.26e-04 | 5246.6ms/step | 11.4 samples/s
+Step 13280/50462 | Loss: 2.6787 | LR: 4.26e-04 | 5228.7ms/step | 11.5 samples/s
+Step 13290/50462 | Loss: 2.7069 | LR: 4.26e-04 | 5232.4ms/step | 11.5 samples/s
+Step 13300/50462 | Loss: 2.6956 | LR: 4.25e-04 | 5251.9ms/step | 11.4 samples/s
+Step 13310/50462 | Loss: 2.6958 | LR: 4.25e-04 | 5256.5ms/step | 11.4 samples/s
+Step 13320/50462 | Loss: 2.6762 | LR: 4.25e-04 | 5219.5ms/step | 11.5 samples/s
+Step 13330/50462 | Loss: 2.6446 | LR: 4.25e-04 | 5230.5ms/step | 11.5 samples/s
+Step 13340/50462 | Loss: 2.6887 | LR: 4.25e-04 | 5217.6ms/step | 11.5 samples/s
+Step 13350/50462 | Loss: 2.6934 | LR: 4.25e-04 | 5219.9ms/step | 11.5 samples/s
+Step 13360/50462 | Loss: 2.7025 | LR: 4.25e-04 | 5236.8ms/step | 11.5 samples/s
+Step 13370/50462 | Loss: 2.6553 | LR: 4.25e-04 | 5230.1ms/step | 11.5 samples/s
+Step 13380/50462 | Loss: 2.6645 | LR: 4.25e-04 | 5214.0ms/step | 11.5 samples/s
+Step 13390/50462 | Loss: 2.6898 | LR: 4.24e-04 | 5228.1ms/step | 11.5 samples/s
+Step 13400/50462 | Loss: 2.6710 | LR: 4.24e-04 | 5220.7ms/step | 11.5 samples/s
+Step 13410/50462 | Loss: 2.6974 | LR: 4.24e-04 | 5224.4ms/step | 11.5 samples/s
+Step 13420/50462 | Loss: 2.6688 | LR: 4.24e-04 | 5220.3ms/step | 11.5 samples/s
+Step 13430/50462 | Loss: 2.7230 | LR: 4.24e-04 | 5218.8ms/step | 11.5 samples/s
+Step 13440/50462 | Loss: 2.6922 | LR: 4.24e-04 | 5225.9ms/step | 11.5 samples/s
+Step 13450/50462 | Loss: 2.6833 | LR: 4.24e-04 | 5248.1ms/step | 11.4 samples/s
+Step 13460/50462 | Loss: 2.7101 | LR: 4.24e-04 | 5245.6ms/step | 11.4 samples/s
+Step 13470/50462 | Loss: 2.6908 | LR: 4.23e-04 | 5244.6ms/step | 11.4 samples/s
+Step 13480/50462 | Loss: 2.6673 | LR: 4.23e-04 | 5248.0ms/step | 11.4 samples/s
+Step 13490/50462 | Loss: 2.6675 | LR: 4.23e-04 | 5247.6ms/step | 11.4 samples/s
+Step 13500/50462 | Loss: 2.6997 | LR: 4.23e-04 | 5254.9ms/step | 11.4 samples/s
+Validation at step 13500: Loss = 2.6943, Perplexity = 14.80
+Step 13510/50462 | Loss: 2.7052 | LR: 4.23e-04 | 7802.9ms/step | 7.7 samples/s
+Step 13520/50462 | Loss: 2.6618 | LR: 4.23e-04 | 5217.2ms/step | 11.5 samples/s
+Step 13530/50462 | Loss: 2.7022 | LR: 4.23e-04 | 5235.5ms/step | 11.5 samples/s
+Step 13540/50462 | Loss: 2.6839 | LR: 4.23e-04 | 5218.8ms/step | 11.5 samples/s
+Step 13550/50462 | Loss: 2.7068 | LR: 4.23e-04 | 5242.9ms/step | 11.4 samples/s
+Step 13560/50462 | Loss: 2.6726 | LR: 4.22e-04 | 5237.5ms/step | 11.5 samples/s
+Step 13570/50462 | Loss: 2.6783 | LR: 4.22e-04 | 5228.0ms/step | 11.5 samples/s
+Step 13580/50462 | Loss: 2.6836 | LR: 4.22e-04 | 5230.7ms/step | 11.5 samples/s
+Step 13590/50462 | Loss: 2.6938 | LR: 4.22e-04 | 5234.5ms/step | 11.5 samples/s
+Step 13600/50462 | Loss: 2.6883 | LR: 4.22e-04 | 5221.4ms/step | 11.5 samples/s
+Step 13610/50462 | Loss: 2.6861 | LR: 4.22e-04 | 5222.2ms/step | 11.5 samples/s
+Step 13620/50462 | Loss: 2.6645 | LR: 4.22e-04 | 5220.5ms/step | 11.5 samples/s
+Step 13630/50462 | Loss: 2.6727 | LR: 4.22e-04 | 5217.2ms/step | 11.5 samples/s
+Step 13640/50462 | Loss: 2.6998 | LR: 4.22e-04 | 5195.7ms/step | 11.5 samples/s
+Step 13650/50462 | Loss: 2.6745 | LR: 4.21e-04 | 5152.4ms/step | 11.6 samples/s
+Step 13660/50462 | Loss: 2.6968 | LR: 4.21e-04 | 5159.9ms/step | 11.6 samples/s
+Step 13670/50462 | Loss: 2.6739 | LR: 4.21e-04 | 5160.9ms/step | 11.6 samples/s
+Step 13680/50462 | Loss: 2.6903 | LR: 4.21e-04 | 5163.9ms/step | 11.6 samples/s
+Step 13690/50462 | Loss: 2.6777 | LR: 4.21e-04 | 5162.6ms/step | 11.6 samples/s
+Step 13700/50462 | Loss: 2.7028 | LR: 4.21e-04 | 5157.9ms/step | 11.6 samples/s
+Step 13710/50462 | Loss: 2.6613 | LR: 4.21e-04 | 5160.3ms/step | 11.6 samples/s
+Step 13720/50462 | Loss: 2.6875 | LR: 4.21e-04 | 5152.8ms/step | 11.6 samples/s
+Step 13730/50462 | Loss: 2.6715 | LR: 4.21e-04 | 5136.4ms/step | 11.7 samples/s
+Step 13740/50462 | Loss: 2.6828 | LR: 4.20e-04 | 5155.1ms/step | 11.6 samples/s
+Step 13750/50462 | Loss: 2.7070 | LR: 4.20e-04 | 5150.5ms/step | 11.6 samples/s
+Validation at step 13750: Loss = 2.6899, Perplexity = 14.73
+Step 13760/50462 | Loss: 2.6735 | LR: 4.20e-04 | 7720.2ms/step | 7.8 samples/s
+Step 13770/50462 | Loss: 2.6768 | LR: 4.20e-04 | 5158.0ms/step | 11.6 samples/s
+Step 13780/50462 | Loss: 2.6933 | LR: 4.20e-04 | 5162.8ms/step | 11.6 samples/s
+Step 13790/50462 | Loss: 2.6721 | LR: 4.20e-04 | 5170.9ms/step | 11.6 samples/s
+Step 13800/50462 | Loss: 2.6848 | LR: 4.20e-04 | 5149.2ms/step | 11.7 samples/s
+Step 13810/50462 | Loss: 2.6557 | LR: 4.20e-04 | 5169.4ms/step | 11.6 samples/s
+Step 13820/50462 | Loss: 2.6716 | LR: 4.19e-04 | 5164.5ms/step | 11.6 samples/s
+Step 13830/50462 | Loss: 2.6801 | LR: 4.19e-04 | 5157.3ms/step | 11.6 samples/s
+Step 13840/50462 | Loss: 2.6665 | LR: 4.19e-04 | 5162.7ms/step | 11.6 samples/s
+Step 13850/50462 | Loss: 2.6504 | LR: 4.19e-04 | 5171.2ms/step | 11.6 samples/s
+Step 13860/50462 | Loss: 2.6624 | LR: 4.19e-04 | 5156.1ms/step | 11.6 samples/s
+Step 13870/50462 | Loss: 2.6687 | LR: 4.19e-04 | 5161.7ms/step | 11.6 samples/s
+Step 13880/50462 | Loss: 2.6801 | LR: 4.19e-04 | 5150.8ms/step | 11.6 samples/s
+Step 13890/50462 | Loss: 2.6792 | LR: 4.19e-04 | 5152.9ms/step | 11.6 samples/s
+Step 13900/50462 | Loss: 2.7017 | LR: 4.19e-04 | 5146.2ms/step | 11.7 samples/s
+Step 13910/50462 | Loss: 2.7129 | LR: 4.18e-04 | 5135.6ms/step | 11.7 samples/s
+Step 13920/50462 | Loss: 2.6687 | LR: 4.18e-04 | 5132.4ms/step | 11.7 samples/s
+Step 13930/50462 | Loss: 2.6709 | LR: 4.18e-04 | 5136.5ms/step | 11.7 samples/s
+Step 13940/50462 | Loss: 2.6850 | LR: 4.18e-04 | 5146.5ms/step | 11.7 samples/s
+Step 13950/50462 | Loss: 2.6726 | LR: 4.18e-04 | 5140.8ms/step | 11.7 samples/s
+Step 13960/50462 | Loss: 2.6689 | LR: 4.18e-04 | 5188.8ms/step | 11.6 samples/s
+Step 13970/50462 | Loss: 2.6963 | LR: 4.18e-04 | 5170.4ms/step | 11.6 samples/s
+Step 13980/50462 | Loss: 2.6517 | LR: 4.18e-04 | 5201.5ms/step | 11.5 samples/s
+Step 13990/50462 | Loss: 2.6695 | LR: 4.17e-04 | 5180.2ms/step | 11.6 samples/s
+Step 14000/50462 | Loss: 2.6948 | LR: 4.17e-04 | 5183.1ms/step | 11.6 samples/s
+Validation at step 14000: Loss = 2.6874, Perplexity = 14.69
+Saved checkpoint to checkpoints/run_20260605_220251/checkpoint-14000
+Step 14010/50462 | Loss: 2.7110 | LR: 4.17e-04 | 8682.4ms/step | 6.9 samples/s
+Step 14020/50462 | Loss: 2.6845 | LR: 4.17e-04 | 5196.6ms/step | 11.5 samples/s
+Step 14030/50462 | Loss: 2.6603 | LR: 4.17e-04 | 5187.8ms/step | 11.6 samples/s
+Step 14040/50462 | Loss: 2.6676 | LR: 4.17e-04 | 5204.1ms/step | 11.5 samples/s
+Step 14050/50462 | Loss: 2.6692 | LR: 4.17e-04 | 5195.3ms/step | 11.5 samples/s
+Step 14060/50462 | Loss: 2.6904 | LR: 4.17e-04 | 5196.0ms/step | 11.5 samples/s
+Step 14070/50462 | Loss: 2.6510 | LR: 4.17e-04 | 5209.1ms/step | 11.5 samples/s
+Step 14080/50462 | Loss: 2.6928 | LR: 4.16e-04 | 5208.3ms/step | 11.5 samples/s
+Step 14090/50462 | Loss: 2.6756 | LR: 4.16e-04 | 5209.2ms/step | 11.5 samples/s
+Step 14100/50462 | Loss: 2.6590 | LR: 4.16e-04 | 5200.5ms/step | 11.5 samples/s
+Step 14110/50462 | Loss: 2.6515 | LR: 4.16e-04 | 5203.3ms/step | 11.5 samples/s
+Step 14120/50462 | Loss: 2.6723 | LR: 4.16e-04 | 5190.8ms/step | 11.6 samples/s
+Step 14130/50462 | Loss: 2.6874 | LR: 4.16e-04 | 5210.3ms/step | 11.5 samples/s
+Step 14140/50462 | Loss: 2.6983 | LR: 4.16e-04 | 5210.2ms/step | 11.5 samples/s
+Step 14150/50462 | Loss: 2.6878 | LR: 4.16e-04 | 5187.6ms/step | 11.6 samples/s
+Step 14160/50462 | Loss: 2.6747 | LR: 4.15e-04 | 5199.2ms/step | 11.5 samples/s
+Step 14170/50462 | Loss: 2.6771 | LR: 4.15e-04 | 5183.6ms/step | 11.6 samples/s
+Step 14180/50462 | Loss: 2.7217 | LR: 4.15e-04 | 5192.9ms/step | 11.6 samples/s
+Step 14190/50462 | Loss: 2.6826 | LR: 4.15e-04 | 5196.5ms/step | 11.5 samples/s
+Step 14200/50462 | Loss: 2.7095 | LR: 4.15e-04 | 5209.1ms/step | 11.5 samples/s
+Step 14210/50462 | Loss: 2.6452 | LR: 4.15e-04 | 5179.9ms/step | 11.6 samples/s
+Step 14220/50462 | Loss: 2.6986 | LR: 4.15e-04 | 5179.2ms/step | 11.6 samples/s
+Step 14230/50462 | Loss: 2.6976 | LR: 4.15e-04 | 5197.9ms/step | 11.5 samples/s
+Step 14240/50462 | Loss: 2.6677 | LR: 4.15e-04 | 5196.3ms/step | 11.5 samples/s
+Step 14250/50462 | Loss: 2.6982 | LR: 4.14e-04 | 5203.6ms/step | 11.5 samples/s
+Validation at step 14250: Loss = 2.6836, Perplexity = 14.64
+Step 14260/50462 | Loss: 2.6702 | LR: 4.14e-04 | 7773.9ms/step | 7.7 samples/s
+Step 14270/50462 | Loss: 2.6936 | LR: 4.14e-04 | 5212.9ms/step | 11.5 samples/s
+Step 14280/50462 | Loss: 2.6553 | LR: 4.14e-04 | 5212.1ms/step | 11.5 samples/s
+Step 14290/50462 | Loss: 2.6669 | LR: 4.14e-04 | 5196.6ms/step | 11.5 samples/s
+Step 14300/50462 | Loss: 2.6945 | LR: 4.14e-04 | 5206.9ms/step | 11.5 samples/s
+Step 14310/50462 | Loss: 2.6466 | LR: 4.14e-04 | 5202.5ms/step | 11.5 samples/s
+Step 14320/50462 | Loss: 2.6667 | LR: 4.14e-04 | 5197.8ms/step | 11.5 samples/s
+Step 14330/50462 | Loss: 2.6615 | LR: 4.13e-04 | 5204.4ms/step | 11.5 samples/s
+Step 14340/50462 | Loss: 2.6541 | LR: 4.13e-04 | 5199.7ms/step | 11.5 samples/s
+Step 14350/50462 | Loss: 2.6686 | LR: 4.13e-04 | 5216.9ms/step | 11.5 samples/s
+Step 14360/50462 | Loss: 2.6513 | LR: 4.13e-04 | 5194.5ms/step | 11.6 samples/s
+Step 14370/50462 | Loss: 2.6865 | LR: 4.13e-04 | 5209.5ms/step | 11.5 samples/s
+Step 14380/50462 | Loss: 2.6945 | LR: 4.13e-04 | 5207.9ms/step | 11.5 samples/s
+Step 14390/50462 | Loss: 2.6884 | LR: 4.13e-04 | 5218.6ms/step | 11.5 samples/s
+Step 14400/50462 | Loss: 2.6817 | LR: 4.13e-04 | 5209.2ms/step | 11.5 samples/s
+Step 14410/50462 | Loss: 2.6937 | LR: 4.12e-04 | 5205.5ms/step | 11.5 samples/s
+Step 14420/50462 | Loss: 2.6597 | LR: 4.12e-04 | 5217.4ms/step | 11.5 samples/s
+Step 14430/50462 | Loss: 2.6699 | LR: 4.12e-04 | 5221.8ms/step | 11.5 samples/s
+Step 14440/50462 | Loss: 2.6910 | LR: 4.12e-04 | 5194.9ms/step | 11.5 samples/s
+Step 14450/50462 | Loss: 2.6761 | LR: 4.12e-04 | 5196.8ms/step | 11.5 samples/s
+Step 14460/50462 | Loss: 2.6597 | LR: 4.12e-04 | 5185.1ms/step | 11.6 samples/s
+Step 14470/50462 | Loss: 2.6777 | LR: 4.12e-04 | 5193.2ms/step | 11.6 samples/s
+Step 14480/50462 | Loss: 2.6406 | LR: 4.12e-04 | 5212.8ms/step | 11.5 samples/s
+Step 14490/50462 | Loss: 2.6950 | LR: 4.12e-04 | 5205.5ms/step | 11.5 samples/s
+Step 14500/50462 | Loss: 2.6743 | LR: 4.11e-04 | 5205.5ms/step | 11.5 samples/s
+Validation at step 14500: Loss = 2.6795, Perplexity = 14.58
+Step 14510/50462 | Loss: 2.6884 | LR: 4.11e-04 | 7783.8ms/step | 7.7 samples/s
+Step 14520/50462 | Loss: 2.6739 | LR: 4.11e-04 | 5201.0ms/step | 11.5 samples/s
+Step 14530/50462 | Loss: 2.6838 | LR: 4.11e-04 | 5217.6ms/step | 11.5 samples/s
+Step 14540/50462 | Loss: 2.6590 | LR: 4.11e-04 | 5214.3ms/step | 11.5 samples/s
+Step 14550/50462 | Loss: 2.6982 | LR: 4.11e-04 | 5198.5ms/step | 11.5 samples/s
+Step 14560/50462 | Loss: 2.6960 | LR: 4.11e-04 | 5217.9ms/step | 11.5 samples/s
+Step 14570/50462 | Loss: 2.6615 | LR: 4.11e-04 | 5208.2ms/step | 11.5 samples/s
+Step 14580/50462 | Loss: 2.6714 | LR: 4.10e-04 | 5215.9ms/step | 11.5 samples/s
+Step 14590/50462 | Loss: 2.6811 | LR: 4.10e-04 | 5217.9ms/step | 11.5 samples/s
+Step 14600/50462 | Loss: 2.6678 | LR: 4.10e-04 | 5227.8ms/step | 11.5 samples/s
+Step 14610/50462 | Loss: 2.6823 | LR: 4.10e-04 | 5216.0ms/step | 11.5 samples/s
+Step 14620/50462 | Loss: 2.6433 | LR: 4.10e-04 | 5204.3ms/step | 11.5 samples/s
+Step 14630/50462 | Loss: 2.6535 | LR: 4.10e-04 | 5205.8ms/step | 11.5 samples/s
+Step 14640/50462 | Loss: 2.6613 | LR: 4.10e-04 | 5209.2ms/step | 11.5 samples/s
+Step 14650/50462 | Loss: 2.6525 | LR: 4.10e-04 | 5199.4ms/step | 11.5 samples/s
+Step 14660/50462 | Loss: 2.7036 | LR: 4.09e-04 | 5193.9ms/step | 11.6 samples/s
+Step 14670/50462 | Loss: 2.6513 | LR: 4.09e-04 | 5189.0ms/step | 11.6 samples/s
+Step 14680/50462 | Loss: 2.6656 | LR: 4.09e-04 | 5189.3ms/step | 11.6 samples/s
+Step 14690/50462 | Loss: 2.6544 | LR: 4.09e-04 | 5183.5ms/step | 11.6 samples/s
+Step 14700/50462 | Loss: 2.6669 | LR: 4.09e-04 | 5204.2ms/step | 11.5 samples/s
+Step 14710/50462 | Loss: 2.6507 | LR: 4.09e-04 | 5200.7ms/step | 11.5 samples/s
+Step 14720/50462 | Loss: 2.6607 | LR: 4.09e-04 | 5201.5ms/step | 11.5 samples/s
+Step 14730/50462 | Loss: 2.6420 | LR: 4.09e-04 | 5205.8ms/step | 11.5 samples/s
+Step 14740/50462 | Loss: 2.6460 | LR: 4.08e-04 | 5203.1ms/step | 11.5 samples/s
+Step 14750/50462 | Loss: 2.6546 | LR: 4.08e-04 | 5200.6ms/step | 11.5 samples/s
+Validation at step 14750: Loss = 2.6774, Perplexity = 14.55
+Step 14760/50462 | Loss: 2.6622 | LR: 4.08e-04 | 7762.3ms/step | 7.7 samples/s
+Step 14770/50462 | Loss: 2.6575 | LR: 4.08e-04 | 5202.4ms/step | 11.5 samples/s
+Step 14780/50462 | Loss: 2.6554 | LR: 4.08e-04 | 5217.6ms/step | 11.5 samples/s
+Step 14790/50462 | Loss: 2.6904 | LR: 4.08e-04 | 5215.9ms/step | 11.5 samples/s
+Step 14800/50462 | Loss: 2.6503 | LR: 4.08e-04 | 5207.9ms/step | 11.5 samples/s
+Step 14810/50462 | Loss: 2.6467 | LR: 4.08e-04 | 5196.8ms/step | 11.5 samples/s
+Step 14820/50462 | Loss: 2.6707 | LR: 4.08e-04 | 5177.1ms/step | 11.6 samples/s
+Step 14830/50462 | Loss: 2.6590 | LR: 4.07e-04 | 5184.2ms/step | 11.6 samples/s
+Step 14840/50462 | Loss: 2.6649 | LR: 4.07e-04 | 5182.7ms/step | 11.6 samples/s
+Step 14850/50462 | Loss: 2.6402 | LR: 4.07e-04 | 5197.8ms/step | 11.5 samples/s
+Step 14860/50462 | Loss: 2.6733 | LR: 4.07e-04 | 5191.6ms/step | 11.6 samples/s
+Step 14870/50462 | Loss: 2.6266 | LR: 4.07e-04 | 5186.9ms/step | 11.6 samples/s
+Step 14880/50462 | Loss: 2.6952 | LR: 4.07e-04 | 5176.9ms/step | 11.6 samples/s
+Step 14890/50462 | Loss: 2.6900 | LR: 4.07e-04 | 5186.8ms/step | 11.6 samples/s
+Step 14900/50462 | Loss: 2.6585 | LR: 4.07e-04 | 5185.1ms/step | 11.6 samples/s
+Step 14910/50462 | Loss: 2.6595 | LR: 4.06e-04 | 5185.5ms/step | 11.6 samples/s
+Step 14920/50462 | Loss: 2.6427 | LR: 4.06e-04 | 5193.4ms/step | 11.6 samples/s
+Step 14930/50462 | Loss: 2.6599 | LR: 4.06e-04 | 5199.1ms/step | 11.5 samples/s
+Step 14940/50462 | Loss: 2.6645 | LR: 4.06e-04 | 5183.0ms/step | 11.6 samples/s
+Step 14950/50462 | Loss: 2.6614 | LR: 4.06e-04 | 5194.2ms/step | 11.6 samples/s
+Step 14960/50462 | Loss: 2.6892 | LR: 4.06e-04 | 5200.6ms/step | 11.5 samples/s
+Step 14970/50462 | Loss: 2.6443 | LR: 4.06e-04 | 5208.0ms/step | 11.5 samples/s
+Step 14980/50462 | Loss: 2.6617 | LR: 4.06e-04 | 5141.2ms/step | 11.7 samples/s
+Step 14990/50462 | Loss: 2.6164 | LR: 4.05e-04 | 5131.5ms/step | 11.7 samples/s
+Step 15000/50462 | Loss: 2.6671 | LR: 4.05e-04 | 5126.4ms/step | 11.7 samples/s
+Validation at step 15000: Loss = 2.6721, Perplexity = 14.47
+Saved checkpoint to checkpoints/run_20260605_220251/checkpoint-15000
+Step 15010/50462 | Loss: 2.6649 | LR: 4.05e-04 | 8686.0ms/step | 6.9 samples/s
+Step 15020/50462 | Loss: 2.6482 | LR: 4.05e-04 | 5140.5ms/step | 11.7 samples/s
+Step 15030/50462 | Loss: 2.6763 | LR: 4.05e-04 | 5149.7ms/step | 11.7 samples/s
+Step 15040/50462 | Loss: 2.6647 | LR: 4.05e-04 | 5137.8ms/step | 11.7 samples/s
+Step 15050/50462 | Loss: 2.6883 | LR: 4.05e-04 | 5136.8ms/step | 11.7 samples/s
+Step 15060/50462 | Loss: 2.6648 | LR: 4.05e-04 | 5134.4ms/step | 11.7 samples/s
+Step 15070/50462 | Loss: 2.6411 | LR: 4.04e-04 | 5174.7ms/step | 11.6 samples/s
+Step 15080/50462 | Loss: 2.6597 | LR: 4.04e-04 | 5138.6ms/step | 11.7 samples/s
+Step 15090/50462 | Loss: 2.6776 | LR: 4.04e-04 | 5146.9ms/step | 11.7 samples/s
+Step 15100/50462 | Loss: 2.6732 | LR: 4.04e-04 | 5136.4ms/step | 11.7 samples/s
+Step 15110/50462 | Loss: 2.6661 | LR: 4.04e-04 | 5140.6ms/step | 11.7 samples/s
+Step 15120/50462 | Loss: 2.6625 | LR: 4.04e-04 | 5150.8ms/step | 11.6 samples/s
+Step 15130/50462 | Loss: 2.6622 | LR: 4.04e-04 | 5534.4ms/step | 10.8 samples/s
+Step 15140/50462 | Loss: 2.6670 | LR: 4.04e-04 | 5191.0ms/step | 11.6 samples/s
+Step 15150/50462 | Loss: 2.6458 | LR: 4.03e-04 | 5181.0ms/step | 11.6 samples/s
+Step 15160/50462 | Loss: 2.6396 | LR: 4.03e-04 | 5178.3ms/step | 11.6 samples/s
+Step 15170/50462 | Loss: 2.6660 | LR: 4.03e-04 | 5167.9ms/step | 11.6 samples/s
+Step 15180/50462 | Loss: 2.6553 | LR: 4.03e-04 | 5173.9ms/step | 11.6 samples/s
+Step 15190/50462 | Loss: 2.6802 | LR: 4.03e-04 | 5188.4ms/step | 11.6 samples/s
+Step 15200/50462 | Loss: 2.6428 | LR: 4.03e-04 | 5188.0ms/step | 11.6 samples/s
+Step 15210/50462 | Loss: 2.6462 | LR: 4.03e-04 | 5177.5ms/step | 11.6 samples/s
+Step 15220/50462 | Loss: 2.6329 | LR: 4.03e-04 | 5189.0ms/step | 11.6 samples/s
+Step 15230/50462 | Loss: 2.6487 | LR: 4.02e-04 | 5187.4ms/step | 11.6 samples/s
+Step 15240/50462 | Loss: 2.6707 | LR: 4.02e-04 | 5180.8ms/step | 11.6 samples/s
+Step 15250/50462 | Loss: 2.6895 | LR: 4.02e-04 | 5185.2ms/step | 11.6 samples/s
+Validation at step 15250: Loss = 2.6699, Perplexity = 14.44
+Step 15260/50462 | Loss: 2.6675 | LR: 4.02e-04 | 8122.8ms/step | 7.4 samples/s
+Step 15270/50462 | Loss: 2.6602 | LR: 4.02e-04 | 5175.4ms/step | 11.6 samples/s
+Step 15280/50462 | Loss: 2.6503 | LR: 4.02e-04 | 5556.1ms/step | 10.8 samples/s
+Step 15290/50462 | Loss: 2.6451 | LR: 4.02e-04 | 5188.5ms/step | 11.6 samples/s
+Step 15300/50462 | Loss: 2.6764 | LR: 4.02e-04 | 5200.8ms/step | 11.5 samples/s
+Step 15310/50462 | Loss: 2.6691 | LR: 4.01e-04 | 5194.8ms/step | 11.6 samples/s
+Step 15320/50462 | Loss: 2.6728 | LR: 4.01e-04 | 5228.0ms/step | 11.5 samples/s
+Step 15330/50462 | Loss: 2.6702 | LR: 4.01e-04 | 5237.6ms/step | 11.5 samples/s
+Step 15340/50462 | Loss: 2.6662 | LR: 4.01e-04 | 5237.0ms/step | 11.5 samples/s
+Step 15350/50462 | Loss: 2.6725 | LR: 4.01e-04 | 5241.8ms/step | 11.4 samples/s
+Step 15360/50462 | Loss: 2.6694 | LR: 4.01e-04 | 5240.1ms/step | 11.5 samples/s
+Step 15370/50462 | Loss: 2.6691 | LR: 4.01e-04 | 5250.9ms/step | 11.4 samples/s
+Step 15380/50462 | Loss: 2.6341 | LR: 4.01e-04 | 5254.8ms/step | 11.4 samples/s
+Step 15390/50462 | Loss: 2.6812 | LR: 4.00e-04 | 5250.7ms/step | 11.4 samples/s
+Step 15400/50462 | Loss: 2.6474 | LR: 4.00e-04 | 5239.2ms/step | 11.5 samples/s
+Step 15410/50462 | Loss: 2.6429 | LR: 4.00e-04 | 5232.5ms/step | 11.5 samples/s
+Step 15420/50462 | Loss: 2.6546 | LR: 4.00e-04 | 5242.6ms/step | 11.4 samples/s
+Step 15430/50462 | Loss: 2.6392 | LR: 4.00e-04 | 5243.1ms/step | 11.4 samples/s
+Step 15440/50462 | Loss: 2.6436 | LR: 4.00e-04 | 5236.9ms/step | 11.5 samples/s
+Step 15450/50462 | Loss: 2.6494 | LR: 4.00e-04 | 5246.4ms/step | 11.4 samples/s
+Step 15460/50462 | Loss: 2.6303 | LR: 4.00e-04 | 5236.3ms/step | 11.5 samples/s
+Step 15470/50462 | Loss: 2.6620 | LR: 3.99e-04 | 5241.3ms/step | 11.4 samples/s
+Step 15480/50462 | Loss: 2.6440 | LR: 3.99e-04 | 5238.2ms/step | 11.5 samples/s
+Step 15490/50462 | Loss: 2.6486 | LR: 3.99e-04 | 5243.7ms/step | 11.4 samples/s
+Step 15500/50462 | Loss: 2.6246 | LR: 3.99e-04 | 5239.0ms/step | 11.5 samples/s
+Validation at step 15500: Loss = 2.6675, Perplexity = 14.40
+Step 15510/50462 | Loss: 2.6598 | LR: 3.99e-04 | 7819.3ms/step | 7.7 samples/s
+Step 15520/50462 | Loss: 2.6377 | LR: 3.99e-04 | 5242.9ms/step | 11.4 samples/s
+Step 15530/50462 | Loss: 2.6384 | LR: 3.99e-04 | 5239.8ms/step | 11.5 samples/s
+Step 15540/50462 | Loss: 2.6400 | LR: 3.99e-04 | 5248.1ms/step | 11.4 samples/s
+Step 15550/50462 | Loss: 2.6461 | LR: 3.98e-04 | 5242.4ms/step | 11.4 samples/s
+Step 15560/50462 | Loss: 2.6640 | LR: 3.98e-04 | 5233.8ms/step | 11.5 samples/s
+Step 15570/50462 | Loss: 2.6992 | LR: 3.98e-04 | 5243.8ms/step | 11.4 samples/s
+Step 15580/50462 | Loss: 2.6449 | LR: 3.98e-04 | 5247.1ms/step | 11.4 samples/s
+Step 15590/50462 | Loss: 2.6213 | LR: 3.98e-04 | 5252.5ms/step | 11.4 samples/s
+Step 15600/50462 | Loss: 2.6363 | LR: 3.98e-04 | 5253.7ms/step | 11.4 samples/s
+Step 15610/50462 | Loss: 2.6535 | LR: 3.98e-04 | 5244.6ms/step | 11.4 samples/s
+Step 15620/50462 | Loss: 2.6578 | LR: 3.98e-04 | 5242.2ms/step | 11.4 samples/s
+Step 15630/50462 | Loss: 2.6611 | LR: 3.97e-04 | 5249.1ms/step | 11.4 samples/s
+Step 15640/50462 | Loss: 2.6161 | LR: 3.97e-04 | 5249.8ms/step | 11.4 samples/s
+Step 15650/50462 | Loss: 2.6647 | LR: 3.97e-04 | 5250.3ms/step | 11.4 samples/s
+Step 15660/50462 | Loss: 2.6564 | LR: 3.97e-04 | 5252.9ms/step | 11.4 samples/s
+Step 15670/50462 | Loss: 2.6361 | LR: 3.97e-04 | 5253.4ms/step | 11.4 samples/s
+Step 15680/50462 | Loss: 2.6531 | LR: 3.97e-04 | 5252.6ms/step | 11.4 samples/s
+Step 15690/50462 | Loss: 2.6337 | LR: 3.97e-04 | 5250.5ms/step | 11.4 samples/s
+Step 15700/50462 | Loss: 2.6537 | LR: 3.96e-04 | 5251.5ms/step | 11.4 samples/s
+Step 15710/50462 | Loss: 2.6611 | LR: 3.96e-04 | 5246.8ms/step | 11.4 samples/s
+Step 15720/50462 | Loss: 2.6183 | LR: 3.96e-04 | 5248.7ms/step | 11.4 samples/s
+Step 15730/50462 | Loss: 2.6297 | LR: 3.96e-04 | 5252.5ms/step | 11.4 samples/s
+Step 15740/50462 | Loss: 2.6484 | LR: 3.96e-04 | 5239.1ms/step | 11.5 samples/s
+Step 15750/50462 | Loss: 2.6452 | LR: 3.96e-04 | 5252.2ms/step | 11.4 samples/s
+Validation at step 15750: Loss = 2.6632, Perplexity = 14.34
+Step 15760/50462 | Loss: 2.6625 | LR: 3.96e-04 | 7810.4ms/step | 7.7 samples/s
+Step 15770/50462 | Loss: 2.6618 | LR: 3.96e-04 | 5238.0ms/step | 11.5 samples/s
+Step 15780/50462 | Loss: 2.6149 | LR: 3.95e-04 | 5237.3ms/step | 11.5 samples/s
+Step 15790/50462 | Loss: 2.6350 | LR: 3.95e-04 | 5237.2ms/step | 11.5 samples/s
+Step 15800/50462 | Loss: 2.6370 | LR: 3.95e-04 | 5233.0ms/step | 11.5 samples/s
+Step 15810/50462 | Loss: 2.6728 | LR: 3.95e-04 | 5190.5ms/step | 11.6 samples/s
+Step 15820/50462 | Loss: 2.6477 | LR: 3.95e-04 | 5198.1ms/step | 11.5 samples/s
+Step 15830/50462 | Loss: 2.6452 | LR: 3.95e-04 | 5212.9ms/step | 11.5 samples/s
+Step 15840/50462 | Loss: 2.6383 | LR: 3.95e-04 | 5220.8ms/step | 11.5 samples/s
+Step 15850/50462 | Loss: 2.6190 | LR: 3.95e-04 | 5207.0ms/step | 11.5 samples/s
+Step 15860/50462 | Loss: 2.6522 | LR: 3.94e-04 | 5208.8ms/step | 11.5 samples/s
+Step 15870/50462 | Loss: 2.6237 | LR: 3.94e-04 | 5171.9ms/step | 11.6 samples/s
+Step 15880/50462 | Loss: 2.6415 | LR: 3.94e-04 | 5166.8ms/step | 11.6 samples/s
+Step 15890/50462 | Loss: 2.6170 | LR: 3.94e-04 | 5183.9ms/step | 11.6 samples/s
+Step 15900/50462 | Loss: 2.6249 | LR: 3.94e-04 | 5177.4ms/step | 11.6 samples/s
+Step 15910/50462 | Loss: 2.5986 | LR: 3.94e-04 | 5168.9ms/step | 11.6 samples/s
+Step 15920/50462 | Loss: 2.6516 | LR: 3.94e-04 | 5164.2ms/step | 11.6 samples/s
+Step 15930/50462 | Loss: 2.6348 | LR: 3.94e-04 | 5170.6ms/step | 11.6 samples/s
+Step 15940/50462 | Loss: 2.6668 | LR: 3.93e-04 | 5166.8ms/step | 11.6 samples/s
+Step 15950/50462 | Loss: 2.6441 | LR: 3.93e-04 | 5166.4ms/step | 11.6 samples/s
+Step 15960/50462 | Loss: 2.6606 | LR: 3.93e-04 | 5189.6ms/step | 11.6 samples/s
+Step 15970/50462 | Loss: 2.6468 | LR: 3.93e-04 | 5172.1ms/step | 11.6 samples/s
+Step 15980/50462 | Loss: 2.6587 | LR: 3.93e-04 | 5170.1ms/step | 11.6 samples/s
+Step 15990/50462 | Loss: 2.6321 | LR: 3.93e-04 | 5171.4ms/step | 11.6 samples/s
+Step 16000/50462 | Loss: 2.6600 | LR: 3.93e-04 | 5184.3ms/step | 11.6 samples/s
+Validation at step 16000: Loss = 2.6597, Perplexity = 14.29
+Saved checkpoint to checkpoints/run_20260605_220251/checkpoint-16000
+Step 16010/50462 | Loss: 2.6497 | LR: 3.92e-04 | 8709.9ms/step | 6.9 samples/s
+Step 16020/50462 | Loss: 2.6543 | LR: 3.92e-04 | 5121.3ms/step | 11.7 samples/s
+Step 16030/50462 | Loss: 2.6697 | LR: 3.92e-04 | 5122.0ms/step | 11.7 samples/s
+Step 16040/50462 | Loss: 2.6473 | LR: 3.92e-04 | 5124.8ms/step | 11.7 samples/s
+Step 16050/50462 | Loss: 2.6724 | LR: 3.92e-04 | 5117.2ms/step | 11.7 samples/s
+Step 16060/50462 | Loss: 2.6465 | LR: 3.92e-04 | 5120.5ms/step | 11.7 samples/s
+Step 16070/50462 | Loss: 2.6360 | LR: 3.92e-04 | 5113.9ms/step | 11.7 samples/s
+Step 16080/50462 | Loss: 2.6518 | LR: 3.92e-04 | 5123.3ms/step | 11.7 samples/s
+Step 16090/50462 | Loss: 2.6467 | LR: 3.91e-04 | 5131.4ms/step | 11.7 samples/s
+Step 16100/50462 | Loss: 2.6800 | LR: 3.91e-04 | 5125.8ms/step | 11.7 samples/s
+Step 16110/50462 | Loss: 2.6542 | LR: 3.91e-04 | 5134.8ms/step | 11.7 samples/s
+Step 16120/50462 | Loss: 2.6477 | LR: 3.91e-04 | 5123.6ms/step | 11.7 samples/s
+Step 16130/50462 | Loss: 2.6405 | LR: 3.91e-04 | 5131.5ms/step | 11.7 samples/s
+Step 16140/50462 | Loss: 2.6359 | LR: 3.91e-04 | 5119.5ms/step | 11.7 samples/s
+Step 16150/50462 | Loss: 2.6545 | LR: 3.91e-04 | 5123.2ms/step | 11.7 samples/s
+Step 16160/50462 | Loss: 2.6394 | LR: 3.91e-04 | 5115.0ms/step | 11.7 samples/s
+Step 16170/50462 | Loss: 2.6622 | LR: 3.90e-04 | 5116.9ms/step | 11.7 samples/s
+Step 16180/50462 | Loss: 2.6438 | LR: 3.90e-04 | 5125.4ms/step | 11.7 samples/s
+Step 16190/50462 | Loss: 2.6444 | LR: 3.90e-04 | 5115.2ms/step | 11.7 samples/s
+Step 16200/50462 | Loss: 2.6381 | LR: 3.90e-04 | 5124.6ms/step | 11.7 samples/s
+Step 16210/50462 | Loss: 2.6582 | LR: 3.90e-04 | 5114.6ms/step | 11.7 samples/s
+Step 16220/50462 | Loss: 2.6423 | LR: 3.90e-04 | 5120.0ms/step | 11.7 samples/s
+Step 16230/50462 | Loss: 2.6440 | LR: 3.90e-04 | 5124.4ms/step | 11.7 samples/s
+Step 16240/50462 | Loss: 2.6453 | LR: 3.89e-04 | 5112.6ms/step | 11.7 samples/s
+Step 16250/50462 | Loss: 2.6434 | LR: 3.89e-04 | 5126.9ms/step | 11.7 samples/s
+Validation at step 16250: Loss = 2.6567, Perplexity = 14.25
+Step 16260/50462 | Loss: 2.6591 | LR: 3.89e-04 | 7675.5ms/step | 7.8 samples/s
+Step 16270/50462 | Loss: 2.6479 | LR: 3.89e-04 | 5113.7ms/step | 11.7 samples/s
+Step 16280/50462 | Loss: 2.6588 | LR: 3.89e-04 | 5126.0ms/step | 11.7 samples/s
+Step 16290/50462 | Loss: 2.6556 | LR: 3.89e-04 | 5120.3ms/step | 11.7 samples/s
+Step 16300/50462 | Loss: 2.6617 | LR: 3.89e-04 | 5116.8ms/step | 11.7 samples/s
+Step 16310/50462 | Loss: 2.6525 | LR: 3.89e-04 | 5122.0ms/step | 11.7 samples/s
+Step 16320/50462 | Loss: 2.6336 | LR: 3.88e-04 | 5128.8ms/step | 11.7 samples/s
+Step 16330/50462 | Loss: 2.7021 | LR: 3.88e-04 | 5119.4ms/step | 11.7 samples/s
+Step 16340/50462 | Loss: 2.6588 | LR: 3.88e-04 | 5136.1ms/step | 11.7 samples/s
+Step 16350/50462 | Loss: 2.6500 | LR: 3.88e-04 | 5119.1ms/step | 11.7 samples/s
+Step 16360/50462 | Loss: 2.6458 | LR: 3.88e-04 | 5124.1ms/step | 11.7 samples/s
+Step 16370/50462 | Loss: 2.6093 | LR: 3.88e-04 | 5128.2ms/step | 11.7 samples/s
+Step 16380/50462 | Loss: 2.6330 | LR: 3.88e-04 | 5133.2ms/step | 11.7 samples/s
+Step 16390/50462 | Loss: 2.6525 | LR: 3.88e-04 | 5122.4ms/step | 11.7 samples/s
+Step 16400/50462 | Loss: 2.6309 | LR: 3.87e-04 | 5121.7ms/step | 11.7 samples/s
+Step 16410/50462 | Loss: 2.6226 | LR: 3.87e-04 | 5131.3ms/step | 11.7 samples/s
+Step 16420/50462 | Loss: 2.6243 | LR: 3.87e-04 | 5125.4ms/step | 11.7 samples/s
+Step 16430/50462 | Loss: 2.6373 | LR: 3.87e-04 | 5139.4ms/step | 11.7 samples/s
+Step 16440/50462 | Loss: 2.6741 | LR: 3.87e-04 | 5123.9ms/step | 11.7 samples/s
+Step 16450/50462 | Loss: 2.6416 | LR: 3.87e-04 | 5130.9ms/step | 11.7 samples/s
+Step 16460/50462 | Loss: 2.6658 | LR: 3.87e-04 | 5134.7ms/step | 11.7 samples/s
+Step 16470/50462 | Loss: 2.6493 | LR: 3.86e-04 | 5122.5ms/step | 11.7 samples/s
+Step 16480/50462 | Loss: 2.6632 | LR: 3.86e-04 | 5129.4ms/step | 11.7 samples/s
+Step 16490/50462 | Loss: 2.6181 | LR: 3.86e-04 | 5124.7ms/step | 11.7 samples/s
+Step 16500/50462 | Loss: 2.6258 | LR: 3.86e-04 | 5128.2ms/step | 11.7 samples/s
+Validation at step 16500: Loss = 2.6528, Perplexity = 14.19
+Step 16510/50462 | Loss: 2.6129 | LR: 3.86e-04 | 7680.6ms/step | 7.8 samples/s
+Step 16520/50462 | Loss: 2.6417 | LR: 3.86e-04 | 5130.0ms/step | 11.7 samples/s
+Step 16530/50462 | Loss: 2.6001 | LR: 3.86e-04 | 5146.0ms/step | 11.7 samples/s
+Step 16540/50462 | Loss: 2.6207 | LR: 3.86e-04 | 5145.2ms/step | 11.7 samples/s
+Step 16550/50462 | Loss: 2.6425 | LR: 3.85e-04 | 5143.6ms/step | 11.7 samples/s
+Step 16560/50462 | Loss: 2.6522 | LR: 3.85e-04 | 5141.5ms/step | 11.7 samples/s
+Step 16570/50462 | Loss: 2.6331 | LR: 3.85e-04 | 5122.9ms/step | 11.7 samples/s
+Step 16580/50462 | Loss: 2.6494 | LR: 3.85e-04 | 5147.9ms/step | 11.7 samples/s
+Step 16590/50462 | Loss: 2.6398 | LR: 3.85e-04 | 5138.4ms/step | 11.7 samples/s
+Step 16600/50462 | Loss: 2.6581 | LR: 3.85e-04 | 5135.8ms/step | 11.7 samples/s
+Step 16610/50462 | Loss: 2.6392 | LR: 3.85e-04 | 5146.3ms/step | 11.7 samples/s
+Step 16620/50462 | Loss: 2.6561 | LR: 3.84e-04 | 5137.8ms/step | 11.7 samples/s
+Step 16630/50462 | Loss: 2.6344 | LR: 3.84e-04 | 5132.5ms/step | 11.7 samples/s
+Step 16640/50462 | Loss: 2.6109 | LR: 3.84e-04 | 5150.8ms/step | 11.6 samples/s
+Step 16650/50462 | Loss: 2.6408 | LR: 3.84e-04 | 5138.4ms/step | 11.7 samples/s
+Step 16660/50462 | Loss: 2.6405 | LR: 3.84e-04 | 5131.0ms/step | 11.7 samples/s
+Step 16670/50462 | Loss: 2.6276 | LR: 3.84e-04 | 5159.4ms/step | 11.6 samples/s
+Step 16680/50462 | Loss: 2.5996 | LR: 3.84e-04 | 5152.1ms/step | 11.6 samples/s
+Step 16690/50462 | Loss: 2.6270 | LR: 3.84e-04 | 5173.3ms/step | 11.6 samples/s
+Step 16700/50462 | Loss: 2.6546 | LR: 3.83e-04 | 5162.1ms/step | 11.6 samples/s
+Step 16710/50462 | Loss: 2.6676 | LR: 3.83e-04 | 5159.6ms/step | 11.6 samples/s
+Step 16720/50462 | Loss: 2.6422 | LR: 3.83e-04 | 5172.1ms/step | 11.6 samples/s
+Step 16730/50462 | Loss: 2.6283 | LR: 3.83e-04 | 5130.0ms/step | 11.7 samples/s
+Step 16740/50462 | Loss: 2.6154 | LR: 3.83e-04 | 5135.5ms/step | 11.7 samples/s
+Step 16750/50462 | Loss: 2.6760 | LR: 3.83e-04 | 5137.5ms/step | 11.7 samples/s
+Validation at step 16750: Loss = 2.6495, Perplexity = 14.15
+Step 16760/50462 | Loss: 2.6388 | LR: 3.83e-04 | 7701.2ms/step | 7.8 samples/s
+Step 16770/50462 | Loss: 2.6382 | LR: 3.82e-04 | 5138.8ms/step | 11.7 samples/s
+Step 16780/50462 | Loss: 2.6211 | LR: 3.82e-04 | 5142.1ms/step | 11.7 samples/s
+Step 16790/50462 | Loss: 2.6726 | LR: 3.82e-04 | 5138.7ms/step | 11.7 samples/s
+Step 16800/50462 | Loss: 2.6455 | LR: 3.82e-04 | 5134.6ms/step | 11.7 samples/s
+Step 16810/50462 | Loss: 2.6598 | LR: 3.82e-04 | 5143.9ms/step | 11.7 samples/s
+Step 16820/50462 | Loss: 2.6452 | LR: 3.82e-04 | 5155.0ms/step | 11.6 samples/s
+Step 16830/50462 | Loss: 2.6433 | LR: 3.82e-04 | 5135.8ms/step | 11.7 samples/s
+Step 16840/50462 | Loss: 2.6478 | LR: 3.82e-04 | 5136.4ms/step | 11.7 samples/s
+Step 16850/50462 | Loss: 2.6267 | LR: 3.81e-04 | 5138.9ms/step | 11.7 samples/s
+Step 16860/50462 | Loss: 2.6314 | LR: 3.81e-04 | 5147.0ms/step | 11.7 samples/s
+Step 16870/50462 | Loss: 2.6246 | LR: 3.81e-04 | 5139.7ms/step | 11.7 samples/s
+Step 16880/50462 | Loss: 2.6202 | LR: 3.81e-04 | 5152.9ms/step | 11.6 samples/s
+Step 16890/50462 | Loss: 2.6218 | LR: 3.81e-04 | 5151.4ms/step | 11.6 samples/s
+Step 16900/50462 | Loss: 2.6448 | LR: 3.81e-04 | 5142.6ms/step | 11.7 samples/s
+Step 16910/50462 | Loss: 2.6354 | LR: 3.81e-04 | 5156.2ms/step | 11.6 samples/s
+Step 16920/50462 | Loss: 2.6556 | LR: 3.80e-04 | 5159.1ms/step | 11.6 samples/s
+Step 16930/50462 | Loss: 2.6309 | LR: 3.80e-04 | 5142.7ms/step | 11.7 samples/s
+Step 16940/50462 | Loss: 2.6432 | LR: 3.80e-04 | 5158.1ms/step | 11.6 samples/s
+Step 16950/50462 | Loss: 2.6309 | LR: 3.80e-04 | 5150.1ms/step | 11.7 samples/s
+Step 16960/50462 | Loss: 2.6518 | LR: 3.80e-04 | 5152.3ms/step | 11.6 samples/s
+Step 16970/50462 | Loss: 2.6239 | LR: 3.80e-04 | 5154.2ms/step | 11.6 samples/s
+Step 16980/50462 | Loss: 2.6241 | LR: 3.80e-04 | 5147.6ms/step | 11.7 samples/s
+Step 16990/50462 | Loss: 2.6621 | LR: 3.79e-04 | 5142.6ms/step | 11.7 samples/s
+Step 17000/50462 | Loss: 2.6326 | LR: 3.79e-04 | 5161.9ms/step | 11.6 samples/s
+Validation at step 17000: Loss = 2.6466, Perplexity = 14.11
+Saved checkpoint to checkpoints/run_20260605_220251/checkpoint-17000
+Step 17010/50462 | Loss: 2.6683 | LR: 3.79e-04 | 8689.1ms/step | 6.9 samples/s
+Step 17020/50462 | Loss: 2.6044 | LR: 3.79e-04 | 5146.2ms/step | 11.7 samples/s
+Step 17030/50462 | Loss: 2.6269 | LR: 3.79e-04 | 5142.6ms/step | 11.7 samples/s
+Step 17040/50462 | Loss: 2.6625 | LR: 3.79e-04 | 5127.4ms/step | 11.7 samples/s
+Step 17050/50462 | Loss: 2.6087 | LR: 3.79e-04 | 5130.4ms/step | 11.7 samples/s
+Step 17060/50462 | Loss: 2.6157 | LR: 3.79e-04 | 5145.9ms/step | 11.7 samples/s
+Step 17070/50462 | Loss: 2.6476 | LR: 3.78e-04 | 5139.7ms/step | 11.7 samples/s
+Step 17080/50462 | Loss: 2.6527 | LR: 3.78e-04 | 5143.7ms/step | 11.7 samples/s
+Step 17090/50462 | Loss: 2.6566 | LR: 3.78e-04 | 5133.9ms/step | 11.7 samples/s
+Step 17100/50462 | Loss: 2.6638 | LR: 3.78e-04 | 5144.4ms/step | 11.7 samples/s
+Step 17110/50462 | Loss: 2.6549 | LR: 3.78e-04 | 5147.7ms/step | 11.7 samples/s
+Step 17120/50462 | Loss: 2.6294 | LR: 3.78e-04 | 5142.6ms/step | 11.7 samples/s
+Step 17130/50462 | Loss: 2.6279 | LR: 3.78e-04 | 5139.0ms/step | 11.7 samples/s
+Step 17140/50462 | Loss: 2.6472 | LR: 3.77e-04 | 5139.7ms/step | 11.7 samples/s
+Step 17150/50462 | Loss: 2.6334 | LR: 3.77e-04 | 5154.8ms/step | 11.6 samples/s
+Step 17160/50462 | Loss: 2.6081 | LR: 3.77e-04 | 5141.0ms/step | 11.7 samples/s
+Step 17170/50462 | Loss: 2.6319 | LR: 3.77e-04 | 5149.5ms/step | 11.7 samples/s
+Step 17180/50462 | Loss: 2.6027 | LR: 3.77e-04 | 5143.4ms/step | 11.7 samples/s
+Step 17190/50462 | Loss: 2.6461 | LR: 3.77e-04 | 5143.0ms/step | 11.7 samples/s
+Step 17200/50462 | Loss: 2.6185 | LR: 3.77e-04 | 5536.0ms/step | 10.8 samples/s
+Step 17210/50462 | Loss: 2.6604 | LR: 3.77e-04 | 5139.9ms/step | 11.7 samples/s
+Step 17220/50462 | Loss: 2.6608 | LR: 3.76e-04 | 5170.9ms/step | 11.6 samples/s
+Step 17230/50462 | Loss: 2.6628 | LR: 3.76e-04 | 5146.2ms/step | 11.7 samples/s
+Step 17240/50462 | Loss: 2.6302 | LR: 3.76e-04 | 5143.3ms/step | 11.7 samples/s
+Step 17250/50462 | Loss: 2.6308 | LR: 3.76e-04 | 5162.5ms/step | 11.6 samples/s
+Validation at step 17250: Loss = 2.6438, Perplexity = 14.07
+Step 17260/50462 | Loss: 2.6515 | LR: 3.76e-04 | 7728.0ms/step | 7.8 samples/s
+Step 17270/50462 | Loss: 2.5973 | LR: 3.76e-04 | 5151.5ms/step | 11.6 samples/s
+Step 17280/50462 | Loss: 2.6185 | LR: 3.76e-04 | 5142.8ms/step | 11.7 samples/s
+Step 17290/50462 | Loss: 2.6140 | LR: 3.75e-04 | 5152.1ms/step | 11.6 samples/s
+Step 17300/50462 | Loss: 2.6148 | LR: 3.75e-04 | 5150.9ms/step | 11.6 samples/s
+Step 17310/50462 | Loss: 2.6123 | LR: 3.75e-04 | 5142.9ms/step | 11.7 samples/s
+Step 17320/50462 | Loss: 2.6230 | LR: 3.75e-04 | 5136.9ms/step | 11.7 samples/s
+Step 17330/50462 | Loss: 2.6377 | LR: 3.75e-04 | 5150.6ms/step | 11.6 samples/s
+Step 17340/50462 | Loss: 2.6199 | LR: 3.75e-04 | 5137.2ms/step | 11.7 samples/s
+Step 17350/50462 | Loss: 2.6467 | LR: 3.75e-04 | 5156.8ms/step | 11.6 samples/s
+Step 17360/50462 | Loss: 2.6443 | LR: 3.74e-04 | 5147.2ms/step | 11.7 samples/s
+Step 17370/50462 | Loss: 2.6223 | LR: 3.74e-04 | 5138.6ms/step | 11.7 samples/s
+Step 17380/50462 | Loss: 2.6270 | LR: 3.74e-04 | 5560.1ms/step | 10.8 samples/s
+Step 17390/50462 | Loss: 2.6331 | LR: 3.74e-04 | 5618.8ms/step | 10.7 samples/s
+Step 17400/50462 | Loss: 2.6224 | LR: 3.74e-04 | 5198.6ms/step | 11.5 samples/s
+Step 17410/50462 | Loss: 2.6278 | LR: 3.74e-04 | 5194.4ms/step | 11.6 samples/s
+Step 17420/50462 | Loss: 2.6214 | LR: 3.74e-04 | 5200.3ms/step | 11.5 samples/s
+Step 17430/50462 | Loss: 2.6176 | LR: 3.74e-04 | 5197.8ms/step | 11.5 samples/s
+Step 17440/50462 | Loss: 2.6190 | LR: 3.73e-04 | 5197.9ms/step | 11.5 samples/s
+Step 17450/50462 | Loss: 2.6382 | LR: 3.73e-04 | 5190.4ms/step | 11.6 samples/s
+Step 17460/50462 | Loss: 2.6390 | LR: 3.73e-04 | 5199.0ms/step | 11.5 samples/s
+Step 17470/50462 | Loss: 2.6156 | LR: 3.73e-04 | 5186.1ms/step | 11.6 samples/s
+Step 17480/50462 | Loss: 2.6319 | LR: 3.73e-04 | 5203.7ms/step | 11.5 samples/s
+Step 17490/50462 | Loss: 2.6562 | LR: 3.73e-04 | 5222.8ms/step | 11.5 samples/s
+Step 17500/50462 | Loss: 2.6105 | LR: 3.73e-04 | 5208.8ms/step | 11.5 samples/s
+Validation at step 17500: Loss = 2.6413, Perplexity = 14.03
+Step 17510/50462 | Loss: 2.6211 | LR: 3.72e-04 | 7772.3ms/step | 7.7 samples/s
+Step 17520/50462 | Loss: 2.6171 | LR: 3.72e-04 | 5189.1ms/step | 11.6 samples/s
+Step 17530/50462 | Loss: 2.6271 | LR: 3.72e-04 | 5213.3ms/step | 11.5 samples/s
+Step 17540/50462 | Loss: 2.6110 | LR: 3.72e-04 | 5181.5ms/step | 11.6 samples/s
+Step 17550/50462 | Loss: 2.6286 | LR: 3.72e-04 | 5192.9ms/step | 11.6 samples/s
+Step 17560/50462 | Loss: 2.6365 | LR: 3.72e-04 | 5186.8ms/step | 11.6 samples/s
+Step 17570/50462 | Loss: 2.6448 | LR: 3.72e-04 | 5198.4ms/step | 11.5 samples/s
+Step 17580/50462 | Loss: 2.6331 | LR: 3.71e-04 | 5198.7ms/step | 11.5 samples/s
+Step 17590/50462 | Loss: 2.5946 | LR: 3.71e-04 | 5195.4ms/step | 11.5 samples/s
+Step 17600/50462 | Loss: 2.6316 | LR: 3.71e-04 | 5179.1ms/step | 11.6 samples/s
+Step 17610/50462 | Loss: 2.6038 | LR: 3.71e-04 | 5176.7ms/step | 11.6 samples/s
+Step 17620/50462 | Loss: 2.6201 | LR: 3.71e-04 | 5202.0ms/step | 11.5 samples/s
+Step 17630/50462 | Loss: 2.6067 | LR: 3.71e-04 | 5210.4ms/step | 11.5 samples/s
+Step 17640/50462 | Loss: 2.6161 | LR: 3.71e-04 | 5223.3ms/step | 11.5 samples/s
+Step 17650/50462 | Loss: 2.6185 | LR: 3.70e-04 | 5209.7ms/step | 11.5 samples/s
+Step 17660/50462 | Loss: 2.6202 | LR: 3.70e-04 | 5208.3ms/step | 11.5 samples/s
+Step 17670/50462 | Loss: 2.6419 | LR: 3.70e-04 | 5196.1ms/step | 11.5 samples/s
+Step 17680/50462 | Loss: 2.6219 | LR: 3.70e-04 | 5195.4ms/step | 11.5 samples/s
+Step 17690/50462 | Loss: 2.6292 | LR: 3.70e-04 | 5181.9ms/step | 11.6 samples/s
+Step 17700/50462 | Loss: 2.5982 | LR: 3.70e-04 | 5181.3ms/step | 11.6 samples/s
+Step 17710/50462 | Loss: 2.6520 | LR: 3.70e-04 | 5179.9ms/step | 11.6 samples/s
+Step 17720/50462 | Loss: 2.6028 | LR: 3.69e-04 | 5177.5ms/step | 11.6 samples/s
+Step 17730/50462 | Loss: 2.6453 | LR: 3.69e-04 | 5182.1ms/step | 11.6 samples/s
+Step 17740/50462 | Loss: 2.6363 | LR: 3.69e-04 | 5184.7ms/step | 11.6 samples/s
+Step 17750/50462 | Loss: 2.6202 | LR: 3.69e-04 | 5200.2ms/step | 11.5 samples/s
+Validation at step 17750: Loss = 2.6385, Perplexity = 13.99
+Step 17760/50462 | Loss: 2.6169 | LR: 3.69e-04 | 7773.4ms/step | 7.7 samples/s
+Step 17770/50462 | Loss: 2.6243 | LR: 3.69e-04 | 5216.8ms/step | 11.5 samples/s
+Step 17780/50462 | Loss: 2.6614 | LR: 3.69e-04 | 5183.8ms/step | 11.6 samples/s
+Step 17790/50462 | Loss: 2.6209 | LR: 3.69e-04 | 5200.6ms/step | 11.5 samples/s
+Step 17800/50462 | Loss: 2.6096 | LR: 3.68e-04 | 5212.9ms/step | 11.5 samples/s
+Step 17810/50462 | Loss: 2.6301 | LR: 3.68e-04 | 5210.4ms/step | 11.5 samples/s
+Step 17820/50462 | Loss: 2.6337 | LR: 3.68e-04 | 5203.6ms/step | 11.5 samples/s
+Step 17830/50462 | Loss: 2.6401 | LR: 3.68e-04 | 5204.0ms/step | 11.5 samples/s
+Step 17840/50462 | Loss: 2.6393 | LR: 3.68e-04 | 5209.9ms/step | 11.5 samples/s
+Step 17850/50462 | Loss: 2.6107 | LR: 3.68e-04 | 5205.0ms/step | 11.5 samples/s
+Step 17860/50462 | Loss: 2.6052 | LR: 3.68e-04 | 5204.6ms/step | 11.5 samples/s
+Step 17870/50462 | Loss: 2.6217 | LR: 3.67e-04 | 5201.1ms/step | 11.5 samples/s
+Step 17880/50462 | Loss: 2.6211 | LR: 3.67e-04 | 5218.7ms/step | 11.5 samples/s
+Step 17890/50462 | Loss: 2.6309 | LR: 3.67e-04 | 5211.4ms/step | 11.5 samples/s
+Step 17900/50462 | Loss: 2.6472 | LR: 3.67e-04 | 5198.8ms/step | 11.5 samples/s
+Step 17910/50462 | Loss: 2.6086 | LR: 3.67e-04 | 5202.8ms/step | 11.5 samples/s
+Step 17920/50462 | Loss: 2.6385 | LR: 3.67e-04 | 5204.5ms/step | 11.5 samples/s
+Step 17930/50462 | Loss: 2.6472 | LR: 3.67e-04 | 5210.1ms/step | 11.5 samples/s
+Step 17940/50462 | Loss: 2.6151 | LR: 3.66e-04 | 5207.7ms/step | 11.5 samples/s
+Step 17950/50462 | Loss: 2.6184 | LR: 3.66e-04 | 5206.9ms/step | 11.5 samples/s
+Step 17960/50462 | Loss: 2.6147 | LR: 3.66e-04 | 5198.4ms/step | 11.5 samples/s
+Step 17970/50462 | Loss: 2.6219 | LR: 3.66e-04 | 5198.2ms/step | 11.5 samples/s
+Step 17980/50462 | Loss: 2.6219 | LR: 3.66e-04 | 5207.4ms/step | 11.5 samples/s
+Step 17990/50462 | Loss: 2.6707 | LR: 3.66e-04 | 5206.3ms/step | 11.5 samples/s
+Step 18000/50462 | Loss: 2.6515 | LR: 3.66e-04 | 5215.9ms/step | 11.5 samples/s
+Validation at step 18000: Loss = 2.6358, Perplexity = 13.95
+Saved checkpoint to checkpoints/run_20260605_220251/checkpoint-18000
+Step 18010/50462 | Loss: 2.6095 | LR: 3.65e-04 | 8679.9ms/step | 6.9 samples/s
+Step 18020/50462 | Loss: 2.6193 | LR: 3.65e-04 | 5147.9ms/step | 11.7 samples/s
+Step 18030/50462 | Loss: 2.6021 | LR: 3.65e-04 | 5143.8ms/step | 11.7 samples/s
+Step 18040/50462 | Loss: 2.6266 | LR: 3.65e-04 | 5126.6ms/step | 11.7 samples/s
+Step 18050/50462 | Loss: 2.6143 | LR: 3.65e-04 | 5143.0ms/step | 11.7 samples/s
+Step 18060/50462 | Loss: 2.6570 | LR: 3.65e-04 | 5150.3ms/step | 11.6 samples/s
+Step 18070/50462 | Loss: 2.6451 | LR: 3.65e-04 | 5149.0ms/step | 11.7 samples/s
+Step 18080/50462 | Loss: 2.6301 | LR: 3.64e-04 | 5143.4ms/step | 11.7 samples/s
+Step 18090/50462 | Loss: 2.5995 | LR: 3.64e-04 | 5143.5ms/step | 11.7 samples/s
+Step 18100/50462 | Loss: 2.6106 | LR: 3.64e-04 | 5148.7ms/step | 11.7 samples/s
+Step 18110/50462 | Loss: 2.6139 | LR: 3.64e-04 | 5158.1ms/step | 11.6 samples/s
+Step 18120/50462 | Loss: 2.5935 | LR: 3.64e-04 | 5140.3ms/step | 11.7 samples/s
+Step 18130/50462 | Loss: 2.5905 | LR: 3.64e-04 | 5134.9ms/step | 11.7 samples/s
+Step 18140/50462 | Loss: 2.6268 | LR: 3.64e-04 | 5151.3ms/step | 11.6 samples/s
+Step 18150/50462 | Loss: 2.6124 | LR: 3.63e-04 | 5139.0ms/step | 11.7 samples/s
+Step 18160/50462 | Loss: 2.6251 | LR: 3.63e-04 | 5138.1ms/step | 11.7 samples/s
+Step 18170/50462 | Loss: 2.6389 | LR: 3.63e-04 | 5137.4ms/step | 11.7 samples/s
+Step 18180/50462 | Loss: 2.6453 | LR: 3.63e-04 | 5148.1ms/step | 11.7 samples/s
+Step 18190/50462 | Loss: 2.6137 | LR: 3.63e-04 | 5160.4ms/step | 11.6 samples/s
+Step 18200/50462 | Loss: 2.6073 | LR: 3.63e-04 | 5136.1ms/step | 11.7 samples/s
+Step 18210/50462 | Loss: 2.6112 | LR: 3.63e-04 | 5133.1ms/step | 11.7 samples/s
+Step 18220/50462 | Loss: 2.6142 | LR: 3.62e-04 | 5151.7ms/step | 11.6 samples/s
+Step 18230/50462 | Loss: 2.6364 | LR: 3.62e-04 | 5148.6ms/step | 11.7 samples/s
+Step 18240/50462 | Loss: 2.5911 | LR: 3.62e-04 | 5137.1ms/step | 11.7 samples/s
+Step 18250/50462 | Loss: 2.6191 | LR: 3.62e-04 | 5151.8ms/step | 11.6 samples/s
+Validation at step 18250: Loss = 2.6334, Perplexity = 13.92
+Step 18260/50462 | Loss: 2.6205 | LR: 3.62e-04 | 7706.3ms/step | 7.8 samples/s
+Step 18270/50462 | Loss: 2.6256 | LR: 3.62e-04 | 5145.6ms/step | 11.7 samples/s
+Step 18280/50462 | Loss: 2.5843 | LR: 3.62e-04 | 5156.3ms/step | 11.6 samples/s
+Step 18290/50462 | Loss: 2.6176 | LR: 3.62e-04 | 5146.2ms/step | 11.7 samples/s
+Step 18300/50462 | Loss: 2.6361 | LR: 3.61e-04 | 5136.1ms/step | 11.7 samples/s
+Step 18310/50462 | Loss: 2.6164 | LR: 3.61e-04 | 5142.9ms/step | 11.7 samples/s
+Step 18320/50462 | Loss: 2.6160 | LR: 3.61e-04 | 5138.4ms/step | 11.7 samples/s
+Step 18330/50462 | Loss: 2.6639 | LR: 3.61e-04 | 5180.2ms/step | 11.6 samples/s
+Step 18340/50462 | Loss: 2.6113 | LR: 3.61e-04 | 5136.5ms/step | 11.7 samples/s
+Step 18350/50462 | Loss: 2.6539 | LR: 3.61e-04 | 5145.7ms/step | 11.7 samples/s
+Step 18360/50462 | Loss: 2.6047 | LR: 3.61e-04 | 5137.1ms/step | 11.7 samples/s
+Step 18370/50462 | Loss: 2.6266 | LR: 3.60e-04 | 5139.8ms/step | 11.7 samples/s
+Step 18380/50462 | Loss: 2.5854 | LR: 3.60e-04 | 5136.1ms/step | 11.7 samples/s
+Step 18390/50462 | Loss: 2.6244 | LR: 3.60e-04 | 5136.5ms/step | 11.7 samples/s
+Step 18400/50462 | Loss: 2.5852 | LR: 3.60e-04 | 5146.6ms/step | 11.7 samples/s
+Step 18410/50462 | Loss: 2.6230 | LR: 3.60e-04 | 5146.7ms/step | 11.7 samples/s
+Step 18420/50462 | Loss: 2.6279 | LR: 3.60e-04 | 5157.2ms/step | 11.6 samples/s
+Step 18430/50462 | Loss: 2.6499 | LR: 3.60e-04 | 5147.6ms/step | 11.7 samples/s
+Step 18440/50462 | Loss: 2.6139 | LR: 3.59e-04 | 5148.0ms/step | 11.7 samples/s
+Step 18450/50462 | Loss: 2.6206 | LR: 3.59e-04 | 5133.8ms/step | 11.7 samples/s
+Step 18460/50462 | Loss: 2.6207 | LR: 3.59e-04 | 5163.2ms/step | 11.6 samples/s
+Step 18470/50462 | Loss: 2.6202 | LR: 3.59e-04 | 5150.0ms/step | 11.7 samples/s
+Step 18480/50462 | Loss: 2.6106 | LR: 3.59e-04 | 5153.4ms/step | 11.6 samples/s
+Step 18490/50462 | Loss: 2.6123 | LR: 3.59e-04 | 5153.0ms/step | 11.6 samples/s
+Step 18500/50462 | Loss: 2.6263 | LR: 3.59e-04 | 5149.5ms/step | 11.7 samples/s
+Validation at step 18500: Loss = 2.6299, Perplexity = 13.87
+Step 18510/50462 | Loss: 2.6066 | LR: 3.58e-04 | 7705.5ms/step | 7.8 samples/s
+Step 18520/50462 | Loss: 2.6185 | LR: 3.58e-04 | 5154.9ms/step | 11.6 samples/s
+Step 18530/50462 | Loss: 2.6343 | LR: 3.58e-04 | 5145.3ms/step | 11.7 samples/s
+Step 18540/50462 | Loss: 2.6050 | LR: 3.58e-04 | 5152.5ms/step | 11.6 samples/s
+Step 18550/50462 | Loss: 2.6385 | LR: 3.58e-04 | 5156.2ms/step | 11.6 samples/s
+Step 18560/50462 | Loss: 2.6030 | LR: 3.58e-04 | 5172.6ms/step | 11.6 samples/s
+Step 18570/50462 | Loss: 2.6082 | LR: 3.58e-04 | 5160.1ms/step | 11.6 samples/s
+Step 18580/50462 | Loss: 2.6445 | LR: 3.57e-04 | 5163.3ms/step | 11.6 samples/s
+Step 18590/50462 | Loss: 2.6213 | LR: 3.57e-04 | 5173.5ms/step | 11.6 samples/s
+Step 18600/50462 | Loss: 2.5923 | LR: 3.57e-04 | 5151.6ms/step | 11.6 samples/s
+Step 18610/50462 | Loss: 2.6216 | LR: 3.57e-04 | 5132.7ms/step | 11.7 samples/s
+Step 18620/50462 | Loss: 2.6230 | LR: 3.57e-04 | 5135.4ms/step | 11.7 samples/s
+Step 18630/50462 | Loss: 2.6365 | LR: 3.57e-04 | 5143.8ms/step | 11.7 samples/s
+Step 18640/50462 | Loss: 2.6284 | LR: 3.57e-04 | 5136.4ms/step | 11.7 samples/s
+Step 18650/50462 | Loss: 2.6360 | LR: 3.56e-04 | 5165.2ms/step | 11.6 samples/s
+Step 18660/50462 | Loss: 2.6086 | LR: 3.56e-04 | 5172.4ms/step | 11.6 samples/s
+Step 18670/50462 | Loss: 2.6029 | LR: 3.56e-04 | 5149.0ms/step | 11.7 samples/s
+Step 18680/50462 | Loss: 2.6274 | LR: 3.56e-04 | 5158.1ms/step | 11.6 samples/s
+Step 18690/50462 | Loss: 2.6291 | LR: 3.56e-04 | 5141.6ms/step | 11.7 samples/s
+Step 18700/50462 | Loss: 2.6287 | LR: 3.56e-04 | 5154.5ms/step | 11.6 samples/s
+Step 18710/50462 | Loss: 2.6193 | LR: 3.56e-04 | 5156.4ms/step | 11.6 samples/s
+Step 18720/50462 | Loss: 2.6201 | LR: 3.55e-04 | 5154.6ms/step | 11.6 samples/s
+Step 18730/50462 | Loss: 2.6268 | LR: 3.55e-04 | 5138.2ms/step | 11.7 samples/s
+Step 18740/50462 | Loss: 2.6105 | LR: 3.55e-04 | 5150.2ms/step | 11.7 samples/s
+Step 18750/50462 | Loss: 2.6361 | LR: 3.55e-04 | 5146.6ms/step | 11.7 samples/s
+Validation at step 18750: Loss = 2.6279, Perplexity = 13.85
+Step 18760/50462 | Loss: 2.6173 | LR: 3.55e-04 | 7709.6ms/step | 7.8 samples/s
+Step 18770/50462 | Loss: 2.6148 | LR: 3.55e-04 | 5152.8ms/step | 11.6 samples/s
+Step 18780/50462 | Loss: 2.6440 | LR: 3.55e-04 | 5160.0ms/step | 11.6 samples/s
+Step 18790/50462 | Loss: 2.6310 | LR: 3.54e-04 | 5143.1ms/step | 11.7 samples/s
+Step 18800/50462 | Loss: 2.6444 | LR: 3.54e-04 | 5131.7ms/step | 11.7 samples/s
+Step 18810/50462 | Loss: 2.6102 | LR: 3.54e-04 | 5162.9ms/step | 11.6 samples/s
+Step 18820/50462 | Loss: 2.5934 | LR: 3.54e-04 | 5139.8ms/step | 11.7 samples/s
+Step 18830/50462 | Loss: 2.6036 | LR: 3.54e-04 | 5163.5ms/step | 11.6 samples/s
+Step 18840/50462 | Loss: 2.6066 | LR: 3.54e-04 | 5149.4ms/step | 11.7 samples/s
+Step 18850/50462 | Loss: 2.5982 | LR: 3.54e-04 | 5170.2ms/step | 11.6 samples/s
+Step 18860/50462 | Loss: 2.6275 | LR: 3.53e-04 | 5166.4ms/step | 11.6 samples/s
+Step 18870/50462 | Loss: 2.6330 | LR: 3.53e-04 | 5164.7ms/step | 11.6 samples/s
+Step 18880/50462 | Loss: 2.6756 | LR: 3.53e-04 | 5167.6ms/step | 11.6 samples/s
+Step 18890/50462 | Loss: 2.6034 | LR: 3.53e-04 | 5190.3ms/step | 11.6 samples/s
+Step 18900/50462 | Loss: 2.6286 | LR: 3.53e-04 | 5180.0ms/step | 11.6 samples/s
+Step 18910/50462 | Loss: 2.6232 | LR: 3.53e-04 | 5170.3ms/step | 11.6 samples/s
+Step 18920/50462 | Loss: 2.6115 | LR: 3.53e-04 | 5166.2ms/step | 11.6 samples/s
+Step 18930/50462 | Loss: 2.6441 | LR: 3.52e-04 | 5150.4ms/step | 11.6 samples/s
+Step 18940/50462 | Loss: 2.6253 | LR: 3.52e-04 | 5152.8ms/step | 11.6 samples/s
+Step 18950/50462 | Loss: 2.6229 | LR: 3.52e-04 | 5171.5ms/step | 11.6 samples/s
+Step 18960/50462 | Loss: 2.6319 | LR: 3.52e-04 | 5178.8ms/step | 11.6 samples/s
+Step 18970/50462 | Loss: 2.6125 | LR: 3.52e-04 | 5174.2ms/step | 11.6 samples/s
+Step 18980/50462 | Loss: 2.6344 | LR: 3.52e-04 | 5163.9ms/step | 11.6 samples/s
+Step 18990/50462 | Loss: 2.6009 | LR: 3.52e-04 | 5181.7ms/step | 11.6 samples/s
+Step 19000/50462 | Loss: 2.6225 | LR: 3.51e-04 | 5148.3ms/step | 11.7 samples/s
+Validation at step 19000: Loss = 2.6239, Perplexity = 13.79
+Saved checkpoint to checkpoints/run_20260605_220251/checkpoint-19000
+Step 19010/50462 | Loss: 2.6148 | LR: 3.51e-04 | 8667.3ms/step | 6.9 samples/s
+Step 19020/50462 | Loss: 2.6317 | LR: 3.51e-04 | 5140.9ms/step | 11.7 samples/s
+Step 19030/50462 | Loss: 2.6327 | LR: 3.51e-04 | 5143.8ms/step | 11.7 samples/s
+Step 19040/50462 | Loss: 2.6120 | LR: 3.51e-04 | 5141.4ms/step | 11.7 samples/s
+Step 19050/50462 | Loss: 2.6143 | LR: 3.51e-04 | 5145.4ms/step | 11.7 samples/s
+Step 19060/50462 | Loss: 2.6035 | LR: 3.50e-04 | 5145.8ms/step | 11.7 samples/s
+Step 19070/50462 | Loss: 2.5970 | LR: 3.50e-04 | 5138.1ms/step | 11.7 samples/s
+Step 19080/50462 | Loss: 2.5962 | LR: 3.50e-04 | 5156.9ms/step | 11.6 samples/s
+Step 19090/50462 | Loss: 2.6099 | LR: 3.50e-04 | 5141.9ms/step | 11.7 samples/s
+Step 19100/50462 | Loss: 2.6112 | LR: 3.50e-04 | 5151.2ms/step | 11.6 samples/s
+Step 19110/50462 | Loss: 2.5926 | LR: 3.50e-04 | 5142.7ms/step | 11.7 samples/s
+Step 19120/50462 | Loss: 2.6182 | LR: 3.50e-04 | 5161.7ms/step | 11.6 samples/s
+Step 19130/50462 | Loss: 2.6510 | LR: 3.49e-04 | 5143.6ms/step | 11.7 samples/s
+Step 19140/50462 | Loss: 2.6090 | LR: 3.49e-04 | 5145.5ms/step | 11.7 samples/s
+Step 19150/50462 | Loss: 2.6233 | LR: 3.49e-04 | 5147.6ms/step | 11.7 samples/s
+Step 19160/50462 | Loss: 2.6155 | LR: 3.49e-04 | 5138.4ms/step | 11.7 samples/s
+Step 19170/50462 | Loss: 2.5994 | LR: 3.49e-04 | 5143.9ms/step | 11.7 samples/s
+Step 19180/50462 | Loss: 2.6140 | LR: 3.49e-04 | 5132.1ms/step | 11.7 samples/s
+Step 19190/50462 | Loss: 2.5901 | LR: 3.49e-04 | 5138.0ms/step | 11.7 samples/s
+Step 19200/50462 | Loss: 2.6044 | LR: 3.48e-04 | 5138.7ms/step | 11.7 samples/s
+Step 19210/50462 | Loss: 2.6001 | LR: 3.48e-04 | 5153.1ms/step | 11.6 samples/s
+Step 19220/50462 | Loss: 2.6136 | LR: 3.48e-04 | 5132.0ms/step | 11.7 samples/s
+Step 19230/50462 | Loss: 2.6014 | LR: 3.48e-04 | 5141.0ms/step | 11.7 samples/s
+Step 19240/50462 | Loss: 2.6166 | LR: 3.48e-04 | 5526.4ms/step | 10.9 samples/s
+Step 19250/50462 | Loss: 2.6036 | LR: 3.48e-04 | 5188.4ms/step | 11.6 samples/s
+Validation at step 19250: Loss = 2.6223, Perplexity = 13.77
+Step 19260/50462 | Loss: 2.5903 | LR: 3.48e-04 | 7754.6ms/step | 7.7 samples/s
+Step 19270/50462 | Loss: 2.6008 | LR: 3.47e-04 | 5208.3ms/step | 11.5 samples/s
+Step 19280/50462 | Loss: 2.6033 | LR: 3.47e-04 | 5180.4ms/step | 11.6 samples/s
+Step 19290/50462 | Loss: 2.6331 | LR: 3.47e-04 | 5202.3ms/step | 11.5 samples/s
+Step 19300/50462 | Loss: 2.5992 | LR: 3.47e-04 | 5199.7ms/step | 11.5 samples/s
+Step 19310/50462 | Loss: 2.5957 | LR: 3.47e-04 | 5193.8ms/step | 11.6 samples/s
+Step 19320/50462 | Loss: 2.5702 | LR: 3.47e-04 | 5192.2ms/step | 11.6 samples/s
+Step 19330/50462 | Loss: 2.6008 | LR: 3.47e-04 | 5187.5ms/step | 11.6 samples/s
+Step 19340/50462 | Loss: 2.5877 | LR: 3.46e-04 | 5189.1ms/step | 11.6 samples/s
+Step 19350/50462 | Loss: 2.6327 | LR: 3.46e-04 | 5200.2ms/step | 11.5 samples/s
+Step 19360/50462 | Loss: 2.6238 | LR: 3.46e-04 | 5197.6ms/step | 11.5 samples/s
+Step 19370/50462 | Loss: 2.5906 | LR: 3.46e-04 | 5185.1ms/step | 11.6 samples/s
+Step 19380/50462 | Loss: 2.6137 | LR: 3.46e-04 | 5181.9ms/step | 11.6 samples/s
+Step 19390/50462 | Loss: 2.6167 | LR: 3.46e-04 | 5188.5ms/step | 11.6 samples/s
+Step 19400/50462 | Loss: 2.5913 | LR: 3.46e-04 | 5183.8ms/step | 11.6 samples/s
+Step 19410/50462 | Loss: 2.6007 | LR: 3.45e-04 | 5192.8ms/step | 11.6 samples/s
+Step 19420/50462 | Loss: 2.5842 | LR: 3.45e-04 | 5181.2ms/step | 11.6 samples/s
+Step 19430/50462 | Loss: 2.6390 | LR: 3.45e-04 | 5185.2ms/step | 11.6 samples/s
+Step 19440/50462 | Loss: 2.6060 | LR: 3.45e-04 | 5170.6ms/step | 11.6 samples/s
+Step 19450/50462 | Loss: 2.6109 | LR: 3.45e-04 | 5200.1ms/step | 11.5 samples/s
+Step 19460/50462 | Loss: 2.6036 | LR: 3.45e-04 | 5185.1ms/step | 11.6 samples/s
+Step 19470/50462 | Loss: 2.6363 | LR: 3.45e-04 | 5178.9ms/step | 11.6 samples/s
+Step 19480/50462 | Loss: 2.5964 | LR: 3.44e-04 | 5629.7ms/step | 10.7 samples/s
+Step 19490/50462 | Loss: 2.6277 | LR: 3.44e-04 | 5179.7ms/step | 11.6 samples/s
+Step 19500/50462 | Loss: 2.6118 | LR: 3.44e-04 | 5195.0ms/step | 11.5 samples/s
+Validation at step 19500: Loss = 2.6204, Perplexity = 13.74
+Step 19510/50462 | Loss: 2.6114 | LR: 3.44e-04 | 8131.2ms/step | 7.4 samples/s
+Step 19520/50462 | Loss: 2.5835 | LR: 3.44e-04 | 5175.3ms/step | 11.6 samples/s
+Step 19530/50462 | Loss: 2.6101 | LR: 3.44e-04 | 5182.5ms/step | 11.6 samples/s
+Step 19540/50462 | Loss: 2.6208 | LR: 3.44e-04 | 5189.9ms/step | 11.6 samples/s
+Step 19550/50462 | Loss: 2.6138 | LR: 3.43e-04 | 5181.9ms/step | 11.6 samples/s
+Step 19560/50462 | Loss: 2.6182 | LR: 3.43e-04 | 5208.2ms/step | 11.5 samples/s
+Step 19570/50462 | Loss: 2.5657 | LR: 3.43e-04 | 5180.0ms/step | 11.6 samples/s
+Step 19580/50462 | Loss: 2.5903 | LR: 3.43e-04 | 5188.9ms/step | 11.6 samples/s
+Step 19590/50462 | Loss: 2.6204 | LR: 3.43e-04 | 5143.1ms/step | 11.7 samples/s
+Step 19600/50462 | Loss: 2.5816 | LR: 3.43e-04 | 5157.9ms/step | 11.6 samples/s
+Step 19610/50462 | Loss: 2.6356 | LR: 3.42e-04 | 5145.3ms/step | 11.7 samples/s
+Step 19620/50462 | Loss: 2.5852 | LR: 3.42e-04 | 5138.5ms/step | 11.7 samples/s
+Step 19630/50462 | Loss: 2.5861 | LR: 3.42e-04 | 5146.4ms/step | 11.7 samples/s
+Step 19640/50462 | Loss: 2.6253 | LR: 3.42e-04 | 5154.2ms/step | 11.6 samples/s
+Step 19650/50462 | Loss: 2.5939 | LR: 3.42e-04 | 5143.3ms/step | 11.7 samples/s
+Step 19660/50462 | Loss: 2.5872 | LR: 3.42e-04 | 5138.5ms/step | 11.7 samples/s
+Step 19670/50462 | Loss: 2.6313 | LR: 3.42e-04 | 5121.2ms/step | 11.7 samples/s
+Step 19680/50462 | Loss: 2.6083 | LR: 3.41e-04 | 5130.8ms/step | 11.7 samples/s
+Step 19690/50462 | Loss: 2.6143 | LR: 3.41e-04 | 5148.2ms/step | 11.7 samples/s
+Step 19700/50462 | Loss: 2.6113 | LR: 3.41e-04 | 5119.9ms/step | 11.7 samples/s
+Step 19710/50462 | Loss: 2.6157 | LR: 3.41e-04 | 5128.1ms/step | 11.7 samples/s
+Step 19720/50462 | Loss: 2.5894 | LR: 3.41e-04 | 5110.4ms/step | 11.7 samples/s
+Step 19730/50462 | Loss: 2.5954 | LR: 3.41e-04 | 5159.3ms/step | 11.6 samples/s
+Step 19740/50462 | Loss: 2.6442 | LR: 3.41e-04 | 5160.8ms/step | 11.6 samples/s
+Step 19750/50462 | Loss: 2.6202 | LR: 3.40e-04 | 5129.1ms/step | 11.7 samples/s
+Validation at step 19750: Loss = 2.6170, Perplexity = 13.70
+Step 19760/50462 | Loss: 2.6084 | LR: 3.40e-04 | 7696.3ms/step | 7.8 samples/s
+Step 19770/50462 | Loss: 2.6369 | LR: 3.40e-04 | 5136.4ms/step | 11.7 samples/s
+Step 19780/50462 | Loss: 2.6298 | LR: 3.40e-04 | 5154.0ms/step | 11.6 samples/s
+Step 19790/50462 | Loss: 2.5609 | LR: 3.40e-04 | 5147.1ms/step | 11.7 samples/s
+Step 19800/50462 | Loss: 2.6199 | LR: 3.40e-04 | 5157.7ms/step | 11.6 samples/s
+Step 19810/50462 | Loss: 2.6037 | LR: 3.40e-04 | 5151.7ms/step | 11.6 samples/s
+Step 19820/50462 | Loss: 2.6086 | LR: 3.39e-04 | 5135.4ms/step | 11.7 samples/s
+Step 19830/50462 | Loss: 2.6241 | LR: 3.39e-04 | 5143.3ms/step | 11.7 samples/s
+Step 19840/50462 | Loss: 2.5920 | LR: 3.39e-04 | 5188.1ms/step | 11.6 samples/s
+Step 19850/50462 | Loss: 2.5902 | LR: 3.39e-04 | 5184.9ms/step | 11.6 samples/s
+Step 19860/50462 | Loss: 2.6000 | LR: 3.39e-04 | 5178.6ms/step | 11.6 samples/s
+Step 19870/50462 | Loss: 2.5907 | LR: 3.39e-04 | 5192.0ms/step | 11.6 samples/s
+Step 19880/50462 | Loss: 2.6032 | LR: 3.39e-04 | 5181.1ms/step | 11.6 samples/s
+Step 19890/50462 | Loss: 2.5946 | LR: 3.38e-04 | 5195.1ms/step | 11.5 samples/s
+Step 19900/50462 | Loss: 2.6101 | LR: 3.38e-04 | 5189.4ms/step | 11.6 samples/s
+Step 19910/50462 | Loss: 2.5723 | LR: 3.38e-04 | 5248.1ms/step | 11.4 samples/s
+Step 19920/50462 | Loss: 2.5953 | LR: 3.38e-04 | 5181.5ms/step | 11.6 samples/s
+Step 19930/50462 | Loss: 2.5903 | LR: 3.38e-04 | 5179.4ms/step | 11.6 samples/s
+Step 19940/50462 | Loss: 2.5683 | LR: 3.38e-04 | 5183.0ms/step | 11.6 samples/s
+Step 19950/50462 | Loss: 2.6182 | LR: 3.37e-04 | 5182.4ms/step | 11.6 samples/s
+Step 19960/50462 | Loss: 2.6072 | LR: 3.37e-04 | 5195.9ms/step | 11.5 samples/s
+Step 19970/50462 | Loss: 2.6108 | LR: 3.37e-04 | 5188.3ms/step | 11.6 samples/s
+Step 19980/50462 | Loss: 2.5686 | LR: 3.37e-04 | 5189.8ms/step | 11.6 samples/s
+Step 19990/50462 | Loss: 2.5887 | LR: 3.37e-04 | 5185.6ms/step | 11.6 samples/s
+Step 20000/50462 | Loss: 2.5833 | LR: 3.37e-04 | 5186.5ms/step | 11.6 samples/s
+Validation at step 20000: Loss = 2.6149, Perplexity = 13.67
+Saved checkpoint to checkpoints/run_20260605_220251/checkpoint-20000
+Step 20010/50462 | Loss: 2.5916 | LR: 3.37e-04 | 8670.2ms/step | 6.9 samples/s
+Step 20020/50462 | Loss: 2.5984 | LR: 3.36e-04 | 5176.6ms/step | 11.6 samples/s
+Step 20030/50462 | Loss: 2.6021 | LR: 3.36e-04 | 5155.2ms/step | 11.6 samples/s
+Step 20040/50462 | Loss: 2.6259 | LR: 3.36e-04 | 5134.9ms/step | 11.7 samples/s
+Step 20050/50462 | Loss: 2.6180 | LR: 3.36e-04 | 5144.5ms/step | 11.7 samples/s
+Step 20060/50462 | Loss: 2.5803 | LR: 3.36e-04 | 5142.6ms/step | 11.7 samples/s
+Step 20070/50462 | Loss: 2.5881 | LR: 3.36e-04 | 5139.1ms/step | 11.7 samples/s
+Step 20080/50462 | Loss: 2.5863 | LR: 3.36e-04 | 5149.7ms/step | 11.7 samples/s
+Step 20090/50462 | Loss: 2.6105 | LR: 3.35e-04 | 5143.4ms/step | 11.7 samples/s
+Step 20100/50462 | Loss: 2.6266 | LR: 3.35e-04 | 5155.1ms/step | 11.6 samples/s
+Step 20110/50462 | Loss: 2.5780 | LR: 3.35e-04 | 5142.5ms/step | 11.7 samples/s
+Step 20120/50462 | Loss: 2.6049 | LR: 3.35e-04 | 5150.5ms/step | 11.6 samples/s
+Step 20130/50462 | Loss: 2.6287 | LR: 3.35e-04 | 5148.7ms/step | 11.7 samples/s
+Step 20140/50462 | Loss: 2.5682 | LR: 3.35e-04 | 5150.2ms/step | 11.6 samples/s
+Step 20150/50462 | Loss: 2.6121 | LR: 3.35e-04 | 5140.9ms/step | 11.7 samples/s
+Step 20160/50462 | Loss: 2.5926 | LR: 3.34e-04 | 5149.7ms/step | 11.7 samples/s
+Step 20170/50462 | Loss: 2.6369 | LR: 3.34e-04 | 5137.2ms/step | 11.7 samples/s
+Step 20180/50462 | Loss: 2.5669 | LR: 3.34e-04 | 5150.1ms/step | 11.7 samples/s
+Step 20190/50462 | Loss: 2.6003 | LR: 3.34e-04 | 5143.1ms/step | 11.7 samples/s
+Step 20200/50462 | Loss: 2.6043 | LR: 3.34e-04 | 5141.1ms/step | 11.7 samples/s
+Step 20210/50462 | Loss: 2.6251 | LR: 3.34e-04 | 5136.7ms/step | 11.7 samples/s
+Step 20220/50462 | Loss: 2.6181 | LR: 3.33e-04 | 5148.5ms/step | 11.7 samples/s
+Step 20230/50462 | Loss: 2.5981 | LR: 3.33e-04 | 5148.1ms/step | 11.7 samples/s
+Step 20240/50462 | Loss: 2.5938 | LR: 3.33e-04 | 5134.9ms/step | 11.7 samples/s
+Step 20250/50462 | Loss: 2.5940 | LR: 3.33e-04 | 5142.2ms/step | 11.7 samples/s
+Validation at step 20250: Loss = 2.6130, Perplexity = 13.64
+Step 20260/50462 | Loss: 2.6207 | LR: 3.33e-04 | 7714.0ms/step | 7.8 samples/s
+Step 20270/50462 | Loss: 2.6458 | LR: 3.33e-04 | 5143.6ms/step | 11.7 samples/s
+Step 20280/50462 | Loss: 2.5617 | LR: 3.33e-04 | 5153.5ms/step | 11.6 samples/s
+Step 20290/50462 | Loss: 2.6268 | LR: 3.32e-04 | 5167.9ms/step | 11.6 samples/s
+Step 20300/50462 | Loss: 2.6034 | LR: 3.32e-04 | 5164.8ms/step | 11.6 samples/s
+Step 20310/50462 | Loss: 2.5994 | LR: 3.32e-04 | 5151.2ms/step | 11.6 samples/s
+Step 20320/50462 | Loss: 2.5947 | LR: 3.32e-04 | 5129.1ms/step | 11.7 samples/s
+Step 20330/50462 | Loss: 2.6248 | LR: 3.32e-04 | 5152.0ms/step | 11.6 samples/s
+Step 20340/50462 | Loss: 2.5865 | LR: 3.32e-04 | 5157.4ms/step | 11.6 samples/s
+Step 20350/50462 | Loss: 2.5823 | LR: 3.32e-04 | 5137.9ms/step | 11.7 samples/s
+Step 20360/50462 | Loss: 2.6226 | LR: 3.31e-04 | 5143.5ms/step | 11.7 samples/s
+Step 20370/50462 | Loss: 2.6072 | LR: 3.31e-04 | 5138.6ms/step | 11.7 samples/s
+Step 20380/50462 | Loss: 2.6250 | LR: 3.31e-04 | 5145.6ms/step | 11.7 samples/s
+Step 20390/50462 | Loss: 2.5893 | LR: 3.31e-04 | 5153.8ms/step | 11.6 samples/s
+Step 20400/50462 | Loss: 2.5941 | LR: 3.31e-04 | 5162.3ms/step | 11.6 samples/s
+Step 20410/50462 | Loss: 2.5883 | LR: 3.31e-04 | 5160.4ms/step | 11.6 samples/s
+Step 20420/50462 | Loss: 2.6071 | LR: 3.30e-04 | 5153.6ms/step | 11.6 samples/s
+Step 20430/50462 | Loss: 2.5862 | LR: 3.30e-04 | 5153.4ms/step | 11.6 samples/s
+Step 20440/50462 | Loss: 2.5980 | LR: 3.30e-04 | 5150.2ms/step | 11.7 samples/s
+Step 20450/50462 | Loss: 2.6060 | LR: 3.30e-04 | 5161.7ms/step | 11.6 samples/s
+Step 20460/50462 | Loss: 2.6036 | LR: 3.30e-04 | 5130.4ms/step | 11.7 samples/s
+Step 20470/50462 | Loss: 2.6243 | LR: 3.30e-04 | 5148.4ms/step | 11.7 samples/s
+Step 20480/50462 | Loss: 2.6192 | LR: 3.30e-04 | 5166.0ms/step | 11.6 samples/s
+Step 20490/50462 | Loss: 2.6080 | LR: 3.29e-04 | 5152.2ms/step | 11.6 samples/s
+Step 20500/50462 | Loss: 2.5790 | LR: 3.29e-04 | 5147.2ms/step | 11.7 samples/s
+Validation at step 20500: Loss = 2.6101, Perplexity = 13.60
+Step 20510/50462 | Loss: 2.5963 | LR: 3.29e-04 | 7700.7ms/step | 7.8 samples/s
+Step 20520/50462 | Loss: 2.5979 | LR: 3.29e-04 | 5139.3ms/step | 11.7 samples/s
+Step 20530/50462 | Loss: 2.5676 | LR: 3.29e-04 | 5145.1ms/step | 11.7 samples/s
+Step 20540/50462 | Loss: 2.5967 | LR: 3.29e-04 | 5152.7ms/step | 11.6 samples/s
+Step 20550/50462 | Loss: 2.6158 | LR: 3.29e-04 | 5140.8ms/step | 11.7 samples/s
+Step 20560/50462 | Loss: 2.6333 | LR: 3.28e-04 | 5136.4ms/step | 11.7 samples/s
+Step 20570/50462 | Loss: 2.6250 | LR: 3.28e-04 | 5141.9ms/step | 11.7 samples/s
+Step 20580/50462 | Loss: 2.6069 | LR: 3.28e-04 | 5150.0ms/step | 11.7 samples/s
+Step 20590/50462 | Loss: 2.6276 | LR: 3.28e-04 | 5139.1ms/step | 11.7 samples/s
+Step 20600/50462 | Loss: 2.5773 | LR: 3.28e-04 | 5142.0ms/step | 11.7 samples/s
+Step 20610/50462 | Loss: 2.5799 | LR: 3.28e-04 | 5140.6ms/step | 11.7 samples/s
+Step 20620/50462 | Loss: 2.5811 | LR: 3.27e-04 | 5152.3ms/step | 11.6 samples/s
+Step 20630/50462 | Loss: 2.5963 | LR: 3.27e-04 | 5144.1ms/step | 11.7 samples/s
+Step 20640/50462 | Loss: 2.6018 | LR: 3.27e-04 | 5146.6ms/step | 11.7 samples/s
+Step 20650/50462 | Loss: 2.5779 | LR: 3.27e-04 | 5145.4ms/step | 11.7 samples/s
+Step 20660/50462 | Loss: 2.6170 | LR: 3.27e-04 | 5145.4ms/step | 11.7 samples/s
+Step 20670/50462 | Loss: 2.6323 | LR: 3.27e-04 | 5140.2ms/step | 11.7 samples/s
+Step 20680/50462 | Loss: 2.5779 | LR: 3.27e-04 | 5151.5ms/step | 11.6 samples/s
+Step 20690/50462 | Loss: 2.5939 | LR: 3.26e-04 | 5158.9ms/step | 11.6 samples/s
+Step 20700/50462 | Loss: 2.5615 | LR: 3.26e-04 | 5140.4ms/step | 11.7 samples/s
+Step 20710/50462 | Loss: 2.6082 | LR: 3.26e-04 | 5150.2ms/step | 11.7 samples/s
+Step 20720/50462 | Loss: 2.6141 | LR: 3.26e-04 | 5144.3ms/step | 11.7 samples/s
+Step 20730/50462 | Loss: 2.5709 | LR: 3.26e-04 | 5159.9ms/step | 11.6 samples/s
+Step 20740/50462 | Loss: 2.5833 | LR: 3.26e-04 | 5150.1ms/step | 11.7 samples/s
+Step 20750/50462 | Loss: 2.5972 | LR: 3.26e-04 | 5143.0ms/step | 11.7 samples/s
+Validation at step 20750: Loss = 2.6072, Perplexity = 13.56
+Step 20760/50462 | Loss: 2.6055 | LR: 3.25e-04 | 7710.5ms/step | 7.8 samples/s
+Step 20770/50462 | Loss: 2.6080 | LR: 3.25e-04 | 5153.9ms/step | 11.6 samples/s
+Step 20780/50462 | Loss: 2.6180 | LR: 3.25e-04 | 5151.8ms/step | 11.6 samples/s
+Step 20790/50462 | Loss: 2.5673 | LR: 3.25e-04 | 5174.2ms/step | 11.6 samples/s
+Step 20800/50462 | Loss: 2.6085 | LR: 3.25e-04 | 5156.8ms/step | 11.6 samples/s
+Step 20810/50462 | Loss: 2.6047 | LR: 3.25e-04 | 5177.7ms/step | 11.6 samples/s
+Step 20820/50462 | Loss: 2.6128 | LR: 3.24e-04 | 5158.2ms/step | 11.6 samples/s
+Step 20830/50462 | Loss: 2.5865 | LR: 3.24e-04 | 5179.9ms/step | 11.6 samples/s
+Step 20840/50462 | Loss: 2.6151 | LR: 3.24e-04 | 5149.8ms/step | 11.7 samples/s
+Step 20850/50462 | Loss: 2.6066 | LR: 3.24e-04 | 5144.4ms/step | 11.7 samples/s
+Step 20860/50462 | Loss: 2.5690 | LR: 3.24e-04 | 5157.5ms/step | 11.6 samples/s
+Step 20870/50462 | Loss: 2.5774 | LR: 3.24e-04 | 5139.0ms/step | 11.7 samples/s
+Step 20880/50462 | Loss: 2.5994 | LR: 3.24e-04 | 5143.5ms/step | 11.7 samples/s
+Step 20890/50462 | Loss: 2.6089 | LR: 3.23e-04 | 5137.2ms/step | 11.7 samples/s
+Step 20900/50462 | Loss: 2.5784 | LR: 3.23e-04 | 5141.1ms/step | 11.7 samples/s
+Step 20910/50462 | Loss: 2.5851 | LR: 3.23e-04 | 5161.5ms/step | 11.6 samples/s
+Step 20920/50462 | Loss: 2.6067 | LR: 3.23e-04 | 5142.7ms/step | 11.7 samples/s
+Step 20930/50462 | Loss: 2.6243 | LR: 3.23e-04 | 5138.1ms/step | 11.7 samples/s
+Step 20940/50462 | Loss: 2.5966 | LR: 3.23e-04 | 5149.0ms/step | 11.7 samples/s
+Step 20950/50462 | Loss: 2.6142 | LR: 3.23e-04 | 5146.5ms/step | 11.7 samples/s
+Step 20960/50462 | Loss: 2.6096 | LR: 3.22e-04 | 5134.0ms/step | 11.7 samples/s
+Step 20970/50462 | Loss: 2.5724 | LR: 3.22e-04 | 5152.2ms/step | 11.6 samples/s
+Step 20980/50462 | Loss: 2.5849 | LR: 3.22e-04 | 5149.2ms/step | 11.7 samples/s
+Step 20990/50462 | Loss: 2.5972 | LR: 3.22e-04 | 5144.2ms/step | 11.7 samples/s
+Step 21000/50462 | Loss: 2.6129 | LR: 3.22e-04 | 5152.4ms/step | 11.6 samples/s
+Validation at step 21000: Loss = 2.6046, Perplexity = 13.53
+Saved checkpoint to checkpoints/run_20260605_220251/checkpoint-21000
+Step 21010/50462 | Loss: 2.6119 | LR: 3.22e-04 | 8687.1ms/step | 6.9 samples/s
+Step 21020/50462 | Loss: 2.6194 | LR: 3.21e-04 | 5187.9ms/step | 11.6 samples/s
+Step 21030/50462 | Loss: 2.5687 | LR: 3.21e-04 | 5168.0ms/step | 11.6 samples/s
+Step 21040/50462 | Loss: 2.5918 | LR: 3.21e-04 | 5147.8ms/step | 11.7 samples/s
+Step 21050/50462 | Loss: 2.5758 | LR: 3.21e-04 | 5174.7ms/step | 11.6 samples/s
+Step 21060/50462 | Loss: 2.5845 | LR: 3.21e-04 | 5188.9ms/step | 11.6 samples/s
+Step 21070/50462 | Loss: 2.5891 | LR: 3.21e-04 | 5187.2ms/step | 11.6 samples/s
+Step 21080/50462 | Loss: 2.5924 | LR: 3.21e-04 | 5180.8ms/step | 11.6 samples/s
+Step 21090/50462 | Loss: 2.6131 | LR: 3.20e-04 | 5189.9ms/step | 11.6 samples/s
+Step 21100/50462 | Loss: 2.6284 | LR: 3.20e-04 | 5171.3ms/step | 11.6 samples/s
+Step 21110/50462 | Loss: 2.6007 | LR: 3.20e-04 | 5139.3ms/step | 11.7 samples/s
+Step 21120/50462 | Loss: 2.5721 | LR: 3.20e-04 | 5161.8ms/step | 11.6 samples/s
+Step 21130/50462 | Loss: 2.6008 | LR: 3.20e-04 | 5196.5ms/step | 11.5 samples/s
+Step 21140/50462 | Loss: 2.6013 | LR: 3.20e-04 | 5181.1ms/step | 11.6 samples/s
+Step 21150/50462 | Loss: 2.5729 | LR: 3.19e-04 | 5185.6ms/step | 11.6 samples/s
+Step 21160/50462 | Loss: 2.5631 | LR: 3.19e-04 | 5180.4ms/step | 11.6 samples/s
+Step 21170/50462 | Loss: 2.5969 | LR: 3.19e-04 | 5177.8ms/step | 11.6 samples/s
+Step 21180/50462 | Loss: 2.6121 | LR: 3.19e-04 | 5183.9ms/step | 11.6 samples/s
+Step 21190/50462 | Loss: 2.6208 | LR: 3.19e-04 | 5179.0ms/step | 11.6 samples/s
+Step 21200/50462 | Loss: 2.6052 | LR: 3.19e-04 | 5197.4ms/step | 11.5 samples/s
+Step 21210/50462 | Loss: 2.6154 | LR: 3.19e-04 | 5191.4ms/step | 11.6 samples/s
+Step 21220/50462 | Loss: 2.6116 | LR: 3.18e-04 | 5203.7ms/step | 11.5 samples/s
+Step 21230/50462 | Loss: 2.5664 | LR: 3.18e-04 | 5198.6ms/step | 11.5 samples/s
+Step 21240/50462 | Loss: 2.6105 | LR: 3.18e-04 | 5206.4ms/step | 11.5 samples/s
+Step 21250/50462 | Loss: 2.5879 | LR: 3.18e-04 | 5199.5ms/step | 11.5 samples/s
+Validation at step 21250: Loss = 2.6017, Perplexity = 13.49
+Step 21260/50462 | Loss: 2.5754 | LR: 3.18e-04 | 7778.9ms/step | 7.7 samples/s
+Step 21270/50462 | Loss: 2.5796 | LR: 3.18e-04 | 5206.1ms/step | 11.5 samples/s
+Step 21280/50462 | Loss: 2.6217 | LR: 3.17e-04 | 5208.1ms/step | 11.5 samples/s
+Step 21290/50462 | Loss: 2.6057 | LR: 3.17e-04 | 5201.4ms/step | 11.5 samples/s
+Step 21300/50462 | Loss: 2.5856 | LR: 3.17e-04 | 5584.1ms/step | 10.7 samples/s
+Step 21310/50462 | Loss: 2.5981 | LR: 3.17e-04 | 5177.6ms/step | 11.6 samples/s
+Step 21320/50462 | Loss: 2.5809 | LR: 3.17e-04 | 5198.3ms/step | 11.5 samples/s
+Step 21330/50462 | Loss: 2.5928 | LR: 3.17e-04 | 5200.7ms/step | 11.5 samples/s
+Step 21340/50462 | Loss: 2.5953 | LR: 3.17e-04 | 5191.7ms/step | 11.6 samples/s
+Step 21350/50462 | Loss: 2.5663 | LR: 3.16e-04 | 5193.7ms/step | 11.6 samples/s

epfl_final/wandb/run-20260605_220539-ns4ua5fr/files/requirements.txt

@@ -0,0 +1,122 @@
+word2number==1.1
+sqlitedict==2.1.0
+pytz==2026.2
+nvidia-cusparselt-cu13==0.8.1
+mpmath==1.3.0
+cuda-toolkit==13.0.2
+zstandard==0.25.0
+xxhash==3.7.0
+urllib3==2.7.0
+tzdata==2026.2
+typing_extensions==4.15.0
+triton==3.7.0
+tqdm==4.68.1
+torchinfo==1.8.0
+threadpoolctl==3.6.0
+tcolorpy==0.1.7
+tabulate==0.10.0
+sympy==1.14.0
+smmap==5.0.3
+six==1.17.0
+shellingham==1.5.4
+safetensors==0.7.0
+regex==2026.5.9
+PyYAML==6.0.3
+Pygments==2.20.0
+pyarrow==24.0.0
+psutil==7.2.2
+protobuf==7.35.0
+propcache==0.5.2
+portalocker==3.2.0
+platformdirs==4.10.0
+pathvalidate==3.3.1
+packaging==26.2
+nvidia-nvtx==13.0.85
+nvidia-nvshmem-cu13==3.4.5
+nvidia-nvjitlink==13.0.88
+nvidia-nccl-cu13==2.29.7
+nvidia-curand==10.4.0.35
+nvidia-cufile==1.15.1.6
+nvidia-cuda-runtime==13.0.96
+nvidia-cuda-nvrtc==13.0.88
+nvidia-cuda-cupti==13.0.85
+numpy==2.2.6
+networkx==3.4.2
+more-itertools==11.1.0
+mdurl==0.1.2
+MarkupSafe==3.0.3
+lxml==6.1.1
+joblib==1.5.3
+idna==3.18
+hf-xet==1.5.0
+h11==0.16.0
+fsspec==2026.4.0
+frozenlist==1.8.0
+filelock==3.29.1
+dill==0.4.1
+cuda-pathfinder==1.5.5
+colorama==0.4.6
+click==8.4.1
+charset-normalizer==3.4.7
+chardet==6.0.0.post1
+certifi==2026.5.20
+attrs==26.1.0
+async-timeout==5.0.1
+annotated-types==0.7.0
+annotated-doc==0.0.4
+aiohappyeyeballs==2.6.2
+absl-py==2.4.0
+typing-inspection==0.4.2
+sentry-sdk==2.61.1
+scipy==1.15.3
+sacrebleu==2.6.0
+requests==2.34.2
+python-dateutil==2.9.0.post0
+pydantic_core==2.46.4
+nvidia-cusparse==12.6.3.3
+nvidia-cufft==12.0.0.61
+nvidia-cublas==13.1.1.3
+nltk==3.9.4
+multiprocess==0.70.19
+multidict==6.7.1
+mbstrdecoder==1.1.5
+markdown-it-py==4.2.0
+Jinja2==3.1.6
+httpcore==1.0.9
+gitdb==4.0.12
+exceptiongroup==1.3.1
+cuda-bindings==13.3.1
+aiosignal==1.4.0
+yarl==1.24.2
+typepy==1.3.5
+scikit-learn==1.7.2
+rouge-score==0.1.2
+rich==15.0.0
+pydantic==2.13.4
+pandas==2.3.3
+nvidia-cusolver==12.0.4.66
+nvidia-cudnn-cu13==9.20.0.48
+GitPython==3.1.50
+anyio==4.13.0
+wandb==0.27.1
+typer==0.25.1
+httpx==0.28.1
+aiohttp==3.14.0
+torch==2.12.0
+huggingface_hub==1.18.0
+DataProperty==1.1.1
+tokenizers==0.22.2
+tabledata==1.3.5
+datasets==5.0.0
+accelerate==1.13.0
+transformers==5.10.2
+pytablewriter==1.2.1
+evaluate==0.4.6
+lm_eval==0.4.12
+PyGObject==3.42.1
+dbus-python==1.2.18
+distro==1.7.0
+pip==22.0.2
+setuptools==59.6.0
+ssh-import-id==5.11
+wheel==0.37.1

epfl_final/wandb/run-20260605_220539-ns4ua5fr/files/wandb-metadata.json

@@ -0,0 +1,54 @@
+{
+  "os": "Linux-6.8.0-124-generic-x86_64-with-glibc2.35",
+  "python": "CPython 3.10.12",
+  "startedAt": "2026-06-05T22:05:39.467239Z",
+  "args": [
+    "--config",
+    "SparseAtt/configs/train_config_dense_resume.json"
+  ],
+  "program": "/mloscratch/homes/navasard/sparse/SparseAtt/train.py",
+  "codePath": "SparseAtt/train.py",
+  "codePathLocal": "SparseAtt/train.py",
+  "email": "ruben.navasardyan@mail.utoronto.ca",
+  "root": "/mloscratch/homes/navasard/sparse",
+  "host": "h100-train-3-gpu-0-0",
+  "executable": "/usr/bin/python3",
+  "cpu_count": 96,
+  "cpu_count_logical": 192,
+  "gpu": "NVIDIA H100 80GB HBM3",
+  "gpu_count": 3,
+  "disk": {
+    "/": {
+      "total": "9374931091456",
+      "used": "6125122981888"
+    }
+  },
+  "memory": {
+    "total": "1623005732864"
+  },
+  "gpu_nvidia": [
+    {
+      "name": "NVIDIA H100 80GB HBM3",
+      "memoryTotal": "85520809984",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-e3358200-f776-7214-8269-c2c219f03c2e"
+    },
+    {
+      "name": "NVIDIA H100 80GB HBM3",
+      "memoryTotal": "85520809984",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-123c77ea-9377-5bde-70b6-99ef1c791ff4"
+    },
+    {
+      "name": "NVIDIA H100 80GB HBM3",
+      "memoryTotal": "85520809984",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-7a2d68b3-dda3-ec4d-f88e-205cb866ebb6"
+    }
+  ],
+  "cudaVersion": "13.0",
+  "writerId": "79bfmd6qpbnas6d81s3yv9kfg01oahes"
+}

epfl_final/wandb/run-20260605_220539-ns4ua5fr/logs/debug.log

@@ -0,0 +1,19 @@
+2026-06-05 22:05:39,485 INFO    MainThread:12012 [wandb_setup.py:_flush():81] Current SDK version is 0.27.1
+2026-06-05 22:05:39,485 INFO    MainThread:12012 [wandb_setup.py:_flush():81] Configure stats pid to 12012
+2026-06-05 22:05:39,485 INFO    MainThread:12012 [wandb_setup.py:_flush():81] Loading settings from environment variables
+2026-06-05 22:05:39,486 INFO    MainThread:12012 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /mloscratch/homes/navasard/sparse/wandb/run-20260605_220539-ns4ua5fr/logs/debug.log
+2026-06-05 22:05:39,487 INFO    MainThread:12012 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /mloscratch/homes/navasard/sparse/wandb/run-20260605_220539-ns4ua5fr/logs/debug-internal.log
+2026-06-05 22:05:39,487 INFO    MainThread:12012 [wandb_init.py:init():766] calling init triggers
+2026-06-05 22:05:39,488 INFO    MainThread:12012 [wandb_init.py:init():771] wandb.init called with sweep_config: {}
+config: {'model_path': '/mloscratch/homes/navasard/qwen3_06b_base', 'model_dtype': 'bfloat16', 'init_from_scratch': True, 'train_path': '/mloscratch/homes/navasard/data/fineweb_large/tokenized', 'val_path': None, 'max_seq_length': 8192, 'num_workers': 8, 'output_dir': './checkpoints/run_20260605_220251', 'resume_from_checkpoint': '/mloscratch/homes/navasard/sparse/checkpoints/run_20260527_053212/checkpoint-10000', 'force_new_run': True, 'num_epochs': 1, 'max_steps': None, 'per_device_batch_size': 2, 'gradient_accumulation_steps': 10, 'learning_rate': 0.0005, 'weight_decay': 0.01, 'warmup_steps': 750, 'lr_scheduler': 'cosine', 'max_grad_norm': 1.0, 'save_steps': 1000, 'eval_steps': 250, 'eval_batches': 50, 'logging_steps': 10, 'seed': 42, 'verbose': True, 'checkpoint_activations': False, 'strategy': 'ddp', 'mixed_precision': 'bf16', 'wandb_enabled': True, 'wandb_project': 'stochastic_sparsity', 'wandb_name': 'dense continued 10k', 'wandb_tags': ['qwen3', '1.2b', 'dense-resume'], 'hidden_size': 1536, 'intermediate_size': 6144, 'num_attention_heads': 12, 'num_key_value_heads': 4, 'num_hidden_layers': None, 'head_dim': None, 'attention_type': 'sparse_stochastic', 'sparse_block_size': 128, 'sparse_num_random_blocks': 31, 'sparse_mask_pool_size': 10000, 'sparse_val_dense_compare': False, 'sparse_layer_indices': None, 'sparse_mask_cache_dir': '/mloscratch/homes/navasard/sparse/mask_cache', 'dense_mode': True, 'dense_step_intervals': None, '_wandb': {}}
+2026-06-05 22:05:39,489 INFO    MainThread:12012 [wandb_init.py:init():814] starting backend
+2026-06-05 22:05:39,489 INFO    MainThread:12012 [wandb_init.py:init():829] sending inform_init request
+2026-06-05 22:05:39,794 INFO    MainThread:12012 [wandb_init.py:init():834] backend started and connected
+2026-06-05 22:05:39,797 INFO    MainThread:12012 [wandb_init.py:init():904] updated telemetry
+2026-06-05 22:05:39,798 INFO    MainThread:12012 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout
+2026-06-05 22:05:40,188 INFO    MainThread:12012 [wandb_init.py:init():972] starting run threads in backend
+2026-06-05 22:05:40,257 INFO    MainThread:12012 [wandb_run.py:_console_start():2509] atexit reg
+2026-06-05 22:05:40,257 INFO    MainThread:12012 [wandb_run.py:_redirect():2359] redirect: wrap_raw
+2026-06-05 22:05:40,257 INFO    MainThread:12012 [wandb_run.py:_redirect():2428] Wrapping output streams.
+2026-06-05 22:05:40,258 INFO    MainThread:12012 [wandb_run.py:_redirect():2451] Redirects installed.
+2026-06-05 22:05:40,260 INFO    MainThread:12012 [wandb_init.py:init():1010] run started, returning control to user process

epfl_final/wandb/run-20260605_220539-ns4ua5fr/run-ns4ua5fr.wandb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cf549030cd42f8a6f661d01a0d6df7dadc960e9a3824e6789e45ae9dfdc81b22
+size 16580608

epfl_final/wandb/run-20260606_174407-ns4ua5fr/files/config.yaml

@@ -0,0 +1,176 @@
+_wandb:
+    value:
+        cli_version: 0.27.2
+        e:
+            961q6ksgorhc60wb5b1o0hgnux1998ag:
+                args:
+                    - --config
+                    - SparseAtt/configs/train_config_dense_resume.json
+                codePath: SparseAtt/train.py
+                codePathLocal: SparseAtt/train.py
+                cpu_count: 96
+                cpu_count_logical: 192
+                cudaVersion: "13.0"
+                disk:
+                    /:
+                        total: "9374931091456"
+                        used: "5545268330496"
+                email: ruben.navasardyan@mail.utoronto.ca
+                executable: /usr/bin/python3
+                gpu: NVIDIA H100 80GB HBM3
+                gpu_count: 3
+                gpu_nvidia:
+                    - architecture: Hopper
+                      cudaCores: 16896
+                      memoryTotal: "85520809984"
+                      name: NVIDIA H100 80GB HBM3
+                      uuid: GPU-b817d241-acce-e5dd-a749-3447239c5207
+                    - architecture: Hopper
+                      cudaCores: 16896
+                      memoryTotal: "85520809984"
+                      name: NVIDIA H100 80GB HBM3
+                      uuid: GPU-165543e5-e049-5219-e717-e1216ac960de
+                    - architecture: Hopper
+                      cudaCores: 16896
+                      memoryTotal: "85520809984"
+                      name: NVIDIA H100 80GB HBM3
+                      uuid: GPU-47444fc7-ea8c-cef6-aac8-ef408dc8047b
+                host: h100-train-3-gpu-0-2
+                memory:
+                    total: "1623005732864"
+                os: Linux-6.8.0-124-generic-x86_64-with-glibc2.35
+                program: /mloscratch/homes/navasard/sparse/SparseAtt/train.py
+                python: CPython 3.10.12
+                root: /mloscratch/homes/navasard/sparse
+                startedAt: "2026-06-06T17:44:07.949203Z"
+                writerId: 961q6ksgorhc60wb5b1o0hgnux1998ag
+        m: []
+        python_version: 3.10.12
+        t:
+            "1":
+                - 1
+                - 5
+                - 11
+                - 49
+                - 51
+                - 53
+                - 71
+            "2":
+                - 1
+                - 5
+                - 11
+                - 49
+                - 51
+                - 53
+                - 71
+            "3":
+                - 2
+                - 5
+                - 13
+                - 14
+                - 15
+                - 16
+                - 61
+                - 62
+            "4": 3.10.12
+            "5": 0.27.2
+            "6": 5.10.2
+            "12": 0.27.2
+            "13": linux-x86_64
+attention_type:
+    value: sparse_stochastic
+checkpoint_activations:
+    value: false
+dense_mode:
+    value: true
+dense_step_intervals:
+    value: null
+eval_batches:
+    value: 50
+eval_steps:
+    value: 250
+force_new_run:
+    value: false
+gradient_accumulation_steps:
+    value: 10
+head_dim:
+    value: null
+hidden_size:
+    value: 1536
+init_from_scratch:
+    value: true
+intermediate_size:
+    value: 6144
+learning_rate:
+    value: 0.0005
+logging_steps:
+    value: 10
+lr_scheduler:
+    value: cosine
+max_grad_norm:
+    value: 1
+max_seq_length:
+    value: 8192
+max_steps:
+    value: null
+mixed_precision:
+    value: bf16
+model_dtype:
+    value: bfloat16
+model_path:
+    value: /mloscratch/homes/navasard/qwen3_06b_base
+num_attention_heads:
+    value: 12
+num_epochs:
+    value: 1
+num_hidden_layers:
+    value: null
+num_key_value_heads:
+    value: 4
+num_workers:
+    value: 8
+output_dir:
+    value: /mloscratch/homes/navasard/sparse/checkpoints/run_20260605_220251
+per_device_batch_size:
+    value: 2
+resume_from_checkpoint:
+    value: /mloscratch/homes/navasard/sparse/checkpoints/run_20260605_220251/checkpoint-21000
+save_steps:
+    value: 1000
+seed:
+    value: 42
+sparse_block_size:
+    value: 128
+sparse_layer_indices:
+    value: null
+sparse_mask_cache_dir:
+    value: /mloscratch/homes/navasard/sparse/mask_cache
+sparse_mask_pool_size:
+    value: 10000
+sparse_num_random_blocks:
+    value: 31
+sparse_val_dense_compare:
+    value: false
+strategy:
+    value: ddp
+train_path:
+    value: /mloscratch/homes/navasard/data/fineweb_large/tokenized
+val_path:
+    value: null
+verbose:
+    value: true
+wandb_enabled:
+    value: true
+wandb_name:
+    value: dense continued 10k
+wandb_project:
+    value: stochastic_sparsity
+wandb_tags:
+    value:
+        - qwen3
+        - 1.2b
+        - dense-resume
+warmup_steps:
+    value: 750
+weight_decay:
+    value: 0.01

epfl_final/wandb/run-20260606_174407-ns4ua5fr/files/requirements.txt

@@ -0,0 +1,122 @@
+word2number==1.1
+sqlitedict==2.1.0
+pytz==2026.2
+nvidia-cusparselt-cu13==0.8.1
+mpmath==1.3.0
+cuda-toolkit==13.0.2
+zstandard==0.25.0
+xxhash==3.7.0
+urllib3==2.7.0
+tzdata==2026.2
+typing_extensions==4.15.0
+triton==3.7.0
+tqdm==4.68.1
+torchinfo==1.8.0
+threadpoolctl==3.6.0
+tcolorpy==0.1.7
+tabulate==0.10.0
+sympy==1.14.0
+smmap==5.0.3
+six==1.17.0
+shellingham==1.5.4
+safetensors==0.7.0
+regex==2026.5.9
+PyYAML==6.0.3
+Pygments==2.20.0
+pyarrow==24.0.0
+psutil==7.2.2
+protobuf==7.35.0
+propcache==0.5.2
+portalocker==3.2.0
+platformdirs==4.10.0
+pathvalidate==3.3.1
+packaging==26.2
+nvidia-nvtx==13.0.85
+nvidia-nvshmem-cu13==3.4.5
+nvidia-nvjitlink==13.0.88
+nvidia-nccl-cu13==2.29.7
+nvidia-curand==10.4.0.35
+nvidia-cufile==1.15.1.6
+nvidia-cuda-runtime==13.0.96
+nvidia-cuda-nvrtc==13.0.88
+nvidia-cuda-cupti==13.0.85
+numpy==2.2.6
+networkx==3.4.2
+more-itertools==11.1.0
+mdurl==0.1.2
+MarkupSafe==3.0.3
+lxml==6.1.1
+joblib==1.5.3
+idna==3.18
+hf-xet==1.5.0
+h11==0.16.0
+fsspec==2026.4.0
+frozenlist==1.8.0
+filelock==3.29.1
+dill==0.4.1
+cuda-pathfinder==1.5.5
+colorama==0.4.6
+click==8.4.1
+charset-normalizer==3.4.7
+chardet==6.0.0.post1
+certifi==2026.5.20
+attrs==26.1.0
+async-timeout==5.0.1
+annotated-types==0.7.0
+annotated-doc==0.0.4
+aiohappyeyeballs==2.6.2
+absl-py==2.4.0
+typing-inspection==0.4.2
+sentry-sdk==2.61.1
+scipy==1.15.3
+sacrebleu==2.6.0
+requests==2.34.2
+python-dateutil==2.9.0.post0
+pydantic_core==2.46.4
+nvidia-cusparse==12.6.3.3
+nvidia-cufft==12.0.0.61
+nvidia-cublas==13.1.1.3
+nltk==3.9.4
+multiprocess==0.70.19
+multidict==6.7.1
+mbstrdecoder==1.1.5
+markdown-it-py==4.2.0
+Jinja2==3.1.6
+httpcore==1.0.9
+gitdb==4.0.12
+exceptiongroup==1.3.1
+cuda-bindings==13.3.1
+aiosignal==1.4.0
+yarl==1.24.2
+typepy==1.3.5
+scikit-learn==1.7.2
+rouge-score==0.1.2
+rich==15.0.0
+pydantic==2.13.4
+pandas==2.3.3
+nvidia-cusolver==12.0.4.66
+nvidia-cudnn-cu13==9.20.0.48
+GitPython==3.1.50
+anyio==4.13.0
+wandb==0.27.2
+typer==0.25.1
+httpx==0.28.1
+aiohttp==3.14.0
+torch==2.12.0
+huggingface_hub==1.18.0
+DataProperty==1.1.1
+tokenizers==0.22.2
+tabledata==1.3.5
+datasets==5.0.0
+accelerate==1.13.0
+transformers==5.10.2
+pytablewriter==1.2.1
+evaluate==0.4.6
+lm_eval==0.4.12
+PyGObject==3.42.1
+dbus-python==1.2.18
+distro==1.7.0
+pip==22.0.2
+setuptools==59.6.0
+ssh-import-id==5.11
+wheel==0.37.1

epfl_final/wandb/run-20260606_174407-ns4ua5fr/files/wandb-metadata.json

@@ -0,0 +1,54 @@
+{
+  "os": "Linux-6.8.0-124-generic-x86_64-with-glibc2.35",
+  "python": "CPython 3.10.12",
+  "startedAt": "2026-06-06T17:44:07.949203Z",
+  "args": [
+    "--config",
+    "SparseAtt/configs/train_config_dense_resume.json"
+  ],
+  "program": "/mloscratch/homes/navasard/sparse/SparseAtt/train.py",
+  "codePath": "SparseAtt/train.py",
+  "codePathLocal": "SparseAtt/train.py",
+  "email": "ruben.navasardyan@mail.utoronto.ca",
+  "root": "/mloscratch/homes/navasard/sparse",
+  "host": "h100-train-3-gpu-0-2",
+  "executable": "/usr/bin/python3",
+  "cpu_count": 96,
+  "cpu_count_logical": 192,
+  "gpu": "NVIDIA H100 80GB HBM3",
+  "gpu_count": 3,
+  "disk": {
+    "/": {
+      "total": "9374931091456",
+      "used": "5545268330496"
+    }
+  },
+  "memory": {
+    "total": "1623005732864"
+  },
+  "gpu_nvidia": [
+    {
+      "name": "NVIDIA H100 80GB HBM3",
+      "memoryTotal": "85520809984",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-b817d241-acce-e5dd-a749-3447239c5207"
+    },
+    {
+      "name": "NVIDIA H100 80GB HBM3",
+      "memoryTotal": "85520809984",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-165543e5-e049-5219-e717-e1216ac960de"
+    },
+    {
+      "name": "NVIDIA H100 80GB HBM3",
+      "memoryTotal": "85520809984",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-47444fc7-ea8c-cef6-aac8-ef408dc8047b"
+    }
+  ],
+  "cudaVersion": "13.0",
+  "writerId": "961q6ksgorhc60wb5b1o0hgnux1998ag"
+}

epfl_final/wandb/run-20260606_174407-ns4ua5fr/files/wandb-summary.json

@@ -0,0 +1 @@
+{"eval/dense_interval_active":0,"train/epoch":0.9999603662161627,"speed/steps_per_sec":0.1763429715538705,"speed/samples_per_sec":10.580578293232229,"speed/ms_per_step":5670.767545700073,"train/loss":2.480617798566818,"train/global_step":50460,"train/learning_rate":1.9968584286100823e-12,"_runtime":226238.05472534,"eval/loss":2.5014872136944066,"_wandb":{"runtime":226238},"_timestamp":1.7809337520442352e+09,"eval/dense_interval_start":-1,"eval/uses_dense_attention":1,"eval/uses_sparse_attention":0,"_step":50460,"eval/dense_interval_end":-1,"eval/perplexity":12.20062541188317}

epfl_final/wandb/run-20260606_174407-ns4ua5fr/logs/debug.log

@@ -0,0 +1,25 @@
+2026-06-06 17:44:07,969 INFO    MainThread:2954 [wandb_setup.py:_flush():81] Current SDK version is 0.27.2
+2026-06-06 17:44:07,969 INFO    MainThread:2954 [wandb_setup.py:_flush():81] Configure stats pid to 2954
+2026-06-06 17:44:07,970 INFO    MainThread:2954 [wandb_setup.py:_flush():81] Loading settings from environment variables
+2026-06-06 17:44:07,971 INFO    MainThread:2954 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /mloscratch/homes/navasard/sparse/wandb/run-20260606_174407-ns4ua5fr/logs/debug.log
+2026-06-06 17:44:07,971 INFO    MainThread:2954 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /mloscratch/homes/navasard/sparse/wandb/run-20260606_174407-ns4ua5fr/logs/debug-internal.log
+2026-06-06 17:44:07,972 INFO    MainThread:2954 [wandb_init.py:init():766] calling init triggers
+2026-06-06 17:44:07,972 INFO    MainThread:2954 [wandb_init.py:init():771] wandb.init called with sweep_config: {}
+config: {'model_path': '/mloscratch/homes/navasard/qwen3_06b_base', 'model_dtype': 'bfloat16', 'init_from_scratch': True, 'train_path': '/mloscratch/homes/navasard/data/fineweb_large/tokenized', 'val_path': None, 'max_seq_length': 8192, 'num_workers': 8, 'output_dir': '/mloscratch/homes/navasard/sparse/checkpoints/run_20260605_220251', 'resume_from_checkpoint': '/mloscratch/homes/navasard/sparse/checkpoints/run_20260605_220251/checkpoint-21000', 'force_new_run': False, 'num_epochs': 1, 'max_steps': None, 'per_device_batch_size': 2, 'gradient_accumulation_steps': 10, 'learning_rate': 0.0005, 'weight_decay': 0.01, 'warmup_steps': 750, 'lr_scheduler': 'cosine', 'max_grad_norm': 1.0, 'save_steps': 1000, 'eval_steps': 250, 'eval_batches': 50, 'logging_steps': 10, 'seed': 42, 'verbose': True, 'checkpoint_activations': False, 'strategy': 'ddp', 'mixed_precision': 'bf16', 'wandb_enabled': True, 'wandb_project': 'stochastic_sparsity', 'wandb_name': 'dense continued 10k', 'wandb_tags': ['qwen3', '1.2b', 'dense-resume'], 'hidden_size': 1536, 'intermediate_size': 6144, 'num_attention_heads': 12, 'num_key_value_heads': 4, 'num_hidden_layers': None, 'head_dim': None, 'attention_type': 'sparse_stochastic', 'sparse_block_size': 128, 'sparse_num_random_blocks': 31, 'sparse_mask_pool_size': 10000, 'sparse_val_dense_compare': False, 'sparse_layer_indices': None, 'sparse_mask_cache_dir': '/mloscratch/homes/navasard/sparse/mask_cache', 'dense_mode': True, 'dense_step_intervals': None, '_wandb': {}}
+2026-06-06 17:44:07,973 INFO    MainThread:2954 [wandb_init.py:init():814] starting backend
+2026-06-06 17:44:07,974 INFO    MainThread:2954 [wandb_init.py:init():829] sending inform_init request
+2026-06-06 17:44:08,303 INFO    MainThread:2954 [wandb_init.py:init():834] backend started and connected
+2026-06-06 17:44:08,306 INFO    MainThread:2954 [wandb_init.py:init():904] updated telemetry
+2026-06-06 17:44:08,308 INFO    MainThread:2954 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout
+2026-06-06 17:44:08,737 INFO    MainThread:2954 [wandb_init.py:init():967] run resumed
+2026-06-06 17:44:08,740 INFO    MainThread:2954 [wandb_init.py:init():972] starting run threads in backend
+2026-06-06 17:44:08,830 INFO    MainThread:2954 [wandb_run.py:_console_start():2523] atexit reg
+2026-06-06 17:44:08,831 INFO    MainThread:2954 [wandb_run.py:_redirect():2373] redirect: wrap_raw
+2026-06-06 17:44:08,831 INFO    MainThread:2954 [wandb_run.py:_redirect():2442] Wrapping output streams.
+2026-06-06 17:44:08,832 INFO    MainThread:2954 [wandb_run.py:_redirect():2465] Redirects installed.
+2026-06-06 17:44:08,833 INFO    MainThread:2954 [wandb_init.py:init():1010] run started, returning control to user process
+2026-06-08 15:49:36,791 INFO    MainThread:2954 [wandb_run.py:_finish():2285] finishing run ruben-navasardyan4869-epfl/stochastic_sparsity/ns4ua5fr
+2026-06-08 15:49:36,791 INFO    MainThread:2954 [wandb_run.py:_atexit_cleanup():2490] got exitcode: 0
+2026-06-08 15:49:36,793 INFO    MainThread:2954 [wandb_run.py:_restore():2472] restore
+2026-06-08 15:49:36,793 INFO    MainThread:2954 [wandb_run.py:_restore():2478] restore done
+2026-06-08 15:49:38,224 INFO    MainThread:2954 [wandb_run.py:_footer_sync_info():3891] logging synced files

epfl_final/wandb/run-20260606_174407-ns4ua5fr/run-ns4ua5fr.wandb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5e6e4651f1a27626d1e359e5563b5f327055bd7850d07b78c40bdc7077602bbd
+size 45661054

epfl_final/wandb/run-20260609_064106-jh4dfz9a/files/requirements.txt

@@ -0,0 +1,122 @@
+word2number==1.1
+sqlitedict==2.1.0
+pytz==2026.2
+nvidia-cusparselt-cu13==0.8.1
+mpmath==1.3.0
+cuda-toolkit==13.0.2
+zstandard==0.25.0
+xxhash==3.7.0
+urllib3==2.7.0
+tzdata==2026.2
+typing_extensions==4.15.0
+triton==3.7.0
+tqdm==4.68.1
+torchinfo==1.8.0
+threadpoolctl==3.6.0
+tcolorpy==0.1.7
+tabulate==0.10.0
+sympy==1.14.0
+smmap==5.0.3
+six==1.17.0
+shellingham==1.5.4
+safetensors==0.7.0
+regex==2026.5.9
+PyYAML==6.0.3
+Pygments==2.20.0
+pyarrow==24.0.0
+psutil==7.2.2
+protobuf==7.35.0
+propcache==0.5.2
+portalocker==3.2.0
+platformdirs==4.10.0
+pathvalidate==3.3.1
+packaging==26.2
+nvidia-nvtx==13.0.85
+nvidia-nvshmem-cu13==3.4.5
+nvidia-nvjitlink==13.0.88
+nvidia-nccl-cu13==2.29.7
+nvidia-curand==10.4.0.35
+nvidia-cufile==1.15.1.6
+nvidia-cuda-runtime==13.0.96
+nvidia-cuda-nvrtc==13.0.88
+nvidia-cuda-cupti==13.0.85
+numpy==2.2.6
+networkx==3.4.2
+more-itertools==11.1.0
+mdurl==0.1.2
+MarkupSafe==3.0.3
+lxml==6.1.1
+joblib==1.5.3
+idna==3.18
+hf-xet==1.5.1
+h11==0.16.0
+fsspec==2026.4.0
+frozenlist==1.8.0
+filelock==3.29.1
+dill==0.4.1
+cuda-pathfinder==1.5.5
+colorama==0.4.6
+click==8.4.1
+charset-normalizer==3.4.7
+chardet==6.0.0.post1
+certifi==2026.5.20
+attrs==26.1.0
+async-timeout==5.0.1
+annotated-types==0.7.0
+annotated-doc==0.0.4
+aiohappyeyeballs==2.6.2
+absl-py==2.4.0
+typing-inspection==0.4.2
+sentry-sdk==2.62.0
+scipy==1.15.3
+sacrebleu==2.6.0
+requests==2.34.2
+python-dateutil==2.9.0.post0
+pydantic_core==2.46.4
+nvidia-cusparse==12.6.3.3
+nvidia-cufft==12.0.0.61
+nvidia-cublas==13.1.1.3
+nltk==3.9.4
+multiprocess==0.70.19
+multidict==6.7.1
+mbstrdecoder==1.1.5
+markdown-it-py==4.2.0
+Jinja2==3.1.6
+httpcore==1.0.9
+gitdb==4.0.12
+exceptiongroup==1.3.1
+cuda-bindings==13.3.1
+aiosignal==1.4.0
+yarl==1.24.2
+typepy==1.3.5
+scikit-learn==1.7.2
+rouge-score==0.1.2
+rich==15.0.0
+pydantic==2.13.4
+pandas==2.3.3
+nvidia-cusolver==12.0.4.66
+nvidia-cudnn-cu13==9.20.0.48
+GitPython==3.1.50
+anyio==4.13.0
+wandb==0.27.2
+typer==0.25.1
+httpx==0.28.1
+aiohttp==3.14.1
+torch==2.12.0
+huggingface_hub==1.18.0
+DataProperty==1.1.1
+tokenizers==0.22.2
+tabledata==1.3.5
+datasets==5.0.0
+accelerate==1.13.0
+transformers==5.10.2
+pytablewriter==1.2.1
+evaluate==0.4.6
+lm_eval==0.4.12
+PyGObject==3.42.1
+dbus-python==1.2.18
+distro==1.7.0
+pip==22.0.2
+setuptools==59.6.0
+ssh-import-id==5.11
+wheel==0.37.1

epfl_final/wandb/run-20260609_064106-jh4dfz9a/files/wandb-metadata.json

@@ -0,0 +1,54 @@
+{
+  "os": "Linux-6.8.0-124-generic-x86_64-with-glibc2.35",
+  "python": "CPython 3.10.12",
+  "startedAt": "2026-06-09T06:41:06.403353Z",
+  "args": [
+    "--config",
+    "SparseAtt/configs/train_config.json"
+  ],
+  "program": "/mloscratch/homes/navasard/sparse/SparseAtt/train.py",
+  "codePath": "SparseAtt/train.py",
+  "codePathLocal": "SparseAtt/train.py",
+  "email": "ruben.navasardyan@mail.utoronto.ca",
+  "root": "/mloscratch/homes/navasard/sparse",
+  "host": "h100-train-3-gpu-0-3",
+  "executable": "/usr/bin/python3",
+  "cpu_count": 96,
+  "cpu_count_logical": 192,
+  "gpu": "NVIDIA H100 80GB HBM3",
+  "gpu_count": 3,
+  "disk": {
+    "/": {
+      "total": "9374931091456",
+      "used": "4936847581184"
+    }
+  },
+  "memory": {
+    "total": "1623005732864"
+  },
+  "gpu_nvidia": [
+    {
+      "name": "NVIDIA H100 80GB HBM3",
+      "memoryTotal": "85520809984",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-f65cda91-ff87-9bff-fb3d-3e1b3f9f8cf0"
+    },
+    {
+      "name": "NVIDIA H100 80GB HBM3",
+      "memoryTotal": "85520809984",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-086f9f0b-a6d8-6263-83e6-c2b66721f47f"
+    },
+    {
+      "name": "NVIDIA H100 80GB HBM3",
+      "memoryTotal": "85520809984",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-bdb5e23e-0700-20a0-0b21-1ad0901f08b7"
+    }
+  ],
+  "cudaVersion": "13.0",
+  "writerId": "zpc9ftoqisn0z5ea81wenrnooio8zy8m"
+}

epfl_final/wandb/run-20260609_064106-jh4dfz9a/logs/debug.log

@@ -0,0 +1,19 @@
+2026-06-09 06:41:06,431 INFO    MainThread:3220 [wandb_setup.py:_flush():81] Current SDK version is 0.27.2
+2026-06-09 06:41:06,431 INFO    MainThread:3220 [wandb_setup.py:_flush():81] Configure stats pid to 3220
+2026-06-09 06:41:06,432 INFO    MainThread:3220 [wandb_setup.py:_flush():81] Loading settings from environment variables
+2026-06-09 06:41:06,433 INFO    MainThread:3220 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /mloscratch/homes/navasard/sparse/wandb/run-20260609_064106-jh4dfz9a/logs/debug.log
+2026-06-09 06:41:06,434 INFO    MainThread:3220 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /mloscratch/homes/navasard/sparse/wandb/run-20260609_064106-jh4dfz9a/logs/debug-internal.log
+2026-06-09 06:41:06,434 INFO    MainThread:3220 [wandb_init.py:init():766] calling init triggers
+2026-06-09 06:41:06,435 INFO    MainThread:3220 [wandb_init.py:init():771] wandb.init called with sweep_config: {}
+config: {'model_path': '/mloscratch/homes/navasard/qwen3_06b_base', 'model_dtype': 'bfloat16', 'init_from_scratch': True, 'train_path': '/mloscratch/homes/navasard/data/fineweb_large/tokenized', 'val_path': None, 'max_seq_length': 8192, 'num_workers': 8, 'output_dir': './checkpoints/run_20260609_063834', 'resume_from_checkpoint': None, 'force_new_run': True, 'num_epochs': 1, 'max_steps': None, 'per_device_batch_size': 2, 'gradient_accumulation_steps': 10, 'learning_rate': 0.0005, 'weight_decay': 0.01, 'warmup_steps': 750, 'lr_scheduler': 'cosine', 'max_grad_norm': 1.0, 'save_steps': 1000, 'eval_steps': 250, 'eval_batches': 50, 'logging_steps': 10, 'seed': 42, 'verbose': True, 'checkpoint_activations': False, 'strategy': 'ddp', 'mixed_precision': 'bf16', 'wandb_enabled': True, 'wandb_project': 'stochastic_sparsity', 'wandb_name': 'sparse 64%', 'wandb_tags': ['qwen3', '1.2b', 'sparse'], 'hidden_size': 1536, 'intermediate_size': 6144, 'num_attention_heads': 12, 'num_key_value_heads': 4, 'num_hidden_layers': None, 'head_dim': None, 'attention_type': 'sparse_stochastic', 'sparse_block_size': 128, 'sparse_num_random_blocks': 22, 'sparse_mask_pool_size': 10000, 'sparse_val_dense_compare': False, 'sparse_layer_indices': None, 'sparse_mask_cache_dir': 'mask_cache_23', 'dense_mode': False, 'dense_step_intervals': None, '_wandb': {}}
+2026-06-09 06:41:06,436 INFO    MainThread:3220 [wandb_init.py:init():814] starting backend
+2026-06-09 06:41:06,436 INFO    MainThread:3220 [wandb_init.py:init():829] sending inform_init request
+2026-06-09 06:41:06,787 INFO    MainThread:3220 [wandb_init.py:init():834] backend started and connected
+2026-06-09 06:41:06,790 INFO    MainThread:3220 [wandb_init.py:init():904] updated telemetry
+2026-06-09 06:41:06,791 INFO    MainThread:3220 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout
+2026-06-09 06:41:07,365 INFO    MainThread:3220 [wandb_init.py:init():972] starting run threads in backend
+2026-06-09 06:41:07,468 INFO    MainThread:3220 [wandb_run.py:_console_start():2523] atexit reg
+2026-06-09 06:41:07,468 INFO    MainThread:3220 [wandb_run.py:_redirect():2373] redirect: wrap_raw
+2026-06-09 06:41:07,468 INFO    MainThread:3220 [wandb_run.py:_redirect():2442] Wrapping output streams.
+2026-06-09 06:41:07,468 INFO    MainThread:3220 [wandb_run.py:_redirect():2465] Redirects installed.
+2026-06-09 06:41:07,470 INFO    MainThread:3220 [wandb_init.py:init():1010] run started, returning control to user process

epfl_final/wandb/run-20260609_064106-jh4dfz9a/run-jh4dfz9a.wandb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d03f0ba0c45127462f1fe12083960a8a89f3a3738cc05a64b4cc7d69abe45699
+size 15564800

epfl_final/wandb/run-20260611_032539-jh4dfz9a/files/requirements.txt

@@ -0,0 +1,122 @@
+word2number==1.1
+sqlitedict==2.1.0
+pytz==2026.2
+nvidia-cusparselt-cu13==0.8.1
+mpmath==1.3.0
+cuda-toolkit==13.0.2
+zstandard==0.25.0
+xxhash==3.7.0
+urllib3==2.7.0
+tzdata==2026.2
+typing_extensions==4.15.0
+triton==3.7.0
+tqdm==4.68.2
+torchinfo==1.8.0
+threadpoolctl==3.6.0
+tcolorpy==0.1.7
+tabulate==0.10.0
+sympy==1.14.0
+smmap==5.0.3
+six==1.17.0
+shellingham==1.5.4
+safetensors==0.8.0
+regex==2026.5.9
+PyYAML==6.0.3
+Pygments==2.20.0
+pyarrow==24.0.0
+psutil==7.2.2
+protobuf==7.35.0
+propcache==0.5.2
+portalocker==3.2.0
+platformdirs==4.10.0
+pathvalidate==3.3.1
+packaging==26.2
+nvidia-nvtx==13.0.85
+nvidia-nvshmem-cu13==3.4.5
+nvidia-nvjitlink==13.0.88
+nvidia-nccl-cu13==2.29.7
+nvidia-curand==10.4.0.35
+nvidia-cufile==1.15.1.6
+nvidia-cuda-runtime==13.0.96
+nvidia-cuda-nvrtc==13.0.88
+nvidia-cuda-cupti==13.0.85
+numpy==2.2.6
+networkx==3.4.2
+more-itertools==11.1.0
+mdurl==0.1.2
+MarkupSafe==3.0.3
+lxml==6.1.1
+joblib==1.5.3
+idna==3.18
+hf-xet==1.5.1
+h11==0.16.0
+fsspec==2026.4.0
+frozenlist==1.8.0
+filelock==3.29.3
+dill==0.4.1
+cuda-pathfinder==1.5.5
+colorama==0.4.6
+click==8.4.1
+charset-normalizer==3.4.7
+chardet==6.0.0.post1
+certifi==2026.5.20
+attrs==26.1.0
+async-timeout==5.0.1
+annotated-types==0.7.0
+annotated-doc==0.0.4
+aiohappyeyeballs==2.6.2
+absl-py==2.4.0
+typing-inspection==0.4.2
+sentry-sdk==2.62.0
+scipy==1.15.3
+sacrebleu==2.6.0
+requests==2.34.2
+python-dateutil==2.9.0.post0
+pydantic_core==2.46.4
+nvidia-cusparse==12.6.3.3
+nvidia-cufft==12.0.0.61
+nvidia-cublas==13.1.1.3
+nltk==3.9.4
+multiprocess==0.70.19
+multidict==6.7.1
+mbstrdecoder==1.1.5
+markdown-it-py==4.2.0
+Jinja2==3.1.6
+httpcore==1.0.9
+gitdb==4.0.12
+exceptiongroup==1.3.1
+cuda-bindings==13.3.1
+aiosignal==1.4.0
+yarl==1.24.2
+typepy==1.3.5
+scikit-learn==1.7.2
+rouge-score==0.1.2
+rich==15.0.0
+pydantic==2.13.4
+pandas==2.3.3
+nvidia-cusolver==12.0.4.66
+nvidia-cudnn-cu13==9.20.0.48
+GitPython==3.1.50
+anyio==4.13.0
+wandb==0.27.2
+typer==0.25.1
+httpx==0.28.1
+aiohttp==3.14.1
+torch==2.12.0
+huggingface_hub==1.18.0
+DataProperty==1.1.1
+tokenizers==0.22.2
+tabledata==1.3.5
+datasets==5.0.0
+accelerate==1.13.0
+transformers==5.11.0
+pytablewriter==1.2.1
+evaluate==0.4.6
+lm_eval==0.4.12
+PyGObject==3.42.1
+dbus-python==1.2.18
+distro==1.7.0
+pip==22.0.2
+setuptools==59.6.0
+ssh-import-id==5.11
+wheel==0.37.1

epfl_final/wandb/run-20260611_032539-jh4dfz9a/files/wandb-metadata.json

@@ -0,0 +1,54 @@
+{
+  "os": "Linux-6.8.0-124-generic-x86_64-with-glibc2.35",
+  "python": "CPython 3.10.12",
+  "startedAt": "2026-06-11T03:25:39.923972Z",
+  "args": [
+    "--config",
+    "SparseAtt/configs/train_config.json"
+  ],
+  "program": "/mloscratch/homes/navasard/sparse/SparseAtt/train.py",
+  "codePath": "SparseAtt/train.py",
+  "codePathLocal": "SparseAtt/train.py",
+  "email": "ruben.navasardyan@mail.utoronto.ca",
+  "root": "/mloscratch/homes/navasard/sparse",
+  "host": "h100-train-3-gpu-0-6",
+  "executable": "/usr/bin/python3",
+  "cpu_count": 96,
+  "cpu_count_logical": 192,
+  "gpu": "NVIDIA H100 80GB HBM3",
+  "gpu_count": 3,
+  "disk": {
+    "/": {
+      "total": "9374931091456",
+      "used": "6602944241664"
+    }
+  },
+  "memory": {
+    "total": "1623005732864"
+  },
+  "gpu_nvidia": [
+    {
+      "name": "NVIDIA H100 80GB HBM3",
+      "memoryTotal": "85520809984",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-e8ce0e9a-801b-03d2-c898-c0cc7d746d53"
+    },
+    {
+      "name": "NVIDIA H100 80GB HBM3",
+      "memoryTotal": "85520809984",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-47444fc7-ea8c-cef6-aac8-ef408dc8047b"
+    },
+    {
+      "name": "NVIDIA H100 80GB HBM3",
+      "memoryTotal": "85520809984",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-37dc97ad-46ef-50d9-3619-3a6eaea1b6aa"
+    }
+  ],
+  "cudaVersion": "13.0",
+  "writerId": "8171ikbdbyxf06dwxoc29cisnnd6avw0"
+}

epfl_final/wandb/run-20260611_032539-jh4dfz9a/logs/debug.log

@@ -0,0 +1,20 @@
+2026-06-11 03:25:39,943 INFO    MainThread:2823 [wandb_setup.py:_flush():81] Current SDK version is 0.27.2
+2026-06-11 03:25:39,943 INFO    MainThread:2823 [wandb_setup.py:_flush():81] Configure stats pid to 2823
+2026-06-11 03:25:39,944 INFO    MainThread:2823 [wandb_setup.py:_flush():81] Loading settings from environment variables
+2026-06-11 03:25:39,945 INFO    MainThread:2823 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /mloscratch/homes/navasard/sparse/wandb/run-20260611_032539-jh4dfz9a/logs/debug.log
+2026-06-11 03:25:39,945 INFO    MainThread:2823 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /mloscratch/homes/navasard/sparse/wandb/run-20260611_032539-jh4dfz9a/logs/debug-internal.log
+2026-06-11 03:25:39,946 INFO    MainThread:2823 [wandb_init.py:init():766] calling init triggers
+2026-06-11 03:25:39,947 INFO    MainThread:2823 [wandb_init.py:init():771] wandb.init called with sweep_config: {}
+config: {'model_path': '/mloscratch/homes/navasard/qwen3_06b_base', 'model_dtype': 'bfloat16', 'init_from_scratch': True, 'train_path': '/mloscratch/homes/navasard/data/fineweb_large/tokenized', 'val_path': None, 'max_seq_length': 8192, 'num_workers': 8, 'output_dir': 'checkpoints/run_20260609_063834', 'resume_from_checkpoint': 'checkpoints/run_20260609_063834/checkpoint-17000', 'force_new_run': False, 'num_epochs': 1, 'max_steps': None, 'per_device_batch_size': 2, 'gradient_accumulation_steps': 10, 'learning_rate': 0.0005, 'weight_decay': 0.01, 'warmup_steps': 750, 'lr_scheduler': 'cosine', 'max_grad_norm': 1.0, 'save_steps': 1000, 'eval_steps': 250, 'eval_batches': 50, 'logging_steps': 10, 'seed': 42, 'verbose': True, 'checkpoint_activations': False, 'strategy': 'ddp', 'mixed_precision': 'bf16', 'wandb_enabled': True, 'wandb_project': 'stochastic_sparsity', 'wandb_name': 'sparse 64%', 'wandb_tags': ['qwen3', '1.2b', 'sparse'], 'hidden_size': 1536, 'intermediate_size': 6144, 'num_attention_heads': 12, 'num_key_value_heads': 4, 'num_hidden_layers': None, 'head_dim': None, 'attention_type': 'sparse_stochastic', 'sparse_block_size': 128, 'sparse_num_random_blocks': 22, 'sparse_mask_pool_size': 10000, 'sparse_val_dense_compare': False, 'sparse_layer_indices': None, 'sparse_mask_cache_dir': 'mask_cache_23', 'dense_mode': False, 'dense_step_intervals': None, '_wandb': {}}
+2026-06-11 03:25:39,947 INFO    MainThread:2823 [wandb_init.py:init():814] starting backend
+2026-06-11 03:25:39,948 INFO    MainThread:2823 [wandb_init.py:init():829] sending inform_init request
+2026-06-11 03:25:40,252 INFO    MainThread:2823 [wandb_init.py:init():834] backend started and connected
+2026-06-11 03:25:40,258 INFO    MainThread:2823 [wandb_init.py:init():904] updated telemetry
+2026-06-11 03:25:40,260 INFO    MainThread:2823 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout
+2026-06-11 03:25:40,757 INFO    MainThread:2823 [wandb_init.py:init():967] run resumed
+2026-06-11 03:25:40,760 INFO    MainThread:2823 [wandb_init.py:init():972] starting run threads in backend
+2026-06-11 03:25:40,862 INFO    MainThread:2823 [wandb_run.py:_console_start():2523] atexit reg
+2026-06-11 03:25:40,863 INFO    MainThread:2823 [wandb_run.py:_redirect():2373] redirect: wrap_raw
+2026-06-11 03:25:40,863 INFO    MainThread:2823 [wandb_run.py:_redirect():2442] Wrapping output streams.
+2026-06-11 03:25:40,863 INFO    MainThread:2823 [wandb_run.py:_redirect():2465] Redirects installed.
+2026-06-11 03:25:40,865 INFO    MainThread:2823 [wandb_init.py:init():1010] run started, returning control to user process

epfl_final/wandb/run-20260611_032539-jh4dfz9a/run-jh4dfz9a.wandb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5c3973a8fec337bb35b41732651cfd8a69590bd5f3d10d90fbc77a6d6223d7e8
+size 19562496

epfl_final/wandb/run-20260612_172414-jh4dfz9a/files/config.yaml

@@ -0,0 +1,176 @@
+_wandb:
+    value:
+        cli_version: 0.27.2
+        e:
+            vka5m5pssehspy0dkt44prm0qmrb8kp6:
+                args:
+                    - --config
+                    - SparseAtt/configs/train_config.json
+                codePath: SparseAtt/train.py
+                codePathLocal: SparseAtt/train.py
+                cpu_count: 96
+                cpu_count_logical: 192
+                cudaVersion: "13.0"
+                disk:
+                    /:
+                        total: "9374931091456"
+                        used: "5130988728320"
+                email: ruben.navasardyan@mail.utoronto.ca
+                executable: /usr/bin/python3
+                gpu: NVIDIA H100 80GB HBM3
+                gpu_count: 3
+                gpu_nvidia:
+                    - architecture: Hopper
+                      cudaCores: 16896
+                      memoryTotal: "85520809984"
+                      name: NVIDIA H100 80GB HBM3
+                      uuid: GPU-bab1e315-c967-43cf-bdfb-cac1a8dfdfac
+                    - architecture: Hopper
+                      cudaCores: 16896
+                      memoryTotal: "85520809984"
+                      name: NVIDIA H100 80GB HBM3
+                      uuid: GPU-f65cda91-ff87-9bff-fb3d-3e1b3f9f8cf0
+                    - architecture: Hopper
+                      cudaCores: 16896
+                      memoryTotal: "85520809984"
+                      name: NVIDIA H100 80GB HBM3
+                      uuid: GPU-086f9f0b-a6d8-6263-83e6-c2b66721f47f
+                host: h100-train-3-gpu-0-8
+                memory:
+                    total: "1623005732864"
+                os: Linux-6.8.0-124-generic-x86_64-with-glibc2.35
+                program: /mloscratch/homes/navasard/sparse/SparseAtt/train.py
+                python: CPython 3.10.12
+                root: /mloscratch/homes/navasard/sparse
+                startedAt: "2026-06-12T17:24:14.822511Z"
+                writerId: vka5m5pssehspy0dkt44prm0qmrb8kp6
+        m: []
+        python_version: 3.10.12
+        t:
+            "1":
+                - 1
+                - 5
+                - 11
+                - 49
+                - 51
+                - 53
+                - 71
+            "2":
+                - 1
+                - 5
+                - 11
+                - 49
+                - 51
+                - 53
+                - 71
+            "3":
+                - 2
+                - 5
+                - 13
+                - 14
+                - 15
+                - 16
+                - 61
+                - 62
+            "4": 3.10.12
+            "5": 0.27.2
+            "6": 5.12.0
+            "12": 0.27.2
+            "13": linux-x86_64
+attention_type:
+    value: sparse_stochastic
+checkpoint_activations:
+    value: false
+dense_mode:
+    value: false
+dense_step_intervals:
+    value: null
+eval_batches:
+    value: 50
+eval_steps:
+    value: 250
+force_new_run:
+    value: false
+gradient_accumulation_steps:
+    value: 10
+head_dim:
+    value: null
+hidden_size:
+    value: 1536
+init_from_scratch:
+    value: true
+intermediate_size:
+    value: 6144
+learning_rate:
+    value: 0.0005
+logging_steps:
+    value: 10
+lr_scheduler:
+    value: cosine
+max_grad_norm:
+    value: 1
+max_seq_length:
+    value: 8192
+max_steps:
+    value: null
+mixed_precision:
+    value: bf16
+model_dtype:
+    value: bfloat16
+model_path:
+    value: /mloscratch/homes/navasard/qwen3_06b_base
+num_attention_heads:
+    value: 12
+num_epochs:
+    value: 1
+num_hidden_layers:
+    value: null
+num_key_value_heads:
+    value: 4
+num_workers:
+    value: 8
+output_dir:
+    value: checkpoints/run_20260609_063834
+per_device_batch_size:
+    value: 2
+resume_from_checkpoint:
+    value: checkpoints/run_20260609_063834/checkpoint-39000
+save_steps:
+    value: 1000
+seed:
+    value: 42
+sparse_block_size:
+    value: 128
+sparse_layer_indices:
+    value: null
+sparse_mask_cache_dir:
+    value: mask_cache_23
+sparse_mask_pool_size:
+    value: 10000
+sparse_num_random_blocks:
+    value: 22
+sparse_val_dense_compare:
+    value: false
+strategy:
+    value: ddp
+train_path:
+    value: /mloscratch/homes/navasard/data/fineweb_large/tokenized
+val_path:
+    value: null
+verbose:
+    value: true
+wandb_enabled:
+    value: true
+wandb_name:
+    value: sparse 64%
+wandb_project:
+    value: stochastic_sparsity
+wandb_tags:
+    value:
+        - qwen3
+        - 1.2b
+        - sparse
+warmup_steps:
+    value: 750
+weight_decay:
+    value: 0.01

epfl_final/wandb/run-20260612_172414-jh4dfz9a/files/output.log

@@ -0,0 +1,1218 @@
+Starting training...
+Validation at step 39000: Loss = 2.5565, Perplexity = 12.89
+wandb: WARNING Tried to log to step 39000 that is less than the current step 39111. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Switching attention mode for step 39001: sparse_stochastic (configured sparse attention)
+Step 39010/50462 | Loss: 2.5751 | LR: 6.27e-05 | 69227.7ms/step | 0.9 samples/s
+wandb: WARNING Tried to log to step 39010 that is less than the current step 39111. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 39020/50462 | Loss: 2.5254 | LR: 6.26e-05 | 2990.5ms/step | 20.1 samples/s
+wandb: WARNING Tried to log to step 39020 that is less than the current step 39111. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 39030/50462 | Loss: 2.5528 | LR: 6.25e-05 | 2990.4ms/step | 20.1 samples/s
+wandb: WARNING Tried to log to step 39030 that is less than the current step 39111. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 39040/50462 | Loss: 2.5419 | LR: 6.23e-05 | 2989.6ms/step | 20.1 samples/s
+wandb: WARNING Tried to log to step 39040 that is less than the current step 39111. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 39050/50462 | Loss: 2.5181 | LR: 6.22e-05 | 2989.8ms/step | 20.1 samples/s
+wandb: WARNING Tried to log to step 39050 that is less than the current step 39111. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 39060/50462 | Loss: 2.5302 | LR: 6.21e-05 | 2995.8ms/step | 20.0 samples/s
+wandb: WARNING Tried to log to step 39060 that is less than the current step 39111. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 39070/50462 | Loss: 2.5524 | LR: 6.20e-05 | 3000.0ms/step | 20.0 samples/s
+wandb: WARNING Tried to log to step 39070 that is less than the current step 39111. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 39080/50462 | Loss: 2.5686 | LR: 6.19e-05 | 2996.9ms/step | 20.0 samples/s
+wandb: WARNING Tried to log to step 39080 that is less than the current step 39111. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 39090/50462 | Loss: 2.5252 | LR: 6.18e-05 | 2993.5ms/step | 20.0 samples/s
+wandb: WARNING Tried to log to step 39090 that is less than the current step 39111. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 39100/50462 | Loss: 2.5487 | LR: 6.17e-05 | 2997.6ms/step | 20.0 samples/s
+wandb: WARNING Tried to log to step 39100 that is less than the current step 39111. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 39110/50462 | Loss: 2.5217 | LR: 6.16e-05 | 2996.5ms/step | 20.0 samples/s
+wandb: WARNING Tried to log to step 39110 that is less than the current step 39111. Steps must be monotonically increasing, so this data will be ignored. See https://wandb.me/define-metric to log data out of order.
+Step 39120/50462 | Loss: 2.5284 | LR: 6.15e-05 | 2996.0ms/step | 20.0 samples/s
+Step 39130/50462 | Loss: 2.5663 | LR: 6.14e-05 | 2995.9ms/step | 20.0 samples/s
+Step 39140/50462 | Loss: 2.5391 | LR: 6.13e-05 | 2995.2ms/step | 20.0 samples/s
+Step 39150/50462 | Loss: 2.5429 | LR: 6.12e-05 | 2995.5ms/step | 20.0 samples/s
+Step 39160/50462 | Loss: 2.5303 | LR: 6.11e-05 | 2993.4ms/step | 20.0 samples/s
+Step 39170/50462 | Loss: 2.5573 | LR: 6.10e-05 | 2995.8ms/step | 20.0 samples/s
+Step 39180/50462 | Loss: 2.5674 | LR: 6.09e-05 | 2994.4ms/step | 20.0 samples/s
+Step 39190/50462 | Loss: 2.5520 | LR: 6.08e-05 | 2997.5ms/step | 20.0 samples/s
+Step 39200/50462 | Loss: 2.5202 | LR: 6.07e-05 | 2996.4ms/step | 20.0 samples/s
+Step 39210/50462 | Loss: 2.5425 | LR: 6.06e-05 | 2995.9ms/step | 20.0 samples/s
+Step 39220/50462 | Loss: 2.5161 | LR: 6.05e-05 | 2998.4ms/step | 20.0 samples/s
+Step 39230/50462 | Loss: 2.5613 | LR: 6.04e-05 | 2996.1ms/step | 20.0 samples/s
+Step 39240/50462 | Loss: 2.5403 | LR: 6.03e-05 | 2993.5ms/step | 20.0 samples/s
+Step 39250/50462 | Loss: 2.5608 | LR: 6.02e-05 | 2996.1ms/step | 20.0 samples/s
+Validation at step 39250: Loss = 2.5562, Perplexity = 12.89
+Step 39260/50462 | Loss: 2.5445 | LR: 6.01e-05 | 4592.2ms/step | 13.1 samples/s
+Step 39270/50462 | Loss: 2.5458 | LR: 6.00e-05 | 2992.1ms/step | 20.1 samples/s
+Step 39280/50462 | Loss: 2.5598 | LR: 5.99e-05 | 2992.0ms/step | 20.1 samples/s
+Step 39290/50462 | Loss: 2.5394 | LR: 5.98e-05 | 2995.8ms/step | 20.0 samples/s
+Step 39300/50462 | Loss: 2.5580 | LR: 5.97e-05 | 2993.1ms/step | 20.0 samples/s
+Step 39310/50462 | Loss: 2.5464 | LR: 5.96e-05 | 2992.9ms/step | 20.0 samples/s
+Step 39320/50462 | Loss: 2.4999 | LR: 5.95e-05 | 2992.6ms/step | 20.0 samples/s
+Step 39330/50462 | Loss: 2.5461 | LR: 5.94e-05 | 2999.1ms/step | 20.0 samples/s
+Step 39340/50462 | Loss: 2.5502 | LR: 5.93e-05 | 2993.8ms/step | 20.0 samples/s
+Step 39350/50462 | Loss: 2.5446 | LR: 5.91e-05 | 2994.3ms/step | 20.0 samples/s
+Step 39360/50462 | Loss: 2.5185 | LR: 5.90e-05 | 2996.5ms/step | 20.0 samples/s
+Step 39370/50462 | Loss: 2.5470 | LR: 5.89e-05 | 2995.1ms/step | 20.0 samples/s
+Step 39380/50462 | Loss: 2.5346 | LR: 5.88e-05 | 2997.2ms/step | 20.0 samples/s
+Step 39390/50462 | Loss: 2.5485 | LR: 5.87e-05 | 2995.0ms/step | 20.0 samples/s
+Step 39400/50462 | Loss: 2.5578 | LR: 5.86e-05 | 2992.6ms/step | 20.0 samples/s
+Step 39410/50462 | Loss: 2.5073 | LR: 5.85e-05 | 2996.2ms/step | 20.0 samples/s
+Step 39420/50462 | Loss: 2.5264 | LR: 5.84e-05 | 2998.1ms/step | 20.0 samples/s
+Step 39430/50462 | Loss: 2.5231 | LR: 5.83e-05 | 2997.9ms/step | 20.0 samples/s
+Step 39440/50462 | Loss: 2.5430 | LR: 5.82e-05 | 2994.9ms/step | 20.0 samples/s
+Step 39450/50462 | Loss: 2.5328 | LR: 5.81e-05 | 2995.6ms/step | 20.0 samples/s
+Step 39460/50462 | Loss: 2.5467 | LR: 5.80e-05 | 2997.4ms/step | 20.0 samples/s
+Step 39470/50462 | Loss: 2.5815 | LR: 5.79e-05 | 2999.9ms/step | 20.0 samples/s
+Step 39480/50462 | Loss: 2.5664 | LR: 5.78e-05 | 2992.3ms/step | 20.1 samples/s
+Step 39490/50462 | Loss: 2.5363 | LR: 5.77e-05 | 2995.6ms/step | 20.0 samples/s
+Step 39500/50462 | Loss: 2.5502 | LR: 5.76e-05 | 2994.6ms/step | 20.0 samples/s
+Validation at step 39500: Loss = 2.5553, Perplexity = 12.88
+Step 39510/50462 | Loss: 2.5479 | LR: 5.75e-05 | 4595.5ms/step | 13.1 samples/s
+Step 39520/50462 | Loss: 2.5591 | LR: 5.74e-05 | 2994.6ms/step | 20.0 samples/s
+Step 39530/50462 | Loss: 2.5463 | LR: 5.73e-05 | 2995.8ms/step | 20.0 samples/s
+Step 39540/50462 | Loss: 2.5464 | LR: 5.72e-05 | 2996.0ms/step | 20.0 samples/s
+Step 39550/50462 | Loss: 2.5498 | LR: 5.71e-05 | 2995.7ms/step | 20.0 samples/s
+Step 39560/50462 | Loss: 2.5390 | LR: 5.70e-05 | 2999.5ms/step | 20.0 samples/s
+Step 39570/50462 | Loss: 2.5503 | LR: 5.69e-05 | 2997.2ms/step | 20.0 samples/s
+Step 39580/50462 | Loss: 2.5316 | LR: 5.68e-05 | 2997.6ms/step | 20.0 samples/s
+Step 39590/50462 | Loss: 2.5555 | LR: 5.67e-05 | 2998.1ms/step | 20.0 samples/s
+Step 39600/50462 | Loss: 2.5391 | LR: 5.66e-05 | 3583.8ms/step | 16.7 samples/s
+Step 39610/50462 | Loss: 2.5316 | LR: 5.65e-05 | 3654.3ms/step | 16.4 samples/s
+Step 39620/50462 | Loss: 2.5260 | LR: 5.64e-05 | 2991.7ms/step | 20.1 samples/s
+Step 39630/50462 | Loss: 2.5168 | LR: 5.63e-05 | 2994.8ms/step | 20.0 samples/s
+Step 39640/50462 | Loss: 2.5715 | LR: 5.62e-05 | 2999.7ms/step | 20.0 samples/s
+Step 39650/50462 | Loss: 2.5465 | LR: 5.61e-05 | 2998.2ms/step | 20.0 samples/s
+Step 39660/50462 | Loss: 2.5464 | LR: 5.60e-05 | 2995.8ms/step | 20.0 samples/s
+Step 39670/50462 | Loss: 2.5459 | LR: 5.59e-05 | 2994.7ms/step | 20.0 samples/s
+Step 39680/50462 | Loss: 2.5548 | LR: 5.58e-05 | 2994.2ms/step | 20.0 samples/s
+Step 39690/50462 | Loss: 2.5408 | LR: 5.57e-05 | 3584.2ms/step | 16.7 samples/s
+Step 39700/50462 | Loss: 2.5370 | LR: 5.56e-05 | 2993.5ms/step | 20.0 samples/s
+Step 39710/50462 | Loss: 2.5269 | LR: 5.55e-05 | 2999.7ms/step | 20.0 samples/s
+Step 39720/50462 | Loss: 2.5184 | LR: 5.54e-05 | 2995.7ms/step | 20.0 samples/s
+Step 39730/50462 | Loss: 2.5206 | LR: 5.53e-05 | 2998.5ms/step | 20.0 samples/s
+Step 39740/50462 | Loss: 2.5410 | LR: 5.52e-05 | 3000.0ms/step | 20.0 samples/s
+Step 39750/50462 | Loss: 2.5507 | LR: 5.51e-05 | 3000.2ms/step | 20.0 samples/s
+Validation at step 39750: Loss = 2.5551, Perplexity = 12.87
+Step 39760/50462 | Loss: 2.5401 | LR: 5.50e-05 | 4595.5ms/step | 13.1 samples/s
+Step 39770/50462 | Loss: 2.5626 | LR: 5.49e-05 | 2996.6ms/step | 20.0 samples/s
+Step 39780/50462 | Loss: 2.5626 | LR: 5.48e-05 | 2999.7ms/step | 20.0 samples/s
+Step 39790/50462 | Loss: 2.5398 | LR: 5.47e-05 | 3000.2ms/step | 20.0 samples/s
+Step 39800/50462 | Loss: 2.5328 | LR: 5.46e-05 | 2999.9ms/step | 20.0 samples/s
+Step 39810/50462 | Loss: 2.5465 | LR: 5.45e-05 | 2999.8ms/step | 20.0 samples/s
+Step 39820/50462 | Loss: 2.5667 | LR: 5.44e-05 | 3000.3ms/step | 20.0 samples/s
+Step 39830/50462 | Loss: 2.5386 | LR: 5.43e-05 | 2999.9ms/step | 20.0 samples/s
+Step 39840/50462 | Loss: 2.5491 | LR: 5.42e-05 | 3000.0ms/step | 20.0 samples/s
+Step 39850/50462 | Loss: 2.5559 | LR: 5.41e-05 | 3000.6ms/step | 20.0 samples/s
+Step 39860/50462 | Loss: 2.5189 | LR: 5.40e-05 | 2995.3ms/step | 20.0 samples/s
+Step 39870/50462 | Loss: 2.5444 | LR: 5.39e-05 | 3000.3ms/step | 20.0 samples/s
+Step 39880/50462 | Loss: 2.5443 | LR: 5.38e-05 | 2996.4ms/step | 20.0 samples/s
+Step 39890/50462 | Loss: 2.5536 | LR: 5.38e-05 | 2997.5ms/step | 20.0 samples/s
+Step 39900/50462 | Loss: 2.5359 | LR: 5.37e-05 | 2998.2ms/step | 20.0 samples/s
+Step 39910/50462 | Loss: 2.5423 | LR: 5.36e-05 | 2998.2ms/step | 20.0 samples/s
+Step 39920/50462 | Loss: 2.5422 | LR: 5.35e-05 | 2997.5ms/step | 20.0 samples/s
+Step 39930/50462 | Loss: 2.5463 | LR: 5.34e-05 | 3000.1ms/step | 20.0 samples/s
+Step 39940/50462 | Loss: 2.5366 | LR: 5.33e-05 | 3000.5ms/step | 20.0 samples/s
+Step 39950/50462 | Loss: 2.5269 | LR: 5.32e-05 | 2999.8ms/step | 20.0 samples/s
+Step 39960/50462 | Loss: 2.5446 | LR: 5.31e-05 | 3000.2ms/step | 20.0 samples/s
+Step 39970/50462 | Loss: 2.5673 | LR: 5.30e-05 | 2995.9ms/step | 20.0 samples/s
+Step 39980/50462 | Loss: 2.5446 | LR: 5.29e-05 | 3000.4ms/step | 20.0 samples/s
+Step 39990/50462 | Loss: 2.5791 | LR: 5.28e-05 | 2999.5ms/step | 20.0 samples/s
+Step 40000/50462 | Loss: 2.5167 | LR: 5.27e-05 | 3000.3ms/step | 20.0 samples/s
+Validation at step 40000: Loss = 2.5550, Perplexity = 12.87
+Saved checkpoint to checkpoints/run_20260609_063834/checkpoint-40000
+Step 40010/50462 | Loss: 2.5602 | LR: 5.26e-05 | 5615.9ms/step | 10.7 samples/s
+Step 40020/50462 | Loss: 2.5821 | LR: 5.25e-05 | 2995.9ms/step | 20.0 samples/s
+Step 40030/50462 | Loss: 2.5278 | LR: 5.24e-05 | 2998.5ms/step | 20.0 samples/s
+Step 40040/50462 | Loss: 2.5587 | LR: 5.23e-05 | 2999.0ms/step | 20.0 samples/s
+Step 40050/50462 | Loss: 2.5460 | LR: 5.22e-05 | 3000.7ms/step | 20.0 samples/s
+Step 40060/50462 | Loss: 2.5399 | LR: 5.21e-05 | 2999.4ms/step | 20.0 samples/s
+Step 40070/50462 | Loss: 2.5506 | LR: 5.20e-05 | 3000.4ms/step | 20.0 samples/s
+Step 40080/50462 | Loss: 2.5255 | LR: 5.19e-05 | 2998.5ms/step | 20.0 samples/s
+Step 40090/50462 | Loss: 2.5585 | LR: 5.18e-05 | 2999.7ms/step | 20.0 samples/s
+Step 40100/50462 | Loss: 2.5554 | LR: 5.17e-05 | 2995.0ms/step | 20.0 samples/s
+Step 40110/50462 | Loss: 2.5421 | LR: 5.16e-05 | 2994.8ms/step | 20.0 samples/s
+Step 40120/50462 | Loss: 2.5564 | LR: 5.15e-05 | 3000.1ms/step | 20.0 samples/s
+Step 40130/50462 | Loss: 2.5390 | LR: 5.14e-05 | 3000.3ms/step | 20.0 samples/s
+Step 40140/50462 | Loss: 2.5353 | LR: 5.13e-05 | 2999.4ms/step | 20.0 samples/s
+Step 40150/50462 | Loss: 2.5135 | LR: 5.12e-05 | 3000.8ms/step | 20.0 samples/s
+Step 40160/50462 | Loss: 2.5109 | LR: 5.11e-05 | 3001.8ms/step | 20.0 samples/s
+Step 40170/50462 | Loss: 2.5495 | LR: 5.10e-05 | 2998.1ms/step | 20.0 samples/s
+Step 40180/50462 | Loss: 2.5517 | LR: 5.09e-05 | 2997.9ms/step | 20.0 samples/s
+Step 40190/50462 | Loss: 2.5482 | LR: 5.09e-05 | 2998.1ms/step | 20.0 samples/s
+Step 40200/50462 | Loss: 2.5573 | LR: 5.08e-05 | 2999.7ms/step | 20.0 samples/s
+Step 40210/50462 | Loss: 2.5622 | LR: 5.07e-05 | 2996.6ms/step | 20.0 samples/s
+Step 40220/50462 | Loss: 2.5620 | LR: 5.06e-05 | 2999.7ms/step | 20.0 samples/s
+Step 40230/50462 | Loss: 2.5539 | LR: 5.05e-05 | 2999.9ms/step | 20.0 samples/s
+Step 40240/50462 | Loss: 2.5841 | LR: 5.04e-05 | 3000.2ms/step | 20.0 samples/s
+Step 40250/50462 | Loss: 2.5438 | LR: 5.03e-05 | 3000.7ms/step | 20.0 samples/s
+Validation at step 40250: Loss = 2.5549, Perplexity = 12.87
+Step 40260/50462 | Loss: 2.5495 | LR: 5.02e-05 | 4597.6ms/step | 13.1 samples/s
+Step 40270/50462 | Loss: 2.5244 | LR: 5.01e-05 | 2999.9ms/step | 20.0 samples/s
+Step 40280/50462 | Loss: 2.5483 | LR: 5.00e-05 | 2999.8ms/step | 20.0 samples/s
+Step 40290/50462 | Loss: 2.5457 | LR: 4.99e-05 | 2999.9ms/step | 20.0 samples/s
+Step 40300/50462 | Loss: 2.5417 | LR: 4.98e-05 | 2995.4ms/step | 20.0 samples/s
+Step 40310/50462 | Loss: 2.5278 | LR: 4.97e-05 | 2994.6ms/step | 20.0 samples/s
+Step 40320/50462 | Loss: 2.5463 | LR: 4.96e-05 | 2999.7ms/step | 20.0 samples/s
+Step 40330/50462 | Loss: 2.5686 | LR: 4.95e-05 | 2998.1ms/step | 20.0 samples/s
+Step 40340/50462 | Loss: 2.5288 | LR: 4.94e-05 | 2998.6ms/step | 20.0 samples/s
+Step 40350/50462 | Loss: 2.5199 | LR: 4.93e-05 | 2999.3ms/step | 20.0 samples/s
+Step 40360/50462 | Loss: 2.5492 | LR: 4.92e-05 | 2996.5ms/step | 20.0 samples/s
+Step 40370/50462 | Loss: 2.5620 | LR: 4.91e-05 | 2998.2ms/step | 20.0 samples/s
+Step 40380/50462 | Loss: 2.5479 | LR: 4.90e-05 | 3000.3ms/step | 20.0 samples/s
+Step 40390/50462 | Loss: 2.5433 | LR: 4.90e-05 | 2999.4ms/step | 20.0 samples/s
+Step 40400/50462 | Loss: 2.5567 | LR: 4.89e-05 | 2998.3ms/step | 20.0 samples/s
+Step 40410/50462 | Loss: 2.5687 | LR: 4.88e-05 | 2999.6ms/step | 20.0 samples/s
+Step 40420/50462 | Loss: 2.5685 | LR: 4.87e-05 | 3000.6ms/step | 20.0 samples/s
+Step 40430/50462 | Loss: 2.5358 | LR: 4.86e-05 | 3000.0ms/step | 20.0 samples/s
+Step 40440/50462 | Loss: 2.5580 | LR: 4.85e-05 | 3000.1ms/step | 20.0 samples/s
+Step 40450/50462 | Loss: 2.5438 | LR: 4.84e-05 | 3000.0ms/step | 20.0 samples/s
+Step 40460/50462 | Loss: 2.5764 | LR: 4.83e-05 | 3000.2ms/step | 20.0 samples/s
+Step 40470/50462 | Loss: 2.5297 | LR: 4.82e-05 | 2999.6ms/step | 20.0 samples/s
+Step 40480/50462 | Loss: 2.5420 | LR: 4.81e-05 | 2998.3ms/step | 20.0 samples/s
+Step 40490/50462 | Loss: 2.5576 | LR: 4.80e-05 | 3001.9ms/step | 20.0 samples/s
+Step 40500/50462 | Loss: 2.5478 | LR: 4.79e-05 | 2999.9ms/step | 20.0 samples/s
+Validation at step 40500: Loss = 2.5547, Perplexity = 12.87
+Step 40510/50462 | Loss: 2.5462 | LR: 4.78e-05 | 4600.3ms/step | 13.0 samples/s
+Step 40520/50462 | Loss: 2.5319 | LR: 4.77e-05 | 2998.1ms/step | 20.0 samples/s
+Step 40530/50462 | Loss: 2.5367 | LR: 4.76e-05 | 2999.3ms/step | 20.0 samples/s
+Step 40540/50462 | Loss: 2.5097 | LR: 4.76e-05 | 2998.7ms/step | 20.0 samples/s
+Step 40550/50462 | Loss: 2.5639 | LR: 4.75e-05 | 3000.0ms/step | 20.0 samples/s
+Step 40560/50462 | Loss: 2.5548 | LR: 4.74e-05 | 3000.0ms/step | 20.0 samples/s
+Step 40570/50462 | Loss: 2.5304 | LR: 4.73e-05 | 2999.5ms/step | 20.0 samples/s
+Step 40580/50462 | Loss: 2.5583 | LR: 4.72e-05 | 3000.7ms/step | 20.0 samples/s
+Step 40590/50462 | Loss: 2.5418 | LR: 4.71e-05 | 3000.1ms/step | 20.0 samples/s
+Step 40600/50462 | Loss: 2.5425 | LR: 4.70e-05 | 2999.7ms/step | 20.0 samples/s
+Step 40610/50462 | Loss: 2.5619 | LR: 4.69e-05 | 3000.1ms/step | 20.0 samples/s
+Step 40620/50462 | Loss: 2.5186 | LR: 4.68e-05 | 3004.1ms/step | 20.0 samples/s
+Step 40630/50462 | Loss: 2.5394 | LR: 4.67e-05 | 2999.7ms/step | 20.0 samples/s
+Step 40640/50462 | Loss: 2.5496 | LR: 4.66e-05 | 3000.4ms/step | 20.0 samples/s
+Step 40650/50462 | Loss: 2.5498 | LR: 4.65e-05 | 2999.9ms/step | 20.0 samples/s
+Step 40660/50462 | Loss: 2.5629 | LR: 4.64e-05 | 3000.2ms/step | 20.0 samples/s
+Step 40670/50462 | Loss: 2.5379 | LR: 4.64e-05 | 2999.8ms/step | 20.0 samples/s
+Step 40680/50462 | Loss: 2.5434 | LR: 4.63e-05 | 3000.3ms/step | 20.0 samples/s
+Step 40690/50462 | Loss: 2.5404 | LR: 4.62e-05 | 2998.0ms/step | 20.0 samples/s
+Step 40700/50462 | Loss: 2.5121 | LR: 4.61e-05 | 3000.0ms/step | 20.0 samples/s
+Step 40710/50462 | Loss: 2.5525 | LR: 4.60e-05 | 3000.0ms/step | 20.0 samples/s
+Step 40720/50462 | Loss: 2.5181 | LR: 4.59e-05 | 3000.5ms/step | 20.0 samples/s
+Step 40730/50462 | Loss: 2.5299 | LR: 4.58e-05 | 2997.4ms/step | 20.0 samples/s
+Step 40740/50462 | Loss: 2.5257 | LR: 4.57e-05 | 2998.1ms/step | 20.0 samples/s
+Step 40750/50462 | Loss: 2.5355 | LR: 4.56e-05 | 2998.1ms/step | 20.0 samples/s
+Validation at step 40750: Loss = 2.5548, Perplexity = 12.87
+Step 40760/50462 | Loss: 2.5616 | LR: 4.55e-05 | 4593.3ms/step | 13.1 samples/s
+Step 40770/50462 | Loss: 2.5420 | LR: 4.54e-05 | 3002.7ms/step | 20.0 samples/s
+Step 40780/50462 | Loss: 2.5568 | LR: 4.54e-05 | 2999.2ms/step | 20.0 samples/s
+Step 40790/50462 | Loss: 2.5417 | LR: 4.53e-05 | 2996.9ms/step | 20.0 samples/s
+Step 40800/50462 | Loss: 2.5508 | LR: 4.52e-05 | 3000.1ms/step | 20.0 samples/s
+Step 40810/50462 | Loss: 2.5080 | LR: 4.51e-05 | 3000.0ms/step | 20.0 samples/s
+Step 40820/50462 | Loss: 2.5822 | LR: 4.50e-05 | 2999.8ms/step | 20.0 samples/s
+Step 40830/50462 | Loss: 2.4968 | LR: 4.49e-05 | 3000.0ms/step | 20.0 samples/s
+Step 40840/50462 | Loss: 2.5220 | LR: 4.48e-05 | 2998.3ms/step | 20.0 samples/s
+Step 40850/50462 | Loss: 2.5653 | LR: 4.47e-05 | 2995.6ms/step | 20.0 samples/s
+Step 40860/50462 | Loss: 2.5104 | LR: 4.46e-05 | 2998.3ms/step | 20.0 samples/s
+Step 40870/50462 | Loss: 2.5655 | LR: 4.45e-05 | 2998.1ms/step | 20.0 samples/s
+Step 40880/50462 | Loss: 2.5715 | LR: 4.45e-05 | 2999.5ms/step | 20.0 samples/s
+Step 40890/50462 | Loss: 2.5041 | LR: 4.44e-05 | 3000.4ms/step | 20.0 samples/s
+Step 40900/50462 | Loss: 2.5293 | LR: 4.43e-05 | 3000.2ms/step | 20.0 samples/s
+Step 40910/50462 | Loss: 2.5504 | LR: 4.42e-05 | 2999.5ms/step | 20.0 samples/s
+Step 40920/50462 | Loss: 2.5588 | LR: 4.41e-05 | 2999.4ms/step | 20.0 samples/s
+Step 40930/50462 | Loss: 2.5494 | LR: 4.40e-05 | 2995.0ms/step | 20.0 samples/s
+Step 40940/50462 | Loss: 2.5672 | LR: 4.39e-05 | 2998.8ms/step | 20.0 samples/s
+Step 40950/50462 | Loss: 2.5218 | LR: 4.38e-05 | 2996.5ms/step | 20.0 samples/s
+Step 40960/50462 | Loss: 2.5510 | LR: 4.37e-05 | 2995.0ms/step | 20.0 samples/s
+Step 40970/50462 | Loss: 2.5414 | LR: 4.36e-05 | 2999.5ms/step | 20.0 samples/s
+Step 40980/50462 | Loss: 2.5417 | LR: 4.36e-05 | 2993.9ms/step | 20.0 samples/s
+Step 40990/50462 | Loss: 2.5098 | LR: 4.35e-05 | 3001.8ms/step | 20.0 samples/s
+Step 41000/50462 | Loss: 2.5550 | LR: 4.34e-05 | 3000.6ms/step | 20.0 samples/s
+Validation at step 41000: Loss = 2.5545, Perplexity = 12.87
+Saved checkpoint to checkpoints/run_20260609_063834/checkpoint-41000
+Step 41010/50462 | Loss: 2.5619 | LR: 4.33e-05 | 5559.2ms/step | 10.8 samples/s
+Step 41020/50462 | Loss: 2.5572 | LR: 4.32e-05 | 2994.2ms/step | 20.0 samples/s
+Step 41030/50462 | Loss: 2.5593 | LR: 4.31e-05 | 2998.2ms/step | 20.0 samples/s
+Step 41040/50462 | Loss: 2.5228 | LR: 4.30e-05 | 2998.6ms/step | 20.0 samples/s
+Step 41050/50462 | Loss: 2.5531 | LR: 4.29e-05 | 2999.9ms/step | 20.0 samples/s
+Step 41060/50462 | Loss: 2.5593 | LR: 4.28e-05 | 2990.7ms/step | 20.1 samples/s
+Step 41070/50462 | Loss: 2.5374 | LR: 4.28e-05 | 2995.2ms/step | 20.0 samples/s
+Step 41080/50462 | Loss: 2.5303 | LR: 4.27e-05 | 2993.6ms/step | 20.0 samples/s
+Step 41090/50462 | Loss: 2.5088 | LR: 4.26e-05 | 2994.5ms/step | 20.0 samples/s
+Step 41100/50462 | Loss: 2.5429 | LR: 4.25e-05 | 2994.4ms/step | 20.0 samples/s
+Step 41110/50462 | Loss: 2.5471 | LR: 4.24e-05 | 3001.7ms/step | 20.0 samples/s
+Step 41120/50462 | Loss: 2.5380 | LR: 4.23e-05 | 2995.1ms/step | 20.0 samples/s
+Step 41130/50462 | Loss: 2.5600 | LR: 4.22e-05 | 2995.1ms/step | 20.0 samples/s
+Step 41140/50462 | Loss: 2.5338 | LR: 4.21e-05 | 2998.1ms/step | 20.0 samples/s
+Step 41150/50462 | Loss: 2.5480 | LR: 4.21e-05 | 2999.7ms/step | 20.0 samples/s
+Step 41160/50462 | Loss: 2.5456 | LR: 4.20e-05 | 2993.9ms/step | 20.0 samples/s
+Step 41170/50462 | Loss: 2.5502 | LR: 4.19e-05 | 2998.3ms/step | 20.0 samples/s
+Step 41180/50462 | Loss: 2.5354 | LR: 4.18e-05 | 3000.1ms/step | 20.0 samples/s
+Step 41190/50462 | Loss: 2.4946 | LR: 4.17e-05 | 2997.5ms/step | 20.0 samples/s
+Step 41200/50462 | Loss: 2.5320 | LR: 4.16e-05 | 2995.5ms/step | 20.0 samples/s
+Step 41210/50462 | Loss: 2.5083 | LR: 4.15e-05 | 2997.1ms/step | 20.0 samples/s
+Step 41220/50462 | Loss: 2.5242 | LR: 4.14e-05 | 2999.9ms/step | 20.0 samples/s
+Step 41230/50462 | Loss: 2.5482 | LR: 4.14e-05 | 3000.0ms/step | 20.0 samples/s
+Step 41240/50462 | Loss: 2.5366 | LR: 4.13e-05 | 2998.0ms/step | 20.0 samples/s
+Step 41250/50462 | Loss: 2.5500 | LR: 4.12e-05 | 2999.5ms/step | 20.0 samples/s
+Validation at step 41250: Loss = 2.5543, Perplexity = 12.86
+Step 41260/50462 | Loss: 2.5464 | LR: 4.11e-05 | 4592.4ms/step | 13.1 samples/s
+Step 41270/50462 | Loss: 2.5208 | LR: 4.10e-05 | 2997.9ms/step | 20.0 samples/s
+Step 41280/50462 | Loss: 2.4995 | LR: 4.09e-05 | 3000.1ms/step | 20.0 samples/s
+Step 41290/50462 | Loss: 2.5605 | LR: 4.08e-05 | 3000.1ms/step | 20.0 samples/s
+Step 41300/50462 | Loss: 2.5517 | LR: 4.07e-05 | 2998.8ms/step | 20.0 samples/s
+Step 41310/50462 | Loss: 2.5129 | LR: 4.07e-05 | 2997.2ms/step | 20.0 samples/s
+Step 41320/50462 | Loss: 2.5948 | LR: 4.06e-05 | 2996.0ms/step | 20.0 samples/s
+Step 41330/50462 | Loss: 2.5310 | LR: 4.05e-05 | 2998.1ms/step | 20.0 samples/s
+Step 41340/50462 | Loss: 2.5241 | LR: 4.04e-05 | 2999.9ms/step | 20.0 samples/s
+Step 41350/50462 | Loss: 2.5461 | LR: 4.03e-05 | 2997.9ms/step | 20.0 samples/s
+Step 41360/50462 | Loss: 2.5722 | LR: 4.02e-05 | 2995.8ms/step | 20.0 samples/s
+Step 41370/50462 | Loss: 2.5433 | LR: 4.01e-05 | 2998.2ms/step | 20.0 samples/s
+Step 41380/50462 | Loss: 2.5363 | LR: 4.01e-05 | 2998.3ms/step | 20.0 samples/s
+Step 41390/50462 | Loss: 2.5118 | LR: 4.00e-05 | 3001.9ms/step | 20.0 samples/s
+Step 41400/50462 | Loss: 2.5525 | LR: 3.99e-05 | 3000.4ms/step | 20.0 samples/s
+Step 41410/50462 | Loss: 2.5579 | LR: 3.98e-05 | 2999.7ms/step | 20.0 samples/s
+Step 41420/50462 | Loss: 2.5210 | LR: 3.97e-05 | 2996.0ms/step | 20.0 samples/s
+Step 41430/50462 | Loss: 2.5532 | LR: 3.96e-05 | 2997.8ms/step | 20.0 samples/s
+Step 41440/50462 | Loss: 2.5218 | LR: 3.95e-05 | 2994.5ms/step | 20.0 samples/s
+Step 41450/50462 | Loss: 2.5417 | LR: 3.95e-05 | 2997.3ms/step | 20.0 samples/s
+Step 41460/50462 | Loss: 2.5240 | LR: 3.94e-05 | 2996.9ms/step | 20.0 samples/s
+Step 41470/50462 | Loss: 2.5026 | LR: 3.93e-05 | 2991.4ms/step | 20.1 samples/s
+Step 41480/50462 | Loss: 2.5617 | LR: 3.92e-05 | 2996.3ms/step | 20.0 samples/s
+Step 41490/50462 | Loss: 2.5149 | LR: 3.91e-05 | 2997.8ms/step | 20.0 samples/s
+Step 41500/50462 | Loss: 2.5579 | LR: 3.90e-05 | 2996.0ms/step | 20.0 samples/s
+Validation at step 41500: Loss = 2.5538, Perplexity = 12.86
+Step 41510/50462 | Loss: 2.5263 | LR: 3.90e-05 | 4593.9ms/step | 13.1 samples/s
+Step 41520/50462 | Loss: 2.5418 | LR: 3.89e-05 | 2995.5ms/step | 20.0 samples/s
+Step 41530/50462 | Loss: 2.5530 | LR: 3.88e-05 | 2996.9ms/step | 20.0 samples/s
+Step 41540/50462 | Loss: 2.5225 | LR: 3.87e-05 | 2995.8ms/step | 20.0 samples/s
+Step 41550/50462 | Loss: 2.5480 | LR: 3.86e-05 | 2995.8ms/step | 20.0 samples/s
+Step 41560/50462 | Loss: 2.5601 | LR: 3.85e-05 | 2998.0ms/step | 20.0 samples/s
+Step 41570/50462 | Loss: 2.5475 | LR: 3.84e-05 | 3000.4ms/step | 20.0 samples/s
+Step 41580/50462 | Loss: 2.5522 | LR: 3.84e-05 | 2998.2ms/step | 20.0 samples/s
+Step 41590/50462 | Loss: 2.5349 | LR: 3.83e-05 | 2999.8ms/step | 20.0 samples/s
+Step 41600/50462 | Loss: 2.5490 | LR: 3.82e-05 | 3000.1ms/step | 20.0 samples/s
+Step 41610/50462 | Loss: 2.5319 | LR: 3.81e-05 | 2993.7ms/step | 20.0 samples/s
+Step 41620/50462 | Loss: 2.5373 | LR: 3.80e-05 | 2996.3ms/step | 20.0 samples/s
+Step 41630/50462 | Loss: 2.5188 | LR: 3.79e-05 | 2994.1ms/step | 20.0 samples/s
+Step 41640/50462 | Loss: 2.5348 | LR: 3.79e-05 | 2998.0ms/step | 20.0 samples/s
+Step 41650/50462 | Loss: 2.5211 | LR: 3.78e-05 | 2995.1ms/step | 20.0 samples/s
+Step 41660/50462 | Loss: 2.5500 | LR: 3.77e-05 | 2998.5ms/step | 20.0 samples/s
+Step 41670/50462 | Loss: 2.5426 | LR: 3.76e-05 | 2996.9ms/step | 20.0 samples/s
+Step 41680/50462 | Loss: 2.5274 | LR: 3.75e-05 | 2995.4ms/step | 20.0 samples/s
+Step 41690/50462 | Loss: 2.5239 | LR: 3.74e-05 | 2993.7ms/step | 20.0 samples/s
+Step 41700/50462 | Loss: 2.5368 | LR: 3.74e-05 | 2993.8ms/step | 20.0 samples/s
+Step 41710/50462 | Loss: 2.5272 | LR: 3.73e-05 | 2996.4ms/step | 20.0 samples/s
+Step 41720/50462 | Loss: 2.5244 | LR: 3.72e-05 | 2998.1ms/step | 20.0 samples/s
+Step 41730/50462 | Loss: 2.5304 | LR: 3.71e-05 | 3540.3ms/step | 16.9 samples/s
+Step 41740/50462 | Loss: 2.5238 | LR: 3.70e-05 | 3626.7ms/step | 16.5 samples/s
+Step 41750/50462 | Loss: 2.5298 | LR: 3.69e-05 | 2993.4ms/step | 20.0 samples/s
+Validation at step 41750: Loss = 2.5538, Perplexity = 12.86
+Step 41760/50462 | Loss: 2.5423 | LR: 3.69e-05 | 4591.8ms/step | 13.1 samples/s
+Step 41770/50462 | Loss: 2.5493 | LR: 3.68e-05 | 2992.0ms/step | 20.1 samples/s
+Step 41780/50462 | Loss: 2.5367 | LR: 3.67e-05 | 2998.4ms/step | 20.0 samples/s
+Step 41790/50462 | Loss: 2.5220 | LR: 3.66e-05 | 3544.7ms/step | 16.9 samples/s
+Step 41800/50462 | Loss: 2.5256 | LR: 3.65e-05 | 2987.0ms/step | 20.1 samples/s
+Step 41810/50462 | Loss: 2.5598 | LR: 3.64e-05 | 2997.8ms/step | 20.0 samples/s
+Step 41820/50462 | Loss: 2.4852 | LR: 3.64e-05 | 3000.4ms/step | 20.0 samples/s
+Step 41830/50462 | Loss: 2.5327 | LR: 3.63e-05 | 2995.1ms/step | 20.0 samples/s
+Step 41840/50462 | Loss: 2.5086 | LR: 3.62e-05 | 2993.6ms/step | 20.0 samples/s
+Step 41850/50462 | Loss: 2.5496 | LR: 3.61e-05 | 2992.8ms/step | 20.0 samples/s
+Step 41860/50462 | Loss: 2.5356 | LR: 3.60e-05 | 2998.5ms/step | 20.0 samples/s
+Step 41870/50462 | Loss: 2.5318 | LR: 3.60e-05 | 2999.8ms/step | 20.0 samples/s
+Step 41880/50462 | Loss: 2.5467 | LR: 3.59e-05 | 2997.1ms/step | 20.0 samples/s
+Step 41890/50462 | Loss: 2.5382 | LR: 3.58e-05 | 2993.1ms/step | 20.0 samples/s
+Step 41900/50462 | Loss: 2.5632 | LR: 3.57e-05 | 2995.1ms/step | 20.0 samples/s
+Step 41910/50462 | Loss: 2.5533 | LR: 3.56e-05 | 2992.4ms/step | 20.1 samples/s
+Step 41920/50462 | Loss: 2.5455 | LR: 3.55e-05 | 2994.0ms/step | 20.0 samples/s
+Step 41930/50462 | Loss: 2.5254 | LR: 3.55e-05 | 2996.7ms/step | 20.0 samples/s
+Step 41940/50462 | Loss: 2.5534 | LR: 3.54e-05 | 2998.1ms/step | 20.0 samples/s
+Step 41950/50462 | Loss: 2.5237 | LR: 3.53e-05 | 2993.4ms/step | 20.0 samples/s
+Step 41960/50462 | Loss: 2.5502 | LR: 3.52e-05 | 3000.7ms/step | 20.0 samples/s
+Step 41970/50462 | Loss: 2.5384 | LR: 3.51e-05 | 2997.6ms/step | 20.0 samples/s
+Step 41980/50462 | Loss: 2.5363 | LR: 3.51e-05 | 2996.2ms/step | 20.0 samples/s
+Step 41990/50462 | Loss: 2.5680 | LR: 3.50e-05 | 2996.3ms/step | 20.0 samples/s
+Step 42000/50462 | Loss: 2.5463 | LR: 3.49e-05 | 3000.1ms/step | 20.0 samples/s
+Validation at step 42000: Loss = 2.5538, Perplexity = 12.86
+Saved checkpoint to checkpoints/run_20260609_063834/checkpoint-42000
+Step 42010/50462 | Loss: 2.5147 | LR: 3.48e-05 | 5600.3ms/step | 10.7 samples/s
+Step 42020/50462 | Loss: 2.5582 | LR: 3.47e-05 | 2999.5ms/step | 20.0 samples/s
+Step 42030/50462 | Loss: 2.5766 | LR: 3.47e-05 | 2998.3ms/step | 20.0 samples/s
+Step 42040/50462 | Loss: 2.5409 | LR: 3.46e-05 | 3002.2ms/step | 20.0 samples/s
+Step 42050/50462 | Loss: 2.5428 | LR: 3.45e-05 | 2999.8ms/step | 20.0 samples/s
+Step 42060/50462 | Loss: 2.5762 | LR: 3.44e-05 | 3000.1ms/step | 20.0 samples/s
+Step 42070/50462 | Loss: 2.5580 | LR: 3.43e-05 | 3000.0ms/step | 20.0 samples/s
+Step 42080/50462 | Loss: 2.5293 | LR: 3.43e-05 | 3000.0ms/step | 20.0 samples/s
+Step 42090/50462 | Loss: 2.5601 | LR: 3.42e-05 | 2999.8ms/step | 20.0 samples/s
+Step 42100/50462 | Loss: 2.5423 | LR: 3.41e-05 | 3000.4ms/step | 20.0 samples/s
+Step 42110/50462 | Loss: 2.5098 | LR: 3.40e-05 | 2998.1ms/step | 20.0 samples/s
+Step 42120/50462 | Loss: 2.5445 | LR: 3.39e-05 | 2995.1ms/step | 20.0 samples/s
+Step 42130/50462 | Loss: 2.5560 | LR: 3.39e-05 | 2992.7ms/step | 20.0 samples/s
+Step 42140/50462 | Loss: 2.5530 | LR: 3.38e-05 | 2997.4ms/step | 20.0 samples/s
+Step 42150/50462 | Loss: 2.5491 | LR: 3.37e-05 | 2989.8ms/step | 20.1 samples/s
+Step 42160/50462 | Loss: 2.5075 | LR: 3.36e-05 | 2990.1ms/step | 20.1 samples/s
+Step 42170/50462 | Loss: 2.5342 | LR: 3.35e-05 | 2994.1ms/step | 20.0 samples/s
+Step 42180/50462 | Loss: 2.5413 | LR: 3.35e-05 | 2991.9ms/step | 20.1 samples/s
+Step 42190/50462 | Loss: 2.5709 | LR: 3.34e-05 | 2988.1ms/step | 20.1 samples/s
+Step 42200/50462 | Loss: 2.5271 | LR: 3.33e-05 | 2990.8ms/step | 20.1 samples/s
+Step 42210/50462 | Loss: 2.5320 | LR: 3.32e-05 | 2993.5ms/step | 20.0 samples/s
+Step 42220/50462 | Loss: 2.5589 | LR: 3.32e-05 | 2994.6ms/step | 20.0 samples/s
+Step 42230/50462 | Loss: 2.5457 | LR: 3.31e-05 | 2994.2ms/step | 20.0 samples/s
+Step 42240/50462 | Loss: 2.5002 | LR: 3.30e-05 | 2998.4ms/step | 20.0 samples/s
+Step 42250/50462 | Loss: 2.5513 | LR: 3.29e-05 | 2990.8ms/step | 20.1 samples/s
+Validation at step 42250: Loss = 2.5537, Perplexity = 12.85
+Step 42260/50462 | Loss: 2.5581 | LR: 3.28e-05 | 4591.9ms/step | 13.1 samples/s
+Step 42270/50462 | Loss: 2.5699 | LR: 3.28e-05 | 2996.1ms/step | 20.0 samples/s
+Step 42280/50462 | Loss: 2.5385 | LR: 3.27e-05 | 2996.4ms/step | 20.0 samples/s
+Step 42290/50462 | Loss: 2.5534 | LR: 3.26e-05 | 2994.5ms/step | 20.0 samples/s
+Step 42300/50462 | Loss: 2.5384 | LR: 3.25e-05 | 2995.9ms/step | 20.0 samples/s
+Step 42310/50462 | Loss: 2.5313 | LR: 3.24e-05 | 2991.5ms/step | 20.1 samples/s
+Step 42320/50462 | Loss: 2.5135 | LR: 3.24e-05 | 2990.6ms/step | 20.1 samples/s
+Step 42330/50462 | Loss: 2.5611 | LR: 3.23e-05 | 2993.4ms/step | 20.0 samples/s
+Step 42340/50462 | Loss: 2.5680 | LR: 3.22e-05 | 2992.1ms/step | 20.1 samples/s
+Step 42350/50462 | Loss: 2.5153 | LR: 3.21e-05 | 2992.0ms/step | 20.1 samples/s
+Step 42360/50462 | Loss: 2.5260 | LR: 3.21e-05 | 2995.1ms/step | 20.0 samples/s
+Step 42370/50462 | Loss: 2.5389 | LR: 3.20e-05 | 2999.6ms/step | 20.0 samples/s
+Step 42380/50462 | Loss: 2.5257 | LR: 3.19e-05 | 2995.7ms/step | 20.0 samples/s
+Step 42390/50462 | Loss: 2.5533 | LR: 3.18e-05 | 2993.3ms/step | 20.0 samples/s
+Step 42400/50462 | Loss: 2.5604 | LR: 3.18e-05 | 2994.9ms/step | 20.0 samples/s
+Step 42410/50462 | Loss: 2.5544 | LR: 3.17e-05 | 2991.6ms/step | 20.1 samples/s
+Step 42420/50462 | Loss: 2.5461 | LR: 3.16e-05 | 2991.9ms/step | 20.1 samples/s
+Step 42430/50462 | Loss: 2.5349 | LR: 3.15e-05 | 2994.8ms/step | 20.0 samples/s
+Step 42440/50462 | Loss: 2.5349 | LR: 3.14e-05 | 2993.9ms/step | 20.0 samples/s
+Step 42450/50462 | Loss: 2.5526 | LR: 3.14e-05 | 2993.3ms/step | 20.0 samples/s
+Step 42460/50462 | Loss: 2.5327 | LR: 3.13e-05 | 2989.0ms/step | 20.1 samples/s
+Step 42470/50462 | Loss: 2.5431 | LR: 3.12e-05 | 2987.3ms/step | 20.1 samples/s
+Step 42480/50462 | Loss: 2.5021 | LR: 3.11e-05 | 2992.1ms/step | 20.1 samples/s
+Step 42490/50462 | Loss: 2.5475 | LR: 3.11e-05 | 2990.0ms/step | 20.1 samples/s
+Step 42500/50462 | Loss: 2.5569 | LR: 3.10e-05 | 2992.6ms/step | 20.0 samples/s
+Validation at step 42500: Loss = 2.5537, Perplexity = 12.85
+Step 42510/50462 | Loss: 2.5537 | LR: 3.09e-05 | 4591.3ms/step | 13.1 samples/s
+Step 42520/50462 | Loss: 2.5379 | LR: 3.08e-05 | 2993.2ms/step | 20.0 samples/s
+Step 42530/50462 | Loss: 2.5310 | LR: 3.08e-05 | 2997.4ms/step | 20.0 samples/s
+Step 42540/50462 | Loss: 2.5359 | LR: 3.07e-05 | 2994.1ms/step | 20.0 samples/s
+Step 42550/50462 | Loss: 2.5619 | LR: 3.06e-05 | 2997.4ms/step | 20.0 samples/s
+Step 42560/50462 | Loss: 2.6203 | LR: 3.05e-05 | 2998.2ms/step | 20.0 samples/s
+Step 42570/50462 | Loss: 2.5497 | LR: 3.05e-05 | 2995.3ms/step | 20.0 samples/s
+Step 42580/50462 | Loss: 2.5433 | LR: 3.04e-05 | 2995.2ms/step | 20.0 samples/s
+Step 42590/50462 | Loss: 2.5372 | LR: 3.03e-05 | 2998.0ms/step | 20.0 samples/s
+Step 42600/50462 | Loss: 2.5851 | LR: 3.02e-05 | 2994.2ms/step | 20.0 samples/s
+Step 42610/50462 | Loss: 2.5338 | LR: 3.02e-05 | 2993.0ms/step | 20.0 samples/s
+Step 42620/50462 | Loss: 2.5281 | LR: 3.01e-05 | 2994.6ms/step | 20.0 samples/s
+Step 42630/50462 | Loss: 2.5588 | LR: 3.00e-05 | 2994.5ms/step | 20.0 samples/s
+Step 42640/50462 | Loss: 2.5847 | LR: 2.99e-05 | 2992.1ms/step | 20.1 samples/s
+Step 42650/50462 | Loss: 2.5542 | LR: 2.99e-05 | 2994.4ms/step | 20.0 samples/s
+Step 42660/50462 | Loss: 2.5569 | LR: 2.98e-05 | 2989.7ms/step | 20.1 samples/s
+Step 42670/50462 | Loss: 2.5274 | LR: 2.97e-05 | 2994.0ms/step | 20.0 samples/s
+Step 42680/50462 | Loss: 2.5273 | LR: 2.96e-05 | 2991.2ms/step | 20.1 samples/s
+Step 42690/50462 | Loss: 2.5459 | LR: 2.96e-05 | 2989.3ms/step | 20.1 samples/s
+Step 42700/50462 | Loss: 2.5544 | LR: 2.95e-05 | 2992.1ms/step | 20.1 samples/s
+Step 42710/50462 | Loss: 2.5354 | LR: 2.94e-05 | 2988.8ms/step | 20.1 samples/s
+Step 42720/50462 | Loss: 2.5372 | LR: 2.93e-05 | 2994.5ms/step | 20.0 samples/s
+Step 42730/50462 | Loss: 2.5370 | LR: 2.93e-05 | 2990.2ms/step | 20.1 samples/s
+Step 42740/50462 | Loss: 2.5414 | LR: 2.92e-05 | 2993.5ms/step | 20.0 samples/s
+Step 42750/50462 | Loss: 2.5382 | LR: 2.91e-05 | 2994.4ms/step | 20.0 samples/s
+Validation at step 42750: Loss = 2.5534, Perplexity = 12.85
+Step 42760/50462 | Loss: 2.5523 | LR: 2.90e-05 | 4592.3ms/step | 13.1 samples/s
+Step 42770/50462 | Loss: 2.5429 | LR: 2.90e-05 | 2995.7ms/step | 20.0 samples/s
+Step 42780/50462 | Loss: 2.5408 | LR: 2.89e-05 | 2993.7ms/step | 20.0 samples/s
+Step 42790/50462 | Loss: 2.5581 | LR: 2.88e-05 | 2998.3ms/step | 20.0 samples/s
+Step 42800/50462 | Loss: 2.5586 | LR: 2.87e-05 | 2996.1ms/step | 20.0 samples/s
+Step 42810/50462 | Loss: 2.5649 | LR: 2.87e-05 | 2993.5ms/step | 20.0 samples/s
+Step 42820/50462 | Loss: 2.5339 | LR: 2.86e-05 | 2992.3ms/step | 20.1 samples/s
+Step 42830/50462 | Loss: 2.5262 | LR: 2.85e-05 | 2998.1ms/step | 20.0 samples/s
+Step 42840/50462 | Loss: 2.5544 | LR: 2.84e-05 | 2993.6ms/step | 20.0 samples/s
+Step 42850/50462 | Loss: 2.5451 | LR: 2.84e-05 | 2996.7ms/step | 20.0 samples/s
+Step 42860/50462 | Loss: 2.5717 | LR: 2.83e-05 | 2991.7ms/step | 20.1 samples/s
+Step 42870/50462 | Loss: 2.5560 | LR: 2.82e-05 | 2996.3ms/step | 20.0 samples/s
+Step 42880/50462 | Loss: 2.5502 | LR: 2.82e-05 | 2994.4ms/step | 20.0 samples/s
+Step 42890/50462 | Loss: 2.5350 | LR: 2.81e-05 | 2993.8ms/step | 20.0 samples/s
+Step 42900/50462 | Loss: 2.5435 | LR: 2.80e-05 | 2991.7ms/step | 20.1 samples/s
+Step 42910/50462 | Loss: 2.5733 | LR: 2.79e-05 | 2993.9ms/step | 20.0 samples/s
+Step 42920/50462 | Loss: 2.5323 | LR: 2.79e-05 | 2997.7ms/step | 20.0 samples/s
+Step 42930/50462 | Loss: 2.5315 | LR: 2.78e-05 | 2992.3ms/step | 20.1 samples/s
+Step 42940/50462 | Loss: 2.5211 | LR: 2.77e-05 | 2997.9ms/step | 20.0 samples/s
+Step 42950/50462 | Loss: 2.5713 | LR: 2.76e-05 | 2997.2ms/step | 20.0 samples/s
+Step 42960/50462 | Loss: 2.5535 | LR: 2.76e-05 | 2995.0ms/step | 20.0 samples/s
+Step 42970/50462 | Loss: 2.5262 | LR: 2.75e-05 | 2995.5ms/step | 20.0 samples/s
+Step 42980/50462 | Loss: 2.5393 | LR: 2.74e-05 | 2994.7ms/step | 20.0 samples/s
+Step 42990/50462 | Loss: 2.5465 | LR: 2.74e-05 | 2997.6ms/step | 20.0 samples/s
+Step 43000/50462 | Loss: 2.5248 | LR: 2.73e-05 | 2991.7ms/step | 20.1 samples/s
+Validation at step 43000: Loss = 2.5537, Perplexity = 12.85
+Saved checkpoint to checkpoints/run_20260609_063834/checkpoint-43000
+Step 43010/50462 | Loss: 2.5654 | LR: 2.72e-05 | 5640.5ms/step | 10.6 samples/s
+Step 43020/50462 | Loss: 2.5385 | LR: 2.71e-05 | 2997.8ms/step | 20.0 samples/s
+Step 43030/50462 | Loss: 2.5444 | LR: 2.71e-05 | 3000.4ms/step | 20.0 samples/s
+Step 43040/50462 | Loss: 2.5310 | LR: 2.70e-05 | 2999.3ms/step | 20.0 samples/s
+Step 43050/50462 | Loss: 2.5520 | LR: 2.69e-05 | 2996.6ms/step | 20.0 samples/s
+Step 43060/50462 | Loss: 2.5268 | LR: 2.69e-05 | 2997.7ms/step | 20.0 samples/s
+Step 43070/50462 | Loss: 2.5447 | LR: 2.68e-05 | 2998.4ms/step | 20.0 samples/s
+Step 43080/50462 | Loss: 2.5228 | LR: 2.67e-05 | 3000.2ms/step | 20.0 samples/s
+Step 43090/50462 | Loss: 2.5203 | LR: 2.66e-05 | 2993.3ms/step | 20.0 samples/s
+Step 43100/50462 | Loss: 2.5185 | LR: 2.66e-05 | 2997.7ms/step | 20.0 samples/s
+Step 43110/50462 | Loss: 2.5276 | LR: 2.65e-05 | 2999.0ms/step | 20.0 samples/s
+Step 43120/50462 | Loss: 2.5422 | LR: 2.64e-05 | 3000.0ms/step | 20.0 samples/s
+Step 43130/50462 | Loss: 2.5531 | LR: 2.64e-05 | 2998.6ms/step | 20.0 samples/s
+Step 43140/50462 | Loss: 2.5231 | LR: 2.63e-05 | 2999.7ms/step | 20.0 samples/s
+Step 43150/50462 | Loss: 2.5241 | LR: 2.62e-05 | 2999.7ms/step | 20.0 samples/s
+Step 43160/50462 | Loss: 2.5633 | LR: 2.61e-05 | 3000.4ms/step | 20.0 samples/s
+Step 43170/50462 | Loss: 2.5547 | LR: 2.61e-05 | 2997.2ms/step | 20.0 samples/s
+Step 43180/50462 | Loss: 2.5083 | LR: 2.60e-05 | 2998.7ms/step | 20.0 samples/s
+Step 43190/50462 | Loss: 2.5639 | LR: 2.59e-05 | 2996.8ms/step | 20.0 samples/s
+Step 43200/50462 | Loss: 2.5419 | LR: 2.59e-05 | 2993.2ms/step | 20.0 samples/s
+Step 43210/50462 | Loss: 2.5471 | LR: 2.58e-05 | 2998.2ms/step | 20.0 samples/s
+Step 43220/50462 | Loss: 2.5553 | LR: 2.57e-05 | 2997.9ms/step | 20.0 samples/s
+Step 43230/50462 | Loss: 2.5404 | LR: 2.57e-05 | 2997.9ms/step | 20.0 samples/s
+Step 43240/50462 | Loss: 2.5614 | LR: 2.56e-05 | 2993.6ms/step | 20.0 samples/s
+Step 43250/50462 | Loss: 2.5121 | LR: 2.55e-05 | 2996.5ms/step | 20.0 samples/s
+Validation at step 43250: Loss = 2.5537, Perplexity = 12.85
+Step 43260/50462 | Loss: 2.5154 | LR: 2.54e-05 | 4587.7ms/step | 13.1 samples/s
+Step 43270/50462 | Loss: 2.5253 | LR: 2.54e-05 | 2996.7ms/step | 20.0 samples/s
+Step 43280/50462 | Loss: 2.5483 | LR: 2.53e-05 | 2995.0ms/step | 20.0 samples/s
+Step 43290/50462 | Loss: 2.5261 | LR: 2.52e-05 | 2990.9ms/step | 20.1 samples/s
+Step 43300/50462 | Loss: 2.5703 | LR: 2.52e-05 | 2995.1ms/step | 20.0 samples/s
+Step 43310/50462 | Loss: 2.5350 | LR: 2.51e-05 | 2997.0ms/step | 20.0 samples/s
+Step 43320/50462 | Loss: 2.5389 | LR: 2.50e-05 | 2997.8ms/step | 20.0 samples/s
+Step 43330/50462 | Loss: 2.5244 | LR: 2.50e-05 | 2991.8ms/step | 20.1 samples/s
+Step 43340/50462 | Loss: 2.5358 | LR: 2.49e-05 | 2994.1ms/step | 20.0 samples/s
+Step 43350/50462 | Loss: 2.5216 | LR: 2.48e-05 | 2996.1ms/step | 20.0 samples/s
+Step 43360/50462 | Loss: 2.5544 | LR: 2.48e-05 | 2993.9ms/step | 20.0 samples/s
+Step 43370/50462 | Loss: 2.5460 | LR: 2.47e-05 | 2994.0ms/step | 20.0 samples/s
+Step 43380/50462 | Loss: 2.5246 | LR: 2.46e-05 | 2997.6ms/step | 20.0 samples/s
+Step 43390/50462 | Loss: 2.5542 | LR: 2.46e-05 | 2992.2ms/step | 20.1 samples/s
+Step 43400/50462 | Loss: 2.5366 | LR: 2.45e-05 | 2994.8ms/step | 20.0 samples/s
+Step 43410/50462 | Loss: 2.5572 | LR: 2.44e-05 | 2996.0ms/step | 20.0 samples/s
+Step 43420/50462 | Loss: 2.5350 | LR: 2.44e-05 | 2995.3ms/step | 20.0 samples/s
+Step 43430/50462 | Loss: 2.5412 | LR: 2.43e-05 | 2996.6ms/step | 20.0 samples/s
+Step 43440/50462 | Loss: 2.5370 | LR: 2.42e-05 | 2992.6ms/step | 20.0 samples/s
+Step 43450/50462 | Loss: 2.5618 | LR: 2.41e-05 | 2990.5ms/step | 20.1 samples/s
+Step 43460/50462 | Loss: 2.5505 | LR: 2.41e-05 | 2996.8ms/step | 20.0 samples/s
+Step 43470/50462 | Loss: 2.5640 | LR: 2.40e-05 | 2993.1ms/step | 20.0 samples/s
+Step 43480/50462 | Loss: 2.5648 | LR: 2.39e-05 | 2992.2ms/step | 20.1 samples/s
+Step 43490/50462 | Loss: 2.5325 | LR: 2.39e-05 | 2995.4ms/step | 20.0 samples/s
+Step 43500/50462 | Loss: 2.5496 | LR: 2.38e-05 | 2996.9ms/step | 20.0 samples/s
+Validation at step 43500: Loss = 2.5534, Perplexity = 12.85
+Step 43510/50462 | Loss: 2.5537 | LR: 2.37e-05 | 4587.8ms/step | 13.1 samples/s
+Step 43520/50462 | Loss: 2.5151 | LR: 2.37e-05 | 2998.6ms/step | 20.0 samples/s
+Step 43530/50462 | Loss: 2.5377 | LR: 2.36e-05 | 2998.1ms/step | 20.0 samples/s
+Step 43540/50462 | Loss: 2.5345 | LR: 2.35e-05 | 2989.4ms/step | 20.1 samples/s
+Step 43550/50462 | Loss: 2.5428 | LR: 2.35e-05 | 2990.9ms/step | 20.1 samples/s
+Step 43560/50462 | Loss: 2.5227 | LR: 2.34e-05 | 2991.3ms/step | 20.1 samples/s
+Step 43570/50462 | Loss: 2.5420 | LR: 2.33e-05 | 2994.7ms/step | 20.0 samples/s
+Step 43580/50462 | Loss: 2.5701 | LR: 2.33e-05 | 2995.4ms/step | 20.0 samples/s
+Step 43590/50462 | Loss: 2.5276 | LR: 2.32e-05 | 2994.2ms/step | 20.0 samples/s
+Step 43600/50462 | Loss: 2.5622 | LR: 2.31e-05 | 2994.0ms/step | 20.0 samples/s
+Step 43610/50462 | Loss: 2.5476 | LR: 2.31e-05 | 2994.6ms/step | 20.0 samples/s
+Step 43620/50462 | Loss: 2.5303 | LR: 2.30e-05 | 2993.0ms/step | 20.0 samples/s
+Step 43630/50462 | Loss: 2.5677 | LR: 2.29e-05 | 2993.9ms/step | 20.0 samples/s
+Step 43640/50462 | Loss: 2.5312 | LR: 2.29e-05 | 2990.5ms/step | 20.1 samples/s
+Step 43650/50462 | Loss: 2.5508 | LR: 2.28e-05 | 2992.5ms/step | 20.1 samples/s
+Step 43660/50462 | Loss: 2.5628 | LR: 2.27e-05 | 2992.0ms/step | 20.1 samples/s
+Step 43670/50462 | Loss: 2.5498 | LR: 2.27e-05 | 2993.0ms/step | 20.0 samples/s
+Step 43680/50462 | Loss: 2.5554 | LR: 2.26e-05 | 2993.1ms/step | 20.0 samples/s
+Step 43690/50462 | Loss: 2.5483 | LR: 2.25e-05 | 2993.7ms/step | 20.0 samples/s
+Step 43700/50462 | Loss: 2.5639 | LR: 2.25e-05 | 2992.4ms/step | 20.1 samples/s
+Step 43710/50462 | Loss: 2.5695 | LR: 2.24e-05 | 2993.1ms/step | 20.0 samples/s
+Step 43720/50462 | Loss: 2.5254 | LR: 2.24e-05 | 2987.2ms/step | 20.1 samples/s
+Step 43730/50462 | Loss: 2.5156 | LR: 2.23e-05 | 2995.6ms/step | 20.0 samples/s
+Step 43740/50462 | Loss: 2.5391 | LR: 2.22e-05 | 2993.9ms/step | 20.0 samples/s
+Step 43750/50462 | Loss: 2.5400 | LR: 2.22e-05 | 2999.0ms/step | 20.0 samples/s
+Validation at step 43750: Loss = 2.5534, Perplexity = 12.85
+Step 43760/50462 | Loss: 2.5459 | LR: 2.21e-05 | 4585.1ms/step | 13.1 samples/s
+Step 43770/50462 | Loss: 2.5300 | LR: 2.20e-05 | 2995.2ms/step | 20.0 samples/s
+Step 43780/50462 | Loss: 2.5396 | LR: 2.20e-05 | 2990.1ms/step | 20.1 samples/s
+Step 43790/50462 | Loss: 2.5449 | LR: 2.19e-05 | 2991.0ms/step | 20.1 samples/s
+Step 43800/50462 | Loss: 2.5605 | LR: 2.18e-05 | 2997.0ms/step | 20.0 samples/s
+Step 43810/50462 | Loss: 2.5039 | LR: 2.18e-05 | 2994.1ms/step | 20.0 samples/s
+Step 43820/50462 | Loss: 2.5084 | LR: 2.17e-05 | 2995.8ms/step | 20.0 samples/s
+Step 43830/50462 | Loss: 2.5514 | LR: 2.16e-05 | 2992.6ms/step | 20.0 samples/s
+Step 43840/50462 | Loss: 2.5673 | LR: 2.16e-05 | 2992.0ms/step | 20.1 samples/s
+Step 43850/50462 | Loss: 2.5622 | LR: 2.15e-05 | 2988.6ms/step | 20.1 samples/s
+Step 43860/50462 | Loss: 2.5445 | LR: 2.14e-05 | 3540.5ms/step | 16.9 samples/s
+Step 43870/50462 | Loss: 2.5338 | LR: 2.14e-05 | 2984.3ms/step | 20.1 samples/s
+Step 43880/50462 | Loss: 2.5550 | LR: 2.13e-05 | 3612.0ms/step | 16.6 samples/s
+Step 43890/50462 | Loss: 2.5451 | LR: 2.13e-05 | 3516.2ms/step | 17.1 samples/s
+Step 43900/50462 | Loss: 2.5992 | LR: 2.12e-05 | 2986.7ms/step | 20.1 samples/s
+Step 43910/50462 | Loss: 2.5639 | LR: 2.11e-05 | 2995.6ms/step | 20.0 samples/s
+Step 43920/50462 | Loss: 2.5404 | LR: 2.11e-05 | 2993.3ms/step | 20.0 samples/s
+Step 43930/50462 | Loss: 2.5662 | LR: 2.10e-05 | 2995.0ms/step | 20.0 samples/s
+Step 43940/50462 | Loss: 2.5408 | LR: 2.09e-05 | 2995.8ms/step | 20.0 samples/s
+Step 43950/50462 | Loss: 2.5527 | LR: 2.09e-05 | 2990.2ms/step | 20.1 samples/s
+Step 43960/50462 | Loss: 2.5356 | LR: 2.08e-05 | 2988.7ms/step | 20.1 samples/s
+Step 43970/50462 | Loss: 2.5304 | LR: 2.07e-05 | 2993.6ms/step | 20.0 samples/s
+Step 43980/50462 | Loss: 2.5233 | LR: 2.07e-05 | 2988.2ms/step | 20.1 samples/s
+Step 43990/50462 | Loss: 2.5595 | LR: 2.06e-05 | 2992.2ms/step | 20.1 samples/s
+Step 44000/50462 | Loss: 2.5656 | LR: 2.06e-05 | 2997.8ms/step | 20.0 samples/s
+Validation at step 44000: Loss = 2.5530, Perplexity = 12.85
+Saved checkpoint to checkpoints/run_20260609_063834/checkpoint-44000
+Step 44010/50462 | Loss: 2.5497 | LR: 2.05e-05 | 5657.0ms/step | 10.6 samples/s
+Step 44020/50462 | Loss: 2.5412 | LR: 2.04e-05 | 2993.3ms/step | 20.0 samples/s
+Step 44030/50462 | Loss: 2.5381 | LR: 2.04e-05 | 2990.5ms/step | 20.1 samples/s
+Step 44040/50462 | Loss: 2.5311 | LR: 2.03e-05 | 2989.3ms/step | 20.1 samples/s
+Step 44050/50462 | Loss: 2.5665 | LR: 2.02e-05 | 2993.2ms/step | 20.0 samples/s
+Step 44060/50462 | Loss: 2.5641 | LR: 2.02e-05 | 2991.0ms/step | 20.1 samples/s
+Step 44070/50462 | Loss: 2.5642 | LR: 2.01e-05 | 2993.3ms/step | 20.0 samples/s
+Step 44080/50462 | Loss: 2.5459 | LR: 2.01e-05 | 2990.4ms/step | 20.1 samples/s
+Step 44090/50462 | Loss: 2.5145 | LR: 2.00e-05 | 2991.5ms/step | 20.1 samples/s
+Step 44100/50462 | Loss: 2.5348 | LR: 1.99e-05 | 2990.9ms/step | 20.1 samples/s
+Step 44110/50462 | Loss: 2.5363 | LR: 1.99e-05 | 2992.3ms/step | 20.1 samples/s
+Step 44120/50462 | Loss: 2.5534 | LR: 1.98e-05 | 2991.2ms/step | 20.1 samples/s
+Step 44130/50462 | Loss: 2.5578 | LR: 1.97e-05 | 2993.1ms/step | 20.0 samples/s
+Step 44140/50462 | Loss: 2.5283 | LR: 1.97e-05 | 2991.4ms/step | 20.1 samples/s
+Step 44150/50462 | Loss: 2.5232 | LR: 1.96e-05 | 2989.4ms/step | 20.1 samples/s
+Step 44160/50462 | Loss: 2.5523 | LR: 1.96e-05 | 2992.4ms/step | 20.1 samples/s
+Step 44170/50462 | Loss: 2.5580 | LR: 1.95e-05 | 2994.1ms/step | 20.0 samples/s
+Step 44180/50462 | Loss: 2.5235 | LR: 1.94e-05 | 2994.1ms/step | 20.0 samples/s
+Step 44190/50462 | Loss: 2.5295 | LR: 1.94e-05 | 2990.9ms/step | 20.1 samples/s
+Step 44200/50462 | Loss: 2.5504 | LR: 1.93e-05 | 2989.1ms/step | 20.1 samples/s
+Step 44210/50462 | Loss: 2.5395 | LR: 1.93e-05 | 2991.3ms/step | 20.1 samples/s
+Step 44220/50462 | Loss: 2.5441 | LR: 1.92e-05 | 2992.8ms/step | 20.0 samples/s
+Step 44230/50462 | Loss: 2.5571 | LR: 1.91e-05 | 2996.1ms/step | 20.0 samples/s
+Step 44240/50462 | Loss: 2.5449 | LR: 1.91e-05 | 2997.3ms/step | 20.0 samples/s
+Step 44250/50462 | Loss: 2.5583 | LR: 1.90e-05 | 2993.7ms/step | 20.0 samples/s
+Validation at step 44250: Loss = 2.5532, Perplexity = 12.85
+Step 44260/50462 | Loss: 2.5507 | LR: 1.90e-05 | 4591.0ms/step | 13.1 samples/s
+Step 44270/50462 | Loss: 2.5239 | LR: 1.89e-05 | 2999.9ms/step | 20.0 samples/s
+Step 44280/50462 | Loss: 2.5457 | LR: 1.88e-05 | 3000.3ms/step | 20.0 samples/s
+Step 44290/50462 | Loss: 2.5434 | LR: 1.88e-05 | 2999.5ms/step | 20.0 samples/s
+Step 44300/50462 | Loss: 2.4946 | LR: 1.87e-05 | 2990.0ms/step | 20.1 samples/s
+Step 44310/50462 | Loss: 2.5233 | LR: 1.87e-05 | 2994.2ms/step | 20.0 samples/s
+Step 44320/50462 | Loss: 2.5563 | LR: 1.86e-05 | 2998.7ms/step | 20.0 samples/s
+Step 44330/50462 | Loss: 2.5466 | LR: 1.85e-05 | 2995.9ms/step | 20.0 samples/s
+Step 44340/50462 | Loss: 2.5418 | LR: 1.85e-05 | 2996.6ms/step | 20.0 samples/s
+Step 44350/50462 | Loss: 2.5488 | LR: 1.84e-05 | 2994.9ms/step | 20.0 samples/s
+Step 44360/50462 | Loss: 2.5206 | LR: 1.84e-05 | 3000.1ms/step | 20.0 samples/s
+Step 44370/50462 | Loss: 2.5456 | LR: 1.83e-05 | 2999.8ms/step | 20.0 samples/s
+Step 44380/50462 | Loss: 2.5369 | LR: 1.82e-05 | 2999.1ms/step | 20.0 samples/s
+Step 44390/50462 | Loss: 2.5571 | LR: 1.82e-05 | 2992.4ms/step | 20.1 samples/s
+Step 44400/50462 | Loss: 2.5235 | LR: 1.81e-05 | 2993.4ms/step | 20.0 samples/s
+Step 44410/50462 | Loss: 2.5319 | LR: 1.81e-05 | 2993.6ms/step | 20.0 samples/s
+Step 44420/50462 | Loss: 2.5423 | LR: 1.80e-05 | 2996.1ms/step | 20.0 samples/s
+Step 44430/50462 | Loss: 2.5502 | LR: 1.79e-05 | 2997.9ms/step | 20.0 samples/s
+Step 44440/50462 | Loss: 2.5395 | LR: 1.79e-05 | 2998.1ms/step | 20.0 samples/s
+Step 44450/50462 | Loss: 2.5277 | LR: 1.78e-05 | 3000.4ms/step | 20.0 samples/s
+Step 44460/50462 | Loss: 2.5435 | LR: 1.78e-05 | 2999.2ms/step | 20.0 samples/s
+Step 44470/50462 | Loss: 2.5568 | LR: 1.77e-05 | 3000.5ms/step | 20.0 samples/s
+Step 44480/50462 | Loss: 2.5706 | LR: 1.77e-05 | 3000.1ms/step | 20.0 samples/s
+Step 44490/50462 | Loss: 2.5667 | LR: 1.76e-05 | 2999.4ms/step | 20.0 samples/s
+Step 44500/50462 | Loss: 2.5282 | LR: 1.75e-05 | 2997.9ms/step | 20.0 samples/s
+Validation at step 44500: Loss = 2.5530, Perplexity = 12.85
+Step 44510/50462 | Loss: 2.5178 | LR: 1.75e-05 | 4592.0ms/step | 13.1 samples/s
+Step 44520/50462 | Loss: 2.5167 | LR: 1.74e-05 | 2998.5ms/step | 20.0 samples/s
+Step 44530/50462 | Loss: 2.5421 | LR: 1.74e-05 | 3000.0ms/step | 20.0 samples/s
+Step 44540/50462 | Loss: 2.5313 | LR: 1.73e-05 | 3000.1ms/step | 20.0 samples/s
+Step 44550/50462 | Loss: 2.5458 | LR: 1.72e-05 | 2998.1ms/step | 20.0 samples/s
+Step 44560/50462 | Loss: 2.5492 | LR: 1.72e-05 | 2999.6ms/step | 20.0 samples/s
+Step 44570/50462 | Loss: 2.5376 | LR: 1.71e-05 | 2996.7ms/step | 20.0 samples/s
+Step 44580/50462 | Loss: 2.5151 | LR: 1.71e-05 | 2999.9ms/step | 20.0 samples/s
+Step 44590/50462 | Loss: 2.5339 | LR: 1.70e-05 | 2997.8ms/step | 20.0 samples/s
+Step 44600/50462 | Loss: 2.5571 | LR: 1.70e-05 | 3000.4ms/step | 20.0 samples/s
+Step 44610/50462 | Loss: 2.5386 | LR: 1.69e-05 | 2999.6ms/step | 20.0 samples/s
+Step 44620/50462 | Loss: 2.5559 | LR: 1.68e-05 | 2996.0ms/step | 20.0 samples/s
+Step 44630/50462 | Loss: 2.5402 | LR: 1.68e-05 | 2994.9ms/step | 20.0 samples/s
+Step 44640/50462 | Loss: 2.5325 | LR: 1.67e-05 | 2999.4ms/step | 20.0 samples/s
+Step 44650/50462 | Loss: 2.5213 | LR: 1.67e-05 | 2995.9ms/step | 20.0 samples/s
+Step 44660/50462 | Loss: 2.5585 | LR: 1.66e-05 | 2994.5ms/step | 20.0 samples/s
+Step 44670/50462 | Loss: 2.5678 | LR: 1.66e-05 | 2999.2ms/step | 20.0 samples/s
+Step 44680/50462 | Loss: 2.5542 | LR: 1.65e-05 | 2998.1ms/step | 20.0 samples/s
+Step 44690/50462 | Loss: 2.5502 | LR: 1.64e-05 | 2995.9ms/step | 20.0 samples/s
+Step 44700/50462 | Loss: 2.5563 | LR: 1.64e-05 | 2996.5ms/step | 20.0 samples/s
+Step 44710/50462 | Loss: 2.5395 | LR: 1.63e-05 | 2997.4ms/step | 20.0 samples/s
+Step 44720/50462 | Loss: 2.5413 | LR: 1.63e-05 | 2992.8ms/step | 20.0 samples/s
+Step 44730/50462 | Loss: 2.5297 | LR: 1.62e-05 | 2999.6ms/step | 20.0 samples/s
+Step 44740/50462 | Loss: 2.5246 | LR: 1.62e-05 | 2998.0ms/step | 20.0 samples/s
+Step 44750/50462 | Loss: 2.5495 | LR: 1.61e-05 | 2996.1ms/step | 20.0 samples/s
+Validation at step 44750: Loss = 2.5528, Perplexity = 12.84
+Step 44760/50462 | Loss: 2.5655 | LR: 1.61e-05 | 4596.1ms/step | 13.1 samples/s
+Step 44770/50462 | Loss: 2.5150 | LR: 1.60e-05 | 2996.2ms/step | 20.0 samples/s
+Step 44780/50462 | Loss: 2.5435 | LR: 1.59e-05 | 2994.0ms/step | 20.0 samples/s
+Step 44790/50462 | Loss: 2.5308 | LR: 1.59e-05 | 2997.8ms/step | 20.0 samples/s
+Step 44800/50462 | Loss: 2.5324 | LR: 1.58e-05 | 3000.4ms/step | 20.0 samples/s
+Step 44810/50462 | Loss: 2.5605 | LR: 1.58e-05 | 2995.9ms/step | 20.0 samples/s
+Step 44820/50462 | Loss: 2.5486 | LR: 1.57e-05 | 2999.7ms/step | 20.0 samples/s
+Step 44830/50462 | Loss: 2.5710 | LR: 1.57e-05 | 3000.5ms/step | 20.0 samples/s
+Step 44840/50462 | Loss: 2.5270 | LR: 1.56e-05 | 2997.6ms/step | 20.0 samples/s
+Step 44850/50462 | Loss: 2.5324 | LR: 1.56e-05 | 2998.7ms/step | 20.0 samples/s
+Step 44860/50462 | Loss: 2.5307 | LR: 1.55e-05 | 2999.0ms/step | 20.0 samples/s
+Step 44870/50462 | Loss: 2.5198 | LR: 1.54e-05 | 2993.6ms/step | 20.0 samples/s
+Step 44880/50462 | Loss: 2.5439 | LR: 1.54e-05 | 2997.2ms/step | 20.0 samples/s
+Step 44890/50462 | Loss: 2.5058 | LR: 1.53e-05 | 2997.0ms/step | 20.0 samples/s
+Step 44900/50462 | Loss: 2.5062 | LR: 1.53e-05 | 2993.2ms/step | 20.0 samples/s
+Step 44910/50462 | Loss: 2.5267 | LR: 1.52e-05 | 2997.9ms/step | 20.0 samples/s
+Step 44920/50462 | Loss: 2.5305 | LR: 1.52e-05 | 2997.8ms/step | 20.0 samples/s
+Step 44930/50462 | Loss: 2.5353 | LR: 1.51e-05 | 2999.9ms/step | 20.0 samples/s
+Step 44940/50462 | Loss: 2.5364 | LR: 1.51e-05 | 2994.2ms/step | 20.0 samples/s
+Step 44950/50462 | Loss: 2.4957 | LR: 1.50e-05 | 2998.4ms/step | 20.0 samples/s
+Step 44960/50462 | Loss: 2.5741 | LR: 1.50e-05 | 2995.9ms/step | 20.0 samples/s
+Step 44970/50462 | Loss: 2.5231 | LR: 1.49e-05 | 2995.8ms/step | 20.0 samples/s
+Step 44980/50462 | Loss: 2.5369 | LR: 1.49e-05 | 2995.6ms/step | 20.0 samples/s
+Step 44990/50462 | Loss: 2.5641 | LR: 1.48e-05 | 2996.2ms/step | 20.0 samples/s
+Step 45000/50462 | Loss: 2.5555 | LR: 1.47e-05 | 2996.3ms/step | 20.0 samples/s
+Validation at step 45000: Loss = 2.5529, Perplexity = 12.84
+Saved checkpoint to checkpoints/run_20260609_063834/checkpoint-45000
+Step 45010/50462 | Loss: 2.5475 | LR: 1.47e-05 | 5703.4ms/step | 10.5 samples/s
+Step 45020/50462 | Loss: 2.5312 | LR: 1.46e-05 | 2992.9ms/step | 20.0 samples/s
+Step 45030/50462 | Loss: 2.5368 | LR: 1.46e-05 | 3000.2ms/step | 20.0 samples/s
+Step 45040/50462 | Loss: 2.5519 | LR: 1.45e-05 | 2996.3ms/step | 20.0 samples/s
+Step 45050/50462 | Loss: 2.5245 | LR: 1.45e-05 | 2999.7ms/step | 20.0 samples/s
+Step 45060/50462 | Loss: 2.5491 | LR: 1.44e-05 | 2999.8ms/step | 20.0 samples/s
+Step 45070/50462 | Loss: 2.5418 | LR: 1.44e-05 | 3000.0ms/step | 20.0 samples/s
+Step 45080/50462 | Loss: 2.5336 | LR: 1.43e-05 | 2998.4ms/step | 20.0 samples/s
+Step 45090/50462 | Loss: 2.5336 | LR: 1.43e-05 | 2999.6ms/step | 20.0 samples/s
+Step 45100/50462 | Loss: 2.5676 | LR: 1.42e-05 | 2994.4ms/step | 20.0 samples/s
+Step 45110/50462 | Loss: 2.5675 | LR: 1.42e-05 | 2993.3ms/step | 20.0 samples/s
+Step 45120/50462 | Loss: 2.5503 | LR: 1.41e-05 | 2998.0ms/step | 20.0 samples/s
+Step 45130/50462 | Loss: 2.5578 | LR: 1.41e-05 | 2998.4ms/step | 20.0 samples/s
+Step 45140/50462 | Loss: 2.5278 | LR: 1.40e-05 | 2993.6ms/step | 20.0 samples/s
+Step 45150/50462 | Loss: 2.5525 | LR: 1.40e-05 | 2995.0ms/step | 20.0 samples/s
+Step 45160/50462 | Loss: 2.5588 | LR: 1.39e-05 | 2991.4ms/step | 20.1 samples/s
+Step 45170/50462 | Loss: 2.5204 | LR: 1.39e-05 | 2994.8ms/step | 20.0 samples/s
+Step 45180/50462 | Loss: 2.5483 | LR: 1.38e-05 | 2999.8ms/step | 20.0 samples/s
+Step 45190/50462 | Loss: 2.5353 | LR: 1.37e-05 | 2995.7ms/step | 20.0 samples/s
+Step 45200/50462 | Loss: 2.5601 | LR: 1.37e-05 | 2997.8ms/step | 20.0 samples/s
+Step 45210/50462 | Loss: 2.5432 | LR: 1.36e-05 | 2998.3ms/step | 20.0 samples/s
+Step 45220/50462 | Loss: 2.5284 | LR: 1.36e-05 | 2995.9ms/step | 20.0 samples/s
+Step 45230/50462 | Loss: 2.5240 | LR: 1.35e-05 | 2994.2ms/step | 20.0 samples/s
+Step 45240/50462 | Loss: 2.5578 | LR: 1.35e-05 | 2999.7ms/step | 20.0 samples/s
+Step 45250/50462 | Loss: 2.5364 | LR: 1.34e-05 | 2999.6ms/step | 20.0 samples/s
+Validation at step 45250: Loss = 2.5533, Perplexity = 12.85
+Step 45260/50462 | Loss: 2.5485 | LR: 1.34e-05 | 4592.9ms/step | 13.1 samples/s
+Step 45270/50462 | Loss: 2.5565 | LR: 1.33e-05 | 2999.7ms/step | 20.0 samples/s
+Step 45280/50462 | Loss: 2.5146 | LR: 1.33e-05 | 2996.0ms/step | 20.0 samples/s
+Step 45290/50462 | Loss: 2.5407 | LR: 1.32e-05 | 2997.6ms/step | 20.0 samples/s
+Step 45300/50462 | Loss: 2.5591 | LR: 1.32e-05 | 2992.2ms/step | 20.1 samples/s
+Step 45310/50462 | Loss: 2.5287 | LR: 1.31e-05 | 2998.5ms/step | 20.0 samples/s
+Step 45320/50462 | Loss: 2.5361 | LR: 1.31e-05 | 2999.8ms/step | 20.0 samples/s
+Step 45330/50462 | Loss: 2.5511 | LR: 1.30e-05 | 2999.9ms/step | 20.0 samples/s
+Step 45340/50462 | Loss: 2.5440 | LR: 1.30e-05 | 2997.5ms/step | 20.0 samples/s
+Step 45350/50462 | Loss: 2.5421 | LR: 1.29e-05 | 2996.6ms/step | 20.0 samples/s
+Step 45360/50462 | Loss: 2.5607 | LR: 1.29e-05 | 2995.7ms/step | 20.0 samples/s
+Step 45370/50462 | Loss: 2.5329 | LR: 1.28e-05 | 2994.6ms/step | 20.0 samples/s
+Step 45380/50462 | Loss: 2.5300 | LR: 1.28e-05 | 2999.5ms/step | 20.0 samples/s
+Step 45390/50462 | Loss: 2.5362 | LR: 1.27e-05 | 3000.5ms/step | 20.0 samples/s
+Step 45400/50462 | Loss: 2.5212 | LR: 1.27e-05 | 2999.9ms/step | 20.0 samples/s
+Step 45410/50462 | Loss: 2.5656 | LR: 1.26e-05 | 2996.1ms/step | 20.0 samples/s
+Step 45420/50462 | Loss: 2.5343 | LR: 1.26e-05 | 2998.9ms/step | 20.0 samples/s
+Step 45430/50462 | Loss: 2.5102 | LR: 1.25e-05 | 2991.5ms/step | 20.1 samples/s
+Step 45440/50462 | Loss: 2.5528 | LR: 1.25e-05 | 2999.7ms/step | 20.0 samples/s
+Step 45450/50462 | Loss: 2.5437 | LR: 1.24e-05 | 3000.0ms/step | 20.0 samples/s
+Step 45460/50462 | Loss: 2.5305 | LR: 1.24e-05 | 2998.6ms/step | 20.0 samples/s
+Step 45470/50462 | Loss: 2.5413 | LR: 1.23e-05 | 2997.1ms/step | 20.0 samples/s
+Step 45480/50462 | Loss: 2.5304 | LR: 1.23e-05 | 2998.3ms/step | 20.0 samples/s
+Step 45490/50462 | Loss: 2.5516 | LR: 1.22e-05 | 3000.0ms/step | 20.0 samples/s
+Step 45500/50462 | Loss: 2.5530 | LR: 1.22e-05 | 2993.7ms/step | 20.0 samples/s
+Validation at step 45500: Loss = 2.5528, Perplexity = 12.84
+Step 45510/50462 | Loss: 2.5269 | LR: 1.21e-05 | 4606.7ms/step | 13.0 samples/s
+Step 45520/50462 | Loss: 2.5724 | LR: 1.21e-05 | 2999.7ms/step | 20.0 samples/s
+Step 45530/50462 | Loss: 2.5189 | LR: 1.20e-05 | 2999.6ms/step | 20.0 samples/s
+Step 45540/50462 | Loss: 2.5676 | LR: 1.20e-05 | 3000.5ms/step | 20.0 samples/s
+Step 45550/50462 | Loss: 2.5266 | LR: 1.19e-05 | 3005.3ms/step | 20.0 samples/s
+Step 45560/50462 | Loss: 2.5507 | LR: 1.19e-05 | 3000.1ms/step | 20.0 samples/s
+Step 45570/50462 | Loss: 2.5557 | LR: 1.19e-05 | 2998.4ms/step | 20.0 samples/s
+Step 45580/50462 | Loss: 2.5332 | LR: 1.18e-05 | 2998.0ms/step | 20.0 samples/s
+Step 45590/50462 | Loss: 2.5213 | LR: 1.18e-05 | 3004.3ms/step | 20.0 samples/s
+Step 45600/50462 | Loss: 2.5295 | LR: 1.17e-05 | 3000.1ms/step | 20.0 samples/s
+Step 45610/50462 | Loss: 2.5091 | LR: 1.17e-05 | 2999.1ms/step | 20.0 samples/s
+Step 45620/50462 | Loss: 2.5407 | LR: 1.16e-05 | 2998.7ms/step | 20.0 samples/s
+Step 45630/50462 | Loss: 2.5582 | LR: 1.16e-05 | 2999.6ms/step | 20.0 samples/s
+Step 45640/50462 | Loss: 2.5359 | LR: 1.15e-05 | 3000.7ms/step | 20.0 samples/s
+Step 45650/50462 | Loss: 2.5613 | LR: 1.15e-05 | 2999.6ms/step | 20.0 samples/s
+Step 45660/50462 | Loss: 2.5454 | LR: 1.14e-05 | 2998.4ms/step | 20.0 samples/s
+Step 45670/50462 | Loss: 2.5117 | LR: 1.14e-05 | 3000.0ms/step | 20.0 samples/s
+Step 45680/50462 | Loss: 2.5318 | LR: 1.13e-05 | 3000.0ms/step | 20.0 samples/s
+Step 45690/50462 | Loss: 2.5524 | LR: 1.13e-05 | 2998.0ms/step | 20.0 samples/s
+Step 45700/50462 | Loss: 2.5457 | LR: 1.12e-05 | 2999.9ms/step | 20.0 samples/s
+Step 45710/50462 | Loss: 2.5468 | LR: 1.12e-05 | 3002.3ms/step | 20.0 samples/s
+Step 45720/50462 | Loss: 2.5380 | LR: 1.11e-05 | 2999.9ms/step | 20.0 samples/s
+Step 45730/50462 | Loss: 2.5214 | LR: 1.11e-05 | 2999.9ms/step | 20.0 samples/s
+Step 45740/50462 | Loss: 2.5346 | LR: 1.10e-05 | 3000.3ms/step | 20.0 samples/s
+Step 45750/50462 | Loss: 2.5518 | LR: 1.10e-05 | 2999.7ms/step | 20.0 samples/s
+Validation at step 45750: Loss = 2.5531, Perplexity = 12.85
+Step 45760/50462 | Loss: 2.5050 | LR: 1.10e-05 | 4592.1ms/step | 13.1 samples/s
+Step 45770/50462 | Loss: 2.5390 | LR: 1.09e-05 | 3001.5ms/step | 20.0 samples/s
+Step 45780/50462 | Loss: 2.5319 | LR: 1.09e-05 | 2998.2ms/step | 20.0 samples/s
+Step 45790/50462 | Loss: 2.5682 | LR: 1.08e-05 | 3000.4ms/step | 20.0 samples/s
+Step 45800/50462 | Loss: 2.4836 | LR: 1.08e-05 | 3000.2ms/step | 20.0 samples/s
+Step 45810/50462 | Loss: 2.5423 | LR: 1.07e-05 | 2999.9ms/step | 20.0 samples/s
+Step 45820/50462 | Loss: 2.5380 | LR: 1.07e-05 | 3000.5ms/step | 20.0 samples/s
+Step 45830/50462 | Loss: 2.5458 | LR: 1.06e-05 | 3001.4ms/step | 20.0 samples/s
+Step 45840/50462 | Loss: 2.5593 | LR: 1.06e-05 | 3000.4ms/step | 20.0 samples/s
+Step 45850/50462 | Loss: 2.5420 | LR: 1.05e-05 | 3000.0ms/step | 20.0 samples/s
+Step 45860/50462 | Loss: 2.5655 | LR: 1.05e-05 | 2999.9ms/step | 20.0 samples/s
+Step 45870/50462 | Loss: 2.5511 | LR: 1.05e-05 | 3000.0ms/step | 20.0 samples/s
+Step 45880/50462 | Loss: 2.5517 | LR: 1.04e-05 | 3000.1ms/step | 20.0 samples/s
+Step 45890/50462 | Loss: 2.5149 | LR: 1.04e-05 | 3000.0ms/step | 20.0 samples/s
+Step 45900/50462 | Loss: 2.5374 | LR: 1.03e-05 | 2999.1ms/step | 20.0 samples/s
+Step 45910/50462 | Loss: 2.5672 | LR: 1.03e-05 | 2996.9ms/step | 20.0 samples/s
+Step 45920/50462 | Loss: 2.5200 | LR: 1.02e-05 | 3000.0ms/step | 20.0 samples/s
+Step 45930/50462 | Loss: 2.5328 | LR: 1.02e-05 | 3000.2ms/step | 20.0 samples/s
+Step 45940/50462 | Loss: 2.5525 | LR: 1.01e-05 | 3000.0ms/step | 20.0 samples/s
+Step 45950/50462 | Loss: 2.5378 | LR: 1.01e-05 | 2999.9ms/step | 20.0 samples/s
+Step 45960/50462 | Loss: 2.5234 | LR: 1.01e-05 | 3001.9ms/step | 20.0 samples/s
+Step 45970/50462 | Loss: 2.5311 | LR: 1.00e-05 | 2998.9ms/step | 20.0 samples/s
+Step 45980/50462 | Loss: 2.5739 | LR: 9.96e-06 | 2999.4ms/step | 20.0 samples/s
+Step 45990/50462 | Loss: 2.5269 | LR: 9.92e-06 | 4116.0ms/step | 14.6 samples/s
+Step 46000/50462 | Loss: 2.5424 | LR: 9.87e-06 | 3617.7ms/step | 16.6 samples/s
+Validation at step 46000: Loss = 2.5529, Perplexity = 12.84
+Saved checkpoint to checkpoints/run_20260609_063834/checkpoint-46000
+Step 46010/50462 | Loss: 2.5447 | LR: 9.83e-06 | 5656.5ms/step | 10.6 samples/s
+Step 46020/50462 | Loss: 2.5330 | LR: 9.79e-06 | 2991.6ms/step | 20.1 samples/s
+Step 46030/50462 | Loss: 2.5550 | LR: 9.74e-06 | 2998.2ms/step | 20.0 samples/s
+Step 46040/50462 | Loss: 2.5542 | LR: 9.70e-06 | 3000.2ms/step | 20.0 samples/s
+Step 46050/50462 | Loss: 2.5372 | LR: 9.65e-06 | 3004.0ms/step | 20.0 samples/s
+Step 46060/50462 | Loss: 2.5343 | LR: 9.61e-06 | 2998.0ms/step | 20.0 samples/s
+Step 46070/50462 | Loss: 2.5518 | LR: 9.57e-06 | 2998.2ms/step | 20.0 samples/s
+Step 46080/50462 | Loss: 2.5461 | LR: 9.52e-06 | 3000.1ms/step | 20.0 samples/s
+Step 46090/50462 | Loss: 2.5158 | LR: 9.48e-06 | 2999.9ms/step | 20.0 samples/s
+Step 46100/50462 | Loss: 2.5426 | LR: 9.44e-06 | 3000.1ms/step | 20.0 samples/s
+Step 46110/50462 | Loss: 2.5383 | LR: 9.40e-06 | 3000.1ms/step | 20.0 samples/s
+Step 46120/50462 | Loss: 2.5337 | LR: 9.35e-06 | 2999.9ms/step | 20.0 samples/s
+Step 46130/50462 | Loss: 2.5357 | LR: 9.31e-06 | 3000.0ms/step | 20.0 samples/s
+Step 46140/50462 | Loss: 2.5346 | LR: 9.27e-06 | 3000.1ms/step | 20.0 samples/s
+Step 46150/50462 | Loss: 2.5320 | LR: 9.22e-06 | 2999.9ms/step | 20.0 samples/s
+Step 46160/50462 | Loss: 2.5516 | LR: 9.18e-06 | 3000.4ms/step | 20.0 samples/s
+Step 46170/50462 | Loss: 2.5511 | LR: 9.14e-06 | 2999.7ms/step | 20.0 samples/s
+Step 46180/50462 | Loss: 2.5208 | LR: 9.10e-06 | 3000.0ms/step | 20.0 samples/s
+Step 46190/50462 | Loss: 2.5222 | LR: 9.06e-06 | 3000.0ms/step | 20.0 samples/s
+Step 46200/50462 | Loss: 2.5282 | LR: 9.01e-06 | 3000.3ms/step | 20.0 samples/s
+Step 46210/50462 | Loss: 2.5471 | LR: 8.97e-06 | 2999.8ms/step | 20.0 samples/s
+Step 46220/50462 | Loss: 2.5489 | LR: 8.93e-06 | 3000.1ms/step | 20.0 samples/s
+Step 46230/50462 | Loss: 2.5539 | LR: 8.89e-06 | 2999.9ms/step | 20.0 samples/s
+Step 46240/50462 | Loss: 2.5629 | LR: 8.85e-06 | 3000.2ms/step | 20.0 samples/s
+Step 46250/50462 | Loss: 2.5604 | LR: 8.80e-06 | 2999.9ms/step | 20.0 samples/s
+Validation at step 46250: Loss = 2.5529, Perplexity = 12.84
+Step 46260/50462 | Loss: 2.5615 | LR: 8.76e-06 | 4597.9ms/step | 13.0 samples/s
+Step 46270/50462 | Loss: 2.5476 | LR: 8.72e-06 | 3000.1ms/step | 20.0 samples/s
+Step 46280/50462 | Loss: 2.5794 | LR: 8.68e-06 | 2995.4ms/step | 20.0 samples/s
+Step 46290/50462 | Loss: 2.5667 | LR: 8.64e-06 | 2996.6ms/step | 20.0 samples/s
+Step 46300/50462 | Loss: 2.5358 | LR: 8.60e-06 | 2999.8ms/step | 20.0 samples/s
+Step 46310/50462 | Loss: 2.5431 | LR: 8.56e-06 | 3000.5ms/step | 20.0 samples/s
+Step 46320/50462 | Loss: 2.5693 | LR: 8.52e-06 | 2999.8ms/step | 20.0 samples/s
+Step 46330/50462 | Loss: 2.5498 | LR: 8.47e-06 | 3000.0ms/step | 20.0 samples/s
+Step 46340/50462 | Loss: 2.5494 | LR: 8.43e-06 | 3000.0ms/step | 20.0 samples/s
+Step 46350/50462 | Loss: 2.5701 | LR: 8.39e-06 | 3000.1ms/step | 20.0 samples/s
+Step 46360/50462 | Loss: 2.5333 | LR: 8.35e-06 | 2999.9ms/step | 20.0 samples/s
+Step 46370/50462 | Loss: 2.5158 | LR: 8.31e-06 | 2998.5ms/step | 20.0 samples/s
+Step 46380/50462 | Loss: 2.5433 | LR: 8.27e-06 | 2999.5ms/step | 20.0 samples/s
+Step 46390/50462 | Loss: 2.5329 | LR: 8.23e-06 | 3000.2ms/step | 20.0 samples/s
+Step 46400/50462 | Loss: 2.5248 | LR: 8.19e-06 | 3000.2ms/step | 20.0 samples/s
+Step 46410/50462 | Loss: 2.5365 | LR: 8.15e-06 | 2994.8ms/step | 20.0 samples/s
+Step 46420/50462 | Loss: 2.5351 | LR: 8.11e-06 | 2998.8ms/step | 20.0 samples/s
+Step 46430/50462 | Loss: 2.5568 | LR: 8.07e-06 | 2998.3ms/step | 20.0 samples/s
+Step 46440/50462 | Loss: 2.5391 | LR: 8.03e-06 | 2999.5ms/step | 20.0 samples/s
+Step 46450/50462 | Loss: 2.5662 | LR: 7.99e-06 | 3000.6ms/step | 20.0 samples/s
+Step 46460/50462 | Loss: 2.5436 | LR: 7.95e-06 | 2999.8ms/step | 20.0 samples/s
+Step 46470/50462 | Loss: 2.5358 | LR: 7.91e-06 | 2999.9ms/step | 20.0 samples/s
+Step 46480/50462 | Loss: 2.5296 | LR: 7.87e-06 | 3000.3ms/step | 20.0 samples/s
+Step 46490/50462 | Loss: 2.5263 | LR: 7.83e-06 | 3000.1ms/step | 20.0 samples/s
+Step 46500/50462 | Loss: 2.5459 | LR: 7.80e-06 | 2999.9ms/step | 20.0 samples/s
+Validation at step 46500: Loss = 2.5526, Perplexity = 12.84
+Step 46510/50462 | Loss: 2.5368 | LR: 7.76e-06 | 4596.0ms/step | 13.1 samples/s
+Step 46520/50462 | Loss: 2.5600 | LR: 7.72e-06 | 3000.0ms/step | 20.0 samples/s
+Step 46530/50462 | Loss: 2.5407 | LR: 7.68e-06 | 2997.9ms/step | 20.0 samples/s
+Step 46540/50462 | Loss: 2.5531 | LR: 7.64e-06 | 3000.3ms/step | 20.0 samples/s
+Step 46550/50462 | Loss: 2.5163 | LR: 7.60e-06 | 2999.9ms/step | 20.0 samples/s
+Step 46560/50462 | Loss: 2.5306 | LR: 7.56e-06 | 2999.9ms/step | 20.0 samples/s
+Step 46570/50462 | Loss: 2.5429 | LR: 7.52e-06 | 3000.9ms/step | 20.0 samples/s
+Step 46580/50462 | Loss: 2.5344 | LR: 7.49e-06 | 2999.2ms/step | 20.0 samples/s
+Step 46590/50462 | Loss: 2.5267 | LR: 7.45e-06 | 3000.1ms/step | 20.0 samples/s
+Step 46600/50462 | Loss: 2.5425 | LR: 7.41e-06 | 2999.3ms/step | 20.0 samples/s
+Step 46610/50462 | Loss: 2.5408 | LR: 7.37e-06 | 3000.9ms/step | 20.0 samples/s
+Step 46620/50462 | Loss: 2.5237 | LR: 7.33e-06 | 3001.6ms/step | 20.0 samples/s
+Step 46630/50462 | Loss: 2.5373 | LR: 7.29e-06 | 2996.6ms/step | 20.0 samples/s
+Step 46640/50462 | Loss: 2.5498 | LR: 7.26e-06 | 2997.6ms/step | 20.0 samples/s
+Step 46650/50462 | Loss: 2.5430 | LR: 7.22e-06 | 2998.2ms/step | 20.0 samples/s
+Step 46660/50462 | Loss: 2.5321 | LR: 7.18e-06 | 2995.8ms/step | 20.0 samples/s
+Step 46670/50462 | Loss: 2.5469 | LR: 7.14e-06 | 2997.9ms/step | 20.0 samples/s
+Step 46680/50462 | Loss: 2.5565 | LR: 7.11e-06 | 2996.0ms/step | 20.0 samples/s
+Step 46690/50462 | Loss: 2.5722 | LR: 7.07e-06 | 3000.4ms/step | 20.0 samples/s
+Step 46700/50462 | Loss: 2.5319 | LR: 7.03e-06 | 2997.5ms/step | 20.0 samples/s
+Step 46710/50462 | Loss: 2.5666 | LR: 6.99e-06 | 2995.5ms/step | 20.0 samples/s
+Step 46720/50462 | Loss: 2.5489 | LR: 6.96e-06 | 2998.7ms/step | 20.0 samples/s
+Step 46730/50462 | Loss: 2.5287 | LR: 6.92e-06 | 2998.0ms/step | 20.0 samples/s
+Step 46740/50462 | Loss: 2.5553 | LR: 6.88e-06 | 3000.5ms/step | 20.0 samples/s
+Step 46750/50462 | Loss: 2.5384 | LR: 6.85e-06 | 2999.5ms/step | 20.0 samples/s
+Validation at step 46750: Loss = 2.5527, Perplexity = 12.84
+Step 46760/50462 | Loss: 2.5527 | LR: 6.81e-06 | 4597.5ms/step | 13.1 samples/s
+Step 46770/50462 | Loss: 2.5432 | LR: 6.77e-06 | 2999.0ms/step | 20.0 samples/s
+Step 46780/50462 | Loss: 2.5668 | LR: 6.74e-06 | 2998.0ms/step | 20.0 samples/s
+Step 46790/50462 | Loss: 2.5245 | LR: 6.70e-06 | 2995.9ms/step | 20.0 samples/s
+Step 46800/50462 | Loss: 2.5629 | LR: 6.66e-06 | 2995.6ms/step | 20.0 samples/s
+Step 46810/50462 | Loss: 2.5575 | LR: 6.63e-06 | 2996.4ms/step | 20.0 samples/s
+Step 46820/50462 | Loss: 2.5227 | LR: 6.59e-06 | 2999.6ms/step | 20.0 samples/s
+Step 46830/50462 | Loss: 2.5206 | LR: 6.56e-06 | 2992.3ms/step | 20.1 samples/s
+Step 46840/50462 | Loss: 2.5464 | LR: 6.52e-06 | 2995.9ms/step | 20.0 samples/s
+Step 46850/50462 | Loss: 2.5414 | LR: 6.48e-06 | 2996.1ms/step | 20.0 samples/s
+Step 46860/50462 | Loss: 2.5420 | LR: 6.45e-06 | 2996.0ms/step | 20.0 samples/s
+Step 46870/50462 | Loss: 2.5001 | LR: 6.41e-06 | 2996.3ms/step | 20.0 samples/s
+Step 46880/50462 | Loss: 2.5365 | LR: 6.38e-06 | 2993.7ms/step | 20.0 samples/s
+Step 46890/50462 | Loss: 2.5402 | LR: 6.34e-06 | 2993.7ms/step | 20.0 samples/s
+Step 46900/50462 | Loss: 2.5342 | LR: 6.31e-06 | 2993.0ms/step | 20.0 samples/s
+Step 46910/50462 | Loss: 2.5267 | LR: 6.27e-06 | 2991.5ms/step | 20.1 samples/s
+Step 46920/50462 | Loss: 2.5648 | LR: 6.24e-06 | 2993.1ms/step | 20.0 samples/s
+Step 46930/50462 | Loss: 2.5412 | LR: 6.20e-06 | 2993.6ms/step | 20.0 samples/s
+Step 46940/50462 | Loss: 2.5432 | LR: 6.17e-06 | 2993.3ms/step | 20.0 samples/s
+Step 46950/50462 | Loss: 2.5269 | LR: 6.13e-06 | 2994.0ms/step | 20.0 samples/s
+Step 46960/50462 | Loss: 2.5509 | LR: 6.10e-06 | 2991.8ms/step | 20.1 samples/s
+Step 46970/50462 | Loss: 2.5524 | LR: 6.06e-06 | 2992.3ms/step | 20.1 samples/s
+Step 46980/50462 | Loss: 2.5511 | LR: 6.03e-06 | 2995.9ms/step | 20.0 samples/s
+Step 46990/50462 | Loss: 2.5724 | LR: 5.99e-06 | 2995.8ms/step | 20.0 samples/s
+Step 47000/50462 | Loss: 2.5515 | LR: 5.96e-06 | 2994.1ms/step | 20.0 samples/s
+Validation at step 47000: Loss = 2.5526, Perplexity = 12.84
+Saved checkpoint to checkpoints/run_20260609_063834/checkpoint-47000
+Step 47010/50462 | Loss: 2.5644 | LR: 5.93e-06 | 5653.6ms/step | 10.6 samples/s
+Step 47020/50462 | Loss: 2.5409 | LR: 5.89e-06 | 2995.0ms/step | 20.0 samples/s
+Step 47030/50462 | Loss: 2.5609 | LR: 5.86e-06 | 3000.1ms/step | 20.0 samples/s
+Step 47040/50462 | Loss: 2.5054 | LR: 5.82e-06 | 2995.6ms/step | 20.0 samples/s
+Step 47050/50462 | Loss: 2.5447 | LR: 5.79e-06 | 2993.8ms/step | 20.0 samples/s
+Step 47060/50462 | Loss: 2.5414 | LR: 5.76e-06 | 2998.7ms/step | 20.0 samples/s
+Step 47070/50462 | Loss: 2.5327 | LR: 5.72e-06 | 2994.8ms/step | 20.0 samples/s
+Step 47080/50462 | Loss: 2.5425 | LR: 5.69e-06 | 2996.5ms/step | 20.0 samples/s
+Step 47090/50462 | Loss: 2.5373 | LR: 5.65e-06 | 2995.6ms/step | 20.0 samples/s
+Step 47100/50462 | Loss: 2.5407 | LR: 5.62e-06 | 2992.5ms/step | 20.1 samples/s
+Step 47110/50462 | Loss: 2.5443 | LR: 5.59e-06 | 2994.7ms/step | 20.0 samples/s
+Step 47120/50462 | Loss: 2.4941 | LR: 5.56e-06 | 2994.9ms/step | 20.0 samples/s
+Step 47130/50462 | Loss: 2.5411 | LR: 5.52e-06 | 2996.5ms/step | 20.0 samples/s
+Step 47140/50462 | Loss: 2.5541 | LR: 5.49e-06 | 2992.5ms/step | 20.0 samples/s
+Step 47150/50462 | Loss: 2.5119 | LR: 5.46e-06 | 2993.4ms/step | 20.0 samples/s
+Step 47160/50462 | Loss: 2.5537 | LR: 5.42e-06 | 2992.0ms/step | 20.1 samples/s
+Step 47170/50462 | Loss: 2.5357 | LR: 5.39e-06 | 2996.6ms/step | 20.0 samples/s
+Step 47180/50462 | Loss: 2.5458 | LR: 5.36e-06 | 2999.8ms/step | 20.0 samples/s
+Step 47190/50462 | Loss: 2.5399 | LR: 5.33e-06 | 3000.5ms/step | 20.0 samples/s
+Step 47200/50462 | Loss: 2.5367 | LR: 5.29e-06 | 2989.7ms/step | 20.1 samples/s
+Step 47210/50462 | Loss: 2.5108 | LR: 5.26e-06 | 2994.3ms/step | 20.0 samples/s
+Step 47220/50462 | Loss: 2.5522 | LR: 5.23e-06 | 2997.2ms/step | 20.0 samples/s
+Step 47230/50462 | Loss: 2.5499 | LR: 5.20e-06 | 2993.7ms/step | 20.0 samples/s
+Step 47240/50462 | Loss: 2.5176 | LR: 5.16e-06 | 2997.1ms/step | 20.0 samples/s
+Step 47250/50462 | Loss: 2.5497 | LR: 5.13e-06 | 2997.8ms/step | 20.0 samples/s
+Validation at step 47250: Loss = 2.5530, Perplexity = 12.85
+Step 47260/50462 | Loss: 2.5425 | LR: 5.10e-06 | 4589.3ms/step | 13.1 samples/s
+Step 47270/50462 | Loss: 2.5549 | LR: 5.07e-06 | 2997.0ms/step | 20.0 samples/s
+Step 47280/50462 | Loss: 2.5296 | LR: 5.04e-06 | 2995.7ms/step | 20.0 samples/s
+Step 47290/50462 | Loss: 2.5568 | LR: 5.01e-06 | 2997.8ms/step | 20.0 samples/s
+Step 47300/50462 | Loss: 2.5564 | LR: 4.97e-06 | 2994.5ms/step | 20.0 samples/s
+Step 47310/50462 | Loss: 2.5173 | LR: 4.94e-06 | 2998.2ms/step | 20.0 samples/s
+Step 47320/50462 | Loss: 2.5336 | LR: 4.91e-06 | 2993.5ms/step | 20.0 samples/s
+Step 47330/50462 | Loss: 2.5369 | LR: 4.88e-06 | 2994.5ms/step | 20.0 samples/s
+Step 47340/50462 | Loss: 2.5019 | LR: 4.85e-06 | 2997.6ms/step | 20.0 samples/s
+Step 47350/50462 | Loss: 2.5510 | LR: 4.82e-06 | 2996.2ms/step | 20.0 samples/s
+Step 47360/50462 | Loss: 2.5198 | LR: 4.79e-06 | 2997.9ms/step | 20.0 samples/s
+Step 47370/50462 | Loss: 2.5587 | LR: 4.76e-06 | 2998.1ms/step | 20.0 samples/s
+Step 47380/50462 | Loss: 2.5516 | LR: 4.73e-06 | 2989.2ms/step | 20.1 samples/s
+Step 47390/50462 | Loss: 2.5346 | LR: 4.70e-06 | 2993.0ms/step | 20.0 samples/s
+Step 47400/50462 | Loss: 2.5368 | LR: 4.67e-06 | 2992.6ms/step | 20.0 samples/s
+Step 47410/50462 | Loss: 2.5496 | LR: 4.64e-06 | 2990.9ms/step | 20.1 samples/s
+Step 47420/50462 | Loss: 2.4998 | LR: 4.61e-06 | 2991.6ms/step | 20.1 samples/s
+Step 47430/50462 | Loss: 2.5276 | LR: 4.58e-06 | 2991.0ms/step | 20.1 samples/s
+Step 47440/50462 | Loss: 2.5362 | LR: 4.55e-06 | 2993.9ms/step | 20.0 samples/s
+Step 47450/50462 | Loss: 2.5105 | LR: 4.52e-06 | 2994.4ms/step | 20.0 samples/s
+Step 47460/50462 | Loss: 2.5555 | LR: 4.49e-06 | 2993.8ms/step | 20.0 samples/s
+Step 47470/50462 | Loss: 2.5543 | LR: 4.46e-06 | 2995.9ms/step | 20.0 samples/s
+Step 47480/50462 | Loss: 2.5519 | LR: 4.43e-06 | 2993.3ms/step | 20.0 samples/s
+Step 47490/50462 | Loss: 2.5456 | LR: 4.40e-06 | 2992.8ms/step | 20.0 samples/s
+Step 47500/50462 | Loss: 2.5367 | LR: 4.37e-06 | 2994.2ms/step | 20.0 samples/s
+Validation at step 47500: Loss = 2.5530, Perplexity = 12.85
+Step 47510/50462 | Loss: 2.5073 | LR: 4.34e-06 | 4594.1ms/step | 13.1 samples/s
+Step 47520/50462 | Loss: 2.5557 | LR: 4.31e-06 | 2993.6ms/step | 20.0 samples/s
+Step 47530/50462 | Loss: 2.5515 | LR: 4.28e-06 | 2994.6ms/step | 20.0 samples/s
+Step 47540/50462 | Loss: 2.5191 | LR: 4.25e-06 | 2991.6ms/step | 20.1 samples/s
+Step 47550/50462 | Loss: 2.5487 | LR: 4.22e-06 | 2989.3ms/step | 20.1 samples/s
+Step 47560/50462 | Loss: 2.5331 | LR: 4.19e-06 | 2997.8ms/step | 20.0 samples/s
+Step 47570/50462 | Loss: 2.5436 | LR: 4.16e-06 | 2995.3ms/step | 20.0 samples/s
+Step 47580/50462 | Loss: 2.5794 | LR: 4.13e-06 | 2998.0ms/step | 20.0 samples/s
+Step 47590/50462 | Loss: 2.5496 | LR: 4.11e-06 | 2993.3ms/step | 20.0 samples/s
+Step 47600/50462 | Loss: 2.5406 | LR: 4.08e-06 | 2992.0ms/step | 20.1 samples/s
+Step 47610/50462 | Loss: 2.5206 | LR: 4.05e-06 | 2994.8ms/step | 20.0 samples/s
+Step 47620/50462 | Loss: 2.5327 | LR: 4.02e-06 | 2991.3ms/step | 20.1 samples/s
+Step 47630/50462 | Loss: 2.5618 | LR: 3.99e-06 | 2992.1ms/step | 20.1 samples/s
+Step 47640/50462 | Loss: 2.5168 | LR: 3.97e-06 | 2992.7ms/step | 20.0 samples/s
+Step 47650/50462 | Loss: 2.5376 | LR: 3.94e-06 | 2990.5ms/step | 20.1 samples/s
+Step 47660/50462 | Loss: 2.5346 | LR: 3.91e-06 | 2993.3ms/step | 20.0 samples/s
+Step 47670/50462 | Loss: 2.5289 | LR: 3.88e-06 | 2994.9ms/step | 20.0 samples/s
+Step 47680/50462 | Loss: 2.5456 | LR: 3.85e-06 | 2994.2ms/step | 20.0 samples/s
+Step 47690/50462 | Loss: 2.5414 | LR: 3.83e-06 | 2994.3ms/step | 20.0 samples/s
+Step 47700/50462 | Loss: 2.5503 | LR: 3.80e-06 | 2999.0ms/step | 20.0 samples/s
+Step 47710/50462 | Loss: 2.5153 | LR: 3.77e-06 | 2999.7ms/step | 20.0 samples/s
+Step 47720/50462 | Loss: 2.5434 | LR: 3.74e-06 | 2999.5ms/step | 20.0 samples/s
+Step 47730/50462 | Loss: 2.5420 | LR: 3.72e-06 | 3000.7ms/step | 20.0 samples/s
+Step 47740/50462 | Loss: 2.5276 | LR: 3.69e-06 | 2999.9ms/step | 20.0 samples/s
+Step 47750/50462 | Loss: 2.5555 | LR: 3.66e-06 | 2998.0ms/step | 20.0 samples/s
+Validation at step 47750: Loss = 2.5531, Perplexity = 12.85
+Step 47760/50462 | Loss: 2.5124 | LR: 3.64e-06 | 4591.5ms/step | 13.1 samples/s
+Step 47770/50462 | Loss: 2.5744 | LR: 3.61e-06 | 2994.5ms/step | 20.0 samples/s
+Step 47780/50462 | Loss: 2.5334 | LR: 3.58e-06 | 2994.0ms/step | 20.0 samples/s
+Step 47790/50462 | Loss: 2.5273 | LR: 3.56e-06 | 2999.3ms/step | 20.0 samples/s
+Step 47800/50462 | Loss: 2.5190 | LR: 3.53e-06 | 2992.7ms/step | 20.0 samples/s
+Step 47810/50462 | Loss: 2.5507 | LR: 3.50e-06 | 2994.6ms/step | 20.0 samples/s
+Step 47820/50462 | Loss: 2.5467 | LR: 3.48e-06 | 2991.8ms/step | 20.1 samples/s
+Step 47830/50462 | Loss: 2.5385 | LR: 3.45e-06 | 2997.2ms/step | 20.0 samples/s
+Step 47840/50462 | Loss: 2.5598 | LR: 3.42e-06 | 2996.7ms/step | 20.0 samples/s
+Step 47850/50462 | Loss: 2.5511 | LR: 3.40e-06 | 3000.0ms/step | 20.0 samples/s
+Step 47860/50462 | Loss: 2.5541 | LR: 3.37e-06 | 2995.9ms/step | 20.0 samples/s
+Step 47870/50462 | Loss: 2.5472 | LR: 3.35e-06 | 2998.4ms/step | 20.0 samples/s
+Step 47880/50462 | Loss: 2.5021 | LR: 3.32e-06 | 2997.9ms/step | 20.0 samples/s
+Step 47890/50462 | Loss: 2.5604 | LR: 3.30e-06 | 2997.8ms/step | 20.0 samples/s
+Step 47900/50462 | Loss: 2.5496 | LR: 3.27e-06 | 2998.4ms/step | 20.0 samples/s
+Step 47910/50462 | Loss: 2.5477 | LR: 3.24e-06 | 2996.5ms/step | 20.0 samples/s
+Step 47920/50462 | Loss: 2.5220 | LR: 3.22e-06 | 2995.5ms/step | 20.0 samples/s
+Step 47930/50462 | Loss: 2.5603 | LR: 3.19e-06 | 2998.0ms/step | 20.0 samples/s
+Step 47940/50462 | Loss: 2.5336 | LR: 3.17e-06 | 2999.0ms/step | 20.0 samples/s
+Step 47950/50462 | Loss: 2.5416 | LR: 3.14e-06 | 2998.8ms/step | 20.0 samples/s
+Step 47960/50462 | Loss: 2.5090 | LR: 3.12e-06 | 2993.8ms/step | 20.0 samples/s
+Step 47970/50462 | Loss: 2.5442 | LR: 3.09e-06 | 2996.3ms/step | 20.0 samples/s
+Step 47980/50462 | Loss: 2.5430 | LR: 3.07e-06 | 2994.1ms/step | 20.0 samples/s
+Step 47990/50462 | Loss: 2.5258 | LR: 3.04e-06 | 2995.9ms/step | 20.0 samples/s
+Step 48000/50462 | Loss: 2.5347 | LR: 3.02e-06 | 2998.2ms/step | 20.0 samples/s
+Validation at step 48000: Loss = 2.5527, Perplexity = 12.84
+Saved checkpoint to checkpoints/run_20260609_063834/checkpoint-48000
+Step 48010/50462 | Loss: 2.5475 | LR: 3.00e-06 | 5655.6ms/step | 10.6 samples/s
+Step 48020/50462 | Loss: 2.5764 | LR: 2.97e-06 | 2992.7ms/step | 20.0 samples/s
+Step 48030/50462 | Loss: 2.5186 | LR: 2.95e-06 | 2995.1ms/step | 20.0 samples/s
+Step 48040/50462 | Loss: 2.5315 | LR: 2.92e-06 | 2992.7ms/step | 20.0 samples/s
+Step 48050/50462 | Loss: 2.5256 | LR: 2.90e-06 | 2994.1ms/step | 20.0 samples/s
+Step 48060/50462 | Loss: 2.5413 | LR: 2.87e-06 | 2994.0ms/step | 20.0 samples/s
+Step 48070/50462 | Loss: 2.5442 | LR: 2.85e-06 | 2992.1ms/step | 20.1 samples/s
+Step 48080/50462 | Loss: 2.5673 | LR: 2.83e-06 | 2993.7ms/step | 20.0 samples/s
+Step 48090/50462 | Loss: 2.5450 | LR: 2.80e-06 | 2992.0ms/step | 20.1 samples/s
+Step 48100/50462 | Loss: 2.5468 | LR: 2.78e-06 | 3546.7ms/step | 16.9 samples/s
+Step 48110/50462 | Loss: 2.5371 | LR: 2.76e-06 | 3678.9ms/step | 16.3 samples/s
+Step 48120/50462 | Loss: 2.5351 | LR: 2.73e-06 | 2984.5ms/step | 20.1 samples/s
+Step 48130/50462 | Loss: 2.5356 | LR: 2.71e-06 | 3564.9ms/step | 16.8 samples/s
+Step 48140/50462 | Loss: 2.5173 | LR: 2.69e-06 | 2993.3ms/step | 20.0 samples/s
+Step 48150/50462 | Loss: 2.5315 | LR: 2.66e-06 | 2993.4ms/step | 20.0 samples/s
+Step 48160/50462 | Loss: 2.5507 | LR: 2.64e-06 | 2992.7ms/step | 20.0 samples/s
+Step 48170/50462 | Loss: 2.5325 | LR: 2.62e-06 | 2994.0ms/step | 20.0 samples/s
+Step 48180/50462 | Loss: 2.5428 | LR: 2.60e-06 | 2995.9ms/step | 20.0 samples/s
+Step 48190/50462 | Loss: 2.5374 | LR: 2.57e-06 | 2993.7ms/step | 20.0 samples/s
+Step 48200/50462 | Loss: 2.5475 | LR: 2.55e-06 | 2996.3ms/step | 20.0 samples/s
+Step 48210/50462 | Loss: 2.5245 | LR: 2.53e-06 | 2997.5ms/step | 20.0 samples/s
+Step 48220/50462 | Loss: 2.5312 | LR: 2.51e-06 | 2999.8ms/step | 20.0 samples/s
+Step 48230/50462 | Loss: 2.5384 | LR: 2.48e-06 | 2993.9ms/step | 20.0 samples/s
+Step 48240/50462 | Loss: 2.5359 | LR: 2.46e-06 | 2996.5ms/step | 20.0 samples/s
+Step 48250/50462 | Loss: 2.5748 | LR: 2.44e-06 | 2992.6ms/step | 20.0 samples/s
+Validation at step 48250: Loss = 2.5529, Perplexity = 12.84
+Step 48260/50462 | Loss: 2.5127 | LR: 2.42e-06 | 4587.9ms/step | 13.1 samples/s
+Step 48270/50462 | Loss: 2.5381 | LR: 2.39e-06 | 2992.3ms/step | 20.1 samples/s
+Step 48280/50462 | Loss: 2.5472 | LR: 2.37e-06 | 2993.8ms/step | 20.0 samples/s
+Step 48290/50462 | Loss: 2.5338 | LR: 2.35e-06 | 2998.2ms/step | 20.0 samples/s
+Step 48300/50462 | Loss: 2.5265 | LR: 2.33e-06 | 2995.3ms/step | 20.0 samples/s
+Step 48310/50462 | Loss: 2.5481 | LR: 2.31e-06 | 3000.4ms/step | 20.0 samples/s
+Step 48320/50462 | Loss: 2.5457 | LR: 2.29e-06 | 2995.7ms/step | 20.0 samples/s
+Step 48330/50462 | Loss: 2.5291 | LR: 2.27e-06 | 2996.2ms/step | 20.0 samples/s
+Step 48340/50462 | Loss: 2.5522 | LR: 2.24e-06 | 3000.2ms/step | 20.0 samples/s
+Step 48350/50462 | Loss: 2.5662 | LR: 2.22e-06 | 3000.3ms/step | 20.0 samples/s
+Step 48360/50462 | Loss: 2.5224 | LR: 2.20e-06 | 2991.8ms/step | 20.1 samples/s
+Step 48370/50462 | Loss: 2.5593 | LR: 2.18e-06 | 2993.9ms/step | 20.0 samples/s
+Step 48380/50462 | Loss: 2.5268 | LR: 2.16e-06 | 2991.9ms/step | 20.1 samples/s
+Step 48390/50462 | Loss: 2.5729 | LR: 2.14e-06 | 2992.6ms/step | 20.0 samples/s
+Step 48400/50462 | Loss: 2.5292 | LR: 2.12e-06 | 2995.6ms/step | 20.0 samples/s
+Step 48410/50462 | Loss: 2.5566 | LR: 2.10e-06 | 2996.2ms/step | 20.0 samples/s
+Step 48420/50462 | Loss: 2.5346 | LR: 2.08e-06 | 2990.4ms/step | 20.1 samples/s
+Step 48430/50462 | Loss: 2.5312 | LR: 2.06e-06 | 2991.4ms/step | 20.1 samples/s
+Step 48440/50462 | Loss: 2.5481 | LR: 2.04e-06 | 2994.3ms/step | 20.0 samples/s
+Step 48450/50462 | Loss: 2.5140 | LR: 2.02e-06 | 2995.4ms/step | 20.0 samples/s
+Step 48460/50462 | Loss: 2.5620 | LR: 2.00e-06 | 2994.0ms/step | 20.0 samples/s
+Step 48470/50462 | Loss: 2.5493 | LR: 1.98e-06 | 2994.3ms/step | 20.0 samples/s
+Step 48480/50462 | Loss: 2.5315 | LR: 1.96e-06 | 2997.9ms/step | 20.0 samples/s
+Step 48490/50462 | Loss: 2.5402 | LR: 1.94e-06 | 2998.3ms/step | 20.0 samples/s
+Step 48500/50462 | Loss: 2.5087 | LR: 1.92e-06 | 2998.0ms/step | 20.0 samples/s
+Validation at step 48500: Loss = 2.5532, Perplexity = 12.85
+Step 48510/50462 | Loss: 2.5428 | LR: 1.90e-06 | 4594.1ms/step | 13.1 samples/s
+Step 48520/50462 | Loss: 2.5427 | LR: 1.88e-06 | 3000.0ms/step | 20.0 samples/s
+Step 48530/50462 | Loss: 2.5598 | LR: 1.86e-06 | 2999.6ms/step | 20.0 samples/s
+Step 48540/50462 | Loss: 2.5156 | LR: 1.84e-06 | 2997.2ms/step | 20.0 samples/s
+Step 48550/50462 | Loss: 2.5251 | LR: 1.82e-06 | 2993.3ms/step | 20.0 samples/s
+Step 48560/50462 | Loss: 2.5356 | LR: 1.80e-06 | 2994.0ms/step | 20.0 samples/s
+Step 48570/50462 | Loss: 2.5371 | LR: 1.78e-06 | 2992.0ms/step | 20.1 samples/s
+Step 48580/50462 | Loss: 2.5305 | LR: 1.77e-06 | 2990.7ms/step | 20.1 samples/s
+Step 48590/50462 | Loss: 2.5549 | LR: 1.75e-06 | 2993.6ms/step | 20.0 samples/s
+Step 48600/50462 | Loss: 2.5462 | LR: 1.73e-06 | 2991.6ms/step | 20.1 samples/s
+Step 48610/50462 | Loss: 2.5323 | LR: 1.71e-06 | 2990.1ms/step | 20.1 samples/s
+Step 48620/50462 | Loss: 2.5358 | LR: 1.69e-06 | 2993.6ms/step | 20.0 samples/s
+Step 48630/50462 | Loss: 2.5588 | LR: 1.67e-06 | 2993.9ms/step | 20.0 samples/s
+Step 48640/50462 | Loss: 2.5410 | LR: 1.66e-06 | 2990.3ms/step | 20.1 samples/s
+Step 48650/50462 | Loss: 2.5593 | LR: 1.64e-06 | 2998.3ms/step | 20.0 samples/s
+Step 48660/50462 | Loss: 2.5535 | LR: 1.62e-06 | 2995.5ms/step | 20.0 samples/s
+Step 48670/50462 | Loss: 2.5553 | LR: 1.60e-06 | 2996.2ms/step | 20.0 samples/s
+Step 48680/50462 | Loss: 2.5361 | LR: 1.58e-06 | 2993.9ms/step | 20.0 samples/s
+Step 48690/50462 | Loss: 2.5508 | LR: 1.57e-06 | 2994.7ms/step | 20.0 samples/s
+Step 48700/50462 | Loss: 2.5120 | LR: 1.55e-06 | 2993.7ms/step | 20.0 samples/s
+Step 48710/50462 | Loss: 2.5348 | LR: 1.53e-06 | 2994.5ms/step | 20.0 samples/s
+Step 48720/50462 | Loss: 2.5544 | LR: 1.51e-06 | 2993.4ms/step | 20.0 samples/s
+Step 48730/50462 | Loss: 2.5401 | LR: 1.50e-06 | 2993.6ms/step | 20.0 samples/s
+Step 48740/50462 | Loss: 2.5185 | LR: 1.48e-06 | 2997.0ms/step | 20.0 samples/s
+Step 48750/50462 | Loss: 2.5616 | LR: 1.46e-06 | 2996.2ms/step | 20.0 samples/s
+Validation at step 48750: Loss = 2.5530, Perplexity = 12.85
+Step 48760/50462 | Loss: 2.5722 | LR: 1.44e-06 | 4593.6ms/step | 13.1 samples/s
+Step 48770/50462 | Loss: 2.5129 | LR: 1.43e-06 | 2998.6ms/step | 20.0 samples/s
+Step 48780/50462 | Loss: 2.5353 | LR: 1.41e-06 | 2993.1ms/step | 20.0 samples/s
+Step 48790/50462 | Loss: 2.5446 | LR: 1.39e-06 | 2998.8ms/step | 20.0 samples/s
+Step 48800/50462 | Loss: 2.5227 | LR: 1.38e-06 | 2997.8ms/step | 20.0 samples/s
+Step 48810/50462 | Loss: 2.5324 | LR: 1.36e-06 | 2995.7ms/step | 20.0 samples/s
+Step 48820/50462 | Loss: 2.5372 | LR: 1.34e-06 | 2996.0ms/step | 20.0 samples/s
+Step 48830/50462 | Loss: 2.5535 | LR: 1.33e-06 | 3000.5ms/step | 20.0 samples/s
+Step 48840/50462 | Loss: 2.5666 | LR: 1.31e-06 | 2999.3ms/step | 20.0 samples/s
+Step 48850/50462 | Loss: 2.5369 | LR: 1.30e-06 | 2994.3ms/step | 20.0 samples/s
+Step 48860/50462 | Loss: 2.5052 | LR: 1.28e-06 | 2996.4ms/step | 20.0 samples/s
+Step 48870/50462 | Loss: 2.5481 | LR: 1.26e-06 | 2991.9ms/step | 20.1 samples/s
+Step 48880/50462 | Loss: 2.5299 | LR: 1.25e-06 | 2996.5ms/step | 20.0 samples/s
+Step 48890/50462 | Loss: 2.5601 | LR: 1.23e-06 | 2995.6ms/step | 20.0 samples/s
+Step 48900/50462 | Loss: 2.5377 | LR: 1.22e-06 | 2996.2ms/step | 20.0 samples/s
+Step 48910/50462 | Loss: 2.5522 | LR: 1.20e-06 | 2999.8ms/step | 20.0 samples/s
+Step 48920/50462 | Loss: 2.5396 | LR: 1.19e-06 | 2993.5ms/step | 20.0 samples/s
+Step 48930/50462 | Loss: 2.5358 | LR: 1.17e-06 | 2996.7ms/step | 20.0 samples/s
+Step 48940/50462 | Loss: 2.5156 | LR: 1.16e-06 | 2990.0ms/step | 20.1 samples/s
+Step 48950/50462 | Loss: 2.5163 | LR: 1.14e-06 | 2999.9ms/step | 20.0 samples/s
+Step 48960/50462 | Loss: 2.5490 | LR: 1.13e-06 | 2998.2ms/step | 20.0 samples/s
+Step 48970/50462 | Loss: 2.5149 | LR: 1.11e-06 | 2996.9ms/step | 20.0 samples/s
+Step 48980/50462 | Loss: 2.5826 | LR: 1.10e-06 | 2995.3ms/step | 20.0 samples/s
+Step 48990/50462 | Loss: 2.5663 | LR: 1.08e-06 | 2995.7ms/step | 20.0 samples/s
+Step 49000/50462 | Loss: 2.5499 | LR: 1.07e-06 | 2994.8ms/step | 20.0 samples/s
+Validation at step 49000: Loss = 2.5530, Perplexity = 12.85
+Saved checkpoint to checkpoints/run_20260609_063834/checkpoint-49000
+Step 49010/50462 | Loss: 2.5380 | LR: 1.05e-06 | 5646.7ms/step | 10.6 samples/s
+Step 49020/50462 | Loss: 2.5451 | LR: 1.04e-06 | 2989.9ms/step | 20.1 samples/s
+Step 49030/50462 | Loss: 2.5462 | LR: 1.02e-06 | 2992.8ms/step | 20.0 samples/s
+Step 49040/50462 | Loss: 2.5429 | LR: 1.01e-06 | 3000.0ms/step | 20.0 samples/s
+Step 49050/50462 | Loss: 2.5433 | LR: 9.95e-07 | 2995.7ms/step | 20.0 samples/s
+Step 49060/50462 | Loss: 2.5632 | LR: 9.81e-07 | 3000.1ms/step | 20.0 samples/s
+Step 49070/50462 | Loss: 2.5603 | LR: 9.67e-07 | 3000.8ms/step | 20.0 samples/s
+Step 49080/50462 | Loss: 2.5632 | LR: 9.53e-07 | 2998.8ms/step | 20.0 samples/s
+Step 49090/50462 | Loss: 2.5171 | LR: 9.39e-07 | 2992.8ms/step | 20.0 samples/s
+Step 49100/50462 | Loss: 2.5658 | LR: 9.25e-07 | 2995.4ms/step | 20.0 samples/s
+Step 49110/50462 | Loss: 2.5497 | LR: 9.12e-07 | 2994.4ms/step | 20.0 samples/s
+Step 49120/50462 | Loss: 2.5200 | LR: 8.99e-07 | 2992.0ms/step | 20.1 samples/s
+Step 49130/50462 | Loss: 2.5491 | LR: 8.85e-07 | 2993.7ms/step | 20.0 samples/s
+Step 49140/50462 | Loss: 2.5206 | LR: 8.72e-07 | 2994.3ms/step | 20.0 samples/s
+Step 49150/50462 | Loss: 2.5527 | LR: 8.59e-07 | 2990.3ms/step | 20.1 samples/s
+Step 49160/50462 | Loss: 2.5422 | LR: 8.46e-07 | 2990.3ms/step | 20.1 samples/s
+Step 49170/50462 | Loss: 2.5400 | LR: 8.33e-07 | 2992.9ms/step | 20.0 samples/s
+Step 49180/50462 | Loss: 2.5303 | LR: 8.20e-07 | 2996.2ms/step | 20.0 samples/s
+Step 49190/50462 | Loss: 2.5177 | LR: 8.07e-07 | 2996.1ms/step | 20.0 samples/s
+Step 49200/50462 | Loss: 2.5737 | LR: 7.95e-07 | 2993.5ms/step | 20.0 samples/s
+Step 49210/50462 | Loss: 2.5431 | LR: 7.82e-07 | 2993.0ms/step | 20.0 samples/s
+Step 49220/50462 | Loss: 2.5526 | LR: 7.70e-07 | 2998.4ms/step | 20.0 samples/s
+Step 49230/50462 | Loss: 2.5559 | LR: 7.57e-07 | 2995.7ms/step | 20.0 samples/s
+Step 49240/50462 | Loss: 2.5452 | LR: 7.45e-07 | 2996.1ms/step | 20.0 samples/s
+Step 49250/50462 | Loss: 2.5401 | LR: 7.33e-07 | 2994.2ms/step | 20.0 samples/s
+Validation at step 49250: Loss = 2.5526, Perplexity = 12.84
+Step 49260/50462 | Loss: 2.5112 | LR: 7.21e-07 | 4589.0ms/step | 13.1 samples/s
+Step 49270/50462 | Loss: 2.5319 | LR: 7.09e-07 | 2995.9ms/step | 20.0 samples/s
+Step 49280/50462 | Loss: 2.5140 | LR: 6.97e-07 | 2996.6ms/step | 20.0 samples/s
+Step 49290/50462 | Loss: 2.5371 | LR: 6.85e-07 | 2993.8ms/step | 20.0 samples/s
+Step 49300/50462 | Loss: 2.5346 | LR: 6.74e-07 | 2996.4ms/step | 20.0 samples/s
+Step 49310/50462 | Loss: 2.5407 | LR: 6.62e-07 | 2993.6ms/step | 20.0 samples/s
+Step 49320/50462 | Loss: 2.5283 | LR: 6.51e-07 | 2995.0ms/step | 20.0 samples/s
+Step 49330/50462 | Loss: 2.5375 | LR: 6.39e-07 | 2995.1ms/step | 20.0 samples/s
+Step 49340/50462 | Loss: 2.5452 | LR: 6.28e-07 | 2992.4ms/step | 20.1 samples/s
+Step 49350/50462 | Loss: 2.5504 | LR: 6.17e-07 | 2996.0ms/step | 20.0 samples/s
+Step 49360/50462 | Loss: 2.5611 | LR: 6.06e-07 | 2993.8ms/step | 20.0 samples/s
+Step 49370/50462 | Loss: 2.5435 | LR: 5.95e-07 | 2992.7ms/step | 20.0 samples/s
+Step 49380/50462 | Loss: 2.5554 | LR: 5.84e-07 | 2990.9ms/step | 20.1 samples/s
+Step 49390/50462 | Loss: 2.5545 | LR: 5.73e-07 | 2994.1ms/step | 20.0 samples/s
+Step 49400/50462 | Loss: 2.5518 | LR: 5.63e-07 | 2993.5ms/step | 20.0 samples/s
+Step 49410/50462 | Loss: 2.5457 | LR: 5.52e-07 | 2997.7ms/step | 20.0 samples/s
+Step 49420/50462 | Loss: 2.5422 | LR: 5.42e-07 | 2995.2ms/step | 20.0 samples/s
+Step 49430/50462 | Loss: 2.5220 | LR: 5.31e-07 | 2992.7ms/step | 20.0 samples/s
+Step 49440/50462 | Loss: 2.5315 | LR: 5.21e-07 | 2995.3ms/step | 20.0 samples/s
+Step 49450/50462 | Loss: 2.5178 | LR: 5.11e-07 | 2994.7ms/step | 20.0 samples/s
+Step 49460/50462 | Loss: 2.5522 | LR: 5.01e-07 | 3000.0ms/step | 20.0 samples/s
+Step 49470/50462 | Loss: 2.5517 | LR: 4.91e-07 | 2999.6ms/step | 20.0 samples/s
+Step 49480/50462 | Loss: 2.5472 | LR: 4.81e-07 | 2996.1ms/step | 20.0 samples/s
+Step 49490/50462 | Loss: 2.5285 | LR: 4.72e-07 | 2998.4ms/step | 20.0 samples/s
+Step 49500/50462 | Loss: 2.5475 | LR: 4.62e-07 | 2997.8ms/step | 20.0 samples/s
+Validation at step 49500: Loss = 2.5525, Perplexity = 12.84
+Step 49510/50462 | Loss: 2.5574 | LR: 4.52e-07 | 4594.2ms/step | 13.1 samples/s
+Step 49520/50462 | Loss: 2.5367 | LR: 4.43e-07 | 3000.2ms/step | 20.0 samples/s
+Step 49530/50462 | Loss: 2.5408 | LR: 4.34e-07 | 3001.9ms/step | 20.0 samples/s
+Step 49540/50462 | Loss: 2.5319 | LR: 4.24e-07 | 3000.1ms/step | 20.0 samples/s
+Step 49550/50462 | Loss: 2.5397 | LR: 4.15e-07 | 2999.8ms/step | 20.0 samples/s
+Step 49560/50462 | Loss: 2.5559 | LR: 4.06e-07 | 3000.3ms/step | 20.0 samples/s
+Step 49570/50462 | Loss: 2.5577 | LR: 3.97e-07 | 3000.3ms/step | 20.0 samples/s
+Step 49580/50462 | Loss: 2.5555 | LR: 3.88e-07 | 2996.9ms/step | 20.0 samples/s
+Step 49590/50462 | Loss: 2.5299 | LR: 3.79e-07 | 2997.0ms/step | 20.0 samples/s
+Step 49600/50462 | Loss: 2.5611 | LR: 3.71e-07 | 2999.9ms/step | 20.0 samples/s
+Step 49610/50462 | Loss: 2.5401 | LR: 3.62e-07 | 3000.1ms/step | 20.0 samples/s
+Step 49620/50462 | Loss: 2.5411 | LR: 3.54e-07 | 2999.6ms/step | 20.0 samples/s
+Step 49630/50462 | Loss: 2.5410 | LR: 3.45e-07 | 2998.5ms/step | 20.0 samples/s
+Step 49640/50462 | Loss: 2.5428 | LR: 3.37e-07 | 3000.0ms/step | 20.0 samples/s
+Step 49650/50462 | Loss: 2.5465 | LR: 3.29e-07 | 2999.9ms/step | 20.0 samples/s
+Step 49660/50462 | Loss: 2.5283 | LR: 3.21e-07 | 3000.0ms/step | 20.0 samples/s
+Step 49670/50462 | Loss: 2.5169 | LR: 3.13e-07 | 3000.2ms/step | 20.0 samples/s
+Step 49680/50462 | Loss: 2.5601 | LR: 3.05e-07 | 2999.0ms/step | 20.0 samples/s
+Step 49690/50462 | Loss: 2.5785 | LR: 2.97e-07 | 2998.7ms/step | 20.0 samples/s
+Step 49700/50462 | Loss: 2.5494 | LR: 2.90e-07 | 2997.4ms/step | 20.0 samples/s
+Step 49710/50462 | Loss: 2.5441 | LR: 2.82e-07 | 2997.0ms/step | 20.0 samples/s
+Step 49720/50462 | Loss: 2.5345 | LR: 2.75e-07 | 2999.1ms/step | 20.0 samples/s
+Step 49730/50462 | Loss: 2.5502 | LR: 2.67e-07 | 2998.8ms/step | 20.0 samples/s
+Step 49740/50462 | Loss: 2.5710 | LR: 2.60e-07 | 3000.1ms/step | 20.0 samples/s
+Step 49750/50462 | Loss: 2.5428 | LR: 2.53e-07 | 3001.0ms/step | 20.0 samples/s
+Validation at step 49750: Loss = 2.5529, Perplexity = 12.84
+Step 49760/50462 | Loss: 2.5593 | LR: 2.46e-07 | 4598.8ms/step | 13.0 samples/s
+Step 49770/50462 | Loss: 2.5487 | LR: 2.39e-07 | 3002.2ms/step | 20.0 samples/s
+Step 49780/50462 | Loss: 2.5345 | LR: 2.32e-07 | 2999.9ms/step | 20.0 samples/s
+Step 49790/50462 | Loss: 2.5260 | LR: 2.25e-07 | 2999.9ms/step | 20.0 samples/s
+Step 49800/50462 | Loss: 2.5693 | LR: 2.19e-07 | 3000.1ms/step | 20.0 samples/s
+Step 49810/50462 | Loss: 2.5576 | LR: 2.12e-07 | 3000.2ms/step | 20.0 samples/s
+Step 49820/50462 | Loss: 2.5538 | LR: 2.06e-07 | 2999.7ms/step | 20.0 samples/s
+Step 49830/50462 | Loss: 2.5443 | LR: 1.99e-07 | 3000.1ms/step | 20.0 samples/s
+Step 49840/50462 | Loss: 2.5580 | LR: 1.93e-07 | 3001.8ms/step | 20.0 samples/s
+Step 49850/50462 | Loss: 2.5344 | LR: 1.87e-07 | 2997.7ms/step | 20.0 samples/s
+Step 49860/50462 | Loss: 2.5355 | LR: 1.81e-07 | 3000.1ms/step | 20.0 samples/s
+Step 49870/50462 | Loss: 2.5512 | LR: 1.75e-07 | 3000.4ms/step | 20.0 samples/s
+Step 49880/50462 | Loss: 2.5342 | LR: 1.69e-07 | 2998.7ms/step | 20.0 samples/s
+Step 49890/50462 | Loss: 2.5424 | LR: 1.63e-07 | 2999.3ms/step | 20.0 samples/s
+Step 49900/50462 | Loss: 2.5570 | LR: 1.58e-07 | 2996.9ms/step | 20.0 samples/s
+Step 49910/50462 | Loss: 2.5419 | LR: 1.52e-07 | 2998.5ms/step | 20.0 samples/s
+Step 49920/50462 | Loss: 2.5570 | LR: 1.47e-07 | 2997.5ms/step | 20.0 samples/s
+Step 49930/50462 | Loss: 2.5267 | LR: 1.41e-07 | 2995.1ms/step | 20.0 samples/s
+Step 49940/50462 | Loss: 2.5395 | LR: 1.36e-07 | 2998.4ms/step | 20.0 samples/s
+Step 49950/50462 | Loss: 2.4992 | LR: 1.31e-07 | 2999.0ms/step | 20.0 samples/s
+Step 49960/50462 | Loss: 2.5296 | LR: 1.26e-07 | 2997.0ms/step | 20.0 samples/s
+Step 49970/50462 | Loss: 2.5220 | LR: 1.21e-07 | 3000.2ms/step | 20.0 samples/s
+Step 49980/50462 | Loss: 2.5370 | LR: 1.16e-07 | 2999.8ms/step | 20.0 samples/s
+Step 49990/50462 | Loss: 2.5137 | LR: 1.11e-07 | 3001.3ms/step | 20.0 samples/s
+Step 50000/50462 | Loss: 2.5245 | LR: 1.07e-07 | 2998.3ms/step | 20.0 samples/s
+Validation at step 50000: Loss = 2.5530, Perplexity = 12.85
+Saved checkpoint to checkpoints/run_20260609_063834/checkpoint-50000
+Step 50010/50462 | Loss: 2.5196 | LR: 1.02e-07 | 5661.7ms/step | 10.6 samples/s
+Step 50020/50462 | Loss: 2.5292 | LR: 9.75e-08 | 2999.0ms/step | 20.0 samples/s
+Step 50030/50462 | Loss: 2.5594 | LR: 9.32e-08 | 3000.5ms/step | 20.0 samples/s
+Step 50040/50462 | Loss: 2.5876 | LR: 8.89e-08 | 2999.6ms/step | 20.0 samples/s
+Step 50050/50462 | Loss: 2.5418 | LR: 8.47e-08 | 2999.6ms/step | 20.0 samples/s
+Step 50060/50462 | Loss: 2.5354 | LR: 8.07e-08 | 3001.5ms/step | 20.0 samples/s
+Step 50070/50462 | Loss: 2.5556 | LR: 7.67e-08 | 2999.1ms/step | 20.0 samples/s
+Step 50080/50462 | Loss: 2.5278 | LR: 7.28e-08 | 2999.6ms/step | 20.0 samples/s
+Step 50090/50462 | Loss: 2.5197 | LR: 6.91e-08 | 3006.8ms/step | 20.0 samples/s
+Step 50100/50462 | Loss: 2.5263 | LR: 6.54e-08 | 2998.4ms/step | 20.0 samples/s
+Step 50110/50462 | Loss: 2.5655 | LR: 6.19e-08 | 3003.1ms/step | 20.0 samples/s
+Step 50120/50462 | Loss: 2.5473 | LR: 5.84e-08 | 3000.0ms/step | 20.0 samples/s
+Step 50130/50462 | Loss: 2.5464 | LR: 5.50e-08 | 3000.1ms/step | 20.0 samples/s
+Step 50140/50462 | Loss: 2.5645 | LR: 5.18e-08 | 3000.2ms/step | 20.0 samples/s
+Step 50150/50462 | Loss: 2.5203 | LR: 4.86e-08 | 2998.8ms/step | 20.0 samples/s
+Step 50160/50462 | Loss: 2.5365 | LR: 4.55e-08 | 2999.3ms/step | 20.0 samples/s
+Step 50170/50462 | Loss: 2.5045 | LR: 4.26e-08 | 3002.1ms/step | 20.0 samples/s
+Step 50180/50462 | Loss: 2.5092 | LR: 3.97e-08 | 3002.4ms/step | 20.0 samples/s
+Step 50190/50462 | Loss: 2.5556 | LR: 3.69e-08 | 3001.2ms/step | 20.0 samples/s
+Step 50200/50462 | Loss: 2.5263 | LR: 3.43e-08 | 3551.7ms/step | 16.9 samples/s
+Step 50210/50462 | Loss: 2.5522 | LR: 3.17e-08 | 2993.8ms/step | 20.0 samples/s
+Step 50220/50462 | Loss: 2.5455 | LR: 2.92e-08 | 2996.2ms/step | 20.0 samples/s
+Step 50230/50462 | Loss: 2.5649 | LR: 2.69e-08 | 3001.2ms/step | 20.0 samples/s
+Step 50240/50462 | Loss: 2.5284 | LR: 2.46e-08 | 4154.8ms/step | 14.4 samples/s
+Step 50250/50462 | Loss: 2.5227 | LR: 2.24e-08 | 2994.0ms/step | 20.0 samples/s
+Validation at step 50250: Loss = 2.5529, Perplexity = 12.84
+Step 50260/50462 | Loss: 2.5702 | LR: 2.04e-08 | 4596.9ms/step | 13.1 samples/s
+Step 50270/50462 | Loss: 2.5814 | LR: 1.84e-08 | 2988.2ms/step | 20.1 samples/s
+Step 50280/50462 | Loss: 2.5519 | LR: 1.65e-08 | 2987.1ms/step | 20.1 samples/s
+Step 50290/50462 | Loss: 2.5665 | LR: 1.48e-08 | 2988.6ms/step | 20.1 samples/s
+Step 50300/50462 | Loss: 2.5208 | LR: 1.31e-08 | 2988.3ms/step | 20.1 samples/s
+Step 50310/50462 | Loss: 2.5170 | LR: 1.15e-08 | 2997.8ms/step | 20.0 samples/s
+Step 50320/50462 | Loss: 2.5372 | LR: 1.01e-08 | 3000.0ms/step | 20.0 samples/s
+Step 50330/50462 | Loss: 2.5651 | LR: 8.70e-09 | 2993.2ms/step | 20.0 samples/s
+Step 50340/50462 | Loss: 2.5465 | LR: 7.43e-09 | 2989.4ms/step | 20.1 samples/s
+Step 50350/50462 | Loss: 2.5261 | LR: 6.26e-09 | 2997.8ms/step | 20.0 samples/s
+Step 50360/50462 | Loss: 2.5492 | LR: 5.19e-09 | 2989.9ms/step | 20.1 samples/s
+Step 50370/50462 | Loss: 2.5247 | LR: 4.23e-09 | 2988.2ms/step | 20.1 samples/s
+Step 50380/50462 | Loss: 2.5187 | LR: 3.36e-09 | 2989.6ms/step | 20.1 samples/s
+Step 50390/50462 | Loss: 2.5567 | LR: 2.59e-09 | 2992.5ms/step | 20.0 samples/s
+Step 50400/50462 | Loss: 2.5410 | LR: 1.92e-09 | 2990.7ms/step | 20.1 samples/s
+Step 50410/50462 | Loss: 2.5311 | LR: 1.35e-09 | 2994.4ms/step | 20.0 samples/s
+Step 50420/50462 | Loss: 2.5571 | LR: 8.81e-10 | 2992.9ms/step | 20.0 samples/s
+Step 50430/50462 | Loss: 2.5479 | LR: 5.11e-10 | 2990.7ms/step | 20.1 samples/s
+Step 50440/50462 | Loss: 2.5175 | LR: 2.42e-10 | 2992.0ms/step | 20.1 samples/s
+Step 50450/50462 | Loss: 2.5583 | LR: 7.19e-11 | 2998.5ms/step | 20.0 samples/s
+Step 50460/50462 | Loss: 2.5309 | LR: 2.00e-12 | 3000.6ms/step | 20.0 samples/s
+Saved checkpoint to checkpoints/run_20260609_063834/checkpoint-50462

epfl_final/wandb/run-20260612_172414-jh4dfz9a/files/requirements.txt

@@ -0,0 +1,122 @@
+word2number==1.1
+sqlitedict==2.1.0
+pytz==2026.2
+nvidia-cusparselt-cu13==0.8.1
+mpmath==1.3.0
+cuda-toolkit==13.0.2
+zstandard==0.25.0
+xxhash==3.7.0
+urllib3==2.7.0
+tzdata==2026.2
+typing_extensions==4.15.0
+triton==3.7.0
+tqdm==4.68.2
+torchinfo==1.8.0
+threadpoolctl==3.6.0
+tcolorpy==0.1.7
+tabulate==0.10.0
+sympy==1.14.0
+smmap==5.0.3
+six==1.17.0
+shellingham==1.5.4
+safetensors==0.8.0
+regex==2026.5.9
+PyYAML==6.0.3
+Pygments==2.20.0
+pyarrow==24.0.0
+psutil==7.2.2
+protobuf==7.35.1
+propcache==0.5.2
+portalocker==3.2.0
+platformdirs==4.10.0
+pathvalidate==3.3.1
+packaging==26.2
+nvidia-nvtx==13.0.85
+nvidia-nvshmem-cu13==3.4.5
+nvidia-nvjitlink==13.0.88
+nvidia-nccl-cu13==2.29.7
+nvidia-curand==10.4.0.35
+nvidia-cufile==1.15.1.6
+nvidia-cuda-runtime==13.0.96
+nvidia-cuda-nvrtc==13.0.88
+nvidia-cuda-cupti==13.0.85
+numpy==2.2.6
+networkx==3.4.2
+more-itertools==11.1.0
+mdurl==0.1.2
+MarkupSafe==3.0.3
+lxml==6.1.1
+joblib==1.5.3
+idna==3.18
+hf-xet==1.5.1
+h11==0.16.0
+fsspec==2026.4.0
+frozenlist==1.8.0
+filelock==3.29.3
+dill==0.4.1
+cuda-pathfinder==1.5.5
+colorama==0.4.6
+click==8.4.1
+charset-normalizer==3.4.7
+chardet==6.0.0.post1
+certifi==2026.5.20
+attrs==26.1.0
+async-timeout==5.0.1
+annotated-types==0.7.0
+annotated-doc==0.0.4
+aiohappyeyeballs==2.6.2
+absl-py==2.4.0
+typing-inspection==0.4.2
+sentry-sdk==2.62.0
+scipy==1.15.3
+sacrebleu==2.6.0
+requests==2.34.2
+python-dateutil==2.9.0.post0
+pydantic_core==2.46.4
+nvidia-cusparse==12.6.3.3
+nvidia-cufft==12.0.0.61
+nvidia-cublas==13.1.1.3
+nltk==3.9.4
+multiprocess==0.70.19
+multidict==6.7.1
+mbstrdecoder==1.1.5
+markdown-it-py==4.2.0
+Jinja2==3.1.6
+httpcore==1.0.9
+gitdb==4.0.12
+exceptiongroup==1.3.1
+cuda-bindings==13.3.1
+aiosignal==1.4.0
+yarl==1.24.2
+typepy==1.3.5
+scikit-learn==1.7.2
+rouge-score==0.1.2
+rich==15.0.0
+pydantic==2.13.4
+pandas==2.3.3
+nvidia-cusolver==12.0.4.66
+nvidia-cudnn-cu13==9.20.0.48
+GitPython==3.1.50
+anyio==4.13.0
+wandb==0.27.2
+typer==0.25.1
+httpx==0.28.1
+aiohttp==3.14.1
+torch==2.12.0
+huggingface_hub==1.19.0
+DataProperty==1.1.1
+tokenizers==0.22.2
+tabledata==1.3.5
+datasets==5.0.0
+accelerate==1.14.0
+transformers==5.12.0
+pytablewriter==1.2.1
+evaluate==0.4.6
+lm_eval==0.4.12
+PyGObject==3.42.1
+dbus-python==1.2.18
+distro==1.7.0
+pip==22.0.2
+setuptools==59.6.0
+ssh-import-id==5.11
+wheel==0.37.1

epfl_final/wandb/run-20260612_172414-jh4dfz9a/files/wandb-metadata.json

@@ -0,0 +1,54 @@
+{
+  "os": "Linux-6.8.0-124-generic-x86_64-with-glibc2.35",
+  "python": "CPython 3.10.12",
+  "startedAt": "2026-06-12T17:24:14.822511Z",
+  "args": [
+    "--config",
+    "SparseAtt/configs/train_config.json"
+  ],
+  "program": "/mloscratch/homes/navasard/sparse/SparseAtt/train.py",
+  "codePath": "SparseAtt/train.py",
+  "codePathLocal": "SparseAtt/train.py",
+  "email": "ruben.navasardyan@mail.utoronto.ca",
+  "root": "/mloscratch/homes/navasard/sparse",
+  "host": "h100-train-3-gpu-0-8",
+  "executable": "/usr/bin/python3",
+  "cpu_count": 96,
+  "cpu_count_logical": 192,
+  "gpu": "NVIDIA H100 80GB HBM3",
+  "gpu_count": 3,
+  "disk": {
+    "/": {
+      "total": "9374931091456",
+      "used": "5130988728320"
+    }
+  },
+  "memory": {
+    "total": "1623005732864"
+  },
+  "gpu_nvidia": [
+    {
+      "name": "NVIDIA H100 80GB HBM3",
+      "memoryTotal": "85520809984",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-bab1e315-c967-43cf-bdfb-cac1a8dfdfac"
+    },
+    {
+      "name": "NVIDIA H100 80GB HBM3",
+      "memoryTotal": "85520809984",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-f65cda91-ff87-9bff-fb3d-3e1b3f9f8cf0"
+    },
+    {
+      "name": "NVIDIA H100 80GB HBM3",
+      "memoryTotal": "85520809984",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-086f9f0b-a6d8-6263-83e6-c2b66721f47f"
+    }
+  ],
+  "cudaVersion": "13.0",
+  "writerId": "vka5m5pssehspy0dkt44prm0qmrb8kp6"
+}

epfl_final/wandb/run-20260612_172414-jh4dfz9a/files/wandb-summary.json

@@ -0,0 +1 @@
+{"eval/perplexity":12.843711844496752,"eval/uses_dense_attention":0,"eval/uses_sparse_attention":1,"eval/loss":2.552854340947317,"train/global_step":50460,"_wandb":{"runtime":157767},"eval/dense_interval_active":0,"speed/samples_per_sec":19.996294231532534,"speed/steps_per_sec":0.33327157052554224,"_step":50460,"train/loss":2.530939458608627,"eval/dense_interval_end":-1,"eval/dense_interval_start":-1,"train/learning_rate":1.9968584286100823e-12,"speed/ms_per_step":3000.555968284607,"_timestamp":1.7813209972941725e+09,"_runtime":157767.553225708,"train/epoch":0.9999603662161627}

epfl_final/wandb/run-20260612_172414-jh4dfz9a/logs/debug.log

@@ -0,0 +1,25 @@
+2026-06-12 17:24:14,839 INFO    MainThread:3037 [wandb_setup.py:_flush():81] Current SDK version is 0.27.2
+2026-06-12 17:24:14,839 INFO    MainThread:3037 [wandb_setup.py:_flush():81] Configure stats pid to 3037
+2026-06-12 17:24:14,840 INFO    MainThread:3037 [wandb_setup.py:_flush():81] Loading settings from environment variables
+2026-06-12 17:24:14,840 INFO    MainThread:3037 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /mloscratch/homes/navasard/sparse/wandb/run-20260612_172414-jh4dfz9a/logs/debug.log
+2026-06-12 17:24:14,841 INFO    MainThread:3037 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /mloscratch/homes/navasard/sparse/wandb/run-20260612_172414-jh4dfz9a/logs/debug-internal.log
+2026-06-12 17:24:14,841 INFO    MainThread:3037 [wandb_init.py:init():766] calling init triggers
+2026-06-12 17:24:14,842 INFO    MainThread:3037 [wandb_init.py:init():771] wandb.init called with sweep_config: {}
+config: {'model_path': '/mloscratch/homes/navasard/qwen3_06b_base', 'model_dtype': 'bfloat16', 'init_from_scratch': True, 'train_path': '/mloscratch/homes/navasard/data/fineweb_large/tokenized', 'val_path': None, 'max_seq_length': 8192, 'num_workers': 8, 'output_dir': 'checkpoints/run_20260609_063834', 'resume_from_checkpoint': 'checkpoints/run_20260609_063834/checkpoint-39000', 'force_new_run': False, 'num_epochs': 1, 'max_steps': None, 'per_device_batch_size': 2, 'gradient_accumulation_steps': 10, 'learning_rate': 0.0005, 'weight_decay': 0.01, 'warmup_steps': 750, 'lr_scheduler': 'cosine', 'max_grad_norm': 1.0, 'save_steps': 1000, 'eval_steps': 250, 'eval_batches': 50, 'logging_steps': 10, 'seed': 42, 'verbose': True, 'checkpoint_activations': False, 'strategy': 'ddp', 'mixed_precision': 'bf16', 'wandb_enabled': True, 'wandb_project': 'stochastic_sparsity', 'wandb_name': 'sparse 64%', 'wandb_tags': ['qwen3', '1.2b', 'sparse'], 'hidden_size': 1536, 'intermediate_size': 6144, 'num_attention_heads': 12, 'num_key_value_heads': 4, 'num_hidden_layers': None, 'head_dim': None, 'attention_type': 'sparse_stochastic', 'sparse_block_size': 128, 'sparse_num_random_blocks': 22, 'sparse_mask_pool_size': 10000, 'sparse_val_dense_compare': False, 'sparse_layer_indices': None, 'sparse_mask_cache_dir': 'mask_cache_23', 'dense_mode': False, 'dense_step_intervals': None, '_wandb': {}}
+2026-06-12 17:24:14,843 INFO    MainThread:3037 [wandb_init.py:init():814] starting backend
+2026-06-12 17:24:14,843 INFO    MainThread:3037 [wandb_init.py:init():829] sending inform_init request
+2026-06-12 17:24:15,199 INFO    MainThread:3037 [wandb_init.py:init():834] backend started and connected
+2026-06-12 17:24:15,206 INFO    MainThread:3037 [wandb_init.py:init():904] updated telemetry
+2026-06-12 17:24:15,207 INFO    MainThread:3037 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout
+2026-06-12 17:24:15,782 INFO    MainThread:3037 [wandb_init.py:init():967] run resumed
+2026-06-12 17:24:15,785 INFO    MainThread:3037 [wandb_init.py:init():972] starting run threads in backend
+2026-06-12 17:24:15,863 INFO    MainThread:3037 [wandb_run.py:_console_start():2523] atexit reg
+2026-06-12 17:24:15,863 INFO    MainThread:3037 [wandb_run.py:_redirect():2373] redirect: wrap_raw
+2026-06-12 17:24:15,863 INFO    MainThread:3037 [wandb_run.py:_redirect():2442] Wrapping output streams.
+2026-06-12 17:24:15,863 INFO    MainThread:3037 [wandb_run.py:_redirect():2465] Redirects installed.
+2026-06-12 17:24:15,865 INFO    MainThread:3037 [wandb_init.py:init():1010] run started, returning control to user process
+2026-06-13 03:23:36,331 INFO    MainThread:3037 [wandb_run.py:_finish():2285] finishing run ruben-navasardyan4869-epfl/stochastic_sparsity/jh4dfz9a
+2026-06-13 03:23:36,333 INFO    MainThread:3037 [wandb_run.py:_atexit_cleanup():2490] got exitcode: 0
+2026-06-13 03:23:36,334 INFO    MainThread:3037 [wandb_run.py:_restore():2472] restore
+2026-06-13 03:23:36,334 INFO    MainThread:3037 [wandb_run.py:_restore():2478] restore done
+2026-06-13 03:23:37,859 INFO    MainThread:3037 [wandb_run.py:_footer_sync_info():3891] logging synced files

epfl_final/wandb/run-20260612_172414-jh4dfz9a/run-jh4dfz9a.wandb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a95755561ec7719e947517c9839f2d60c2c04c8e0da987e9a97410bd57f03ffa
+size 10319572

epfl_final/wandb/run-20260613_222959-rp96qdel/files/config.yaml

@@ -0,0 +1,173 @@
+_wandb:
+    value:
+        cli_version: 0.27.2
+        e:
+            2d0i9uu80tocjigcv6jnloatmmzovbz6:
+                args:
+                    - --config
+                    - SparseAtt/configs/train_config.json
+                codePath: SparseAtt/train.py
+                codePathLocal: SparseAtt/train.py
+                cpu_count: 96
+                cpu_count_logical: 192
+                cudaVersion: "13.0"
+                disk:
+                    /:
+                        total: "9374931091456"
+                        used: "6429901119488"
+                email: ruben.navasardyan@mail.utoronto.ca
+                executable: /usr/bin/python3
+                gpu: NVIDIA H100 80GB HBM3
+                gpu_count: 3
+                gpu_nvidia:
+                    - architecture: Hopper
+                      cudaCores: 16896
+                      memoryTotal: "85520809984"
+                      name: NVIDIA H100 80GB HBM3
+                      uuid: GPU-b97c2df9-b38b-c4e9-4d9d-780e91b62380
+                    - architecture: Hopper
+                      cudaCores: 16896
+                      memoryTotal: "85520809984"
+                      name: NVIDIA H100 80GB HBM3
+                      uuid: GPU-337db0d9-bb90-c768-bb02-7b5dfde9a5d1
+                    - architecture: Hopper
+                      cudaCores: 16896
+                      memoryTotal: "85520809984"
+                      name: NVIDIA H100 80GB HBM3
+                      uuid: GPU-129f6620-c2e8-b738-a48f-5b9473847237
+                host: h100-train-3-gpu-0-9
+                memory:
+                    total: "1623005732864"
+                os: Linux-6.8.0-124-generic-x86_64-with-glibc2.35
+                program: /mloscratch/homes/navasard/sparse/SparseAtt/train.py
+                python: CPython 3.10.12
+                root: /mloscratch/homes/navasard/sparse
+                startedAt: "2026-06-13T22:29:59.446565Z"
+                writerId: 2d0i9uu80tocjigcv6jnloatmmzovbz6
+        m: []
+        python_version: 3.10.12
+        t:
+            "1":
+                - 1
+                - 5
+                - 11
+                - 49
+                - 51
+                - 53
+                - 71
+            "2":
+                - 1
+                - 5
+                - 11
+                - 49
+                - 51
+                - 53
+                - 71
+            "3":
+                - 13
+                - 15
+                - 16
+                - 41
+                - 61
+            "4": 3.10.12
+            "5": 0.27.2
+            "6": 5.12.0
+            "12": 0.27.2
+            "13": linux-x86_64
+attention_type:
+    value: sparse_stochastic
+checkpoint_activations:
+    value: false
+dense_mode:
+    value: false
+dense_step_intervals:
+    value: null
+eval_batches:
+    value: 50
+eval_steps:
+    value: 250
+force_new_run:
+    value: true
+gradient_accumulation_steps:
+    value: 10
+head_dim:
+    value: null
+hidden_size:
+    value: 1536
+init_from_scratch:
+    value: true
+intermediate_size:
+    value: 6144
+learning_rate:
+    value: 0.0005
+logging_steps:
+    value: 10
+lr_scheduler:
+    value: cosine
+max_grad_norm:
+    value: 1
+max_seq_length:
+    value: 8192
+max_steps:
+    value: null
+mixed_precision:
+    value: bf16
+model_dtype:
+    value: bfloat16
+model_path:
+    value: /mloscratch/homes/navasard/qwen3_06b_base
+num_attention_heads:
+    value: 12
+num_epochs:
+    value: 1
+num_hidden_layers:
+    value: null
+num_key_value_heads:
+    value: 4
+num_workers:
+    value: 8
+output_dir:
+    value: ./checkpoints/run_20260613_222732
+per_device_batch_size:
+    value: 2
+resume_from_checkpoint:
+    value: null
+save_steps:
+    value: 1000
+seed:
+    value: 42
+sparse_block_size:
+    value: 128
+sparse_layer_indices:
+    value: null
+sparse_mask_cache_dir:
+    value: mask_cache_10
+sparse_mask_pool_size:
+    value: 10000
+sparse_num_random_blocks:
+    value: 9
+sparse_val_dense_compare:
+    value: false
+strategy:
+    value: ddp
+train_path:
+    value: /mloscratch/homes/navasard/data/fineweb_large/tokenized
+val_path:
+    value: null
+verbose:
+    value: true
+wandb_enabled:
+    value: true
+wandb_name:
+    value: sparse 85%
+wandb_project:
+    value: stochastic_sparsity
+wandb_tags:
+    value:
+        - qwen3
+        - 1.2b
+        - sparse
+warmup_steps:
+    value: 750
+weight_decay:
+    value: 0.01

epfl_final/wandb/run-20260613_222959-rp96qdel/files/output.log

@@ -0,0 +1,500 @@
+Starting training...
+Switching attention mode for step 1: sparse_stochastic (configured sparse attention)
+Step 10/50462 | Loss: 12.2625 | LR: 6.67e-06 | 8182.9ms/step | 7.3 samples/s
+Step 20/50462 | Loss: 12.0390 | LR: 1.33e-05 | 2751.4ms/step | 21.8 samples/s
+Step 30/50462 | Loss: 11.2242 | LR: 2.00e-05 | 2756.9ms/step | 21.8 samples/s
+Step 40/50462 | Loss: 10.2579 | LR: 2.67e-05 | 2751.4ms/step | 21.8 samples/s
+Step 50/50462 | Loss: 9.8189 | LR: 3.33e-05 | 2750.1ms/step | 21.8 samples/s
+Step 60/50462 | Loss: 9.3643 | LR: 4.00e-05 | 2749.4ms/step | 21.8 samples/s
+Step 70/50462 | Loss: 8.8529 | LR: 4.67e-05 | 2749.5ms/step | 21.8 samples/s
+Step 80/50462 | Loss: 8.3457 | LR: 5.33e-05 | 2750.8ms/step | 21.8 samples/s
+Step 90/50462 | Loss: 7.9054 | LR: 6.00e-05 | 2753.4ms/step | 21.8 samples/s
+Step 100/50462 | Loss: 7.5376 | LR: 6.67e-05 | 2751.6ms/step | 21.8 samples/s
+Step 110/50462 | Loss: 7.2801 | LR: 7.33e-05 | 2754.5ms/step | 21.8 samples/s
+Step 120/50462 | Loss: 7.1211 | LR: 8.00e-05 | 2758.6ms/step | 21.7 samples/s
+Step 130/50462 | Loss: 7.0000 | LR: 8.67e-05 | 2761.6ms/step | 21.7 samples/s
+Step 140/50462 | Loss: 6.8729 | LR: 9.33e-05 | 2761.3ms/step | 21.7 samples/s
+Step 150/50462 | Loss: 6.7415 | LR: 1.00e-04 | 2760.7ms/step | 21.7 samples/s
+Step 160/50462 | Loss: 6.6354 | LR: 1.07e-04 | 2761.9ms/step | 21.7 samples/s
+Step 170/50462 | Loss: 6.5196 | LR: 1.13e-04 | 2759.5ms/step | 21.7 samples/s
+Step 180/50462 | Loss: 6.4332 | LR: 1.20e-04 | 2759.1ms/step | 21.7 samples/s
+Step 190/50462 | Loss: 6.3651 | LR: 1.27e-04 | 2762.1ms/step | 21.7 samples/s
+Step 200/50462 | Loss: 6.2856 | LR: 1.33e-04 | 2765.8ms/step | 21.7 samples/s
+Step 210/50462 | Loss: 6.2157 | LR: 1.40e-04 | 2763.2ms/step | 21.7 samples/s
+Step 220/50462 | Loss: 6.1514 | LR: 1.47e-04 | 2764.4ms/step | 21.7 samples/s
+Step 230/50462 | Loss: 6.1246 | LR: 1.53e-04 | 2764.8ms/step | 21.7 samples/s
+Step 240/50462 | Loss: 6.0809 | LR: 1.60e-04 | 2765.7ms/step | 21.7 samples/s
+Step 250/50462 | Loss: 6.0134 | LR: 1.67e-04 | 2766.9ms/step | 21.7 samples/s
+Validation at step 250: Loss = 5.9597, Perplexity = 387.50
+Step 260/50462 | Loss: 5.9535 | LR: 1.73e-04 | 6912.4ms/step | 8.7 samples/s
+Step 270/50462 | Loss: 5.9490 | LR: 1.80e-04 | 2761.4ms/step | 21.7 samples/s
+Step 280/50462 | Loss: 5.8735 | LR: 1.87e-04 | 2759.1ms/step | 21.7 samples/s
+Step 290/50462 | Loss: 5.8079 | LR: 1.93e-04 | 2761.0ms/step | 21.7 samples/s
+Step 300/50462 | Loss: 5.7372 | LR: 2.00e-04 | 2762.7ms/step | 21.7 samples/s
+Step 310/50462 | Loss: 5.6924 | LR: 2.07e-04 | 2761.0ms/step | 21.7 samples/s
+Step 320/50462 | Loss: 5.6409 | LR: 2.13e-04 | 2760.7ms/step | 21.7 samples/s
+Step 330/50462 | Loss: 5.6141 | LR: 2.20e-04 | 2760.6ms/step | 21.7 samples/s
+Step 340/50462 | Loss: 5.5656 | LR: 2.27e-04 | 2760.9ms/step | 21.7 samples/s
+Step 350/50462 | Loss: 5.5115 | LR: 2.33e-04 | 2759.4ms/step | 21.7 samples/s
+Step 360/50462 | Loss: 5.4540 | LR: 2.40e-04 | 2760.6ms/step | 21.7 samples/s
+Step 370/50462 | Loss: 5.3852 | LR: 2.47e-04 | 2760.4ms/step | 21.7 samples/s
+Step 380/50462 | Loss: 5.3676 | LR: 2.53e-04 | 2761.4ms/step | 21.7 samples/s
+Step 390/50462 | Loss: 5.3260 | LR: 2.60e-04 | 2759.9ms/step | 21.7 samples/s
+Step 400/50462 | Loss: 5.2715 | LR: 2.67e-04 | 2760.3ms/step | 21.7 samples/s
+Step 410/50462 | Loss: 5.2183 | LR: 2.73e-04 | 2759.8ms/step | 21.7 samples/s
+Step 420/50462 | Loss: 5.1942 | LR: 2.80e-04 | 2761.0ms/step | 21.7 samples/s
+Step 430/50462 | Loss: 5.1278 | LR: 2.87e-04 | 2761.1ms/step | 21.7 samples/s
+Step 440/50462 | Loss: 5.1252 | LR: 2.93e-04 | 2759.0ms/step | 21.7 samples/s
+Step 450/50462 | Loss: 5.0873 | LR: 3.00e-04 | 2758.1ms/step | 21.8 samples/s
+Step 460/50462 | Loss: 5.0026 | LR: 3.07e-04 | 2756.5ms/step | 21.8 samples/s
+Step 470/50462 | Loss: 5.0290 | LR: 3.13e-04 | 2759.6ms/step | 21.7 samples/s
+Step 480/50462 | Loss: 5.0300 | LR: 3.20e-04 | 2760.6ms/step | 21.7 samples/s
+Step 490/50462 | Loss: 4.9552 | LR: 3.27e-04 | 2760.2ms/step | 21.7 samples/s
+Step 500/50462 | Loss: 4.9114 | LR: 3.33e-04 | 2759.2ms/step | 21.7 samples/s
+Validation at step 500: Loss = 4.8836, Perplexity = 132.10
+Step 510/50462 | Loss: 4.9205 | LR: 3.40e-04 | 4253.1ms/step | 14.1 samples/s
+Step 520/50462 | Loss: 4.8721 | LR: 3.47e-04 | 2759.4ms/step | 21.7 samples/s
+Step 530/50462 | Loss: 4.8278 | LR: 3.53e-04 | 2758.7ms/step | 21.7 samples/s
+Step 540/50462 | Loss: 4.7683 | LR: 3.60e-04 | 2756.9ms/step | 21.8 samples/s
+Step 550/50462 | Loss: 4.8274 | LR: 3.67e-04 | 2758.3ms/step | 21.8 samples/s
+Step 560/50462 | Loss: 4.7601 | LR: 3.73e-04 | 2758.6ms/step | 21.7 samples/s
+Step 570/50462 | Loss: 4.6603 | LR: 3.80e-04 | 2757.7ms/step | 21.8 samples/s
+Step 580/50462 | Loss: 4.6139 | LR: 3.87e-04 | 2758.0ms/step | 21.8 samples/s
+Step 590/50462 | Loss: 4.7223 | LR: 3.93e-04 | 2757.1ms/step | 21.8 samples/s
+Step 600/50462 | Loss: 4.7198 | LR: 4.00e-04 | 2757.4ms/step | 21.8 samples/s
+Step 610/50462 | Loss: 4.5945 | LR: 4.07e-04 | 3350.6ms/step | 17.9 samples/s
+Step 620/50462 | Loss: 4.5770 | LR: 4.13e-04 | 2751.4ms/step | 21.8 samples/s
+Step 630/50462 | Loss: 4.5489 | LR: 4.20e-04 | 2755.7ms/step | 21.8 samples/s
+Step 640/50462 | Loss: 4.5989 | LR: 4.27e-04 | 3351.4ms/step | 17.9 samples/s
+Step 650/50462 | Loss: 4.5378 | LR: 4.33e-04 | 2757.0ms/step | 21.8 samples/s
+Step 660/50462 | Loss: 4.4513 | LR: 4.40e-04 | 2759.7ms/step | 21.7 samples/s
+Step 670/50462 | Loss: 4.4261 | LR: 4.47e-04 | 2759.6ms/step | 21.7 samples/s
+Step 680/50462 | Loss: 4.4397 | LR: 4.53e-04 | 2759.4ms/step | 21.7 samples/s
+Step 690/50462 | Loss: 4.4345 | LR: 4.60e-04 | 2758.6ms/step | 21.8 samples/s
+Step 700/50462 | Loss: 4.4282 | LR: 4.67e-04 | 2756.7ms/step | 21.8 samples/s
+Step 710/50462 | Loss: 4.3494 | LR: 4.73e-04 | 2759.1ms/step | 21.7 samples/s
+Step 720/50462 | Loss: 4.3021 | LR: 4.80e-04 | 2758.6ms/step | 21.7 samples/s
+Step 730/50462 | Loss: 4.3007 | LR: 4.87e-04 | 2761.2ms/step | 21.7 samples/s
+Step 740/50462 | Loss: 4.2722 | LR: 4.93e-04 | 2761.0ms/step | 21.7 samples/s
+Step 750/50462 | Loss: 4.2957 | LR: 5.00e-04 | 2760.2ms/step | 21.7 samples/s
+Validation at step 750: Loss = 4.2539, Perplexity = 70.38
+Step 760/50462 | Loss: 4.2268 | LR: 5.00e-04 | 4247.4ms/step | 14.1 samples/s
+Step 770/50462 | Loss: 4.2373 | LR: 5.00e-04 | 2755.4ms/step | 21.8 samples/s
+Step 780/50462 | Loss: 4.1921 | LR: 5.00e-04 | 2756.7ms/step | 21.8 samples/s
+Step 790/50462 | Loss: 4.1957 | LR: 5.00e-04 | 2755.7ms/step | 21.8 samples/s
+Step 800/50462 | Loss: 4.1411 | LR: 5.00e-04 | 2759.5ms/step | 21.7 samples/s
+Step 810/50462 | Loss: 4.1465 | LR: 5.00e-04 | 2758.6ms/step | 21.7 samples/s
+Step 820/50462 | Loss: 4.1310 | LR: 5.00e-04 | 2757.7ms/step | 21.8 samples/s
+Step 830/50462 | Loss: 4.1071 | LR: 5.00e-04 | 2753.5ms/step | 21.8 samples/s
+Step 840/50462 | Loss: 4.0836 | LR: 5.00e-04 | 2758.5ms/step | 21.8 samples/s
+Step 850/50462 | Loss: 4.0653 | LR: 5.00e-04 | 2759.2ms/step | 21.7 samples/s
+Step 860/50462 | Loss: 4.0364 | LR: 5.00e-04 | 2757.6ms/step | 21.8 samples/s
+Step 870/50462 | Loss: 4.0130 | LR: 5.00e-04 | 2759.9ms/step | 21.7 samples/s
+Step 880/50462 | Loss: 3.9941 | LR: 5.00e-04 | 2756.1ms/step | 21.8 samples/s
+Step 890/50462 | Loss: 3.9949 | LR: 5.00e-04 | 2758.7ms/step | 21.7 samples/s
+Step 900/50462 | Loss: 3.9825 | LR: 5.00e-04 | 2759.2ms/step | 21.7 samples/s
+Step 910/50462 | Loss: 3.9619 | LR: 5.00e-04 | 2759.3ms/step | 21.7 samples/s
+Step 920/50462 | Loss: 3.9121 | LR: 5.00e-04 | 2758.2ms/step | 21.8 samples/s
+Step 930/50462 | Loss: 3.9619 | LR: 5.00e-04 | 2760.1ms/step | 21.7 samples/s
+Step 940/50462 | Loss: 3.9100 | LR: 5.00e-04 | 2756.4ms/step | 21.8 samples/s
+Step 950/50462 | Loss: 3.8990 | LR: 5.00e-04 | 2757.9ms/step | 21.8 samples/s
+Step 960/50462 | Loss: 3.8983 | LR: 5.00e-04 | 2756.9ms/step | 21.8 samples/s
+Step 970/50462 | Loss: 3.8782 | LR: 5.00e-04 | 2759.8ms/step | 21.7 samples/s
+Step 980/50462 | Loss: 3.8480 | LR: 5.00e-04 | 2759.5ms/step | 21.7 samples/s
+Step 990/50462 | Loss: 3.8627 | LR: 5.00e-04 | 2759.6ms/step | 21.7 samples/s
+Step 1000/50462 | Loss: 3.8628 | LR: 5.00e-04 | 2758.8ms/step | 21.7 samples/s
+Validation at step 1000: Loss = 3.8567, Perplexity = 47.31
+Saved checkpoint to checkpoints/run_20260613_222732/checkpoint-1000
+Step 1010/50462 | Loss: 3.8364 | LR: 5.00e-04 | 5335.5ms/step | 11.2 samples/s
+Step 1020/50462 | Loss: 3.8440 | LR: 5.00e-04 | 2758.6ms/step | 21.8 samples/s
+Step 1030/50462 | Loss: 3.8335 | LR: 5.00e-04 | 2758.5ms/step | 21.8 samples/s
+Step 1040/50462 | Loss: 3.8327 | LR: 5.00e-04 | 2757.2ms/step | 21.8 samples/s
+Step 1050/50462 | Loss: 3.8008 | LR: 5.00e-04 | 2755.6ms/step | 21.8 samples/s
+Step 1060/50462 | Loss: 3.7957 | LR: 5.00e-04 | 2757.5ms/step | 21.8 samples/s
+Step 1070/50462 | Loss: 3.7822 | LR: 5.00e-04 | 2755.4ms/step | 21.8 samples/s
+Step 1080/50462 | Loss: 3.7829 | LR: 5.00e-04 | 2755.4ms/step | 21.8 samples/s
+Step 1090/50462 | Loss: 3.7583 | LR: 5.00e-04 | 2757.7ms/step | 21.8 samples/s
+Step 1100/50462 | Loss: 3.7709 | LR: 5.00e-04 | 2754.8ms/step | 21.8 samples/s
+Step 1110/50462 | Loss: 3.7790 | LR: 5.00e-04 | 2753.9ms/step | 21.8 samples/s
+Step 1120/50462 | Loss: 3.7341 | LR: 5.00e-04 | 2755.7ms/step | 21.8 samples/s
+Step 1130/50462 | Loss: 3.7454 | LR: 5.00e-04 | 2754.9ms/step | 21.8 samples/s
+Step 1140/50462 | Loss: 3.7167 | LR: 5.00e-04 | 2754.6ms/step | 21.8 samples/s
+Step 1150/50462 | Loss: 3.6859 | LR: 5.00e-04 | 2754.8ms/step | 21.8 samples/s
+Step 1160/50462 | Loss: 3.7077 | LR: 5.00e-04 | 2755.9ms/step | 21.8 samples/s
+Step 1170/50462 | Loss: 3.6990 | LR: 5.00e-04 | 2757.6ms/step | 21.8 samples/s
+Step 1180/50462 | Loss: 3.6600 | LR: 5.00e-04 | 2754.0ms/step | 21.8 samples/s
+Step 1190/50462 | Loss: 3.6831 | LR: 5.00e-04 | 2752.8ms/step | 21.8 samples/s
+Step 1200/50462 | Loss: 3.6980 | LR: 5.00e-04 | 2756.9ms/step | 21.8 samples/s
+Step 1210/50462 | Loss: 3.6939 | LR: 5.00e-04 | 2754.9ms/step | 21.8 samples/s
+Step 1220/50462 | Loss: 3.6551 | LR: 5.00e-04 | 2753.8ms/step | 21.8 samples/s
+Step 1230/50462 | Loss: 3.6261 | LR: 5.00e-04 | 2754.7ms/step | 21.8 samples/s
+Step 1240/50462 | Loss: 3.6812 | LR: 5.00e-04 | 2754.7ms/step | 21.8 samples/s
+Step 1250/50462 | Loss: 3.6441 | LR: 5.00e-04 | 2755.0ms/step | 21.8 samples/s
+Validation at step 1250: Loss = 3.6542, Perplexity = 38.64
+Step 1260/50462 | Loss: 3.6657 | LR: 5.00e-04 | 4239.3ms/step | 14.2 samples/s
+Step 1270/50462 | Loss: 3.6287 | LR: 5.00e-04 | 2756.0ms/step | 21.8 samples/s
+Step 1280/50462 | Loss: 3.6062 | LR: 5.00e-04 | 2758.2ms/step | 21.8 samples/s
+Step 1290/50462 | Loss: 3.6149 | LR: 5.00e-04 | 2756.7ms/step | 21.8 samples/s
+Step 1300/50462 | Loss: 3.6040 | LR: 5.00e-04 | 2757.3ms/step | 21.8 samples/s
+Step 1310/50462 | Loss: 3.6104 | LR: 5.00e-04 | 2758.2ms/step | 21.8 samples/s
+Step 1320/50462 | Loss: 3.6035 | LR: 5.00e-04 | 2755.7ms/step | 21.8 samples/s
+Step 1330/50462 | Loss: 3.5944 | LR: 5.00e-04 | 2757.9ms/step | 21.8 samples/s
+Step 1340/50462 | Loss: 3.5838 | LR: 5.00e-04 | 2753.3ms/step | 21.8 samples/s
+Step 1350/50462 | Loss: 3.5655 | LR: 5.00e-04 | 2752.6ms/step | 21.8 samples/s
+Step 1360/50462 | Loss: 3.5793 | LR: 5.00e-04 | 2759.1ms/step | 21.7 samples/s
+Step 1370/50462 | Loss: 3.5590 | LR: 5.00e-04 | 2760.2ms/step | 21.7 samples/s
+Step 1380/50462 | Loss: 3.5983 | LR: 5.00e-04 | 2757.8ms/step | 21.8 samples/s
+Step 1390/50462 | Loss: 3.5783 | LR: 5.00e-04 | 2750.2ms/step | 21.8 samples/s
+Step 1400/50462 | Loss: 3.5413 | LR: 5.00e-04 | 2751.5ms/step | 21.8 samples/s
+Step 1410/50462 | Loss: 3.5678 | LR: 5.00e-04 | 2750.0ms/step | 21.8 samples/s
+Step 1420/50462 | Loss: 3.5473 | LR: 5.00e-04 | 2751.2ms/step | 21.8 samples/s
+Step 1430/50462 | Loss: 3.5228 | LR: 5.00e-04 | 2751.3ms/step | 21.8 samples/s
+Step 1440/50462 | Loss: 3.5522 | LR: 5.00e-04 | 2750.7ms/step | 21.8 samples/s
+Step 1450/50462 | Loss: 3.5181 | LR: 5.00e-04 | 2750.4ms/step | 21.8 samples/s
+Step 1460/50462 | Loss: 3.5244 | LR: 5.00e-04 | 2752.1ms/step | 21.8 samples/s
+Step 1470/50462 | Loss: 3.4998 | LR: 5.00e-04 | 2749.7ms/step | 21.8 samples/s
+Step 1480/50462 | Loss: 3.5392 | LR: 5.00e-04 | 2749.8ms/step | 21.8 samples/s
+Step 1490/50462 | Loss: 3.5190 | LR: 5.00e-04 | 2754.1ms/step | 21.8 samples/s
+Step 1500/50462 | Loss: 3.4885 | LR: 5.00e-04 | 2752.7ms/step | 21.8 samples/s
+Validation at step 1500: Loss = 3.5274, Perplexity = 34.03
+Step 1510/50462 | Loss: 3.5219 | LR: 5.00e-04 | 4239.9ms/step | 14.2 samples/s
+Step 1520/50462 | Loss: 3.4904 | LR: 5.00e-04 | 2756.5ms/step | 21.8 samples/s
+Step 1530/50462 | Loss: 3.5169 | LR: 5.00e-04 | 2756.5ms/step | 21.8 samples/s
+Step 1540/50462 | Loss: 3.4818 | LR: 5.00e-04 | 2756.4ms/step | 21.8 samples/s
+Step 1550/50462 | Loss: 3.4889 | LR: 5.00e-04 | 2755.1ms/step | 21.8 samples/s
+Step 1560/50462 | Loss: 3.4817 | LR: 5.00e-04 | 2754.7ms/step | 21.8 samples/s
+Step 1570/50462 | Loss: 3.5134 | LR: 5.00e-04 | 2756.2ms/step | 21.8 samples/s
+Step 1580/50462 | Loss: 3.4805 | LR: 5.00e-04 | 2757.0ms/step | 21.8 samples/s
+Step 1590/50462 | Loss: 3.4989 | LR: 5.00e-04 | 2755.5ms/step | 21.8 samples/s
+Step 1600/50462 | Loss: 3.4897 | LR: 5.00e-04 | 2756.5ms/step | 21.8 samples/s
+Step 1610/50462 | Loss: 3.4518 | LR: 5.00e-04 | 2756.2ms/step | 21.8 samples/s
+Step 1620/50462 | Loss: 3.4727 | LR: 5.00e-04 | 2756.0ms/step | 21.8 samples/s
+Step 1630/50462 | Loss: 3.4882 | LR: 5.00e-04 | 2755.3ms/step | 21.8 samples/s
+Step 1640/50462 | Loss: 3.4619 | LR: 5.00e-04 | 2755.1ms/step | 21.8 samples/s
+Step 1650/50462 | Loss: 3.4196 | LR: 5.00e-04 | 2754.2ms/step | 21.8 samples/s
+Step 1660/50462 | Loss: 3.4413 | LR: 5.00e-04 | 2757.2ms/step | 21.8 samples/s
+Step 1670/50462 | Loss: 3.4653 | LR: 5.00e-04 | 2755.3ms/step | 21.8 samples/s
+Step 1680/50462 | Loss: 3.4382 | LR: 5.00e-04 | 2757.2ms/step | 21.8 samples/s
+Step 1690/50462 | Loss: 3.4365 | LR: 5.00e-04 | 2757.5ms/step | 21.8 samples/s
+Step 1700/50462 | Loss: 3.4450 | LR: 5.00e-04 | 2754.2ms/step | 21.8 samples/s
+Step 1710/50462 | Loss: 3.4607 | LR: 5.00e-04 | 2753.4ms/step | 21.8 samples/s
+Step 1720/50462 | Loss: 3.4240 | LR: 5.00e-04 | 2753.0ms/step | 21.8 samples/s
+Step 1730/50462 | Loss: 3.4246 | LR: 5.00e-04 | 2758.0ms/step | 21.8 samples/s
+Step 1740/50462 | Loss: 3.4535 | LR: 5.00e-04 | 2757.1ms/step | 21.8 samples/s
+Step 1750/50462 | Loss: 3.4654 | LR: 5.00e-04 | 2755.9ms/step | 21.8 samples/s
+Validation at step 1750: Loss = 3.4314, Perplexity = 30.92
+Step 1760/50462 | Loss: 3.4416 | LR: 4.99e-04 | 4244.7ms/step | 14.1 samples/s
+Step 1770/50462 | Loss: 3.4231 | LR: 4.99e-04 | 2757.2ms/step | 21.8 samples/s
+Step 1780/50462 | Loss: 3.3943 | LR: 4.99e-04 | 2756.0ms/step | 21.8 samples/s
+Step 1790/50462 | Loss: 3.3960 | LR: 4.99e-04 | 2753.5ms/step | 21.8 samples/s
+Step 1800/50462 | Loss: 3.3913 | LR: 4.99e-04 | 2755.6ms/step | 21.8 samples/s
+Step 1810/50462 | Loss: 3.4023 | LR: 4.99e-04 | 2754.3ms/step | 21.8 samples/s
+Step 1820/50462 | Loss: 3.3828 | LR: 4.99e-04 | 2755.6ms/step | 21.8 samples/s
+Step 1830/50462 | Loss: 3.3855 | LR: 4.99e-04 | 2755.8ms/step | 21.8 samples/s
+Step 1840/50462 | Loss: 3.3854 | LR: 4.99e-04 | 2757.8ms/step | 21.8 samples/s
+Step 1850/50462 | Loss: 3.3936 | LR: 4.99e-04 | 2758.2ms/step | 21.8 samples/s
+Step 1860/50462 | Loss: 3.3723 | LR: 4.99e-04 | 2754.1ms/step | 21.8 samples/s
+Step 1870/50462 | Loss: 3.4110 | LR: 4.99e-04 | 2752.6ms/step | 21.8 samples/s
+Step 1880/50462 | Loss: 3.3946 | LR: 4.99e-04 | 2755.1ms/step | 21.8 samples/s
+Step 1890/50462 | Loss: 3.3591 | LR: 4.99e-04 | 2754.6ms/step | 21.8 samples/s
+Step 1900/50462 | Loss: 3.3619 | LR: 4.99e-04 | 2752.5ms/step | 21.8 samples/s
+Step 1910/50462 | Loss: 3.3959 | LR: 4.99e-04 | 2753.5ms/step | 21.8 samples/s
+Step 1920/50462 | Loss: 3.3929 | LR: 4.99e-04 | 2755.1ms/step | 21.8 samples/s
+Step 1930/50462 | Loss: 3.3972 | LR: 4.99e-04 | 2748.4ms/step | 21.8 samples/s
+Step 1940/50462 | Loss: 3.3899 | LR: 4.99e-04 | 2755.8ms/step | 21.8 samples/s
+Step 1950/50462 | Loss: 3.3772 | LR: 4.99e-04 | 2753.1ms/step | 21.8 samples/s
+Step 1960/50462 | Loss: 3.3742 | LR: 4.99e-04 | 2752.0ms/step | 21.8 samples/s
+Step 1970/50462 | Loss: 3.3719 | LR: 4.99e-04 | 2753.3ms/step | 21.8 samples/s
+Step 1980/50462 | Loss: 3.3642 | LR: 4.99e-04 | 2752.9ms/step | 21.8 samples/s
+Step 1990/50462 | Loss: 3.3920 | LR: 4.99e-04 | 2751.7ms/step | 21.8 samples/s
+Step 2000/50462 | Loss: 3.3910 | LR: 4.99e-04 | 2751.7ms/step | 21.8 samples/s
+Validation at step 2000: Loss = 3.3635, Perplexity = 28.89
+Saved checkpoint to checkpoints/run_20260613_222732/checkpoint-2000
+Step 2010/50462 | Loss: 3.3737 | LR: 4.99e-04 | 5303.6ms/step | 11.3 samples/s
+Step 2020/50462 | Loss: 3.3731 | LR: 4.99e-04 | 2749.0ms/step | 21.8 samples/s
+Step 2030/50462 | Loss: 3.3423 | LR: 4.99e-04 | 2750.6ms/step | 21.8 samples/s
+Step 2040/50462 | Loss: 3.3541 | LR: 4.99e-04 | 2754.8ms/step | 21.8 samples/s
+Step 2050/50462 | Loss: 3.3236 | LR: 4.99e-04 | 2753.4ms/step | 21.8 samples/s
+Step 2060/50462 | Loss: 3.3272 | LR: 4.99e-04 | 2754.1ms/step | 21.8 samples/s
+Step 2070/50462 | Loss: 3.3468 | LR: 4.99e-04 | 2755.6ms/step | 21.8 samples/s
+Step 2080/50462 | Loss: 3.3361 | LR: 4.99e-04 | 2752.7ms/step | 21.8 samples/s
+Step 2090/50462 | Loss: 3.3368 | LR: 4.99e-04 | 2752.5ms/step | 21.8 samples/s
+Step 2100/50462 | Loss: 3.3238 | LR: 4.99e-04 | 2754.0ms/step | 21.8 samples/s
+Step 2110/50462 | Loss: 3.3676 | LR: 4.99e-04 | 2754.5ms/step | 21.8 samples/s
+Step 2120/50462 | Loss: 3.3574 | LR: 4.99e-04 | 2751.1ms/step | 21.8 samples/s
+Step 2130/50462 | Loss: 3.3197 | LR: 4.99e-04 | 2754.1ms/step | 21.8 samples/s
+Step 2140/50462 | Loss: 3.3035 | LR: 4.99e-04 | 2754.4ms/step | 21.8 samples/s
+Step 2150/50462 | Loss: 3.3093 | LR: 4.99e-04 | 2751.5ms/step | 21.8 samples/s
+Step 2160/50462 | Loss: 3.3205 | LR: 4.99e-04 | 2752.5ms/step | 21.8 samples/s
+Step 2170/50462 | Loss: 3.3091 | LR: 4.99e-04 | 2753.6ms/step | 21.8 samples/s
+Step 2180/50462 | Loss: 3.3326 | LR: 4.99e-04 | 2754.3ms/step | 21.8 samples/s
+Step 2190/50462 | Loss: 3.3087 | LR: 4.99e-04 | 2754.1ms/step | 21.8 samples/s
+Step 2200/50462 | Loss: 3.3012 | LR: 4.99e-04 | 2752.7ms/step | 21.8 samples/s
+Step 2210/50462 | Loss: 3.3139 | LR: 4.99e-04 | 2755.2ms/step | 21.8 samples/s
+Step 2220/50462 | Loss: 3.2971 | LR: 4.99e-04 | 2753.7ms/step | 21.8 samples/s
+Step 2230/50462 | Loss: 3.3110 | LR: 4.99e-04 | 2752.6ms/step | 21.8 samples/s
+Step 2240/50462 | Loss: 3.3018 | LR: 4.99e-04 | 2756.0ms/step | 21.8 samples/s
+Step 2250/50462 | Loss: 3.3460 | LR: 4.99e-04 | 2755.3ms/step | 21.8 samples/s
+Validation at step 2250: Loss = 3.3093, Perplexity = 27.37
+Step 2260/50462 | Loss: 3.3055 | LR: 4.99e-04 | 4240.8ms/step | 14.1 samples/s
+Step 2270/50462 | Loss: 3.3194 | LR: 4.99e-04 | 2754.1ms/step | 21.8 samples/s
+Step 2280/50462 | Loss: 3.2712 | LR: 4.99e-04 | 2754.0ms/step | 21.8 samples/s
+Step 2290/50462 | Loss: 3.2688 | LR: 4.99e-04 | 2753.0ms/step | 21.8 samples/s
+Step 2300/50462 | Loss: 3.2727 | LR: 4.99e-04 | 2758.6ms/step | 21.8 samples/s
+Step 2310/50462 | Loss: 3.2953 | LR: 4.99e-04 | 2757.3ms/step | 21.8 samples/s
+Step 2320/50462 | Loss: 3.2905 | LR: 4.99e-04 | 2754.0ms/step | 21.8 samples/s
+Step 2330/50462 | Loss: 3.2547 | LR: 4.99e-04 | 2754.7ms/step | 21.8 samples/s
+Step 2340/50462 | Loss: 3.2857 | LR: 4.99e-04 | 2753.4ms/step | 21.8 samples/s
+Step 2350/50462 | Loss: 3.2591 | LR: 4.99e-04 | 2753.4ms/step | 21.8 samples/s
+Step 2360/50462 | Loss: 3.2575 | LR: 4.99e-04 | 2755.5ms/step | 21.8 samples/s
+Step 2370/50462 | Loss: 3.2791 | LR: 4.99e-04 | 2752.6ms/step | 21.8 samples/s
+Step 2380/50462 | Loss: 3.2312 | LR: 4.99e-04 | 2757.2ms/step | 21.8 samples/s
+Step 2390/50462 | Loss: 3.2538 | LR: 4.99e-04 | 2752.6ms/step | 21.8 samples/s
+Step 2400/50462 | Loss: 3.2761 | LR: 4.99e-04 | 2754.2ms/step | 21.8 samples/s
+Step 2410/50462 | Loss: 3.2397 | LR: 4.99e-04 | 2753.6ms/step | 21.8 samples/s
+Step 2420/50462 | Loss: 3.2826 | LR: 4.99e-04 | 2753.9ms/step | 21.8 samples/s
+Step 2430/50462 | Loss: 3.2701 | LR: 4.99e-04 | 2753.9ms/step | 21.8 samples/s
+Step 2440/50462 | Loss: 3.2825 | LR: 4.99e-04 | 2754.0ms/step | 21.8 samples/s
+Step 2450/50462 | Loss: 3.3077 | LR: 4.99e-04 | 2754.3ms/step | 21.8 samples/s
+Step 2460/50462 | Loss: 3.2552 | LR: 4.99e-04 | 2754.5ms/step | 21.8 samples/s
+Step 2470/50462 | Loss: 3.2625 | LR: 4.99e-04 | 2750.3ms/step | 21.8 samples/s
+Step 2480/50462 | Loss: 3.2772 | LR: 4.99e-04 | 2749.9ms/step | 21.8 samples/s
+Step 2490/50462 | Loss: 3.2593 | LR: 4.98e-04 | 2750.2ms/step | 21.8 samples/s
+Step 2500/50462 | Loss: 3.2519 | LR: 4.98e-04 | 2752.0ms/step | 21.8 samples/s
+Validation at step 2500: Loss = 3.2614, Perplexity = 26.09
+Step 2510/50462 | Loss: 3.2464 | LR: 4.98e-04 | 4242.9ms/step | 14.1 samples/s
+Step 2520/50462 | Loss: 3.2366 | LR: 4.98e-04 | 2755.4ms/step | 21.8 samples/s
+Step 2530/50462 | Loss: 3.2379 | LR: 4.98e-04 | 2752.0ms/step | 21.8 samples/s
+Step 2540/50462 | Loss: 3.2605 | LR: 4.98e-04 | 2752.4ms/step | 21.8 samples/s
+Step 2550/50462 | Loss: 3.2680 | LR: 4.98e-04 | 2750.5ms/step | 21.8 samples/s
+Step 2560/50462 | Loss: 3.2758 | LR: 4.98e-04 | 2752.0ms/step | 21.8 samples/s
+Step 2570/50462 | Loss: 3.2169 | LR: 4.98e-04 | 2752.2ms/step | 21.8 samples/s
+Step 2580/50462 | Loss: 3.2281 | LR: 4.98e-04 | 2751.8ms/step | 21.8 samples/s
+Step 2590/50462 | Loss: 3.2566 | LR: 4.98e-04 | 2750.7ms/step | 21.8 samples/s
+Step 2600/50462 | Loss: 3.2153 | LR: 4.98e-04 | 2752.6ms/step | 21.8 samples/s
+Step 2610/50462 | Loss: 3.2294 | LR: 4.98e-04 | 2752.3ms/step | 21.8 samples/s
+Step 2620/50462 | Loss: 3.2509 | LR: 4.98e-04 | 2754.3ms/step | 21.8 samples/s
+Step 2630/50462 | Loss: 3.2172 | LR: 4.98e-04 | 2751.2ms/step | 21.8 samples/s
+Step 2640/50462 | Loss: 3.2446 | LR: 4.98e-04 | 2750.6ms/step | 21.8 samples/s
+Step 2650/50462 | Loss: 3.2413 | LR: 4.98e-04 | 2753.6ms/step | 21.8 samples/s
+Step 2660/50462 | Loss: 3.2172 | LR: 4.98e-04 | 2749.9ms/step | 21.8 samples/s
+Step 2670/50462 | Loss: 3.2744 | LR: 4.98e-04 | 2748.6ms/step | 21.8 samples/s
+Step 2680/50462 | Loss: 3.2197 | LR: 4.98e-04 | 2751.7ms/step | 21.8 samples/s
+Step 2690/50462 | Loss: 3.2382 | LR: 4.98e-04 | 2752.3ms/step | 21.8 samples/s
+Step 2700/50462 | Loss: 3.2237 | LR: 4.98e-04 | 2751.2ms/step | 21.8 samples/s
+Step 2710/50462 | Loss: 3.2030 | LR: 4.98e-04 | 2753.0ms/step | 21.8 samples/s
+Step 2720/50462 | Loss: 3.2118 | LR: 4.98e-04 | 2754.1ms/step | 21.8 samples/s
+Step 2730/50462 | Loss: 3.2279 | LR: 4.98e-04 | 3265.0ms/step | 18.4 samples/s
+Step 2740/50462 | Loss: 3.1963 | LR: 4.98e-04 | 2748.0ms/step | 21.8 samples/s
+Step 2750/50462 | Loss: 3.2225 | LR: 4.98e-04 | 2750.9ms/step | 21.8 samples/s
+Validation at step 2750: Loss = 3.2229, Perplexity = 25.10
+Step 2760/50462 | Loss: 3.2236 | LR: 4.98e-04 | 4236.9ms/step | 14.2 samples/s
+Step 2770/50462 | Loss: 3.2081 | LR: 4.98e-04 | 2752.8ms/step | 21.8 samples/s
+Step 2780/50462 | Loss: 3.2147 | LR: 4.98e-04 | 3289.5ms/step | 18.2 samples/s
+Step 2790/50462 | Loss: 3.1886 | LR: 4.98e-04 | 2750.6ms/step | 21.8 samples/s
+Step 2800/50462 | Loss: 3.2108 | LR: 4.98e-04 | 2750.3ms/step | 21.8 samples/s
+Step 2810/50462 | Loss: 3.2195 | LR: 4.98e-04 | 2752.1ms/step | 21.8 samples/s
+Step 2820/50462 | Loss: 3.2034 | LR: 4.98e-04 | 2753.2ms/step | 21.8 samples/s
+Step 2830/50462 | Loss: 3.1613 | LR: 4.98e-04 | 2752.7ms/step | 21.8 samples/s
+Step 2840/50462 | Loss: 3.2031 | LR: 4.98e-04 | 2753.9ms/step | 21.8 samples/s
+Step 2850/50462 | Loss: 3.2032 | LR: 4.98e-04 | 2753.0ms/step | 21.8 samples/s
+Step 2860/50462 | Loss: 3.2109 | LR: 4.98e-04 | 2750.1ms/step | 21.8 samples/s
+Step 2870/50462 | Loss: 3.1961 | LR: 4.98e-04 | 2751.8ms/step | 21.8 samples/s
+Step 2880/50462 | Loss: 3.1908 | LR: 4.98e-04 | 2752.1ms/step | 21.8 samples/s
+Step 2890/50462 | Loss: 3.1746 | LR: 4.98e-04 | 2752.3ms/step | 21.8 samples/s
+Step 2900/50462 | Loss: 3.1946 | LR: 4.98e-04 | 2751.9ms/step | 21.8 samples/s
+Step 2910/50462 | Loss: 3.2074 | LR: 4.98e-04 | 2752.4ms/step | 21.8 samples/s
+Step 2920/50462 | Loss: 3.1985 | LR: 4.98e-04 | 2750.9ms/step | 21.8 samples/s
+Step 2930/50462 | Loss: 3.1761 | LR: 4.98e-04 | 2751.8ms/step | 21.8 samples/s
+Step 2940/50462 | Loss: 3.1707 | LR: 4.98e-04 | 2752.2ms/step | 21.8 samples/s
+Step 2950/50462 | Loss: 3.2039 | LR: 4.98e-04 | 2752.7ms/step | 21.8 samples/s
+Step 2960/50462 | Loss: 3.1663 | LR: 4.98e-04 | 2750.3ms/step | 21.8 samples/s
+Step 2970/50462 | Loss: 3.1965 | LR: 4.98e-04 | 2750.3ms/step | 21.8 samples/s
+Step 2980/50462 | Loss: 3.2059 | LR: 4.98e-04 | 2751.1ms/step | 21.8 samples/s
+Step 2990/50462 | Loss: 3.1836 | LR: 4.97e-04 | 2752.6ms/step | 21.8 samples/s
+Step 3000/50462 | Loss: 3.1867 | LR: 4.97e-04 | 2751.1ms/step | 21.8 samples/s
+Validation at step 3000: Loss = 3.1912, Perplexity = 24.32
+Saved checkpoint to checkpoints/run_20260613_222732/checkpoint-3000
+Step 3010/50462 | Loss: 3.1856 | LR: 4.97e-04 | 5336.6ms/step | 11.2 samples/s
+Step 3020/50462 | Loss: 3.1742 | LR: 4.97e-04 | 2747.0ms/step | 21.8 samples/s
+Step 3030/50462 | Loss: 3.1650 | LR: 4.97e-04 | 2749.1ms/step | 21.8 samples/s
+Traceback (most recent call last):
+  File "/mloscratch/homes/navasard/sparse/SparseAtt/train.py", line 1440, in <module>
+    main()
+  File "/mloscratch/homes/navasard/sparse/SparseAtt/train.py", line 1437, in main
+    train(config)
+  File "/mloscratch/homes/navasard/sparse/SparseAtt/train.py", line 1301, in train
+    outputs = model(
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
+    return self._call_impl(*args, **kwargs)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
+    return forward_call(*args, **kwargs)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1832, in forward
+    else self._run_ddp_forward(*inputs, **kwargs)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1657, in _run_ddp_forward
+    return self.module(*inputs, **kwargs)  # type: ignore[index]
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 473, in __call__
+    return super().__call__(*args, **kwargs)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
+    return self._call_impl(*args, **kwargs)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
+    return forward_call(*args, **kwargs)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1047, in compile_wrapper
+    result = fn(*args, **kwargs)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
+    return self._call_impl(*args, **kwargs)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
+    return forward_call(*args, **kwargs)
+  File "/mloscratch/homes/navasard/sparse/SparseAtt/sparseatt/model.py", line 285, in forward
+    hidden_states, next_cache, all_hidden_states, all_attentions = self.model(
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
+    return self._call_impl(*args, **kwargs)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
+    return forward_call(*args, **kwargs)
+  File "/mloscratch/homes/navasard/sparse/SparseAtt/sparseatt/model.py", line 139, in forward
+    if has_any_truly_sparse and (attention_mask == 0).any():
+  File "/mloscratch/homes/navasard/sparse/SparseAtt/sparseatt/model.py", line 139, in torch_dynamo_resume_in_forward_at_139
+    if has_any_truly_sparse and (attention_mask == 0).any():
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 473, in __call__
+    return super().__call__(*args, **kwargs)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
+    return self._call_impl(*args, **kwargs)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
+    return forward_call(*args, **kwargs)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1298, in _fn
+    return fn(*args, **kwargs)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/fx/graph_module.py", line 949, in call_wrapped
+    return self._wrapped_call(self, *args, **kwargs)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/fx/graph_module.py", line 447, in __call__
+    return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
+    return self._call_impl(*args, **kwargs)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
+    return forward_call(*args, **kwargs)
+  File "<eval_with_key>.720", line 656, in forward
+    submod_16 = self.compiled_submod_16(submod_15, l_self_modules_layers_modules_7_modules_mlp_modules_down_proj_parameters_weight_, getitem_15, l_self_modules_layers_modules_8_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_8_modules_self_attn_modules_q_proj_parameters_weight_, l_self_modules_layers_modules_8_modules_self_attn_modules_k_proj_parameters_weight_, l_self_modules_layers_modules_8_modules_self_attn_modules_v_proj_parameters_weight_, l_self_modules_layers_modules_8_modules_self_attn_modules_q_norm_parameters_weight_, l_self_modules_layers_modules_8_modules_self_attn_modules_k_norm_parameters_weight_, l_self_modules_layers_modules_8_modules_self_attn_modules_rotary_emb_buffers_inv_freq_, l_position_ids_, l_sparse_indices_8_kv_num_blocks, l_sparse_indices_8_kv_indices, l_sparse_indices_8_full_kv_num_blocks, l_sparse_indices_8_full_kv_indices, l_sparse_indices_8_q_num_blocks, l_sparse_indices_8_q_indices, l_sparse_indices_8_full_q_num_blocks, l_sparse_indices_8_full_q_indices, l_self_modules_layers_modules_8_modules_self_attn_modules_o_proj_parameters_weight_, l_self_modules_layers_modules_8_modules_post_attention_layernorm_parameters_weight_);  submod_15 = l_self_modules_layers_modules_7_modules_mlp_modules_down_proj_parameters_weight_ = getitem_15 = l_self_modules_layers_modules_8_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_8_modules_self_attn_modules_q_proj_parameters_weight_ = l_self_modules_layers_modules_8_modules_self_attn_modules_k_proj_parameters_weight_ = l_self_modules_layers_modules_8_modules_self_attn_modules_v_proj_parameters_weight_ = l_self_modules_layers_modules_8_modules_self_attn_modules_q_norm_parameters_weight_ = l_self_modules_layers_modules_8_modules_self_attn_modules_k_norm_parameters_weight_ = l_self_modules_layers_modules_8_modules_self_attn_modules_rotary_emb_buffers_inv_freq_ = l_sparse_indices_8_kv_num_blocks = l_sparse_indices_8_kv_indices = l_sparse_indices_8_full_kv_num_blocks = l_sparse_indices_8_full_kv_indices = l_sparse_indices_8_q_num_blocks = l_sparse_indices_8_q_indices = l_sparse_indices_8_full_q_num_blocks = l_sparse_indices_8_full_q_indices = l_self_modules_layers_modules_8_modules_self_attn_modules_o_proj_parameters_weight_ = l_self_modules_layers_modules_8_modules_post_attention_layernorm_parameters_weight_ = None
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
+    return self._call_impl(*args, **kwargs)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
+    return forward_call(*args, **kwargs)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/_dynamo/backends/distributed.py", line 215, in forward
+    x = self.submod(*args)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1298, in _fn
+    return fn(*args, **kwargs)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1273, in forward
+    return compiled_fn(full_args)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 777, in runtime_wrapper
+    all_outs = compiled_invoker.run(args, on_before_call=exit_prologue)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 506, in run
+    return call_func_at_runtime_with_args(
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 126, in call_func_at_runtime_with_args
+    out = normalize_as_list(f(args))
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 93, in g
+    return f(*args)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/autograd/function.py", line 596, in apply
+    return super().apply(*args, **kwargs)  # type: ignore[misc]
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 2844, in forward
+    fw_outs = call_func_at_runtime_with_args(
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 126, in call_func_at_runtime_with_args
+    out = normalize_as_list(f(args))
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 961, in wrapper
+    return compiled_fn(runtime_args)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 850, in wrapper
+    return compiled_fn(runtime_args)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 1055, in inner_fn
+    outs = compiled_fn(args)
+  File "/home/navasard/.local/lib/python3.10/site-packages/torch/_inductor/output_code.py", line 725, in __call__
+    return self.current_callable(inputs)
+  File "/home/navasard/.cache/torch_inductor/oq/coq6impgppmp6dnp26psegq3avd7bsgoow4agrrdejd2tgokhclc.py", line 1390, in call
+    buf19.copy_(primals_12, False)
+KeyboardInterrupt
+[rank0]: Traceback (most recent call last):
+[rank0]:   File "/mloscratch/homes/navasard/sparse/SparseAtt/train.py", line 1440, in <module>
+[rank0]:     main()
+[rank0]:   File "/mloscratch/homes/navasard/sparse/SparseAtt/train.py", line 1437, in main
+[rank0]:     train(config)
+[rank0]:   File "/mloscratch/homes/navasard/sparse/SparseAtt/train.py", line 1301, in train
+[rank0]:     outputs = model(
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
+[rank0]:     return self._call_impl(*args, **kwargs)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
+[rank0]:     return forward_call(*args, **kwargs)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1832, in forward
+[rank0]:     else self._run_ddp_forward(*inputs, **kwargs)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1657, in _run_ddp_forward
+[rank0]:     return self.module(*inputs, **kwargs)  # type: ignore[index]
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 473, in __call__
+[rank0]:     return super().__call__(*args, **kwargs)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
+[rank0]:     return self._call_impl(*args, **kwargs)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
+[rank0]:     return forward_call(*args, **kwargs)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1047, in compile_wrapper
+[rank0]:     result = fn(*args, **kwargs)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
+[rank0]:     return self._call_impl(*args, **kwargs)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
+[rank0]:     return forward_call(*args, **kwargs)
+[rank0]:   File "/mloscratch/homes/navasard/sparse/SparseAtt/sparseatt/model.py", line 285, in forward
+[rank0]:     hidden_states, next_cache, all_hidden_states, all_attentions = self.model(
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
+[rank0]:     return self._call_impl(*args, **kwargs)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
+[rank0]:     return forward_call(*args, **kwargs)
+[rank0]:   File "/mloscratch/homes/navasard/sparse/SparseAtt/sparseatt/model.py", line 139, in forward
+[rank0]:     if has_any_truly_sparse and (attention_mask == 0).any():
+[rank0]:   File "/mloscratch/homes/navasard/sparse/SparseAtt/sparseatt/model.py", line 139, in torch_dynamo_resume_in_forward_at_139
+[rank0]:     if has_any_truly_sparse and (attention_mask == 0).any():
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 473, in __call__
+[rank0]:     return super().__call__(*args, **kwargs)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
+[rank0]:     return self._call_impl(*args, **kwargs)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
+[rank0]:     return forward_call(*args, **kwargs)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1298, in _fn
+[rank0]:     return fn(*args, **kwargs)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/fx/graph_module.py", line 949, in call_wrapped
+[rank0]:     return self._wrapped_call(self, *args, **kwargs)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/fx/graph_module.py", line 447, in __call__
+[rank0]:     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
+[rank0]:     return self._call_impl(*args, **kwargs)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
+[rank0]:     return forward_call(*args, **kwargs)
+[rank0]:   File "<eval_with_key>.720", line 656, in forward
+[rank0]:     submod_16 = self.compiled_submod_16(submod_15, l_self_modules_layers_modules_7_modules_mlp_modules_down_proj_parameters_weight_, getitem_15, l_self_modules_layers_modules_8_modules_input_layernorm_parameters_weight_, l_self_modules_layers_modules_8_modules_self_attn_modules_q_proj_parameters_weight_, l_self_modules_layers_modules_8_modules_self_attn_modules_k_proj_parameters_weight_, l_self_modules_layers_modules_8_modules_self_attn_modules_v_proj_parameters_weight_, l_self_modules_layers_modules_8_modules_self_attn_modules_q_norm_parameters_weight_, l_self_modules_layers_modules_8_modules_self_attn_modules_k_norm_parameters_weight_, l_self_modules_layers_modules_8_modules_self_attn_modules_rotary_emb_buffers_inv_freq_, l_position_ids_, l_sparse_indices_8_kv_num_blocks, l_sparse_indices_8_kv_indices, l_sparse_indices_8_full_kv_num_blocks, l_sparse_indices_8_full_kv_indices, l_sparse_indices_8_q_num_blocks, l_sparse_indices_8_q_indices, l_sparse_indices_8_full_q_num_blocks, l_sparse_indices_8_full_q_indices, l_self_modules_layers_modules_8_modules_self_attn_modules_o_proj_parameters_weight_, l_self_modules_layers_modules_8_modules_post_attention_layernorm_parameters_weight_);  submod_15 = l_self_modules_layers_modules_7_modules_mlp_modules_down_proj_parameters_weight_ = getitem_15 = l_self_modules_layers_modules_8_modules_input_layernorm_parameters_weight_ = l_self_modules_layers_modules_8_modules_self_attn_modules_q_proj_parameters_weight_ = l_self_modules_layers_modules_8_modules_self_attn_modules_k_proj_parameters_weight_ = l_self_modules_layers_modules_8_modules_self_attn_modules_v_proj_parameters_weight_ = l_self_modules_layers_modules_8_modules_self_attn_modules_q_norm_parameters_weight_ = l_self_modules_layers_modules_8_modules_self_attn_modules_k_norm_parameters_weight_ = l_self_modules_layers_modules_8_modules_self_attn_modules_rotary_emb_buffers_inv_freq_ = l_sparse_indices_8_kv_num_blocks = l_sparse_indices_8_kv_indices = l_sparse_indices_8_full_kv_num_blocks = l_sparse_indices_8_full_kv_indices = l_sparse_indices_8_q_num_blocks = l_sparse_indices_8_q_indices = l_sparse_indices_8_full_q_num_blocks = l_sparse_indices_8_full_q_indices = l_self_modules_layers_modules_8_modules_self_attn_modules_o_proj_parameters_weight_ = l_self_modules_layers_modules_8_modules_post_attention_layernorm_parameters_weight_ = None
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1778, in _wrapped_call_impl
+[rank0]:     return self._call_impl(*args, **kwargs)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1789, in _call_impl
+[rank0]:     return forward_call(*args, **kwargs)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/_dynamo/backends/distributed.py", line 215, in forward
+[rank0]:     x = self.submod(*args)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1298, in _fn
+[rank0]:     return fn(*args, **kwargs)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1273, in forward
+[rank0]:     return compiled_fn(full_args)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 777, in runtime_wrapper
+[rank0]:     all_outs = compiled_invoker.run(args, on_before_call=exit_prologue)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 506, in run
+[rank0]:     return call_func_at_runtime_with_args(
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 126, in call_func_at_runtime_with_args
+[rank0]:     out = normalize_as_list(f(args))
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 93, in g
+[rank0]:     return f(*args)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/autograd/function.py", line 596, in apply
+[rank0]:     return super().apply(*args, **kwargs)  # type: ignore[misc]
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 2844, in forward
+[rank0]:     fw_outs = call_func_at_runtime_with_args(
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 126, in call_func_at_runtime_with_args
+[rank0]:     out = normalize_as_list(f(args))
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 961, in wrapper
+[rank0]:     return compiled_fn(runtime_args)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 850, in wrapper
+[rank0]:     return compiled_fn(runtime_args)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 1055, in inner_fn
+[rank0]:     outs = compiled_fn(args)
+[rank0]:   File "/home/navasard/.local/lib/python3.10/site-packages/torch/_inductor/output_code.py", line 725, in __call__
+[rank0]:     return self.current_callable(inputs)
+[rank0]:   File "/home/navasard/.cache/torch_inductor/oq/coq6impgppmp6dnp26psegq3avd7bsgoow4agrrdejd2tgokhclc.py", line 1390, in call
+[rank0]:     buf19.copy_(primals_12, False)
+[rank0]: KeyboardInterrupt

epfl_final/wandb/run-20260613_222959-rp96qdel/files/requirements.txt

@@ -0,0 +1,122 @@
+word2number==1.1
+sqlitedict==2.1.0
+pytz==2026.2
+nvidia-cusparselt-cu13==0.8.1
+mpmath==1.3.0
+cuda-toolkit==13.0.2
+zstandard==0.25.0
+xxhash==3.7.0
+urllib3==2.7.0
+tzdata==2026.2
+typing_extensions==4.15.0
+triton==3.7.0
+tqdm==4.68.2
+torchinfo==1.8.0
+threadpoolctl==3.6.0
+tcolorpy==0.1.7
+tabulate==0.10.0
+sympy==1.14.0
+smmap==5.0.3
+six==1.17.0
+shellingham==1.5.4
+safetensors==0.8.0
+regex==2026.5.9
+PyYAML==6.0.3
+Pygments==2.20.0
+pyarrow==24.0.0
+psutil==7.2.2
+protobuf==7.35.1
+propcache==0.5.2
+portalocker==3.2.0
+platformdirs==4.10.0
+pathvalidate==3.3.1
+packaging==26.2
+nvidia-nvtx==13.0.85
+nvidia-nvshmem-cu13==3.4.5
+nvidia-nvjitlink==13.0.88
+nvidia-nccl-cu13==2.29.7
+nvidia-curand==10.4.0.35
+nvidia-cufile==1.15.1.6
+nvidia-cuda-runtime==13.0.96
+nvidia-cuda-nvrtc==13.0.88
+nvidia-cuda-cupti==13.0.85
+numpy==2.2.6
+networkx==3.4.2
+more-itertools==11.1.0
+mdurl==0.1.2
+MarkupSafe==3.0.3
+lxml==6.1.1
+joblib==1.5.3
+idna==3.18
+hf-xet==1.5.1
+h11==0.16.0
+fsspec==2026.4.0
+frozenlist==1.8.0
+filelock==3.29.4
+dill==0.4.1
+cuda-pathfinder==1.5.5
+colorama==0.4.6
+click==8.4.1
+charset-normalizer==3.4.7
+chardet==6.0.0.post1
+certifi==2026.5.20
+attrs==26.1.0
+async-timeout==5.0.1
+annotated-types==0.7.0
+annotated-doc==0.0.4
+aiohappyeyeballs==2.6.2
+absl-py==2.4.0
+typing-inspection==0.4.2
+sentry-sdk==2.62.0
+scipy==1.15.3
+sacrebleu==2.6.0
+requests==2.34.2
+python-dateutil==2.9.0.post0
+pydantic_core==2.46.4
+nvidia-cusparse==12.6.3.3
+nvidia-cufft==12.0.0.61
+nvidia-cublas==13.1.1.3
+nltk==3.9.4
+multiprocess==0.70.19
+multidict==6.7.1
+mbstrdecoder==1.1.5
+markdown-it-py==4.2.0
+Jinja2==3.1.6
+httpcore==1.0.9
+gitdb==4.0.12
+exceptiongroup==1.3.1
+cuda-bindings==13.3.1
+aiosignal==1.4.0
+yarl==1.24.2
+typepy==1.3.5
+scikit-learn==1.7.2
+rouge-score==0.1.2
+rich==15.0.0
+pydantic==2.13.4
+pandas==2.3.3
+nvidia-cusolver==12.0.4.66
+nvidia-cudnn-cu13==9.20.0.48
+GitPython==3.1.50
+anyio==4.13.0
+wandb==0.27.2
+typer==0.25.1
+httpx==0.28.1
+aiohttp==3.14.1
+torch==2.12.0
+huggingface_hub==1.19.0
+DataProperty==1.1.1
+tokenizers==0.22.2
+tabledata==1.3.5
+datasets==5.0.0
+accelerate==1.14.0
+transformers==5.12.0
+pytablewriter==1.2.1
+evaluate==0.4.6
+lm_eval==0.4.12
+PyGObject==3.42.1
+dbus-python==1.2.18
+distro==1.7.0
+pip==22.0.2
+setuptools==59.6.0
+ssh-import-id==5.11
+wheel==0.37.1

epfl_final/wandb/run-20260613_222959-rp96qdel/files/wandb-metadata.json

@@ -0,0 +1,54 @@
+{
+  "os": "Linux-6.8.0-124-generic-x86_64-with-glibc2.35",
+  "python": "CPython 3.10.12",
+  "startedAt": "2026-06-13T22:29:59.446565Z",
+  "args": [
+    "--config",
+    "SparseAtt/configs/train_config.json"
+  ],
+  "program": "/mloscratch/homes/navasard/sparse/SparseAtt/train.py",
+  "codePath": "SparseAtt/train.py",
+  "codePathLocal": "SparseAtt/train.py",
+  "email": "ruben.navasardyan@mail.utoronto.ca",
+  "root": "/mloscratch/homes/navasard/sparse",
+  "host": "h100-train-3-gpu-0-9",
+  "executable": "/usr/bin/python3",
+  "cpu_count": 96,
+  "cpu_count_logical": 192,
+  "gpu": "NVIDIA H100 80GB HBM3",
+  "gpu_count": 3,
+  "disk": {
+    "/": {
+      "total": "9374931091456",
+      "used": "6429901119488"
+    }
+  },
+  "memory": {
+    "total": "1623005732864"
+  },
+  "gpu_nvidia": [
+    {
+      "name": "NVIDIA H100 80GB HBM3",
+      "memoryTotal": "85520809984",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-b97c2df9-b38b-c4e9-4d9d-780e91b62380"
+    },
+    {
+      "name": "NVIDIA H100 80GB HBM3",
+      "memoryTotal": "85520809984",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-337db0d9-bb90-c768-bb02-7b5dfde9a5d1"
+    },
+    {
+      "name": "NVIDIA H100 80GB HBM3",
+      "memoryTotal": "85520809984",
+      "cudaCores": 16896,
+      "architecture": "Hopper",
+      "uuid": "GPU-129f6620-c2e8-b738-a48f-5b9473847237"
+    }
+  ],
+  "cudaVersion": "13.0",
+  "writerId": "2d0i9uu80tocjigcv6jnloatmmzovbz6"
+}

epfl_final/wandb/run-20260613_222959-rp96qdel/files/wandb-summary.json

@@ -0,0 +1 @@
+{"speed/steps_per_sec":0.3637516071577971,"train/learning_rate":0.0004974093694230931,"eval/uses_sparse_attention":1,"speed/ms_per_step":2749.128746986389,"_wandb":{"runtime":8762},"eval/perplexity":24.317472129382285,"eval/dense_interval_end":-1,"speed/samples_per_sec":21.825096429467827,"eval/uses_dense_attention":0,"train/global_step":3030,"eval/dense_interval_active":0,"_timestamp":1.7813984633541067e+09,"_step":3030,"train/loss":3.1649989700317382,"eval/loss":3.1911951096471913,"_runtime":8762.641483934,"train/epoch":0.060045182513574574,"eval/dense_interval_start":-1}

epfl_final/wandb/run-20260613_222959-rp96qdel/logs/debug.log

@@ -0,0 +1,42 @@
+2026-06-13 22:29:59,465 INFO    MainThread:3072 [wandb_setup.py:_flush():81] Current SDK version is 0.27.2
+2026-06-13 22:29:59,465 INFO    MainThread:3072 [wandb_setup.py:_flush():81] Configure stats pid to 3072
+2026-06-13 22:29:59,465 INFO    MainThread:3072 [wandb_setup.py:_flush():81] Loading settings from environment variables
+2026-06-13 22:29:59,466 INFO    MainThread:3072 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /mloscratch/homes/navasard/sparse/wandb/run-20260613_222959-rp96qdel/logs/debug.log
+2026-06-13 22:29:59,466 INFO    MainThread:3072 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /mloscratch/homes/navasard/sparse/wandb/run-20260613_222959-rp96qdel/logs/debug-internal.log
+2026-06-13 22:29:59,467 INFO    MainThread:3072 [wandb_init.py:init():766] calling init triggers
+2026-06-13 22:29:59,468 INFO    MainThread:3072 [wandb_init.py:init():771] wandb.init called with sweep_config: {}
+config: {'model_path': '/mloscratch/homes/navasard/qwen3_06b_base', 'model_dtype': 'bfloat16', 'init_from_scratch': True, 'train_path': '/mloscratch/homes/navasard/data/fineweb_large/tokenized', 'val_path': None, 'max_seq_length': 8192, 'num_workers': 8, 'output_dir': './checkpoints/run_20260613_222732', 'resume_from_checkpoint': None, 'force_new_run': True, 'num_epochs': 1, 'max_steps': None, 'per_device_batch_size': 2, 'gradient_accumulation_steps': 10, 'learning_rate': 0.0005, 'weight_decay': 0.01, 'warmup_steps': 750, 'lr_scheduler': 'cosine', 'max_grad_norm': 1.0, 'save_steps': 1000, 'eval_steps': 250, 'eval_batches': 50, 'logging_steps': 10, 'seed': 42, 'verbose': True, 'checkpoint_activations': False, 'strategy': 'ddp', 'mixed_precision': 'bf16', 'wandb_enabled': True, 'wandb_project': 'stochastic_sparsity', 'wandb_name': 'sparse 85%', 'wandb_tags': ['qwen3', '1.2b', 'sparse'], 'hidden_size': 1536, 'intermediate_size': 6144, 'num_attention_heads': 12, 'num_key_value_heads': 4, 'num_hidden_layers': None, 'head_dim': None, 'attention_type': 'sparse_stochastic', 'sparse_block_size': 128, 'sparse_num_random_blocks': 9, 'sparse_mask_pool_size': 10000, 'sparse_val_dense_compare': False, 'sparse_layer_indices': None, 'sparse_mask_cache_dir': 'mask_cache_10', 'dense_mode': False, 'dense_step_intervals': None, '_wandb': {}}
+2026-06-13 22:29:59,469 INFO    MainThread:3072 [wandb_init.py:init():814] starting backend
+2026-06-13 22:29:59,469 INFO    MainThread:3072 [wandb_init.py:init():829] sending inform_init request
+2026-06-13 22:29:59,802 INFO    MainThread:3072 [wandb_init.py:init():834] backend started and connected
+2026-06-13 22:29:59,816 INFO    MainThread:3072 [wandb_init.py:init():904] updated telemetry
+2026-06-13 22:29:59,817 INFO    MainThread:3072 [wandb_init.py:init():927] communicating run to backend with 90.0 second timeout
+2026-06-13 22:30:00,131 INFO    MainThread:3072 [wandb_init.py:init():972] starting run threads in backend
+2026-06-13 22:30:00,237 INFO    MainThread:3072 [wandb_run.py:_console_start():2523] atexit reg
+2026-06-13 22:30:00,237 INFO    MainThread:3072 [wandb_run.py:_redirect():2373] redirect: wrap_raw
+2026-06-13 22:30:00,237 INFO    MainThread:3072 [wandb_run.py:_redirect():2442] Wrapping output streams.
+2026-06-13 22:30:00,237 INFO    MainThread:3072 [wandb_run.py:_redirect():2465] Redirects installed.
+2026-06-13 22:30:00,240 INFO    MainThread:3072 [wandb_init.py:init():1010] run started, returning control to user process
+2026-06-14 00:56:02,774 INFO    wandb-AsyncioManager-main:3072 [service_client.py:_forward_responses():122] [no run ID] Reached EOF.
+2026-06-14 00:56:02,775 INFO    wandb-AsyncioManager-main:3072 [mailbox.py:close():154] [no run ID] Closing mailbox, abandoning 2 handles.
+2026-06-14 00:56:02,776 ERROR   wandb-AsyncioManager-main:3072 [asyncio_manager.py:fn_wrap_exceptions():184] Uncaught exception in run_soon callback.
+Traceback (most recent call last):
+  File "/home/navasard/.local/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_manager.py", line 182, in fn_wrap_exceptions
+    await fn()
+  File "/home/navasard/.local/lib/python3.10/site-packages/wandb/sdk/lib/run_messages.py", line 90, in loop
+    await asyncio_compat.race(
+  File "/home/navasard/.local/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_compat.py", line 278, in race
+    async with open_task_group(race=True) as tg:
+  File "/usr/lib/python3.10/contextlib.py", line 206, in __aexit__
+    await anext(self.gen)
+  File "/home/navasard/.local/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_compat.py", line 240, in open_task_group
+    await task_group._wait_all(race=race, timeout=exit_timeout)
+  File "/home/navasard/.local/lib/python3.10/site-packages/wandb/sdk/lib/asyncio_compat.py", line 180, in _wait_all
+    raise exc
+  File "/home/navasard/.local/lib/python3.10/site-packages/wandb/sdk/lib/run_messages.py", line 113, in _print_all
+    result = await handle.wait_async(timeout=None)
+  File "/home/navasard/.local/lib/python3.10/site-packages/wandb/sdk/mailbox/mailbox_handle.py", line 127, in wait_async
+    response = await self._handle.wait_async(timeout=timeout)
+  File "/home/navasard/.local/lib/python3.10/site-packages/wandb/sdk/mailbox/response_handle.py", line 123, in wait_async
+    raise HandleAbandonedError()
+wandb.sdk.mailbox.mailbox_handle.HandleAbandonedError

epfl_final/wandb/run-20260613_222959-rp96qdel/run-rp96qdel.wandb

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:82ebd8daefdfa40b01abe2ca73fab057a9421080b8dac19ddce10c03cce4973a
+size 2535726