Hugging Face's logo

rubentium
/
sparse-checkpoints

Model card Files
xet
 Community
rubentium commited on
Commit
32e2d99
·
verified ·
1 Parent(s): cad088e

EPFL ALiBi and training provenance 1051-1100 of 1184

Browse files
Files changed (49) hide show
  1. epfl_final/train_logs/dense72_eval_queue_20260909.log +114 -0
  2. epfl_final/train_logs/dropout_p28_resume/seed42.20260819T170628Z.log +180 -0
  3. epfl_final/train_logs/dropout_p28_resume/seed42.20260819T171033Z.log +0 -0
  4. epfl_final/train_logs/dropout_p28_resume/seed43.20260819T170628Z.log +170 -0
  5. epfl_final/train_logs/dropout_p28_resume/seed43.20260819T171033Z.log +0 -0
  6. epfl_final/train_logs/dropout_p28_resume/seed44.20260819T170628Z.log +170 -0
  7. epfl_final/train_logs/dropout_p28_resume/seed44.20260819T171033Z.log +0 -0
  8. epfl_final/train_logs/dropout_p28_resume/seed45.20260819T170628Z.log +186 -0
  9. epfl_final/train_logs/dropout_p28_resume/seed45.20260819T171033Z.log +0 -0
  10. epfl_final/train_logs/dropout_p28_resume/seed46.20260819T170628Z.log +170 -0
  11. epfl_final/train_logs/dropout_p28_resume/seed46.20260819T171033Z.log +0 -0
  12. epfl_final/train_logs/hf_upload_final_checkpoints.log +0 -0
  13. epfl_final/train_logs/hf_upload_paper_compendium.log +4 -0
  14. epfl_final/train_logs/hf_upload_september_results.log +0 -0
  15. epfl_final/train_logs/mixed8k3b_mask_build/gpu0.log +4 -0
  16. epfl_final/train_logs/mixed8k3b_mask_build/gpu1.log +4 -0
  17. epfl_final/train_logs/mixed8k3b_mask_build/gpu2.log +4 -0
  18. epfl_final/train_logs/mixed8k3b_mask_build/gpu3.log +4 -0
  19. epfl_final/train_logs/p2_1p2b_dense_seed72_6gpu_resume48000.log +458 -0
  20. epfl_final/train_logs/p2_1p2b_dense_seed72_6gpu_resume48000.pid +1 -0
  21. epfl_final/train_logs/p2_1p2b_mixed_seed70_repaired_from_scratch_6gpu.log +0 -0
  22. epfl_final/train_logs/p2_1p2b_mixed_seed70_repaired_from_scratch_6gpu.pid +1 -0
  23. epfl_final/train_logs/p2_1p2b_mixed_seed70_repaired_resume12750_6gpu.log +337 -0
  24. epfl_final/train_logs/p2_1p2b_mixed_seed70_repaired_resume12750_6gpu.pid +1 -0
  25. epfl_final/train_logs/p2_1p2b_mixed_seed70_repaired_resume14500_6gpu.log +658 -0
  26. epfl_final/train_logs/p2_1p2b_mixed_seed70_repaired_resume14500_6gpu.pid +1 -0
  27. epfl_final/train_logs/p2_1p2b_mixed_seed70_repaired_resume18500_6gpu.log +0 -0
  28. epfl_final/train_logs/p2_1p2b_mixed_seed70_repaired_resume33750_6gpu.log +396 -0
  29. epfl_final/train_logs/p2_1p2b_mixed_seed70_repaired_resume35500_6gpu.log +208 -0
  30. epfl_final/train_logs/p2_1p2b_mixed_seed70_repaired_resume35500_pod46.log +0 -0
  31. epfl_final/train_logs/p2_1p2b_mixed_seed70_repaired_resume49250_6gpu.log +310 -0
  32. epfl_final/train_logs/p2_1p2b_mixed_seed70_repaired_resume50000_6gpu.log +256 -0
  33. epfl_final/train_logs/queue_seed70_mixed_matched_evals.log +0 -0
  34. epfl_final/train_logs/queue_seed70_mixed_repaired_after_seed72.sh +38 -0
  35. epfl_final/train_logs/ropectl_a1p0_h100_resume/seed42.20260820T011048Z.log +903 -0
  36. epfl_final/train_logs/ropectl_a1p0_h100_resume/seed43.20260820T011048Z.log +900 -0
  37. epfl_final/train_logs/ropectl_a1p0_h100_resume/seed44.20260820T011048Z.log +918 -0
  38. epfl_final/train_logs/ropectl_a1p0_h100_resume/seed45.20260820T011048Z.log +915 -0
  39. epfl_final/train_logs/ropectl_a1p0_h100_resume/seed46.20260820T011048Z.log +905 -0
  40. epfl_final/train_logs/ropectl_calib/a0p5.log +213 -0
  41. epfl_final/train_logs/ropectl_calib/a1p0.log +213 -0
  42. epfl_final/train_logs/ropectl_calib/a1p5.log +213 -0
  43. epfl_final/train_logs/ropectl_calib/a2p0.log +213 -0
  44. epfl_final/train_logs/ropectl_calib/a3p0.log +213 -0
  45. epfl_final/train_logs/ropectl_full_a1p0/seed42.log +271 -0
  46. epfl_final/train_logs/ropectl_full_a1p0/seed43.log +287 -0
  47. epfl_final/train_logs/ropectl_full_a1p0/seed44.log +271 -0
  48. epfl_final/train_logs/ropectl_full_a1p0/seed45.log +271 -0
  49. epfl_final/train_logs/ropectl_full_a1p0/seed46.log +271 -0
epfl_final/train_logs/dense72_eval_queue_20260909.log ADDED
@@ -0,0 +1,114 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ TOTAL 56
2
+ START 0 ppl_8192_fineweb
3
+ START 1 ppl_8192_pg19
4
+ START 2 ppl_8192_arxiv
5
+ START 3 ppl_16384_fineweb
6
+ DONE ppl_8192_fineweb 0 True
7
+ DONE ppl_8192_pg19 0 True
8
+ DONE ppl_8192_arxiv 0 True
9
+ DONE ppl_16384_fineweb 0 True
10
+ START 0 ppl_16384_pg19
11
+ START 1 ppl_16384_arxiv
12
+ START 2 ppl_24576_fineweb
13
+ START 3 ppl_24576_pg19
14
+ DONE ppl_16384_pg19 0 True
15
+ DONE ppl_24576_pg19 0 True
16
+ START 0 ppl_24576_arxiv
17
+ START 3 ppl_30720_fineweb
18
+ DONE ppl_16384_arxiv 0 True
19
+ DONE ppl_24576_fineweb 0 True
20
+ START 1 ppl_30720_pg19
21
+ START 2 ppl_30720_arxiv
22
+ DONE ppl_24576_arxiv 0 True
23
+ START 0 cal_8192
24
+ DONE ppl_30720_fineweb 0 True
25
+ DONE ppl_30720_arxiv 0 True
26
+ START 2 cal_16384
27
+ START 3 cal_24576
28
+ DONE ppl_30720_pg19 0 True
29
+ START 1 cal_30720
30
+ DONE cal_8192 0 True
31
+ START 0 retr_8192_passkey_tau1
32
+ DONE cal_16384 0 True
33
+ START 2 retr_8192_niah_multikey_tau1
34
+ DONE cal_30720 0 True
35
+ START 1 retr_8192_niah_multivalue_tau1
36
+ DONE cal_24576 0 True
37
+ START 3 retr_8192_vt_tau1
38
+ DONE retr_8192_passkey_tau1 0 True
39
+ START 0 retr_8192_passkey_selected
40
+ DONE retr_8192_niah_multikey_tau1 0 True
41
+ START 2 retr_8192_niah_multikey_selected
42
+ DONE retr_8192_niah_multivalue_tau1 0 True
43
+ START 1 retr_8192_niah_multivalue_selected
44
+ DONE retr_8192_vt_tau1 0 True
45
+ START 3 retr_8192_vt_selected
46
+ DONE retr_8192_passkey_selected 0 True
47
+ START 0 retr_16384_passkey_tau1
48
+ DONE retr_8192_niah_multikey_selected 0 True
49
+ START 2 retr_16384_niah_multikey_tau1
50
+ DONE retr_8192_niah_multivalue_selected 0 True
51
+ START 1 retr_16384_niah_multivalue_tau1
52
+ DONE retr_8192_vt_selected 0 True
53
+ START 3 retr_16384_vt_tau1
54
+ DONE retr_16384_passkey_tau1 0 True
55
+ START 0 retr_24576_passkey_tau1
56
+ DONE retr_16384_niah_multikey_tau1 0 True
57
+ START 2 retr_24576_niah_multikey_tau1
58
+ DONE retr_16384_niah_multivalue_tau1 0 True
59
+ START 1 retr_24576_niah_multivalue_tau1
60
+ DONE retr_16384_vt_tau1 0 True
61
+ START 3 retr_24576_vt_tau1
62
+ DONE retr_24576_passkey_tau1 0 True
63
+ DONE retr_24576_niah_multikey_tau1 0 True
64
+ START 0 retr_30720_passkey_tau1
65
+ START 2 retr_30720_niah_multikey_tau1
66
+ DONE retr_24576_niah_multivalue_tau1 0 True
67
+ START 1 retr_30720_niah_multivalue_tau1
68
+ DONE retr_24576_vt_tau1 0 True
69
+ START 3 retr_30720_vt_tau1
70
+ DONE retr_30720_niah_multikey_tau1 0 True
71
+ START 2 retr_30720_passkey_tau08
72
+ DONE retr_30720_passkey_tau1 0 True
73
+ START 0 retr_30720_niah_multikey_tau08
74
+ DONE retr_30720_niah_multivalue_tau1 0 True
75
+ START 1 retr_30720_niah_multivalue_tau08
76
+ DONE retr_30720_vt_tau1 0 True
77
+ START 3 retr_30720_vt_tau08
78
+ DONE retr_30720_passkey_tau08 0 True
79
+ START 2 retr_30720_passkey_tau085
80
+ DONE retr_30720_niah_multikey_tau08 0 True
81
+ START 0 retr_30720_niah_multikey_tau085
82
+ DONE retr_30720_niah_multivalue_tau08 0 True
83
+ START 1 retr_30720_niah_multivalue_tau085
84
+ DONE retr_30720_vt_tau08 0 True
85
+ START 3 retr_30720_vt_tau085
86
+ DONE retr_30720_passkey_tau085 0 True
87
+ START 2 retr_30720_passkey_selected
88
+ DONE retr_30720_niah_multikey_tau085 0 True
89
+ START 0 retr_30720_niah_multikey_selected
90
+ DONE retr_30720_niah_multivalue_tau085 0 True
91
+ START 1 retr_30720_niah_multivalue_selected
92
+ DONE retr_30720_vt_tau085 0 True
93
+ START 3 retr_30720_vt_selected
94
+ DONE retr_30720_passkey_selected 0 True
95
+ DONE retr_30720_niah_multikey_selected 0 True
96
+ START 0 gen_30720_passkey_tau1
97
+ START 2 gen_30720_passkey_selected
98
+ DONE retr_30720_niah_multivalue_selected 0 True
99
+ START 1 gen_30720_niah_multikey_tau1
100
+ DONE retr_30720_vt_selected 0 True
101
+ START 3 gen_30720_niah_multikey_selected
102
+ DONE gen_30720_passkey_tau1 0 True
103
+ DONE gen_30720_passkey_selected 0 True
104
+ DONE gen_30720_niah_multikey_tau1 0 True
105
+ START 0 diag_8192
106
+ START 1 diag_16384
107
+ START 2 diag_24576
108
+ DONE gen_30720_niah_multikey_selected 0 True
109
+ START 3 diag_30720
110
+ DONE diag_8192 0 True
111
+ DONE diag_16384 0 True
112
+ DONE diag_24576 0 True
113
+ DONE diag_30720 0 True
114
+ COMPLETE 56 / 56 INCOMPLETE []
epfl_final/train_logs/dropout_p28_resume/seed42.20260819T170628Z.log ADDED
@@ -0,0 +1,180 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Auto-resuming from latest checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed42/run_20260819_050018/checkpoint-27000
2
+ Resuming: Using existing output directory /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed42/run_20260819_050018
3
+ Running config:
4
+ {
5
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
6
+ "model_dtype": "bfloat16",
7
+ "init_from_scratch": true,
8
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
9
+ "val_path": null,
10
+ "max_seq_length": 8192,
11
+ "num_workers": 8,
12
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed42/run_20260819_050018",
13
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed42/run_20260819_050018/checkpoint-27000",
14
+ "force_new_run": false,
15
+ "num_epochs": 1,
16
+ "max_steps": 44000,
17
+ "per_device_batch_size": 2,
18
+ "gradient_accumulation_steps": 10,
19
+ "learning_rate": 0.0005,
20
+ "weight_decay": 0.01,
21
+ "warmup_steps": 750,
22
+ "lr_scheduler": "cosine",
23
+ "max_grad_norm": 1.0,
24
+ "save_steps": 1000,
25
+ "eval_steps": 1000,
26
+ "eval_batches": 50,
27
+ "logging_steps": 10,
28
+ "seed": 42,
29
+ "verbose": true,
30
+ "checkpoint_activations": false,
31
+ "strategy": "ddp",
32
+ "mixed_precision": "bf16",
33
+ "wandb_enabled": false,
34
+ "wandb_project": "stochastic_sparsity",
35
+ "wandb_name": "l8k365m_dense_dropout_p28_seed42",
36
+ "wandb_tags": [
37
+ "qwen3",
38
+ "365m",
39
+ "8k",
40
+ "scaleladder",
41
+ "dense"
42
+ ],
43
+ "hidden_size": 768,
44
+ "intermediate_size": 3072,
45
+ "num_attention_heads": 6,
46
+ "num_key_value_heads": 2,
47
+ "num_hidden_layers": 28,
48
+ "head_dim": null,
49
+ "attention_dropout": 0.28,
50
+ "rope_train_position_scale_set": null,
51
+ "attention_type": "sparse_stochastic",
52
+ "sparse_block_size": 128,
53
+ "sparse_num_random_blocks": 31,
54
+ "sparse_mask_pool_size": 1,
55
+ "sparse_val_dense_compare": false,
56
+ "sparse_layer_indices": null,
57
+ "sparse_mask_cache_dir": null,
58
+ "sparse_mask_seed": 0,
59
+ "sparse_mixed_nrb_set": null,
60
+ "dense_mode": true,
61
+ "dense_step_intervals": null
62
+ }
63
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
64
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed42/run_20260819_050018/checkpoint-27000
65
+ Checkpoint activations: False
66
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
67
+
68
+
69
+ ================================================================================
70
+ MODEL ARCHITECTURE
71
+ ================================================================================
72
+ Model config:
73
+ vocab_size: 151936
74
+ hidden_size: 768
75
+ num_hidden_layers: 28
76
+ num_attention_heads: 6
77
+ num_key_value_heads: 2
78
+ head_dim: 128
79
+ intermediate_size: 3072
80
+ max_position_embeddings: 32768
81
+ rope_theta: 1000000
82
+ rope_train_position_scale_set: None
83
+ attention_type: sparse_stochastic
84
+ sparse_block_size: 128
85
+ sparse_num_random_blocks: 31
86
+ sparse_mask_pool_size: 1
87
+ sparse_max_seq_length: 8192
88
+
89
+ Parameter count:
90
+ Total parameters: 358,958,848
91
+ Trainable parameters: 358,958,848
92
+ Non-trainable parameters: 0
93
+ Model size (BF16): 0.72 GB
94
+
95
+ --------------------------------------------------------------------------------
96
+ LAYER-BY-LAYER SUMMARY
97
+ --------------------------------------------------------------------------------
98
+ ================================================================================================================================================================
99
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
100
+ ================================================================================================================================================================
101
+ Qwen3ForCausalLM [2, 8192] -- -- True
102
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
103
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
104
+ │ └─ModuleList: 2-2 -- -- -- True
105
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
106
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
107
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
108
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
109
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
110
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
111
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
132
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
133
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
134
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
135
+ ================================================================================================================================================================
136
+ Total params: 475,645,696
137
+ Trainable params: 475,645,696
138
+ Non-trainable params: 0
139
+ Total mult-adds (Units.MEGABYTES): 951.29
140
+ ================================================================================================================================================================
141
+ Input size (MB): 0.13
142
+ Forward/backward pass size (MB): 31197.23
143
+ Params size (MB): 951.29
144
+ Estimated Total Size (MB): 32148.66
145
+ ================================================================================================================================================================
146
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
147
+ ================================================================================
148
+
149
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
150
+ Using pre-tokenized dataset (no runtime tokenization)
151
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
152
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
153
+ Dataset split: 3027764 train, 159356 validation
154
+ Training examples (packed, train split): 3027764
155
+ Validation examples (packed, val split): 159356
156
+ Per-device batch size: 2
157
+ World size (GPUs): 1
158
+ Gradient accumulation steps: 10
159
+ Effective batch size (seqs/step): 20
160
+ Sequence length: 8192
161
+ Tokens per optimizer step: 163,840
162
+ Micro-batches per epoch (per rank): 1513882
163
+ Optimizer steps per epoch: 151388
164
+ Total optimizer steps: 44000 (capped by max_steps; 0.29 epochs)
165
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed42/run_20260819_050018/checkpoint-27000...
166
+ Overriding checkpoint LR with config LR: 0.0005
167
+ Resumed from checkpoint at step 27000 (epoch 0, micro_step 270000)
168
+ Resuming training from step 27000
169
+ Starting training...
170
+ Validation at step 27000: Loss = 2.9398, Perplexity = 18.91
171
+ Traceback (most recent call last):
172
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/resource_sharer.py", line 139, in _serve
173
+ msg = conn.recv()
174
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 251, in recv
175
+ buf = self._recv_bytes()
176
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 431, in _recv_bytes
177
+ buf = self._recv(4)
178
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 400, in _recv
179
+ raise EOFError
180
+ EOFError
epfl_final/train_logs/dropout_p28_resume/seed42.20260819T171033Z.log ADDED
The diff for this file is too large to render. See raw diff
 
epfl_final/train_logs/dropout_p28_resume/seed43.20260819T170628Z.log ADDED
@@ -0,0 +1,170 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Auto-resuming from latest checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed43/run_20260819_050018/checkpoint-27000
2
+ Resuming: Using existing output directory /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed43/run_20260819_050018
3
+ Running config:
4
+ {
5
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
6
+ "model_dtype": "bfloat16",
7
+ "init_from_scratch": true,
8
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
9
+ "val_path": null,
10
+ "max_seq_length": 8192,
11
+ "num_workers": 8,
12
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed43/run_20260819_050018",
13
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed43/run_20260819_050018/checkpoint-27000",
14
+ "force_new_run": false,
15
+ "num_epochs": 1,
16
+ "max_steps": 44000,
17
+ "per_device_batch_size": 2,
18
+ "gradient_accumulation_steps": 10,
19
+ "learning_rate": 0.0005,
20
+ "weight_decay": 0.01,
21
+ "warmup_steps": 750,
22
+ "lr_scheduler": "cosine",
23
+ "max_grad_norm": 1.0,
24
+ "save_steps": 1000,
25
+ "eval_steps": 1000,
26
+ "eval_batches": 50,
27
+ "logging_steps": 10,
28
+ "seed": 43,
29
+ "verbose": true,
30
+ "checkpoint_activations": false,
31
+ "strategy": "ddp",
32
+ "mixed_precision": "bf16",
33
+ "wandb_enabled": false,
34
+ "wandb_project": "stochastic_sparsity",
35
+ "wandb_name": "l8k365m_dense_dropout_p28_seed43",
36
+ "wandb_tags": [
37
+ "qwen3",
38
+ "365m",
39
+ "8k",
40
+ "scaleladder",
41
+ "dense"
42
+ ],
43
+ "hidden_size": 768,
44
+ "intermediate_size": 3072,
45
+ "num_attention_heads": 6,
46
+ "num_key_value_heads": 2,
47
+ "num_hidden_layers": 28,
48
+ "head_dim": null,
49
+ "attention_dropout": 0.28,
50
+ "rope_train_position_scale_set": null,
51
+ "attention_type": "sparse_stochastic",
52
+ "sparse_block_size": 128,
53
+ "sparse_num_random_blocks": 31,
54
+ "sparse_mask_pool_size": 1,
55
+ "sparse_val_dense_compare": false,
56
+ "sparse_layer_indices": null,
57
+ "sparse_mask_cache_dir": null,
58
+ "sparse_mask_seed": 0,
59
+ "sparse_mixed_nrb_set": null,
60
+ "dense_mode": true,
61
+ "dense_step_intervals": null
62
+ }
63
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
64
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed43/run_20260819_050018/checkpoint-27000
65
+ Checkpoint activations: False
66
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
67
+
68
+
69
+ ================================================================================
70
+ MODEL ARCHITECTURE
71
+ ================================================================================
72
+ Model config:
73
+ vocab_size: 151936
74
+ hidden_size: 768
75
+ num_hidden_layers: 28
76
+ num_attention_heads: 6
77
+ num_key_value_heads: 2
78
+ head_dim: 128
79
+ intermediate_size: 3072
80
+ max_position_embeddings: 32768
81
+ rope_theta: 1000000
82
+ rope_train_position_scale_set: None
83
+ attention_type: sparse_stochastic
84
+ sparse_block_size: 128
85
+ sparse_num_random_blocks: 31
86
+ sparse_mask_pool_size: 1
87
+ sparse_max_seq_length: 8192
88
+
89
+ Parameter count:
90
+ Total parameters: 358,958,848
91
+ Trainable parameters: 358,958,848
92
+ Non-trainable parameters: 0
93
+ Model size (BF16): 0.72 GB
94
+
95
+ --------------------------------------------------------------------------------
96
+ LAYER-BY-LAYER SUMMARY
97
+ --------------------------------------------------------------------------------
98
+ ================================================================================================================================================================
99
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
100
+ ================================================================================================================================================================
101
+ Qwen3ForCausalLM [2, 8192] -- -- True
102
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
103
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
104
+ │ └─ModuleList: 2-2 -- -- -- True
105
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
106
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
107
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
108
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
109
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
110
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
111
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
132
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
133
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
134
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
135
+ ================================================================================================================================================================
136
+ Total params: 475,645,696
137
+ Trainable params: 475,645,696
138
+ Non-trainable params: 0
139
+ Total mult-adds (Units.MEGABYTES): 951.29
140
+ ================================================================================================================================================================
141
+ Input size (MB): 0.13
142
+ Forward/backward pass size (MB): 31197.23
143
+ Params size (MB): 951.29
144
+ Estimated Total Size (MB): 32148.66
145
+ ================================================================================================================================================================
146
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
147
+ ================================================================================
148
+
149
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
150
+ Using pre-tokenized dataset (no runtime tokenization)
151
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
152
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
153
+ Dataset split: 3027764 train, 159356 validation
154
+ Training examples (packed, train split): 3027764
155
+ Validation examples (packed, val split): 159356
156
+ Per-device batch size: 2
157
+ World size (GPUs): 1
158
+ Gradient accumulation steps: 10
159
+ Effective batch size (seqs/step): 20
160
+ Sequence length: 8192
161
+ Tokens per optimizer step: 163,840
162
+ Micro-batches per epoch (per rank): 1513882
163
+ Optimizer steps per epoch: 151388
164
+ Total optimizer steps: 44000 (capped by max_steps; 0.29 epochs)
165
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed43/run_20260819_050018/checkpoint-27000...
166
+ Overriding checkpoint LR with config LR: 0.0005
167
+ Resumed from checkpoint at step 27000 (epoch 0, micro_step 270000)
168
+ Resuming training from step 27000
169
+ Starting training...
170
+ Validation at step 27000: Loss = 2.9169, Perplexity = 18.48
epfl_final/train_logs/dropout_p28_resume/seed43.20260819T171033Z.log ADDED
The diff for this file is too large to render. See raw diff
 
epfl_final/train_logs/dropout_p28_resume/seed44.20260819T170628Z.log ADDED
@@ -0,0 +1,170 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Auto-resuming from latest checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed44/run_20260819_050018/checkpoint-27000
2
+ Resuming: Using existing output directory /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed44/run_20260819_050018
3
+ Running config:
4
+ {
5
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
6
+ "model_dtype": "bfloat16",
7
+ "init_from_scratch": true,
8
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
9
+ "val_path": null,
10
+ "max_seq_length": 8192,
11
+ "num_workers": 8,
12
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed44/run_20260819_050018",
13
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed44/run_20260819_050018/checkpoint-27000",
14
+ "force_new_run": false,
15
+ "num_epochs": 1,
16
+ "max_steps": 44000,
17
+ "per_device_batch_size": 2,
18
+ "gradient_accumulation_steps": 10,
19
+ "learning_rate": 0.0005,
20
+ "weight_decay": 0.01,
21
+ "warmup_steps": 750,
22
+ "lr_scheduler": "cosine",
23
+ "max_grad_norm": 1.0,
24
+ "save_steps": 1000,
25
+ "eval_steps": 1000,
26
+ "eval_batches": 50,
27
+ "logging_steps": 10,
28
+ "seed": 44,
29
+ "verbose": true,
30
+ "checkpoint_activations": false,
31
+ "strategy": "ddp",
32
+ "mixed_precision": "bf16",
33
+ "wandb_enabled": false,
34
+ "wandb_project": "stochastic_sparsity",
35
+ "wandb_name": "l8k365m_dense_dropout_p28_seed44",
36
+ "wandb_tags": [
37
+ "qwen3",
38
+ "365m",
39
+ "8k",
40
+ "scaleladder",
41
+ "dense"
42
+ ],
43
+ "hidden_size": 768,
44
+ "intermediate_size": 3072,
45
+ "num_attention_heads": 6,
46
+ "num_key_value_heads": 2,
47
+ "num_hidden_layers": 28,
48
+ "head_dim": null,
49
+ "attention_dropout": 0.28,
50
+ "rope_train_position_scale_set": null,
51
+ "attention_type": "sparse_stochastic",
52
+ "sparse_block_size": 128,
53
+ "sparse_num_random_blocks": 31,
54
+ "sparse_mask_pool_size": 1,
55
+ "sparse_val_dense_compare": false,
56
+ "sparse_layer_indices": null,
57
+ "sparse_mask_cache_dir": null,
58
+ "sparse_mask_seed": 0,
59
+ "sparse_mixed_nrb_set": null,
60
+ "dense_mode": true,
61
+ "dense_step_intervals": null
62
+ }
63
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
64
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed44/run_20260819_050018/checkpoint-27000
65
+ Checkpoint activations: False
66
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
67
+
68
+
69
+ ================================================================================
70
+ MODEL ARCHITECTURE
71
+ ================================================================================
72
+ Model config:
73
+ vocab_size: 151936
74
+ hidden_size: 768
75
+ num_hidden_layers: 28
76
+ num_attention_heads: 6
77
+ num_key_value_heads: 2
78
+ head_dim: 128
79
+ intermediate_size: 3072
80
+ max_position_embeddings: 32768
81
+ rope_theta: 1000000
82
+ rope_train_position_scale_set: None
83
+ attention_type: sparse_stochastic
84
+ sparse_block_size: 128
85
+ sparse_num_random_blocks: 31
86
+ sparse_mask_pool_size: 1
87
+ sparse_max_seq_length: 8192
88
+
89
+ Parameter count:
90
+ Total parameters: 358,958,848
91
+ Trainable parameters: 358,958,848
92
+ Non-trainable parameters: 0
93
+ Model size (BF16): 0.72 GB
94
+
95
+ --------------------------------------------------------------------------------
96
+ LAYER-BY-LAYER SUMMARY
97
+ --------------------------------------------------------------------------------
98
+ ================================================================================================================================================================
99
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
100
+ ================================================================================================================================================================
101
+ Qwen3ForCausalLM [2, 8192] -- -- True
102
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
103
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
104
+ │ └─ModuleList: 2-2 -- -- -- True
105
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
106
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
107
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
108
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
109
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
110
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
111
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
132
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
133
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
134
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
135
+ ================================================================================================================================================================
136
+ Total params: 475,645,696
137
+ Trainable params: 475,645,696
138
+ Non-trainable params: 0
139
+ Total mult-adds (Units.MEGABYTES): 951.29
140
+ ================================================================================================================================================================
141
+ Input size (MB): 0.13
142
+ Forward/backward pass size (MB): 31197.23
143
+ Params size (MB): 951.29
144
+ Estimated Total Size (MB): 32148.66
145
+ ================================================================================================================================================================
146
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
147
+ ================================================================================
148
+
149
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
150
+ Using pre-tokenized dataset (no runtime tokenization)
151
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
152
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
153
+ Dataset split: 3027764 train, 159356 validation
154
+ Training examples (packed, train split): 3027764
155
+ Validation examples (packed, val split): 159356
156
+ Per-device batch size: 2
157
+ World size (GPUs): 1
158
+ Gradient accumulation steps: 10
159
+ Effective batch size (seqs/step): 20
160
+ Sequence length: 8192
161
+ Tokens per optimizer step: 163,840
162
+ Micro-batches per epoch (per rank): 1513882
163
+ Optimizer steps per epoch: 151388
164
+ Total optimizer steps: 44000 (capped by max_steps; 0.29 epochs)
165
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed44/run_20260819_050018/checkpoint-27000...
166
+ Overriding checkpoint LR with config LR: 0.0005
167
+ Resumed from checkpoint at step 27000 (epoch 0, micro_step 270000)
168
+ Resuming training from step 27000
169
+ Starting training...
170
+ Validation at step 27000: Loss = 2.9375, Perplexity = 18.87
epfl_final/train_logs/dropout_p28_resume/seed44.20260819T171033Z.log ADDED
The diff for this file is too large to render. See raw diff
 
epfl_final/train_logs/dropout_p28_resume/seed45.20260819T170628Z.log ADDED
@@ -0,0 +1,186 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Auto-resuming from latest checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed45/run_20260819_050018/checkpoint-26000
2
+ Resuming: Using existing output directory /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed45/run_20260819_050018
3
+ Running config:
4
+ {
5
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
6
+ "model_dtype": "bfloat16",
7
+ "init_from_scratch": true,
8
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
9
+ "val_path": null,
10
+ "max_seq_length": 8192,
11
+ "num_workers": 8,
12
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed45/run_20260819_050018",
13
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed45/run_20260819_050018/checkpoint-26000",
14
+ "force_new_run": false,
15
+ "num_epochs": 1,
16
+ "max_steps": 44000,
17
+ "per_device_batch_size": 2,
18
+ "gradient_accumulation_steps": 10,
19
+ "learning_rate": 0.0005,
20
+ "weight_decay": 0.01,
21
+ "warmup_steps": 750,
22
+ "lr_scheduler": "cosine",
23
+ "max_grad_norm": 1.0,
24
+ "save_steps": 1000,
25
+ "eval_steps": 1000,
26
+ "eval_batches": 50,
27
+ "logging_steps": 10,
28
+ "seed": 45,
29
+ "verbose": true,
30
+ "checkpoint_activations": false,
31
+ "strategy": "ddp",
32
+ "mixed_precision": "bf16",
33
+ "wandb_enabled": false,
34
+ "wandb_project": "stochastic_sparsity",
35
+ "wandb_name": "l8k365m_dense_dropout_p28_seed45",
36
+ "wandb_tags": [
37
+ "qwen3",
38
+ "365m",
39
+ "8k",
40
+ "scaleladder",
41
+ "dense"
42
+ ],
43
+ "hidden_size": 768,
44
+ "intermediate_size": 3072,
45
+ "num_attention_heads": 6,
46
+ "num_key_value_heads": 2,
47
+ "num_hidden_layers": 28,
48
+ "head_dim": null,
49
+ "attention_dropout": 0.28,
50
+ "rope_train_position_scale_set": null,
51
+ "attention_type": "sparse_stochastic",
52
+ "sparse_block_size": 128,
53
+ "sparse_num_random_blocks": 31,
54
+ "sparse_mask_pool_size": 1,
55
+ "sparse_val_dense_compare": false,
56
+ "sparse_layer_indices": null,
57
+ "sparse_mask_cache_dir": null,
58
+ "sparse_mask_seed": 0,
59
+ "sparse_mixed_nrb_set": null,
60
+ "dense_mode": true,
61
+ "dense_step_intervals": null
62
+ }
63
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
64
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed45/run_20260819_050018/checkpoint-26000
65
+ Checkpoint activations: False
66
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
67
+
68
+
69
+ ================================================================================
70
+ MODEL ARCHITECTURE
71
+ ================================================================================
72
+ Model config:
73
+ vocab_size: 151936
74
+ hidden_size: 768
75
+ num_hidden_layers: 28
76
+ num_attention_heads: 6
77
+ num_key_value_heads: 2
78
+ head_dim: 128
79
+ intermediate_size: 3072
80
+ max_position_embeddings: 32768
81
+ rope_theta: 1000000
82
+ rope_train_position_scale_set: None
83
+ attention_type: sparse_stochastic
84
+ sparse_block_size: 128
85
+ sparse_num_random_blocks: 31
86
+ sparse_mask_pool_size: 1
87
+ sparse_max_seq_length: 8192
88
+
89
+ Parameter count:
90
+ Total parameters: 358,958,848
91
+ Trainable parameters: 358,958,848
92
+ Non-trainable parameters: 0
93
+ Model size (BF16): 0.72 GB
94
+
95
+ --------------------------------------------------------------------------------
96
+ LAYER-BY-LAYER SUMMARY
97
+ --------------------------------------------------------------------------------
98
+ ================================================================================================================================================================
99
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
100
+ ================================================================================================================================================================
101
+ Qwen3ForCausalLM [2, 8192] -- -- True
102
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
103
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
104
+ │ └─ModuleList: 2-2 -- -- -- True
105
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
106
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
107
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
108
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
109
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
110
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
111
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
132
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
133
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
134
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
135
+ ================================================================================================================================================================
136
+ Total params: 475,645,696
137
+ Trainable params: 475,645,696
138
+ Non-trainable params: 0
139
+ Total mult-adds (Units.MEGABYTES): 951.29
140
+ ================================================================================================================================================================
141
+ Input size (MB): 0.13
142
+ Forward/backward pass size (MB): 31197.23
143
+ Params size (MB): 951.29
144
+ Estimated Total Size (MB): 32148.66
145
+ ================================================================================================================================================================
146
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
147
+ ================================================================================
148
+
149
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
150
+ Using pre-tokenized dataset (no runtime tokenization)
151
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
152
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
153
+ Dataset split: 3027764 train, 159356 validation
154
+ Training examples (packed, train split): 3027764
155
+ Validation examples (packed, val split): 159356
156
+ Per-device batch size: 2
157
+ World size (GPUs): 1
158
+ Gradient accumulation steps: 10
159
+ Effective batch size (seqs/step): 20
160
+ Sequence length: 8192
161
+ Tokens per optimizer step: 163,840
162
+ Micro-batches per epoch (per rank): 1513882
163
+ Optimizer steps per epoch: 151388
164
+ Total optimizer steps: 44000 (capped by max_steps; 0.29 epochs)
165
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed45/run_20260819_050018/checkpoint-26000...
166
+ Overriding checkpoint LR with config LR: 0.0005
167
+ Resumed from checkpoint at step 26000 (epoch 0, micro_step 260000)
168
+ Resuming training from step 26000
169
+ Starting training...
170
+ Validation at step 26000: Loss = 2.9390, Perplexity = 18.90
171
+ Traceback (most recent call last):
172
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/resource_sharer.py", line 138, in _serve
173
+ with self._listener.accept() as conn:
174
+ ~~~~~~~~~~~~~~~~~~~~~^^
175
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 496, in accept
176
+ answer_challenge(c, self._authkey)
177
+ ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
178
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 975, in answer_challenge
179
+ message = connection.recv_bytes(256) # reject large message
180
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 217, in recv_bytes
181
+ buf = self._recv_bytes(maxlength)
182
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 431, in _recv_bytes
183
+ buf = self._recv(4)
184
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 396, in _recv
185
+ chunk = read(handle, remaining)
186
+ ConnectionResetError: [Errno 104] Connection reset by peer
epfl_final/train_logs/dropout_p28_resume/seed45.20260819T171033Z.log ADDED
The diff for this file is too large to render. See raw diff
 
epfl_final/train_logs/dropout_p28_resume/seed46.20260819T170628Z.log ADDED
@@ -0,0 +1,170 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Auto-resuming from latest checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed46/run_20260819_050018/checkpoint-27000
2
+ Resuming: Using existing output directory /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed46/run_20260819_050018
3
+ Running config:
4
+ {
5
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
6
+ "model_dtype": "bfloat16",
7
+ "init_from_scratch": true,
8
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
9
+ "val_path": null,
10
+ "max_seq_length": 8192,
11
+ "num_workers": 8,
12
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed46/run_20260819_050018",
13
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed46/run_20260819_050018/checkpoint-27000",
14
+ "force_new_run": false,
15
+ "num_epochs": 1,
16
+ "max_steps": 44000,
17
+ "per_device_batch_size": 2,
18
+ "gradient_accumulation_steps": 10,
19
+ "learning_rate": 0.0005,
20
+ "weight_decay": 0.01,
21
+ "warmup_steps": 750,
22
+ "lr_scheduler": "cosine",
23
+ "max_grad_norm": 1.0,
24
+ "save_steps": 1000,
25
+ "eval_steps": 1000,
26
+ "eval_batches": 50,
27
+ "logging_steps": 10,
28
+ "seed": 46,
29
+ "verbose": true,
30
+ "checkpoint_activations": false,
31
+ "strategy": "ddp",
32
+ "mixed_precision": "bf16",
33
+ "wandb_enabled": false,
34
+ "wandb_project": "stochastic_sparsity",
35
+ "wandb_name": "l8k365m_dense_dropout_p28_seed46",
36
+ "wandb_tags": [
37
+ "qwen3",
38
+ "365m",
39
+ "8k",
40
+ "scaleladder",
41
+ "dense"
42
+ ],
43
+ "hidden_size": 768,
44
+ "intermediate_size": 3072,
45
+ "num_attention_heads": 6,
46
+ "num_key_value_heads": 2,
47
+ "num_hidden_layers": 28,
48
+ "head_dim": null,
49
+ "attention_dropout": 0.28,
50
+ "rope_train_position_scale_set": null,
51
+ "attention_type": "sparse_stochastic",
52
+ "sparse_block_size": 128,
53
+ "sparse_num_random_blocks": 31,
54
+ "sparse_mask_pool_size": 1,
55
+ "sparse_val_dense_compare": false,
56
+ "sparse_layer_indices": null,
57
+ "sparse_mask_cache_dir": null,
58
+ "sparse_mask_seed": 0,
59
+ "sparse_mixed_nrb_set": null,
60
+ "dense_mode": true,
61
+ "dense_step_intervals": null
62
+ }
63
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
64
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed46/run_20260819_050018/checkpoint-27000
65
+ Checkpoint activations: False
66
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
67
+
68
+
69
+ ================================================================================
70
+ MODEL ARCHITECTURE
71
+ ================================================================================
72
+ Model config:
73
+ vocab_size: 151936
74
+ hidden_size: 768
75
+ num_hidden_layers: 28
76
+ num_attention_heads: 6
77
+ num_key_value_heads: 2
78
+ head_dim: 128
79
+ intermediate_size: 3072
80
+ max_position_embeddings: 32768
81
+ rope_theta: 1000000
82
+ rope_train_position_scale_set: None
83
+ attention_type: sparse_stochastic
84
+ sparse_block_size: 128
85
+ sparse_num_random_blocks: 31
86
+ sparse_mask_pool_size: 1
87
+ sparse_max_seq_length: 8192
88
+
89
+ Parameter count:
90
+ Total parameters: 358,958,848
91
+ Trainable parameters: 358,958,848
92
+ Non-trainable parameters: 0
93
+ Model size (BF16): 0.72 GB
94
+
95
+ --------------------------------------------------------------------------------
96
+ LAYER-BY-LAYER SUMMARY
97
+ --------------------------------------------------------------------------------
98
+ ================================================================================================================================================================
99
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
100
+ ================================================================================================================================================================
101
+ Qwen3ForCausalLM [2, 8192] -- -- True
102
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
103
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
104
+ │ └─ModuleList: 2-2 -- -- -- True
105
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
106
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
107
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
108
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
109
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
110
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
111
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
132
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
133
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
134
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
135
+ ================================================================================================================================================================
136
+ Total params: 475,645,696
137
+ Trainable params: 475,645,696
138
+ Non-trainable params: 0
139
+ Total mult-adds (Units.MEGABYTES): 951.29
140
+ ================================================================================================================================================================
141
+ Input size (MB): 0.13
142
+ Forward/backward pass size (MB): 31197.23
143
+ Params size (MB): 951.29
144
+ Estimated Total Size (MB): 32148.66
145
+ ================================================================================================================================================================
146
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
147
+ ================================================================================
148
+
149
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
150
+ Using pre-tokenized dataset (no runtime tokenization)
151
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
152
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
153
+ Dataset split: 3027764 train, 159356 validation
154
+ Training examples (packed, train split): 3027764
155
+ Validation examples (packed, val split): 159356
156
+ Per-device batch size: 2
157
+ World size (GPUs): 1
158
+ Gradient accumulation steps: 10
159
+ Effective batch size (seqs/step): 20
160
+ Sequence length: 8192
161
+ Tokens per optimizer step: 163,840
162
+ Micro-batches per epoch (per rank): 1513882
163
+ Optimizer steps per epoch: 151388
164
+ Total optimizer steps: 44000 (capped by max_steps; 0.29 epochs)
165
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_dropout_p28_seed46/run_20260819_050018/checkpoint-27000...
166
+ Overriding checkpoint LR with config LR: 0.0005
167
+ Resumed from checkpoint at step 27000 (epoch 0, micro_step 270000)
168
+ Resuming training from step 27000
169
+ Starting training...
170
+ Validation at step 27000: Loss = 2.9277, Perplexity = 18.68
epfl_final/train_logs/dropout_p28_resume/seed46.20260819T171033Z.log ADDED
The diff for this file is too large to render. See raw diff
 
epfl_final/train_logs/hf_upload_final_checkpoints.log ADDED
The diff for this file is too large to render. See raw diff
 
epfl_final/train_logs/hf_upload_paper_compendium.log ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ https://huggingface.co/datasets/rubentium/sparse-results/commit/7ac9f9a6b4ea9cda8416e92fa32530882336328a
2
+ https://huggingface.co/datasets/rubentium/sparse-results/commit/00fff037c5f74b97476def354d54da9232146008
3
+ No files have been modified since last commit. Skipping to prevent empty commit.
4
+ https://huggingface.co/datasets/rubentium/sparse-results/commit/00fff037c5f74b97476def354d54da9232146008
epfl_final/train_logs/hf_upload_september_results.log ADDED
The diff for this file is too large to render. See raw diff
 
epfl_final/train_logs/mixed8k3b_mask_build/gpu0.log ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ layer=28 building
2
+ layer=28 complete bytes=769559032
3
+ layer=32 building
4
+ layer=32 complete bytes=769559032
epfl_final/train_logs/mixed8k3b_mask_build/gpu1.log ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ layer=29 building
2
+ layer=29 complete bytes=769559032
3
+ layer=33 building
4
+ layer=33 complete bytes=769559032
epfl_final/train_logs/mixed8k3b_mask_build/gpu2.log ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ layer=30 building
2
+ layer=30 complete bytes=769559032
3
+ layer=34 building
4
+ layer=34 complete bytes=769559032
epfl_final/train_logs/mixed8k3b_mask_build/gpu3.log ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ layer=31 building
2
+ layer=31 complete bytes=769559032
3
+ layer=35 building
4
+ layer=35 complete bytes=769559032
epfl_final/train_logs/p2_1p2b_dense_seed72_6gpu_resume48000.log ADDED
@@ -0,0 +1,458 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ *****************************************
3
+ Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
4
+ *****************************************
5
+ Resuming: Using existing output directory /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_dense_seed72_repaired/run_20260906_repair_from9250
6
+ Running config:
7
+ {
8
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
9
+ "model_dtype": "bfloat16",
10
+ "init_from_scratch": true,
11
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
12
+ "val_path": null,
13
+ "max_seq_length": 8192,
14
+ "num_workers": 8,
15
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_dense_seed72_repaired/run_20260906_repair_from9250",
16
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_dense_seed72_repaired/run_20260906_repair_from9250/checkpoint-48000",
17
+ "force_new_run": false,
18
+ "num_epochs": 1,
19
+ "max_steps": 50462,
20
+ "per_device_batch_size": 1,
21
+ "gradient_accumulation_steps": 10,
22
+ "learning_rate": 0.0005,
23
+ "weight_decay": 0.01,
24
+ "warmup_steps": 750,
25
+ "lr_scheduler": "cosine",
26
+ "max_grad_norm": 1.0,
27
+ "save_steps": 250,
28
+ "eval_steps": 250,
29
+ "eval_batches": 50,
30
+ "logging_steps": 10,
31
+ "seed": 72,
32
+ "verbose": true,
33
+ "checkpoint_activations": false,
34
+ "resume_micro_step_override": 480000,
35
+ "strategy": "ddp",
36
+ "mixed_precision": "bf16",
37
+ "wandb_enabled": false,
38
+ "wandb_project": "stochastic_sparsity",
39
+ "wandb_name": "1.2b dense 8k seed72 6gpu resume48000",
40
+ "wandb_tags": [
41
+ "qwen3",
42
+ "1.2b",
43
+ "8k",
44
+ "p2",
45
+ "dense",
46
+ "seed72"
47
+ ],
48
+ "hidden_size": 1536,
49
+ "intermediate_size": 6144,
50
+ "num_attention_heads": 12,
51
+ "num_key_value_heads": 4,
52
+ "num_hidden_layers": 28,
53
+ "head_dim": null,
54
+ "attention_dropout": null,
55
+ "rope_train_position_scale_set": null,
56
+ "attention_type": "sparse_stochastic",
57
+ "sparse_block_size": 128,
58
+ "sparse_num_random_blocks": 31,
59
+ "sparse_mask_pool_size": 1,
60
+ "sparse_val_dense_compare": false,
61
+ "sparse_layer_indices": null,
62
+ "sparse_mask_cache_dir": null,
63
+ "sparse_mask_seed": 0,
64
+ "sparse_mixed_nrb_set": null,
65
+ "dense_mode": true,
66
+ "dense_step_intervals": null
67
+ }
68
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
69
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_dense_seed72_repaired/run_20260906_repair_from9250/checkpoint-48000
70
+ Checkpoint activations: False
71
+ Checkpoint activations: FalseCheckpoint activations: FalseCheckpoint activations: False
72
+
73
+
74
+ Checkpoint activations: False
75
+ Checkpoint activations: False
76
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
77
+
78
+
79
+ ================================================================================
80
+ MODEL ARCHITECTURE
81
+ ================================================================================
82
+ Model config:
83
+ vocab_size: 151936
84
+ hidden_size: 1536
85
+ num_hidden_layers: 28
86
+ num_attention_heads: 12
87
+ num_key_value_heads: 4
88
+ head_dim: 128
89
+ intermediate_size: 6144
90
+ max_position_embeddings: 32768
91
+ rope_theta: 1000000
92
+ rope_train_position_scale_set: None
93
+ attention_type: sparse_stochastic
94
+ sparse_block_size: 128
95
+ sparse_num_random_blocks: 31
96
+ sparse_mask_pool_size: 1
97
+ sparse_max_seq_length: 8192
98
+
99
+ Parameter count:
100
+ Total parameters: 1,202,352,640
101
+ Trainable parameters: 1,202,352,640
102
+ Non-trainable parameters: 0
103
+ Model size (BF16): 2.40 GB
104
+
105
+ --------------------------------------------------------------------------------
106
+ LAYER-BY-LAYER SUMMARY
107
+ --------------------------------------------------------------------------------
108
+ ================================================================================================================================================================
109
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
110
+ ================================================================================================================================================================
111
+ Qwen3ForCausalLM [1, 8192] -- -- True
112
+ ├─Qwen3Model: 1-1 -- [1, 8192, 1536] -- True
113
+ │ └─Embedding: 2-1 [1, 8192] [1, 8192, 1536] 233,373,696 True
114
+ │ └─ModuleList: 2-2 -- -- -- True
115
+ │ │ └─Qwen3DecoderLayer: 3-1 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
116
+ │ │ └─Qwen3DecoderLayer: 3-2 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
117
+ │ │ └─Qwen3DecoderLayer: 3-3 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
118
+ │ │ └─Qwen3DecoderLayer: 3-4 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
119
+ │ │ └─Qwen3DecoderLayer: 3-5 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
120
+ │ │ └─Qwen3DecoderLayer: 3-6 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
121
+ │ │ └─Qwen3DecoderLayer: 3-7 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
122
+ │ │ └─Qwen3DecoderLayer: 3-8 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
123
+ │ │ └─Qwen3DecoderLayer: 3-9 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
124
+ │ │ └─Qwen3DecoderLayer: 3-10 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
125
+ │ │ └─Qwen3DecoderLayer: 3-11 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
126
+ │ │ └─Qwen3DecoderLayer: 3-12 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
127
+ │ │ └─Qwen3DecoderLayer: 3-13 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
128
+ │ │ └─Qwen3DecoderLayer: 3-14 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
129
+ │ │ └─Qwen3DecoderLayer: 3-15 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
130
+ │ │ └─Qwen3DecoderLayer: 3-16 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
131
+ │ │ └─Qwen3DecoderLayer: 3-17 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
132
+ │ │ └─Qwen3DecoderLayer: 3-18 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
133
+ │ │ └─Qwen3DecoderLayer: 3-19 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
134
+ │ │ └─Qwen3DecoderLayer: 3-20 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
135
+ │ │ └─Qwen3DecoderLayer: 3-21 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
136
+ │ │ └─Qwen3DecoderLayer: 3-22 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
137
+ │ │ └─Qwen3DecoderLayer: 3-23 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
138
+ │ │ └─Qwen3DecoderLayer: 3-24 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
139
+ │ │ └─Qwen3DecoderLayer: 3-25 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
140
+ │ │ └─Qwen3DecoderLayer: 3-26 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
141
+ │ │ └─Qwen3DecoderLayer: 3-27 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
142
+ │ │ └─Qwen3DecoderLayer: 3-28 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
143
+ │ └─Qwen3RMSNorm: 2-3 [1, 8192, 1536] [1, 8192, 1536] 1,536 True
144
+ ├─Linear: 1-2 [1, 8192, 1536] [1, 8192, 151936] 233,373,696 True
145
+ ================================================================================================================================================================
146
+ Total params: 1,435,726,336
147
+ Trainable params: 1,435,726,336
148
+ Non-trainable params: 0
149
+ Total mult-adds (Units.GIGABYTES): 1.44
150
+ ================================================================================================================================================================
151
+ Input size (MB): 0.07
152
+ Forward/backward pass size (MB): 26218.59
153
+ Params size (MB): 2871.45
154
+ Estimated Total Size (MB): 29090.11
155
+ ================================================================================================================================================================
156
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (233,373,696 params). True unique parameter count: 1,202,352,640 (1202.4M)
157
+ ================================================================================
158
+
159
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
160
+ Using pre-tokenized dataset (no runtime tokenization)
161
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
162
+
163
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
164
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
165
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
166
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
167
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
168
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
169
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
170
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
171
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
172
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
173
+ Dataset split: 3027764 train, 159356 validation
174
+ Training examples (packed, train split): 3027764
175
+ Validation examples (packed, val split): 159356
176
+ Per-device batch size: 1
177
+ World size (GPUs): 6
178
+ Gradient accumulation steps: 10
179
+ Effective batch size (seqs/step): 60
180
+ Sequence length: 8192
181
+ Tokens per optimizer step: 491,520
182
+ Micro-batches per epoch (per rank): 504628
183
+ Optimizer steps per epoch: 50462
184
+ Total optimizer steps: 50462 (capped by max_steps; 1.00 epochs)
185
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_dense_seed72_repaired/run_20260906_repair_from9250/checkpoint-48000...
186
+ Preserving checkpoint LR state for exact resume: current=[3.019862028798087e-06], schedule_base=[0.0005]
187
+ Resumed from checkpoint at step 48000 (epoch 0, micro_step 480000)
188
+ Overriding resume micro-step 480000 -> 480000 for DDP migration
189
+ Resuming training from step 48000
190
+ Starting training...
191
+ Validation at step 48000: Loss = 2.5308, Perplexity = 12.56
192
+ Switching attention mode for step 48001: dense (dense step interval)
193
+ Step 48010/50462 | Loss: 2.5039 | LR: 3.00e-06 | 67290.4ms/step | 0.9 samples/s
194
+ Step 48020/50462 | Loss: 2.4784 | LR: 2.97e-06 | 1890.7ms/step | 31.7 samples/s
195
+ Step 48030/50462 | Loss: 2.4669 | LR: 2.95e-06 | 1893.5ms/step | 31.7 samples/s
196
+ Step 48040/50462 | Loss: 2.4977 | LR: 2.92e-06 | 1899.6ms/step | 31.6 samples/s
197
+ Step 48050/50462 | Loss: 2.5004 | LR: 2.90e-06 | 1901.5ms/step | 31.6 samples/s
198
+ Step 48060/50462 | Loss: 2.4906 | LR: 2.87e-06 | 1903.3ms/step | 31.5 samples/s
199
+ Step 48070/50462 | Loss: 2.4853 | LR: 2.85e-06 | 1903.8ms/step | 31.5 samples/s
200
+ Step 48080/50462 | Loss: 2.4770 | LR: 2.83e-06 | 1907.0ms/step | 31.5 samples/s
201
+ Step 48090/50462 | Loss: 2.4893 | LR: 2.80e-06 | 1907.5ms/step | 31.5 samples/s
202
+ Step 48100/50462 | Loss: 2.5210 | LR: 2.78e-06 | 1909.4ms/step | 31.4 samples/s
203
+ Step 48110/50462 | Loss: 2.4794 | LR: 2.76e-06 | 1909.7ms/step | 31.4 samples/s
204
+ Step 48120/50462 | Loss: 2.4560 | LR: 2.73e-06 | 1913.5ms/step | 31.4 samples/s
205
+ Step 48130/50462 | Loss: 2.5125 | LR: 2.71e-06 | 1911.5ms/step | 31.4 samples/s
206
+ Step 48140/50462 | Loss: 2.4971 | LR: 2.69e-06 | 1914.7ms/step | 31.3 samples/s
207
+ Step 48150/50462 | Loss: 2.4777 | LR: 2.66e-06 | 1909.1ms/step | 31.4 samples/s
208
+ Step 48160/50462 | Loss: 2.4904 | LR: 2.64e-06 | 1910.7ms/step | 31.4 samples/s
209
+ Step 48170/50462 | Loss: 2.4812 | LR: 2.62e-06 | 1914.6ms/step | 31.3 samples/s
210
+ Step 48180/50462 | Loss: 2.4968 | LR: 2.60e-06 | 1912.6ms/step | 31.4 samples/s
211
+ Step 48190/50462 | Loss: 2.5074 | LR: 2.57e-06 | 1914.8ms/step | 31.3 samples/s
212
+ Step 48200/50462 | Loss: 2.5158 | LR: 2.55e-06 | 1916.4ms/step | 31.3 samples/s
213
+ Step 48210/50462 | Loss: 2.5322 | LR: 2.53e-06 | 1916.7ms/step | 31.3 samples/s
214
+ Step 48220/50462 | Loss: 2.4967 | LR: 2.51e-06 | 1915.3ms/step | 31.3 samples/s
215
+ Step 48230/50462 | Loss: 2.4765 | LR: 2.48e-06 | 1915.8ms/step | 31.3 samples/s
216
+ Step 48240/50462 | Loss: 2.4561 | LR: 2.46e-06 | 1909.7ms/step | 31.4 samples/s
217
+ Step 48250/50462 | Loss: 2.5071 | LR: 2.44e-06 | 1911.4ms/step | 31.4 samples/s
218
+ Validation at step 48250: Loss = 2.5308, Perplexity = 12.56
219
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_dense_seed72_repaired/run_20260906_repair_from9250/checkpoint-48250
220
+ Step 48260/50462 | Loss: 2.4908 | LR: 2.42e-06 | 3326.7ms/step | 18.0 samples/s
221
+ Step 48270/50462 | Loss: 2.5435 | LR: 2.39e-06 | 1901.7ms/step | 31.6 samples/s
222
+ Step 48280/50462 | Loss: 2.5034 | LR: 2.37e-06 | 1905.4ms/step | 31.5 samples/s
223
+ Step 48290/50462 | Loss: 2.4704 | LR: 2.35e-06 | 1913.0ms/step | 31.4 samples/s
224
+ Step 48300/50462 | Loss: 2.4629 | LR: 2.33e-06 | 1915.0ms/step | 31.3 samples/s
225
+ Step 48310/50462 | Loss: 2.5134 | LR: 2.31e-06 | 1915.8ms/step | 31.3 samples/s
226
+ Step 48320/50462 | Loss: 2.4767 | LR: 2.29e-06 | 1918.6ms/step | 31.3 samples/s
227
+ Step 48330/50462 | Loss: 2.4985 | LR: 2.27e-06 | 1920.9ms/step | 31.2 samples/s
228
+ Step 48340/50462 | Loss: 2.5163 | LR: 2.24e-06 | 1916.9ms/step | 31.3 samples/s
229
+ Step 48350/50462 | Loss: 2.4515 | LR: 2.22e-06 | 1919.0ms/step | 31.3 samples/s
230
+ Step 48360/50462 | Loss: 2.5309 | LR: 2.20e-06 | 1920.5ms/step | 31.2 samples/s
231
+ Step 48370/50462 | Loss: 2.4337 | LR: 2.18e-06 | 1917.6ms/step | 31.3 samples/s
232
+ Step 48380/50462 | Loss: 2.4947 | LR: 2.16e-06 | 1920.2ms/step | 31.2 samples/s
233
+ Step 48390/50462 | Loss: 2.4810 | LR: 2.14e-06 | 1919.1ms/step | 31.3 samples/s
234
+ Step 48400/50462 | Loss: 2.5271 | LR: 2.12e-06 | 1919.0ms/step | 31.3 samples/s
235
+ Step 48410/50462 | Loss: 2.4509 | LR: 2.10e-06 | 1920.2ms/step | 31.2 samples/s
236
+ Step 48420/50462 | Loss: 2.5221 | LR: 2.08e-06 | 1913.4ms/step | 31.4 samples/s
237
+ Step 48430/50462 | Loss: 2.4440 | LR: 2.06e-06 | 1914.9ms/step | 31.3 samples/s
238
+ Step 48440/50462 | Loss: 2.5301 | LR: 2.04e-06 | 1918.2ms/step | 31.3 samples/s
239
+ Step 48450/50462 | Loss: 2.5095 | LR: 2.02e-06 | 1919.6ms/step | 31.3 samples/s
240
+ Step 48460/50462 | Loss: 2.4772 | LR: 2.00e-06 | 1920.1ms/step | 31.2 samples/s
241
+ Step 48470/50462 | Loss: 2.5050 | LR: 1.98e-06 | 1918.8ms/step | 31.3 samples/s
242
+ Step 48480/50462 | Loss: 2.4982 | LR: 1.96e-06 | 1917.1ms/step | 31.3 samples/s
243
+ Step 48490/50462 | Loss: 2.4799 | LR: 1.94e-06 | 1920.1ms/step | 31.2 samples/s
244
+ Step 48500/50462 | Loss: 2.4782 | LR: 1.92e-06 | 1919.4ms/step | 31.3 samples/s
245
+ Validation at step 48500: Loss = 2.5308, Perplexity = 12.56
246
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_dense_seed72_repaired/run_20260906_repair_from9250/checkpoint-48500
247
+ Step 48510/50462 | Loss: 2.5053 | LR: 1.90e-06 | 3299.9ms/step | 18.2 samples/s
248
+ Step 48520/50462 | Loss: 2.5344 | LR: 1.88e-06 | 1904.4ms/step | 31.5 samples/s
249
+ Step 48530/50462 | Loss: 2.4712 | LR: 1.86e-06 | 1908.9ms/step | 31.4 samples/s
250
+ Step 48540/50462 | Loss: 2.4797 | LR: 1.84e-06 | 1913.2ms/step | 31.4 samples/s
251
+ Step 48550/50462 | Loss: 2.4895 | LR: 1.82e-06 | 1914.4ms/step | 31.3 samples/s
252
+ Step 48560/50462 | Loss: 2.4783 | LR: 1.80e-06 | 1916.7ms/step | 31.3 samples/s
253
+ Step 48570/50462 | Loss: 2.4894 | LR: 1.78e-06 | 1918.4ms/step | 31.3 samples/s
254
+ Step 48580/50462 | Loss: 2.4707 | LR: 1.77e-06 | 1919.6ms/step | 31.3 samples/s
255
+ Step 48590/50462 | Loss: 2.5240 | LR: 1.75e-06 | 1919.6ms/step | 31.3 samples/s
256
+ Step 48600/50462 | Loss: 2.4720 | LR: 1.73e-06 | 1919.5ms/step | 31.3 samples/s
257
+ Step 48610/50462 | Loss: 2.5031 | LR: 1.71e-06 | 1919.3ms/step | 31.3 samples/s
258
+ Step 48620/50462 | Loss: 2.4785 | LR: 1.69e-06 | 1921.2ms/step | 31.2 samples/s
259
+ Step 48630/50462 | Loss: 2.5152 | LR: 1.67e-06 | 1924.5ms/step | 31.2 samples/s
260
+ Step 48640/50462 | Loss: 2.4918 | LR: 1.66e-06 | 1922.6ms/step | 31.2 samples/s
261
+ Step 48650/50462 | Loss: 2.4886 | LR: 1.64e-06 | 1917.7ms/step | 31.3 samples/s
262
+ Step 48660/50462 | Loss: 2.5037 | LR: 1.62e-06 | 1920.7ms/step | 31.2 samples/s
263
+ Step 48670/50462 | Loss: 2.4983 | LR: 1.60e-06 | 1919.6ms/step | 31.3 samples/s
264
+ Step 48680/50462 | Loss: 2.4882 | LR: 1.58e-06 | 1918.8ms/step | 31.3 samples/s
265
+ Step 48690/50462 | Loss: 2.4700 | LR: 1.57e-06 | 1921.6ms/step | 31.2 samples/s
266
+ Step 48700/50462 | Loss: 2.4808 | LR: 1.55e-06 | 1918.7ms/step | 31.3 samples/s
267
+ Step 48710/50462 | Loss: 2.5029 | LR: 1.53e-06 | 1918.3ms/step | 31.3 samples/s
268
+ Step 48720/50462 | Loss: 2.4767 | LR: 1.51e-06 | 1919.8ms/step | 31.3 samples/s
269
+ Step 48730/50462 | Loss: 2.4474 | LR: 1.50e-06 | 1914.2ms/step | 31.3 samples/s
270
+ Step 48740/50462 | Loss: 2.4804 | LR: 1.48e-06 | 1918.1ms/step | 31.3 samples/s
271
+ Step 48750/50462 | Loss: 2.4528 | LR: 1.46e-06 | 1921.8ms/step | 31.2 samples/s
272
+ Validation at step 48750: Loss = 2.5308, Perplexity = 12.56
273
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_dense_seed72_repaired/run_20260906_repair_from9250/checkpoint-48750
274
+ Step 48760/50462 | Loss: 2.4894 | LR: 1.44e-06 | 3282.2ms/step | 18.3 samples/s
275
+ Step 48770/50462 | Loss: 2.4816 | LR: 1.43e-06 | 1897.4ms/step | 31.6 samples/s
276
+ Step 48780/50462 | Loss: 2.4737 | LR: 1.41e-06 | 1904.5ms/step | 31.5 samples/s
277
+ Step 48790/50462 | Loss: 2.4629 | LR: 1.39e-06 | 1910.0ms/step | 31.4 samples/s
278
+ Step 48800/50462 | Loss: 2.4708 | LR: 1.38e-06 | 1916.5ms/step | 31.3 samples/s
279
+ Step 48810/50462 | Loss: 2.4680 | LR: 1.36e-06 | 1916.3ms/step | 31.3 samples/s
280
+ Step 48820/50462 | Loss: 2.5239 | LR: 1.34e-06 | 1920.4ms/step | 31.2 samples/s
281
+ Step 48830/50462 | Loss: 2.5319 | LR: 1.33e-06 | 1918.8ms/step | 31.3 samples/s
282
+ Step 48840/50462 | Loss: 2.5208 | LR: 1.31e-06 | 1920.1ms/step | 31.2 samples/s
283
+ Step 48850/50462 | Loss: 2.4690 | LR: 1.30e-06 | 1921.4ms/step | 31.2 samples/s
284
+ Step 48860/50462 | Loss: 2.4558 | LR: 1.28e-06 | 1918.9ms/step | 31.3 samples/s
285
+ Step 48870/50462 | Loss: 2.5005 | LR: 1.26e-06 | 1916.9ms/step | 31.3 samples/s
286
+ Step 48880/50462 | Loss: 2.4516 | LR: 1.25e-06 | 1916.5ms/step | 31.3 samples/s
287
+ Step 48890/50462 | Loss: 2.5068 | LR: 1.23e-06 | 1916.6ms/step | 31.3 samples/s
288
+ Step 48900/50462 | Loss: 2.4834 | LR: 1.22e-06 | 1917.5ms/step | 31.3 samples/s
289
+ Step 48910/50462 | Loss: 2.5049 | LR: 1.20e-06 | 1917.7ms/step | 31.3 samples/s
290
+ Step 48920/50462 | Loss: 2.5018 | LR: 1.19e-06 | 1919.8ms/step | 31.3 samples/s
291
+ Step 48930/50462 | Loss: 2.4797 | LR: 1.17e-06 | 1922.2ms/step | 31.2 samples/s
292
+ Step 48940/50462 | Loss: 2.5214 | LR: 1.16e-06 | 1917.8ms/step | 31.3 samples/s
293
+ Step 48950/50462 | Loss: 2.4621 | LR: 1.14e-06 | 1915.9ms/step | 31.3 samples/s
294
+ Step 48960/50462 | Loss: 2.4932 | LR: 1.13e-06 | 1915.4ms/step | 31.3 samples/s
295
+ Step 48970/50462 | Loss: 2.5043 | LR: 1.11e-06 | 1917.4ms/step | 31.3 samples/s
296
+ Step 48980/50462 | Loss: 2.5364 | LR: 1.10e-06 | 1917.2ms/step | 31.3 samples/s
297
+ Step 48990/50462 | Loss: 2.5346 | LR: 1.08e-06 | 1918.9ms/step | 31.3 samples/s
298
+ Step 49000/50462 | Loss: 2.4763 | LR: 1.07e-06 | 1919.1ms/step | 31.3 samples/s
299
+ Validation at step 49000: Loss = 2.5308, Perplexity = 12.56
300
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_dense_seed72_repaired/run_20260906_repair_from9250/checkpoint-49000
301
+ Step 49010/50462 | Loss: 2.4633 | LR: 1.05e-06 | 3236.9ms/step | 18.5 samples/s
302
+ Step 49020/50462 | Loss: 2.5051 | LR: 1.04e-06 | 1903.6ms/step | 31.5 samples/s
303
+ Step 49030/50462 | Loss: 2.4992 | LR: 1.02e-06 | 1908.8ms/step | 31.4 samples/s
304
+ Step 49040/50462 | Loss: 2.4629 | LR: 1.01e-06 | 1909.2ms/step | 31.4 samples/s
305
+ Step 49050/50462 | Loss: 2.4594 | LR: 9.95e-07 | 1913.0ms/step | 31.4 samples/s
306
+ Step 49060/50462 | Loss: 2.5120 | LR: 9.81e-07 | 1919.0ms/step | 31.3 samples/s
307
+ Step 49070/50462 | Loss: 2.5279 | LR: 9.67e-07 | 1920.3ms/step | 31.2 samples/s
308
+ Step 49080/50462 | Loss: 2.4668 | LR: 9.53e-07 | 1916.2ms/step | 31.3 samples/s
309
+ Step 49090/50462 | Loss: 2.5100 | LR: 9.39e-07 | 1918.8ms/step | 31.3 samples/s
310
+ Step 49100/50462 | Loss: 2.5228 | LR: 9.25e-07 | 1921.0ms/step | 31.2 samples/s
311
+ Step 49110/50462 | Loss: 2.4796 | LR: 9.12e-07 | 1919.3ms/step | 31.3 samples/s
312
+ Step 49120/50462 | Loss: 2.5117 | LR: 8.99e-07 | 1921.8ms/step | 31.2 samples/s
313
+ Step 49130/50462 | Loss: 2.5019 | LR: 8.85e-07 | 1923.0ms/step | 31.2 samples/s
314
+ Step 49140/50462 | Loss: 2.4792 | LR: 8.72e-07 | 1923.0ms/step | 31.2 samples/s
315
+ Step 49150/50462 | Loss: 2.4853 | LR: 8.59e-07 | 1922.7ms/step | 31.2 samples/s
316
+ Step 49160/50462 | Loss: 2.5095 | LR: 8.46e-07 | 1922.9ms/step | 31.2 samples/s
317
+ Step 49170/50462 | Loss: 2.4560 | LR: 8.33e-07 | 1924.0ms/step | 31.2 samples/s
318
+ Step 49180/50462 | Loss: 2.4815 | LR: 8.20e-07 | 1922.4ms/step | 31.2 samples/s
319
+ Step 49190/50462 | Loss: 2.4554 | LR: 8.07e-07 | 1928.8ms/step | 31.1 samples/s
320
+ Step 49200/50462 | Loss: 2.4793 | LR: 7.95e-07 | 1923.9ms/step | 31.2 samples/s
321
+ Step 49210/50462 | Loss: 2.5263 | LR: 7.82e-07 | 1923.5ms/step | 31.2 samples/s
322
+ Step 49220/50462 | Loss: 2.4810 | LR: 7.70e-07 | 1925.1ms/step | 31.2 samples/s
323
+ Step 49230/50462 | Loss: 2.5096 | LR: 7.57e-07 | 1921.1ms/step | 31.2 samples/s
324
+ Step 49240/50462 | Loss: 2.4823 | LR: 7.45e-07 | 1923.4ms/step | 31.2 samples/s
325
+ Step 49250/50462 | Loss: 2.4983 | LR: 7.33e-07 | 1924.0ms/step | 31.2 samples/s
326
+ Validation at step 49250: Loss = 2.5308, Perplexity = 12.56
327
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_dense_seed72_repaired/run_20260906_repair_from9250/checkpoint-49250
328
+ Step 49260/50462 | Loss: 2.5546 | LR: 7.21e-07 | 3277.3ms/step | 18.3 samples/s
329
+ Step 49270/50462 | Loss: 2.5029 | LR: 7.09e-07 | 1903.8ms/step | 31.5 samples/s
330
+ Step 49280/50462 | Loss: 2.4629 | LR: 6.97e-07 | 1910.0ms/step | 31.4 samples/s
331
+ Step 49290/50462 | Loss: 2.4968 | LR: 6.85e-07 | 1916.8ms/step | 31.3 samples/s
332
+ Step 49300/50462 | Loss: 2.4807 | LR: 6.74e-07 | 1918.1ms/step | 31.3 samples/s
333
+ Step 49310/50462 | Loss: 2.4897 | LR: 6.62e-07 | 1919.6ms/step | 31.3 samples/s
334
+ Step 49320/50462 | Loss: 2.4898 | LR: 6.51e-07 | 1922.0ms/step | 31.2 samples/s
335
+ Step 49330/50462 | Loss: 2.4795 | LR: 6.39e-07 | 1923.9ms/step | 31.2 samples/s
336
+ Step 49340/50462 | Loss: 2.4969 | LR: 6.28e-07 | 1921.8ms/step | 31.2 samples/s
337
+ Step 49350/50462 | Loss: 2.4734 | LR: 6.17e-07 | 1921.9ms/step | 31.2 samples/s
338
+ Step 49360/50462 | Loss: 2.4803 | LR: 6.06e-07 | 1922.1ms/step | 31.2 samples/s
339
+ Step 49370/50462 | Loss: 2.4921 | LR: 5.95e-07 | 1923.3ms/step | 31.2 samples/s
340
+ Step 49380/50462 | Loss: 2.4945 | LR: 5.84e-07 | 1922.1ms/step | 31.2 samples/s
341
+ Step 49390/50462 | Loss: 2.5005 | LR: 5.73e-07 | 1923.5ms/step | 31.2 samples/s
342
+ Step 49400/50462 | Loss: 2.5165 | LR: 5.63e-07 | 1924.0ms/step | 31.2 samples/s
343
+ Step 49410/50462 | Loss: 2.5049 | LR: 5.52e-07 | 1923.0ms/step | 31.2 samples/s
344
+ Step 49420/50462 | Loss: 2.5230 | LR: 5.42e-07 | 1920.1ms/step | 31.2 samples/s
345
+ Step 49430/50462 | Loss: 2.5225 | LR: 5.31e-07 | 1923.1ms/step | 31.2 samples/s
346
+ Step 49440/50462 | Loss: 2.5137 | LR: 5.21e-07 | 1923.1ms/step | 31.2 samples/s
347
+ Step 49450/50462 | Loss: 2.4978 | LR: 5.11e-07 | 1922.6ms/step | 31.2 samples/s
348
+ Step 49460/50462 | Loss: 2.4782 | LR: 5.01e-07 | 1921.3ms/step | 31.2 samples/s
349
+ Step 49470/50462 | Loss: 2.5126 | LR: 4.91e-07 | 1918.2ms/step | 31.3 samples/s
350
+ Step 49480/50462 | Loss: 2.4859 | LR: 4.81e-07 | 1920.8ms/step | 31.2 samples/s
351
+ Step 49490/50462 | Loss: 2.4815 | LR: 4.72e-07 | 1923.3ms/step | 31.2 samples/s
352
+ Step 49500/50462 | Loss: 2.4959 | LR: 4.62e-07 | 1922.0ms/step | 31.2 samples/s
353
+ Validation at step 49500: Loss = 2.5308, Perplexity = 12.56
354
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_dense_seed72_repaired/run_20260906_repair_from9250/checkpoint-49500
355
+ Step 49510/50462 | Loss: 2.5362 | LR: 4.52e-07 | 3278.0ms/step | 18.3 samples/s
356
+ Step 49520/50462 | Loss: 2.5210 | LR: 4.43e-07 | 1902.6ms/step | 31.5 samples/s
357
+ Step 49530/50462 | Loss: 2.4897 | LR: 4.34e-07 | 1906.3ms/step | 31.5 samples/s
358
+ Step 49540/50462 | Loss: 2.4729 | LR: 4.24e-07 | 1907.3ms/step | 31.5 samples/s
359
+ Step 49550/50462 | Loss: 2.4915 | LR: 4.15e-07 | 1911.4ms/step | 31.4 samples/s
360
+ Step 49560/50462 | Loss: 2.5238 | LR: 4.06e-07 | 1916.1ms/step | 31.3 samples/s
361
+ Step 49570/50462 | Loss: 2.4519 | LR: 3.97e-07 | 1918.4ms/step | 31.3 samples/s
362
+ Step 49580/50462 | Loss: 2.5309 | LR: 3.88e-07 | 1917.4ms/step | 31.3 samples/s
363
+ Step 49590/50462 | Loss: 2.5227 | LR: 3.79e-07 | 1916.6ms/step | 31.3 samples/s
364
+ Step 49600/50462 | Loss: 2.5171 | LR: 3.71e-07 | 1917.8ms/step | 31.3 samples/s
365
+ Step 49610/50462 | Loss: 2.4844 | LR: 3.62e-07 | 1920.4ms/step | 31.2 samples/s
366
+ Step 49620/50462 | Loss: 2.4825 | LR: 3.54e-07 | 1923.0ms/step | 31.2 samples/s
367
+ Step 49630/50462 | Loss: 2.4869 | LR: 3.45e-07 | 1925.0ms/step | 31.2 samples/s
368
+ Step 49640/50462 | Loss: 2.4633 | LR: 3.37e-07 | 1920.1ms/step | 31.2 samples/s
369
+ Step 49650/50462 | Loss: 2.5311 | LR: 3.29e-07 | 1915.8ms/step | 31.3 samples/s
370
+ Step 49660/50462 | Loss: 2.5111 | LR: 3.21e-07 | 1921.2ms/step | 31.2 samples/s
371
+ Step 49670/50462 | Loss: 2.4699 | LR: 3.13e-07 | 1917.1ms/step | 31.3 samples/s
372
+ Step 49680/50462 | Loss: 2.4401 | LR: 3.05e-07 | 1917.9ms/step | 31.3 samples/s
373
+ Step 49690/50462 | Loss: 2.4989 | LR: 2.97e-07 | 1917.7ms/step | 31.3 samples/s
374
+ Step 49700/50462 | Loss: 2.4935 | LR: 2.90e-07 | 1919.1ms/step | 31.3 samples/s
375
+ Step 49710/50462 | Loss: 2.5247 | LR: 2.82e-07 | 1917.7ms/step | 31.3 samples/s
376
+ Step 49720/50462 | Loss: 2.5001 | LR: 2.75e-07 | 1919.4ms/step | 31.3 samples/s
377
+ Step 49730/50462 | Loss: 2.4878 | LR: 2.67e-07 | 1919.2ms/step | 31.3 samples/s
378
+ Step 49740/50462 | Loss: 2.5140 | LR: 2.60e-07 | 1916.7ms/step | 31.3 samples/s
379
+ Step 49750/50462 | Loss: 2.5397 | LR: 2.53e-07 | 1917.9ms/step | 31.3 samples/s
380
+ Validation at step 49750: Loss = 2.5308, Perplexity = 12.56
381
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_dense_seed72_repaired/run_20260906_repair_from9250/checkpoint-49750
382
+ Step 49760/50462 | Loss: 2.5290 | LR: 2.46e-07 | 3269.0ms/step | 18.4 samples/s
383
+ Step 49770/50462 | Loss: 2.5168 | LR: 2.39e-07 | 1903.7ms/step | 31.5 samples/s
384
+ Step 49780/50462 | Loss: 2.4945 | LR: 2.32e-07 | 1905.4ms/step | 31.5 samples/s
385
+ Step 49790/50462 | Loss: 2.5125 | LR: 2.25e-07 | 1908.4ms/step | 31.4 samples/s
386
+ Step 49800/50462 | Loss: 2.4581 | LR: 2.19e-07 | 1913.8ms/step | 31.4 samples/s
387
+ Step 49810/50462 | Loss: 2.4924 | LR: 2.12e-07 | 1916.3ms/step | 31.3 samples/s
388
+ Step 49820/50462 | Loss: 2.4536 | LR: 2.06e-07 | 1915.7ms/step | 31.3 samples/s
389
+ Step 49830/50462 | Loss: 2.4274 | LR: 1.99e-07 | 1916.3ms/step | 31.3 samples/s
390
+ Step 49840/50462 | Loss: 2.4900 | LR: 1.93e-07 | 1917.8ms/step | 31.3 samples/s
391
+ Step 49850/50462 | Loss: 2.5182 | LR: 1.87e-07 | 1918.0ms/step | 31.3 samples/s
392
+ Step 49860/50462 | Loss: 2.4912 | LR: 1.81e-07 | 1920.1ms/step | 31.2 samples/s
393
+ Step 49870/50462 | Loss: 2.4951 | LR: 1.75e-07 | 1920.4ms/step | 31.2 samples/s
394
+ Step 49880/50462 | Loss: 2.4278 | LR: 1.69e-07 | 1920.8ms/step | 31.2 samples/s
395
+ Step 49890/50462 | Loss: 2.4870 | LR: 1.63e-07 | 1918.2ms/step | 31.3 samples/s
396
+ Step 49900/50462 | Loss: 2.4841 | LR: 1.58e-07 | 1917.3ms/step | 31.3 samples/s
397
+ Step 49910/50462 | Loss: 2.4832 | LR: 1.52e-07 | 1918.3ms/step | 31.3 samples/s
398
+ Step 49920/50462 | Loss: 2.4648 | LR: 1.47e-07 | 1919.1ms/step | 31.3 samples/s
399
+ Step 49930/50462 | Loss: 2.4839 | LR: 1.41e-07 | 1915.0ms/step | 31.3 samples/s
400
+ Step 49940/50462 | Loss: 2.5264 | LR: 1.36e-07 | 1919.2ms/step | 31.3 samples/s
401
+ Step 49950/50462 | Loss: 2.5001 | LR: 1.31e-07 | 1916.2ms/step | 31.3 samples/s
402
+ Step 49960/50462 | Loss: 2.4862 | LR: 1.26e-07 | 1918.0ms/step | 31.3 samples/s
403
+ Step 49970/50462 | Loss: 2.4902 | LR: 1.21e-07 | 1919.9ms/step | 31.3 samples/s
404
+ Step 49980/50462 | Loss: 2.5009 | LR: 1.16e-07 | 1919.4ms/step | 31.3 samples/s
405
+ Step 49990/50462 | Loss: 2.5012 | LR: 1.11e-07 | 1919.8ms/step | 31.3 samples/s
406
+ Step 50000/50462 | Loss: 2.5683 | LR: 1.07e-07 | 1919.2ms/step | 31.3 samples/s
407
+ Validation at step 50000: Loss = 2.5308, Perplexity = 12.56
408
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_dense_seed72_repaired/run_20260906_repair_from9250/checkpoint-50000
409
+ Step 50010/50462 | Loss: 2.4902 | LR: 1.02e-07 | 3284.1ms/step | 18.3 samples/s
410
+ Step 50020/50462 | Loss: 2.4732 | LR: 9.75e-08 | 1906.0ms/step | 31.5 samples/s
411
+ Step 50030/50462 | Loss: 2.5221 | LR: 9.32e-08 | 1912.2ms/step | 31.4 samples/s
412
+ Step 50040/50462 | Loss: 2.4701 | LR: 8.89e-08 | 1917.2ms/step | 31.3 samples/s
413
+ Step 50050/50462 | Loss: 2.4753 | LR: 8.47e-08 | 1917.3ms/step | 31.3 samples/s
414
+ Step 50060/50462 | Loss: 2.5255 | LR: 8.07e-08 | 1921.1ms/step | 31.2 samples/s
415
+ Step 50070/50462 | Loss: 2.4822 | LR: 7.67e-08 | 1916.1ms/step | 31.3 samples/s
416
+ Step 50080/50462 | Loss: 2.4901 | LR: 7.28e-08 | 1917.1ms/step | 31.3 samples/s
417
+ Step 50090/50462 | Loss: 2.5373 | LR: 6.91e-08 | 1918.8ms/step | 31.3 samples/s
418
+ Step 50100/50462 | Loss: 2.4365 | LR: 6.54e-08 | 1921.8ms/step | 31.2 samples/s
419
+ Step 50110/50462 | Loss: 2.5263 | LR: 6.19e-08 | 1920.5ms/step | 31.2 samples/s
420
+ Step 50120/50462 | Loss: 2.4978 | LR: 5.84e-08 | 1922.0ms/step | 31.2 samples/s
421
+ Step 50130/50462 | Loss: 2.4779 | LR: 5.50e-08 | 1919.6ms/step | 31.3 samples/s
422
+ Step 50140/50462 | Loss: 2.4874 | LR: 5.18e-08 | 1918.9ms/step | 31.3 samples/s
423
+ Step 50150/50462 | Loss: 2.4717 | LR: 4.86e-08 | 1919.9ms/step | 31.3 samples/s
424
+ Step 50160/50462 | Loss: 2.4594 | LR: 4.55e-08 | 1919.6ms/step | 31.3 samples/s
425
+ Step 50170/50462 | Loss: 2.5121 | LR: 4.26e-08 | 1919.9ms/step | 31.3 samples/s
426
+ Step 50180/50462 | Loss: 2.5068 | LR: 3.97e-08 | 1916.7ms/step | 31.3 samples/s
427
+ Step 50190/50462 | Loss: 2.4909 | LR: 3.69e-08 | 1920.3ms/step | 31.2 samples/s
428
+ Step 50200/50462 | Loss: 2.4659 | LR: 3.43e-08 | 1920.5ms/step | 31.2 samples/s
429
+ Step 50210/50462 | Loss: 2.4972 | LR: 3.17e-08 | 1918.3ms/step | 31.3 samples/s
430
+ Step 50220/50462 | Loss: 2.4950 | LR: 2.92e-08 | 1917.8ms/step | 31.3 samples/s
431
+ Step 50230/50462 | Loss: 2.5092 | LR: 2.69e-08 | 1917.7ms/step | 31.3 samples/s
432
+ Step 50240/50462 | Loss: 2.4798 | LR: 2.46e-08 | 1920.1ms/step | 31.2 samples/s
433
+ Step 50250/50462 | Loss: 2.4475 | LR: 2.24e-08 | 1920.7ms/step | 31.2 samples/s
434
+ Validation at step 50250: Loss = 2.5308, Perplexity = 12.56
435
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_dense_seed72_repaired/run_20260906_repair_from9250/checkpoint-50250
436
+ Step 50260/50462 | Loss: 2.5138 | LR: 2.04e-08 | 3290.8ms/step | 18.2 samples/s
437
+ Step 50270/50462 | Loss: 2.5389 | LR: 1.84e-08 | 1903.2ms/step | 31.5 samples/s
438
+ Step 50280/50462 | Loss: 2.4606 | LR: 1.65e-08 | 1904.3ms/step | 31.5 samples/s
439
+ Step 50290/50462 | Loss: 2.4707 | LR: 1.48e-08 | 1910.3ms/step | 31.4 samples/s
440
+ Step 50300/50462 | Loss: 2.4740 | LR: 1.31e-08 | 1913.1ms/step | 31.4 samples/s
441
+ Step 50310/50462 | Loss: 2.4969 | LR: 1.15e-08 | 1917.7ms/step | 31.3 samples/s
442
+ Step 50320/50462 | Loss: 2.4832 | LR: 1.01e-08 | 1914.5ms/step | 31.3 samples/s
443
+ Step 50330/50462 | Loss: 2.4591 | LR: 8.70e-09 | 1916.7ms/step | 31.3 samples/s
444
+ Step 50340/50462 | Loss: 2.4950 | LR: 7.43e-09 | 1913.6ms/step | 31.4 samples/s
445
+ Step 50350/50462 | Loss: 2.4247 | LR: 6.26e-09 | 1918.0ms/step | 31.3 samples/s
446
+ Step 50360/50462 | Loss: 2.5139 | LR: 5.19e-09 | 1915.5ms/step | 31.3 samples/s
447
+ Step 50370/50462 | Loss: 2.5117 | LR: 4.23e-09 | 1918.0ms/step | 31.3 samples/s
448
+ Step 50380/50462 | Loss: 2.5277 | LR: 3.36e-09 | 1914.0ms/step | 31.3 samples/s
449
+ Step 50390/50462 | Loss: 2.5300 | LR: 2.59e-09 | 1915.4ms/step | 31.3 samples/s
450
+ Step 50400/50462 | Loss: 2.5118 | LR: 1.92e-09 | 1917.8ms/step | 31.3 samples/s
451
+ Step 50410/50462 | Loss: 2.4657 | LR: 1.35e-09 | 1917.2ms/step | 31.3 samples/s
452
+ Step 50420/50462 | Loss: 2.5122 | LR: 8.81e-10 | 1916.2ms/step | 31.3 samples/s
453
+ Step 50430/50462 | Loss: 2.4837 | LR: 5.11e-10 | 1913.5ms/step | 31.4 samples/s
454
+ Step 50440/50462 | Loss: 2.5357 | LR: 2.42e-10 | 1915.0ms/step | 31.3 samples/s
455
+ Step 50450/50462 | Loss: 2.4567 | LR: 7.19e-11 | 1916.0ms/step | 31.3 samples/s
456
+ Step 50460/50462 | Loss: 2.4554 | LR: 2.00e-12 | 1916.9ms/step | 31.3 samples/s
457
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_dense_seed72_repaired/run_20260906_repair_from9250/checkpoint-50462
458
+ Training complete!
epfl_final/train_logs/p2_1p2b_dense_seed72_6gpu_resume48000.pid ADDED
@@ -0,0 +1 @@
 
 
1
+ 108
epfl_final/train_logs/p2_1p2b_mixed_seed70_repaired_from_scratch_6gpu.log ADDED
The diff for this file is too large to render. See raw diff
 
epfl_final/train_logs/p2_1p2b_mixed_seed70_repaired_from_scratch_6gpu.pid ADDED
@@ -0,0 +1 @@
 
 
1
+ 116
epfl_final/train_logs/p2_1p2b_mixed_seed70_repaired_resume12750_6gpu.log ADDED
@@ -0,0 +1,337 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ *****************************************
3
+ Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
4
+ *****************************************
5
+ Resuming: Using existing output directory /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557
6
+ Running config:
7
+ {
8
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
9
+ "model_dtype": "bfloat16",
10
+ "init_from_scratch": true,
11
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
12
+ "val_path": null,
13
+ "max_seq_length": 8192,
14
+ "num_workers": 8,
15
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557",
16
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-12750",
17
+ "force_new_run": false,
18
+ "num_epochs": 1,
19
+ "max_steps": 50462,
20
+ "per_device_batch_size": 1,
21
+ "gradient_accumulation_steps": 10,
22
+ "learning_rate": 0.0005,
23
+ "weight_decay": 0.01,
24
+ "warmup_steps": 750,
25
+ "lr_scheduler": "cosine",
26
+ "max_grad_norm": 1.0,
27
+ "save_steps": 250,
28
+ "eval_steps": 250,
29
+ "eval_batches": 50,
30
+ "logging_steps": 10,
31
+ "seed": 70,
32
+ "verbose": true,
33
+ "checkpoint_activations": true,
34
+ "resume_micro_step_override": null,
35
+ "strategy": "ddp",
36
+ "mixed_precision": "bf16",
37
+ "wandb_enabled": false,
38
+ "wandb_project": "stochastic_sparsity",
39
+ "wandb_name": "1.2b mixed 8k seed70 repaired resume12750",
40
+ "wandb_tags": [
41
+ "qwen3",
42
+ "1.2b",
43
+ "8k",
44
+ "p2",
45
+ "mixed",
46
+ "seed70",
47
+ "repaired"
48
+ ],
49
+ "hidden_size": 1536,
50
+ "intermediate_size": 6144,
51
+ "num_attention_heads": 12,
52
+ "num_key_value_heads": 4,
53
+ "num_hidden_layers": 28,
54
+ "head_dim": null,
55
+ "attention_dropout": null,
56
+ "rope_train_position_scale_set": null,
57
+ "attention_type": "sparse_stochastic",
58
+ "sparse_block_size": 128,
59
+ "sparse_num_random_blocks": 35,
60
+ "sparse_mask_pool_size": 10000,
61
+ "sparse_val_dense_compare": false,
62
+ "sparse_layer_indices": null,
63
+ "sparse_mask_cache_dir": "/mloscratch/homes/navasard/sparse/mask_cache_8k_mixed",
64
+ "sparse_mask_seed": 70,
65
+ "sparse_mixed_nrb_set": [
66
+ 63,
67
+ 47,
68
+ 31,
69
+ 21,
70
+ 11
71
+ ],
72
+ "dense_mode": false,
73
+ "dense_step_intervals": null
74
+ }
75
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
76
+ Checkpoint activations: True
77
+ Checkpoint activations: True
78
+ Checkpoint activations: True
79
+ Checkpoint activations: True
80
+ Checkpoint activations: True
81
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-12750
82
+ Checkpoint activations: True
83
+
84
+
85
+ ================================================================================
86
+ MODEL ARCHITECTURE
87
+ ================================================================================
88
+ Model config:
89
+ vocab_size: 151936
90
+ hidden_size: 1536
91
+ num_hidden_layers: 28
92
+ num_attention_heads: 12
93
+ num_key_value_heads: 4
94
+ head_dim: 128
95
+ intermediate_size: 6144
96
+ max_position_embeddings: 32768
97
+ rope_theta: 1000000
98
+ rope_train_position_scale_set: None
99
+ attention_type: sparse_stochastic
100
+ sparse_block_size: 128
101
+ sparse_num_random_blocks: 35
102
+ sparse_mask_pool_size: 10000
103
+ sparse_max_seq_length: 8192
104
+
105
+ Parameter count:
106
+ Total parameters: 1,202,352,640
107
+ Trainable parameters: 1,202,352,640
108
+ Non-trainable parameters: 0
109
+ Model size (BF16): 2.40 GB
110
+
111
+ --------------------------------------------------------------------------------
112
+ LAYER-BY-LAYER SUMMARY
113
+ --------------------------------------------------------------------------------
114
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/nn/attention/flex_attention.py:1704: UserWarning: flex_attention called without torch.compile() - this will use an unfused implementation that materializes the full scores matrix instead of generating a fused kernel.
115
+
116
+ SOLUTION: Use torch.compile(flex_attention)(...)
117
+
118
+ If you want to debug your score_mod/mask_mod, you can set:
119
+ torch.nn.attention.flex_attention._FLEX_ATTENTION_DISABLE_COMPILE_DEBUG = True
120
+
121
+ This will allow you to use print statements or breakpoints. Note: This doesn't work with the backwards pass and may produce incorrect results.
122
+ _warn_once(
123
+ ================================================================================================================================================================
124
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
125
+ ================================================================================================================================================================
126
+ Qwen3ForCausalLM [1, 8192] -- -- True
127
+ ├─Qwen3Model: 1-1 -- [1, 8192, 1536] -- True
128
+ │ └─Embedding: 2-1 [1, 8192] [1, 8192, 1536] 233,373,696 True
129
+ │ └─ModuleList: 2-2 -- -- -- True
130
+ │ │ └─Qwen3DecoderLayer: 3-1 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
131
+ │ │ └─Qwen3DecoderLayer: 3-2 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
132
+ │ │ └─Qwen3DecoderLayer: 3-3 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
133
+ │ │ └─Qwen3DecoderLayer: 3-4 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
134
+ │ │ └─Qwen3DecoderLayer: 3-5 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
135
+ │ │ └─Qwen3DecoderLayer: 3-6 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
136
+ │ │ └─Qwen3DecoderLayer: 3-7 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
137
+ │ │ └─Qwen3DecoderLayer: 3-8 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
138
+ │ │ └─Qwen3DecoderLayer: 3-9 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
139
+ │ │ └─Qwen3DecoderLayer: 3-10 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
140
+ │ │ └─Qwen3DecoderLayer: 3-11 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
141
+ │ │ └─Qwen3DecoderLayer: 3-12 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
142
+ │ │ └─Qwen3DecoderLayer: 3-13 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
143
+ │ │ └─Qwen3DecoderLayer: 3-14 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
144
+ │ │ └─Qwen3DecoderLayer: 3-15 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
145
+ │ │ └─Qwen3DecoderLayer: 3-16 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
146
+ │ │ └─Qwen3DecoderLayer: 3-17 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
147
+ │ │ └─Qwen3DecoderLayer: 3-18 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
148
+ │ │ └─Qwen3DecoderLayer: 3-19 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
149
+ │ │ └─Qwen3DecoderLayer: 3-20 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
150
+ │ │ └─Qwen3DecoderLayer: 3-21 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
151
+ │ │ └─Qwen3DecoderLayer: 3-22 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
152
+ │ │ └─Qwen3DecoderLayer: 3-23 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
153
+ │ │ └─Qwen3DecoderLayer: 3-24 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
154
+ │ │ └─Qwen3DecoderLayer: 3-25 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
155
+ │ │ └─Qwen3DecoderLayer: 3-26 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
156
+ │ │ └─Qwen3DecoderLayer: 3-27 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
157
+ │ │ └─Qwen3DecoderLayer: 3-28 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
158
+ │ └─Qwen3RMSNorm: 2-3 [1, 8192, 1536] [1, 8192, 1536] 1,536 True
159
+ ├─Linear: 1-2 [1, 8192, 1536] [1, 8192, 151936] 233,373,696 True
160
+ ================================================================================================================================================================
161
+ Total params: 1,435,726,336
162
+ Trainable params: 1,435,726,336
163
+ Non-trainable params: 0
164
+ Total mult-adds (Units.GIGABYTES): 1.44
165
+ ================================================================================================================================================================
166
+ Input size (MB): 0.07
167
+ Forward/backward pass size (MB): 26218.59
168
+ Params size (MB): 2871.45
169
+ Estimated Total Size (MB): 29090.11
170
+ ================================================================================================================================================================
171
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (233,373,696 params). True unique parameter count: 1,202,352,640 (1202.4M)
172
+ ================================================================================
173
+
174
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
175
+ Using pre-tokenized dataset (no runtime tokenization)
176
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
177
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
178
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
179
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
180
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
181
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
182
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
183
+
184
+
185
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
186
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
187
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
188
+ Dataset split: 3027764 train, 159356 validation
189
+ Training examples (packed, train split): 3027764
190
+ Validation examples (packed, val split): 159356
191
+ Per-device batch size: 1
192
+ World size (GPUs): 6
193
+ Gradient accumulation steps: 10
194
+ Effective batch size (seqs/step): 60
195
+ Sequence length: 8192
196
+ Tokens per optimizer step: 491,520
197
+ Micro-batches per epoch (per rank): 504628
198
+ Optimizer steps per epoch: 50462
199
+ Total optimizer steps: 50462 (capped by max_steps; 1.00 epochs)
200
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-12750...
201
+ Preserving checkpoint LR state for exact resume: current=[0.00043149287842086514], schedule_base=[0.0005]
202
+ Resumed from checkpoint at step 12750 (epoch 0, micro_step 127500)
203
+ Resuming training from step 12750
204
+ Starting training...
205
+ Validation at step 12750: Loss = 2.7270, Perplexity = 15.29
206
+ W0909 17:09:22.839000 417 site-packages/torch/distributed/elastic/agent/server/api.py:739] Received 15 death signal, shutting down workers
207
+ [rank0]: Traceback (most recent call last):
208
+ [rank0]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/resource_sharer.py", line 138, in _serve
209
+ [rank0]: with self._listener.accept() as conn:
210
+ [rank0]: ~~~~~~~~~~~~~~~~~~~~~^^
211
+ [rank0]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 496, in accept
212
+ [rank0]: answer_challenge(c, self._authkey)
213
+ [rank0]: ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
214
+ [rank0]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 984, in answer_challenge
215
+ [rank0]: response = connection.recv_bytes(256) # reject large message
216
+ [rank0]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 217, in recv_bytes
217
+ [rank0]: buf = self._recv_bytes(maxlength)
218
+ [rank0]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 431, in _recv_bytes
219
+ [rank0]: buf = self._recv(4)
220
+ [rank0]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 400, in _recv
221
+ [rank0]: raise EOFError
222
+ [rank0]: EOFError
223
+ [rank2]: Traceback (most recent call last):
224
+ [rank2]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/resource_sharer.py", line 138, in _serve
225
+ [rank2]: with self._listener.accept() as conn:
226
+ [rank2]: ~~~~~~~~~~~~~~~~~~~~~^^
227
+ [rank2]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 496, in accept
228
+ [rank2]: answer_challenge(c, self._authkey)
229
+ [rank2]: ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
230
+ [rank2]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 975, in answer_challenge
231
+ [rank2]: message = connection.recv_bytes(256) # reject large message
232
+ [rank2]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 217, in recv_bytes
233
+ [rank2]: buf = self._recv_bytes(maxlength)
234
+ [rank2]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 431, in _recv_bytes
235
+ [rank2]: buf = self._recv(4)
236
+ [rank2]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 396, in _recv
237
+ [rank2]: chunk = read(handle, remaining)
238
+ [rank2]: ConnectionResetError: [Errno 104] Connection reset by peer
239
+ [rank1]: Traceback (most recent call last):
240
+ [rank1]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/resource_sharer.py", line 138, in _serve
241
+ [rank1]: with self._listener.accept() as conn:
242
+ [rank1]: ~~~~~~~~~~~~~~~~~~~~~^^
243
+ [rank1]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 496, in accept
244
+ [rank1]: answer_challenge(c, self._authkey)
245
+ [rank1]: ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
246
+ [rank1]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 975, in answer_challenge
247
+ [rank1]: message = connection.recv_bytes(256) # reject large message
248
+ [rank1]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 217, in recv_bytes
249
+ [rank1]: buf = self._recv_bytes(maxlength)
250
+ [rank1]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 431, in _recv_bytes
251
+ [rank1]: buf = self._recv(4)
252
+ [rank1]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 396, in _recv
253
+ [rank1]: chunk = read(handle, remaining)
254
+ [rank1]: ConnectionResetError: [Errno 104] Connection reset by peer
255
+ [rank4]: Traceback (most recent call last):
256
+ [rank4]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/resource_sharer.py", line 138, in _serve
257
+ [rank4]: with self._listener.accept() as conn:
258
+ [rank4]: ~~~~~~~~~~~~~~~~~~~~~^^
259
+ [rank4]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 496, in accept
260
+ [rank4]: answer_challenge(c, self._authkey)
261
+ [rank4]: ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
262
+ [rank4]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 975, in answer_challenge
263
+ [rank4]: message = connection.recv_bytes(256) # reject large message
264
+ [rank4]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 217, in recv_bytes
265
+ [rank4]: buf = self._recv_bytes(maxlength)
266
+ [rank4]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 431, in _recv_bytes
267
+ [rank4]: buf = self._recv(4)
268
+ [rank4]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 400, in _recv
269
+ [rank4]: raise EOFError
270
+ [rank4]: EOFError
271
+ [rank3]: Traceback (most recent call last):
272
+ [rank3]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/resource_sharer.py", line 138, in _serve
273
+ [rank3]: with self._listener.accept() as conn:
274
+ [rank3]: ~~~~~~~~~~~~~~~~~~~~~^^
275
+ [rank3]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 496, in accept
276
+ [rank3]: answer_challenge(c, self._authkey)
277
+ [rank3]: ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
278
+ [rank3]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 984, in answer_challenge
279
+ [rank3]: response = connection.recv_bytes(256) # reject large message
280
+ [rank3]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 217, in recv_bytes
281
+ [rank3]: buf = self._recv_bytes(maxlength)
282
+ [rank3]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 431, in _recv_bytes
283
+ [rank3]: buf = self._recv(4)
284
+ [rank3]: File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 400, in _recv
285
+ [rank3]: raise EOFError
286
+ [rank3]: EOFError
287
+ Traceback (most recent call last):
288
+ File "<frozen runpy>", line 198, in _run_module_as_main
289
+ File "<frozen runpy>", line 88, in _run_code
290
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/distributed/run.py", line 994, in <module>
291
+ main()
292
+ ~~~~^^
293
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 362, in wrapper
294
+ return f(*args, **kwargs)
295
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/distributed/run.py", line 990, in main
296
+ run(args)
297
+ ~~~^^^^^^
298
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/distributed/run.py", line 981, in run
299
+ elastic_launch(
300
+ ~~~~~~~~~~~~~~~
301
+ config=config,
302
+ ~~~~~~~~~~~~~~
303
+ entrypoint=cmd,
304
+ ~~~~~~~~~~~~~~~
305
+ )(*cmd_args)
306
+ ~^^^^^^^^^^^
307
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/distributed/launcher/api.py", line 170, in __call__
308
+ return launch_agent(self._config, self._entrypoint, list(args))
309
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/distributed/launcher/api.py", line 308, in launch_agent
310
+ result = agent.run()
311
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/distributed/elastic/metrics/api.py", line 134, in wrapper
312
+ result = f(*args, **kwargs)
313
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/distributed/elastic/agent/server/api.py", line 731, in run
314
+ result = self._invoke_run(role)
315
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/distributed/elastic/agent/server/api.py", line 909, in _invoke_run
316
+ run_result = self._monitor_workers(self._worker_group)
317
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/distributed/elastic/metrics/api.py", line 134, in wrapper
318
+ result = f(*args, **kwargs)
319
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/distributed/elastic/agent/server/local_elastic_agent.py", line 436, in _monitor_workers
320
+ result = self._pcontext.wait(0)
321
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 622, in wait
322
+ return self._poll()
323
+ ~~~~~~~~~~^^
324
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 968, in _poll
325
+ self._capture_process_failures(done_local_ranks)
326
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
327
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 954, in _capture_process_failures
328
+ exitcode = handler.proc.poll()
329
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/subprocess.py", line 1252, in poll
330
+ return self._internal_poll()
331
+ ~~~~~~~~~~~~~~~~~~~^^
332
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/subprocess.py", line 2022, in _internal_poll
333
+ pid, sts = _del_safe.waitpid(self.pid, _del_safe.WNOHANG)
334
+ ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
335
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/distributed/elastic/multiprocessing/api.py", line 86, in _terminate_process_handler
336
+ raise SignalException(f"Process {os.getpid()} got signal: {sigval}", sigval=sigval)
337
+ torch.distributed.elastic.multiprocessing.api.SignalException: Process 417 got signal: 15
epfl_final/train_logs/p2_1p2b_mixed_seed70_repaired_resume12750_6gpu.pid ADDED
@@ -0,0 +1 @@
 
 
1
+ 417
epfl_final/train_logs/p2_1p2b_mixed_seed70_repaired_resume14500_6gpu.log ADDED
@@ -0,0 +1,658 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ *****************************************
3
+ Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
4
+ *****************************************
5
+ Resuming: Using existing output directory /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557
6
+ Running config:
7
+ {
8
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
9
+ "model_dtype": "bfloat16",
10
+ "init_from_scratch": true,
11
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
12
+ "val_path": null,
13
+ "max_seq_length": 8192,
14
+ "num_workers": 8,
15
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557",
16
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-14500",
17
+ "force_new_run": false,
18
+ "num_epochs": 1,
19
+ "max_steps": 50462,
20
+ "per_device_batch_size": 1,
21
+ "gradient_accumulation_steps": 10,
22
+ "learning_rate": 0.0005,
23
+ "weight_decay": 0.01,
24
+ "warmup_steps": 750,
25
+ "lr_scheduler": "cosine",
26
+ "max_grad_norm": 1.0,
27
+ "save_steps": 250,
28
+ "eval_steps": 250,
29
+ "eval_batches": 50,
30
+ "logging_steps": 10,
31
+ "seed": 70,
32
+ "verbose": true,
33
+ "checkpoint_activations": true,
34
+ "resume_micro_step_override": null,
35
+ "strategy": "ddp",
36
+ "mixed_precision": "bf16",
37
+ "wandb_enabled": false,
38
+ "wandb_project": "stochastic_sparsity",
39
+ "wandb_name": "1.2b mixed 8k seed70 repaired resume14500",
40
+ "wandb_tags": [
41
+ "qwen3",
42
+ "1.2b",
43
+ "8k",
44
+ "p2",
45
+ "mixed",
46
+ "seed70",
47
+ "repaired"
48
+ ],
49
+ "hidden_size": 1536,
50
+ "intermediate_size": 6144,
51
+ "num_attention_heads": 12,
52
+ "num_key_value_heads": 4,
53
+ "num_hidden_layers": 28,
54
+ "head_dim": null,
55
+ "attention_dropout": null,
56
+ "rope_train_position_scale_set": null,
57
+ "attention_type": "sparse_stochastic",
58
+ "sparse_block_size": 128,
59
+ "sparse_num_random_blocks": 35,
60
+ "sparse_mask_pool_size": 10000,
61
+ "sparse_val_dense_compare": false,
62
+ "sparse_layer_indices": null,
63
+ "sparse_mask_cache_dir": "/mloscratch/homes/navasard/sparse/mask_cache_8k_mixed",
64
+ "sparse_mask_seed": 70,
65
+ "sparse_mixed_nrb_set": [
66
+ 63,
67
+ 47,
68
+ 31,
69
+ 21,
70
+ 11
71
+ ],
72
+ "dense_mode": false,
73
+ "dense_step_intervals": null
74
+ }
75
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
76
+ Checkpoint activations: True
77
+ Checkpoint activations: True
78
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-14500
79
+ Checkpoint activations: True
80
+ Checkpoint activations: True
81
+ Checkpoint activations: True
82
+ Checkpoint activations: True
83
+
84
+
85
+ ================================================================================
86
+ MODEL ARCHITECTURE
87
+ ================================================================================
88
+ Model config:
89
+ vocab_size: 151936
90
+ hidden_size: 1536
91
+ num_hidden_layers: 28
92
+ num_attention_heads: 12
93
+ num_key_value_heads: 4
94
+ head_dim: 128
95
+ intermediate_size: 6144
96
+ max_position_embeddings: 32768
97
+ rope_theta: 1000000
98
+ rope_train_position_scale_set: None
99
+ attention_type: sparse_stochastic
100
+ sparse_block_size: 128
101
+ sparse_num_random_blocks: 35
102
+ sparse_mask_pool_size: 10000
103
+ sparse_max_seq_length: 8192
104
+
105
+ Parameter count:
106
+ Total parameters: 1,202,352,640
107
+ Trainable parameters: 1,202,352,640
108
+ Non-trainable parameters: 0
109
+ Model size (BF16): 2.40 GB
110
+
111
+ --------------------------------------------------------------------------------
112
+ LAYER-BY-LAYER SUMMARY
113
+ --------------------------------------------------------------------------------
114
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/nn/attention/flex_attention.py:1704: UserWarning: flex_attention called without torch.compile() - this will use an unfused implementation that materializes the full scores matrix instead of generating a fused kernel.
115
+
116
+ SOLUTION: Use torch.compile(flex_attention)(...)
117
+
118
+ If you want to debug your score_mod/mask_mod, you can set:
119
+ torch.nn.attention.flex_attention._FLEX_ATTENTION_DISABLE_COMPILE_DEBUG = True
120
+
121
+ This will allow you to use print statements or breakpoints. Note: This doesn't work with the backwards pass and may produce incorrect results.
122
+ _warn_once(
123
+ ================================================================================================================================================================
124
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
125
+ ================================================================================================================================================================
126
+ Qwen3ForCausalLM [1, 8192] -- -- True
127
+ ├─Qwen3Model: 1-1 -- [1, 8192, 1536] -- True
128
+ │ └─Embedding: 2-1 [1, 8192] [1, 8192, 1536] 233,373,696 True
129
+ │ └─ModuleList: 2-2 -- -- -- True
130
+ │ │ └─Qwen3DecoderLayer: 3-1 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
131
+ │ │ └─Qwen3DecoderLayer: 3-2 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
132
+ │ │ └─Qwen3DecoderLayer: 3-3 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
133
+ │ │ └─Qwen3DecoderLayer: 3-4 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
134
+ │ │ └─Qwen3DecoderLayer: 3-5 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
135
+ │ │ └─Qwen3DecoderLayer: 3-6 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
136
+ │ │ └─Qwen3DecoderLayer: 3-7 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
137
+ │ │ └─Qwen3DecoderLayer: 3-8 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
138
+ │ │ └─Qwen3DecoderLayer: 3-9 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
139
+ │ │ └─Qwen3DecoderLayer: 3-10 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
140
+ │ │ └─Qwen3DecoderLayer: 3-11 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
141
+ │ │ └─Qwen3DecoderLayer: 3-12 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
142
+ │ │ └─Qwen3DecoderLayer: 3-13 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
143
+ │ │ └─Qwen3DecoderLayer: 3-14 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
144
+ │ │ └─Qwen3DecoderLayer: 3-15 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
145
+ │ │ └─Qwen3DecoderLayer: 3-16 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
146
+ │ │ └─Qwen3DecoderLayer: 3-17 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
147
+ │ │ └─Qwen3DecoderLayer: 3-18 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
148
+ │ │ └─Qwen3DecoderLayer: 3-19 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
149
+ │ │ └─Qwen3DecoderLayer: 3-20 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
150
+ │ │ └─Qwen3DecoderLayer: 3-21 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
151
+ │ │ └─Qwen3DecoderLayer: 3-22 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
152
+ │ │ └─Qwen3DecoderLayer: 3-23 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
153
+ │ │ └─Qwen3DecoderLayer: 3-24 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
154
+ │ │ └─Qwen3DecoderLayer: 3-25 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
155
+ │ │ └─Qwen3DecoderLayer: 3-26 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
156
+ │ │ └─Qwen3DecoderLayer: 3-27 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
157
+ │ │ └─Qwen3DecoderLayer: 3-28 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
158
+ │ └─Qwen3RMSNorm: 2-3 [1, 8192, 1536] [1, 8192, 1536] 1,536 True
159
+ ├─Linear: 1-2 [1, 8192, 1536] [1, 8192, 151936] 233,373,696 True
160
+ ================================================================================================================================================================
161
+ Total params: 1,435,726,336
162
+ Trainable params: 1,435,726,336
163
+ Non-trainable params: 0
164
+ Total mult-adds (Units.GIGABYTES): 1.44
165
+ ================================================================================================================================================================
166
+ Input size (MB): 0.07
167
+ Forward/backward pass size (MB): 26218.59
168
+ Params size (MB): 2871.45
169
+ Estimated Total Size (MB): 29090.11
170
+ ================================================================================================================================================================
171
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (233,373,696 params). True unique parameter count: 1,202,352,640 (1202.4M)
172
+ ================================================================================
173
+
174
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
175
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
176
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
177
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
178
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
179
+ Using pre-tokenized dataset (no runtime tokenization)
180
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
181
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
182
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
183
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
184
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
185
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
186
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
187
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
188
+ Dataset split: 3027764 train, 159356 validation
189
+ Training examples (packed, train split): 3027764
190
+ Validation examples (packed, val split): 159356
191
+ Per-device batch size: 1
192
+ World size (GPUs): 6
193
+ Gradient accumulation steps: 10
194
+ Effective batch size (seqs/step): 60
195
+ Sequence length: 8192
196
+ Tokens per optimizer step: 491,520
197
+ Micro-batches per epoch (per rank): 504628
198
+ Optimizer steps per epoch: 50462
199
+ Total optimizer steps: 50462 (capped by max_steps; 1.00 epochs)
200
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-14500...
201
+ Preserving checkpoint LR state for exact resume: current=[0.00041140850308630676], schedule_base=[0.0005]
202
+ Resumed from checkpoint at step 14500 (epoch 0, micro_step 145000)
203
+ Resuming training from step 14500
204
+ Starting training...
205
+ Validation at step 14500: Loss = 2.7038, Perplexity = 14.94
206
+ Switching attention mode for step 14501: sparse_stochastic (configured sparse attention)
207
+ Step 14510/50462 | Loss: 2.6802 | LR: 4.11e-04 | 26351.6ms/step | 2.3 samples/s
208
+ Step 14520/50462 | Loss: 2.6932 | LR: 4.11e-04 | 2728.1ms/step | 22.0 samples/s
209
+ Step 14530/50462 | Loss: 2.6772 | LR: 4.11e-04 | 2734.2ms/step | 21.9 samples/s
210
+ Step 14540/50462 | Loss: 2.7197 | LR: 4.11e-04 | 2740.7ms/step | 21.9 samples/s
211
+ Step 14550/50462 | Loss: 2.7075 | LR: 4.11e-04 | 2741.1ms/step | 21.9 samples/s
212
+ Step 14560/50462 | Loss: 2.6983 | LR: 4.11e-04 | 2735.7ms/step | 21.9 samples/s
213
+ Step 14570/50462 | Loss: 2.6948 | LR: 4.11e-04 | 2738.9ms/step | 21.9 samples/s
214
+ Step 14580/50462 | Loss: 2.7184 | LR: 4.10e-04 | 2730.8ms/step | 22.0 samples/s
215
+ Step 14590/50462 | Loss: 2.7486 | LR: 4.10e-04 | 2732.9ms/step | 22.0 samples/s
216
+ Step 14600/50462 | Loss: 2.6688 | LR: 4.10e-04 | 2733.5ms/step | 21.9 samples/s
217
+ Step 14610/50462 | Loss: 2.7061 | LR: 4.10e-04 | 2726.2ms/step | 22.0 samples/s
218
+ Step 14620/50462 | Loss: 2.7024 | LR: 4.10e-04 | 2731.9ms/step | 22.0 samples/s
219
+ Step 14630/50462 | Loss: 2.6779 | LR: 4.10e-04 | 2733.1ms/step | 22.0 samples/s
220
+ Step 14640/50462 | Loss: 2.7269 | LR: 4.10e-04 | 2738.7ms/step | 21.9 samples/s
221
+ Step 14650/50462 | Loss: 2.6811 | LR: 4.10e-04 | 2731.6ms/step | 22.0 samples/s
222
+ Step 14660/50462 | Loss: 2.6838 | LR: 4.09e-04 | 2735.3ms/step | 21.9 samples/s
223
+ Step 14670/50462 | Loss: 2.6862 | LR: 4.09e-04 | 2737.7ms/step | 21.9 samples/s
224
+ Step 14680/50462 | Loss: 2.6830 | LR: 4.09e-04 | 2718.8ms/step | 22.1 samples/s
225
+ Step 14690/50462 | Loss: 2.7249 | LR: 4.09e-04 | 2730.0ms/step | 22.0 samples/s
226
+ Step 14700/50462 | Loss: 2.7237 | LR: 4.09e-04 | 2728.3ms/step | 22.0 samples/s
227
+ Step 14710/50462 | Loss: 2.7157 | LR: 4.09e-04 | 2739.1ms/step | 21.9 samples/s
228
+ Step 14720/50462 | Loss: 2.6909 | LR: 4.09e-04 | 2727.3ms/step | 22.0 samples/s
229
+ Step 14730/50462 | Loss: 2.7083 | LR: 4.09e-04 | 2729.9ms/step | 22.0 samples/s
230
+ Step 14740/50462 | Loss: 2.7379 | LR: 4.08e-04 | 2740.1ms/step | 21.9 samples/s
231
+ Step 14750/50462 | Loss: 2.6895 | LR: 4.08e-04 | 2728.2ms/step | 22.0 samples/s
232
+ Validation at step 14750: Loss = 2.7010, Perplexity = 14.89
233
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-14750
234
+ Step 14760/50462 | Loss: 2.7208 | LR: 4.08e-04 | 4035.4ms/step | 14.9 samples/s
235
+ Step 14770/50462 | Loss: 2.6992 | LR: 4.08e-04 | 2737.7ms/step | 21.9 samples/s
236
+ Step 14780/50462 | Loss: 2.7098 | LR: 4.08e-04 | 2732.5ms/step | 22.0 samples/s
237
+ Step 14790/50462 | Loss: 2.7173 | LR: 4.08e-04 | 2731.1ms/step | 22.0 samples/s
238
+ Step 14800/50462 | Loss: 2.7230 | LR: 4.08e-04 | 2734.7ms/step | 21.9 samples/s
239
+ Step 14810/50462 | Loss: 2.6599 | LR: 4.08e-04 | 2727.7ms/step | 22.0 samples/s
240
+ Step 14820/50462 | Loss: 2.7189 | LR: 4.08e-04 | 2733.1ms/step | 22.0 samples/s
241
+ Step 14830/50462 | Loss: 2.6919 | LR: 4.07e-04 | 2733.5ms/step | 21.9 samples/s
242
+ Step 14840/50462 | Loss: 2.6874 | LR: 4.07e-04 | 2731.5ms/step | 22.0 samples/s
243
+ Step 14850/50462 | Loss: 2.6618 | LR: 4.07e-04 | 2725.9ms/step | 22.0 samples/s
244
+ Step 14860/50462 | Loss: 2.6890 | LR: 4.07e-04 | 2737.6ms/step | 21.9 samples/s
245
+ Step 14870/50462 | Loss: 2.7150 | LR: 4.07e-04 | 2731.6ms/step | 22.0 samples/s
246
+ Step 14880/50462 | Loss: 2.6636 | LR: 4.07e-04 | 2726.0ms/step | 22.0 samples/s
247
+ Step 14890/50462 | Loss: 2.6672 | LR: 4.07e-04 | 2729.2ms/step | 22.0 samples/s
248
+ Step 14900/50462 | Loss: 2.7314 | LR: 4.07e-04 | 2734.2ms/step | 21.9 samples/s
249
+ Step 14910/50462 | Loss: 2.7022 | LR: 4.06e-04 | 2737.5ms/step | 21.9 samples/s
250
+ Step 14920/50462 | Loss: 2.6917 | LR: 4.06e-04 | 2736.6ms/step | 21.9 samples/s
251
+ Step 14930/50462 | Loss: 2.6793 | LR: 4.06e-04 | 2733.9ms/step | 21.9 samples/s
252
+ Step 14940/50462 | Loss: 2.6542 | LR: 4.06e-04 | 2736.0ms/step | 21.9 samples/s
253
+ Step 14950/50462 | Loss: 2.6956 | LR: 4.06e-04 | 2719.5ms/step | 22.1 samples/s
254
+ Step 14960/50462 | Loss: 2.7381 | LR: 4.06e-04 | 2736.2ms/step | 21.9 samples/s
255
+ Step 14970/50462 | Loss: 2.6663 | LR: 4.06e-04 | 2729.9ms/step | 22.0 samples/s
256
+ Step 14980/50462 | Loss: 2.7270 | LR: 4.06e-04 | 2726.5ms/step | 22.0 samples/s
257
+ Step 14990/50462 | Loss: 2.7040 | LR: 4.05e-04 | 2728.0ms/step | 22.0 samples/s
258
+ Step 15000/50462 | Loss: 2.6480 | LR: 4.05e-04 | 2728.2ms/step | 22.0 samples/s
259
+ Validation at step 15000: Loss = 2.6963, Perplexity = 14.83
260
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-15000
261
+ Step 15010/50462 | Loss: 2.7081 | LR: 4.05e-04 | 4031.3ms/step | 14.9 samples/s
262
+ Step 15020/50462 | Loss: 2.7032 | LR: 4.05e-04 | 2730.9ms/step | 22.0 samples/s
263
+ Step 15030/50462 | Loss: 2.6874 | LR: 4.05e-04 | 2743.9ms/step | 21.9 samples/s
264
+ Step 15040/50462 | Loss: 2.6827 | LR: 4.05e-04 | 2733.4ms/step | 22.0 samples/s
265
+ Step 15050/50462 | Loss: 2.7415 | LR: 4.05e-04 | 2730.6ms/step | 22.0 samples/s
266
+ Step 15060/50462 | Loss: 2.6908 | LR: 4.05e-04 | 2737.4ms/step | 21.9 samples/s
267
+ Step 15070/50462 | Loss: 2.6913 | LR: 4.04e-04 | 2734.5ms/step | 21.9 samples/s
268
+ Step 15080/50462 | Loss: 2.7049 | LR: 4.04e-04 | 2738.7ms/step | 21.9 samples/s
269
+ Step 15090/50462 | Loss: 2.6858 | LR: 4.04e-04 | 2741.7ms/step | 21.9 samples/s
270
+ Step 15100/50462 | Loss: 2.6873 | LR: 4.04e-04 | 2738.2ms/step | 21.9 samples/s
271
+ Step 15110/50462 | Loss: 2.6862 | LR: 4.04e-04 | 2743.6ms/step | 21.9 samples/s
272
+ Step 15120/50462 | Loss: 2.6991 | LR: 4.04e-04 | 2733.7ms/step | 21.9 samples/s
273
+ Step 15130/50462 | Loss: 2.6940 | LR: 4.04e-04 | 2732.3ms/step | 22.0 samples/s
274
+ Step 15140/50462 | Loss: 2.7233 | LR: 4.04e-04 | 2735.6ms/step | 21.9 samples/s
275
+ Step 15150/50462 | Loss: 2.7224 | LR: 4.03e-04 | 2733.8ms/step | 21.9 samples/s
276
+ Step 15160/50462 | Loss: 2.7306 | LR: 4.03e-04 | 2739.8ms/step | 21.9 samples/s
277
+ Step 15170/50462 | Loss: 2.6782 | LR: 4.03e-04 | 2732.5ms/step | 22.0 samples/s
278
+ Step 15180/50462 | Loss: 2.6889 | LR: 4.03e-04 | 2732.3ms/step | 22.0 samples/s
279
+ Step 15190/50462 | Loss: 2.7079 | LR: 4.03e-04 | 2733.8ms/step | 21.9 samples/s
280
+ Step 15200/50462 | Loss: 2.6896 | LR: 4.03e-04 | 2736.5ms/step | 21.9 samples/s
281
+ Step 15210/50462 | Loss: 2.6970 | LR: 4.03e-04 | 2734.5ms/step | 21.9 samples/s
282
+ Step 15220/50462 | Loss: 2.7176 | LR: 4.03e-04 | 2734.6ms/step | 21.9 samples/s
283
+ Step 15230/50462 | Loss: 2.7171 | LR: 4.02e-04 | 2738.7ms/step | 21.9 samples/s
284
+ Step 15240/50462 | Loss: 2.6954 | LR: 4.02e-04 | 2739.6ms/step | 21.9 samples/s
285
+ Step 15250/50462 | Loss: 2.6787 | LR: 4.02e-04 | 2739.2ms/step | 21.9 samples/s
286
+ Validation at step 15250: Loss = 2.6947, Perplexity = 14.80
287
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-15250
288
+ Step 15260/50462 | Loss: 2.6734 | LR: 4.02e-04 | 4027.6ms/step | 14.9 samples/s
289
+ Step 15270/50462 | Loss: 2.6553 | LR: 4.02e-04 | 2732.1ms/step | 22.0 samples/s
290
+ Step 15280/50462 | Loss: 2.6945 | LR: 4.02e-04 | 2734.4ms/step | 21.9 samples/s
291
+ Step 15290/50462 | Loss: 2.7318 | LR: 4.02e-04 | 2734.8ms/step | 21.9 samples/s
292
+ Step 15300/50462 | Loss: 2.6513 | LR: 4.02e-04 | 2735.6ms/step | 21.9 samples/s
293
+ Step 15310/50462 | Loss: 2.6937 | LR: 4.01e-04 | 2740.1ms/step | 21.9 samples/s
294
+ Step 15320/50462 | Loss: 2.6378 | LR: 4.01e-04 | 2738.6ms/step | 21.9 samples/s
295
+ Step 15330/50462 | Loss: 2.6854 | LR: 4.01e-04 | 2738.6ms/step | 21.9 samples/s
296
+ Step 15340/50462 | Loss: 2.6956 | LR: 4.01e-04 | 2727.5ms/step | 22.0 samples/s
297
+ Step 15350/50462 | Loss: 2.7178 | LR: 4.01e-04 | 2735.0ms/step | 21.9 samples/s
298
+ Step 15360/50462 | Loss: 2.6498 | LR: 4.01e-04 | 2738.3ms/step | 21.9 samples/s
299
+ Step 15370/50462 | Loss: 2.7116 | LR: 4.01e-04 | 2739.3ms/step | 21.9 samples/s
300
+ Step 15380/50462 | Loss: 2.6889 | LR: 4.01e-04 | 2731.2ms/step | 22.0 samples/s
301
+ Step 15390/50462 | Loss: 2.6690 | LR: 4.00e-04 | 2730.4ms/step | 22.0 samples/s
302
+ Step 15400/50462 | Loss: 2.6968 | LR: 4.00e-04 | 2740.6ms/step | 21.9 samples/s
303
+ Step 15410/50462 | Loss: 2.7075 | LR: 4.00e-04 | 2728.9ms/step | 22.0 samples/s
304
+ Step 15420/50462 | Loss: 2.6676 | LR: 4.00e-04 | 2728.6ms/step | 22.0 samples/s
305
+ Step 15430/50462 | Loss: 2.6998 | LR: 4.00e-04 | 2733.9ms/step | 21.9 samples/s
306
+ Step 15440/50462 | Loss: 2.7171 | LR: 4.00e-04 | 2730.9ms/step | 22.0 samples/s
307
+ Step 15450/50462 | Loss: 2.6667 | LR: 4.00e-04 | 2730.8ms/step | 22.0 samples/s
308
+ Step 15460/50462 | Loss: 2.6591 | LR: 4.00e-04 | 2735.4ms/step | 21.9 samples/s
309
+ Step 15470/50462 | Loss: 2.6719 | LR: 3.99e-04 | 2733.4ms/step | 22.0 samples/s
310
+ Step 15480/50462 | Loss: 2.7023 | LR: 3.99e-04 | 2732.2ms/step | 22.0 samples/s
311
+ Step 15490/50462 | Loss: 2.6598 | LR: 3.99e-04 | 2742.3ms/step | 21.9 samples/s
312
+ Step 15500/50462 | Loss: 2.6858 | LR: 3.99e-04 | 2736.0ms/step | 21.9 samples/s
313
+ Validation at step 15500: Loss = 2.6908, Perplexity = 14.74
314
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-15500
315
+ Step 15510/50462 | Loss: 2.6808 | LR: 3.99e-04 | 4037.6ms/step | 14.9 samples/s
316
+ Step 15520/50462 | Loss: 2.6742 | LR: 3.99e-04 | 2733.2ms/step | 22.0 samples/s
317
+ Step 15530/50462 | Loss: 2.6703 | LR: 3.99e-04 | 2728.8ms/step | 22.0 samples/s
318
+ Step 15540/50462 | Loss: 2.6513 | LR: 3.99e-04 | 2744.4ms/step | 21.9 samples/s
319
+ Step 15550/50462 | Loss: 2.6759 | LR: 3.98e-04 | 2731.5ms/step | 22.0 samples/s
320
+ Step 15560/50462 | Loss: 2.6704 | LR: 3.98e-04 | 2727.4ms/step | 22.0 samples/s
321
+ Step 15570/50462 | Loss: 2.6470 | LR: 3.98e-04 | 2729.6ms/step | 22.0 samples/s
322
+ Step 15580/50462 | Loss: 2.6905 | LR: 3.98e-04 | 2738.2ms/step | 21.9 samples/s
323
+ Step 15590/50462 | Loss: 2.6995 | LR: 3.98e-04 | 2735.5ms/step | 21.9 samples/s
324
+ Step 15600/50462 | Loss: 2.6860 | LR: 3.98e-04 | 2732.1ms/step | 22.0 samples/s
325
+ Step 15610/50462 | Loss: 2.6571 | LR: 3.98e-04 | 2734.6ms/step | 21.9 samples/s
326
+ Step 15620/50462 | Loss: 2.6895 | LR: 3.98e-04 | 2725.8ms/step | 22.0 samples/s
327
+ Step 15630/50462 | Loss: 2.6696 | LR: 3.97e-04 | 2732.6ms/step | 22.0 samples/s
328
+ Step 15640/50462 | Loss: 2.7605 | LR: 3.97e-04 | 2732.2ms/step | 22.0 samples/s
329
+ Step 15650/50462 | Loss: 2.6782 | LR: 3.97e-04 | 2730.5ms/step | 22.0 samples/s
330
+ Step 15660/50462 | Loss: 2.7057 | LR: 3.97e-04 | 2731.9ms/step | 22.0 samples/s
331
+ Step 15670/50462 | Loss: 2.7086 | LR: 3.97e-04 | 2732.0ms/step | 22.0 samples/s
332
+ Step 15680/50462 | Loss: 2.6767 | LR: 3.97e-04 | 2728.3ms/step | 22.0 samples/s
333
+ Step 15690/50462 | Loss: 2.7101 | LR: 3.97e-04 | 2729.9ms/step | 22.0 samples/s
334
+ Step 15700/50462 | Loss: 2.6781 | LR: 3.96e-04 | 2730.6ms/step | 22.0 samples/s
335
+ Step 15710/50462 | Loss: 2.6433 | LR: 3.96e-04 | 2733.3ms/step | 22.0 samples/s
336
+ Step 15720/50462 | Loss: 2.6659 | LR: 3.96e-04 | 2727.8ms/step | 22.0 samples/s
337
+ Step 15730/50462 | Loss: 2.7169 | LR: 3.96e-04 | 2728.9ms/step | 22.0 samples/s
338
+ Step 15740/50462 | Loss: 2.7310 | LR: 3.96e-04 | 2728.0ms/step | 22.0 samples/s
339
+ Step 15750/50462 | Loss: 2.6685 | LR: 3.96e-04 | 2734.7ms/step | 21.9 samples/s
340
+ Validation at step 15750: Loss = 2.6887, Perplexity = 14.71
341
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-15750
342
+ Step 15760/50462 | Loss: 2.6983 | LR: 3.96e-04 | 4035.5ms/step | 14.9 samples/s
343
+ Step 15770/50462 | Loss: 2.7154 | LR: 3.96e-04 | 2728.8ms/step | 22.0 samples/s
344
+ Step 15780/50462 | Loss: 2.6643 | LR: 3.95e-04 | 2728.0ms/step | 22.0 samples/s
345
+ Step 15790/50462 | Loss: 2.6974 | LR: 3.95e-04 | 2731.6ms/step | 22.0 samples/s
346
+ Step 15800/50462 | Loss: 2.7577 | LR: 3.95e-04 | 2730.4ms/step | 22.0 samples/s
347
+ Step 15810/50462 | Loss: 2.6649 | LR: 3.95e-04 | 2735.1ms/step | 21.9 samples/s
348
+ Step 15820/50462 | Loss: 2.6920 | LR: 3.95e-04 | 2726.1ms/step | 22.0 samples/s
349
+ Step 15830/50462 | Loss: 2.6900 | LR: 3.95e-04 | 2727.4ms/step | 22.0 samples/s
350
+ Step 15840/50462 | Loss: 2.6683 | LR: 3.95e-04 | 2729.3ms/step | 22.0 samples/s
351
+ Step 15850/50462 | Loss: 2.7121 | LR: 3.95e-04 | 2735.5ms/step | 21.9 samples/s
352
+ Step 15860/50462 | Loss: 2.6791 | LR: 3.94e-04 | 2728.4ms/step | 22.0 samples/s
353
+ Step 15870/50462 | Loss: 2.6754 | LR: 3.94e-04 | 2722.8ms/step | 22.0 samples/s
354
+ Step 15880/50462 | Loss: 2.6608 | LR: 3.94e-04 | 2733.0ms/step | 22.0 samples/s
355
+ Step 15890/50462 | Loss: 2.6629 | LR: 3.94e-04 | 2737.6ms/step | 21.9 samples/s
356
+ Step 15900/50462 | Loss: 2.6572 | LR: 3.94e-04 | 2727.3ms/step | 22.0 samples/s
357
+ Step 15910/50462 | Loss: 2.7081 | LR: 3.94e-04 | 2729.6ms/step | 22.0 samples/s
358
+ Step 15920/50462 | Loss: 2.6879 | LR: 3.94e-04 | 2727.5ms/step | 22.0 samples/s
359
+ Step 15930/50462 | Loss: 2.7426 | LR: 3.94e-04 | 2720.2ms/step | 22.1 samples/s
360
+ Step 15940/50462 | Loss: 2.6790 | LR: 3.93e-04 | 2724.8ms/step | 22.0 samples/s
361
+ Step 15950/50462 | Loss: 2.7150 | LR: 3.93e-04 | 2734.6ms/step | 21.9 samples/s
362
+ Step 15960/50462 | Loss: 2.6615 | LR: 3.93e-04 | 2730.6ms/step | 22.0 samples/s
363
+ Step 15970/50462 | Loss: 2.6544 | LR: 3.93e-04 | 2727.5ms/step | 22.0 samples/s
364
+ Step 15980/50462 | Loss: 2.6717 | LR: 3.93e-04 | 2725.8ms/step | 22.0 samples/s
365
+ Step 15990/50462 | Loss: 2.6961 | LR: 3.93e-04 | 2732.8ms/step | 22.0 samples/s
366
+ Step 16000/50462 | Loss: 2.6844 | LR: 3.93e-04 | 2737.9ms/step | 21.9 samples/s
367
+ Validation at step 16000: Loss = 2.6820, Perplexity = 14.61
368
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-16000
369
+ Step 16010/50462 | Loss: 2.6616 | LR: 3.92e-04 | 4026.2ms/step | 14.9 samples/s
370
+ Step 16020/50462 | Loss: 2.6630 | LR: 3.92e-04 | 2734.1ms/step | 21.9 samples/s
371
+ Step 16030/50462 | Loss: 2.6711 | LR: 3.92e-04 | 2727.0ms/step | 22.0 samples/s
372
+ Step 16040/50462 | Loss: 2.6906 | LR: 3.92e-04 | 2736.8ms/step | 21.9 samples/s
373
+ Step 16050/50462 | Loss: 2.6759 | LR: 3.92e-04 | 2731.2ms/step | 22.0 samples/s
374
+ Step 16060/50462 | Loss: 2.6862 | LR: 3.92e-04 | 2735.8ms/step | 21.9 samples/s
375
+ Step 16070/50462 | Loss: 2.6932 | LR: 3.92e-04 | 2728.2ms/step | 22.0 samples/s
376
+ Step 16080/50462 | Loss: 2.6654 | LR: 3.92e-04 | 2725.8ms/step | 22.0 samples/s
377
+ Step 16090/50462 | Loss: 2.6713 | LR: 3.91e-04 | 2733.9ms/step | 21.9 samples/s
378
+ Step 16100/50462 | Loss: 2.7215 | LR: 3.91e-04 | 2722.4ms/step | 22.0 samples/s
379
+ Step 16110/50462 | Loss: 2.6996 | LR: 3.91e-04 | 2723.4ms/step | 22.0 samples/s
380
+ Step 16120/50462 | Loss: 2.6920 | LR: 3.91e-04 | 2726.7ms/step | 22.0 samples/s
381
+ Step 16130/50462 | Loss: 2.6931 | LR: 3.91e-04 | 2726.7ms/step | 22.0 samples/s
382
+ Step 16140/50462 | Loss: 2.7207 | LR: 3.91e-04 | 2731.8ms/step | 22.0 samples/s
383
+ Step 16150/50462 | Loss: 2.6438 | LR: 3.91e-04 | 2731.7ms/step | 22.0 samples/s
384
+ Step 16160/50462 | Loss: 2.6995 | LR: 3.91e-04 | 2734.7ms/step | 21.9 samples/s
385
+ Step 16170/50462 | Loss: 2.7005 | LR: 3.90e-04 | 2731.2ms/step | 22.0 samples/s
386
+ Step 16180/50462 | Loss: 2.6840 | LR: 3.90e-04 | 2731.0ms/step | 22.0 samples/s
387
+ Step 16190/50462 | Loss: 2.6843 | LR: 3.90e-04 | 2739.0ms/step | 21.9 samples/s
388
+ Step 16200/50462 | Loss: 2.7626 | LR: 3.90e-04 | 2730.3ms/step | 22.0 samples/s
389
+ Step 16210/50462 | Loss: 2.6700 | LR: 3.90e-04 | 2730.5ms/step | 22.0 samples/s
390
+ Step 16220/50462 | Loss: 2.6526 | LR: 3.90e-04 | 2729.5ms/step | 22.0 samples/s
391
+ Step 16230/50462 | Loss: 2.6741 | LR: 3.90e-04 | 2729.1ms/step | 22.0 samples/s
392
+ Step 16240/50462 | Loss: 2.6928 | LR: 3.89e-04 | 2734.2ms/step | 21.9 samples/s
393
+ Step 16250/50462 | Loss: 2.6829 | LR: 3.89e-04 | 2728.7ms/step | 22.0 samples/s
394
+ Validation at step 16250: Loss = 2.6787, Perplexity = 14.57
395
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-16250
396
+ Step 16260/50462 | Loss: 2.6987 | LR: 3.89e-04 | 4025.6ms/step | 14.9 samples/s
397
+ Step 16270/50462 | Loss: 2.6842 | LR: 3.89e-04 | 2728.5ms/step | 22.0 samples/s
398
+ Step 16280/50462 | Loss: 2.6518 | LR: 3.89e-04 | 2727.2ms/step | 22.0 samples/s
399
+ Step 16290/50462 | Loss: 2.6509 | LR: 3.89e-04 | 2726.9ms/step | 22.0 samples/s
400
+ Step 16300/50462 | Loss: 2.7453 | LR: 3.89e-04 | 2729.2ms/step | 22.0 samples/s
401
+ Step 16310/50462 | Loss: 2.6743 | LR: 3.89e-04 | 2731.6ms/step | 22.0 samples/s
402
+ Step 16320/50462 | Loss: 2.6633 | LR: 3.88e-04 | 2731.9ms/step | 22.0 samples/s
403
+ Step 16330/50462 | Loss: 2.6575 | LR: 3.88e-04 | 2724.6ms/step | 22.0 samples/s
404
+ Step 16340/50462 | Loss: 2.6796 | LR: 3.88e-04 | 2734.1ms/step | 21.9 samples/s
405
+ Step 16350/50462 | Loss: 2.6985 | LR: 3.88e-04 | 2727.5ms/step | 22.0 samples/s
406
+ Step 16360/50462 | Loss: 2.6661 | LR: 3.88e-04 | 2728.7ms/step | 22.0 samples/s
407
+ Step 16370/50462 | Loss: 2.6513 | LR: 3.88e-04 | 2723.2ms/step | 22.0 samples/s
408
+ Step 16380/50462 | Loss: 2.7128 | LR: 3.88e-04 | 2736.1ms/step | 21.9 samples/s
409
+ Step 16390/50462 | Loss: 2.6768 | LR: 3.88e-04 | 2735.1ms/step | 21.9 samples/s
410
+ Step 16400/50462 | Loss: 2.6710 | LR: 3.87e-04 | 2735.1ms/step | 21.9 samples/s
411
+ Step 16410/50462 | Loss: 2.6876 | LR: 3.87e-04 | 2723.7ms/step | 22.0 samples/s
412
+ Step 16420/50462 | Loss: 2.6744 | LR: 3.87e-04 | 2737.3ms/step | 21.9 samples/s
413
+ Step 16430/50462 | Loss: 2.7146 | LR: 3.87e-04 | 2727.8ms/step | 22.0 samples/s
414
+ Step 16440/50462 | Loss: 2.6900 | LR: 3.87e-04 | 2727.5ms/step | 22.0 samples/s
415
+ Step 16450/50462 | Loss: 2.6396 | LR: 3.87e-04 | 2731.3ms/step | 22.0 samples/s
416
+ Step 16460/50462 | Loss: 2.6848 | LR: 3.87e-04 | 2723.3ms/step | 22.0 samples/s
417
+ Step 16470/50462 | Loss: 2.6838 | LR: 3.86e-04 | 2729.0ms/step | 22.0 samples/s
418
+ Step 16480/50462 | Loss: 2.6758 | LR: 3.86e-04 | 2726.8ms/step | 22.0 samples/s
419
+ Step 16490/50462 | Loss: 2.7139 | LR: 3.86e-04 | 2730.4ms/step | 22.0 samples/s
420
+ Step 16500/50462 | Loss: 2.6857 | LR: 3.86e-04 | 2742.6ms/step | 21.9 samples/s
421
+ Validation at step 16500: Loss = 2.6772, Perplexity = 14.54
422
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-16500
423
+ Step 16510/50462 | Loss: 2.6706 | LR: 3.86e-04 | 4014.5ms/step | 14.9 samples/s
424
+ Step 16520/50462 | Loss: 2.6292 | LR: 3.86e-04 | 2723.9ms/step | 22.0 samples/s
425
+ Step 16530/50462 | Loss: 2.6667 | LR: 3.86e-04 | 2737.7ms/step | 21.9 samples/s
426
+ Step 16540/50462 | Loss: 2.6934 | LR: 3.86e-04 | 2728.9ms/step | 22.0 samples/s
427
+ Step 16550/50462 | Loss: 2.6319 | LR: 3.85e-04 | 2727.0ms/step | 22.0 samples/s
428
+ Step 16560/50462 | Loss: 2.6274 | LR: 3.85e-04 | 2740.5ms/step | 21.9 samples/s
429
+ Step 16570/50462 | Loss: 2.6816 | LR: 3.85e-04 | 2735.4ms/step | 21.9 samples/s
430
+ Step 16580/50462 | Loss: 2.6712 | LR: 3.85e-04 | 2725.1ms/step | 22.0 samples/s
431
+ Step 16590/50462 | Loss: 2.7125 | LR: 3.85e-04 | 2732.1ms/step | 22.0 samples/s
432
+ Step 16600/50462 | Loss: 2.7226 | LR: 3.85e-04 | 2730.2ms/step | 22.0 samples/s
433
+ Step 16610/50462 | Loss: 2.6814 | LR: 3.85e-04 | 2727.2ms/step | 22.0 samples/s
434
+ Step 16620/50462 | Loss: 2.6750 | LR: 3.84e-04 | 2734.1ms/step | 21.9 samples/s
435
+ Step 16630/50462 | Loss: 2.7023 | LR: 3.84e-04 | 2726.1ms/step | 22.0 samples/s
436
+ Step 16640/50462 | Loss: 2.7165 | LR: 3.84e-04 | 2728.8ms/step | 22.0 samples/s
437
+ Step 16650/50462 | Loss: 2.6795 | LR: 3.84e-04 | 2735.3ms/step | 21.9 samples/s
438
+ Step 16660/50462 | Loss: 2.6877 | LR: 3.84e-04 | 2733.3ms/step | 22.0 samples/s
439
+ Step 16670/50462 | Loss: 2.7014 | LR: 3.84e-04 | 2720.0ms/step | 22.1 samples/s
440
+ Step 16680/50462 | Loss: 2.7186 | LR: 3.84e-04 | 2727.0ms/step | 22.0 samples/s
441
+ Step 16690/50462 | Loss: 2.6555 | LR: 3.84e-04 | 2735.0ms/step | 21.9 samples/s
442
+ Step 16700/50462 | Loss: 2.6728 | LR: 3.83e-04 | 2739.1ms/step | 21.9 samples/s
443
+ Step 16710/50462 | Loss: 2.6478 | LR: 3.83e-04 | 2732.2ms/step | 22.0 samples/s
444
+ Step 16720/50462 | Loss: 2.6694 | LR: 3.83e-04 | 2729.6ms/step | 22.0 samples/s
445
+ Step 16730/50462 | Loss: 2.6525 | LR: 3.83e-04 | 2728.8ms/step | 22.0 samples/s
446
+ Step 16740/50462 | Loss: 2.6656 | LR: 3.83e-04 | 2729.9ms/step | 22.0 samples/s
447
+ Step 16750/50462 | Loss: 2.6401 | LR: 3.83e-04 | 2731.4ms/step | 22.0 samples/s
448
+ Validation at step 16750: Loss = 2.6760, Perplexity = 14.53
449
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-16750
450
+ Step 16760/50462 | Loss: 2.7250 | LR: 3.83e-04 | 4036.8ms/step | 14.9 samples/s
451
+ Step 16770/50462 | Loss: 2.6948 | LR: 3.82e-04 | 2735.5ms/step | 21.9 samples/s
452
+ Step 16780/50462 | Loss: 2.7052 | LR: 3.82e-04 | 2731.9ms/step | 22.0 samples/s
453
+ Step 16790/50462 | Loss: 2.7094 | LR: 3.82e-04 | 2731.1ms/step | 22.0 samples/s
454
+ Step 16800/50462 | Loss: 2.6801 | LR: 3.82e-04 | 2733.2ms/step | 22.0 samples/s
455
+ Step 16810/50462 | Loss: 2.6644 | LR: 3.82e-04 | 2731.2ms/step | 22.0 samples/s
456
+ Step 16820/50462 | Loss: 2.6614 | LR: 3.82e-04 | 2738.5ms/step | 21.9 samples/s
457
+ Step 16830/50462 | Loss: 2.6901 | LR: 3.82e-04 | 2724.5ms/step | 22.0 samples/s
458
+ Step 16840/50462 | Loss: 2.6887 | LR: 3.82e-04 | 2731.6ms/step | 22.0 samples/s
459
+ Step 16850/50462 | Loss: 2.6475 | LR: 3.81e-04 | 2732.6ms/step | 22.0 samples/s
460
+ Step 16860/50462 | Loss: 2.6849 | LR: 3.81e-04 | 2727.2ms/step | 22.0 samples/s
461
+ Step 16870/50462 | Loss: 2.6724 | LR: 3.81e-04 | 2728.5ms/step | 22.0 samples/s
462
+ Step 16880/50462 | Loss: 2.6609 | LR: 3.81e-04 | 2730.6ms/step | 22.0 samples/s
463
+ Step 16890/50462 | Loss: 2.6615 | LR: 3.81e-04 | 2730.1ms/step | 22.0 samples/s
464
+ Step 16900/50462 | Loss: 2.7112 | LR: 3.81e-04 | 2727.0ms/step | 22.0 samples/s
465
+ Step 16910/50462 | Loss: 2.6575 | LR: 3.81e-04 | 2732.6ms/step | 22.0 samples/s
466
+ Step 16920/50462 | Loss: 2.6459 | LR: 3.80e-04 | 2731.9ms/step | 22.0 samples/s
467
+ Step 16930/50462 | Loss: 2.6466 | LR: 3.80e-04 | 2745.2ms/step | 21.9 samples/s
468
+ Step 16940/50462 | Loss: 2.6950 | LR: 3.80e-04 | 2725.9ms/step | 22.0 samples/s
469
+ Step 16950/50462 | Loss: 2.6664 | LR: 3.80e-04 | 2731.0ms/step | 22.0 samples/s
470
+ Step 16960/50462 | Loss: 2.6897 | LR: 3.80e-04 | 2726.9ms/step | 22.0 samples/s
471
+ Step 16970/50462 | Loss: 2.6162 | LR: 3.80e-04 | 2733.0ms/step | 22.0 samples/s
472
+ Step 16980/50462 | Loss: 2.6773 | LR: 3.80e-04 | 2731.8ms/step | 22.0 samples/s
473
+ Step 16990/50462 | Loss: 2.6693 | LR: 3.79e-04 | 2724.9ms/step | 22.0 samples/s
474
+ Step 17000/50462 | Loss: 2.6845 | LR: 3.79e-04 | 2721.7ms/step | 22.0 samples/s
475
+ Validation at step 17000: Loss = 2.6720, Perplexity = 14.47
476
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-17000
477
+ Step 17010/50462 | Loss: 2.6804 | LR: 3.79e-04 | 4025.6ms/step | 14.9 samples/s
478
+ Step 17020/50462 | Loss: 2.6320 | LR: 3.79e-04 | 2734.4ms/step | 21.9 samples/s
479
+ Step 17030/50462 | Loss: 2.6923 | LR: 3.79e-04 | 2728.6ms/step | 22.0 samples/s
480
+ Step 17040/50462 | Loss: 2.7082 | LR: 3.79e-04 | 2730.0ms/step | 22.0 samples/s
481
+ Step 17050/50462 | Loss: 2.6657 | LR: 3.79e-04 | 2727.4ms/step | 22.0 samples/s
482
+ Step 17060/50462 | Loss: 2.6981 | LR: 3.79e-04 | 2733.2ms/step | 22.0 samples/s
483
+ Step 17070/50462 | Loss: 2.6774 | LR: 3.78e-04 | 2732.4ms/step | 22.0 samples/s
484
+ Step 17080/50462 | Loss: 2.6513 | LR: 3.78e-04 | 2729.9ms/step | 22.0 samples/s
485
+ Step 17090/50462 | Loss: 2.6867 | LR: 3.78e-04 | 2733.9ms/step | 21.9 samples/s
486
+ Step 17100/50462 | Loss: 2.6801 | LR: 3.78e-04 | 2721.5ms/step | 22.0 samples/s
487
+ Step 17110/50462 | Loss: 2.7076 | LR: 3.78e-04 | 2725.0ms/step | 22.0 samples/s
488
+ Step 17120/50462 | Loss: 2.6380 | LR: 3.78e-04 | 2739.3ms/step | 21.9 samples/s
489
+ Step 17130/50462 | Loss: 2.6618 | LR: 3.78e-04 | 2725.7ms/step | 22.0 samples/s
490
+ Step 17140/50462 | Loss: 2.6324 | LR: 3.77e-04 | 2728.8ms/step | 22.0 samples/s
491
+ Step 17150/50462 | Loss: 2.7325 | LR: 3.77e-04 | 2734.6ms/step | 21.9 samples/s
492
+ Step 17160/50462 | Loss: 2.6569 | LR: 3.77e-04 | 2728.5ms/step | 22.0 samples/s
493
+ Step 17170/50462 | Loss: 2.6587 | LR: 3.77e-04 | 2732.7ms/step | 22.0 samples/s
494
+ Step 17180/50462 | Loss: 2.6700 | LR: 3.77e-04 | 2734.9ms/step | 21.9 samples/s
495
+ Step 17190/50462 | Loss: 2.7040 | LR: 3.77e-04 | 2731.6ms/step | 22.0 samples/s
496
+ Step 17200/50462 | Loss: 2.6768 | LR: 3.77e-04 | 2726.0ms/step | 22.0 samples/s
497
+ Step 17210/50462 | Loss: 2.6773 | LR: 3.77e-04 | 2739.7ms/step | 21.9 samples/s
498
+ Step 17220/50462 | Loss: 2.6879 | LR: 3.76e-04 | 2732.3ms/step | 22.0 samples/s
499
+ Step 17230/50462 | Loss: 2.6379 | LR: 3.76e-04 | 2737.1ms/step | 21.9 samples/s
500
+ Step 17240/50462 | Loss: 2.6608 | LR: 3.76e-04 | 2727.3ms/step | 22.0 samples/s
501
+ Step 17250/50462 | Loss: 2.5845 | LR: 3.76e-04 | 2731.1ms/step | 22.0 samples/s
502
+ Validation at step 17250: Loss = 2.6706, Perplexity = 14.45
503
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-17250
504
+ Step 17260/50462 | Loss: 2.6934 | LR: 3.76e-04 | 4019.2ms/step | 14.9 samples/s
505
+ Step 17270/50462 | Loss: 2.6692 | LR: 3.76e-04 | 2736.4ms/step | 21.9 samples/s
506
+ Step 17280/50462 | Loss: 2.6980 | LR: 3.76e-04 | 2737.9ms/step | 21.9 samples/s
507
+ Step 17290/50462 | Loss: 2.6474 | LR: 3.75e-04 | 2738.1ms/step | 21.9 samples/s
508
+ Step 17300/50462 | Loss: 2.6646 | LR: 3.75e-04 | 2735.4ms/step | 21.9 samples/s
509
+ Step 17310/50462 | Loss: 2.6258 | LR: 3.75e-04 | 2730.3ms/step | 22.0 samples/s
510
+ Step 17320/50462 | Loss: 2.6844 | LR: 3.75e-04 | 2733.1ms/step | 22.0 samples/s
511
+ Step 17330/50462 | Loss: 2.6457 | LR: 3.75e-04 | 2733.9ms/step | 21.9 samples/s
512
+ Step 17340/50462 | Loss: 2.6843 | LR: 3.75e-04 | 2733.9ms/step | 21.9 samples/s
513
+ Step 17350/50462 | Loss: 2.6594 | LR: 3.75e-04 | 2730.7ms/step | 22.0 samples/s
514
+ Step 17360/50462 | Loss: 2.6924 | LR: 3.74e-04 | 2732.0ms/step | 22.0 samples/s
515
+ Step 17370/50462 | Loss: 2.6987 | LR: 3.74e-04 | 2732.3ms/step | 22.0 samples/s
516
+ Step 17380/50462 | Loss: 2.6502 | LR: 3.74e-04 | 2741.7ms/step | 21.9 samples/s
517
+ Step 17390/50462 | Loss: 2.6722 | LR: 3.74e-04 | 2735.6ms/step | 21.9 samples/s
518
+ Step 17400/50462 | Loss: 2.6416 | LR: 3.74e-04 | 2728.0ms/step | 22.0 samples/s
519
+ Step 17410/50462 | Loss: 2.6921 | LR: 3.74e-04 | 2734.2ms/step | 21.9 samples/s
520
+ Step 17420/50462 | Loss: 2.6474 | LR: 3.74e-04 | 2734.3ms/step | 21.9 samples/s
521
+ Step 17430/50462 | Loss: 2.6581 | LR: 3.74e-04 | 2725.7ms/step | 22.0 samples/s
522
+ Step 17440/50462 | Loss: 2.6892 | LR: 3.73e-04 | 2731.4ms/step | 22.0 samples/s
523
+ Step 17450/50462 | Loss: 2.6724 | LR: 3.73e-04 | 2735.1ms/step | 21.9 samples/s
524
+ Step 17460/50462 | Loss: 2.6684 | LR: 3.73e-04 | 2731.2ms/step | 22.0 samples/s
525
+ Step 17470/50462 | Loss: 2.6581 | LR: 3.73e-04 | 2741.6ms/step | 21.9 samples/s
526
+ Step 17480/50462 | Loss: 2.6565 | LR: 3.73e-04 | 2735.7ms/step | 21.9 samples/s
527
+ Step 17490/50462 | Loss: 2.6756 | LR: 3.73e-04 | 2733.0ms/step | 22.0 samples/s
528
+ Step 17500/50462 | Loss: 2.6783 | LR: 3.73e-04 | 2727.5ms/step | 22.0 samples/s
529
+ Validation at step 17500: Loss = 2.6676, Perplexity = 14.41
530
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-17500
531
+ Step 17510/50462 | Loss: 2.6520 | LR: 3.72e-04 | 4045.4ms/step | 14.8 samples/s
532
+ Step 17520/50462 | Loss: 2.6620 | LR: 3.72e-04 | 2725.5ms/step | 22.0 samples/s
533
+ Step 17530/50462 | Loss: 2.6621 | LR: 3.72e-04 | 2724.3ms/step | 22.0 samples/s
534
+ Step 17540/50462 | Loss: 2.6933 | LR: 3.72e-04 | 2733.9ms/step | 21.9 samples/s
535
+ Step 17550/50462 | Loss: 2.6699 | LR: 3.72e-04 | 2728.7ms/step | 22.0 samples/s
536
+ Step 17560/50462 | Loss: 2.6537 | LR: 3.72e-04 | 2727.9ms/step | 22.0 samples/s
537
+ Step 17570/50462 | Loss: 2.6596 | LR: 3.72e-04 | 2731.2ms/step | 22.0 samples/s
538
+ Step 17580/50462 | Loss: 2.6499 | LR: 3.71e-04 | 2724.2ms/step | 22.0 samples/s
539
+ Step 17590/50462 | Loss: 2.6620 | LR: 3.71e-04 | 2733.8ms/step | 21.9 samples/s
540
+ Step 17600/50462 | Loss: 2.6988 | LR: 3.71e-04 | 2731.7ms/step | 22.0 samples/s
541
+ Step 17610/50462 | Loss: 2.6824 | LR: 3.71e-04 | 2732.8ms/step | 22.0 samples/s
542
+ Step 17620/50462 | Loss: 2.6494 | LR: 3.71e-04 | 2731.3ms/step | 22.0 samples/s
543
+ Step 17630/50462 | Loss: 2.6220 | LR: 3.71e-04 | 2731.0ms/step | 22.0 samples/s
544
+ Step 17640/50462 | Loss: 2.6543 | LR: 3.71e-04 | 2729.9ms/step | 22.0 samples/s
545
+ Step 17650/50462 | Loss: 2.6947 | LR: 3.70e-04 | 2738.9ms/step | 21.9 samples/s
546
+ Step 17660/50462 | Loss: 2.6519 | LR: 3.70e-04 | 2735.2ms/step | 21.9 samples/s
547
+ Step 17670/50462 | Loss: 2.6739 | LR: 3.70e-04 | 2732.6ms/step | 22.0 samples/s
548
+ Step 17680/50462 | Loss: 2.6650 | LR: 3.70e-04 | 2730.6ms/step | 22.0 samples/s
549
+ Step 17690/50462 | Loss: 2.6549 | LR: 3.70e-04 | 2730.3ms/step | 22.0 samples/s
550
+ Step 17700/50462 | Loss: 2.6899 | LR: 3.70e-04 | 2733.2ms/step | 22.0 samples/s
551
+ Step 17710/50462 | Loss: 2.6135 | LR: 3.70e-04 | 2735.9ms/step | 21.9 samples/s
552
+ Step 17720/50462 | Loss: 2.6713 | LR: 3.69e-04 | 2722.1ms/step | 22.0 samples/s
553
+ Step 17730/50462 | Loss: 2.6548 | LR: 3.69e-04 | 2733.4ms/step | 22.0 samples/s
554
+ Step 17740/50462 | Loss: 2.6785 | LR: 3.69e-04 | 2724.4ms/step | 22.0 samples/s
555
+ Step 17750/50462 | Loss: 2.6427 | LR: 3.69e-04 | 2735.9ms/step | 21.9 samples/s
556
+ Validation at step 17750: Loss = 2.6657, Perplexity = 14.38
557
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-17750
558
+ Step 17760/50462 | Loss: 2.6862 | LR: 3.69e-04 | 4031.1ms/step | 14.9 samples/s
559
+ Step 17770/50462 | Loss: 2.6702 | LR: 3.69e-04 | 2725.0ms/step | 22.0 samples/s
560
+ Step 17780/50462 | Loss: 2.6685 | LR: 3.69e-04 | 2726.4ms/step | 22.0 samples/s
561
+ Step 17790/50462 | Loss: 2.6788 | LR: 3.69e-04 | 2727.6ms/step | 22.0 samples/s
562
+ Step 17800/50462 | Loss: 2.6867 | LR: 3.68e-04 | 2731.8ms/step | 22.0 samples/s
563
+ Step 17810/50462 | Loss: 2.6818 | LR: 3.68e-04 | 2727.3ms/step | 22.0 samples/s
564
+ Step 17820/50462 | Loss: 2.6717 | LR: 3.68e-04 | 2737.5ms/step | 21.9 samples/s
565
+ Step 17830/50462 | Loss: 2.6402 | LR: 3.68e-04 | 2729.9ms/step | 22.0 samples/s
566
+ Step 17840/50462 | Loss: 2.6533 | LR: 3.68e-04 | 2736.1ms/step | 21.9 samples/s
567
+ Step 17850/50462 | Loss: 2.6653 | LR: 3.68e-04 | 2734.1ms/step | 21.9 samples/s
568
+ Step 17860/50462 | Loss: 2.6662 | LR: 3.68e-04 | 2726.4ms/step | 22.0 samples/s
569
+ Step 17870/50462 | Loss: 2.6795 | LR: 3.67e-04 | 2736.3ms/step | 21.9 samples/s
570
+ Step 17880/50462 | Loss: 2.7062 | LR: 3.67e-04 | 2733.8ms/step | 21.9 samples/s
571
+ Step 17890/50462 | Loss: 2.6410 | LR: 3.67e-04 | 2732.1ms/step | 22.0 samples/s
572
+ Step 17900/50462 | Loss: 2.6764 | LR: 3.67e-04 | 2730.7ms/step | 22.0 samples/s
573
+ Step 17910/50462 | Loss: 2.6859 | LR: 3.67e-04 | 2732.7ms/step | 22.0 samples/s
574
+ Step 17920/50462 | Loss: 2.6040 | LR: 3.67e-04 | 2730.7ms/step | 22.0 samples/s
575
+ Step 17930/50462 | Loss: 2.6434 | LR: 3.67e-04 | 2731.5ms/step | 22.0 samples/s
576
+ Step 17940/50462 | Loss: 2.6653 | LR: 3.66e-04 | 2735.0ms/step | 21.9 samples/s
577
+ Step 17950/50462 | Loss: 2.7027 | LR: 3.66e-04 | 2729.6ms/step | 22.0 samples/s
578
+ Step 17960/50462 | Loss: 2.6504 | LR: 3.66e-04 | 2735.2ms/step | 21.9 samples/s
579
+ Step 17970/50462 | Loss: 2.6545 | LR: 3.66e-04 | 2735.8ms/step | 21.9 samples/s
580
+ Step 17980/50462 | Loss: 2.6767 | LR: 3.66e-04 | 2736.4ms/step | 21.9 samples/s
581
+ Step 17990/50462 | Loss: 2.6453 | LR: 3.66e-04 | 2731.8ms/step | 22.0 samples/s
582
+ Step 18000/50462 | Loss: 2.6472 | LR: 3.66e-04 | 2733.8ms/step | 21.9 samples/s
583
+ Validation at step 18000: Loss = 2.6609, Perplexity = 14.31
584
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-18000
585
+ Step 18010/50462 | Loss: 2.6863 | LR: 3.65e-04 | 4030.9ms/step | 14.9 samples/s
586
+ Step 18020/50462 | Loss: 2.6679 | LR: 3.65e-04 | 2728.2ms/step | 22.0 samples/s
587
+ Step 18030/50462 | Loss: 2.6792 | LR: 3.65e-04 | 2730.3ms/step | 22.0 samples/s
588
+ Step 18040/50462 | Loss: 2.7246 | LR: 3.65e-04 | 2731.9ms/step | 22.0 samples/s
589
+ Step 18050/50462 | Loss: 2.6650 | LR: 3.65e-04 | 2727.1ms/step | 22.0 samples/s
590
+ Step 18060/50462 | Loss: 2.6979 | LR: 3.65e-04 | 2724.1ms/step | 22.0 samples/s
591
+ Step 18070/50462 | Loss: 2.6626 | LR: 3.65e-04 | 2729.9ms/step | 22.0 samples/s
592
+ Step 18080/50462 | Loss: 2.6854 | LR: 3.64e-04 | 2726.1ms/step | 22.0 samples/s
593
+ Step 18090/50462 | Loss: 2.6857 | LR: 3.64e-04 | 2728.2ms/step | 22.0 samples/s
594
+ Step 18100/50462 | Loss: 2.6748 | LR: 3.64e-04 | 2736.8ms/step | 21.9 samples/s
595
+ Step 18110/50462 | Loss: 2.6414 | LR: 3.64e-04 | 2737.4ms/step | 21.9 samples/s
596
+ Step 18120/50462 | Loss: 2.6935 | LR: 3.64e-04 | 2729.0ms/step | 22.0 samples/s
597
+ Step 18130/50462 | Loss: 2.6627 | LR: 3.64e-04 | 2733.6ms/step | 21.9 samples/s
598
+ Step 18140/50462 | Loss: 2.6743 | LR: 3.64e-04 | 2732.9ms/step | 22.0 samples/s
599
+ Step 18150/50462 | Loss: 2.6629 | LR: 3.63e-04 | 2732.2ms/step | 22.0 samples/s
600
+ Step 18160/50462 | Loss: 2.5906 | LR: 3.63e-04 | 2734.2ms/step | 21.9 samples/s
601
+ Step 18170/50462 | Loss: 2.6353 | LR: 3.63e-04 | 2728.2ms/step | 22.0 samples/s
602
+ Step 18180/50462 | Loss: 2.6579 | LR: 3.63e-04 | 2728.6ms/step | 22.0 samples/s
603
+ Step 18190/50462 | Loss: 2.6562 | LR: 3.63e-04 | 2728.3ms/step | 22.0 samples/s
604
+ Step 18200/50462 | Loss: 2.6539 | LR: 3.63e-04 | 2724.8ms/step | 22.0 samples/s
605
+ Step 18210/50462 | Loss: 2.6726 | LR: 3.63e-04 | 2725.7ms/step | 22.0 samples/s
606
+ Step 18220/50462 | Loss: 2.6624 | LR: 3.62e-04 | 2732.0ms/step | 22.0 samples/s
607
+ Step 18230/50462 | Loss: 2.6386 | LR: 3.62e-04 | 2719.5ms/step | 22.1 samples/s
608
+ Step 18240/50462 | Loss: 2.7183 | LR: 3.62e-04 | 2729.0ms/step | 22.0 samples/s
609
+ Step 18250/50462 | Loss: 2.7067 | LR: 3.62e-04 | 2733.8ms/step | 21.9 samples/s
610
+ Validation at step 18250: Loss = 2.6583, Perplexity = 14.27
611
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-18250
612
+ Step 18260/50462 | Loss: 2.6648 | LR: 3.62e-04 | 4036.6ms/step | 14.9 samples/s
613
+ Step 18270/50462 | Loss: 2.6706 | LR: 3.62e-04 | 2726.0ms/step | 22.0 samples/s
614
+ Step 18280/50462 | Loss: 2.6713 | LR: 3.62e-04 | 2736.8ms/step | 21.9 samples/s
615
+ Step 18290/50462 | Loss: 2.6560 | LR: 3.62e-04 | 2728.3ms/step | 22.0 samples/s
616
+ Step 18300/50462 | Loss: 2.6325 | LR: 3.61e-04 | 2724.0ms/step | 22.0 samples/s
617
+ Step 18310/50462 | Loss: 2.6571 | LR: 3.61e-04 | 2745.0ms/step | 21.9 samples/s
618
+ Step 18320/50462 | Loss: 2.6715 | LR: 3.61e-04 | 2725.7ms/step | 22.0 samples/s
619
+ Step 18330/50462 | Loss: 2.6486 | LR: 3.61e-04 | 2737.6ms/step | 21.9 samples/s
620
+ Step 18340/50462 | Loss: 2.6697 | LR: 3.61e-04 | 2736.9ms/step | 21.9 samples/s
621
+ Step 18350/50462 | Loss: 2.6419 | LR: 3.61e-04 | 2731.0ms/step | 22.0 samples/s
622
+ Step 18360/50462 | Loss: 2.6722 | LR: 3.61e-04 | 2728.5ms/step | 22.0 samples/s
623
+ Step 18370/50462 | Loss: 2.7074 | LR: 3.60e-04 | 2726.4ms/step | 22.0 samples/s
624
+ Step 18380/50462 | Loss: 2.6623 | LR: 3.60e-04 | 2730.2ms/step | 22.0 samples/s
625
+ Step 18390/50462 | Loss: 2.6379 | LR: 3.60e-04 | 2727.6ms/step | 22.0 samples/s
626
+ Step 18400/50462 | Loss: 2.6563 | LR: 3.60e-04 | 2735.7ms/step | 21.9 samples/s
627
+ Step 18410/50462 | Loss: 2.7057 | LR: 3.60e-04 | 2726.7ms/step | 22.0 samples/s
628
+ Step 18420/50462 | Loss: 2.6566 | LR: 3.60e-04 | 2735.7ms/step | 21.9 samples/s
629
+ Step 18430/50462 | Loss: 2.6818 | LR: 3.60e-04 | 2729.2ms/step | 22.0 samples/s
630
+ Step 18440/50462 | Loss: 2.6553 | LR: 3.59e-04 | 2728.9ms/step | 22.0 samples/s
631
+ Step 18450/50462 | Loss: 2.6647 | LR: 3.59e-04 | 2732.8ms/step | 22.0 samples/s
632
+ Step 18460/50462 | Loss: 2.6571 | LR: 3.59e-04 | 2726.5ms/step | 22.0 samples/s
633
+ Step 18470/50462 | Loss: 2.6349 | LR: 3.59e-04 | 2729.7ms/step | 22.0 samples/s
634
+ Step 18480/50462 | Loss: 2.6575 | LR: 3.59e-04 | 2728.7ms/step | 22.0 samples/s
635
+ Step 18490/50462 | Loss: 2.6413 | LR: 3.59e-04 | 2734.1ms/step | 21.9 samples/s
636
+ Step 18500/50462 | Loss: 2.6846 | LR: 3.59e-04 | 2729.3ms/step | 22.0 samples/s
637
+ Validation at step 18500: Loss = 2.6572, Perplexity = 14.26
638
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-18500
639
+ Step 18510/50462 | Loss: 2.6338 | LR: 3.58e-04 | 4032.9ms/step | 14.9 samples/s
640
+ Step 18520/50462 | Loss: 2.6985 | LR: 3.58e-04 | 2724.8ms/step | 22.0 samples/s
641
+ Step 18530/50462 | Loss: 2.6444 | LR: 3.58e-04 | 2728.8ms/step | 22.0 samples/s
642
+ Step 18540/50462 | Loss: 2.6799 | LR: 3.58e-04 | 2732.3ms/step | 22.0 samples/s
643
+ Step 18550/50462 | Loss: 2.6887 | LR: 3.58e-04 | 2726.3ms/step | 22.0 samples/s
644
+ Step 18560/50462 | Loss: 2.6246 | LR: 3.58e-04 | 2735.8ms/step | 21.9 samples/s
645
+ Step 18570/50462 | Loss: 2.6363 | LR: 3.58e-04 | 2739.5ms/step | 21.9 samples/s
646
+ Step 18580/50462 | Loss: 2.7045 | LR: 3.57e-04 | 2728.2ms/step | 22.0 samples/s
647
+ Step 18590/50462 | Loss: 2.6486 | LR: 3.57e-04 | 2732.1ms/step | 22.0 samples/s
648
+ Step 18600/50462 | Loss: 2.6317 | LR: 3.57e-04 | 2735.8ms/step | 21.9 samples/s
649
+ Step 18610/50462 | Loss: 2.6335 | LR: 3.57e-04 | 2736.5ms/step | 21.9 samples/s
650
+ Step 18620/50462 | Loss: 2.6356 | LR: 3.57e-04 | 2732.9ms/step | 22.0 samples/s
651
+ Step 18630/50462 | Loss: 2.7029 | LR: 3.57e-04 | 2728.7ms/step | 22.0 samples/s
652
+ Step 18640/50462 | Loss: 2.6761 | LR: 3.57e-04 | 2731.9ms/step | 22.0 samples/s
653
+ Step 18650/50462 | Loss: 2.6528 | LR: 3.56e-04 | 2731.4ms/step | 22.0 samples/s
654
+ Step 18660/50462 | Loss: 2.6383 | LR: 3.56e-04 | 2728.3ms/step | 22.0 samples/s
655
+ Step 18670/50462 | Loss: 2.6409 | LR: 3.56e-04 | 2732.7ms/step | 22.0 samples/s
656
+ Step 18680/50462 | Loss: 2.6494 | LR: 3.56e-04 | 2731.7ms/step | 22.0 samples/s
657
+ Step 18690/50462 | Loss: 2.6447 | LR: 3.56e-04 | 2733.4ms/step | 22.0 samples/s
658
+ Step 18700/50462 | Loss: 2.6250 | LR: 3.56e-04 | 2724.8ms/step | 22.0 samples/s
epfl_final/train_logs/p2_1p2b_mixed_seed70_repaired_resume14500_6gpu.pid ADDED
@@ -0,0 +1 @@
 
 
1
+ 7952
epfl_final/train_logs/p2_1p2b_mixed_seed70_repaired_resume18500_6gpu.log ADDED
The diff for this file is too large to render. See raw diff
 
epfl_final/train_logs/p2_1p2b_mixed_seed70_repaired_resume33750_6gpu.log ADDED
@@ -0,0 +1,396 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ *****************************************
3
+ Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
4
+ *****************************************
5
+ Resuming: Using existing output directory /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557
6
+ Running config:
7
+ {
8
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
9
+ "model_dtype": "bfloat16",
10
+ "init_from_scratch": true,
11
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
12
+ "val_path": null,
13
+ "max_seq_length": 8192,
14
+ "num_workers": 8,
15
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557",
16
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-33750",
17
+ "force_new_run": false,
18
+ "num_epochs": 1,
19
+ "max_steps": 50462,
20
+ "per_device_batch_size": 1,
21
+ "gradient_accumulation_steps": 10,
22
+ "learning_rate": 0.0005,
23
+ "weight_decay": 0.01,
24
+ "warmup_steps": 750,
25
+ "lr_scheduler": "cosine",
26
+ "max_grad_norm": 1.0,
27
+ "save_steps": 250,
28
+ "eval_steps": 250,
29
+ "eval_batches": 50,
30
+ "logging_steps": 10,
31
+ "seed": 70,
32
+ "verbose": true,
33
+ "checkpoint_activations": true,
34
+ "resume_micro_step_override": null,
35
+ "strategy": "ddp",
36
+ "mixed_precision": "bf16",
37
+ "wandb_enabled": false,
38
+ "wandb_project": "stochastic_sparsity",
39
+ "wandb_name": "1.2b mixed 8k seed70 repaired resume33750",
40
+ "wandb_tags": [
41
+ "qwen3",
42
+ "1.2b",
43
+ "8k",
44
+ "p2",
45
+ "mixed",
46
+ "seed70",
47
+ "repaired"
48
+ ],
49
+ "hidden_size": 1536,
50
+ "intermediate_size": 6144,
51
+ "num_attention_heads": 12,
52
+ "num_key_value_heads": 4,
53
+ "num_hidden_layers": 28,
54
+ "head_dim": null,
55
+ "attention_dropout": null,
56
+ "rope_train_position_scale_set": null,
57
+ "attention_type": "sparse_stochastic",
58
+ "sparse_block_size": 128,
59
+ "sparse_num_random_blocks": 35,
60
+ "sparse_mask_pool_size": 10000,
61
+ "sparse_val_dense_compare": false,
62
+ "sparse_layer_indices": null,
63
+ "sparse_mask_cache_dir": "/mloscratch/homes/navasard/sparse/mask_cache_8k_mixed",
64
+ "sparse_mask_seed": 70,
65
+ "sparse_mixed_nrb_set": [
66
+ 63,
67
+ 47,
68
+ 31,
69
+ 21,
70
+ 11
71
+ ],
72
+ "dense_mode": false,
73
+ "dense_step_intervals": null
74
+ }
75
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
76
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-33750
77
+ Checkpoint activations: TrueCheckpoint activations: TrueCheckpoint activations: True
78
+ Checkpoint activations: TrueCheckpoint activations: True
79
+
80
+
81
+
82
+ Checkpoint activations: True
83
+
84
+
85
+ ================================================================================
86
+ MODEL ARCHITECTURE
87
+ ================================================================================
88
+ Model config:
89
+ vocab_size: 151936
90
+ hidden_size: 1536
91
+ num_hidden_layers: 28
92
+ num_attention_heads: 12
93
+ num_key_value_heads: 4
94
+ head_dim: 128
95
+ intermediate_size: 6144
96
+ max_position_embeddings: 32768
97
+ rope_theta: 1000000
98
+ rope_train_position_scale_set: None
99
+ attention_type: sparse_stochastic
100
+ sparse_block_size: 128
101
+ sparse_num_random_blocks: 35
102
+ sparse_mask_pool_size: 10000
103
+ sparse_max_seq_length: 8192
104
+
105
+ Parameter count:
106
+ Total parameters: 1,202,352,640
107
+ Trainable parameters: 1,202,352,640
108
+ Non-trainable parameters: 0
109
+ Model size (BF16): 2.40 GB
110
+
111
+ --------------------------------------------------------------------------------
112
+ LAYER-BY-LAYER SUMMARY
113
+ --------------------------------------------------------------------------------
114
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/nn/attention/flex_attention.py:1704: UserWarning: flex_attention called without torch.compile() - this will use an unfused implementation that materializes the full scores matrix instead of generating a fused kernel.
115
+
116
+ SOLUTION: Use torch.compile(flex_attention)(...)
117
+
118
+ If you want to debug your score_mod/mask_mod, you can set:
119
+ torch.nn.attention.flex_attention._FLEX_ATTENTION_DISABLE_COMPILE_DEBUG = True
120
+
121
+ This will allow you to use print statements or breakpoints. Note: This doesn't work with the backwards pass and may produce incorrect results.
122
+ _warn_once(
123
+ ================================================================================================================================================================
124
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
125
+ ================================================================================================================================================================
126
+ Qwen3ForCausalLM [1, 8192] -- -- True
127
+ ├─Qwen3Model: 1-1 -- [1, 8192, 1536] -- True
128
+ │ └─Embedding: 2-1 [1, 8192] [1, 8192, 1536] 233,373,696 True
129
+ │ └─ModuleList: 2-2 -- -- -- True
130
+ │ │ └─Qwen3DecoderLayer: 3-1 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
131
+ │ │ └─Qwen3DecoderLayer: 3-2 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
132
+ │ │ └─Qwen3DecoderLayer: 3-3 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
133
+ │ │ └─Qwen3DecoderLayer: 3-4 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
134
+ │ │ └─Qwen3DecoderLayer: 3-5 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
135
+ │ │ └─Qwen3DecoderLayer: 3-6 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
136
+ │ │ └─Qwen3DecoderLayer: 3-7 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
137
+ │ │ └─Qwen3DecoderLayer: 3-8 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
138
+ │ │ └─Qwen3DecoderLayer: 3-9 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
139
+ │ │ └─Qwen3DecoderLayer: 3-10 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
140
+ │ │ └─Qwen3DecoderLayer: 3-11 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
141
+ │ │ └─Qwen3DecoderLayer: 3-12 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
142
+ │ │ └─Qwen3DecoderLayer: 3-13 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
143
+ │ │ └─Qwen3DecoderLayer: 3-14 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
144
+ │ │ └─Qwen3DecoderLayer: 3-15 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
145
+ │ │ └─Qwen3DecoderLayer: 3-16 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
146
+ │ │ └─Qwen3DecoderLayer: 3-17 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
147
+ │ │ └─Qwen3DecoderLayer: 3-18 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
148
+ │ │ └─Qwen3DecoderLayer: 3-19 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
149
+ │ │ └─Qwen3DecoderLayer: 3-20 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
150
+ │ │ └─Qwen3DecoderLayer: 3-21 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
151
+ │ │ └─Qwen3DecoderLayer: 3-22 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
152
+ │ │ └─Qwen3DecoderLayer: 3-23 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
153
+ │ │ └─Qwen3DecoderLayer: 3-24 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
154
+ │ │ └─Qwen3DecoderLayer: 3-25 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
155
+ │ │ └─Qwen3DecoderLayer: 3-26 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
156
+ │ │ └─Qwen3DecoderLayer: 3-27 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
157
+ │ │ └─Qwen3DecoderLayer: 3-28 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
158
+ │ └─Qwen3RMSNorm: 2-3 [1, 8192, 1536] [1, 8192, 1536] 1,536 True
159
+ ├─Linear: 1-2 [1, 8192, 1536] [1, 8192, 151936] 233,373,696 True
160
+ ================================================================================================================================================================
161
+ Total params: 1,435,726,336
162
+ Trainable params: 1,435,726,336
163
+ Non-trainable params: 0
164
+ Total mult-adds (Units.GIGABYTES): 1.44
165
+ ================================================================================================================================================================
166
+ Input size (MB): 0.07
167
+ Forward/backward pass size (MB): 26218.59
168
+ Params size (MB): 2871.45
169
+ Estimated Total Size (MB): 29090.11
170
+ ================================================================================================================================================================
171
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (233,373,696 params). True unique parameter count: 1,202,352,640 (1202.4M)
172
+ ================================================================================
173
+
174
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
175
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
176
+ Using pre-tokenized dataset (no runtime tokenization)
177
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
178
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
179
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
180
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
181
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
182
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
183
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
184
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
185
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
186
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
187
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
188
+ Dataset split: 3027764 train, 159356 validation
189
+ Training examples (packed, train split): 3027764
190
+ Validation examples (packed, val split): 159356
191
+ Per-device batch size: 1
192
+ World size (GPUs): 6
193
+ Gradient accumulation steps: 10
194
+ Effective batch size (seqs/step): 60
195
+ Sequence length: 8192
196
+ Tokens per optimizer step: 491,520
197
+ Micro-batches per epoch (per rank): 504628
198
+ Optimizer steps per epoch: 50462
199
+ Total optimizer steps: 50462 (capped by max_steps; 1.00 epochs)
200
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-33750...
201
+ Preserving checkpoint LR state for exact resume: current=[0.00012693872600679152], schedule_base=[0.0005]
202
+ Resumed from checkpoint at step 33750 (epoch 0, micro_step 337500)
203
+ Resuming training from step 33750
204
+ Starting training...
205
+ Validation at step 33750: Loss = 2.5441, Perplexity = 12.73
206
+ Switching attention mode for step 33751: sparse_stochastic (configured sparse attention)
207
+ Step 33760/50462 | Loss: 2.5224 | LR: 1.27e-04 | 58286.4ms/step | 1.0 samples/s
208
+ Step 33770/50462 | Loss: 2.5431 | LR: 1.27e-04 | 2729.4ms/step | 22.0 samples/s
209
+ Step 33780/50462 | Loss: 2.5172 | LR: 1.27e-04 | 2732.4ms/step | 22.0 samples/s
210
+ Step 33790/50462 | Loss: 2.5535 | LR: 1.26e-04 | 2733.5ms/step | 21.9 samples/s
211
+ Step 33800/50462 | Loss: 2.5407 | LR: 1.26e-04 | 2741.5ms/step | 21.9 samples/s
212
+ Step 33810/50462 | Loss: 2.5477 | LR: 1.26e-04 | 2734.1ms/step | 21.9 samples/s
213
+ Step 33820/50462 | Loss: 2.5370 | LR: 1.26e-04 | 2731.9ms/step | 22.0 samples/s
214
+ Step 33830/50462 | Loss: 2.5127 | LR: 1.26e-04 | 2741.2ms/step | 21.9 samples/s
215
+ Step 33840/50462 | Loss: 2.5255 | LR: 1.26e-04 | 2735.2ms/step | 21.9 samples/s
216
+ Step 33850/50462 | Loss: 2.5279 | LR: 1.26e-04 | 2736.6ms/step | 21.9 samples/s
217
+ Step 33860/50462 | Loss: 2.5843 | LR: 1.25e-04 | 2729.8ms/step | 22.0 samples/s
218
+ Step 33870/50462 | Loss: 2.5187 | LR: 1.25e-04 | 2735.0ms/step | 21.9 samples/s
219
+ Step 33880/50462 | Loss: 2.5502 | LR: 1.25e-04 | 2738.5ms/step | 21.9 samples/s
220
+ Step 33890/50462 | Loss: 2.5276 | LR: 1.25e-04 | 2738.9ms/step | 21.9 samples/s
221
+ Step 33900/50462 | Loss: 2.5618 | LR: 1.25e-04 | 2732.9ms/step | 22.0 samples/s
222
+ Step 33910/50462 | Loss: 2.5378 | LR: 1.25e-04 | 2736.2ms/step | 21.9 samples/s
223
+ Step 33920/50462 | Loss: 2.5956 | LR: 1.25e-04 | 2737.1ms/step | 21.9 samples/s
224
+ Step 33930/50462 | Loss: 2.5315 | LR: 1.24e-04 | 2746.3ms/step | 21.8 samples/s
225
+ Step 33940/50462 | Loss: 2.5399 | LR: 1.24e-04 | 2733.1ms/step | 22.0 samples/s
226
+ Step 33950/50462 | Loss: 2.5540 | LR: 1.24e-04 | 2729.1ms/step | 22.0 samples/s
227
+ Step 33960/50462 | Loss: 2.5436 | LR: 1.24e-04 | 2736.1ms/step | 21.9 samples/s
228
+ Step 33970/50462 | Loss: 2.5241 | LR: 1.24e-04 | 2733.2ms/step | 22.0 samples/s
229
+ Step 33980/50462 | Loss: 2.5547 | LR: 1.24e-04 | 2735.8ms/step | 21.9 samples/s
230
+ Step 33990/50462 | Loss: 2.5109 | LR: 1.24e-04 | 2731.5ms/step | 22.0 samples/s
231
+ Step 34000/50462 | Loss: 2.4949 | LR: 1.24e-04 | 2725.5ms/step | 22.0 samples/s
232
+ Validation at step 34000: Loss = 2.5419, Perplexity = 12.70
233
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-34000
234
+ Step 34010/50462 | Loss: 2.5240 | LR: 1.23e-04 | 4130.7ms/step | 14.5 samples/s
235
+ Step 34020/50462 | Loss: 2.5449 | LR: 1.23e-04 | 2732.5ms/step | 22.0 samples/s
236
+ Step 34030/50462 | Loss: 2.5558 | LR: 1.23e-04 | 2739.1ms/step | 21.9 samples/s
237
+ Step 34040/50462 | Loss: 2.5722 | LR: 1.23e-04 | 2739.2ms/step | 21.9 samples/s
238
+ Step 34050/50462 | Loss: 2.5127 | LR: 1.23e-04 | 2743.9ms/step | 21.9 samples/s
239
+ Step 34060/50462 | Loss: 2.5249 | LR: 1.23e-04 | 2741.4ms/step | 21.9 samples/s
240
+ Step 34070/50462 | Loss: 2.5500 | LR: 1.23e-04 | 2741.4ms/step | 21.9 samples/s
241
+ Step 34080/50462 | Loss: 2.5539 | LR: 1.22e-04 | 2736.3ms/step | 21.9 samples/s
242
+ Step 34090/50462 | Loss: 2.5080 | LR: 1.22e-04 | 2736.5ms/step | 21.9 samples/s
243
+ Step 34100/50462 | Loss: 2.5014 | LR: 1.22e-04 | 2740.0ms/step | 21.9 samples/s
244
+ Step 34110/50462 | Loss: 2.5269 | LR: 1.22e-04 | 2738.8ms/step | 21.9 samples/s
245
+ Step 34120/50462 | Loss: 2.5424 | LR: 1.22e-04 | 2729.0ms/step | 22.0 samples/s
246
+ Step 34130/50462 | Loss: 2.5753 | LR: 1.22e-04 | 2730.9ms/step | 22.0 samples/s
247
+ Step 34140/50462 | Loss: 2.5395 | LR: 1.22e-04 | 2727.4ms/step | 22.0 samples/s
248
+ Step 34150/50462 | Loss: 2.5038 | LR: 1.21e-04 | 2738.6ms/step | 21.9 samples/s
249
+ Step 34160/50462 | Loss: 2.5317 | LR: 1.21e-04 | 2743.4ms/step | 21.9 samples/s
250
+ Step 34170/50462 | Loss: 2.5024 | LR: 1.21e-04 | 2730.5ms/step | 22.0 samples/s
251
+ Step 34180/50462 | Loss: 2.5678 | LR: 1.21e-04 | 2725.4ms/step | 22.0 samples/s
252
+ Step 34190/50462 | Loss: 2.5463 | LR: 1.21e-04 | 2740.6ms/step | 21.9 samples/s
253
+ Step 34200/50462 | Loss: 2.5071 | LR: 1.21e-04 | 2727.6ms/step | 22.0 samples/s
254
+ Step 34210/50462 | Loss: 2.5584 | LR: 1.21e-04 | 2727.0ms/step | 22.0 samples/s
255
+ Step 34220/50462 | Loss: 2.5394 | LR: 1.21e-04 | 2738.2ms/step | 21.9 samples/s
256
+ Step 34230/50462 | Loss: 2.5460 | LR: 1.20e-04 | 2727.8ms/step | 22.0 samples/s
257
+ Step 34240/50462 | Loss: 2.5257 | LR: 1.20e-04 | 2741.2ms/step | 21.9 samples/s
258
+ Step 34250/50462 | Loss: 2.5730 | LR: 1.20e-04 | 2738.2ms/step | 21.9 samples/s
259
+ Validation at step 34250: Loss = 2.5403, Perplexity = 12.68
260
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-34250
261
+ Step 34260/50462 | Loss: 2.5114 | LR: 1.20e-04 | 4123.2ms/step | 14.6 samples/s
262
+ Step 34270/50462 | Loss: 2.4958 | LR: 1.20e-04 | 2728.8ms/step | 22.0 samples/s
263
+ Step 34280/50462 | Loss: 2.5627 | LR: 1.20e-04 | 2735.7ms/step | 21.9 samples/s
264
+ Step 34290/50462 | Loss: 2.5323 | LR: 1.20e-04 | 2740.3ms/step | 21.9 samples/s
265
+ Step 34300/50462 | Loss: 2.5131 | LR: 1.19e-04 | 2740.8ms/step | 21.9 samples/s
266
+ Step 34310/50462 | Loss: 2.5591 | LR: 1.19e-04 | 2735.1ms/step | 21.9 samples/s
267
+ Step 34320/50462 | Loss: 2.5166 | LR: 1.19e-04 | 2742.0ms/step | 21.9 samples/s
268
+ Step 34330/50462 | Loss: 2.5768 | LR: 1.19e-04 | 2737.0ms/step | 21.9 samples/s
269
+ Step 34340/50462 | Loss: 2.5503 | LR: 1.19e-04 | 2721.7ms/step | 22.0 samples/s
270
+ Step 34350/50462 | Loss: 2.5362 | LR: 1.19e-04 | 2728.6ms/step | 22.0 samples/s
271
+ Step 34360/50462 | Loss: 2.5423 | LR: 1.19e-04 | 2741.0ms/step | 21.9 samples/s
272
+ Step 34370/50462 | Loss: 2.5792 | LR: 1.19e-04 | 2731.2ms/step | 22.0 samples/s
273
+ Step 34380/50462 | Loss: 2.5308 | LR: 1.18e-04 | 2728.1ms/step | 22.0 samples/s
274
+ Step 34390/50462 | Loss: 2.5912 | LR: 1.18e-04 | 2737.9ms/step | 21.9 samples/s
275
+ Step 34400/50462 | Loss: 2.5378 | LR: 1.18e-04 | 2738.4ms/step | 21.9 samples/s
276
+ Step 34410/50462 | Loss: 2.5832 | LR: 1.18e-04 | 2729.6ms/step | 22.0 samples/s
277
+ Step 34420/50462 | Loss: 2.5102 | LR: 1.18e-04 | 2734.9ms/step | 21.9 samples/s
278
+ Step 34430/50462 | Loss: 2.5407 | LR: 1.18e-04 | 2730.3ms/step | 22.0 samples/s
279
+ Step 34440/50462 | Loss: 2.5243 | LR: 1.18e-04 | 2729.8ms/step | 22.0 samples/s
280
+ Step 34450/50462 | Loss: 2.5329 | LR: 1.17e-04 | 2731.7ms/step | 22.0 samples/s
281
+ Step 34460/50462 | Loss: 2.5277 | LR: 1.17e-04 | 2731.2ms/step | 22.0 samples/s
282
+ Step 34470/50462 | Loss: 2.5015 | LR: 1.17e-04 | 2735.8ms/step | 21.9 samples/s
283
+ Step 34480/50462 | Loss: 2.5375 | LR: 1.17e-04 | 2734.1ms/step | 21.9 samples/s
284
+ Step 34490/50462 | Loss: 2.5238 | LR: 1.17e-04 | 2732.6ms/step | 22.0 samples/s
285
+ Step 34500/50462 | Loss: 2.5346 | LR: 1.17e-04 | 2735.8ms/step | 21.9 samples/s
286
+ Validation at step 34500: Loss = 2.5401, Perplexity = 12.68
287
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-34500
288
+ Step 34510/50462 | Loss: 2.5113 | LR: 1.17e-04 | 4116.6ms/step | 14.6 samples/s
289
+ Step 34520/50462 | Loss: 2.5376 | LR: 1.16e-04 | 2731.2ms/step | 22.0 samples/s
290
+ Step 34530/50462 | Loss: 2.5175 | LR: 1.16e-04 | 2735.3ms/step | 21.9 samples/s
291
+ Step 34540/50462 | Loss: 2.5192 | LR: 1.16e-04 | 2728.6ms/step | 22.0 samples/s
292
+ Step 34550/50462 | Loss: 2.4967 | LR: 1.16e-04 | 2732.2ms/step | 22.0 samples/s
293
+ Step 34560/50462 | Loss: 2.5574 | LR: 1.16e-04 | 2735.2ms/step | 21.9 samples/s
294
+ Step 34570/50462 | Loss: 2.5261 | LR: 1.16e-04 | 2734.0ms/step | 21.9 samples/s
295
+ Step 34580/50462 | Loss: 2.5486 | LR: 1.16e-04 | 2735.5ms/step | 21.9 samples/s
296
+ Step 34590/50462 | Loss: 2.5422 | LR: 1.16e-04 | 2733.4ms/step | 22.0 samples/s
297
+ Step 34600/50462 | Loss: 2.5749 | LR: 1.15e-04 | 2727.5ms/step | 22.0 samples/s
298
+ Step 34610/50462 | Loss: 2.5501 | LR: 1.15e-04 | 2727.9ms/step | 22.0 samples/s
299
+ Step 34620/50462 | Loss: 2.5237 | LR: 1.15e-04 | 2731.1ms/step | 22.0 samples/s
300
+ Step 34630/50462 | Loss: 2.5372 | LR: 1.15e-04 | 2730.6ms/step | 22.0 samples/s
301
+ Step 34640/50462 | Loss: 2.5230 | LR: 1.15e-04 | 2735.8ms/step | 21.9 samples/s
302
+ Step 34650/50462 | Loss: 2.4897 | LR: 1.15e-04 | 2728.4ms/step | 22.0 samples/s
303
+ Step 34660/50462 | Loss: 2.5367 | LR: 1.15e-04 | 2725.6ms/step | 22.0 samples/s
304
+ Step 34670/50462 | Loss: 2.5826 | LR: 1.15e-04 | 2742.4ms/step | 21.9 samples/s
305
+ Step 34680/50462 | Loss: 2.5771 | LR: 1.14e-04 | 2731.5ms/step | 22.0 samples/s
306
+ Step 34690/50462 | Loss: 2.5258 | LR: 1.14e-04 | 2727.7ms/step | 22.0 samples/s
307
+ Step 34700/50462 | Loss: 2.5154 | LR: 1.14e-04 | 2737.7ms/step | 21.9 samples/s
308
+ Step 34710/50462 | Loss: 2.5414 | LR: 1.14e-04 | 2734.8ms/step | 21.9 samples/s
309
+ Step 34720/50462 | Loss: 2.5453 | LR: 1.14e-04 | 2730.7ms/step | 22.0 samples/s
310
+ Step 34730/50462 | Loss: 2.5258 | LR: 1.14e-04 | 2734.0ms/step | 21.9 samples/s
311
+ Step 34740/50462 | Loss: 2.5195 | LR: 1.14e-04 | 2731.4ms/step | 22.0 samples/s
312
+ Step 34750/50462 | Loss: 2.5358 | LR: 1.13e-04 | 2739.7ms/step | 21.9 samples/s
313
+ Validation at step 34750: Loss = 2.5385, Perplexity = 12.66
314
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-34750
315
+ Step 34760/50462 | Loss: 2.5584 | LR: 1.13e-04 | 4106.4ms/step | 14.6 samples/s
316
+ Step 34770/50462 | Loss: 2.5253 | LR: 1.13e-04 | 2731.3ms/step | 22.0 samples/s
317
+ Step 34780/50462 | Loss: 2.5328 | LR: 1.13e-04 | 2721.7ms/step | 22.0 samples/s
318
+ Step 34790/50462 | Loss: 2.5376 | LR: 1.13e-04 | 2736.0ms/step | 21.9 samples/s
319
+ Step 34800/50462 | Loss: 2.5378 | LR: 1.13e-04 | 2734.3ms/step | 21.9 samples/s
320
+ Step 34810/50462 | Loss: 2.5314 | LR: 1.13e-04 | 2717.1ms/step | 22.1 samples/s
321
+ Step 34820/50462 | Loss: 2.5096 | LR: 1.13e-04 | 2739.7ms/step | 21.9 samples/s
322
+ Step 34830/50462 | Loss: 2.5431 | LR: 1.12e-04 | 2723.0ms/step | 22.0 samples/s
323
+ Step 34840/50462 | Loss: 2.5128 | LR: 1.12e-04 | 2733.3ms/step | 22.0 samples/s
324
+ Step 34850/50462 | Loss: 2.5493 | LR: 1.12e-04 | 2731.3ms/step | 22.0 samples/s
325
+ Step 34860/50462 | Loss: 2.5711 | LR: 1.12e-04 | 2732.1ms/step | 22.0 samples/s
326
+ Step 34870/50462 | Loss: 2.5636 | LR: 1.12e-04 | 2738.5ms/step | 21.9 samples/s
327
+ Step 34880/50462 | Loss: 2.5552 | LR: 1.12e-04 | 2731.2ms/step | 22.0 samples/s
328
+ Step 34890/50462 | Loss: 2.4976 | LR: 1.12e-04 | 2727.6ms/step | 22.0 samples/s
329
+ Step 34900/50462 | Loss: 2.5187 | LR: 1.11e-04 | 2732.7ms/step | 22.0 samples/s
330
+ Step 34910/50462 | Loss: 2.5516 | LR: 1.11e-04 | 2730.5ms/step | 22.0 samples/s
331
+ Step 34920/50462 | Loss: 2.5359 | LR: 1.11e-04 | 2739.8ms/step | 21.9 samples/s
332
+ Step 34930/50462 | Loss: 2.5522 | LR: 1.11e-04 | 2740.7ms/step | 21.9 samples/s
333
+ Step 34940/50462 | Loss: 2.5421 | LR: 1.11e-04 | 2730.4ms/step | 22.0 samples/s
334
+ Step 34950/50462 | Loss: 2.5037 | LR: 1.11e-04 | 2740.9ms/step | 21.9 samples/s
335
+ Step 34960/50462 | Loss: 2.5385 | LR: 1.11e-04 | 2728.8ms/step | 22.0 samples/s
336
+ Step 34970/50462 | Loss: 2.5568 | LR: 1.11e-04 | 2735.8ms/step | 21.9 samples/s
337
+ Step 34980/50462 | Loss: 2.5153 | LR: 1.10e-04 | 2729.7ms/step | 22.0 samples/s
338
+ Step 34990/50462 | Loss: 2.5871 | LR: 1.10e-04 | 2730.4ms/step | 22.0 samples/s
339
+ Step 35000/50462 | Loss: 2.5608 | LR: 1.10e-04 | 2728.4ms/step | 22.0 samples/s
340
+ Validation at step 35000: Loss = 2.5383, Perplexity = 12.66
341
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-35000
342
+ Step 35010/50462 | Loss: 2.5467 | LR: 1.10e-04 | 4123.3ms/step | 14.6 samples/s
343
+ Step 35020/50462 | Loss: 2.5173 | LR: 1.10e-04 | 2739.9ms/step | 21.9 samples/s
344
+ Step 35030/50462 | Loss: 2.5690 | LR: 1.10e-04 | 2738.2ms/step | 21.9 samples/s
345
+ Step 35040/50462 | Loss: 2.5296 | LR: 1.10e-04 | 2728.9ms/step | 22.0 samples/s
346
+ Step 35050/50462 | Loss: 2.5198 | LR: 1.09e-04 | 2730.3ms/step | 22.0 samples/s
347
+ Step 35060/50462 | Loss: 2.5126 | LR: 1.09e-04 | 2735.7ms/step | 21.9 samples/s
348
+ Step 35070/50462 | Loss: 2.5346 | LR: 1.09e-04 | 2730.5ms/step | 22.0 samples/s
349
+ Step 35080/50462 | Loss: 2.5103 | LR: 1.09e-04 | 2732.4ms/step | 22.0 samples/s
350
+ Step 35090/50462 | Loss: 2.5597 | LR: 1.09e-04 | 2734.7ms/step | 21.9 samples/s
351
+ Step 35100/50462 | Loss: 2.5708 | LR: 1.09e-04 | 2731.6ms/step | 22.0 samples/s
352
+ Step 35110/50462 | Loss: 2.5626 | LR: 1.09e-04 | 2731.8ms/step | 22.0 samples/s
353
+ Step 35120/50462 | Loss: 2.5698 | LR: 1.09e-04 | 2741.2ms/step | 21.9 samples/s
354
+ Step 35130/50462 | Loss: 2.5458 | LR: 1.08e-04 | 2738.6ms/step | 21.9 samples/s
355
+ Step 35140/50462 | Loss: 2.5508 | LR: 1.08e-04 | 2724.9ms/step | 22.0 samples/s
356
+ Step 35150/50462 | Loss: 2.5787 | LR: 1.08e-04 | 2736.3ms/step | 21.9 samples/s
357
+ Step 35160/50462 | Loss: 2.5367 | LR: 1.08e-04 | 2754.9ms/step | 21.8 samples/s
358
+ Step 35170/50462 | Loss: 2.5561 | LR: 1.08e-04 | 2734.8ms/step | 21.9 samples/s
359
+ Step 35180/50462 | Loss: 2.5416 | LR: 1.08e-04 | 2732.2ms/step | 22.0 samples/s
360
+ Step 35190/50462 | Loss: 2.5311 | LR: 1.08e-04 | 2736.0ms/step | 21.9 samples/s
361
+ Step 35200/50462 | Loss: 2.5334 | LR: 1.08e-04 | 2741.7ms/step | 21.9 samples/s
362
+ Step 35210/50462 | Loss: 2.5394 | LR: 1.07e-04 | 2728.5ms/step | 22.0 samples/s
363
+ Step 35220/50462 | Loss: 2.5587 | LR: 1.07e-04 | 2731.5ms/step | 22.0 samples/s
364
+ Step 35230/50462 | Loss: 2.5640 | LR: 1.07e-04 | 2730.0ms/step | 22.0 samples/s
365
+ Step 35240/50462 | Loss: 2.5664 | LR: 1.07e-04 | 2731.1ms/step | 22.0 samples/s
366
+ Step 35250/50462 | Loss: 2.5404 | LR: 1.07e-04 | 2744.7ms/step | 21.9 samples/s
367
+ Validation at step 35250: Loss = 2.5394, Perplexity = 12.67
368
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-35250
369
+ Step 35260/50462 | Loss: 2.5307 | LR: 1.07e-04 | 4127.3ms/step | 14.5 samples/s
370
+ Step 35270/50462 | Loss: 2.5304 | LR: 1.07e-04 | 2731.9ms/step | 22.0 samples/s
371
+ Step 35280/50462 | Loss: 2.5314 | LR: 1.07e-04 | 2734.1ms/step | 21.9 samples/s
372
+ Step 35290/50462 | Loss: 2.5427 | LR: 1.06e-04 | 2730.8ms/step | 22.0 samples/s
373
+ Step 35300/50462 | Loss: 2.5254 | LR: 1.06e-04 | 2740.4ms/step | 21.9 samples/s
374
+ Step 35310/50462 | Loss: 2.5743 | LR: 1.06e-04 | 2735.0ms/step | 21.9 samples/s
375
+ Step 35320/50462 | Loss: 2.5612 | LR: 1.06e-04 | 2724.9ms/step | 22.0 samples/s
376
+ Step 35330/50462 | Loss: 2.5191 | LR: 1.06e-04 | 2739.8ms/step | 21.9 samples/s
377
+ Step 35340/50462 | Loss: 2.5555 | LR: 1.06e-04 | 2740.1ms/step | 21.9 samples/s
378
+ Step 35350/50462 | Loss: 2.5137 | LR: 1.06e-04 | 2740.2ms/step | 21.9 samples/s
379
+ Step 35360/50462 | Loss: 2.5804 | LR: 1.05e-04 | 2744.5ms/step | 21.9 samples/s
380
+ Step 35370/50462 | Loss: 2.5308 | LR: 1.05e-04 | 2732.1ms/step | 22.0 samples/s
381
+ Step 35380/50462 | Loss: 2.5730 | LR: 1.05e-04 | 2736.8ms/step | 21.9 samples/s
382
+ Step 35390/50462 | Loss: 2.5919 | LR: 1.05e-04 | 2746.9ms/step | 21.8 samples/s
383
+ Step 35400/50462 | Loss: 2.5692 | LR: 1.05e-04 | 2738.9ms/step | 21.9 samples/s
384
+ Step 35410/50462 | Loss: 2.5797 | LR: 1.05e-04 | 2733.7ms/step | 21.9 samples/s
385
+ Step 35420/50462 | Loss: 2.4925 | LR: 1.05e-04 | 2731.8ms/step | 22.0 samples/s
386
+ Step 35430/50462 | Loss: 2.5453 | LR: 1.05e-04 | 2734.7ms/step | 21.9 samples/s
387
+ Step 35440/50462 | Loss: 2.5308 | LR: 1.04e-04 | 2730.3ms/step | 22.0 samples/s
388
+ Step 35450/50462 | Loss: 2.5664 | LR: 1.04e-04 | 2725.9ms/step | 22.0 samples/s
389
+ Step 35460/50462 | Loss: 2.5462 | LR: 1.04e-04 | 2729.8ms/step | 22.0 samples/s
390
+ Step 35470/50462 | Loss: 2.5257 | LR: 1.04e-04 | 2728.9ms/step | 22.0 samples/s
391
+ Step 35480/50462 | Loss: 2.5453 | LR: 1.04e-04 | 2735.7ms/step | 21.9 samples/s
392
+ Step 35490/50462 | Loss: 2.5457 | LR: 1.04e-04 | 2734.8ms/step | 21.9 samples/s
393
+ Step 35500/50462 | Loss: 2.5566 | LR: 1.04e-04 | 2734.0ms/step | 21.9 samples/s
394
+ Validation at step 35500: Loss = 2.5359, Perplexity = 12.63
395
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-35500
396
+ Step 35510/50462 | Loss: 2.5122 | LR: 1.04e-04 | 4165.0ms/step | 14.4 samples/s
epfl_final/train_logs/p2_1p2b_mixed_seed70_repaired_resume35500_6gpu.log ADDED
@@ -0,0 +1,208 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ *****************************************
3
+ Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
4
+ *****************************************
5
+ Resuming: Using existing output directory /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557
6
+ Running config:
7
+ {
8
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
9
+ "model_dtype": "bfloat16",
10
+ "init_from_scratch": true,
11
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
12
+ "val_path": null,
13
+ "max_seq_length": 8192,
14
+ "num_workers": 8,
15
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557",
16
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-35500",
17
+ "force_new_run": false,
18
+ "num_epochs": 1,
19
+ "max_steps": 50462,
20
+ "per_device_batch_size": 1,
21
+ "gradient_accumulation_steps": 10,
22
+ "learning_rate": 0.0005,
23
+ "weight_decay": 0.01,
24
+ "warmup_steps": 750,
25
+ "lr_scheduler": "cosine",
26
+ "max_grad_norm": 1.0,
27
+ "save_steps": 250,
28
+ "eval_steps": 250,
29
+ "eval_batches": 50,
30
+ "logging_steps": 10,
31
+ "seed": 70,
32
+ "verbose": true,
33
+ "checkpoint_activations": true,
34
+ "resume_micro_step_override": null,
35
+ "strategy": "ddp",
36
+ "mixed_precision": "bf16",
37
+ "wandb_enabled": false,
38
+ "wandb_project": "stochastic_sparsity",
39
+ "wandb_name": "1.2b mixed 8k seed70 repaired resume35500",
40
+ "wandb_tags": [
41
+ "qwen3",
42
+ "1.2b",
43
+ "8k",
44
+ "p2",
45
+ "mixed",
46
+ "seed70",
47
+ "repaired"
48
+ ],
49
+ "hidden_size": 1536,
50
+ "intermediate_size": 6144,
51
+ "num_attention_heads": 12,
52
+ "num_key_value_heads": 4,
53
+ "num_hidden_layers": 28,
54
+ "head_dim": null,
55
+ "attention_dropout": null,
56
+ "rope_train_position_scale_set": null,
57
+ "attention_type": "sparse_stochastic",
58
+ "sparse_block_size": 128,
59
+ "sparse_num_random_blocks": 35,
60
+ "sparse_mask_pool_size": 10000,
61
+ "sparse_val_dense_compare": false,
62
+ "sparse_layer_indices": null,
63
+ "sparse_mask_cache_dir": "/mloscratch/homes/navasard/sparse/mask_cache_8k_mixed",
64
+ "sparse_mask_seed": 70,
65
+ "sparse_mixed_nrb_set": [
66
+ 63,
67
+ 47,
68
+ 31,
69
+ 21,
70
+ 11
71
+ ],
72
+ "dense_mode": false,
73
+ "dense_step_intervals": null
74
+ }
75
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
76
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-35500
77
+ Checkpoint activations: TrueCheckpoint activations: True
78
+ Checkpoint activations: True
79
+
80
+ Checkpoint activations: True
81
+ Checkpoint activations: True
82
+ Checkpoint activations: True
83
+
84
+
85
+ ================================================================================
86
+ MODEL ARCHITECTURE
87
+ ================================================================================
88
+ Model config:
89
+ vocab_size: 151936
90
+ hidden_size: 1536
91
+ num_hidden_layers: 28
92
+ num_attention_heads: 12
93
+ num_key_value_heads: 4
94
+ head_dim: 128
95
+ intermediate_size: 6144
96
+ max_position_embeddings: 32768
97
+ rope_theta: 1000000
98
+ rope_train_position_scale_set: None
99
+ attention_type: sparse_stochastic
100
+ sparse_block_size: 128
101
+ sparse_num_random_blocks: 35
102
+ sparse_mask_pool_size: 10000
103
+ sparse_max_seq_length: 8192
104
+
105
+ Parameter count:
106
+ Total parameters: 1,202,352,640
107
+ Trainable parameters: 1,202,352,640
108
+ Non-trainable parameters: 0
109
+ Model size (BF16): 2.40 GB
110
+
111
+ --------------------------------------------------------------------------------
112
+ LAYER-BY-LAYER SUMMARY
113
+ --------------------------------------------------------------------------------
114
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/nn/attention/flex_attention.py:1704: UserWarning: flex_attention called without torch.compile() - this will use an unfused implementation that materializes the full scores matrix instead of generating a fused kernel.
115
+
116
+ SOLUTION: Use torch.compile(flex_attention)(...)
117
+
118
+ If you want to debug your score_mod/mask_mod, you can set:
119
+ torch.nn.attention.flex_attention._FLEX_ATTENTION_DISABLE_COMPILE_DEBUG = True
120
+
121
+ This will allow you to use print statements or breakpoints. Note: This doesn't work with the backwards pass and may produce incorrect results.
122
+ _warn_once(
123
+ ================================================================================================================================================================
124
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
125
+ ================================================================================================================================================================
126
+ Qwen3ForCausalLM [1, 8192] -- -- True
127
+ ├─Qwen3Model: 1-1 -- [1, 8192, 1536] -- True
128
+ │ └─Embedding: 2-1 [1, 8192] [1, 8192, 1536] 233,373,696 True
129
+ │ └─ModuleList: 2-2 -- -- -- True
130
+ │ │ └─Qwen3DecoderLayer: 3-1 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
131
+ │ │ └─Qwen3DecoderLayer: 3-2 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
132
+ │ │ └─Qwen3DecoderLayer: 3-3 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
133
+ │ │ └─Qwen3DecoderLayer: 3-4 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
134
+ │ │ └─Qwen3DecoderLayer: 3-5 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
135
+ │ │ └─Qwen3DecoderLayer: 3-6 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
136
+ │ │ └─Qwen3DecoderLayer: 3-7 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
137
+ │ │ └─Qwen3DecoderLayer: 3-8 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
138
+ │ │ └─Qwen3DecoderLayer: 3-9 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
139
+ │ │ └─Qwen3DecoderLayer: 3-10 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
140
+ │ │ └─Qwen3DecoderLayer: 3-11 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
141
+ │ │ └─Qwen3DecoderLayer: 3-12 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
142
+ │ │ └─Qwen3DecoderLayer: 3-13 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
143
+ │ │ └─Qwen3DecoderLayer: 3-14 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
144
+ │ │ └─Qwen3DecoderLayer: 3-15 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
145
+ │ │ └─Qwen3DecoderLayer: 3-16 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
146
+ │ │ └─Qwen3DecoderLayer: 3-17 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
147
+ │ │ └─Qwen3DecoderLayer: 3-18 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
148
+ │ │ └─Qwen3DecoderLayer: 3-19 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
149
+ │ │ └─Qwen3DecoderLayer: 3-20 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
150
+ │ │ └─Qwen3DecoderLayer: 3-21 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
151
+ │ │ └─Qwen3DecoderLayer: 3-22 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
152
+ │ │ └─Qwen3DecoderLayer: 3-23 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
153
+ │ │ └─Qwen3DecoderLayer: 3-24 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
154
+ │ │ └─Qwen3DecoderLayer: 3-25 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
155
+ │ │ └─Qwen3DecoderLayer: 3-26 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
156
+ │ │ └─Qwen3DecoderLayer: 3-27 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
157
+ │ │ └─Qwen3DecoderLayer: 3-28 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
158
+ │ └─Qwen3RMSNorm: 2-3 [1, 8192, 1536] [1, 8192, 1536] 1,536 True
159
+ ├─Linear: 1-2 [1, 8192, 1536] [1, 8192, 151936] 233,373,696 True
160
+ ================================================================================================================================================================
161
+ Total params: 1,435,726,336
162
+ Trainable params: 1,435,726,336
163
+ Non-trainable params: 0
164
+ Total mult-adds (Units.GIGABYTES): 1.44
165
+ ================================================================================================================================================================
166
+ Input size (MB): 0.07
167
+ Forward/backward pass size (MB): 26218.59
168
+ Params size (MB): 2871.45
169
+ Estimated Total Size (MB): 29090.11
170
+ ================================================================================================================================================================
171
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (233,373,696 params). True unique parameter count: 1,202,352,640 (1202.4M)
172
+ ================================================================================
173
+
174
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
175
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
176
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
177
+ Using pre-tokenized dataset (no runtime tokenization)
178
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
179
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
180
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
181
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
182
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
183
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
184
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
185
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
186
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
187
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
188
+ Dataset split: 3027764 train, 159356 validation
189
+ Training examples (packed, train split): 3027764
190
+ Validation examples (packed, val split): 159356
191
+ Per-device batch size: 1
192
+ World size (GPUs): 6
193
+ Gradient accumulation steps: 10
194
+ Effective batch size (seqs/step): 60
195
+ Sequence length: 8192
196
+ Tokens per optimizer step: 491,520
197
+ Micro-batches per epoch (per rank): 504628
198
+ Optimizer steps per epoch: 50462
199
+ Total optimizer steps: 50462 (capped by max_steps; 1.00 epochs)
200
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-35500...
201
+ Preserving checkpoint LR state for exact resume: current=[0.00010367300430253871], schedule_base=[0.0005]
202
+ Resumed from checkpoint at step 35500 (epoch 0, micro_step 355000)
203
+ Resuming training from step 35500
204
+ Starting training...
205
+ Validation at step 35500: Loss = 2.5371, Perplexity = 12.64
206
+ Switching attention mode for step 35501: sparse_stochastic (configured sparse attention)
207
+ Step 35510/50462 | Loss: 2.5123 | LR: 1.04e-04 | 56080.5ms/step | 1.1 samples/s
208
+ Step 35520/50462 | Loss: 2.5403 | LR: 1.03e-04 | 2724.8ms/step | 22.0 samples/s
epfl_final/train_logs/p2_1p2b_mixed_seed70_repaired_resume35500_pod46.log ADDED
The diff for this file is too large to render. See raw diff
 
epfl_final/train_logs/p2_1p2b_mixed_seed70_repaired_resume49250_6gpu.log ADDED
@@ -0,0 +1,310 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ *****************************************
3
+ Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
4
+ *****************************************
5
+ Resuming: Using existing output directory /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557
6
+ Running config:
7
+ {
8
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
9
+ "model_dtype": "bfloat16",
10
+ "init_from_scratch": true,
11
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
12
+ "val_path": null,
13
+ "max_seq_length": 8192,
14
+ "num_workers": 8,
15
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557",
16
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-49250",
17
+ "force_new_run": false,
18
+ "num_epochs": 1,
19
+ "max_steps": 50462,
20
+ "per_device_batch_size": 1,
21
+ "gradient_accumulation_steps": 10,
22
+ "learning_rate": 0.0005,
23
+ "weight_decay": 0.01,
24
+ "warmup_steps": 750,
25
+ "lr_scheduler": "cosine",
26
+ "max_grad_norm": 1.0,
27
+ "save_steps": 250,
28
+ "eval_steps": 250,
29
+ "eval_batches": 50,
30
+ "logging_steps": 10,
31
+ "seed": 70,
32
+ "verbose": true,
33
+ "checkpoint_activations": true,
34
+ "resume_micro_step_override": null,
35
+ "strategy": "ddp",
36
+ "mixed_precision": "bf16",
37
+ "wandb_enabled": false,
38
+ "wandb_project": "stochastic_sparsity",
39
+ "wandb_name": "1.2b mixed 8k seed70 repaired resume49250",
40
+ "wandb_tags": [
41
+ "qwen3",
42
+ "1.2b",
43
+ "8k",
44
+ "p2",
45
+ "mixed",
46
+ "seed70",
47
+ "repaired"
48
+ ],
49
+ "hidden_size": 1536,
50
+ "intermediate_size": 6144,
51
+ "num_attention_heads": 12,
52
+ "num_key_value_heads": 4,
53
+ "num_hidden_layers": 28,
54
+ "head_dim": null,
55
+ "attention_dropout": null,
56
+ "rope_train_position_scale_set": null,
57
+ "attention_type": "sparse_stochastic",
58
+ "sparse_block_size": 128,
59
+ "sparse_num_random_blocks": 35,
60
+ "sparse_mask_pool_size": 10000,
61
+ "sparse_val_dense_compare": false,
62
+ "sparse_layer_indices": null,
63
+ "sparse_mask_cache_dir": "/mloscratch/homes/navasard/sparse/mask_cache_8k_mixed",
64
+ "sparse_mask_seed": 70,
65
+ "sparse_mixed_nrb_set": [
66
+ 63,
67
+ 47,
68
+ 31,
69
+ 21,
70
+ 11
71
+ ],
72
+ "dense_mode": false,
73
+ "dense_step_intervals": null
74
+ }
75
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
76
+ Checkpoint activations: TrueCheckpoint activations: TrueCheckpoint activations: True
77
+
78
+
79
+ Checkpoint activations: True
80
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-49250
81
+ Checkpoint activations: True
82
+ Checkpoint activations: True
83
+
84
+
85
+ ================================================================================
86
+ MODEL ARCHITECTURE
87
+ ================================================================================
88
+ Model config:
89
+ vocab_size: 151936
90
+ hidden_size: 1536
91
+ num_hidden_layers: 28
92
+ num_attention_heads: 12
93
+ num_key_value_heads: 4
94
+ head_dim: 128
95
+ intermediate_size: 6144
96
+ max_position_embeddings: 32768
97
+ rope_theta: 1000000
98
+ rope_train_position_scale_set: None
99
+ attention_type: sparse_stochastic
100
+ sparse_block_size: 128
101
+ sparse_num_random_blocks: 35
102
+ sparse_mask_pool_size: 10000
103
+ sparse_max_seq_length: 8192
104
+
105
+ Parameter count:
106
+ Total parameters: 1,202,352,640
107
+ Trainable parameters: 1,202,352,640
108
+ Non-trainable parameters: 0
109
+ Model size (BF16): 2.40 GB
110
+
111
+ --------------------------------------------------------------------------------
112
+ LAYER-BY-LAYER SUMMARY
113
+ --------------------------------------------------------------------------------
114
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/nn/attention/flex_attention.py:1704: UserWarning: flex_attention called without torch.compile() - this will use an unfused implementation that materializes the full scores matrix instead of generating a fused kernel.
115
+
116
+ SOLUTION: Use torch.compile(flex_attention)(...)
117
+
118
+ If you want to debug your score_mod/mask_mod, you can set:
119
+ torch.nn.attention.flex_attention._FLEX_ATTENTION_DISABLE_COMPILE_DEBUG = True
120
+
121
+ This will allow you to use print statements or breakpoints. Note: This doesn't work with the backwards pass and may produce incorrect results.
122
+ _warn_once(
123
+ ================================================================================================================================================================
124
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
125
+ ================================================================================================================================================================
126
+ Qwen3ForCausalLM [1, 8192] -- -- True
127
+ ├─Qwen3Model: 1-1 -- [1, 8192, 1536] -- True
128
+ │ └─Embedding: 2-1 [1, 8192] [1, 8192, 1536] 233,373,696 True
129
+ │ └─ModuleList: 2-2 -- -- -- True
130
+ │ │ └─Qwen3DecoderLayer: 3-1 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
131
+ │ │ └─Qwen3DecoderLayer: 3-2 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
132
+ │ │ └─Qwen3DecoderLayer: 3-3 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
133
+ │ │ └─Qwen3DecoderLayer: 3-4 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
134
+ │ │ └─Qwen3DecoderLayer: 3-5 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
135
+ │ │ └─Qwen3DecoderLayer: 3-6 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
136
+ │ │ └─Qwen3DecoderLayer: 3-7 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
137
+ │ │ └─Qwen3DecoderLayer: 3-8 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
138
+ │ │ └─Qwen3DecoderLayer: 3-9 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
139
+ │ │ └─Qwen3DecoderLayer: 3-10 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
140
+ │ │ └─Qwen3DecoderLayer: 3-11 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
141
+ │ │ └─Qwen3DecoderLayer: 3-12 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
142
+ │ │ └─Qwen3DecoderLayer: 3-13 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
143
+ │ │ └─Qwen3DecoderLayer: 3-14 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
144
+ │ │ └─Qwen3DecoderLayer: 3-15 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
145
+ │ │ └─Qwen3DecoderLayer: 3-16 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
146
+ │ │ └─Qwen3DecoderLayer: 3-17 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
147
+ │ │ └─Qwen3DecoderLayer: 3-18 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
148
+ │ │ └─Qwen3DecoderLayer: 3-19 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
149
+ │ │ └─Qwen3DecoderLayer: 3-20 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
150
+ │ │ └─Qwen3DecoderLayer: 3-21 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
151
+ │ │ └─Qwen3DecoderLayer: 3-22 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
152
+ │ │ └─Qwen3DecoderLayer: 3-23 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
153
+ │ │ └─Qwen3DecoderLayer: 3-24 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
154
+ │ │ └─Qwen3DecoderLayer: 3-25 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
155
+ │ │ └─Qwen3DecoderLayer: 3-26 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
156
+ │ │ └─Qwen3DecoderLayer: 3-27 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
157
+ │ │ └─Qwen3DecoderLayer: 3-28 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
158
+ │ └─Qwen3RMSNorm: 2-3 [1, 8192, 1536] [1, 8192, 1536] 1,536 True
159
+ ├─Linear: 1-2 [1, 8192, 1536] [1, 8192, 151936] 233,373,696 True
160
+ ================================================================================================================================================================
161
+ Total params: 1,435,726,336
162
+ Trainable params: 1,435,726,336
163
+ Non-trainable params: 0
164
+ Total mult-adds (Units.GIGABYTES): 1.44
165
+ ================================================================================================================================================================
166
+ Input size (MB): 0.07
167
+ Forward/backward pass size (MB): 26218.59
168
+ Params size (MB): 2871.45
169
+ Estimated Total Size (MB): 29090.11
170
+ ================================================================================================================================================================
171
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (233,373,696 params). True unique parameter count: 1,202,352,640 (1202.4M)
172
+ ================================================================================
173
+
174
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
175
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
176
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
177
+
178
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
179
+ Using pre-tokenized dataset (no runtime tokenization)
180
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
181
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
182
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
183
+
184
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
185
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
186
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
187
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
188
+ Dataset split: 3027764 train, 159356 validation
189
+ Training examples (packed, train split): 3027764
190
+ Validation examples (packed, val split): 159356
191
+ Per-device batch size: 1
192
+ World size (GPUs): 6
193
+ Gradient accumulation steps: 10
194
+ Effective batch size (seqs/step): 60
195
+ Sequence length: 8192
196
+ Tokens per optimizer step: 491,520
197
+ Micro-batches per epoch (per rank): 504628
198
+ Optimizer steps per epoch: 50462
199
+ Total optimizer steps: 50462 (capped by max_steps; 1.00 epochs)
200
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-49250...
201
+ Preserving checkpoint LR state for exact resume: current=[7.329598717795427e-07], schedule_base=[0.0005]
202
+ Resumed from checkpoint at step 49250 (epoch 0, micro_step 492500)
203
+ Resuming training from step 49250
204
+ Starting training...
205
+ Validation at step 49250: Loss = 2.5265, Perplexity = 12.51
206
+ Switching attention mode for step 49251: sparse_stochastic (configured sparse attention)
207
+ Step 49260/50462 | Loss: 2.5502 | LR: 7.21e-07 | 60542.8ms/step | 1.0 samples/s
208
+ Step 49270/50462 | Loss: 2.5046 | LR: 7.09e-07 | 2716.2ms/step | 22.1 samples/s
209
+ Step 49280/50462 | Loss: 2.5426 | LR: 6.97e-07 | 2715.4ms/step | 22.1 samples/s
210
+ Step 49290/50462 | Loss: 2.5617 | LR: 6.85e-07 | 2713.0ms/step | 22.1 samples/s
211
+ Step 49300/50462 | Loss: 2.5117 | LR: 6.74e-07 | 2713.5ms/step | 22.1 samples/s
212
+ Step 49310/50462 | Loss: 2.5848 | LR: 6.62e-07 | 2728.3ms/step | 22.0 samples/s
213
+ Step 49320/50462 | Loss: 2.5156 | LR: 6.51e-07 | 2714.3ms/step | 22.1 samples/s
214
+ Step 49330/50462 | Loss: 2.4870 | LR: 6.39e-07 | 2720.8ms/step | 22.1 samples/s
215
+ Step 49340/50462 | Loss: 2.5393 | LR: 6.28e-07 | 2723.8ms/step | 22.0 samples/s
216
+ Step 49350/50462 | Loss: 2.5357 | LR: 6.17e-07 | 2719.2ms/step | 22.1 samples/s
217
+ Step 49360/50462 | Loss: 2.5498 | LR: 6.06e-07 | 2717.4ms/step | 22.1 samples/s
218
+ Step 49370/50462 | Loss: 2.5523 | LR: 5.95e-07 | 2713.6ms/step | 22.1 samples/s
219
+ Step 49380/50462 | Loss: 2.4941 | LR: 5.84e-07 | 2716.4ms/step | 22.1 samples/s
220
+ Step 49390/50462 | Loss: 2.5116 | LR: 5.73e-07 | 2712.0ms/step | 22.1 samples/s
221
+ Step 49400/50462 | Loss: 2.5177 | LR: 5.63e-07 | 2717.1ms/step | 22.1 samples/s
222
+ Step 49410/50462 | Loss: 2.5133 | LR: 5.52e-07 | 2715.4ms/step | 22.1 samples/s
223
+ Step 49420/50462 | Loss: 2.5387 | LR: 5.42e-07 | 2717.4ms/step | 22.1 samples/s
224
+ Step 49430/50462 | Loss: 2.5739 | LR: 5.31e-07 | 2721.6ms/step | 22.0 samples/s
225
+ Step 49440/50462 | Loss: 2.5550 | LR: 5.21e-07 | 2720.2ms/step | 22.1 samples/s
226
+ Step 49450/50462 | Loss: 2.5484 | LR: 5.11e-07 | 2719.5ms/step | 22.1 samples/s
227
+ Step 49460/50462 | Loss: 2.5116 | LR: 5.01e-07 | 2714.0ms/step | 22.1 samples/s
228
+ Step 49470/50462 | Loss: 2.5549 | LR: 4.91e-07 | 2723.4ms/step | 22.0 samples/s
229
+ Step 49480/50462 | Loss: 2.5188 | LR: 4.81e-07 | 2721.8ms/step | 22.0 samples/s
230
+ Step 49490/50462 | Loss: 2.5168 | LR: 4.72e-07 | 2715.2ms/step | 22.1 samples/s
231
+ Step 49500/50462 | Loss: 2.5232 | LR: 4.62e-07 | 2719.5ms/step | 22.1 samples/s
232
+ Validation at step 49500: Loss = 2.5278, Perplexity = 12.53
233
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-49500
234
+ Step 49510/50462 | Loss: 2.5207 | LR: 4.52e-07 | 4109.6ms/step | 14.6 samples/s
235
+ Step 49520/50462 | Loss: 2.5645 | LR: 4.43e-07 | 2720.2ms/step | 22.1 samples/s
236
+ Step 49530/50462 | Loss: 2.5883 | LR: 4.34e-07 | 2724.0ms/step | 22.0 samples/s
237
+ Step 49540/50462 | Loss: 2.5576 | LR: 4.24e-07 | 2722.4ms/step | 22.0 samples/s
238
+ Step 49550/50462 | Loss: 2.4916 | LR: 4.15e-07 | 2721.2ms/step | 22.0 samples/s
239
+ Step 49560/50462 | Loss: 2.5499 | LR: 4.06e-07 | 2724.9ms/step | 22.0 samples/s
240
+ Step 49570/50462 | Loss: 2.5466 | LR: 3.97e-07 | 2723.7ms/step | 22.0 samples/s
241
+ Step 49580/50462 | Loss: 2.4421 | LR: 3.88e-07 | 2724.7ms/step | 22.0 samples/s
242
+ Step 49590/50462 | Loss: 2.5889 | LR: 3.79e-07 | 2724.4ms/step | 22.0 samples/s
243
+ Step 49600/50462 | Loss: 2.5172 | LR: 3.71e-07 | 2722.7ms/step | 22.0 samples/s
244
+ Step 49610/50462 | Loss: 2.5742 | LR: 3.62e-07 | 2723.2ms/step | 22.0 samples/s
245
+ Step 49620/50462 | Loss: 2.5402 | LR: 3.54e-07 | 2727.6ms/step | 22.0 samples/s
246
+ Step 49630/50462 | Loss: 2.5440 | LR: 3.45e-07 | 2722.8ms/step | 22.0 samples/s
247
+ Step 49640/50462 | Loss: 2.5382 | LR: 3.37e-07 | 2725.2ms/step | 22.0 samples/s
248
+ Step 49650/50462 | Loss: 2.5587 | LR: 3.29e-07 | 2724.5ms/step | 22.0 samples/s
249
+ Step 49660/50462 | Loss: 2.5242 | LR: 3.21e-07 | 2726.1ms/step | 22.0 samples/s
250
+ Step 49670/50462 | Loss: 2.5783 | LR: 3.13e-07 | 2722.6ms/step | 22.0 samples/s
251
+ Step 49680/50462 | Loss: 2.5206 | LR: 3.05e-07 | 2727.5ms/step | 22.0 samples/s
252
+ Step 49690/50462 | Loss: 2.5387 | LR: 2.97e-07 | 2718.8ms/step | 22.1 samples/s
253
+ Step 49700/50462 | Loss: 2.5098 | LR: 2.90e-07 | 2716.7ms/step | 22.1 samples/s
254
+ Step 49710/50462 | Loss: 2.5235 | LR: 2.82e-07 | 2720.5ms/step | 22.1 samples/s
255
+ Step 49720/50462 | Loss: 2.5107 | LR: 2.75e-07 | 2718.7ms/step | 22.1 samples/s
256
+ Step 49730/50462 | Loss: 2.5483 | LR: 2.67e-07 | 2726.2ms/step | 22.0 samples/s
257
+ Step 49740/50462 | Loss: 2.5039 | LR: 2.60e-07 | 2720.2ms/step | 22.1 samples/s
258
+ Step 49750/50462 | Loss: 2.5121 | LR: 2.53e-07 | 2721.1ms/step | 22.0 samples/s
259
+ Validation at step 49750: Loss = 2.5268, Perplexity = 12.51
260
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-49750
261
+ Step 49760/50462 | Loss: 2.5655 | LR: 2.46e-07 | 4115.6ms/step | 14.6 samples/s
262
+ Step 49770/50462 | Loss: 2.5182 | LR: 2.39e-07 | 2723.1ms/step | 22.0 samples/s
263
+ Step 49780/50462 | Loss: 2.5419 | LR: 2.32e-07 | 2724.4ms/step | 22.0 samples/s
264
+ Step 49790/50462 | Loss: 2.5284 | LR: 2.25e-07 | 2725.9ms/step | 22.0 samples/s
265
+ Step 49800/50462 | Loss: 2.5086 | LR: 2.19e-07 | 2731.6ms/step | 22.0 samples/s
266
+ Step 49810/50462 | Loss: 2.5164 | LR: 2.12e-07 | 2716.3ms/step | 22.1 samples/s
267
+ Step 49820/50462 | Loss: 2.5540 | LR: 2.06e-07 | 2720.5ms/step | 22.1 samples/s
268
+ Step 49830/50462 | Loss: 2.4912 | LR: 1.99e-07 | 2725.0ms/step | 22.0 samples/s
269
+ Step 49840/50462 | Loss: 2.5197 | LR: 1.93e-07 | 2717.4ms/step | 22.1 samples/s
270
+ Step 49850/50462 | Loss: 2.5432 | LR: 1.87e-07 | 2724.4ms/step | 22.0 samples/s
271
+ Step 49860/50462 | Loss: 2.5363 | LR: 1.81e-07 | 2712.0ms/step | 22.1 samples/s
272
+ Step 49870/50462 | Loss: 2.5077 | LR: 1.75e-07 | 2723.6ms/step | 22.0 samples/s
273
+ Step 49880/50462 | Loss: 2.5234 | LR: 1.69e-07 | 2710.5ms/step | 22.1 samples/s
274
+ Step 49890/50462 | Loss: 2.5352 | LR: 1.63e-07 | 2724.8ms/step | 22.0 samples/s
275
+ Step 49900/50462 | Loss: 2.5425 | LR: 1.58e-07 | 2722.3ms/step | 22.0 samples/s
276
+ Step 49910/50462 | Loss: 2.5370 | LR: 1.52e-07 | 2728.7ms/step | 22.0 samples/s
277
+ Step 49920/50462 | Loss: 2.5319 | LR: 1.47e-07 | 2721.2ms/step | 22.0 samples/s
278
+ Step 49930/50462 | Loss: 2.4920 | LR: 1.41e-07 | 2744.2ms/step | 21.9 samples/s
279
+ Step 49940/50462 | Loss: 2.5003 | LR: 1.36e-07 | 2733.5ms/step | 22.0 samples/s
280
+ Step 49950/50462 | Loss: 2.5189 | LR: 1.31e-07 | 2729.8ms/step | 22.0 samples/s
281
+ Step 49960/50462 | Loss: 2.5093 | LR: 1.26e-07 | 2738.3ms/step | 21.9 samples/s
282
+ Step 49970/50462 | Loss: 2.5494 | LR: 1.21e-07 | 2731.0ms/step | 22.0 samples/s
283
+ Step 49980/50462 | Loss: 2.5429 | LR: 1.16e-07 | 2732.5ms/step | 22.0 samples/s
284
+ Step 49990/50462 | Loss: 2.5438 | LR: 1.11e-07 | 2731.4ms/step | 22.0 samples/s
285
+ Step 50000/50462 | Loss: 2.5543 | LR: 1.07e-07 | 2745.7ms/step | 21.9 samples/s
286
+ Validation at step 50000: Loss = 2.5275, Perplexity = 12.52
287
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-50000
288
+ Step 50010/50462 | Loss: 2.5265 | LR: 1.02e-07 | 4167.6ms/step | 14.4 samples/s
289
+ Step 50020/50462 | Loss: 2.5202 | LR: 9.75e-08 | 2728.1ms/step | 22.0 samples/s
290
+ Step 50030/50462 | Loss: 2.5561 | LR: 9.32e-08 | 2721.3ms/step | 22.0 samples/s
291
+ Step 50040/50462 | Loss: 2.5413 | LR: 8.89e-08 | 2726.1ms/step | 22.0 samples/s
292
+ Step 50050/50462 | Loss: 2.5734 | LR: 8.47e-08 | 2722.3ms/step | 22.0 samples/s
293
+ Step 50060/50462 | Loss: 2.5482 | LR: 8.07e-08 | 2734.9ms/step | 21.9 samples/s
294
+ Step 50070/50462 | Loss: 2.5446 | LR: 7.67e-08 | 2719.2ms/step | 22.1 samples/s
295
+ Step 50080/50462 | Loss: 2.5118 | LR: 7.28e-08 | 2727.5ms/step | 22.0 samples/s
296
+ Step 50090/50462 | Loss: 2.5393 | LR: 6.91e-08 | 2722.9ms/step | 22.0 samples/s
297
+ Step 50100/50462 | Loss: 2.5202 | LR: 6.54e-08 | 2717.8ms/step | 22.1 samples/s
298
+ Step 50110/50462 | Loss: 2.5335 | LR: 6.19e-08 | 2729.1ms/step | 22.0 samples/s
299
+ Step 50120/50462 | Loss: 2.4926 | LR: 5.84e-08 | 2733.1ms/step | 22.0 samples/s
300
+ Step 50130/50462 | Loss: 2.4960 | LR: 5.50e-08 | 2726.2ms/step | 22.0 samples/s
301
+ Step 50140/50462 | Loss: 2.5266 | LR: 5.18e-08 | 2728.5ms/step | 22.0 samples/s
302
+ Step 50150/50462 | Loss: 2.4959 | LR: 4.86e-08 | 2727.6ms/step | 22.0 samples/s
303
+ Step 50160/50462 | Loss: 2.5119 | LR: 4.55e-08 | 2733.1ms/step | 22.0 samples/s
304
+ Step 50170/50462 | Loss: 2.5458 | LR: 4.26e-08 | 2731.3ms/step | 22.0 samples/s
305
+ Step 50180/50462 | Loss: 2.5197 | LR: 3.97e-08 | 2726.7ms/step | 22.0 samples/s
306
+ Step 50190/50462 | Loss: 2.5388 | LR: 3.69e-08 | 2735.8ms/step | 21.9 samples/s
307
+ Step 50200/50462 | Loss: 2.5461 | LR: 3.43e-08 | 2731.8ms/step | 22.0 samples/s
308
+ Step 50210/50462 | Loss: 2.5258 | LR: 3.17e-08 | 2738.0ms/step | 21.9 samples/s
309
+ Step 50220/50462 | Loss: 2.5085 | LR: 2.92e-08 | 2726.6ms/step | 22.0 samples/s
310
+ Step 50230/50462 | Loss: 2.5312 | LR: 2.69e-08 | 2724.1ms/step | 22.0 samples/s
epfl_final/train_logs/p2_1p2b_mixed_seed70_repaired_resume50000_6gpu.log ADDED
@@ -0,0 +1,256 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ *****************************************
3
+ Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
4
+ *****************************************
5
+ Resuming: Using existing output directory /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557
6
+ Running config:
7
+ {
8
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
9
+ "model_dtype": "bfloat16",
10
+ "init_from_scratch": true,
11
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
12
+ "val_path": null,
13
+ "max_seq_length": 8192,
14
+ "num_workers": 8,
15
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557",
16
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-50000",
17
+ "force_new_run": false,
18
+ "num_epochs": 1,
19
+ "max_steps": 50462,
20
+ "per_device_batch_size": 1,
21
+ "gradient_accumulation_steps": 10,
22
+ "learning_rate": 0.0005,
23
+ "weight_decay": 0.01,
24
+ "warmup_steps": 750,
25
+ "lr_scheduler": "cosine",
26
+ "max_grad_norm": 1.0,
27
+ "save_steps": 250,
28
+ "eval_steps": 250,
29
+ "eval_batches": 50,
30
+ "logging_steps": 10,
31
+ "seed": 70,
32
+ "verbose": true,
33
+ "checkpoint_activations": true,
34
+ "resume_micro_step_override": null,
35
+ "strategy": "ddp",
36
+ "mixed_precision": "bf16",
37
+ "wandb_enabled": false,
38
+ "wandb_project": "stochastic_sparsity",
39
+ "wandb_name": "1.2b mixed 8k seed70 repaired resume50000",
40
+ "wandb_tags": [
41
+ "qwen3",
42
+ "1.2b",
43
+ "8k",
44
+ "p2",
45
+ "mixed",
46
+ "seed70",
47
+ "repaired"
48
+ ],
49
+ "hidden_size": 1536,
50
+ "intermediate_size": 6144,
51
+ "num_attention_heads": 12,
52
+ "num_key_value_heads": 4,
53
+ "num_hidden_layers": 28,
54
+ "head_dim": null,
55
+ "attention_dropout": null,
56
+ "rope_train_position_scale_set": null,
57
+ "attention_type": "sparse_stochastic",
58
+ "sparse_block_size": 128,
59
+ "sparse_num_random_blocks": 35,
60
+ "sparse_mask_pool_size": 10000,
61
+ "sparse_val_dense_compare": false,
62
+ "sparse_layer_indices": null,
63
+ "sparse_mask_cache_dir": "/mloscratch/homes/navasard/sparse/mask_cache_8k_mixed",
64
+ "sparse_mask_seed": 70,
65
+ "sparse_mixed_nrb_set": [
66
+ 63,
67
+ 47,
68
+ 31,
69
+ 21,
70
+ 11
71
+ ],
72
+ "dense_mode": false,
73
+ "dense_step_intervals": null
74
+ }
75
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
76
+ Checkpoint activations: True
77
+ Checkpoint activations: True
78
+ Checkpoint activations: True
79
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-50000
80
+ Checkpoint activations: True
81
+ Checkpoint activations: True
82
+ Checkpoint activations: True
83
+
84
+
85
+ ================================================================================
86
+ MODEL ARCHITECTURE
87
+ ================================================================================
88
+ Model config:
89
+ vocab_size: 151936
90
+ hidden_size: 1536
91
+ num_hidden_layers: 28
92
+ num_attention_heads: 12
93
+ num_key_value_heads: 4
94
+ head_dim: 128
95
+ intermediate_size: 6144
96
+ max_position_embeddings: 32768
97
+ rope_theta: 1000000
98
+ rope_train_position_scale_set: None
99
+ attention_type: sparse_stochastic
100
+ sparse_block_size: 128
101
+ sparse_num_random_blocks: 35
102
+ sparse_mask_pool_size: 10000
103
+ sparse_max_seq_length: 8192
104
+
105
+ Parameter count:
106
+ Total parameters: 1,202,352,640
107
+ Trainable parameters: 1,202,352,640
108
+ Non-trainable parameters: 0
109
+ Model size (BF16): 2.40 GB
110
+
111
+ --------------------------------------------------------------------------------
112
+ LAYER-BY-LAYER SUMMARY
113
+ --------------------------------------------------------------------------------
114
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/nn/attention/flex_attention.py:1704: UserWarning: flex_attention called without torch.compile() - this will use an unfused implementation that materializes the full scores matrix instead of generating a fused kernel.
115
+
116
+ SOLUTION: Use torch.compile(flex_attention)(...)
117
+
118
+ If you want to debug your score_mod/mask_mod, you can set:
119
+ torch.nn.attention.flex_attention._FLEX_ATTENTION_DISABLE_COMPILE_DEBUG = True
120
+
121
+ This will allow you to use print statements or breakpoints. Note: This doesn't work with the backwards pass and may produce incorrect results.
122
+ _warn_once(
123
+ ================================================================================================================================================================
124
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
125
+ ================================================================================================================================================================
126
+ Qwen3ForCausalLM [1, 8192] -- -- True
127
+ ├─Qwen3Model: 1-1 -- [1, 8192, 1536] -- True
128
+ │ └─Embedding: 2-1 [1, 8192] [1, 8192, 1536] 233,373,696 True
129
+ │ └─ModuleList: 2-2 -- -- -- True
130
+ │ │ └─Qwen3DecoderLayer: 3-1 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
131
+ │ │ └─Qwen3DecoderLayer: 3-2 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
132
+ │ │ └─Qwen3DecoderLayer: 3-3 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
133
+ │ │ └─Qwen3DecoderLayer: 3-4 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
134
+ │ │ └─Qwen3DecoderLayer: 3-5 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
135
+ │ │ └─Qwen3DecoderLayer: 3-6 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
136
+ │ │ └─Qwen3DecoderLayer: 3-7 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
137
+ │ │ └─Qwen3DecoderLayer: 3-8 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
138
+ │ │ └─Qwen3DecoderLayer: 3-9 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
139
+ │ │ └─Qwen3DecoderLayer: 3-10 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
140
+ │ │ └─Qwen3DecoderLayer: 3-11 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
141
+ │ │ └─Qwen3DecoderLayer: 3-12 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
142
+ │ │ └─Qwen3DecoderLayer: 3-13 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
143
+ │ │ └─Qwen3DecoderLayer: 3-14 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
144
+ │ │ └─Qwen3DecoderLayer: 3-15 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
145
+ │ │ └─Qwen3DecoderLayer: 3-16 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
146
+ │ │ └─Qwen3DecoderLayer: 3-17 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
147
+ │ │ └─Qwen3DecoderLayer: 3-18 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
148
+ │ │ └─Qwen3DecoderLayer: 3-19 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
149
+ │ │ └─Qwen3DecoderLayer: 3-20 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
150
+ │ │ └─Qwen3DecoderLayer: 3-21 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
151
+ │ │ └─Qwen3DecoderLayer: 3-22 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
152
+ │ │ └─Qwen3DecoderLayer: 3-23 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
153
+ │ │ └─Qwen3DecoderLayer: 3-24 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
154
+ │ │ └─Qwen3DecoderLayer: 3-25 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
155
+ │ │ └─Qwen3DecoderLayer: 3-26 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
156
+ │ │ └─Qwen3DecoderLayer: 3-27 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
157
+ │ │ └─Qwen3DecoderLayer: 3-28 [1, 8192, 1536] [1, 8192, 1536] 34,606,336 True
158
+ │ └─Qwen3RMSNorm: 2-3 [1, 8192, 1536] [1, 8192, 1536] 1,536 True
159
+ ├─Linear: 1-2 [1, 8192, 1536] [1, 8192, 151936] 233,373,696 True
160
+ ================================================================================================================================================================
161
+ Total params: 1,435,726,336
162
+ Trainable params: 1,435,726,336
163
+ Non-trainable params: 0
164
+ Total mult-adds (Units.GIGABYTES): 1.44
165
+ ================================================================================================================================================================
166
+ Input size (MB): 0.07
167
+ Forward/backward pass size (MB): 26218.59
168
+ Params size (MB): 2871.45
169
+ Estimated Total Size (MB): 29090.11
170
+ ================================================================================================================================================================
171
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (233,373,696 params). True unique parameter count: 1,202,352,640 (1202.4M)
172
+ ================================================================================
173
+
174
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
175
+ Using pre-tokenized dataset (no runtime tokenization)
176
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
177
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
178
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
179
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
180
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
181
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
182
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
183
+
184
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
185
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
186
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
187
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
188
+ Dataset split: 3027764 train, 159356 validation
189
+ Training examples (packed, train split): 3027764
190
+ Validation examples (packed, val split): 159356
191
+ Per-device batch size: 1
192
+ World size (GPUs): 6
193
+ Gradient accumulation steps: 10
194
+ Effective batch size (seqs/step): 60
195
+ Sequence length: 8192
196
+ Tokens per optimizer step: 491,520
197
+ Micro-batches per epoch (per rank): 504628
198
+ Optimizer steps per epoch: 50462
199
+ Total optimizer steps: 50462 (capped by max_steps; 1.00 epochs)
200
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-50000...
201
+ Preserving checkpoint LR state for exact resume: current=[1.0654679412810953e-07], schedule_base=[0.0005]
202
+ Resumed from checkpoint at step 50000 (epoch 0, micro_step 500000)
203
+ Resuming training from step 50000
204
+ Starting training...
205
+ Validation at step 50000: Loss = 2.5266, Perplexity = 12.51
206
+ Switching attention mode for step 50001: sparse_stochastic (configured sparse attention)
207
+ Step 50010/50462 | Loss: 2.5273 | LR: 1.02e-07 | 64935.1ms/step | 0.9 samples/s
208
+ Step 50020/50462 | Loss: 2.5219 | LR: 9.75e-08 | 2720.4ms/step | 22.1 samples/s
209
+ Step 50030/50462 | Loss: 2.5553 | LR: 9.32e-08 | 2724.2ms/step | 22.0 samples/s
210
+ Step 50040/50462 | Loss: 2.5397 | LR: 8.89e-08 | 2720.4ms/step | 22.1 samples/s
211
+ Step 50050/50462 | Loss: 2.5730 | LR: 8.47e-08 | 2738.4ms/step | 21.9 samples/s
212
+ Step 50060/50462 | Loss: 2.5485 | LR: 8.07e-08 | 2721.5ms/step | 22.0 samples/s
213
+ Step 50070/50462 | Loss: 2.5446 | LR: 7.67e-08 | 2724.3ms/step | 22.0 samples/s
214
+ Step 50080/50462 | Loss: 2.5125 | LR: 7.28e-08 | 2723.5ms/step | 22.0 samples/s
215
+ Step 50090/50462 | Loss: 2.5384 | LR: 6.91e-08 | 2719.8ms/step | 22.1 samples/s
216
+ Step 50100/50462 | Loss: 2.5209 | LR: 6.54e-08 | 2727.9ms/step | 22.0 samples/s
217
+ Step 50110/50462 | Loss: 2.5327 | LR: 6.19e-08 | 2731.2ms/step | 22.0 samples/s
218
+ Step 50120/50462 | Loss: 2.4927 | LR: 5.84e-08 | 2727.5ms/step | 22.0 samples/s
219
+ Step 50130/50462 | Loss: 2.4963 | LR: 5.50e-08 | 2726.9ms/step | 22.0 samples/s
220
+ Step 50140/50462 | Loss: 2.5261 | LR: 5.18e-08 | 2727.8ms/step | 22.0 samples/s
221
+ Step 50150/50462 | Loss: 2.4961 | LR: 4.86e-08 | 2732.9ms/step | 22.0 samples/s
222
+ Step 50160/50462 | Loss: 2.5112 | LR: 4.55e-08 | 2727.5ms/step | 22.0 samples/s
223
+ Step 50170/50462 | Loss: 2.5462 | LR: 4.26e-08 | 2730.1ms/step | 22.0 samples/s
224
+ Step 50180/50462 | Loss: 2.5206 | LR: 3.97e-08 | 2730.0ms/step | 22.0 samples/s
225
+ Step 50190/50462 | Loss: 2.5375 | LR: 3.69e-08 | 2730.4ms/step | 22.0 samples/s
226
+ Step 50200/50462 | Loss: 2.5454 | LR: 3.43e-08 | 2728.8ms/step | 22.0 samples/s
227
+ Step 50210/50462 | Loss: 2.5274 | LR: 3.17e-08 | 2722.4ms/step | 22.0 samples/s
228
+ Step 50220/50462 | Loss: 2.5085 | LR: 2.92e-08 | 2722.3ms/step | 22.0 samples/s
229
+ Step 50230/50462 | Loss: 2.5315 | LR: 2.69e-08 | 2723.6ms/step | 22.0 samples/s
230
+ Step 50240/50462 | Loss: 2.5373 | LR: 2.46e-08 | 2724.3ms/step | 22.0 samples/s
231
+ Step 50250/50462 | Loss: 2.5141 | LR: 2.24e-08 | 2729.2ms/step | 22.0 samples/s
232
+ Validation at step 50250: Loss = 2.5263, Perplexity = 12.51
233
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-50250
234
+ Step 50260/50462 | Loss: 2.5046 | LR: 2.04e-08 | 4093.2ms/step | 14.7 samples/s
235
+ Step 50270/50462 | Loss: 2.5177 | LR: 1.84e-08 | 2735.2ms/step | 21.9 samples/s
236
+ Step 50280/50462 | Loss: 2.5188 | LR: 1.65e-08 | 2730.4ms/step | 22.0 samples/s
237
+ Step 50290/50462 | Loss: 2.5564 | LR: 1.48e-08 | 2738.2ms/step | 21.9 samples/s
238
+ Step 50300/50462 | Loss: 2.5350 | LR: 1.31e-08 | 2726.6ms/step | 22.0 samples/s
239
+ Step 50310/50462 | Loss: 2.5301 | LR: 1.15e-08 | 2728.9ms/step | 22.0 samples/s
240
+ Step 50320/50462 | Loss: 2.5350 | LR: 1.01e-08 | 2731.1ms/step | 22.0 samples/s
241
+ Step 50330/50462 | Loss: 2.5111 | LR: 8.70e-09 | 2728.9ms/step | 22.0 samples/s
242
+ Step 50340/50462 | Loss: 2.5329 | LR: 7.43e-09 | 2733.1ms/step | 22.0 samples/s
243
+ Step 50350/50462 | Loss: 2.5297 | LR: 6.26e-09 | 2727.2ms/step | 22.0 samples/s
244
+ Step 50360/50462 | Loss: 2.4891 | LR: 5.19e-09 | 2732.2ms/step | 22.0 samples/s
245
+ Step 50370/50462 | Loss: 2.5264 | LR: 4.23e-09 | 2726.6ms/step | 22.0 samples/s
246
+ Step 50380/50462 | Loss: 2.5141 | LR: 3.36e-09 | 2724.8ms/step | 22.0 samples/s
247
+ Step 50390/50462 | Loss: 2.5031 | LR: 2.59e-09 | 2729.4ms/step | 22.0 samples/s
248
+ Step 50400/50462 | Loss: 2.5148 | LR: 1.92e-09 | 2728.2ms/step | 22.0 samples/s
249
+ Step 50410/50462 | Loss: 2.5080 | LR: 1.35e-09 | 2733.9ms/step | 21.9 samples/s
250
+ Step 50420/50462 | Loss: 2.4847 | LR: 8.81e-10 | 2718.9ms/step | 22.1 samples/s
251
+ Step 50430/50462 | Loss: 2.5528 | LR: 5.11e-10 | 2736.3ms/step | 21.9 samples/s
252
+ Step 50440/50462 | Loss: 2.4466 | LR: 2.42e-10 | 2728.0ms/step | 22.0 samples/s
253
+ Step 50450/50462 | Loss: 2.5183 | LR: 7.19e-11 | 2719.5ms/step | 22.1 samples/s
254
+ Step 50460/50462 | Loss: 2.5300 | LR: 2.00e-12 | 2734.5ms/step | 21.9 samples/s
255
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired/run_20260909_054557/checkpoint-50462
256
+ Training complete!
epfl_final/train_logs/queue_seed70_mixed_matched_evals.log ADDED
File without changes
epfl_final/train_logs/queue_seed70_mixed_repaired_after_seed72.sh ADDED
@@ -0,0 +1,38 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env bash
2
+ set -euo pipefail
3
+
4
+ repo=/mloscratch/homes/navasard/sparse/SparseAtt
5
+ logs=/mloscratch/homes/navasard/sparse/train_logs
6
+ seed72_endpoint=/mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_dense_seed72_repaired/run_20260906_repair_from9250/checkpoint-50462
7
+ seed70_root=/mloscratch/homes/navasard/sparse/checkpoints/l8k1p2b_mixed_seed70_repaired
8
+ seed70_log="$logs/p2_1p2b_mixed_seed70_repaired_from_scratch_6gpu.log"
9
+ py=/mloscratch/homes/navasard/conda/bin/python3.13
10
+
11
+ # Start only after the active corrected Dense run reaches its intended endpoint.
12
+ while [[ ! -d "$seed72_endpoint" ]]; do
13
+ sleep 60
14
+ done
15
+
16
+ cd "$repo"
17
+ nohup env USER=navasard LOGNAME=navasard WANDB_MODE=disabled \
18
+ TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_navasard \
19
+ "$py" -m torch.distributed.run --nproc_per_node=6 train.py \
20
+ --config configs/p2_1p2b_mixed_seed70_repaired_from_scratch_6gpu.json \
21
+ > "$seed70_log" 2>&1 < /dev/null &
22
+ train_pid=$!
23
+ echo "$train_pid" > "$logs/p2_1p2b_mixed_seed70_repaired_from_scratch_6gpu.pid"
24
+
25
+ # Keep the four most recent recoverable checkpoints. This avoids exhausting the
26
+ # shared scratch volume while preserving at most 1,000 steps of replay on a preemption.
27
+ while kill -0 "$train_pid" 2>/dev/null; do
28
+ run_dir=$(find "$seed70_root" -mindepth 1 -maxdepth 1 -type d -name 'run_*' -printf '%T@ %p\n' | sort -n | tail -n 1 | cut -d' ' -f2-)
29
+ if [[ -n "$run_dir" ]]; then
30
+ mapfile -t checkpoints < <(find "$run_dir" -mindepth 1 -maxdepth 1 -type d -name 'checkpoint-*' -printf '%f\n' | sort -t- -k2n)
31
+ if (( ${#checkpoints[@]} > 4 )); then
32
+ for checkpoint in "${checkpoints[@]:0:${#checkpoints[@]}-4}"; do
33
+ rm -rf -- "$run_dir/$checkpoint"
34
+ done
35
+ fi
36
+ fi
37
+ sleep 300
38
+ done
epfl_final/train_logs/ropectl_a1p0_h100_resume/seed42.20260820T011048Z.log ADDED
@@ -0,0 +1,903 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Auto-resuming from latest checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-6000
2
+ Resuming: Using existing output directory /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735
3
+ Running config:
4
+ {
5
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
6
+ "model_dtype": "bfloat16",
7
+ "init_from_scratch": true,
8
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
9
+ "val_path": null,
10
+ "max_seq_length": 8192,
11
+ "num_workers": 8,
12
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735",
13
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-6000",
14
+ "force_new_run": false,
15
+ "num_epochs": 1,
16
+ "max_steps": 44000,
17
+ "per_device_batch_size": 2,
18
+ "gradient_accumulation_steps": 10,
19
+ "learning_rate": 0.0005,
20
+ "weight_decay": 0.01,
21
+ "warmup_steps": 750,
22
+ "lr_scheduler": "cosine",
23
+ "max_grad_norm": 1.0,
24
+ "save_steps": 1000,
25
+ "eval_steps": 1000,
26
+ "eval_batches": 50,
27
+ "logging_steps": 10,
28
+ "seed": 42,
29
+ "verbose": true,
30
+ "checkpoint_activations": false,
31
+ "strategy": "ddp",
32
+ "mixed_precision": "bf16",
33
+ "wandb_enabled": true,
34
+ "wandb_project": "stochastic_sparsity",
35
+ "wandb_name": "l8k365m_dense_ropectl_a1p0_seed42",
36
+ "wandb_tags": [
37
+ "qwen3",
38
+ "365m",
39
+ "8k",
40
+ "dense",
41
+ "rope-control"
42
+ ],
43
+ "hidden_size": 768,
44
+ "intermediate_size": 3072,
45
+ "num_attention_heads": 6,
46
+ "num_key_value_heads": 2,
47
+ "num_hidden_layers": 28,
48
+ "head_dim": null,
49
+ "attention_dropout": 0.0,
50
+ "rope_train_position_scale_set": [
51
+ 1.0,
52
+ 0.9896339488176222,
53
+ 0.9692159692159692,
54
+ 0.9420289855072463,
55
+ 0.8730158730158729
56
+ ],
57
+ "attention_type": "sparse_stochastic",
58
+ "sparse_block_size": 128,
59
+ "sparse_num_random_blocks": 31,
60
+ "sparse_mask_pool_size": 1,
61
+ "sparse_val_dense_compare": false,
62
+ "sparse_layer_indices": null,
63
+ "sparse_mask_cache_dir": null,
64
+ "sparse_mask_seed": 0,
65
+ "sparse_mixed_nrb_set": null,
66
+ "dense_mode": true,
67
+ "dense_step_intervals": null
68
+ }
69
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
70
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-6000
71
+ Checkpoint activations: False
72
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
73
+
74
+
75
+ ================================================================================
76
+ MODEL ARCHITECTURE
77
+ ================================================================================
78
+ Model config:
79
+ vocab_size: 151936
80
+ hidden_size: 768
81
+ num_hidden_layers: 28
82
+ num_attention_heads: 6
83
+ num_key_value_heads: 2
84
+ head_dim: 128
85
+ intermediate_size: 3072
86
+ max_position_embeddings: 32768
87
+ rope_theta: 1000000
88
+ rope_train_position_scale_set: [1.0, 0.9896339488176222, 0.9692159692159692, 0.9420289855072463, 0.8730158730158729]
89
+ attention_type: sparse_stochastic
90
+ sparse_block_size: 128
91
+ sparse_num_random_blocks: 31
92
+ sparse_mask_pool_size: 1
93
+ sparse_max_seq_length: 8192
94
+
95
+ Parameter count:
96
+ Total parameters: 358,958,848
97
+ Trainable parameters: 358,958,848
98
+ Non-trainable parameters: 0
99
+ Model size (BF16): 0.72 GB
100
+
101
+ --------------------------------------------------------------------------------
102
+ LAYER-BY-LAYER SUMMARY
103
+ --------------------------------------------------------------------------------
104
+ ================================================================================================================================================================
105
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
106
+ ================================================================================================================================================================
107
+ Qwen3ForCausalLM [2, 8192] -- -- True
108
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
109
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
110
+ │ └─ModuleList: 2-2 -- -- -- True
111
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
132
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
133
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
134
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
135
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
136
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
137
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
138
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
139
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
140
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
141
+ ================================================================================================================================================================
142
+ Total params: 475,645,696
143
+ Trainable params: 475,645,696
144
+ Non-trainable params: 0
145
+ Total mult-adds (Units.MEGABYTES): 951.29
146
+ ================================================================================================================================================================
147
+ Input size (MB): 0.13
148
+ Forward/backward pass size (MB): 31197.23
149
+ Params size (MB): 951.29
150
+ Estimated Total Size (MB): 32148.66
151
+ ================================================================================================================================================================
152
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
153
+ ================================================================================
154
+
155
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
156
+ Using pre-tokenized dataset (no runtime tokenization)
157
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
158
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
159
+ Dataset split: 3027764 train, 159356 validation
160
+ Training examples (packed, train split): 3027764
161
+ Validation examples (packed, val split): 159356
162
+ Per-device batch size: 2
163
+ World size (GPUs): 1
164
+ Gradient accumulation steps: 10
165
+ Effective batch size (seqs/step): 20
166
+ Sequence length: 8192
167
+ Tokens per optimizer step: 163,840
168
+ Micro-batches per epoch (per rank): 1513882
169
+ Optimizer steps per epoch: 151388
170
+ Total optimizer steps: 44000 (capped by max_steps; 0.29 epochs)
171
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-6000...
172
+ Preserving checkpoint LR state for exact resume: current=[0.0004820408177936365], schedule_base=[0.0005]
173
+ Resumed from checkpoint at step 6000 (epoch 0, micro_step 60000)
174
+ Resuming training from step 6000
175
+ wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from WANDB_API_KEY.
176
+ wandb: Currently logged in as: ruben-navasardyan4869 (ruben-navasardyan4869-epfl) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
177
+ wandb: setting up run jqstnvej
178
+ wandb: Tracking run with wandb version 0.26.1
179
+ wandb: Run data is saved locally in /mloscratch/homes/navasard/sparse/SparseAtt/wandb/run-20260820_011125-jqstnvej
180
+ wandb: Run `wandb offline` to turn off syncing.
181
+ wandb: Syncing run l8k365m_dense_ropectl_a1p0_seed42
182
+ wandb: ⭐️ View project at https://wandb.ai/ruben-navasardyan4869-epfl/stochastic_sparsity
183
+ wandb: 🚀 View run at https://wandb.ai/ruben-navasardyan4869-epfl/stochastic_sparsity/runs/jqstnvej
184
+ Starting training...
185
+ Validation at step 6000: Loss = 3.2502, Perplexity = 25.80
186
+ Switching attention mode for step 6001: dense (dense step interval)
187
+ Step 6010/44000 | Loss: 3.2337 | LR: 4.82e-04 | 16593.7ms/step | 1.2 samples/s
188
+ Step 6020/44000 | Loss: 3.2222 | LR: 4.82e-04 | 1399.8ms/step | 14.3 samples/s
189
+ Step 6030/44000 | Loss: 3.2076 | LR: 4.82e-04 | 1392.7ms/step | 14.4 samples/s
190
+ Step 6040/44000 | Loss: 3.2263 | LR: 4.82e-04 | 1387.6ms/step | 14.4 samples/s
191
+ Step 6050/44000 | Loss: 3.2102 | LR: 4.82e-04 | 1395.9ms/step | 14.3 samples/s
192
+ Step 6060/44000 | Loss: 3.2228 | LR: 4.82e-04 | 1393.7ms/step | 14.4 samples/s
193
+ Step 6070/44000 | Loss: 3.2344 | LR: 4.82e-04 | 1394.0ms/step | 14.3 samples/s
194
+ Step 6080/44000 | Loss: 3.2540 | LR: 4.81e-04 | 1400.3ms/step | 14.3 samples/s
195
+ Step 6090/44000 | Loss: 3.2133 | LR: 4.81e-04 | 1390.6ms/step | 14.4 samples/s
196
+ Step 6100/44000 | Loss: 3.2561 | LR: 4.81e-04 | 1388.6ms/step | 14.4 samples/s
197
+ Step 6110/44000 | Loss: 3.2216 | LR: 4.81e-04 | 1386.9ms/step | 14.4 samples/s
198
+ Step 6120/44000 | Loss: 3.2246 | LR: 4.81e-04 | 1388.8ms/step | 14.4 samples/s
199
+ Step 6130/44000 | Loss: 3.2400 | LR: 4.81e-04 | 1390.8ms/step | 14.4 samples/s
200
+ Step 6140/44000 | Loss: 3.2495 | LR: 4.81e-04 | 1394.3ms/step | 14.3 samples/s
201
+ Step 6150/44000 | Loss: 3.2510 | LR: 4.81e-04 | 1396.0ms/step | 14.3 samples/s
202
+ Step 6160/44000 | Loss: 3.1790 | LR: 4.81e-04 | 1396.0ms/step | 14.3 samples/s
203
+ Step 6170/44000 | Loss: 3.2164 | LR: 4.81e-04 | 1397.7ms/step | 14.3 samples/s
204
+ Step 6180/44000 | Loss: 3.2367 | LR: 4.81e-04 | 1390.2ms/step | 14.4 samples/s
205
+ Step 6190/44000 | Loss: 3.2409 | LR: 4.81e-04 | 1393.8ms/step | 14.3 samples/s
206
+ Step 6200/44000 | Loss: 3.2226 | LR: 4.81e-04 | 1388.0ms/step | 14.4 samples/s
207
+ Step 6210/44000 | Loss: 3.1901 | LR: 4.81e-04 | 1390.1ms/step | 14.4 samples/s
208
+ Step 6220/44000 | Loss: 3.2101 | LR: 4.81e-04 | 1394.0ms/step | 14.3 samples/s
209
+ Step 6230/44000 | Loss: 3.2405 | LR: 4.80e-04 | 1396.1ms/step | 14.3 samples/s
210
+ Step 6240/44000 | Loss: 3.2258 | LR: 4.80e-04 | 1398.1ms/step | 14.3 samples/s
211
+ Step 6250/44000 | Loss: 3.2335 | LR: 4.80e-04 | 1400.2ms/step | 14.3 samples/s
212
+ Step 6260/44000 | Loss: 3.2253 | LR: 4.80e-04 | 1399.8ms/step | 14.3 samples/s
213
+ Step 6270/44000 | Loss: 3.2269 | LR: 4.80e-04 | 1399.9ms/step | 14.3 samples/s
214
+ Step 6280/44000 | Loss: 3.2451 | LR: 4.80e-04 | 1396.0ms/step | 14.3 samples/s
215
+ Step 6290/44000 | Loss: 3.2305 | LR: 4.80e-04 | 1398.3ms/step | 14.3 samples/s
216
+ Step 6300/44000 | Loss: 3.2300 | LR: 4.80e-04 | 1398.9ms/step | 14.3 samples/s
217
+ Step 6310/44000 | Loss: 3.2220 | LR: 4.80e-04 | 1396.5ms/step | 14.3 samples/s
218
+ Step 6320/44000 | Loss: 3.2081 | LR: 4.80e-04 | 1391.7ms/step | 14.4 samples/s
219
+ Step 6330/44000 | Loss: 3.1989 | LR: 4.80e-04 | 1394.8ms/step | 14.3 samples/s
220
+ Step 6340/44000 | Loss: 3.1960 | LR: 4.80e-04 | 1396.0ms/step | 14.3 samples/s
221
+ Step 6350/44000 | Loss: 3.2265 | LR: 4.80e-04 | 1399.8ms/step | 14.3 samples/s
222
+ Step 6360/44000 | Loss: 3.2153 | LR: 4.80e-04 | 1400.4ms/step | 14.3 samples/s
223
+ Step 6370/44000 | Loss: 3.2195 | LR: 4.79e-04 | 1399.8ms/step | 14.3 samples/s
224
+ Step 6380/44000 | Loss: 3.1814 | LR: 4.79e-04 | 1400.1ms/step | 14.3 samples/s
225
+ Step 6390/44000 | Loss: 3.2188 | LR: 4.79e-04 | 1400.0ms/step | 14.3 samples/s
226
+ Step 6400/44000 | Loss: 3.2153 | LR: 4.79e-04 | 1398.7ms/step | 14.3 samples/s
227
+ Step 6410/44000 | Loss: 3.2190 | LR: 4.79e-04 | 1393.3ms/step | 14.4 samples/s
228
+ Step 6420/44000 | Loss: 3.2389 | LR: 4.79e-04 | 1396.1ms/step | 14.3 samples/s
229
+ Step 6430/44000 | Loss: 3.2000 | LR: 4.79e-04 | 1399.9ms/step | 14.3 samples/s
230
+ Step 6440/44000 | Loss: 3.2238 | LR: 4.79e-04 | 1400.0ms/step | 14.3 samples/s
231
+ Step 6450/44000 | Loss: 3.2301 | LR: 4.79e-04 | 1399.9ms/step | 14.3 samples/s
232
+ Step 6460/44000 | Loss: 3.2083 | LR: 4.79e-04 | 1398.3ms/step | 14.3 samples/s
233
+ Step 6470/44000 | Loss: 3.1976 | LR: 4.79e-04 | 1393.6ms/step | 14.4 samples/s
234
+ Step 6480/44000 | Loss: 3.2114 | LR: 4.79e-04 | 1398.2ms/step | 14.3 samples/s
235
+ Step 6490/44000 | Loss: 3.2181 | LR: 4.79e-04 | 1400.1ms/step | 14.3 samples/s
236
+ Step 6500/44000 | Loss: 3.2047 | LR: 4.79e-04 | 1399.9ms/step | 14.3 samples/s
237
+ Step 6510/44000 | Loss: 3.2207 | LR: 4.78e-04 | 1400.1ms/step | 14.3 samples/s
238
+ Step 6520/44000 | Loss: 3.2099 | LR: 4.78e-04 | 1399.8ms/step | 14.3 samples/s
239
+ Step 6530/44000 | Loss: 3.1820 | LR: 4.78e-04 | 1399.1ms/step | 14.3 samples/s
240
+ Step 6540/44000 | Loss: 3.2409 | LR: 4.78e-04 | 1392.6ms/step | 14.4 samples/s
241
+ Step 6550/44000 | Loss: 3.2100 | LR: 4.78e-04 | 1395.7ms/step | 14.3 samples/s
242
+ Step 6560/44000 | Loss: 3.1882 | LR: 4.78e-04 | 1398.5ms/step | 14.3 samples/s
243
+ Step 6570/44000 | Loss: 3.1988 | LR: 4.78e-04 | 1400.2ms/step | 14.3 samples/s
244
+ Step 6580/44000 | Loss: 3.2253 | LR: 4.78e-04 | 1400.0ms/step | 14.3 samples/s
245
+ Step 6590/44000 | Loss: 3.2185 | LR: 4.78e-04 | 1399.7ms/step | 14.3 samples/s
246
+ Step 6600/44000 | Loss: 3.2354 | LR: 4.78e-04 | 1400.3ms/step | 14.3 samples/s
247
+ Step 6610/44000 | Loss: 3.1961 | LR: 4.78e-04 | 1400.0ms/step | 14.3 samples/s
248
+ Step 6620/44000 | Loss: 3.2088 | LR: 4.78e-04 | 1399.9ms/step | 14.3 samples/s
249
+ Step 6630/44000 | Loss: 3.2174 | LR: 4.78e-04 | 1400.1ms/step | 14.3 samples/s
250
+ Step 6640/44000 | Loss: 3.2137 | LR: 4.77e-04 | 1391.5ms/step | 14.4 samples/s
251
+ Step 6650/44000 | Loss: 3.2045 | LR: 4.77e-04 | 1390.3ms/step | 14.4 samples/s
252
+ Step 6660/44000 | Loss: 3.2084 | LR: 4.77e-04 | 1397.8ms/step | 14.3 samples/s
253
+ Step 6670/44000 | Loss: 3.1901 | LR: 4.77e-04 | 1398.3ms/step | 14.3 samples/s
254
+ Step 6680/44000 | Loss: 3.1990 | LR: 4.77e-04 | 1394.0ms/step | 14.3 samples/s
255
+ Step 6690/44000 | Loss: 3.2042 | LR: 4.77e-04 | 1400.0ms/step | 14.3 samples/s
256
+ Step 6700/44000 | Loss: 3.2350 | LR: 4.77e-04 | 1397.9ms/step | 14.3 samples/s
257
+ Step 6710/44000 | Loss: 3.2056 | LR: 4.77e-04 | 1400.1ms/step | 14.3 samples/s
258
+ Step 6720/44000 | Loss: 3.2040 | LR: 4.77e-04 | 1398.0ms/step | 14.3 samples/s
259
+ Step 6730/44000 | Loss: 3.2183 | LR: 4.77e-04 | 1397.0ms/step | 14.3 samples/s
260
+ Step 6740/44000 | Loss: 3.2134 | LR: 4.77e-04 | 1395.1ms/step | 14.3 samples/s
261
+ Step 6750/44000 | Loss: 3.2227 | LR: 4.77e-04 | 1400.0ms/step | 14.3 samples/s
262
+ Step 6760/44000 | Loss: 3.2024 | LR: 4.77e-04 | 1396.0ms/step | 14.3 samples/s
263
+ Step 6770/44000 | Loss: 3.2099 | LR: 4.76e-04 | 1400.2ms/step | 14.3 samples/s
264
+ Step 6780/44000 | Loss: 3.2022 | LR: 4.76e-04 | 1397.9ms/step | 14.3 samples/s
265
+ Step 6790/44000 | Loss: 3.1931 | LR: 4.76e-04 | 1399.8ms/step | 14.3 samples/s
266
+ Step 6800/44000 | Loss: 3.2085 | LR: 4.76e-04 | 1389.2ms/step | 14.4 samples/s
267
+ Step 6810/44000 | Loss: 3.2231 | LR: 4.76e-04 | 1398.7ms/step | 14.3 samples/s
268
+ Step 6820/44000 | Loss: 3.2086 | LR: 4.76e-04 | 1398.4ms/step | 14.3 samples/s
269
+ Step 6830/44000 | Loss: 3.2095 | LR: 4.76e-04 | 1397.8ms/step | 14.3 samples/s
270
+ Step 6840/44000 | Loss: 3.1580 | LR: 4.76e-04 | 1393.5ms/step | 14.4 samples/s
271
+ Step 6850/44000 | Loss: 3.2156 | LR: 4.76e-04 | 1396.4ms/step | 14.3 samples/s
272
+ Step 6860/44000 | Loss: 3.1672 | LR: 4.76e-04 | 1398.2ms/step | 14.3 samples/s
273
+ Step 6870/44000 | Loss: 3.2054 | LR: 4.76e-04 | 1399.9ms/step | 14.3 samples/s
274
+ Step 6880/44000 | Loss: 3.1802 | LR: 4.76e-04 | 1388.8ms/step | 14.4 samples/s
275
+ Step 6890/44000 | Loss: 3.2096 | LR: 4.76e-04 | 1387.8ms/step | 14.4 samples/s
276
+ Step 6900/44000 | Loss: 3.2009 | LR: 4.75e-04 | 1389.5ms/step | 14.4 samples/s
277
+ Step 6910/44000 | Loss: 3.2194 | LR: 4.75e-04 | 1398.1ms/step | 14.3 samples/s
278
+ Step 6920/44000 | Loss: 3.2419 | LR: 4.75e-04 | 1399.7ms/step | 14.3 samples/s
279
+ Step 6930/44000 | Loss: 3.2081 | LR: 4.75e-04 | 1398.2ms/step | 14.3 samples/s
280
+ Step 6940/44000 | Loss: 3.1963 | LR: 4.75e-04 | 1398.1ms/step | 14.3 samples/s
281
+ Step 6950/44000 | Loss: 3.2057 | LR: 4.75e-04 | 1394.0ms/step | 14.3 samples/s
282
+ Step 6960/44000 | Loss: 3.2188 | LR: 4.75e-04 | 1394.9ms/step | 14.3 samples/s
283
+ Step 6970/44000 | Loss: 3.1951 | LR: 4.75e-04 | 1399.0ms/step | 14.3 samples/s
284
+ Step 6980/44000 | Loss: 3.1828 | LR: 4.75e-04 | 1395.9ms/step | 14.3 samples/s
285
+ Step 6990/44000 | Loss: 3.1797 | LR: 4.75e-04 | 1397.8ms/step | 14.3 samples/s
286
+ Step 7000/44000 | Loss: 3.2119 | LR: 4.75e-04 | 1390.0ms/step | 14.4 samples/s
287
+ Validation at step 7000: Loss = 3.2094, Perplexity = 24.76
288
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-7000
289
+ Step 7010/44000 | Loss: 3.1706 | LR: 4.75e-04 | 3885.9ms/step | 5.1 samples/s
290
+ Step 7020/44000 | Loss: 3.1734 | LR: 4.75e-04 | 1396.5ms/step | 14.3 samples/s
291
+ Step 7030/44000 | Loss: 3.2028 | LR: 4.74e-04 | 1399.8ms/step | 14.3 samples/s
292
+ Step 7040/44000 | Loss: 3.2086 | LR: 4.74e-04 | 1400.1ms/step | 14.3 samples/s
293
+ Step 7050/44000 | Loss: 3.1690 | LR: 4.74e-04 | 1400.2ms/step | 14.3 samples/s
294
+ Step 7060/44000 | Loss: 3.1921 | LR: 4.74e-04 | 1400.0ms/step | 14.3 samples/s
295
+ Step 7070/44000 | Loss: 3.2195 | LR: 4.74e-04 | 1400.0ms/step | 14.3 samples/s
296
+ Step 7080/44000 | Loss: 3.1948 | LR: 4.74e-04 | 1396.0ms/step | 14.3 samples/s
297
+ Step 7090/44000 | Loss: 3.1997 | LR: 4.74e-04 | 1400.0ms/step | 14.3 samples/s
298
+ Step 7100/44000 | Loss: 3.1714 | LR: 4.74e-04 | 1400.1ms/step | 14.3 samples/s
299
+ Step 7110/44000 | Loss: 3.2021 | LR: 4.74e-04 | 1400.1ms/step | 14.3 samples/s
300
+ Step 7120/44000 | Loss: 3.1926 | LR: 4.74e-04 | 1399.8ms/step | 14.3 samples/s
301
+ Step 7130/44000 | Loss: 3.1867 | LR: 4.74e-04 | 1398.4ms/step | 14.3 samples/s
302
+ Step 7140/44000 | Loss: 3.1698 | LR: 4.74e-04 | 1399.7ms/step | 14.3 samples/s
303
+ Step 7150/44000 | Loss: 3.1920 | LR: 4.73e-04 | 1397.9ms/step | 14.3 samples/s
304
+ Step 7160/44000 | Loss: 3.2068 | LR: 4.73e-04 | 1400.0ms/step | 14.3 samples/s
305
+ Step 7170/44000 | Loss: 3.2037 | LR: 4.73e-04 | 1400.1ms/step | 14.3 samples/s
306
+ Step 7180/44000 | Loss: 3.1530 | LR: 4.73e-04 | 1399.7ms/step | 14.3 samples/s
307
+ Step 7190/44000 | Loss: 3.1693 | LR: 4.73e-04 | 1398.1ms/step | 14.3 samples/s
308
+ Step 7200/44000 | Loss: 3.1628 | LR: 4.73e-04 | 1400.1ms/step | 14.3 samples/s
309
+ Step 7210/44000 | Loss: 3.1841 | LR: 4.73e-04 | 1397.8ms/step | 14.3 samples/s
310
+ Step 7220/44000 | Loss: 3.1798 | LR: 4.73e-04 | 1400.2ms/step | 14.3 samples/s
311
+ Step 7230/44000 | Loss: 3.1796 | LR: 4.73e-04 | 1400.2ms/step | 14.3 samples/s
312
+ Step 7240/44000 | Loss: 3.2007 | LR: 4.73e-04 | 1399.9ms/step | 14.3 samples/s
313
+ Step 7250/44000 | Loss: 3.1939 | LR: 4.73e-04 | 1400.1ms/step | 14.3 samples/s
314
+ Step 7260/44000 | Loss: 3.1762 | LR: 4.73e-04 | 1400.0ms/step | 14.3 samples/s
315
+ Step 7270/44000 | Loss: 3.1727 | LR: 4.72e-04 | 1399.8ms/step | 14.3 samples/s
316
+ Step 7280/44000 | Loss: 3.1652 | LR: 4.72e-04 | 1400.0ms/step | 14.3 samples/s
317
+ Step 7290/44000 | Loss: 3.2181 | LR: 4.72e-04 | 1399.9ms/step | 14.3 samples/s
318
+ Step 7300/44000 | Loss: 3.1833 | LR: 4.72e-04 | 1400.1ms/step | 14.3 samples/s
319
+ Step 7310/44000 | Loss: 3.1544 | LR: 4.72e-04 | 1400.1ms/step | 14.3 samples/s
320
+ Step 7320/44000 | Loss: 3.1951 | LR: 4.72e-04 | 1400.2ms/step | 14.3 samples/s
321
+ Step 7330/44000 | Loss: 3.1797 | LR: 4.72e-04 | 1399.6ms/step | 14.3 samples/s
322
+ Step 7340/44000 | Loss: 3.1718 | LR: 4.72e-04 | 1394.4ms/step | 14.3 samples/s
323
+ Step 7350/44000 | Loss: 3.1804 | LR: 4.72e-04 | 1395.9ms/step | 14.3 samples/s
324
+ Step 7360/44000 | Loss: 3.1765 | LR: 4.72e-04 | 1399.6ms/step | 14.3 samples/s
325
+ Step 7370/44000 | Loss: 3.2072 | LR: 4.72e-04 | 1400.4ms/step | 14.3 samples/s
326
+ Step 7380/44000 | Loss: 3.1890 | LR: 4.72e-04 | 1399.9ms/step | 14.3 samples/s
327
+ Step 7390/44000 | Loss: 3.2129 | LR: 4.71e-04 | 1400.0ms/step | 14.3 samples/s
328
+ Step 7400/44000 | Loss: 3.1746 | LR: 4.71e-04 | 1400.0ms/step | 14.3 samples/s
329
+ Step 7410/44000 | Loss: 3.1473 | LR: 4.71e-04 | 1400.0ms/step | 14.3 samples/s
330
+ Step 7420/44000 | Loss: 3.1881 | LR: 4.71e-04 | 1400.1ms/step | 14.3 samples/s
331
+ Step 7430/44000 | Loss: 3.1742 | LR: 4.71e-04 | 1399.7ms/step | 14.3 samples/s
332
+ Step 7440/44000 | Loss: 3.1965 | LR: 4.71e-04 | 1398.3ms/step | 14.3 samples/s
333
+ Step 7450/44000 | Loss: 3.2059 | LR: 4.71e-04 | 1399.9ms/step | 14.3 samples/s
334
+ Step 7460/44000 | Loss: 3.1509 | LR: 4.71e-04 | 1400.2ms/step | 14.3 samples/s
335
+ Step 7470/44000 | Loss: 3.1856 | LR: 4.71e-04 | 1400.1ms/step | 14.3 samples/s
336
+ Step 7480/44000 | Loss: 3.1633 | LR: 4.71e-04 | 1399.9ms/step | 14.3 samples/s
337
+ Step 7490/44000 | Loss: 3.1923 | LR: 4.71e-04 | 1400.0ms/step | 14.3 samples/s
338
+ Step 7500/44000 | Loss: 3.1684 | LR: 4.71e-04 | 1399.8ms/step | 14.3 samples/s
339
+ Step 7510/44000 | Loss: 3.1538 | LR: 4.70e-04 | 1400.1ms/step | 14.3 samples/s
340
+ Step 7520/44000 | Loss: 3.1845 | LR: 4.70e-04 | 1400.1ms/step | 14.3 samples/s
341
+ Step 7530/44000 | Loss: 3.1555 | LR: 4.70e-04 | 1400.1ms/step | 14.3 samples/s
342
+ Step 7540/44000 | Loss: 3.1816 | LR: 4.70e-04 | 1400.1ms/step | 14.3 samples/s
343
+ Step 7550/44000 | Loss: 3.2068 | LR: 4.70e-04 | 1399.6ms/step | 14.3 samples/s
344
+ Step 7560/44000 | Loss: 3.1942 | LR: 4.70e-04 | 1400.1ms/step | 14.3 samples/s
345
+ Step 7570/44000 | Loss: 3.1670 | LR: 4.70e-04 | 1400.0ms/step | 14.3 samples/s
346
+ Step 7580/44000 | Loss: 3.1666 | LR: 4.70e-04 | 1400.2ms/step | 14.3 samples/s
347
+ Step 7590/44000 | Loss: 3.1857 | LR: 4.70e-04 | 1400.0ms/step | 14.3 samples/s
348
+ Step 7600/44000 | Loss: 3.1729 | LR: 4.70e-04 | 1399.8ms/step | 14.3 samples/s
349
+ Step 7610/44000 | Loss: 3.1520 | LR: 4.70e-04 | 1399.9ms/step | 14.3 samples/s
350
+ Step 7620/44000 | Loss: 3.1661 | LR: 4.70e-04 | 1398.2ms/step | 14.3 samples/s
351
+ Step 7630/44000 | Loss: 3.1763 | LR: 4.69e-04 | 1398.0ms/step | 14.3 samples/s
352
+ Step 7640/44000 | Loss: 3.1813 | LR: 4.69e-04 | 1397.9ms/step | 14.3 samples/s
353
+ Step 7650/44000 | Loss: 3.1691 | LR: 4.69e-04 | 1393.9ms/step | 14.3 samples/s
354
+ Step 7660/44000 | Loss: 3.1849 | LR: 4.69e-04 | 1394.0ms/step | 14.3 samples/s
355
+ Step 7670/44000 | Loss: 3.1622 | LR: 4.69e-04 | 1398.0ms/step | 14.3 samples/s
356
+ Step 7680/44000 | Loss: 3.1633 | LR: 4.69e-04 | 1396.6ms/step | 14.3 samples/s
357
+ Step 7690/44000 | Loss: 3.1728 | LR: 4.69e-04 | 1399.6ms/step | 14.3 samples/s
358
+ Step 7700/44000 | Loss: 3.1444 | LR: 4.69e-04 | 1398.0ms/step | 14.3 samples/s
359
+ Step 7710/44000 | Loss: 3.1750 | LR: 4.69e-04 | 1398.0ms/step | 14.3 samples/s
360
+ Step 7720/44000 | Loss: 3.1923 | LR: 4.69e-04 | 1400.0ms/step | 14.3 samples/s
361
+ Step 7730/44000 | Loss: 3.2134 | LR: 4.69e-04 | 1398.0ms/step | 14.3 samples/s
362
+ Step 7740/44000 | Loss: 3.1395 | LR: 4.68e-04 | 1399.9ms/step | 14.3 samples/s
363
+ Step 7750/44000 | Loss: 3.1764 | LR: 4.68e-04 | 1396.2ms/step | 14.3 samples/s
364
+ Step 7760/44000 | Loss: 3.1382 | LR: 4.68e-04 | 1397.9ms/step | 14.3 samples/s
365
+ Step 7770/44000 | Loss: 3.1704 | LR: 4.68e-04 | 1400.0ms/step | 14.3 samples/s
366
+ Step 7780/44000 | Loss: 3.1400 | LR: 4.68e-04 | 1399.6ms/step | 14.3 samples/s
367
+ Step 7790/44000 | Loss: 3.1871 | LR: 4.68e-04 | 1400.3ms/step | 14.3 samples/s
368
+ Step 7800/44000 | Loss: 3.1762 | LR: 4.68e-04 | 1400.2ms/step | 14.3 samples/s
369
+ Step 7810/44000 | Loss: 3.1836 | LR: 4.68e-04 | 1400.1ms/step | 14.3 samples/s
370
+ Step 7820/44000 | Loss: 3.1581 | LR: 4.68e-04 | 1400.0ms/step | 14.3 samples/s
371
+ Step 7830/44000 | Loss: 3.1480 | LR: 4.68e-04 | 1400.1ms/step | 14.3 samples/s
372
+ Step 7840/44000 | Loss: 3.1438 | LR: 4.68e-04 | 1399.8ms/step | 14.3 samples/s
373
+ Step 7850/44000 | Loss: 3.1499 | LR: 4.67e-04 | 1400.2ms/step | 14.3 samples/s
374
+ Step 7860/44000 | Loss: 3.1699 | LR: 4.67e-04 | 1400.2ms/step | 14.3 samples/s
375
+ Step 7870/44000 | Loss: 3.1577 | LR: 4.67e-04 | 1399.8ms/step | 14.3 samples/s
376
+ Step 7880/44000 | Loss: 3.1785 | LR: 4.67e-04 | 1400.0ms/step | 14.3 samples/s
377
+ Step 7890/44000 | Loss: 3.1615 | LR: 4.67e-04 | 1399.9ms/step | 14.3 samples/s
378
+ Step 7900/44000 | Loss: 3.1566 | LR: 4.67e-04 | 1399.7ms/step | 14.3 samples/s
379
+ Step 7910/44000 | Loss: 3.1652 | LR: 4.67e-04 | 1400.4ms/step | 14.3 samples/s
380
+ Step 7920/44000 | Loss: 3.1578 | LR: 4.67e-04 | 1400.2ms/step | 14.3 samples/s
381
+ Step 7930/44000 | Loss: 3.1648 | LR: 4.67e-04 | 1399.8ms/step | 14.3 samples/s
382
+ Step 7940/44000 | Loss: 3.1605 | LR: 4.67e-04 | 1399.9ms/step | 14.3 samples/s
383
+ Step 7950/44000 | Loss: 3.1716 | LR: 4.67e-04 | 1399.9ms/step | 14.3 samples/s
384
+ Step 7960/44000 | Loss: 3.1380 | LR: 4.66e-04 | 1400.1ms/step | 14.3 samples/s
385
+ Step 7970/44000 | Loss: 3.1668 | LR: 4.66e-04 | 1400.1ms/step | 14.3 samples/s
386
+ Step 7980/44000 | Loss: 3.1474 | LR: 4.66e-04 | 1400.0ms/step | 14.3 samples/s
387
+ Step 7990/44000 | Loss: 3.1581 | LR: 4.66e-04 | 1399.8ms/step | 14.3 samples/s
388
+ Step 8000/44000 | Loss: 3.1318 | LR: 4.66e-04 | 1398.1ms/step | 14.3 samples/s
389
+ Validation at step 8000: Loss = 3.1769, Perplexity = 23.97
390
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-8000
391
+ Step 8010/44000 | Loss: 3.1246 | LR: 4.66e-04 | 3910.0ms/step | 5.1 samples/s
392
+ Step 8020/44000 | Loss: 3.1631 | LR: 4.66e-04 | 1400.0ms/step | 14.3 samples/s
393
+ Step 8030/44000 | Loss: 3.1571 | LR: 4.66e-04 | 1399.9ms/step | 14.3 samples/s
394
+ Step 8040/44000 | Loss: 3.1638 | LR: 4.66e-04 | 1400.0ms/step | 14.3 samples/s
395
+ Step 8050/44000 | Loss: 3.1684 | LR: 4.66e-04 | 1400.1ms/step | 14.3 samples/s
396
+ Step 8060/44000 | Loss: 3.1380 | LR: 4.66e-04 | 1394.6ms/step | 14.3 samples/s
397
+ Step 8070/44000 | Loss: 3.1747 | LR: 4.65e-04 | 1399.5ms/step | 14.3 samples/s
398
+ Step 8080/44000 | Loss: 3.1615 | LR: 4.65e-04 | 1399.9ms/step | 14.3 samples/s
399
+ Step 8090/44000 | Loss: 3.1685 | LR: 4.65e-04 | 1400.1ms/step | 14.3 samples/s
400
+ Step 8100/44000 | Loss: 3.1373 | LR: 4.65e-04 | 1400.0ms/step | 14.3 samples/s
401
+ Step 8110/44000 | Loss: 3.1465 | LR: 4.65e-04 | 1400.0ms/step | 14.3 samples/s
402
+ Step 8120/44000 | Loss: 3.1858 | LR: 4.65e-04 | 1400.1ms/step | 14.3 samples/s
403
+ Step 8130/44000 | Loss: 3.1224 | LR: 4.65e-04 | 1400.0ms/step | 14.3 samples/s
404
+ Step 8140/44000 | Loss: 3.1562 | LR: 4.65e-04 | 1400.2ms/step | 14.3 samples/s
405
+ Step 8150/44000 | Loss: 3.1531 | LR: 4.65e-04 | 1399.9ms/step | 14.3 samples/s
406
+ Step 8160/44000 | Loss: 3.2047 | LR: 4.65e-04 | 1399.9ms/step | 14.3 samples/s
407
+ Step 8170/44000 | Loss: 3.1545 | LR: 4.65e-04 | 1400.2ms/step | 14.3 samples/s
408
+ Step 8180/44000 | Loss: 3.1403 | LR: 4.64e-04 | 1400.1ms/step | 14.3 samples/s
409
+ Step 8190/44000 | Loss: 3.1623 | LR: 4.64e-04 | 1399.9ms/step | 14.3 samples/s
410
+ Step 8200/44000 | Loss: 3.1441 | LR: 4.64e-04 | 1400.0ms/step | 14.3 samples/s
411
+ Step 8210/44000 | Loss: 3.1126 | LR: 4.64e-04 | 1399.9ms/step | 14.3 samples/s
412
+ Step 8220/44000 | Loss: 3.1763 | LR: 4.64e-04 | 1400.0ms/step | 14.3 samples/s
413
+ Step 8230/44000 | Loss: 3.1313 | LR: 4.64e-04 | 1400.1ms/step | 14.3 samples/s
414
+ Step 8240/44000 | Loss: 3.1570 | LR: 4.64e-04 | 1400.0ms/step | 14.3 samples/s
415
+ Step 8250/44000 | Loss: 3.1813 | LR: 4.64e-04 | 1400.0ms/step | 14.3 samples/s
416
+ Step 8260/44000 | Loss: 3.1404 | LR: 4.64e-04 | 1400.0ms/step | 14.3 samples/s
417
+ Step 8270/44000 | Loss: 3.1697 | LR: 4.64e-04 | 1400.2ms/step | 14.3 samples/s
418
+ Step 8280/44000 | Loss: 3.1685 | LR: 4.64e-04 | 1400.0ms/step | 14.3 samples/s
419
+ Step 8290/44000 | Loss: 3.1695 | LR: 4.63e-04 | 1399.9ms/step | 14.3 samples/s
420
+ Step 8300/44000 | Loss: 3.1603 | LR: 4.63e-04 | 1400.0ms/step | 14.3 samples/s
421
+ Step 8310/44000 | Loss: 3.1182 | LR: 4.63e-04 | 1400.3ms/step | 14.3 samples/s
422
+ Step 8320/44000 | Loss: 3.1691 | LR: 4.63e-04 | 1399.6ms/step | 14.3 samples/s
423
+ Step 8330/44000 | Loss: 3.1287 | LR: 4.63e-04 | 1399.9ms/step | 14.3 samples/s
424
+ Step 8340/44000 | Loss: 3.1401 | LR: 4.63e-04 | 1400.1ms/step | 14.3 samples/s
425
+ Step 8350/44000 | Loss: 3.1345 | LR: 4.63e-04 | 1400.2ms/step | 14.3 samples/s
426
+ Step 8360/44000 | Loss: 3.1484 | LR: 4.63e-04 | 1399.7ms/step | 14.3 samples/s
427
+ Step 8370/44000 | Loss: 3.1483 | LR: 4.63e-04 | 1400.2ms/step | 14.3 samples/s
428
+ Step 8380/44000 | Loss: 3.1654 | LR: 4.63e-04 | 1400.0ms/step | 14.3 samples/s
429
+ Step 8390/44000 | Loss: 3.1409 | LR: 4.62e-04 | 1399.9ms/step | 14.3 samples/s
430
+ Step 8400/44000 | Loss: 3.1687 | LR: 4.62e-04 | 1400.0ms/step | 14.3 samples/s
431
+ Step 8410/44000 | Loss: 3.1396 | LR: 4.62e-04 | 1400.1ms/step | 14.3 samples/s
432
+ Step 8420/44000 | Loss: 3.1560 | LR: 4.62e-04 | 1399.9ms/step | 14.3 samples/s
433
+ Step 8430/44000 | Loss: 3.1513 | LR: 4.62e-04 | 1400.1ms/step | 14.3 samples/s
434
+ Step 8440/44000 | Loss: 3.1248 | LR: 4.62e-04 | 1400.1ms/step | 14.3 samples/s
435
+ Step 8450/44000 | Loss: 3.1501 | LR: 4.62e-04 | 1400.0ms/step | 14.3 samples/s
436
+ Step 8460/44000 | Loss: 3.1496 | LR: 4.62e-04 | 1399.8ms/step | 14.3 samples/s
437
+ Step 8470/44000 | Loss: 3.1656 | LR: 4.62e-04 | 1400.1ms/step | 14.3 samples/s
438
+ Step 8480/44000 | Loss: 3.1364 | LR: 4.62e-04 | 1398.5ms/step | 14.3 samples/s
439
+ Step 8490/44000 | Loss: 3.1305 | LR: 4.62e-04 | 1399.2ms/step | 14.3 samples/s
440
+ Step 8500/44000 | Loss: 3.1453 | LR: 4.61e-04 | 1400.2ms/step | 14.3 samples/s
441
+ Step 8510/44000 | Loss: 3.1343 | LR: 4.61e-04 | 1400.1ms/step | 14.3 samples/s
442
+ Step 8520/44000 | Loss: 3.1421 | LR: 4.61e-04 | 1399.9ms/step | 14.3 samples/s
443
+ Step 8530/44000 | Loss: 3.1488 | LR: 4.61e-04 | 1400.4ms/step | 14.3 samples/s
444
+ Step 8540/44000 | Loss: 3.1422 | LR: 4.61e-04 | 1399.7ms/step | 14.3 samples/s
445
+ Step 8550/44000 | Loss: 3.1493 | LR: 4.61e-04 | 1399.9ms/step | 14.3 samples/s
446
+ Step 8560/44000 | Loss: 3.1500 | LR: 4.61e-04 | 1399.6ms/step | 14.3 samples/s
447
+ Step 8570/44000 | Loss: 3.1369 | LR: 4.61e-04 | 1400.4ms/step | 14.3 samples/s
448
+ Step 8580/44000 | Loss: 3.1313 | LR: 4.61e-04 | 1400.1ms/step | 14.3 samples/s
449
+ Step 8590/44000 | Loss: 3.1588 | LR: 4.61e-04 | 1397.9ms/step | 14.3 samples/s
450
+ Step 8600/44000 | Loss: 3.1659 | LR: 4.60e-04 | 1400.0ms/step | 14.3 samples/s
451
+ Step 8610/44000 | Loss: 3.1639 | LR: 4.60e-04 | 1399.7ms/step | 14.3 samples/s
452
+ Step 8620/44000 | Loss: 3.1396 | LR: 4.60e-04 | 1394.2ms/step | 14.3 samples/s
453
+ Step 8630/44000 | Loss: 3.1668 | LR: 4.60e-04 | 1398.0ms/step | 14.3 samples/s
454
+ Step 8640/44000 | Loss: 3.1610 | LR: 4.60e-04 | 1393.7ms/step | 14.4 samples/s
455
+ Step 8650/44000 | Loss: 3.1393 | LR: 4.60e-04 | 1392.1ms/step | 14.4 samples/s
456
+ Step 8660/44000 | Loss: 3.1538 | LR: 4.60e-04 | 1394.1ms/step | 14.3 samples/s
457
+ Step 8670/44000 | Loss: 3.1616 | LR: 4.60e-04 | 1397.8ms/step | 14.3 samples/s
458
+ Step 8680/44000 | Loss: 3.1305 | LR: 4.60e-04 | 1394.4ms/step | 14.3 samples/s
459
+ Step 8690/44000 | Loss: 3.1674 | LR: 4.60e-04 | 1397.7ms/step | 14.3 samples/s
460
+ Step 8700/44000 | Loss: 3.1270 | LR: 4.59e-04 | 1398.3ms/step | 14.3 samples/s
461
+ Step 8710/44000 | Loss: 3.1561 | LR: 4.59e-04 | 1399.3ms/step | 14.3 samples/s
462
+ Step 8720/44000 | Loss: 3.1537 | LR: 4.59e-04 | 1396.7ms/step | 14.3 samples/s
463
+ Step 8730/44000 | Loss: 3.1659 | LR: 4.59e-04 | 1400.1ms/step | 14.3 samples/s
464
+ Step 8740/44000 | Loss: 3.1363 | LR: 4.59e-04 | 1400.0ms/step | 14.3 samples/s
465
+ Step 8750/44000 | Loss: 3.1480 | LR: 4.59e-04 | 1395.9ms/step | 14.3 samples/s
466
+ Step 8760/44000 | Loss: 3.1150 | LR: 4.59e-04 | 1393.6ms/step | 14.4 samples/s
467
+ Step 8770/44000 | Loss: 3.1067 | LR: 4.59e-04 | 1398.6ms/step | 14.3 samples/s
468
+ Step 8780/44000 | Loss: 3.1357 | LR: 4.59e-04 | 1398.8ms/step | 14.3 samples/s
469
+ Step 8790/44000 | Loss: 3.1057 | LR: 4.59e-04 | 1399.1ms/step | 14.3 samples/s
470
+ Step 8800/44000 | Loss: 3.1179 | LR: 4.58e-04 | 1400.2ms/step | 14.3 samples/s
471
+ Step 8810/44000 | Loss: 3.1456 | LR: 4.58e-04 | 1400.0ms/step | 14.3 samples/s
472
+ Step 8820/44000 | Loss: 3.1479 | LR: 4.58e-04 | 1399.8ms/step | 14.3 samples/s
473
+ Step 8830/44000 | Loss: 3.1475 | LR: 4.58e-04 | 1400.1ms/step | 14.3 samples/s
474
+ Step 8840/44000 | Loss: 3.1286 | LR: 4.58e-04 | 1399.8ms/step | 14.3 samples/s
475
+ Step 8850/44000 | Loss: 3.1461 | LR: 4.58e-04 | 1400.1ms/step | 14.3 samples/s
476
+ Step 8860/44000 | Loss: 3.1630 | LR: 4.58e-04 | 1400.2ms/step | 14.3 samples/s
477
+ Step 8870/44000 | Loss: 3.1165 | LR: 4.58e-04 | 1400.1ms/step | 14.3 samples/s
478
+ Step 8880/44000 | Loss: 3.1052 | LR: 4.58e-04 | 1399.9ms/step | 14.3 samples/s
479
+ Step 8890/44000 | Loss: 3.1753 | LR: 4.58e-04 | 1400.1ms/step | 14.3 samples/s
480
+ Step 8900/44000 | Loss: 3.1244 | LR: 4.57e-04 | 1399.8ms/step | 14.3 samples/s
481
+ Step 8910/44000 | Loss: 3.1219 | LR: 4.57e-04 | 1400.2ms/step | 14.3 samples/s
482
+ Step 8920/44000 | Loss: 3.1438 | LR: 4.57e-04 | 1400.1ms/step | 14.3 samples/s
483
+ Step 8930/44000 | Loss: 3.1092 | LR: 4.57e-04 | 1399.7ms/step | 14.3 samples/s
484
+ Step 8940/44000 | Loss: 3.1655 | LR: 4.57e-04 | 1400.3ms/step | 14.3 samples/s
485
+ Step 8950/44000 | Loss: 3.1531 | LR: 4.57e-04 | 1399.9ms/step | 14.3 samples/s
486
+ Step 8960/44000 | Loss: 3.1320 | LR: 4.57e-04 | 1400.1ms/step | 14.3 samples/s
487
+ Step 8970/44000 | Loss: 3.1397 | LR: 4.57e-04 | 1399.9ms/step | 14.3 samples/s
488
+ Step 8980/44000 | Loss: 3.1739 | LR: 4.57e-04 | 1400.2ms/step | 14.3 samples/s
489
+ Step 8990/44000 | Loss: 3.1377 | LR: 4.57e-04 | 1399.8ms/step | 14.3 samples/s
490
+ Step 9000/44000 | Loss: 3.1085 | LR: 4.56e-04 | 1400.1ms/step | 14.3 samples/s
491
+ Validation at step 9000: Loss = 3.1476, Perplexity = 23.28
492
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-9000
493
+ Step 9010/44000 | Loss: 3.1102 | LR: 4.56e-04 | 3921.7ms/step | 5.1 samples/s
494
+ Step 9020/44000 | Loss: 3.1337 | LR: 4.56e-04 | 1397.4ms/step | 14.3 samples/s
495
+ Step 9030/44000 | Loss: 3.1498 | LR: 4.56e-04 | 1397.0ms/step | 14.3 samples/s
496
+ Step 9040/44000 | Loss: 3.1267 | LR: 4.56e-04 | 1399.9ms/step | 14.3 samples/s
497
+ Step 9050/44000 | Loss: 3.1293 | LR: 4.56e-04 | 1399.9ms/step | 14.3 samples/s
498
+ Step 9060/44000 | Loss: 3.1408 | LR: 4.56e-04 | 1400.2ms/step | 14.3 samples/s
499
+ Step 9070/44000 | Loss: 3.1244 | LR: 4.56e-04 | 1399.7ms/step | 14.3 samples/s
500
+ Step 9080/44000 | Loss: 3.1023 | LR: 4.56e-04 | 1400.4ms/step | 14.3 samples/s
501
+ Step 9090/44000 | Loss: 3.1336 | LR: 4.56e-04 | 1399.8ms/step | 14.3 samples/s
502
+ Step 9100/44000 | Loss: 3.1162 | LR: 4.55e-04 | 1400.3ms/step | 14.3 samples/s
503
+ Step 9110/44000 | Loss: 3.1291 | LR: 4.55e-04 | 1400.0ms/step | 14.3 samples/s
504
+ Step 9120/44000 | Loss: 3.1061 | LR: 4.55e-04 | 1400.1ms/step | 14.3 samples/s
505
+ Step 9130/44000 | Loss: 3.1603 | LR: 4.55e-04 | 1399.8ms/step | 14.3 samples/s
506
+ Step 9140/44000 | Loss: 3.1127 | LR: 4.55e-04 | 1400.1ms/step | 14.3 samples/s
507
+ Step 9150/44000 | Loss: 3.0816 | LR: 4.55e-04 | 1400.0ms/step | 14.3 samples/s
508
+ Step 9160/44000 | Loss: 3.1397 | LR: 4.55e-04 | 1400.0ms/step | 14.3 samples/s
509
+ Step 9170/44000 | Loss: 3.1230 | LR: 4.55e-04 | 1399.9ms/step | 14.3 samples/s
510
+ Step 9180/44000 | Loss: 3.1102 | LR: 4.55e-04 | 1400.2ms/step | 14.3 samples/s
511
+ Step 9190/44000 | Loss: 3.1167 | LR: 4.54e-04 | 1400.0ms/step | 14.3 samples/s
512
+ Step 9200/44000 | Loss: 3.1518 | LR: 4.54e-04 | 1400.2ms/step | 14.3 samples/s
513
+ Step 9210/44000 | Loss: 3.1201 | LR: 4.54e-04 | 1399.8ms/step | 14.3 samples/s
514
+ Step 9220/44000 | Loss: 3.1221 | LR: 4.54e-04 | 1400.0ms/step | 14.3 samples/s
515
+ Step 9230/44000 | Loss: 3.1169 | LR: 4.54e-04 | 1399.8ms/step | 14.3 samples/s
516
+ Step 9240/44000 | Loss: 3.0789 | LR: 4.54e-04 | 1400.2ms/step | 14.3 samples/s
517
+ Step 9250/44000 | Loss: 3.1280 | LR: 4.54e-04 | 1400.1ms/step | 14.3 samples/s
518
+ Step 9260/44000 | Loss: 3.1571 | LR: 4.54e-04 | 1399.9ms/step | 14.3 samples/s
519
+ Step 9270/44000 | Loss: 3.1059 | LR: 4.54e-04 | 1399.9ms/step | 14.3 samples/s
520
+ Step 9280/44000 | Loss: 3.1194 | LR: 4.54e-04 | 1400.2ms/step | 14.3 samples/s
521
+ Step 9290/44000 | Loss: 3.1404 | LR: 4.53e-04 | 1400.0ms/step | 14.3 samples/s
522
+ Step 9300/44000 | Loss: 3.1344 | LR: 4.53e-04 | 1399.9ms/step | 14.3 samples/s
523
+ Step 9310/44000 | Loss: 3.1219 | LR: 4.53e-04 | 1400.1ms/step | 14.3 samples/s
524
+ Step 9320/44000 | Loss: 3.1344 | LR: 4.53e-04 | 1399.9ms/step | 14.3 samples/s
525
+ Step 9330/44000 | Loss: 3.1366 | LR: 4.53e-04 | 1399.9ms/step | 14.3 samples/s
526
+ Step 9340/44000 | Loss: 3.1292 | LR: 4.53e-04 | 1400.0ms/step | 14.3 samples/s
527
+ Step 9350/44000 | Loss: 3.1456 | LR: 4.53e-04 | 1400.0ms/step | 14.3 samples/s
528
+ Step 9360/44000 | Loss: 3.1321 | LR: 4.53e-04 | 1400.3ms/step | 14.3 samples/s
529
+ Step 9370/44000 | Loss: 3.1277 | LR: 4.53e-04 | 1399.9ms/step | 14.3 samples/s
530
+ Step 9380/44000 | Loss: 3.1275 | LR: 4.52e-04 | 1399.7ms/step | 14.3 samples/s
531
+ Step 9390/44000 | Loss: 3.1112 | LR: 4.52e-04 | 1400.3ms/step | 14.3 samples/s
532
+ Step 9400/44000 | Loss: 3.1027 | LR: 4.52e-04 | 1399.9ms/step | 14.3 samples/s
533
+ Step 9410/44000 | Loss: 3.1364 | LR: 4.52e-04 | 1400.1ms/step | 14.3 samples/s
534
+ Step 9420/44000 | Loss: 3.1275 | LR: 4.52e-04 | 1400.0ms/step | 14.3 samples/s
535
+ Step 9430/44000 | Loss: 3.1499 | LR: 4.52e-04 | 1400.1ms/step | 14.3 samples/s
536
+ Step 9440/44000 | Loss: 3.1440 | LR: 4.52e-04 | 1399.7ms/step | 14.3 samples/s
537
+ Step 9450/44000 | Loss: 3.1354 | LR: 4.52e-04 | 1399.9ms/step | 14.3 samples/s
538
+ Step 9460/44000 | Loss: 3.1284 | LR: 4.52e-04 | 1400.1ms/step | 14.3 samples/s
539
+ Step 9470/44000 | Loss: 3.1148 | LR: 4.52e-04 | 1399.9ms/step | 14.3 samples/s
540
+ Step 9480/44000 | Loss: 3.1233 | LR: 4.51e-04 | 1399.9ms/step | 14.3 samples/s
541
+ Step 9490/44000 | Loss: 3.1318 | LR: 4.51e-04 | 1400.3ms/step | 14.3 samples/s
542
+ Step 9500/44000 | Loss: 3.1233 | LR: 4.51e-04 | 1399.8ms/step | 14.3 samples/s
543
+ Step 9510/44000 | Loss: 3.1156 | LR: 4.51e-04 | 1400.0ms/step | 14.3 samples/s
544
+ Step 9520/44000 | Loss: 3.0813 | LR: 4.51e-04 | 1400.2ms/step | 14.3 samples/s
545
+ Step 9530/44000 | Loss: 3.1250 | LR: 4.51e-04 | 1400.0ms/step | 14.3 samples/s
546
+ Step 9540/44000 | Loss: 3.1241 | LR: 4.51e-04 | 1399.8ms/step | 14.3 samples/s
547
+ Step 9550/44000 | Loss: 3.0836 | LR: 4.51e-04 | 1400.0ms/step | 14.3 samples/s
548
+ Step 9560/44000 | Loss: 3.1214 | LR: 4.51e-04 | 1399.9ms/step | 14.3 samples/s
549
+ Step 9570/44000 | Loss: 3.1250 | LR: 4.50e-04 | 1399.8ms/step | 14.3 samples/s
550
+ Step 9580/44000 | Loss: 3.1534 | LR: 4.50e-04 | 1398.2ms/step | 14.3 samples/s
551
+ Step 9590/44000 | Loss: 3.1195 | LR: 4.50e-04 | 1400.1ms/step | 14.3 samples/s
552
+ Step 9600/44000 | Loss: 3.1071 | LR: 4.50e-04 | 1400.2ms/step | 14.3 samples/s
553
+ Step 9610/44000 | Loss: 3.0950 | LR: 4.50e-04 | 1400.0ms/step | 14.3 samples/s
554
+ Step 9620/44000 | Loss: 3.1531 | LR: 4.50e-04 | 1397.9ms/step | 14.3 samples/s
555
+ Step 9630/44000 | Loss: 3.1302 | LR: 4.50e-04 | 1395.8ms/step | 14.3 samples/s
556
+ Step 9640/44000 | Loss: 3.1117 | LR: 4.50e-04 | 1400.1ms/step | 14.3 samples/s
557
+ Step 9650/44000 | Loss: 3.0827 | LR: 4.50e-04 | 1400.1ms/step | 14.3 samples/s
558
+ Step 9660/44000 | Loss: 3.0915 | LR: 4.49e-04 | 1399.9ms/step | 14.3 samples/s
559
+ Step 9670/44000 | Loss: 3.0828 | LR: 4.49e-04 | 1398.3ms/step | 14.3 samples/s
560
+ Step 9680/44000 | Loss: 3.1133 | LR: 4.49e-04 | 1399.9ms/step | 14.3 samples/s
561
+ Step 9690/44000 | Loss: 3.0771 | LR: 4.49e-04 | 1400.0ms/step | 14.3 samples/s
562
+ Step 9700/44000 | Loss: 3.1275 | LR: 4.49e-04 | 1399.8ms/step | 14.3 samples/s
563
+ Step 9710/44000 | Loss: 3.1102 | LR: 4.49e-04 | 1400.1ms/step | 14.3 samples/s
564
+ Step 9720/44000 | Loss: 3.1292 | LR: 4.49e-04 | 1400.0ms/step | 14.3 samples/s
565
+ Step 9730/44000 | Loss: 3.1022 | LR: 4.49e-04 | 1399.9ms/step | 14.3 samples/s
566
+ Step 9740/44000 | Loss: 3.1065 | LR: 4.49e-04 | 1400.0ms/step | 14.3 samples/s
567
+ Step 9750/44000 | Loss: 3.1091 | LR: 4.48e-04 | 1400.2ms/step | 14.3 samples/s
568
+ Step 9760/44000 | Loss: 3.1037 | LR: 4.48e-04 | 1399.9ms/step | 14.3 samples/s
569
+ Step 9770/44000 | Loss: 3.0946 | LR: 4.48e-04 | 1399.9ms/step | 14.3 samples/s
570
+ Step 9780/44000 | Loss: 3.1066 | LR: 4.48e-04 | 1400.0ms/step | 14.3 samples/s
571
+ Step 9790/44000 | Loss: 3.1279 | LR: 4.48e-04 | 1400.0ms/step | 14.3 samples/s
572
+ Step 9800/44000 | Loss: 3.1183 | LR: 4.48e-04 | 1400.4ms/step | 14.3 samples/s
573
+ Step 9810/44000 | Loss: 3.1193 | LR: 4.48e-04 | 1399.8ms/step | 14.3 samples/s
574
+ Step 9820/44000 | Loss: 3.1053 | LR: 4.48e-04 | 1399.6ms/step | 14.3 samples/s
575
+ Step 9830/44000 | Loss: 3.1378 | LR: 4.48e-04 | 1400.3ms/step | 14.3 samples/s
576
+ Step 9840/44000 | Loss: 3.1499 | LR: 4.47e-04 | 1399.9ms/step | 14.3 samples/s
577
+ Step 9850/44000 | Loss: 3.1194 | LR: 4.47e-04 | 1400.2ms/step | 14.3 samples/s
578
+ Step 9860/44000 | Loss: 3.1209 | LR: 4.47e-04 | 1397.6ms/step | 14.3 samples/s
579
+ Step 9870/44000 | Loss: 3.1155 | LR: 4.47e-04 | 1400.4ms/step | 14.3 samples/s
580
+ Step 9880/44000 | Loss: 3.0882 | LR: 4.47e-04 | 1399.9ms/step | 14.3 samples/s
581
+ Step 9890/44000 | Loss: 3.1251 | LR: 4.47e-04 | 1400.1ms/step | 14.3 samples/s
582
+ Step 9900/44000 | Loss: 3.0947 | LR: 4.47e-04 | 1399.9ms/step | 14.3 samples/s
583
+ Step 9910/44000 | Loss: 3.1215 | LR: 4.47e-04 | 1400.1ms/step | 14.3 samples/s
584
+ Step 9920/44000 | Loss: 3.1253 | LR: 4.47e-04 | 1400.0ms/step | 14.3 samples/s
585
+ Step 9930/44000 | Loss: 3.1201 | LR: 4.46e-04 | 1400.0ms/step | 14.3 samples/s
586
+ Step 9940/44000 | Loss: 3.1165 | LR: 4.46e-04 | 1400.1ms/step | 14.3 samples/s
587
+ Step 9950/44000 | Loss: 3.1119 | LR: 4.46e-04 | 1400.0ms/step | 14.3 samples/s
588
+ Step 9960/44000 | Loss: 3.1338 | LR: 4.46e-04 | 1399.9ms/step | 14.3 samples/s
589
+ Step 9970/44000 | Loss: 3.0999 | LR: 4.46e-04 | 1400.1ms/step | 14.3 samples/s
590
+ Step 9980/44000 | Loss: 3.1242 | LR: 4.46e-04 | 1400.2ms/step | 14.3 samples/s
591
+ Step 9990/44000 | Loss: 3.0853 | LR: 4.46e-04 | 1400.0ms/step | 14.3 samples/s
592
+ Step 10000/44000 | Loss: 3.1278 | LR: 4.46e-04 | 1399.7ms/step | 14.3 samples/s
593
+ Validation at step 10000: Loss = 3.1243, Perplexity = 22.74
594
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-10000
595
+ Step 10010/44000 | Loss: 3.1144 | LR: 4.46e-04 | 3918.1ms/step | 5.1 samples/s
596
+ Step 10020/44000 | Loss: 3.0922 | LR: 4.45e-04 | 1400.0ms/step | 14.3 samples/s
597
+ Step 10030/44000 | Loss: 3.1350 | LR: 4.45e-04 | 1400.0ms/step | 14.3 samples/s
598
+ Step 10040/44000 | Loss: 3.0928 | LR: 4.45e-04 | 1400.0ms/step | 14.3 samples/s
599
+ Step 10050/44000 | Loss: 3.1075 | LR: 4.45e-04 | 1400.1ms/step | 14.3 samples/s
600
+ Step 10060/44000 | Loss: 3.1406 | LR: 4.45e-04 | 1400.1ms/step | 14.3 samples/s
601
+ Step 10070/44000 | Loss: 3.0921 | LR: 4.45e-04 | 1399.7ms/step | 14.3 samples/s
602
+ Step 10080/44000 | Loss: 3.1016 | LR: 4.45e-04 | 1400.3ms/step | 14.3 samples/s
603
+ Step 10090/44000 | Loss: 3.1493 | LR: 4.45e-04 | 1400.2ms/step | 14.3 samples/s
604
+ Step 10100/44000 | Loss: 3.0981 | LR: 4.45e-04 | 1400.0ms/step | 14.3 samples/s
605
+ Step 10110/44000 | Loss: 3.1528 | LR: 4.44e-04 | 1400.2ms/step | 14.3 samples/s
606
+ Step 10120/44000 | Loss: 3.1058 | LR: 4.44e-04 | 1399.9ms/step | 14.3 samples/s
607
+ Step 10130/44000 | Loss: 3.0938 | LR: 4.44e-04 | 1400.1ms/step | 14.3 samples/s
608
+ Step 10140/44000 | Loss: 3.1111 | LR: 4.44e-04 | 1399.9ms/step | 14.3 samples/s
609
+ Step 10150/44000 | Loss: 3.1392 | LR: 4.44e-04 | 1400.0ms/step | 14.3 samples/s
610
+ Step 10160/44000 | Loss: 3.1183 | LR: 4.44e-04 | 1399.9ms/step | 14.3 samples/s
611
+ Step 10170/44000 | Loss: 3.1131 | LR: 4.44e-04 | 1399.9ms/step | 14.3 samples/s
612
+ Step 10180/44000 | Loss: 3.1047 | LR: 4.44e-04 | 1400.1ms/step | 14.3 samples/s
613
+ Step 10190/44000 | Loss: 3.0889 | LR: 4.43e-04 | 1400.0ms/step | 14.3 samples/s
614
+ Step 10200/44000 | Loss: 3.1069 | LR: 4.43e-04 | 1400.0ms/step | 14.3 samples/s
615
+ Step 10210/44000 | Loss: 3.0820 | LR: 4.43e-04 | 1399.8ms/step | 14.3 samples/s
616
+ Step 10220/44000 | Loss: 3.1049 | LR: 4.43e-04 | 1400.1ms/step | 14.3 samples/s
617
+ Step 10230/44000 | Loss: 3.1048 | LR: 4.43e-04 | 1400.2ms/step | 14.3 samples/s
618
+ Step 10240/44000 | Loss: 3.0975 | LR: 4.43e-04 | 1400.1ms/step | 14.3 samples/s
619
+ Step 10250/44000 | Loss: 3.1140 | LR: 4.43e-04 | 1400.0ms/step | 14.3 samples/s
620
+ Step 10260/44000 | Loss: 3.0934 | LR: 4.43e-04 | 1400.0ms/step | 14.3 samples/s
621
+ Step 10270/44000 | Loss: 3.1113 | LR: 4.43e-04 | 1399.8ms/step | 14.3 samples/s
622
+ Step 10280/44000 | Loss: 3.1084 | LR: 4.42e-04 | 1400.0ms/step | 14.3 samples/s
623
+ Step 10290/44000 | Loss: 3.1086 | LR: 4.42e-04 | 1400.1ms/step | 14.3 samples/s
624
+ Step 10300/44000 | Loss: 3.1209 | LR: 4.42e-04 | 1399.9ms/step | 14.3 samples/s
625
+ Step 10310/44000 | Loss: 3.1195 | LR: 4.42e-04 | 1400.1ms/step | 14.3 samples/s
626
+ Step 10320/44000 | Loss: 3.1143 | LR: 4.42e-04 | 1399.9ms/step | 14.3 samples/s
627
+ Step 10330/44000 | Loss: 3.1046 | LR: 4.42e-04 | 1400.2ms/step | 14.3 samples/s
628
+ Step 10340/44000 | Loss: 3.1161 | LR: 4.42e-04 | 1400.1ms/step | 14.3 samples/s
629
+ Step 10350/44000 | Loss: 3.1169 | LR: 4.42e-04 | 1400.0ms/step | 14.3 samples/s
630
+ Step 10360/44000 | Loss: 3.1209 | LR: 4.42e-04 | 1400.0ms/step | 14.3 samples/s
631
+ Step 10370/44000 | Loss: 3.1060 | LR: 4.41e-04 | 1399.9ms/step | 14.3 samples/s
632
+ Step 10380/44000 | Loss: 3.0900 | LR: 4.41e-04 | 1400.1ms/step | 14.3 samples/s
633
+ Step 10390/44000 | Loss: 3.0762 | LR: 4.41e-04 | 1399.9ms/step | 14.3 samples/s
634
+ Step 10400/44000 | Loss: 3.0853 | LR: 4.41e-04 | 1400.0ms/step | 14.3 samples/s
635
+ Step 10410/44000 | Loss: 3.0715 | LR: 4.41e-04 | 1400.2ms/step | 14.3 samples/s
636
+ Step 10420/44000 | Loss: 3.0782 | LR: 4.41e-04 | 1400.0ms/step | 14.3 samples/s
637
+ Step 10430/44000 | Loss: 3.1072 | LR: 4.41e-04 | 1399.9ms/step | 14.3 samples/s
638
+ Step 10440/44000 | Loss: 3.1115 | LR: 4.41e-04 | 1400.1ms/step | 14.3 samples/s
639
+ Step 10450/44000 | Loss: 3.1197 | LR: 4.40e-04 | 1399.9ms/step | 14.3 samples/s
640
+ Step 10460/44000 | Loss: 3.0849 | LR: 4.40e-04 | 1399.9ms/step | 14.3 samples/s
641
+ Step 10470/44000 | Loss: 3.1088 | LR: 4.40e-04 | 1400.2ms/step | 14.3 samples/s
642
+ Step 10480/44000 | Loss: 3.0904 | LR: 4.40e-04 | 1399.9ms/step | 14.3 samples/s
643
+ Step 10490/44000 | Loss: 3.1139 | LR: 4.40e-04 | 1399.8ms/step | 14.3 samples/s
644
+ Step 10500/44000 | Loss: 3.1166 | LR: 4.40e-04 | 1400.3ms/step | 14.3 samples/s
645
+ Step 10510/44000 | Loss: 3.0761 | LR: 4.40e-04 | 1399.8ms/step | 14.3 samples/s
646
+ Step 10520/44000 | Loss: 3.0832 | LR: 4.40e-04 | 1400.3ms/step | 14.3 samples/s
647
+ Step 10530/44000 | Loss: 3.1197 | LR: 4.40e-04 | 1400.0ms/step | 14.3 samples/s
648
+ Step 10540/44000 | Loss: 3.0981 | LR: 4.39e-04 | 1399.8ms/step | 14.3 samples/s
649
+ Step 10550/44000 | Loss: 3.0858 | LR: 4.39e-04 | 1398.4ms/step | 14.3 samples/s
650
+ Step 10560/44000 | Loss: 3.0926 | LR: 4.39e-04 | 1393.7ms/step | 14.4 samples/s
651
+ Step 10570/44000 | Loss: 3.1083 | LR: 4.39e-04 | 1395.7ms/step | 14.3 samples/s
652
+ Step 10580/44000 | Loss: 3.1064 | LR: 4.39e-04 | 1398.4ms/step | 14.3 samples/s
653
+ Step 10590/44000 | Loss: 3.0617 | LR: 4.39e-04 | 1397.9ms/step | 14.3 samples/s
654
+ Step 10600/44000 | Loss: 3.0855 | LR: 4.39e-04 | 1399.5ms/step | 14.3 samples/s
655
+ Step 10610/44000 | Loss: 3.0921 | LR: 4.39e-04 | 1398.4ms/step | 14.3 samples/s
656
+ Step 10620/44000 | Loss: 3.1399 | LR: 4.38e-04 | 1400.1ms/step | 14.3 samples/s
657
+ Step 10630/44000 | Loss: 3.1027 | LR: 4.38e-04 | 1400.0ms/step | 14.3 samples/s
658
+ Step 10640/44000 | Loss: 3.1140 | LR: 4.38e-04 | 1397.8ms/step | 14.3 samples/s
659
+ Step 10650/44000 | Loss: 3.0831 | LR: 4.38e-04 | 1398.2ms/step | 14.3 samples/s
660
+ Step 10660/44000 | Loss: 3.0934 | LR: 4.38e-04 | 1394.0ms/step | 14.3 samples/s
661
+ Step 10670/44000 | Loss: 3.1039 | LR: 4.38e-04 | 1400.0ms/step | 14.3 samples/s
662
+ Step 10680/44000 | Loss: 3.1010 | LR: 4.38e-04 | 1399.8ms/step | 14.3 samples/s
663
+ Step 10690/44000 | Loss: 3.0980 | LR: 4.38e-04 | 1400.0ms/step | 14.3 samples/s
664
+ Step 10700/44000 | Loss: 3.0888 | LR: 4.37e-04 | 1400.2ms/step | 14.3 samples/s
665
+ Step 10710/44000 | Loss: 3.1158 | LR: 4.37e-04 | 1395.8ms/step | 14.3 samples/s
666
+ Step 10720/44000 | Loss: 3.1045 | LR: 4.37e-04 | 1398.1ms/step | 14.3 samples/s
667
+ Step 10730/44000 | Loss: 3.0969 | LR: 4.37e-04 | 1399.9ms/step | 14.3 samples/s
668
+ Step 10740/44000 | Loss: 3.1057 | LR: 4.37e-04 | 1400.2ms/step | 14.3 samples/s
669
+ Step 10750/44000 | Loss: 3.1237 | LR: 4.37e-04 | 1399.6ms/step | 14.3 samples/s
670
+ Step 10760/44000 | Loss: 3.0859 | LR: 4.37e-04 | 1400.1ms/step | 14.3 samples/s
671
+ Step 10770/44000 | Loss: 3.1197 | LR: 4.37e-04 | 1400.2ms/step | 14.3 samples/s
672
+ Step 10780/44000 | Loss: 3.0862 | LR: 4.37e-04 | 1399.8ms/step | 14.3 samples/s
673
+ Step 10790/44000 | Loss: 3.1234 | LR: 4.36e-04 | 1393.4ms/step | 14.4 samples/s
674
+ Step 10800/44000 | Loss: 3.1156 | LR: 4.36e-04 | 1396.3ms/step | 14.3 samples/s
675
+ Step 10810/44000 | Loss: 3.0992 | LR: 4.36e-04 | 1394.0ms/step | 14.3 samples/s
676
+ Step 10820/44000 | Loss: 3.0855 | LR: 4.36e-04 | 1394.4ms/step | 14.3 samples/s
677
+ Step 10830/44000 | Loss: 3.0816 | LR: 4.36e-04 | 1398.1ms/step | 14.3 samples/s
678
+ Step 10840/44000 | Loss: 3.0898 | LR: 4.36e-04 | 1400.1ms/step | 14.3 samples/s
679
+ Step 10850/44000 | Loss: 3.0961 | LR: 4.36e-04 | 1395.9ms/step | 14.3 samples/s
680
+ Step 10860/44000 | Loss: 3.0921 | LR: 4.36e-04 | 1397.6ms/step | 14.3 samples/s
681
+ Step 10870/44000 | Loss: 3.1124 | LR: 4.35e-04 | 1400.3ms/step | 14.3 samples/s
682
+ Step 10880/44000 | Loss: 3.1131 | LR: 4.35e-04 | 1400.1ms/step | 14.3 samples/s
683
+ Step 10890/44000 | Loss: 3.1197 | LR: 4.35e-04 | 1397.8ms/step | 14.3 samples/s
684
+ Step 10900/44000 | Loss: 3.0636 | LR: 4.35e-04 | 1398.2ms/step | 14.3 samples/s
685
+ Step 10910/44000 | Loss: 3.0977 | LR: 4.35e-04 | 1400.2ms/step | 14.3 samples/s
686
+ Step 10920/44000 | Loss: 3.0996 | LR: 4.35e-04 | 1399.6ms/step | 14.3 samples/s
687
+ Step 10930/44000 | Loss: 3.0850 | LR: 4.35e-04 | 1400.3ms/step | 14.3 samples/s
688
+ Step 10940/44000 | Loss: 3.1079 | LR: 4.35e-04 | 1400.2ms/step | 14.3 samples/s
689
+ Step 10950/44000 | Loss: 3.0414 | LR: 4.34e-04 | 1399.9ms/step | 14.3 samples/s
690
+ Step 10960/44000 | Loss: 3.0533 | LR: 4.34e-04 | 1399.9ms/step | 14.3 samples/s
691
+ Step 10970/44000 | Loss: 3.0850 | LR: 4.34e-04 | 1400.0ms/step | 14.3 samples/s
692
+ Step 10980/44000 | Loss: 3.0935 | LR: 4.34e-04 | 1400.0ms/step | 14.3 samples/s
693
+ Step 10990/44000 | Loss: 3.0797 | LR: 4.34e-04 | 1400.0ms/step | 14.3 samples/s
694
+ Step 11000/44000 | Loss: 3.0770 | LR: 4.34e-04 | 1400.2ms/step | 14.3 samples/s
695
+ Validation at step 11000: Loss = 3.1025, Perplexity = 22.25
696
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-11000
697
+ Step 11010/44000 | Loss: 3.0844 | LR: 4.34e-04 | 3909.8ms/step | 5.1 samples/s
698
+ Step 11020/44000 | Loss: 3.0675 | LR: 4.34e-04 | 1400.0ms/step | 14.3 samples/s
699
+ Step 11030/44000 | Loss: 3.1045 | LR: 4.33e-04 | 1400.1ms/step | 14.3 samples/s
700
+ Step 11040/44000 | Loss: 3.0954 | LR: 4.33e-04 | 1400.0ms/step | 14.3 samples/s
701
+ Step 11050/44000 | Loss: 3.1100 | LR: 4.33e-04 | 1400.0ms/step | 14.3 samples/s
702
+ Step 11060/44000 | Loss: 3.0740 | LR: 4.33e-04 | 1400.0ms/step | 14.3 samples/s
703
+ Step 11070/44000 | Loss: 3.0763 | LR: 4.33e-04 | 1400.3ms/step | 14.3 samples/s
704
+ Step 11080/44000 | Loss: 3.0815 | LR: 4.33e-04 | 1399.8ms/step | 14.3 samples/s
705
+ Step 11090/44000 | Loss: 3.0846 | LR: 4.33e-04 | 1400.2ms/step | 14.3 samples/s
706
+ Step 11100/44000 | Loss: 3.0831 | LR: 4.33e-04 | 1400.1ms/step | 14.3 samples/s
707
+ Step 11110/44000 | Loss: 3.0839 | LR: 4.32e-04 | 1399.9ms/step | 14.3 samples/s
708
+ Step 11120/44000 | Loss: 3.0838 | LR: 4.32e-04 | 1400.1ms/step | 14.3 samples/s
709
+ Step 11130/44000 | Loss: 3.0771 | LR: 4.32e-04 | 1400.0ms/step | 14.3 samples/s
710
+ Step 11140/44000 | Loss: 3.0841 | LR: 4.32e-04 | 1399.9ms/step | 14.3 samples/s
711
+ Step 11150/44000 | Loss: 3.0290 | LR: 4.32e-04 | 1399.8ms/step | 14.3 samples/s
712
+ Step 11160/44000 | Loss: 3.0606 | LR: 4.32e-04 | 1400.2ms/step | 14.3 samples/s
713
+ Step 11170/44000 | Loss: 3.0876 | LR: 4.32e-04 | 1400.0ms/step | 14.3 samples/s
714
+ Step 11180/44000 | Loss: 3.0587 | LR: 4.32e-04 | 1399.9ms/step | 14.3 samples/s
715
+ Step 11190/44000 | Loss: 3.0833 | LR: 4.31e-04 | 1399.9ms/step | 14.3 samples/s
716
+ Step 11200/44000 | Loss: 3.1062 | LR: 4.31e-04 | 1400.1ms/step | 14.3 samples/s
717
+ Step 11210/44000 | Loss: 3.0599 | LR: 4.31e-04 | 1400.0ms/step | 14.3 samples/s
718
+ Step 11220/44000 | Loss: 3.0974 | LR: 4.31e-04 | 1399.9ms/step | 14.3 samples/s
719
+ Step 11230/44000 | Loss: 3.1131 | LR: 4.31e-04 | 1400.2ms/step | 14.3 samples/s
720
+ Step 11240/44000 | Loss: 3.0741 | LR: 4.31e-04 | 1400.0ms/step | 14.3 samples/s
721
+ Step 11250/44000 | Loss: 3.0929 | LR: 4.31e-04 | 1400.1ms/step | 14.3 samples/s
722
+ Step 11260/44000 | Loss: 3.0826 | LR: 4.31e-04 | 1399.9ms/step | 14.3 samples/s
723
+ Step 11270/44000 | Loss: 3.0620 | LR: 4.30e-04 | 1400.0ms/step | 14.3 samples/s
724
+ Step 11280/44000 | Loss: 3.1094 | LR: 4.30e-04 | 1400.0ms/step | 14.3 samples/s
725
+ Step 11290/44000 | Loss: 3.0688 | LR: 4.30e-04 | 1399.9ms/step | 14.3 samples/s
726
+ Step 11300/44000 | Loss: 3.0718 | LR: 4.30e-04 | 1400.5ms/step | 14.3 samples/s
727
+ Step 11310/44000 | Loss: 3.0525 | LR: 4.30e-04 | 1399.6ms/step | 14.3 samples/s
728
+ Step 11320/44000 | Loss: 3.0856 | LR: 4.30e-04 | 1400.0ms/step | 14.3 samples/s
729
+ Step 11330/44000 | Loss: 3.0492 | LR: 4.30e-04 | 1400.0ms/step | 14.3 samples/s
730
+ Step 11340/44000 | Loss: 3.0825 | LR: 4.30e-04 | 1400.0ms/step | 14.3 samples/s
731
+ Step 11350/44000 | Loss: 3.0830 | LR: 4.29e-04 | 1400.1ms/step | 14.3 samples/s
732
+ Step 11360/44000 | Loss: 3.1039 | LR: 4.29e-04 | 1400.0ms/step | 14.3 samples/s
733
+ Step 11370/44000 | Loss: 3.0300 | LR: 4.29e-04 | 1400.0ms/step | 14.3 samples/s
734
+ Step 11380/44000 | Loss: 3.0926 | LR: 4.29e-04 | 1399.8ms/step | 14.3 samples/s
735
+ Step 11390/44000 | Loss: 3.0753 | LR: 4.29e-04 | 1400.0ms/step | 14.3 samples/s
736
+ Step 11400/44000 | Loss: 3.1003 | LR: 4.29e-04 | 1400.3ms/step | 14.3 samples/s
737
+ Step 11410/44000 | Loss: 3.0691 | LR: 4.29e-04 | 1400.2ms/step | 14.3 samples/s
738
+ Step 11420/44000 | Loss: 3.0666 | LR: 4.29e-04 | 1399.9ms/step | 14.3 samples/s
739
+ Step 11430/44000 | Loss: 3.0662 | LR: 4.28e-04 | 1399.8ms/step | 14.3 samples/s
740
+ Step 11440/44000 | Loss: 3.0694 | LR: 4.28e-04 | 1400.1ms/step | 14.3 samples/s
741
+ Step 11450/44000 | Loss: 3.0693 | LR: 4.28e-04 | 1399.9ms/step | 14.3 samples/s
742
+ Step 11460/44000 | Loss: 3.0781 | LR: 4.28e-04 | 1400.0ms/step | 14.3 samples/s
743
+ Step 11470/44000 | Loss: 3.0516 | LR: 4.28e-04 | 1400.2ms/step | 14.3 samples/s
744
+ Step 11480/44000 | Loss: 3.0841 | LR: 4.28e-04 | 1400.0ms/step | 14.3 samples/s
745
+ Step 11490/44000 | Loss: 3.0653 | LR: 4.28e-04 | 1399.7ms/step | 14.3 samples/s
746
+ Step 11500/44000 | Loss: 3.0752 | LR: 4.28e-04 | 1396.2ms/step | 14.3 samples/s
747
+ Step 11510/44000 | Loss: 3.0928 | LR: 4.27e-04 | 1395.7ms/step | 14.3 samples/s
748
+ Step 11520/44000 | Loss: 3.0803 | LR: 4.27e-04 | 1400.0ms/step | 14.3 samples/s
749
+ Step 11530/44000 | Loss: 3.0878 | LR: 4.27e-04 | 1399.9ms/step | 14.3 samples/s
750
+ Step 11540/44000 | Loss: 3.0988 | LR: 4.27e-04 | 1400.3ms/step | 14.3 samples/s
751
+ Step 11550/44000 | Loss: 3.0971 | LR: 4.27e-04 | 1392.4ms/step | 14.4 samples/s
752
+ Step 11560/44000 | Loss: 3.0969 | LR: 4.27e-04 | 1397.5ms/step | 14.3 samples/s
753
+ Step 11570/44000 | Loss: 3.0709 | LR: 4.27e-04 | 1398.2ms/step | 14.3 samples/s
754
+ Step 11580/44000 | Loss: 3.0928 | LR: 4.27e-04 | 1395.9ms/step | 14.3 samples/s
755
+ Step 11590/44000 | Loss: 3.0944 | LR: 4.26e-04 | 1391.9ms/step | 14.4 samples/s
756
+ Step 11600/44000 | Loss: 3.0536 | LR: 4.26e-04 | 1391.2ms/step | 14.4 samples/s
757
+ Step 11610/44000 | Loss: 3.1053 | LR: 4.26e-04 | 1396.9ms/step | 14.3 samples/s
758
+ Step 11620/44000 | Loss: 3.0639 | LR: 4.26e-04 | 1400.1ms/step | 14.3 samples/s
759
+ Step 11630/44000 | Loss: 3.0860 | LR: 4.26e-04 | 1399.9ms/step | 14.3 samples/s
760
+ Step 11640/44000 | Loss: 3.0760 | LR: 4.26e-04 | 1400.0ms/step | 14.3 samples/s
761
+ Step 11650/44000 | Loss: 3.1006 | LR: 4.26e-04 | 1400.3ms/step | 14.3 samples/s
762
+ Step 11660/44000 | Loss: 3.0691 | LR: 4.26e-04 | 1399.9ms/step | 14.3 samples/s
763
+ Step 11670/44000 | Loss: 3.0699 | LR: 4.25e-04 | 1399.9ms/step | 14.3 samples/s
764
+ Step 11680/44000 | Loss: 3.0746 | LR: 4.25e-04 | 1399.7ms/step | 14.3 samples/s
765
+ Step 11690/44000 | Loss: 3.0698 | LR: 4.25e-04 | 1398.2ms/step | 14.3 samples/s
766
+ Step 11700/44000 | Loss: 3.1018 | LR: 4.25e-04 | 1398.1ms/step | 14.3 samples/s
767
+ Step 11710/44000 | Loss: 3.0687 | LR: 4.25e-04 | 1397.9ms/step | 14.3 samples/s
768
+ Step 11720/44000 | Loss: 3.0881 | LR: 4.25e-04 | 1396.2ms/step | 14.3 samples/s
769
+ Step 11730/44000 | Loss: 3.0631 | LR: 4.25e-04 | 1397.8ms/step | 14.3 samples/s
770
+ Step 11740/44000 | Loss: 3.0646 | LR: 4.24e-04 | 1398.2ms/step | 14.3 samples/s
771
+ Step 11750/44000 | Loss: 3.0789 | LR: 4.24e-04 | 1399.8ms/step | 14.3 samples/s
772
+ Step 11760/44000 | Loss: 3.0717 | LR: 4.24e-04 | 1399.7ms/step | 14.3 samples/s
773
+ Step 11770/44000 | Loss: 3.0595 | LR: 4.24e-04 | 1398.1ms/step | 14.3 samples/s
774
+ Step 11780/44000 | Loss: 3.0797 | LR: 4.24e-04 | 1398.1ms/step | 14.3 samples/s
775
+ Step 11790/44000 | Loss: 3.0711 | LR: 4.24e-04 | 1396.1ms/step | 14.3 samples/s
776
+ Step 11800/44000 | Loss: 3.0779 | LR: 4.24e-04 | 1398.0ms/step | 14.3 samples/s
777
+ Step 11810/44000 | Loss: 3.0899 | LR: 4.24e-04 | 1398.2ms/step | 14.3 samples/s
778
+ Step 11820/44000 | Loss: 3.0694 | LR: 4.23e-04 | 1395.7ms/step | 14.3 samples/s
779
+ Step 11830/44000 | Loss: 3.0725 | LR: 4.23e-04 | 1400.3ms/step | 14.3 samples/s
780
+ Step 11840/44000 | Loss: 3.0797 | LR: 4.23e-04 | 1400.2ms/step | 14.3 samples/s
781
+ Step 11850/44000 | Loss: 3.0988 | LR: 4.23e-04 | 1399.5ms/step | 14.3 samples/s
782
+ Step 11860/44000 | Loss: 3.0664 | LR: 4.23e-04 | 1399.0ms/step | 14.3 samples/s
783
+ Step 11870/44000 | Loss: 3.0687 | LR: 4.23e-04 | 1395.5ms/step | 14.3 samples/s
784
+ Step 11880/44000 | Loss: 3.0676 | LR: 4.23e-04 | 1398.0ms/step | 14.3 samples/s
785
+ Step 11890/44000 | Loss: 3.0505 | LR: 4.23e-04 | 1400.1ms/step | 14.3 samples/s
786
+ Step 11900/44000 | Loss: 3.0689 | LR: 4.22e-04 | 1393.4ms/step | 14.4 samples/s
787
+ Step 11910/44000 | Loss: 3.0724 | LR: 4.22e-04 | 1389.2ms/step | 14.4 samples/s
788
+ Step 11920/44000 | Loss: 3.1088 | LR: 4.22e-04 | 1389.2ms/step | 14.4 samples/s
789
+ Step 11930/44000 | Loss: 3.0605 | LR: 4.22e-04 | 1393.9ms/step | 14.3 samples/s
790
+ Step 11940/44000 | Loss: 3.0641 | LR: 4.22e-04 | 1394.3ms/step | 14.3 samples/s
791
+ Step 11950/44000 | Loss: 3.0991 | LR: 4.22e-04 | 1400.0ms/step | 14.3 samples/s
792
+ Step 11960/44000 | Loss: 3.0650 | LR: 4.22e-04 | 1400.0ms/step | 14.3 samples/s
793
+ Step 11970/44000 | Loss: 3.0352 | LR: 4.21e-04 | 1400.1ms/step | 14.3 samples/s
794
+ Step 11980/44000 | Loss: 3.0434 | LR: 4.21e-04 | 1400.1ms/step | 14.3 samples/s
795
+ Step 11990/44000 | Loss: 3.0760 | LR: 4.21e-04 | 1400.0ms/step | 14.3 samples/s
796
+ Step 12000/44000 | Loss: 3.0750 | LR: 4.21e-04 | 1400.0ms/step | 14.3 samples/s
797
+ Validation at step 12000: Loss = 3.0836, Perplexity = 21.84
798
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-12000
799
+ Step 12010/44000 | Loss: 3.0598 | LR: 4.21e-04 | 3900.0ms/step | 5.1 samples/s
800
+ Step 12020/44000 | Loss: 3.0493 | LR: 4.21e-04 | 1400.0ms/step | 14.3 samples/s
801
+ Step 12030/44000 | Loss: 3.0510 | LR: 4.21e-04 | 1400.0ms/step | 14.3 samples/s
802
+ Step 12040/44000 | Loss: 3.0973 | LR: 4.21e-04 | 1400.0ms/step | 14.3 samples/s
803
+ Step 12050/44000 | Loss: 3.1019 | LR: 4.20e-04 | 1400.1ms/step | 14.3 samples/s
804
+ Step 12060/44000 | Loss: 3.0439 | LR: 4.20e-04 | 1400.1ms/step | 14.3 samples/s
805
+ Step 12070/44000 | Loss: 3.0662 | LR: 4.20e-04 | 1400.2ms/step | 14.3 samples/s
806
+ Step 12080/44000 | Loss: 3.0577 | LR: 4.20e-04 | 1399.5ms/step | 14.3 samples/s
807
+ Step 12090/44000 | Loss: 3.0533 | LR: 4.20e-04 | 1400.4ms/step | 14.3 samples/s
808
+ Step 12100/44000 | Loss: 3.0941 | LR: 4.20e-04 | 1400.0ms/step | 14.3 samples/s
809
+ Step 12110/44000 | Loss: 3.0935 | LR: 4.20e-04 | 1400.1ms/step | 14.3 samples/s
810
+ Step 12120/44000 | Loss: 3.0815 | LR: 4.19e-04 | 1400.1ms/step | 14.3 samples/s
811
+ Step 12130/44000 | Loss: 3.0591 | LR: 4.19e-04 | 1399.9ms/step | 14.3 samples/s
812
+ Step 12140/44000 | Loss: 3.0769 | LR: 4.19e-04 | 1399.9ms/step | 14.3 samples/s
813
+ Step 12150/44000 | Loss: 3.0604 | LR: 4.19e-04 | 1399.8ms/step | 14.3 samples/s
814
+ Step 12160/44000 | Loss: 3.0619 | LR: 4.19e-04 | 1400.0ms/step | 14.3 samples/s
815
+ Step 12170/44000 | Loss: 3.0803 | LR: 4.19e-04 | 1400.2ms/step | 14.3 samples/s
816
+ Step 12180/44000 | Loss: 3.0611 | LR: 4.19e-04 | 1400.0ms/step | 14.3 samples/s
817
+ Step 12190/44000 | Loss: 3.0662 | LR: 4.19e-04 | 1400.1ms/step | 14.3 samples/s
818
+ Step 12200/44000 | Loss: 3.0830 | LR: 4.18e-04 | 1400.0ms/step | 14.3 samples/s
819
+ Step 12210/44000 | Loss: 3.0371 | LR: 4.18e-04 | 1399.9ms/step | 14.3 samples/s
820
+ Step 12220/44000 | Loss: 3.0783 | LR: 4.18e-04 | 1400.0ms/step | 14.3 samples/s
821
+ Step 12230/44000 | Loss: 3.0382 | LR: 4.18e-04 | 1400.0ms/step | 14.3 samples/s
822
+ Step 12240/44000 | Loss: 3.0788 | LR: 4.18e-04 | 1400.1ms/step | 14.3 samples/s
823
+ Step 12250/44000 | Loss: 3.0466 | LR: 4.18e-04 | 1399.9ms/step | 14.3 samples/s
824
+ Step 12260/44000 | Loss: 3.0745 | LR: 4.18e-04 | 1400.2ms/step | 14.3 samples/s
825
+ Step 12270/44000 | Loss: 3.0509 | LR: 4.17e-04 | 1400.2ms/step | 14.3 samples/s
826
+ Step 12280/44000 | Loss: 3.0736 | LR: 4.17e-04 | 1399.7ms/step | 14.3 samples/s
827
+ Step 12290/44000 | Loss: 3.0859 | LR: 4.17e-04 | 1399.8ms/step | 14.3 samples/s
828
+ Step 12300/44000 | Loss: 3.0647 | LR: 4.17e-04 | 1400.1ms/step | 14.3 samples/s
829
+ Step 12310/44000 | Loss: 3.0366 | LR: 4.17e-04 | 1400.1ms/step | 14.3 samples/s
830
+ Step 12320/44000 | Loss: 3.0726 | LR: 4.17e-04 | 1399.9ms/step | 14.3 samples/s
831
+ Step 12330/44000 | Loss: 3.0638 | LR: 4.17e-04 | 1400.0ms/step | 14.3 samples/s
832
+ Step 12340/44000 | Loss: 3.0636 | LR: 4.17e-04 | 1400.0ms/step | 14.3 samples/s
833
+ Step 12350/44000 | Loss: 3.0525 | LR: 4.16e-04 | 1400.2ms/step | 14.3 samples/s
834
+ Step 12360/44000 | Loss: 3.1035 | LR: 4.16e-04 | 1399.9ms/step | 14.3 samples/s
835
+ Step 12370/44000 | Loss: 3.0440 | LR: 4.16e-04 | 1400.0ms/step | 14.3 samples/s
836
+ Step 12380/44000 | Loss: 3.0771 | LR: 4.16e-04 | 1400.1ms/step | 14.3 samples/s
837
+ Step 12390/44000 | Loss: 3.0714 | LR: 4.16e-04 | 1399.8ms/step | 14.3 samples/s
838
+ Step 12400/44000 | Loss: 3.0821 | LR: 4.16e-04 | 1400.1ms/step | 14.3 samples/s
839
+ Step 12410/44000 | Loss: 3.0867 | LR: 4.16e-04 | 1399.9ms/step | 14.3 samples/s
840
+ Step 12420/44000 | Loss: 3.0778 | LR: 4.15e-04 | 1399.9ms/step | 14.3 samples/s
841
+ Step 12430/44000 | Loss: 3.0762 | LR: 4.15e-04 | 1400.2ms/step | 14.3 samples/s
842
+ Step 12440/44000 | Loss: 3.0654 | LR: 4.15e-04 | 1399.5ms/step | 14.3 samples/s
843
+ Step 12450/44000 | Loss: 3.0913 | LR: 4.15e-04 | 1398.1ms/step | 14.3 samples/s
844
+ Step 12460/44000 | Loss: 3.0649 | LR: 4.15e-04 | 1400.3ms/step | 14.3 samples/s
845
+ Step 12470/44000 | Loss: 3.0787 | LR: 4.15e-04 | 1400.0ms/step | 14.3 samples/s
846
+ Step 12480/44000 | Loss: 3.0563 | LR: 4.15e-04 | 1397.8ms/step | 14.3 samples/s
847
+ Step 12490/44000 | Loss: 3.0607 | LR: 4.14e-04 | 1394.2ms/step | 14.3 samples/s
848
+ Step 12500/44000 | Loss: 3.1091 | LR: 4.14e-04 | 1399.8ms/step | 14.3 samples/s
849
+ Step 12510/44000 | Loss: 3.0616 | LR: 4.14e-04 | 1398.1ms/step | 14.3 samples/s
850
+ Step 12520/44000 | Loss: 3.0492 | LR: 4.14e-04 | 1398.1ms/step | 14.3 samples/s
851
+ Step 12530/44000 | Loss: 3.0561 | LR: 4.14e-04 | 1400.0ms/step | 14.3 samples/s
852
+ Step 12540/44000 | Loss: 3.0669 | LR: 4.14e-04 | 1400.0ms/step | 14.3 samples/s
853
+ Step 12550/44000 | Loss: 3.0621 | LR: 4.14e-04 | 1400.1ms/step | 14.3 samples/s
854
+ Step 12560/44000 | Loss: 3.0383 | LR: 4.14e-04 | 1400.0ms/step | 14.3 samples/s
855
+ Step 12570/44000 | Loss: 3.0454 | LR: 4.13e-04 | 1399.9ms/step | 14.3 samples/s
856
+ Step 12580/44000 | Loss: 3.0636 | LR: 4.13e-04 | 1400.1ms/step | 14.3 samples/s
857
+ Step 12590/44000 | Loss: 3.0659 | LR: 4.13e-04 | 1398.8ms/step | 14.3 samples/s
858
+ Step 12600/44000 | Loss: 3.0398 | LR: 4.13e-04 | 1399.4ms/step | 14.3 samples/s
859
+ Step 12610/44000 | Loss: 3.0751 | LR: 4.13e-04 | 1399.7ms/step | 14.3 samples/s
860
+ Step 12620/44000 | Loss: 3.0318 | LR: 4.13e-04 | 1400.2ms/step | 14.3 samples/s
861
+ Step 12630/44000 | Loss: 3.0636 | LR: 4.13e-04 | 1400.1ms/step | 14.3 samples/s
862
+ Step 12640/44000 | Loss: 3.0989 | LR: 4.12e-04 | 1400.0ms/step | 14.3 samples/s
863
+ Step 12650/44000 | Loss: 3.0642 | LR: 4.12e-04 | 1400.1ms/step | 14.3 samples/s
864
+ Step 12660/44000 | Loss: 3.0632 | LR: 4.12e-04 | 1399.8ms/step | 14.3 samples/s
865
+ Step 12670/44000 | Loss: 3.0957 | LR: 4.12e-04 | 1400.1ms/step | 14.3 samples/s
866
+ Step 12680/44000 | Loss: 3.0741 | LR: 4.12e-04 | 1399.7ms/step | 14.3 samples/s
867
+ Step 12690/44000 | Loss: 3.0601 | LR: 4.12e-04 | 1400.0ms/step | 14.3 samples/s
868
+ Step 12700/44000 | Loss: 3.0631 | LR: 4.12e-04 | 1400.1ms/step | 14.3 samples/s
869
+ Step 12710/44000 | Loss: 3.0496 | LR: 4.11e-04 | 1400.1ms/step | 14.3 samples/s
870
+ Step 12720/44000 | Loss: 3.0457 | LR: 4.11e-04 | 1400.1ms/step | 14.3 samples/s
871
+ Step 12730/44000 | Loss: 3.0307 | LR: 4.11e-04 | 1400.2ms/step | 14.3 samples/s
872
+ Step 12740/44000 | Loss: 3.0825 | LR: 4.11e-04 | 1399.6ms/step | 14.3 samples/s
873
+ Step 12750/44000 | Loss: 3.0599 | LR: 4.11e-04 | 1399.9ms/step | 14.3 samples/s
874
+ Step 12760/44000 | Loss: 3.0414 | LR: 4.11e-04 | 1400.1ms/step | 14.3 samples/s
875
+ Step 12770/44000 | Loss: 3.0473 | LR: 4.11e-04 | 1399.9ms/step | 14.3 samples/s
876
+ Step 12780/44000 | Loss: 3.0775 | LR: 4.10e-04 | 1398.2ms/step | 14.3 samples/s
877
+ Step 12790/44000 | Loss: 3.0768 | LR: 4.10e-04 | 1395.6ms/step | 14.3 samples/s
878
+ Step 12800/44000 | Loss: 3.0599 | LR: 4.10e-04 | 1398.3ms/step | 14.3 samples/s
879
+ Step 12810/44000 | Loss: 3.0858 | LR: 4.10e-04 | 1400.1ms/step | 14.3 samples/s
880
+ Step 12820/44000 | Loss: 3.0694 | LR: 4.10e-04 | 1399.9ms/step | 14.3 samples/s
881
+ Step 12830/44000 | Loss: 3.0592 | LR: 4.10e-04 | 1400.0ms/step | 14.3 samples/s
882
+ Step 12840/44000 | Loss: 3.0617 | LR: 4.10e-04 | 1399.9ms/step | 14.3 samples/s
883
+ Step 12850/44000 | Loss: 3.0265 | LR: 4.09e-04 | 1400.2ms/step | 14.3 samples/s
884
+ Step 12860/44000 | Loss: 3.0322 | LR: 4.09e-04 | 1400.0ms/step | 14.3 samples/s
885
+ Step 12870/44000 | Loss: 3.0899 | LR: 4.09e-04 | 1397.2ms/step | 14.3 samples/s
886
+ Step 12880/44000 | Loss: 3.0282 | LR: 4.09e-04 | 1389.2ms/step | 14.4 samples/s
887
+ Step 12890/44000 | Loss: 3.0903 | LR: 4.09e-04 | 1389.3ms/step | 14.4 samples/s
888
+ Step 12900/44000 | Loss: 3.0843 | LR: 4.09e-04 | 1391.8ms/step | 14.4 samples/s
889
+ Step 12910/44000 | Loss: 3.0160 | LR: 4.09e-04 | 1392.1ms/step | 14.4 samples/s
890
+ Step 12920/44000 | Loss: 3.0437 | LR: 4.09e-04 | 1388.2ms/step | 14.4 samples/s
891
+ Step 12930/44000 | Loss: 3.0870 | LR: 4.08e-04 | 1395.8ms/step | 14.3 samples/s
892
+ Step 12940/44000 | Loss: 3.0673 | LR: 4.08e-04 | 1393.1ms/step | 14.4 samples/s
893
+ Step 12950/44000 | Loss: 3.0260 | LR: 4.08e-04 | 1392.9ms/step | 14.4 samples/s
894
+ Step 12960/44000 | Loss: 3.0447 | LR: 4.08e-04 | 1398.2ms/step | 14.3 samples/s
895
+ Step 12970/44000 | Loss: 3.0355 | LR: 4.08e-04 | 1397.3ms/step | 14.3 samples/s
896
+ Step 12980/44000 | Loss: 3.0511 | LR: 4.08e-04 | 1392.6ms/step | 14.4 samples/s
897
+ Step 12990/44000 | Loss: 3.0267 | LR: 4.08e-04 | 1400.4ms/step | 14.3 samples/s
898
+ Step 13000/44000 | Loss: 3.0597 | LR: 4.07e-04 | 1396.2ms/step | 14.3 samples/s
899
+ Validation at step 13000: Loss = 3.0665, Perplexity = 21.47
900
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735/checkpoint-13000
901
+ Step 13010/44000 | Loss: 3.0247 | LR: 4.07e-04 | 3915.6ms/step | 5.1 samples/s
902
+ Step 13020/44000 | Loss: 3.0659 | LR: 4.07e-04 | 1400.1ms/step | 14.3 samples/s
903
+ Step 13030/44000 | Loss: 3.0623 | LR: 4.07e-04 | 1399.9ms/step | 14.3 samples/s
epfl_final/train_logs/ropectl_a1p0_h100_resume/seed43.20260820T011048Z.log ADDED
@@ -0,0 +1,900 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Auto-resuming from latest checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-7000
2
+ Resuming: Using existing output directory /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735
3
+ Running config:
4
+ {
5
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
6
+ "model_dtype": "bfloat16",
7
+ "init_from_scratch": true,
8
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
9
+ "val_path": null,
10
+ "max_seq_length": 8192,
11
+ "num_workers": 8,
12
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735",
13
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-7000",
14
+ "force_new_run": false,
15
+ "num_epochs": 1,
16
+ "max_steps": 44000,
17
+ "per_device_batch_size": 2,
18
+ "gradient_accumulation_steps": 10,
19
+ "learning_rate": 0.0005,
20
+ "weight_decay": 0.01,
21
+ "warmup_steps": 750,
22
+ "lr_scheduler": "cosine",
23
+ "max_grad_norm": 1.0,
24
+ "save_steps": 1000,
25
+ "eval_steps": 1000,
26
+ "eval_batches": 50,
27
+ "logging_steps": 10,
28
+ "seed": 43,
29
+ "verbose": true,
30
+ "checkpoint_activations": false,
31
+ "strategy": "ddp",
32
+ "mixed_precision": "bf16",
33
+ "wandb_enabled": true,
34
+ "wandb_project": "stochastic_sparsity",
35
+ "wandb_name": "l8k365m_dense_ropectl_a1p0_seed43",
36
+ "wandb_tags": [
37
+ "qwen3",
38
+ "365m",
39
+ "8k",
40
+ "dense",
41
+ "rope-control"
42
+ ],
43
+ "hidden_size": 768,
44
+ "intermediate_size": 3072,
45
+ "num_attention_heads": 6,
46
+ "num_key_value_heads": 2,
47
+ "num_hidden_layers": 28,
48
+ "head_dim": null,
49
+ "attention_dropout": 0.0,
50
+ "rope_train_position_scale_set": [
51
+ 1.0,
52
+ 0.9896339488176222,
53
+ 0.9692159692159692,
54
+ 0.9420289855072463,
55
+ 0.8730158730158729
56
+ ],
57
+ "attention_type": "sparse_stochastic",
58
+ "sparse_block_size": 128,
59
+ "sparse_num_random_blocks": 31,
60
+ "sparse_mask_pool_size": 1,
61
+ "sparse_val_dense_compare": false,
62
+ "sparse_layer_indices": null,
63
+ "sparse_mask_cache_dir": null,
64
+ "sparse_mask_seed": 0,
65
+ "sparse_mixed_nrb_set": null,
66
+ "dense_mode": true,
67
+ "dense_step_intervals": null
68
+ }
69
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
70
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-7000
71
+ Checkpoint activations: False
72
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
73
+
74
+
75
+ ================================================================================
76
+ MODEL ARCHITECTURE
77
+ ================================================================================
78
+ Model config:
79
+ vocab_size: 151936
80
+ hidden_size: 768
81
+ num_hidden_layers: 28
82
+ num_attention_heads: 6
83
+ num_key_value_heads: 2
84
+ head_dim: 128
85
+ intermediate_size: 3072
86
+ max_position_embeddings: 32768
87
+ rope_theta: 1000000
88
+ rope_train_position_scale_set: [1.0, 0.9896339488176222, 0.9692159692159692, 0.9420289855072463, 0.8730158730158729]
89
+ attention_type: sparse_stochastic
90
+ sparse_block_size: 128
91
+ sparse_num_random_blocks: 31
92
+ sparse_mask_pool_size: 1
93
+ sparse_max_seq_length: 8192
94
+
95
+ Parameter count:
96
+ Total parameters: 358,958,848
97
+ Trainable parameters: 358,958,848
98
+ Non-trainable parameters: 0
99
+ Model size (BF16): 0.72 GB
100
+
101
+ --------------------------------------------------------------------------------
102
+ LAYER-BY-LAYER SUMMARY
103
+ --------------------------------------------------------------------------------
104
+ ================================================================================================================================================================
105
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
106
+ ================================================================================================================================================================
107
+ Qwen3ForCausalLM [2, 8192] -- -- True
108
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
109
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
110
+ │ └─ModuleList: 2-2 -- -- -- True
111
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
132
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
133
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
134
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
135
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
136
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
137
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
138
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
139
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
140
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
141
+ ================================================================================================================================================================
142
+ Total params: 475,645,696
143
+ Trainable params: 475,645,696
144
+ Non-trainable params: 0
145
+ Total mult-adds (Units.MEGABYTES): 951.29
146
+ ================================================================================================================================================================
147
+ Input size (MB): 0.13
148
+ Forward/backward pass size (MB): 31197.23
149
+ Params size (MB): 951.29
150
+ Estimated Total Size (MB): 32148.66
151
+ ================================================================================================================================================================
152
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
153
+ ================================================================================
154
+
155
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
156
+ Using pre-tokenized dataset (no runtime tokenization)
157
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
158
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
159
+ Dataset split: 3027764 train, 159356 validation
160
+ Training examples (packed, train split): 3027764
161
+ Validation examples (packed, val split): 159356
162
+ Per-device batch size: 2
163
+ World size (GPUs): 1
164
+ Gradient accumulation steps: 10
165
+ Effective batch size (seqs/step): 20
166
+ Sequence length: 8192
167
+ Tokens per optimizer step: 163,840
168
+ Micro-batches per epoch (per rank): 1513882
169
+ Optimizer steps per epoch: 151388
170
+ Total optimizer steps: 44000 (capped by max_steps; 0.29 epochs)
171
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-7000...
172
+ Preserving checkpoint LR state for exact resume: current=[0.0004746763942268243], schedule_base=[0.0005]
173
+ Resumed from checkpoint at step 7000 (epoch 0, micro_step 70000)
174
+ Resuming training from step 7000
175
+ wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from WANDB_API_KEY.
176
+ wandb: Currently logged in as: ruben-navasardyan4869 (ruben-navasardyan4869-epfl) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
177
+ wandb: setting up run ysa9iyov
178
+ wandb: Tracking run with wandb version 0.26.1
179
+ wandb: Run data is saved locally in /mloscratch/homes/navasard/sparse/SparseAtt/wandb/run-20260820_011128-ysa9iyov
180
+ wandb: Run `wandb offline` to turn off syncing.
181
+ wandb: Syncing run l8k365m_dense_ropectl_a1p0_seed43
182
+ wandb: ⭐️ View project at https://wandb.ai/ruben-navasardyan4869-epfl/stochastic_sparsity
183
+ wandb: 🚀 View run at https://wandb.ai/ruben-navasardyan4869-epfl/stochastic_sparsity/runs/ysa9iyov
184
+ Starting training...
185
+ Validation at step 7000: Loss = 3.1887, Perplexity = 24.26
186
+ Switching attention mode for step 7001: dense (dense step interval)
187
+ Step 7010/44000 | Loss: 3.2378 | LR: 4.75e-04 | 16610.4ms/step | 1.2 samples/s
188
+ Step 7020/44000 | Loss: 3.2161 | LR: 4.75e-04 | 1400.2ms/step | 14.3 samples/s
189
+ Step 7030/44000 | Loss: 3.1930 | LR: 4.74e-04 | 1400.1ms/step | 14.3 samples/s
190
+ Step 7040/44000 | Loss: 3.2021 | LR: 4.74e-04 | 1400.0ms/step | 14.3 samples/s
191
+ Step 7050/44000 | Loss: 3.1891 | LR: 4.74e-04 | 1400.0ms/step | 14.3 samples/s
192
+ Step 7060/44000 | Loss: 3.1711 | LR: 4.74e-04 | 1400.1ms/step | 14.3 samples/s
193
+ Step 7070/44000 | Loss: 3.2081 | LR: 4.74e-04 | 1400.0ms/step | 14.3 samples/s
194
+ Step 7080/44000 | Loss: 3.1494 | LR: 4.74e-04 | 1399.9ms/step | 14.3 samples/s
195
+ Step 7090/44000 | Loss: 3.2019 | LR: 4.74e-04 | 1400.0ms/step | 14.3 samples/s
196
+ Step 7100/44000 | Loss: 3.1880 | LR: 4.74e-04 | 1400.0ms/step | 14.3 samples/s
197
+ Step 7110/44000 | Loss: 3.2000 | LR: 4.74e-04 | 1400.0ms/step | 14.3 samples/s
198
+ Step 7120/44000 | Loss: 3.2023 | LR: 4.74e-04 | 1399.9ms/step | 14.3 samples/s
199
+ Step 7130/44000 | Loss: 3.2013 | LR: 4.74e-04 | 1400.3ms/step | 14.3 samples/s
200
+ Step 7140/44000 | Loss: 3.1745 | LR: 4.74e-04 | 1400.3ms/step | 14.3 samples/s
201
+ Step 7150/44000 | Loss: 3.2055 | LR: 4.73e-04 | 1399.9ms/step | 14.3 samples/s
202
+ Step 7160/44000 | Loss: 3.1701 | LR: 4.73e-04 | 1399.6ms/step | 14.3 samples/s
203
+ Step 7170/44000 | Loss: 3.1819 | LR: 4.73e-04 | 1400.1ms/step | 14.3 samples/s
204
+ Step 7180/44000 | Loss: 3.2145 | LR: 4.73e-04 | 1400.0ms/step | 14.3 samples/s
205
+ Step 7190/44000 | Loss: 3.2191 | LR: 4.73e-04 | 1399.8ms/step | 14.3 samples/s
206
+ Step 7200/44000 | Loss: 3.2221 | LR: 4.73e-04 | 1399.9ms/step | 14.3 samples/s
207
+ Step 7210/44000 | Loss: 3.1904 | LR: 4.73e-04 | 1400.5ms/step | 14.3 samples/s
208
+ Step 7220/44000 | Loss: 3.2112 | LR: 4.73e-04 | 1401.8ms/step | 14.3 samples/s
209
+ Step 7230/44000 | Loss: 3.1888 | LR: 4.73e-04 | 1401.2ms/step | 14.3 samples/s
210
+ Step 7240/44000 | Loss: 3.2273 | LR: 4.73e-04 | 1404.1ms/step | 14.2 samples/s
211
+ Step 7250/44000 | Loss: 3.1857 | LR: 4.73e-04 | 1404.8ms/step | 14.2 samples/s
212
+ Step 7260/44000 | Loss: 3.1968 | LR: 4.73e-04 | 1406.1ms/step | 14.2 samples/s
213
+ Step 7270/44000 | Loss: 3.1982 | LR: 4.72e-04 | 1402.1ms/step | 14.3 samples/s
214
+ Step 7280/44000 | Loss: 3.1973 | LR: 4.72e-04 | 1399.7ms/step | 14.3 samples/s
215
+ Step 7290/44000 | Loss: 3.1980 | LR: 4.72e-04 | 1400.1ms/step | 14.3 samples/s
216
+ Step 7300/44000 | Loss: 3.1992 | LR: 4.72e-04 | 1399.8ms/step | 14.3 samples/s
217
+ Step 7310/44000 | Loss: 3.2175 | LR: 4.72e-04 | 1399.9ms/step | 14.3 samples/s
218
+ Step 7320/44000 | Loss: 3.1606 | LR: 4.72e-04 | 1400.1ms/step | 14.3 samples/s
219
+ Step 7330/44000 | Loss: 3.1815 | LR: 4.72e-04 | 1401.9ms/step | 14.3 samples/s
220
+ Step 7340/44000 | Loss: 3.1827 | LR: 4.72e-04 | 1400.1ms/step | 14.3 samples/s
221
+ Step 7350/44000 | Loss: 3.1640 | LR: 4.72e-04 | 1404.0ms/step | 14.2 samples/s
222
+ Step 7360/44000 | Loss: 3.2107 | LR: 4.72e-04 | 1400.5ms/step | 14.3 samples/s
223
+ Step 7370/44000 | Loss: 3.1788 | LR: 4.72e-04 | 1408.4ms/step | 14.2 samples/s
224
+ Step 7380/44000 | Loss: 3.2138 | LR: 4.72e-04 | 1414.2ms/step | 14.1 samples/s
225
+ Step 7390/44000 | Loss: 3.1946 | LR: 4.71e-04 | 1413.2ms/step | 14.2 samples/s
226
+ Step 7400/44000 | Loss: 3.1901 | LR: 4.71e-04 | 1408.4ms/step | 14.2 samples/s
227
+ Step 7410/44000 | Loss: 3.2026 | LR: 4.71e-04 | 1409.3ms/step | 14.2 samples/s
228
+ Step 7420/44000 | Loss: 3.1861 | LR: 4.71e-04 | 1414.2ms/step | 14.1 samples/s
229
+ Step 7430/44000 | Loss: 3.2039 | LR: 4.71e-04 | 1416.1ms/step | 14.1 samples/s
230
+ Step 7440/44000 | Loss: 3.2059 | LR: 4.71e-04 | 1415.0ms/step | 14.1 samples/s
231
+ Step 7450/44000 | Loss: 3.1806 | LR: 4.71e-04 | 1414.1ms/step | 14.1 samples/s
232
+ Step 7460/44000 | Loss: 3.1856 | LR: 4.71e-04 | 1412.9ms/step | 14.2 samples/s
233
+ Step 7470/44000 | Loss: 3.1765 | LR: 4.71e-04 | 1410.5ms/step | 14.2 samples/s
234
+ Step 7480/44000 | Loss: 3.1932 | LR: 4.71e-04 | 1407.6ms/step | 14.2 samples/s
235
+ Step 7490/44000 | Loss: 3.1632 | LR: 4.71e-04 | 1404.3ms/step | 14.2 samples/s
236
+ Step 7500/44000 | Loss: 3.1904 | LR: 4.71e-04 | 1406.2ms/step | 14.2 samples/s
237
+ Step 7510/44000 | Loss: 3.1920 | LR: 4.70e-04 | 1406.9ms/step | 14.2 samples/s
238
+ Step 7520/44000 | Loss: 3.1792 | LR: 4.70e-04 | 1406.2ms/step | 14.2 samples/s
239
+ Step 7530/44000 | Loss: 3.2126 | LR: 4.70e-04 | 1399.9ms/step | 14.3 samples/s
240
+ Step 7540/44000 | Loss: 3.1964 | LR: 4.70e-04 | 1400.3ms/step | 14.3 samples/s
241
+ Step 7550/44000 | Loss: 3.1889 | LR: 4.70e-04 | 1401.7ms/step | 14.3 samples/s
242
+ Step 7560/44000 | Loss: 3.1665 | LR: 4.70e-04 | 1402.1ms/step | 14.3 samples/s
243
+ Step 7570/44000 | Loss: 3.1858 | LR: 4.70e-04 | 1404.1ms/step | 14.2 samples/s
244
+ Step 7580/44000 | Loss: 3.1450 | LR: 4.70e-04 | 1405.4ms/step | 14.2 samples/s
245
+ Step 7590/44000 | Loss: 3.1720 | LR: 4.70e-04 | 1405.2ms/step | 14.2 samples/s
246
+ Step 7600/44000 | Loss: 3.1793 | LR: 4.70e-04 | 1407.3ms/step | 14.2 samples/s
247
+ Step 7610/44000 | Loss: 3.1825 | LR: 4.70e-04 | 1402.1ms/step | 14.3 samples/s
248
+ Step 7620/44000 | Loss: 3.1902 | LR: 4.70e-04 | 1402.8ms/step | 14.3 samples/s
249
+ Step 7630/44000 | Loss: 3.1696 | LR: 4.69e-04 | 1400.8ms/step | 14.3 samples/s
250
+ Step 7640/44000 | Loss: 3.1666 | LR: 4.69e-04 | 1400.0ms/step | 14.3 samples/s
251
+ Step 7650/44000 | Loss: 3.1633 | LR: 4.69e-04 | 1402.7ms/step | 14.3 samples/s
252
+ Step 7660/44000 | Loss: 3.1983 | LR: 4.69e-04 | 1399.6ms/step | 14.3 samples/s
253
+ Step 7670/44000 | Loss: 3.1791 | LR: 4.69e-04 | 1400.2ms/step | 14.3 samples/s
254
+ Step 7680/44000 | Loss: 3.1686 | LR: 4.69e-04 | 1401.8ms/step | 14.3 samples/s
255
+ Step 7690/44000 | Loss: 3.1692 | LR: 4.69e-04 | 1405.8ms/step | 14.2 samples/s
256
+ Step 7700/44000 | Loss: 3.2092 | LR: 4.69e-04 | 1402.1ms/step | 14.3 samples/s
257
+ Step 7710/44000 | Loss: 3.1626 | LR: 4.69e-04 | 1402.1ms/step | 14.3 samples/s
258
+ Step 7720/44000 | Loss: 3.1576 | LR: 4.69e-04 | 1402.2ms/step | 14.3 samples/s
259
+ Step 7730/44000 | Loss: 3.1353 | LR: 4.69e-04 | 1402.0ms/step | 14.3 samples/s
260
+ Step 7740/44000 | Loss: 3.1613 | LR: 4.68e-04 | 1400.2ms/step | 14.3 samples/s
261
+ Step 7750/44000 | Loss: 3.2006 | LR: 4.68e-04 | 1404.0ms/step | 14.2 samples/s
262
+ Step 7760/44000 | Loss: 3.1672 | LR: 4.68e-04 | 1400.0ms/step | 14.3 samples/s
263
+ Step 7770/44000 | Loss: 3.1725 | LR: 4.68e-04 | 1402.9ms/step | 14.3 samples/s
264
+ Step 7780/44000 | Loss: 3.1748 | LR: 4.68e-04 | 1401.3ms/step | 14.3 samples/s
265
+ Step 7790/44000 | Loss: 3.1730 | LR: 4.68e-04 | 1401.2ms/step | 14.3 samples/s
266
+ Step 7800/44000 | Loss: 3.1990 | LR: 4.68e-04 | 1400.0ms/step | 14.3 samples/s
267
+ Step 7810/44000 | Loss: 3.1459 | LR: 4.68e-04 | 1400.4ms/step | 14.3 samples/s
268
+ Step 7820/44000 | Loss: 3.1935 | LR: 4.68e-04 | 1401.9ms/step | 14.3 samples/s
269
+ Step 7830/44000 | Loss: 3.1653 | LR: 4.68e-04 | 1399.8ms/step | 14.3 samples/s
270
+ Step 7840/44000 | Loss: 3.1482 | LR: 4.68e-04 | 1400.1ms/step | 14.3 samples/s
271
+ Step 7850/44000 | Loss: 3.1597 | LR: 4.67e-04 | 1400.4ms/step | 14.3 samples/s
272
+ Step 7860/44000 | Loss: 3.1516 | LR: 4.67e-04 | 1403.7ms/step | 14.2 samples/s
273
+ Step 7870/44000 | Loss: 3.1740 | LR: 4.67e-04 | 1400.1ms/step | 14.3 samples/s
274
+ Step 7880/44000 | Loss: 3.1948 | LR: 4.67e-04 | 1399.5ms/step | 14.3 samples/s
275
+ Step 7890/44000 | Loss: 3.1725 | LR: 4.67e-04 | 1400.2ms/step | 14.3 samples/s
276
+ Step 7900/44000 | Loss: 3.1784 | LR: 4.67e-04 | 1400.3ms/step | 14.3 samples/s
277
+ Step 7910/44000 | Loss: 3.1622 | LR: 4.67e-04 | 1401.8ms/step | 14.3 samples/s
278
+ Step 7920/44000 | Loss: 3.1737 | LR: 4.67e-04 | 1399.9ms/step | 14.3 samples/s
279
+ Step 7930/44000 | Loss: 3.1807 | LR: 4.67e-04 | 1400.0ms/step | 14.3 samples/s
280
+ Step 7940/44000 | Loss: 3.1674 | LR: 4.67e-04 | 1399.9ms/step | 14.3 samples/s
281
+ Step 7950/44000 | Loss: 3.1637 | LR: 4.67e-04 | 1400.1ms/step | 14.3 samples/s
282
+ Step 7960/44000 | Loss: 3.1523 | LR: 4.66e-04 | 1400.1ms/step | 14.3 samples/s
283
+ Step 7970/44000 | Loss: 3.1872 | LR: 4.66e-04 | 1399.9ms/step | 14.3 samples/s
284
+ Step 7980/44000 | Loss: 3.1797 | LR: 4.66e-04 | 1400.2ms/step | 14.3 samples/s
285
+ Step 7990/44000 | Loss: 3.1744 | LR: 4.66e-04 | 1399.9ms/step | 14.3 samples/s
286
+ Step 8000/44000 | Loss: 3.1619 | LR: 4.66e-04 | 1400.0ms/step | 14.3 samples/s
287
+ Validation at step 8000: Loss = 3.1563, Perplexity = 23.48
288
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-8000
289
+ Step 8010/44000 | Loss: 3.1459 | LR: 4.66e-04 | 3908.1ms/step | 5.1 samples/s
290
+ Step 8020/44000 | Loss: 3.1432 | LR: 4.66e-04 | 1404.6ms/step | 14.2 samples/s
291
+ Step 8030/44000 | Loss: 3.1785 | LR: 4.66e-04 | 1406.6ms/step | 14.2 samples/s
292
+ Step 8040/44000 | Loss: 3.1637 | LR: 4.66e-04 | 1402.9ms/step | 14.3 samples/s
293
+ Step 8050/44000 | Loss: 3.1979 | LR: 4.66e-04 | 1405.0ms/step | 14.2 samples/s
294
+ Step 8060/44000 | Loss: 3.1732 | LR: 4.66e-04 | 1407.1ms/step | 14.2 samples/s
295
+ Step 8070/44000 | Loss: 3.1650 | LR: 4.65e-04 | 1399.8ms/step | 14.3 samples/s
296
+ Step 8080/44000 | Loss: 3.1598 | LR: 4.65e-04 | 1402.4ms/step | 14.3 samples/s
297
+ Step 8090/44000 | Loss: 3.1856 | LR: 4.65e-04 | 1399.5ms/step | 14.3 samples/s
298
+ Step 8100/44000 | Loss: 3.1875 | LR: 4.65e-04 | 1400.1ms/step | 14.3 samples/s
299
+ Step 8110/44000 | Loss: 3.1602 | LR: 4.65e-04 | 1403.1ms/step | 14.3 samples/s
300
+ Step 8120/44000 | Loss: 3.1507 | LR: 4.65e-04 | 1404.7ms/step | 14.2 samples/s
301
+ Step 8130/44000 | Loss: 3.1630 | LR: 4.65e-04 | 1400.9ms/step | 14.3 samples/s
302
+ Step 8140/44000 | Loss: 3.1921 | LR: 4.65e-04 | 1402.0ms/step | 14.3 samples/s
303
+ Step 8150/44000 | Loss: 3.1427 | LR: 4.65e-04 | 1403.5ms/step | 14.3 samples/s
304
+ Step 8160/44000 | Loss: 3.1760 | LR: 4.65e-04 | 1404.2ms/step | 14.2 samples/s
305
+ Step 8170/44000 | Loss: 3.1352 | LR: 4.65e-04 | 1405.8ms/step | 14.2 samples/s
306
+ Step 8180/44000 | Loss: 3.1670 | LR: 4.64e-04 | 1400.0ms/step | 14.3 samples/s
307
+ Step 8190/44000 | Loss: 3.1871 | LR: 4.64e-04 | 1399.8ms/step | 14.3 samples/s
308
+ Step 8200/44000 | Loss: 3.1693 | LR: 4.64e-04 | 1401.0ms/step | 14.3 samples/s
309
+ Step 8210/44000 | Loss: 3.1524 | LR: 4.64e-04 | 1403.7ms/step | 14.2 samples/s
310
+ Step 8220/44000 | Loss: 3.1969 | LR: 4.64e-04 | 1407.3ms/step | 14.2 samples/s
311
+ Step 8230/44000 | Loss: 3.1496 | LR: 4.64e-04 | 1410.8ms/step | 14.2 samples/s
312
+ Step 8240/44000 | Loss: 3.1233 | LR: 4.64e-04 | 1407.9ms/step | 14.2 samples/s
313
+ Step 8250/44000 | Loss: 3.1729 | LR: 4.64e-04 | 1409.8ms/step | 14.2 samples/s
314
+ Step 8260/44000 | Loss: 3.1513 | LR: 4.64e-04 | 1407.8ms/step | 14.2 samples/s
315
+ Step 8270/44000 | Loss: 3.1564 | LR: 4.64e-04 | 1402.0ms/step | 14.3 samples/s
316
+ Step 8280/44000 | Loss: 3.1445 | LR: 4.64e-04 | 1406.9ms/step | 14.2 samples/s
317
+ Step 8290/44000 | Loss: 3.1641 | LR: 4.63e-04 | 1407.6ms/step | 14.2 samples/s
318
+ Step 8300/44000 | Loss: 3.1809 | LR: 4.63e-04 | 1410.4ms/step | 14.2 samples/s
319
+ Step 8310/44000 | Loss: 3.1408 | LR: 4.63e-04 | 1409.3ms/step | 14.2 samples/s
320
+ Step 8320/44000 | Loss: 3.1668 | LR: 4.63e-04 | 1407.8ms/step | 14.2 samples/s
321
+ Step 8330/44000 | Loss: 3.1570 | LR: 4.63e-04 | 1402.1ms/step | 14.3 samples/s
322
+ Step 8340/44000 | Loss: 3.1659 | LR: 4.63e-04 | 1401.7ms/step | 14.3 samples/s
323
+ Step 8350/44000 | Loss: 3.1385 | LR: 4.63e-04 | 1405.4ms/step | 14.2 samples/s
324
+ Step 8360/44000 | Loss: 3.1767 | LR: 4.63e-04 | 1407.3ms/step | 14.2 samples/s
325
+ Step 8370/44000 | Loss: 3.1355 | LR: 4.63e-04 | 1407.5ms/step | 14.2 samples/s
326
+ Step 8380/44000 | Loss: 3.1747 | LR: 4.63e-04 | 1407.8ms/step | 14.2 samples/s
327
+ Step 8390/44000 | Loss: 3.1449 | LR: 4.62e-04 | 1408.7ms/step | 14.2 samples/s
328
+ Step 8400/44000 | Loss: 3.1249 | LR: 4.62e-04 | 1407.6ms/step | 14.2 samples/s
329
+ Step 8410/44000 | Loss: 3.1510 | LR: 4.62e-04 | 1409.7ms/step | 14.2 samples/s
330
+ Step 8420/44000 | Loss: 3.1495 | LR: 4.62e-04 | 1405.9ms/step | 14.2 samples/s
331
+ Step 8430/44000 | Loss: 3.1598 | LR: 4.62e-04 | 1402.2ms/step | 14.3 samples/s
332
+ Step 8440/44000 | Loss: 3.1814 | LR: 4.62e-04 | 1405.9ms/step | 14.2 samples/s
333
+ Step 8450/44000 | Loss: 3.1653 | LR: 4.62e-04 | 1409.0ms/step | 14.2 samples/s
334
+ Step 8460/44000 | Loss: 3.1412 | LR: 4.62e-04 | 1409.4ms/step | 14.2 samples/s
335
+ Step 8470/44000 | Loss: 3.1490 | LR: 4.62e-04 | 1411.8ms/step | 14.2 samples/s
336
+ Step 8480/44000 | Loss: 3.1478 | LR: 4.62e-04 | 1407.9ms/step | 14.2 samples/s
337
+ Step 8490/44000 | Loss: 3.1341 | LR: 4.62e-04 | 1408.0ms/step | 14.2 samples/s
338
+ Step 8500/44000 | Loss: 3.1498 | LR: 4.61e-04 | 1410.3ms/step | 14.2 samples/s
339
+ Step 8510/44000 | Loss: 3.1437 | LR: 4.61e-04 | 1409.0ms/step | 14.2 samples/s
340
+ Step 8520/44000 | Loss: 3.1271 | LR: 4.61e-04 | 1412.0ms/step | 14.2 samples/s
341
+ Step 8530/44000 | Loss: 3.1414 | LR: 4.61e-04 | 1410.6ms/step | 14.2 samples/s
342
+ Step 8540/44000 | Loss: 3.1360 | LR: 4.61e-04 | 1408.1ms/step | 14.2 samples/s
343
+ Step 8550/44000 | Loss: 3.1629 | LR: 4.61e-04 | 1400.0ms/step | 14.3 samples/s
344
+ Step 8560/44000 | Loss: 3.1589 | LR: 4.61e-04 | 1400.0ms/step | 14.3 samples/s
345
+ Step 8570/44000 | Loss: 3.1710 | LR: 4.61e-04 | 1404.1ms/step | 14.2 samples/s
346
+ Step 8580/44000 | Loss: 3.1549 | LR: 4.61e-04 | 1405.2ms/step | 14.2 samples/s
347
+ Step 8590/44000 | Loss: 3.1574 | LR: 4.61e-04 | 1408.8ms/step | 14.2 samples/s
348
+ Step 8600/44000 | Loss: 3.1681 | LR: 4.60e-04 | 1403.7ms/step | 14.2 samples/s
349
+ Step 8610/44000 | Loss: 3.1451 | LR: 4.60e-04 | 1400.0ms/step | 14.3 samples/s
350
+ Step 8620/44000 | Loss: 3.1428 | LR: 4.60e-04 | 1400.1ms/step | 14.3 samples/s
351
+ Step 8630/44000 | Loss: 3.1948 | LR: 4.60e-04 | 1400.1ms/step | 14.3 samples/s
352
+ Step 8640/44000 | Loss: 3.1635 | LR: 4.60e-04 | 1400.0ms/step | 14.3 samples/s
353
+ Step 8650/44000 | Loss: 3.1668 | LR: 4.60e-04 | 1400.1ms/step | 14.3 samples/s
354
+ Step 8660/44000 | Loss: 3.1750 | LR: 4.60e-04 | 1399.7ms/step | 14.3 samples/s
355
+ Step 8670/44000 | Loss: 3.1501 | LR: 4.60e-04 | 1400.3ms/step | 14.3 samples/s
356
+ Step 8680/44000 | Loss: 3.1544 | LR: 4.60e-04 | 1401.8ms/step | 14.3 samples/s
357
+ Step 8690/44000 | Loss: 3.1439 | LR: 4.60e-04 | 1400.0ms/step | 14.3 samples/s
358
+ Step 8700/44000 | Loss: 3.1635 | LR: 4.59e-04 | 1400.1ms/step | 14.3 samples/s
359
+ Step 8710/44000 | Loss: 3.1489 | LR: 4.59e-04 | 1399.9ms/step | 14.3 samples/s
360
+ Step 8720/44000 | Loss: 3.1516 | LR: 4.59e-04 | 1402.0ms/step | 14.3 samples/s
361
+ Step 8730/44000 | Loss: 3.1324 | LR: 4.59e-04 | 1402.3ms/step | 14.3 samples/s
362
+ Step 8740/44000 | Loss: 3.1523 | LR: 4.59e-04 | 1399.7ms/step | 14.3 samples/s
363
+ Step 8750/44000 | Loss: 3.1518 | LR: 4.59e-04 | 1402.3ms/step | 14.3 samples/s
364
+ Step 8760/44000 | Loss: 3.1588 | LR: 4.59e-04 | 1403.7ms/step | 14.2 samples/s
365
+ Step 8770/44000 | Loss: 3.1553 | LR: 4.59e-04 | 1402.5ms/step | 14.3 samples/s
366
+ Step 8780/44000 | Loss: 3.1433 | LR: 4.59e-04 | 1399.8ms/step | 14.3 samples/s
367
+ Step 8790/44000 | Loss: 3.1506 | LR: 4.59e-04 | 1403.8ms/step | 14.2 samples/s
368
+ Step 8800/44000 | Loss: 3.1292 | LR: 4.58e-04 | 1403.9ms/step | 14.2 samples/s
369
+ Step 8810/44000 | Loss: 3.1508 | LR: 4.58e-04 | 1404.4ms/step | 14.2 samples/s
370
+ Step 8820/44000 | Loss: 3.1126 | LR: 4.58e-04 | 1403.9ms/step | 14.2 samples/s
371
+ Step 8830/44000 | Loss: 3.1109 | LR: 4.58e-04 | 1403.9ms/step | 14.2 samples/s
372
+ Step 8840/44000 | Loss: 3.1422 | LR: 4.58e-04 | 1404.6ms/step | 14.2 samples/s
373
+ Step 8850/44000 | Loss: 3.1418 | LR: 4.58e-04 | 1405.9ms/step | 14.2 samples/s
374
+ Step 8860/44000 | Loss: 3.1213 | LR: 4.58e-04 | 1408.0ms/step | 14.2 samples/s
375
+ Step 8870/44000 | Loss: 3.1615 | LR: 4.58e-04 | 1407.5ms/step | 14.2 samples/s
376
+ Step 8880/44000 | Loss: 3.1442 | LR: 4.58e-04 | 1403.8ms/step | 14.2 samples/s
377
+ Step 8890/44000 | Loss: 3.1357 | LR: 4.58e-04 | 1400.1ms/step | 14.3 samples/s
378
+ Step 8900/44000 | Loss: 3.1370 | LR: 4.57e-04 | 1404.8ms/step | 14.2 samples/s
379
+ Step 8910/44000 | Loss: 3.1190 | LR: 4.57e-04 | 1405.3ms/step | 14.2 samples/s
380
+ Step 8920/44000 | Loss: 3.1601 | LR: 4.57e-04 | 1404.2ms/step | 14.2 samples/s
381
+ Step 8930/44000 | Loss: 3.1652 | LR: 4.57e-04 | 1406.1ms/step | 14.2 samples/s
382
+ Step 8940/44000 | Loss: 3.1143 | LR: 4.57e-04 | 1407.7ms/step | 14.2 samples/s
383
+ Step 8950/44000 | Loss: 3.1463 | LR: 4.57e-04 | 1402.3ms/step | 14.3 samples/s
384
+ Step 8960/44000 | Loss: 3.1255 | LR: 4.57e-04 | 1408.1ms/step | 14.2 samples/s
385
+ Step 8970/44000 | Loss: 3.1358 | LR: 4.57e-04 | 1407.5ms/step | 14.2 samples/s
386
+ Step 8980/44000 | Loss: 3.1236 | LR: 4.57e-04 | 1401.6ms/step | 14.3 samples/s
387
+ Step 8990/44000 | Loss: 3.1252 | LR: 4.57e-04 | 1400.3ms/step | 14.3 samples/s
388
+ Step 9000/44000 | Loss: 3.1499 | LR: 4.56e-04 | 1401.9ms/step | 14.3 samples/s
389
+ Validation at step 9000: Loss = 3.1276, Perplexity = 22.82
390
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-9000
391
+ Step 9010/44000 | Loss: 3.1391 | LR: 4.56e-04 | 3930.4ms/step | 5.1 samples/s
392
+ Step 9020/44000 | Loss: 3.1061 | LR: 4.56e-04 | 1403.9ms/step | 14.2 samples/s
393
+ Step 9030/44000 | Loss: 3.1733 | LR: 4.56e-04 | 1401.8ms/step | 14.3 samples/s
394
+ Step 9040/44000 | Loss: 3.1372 | LR: 4.56e-04 | 1404.7ms/step | 14.2 samples/s
395
+ Step 9050/44000 | Loss: 3.1105 | LR: 4.56e-04 | 1401.3ms/step | 14.3 samples/s
396
+ Step 9060/44000 | Loss: 3.1341 | LR: 4.56e-04 | 1401.2ms/step | 14.3 samples/s
397
+ Step 9070/44000 | Loss: 3.1049 | LR: 4.56e-04 | 1400.7ms/step | 14.3 samples/s
398
+ Step 9080/44000 | Loss: 3.1517 | LR: 4.56e-04 | 1400.1ms/step | 14.3 samples/s
399
+ Step 9090/44000 | Loss: 3.1421 | LR: 4.56e-04 | 1401.9ms/step | 14.3 samples/s
400
+ Step 9100/44000 | Loss: 3.1232 | LR: 4.55e-04 | 1402.7ms/step | 14.3 samples/s
401
+ Step 9110/44000 | Loss: 3.1191 | LR: 4.55e-04 | 1405.5ms/step | 14.2 samples/s
402
+ Step 9120/44000 | Loss: 3.1411 | LR: 4.55e-04 | 1402.2ms/step | 14.3 samples/s
403
+ Step 9130/44000 | Loss: 3.1191 | LR: 4.55e-04 | 1403.6ms/step | 14.2 samples/s
404
+ Step 9140/44000 | Loss: 3.0964 | LR: 4.55e-04 | 1408.9ms/step | 14.2 samples/s
405
+ Step 9150/44000 | Loss: 3.1481 | LR: 4.55e-04 | 1409.7ms/step | 14.2 samples/s
406
+ Step 9160/44000 | Loss: 3.1581 | LR: 4.55e-04 | 1409.6ms/step | 14.2 samples/s
407
+ Step 9170/44000 | Loss: 3.1439 | LR: 4.55e-04 | 1410.3ms/step | 14.2 samples/s
408
+ Step 9180/44000 | Loss: 3.1059 | LR: 4.55e-04 | 1410.0ms/step | 14.2 samples/s
409
+ Step 9190/44000 | Loss: 3.1228 | LR: 4.54e-04 | 1409.9ms/step | 14.2 samples/s
410
+ Step 9200/44000 | Loss: 3.1262 | LR: 4.54e-04 | 1408.3ms/step | 14.2 samples/s
411
+ Step 9210/44000 | Loss: 3.1231 | LR: 4.54e-04 | 1410.7ms/step | 14.2 samples/s
412
+ Step 9220/44000 | Loss: 3.1166 | LR: 4.54e-04 | 1409.1ms/step | 14.2 samples/s
413
+ Step 9230/44000 | Loss: 3.1366 | LR: 4.54e-04 | 1408.1ms/step | 14.2 samples/s
414
+ Step 9240/44000 | Loss: 3.1300 | LR: 4.54e-04 | 1409.1ms/step | 14.2 samples/s
415
+ Step 9250/44000 | Loss: 3.1313 | LR: 4.54e-04 | 1409.7ms/step | 14.2 samples/s
416
+ Step 9260/44000 | Loss: 3.1668 | LR: 4.54e-04 | 1410.7ms/step | 14.2 samples/s
417
+ Step 9270/44000 | Loss: 3.1387 | LR: 4.54e-04 | 1412.4ms/step | 14.2 samples/s
418
+ Step 9280/44000 | Loss: 3.1438 | LR: 4.54e-04 | 1413.7ms/step | 14.1 samples/s
419
+ Step 9290/44000 | Loss: 3.1241 | LR: 4.53e-04 | 1414.8ms/step | 14.1 samples/s
420
+ Step 9300/44000 | Loss: 3.0910 | LR: 4.53e-04 | 1414.6ms/step | 14.1 samples/s
421
+ Step 9310/44000 | Loss: 3.1358 | LR: 4.53e-04 | 1411.7ms/step | 14.2 samples/s
422
+ Step 9320/44000 | Loss: 3.1396 | LR: 4.53e-04 | 1411.3ms/step | 14.2 samples/s
423
+ Step 9330/44000 | Loss: 3.1287 | LR: 4.53e-04 | 1410.0ms/step | 14.2 samples/s
424
+ Step 9340/44000 | Loss: 3.1039 | LR: 4.53e-04 | 1410.5ms/step | 14.2 samples/s
425
+ Step 9350/44000 | Loss: 3.1361 | LR: 4.53e-04 | 1411.1ms/step | 14.2 samples/s
426
+ Step 9360/44000 | Loss: 3.1379 | LR: 4.53e-04 | 1406.7ms/step | 14.2 samples/s
427
+ Step 9370/44000 | Loss: 3.1155 | LR: 4.53e-04 | 1409.3ms/step | 14.2 samples/s
428
+ Step 9380/44000 | Loss: 3.1364 | LR: 4.52e-04 | 1406.1ms/step | 14.2 samples/s
429
+ Step 9390/44000 | Loss: 3.1441 | LR: 4.52e-04 | 1403.9ms/step | 14.2 samples/s
430
+ Step 9400/44000 | Loss: 3.1410 | LR: 4.52e-04 | 1404.1ms/step | 14.2 samples/s
431
+ Step 9410/44000 | Loss: 3.1347 | LR: 4.52e-04 | 1408.2ms/step | 14.2 samples/s
432
+ Step 9420/44000 | Loss: 3.1305 | LR: 4.52e-04 | 1409.9ms/step | 14.2 samples/s
433
+ Step 9430/44000 | Loss: 3.1292 | LR: 4.52e-04 | 1410.6ms/step | 14.2 samples/s
434
+ Step 9440/44000 | Loss: 3.1208 | LR: 4.52e-04 | 1410.2ms/step | 14.2 samples/s
435
+ Step 9450/44000 | Loss: 3.1344 | LR: 4.52e-04 | 1409.5ms/step | 14.2 samples/s
436
+ Step 9460/44000 | Loss: 3.1122 | LR: 4.52e-04 | 1410.8ms/step | 14.2 samples/s
437
+ Step 9470/44000 | Loss: 3.1421 | LR: 4.52e-04 | 1403.0ms/step | 14.3 samples/s
438
+ Step 9480/44000 | Loss: 3.1245 | LR: 4.51e-04 | 1402.5ms/step | 14.3 samples/s
439
+ Step 9490/44000 | Loss: 3.1188 | LR: 4.51e-04 | 1405.5ms/step | 14.2 samples/s
440
+ Step 9500/44000 | Loss: 3.1198 | LR: 4.51e-04 | 1408.4ms/step | 14.2 samples/s
441
+ Step 9510/44000 | Loss: 3.1178 | LR: 4.51e-04 | 1408.0ms/step | 14.2 samples/s
442
+ Step 9520/44000 | Loss: 3.1111 | LR: 4.51e-04 | 1409.9ms/step | 14.2 samples/s
443
+ Step 9530/44000 | Loss: 3.1394 | LR: 4.51e-04 | 1409.8ms/step | 14.2 samples/s
444
+ Step 9540/44000 | Loss: 3.1264 | LR: 4.51e-04 | 1406.7ms/step | 14.2 samples/s
445
+ Step 9550/44000 | Loss: 3.0934 | LR: 4.51e-04 | 1402.8ms/step | 14.3 samples/s
446
+ Step 9560/44000 | Loss: 3.1112 | LR: 4.51e-04 | 1402.1ms/step | 14.3 samples/s
447
+ Step 9570/44000 | Loss: 3.1347 | LR: 4.50e-04 | 1403.9ms/step | 14.2 samples/s
448
+ Step 9580/44000 | Loss: 3.1207 | LR: 4.50e-04 | 1404.3ms/step | 14.2 samples/s
449
+ Step 9590/44000 | Loss: 3.1159 | LR: 4.50e-04 | 1403.8ms/step | 14.2 samples/s
450
+ Step 9600/44000 | Loss: 3.1566 | LR: 4.50e-04 | 1402.1ms/step | 14.3 samples/s
451
+ Step 9610/44000 | Loss: 3.1396 | LR: 4.50e-04 | 1402.1ms/step | 14.3 samples/s
452
+ Step 9620/44000 | Loss: 3.1584 | LR: 4.50e-04 | 1401.9ms/step | 14.3 samples/s
453
+ Step 9630/44000 | Loss: 3.0924 | LR: 4.50e-04 | 1399.9ms/step | 14.3 samples/s
454
+ Step 9640/44000 | Loss: 3.1430 | LR: 4.50e-04 | 1399.9ms/step | 14.3 samples/s
455
+ Step 9650/44000 | Loss: 3.1381 | LR: 4.50e-04 | 1400.4ms/step | 14.3 samples/s
456
+ Step 9660/44000 | Loss: 3.1325 | LR: 4.49e-04 | 1399.6ms/step | 14.3 samples/s
457
+ Step 9670/44000 | Loss: 3.0682 | LR: 4.49e-04 | 1400.1ms/step | 14.3 samples/s
458
+ Step 9680/44000 | Loss: 3.1254 | LR: 4.49e-04 | 1402.3ms/step | 14.3 samples/s
459
+ Step 9690/44000 | Loss: 3.1294 | LR: 4.49e-04 | 1402.7ms/step | 14.3 samples/s
460
+ Step 9700/44000 | Loss: 3.1663 | LR: 4.49e-04 | 1401.1ms/step | 14.3 samples/s
461
+ Step 9710/44000 | Loss: 3.1534 | LR: 4.49e-04 | 1402.4ms/step | 14.3 samples/s
462
+ Step 9720/44000 | Loss: 3.0960 | LR: 4.49e-04 | 1407.7ms/step | 14.2 samples/s
463
+ Step 9730/44000 | Loss: 3.1048 | LR: 4.49e-04 | 1405.9ms/step | 14.2 samples/s
464
+ Step 9740/44000 | Loss: 3.1167 | LR: 4.49e-04 | 1399.8ms/step | 14.3 samples/s
465
+ Step 9750/44000 | Loss: 3.1323 | LR: 4.48e-04 | 1400.9ms/step | 14.3 samples/s
466
+ Step 9760/44000 | Loss: 3.0747 | LR: 4.48e-04 | 1401.5ms/step | 14.3 samples/s
467
+ Step 9770/44000 | Loss: 3.1140 | LR: 4.48e-04 | 1399.9ms/step | 14.3 samples/s
468
+ Step 9780/44000 | Loss: 3.1137 | LR: 4.48e-04 | 1401.9ms/step | 14.3 samples/s
469
+ Step 9790/44000 | Loss: 3.1068 | LR: 4.48e-04 | 1406.6ms/step | 14.2 samples/s
470
+ Step 9800/44000 | Loss: 3.1330 | LR: 4.48e-04 | 1404.4ms/step | 14.2 samples/s
471
+ Step 9810/44000 | Loss: 3.1034 | LR: 4.48e-04 | 1405.3ms/step | 14.2 samples/s
472
+ Step 9820/44000 | Loss: 3.1293 | LR: 4.48e-04 | 1402.6ms/step | 14.3 samples/s
473
+ Step 9830/44000 | Loss: 3.1044 | LR: 4.48e-04 | 1402.5ms/step | 14.3 samples/s
474
+ Step 9840/44000 | Loss: 3.1018 | LR: 4.47e-04 | 1400.5ms/step | 14.3 samples/s
475
+ Step 9850/44000 | Loss: 3.1160 | LR: 4.47e-04 | 1400.4ms/step | 14.3 samples/s
476
+ Step 9860/44000 | Loss: 3.1473 | LR: 4.47e-04 | 1402.1ms/step | 14.3 samples/s
477
+ Step 9870/44000 | Loss: 3.1385 | LR: 4.47e-04 | 1402.1ms/step | 14.3 samples/s
478
+ Step 9880/44000 | Loss: 3.1358 | LR: 4.47e-04 | 1404.7ms/step | 14.2 samples/s
479
+ Step 9890/44000 | Loss: 3.1261 | LR: 4.47e-04 | 1406.9ms/step | 14.2 samples/s
480
+ Step 9900/44000 | Loss: 3.1528 | LR: 4.47e-04 | 1404.9ms/step | 14.2 samples/s
481
+ Step 9910/44000 | Loss: 3.1018 | LR: 4.47e-04 | 1407.4ms/step | 14.2 samples/s
482
+ Step 9920/44000 | Loss: 3.1114 | LR: 4.47e-04 | 1402.5ms/step | 14.3 samples/s
483
+ Step 9930/44000 | Loss: 3.1072 | LR: 4.46e-04 | 1405.7ms/step | 14.2 samples/s
484
+ Step 9940/44000 | Loss: 3.1244 | LR: 4.46e-04 | 1405.1ms/step | 14.2 samples/s
485
+ Step 9950/44000 | Loss: 3.1176 | LR: 4.46e-04 | 1405.7ms/step | 14.2 samples/s
486
+ Step 9960/44000 | Loss: 3.1344 | LR: 4.46e-04 | 1407.7ms/step | 14.2 samples/s
487
+ Step 9970/44000 | Loss: 3.1121 | LR: 4.46e-04 | 1408.1ms/step | 14.2 samples/s
488
+ Step 9980/44000 | Loss: 3.1522 | LR: 4.46e-04 | 1407.1ms/step | 14.2 samples/s
489
+ Step 9990/44000 | Loss: 3.1056 | LR: 4.46e-04 | 1402.3ms/step | 14.3 samples/s
490
+ Step 10000/44000 | Loss: 3.1291 | LR: 4.46e-04 | 1406.2ms/step | 14.2 samples/s
491
+ Validation at step 10000: Loss = 3.1027, Perplexity = 22.26
492
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-10000
493
+ Step 10010/44000 | Loss: 3.1139 | LR: 4.46e-04 | 3929.3ms/step | 5.1 samples/s
494
+ Step 10020/44000 | Loss: 3.0857 | LR: 4.45e-04 | 1400.5ms/step | 14.3 samples/s
495
+ Step 10030/44000 | Loss: 3.1113 | LR: 4.45e-04 | 1399.6ms/step | 14.3 samples/s
496
+ Step 10040/44000 | Loss: 3.1253 | LR: 4.45e-04 | 1402.4ms/step | 14.3 samples/s
497
+ Step 10050/44000 | Loss: 3.1238 | LR: 4.45e-04 | 1401.7ms/step | 14.3 samples/s
498
+ Step 10060/44000 | Loss: 3.1100 | LR: 4.45e-04 | 1400.2ms/step | 14.3 samples/s
499
+ Step 10070/44000 | Loss: 3.1107 | LR: 4.45e-04 | 1399.6ms/step | 14.3 samples/s
500
+ Step 10080/44000 | Loss: 3.1344 | LR: 4.45e-04 | 1400.7ms/step | 14.3 samples/s
501
+ Step 10090/44000 | Loss: 3.1042 | LR: 4.45e-04 | 1405.5ms/step | 14.2 samples/s
502
+ Step 10100/44000 | Loss: 3.1371 | LR: 4.45e-04 | 1406.7ms/step | 14.2 samples/s
503
+ Step 10110/44000 | Loss: 3.0871 | LR: 4.44e-04 | 1406.5ms/step | 14.2 samples/s
504
+ Step 10120/44000 | Loss: 3.0956 | LR: 4.44e-04 | 1407.3ms/step | 14.2 samples/s
505
+ Step 10130/44000 | Loss: 3.1282 | LR: 4.44e-04 | 1410.1ms/step | 14.2 samples/s
506
+ Step 10140/44000 | Loss: 3.0656 | LR: 4.44e-04 | 1410.1ms/step | 14.2 samples/s
507
+ Step 10150/44000 | Loss: 3.0713 | LR: 4.44e-04 | 1411.5ms/step | 14.2 samples/s
508
+ Step 10160/44000 | Loss: 3.0989 | LR: 4.44e-04 | 1410.6ms/step | 14.2 samples/s
509
+ Step 10170/44000 | Loss: 3.0823 | LR: 4.44e-04 | 1411.8ms/step | 14.2 samples/s
510
+ Step 10180/44000 | Loss: 3.1069 | LR: 4.44e-04 | 1411.8ms/step | 14.2 samples/s
511
+ Step 10190/44000 | Loss: 3.1158 | LR: 4.43e-04 | 1409.6ms/step | 14.2 samples/s
512
+ Step 10200/44000 | Loss: 3.0979 | LR: 4.43e-04 | 1410.8ms/step | 14.2 samples/s
513
+ Step 10210/44000 | Loss: 3.1027 | LR: 4.43e-04 | 1410.5ms/step | 14.2 samples/s
514
+ Step 10220/44000 | Loss: 3.0791 | LR: 4.43e-04 | 1408.4ms/step | 14.2 samples/s
515
+ Step 10230/44000 | Loss: 3.0968 | LR: 4.43e-04 | 1410.3ms/step | 14.2 samples/s
516
+ Step 10240/44000 | Loss: 3.1259 | LR: 4.43e-04 | 1410.7ms/step | 14.2 samples/s
517
+ Step 10250/44000 | Loss: 3.1145 | LR: 4.43e-04 | 1410.1ms/step | 14.2 samples/s
518
+ Step 10260/44000 | Loss: 3.0700 | LR: 4.43e-04 | 1412.6ms/step | 14.2 samples/s
519
+ Step 10270/44000 | Loss: 3.0649 | LR: 4.43e-04 | 1412.1ms/step | 14.2 samples/s
520
+ Step 10280/44000 | Loss: 3.1561 | LR: 4.42e-04 | 1412.3ms/step | 14.2 samples/s
521
+ Step 10290/44000 | Loss: 3.0974 | LR: 4.42e-04 | 1408.4ms/step | 14.2 samples/s
522
+ Step 10300/44000 | Loss: 3.1285 | LR: 4.42e-04 | 1406.4ms/step | 14.2 samples/s
523
+ Step 10310/44000 | Loss: 3.0942 | LR: 4.42e-04 | 1409.8ms/step | 14.2 samples/s
524
+ Step 10320/44000 | Loss: 3.1015 | LR: 4.42e-04 | 1410.3ms/step | 14.2 samples/s
525
+ Step 10330/44000 | Loss: 3.1418 | LR: 4.42e-04 | 1412.2ms/step | 14.2 samples/s
526
+ Step 10340/44000 | Loss: 3.0753 | LR: 4.42e-04 | 1409.3ms/step | 14.2 samples/s
527
+ Step 10350/44000 | Loss: 3.0967 | LR: 4.42e-04 | 1412.7ms/step | 14.2 samples/s
528
+ Step 10360/44000 | Loss: 3.0916 | LR: 4.42e-04 | 1411.4ms/step | 14.2 samples/s
529
+ Step 10370/44000 | Loss: 3.0797 | LR: 4.41e-04 | 1410.8ms/step | 14.2 samples/s
530
+ Step 10380/44000 | Loss: 3.0792 | LR: 4.41e-04 | 1410.1ms/step | 14.2 samples/s
531
+ Step 10390/44000 | Loss: 3.1106 | LR: 4.41e-04 | 1409.5ms/step | 14.2 samples/s
532
+ Step 10400/44000 | Loss: 3.0695 | LR: 4.41e-04 | 1409.6ms/step | 14.2 samples/s
533
+ Step 10410/44000 | Loss: 3.1261 | LR: 4.41e-04 | 1411.5ms/step | 14.2 samples/s
534
+ Step 10420/44000 | Loss: 3.1178 | LR: 4.41e-04 | 1413.4ms/step | 14.2 samples/s
535
+ Step 10430/44000 | Loss: 3.1224 | LR: 4.41e-04 | 1411.9ms/step | 14.2 samples/s
536
+ Step 10440/44000 | Loss: 3.1033 | LR: 4.41e-04 | 1409.6ms/step | 14.2 samples/s
537
+ Step 10450/44000 | Loss: 3.1016 | LR: 4.40e-04 | 1406.7ms/step | 14.2 samples/s
538
+ Step 10460/44000 | Loss: 3.0957 | LR: 4.40e-04 | 1405.4ms/step | 14.2 samples/s
539
+ Step 10470/44000 | Loss: 3.1374 | LR: 4.40e-04 | 1407.9ms/step | 14.2 samples/s
540
+ Step 10480/44000 | Loss: 3.1097 | LR: 4.40e-04 | 1405.5ms/step | 14.2 samples/s
541
+ Step 10490/44000 | Loss: 3.1210 | LR: 4.40e-04 | 1404.4ms/step | 14.2 samples/s
542
+ Step 10500/44000 | Loss: 3.1224 | LR: 4.40e-04 | 1406.3ms/step | 14.2 samples/s
543
+ Step 10510/44000 | Loss: 3.1040 | LR: 4.40e-04 | 1408.9ms/step | 14.2 samples/s
544
+ Step 10520/44000 | Loss: 3.0971 | LR: 4.40e-04 | 1407.6ms/step | 14.2 samples/s
545
+ Step 10530/44000 | Loss: 3.1116 | LR: 4.40e-04 | 1405.6ms/step | 14.2 samples/s
546
+ Step 10540/44000 | Loss: 3.0976 | LR: 4.39e-04 | 1409.7ms/step | 14.2 samples/s
547
+ Step 10550/44000 | Loss: 3.0950 | LR: 4.39e-04 | 1403.7ms/step | 14.2 samples/s
548
+ Step 10560/44000 | Loss: 3.1155 | LR: 4.39e-04 | 1400.0ms/step | 14.3 samples/s
549
+ Step 10570/44000 | Loss: 3.0834 | LR: 4.39e-04 | 1402.3ms/step | 14.3 samples/s
550
+ Step 10580/44000 | Loss: 3.1006 | LR: 4.39e-04 | 1403.6ms/step | 14.2 samples/s
551
+ Step 10590/44000 | Loss: 3.1374 | LR: 4.39e-04 | 1404.3ms/step | 14.2 samples/s
552
+ Step 10600/44000 | Loss: 3.0777 | LR: 4.39e-04 | 1399.8ms/step | 14.3 samples/s
553
+ Step 10610/44000 | Loss: 3.1259 | LR: 4.39e-04 | 1400.5ms/step | 14.3 samples/s
554
+ Step 10620/44000 | Loss: 3.1061 | LR: 4.38e-04 | 1402.2ms/step | 14.3 samples/s
555
+ Step 10630/44000 | Loss: 3.1307 | LR: 4.38e-04 | 1401.7ms/step | 14.3 samples/s
556
+ Step 10640/44000 | Loss: 3.1174 | LR: 4.38e-04 | 1403.9ms/step | 14.2 samples/s
557
+ Step 10650/44000 | Loss: 3.1028 | LR: 4.38e-04 | 1400.5ms/step | 14.3 samples/s
558
+ Step 10660/44000 | Loss: 3.0819 | LR: 4.38e-04 | 1404.1ms/step | 14.2 samples/s
559
+ Step 10670/44000 | Loss: 3.0769 | LR: 4.38e-04 | 1407.3ms/step | 14.2 samples/s
560
+ Step 10680/44000 | Loss: 3.0926 | LR: 4.38e-04 | 1406.3ms/step | 14.2 samples/s
561
+ Step 10690/44000 | Loss: 3.0938 | LR: 4.38e-04 | 1406.0ms/step | 14.2 samples/s
562
+ Step 10700/44000 | Loss: 3.1088 | LR: 4.37e-04 | 1407.7ms/step | 14.2 samples/s
563
+ Step 10710/44000 | Loss: 3.1150 | LR: 4.37e-04 | 1408.0ms/step | 14.2 samples/s
564
+ Step 10720/44000 | Loss: 3.0955 | LR: 4.37e-04 | 1406.6ms/step | 14.2 samples/s
565
+ Step 10730/44000 | Loss: 3.1439 | LR: 4.37e-04 | 1407.9ms/step | 14.2 samples/s
566
+ Step 10740/44000 | Loss: 3.1324 | LR: 4.37e-04 | 1405.7ms/step | 14.2 samples/s
567
+ Step 10750/44000 | Loss: 3.0899 | LR: 4.37e-04 | 1406.2ms/step | 14.2 samples/s
568
+ Step 10760/44000 | Loss: 3.0951 | LR: 4.37e-04 | 1401.0ms/step | 14.3 samples/s
569
+ Step 10770/44000 | Loss: 3.0855 | LR: 4.37e-04 | 1406.9ms/step | 14.2 samples/s
570
+ Step 10780/44000 | Loss: 3.1027 | LR: 4.37e-04 | 1407.2ms/step | 14.2 samples/s
571
+ Step 10790/44000 | Loss: 3.0676 | LR: 4.36e-04 | 1404.9ms/step | 14.2 samples/s
572
+ Step 10800/44000 | Loss: 3.0920 | LR: 4.36e-04 | 1405.0ms/step | 14.2 samples/s
573
+ Step 10810/44000 | Loss: 3.0940 | LR: 4.36e-04 | 1404.5ms/step | 14.2 samples/s
574
+ Step 10820/44000 | Loss: 3.1248 | LR: 4.36e-04 | 1404.3ms/step | 14.2 samples/s
575
+ Step 10830/44000 | Loss: 3.0871 | LR: 4.36e-04 | 1406.0ms/step | 14.2 samples/s
576
+ Step 10840/44000 | Loss: 3.0912 | LR: 4.36e-04 | 1401.7ms/step | 14.3 samples/s
577
+ Step 10850/44000 | Loss: 3.0636 | LR: 4.36e-04 | 1402.2ms/step | 14.3 samples/s
578
+ Step 10860/44000 | Loss: 3.1123 | LR: 4.36e-04 | 1404.1ms/step | 14.2 samples/s
579
+ Step 10870/44000 | Loss: 3.1204 | LR: 4.35e-04 | 1402.5ms/step | 14.3 samples/s
580
+ Step 10880/44000 | Loss: 3.0951 | LR: 4.35e-04 | 1401.9ms/step | 14.3 samples/s
581
+ Step 10890/44000 | Loss: 3.0830 | LR: 4.35e-04 | 1401.6ms/step | 14.3 samples/s
582
+ Step 10900/44000 | Loss: 3.0962 | LR: 4.35e-04 | 1402.0ms/step | 14.3 samples/s
583
+ Step 10910/44000 | Loss: 3.1161 | LR: 4.35e-04 | 1404.0ms/step | 14.2 samples/s
584
+ Step 10920/44000 | Loss: 3.0845 | LR: 4.35e-04 | 1403.9ms/step | 14.2 samples/s
585
+ Step 10930/44000 | Loss: 3.1032 | LR: 4.35e-04 | 1402.3ms/step | 14.3 samples/s
586
+ Step 10940/44000 | Loss: 3.1167 | LR: 4.35e-04 | 1407.8ms/step | 14.2 samples/s
587
+ Step 10950/44000 | Loss: 3.1012 | LR: 4.34e-04 | 1404.3ms/step | 14.2 samples/s
588
+ Step 10960/44000 | Loss: 3.0833 | LR: 4.34e-04 | 1406.1ms/step | 14.2 samples/s
589
+ Step 10970/44000 | Loss: 3.1097 | LR: 4.34e-04 | 1407.6ms/step | 14.2 samples/s
590
+ Step 10980/44000 | Loss: 3.0889 | LR: 4.34e-04 | 1405.9ms/step | 14.2 samples/s
591
+ Step 10990/44000 | Loss: 3.1079 | LR: 4.34e-04 | 1405.6ms/step | 14.2 samples/s
592
+ Step 11000/44000 | Loss: 3.1145 | LR: 4.34e-04 | 1404.7ms/step | 14.2 samples/s
593
+ Validation at step 11000: Loss = 3.0832, Perplexity = 21.83
594
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-11000
595
+ Step 11010/44000 | Loss: 3.1048 | LR: 4.34e-04 | 3923.8ms/step | 5.1 samples/s
596
+ Step 11020/44000 | Loss: 3.0973 | LR: 4.34e-04 | 1400.7ms/step | 14.3 samples/s
597
+ Step 11030/44000 | Loss: 3.1211 | LR: 4.33e-04 | 1405.1ms/step | 14.2 samples/s
598
+ Step 11040/44000 | Loss: 3.0705 | LR: 4.33e-04 | 1408.2ms/step | 14.2 samples/s
599
+ Step 11050/44000 | Loss: 3.0568 | LR: 4.33e-04 | 1404.2ms/step | 14.2 samples/s
600
+ Step 11060/44000 | Loss: 3.0985 | LR: 4.33e-04 | 1402.1ms/step | 14.3 samples/s
601
+ Step 11070/44000 | Loss: 3.1213 | LR: 4.33e-04 | 1407.4ms/step | 14.2 samples/s
602
+ Step 11080/44000 | Loss: 3.0798 | LR: 4.33e-04 | 1408.2ms/step | 14.2 samples/s
603
+ Step 11090/44000 | Loss: 3.0937 | LR: 4.33e-04 | 1411.0ms/step | 14.2 samples/s
604
+ Step 11100/44000 | Loss: 3.0921 | LR: 4.33e-04 | 1411.5ms/step | 14.2 samples/s
605
+ Step 11110/44000 | Loss: 3.0640 | LR: 4.32e-04 | 1412.3ms/step | 14.2 samples/s
606
+ Step 11120/44000 | Loss: 3.0772 | LR: 4.32e-04 | 1411.4ms/step | 14.2 samples/s
607
+ Step 11130/44000 | Loss: 3.0501 | LR: 4.32e-04 | 1408.8ms/step | 14.2 samples/s
608
+ Step 11140/44000 | Loss: 3.0983 | LR: 4.32e-04 | 1411.3ms/step | 14.2 samples/s
609
+ Step 11150/44000 | Loss: 3.0510 | LR: 4.32e-04 | 1407.8ms/step | 14.2 samples/s
610
+ Step 11160/44000 | Loss: 3.1102 | LR: 4.32e-04 | 1408.9ms/step | 14.2 samples/s
611
+ Step 11170/44000 | Loss: 3.1115 | LR: 4.32e-04 | 1409.6ms/step | 14.2 samples/s
612
+ Step 11180/44000 | Loss: 3.1016 | LR: 4.32e-04 | 1409.7ms/step | 14.2 samples/s
613
+ Step 11190/44000 | Loss: 3.0904 | LR: 4.31e-04 | 1410.6ms/step | 14.2 samples/s
614
+ Step 11200/44000 | Loss: 3.1061 | LR: 4.31e-04 | 1409.9ms/step | 14.2 samples/s
615
+ Step 11210/44000 | Loss: 3.0956 | LR: 4.31e-04 | 1412.3ms/step | 14.2 samples/s
616
+ Step 11220/44000 | Loss: 3.0955 | LR: 4.31e-04 | 1411.1ms/step | 14.2 samples/s
617
+ Step 11230/44000 | Loss: 3.0797 | LR: 4.31e-04 | 1412.1ms/step | 14.2 samples/s
618
+ Step 11240/44000 | Loss: 3.0769 | LR: 4.31e-04 | 1410.0ms/step | 14.2 samples/s
619
+ Step 11250/44000 | Loss: 3.0783 | LR: 4.31e-04 | 1410.4ms/step | 14.2 samples/s
620
+ Step 11260/44000 | Loss: 3.0971 | LR: 4.31e-04 | 1409.9ms/step | 14.2 samples/s
621
+ Step 11270/44000 | Loss: 3.0659 | LR: 4.30e-04 | 1409.7ms/step | 14.2 samples/s
622
+ Step 11280/44000 | Loss: 3.0995 | LR: 4.30e-04 | 1409.9ms/step | 14.2 samples/s
623
+ Step 11290/44000 | Loss: 3.0663 | LR: 4.30e-04 | 1409.8ms/step | 14.2 samples/s
624
+ Step 11300/44000 | Loss: 3.0665 | LR: 4.30e-04 | 1409.2ms/step | 14.2 samples/s
625
+ Step 11310/44000 | Loss: 3.0765 | LR: 4.30e-04 | 1409.3ms/step | 14.2 samples/s
626
+ Step 11320/44000 | Loss: 3.0624 | LR: 4.30e-04 | 1408.4ms/step | 14.2 samples/s
627
+ Step 11330/44000 | Loss: 3.1223 | LR: 4.30e-04 | 1409.0ms/step | 14.2 samples/s
628
+ Step 11340/44000 | Loss: 3.0610 | LR: 4.30e-04 | 1408.7ms/step | 14.2 samples/s
629
+ Step 11350/44000 | Loss: 3.1035 | LR: 4.29e-04 | 1406.5ms/step | 14.2 samples/s
630
+ Step 11360/44000 | Loss: 3.1070 | LR: 4.29e-04 | 1406.9ms/step | 14.2 samples/s
631
+ Step 11370/44000 | Loss: 3.0720 | LR: 4.29e-04 | 1402.0ms/step | 14.3 samples/s
632
+ Step 11380/44000 | Loss: 3.0813 | LR: 4.29e-04 | 1403.0ms/step | 14.3 samples/s
633
+ Step 11390/44000 | Loss: 3.0752 | LR: 4.29e-04 | 1403.7ms/step | 14.2 samples/s
634
+ Step 11400/44000 | Loss: 3.1073 | LR: 4.29e-04 | 1407.2ms/step | 14.2 samples/s
635
+ Step 11410/44000 | Loss: 3.1016 | LR: 4.29e-04 | 1408.4ms/step | 14.2 samples/s
636
+ Step 11420/44000 | Loss: 3.1016 | LR: 4.29e-04 | 1407.9ms/step | 14.2 samples/s
637
+ Step 11430/44000 | Loss: 3.0792 | LR: 4.28e-04 | 1406.9ms/step | 14.2 samples/s
638
+ Step 11440/44000 | Loss: 3.0864 | LR: 4.28e-04 | 1403.5ms/step | 14.3 samples/s
639
+ Step 11450/44000 | Loss: 3.0733 | LR: 4.28e-04 | 1405.6ms/step | 14.2 samples/s
640
+ Step 11460/44000 | Loss: 3.0638 | LR: 4.28e-04 | 1404.3ms/step | 14.2 samples/s
641
+ Step 11470/44000 | Loss: 3.0840 | LR: 4.28e-04 | 1408.0ms/step | 14.2 samples/s
642
+ Step 11480/44000 | Loss: 3.0510 | LR: 4.28e-04 | 1406.5ms/step | 14.2 samples/s
643
+ Step 11490/44000 | Loss: 3.0728 | LR: 4.28e-04 | 1407.0ms/step | 14.2 samples/s
644
+ Step 11500/44000 | Loss: 3.0773 | LR: 4.28e-04 | 1408.8ms/step | 14.2 samples/s
645
+ Step 11510/44000 | Loss: 3.0577 | LR: 4.27e-04 | 1407.5ms/step | 14.2 samples/s
646
+ Step 11520/44000 | Loss: 3.0858 | LR: 4.27e-04 | 1403.7ms/step | 14.2 samples/s
647
+ Step 11530/44000 | Loss: 3.0852 | LR: 4.27e-04 | 1400.3ms/step | 14.3 samples/s
648
+ Step 11540/44000 | Loss: 3.0854 | LR: 4.27e-04 | 1399.6ms/step | 14.3 samples/s
649
+ Step 11550/44000 | Loss: 3.0739 | LR: 4.27e-04 | 1400.0ms/step | 14.3 samples/s
650
+ Step 11560/44000 | Loss: 3.0739 | LR: 4.27e-04 | 1400.0ms/step | 14.3 samples/s
651
+ Step 11570/44000 | Loss: 3.0712 | LR: 4.27e-04 | 1402.0ms/step | 14.3 samples/s
652
+ Step 11580/44000 | Loss: 3.0592 | LR: 4.27e-04 | 1402.0ms/step | 14.3 samples/s
653
+ Step 11590/44000 | Loss: 3.0741 | LR: 4.26e-04 | 1404.2ms/step | 14.2 samples/s
654
+ Step 11600/44000 | Loss: 3.0738 | LR: 4.26e-04 | 1416.0ms/step | 14.1 samples/s
655
+ Step 11610/44000 | Loss: 3.1089 | LR: 4.26e-04 | 1417.6ms/step | 14.1 samples/s
656
+ Step 11620/44000 | Loss: 3.0818 | LR: 4.26e-04 | 1402.3ms/step | 14.3 samples/s
657
+ Step 11630/44000 | Loss: 3.0894 | LR: 4.26e-04 | 1404.8ms/step | 14.2 samples/s
658
+ Step 11640/44000 | Loss: 3.0970 | LR: 4.26e-04 | 1406.0ms/step | 14.2 samples/s
659
+ Step 11650/44000 | Loss: 3.1157 | LR: 4.26e-04 | 1403.6ms/step | 14.2 samples/s
660
+ Step 11660/44000 | Loss: 3.0692 | LR: 4.26e-04 | 1406.2ms/step | 14.2 samples/s
661
+ Step 11670/44000 | Loss: 3.0894 | LR: 4.25e-04 | 1405.4ms/step | 14.2 samples/s
662
+ Step 11680/44000 | Loss: 3.0619 | LR: 4.25e-04 | 1400.0ms/step | 14.3 samples/s
663
+ Step 11690/44000 | Loss: 3.0609 | LR: 4.25e-04 | 1400.0ms/step | 14.3 samples/s
664
+ Step 11700/44000 | Loss: 3.0820 | LR: 4.25e-04 | 1400.2ms/step | 14.3 samples/s
665
+ Step 11710/44000 | Loss: 3.0242 | LR: 4.25e-04 | 1400.1ms/step | 14.3 samples/s
666
+ Step 11720/44000 | Loss: 3.0831 | LR: 4.25e-04 | 1403.9ms/step | 14.2 samples/s
667
+ Step 11730/44000 | Loss: 3.0492 | LR: 4.25e-04 | 1405.7ms/step | 14.2 samples/s
668
+ Step 11740/44000 | Loss: 3.0424 | LR: 4.24e-04 | 1400.2ms/step | 14.3 samples/s
669
+ Step 11750/44000 | Loss: 3.0564 | LR: 4.24e-04 | 1402.7ms/step | 14.3 samples/s
670
+ Step 11760/44000 | Loss: 3.0838 | LR: 4.24e-04 | 1401.1ms/step | 14.3 samples/s
671
+ Step 11770/44000 | Loss: 3.0887 | LR: 4.24e-04 | 1400.1ms/step | 14.3 samples/s
672
+ Step 11780/44000 | Loss: 3.0900 | LR: 4.24e-04 | 1400.0ms/step | 14.3 samples/s
673
+ Step 11790/44000 | Loss: 3.0547 | LR: 4.24e-04 | 1399.9ms/step | 14.3 samples/s
674
+ Step 11800/44000 | Loss: 3.0719 | LR: 4.24e-04 | 1400.1ms/step | 14.3 samples/s
675
+ Step 11810/44000 | Loss: 3.0761 | LR: 4.24e-04 | 1400.4ms/step | 14.3 samples/s
676
+ Step 11820/44000 | Loss: 3.0725 | LR: 4.23e-04 | 1401.5ms/step | 14.3 samples/s
677
+ Step 11830/44000 | Loss: 3.1007 | LR: 4.23e-04 | 1400.0ms/step | 14.3 samples/s
678
+ Step 11840/44000 | Loss: 3.1051 | LR: 4.23e-04 | 1400.1ms/step | 14.3 samples/s
679
+ Step 11850/44000 | Loss: 3.0711 | LR: 4.23e-04 | 1401.8ms/step | 14.3 samples/s
680
+ Step 11860/44000 | Loss: 3.0656 | LR: 4.23e-04 | 1400.1ms/step | 14.3 samples/s
681
+ Step 11870/44000 | Loss: 3.0594 | LR: 4.23e-04 | 1400.0ms/step | 14.3 samples/s
682
+ Step 11880/44000 | Loss: 3.0579 | LR: 4.23e-04 | 1401.8ms/step | 14.3 samples/s
683
+ Step 11890/44000 | Loss: 3.0589 | LR: 4.23e-04 | 1404.5ms/step | 14.2 samples/s
684
+ Step 11900/44000 | Loss: 3.0828 | LR: 4.22e-04 | 1401.8ms/step | 14.3 samples/s
685
+ Step 11910/44000 | Loss: 3.0870 | LR: 4.22e-04 | 1405.9ms/step | 14.2 samples/s
686
+ Step 11920/44000 | Loss: 3.0685 | LR: 4.22e-04 | 1406.6ms/step | 14.2 samples/s
687
+ Step 11930/44000 | Loss: 3.0591 | LR: 4.22e-04 | 1405.8ms/step | 14.2 samples/s
688
+ Step 11940/44000 | Loss: 3.0915 | LR: 4.22e-04 | 1407.9ms/step | 14.2 samples/s
689
+ Step 11950/44000 | Loss: 3.0698 | LR: 4.22e-04 | 1406.7ms/step | 14.2 samples/s
690
+ Step 11960/44000 | Loss: 3.0657 | LR: 4.22e-04 | 1408.9ms/step | 14.2 samples/s
691
+ Step 11970/44000 | Loss: 3.0955 | LR: 4.21e-04 | 1404.9ms/step | 14.2 samples/s
692
+ Step 11980/44000 | Loss: 3.0901 | LR: 4.21e-04 | 1406.2ms/step | 14.2 samples/s
693
+ Step 11990/44000 | Loss: 3.0444 | LR: 4.21e-04 | 1401.3ms/step | 14.3 samples/s
694
+ Step 12000/44000 | Loss: 3.0649 | LR: 4.21e-04 | 1404.2ms/step | 14.2 samples/s
695
+ Validation at step 12000: Loss = 3.0646, Perplexity = 21.43
696
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-12000
697
+ Step 12010/44000 | Loss: 3.0627 | LR: 4.21e-04 | 3921.4ms/step | 5.1 samples/s
698
+ Step 12020/44000 | Loss: 3.1026 | LR: 4.21e-04 | 1400.3ms/step | 14.3 samples/s
699
+ Step 12030/44000 | Loss: 3.0989 | LR: 4.21e-04 | 1402.0ms/step | 14.3 samples/s
700
+ Step 12040/44000 | Loss: 3.0758 | LR: 4.21e-04 | 1406.9ms/step | 14.2 samples/s
701
+ Step 12050/44000 | Loss: 3.0446 | LR: 4.20e-04 | 1409.8ms/step | 14.2 samples/s
702
+ Step 12060/44000 | Loss: 3.0759 | LR: 4.20e-04 | 1410.0ms/step | 14.2 samples/s
703
+ Step 12070/44000 | Loss: 3.0751 | LR: 4.20e-04 | 1413.2ms/step | 14.2 samples/s
704
+ Step 12080/44000 | Loss: 3.0718 | LR: 4.20e-04 | 1411.0ms/step | 14.2 samples/s
705
+ Step 12090/44000 | Loss: 3.0596 | LR: 4.20e-04 | 1413.9ms/step | 14.1 samples/s
706
+ Step 12100/44000 | Loss: 3.0403 | LR: 4.20e-04 | 1412.7ms/step | 14.2 samples/s
707
+ Step 12110/44000 | Loss: 3.0812 | LR: 4.20e-04 | 1412.5ms/step | 14.2 samples/s
708
+ Step 12120/44000 | Loss: 3.0636 | LR: 4.19e-04 | 1408.6ms/step | 14.2 samples/s
709
+ Step 12130/44000 | Loss: 3.0582 | LR: 4.19e-04 | 1413.1ms/step | 14.2 samples/s
710
+ Step 12140/44000 | Loss: 3.0897 | LR: 4.19e-04 | 1411.0ms/step | 14.2 samples/s
711
+ Step 12150/44000 | Loss: 3.1058 | LR: 4.19e-04 | 1414.6ms/step | 14.1 samples/s
712
+ Step 12160/44000 | Loss: 3.0544 | LR: 4.19e-04 | 1412.5ms/step | 14.2 samples/s
713
+ Step 12170/44000 | Loss: 3.0819 | LR: 4.19e-04 | 1408.6ms/step | 14.2 samples/s
714
+ Step 12180/44000 | Loss: 3.0369 | LR: 4.19e-04 | 1410.5ms/step | 14.2 samples/s
715
+ Step 12190/44000 | Loss: 3.0773 | LR: 4.19e-04 | 1409.9ms/step | 14.2 samples/s
716
+ Step 12200/44000 | Loss: 3.0810 | LR: 4.18e-04 | 1408.9ms/step | 14.2 samples/s
717
+ Step 12210/44000 | Loss: 3.0895 | LR: 4.18e-04 | 1410.0ms/step | 14.2 samples/s
718
+ Step 12220/44000 | Loss: 3.0528 | LR: 4.18e-04 | 1410.8ms/step | 14.2 samples/s
719
+ Step 12230/44000 | Loss: 3.0559 | LR: 4.18e-04 | 1411.7ms/step | 14.2 samples/s
720
+ Step 12240/44000 | Loss: 3.0583 | LR: 4.18e-04 | 1413.3ms/step | 14.2 samples/s
721
+ Step 12250/44000 | Loss: 3.0711 | LR: 4.18e-04 | 1411.1ms/step | 14.2 samples/s
722
+ Step 12260/44000 | Loss: 3.1340 | LR: 4.18e-04 | 1409.9ms/step | 14.2 samples/s
723
+ Step 12270/44000 | Loss: 3.0843 | LR: 4.17e-04 | 1411.1ms/step | 14.2 samples/s
724
+ Step 12280/44000 | Loss: 3.0585 | LR: 4.17e-04 | 1412.4ms/step | 14.2 samples/s
725
+ Step 12290/44000 | Loss: 3.0507 | LR: 4.17e-04 | 1407.9ms/step | 14.2 samples/s
726
+ Step 12300/44000 | Loss: 3.0736 | LR: 4.17e-04 | 1405.9ms/step | 14.2 samples/s
727
+ Step 12310/44000 | Loss: 3.0615 | LR: 4.17e-04 | 1406.8ms/step | 14.2 samples/s
728
+ Step 12320/44000 | Loss: 3.0674 | LR: 4.17e-04 | 1407.6ms/step | 14.2 samples/s
729
+ Step 12330/44000 | Loss: 3.0688 | LR: 4.17e-04 | 1407.3ms/step | 14.2 samples/s
730
+ Step 12340/44000 | Loss: 3.0804 | LR: 4.17e-04 | 1403.9ms/step | 14.2 samples/s
731
+ Step 12350/44000 | Loss: 3.0697 | LR: 4.16e-04 | 1402.9ms/step | 14.3 samples/s
732
+ Step 12360/44000 | Loss: 3.0744 | LR: 4.16e-04 | 1406.4ms/step | 14.2 samples/s
733
+ Step 12370/44000 | Loss: 3.0824 | LR: 4.16e-04 | 1407.2ms/step | 14.2 samples/s
734
+ Step 12380/44000 | Loss: 3.0552 | LR: 4.16e-04 | 1410.3ms/step | 14.2 samples/s
735
+ Step 12390/44000 | Loss: 3.0683 | LR: 4.16e-04 | 1407.7ms/step | 14.2 samples/s
736
+ Step 12400/44000 | Loss: 3.0548 | LR: 4.16e-04 | 1407.9ms/step | 14.2 samples/s
737
+ Step 12410/44000 | Loss: 3.0833 | LR: 4.16e-04 | 1407.0ms/step | 14.2 samples/s
738
+ Step 12420/44000 | Loss: 3.0618 | LR: 4.15e-04 | 1408.7ms/step | 14.2 samples/s
739
+ Step 12430/44000 | Loss: 3.0875 | LR: 4.15e-04 | 1406.8ms/step | 14.2 samples/s
740
+ Step 12440/44000 | Loss: 3.0682 | LR: 4.15e-04 | 1405.5ms/step | 14.2 samples/s
741
+ Step 12450/44000 | Loss: 3.0752 | LR: 4.15e-04 | 1409.7ms/step | 14.2 samples/s
742
+ Step 12460/44000 | Loss: 3.0793 | LR: 4.15e-04 | 1402.1ms/step | 14.3 samples/s
743
+ Step 12470/44000 | Loss: 3.0682 | LR: 4.15e-04 | 1399.7ms/step | 14.3 samples/s
744
+ Step 12480/44000 | Loss: 3.0646 | LR: 4.15e-04 | 1400.1ms/step | 14.3 samples/s
745
+ Step 12490/44000 | Loss: 3.0717 | LR: 4.14e-04 | 1400.0ms/step | 14.3 samples/s
746
+ Step 12500/44000 | Loss: 3.0976 | LR: 4.14e-04 | 1400.0ms/step | 14.3 samples/s
747
+ Step 12510/44000 | Loss: 3.0680 | LR: 4.14e-04 | 1400.1ms/step | 14.3 samples/s
748
+ Step 12520/44000 | Loss: 3.0528 | LR: 4.14e-04 | 1400.0ms/step | 14.3 samples/s
749
+ Step 12530/44000 | Loss: 3.0819 | LR: 4.14e-04 | 1399.9ms/step | 14.3 samples/s
750
+ Step 12540/44000 | Loss: 3.0675 | LR: 4.14e-04 | 1400.5ms/step | 14.3 samples/s
751
+ Step 12550/44000 | Loss: 3.0520 | LR: 4.14e-04 | 1401.6ms/step | 14.3 samples/s
752
+ Step 12560/44000 | Loss: 3.0732 | LR: 4.14e-04 | 1399.8ms/step | 14.3 samples/s
753
+ Step 12570/44000 | Loss: 3.0631 | LR: 4.13e-04 | 1400.0ms/step | 14.3 samples/s
754
+ Step 12580/44000 | Loss: 3.0742 | LR: 4.13e-04 | 1400.3ms/step | 14.3 samples/s
755
+ Step 12590/44000 | Loss: 3.1108 | LR: 4.13e-04 | 1400.8ms/step | 14.3 samples/s
756
+ Step 12600/44000 | Loss: 3.0451 | LR: 4.13e-04 | 1403.3ms/step | 14.3 samples/s
757
+ Step 12610/44000 | Loss: 3.0410 | LR: 4.13e-04 | 1403.9ms/step | 14.2 samples/s
758
+ Step 12620/44000 | Loss: 3.0618 | LR: 4.13e-04 | 1404.8ms/step | 14.2 samples/s
759
+ Step 12630/44000 | Loss: 3.0638 | LR: 4.13e-04 | 1401.3ms/step | 14.3 samples/s
760
+ Step 12640/44000 | Loss: 3.0559 | LR: 4.12e-04 | 1404.8ms/step | 14.2 samples/s
761
+ Step 12650/44000 | Loss: 3.0699 | LR: 4.12e-04 | 1404.1ms/step | 14.2 samples/s
762
+ Step 12660/44000 | Loss: 3.0675 | LR: 4.12e-04 | 1401.6ms/step | 14.3 samples/s
763
+ Step 12670/44000 | Loss: 3.0641 | LR: 4.12e-04 | 1401.1ms/step | 14.3 samples/s
764
+ Step 12680/44000 | Loss: 3.0487 | LR: 4.12e-04 | 1400.1ms/step | 14.3 samples/s
765
+ Step 12690/44000 | Loss: 3.0641 | LR: 4.12e-04 | 1400.2ms/step | 14.3 samples/s
766
+ Step 12700/44000 | Loss: 3.0806 | LR: 4.12e-04 | 1399.9ms/step | 14.3 samples/s
767
+ Step 12710/44000 | Loss: 3.0648 | LR: 4.11e-04 | 1400.0ms/step | 14.3 samples/s
768
+ Step 12720/44000 | Loss: 3.0512 | LR: 4.11e-04 | 1400.0ms/step | 14.3 samples/s
769
+ Step 12730/44000 | Loss: 3.0933 | LR: 4.11e-04 | 1399.9ms/step | 14.3 samples/s
770
+ Step 12740/44000 | Loss: 3.0093 | LR: 4.11e-04 | 1400.4ms/step | 14.3 samples/s
771
+ Step 12750/44000 | Loss: 3.0656 | LR: 4.11e-04 | 1401.7ms/step | 14.3 samples/s
772
+ Step 12760/44000 | Loss: 3.0410 | LR: 4.11e-04 | 1400.3ms/step | 14.3 samples/s
773
+ Step 12770/44000 | Loss: 3.0597 | LR: 4.11e-04 | 1399.9ms/step | 14.3 samples/s
774
+ Step 12780/44000 | Loss: 3.0610 | LR: 4.10e-04 | 1399.7ms/step | 14.3 samples/s
775
+ Step 12790/44000 | Loss: 3.0594 | LR: 4.10e-04 | 1400.1ms/step | 14.3 samples/s
776
+ Step 12800/44000 | Loss: 3.0464 | LR: 4.10e-04 | 1400.2ms/step | 14.3 samples/s
777
+ Step 12810/44000 | Loss: 3.0682 | LR: 4.10e-04 | 1401.0ms/step | 14.3 samples/s
778
+ Step 12820/44000 | Loss: 3.0856 | LR: 4.10e-04 | 1403.5ms/step | 14.2 samples/s
779
+ Step 12830/44000 | Loss: 3.0819 | LR: 4.10e-04 | 1403.6ms/step | 14.2 samples/s
780
+ Step 12840/44000 | Loss: 3.0853 | LR: 4.10e-04 | 1400.0ms/step | 14.3 samples/s
781
+ Step 12850/44000 | Loss: 3.0428 | LR: 4.09e-04 | 1404.1ms/step | 14.2 samples/s
782
+ Step 12860/44000 | Loss: 3.0628 | LR: 4.09e-04 | 1405.7ms/step | 14.2 samples/s
783
+ Step 12870/44000 | Loss: 3.0766 | LR: 4.09e-04 | 1400.0ms/step | 14.3 samples/s
784
+ Step 12880/44000 | Loss: 3.0978 | LR: 4.09e-04 | 1399.8ms/step | 14.3 samples/s
785
+ Step 12890/44000 | Loss: 3.0582 | LR: 4.09e-04 | 1400.3ms/step | 14.3 samples/s
786
+ Step 12900/44000 | Loss: 3.0548 | LR: 4.09e-04 | 1400.0ms/step | 14.3 samples/s
787
+ Step 12910/44000 | Loss: 3.0661 | LR: 4.09e-04 | 1399.9ms/step | 14.3 samples/s
788
+ Step 12920/44000 | Loss: 3.0360 | LR: 4.09e-04 | 1400.7ms/step | 14.3 samples/s
789
+ Step 12930/44000 | Loss: 3.0695 | LR: 4.08e-04 | 1400.6ms/step | 14.3 samples/s
790
+ Step 12940/44000 | Loss: 3.0929 | LR: 4.08e-04 | 1403.5ms/step | 14.3 samples/s
791
+ Step 12950/44000 | Loss: 3.0696 | LR: 4.08e-04 | 1403.5ms/step | 14.2 samples/s
792
+ Step 12960/44000 | Loss: 3.0721 | LR: 4.08e-04 | 1404.6ms/step | 14.2 samples/s
793
+ Step 12970/44000 | Loss: 3.0563 | LR: 4.08e-04 | 1407.6ms/step | 14.2 samples/s
794
+ Step 12980/44000 | Loss: 3.0434 | LR: 4.08e-04 | 1401.6ms/step | 14.3 samples/s
795
+ Step 12990/44000 | Loss: 3.0289 | LR: 4.08e-04 | 1400.3ms/step | 14.3 samples/s
796
+ Step 13000/44000 | Loss: 3.0754 | LR: 4.07e-04 | 1403.7ms/step | 14.2 samples/s
797
+ Validation at step 13000: Loss = 3.0468, Perplexity = 21.05
798
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-13000
799
+ Step 13010/44000 | Loss: 3.0428 | LR: 4.07e-04 | 3936.1ms/step | 5.1 samples/s
800
+ Step 13020/44000 | Loss: 3.0560 | LR: 4.07e-04 | 1405.9ms/step | 14.2 samples/s
801
+ Step 13030/44000 | Loss: 3.0365 | LR: 4.07e-04 | 1408.3ms/step | 14.2 samples/s
802
+ Step 13040/44000 | Loss: 3.0766 | LR: 4.07e-04 | 1410.0ms/step | 14.2 samples/s
803
+ Step 13050/44000 | Loss: 3.0779 | LR: 4.07e-04 | 1410.2ms/step | 14.2 samples/s
804
+ Step 13060/44000 | Loss: 3.0487 | LR: 4.07e-04 | 1407.8ms/step | 14.2 samples/s
805
+ Step 13070/44000 | Loss: 3.0610 | LR: 4.06e-04 | 1407.8ms/step | 14.2 samples/s
806
+ Step 13080/44000 | Loss: 3.0886 | LR: 4.06e-04 | 1410.4ms/step | 14.2 samples/s
807
+ Step 13090/44000 | Loss: 3.0555 | LR: 4.06e-04 | 1410.6ms/step | 14.2 samples/s
808
+ Step 13100/44000 | Loss: 3.0537 | LR: 4.06e-04 | 1411.2ms/step | 14.2 samples/s
809
+ Step 13110/44000 | Loss: 3.0510 | LR: 4.06e-04 | 1408.7ms/step | 14.2 samples/s
810
+ Step 13120/44000 | Loss: 3.0669 | LR: 4.06e-04 | 1410.6ms/step | 14.2 samples/s
811
+ Step 13130/44000 | Loss: 3.0551 | LR: 4.06e-04 | 1410.8ms/step | 14.2 samples/s
812
+ Step 13140/44000 | Loss: 3.0557 | LR: 4.05e-04 | 1410.0ms/step | 14.2 samples/s
813
+ Step 13150/44000 | Loss: 3.0673 | LR: 4.05e-04 | 1410.0ms/step | 14.2 samples/s
814
+ Step 13160/44000 | Loss: 3.0161 | LR: 4.05e-04 | 1411.3ms/step | 14.2 samples/s
815
+ Step 13170/44000 | Loss: 3.0689 | LR: 4.05e-04 | 1412.9ms/step | 14.2 samples/s
816
+ Step 13180/44000 | Loss: 3.0458 | LR: 4.05e-04 | 1412.5ms/step | 14.2 samples/s
817
+ Step 13190/44000 | Loss: 3.0277 | LR: 4.05e-04 | 1413.7ms/step | 14.1 samples/s
818
+ Step 13200/44000 | Loss: 3.0752 | LR: 4.05e-04 | 1409.7ms/step | 14.2 samples/s
819
+ Step 13210/44000 | Loss: 3.0489 | LR: 4.04e-04 | 1410.0ms/step | 14.2 samples/s
820
+ Step 13220/44000 | Loss: 3.0476 | LR: 4.04e-04 | 1412.6ms/step | 14.2 samples/s
821
+ Step 13230/44000 | Loss: 3.0879 | LR: 4.04e-04 | 1410.0ms/step | 14.2 samples/s
822
+ Step 13240/44000 | Loss: 3.1028 | LR: 4.04e-04 | 1411.2ms/step | 14.2 samples/s
823
+ Step 13250/44000 | Loss: 3.0594 | LR: 4.04e-04 | 1411.1ms/step | 14.2 samples/s
824
+ Step 13260/44000 | Loss: 3.0662 | LR: 4.04e-04 | 1412.0ms/step | 14.2 samples/s
825
+ Step 13270/44000 | Loss: 3.0832 | LR: 4.04e-04 | 1411.4ms/step | 14.2 samples/s
826
+ Step 13280/44000 | Loss: 3.0850 | LR: 4.03e-04 | 1411.6ms/step | 14.2 samples/s
827
+ Step 13290/44000 | Loss: 3.0657 | LR: 4.03e-04 | 1408.4ms/step | 14.2 samples/s
828
+ Step 13300/44000 | Loss: 3.0562 | LR: 4.03e-04 | 1408.6ms/step | 14.2 samples/s
829
+ Step 13310/44000 | Loss: 3.0780 | LR: 4.03e-04 | 1411.2ms/step | 14.2 samples/s
830
+ Step 13320/44000 | Loss: 3.0437 | LR: 4.03e-04 | 1411.8ms/step | 14.2 samples/s
831
+ Step 13330/44000 | Loss: 3.0678 | LR: 4.03e-04 | 1410.4ms/step | 14.2 samples/s
832
+ Step 13340/44000 | Loss: 3.0267 | LR: 4.03e-04 | 1409.5ms/step | 14.2 samples/s
833
+ Step 13350/44000 | Loss: 3.0744 | LR: 4.02e-04 | 1410.3ms/step | 14.2 samples/s
834
+ Step 13360/44000 | Loss: 3.0552 | LR: 4.02e-04 | 1412.5ms/step | 14.2 samples/s
835
+ Step 13370/44000 | Loss: 3.0747 | LR: 4.02e-04 | 1413.4ms/step | 14.2 samples/s
836
+ Step 13380/44000 | Loss: 3.0651 | LR: 4.02e-04 | 1414.3ms/step | 14.1 samples/s
837
+ Step 13390/44000 | Loss: 3.0648 | LR: 4.02e-04 | 1412.2ms/step | 14.2 samples/s
838
+ Step 13400/44000 | Loss: 3.0421 | LR: 4.02e-04 | 1401.8ms/step | 14.3 samples/s
839
+ Step 13410/44000 | Loss: 3.0544 | LR: 4.02e-04 | 1399.8ms/step | 14.3 samples/s
840
+ Step 13420/44000 | Loss: 3.0155 | LR: 4.01e-04 | 1400.0ms/step | 14.3 samples/s
841
+ Step 13430/44000 | Loss: 3.0720 | LR: 4.01e-04 | 1399.9ms/step | 14.3 samples/s
842
+ Step 13440/44000 | Loss: 3.0407 | LR: 4.01e-04 | 1400.7ms/step | 14.3 samples/s
843
+ Step 13450/44000 | Loss: 3.0596 | LR: 4.01e-04 | 1401.4ms/step | 14.3 samples/s
844
+ Step 13460/44000 | Loss: 3.0534 | LR: 4.01e-04 | 1400.1ms/step | 14.3 samples/s
845
+ Step 13470/44000 | Loss: 3.0873 | LR: 4.01e-04 | 1402.3ms/step | 14.3 samples/s
846
+ Step 13480/44000 | Loss: 3.0607 | LR: 4.01e-04 | 1401.5ms/step | 14.3 samples/s
847
+ Step 13490/44000 | Loss: 3.0447 | LR: 4.00e-04 | 1400.5ms/step | 14.3 samples/s
848
+ Step 13500/44000 | Loss: 3.0687 | LR: 4.00e-04 | 1402.8ms/step | 14.3 samples/s
849
+ Step 13510/44000 | Loss: 3.0510 | LR: 4.00e-04 | 1405.3ms/step | 14.2 samples/s
850
+ Step 13520/44000 | Loss: 3.0207 | LR: 4.00e-04 | 1405.8ms/step | 14.2 samples/s
851
+ Step 13530/44000 | Loss: 3.0419 | LR: 4.00e-04 | 1406.6ms/step | 14.2 samples/s
852
+ Step 13540/44000 | Loss: 3.0400 | LR: 4.00e-04 | 1404.4ms/step | 14.2 samples/s
853
+ Step 13550/44000 | Loss: 3.0316 | LR: 4.00e-04 | 1400.7ms/step | 14.3 samples/s
854
+ Step 13560/44000 | Loss: 3.0495 | LR: 3.99e-04 | 1400.3ms/step | 14.3 samples/s
855
+ Step 13570/44000 | Loss: 3.0637 | LR: 3.99e-04 | 1402.0ms/step | 14.3 samples/s
856
+ Step 13580/44000 | Loss: 3.0523 | LR: 3.99e-04 | 1402.6ms/step | 14.3 samples/s
857
+ Step 13590/44000 | Loss: 3.0280 | LR: 3.99e-04 | 1403.5ms/step | 14.3 samples/s
858
+ Step 13600/44000 | Loss: 3.0588 | LR: 3.99e-04 | 1404.0ms/step | 14.2 samples/s
859
+ Step 13610/44000 | Loss: 3.0797 | LR: 3.99e-04 | 1406.4ms/step | 14.2 samples/s
860
+ Step 13620/44000 | Loss: 3.0541 | LR: 3.98e-04 | 1401.4ms/step | 14.3 samples/s
861
+ Step 13630/44000 | Loss: 3.0294 | LR: 3.98e-04 | 1402.4ms/step | 14.3 samples/s
862
+ Step 13640/44000 | Loss: 3.0598 | LR: 3.98e-04 | 1406.4ms/step | 14.2 samples/s
863
+ Step 13650/44000 | Loss: 3.0561 | LR: 3.98e-04 | 1405.0ms/step | 14.2 samples/s
864
+ Step 13660/44000 | Loss: 3.0666 | LR: 3.98e-04 | 1400.2ms/step | 14.3 samples/s
865
+ Step 13670/44000 | Loss: 3.0718 | LR: 3.98e-04 | 1402.6ms/step | 14.3 samples/s
866
+ Step 13680/44000 | Loss: 3.0343 | LR: 3.98e-04 | 1403.7ms/step | 14.2 samples/s
867
+ Step 13690/44000 | Loss: 3.0674 | LR: 3.97e-04 | 1401.6ms/step | 14.3 samples/s
868
+ Step 13700/44000 | Loss: 3.0214 | LR: 3.97e-04 | 1404.5ms/step | 14.2 samples/s
869
+ Step 13710/44000 | Loss: 3.0526 | LR: 3.97e-04 | 1405.5ms/step | 14.2 samples/s
870
+ Step 13720/44000 | Loss: 3.0501 | LR: 3.97e-04 | 1400.9ms/step | 14.3 samples/s
871
+ Step 13730/44000 | Loss: 3.0441 | LR: 3.97e-04 | 1403.1ms/step | 14.3 samples/s
872
+ Step 13740/44000 | Loss: 3.0598 | LR: 3.97e-04 | 1402.1ms/step | 14.3 samples/s
873
+ Step 13750/44000 | Loss: 3.0645 | LR: 3.97e-04 | 1400.0ms/step | 14.3 samples/s
874
+ Step 13760/44000 | Loss: 3.0413 | LR: 3.96e-04 | 1400.1ms/step | 14.3 samples/s
875
+ Step 13770/44000 | Loss: 3.0701 | LR: 3.96e-04 | 1401.8ms/step | 14.3 samples/s
876
+ Step 13780/44000 | Loss: 3.0331 | LR: 3.96e-04 | 1402.2ms/step | 14.3 samples/s
877
+ Step 13790/44000 | Loss: 3.0386 | LR: 3.96e-04 | 1399.9ms/step | 14.3 samples/s
878
+ Step 13800/44000 | Loss: 3.0633 | LR: 3.96e-04 | 1402.2ms/step | 14.3 samples/s
879
+ Step 13810/44000 | Loss: 2.9968 | LR: 3.96e-04 | 1402.5ms/step | 14.3 samples/s
880
+ Step 13820/44000 | Loss: 3.0581 | LR: 3.96e-04 | 1403.6ms/step | 14.2 samples/s
881
+ Step 13830/44000 | Loss: 3.0633 | LR: 3.95e-04 | 1404.1ms/step | 14.2 samples/s
882
+ Step 13840/44000 | Loss: 3.0076 | LR: 3.95e-04 | 1399.6ms/step | 14.3 samples/s
883
+ Step 13850/44000 | Loss: 3.0434 | LR: 3.95e-04 | 1400.0ms/step | 14.3 samples/s
884
+ Step 13860/44000 | Loss: 3.0491 | LR: 3.95e-04 | 1399.9ms/step | 14.3 samples/s
885
+ Step 13870/44000 | Loss: 3.0459 | LR: 3.95e-04 | 1400.2ms/step | 14.3 samples/s
886
+ Step 13880/44000 | Loss: 3.0587 | LR: 3.95e-04 | 1399.6ms/step | 14.3 samples/s
887
+ Step 13890/44000 | Loss: 3.0506 | LR: 3.95e-04 | 1400.4ms/step | 14.3 samples/s
888
+ Step 13900/44000 | Loss: 3.0759 | LR: 3.94e-04 | 1399.7ms/step | 14.3 samples/s
889
+ Step 13910/44000 | Loss: 3.0439 | LR: 3.94e-04 | 1400.2ms/step | 14.3 samples/s
890
+ Step 13920/44000 | Loss: 3.0446 | LR: 3.94e-04 | 1400.0ms/step | 14.3 samples/s
891
+ Step 13930/44000 | Loss: 3.0302 | LR: 3.94e-04 | 1401.1ms/step | 14.3 samples/s
892
+ Step 13940/44000 | Loss: 3.0302 | LR: 3.94e-04 | 1400.9ms/step | 14.3 samples/s
893
+ Step 13950/44000 | Loss: 3.0159 | LR: 3.94e-04 | 1400.3ms/step | 14.3 samples/s
894
+ Step 13960/44000 | Loss: 3.0230 | LR: 3.93e-04 | 1403.5ms/step | 14.2 samples/s
895
+ Step 13970/44000 | Loss: 3.0529 | LR: 3.93e-04 | 1400.2ms/step | 14.3 samples/s
896
+ Step 13980/44000 | Loss: 3.0109 | LR: 3.93e-04 | 1402.2ms/step | 14.3 samples/s
897
+ Step 13990/44000 | Loss: 3.0353 | LR: 3.93e-04 | 1406.4ms/step | 14.2 samples/s
898
+ Step 14000/44000 | Loss: 3.0585 | LR: 3.93e-04 | 1405.3ms/step | 14.2 samples/s
899
+ Validation at step 14000: Loss = 3.0315, Perplexity = 20.73
900
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735/checkpoint-14000
epfl_final/train_logs/ropectl_a1p0_h100_resume/seed44.20260820T011048Z.log ADDED
@@ -0,0 +1,918 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Auto-resuming from latest checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858/checkpoint-6000
2
+ Resuming: Using existing output directory /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858
3
+ Running config:
4
+ {
5
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
6
+ "model_dtype": "bfloat16",
7
+ "init_from_scratch": true,
8
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
9
+ "val_path": null,
10
+ "max_seq_length": 8192,
11
+ "num_workers": 8,
12
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858",
13
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858/checkpoint-6000",
14
+ "force_new_run": false,
15
+ "num_epochs": 1,
16
+ "max_steps": 44000,
17
+ "per_device_batch_size": 2,
18
+ "gradient_accumulation_steps": 10,
19
+ "learning_rate": 0.0005,
20
+ "weight_decay": 0.01,
21
+ "warmup_steps": 750,
22
+ "lr_scheduler": "cosine",
23
+ "max_grad_norm": 1.0,
24
+ "save_steps": 1000,
25
+ "eval_steps": 1000,
26
+ "eval_batches": 50,
27
+ "logging_steps": 10,
28
+ "seed": 44,
29
+ "verbose": true,
30
+ "checkpoint_activations": false,
31
+ "strategy": "ddp",
32
+ "mixed_precision": "bf16",
33
+ "wandb_enabled": true,
34
+ "wandb_project": "stochastic_sparsity",
35
+ "wandb_name": "l8k365m_dense_ropectl_a1p0_seed44",
36
+ "wandb_tags": [
37
+ "qwen3",
38
+ "365m",
39
+ "8k",
40
+ "dense",
41
+ "rope-control"
42
+ ],
43
+ "hidden_size": 768,
44
+ "intermediate_size": 3072,
45
+ "num_attention_heads": 6,
46
+ "num_key_value_heads": 2,
47
+ "num_hidden_layers": 28,
48
+ "head_dim": null,
49
+ "attention_dropout": 0.0,
50
+ "rope_train_position_scale_set": [
51
+ 1.0,
52
+ 0.9896339488176222,
53
+ 0.9692159692159692,
54
+ 0.9420289855072463,
55
+ 0.8730158730158729
56
+ ],
57
+ "attention_type": "sparse_stochastic",
58
+ "sparse_block_size": 128,
59
+ "sparse_num_random_blocks": 31,
60
+ "sparse_mask_pool_size": 1,
61
+ "sparse_val_dense_compare": false,
62
+ "sparse_layer_indices": null,
63
+ "sparse_mask_cache_dir": null,
64
+ "sparse_mask_seed": 0,
65
+ "sparse_mixed_nrb_set": null,
66
+ "dense_mode": true,
67
+ "dense_step_intervals": null
68
+ }
69
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
70
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858/checkpoint-6000
71
+ Checkpoint activations: False
72
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
73
+
74
+
75
+ ================================================================================
76
+ MODEL ARCHITECTURE
77
+ ================================================================================
78
+ Model config:
79
+ vocab_size: 151936
80
+ hidden_size: 768
81
+ num_hidden_layers: 28
82
+ num_attention_heads: 6
83
+ num_key_value_heads: 2
84
+ head_dim: 128
85
+ intermediate_size: 3072
86
+ max_position_embeddings: 32768
87
+ rope_theta: 1000000
88
+ rope_train_position_scale_set: [1.0, 0.9896339488176222, 0.9692159692159692, 0.9420289855072463, 0.8730158730158729]
89
+ attention_type: sparse_stochastic
90
+ sparse_block_size: 128
91
+ sparse_num_random_blocks: 31
92
+ sparse_mask_pool_size: 1
93
+ sparse_max_seq_length: 8192
94
+
95
+ Parameter count:
96
+ Total parameters: 358,958,848
97
+ Trainable parameters: 358,958,848
98
+ Non-trainable parameters: 0
99
+ Model size (BF16): 0.72 GB
100
+
101
+ --------------------------------------------------------------------------------
102
+ LAYER-BY-LAYER SUMMARY
103
+ --------------------------------------------------------------------------------
104
+ ================================================================================================================================================================
105
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
106
+ ================================================================================================================================================================
107
+ Qwen3ForCausalLM [2, 8192] -- -- True
108
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
109
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
110
+ │ └─ModuleList: 2-2 -- -- -- True
111
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
132
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
133
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
134
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
135
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
136
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
137
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
138
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
139
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
140
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
141
+ ================================================================================================================================================================
142
+ Total params: 475,645,696
143
+ Trainable params: 475,645,696
144
+ Non-trainable params: 0
145
+ Total mult-adds (Units.MEGABYTES): 951.29
146
+ ================================================================================================================================================================
147
+ Input size (MB): 0.13
148
+ Forward/backward pass size (MB): 31197.23
149
+ Params size (MB): 951.29
150
+ Estimated Total Size (MB): 32148.66
151
+ ================================================================================================================================================================
152
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
153
+ ================================================================================
154
+
155
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
156
+ Using pre-tokenized dataset (no runtime tokenization)
157
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
158
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
159
+ Dataset split: 3027764 train, 159356 validation
160
+ Training examples (packed, train split): 3027764
161
+ Validation examples (packed, val split): 159356
162
+ Per-device batch size: 2
163
+ World size (GPUs): 1
164
+ Gradient accumulation steps: 10
165
+ Effective batch size (seqs/step): 20
166
+ Sequence length: 8192
167
+ Tokens per optimizer step: 163,840
168
+ Micro-batches per epoch (per rank): 1513882
169
+ Optimizer steps per epoch: 151388
170
+ Total optimizer steps: 44000 (capped by max_steps; 0.29 epochs)
171
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858/checkpoint-6000...
172
+ Preserving checkpoint LR state for exact resume: current=[0.0004820408177936365], schedule_base=[0.0005]
173
+ Resumed from checkpoint at step 6000 (epoch 0, micro_step 60000)
174
+ Resuming training from step 6000
175
+ wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from WANDB_API_KEY.
176
+ wandb: Currently logged in as: ruben-navasardyan4869 (ruben-navasardyan4869-epfl) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
177
+ wandb: setting up run bajm6b15
178
+ wandb: Tracking run with wandb version 0.26.1
179
+ wandb: Run data is saved locally in /mloscratch/homes/navasard/sparse/SparseAtt/wandb/run-20260820_011125-bajm6b15
180
+ wandb: Run `wandb offline` to turn off syncing.
181
+ wandb: Syncing run l8k365m_dense_ropectl_a1p0_seed44
182
+ wandb: ⭐️ View project at https://wandb.ai/ruben-navasardyan4869-epfl/stochastic_sparsity
183
+ wandb: 🚀 View run at https://wandb.ai/ruben-navasardyan4869-epfl/stochastic_sparsity/runs/bajm6b15
184
+ Starting training...
185
+ Validation at step 6000: Loss = 3.2526, Perplexity = 25.86
186
+ Switching attention mode for step 6001: dense (dense step interval)
187
+ Step 6010/44000 | Loss: 3.2834 | LR: 4.82e-04 | 16026.6ms/step | 1.2 samples/s
188
+ Step 6020/44000 | Loss: 3.2339 | LR: 4.82e-04 | 1361.4ms/step | 14.7 samples/s
189
+ Step 6030/44000 | Loss: 3.2670 | LR: 4.82e-04 | 1362.6ms/step | 14.7 samples/s
190
+ Step 6040/44000 | Loss: 3.2177 | LR: 4.82e-04 | 1364.0ms/step | 14.7 samples/s
191
+ Step 6050/44000 | Loss: 3.2302 | LR: 4.82e-04 | 1364.8ms/step | 14.7 samples/s
192
+ Step 6060/44000 | Loss: 3.2611 | LR: 4.82e-04 | 1366.7ms/step | 14.6 samples/s
193
+ Step 6070/44000 | Loss: 3.2658 | LR: 4.82e-04 | 1368.0ms/step | 14.6 samples/s
194
+ Step 6080/44000 | Loss: 3.2367 | LR: 4.81e-04 | 1369.1ms/step | 14.6 samples/s
195
+ Step 6090/44000 | Loss: 3.2179 | LR: 4.81e-04 | 1366.9ms/step | 14.6 samples/s
196
+ Step 6100/44000 | Loss: 3.2500 | LR: 4.81e-04 | 1365.0ms/step | 14.7 samples/s
197
+ Step 6110/44000 | Loss: 3.2168 | LR: 4.81e-04 | 1364.8ms/step | 14.7 samples/s
198
+ Step 6120/44000 | Loss: 3.2170 | LR: 4.81e-04 | 1365.3ms/step | 14.6 samples/s
199
+ Step 6130/44000 | Loss: 3.2318 | LR: 4.81e-04 | 1364.4ms/step | 14.7 samples/s
200
+ Step 6140/44000 | Loss: 3.2286 | LR: 4.81e-04 | 1365.4ms/step | 14.6 samples/s
201
+ Step 6150/44000 | Loss: 3.2382 | LR: 4.81e-04 | 1368.1ms/step | 14.6 samples/s
202
+ Step 6160/44000 | Loss: 3.2045 | LR: 4.81e-04 | 1366.9ms/step | 14.6 samples/s
203
+ Step 6170/44000 | Loss: 3.2695 | LR: 4.81e-04 | 1366.7ms/step | 14.6 samples/s
204
+ Step 6180/44000 | Loss: 3.2533 | LR: 4.81e-04 | 1365.2ms/step | 14.6 samples/s
205
+ Step 6190/44000 | Loss: 3.2448 | LR: 4.81e-04 | 1365.7ms/step | 14.6 samples/s
206
+ Step 6200/44000 | Loss: 3.2758 | LR: 4.81e-04 | 1365.6ms/step | 14.6 samples/s
207
+ Step 6210/44000 | Loss: 3.2127 | LR: 4.81e-04 | 1366.4ms/step | 14.6 samples/s
208
+ Step 6220/44000 | Loss: 3.2185 | LR: 4.81e-04 | 1367.8ms/step | 14.6 samples/s
209
+ Step 6230/44000 | Loss: 3.2120 | LR: 4.80e-04 | 1366.1ms/step | 14.6 samples/s
210
+ Step 6240/44000 | Loss: 3.2668 | LR: 4.80e-04 | 1367.8ms/step | 14.6 samples/s
211
+ Step 6250/44000 | Loss: 3.2345 | LR: 4.80e-04 | 1368.0ms/step | 14.6 samples/s
212
+ Step 6260/44000 | Loss: 3.2117 | LR: 4.80e-04 | 1368.4ms/step | 14.6 samples/s
213
+ Step 6270/44000 | Loss: 3.2646 | LR: 4.80e-04 | 1362.5ms/step | 14.7 samples/s
214
+ Step 6280/44000 | Loss: 3.2203 | LR: 4.80e-04 | 1361.9ms/step | 14.7 samples/s
215
+ Step 6290/44000 | Loss: 3.2357 | LR: 4.80e-04 | 1367.1ms/step | 14.6 samples/s
216
+ Step 6300/44000 | Loss: 3.2326 | LR: 4.80e-04 | 1366.6ms/step | 14.6 samples/s
217
+ Step 6310/44000 | Loss: 3.2345 | LR: 4.80e-04 | 1367.1ms/step | 14.6 samples/s
218
+ Step 6320/44000 | Loss: 3.2357 | LR: 4.80e-04 | 1366.5ms/step | 14.6 samples/s
219
+ Step 6330/44000 | Loss: 3.2528 | LR: 4.80e-04 | 1365.7ms/step | 14.6 samples/s
220
+ Step 6340/44000 | Loss: 3.2428 | LR: 4.80e-04 | 1368.6ms/step | 14.6 samples/s
221
+ Step 6350/44000 | Loss: 3.2072 | LR: 4.80e-04 | 1368.9ms/step | 14.6 samples/s
222
+ Step 6360/44000 | Loss: 3.2461 | LR: 4.80e-04 | 1367.1ms/step | 14.6 samples/s
223
+ Step 6370/44000 | Loss: 3.2063 | LR: 4.79e-04 | 1367.2ms/step | 14.6 samples/s
224
+ Step 6380/44000 | Loss: 3.2518 | LR: 4.79e-04 | 1361.5ms/step | 14.7 samples/s
225
+ Step 6390/44000 | Loss: 3.2535 | LR: 4.79e-04 | 1361.2ms/step | 14.7 samples/s
226
+ Step 6400/44000 | Loss: 3.2132 | LR: 4.79e-04 | 1360.8ms/step | 14.7 samples/s
227
+ Step 6410/44000 | Loss: 3.2423 | LR: 4.79e-04 | 1361.2ms/step | 14.7 samples/s
228
+ Step 6420/44000 | Loss: 3.1996 | LR: 4.79e-04 | 1366.4ms/step | 14.6 samples/s
229
+ Step 6430/44000 | Loss: 3.2173 | LR: 4.79e-04 | 1368.0ms/step | 14.6 samples/s
230
+ Step 6440/44000 | Loss: 3.2024 | LR: 4.79e-04 | 1369.0ms/step | 14.6 samples/s
231
+ Step 6450/44000 | Loss: 3.2338 | LR: 4.79e-04 | 1361.0ms/step | 14.7 samples/s
232
+ Step 6460/44000 | Loss: 3.2319 | LR: 4.79e-04 | 1361.8ms/step | 14.7 samples/s
233
+ Step 6470/44000 | Loss: 3.2427 | LR: 4.79e-04 | 1360.2ms/step | 14.7 samples/s
234
+ Step 6480/44000 | Loss: 3.2467 | LR: 4.79e-04 | 1359.9ms/step | 14.7 samples/s
235
+ Step 6490/44000 | Loss: 3.2292 | LR: 4.79e-04 | 1360.2ms/step | 14.7 samples/s
236
+ Step 6500/44000 | Loss: 3.2136 | LR: 4.79e-04 | 1360.0ms/step | 14.7 samples/s
237
+ Step 6510/44000 | Loss: 3.2229 | LR: 4.78e-04 | 1360.0ms/step | 14.7 samples/s
238
+ Step 6520/44000 | Loss: 3.2126 | LR: 4.78e-04 | 1361.3ms/step | 14.7 samples/s
239
+ Step 6530/44000 | Loss: 3.2380 | LR: 4.78e-04 | 1360.5ms/step | 14.7 samples/s
240
+ Step 6540/44000 | Loss: 3.2147 | LR: 4.78e-04 | 1360.1ms/step | 14.7 samples/s
241
+ Step 6550/44000 | Loss: 3.2344 | LR: 4.78e-04 | 1362.4ms/step | 14.7 samples/s
242
+ Step 6560/44000 | Loss: 3.2334 | LR: 4.78e-04 | 1367.2ms/step | 14.6 samples/s
243
+ Step 6570/44000 | Loss: 3.2437 | LR: 4.78e-04 | 1368.5ms/step | 14.6 samples/s
244
+ Step 6580/44000 | Loss: 3.2233 | LR: 4.78e-04 | 1360.5ms/step | 14.7 samples/s
245
+ Step 6590/44000 | Loss: 3.2346 | LR: 4.78e-04 | 1360.3ms/step | 14.7 samples/s
246
+ Step 6600/44000 | Loss: 3.2020 | LR: 4.78e-04 | 1362.0ms/step | 14.7 samples/s
247
+ Step 6610/44000 | Loss: 3.2356 | LR: 4.78e-04 | 1362.2ms/step | 14.7 samples/s
248
+ Step 6620/44000 | Loss: 3.2035 | LR: 4.78e-04 | 1361.2ms/step | 14.7 samples/s
249
+ Step 6630/44000 | Loss: 3.2203 | LR: 4.78e-04 | 1359.8ms/step | 14.7 samples/s
250
+ Step 6640/44000 | Loss: 3.2503 | LR: 4.77e-04 | 1360.2ms/step | 14.7 samples/s
251
+ Step 6650/44000 | Loss: 3.2422 | LR: 4.77e-04 | 1359.5ms/step | 14.7 samples/s
252
+ Step 6660/44000 | Loss: 3.2359 | LR: 4.77e-04 | 1363.7ms/step | 14.7 samples/s
253
+ Step 6670/44000 | Loss: 3.2191 | LR: 4.77e-04 | 1366.0ms/step | 14.6 samples/s
254
+ Step 6680/44000 | Loss: 3.2309 | LR: 4.77e-04 | 1362.4ms/step | 14.7 samples/s
255
+ Step 6690/44000 | Loss: 3.2317 | LR: 4.77e-04 | 1360.1ms/step | 14.7 samples/s
256
+ Step 6700/44000 | Loss: 3.2150 | LR: 4.77e-04 | 1360.0ms/step | 14.7 samples/s
257
+ Step 6710/44000 | Loss: 3.1976 | LR: 4.77e-04 | 1358.3ms/step | 14.7 samples/s
258
+ Step 6720/44000 | Loss: 3.1971 | LR: 4.77e-04 | 1358.7ms/step | 14.7 samples/s
259
+ Step 6730/44000 | Loss: 3.2091 | LR: 4.77e-04 | 1359.3ms/step | 14.7 samples/s
260
+ Step 6740/44000 | Loss: 3.2273 | LR: 4.77e-04 | 1359.7ms/step | 14.7 samples/s
261
+ Step 6750/44000 | Loss: 3.1980 | LR: 4.77e-04 | 1360.1ms/step | 14.7 samples/s
262
+ Step 6760/44000 | Loss: 3.2146 | LR: 4.77e-04 | 1366.0ms/step | 14.6 samples/s
263
+ Step 6770/44000 | Loss: 3.2346 | LR: 4.76e-04 | 1365.7ms/step | 14.6 samples/s
264
+ Step 6780/44000 | Loss: 3.2048 | LR: 4.76e-04 | 1360.0ms/step | 14.7 samples/s
265
+ Step 6790/44000 | Loss: 3.2221 | LR: 4.76e-04 | 1360.4ms/step | 14.7 samples/s
266
+ Step 6800/44000 | Loss: 3.2050 | LR: 4.76e-04 | 1359.3ms/step | 14.7 samples/s
267
+ Step 6810/44000 | Loss: 3.2289 | LR: 4.76e-04 | 1358.9ms/step | 14.7 samples/s
268
+ Step 6820/44000 | Loss: 3.1924 | LR: 4.76e-04 | 1363.5ms/step | 14.7 samples/s
269
+ Step 6830/44000 | Loss: 3.2256 | LR: 4.76e-04 | 1366.5ms/step | 14.6 samples/s
270
+ Step 6840/44000 | Loss: 3.2046 | LR: 4.76e-04 | 1368.7ms/step | 14.6 samples/s
271
+ Step 6850/44000 | Loss: 3.1972 | LR: 4.76e-04 | 1361.4ms/step | 14.7 samples/s
272
+ Step 6860/44000 | Loss: 3.2150 | LR: 4.76e-04 | 1359.5ms/step | 14.7 samples/s
273
+ Step 6870/44000 | Loss: 3.2143 | LR: 4.76e-04 | 1360.1ms/step | 14.7 samples/s
274
+ Step 6880/44000 | Loss: 3.2001 | LR: 4.76e-04 | 1359.9ms/step | 14.7 samples/s
275
+ Step 6890/44000 | Loss: 3.2245 | LR: 4.76e-04 | 1360.5ms/step | 14.7 samples/s
276
+ Step 6900/44000 | Loss: 3.2229 | LR: 4.75e-04 | 1359.1ms/step | 14.7 samples/s
277
+ Step 6910/44000 | Loss: 3.2345 | LR: 4.75e-04 | 1365.8ms/step | 14.6 samples/s
278
+ Step 6920/44000 | Loss: 3.2039 | LR: 4.75e-04 | 1366.1ms/step | 14.6 samples/s
279
+ Step 6930/44000 | Loss: 3.2063 | LR: 4.75e-04 | 1366.5ms/step | 14.6 samples/s
280
+ Step 6940/44000 | Loss: 3.1959 | LR: 4.75e-04 | 1362.3ms/step | 14.7 samples/s
281
+ Step 6950/44000 | Loss: 3.2157 | LR: 4.75e-04 | 1360.7ms/step | 14.7 samples/s
282
+ Step 6960/44000 | Loss: 3.2238 | LR: 4.75e-04 | 1365.7ms/step | 14.6 samples/s
283
+ Step 6970/44000 | Loss: 3.2117 | LR: 4.75e-04 | 1359.4ms/step | 14.7 samples/s
284
+ Step 6980/44000 | Loss: 3.2230 | LR: 4.75e-04 | 1367.1ms/step | 14.6 samples/s
285
+ Step 6990/44000 | Loss: 3.1790 | LR: 4.75e-04 | 1366.6ms/step | 14.6 samples/s
286
+ Step 7000/44000 | Loss: 3.2056 | LR: 4.75e-04 | 1363.1ms/step | 14.7 samples/s
287
+ Validation at step 7000: Loss = 3.2117, Perplexity = 24.82
288
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858/checkpoint-7000
289
+ Step 7010/44000 | Loss: 3.1991 | LR: 4.75e-04 | 3844.7ms/step | 5.2 samples/s
290
+ Step 7020/44000 | Loss: 3.1938 | LR: 4.75e-04 | 1368.5ms/step | 14.6 samples/s
291
+ Step 7030/44000 | Loss: 3.2163 | LR: 4.74e-04 | 1367.6ms/step | 14.6 samples/s
292
+ Step 7040/44000 | Loss: 3.1965 | LR: 4.74e-04 | 1369.9ms/step | 14.6 samples/s
293
+ Step 7050/44000 | Loss: 3.2026 | LR: 4.74e-04 | 1371.2ms/step | 14.6 samples/s
294
+ Step 7060/44000 | Loss: 3.2350 | LR: 4.74e-04 | 1371.6ms/step | 14.6 samples/s
295
+ Step 7070/44000 | Loss: 3.1964 | LR: 4.74e-04 | 1371.4ms/step | 14.6 samples/s
296
+ Step 7080/44000 | Loss: 3.2121 | LR: 4.74e-04 | 1372.3ms/step | 14.6 samples/s
297
+ Step 7090/44000 | Loss: 3.2047 | LR: 4.74e-04 | 1372.2ms/step | 14.6 samples/s
298
+ Step 7100/44000 | Loss: 3.2195 | LR: 4.74e-04 | 1371.9ms/step | 14.6 samples/s
299
+ Step 7110/44000 | Loss: 3.1956 | LR: 4.74e-04 | 1369.2ms/step | 14.6 samples/s
300
+ Step 7120/44000 | Loss: 3.1908 | LR: 4.74e-04 | 1370.5ms/step | 14.6 samples/s
301
+ Step 7130/44000 | Loss: 3.2035 | LR: 4.74e-04 | 1369.1ms/step | 14.6 samples/s
302
+ Step 7140/44000 | Loss: 3.1852 | LR: 4.74e-04 | 1369.3ms/step | 14.6 samples/s
303
+ Step 7150/44000 | Loss: 3.2132 | LR: 4.73e-04 | 1373.4ms/step | 14.6 samples/s
304
+ Step 7160/44000 | Loss: 3.1930 | LR: 4.73e-04 | 1371.1ms/step | 14.6 samples/s
305
+ Step 7170/44000 | Loss: 3.1950 | LR: 4.73e-04 | 1369.1ms/step | 14.6 samples/s
306
+ Step 7180/44000 | Loss: 3.1869 | LR: 4.73e-04 | 1369.6ms/step | 14.6 samples/s
307
+ Step 7190/44000 | Loss: 3.1600 | LR: 4.73e-04 | 1371.1ms/step | 14.6 samples/s
308
+ Step 7200/44000 | Loss: 3.1963 | LR: 4.73e-04 | 1370.1ms/step | 14.6 samples/s
309
+ Step 7210/44000 | Loss: 3.2094 | LR: 4.73e-04 | 1371.6ms/step | 14.6 samples/s
310
+ Step 7220/44000 | Loss: 3.1936 | LR: 4.73e-04 | 1370.7ms/step | 14.6 samples/s
311
+ Step 7230/44000 | Loss: 3.1979 | LR: 4.73e-04 | 1373.1ms/step | 14.6 samples/s
312
+ Step 7240/44000 | Loss: 3.1872 | LR: 4.73e-04 | 1371.4ms/step | 14.6 samples/s
313
+ Step 7250/44000 | Loss: 3.2161 | LR: 4.73e-04 | 1372.1ms/step | 14.6 samples/s
314
+ Step 7260/44000 | Loss: 3.1814 | LR: 4.73e-04 | 1373.9ms/step | 14.6 samples/s
315
+ Step 7270/44000 | Loss: 3.2205 | LR: 4.72e-04 | 1371.7ms/step | 14.6 samples/s
316
+ Step 7280/44000 | Loss: 3.2101 | LR: 4.72e-04 | 1370.8ms/step | 14.6 samples/s
317
+ Step 7290/44000 | Loss: 3.2176 | LR: 4.72e-04 | 1371.5ms/step | 14.6 samples/s
318
+ Step 7300/44000 | Loss: 3.2315 | LR: 4.72e-04 | 1371.0ms/step | 14.6 samples/s
319
+ Step 7310/44000 | Loss: 3.2000 | LR: 4.72e-04 | 1371.7ms/step | 14.6 samples/s
320
+ Step 7320/44000 | Loss: 3.1880 | LR: 4.72e-04 | 1372.9ms/step | 14.6 samples/s
321
+ Step 7330/44000 | Loss: 3.2177 | LR: 4.72e-04 | 1372.3ms/step | 14.6 samples/s
322
+ Step 7340/44000 | Loss: 3.1895 | LR: 4.72e-04 | 1373.8ms/step | 14.6 samples/s
323
+ Step 7350/44000 | Loss: 3.1816 | LR: 4.72e-04 | 1374.2ms/step | 14.6 samples/s
324
+ Step 7360/44000 | Loss: 3.1582 | LR: 4.72e-04 | 1373.7ms/step | 14.6 samples/s
325
+ Step 7370/44000 | Loss: 3.2077 | LR: 4.72e-04 | 1369.5ms/step | 14.6 samples/s
326
+ Step 7380/44000 | Loss: 3.1892 | LR: 4.72e-04 | 1368.8ms/step | 14.6 samples/s
327
+ Step 7390/44000 | Loss: 3.1618 | LR: 4.71e-04 | 1370.3ms/step | 14.6 samples/s
328
+ Step 7400/44000 | Loss: 3.2192 | LR: 4.71e-04 | 1371.8ms/step | 14.6 samples/s
329
+ Step 7410/44000 | Loss: 3.1851 | LR: 4.71e-04 | 1370.1ms/step | 14.6 samples/s
330
+ Step 7420/44000 | Loss: 3.1881 | LR: 4.71e-04 | 1372.7ms/step | 14.6 samples/s
331
+ Step 7430/44000 | Loss: 3.2146 | LR: 4.71e-04 | 1373.2ms/step | 14.6 samples/s
332
+ Step 7440/44000 | Loss: 3.1870 | LR: 4.71e-04 | 1371.2ms/step | 14.6 samples/s
333
+ Step 7450/44000 | Loss: 3.1656 | LR: 4.71e-04 | 1372.6ms/step | 14.6 samples/s
334
+ Step 7460/44000 | Loss: 3.2135 | LR: 4.71e-04 | 1371.7ms/step | 14.6 samples/s
335
+ Step 7470/44000 | Loss: 3.1982 | LR: 4.71e-04 | 1369.5ms/step | 14.6 samples/s
336
+ Step 7480/44000 | Loss: 3.2056 | LR: 4.71e-04 | 1371.3ms/step | 14.6 samples/s
337
+ Step 7490/44000 | Loss: 3.2005 | LR: 4.71e-04 | 1372.4ms/step | 14.6 samples/s
338
+ Step 7500/44000 | Loss: 3.1832 | LR: 4.71e-04 | 1371.9ms/step | 14.6 samples/s
339
+ Step 7510/44000 | Loss: 3.1712 | LR: 4.70e-04 | 1373.9ms/step | 14.6 samples/s
340
+ Step 7520/44000 | Loss: 3.1821 | LR: 4.70e-04 | 1372.8ms/step | 14.6 samples/s
341
+ Step 7530/44000 | Loss: 3.1801 | LR: 4.70e-04 | 1372.2ms/step | 14.6 samples/s
342
+ Step 7540/44000 | Loss: 3.1853 | LR: 4.70e-04 | 1373.3ms/step | 14.6 samples/s
343
+ Step 7550/44000 | Loss: 3.2051 | LR: 4.70e-04 | 1372.5ms/step | 14.6 samples/s
344
+ Step 7560/44000 | Loss: 3.1683 | LR: 4.70e-04 | 1372.1ms/step | 14.6 samples/s
345
+ Step 7570/44000 | Loss: 3.1790 | LR: 4.70e-04 | 1375.4ms/step | 14.5 samples/s
346
+ Step 7580/44000 | Loss: 3.1880 | LR: 4.70e-04 | 1372.6ms/step | 14.6 samples/s
347
+ Step 7590/44000 | Loss: 3.1700 | LR: 4.70e-04 | 1373.1ms/step | 14.6 samples/s
348
+ Step 7600/44000 | Loss: 3.2045 | LR: 4.70e-04 | 1370.4ms/step | 14.6 samples/s
349
+ Step 7610/44000 | Loss: 3.1867 | LR: 4.70e-04 | 1372.0ms/step | 14.6 samples/s
350
+ Step 7620/44000 | Loss: 3.1722 | LR: 4.70e-04 | 1371.3ms/step | 14.6 samples/s
351
+ Step 7630/44000 | Loss: 3.1963 | LR: 4.69e-04 | 1372.9ms/step | 14.6 samples/s
352
+ Step 7640/44000 | Loss: 3.1895 | LR: 4.69e-04 | 1371.9ms/step | 14.6 samples/s
353
+ Step 7650/44000 | Loss: 3.1872 | LR: 4.69e-04 | 1370.2ms/step | 14.6 samples/s
354
+ Step 7660/44000 | Loss: 3.1525 | LR: 4.69e-04 | 1369.5ms/step | 14.6 samples/s
355
+ Step 7670/44000 | Loss: 3.1876 | LR: 4.69e-04 | 1370.9ms/step | 14.6 samples/s
356
+ Step 7680/44000 | Loss: 3.2165 | LR: 4.69e-04 | 1371.9ms/step | 14.6 samples/s
357
+ Step 7690/44000 | Loss: 3.1640 | LR: 4.69e-04 | 1370.7ms/step | 14.6 samples/s
358
+ Step 7700/44000 | Loss: 3.1839 | LR: 4.69e-04 | 1371.4ms/step | 14.6 samples/s
359
+ Step 7710/44000 | Loss: 3.1538 | LR: 4.69e-04 | 1370.9ms/step | 14.6 samples/s
360
+ Step 7720/44000 | Loss: 3.1707 | LR: 4.69e-04 | 1371.0ms/step | 14.6 samples/s
361
+ Step 7730/44000 | Loss: 3.1707 | LR: 4.69e-04 | 1371.8ms/step | 14.6 samples/s
362
+ Step 7740/44000 | Loss: 3.1849 | LR: 4.68e-04 | 1371.1ms/step | 14.6 samples/s
363
+ Step 7750/44000 | Loss: 3.1623 | LR: 4.68e-04 | 1371.1ms/step | 14.6 samples/s
364
+ Step 7760/44000 | Loss: 3.1816 | LR: 4.68e-04 | 1370.6ms/step | 14.6 samples/s
365
+ Step 7770/44000 | Loss: 3.1607 | LR: 4.68e-04 | 1370.8ms/step | 14.6 samples/s
366
+ Step 7780/44000 | Loss: 3.1694 | LR: 4.68e-04 | 1371.3ms/step | 14.6 samples/s
367
+ Step 7790/44000 | Loss: 3.1881 | LR: 4.68e-04 | 1371.0ms/step | 14.6 samples/s
368
+ Step 7800/44000 | Loss: 3.1966 | LR: 4.68e-04 | 1371.2ms/step | 14.6 samples/s
369
+ Step 7810/44000 | Loss: 3.1855 | LR: 4.68e-04 | 1372.1ms/step | 14.6 samples/s
370
+ Step 7820/44000 | Loss: 3.1830 | LR: 4.68e-04 | 1371.3ms/step | 14.6 samples/s
371
+ Step 7830/44000 | Loss: 3.1748 | LR: 4.68e-04 | 1371.3ms/step | 14.6 samples/s
372
+ Step 7840/44000 | Loss: 3.1532 | LR: 4.68e-04 | 1371.3ms/step | 14.6 samples/s
373
+ Step 7850/44000 | Loss: 3.1913 | LR: 4.67e-04 | 1371.8ms/step | 14.6 samples/s
374
+ Step 7860/44000 | Loss: 3.2097 | LR: 4.67e-04 | 1371.5ms/step | 14.6 samples/s
375
+ Step 7870/44000 | Loss: 3.1518 | LR: 4.67e-04 | 1372.6ms/step | 14.6 samples/s
376
+ Step 7880/44000 | Loss: 3.1660 | LR: 4.67e-04 | 1370.3ms/step | 14.6 samples/s
377
+ Step 7890/44000 | Loss: 3.1829 | LR: 4.67e-04 | 1371.3ms/step | 14.6 samples/s
378
+ Step 7900/44000 | Loss: 3.1652 | LR: 4.67e-04 | 1370.1ms/step | 14.6 samples/s
379
+ Step 7910/44000 | Loss: 3.1394 | LR: 4.67e-04 | 1371.4ms/step | 14.6 samples/s
380
+ Step 7920/44000 | Loss: 3.1678 | LR: 4.67e-04 | 1372.1ms/step | 14.6 samples/s
381
+ Step 7930/44000 | Loss: 3.1943 | LR: 4.67e-04 | 1372.1ms/step | 14.6 samples/s
382
+ Step 7940/44000 | Loss: 3.1828 | LR: 4.67e-04 | 1370.8ms/step | 14.6 samples/s
383
+ Step 7950/44000 | Loss: 3.1440 | LR: 4.67e-04 | 1369.5ms/step | 14.6 samples/s
384
+ Step 7960/44000 | Loss: 3.1536 | LR: 4.66e-04 | 1370.6ms/step | 14.6 samples/s
385
+ Step 7970/44000 | Loss: 3.1806 | LR: 4.66e-04 | 1370.5ms/step | 14.6 samples/s
386
+ Step 7980/44000 | Loss: 3.1987 | LR: 4.66e-04 | 1369.2ms/step | 14.6 samples/s
387
+ Step 7990/44000 | Loss: 3.1762 | LR: 4.66e-04 | 1371.3ms/step | 14.6 samples/s
388
+ Step 8000/44000 | Loss: 3.1675 | LR: 4.66e-04 | 1371.2ms/step | 14.6 samples/s
389
+ Validation at step 8000: Loss = 3.1788, Perplexity = 24.02
390
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858/checkpoint-8000
391
+ Step 8010/44000 | Loss: 3.1949 | LR: 4.66e-04 | 3831.0ms/step | 5.2 samples/s
392
+ Step 8020/44000 | Loss: 3.1484 | LR: 4.66e-04 | 1368.6ms/step | 14.6 samples/s
393
+ Step 8030/44000 | Loss: 3.1718 | LR: 4.66e-04 | 1368.4ms/step | 14.6 samples/s
394
+ Step 8040/44000 | Loss: 3.1659 | LR: 4.66e-04 | 1371.1ms/step | 14.6 samples/s
395
+ Step 8050/44000 | Loss: 3.1925 | LR: 4.66e-04 | 1371.8ms/step | 14.6 samples/s
396
+ Step 8060/44000 | Loss: 3.1944 | LR: 4.66e-04 | 1372.3ms/step | 14.6 samples/s
397
+ Step 8070/44000 | Loss: 3.1803 | LR: 4.65e-04 | 1371.1ms/step | 14.6 samples/s
398
+ Step 8080/44000 | Loss: 3.1562 | LR: 4.65e-04 | 1371.5ms/step | 14.6 samples/s
399
+ Step 8090/44000 | Loss: 3.1827 | LR: 4.65e-04 | 1369.5ms/step | 14.6 samples/s
400
+ Step 8100/44000 | Loss: 3.1863 | LR: 4.65e-04 | 1369.1ms/step | 14.6 samples/s
401
+ Step 8110/44000 | Loss: 3.1640 | LR: 4.65e-04 | 1370.7ms/step | 14.6 samples/s
402
+ Step 8120/44000 | Loss: 3.1851 | LR: 4.65e-04 | 1370.6ms/step | 14.6 samples/s
403
+ Step 8130/44000 | Loss: 3.1402 | LR: 4.65e-04 | 1369.3ms/step | 14.6 samples/s
404
+ Step 8140/44000 | Loss: 3.1396 | LR: 4.65e-04 | 1370.7ms/step | 14.6 samples/s
405
+ Step 8150/44000 | Loss: 3.1653 | LR: 4.65e-04 | 1370.6ms/step | 14.6 samples/s
406
+ Step 8160/44000 | Loss: 3.1889 | LR: 4.65e-04 | 1371.5ms/step | 14.6 samples/s
407
+ Step 8170/44000 | Loss: 3.1575 | LR: 4.65e-04 | 1371.0ms/step | 14.6 samples/s
408
+ Step 8180/44000 | Loss: 3.1620 | LR: 4.64e-04 | 1372.6ms/step | 14.6 samples/s
409
+ Step 8190/44000 | Loss: 3.1678 | LR: 4.64e-04 | 1371.6ms/step | 14.6 samples/s
410
+ Step 8200/44000 | Loss: 3.1512 | LR: 4.64e-04 | 1368.2ms/step | 14.6 samples/s
411
+ Step 8210/44000 | Loss: 3.1582 | LR: 4.64e-04 | 1364.2ms/step | 14.7 samples/s
412
+ Step 8220/44000 | Loss: 3.1788 | LR: 4.64e-04 | 1362.8ms/step | 14.7 samples/s
413
+ Step 8230/44000 | Loss: 3.1609 | LR: 4.64e-04 | 1366.0ms/step | 14.6 samples/s
414
+ Step 8240/44000 | Loss: 3.1754 | LR: 4.64e-04 | 1365.5ms/step | 14.6 samples/s
415
+ Step 8250/44000 | Loss: 3.1752 | LR: 4.64e-04 | 1364.1ms/step | 14.7 samples/s
416
+ Step 8260/44000 | Loss: 3.1768 | LR: 4.64e-04 | 1366.0ms/step | 14.6 samples/s
417
+ Step 8270/44000 | Loss: 3.1853 | LR: 4.64e-04 | 1365.3ms/step | 14.6 samples/s
418
+ Step 8280/44000 | Loss: 3.1751 | LR: 4.64e-04 | 1365.8ms/step | 14.6 samples/s
419
+ Step 8290/44000 | Loss: 3.1434 | LR: 4.63e-04 | 1365.0ms/step | 14.7 samples/s
420
+ Step 8300/44000 | Loss: 3.1454 | LR: 4.63e-04 | 1365.0ms/step | 14.7 samples/s
421
+ Step 8310/44000 | Loss: 3.1625 | LR: 4.63e-04 | 1365.4ms/step | 14.6 samples/s
422
+ Step 8320/44000 | Loss: 3.1724 | LR: 4.63e-04 | 1366.3ms/step | 14.6 samples/s
423
+ Step 8330/44000 | Loss: 3.1607 | LR: 4.63e-04 | 1365.6ms/step | 14.6 samples/s
424
+ Step 8340/44000 | Loss: 3.1391 | LR: 4.63e-04 | 1366.1ms/step | 14.6 samples/s
425
+ Step 8350/44000 | Loss: 3.1616 | LR: 4.63e-04 | 1367.2ms/step | 14.6 samples/s
426
+ Step 8360/44000 | Loss: 3.1575 | LR: 4.63e-04 | 1366.7ms/step | 14.6 samples/s
427
+ Step 8370/44000 | Loss: 3.1548 | LR: 4.63e-04 | 1366.8ms/step | 14.6 samples/s
428
+ Step 8380/44000 | Loss: 3.1351 | LR: 4.63e-04 | 1365.4ms/step | 14.6 samples/s
429
+ Step 8390/44000 | Loss: 3.1596 | LR: 4.62e-04 | 1365.4ms/step | 14.6 samples/s
430
+ Step 8400/44000 | Loss: 3.1524 | LR: 4.62e-04 | 1366.2ms/step | 14.6 samples/s
431
+ Step 8410/44000 | Loss: 3.1750 | LR: 4.62e-04 | 1366.9ms/step | 14.6 samples/s
432
+ Step 8420/44000 | Loss: 3.1788 | LR: 4.62e-04 | 1365.5ms/step | 14.6 samples/s
433
+ Step 8430/44000 | Loss: 3.1851 | LR: 4.62e-04 | 1364.7ms/step | 14.7 samples/s
434
+ Step 8440/44000 | Loss: 3.1603 | LR: 4.62e-04 | 1365.1ms/step | 14.7 samples/s
435
+ Step 8450/44000 | Loss: 3.1447 | LR: 4.62e-04 | 1366.0ms/step | 14.6 samples/s
436
+ Step 8460/44000 | Loss: 3.1712 | LR: 4.62e-04 | 1370.9ms/step | 14.6 samples/s
437
+ Step 8470/44000 | Loss: 3.1334 | LR: 4.62e-04 | 1371.1ms/step | 14.6 samples/s
438
+ Step 8480/44000 | Loss: 3.1660 | LR: 4.62e-04 | 1373.5ms/step | 14.6 samples/s
439
+ Step 8490/44000 | Loss: 3.1532 | LR: 4.62e-04 | 1374.0ms/step | 14.6 samples/s
440
+ Step 8500/44000 | Loss: 3.1446 | LR: 4.61e-04 | 1373.1ms/step | 14.6 samples/s
441
+ Step 8510/44000 | Loss: 3.1881 | LR: 4.61e-04 | 1366.6ms/step | 14.6 samples/s
442
+ Step 8520/44000 | Loss: 3.1699 | LR: 4.61e-04 | 1365.3ms/step | 14.6 samples/s
443
+ Step 8530/44000 | Loss: 3.1333 | LR: 4.61e-04 | 1363.9ms/step | 14.7 samples/s
444
+ Step 8540/44000 | Loss: 3.1746 | LR: 4.61e-04 | 1365.5ms/step | 14.6 samples/s
445
+ Step 8550/44000 | Loss: 3.1550 | LR: 4.61e-04 | 1370.9ms/step | 14.6 samples/s
446
+ Step 8560/44000 | Loss: 3.1430 | LR: 4.61e-04 | 1370.1ms/step | 14.6 samples/s
447
+ Step 8570/44000 | Loss: 3.1668 | LR: 4.61e-04 | 1363.8ms/step | 14.7 samples/s
448
+ Step 8580/44000 | Loss: 3.1332 | LR: 4.61e-04 | 1363.6ms/step | 14.7 samples/s
449
+ Step 8590/44000 | Loss: 3.1373 | LR: 4.61e-04 | 1364.8ms/step | 14.7 samples/s
450
+ Step 8600/44000 | Loss: 3.1608 | LR: 4.60e-04 | 1364.5ms/step | 14.7 samples/s
451
+ Step 8610/44000 | Loss: 3.1526 | LR: 4.60e-04 | 1363.9ms/step | 14.7 samples/s
452
+ Step 8620/44000 | Loss: 3.1480 | LR: 4.60e-04 | 1362.7ms/step | 14.7 samples/s
453
+ Step 8630/44000 | Loss: 3.1965 | LR: 4.60e-04 | 1368.5ms/step | 14.6 samples/s
454
+ Step 8640/44000 | Loss: 3.1583 | LR: 4.60e-04 | 1371.2ms/step | 14.6 samples/s
455
+ Step 8650/44000 | Loss: 3.1518 | LR: 4.60e-04 | 1366.0ms/step | 14.6 samples/s
456
+ Step 8660/44000 | Loss: 3.1325 | LR: 4.60e-04 | 1366.4ms/step | 14.6 samples/s
457
+ Step 8670/44000 | Loss: 3.1488 | LR: 4.60e-04 | 1371.7ms/step | 14.6 samples/s
458
+ Step 8680/44000 | Loss: 3.1448 | LR: 4.60e-04 | 1370.1ms/step | 14.6 samples/s
459
+ Step 8690/44000 | Loss: 3.1505 | LR: 4.60e-04 | 1370.6ms/step | 14.6 samples/s
460
+ Step 8700/44000 | Loss: 3.1432 | LR: 4.59e-04 | 1369.2ms/step | 14.6 samples/s
461
+ Step 8710/44000 | Loss: 3.1467 | LR: 4.59e-04 | 1368.8ms/step | 14.6 samples/s
462
+ Step 8720/44000 | Loss: 3.1516 | LR: 4.59e-04 | 1369.3ms/step | 14.6 samples/s
463
+ Step 8730/44000 | Loss: 3.1339 | LR: 4.59e-04 | 1371.0ms/step | 14.6 samples/s
464
+ Step 8740/44000 | Loss: 3.1663 | LR: 4.59e-04 | 1369.8ms/step | 14.6 samples/s
465
+ Step 8750/44000 | Loss: 3.1345 | LR: 4.59e-04 | 1370.9ms/step | 14.6 samples/s
466
+ Step 8760/44000 | Loss: 3.1422 | LR: 4.59e-04 | 1371.4ms/step | 14.6 samples/s
467
+ Step 8770/44000 | Loss: 3.1473 | LR: 4.59e-04 | 1371.1ms/step | 14.6 samples/s
468
+ Step 8780/44000 | Loss: 3.1596 | LR: 4.59e-04 | 1369.8ms/step | 14.6 samples/s
469
+ Step 8790/44000 | Loss: 3.1537 | LR: 4.59e-04 | 1372.0ms/step | 14.6 samples/s
470
+ Step 8800/44000 | Loss: 3.1541 | LR: 4.58e-04 | 1370.3ms/step | 14.6 samples/s
471
+ Step 8810/44000 | Loss: 3.1830 | LR: 4.58e-04 | 1366.0ms/step | 14.6 samples/s
472
+ Step 8820/44000 | Loss: 3.1433 | LR: 4.58e-04 | 1370.2ms/step | 14.6 samples/s
473
+ Step 8830/44000 | Loss: 3.1581 | LR: 4.58e-04 | 1369.0ms/step | 14.6 samples/s
474
+ Step 8840/44000 | Loss: 3.1525 | LR: 4.58e-04 | 1370.6ms/step | 14.6 samples/s
475
+ Step 8850/44000 | Loss: 3.1450 | LR: 4.58e-04 | 1370.4ms/step | 14.6 samples/s
476
+ Step 8860/44000 | Loss: 3.1581 | LR: 4.58e-04 | 1368.0ms/step | 14.6 samples/s
477
+ Step 8870/44000 | Loss: 3.1744 | LR: 4.58e-04 | 1367.2ms/step | 14.6 samples/s
478
+ Step 8880/44000 | Loss: 3.1287 | LR: 4.58e-04 | 1362.9ms/step | 14.7 samples/s
479
+ Step 8890/44000 | Loss: 3.1725 | LR: 4.58e-04 | 1364.5ms/step | 14.7 samples/s
480
+ Step 8900/44000 | Loss: 3.1442 | LR: 4.57e-04 | 1361.8ms/step | 14.7 samples/s
481
+ Step 8910/44000 | Loss: 3.1452 | LR: 4.57e-04 | 1363.3ms/step | 14.7 samples/s
482
+ Step 8920/44000 | Loss: 3.1461 | LR: 4.57e-04 | 1364.6ms/step | 14.7 samples/s
483
+ Step 8930/44000 | Loss: 3.1478 | LR: 4.57e-04 | 1369.5ms/step | 14.6 samples/s
484
+ Step 8940/44000 | Loss: 3.1684 | LR: 4.57e-04 | 1369.3ms/step | 14.6 samples/s
485
+ Step 8950/44000 | Loss: 3.1390 | LR: 4.57e-04 | 1362.6ms/step | 14.7 samples/s
486
+ Step 8960/44000 | Loss: 3.1386 | LR: 4.57e-04 | 1363.4ms/step | 14.7 samples/s
487
+ Step 8970/44000 | Loss: 3.1558 | LR: 4.57e-04 | 1364.0ms/step | 14.7 samples/s
488
+ Step 8980/44000 | Loss: 3.1260 | LR: 4.57e-04 | 1363.5ms/step | 14.7 samples/s
489
+ Step 8990/44000 | Loss: 3.1274 | LR: 4.57e-04 | 1364.5ms/step | 14.7 samples/s
490
+ Step 9000/44000 | Loss: 3.1304 | LR: 4.56e-04 | 1363.8ms/step | 14.7 samples/s
491
+ Validation at step 9000: Loss = 3.1493, Perplexity = 23.32
492
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858/checkpoint-9000
493
+ Step 9010/44000 | Loss: 3.1575 | LR: 4.56e-04 | 3816.0ms/step | 5.2 samples/s
494
+ Step 9020/44000 | Loss: 3.1574 | LR: 4.56e-04 | 1371.5ms/step | 14.6 samples/s
495
+ Step 9030/44000 | Loss: 3.1434 | LR: 4.56e-04 | 1371.5ms/step | 14.6 samples/s
496
+ Step 9040/44000 | Loss: 3.1447 | LR: 4.56e-04 | 1372.3ms/step | 14.6 samples/s
497
+ Step 9050/44000 | Loss: 3.1582 | LR: 4.56e-04 | 1371.4ms/step | 14.6 samples/s
498
+ Step 9060/44000 | Loss: 3.1683 | LR: 4.56e-04 | 1370.8ms/step | 14.6 samples/s
499
+ Step 9070/44000 | Loss: 3.1534 | LR: 4.56e-04 | 1370.3ms/step | 14.6 samples/s
500
+ Step 9080/44000 | Loss: 3.1622 | LR: 4.56e-04 | 1369.7ms/step | 14.6 samples/s
501
+ Step 9090/44000 | Loss: 3.1815 | LR: 4.56e-04 | 1367.8ms/step | 14.6 samples/s
502
+ Step 9100/44000 | Loss: 3.1304 | LR: 4.55e-04 | 1369.8ms/step | 14.6 samples/s
503
+ Step 9110/44000 | Loss: 3.1559 | LR: 4.55e-04 | 1370.9ms/step | 14.6 samples/s
504
+ Step 9120/44000 | Loss: 3.1457 | LR: 4.55e-04 | 1370.2ms/step | 14.6 samples/s
505
+ Step 9130/44000 | Loss: 3.1556 | LR: 4.55e-04 | 1372.1ms/step | 14.6 samples/s
506
+ Step 9140/44000 | Loss: 3.1330 | LR: 4.55e-04 | 1370.9ms/step | 14.6 samples/s
507
+ Step 9150/44000 | Loss: 3.1259 | LR: 4.55e-04 | 1369.2ms/step | 14.6 samples/s
508
+ Step 9160/44000 | Loss: 3.1532 | LR: 4.55e-04 | 1369.1ms/step | 14.6 samples/s
509
+ Step 9170/44000 | Loss: 3.1370 | LR: 4.55e-04 | 1371.9ms/step | 14.6 samples/s
510
+ Step 9180/44000 | Loss: 3.1497 | LR: 4.55e-04 | 1371.9ms/step | 14.6 samples/s
511
+ Step 9190/44000 | Loss: 3.1488 | LR: 4.54e-04 | 1372.1ms/step | 14.6 samples/s
512
+ Step 9200/44000 | Loss: 3.1245 | LR: 4.54e-04 | 1369.8ms/step | 14.6 samples/s
513
+ Step 9210/44000 | Loss: 3.1570 | LR: 4.54e-04 | 1371.6ms/step | 14.6 samples/s
514
+ Step 9220/44000 | Loss: 3.1306 | LR: 4.54e-04 | 1372.5ms/step | 14.6 samples/s
515
+ Step 9230/44000 | Loss: 3.1472 | LR: 4.54e-04 | 1372.3ms/step | 14.6 samples/s
516
+ Step 9240/44000 | Loss: 3.1317 | LR: 4.54e-04 | 1373.2ms/step | 14.6 samples/s
517
+ Step 9250/44000 | Loss: 3.1572 | LR: 4.54e-04 | 1373.5ms/step | 14.6 samples/s
518
+ Step 9260/44000 | Loss: 3.1418 | LR: 4.54e-04 | 1372.8ms/step | 14.6 samples/s
519
+ Step 9270/44000 | Loss: 3.1434 | LR: 4.54e-04 | 1370.1ms/step | 14.6 samples/s
520
+ Step 9280/44000 | Loss: 3.1314 | LR: 4.54e-04 | 1370.6ms/step | 14.6 samples/s
521
+ Step 9290/44000 | Loss: 3.1122 | LR: 4.53e-04 | 1373.7ms/step | 14.6 samples/s
522
+ Step 9300/44000 | Loss: 3.1182 | LR: 4.53e-04 | 1373.7ms/step | 14.6 samples/s
523
+ Step 9310/44000 | Loss: 3.1003 | LR: 4.53e-04 | 1372.5ms/step | 14.6 samples/s
524
+ Step 9320/44000 | Loss: 3.1147 | LR: 4.53e-04 | 1373.4ms/step | 14.6 samples/s
525
+ Step 9330/44000 | Loss: 3.1124 | LR: 4.53e-04 | 1374.1ms/step | 14.6 samples/s
526
+ Step 9340/44000 | Loss: 3.1101 | LR: 4.53e-04 | 1373.0ms/step | 14.6 samples/s
527
+ Step 9350/44000 | Loss: 3.1290 | LR: 4.53e-04 | 1372.0ms/step | 14.6 samples/s
528
+ Step 9360/44000 | Loss: 3.1361 | LR: 4.53e-04 | 1374.3ms/step | 14.6 samples/s
529
+ Step 9370/44000 | Loss: 3.1418 | LR: 4.53e-04 | 1370.6ms/step | 14.6 samples/s
530
+ Step 9380/44000 | Loss: 3.1538 | LR: 4.52e-04 | 1364.9ms/step | 14.7 samples/s
531
+ Step 9390/44000 | Loss: 3.1390 | LR: 4.52e-04 | 1371.0ms/step | 14.6 samples/s
532
+ Step 9400/44000 | Loss: 3.1465 | LR: 4.52e-04 | 1372.5ms/step | 14.6 samples/s
533
+ Step 9410/44000 | Loss: 3.1228 | LR: 4.52e-04 | 1372.5ms/step | 14.6 samples/s
534
+ Step 9420/44000 | Loss: 3.1152 | LR: 4.52e-04 | 1371.8ms/step | 14.6 samples/s
535
+ Step 9430/44000 | Loss: 3.1366 | LR: 4.52e-04 | 1365.9ms/step | 14.6 samples/s
536
+ Step 9440/44000 | Loss: 3.1458 | LR: 4.52e-04 | 1366.2ms/step | 14.6 samples/s
537
+ Step 9450/44000 | Loss: 3.1401 | LR: 4.52e-04 | 1364.3ms/step | 14.7 samples/s
538
+ Step 9460/44000 | Loss: 3.1425 | LR: 4.52e-04 | 1364.7ms/step | 14.7 samples/s
539
+ Step 9470/44000 | Loss: 3.1373 | LR: 4.52e-04 | 1364.5ms/step | 14.7 samples/s
540
+ Step 9480/44000 | Loss: 3.1224 | LR: 4.51e-04 | 1364.7ms/step | 14.7 samples/s
541
+ Step 9490/44000 | Loss: 3.0991 | LR: 4.51e-04 | 1364.2ms/step | 14.7 samples/s
542
+ Step 9500/44000 | Loss: 3.1419 | LR: 4.51e-04 | 1365.5ms/step | 14.6 samples/s
543
+ Step 9510/44000 | Loss: 3.1438 | LR: 4.51e-04 | 1366.5ms/step | 14.6 samples/s
544
+ Step 9520/44000 | Loss: 3.1374 | LR: 4.51e-04 | 1365.3ms/step | 14.6 samples/s
545
+ Step 9530/44000 | Loss: 3.1433 | LR: 4.51e-04 | 1363.9ms/step | 14.7 samples/s
546
+ Step 9540/44000 | Loss: 3.1349 | LR: 4.51e-04 | 1363.1ms/step | 14.7 samples/s
547
+ Step 9550/44000 | Loss: 3.1132 | LR: 4.51e-04 | 1364.5ms/step | 14.7 samples/s
548
+ Step 9560/44000 | Loss: 3.1451 | LR: 4.51e-04 | 1369.7ms/step | 14.6 samples/s
549
+ Step 9570/44000 | Loss: 3.1586 | LR: 4.50e-04 | 1373.0ms/step | 14.6 samples/s
550
+ Step 9580/44000 | Loss: 3.1464 | LR: 4.50e-04 | 1372.4ms/step | 14.6 samples/s
551
+ Step 9590/44000 | Loss: 3.1226 | LR: 4.50e-04 | 1371.9ms/step | 14.6 samples/s
552
+ Step 9600/44000 | Loss: 3.1415 | LR: 4.50e-04 | 1371.5ms/step | 14.6 samples/s
553
+ Step 9610/44000 | Loss: 3.1208 | LR: 4.50e-04 | 1373.1ms/step | 14.6 samples/s
554
+ Step 9620/44000 | Loss: 3.1378 | LR: 4.50e-04 | 1372.8ms/step | 14.6 samples/s
555
+ Step 9630/44000 | Loss: 3.1528 | LR: 4.50e-04 | 1372.0ms/step | 14.6 samples/s
556
+ Step 9640/44000 | Loss: 3.1557 | LR: 4.50e-04 | 1372.1ms/step | 14.6 samples/s
557
+ Step 9650/44000 | Loss: 3.1020 | LR: 4.50e-04 | 1371.3ms/step | 14.6 samples/s
558
+ Step 9660/44000 | Loss: 3.1013 | LR: 4.49e-04 | 1369.1ms/step | 14.6 samples/s
559
+ Step 9670/44000 | Loss: 3.1299 | LR: 4.49e-04 | 1370.2ms/step | 14.6 samples/s
560
+ Step 9680/44000 | Loss: 3.1137 | LR: 4.49e-04 | 1371.5ms/step | 14.6 samples/s
561
+ Step 9690/44000 | Loss: 3.1423 | LR: 4.49e-04 | 1371.1ms/step | 14.6 samples/s
562
+ Step 9700/44000 | Loss: 3.1236 | LR: 4.49e-04 | 1370.5ms/step | 14.6 samples/s
563
+ Step 9710/44000 | Loss: 3.1395 | LR: 4.49e-04 | 1369.4ms/step | 14.6 samples/s
564
+ Step 9720/44000 | Loss: 3.1200 | LR: 4.49e-04 | 1369.6ms/step | 14.6 samples/s
565
+ Step 9730/44000 | Loss: 3.0902 | LR: 4.49e-04 | 1370.2ms/step | 14.6 samples/s
566
+ Step 9740/44000 | Loss: 3.1300 | LR: 4.49e-04 | 1372.1ms/step | 14.6 samples/s
567
+ Step 9750/44000 | Loss: 3.0976 | LR: 4.48e-04 | 1371.8ms/step | 14.6 samples/s
568
+ Step 9760/44000 | Loss: 3.1055 | LR: 4.48e-04 | 1370.9ms/step | 14.6 samples/s
569
+ Step 9770/44000 | Loss: 3.1518 | LR: 4.48e-04 | 1372.5ms/step | 14.6 samples/s
570
+ Step 9780/44000 | Loss: 3.1365 | LR: 4.48e-04 | 1371.6ms/step | 14.6 samples/s
571
+ Step 9790/44000 | Loss: 3.1124 | LR: 4.48e-04 | 1372.0ms/step | 14.6 samples/s
572
+ Step 9800/44000 | Loss: 3.1266 | LR: 4.48e-04 | 1371.9ms/step | 14.6 samples/s
573
+ Step 9810/44000 | Loss: 3.1589 | LR: 4.48e-04 | 1371.4ms/step | 14.6 samples/s
574
+ Step 9820/44000 | Loss: 3.1135 | LR: 4.48e-04 | 1371.1ms/step | 14.6 samples/s
575
+ Step 9830/44000 | Loss: 3.1231 | LR: 4.48e-04 | 1371.5ms/step | 14.6 samples/s
576
+ Step 9840/44000 | Loss: 3.1238 | LR: 4.47e-04 | 1371.1ms/step | 14.6 samples/s
577
+ Step 9850/44000 | Loss: 3.1231 | LR: 4.47e-04 | 1372.3ms/step | 14.6 samples/s
578
+ Step 9860/44000 | Loss: 3.1071 | LR: 4.47e-04 | 1371.2ms/step | 14.6 samples/s
579
+ Step 9870/44000 | Loss: 3.1276 | LR: 4.47e-04 | 1369.3ms/step | 14.6 samples/s
580
+ Step 9880/44000 | Loss: 3.1255 | LR: 4.47e-04 | 1371.1ms/step | 14.6 samples/s
581
+ Step 9890/44000 | Loss: 3.1398 | LR: 4.47e-04 | 1373.1ms/step | 14.6 samples/s
582
+ Step 9900/44000 | Loss: 3.1082 | LR: 4.47e-04 | 1371.7ms/step | 14.6 samples/s
583
+ Step 9910/44000 | Loss: 3.1222 | LR: 4.47e-04 | 1371.4ms/step | 14.6 samples/s
584
+ Step 9920/44000 | Loss: 3.1418 | LR: 4.47e-04 | 1370.2ms/step | 14.6 samples/s
585
+ Step 9930/44000 | Loss: 3.1149 | LR: 4.46e-04 | 1373.0ms/step | 14.6 samples/s
586
+ Step 9940/44000 | Loss: 3.1001 | LR: 4.46e-04 | 1371.4ms/step | 14.6 samples/s
587
+ Step 9950/44000 | Loss: 3.0831 | LR: 4.46e-04 | 1371.8ms/step | 14.6 samples/s
588
+ Step 9960/44000 | Loss: 3.1435 | LR: 4.46e-04 | 1370.2ms/step | 14.6 samples/s
589
+ Step 9970/44000 | Loss: 3.1091 | LR: 4.46e-04 | 1371.0ms/step | 14.6 samples/s
590
+ Step 9980/44000 | Loss: 3.1291 | LR: 4.46e-04 | 1371.9ms/step | 14.6 samples/s
591
+ Step 9990/44000 | Loss: 3.0964 | LR: 4.46e-04 | 1370.2ms/step | 14.6 samples/s
592
+ Step 10000/44000 | Loss: 3.1122 | LR: 4.46e-04 | 1369.8ms/step | 14.6 samples/s
593
+ Validation at step 10000: Loss = 3.1268, Perplexity = 22.80
594
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858/checkpoint-10000
595
+ Step 10010/44000 | Loss: 3.1284 | LR: 4.46e-04 | 3829.4ms/step | 5.2 samples/s
596
+ Step 10020/44000 | Loss: 3.1303 | LR: 4.45e-04 | 1369.7ms/step | 14.6 samples/s
597
+ Step 10030/44000 | Loss: 3.1130 | LR: 4.45e-04 | 1372.8ms/step | 14.6 samples/s
598
+ Step 10040/44000 | Loss: 3.1328 | LR: 4.45e-04 | 1370.2ms/step | 14.6 samples/s
599
+ Step 10050/44000 | Loss: 3.1539 | LR: 4.45e-04 | 1372.2ms/step | 14.6 samples/s
600
+ Step 10060/44000 | Loss: 3.1144 | LR: 4.45e-04 | 1372.0ms/step | 14.6 samples/s
601
+ Step 10070/44000 | Loss: 3.0831 | LR: 4.45e-04 | 1372.7ms/step | 14.6 samples/s
602
+ Step 10080/44000 | Loss: 3.1575 | LR: 4.45e-04 | 1371.5ms/step | 14.6 samples/s
603
+ Step 10090/44000 | Loss: 3.0910 | LR: 4.45e-04 | 1373.9ms/step | 14.6 samples/s
604
+ Step 10100/44000 | Loss: 3.1012 | LR: 4.45e-04 | 1370.4ms/step | 14.6 samples/s
605
+ Step 10110/44000 | Loss: 3.1353 | LR: 4.44e-04 | 1370.7ms/step | 14.6 samples/s
606
+ Step 10120/44000 | Loss: 3.1264 | LR: 4.44e-04 | 1370.5ms/step | 14.6 samples/s
607
+ Step 10130/44000 | Loss: 3.0805 | LR: 4.44e-04 | 1371.1ms/step | 14.6 samples/s
608
+ Step 10140/44000 | Loss: 3.1188 | LR: 4.44e-04 | 1371.3ms/step | 14.6 samples/s
609
+ Step 10150/44000 | Loss: 3.1124 | LR: 4.44e-04 | 1372.5ms/step | 14.6 samples/s
610
+ Step 10160/44000 | Loss: 3.1347 | LR: 4.44e-04 | 1371.1ms/step | 14.6 samples/s
611
+ Step 10170/44000 | Loss: 3.1215 | LR: 4.44e-04 | 1370.2ms/step | 14.6 samples/s
612
+ Step 10180/44000 | Loss: 3.1333 | LR: 4.44e-04 | 1371.3ms/step | 14.6 samples/s
613
+ Step 10190/44000 | Loss: 3.0974 | LR: 4.43e-04 | 1370.7ms/step | 14.6 samples/s
614
+ Step 10200/44000 | Loss: 3.1066 | LR: 4.43e-04 | 1369.8ms/step | 14.6 samples/s
615
+ Step 10210/44000 | Loss: 3.1245 | LR: 4.43e-04 | 1364.7ms/step | 14.7 samples/s
616
+ Step 10220/44000 | Loss: 3.1076 | LR: 4.43e-04 | 1364.1ms/step | 14.7 samples/s
617
+ Step 10230/44000 | Loss: 3.1563 | LR: 4.43e-04 | 1363.6ms/step | 14.7 samples/s
618
+ Step 10240/44000 | Loss: 3.1287 | LR: 4.43e-04 | 1363.7ms/step | 14.7 samples/s
619
+ Step 10250/44000 | Loss: 3.1143 | LR: 4.43e-04 | 1362.9ms/step | 14.7 samples/s
620
+ Step 10260/44000 | Loss: 3.1443 | LR: 4.43e-04 | 1364.4ms/step | 14.7 samples/s
621
+ Step 10270/44000 | Loss: 3.1337 | LR: 4.43e-04 | 1364.2ms/step | 14.7 samples/s
622
+ Step 10280/44000 | Loss: 3.1291 | LR: 4.42e-04 | 1363.0ms/step | 14.7 samples/s
623
+ Step 10290/44000 | Loss: 3.1198 | LR: 4.42e-04 | 1365.1ms/step | 14.7 samples/s
624
+ Step 10300/44000 | Loss: 3.1388 | LR: 4.42e-04 | 1364.0ms/step | 14.7 samples/s
625
+ Step 10310/44000 | Loss: 3.1481 | LR: 4.42e-04 | 1363.3ms/step | 14.7 samples/s
626
+ Step 10320/44000 | Loss: 3.1257 | LR: 4.42e-04 | 1363.6ms/step | 14.7 samples/s
627
+ Step 10330/44000 | Loss: 3.1002 | LR: 4.42e-04 | 1365.0ms/step | 14.7 samples/s
628
+ Step 10340/44000 | Loss: 3.0863 | LR: 4.42e-04 | 1364.8ms/step | 14.7 samples/s
629
+ Step 10350/44000 | Loss: 3.0927 | LR: 4.42e-04 | 1365.9ms/step | 14.6 samples/s
630
+ Step 10360/44000 | Loss: 3.0868 | LR: 4.42e-04 | 1366.0ms/step | 14.6 samples/s
631
+ Step 10370/44000 | Loss: 3.1175 | LR: 4.41e-04 | 1362.8ms/step | 14.7 samples/s
632
+ Step 10380/44000 | Loss: 3.1145 | LR: 4.41e-04 | 1369.9ms/step | 14.6 samples/s
633
+ Step 10390/44000 | Loss: 3.1393 | LR: 4.41e-04 | 1371.4ms/step | 14.6 samples/s
634
+ Step 10400/44000 | Loss: 3.0928 | LR: 4.41e-04 | 1372.0ms/step | 14.6 samples/s
635
+ Step 10410/44000 | Loss: 3.1260 | LR: 4.41e-04 | 1366.4ms/step | 14.6 samples/s
636
+ Step 10420/44000 | Loss: 3.1007 | LR: 4.41e-04 | 1364.5ms/step | 14.7 samples/s
637
+ Step 10430/44000 | Loss: 3.1123 | LR: 4.41e-04 | 1365.4ms/step | 14.6 samples/s
638
+ Step 10440/44000 | Loss: 3.1330 | LR: 4.41e-04 | 1366.5ms/step | 14.6 samples/s
639
+ Step 10450/44000 | Loss: 3.1158 | LR: 4.40e-04 | 1362.7ms/step | 14.7 samples/s
640
+ Step 10460/44000 | Loss: 3.1149 | LR: 4.40e-04 | 1364.0ms/step | 14.7 samples/s
641
+ Step 10470/44000 | Loss: 3.1197 | LR: 4.40e-04 | 1362.1ms/step | 14.7 samples/s
642
+ Step 10480/44000 | Loss: 3.1090 | LR: 4.40e-04 | 1362.2ms/step | 14.7 samples/s
643
+ Step 10490/44000 | Loss: 3.1216 | LR: 4.40e-04 | 1368.4ms/step | 14.6 samples/s
644
+ Step 10500/44000 | Loss: 3.0936 | LR: 4.40e-04 | 1370.3ms/step | 14.6 samples/s
645
+ Step 10510/44000 | Loss: 3.1226 | LR: 4.40e-04 | 1370.7ms/step | 14.6 samples/s
646
+ Step 10520/44000 | Loss: 3.0953 | LR: 4.40e-04 | 1371.7ms/step | 14.6 samples/s
647
+ Step 10530/44000 | Loss: 3.0795 | LR: 4.40e-04 | 1367.0ms/step | 14.6 samples/s
648
+ Step 10540/44000 | Loss: 3.1136 | LR: 4.39e-04 | 1364.3ms/step | 14.7 samples/s
649
+ Step 10550/44000 | Loss: 3.0996 | LR: 4.39e-04 | 1362.4ms/step | 14.7 samples/s
650
+ Step 10560/44000 | Loss: 3.1018 | LR: 4.39e-04 | 1366.6ms/step | 14.6 samples/s
651
+ Step 10570/44000 | Loss: 3.1023 | LR: 4.39e-04 | 1369.2ms/step | 14.6 samples/s
652
+ Step 10580/44000 | Loss: 3.1286 | LR: 4.39e-04 | 1370.5ms/step | 14.6 samples/s
653
+ Step 10590/44000 | Loss: 3.1010 | LR: 4.39e-04 | 1371.8ms/step | 14.6 samples/s
654
+ Step 10600/44000 | Loss: 3.1012 | LR: 4.39e-04 | 1368.3ms/step | 14.6 samples/s
655
+ Step 10610/44000 | Loss: 3.0906 | LR: 4.39e-04 | 1364.5ms/step | 14.7 samples/s
656
+ Step 10620/44000 | Loss: 3.1185 | LR: 4.38e-04 | 1370.8ms/step | 14.6 samples/s
657
+ Step 10630/44000 | Loss: 3.0869 | LR: 4.38e-04 | 1369.0ms/step | 14.6 samples/s
658
+ Step 10640/44000 | Loss: 3.1099 | LR: 4.38e-04 | 1370.4ms/step | 14.6 samples/s
659
+ Step 10650/44000 | Loss: 3.1283 | LR: 4.38e-04 | 1372.3ms/step | 14.6 samples/s
660
+ Step 10660/44000 | Loss: 3.1207 | LR: 4.38e-04 | 1369.9ms/step | 14.6 samples/s
661
+ Step 10670/44000 | Loss: 3.0895 | LR: 4.38e-04 | 1370.4ms/step | 14.6 samples/s
662
+ Step 10680/44000 | Loss: 3.1130 | LR: 4.38e-04 | 1371.2ms/step | 14.6 samples/s
663
+ Step 10690/44000 | Loss: 3.1170 | LR: 4.38e-04 | 1369.9ms/step | 14.6 samples/s
664
+ Step 10700/44000 | Loss: 3.0896 | LR: 4.37e-04 | 1372.0ms/step | 14.6 samples/s
665
+ Step 10710/44000 | Loss: 3.0897 | LR: 4.37e-04 | 1370.9ms/step | 14.6 samples/s
666
+ Step 10720/44000 | Loss: 3.1149 | LR: 4.37e-04 | 1372.0ms/step | 14.6 samples/s
667
+ Step 10730/44000 | Loss: 3.0815 | LR: 4.37e-04 | 1372.3ms/step | 14.6 samples/s
668
+ Step 10740/44000 | Loss: 3.1070 | LR: 4.37e-04 | 1363.8ms/step | 14.7 samples/s
669
+ Step 10750/44000 | Loss: 3.1263 | LR: 4.37e-04 | 1370.0ms/step | 14.6 samples/s
670
+ Step 10760/44000 | Loss: 3.1169 | LR: 4.37e-04 | 1370.4ms/step | 14.6 samples/s
671
+ Step 10770/44000 | Loss: 3.1081 | LR: 4.37e-04 | 1369.2ms/step | 14.6 samples/s
672
+ Step 10780/44000 | Loss: 3.0939 | LR: 4.37e-04 | 1372.3ms/step | 14.6 samples/s
673
+ Step 10790/44000 | Loss: 3.1270 | LR: 4.36e-04 | 1370.1ms/step | 14.6 samples/s
674
+ Step 10800/44000 | Loss: 3.1149 | LR: 4.36e-04 | 1371.6ms/step | 14.6 samples/s
675
+ Step 10810/44000 | Loss: 3.1097 | LR: 4.36e-04 | 1369.3ms/step | 14.6 samples/s
676
+ Step 10820/44000 | Loss: 3.0903 | LR: 4.36e-04 | 1369.0ms/step | 14.6 samples/s
677
+ Step 10830/44000 | Loss: 3.1010 | LR: 4.36e-04 | 1370.2ms/step | 14.6 samples/s
678
+ Step 10840/44000 | Loss: 3.0978 | LR: 4.36e-04 | 1371.3ms/step | 14.6 samples/s
679
+ Step 10850/44000 | Loss: 3.0944 | LR: 4.36e-04 | 1371.8ms/step | 14.6 samples/s
680
+ Step 10860/44000 | Loss: 3.1173 | LR: 4.36e-04 | 1370.0ms/step | 14.6 samples/s
681
+ Step 10870/44000 | Loss: 3.0866 | LR: 4.35e-04 | 1369.1ms/step | 14.6 samples/s
682
+ Step 10880/44000 | Loss: 3.0863 | LR: 4.35e-04 | 1371.1ms/step | 14.6 samples/s
683
+ Step 10890/44000 | Loss: 3.1191 | LR: 4.35e-04 | 1370.6ms/step | 14.6 samples/s
684
+ Step 10900/44000 | Loss: 3.0936 | LR: 4.35e-04 | 1369.7ms/step | 14.6 samples/s
685
+ Step 10910/44000 | Loss: 3.0997 | LR: 4.35e-04 | 1370.8ms/step | 14.6 samples/s
686
+ Step 10920/44000 | Loss: 3.1259 | LR: 4.35e-04 | 1370.0ms/step | 14.6 samples/s
687
+ Step 10930/44000 | Loss: 3.1051 | LR: 4.35e-04 | 1368.1ms/step | 14.6 samples/s
688
+ Step 10940/44000 | Loss: 3.0899 | LR: 4.35e-04 | 1369.3ms/step | 14.6 samples/s
689
+ Step 10950/44000 | Loss: 3.1111 | LR: 4.34e-04 | 1372.2ms/step | 14.6 samples/s
690
+ Step 10960/44000 | Loss: 3.1128 | LR: 4.34e-04 | 1369.7ms/step | 14.6 samples/s
691
+ Step 10970/44000 | Loss: 3.0779 | LR: 4.34e-04 | 1368.8ms/step | 14.6 samples/s
692
+ Step 10980/44000 | Loss: 3.0927 | LR: 4.34e-04 | 1371.3ms/step | 14.6 samples/s
693
+ Step 10990/44000 | Loss: 3.0889 | LR: 4.34e-04 | 1371.8ms/step | 14.6 samples/s
694
+ Step 11000/44000 | Loss: 3.0940 | LR: 4.34e-04 | 1370.6ms/step | 14.6 samples/s
695
+ Validation at step 11000: Loss = 3.1050, Perplexity = 22.31
696
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858/checkpoint-11000
697
+ Step 11010/44000 | Loss: 3.1130 | LR: 4.34e-04 | 3824.2ms/step | 5.2 samples/s
698
+ Step 11020/44000 | Loss: 3.1013 | LR: 4.34e-04 | 1370.0ms/step | 14.6 samples/s
699
+ Step 11030/44000 | Loss: 3.1208 | LR: 4.33e-04 | 1371.2ms/step | 14.6 samples/s
700
+ Step 11040/44000 | Loss: 3.0878 | LR: 4.33e-04 | 1369.9ms/step | 14.6 samples/s
701
+ Step 11050/44000 | Loss: 3.1203 | LR: 4.33e-04 | 1369.9ms/step | 14.6 samples/s
702
+ Step 11060/44000 | Loss: 3.1071 | LR: 4.33e-04 | 1371.8ms/step | 14.6 samples/s
703
+ Step 11070/44000 | Loss: 3.1131 | LR: 4.33e-04 | 1372.0ms/step | 14.6 samples/s
704
+ Step 11080/44000 | Loss: 3.0985 | LR: 4.33e-04 | 1371.1ms/step | 14.6 samples/s
705
+ Step 11090/44000 | Loss: 3.1116 | LR: 4.33e-04 | 1372.3ms/step | 14.6 samples/s
706
+ Step 11100/44000 | Loss: 3.1104 | LR: 4.33e-04 | 1370.4ms/step | 14.6 samples/s
707
+ Step 11110/44000 | Loss: 3.1105 | LR: 4.32e-04 | 1371.3ms/step | 14.6 samples/s
708
+ Step 11120/44000 | Loss: 3.1061 | LR: 4.32e-04 | 1372.6ms/step | 14.6 samples/s
709
+ Step 11130/44000 | Loss: 3.1120 | LR: 4.32e-04 | 1370.7ms/step | 14.6 samples/s
710
+ Step 11140/44000 | Loss: 3.1338 | LR: 4.32e-04 | 1371.7ms/step | 14.6 samples/s
711
+ Step 11150/44000 | Loss: 3.0906 | LR: 4.32e-04 | 1369.1ms/step | 14.6 samples/s
712
+ Step 11160/44000 | Loss: 3.0975 | LR: 4.32e-04 | 1371.2ms/step | 14.6 samples/s
713
+ Step 11170/44000 | Loss: 3.1170 | LR: 4.32e-04 | 1370.9ms/step | 14.6 samples/s
714
+ Step 11180/44000 | Loss: 3.0924 | LR: 4.32e-04 | 1371.2ms/step | 14.6 samples/s
715
+ Step 11190/44000 | Loss: 3.1058 | LR: 4.31e-04 | 1371.9ms/step | 14.6 samples/s
716
+ Step 11200/44000 | Loss: 3.1037 | LR: 4.31e-04 | 1373.0ms/step | 14.6 samples/s
717
+ Step 11210/44000 | Loss: 3.1107 | LR: 4.31e-04 | 1373.9ms/step | 14.6 samples/s
718
+ Step 11220/44000 | Loss: 3.0845 | LR: 4.31e-04 | 1374.7ms/step | 14.5 samples/s
719
+ Step 11230/44000 | Loss: 3.0784 | LR: 4.31e-04 | 1373.6ms/step | 14.6 samples/s
720
+ Step 11240/44000 | Loss: 3.1133 | LR: 4.31e-04 | 1376.0ms/step | 14.5 samples/s
721
+ Step 11250/44000 | Loss: 3.0972 | LR: 4.31e-04 | 1374.6ms/step | 14.5 samples/s
722
+ Step 11260/44000 | Loss: 3.0690 | LR: 4.31e-04 | 1372.4ms/step | 14.6 samples/s
723
+ Step 11270/44000 | Loss: 3.1069 | LR: 4.30e-04 | 1373.2ms/step | 14.6 samples/s
724
+ Step 11280/44000 | Loss: 3.0847 | LR: 4.30e-04 | 1373.4ms/step | 14.6 samples/s
725
+ Step 11290/44000 | Loss: 3.0714 | LR: 4.30e-04 | 1373.8ms/step | 14.6 samples/s
726
+ Step 11300/44000 | Loss: 3.0588 | LR: 4.30e-04 | 1373.3ms/step | 14.6 samples/s
727
+ Step 11310/44000 | Loss: 3.0898 | LR: 4.30e-04 | 1373.5ms/step | 14.6 samples/s
728
+ Step 11320/44000 | Loss: 3.0941 | LR: 4.30e-04 | 1373.1ms/step | 14.6 samples/s
729
+ Step 11330/44000 | Loss: 3.0902 | LR: 4.30e-04 | 1374.2ms/step | 14.6 samples/s
730
+ Step 11340/44000 | Loss: 3.0895 | LR: 4.30e-04 | 1373.8ms/step | 14.6 samples/s
731
+ Step 11350/44000 | Loss: 3.0811 | LR: 4.29e-04 | 1373.1ms/step | 14.6 samples/s
732
+ Step 11360/44000 | Loss: 3.0956 | LR: 4.29e-04 | 1371.4ms/step | 14.6 samples/s
733
+ Step 11370/44000 | Loss: 3.0725 | LR: 4.29e-04 | 1373.1ms/step | 14.6 samples/s
734
+ Step 11380/44000 | Loss: 3.0776 | LR: 4.29e-04 | 1372.4ms/step | 14.6 samples/s
735
+ Step 11390/44000 | Loss: 3.1036 | LR: 4.29e-04 | 1374.3ms/step | 14.6 samples/s
736
+ Step 11400/44000 | Loss: 3.1053 | LR: 4.29e-04 | 1374.2ms/step | 14.6 samples/s
737
+ Step 11410/44000 | Loss: 3.1056 | LR: 4.29e-04 | 1372.3ms/step | 14.6 samples/s
738
+ Step 11420/44000 | Loss: 3.0508 | LR: 4.29e-04 | 1370.9ms/step | 14.6 samples/s
739
+ Step 11430/44000 | Loss: 3.0850 | LR: 4.28e-04 | 1374.5ms/step | 14.6 samples/s
740
+ Step 11440/44000 | Loss: 3.1066 | LR: 4.28e-04 | 1373.7ms/step | 14.6 samples/s
741
+ Step 11450/44000 | Loss: 3.0954 | LR: 4.28e-04 | 1371.1ms/step | 14.6 samples/s
742
+ Step 11460/44000 | Loss: 3.0567 | LR: 4.28e-04 | 1372.9ms/step | 14.6 samples/s
743
+ Step 11470/44000 | Loss: 3.0677 | LR: 4.28e-04 | 1372.3ms/step | 14.6 samples/s
744
+ Step 11480/44000 | Loss: 3.0649 | LR: 4.28e-04 | 1372.0ms/step | 14.6 samples/s
745
+ Step 11490/44000 | Loss: 3.0850 | LR: 4.28e-04 | 1370.8ms/step | 14.6 samples/s
746
+ Step 11500/44000 | Loss: 3.0858 | LR: 4.28e-04 | 1373.6ms/step | 14.6 samples/s
747
+ Step 11510/44000 | Loss: 3.0878 | LR: 4.27e-04 | 1371.5ms/step | 14.6 samples/s
748
+ Step 11520/44000 | Loss: 3.0770 | LR: 4.27e-04 | 1372.8ms/step | 14.6 samples/s
749
+ Step 11530/44000 | Loss: 3.0845 | LR: 4.27e-04 | 1371.8ms/step | 14.6 samples/s
750
+ Step 11540/44000 | Loss: 3.0670 | LR: 4.27e-04 | 1372.8ms/step | 14.6 samples/s
751
+ Step 11550/44000 | Loss: 3.0670 | LR: 4.27e-04 | 1372.8ms/step | 14.6 samples/s
752
+ Step 11560/44000 | Loss: 3.1072 | LR: 4.27e-04 | 1372.8ms/step | 14.6 samples/s
753
+ Step 11570/44000 | Loss: 3.0721 | LR: 4.27e-04 | 1371.6ms/step | 14.6 samples/s
754
+ Step 11580/44000 | Loss: 3.1176 | LR: 4.27e-04 | 1371.9ms/step | 14.6 samples/s
755
+ Step 11590/44000 | Loss: 3.0927 | LR: 4.26e-04 | 1369.5ms/step | 14.6 samples/s
756
+ Step 11600/44000 | Loss: 3.0796 | LR: 4.26e-04 | 1371.5ms/step | 14.6 samples/s
757
+ Step 11610/44000 | Loss: 3.1038 | LR: 4.26e-04 | 1371.9ms/step | 14.6 samples/s
758
+ Step 11620/44000 | Loss: 3.0931 | LR: 4.26e-04 | 1371.6ms/step | 14.6 samples/s
759
+ Step 11630/44000 | Loss: 3.1091 | LR: 4.26e-04 | 1368.8ms/step | 14.6 samples/s
760
+ Step 11640/44000 | Loss: 3.1026 | LR: 4.26e-04 | 1369.8ms/step | 14.6 samples/s
761
+ Step 11650/44000 | Loss: 3.0679 | LR: 4.26e-04 | 1371.0ms/step | 14.6 samples/s
762
+ Step 11660/44000 | Loss: 3.1061 | LR: 4.26e-04 | 1371.2ms/step | 14.6 samples/s
763
+ Step 11670/44000 | Loss: 3.0922 | LR: 4.25e-04 | 1368.9ms/step | 14.6 samples/s
764
+ Step 11680/44000 | Loss: 3.1060 | LR: 4.25e-04 | 1369.6ms/step | 14.6 samples/s
765
+ Step 11690/44000 | Loss: 3.0689 | LR: 4.25e-04 | 1371.7ms/step | 14.6 samples/s
766
+ Step 11700/44000 | Loss: 3.0935 | LR: 4.25e-04 | 1370.7ms/step | 14.6 samples/s
767
+ Step 11710/44000 | Loss: 3.0586 | LR: 4.25e-04 | 1368.6ms/step | 14.6 samples/s
768
+ Step 11720/44000 | Loss: 3.1061 | LR: 4.25e-04 | 1369.1ms/step | 14.6 samples/s
769
+ Step 11730/44000 | Loss: 3.1133 | LR: 4.25e-04 | 1367.2ms/step | 14.6 samples/s
770
+ Step 11740/44000 | Loss: 3.0753 | LR: 4.24e-04 | 1368.9ms/step | 14.6 samples/s
771
+ Step 11750/44000 | Loss: 3.0914 | LR: 4.24e-04 | 1370.3ms/step | 14.6 samples/s
772
+ Step 11760/44000 | Loss: 3.0753 | LR: 4.24e-04 | 1372.1ms/step | 14.6 samples/s
773
+ Step 11770/44000 | Loss: 3.0982 | LR: 4.24e-04 | 1385.4ms/step | 14.4 samples/s
774
+ Step 11780/44000 | Loss: 3.0652 | LR: 4.24e-04 | 1378.0ms/step | 14.5 samples/s
775
+ Step 11790/44000 | Loss: 3.0786 | LR: 4.24e-04 | 1378.8ms/step | 14.5 samples/s
776
+ Step 11800/44000 | Loss: 3.0950 | LR: 4.24e-04 | 1379.5ms/step | 14.5 samples/s
777
+ Step 11810/44000 | Loss: 3.0756 | LR: 4.24e-04 | 1378.9ms/step | 14.5 samples/s
778
+ Step 11820/44000 | Loss: 3.0947 | LR: 4.23e-04 | 1378.8ms/step | 14.5 samples/s
779
+ Step 11830/44000 | Loss: 3.1080 | LR: 4.23e-04 | 1375.8ms/step | 14.5 samples/s
780
+ Step 11840/44000 | Loss: 3.1034 | LR: 4.23e-04 | 1375.8ms/step | 14.5 samples/s
781
+ Step 11850/44000 | Loss: 3.0675 | LR: 4.23e-04 | 1376.8ms/step | 14.5 samples/s
782
+ Step 11860/44000 | Loss: 3.0638 | LR: 4.23e-04 | 1377.0ms/step | 14.5 samples/s
783
+ Step 11870/44000 | Loss: 3.0803 | LR: 4.23e-04 | 1377.6ms/step | 14.5 samples/s
784
+ Step 11880/44000 | Loss: 3.0804 | LR: 4.23e-04 | 1376.7ms/step | 14.5 samples/s
785
+ Step 11890/44000 | Loss: 3.0736 | LR: 4.23e-04 | 1376.1ms/step | 14.5 samples/s
786
+ Step 11900/44000 | Loss: 3.0771 | LR: 4.22e-04 | 1376.4ms/step | 14.5 samples/s
787
+ Step 11910/44000 | Loss: 3.0783 | LR: 4.22e-04 | 1374.0ms/step | 14.6 samples/s
788
+ Step 11920/44000 | Loss: 3.0904 | LR: 4.22e-04 | 1376.2ms/step | 14.5 samples/s
789
+ Step 11930/44000 | Loss: 3.0737 | LR: 4.22e-04 | 1375.1ms/step | 14.5 samples/s
790
+ Step 11940/44000 | Loss: 3.0770 | LR: 4.22e-04 | 1376.1ms/step | 14.5 samples/s
791
+ Step 11950/44000 | Loss: 3.0838 | LR: 4.22e-04 | 1376.5ms/step | 14.5 samples/s
792
+ Step 11960/44000 | Loss: 3.0956 | LR: 4.22e-04 | 1376.8ms/step | 14.5 samples/s
793
+ Step 11970/44000 | Loss: 3.0730 | LR: 4.21e-04 | 1377.4ms/step | 14.5 samples/s
794
+ Step 11980/44000 | Loss: 3.1026 | LR: 4.21e-04 | 1379.1ms/step | 14.5 samples/s
795
+ Step 11990/44000 | Loss: 3.0935 | LR: 4.21e-04 | 1378.5ms/step | 14.5 samples/s
796
+ Step 12000/44000 | Loss: 3.0629 | LR: 4.21e-04 | 1376.6ms/step | 14.5 samples/s
797
+ Validation at step 12000: Loss = 3.0858, Perplexity = 21.89
798
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858/checkpoint-12000
799
+ Step 12010/44000 | Loss: 3.0502 | LR: 4.21e-04 | 3865.1ms/step | 5.2 samples/s
800
+ Step 12020/44000 | Loss: 3.0628 | LR: 4.21e-04 | 1373.0ms/step | 14.6 samples/s
801
+ Step 12030/44000 | Loss: 3.0656 | LR: 4.21e-04 | 1373.6ms/step | 14.6 samples/s
802
+ Step 12040/44000 | Loss: 3.0708 | LR: 4.21e-04 | 1375.7ms/step | 14.5 samples/s
803
+ Step 12050/44000 | Loss: 3.0959 | LR: 4.20e-04 | 1376.5ms/step | 14.5 samples/s
804
+ Step 12060/44000 | Loss: 3.0834 | LR: 4.20e-04 | 1375.2ms/step | 14.5 samples/s
805
+ Step 12070/44000 | Loss: 3.0916 | LR: 4.20e-04 | 1375.2ms/step | 14.5 samples/s
806
+ Step 12080/44000 | Loss: 3.0698 | LR: 4.20e-04 | 1376.2ms/step | 14.5 samples/s
807
+ Step 12090/44000 | Loss: 3.0733 | LR: 4.20e-04 | 1375.4ms/step | 14.5 samples/s
808
+ Step 12100/44000 | Loss: 3.0801 | LR: 4.20e-04 | 1377.5ms/step | 14.5 samples/s
809
+ Step 12110/44000 | Loss: 3.1052 | LR: 4.20e-04 | 1377.9ms/step | 14.5 samples/s
810
+ Step 12120/44000 | Loss: 3.0691 | LR: 4.19e-04 | 1377.0ms/step | 14.5 samples/s
811
+ Step 12130/44000 | Loss: 3.0864 | LR: 4.19e-04 | 1374.7ms/step | 14.5 samples/s
812
+ Step 12140/44000 | Loss: 3.1027 | LR: 4.19e-04 | 1377.3ms/step | 14.5 samples/s
813
+ Step 12150/44000 | Loss: 3.1183 | LR: 4.19e-04 | 1377.8ms/step | 14.5 samples/s
814
+ Step 12160/44000 | Loss: 3.0630 | LR: 4.19e-04 | 1376.4ms/step | 14.5 samples/s
815
+ Step 12170/44000 | Loss: 3.0823 | LR: 4.19e-04 | 1378.0ms/step | 14.5 samples/s
816
+ Step 12180/44000 | Loss: 3.0791 | LR: 4.19e-04 | 1378.6ms/step | 14.5 samples/s
817
+ Step 12190/44000 | Loss: 3.0595 | LR: 4.19e-04 | 1379.6ms/step | 14.5 samples/s
818
+ Step 12200/44000 | Loss: 3.0538 | LR: 4.18e-04 | 1380.3ms/step | 14.5 samples/s
819
+ Step 12210/44000 | Loss: 3.0852 | LR: 4.18e-04 | 1380.1ms/step | 14.5 samples/s
820
+ Step 12220/44000 | Loss: 3.0739 | LR: 4.18e-04 | 1378.5ms/step | 14.5 samples/s
821
+ Step 12230/44000 | Loss: 3.0589 | LR: 4.18e-04 | 1378.1ms/step | 14.5 samples/s
822
+ Step 12240/44000 | Loss: 3.0904 | LR: 4.18e-04 | 1377.2ms/step | 14.5 samples/s
823
+ Step 12250/44000 | Loss: 3.0750 | LR: 4.18e-04 | 1379.4ms/step | 14.5 samples/s
824
+ Step 12260/44000 | Loss: 3.0823 | LR: 4.18e-04 | 1379.2ms/step | 14.5 samples/s
825
+ Step 12270/44000 | Loss: 3.0495 | LR: 4.17e-04 | 1379.2ms/step | 14.5 samples/s
826
+ Step 12280/44000 | Loss: 3.0910 | LR: 4.17e-04 | 1381.0ms/step | 14.5 samples/s
827
+ Step 12290/44000 | Loss: 3.0735 | LR: 4.17e-04 | 1380.9ms/step | 14.5 samples/s
828
+ Step 12300/44000 | Loss: 3.0784 | LR: 4.17e-04 | 1381.5ms/step | 14.5 samples/s
829
+ Step 12310/44000 | Loss: 3.0779 | LR: 4.17e-04 | 1380.0ms/step | 14.5 samples/s
830
+ Step 12320/44000 | Loss: 3.0892 | LR: 4.17e-04 | 1380.1ms/step | 14.5 samples/s
831
+ Step 12330/44000 | Loss: 3.0582 | LR: 4.17e-04 | 1380.2ms/step | 14.5 samples/s
832
+ Step 12340/44000 | Loss: 3.0768 | LR: 4.17e-04 | 1381.4ms/step | 14.5 samples/s
833
+ Step 12350/44000 | Loss: 3.0657 | LR: 4.16e-04 | 1380.6ms/step | 14.5 samples/s
834
+ Step 12360/44000 | Loss: 3.0636 | LR: 4.16e-04 | 1379.8ms/step | 14.5 samples/s
835
+ Step 12370/44000 | Loss: 3.0587 | LR: 4.16e-04 | 1378.2ms/step | 14.5 samples/s
836
+ Step 12380/44000 | Loss: 3.1030 | LR: 4.16e-04 | 1378.3ms/step | 14.5 samples/s
837
+ Step 12390/44000 | Loss: 3.0694 | LR: 4.16e-04 | 1379.6ms/step | 14.5 samples/s
838
+ Step 12400/44000 | Loss: 3.0639 | LR: 4.16e-04 | 1380.0ms/step | 14.5 samples/s
839
+ Step 12410/44000 | Loss: 3.0597 | LR: 4.16e-04 | 1381.7ms/step | 14.5 samples/s
840
+ Step 12420/44000 | Loss: 3.0780 | LR: 4.15e-04 | 1381.8ms/step | 14.5 samples/s
841
+ Step 12430/44000 | Loss: 3.0437 | LR: 4.15e-04 | 1382.4ms/step | 14.5 samples/s
842
+ Step 12440/44000 | Loss: 3.0946 | LR: 4.15e-04 | 1378.8ms/step | 14.5 samples/s
843
+ Step 12450/44000 | Loss: 3.0678 | LR: 4.15e-04 | 1380.7ms/step | 14.5 samples/s
844
+ Step 12460/44000 | Loss: 3.0680 | LR: 4.15e-04 | 1378.4ms/step | 14.5 samples/s
845
+ Step 12470/44000 | Loss: 3.0530 | LR: 4.15e-04 | 1380.2ms/step | 14.5 samples/s
846
+ Step 12480/44000 | Loss: 3.0718 | LR: 4.15e-04 | 1380.7ms/step | 14.5 samples/s
847
+ Step 12490/44000 | Loss: 3.0605 | LR: 4.14e-04 | 1381.9ms/step | 14.5 samples/s
848
+ Step 12500/44000 | Loss: 3.0704 | LR: 4.14e-04 | 1379.6ms/step | 14.5 samples/s
849
+ Step 12510/44000 | Loss: 3.0799 | LR: 4.14e-04 | 1379.7ms/step | 14.5 samples/s
850
+ Step 12520/44000 | Loss: 3.1041 | LR: 4.14e-04 | 1382.1ms/step | 14.5 samples/s
851
+ Step 12530/44000 | Loss: 3.0844 | LR: 4.14e-04 | 1383.3ms/step | 14.5 samples/s
852
+ Step 12540/44000 | Loss: 3.0609 | LR: 4.14e-04 | 1381.6ms/step | 14.5 samples/s
853
+ Step 12550/44000 | Loss: 3.0604 | LR: 4.14e-04 | 1381.1ms/step | 14.5 samples/s
854
+ Step 12560/44000 | Loss: 3.0914 | LR: 4.14e-04 | 1381.9ms/step | 14.5 samples/s
855
+ Step 12570/44000 | Loss: 3.0948 | LR: 4.13e-04 | 1378.3ms/step | 14.5 samples/s
856
+ Step 12580/44000 | Loss: 3.0393 | LR: 4.13e-04 | 1377.2ms/step | 14.5 samples/s
857
+ Step 12590/44000 | Loss: 3.0477 | LR: 4.13e-04 | 1377.8ms/step | 14.5 samples/s
858
+ Step 12600/44000 | Loss: 3.0638 | LR: 4.13e-04 | 1376.4ms/step | 14.5 samples/s
859
+ Step 12610/44000 | Loss: 3.0745 | LR: 4.13e-04 | 1378.1ms/step | 14.5 samples/s
860
+ Step 12620/44000 | Loss: 3.0927 | LR: 4.13e-04 | 1379.3ms/step | 14.5 samples/s
861
+ Step 12630/44000 | Loss: 3.0446 | LR: 4.13e-04 | 1375.8ms/step | 14.5 samples/s
862
+ Step 12640/44000 | Loss: 3.0875 | LR: 4.12e-04 | 1377.1ms/step | 14.5 samples/s
863
+ Step 12650/44000 | Loss: 3.0934 | LR: 4.12e-04 | 1377.2ms/step | 14.5 samples/s
864
+ Step 12660/44000 | Loss: 3.0546 | LR: 4.12e-04 | 1377.0ms/step | 14.5 samples/s
865
+ Step 12670/44000 | Loss: 3.0460 | LR: 4.12e-04 | 1379.9ms/step | 14.5 samples/s
866
+ Step 12680/44000 | Loss: 3.0596 | LR: 4.12e-04 | 1379.9ms/step | 14.5 samples/s
867
+ Step 12690/44000 | Loss: 3.0822 | LR: 4.12e-04 | 1379.1ms/step | 14.5 samples/s
868
+ Step 12700/44000 | Loss: 3.0389 | LR: 4.12e-04 | 1376.7ms/step | 14.5 samples/s
869
+ Step 12710/44000 | Loss: 3.0955 | LR: 4.11e-04 | 1379.4ms/step | 14.5 samples/s
870
+ Step 12720/44000 | Loss: 3.0794 | LR: 4.11e-04 | 1376.8ms/step | 14.5 samples/s
871
+ Step 12730/44000 | Loss: 3.1128 | LR: 4.11e-04 | 1377.4ms/step | 14.5 samples/s
872
+ Step 12740/44000 | Loss: 3.0822 | LR: 4.11e-04 | 1376.8ms/step | 14.5 samples/s
873
+ Step 12750/44000 | Loss: 3.0868 | LR: 4.11e-04 | 1377.3ms/step | 14.5 samples/s
874
+ Step 12760/44000 | Loss: 3.0433 | LR: 4.11e-04 | 1374.4ms/step | 14.6 samples/s
875
+ Step 12770/44000 | Loss: 3.0865 | LR: 4.11e-04 | 1377.3ms/step | 14.5 samples/s
876
+ Step 12780/44000 | Loss: 3.0887 | LR: 4.10e-04 | 1377.0ms/step | 14.5 samples/s
877
+ Step 12790/44000 | Loss: 3.0771 | LR: 4.10e-04 | 1377.0ms/step | 14.5 samples/s
878
+ Step 12800/44000 | Loss: 3.0649 | LR: 4.10e-04 | 1377.0ms/step | 14.5 samples/s
879
+ Step 12810/44000 | Loss: 3.0678 | LR: 4.10e-04 | 1378.6ms/step | 14.5 samples/s
880
+ Step 12820/44000 | Loss: 3.0428 | LR: 4.10e-04 | 1379.5ms/step | 14.5 samples/s
881
+ Step 12830/44000 | Loss: 3.0688 | LR: 4.10e-04 | 1378.0ms/step | 14.5 samples/s
882
+ Step 12840/44000 | Loss: 3.0678 | LR: 4.10e-04 | 1377.4ms/step | 14.5 samples/s
883
+ Step 12850/44000 | Loss: 3.0528 | LR: 4.09e-04 | 1378.7ms/step | 14.5 samples/s
884
+ Step 12860/44000 | Loss: 3.0376 | LR: 4.09e-04 | 1379.2ms/step | 14.5 samples/s
885
+ Step 12870/44000 | Loss: 3.0750 | LR: 4.09e-04 | 1377.8ms/step | 14.5 samples/s
886
+ Step 12880/44000 | Loss: 3.0496 | LR: 4.09e-04 | 1380.9ms/step | 14.5 samples/s
887
+ Step 12890/44000 | Loss: 3.0862 | LR: 4.09e-04 | 1378.0ms/step | 14.5 samples/s
888
+ Step 12900/44000 | Loss: 3.0692 | LR: 4.09e-04 | 1374.7ms/step | 14.5 samples/s
889
+ Step 12910/44000 | Loss: 3.0758 | LR: 4.09e-04 | 1377.8ms/step | 14.5 samples/s
890
+ Step 12920/44000 | Loss: 3.0891 | LR: 4.09e-04 | 1374.6ms/step | 14.5 samples/s
891
+ Step 12930/44000 | Loss: 3.0639 | LR: 4.08e-04 | 1375.2ms/step | 14.5 samples/s
892
+ Step 12940/44000 | Loss: 3.0760 | LR: 4.08e-04 | 1373.8ms/step | 14.6 samples/s
893
+ Step 12950/44000 | Loss: 3.0796 | LR: 4.08e-04 | 1374.3ms/step | 14.6 samples/s
894
+ Step 12960/44000 | Loss: 3.0763 | LR: 4.08e-04 | 1374.4ms/step | 14.6 samples/s
895
+ Step 12970/44000 | Loss: 3.0734 | LR: 4.08e-04 | 1373.6ms/step | 14.6 samples/s
896
+ Step 12980/44000 | Loss: 3.0701 | LR: 4.08e-04 | 1373.7ms/step | 14.6 samples/s
897
+ Step 12990/44000 | Loss: 3.0790 | LR: 4.08e-04 | 1375.3ms/step | 14.5 samples/s
898
+ Step 13000/44000 | Loss: 3.0430 | LR: 4.07e-04 | 1374.8ms/step | 14.5 samples/s
899
+ Validation at step 13000: Loss = 3.0696, Perplexity = 21.53
900
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858/checkpoint-13000
901
+ Step 13010/44000 | Loss: 3.0653 | LR: 4.07e-04 | 3847.7ms/step | 5.2 samples/s
902
+ Step 13020/44000 | Loss: 3.0749 | LR: 4.07e-04 | 1371.7ms/step | 14.6 samples/s
903
+ Step 13030/44000 | Loss: 3.0467 | LR: 4.07e-04 | 1372.5ms/step | 14.6 samples/s
904
+ Step 13040/44000 | Loss: 3.0705 | LR: 4.07e-04 | 1372.4ms/step | 14.6 samples/s
905
+ Step 13050/44000 | Loss: 3.0500 | LR: 4.07e-04 | 1372.9ms/step | 14.6 samples/s
906
+ Step 13060/44000 | Loss: 3.0482 | LR: 4.07e-04 | 1374.0ms/step | 14.6 samples/s
907
+ Step 13070/44000 | Loss: 3.0519 | LR: 4.06e-04 | 1373.2ms/step | 14.6 samples/s
908
+ Step 13080/44000 | Loss: 3.0577 | LR: 4.06e-04 | 1373.5ms/step | 14.6 samples/s
909
+ Step 13090/44000 | Loss: 3.0592 | LR: 4.06e-04 | 1370.1ms/step | 14.6 samples/s
910
+ Step 13100/44000 | Loss: 3.0700 | LR: 4.06e-04 | 1367.2ms/step | 14.6 samples/s
911
+ Step 13110/44000 | Loss: 3.0407 | LR: 4.06e-04 | 1371.5ms/step | 14.6 samples/s
912
+ Step 13120/44000 | Loss: 3.0724 | LR: 4.06e-04 | 1375.2ms/step | 14.5 samples/s
913
+ Step 13130/44000 | Loss: 3.0795 | LR: 4.06e-04 | 1374.2ms/step | 14.6 samples/s
914
+ Step 13140/44000 | Loss: 3.0778 | LR: 4.05e-04 | 1373.9ms/step | 14.6 samples/s
915
+ Step 13150/44000 | Loss: 3.0502 | LR: 4.05e-04 | 1371.0ms/step | 14.6 samples/s
916
+ Step 13160/44000 | Loss: 3.0820 | LR: 4.05e-04 | 1367.0ms/step | 14.6 samples/s
917
+ Step 13170/44000 | Loss: 3.0622 | LR: 4.05e-04 | 1363.8ms/step | 14.7 samples/s
918
+ Step 13180/44000 | Loss: 3.0623 | LR: 4.05e-04 | 1367.4ms/step | 14.6 samples/s
epfl_final/train_logs/ropectl_a1p0_h100_resume/seed45.20260820T011048Z.log ADDED
@@ -0,0 +1,915 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Auto-resuming from latest checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735/checkpoint-6000
2
+ Resuming: Using existing output directory /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735
3
+ Running config:
4
+ {
5
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
6
+ "model_dtype": "bfloat16",
7
+ "init_from_scratch": true,
8
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
9
+ "val_path": null,
10
+ "max_seq_length": 8192,
11
+ "num_workers": 8,
12
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735",
13
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735/checkpoint-6000",
14
+ "force_new_run": false,
15
+ "num_epochs": 1,
16
+ "max_steps": 44000,
17
+ "per_device_batch_size": 2,
18
+ "gradient_accumulation_steps": 10,
19
+ "learning_rate": 0.0005,
20
+ "weight_decay": 0.01,
21
+ "warmup_steps": 750,
22
+ "lr_scheduler": "cosine",
23
+ "max_grad_norm": 1.0,
24
+ "save_steps": 1000,
25
+ "eval_steps": 1000,
26
+ "eval_batches": 50,
27
+ "logging_steps": 10,
28
+ "seed": 45,
29
+ "verbose": true,
30
+ "checkpoint_activations": false,
31
+ "strategy": "ddp",
32
+ "mixed_precision": "bf16",
33
+ "wandb_enabled": true,
34
+ "wandb_project": "stochastic_sparsity",
35
+ "wandb_name": "l8k365m_dense_ropectl_a1p0_seed45",
36
+ "wandb_tags": [
37
+ "qwen3",
38
+ "365m",
39
+ "8k",
40
+ "dense",
41
+ "rope-control"
42
+ ],
43
+ "hidden_size": 768,
44
+ "intermediate_size": 3072,
45
+ "num_attention_heads": 6,
46
+ "num_key_value_heads": 2,
47
+ "num_hidden_layers": 28,
48
+ "head_dim": null,
49
+ "attention_dropout": 0.0,
50
+ "rope_train_position_scale_set": [
51
+ 1.0,
52
+ 0.9896339488176222,
53
+ 0.9692159692159692,
54
+ 0.9420289855072463,
55
+ 0.8730158730158729
56
+ ],
57
+ "attention_type": "sparse_stochastic",
58
+ "sparse_block_size": 128,
59
+ "sparse_num_random_blocks": 31,
60
+ "sparse_mask_pool_size": 1,
61
+ "sparse_val_dense_compare": false,
62
+ "sparse_layer_indices": null,
63
+ "sparse_mask_cache_dir": null,
64
+ "sparse_mask_seed": 0,
65
+ "sparse_mixed_nrb_set": null,
66
+ "dense_mode": true,
67
+ "dense_step_intervals": null
68
+ }
69
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
70
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735/checkpoint-6000
71
+ Checkpoint activations: False
72
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
73
+
74
+
75
+ ================================================================================
76
+ MODEL ARCHITECTURE
77
+ ================================================================================
78
+ Model config:
79
+ vocab_size: 151936
80
+ hidden_size: 768
81
+ num_hidden_layers: 28
82
+ num_attention_heads: 6
83
+ num_key_value_heads: 2
84
+ head_dim: 128
85
+ intermediate_size: 3072
86
+ max_position_embeddings: 32768
87
+ rope_theta: 1000000
88
+ rope_train_position_scale_set: [1.0, 0.9896339488176222, 0.9692159692159692, 0.9420289855072463, 0.8730158730158729]
89
+ attention_type: sparse_stochastic
90
+ sparse_block_size: 128
91
+ sparse_num_random_blocks: 31
92
+ sparse_mask_pool_size: 1
93
+ sparse_max_seq_length: 8192
94
+
95
+ Parameter count:
96
+ Total parameters: 358,958,848
97
+ Trainable parameters: 358,958,848
98
+ Non-trainable parameters: 0
99
+ Model size (BF16): 0.72 GB
100
+
101
+ --------------------------------------------------------------------------------
102
+ LAYER-BY-LAYER SUMMARY
103
+ --------------------------------------------------------------------------------
104
+ ================================================================================================================================================================
105
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
106
+ ================================================================================================================================================================
107
+ Qwen3ForCausalLM [2, 8192] -- -- True
108
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
109
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
110
+ │ └─ModuleList: 2-2 -- -- -- True
111
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
132
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
133
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
134
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
135
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
136
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
137
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
138
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
139
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
140
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
141
+ ================================================================================================================================================================
142
+ Total params: 475,645,696
143
+ Trainable params: 475,645,696
144
+ Non-trainable params: 0
145
+ Total mult-adds (Units.MEGABYTES): 951.29
146
+ ================================================================================================================================================================
147
+ Input size (MB): 0.13
148
+ Forward/backward pass size (MB): 31197.23
149
+ Params size (MB): 951.29
150
+ Estimated Total Size (MB): 32148.66
151
+ ================================================================================================================================================================
152
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
153
+ ================================================================================
154
+
155
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
156
+ Using pre-tokenized dataset (no runtime tokenization)
157
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
158
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
159
+ Dataset split: 3027764 train, 159356 validation
160
+ Training examples (packed, train split): 3027764
161
+ Validation examples (packed, val split): 159356
162
+ Per-device batch size: 2
163
+ World size (GPUs): 1
164
+ Gradient accumulation steps: 10
165
+ Effective batch size (seqs/step): 20
166
+ Sequence length: 8192
167
+ Tokens per optimizer step: 163,840
168
+ Micro-batches per epoch (per rank): 1513882
169
+ Optimizer steps per epoch: 151388
170
+ Total optimizer steps: 44000 (capped by max_steps; 0.29 epochs)
171
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735/checkpoint-6000...
172
+ Preserving checkpoint LR state for exact resume: current=[0.0004820408177936365], schedule_base=[0.0005]
173
+ Resumed from checkpoint at step 6000 (epoch 0, micro_step 60000)
174
+ Resuming training from step 6000
175
+ wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from WANDB_API_KEY.
176
+ wandb: Currently logged in as: ruben-navasardyan4869 (ruben-navasardyan4869-epfl) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
177
+ wandb: Tracking run with wandb version 0.26.1
178
+ wandb: Run data is saved locally in /mloscratch/homes/navasard/sparse/SparseAtt/wandb/run-20260820_011125-xfnxdn24
179
+ wandb: Run `wandb offline` to turn off syncing.
180
+ wandb: Syncing run l8k365m_dense_ropectl_a1p0_seed45
181
+ wandb: ⭐️ View project at https://wandb.ai/ruben-navasardyan4869-epfl/stochastic_sparsity
182
+ wandb: 🚀 View run at https://wandb.ai/ruben-navasardyan4869-epfl/stochastic_sparsity/runs/xfnxdn24
183
+ Starting training...
184
+ Validation at step 6000: Loss = 3.2430, Perplexity = 25.61
185
+ Switching attention mode for step 6001: dense (dense step interval)
186
+ Step 6010/44000 | Loss: 3.2445 | LR: 4.82e-04 | 15921.6ms/step | 1.3 samples/s
187
+ Step 6020/44000 | Loss: 3.2320 | LR: 4.82e-04 | 1363.3ms/step | 14.7 samples/s
188
+ Step 6030/44000 | Loss: 3.2292 | LR: 4.82e-04 | 1365.0ms/step | 14.7 samples/s
189
+ Step 6040/44000 | Loss: 3.2477 | LR: 4.82e-04 | 1366.2ms/step | 14.6 samples/s
190
+ Step 6050/44000 | Loss: 3.2273 | LR: 4.82e-04 | 1368.1ms/step | 14.6 samples/s
191
+ Step 6060/44000 | Loss: 3.2241 | LR: 4.82e-04 | 1370.5ms/step | 14.6 samples/s
192
+ Step 6070/44000 | Loss: 3.2377 | LR: 4.82e-04 | 1373.3ms/step | 14.6 samples/s
193
+ Step 6080/44000 | Loss: 3.2289 | LR: 4.81e-04 | 1373.2ms/step | 14.6 samples/s
194
+ Step 6090/44000 | Loss: 3.1882 | LR: 4.81e-04 | 1371.1ms/step | 14.6 samples/s
195
+ Step 6100/44000 | Loss: 3.2602 | LR: 4.81e-04 | 1370.5ms/step | 14.6 samples/s
196
+ Step 6110/44000 | Loss: 3.2077 | LR: 4.81e-04 | 1369.7ms/step | 14.6 samples/s
197
+ Step 6120/44000 | Loss: 3.2222 | LR: 4.81e-04 | 1371.2ms/step | 14.6 samples/s
198
+ Step 6130/44000 | Loss: 3.2656 | LR: 4.81e-04 | 1370.4ms/step | 14.6 samples/s
199
+ Step 6140/44000 | Loss: 3.2513 | LR: 4.81e-04 | 1373.1ms/step | 14.6 samples/s
200
+ Step 6150/44000 | Loss: 3.2022 | LR: 4.81e-04 | 1373.7ms/step | 14.6 samples/s
201
+ Step 6160/44000 | Loss: 3.2491 | LR: 4.81e-04 | 1373.4ms/step | 14.6 samples/s
202
+ Step 6170/44000 | Loss: 3.2120 | LR: 4.81e-04 | 1373.2ms/step | 14.6 samples/s
203
+ Step 6180/44000 | Loss: 3.2312 | LR: 4.81e-04 | 1373.3ms/step | 14.6 samples/s
204
+ Step 6190/44000 | Loss: 3.2278 | LR: 4.81e-04 | 1372.8ms/step | 14.6 samples/s
205
+ Step 6200/44000 | Loss: 3.2231 | LR: 4.81e-04 | 1375.2ms/step | 14.5 samples/s
206
+ Step 6210/44000 | Loss: 3.2128 | LR: 4.81e-04 | 1372.8ms/step | 14.6 samples/s
207
+ Step 6220/44000 | Loss: 3.1997 | LR: 4.81e-04 | 1373.8ms/step | 14.6 samples/s
208
+ Step 6230/44000 | Loss: 3.2366 | LR: 4.80e-04 | 1372.1ms/step | 14.6 samples/s
209
+ Step 6240/44000 | Loss: 3.2222 | LR: 4.80e-04 | 1372.8ms/step | 14.6 samples/s
210
+ Step 6250/44000 | Loss: 3.2532 | LR: 4.80e-04 | 1373.5ms/step | 14.6 samples/s
211
+ Step 6260/44000 | Loss: 3.1944 | LR: 4.80e-04 | 1373.7ms/step | 14.6 samples/s
212
+ Step 6270/44000 | Loss: 3.2453 | LR: 4.80e-04 | 1374.5ms/step | 14.6 samples/s
213
+ Step 6280/44000 | Loss: 3.2447 | LR: 4.80e-04 | 1373.9ms/step | 14.6 samples/s
214
+ Step 6290/44000 | Loss: 3.2012 | LR: 4.80e-04 | 1371.8ms/step | 14.6 samples/s
215
+ Step 6300/44000 | Loss: 3.1958 | LR: 4.80e-04 | 1373.1ms/step | 14.6 samples/s
216
+ Step 6310/44000 | Loss: 3.2182 | LR: 4.80e-04 | 1372.9ms/step | 14.6 samples/s
217
+ Step 6320/44000 | Loss: 3.2276 | LR: 4.80e-04 | 1371.1ms/step | 14.6 samples/s
218
+ Step 6330/44000 | Loss: 3.2425 | LR: 4.80e-04 | 1370.4ms/step | 14.6 samples/s
219
+ Step 6340/44000 | Loss: 3.2281 | LR: 4.80e-04 | 1370.4ms/step | 14.6 samples/s
220
+ Step 6350/44000 | Loss: 3.2387 | LR: 4.80e-04 | 1374.1ms/step | 14.6 samples/s
221
+ Step 6360/44000 | Loss: 3.1916 | LR: 4.80e-04 | 1373.8ms/step | 14.6 samples/s
222
+ Step 6370/44000 | Loss: 3.2288 | LR: 4.79e-04 | 1372.6ms/step | 14.6 samples/s
223
+ Step 6380/44000 | Loss: 3.1844 | LR: 4.79e-04 | 1373.2ms/step | 14.6 samples/s
224
+ Step 6390/44000 | Loss: 3.2133 | LR: 4.79e-04 | 1374.2ms/step | 14.6 samples/s
225
+ Step 6400/44000 | Loss: 3.1995 | LR: 4.79e-04 | 1374.8ms/step | 14.5 samples/s
226
+ Step 6410/44000 | Loss: 3.2246 | LR: 4.79e-04 | 1374.4ms/step | 14.6 samples/s
227
+ Step 6420/44000 | Loss: 3.2090 | LR: 4.79e-04 | 1372.9ms/step | 14.6 samples/s
228
+ Step 6430/44000 | Loss: 3.2159 | LR: 4.79e-04 | 1374.4ms/step | 14.6 samples/s
229
+ Step 6440/44000 | Loss: 3.2074 | LR: 4.79e-04 | 1374.1ms/step | 14.6 samples/s
230
+ Step 6450/44000 | Loss: 3.1822 | LR: 4.79e-04 | 1376.1ms/step | 14.5 samples/s
231
+ Step 6460/44000 | Loss: 3.2137 | LR: 4.79e-04 | 1373.7ms/step | 14.6 samples/s
232
+ Step 6470/44000 | Loss: 3.2443 | LR: 4.79e-04 | 1372.7ms/step | 14.6 samples/s
233
+ Step 6480/44000 | Loss: 3.2253 | LR: 4.79e-04 | 1374.7ms/step | 14.5 samples/s
234
+ Step 6490/44000 | Loss: 3.2224 | LR: 4.79e-04 | 1373.9ms/step | 14.6 samples/s
235
+ Step 6500/44000 | Loss: 3.2266 | LR: 4.79e-04 | 1374.1ms/step | 14.6 samples/s
236
+ Step 6510/44000 | Loss: 3.2032 | LR: 4.78e-04 | 1373.3ms/step | 14.6 samples/s
237
+ Step 6520/44000 | Loss: 3.2297 | LR: 4.78e-04 | 1375.0ms/step | 14.5 samples/s
238
+ Step 6530/44000 | Loss: 3.1851 | LR: 4.78e-04 | 1371.5ms/step | 14.6 samples/s
239
+ Step 6540/44000 | Loss: 3.2026 | LR: 4.78e-04 | 1374.1ms/step | 14.6 samples/s
240
+ Step 6550/44000 | Loss: 3.2072 | LR: 4.78e-04 | 1371.2ms/step | 14.6 samples/s
241
+ Step 6560/44000 | Loss: 3.2505 | LR: 4.78e-04 | 1372.1ms/step | 14.6 samples/s
242
+ Step 6570/44000 | Loss: 3.2384 | LR: 4.78e-04 | 1372.8ms/step | 14.6 samples/s
243
+ Step 6580/44000 | Loss: 3.2137 | LR: 4.78e-04 | 1375.6ms/step | 14.5 samples/s
244
+ Step 6590/44000 | Loss: 3.1758 | LR: 4.78e-04 | 1374.9ms/step | 14.5 samples/s
245
+ Step 6600/44000 | Loss: 3.2200 | LR: 4.78e-04 | 1375.7ms/step | 14.5 samples/s
246
+ Step 6610/44000 | Loss: 3.2341 | LR: 4.78e-04 | 1375.1ms/step | 14.5 samples/s
247
+ Step 6620/44000 | Loss: 3.2203 | LR: 4.78e-04 | 1375.3ms/step | 14.5 samples/s
248
+ Step 6630/44000 | Loss: 3.1918 | LR: 4.78e-04 | 1374.5ms/step | 14.6 samples/s
249
+ Step 6640/44000 | Loss: 3.2142 | LR: 4.77e-04 | 1374.7ms/step | 14.5 samples/s
250
+ Step 6650/44000 | Loss: 3.2391 | LR: 4.77e-04 | 1375.3ms/step | 14.5 samples/s
251
+ Step 6660/44000 | Loss: 3.2148 | LR: 4.77e-04 | 1373.0ms/step | 14.6 samples/s
252
+ Step 6670/44000 | Loss: 3.1899 | LR: 4.77e-04 | 1375.0ms/step | 14.5 samples/s
253
+ Step 6680/44000 | Loss: 3.1889 | LR: 4.77e-04 | 1373.6ms/step | 14.6 samples/s
254
+ Step 6690/44000 | Loss: 3.2476 | LR: 4.77e-04 | 1373.0ms/step | 14.6 samples/s
255
+ Step 6700/44000 | Loss: 3.1795 | LR: 4.77e-04 | 1374.7ms/step | 14.5 samples/s
256
+ Step 6710/44000 | Loss: 3.2118 | LR: 4.77e-04 | 1374.2ms/step | 14.6 samples/s
257
+ Step 6720/44000 | Loss: 3.1901 | LR: 4.77e-04 | 1374.0ms/step | 14.6 samples/s
258
+ Step 6730/44000 | Loss: 3.1923 | LR: 4.77e-04 | 1375.2ms/step | 14.5 samples/s
259
+ Step 6740/44000 | Loss: 3.2191 | LR: 4.77e-04 | 1373.8ms/step | 14.6 samples/s
260
+ Step 6750/44000 | Loss: 3.1745 | LR: 4.77e-04 | 1372.6ms/step | 14.6 samples/s
261
+ Step 6760/44000 | Loss: 3.2194 | LR: 4.77e-04 | 1373.5ms/step | 14.6 samples/s
262
+ Step 6770/44000 | Loss: 3.2216 | LR: 4.76e-04 | 1373.3ms/step | 14.6 samples/s
263
+ Step 6780/44000 | Loss: 3.2298 | LR: 4.76e-04 | 1373.5ms/step | 14.6 samples/s
264
+ Step 6790/44000 | Loss: 3.2327 | LR: 4.76e-04 | 1373.2ms/step | 14.6 samples/s
265
+ Step 6800/44000 | Loss: 3.2005 | LR: 4.76e-04 | 1374.5ms/step | 14.6 samples/s
266
+ Step 6810/44000 | Loss: 3.2230 | LR: 4.76e-04 | 1372.1ms/step | 14.6 samples/s
267
+ Step 6820/44000 | Loss: 3.2070 | LR: 4.76e-04 | 1371.9ms/step | 14.6 samples/s
268
+ Step 6830/44000 | Loss: 3.2384 | LR: 4.76e-04 | 1374.1ms/step | 14.6 samples/s
269
+ Step 6840/44000 | Loss: 3.1965 | LR: 4.76e-04 | 1374.2ms/step | 14.6 samples/s
270
+ Step 6850/44000 | Loss: 3.2126 | LR: 4.76e-04 | 1374.4ms/step | 14.6 samples/s
271
+ Step 6860/44000 | Loss: 3.2079 | LR: 4.76e-04 | 1373.0ms/step | 14.6 samples/s
272
+ Step 6870/44000 | Loss: 3.2050 | LR: 4.76e-04 | 1374.5ms/step | 14.6 samples/s
273
+ Step 6880/44000 | Loss: 3.2106 | LR: 4.76e-04 | 1373.8ms/step | 14.6 samples/s
274
+ Step 6890/44000 | Loss: 3.2017 | LR: 4.76e-04 | 1370.7ms/step | 14.6 samples/s
275
+ Step 6900/44000 | Loss: 3.2290 | LR: 4.75e-04 | 1371.5ms/step | 14.6 samples/s
276
+ Step 6910/44000 | Loss: 3.1918 | LR: 4.75e-04 | 1372.2ms/step | 14.6 samples/s
277
+ Step 6920/44000 | Loss: 3.2006 | LR: 4.75e-04 | 1372.4ms/step | 14.6 samples/s
278
+ Step 6930/44000 | Loss: 3.1870 | LR: 4.75e-04 | 1373.3ms/step | 14.6 samples/s
279
+ Step 6940/44000 | Loss: 3.1844 | LR: 4.75e-04 | 1371.0ms/step | 14.6 samples/s
280
+ Step 6950/44000 | Loss: 3.2201 | LR: 4.75e-04 | 1371.1ms/step | 14.6 samples/s
281
+ Step 6960/44000 | Loss: 3.1994 | LR: 4.75e-04 | 1372.0ms/step | 14.6 samples/s
282
+ Step 6970/44000 | Loss: 3.2199 | LR: 4.75e-04 | 1373.5ms/step | 14.6 samples/s
283
+ Step 6980/44000 | Loss: 3.1966 | LR: 4.75e-04 | 1373.0ms/step | 14.6 samples/s
284
+ Step 6990/44000 | Loss: 3.2020 | LR: 4.75e-04 | 1373.2ms/step | 14.6 samples/s
285
+ Step 7000/44000 | Loss: 3.1869 | LR: 4.75e-04 | 1372.6ms/step | 14.6 samples/s
286
+ Validation at step 7000: Loss = 3.2020, Perplexity = 24.58
287
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735/checkpoint-7000
288
+ Step 7010/44000 | Loss: 3.2182 | LR: 4.75e-04 | 3855.1ms/step | 5.2 samples/s
289
+ Step 7020/44000 | Loss: 3.1986 | LR: 4.75e-04 | 1374.7ms/step | 14.5 samples/s
290
+ Step 7030/44000 | Loss: 3.1884 | LR: 4.74e-04 | 1374.2ms/step | 14.6 samples/s
291
+ Step 7040/44000 | Loss: 3.2143 | LR: 4.74e-04 | 1375.4ms/step | 14.5 samples/s
292
+ Step 7050/44000 | Loss: 3.2066 | LR: 4.74e-04 | 1374.1ms/step | 14.6 samples/s
293
+ Step 7060/44000 | Loss: 3.1997 | LR: 4.74e-04 | 1375.0ms/step | 14.5 samples/s
294
+ Step 7070/44000 | Loss: 3.2099 | LR: 4.74e-04 | 1376.1ms/step | 14.5 samples/s
295
+ Step 7080/44000 | Loss: 3.1954 | LR: 4.74e-04 | 1376.0ms/step | 14.5 samples/s
296
+ Step 7090/44000 | Loss: 3.2054 | LR: 4.74e-04 | 1375.5ms/step | 14.5 samples/s
297
+ Step 7100/44000 | Loss: 3.1935 | LR: 4.74e-04 | 1375.8ms/step | 14.5 samples/s
298
+ Step 7110/44000 | Loss: 3.1885 | LR: 4.74e-04 | 1374.7ms/step | 14.5 samples/s
299
+ Step 7120/44000 | Loss: 3.1837 | LR: 4.74e-04 | 1376.7ms/step | 14.5 samples/s
300
+ Step 7130/44000 | Loss: 3.1964 | LR: 4.74e-04 | 1376.5ms/step | 14.5 samples/s
301
+ Step 7140/44000 | Loss: 3.1631 | LR: 4.74e-04 | 1379.7ms/step | 14.5 samples/s
302
+ Step 7150/44000 | Loss: 3.2022 | LR: 4.73e-04 | 1377.4ms/step | 14.5 samples/s
303
+ Step 7160/44000 | Loss: 3.1835 | LR: 4.73e-04 | 1376.6ms/step | 14.5 samples/s
304
+ Step 7170/44000 | Loss: 3.2052 | LR: 4.73e-04 | 1375.8ms/step | 14.5 samples/s
305
+ Step 7180/44000 | Loss: 3.2003 | LR: 4.73e-04 | 1374.5ms/step | 14.6 samples/s
306
+ Step 7190/44000 | Loss: 3.1781 | LR: 4.73e-04 | 1376.2ms/step | 14.5 samples/s
307
+ Step 7200/44000 | Loss: 3.1821 | LR: 4.73e-04 | 1376.2ms/step | 14.5 samples/s
308
+ Step 7210/44000 | Loss: 3.1756 | LR: 4.73e-04 | 1375.9ms/step | 14.5 samples/s
309
+ Step 7220/44000 | Loss: 3.2084 | LR: 4.73e-04 | 1375.5ms/step | 14.5 samples/s
310
+ Step 7230/44000 | Loss: 3.1840 | LR: 4.73e-04 | 1374.1ms/step | 14.6 samples/s
311
+ Step 7240/44000 | Loss: 3.1505 | LR: 4.73e-04 | 1376.3ms/step | 14.5 samples/s
312
+ Step 7250/44000 | Loss: 3.1570 | LR: 4.73e-04 | 1377.6ms/step | 14.5 samples/s
313
+ Step 7260/44000 | Loss: 3.1761 | LR: 4.73e-04 | 1376.3ms/step | 14.5 samples/s
314
+ Step 7270/44000 | Loss: 3.1758 | LR: 4.72e-04 | 1376.2ms/step | 14.5 samples/s
315
+ Step 7280/44000 | Loss: 3.2155 | LR: 4.72e-04 | 1372.6ms/step | 14.6 samples/s
316
+ Step 7290/44000 | Loss: 3.2225 | LR: 4.72e-04 | 1367.7ms/step | 14.6 samples/s
317
+ Step 7300/44000 | Loss: 3.2045 | LR: 4.72e-04 | 1369.7ms/step | 14.6 samples/s
318
+ Step 7310/44000 | Loss: 3.1886 | LR: 4.72e-04 | 1376.8ms/step | 14.5 samples/s
319
+ Step 7320/44000 | Loss: 3.1948 | LR: 4.72e-04 | 1376.1ms/step | 14.5 samples/s
320
+ Step 7330/44000 | Loss: 3.2054 | LR: 4.72e-04 | 1372.8ms/step | 14.6 samples/s
321
+ Step 7340/44000 | Loss: 3.1975 | LR: 4.72e-04 | 1368.6ms/step | 14.6 samples/s
322
+ Step 7350/44000 | Loss: 3.2064 | LR: 4.72e-04 | 1368.0ms/step | 14.6 samples/s
323
+ Step 7360/44000 | Loss: 3.1890 | LR: 4.72e-04 | 1371.0ms/step | 14.6 samples/s
324
+ Step 7370/44000 | Loss: 3.2000 | LR: 4.72e-04 | 1374.4ms/step | 14.6 samples/s
325
+ Step 7380/44000 | Loss: 3.1468 | LR: 4.72e-04 | 1376.1ms/step | 14.5 samples/s
326
+ Step 7390/44000 | Loss: 3.2038 | LR: 4.71e-04 | 1376.4ms/step | 14.5 samples/s
327
+ Step 7400/44000 | Loss: 3.1729 | LR: 4.71e-04 | 1375.2ms/step | 14.5 samples/s
328
+ Step 7410/44000 | Loss: 3.1946 | LR: 4.71e-04 | 1369.3ms/step | 14.6 samples/s
329
+ Step 7420/44000 | Loss: 3.1607 | LR: 4.71e-04 | 1368.3ms/step | 14.6 samples/s
330
+ Step 7430/44000 | Loss: 3.1734 | LR: 4.71e-04 | 1368.5ms/step | 14.6 samples/s
331
+ Step 7440/44000 | Loss: 3.1780 | LR: 4.71e-04 | 1368.3ms/step | 14.6 samples/s
332
+ Step 7450/44000 | Loss: 3.1762 | LR: 4.71e-04 | 1367.9ms/step | 14.6 samples/s
333
+ Step 7460/44000 | Loss: 3.1928 | LR: 4.71e-04 | 1367.6ms/step | 14.6 samples/s
334
+ Step 7470/44000 | Loss: 3.1691 | LR: 4.71e-04 | 1376.8ms/step | 14.5 samples/s
335
+ Step 7480/44000 | Loss: 3.1610 | LR: 4.71e-04 | 1380.8ms/step | 14.5 samples/s
336
+ Step 7490/44000 | Loss: 3.1791 | LR: 4.71e-04 | 1378.5ms/step | 14.5 samples/s
337
+ Step 7500/44000 | Loss: 3.1840 | LR: 4.71e-04 | 1369.5ms/step | 14.6 samples/s
338
+ Step 7510/44000 | Loss: 3.1856 | LR: 4.70e-04 | 1369.8ms/step | 14.6 samples/s
339
+ Step 7520/44000 | Loss: 3.1711 | LR: 4.70e-04 | 1368.2ms/step | 14.6 samples/s
340
+ Step 7530/44000 | Loss: 3.2171 | LR: 4.70e-04 | 1368.4ms/step | 14.6 samples/s
341
+ Step 7540/44000 | Loss: 3.1577 | LR: 4.70e-04 | 1370.6ms/step | 14.6 samples/s
342
+ Step 7550/44000 | Loss: 3.1843 | LR: 4.70e-04 | 1369.6ms/step | 14.6 samples/s
343
+ Step 7560/44000 | Loss: 3.1689 | LR: 4.70e-04 | 1369.0ms/step | 14.6 samples/s
344
+ Step 7570/44000 | Loss: 3.1876 | LR: 4.70e-04 | 1368.8ms/step | 14.6 samples/s
345
+ Step 7580/44000 | Loss: 3.1437 | LR: 4.70e-04 | 1368.0ms/step | 14.6 samples/s
346
+ Step 7590/44000 | Loss: 3.1687 | LR: 4.70e-04 | 1368.9ms/step | 14.6 samples/s
347
+ Step 7600/44000 | Loss: 3.1725 | LR: 4.70e-04 | 1375.2ms/step | 14.5 samples/s
348
+ Step 7610/44000 | Loss: 3.1939 | LR: 4.70e-04 | 1374.8ms/step | 14.5 samples/s
349
+ Step 7620/44000 | Loss: 3.1872 | LR: 4.70e-04 | 1375.7ms/step | 14.5 samples/s
350
+ Step 7630/44000 | Loss: 3.1633 | LR: 4.69e-04 | 1369.8ms/step | 14.6 samples/s
351
+ Step 7640/44000 | Loss: 3.1707 | LR: 4.69e-04 | 1367.9ms/step | 14.6 samples/s
352
+ Step 7650/44000 | Loss: 3.1745 | LR: 4.69e-04 | 1373.1ms/step | 14.6 samples/s
353
+ Step 7660/44000 | Loss: 3.1699 | LR: 4.69e-04 | 1374.3ms/step | 14.6 samples/s
354
+ Step 7670/44000 | Loss: 3.1805 | LR: 4.69e-04 | 1376.9ms/step | 14.5 samples/s
355
+ Step 7680/44000 | Loss: 3.1818 | LR: 4.69e-04 | 1373.2ms/step | 14.6 samples/s
356
+ Step 7690/44000 | Loss: 3.1588 | LR: 4.69e-04 | 1373.1ms/step | 14.6 samples/s
357
+ Step 7700/44000 | Loss: 3.2160 | LR: 4.69e-04 | 1376.1ms/step | 14.5 samples/s
358
+ Step 7710/44000 | Loss: 3.1482 | LR: 4.69e-04 | 1374.5ms/step | 14.6 samples/s
359
+ Step 7720/44000 | Loss: 3.1540 | LR: 4.69e-04 | 1374.3ms/step | 14.6 samples/s
360
+ Step 7730/44000 | Loss: 3.1792 | LR: 4.69e-04 | 1376.6ms/step | 14.5 samples/s
361
+ Step 7740/44000 | Loss: 3.1868 | LR: 4.68e-04 | 1373.0ms/step | 14.6 samples/s
362
+ Step 7750/44000 | Loss: 3.2058 | LR: 4.68e-04 | 1374.0ms/step | 14.6 samples/s
363
+ Step 7760/44000 | Loss: 3.1753 | LR: 4.68e-04 | 1375.5ms/step | 14.5 samples/s
364
+ Step 7770/44000 | Loss: 3.1804 | LR: 4.68e-04 | 1373.9ms/step | 14.6 samples/s
365
+ Step 7780/44000 | Loss: 3.1564 | LR: 4.68e-04 | 1374.1ms/step | 14.6 samples/s
366
+ Step 7790/44000 | Loss: 3.1938 | LR: 4.68e-04 | 1372.9ms/step | 14.6 samples/s
367
+ Step 7800/44000 | Loss: 3.1710 | LR: 4.68e-04 | 1374.8ms/step | 14.5 samples/s
368
+ Step 7810/44000 | Loss: 3.1718 | LR: 4.68e-04 | 1376.5ms/step | 14.5 samples/s
369
+ Step 7820/44000 | Loss: 3.1649 | LR: 4.68e-04 | 1373.9ms/step | 14.6 samples/s
370
+ Step 7830/44000 | Loss: 3.1690 | LR: 4.68e-04 | 1373.8ms/step | 14.6 samples/s
371
+ Step 7840/44000 | Loss: 3.1664 | LR: 4.68e-04 | 1375.9ms/step | 14.5 samples/s
372
+ Step 7850/44000 | Loss: 3.1873 | LR: 4.67e-04 | 1373.3ms/step | 14.6 samples/s
373
+ Step 7860/44000 | Loss: 3.1695 | LR: 4.67e-04 | 1368.7ms/step | 14.6 samples/s
374
+ Step 7870/44000 | Loss: 3.1495 | LR: 4.67e-04 | 1367.3ms/step | 14.6 samples/s
375
+ Step 7880/44000 | Loss: 3.1589 | LR: 4.67e-04 | 1367.5ms/step | 14.6 samples/s
376
+ Step 7890/44000 | Loss: 3.1899 | LR: 4.67e-04 | 1367.2ms/step | 14.6 samples/s
377
+ Step 7900/44000 | Loss: 3.1443 | LR: 4.67e-04 | 1367.8ms/step | 14.6 samples/s
378
+ Step 7910/44000 | Loss: 3.1807 | LR: 4.67e-04 | 1367.8ms/step | 14.6 samples/s
379
+ Step 7920/44000 | Loss: 3.1729 | LR: 4.67e-04 | 1367.7ms/step | 14.6 samples/s
380
+ Step 7930/44000 | Loss: 3.1886 | LR: 4.67e-04 | 1368.0ms/step | 14.6 samples/s
381
+ Step 7940/44000 | Loss: 3.1851 | LR: 4.67e-04 | 1372.6ms/step | 14.6 samples/s
382
+ Step 7950/44000 | Loss: 3.1525 | LR: 4.67e-04 | 1374.8ms/step | 14.5 samples/s
383
+ Step 7960/44000 | Loss: 3.1523 | LR: 4.66e-04 | 1373.7ms/step | 14.6 samples/s
384
+ Step 7970/44000 | Loss: 3.1691 | LR: 4.66e-04 | 1374.9ms/step | 14.5 samples/s
385
+ Step 7980/44000 | Loss: 3.1590 | LR: 4.66e-04 | 1374.1ms/step | 14.6 samples/s
386
+ Step 7990/44000 | Loss: 3.1811 | LR: 4.66e-04 | 1368.8ms/step | 14.6 samples/s
387
+ Step 8000/44000 | Loss: 3.1724 | LR: 4.66e-04 | 1366.4ms/step | 14.6 samples/s
388
+ Validation at step 8000: Loss = 3.1700, Perplexity = 23.81
389
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735/checkpoint-8000
390
+ Step 8010/44000 | Loss: 3.1763 | LR: 4.66e-04 | 3825.9ms/step | 5.2 samples/s
391
+ Step 8020/44000 | Loss: 3.1703 | LR: 4.66e-04 | 1371.8ms/step | 14.6 samples/s
392
+ Step 8030/44000 | Loss: 3.1998 | LR: 4.66e-04 | 1376.1ms/step | 14.5 samples/s
393
+ Step 8040/44000 | Loss: 3.1558 | LR: 4.66e-04 | 1378.0ms/step | 14.5 samples/s
394
+ Step 8050/44000 | Loss: 3.2043 | LR: 4.66e-04 | 1379.9ms/step | 14.5 samples/s
395
+ Step 8060/44000 | Loss: 3.1568 | LR: 4.66e-04 | 1376.6ms/step | 14.5 samples/s
396
+ Step 8070/44000 | Loss: 3.1673 | LR: 4.65e-04 | 1379.6ms/step | 14.5 samples/s
397
+ Step 8080/44000 | Loss: 3.1835 | LR: 4.65e-04 | 1375.9ms/step | 14.5 samples/s
398
+ Step 8090/44000 | Loss: 3.1719 | LR: 4.65e-04 | 1379.2ms/step | 14.5 samples/s
399
+ Step 8100/44000 | Loss: 3.1674 | LR: 4.65e-04 | 1373.8ms/step | 14.6 samples/s
400
+ Step 8110/44000 | Loss: 3.1428 | LR: 4.65e-04 | 1377.5ms/step | 14.5 samples/s
401
+ Step 8120/44000 | Loss: 3.1635 | LR: 4.65e-04 | 1378.1ms/step | 14.5 samples/s
402
+ Step 8130/44000 | Loss: 3.1580 | LR: 4.65e-04 | 1376.4ms/step | 14.5 samples/s
403
+ Step 8140/44000 | Loss: 3.1821 | LR: 4.65e-04 | 1376.6ms/step | 14.5 samples/s
404
+ Step 8150/44000 | Loss: 3.1520 | LR: 4.65e-04 | 1374.4ms/step | 14.6 samples/s
405
+ Step 8160/44000 | Loss: 3.1861 | LR: 4.65e-04 | 1375.4ms/step | 14.5 samples/s
406
+ Step 8170/44000 | Loss: 3.1925 | LR: 4.65e-04 | 1377.8ms/step | 14.5 samples/s
407
+ Step 8180/44000 | Loss: 3.1330 | LR: 4.64e-04 | 1377.2ms/step | 14.5 samples/s
408
+ Step 8190/44000 | Loss: 3.1635 | LR: 4.64e-04 | 1379.0ms/step | 14.5 samples/s
409
+ Step 8200/44000 | Loss: 3.1542 | LR: 4.64e-04 | 1378.0ms/step | 14.5 samples/s
410
+ Step 8210/44000 | Loss: 3.1401 | LR: 4.64e-04 | 1380.1ms/step | 14.5 samples/s
411
+ Step 8220/44000 | Loss: 3.1462 | LR: 4.64e-04 | 1378.1ms/step | 14.5 samples/s
412
+ Step 8230/44000 | Loss: 3.1451 | LR: 4.64e-04 | 1376.3ms/step | 14.5 samples/s
413
+ Step 8240/44000 | Loss: 3.1620 | LR: 4.64e-04 | 1382.4ms/step | 14.5 samples/s
414
+ Step 8250/44000 | Loss: 3.1848 | LR: 4.64e-04 | 1379.8ms/step | 14.5 samples/s
415
+ Step 8260/44000 | Loss: 3.1417 | LR: 4.64e-04 | 1379.6ms/step | 14.5 samples/s
416
+ Step 8270/44000 | Loss: 3.1524 | LR: 4.64e-04 | 1380.5ms/step | 14.5 samples/s
417
+ Step 8280/44000 | Loss: 3.1416 | LR: 4.64e-04 | 1379.1ms/step | 14.5 samples/s
418
+ Step 8290/44000 | Loss: 3.1631 | LR: 4.63e-04 | 1379.0ms/step | 14.5 samples/s
419
+ Step 8300/44000 | Loss: 3.1837 | LR: 4.63e-04 | 1380.0ms/step | 14.5 samples/s
420
+ Step 8310/44000 | Loss: 3.1415 | LR: 4.63e-04 | 1378.4ms/step | 14.5 samples/s
421
+ Step 8320/44000 | Loss: 3.1485 | LR: 4.63e-04 | 1381.4ms/step | 14.5 samples/s
422
+ Step 8330/44000 | Loss: 3.1503 | LR: 4.63e-04 | 1380.7ms/step | 14.5 samples/s
423
+ Step 8340/44000 | Loss: 3.1598 | LR: 4.63e-04 | 1379.3ms/step | 14.5 samples/s
424
+ Step 8350/44000 | Loss: 3.1651 | LR: 4.63e-04 | 1381.6ms/step | 14.5 samples/s
425
+ Step 8360/44000 | Loss: 3.1267 | LR: 4.63e-04 | 1380.2ms/step | 14.5 samples/s
426
+ Step 8370/44000 | Loss: 3.1714 | LR: 4.63e-04 | 1378.6ms/step | 14.5 samples/s
427
+ Step 8380/44000 | Loss: 3.1538 | LR: 4.63e-04 | 1377.9ms/step | 14.5 samples/s
428
+ Step 8390/44000 | Loss: 3.1354 | LR: 4.62e-04 | 1379.9ms/step | 14.5 samples/s
429
+ Step 8400/44000 | Loss: 3.1465 | LR: 4.62e-04 | 1381.3ms/step | 14.5 samples/s
430
+ Step 8410/44000 | Loss: 3.1233 | LR: 4.62e-04 | 1378.9ms/step | 14.5 samples/s
431
+ Step 8420/44000 | Loss: 3.1918 | LR: 4.62e-04 | 1378.9ms/step | 14.5 samples/s
432
+ Step 8430/44000 | Loss: 3.1699 | LR: 4.62e-04 | 1379.2ms/step | 14.5 samples/s
433
+ Step 8440/44000 | Loss: 3.1743 | LR: 4.62e-04 | 1377.5ms/step | 14.5 samples/s
434
+ Step 8450/44000 | Loss: 3.1423 | LR: 4.62e-04 | 1376.7ms/step | 14.5 samples/s
435
+ Step 8460/44000 | Loss: 3.1477 | LR: 4.62e-04 | 1377.6ms/step | 14.5 samples/s
436
+ Step 8470/44000 | Loss: 3.1451 | LR: 4.62e-04 | 1377.9ms/step | 14.5 samples/s
437
+ Step 8480/44000 | Loss: 3.1738 | LR: 4.62e-04 | 1380.1ms/step | 14.5 samples/s
438
+ Step 8490/44000 | Loss: 3.1549 | LR: 4.62e-04 | 1380.6ms/step | 14.5 samples/s
439
+ Step 8500/44000 | Loss: 3.1455 | LR: 4.61e-04 | 1378.6ms/step | 14.5 samples/s
440
+ Step 8510/44000 | Loss: 3.1410 | LR: 4.61e-04 | 1379.4ms/step | 14.5 samples/s
441
+ Step 8520/44000 | Loss: 3.1421 | LR: 4.61e-04 | 1379.4ms/step | 14.5 samples/s
442
+ Step 8530/44000 | Loss: 3.1647 | LR: 4.61e-04 | 1378.2ms/step | 14.5 samples/s
443
+ Step 8540/44000 | Loss: 3.1325 | LR: 4.61e-04 | 1378.8ms/step | 14.5 samples/s
444
+ Step 8550/44000 | Loss: 3.1571 | LR: 4.61e-04 | 1378.7ms/step | 14.5 samples/s
445
+ Step 8560/44000 | Loss: 3.1742 | LR: 4.61e-04 | 1380.5ms/step | 14.5 samples/s
446
+ Step 8570/44000 | Loss: 3.1410 | LR: 4.61e-04 | 1379.6ms/step | 14.5 samples/s
447
+ Step 8580/44000 | Loss: 3.1554 | LR: 4.61e-04 | 1379.5ms/step | 14.5 samples/s
448
+ Step 8590/44000 | Loss: 3.1431 | LR: 4.61e-04 | 1379.8ms/step | 14.5 samples/s
449
+ Step 8600/44000 | Loss: 3.1456 | LR: 4.60e-04 | 1378.9ms/step | 14.5 samples/s
450
+ Step 8610/44000 | Loss: 3.1638 | LR: 4.60e-04 | 1376.4ms/step | 14.5 samples/s
451
+ Step 8620/44000 | Loss: 3.1544 | LR: 4.60e-04 | 1376.5ms/step | 14.5 samples/s
452
+ Step 8630/44000 | Loss: 3.1507 | LR: 4.60e-04 | 1375.8ms/step | 14.5 samples/s
453
+ Step 8640/44000 | Loss: 3.1333 | LR: 4.60e-04 | 1377.1ms/step | 14.5 samples/s
454
+ Step 8650/44000 | Loss: 3.1695 | LR: 4.60e-04 | 1376.8ms/step | 14.5 samples/s
455
+ Step 8660/44000 | Loss: 3.1677 | LR: 4.60e-04 | 1376.8ms/step | 14.5 samples/s
456
+ Step 8670/44000 | Loss: 3.1711 | LR: 4.60e-04 | 1376.4ms/step | 14.5 samples/s
457
+ Step 8680/44000 | Loss: 3.1458 | LR: 4.60e-04 | 1376.4ms/step | 14.5 samples/s
458
+ Step 8690/44000 | Loss: 3.1586 | LR: 4.60e-04 | 1376.0ms/step | 14.5 samples/s
459
+ Step 8700/44000 | Loss: 3.1438 | LR: 4.59e-04 | 1378.7ms/step | 14.5 samples/s
460
+ Step 8710/44000 | Loss: 3.1476 | LR: 4.59e-04 | 1378.5ms/step | 14.5 samples/s
461
+ Step 8720/44000 | Loss: 3.1529 | LR: 4.59e-04 | 1377.6ms/step | 14.5 samples/s
462
+ Step 8730/44000 | Loss: 3.1643 | LR: 4.59e-04 | 1374.8ms/step | 14.5 samples/s
463
+ Step 8740/44000 | Loss: 3.1561 | LR: 4.59e-04 | 1378.4ms/step | 14.5 samples/s
464
+ Step 8750/44000 | Loss: 3.1321 | LR: 4.59e-04 | 1378.2ms/step | 14.5 samples/s
465
+ Step 8760/44000 | Loss: 3.1329 | LR: 4.59e-04 | 1376.8ms/step | 14.5 samples/s
466
+ Step 8770/44000 | Loss: 3.1366 | LR: 4.59e-04 | 1377.1ms/step | 14.5 samples/s
467
+ Step 8780/44000 | Loss: 3.1257 | LR: 4.59e-04 | 1377.8ms/step | 14.5 samples/s
468
+ Step 8790/44000 | Loss: 3.1727 | LR: 4.59e-04 | 1378.3ms/step | 14.5 samples/s
469
+ Step 8800/44000 | Loss: 3.1205 | LR: 4.58e-04 | 1377.0ms/step | 14.5 samples/s
470
+ Step 8810/44000 | Loss: 3.1371 | LR: 4.58e-04 | 1376.3ms/step | 14.5 samples/s
471
+ Step 8820/44000 | Loss: 3.1782 | LR: 4.58e-04 | 1378.2ms/step | 14.5 samples/s
472
+ Step 8830/44000 | Loss: 3.1524 | LR: 4.58e-04 | 1378.4ms/step | 14.5 samples/s
473
+ Step 8840/44000 | Loss: 3.1315 | LR: 4.58e-04 | 1376.1ms/step | 14.5 samples/s
474
+ Step 8850/44000 | Loss: 3.1351 | LR: 4.58e-04 | 1376.1ms/step | 14.5 samples/s
475
+ Step 8860/44000 | Loss: 3.1553 | LR: 4.58e-04 | 1378.3ms/step | 14.5 samples/s
476
+ Step 8870/44000 | Loss: 3.1625 | LR: 4.58e-04 | 1376.8ms/step | 14.5 samples/s
477
+ Step 8880/44000 | Loss: 3.1433 | LR: 4.58e-04 | 1376.7ms/step | 14.5 samples/s
478
+ Step 8890/44000 | Loss: 3.1242 | LR: 4.58e-04 | 1376.2ms/step | 14.5 samples/s
479
+ Step 8900/44000 | Loss: 3.1515 | LR: 4.57e-04 | 1376.0ms/step | 14.5 samples/s
480
+ Step 8910/44000 | Loss: 3.1420 | LR: 4.57e-04 | 1377.8ms/step | 14.5 samples/s
481
+ Step 8920/44000 | Loss: 3.1557 | LR: 4.57e-04 | 1377.6ms/step | 14.5 samples/s
482
+ Step 8930/44000 | Loss: 3.0762 | LR: 4.57e-04 | 1378.3ms/step | 14.5 samples/s
483
+ Step 8940/44000 | Loss: 3.1286 | LR: 4.57e-04 | 1377.3ms/step | 14.5 samples/s
484
+ Step 8950/44000 | Loss: 3.1349 | LR: 4.57e-04 | 1378.2ms/step | 14.5 samples/s
485
+ Step 8960/44000 | Loss: 3.1219 | LR: 4.57e-04 | 1376.6ms/step | 14.5 samples/s
486
+ Step 8970/44000 | Loss: 3.1607 | LR: 4.57e-04 | 1376.6ms/step | 14.5 samples/s
487
+ Step 8980/44000 | Loss: 3.1704 | LR: 4.57e-04 | 1378.6ms/step | 14.5 samples/s
488
+ Step 8990/44000 | Loss: 3.1504 | LR: 4.57e-04 | 1377.9ms/step | 14.5 samples/s
489
+ Step 9000/44000 | Loss: 3.1532 | LR: 4.56e-04 | 1376.9ms/step | 14.5 samples/s
490
+ Validation at step 9000: Loss = 3.1415, Perplexity = 23.14
491
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735/checkpoint-9000
492
+ Step 9010/44000 | Loss: 3.1069 | LR: 4.56e-04 | 3835.4ms/step | 5.2 samples/s
493
+ Step 9020/44000 | Loss: 3.1067 | LR: 4.56e-04 | 1373.9ms/step | 14.6 samples/s
494
+ Step 9030/44000 | Loss: 3.1065 | LR: 4.56e-04 | 1374.5ms/step | 14.6 samples/s
495
+ Step 9040/44000 | Loss: 3.1691 | LR: 4.56e-04 | 1372.8ms/step | 14.6 samples/s
496
+ Step 9050/44000 | Loss: 3.1311 | LR: 4.56e-04 | 1376.6ms/step | 14.5 samples/s
497
+ Step 9060/44000 | Loss: 3.1256 | LR: 4.56e-04 | 1377.3ms/step | 14.5 samples/s
498
+ Step 9070/44000 | Loss: 3.1164 | LR: 4.56e-04 | 1375.3ms/step | 14.5 samples/s
499
+ Step 9080/44000 | Loss: 3.1446 | LR: 4.56e-04 | 1372.6ms/step | 14.6 samples/s
500
+ Step 9090/44000 | Loss: 3.1264 | LR: 4.56e-04 | 1375.7ms/step | 14.5 samples/s
501
+ Step 9100/44000 | Loss: 3.1010 | LR: 4.55e-04 | 1376.1ms/step | 14.5 samples/s
502
+ Step 9110/44000 | Loss: 3.1456 | LR: 4.55e-04 | 1373.8ms/step | 14.6 samples/s
503
+ Step 9120/44000 | Loss: 3.1560 | LR: 4.55e-04 | 1375.7ms/step | 14.5 samples/s
504
+ Step 9130/44000 | Loss: 3.1311 | LR: 4.55e-04 | 1376.2ms/step | 14.5 samples/s
505
+ Step 9140/44000 | Loss: 3.1120 | LR: 4.55e-04 | 1374.2ms/step | 14.6 samples/s
506
+ Step 9150/44000 | Loss: 3.1302 | LR: 4.55e-04 | 1376.9ms/step | 14.5 samples/s
507
+ Step 9160/44000 | Loss: 3.1368 | LR: 4.55e-04 | 1375.2ms/step | 14.5 samples/s
508
+ Step 9170/44000 | Loss: 3.1315 | LR: 4.55e-04 | 1377.5ms/step | 14.5 samples/s
509
+ Step 9180/44000 | Loss: 3.1249 | LR: 4.55e-04 | 1375.9ms/step | 14.5 samples/s
510
+ Step 9190/44000 | Loss: 3.1148 | LR: 4.54e-04 | 1378.4ms/step | 14.5 samples/s
511
+ Step 9200/44000 | Loss: 3.1521 | LR: 4.54e-04 | 1375.1ms/step | 14.5 samples/s
512
+ Step 9210/44000 | Loss: 3.1235 | LR: 4.54e-04 | 1377.0ms/step | 14.5 samples/s
513
+ Step 9220/44000 | Loss: 3.1273 | LR: 4.54e-04 | 1373.7ms/step | 14.6 samples/s
514
+ Step 9230/44000 | Loss: 3.1205 | LR: 4.54e-04 | 1368.1ms/step | 14.6 samples/s
515
+ Step 9240/44000 | Loss: 3.1336 | LR: 4.54e-04 | 1368.8ms/step | 14.6 samples/s
516
+ Step 9250/44000 | Loss: 3.1386 | LR: 4.54e-04 | 1371.4ms/step | 14.6 samples/s
517
+ Step 9260/44000 | Loss: 3.1111 | LR: 4.54e-04 | 1368.8ms/step | 14.6 samples/s
518
+ Step 9270/44000 | Loss: 3.1040 | LR: 4.54e-04 | 1369.3ms/step | 14.6 samples/s
519
+ Step 9280/44000 | Loss: 3.1550 | LR: 4.54e-04 | 1369.5ms/step | 14.6 samples/s
520
+ Step 9290/44000 | Loss: 3.1560 | LR: 4.53e-04 | 1370.4ms/step | 14.6 samples/s
521
+ Step 9300/44000 | Loss: 3.1248 | LR: 4.53e-04 | 1367.7ms/step | 14.6 samples/s
522
+ Step 9310/44000 | Loss: 3.1531 | LR: 4.53e-04 | 1370.8ms/step | 14.6 samples/s
523
+ Step 9320/44000 | Loss: 3.1440 | LR: 4.53e-04 | 1369.8ms/step | 14.6 samples/s
524
+ Step 9330/44000 | Loss: 3.0876 | LR: 4.53e-04 | 1371.1ms/step | 14.6 samples/s
525
+ Step 9340/44000 | Loss: 3.1292 | LR: 4.53e-04 | 1370.0ms/step | 14.6 samples/s
526
+ Step 9350/44000 | Loss: 3.1324 | LR: 4.53e-04 | 1369.7ms/step | 14.6 samples/s
527
+ Step 9360/44000 | Loss: 3.1445 | LR: 4.53e-04 | 1370.8ms/step | 14.6 samples/s
528
+ Step 9370/44000 | Loss: 3.1263 | LR: 4.53e-04 | 1368.1ms/step | 14.6 samples/s
529
+ Step 9380/44000 | Loss: 3.1299 | LR: 4.52e-04 | 1368.3ms/step | 14.6 samples/s
530
+ Step 9390/44000 | Loss: 3.1283 | LR: 4.52e-04 | 1368.7ms/step | 14.6 samples/s
531
+ Step 9400/44000 | Loss: 3.1297 | LR: 4.52e-04 | 1369.9ms/step | 14.6 samples/s
532
+ Step 9410/44000 | Loss: 3.1502 | LR: 4.52e-04 | 1369.2ms/step | 14.6 samples/s
533
+ Step 9420/44000 | Loss: 3.1017 | LR: 4.52e-04 | 1368.8ms/step | 14.6 samples/s
534
+ Step 9430/44000 | Loss: 3.1365 | LR: 4.52e-04 | 1370.0ms/step | 14.6 samples/s
535
+ Step 9440/44000 | Loss: 3.1403 | LR: 4.52e-04 | 1369.7ms/step | 14.6 samples/s
536
+ Step 9450/44000 | Loss: 3.0910 | LR: 4.52e-04 | 1369.4ms/step | 14.6 samples/s
537
+ Step 9460/44000 | Loss: 3.1217 | LR: 4.52e-04 | 1368.1ms/step | 14.6 samples/s
538
+ Step 9470/44000 | Loss: 3.0866 | LR: 4.52e-04 | 1369.1ms/step | 14.6 samples/s
539
+ Step 9480/44000 | Loss: 3.1197 | LR: 4.51e-04 | 1368.5ms/step | 14.6 samples/s
540
+ Step 9490/44000 | Loss: 3.1031 | LR: 4.51e-04 | 1368.7ms/step | 14.6 samples/s
541
+ Step 9500/44000 | Loss: 3.0980 | LR: 4.51e-04 | 1367.9ms/step | 14.6 samples/s
542
+ Step 9510/44000 | Loss: 3.1006 | LR: 4.51e-04 | 1368.3ms/step | 14.6 samples/s
543
+ Step 9520/44000 | Loss: 3.1351 | LR: 4.51e-04 | 1369.2ms/step | 14.6 samples/s
544
+ Step 9530/44000 | Loss: 3.0958 | LR: 4.51e-04 | 1367.3ms/step | 14.6 samples/s
545
+ Step 9540/44000 | Loss: 3.1327 | LR: 4.51e-04 | 1368.9ms/step | 14.6 samples/s
546
+ Step 9550/44000 | Loss: 3.1260 | LR: 4.51e-04 | 1372.4ms/step | 14.6 samples/s
547
+ Step 9560/44000 | Loss: 3.1151 | LR: 4.51e-04 | 1375.7ms/step | 14.5 samples/s
548
+ Step 9570/44000 | Loss: 3.1477 | LR: 4.50e-04 | 1374.6ms/step | 14.5 samples/s
549
+ Step 9580/44000 | Loss: 3.1397 | LR: 4.50e-04 | 1374.7ms/step | 14.5 samples/s
550
+ Step 9590/44000 | Loss: 3.1256 | LR: 4.50e-04 | 1374.4ms/step | 14.6 samples/s
551
+ Step 9600/44000 | Loss: 3.1267 | LR: 4.50e-04 | 1375.2ms/step | 14.5 samples/s
552
+ Step 9610/44000 | Loss: 3.1327 | LR: 4.50e-04 | 1368.1ms/step | 14.6 samples/s
553
+ Step 9620/44000 | Loss: 3.0977 | LR: 4.50e-04 | 1369.6ms/step | 14.6 samples/s
554
+ Step 9630/44000 | Loss: 3.1360 | LR: 4.50e-04 | 1370.1ms/step | 14.6 samples/s
555
+ Step 9640/44000 | Loss: 3.1195 | LR: 4.50e-04 | 1369.0ms/step | 14.6 samples/s
556
+ Step 9650/44000 | Loss: 3.1382 | LR: 4.50e-04 | 1373.3ms/step | 14.6 samples/s
557
+ Step 9660/44000 | Loss: 3.1296 | LR: 4.49e-04 | 1375.8ms/step | 14.5 samples/s
558
+ Step 9670/44000 | Loss: 3.1300 | LR: 4.49e-04 | 1377.9ms/step | 14.5 samples/s
559
+ Step 9680/44000 | Loss: 3.0823 | LR: 4.49e-04 | 1374.5ms/step | 14.6 samples/s
560
+ Step 9690/44000 | Loss: 3.0979 | LR: 4.49e-04 | 1373.5ms/step | 14.6 samples/s
561
+ Step 9700/44000 | Loss: 3.1471 | LR: 4.49e-04 | 1375.7ms/step | 14.5 samples/s
562
+ Step 9710/44000 | Loss: 3.1030 | LR: 4.49e-04 | 1376.1ms/step | 14.5 samples/s
563
+ Step 9720/44000 | Loss: 3.1351 | LR: 4.49e-04 | 1375.4ms/step | 14.5 samples/s
564
+ Step 9730/44000 | Loss: 3.1620 | LR: 4.49e-04 | 1377.0ms/step | 14.5 samples/s
565
+ Step 9740/44000 | Loss: 3.1316 | LR: 4.49e-04 | 1375.2ms/step | 14.5 samples/s
566
+ Step 9750/44000 | Loss: 3.1529 | LR: 4.48e-04 | 1377.9ms/step | 14.5 samples/s
567
+ Step 9760/44000 | Loss: 3.0992 | LR: 4.48e-04 | 1374.1ms/step | 14.6 samples/s
568
+ Step 9770/44000 | Loss: 3.1378 | LR: 4.48e-04 | 1368.1ms/step | 14.6 samples/s
569
+ Step 9780/44000 | Loss: 3.1098 | LR: 4.48e-04 | 1369.1ms/step | 14.6 samples/s
570
+ Step 9790/44000 | Loss: 3.1250 | LR: 4.48e-04 | 1368.2ms/step | 14.6 samples/s
571
+ Step 9800/44000 | Loss: 3.1183 | LR: 4.48e-04 | 1369.1ms/step | 14.6 samples/s
572
+ Step 9810/44000 | Loss: 3.1355 | LR: 4.48e-04 | 1367.0ms/step | 14.6 samples/s
573
+ Step 9820/44000 | Loss: 3.1089 | LR: 4.48e-04 | 1368.0ms/step | 14.6 samples/s
574
+ Step 9830/44000 | Loss: 3.1147 | LR: 4.48e-04 | 1368.0ms/step | 14.6 samples/s
575
+ Step 9840/44000 | Loss: 3.1440 | LR: 4.47e-04 | 1367.0ms/step | 14.6 samples/s
576
+ Step 9850/44000 | Loss: 3.0835 | LR: 4.47e-04 | 1374.2ms/step | 14.6 samples/s
577
+ Step 9860/44000 | Loss: 3.1299 | LR: 4.47e-04 | 1376.5ms/step | 14.5 samples/s
578
+ Step 9870/44000 | Loss: 3.0873 | LR: 4.47e-04 | 1370.5ms/step | 14.6 samples/s
579
+ Step 9880/44000 | Loss: 3.1159 | LR: 4.47e-04 | 1369.6ms/step | 14.6 samples/s
580
+ Step 9890/44000 | Loss: 3.1313 | LR: 4.47e-04 | 1367.1ms/step | 14.6 samples/s
581
+ Step 9900/44000 | Loss: 3.0940 | LR: 4.47e-04 | 1368.1ms/step | 14.6 samples/s
582
+ Step 9910/44000 | Loss: 3.1102 | LR: 4.47e-04 | 1368.1ms/step | 14.6 samples/s
583
+ Step 9920/44000 | Loss: 3.1081 | LR: 4.47e-04 | 1367.9ms/step | 14.6 samples/s
584
+ Step 9930/44000 | Loss: 3.1068 | LR: 4.46e-04 | 1367.8ms/step | 14.6 samples/s
585
+ Step 9940/44000 | Loss: 3.1374 | LR: 4.46e-04 | 1376.4ms/step | 14.5 samples/s
586
+ Step 9950/44000 | Loss: 3.1058 | LR: 4.46e-04 | 1374.8ms/step | 14.5 samples/s
587
+ Step 9960/44000 | Loss: 3.1421 | LR: 4.46e-04 | 1376.6ms/step | 14.5 samples/s
588
+ Step 9970/44000 | Loss: 3.1365 | LR: 4.46e-04 | 1375.3ms/step | 14.5 samples/s
589
+ Step 9980/44000 | Loss: 3.1054 | LR: 4.46e-04 | 1375.0ms/step | 14.5 samples/s
590
+ Step 9990/44000 | Loss: 3.1092 | LR: 4.46e-04 | 1375.7ms/step | 14.5 samples/s
591
+ Step 10000/44000 | Loss: 3.0955 | LR: 4.46e-04 | 1376.0ms/step | 14.5 samples/s
592
+ Validation at step 10000: Loss = 3.1172, Perplexity = 22.58
593
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735/checkpoint-10000
594
+ Step 10010/44000 | Loss: 3.1577 | LR: 4.46e-04 | 3850.3ms/step | 5.2 samples/s
595
+ Step 10020/44000 | Loss: 3.1321 | LR: 4.45e-04 | 1374.9ms/step | 14.5 samples/s
596
+ Step 10030/44000 | Loss: 3.1147 | LR: 4.45e-04 | 1378.1ms/step | 14.5 samples/s
597
+ Step 10040/44000 | Loss: 3.1116 | LR: 4.45e-04 | 1377.0ms/step | 14.5 samples/s
598
+ Step 10050/44000 | Loss: 3.1121 | LR: 4.45e-04 | 1376.2ms/step | 14.5 samples/s
599
+ Step 10060/44000 | Loss: 3.1021 | LR: 4.45e-04 | 1375.9ms/step | 14.5 samples/s
600
+ Step 10070/44000 | Loss: 3.1353 | LR: 4.45e-04 | 1373.1ms/step | 14.6 samples/s
601
+ Step 10080/44000 | Loss: 3.0877 | LR: 4.45e-04 | 1374.4ms/step | 14.6 samples/s
602
+ Step 10090/44000 | Loss: 3.1343 | LR: 4.45e-04 | 1374.7ms/step | 14.5 samples/s
603
+ Step 10100/44000 | Loss: 3.1085 | LR: 4.45e-04 | 1375.5ms/step | 14.5 samples/s
604
+ Step 10110/44000 | Loss: 3.1264 | LR: 4.44e-04 | 1374.8ms/step | 14.5 samples/s
605
+ Step 10120/44000 | Loss: 3.1049 | LR: 4.44e-04 | 1375.3ms/step | 14.5 samples/s
606
+ Step 10130/44000 | Loss: 3.1305 | LR: 4.44e-04 | 1374.9ms/step | 14.5 samples/s
607
+ Step 10140/44000 | Loss: 3.1432 | LR: 4.44e-04 | 1377.1ms/step | 14.5 samples/s
608
+ Step 10150/44000 | Loss: 3.1200 | LR: 4.44e-04 | 1374.5ms/step | 14.6 samples/s
609
+ Step 10160/44000 | Loss: 3.1051 | LR: 4.44e-04 | 1374.4ms/step | 14.6 samples/s
610
+ Step 10170/44000 | Loss: 3.0985 | LR: 4.44e-04 | 1376.4ms/step | 14.5 samples/s
611
+ Step 10180/44000 | Loss: 3.1284 | LR: 4.44e-04 | 1377.2ms/step | 14.5 samples/s
612
+ Step 10190/44000 | Loss: 3.0833 | LR: 4.43e-04 | 1377.8ms/step | 14.5 samples/s
613
+ Step 10200/44000 | Loss: 3.1028 | LR: 4.43e-04 | 1377.6ms/step | 14.5 samples/s
614
+ Step 10210/44000 | Loss: 3.1018 | LR: 4.43e-04 | 1377.3ms/step | 14.5 samples/s
615
+ Step 10220/44000 | Loss: 3.1118 | LR: 4.43e-04 | 1378.3ms/step | 14.5 samples/s
616
+ Step 10230/44000 | Loss: 3.1247 | LR: 4.43e-04 | 1376.4ms/step | 14.5 samples/s
617
+ Step 10240/44000 | Loss: 3.1250 | LR: 4.43e-04 | 1376.5ms/step | 14.5 samples/s
618
+ Step 10250/44000 | Loss: 3.1296 | LR: 4.43e-04 | 1377.3ms/step | 14.5 samples/s
619
+ Step 10260/44000 | Loss: 3.1431 | LR: 4.43e-04 | 1376.7ms/step | 14.5 samples/s
620
+ Step 10270/44000 | Loss: 3.0910 | LR: 4.43e-04 | 1377.4ms/step | 14.5 samples/s
621
+ Step 10280/44000 | Loss: 3.0994 | LR: 4.42e-04 | 1376.3ms/step | 14.5 samples/s
622
+ Step 10290/44000 | Loss: 3.1131 | LR: 4.42e-04 | 1375.8ms/step | 14.5 samples/s
623
+ Step 10300/44000 | Loss: 3.0805 | LR: 4.42e-04 | 1378.3ms/step | 14.5 samples/s
624
+ Step 10310/44000 | Loss: 3.1373 | LR: 4.42e-04 | 1376.0ms/step | 14.5 samples/s
625
+ Step 10320/44000 | Loss: 3.1321 | LR: 4.42e-04 | 1376.0ms/step | 14.5 samples/s
626
+ Step 10330/44000 | Loss: 3.1254 | LR: 4.42e-04 | 1375.7ms/step | 14.5 samples/s
627
+ Step 10340/44000 | Loss: 3.1153 | LR: 4.42e-04 | 1376.1ms/step | 14.5 samples/s
628
+ Step 10350/44000 | Loss: 3.1168 | LR: 4.42e-04 | 1377.3ms/step | 14.5 samples/s
629
+ Step 10360/44000 | Loss: 3.0953 | LR: 4.42e-04 | 1374.8ms/step | 14.5 samples/s
630
+ Step 10370/44000 | Loss: 3.1419 | LR: 4.41e-04 | 1374.5ms/step | 14.6 samples/s
631
+ Step 10380/44000 | Loss: 3.1200 | LR: 4.41e-04 | 1374.4ms/step | 14.6 samples/s
632
+ Step 10390/44000 | Loss: 3.1170 | LR: 4.41e-04 | 1373.9ms/step | 14.6 samples/s
633
+ Step 10400/44000 | Loss: 3.1196 | LR: 4.41e-04 | 1376.6ms/step | 14.5 samples/s
634
+ Step 10410/44000 | Loss: 3.1096 | LR: 4.41e-04 | 1376.4ms/step | 14.5 samples/s
635
+ Step 10420/44000 | Loss: 3.1237 | LR: 4.41e-04 | 1375.9ms/step | 14.5 samples/s
636
+ Step 10430/44000 | Loss: 3.1297 | LR: 4.41e-04 | 1376.1ms/step | 14.5 samples/s
637
+ Step 10440/44000 | Loss: 3.1274 | LR: 4.41e-04 | 1377.7ms/step | 14.5 samples/s
638
+ Step 10450/44000 | Loss: 3.1136 | LR: 4.40e-04 | 1376.5ms/step | 14.5 samples/s
639
+ Step 10460/44000 | Loss: 3.1078 | LR: 4.40e-04 | 1378.2ms/step | 14.5 samples/s
640
+ Step 10470/44000 | Loss: 3.1171 | LR: 4.40e-04 | 1376.6ms/step | 14.5 samples/s
641
+ Step 10480/44000 | Loss: 3.0902 | LR: 4.40e-04 | 1374.9ms/step | 14.5 samples/s
642
+ Step 10490/44000 | Loss: 3.0957 | LR: 4.40e-04 | 1375.9ms/step | 14.5 samples/s
643
+ Step 10500/44000 | Loss: 3.1322 | LR: 4.40e-04 | 1376.0ms/step | 14.5 samples/s
644
+ Step 10510/44000 | Loss: 3.1123 | LR: 4.40e-04 | 1378.0ms/step | 14.5 samples/s
645
+ Step 10520/44000 | Loss: 3.0964 | LR: 4.40e-04 | 1376.6ms/step | 14.5 samples/s
646
+ Step 10530/44000 | Loss: 3.1096 | LR: 4.40e-04 | 1374.9ms/step | 14.5 samples/s
647
+ Step 10540/44000 | Loss: 3.1352 | LR: 4.39e-04 | 1374.9ms/step | 14.5 samples/s
648
+ Step 10550/44000 | Loss: 3.0738 | LR: 4.39e-04 | 1374.7ms/step | 14.5 samples/s
649
+ Step 10560/44000 | Loss: 3.1273 | LR: 4.39e-04 | 1376.7ms/step | 14.5 samples/s
650
+ Step 10570/44000 | Loss: 3.1409 | LR: 4.39e-04 | 1376.0ms/step | 14.5 samples/s
651
+ Step 10580/44000 | Loss: 3.1175 | LR: 4.39e-04 | 1376.0ms/step | 14.5 samples/s
652
+ Step 10590/44000 | Loss: 3.0960 | LR: 4.39e-04 | 1376.3ms/step | 14.5 samples/s
653
+ Step 10600/44000 | Loss: 3.1233 | LR: 4.39e-04 | 1376.0ms/step | 14.5 samples/s
654
+ Step 10610/44000 | Loss: 3.1147 | LR: 4.39e-04 | 1375.2ms/step | 14.5 samples/s
655
+ Step 10620/44000 | Loss: 3.0829 | LR: 4.38e-04 | 1377.1ms/step | 14.5 samples/s
656
+ Step 10630/44000 | Loss: 3.0911 | LR: 4.38e-04 | 1376.0ms/step | 14.5 samples/s
657
+ Step 10640/44000 | Loss: 3.1074 | LR: 4.38e-04 | 1375.3ms/step | 14.5 samples/s
658
+ Step 10650/44000 | Loss: 3.1211 | LR: 4.38e-04 | 1375.5ms/step | 14.5 samples/s
659
+ Step 10660/44000 | Loss: 3.0993 | LR: 4.38e-04 | 1374.4ms/step | 14.6 samples/s
660
+ Step 10670/44000 | Loss: 3.1174 | LR: 4.38e-04 | 1374.4ms/step | 14.6 samples/s
661
+ Step 10680/44000 | Loss: 3.1108 | LR: 4.38e-04 | 1375.5ms/step | 14.5 samples/s
662
+ Step 10690/44000 | Loss: 3.0950 | LR: 4.38e-04 | 1376.0ms/step | 14.5 samples/s
663
+ Step 10700/44000 | Loss: 3.0975 | LR: 4.37e-04 | 1376.6ms/step | 14.5 samples/s
664
+ Step 10710/44000 | Loss: 3.0881 | LR: 4.37e-04 | 1375.9ms/step | 14.5 samples/s
665
+ Step 10720/44000 | Loss: 3.0976 | LR: 4.37e-04 | 1376.4ms/step | 14.5 samples/s
666
+ Step 10730/44000 | Loss: 3.1341 | LR: 4.37e-04 | 1375.7ms/step | 14.5 samples/s
667
+ Step 10740/44000 | Loss: 3.1056 | LR: 4.37e-04 | 1373.7ms/step | 14.6 samples/s
668
+ Step 10750/44000 | Loss: 3.1045 | LR: 4.37e-04 | 1374.4ms/step | 14.6 samples/s
669
+ Step 10760/44000 | Loss: 3.0856 | LR: 4.37e-04 | 1376.2ms/step | 14.5 samples/s
670
+ Step 10770/44000 | Loss: 3.0984 | LR: 4.37e-04 | 1374.9ms/step | 14.5 samples/s
671
+ Step 10780/44000 | Loss: 3.1071 | LR: 4.37e-04 | 1375.7ms/step | 14.5 samples/s
672
+ Step 10790/44000 | Loss: 3.0717 | LR: 4.36e-04 | 1375.2ms/step | 14.5 samples/s
673
+ Step 10800/44000 | Loss: 3.1103 | LR: 4.36e-04 | 1372.6ms/step | 14.6 samples/s
674
+ Step 10810/44000 | Loss: 3.0996 | LR: 4.36e-04 | 1372.8ms/step | 14.6 samples/s
675
+ Step 10820/44000 | Loss: 3.1098 | LR: 4.36e-04 | 1374.4ms/step | 14.6 samples/s
676
+ Step 10830/44000 | Loss: 3.1121 | LR: 4.36e-04 | 1376.3ms/step | 14.5 samples/s
677
+ Step 10840/44000 | Loss: 3.1197 | LR: 4.36e-04 | 1373.8ms/step | 14.6 samples/s
678
+ Step 10850/44000 | Loss: 3.1029 | LR: 4.36e-04 | 1374.6ms/step | 14.6 samples/s
679
+ Step 10860/44000 | Loss: 3.1019 | LR: 4.36e-04 | 1375.8ms/step | 14.5 samples/s
680
+ Step 10870/44000 | Loss: 3.0969 | LR: 4.35e-04 | 1376.1ms/step | 14.5 samples/s
681
+ Step 10880/44000 | Loss: 3.1032 | LR: 4.35e-04 | 1373.3ms/step | 14.6 samples/s
682
+ Step 10890/44000 | Loss: 3.0805 | LR: 4.35e-04 | 1376.9ms/step | 14.5 samples/s
683
+ Step 10900/44000 | Loss: 3.0826 | LR: 4.35e-04 | 1373.6ms/step | 14.6 samples/s
684
+ Step 10910/44000 | Loss: 3.0908 | LR: 4.35e-04 | 1374.5ms/step | 14.6 samples/s
685
+ Step 10920/44000 | Loss: 3.0796 | LR: 4.35e-04 | 1377.3ms/step | 14.5 samples/s
686
+ Step 10930/44000 | Loss: 3.0787 | LR: 4.35e-04 | 1374.5ms/step | 14.6 samples/s
687
+ Step 10940/44000 | Loss: 3.1248 | LR: 4.35e-04 | 1374.7ms/step | 14.5 samples/s
688
+ Step 10950/44000 | Loss: 3.0740 | LR: 4.34e-04 | 1374.6ms/step | 14.5 samples/s
689
+ Step 10960/44000 | Loss: 3.0370 | LR: 4.34e-04 | 1375.5ms/step | 14.5 samples/s
690
+ Step 10970/44000 | Loss: 3.0935 | LR: 4.34e-04 | 1376.2ms/step | 14.5 samples/s
691
+ Step 10980/44000 | Loss: 3.1175 | LR: 4.34e-04 | 1375.0ms/step | 14.5 samples/s
692
+ Step 10990/44000 | Loss: 3.1092 | LR: 4.34e-04 | 1376.0ms/step | 14.5 samples/s
693
+ Step 11000/44000 | Loss: 3.0943 | LR: 4.34e-04 | 1376.3ms/step | 14.5 samples/s
694
+ Validation at step 11000: Loss = 3.0969, Perplexity = 22.13
695
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735/checkpoint-11000
696
+ Step 11010/44000 | Loss: 3.1211 | LR: 4.34e-04 | 3832.1ms/step | 5.2 samples/s
697
+ Step 11020/44000 | Loss: 3.0622 | LR: 4.34e-04 | 1377.8ms/step | 14.5 samples/s
698
+ Step 11030/44000 | Loss: 3.0867 | LR: 4.33e-04 | 1375.7ms/step | 14.5 samples/s
699
+ Step 11040/44000 | Loss: 3.0748 | LR: 4.33e-04 | 1376.4ms/step | 14.5 samples/s
700
+ Step 11050/44000 | Loss: 3.1056 | LR: 4.33e-04 | 1376.0ms/step | 14.5 samples/s
701
+ Step 11060/44000 | Loss: 3.1100 | LR: 4.33e-04 | 1376.3ms/step | 14.5 samples/s
702
+ Step 11070/44000 | Loss: 3.0880 | LR: 4.33e-04 | 1376.7ms/step | 14.5 samples/s
703
+ Step 11080/44000 | Loss: 3.0939 | LR: 4.33e-04 | 1375.4ms/step | 14.5 samples/s
704
+ Step 11090/44000 | Loss: 3.0772 | LR: 4.33e-04 | 1376.6ms/step | 14.5 samples/s
705
+ Step 11100/44000 | Loss: 3.1127 | LR: 4.33e-04 | 1375.7ms/step | 14.5 samples/s
706
+ Step 11110/44000 | Loss: 3.0798 | LR: 4.32e-04 | 1375.4ms/step | 14.5 samples/s
707
+ Step 11120/44000 | Loss: 3.1024 | LR: 4.32e-04 | 1375.3ms/step | 14.5 samples/s
708
+ Step 11130/44000 | Loss: 3.0381 | LR: 4.32e-04 | 1374.9ms/step | 14.5 samples/s
709
+ Step 11140/44000 | Loss: 3.1133 | LR: 4.32e-04 | 1374.8ms/step | 14.5 samples/s
710
+ Step 11150/44000 | Loss: 3.0788 | LR: 4.32e-04 | 1376.0ms/step | 14.5 samples/s
711
+ Step 11160/44000 | Loss: 3.0934 | LR: 4.32e-04 | 1376.2ms/step | 14.5 samples/s
712
+ Step 11170/44000 | Loss: 3.1054 | LR: 4.32e-04 | 1376.3ms/step | 14.5 samples/s
713
+ Step 11180/44000 | Loss: 3.1096 | LR: 4.32e-04 | 1377.8ms/step | 14.5 samples/s
714
+ Step 11190/44000 | Loss: 3.0749 | LR: 4.31e-04 | 1377.4ms/step | 14.5 samples/s
715
+ Step 11200/44000 | Loss: 3.0931 | LR: 4.31e-04 | 1376.8ms/step | 14.5 samples/s
716
+ Step 11210/44000 | Loss: 3.0849 | LR: 4.31e-04 | 1377.7ms/step | 14.5 samples/s
717
+ Step 11220/44000 | Loss: 3.0990 | LR: 4.31e-04 | 1377.3ms/step | 14.5 samples/s
718
+ Step 11230/44000 | Loss: 3.1051 | LR: 4.31e-04 | 1368.3ms/step | 14.6 samples/s
719
+ Step 11240/44000 | Loss: 3.0623 | LR: 4.31e-04 | 1368.3ms/step | 14.6 samples/s
720
+ Step 11250/44000 | Loss: 3.0780 | LR: 4.31e-04 | 1366.1ms/step | 14.6 samples/s
721
+ Step 11260/44000 | Loss: 3.0475 | LR: 4.31e-04 | 1369.3ms/step | 14.6 samples/s
722
+ Step 11270/44000 | Loss: 3.0910 | LR: 4.30e-04 | 1368.2ms/step | 14.6 samples/s
723
+ Step 11280/44000 | Loss: 3.1075 | LR: 4.30e-04 | 1369.7ms/step | 14.6 samples/s
724
+ Step 11290/44000 | Loss: 3.0896 | LR: 4.30e-04 | 1369.1ms/step | 14.6 samples/s
725
+ Step 11300/44000 | Loss: 3.0585 | LR: 4.30e-04 | 1368.4ms/step | 14.6 samples/s
726
+ Step 11310/44000 | Loss: 3.0636 | LR: 4.30e-04 | 1367.8ms/step | 14.6 samples/s
727
+ Step 11320/44000 | Loss: 3.0840 | LR: 4.30e-04 | 1371.1ms/step | 14.6 samples/s
728
+ Step 11330/44000 | Loss: 3.0912 | LR: 4.30e-04 | 1369.0ms/step | 14.6 samples/s
729
+ Step 11340/44000 | Loss: 3.1084 | LR: 4.30e-04 | 1368.5ms/step | 14.6 samples/s
730
+ Step 11350/44000 | Loss: 3.0879 | LR: 4.29e-04 | 1368.4ms/step | 14.6 samples/s
731
+ Step 11360/44000 | Loss: 3.1062 | LR: 4.29e-04 | 1367.0ms/step | 14.6 samples/s
732
+ Step 11370/44000 | Loss: 3.0955 | LR: 4.29e-04 | 1368.4ms/step | 14.6 samples/s
733
+ Step 11380/44000 | Loss: 3.0599 | LR: 4.29e-04 | 1369.2ms/step | 14.6 samples/s
734
+ Step 11390/44000 | Loss: 3.0934 | LR: 4.29e-04 | 1375.0ms/step | 14.5 samples/s
735
+ Step 11400/44000 | Loss: 3.0717 | LR: 4.29e-04 | 1374.7ms/step | 14.5 samples/s
736
+ Step 11410/44000 | Loss: 3.0621 | LR: 4.29e-04 | 1369.5ms/step | 14.6 samples/s
737
+ Step 11420/44000 | Loss: 3.0803 | LR: 4.29e-04 | 1370.0ms/step | 14.6 samples/s
738
+ Step 11430/44000 | Loss: 3.0850 | LR: 4.28e-04 | 1374.5ms/step | 14.6 samples/s
739
+ Step 11440/44000 | Loss: 3.0717 | LR: 4.28e-04 | 1377.2ms/step | 14.5 samples/s
740
+ Step 11450/44000 | Loss: 3.0824 | LR: 4.28e-04 | 1376.2ms/step | 14.5 samples/s
741
+ Step 11460/44000 | Loss: 3.0860 | LR: 4.28e-04 | 1375.9ms/step | 14.5 samples/s
742
+ Step 11470/44000 | Loss: 3.0964 | LR: 4.28e-04 | 1375.0ms/step | 14.5 samples/s
743
+ Step 11480/44000 | Loss: 3.0860 | LR: 4.28e-04 | 1377.0ms/step | 14.5 samples/s
744
+ Step 11490/44000 | Loss: 3.0804 | LR: 4.28e-04 | 1377.5ms/step | 14.5 samples/s
745
+ Step 11500/44000 | Loss: 3.0837 | LR: 4.28e-04 | 1377.6ms/step | 14.5 samples/s
746
+ Step 11510/44000 | Loss: 3.0899 | LR: 4.27e-04 | 1375.6ms/step | 14.5 samples/s
747
+ Step 11520/44000 | Loss: 3.1188 | LR: 4.27e-04 | 1376.7ms/step | 14.5 samples/s
748
+ Step 11530/44000 | Loss: 3.0671 | LR: 4.27e-04 | 1378.3ms/step | 14.5 samples/s
749
+ Step 11540/44000 | Loss: 3.1175 | LR: 4.27e-04 | 1375.0ms/step | 14.5 samples/s
750
+ Step 11550/44000 | Loss: 3.1095 | LR: 4.27e-04 | 1381.8ms/step | 14.5 samples/s
751
+ Step 11560/44000 | Loss: 3.0602 | LR: 4.27e-04 | 1377.3ms/step | 14.5 samples/s
752
+ Step 11570/44000 | Loss: 3.0885 | LR: 4.27e-04 | 1373.3ms/step | 14.6 samples/s
753
+ Step 11580/44000 | Loss: 3.0708 | LR: 4.27e-04 | 1376.1ms/step | 14.5 samples/s
754
+ Step 11590/44000 | Loss: 3.1099 | LR: 4.26e-04 | 1374.6ms/step | 14.5 samples/s
755
+ Step 11600/44000 | Loss: 3.0879 | LR: 4.26e-04 | 1375.2ms/step | 14.5 samples/s
756
+ Step 11610/44000 | Loss: 3.0879 | LR: 4.26e-04 | 1373.0ms/step | 14.6 samples/s
757
+ Step 11620/44000 | Loss: 3.0705 | LR: 4.26e-04 | 1375.4ms/step | 14.5 samples/s
758
+ Step 11630/44000 | Loss: 3.0633 | LR: 4.26e-04 | 1375.8ms/step | 14.5 samples/s
759
+ Step 11640/44000 | Loss: 3.0600 | LR: 4.26e-04 | 1374.4ms/step | 14.6 samples/s
760
+ Step 11650/44000 | Loss: 3.0727 | LR: 4.26e-04 | 1373.5ms/step | 14.6 samples/s
761
+ Step 11660/44000 | Loss: 3.0803 | LR: 4.26e-04 | 1374.9ms/step | 14.5 samples/s
762
+ Step 11670/44000 | Loss: 3.0840 | LR: 4.25e-04 | 1377.1ms/step | 14.5 samples/s
763
+ Step 11680/44000 | Loss: 3.0877 | LR: 4.25e-04 | 1376.5ms/step | 14.5 samples/s
764
+ Step 11690/44000 | Loss: 3.0806 | LR: 4.25e-04 | 1375.2ms/step | 14.5 samples/s
765
+ Step 11700/44000 | Loss: 3.0938 | LR: 4.25e-04 | 1373.2ms/step | 14.6 samples/s
766
+ Step 11710/44000 | Loss: 3.0897 | LR: 4.25e-04 | 1374.5ms/step | 14.6 samples/s
767
+ Step 11720/44000 | Loss: 3.0731 | LR: 4.25e-04 | 1372.8ms/step | 14.6 samples/s
768
+ Step 11730/44000 | Loss: 3.0878 | LR: 4.25e-04 | 1375.7ms/step | 14.5 samples/s
769
+ Step 11740/44000 | Loss: 3.0579 | LR: 4.24e-04 | 1375.4ms/step | 14.5 samples/s
770
+ Step 11750/44000 | Loss: 3.1060 | LR: 4.24e-04 | 1376.4ms/step | 14.5 samples/s
771
+ Step 11760/44000 | Loss: 3.0475 | LR: 4.24e-04 | 1376.7ms/step | 14.5 samples/s
772
+ Step 11770/44000 | Loss: 3.0894 | LR: 4.24e-04 | 1375.9ms/step | 14.5 samples/s
773
+ Step 11780/44000 | Loss: 3.1142 | LR: 4.24e-04 | 1376.1ms/step | 14.5 samples/s
774
+ Step 11790/44000 | Loss: 3.0960 | LR: 4.24e-04 | 1380.3ms/step | 14.5 samples/s
775
+ Step 11800/44000 | Loss: 3.0810 | LR: 4.24e-04 | 1381.3ms/step | 14.5 samples/s
776
+ Step 11810/44000 | Loss: 3.0988 | LR: 4.24e-04 | 1378.1ms/step | 14.5 samples/s
777
+ Step 11820/44000 | Loss: 3.0816 | LR: 4.23e-04 | 1378.4ms/step | 14.5 samples/s
778
+ Step 11830/44000 | Loss: 3.1038 | LR: 4.23e-04 | 1379.9ms/step | 14.5 samples/s
779
+ Step 11840/44000 | Loss: 3.0876 | LR: 4.23e-04 | 1380.2ms/step | 14.5 samples/s
780
+ Step 11850/44000 | Loss: 3.0735 | LR: 4.23e-04 | 1378.7ms/step | 14.5 samples/s
781
+ Step 11860/44000 | Loss: 3.0879 | LR: 4.23e-04 | 1379.4ms/step | 14.5 samples/s
782
+ Step 11870/44000 | Loss: 3.0623 | LR: 4.23e-04 | 1379.9ms/step | 14.5 samples/s
783
+ Step 11880/44000 | Loss: 3.0797 | LR: 4.23e-04 | 1378.1ms/step | 14.5 samples/s
784
+ Step 11890/44000 | Loss: 3.0711 | LR: 4.23e-04 | 1377.7ms/step | 14.5 samples/s
785
+ Step 11900/44000 | Loss: 3.0991 | LR: 4.22e-04 | 1377.7ms/step | 14.5 samples/s
786
+ Step 11910/44000 | Loss: 3.0638 | LR: 4.22e-04 | 1378.2ms/step | 14.5 samples/s
787
+ Step 11920/44000 | Loss: 3.0691 | LR: 4.22e-04 | 1375.3ms/step | 14.5 samples/s
788
+ Step 11930/44000 | Loss: 3.0873 | LR: 4.22e-04 | 1374.9ms/step | 14.5 samples/s
789
+ Step 11940/44000 | Loss: 3.0698 | LR: 4.22e-04 | 1375.0ms/step | 14.5 samples/s
790
+ Step 11950/44000 | Loss: 3.0666 | LR: 4.22e-04 | 1376.1ms/step | 14.5 samples/s
791
+ Step 11960/44000 | Loss: 3.0588 | LR: 4.22e-04 | 1375.0ms/step | 14.5 samples/s
792
+ Step 11970/44000 | Loss: 3.1018 | LR: 4.21e-04 | 1376.0ms/step | 14.5 samples/s
793
+ Step 11980/44000 | Loss: 3.0509 | LR: 4.21e-04 | 1373.8ms/step | 14.6 samples/s
794
+ Step 11990/44000 | Loss: 3.0913 | LR: 4.21e-04 | 1375.4ms/step | 14.5 samples/s
795
+ Step 12000/44000 | Loss: 3.1078 | LR: 4.21e-04 | 1377.3ms/step | 14.5 samples/s
796
+ Validation at step 12000: Loss = 3.0785, Perplexity = 21.73
797
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735/checkpoint-12000
798
+ Step 12010/44000 | Loss: 3.0695 | LR: 4.21e-04 | 3943.1ms/step | 5.1 samples/s
799
+ Step 12020/44000 | Loss: 3.0984 | LR: 4.21e-04 | 1372.6ms/step | 14.6 samples/s
800
+ Step 12030/44000 | Loss: 3.0893 | LR: 4.21e-04 | 1372.3ms/step | 14.6 samples/s
801
+ Step 12040/44000 | Loss: 3.0485 | LR: 4.21e-04 | 1375.4ms/step | 14.5 samples/s
802
+ Step 12050/44000 | Loss: 3.0721 | LR: 4.20e-04 | 1377.5ms/step | 14.5 samples/s
803
+ Step 12060/44000 | Loss: 3.1008 | LR: 4.20e-04 | 1376.6ms/step | 14.5 samples/s
804
+ Step 12070/44000 | Loss: 3.0752 | LR: 4.20e-04 | 1377.8ms/step | 14.5 samples/s
805
+ Step 12080/44000 | Loss: 3.0788 | LR: 4.20e-04 | 1379.0ms/step | 14.5 samples/s
806
+ Step 12090/44000 | Loss: 3.1005 | LR: 4.20e-04 | 1377.9ms/step | 14.5 samples/s
807
+ Step 12100/44000 | Loss: 3.0851 | LR: 4.20e-04 | 1377.5ms/step | 14.5 samples/s
808
+ Step 12110/44000 | Loss: 3.0940 | LR: 4.20e-04 | 1376.9ms/step | 14.5 samples/s
809
+ Step 12120/44000 | Loss: 3.0943 | LR: 4.19e-04 | 1378.5ms/step | 14.5 samples/s
810
+ Step 12130/44000 | Loss: 3.0485 | LR: 4.19e-04 | 1376.3ms/step | 14.5 samples/s
811
+ Step 12140/44000 | Loss: 3.0564 | LR: 4.19e-04 | 1376.4ms/step | 14.5 samples/s
812
+ Step 12150/44000 | Loss: 3.0719 | LR: 4.19e-04 | 1379.6ms/step | 14.5 samples/s
813
+ Step 12160/44000 | Loss: 3.1018 | LR: 4.19e-04 | 1379.6ms/step | 14.5 samples/s
814
+ Step 12170/44000 | Loss: 3.0638 | LR: 4.19e-04 | 1378.5ms/step | 14.5 samples/s
815
+ Step 12180/44000 | Loss: 3.0824 | LR: 4.19e-04 | 1380.0ms/step | 14.5 samples/s
816
+ Step 12190/44000 | Loss: 3.0772 | LR: 4.19e-04 | 1377.9ms/step | 14.5 samples/s
817
+ Step 12200/44000 | Loss: 3.0660 | LR: 4.18e-04 | 1377.0ms/step | 14.5 samples/s
818
+ Step 12210/44000 | Loss: 3.0686 | LR: 4.18e-04 | 1379.0ms/step | 14.5 samples/s
819
+ Step 12220/44000 | Loss: 3.0834 | LR: 4.18e-04 | 1379.9ms/step | 14.5 samples/s
820
+ Step 12230/44000 | Loss: 3.0601 | LR: 4.18e-04 | 1379.9ms/step | 14.5 samples/s
821
+ Step 12240/44000 | Loss: 3.1170 | LR: 4.18e-04 | 1378.3ms/step | 14.5 samples/s
822
+ Step 12250/44000 | Loss: 3.0921 | LR: 4.18e-04 | 1378.6ms/step | 14.5 samples/s
823
+ Step 12260/44000 | Loss: 3.0651 | LR: 4.18e-04 | 1378.3ms/step | 14.5 samples/s
824
+ Step 12270/44000 | Loss: 3.0860 | LR: 4.17e-04 | 1378.2ms/step | 14.5 samples/s
825
+ Step 12280/44000 | Loss: 3.0773 | LR: 4.17e-04 | 1378.9ms/step | 14.5 samples/s
826
+ Step 12290/44000 | Loss: 3.0516 | LR: 4.17e-04 | 1377.2ms/step | 14.5 samples/s
827
+ Step 12300/44000 | Loss: 3.0851 | LR: 4.17e-04 | 1377.8ms/step | 14.5 samples/s
828
+ Step 12310/44000 | Loss: 3.0693 | LR: 4.17e-04 | 1379.4ms/step | 14.5 samples/s
829
+ Step 12320/44000 | Loss: 3.1017 | LR: 4.17e-04 | 1378.0ms/step | 14.5 samples/s
830
+ Step 12330/44000 | Loss: 3.0398 | LR: 4.17e-04 | 1379.1ms/step | 14.5 samples/s
831
+ Step 12340/44000 | Loss: 3.0679 | LR: 4.17e-04 | 1378.3ms/step | 14.5 samples/s
832
+ Step 12350/44000 | Loss: 3.0735 | LR: 4.16e-04 | 1378.3ms/step | 14.5 samples/s
833
+ Step 12360/44000 | Loss: 3.0593 | LR: 4.16e-04 | 1379.8ms/step | 14.5 samples/s
834
+ Step 12370/44000 | Loss: 3.0761 | LR: 4.16e-04 | 1378.0ms/step | 14.5 samples/s
835
+ Step 12380/44000 | Loss: 3.0792 | LR: 4.16e-04 | 1377.4ms/step | 14.5 samples/s
836
+ Step 12390/44000 | Loss: 3.0583 | LR: 4.16e-04 | 1378.3ms/step | 14.5 samples/s
837
+ Step 12400/44000 | Loss: 3.0592 | LR: 4.16e-04 | 1378.4ms/step | 14.5 samples/s
838
+ Step 12410/44000 | Loss: 3.0648 | LR: 4.16e-04 | 1380.0ms/step | 14.5 samples/s
839
+ Step 12420/44000 | Loss: 3.0735 | LR: 4.15e-04 | 1378.0ms/step | 14.5 samples/s
840
+ Step 12430/44000 | Loss: 3.0795 | LR: 4.15e-04 | 1377.4ms/step | 14.5 samples/s
841
+ Step 12440/44000 | Loss: 3.0746 | LR: 4.15e-04 | 1378.8ms/step | 14.5 samples/s
842
+ Step 12450/44000 | Loss: 3.0797 | LR: 4.15e-04 | 1379.5ms/step | 14.5 samples/s
843
+ Step 12460/44000 | Loss: 3.0944 | LR: 4.15e-04 | 1380.4ms/step | 14.5 samples/s
844
+ Step 12470/44000 | Loss: 3.0832 | LR: 4.15e-04 | 1380.1ms/step | 14.5 samples/s
845
+ Step 12480/44000 | Loss: 3.0779 | LR: 4.15e-04 | 1381.2ms/step | 14.5 samples/s
846
+ Step 12490/44000 | Loss: 3.0669 | LR: 4.14e-04 | 1377.8ms/step | 14.5 samples/s
847
+ Step 12500/44000 | Loss: 3.0753 | LR: 4.14e-04 | 1380.9ms/step | 14.5 samples/s
848
+ Step 12510/44000 | Loss: 3.0940 | LR: 4.14e-04 | 1381.9ms/step | 14.5 samples/s
849
+ Step 12520/44000 | Loss: 3.0361 | LR: 4.14e-04 | 1380.1ms/step | 14.5 samples/s
850
+ Step 12530/44000 | Loss: 3.0890 | LR: 4.14e-04 | 1380.2ms/step | 14.5 samples/s
851
+ Step 12540/44000 | Loss: 3.0586 | LR: 4.14e-04 | 1379.7ms/step | 14.5 samples/s
852
+ Step 12550/44000 | Loss: 3.0954 | LR: 4.14e-04 | 1376.1ms/step | 14.5 samples/s
853
+ Step 12560/44000 | Loss: 3.0942 | LR: 4.14e-04 | 1375.5ms/step | 14.5 samples/s
854
+ Step 12570/44000 | Loss: 3.0790 | LR: 4.13e-04 | 1376.9ms/step | 14.5 samples/s
855
+ Step 12580/44000 | Loss: 3.0625 | LR: 4.13e-04 | 1377.7ms/step | 14.5 samples/s
856
+ Step 12590/44000 | Loss: 3.0806 | LR: 4.13e-04 | 1378.0ms/step | 14.5 samples/s
857
+ Step 12600/44000 | Loss: 3.0706 | LR: 4.13e-04 | 1374.8ms/step | 14.5 samples/s
858
+ Step 12610/44000 | Loss: 3.0925 | LR: 4.13e-04 | 1376.4ms/step | 14.5 samples/s
859
+ Step 12620/44000 | Loss: 3.0754 | LR: 4.13e-04 | 1376.4ms/step | 14.5 samples/s
860
+ Step 12630/44000 | Loss: 3.0472 | LR: 4.13e-04 | 1376.2ms/step | 14.5 samples/s
861
+ Step 12640/44000 | Loss: 3.0778 | LR: 4.12e-04 | 1376.5ms/step | 14.5 samples/s
862
+ Step 12650/44000 | Loss: 3.0278 | LR: 4.12e-04 | 1377.2ms/step | 14.5 samples/s
863
+ Step 12660/44000 | Loss: 3.0597 | LR: 4.12e-04 | 1377.8ms/step | 14.5 samples/s
864
+ Step 12670/44000 | Loss: 3.0610 | LR: 4.12e-04 | 1379.1ms/step | 14.5 samples/s
865
+ Step 12680/44000 | Loss: 3.0325 | LR: 4.12e-04 | 1377.5ms/step | 14.5 samples/s
866
+ Step 12690/44000 | Loss: 3.0713 | LR: 4.12e-04 | 1379.0ms/step | 14.5 samples/s
867
+ Step 12700/44000 | Loss: 3.0465 | LR: 4.12e-04 | 1375.3ms/step | 14.5 samples/s
868
+ Step 12710/44000 | Loss: 3.0677 | LR: 4.11e-04 | 1375.4ms/step | 14.5 samples/s
869
+ Step 12720/44000 | Loss: 3.0690 | LR: 4.11e-04 | 1376.7ms/step | 14.5 samples/s
870
+ Step 12730/44000 | Loss: 3.0304 | LR: 4.11e-04 | 1380.4ms/step | 14.5 samples/s
871
+ Step 12740/44000 | Loss: 3.0626 | LR: 4.11e-04 | 1377.2ms/step | 14.5 samples/s
872
+ Step 12750/44000 | Loss: 3.0513 | LR: 4.11e-04 | 1376.6ms/step | 14.5 samples/s
873
+ Step 12760/44000 | Loss: 3.0534 | LR: 4.11e-04 | 1377.9ms/step | 14.5 samples/s
874
+ Step 12770/44000 | Loss: 3.0609 | LR: 4.11e-04 | 1377.8ms/step | 14.5 samples/s
875
+ Step 12780/44000 | Loss: 3.0386 | LR: 4.10e-04 | 1378.0ms/step | 14.5 samples/s
876
+ Step 12790/44000 | Loss: 3.0572 | LR: 4.10e-04 | 1377.8ms/step | 14.5 samples/s
877
+ Step 12800/44000 | Loss: 3.0674 | LR: 4.10e-04 | 1378.1ms/step | 14.5 samples/s
878
+ Step 12810/44000 | Loss: 3.0511 | LR: 4.10e-04 | 1378.0ms/step | 14.5 samples/s
879
+ Step 12820/44000 | Loss: 3.0646 | LR: 4.10e-04 | 1376.2ms/step | 14.5 samples/s
880
+ Step 12830/44000 | Loss: 3.0470 | LR: 4.10e-04 | 1378.0ms/step | 14.5 samples/s
881
+ Step 12840/44000 | Loss: 3.0640 | LR: 4.10e-04 | 1379.9ms/step | 14.5 samples/s
882
+ Step 12850/44000 | Loss: 3.0479 | LR: 4.09e-04 | 1376.8ms/step | 14.5 samples/s
883
+ Step 12860/44000 | Loss: 3.0857 | LR: 4.09e-04 | 1379.3ms/step | 14.5 samples/s
884
+ Step 12870/44000 | Loss: 3.0733 | LR: 4.09e-04 | 1376.9ms/step | 14.5 samples/s
885
+ Step 12880/44000 | Loss: 3.0709 | LR: 4.09e-04 | 1378.5ms/step | 14.5 samples/s
886
+ Step 12890/44000 | Loss: 3.0448 | LR: 4.09e-04 | 1377.7ms/step | 14.5 samples/s
887
+ Step 12900/44000 | Loss: 3.0671 | LR: 4.09e-04 | 1375.8ms/step | 14.5 samples/s
888
+ Step 12910/44000 | Loss: 3.0943 | LR: 4.09e-04 | 1378.7ms/step | 14.5 samples/s
889
+ Step 12920/44000 | Loss: 3.0598 | LR: 4.09e-04 | 1376.1ms/step | 14.5 samples/s
890
+ Step 12930/44000 | Loss: 3.0691 | LR: 4.08e-04 | 1375.7ms/step | 14.5 samples/s
891
+ Step 12940/44000 | Loss: 3.0809 | LR: 4.08e-04 | 1376.7ms/step | 14.5 samples/s
892
+ Step 12950/44000 | Loss: 3.0521 | LR: 4.08e-04 | 1376.0ms/step | 14.5 samples/s
893
+ Step 12960/44000 | Loss: 3.0496 | LR: 4.08e-04 | 1375.9ms/step | 14.5 samples/s
894
+ Step 12970/44000 | Loss: 3.0801 | LR: 4.08e-04 | 1377.8ms/step | 14.5 samples/s
895
+ Step 12980/44000 | Loss: 3.0637 | LR: 4.08e-04 | 1378.4ms/step | 14.5 samples/s
896
+ Step 12990/44000 | Loss: 3.0781 | LR: 4.08e-04 | 1378.1ms/step | 14.5 samples/s
897
+ Step 13000/44000 | Loss: 3.0612 | LR: 4.07e-04 | 1375.0ms/step | 14.5 samples/s
898
+ Validation at step 13000: Loss = 3.0601, Perplexity = 21.33
899
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735/checkpoint-13000
900
+ Step 13010/44000 | Loss: 3.0525 | LR: 4.07e-04 | 3820.7ms/step | 5.2 samples/s
901
+ Step 13020/44000 | Loss: 3.0433 | LR: 4.07e-04 | 1372.0ms/step | 14.6 samples/s
902
+ Step 13030/44000 | Loss: 3.0306 | LR: 4.07e-04 | 1369.9ms/step | 14.6 samples/s
903
+ Step 13040/44000 | Loss: 3.0661 | LR: 4.07e-04 | 1375.7ms/step | 14.5 samples/s
904
+ Step 13050/44000 | Loss: 3.0451 | LR: 4.07e-04 | 1374.6ms/step | 14.6 samples/s
905
+ Step 13060/44000 | Loss: 3.0528 | LR: 4.07e-04 | 1377.3ms/step | 14.5 samples/s
906
+ Step 13070/44000 | Loss: 3.0593 | LR: 4.06e-04 | 1378.6ms/step | 14.5 samples/s
907
+ Step 13080/44000 | Loss: 3.0376 | LR: 4.06e-04 | 1377.0ms/step | 14.5 samples/s
908
+ Step 13090/44000 | Loss: 3.0893 | LR: 4.06e-04 | 1376.0ms/step | 14.5 samples/s
909
+ Step 13100/44000 | Loss: 3.0524 | LR: 4.06e-04 | 1375.1ms/step | 14.5 samples/s
910
+ Step 13110/44000 | Loss: 3.0893 | LR: 4.06e-04 | 1374.7ms/step | 14.5 samples/s
911
+ Step 13120/44000 | Loss: 3.0876 | LR: 4.06e-04 | 1375.2ms/step | 14.5 samples/s
912
+ Step 13130/44000 | Loss: 3.0559 | LR: 4.06e-04 | 1378.2ms/step | 14.5 samples/s
913
+ Step 13140/44000 | Loss: 3.0262 | LR: 4.05e-04 | 1378.9ms/step | 14.5 samples/s
914
+ Step 13150/44000 | Loss: 3.0955 | LR: 4.05e-04 | 1376.5ms/step | 14.5 samples/s
915
+ Step 13160/44000 | Loss: 3.0439 | LR: 4.05e-04 | 1376.2ms/step | 14.5 samples/s
epfl_final/train_logs/ropectl_a1p0_h100_resume/seed46.20260820T011048Z.log ADDED
@@ -0,0 +1,905 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Auto-resuming from latest checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735/checkpoint-6000
2
+ Resuming: Using existing output directory /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735
3
+ Running config:
4
+ {
5
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
6
+ "model_dtype": "bfloat16",
7
+ "init_from_scratch": true,
8
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
9
+ "val_path": null,
10
+ "max_seq_length": 8192,
11
+ "num_workers": 8,
12
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735",
13
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735/checkpoint-6000",
14
+ "force_new_run": false,
15
+ "num_epochs": 1,
16
+ "max_steps": 44000,
17
+ "per_device_batch_size": 2,
18
+ "gradient_accumulation_steps": 10,
19
+ "learning_rate": 0.0005,
20
+ "weight_decay": 0.01,
21
+ "warmup_steps": 750,
22
+ "lr_scheduler": "cosine",
23
+ "max_grad_norm": 1.0,
24
+ "save_steps": 1000,
25
+ "eval_steps": 1000,
26
+ "eval_batches": 50,
27
+ "logging_steps": 10,
28
+ "seed": 46,
29
+ "verbose": true,
30
+ "checkpoint_activations": false,
31
+ "strategy": "ddp",
32
+ "mixed_precision": "bf16",
33
+ "wandb_enabled": true,
34
+ "wandb_project": "stochastic_sparsity",
35
+ "wandb_name": "l8k365m_dense_ropectl_a1p0_seed46",
36
+ "wandb_tags": [
37
+ "qwen3",
38
+ "365m",
39
+ "8k",
40
+ "dense",
41
+ "rope-control"
42
+ ],
43
+ "hidden_size": 768,
44
+ "intermediate_size": 3072,
45
+ "num_attention_heads": 6,
46
+ "num_key_value_heads": 2,
47
+ "num_hidden_layers": 28,
48
+ "head_dim": null,
49
+ "attention_dropout": 0.0,
50
+ "rope_train_position_scale_set": [
51
+ 1.0,
52
+ 0.9896339488176222,
53
+ 0.9692159692159692,
54
+ 0.9420289855072463,
55
+ 0.8730158730158729
56
+ ],
57
+ "attention_type": "sparse_stochastic",
58
+ "sparse_block_size": 128,
59
+ "sparse_num_random_blocks": 31,
60
+ "sparse_mask_pool_size": 1,
61
+ "sparse_val_dense_compare": false,
62
+ "sparse_layer_indices": null,
63
+ "sparse_mask_cache_dir": null,
64
+ "sparse_mask_seed": 0,
65
+ "sparse_mixed_nrb_set": null,
66
+ "dense_mode": true,
67
+ "dense_step_intervals": null
68
+ }
69
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
70
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735/checkpoint-6000
71
+ Checkpoint activations: False
72
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
73
+
74
+
75
+ ================================================================================
76
+ MODEL ARCHITECTURE
77
+ ================================================================================
78
+ Model config:
79
+ vocab_size: 151936
80
+ hidden_size: 768
81
+ num_hidden_layers: 28
82
+ num_attention_heads: 6
83
+ num_key_value_heads: 2
84
+ head_dim: 128
85
+ intermediate_size: 3072
86
+ max_position_embeddings: 32768
87
+ rope_theta: 1000000
88
+ rope_train_position_scale_set: [1.0, 0.9896339488176222, 0.9692159692159692, 0.9420289855072463, 0.8730158730158729]
89
+ attention_type: sparse_stochastic
90
+ sparse_block_size: 128
91
+ sparse_num_random_blocks: 31
92
+ sparse_mask_pool_size: 1
93
+ sparse_max_seq_length: 8192
94
+
95
+ Parameter count:
96
+ Total parameters: 358,958,848
97
+ Trainable parameters: 358,958,848
98
+ Non-trainable parameters: 0
99
+ Model size (BF16): 0.72 GB
100
+
101
+ --------------------------------------------------------------------------------
102
+ LAYER-BY-LAYER SUMMARY
103
+ --------------------------------------------------------------------------------
104
+ ================================================================================================================================================================
105
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
106
+ ================================================================================================================================================================
107
+ Qwen3ForCausalLM [2, 8192] -- -- True
108
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
109
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
110
+ │ └─ModuleList: 2-2 -- -- -- True
111
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
132
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
133
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
134
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
135
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
136
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
137
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
138
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
139
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
140
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
141
+ ================================================================================================================================================================
142
+ Total params: 475,645,696
143
+ Trainable params: 475,645,696
144
+ Non-trainable params: 0
145
+ Total mult-adds (Units.MEGABYTES): 951.29
146
+ ================================================================================================================================================================
147
+ Input size (MB): 0.13
148
+ Forward/backward pass size (MB): 31197.23
149
+ Params size (MB): 951.29
150
+ Estimated Total Size (MB): 32148.66
151
+ ================================================================================================================================================================
152
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
153
+ ================================================================================
154
+
155
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
156
+ Using pre-tokenized dataset (no runtime tokenization)
157
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
158
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
159
+ Dataset split: 3027764 train, 159356 validation
160
+ Training examples (packed, train split): 3027764
161
+ Validation examples (packed, val split): 159356
162
+ Per-device batch size: 2
163
+ World size (GPUs): 1
164
+ Gradient accumulation steps: 10
165
+ Effective batch size (seqs/step): 20
166
+ Sequence length: 8192
167
+ Tokens per optimizer step: 163,840
168
+ Micro-batches per epoch (per rank): 1513882
169
+ Optimizer steps per epoch: 151388
170
+ Total optimizer steps: 44000 (capped by max_steps; 0.29 epochs)
171
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735/checkpoint-6000...
172
+ Preserving checkpoint LR state for exact resume: current=[0.0004820408177936365], schedule_base=[0.0005]
173
+ Resumed from checkpoint at step 6000 (epoch 0, micro_step 60000)
174
+ Resuming training from step 6000
175
+ wandb: [wandb.login()] Loaded credentials for https://api.wandb.ai from WANDB_API_KEY.
176
+ wandb: Currently logged in as: ruben-navasardyan4869 (ruben-navasardyan4869-epfl) to https://api.wandb.ai. Use `wandb login --relogin` to force relogin
177
+ wandb: Tracking run with wandb version 0.26.1
178
+ wandb: Run data is saved locally in /mloscratch/homes/navasard/sparse/SparseAtt/wandb/run-20260820_011124-zovy63wq
179
+ wandb: Run `wandb offline` to turn off syncing.
180
+ wandb: Syncing run l8k365m_dense_ropectl_a1p0_seed46
181
+ wandb: ⭐️ View project at https://wandb.ai/ruben-navasardyan4869-epfl/stochastic_sparsity
182
+ wandb: 🚀 View run at https://wandb.ai/ruben-navasardyan4869-epfl/stochastic_sparsity/runs/zovy63wq
183
+ Starting training...
184
+ Validation at step 6000: Loss = 3.2430, Perplexity = 25.61
185
+ Switching attention mode for step 6001: dense (dense step interval)
186
+ Step 6010/44000 | Loss: 3.2618 | LR: 4.82e-04 | 16291.9ms/step | 1.2 samples/s
187
+ Step 6020/44000 | Loss: 3.2425 | LR: 4.82e-04 | 1377.7ms/step | 14.5 samples/s
188
+ Step 6030/44000 | Loss: 3.2139 | LR: 4.82e-04 | 1381.2ms/step | 14.5 samples/s
189
+ Step 6040/44000 | Loss: 3.2182 | LR: 4.82e-04 | 1382.9ms/step | 14.5 samples/s
190
+ Step 6050/44000 | Loss: 3.2339 | LR: 4.82e-04 | 1385.9ms/step | 14.4 samples/s
191
+ Step 6060/44000 | Loss: 3.2441 | LR: 4.82e-04 | 1388.0ms/step | 14.4 samples/s
192
+ Step 6070/44000 | Loss: 3.2609 | LR: 4.82e-04 | 1387.1ms/step | 14.4 samples/s
193
+ Step 6080/44000 | Loss: 3.2565 | LR: 4.81e-04 | 1387.6ms/step | 14.4 samples/s
194
+ Step 6090/44000 | Loss: 3.2222 | LR: 4.81e-04 | 1385.2ms/step | 14.4 samples/s
195
+ Step 6100/44000 | Loss: 3.2387 | LR: 4.81e-04 | 1387.0ms/step | 14.4 samples/s
196
+ Step 6110/44000 | Loss: 3.2742 | LR: 4.81e-04 | 1387.2ms/step | 14.4 samples/s
197
+ Step 6120/44000 | Loss: 3.2595 | LR: 4.81e-04 | 1383.6ms/step | 14.5 samples/s
198
+ Step 6130/44000 | Loss: 3.2357 | LR: 4.81e-04 | 1388.5ms/step | 14.4 samples/s
199
+ Step 6140/44000 | Loss: 3.2400 | LR: 4.81e-04 | 1385.8ms/step | 14.4 samples/s
200
+ Step 6150/44000 | Loss: 3.2329 | LR: 4.81e-04 | 1387.5ms/step | 14.4 samples/s
201
+ Step 6160/44000 | Loss: 3.2459 | LR: 4.81e-04 | 1386.9ms/step | 14.4 samples/s
202
+ Step 6170/44000 | Loss: 3.2159 | LR: 4.81e-04 | 1390.4ms/step | 14.4 samples/s
203
+ Step 6180/44000 | Loss: 3.1839 | LR: 4.81e-04 | 1385.5ms/step | 14.4 samples/s
204
+ Step 6190/44000 | Loss: 3.2174 | LR: 4.81e-04 | 1388.0ms/step | 14.4 samples/s
205
+ Step 6200/44000 | Loss: 3.2140 | LR: 4.81e-04 | 1386.1ms/step | 14.4 samples/s
206
+ Step 6210/44000 | Loss: 3.2210 | LR: 4.81e-04 | 1386.4ms/step | 14.4 samples/s
207
+ Step 6220/44000 | Loss: 3.2578 | LR: 4.81e-04 | 1391.4ms/step | 14.4 samples/s
208
+ Step 6230/44000 | Loss: 3.2381 | LR: 4.80e-04 | 1388.4ms/step | 14.4 samples/s
209
+ Step 6240/44000 | Loss: 3.2345 | LR: 4.80e-04 | 1389.7ms/step | 14.4 samples/s
210
+ Step 6250/44000 | Loss: 3.2388 | LR: 4.80e-04 | 1389.9ms/step | 14.4 samples/s
211
+ Step 6260/44000 | Loss: 3.2367 | LR: 4.80e-04 | 1387.8ms/step | 14.4 samples/s
212
+ Step 6270/44000 | Loss: 3.2370 | LR: 4.80e-04 | 1388.7ms/step | 14.4 samples/s
213
+ Step 6280/44000 | Loss: 3.2396 | LR: 4.80e-04 | 1387.6ms/step | 14.4 samples/s
214
+ Step 6290/44000 | Loss: 3.1966 | LR: 4.80e-04 | 1388.5ms/step | 14.4 samples/s
215
+ Step 6300/44000 | Loss: 3.2441 | LR: 4.80e-04 | 1389.4ms/step | 14.4 samples/s
216
+ Step 6310/44000 | Loss: 3.2216 | LR: 4.80e-04 | 1386.5ms/step | 14.4 samples/s
217
+ Step 6320/44000 | Loss: 3.2480 | LR: 4.80e-04 | 1389.6ms/step | 14.4 samples/s
218
+ Step 6330/44000 | Loss: 3.2172 | LR: 4.80e-04 | 1388.0ms/step | 14.4 samples/s
219
+ Step 6340/44000 | Loss: 3.2368 | LR: 4.80e-04 | 1388.1ms/step | 14.4 samples/s
220
+ Step 6350/44000 | Loss: 3.2257 | LR: 4.80e-04 | 1389.8ms/step | 14.4 samples/s
221
+ Step 6360/44000 | Loss: 3.2329 | LR: 4.80e-04 | 1389.0ms/step | 14.4 samples/s
222
+ Step 6370/44000 | Loss: 3.2362 | LR: 4.79e-04 | 1389.1ms/step | 14.4 samples/s
223
+ Step 6380/44000 | Loss: 3.2422 | LR: 4.79e-04 | 1388.9ms/step | 14.4 samples/s
224
+ Step 6390/44000 | Loss: 3.2221 | LR: 4.79e-04 | 1390.9ms/step | 14.4 samples/s
225
+ Step 6400/44000 | Loss: 3.2560 | LR: 4.79e-04 | 1388.2ms/step | 14.4 samples/s
226
+ Step 6410/44000 | Loss: 3.1829 | LR: 4.79e-04 | 1387.9ms/step | 14.4 samples/s
227
+ Step 6420/44000 | Loss: 3.1887 | LR: 4.79e-04 | 1388.0ms/step | 14.4 samples/s
228
+ Step 6430/44000 | Loss: 3.2313 | LR: 4.79e-04 | 1388.1ms/step | 14.4 samples/s
229
+ Step 6440/44000 | Loss: 3.2047 | LR: 4.79e-04 | 1388.3ms/step | 14.4 samples/s
230
+ Step 6450/44000 | Loss: 3.2025 | LR: 4.79e-04 | 1387.3ms/step | 14.4 samples/s
231
+ Step 6460/44000 | Loss: 3.2089 | LR: 4.79e-04 | 1390.0ms/step | 14.4 samples/s
232
+ Step 6470/44000 | Loss: 3.2698 | LR: 4.79e-04 | 1385.6ms/step | 14.4 samples/s
233
+ Step 6480/44000 | Loss: 3.2303 | LR: 4.79e-04 | 1387.0ms/step | 14.4 samples/s
234
+ Step 6490/44000 | Loss: 3.2290 | LR: 4.79e-04 | 1386.0ms/step | 14.4 samples/s
235
+ Step 6500/44000 | Loss: 3.2570 | LR: 4.79e-04 | 1388.8ms/step | 14.4 samples/s
236
+ Step 6510/44000 | Loss: 3.1875 | LR: 4.78e-04 | 1390.1ms/step | 14.4 samples/s
237
+ Step 6520/44000 | Loss: 3.2040 | LR: 4.78e-04 | 1389.4ms/step | 14.4 samples/s
238
+ Step 6530/44000 | Loss: 3.2179 | LR: 4.78e-04 | 1390.4ms/step | 14.4 samples/s
239
+ Step 6540/44000 | Loss: 3.2192 | LR: 4.78e-04 | 1385.8ms/step | 14.4 samples/s
240
+ Step 6550/44000 | Loss: 3.2178 | LR: 4.78e-04 | 1385.4ms/step | 14.4 samples/s
241
+ Step 6560/44000 | Loss: 3.1976 | LR: 4.78e-04 | 1386.3ms/step | 14.4 samples/s
242
+ Step 6570/44000 | Loss: 3.2365 | LR: 4.78e-04 | 1387.2ms/step | 14.4 samples/s
243
+ Step 6580/44000 | Loss: 3.1924 | LR: 4.78e-04 | 1390.0ms/step | 14.4 samples/s
244
+ Step 6590/44000 | Loss: 3.2235 | LR: 4.78e-04 | 1390.6ms/step | 14.4 samples/s
245
+ Step 6600/44000 | Loss: 3.2035 | LR: 4.78e-04 | 1391.0ms/step | 14.4 samples/s
246
+ Step 6610/44000 | Loss: 3.2271 | LR: 4.78e-04 | 1390.2ms/step | 14.4 samples/s
247
+ Step 6620/44000 | Loss: 3.2095 | LR: 4.78e-04 | 1390.1ms/step | 14.4 samples/s
248
+ Step 6630/44000 | Loss: 3.1996 | LR: 4.78e-04 | 1387.9ms/step | 14.4 samples/s
249
+ Step 6640/44000 | Loss: 3.2154 | LR: 4.77e-04 | 1385.4ms/step | 14.4 samples/s
250
+ Step 6650/44000 | Loss: 3.1864 | LR: 4.77e-04 | 1386.2ms/step | 14.4 samples/s
251
+ Step 6660/44000 | Loss: 3.2061 | LR: 4.77e-04 | 1387.6ms/step | 14.4 samples/s
252
+ Step 6670/44000 | Loss: 3.2146 | LR: 4.77e-04 | 1388.0ms/step | 14.4 samples/s
253
+ Step 6680/44000 | Loss: 3.2490 | LR: 4.77e-04 | 1391.7ms/step | 14.4 samples/s
254
+ Step 6690/44000 | Loss: 3.2343 | LR: 4.77e-04 | 1387.8ms/step | 14.4 samples/s
255
+ Step 6700/44000 | Loss: 3.2315 | LR: 4.77e-04 | 1390.3ms/step | 14.4 samples/s
256
+ Step 6710/44000 | Loss: 3.1859 | LR: 4.77e-04 | 1389.8ms/step | 14.4 samples/s
257
+ Step 6720/44000 | Loss: 3.1985 | LR: 4.77e-04 | 1389.2ms/step | 14.4 samples/s
258
+ Step 6730/44000 | Loss: 3.2068 | LR: 4.77e-04 | 1390.5ms/step | 14.4 samples/s
259
+ Step 6740/44000 | Loss: 3.2252 | LR: 4.77e-04 | 1386.1ms/step | 14.4 samples/s
260
+ Step 6750/44000 | Loss: 3.2114 | LR: 4.77e-04 | 1386.0ms/step | 14.4 samples/s
261
+ Step 6760/44000 | Loss: 3.2105 | LR: 4.77e-04 | 1390.3ms/step | 14.4 samples/s
262
+ Step 6770/44000 | Loss: 3.1989 | LR: 4.76e-04 | 1387.6ms/step | 14.4 samples/s
263
+ Step 6780/44000 | Loss: 3.1956 | LR: 4.76e-04 | 1387.5ms/step | 14.4 samples/s
264
+ Step 6790/44000 | Loss: 3.2094 | LR: 4.76e-04 | 1389.0ms/step | 14.4 samples/s
265
+ Step 6800/44000 | Loss: 3.2156 | LR: 4.76e-04 | 1387.8ms/step | 14.4 samples/s
266
+ Step 6810/44000 | Loss: 3.2052 | LR: 4.76e-04 | 1386.4ms/step | 14.4 samples/s
267
+ Step 6820/44000 | Loss: 3.1752 | LR: 4.76e-04 | 1385.1ms/step | 14.4 samples/s
268
+ Step 6830/44000 | Loss: 3.1942 | LR: 4.76e-04 | 1390.5ms/step | 14.4 samples/s
269
+ Step 6840/44000 | Loss: 3.2224 | LR: 4.76e-04 | 1387.7ms/step | 14.4 samples/s
270
+ Step 6850/44000 | Loss: 3.1947 | LR: 4.76e-04 | 1387.6ms/step | 14.4 samples/s
271
+ Step 6860/44000 | Loss: 3.2130 | LR: 4.76e-04 | 1387.4ms/step | 14.4 samples/s
272
+ Step 6870/44000 | Loss: 3.2155 | LR: 4.76e-04 | 1387.6ms/step | 14.4 samples/s
273
+ Step 6880/44000 | Loss: 3.2202 | LR: 4.76e-04 | 1388.0ms/step | 14.4 samples/s
274
+ Step 6890/44000 | Loss: 3.1733 | LR: 4.76e-04 | 1385.4ms/step | 14.4 samples/s
275
+ Step 6900/44000 | Loss: 3.2030 | LR: 4.75e-04 | 1385.2ms/step | 14.4 samples/s
276
+ Step 6910/44000 | Loss: 3.1921 | LR: 4.75e-04 | 1388.9ms/step | 14.4 samples/s
277
+ Step 6920/44000 | Loss: 3.2234 | LR: 4.75e-04 | 1385.5ms/step | 14.4 samples/s
278
+ Step 6930/44000 | Loss: 3.2028 | LR: 4.75e-04 | 1386.7ms/step | 14.4 samples/s
279
+ Step 6940/44000 | Loss: 3.2089 | LR: 4.75e-04 | 1388.6ms/step | 14.4 samples/s
280
+ Step 6950/44000 | Loss: 3.1889 | LR: 4.75e-04 | 1386.6ms/step | 14.4 samples/s
281
+ Step 6960/44000 | Loss: 3.2114 | LR: 4.75e-04 | 1387.8ms/step | 14.4 samples/s
282
+ Step 6970/44000 | Loss: 3.2240 | LR: 4.75e-04 | 1387.2ms/step | 14.4 samples/s
283
+ Step 6980/44000 | Loss: 3.2464 | LR: 4.75e-04 | 1390.0ms/step | 14.4 samples/s
284
+ Step 6990/44000 | Loss: 3.1767 | LR: 4.75e-04 | 1388.5ms/step | 14.4 samples/s
285
+ Step 7000/44000 | Loss: 3.1913 | LR: 4.75e-04 | 1389.2ms/step | 14.4 samples/s
286
+ Validation at step 7000: Loss = 3.2028, Perplexity = 24.60
287
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735/checkpoint-7000
288
+ Step 7010/44000 | Loss: 3.2040 | LR: 4.75e-04 | 3866.4ms/step | 5.2 samples/s
289
+ Step 7020/44000 | Loss: 3.1819 | LR: 4.75e-04 | 1390.8ms/step | 14.4 samples/s
290
+ Step 7030/44000 | Loss: 3.1963 | LR: 4.74e-04 | 1393.1ms/step | 14.4 samples/s
291
+ Step 7040/44000 | Loss: 3.1890 | LR: 4.74e-04 | 1395.8ms/step | 14.3 samples/s
292
+ Step 7050/44000 | Loss: 3.2050 | LR: 4.74e-04 | 1396.0ms/step | 14.3 samples/s
293
+ Step 7060/44000 | Loss: 3.1915 | LR: 4.74e-04 | 1392.3ms/step | 14.4 samples/s
294
+ Step 7070/44000 | Loss: 3.1876 | LR: 4.74e-04 | 1396.1ms/step | 14.3 samples/s
295
+ Step 7080/44000 | Loss: 3.1768 | LR: 4.74e-04 | 1391.1ms/step | 14.4 samples/s
296
+ Step 7090/44000 | Loss: 3.2106 | LR: 4.74e-04 | 1389.5ms/step | 14.4 samples/s
297
+ Step 7100/44000 | Loss: 3.1740 | LR: 4.74e-04 | 1393.0ms/step | 14.4 samples/s
298
+ Step 7110/44000 | Loss: 3.1790 | LR: 4.74e-04 | 1392.3ms/step | 14.4 samples/s
299
+ Step 7120/44000 | Loss: 3.1790 | LR: 4.74e-04 | 1393.4ms/step | 14.4 samples/s
300
+ Step 7130/44000 | Loss: 3.2003 | LR: 4.74e-04 | 1396.6ms/step | 14.3 samples/s
301
+ Step 7140/44000 | Loss: 3.2084 | LR: 4.74e-04 | 1393.8ms/step | 14.3 samples/s
302
+ Step 7150/44000 | Loss: 3.2104 | LR: 4.73e-04 | 1394.2ms/step | 14.3 samples/s
303
+ Step 7160/44000 | Loss: 3.1920 | LR: 4.73e-04 | 1392.2ms/step | 14.4 samples/s
304
+ Step 7170/44000 | Loss: 3.1771 | LR: 4.73e-04 | 1397.6ms/step | 14.3 samples/s
305
+ Step 7180/44000 | Loss: 3.1761 | LR: 4.73e-04 | 1396.1ms/step | 14.3 samples/s
306
+ Step 7190/44000 | Loss: 3.1931 | LR: 4.73e-04 | 1394.8ms/step | 14.3 samples/s
307
+ Step 7200/44000 | Loss: 3.1981 | LR: 4.73e-04 | 1391.4ms/step | 14.4 samples/s
308
+ Step 7210/44000 | Loss: 3.1557 | LR: 4.73e-04 | 1390.5ms/step | 14.4 samples/s
309
+ Step 7220/44000 | Loss: 3.1993 | LR: 4.73e-04 | 1391.7ms/step | 14.4 samples/s
310
+ Step 7230/44000 | Loss: 3.1677 | LR: 4.73e-04 | 1393.9ms/step | 14.3 samples/s
311
+ Step 7240/44000 | Loss: 3.1904 | LR: 4.73e-04 | 1396.0ms/step | 14.3 samples/s
312
+ Step 7250/44000 | Loss: 3.2068 | LR: 4.73e-04 | 1398.0ms/step | 14.3 samples/s
313
+ Step 7260/44000 | Loss: 3.1958 | LR: 4.73e-04 | 1398.1ms/step | 14.3 samples/s
314
+ Step 7270/44000 | Loss: 3.1869 | LR: 4.72e-04 | 1393.7ms/step | 14.4 samples/s
315
+ Step 7280/44000 | Loss: 3.2069 | LR: 4.72e-04 | 1393.9ms/step | 14.3 samples/s
316
+ Step 7290/44000 | Loss: 3.1941 | LR: 4.72e-04 | 1392.5ms/step | 14.4 samples/s
317
+ Step 7300/44000 | Loss: 3.2220 | LR: 4.72e-04 | 1395.7ms/step | 14.3 samples/s
318
+ Step 7310/44000 | Loss: 3.1818 | LR: 4.72e-04 | 1396.2ms/step | 14.3 samples/s
319
+ Step 7320/44000 | Loss: 3.2174 | LR: 4.72e-04 | 1397.7ms/step | 14.3 samples/s
320
+ Step 7330/44000 | Loss: 3.1681 | LR: 4.72e-04 | 1394.2ms/step | 14.3 samples/s
321
+ Step 7340/44000 | Loss: 3.1631 | LR: 4.72e-04 | 1392.2ms/step | 14.4 samples/s
322
+ Step 7350/44000 | Loss: 3.1890 | LR: 4.72e-04 | 1391.7ms/step | 14.4 samples/s
323
+ Step 7360/44000 | Loss: 3.2176 | LR: 4.72e-04 | 1389.9ms/step | 14.4 samples/s
324
+ Step 7370/44000 | Loss: 3.1923 | LR: 4.72e-04 | 1388.9ms/step | 14.4 samples/s
325
+ Step 7380/44000 | Loss: 3.1718 | LR: 4.72e-04 | 1393.1ms/step | 14.4 samples/s
326
+ Step 7390/44000 | Loss: 3.2335 | LR: 4.71e-04 | 1394.1ms/step | 14.3 samples/s
327
+ Step 7400/44000 | Loss: 3.1794 | LR: 4.71e-04 | 1392.2ms/step | 14.4 samples/s
328
+ Step 7410/44000 | Loss: 3.1926 | LR: 4.71e-04 | 1395.4ms/step | 14.3 samples/s
329
+ Step 7420/44000 | Loss: 3.1617 | LR: 4.71e-04 | 1392.7ms/step | 14.4 samples/s
330
+ Step 7430/44000 | Loss: 3.1824 | LR: 4.71e-04 | 1395.9ms/step | 14.3 samples/s
331
+ Step 7440/44000 | Loss: 3.1942 | LR: 4.71e-04 | 1396.2ms/step | 14.3 samples/s
332
+ Step 7450/44000 | Loss: 3.1815 | LR: 4.71e-04 | 1400.2ms/step | 14.3 samples/s
333
+ Step 7460/44000 | Loss: 3.1918 | LR: 4.71e-04 | 1399.8ms/step | 14.3 samples/s
334
+ Step 7470/44000 | Loss: 3.1799 | LR: 4.71e-04 | 1400.1ms/step | 14.3 samples/s
335
+ Step 7480/44000 | Loss: 3.1565 | LR: 4.71e-04 | 1399.9ms/step | 14.3 samples/s
336
+ Step 7490/44000 | Loss: 3.1482 | LR: 4.71e-04 | 1400.1ms/step | 14.3 samples/s
337
+ Step 7500/44000 | Loss: 3.1661 | LR: 4.71e-04 | 1400.1ms/step | 14.3 samples/s
338
+ Step 7510/44000 | Loss: 3.2178 | LR: 4.70e-04 | 1399.9ms/step | 14.3 samples/s
339
+ Step 7520/44000 | Loss: 3.1731 | LR: 4.70e-04 | 1400.2ms/step | 14.3 samples/s
340
+ Step 7530/44000 | Loss: 3.1832 | LR: 4.70e-04 | 1400.0ms/step | 14.3 samples/s
341
+ Step 7540/44000 | Loss: 3.1903 | LR: 4.70e-04 | 1399.9ms/step | 14.3 samples/s
342
+ Step 7550/44000 | Loss: 3.1650 | LR: 4.70e-04 | 1399.7ms/step | 14.3 samples/s
343
+ Step 7560/44000 | Loss: 3.1794 | LR: 4.70e-04 | 1400.2ms/step | 14.3 samples/s
344
+ Step 7570/44000 | Loss: 3.1928 | LR: 4.70e-04 | 1399.7ms/step | 14.3 samples/s
345
+ Step 7580/44000 | Loss: 3.1445 | LR: 4.70e-04 | 1400.2ms/step | 14.3 samples/s
346
+ Step 7590/44000 | Loss: 3.2034 | LR: 4.70e-04 | 1399.9ms/step | 14.3 samples/s
347
+ Step 7600/44000 | Loss: 3.2297 | LR: 4.70e-04 | 1400.1ms/step | 14.3 samples/s
348
+ Step 7610/44000 | Loss: 3.1536 | LR: 4.70e-04 | 1400.1ms/step | 14.3 samples/s
349
+ Step 7620/44000 | Loss: 3.1957 | LR: 4.70e-04 | 1399.9ms/step | 14.3 samples/s
350
+ Step 7630/44000 | Loss: 3.1781 | LR: 4.69e-04 | 1399.9ms/step | 14.3 samples/s
351
+ Step 7640/44000 | Loss: 3.1881 | LR: 4.69e-04 | 1400.0ms/step | 14.3 samples/s
352
+ Step 7650/44000 | Loss: 3.1903 | LR: 4.69e-04 | 1399.9ms/step | 14.3 samples/s
353
+ Step 7660/44000 | Loss: 3.1644 | LR: 4.69e-04 | 1397.9ms/step | 14.3 samples/s
354
+ Step 7670/44000 | Loss: 3.1611 | LR: 4.69e-04 | 1398.2ms/step | 14.3 samples/s
355
+ Step 7680/44000 | Loss: 3.1664 | LR: 4.69e-04 | 1400.3ms/step | 14.3 samples/s
356
+ Step 7690/44000 | Loss: 3.1680 | LR: 4.69e-04 | 1399.8ms/step | 14.3 samples/s
357
+ Step 7700/44000 | Loss: 3.1474 | LR: 4.69e-04 | 1400.0ms/step | 14.3 samples/s
358
+ Step 7710/44000 | Loss: 3.1679 | LR: 4.69e-04 | 1400.0ms/step | 14.3 samples/s
359
+ Step 7720/44000 | Loss: 3.1584 | LR: 4.69e-04 | 1399.9ms/step | 14.3 samples/s
360
+ Step 7730/44000 | Loss: 3.1900 | LR: 4.69e-04 | 1400.2ms/step | 14.3 samples/s
361
+ Step 7740/44000 | Loss: 3.1768 | LR: 4.68e-04 | 1399.9ms/step | 14.3 samples/s
362
+ Step 7750/44000 | Loss: 3.1850 | LR: 4.68e-04 | 1399.4ms/step | 14.3 samples/s
363
+ Step 7760/44000 | Loss: 3.1794 | LR: 4.68e-04 | 1398.8ms/step | 14.3 samples/s
364
+ Step 7770/44000 | Loss: 3.1580 | LR: 4.68e-04 | 1399.9ms/step | 14.3 samples/s
365
+ Step 7780/44000 | Loss: 3.1712 | LR: 4.68e-04 | 1400.1ms/step | 14.3 samples/s
366
+ Step 7790/44000 | Loss: 3.1741 | LR: 4.68e-04 | 1400.0ms/step | 14.3 samples/s
367
+ Step 7800/44000 | Loss: 3.1699 | LR: 4.68e-04 | 1399.9ms/step | 14.3 samples/s
368
+ Step 7810/44000 | Loss: 3.1709 | LR: 4.68e-04 | 1400.2ms/step | 14.3 samples/s
369
+ Step 7820/44000 | Loss: 3.1865 | LR: 4.68e-04 | 1399.9ms/step | 14.3 samples/s
370
+ Step 7830/44000 | Loss: 3.1565 | LR: 4.68e-04 | 1400.1ms/step | 14.3 samples/s
371
+ Step 7840/44000 | Loss: 3.1769 | LR: 4.68e-04 | 1398.0ms/step | 14.3 samples/s
372
+ Step 7850/44000 | Loss: 3.1616 | LR: 4.67e-04 | 1397.8ms/step | 14.3 samples/s
373
+ Step 7860/44000 | Loss: 3.1730 | LR: 4.67e-04 | 1400.2ms/step | 14.3 samples/s
374
+ Step 7870/44000 | Loss: 3.1792 | LR: 4.67e-04 | 1399.9ms/step | 14.3 samples/s
375
+ Step 7880/44000 | Loss: 3.1620 | LR: 4.67e-04 | 1400.0ms/step | 14.3 samples/s
376
+ Step 7890/44000 | Loss: 3.1716 | LR: 4.67e-04 | 1400.1ms/step | 14.3 samples/s
377
+ Step 7900/44000 | Loss: 3.1890 | LR: 4.67e-04 | 1400.1ms/step | 14.3 samples/s
378
+ Step 7910/44000 | Loss: 3.1509 | LR: 4.67e-04 | 1399.9ms/step | 14.3 samples/s
379
+ Step 7920/44000 | Loss: 3.1629 | LR: 4.67e-04 | 1400.0ms/step | 14.3 samples/s
380
+ Step 7930/44000 | Loss: 3.1675 | LR: 4.67e-04 | 1399.7ms/step | 14.3 samples/s
381
+ Step 7940/44000 | Loss: 3.1713 | LR: 4.67e-04 | 1400.2ms/step | 14.3 samples/s
382
+ Step 7950/44000 | Loss: 3.2018 | LR: 4.67e-04 | 1399.9ms/step | 14.3 samples/s
383
+ Step 7960/44000 | Loss: 3.1385 | LR: 4.66e-04 | 1400.1ms/step | 14.3 samples/s
384
+ Step 7970/44000 | Loss: 3.1782 | LR: 4.66e-04 | 1400.0ms/step | 14.3 samples/s
385
+ Step 7980/44000 | Loss: 3.1577 | LR: 4.66e-04 | 1400.0ms/step | 14.3 samples/s
386
+ Step 7990/44000 | Loss: 3.1700 | LR: 4.66e-04 | 1400.2ms/step | 14.3 samples/s
387
+ Step 8000/44000 | Loss: 3.1558 | LR: 4.66e-04 | 1399.8ms/step | 14.3 samples/s
388
+ Validation at step 8000: Loss = 3.1684, Perplexity = 23.77
389
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735/checkpoint-8000
390
+ Step 8010/44000 | Loss: 3.1654 | LR: 4.66e-04 | 3902.0ms/step | 5.1 samples/s
391
+ Step 8020/44000 | Loss: 3.1580 | LR: 4.66e-04 | 1400.1ms/step | 14.3 samples/s
392
+ Step 8030/44000 | Loss: 3.1808 | LR: 4.66e-04 | 1400.0ms/step | 14.3 samples/s
393
+ Step 8040/44000 | Loss: 3.1810 | LR: 4.66e-04 | 1399.8ms/step | 14.3 samples/s
394
+ Step 8050/44000 | Loss: 3.1900 | LR: 4.66e-04 | 1400.2ms/step | 14.3 samples/s
395
+ Step 8060/44000 | Loss: 3.1520 | LR: 4.66e-04 | 1400.0ms/step | 14.3 samples/s
396
+ Step 8070/44000 | Loss: 3.1502 | LR: 4.65e-04 | 1399.8ms/step | 14.3 samples/s
397
+ Step 8080/44000 | Loss: 3.1666 | LR: 4.65e-04 | 1400.1ms/step | 14.3 samples/s
398
+ Step 8090/44000 | Loss: 3.1424 | LR: 4.65e-04 | 1400.2ms/step | 14.3 samples/s
399
+ Step 8100/44000 | Loss: 3.1585 | LR: 4.65e-04 | 1400.1ms/step | 14.3 samples/s
400
+ Step 8110/44000 | Loss: 3.1653 | LR: 4.65e-04 | 1399.9ms/step | 14.3 samples/s
401
+ Step 8120/44000 | Loss: 3.1503 | LR: 4.65e-04 | 1399.8ms/step | 14.3 samples/s
402
+ Step 8130/44000 | Loss: 3.1493 | LR: 4.65e-04 | 1400.1ms/step | 14.3 samples/s
403
+ Step 8140/44000 | Loss: 3.1413 | LR: 4.65e-04 | 1400.1ms/step | 14.3 samples/s
404
+ Step 8150/44000 | Loss: 3.1972 | LR: 4.65e-04 | 1400.1ms/step | 14.3 samples/s
405
+ Step 8160/44000 | Loss: 3.1883 | LR: 4.65e-04 | 1400.0ms/step | 14.3 samples/s
406
+ Step 8170/44000 | Loss: 3.1416 | LR: 4.65e-04 | 1399.9ms/step | 14.3 samples/s
407
+ Step 8180/44000 | Loss: 3.1431 | LR: 4.64e-04 | 1400.1ms/step | 14.3 samples/s
408
+ Step 8190/44000 | Loss: 3.1569 | LR: 4.64e-04 | 1399.8ms/step | 14.3 samples/s
409
+ Step 8200/44000 | Loss: 3.1633 | LR: 4.64e-04 | 1400.1ms/step | 14.3 samples/s
410
+ Step 8210/44000 | Loss: 3.1464 | LR: 4.64e-04 | 1400.1ms/step | 14.3 samples/s
411
+ Step 8220/44000 | Loss: 3.1618 | LR: 4.64e-04 | 1400.0ms/step | 14.3 samples/s
412
+ Step 8230/44000 | Loss: 3.1589 | LR: 4.64e-04 | 1400.0ms/step | 14.3 samples/s
413
+ Step 8240/44000 | Loss: 3.1483 | LR: 4.64e-04 | 1399.7ms/step | 14.3 samples/s
414
+ Step 8250/44000 | Loss: 3.1790 | LR: 4.64e-04 | 1400.4ms/step | 14.3 samples/s
415
+ Step 8260/44000 | Loss: 3.1741 | LR: 4.64e-04 | 1399.9ms/step | 14.3 samples/s
416
+ Step 8270/44000 | Loss: 3.2039 | LR: 4.64e-04 | 1400.0ms/step | 14.3 samples/s
417
+ Step 8280/44000 | Loss: 3.1456 | LR: 4.64e-04 | 1400.0ms/step | 14.3 samples/s
418
+ Step 8290/44000 | Loss: 3.1591 | LR: 4.63e-04 | 1400.1ms/step | 14.3 samples/s
419
+ Step 8300/44000 | Loss: 3.1783 | LR: 4.63e-04 | 1399.7ms/step | 14.3 samples/s
420
+ Step 8310/44000 | Loss: 3.1482 | LR: 4.63e-04 | 1400.2ms/step | 14.3 samples/s
421
+ Step 8320/44000 | Loss: 3.1596 | LR: 4.63e-04 | 1400.1ms/step | 14.3 samples/s
422
+ Step 8330/44000 | Loss: 3.1674 | LR: 4.63e-04 | 1399.8ms/step | 14.3 samples/s
423
+ Step 8340/44000 | Loss: 3.1516 | LR: 4.63e-04 | 1400.2ms/step | 14.3 samples/s
424
+ Step 8350/44000 | Loss: 3.1934 | LR: 4.63e-04 | 1399.9ms/step | 14.3 samples/s
425
+ Step 8360/44000 | Loss: 3.1542 | LR: 4.63e-04 | 1400.1ms/step | 14.3 samples/s
426
+ Step 8370/44000 | Loss: 3.1270 | LR: 4.63e-04 | 1399.8ms/step | 14.3 samples/s
427
+ Step 8380/44000 | Loss: 3.1499 | LR: 4.63e-04 | 1400.0ms/step | 14.3 samples/s
428
+ Step 8390/44000 | Loss: 3.1700 | LR: 4.62e-04 | 1399.9ms/step | 14.3 samples/s
429
+ Step 8400/44000 | Loss: 3.1795 | LR: 4.62e-04 | 1400.0ms/step | 14.3 samples/s
430
+ Step 8410/44000 | Loss: 3.1382 | LR: 4.62e-04 | 1400.1ms/step | 14.3 samples/s
431
+ Step 8420/44000 | Loss: 3.1290 | LR: 4.62e-04 | 1400.1ms/step | 14.3 samples/s
432
+ Step 8430/44000 | Loss: 3.2009 | LR: 4.62e-04 | 1400.1ms/step | 14.3 samples/s
433
+ Step 8440/44000 | Loss: 3.1520 | LR: 4.62e-04 | 1400.0ms/step | 14.3 samples/s
434
+ Step 8450/44000 | Loss: 3.1658 | LR: 4.62e-04 | 1399.9ms/step | 14.3 samples/s
435
+ Step 8460/44000 | Loss: 3.1397 | LR: 4.62e-04 | 1400.1ms/step | 14.3 samples/s
436
+ Step 8470/44000 | Loss: 3.1342 | LR: 4.62e-04 | 1400.0ms/step | 14.3 samples/s
437
+ Step 8480/44000 | Loss: 3.1290 | LR: 4.62e-04 | 1399.9ms/step | 14.3 samples/s
438
+ Step 8490/44000 | Loss: 3.1448 | LR: 4.62e-04 | 1399.9ms/step | 14.3 samples/s
439
+ Step 8500/44000 | Loss: 3.1484 | LR: 4.61e-04 | 1400.1ms/step | 14.3 samples/s
440
+ Step 8510/44000 | Loss: 3.1634 | LR: 4.61e-04 | 1400.0ms/step | 14.3 samples/s
441
+ Step 8520/44000 | Loss: 3.1307 | LR: 4.61e-04 | 1399.9ms/step | 14.3 samples/s
442
+ Step 8530/44000 | Loss: 3.1476 | LR: 4.61e-04 | 1400.2ms/step | 14.3 samples/s
443
+ Step 8540/44000 | Loss: 3.1646 | LR: 4.61e-04 | 1400.0ms/step | 14.3 samples/s
444
+ Step 8550/44000 | Loss: 3.1357 | LR: 4.61e-04 | 1399.7ms/step | 14.3 samples/s
445
+ Step 8560/44000 | Loss: 3.1662 | LR: 4.61e-04 | 1400.2ms/step | 14.3 samples/s
446
+ Step 8570/44000 | Loss: 3.1148 | LR: 4.61e-04 | 1400.1ms/step | 14.3 samples/s
447
+ Step 8580/44000 | Loss: 3.1349 | LR: 4.61e-04 | 1399.8ms/step | 14.3 samples/s
448
+ Step 8590/44000 | Loss: 3.1359 | LR: 4.61e-04 | 1400.3ms/step | 14.3 samples/s
449
+ Step 8600/44000 | Loss: 3.1807 | LR: 4.60e-04 | 1399.9ms/step | 14.3 samples/s
450
+ Step 8610/44000 | Loss: 3.1706 | LR: 4.60e-04 | 1400.0ms/step | 14.3 samples/s
451
+ Step 8620/44000 | Loss: 3.1686 | LR: 4.60e-04 | 1400.0ms/step | 14.3 samples/s
452
+ Step 8630/44000 | Loss: 3.1545 | LR: 4.60e-04 | 1399.9ms/step | 14.3 samples/s
453
+ Step 8640/44000 | Loss: 3.1465 | LR: 4.60e-04 | 1400.0ms/step | 14.3 samples/s
454
+ Step 8650/44000 | Loss: 3.1621 | LR: 4.60e-04 | 1400.2ms/step | 14.3 samples/s
455
+ Step 8660/44000 | Loss: 3.1431 | LR: 4.60e-04 | 1399.9ms/step | 14.3 samples/s
456
+ Step 8670/44000 | Loss: 3.1478 | LR: 4.60e-04 | 1399.9ms/step | 14.3 samples/s
457
+ Step 8680/44000 | Loss: 3.1419 | LR: 4.60e-04 | 1399.9ms/step | 14.3 samples/s
458
+ Step 8690/44000 | Loss: 3.1386 | LR: 4.60e-04 | 1399.9ms/step | 14.3 samples/s
459
+ Step 8700/44000 | Loss: 3.1100 | LR: 4.59e-04 | 1400.1ms/step | 14.3 samples/s
460
+ Step 8710/44000 | Loss: 3.1617 | LR: 4.59e-04 | 1400.1ms/step | 14.3 samples/s
461
+ Step 8720/44000 | Loss: 3.1332 | LR: 4.59e-04 | 1400.1ms/step | 14.3 samples/s
462
+ Step 8730/44000 | Loss: 3.1659 | LR: 4.59e-04 | 1399.8ms/step | 14.3 samples/s
463
+ Step 8740/44000 | Loss: 3.1621 | LR: 4.59e-04 | 1400.3ms/step | 14.3 samples/s
464
+ Step 8750/44000 | Loss: 3.1659 | LR: 4.59e-04 | 1399.9ms/step | 14.3 samples/s
465
+ Step 8760/44000 | Loss: 3.1589 | LR: 4.59e-04 | 1399.9ms/step | 14.3 samples/s
466
+ Step 8770/44000 | Loss: 3.1547 | LR: 4.59e-04 | 1397.9ms/step | 14.3 samples/s
467
+ Step 8780/44000 | Loss: 3.1419 | LR: 4.59e-04 | 1399.9ms/step | 14.3 samples/s
468
+ Step 8790/44000 | Loss: 3.1589 | LR: 4.59e-04 | 1400.1ms/step | 14.3 samples/s
469
+ Step 8800/44000 | Loss: 3.1637 | LR: 4.58e-04 | 1400.2ms/step | 14.3 samples/s
470
+ Step 8810/44000 | Loss: 3.1420 | LR: 4.58e-04 | 1400.1ms/step | 14.3 samples/s
471
+ Step 8820/44000 | Loss: 3.1474 | LR: 4.58e-04 | 1399.8ms/step | 14.3 samples/s
472
+ Step 8830/44000 | Loss: 3.1160 | LR: 4.58e-04 | 1400.0ms/step | 14.3 samples/s
473
+ Step 8840/44000 | Loss: 3.1848 | LR: 4.58e-04 | 1400.0ms/step | 14.3 samples/s
474
+ Step 8850/44000 | Loss: 3.1428 | LR: 4.58e-04 | 1400.1ms/step | 14.3 samples/s
475
+ Step 8860/44000 | Loss: 3.1632 | LR: 4.58e-04 | 1399.9ms/step | 14.3 samples/s
476
+ Step 8870/44000 | Loss: 3.1569 | LR: 4.58e-04 | 1400.1ms/step | 14.3 samples/s
477
+ Step 8880/44000 | Loss: 3.1494 | LR: 4.58e-04 | 1400.0ms/step | 14.3 samples/s
478
+ Step 8890/44000 | Loss: 3.1077 | LR: 4.58e-04 | 1400.0ms/step | 14.3 samples/s
479
+ Step 8900/44000 | Loss: 3.1492 | LR: 4.57e-04 | 1399.9ms/step | 14.3 samples/s
480
+ Step 8910/44000 | Loss: 3.1355 | LR: 4.57e-04 | 1400.0ms/step | 14.3 samples/s
481
+ Step 8920/44000 | Loss: 3.1555 | LR: 4.57e-04 | 1400.0ms/step | 14.3 samples/s
482
+ Step 8930/44000 | Loss: 3.1481 | LR: 4.57e-04 | 1400.2ms/step | 14.3 samples/s
483
+ Step 8940/44000 | Loss: 3.1551 | LR: 4.57e-04 | 1400.0ms/step | 14.3 samples/s
484
+ Step 8950/44000 | Loss: 3.1164 | LR: 4.57e-04 | 1400.0ms/step | 14.3 samples/s
485
+ Step 8960/44000 | Loss: 3.1534 | LR: 4.57e-04 | 1399.8ms/step | 14.3 samples/s
486
+ Step 8970/44000 | Loss: 3.1250 | LR: 4.57e-04 | 1400.1ms/step | 14.3 samples/s
487
+ Step 8980/44000 | Loss: 3.1240 | LR: 4.57e-04 | 1399.9ms/step | 14.3 samples/s
488
+ Step 8990/44000 | Loss: 3.0993 | LR: 4.57e-04 | 1400.1ms/step | 14.3 samples/s
489
+ Step 9000/44000 | Loss: 3.1222 | LR: 4.56e-04 | 1400.0ms/step | 14.3 samples/s
490
+ Validation at step 9000: Loss = 3.1415, Perplexity = 23.14
491
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735/checkpoint-9000
492
+ Step 9010/44000 | Loss: 3.1551 | LR: 4.56e-04 | 3919.9ms/step | 5.1 samples/s
493
+ Step 9020/44000 | Loss: 3.1298 | LR: 4.56e-04 | 1399.9ms/step | 14.3 samples/s
494
+ Step 9030/44000 | Loss: 3.1637 | LR: 4.56e-04 | 1396.1ms/step | 14.3 samples/s
495
+ Step 9040/44000 | Loss: 3.1270 | LR: 4.56e-04 | 1400.0ms/step | 14.3 samples/s
496
+ Step 9050/44000 | Loss: 3.1374 | LR: 4.56e-04 | 1399.8ms/step | 14.3 samples/s
497
+ Step 9060/44000 | Loss: 3.1588 | LR: 4.56e-04 | 1400.3ms/step | 14.3 samples/s
498
+ Step 9070/44000 | Loss: 3.1683 | LR: 4.56e-04 | 1400.2ms/step | 14.3 samples/s
499
+ Step 9080/44000 | Loss: 3.1372 | LR: 4.56e-04 | 1399.9ms/step | 14.3 samples/s
500
+ Step 9090/44000 | Loss: 3.1291 | LR: 4.56e-04 | 1399.9ms/step | 14.3 samples/s
501
+ Step 9100/44000 | Loss: 3.1074 | LR: 4.55e-04 | 1399.8ms/step | 14.3 samples/s
502
+ Step 9110/44000 | Loss: 3.1548 | LR: 4.55e-04 | 1400.4ms/step | 14.3 samples/s
503
+ Step 9120/44000 | Loss: 3.1314 | LR: 4.55e-04 | 1399.9ms/step | 14.3 samples/s
504
+ Step 9130/44000 | Loss: 3.1311 | LR: 4.55e-04 | 1400.0ms/step | 14.3 samples/s
505
+ Step 9140/44000 | Loss: 3.1489 | LR: 4.55e-04 | 1400.0ms/step | 14.3 samples/s
506
+ Step 9150/44000 | Loss: 3.1381 | LR: 4.55e-04 | 1400.0ms/step | 14.3 samples/s
507
+ Step 9160/44000 | Loss: 3.1213 | LR: 4.55e-04 | 1400.1ms/step | 14.3 samples/s
508
+ Step 9170/44000 | Loss: 3.1483 | LR: 4.55e-04 | 1400.2ms/step | 14.3 samples/s
509
+ Step 9180/44000 | Loss: 3.1596 | LR: 4.55e-04 | 1400.0ms/step | 14.3 samples/s
510
+ Step 9190/44000 | Loss: 3.1291 | LR: 4.54e-04 | 1399.9ms/step | 14.3 samples/s
511
+ Step 9200/44000 | Loss: 3.1360 | LR: 4.54e-04 | 1400.2ms/step | 14.3 samples/s
512
+ Step 9210/44000 | Loss: 3.1144 | LR: 4.54e-04 | 1400.0ms/step | 14.3 samples/s
513
+ Step 9220/44000 | Loss: 3.1538 | LR: 4.54e-04 | 1399.8ms/step | 14.3 samples/s
514
+ Step 9230/44000 | Loss: 3.1140 | LR: 4.54e-04 | 1400.1ms/step | 14.3 samples/s
515
+ Step 9240/44000 | Loss: 3.1497 | LR: 4.54e-04 | 1399.8ms/step | 14.3 samples/s
516
+ Step 9250/44000 | Loss: 3.1358 | LR: 4.54e-04 | 1400.1ms/step | 14.3 samples/s
517
+ Step 9260/44000 | Loss: 3.1301 | LR: 4.54e-04 | 1400.0ms/step | 14.3 samples/s
518
+ Step 9270/44000 | Loss: 3.1458 | LR: 4.54e-04 | 1400.2ms/step | 14.3 samples/s
519
+ Step 9280/44000 | Loss: 3.1116 | LR: 4.54e-04 | 1399.9ms/step | 14.3 samples/s
520
+ Step 9290/44000 | Loss: 3.1434 | LR: 4.53e-04 | 1400.2ms/step | 14.3 samples/s
521
+ Step 9300/44000 | Loss: 3.1292 | LR: 4.53e-04 | 1399.8ms/step | 14.3 samples/s
522
+ Step 9310/44000 | Loss: 3.0980 | LR: 4.53e-04 | 1400.2ms/step | 14.3 samples/s
523
+ Step 9320/44000 | Loss: 3.1430 | LR: 4.53e-04 | 1399.6ms/step | 14.3 samples/s
524
+ Step 9330/44000 | Loss: 3.1176 | LR: 4.53e-04 | 1400.2ms/step | 14.3 samples/s
525
+ Step 9340/44000 | Loss: 3.0990 | LR: 4.53e-04 | 1400.1ms/step | 14.3 samples/s
526
+ Step 9350/44000 | Loss: 3.1704 | LR: 4.53e-04 | 1400.1ms/step | 14.3 samples/s
527
+ Step 9360/44000 | Loss: 3.1231 | LR: 4.53e-04 | 1399.7ms/step | 14.3 samples/s
528
+ Step 9370/44000 | Loss: 3.1344 | LR: 4.53e-04 | 1400.3ms/step | 14.3 samples/s
529
+ Step 9380/44000 | Loss: 3.1209 | LR: 4.52e-04 | 1399.8ms/step | 14.3 samples/s
530
+ Step 9390/44000 | Loss: 3.1524 | LR: 4.52e-04 | 1400.0ms/step | 14.3 samples/s
531
+ Step 9400/44000 | Loss: 3.1279 | LR: 4.52e-04 | 1399.9ms/step | 14.3 samples/s
532
+ Step 9410/44000 | Loss: 3.1209 | LR: 4.52e-04 | 1400.1ms/step | 14.3 samples/s
533
+ Step 9420/44000 | Loss: 3.1330 | LR: 4.52e-04 | 1399.8ms/step | 14.3 samples/s
534
+ Step 9430/44000 | Loss: 3.1219 | LR: 4.52e-04 | 1400.4ms/step | 14.3 samples/s
535
+ Step 9440/44000 | Loss: 3.1077 | LR: 4.52e-04 | 1399.5ms/step | 14.3 samples/s
536
+ Step 9450/44000 | Loss: 3.1240 | LR: 4.52e-04 | 1400.3ms/step | 14.3 samples/s
537
+ Step 9460/44000 | Loss: 3.1225 | LR: 4.52e-04 | 1400.0ms/step | 14.3 samples/s
538
+ Step 9470/44000 | Loss: 3.1452 | LR: 4.52e-04 | 1400.0ms/step | 14.3 samples/s
539
+ Step 9480/44000 | Loss: 3.1492 | LR: 4.51e-04 | 1399.3ms/step | 14.3 samples/s
540
+ Step 9490/44000 | Loss: 3.1444 | LR: 4.51e-04 | 1398.7ms/step | 14.3 samples/s
541
+ Step 9500/44000 | Loss: 3.1237 | LR: 4.51e-04 | 1399.9ms/step | 14.3 samples/s
542
+ Step 9510/44000 | Loss: 3.1275 | LR: 4.51e-04 | 1400.2ms/step | 14.3 samples/s
543
+ Step 9520/44000 | Loss: 3.1373 | LR: 4.51e-04 | 1400.2ms/step | 14.3 samples/s
544
+ Step 9530/44000 | Loss: 3.1049 | LR: 4.51e-04 | 1399.8ms/step | 14.3 samples/s
545
+ Step 9540/44000 | Loss: 3.1086 | LR: 4.51e-04 | 1400.0ms/step | 14.3 samples/s
546
+ Step 9550/44000 | Loss: 3.1266 | LR: 4.51e-04 | 1399.9ms/step | 14.3 samples/s
547
+ Step 9560/44000 | Loss: 3.1335 | LR: 4.51e-04 | 1400.3ms/step | 14.3 samples/s
548
+ Step 9570/44000 | Loss: 3.1237 | LR: 4.50e-04 | 1400.1ms/step | 14.3 samples/s
549
+ Step 9580/44000 | Loss: 3.1317 | LR: 4.50e-04 | 1399.5ms/step | 14.3 samples/s
550
+ Step 9590/44000 | Loss: 3.1340 | LR: 4.50e-04 | 1394.2ms/step | 14.3 samples/s
551
+ Step 9600/44000 | Loss: 3.1433 | LR: 4.50e-04 | 1400.0ms/step | 14.3 samples/s
552
+ Step 9610/44000 | Loss: 3.1162 | LR: 4.50e-04 | 1400.0ms/step | 14.3 samples/s
553
+ Step 9620/44000 | Loss: 3.1359 | LR: 4.50e-04 | 1400.2ms/step | 14.3 samples/s
554
+ Step 9630/44000 | Loss: 3.1458 | LR: 4.50e-04 | 1397.1ms/step | 14.3 samples/s
555
+ Step 9640/44000 | Loss: 3.1546 | LR: 4.50e-04 | 1392.8ms/step | 14.4 samples/s
556
+ Step 9650/44000 | Loss: 3.1210 | LR: 4.50e-04 | 1393.6ms/step | 14.4 samples/s
557
+ Step 9660/44000 | Loss: 3.1251 | LR: 4.49e-04 | 1396.3ms/step | 14.3 samples/s
558
+ Step 9670/44000 | Loss: 3.1340 | LR: 4.49e-04 | 1392.0ms/step | 14.4 samples/s
559
+ Step 9680/44000 | Loss: 3.1295 | LR: 4.49e-04 | 1392.6ms/step | 14.4 samples/s
560
+ Step 9690/44000 | Loss: 3.1212 | LR: 4.49e-04 | 1395.4ms/step | 14.3 samples/s
561
+ Step 9700/44000 | Loss: 3.1138 | LR: 4.49e-04 | 1392.2ms/step | 14.4 samples/s
562
+ Step 9710/44000 | Loss: 3.1560 | LR: 4.49e-04 | 1393.9ms/step | 14.3 samples/s
563
+ Step 9720/44000 | Loss: 3.1366 | LR: 4.49e-04 | 1391.7ms/step | 14.4 samples/s
564
+ Step 9730/44000 | Loss: 3.0881 | LR: 4.49e-04 | 1390.4ms/step | 14.4 samples/s
565
+ Step 9740/44000 | Loss: 3.1071 | LR: 4.49e-04 | 1395.0ms/step | 14.3 samples/s
566
+ Step 9750/44000 | Loss: 3.1136 | LR: 4.48e-04 | 1391.2ms/step | 14.4 samples/s
567
+ Step 9760/44000 | Loss: 3.1213 | LR: 4.48e-04 | 1393.6ms/step | 14.4 samples/s
568
+ Step 9770/44000 | Loss: 3.1276 | LR: 4.48e-04 | 1395.0ms/step | 14.3 samples/s
569
+ Step 9780/44000 | Loss: 3.1037 | LR: 4.48e-04 | 1393.0ms/step | 14.4 samples/s
570
+ Step 9790/44000 | Loss: 3.1279 | LR: 4.48e-04 | 1390.1ms/step | 14.4 samples/s
571
+ Step 9800/44000 | Loss: 3.1543 | LR: 4.48e-04 | 1395.8ms/step | 14.3 samples/s
572
+ Step 9810/44000 | Loss: 3.1072 | LR: 4.48e-04 | 1392.3ms/step | 14.4 samples/s
573
+ Step 9820/44000 | Loss: 3.0958 | LR: 4.48e-04 | 1389.7ms/step | 14.4 samples/s
574
+ Step 9830/44000 | Loss: 3.1120 | LR: 4.48e-04 | 1388.4ms/step | 14.4 samples/s
575
+ Step 9840/44000 | Loss: 3.1082 | LR: 4.47e-04 | 1390.0ms/step | 14.4 samples/s
576
+ Step 9850/44000 | Loss: 3.1178 | LR: 4.47e-04 | 1393.8ms/step | 14.3 samples/s
577
+ Step 9860/44000 | Loss: 3.1152 | LR: 4.47e-04 | 1390.2ms/step | 14.4 samples/s
578
+ Step 9870/44000 | Loss: 3.1118 | LR: 4.47e-04 | 1389.7ms/step | 14.4 samples/s
579
+ Step 9880/44000 | Loss: 3.1113 | LR: 4.47e-04 | 1392.4ms/step | 14.4 samples/s
580
+ Step 9890/44000 | Loss: 3.1419 | LR: 4.47e-04 | 1393.6ms/step | 14.4 samples/s
581
+ Step 9900/44000 | Loss: 3.1243 | LR: 4.47e-04 | 1390.1ms/step | 14.4 samples/s
582
+ Step 9910/44000 | Loss: 3.0955 | LR: 4.47e-04 | 1392.2ms/step | 14.4 samples/s
583
+ Step 9920/44000 | Loss: 3.0859 | LR: 4.47e-04 | 1388.9ms/step | 14.4 samples/s
584
+ Step 9930/44000 | Loss: 3.0977 | LR: 4.46e-04 | 1391.9ms/step | 14.4 samples/s
585
+ Step 9940/44000 | Loss: 3.1274 | LR: 4.46e-04 | 1393.5ms/step | 14.4 samples/s
586
+ Step 9950/44000 | Loss: 3.1192 | LR: 4.46e-04 | 1389.9ms/step | 14.4 samples/s
587
+ Step 9960/44000 | Loss: 3.1154 | LR: 4.46e-04 | 1389.9ms/step | 14.4 samples/s
588
+ Step 9970/44000 | Loss: 3.0883 | LR: 4.46e-04 | 1391.7ms/step | 14.4 samples/s
589
+ Step 9980/44000 | Loss: 3.1335 | LR: 4.46e-04 | 1393.1ms/step | 14.4 samples/s
590
+ Step 9990/44000 | Loss: 3.1216 | LR: 4.46e-04 | 1393.0ms/step | 14.4 samples/s
591
+ Step 10000/44000 | Loss: 3.1213 | LR: 4.46e-04 | 1392.1ms/step | 14.4 samples/s
592
+ Validation at step 10000: Loss = 3.1163, Perplexity = 22.56
593
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735/checkpoint-10000
594
+ Step 10010/44000 | Loss: 3.1047 | LR: 4.46e-04 | 3868.3ms/step | 5.2 samples/s
595
+ Step 10020/44000 | Loss: 3.1035 | LR: 4.45e-04 | 1391.9ms/step | 14.4 samples/s
596
+ Step 10030/44000 | Loss: 3.1323 | LR: 4.45e-04 | 1389.3ms/step | 14.4 samples/s
597
+ Step 10040/44000 | Loss: 3.1013 | LR: 4.45e-04 | 1387.5ms/step | 14.4 samples/s
598
+ Step 10050/44000 | Loss: 3.1129 | LR: 4.45e-04 | 1389.4ms/step | 14.4 samples/s
599
+ Step 10060/44000 | Loss: 3.1054 | LR: 4.45e-04 | 1390.2ms/step | 14.4 samples/s
600
+ Step 10070/44000 | Loss: 3.0999 | LR: 4.45e-04 | 1394.7ms/step | 14.3 samples/s
601
+ Step 10080/44000 | Loss: 3.1198 | LR: 4.45e-04 | 1388.9ms/step | 14.4 samples/s
602
+ Step 10090/44000 | Loss: 3.1138 | LR: 4.45e-04 | 1392.2ms/step | 14.4 samples/s
603
+ Step 10100/44000 | Loss: 3.0917 | LR: 4.45e-04 | 1392.9ms/step | 14.4 samples/s
604
+ Step 10110/44000 | Loss: 3.1124 | LR: 4.44e-04 | 1394.9ms/step | 14.3 samples/s
605
+ Step 10120/44000 | Loss: 3.1201 | LR: 4.44e-04 | 1400.1ms/step | 14.3 samples/s
606
+ Step 10130/44000 | Loss: 3.0910 | LR: 4.44e-04 | 1395.7ms/step | 14.3 samples/s
607
+ Step 10140/44000 | Loss: 3.0957 | LR: 4.44e-04 | 1396.1ms/step | 14.3 samples/s
608
+ Step 10150/44000 | Loss: 3.1162 | LR: 4.44e-04 | 1394.2ms/step | 14.3 samples/s
609
+ Step 10160/44000 | Loss: 3.1453 | LR: 4.44e-04 | 1398.0ms/step | 14.3 samples/s
610
+ Step 10170/44000 | Loss: 3.1228 | LR: 4.44e-04 | 1391.9ms/step | 14.4 samples/s
611
+ Step 10180/44000 | Loss: 3.1130 | LR: 4.44e-04 | 1394.0ms/step | 14.3 samples/s
612
+ Step 10190/44000 | Loss: 3.1251 | LR: 4.43e-04 | 1395.6ms/step | 14.3 samples/s
613
+ Step 10200/44000 | Loss: 3.1176 | LR: 4.43e-04 | 1392.4ms/step | 14.4 samples/s
614
+ Step 10210/44000 | Loss: 3.1233 | LR: 4.43e-04 | 1398.0ms/step | 14.3 samples/s
615
+ Step 10220/44000 | Loss: 3.1277 | LR: 4.43e-04 | 1395.7ms/step | 14.3 samples/s
616
+ Step 10230/44000 | Loss: 3.1004 | LR: 4.43e-04 | 1396.2ms/step | 14.3 samples/s
617
+ Step 10240/44000 | Loss: 3.1189 | LR: 4.43e-04 | 1395.5ms/step | 14.3 samples/s
618
+ Step 10250/44000 | Loss: 3.0939 | LR: 4.43e-04 | 1394.3ms/step | 14.3 samples/s
619
+ Step 10260/44000 | Loss: 3.1409 | LR: 4.43e-04 | 1394.4ms/step | 14.3 samples/s
620
+ Step 10270/44000 | Loss: 3.1171 | LR: 4.43e-04 | 1397.7ms/step | 14.3 samples/s
621
+ Step 10280/44000 | Loss: 3.0862 | LR: 4.42e-04 | 1396.1ms/step | 14.3 samples/s
622
+ Step 10290/44000 | Loss: 3.1439 | LR: 4.42e-04 | 1396.3ms/step | 14.3 samples/s
623
+ Step 10300/44000 | Loss: 3.1124 | LR: 4.42e-04 | 1395.8ms/step | 14.3 samples/s
624
+ Step 10310/44000 | Loss: 3.0686 | LR: 4.42e-04 | 1392.1ms/step | 14.4 samples/s
625
+ Step 10320/44000 | Loss: 3.0959 | LR: 4.42e-04 | 1392.6ms/step | 14.4 samples/s
626
+ Step 10330/44000 | Loss: 3.1015 | LR: 4.42e-04 | 1395.0ms/step | 14.3 samples/s
627
+ Step 10340/44000 | Loss: 3.1183 | LR: 4.42e-04 | 1394.5ms/step | 14.3 samples/s
628
+ Step 10350/44000 | Loss: 3.1194 | LR: 4.42e-04 | 1396.1ms/step | 14.3 samples/s
629
+ Step 10360/44000 | Loss: 3.1002 | LR: 4.42e-04 | 1398.0ms/step | 14.3 samples/s
630
+ Step 10370/44000 | Loss: 3.1364 | LR: 4.41e-04 | 1396.0ms/step | 14.3 samples/s
631
+ Step 10380/44000 | Loss: 3.1361 | LR: 4.41e-04 | 1397.8ms/step | 14.3 samples/s
632
+ Step 10390/44000 | Loss: 3.0966 | LR: 4.41e-04 | 1396.6ms/step | 14.3 samples/s
633
+ Step 10400/44000 | Loss: 3.1222 | LR: 4.41e-04 | 1389.9ms/step | 14.4 samples/s
634
+ Step 10410/44000 | Loss: 3.1257 | LR: 4.41e-04 | 1393.6ms/step | 14.4 samples/s
635
+ Step 10420/44000 | Loss: 3.0791 | LR: 4.41e-04 | 1392.0ms/step | 14.4 samples/s
636
+ Step 10430/44000 | Loss: 3.1054 | LR: 4.41e-04 | 1393.9ms/step | 14.3 samples/s
637
+ Step 10440/44000 | Loss: 3.1114 | LR: 4.41e-04 | 1395.7ms/step | 14.3 samples/s
638
+ Step 10450/44000 | Loss: 3.1067 | LR: 4.40e-04 | 1394.7ms/step | 14.3 samples/s
639
+ Step 10460/44000 | Loss: 3.0836 | LR: 4.40e-04 | 1394.0ms/step | 14.3 samples/s
640
+ Step 10470/44000 | Loss: 3.0916 | LR: 4.40e-04 | 1391.6ms/step | 14.4 samples/s
641
+ Step 10480/44000 | Loss: 3.1031 | LR: 4.40e-04 | 1394.3ms/step | 14.3 samples/s
642
+ Step 10490/44000 | Loss: 3.0689 | LR: 4.40e-04 | 1395.9ms/step | 14.3 samples/s
643
+ Step 10500/44000 | Loss: 3.1028 | LR: 4.40e-04 | 1390.4ms/step | 14.4 samples/s
644
+ Step 10510/44000 | Loss: 3.1085 | LR: 4.40e-04 | 1397.3ms/step | 14.3 samples/s
645
+ Step 10520/44000 | Loss: 3.1095 | LR: 4.40e-04 | 1392.9ms/step | 14.4 samples/s
646
+ Step 10530/44000 | Loss: 3.1270 | LR: 4.40e-04 | 1391.4ms/step | 14.4 samples/s
647
+ Step 10540/44000 | Loss: 3.1258 | LR: 4.39e-04 | 1393.8ms/step | 14.3 samples/s
648
+ Step 10550/44000 | Loss: 3.1253 | LR: 4.39e-04 | 1393.8ms/step | 14.3 samples/s
649
+ Step 10560/44000 | Loss: 3.1050 | LR: 4.39e-04 | 1388.7ms/step | 14.4 samples/s
650
+ Step 10570/44000 | Loss: 3.1137 | LR: 4.39e-04 | 1391.8ms/step | 14.4 samples/s
651
+ Step 10580/44000 | Loss: 3.1295 | LR: 4.39e-04 | 1390.7ms/step | 14.4 samples/s
652
+ Step 10590/44000 | Loss: 3.0973 | LR: 4.39e-04 | 1391.0ms/step | 14.4 samples/s
653
+ Step 10600/44000 | Loss: 3.0929 | LR: 4.39e-04 | 1391.7ms/step | 14.4 samples/s
654
+ Step 10610/44000 | Loss: 3.0905 | LR: 4.39e-04 | 1396.4ms/step | 14.3 samples/s
655
+ Step 10620/44000 | Loss: 3.0762 | LR: 4.38e-04 | 1394.0ms/step | 14.3 samples/s
656
+ Step 10630/44000 | Loss: 3.1209 | LR: 4.38e-04 | 1392.4ms/step | 14.4 samples/s
657
+ Step 10640/44000 | Loss: 3.0923 | LR: 4.38e-04 | 1393.7ms/step | 14.4 samples/s
658
+ Step 10650/44000 | Loss: 3.1008 | LR: 4.38e-04 | 1393.3ms/step | 14.4 samples/s
659
+ Step 10660/44000 | Loss: 3.1019 | LR: 4.38e-04 | 1392.1ms/step | 14.4 samples/s
660
+ Step 10670/44000 | Loss: 3.0922 | LR: 4.38e-04 | 1388.5ms/step | 14.4 samples/s
661
+ Step 10680/44000 | Loss: 3.0709 | LR: 4.38e-04 | 1388.4ms/step | 14.4 samples/s
662
+ Step 10690/44000 | Loss: 3.0866 | LR: 4.38e-04 | 1392.1ms/step | 14.4 samples/s
663
+ Step 10700/44000 | Loss: 3.1307 | LR: 4.37e-04 | 1389.5ms/step | 14.4 samples/s
664
+ Step 10710/44000 | Loss: 3.1036 | LR: 4.37e-04 | 1390.4ms/step | 14.4 samples/s
665
+ Step 10720/44000 | Loss: 3.1113 | LR: 4.37e-04 | 1389.6ms/step | 14.4 samples/s
666
+ Step 10730/44000 | Loss: 3.1002 | LR: 4.37e-04 | 1389.3ms/step | 14.4 samples/s
667
+ Step 10740/44000 | Loss: 3.0982 | LR: 4.37e-04 | 1391.2ms/step | 14.4 samples/s
668
+ Step 10750/44000 | Loss: 3.1171 | LR: 4.37e-04 | 1389.4ms/step | 14.4 samples/s
669
+ Step 10760/44000 | Loss: 3.1164 | LR: 4.37e-04 | 1394.0ms/step | 14.3 samples/s
670
+ Step 10770/44000 | Loss: 3.0732 | LR: 4.37e-04 | 1392.7ms/step | 14.4 samples/s
671
+ Step 10780/44000 | Loss: 3.1094 | LR: 4.37e-04 | 1393.8ms/step | 14.3 samples/s
672
+ Step 10790/44000 | Loss: 3.1249 | LR: 4.36e-04 | 1392.2ms/step | 14.4 samples/s
673
+ Step 10800/44000 | Loss: 3.0932 | LR: 4.36e-04 | 1391.3ms/step | 14.4 samples/s
674
+ Step 10810/44000 | Loss: 3.0814 | LR: 4.36e-04 | 1390.5ms/step | 14.4 samples/s
675
+ Step 10820/44000 | Loss: 3.1016 | LR: 4.36e-04 | 1390.3ms/step | 14.4 samples/s
676
+ Step 10830/44000 | Loss: 3.0962 | LR: 4.36e-04 | 1391.7ms/step | 14.4 samples/s
677
+ Step 10840/44000 | Loss: 3.0945 | LR: 4.36e-04 | 1392.0ms/step | 14.4 samples/s
678
+ Step 10850/44000 | Loss: 3.1191 | LR: 4.36e-04 | 1390.2ms/step | 14.4 samples/s
679
+ Step 10860/44000 | Loss: 3.1179 | LR: 4.36e-04 | 1390.6ms/step | 14.4 samples/s
680
+ Step 10870/44000 | Loss: 3.0693 | LR: 4.35e-04 | 1391.6ms/step | 14.4 samples/s
681
+ Step 10880/44000 | Loss: 3.0698 | LR: 4.35e-04 | 1393.2ms/step | 14.4 samples/s
682
+ Step 10890/44000 | Loss: 3.1339 | LR: 4.35e-04 | 1392.8ms/step | 14.4 samples/s
683
+ Step 10900/44000 | Loss: 3.1124 | LR: 4.35e-04 | 1391.5ms/step | 14.4 samples/s
684
+ Step 10910/44000 | Loss: 3.0632 | LR: 4.35e-04 | 1389.7ms/step | 14.4 samples/s
685
+ Step 10920/44000 | Loss: 3.1180 | LR: 4.35e-04 | 1390.0ms/step | 14.4 samples/s
686
+ Step 10930/44000 | Loss: 3.0972 | LR: 4.35e-04 | 1390.9ms/step | 14.4 samples/s
687
+ Step 10940/44000 | Loss: 3.1109 | LR: 4.35e-04 | 1396.0ms/step | 14.3 samples/s
688
+ Step 10950/44000 | Loss: 3.1414 | LR: 4.34e-04 | 1395.7ms/step | 14.3 samples/s
689
+ Step 10960/44000 | Loss: 3.0743 | LR: 4.34e-04 | 1396.3ms/step | 14.3 samples/s
690
+ Step 10970/44000 | Loss: 3.0889 | LR: 4.34e-04 | 1390.2ms/step | 14.4 samples/s
691
+ Step 10980/44000 | Loss: 3.0992 | LR: 4.34e-04 | 1394.1ms/step | 14.3 samples/s
692
+ Step 10990/44000 | Loss: 3.1109 | LR: 4.34e-04 | 1393.9ms/step | 14.3 samples/s
693
+ Step 11000/44000 | Loss: 3.0982 | LR: 4.34e-04 | 1395.9ms/step | 14.3 samples/s
694
+ Validation at step 11000: Loss = 3.0984, Perplexity = 22.16
695
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735/checkpoint-11000
696
+ Step 11010/44000 | Loss: 3.0973 | LR: 4.34e-04 | 3868.0ms/step | 5.2 samples/s
697
+ Step 11020/44000 | Loss: 3.0997 | LR: 4.34e-04 | 1389.9ms/step | 14.4 samples/s
698
+ Step 11030/44000 | Loss: 3.0926 | LR: 4.33e-04 | 1391.9ms/step | 14.4 samples/s
699
+ Step 11040/44000 | Loss: 3.0832 | LR: 4.33e-04 | 1390.3ms/step | 14.4 samples/s
700
+ Step 11050/44000 | Loss: 3.1004 | LR: 4.33e-04 | 1388.0ms/step | 14.4 samples/s
701
+ Step 11060/44000 | Loss: 3.0932 | LR: 4.33e-04 | 1390.0ms/step | 14.4 samples/s
702
+ Step 11070/44000 | Loss: 3.0779 | LR: 4.33e-04 | 1391.6ms/step | 14.4 samples/s
703
+ Step 11080/44000 | Loss: 3.0684 | LR: 4.33e-04 | 1396.3ms/step | 14.3 samples/s
704
+ Step 11090/44000 | Loss: 3.1000 | LR: 4.33e-04 | 1392.3ms/step | 14.4 samples/s
705
+ Step 11100/44000 | Loss: 3.1147 | LR: 4.33e-04 | 1398.0ms/step | 14.3 samples/s
706
+ Step 11110/44000 | Loss: 3.1112 | LR: 4.32e-04 | 1394.0ms/step | 14.3 samples/s
707
+ Step 11120/44000 | Loss: 3.0900 | LR: 4.32e-04 | 1395.9ms/step | 14.3 samples/s
708
+ Step 11130/44000 | Loss: 3.0957 | LR: 4.32e-04 | 1393.9ms/step | 14.3 samples/s
709
+ Step 11140/44000 | Loss: 3.1268 | LR: 4.32e-04 | 1400.5ms/step | 14.3 samples/s
710
+ Step 11150/44000 | Loss: 3.0921 | LR: 4.32e-04 | 1399.9ms/step | 14.3 samples/s
711
+ Step 11160/44000 | Loss: 3.0887 | LR: 4.32e-04 | 1393.8ms/step | 14.3 samples/s
712
+ Step 11170/44000 | Loss: 3.0933 | LR: 4.32e-04 | 1395.1ms/step | 14.3 samples/s
713
+ Step 11180/44000 | Loss: 3.0909 | LR: 4.32e-04 | 1398.9ms/step | 14.3 samples/s
714
+ Step 11190/44000 | Loss: 3.0836 | LR: 4.31e-04 | 1396.2ms/step | 14.3 samples/s
715
+ Step 11200/44000 | Loss: 3.0731 | LR: 4.31e-04 | 1397.1ms/step | 14.3 samples/s
716
+ Step 11210/44000 | Loss: 3.0764 | LR: 4.31e-04 | 1394.6ms/step | 14.3 samples/s
717
+ Step 11220/44000 | Loss: 3.0901 | LR: 4.31e-04 | 1392.0ms/step | 14.4 samples/s
718
+ Step 11230/44000 | Loss: 3.0655 | LR: 4.31e-04 | 1396.2ms/step | 14.3 samples/s
719
+ Step 11240/44000 | Loss: 3.0886 | LR: 4.31e-04 | 1395.9ms/step | 14.3 samples/s
720
+ Step 11250/44000 | Loss: 3.0726 | LR: 4.31e-04 | 1392.3ms/step | 14.4 samples/s
721
+ Step 11260/44000 | Loss: 3.0844 | LR: 4.31e-04 | 1391.7ms/step | 14.4 samples/s
722
+ Step 11270/44000 | Loss: 3.0644 | LR: 4.30e-04 | 1392.2ms/step | 14.4 samples/s
723
+ Step 11280/44000 | Loss: 3.0716 | LR: 4.30e-04 | 1394.0ms/step | 14.3 samples/s
724
+ Step 11290/44000 | Loss: 3.0758 | LR: 4.30e-04 | 1392.4ms/step | 14.4 samples/s
725
+ Step 11300/44000 | Loss: 3.1062 | LR: 4.30e-04 | 1393.6ms/step | 14.4 samples/s
726
+ Step 11310/44000 | Loss: 3.1046 | LR: 4.30e-04 | 1392.4ms/step | 14.4 samples/s
727
+ Step 11320/44000 | Loss: 3.1072 | LR: 4.30e-04 | 1397.6ms/step | 14.3 samples/s
728
+ Step 11330/44000 | Loss: 3.0566 | LR: 4.30e-04 | 1395.5ms/step | 14.3 samples/s
729
+ Step 11340/44000 | Loss: 3.0935 | LR: 4.30e-04 | 1390.4ms/step | 14.4 samples/s
730
+ Step 11350/44000 | Loss: 3.0881 | LR: 4.29e-04 | 1398.1ms/step | 14.3 samples/s
731
+ Step 11360/44000 | Loss: 3.0593 | LR: 4.29e-04 | 1395.6ms/step | 14.3 samples/s
732
+ Step 11370/44000 | Loss: 3.1109 | LR: 4.29e-04 | 1392.5ms/step | 14.4 samples/s
733
+ Step 11380/44000 | Loss: 3.0923 | LR: 4.29e-04 | 1393.6ms/step | 14.4 samples/s
734
+ Step 11390/44000 | Loss: 3.0982 | LR: 4.29e-04 | 1390.5ms/step | 14.4 samples/s
735
+ Step 11400/44000 | Loss: 3.1076 | LR: 4.29e-04 | 1397.4ms/step | 14.3 samples/s
736
+ Step 11410/44000 | Loss: 3.0745 | LR: 4.29e-04 | 1393.7ms/step | 14.4 samples/s
737
+ Step 11420/44000 | Loss: 3.0955 | LR: 4.29e-04 | 1390.7ms/step | 14.4 samples/s
738
+ Step 11430/44000 | Loss: 3.0683 | LR: 4.28e-04 | 1395.9ms/step | 14.3 samples/s
739
+ Step 11440/44000 | Loss: 3.0805 | LR: 4.28e-04 | 1392.2ms/step | 14.4 samples/s
740
+ Step 11450/44000 | Loss: 3.1192 | LR: 4.28e-04 | 1391.9ms/step | 14.4 samples/s
741
+ Step 11460/44000 | Loss: 3.0962 | LR: 4.28e-04 | 1390.3ms/step | 14.4 samples/s
742
+ Step 11470/44000 | Loss: 3.0403 | LR: 4.28e-04 | 1393.6ms/step | 14.4 samples/s
743
+ Step 11480/44000 | Loss: 3.0712 | LR: 4.28e-04 | 1391.8ms/step | 14.4 samples/s
744
+ Step 11490/44000 | Loss: 3.0584 | LR: 4.28e-04 | 1393.0ms/step | 14.4 samples/s
745
+ Step 11500/44000 | Loss: 3.0924 | LR: 4.28e-04 | 1395.2ms/step | 14.3 samples/s
746
+ Step 11510/44000 | Loss: 3.0917 | LR: 4.27e-04 | 1390.1ms/step | 14.4 samples/s
747
+ Step 11520/44000 | Loss: 3.0599 | LR: 4.27e-04 | 1390.0ms/step | 14.4 samples/s
748
+ Step 11530/44000 | Loss: 3.0541 | LR: 4.27e-04 | 1391.5ms/step | 14.4 samples/s
749
+ Step 11540/44000 | Loss: 3.0803 | LR: 4.27e-04 | 1389.4ms/step | 14.4 samples/s
750
+ Step 11550/44000 | Loss: 3.0936 | LR: 4.27e-04 | 1393.0ms/step | 14.4 samples/s
751
+ Step 11560/44000 | Loss: 3.0637 | LR: 4.27e-04 | 1391.5ms/step | 14.4 samples/s
752
+ Step 11570/44000 | Loss: 3.0718 | LR: 4.27e-04 | 1388.7ms/step | 14.4 samples/s
753
+ Step 11580/44000 | Loss: 3.0918 | LR: 4.27e-04 | 1391.9ms/step | 14.4 samples/s
754
+ Step 11590/44000 | Loss: 3.0481 | LR: 4.26e-04 | 1391.5ms/step | 14.4 samples/s
755
+ Step 11600/44000 | Loss: 3.0861 | LR: 4.26e-04 | 1388.7ms/step | 14.4 samples/s
756
+ Step 11610/44000 | Loss: 3.0628 | LR: 4.26e-04 | 1392.1ms/step | 14.4 samples/s
757
+ Step 11620/44000 | Loss: 3.1109 | LR: 4.26e-04 | 1391.6ms/step | 14.4 samples/s
758
+ Step 11630/44000 | Loss: 3.0839 | LR: 4.26e-04 | 1389.8ms/step | 14.4 samples/s
759
+ Step 11640/44000 | Loss: 3.0468 | LR: 4.26e-04 | 1390.4ms/step | 14.4 samples/s
760
+ Step 11650/44000 | Loss: 3.0696 | LR: 4.26e-04 | 1390.2ms/step | 14.4 samples/s
761
+ Step 11660/44000 | Loss: 3.1045 | LR: 4.26e-04 | 1389.5ms/step | 14.4 samples/s
762
+ Step 11670/44000 | Loss: 3.0740 | LR: 4.25e-04 | 1392.1ms/step | 14.4 samples/s
763
+ Step 11680/44000 | Loss: 3.0556 | LR: 4.25e-04 | 1390.0ms/step | 14.4 samples/s
764
+ Step 11690/44000 | Loss: 3.0948 | LR: 4.25e-04 | 1392.5ms/step | 14.4 samples/s
765
+ Step 11700/44000 | Loss: 3.0689 | LR: 4.25e-04 | 1391.6ms/step | 14.4 samples/s
766
+ Step 11710/44000 | Loss: 3.0805 | LR: 4.25e-04 | 1392.1ms/step | 14.4 samples/s
767
+ Step 11720/44000 | Loss: 3.1107 | LR: 4.25e-04 | 1388.1ms/step | 14.4 samples/s
768
+ Step 11730/44000 | Loss: 3.1131 | LR: 4.25e-04 | 1390.0ms/step | 14.4 samples/s
769
+ Step 11740/44000 | Loss: 3.0815 | LR: 4.24e-04 | 1388.3ms/step | 14.4 samples/s
770
+ Step 11750/44000 | Loss: 3.0877 | LR: 4.24e-04 | 1387.4ms/step | 14.4 samples/s
771
+ Step 11760/44000 | Loss: 3.0858 | LR: 4.24e-04 | 1389.8ms/step | 14.4 samples/s
772
+ Step 11770/44000 | Loss: 3.0789 | LR: 4.24e-04 | 1392.7ms/step | 14.4 samples/s
773
+ Step 11780/44000 | Loss: 3.0779 | LR: 4.24e-04 | 1389.9ms/step | 14.4 samples/s
774
+ Step 11790/44000 | Loss: 3.0595 | LR: 4.24e-04 | 1390.1ms/step | 14.4 samples/s
775
+ Step 11800/44000 | Loss: 3.0799 | LR: 4.24e-04 | 1388.4ms/step | 14.4 samples/s
776
+ Step 11810/44000 | Loss: 3.1019 | LR: 4.24e-04 | 1390.1ms/step | 14.4 samples/s
777
+ Step 11820/44000 | Loss: 3.0517 | LR: 4.23e-04 | 1393.5ms/step | 14.4 samples/s
778
+ Step 11830/44000 | Loss: 3.1006 | LR: 4.23e-04 | 1391.2ms/step | 14.4 samples/s
779
+ Step 11840/44000 | Loss: 3.1044 | LR: 4.23e-04 | 1389.3ms/step | 14.4 samples/s
780
+ Step 11850/44000 | Loss: 3.0668 | LR: 4.23e-04 | 1393.5ms/step | 14.4 samples/s
781
+ Step 11860/44000 | Loss: 3.0715 | LR: 4.23e-04 | 1389.9ms/step | 14.4 samples/s
782
+ Step 11870/44000 | Loss: 3.0814 | LR: 4.23e-04 | 1391.9ms/step | 14.4 samples/s
783
+ Step 11880/44000 | Loss: 3.0977 | LR: 4.23e-04 | 1392.0ms/step | 14.4 samples/s
784
+ Step 11890/44000 | Loss: 3.0771 | LR: 4.23e-04 | 1390.2ms/step | 14.4 samples/s
785
+ Step 11900/44000 | Loss: 3.0755 | LR: 4.22e-04 | 1397.5ms/step | 14.3 samples/s
786
+ Step 11910/44000 | Loss: 3.0631 | LR: 4.22e-04 | 1394.4ms/step | 14.3 samples/s
787
+ Step 11920/44000 | Loss: 3.0929 | LR: 4.22e-04 | 1389.1ms/step | 14.4 samples/s
788
+ Step 11930/44000 | Loss: 3.0730 | LR: 4.22e-04 | 1387.0ms/step | 14.4 samples/s
789
+ Step 11940/44000 | Loss: 3.0599 | LR: 4.22e-04 | 1387.8ms/step | 14.4 samples/s
790
+ Step 11950/44000 | Loss: 3.0576 | LR: 4.22e-04 | 1389.8ms/step | 14.4 samples/s
791
+ Step 11960/44000 | Loss: 3.0581 | LR: 4.22e-04 | 1388.2ms/step | 14.4 samples/s
792
+ Step 11970/44000 | Loss: 3.0841 | LR: 4.21e-04 | 1393.5ms/step | 14.4 samples/s
793
+ Step 11980/44000 | Loss: 3.0820 | LR: 4.21e-04 | 1390.3ms/step | 14.4 samples/s
794
+ Step 11990/44000 | Loss: 3.1163 | LR: 4.21e-04 | 1391.0ms/step | 14.4 samples/s
795
+ Step 12000/44000 | Loss: 3.0848 | LR: 4.21e-04 | 1389.8ms/step | 14.4 samples/s
796
+ Validation at step 12000: Loss = 3.0770, Perplexity = 21.69
797
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735/checkpoint-12000
798
+ Step 12010/44000 | Loss: 3.0410 | LR: 4.21e-04 | 3875.5ms/step | 5.2 samples/s
799
+ Step 12020/44000 | Loss: 3.0870 | LR: 4.21e-04 | 1390.0ms/step | 14.4 samples/s
800
+ Step 12030/44000 | Loss: 3.0730 | LR: 4.21e-04 | 1389.9ms/step | 14.4 samples/s
801
+ Step 12040/44000 | Loss: 3.0558 | LR: 4.21e-04 | 1390.2ms/step | 14.4 samples/s
802
+ Step 12050/44000 | Loss: 3.0739 | LR: 4.20e-04 | 1394.1ms/step | 14.3 samples/s
803
+ Step 12060/44000 | Loss: 3.1198 | LR: 4.20e-04 | 1394.0ms/step | 14.3 samples/s
804
+ Step 12070/44000 | Loss: 3.0567 | LR: 4.20e-04 | 1396.2ms/step | 14.3 samples/s
805
+ Step 12080/44000 | Loss: 3.0858 | LR: 4.20e-04 | 1392.4ms/step | 14.4 samples/s
806
+ Step 12090/44000 | Loss: 3.0821 | LR: 4.20e-04 | 1397.1ms/step | 14.3 samples/s
807
+ Step 12100/44000 | Loss: 3.0769 | LR: 4.20e-04 | 1394.1ms/step | 14.3 samples/s
808
+ Step 12110/44000 | Loss: 3.0754 | LR: 4.20e-04 | 1394.4ms/step | 14.3 samples/s
809
+ Step 12120/44000 | Loss: 3.0978 | LR: 4.19e-04 | 1394.7ms/step | 14.3 samples/s
810
+ Step 12130/44000 | Loss: 3.0825 | LR: 4.19e-04 | 1397.3ms/step | 14.3 samples/s
811
+ Step 12140/44000 | Loss: 3.0951 | LR: 4.19e-04 | 1396.0ms/step | 14.3 samples/s
812
+ Step 12150/44000 | Loss: 3.0849 | LR: 4.19e-04 | 1399.9ms/step | 14.3 samples/s
813
+ Step 12160/44000 | Loss: 3.0574 | LR: 4.19e-04 | 1397.9ms/step | 14.3 samples/s
814
+ Step 12170/44000 | Loss: 3.0993 | LR: 4.19e-04 | 1396.0ms/step | 14.3 samples/s
815
+ Step 12180/44000 | Loss: 3.0875 | LR: 4.19e-04 | 1397.9ms/step | 14.3 samples/s
816
+ Step 12190/44000 | Loss: 3.0316 | LR: 4.19e-04 | 1394.3ms/step | 14.3 samples/s
817
+ Step 12200/44000 | Loss: 3.0653 | LR: 4.18e-04 | 1399.8ms/step | 14.3 samples/s
818
+ Step 12210/44000 | Loss: 3.0779 | LR: 4.18e-04 | 1397.9ms/step | 14.3 samples/s
819
+ Step 12220/44000 | Loss: 3.0890 | LR: 4.18e-04 | 1396.3ms/step | 14.3 samples/s
820
+ Step 12230/44000 | Loss: 3.0564 | LR: 4.18e-04 | 1396.1ms/step | 14.3 samples/s
821
+ Step 12240/44000 | Loss: 3.0726 | LR: 4.18e-04 | 1395.6ms/step | 14.3 samples/s
822
+ Step 12250/44000 | Loss: 3.0603 | LR: 4.18e-04 | 1394.2ms/step | 14.3 samples/s
823
+ Step 12260/44000 | Loss: 3.0661 | LR: 4.18e-04 | 1393.7ms/step | 14.4 samples/s
824
+ Step 12270/44000 | Loss: 3.0923 | LR: 4.17e-04 | 1396.4ms/step | 14.3 samples/s
825
+ Step 12280/44000 | Loss: 3.0676 | LR: 4.17e-04 | 1396.0ms/step | 14.3 samples/s
826
+ Step 12290/44000 | Loss: 3.0742 | LR: 4.17e-04 | 1395.3ms/step | 14.3 samples/s
827
+ Step 12300/44000 | Loss: 3.0912 | LR: 4.17e-04 | 1393.2ms/step | 14.4 samples/s
828
+ Step 12310/44000 | Loss: 3.0801 | LR: 4.17e-04 | 1395.7ms/step | 14.3 samples/s
829
+ Step 12320/44000 | Loss: 3.0670 | LR: 4.17e-04 | 1399.4ms/step | 14.3 samples/s
830
+ Step 12330/44000 | Loss: 3.0500 | LR: 4.17e-04 | 1394.3ms/step | 14.3 samples/s
831
+ Step 12340/44000 | Loss: 3.1003 | LR: 4.17e-04 | 1397.8ms/step | 14.3 samples/s
832
+ Step 12350/44000 | Loss: 3.0283 | LR: 4.16e-04 | 1394.0ms/step | 14.3 samples/s
833
+ Step 12360/44000 | Loss: 3.0728 | LR: 4.16e-04 | 1396.2ms/step | 14.3 samples/s
834
+ Step 12370/44000 | Loss: 3.0635 | LR: 4.16e-04 | 1399.7ms/step | 14.3 samples/s
835
+ Step 12380/44000 | Loss: 3.0681 | LR: 4.16e-04 | 1398.6ms/step | 14.3 samples/s
836
+ Step 12390/44000 | Loss: 3.1175 | LR: 4.16e-04 | 1398.0ms/step | 14.3 samples/s
837
+ Step 12400/44000 | Loss: 3.0510 | LR: 4.16e-04 | 1397.0ms/step | 14.3 samples/s
838
+ Step 12410/44000 | Loss: 3.0790 | LR: 4.16e-04 | 1395.2ms/step | 14.3 samples/s
839
+ Step 12420/44000 | Loss: 3.0925 | LR: 4.15e-04 | 1399.7ms/step | 14.3 samples/s
840
+ Step 12430/44000 | Loss: 3.0356 | LR: 4.15e-04 | 1398.3ms/step | 14.3 samples/s
841
+ Step 12440/44000 | Loss: 3.0805 | LR: 4.15e-04 | 1399.8ms/step | 14.3 samples/s
842
+ Step 12450/44000 | Loss: 3.0693 | LR: 4.15e-04 | 1393.9ms/step | 14.3 samples/s
843
+ Step 12460/44000 | Loss: 3.0685 | LR: 4.15e-04 | 1392.3ms/step | 14.4 samples/s
844
+ Step 12470/44000 | Loss: 3.0438 | LR: 4.15e-04 | 1391.8ms/step | 14.4 samples/s
845
+ Step 12480/44000 | Loss: 3.0403 | LR: 4.15e-04 | 1391.9ms/step | 14.4 samples/s
846
+ Step 12490/44000 | Loss: 3.0781 | LR: 4.14e-04 | 1392.2ms/step | 14.4 samples/s
847
+ Step 12500/44000 | Loss: 3.0665 | LR: 4.14e-04 | 1395.5ms/step | 14.3 samples/s
848
+ Step 12510/44000 | Loss: 3.0784 | LR: 4.14e-04 | 1389.2ms/step | 14.4 samples/s
849
+ Step 12520/44000 | Loss: 3.0439 | LR: 4.14e-04 | 1391.3ms/step | 14.4 samples/s
850
+ Step 12530/44000 | Loss: 3.0660 | LR: 4.14e-04 | 1394.3ms/step | 14.3 samples/s
851
+ Step 12540/44000 | Loss: 3.0581 | LR: 4.14e-04 | 1392.0ms/step | 14.4 samples/s
852
+ Step 12550/44000 | Loss: 3.0667 | LR: 4.14e-04 | 1391.4ms/step | 14.4 samples/s
853
+ Step 12560/44000 | Loss: 3.0939 | LR: 4.14e-04 | 1398.1ms/step | 14.3 samples/s
854
+ Step 12570/44000 | Loss: 3.0859 | LR: 4.13e-04 | 1392.2ms/step | 14.4 samples/s
855
+ Step 12580/44000 | Loss: 3.0843 | LR: 4.13e-04 | 1394.3ms/step | 14.3 samples/s
856
+ Step 12590/44000 | Loss: 3.0230 | LR: 4.13e-04 | 1393.8ms/step | 14.3 samples/s
857
+ Step 12600/44000 | Loss: 3.0786 | LR: 4.13e-04 | 1391.6ms/step | 14.4 samples/s
858
+ Step 12610/44000 | Loss: 3.0493 | LR: 4.13e-04 | 1392.5ms/step | 14.4 samples/s
859
+ Step 12620/44000 | Loss: 3.0618 | LR: 4.13e-04 | 1393.1ms/step | 14.4 samples/s
860
+ Step 12630/44000 | Loss: 3.0512 | LR: 4.13e-04 | 1392.2ms/step | 14.4 samples/s
861
+ Step 12640/44000 | Loss: 3.0555 | LR: 4.12e-04 | 1390.6ms/step | 14.4 samples/s
862
+ Step 12650/44000 | Loss: 3.0647 | LR: 4.12e-04 | 1392.0ms/step | 14.4 samples/s
863
+ Step 12660/44000 | Loss: 3.0778 | LR: 4.12e-04 | 1393.4ms/step | 14.4 samples/s
864
+ Step 12670/44000 | Loss: 3.0788 | LR: 4.12e-04 | 1392.1ms/step | 14.4 samples/s
865
+ Step 12680/44000 | Loss: 3.0819 | LR: 4.12e-04 | 1392.2ms/step | 14.4 samples/s
866
+ Step 12690/44000 | Loss: 3.0734 | LR: 4.12e-04 | 1394.3ms/step | 14.3 samples/s
867
+ Step 12700/44000 | Loss: 3.0408 | LR: 4.12e-04 | 1394.6ms/step | 14.3 samples/s
868
+ Step 12710/44000 | Loss: 3.0331 | LR: 4.11e-04 | 1393.0ms/step | 14.4 samples/s
869
+ Step 12720/44000 | Loss: 3.0674 | LR: 4.11e-04 | 1391.4ms/step | 14.4 samples/s
870
+ Step 12730/44000 | Loss: 3.0537 | LR: 4.11e-04 | 1394.3ms/step | 14.3 samples/s
871
+ Step 12740/44000 | Loss: 3.0479 | LR: 4.11e-04 | 1393.1ms/step | 14.4 samples/s
872
+ Step 12750/44000 | Loss: 3.0687 | LR: 4.11e-04 | 1389.0ms/step | 14.4 samples/s
873
+ Step 12760/44000 | Loss: 3.0718 | LR: 4.11e-04 | 1396.7ms/step | 14.3 samples/s
874
+ Step 12770/44000 | Loss: 3.0743 | LR: 4.11e-04 | 1392.2ms/step | 14.4 samples/s
875
+ Step 12780/44000 | Loss: 3.0343 | LR: 4.10e-04 | 1394.0ms/step | 14.3 samples/s
876
+ Step 12790/44000 | Loss: 3.0585 | LR: 4.10e-04 | 1393.1ms/step | 14.4 samples/s
877
+ Step 12800/44000 | Loss: 3.0235 | LR: 4.10e-04 | 1389.1ms/step | 14.4 samples/s
878
+ Step 12810/44000 | Loss: 3.0679 | LR: 4.10e-04 | 1391.0ms/step | 14.4 samples/s
879
+ Step 12820/44000 | Loss: 3.0398 | LR: 4.10e-04 | 1389.5ms/step | 14.4 samples/s
880
+ Step 12830/44000 | Loss: 3.0493 | LR: 4.10e-04 | 1389.1ms/step | 14.4 samples/s
881
+ Step 12840/44000 | Loss: 3.0656 | LR: 4.10e-04 | 1394.0ms/step | 14.3 samples/s
882
+ Step 12850/44000 | Loss: 3.0478 | LR: 4.09e-04 | 1392.1ms/step | 14.4 samples/s
883
+ Step 12860/44000 | Loss: 3.0675 | LR: 4.09e-04 | 1390.9ms/step | 14.4 samples/s
884
+ Step 12870/44000 | Loss: 3.0471 | LR: 4.09e-04 | 1391.3ms/step | 14.4 samples/s
885
+ Step 12880/44000 | Loss: 3.0776 | LR: 4.09e-04 | 1393.2ms/step | 14.4 samples/s
886
+ Step 12890/44000 | Loss: 3.0576 | LR: 4.09e-04 | 1388.8ms/step | 14.4 samples/s
887
+ Step 12900/44000 | Loss: 3.0603 | LR: 4.09e-04 | 1389.0ms/step | 14.4 samples/s
888
+ Step 12910/44000 | Loss: 3.0465 | LR: 4.09e-04 | 1387.4ms/step | 14.4 samples/s
889
+ Step 12920/44000 | Loss: 3.0314 | LR: 4.09e-04 | 1389.0ms/step | 14.4 samples/s
890
+ Step 12930/44000 | Loss: 3.0245 | LR: 4.08e-04 | 1388.9ms/step | 14.4 samples/s
891
+ Step 12940/44000 | Loss: 3.0556 | LR: 4.08e-04 | 1388.0ms/step | 14.4 samples/s
892
+ Step 12950/44000 | Loss: 3.0715 | LR: 4.08e-04 | 1389.0ms/step | 14.4 samples/s
893
+ Step 12960/44000 | Loss: 3.0521 | LR: 4.08e-04 | 1390.0ms/step | 14.4 samples/s
894
+ Step 12970/44000 | Loss: 3.0529 | LR: 4.08e-04 | 1388.5ms/step | 14.4 samples/s
895
+ Step 12980/44000 | Loss: 3.0697 | LR: 4.08e-04 | 1388.9ms/step | 14.4 samples/s
896
+ Step 12990/44000 | Loss: 3.0716 | LR: 4.08e-04 | 1390.0ms/step | 14.4 samples/s
897
+ Step 13000/44000 | Loss: 3.0590 | LR: 4.07e-04 | 1390.5ms/step | 14.4 samples/s
898
+ Validation at step 13000: Loss = 3.0598, Perplexity = 21.32
899
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735/checkpoint-13000
900
+ Step 13010/44000 | Loss: 3.0453 | LR: 4.07e-04 | 3876.8ms/step | 5.2 samples/s
901
+ Step 13020/44000 | Loss: 3.0595 | LR: 4.07e-04 | 1390.4ms/step | 14.4 samples/s
902
+ Step 13030/44000 | Loss: 3.0810 | LR: 4.07e-04 | 1395.7ms/step | 14.3 samples/s
903
+ Step 13040/44000 | Loss: 3.0743 | LR: 4.07e-04 | 1393.9ms/step | 14.3 samples/s
904
+ Step 13050/44000 | Loss: 3.0341 | LR: 4.07e-04 | 1391.0ms/step | 14.4 samples/s
905
+ Step 13060/44000 | Loss: 3.1067 | LR: 4.07e-04 | 1393.4ms/step | 14.4 samples/s
epfl_final/train_logs/ropectl_calib/a0p5.log ADDED
@@ -0,0 +1,213 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ New run: Checkpoints will be saved to /mloscratch/homes/navasard/sparse/checkpoints/ropectl_calib_a0p5/run_20260819_144428
2
+ Running config:
3
+ {
4
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
5
+ "model_dtype": "bfloat16",
6
+ "init_from_scratch": false,
7
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
8
+ "val_path": "/mloscratch/homes/navasard/data/fineweb_val_seed42",
9
+ "max_seq_length": 8192,
10
+ "num_workers": 8,
11
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/ropectl_calib_a0p5/run_20260819_144428",
12
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000",
13
+ "force_new_run": true,
14
+ "num_epochs": 1,
15
+ "max_steps": 44300,
16
+ "per_device_batch_size": 2,
17
+ "gradient_accumulation_steps": 10,
18
+ "learning_rate": 5e-05,
19
+ "weight_decay": 0.01,
20
+ "warmup_steps": 0,
21
+ "lr_scheduler": "constant",
22
+ "max_grad_norm": 1.0,
23
+ "save_steps": 300,
24
+ "eval_steps": 75,
25
+ "eval_batches": 50,
26
+ "logging_steps": 10,
27
+ "seed": 42,
28
+ "verbose": true,
29
+ "checkpoint_activations": false,
30
+ "strategy": "ddp",
31
+ "mixed_precision": "bf16",
32
+ "wandb_enabled": false,
33
+ "wandb_project": "stochastic_sparsity",
34
+ "wandb_name": "ropectl_calib_a0p5",
35
+ "wandb_tags": [
36
+ "qwen3",
37
+ "365m",
38
+ "8k",
39
+ "scaleladder",
40
+ "dense"
41
+ ],
42
+ "hidden_size": 768,
43
+ "intermediate_size": 3072,
44
+ "num_attention_heads": 6,
45
+ "num_key_value_heads": 2,
46
+ "num_hidden_layers": 28,
47
+ "head_dim": null,
48
+ "attention_dropout": 0.0,
49
+ "rope_train_position_scale_set": [
50
+ 1.0,
51
+ 0.9948169744088111,
52
+ 0.9846079846079846,
53
+ 0.9710144927536232,
54
+ 0.9365079365079365
55
+ ],
56
+ "attention_type": "sparse_stochastic",
57
+ "sparse_block_size": 128,
58
+ "sparse_num_random_blocks": 31,
59
+ "sparse_mask_pool_size": 1,
60
+ "sparse_val_dense_compare": false,
61
+ "sparse_layer_indices": null,
62
+ "sparse_mask_cache_dir": null,
63
+ "sparse_mask_seed": 0,
64
+ "sparse_mixed_nrb_set": null,
65
+ "dense_mode": true,
66
+ "dense_step_intervals": null
67
+ }
68
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
69
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000
70
+ Checkpoint activations: False
71
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
72
+
73
+
74
+ ================================================================================
75
+ MODEL ARCHITECTURE
76
+ ================================================================================
77
+ Model config:
78
+ vocab_size: 151936
79
+ hidden_size: 768
80
+ num_hidden_layers: 28
81
+ num_attention_heads: 6
82
+ num_key_value_heads: 2
83
+ head_dim: 128
84
+ intermediate_size: 3072
85
+ max_position_embeddings: 32768
86
+ rope_theta: 1000000
87
+ rope_train_position_scale_set: [1.0, 0.9948169744088111, 0.9846079846079846, 0.9710144927536232, 0.9365079365079365]
88
+ attention_type: sparse_stochastic
89
+ sparse_block_size: 128
90
+ sparse_num_random_blocks: 31
91
+ sparse_mask_pool_size: 1
92
+ sparse_max_seq_length: 8192
93
+
94
+ Parameter count:
95
+ Total parameters: 358,958,848
96
+ Trainable parameters: 358,958,848
97
+ Non-trainable parameters: 0
98
+ Model size (BF16): 0.72 GB
99
+
100
+ --------------------------------------------------------------------------------
101
+ LAYER-BY-LAYER SUMMARY
102
+ --------------------------------------------------------------------------------
103
+ ================================================================================================================================================================
104
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
105
+ ================================================================================================================================================================
106
+ Qwen3ForCausalLM [2, 8192] -- -- True
107
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
108
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
109
+ │ └─ModuleList: 2-2 -- -- -- True
110
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
111
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
132
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
133
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
134
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
135
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
136
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
137
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
138
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
139
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
140
+ ================================================================================================================================================================
141
+ Total params: 475,645,696
142
+ Trainable params: 475,645,696
143
+ Non-trainable params: 0
144
+ Total mult-adds (Units.MEGABYTES): 951.29
145
+ ================================================================================================================================================================
146
+ Input size (MB): 0.13
147
+ Forward/backward pass size (MB): 31197.23
148
+ Params size (MB): 951.29
149
+ Estimated Total Size (MB): 32148.66
150
+ ================================================================================================================================================================
151
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
152
+ ================================================================================
153
+
154
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
155
+ Using pre-tokenized dataset (no runtime tokenization)
156
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
157
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
158
+ Dataset split: 3027764 train, 159356 validation
159
+ Training examples (packed, train split): 3027764
160
+ Validation examples (packed, val split): 159356
161
+ Per-device batch size: 2
162
+ World size (GPUs): 1
163
+ Gradient accumulation steps: 10
164
+ Effective batch size (seqs/step): 20
165
+ Sequence length: 8192
166
+ Tokens per optimizer step: 163,840
167
+ Micro-batches per epoch (per rank): 1513882
168
+ Optimizer steps per epoch: 151388
169
+ Total optimizer steps: 44300 (capped by max_steps; 0.29 epochs)
170
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000...
171
+ Overriding checkpoint LR with config LR: 5e-05
172
+ Resumed from checkpoint at step 44000 (epoch 0, micro_step 440000)
173
+ Resuming training from step 44000
174
+ Starting training...
175
+ Validation at step 44000: Loss = 2.9127, Perplexity = 18.41
176
+ Switching attention mode for step 44001: dense (dense step interval)
177
+ Step 44010/44300 | Loss: 2.8868 | LR: 5.00e-05 | 118070.9ms/step | 0.2 samples/s
178
+ Step 44020/44300 | Loss: 2.9179 | LR: 5.00e-05 | 5114.0ms/step | 3.9 samples/s
179
+ Validation at step 44025: Loss = 2.8964, Perplexity = 18.11
180
+ Step 44030/44300 | Loss: 2.8595 | LR: 5.00e-05 | 14254.5ms/step | 1.4 samples/s
181
+ Step 44040/44300 | Loss: 2.8977 | LR: 5.00e-05 | 5036.7ms/step | 4.0 samples/s
182
+ Step 44050/44300 | Loss: 2.8830 | LR: 5.00e-05 | 5275.4ms/step | 3.8 samples/s
183
+ Step 44060/44300 | Loss: 2.9123 | LR: 5.00e-05 | 5059.5ms/step | 4.0 samples/s
184
+ Step 44070/44300 | Loss: 2.8704 | LR: 5.00e-05 | 5200.1ms/step | 3.8 samples/s
185
+ Step 44080/44300 | Loss: 2.8780 | LR: 5.00e-05 | 5216.8ms/step | 3.8 samples/s
186
+ Step 44090/44300 | Loss: 2.9049 | LR: 5.00e-05 | 5150.0ms/step | 3.9 samples/s
187
+ Step 44100/44300 | Loss: 2.9039 | LR: 5.00e-05 | 5091.6ms/step | 3.9 samples/s
188
+ Validation at step 44100: Loss = 2.8949, Perplexity = 18.08
189
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/ropectl_calib_a0p5/run_20260819_144428/checkpoint-44100
190
+ Step 44110/44300 | Loss: 2.8834 | LR: 5.00e-05 | 14813.2ms/step | 1.4 samples/s
191
+ Step 44120/44300 | Loss: 2.8711 | LR: 5.00e-05 | 4750.2ms/step | 4.2 samples/s
192
+ Step 44130/44300 | Loss: 2.8868 | LR: 5.00e-05 | 4813.4ms/step | 4.2 samples/s
193
+ Step 44140/44300 | Loss: 2.8773 | LR: 5.00e-05 | 4846.9ms/step | 4.1 samples/s
194
+ Step 44150/44300 | Loss: 2.8801 | LR: 5.00e-05 | 4969.3ms/step | 4.0 samples/s
195
+ Step 44160/44300 | Loss: 2.8650 | LR: 5.00e-05 | 4729.1ms/step | 4.2 samples/s
196
+ Step 44170/44300 | Loss: 2.8637 | LR: 5.00e-05 | 4804.3ms/step | 4.2 samples/s
197
+ Validation at step 44175: Loss = 2.8948, Perplexity = 18.08
198
+ Step 44180/44300 | Loss: 2.8971 | LR: 5.00e-05 | 13646.1ms/step | 1.5 samples/s
199
+ Step 44190/44300 | Loss: 2.9047 | LR: 5.00e-05 | 4866.6ms/step | 4.1 samples/s
200
+ Step 44200/44300 | Loss: 2.9014 | LR: 5.00e-05 | 4887.9ms/step | 4.1 samples/s
201
+ Step 44210/44300 | Loss: 2.8816 | LR: 5.00e-05 | 4851.9ms/step | 4.1 samples/s
202
+ Step 44220/44300 | Loss: 2.8553 | LR: 5.00e-05 | 4834.7ms/step | 4.1 samples/s
203
+ Step 44230/44300 | Loss: 2.8613 | LR: 5.00e-05 | 4992.1ms/step | 4.0 samples/s
204
+ Step 44240/44300 | Loss: 2.8832 | LR: 5.00e-05 | 4898.0ms/step | 4.1 samples/s
205
+ Step 44250/44300 | Loss: 2.9032 | LR: 5.00e-05 | 4746.6ms/step | 4.2 samples/s
206
+ Validation at step 44250: Loss = 2.8947, Perplexity = 18.08
207
+ Step 44260/44300 | Loss: 2.8808 | LR: 5.00e-05 | 13916.6ms/step | 1.4 samples/s
208
+ Step 44270/44300 | Loss: 2.8629 | LR: 5.00e-05 | 4803.1ms/step | 4.2 samples/s
209
+ Step 44280/44300 | Loss: 2.8770 | LR: 5.00e-05 | 4772.6ms/step | 4.2 samples/s
210
+ Step 44290/44300 | Loss: 2.8902 | LR: 5.00e-05 | 4624.9ms/step | 4.3 samples/s
211
+ Step 44300/44300 | Loss: 2.8962 | LR: 5.00e-05 | 3712.2ms/step | 5.4 samples/s
212
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/ropectl_calib_a0p5/run_20260819_144428/checkpoint-44300
213
+ Training complete!
epfl_final/train_logs/ropectl_calib/a1p0.log ADDED
@@ -0,0 +1,213 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ New run: Checkpoints will be saved to /mloscratch/homes/navasard/sparse/checkpoints/ropectl_calib_a1p0/run_20260819_144428
2
+ Running config:
3
+ {
4
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
5
+ "model_dtype": "bfloat16",
6
+ "init_from_scratch": false,
7
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
8
+ "val_path": "/mloscratch/homes/navasard/data/fineweb_val_seed42",
9
+ "max_seq_length": 8192,
10
+ "num_workers": 8,
11
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/ropectl_calib_a1p0/run_20260819_144428",
12
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000",
13
+ "force_new_run": true,
14
+ "num_epochs": 1,
15
+ "max_steps": 44300,
16
+ "per_device_batch_size": 2,
17
+ "gradient_accumulation_steps": 10,
18
+ "learning_rate": 5e-05,
19
+ "weight_decay": 0.01,
20
+ "warmup_steps": 0,
21
+ "lr_scheduler": "constant",
22
+ "max_grad_norm": 1.0,
23
+ "save_steps": 300,
24
+ "eval_steps": 75,
25
+ "eval_batches": 50,
26
+ "logging_steps": 10,
27
+ "seed": 42,
28
+ "verbose": true,
29
+ "checkpoint_activations": false,
30
+ "strategy": "ddp",
31
+ "mixed_precision": "bf16",
32
+ "wandb_enabled": false,
33
+ "wandb_project": "stochastic_sparsity",
34
+ "wandb_name": "ropectl_calib_a1p0",
35
+ "wandb_tags": [
36
+ "qwen3",
37
+ "365m",
38
+ "8k",
39
+ "scaleladder",
40
+ "dense"
41
+ ],
42
+ "hidden_size": 768,
43
+ "intermediate_size": 3072,
44
+ "num_attention_heads": 6,
45
+ "num_key_value_heads": 2,
46
+ "num_hidden_layers": 28,
47
+ "head_dim": null,
48
+ "attention_dropout": 0.0,
49
+ "rope_train_position_scale_set": [
50
+ 1.0,
51
+ 0.9896339488176222,
52
+ 0.9692159692159692,
53
+ 0.9420289855072463,
54
+ 0.8730158730158729
55
+ ],
56
+ "attention_type": "sparse_stochastic",
57
+ "sparse_block_size": 128,
58
+ "sparse_num_random_blocks": 31,
59
+ "sparse_mask_pool_size": 1,
60
+ "sparse_val_dense_compare": false,
61
+ "sparse_layer_indices": null,
62
+ "sparse_mask_cache_dir": null,
63
+ "sparse_mask_seed": 0,
64
+ "sparse_mixed_nrb_set": null,
65
+ "dense_mode": true,
66
+ "dense_step_intervals": null
67
+ }
68
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
69
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000
70
+ Checkpoint activations: False
71
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
72
+
73
+
74
+ ================================================================================
75
+ MODEL ARCHITECTURE
76
+ ================================================================================
77
+ Model config:
78
+ vocab_size: 151936
79
+ hidden_size: 768
80
+ num_hidden_layers: 28
81
+ num_attention_heads: 6
82
+ num_key_value_heads: 2
83
+ head_dim: 128
84
+ intermediate_size: 3072
85
+ max_position_embeddings: 32768
86
+ rope_theta: 1000000
87
+ rope_train_position_scale_set: [1.0, 0.9896339488176222, 0.9692159692159692, 0.9420289855072463, 0.8730158730158729]
88
+ attention_type: sparse_stochastic
89
+ sparse_block_size: 128
90
+ sparse_num_random_blocks: 31
91
+ sparse_mask_pool_size: 1
92
+ sparse_max_seq_length: 8192
93
+
94
+ Parameter count:
95
+ Total parameters: 358,958,848
96
+ Trainable parameters: 358,958,848
97
+ Non-trainable parameters: 0
98
+ Model size (BF16): 0.72 GB
99
+
100
+ --------------------------------------------------------------------------------
101
+ LAYER-BY-LAYER SUMMARY
102
+ --------------------------------------------------------------------------------
103
+ ================================================================================================================================================================
104
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
105
+ ================================================================================================================================================================
106
+ Qwen3ForCausalLM [2, 8192] -- -- True
107
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
108
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
109
+ │ └─ModuleList: 2-2 -- -- -- True
110
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
111
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
132
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
133
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
134
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
135
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
136
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
137
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
138
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
139
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
140
+ ================================================================================================================================================================
141
+ Total params: 475,645,696
142
+ Trainable params: 475,645,696
143
+ Non-trainable params: 0
144
+ Total mult-adds (Units.MEGABYTES): 951.29
145
+ ================================================================================================================================================================
146
+ Input size (MB): 0.13
147
+ Forward/backward pass size (MB): 31197.23
148
+ Params size (MB): 951.29
149
+ Estimated Total Size (MB): 32148.66
150
+ ================================================================================================================================================================
151
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
152
+ ================================================================================
153
+
154
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
155
+ Using pre-tokenized dataset (no runtime tokenization)
156
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
157
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
158
+ Dataset split: 3027764 train, 159356 validation
159
+ Training examples (packed, train split): 3027764
160
+ Validation examples (packed, val split): 159356
161
+ Per-device batch size: 2
162
+ World size (GPUs): 1
163
+ Gradient accumulation steps: 10
164
+ Effective batch size (seqs/step): 20
165
+ Sequence length: 8192
166
+ Tokens per optimizer step: 163,840
167
+ Micro-batches per epoch (per rank): 1513882
168
+ Optimizer steps per epoch: 151388
169
+ Total optimizer steps: 44300 (capped by max_steps; 0.29 epochs)
170
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000...
171
+ Overriding checkpoint LR with config LR: 5e-05
172
+ Resumed from checkpoint at step 44000 (epoch 0, micro_step 440000)
173
+ Resuming training from step 44000
174
+ Starting training...
175
+ Validation at step 44000: Loss = 2.9127, Perplexity = 18.41
176
+ Switching attention mode for step 44001: dense (dense step interval)
177
+ Step 44010/44300 | Loss: 2.8880 | LR: 5.00e-05 | 118137.6ms/step | 0.2 samples/s
178
+ Step 44020/44300 | Loss: 2.9188 | LR: 5.00e-05 | 5206.2ms/step | 3.8 samples/s
179
+ Validation at step 44025: Loss = 2.8971, Perplexity = 18.12
180
+ Step 44030/44300 | Loss: 2.8603 | LR: 5.00e-05 | 14192.7ms/step | 1.4 samples/s
181
+ Step 44040/44300 | Loss: 2.8985 | LR: 5.00e-05 | 5079.6ms/step | 3.9 samples/s
182
+ Step 44050/44300 | Loss: 2.8837 | LR: 5.00e-05 | 5076.2ms/step | 3.9 samples/s
183
+ Step 44060/44300 | Loss: 2.9131 | LR: 5.00e-05 | 5676.3ms/step | 3.5 samples/s
184
+ Step 44070/44300 | Loss: 2.8710 | LR: 5.00e-05 | 5429.4ms/step | 3.7 samples/s
185
+ Step 44080/44300 | Loss: 2.8788 | LR: 5.00e-05 | 5102.1ms/step | 3.9 samples/s
186
+ Step 44090/44300 | Loss: 2.9057 | LR: 5.00e-05 | 4990.1ms/step | 4.0 samples/s
187
+ Step 44100/44300 | Loss: 2.9045 | LR: 5.00e-05 | 5055.1ms/step | 4.0 samples/s
188
+ Validation at step 44100: Loss = 2.8954, Perplexity = 18.09
189
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/ropectl_calib_a1p0/run_20260819_144428/checkpoint-44100
190
+ Step 44110/44300 | Loss: 2.8840 | LR: 5.00e-05 | 14589.4ms/step | 1.4 samples/s
191
+ Step 44120/44300 | Loss: 2.8718 | LR: 5.00e-05 | 4660.1ms/step | 4.3 samples/s
192
+ Step 44130/44300 | Loss: 2.8874 | LR: 5.00e-05 | 4802.1ms/step | 4.2 samples/s
193
+ Step 44140/44300 | Loss: 2.8779 | LR: 5.00e-05 | 4847.0ms/step | 4.1 samples/s
194
+ Step 44150/44300 | Loss: 2.8809 | LR: 5.00e-05 | 4821.3ms/step | 4.1 samples/s
195
+ Step 44160/44300 | Loss: 2.8657 | LR: 5.00e-05 | 4960.3ms/step | 4.0 samples/s
196
+ Step 44170/44300 | Loss: 2.8644 | LR: 5.00e-05 | 4898.8ms/step | 4.1 samples/s
197
+ Validation at step 44175: Loss = 2.8954, Perplexity = 18.09
198
+ Step 44180/44300 | Loss: 2.8978 | LR: 5.00e-05 | 13833.2ms/step | 1.4 samples/s
199
+ Step 44190/44300 | Loss: 2.9055 | LR: 5.00e-05 | 4937.5ms/step | 4.1 samples/s
200
+ Step 44200/44300 | Loss: 2.9020 | LR: 5.00e-05 | 5036.3ms/step | 4.0 samples/s
201
+ Step 44210/44300 | Loss: 2.8823 | LR: 5.00e-05 | 4872.2ms/step | 4.1 samples/s
202
+ Step 44220/44300 | Loss: 2.8560 | LR: 5.00e-05 | 4955.6ms/step | 4.0 samples/s
203
+ Step 44230/44300 | Loss: 2.8620 | LR: 5.00e-05 | 4914.8ms/step | 4.1 samples/s
204
+ Step 44240/44300 | Loss: 2.8839 | LR: 5.00e-05 | 4879.3ms/step | 4.1 samples/s
205
+ Step 44250/44300 | Loss: 2.9038 | LR: 5.00e-05 | 4830.3ms/step | 4.1 samples/s
206
+ Validation at step 44250: Loss = 2.8954, Perplexity = 18.09
207
+ Step 44260/44300 | Loss: 2.8815 | LR: 5.00e-05 | 14027.3ms/step | 1.4 samples/s
208
+ Step 44270/44300 | Loss: 2.8638 | LR: 5.00e-05 | 4945.2ms/step | 4.0 samples/s
209
+ Step 44280/44300 | Loss: 2.8777 | LR: 5.00e-05 | 4881.2ms/step | 4.1 samples/s
210
+ Step 44290/44300 | Loss: 2.8909 | LR: 5.00e-05 | 4496.1ms/step | 4.4 samples/s
211
+ Step 44300/44300 | Loss: 2.8968 | LR: 5.00e-05 | 3747.9ms/step | 5.3 samples/s
212
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/ropectl_calib_a1p0/run_20260819_144428/checkpoint-44300
213
+ Training complete!
epfl_final/train_logs/ropectl_calib/a1p5.log ADDED
@@ -0,0 +1,213 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ New run: Checkpoints will be saved to /mloscratch/homes/navasard/sparse/checkpoints/ropectl_calib_a1p5/run_20260819_144428
2
+ Running config:
3
+ {
4
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
5
+ "model_dtype": "bfloat16",
6
+ "init_from_scratch": false,
7
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
8
+ "val_path": "/mloscratch/homes/navasard/data/fineweb_val_seed42",
9
+ "max_seq_length": 8192,
10
+ "num_workers": 8,
11
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/ropectl_calib_a1p5/run_20260819_144428",
12
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000",
13
+ "force_new_run": true,
14
+ "num_epochs": 1,
15
+ "max_steps": 44300,
16
+ "per_device_batch_size": 2,
17
+ "gradient_accumulation_steps": 10,
18
+ "learning_rate": 5e-05,
19
+ "weight_decay": 0.01,
20
+ "warmup_steps": 0,
21
+ "lr_scheduler": "constant",
22
+ "max_grad_norm": 1.0,
23
+ "save_steps": 300,
24
+ "eval_steps": 75,
25
+ "eval_batches": 50,
26
+ "logging_steps": 10,
27
+ "seed": 42,
28
+ "verbose": true,
29
+ "checkpoint_activations": false,
30
+ "strategy": "ddp",
31
+ "mixed_precision": "bf16",
32
+ "wandb_enabled": false,
33
+ "wandb_project": "stochastic_sparsity",
34
+ "wandb_name": "ropectl_calib_a1p5",
35
+ "wandb_tags": [
36
+ "qwen3",
37
+ "365m",
38
+ "8k",
39
+ "scaleladder",
40
+ "dense"
41
+ ],
42
+ "hidden_size": 768,
43
+ "intermediate_size": 3072,
44
+ "num_attention_heads": 6,
45
+ "num_key_value_heads": 2,
46
+ "num_hidden_layers": 28,
47
+ "head_dim": null,
48
+ "attention_dropout": 0.0,
49
+ "rope_train_position_scale_set": [
50
+ 1.0,
51
+ 0.9844509232264333,
52
+ 0.9538239538239538,
53
+ 0.9130434782608695,
54
+ 0.8095238095238093
55
+ ],
56
+ "attention_type": "sparse_stochastic",
57
+ "sparse_block_size": 128,
58
+ "sparse_num_random_blocks": 31,
59
+ "sparse_mask_pool_size": 1,
60
+ "sparse_val_dense_compare": false,
61
+ "sparse_layer_indices": null,
62
+ "sparse_mask_cache_dir": null,
63
+ "sparse_mask_seed": 0,
64
+ "sparse_mixed_nrb_set": null,
65
+ "dense_mode": true,
66
+ "dense_step_intervals": null
67
+ }
68
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
69
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000
70
+ Checkpoint activations: False
71
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
72
+
73
+
74
+ ================================================================================
75
+ MODEL ARCHITECTURE
76
+ ================================================================================
77
+ Model config:
78
+ vocab_size: 151936
79
+ hidden_size: 768
80
+ num_hidden_layers: 28
81
+ num_attention_heads: 6
82
+ num_key_value_heads: 2
83
+ head_dim: 128
84
+ intermediate_size: 3072
85
+ max_position_embeddings: 32768
86
+ rope_theta: 1000000
87
+ rope_train_position_scale_set: [1.0, 0.9844509232264333, 0.9538239538239538, 0.9130434782608695, 0.8095238095238093]
88
+ attention_type: sparse_stochastic
89
+ sparse_block_size: 128
90
+ sparse_num_random_blocks: 31
91
+ sparse_mask_pool_size: 1
92
+ sparse_max_seq_length: 8192
93
+
94
+ Parameter count:
95
+ Total parameters: 358,958,848
96
+ Trainable parameters: 358,958,848
97
+ Non-trainable parameters: 0
98
+ Model size (BF16): 0.72 GB
99
+
100
+ --------------------------------------------------------------------------------
101
+ LAYER-BY-LAYER SUMMARY
102
+ --------------------------------------------------------------------------------
103
+ ================================================================================================================================================================
104
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
105
+ ================================================================================================================================================================
106
+ Qwen3ForCausalLM [2, 8192] -- -- True
107
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
108
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
109
+ │ └─ModuleList: 2-2 -- -- -- True
110
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
111
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
132
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
133
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
134
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
135
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
136
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
137
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
138
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
139
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
140
+ ================================================================================================================================================================
141
+ Total params: 475,645,696
142
+ Trainable params: 475,645,696
143
+ Non-trainable params: 0
144
+ Total mult-adds (Units.MEGABYTES): 951.29
145
+ ================================================================================================================================================================
146
+ Input size (MB): 0.13
147
+ Forward/backward pass size (MB): 31197.23
148
+ Params size (MB): 951.29
149
+ Estimated Total Size (MB): 32148.66
150
+ ================================================================================================================================================================
151
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
152
+ ================================================================================
153
+
154
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
155
+ Using pre-tokenized dataset (no runtime tokenization)
156
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
157
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
158
+ Dataset split: 3027764 train, 159356 validation
159
+ Training examples (packed, train split): 3027764
160
+ Validation examples (packed, val split): 159356
161
+ Per-device batch size: 2
162
+ World size (GPUs): 1
163
+ Gradient accumulation steps: 10
164
+ Effective batch size (seqs/step): 20
165
+ Sequence length: 8192
166
+ Tokens per optimizer step: 163,840
167
+ Micro-batches per epoch (per rank): 1513882
168
+ Optimizer steps per epoch: 151388
169
+ Total optimizer steps: 44300 (capped by max_steps; 0.29 epochs)
170
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000...
171
+ Overriding checkpoint LR with config LR: 5e-05
172
+ Resumed from checkpoint at step 44000 (epoch 0, micro_step 440000)
173
+ Resuming training from step 44000
174
+ Starting training...
175
+ Validation at step 44000: Loss = 2.9127, Perplexity = 18.41
176
+ Switching attention mode for step 44001: dense (dense step interval)
177
+ Step 44010/44300 | Loss: 2.8905 | LR: 5.00e-05 | 120467.3ms/step | 0.2 samples/s
178
+ Step 44020/44300 | Loss: 2.9205 | LR: 5.00e-05 | 5450.3ms/step | 3.7 samples/s
179
+ Validation at step 44025: Loss = 2.8982, Perplexity = 18.14
180
+ Step 44030/44300 | Loss: 2.8619 | LR: 5.00e-05 | 15127.6ms/step | 1.3 samples/s
181
+ Step 44040/44300 | Loss: 2.8999 | LR: 5.00e-05 | 6216.4ms/step | 3.2 samples/s
182
+ Step 44050/44300 | Loss: 2.8852 | LR: 5.00e-05 | 5282.3ms/step | 3.8 samples/s
183
+ Step 44060/44300 | Loss: 2.9144 | LR: 5.00e-05 | 5623.0ms/step | 3.6 samples/s
184
+ Step 44070/44300 | Loss: 2.8722 | LR: 5.00e-05 | 6160.7ms/step | 3.2 samples/s
185
+ Step 44080/44300 | Loss: 2.8802 | LR: 5.00e-05 | 5632.0ms/step | 3.6 samples/s
186
+ Step 44090/44300 | Loss: 2.9070 | LR: 5.00e-05 | 5213.6ms/step | 3.8 samples/s
187
+ Step 44100/44300 | Loss: 2.9057 | LR: 5.00e-05 | 5049.6ms/step | 4.0 samples/s
188
+ Validation at step 44100: Loss = 2.8963, Perplexity = 18.11
189
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/ropectl_calib_a1p5/run_20260819_144428/checkpoint-44100
190
+ Step 44110/44300 | Loss: 2.8852 | LR: 5.00e-05 | 19937.7ms/step | 1.0 samples/s
191
+ Step 44120/44300 | Loss: 2.8730 | LR: 5.00e-05 | 5213.6ms/step | 3.8 samples/s
192
+ Step 44130/44300 | Loss: 2.8887 | LR: 5.00e-05 | 5564.4ms/step | 3.6 samples/s
193
+ Step 44140/44300 | Loss: 2.8792 | LR: 5.00e-05 | 5126.2ms/step | 3.9 samples/s
194
+ Step 44150/44300 | Loss: 2.8823 | LR: 5.00e-05 | 4782.9ms/step | 4.2 samples/s
195
+ Step 44160/44300 | Loss: 2.8670 | LR: 5.00e-05 | 4742.0ms/step | 4.2 samples/s
196
+ Step 44170/44300 | Loss: 2.8656 | LR: 5.00e-05 | 5169.8ms/step | 3.9 samples/s
197
+ Validation at step 44175: Loss = 2.8965, Perplexity = 18.11
198
+ Step 44180/44300 | Loss: 2.8992 | LR: 5.00e-05 | 15065.0ms/step | 1.3 samples/s
199
+ Step 44190/44300 | Loss: 2.9065 | LR: 5.00e-05 | 4976.8ms/step | 4.0 samples/s
200
+ Step 44200/44300 | Loss: 2.9033 | LR: 5.00e-05 | 5575.3ms/step | 3.6 samples/s
201
+ Step 44210/44300 | Loss: 2.8835 | LR: 5.00e-05 | 5101.6ms/step | 3.9 samples/s
202
+ Step 44220/44300 | Loss: 2.8571 | LR: 5.00e-05 | 4878.4ms/step | 4.1 samples/s
203
+ Step 44230/44300 | Loss: 2.8632 | LR: 5.00e-05 | 5076.5ms/step | 3.9 samples/s
204
+ Step 44240/44300 | Loss: 2.8851 | LR: 5.00e-05 | 5499.1ms/step | 3.6 samples/s
205
+ Step 44250/44300 | Loss: 2.9051 | LR: 5.00e-05 | 5201.1ms/step | 3.8 samples/s
206
+ Validation at step 44250: Loss = 2.8965, Perplexity = 18.11
207
+ Step 44260/44300 | Loss: 2.8827 | LR: 5.00e-05 | 13822.4ms/step | 1.4 samples/s
208
+ Step 44270/44300 | Loss: 2.8651 | LR: 5.00e-05 | 3723.6ms/step | 5.4 samples/s
209
+ Step 44280/44300 | Loss: 2.8789 | LR: 5.00e-05 | 3722.7ms/step | 5.4 samples/s
210
+ Step 44290/44300 | Loss: 2.8922 | LR: 5.00e-05 | 3719.5ms/step | 5.4 samples/s
211
+ Step 44300/44300 | Loss: 2.8980 | LR: 5.00e-05 | 3718.6ms/step | 5.4 samples/s
212
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/ropectl_calib_a1p5/run_20260819_144428/checkpoint-44300
213
+ Training complete!
epfl_final/train_logs/ropectl_calib/a2p0.log ADDED
@@ -0,0 +1,213 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ New run: Checkpoints will be saved to /mloscratch/homes/navasard/sparse/checkpoints/ropectl_calib_a2p0/run_20260819_144428
2
+ Running config:
3
+ {
4
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
5
+ "model_dtype": "bfloat16",
6
+ "init_from_scratch": false,
7
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
8
+ "val_path": "/mloscratch/homes/navasard/data/fineweb_val_seed42",
9
+ "max_seq_length": 8192,
10
+ "num_workers": 8,
11
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/ropectl_calib_a2p0/run_20260819_144428",
12
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000",
13
+ "force_new_run": true,
14
+ "num_epochs": 1,
15
+ "max_steps": 44300,
16
+ "per_device_batch_size": 2,
17
+ "gradient_accumulation_steps": 10,
18
+ "learning_rate": 5e-05,
19
+ "weight_decay": 0.01,
20
+ "warmup_steps": 0,
21
+ "lr_scheduler": "constant",
22
+ "max_grad_norm": 1.0,
23
+ "save_steps": 300,
24
+ "eval_steps": 75,
25
+ "eval_batches": 50,
26
+ "logging_steps": 10,
27
+ "seed": 42,
28
+ "verbose": true,
29
+ "checkpoint_activations": false,
30
+ "strategy": "ddp",
31
+ "mixed_precision": "bf16",
32
+ "wandb_enabled": false,
33
+ "wandb_project": "stochastic_sparsity",
34
+ "wandb_name": "ropectl_calib_a2p0",
35
+ "wandb_tags": [
36
+ "qwen3",
37
+ "365m",
38
+ "8k",
39
+ "scaleladder",
40
+ "dense"
41
+ ],
42
+ "hidden_size": 768,
43
+ "intermediate_size": 3072,
44
+ "num_attention_heads": 6,
45
+ "num_key_value_heads": 2,
46
+ "num_hidden_layers": 28,
47
+ "head_dim": null,
48
+ "attention_dropout": 0.0,
49
+ "rope_train_position_scale_set": [
50
+ 1.0,
51
+ 0.9792678976352445,
52
+ 0.9384319384319384,
53
+ 0.8840579710144927,
54
+ 0.7460317460317458
55
+ ],
56
+ "attention_type": "sparse_stochastic",
57
+ "sparse_block_size": 128,
58
+ "sparse_num_random_blocks": 31,
59
+ "sparse_mask_pool_size": 1,
60
+ "sparse_val_dense_compare": false,
61
+ "sparse_layer_indices": null,
62
+ "sparse_mask_cache_dir": null,
63
+ "sparse_mask_seed": 0,
64
+ "sparse_mixed_nrb_set": null,
65
+ "dense_mode": true,
66
+ "dense_step_intervals": null
67
+ }
68
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
69
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000
70
+ Checkpoint activations: False
71
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
72
+
73
+
74
+ ================================================================================
75
+ MODEL ARCHITECTURE
76
+ ================================================================================
77
+ Model config:
78
+ vocab_size: 151936
79
+ hidden_size: 768
80
+ num_hidden_layers: 28
81
+ num_attention_heads: 6
82
+ num_key_value_heads: 2
83
+ head_dim: 128
84
+ intermediate_size: 3072
85
+ max_position_embeddings: 32768
86
+ rope_theta: 1000000
87
+ rope_train_position_scale_set: [1.0, 0.9792678976352445, 0.9384319384319384, 0.8840579710144927, 0.7460317460317458]
88
+ attention_type: sparse_stochastic
89
+ sparse_block_size: 128
90
+ sparse_num_random_blocks: 31
91
+ sparse_mask_pool_size: 1
92
+ sparse_max_seq_length: 8192
93
+
94
+ Parameter count:
95
+ Total parameters: 358,958,848
96
+ Trainable parameters: 358,958,848
97
+ Non-trainable parameters: 0
98
+ Model size (BF16): 0.72 GB
99
+
100
+ --------------------------------------------------------------------------------
101
+ LAYER-BY-LAYER SUMMARY
102
+ --------------------------------------------------------------------------------
103
+ ================================================================================================================================================================
104
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
105
+ ================================================================================================================================================================
106
+ Qwen3ForCausalLM [2, 8192] -- -- True
107
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
108
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
109
+ │ └─ModuleList: 2-2 -- -- -- True
110
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
111
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
132
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
133
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
134
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
135
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
136
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
137
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
138
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
139
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
140
+ ================================================================================================================================================================
141
+ Total params: 475,645,696
142
+ Trainable params: 475,645,696
143
+ Non-trainable params: 0
144
+ Total mult-adds (Units.MEGABYTES): 951.29
145
+ ================================================================================================================================================================
146
+ Input size (MB): 0.13
147
+ Forward/backward pass size (MB): 31197.23
148
+ Params size (MB): 951.29
149
+ Estimated Total Size (MB): 32148.66
150
+ ================================================================================================================================================================
151
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
152
+ ================================================================================
153
+
154
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
155
+ Using pre-tokenized dataset (no runtime tokenization)
156
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
157
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
158
+ Dataset split: 3027764 train, 159356 validation
159
+ Training examples (packed, train split): 3027764
160
+ Validation examples (packed, val split): 159356
161
+ Per-device batch size: 2
162
+ World size (GPUs): 1
163
+ Gradient accumulation steps: 10
164
+ Effective batch size (seqs/step): 20
165
+ Sequence length: 8192
166
+ Tokens per optimizer step: 163,840
167
+ Micro-batches per epoch (per rank): 1513882
168
+ Optimizer steps per epoch: 151388
169
+ Total optimizer steps: 44300 (capped by max_steps; 0.29 epochs)
170
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000...
171
+ Overriding checkpoint LR with config LR: 5e-05
172
+ Resumed from checkpoint at step 44000 (epoch 0, micro_step 440000)
173
+ Resuming training from step 44000
174
+ Starting training...
175
+ Validation at step 44000: Loss = 2.9127, Perplexity = 18.41
176
+ Switching attention mode for step 44001: dense (dense step interval)
177
+ Step 44010/44300 | Loss: 2.8947 | LR: 5.00e-05 | 118180.4ms/step | 0.2 samples/s
178
+ Step 44020/44300 | Loss: 2.9234 | LR: 5.00e-05 | 5022.6ms/step | 4.0 samples/s
179
+ Validation at step 44025: Loss = 2.9001, Perplexity = 18.18
180
+ Step 44030/44300 | Loss: 2.8645 | LR: 5.00e-05 | 13763.6ms/step | 1.5 samples/s
181
+ Step 44040/44300 | Loss: 2.9021 | LR: 5.00e-05 | 4782.3ms/step | 4.2 samples/s
182
+ Step 44050/44300 | Loss: 2.8876 | LR: 5.00e-05 | 5159.4ms/step | 3.9 samples/s
183
+ Step 44060/44300 | Loss: 2.9166 | LR: 5.00e-05 | 5044.8ms/step | 4.0 samples/s
184
+ Step 44070/44300 | Loss: 2.8742 | LR: 5.00e-05 | 4986.3ms/step | 4.0 samples/s
185
+ Step 44080/44300 | Loss: 2.8825 | LR: 5.00e-05 | 4901.5ms/step | 4.1 samples/s
186
+ Step 44090/44300 | Loss: 2.9090 | LR: 5.00e-05 | 4868.6ms/step | 4.1 samples/s
187
+ Step 44100/44300 | Loss: 2.9076 | LR: 5.00e-05 | 4874.1ms/step | 4.1 samples/s
188
+ Validation at step 44100: Loss = 2.8978, Perplexity = 18.13
189
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/ropectl_calib_a2p0/run_20260819_144428/checkpoint-44100
190
+ Step 44110/44300 | Loss: 2.8870 | LR: 5.00e-05 | 14735.0ms/step | 1.4 samples/s
191
+ Step 44120/44300 | Loss: 2.8749 | LR: 5.00e-05 | 4879.9ms/step | 4.1 samples/s
192
+ Step 44130/44300 | Loss: 2.8908 | LR: 5.00e-05 | 4600.9ms/step | 4.3 samples/s
193
+ Step 44140/44300 | Loss: 2.8812 | LR: 5.00e-05 | 4840.7ms/step | 4.1 samples/s
194
+ Step 44150/44300 | Loss: 2.8845 | LR: 5.00e-05 | 4783.1ms/step | 4.2 samples/s
195
+ Step 44160/44300 | Loss: 2.8691 | LR: 5.00e-05 | 4584.8ms/step | 4.4 samples/s
196
+ Step 44170/44300 | Loss: 2.8675 | LR: 5.00e-05 | 4688.3ms/step | 4.3 samples/s
197
+ Validation at step 44175: Loss = 2.8982, Perplexity = 18.14
198
+ Step 44180/44300 | Loss: 2.9012 | LR: 5.00e-05 | 13523.1ms/step | 1.5 samples/s
199
+ Step 44190/44300 | Loss: 2.9083 | LR: 5.00e-05 | 4710.1ms/step | 4.2 samples/s
200
+ Step 44200/44300 | Loss: 2.9051 | LR: 5.00e-05 | 4681.0ms/step | 4.3 samples/s
201
+ Step 44210/44300 | Loss: 2.8855 | LR: 5.00e-05 | 4810.9ms/step | 4.2 samples/s
202
+ Step 44220/44300 | Loss: 2.8588 | LR: 5.00e-05 | 4766.6ms/step | 4.2 samples/s
203
+ Step 44230/44300 | Loss: 2.8650 | LR: 5.00e-05 | 4759.6ms/step | 4.2 samples/s
204
+ Step 44240/44300 | Loss: 2.8868 | LR: 5.00e-05 | 4767.4ms/step | 4.2 samples/s
205
+ Step 44250/44300 | Loss: 2.9070 | LR: 5.00e-05 | 4738.9ms/step | 4.2 samples/s
206
+ Validation at step 44250: Loss = 2.8982, Perplexity = 18.14
207
+ Step 44260/44300 | Loss: 2.8847 | LR: 5.00e-05 | 13481.8ms/step | 1.5 samples/s
208
+ Step 44270/44300 | Loss: 2.8670 | LR: 5.00e-05 | 4691.0ms/step | 4.3 samples/s
209
+ Step 44280/44300 | Loss: 2.8809 | LR: 5.00e-05 | 4732.0ms/step | 4.2 samples/s
210
+ Step 44290/44300 | Loss: 2.8941 | LR: 5.00e-05 | 4587.0ms/step | 4.4 samples/s
211
+ Step 44300/44300 | Loss: 2.8998 | LR: 5.00e-05 | 4586.5ms/step | 4.4 samples/s
212
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/ropectl_calib_a2p0/run_20260819_144428/checkpoint-44300
213
+ Training complete!
epfl_final/train_logs/ropectl_calib/a3p0.log ADDED
@@ -0,0 +1,213 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ New run: Checkpoints will be saved to /mloscratch/homes/navasard/sparse/checkpoints/ropectl_calib_a3p0/run_20260819_144428
2
+ Running config:
3
+ {
4
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
5
+ "model_dtype": "bfloat16",
6
+ "init_from_scratch": false,
7
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
8
+ "val_path": "/mloscratch/homes/navasard/data/fineweb_val_seed42",
9
+ "max_seq_length": 8192,
10
+ "num_workers": 8,
11
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/ropectl_calib_a3p0/run_20260819_144428",
12
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000",
13
+ "force_new_run": true,
14
+ "num_epochs": 1,
15
+ "max_steps": 44300,
16
+ "per_device_batch_size": 2,
17
+ "gradient_accumulation_steps": 10,
18
+ "learning_rate": 5e-05,
19
+ "weight_decay": 0.01,
20
+ "warmup_steps": 0,
21
+ "lr_scheduler": "constant",
22
+ "max_grad_norm": 1.0,
23
+ "save_steps": 300,
24
+ "eval_steps": 75,
25
+ "eval_batches": 50,
26
+ "logging_steps": 10,
27
+ "seed": 42,
28
+ "verbose": true,
29
+ "checkpoint_activations": false,
30
+ "strategy": "ddp",
31
+ "mixed_precision": "bf16",
32
+ "wandb_enabled": false,
33
+ "wandb_project": "stochastic_sparsity",
34
+ "wandb_name": "ropectl_calib_a3p0",
35
+ "wandb_tags": [
36
+ "qwen3",
37
+ "365m",
38
+ "8k",
39
+ "scaleladder",
40
+ "dense"
41
+ ],
42
+ "hidden_size": 768,
43
+ "intermediate_size": 3072,
44
+ "num_attention_heads": 6,
45
+ "num_key_value_heads": 2,
46
+ "num_hidden_layers": 28,
47
+ "head_dim": null,
48
+ "attention_dropout": 0.0,
49
+ "rope_train_position_scale_set": [
50
+ 1.0,
51
+ 0.9689018464528667,
52
+ 0.9076479076479076,
53
+ 0.826086956521739,
54
+ 0.6190476190476187
55
+ ],
56
+ "attention_type": "sparse_stochastic",
57
+ "sparse_block_size": 128,
58
+ "sparse_num_random_blocks": 31,
59
+ "sparse_mask_pool_size": 1,
60
+ "sparse_val_dense_compare": false,
61
+ "sparse_layer_indices": null,
62
+ "sparse_mask_cache_dir": null,
63
+ "sparse_mask_seed": 0,
64
+ "sparse_mixed_nrb_set": null,
65
+ "dense_mode": true,
66
+ "dense_step_intervals": null
67
+ }
68
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
69
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000
70
+ Checkpoint activations: False
71
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
72
+
73
+
74
+ ================================================================================
75
+ MODEL ARCHITECTURE
76
+ ================================================================================
77
+ Model config:
78
+ vocab_size: 151936
79
+ hidden_size: 768
80
+ num_hidden_layers: 28
81
+ num_attention_heads: 6
82
+ num_key_value_heads: 2
83
+ head_dim: 128
84
+ intermediate_size: 3072
85
+ max_position_embeddings: 32768
86
+ rope_theta: 1000000
87
+ rope_train_position_scale_set: [1.0, 0.9689018464528667, 0.9076479076479076, 0.826086956521739, 0.6190476190476187]
88
+ attention_type: sparse_stochastic
89
+ sparse_block_size: 128
90
+ sparse_num_random_blocks: 31
91
+ sparse_mask_pool_size: 1
92
+ sparse_max_seq_length: 8192
93
+
94
+ Parameter count:
95
+ Total parameters: 358,958,848
96
+ Trainable parameters: 358,958,848
97
+ Non-trainable parameters: 0
98
+ Model size (BF16): 0.72 GB
99
+
100
+ --------------------------------------------------------------------------------
101
+ LAYER-BY-LAYER SUMMARY
102
+ --------------------------------------------------------------------------------
103
+ ================================================================================================================================================================
104
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
105
+ ================================================================================================================================================================
106
+ Qwen3ForCausalLM [2, 8192] -- -- True
107
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
108
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
109
+ │ └─ModuleList: 2-2 -- -- -- True
110
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
111
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
132
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
133
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
134
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
135
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
136
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
137
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
138
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
139
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
140
+ ================================================================================================================================================================
141
+ Total params: 475,645,696
142
+ Trainable params: 475,645,696
143
+ Non-trainable params: 0
144
+ Total mult-adds (Units.MEGABYTES): 951.29
145
+ ================================================================================================================================================================
146
+ Input size (MB): 0.13
147
+ Forward/backward pass size (MB): 31197.23
148
+ Params size (MB): 951.29
149
+ Estimated Total Size (MB): 32148.66
150
+ ================================================================================================================================================================
151
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
152
+ ================================================================================
153
+
154
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
155
+ Using pre-tokenized dataset (no runtime tokenization)
156
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
157
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
158
+ Dataset split: 3027764 train, 159356 validation
159
+ Training examples (packed, train split): 3027764
160
+ Validation examples (packed, val split): 159356
161
+ Per-device batch size: 2
162
+ World size (GPUs): 1
163
+ Gradient accumulation steps: 10
164
+ Effective batch size (seqs/step): 20
165
+ Sequence length: 8192
166
+ Tokens per optimizer step: 163,840
167
+ Micro-batches per epoch (per rank): 1513882
168
+ Optimizer steps per epoch: 151388
169
+ Total optimizer steps: 44300 (capped by max_steps; 0.29 epochs)
170
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000...
171
+ Overriding checkpoint LR with config LR: 5e-05
172
+ Resumed from checkpoint at step 44000 (epoch 0, micro_step 440000)
173
+ Resuming training from step 44000
174
+ Starting training...
175
+ Validation at step 44000: Loss = 2.9127, Perplexity = 18.41
176
+ Switching attention mode for step 44001: dense (dense step interval)
177
+ Step 44010/44300 | Loss: 2.9119 | LR: 5.00e-05 | 119135.0ms/step | 0.2 samples/s
178
+ Step 44020/44300 | Loss: 2.9345 | LR: 5.00e-05 | 5052.9ms/step | 4.0 samples/s
179
+ Validation at step 44025: Loss = 2.9085, Perplexity = 18.33
180
+ Step 44030/44300 | Loss: 2.8745 | LR: 5.00e-05 | 13709.8ms/step | 1.5 samples/s
181
+ Step 44040/44300 | Loss: 2.9102 | LR: 5.00e-05 | 4884.2ms/step | 4.1 samples/s
182
+ Step 44050/44300 | Loss: 2.8963 | LR: 5.00e-05 | 4965.9ms/step | 4.0 samples/s
183
+ Step 44060/44300 | Loss: 2.9242 | LR: 5.00e-05 | 4832.4ms/step | 4.1 samples/s
184
+ Step 44070/44300 | Loss: 2.8813 | LR: 5.00e-05 | 4901.6ms/step | 4.1 samples/s
185
+ Step 44080/44300 | Loss: 2.8910 | LR: 5.00e-05 | 5078.6ms/step | 3.9 samples/s
186
+ Step 44090/44300 | Loss: 2.9165 | LR: 5.00e-05 | 4757.9ms/step | 4.2 samples/s
187
+ Step 44100/44300 | Loss: 2.9146 | LR: 5.00e-05 | 4922.5ms/step | 4.1 samples/s
188
+ Validation at step 44100: Loss = 2.9026, Perplexity = 18.22
189
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/ropectl_calib_a3p0/run_20260819_144428/checkpoint-44100
190
+ Step 44110/44300 | Loss: 2.8936 | LR: 5.00e-05 | 13929.7ms/step | 1.4 samples/s
191
+ Step 44120/44300 | Loss: 2.8817 | LR: 5.00e-05 | 4671.1ms/step | 4.3 samples/s
192
+ Step 44130/44300 | Loss: 2.8981 | LR: 5.00e-05 | 4507.7ms/step | 4.4 samples/s
193
+ Step 44140/44300 | Loss: 2.8883 | LR: 5.00e-05 | 4717.0ms/step | 4.2 samples/s
194
+ Step 44150/44300 | Loss: 2.8919 | LR: 5.00e-05 | 4614.0ms/step | 4.3 samples/s
195
+ Step 44160/44300 | Loss: 2.8765 | LR: 5.00e-05 | 4569.6ms/step | 4.4 samples/s
196
+ Step 44170/44300 | Loss: 2.8740 | LR: 5.00e-05 | 4583.4ms/step | 4.4 samples/s
197
+ Validation at step 44175: Loss = 2.9036, Perplexity = 18.24
198
+ Step 44180/44300 | Loss: 2.9087 | LR: 5.00e-05 | 13416.7ms/step | 1.5 samples/s
199
+ Step 44190/44300 | Loss: 2.9145 | LR: 5.00e-05 | 4648.9ms/step | 4.3 samples/s
200
+ Step 44200/44300 | Loss: 2.9117 | LR: 5.00e-05 | 4661.5ms/step | 4.3 samples/s
201
+ Step 44210/44300 | Loss: 2.8922 | LR: 5.00e-05 | 4684.6ms/step | 4.3 samples/s
202
+ Step 44220/44300 | Loss: 2.8649 | LR: 5.00e-05 | 4713.9ms/step | 4.2 samples/s
203
+ Step 44230/44300 | Loss: 2.8716 | LR: 5.00e-05 | 4875.5ms/step | 4.1 samples/s
204
+ Step 44240/44300 | Loss: 2.8929 | LR: 5.00e-05 | 4787.4ms/step | 4.2 samples/s
205
+ Step 44250/44300 | Loss: 2.9136 | LR: 5.00e-05 | 4764.2ms/step | 4.2 samples/s
206
+ Validation at step 44250: Loss = 2.9037, Perplexity = 18.24
207
+ Step 44260/44300 | Loss: 2.8914 | LR: 5.00e-05 | 13293.6ms/step | 1.5 samples/s
208
+ Step 44270/44300 | Loss: 2.8740 | LR: 5.00e-05 | 4685.5ms/step | 4.3 samples/s
209
+ Step 44280/44300 | Loss: 2.8878 | LR: 5.00e-05 | 4658.9ms/step | 4.3 samples/s
210
+ Step 44290/44300 | Loss: 2.9009 | LR: 5.00e-05 | 4632.9ms/step | 4.3 samples/s
211
+ Step 44300/44300 | Loss: 2.9061 | LR: 5.00e-05 | 4650.2ms/step | 4.3 samples/s
212
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/ropectl_calib_a3p0/run_20260819_144428/checkpoint-44300
213
+ Training complete!
epfl_final/train_logs/ropectl_full_a1p0/seed42.log ADDED
@@ -0,0 +1,271 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ New run: Checkpoints will be saved to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735
2
+ Running config:
3
+ {
4
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
5
+ "model_dtype": "bfloat16",
6
+ "init_from_scratch": true,
7
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
8
+ "val_path": null,
9
+ "max_seq_length": 8192,
10
+ "num_workers": 8,
11
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed42/run_20260819_153735",
12
+ "resume_from_checkpoint": null,
13
+ "force_new_run": true,
14
+ "num_epochs": 1,
15
+ "max_steps": 44000,
16
+ "per_device_batch_size": 2,
17
+ "gradient_accumulation_steps": 10,
18
+ "learning_rate": 0.0005,
19
+ "weight_decay": 0.01,
20
+ "warmup_steps": 750,
21
+ "lr_scheduler": "cosine",
22
+ "max_grad_norm": 1.0,
23
+ "save_steps": 1000,
24
+ "eval_steps": 1000,
25
+ "eval_batches": 50,
26
+ "logging_steps": 10,
27
+ "seed": 42,
28
+ "verbose": true,
29
+ "checkpoint_activations": false,
30
+ "strategy": "ddp",
31
+ "mixed_precision": "bf16",
32
+ "wandb_enabled": false,
33
+ "wandb_project": "stochastic_sparsity",
34
+ "wandb_name": "l8k365m_dense_ropectl_a1p0_seed42",
35
+ "wandb_tags": [
36
+ "qwen3",
37
+ "365m",
38
+ "8k",
39
+ "dense",
40
+ "rope-control"
41
+ ],
42
+ "hidden_size": 768,
43
+ "intermediate_size": 3072,
44
+ "num_attention_heads": 6,
45
+ "num_key_value_heads": 2,
46
+ "num_hidden_layers": 28,
47
+ "head_dim": null,
48
+ "attention_dropout": 0.0,
49
+ "rope_train_position_scale_set": [
50
+ 1.0,
51
+ 0.9896339488176222,
52
+ 0.9692159692159692,
53
+ 0.9420289855072463,
54
+ 0.8730158730158729
55
+ ],
56
+ "attention_type": "sparse_stochastic",
57
+ "sparse_block_size": 128,
58
+ "sparse_num_random_blocks": 31,
59
+ "sparse_mask_pool_size": 1,
60
+ "sparse_val_dense_compare": false,
61
+ "sparse_layer_indices": null,
62
+ "sparse_mask_cache_dir": null,
63
+ "sparse_mask_seed": 0,
64
+ "sparse_mixed_nrb_set": null,
65
+ "dense_mode": true,
66
+ "dense_step_intervals": null
67
+ }
68
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
69
+ Initializing model from scratch (random weights)...
70
+ Checkpoint activations: False
71
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
72
+
73
+
74
+ ================================================================================
75
+ MODEL ARCHITECTURE
76
+ ================================================================================
77
+ Model config:
78
+ vocab_size: 151936
79
+ hidden_size: 768
80
+ num_hidden_layers: 28
81
+ num_attention_heads: 6
82
+ num_key_value_heads: 2
83
+ head_dim: 128
84
+ intermediate_size: 3072
85
+ max_position_embeddings: 32768
86
+ rope_theta: 1000000
87
+ rope_train_position_scale_set: [1.0, 0.9896339488176222, 0.9692159692159692, 0.9420289855072463, 0.8730158730158729]
88
+ attention_type: sparse_stochastic
89
+ sparse_block_size: 128
90
+ sparse_num_random_blocks: 31
91
+ sparse_mask_pool_size: 1
92
+ sparse_max_seq_length: 8192
93
+
94
+ Parameter count:
95
+ Total parameters: 358,958,848
96
+ Trainable parameters: 358,958,848
97
+ Non-trainable parameters: 0
98
+ Model size (BF16): 0.72 GB
99
+
100
+ --------------------------------------------------------------------------------
101
+ LAYER-BY-LAYER SUMMARY
102
+ --------------------------------------------------------------------------------
103
+ ================================================================================================================================================================
104
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
105
+ ================================================================================================================================================================
106
+ Qwen3ForCausalLM [2, 8192] -- -- True
107
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
108
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
109
+ │ └─ModuleList: 2-2 -- -- -- True
110
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
111
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
132
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
133
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
134
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
135
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
136
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
137
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
138
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
139
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
140
+ ================================================================================================================================================================
141
+ Total params: 475,645,696
142
+ Trainable params: 475,645,696
143
+ Non-trainable params: 0
144
+ Total mult-adds (Units.MEGABYTES): 951.29
145
+ ================================================================================================================================================================
146
+ Input size (MB): 0.13
147
+ Forward/backward pass size (MB): 31197.23
148
+ Params size (MB): 951.29
149
+ Estimated Total Size (MB): 32148.66
150
+ ================================================================================================================================================================
151
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
152
+ ================================================================================
153
+
154
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
155
+ Using pre-tokenized dataset (no runtime tokenization)
156
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
157
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
158
+ Dataset split: 3027764 train, 159356 validation
159
+ Training examples (packed, train split): 3027764
160
+ Validation examples (packed, val split): 159356
161
+ Per-device batch size: 2
162
+ World size (GPUs): 1
163
+ Gradient accumulation steps: 10
164
+ Effective batch size (seqs/step): 20
165
+ Sequence length: 8192
166
+ Tokens per optimizer step: 163,840
167
+ Micro-batches per epoch (per rank): 1513882
168
+ Optimizer steps per epoch: 151388
169
+ Total optimizer steps: 44000 (capped by max_steps; 0.29 epochs)
170
+ Starting training...
171
+ Switching attention mode for step 1: dense (dense step interval)
172
+ Step 10/44000 | Loss: 12.0395 | LR: 6.67e-06 | 4883.1ms/step | 4.1 samples/s
173
+ Step 20/44000 | Loss: 11.9277 | LR: 1.33e-05 | 5113.5ms/step | 3.9 samples/s
174
+ Step 30/44000 | Loss: 11.5669 | LR: 2.00e-05 | 4770.4ms/step | 4.2 samples/s
175
+ Step 40/44000 | Loss: 11.0214 | LR: 2.67e-05 | 4761.6ms/step | 4.2 samples/s
176
+ Step 50/44000 | Loss: 10.7365 | LR: 3.33e-05 | 4732.7ms/step | 4.2 samples/s
177
+ Step 60/44000 | Loss: 10.4926 | LR: 4.00e-05 | 5058.3ms/step | 4.0 samples/s
178
+ Step 70/44000 | Loss: 10.2585 | LR: 4.67e-05 | 4873.2ms/step | 4.1 samples/s
179
+ Step 80/44000 | Loss: 9.9603 | LR: 5.33e-05 | 5193.8ms/step | 3.9 samples/s
180
+ Step 90/44000 | Loss: 9.6057 | LR: 6.00e-05 | 5418.1ms/step | 3.7 samples/s
181
+ Step 100/44000 | Loss: 9.1661 | LR: 6.67e-05 | 5238.9ms/step | 3.8 samples/s
182
+ Step 110/44000 | Loss: 8.6374 | LR: 7.33e-05 | 5866.4ms/step | 3.4 samples/s
183
+ Step 120/44000 | Loss: 8.1739 | LR: 8.00e-05 | 4938.2ms/step | 4.1 samples/s
184
+ Step 130/44000 | Loss: 7.7860 | LR: 8.67e-05 | 5071.2ms/step | 3.9 samples/s
185
+ Step 140/44000 | Loss: 7.5487 | LR: 9.33e-05 | 5174.6ms/step | 3.9 samples/s
186
+ Step 150/44000 | Loss: 7.3601 | LR: 1.00e-04 | 5783.8ms/step | 3.5 samples/s
187
+ Step 160/44000 | Loss: 7.2463 | LR: 1.07e-04 | 5234.8ms/step | 3.8 samples/s
188
+ Step 170/44000 | Loss: 7.1278 | LR: 1.13e-04 | 4996.6ms/step | 4.0 samples/s
189
+ Step 180/44000 | Loss: 7.0250 | LR: 1.20e-04 | 5094.3ms/step | 3.9 samples/s
190
+ Step 190/44000 | Loss: 6.9282 | LR: 1.27e-04 | 5724.2ms/step | 3.5 samples/s
191
+ Step 200/44000 | Loss: 6.8681 | LR: 1.33e-04 | 5346.8ms/step | 3.7 samples/s
192
+ Step 210/44000 | Loss: 6.8133 | LR: 1.40e-04 | 5064.1ms/step | 3.9 samples/s
193
+ Step 220/44000 | Loss: 6.7121 | LR: 1.47e-04 | 5230.0ms/step | 3.8 samples/s
194
+ Step 230/44000 | Loss: 6.6114 | LR: 1.53e-04 | 5350.5ms/step | 3.7 samples/s
195
+ Step 240/44000 | Loss: 6.5509 | LR: 1.60e-04 | 5285.5ms/step | 3.8 samples/s
196
+ Step 250/44000 | Loss: 6.4644 | LR: 1.67e-04 | 5113.7ms/step | 3.9 samples/s
197
+ Step 260/44000 | Loss: 6.4375 | LR: 1.73e-04 | 5193.9ms/step | 3.9 samples/s
198
+ Step 270/44000 | Loss: 6.3662 | LR: 1.80e-04 | 4973.9ms/step | 4.0 samples/s
199
+ Step 280/44000 | Loss: 6.3324 | LR: 1.87e-04 | 5464.6ms/step | 3.7 samples/s
200
+ Step 290/44000 | Loss: 6.2935 | LR: 1.93e-04 | 5351.2ms/step | 3.7 samples/s
201
+ Step 300/44000 | Loss: 6.2227 | LR: 2.00e-04 | 5222.5ms/step | 3.8 samples/s
202
+ Step 310/44000 | Loss: 6.1932 | LR: 2.07e-04 | 5323.0ms/step | 3.8 samples/s
203
+ Step 320/44000 | Loss: 6.1458 | LR: 2.13e-04 | 5620.9ms/step | 3.6 samples/s
204
+ Step 330/44000 | Loss: 6.0882 | LR: 2.20e-04 | 5285.5ms/step | 3.8 samples/s
205
+ Step 340/44000 | Loss: 6.0720 | LR: 2.27e-04 | 5323.5ms/step | 3.8 samples/s
206
+ Step 350/44000 | Loss: 5.9998 | LR: 2.33e-04 | 5315.2ms/step | 3.8 samples/s
207
+ Step 360/44000 | Loss: 5.9719 | LR: 2.40e-04 | 5312.1ms/step | 3.8 samples/s
208
+ Step 370/44000 | Loss: 5.9381 | LR: 2.47e-04 | 5335.5ms/step | 3.7 samples/s
209
+ Step 380/44000 | Loss: 5.9201 | LR: 2.53e-04 | 5123.4ms/step | 3.9 samples/s
210
+ Step 390/44000 | Loss: 5.8292 | LR: 2.60e-04 | 5223.7ms/step | 3.8 samples/s
211
+ Step 400/44000 | Loss: 5.8476 | LR: 2.67e-04 | 5426.5ms/step | 3.7 samples/s
212
+ Step 410/44000 | Loss: 5.8220 | LR: 2.73e-04 | 5087.2ms/step | 3.9 samples/s
213
+ Step 420/44000 | Loss: 5.8215 | LR: 2.80e-04 | 5022.6ms/step | 4.0 samples/s
214
+ Step 430/44000 | Loss: 5.7436 | LR: 2.87e-04 | 5220.2ms/step | 3.8 samples/s
215
+ Step 440/44000 | Loss: 5.7146 | LR: 2.93e-04 | 5142.7ms/step | 3.9 samples/s
216
+ Step 450/44000 | Loss: 5.6766 | LR: 3.00e-04 | 5458.5ms/step | 3.7 samples/s
217
+ Step 460/44000 | Loss: 5.6101 | LR: 3.07e-04 | 5047.6ms/step | 4.0 samples/s
218
+ Step 470/44000 | Loss: 5.6127 | LR: 3.13e-04 | 4989.4ms/step | 4.0 samples/s
219
+ Step 480/44000 | Loss: 5.5602 | LR: 3.20e-04 | 5143.1ms/step | 3.9 samples/s
220
+ Step 490/44000 | Loss: 5.5210 | LR: 3.27e-04 | 5317.3ms/step | 3.8 samples/s
221
+ Step 500/44000 | Loss: 5.5072 | LR: 3.33e-04 | 4855.9ms/step | 4.1 samples/s
222
+ Step 510/44000 | Loss: 5.4657 | LR: 3.40e-04 | 4861.1ms/step | 4.1 samples/s
223
+ Step 520/44000 | Loss: 5.4204 | LR: 3.47e-04 | 4913.3ms/step | 4.1 samples/s
224
+ Step 530/44000 | Loss: 5.4431 | LR: 3.53e-04 | 4814.3ms/step | 4.2 samples/s
225
+ Step 540/44000 | Loss: 5.3591 | LR: 3.60e-04 | 4929.1ms/step | 4.1 samples/s
226
+ Step 550/44000 | Loss: 5.3290 | LR: 3.67e-04 | 4944.6ms/step | 4.0 samples/s
227
+ Step 560/44000 | Loss: 5.3100 | LR: 3.73e-04 | 4729.3ms/step | 4.2 samples/s
228
+ Step 570/44000 | Loss: 5.3070 | LR: 3.80e-04 | 4809.8ms/step | 4.2 samples/s
229
+ Step 580/44000 | Loss: 5.2510 | LR: 3.87e-04 | 5031.4ms/step | 4.0 samples/s
230
+ Step 590/44000 | Loss: 5.2145 | LR: 3.93e-04 | 4937.5ms/step | 4.1 samples/s
231
+ Step 600/44000 | Loss: 5.2018 | LR: 4.00e-04 | 4797.9ms/step | 4.2 samples/s
232
+ Step 610/44000 | Loss: 5.1676 | LR: 4.07e-04 | 4820.4ms/step | 4.1 samples/s
233
+ Step 620/44000 | Loss: 5.1468 | LR: 4.13e-04 | 4715.6ms/step | 4.2 samples/s
234
+ Step 630/44000 | Loss: 5.0701 | LR: 4.20e-04 | 4904.7ms/step | 4.1 samples/s
235
+ Step 640/44000 | Loss: 5.0461 | LR: 4.27e-04 | 4849.7ms/step | 4.1 samples/s
236
+ Step 650/44000 | Loss: 5.0416 | LR: 4.33e-04 | 4932.6ms/step | 4.1 samples/s
237
+ Step 660/44000 | Loss: 5.0051 | LR: 4.40e-04 | 5000.1ms/step | 4.0 samples/s
238
+ Step 670/44000 | Loss: 4.9702 | LR: 4.47e-04 | 5006.8ms/step | 4.0 samples/s
239
+ Step 680/44000 | Loss: 4.9488 | LR: 4.53e-04 | 5075.8ms/step | 3.9 samples/s
240
+ Step 690/44000 | Loss: 4.9240 | LR: 4.60e-04 | 4917.0ms/step | 4.1 samples/s
241
+ Step 700/44000 | Loss: 4.8984 | LR: 4.67e-04 | 4802.1ms/step | 4.2 samples/s
242
+ Step 710/44000 | Loss: 4.8487 | LR: 4.73e-04 | 4701.5ms/step | 4.3 samples/s
243
+ Step 720/44000 | Loss: 4.7877 | LR: 4.80e-04 | 5057.4ms/step | 4.0 samples/s
244
+ Step 730/44000 | Loss: 4.7968 | LR: 4.87e-04 | 4948.7ms/step | 4.0 samples/s
245
+ Step 740/44000 | Loss: 4.7397 | LR: 4.93e-04 | 4843.8ms/step | 4.1 samples/s
246
+ Step 750/44000 | Loss: 4.7458 | LR: 5.00e-04 | 4910.7ms/step | 4.1 samples/s
247
+ Step 760/44000 | Loss: 4.7154 | LR: 5.00e-04 | 4908.4ms/step | 4.1 samples/s
248
+ Step 770/44000 | Loss: 4.6763 | LR: 5.00e-04 | 4974.8ms/step | 4.0 samples/s
249
+ Step 780/44000 | Loss: 4.6752 | LR: 5.00e-04 | 4902.8ms/step | 4.1 samples/s
250
+ Step 790/44000 | Loss: 4.6364 | LR: 5.00e-04 | 5002.8ms/step | 4.0 samples/s
251
+ Step 800/44000 | Loss: 4.5967 | LR: 5.00e-04 | 5115.8ms/step | 3.9 samples/s
252
+ Step 810/44000 | Loss: 4.5947 | LR: 5.00e-04 | 5245.3ms/step | 3.8 samples/s
253
+ Step 820/44000 | Loss: 4.5604 | LR: 5.00e-04 | 4983.6ms/step | 4.0 samples/s
254
+ Step 830/44000 | Loss: 4.5392 | LR: 5.00e-04 | 5088.9ms/step | 3.9 samples/s
255
+ Step 840/44000 | Loss: 4.5058 | LR: 5.00e-04 | 5098.8ms/step | 3.9 samples/s
256
+ Step 850/44000 | Loss: 4.5017 | LR: 5.00e-04 | 5118.7ms/step | 3.9 samples/s
257
+ Step 860/44000 | Loss: 4.5260 | LR: 5.00e-04 | 5079.8ms/step | 3.9 samples/s
258
+ Step 870/44000 | Loss: 4.4369 | LR: 5.00e-04 | 4828.7ms/step | 4.1 samples/s
259
+ Step 880/44000 | Loss: 4.4471 | LR: 5.00e-04 | 5112.6ms/step | 3.9 samples/s
260
+ Step 890/44000 | Loss: 4.4276 | LR: 5.00e-04 | 4831.0ms/step | 4.1 samples/s
261
+ Step 900/44000 | Loss: 4.4048 | LR: 5.00e-04 | 4808.5ms/step | 4.2 samples/s
262
+ Step 910/44000 | Loss: 4.3736 | LR: 5.00e-04 | 4948.7ms/step | 4.0 samples/s
263
+ Step 920/44000 | Loss: 4.3707 | LR: 5.00e-04 | 4748.5ms/step | 4.2 samples/s
264
+ Step 930/44000 | Loss: 4.3447 | LR: 5.00e-04 | 4300.3ms/step | 4.7 samples/s
265
+ Step 940/44000 | Loss: 4.3358 | LR: 5.00e-04 | 4474.4ms/step | 4.5 samples/s
266
+ Step 950/44000 | Loss: 4.3277 | LR: 5.00e-04 | 4646.0ms/step | 4.3 samples/s
267
+ Step 960/44000 | Loss: 4.2854 | LR: 5.00e-04 | 4344.1ms/step | 4.6 samples/s
268
+ Step 970/44000 | Loss: 4.3107 | LR: 5.00e-04 | 4835.4ms/step | 4.1 samples/s
269
+ Step 980/44000 | Loss: 4.2947 | LR: 5.00e-04 | 4812.0ms/step | 4.2 samples/s
270
+ Step 990/44000 | Loss: 4.2751 | LR: 5.00e-04 | 4406.5ms/step | 4.5 samples/s
271
+ Step 1000/44000 | Loss: 4.2679 | LR: 5.00e-04 | 4713.6ms/step | 4.2 samples/s
epfl_final/train_logs/ropectl_full_a1p0/seed43.log ADDED
@@ -0,0 +1,287 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ New run: Checkpoints will be saved to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735
2
+ Running config:
3
+ {
4
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
5
+ "model_dtype": "bfloat16",
6
+ "init_from_scratch": true,
7
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
8
+ "val_path": null,
9
+ "max_seq_length": 8192,
10
+ "num_workers": 8,
11
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed43/run_20260819_153735",
12
+ "resume_from_checkpoint": null,
13
+ "force_new_run": true,
14
+ "num_epochs": 1,
15
+ "max_steps": 44000,
16
+ "per_device_batch_size": 2,
17
+ "gradient_accumulation_steps": 10,
18
+ "learning_rate": 0.0005,
19
+ "weight_decay": 0.01,
20
+ "warmup_steps": 750,
21
+ "lr_scheduler": "cosine",
22
+ "max_grad_norm": 1.0,
23
+ "save_steps": 1000,
24
+ "eval_steps": 1000,
25
+ "eval_batches": 50,
26
+ "logging_steps": 10,
27
+ "seed": 43,
28
+ "verbose": true,
29
+ "checkpoint_activations": false,
30
+ "strategy": "ddp",
31
+ "mixed_precision": "bf16",
32
+ "wandb_enabled": false,
33
+ "wandb_project": "stochastic_sparsity",
34
+ "wandb_name": "l8k365m_dense_ropectl_a1p0_seed43",
35
+ "wandb_tags": [
36
+ "qwen3",
37
+ "365m",
38
+ "8k",
39
+ "dense",
40
+ "rope-control"
41
+ ],
42
+ "hidden_size": 768,
43
+ "intermediate_size": 3072,
44
+ "num_attention_heads": 6,
45
+ "num_key_value_heads": 2,
46
+ "num_hidden_layers": 28,
47
+ "head_dim": null,
48
+ "attention_dropout": 0.0,
49
+ "rope_train_position_scale_set": [
50
+ 1.0,
51
+ 0.9896339488176222,
52
+ 0.9692159692159692,
53
+ 0.9420289855072463,
54
+ 0.8730158730158729
55
+ ],
56
+ "attention_type": "sparse_stochastic",
57
+ "sparse_block_size": 128,
58
+ "sparse_num_random_blocks": 31,
59
+ "sparse_mask_pool_size": 1,
60
+ "sparse_val_dense_compare": false,
61
+ "sparse_layer_indices": null,
62
+ "sparse_mask_cache_dir": null,
63
+ "sparse_mask_seed": 0,
64
+ "sparse_mixed_nrb_set": null,
65
+ "dense_mode": true,
66
+ "dense_step_intervals": null
67
+ }
68
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
69
+ Initializing model from scratch (random weights)...
70
+ Checkpoint activations: False
71
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
72
+
73
+
74
+ ================================================================================
75
+ MODEL ARCHITECTURE
76
+ ================================================================================
77
+ Model config:
78
+ vocab_size: 151936
79
+ hidden_size: 768
80
+ num_hidden_layers: 28
81
+ num_attention_heads: 6
82
+ num_key_value_heads: 2
83
+ head_dim: 128
84
+ intermediate_size: 3072
85
+ max_position_embeddings: 32768
86
+ rope_theta: 1000000
87
+ rope_train_position_scale_set: [1.0, 0.9896339488176222, 0.9692159692159692, 0.9420289855072463, 0.8730158730158729]
88
+ attention_type: sparse_stochastic
89
+ sparse_block_size: 128
90
+ sparse_num_random_blocks: 31
91
+ sparse_mask_pool_size: 1
92
+ sparse_max_seq_length: 8192
93
+
94
+ Parameter count:
95
+ Total parameters: 358,958,848
96
+ Trainable parameters: 358,958,848
97
+ Non-trainable parameters: 0
98
+ Model size (BF16): 0.72 GB
99
+
100
+ --------------------------------------------------------------------------------
101
+ LAYER-BY-LAYER SUMMARY
102
+ --------------------------------------------------------------------------------
103
+ ================================================================================================================================================================
104
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
105
+ ================================================================================================================================================================
106
+ Qwen3ForCausalLM [2, 8192] -- -- True
107
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
108
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
109
+ │ └─ModuleList: 2-2 -- -- -- True
110
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
111
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
132
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
133
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
134
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
135
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
136
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
137
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
138
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
139
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
140
+ ================================================================================================================================================================
141
+ Total params: 475,645,696
142
+ Trainable params: 475,645,696
143
+ Non-trainable params: 0
144
+ Total mult-adds (Units.MEGABYTES): 951.29
145
+ ================================================================================================================================================================
146
+ Input size (MB): 0.13
147
+ Forward/backward pass size (MB): 31197.23
148
+ Params size (MB): 951.29
149
+ Estimated Total Size (MB): 32148.66
150
+ ================================================================================================================================================================
151
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
152
+ ================================================================================
153
+
154
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
155
+ Using pre-tokenized dataset (no runtime tokenization)
156
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
157
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
158
+ Dataset split: 3027764 train, 159356 validation
159
+ Training examples (packed, train split): 3027764
160
+ Validation examples (packed, val split): 159356
161
+ Per-device batch size: 2
162
+ World size (GPUs): 1
163
+ Gradient accumulation steps: 10
164
+ Effective batch size (seqs/step): 20
165
+ Sequence length: 8192
166
+ Tokens per optimizer step: 163,840
167
+ Micro-batches per epoch (per rank): 1513882
168
+ Optimizer steps per epoch: 151388
169
+ Total optimizer steps: 44000 (capped by max_steps; 0.29 epochs)
170
+ Starting training...
171
+ Switching attention mode for step 1: dense (dense step interval)
172
+ Step 10/44000 | Loss: 12.0471 | LR: 6.67e-06 | 4981.9ms/step | 4.0 samples/s
173
+ Step 20/44000 | Loss: 11.9631 | LR: 1.33e-05 | 4496.1ms/step | 4.4 samples/s
174
+ Step 30/44000 | Loss: 11.6650 | LR: 2.00e-05 | 4407.3ms/step | 4.5 samples/s
175
+ Step 40/44000 | Loss: 11.1311 | LR: 2.67e-05 | 4514.6ms/step | 4.4 samples/s
176
+ Step 50/44000 | Loss: 10.7789 | LR: 3.33e-05 | 4504.9ms/step | 4.4 samples/s
177
+ Step 60/44000 | Loss: 10.5403 | LR: 4.00e-05 | 4662.1ms/step | 4.3 samples/s
178
+ Step 70/44000 | Loss: 10.2980 | LR: 4.67e-05 | 4477.2ms/step | 4.5 samples/s
179
+ Step 80/44000 | Loss: 10.0010 | LR: 5.33e-05 | 4524.1ms/step | 4.4 samples/s
180
+ Step 90/44000 | Loss: 9.6513 | LR: 6.00e-05 | 4714.8ms/step | 4.2 samples/s
181
+ Step 100/44000 | Loss: 9.1544 | LR: 6.67e-05 | 4565.8ms/step | 4.4 samples/s
182
+ Step 110/44000 | Loss: 8.6366 | LR: 7.33e-05 | 4722.0ms/step | 4.2 samples/s
183
+ Step 120/44000 | Loss: 8.1371 | LR: 8.00e-05 | 4679.5ms/step | 4.3 samples/s
184
+ Step 130/44000 | Loss: 7.7777 | LR: 8.67e-05 | 4655.0ms/step | 4.3 samples/s
185
+ Step 140/44000 | Loss: 7.5066 | LR: 9.33e-05 | 4602.1ms/step | 4.3 samples/s
186
+ Step 150/44000 | Loss: 7.3495 | LR: 1.00e-04 | 4540.9ms/step | 4.4 samples/s
187
+ Step 160/44000 | Loss: 7.2203 | LR: 1.07e-04 | 4619.2ms/step | 4.3 samples/s
188
+ Step 170/44000 | Loss: 7.1301 | LR: 1.13e-04 | 4591.8ms/step | 4.4 samples/s
189
+ Step 180/44000 | Loss: 7.0444 | LR: 1.20e-04 | 4528.3ms/step | 4.4 samples/s
190
+ Step 190/44000 | Loss: 6.9277 | LR: 1.27e-04 | 4632.3ms/step | 4.3 samples/s
191
+ Step 200/44000 | Loss: 6.8284 | LR: 1.33e-04 | 4645.6ms/step | 4.3 samples/s
192
+ Step 210/44000 | Loss: 6.7509 | LR: 1.40e-04 | 4616.1ms/step | 4.3 samples/s
193
+ Step 220/44000 | Loss: 6.6871 | LR: 1.47e-04 | 4630.3ms/step | 4.3 samples/s
194
+ Step 230/44000 | Loss: 6.6048 | LR: 1.53e-04 | 4614.2ms/step | 4.3 samples/s
195
+ Step 240/44000 | Loss: 6.5302 | LR: 1.60e-04 | 4692.2ms/step | 4.3 samples/s
196
+ Step 250/44000 | Loss: 6.4602 | LR: 1.67e-04 | 4669.0ms/step | 4.3 samples/s
197
+ Step 260/44000 | Loss: 6.4182 | LR: 1.73e-04 | 4615.8ms/step | 4.3 samples/s
198
+ Step 270/44000 | Loss: 6.3615 | LR: 1.80e-04 | 4575.4ms/step | 4.4 samples/s
199
+ Step 280/44000 | Loss: 6.2961 | LR: 1.87e-04 | 4569.9ms/step | 4.4 samples/s
200
+ Step 290/44000 | Loss: 6.2423 | LR: 1.93e-04 | 4636.1ms/step | 4.3 samples/s
201
+ Step 300/44000 | Loss: 6.1843 | LR: 2.00e-04 | 4637.6ms/step | 4.3 samples/s
202
+ Step 310/44000 | Loss: 6.1488 | LR: 2.07e-04 | 4574.4ms/step | 4.4 samples/s
203
+ Step 320/44000 | Loss: 6.1278 | LR: 2.13e-04 | 4703.0ms/step | 4.3 samples/s
204
+ Step 330/44000 | Loss: 6.0744 | LR: 2.20e-04 | 4808.2ms/step | 4.2 samples/s
205
+ Step 340/44000 | Loss: 6.0686 | LR: 2.27e-04 | 4788.9ms/step | 4.2 samples/s
206
+ Step 350/44000 | Loss: 6.0174 | LR: 2.33e-04 | 4753.2ms/step | 4.2 samples/s
207
+ Step 360/44000 | Loss: 5.9457 | LR: 2.40e-04 | 4711.8ms/step | 4.2 samples/s
208
+ Step 370/44000 | Loss: 5.9369 | LR: 2.47e-04 | 4723.0ms/step | 4.2 samples/s
209
+ Step 380/44000 | Loss: 5.8989 | LR: 2.53e-04 | 4744.4ms/step | 4.2 samples/s
210
+ Step 390/44000 | Loss: 5.8345 | LR: 2.60e-04 | 4796.3ms/step | 4.2 samples/s
211
+ Step 400/44000 | Loss: 5.8239 | LR: 2.67e-04 | 4647.3ms/step | 4.3 samples/s
212
+ Step 410/44000 | Loss: 5.7631 | LR: 2.73e-04 | 4632.9ms/step | 4.3 samples/s
213
+ Step 420/44000 | Loss: 5.7329 | LR: 2.80e-04 | 4668.0ms/step | 4.3 samples/s
214
+ Step 430/44000 | Loss: 5.7342 | LR: 2.87e-04 | 4579.1ms/step | 4.4 samples/s
215
+ Step 440/44000 | Loss: 5.6684 | LR: 2.93e-04 | 4673.1ms/step | 4.3 samples/s
216
+ Step 450/44000 | Loss: 5.6497 | LR: 3.00e-04 | 4561.7ms/step | 4.4 samples/s
217
+ Step 460/44000 | Loss: 5.5924 | LR: 3.07e-04 | 4616.2ms/step | 4.3 samples/s
218
+ Step 470/44000 | Loss: 5.5569 | LR: 3.13e-04 | 4579.5ms/step | 4.4 samples/s
219
+ Step 480/44000 | Loss: 5.5548 | LR: 3.20e-04 | 4662.7ms/step | 4.3 samples/s
220
+ Step 490/44000 | Loss: 5.4914 | LR: 3.27e-04 | 4697.9ms/step | 4.3 samples/s
221
+ Step 500/44000 | Loss: 5.4652 | LR: 3.33e-04 | 4577.4ms/step | 4.4 samples/s
222
+ Step 510/44000 | Loss: 5.4546 | LR: 3.40e-04 | 4620.3ms/step | 4.3 samples/s
223
+ Step 520/44000 | Loss: 5.4160 | LR: 3.47e-04 | 4611.7ms/step | 4.3 samples/s
224
+ Step 530/44000 | Loss: 5.3777 | LR: 3.53e-04 | 4549.6ms/step | 4.4 samples/s
225
+ Step 540/44000 | Loss: 5.3866 | LR: 3.60e-04 | 4696.2ms/step | 4.3 samples/s
226
+ Step 550/44000 | Loss: 5.3163 | LR: 3.67e-04 | 4644.8ms/step | 4.3 samples/s
227
+ Step 560/44000 | Loss: 5.2820 | LR: 3.73e-04 | 4614.6ms/step | 4.3 samples/s
228
+ Step 570/44000 | Loss: 5.2639 | LR: 3.80e-04 | 4787.4ms/step | 4.2 samples/s
229
+ Step 580/44000 | Loss: 5.2425 | LR: 3.87e-04 | 4597.1ms/step | 4.4 samples/s
230
+ Step 590/44000 | Loss: 5.2022 | LR: 3.93e-04 | 4655.7ms/step | 4.3 samples/s
231
+ Step 600/44000 | Loss: 5.1974 | LR: 4.00e-04 | 4616.6ms/step | 4.3 samples/s
232
+ Step 610/44000 | Loss: 5.1638 | LR: 4.07e-04 | 4618.7ms/step | 4.3 samples/s
233
+ Step 620/44000 | Loss: 5.1347 | LR: 4.13e-04 | 4645.6ms/step | 4.3 samples/s
234
+ Step 630/44000 | Loss: 5.0720 | LR: 4.20e-04 | 4708.2ms/step | 4.2 samples/s
235
+ Step 640/44000 | Loss: 5.0443 | LR: 4.27e-04 | 4646.8ms/step | 4.3 samples/s
236
+ Step 650/44000 | Loss: 5.0376 | LR: 4.33e-04 | 4810.4ms/step | 4.2 samples/s
237
+ Step 660/44000 | Loss: 4.9885 | LR: 4.40e-04 | 4618.5ms/step | 4.3 samples/s
238
+ Step 670/44000 | Loss: 4.9642 | LR: 4.47e-04 | 4681.0ms/step | 4.3 samples/s
239
+ Step 680/44000 | Loss: 4.9169 | LR: 4.53e-04 | 4750.6ms/step | 4.2 samples/s
240
+ Step 690/44000 | Loss: 4.9120 | LR: 4.60e-04 | 4544.5ms/step | 4.4 samples/s
241
+ Step 700/44000 | Loss: 4.8471 | LR: 4.67e-04 | 4759.7ms/step | 4.2 samples/s
242
+ Step 710/44000 | Loss: 4.8069 | LR: 4.73e-04 | 4670.9ms/step | 4.3 samples/s
243
+ Step 720/44000 | Loss: 4.8372 | LR: 4.80e-04 | 4747.8ms/step | 4.2 samples/s
244
+ Step 730/44000 | Loss: 4.7648 | LR: 4.87e-04 | 4872.4ms/step | 4.1 samples/s
245
+ Step 740/44000 | Loss: 4.7706 | LR: 4.93e-04 | 4686.4ms/step | 4.3 samples/s
246
+ Step 750/44000 | Loss: 4.7215 | LR: 5.00e-04 | 4750.0ms/step | 4.2 samples/s
247
+ Step 760/44000 | Loss: 4.7329 | LR: 5.00e-04 | 4729.0ms/step | 4.2 samples/s
248
+ Step 770/44000 | Loss: 4.6652 | LR: 5.00e-04 | 4639.5ms/step | 4.3 samples/s
249
+ Step 780/44000 | Loss: 4.6739 | LR: 5.00e-04 | 4530.0ms/step | 4.4 samples/s
250
+ Step 790/44000 | Loss: 4.5860 | LR: 5.00e-04 | 4570.9ms/step | 4.4 samples/s
251
+ Step 800/44000 | Loss: 4.5480 | LR: 5.00e-04 | 4690.4ms/step | 4.3 samples/s
252
+ Step 810/44000 | Loss: 4.5601 | LR: 5.00e-04 | 4665.9ms/step | 4.3 samples/s
253
+ Step 820/44000 | Loss: 4.5444 | LR: 5.00e-04 | 4733.5ms/step | 4.2 samples/s
254
+ Step 830/44000 | Loss: 4.5112 | LR: 5.00e-04 | 4773.1ms/step | 4.2 samples/s
255
+ Step 840/44000 | Loss: 4.4828 | LR: 5.00e-04 | 4686.7ms/step | 4.3 samples/s
256
+ Step 850/44000 | Loss: 4.4625 | LR: 5.00e-04 | 4584.9ms/step | 4.4 samples/s
257
+ Step 860/44000 | Loss: 4.4707 | LR: 5.00e-04 | 4921.4ms/step | 4.1 samples/s
258
+ Step 870/44000 | Loss: 4.4304 | LR: 5.00e-04 | 4853.2ms/step | 4.1 samples/s
259
+ Step 880/44000 | Loss: 4.4372 | LR: 5.00e-04 | 4804.0ms/step | 4.2 samples/s
260
+ Step 890/44000 | Loss: 4.4242 | LR: 5.00e-04 | 4822.0ms/step | 4.1 samples/s
261
+ Step 900/44000 | Loss: 4.3993 | LR: 5.00e-04 | 4842.6ms/step | 4.1 samples/s
262
+ Step 910/44000 | Loss: 4.3485 | LR: 5.00e-04 | 4838.8ms/step | 4.1 samples/s
263
+ Step 920/44000 | Loss: 4.3770 | LR: 5.00e-04 | 4843.7ms/step | 4.1 samples/s
264
+ Step 930/44000 | Loss: 4.3407 | LR: 5.00e-04 | 4691.3ms/step | 4.3 samples/s
265
+ Step 940/44000 | Loss: 4.3288 | LR: 5.00e-04 | 4666.4ms/step | 4.3 samples/s
266
+ Step 950/44000 | Loss: 4.3385 | LR: 5.00e-04 | 4595.2ms/step | 4.4 samples/s
267
+ Step 960/44000 | Loss: 4.3093 | LR: 5.00e-04 | 4685.6ms/step | 4.3 samples/s
268
+ Step 970/44000 | Loss: 4.3016 | LR: 5.00e-04 | 4616.5ms/step | 4.3 samples/s
269
+ Step 980/44000 | Loss: 4.2642 | LR: 5.00e-04 | 4630.1ms/step | 4.3 samples/s
270
+ Step 990/44000 | Loss: 4.2538 | LR: 5.00e-04 | 4666.3ms/step | 4.3 samples/s
271
+ Step 1000/44000 | Loss: 4.2567 | LR: 5.00e-04 | 4693.2ms/step | 4.3 samples/s
272
+ Traceback (most recent call last):
273
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/resource_sharer.py", line 138, in _serve
274
+ with self._listener.accept() as conn:
275
+ ~~~~~~~~~~~~~~~~~~~~~^^
276
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 495, in accept
277
+ deliver_challenge(c, self._authkey)
278
+ ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
279
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 961, in deliver_challenge
280
+ response = connection.recv_bytes(256) # reject large message
281
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 217, in recv_bytes
282
+ buf = self._recv_bytes(maxlength)
283
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 431, in _recv_bytes
284
+ buf = self._recv(4)
285
+ File "/mloscratch/homes/navasard/conda/lib/python3.13/multiprocessing/connection.py", line 396, in _recv
286
+ chunk = read(handle, remaining)
287
+ ConnectionResetError: [Errno 104] Connection reset by peer
epfl_final/train_logs/ropectl_full_a1p0/seed44.log ADDED
@@ -0,0 +1,271 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ New run: Checkpoints will be saved to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858
2
+ Running config:
3
+ {
4
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
5
+ "model_dtype": "bfloat16",
6
+ "init_from_scratch": true,
7
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
8
+ "val_path": null,
9
+ "max_seq_length": 8192,
10
+ "num_workers": 8,
11
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed44/run_20260819_153858",
12
+ "resume_from_checkpoint": null,
13
+ "force_new_run": true,
14
+ "num_epochs": 1,
15
+ "max_steps": 44000,
16
+ "per_device_batch_size": 2,
17
+ "gradient_accumulation_steps": 10,
18
+ "learning_rate": 0.0005,
19
+ "weight_decay": 0.01,
20
+ "warmup_steps": 750,
21
+ "lr_scheduler": "cosine",
22
+ "max_grad_norm": 1.0,
23
+ "save_steps": 1000,
24
+ "eval_steps": 1000,
25
+ "eval_batches": 50,
26
+ "logging_steps": 10,
27
+ "seed": 44,
28
+ "verbose": true,
29
+ "checkpoint_activations": false,
30
+ "strategy": "ddp",
31
+ "mixed_precision": "bf16",
32
+ "wandb_enabled": false,
33
+ "wandb_project": "stochastic_sparsity",
34
+ "wandb_name": "l8k365m_dense_ropectl_a1p0_seed44",
35
+ "wandb_tags": [
36
+ "qwen3",
37
+ "365m",
38
+ "8k",
39
+ "dense",
40
+ "rope-control"
41
+ ],
42
+ "hidden_size": 768,
43
+ "intermediate_size": 3072,
44
+ "num_attention_heads": 6,
45
+ "num_key_value_heads": 2,
46
+ "num_hidden_layers": 28,
47
+ "head_dim": null,
48
+ "attention_dropout": 0.0,
49
+ "rope_train_position_scale_set": [
50
+ 1.0,
51
+ 0.9896339488176222,
52
+ 0.9692159692159692,
53
+ 0.9420289855072463,
54
+ 0.8730158730158729
55
+ ],
56
+ "attention_type": "sparse_stochastic",
57
+ "sparse_block_size": 128,
58
+ "sparse_num_random_blocks": 31,
59
+ "sparse_mask_pool_size": 1,
60
+ "sparse_val_dense_compare": false,
61
+ "sparse_layer_indices": null,
62
+ "sparse_mask_cache_dir": null,
63
+ "sparse_mask_seed": 0,
64
+ "sparse_mixed_nrb_set": null,
65
+ "dense_mode": true,
66
+ "dense_step_intervals": null
67
+ }
68
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
69
+ Initializing model from scratch (random weights)...
70
+ Checkpoint activations: False
71
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
72
+
73
+
74
+ ================================================================================
75
+ MODEL ARCHITECTURE
76
+ ================================================================================
77
+ Model config:
78
+ vocab_size: 151936
79
+ hidden_size: 768
80
+ num_hidden_layers: 28
81
+ num_attention_heads: 6
82
+ num_key_value_heads: 2
83
+ head_dim: 128
84
+ intermediate_size: 3072
85
+ max_position_embeddings: 32768
86
+ rope_theta: 1000000
87
+ rope_train_position_scale_set: [1.0, 0.9896339488176222, 0.9692159692159692, 0.9420289855072463, 0.8730158730158729]
88
+ attention_type: sparse_stochastic
89
+ sparse_block_size: 128
90
+ sparse_num_random_blocks: 31
91
+ sparse_mask_pool_size: 1
92
+ sparse_max_seq_length: 8192
93
+
94
+ Parameter count:
95
+ Total parameters: 358,958,848
96
+ Trainable parameters: 358,958,848
97
+ Non-trainable parameters: 0
98
+ Model size (BF16): 0.72 GB
99
+
100
+ --------------------------------------------------------------------------------
101
+ LAYER-BY-LAYER SUMMARY
102
+ --------------------------------------------------------------------------------
103
+ ================================================================================================================================================================
104
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
105
+ ================================================================================================================================================================
106
+ Qwen3ForCausalLM [2, 8192] -- -- True
107
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
108
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
109
+ │ └─ModuleList: 2-2 -- -- -- True
110
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
111
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
132
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
133
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
134
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
135
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
136
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
137
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
138
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
139
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
140
+ ================================================================================================================================================================
141
+ Total params: 475,645,696
142
+ Trainable params: 475,645,696
143
+ Non-trainable params: 0
144
+ Total mult-adds (Units.MEGABYTES): 951.29
145
+ ================================================================================================================================================================
146
+ Input size (MB): 0.13
147
+ Forward/backward pass size (MB): 31197.23
148
+ Params size (MB): 951.29
149
+ Estimated Total Size (MB): 32148.66
150
+ ================================================================================================================================================================
151
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
152
+ ================================================================================
153
+
154
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
155
+ Using pre-tokenized dataset (no runtime tokenization)
156
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
157
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
158
+ Dataset split: 3027764 train, 159356 validation
159
+ Training examples (packed, train split): 3027764
160
+ Validation examples (packed, val split): 159356
161
+ Per-device batch size: 2
162
+ World size (GPUs): 1
163
+ Gradient accumulation steps: 10
164
+ Effective batch size (seqs/step): 20
165
+ Sequence length: 8192
166
+ Tokens per optimizer step: 163,840
167
+ Micro-batches per epoch (per rank): 1513882
168
+ Optimizer steps per epoch: 151388
169
+ Total optimizer steps: 44000 (capped by max_steps; 0.29 epochs)
170
+ Starting training...
171
+ Switching attention mode for step 1: dense (dense step interval)
172
+ Step 10/44000 | Loss: 12.0402 | LR: 6.67e-06 | 13510.5ms/step | 1.5 samples/s
173
+ Step 20/44000 | Loss: 11.9576 | LR: 1.33e-05 | 4924.1ms/step | 4.1 samples/s
174
+ Step 30/44000 | Loss: 11.6481 | LR: 2.00e-05 | 4641.4ms/step | 4.3 samples/s
175
+ Step 40/44000 | Loss: 11.1109 | LR: 2.67e-05 | 4674.2ms/step | 4.3 samples/s
176
+ Step 50/44000 | Loss: 10.7733 | LR: 3.33e-05 | 4739.5ms/step | 4.2 samples/s
177
+ Step 60/44000 | Loss: 10.5185 | LR: 4.00e-05 | 4694.1ms/step | 4.3 samples/s
178
+ Step 70/44000 | Loss: 10.2953 | LR: 4.67e-05 | 4596.0ms/step | 4.4 samples/s
179
+ Step 80/44000 | Loss: 9.9988 | LR: 5.33e-05 | 4606.4ms/step | 4.3 samples/s
180
+ Step 90/44000 | Loss: 9.6238 | LR: 6.00e-05 | 4650.7ms/step | 4.3 samples/s
181
+ Step 100/44000 | Loss: 9.1573 | LR: 6.67e-05 | 4718.3ms/step | 4.2 samples/s
182
+ Step 110/44000 | Loss: 8.6288 | LR: 7.33e-05 | 4660.3ms/step | 4.3 samples/s
183
+ Step 120/44000 | Loss: 8.1340 | LR: 8.00e-05 | 4685.5ms/step | 4.3 samples/s
184
+ Step 130/44000 | Loss: 7.7702 | LR: 8.67e-05 | 4696.1ms/step | 4.3 samples/s
185
+ Step 140/44000 | Loss: 7.5538 | LR: 9.33e-05 | 4743.5ms/step | 4.2 samples/s
186
+ Step 150/44000 | Loss: 7.3426 | LR: 1.00e-04 | 4776.7ms/step | 4.2 samples/s
187
+ Step 160/44000 | Loss: 7.2341 | LR: 1.07e-04 | 4848.9ms/step | 4.1 samples/s
188
+ Step 170/44000 | Loss: 7.1072 | LR: 1.13e-04 | 4712.7ms/step | 4.2 samples/s
189
+ Step 180/44000 | Loss: 7.0168 | LR: 1.20e-04 | 4680.9ms/step | 4.3 samples/s
190
+ Step 190/44000 | Loss: 6.8978 | LR: 1.27e-04 | 4655.4ms/step | 4.3 samples/s
191
+ Step 200/44000 | Loss: 6.8286 | LR: 1.33e-04 | 4658.2ms/step | 4.3 samples/s
192
+ Step 210/44000 | Loss: 6.7548 | LR: 1.40e-04 | 4651.1ms/step | 4.3 samples/s
193
+ Step 220/44000 | Loss: 6.6723 | LR: 1.47e-04 | 4602.1ms/step | 4.3 samples/s
194
+ Step 230/44000 | Loss: 6.6017 | LR: 1.53e-04 | 4671.8ms/step | 4.3 samples/s
195
+ Step 240/44000 | Loss: 6.5023 | LR: 1.60e-04 | 4755.8ms/step | 4.2 samples/s
196
+ Step 250/44000 | Loss: 6.4628 | LR: 1.67e-04 | 4835.9ms/step | 4.1 samples/s
197
+ Step 260/44000 | Loss: 6.4062 | LR: 1.73e-04 | 5063.1ms/step | 4.0 samples/s
198
+ Step 270/44000 | Loss: 6.3694 | LR: 1.80e-04 | 4930.3ms/step | 4.1 samples/s
199
+ Step 280/44000 | Loss: 6.3030 | LR: 1.87e-04 | 4972.6ms/step | 4.0 samples/s
200
+ Step 290/44000 | Loss: 6.2823 | LR: 1.93e-04 | 4923.1ms/step | 4.1 samples/s
201
+ Step 300/44000 | Loss: 6.2608 | LR: 2.00e-04 | 4897.1ms/step | 4.1 samples/s
202
+ Step 310/44000 | Loss: 6.2176 | LR: 2.07e-04 | 4802.9ms/step | 4.2 samples/s
203
+ Step 320/44000 | Loss: 6.1558 | LR: 2.13e-04 | 4732.1ms/step | 4.2 samples/s
204
+ Step 330/44000 | Loss: 6.1201 | LR: 2.20e-04 | 4818.3ms/step | 4.2 samples/s
205
+ Step 340/44000 | Loss: 6.0567 | LR: 2.27e-04 | 4617.7ms/step | 4.3 samples/s
206
+ Step 350/44000 | Loss: 6.0206 | LR: 2.33e-04 | 4733.7ms/step | 4.2 samples/s
207
+ Step 360/44000 | Loss: 5.9602 | LR: 2.40e-04 | 4708.3ms/step | 4.2 samples/s
208
+ Step 370/44000 | Loss: 5.9206 | LR: 2.47e-04 | 4712.3ms/step | 4.2 samples/s
209
+ Step 380/44000 | Loss: 5.8908 | LR: 2.53e-04 | 4629.4ms/step | 4.3 samples/s
210
+ Step 390/44000 | Loss: 5.8408 | LR: 2.60e-04 | 4590.7ms/step | 4.4 samples/s
211
+ Step 400/44000 | Loss: 5.8375 | LR: 2.67e-04 | 4639.5ms/step | 4.3 samples/s
212
+ Step 410/44000 | Loss: 5.7954 | LR: 2.73e-04 | 4629.7ms/step | 4.3 samples/s
213
+ Step 420/44000 | Loss: 5.7558 | LR: 2.80e-04 | 4674.0ms/step | 4.3 samples/s
214
+ Step 430/44000 | Loss: 5.7681 | LR: 2.87e-04 | 4722.7ms/step | 4.2 samples/s
215
+ Step 440/44000 | Loss: 5.7153 | LR: 2.93e-04 | 4686.7ms/step | 4.3 samples/s
216
+ Step 450/44000 | Loss: 5.6459 | LR: 3.00e-04 | 4623.2ms/step | 4.3 samples/s
217
+ Step 460/44000 | Loss: 5.6284 | LR: 3.07e-04 | 4632.0ms/step | 4.3 samples/s
218
+ Step 470/44000 | Loss: 5.5750 | LR: 3.13e-04 | 4639.7ms/step | 4.3 samples/s
219
+ Step 480/44000 | Loss: 5.5161 | LR: 3.20e-04 | 4643.0ms/step | 4.3 samples/s
220
+ Step 490/44000 | Loss: 5.5343 | LR: 3.27e-04 | 4641.5ms/step | 4.3 samples/s
221
+ Step 500/44000 | Loss: 5.4819 | LR: 3.33e-04 | 4728.4ms/step | 4.2 samples/s
222
+ Step 510/44000 | Loss: 5.4552 | LR: 3.40e-04 | 4785.1ms/step | 4.2 samples/s
223
+ Step 520/44000 | Loss: 5.4290 | LR: 3.47e-04 | 4688.4ms/step | 4.3 samples/s
224
+ Step 530/44000 | Loss: 5.3787 | LR: 3.53e-04 | 4699.8ms/step | 4.3 samples/s
225
+ Step 540/44000 | Loss: 5.3513 | LR: 3.60e-04 | 4741.0ms/step | 4.2 samples/s
226
+ Step 550/44000 | Loss: 5.3495 | LR: 3.67e-04 | 4613.1ms/step | 4.3 samples/s
227
+ Step 560/44000 | Loss: 5.3034 | LR: 3.73e-04 | 4833.9ms/step | 4.1 samples/s
228
+ Step 570/44000 | Loss: 5.2613 | LR: 3.80e-04 | 4701.8ms/step | 4.3 samples/s
229
+ Step 580/44000 | Loss: 5.2069 | LR: 3.87e-04 | 4650.8ms/step | 4.3 samples/s
230
+ Step 590/44000 | Loss: 5.2333 | LR: 3.93e-04 | 4794.8ms/step | 4.2 samples/s
231
+ Step 600/44000 | Loss: 5.1799 | LR: 4.00e-04 | 4660.2ms/step | 4.3 samples/s
232
+ Step 610/44000 | Loss: 5.1666 | LR: 4.07e-04 | 4733.9ms/step | 4.2 samples/s
233
+ Step 620/44000 | Loss: 5.1160 | LR: 4.13e-04 | 4676.2ms/step | 4.3 samples/s
234
+ Step 630/44000 | Loss: 5.1012 | LR: 4.20e-04 | 4660.0ms/step | 4.3 samples/s
235
+ Step 640/44000 | Loss: 5.0810 | LR: 4.27e-04 | 4917.7ms/step | 4.1 samples/s
236
+ Step 650/44000 | Loss: 5.0127 | LR: 4.33e-04 | 4667.1ms/step | 4.3 samples/s
237
+ Step 660/44000 | Loss: 4.9605 | LR: 4.40e-04 | 4811.9ms/step | 4.2 samples/s
238
+ Step 670/44000 | Loss: 4.9943 | LR: 4.47e-04 | 4712.5ms/step | 4.2 samples/s
239
+ Step 680/44000 | Loss: 4.9695 | LR: 4.53e-04 | 4601.9ms/step | 4.3 samples/s
240
+ Step 690/44000 | Loss: 4.9122 | LR: 4.60e-04 | 4574.6ms/step | 4.4 samples/s
241
+ Step 700/44000 | Loss: 4.8777 | LR: 4.67e-04 | 4587.8ms/step | 4.4 samples/s
242
+ Step 710/44000 | Loss: 4.8364 | LR: 4.73e-04 | 4666.5ms/step | 4.3 samples/s
243
+ Step 720/44000 | Loss: 4.8480 | LR: 4.80e-04 | 4652.5ms/step | 4.3 samples/s
244
+ Step 730/44000 | Loss: 4.8160 | LR: 4.87e-04 | 4623.6ms/step | 4.3 samples/s
245
+ Step 740/44000 | Loss: 4.7709 | LR: 4.93e-04 | 4719.4ms/step | 4.2 samples/s
246
+ Step 750/44000 | Loss: 4.6810 | LR: 5.00e-04 | 4648.5ms/step | 4.3 samples/s
247
+ Step 760/44000 | Loss: 4.7215 | LR: 5.00e-04 | 4702.2ms/step | 4.3 samples/s
248
+ Step 770/44000 | Loss: 4.6970 | LR: 5.00e-04 | 4770.1ms/step | 4.2 samples/s
249
+ Step 780/44000 | Loss: 4.6712 | LR: 5.00e-04 | 4900.6ms/step | 4.1 samples/s
250
+ Step 790/44000 | Loss: 4.6512 | LR: 5.00e-04 | 4917.3ms/step | 4.1 samples/s
251
+ Step 800/44000 | Loss: 4.6234 | LR: 5.00e-04 | 4811.9ms/step | 4.2 samples/s
252
+ Step 810/44000 | Loss: 4.5839 | LR: 5.00e-04 | 4912.3ms/step | 4.1 samples/s
253
+ Step 820/44000 | Loss: 4.5824 | LR: 5.00e-04 | 4935.2ms/step | 4.1 samples/s
254
+ Step 830/44000 | Loss: 4.5488 | LR: 5.00e-04 | 4815.8ms/step | 4.2 samples/s
255
+ Step 840/44000 | Loss: 4.4994 | LR: 5.00e-04 | 4732.3ms/step | 4.2 samples/s
256
+ Step 850/44000 | Loss: 4.4889 | LR: 5.00e-04 | 4598.7ms/step | 4.3 samples/s
257
+ Step 860/44000 | Loss: 4.4952 | LR: 5.00e-04 | 4705.7ms/step | 4.3 samples/s
258
+ Step 870/44000 | Loss: 4.4365 | LR: 5.00e-04 | 4762.7ms/step | 4.2 samples/s
259
+ Step 880/44000 | Loss: 4.4366 | LR: 5.00e-04 | 4741.0ms/step | 4.2 samples/s
260
+ Step 890/44000 | Loss: 4.4334 | LR: 5.00e-04 | 4752.9ms/step | 4.2 samples/s
261
+ Step 900/44000 | Loss: 4.3964 | LR: 5.00e-04 | 4735.3ms/step | 4.2 samples/s
262
+ Step 910/44000 | Loss: 4.3748 | LR: 5.00e-04 | 4664.5ms/step | 4.3 samples/s
263
+ Step 920/44000 | Loss: 4.3545 | LR: 5.00e-04 | 4427.3ms/step | 4.5 samples/s
264
+ Step 930/44000 | Loss: 4.3499 | LR: 5.00e-04 | 4238.7ms/step | 4.7 samples/s
265
+ Step 940/44000 | Loss: 4.3521 | LR: 5.00e-04 | 4584.5ms/step | 4.4 samples/s
266
+ Step 950/44000 | Loss: 4.3020 | LR: 5.00e-04 | 4342.7ms/step | 4.6 samples/s
267
+ Step 960/44000 | Loss: 4.3416 | LR: 5.00e-04 | 4640.7ms/step | 4.3 samples/s
268
+ Step 970/44000 | Loss: 4.3052 | LR: 5.00e-04 | 5014.6ms/step | 4.0 samples/s
269
+ Step 980/44000 | Loss: 4.2775 | LR: 5.00e-04 | 4379.9ms/step | 4.6 samples/s
270
+ Step 990/44000 | Loss: 4.2771 | LR: 5.00e-04 | 4525.6ms/step | 4.4 samples/s
271
+ Step 1000/44000 | Loss: 4.2480 | LR: 5.00e-04 | 4533.0ms/step | 4.4 samples/s
epfl_final/train_logs/ropectl_full_a1p0/seed45.log ADDED
@@ -0,0 +1,271 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ New run: Checkpoints will be saved to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735
2
+ Running config:
3
+ {
4
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
5
+ "model_dtype": "bfloat16",
6
+ "init_from_scratch": true,
7
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
8
+ "val_path": null,
9
+ "max_seq_length": 8192,
10
+ "num_workers": 8,
11
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed45/run_20260819_153735",
12
+ "resume_from_checkpoint": null,
13
+ "force_new_run": true,
14
+ "num_epochs": 1,
15
+ "max_steps": 44000,
16
+ "per_device_batch_size": 2,
17
+ "gradient_accumulation_steps": 10,
18
+ "learning_rate": 0.0005,
19
+ "weight_decay": 0.01,
20
+ "warmup_steps": 750,
21
+ "lr_scheduler": "cosine",
22
+ "max_grad_norm": 1.0,
23
+ "save_steps": 1000,
24
+ "eval_steps": 1000,
25
+ "eval_batches": 50,
26
+ "logging_steps": 10,
27
+ "seed": 45,
28
+ "verbose": true,
29
+ "checkpoint_activations": false,
30
+ "strategy": "ddp",
31
+ "mixed_precision": "bf16",
32
+ "wandb_enabled": false,
33
+ "wandb_project": "stochastic_sparsity",
34
+ "wandb_name": "l8k365m_dense_ropectl_a1p0_seed45",
35
+ "wandb_tags": [
36
+ "qwen3",
37
+ "365m",
38
+ "8k",
39
+ "dense",
40
+ "rope-control"
41
+ ],
42
+ "hidden_size": 768,
43
+ "intermediate_size": 3072,
44
+ "num_attention_heads": 6,
45
+ "num_key_value_heads": 2,
46
+ "num_hidden_layers": 28,
47
+ "head_dim": null,
48
+ "attention_dropout": 0.0,
49
+ "rope_train_position_scale_set": [
50
+ 1.0,
51
+ 0.9896339488176222,
52
+ 0.9692159692159692,
53
+ 0.9420289855072463,
54
+ 0.8730158730158729
55
+ ],
56
+ "attention_type": "sparse_stochastic",
57
+ "sparse_block_size": 128,
58
+ "sparse_num_random_blocks": 31,
59
+ "sparse_mask_pool_size": 1,
60
+ "sparse_val_dense_compare": false,
61
+ "sparse_layer_indices": null,
62
+ "sparse_mask_cache_dir": null,
63
+ "sparse_mask_seed": 0,
64
+ "sparse_mixed_nrb_set": null,
65
+ "dense_mode": true,
66
+ "dense_step_intervals": null
67
+ }
68
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
69
+ Initializing model from scratch (random weights)...
70
+ Checkpoint activations: False
71
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
72
+
73
+
74
+ ================================================================================
75
+ MODEL ARCHITECTURE
76
+ ================================================================================
77
+ Model config:
78
+ vocab_size: 151936
79
+ hidden_size: 768
80
+ num_hidden_layers: 28
81
+ num_attention_heads: 6
82
+ num_key_value_heads: 2
83
+ head_dim: 128
84
+ intermediate_size: 3072
85
+ max_position_embeddings: 32768
86
+ rope_theta: 1000000
87
+ rope_train_position_scale_set: [1.0, 0.9896339488176222, 0.9692159692159692, 0.9420289855072463, 0.8730158730158729]
88
+ attention_type: sparse_stochastic
89
+ sparse_block_size: 128
90
+ sparse_num_random_blocks: 31
91
+ sparse_mask_pool_size: 1
92
+ sparse_max_seq_length: 8192
93
+
94
+ Parameter count:
95
+ Total parameters: 358,958,848
96
+ Trainable parameters: 358,958,848
97
+ Non-trainable parameters: 0
98
+ Model size (BF16): 0.72 GB
99
+
100
+ --------------------------------------------------------------------------------
101
+ LAYER-BY-LAYER SUMMARY
102
+ --------------------------------------------------------------------------------
103
+ ================================================================================================================================================================
104
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
105
+ ================================================================================================================================================================
106
+ Qwen3ForCausalLM [2, 8192] -- -- True
107
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
108
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
109
+ │ └─ModuleList: 2-2 -- -- -- True
110
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
111
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
132
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
133
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
134
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
135
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
136
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
137
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
138
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
139
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
140
+ ================================================================================================================================================================
141
+ Total params: 475,645,696
142
+ Trainable params: 475,645,696
143
+ Non-trainable params: 0
144
+ Total mult-adds (Units.MEGABYTES): 951.29
145
+ ================================================================================================================================================================
146
+ Input size (MB): 0.13
147
+ Forward/backward pass size (MB): 31197.23
148
+ Params size (MB): 951.29
149
+ Estimated Total Size (MB): 32148.66
150
+ ================================================================================================================================================================
151
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
152
+ ================================================================================
153
+
154
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
155
+ Using pre-tokenized dataset (no runtime tokenization)
156
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
157
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
158
+ Dataset split: 3027764 train, 159356 validation
159
+ Training examples (packed, train split): 3027764
160
+ Validation examples (packed, val split): 159356
161
+ Per-device batch size: 2
162
+ World size (GPUs): 1
163
+ Gradient accumulation steps: 10
164
+ Effective batch size (seqs/step): 20
165
+ Sequence length: 8192
166
+ Tokens per optimizer step: 163,840
167
+ Micro-batches per epoch (per rank): 1513882
168
+ Optimizer steps per epoch: 151388
169
+ Total optimizer steps: 44000 (capped by max_steps; 0.29 epochs)
170
+ Starting training...
171
+ Switching attention mode for step 1: dense (dense step interval)
172
+ Step 10/44000 | Loss: 12.0744 | LR: 6.67e-06 | 4955.0ms/step | 4.0 samples/s
173
+ Step 20/44000 | Loss: 11.9917 | LR: 1.33e-05 | 4392.0ms/step | 4.6 samples/s
174
+ Step 30/44000 | Loss: 11.6999 | LR: 2.00e-05 | 4413.1ms/step | 4.5 samples/s
175
+ Step 40/44000 | Loss: 11.1842 | LR: 2.67e-05 | 4525.0ms/step | 4.4 samples/s
176
+ Step 50/44000 | Loss: 10.8391 | LR: 3.33e-05 | 4503.4ms/step | 4.4 samples/s
177
+ Step 60/44000 | Loss: 10.5941 | LR: 4.00e-05 | 4623.0ms/step | 4.3 samples/s
178
+ Step 70/44000 | Loss: 10.3541 | LR: 4.67e-05 | 4601.6ms/step | 4.3 samples/s
179
+ Step 80/44000 | Loss: 10.0574 | LR: 5.33e-05 | 4623.6ms/step | 4.3 samples/s
180
+ Step 90/44000 | Loss: 9.7045 | LR: 6.00e-05 | 4920.2ms/step | 4.1 samples/s
181
+ Step 100/44000 | Loss: 9.2098 | LR: 6.67e-05 | 4836.6ms/step | 4.1 samples/s
182
+ Step 110/44000 | Loss: 8.6573 | LR: 7.33e-05 | 4843.0ms/step | 4.1 samples/s
183
+ Step 120/44000 | Loss: 8.1762 | LR: 8.00e-05 | 4675.0ms/step | 4.3 samples/s
184
+ Step 130/44000 | Loss: 7.7989 | LR: 8.67e-05 | 4788.5ms/step | 4.2 samples/s
185
+ Step 140/44000 | Loss: 7.5242 | LR: 9.33e-05 | 4904.7ms/step | 4.1 samples/s
186
+ Step 150/44000 | Loss: 7.3578 | LR: 1.00e-04 | 4810.4ms/step | 4.2 samples/s
187
+ Step 160/44000 | Loss: 7.2185 | LR: 1.07e-04 | 4691.9ms/step | 4.3 samples/s
188
+ Step 170/44000 | Loss: 7.1179 | LR: 1.13e-04 | 4726.5ms/step | 4.2 samples/s
189
+ Step 180/44000 | Loss: 7.0186 | LR: 1.20e-04 | 4744.4ms/step | 4.2 samples/s
190
+ Step 190/44000 | Loss: 6.9409 | LR: 1.27e-04 | 4756.0ms/step | 4.2 samples/s
191
+ Step 200/44000 | Loss: 6.8452 | LR: 1.33e-04 | 4790.8ms/step | 4.2 samples/s
192
+ Step 210/44000 | Loss: 6.7685 | LR: 1.40e-04 | 4884.3ms/step | 4.1 samples/s
193
+ Step 220/44000 | Loss: 6.6751 | LR: 1.47e-04 | 4832.6ms/step | 4.1 samples/s
194
+ Step 230/44000 | Loss: 6.6086 | LR: 1.53e-04 | 4759.9ms/step | 4.2 samples/s
195
+ Step 240/44000 | Loss: 6.5487 | LR: 1.60e-04 | 4783.5ms/step | 4.2 samples/s
196
+ Step 250/44000 | Loss: 6.4796 | LR: 1.67e-04 | 4862.3ms/step | 4.1 samples/s
197
+ Step 260/44000 | Loss: 6.4289 | LR: 1.73e-04 | 4906.7ms/step | 4.1 samples/s
198
+ Step 270/44000 | Loss: 6.3534 | LR: 1.80e-04 | 4982.1ms/step | 4.0 samples/s
199
+ Step 280/44000 | Loss: 6.3480 | LR: 1.87e-04 | 4798.6ms/step | 4.2 samples/s
200
+ Step 290/44000 | Loss: 6.2651 | LR: 1.93e-04 | 4894.7ms/step | 4.1 samples/s
201
+ Step 300/44000 | Loss: 6.2101 | LR: 2.00e-04 | 5023.5ms/step | 4.0 samples/s
202
+ Step 310/44000 | Loss: 6.2070 | LR: 2.07e-04 | 4710.7ms/step | 4.2 samples/s
203
+ Step 320/44000 | Loss: 6.1499 | LR: 2.13e-04 | 4959.2ms/step | 4.0 samples/s
204
+ Step 330/44000 | Loss: 6.0913 | LR: 2.20e-04 | 5114.9ms/step | 3.9 samples/s
205
+ Step 340/44000 | Loss: 6.0304 | LR: 2.27e-04 | 5006.6ms/step | 4.0 samples/s
206
+ Step 350/44000 | Loss: 6.0361 | LR: 2.33e-04 | 5026.6ms/step | 4.0 samples/s
207
+ Step 360/44000 | Loss: 5.9918 | LR: 2.40e-04 | 5033.9ms/step | 4.0 samples/s
208
+ Step 370/44000 | Loss: 5.9324 | LR: 2.47e-04 | 4893.6ms/step | 4.1 samples/s
209
+ Step 380/44000 | Loss: 5.9135 | LR: 2.53e-04 | 4980.1ms/step | 4.0 samples/s
210
+ Step 390/44000 | Loss: 5.8840 | LR: 2.60e-04 | 4823.7ms/step | 4.1 samples/s
211
+ Step 400/44000 | Loss: 5.8480 | LR: 2.67e-04 | 4842.0ms/step | 4.1 samples/s
212
+ Step 410/44000 | Loss: 5.7986 | LR: 2.73e-04 | 4794.6ms/step | 4.2 samples/s
213
+ Step 420/44000 | Loss: 5.8002 | LR: 2.80e-04 | 4885.4ms/step | 4.1 samples/s
214
+ Step 430/44000 | Loss: 5.7189 | LR: 2.87e-04 | 4716.9ms/step | 4.2 samples/s
215
+ Step 440/44000 | Loss: 5.7027 | LR: 2.93e-04 | 5105.3ms/step | 3.9 samples/s
216
+ Step 450/44000 | Loss: 5.6828 | LR: 3.00e-04 | 4827.2ms/step | 4.1 samples/s
217
+ Step 460/44000 | Loss: 5.6525 | LR: 3.07e-04 | 4819.9ms/step | 4.1 samples/s
218
+ Step 470/44000 | Loss: 5.5984 | LR: 3.13e-04 | 4836.0ms/step | 4.1 samples/s
219
+ Step 480/44000 | Loss: 5.5502 | LR: 3.20e-04 | 4768.0ms/step | 4.2 samples/s
220
+ Step 490/44000 | Loss: 5.5581 | LR: 3.27e-04 | 4811.1ms/step | 4.2 samples/s
221
+ Step 500/44000 | Loss: 5.4899 | LR: 3.33e-04 | 4793.0ms/step | 4.2 samples/s
222
+ Step 510/44000 | Loss: 5.4325 | LR: 3.40e-04 | 4754.3ms/step | 4.2 samples/s
223
+ Step 520/44000 | Loss: 5.4335 | LR: 3.47e-04 | 4891.1ms/step | 4.1 samples/s
224
+ Step 530/44000 | Loss: 5.3808 | LR: 3.53e-04 | 4937.9ms/step | 4.1 samples/s
225
+ Step 540/44000 | Loss: 5.3720 | LR: 3.60e-04 | 4874.9ms/step | 4.1 samples/s
226
+ Step 550/44000 | Loss: 5.3211 | LR: 3.67e-04 | 4961.9ms/step | 4.0 samples/s
227
+ Step 560/44000 | Loss: 5.3098 | LR: 3.73e-04 | 4926.1ms/step | 4.1 samples/s
228
+ Step 570/44000 | Loss: 5.3069 | LR: 3.80e-04 | 4764.1ms/step | 4.2 samples/s
229
+ Step 580/44000 | Loss: 5.2648 | LR: 3.87e-04 | 4939.8ms/step | 4.0 samples/s
230
+ Step 590/44000 | Loss: 5.2003 | LR: 3.93e-04 | 4811.7ms/step | 4.2 samples/s
231
+ Step 600/44000 | Loss: 5.1619 | LR: 4.00e-04 | 4680.3ms/step | 4.3 samples/s
232
+ Step 610/44000 | Loss: 5.1348 | LR: 4.07e-04 | 4624.3ms/step | 4.3 samples/s
233
+ Step 620/44000 | Loss: 5.1263 | LR: 4.13e-04 | 4838.6ms/step | 4.1 samples/s
234
+ Step 630/44000 | Loss: 5.1120 | LR: 4.20e-04 | 4881.7ms/step | 4.1 samples/s
235
+ Step 640/44000 | Loss: 5.0980 | LR: 4.27e-04 | 4794.2ms/step | 4.2 samples/s
236
+ Step 650/44000 | Loss: 5.0563 | LR: 4.33e-04 | 5029.4ms/step | 4.0 samples/s
237
+ Step 660/44000 | Loss: 5.0270 | LR: 4.40e-04 | 4833.3ms/step | 4.1 samples/s
238
+ Step 670/44000 | Loss: 4.9956 | LR: 4.47e-04 | 4727.1ms/step | 4.2 samples/s
239
+ Step 680/44000 | Loss: 4.9640 | LR: 4.53e-04 | 4750.6ms/step | 4.2 samples/s
240
+ Step 690/44000 | Loss: 4.8893 | LR: 4.60e-04 | 4718.2ms/step | 4.2 samples/s
241
+ Step 700/44000 | Loss: 4.8953 | LR: 4.67e-04 | 4791.4ms/step | 4.2 samples/s
242
+ Step 710/44000 | Loss: 4.8613 | LR: 4.73e-04 | 4860.3ms/step | 4.1 samples/s
243
+ Step 720/44000 | Loss: 4.8417 | LR: 4.80e-04 | 4766.2ms/step | 4.2 samples/s
244
+ Step 730/44000 | Loss: 4.7862 | LR: 4.87e-04 | 4862.4ms/step | 4.1 samples/s
245
+ Step 740/44000 | Loss: 4.7912 | LR: 4.93e-04 | 4804.8ms/step | 4.2 samples/s
246
+ Step 750/44000 | Loss: 4.7387 | LR: 5.00e-04 | 4761.0ms/step | 4.2 samples/s
247
+ Step 760/44000 | Loss: 4.7204 | LR: 5.00e-04 | 4860.6ms/step | 4.1 samples/s
248
+ Step 770/44000 | Loss: 4.7007 | LR: 5.00e-04 | 4968.1ms/step | 4.0 samples/s
249
+ Step 780/44000 | Loss: 4.6794 | LR: 5.00e-04 | 5174.1ms/step | 3.9 samples/s
250
+ Step 790/44000 | Loss: 4.6193 | LR: 5.00e-04 | 5390.2ms/step | 3.7 samples/s
251
+ Step 800/44000 | Loss: 4.6028 | LR: 5.00e-04 | 5149.3ms/step | 3.9 samples/s
252
+ Step 810/44000 | Loss: 4.6039 | LR: 5.00e-04 | 5737.4ms/step | 3.5 samples/s
253
+ Step 820/44000 | Loss: 4.5670 | LR: 5.00e-04 | 5154.7ms/step | 3.9 samples/s
254
+ Step 830/44000 | Loss: 4.5433 | LR: 5.00e-04 | 5317.9ms/step | 3.8 samples/s
255
+ Step 840/44000 | Loss: 4.5283 | LR: 5.00e-04 | 5406.6ms/step | 3.7 samples/s
256
+ Step 850/44000 | Loss: 4.4838 | LR: 5.00e-04 | 5840.0ms/step | 3.4 samples/s
257
+ Step 860/44000 | Loss: 4.4818 | LR: 5.00e-04 | 5554.5ms/step | 3.6 samples/s
258
+ Step 870/44000 | Loss: 4.4403 | LR: 5.00e-04 | 5345.5ms/step | 3.7 samples/s
259
+ Step 880/44000 | Loss: 4.4495 | LR: 5.00e-04 | 5266.3ms/step | 3.8 samples/s
260
+ Step 890/44000 | Loss: 4.4343 | LR: 5.00e-04 | 5995.9ms/step | 3.3 samples/s
261
+ Step 900/44000 | Loss: 4.3761 | LR: 5.00e-04 | 5052.8ms/step | 4.0 samples/s
262
+ Step 910/44000 | Loss: 4.3897 | LR: 5.00e-04 | 5182.1ms/step | 3.9 samples/s
263
+ Step 920/44000 | Loss: 4.3440 | LR: 5.00e-04 | 5172.9ms/step | 3.9 samples/s
264
+ Step 930/44000 | Loss: 4.3465 | LR: 5.00e-04 | 5822.5ms/step | 3.4 samples/s
265
+ Step 940/44000 | Loss: 4.3114 | LR: 5.00e-04 | 5445.9ms/step | 3.7 samples/s
266
+ Step 950/44000 | Loss: 4.3111 | LR: 5.00e-04 | 4921.4ms/step | 4.1 samples/s
267
+ Step 960/44000 | Loss: 4.3476 | LR: 5.00e-04 | 4520.5ms/step | 4.4 samples/s
268
+ Step 970/44000 | Loss: 4.3198 | LR: 5.00e-04 | 4613.9ms/step | 4.3 samples/s
269
+ Step 980/44000 | Loss: 4.2712 | LR: 5.00e-04 | 5348.6ms/step | 3.7 samples/s
270
+ Step 990/44000 | Loss: 4.2731 | LR: 5.00e-04 | 4634.2ms/step | 4.3 samples/s
271
+ Step 1000/44000 | Loss: 4.2224 | LR: 5.00e-04 | 5219.9ms/step | 3.8 samples/s
epfl_final/train_logs/ropectl_full_a1p0/seed46.log ADDED
@@ -0,0 +1,271 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ New run: Checkpoints will be saved to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735
2
+ Running config:
3
+ {
4
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
5
+ "model_dtype": "bfloat16",
6
+ "init_from_scratch": true,
7
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
8
+ "val_path": null,
9
+ "max_seq_length": 8192,
10
+ "num_workers": 8,
11
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_ropectl_a1p0_seed46/run_20260819_153735",
12
+ "resume_from_checkpoint": null,
13
+ "force_new_run": true,
14
+ "num_epochs": 1,
15
+ "max_steps": 44000,
16
+ "per_device_batch_size": 2,
17
+ "gradient_accumulation_steps": 10,
18
+ "learning_rate": 0.0005,
19
+ "weight_decay": 0.01,
20
+ "warmup_steps": 750,
21
+ "lr_scheduler": "cosine",
22
+ "max_grad_norm": 1.0,
23
+ "save_steps": 1000,
24
+ "eval_steps": 1000,
25
+ "eval_batches": 50,
26
+ "logging_steps": 10,
27
+ "seed": 46,
28
+ "verbose": true,
29
+ "checkpoint_activations": false,
30
+ "strategy": "ddp",
31
+ "mixed_precision": "bf16",
32
+ "wandb_enabled": false,
33
+ "wandb_project": "stochastic_sparsity",
34
+ "wandb_name": "l8k365m_dense_ropectl_a1p0_seed46",
35
+ "wandb_tags": [
36
+ "qwen3",
37
+ "365m",
38
+ "8k",
39
+ "dense",
40
+ "rope-control"
41
+ ],
42
+ "hidden_size": 768,
43
+ "intermediate_size": 3072,
44
+ "num_attention_heads": 6,
45
+ "num_key_value_heads": 2,
46
+ "num_hidden_layers": 28,
47
+ "head_dim": null,
48
+ "attention_dropout": 0.0,
49
+ "rope_train_position_scale_set": [
50
+ 1.0,
51
+ 0.9896339488176222,
52
+ 0.9692159692159692,
53
+ 0.9420289855072463,
54
+ 0.8730158730158729
55
+ ],
56
+ "attention_type": "sparse_stochastic",
57
+ "sparse_block_size": 128,
58
+ "sparse_num_random_blocks": 31,
59
+ "sparse_mask_pool_size": 1,
60
+ "sparse_val_dense_compare": false,
61
+ "sparse_layer_indices": null,
62
+ "sparse_mask_cache_dir": null,
63
+ "sparse_mask_seed": 0,
64
+ "sparse_mixed_nrb_set": null,
65
+ "dense_mode": true,
66
+ "dense_step_intervals": null
67
+ }
68
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
69
+ Initializing model from scratch (random weights)...
70
+ Checkpoint activations: False
71
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
72
+
73
+
74
+ ================================================================================
75
+ MODEL ARCHITECTURE
76
+ ================================================================================
77
+ Model config:
78
+ vocab_size: 151936
79
+ hidden_size: 768
80
+ num_hidden_layers: 28
81
+ num_attention_heads: 6
82
+ num_key_value_heads: 2
83
+ head_dim: 128
84
+ intermediate_size: 3072
85
+ max_position_embeddings: 32768
86
+ rope_theta: 1000000
87
+ rope_train_position_scale_set: [1.0, 0.9896339488176222, 0.9692159692159692, 0.9420289855072463, 0.8730158730158729]
88
+ attention_type: sparse_stochastic
89
+ sparse_block_size: 128
90
+ sparse_num_random_blocks: 31
91
+ sparse_mask_pool_size: 1
92
+ sparse_max_seq_length: 8192
93
+
94
+ Parameter count:
95
+ Total parameters: 358,958,848
96
+ Trainable parameters: 358,958,848
97
+ Non-trainable parameters: 0
98
+ Model size (BF16): 0.72 GB
99
+
100
+ --------------------------------------------------------------------------------
101
+ LAYER-BY-LAYER SUMMARY
102
+ --------------------------------------------------------------------------------
103
+ ================================================================================================================================================================
104
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
105
+ ================================================================================================================================================================
106
+ Qwen3ForCausalLM [2, 8192] -- -- True
107
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
108
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
109
+ │ └─ModuleList: 2-2 -- -- -- True
110
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
111
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
132
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
133
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
134
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
135
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
136
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
137
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
138
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
139
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
140
+ ================================================================================================================================================================
141
+ Total params: 475,645,696
142
+ Trainable params: 475,645,696
143
+ Non-trainable params: 0
144
+ Total mult-adds (Units.MEGABYTES): 951.29
145
+ ================================================================================================================================================================
146
+ Input size (MB): 0.13
147
+ Forward/backward pass size (MB): 31197.23
148
+ Params size (MB): 951.29
149
+ Estimated Total Size (MB): 32148.66
150
+ ================================================================================================================================================================
151
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
152
+ ================================================================================
153
+
154
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
155
+ Using pre-tokenized dataset (no runtime tokenization)
156
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
157
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
158
+ Dataset split: 3027764 train, 159356 validation
159
+ Training examples (packed, train split): 3027764
160
+ Validation examples (packed, val split): 159356
161
+ Per-device batch size: 2
162
+ World size (GPUs): 1
163
+ Gradient accumulation steps: 10
164
+ Effective batch size (seqs/step): 20
165
+ Sequence length: 8192
166
+ Tokens per optimizer step: 163,840
167
+ Micro-batches per epoch (per rank): 1513882
168
+ Optimizer steps per epoch: 151388
169
+ Total optimizer steps: 44000 (capped by max_steps; 0.29 epochs)
170
+ Starting training...
171
+ Switching attention mode for step 1: dense (dense step interval)
172
+ Step 10/44000 | Loss: 12.0605 | LR: 6.67e-06 | 4895.7ms/step | 4.1 samples/s
173
+ Step 20/44000 | Loss: 11.9673 | LR: 1.33e-05 | 4391.0ms/step | 4.6 samples/s
174
+ Step 30/44000 | Loss: 11.6573 | LR: 2.00e-05 | 4413.3ms/step | 4.5 samples/s
175
+ Step 40/44000 | Loss: 11.1194 | LR: 2.67e-05 | 4477.1ms/step | 4.5 samples/s
176
+ Step 50/44000 | Loss: 10.7791 | LR: 3.33e-05 | 4503.8ms/step | 4.4 samples/s
177
+ Step 60/44000 | Loss: 10.5590 | LR: 4.00e-05 | 4471.5ms/step | 4.5 samples/s
178
+ Step 70/44000 | Loss: 10.3036 | LR: 4.67e-05 | 4429.0ms/step | 4.5 samples/s
179
+ Step 80/44000 | Loss: 10.0085 | LR: 5.33e-05 | 4480.7ms/step | 4.5 samples/s
180
+ Step 90/44000 | Loss: 9.5904 | LR: 6.00e-05 | 4800.0ms/step | 4.2 samples/s
181
+ Step 100/44000 | Loss: 9.1686 | LR: 6.67e-05 | 4673.2ms/step | 4.3 samples/s
182
+ Step 110/44000 | Loss: 8.6288 | LR: 7.33e-05 | 4638.9ms/step | 4.3 samples/s
183
+ Step 120/44000 | Loss: 8.1674 | LR: 8.00e-05 | 4625.5ms/step | 4.3 samples/s
184
+ Step 130/44000 | Loss: 7.7940 | LR: 8.67e-05 | 4651.2ms/step | 4.3 samples/s
185
+ Step 140/44000 | Loss: 7.5257 | LR: 9.33e-05 | 4733.1ms/step | 4.2 samples/s
186
+ Step 150/44000 | Loss: 7.3621 | LR: 1.00e-04 | 4636.8ms/step | 4.3 samples/s
187
+ Step 160/44000 | Loss: 7.2137 | LR: 1.07e-04 | 4708.1ms/step | 4.2 samples/s
188
+ Step 170/44000 | Loss: 7.1333 | LR: 1.13e-04 | 4584.6ms/step | 4.4 samples/s
189
+ Step 180/44000 | Loss: 7.0559 | LR: 1.20e-04 | 4577.3ms/step | 4.4 samples/s
190
+ Step 190/44000 | Loss: 6.9389 | LR: 1.27e-04 | 4684.5ms/step | 4.3 samples/s
191
+ Step 200/44000 | Loss: 6.8668 | LR: 1.33e-04 | 4722.9ms/step | 4.2 samples/s
192
+ Step 210/44000 | Loss: 6.7607 | LR: 1.40e-04 | 4723.0ms/step | 4.2 samples/s
193
+ Step 220/44000 | Loss: 6.6681 | LR: 1.47e-04 | 4678.1ms/step | 4.3 samples/s
194
+ Step 230/44000 | Loss: 6.5869 | LR: 1.53e-04 | 4615.4ms/step | 4.3 samples/s
195
+ Step 240/44000 | Loss: 6.5316 | LR: 1.60e-04 | 4824.2ms/step | 4.1 samples/s
196
+ Step 250/44000 | Loss: 6.4987 | LR: 1.67e-04 | 4618.4ms/step | 4.3 samples/s
197
+ Step 260/44000 | Loss: 6.3916 | LR: 1.73e-04 | 4744.5ms/step | 4.2 samples/s
198
+ Step 270/44000 | Loss: 6.3680 | LR: 1.80e-04 | 4695.9ms/step | 4.3 samples/s
199
+ Step 280/44000 | Loss: 6.3096 | LR: 1.87e-04 | 4654.6ms/step | 4.3 samples/s
200
+ Step 290/44000 | Loss: 6.2770 | LR: 1.93e-04 | 4680.0ms/step | 4.3 samples/s
201
+ Step 300/44000 | Loss: 6.2120 | LR: 2.00e-04 | 4619.3ms/step | 4.3 samples/s
202
+ Step 310/44000 | Loss: 6.1608 | LR: 2.07e-04 | 4562.4ms/step | 4.4 samples/s
203
+ Step 320/44000 | Loss: 6.1311 | LR: 2.13e-04 | 4630.3ms/step | 4.3 samples/s
204
+ Step 330/44000 | Loss: 6.0853 | LR: 2.20e-04 | 4666.6ms/step | 4.3 samples/s
205
+ Step 340/44000 | Loss: 6.0326 | LR: 2.27e-04 | 5005.7ms/step | 4.0 samples/s
206
+ Step 350/44000 | Loss: 5.9896 | LR: 2.33e-04 | 4860.4ms/step | 4.1 samples/s
207
+ Step 360/44000 | Loss: 5.9760 | LR: 2.40e-04 | 4827.0ms/step | 4.1 samples/s
208
+ Step 370/44000 | Loss: 5.9269 | LR: 2.47e-04 | 4863.2ms/step | 4.1 samples/s
209
+ Step 380/44000 | Loss: 5.8925 | LR: 2.53e-04 | 4761.4ms/step | 4.2 samples/s
210
+ Step 390/44000 | Loss: 5.8783 | LR: 2.60e-04 | 4820.9ms/step | 4.1 samples/s
211
+ Step 400/44000 | Loss: 5.8371 | LR: 2.67e-04 | 4801.3ms/step | 4.2 samples/s
212
+ Step 410/44000 | Loss: 5.7788 | LR: 2.73e-04 | 4772.6ms/step | 4.2 samples/s
213
+ Step 420/44000 | Loss: 5.7767 | LR: 2.80e-04 | 4711.7ms/step | 4.2 samples/s
214
+ Step 430/44000 | Loss: 5.7324 | LR: 2.87e-04 | 4757.6ms/step | 4.2 samples/s
215
+ Step 440/44000 | Loss: 5.7008 | LR: 2.93e-04 | 4678.7ms/step | 4.3 samples/s
216
+ Step 450/44000 | Loss: 5.6651 | LR: 3.00e-04 | 4751.6ms/step | 4.2 samples/s
217
+ Step 460/44000 | Loss: 5.5993 | LR: 3.07e-04 | 4730.4ms/step | 4.2 samples/s
218
+ Step 470/44000 | Loss: 5.5551 | LR: 3.13e-04 | 4762.9ms/step | 4.2 samples/s
219
+ Step 480/44000 | Loss: 5.5728 | LR: 3.20e-04 | 4654.9ms/step | 4.3 samples/s
220
+ Step 490/44000 | Loss: 5.5186 | LR: 3.27e-04 | 4572.2ms/step | 4.4 samples/s
221
+ Step 500/44000 | Loss: 5.4752 | LR: 3.33e-04 | 4661.4ms/step | 4.3 samples/s
222
+ Step 510/44000 | Loss: 5.4343 | LR: 3.40e-04 | 4650.1ms/step | 4.3 samples/s
223
+ Step 520/44000 | Loss: 5.4029 | LR: 3.47e-04 | 4707.3ms/step | 4.2 samples/s
224
+ Step 530/44000 | Loss: 5.3977 | LR: 3.53e-04 | 4729.8ms/step | 4.2 samples/s
225
+ Step 540/44000 | Loss: 5.3520 | LR: 3.60e-04 | 4705.7ms/step | 4.3 samples/s
226
+ Step 550/44000 | Loss: 5.3023 | LR: 3.67e-04 | 4647.3ms/step | 4.3 samples/s
227
+ Step 560/44000 | Loss: 5.2960 | LR: 3.73e-04 | 4657.9ms/step | 4.3 samples/s
228
+ Step 570/44000 | Loss: 5.2328 | LR: 3.80e-04 | 4653.2ms/step | 4.3 samples/s
229
+ Step 580/44000 | Loss: 5.2121 | LR: 3.87e-04 | 4742.9ms/step | 4.2 samples/s
230
+ Step 590/44000 | Loss: 5.2089 | LR: 3.93e-04 | 4687.1ms/step | 4.3 samples/s
231
+ Step 600/44000 | Loss: 5.1776 | LR: 4.00e-04 | 4706.6ms/step | 4.2 samples/s
232
+ Step 610/44000 | Loss: 5.1638 | LR: 4.07e-04 | 4657.8ms/step | 4.3 samples/s
233
+ Step 620/44000 | Loss: 5.1021 | LR: 4.13e-04 | 4540.7ms/step | 4.4 samples/s
234
+ Step 630/44000 | Loss: 5.0679 | LR: 4.20e-04 | 4696.7ms/step | 4.3 samples/s
235
+ Step 640/44000 | Loss: 5.0550 | LR: 4.27e-04 | 4989.7ms/step | 4.0 samples/s
236
+ Step 650/44000 | Loss: 5.0510 | LR: 4.33e-04 | 4805.2ms/step | 4.2 samples/s
237
+ Step 660/44000 | Loss: 5.0292 | LR: 4.40e-04 | 4739.3ms/step | 4.2 samples/s
238
+ Step 670/44000 | Loss: 4.9869 | LR: 4.47e-04 | 4984.4ms/step | 4.0 samples/s
239
+ Step 680/44000 | Loss: 4.9637 | LR: 4.53e-04 | 4721.6ms/step | 4.2 samples/s
240
+ Step 690/44000 | Loss: 4.8999 | LR: 4.60e-04 | 4874.4ms/step | 4.1 samples/s
241
+ Step 700/44000 | Loss: 4.9207 | LR: 4.67e-04 | 4880.8ms/step | 4.1 samples/s
242
+ Step 710/44000 | Loss: 4.8444 | LR: 4.73e-04 | 4791.4ms/step | 4.2 samples/s
243
+ Step 720/44000 | Loss: 4.8104 | LR: 4.80e-04 | 4830.1ms/step | 4.1 samples/s
244
+ Step 730/44000 | Loss: 4.8303 | LR: 4.87e-04 | 4726.8ms/step | 4.2 samples/s
245
+ Step 740/44000 | Loss: 4.7742 | LR: 4.93e-04 | 4770.1ms/step | 4.2 samples/s
246
+ Step 750/44000 | Loss: 4.7188 | LR: 5.00e-04 | 4755.4ms/step | 4.2 samples/s
247
+ Step 760/44000 | Loss: 4.7019 | LR: 5.00e-04 | 4634.1ms/step | 4.3 samples/s
248
+ Step 770/44000 | Loss: 4.6880 | LR: 5.00e-04 | 4583.7ms/step | 4.4 samples/s
249
+ Step 780/44000 | Loss: 4.6344 | LR: 5.00e-04 | 4675.1ms/step | 4.3 samples/s
250
+ Step 790/44000 | Loss: 4.6372 | LR: 5.00e-04 | 4774.9ms/step | 4.2 samples/s
251
+ Step 800/44000 | Loss: 4.6152 | LR: 5.00e-04 | 4640.8ms/step | 4.3 samples/s
252
+ Step 810/44000 | Loss: 4.6136 | LR: 5.00e-04 | 4624.2ms/step | 4.3 samples/s
253
+ Step 820/44000 | Loss: 4.5447 | LR: 5.00e-04 | 4821.2ms/step | 4.1 samples/s
254
+ Step 830/44000 | Loss: 4.5456 | LR: 5.00e-04 | 4723.3ms/step | 4.2 samples/s
255
+ Step 840/44000 | Loss: 4.5518 | LR: 5.00e-04 | 4714.7ms/step | 4.2 samples/s
256
+ Step 850/44000 | Loss: 4.4382 | LR: 5.00e-04 | 4828.8ms/step | 4.1 samples/s
257
+ Step 860/44000 | Loss: 4.5036 | LR: 5.00e-04 | 4961.2ms/step | 4.0 samples/s
258
+ Step 870/44000 | Loss: 4.4355 | LR: 5.00e-04 | 4862.7ms/step | 4.1 samples/s
259
+ Step 880/44000 | Loss: 4.4336 | LR: 5.00e-04 | 4910.1ms/step | 4.1 samples/s
260
+ Step 890/44000 | Loss: 4.4458 | LR: 5.00e-04 | 4848.7ms/step | 4.1 samples/s
261
+ Step 900/44000 | Loss: 4.3783 | LR: 5.00e-04 | 4848.4ms/step | 4.1 samples/s
262
+ Step 910/44000 | Loss: 4.3739 | LR: 5.00e-04 | 4904.9ms/step | 4.1 samples/s
263
+ Step 920/44000 | Loss: 4.3259 | LR: 5.00e-04 | 4814.5ms/step | 4.2 samples/s
264
+ Step 930/44000 | Loss: 4.3706 | LR: 5.00e-04 | 4734.7ms/step | 4.2 samples/s
265
+ Step 940/44000 | Loss: 4.3025 | LR: 5.00e-04 | 4658.6ms/step | 4.3 samples/s
266
+ Step 950/44000 | Loss: 4.3169 | LR: 5.00e-04 | 4753.4ms/step | 4.2 samples/s
267
+ Step 960/44000 | Loss: 4.2929 | LR: 5.00e-04 | 4732.9ms/step | 4.2 samples/s
268
+ Step 970/44000 | Loss: 4.2888 | LR: 5.00e-04 | 4674.3ms/step | 4.3 samples/s
269
+ Step 980/44000 | Loss: 4.2544 | LR: 5.00e-04 | 4642.0ms/step | 4.3 samples/s
270
+ Step 990/44000 | Loss: 4.2404 | LR: 5.00e-04 | 4824.8ms/step | 4.1 samples/s
271
+ Step 1000/44000 | Loss: 4.2095 | LR: 5.00e-04 | 4369.4ms/step | 4.6 samples/s