Hugging Face's logo

rubentium
/
sparse-checkpoints

Model card Files
xet
 Community
rubentium commited on
Commit
91e8eee
·
verified ·
1 Parent(s): 1a0e2cd

EPFL ALiBi and training provenance 1151-1184 of 1184

Browse files
Files changed (34) hide show
  1. epfl_final/dropctl_logs/calib_p00.log +204 -0
  2. epfl_final/dropctl_logs/calib_p05.log +204 -0
  3. epfl_final/dropctl_logs/calib_p10.log +204 -0
  4. epfl_final/dropctl_logs/calib_p20.log +204 -0
  5. epfl_final/dropctl_logs/calib_p28_verify.log +168 -0
  6. epfl_final/dropctl_logs/calib_p28_verify_r2.log +206 -0
  7. epfl_final/dropctl_logs/calib_p30_verify.log +168 -0
  8. epfl_final/dropctl_logs/calib_p30_verify_r2.log +206 -0
  9. epfl_final/dropctl_logs/diag_p28_20.log +174 -0
  10. epfl_final/eval_logs/r2_heldout_expanded_20260820.log +164 -0
  11. epfl_final/eval_logs/r2_temperature_ruler_20260820.log +1 -0
  12. epfl_final/eval_logs/r2_temperature_shards/a100_gpu4.log +3 -0
  13. epfl_final/eval_logs/r2_temperature_shards/h100_gpu2.log +64 -0
  14. epfl_final/eval_logs/r2_temperature_shards/h100_gpu2_tail.log +8 -0
  15. epfl_final/eval_logs/r2_temperature_shards/h100_gpu3.log +64 -0
  16. epfl_final/eval_logs/r2_temperature_shards/h100_gpu3_tail.log +8 -0
  17. epfl_final/eval_logs/r2_temperature_shards/h100_tail_watcher.log +0 -0
  18. epfl_final/eval_logs/rope42_sharded_gpu0_20260821.log +7 -0
  19. epfl_final/eval_logs/rope42_sharded_gpu1_20260821.log +7 -0
  20. epfl_final/eval_logs/rope42_sharded_gpu2_20260821.log +7 -0
  21. epfl_final/eval_logs/rope42_sharded_gpu3_20260821.log +7 -0
  22. epfl_final/eval_logs/rope43_sharded_gpu1_20260821.log +9 -0
  23. epfl_final/eval_logs/rope43_sharded_gpu2_20260821.log +9 -0
  24. epfl_final/eval_logs/rope43_sharded_gpu3_20260821.log +9 -0
  25. epfl_final/eval_logs/rope44_a1005_20260821/sharded.log +74 -0
  26. epfl_final/eval_logs/rope44_after_mixed48_20260821/watcher.log +15 -0
  27. epfl_final/eval_logs/rope44_auto_20260821/watcher.log +332 -0
  28. epfl_final/eval_logs/rope45_auto_20260821/watcher.log +235 -0
  29. epfl_final/eval_logs/rope_seed46_matched_20260820.log +24 -0
  30. epfl_final/eval_logs/seed47_dense_matched_watcher_20260820.log +27 -0
  31. epfl_final/eval_logs/seed47_mixed_matched_watcher_20260820.log +26 -0
  32. epfl_final/eval_logs/seed48_dense_auto_20260821/watcher.log +222 -0
  33. epfl_final/eval_logs/seed48_mixed_auto_20260821/watcher.log +164 -0
  34. epfl_final/train_logs/p1_scheduled_a100_seed47_20260822/l8k365m_resampled_nrb35_seed47.log +0 -0
epfl_final/dropctl_logs/calib_p00.log ADDED
@@ -0,0 +1,204 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ New run: Checkpoints will be saved to /mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p00/run_20260807_080949
2
+ Running config:
3
+ {
4
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
5
+ "model_dtype": "bfloat16",
6
+ "init_from_scratch": false,
7
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
8
+ "val_path": "/mloscratch/homes/navasard/data/fineweb_val_seed42",
9
+ "max_seq_length": 8192,
10
+ "num_workers": 8,
11
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p00/run_20260807_080949",
12
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000",
13
+ "force_new_run": true,
14
+ "num_epochs": 1,
15
+ "max_steps": 44300,
16
+ "per_device_batch_size": 2,
17
+ "gradient_accumulation_steps": 10,
18
+ "learning_rate": 5e-05,
19
+ "weight_decay": 0.01,
20
+ "warmup_steps": 0,
21
+ "lr_scheduler": "constant",
22
+ "max_grad_norm": 1.0,
23
+ "save_steps": 300,
24
+ "eval_steps": 75,
25
+ "eval_batches": 50,
26
+ "logging_steps": 10,
27
+ "seed": 42,
28
+ "verbose": true,
29
+ "checkpoint_activations": false,
30
+ "strategy": "ddp",
31
+ "mixed_precision": "bf16",
32
+ "wandb_enabled": false,
33
+ "wandb_project": "stochastic_sparsity",
34
+ "wandb_name": "dropctl_calib_p00",
35
+ "wandb_tags": [
36
+ "qwen3",
37
+ "365m",
38
+ "8k",
39
+ "scaleladder",
40
+ "dense"
41
+ ],
42
+ "hidden_size": 768,
43
+ "intermediate_size": 3072,
44
+ "num_attention_heads": 6,
45
+ "num_key_value_heads": 2,
46
+ "num_hidden_layers": 28,
47
+ "head_dim": null,
48
+ "attention_dropout": 0.0,
49
+ "attention_type": "sparse_stochastic",
50
+ "sparse_block_size": 128,
51
+ "sparse_num_random_blocks": 31,
52
+ "sparse_mask_pool_size": 1,
53
+ "sparse_val_dense_compare": false,
54
+ "sparse_layer_indices": null,
55
+ "sparse_mask_cache_dir": null,
56
+ "sparse_mask_seed": 0,
57
+ "dense_mode": true,
58
+ "dense_step_intervals": null
59
+ }
60
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
61
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000
62
+ Checkpoint activations: False
63
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
64
+
65
+
66
+ ================================================================================
67
+ MODEL ARCHITECTURE
68
+ ================================================================================
69
+ Model config:
70
+ vocab_size: 151936
71
+ hidden_size: 768
72
+ num_hidden_layers: 28
73
+ num_attention_heads: 6
74
+ num_key_value_heads: 2
75
+ head_dim: 128
76
+ intermediate_size: 3072
77
+ max_position_embeddings: 32768
78
+ rope_theta: 1000000
79
+ attention_type: sparse_stochastic
80
+ sparse_block_size: 128
81
+ sparse_num_random_blocks: 31
82
+ sparse_mask_pool_size: 1
83
+ sparse_max_seq_length: 8192
84
+
85
+ Parameter count:
86
+ Total parameters: 358,958,848
87
+ Trainable parameters: 358,958,848
88
+ Non-trainable parameters: 0
89
+ Model size (BF16): 0.72 GB
90
+
91
+ --------------------------------------------------------------------------------
92
+ LAYER-BY-LAYER SUMMARY
93
+ --------------------------------------------------------------------------------
94
+ ================================================================================================================================================================
95
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
96
+ ================================================================================================================================================================
97
+ Qwen3ForCausalLM [2, 8192] -- -- True
98
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
99
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
100
+ │ └─ModuleList: 2-2 -- -- -- True
101
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
102
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
103
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
104
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
105
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
106
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
107
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
108
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
109
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
110
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
111
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
130
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
131
+ ================================================================================================================================================================
132
+ Total params: 475,645,696
133
+ Trainable params: 475,645,696
134
+ Non-trainable params: 0
135
+ Total mult-adds (Units.MEGABYTES): 951.29
136
+ ================================================================================================================================================================
137
+ Input size (MB): 0.13
138
+ Forward/backward pass size (MB): 31197.23
139
+ Params size (MB): 951.29
140
+ Estimated Total Size (MB): 32148.66
141
+ ================================================================================================================================================================
142
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
143
+ ================================================================================
144
+
145
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
146
+ Using pre-tokenized dataset (no runtime tokenization)
147
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
148
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
149
+ Dataset split: 3027764 train, 159356 validation
150
+ Training examples (packed, train split): 3027764
151
+ Validation examples (packed, val split): 159356
152
+ Per-device batch size: 2
153
+ World size (GPUs): 1
154
+ Gradient accumulation steps: 10
155
+ Effective batch size (seqs/step): 20
156
+ Sequence length: 8192
157
+ Tokens per optimizer step: 163,840
158
+ Micro-batches per epoch (per rank): 1513882
159
+ Optimizer steps per epoch: 151388
160
+ Total optimizer steps: 44300 (capped by max_steps; 0.29 epochs)
161
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000...
162
+ Overriding checkpoint LR with config LR: 5e-05
163
+ Resumed from checkpoint at step 44000 (epoch 0, micro_step 440000)
164
+ Resuming training from step 44000
165
+ Starting training...
166
+ Validation at step 44000: Loss = 2.9127, Perplexity = 18.41
167
+ Switching attention mode for step 44001: dense (dense step interval)
168
+ Step 44010/44300 | Loss: 2.8865 | LR: 5.00e-05 | 62565.3ms/step | 0.3 samples/s
169
+ Step 44020/44300 | Loss: 2.9177 | LR: 5.00e-05 | 1367.1ms/step | 14.6 samples/s
170
+ Validation at step 44025: Loss = 2.8962, Perplexity = 18.11
171
+ Step 44030/44300 | Loss: 2.8593 | LR: 5.00e-05 | 3540.7ms/step | 5.6 samples/s
172
+ Step 44040/44300 | Loss: 2.8975 | LR: 5.00e-05 | 1372.3ms/step | 14.6 samples/s
173
+ Step 44050/44300 | Loss: 2.8827 | LR: 5.00e-05 | 1377.1ms/step | 14.5 samples/s
174
+ Step 44060/44300 | Loss: 2.9121 | LR: 5.00e-05 | 1381.7ms/step | 14.5 samples/s
175
+ Step 44070/44300 | Loss: 2.8701 | LR: 5.00e-05 | 1382.9ms/step | 14.5 samples/s
176
+ Step 44080/44300 | Loss: 2.8777 | LR: 5.00e-05 | 1384.2ms/step | 14.4 samples/s
177
+ Step 44090/44300 | Loss: 2.9046 | LR: 5.00e-05 | 1383.8ms/step | 14.5 samples/s
178
+ Step 44100/44300 | Loss: 2.9037 | LR: 5.00e-05 | 1387.5ms/step | 14.4 samples/s
179
+ Validation at step 44100: Loss = 2.8947, Perplexity = 18.08
180
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p00/run_20260807_080949/checkpoint-44100
181
+ Step 44110/44300 | Loss: 2.8832 | LR: 5.00e-05 | 3868.4ms/step | 5.2 samples/s
182
+ Step 44120/44300 | Loss: 2.8709 | LR: 5.00e-05 | 1382.6ms/step | 14.5 samples/s
183
+ Step 44130/44300 | Loss: 2.8866 | LR: 5.00e-05 | 1382.8ms/step | 14.5 samples/s
184
+ Step 44140/44300 | Loss: 2.8771 | LR: 5.00e-05 | 1385.2ms/step | 14.4 samples/s
185
+ Step 44150/44300 | Loss: 2.8798 | LR: 5.00e-05 | 1385.2ms/step | 14.4 samples/s
186
+ Step 44160/44300 | Loss: 2.8648 | LR: 5.00e-05 | 1384.5ms/step | 14.4 samples/s
187
+ Step 44170/44300 | Loss: 2.8635 | LR: 5.00e-05 | 1385.3ms/step | 14.4 samples/s
188
+ Validation at step 44175: Loss = 2.8946, Perplexity = 18.08
189
+ Step 44180/44300 | Loss: 2.8969 | LR: 5.00e-05 | 3575.9ms/step | 5.6 samples/s
190
+ Step 44190/44300 | Loss: 2.9045 | LR: 5.00e-05 | 1387.1ms/step | 14.4 samples/s
191
+ Step 44200/44300 | Loss: 2.9012 | LR: 5.00e-05 | 1387.7ms/step | 14.4 samples/s
192
+ Step 44210/44300 | Loss: 2.8813 | LR: 5.00e-05 | 1385.9ms/step | 14.4 samples/s
193
+ Step 44220/44300 | Loss: 2.8551 | LR: 5.00e-05 | 1387.3ms/step | 14.4 samples/s
194
+ Step 44230/44300 | Loss: 2.8611 | LR: 5.00e-05 | 1387.0ms/step | 14.4 samples/s
195
+ Step 44240/44300 | Loss: 2.8830 | LR: 5.00e-05 | 1385.8ms/step | 14.4 samples/s
196
+ Step 44250/44300 | Loss: 2.9029 | LR: 5.00e-05 | 1389.4ms/step | 14.4 samples/s
197
+ Validation at step 44250: Loss = 2.8945, Perplexity = 18.07
198
+ Step 44260/44300 | Loss: 2.8805 | LR: 5.00e-05 | 3580.5ms/step | 5.6 samples/s
199
+ Step 44270/44300 | Loss: 2.8627 | LR: 5.00e-05 | 1387.6ms/step | 14.4 samples/s
200
+ Step 44280/44300 | Loss: 2.8768 | LR: 5.00e-05 | 1386.8ms/step | 14.4 samples/s
201
+ Step 44290/44300 | Loss: 2.8900 | LR: 5.00e-05 | 1389.1ms/step | 14.4 samples/s
202
+ Step 44300/44300 | Loss: 2.8960 | LR: 5.00e-05 | 1387.8ms/step | 14.4 samples/s
203
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p00/run_20260807_080949/checkpoint-44300
204
+ Training complete!
epfl_final/dropctl_logs/calib_p05.log ADDED
@@ -0,0 +1,204 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ New run: Checkpoints will be saved to /mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p05/run_20260807_080949
2
+ Running config:
3
+ {
4
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
5
+ "model_dtype": "bfloat16",
6
+ "init_from_scratch": false,
7
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
8
+ "val_path": "/mloscratch/homes/navasard/data/fineweb_val_seed42",
9
+ "max_seq_length": 8192,
10
+ "num_workers": 8,
11
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p05/run_20260807_080949",
12
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000",
13
+ "force_new_run": true,
14
+ "num_epochs": 1,
15
+ "max_steps": 44300,
16
+ "per_device_batch_size": 2,
17
+ "gradient_accumulation_steps": 10,
18
+ "learning_rate": 5e-05,
19
+ "weight_decay": 0.01,
20
+ "warmup_steps": 0,
21
+ "lr_scheduler": "constant",
22
+ "max_grad_norm": 1.0,
23
+ "save_steps": 300,
24
+ "eval_steps": 75,
25
+ "eval_batches": 50,
26
+ "logging_steps": 10,
27
+ "seed": 42,
28
+ "verbose": true,
29
+ "checkpoint_activations": false,
30
+ "strategy": "ddp",
31
+ "mixed_precision": "bf16",
32
+ "wandb_enabled": false,
33
+ "wandb_project": "stochastic_sparsity",
34
+ "wandb_name": "dropctl_calib_p05",
35
+ "wandb_tags": [
36
+ "qwen3",
37
+ "365m",
38
+ "8k",
39
+ "scaleladder",
40
+ "dense"
41
+ ],
42
+ "hidden_size": 768,
43
+ "intermediate_size": 3072,
44
+ "num_attention_heads": 6,
45
+ "num_key_value_heads": 2,
46
+ "num_hidden_layers": 28,
47
+ "head_dim": null,
48
+ "attention_dropout": 0.05,
49
+ "attention_type": "sparse_stochastic",
50
+ "sparse_block_size": 128,
51
+ "sparse_num_random_blocks": 31,
52
+ "sparse_mask_pool_size": 1,
53
+ "sparse_val_dense_compare": false,
54
+ "sparse_layer_indices": null,
55
+ "sparse_mask_cache_dir": null,
56
+ "sparse_mask_seed": 0,
57
+ "dense_mode": true,
58
+ "dense_step_intervals": null
59
+ }
60
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
61
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000
62
+ Checkpoint activations: False
63
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
64
+
65
+
66
+ ================================================================================
67
+ MODEL ARCHITECTURE
68
+ ================================================================================
69
+ Model config:
70
+ vocab_size: 151936
71
+ hidden_size: 768
72
+ num_hidden_layers: 28
73
+ num_attention_heads: 6
74
+ num_key_value_heads: 2
75
+ head_dim: 128
76
+ intermediate_size: 3072
77
+ max_position_embeddings: 32768
78
+ rope_theta: 1000000
79
+ attention_type: sparse_stochastic
80
+ sparse_block_size: 128
81
+ sparse_num_random_blocks: 31
82
+ sparse_mask_pool_size: 1
83
+ sparse_max_seq_length: 8192
84
+
85
+ Parameter count:
86
+ Total parameters: 358,958,848
87
+ Trainable parameters: 358,958,848
88
+ Non-trainable parameters: 0
89
+ Model size (BF16): 0.72 GB
90
+
91
+ --------------------------------------------------------------------------------
92
+ LAYER-BY-LAYER SUMMARY
93
+ --------------------------------------------------------------------------------
94
+ ================================================================================================================================================================
95
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
96
+ ================================================================================================================================================================
97
+ Qwen3ForCausalLM [2, 8192] -- -- True
98
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
99
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
100
+ │ └─ModuleList: 2-2 -- -- -- True
101
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
102
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
103
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
104
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
105
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
106
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
107
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
108
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
109
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
110
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
111
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
130
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
131
+ ================================================================================================================================================================
132
+ Total params: 475,645,696
133
+ Trainable params: 475,645,696
134
+ Non-trainable params: 0
135
+ Total mult-adds (Units.MEGABYTES): 951.29
136
+ ================================================================================================================================================================
137
+ Input size (MB): 0.13
138
+ Forward/backward pass size (MB): 31197.23
139
+ Params size (MB): 951.29
140
+ Estimated Total Size (MB): 32148.66
141
+ ================================================================================================================================================================
142
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
143
+ ================================================================================
144
+
145
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
146
+ Using pre-tokenized dataset (no runtime tokenization)
147
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
148
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
149
+ Dataset split: 3027764 train, 159356 validation
150
+ Training examples (packed, train split): 3027764
151
+ Validation examples (packed, val split): 159356
152
+ Per-device batch size: 2
153
+ World size (GPUs): 1
154
+ Gradient accumulation steps: 10
155
+ Effective batch size (seqs/step): 20
156
+ Sequence length: 8192
157
+ Tokens per optimizer step: 163,840
158
+ Micro-batches per epoch (per rank): 1513882
159
+ Optimizer steps per epoch: 151388
160
+ Total optimizer steps: 44300 (capped by max_steps; 0.29 epochs)
161
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000...
162
+ Overriding checkpoint LR with config LR: 5e-05
163
+ Resumed from checkpoint at step 44000 (epoch 0, micro_step 440000)
164
+ Resuming training from step 44000
165
+ Starting training...
166
+ Validation at step 44000: Loss = 2.9127, Perplexity = 18.41
167
+ Switching attention mode for step 44001: dense (dense step interval)
168
+ Step 44010/44300 | Loss: 2.9188 | LR: 5.00e-05 | 64385.9ms/step | 0.3 samples/s
169
+ Step 44020/44300 | Loss: 2.9483 | LR: 5.00e-05 | 1551.9ms/step | 12.9 samples/s
170
+ Validation at step 44025: Loss = 2.8976, Perplexity = 18.13
171
+ Step 44030/44300 | Loss: 2.8886 | LR: 5.00e-05 | 3760.9ms/step | 5.3 samples/s
172
+ Step 44040/44300 | Loss: 2.9270 | LR: 5.00e-05 | 1565.8ms/step | 12.8 samples/s
173
+ Step 44050/44300 | Loss: 2.9117 | LR: 5.00e-05 | 1567.9ms/step | 12.8 samples/s
174
+ Step 44060/44300 | Loss: 2.9404 | LR: 5.00e-05 | 1566.4ms/step | 12.8 samples/s
175
+ Step 44070/44300 | Loss: 2.8980 | LR: 5.00e-05 | 1568.3ms/step | 12.8 samples/s
176
+ Step 44080/44300 | Loss: 2.9058 | LR: 5.00e-05 | 1565.3ms/step | 12.8 samples/s
177
+ Step 44090/44300 | Loss: 2.9329 | LR: 5.00e-05 | 1567.9ms/step | 12.8 samples/s
178
+ Step 44100/44300 | Loss: 2.9311 | LR: 5.00e-05 | 1565.8ms/step | 12.8 samples/s
179
+ Validation at step 44100: Loss = 2.8961, Perplexity = 18.10
180
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p05/run_20260807_080949/checkpoint-44100
181
+ Step 44110/44300 | Loss: 2.9106 | LR: 5.00e-05 | 4078.9ms/step | 4.9 samples/s
182
+ Step 44120/44300 | Loss: 2.8988 | LR: 5.00e-05 | 1564.3ms/step | 12.8 samples/s
183
+ Step 44130/44300 | Loss: 2.9137 | LR: 5.00e-05 | 1564.1ms/step | 12.8 samples/s
184
+ Step 44140/44300 | Loss: 2.9047 | LR: 5.00e-05 | 1564.1ms/step | 12.8 samples/s
185
+ Step 44150/44300 | Loss: 2.9071 | LR: 5.00e-05 | 1567.2ms/step | 12.8 samples/s
186
+ Step 44160/44300 | Loss: 2.8925 | LR: 5.00e-05 | 1569.7ms/step | 12.7 samples/s
187
+ Step 44170/44300 | Loss: 2.8908 | LR: 5.00e-05 | 1569.0ms/step | 12.7 samples/s
188
+ Validation at step 44175: Loss = 2.8960, Perplexity = 18.10
189
+ Step 44180/44300 | Loss: 2.9239 | LR: 5.00e-05 | 3774.4ms/step | 5.3 samples/s
190
+ Step 44190/44300 | Loss: 2.9316 | LR: 5.00e-05 | 1568.1ms/step | 12.8 samples/s
191
+ Step 44200/44300 | Loss: 2.9276 | LR: 5.00e-05 | 1566.1ms/step | 12.8 samples/s
192
+ Step 44210/44300 | Loss: 2.9084 | LR: 5.00e-05 | 1568.6ms/step | 12.8 samples/s
193
+ Step 44220/44300 | Loss: 2.8819 | LR: 5.00e-05 | 1568.4ms/step | 12.8 samples/s
194
+ Step 44230/44300 | Loss: 2.8878 | LR: 5.00e-05 | 1571.9ms/step | 12.7 samples/s
195
+ Step 44240/44300 | Loss: 2.9094 | LR: 5.00e-05 | 1567.7ms/step | 12.8 samples/s
196
+ Step 44250/44300 | Loss: 2.9297 | LR: 5.00e-05 | 1569.1ms/step | 12.7 samples/s
197
+ Validation at step 44250: Loss = 2.8960, Perplexity = 18.10
198
+ Step 44260/44300 | Loss: 2.9066 | LR: 5.00e-05 | 3774.3ms/step | 5.3 samples/s
199
+ Step 44270/44300 | Loss: 2.8892 | LR: 5.00e-05 | 1565.7ms/step | 12.8 samples/s
200
+ Step 44280/44300 | Loss: 2.9026 | LR: 5.00e-05 | 1569.0ms/step | 12.7 samples/s
201
+ Step 44290/44300 | Loss: 2.9167 | LR: 5.00e-05 | 1566.9ms/step | 12.8 samples/s
202
+ Step 44300/44300 | Loss: 2.9220 | LR: 5.00e-05 | 1567.7ms/step | 12.8 samples/s
203
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p05/run_20260807_080949/checkpoint-44300
204
+ Training complete!
epfl_final/dropctl_logs/calib_p10.log ADDED
@@ -0,0 +1,204 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ New run: Checkpoints will be saved to /mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p10/run_20260807_080949
2
+ Running config:
3
+ {
4
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
5
+ "model_dtype": "bfloat16",
6
+ "init_from_scratch": false,
7
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
8
+ "val_path": "/mloscratch/homes/navasard/data/fineweb_val_seed42",
9
+ "max_seq_length": 8192,
10
+ "num_workers": 8,
11
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p10/run_20260807_080949",
12
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000",
13
+ "force_new_run": true,
14
+ "num_epochs": 1,
15
+ "max_steps": 44300,
16
+ "per_device_batch_size": 2,
17
+ "gradient_accumulation_steps": 10,
18
+ "learning_rate": 5e-05,
19
+ "weight_decay": 0.01,
20
+ "warmup_steps": 0,
21
+ "lr_scheduler": "constant",
22
+ "max_grad_norm": 1.0,
23
+ "save_steps": 300,
24
+ "eval_steps": 75,
25
+ "eval_batches": 50,
26
+ "logging_steps": 10,
27
+ "seed": 42,
28
+ "verbose": true,
29
+ "checkpoint_activations": false,
30
+ "strategy": "ddp",
31
+ "mixed_precision": "bf16",
32
+ "wandb_enabled": false,
33
+ "wandb_project": "stochastic_sparsity",
34
+ "wandb_name": "dropctl_calib_p10",
35
+ "wandb_tags": [
36
+ "qwen3",
37
+ "365m",
38
+ "8k",
39
+ "scaleladder",
40
+ "dense"
41
+ ],
42
+ "hidden_size": 768,
43
+ "intermediate_size": 3072,
44
+ "num_attention_heads": 6,
45
+ "num_key_value_heads": 2,
46
+ "num_hidden_layers": 28,
47
+ "head_dim": null,
48
+ "attention_dropout": 0.1,
49
+ "attention_type": "sparse_stochastic",
50
+ "sparse_block_size": 128,
51
+ "sparse_num_random_blocks": 31,
52
+ "sparse_mask_pool_size": 1,
53
+ "sparse_val_dense_compare": false,
54
+ "sparse_layer_indices": null,
55
+ "sparse_mask_cache_dir": null,
56
+ "sparse_mask_seed": 0,
57
+ "dense_mode": true,
58
+ "dense_step_intervals": null
59
+ }
60
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
61
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000
62
+ Checkpoint activations: False
63
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
64
+
65
+
66
+ ================================================================================
67
+ MODEL ARCHITECTURE
68
+ ================================================================================
69
+ Model config:
70
+ vocab_size: 151936
71
+ hidden_size: 768
72
+ num_hidden_layers: 28
73
+ num_attention_heads: 6
74
+ num_key_value_heads: 2
75
+ head_dim: 128
76
+ intermediate_size: 3072
77
+ max_position_embeddings: 32768
78
+ rope_theta: 1000000
79
+ attention_type: sparse_stochastic
80
+ sparse_block_size: 128
81
+ sparse_num_random_blocks: 31
82
+ sparse_mask_pool_size: 1
83
+ sparse_max_seq_length: 8192
84
+
85
+ Parameter count:
86
+ Total parameters: 358,958,848
87
+ Trainable parameters: 358,958,848
88
+ Non-trainable parameters: 0
89
+ Model size (BF16): 0.72 GB
90
+
91
+ --------------------------------------------------------------------------------
92
+ LAYER-BY-LAYER SUMMARY
93
+ --------------------------------------------------------------------------------
94
+ ================================================================================================================================================================
95
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
96
+ ================================================================================================================================================================
97
+ Qwen3ForCausalLM [2, 8192] -- -- True
98
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
99
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
100
+ │ └─ModuleList: 2-2 -- -- -- True
101
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
102
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
103
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
104
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
105
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
106
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
107
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
108
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
109
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
110
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
111
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
130
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
131
+ ================================================================================================================================================================
132
+ Total params: 475,645,696
133
+ Trainable params: 475,645,696
134
+ Non-trainable params: 0
135
+ Total mult-adds (Units.MEGABYTES): 951.29
136
+ ================================================================================================================================================================
137
+ Input size (MB): 0.13
138
+ Forward/backward pass size (MB): 31197.23
139
+ Params size (MB): 951.29
140
+ Estimated Total Size (MB): 32148.66
141
+ ================================================================================================================================================================
142
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
143
+ ================================================================================
144
+
145
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
146
+ Using pre-tokenized dataset (no runtime tokenization)
147
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
148
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
149
+ Dataset split: 3027764 train, 159356 validation
150
+ Training examples (packed, train split): 3027764
151
+ Validation examples (packed, val split): 159356
152
+ Per-device batch size: 2
153
+ World size (GPUs): 1
154
+ Gradient accumulation steps: 10
155
+ Effective batch size (seqs/step): 20
156
+ Sequence length: 8192
157
+ Tokens per optimizer step: 163,840
158
+ Micro-batches per epoch (per rank): 1513882
159
+ Optimizer steps per epoch: 151388
160
+ Total optimizer steps: 44300 (capped by max_steps; 0.29 epochs)
161
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000...
162
+ Overriding checkpoint LR with config LR: 5e-05
163
+ Resumed from checkpoint at step 44000 (epoch 0, micro_step 440000)
164
+ Resuming training from step 44000
165
+ Starting training...
166
+ Validation at step 44000: Loss = 2.9127, Perplexity = 18.41
167
+ Switching attention mode for step 44001: dense (dense step interval)
168
+ Step 44010/44300 | Loss: 2.9557 | LR: 5.00e-05 | 64949.7ms/step | 0.3 samples/s
169
+ Step 44020/44300 | Loss: 2.9815 | LR: 5.00e-05 | 1547.3ms/step | 12.9 samples/s
170
+ Validation at step 44025: Loss = 2.9003, Perplexity = 18.18
171
+ Step 44030/44300 | Loss: 2.9203 | LR: 5.00e-05 | 3742.6ms/step | 5.3 samples/s
172
+ Step 44040/44300 | Loss: 2.9581 | LR: 5.00e-05 | 1558.5ms/step | 12.8 samples/s
173
+ Step 44050/44300 | Loss: 2.9420 | LR: 5.00e-05 | 1559.7ms/step | 12.8 samples/s
174
+ Step 44060/44300 | Loss: 2.9706 | LR: 5.00e-05 | 1563.8ms/step | 12.8 samples/s
175
+ Step 44070/44300 | Loss: 2.9275 | LR: 5.00e-05 | 1563.1ms/step | 12.8 samples/s
176
+ Step 44080/44300 | Loss: 2.9352 | LR: 5.00e-05 | 1565.1ms/step | 12.8 samples/s
177
+ Step 44090/44300 | Loss: 2.9622 | LR: 5.00e-05 | 1565.1ms/step | 12.8 samples/s
178
+ Step 44100/44300 | Loss: 2.9599 | LR: 5.00e-05 | 1564.6ms/step | 12.8 samples/s
179
+ Validation at step 44100: Loss = 2.8988, Perplexity = 18.15
180
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p10/run_20260807_080949/checkpoint-44100
181
+ Step 44110/44300 | Loss: 2.9389 | LR: 5.00e-05 | 4044.2ms/step | 4.9 samples/s
182
+ Step 44120/44300 | Loss: 2.9274 | LR: 5.00e-05 | 1562.7ms/step | 12.8 samples/s
183
+ Step 44130/44300 | Loss: 2.9415 | LR: 5.00e-05 | 1566.2ms/step | 12.8 samples/s
184
+ Step 44140/44300 | Loss: 2.9329 | LR: 5.00e-05 | 1567.1ms/step | 12.8 samples/s
185
+ Step 44150/44300 | Loss: 2.9354 | LR: 5.00e-05 | 1565.5ms/step | 12.8 samples/s
186
+ Step 44160/44300 | Loss: 2.9205 | LR: 5.00e-05 | 1567.1ms/step | 12.8 samples/s
187
+ Step 44170/44300 | Loss: 2.9185 | LR: 5.00e-05 | 1562.8ms/step | 12.8 samples/s
188
+ Validation at step 44175: Loss = 2.8989, Perplexity = 18.15
189
+ Step 44180/44300 | Loss: 2.9515 | LR: 5.00e-05 | 3767.7ms/step | 5.3 samples/s
190
+ Step 44190/44300 | Loss: 2.9594 | LR: 5.00e-05 | 1563.7ms/step | 12.8 samples/s
191
+ Step 44200/44300 | Loss: 2.9551 | LR: 5.00e-05 | 1567.5ms/step | 12.8 samples/s
192
+ Step 44210/44300 | Loss: 2.9356 | LR: 5.00e-05 | 1564.7ms/step | 12.8 samples/s
193
+ Step 44220/44300 | Loss: 2.9089 | LR: 5.00e-05 | 1567.3ms/step | 12.8 samples/s
194
+ Step 44230/44300 | Loss: 2.9148 | LR: 5.00e-05 | 1568.0ms/step | 12.8 samples/s
195
+ Step 44240/44300 | Loss: 2.9359 | LR: 5.00e-05 | 1564.2ms/step | 12.8 samples/s
196
+ Step 44250/44300 | Loss: 2.9564 | LR: 5.00e-05 | 1565.3ms/step | 12.8 samples/s
197
+ Validation at step 44250: Loss = 2.8989, Perplexity = 18.15
198
+ Step 44260/44300 | Loss: 2.9324 | LR: 5.00e-05 | 3774.1ms/step | 5.3 samples/s
199
+ Step 44270/44300 | Loss: 2.9159 | LR: 5.00e-05 | 1567.7ms/step | 12.8 samples/s
200
+ Step 44280/44300 | Loss: 2.9287 | LR: 5.00e-05 | 1567.3ms/step | 12.8 samples/s
201
+ Step 44290/44300 | Loss: 2.9432 | LR: 5.00e-05 | 1562.7ms/step | 12.8 samples/s
202
+ Step 44300/44300 | Loss: 2.9485 | LR: 5.00e-05 | 1561.4ms/step | 12.8 samples/s
203
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p10/run_20260807_080949/checkpoint-44300
204
+ Training complete!
epfl_final/dropctl_logs/calib_p20.log ADDED
@@ -0,0 +1,204 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ New run: Checkpoints will be saved to /mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p20/run_20260807_080949
2
+ Running config:
3
+ {
4
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
5
+ "model_dtype": "bfloat16",
6
+ "init_from_scratch": false,
7
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
8
+ "val_path": "/mloscratch/homes/navasard/data/fineweb_val_seed42",
9
+ "max_seq_length": 8192,
10
+ "num_workers": 8,
11
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p20/run_20260807_080949",
12
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000",
13
+ "force_new_run": true,
14
+ "num_epochs": 1,
15
+ "max_steps": 44300,
16
+ "per_device_batch_size": 2,
17
+ "gradient_accumulation_steps": 10,
18
+ "learning_rate": 5e-05,
19
+ "weight_decay": 0.01,
20
+ "warmup_steps": 0,
21
+ "lr_scheduler": "constant",
22
+ "max_grad_norm": 1.0,
23
+ "save_steps": 300,
24
+ "eval_steps": 75,
25
+ "eval_batches": 50,
26
+ "logging_steps": 10,
27
+ "seed": 42,
28
+ "verbose": true,
29
+ "checkpoint_activations": false,
30
+ "strategy": "ddp",
31
+ "mixed_precision": "bf16",
32
+ "wandb_enabled": false,
33
+ "wandb_project": "stochastic_sparsity",
34
+ "wandb_name": "dropctl_calib_p20",
35
+ "wandb_tags": [
36
+ "qwen3",
37
+ "365m",
38
+ "8k",
39
+ "scaleladder",
40
+ "dense"
41
+ ],
42
+ "hidden_size": 768,
43
+ "intermediate_size": 3072,
44
+ "num_attention_heads": 6,
45
+ "num_key_value_heads": 2,
46
+ "num_hidden_layers": 28,
47
+ "head_dim": null,
48
+ "attention_dropout": 0.2,
49
+ "attention_type": "sparse_stochastic",
50
+ "sparse_block_size": 128,
51
+ "sparse_num_random_blocks": 31,
52
+ "sparse_mask_pool_size": 1,
53
+ "sparse_val_dense_compare": false,
54
+ "sparse_layer_indices": null,
55
+ "sparse_mask_cache_dir": null,
56
+ "sparse_mask_seed": 0,
57
+ "dense_mode": true,
58
+ "dense_step_intervals": null
59
+ }
60
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
61
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000
62
+ Checkpoint activations: False
63
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
64
+
65
+
66
+ ================================================================================
67
+ MODEL ARCHITECTURE
68
+ ================================================================================
69
+ Model config:
70
+ vocab_size: 151936
71
+ hidden_size: 768
72
+ num_hidden_layers: 28
73
+ num_attention_heads: 6
74
+ num_key_value_heads: 2
75
+ head_dim: 128
76
+ intermediate_size: 3072
77
+ max_position_embeddings: 32768
78
+ rope_theta: 1000000
79
+ attention_type: sparse_stochastic
80
+ sparse_block_size: 128
81
+ sparse_num_random_blocks: 31
82
+ sparse_mask_pool_size: 1
83
+ sparse_max_seq_length: 8192
84
+
85
+ Parameter count:
86
+ Total parameters: 358,958,848
87
+ Trainable parameters: 358,958,848
88
+ Non-trainable parameters: 0
89
+ Model size (BF16): 0.72 GB
90
+
91
+ --------------------------------------------------------------------------------
92
+ LAYER-BY-LAYER SUMMARY
93
+ --------------------------------------------------------------------------------
94
+ ================================================================================================================================================================
95
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
96
+ ================================================================================================================================================================
97
+ Qwen3ForCausalLM [2, 8192] -- -- True
98
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
99
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
100
+ │ └─ModuleList: 2-2 -- -- -- True
101
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
102
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
103
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
104
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
105
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
106
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
107
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
108
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
109
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
110
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
111
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
130
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
131
+ ================================================================================================================================================================
132
+ Total params: 475,645,696
133
+ Trainable params: 475,645,696
134
+ Non-trainable params: 0
135
+ Total mult-adds (Units.MEGABYTES): 951.29
136
+ ================================================================================================================================================================
137
+ Input size (MB): 0.13
138
+ Forward/backward pass size (MB): 31197.23
139
+ Params size (MB): 951.29
140
+ Estimated Total Size (MB): 32148.66
141
+ ================================================================================================================================================================
142
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
143
+ ================================================================================
144
+
145
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
146
+ Using pre-tokenized dataset (no runtime tokenization)
147
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
148
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
149
+ Dataset split: 3027764 train, 159356 validation
150
+ Training examples (packed, train split): 3027764
151
+ Validation examples (packed, val split): 159356
152
+ Per-device batch size: 2
153
+ World size (GPUs): 1
154
+ Gradient accumulation steps: 10
155
+ Effective batch size (seqs/step): 20
156
+ Sequence length: 8192
157
+ Tokens per optimizer step: 163,840
158
+ Micro-batches per epoch (per rank): 1513882
159
+ Optimizer steps per epoch: 151388
160
+ Total optimizer steps: 44300 (capped by max_steps; 0.29 epochs)
161
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000...
162
+ Overriding checkpoint LR with config LR: 5e-05
163
+ Resumed from checkpoint at step 44000 (epoch 0, micro_step 440000)
164
+ Resuming training from step 44000
165
+ Starting training...
166
+ Validation at step 44000: Loss = 2.9127, Perplexity = 18.41
167
+ Switching attention mode for step 44001: dense (dense step interval)
168
+ Step 44010/44300 | Loss: 3.0486 | LR: 5.00e-05 | 61304.9ms/step | 0.3 samples/s
169
+ Step 44020/44300 | Loss: 3.0618 | LR: 5.00e-05 | 1551.6ms/step | 12.9 samples/s
170
+ Validation at step 44025: Loss = 2.9085, Perplexity = 18.33
171
+ Step 44030/44300 | Loss: 2.9961 | LR: 5.00e-05 | 3753.2ms/step | 5.3 samples/s
172
+ Step 44040/44300 | Loss: 3.0311 | LR: 5.00e-05 | 1557.1ms/step | 12.8 samples/s
173
+ Step 44050/44300 | Loss: 3.0129 | LR: 5.00e-05 | 1553.6ms/step | 12.9 samples/s
174
+ Step 44060/44300 | Loss: 3.0402 | LR: 5.00e-05 | 1556.8ms/step | 12.8 samples/s
175
+ Step 44070/44300 | Loss: 2.9953 | LR: 5.00e-05 | 1553.7ms/step | 12.9 samples/s
176
+ Step 44080/44300 | Loss: 3.0027 | LR: 5.00e-05 | 1555.3ms/step | 12.9 samples/s
177
+ Step 44090/44300 | Loss: 3.0302 | LR: 5.00e-05 | 1557.7ms/step | 12.8 samples/s
178
+ Step 44100/44300 | Loss: 3.0252 | LR: 5.00e-05 | 1555.8ms/step | 12.9 samples/s
179
+ Validation at step 44100: Loss = 2.9070, Perplexity = 18.30
180
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p20/run_20260807_080949/checkpoint-44100
181
+ Step 44110/44300 | Loss: 3.0033 | LR: 5.00e-05 | 4055.6ms/step | 4.9 samples/s
182
+ Step 44120/44300 | Loss: 2.9918 | LR: 5.00e-05 | 1555.2ms/step | 12.9 samples/s
183
+ Step 44130/44300 | Loss: 3.0055 | LR: 5.00e-05 | 1557.9ms/step | 12.8 samples/s
184
+ Step 44140/44300 | Loss: 2.9971 | LR: 5.00e-05 | 1555.8ms/step | 12.9 samples/s
185
+ Step 44150/44300 | Loss: 2.9992 | LR: 5.00e-05 | 1555.7ms/step | 12.9 samples/s
186
+ Step 44160/44300 | Loss: 2.9835 | LR: 5.00e-05 | 1556.7ms/step | 12.8 samples/s
187
+ Step 44170/44300 | Loss: 2.9813 | LR: 5.00e-05 | 1555.7ms/step | 12.9 samples/s
188
+ Validation at step 44175: Loss = 2.9076, Perplexity = 18.31
189
+ Step 44180/44300 | Loss: 3.0140 | LR: 5.00e-05 | 3742.1ms/step | 5.3 samples/s
190
+ Step 44190/44300 | Loss: 3.0215 | LR: 5.00e-05 | 1559.1ms/step | 12.8 samples/s
191
+ Step 44200/44300 | Loss: 3.0155 | LR: 5.00e-05 | 1556.6ms/step | 12.8 samples/s
192
+ Step 44210/44300 | Loss: 2.9962 | LR: 5.00e-05 | 1559.2ms/step | 12.8 samples/s
193
+ Step 44220/44300 | Loss: 2.9691 | LR: 5.00e-05 | 1555.5ms/step | 12.9 samples/s
194
+ Step 44230/44300 | Loss: 2.9751 | LR: 5.00e-05 | 1557.5ms/step | 12.8 samples/s
195
+ Step 44240/44300 | Loss: 2.9956 | LR: 5.00e-05 | 1558.4ms/step | 12.8 samples/s
196
+ Step 44250/44300 | Loss: 3.0149 | LR: 5.00e-05 | 1556.8ms/step | 12.8 samples/s
197
+ Validation at step 44250: Loss = 2.9077, Perplexity = 18.31
198
+ Step 44260/44300 | Loss: 2.9913 | LR: 5.00e-05 | 3742.6ms/step | 5.3 samples/s
199
+ Step 44270/44300 | Loss: 2.9756 | LR: 5.00e-05 | 1554.8ms/step | 12.9 samples/s
200
+ Step 44280/44300 | Loss: 2.9867 | LR: 5.00e-05 | 1557.6ms/step | 12.8 samples/s
201
+ Step 44290/44300 | Loss: 3.0018 | LR: 5.00e-05 | 1558.7ms/step | 12.8 samples/s
202
+ Step 44300/44300 | Loss: 3.0063 | LR: 5.00e-05 | 1556.8ms/step | 12.8 samples/s
203
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p20/run_20260807_080949/checkpoint-44300
204
+ Training complete!
epfl_final/dropctl_logs/calib_p28_verify.log ADDED
@@ -0,0 +1,168 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ nohup: ignoring input
2
+ New run: Checkpoints will be saved to /mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p28_verify/run_20260819_034938
3
+ Running config:
4
+ {
5
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
6
+ "model_dtype": "bfloat16",
7
+ "init_from_scratch": false,
8
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
9
+ "val_path": "/mloscratch/homes/navasard/data/fineweb_val_seed42",
10
+ "max_seq_length": 8192,
11
+ "num_workers": 8,
12
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p28_verify/run_20260819_034938",
13
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000",
14
+ "force_new_run": true,
15
+ "num_epochs": 1,
16
+ "max_steps": 44300,
17
+ "per_device_batch_size": 2,
18
+ "gradient_accumulation_steps": 10,
19
+ "learning_rate": 5e-05,
20
+ "weight_decay": 0.01,
21
+ "warmup_steps": 0,
22
+ "lr_scheduler": "constant",
23
+ "max_grad_norm": 1.0,
24
+ "save_steps": 300,
25
+ "eval_steps": 75,
26
+ "eval_batches": 50,
27
+ "logging_steps": 10,
28
+ "seed": 42,
29
+ "verbose": true,
30
+ "checkpoint_activations": false,
31
+ "strategy": "ddp",
32
+ "mixed_precision": "bf16",
33
+ "wandb_enabled": false,
34
+ "wandb_project": "stochastic_sparsity",
35
+ "wandb_name": "dropctl_calib_p28_verify",
36
+ "wandb_tags": [
37
+ "qwen3",
38
+ "365m",
39
+ "8k",
40
+ "scaleladder",
41
+ "dense"
42
+ ],
43
+ "hidden_size": 768,
44
+ "intermediate_size": 3072,
45
+ "num_attention_heads": 6,
46
+ "num_key_value_heads": 2,
47
+ "num_hidden_layers": 28,
48
+ "head_dim": null,
49
+ "attention_dropout": 0.28,
50
+ "attention_type": "sparse_stochastic",
51
+ "sparse_block_size": 128,
52
+ "sparse_num_random_blocks": 31,
53
+ "sparse_mask_pool_size": 1,
54
+ "sparse_val_dense_compare": false,
55
+ "sparse_layer_indices": null,
56
+ "sparse_mask_cache_dir": null,
57
+ "sparse_mask_seed": 0,
58
+ "sparse_mixed_nrb_set": null,
59
+ "dense_mode": true,
60
+ "dense_step_intervals": null
61
+ }
62
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
63
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000
64
+ Checkpoint activations: False
65
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
66
+
67
+
68
+ ================================================================================
69
+ MODEL ARCHITECTURE
70
+ ================================================================================
71
+ Model config:
72
+ vocab_size: 151936
73
+ hidden_size: 768
74
+ num_hidden_layers: 28
75
+ num_attention_heads: 6
76
+ num_key_value_heads: 2
77
+ head_dim: 128
78
+ intermediate_size: 3072
79
+ max_position_embeddings: 32768
80
+ rope_theta: 1000000
81
+ attention_type: sparse_stochastic
82
+ sparse_block_size: 128
83
+ sparse_num_random_blocks: 31
84
+ sparse_mask_pool_size: 1
85
+ sparse_max_seq_length: 8192
86
+
87
+ Parameter count:
88
+ Total parameters: 358,958,848
89
+ Trainable parameters: 358,958,848
90
+ Non-trainable parameters: 0
91
+ Model size (BF16): 0.72 GB
92
+
93
+ --------------------------------------------------------------------------------
94
+ LAYER-BY-LAYER SUMMARY
95
+ --------------------------------------------------------------------------------
96
+ ================================================================================================================================================================
97
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
98
+ ================================================================================================================================================================
99
+ Qwen3ForCausalLM [2, 8192] -- -- True
100
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
101
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
102
+ │ └─ModuleList: 2-2 -- -- -- True
103
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
104
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
105
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
106
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
107
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
108
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
109
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
110
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
111
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
132
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
133
+ ================================================================================================================================================================
134
+ Total params: 475,645,696
135
+ Trainable params: 475,645,696
136
+ Non-trainable params: 0
137
+ Total mult-adds (Units.MEGABYTES): 951.29
138
+ ================================================================================================================================================================
139
+ Input size (MB): 0.13
140
+ Forward/backward pass size (MB): 31197.23
141
+ Params size (MB): 951.29
142
+ Estimated Total Size (MB): 32148.66
143
+ ================================================================================================================================================================
144
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
145
+ ================================================================================
146
+
147
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
148
+ Using pre-tokenized dataset (no runtime tokenization)
149
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
150
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
151
+ Dataset split: 3027764 train, 159356 validation
152
+ Training examples (packed, train split): 3027764
153
+ Validation examples (packed, val split): 159356
154
+ Per-device batch size: 2
155
+ World size (GPUs): 1
156
+ Gradient accumulation steps: 10
157
+ Effective batch size (seqs/step): 20
158
+ Sequence length: 8192
159
+ Tokens per optimizer step: 163,840
160
+ Micro-batches per epoch (per rank): 1513882
161
+ Optimizer steps per epoch: 151388
162
+ Total optimizer steps: 44300 (capped by max_steps; 0.29 epochs)
163
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000...
164
+ Overriding checkpoint LR with config LR: 5e-05
165
+ Resumed from checkpoint at step 44000 (epoch 0, micro_step 440000)
166
+ Resuming training from step 44000
167
+ Starting training...
168
+ Validation at step 44000: Loss = 2.9127, Perplexity = 18.41
epfl_final/dropctl_logs/calib_p28_verify_r2.log ADDED
@@ -0,0 +1,206 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ nohup: ignoring input
2
+ New run: Checkpoints will be saved to /mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p28_verify/run_20260819_043051
3
+ Running config:
4
+ {
5
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
6
+ "model_dtype": "bfloat16",
7
+ "init_from_scratch": false,
8
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
9
+ "val_path": "/mloscratch/homes/navasard/data/fineweb_val_seed42",
10
+ "max_seq_length": 8192,
11
+ "num_workers": 8,
12
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p28_verify/run_20260819_043051",
13
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000",
14
+ "force_new_run": true,
15
+ "num_epochs": 1,
16
+ "max_steps": 44300,
17
+ "per_device_batch_size": 2,
18
+ "gradient_accumulation_steps": 10,
19
+ "learning_rate": 5e-05,
20
+ "weight_decay": 0.01,
21
+ "warmup_steps": 0,
22
+ "lr_scheduler": "constant",
23
+ "max_grad_norm": 1.0,
24
+ "save_steps": 300,
25
+ "eval_steps": 75,
26
+ "eval_batches": 50,
27
+ "logging_steps": 10,
28
+ "seed": 42,
29
+ "verbose": true,
30
+ "checkpoint_activations": false,
31
+ "strategy": "ddp",
32
+ "mixed_precision": "bf16",
33
+ "wandb_enabled": false,
34
+ "wandb_project": "stochastic_sparsity",
35
+ "wandb_name": "dropctl_calib_p28_verify",
36
+ "wandb_tags": [
37
+ "qwen3",
38
+ "365m",
39
+ "8k",
40
+ "scaleladder",
41
+ "dense"
42
+ ],
43
+ "hidden_size": 768,
44
+ "intermediate_size": 3072,
45
+ "num_attention_heads": 6,
46
+ "num_key_value_heads": 2,
47
+ "num_hidden_layers": 28,
48
+ "head_dim": null,
49
+ "attention_dropout": 0.28,
50
+ "attention_type": "sparse_stochastic",
51
+ "sparse_block_size": 128,
52
+ "sparse_num_random_blocks": 31,
53
+ "sparse_mask_pool_size": 1,
54
+ "sparse_val_dense_compare": false,
55
+ "sparse_layer_indices": null,
56
+ "sparse_mask_cache_dir": null,
57
+ "sparse_mask_seed": 0,
58
+ "sparse_mixed_nrb_set": null,
59
+ "dense_mode": true,
60
+ "dense_step_intervals": null
61
+ }
62
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
63
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000
64
+ Checkpoint activations: False
65
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
66
+
67
+
68
+ ================================================================================
69
+ MODEL ARCHITECTURE
70
+ ================================================================================
71
+ Model config:
72
+ vocab_size: 151936
73
+ hidden_size: 768
74
+ num_hidden_layers: 28
75
+ num_attention_heads: 6
76
+ num_key_value_heads: 2
77
+ head_dim: 128
78
+ intermediate_size: 3072
79
+ max_position_embeddings: 32768
80
+ rope_theta: 1000000
81
+ attention_type: sparse_stochastic
82
+ sparse_block_size: 128
83
+ sparse_num_random_blocks: 31
84
+ sparse_mask_pool_size: 1
85
+ sparse_max_seq_length: 8192
86
+
87
+ Parameter count:
88
+ Total parameters: 358,958,848
89
+ Trainable parameters: 358,958,848
90
+ Non-trainable parameters: 0
91
+ Model size (BF16): 0.72 GB
92
+
93
+ --------------------------------------------------------------------------------
94
+ LAYER-BY-LAYER SUMMARY
95
+ --------------------------------------------------------------------------------
96
+ ================================================================================================================================================================
97
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
98
+ ================================================================================================================================================================
99
+ Qwen3ForCausalLM [2, 8192] -- -- True
100
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
101
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
102
+ │ └─ModuleList: 2-2 -- -- -- True
103
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
104
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
105
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
106
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
107
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
108
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
109
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
110
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
111
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
132
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
133
+ ================================================================================================================================================================
134
+ Total params: 475,645,696
135
+ Trainable params: 475,645,696
136
+ Non-trainable params: 0
137
+ Total mult-adds (Units.MEGABYTES): 951.29
138
+ ================================================================================================================================================================
139
+ Input size (MB): 0.13
140
+ Forward/backward pass size (MB): 31197.23
141
+ Params size (MB): 951.29
142
+ Estimated Total Size (MB): 32148.66
143
+ ================================================================================================================================================================
144
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
145
+ ================================================================================
146
+
147
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
148
+ Using pre-tokenized dataset (no runtime tokenization)
149
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
150
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
151
+ Dataset split: 3027764 train, 159356 validation
152
+ Training examples (packed, train split): 3027764
153
+ Validation examples (packed, val split): 159356
154
+ Per-device batch size: 2
155
+ World size (GPUs): 1
156
+ Gradient accumulation steps: 10
157
+ Effective batch size (seqs/step): 20
158
+ Sequence length: 8192
159
+ Tokens per optimizer step: 163,840
160
+ Micro-batches per epoch (per rank): 1513882
161
+ Optimizer steps per epoch: 151388
162
+ Total optimizer steps: 44300 (capped by max_steps; 0.29 epochs)
163
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000...
164
+ Overriding checkpoint LR with config LR: 5e-05
165
+ Resumed from checkpoint at step 44000 (epoch 0, micro_step 440000)
166
+ Resuming training from step 44000
167
+ Starting training...
168
+ Validation at step 44000: Loss = 2.9127, Perplexity = 18.41
169
+ Switching attention mode for step 44001: dense (dense step interval)
170
+ Step 44010/44300 | Loss: 3.1491 | LR: 5.00e-05 | 55165.6ms/step | 0.4 samples/s
171
+ Step 44020/44300 | Loss: 3.1438 | LR: 5.00e-05 | 1530.2ms/step | 13.1 samples/s
172
+ Validation at step 44025: Loss = 2.9172, Perplexity = 18.49
173
+ Step 44030/44300 | Loss: 3.0722 | LR: 5.00e-05 | 3695.0ms/step | 5.4 samples/s
174
+ Step 44040/44300 | Loss: 3.1044 | LR: 5.00e-05 | 1540.3ms/step | 13.0 samples/s
175
+ Step 44050/44300 | Loss: 3.0836 | LR: 5.00e-05 | 1544.1ms/step | 13.0 samples/s
176
+ Step 44060/44300 | Loss: 3.1085 | LR: 5.00e-05 | 1547.5ms/step | 12.9 samples/s
177
+ Step 44070/44300 | Loss: 3.0621 | LR: 5.00e-05 | 1551.1ms/step | 12.9 samples/s
178
+ Step 44080/44300 | Loss: 3.0690 | LR: 5.00e-05 | 1551.2ms/step | 12.9 samples/s
179
+ Step 44090/44300 | Loss: 3.0966 | LR: 5.00e-05 | 1552.2ms/step | 12.9 samples/s
180
+ Step 44100/44300 | Loss: 3.0894 | LR: 5.00e-05 | 1552.2ms/step | 12.9 samples/s
181
+ Validation at step 44100: Loss = 2.9158, Perplexity = 18.46
182
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p28_verify/run_20260819_043051/checkpoint-44100
183
+ Step 44110/44300 | Loss: 3.0661 | LR: 5.00e-05 | 4087.1ms/step | 4.9 samples/s
184
+ Step 44120/44300 | Loss: 3.0542 | LR: 5.00e-05 | 1543.5ms/step | 13.0 samples/s
185
+ Step 44130/44300 | Loss: 3.0669 | LR: 5.00e-05 | 1551.1ms/step | 12.9 samples/s
186
+ Step 44140/44300 | Loss: 3.0582 | LR: 5.00e-05 | 1553.8ms/step | 12.9 samples/s
187
+ Step 44150/44300 | Loss: 3.0598 | LR: 5.00e-05 | 1554.4ms/step | 12.9 samples/s
188
+ Step 44160/44300 | Loss: 3.0433 | LR: 5.00e-05 | 1555.1ms/step | 12.9 samples/s
189
+ Step 44170/44300 | Loss: 3.0410 | LR: 5.00e-05 | 1555.9ms/step | 12.9 samples/s
190
+ Validation at step 44175: Loss = 2.9171, Perplexity = 18.49
191
+ Step 44180/44300 | Loss: 3.0727 | LR: 5.00e-05 | 3745.2ms/step | 5.3 samples/s
192
+ Step 44190/44300 | Loss: 3.0802 | LR: 5.00e-05 | 1556.9ms/step | 12.8 samples/s
193
+ Step 44200/44300 | Loss: 3.0718 | LR: 5.00e-05 | 1559.2ms/step | 12.8 samples/s
194
+ Step 44210/44300 | Loss: 3.0533 | LR: 5.00e-05 | 1559.4ms/step | 12.8 samples/s
195
+ Step 44220/44300 | Loss: 3.0267 | LR: 5.00e-05 | 1555.5ms/step | 12.9 samples/s
196
+ Step 44230/44300 | Loss: 3.0324 | LR: 5.00e-05 | 1560.0ms/step | 12.8 samples/s
197
+ Step 44240/44300 | Loss: 3.0521 | LR: 5.00e-05 | 1557.1ms/step | 12.8 samples/s
198
+ Step 44250/44300 | Loss: 3.0699 | LR: 5.00e-05 | 1557.4ms/step | 12.8 samples/s
199
+ Validation at step 44250: Loss = 2.9175, Perplexity = 18.50
200
+ Step 44260/44300 | Loss: 3.0461 | LR: 5.00e-05 | 3753.8ms/step | 5.3 samples/s
201
+ Step 44270/44300 | Loss: 3.0314 | LR: 5.00e-05 | 1560.3ms/step | 12.8 samples/s
202
+ Step 44280/44300 | Loss: 3.0416 | LR: 5.00e-05 | 1559.0ms/step | 12.8 samples/s
203
+ Step 44290/44300 | Loss: 3.0569 | LR: 5.00e-05 | 1560.3ms/step | 12.8 samples/s
204
+ Step 44300/44300 | Loss: 3.0606 | LR: 5.00e-05 | 1560.2ms/step | 12.8 samples/s
205
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p28_verify/run_20260819_043051/checkpoint-44300
206
+ Training complete!
epfl_final/dropctl_logs/calib_p30_verify.log ADDED
@@ -0,0 +1,168 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ nohup: ignoring input
2
+ New run: Checkpoints will be saved to /mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p30_verify/run_20260819_034938
3
+ Running config:
4
+ {
5
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
6
+ "model_dtype": "bfloat16",
7
+ "init_from_scratch": false,
8
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
9
+ "val_path": "/mloscratch/homes/navasard/data/fineweb_val_seed42",
10
+ "max_seq_length": 8192,
11
+ "num_workers": 8,
12
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p30_verify/run_20260819_034938",
13
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000",
14
+ "force_new_run": true,
15
+ "num_epochs": 1,
16
+ "max_steps": 44300,
17
+ "per_device_batch_size": 2,
18
+ "gradient_accumulation_steps": 10,
19
+ "learning_rate": 5e-05,
20
+ "weight_decay": 0.01,
21
+ "warmup_steps": 0,
22
+ "lr_scheduler": "constant",
23
+ "max_grad_norm": 1.0,
24
+ "save_steps": 300,
25
+ "eval_steps": 75,
26
+ "eval_batches": 50,
27
+ "logging_steps": 10,
28
+ "seed": 42,
29
+ "verbose": true,
30
+ "checkpoint_activations": false,
31
+ "strategy": "ddp",
32
+ "mixed_precision": "bf16",
33
+ "wandb_enabled": false,
34
+ "wandb_project": "stochastic_sparsity",
35
+ "wandb_name": "dropctl_calib_p30_verify",
36
+ "wandb_tags": [
37
+ "qwen3",
38
+ "365m",
39
+ "8k",
40
+ "scaleladder",
41
+ "dense"
42
+ ],
43
+ "hidden_size": 768,
44
+ "intermediate_size": 3072,
45
+ "num_attention_heads": 6,
46
+ "num_key_value_heads": 2,
47
+ "num_hidden_layers": 28,
48
+ "head_dim": null,
49
+ "attention_dropout": 0.3,
50
+ "attention_type": "sparse_stochastic",
51
+ "sparse_block_size": 128,
52
+ "sparse_num_random_blocks": 31,
53
+ "sparse_mask_pool_size": 1,
54
+ "sparse_val_dense_compare": false,
55
+ "sparse_layer_indices": null,
56
+ "sparse_mask_cache_dir": null,
57
+ "sparse_mask_seed": 0,
58
+ "sparse_mixed_nrb_set": null,
59
+ "dense_mode": true,
60
+ "dense_step_intervals": null
61
+ }
62
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
63
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000
64
+ Checkpoint activations: False
65
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
66
+
67
+
68
+ ================================================================================
69
+ MODEL ARCHITECTURE
70
+ ================================================================================
71
+ Model config:
72
+ vocab_size: 151936
73
+ hidden_size: 768
74
+ num_hidden_layers: 28
75
+ num_attention_heads: 6
76
+ num_key_value_heads: 2
77
+ head_dim: 128
78
+ intermediate_size: 3072
79
+ max_position_embeddings: 32768
80
+ rope_theta: 1000000
81
+ attention_type: sparse_stochastic
82
+ sparse_block_size: 128
83
+ sparse_num_random_blocks: 31
84
+ sparse_mask_pool_size: 1
85
+ sparse_max_seq_length: 8192
86
+
87
+ Parameter count:
88
+ Total parameters: 358,958,848
89
+ Trainable parameters: 358,958,848
90
+ Non-trainable parameters: 0
91
+ Model size (BF16): 0.72 GB
92
+
93
+ --------------------------------------------------------------------------------
94
+ LAYER-BY-LAYER SUMMARY
95
+ --------------------------------------------------------------------------------
96
+ ================================================================================================================================================================
97
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
98
+ ================================================================================================================================================================
99
+ Qwen3ForCausalLM [2, 8192] -- -- True
100
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
101
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
102
+ │ └─ModuleList: 2-2 -- -- -- True
103
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
104
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
105
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
106
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
107
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
108
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
109
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
110
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
111
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
132
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
133
+ ================================================================================================================================================================
134
+ Total params: 475,645,696
135
+ Trainable params: 475,645,696
136
+ Non-trainable params: 0
137
+ Total mult-adds (Units.MEGABYTES): 951.29
138
+ ================================================================================================================================================================
139
+ Input size (MB): 0.13
140
+ Forward/backward pass size (MB): 31197.23
141
+ Params size (MB): 951.29
142
+ Estimated Total Size (MB): 32148.66
143
+ ================================================================================================================================================================
144
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
145
+ ================================================================================
146
+
147
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
148
+ Using pre-tokenized dataset (no runtime tokenization)
149
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
150
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
151
+ Dataset split: 3027764 train, 159356 validation
152
+ Training examples (packed, train split): 3027764
153
+ Validation examples (packed, val split): 159356
154
+ Per-device batch size: 2
155
+ World size (GPUs): 1
156
+ Gradient accumulation steps: 10
157
+ Effective batch size (seqs/step): 20
158
+ Sequence length: 8192
159
+ Tokens per optimizer step: 163,840
160
+ Micro-batches per epoch (per rank): 1513882
161
+ Optimizer steps per epoch: 151388
162
+ Total optimizer steps: 44300 (capped by max_steps; 0.29 epochs)
163
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000...
164
+ Overriding checkpoint LR with config LR: 5e-05
165
+ Resumed from checkpoint at step 44000 (epoch 0, micro_step 440000)
166
+ Resuming training from step 44000
167
+ Starting training...
168
+ Validation at step 44000: Loss = 2.9127, Perplexity = 18.41
epfl_final/dropctl_logs/calib_p30_verify_r2.log ADDED
@@ -0,0 +1,206 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ nohup: ignoring input
2
+ New run: Checkpoints will be saved to /mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p30_verify/run_20260819_043051
3
+ Running config:
4
+ {
5
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
6
+ "model_dtype": "bfloat16",
7
+ "init_from_scratch": false,
8
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
9
+ "val_path": "/mloscratch/homes/navasard/data/fineweb_val_seed42",
10
+ "max_seq_length": 8192,
11
+ "num_workers": 8,
12
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p30_verify/run_20260819_043051",
13
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000",
14
+ "force_new_run": true,
15
+ "num_epochs": 1,
16
+ "max_steps": 44300,
17
+ "per_device_batch_size": 2,
18
+ "gradient_accumulation_steps": 10,
19
+ "learning_rate": 5e-05,
20
+ "weight_decay": 0.01,
21
+ "warmup_steps": 0,
22
+ "lr_scheduler": "constant",
23
+ "max_grad_norm": 1.0,
24
+ "save_steps": 300,
25
+ "eval_steps": 75,
26
+ "eval_batches": 50,
27
+ "logging_steps": 10,
28
+ "seed": 42,
29
+ "verbose": true,
30
+ "checkpoint_activations": false,
31
+ "strategy": "ddp",
32
+ "mixed_precision": "bf16",
33
+ "wandb_enabled": false,
34
+ "wandb_project": "stochastic_sparsity",
35
+ "wandb_name": "dropctl_calib_p30_verify",
36
+ "wandb_tags": [
37
+ "qwen3",
38
+ "365m",
39
+ "8k",
40
+ "scaleladder",
41
+ "dense"
42
+ ],
43
+ "hidden_size": 768,
44
+ "intermediate_size": 3072,
45
+ "num_attention_heads": 6,
46
+ "num_key_value_heads": 2,
47
+ "num_hidden_layers": 28,
48
+ "head_dim": null,
49
+ "attention_dropout": 0.3,
50
+ "attention_type": "sparse_stochastic",
51
+ "sparse_block_size": 128,
52
+ "sparse_num_random_blocks": 31,
53
+ "sparse_mask_pool_size": 1,
54
+ "sparse_val_dense_compare": false,
55
+ "sparse_layer_indices": null,
56
+ "sparse_mask_cache_dir": null,
57
+ "sparse_mask_seed": 0,
58
+ "sparse_mixed_nrb_set": null,
59
+ "dense_mode": true,
60
+ "dense_step_intervals": null
61
+ }
62
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
63
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000
64
+ Checkpoint activations: False
65
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
66
+
67
+
68
+ ================================================================================
69
+ MODEL ARCHITECTURE
70
+ ================================================================================
71
+ Model config:
72
+ vocab_size: 151936
73
+ hidden_size: 768
74
+ num_hidden_layers: 28
75
+ num_attention_heads: 6
76
+ num_key_value_heads: 2
77
+ head_dim: 128
78
+ intermediate_size: 3072
79
+ max_position_embeddings: 32768
80
+ rope_theta: 1000000
81
+ attention_type: sparse_stochastic
82
+ sparse_block_size: 128
83
+ sparse_num_random_blocks: 31
84
+ sparse_mask_pool_size: 1
85
+ sparse_max_seq_length: 8192
86
+
87
+ Parameter count:
88
+ Total parameters: 358,958,848
89
+ Trainable parameters: 358,958,848
90
+ Non-trainable parameters: 0
91
+ Model size (BF16): 0.72 GB
92
+
93
+ --------------------------------------------------------------------------------
94
+ LAYER-BY-LAYER SUMMARY
95
+ --------------------------------------------------------------------------------
96
+ ================================================================================================================================================================
97
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
98
+ ================================================================================================================================================================
99
+ Qwen3ForCausalLM [2, 8192] -- -- True
100
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
101
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
102
+ │ └─ModuleList: 2-2 -- -- -- True
103
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
104
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
105
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
106
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
107
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
108
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
109
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
110
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
111
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
132
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
133
+ ================================================================================================================================================================
134
+ Total params: 475,645,696
135
+ Trainable params: 475,645,696
136
+ Non-trainable params: 0
137
+ Total mult-adds (Units.MEGABYTES): 951.29
138
+ ================================================================================================================================================================
139
+ Input size (MB): 0.13
140
+ Forward/backward pass size (MB): 31197.23
141
+ Params size (MB): 951.29
142
+ Estimated Total Size (MB): 32148.66
143
+ ================================================================================================================================================================
144
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
145
+ ================================================================================
146
+
147
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
148
+ Using pre-tokenized dataset (no runtime tokenization)
149
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
150
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
151
+ Dataset split: 3027764 train, 159356 validation
152
+ Training examples (packed, train split): 3027764
153
+ Validation examples (packed, val split): 159356
154
+ Per-device batch size: 2
155
+ World size (GPUs): 1
156
+ Gradient accumulation steps: 10
157
+ Effective batch size (seqs/step): 20
158
+ Sequence length: 8192
159
+ Tokens per optimizer step: 163,840
160
+ Micro-batches per epoch (per rank): 1513882
161
+ Optimizer steps per epoch: 151388
162
+ Total optimizer steps: 44300 (capped by max_steps; 0.29 epochs)
163
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000...
164
+ Overriding checkpoint LR with config LR: 5e-05
165
+ Resumed from checkpoint at step 44000 (epoch 0, micro_step 440000)
166
+ Resuming training from step 44000
167
+ Starting training...
168
+ Validation at step 44000: Loss = 2.9127, Perplexity = 18.41
169
+ Switching attention mode for step 44001: dense (dense step interval)
170
+ Step 44010/44300 | Loss: 3.1793 | LR: 5.00e-05 | 55394.6ms/step | 0.4 samples/s
171
+ Step 44020/44300 | Loss: 3.1677 | LR: 5.00e-05 | 1540.3ms/step | 13.0 samples/s
172
+ Validation at step 44025: Loss = 2.9197, Perplexity = 18.54
173
+ Step 44030/44300 | Loss: 3.0941 | LR: 5.00e-05 | 3723.5ms/step | 5.4 samples/s
174
+ Step 44040/44300 | Loss: 3.1249 | LR: 5.00e-05 | 1548.2ms/step | 12.9 samples/s
175
+ Step 44050/44300 | Loss: 3.1031 | LR: 5.00e-05 | 1546.5ms/step | 12.9 samples/s
176
+ Step 44060/44300 | Loss: 3.1276 | LR: 5.00e-05 | 1540.9ms/step | 13.0 samples/s
177
+ Step 44070/44300 | Loss: 3.0808 | LR: 5.00e-05 | 1545.7ms/step | 12.9 samples/s
178
+ Step 44080/44300 | Loss: 3.0874 | LR: 5.00e-05 | 1546.0ms/step | 12.9 samples/s
179
+ Step 44090/44300 | Loss: 3.1147 | LR: 5.00e-05 | 1544.6ms/step | 12.9 samples/s
180
+ Step 44100/44300 | Loss: 3.1072 | LR: 5.00e-05 | 1546.0ms/step | 12.9 samples/s
181
+ Validation at step 44100: Loss = 2.9183, Perplexity = 18.51
182
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p30_verify/run_20260819_043051/checkpoint-44100
183
+ Step 44110/44300 | Loss: 3.0837 | LR: 5.00e-05 | 4070.3ms/step | 4.9 samples/s
184
+ Step 44120/44300 | Loss: 3.0716 | LR: 5.00e-05 | 1548.4ms/step | 12.9 samples/s
185
+ Step 44130/44300 | Loss: 3.0840 | LR: 5.00e-05 | 1544.5ms/step | 12.9 samples/s
186
+ Step 44140/44300 | Loss: 3.0751 | LR: 5.00e-05 | 1545.5ms/step | 12.9 samples/s
187
+ Step 44150/44300 | Loss: 3.0768 | LR: 5.00e-05 | 1546.9ms/step | 12.9 samples/s
188
+ Step 44160/44300 | Loss: 3.0596 | LR: 5.00e-05 | 1546.5ms/step | 12.9 samples/s
189
+ Step 44170/44300 | Loss: 3.0576 | LR: 5.00e-05 | 1547.0ms/step | 12.9 samples/s
190
+ Validation at step 44175: Loss = 2.9198, Perplexity = 18.54
191
+ Step 44180/44300 | Loss: 3.0889 | LR: 5.00e-05 | 3721.4ms/step | 5.4 samples/s
192
+ Step 44190/44300 | Loss: 3.0964 | LR: 5.00e-05 | 1543.5ms/step | 13.0 samples/s
193
+ Step 44200/44300 | Loss: 3.0876 | LR: 5.00e-05 | 1549.3ms/step | 12.9 samples/s
194
+ Step 44210/44300 | Loss: 3.0693 | LR: 5.00e-05 | 1547.0ms/step | 12.9 samples/s
195
+ Step 44220/44300 | Loss: 3.0425 | LR: 5.00e-05 | 1546.7ms/step | 12.9 samples/s
196
+ Step 44230/44300 | Loss: 3.0484 | LR: 5.00e-05 | 1546.1ms/step | 12.9 samples/s
197
+ Step 44240/44300 | Loss: 3.0679 | LR: 5.00e-05 | 1548.6ms/step | 12.9 samples/s
198
+ Step 44250/44300 | Loss: 3.0854 | LR: 5.00e-05 | 1546.8ms/step | 12.9 samples/s
199
+ Validation at step 44250: Loss = 2.9203, Perplexity = 18.55
200
+ Step 44260/44300 | Loss: 3.0616 | LR: 5.00e-05 | 3723.4ms/step | 5.4 samples/s
201
+ Step 44270/44300 | Loss: 3.0468 | LR: 5.00e-05 | 1546.4ms/step | 12.9 samples/s
202
+ Step 44280/44300 | Loss: 3.0566 | LR: 5.00e-05 | 1547.0ms/step | 12.9 samples/s
203
+ Step 44290/44300 | Loss: 3.0719 | LR: 5.00e-05 | 1546.3ms/step | 12.9 samples/s
204
+ Step 44300/44300 | Loss: 3.0755 | LR: 5.00e-05 | 1545.8ms/step | 12.9 samples/s
205
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/dropctl_calib_p30_verify/run_20260819_043051/checkpoint-44300
206
+ Training complete!
epfl_final/dropctl_logs/diag_p28_20.log ADDED
@@ -0,0 +1,174 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ nohup: ignoring input
2
+ New run: Checkpoints will be saved to /mloscratch/homes/navasard/sparse/checkpoints/dropctl_diag_p28_20/run_20260819_041508
3
+ Running config:
4
+ {
5
+ "model_path": "/mloscratch/homes/navasard/qwen3_06b_base",
6
+ "model_dtype": "bfloat16",
7
+ "init_from_scratch": false,
8
+ "train_path": "/mloscratch/homes/navasard/data/fineweb_large/tokenized",
9
+ "val_path": "/mloscratch/homes/navasard/data/fineweb_val_seed42",
10
+ "max_seq_length": 8192,
11
+ "num_workers": 8,
12
+ "output_dir": "/mloscratch/homes/navasard/sparse/checkpoints/dropctl_diag_p28_20/run_20260819_041508",
13
+ "resume_from_checkpoint": "/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000",
14
+ "force_new_run": true,
15
+ "num_epochs": 1,
16
+ "max_steps": 44020,
17
+ "per_device_batch_size": 2,
18
+ "gradient_accumulation_steps": 10,
19
+ "learning_rate": 5e-05,
20
+ "weight_decay": 0.01,
21
+ "warmup_steps": 0,
22
+ "lr_scheduler": "constant",
23
+ "max_grad_norm": 1.0,
24
+ "save_steps": 20,
25
+ "eval_steps": 25,
26
+ "eval_batches": 50,
27
+ "logging_steps": 10,
28
+ "seed": 42,
29
+ "verbose": true,
30
+ "checkpoint_activations": false,
31
+ "strategy": "ddp",
32
+ "mixed_precision": "bf16",
33
+ "wandb_enabled": false,
34
+ "wandb_project": "stochastic_sparsity",
35
+ "wandb_name": "dropctl_diag_p28_20",
36
+ "wandb_tags": [
37
+ "qwen3",
38
+ "365m",
39
+ "8k",
40
+ "scaleladder",
41
+ "dense"
42
+ ],
43
+ "hidden_size": 768,
44
+ "intermediate_size": 3072,
45
+ "num_attention_heads": 6,
46
+ "num_key_value_heads": 2,
47
+ "num_hidden_layers": 28,
48
+ "head_dim": null,
49
+ "attention_dropout": 0.28,
50
+ "attention_type": "sparse_stochastic",
51
+ "sparse_block_size": 128,
52
+ "sparse_num_random_blocks": 31,
53
+ "sparse_mask_pool_size": 1,
54
+ "sparse_val_dense_compare": false,
55
+ "sparse_layer_indices": null,
56
+ "sparse_mask_cache_dir": null,
57
+ "sparse_mask_seed": 0,
58
+ "sparse_mixed_nrb_set": null,
59
+ "dense_mode": true,
60
+ "dense_step_intervals": null
61
+ }
62
+ Loading tokenizer from /mloscratch/homes/navasard/qwen3_06b_base...
63
+ Resuming from checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000
64
+ Checkpoint activations: False
65
+ dense_mode=True: sparse attention layers will use standard causal SDPA.
66
+
67
+
68
+ ================================================================================
69
+ MODEL ARCHITECTURE
70
+ ================================================================================
71
+ Model config:
72
+ vocab_size: 151936
73
+ hidden_size: 768
74
+ num_hidden_layers: 28
75
+ num_attention_heads: 6
76
+ num_key_value_heads: 2
77
+ head_dim: 128
78
+ intermediate_size: 3072
79
+ max_position_embeddings: 32768
80
+ rope_theta: 1000000
81
+ attention_type: sparse_stochastic
82
+ sparse_block_size: 128
83
+ sparse_num_random_blocks: 31
84
+ sparse_mask_pool_size: 1
85
+ sparse_max_seq_length: 8192
86
+
87
+ Parameter count:
88
+ Total parameters: 358,958,848
89
+ Trainable parameters: 358,958,848
90
+ Non-trainable parameters: 0
91
+ Model size (BF16): 0.72 GB
92
+
93
+ --------------------------------------------------------------------------------
94
+ LAYER-BY-LAYER SUMMARY
95
+ --------------------------------------------------------------------------------
96
+ ================================================================================================================================================================
97
+ Layer (type:depth-idx) Input Shape Output Shape Param # Trainable
98
+ ================================================================================================================================================================
99
+ Qwen3ForCausalLM [2, 8192] -- -- True
100
+ ├─Qwen3Model: 1-1 -- [2, 8192, 768] -- True
101
+ │ └─Embedding: 2-1 [2, 8192] [2, 8192, 768] 116,686,848 True
102
+ │ └─ModuleList: 2-2 -- -- -- True
103
+ │ │ └─Qwen3DecoderLayer: 3-1 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
104
+ │ │ └─Qwen3DecoderLayer: 3-2 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
105
+ │ │ └─Qwen3DecoderLayer: 3-3 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
106
+ │ │ └─Qwen3DecoderLayer: 3-4 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
107
+ │ │ └─Qwen3DecoderLayer: 3-5 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
108
+ │ │ └─Qwen3DecoderLayer: 3-6 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
109
+ │ │ └─Qwen3DecoderLayer: 3-7 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
110
+ │ │ └─Qwen3DecoderLayer: 3-8 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
111
+ │ │ └─Qwen3DecoderLayer: 3-9 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
112
+ │ │ └─Qwen3DecoderLayer: 3-10 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
113
+ │ │ └─Qwen3DecoderLayer: 3-11 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
114
+ │ │ └─Qwen3DecoderLayer: 3-12 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
115
+ │ │ └─Qwen3DecoderLayer: 3-13 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
116
+ │ │ └─Qwen3DecoderLayer: 3-14 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
117
+ │ │ └─Qwen3DecoderLayer: 3-15 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
118
+ │ │ └─Qwen3DecoderLayer: 3-16 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
119
+ │ │ └─Qwen3DecoderLayer: 3-17 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
120
+ │ │ └─Qwen3DecoderLayer: 3-18 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
121
+ │ │ └─Qwen3DecoderLayer: 3-19 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
122
+ │ │ └─Qwen3DecoderLayer: 3-20 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
123
+ │ │ └─Qwen3DecoderLayer: 3-21 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
124
+ │ │ └─Qwen3DecoderLayer: 3-22 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
125
+ │ │ └─Qwen3DecoderLayer: 3-23 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
126
+ │ │ └─Qwen3DecoderLayer: 3-24 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
127
+ │ │ └─Qwen3DecoderLayer: 3-25 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
128
+ │ │ └─Qwen3DecoderLayer: 3-26 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
129
+ │ │ └─Qwen3DecoderLayer: 3-27 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
130
+ │ │ └─Qwen3DecoderLayer: 3-28 [2, 8192, 768] [2, 8192, 768] 8,652,544 True
131
+ │ └─Qwen3RMSNorm: 2-3 [2, 8192, 768] [2, 8192, 768] 768 True
132
+ ├─Linear: 1-2 [2, 8192, 768] [2, 8192, 151936] 116,686,848 True
133
+ ================================================================================================================================================================
134
+ Total params: 475,645,696
135
+ Trainable params: 475,645,696
136
+ Non-trainable params: 0
137
+ Total mult-adds (Units.MEGABYTES): 951.29
138
+ ================================================================================================================================================================
139
+ Input size (MB): 0.13
140
+ Forward/backward pass size (MB): 31197.23
141
+ Params size (MB): 951.29
142
+ Estimated Total Size (MB): 32148.66
143
+ ================================================================================================================================================================
144
+ NOTE: torchinfo double-counts tied embed_tokens/lm_head weight (116,686,848 params). True unique parameter count: 358,958,848 (359.0M)
145
+ ================================================================================
146
+
147
+ Loading dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
148
+ Using pre-tokenized dataset (no runtime tokenization)
149
+ Loading pre-tokenized HF dataset from /mloscratch/homes/navasard/data/fineweb_large/tokenized...
150
+ Loaded 12748481 pre-tokenized examples (stored_len=2048, pack_factor=4, packed_examples=3187120)
151
+ Dataset split: 3027764 train, 159356 validation
152
+ Training examples (packed, train split): 3027764
153
+ Validation examples (packed, val split): 159356
154
+ Per-device batch size: 2
155
+ World size (GPUs): 1
156
+ Gradient accumulation steps: 10
157
+ Effective batch size (seqs/step): 20
158
+ Sequence length: 8192
159
+ Tokens per optimizer step: 163,840
160
+ Micro-batches per epoch (per rank): 1513882
161
+ Optimizer steps per epoch: 151388
162
+ Total optimizer steps: 44020 (capped by max_steps; 0.29 epochs)
163
+ Loading checkpoint from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000...
164
+ Overriding checkpoint LR with config LR: 5e-05
165
+ Resumed from checkpoint at step 44000 (epoch 0, micro_step 440000)
166
+ Resuming training from step 44000
167
+ Starting training...
168
+ Validation at step 44000: Loss = 2.9127, Perplexity = 18.41
169
+ Switching attention mode for step 44001: dense (dense step interval)
170
+ Step 44010/44020 | Loss: 3.1491 | LR: 5.00e-05 | 54963.6ms/step | 0.4 samples/s
171
+ Step 44020/44020 | Loss: 3.1438 | LR: 5.00e-05 | 1528.9ms/step | 13.1 samples/s
172
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/dropctl_diag_p28_20/run_20260819_041508/checkpoint-44020
173
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/dropctl_diag_p28_20/run_20260819_041508/checkpoint-44020
174
+ Training complete!
epfl_final/eval_logs/r2_heldout_expanded_20260820.log ADDED
@@ -0,0 +1,164 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [data] arxiv: held-out windows [48:72)
2
+ [data] fineweb: held-out windows [48:72)
3
+ [data] pg19: held-out windows [48:72)
4
+ [load] dense seed=42 k=64 checkpoint=/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000
5
+ Checkpoint activations: False
6
+ [done] dense s42 k=64 tau=1.0 arxiv ppl=20.827807
7
+ [done] dense s42 k=64 tau=1.0 fineweb ppl=17.590889
8
+ [done] dense s42 k=64 tau=1.0 pg19 ppl=51.644387
9
+ [done] dense s42 k=64 tau=1.0 arxiv ppl=20.827807
10
+ [done] dense s42 k=64 tau=1.0 fineweb ppl=17.590889
11
+ [done] dense s42 k=64 tau=1.0 pg19 ppl=51.644387
12
+ [load] dense seed=42 k=4 checkpoint=/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense/run_20260707_034553/checkpoint-44000
13
+ Checkpoint activations: False
14
+ [done] dense s42 k=4 tau=1.0 arxiv ppl=28.062414
15
+ [done] dense s42 k=4 tau=1.0 fineweb ppl=18.855334
16
+ [done] dense s42 k=4 tau=1.0 pg19 ppl=62.142987
17
+ [done] dense s42 k=4 tau=1.1 arxiv ppl=26.810457
18
+ [done] dense s42 k=4 tau=1.1 fineweb ppl=18.489435
19
+ [done] dense s42 k=4 tau=1.1 pg19 ppl=59.543974
20
+ [load] dense seed=43 k=64 checkpoint=/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_seed43/run_20260722_221755/checkpoint-44000
21
+ Checkpoint activations: False
22
+ [done] dense s43 k=64 tau=1.0 arxiv ppl=20.679261
23
+ [done] dense s43 k=64 tau=1.0 fineweb ppl=17.272070
24
+ [done] dense s43 k=64 tau=1.0 pg19 ppl=51.917639
25
+ [done] dense s43 k=64 tau=1.0 arxiv ppl=20.679261
26
+ [done] dense s43 k=64 tau=1.0 fineweb ppl=17.272070
27
+ [done] dense s43 k=64 tau=1.0 pg19 ppl=51.917639
28
+ [load] dense seed=43 k=4 checkpoint=/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_seed43/run_20260722_221755/checkpoint-44000
29
+ Checkpoint activations: False
30
+ [done] dense s43 k=4 tau=1.0 arxiv ppl=27.665412
31
+ [done] dense s43 k=4 tau=1.0 fineweb ppl=18.279679
32
+ [done] dense s43 k=4 tau=1.0 pg19 ppl=60.683716
33
+ [done] dense s43 k=4 tau=1.1 arxiv ppl=26.998110
34
+ [done] dense s43 k=4 tau=1.1 fineweb ppl=18.077326
35
+ [done] dense s43 k=4 tau=1.1 pg19 ppl=59.381247
36
+ [load] dense seed=44 k=64 checkpoint=/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_seed44/run_20260722_223953/checkpoint-44000
37
+ Checkpoint activations: False
38
+ [done] dense s44 k=64 tau=1.0 arxiv ppl=21.218933
39
+ [done] dense s44 k=64 tau=1.0 fineweb ppl=17.346678
40
+ [done] dense s44 k=64 tau=1.0 pg19 ppl=51.723888
41
+ [done] dense s44 k=64 tau=1.0 arxiv ppl=21.218933
42
+ [done] dense s44 k=64 tau=1.0 fineweb ppl=17.346678
43
+ [done] dense s44 k=64 tau=1.0 pg19 ppl=51.723888
44
+ [load] dense seed=44 k=4 checkpoint=/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_seed44/run_20260722_223953/checkpoint-44000
45
+ Checkpoint activations: False
46
+ [done] dense s44 k=4 tau=1.0 arxiv ppl=28.020652
47
+ [done] dense s44 k=4 tau=1.0 fineweb ppl=18.379780
48
+ [done] dense s44 k=4 tau=1.0 pg19 ppl=60.211017
49
+ [done] dense s44 k=4 tau=1.1 arxiv ppl=27.541957
50
+ [done] dense s44 k=4 tau=1.1 fineweb ppl=18.232032
51
+ [done] dense s44 k=4 tau=1.1 pg19 ppl=59.030090
52
+ [load] dense seed=45 k=64 checkpoint=/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_seed45/run_20260722_221755/checkpoint-44000
53
+ Checkpoint activations: False
54
+ [done] dense s45 k=64 tau=1.0 arxiv ppl=20.126801
55
+ [done] dense s45 k=64 tau=1.0 fineweb ppl=17.252433
56
+ [done] dense s45 k=64 tau=1.0 pg19 ppl=51.048448
57
+ [done] dense s45 k=64 tau=1.0 arxiv ppl=20.126801
58
+ [done] dense s45 k=64 tau=1.0 fineweb ppl=17.252433
59
+ [done] dense s45 k=64 tau=1.0 pg19 ppl=51.048448
60
+ [load] dense seed=45 k=4 checkpoint=/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_seed45/run_20260722_221755/checkpoint-44000
61
+ Checkpoint activations: False
62
+ [done] dense s45 k=4 tau=1.0 arxiv ppl=27.289920
63
+ [done] dense s45 k=4 tau=1.0 fineweb ppl=18.366931
64
+ [done] dense s45 k=4 tau=1.0 pg19 ppl=60.491529
65
+ [done] dense s45 k=4 tau=1.1 arxiv ppl=26.478482
66
+ [done] dense s45 k=4 tau=1.1 fineweb ppl=18.114687
67
+ [done] dense s45 k=4 tau=1.1 pg19 ppl=58.885984
68
+ [load] dense seed=46 k=64 checkpoint=/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_seed46/run_20260722_223953/checkpoint-44000
69
+ Checkpoint activations: False
70
+ [done] dense s46 k=64 tau=1.0 arxiv ppl=20.378415
71
+ [done] dense s46 k=64 tau=1.0 fineweb ppl=17.239947
72
+ [done] dense s46 k=64 tau=1.0 pg19 ppl=51.087885
73
+ [done] dense s46 k=64 tau=1.0 arxiv ppl=20.378415
74
+ [done] dense s46 k=64 tau=1.0 fineweb ppl=17.239947
75
+ [done] dense s46 k=64 tau=1.0 pg19 ppl=51.087885
76
+ [load] dense seed=46 k=4 checkpoint=/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_seed46/run_20260722_223953/checkpoint-44000
77
+ Checkpoint activations: False
78
+ [done] dense s46 k=4 tau=1.0 arxiv ppl=29.063718
79
+ [done] dense s46 k=4 tau=1.0 fineweb ppl=18.346118
80
+ [done] dense s46 k=4 tau=1.0 pg19 ppl=61.549315
81
+ [done] dense s46 k=4 tau=1.1 arxiv ppl=27.720753
82
+ [done] dense s46 k=4 tau=1.1 fineweb ppl=18.116804
83
+ [done] dense s46 k=4 tau=1.1 pg19 ppl=59.443352
84
+ [load] mixed seed=42 k=64 checkpoint=/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_seed42/run_20260719_234149/checkpoint-44000
85
+ Checkpoint activations: False
86
+ [done] mixed s42 k=64 tau=1.0 arxiv ppl=21.310017
87
+ [done] mixed s42 k=64 tau=1.0 fineweb ppl=17.677243
88
+ [done] mixed s42 k=64 tau=1.0 pg19 ppl=52.384668
89
+ [done] mixed s42 k=64 tau=0.95 arxiv ppl=21.078126
90
+ [done] mixed s42 k=64 tau=0.95 fineweb ppl=17.590759
91
+ [done] mixed s42 k=64 tau=0.95 pg19 ppl=51.913180
92
+ [load] mixed seed=42 k=4 checkpoint=/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_seed42/run_20260719_234149/checkpoint-44000
93
+ Checkpoint activations: False
94
+ [done] mixed s42 k=4 tau=1.0 arxiv ppl=25.720382
95
+ [done] mixed s42 k=4 tau=1.0 fineweb ppl=18.078353
96
+ [done] mixed s42 k=4 tau=1.0 pg19 ppl=57.705661
97
+ [done] mixed s42 k=4 tau=1.05 arxiv ppl=25.515733
98
+ [done] mixed s42 k=4 tau=1.05 fineweb ppl=18.045929
99
+ [done] mixed s42 k=4 tau=1.05 pg19 ppl=57.407884
100
+ [load] mixed seed=43 k=64 checkpoint=/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_seed43/run_20260719_234149/checkpoint-44000
101
+ Checkpoint activations: False
102
+ [done] mixed s43 k=64 tau=1.0 arxiv ppl=21.827911
103
+ [done] mixed s43 k=64 tau=1.0 fineweb ppl=17.666934
104
+ [done] mixed s43 k=64 tau=1.0 pg19 ppl=52.213548
105
+ [done] mixed s43 k=64 tau=0.95 arxiv ppl=21.615961
106
+ [done] mixed s43 k=64 tau=0.95 fineweb ppl=17.578845
107
+ [done] mixed s43 k=64 tau=0.95 pg19 ppl=51.759837
108
+ [load] mixed seed=43 k=4 checkpoint=/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_seed43/run_20260719_234149/checkpoint-44000
109
+ Checkpoint activations: False
110
+ [done] mixed s43 k=4 tau=1.0 arxiv ppl=26.623032
111
+ [done] mixed s43 k=4 tau=1.0 fineweb ppl=18.141605
112
+ [done] mixed s43 k=4 tau=1.0 pg19 ppl=57.846202
113
+ [done] mixed s43 k=4 tau=1.05 arxiv ppl=26.427125
114
+ [done] mixed s43 k=4 tau=1.05 fineweb ppl=18.096165
115
+ [done] mixed s43 k=4 tau=1.05 pg19 ppl=57.532262
116
+ [load] mixed seed=44 k=64 checkpoint=/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_seed44/run_20260722_221755/checkpoint-44000
117
+ Checkpoint activations: False
118
+ [done] mixed s44 k=64 tau=1.0 arxiv ppl=21.488965
119
+ [done] mixed s44 k=64 tau=1.0 fineweb ppl=17.713621
120
+ [done] mixed s44 k=64 tau=1.0 pg19 ppl=52.943689
121
+ [done] mixed s44 k=64 tau=0.95 arxiv ppl=21.257669
122
+ [done] mixed s44 k=64 tau=0.95 fineweb ppl=17.625584
123
+ [done] mixed s44 k=64 tau=0.95 pg19 ppl=52.448237
124
+ [load] mixed seed=44 k=4 checkpoint=/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_seed44/run_20260722_221755/checkpoint-44000
125
+ Checkpoint activations: False
126
+ [done] mixed s44 k=4 tau=1.0 arxiv ppl=26.238666
127
+ [done] mixed s44 k=4 tau=1.0 fineweb ppl=18.079179
128
+ [done] mixed s44 k=4 tau=1.0 pg19 ppl=58.352118
129
+ [done] mixed s44 k=4 tau=1.05 arxiv ppl=26.008491
130
+ [done] mixed s44 k=4 tau=1.05 fineweb ppl=18.044383
131
+ [done] mixed s44 k=4 tau=1.05 pg19 ppl=58.014340
132
+ [load] mixed seed=45 k=64 checkpoint=/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_seed45/run_20260722_223953/checkpoint-44000
133
+ Checkpoint activations: False
134
+ [done] mixed s45 k=64 tau=1.0 arxiv ppl=21.225616
135
+ [done] mixed s45 k=64 tau=1.0 fineweb ppl=17.697083
136
+ [done] mixed s45 k=64 tau=1.0 pg19 ppl=53.116345
137
+ [done] mixed s45 k=64 tau=0.95 arxiv ppl=21.024314
138
+ [done] mixed s45 k=64 tau=0.95 fineweb ppl=17.603223
139
+ [done] mixed s45 k=64 tau=0.95 pg19 ppl=52.729393
140
+ [load] mixed seed=45 k=4 checkpoint=/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_seed45/run_20260722_223953/checkpoint-44000
141
+ Checkpoint activations: False
142
+ [done] mixed s45 k=4 tau=1.0 arxiv ppl=25.961290
143
+ [done] mixed s45 k=4 tau=1.0 fineweb ppl=18.093054
144
+ [done] mixed s45 k=4 tau=1.0 pg19 ppl=59.100388
145
+ [done] mixed s45 k=4 tau=1.05 arxiv ppl=25.759159
146
+ [done] mixed s45 k=4 tau=1.05 fineweb ppl=18.071190
147
+ [done] mixed s45 k=4 tau=1.05 pg19 ppl=58.739943
148
+ [load] mixed seed=46 k=64 checkpoint=/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_seed46/run_20260722_223953/checkpoint-44000
149
+ Checkpoint activations: False
150
+ [done] mixed s46 k=64 tau=1.0 arxiv ppl=20.873182
151
+ [done] mixed s46 k=64 tau=1.0 fineweb ppl=17.582663
152
+ [done] mixed s46 k=64 tau=1.0 pg19 ppl=52.113241
153
+ [done] mixed s46 k=64 tau=0.95 arxiv ppl=20.726842
154
+ [done] mixed s46 k=64 tau=0.95 fineweb ppl=17.486791
155
+ [done] mixed s46 k=64 tau=0.95 pg19 ppl=51.603325
156
+ [load] mixed seed=46 k=4 checkpoint=/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_seed46/run_20260722_223953/checkpoint-44000
157
+ Checkpoint activations: False
158
+ [done] mixed s46 k=4 tau=1.0 arxiv ppl=25.590572
159
+ [done] mixed s46 k=4 tau=1.0 fineweb ppl=18.008759
160
+ [done] mixed s46 k=4 tau=1.0 pg19 ppl=57.811801
161
+ [done] mixed s46 k=4 tau=1.05 arxiv ppl=25.323875
162
+ [done] mixed s46 k=4 tau=1.05 fineweb ppl=17.964840
163
+ [done] mixed s46 k=4 tau=1.05 pg19 ppl=57.480996
164
+ [wrote] /mloscratch/homes/navasard/sparse/SparseAtt/analysis/R2_HELDOUT_FIXEDTAU_EXPANDED_20260820/shard_0.tsv rows=120
epfl_final/eval_logs/r2_temperature_ruler_20260820.log ADDED
@@ -0,0 +1 @@
 
 
1
+ serial worker stopped for sharding
epfl_final/eval_logs/r2_temperature_shards/a100_gpu4.log ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ [a100_gpu4 96/105] START mixed_s46_k64_tau1_vt
2
+ [a100_gpu4 96/105] DONE mixed_s46_k64_tau1_vt
3
+ [a100_gpu4 97/105] START mixed_s46_k64_fixed_niah_multikey
epfl_final/eval_logs/r2_temperature_shards/h100_gpu2.log ADDED
@@ -0,0 +1,64 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [h100_gpu2 32/105] START dense_s45_k4_tau1_niah_multivalue
2
+ [h100_gpu2 32/105] DONE dense_s45_k4_tau1_niah_multivalue
3
+ [h100_gpu2 33/105] START dense_s45_k4_tau1_vt
4
+ [h100_gpu2 33/105] DONE dense_s45_k4_tau1_vt
5
+ [h100_gpu2 34/105] START dense_s45_k4_fixed_niah_multikey
6
+ [h100_gpu2 34/105] DONE dense_s45_k4_fixed_niah_multikey
7
+ [h100_gpu2 35/105] START dense_s45_k4_fixed_niah_multivalue
8
+ [h100_gpu2 35/105] DONE dense_s45_k4_fixed_niah_multivalue
9
+ [h100_gpu2 36/105] START dense_s45_k4_fixed_vt
10
+ [h100_gpu2 36/105] DONE dense_s45_k4_fixed_vt
11
+ [h100_gpu2 37/105] START dense_s46_k64_tau1_niah_multikey
12
+ [h100_gpu2 37/105] DONE dense_s46_k64_tau1_niah_multikey
13
+ [h100_gpu2 38/105] START dense_s46_k64_tau1_niah_multivalue
14
+ [h100_gpu2 38/105] DONE dense_s46_k64_tau1_niah_multivalue
15
+ [h100_gpu2 39/105] START dense_s46_k64_tau1_vt
16
+ [h100_gpu2 39/105] DONE dense_s46_k64_tau1_vt
17
+ [h100_gpu2 40/105] START dense_s46_k4_tau1_niah_multikey
18
+ [h100_gpu2 40/105] DONE dense_s46_k4_tau1_niah_multikey
19
+ [h100_gpu2 41/105] START dense_s46_k4_tau1_niah_multivalue
20
+ [h100_gpu2 41/105] DONE dense_s46_k4_tau1_niah_multivalue
21
+ [h100_gpu2 42/105] START dense_s46_k4_tau1_vt
22
+ [h100_gpu2 42/105] DONE dense_s46_k4_tau1_vt
23
+ [h100_gpu2 43/105] START dense_s46_k4_fixed_niah_multikey
24
+ [h100_gpu2 43/105] DONE dense_s46_k4_fixed_niah_multikey
25
+ [h100_gpu2 44/105] START dense_s46_k4_fixed_niah_multivalue
26
+ [h100_gpu2 44/105] DONE dense_s46_k4_fixed_niah_multivalue
27
+ [h100_gpu2 45/105] START dense_s46_k4_fixed_vt
28
+ [h100_gpu2 45/105] DONE dense_s46_k4_fixed_vt
29
+ [h100_gpu2 46/105] START mixed_s42_k64_tau1_niah_multikey
30
+ [h100_gpu2 46/105] DONE mixed_s42_k64_tau1_niah_multikey
31
+ [h100_gpu2 47/105] START mixed_s42_k64_tau1_niah_multivalue
32
+ [h100_gpu2 47/105] DONE mixed_s42_k64_tau1_niah_multivalue
33
+ [h100_gpu2 48/105] START mixed_s42_k64_tau1_vt
34
+ [h100_gpu2 48/105] DONE mixed_s42_k64_tau1_vt
35
+ [h100_gpu2 49/105] START mixed_s42_k64_fixed_niah_multikey
36
+ [h100_gpu2 49/105] DONE mixed_s42_k64_fixed_niah_multikey
37
+ [h100_gpu2 50/105] START mixed_s42_k64_fixed_niah_multivalue
38
+ [h100_gpu2 50/105] DONE mixed_s42_k64_fixed_niah_multivalue
39
+ [h100_gpu2 51/105] START mixed_s42_k64_fixed_vt
40
+ [h100_gpu2 51/105] DONE mixed_s42_k64_fixed_vt
41
+ [h100_gpu2 52/105] START mixed_s42_k4_tau1_niah_multikey
42
+ [h100_gpu2 52/105] DONE mixed_s42_k4_tau1_niah_multikey
43
+ [h100_gpu2 53/105] START mixed_s42_k4_tau1_niah_multivalue
44
+ [h100_gpu2 53/105] DONE mixed_s42_k4_tau1_niah_multivalue
45
+ [h100_gpu2 54/105] START mixed_s42_k4_tau1_vt
46
+ [h100_gpu2 54/105] DONE mixed_s42_k4_tau1_vt
47
+ [h100_gpu2 55/105] START mixed_s42_k4_fixed_niah_multikey
48
+ [h100_gpu2 55/105] DONE mixed_s42_k4_fixed_niah_multikey
49
+ [h100_gpu2 56/105] START mixed_s42_k4_fixed_niah_multivalue
50
+ [h100_gpu2 56/105] DONE mixed_s42_k4_fixed_niah_multivalue
51
+ [h100_gpu2 57/105] START mixed_s42_k4_fixed_vt
52
+ [h100_gpu2 57/105] DONE mixed_s42_k4_fixed_vt
53
+ [h100_gpu2 58/105] START mixed_s43_k64_tau1_niah_multikey
54
+ [h100_gpu2 58/105] DONE mixed_s43_k64_tau1_niah_multikey
55
+ [h100_gpu2 59/105] START mixed_s43_k64_tau1_niah_multivalue
56
+ [h100_gpu2 59/105] DONE mixed_s43_k64_tau1_niah_multivalue
57
+ [h100_gpu2 60/105] START mixed_s43_k64_tau1_vt
58
+ [h100_gpu2 60/105] DONE mixed_s43_k64_tau1_vt
59
+ [h100_gpu2 61/105] START mixed_s43_k64_fixed_niah_multikey
60
+ [h100_gpu2 61/105] DONE mixed_s43_k64_fixed_niah_multikey
61
+ [h100_gpu2 62/105] START mixed_s43_k64_fixed_niah_multivalue
62
+ [h100_gpu2 62/105] DONE mixed_s43_k64_fixed_niah_multivalue
63
+ [h100_gpu2 63/105] START mixed_s43_k64_fixed_vt
64
+ [h100_gpu2 63/105] DONE mixed_s43_k64_fixed_vt
epfl_final/eval_logs/r2_temperature_shards/h100_gpu2_tail.log ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ [h100_gpu2_tail 98/105] START mixed_s46_k64_fixed_niah_multivalue
2
+ [h100_gpu2_tail 98/105] DONE mixed_s46_k64_fixed_niah_multivalue
3
+ [h100_gpu2_tail 99/105] START mixed_s46_k64_fixed_vt
4
+ [h100_gpu2_tail 99/105] DONE mixed_s46_k64_fixed_vt
5
+ [h100_gpu2_tail 100/105] START mixed_s46_k4_tau1_niah_multikey
6
+ [h100_gpu2_tail 100/105] DONE mixed_s46_k4_tau1_niah_multikey
7
+ [h100_gpu2_tail 101/105] START mixed_s46_k4_tau1_niah_multivalue
8
+ [h100_gpu2_tail 101/105] DONE mixed_s46_k4_tau1_niah_multivalue
epfl_final/eval_logs/r2_temperature_shards/h100_gpu3.log ADDED
@@ -0,0 +1,64 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [h100_gpu3 64/105] START mixed_s43_k4_tau1_niah_multikey
2
+ [h100_gpu3 64/105] DONE mixed_s43_k4_tau1_niah_multikey
3
+ [h100_gpu3 65/105] START mixed_s43_k4_tau1_niah_multivalue
4
+ [h100_gpu3 65/105] DONE mixed_s43_k4_tau1_niah_multivalue
5
+ [h100_gpu3 66/105] START mixed_s43_k4_tau1_vt
6
+ [h100_gpu3 66/105] DONE mixed_s43_k4_tau1_vt
7
+ [h100_gpu3 67/105] START mixed_s43_k4_fixed_niah_multikey
8
+ [h100_gpu3 67/105] DONE mixed_s43_k4_fixed_niah_multikey
9
+ [h100_gpu3 68/105] START mixed_s43_k4_fixed_niah_multivalue
10
+ [h100_gpu3 68/105] DONE mixed_s43_k4_fixed_niah_multivalue
11
+ [h100_gpu3 69/105] START mixed_s43_k4_fixed_vt
12
+ [h100_gpu3 69/105] DONE mixed_s43_k4_fixed_vt
13
+ [h100_gpu3 70/105] START mixed_s44_k64_tau1_niah_multikey
14
+ [h100_gpu3 70/105] DONE mixed_s44_k64_tau1_niah_multikey
15
+ [h100_gpu3 71/105] START mixed_s44_k64_tau1_niah_multivalue
16
+ [h100_gpu3 71/105] DONE mixed_s44_k64_tau1_niah_multivalue
17
+ [h100_gpu3 72/105] START mixed_s44_k64_tau1_vt
18
+ [h100_gpu3 72/105] DONE mixed_s44_k64_tau1_vt
19
+ [h100_gpu3 73/105] START mixed_s44_k64_fixed_niah_multikey
20
+ [h100_gpu3 73/105] DONE mixed_s44_k64_fixed_niah_multikey
21
+ [h100_gpu3 74/105] START mixed_s44_k64_fixed_niah_multivalue
22
+ [h100_gpu3 74/105] DONE mixed_s44_k64_fixed_niah_multivalue
23
+ [h100_gpu3 75/105] START mixed_s44_k64_fixed_vt
24
+ [h100_gpu3 75/105] DONE mixed_s44_k64_fixed_vt
25
+ [h100_gpu3 76/105] START mixed_s44_k4_tau1_niah_multikey
26
+ [h100_gpu3 76/105] DONE mixed_s44_k4_tau1_niah_multikey
27
+ [h100_gpu3 77/105] START mixed_s44_k4_tau1_niah_multivalue
28
+ [h100_gpu3 77/105] DONE mixed_s44_k4_tau1_niah_multivalue
29
+ [h100_gpu3 78/105] START mixed_s44_k4_tau1_vt
30
+ [h100_gpu3 78/105] DONE mixed_s44_k4_tau1_vt
31
+ [h100_gpu3 79/105] START mixed_s44_k4_fixed_niah_multikey
32
+ [h100_gpu3 79/105] DONE mixed_s44_k4_fixed_niah_multikey
33
+ [h100_gpu3 80/105] START mixed_s44_k4_fixed_niah_multivalue
34
+ [h100_gpu3 80/105] DONE mixed_s44_k4_fixed_niah_multivalue
35
+ [h100_gpu3 81/105] START mixed_s44_k4_fixed_vt
36
+ [h100_gpu3 81/105] DONE mixed_s44_k4_fixed_vt
37
+ [h100_gpu3 82/105] START mixed_s45_k64_tau1_niah_multikey
38
+ [h100_gpu3 82/105] DONE mixed_s45_k64_tau1_niah_multikey
39
+ [h100_gpu3 83/105] START mixed_s45_k64_tau1_niah_multivalue
40
+ [h100_gpu3 83/105] DONE mixed_s45_k64_tau1_niah_multivalue
41
+ [h100_gpu3 84/105] START mixed_s45_k64_tau1_vt
42
+ [h100_gpu3 84/105] DONE mixed_s45_k64_tau1_vt
43
+ [h100_gpu3 85/105] START mixed_s45_k64_fixed_niah_multikey
44
+ [h100_gpu3 85/105] DONE mixed_s45_k64_fixed_niah_multikey
45
+ [h100_gpu3 86/105] START mixed_s45_k64_fixed_niah_multivalue
46
+ [h100_gpu3 86/105] DONE mixed_s45_k64_fixed_niah_multivalue
47
+ [h100_gpu3 87/105] START mixed_s45_k64_fixed_vt
48
+ [h100_gpu3 87/105] DONE mixed_s45_k64_fixed_vt
49
+ [h100_gpu3 88/105] START mixed_s45_k4_tau1_niah_multikey
50
+ [h100_gpu3 88/105] DONE mixed_s45_k4_tau1_niah_multikey
51
+ [h100_gpu3 89/105] START mixed_s45_k4_tau1_niah_multivalue
52
+ [h100_gpu3 89/105] DONE mixed_s45_k4_tau1_niah_multivalue
53
+ [h100_gpu3 90/105] START mixed_s45_k4_tau1_vt
54
+ [h100_gpu3 90/105] DONE mixed_s45_k4_tau1_vt
55
+ [h100_gpu3 91/105] START mixed_s45_k4_fixed_niah_multikey
56
+ [h100_gpu3 91/105] DONE mixed_s45_k4_fixed_niah_multikey
57
+ [h100_gpu3 92/105] START mixed_s45_k4_fixed_niah_multivalue
58
+ [h100_gpu3 92/105] DONE mixed_s45_k4_fixed_niah_multivalue
59
+ [h100_gpu3 93/105] START mixed_s45_k4_fixed_vt
60
+ [h100_gpu3 93/105] DONE mixed_s45_k4_fixed_vt
61
+ [h100_gpu3 94/105] START mixed_s46_k64_tau1_niah_multikey
62
+ [h100_gpu3 94/105] DONE mixed_s46_k64_tau1_niah_multikey
63
+ [h100_gpu3 95/105] START mixed_s46_k64_tau1_niah_multivalue
64
+ [h100_gpu3 95/105] DONE mixed_s46_k64_tau1_niah_multivalue
epfl_final/eval_logs/r2_temperature_shards/h100_gpu3_tail.log ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ [h100_gpu3_tail 102/105] START mixed_s46_k4_tau1_vt
2
+ [h100_gpu3_tail 102/105] DONE mixed_s46_k4_tau1_vt
3
+ [h100_gpu3_tail 103/105] START mixed_s46_k4_fixed_niah_multikey
4
+ [h100_gpu3_tail 103/105] DONE mixed_s46_k4_fixed_niah_multikey
5
+ [h100_gpu3_tail 104/105] START mixed_s46_k4_fixed_niah_multivalue
6
+ [h100_gpu3_tail 104/105] DONE mixed_s46_k4_fixed_niah_multivalue
7
+ [h100_gpu3_tail 105/105] START mixed_s46_k4_fixed_vt
8
+ [h100_gpu3_tail 105/105] DONE mixed_s46_k4_fixed_vt
epfl_final/eval_logs/r2_temperature_shards/h100_tail_watcher.log ADDED
File without changes
epfl_final/eval_logs/rope42_sharded_gpu0_20260821.log ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ [start] 8192_rope42_arxiv
2
+ [done] 8192_rope42_arxiv
3
+ [start] 8192_rope42_fineweb
4
+ [done] 8192_rope42_fineweb
5
+ [start] 8192_rope42_pg19
6
+ [done] 8192_rope42_pg19
7
+ [complete] worker gpu0, len8192
epfl_final/eval_logs/rope42_sharded_gpu1_20260821.log ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ [start] 12288_rope42_arxiv
2
+ [done] 12288_rope42_arxiv
3
+ [start] 12288_rope42_fineweb
4
+ [done] 12288_rope42_fineweb
5
+ [start] 12288_rope42_pg19
6
+ [done] 12288_rope42_pg19
7
+ [complete] worker gpu1, len12288
epfl_final/eval_logs/rope42_sharded_gpu2_20260821.log ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ [start] 16384_rope42_arxiv
2
+ [done] 16384_rope42_arxiv
3
+ [start] 16384_rope42_fineweb
4
+ [done] 16384_rope42_fineweb
5
+ [start] 16384_rope42_pg19
6
+ [done] 16384_rope42_pg19
7
+ [complete] worker gpu2, len16384
epfl_final/eval_logs/rope42_sharded_gpu3_20260821.log ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ [start] 24576_rope42_arxiv
2
+ [done] 24576_rope42_arxiv
3
+ [start] 24576_rope42_fineweb
4
+ [done] 24576_rope42_fineweb
5
+ [start] 24576_rope42_pg19
6
+ [done] 24576_rope42_pg19
7
+ [complete] worker gpu3, len24576
epfl_final/eval_logs/rope43_sharded_gpu1_20260821.log ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ [start] 8192_rope43_arxiv
2
+ [done] 8192_rope43_arxiv
3
+ [start] 8192_rope43_fineweb
4
+ [done] 8192_rope43_fineweb
5
+ [start] 8192_rope43_pg19
6
+ [done] 8192_rope43_pg19
7
+ [start] 12288_rope43_arxiv
8
+ [done] 12288_rope43_arxiv
9
+ [complete] worker gpu1
epfl_final/eval_logs/rope43_sharded_gpu2_20260821.log ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ [start] 12288_rope43_fineweb
2
+ [done] 12288_rope43_fineweb
3
+ [start] 12288_rope43_pg19
4
+ [done] 12288_rope43_pg19
5
+ [start] 16384_rope43_arxiv
6
+ [done] 16384_rope43_arxiv
7
+ [start] 16384_rope43_fineweb
8
+ [done] 16384_rope43_fineweb
9
+ [complete] worker gpu2
epfl_final/eval_logs/rope43_sharded_gpu3_20260821.log ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ [start] 16384_rope43_pg19
2
+ [done] 16384_rope43_pg19
3
+ [start] 24576_rope43_arxiv
4
+ [done] 24576_rope43_arxiv
5
+ [start] 24576_rope43_fineweb
6
+ [done] 24576_rope43_fineweb
7
+ [start] 24576_rope43_pg19
8
+ [done] 24576_rope43_pg19
9
+ [complete] worker gpu3
epfl_final/eval_logs/rope44_a1005_20260821/sharded.log ADDED
@@ -0,0 +1,74 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [start] 2026-08-21T12:46:54+00:00: seed44 matched eval, five A100 shards
2
+ [gpu4] 2026-08-21T12:46:54+00:00 12288 fineweb
3
+ [gpu3] 2026-08-21T12:46:54+00:00 12288 arxiv
4
+ [gpu0] 2026-08-21T12:46:54+00:00 8192 arxiv
5
+ [gpu2] 2026-08-21T12:46:54+00:00 8192 pg19
6
+ [gpu1] 2026-08-21T12:46:54+00:00 8192 fineweb
7
+ Checkpoint activations: False
8
+ Checkpoint activations: False
9
+ Checkpoint activations: False
10
+ Checkpoint activations: False
11
+ Checkpoint activations: False
12
+ Checkpoint activations: False
13
+ Checkpoint activations: False
14
+ Checkpoint activations: False
15
+ Checkpoint activations: False
16
+ Checkpoint activations: False
17
+ Checkpoint activations: False
18
+ Checkpoint activations: False
19
+ Checkpoint activations: False
20
+ Checkpoint activations: False
21
+ Checkpoint activations: False
22
+ Checkpoint activations: False
23
+ Checkpoint activations: False
24
+ Checkpoint activations: False
25
+ Checkpoint activations: False
26
+ Checkpoint activations: False
27
+ [gpu4] 2026-08-21T12:49:22+00:00 24576 arxiv
28
+ [gpu0] 2026-08-21T12:49:22+00:00 12288 pg19
29
+ [gpu2] 2026-08-21T12:49:22+00:00 16384 fineweb
30
+ Checkpoint activations: False
31
+ Checkpoint activations: False
32
+ Checkpoint activations: False
33
+ Checkpoint activations: False
34
+ Checkpoint activations: False
35
+ [gpu3] 2026-08-21T12:49:25+00:00 16384 pg19
36
+ [gpu1] 2026-08-21T12:49:26+00:00 16384 arxiv
37
+ Checkpoint activations: False
38
+ Checkpoint activations: False
39
+ Checkpoint activations: False
40
+ Checkpoint activations: False
41
+ Checkpoint activations: False
42
+ Checkpoint activations: False
43
+ Checkpoint activations: False
44
+ Checkpoint activations: False
45
+ Checkpoint activations: False
46
+ Checkpoint activations: False
47
+ Checkpoint activations: False
48
+ Checkpoint activations: False
49
+ Checkpoint activations: False
50
+ Checkpoint activations: False
51
+ Checkpoint activations: False
52
+ Checkpoint activations: False
53
+ Checkpoint activations: False
54
+ Checkpoint activations: False
55
+ Checkpoint activations: False
56
+ Checkpoint activations: False
57
+ Checkpoint activations: False
58
+ Checkpoint activations: False
59
+ Checkpoint activations: False
60
+ Checkpoint activations: False
61
+ Checkpoint activations: False
62
+ [gpu1] 2026-08-21T12:51:54+00:00 24576 pg19
63
+ [gpu0] 2026-08-21T12:51:54+00:00 24576 fineweb
64
+ Checkpoint activations: False
65
+ Checkpoint activations: False
66
+ Checkpoint activations: False
67
+ Checkpoint activations: False
68
+ Checkpoint activations: False
69
+ Checkpoint activations: False
70
+ Checkpoint activations: False
71
+ Checkpoint activations: False
72
+ Checkpoint activations: False
73
+ Checkpoint activations: False
74
+ [done] 2026-08-21T12:54:14+00:00: seed44 all matched evals complete
epfl_final/eval_logs/rope44_after_mixed48_20260821/watcher.log ADDED
@@ -0,0 +1,15 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [wait] 2026-08-21T12:31:23+00:00: Mixed48 not yet complete
2
+ [wait] 2026-08-21T12:32:23+00:00: Mixed48 not yet complete
3
+ [wait] 2026-08-21T12:33:23+00:00: Mixed48 not yet complete
4
+ [wait] 2026-08-21T12:34:23+00:00: Mixed48 not yet complete
5
+ [wait] 2026-08-21T12:35:23+00:00: Mixed48 not yet complete
6
+ [wait] 2026-08-21T12:36:23+00:00: Mixed48 not yet complete
7
+ [wait] 2026-08-21T12:37:23+00:00: Mixed48 not yet complete
8
+ [wait] 2026-08-21T12:38:23+00:00: Mixed48 not yet complete
9
+ [wait] 2026-08-21T12:39:23+00:00: Mixed48 not yet complete
10
+ [wait] 2026-08-21T12:40:23+00:00: Mixed48 not yet complete
11
+ [wait] 2026-08-21T12:41:23+00:00: Mixed48 not yet complete
12
+ [wait] 2026-08-21T12:42:23+00:00: Mixed48 not yet complete
13
+ [wait] 2026-08-21T12:43:23+00:00: Mixed48 not yet complete
14
+ [wait] 2026-08-21T12:44:23+00:00: Mixed48 not yet complete
15
+ [wait] 2026-08-21T12:45:23+00:00: Mixed48 not yet complete
epfl_final/eval_logs/rope44_auto_20260821/watcher.log ADDED
@@ -0,0 +1,332 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [wait] 2026-08-21T05:03:56+00:00: seed44 checkpoint not yet complete
2
+ [wait] 2026-08-21T05:04:56+00:00: seed44 checkpoint not yet complete
3
+ [wait] 2026-08-21T05:05:56+00:00: seed44 checkpoint not yet complete
4
+ [wait] 2026-08-21T05:06:56+00:00: seed44 checkpoint not yet complete
5
+ [wait] 2026-08-21T05:07:56+00:00: seed44 checkpoint not yet complete
6
+ [wait] 2026-08-21T05:08:56+00:00: seed44 checkpoint not yet complete
7
+ [wait] 2026-08-21T05:09:56+00:00: seed44 checkpoint not yet complete
8
+ [wait] 2026-08-21T05:10:56+00:00: seed44 checkpoint not yet complete
9
+ [wait] 2026-08-21T05:11:56+00:00: seed44 checkpoint not yet complete
10
+ [wait] 2026-08-21T05:12:56+00:00: seed44 checkpoint not yet complete
11
+ [wait] 2026-08-21T05:13:56+00:00: seed44 checkpoint not yet complete
12
+ [wait] 2026-08-21T05:14:56+00:00: seed44 checkpoint not yet complete
13
+ [wait] 2026-08-21T05:15:56+00:00: seed44 checkpoint not yet complete
14
+ [wait] 2026-08-21T05:16:56+00:00: seed44 checkpoint not yet complete
15
+ [wait] 2026-08-21T05:17:56+00:00: seed44 checkpoint not yet complete
16
+ [wait] 2026-08-21T05:18:56+00:00: seed44 checkpoint not yet complete
17
+ [wait] 2026-08-21T05:19:57+00:00: seed44 checkpoint not yet complete
18
+ [wait] 2026-08-21T05:20:57+00:00: seed44 checkpoint not yet complete
19
+ [wait] 2026-08-21T05:21:57+00:00: seed44 checkpoint not yet complete
20
+ [wait] 2026-08-21T05:22:57+00:00: seed44 checkpoint not yet complete
21
+ [wait] 2026-08-21T05:23:57+00:00: seed44 checkpoint not yet complete
22
+ [wait] 2026-08-21T05:24:57+00:00: seed44 checkpoint not yet complete
23
+ [wait] 2026-08-21T05:25:57+00:00: seed44 checkpoint not yet complete
24
+ [wait] 2026-08-21T05:26:57+00:00: seed44 checkpoint not yet complete
25
+ [wait] 2026-08-21T05:27:57+00:00: seed44 checkpoint not yet complete
26
+ [wait] 2026-08-21T05:28:57+00:00: seed44 checkpoint not yet complete
27
+ [wait] 2026-08-21T05:29:57+00:00: seed44 checkpoint not yet complete
28
+ [wait] 2026-08-21T05:30:57+00:00: seed44 checkpoint not yet complete
29
+ [wait] 2026-08-21T05:31:57+00:00: seed44 checkpoint not yet complete
30
+ [wait] 2026-08-21T05:32:57+00:00: seed44 checkpoint not yet complete
31
+ [wait] 2026-08-21T05:33:57+00:00: seed44 checkpoint not yet complete
32
+ [wait] 2026-08-21T05:34:57+00:00: seed44 checkpoint not yet complete
33
+ [wait] 2026-08-21T05:35:57+00:00: seed44 checkpoint not yet complete
34
+ [wait] 2026-08-21T05:36:57+00:00: seed44 checkpoint not yet complete
35
+ [wait] 2026-08-21T05:37:57+00:00: seed44 checkpoint not yet complete
36
+ [wait] 2026-08-21T05:38:57+00:00: seed44 checkpoint not yet complete
37
+ [wait] 2026-08-21T05:39:57+00:00: seed44 checkpoint not yet complete
38
+ [wait] 2026-08-21T05:40:57+00:00: seed44 checkpoint not yet complete
39
+ [wait] 2026-08-21T05:41:57+00:00: seed44 checkpoint not yet complete
40
+ [wait] 2026-08-21T05:42:57+00:00: seed44 checkpoint not yet complete
41
+ [wait] 2026-08-21T05:43:57+00:00: seed44 checkpoint not yet complete
42
+ [wait] 2026-08-21T05:44:57+00:00: seed44 checkpoint not yet complete
43
+ [wait] 2026-08-21T05:45:57+00:00: seed44 checkpoint not yet complete
44
+ [wait] 2026-08-21T05:46:57+00:00: seed44 checkpoint not yet complete
45
+ [wait] 2026-08-21T05:47:57+00:00: seed44 checkpoint not yet complete
46
+ [wait] 2026-08-21T05:48:57+00:00: seed44 checkpoint not yet complete
47
+ [wait] 2026-08-21T05:49:57+00:00: seed44 checkpoint not yet complete
48
+ [wait] 2026-08-21T05:50:57+00:00: seed44 checkpoint not yet complete
49
+ [wait] 2026-08-21T05:51:58+00:00: seed44 checkpoint not yet complete
50
+ [wait] 2026-08-21T05:52:58+00:00: seed44 checkpoint not yet complete
51
+ [wait] 2026-08-21T05:53:58+00:00: seed44 checkpoint not yet complete
52
+ [wait] 2026-08-21T05:54:58+00:00: seed44 checkpoint not yet complete
53
+ [wait] 2026-08-21T05:55:58+00:00: seed44 checkpoint not yet complete
54
+ [wait] 2026-08-21T05:56:58+00:00: seed44 checkpoint not yet complete
55
+ [wait] 2026-08-21T05:57:58+00:00: seed44 checkpoint not yet complete
56
+ [wait] 2026-08-21T05:58:58+00:00: seed44 checkpoint not yet complete
57
+ [wait] 2026-08-21T05:59:58+00:00: seed44 checkpoint not yet complete
58
+ [wait] 2026-08-21T06:00:58+00:00: seed44 checkpoint not yet complete
59
+ [wait] 2026-08-21T06:01:58+00:00: seed44 checkpoint not yet complete
60
+ [wait] 2026-08-21T06:02:58+00:00: seed44 checkpoint not yet complete
61
+ [wait] 2026-08-21T06:03:58+00:00: seed44 checkpoint not yet complete
62
+ [wait] 2026-08-21T06:04:58+00:00: seed44 checkpoint not yet complete
63
+ [wait] 2026-08-21T06:05:58+00:00: seed44 checkpoint not yet complete
64
+ [wait] 2026-08-21T06:06:58+00:00: seed44 checkpoint not yet complete
65
+ [wait] 2026-08-21T06:07:58+00:00: seed44 checkpoint not yet complete
66
+ [wait] 2026-08-21T06:08:58+00:00: seed44 checkpoint not yet complete
67
+ [wait] 2026-08-21T06:09:58+00:00: seed44 checkpoint not yet complete
68
+ [wait] 2026-08-21T06:10:58+00:00: seed44 checkpoint not yet complete
69
+ [wait] 2026-08-21T06:11:58+00:00: seed44 checkpoint not yet complete
70
+ [wait] 2026-08-21T06:12:58+00:00: seed44 checkpoint not yet complete
71
+ [wait] 2026-08-21T06:13:58+00:00: seed44 checkpoint not yet complete
72
+ [wait] 2026-08-21T06:14:58+00:00: seed44 checkpoint not yet complete
73
+ [wait] 2026-08-21T06:15:58+00:00: seed44 checkpoint not yet complete
74
+ [wait] 2026-08-21T06:16:58+00:00: seed44 checkpoint not yet complete
75
+ [wait] 2026-08-21T06:17:58+00:00: seed44 checkpoint not yet complete
76
+ [wait] 2026-08-21T06:18:58+00:00: seed44 checkpoint not yet complete
77
+ [wait] 2026-08-21T06:19:58+00:00: seed44 checkpoint not yet complete
78
+ [wait] 2026-08-21T06:20:58+00:00: seed44 checkpoint not yet complete
79
+ [wait] 2026-08-21T06:21:58+00:00: seed44 checkpoint not yet complete
80
+ [wait] 2026-08-21T06:22:59+00:00: seed44 checkpoint not yet complete
81
+ [wait] 2026-08-21T06:23:59+00:00: seed44 checkpoint not yet complete
82
+ [wait] 2026-08-21T06:24:59+00:00: seed44 checkpoint not yet complete
83
+ [wait] 2026-08-21T06:25:59+00:00: seed44 checkpoint not yet complete
84
+ [wait] 2026-08-21T06:26:59+00:00: seed44 checkpoint not yet complete
85
+ [wait] 2026-08-21T06:27:59+00:00: seed44 checkpoint not yet complete
86
+ [wait] 2026-08-21T06:28:59+00:00: seed44 checkpoint not yet complete
87
+ [wait] 2026-08-21T06:29:59+00:00: seed44 checkpoint not yet complete
88
+ [wait] 2026-08-21T06:30:59+00:00: seed44 checkpoint not yet complete
89
+ [wait] 2026-08-21T06:31:59+00:00: seed44 checkpoint not yet complete
90
+ [wait] 2026-08-21T06:32:59+00:00: seed44 checkpoint not yet complete
91
+ [wait] 2026-08-21T06:33:59+00:00: seed44 checkpoint not yet complete
92
+ [wait] 2026-08-21T06:34:59+00:00: seed44 checkpoint not yet complete
93
+ [wait] 2026-08-21T06:35:59+00:00: seed44 checkpoint not yet complete
94
+ [wait] 2026-08-21T06:36:59+00:00: seed44 checkpoint not yet complete
95
+ [wait] 2026-08-21T06:37:59+00:00: seed44 checkpoint not yet complete
96
+ [wait] 2026-08-21T06:38:59+00:00: seed44 checkpoint not yet complete
97
+ [wait] 2026-08-21T06:39:59+00:00: seed44 checkpoint not yet complete
98
+ [wait] 2026-08-21T06:40:59+00:00: seed44 checkpoint not yet complete
99
+ [wait] 2026-08-21T06:41:59+00:00: seed44 checkpoint not yet complete
100
+ [wait] 2026-08-21T06:42:59+00:00: seed44 checkpoint not yet complete
101
+ [wait] 2026-08-21T06:43:59+00:00: seed44 checkpoint not yet complete
102
+ [wait] 2026-08-21T06:44:59+00:00: seed44 checkpoint not yet complete
103
+ [wait] 2026-08-21T06:45:59+00:00: seed44 checkpoint not yet complete
104
+ [wait] 2026-08-21T06:46:59+00:00: seed44 checkpoint not yet complete
105
+ [wait] 2026-08-21T06:47:59+00:00: seed44 checkpoint not yet complete
106
+ [wait] 2026-08-21T06:48:59+00:00: seed44 checkpoint not yet complete
107
+ [wait] 2026-08-21T06:49:59+00:00: seed44 checkpoint not yet complete
108
+ [wait] 2026-08-21T06:51:00+00:00: seed44 checkpoint not yet complete
109
+ [wait] 2026-08-21T06:52:00+00:00: seed44 checkpoint not yet complete
110
+ [wait] 2026-08-21T06:53:00+00:00: seed44 checkpoint not yet complete
111
+ [wait] 2026-08-21T06:54:00+00:00: seed44 checkpoint not yet complete
112
+ [wait] 2026-08-21T06:55:00+00:00: seed44 checkpoint not yet complete
113
+ [wait] 2026-08-21T06:56:00+00:00: seed44 checkpoint not yet complete
114
+ [wait] 2026-08-21T06:57:00+00:00: seed44 checkpoint not yet complete
115
+ [wait] 2026-08-21T06:58:00+00:00: seed44 checkpoint not yet complete
116
+ [wait] 2026-08-21T06:59:00+00:00: seed44 checkpoint not yet complete
117
+ [wait] 2026-08-21T07:00:00+00:00: seed44 checkpoint not yet complete
118
+ [wait] 2026-08-21T07:01:00+00:00: seed44 checkpoint not yet complete
119
+ [wait] 2026-08-21T07:02:00+00:00: seed44 checkpoint not yet complete
120
+ [wait] 2026-08-21T07:03:00+00:00: seed44 checkpoint not yet complete
121
+ [wait] 2026-08-21T07:04:00+00:00: seed44 checkpoint not yet complete
122
+ [wait] 2026-08-21T07:05:00+00:00: seed44 checkpoint not yet complete
123
+ [wait] 2026-08-21T07:06:00+00:00: seed44 checkpoint not yet complete
124
+ [wait] 2026-08-21T07:07:00+00:00: seed44 checkpoint not yet complete
125
+ [wait] 2026-08-21T07:08:00+00:00: seed44 checkpoint not yet complete
126
+ [wait] 2026-08-21T07:09:00+00:00: seed44 checkpoint not yet complete
127
+ [wait] 2026-08-21T07:10:00+00:00: seed44 checkpoint not yet complete
128
+ [wait] 2026-08-21T07:11:00+00:00: seed44 checkpoint not yet complete
129
+ [wait] 2026-08-21T07:12:00+00:00: seed44 checkpoint not yet complete
130
+ [wait] 2026-08-21T07:13:00+00:00: seed44 checkpoint not yet complete
131
+ [wait] 2026-08-21T07:14:00+00:00: seed44 checkpoint not yet complete
132
+ [wait] 2026-08-21T07:15:00+00:00: seed44 checkpoint not yet complete
133
+ [wait] 2026-08-21T07:16:00+00:00: seed44 checkpoint not yet complete
134
+ [wait] 2026-08-21T07:17:00+00:00: seed44 checkpoint not yet complete
135
+ [wait] 2026-08-21T07:18:00+00:00: seed44 checkpoint not yet complete
136
+ [wait] 2026-08-21T07:19:00+00:00: seed44 checkpoint not yet complete
137
+ [wait] 2026-08-21T07:20:01+00:00: seed44 checkpoint not yet complete
138
+ [wait] 2026-08-21T07:21:01+00:00: seed44 checkpoint not yet complete
139
+ [wait] 2026-08-21T07:22:01+00:00: seed44 checkpoint not yet complete
140
+ [wait] 2026-08-21T07:23:01+00:00: seed44 checkpoint not yet complete
141
+ [wait] 2026-08-21T07:24:01+00:00: seed44 checkpoint not yet complete
142
+ [wait] 2026-08-21T07:25:01+00:00: seed44 checkpoint not yet complete
143
+ [wait] 2026-08-21T07:26:01+00:00: seed44 checkpoint not yet complete
144
+ [wait] 2026-08-21T07:27:01+00:00: seed44 checkpoint not yet complete
145
+ [wait] 2026-08-21T07:28:01+00:00: seed44 checkpoint not yet complete
146
+ [wait] 2026-08-21T07:29:01+00:00: seed44 checkpoint not yet complete
147
+ [wait] 2026-08-21T07:30:01+00:00: seed44 checkpoint not yet complete
148
+ [wait] 2026-08-21T07:31:01+00:00: seed44 checkpoint not yet complete
149
+ [wait] 2026-08-21T07:32:01+00:00: seed44 checkpoint not yet complete
150
+ [wait] 2026-08-21T07:33:01+00:00: seed44 checkpoint not yet complete
151
+ [wait] 2026-08-21T07:34:01+00:00: seed44 checkpoint not yet complete
152
+ [wait] 2026-08-21T07:35:01+00:00: seed44 checkpoint not yet complete
153
+ [wait] 2026-08-21T07:36:01+00:00: seed44 checkpoint not yet complete
154
+ [wait] 2026-08-21T07:37:01+00:00: seed44 checkpoint not yet complete
155
+ [wait] 2026-08-21T07:38:01+00:00: seed44 checkpoint not yet complete
156
+ [wait] 2026-08-21T07:39:01+00:00: seed44 checkpoint not yet complete
157
+ [wait] 2026-08-21T07:40:01+00:00: seed44 checkpoint not yet complete
158
+ [wait] 2026-08-21T07:41:01+00:00: seed44 checkpoint not yet complete
159
+ [wait] 2026-08-21T07:42:01+00:00: seed44 checkpoint not yet complete
160
+ [wait] 2026-08-21T07:43:01+00:00: seed44 checkpoint not yet complete
161
+ [wait] 2026-08-21T07:44:01+00:00: seed44 checkpoint not yet complete
162
+ [wait] 2026-08-21T07:45:01+00:00: seed44 checkpoint not yet complete
163
+ [wait] 2026-08-21T07:46:01+00:00: seed44 checkpoint not yet complete
164
+ [wait] 2026-08-21T07:47:01+00:00: seed44 checkpoint not yet complete
165
+ [wait] 2026-08-21T07:48:01+00:00: seed44 checkpoint not yet complete
166
+ [wait] 2026-08-21T07:49:01+00:00: seed44 checkpoint not yet complete
167
+ [wait] 2026-08-21T07:50:01+00:00: seed44 checkpoint not yet complete
168
+ [wait] 2026-08-21T07:51:01+00:00: seed44 checkpoint not yet complete
169
+ [wait] 2026-08-21T07:52:01+00:00: seed44 checkpoint not yet complete
170
+ [wait] 2026-08-21T07:53:01+00:00: seed44 checkpoint not yet complete
171
+ [wait] 2026-08-21T07:54:01+00:00: seed44 checkpoint not yet complete
172
+ [wait] 2026-08-21T07:55:01+00:00: seed44 checkpoint not yet complete
173
+ [wait] 2026-08-21T07:56:01+00:00: seed44 checkpoint not yet complete
174
+ [wait] 2026-08-21T07:57:01+00:00: seed44 checkpoint not yet complete
175
+ [wait] 2026-08-21T07:58:01+00:00: seed44 checkpoint not yet complete
176
+ [wait] 2026-08-21T07:59:01+00:00: seed44 checkpoint not yet complete
177
+ [wait] 2026-08-21T08:00:01+00:00: seed44 checkpoint not yet complete
178
+ [wait] 2026-08-21T08:01:01+00:00: seed44 checkpoint not yet complete
179
+ [wait] 2026-08-21T08:02:01+00:00: seed44 checkpoint not yet complete
180
+ [wait] 2026-08-21T08:03:01+00:00: seed44 checkpoint not yet complete
181
+ [wait] 2026-08-21T08:04:01+00:00: seed44 checkpoint not yet complete
182
+ [wait] 2026-08-21T08:05:01+00:00: seed44 checkpoint not yet complete
183
+ [wait] 2026-08-21T08:06:01+00:00: seed44 checkpoint not yet complete
184
+ [wait] 2026-08-21T08:07:01+00:00: seed44 checkpoint not yet complete
185
+ [wait] 2026-08-21T08:08:01+00:00: seed44 checkpoint not yet complete
186
+ [wait] 2026-08-21T08:09:01+00:00: seed44 checkpoint not yet complete
187
+ [wait] 2026-08-21T08:10:01+00:00: seed44 checkpoint not yet complete
188
+ [wait] 2026-08-21T08:11:01+00:00: seed44 checkpoint not yet complete
189
+ [wait] 2026-08-21T08:12:01+00:00: seed44 checkpoint not yet complete
190
+ [wait] 2026-08-21T08:13:01+00:00: seed44 checkpoint not yet complete
191
+ [wait] 2026-08-21T08:14:01+00:00: seed44 checkpoint not yet complete
192
+ [wait] 2026-08-21T08:15:01+00:00: seed44 checkpoint not yet complete
193
+ [wait] 2026-08-21T08:16:01+00:00: seed44 checkpoint not yet complete
194
+ [wait] 2026-08-21T08:17:01+00:00: seed44 checkpoint not yet complete
195
+ [wait] 2026-08-21T08:18:01+00:00: seed44 checkpoint not yet complete
196
+ [wait] 2026-08-21T08:19:01+00:00: seed44 checkpoint not yet complete
197
+ [wait] 2026-08-21T08:20:01+00:00: seed44 checkpoint not yet complete
198
+ [wait] 2026-08-21T08:21:01+00:00: seed44 checkpoint not yet complete
199
+ [wait] 2026-08-21T08:22:01+00:00: seed44 checkpoint not yet complete
200
+ [wait] 2026-08-21T08:23:01+00:00: seed44 checkpoint not yet complete
201
+ [wait] 2026-08-21T08:24:01+00:00: seed44 checkpoint not yet complete
202
+ [wait] 2026-08-21T08:25:01+00:00: seed44 checkpoint not yet complete
203
+ [wait] 2026-08-21T08:26:01+00:00: seed44 checkpoint not yet complete
204
+ [wait] 2026-08-21T08:27:01+00:00: seed44 checkpoint not yet complete
205
+ [wait] 2026-08-21T08:28:01+00:00: seed44 checkpoint not yet complete
206
+ [wait] 2026-08-21T08:29:01+00:00: seed44 checkpoint not yet complete
207
+ [wait] 2026-08-21T08:30:01+00:00: seed44 checkpoint not yet complete
208
+ [wait] 2026-08-21T08:31:01+00:00: seed44 checkpoint not yet complete
209
+ [wait] 2026-08-21T08:32:01+00:00: seed44 checkpoint not yet complete
210
+ [wait] 2026-08-21T08:33:01+00:00: seed44 checkpoint not yet complete
211
+ [wait] 2026-08-21T08:34:01+00:00: seed44 checkpoint not yet complete
212
+ [wait] 2026-08-21T08:35:01+00:00: seed44 checkpoint not yet complete
213
+ [wait] 2026-08-21T08:36:01+00:00: seed44 checkpoint not yet complete
214
+ [wait] 2026-08-21T08:37:01+00:00: seed44 checkpoint not yet complete
215
+ [wait] 2026-08-21T08:38:01+00:00: seed44 checkpoint not yet complete
216
+ [wait] 2026-08-21T08:39:01+00:00: seed44 checkpoint not yet complete
217
+ [wait] 2026-08-21T08:40:01+00:00: seed44 checkpoint not yet complete
218
+ [wait] 2026-08-21T08:41:01+00:00: seed44 checkpoint not yet complete
219
+ [wait] 2026-08-21T08:42:01+00:00: seed44 checkpoint not yet complete
220
+ [wait] 2026-08-21T08:43:01+00:00: seed44 checkpoint not yet complete
221
+ [wait] 2026-08-21T08:44:01+00:00: seed44 checkpoint not yet complete
222
+ [wait] 2026-08-21T08:45:01+00:00: seed44 checkpoint not yet complete
223
+ [wait] 2026-08-21T08:46:01+00:00: seed44 checkpoint not yet complete
224
+ [wait] 2026-08-21T08:47:01+00:00: seed44 checkpoint not yet complete
225
+ [wait] 2026-08-21T08:48:02+00:00: seed44 checkpoint not yet complete
226
+ [wait] 2026-08-21T08:49:02+00:00: seed44 checkpoint not yet complete
227
+ [wait] 2026-08-21T08:50:02+00:00: seed44 checkpoint not yet complete
228
+ [wait] 2026-08-21T08:51:02+00:00: seed44 checkpoint not yet complete
229
+ [wait] 2026-08-21T08:52:02+00:00: seed44 checkpoint not yet complete
230
+ [wait] 2026-08-21T08:53:02+00:00: seed44 checkpoint not yet complete
231
+ [wait] 2026-08-21T08:54:02+00:00: seed44 checkpoint not yet complete
232
+ [wait] 2026-08-21T08:55:02+00:00: seed44 checkpoint not yet complete
233
+ [wait] 2026-08-21T08:56:02+00:00: seed44 checkpoint not yet complete
234
+ [wait] 2026-08-21T08:57:02+00:00: seed44 checkpoint not yet complete
235
+ [wait] 2026-08-21T08:58:02+00:00: seed44 checkpoint not yet complete
236
+ [wait] 2026-08-21T08:59:02+00:00: seed44 checkpoint not yet complete
237
+ [wait] 2026-08-21T09:00:02+00:00: seed44 checkpoint not yet complete
238
+ [wait] 2026-08-21T09:01:02+00:00: seed44 checkpoint not yet complete
239
+ [wait] 2026-08-21T09:02:02+00:00: seed44 checkpoint not yet complete
240
+ [wait] 2026-08-21T09:03:02+00:00: seed44 checkpoint not yet complete
241
+ [wait] 2026-08-21T09:04:02+00:00: seed44 checkpoint not yet complete
242
+ [wait] 2026-08-21T09:05:02+00:00: seed44 checkpoint not yet complete
243
+ [wait] 2026-08-21T09:06:02+00:00: seed44 checkpoint not yet complete
244
+ [wait] 2026-08-21T09:07:02+00:00: seed44 checkpoint not yet complete
245
+ [wait] 2026-08-21T09:08:02+00:00: seed44 checkpoint not yet complete
246
+ [wait] 2026-08-21T09:09:02+00:00: seed44 checkpoint not yet complete
247
+ [wait] 2026-08-21T09:10:02+00:00: seed44 checkpoint not yet complete
248
+ [wait] 2026-08-21T09:11:02+00:00: seed44 checkpoint not yet complete
249
+ [wait] 2026-08-21T09:12:02+00:00: seed44 checkpoint not yet complete
250
+ [wait] 2026-08-21T09:13:02+00:00: seed44 checkpoint not yet complete
251
+ [wait] 2026-08-21T09:14:02+00:00: seed44 checkpoint not yet complete
252
+ [wait] 2026-08-21T09:15:02+00:00: seed44 checkpoint not yet complete
253
+ [wait] 2026-08-21T09:16:02+00:00: seed44 checkpoint not yet complete
254
+ [wait] 2026-08-21T09:17:02+00:00: seed44 checkpoint not yet complete
255
+ [wait] 2026-08-21T09:18:02+00:00: seed44 checkpoint not yet complete
256
+ [wait] 2026-08-21T09:19:02+00:00: seed44 checkpoint not yet complete
257
+ [wait] 2026-08-21T09:20:02+00:00: seed44 checkpoint not yet complete
258
+ [wait] 2026-08-21T09:21:02+00:00: seed44 checkpoint not yet complete
259
+ [wait] 2026-08-21T09:22:02+00:00: seed44 checkpoint not yet complete
260
+ [wait] 2026-08-21T09:23:02+00:00: seed44 checkpoint not yet complete
261
+ [wait] 2026-08-21T09:24:02+00:00: seed44 checkpoint not yet complete
262
+ [wait] 2026-08-21T09:25:02+00:00: seed44 checkpoint not yet complete
263
+ [wait] 2026-08-21T09:26:02+00:00: seed44 checkpoint not yet complete
264
+ [wait] 2026-08-21T09:27:02+00:00: seed44 checkpoint not yet complete
265
+ [wait] 2026-08-21T09:28:02+00:00: seed44 checkpoint not yet complete
266
+ [wait] 2026-08-21T09:29:02+00:00: seed44 checkpoint not yet complete
267
+ [wait] 2026-08-21T09:30:02+00:00: seed44 checkpoint not yet complete
268
+ [wait] 2026-08-21T09:31:02+00:00: seed44 checkpoint not yet complete
269
+ [wait] 2026-08-21T09:32:02+00:00: seed44 checkpoint not yet complete
270
+ [wait] 2026-08-21T09:33:02+00:00: seed44 checkpoint not yet complete
271
+ [wait] 2026-08-21T09:34:02+00:00: seed44 checkpoint not yet complete
272
+ [wait] 2026-08-21T09:35:02+00:00: seed44 checkpoint not yet complete
273
+ [wait] 2026-08-21T09:36:02+00:00: seed44 checkpoint not yet complete
274
+ [wait] 2026-08-21T09:37:02+00:00: seed44 checkpoint not yet complete
275
+ [wait] 2026-08-21T09:38:02+00:00: seed44 checkpoint not yet complete
276
+ [wait] 2026-08-21T09:39:02+00:00: seed44 checkpoint not yet complete
277
+ [wait] 2026-08-21T09:40:02+00:00: seed44 checkpoint not yet complete
278
+ [wait] 2026-08-21T09:41:02+00:00: seed44 checkpoint not yet complete
279
+ [wait] 2026-08-21T09:42:02+00:00: seed44 checkpoint not yet complete
280
+ [wait] 2026-08-21T09:43:02+00:00: seed44 checkpoint not yet complete
281
+ [wait] 2026-08-21T09:44:02+00:00: seed44 checkpoint not yet complete
282
+ [wait] 2026-08-21T09:45:02+00:00: seed44 checkpoint not yet complete
283
+ [wait] 2026-08-21T09:46:02+00:00: seed44 checkpoint not yet complete
284
+ [wait] 2026-08-21T09:47:02+00:00: seed44 checkpoint not yet complete
285
+ [wait] 2026-08-21T09:48:02+00:00: seed44 checkpoint not yet complete
286
+ [wait] 2026-08-21T09:49:02+00:00: seed44 checkpoint not yet complete
287
+ [wait] 2026-08-21T09:50:02+00:00: seed44 checkpoint not yet complete
288
+ [wait] 2026-08-21T09:51:02+00:00: seed44 checkpoint not yet complete
289
+ [wait] 2026-08-21T09:52:02+00:00: seed44 checkpoint not yet complete
290
+ [wait] 2026-08-21T09:53:02+00:00: seed44 checkpoint not yet complete
291
+ [wait] 2026-08-21T09:54:02+00:00: seed44 checkpoint not yet complete
292
+ [wait] 2026-08-21T09:55:02+00:00: seed44 checkpoint not yet complete
293
+ [wait] 2026-08-21T09:56:02+00:00: seed44 checkpoint not yet complete
294
+ [wait] 2026-08-21T09:57:02+00:00: seed44 checkpoint not yet complete
295
+ [wait] 2026-08-21T09:58:02+00:00: seed44 checkpoint not yet complete
296
+ [wait] 2026-08-21T09:59:02+00:00: seed44 checkpoint not yet complete
297
+ [wait] 2026-08-21T10:00:02+00:00: seed44 checkpoint not yet complete
298
+ [wait] 2026-08-21T10:01:02+00:00: seed44 checkpoint not yet complete
299
+ [wait] 2026-08-21T10:02:02+00:00: seed44 checkpoint not yet complete
300
+ [wait] 2026-08-21T10:03:02+00:00: seed44 checkpoint not yet complete
301
+ [wait] 2026-08-21T10:04:02+00:00: seed44 checkpoint not yet complete
302
+ [wait] 2026-08-21T10:05:02+00:00: seed44 checkpoint not yet complete
303
+ [wait] 2026-08-21T10:06:02+00:00: seed44 checkpoint not yet complete
304
+ [wait] 2026-08-21T10:07:02+00:00: seed44 checkpoint not yet complete
305
+ [wait] 2026-08-21T10:08:02+00:00: seed44 checkpoint not yet complete
306
+ [wait] 2026-08-21T10:09:02+00:00: seed44 checkpoint not yet complete
307
+ [wait] 2026-08-21T10:10:02+00:00: seed44 checkpoint not yet complete
308
+ [wait] 2026-08-21T10:11:02+00:00: seed44 checkpoint not yet complete
309
+ [wait] 2026-08-21T10:12:02+00:00: seed44 checkpoint not yet complete
310
+ [wait] 2026-08-21T10:13:02+00:00: seed44 checkpoint not yet complete
311
+ [wait] 2026-08-21T10:14:02+00:00: seed44 checkpoint not yet complete
312
+ [wait] 2026-08-21T10:15:02+00:00: seed44 checkpoint not yet complete
313
+ [wait] 2026-08-21T10:16:02+00:00: seed44 checkpoint not yet complete
314
+ [wait] 2026-08-21T10:17:02+00:00: seed44 checkpoint not yet complete
315
+ [wait] 2026-08-21T10:18:02+00:00: seed44 checkpoint not yet complete
316
+ [wait] 2026-08-21T10:19:02+00:00: seed44 checkpoint not yet complete
317
+ [wait] 2026-08-21T10:20:02+00:00: seed44 checkpoint not yet complete
318
+ [wait] 2026-08-21T10:21:02+00:00: seed44 checkpoint not yet complete
319
+ [wait] 2026-08-21T10:22:02+00:00: seed44 checkpoint not yet complete
320
+ [wait] 2026-08-21T10:23:02+00:00: seed44 checkpoint not yet complete
321
+ [wait] 2026-08-21T10:24:02+00:00: seed44 checkpoint not yet complete
322
+ [wait] 2026-08-21T10:25:02+00:00: seed44 checkpoint not yet complete
323
+ [wait] 2026-08-21T10:26:02+00:00: seed44 checkpoint not yet complete
324
+ [wait] 2026-08-21T10:27:02+00:00: seed44 checkpoint not yet complete
325
+ [wait] 2026-08-21T10:28:02+00:00: seed44 checkpoint not yet complete
326
+ [wait] 2026-08-21T10:29:02+00:00: seed44 checkpoint not yet complete
327
+ [wait] 2026-08-21T10:30:02+00:00: seed44 checkpoint not yet complete
328
+ [wait] 2026-08-21T10:31:02+00:00: seed44 checkpoint not yet complete
329
+ [wait] 2026-08-21T10:32:02+00:00: seed44 checkpoint not yet complete
330
+ [wait] 2026-08-21T10:33:02+00:00: seed44 checkpoint not yet complete
331
+ [wait] 2026-08-21T10:34:02+00:00: seed44 checkpoint not yet complete
332
+ [wait] 2026-08-21T10:35:02+00:00: seed44 checkpoint not yet complete
epfl_final/eval_logs/rope45_auto_20260821/watcher.log ADDED
@@ -0,0 +1,235 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [wait] 2026-08-21T05:05:14+00:00: seed45 checkpoint not yet complete
2
+ [wait] 2026-08-21T05:06:15+00:00: seed45 checkpoint not yet complete
3
+ [wait] 2026-08-21T05:07:15+00:00: seed45 checkpoint not yet complete
4
+ [wait] 2026-08-21T05:08:15+00:00: seed45 checkpoint not yet complete
5
+ [wait] 2026-08-21T05:09:15+00:00: seed45 checkpoint not yet complete
6
+ [wait] 2026-08-21T05:10:15+00:00: seed45 checkpoint not yet complete
7
+ [wait] 2026-08-21T05:11:15+00:00: seed45 checkpoint not yet complete
8
+ [wait] 2026-08-21T05:12:15+00:00: seed45 checkpoint not yet complete
9
+ [wait] 2026-08-21T05:13:15+00:00: seed45 checkpoint not yet complete
10
+ [wait] 2026-08-21T05:14:15+00:00: seed45 checkpoint not yet complete
11
+ [wait] 2026-08-21T05:15:15+00:00: seed45 checkpoint not yet complete
12
+ [wait] 2026-08-21T05:16:15+00:00: seed45 checkpoint not yet complete
13
+ [wait] 2026-08-21T05:17:15+00:00: seed45 checkpoint not yet complete
14
+ [wait] 2026-08-21T05:18:15+00:00: seed45 checkpoint not yet complete
15
+ [wait] 2026-08-21T05:19:15+00:00: seed45 checkpoint not yet complete
16
+ [wait] 2026-08-21T05:20:15+00:00: seed45 checkpoint not yet complete
17
+ [wait] 2026-08-21T05:21:15+00:00: seed45 checkpoint not yet complete
18
+ [wait] 2026-08-21T05:22:15+00:00: seed45 checkpoint not yet complete
19
+ [wait] 2026-08-21T05:23:15+00:00: seed45 checkpoint not yet complete
20
+ [wait] 2026-08-21T05:24:15+00:00: seed45 checkpoint not yet complete
21
+ [wait] 2026-08-21T05:25:15+00:00: seed45 checkpoint not yet complete
22
+ [wait] 2026-08-21T05:26:15+00:00: seed45 checkpoint not yet complete
23
+ [wait] 2026-08-21T05:27:15+00:00: seed45 checkpoint not yet complete
24
+ [wait] 2026-08-21T05:28:15+00:00: seed45 checkpoint not yet complete
25
+ [wait] 2026-08-21T05:29:15+00:00: seed45 checkpoint not yet complete
26
+ [wait] 2026-08-21T05:30:16+00:00: seed45 checkpoint not yet complete
27
+ [wait] 2026-08-21T05:31:16+00:00: seed45 checkpoint not yet complete
28
+ [wait] 2026-08-21T05:32:16+00:00: seed45 checkpoint not yet complete
29
+ [wait] 2026-08-21T05:33:16+00:00: seed45 checkpoint not yet complete
30
+ [wait] 2026-08-21T05:34:16+00:00: seed45 checkpoint not yet complete
31
+ [wait] 2026-08-21T05:35:16+00:00: seed45 checkpoint not yet complete
32
+ [wait] 2026-08-21T05:36:16+00:00: seed45 checkpoint not yet complete
33
+ [wait] 2026-08-21T05:37:16+00:00: seed45 checkpoint not yet complete
34
+ [wait] 2026-08-21T05:38:16+00:00: seed45 checkpoint not yet complete
35
+ [wait] 2026-08-21T05:39:16+00:00: seed45 checkpoint not yet complete
36
+ [wait] 2026-08-21T05:40:16+00:00: seed45 checkpoint not yet complete
37
+ [wait] 2026-08-21T05:41:16+00:00: seed45 checkpoint not yet complete
38
+ [wait] 2026-08-21T05:42:16+00:00: seed45 checkpoint not yet complete
39
+ [wait] 2026-08-21T05:43:16+00:00: seed45 checkpoint not yet complete
40
+ [wait] 2026-08-21T05:44:16+00:00: seed45 checkpoint not yet complete
41
+ [wait] 2026-08-21T05:45:16+00:00: seed45 checkpoint not yet complete
42
+ [wait] 2026-08-21T05:46:16+00:00: seed45 checkpoint not yet complete
43
+ [wait] 2026-08-21T05:47:16+00:00: seed45 checkpoint not yet complete
44
+ [wait] 2026-08-21T05:48:16+00:00: seed45 checkpoint not yet complete
45
+ [wait] 2026-08-21T05:49:16+00:00: seed45 checkpoint not yet complete
46
+ [wait] 2026-08-21T05:50:16+00:00: seed45 checkpoint not yet complete
47
+ [wait] 2026-08-21T05:51:16+00:00: seed45 checkpoint not yet complete
48
+ [wait] 2026-08-21T05:52:16+00:00: seed45 checkpoint not yet complete
49
+ [wait] 2026-08-21T05:53:16+00:00: seed45 checkpoint not yet complete
50
+ [wait] 2026-08-21T05:54:16+00:00: seed45 checkpoint not yet complete
51
+ [wait] 2026-08-21T05:55:17+00:00: seed45 checkpoint not yet complete
52
+ [wait] 2026-08-21T05:56:17+00:00: seed45 checkpoint not yet complete
53
+ [wait] 2026-08-21T05:57:17+00:00: seed45 checkpoint not yet complete
54
+ [wait] 2026-08-21T05:58:17+00:00: seed45 checkpoint not yet complete
55
+ [wait] 2026-08-21T05:59:17+00:00: seed45 checkpoint not yet complete
56
+ [wait] 2026-08-21T06:00:17+00:00: seed45 checkpoint not yet complete
57
+ [wait] 2026-08-21T06:01:17+00:00: seed45 checkpoint not yet complete
58
+ [wait] 2026-08-21T06:02:17+00:00: seed45 checkpoint not yet complete
59
+ [wait] 2026-08-21T06:03:17+00:00: seed45 checkpoint not yet complete
60
+ [wait] 2026-08-21T06:04:17+00:00: seed45 checkpoint not yet complete
61
+ [wait] 2026-08-21T06:05:17+00:00: seed45 checkpoint not yet complete
62
+ [wait] 2026-08-21T06:06:17+00:00: seed45 checkpoint not yet complete
63
+ [wait] 2026-08-21T06:07:17+00:00: seed45 checkpoint not yet complete
64
+ [wait] 2026-08-21T06:08:17+00:00: seed45 checkpoint not yet complete
65
+ [wait] 2026-08-21T06:09:17+00:00: seed45 checkpoint not yet complete
66
+ [wait] 2026-08-21T06:10:17+00:00: seed45 checkpoint not yet complete
67
+ [wait] 2026-08-21T06:11:17+00:00: seed45 checkpoint not yet complete
68
+ [wait] 2026-08-21T06:12:17+00:00: seed45 checkpoint not yet complete
69
+ [wait] 2026-08-21T06:13:17+00:00: seed45 checkpoint not yet complete
70
+ [wait] 2026-08-21T06:14:17+00:00: seed45 checkpoint not yet complete
71
+ [wait] 2026-08-21T06:15:17+00:00: seed45 checkpoint not yet complete
72
+ [wait] 2026-08-21T06:16:17+00:00: seed45 checkpoint not yet complete
73
+ [wait] 2026-08-21T06:17:17+00:00: seed45 checkpoint not yet complete
74
+ [wait] 2026-08-21T06:18:17+00:00: seed45 checkpoint not yet complete
75
+ [wait] 2026-08-21T06:19:17+00:00: seed45 checkpoint not yet complete
76
+ [wait] 2026-08-21T06:20:17+00:00: seed45 checkpoint not yet complete
77
+ [wait] 2026-08-21T06:21:18+00:00: seed45 checkpoint not yet complete
78
+ [wait] 2026-08-21T06:22:18+00:00: seed45 checkpoint not yet complete
79
+ [wait] 2026-08-21T06:23:18+00:00: seed45 checkpoint not yet complete
80
+ [wait] 2026-08-21T06:24:18+00:00: seed45 checkpoint not yet complete
81
+ [wait] 2026-08-21T06:25:18+00:00: seed45 checkpoint not yet complete
82
+ [wait] 2026-08-21T06:26:18+00:00: seed45 checkpoint not yet complete
83
+ [wait] 2026-08-21T06:27:18+00:00: seed45 checkpoint not yet complete
84
+ [wait] 2026-08-21T06:28:18+00:00: seed45 checkpoint not yet complete
85
+ [wait] 2026-08-21T06:29:18+00:00: seed45 checkpoint not yet complete
86
+ [wait] 2026-08-21T06:30:18+00:00: seed45 checkpoint not yet complete
87
+ [wait] 2026-08-21T06:31:18+00:00: seed45 checkpoint not yet complete
88
+ [wait] 2026-08-21T06:32:18+00:00: seed45 checkpoint not yet complete
89
+ [wait] 2026-08-21T06:33:18+00:00: seed45 checkpoint not yet complete
90
+ [wait] 2026-08-21T06:34:18+00:00: seed45 checkpoint not yet complete
91
+ [wait] 2026-08-21T06:35:18+00:00: seed45 checkpoint not yet complete
92
+ [wait] 2026-08-21T06:36:18+00:00: seed45 checkpoint not yet complete
93
+ [wait] 2026-08-21T06:37:18+00:00: seed45 checkpoint not yet complete
94
+ [wait] 2026-08-21T06:38:18+00:00: seed45 checkpoint not yet complete
95
+ [wait] 2026-08-21T06:39:18+00:00: seed45 checkpoint not yet complete
96
+ [wait] 2026-08-21T06:40:18+00:00: seed45 checkpoint not yet complete
97
+ [wait] 2026-08-21T06:41:18+00:00: seed45 checkpoint not yet complete
98
+ [wait] 2026-08-21T06:42:18+00:00: seed45 checkpoint not yet complete
99
+ [wait] 2026-08-21T06:43:18+00:00: seed45 checkpoint not yet complete
100
+ [wait] 2026-08-21T06:44:18+00:00: seed45 checkpoint not yet complete
101
+ [wait] 2026-08-21T06:45:19+00:00: seed45 checkpoint not yet complete
102
+ [wait] 2026-08-21T06:46:19+00:00: seed45 checkpoint not yet complete
103
+ [wait] 2026-08-21T06:47:19+00:00: seed45 checkpoint not yet complete
104
+ [wait] 2026-08-21T06:48:19+00:00: seed45 checkpoint not yet complete
105
+ [wait] 2026-08-21T06:49:19+00:00: seed45 checkpoint not yet complete
106
+ [wait] 2026-08-21T06:50:19+00:00: seed45 checkpoint not yet complete
107
+ [wait] 2026-08-21T06:51:19+00:00: seed45 checkpoint not yet complete
108
+ [wait] 2026-08-21T06:52:19+00:00: seed45 checkpoint not yet complete
109
+ [wait] 2026-08-21T06:53:19+00:00: seed45 checkpoint not yet complete
110
+ [wait] 2026-08-21T06:54:19+00:00: seed45 checkpoint not yet complete
111
+ [wait] 2026-08-21T06:55:19+00:00: seed45 checkpoint not yet complete
112
+ [wait] 2026-08-21T06:56:19+00:00: seed45 checkpoint not yet complete
113
+ [wait] 2026-08-21T06:57:19+00:00: seed45 checkpoint not yet complete
114
+ [wait] 2026-08-21T06:58:19+00:00: seed45 checkpoint not yet complete
115
+ [wait] 2026-08-21T06:59:19+00:00: seed45 checkpoint not yet complete
116
+ [wait] 2026-08-21T07:00:19+00:00: seed45 checkpoint not yet complete
117
+ [wait] 2026-08-21T07:01:19+00:00: seed45 checkpoint not yet complete
118
+ [wait] 2026-08-21T07:02:19+00:00: seed45 checkpoint not yet complete
119
+ [wait] 2026-08-21T07:03:19+00:00: seed45 checkpoint not yet complete
120
+ [wait] 2026-08-21T07:04:19+00:00: seed45 checkpoint not yet complete
121
+ [wait] 2026-08-21T07:05:19+00:00: seed45 checkpoint not yet complete
122
+ [wait] 2026-08-21T07:06:19+00:00: seed45 checkpoint not yet complete
123
+ [wait] 2026-08-21T07:07:19+00:00: seed45 checkpoint not yet complete
124
+ [wait] 2026-08-21T07:08:19+00:00: seed45 checkpoint not yet complete
125
+ [wait] 2026-08-21T07:09:19+00:00: seed45 checkpoint not yet complete
126
+ [wait] 2026-08-21T07:10:20+00:00: seed45 checkpoint not yet complete
127
+ [wait] 2026-08-21T07:11:20+00:00: seed45 checkpoint not yet complete
128
+ [wait] 2026-08-21T07:12:20+00:00: seed45 checkpoint not yet complete
129
+ [wait] 2026-08-21T07:13:20+00:00: seed45 checkpoint not yet complete
130
+ [wait] 2026-08-21T07:14:20+00:00: seed45 checkpoint not yet complete
131
+ [wait] 2026-08-21T07:15:20+00:00: seed45 checkpoint not yet complete
132
+ [wait] 2026-08-21T07:16:20+00:00: seed45 checkpoint not yet complete
133
+ [wait] 2026-08-21T07:17:20+00:00: seed45 checkpoint not yet complete
134
+ [wait] 2026-08-21T07:18:20+00:00: seed45 checkpoint not yet complete
135
+ [wait] 2026-08-21T07:19:20+00:00: seed45 checkpoint not yet complete
136
+ [wait] 2026-08-21T07:20:20+00:00: seed45 checkpoint not yet complete
137
+ [wait] 2026-08-21T07:21:20+00:00: seed45 checkpoint not yet complete
138
+ [wait] 2026-08-21T07:22:20+00:00: seed45 checkpoint not yet complete
139
+ [wait] 2026-08-21T07:23:20+00:00: seed45 checkpoint not yet complete
140
+ [wait] 2026-08-21T07:24:20+00:00: seed45 checkpoint not yet complete
141
+ [wait] 2026-08-21T07:25:20+00:00: seed45 checkpoint not yet complete
142
+ [wait] 2026-08-21T07:26:20+00:00: seed45 checkpoint not yet complete
143
+ [wait] 2026-08-21T07:27:20+00:00: seed45 checkpoint not yet complete
144
+ [wait] 2026-08-21T07:28:20+00:00: seed45 checkpoint not yet complete
145
+ [wait] 2026-08-21T07:29:20+00:00: seed45 checkpoint not yet complete
146
+ [start] 2026-08-21T07:30:20+00:00: seed45 matched eval
147
+ [gpu2] 2026-08-21T07:30:20+00:00 8192 arxiv
148
+ [gpu3] 2026-08-21T07:30:20+00:00 8192 fineweb
149
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
150
+ warnings.warn(
151
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
152
+ warnings.warn(
153
+ Checkpoint activations: False
154
+ Checkpoint activations: False
155
+ Checkpoint activations: False
156
+ Checkpoint activations: False
157
+ Checkpoint activations: False
158
+ Checkpoint activations: False
159
+ Checkpoint activations: False
160
+ Checkpoint activations: False
161
+ Checkpoint activations: False
162
+ Checkpoint activations: False
163
+ [gpu2] 2026-08-21T07:37:08+00:00 8192 pg19
164
+ [gpu3] 2026-08-21T07:37:09+00:00 12288 arxiv
165
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
166
+ warnings.warn(
167
+ Checkpoint activations: False
168
+ Checkpoint activations: False
169
+ Checkpoint activations: False
170
+ Checkpoint activations: False
171
+ Checkpoint activations: False
172
+ [gpu2] 2026-08-21T07:39:26+00:00 12288 fineweb
173
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
174
+ warnings.warn(
175
+ Checkpoint activations: False
176
+ Checkpoint activations: False
177
+ Checkpoint activations: False
178
+ Checkpoint activations: False
179
+ Checkpoint activations: False
180
+ Checkpoint activations: False
181
+ Checkpoint activations: False
182
+ Checkpoint activations: False
183
+ Checkpoint activations: False
184
+ Checkpoint activations: False
185
+ [gpu3] 2026-08-21T07:43:41+00:00 12288 pg19
186
+ [gpu2] 2026-08-21T07:43:43+00:00 16384 arxiv
187
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
188
+ warnings.warn(
189
+ Checkpoint activations: False
190
+ Checkpoint activations: False
191
+ Checkpoint activations: False
192
+ Checkpoint activations: False
193
+ Checkpoint activations: False
194
+ [gpu3] 2026-08-21T07:45:58+00:00 16384 fineweb
195
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
196
+ warnings.warn(
197
+ Checkpoint activations: False
198
+ Checkpoint activations: False
199
+ Checkpoint activations: False
200
+ Checkpoint activations: False
201
+ Checkpoint activations: False
202
+ Checkpoint activations: False
203
+ Checkpoint activations: False
204
+ Checkpoint activations: False
205
+ Checkpoint activations: False
206
+ Checkpoint activations: False
207
+ [gpu2] 2026-08-21T07:49:50+00:00 16384 pg19
208
+ [gpu3] 2026-08-21T07:49:54+00:00 24576 arxiv
209
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
210
+ warnings.warn(
211
+ Checkpoint activations: False
212
+ Checkpoint activations: False
213
+ Checkpoint activations: False
214
+ Checkpoint activations: False
215
+ Checkpoint activations: False
216
+ [gpu2] 2026-08-21T07:52:07+00:00 24576 fineweb
217
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
218
+ warnings.warn(
219
+ Checkpoint activations: False
220
+ Checkpoint activations: False
221
+ Checkpoint activations: False
222
+ Checkpoint activations: False
223
+ Checkpoint activations: False
224
+ Checkpoint activations: False
225
+ Checkpoint activations: False
226
+ Checkpoint activations: False
227
+ Checkpoint activations: False
228
+ Checkpoint activations: False
229
+ [gpu3] 2026-08-21T07:56:45+00:00 24576 pg19
230
+ Checkpoint activations: False
231
+ Checkpoint activations: False
232
+ Checkpoint activations: False
233
+ Checkpoint activations: False
234
+ Checkpoint activations: False
235
+ [done] 2026-08-21T07:59:05+00:00: seed45 all matched evals complete
epfl_final/eval_logs/rope_seed46_matched_20260820.log ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [start] 8192_rope_seed46_arxiv (1/12)
2
+ [done] 8192_rope_seed46_arxiv (3.6m)
3
+ [start] 8192_rope_seed46_fineweb (2/12)
4
+ [done] 8192_rope_seed46_fineweb (1.5m)
5
+ [start] 8192_rope_seed46_pg19 (3/12)
6
+ [done] 8192_rope_seed46_pg19 (1.5m)
7
+ [start] 12288_rope_seed46_arxiv (4/12)
8
+ [done] 12288_rope_seed46_arxiv (3.8m)
9
+ [start] 12288_rope_seed46_fineweb (5/12)
10
+ [done] 12288_rope_seed46_fineweb (1.5m)
11
+ [start] 12288_rope_seed46_pg19 (6/12)
12
+ [done] 12288_rope_seed46_pg19 (1.5m)
13
+ [start] 16384_rope_seed46_arxiv (7/12)
14
+ [done] 16384_rope_seed46_arxiv (3.6m)
15
+ [start] 16384_rope_seed46_fineweb (8/12)
16
+ [done] 16384_rope_seed46_fineweb (1.6m)
17
+ [start] 16384_rope_seed46_pg19 (9/12)
18
+ [done] 16384_rope_seed46_pg19 (1.5m)
19
+ [start] 24576_rope_seed46_arxiv (10/12)
20
+ [done] 24576_rope_seed46_arxiv (3.9m)
21
+ [start] 24576_rope_seed46_fineweb (11/12)
22
+ [done] 24576_rope_seed46_fineweb (1.6m)
23
+ [start] 24576_rope_seed46_pg19 (12/12)
24
+ [done] 24576_rope_seed46_pg19 (1.5m)
epfl_final/eval_logs/seed47_dense_matched_watcher_20260820.log ADDED
@@ -0,0 +1,27 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [watch] dense seed47: waiting for training PID and final checkpoint
2
+ [ready] dense seed47 checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_seed47/run_20260820_031247/checkpoint-44000
3
+ [start] 8192_dense_seed47_arxiv
4
+ [done] 8192_dense_seed47_arxiv
5
+ [start] 8192_dense_seed47_fineweb
6
+ [done] 8192_dense_seed47_fineweb
7
+ [start] 8192_dense_seed47_pg19
8
+ [done] 8192_dense_seed47_pg19
9
+ [start] 12288_dense_seed47_arxiv
10
+ [done] 12288_dense_seed47_arxiv
11
+ [start] 12288_dense_seed47_fineweb
12
+ [done] 12288_dense_seed47_fineweb
13
+ [start] 12288_dense_seed47_pg19
14
+ [done] 12288_dense_seed47_pg19
15
+ [start] 16384_dense_seed47_arxiv
16
+ [done] 16384_dense_seed47_arxiv
17
+ [start] 16384_dense_seed47_fineweb
18
+ [done] 16384_dense_seed47_fineweb
19
+ [start] 16384_dense_seed47_pg19
20
+ [done] 16384_dense_seed47_pg19
21
+ [start] 24576_dense_seed47_arxiv
22
+ [done] 24576_dense_seed47_arxiv
23
+ [start] 24576_dense_seed47_fineweb
24
+ [done] 24576_dense_seed47_fineweb
25
+ [start] 24576_dense_seed47_pg19
26
+ [done] 24576_dense_seed47_pg19
27
+ [complete] dense seed47 matched grid
epfl_final/eval_logs/seed47_mixed_matched_watcher_20260820.log ADDED
@@ -0,0 +1,26 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [ready] mixed seed47 checkpoint: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_seed47/run_20260820_031812/checkpoint-44000
2
+ [start] 8192_mixed_seed47_arxiv
3
+ [done] 8192_mixed_seed47_arxiv
4
+ [start] 8192_mixed_seed47_fineweb
5
+ [done] 8192_mixed_seed47_fineweb
6
+ [start] 8192_mixed_seed47_pg19
7
+ [done] 8192_mixed_seed47_pg19
8
+ [start] 12288_mixed_seed47_arxiv
9
+ [done] 12288_mixed_seed47_arxiv
10
+ [start] 12288_mixed_seed47_fineweb
11
+ [done] 12288_mixed_seed47_fineweb
12
+ [start] 12288_mixed_seed47_pg19
13
+ [done] 12288_mixed_seed47_pg19
14
+ [start] 16384_mixed_seed47_arxiv
15
+ [done] 16384_mixed_seed47_arxiv
16
+ [start] 16384_mixed_seed47_fineweb
17
+ [done] 16384_mixed_seed47_fineweb
18
+ [start] 16384_mixed_seed47_pg19
19
+ [done] 16384_mixed_seed47_pg19
20
+ [start] 24576_mixed_seed47_arxiv
21
+ [done] 24576_mixed_seed47_arxiv
22
+ [start] 24576_mixed_seed47_fineweb
23
+ [done] 24576_mixed_seed47_fineweb
24
+ [start] 24576_mixed_seed47_pg19
25
+ [done] 24576_mixed_seed47_pg19
26
+ [complete] mixed seed47 matched grid
epfl_final/eval_logs/seed48_dense_auto_20260821/watcher.log ADDED
@@ -0,0 +1,222 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [wait] 2026-08-21T12:53:23+00:00: dense48 final checkpoint
2
+ [wait] 2026-08-21T12:54:23+00:00: dense48 final checkpoint
3
+ [wait] 2026-08-21T12:55:24+00:00: dense48 final checkpoint
4
+ [wait] 2026-08-21T12:56:24+00:00: dense48 final checkpoint
5
+ [wait] 2026-08-21T12:57:24+00:00: dense48 final checkpoint
6
+ [wait] 2026-08-21T12:58:24+00:00: dense48 final checkpoint
7
+ [wait] 2026-08-21T12:59:24+00:00: dense48 final checkpoint
8
+ [wait] 2026-08-21T13:00:24+00:00: dense48 final checkpoint
9
+ [wait] 2026-08-21T13:01:24+00:00: dense48 final checkpoint
10
+ [wait] 2026-08-21T13:02:24+00:00: dense48 final checkpoint
11
+ [wait] 2026-08-21T13:03:24+00:00: dense48 final checkpoint
12
+ [wait] 2026-08-21T13:04:24+00:00: dense48 final checkpoint
13
+ [wait] 2026-08-21T13:05:24+00:00: dense48 final checkpoint
14
+ [wait] 2026-08-21T13:06:24+00:00: dense48 final checkpoint
15
+ [wait] 2026-08-21T13:07:24+00:00: dense48 final checkpoint
16
+ [wait] 2026-08-21T13:08:24+00:00: dense48 final checkpoint
17
+ [wait] 2026-08-21T13:09:24+00:00: dense48 final checkpoint
18
+ [wait] 2026-08-21T13:10:24+00:00: dense48 final checkpoint
19
+ [wait] 2026-08-21T13:11:24+00:00: dense48 final checkpoint
20
+ [wait] 2026-08-21T13:12:24+00:00: dense48 final checkpoint
21
+ [wait] 2026-08-21T13:13:24+00:00: dense48 final checkpoint
22
+ [wait] 2026-08-21T13:14:24+00:00: dense48 final checkpoint
23
+ [wait] 2026-08-21T13:15:24+00:00: dense48 final checkpoint
24
+ [wait] 2026-08-21T13:16:24+00:00: dense48 final checkpoint
25
+ [wait] 2026-08-21T13:17:24+00:00: dense48 final checkpoint
26
+ [wait] 2026-08-21T13:18:24+00:00: dense48 final checkpoint
27
+ [wait] 2026-08-21T13:19:24+00:00: dense48 final checkpoint
28
+ [wait] 2026-08-21T13:20:24+00:00: dense48 final checkpoint
29
+ [wait] 2026-08-21T13:21:24+00:00: dense48 final checkpoint
30
+ [wait] 2026-08-21T13:22:24+00:00: dense48 final checkpoint
31
+ [wait] 2026-08-21T13:23:24+00:00: dense48 final checkpoint
32
+ [wait] 2026-08-21T13:24:24+00:00: dense48 final checkpoint
33
+ [wait] 2026-08-21T13:25:24+00:00: dense48 final checkpoint
34
+ [wait] 2026-08-21T13:26:24+00:00: dense48 final checkpoint
35
+ [wait] 2026-08-21T13:27:24+00:00: dense48 final checkpoint
36
+ [wait] 2026-08-21T13:28:24+00:00: dense48 final checkpoint
37
+ [wait] 2026-08-21T13:29:24+00:00: dense48 final checkpoint
38
+ [wait] 2026-08-21T13:30:24+00:00: dense48 final checkpoint
39
+ [wait] 2026-08-21T13:31:24+00:00: dense48 final checkpoint
40
+ [wait] 2026-08-21T13:32:24+00:00: dense48 final checkpoint
41
+ [wait] 2026-08-21T13:33:24+00:00: dense48 final checkpoint
42
+ [wait] 2026-08-21T13:34:24+00:00: dense48 final checkpoint
43
+ [wait] 2026-08-21T13:35:24+00:00: dense48 final checkpoint
44
+ [wait] 2026-08-21T13:36:24+00:00: dense48 final checkpoint
45
+ [wait] 2026-08-21T13:37:24+00:00: dense48 final checkpoint
46
+ [wait] 2026-08-21T13:38:24+00:00: dense48 final checkpoint
47
+ [wait] 2026-08-21T13:39:24+00:00: dense48 final checkpoint
48
+ [wait] 2026-08-21T13:40:24+00:00: dense48 final checkpoint
49
+ [wait] 2026-08-21T13:41:24+00:00: dense48 final checkpoint
50
+ [wait] 2026-08-21T13:42:24+00:00: dense48 final checkpoint
51
+ [wait] 2026-08-21T13:43:24+00:00: dense48 final checkpoint
52
+ [wait] 2026-08-21T13:44:24+00:00: dense48 final checkpoint
53
+ [wait] 2026-08-21T13:45:24+00:00: dense48 final checkpoint
54
+ [wait] 2026-08-21T13:46:24+00:00: dense48 final checkpoint
55
+ [wait] 2026-08-21T13:47:24+00:00: dense48 final checkpoint
56
+ [wait] 2026-08-21T13:48:24+00:00: dense48 final checkpoint
57
+ [wait] 2026-08-21T13:49:24+00:00: dense48 final checkpoint
58
+ [wait] 2026-08-21T13:50:24+00:00: dense48 final checkpoint
59
+ [wait] 2026-08-21T13:51:24+00:00: dense48 final checkpoint
60
+ [wait] 2026-08-21T13:52:24+00:00: dense48 final checkpoint
61
+ [wait] 2026-08-21T13:53:24+00:00: dense48 final checkpoint
62
+ [wait] 2026-08-21T13:54:24+00:00: dense48 final checkpoint
63
+ [wait] 2026-08-21T13:55:24+00:00: dense48 final checkpoint
64
+ [wait] 2026-08-21T13:56:24+00:00: dense48 final checkpoint
65
+ [wait] 2026-08-21T13:57:24+00:00: dense48 final checkpoint
66
+ [wait] 2026-08-21T13:58:24+00:00: dense48 final checkpoint
67
+ [wait] 2026-08-21T13:59:24+00:00: dense48 final checkpoint
68
+ [wait] 2026-08-21T14:00:24+00:00: dense48 final checkpoint
69
+ [wait] 2026-08-21T14:01:24+00:00: dense48 final checkpoint
70
+ [wait] 2026-08-21T14:02:24+00:00: dense48 final checkpoint
71
+ [wait] 2026-08-21T14:03:24+00:00: dense48 final checkpoint
72
+ [wait] 2026-08-21T14:04:24+00:00: dense48 final checkpoint
73
+ [wait] 2026-08-21T14:05:24+00:00: dense48 final checkpoint
74
+ [wait] 2026-08-21T14:06:24+00:00: dense48 final checkpoint
75
+ [wait] 2026-08-21T14:07:24+00:00: dense48 final checkpoint
76
+ [wait] 2026-08-21T14:08:24+00:00: dense48 final checkpoint
77
+ [wait] 2026-08-21T14:09:24+00:00: dense48 final checkpoint
78
+ [wait] 2026-08-21T14:10:24+00:00: dense48 final checkpoint
79
+ [wait] 2026-08-21T14:11:24+00:00: dense48 final checkpoint
80
+ [wait] 2026-08-21T14:12:24+00:00: dense48 final checkpoint
81
+ [wait] 2026-08-21T14:13:24+00:00: dense48 final checkpoint
82
+ [wait] 2026-08-21T14:14:24+00:00: dense48 final checkpoint
83
+ [wait] 2026-08-21T14:15:24+00:00: dense48 final checkpoint
84
+ [wait] 2026-08-21T14:16:24+00:00: dense48 final checkpoint
85
+ [wait] 2026-08-21T14:17:24+00:00: dense48 final checkpoint
86
+ [wait] 2026-08-21T14:18:24+00:00: dense48 final checkpoint
87
+ [wait] 2026-08-21T14:19:24+00:00: dense48 final checkpoint
88
+ [wait] 2026-08-21T14:20:24+00:00: dense48 final checkpoint
89
+ [wait] 2026-08-21T14:21:24+00:00: dense48 final checkpoint
90
+ [wait] 2026-08-21T14:22:24+00:00: dense48 final checkpoint
91
+ [wait] 2026-08-21T14:23:24+00:00: dense48 final checkpoint
92
+ [wait] 2026-08-21T14:24:24+00:00: dense48 final checkpoint
93
+ [wait] 2026-08-21T14:25:24+00:00: dense48 final checkpoint
94
+ [wait] 2026-08-21T14:26:24+00:00: dense48 final checkpoint
95
+ [wait] 2026-08-21T14:27:24+00:00: dense48 final checkpoint
96
+ [wait] 2026-08-21T14:28:24+00:00: dense48 final checkpoint
97
+ [wait] 2026-08-21T14:29:24+00:00: dense48 final checkpoint
98
+ [wait] 2026-08-21T14:30:24+00:00: dense48 final checkpoint
99
+ [wait] 2026-08-21T14:31:25+00:00: dense48 final checkpoint
100
+ [wait] 2026-08-21T14:32:25+00:00: dense48 final checkpoint
101
+ [wait] 2026-08-21T14:33:25+00:00: dense48 final checkpoint
102
+ [wait] 2026-08-21T14:34:25+00:00: dense48 final checkpoint
103
+ [wait] 2026-08-21T14:35:25+00:00: dense48 final checkpoint
104
+ [wait] 2026-08-21T14:36:25+00:00: dense48 final checkpoint
105
+ [wait] 2026-08-21T14:37:25+00:00: dense48 final checkpoint
106
+ [wait] 2026-08-21T14:38:25+00:00: dense48 final checkpoint
107
+ [wait] 2026-08-21T14:39:25+00:00: dense48 final checkpoint
108
+ [wait] 2026-08-21T14:40:25+00:00: dense48 final checkpoint
109
+ [wait] 2026-08-21T14:41:25+00:00: dense48 final checkpoint
110
+ [wait] 2026-08-21T14:42:25+00:00: dense48 final checkpoint
111
+ [wait] 2026-08-21T14:43:25+00:00: dense48 final checkpoint
112
+ [wait] 2026-08-21T14:44:25+00:00: dense48 final checkpoint
113
+ [wait] 2026-08-21T14:45:25+00:00: dense48 final checkpoint
114
+ [wait] 2026-08-21T14:46:25+00:00: dense48 final checkpoint
115
+ [wait] 2026-08-21T14:47:25+00:00: dense48 final checkpoint
116
+ [wait] 2026-08-21T14:48:25+00:00: dense48 final checkpoint
117
+ [wait] 2026-08-21T14:49:25+00:00: dense48 final checkpoint
118
+ [wait] 2026-08-21T14:50:25+00:00: dense48 final checkpoint
119
+ [wait] 2026-08-21T14:51:25+00:00: dense48 final checkpoint
120
+ [wait] 2026-08-21T14:52:25+00:00: dense48 final checkpoint
121
+ [wait] 2026-08-21T14:53:25+00:00: dense48 final checkpoint
122
+ [wait] 2026-08-21T14:54:25+00:00: dense48 final checkpoint
123
+ [wait] 2026-08-21T14:55:25+00:00: dense48 final checkpoint
124
+ [wait] 2026-08-21T14:56:25+00:00: dense48 final checkpoint
125
+ [wait] 2026-08-21T14:57:25+00:00: dense48 final checkpoint
126
+ [wait] 2026-08-21T14:58:25+00:00: dense48 final checkpoint
127
+ [wait] 2026-08-21T14:59:25+00:00: dense48 final checkpoint
128
+ [wait] 2026-08-21T15:00:25+00:00: dense48 final checkpoint
129
+ [start] 2026-08-21T15:01:25+00:00: dense48 matched eval from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_dense_seed48/run_20260821_043824/checkpoint-44000
130
+ [gpu0] 2026-08-21T15:01:25+00:00 8192 arxiv
131
+ [gpu3] 2026-08-21T15:01:25+00:00 12288 arxiv
132
+ [gpu2] 2026-08-21T15:01:25+00:00 8192 pg19
133
+ [gpu1] 2026-08-21T15:01:25+00:00 8192 fineweb
134
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
135
+ warnings.warn(
136
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
137
+ warnings.warn(
138
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
139
+ warnings.warn(
140
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
141
+ warnings.warn(
142
+ Checkpoint activations: False
143
+ Checkpoint activations: False
144
+ Checkpoint activations: False
145
+ Checkpoint activations: False
146
+ Checkpoint activations: False
147
+ Checkpoint activations: False
148
+ Checkpoint activations: False
149
+ Checkpoint activations: False
150
+ Checkpoint activations: False
151
+ Checkpoint activations: False
152
+ Checkpoint activations: False
153
+ Checkpoint activations: False
154
+ Checkpoint activations: False
155
+ Checkpoint activations: False
156
+ Checkpoint activations: False
157
+ [gpu1] 2026-08-21T15:05:11+00:00 12288 pg19
158
+ [gpu2] 2026-08-21T15:05:12+00:00 16384 arxiv
159
+ [gpu0] 2026-08-21T15:05:17+00:00 12288 fineweb
160
+ Checkpoint activations: False
161
+ Checkpoint activations: False
162
+ Checkpoint activations: False
163
+ Checkpoint activations: False
164
+ Checkpoint activations: False
165
+ [gpu3] 2026-08-21T15:05:31+00:00 16384 fineweb
166
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
167
+ warnings.warn(
168
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
169
+ warnings.warn(
170
+ Checkpoint activations: False
171
+ Checkpoint activations: False
172
+ Checkpoint activations: False
173
+ Checkpoint activations: False
174
+ Checkpoint activations: False
175
+ Checkpoint activations: False
176
+ Checkpoint activations: False
177
+ Checkpoint activations: False
178
+ Checkpoint activations: False
179
+ Checkpoint activations: False
180
+ [gpu1] 2026-08-21T15:06:51+00:00 24576 arxiv
181
+ [gpu0] 2026-08-21T15:06:51+00:00 16384 pg19
182
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
183
+ warnings.warn(
184
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
185
+ warnings.warn(
186
+ Checkpoint activations: False
187
+ Checkpoint activations: False
188
+ Checkpoint activations: False
189
+ Checkpoint activations: False
190
+ Checkpoint activations: False
191
+ Checkpoint activations: False
192
+ Checkpoint activations: False
193
+ Checkpoint activations: False
194
+ Checkpoint activations: False
195
+ Checkpoint activations: False
196
+ Checkpoint activations: False
197
+ Checkpoint activations: False
198
+ Checkpoint activations: False
199
+ Checkpoint activations: False
200
+ Checkpoint activations: False
201
+ [gpu2] 2026-08-21T15:09:03+00:00 24576 fineweb
202
+ [gpu3] 2026-08-21T15:09:04+00:00 24576 pg19
203
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
204
+ warnings.warn(
205
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
206
+ warnings.warn(
207
+ Checkpoint activations: False
208
+ Checkpoint activations: False
209
+ Checkpoint activations: False
210
+ Checkpoint activations: False
211
+ Checkpoint activations: False
212
+ Checkpoint activations: False
213
+ Checkpoint activations: False
214
+ Checkpoint activations: False
215
+ Checkpoint activations: False
216
+ Checkpoint activations: False
217
+ Checkpoint activations: False
218
+ Checkpoint activations: False
219
+ Checkpoint activations: False
220
+ Checkpoint activations: False
221
+ Checkpoint activations: False
222
+ [done] 2026-08-21T15:11:04+00:00: dense48 eval complete
epfl_final/eval_logs/seed48_mixed_auto_20260821/watcher.log ADDED
@@ -0,0 +1,164 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [wait] 2026-08-21T12:52:32+00:00: mixed48 final checkpoint
2
+ [wait] 2026-08-21T12:53:32+00:00: mixed48 final checkpoint
3
+ [wait] 2026-08-21T12:54:32+00:00: mixed48 final checkpoint
4
+ [wait] 2026-08-21T12:55:32+00:00: mixed48 final checkpoint
5
+ [wait] 2026-08-21T12:56:32+00:00: mixed48 final checkpoint
6
+ [wait] 2026-08-21T12:57:32+00:00: mixed48 final checkpoint
7
+ [wait] 2026-08-21T12:58:32+00:00: mixed48 final checkpoint
8
+ [wait] 2026-08-21T12:59:32+00:00: mixed48 final checkpoint
9
+ [wait] 2026-08-21T13:00:32+00:00: mixed48 final checkpoint
10
+ [wait] 2026-08-21T13:01:33+00:00: mixed48 final checkpoint
11
+ [wait] 2026-08-21T13:02:33+00:00: mixed48 final checkpoint
12
+ [wait] 2026-08-21T13:03:33+00:00: mixed48 final checkpoint
13
+ [wait] 2026-08-21T13:04:33+00:00: mixed48 final checkpoint
14
+ [wait] 2026-08-21T13:05:33+00:00: mixed48 final checkpoint
15
+ [wait] 2026-08-21T13:06:33+00:00: mixed48 final checkpoint
16
+ [wait] 2026-08-21T13:07:33+00:00: mixed48 final checkpoint
17
+ [wait] 2026-08-21T13:08:33+00:00: mixed48 final checkpoint
18
+ [wait] 2026-08-21T13:09:33+00:00: mixed48 final checkpoint
19
+ [wait] 2026-08-21T13:10:33+00:00: mixed48 final checkpoint
20
+ [wait] 2026-08-21T13:11:33+00:00: mixed48 final checkpoint
21
+ [wait] 2026-08-21T13:12:33+00:00: mixed48 final checkpoint
22
+ [wait] 2026-08-21T13:13:33+00:00: mixed48 final checkpoint
23
+ [wait] 2026-08-21T13:14:33+00:00: mixed48 final checkpoint
24
+ [wait] 2026-08-21T13:15:33+00:00: mixed48 final checkpoint
25
+ [wait] 2026-08-21T13:16:33+00:00: mixed48 final checkpoint
26
+ [wait] 2026-08-21T13:17:33+00:00: mixed48 final checkpoint
27
+ [wait] 2026-08-21T13:18:33+00:00: mixed48 final checkpoint
28
+ [wait] 2026-08-21T13:19:33+00:00: mixed48 final checkpoint
29
+ [wait] 2026-08-21T13:20:33+00:00: mixed48 final checkpoint
30
+ [wait] 2026-08-21T13:21:33+00:00: mixed48 final checkpoint
31
+ [wait] 2026-08-21T13:22:33+00:00: mixed48 final checkpoint
32
+ [wait] 2026-08-21T13:23:33+00:00: mixed48 final checkpoint
33
+ [wait] 2026-08-21T13:24:33+00:00: mixed48 final checkpoint
34
+ [wait] 2026-08-21T13:25:33+00:00: mixed48 final checkpoint
35
+ [wait] 2026-08-21T13:26:33+00:00: mixed48 final checkpoint
36
+ [wait] 2026-08-21T13:27:33+00:00: mixed48 final checkpoint
37
+ [wait] 2026-08-21T13:28:33+00:00: mixed48 final checkpoint
38
+ [wait] 2026-08-21T13:29:33+00:00: mixed48 final checkpoint
39
+ [wait] 2026-08-21T13:30:33+00:00: mixed48 final checkpoint
40
+ [wait] 2026-08-21T13:31:33+00:00: mixed48 final checkpoint
41
+ [wait] 2026-08-21T13:32:33+00:00: mixed48 final checkpoint
42
+ [wait] 2026-08-21T13:33:33+00:00: mixed48 final checkpoint
43
+ [wait] 2026-08-21T13:34:33+00:00: mixed48 final checkpoint
44
+ [wait] 2026-08-21T13:35:33+00:00: mixed48 final checkpoint
45
+ [wait] 2026-08-21T13:36:33+00:00: mixed48 final checkpoint
46
+ [wait] 2026-08-21T13:37:33+00:00: mixed48 final checkpoint
47
+ [wait] 2026-08-21T13:38:33+00:00: mixed48 final checkpoint
48
+ [wait] 2026-08-21T13:39:33+00:00: mixed48 final checkpoint
49
+ [wait] 2026-08-21T13:40:33+00:00: mixed48 final checkpoint
50
+ [wait] 2026-08-21T13:41:33+00:00: mixed48 final checkpoint
51
+ [wait] 2026-08-21T13:42:33+00:00: mixed48 final checkpoint
52
+ [wait] 2026-08-21T13:43:33+00:00: mixed48 final checkpoint
53
+ [wait] 2026-08-21T13:44:33+00:00: mixed48 final checkpoint
54
+ [wait] 2026-08-21T13:45:33+00:00: mixed48 final checkpoint
55
+ [wait] 2026-08-21T13:46:33+00:00: mixed48 final checkpoint
56
+ [wait] 2026-08-21T13:47:33+00:00: mixed48 final checkpoint
57
+ [wait] 2026-08-21T13:48:33+00:00: mixed48 final checkpoint
58
+ [wait] 2026-08-21T13:49:33+00:00: mixed48 final checkpoint
59
+ [wait] 2026-08-21T13:50:33+00:00: mixed48 final checkpoint
60
+ [wait] 2026-08-21T13:51:33+00:00: mixed48 final checkpoint
61
+ [wait] 2026-08-21T13:52:33+00:00: mixed48 final checkpoint
62
+ [wait] 2026-08-21T13:53:33+00:00: mixed48 final checkpoint
63
+ [wait] 2026-08-21T13:54:33+00:00: mixed48 final checkpoint
64
+ [wait] 2026-08-21T13:55:33+00:00: mixed48 final checkpoint
65
+ [wait] 2026-08-21T13:56:33+00:00: mixed48 final checkpoint
66
+ [wait] 2026-08-21T13:57:33+00:00: mixed48 final checkpoint
67
+ [wait] 2026-08-21T13:58:33+00:00: mixed48 final checkpoint
68
+ [wait] 2026-08-21T13:59:33+00:00: mixed48 final checkpoint
69
+ [wait] 2026-08-21T14:00:33+00:00: mixed48 final checkpoint
70
+ [wait] 2026-08-21T14:01:34+00:00: mixed48 final checkpoint
71
+ [wait] 2026-08-21T14:02:34+00:00: mixed48 final checkpoint
72
+ [wait] 2026-08-21T14:03:34+00:00: mixed48 final checkpoint
73
+ [wait] 2026-08-21T14:04:34+00:00: mixed48 final checkpoint
74
+ [wait] 2026-08-21T14:05:34+00:00: mixed48 final checkpoint
75
+ [start] 2026-08-21T14:06:34+00:00: mixed48 matched eval from /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_seed48/run_20260821_050923/checkpoint-44000
76
+ [gpu0] 2026-08-21T14:06:34+00:00 8192 arxiv
77
+ [gpu1] 2026-08-21T14:06:34+00:00 8192 fineweb
78
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
79
+ warnings.warn(
80
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
81
+ warnings.warn(
82
+ Checkpoint activations: False
83
+ Checkpoint activations: False
84
+ Checkpoint activations: False
85
+ Checkpoint activations: False
86
+ Checkpoint activations: False
87
+ Checkpoint activations: False
88
+ Checkpoint activations: False
89
+ Checkpoint activations: False
90
+ Checkpoint activations: False
91
+ Checkpoint activations: False
92
+ [gpu1] 2026-08-21T14:10:12+00:00 12288 arxiv
93
+ [gpu0] 2026-08-21T14:10:12+00:00 8192 pg19
94
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
95
+ warnings.warn(
96
+ Checkpoint activations: False
97
+ Checkpoint activations: False
98
+ Checkpoint activations: False
99
+ Checkpoint activations: False
100
+ Checkpoint activations: False
101
+ [gpu0] 2026-08-21T14:11:46+00:00 12288 fineweb
102
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
103
+ warnings.warn(
104
+ Checkpoint activations: False
105
+ Checkpoint activations: False
106
+ Checkpoint activations: False
107
+ Checkpoint activations: False
108
+ Checkpoint activations: False
109
+ [gpu1] 2026-08-21T14:14:05+00:00 12288 pg19
110
+ Checkpoint activations: False
111
+ Checkpoint activations: False
112
+ Checkpoint activations: False
113
+ Checkpoint activations: False
114
+ Checkpoint activations: False
115
+ [gpu0] 2026-08-21T14:14:11+00:00 16384 arxiv
116
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
117
+ warnings.warn(
118
+ Checkpoint activations: False
119
+ Checkpoint activations: False
120
+ Checkpoint activations: False
121
+ Checkpoint activations: False
122
+ Checkpoint activations: False
123
+ [gpu1] 2026-08-21T14:15:43+00:00 16384 fineweb
124
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
125
+ warnings.warn(
126
+ Checkpoint activations: False
127
+ Checkpoint activations: False
128
+ Checkpoint activations: False
129
+ Checkpoint activations: False
130
+ Checkpoint activations: False
131
+ [gpu0] 2026-08-21T14:17:51+00:00 16384 pg19
132
+ Checkpoint activations: False
133
+ Checkpoint activations: False
134
+ Checkpoint activations: False
135
+ Checkpoint activations: False
136
+ Checkpoint activations: False
137
+ [gpu1] 2026-08-21T14:17:56+00:00 24576 arxiv
138
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
139
+ warnings.warn(
140
+ Checkpoint activations: False
141
+ Checkpoint activations: False
142
+ Checkpoint activations: False
143
+ Checkpoint activations: False
144
+ Checkpoint activations: False
145
+ [gpu0] 2026-08-21T14:19:26+00:00 24576 fineweb
146
+ /mloscratch/homes/navasard/conda/lib/python3.13/site-packages/torch/_inductor/compile_fx.py:322: UserWarning: TensorFloat32 tensor cores for float32 matrix multiplication available but not enabled. Consider setting `torch.set_float32_matmul_precision('high')` for better performance.
147
+ warnings.warn(
148
+ Checkpoint activations: False
149
+ Checkpoint activations: False
150
+ Checkpoint activations: False
151
+ Checkpoint activations: False
152
+ Checkpoint activations: False
153
+ Checkpoint activations: False
154
+ Checkpoint activations: False
155
+ Checkpoint activations: False
156
+ Checkpoint activations: False
157
+ Checkpoint activations: False
158
+ [gpu1] 2026-08-21T14:21:53+00:00 24576 pg19
159
+ Checkpoint activations: False
160
+ Checkpoint activations: False
161
+ Checkpoint activations: False
162
+ Checkpoint activations: False
163
+ Checkpoint activations: False
164
+ [done] 2026-08-21T14:23:36+00:00: mixed48 eval complete
epfl_final/train_logs/p1_scheduled_a100_seed47_20260822/l8k365m_resampled_nrb35_seed47.log ADDED
The diff for this file is too large to render. See raw diff