Hugging Face's logo

rubentium
/
sparse-checkpoints

Model card Files
xet
 Community
rubentium commited on
Commit
83d26d0
·
verified ·
1 Parent(s): 6e2e655

EPFL final checkpoints and W&B history 551-600 of 1463

Browse files
This view is limited to 50 files because it contains too many changes.   See raw diff
Files changed (50) hide show
  1. .gitattributes +7 -0
  2. epfl_final/SparseAtt/wandb/run-20260726_073805-33yw8345/files/config.yaml +192 -0
  3. epfl_final/SparseAtt/wandb/run-20260726_073805-33yw8345/files/wandb-summary.json +1 -0
  4. epfl_final/SparseAtt/wandb/run-20260726_073805-33yw8345/run-33yw8345.wandb +3 -0
  5. epfl_final/SparseAtt/wandb/run-20260730_174705-h5tf7yly/files/output.log +13 -0
  6. epfl_final/SparseAtt/wandb/run-20260730_174705-h5tf7yly/files/requirements.txt +142 -0
  7. epfl_final/SparseAtt/wandb/run-20260730_174705-h5tf7yly/files/wandb-metadata.json +79 -0
  8. epfl_final/SparseAtt/wandb/run-20260730_174705-h5tf7yly/logs/debug-internal.log +161 -0
  9. epfl_final/SparseAtt/wandb/run-20260730_174705-h5tf7yly/logs/debug.log +19 -0
  10. epfl_final/SparseAtt/wandb/run-20260730_174705-h5tf7yly/run-h5tf7yly.wandb +3 -0
  11. epfl_final/SparseAtt/wandb/run-20260730_180915-0a2yphds/files/config.yaml +188 -0
  12. epfl_final/SparseAtt/wandb/run-20260730_180915-0a2yphds/files/output.log +788 -0
  13. epfl_final/SparseAtt/wandb/run-20260730_180915-0a2yphds/files/requirements.txt +142 -0
  14. epfl_final/SparseAtt/wandb/run-20260730_180915-0a2yphds/files/wandb-metadata.json +65 -0
  15. epfl_final/SparseAtt/wandb/run-20260730_180915-0a2yphds/files/wandb-summary.json +1 -0
  16. epfl_final/SparseAtt/wandb/run-20260730_180915-0a2yphds/logs/debug-internal.log +0 -0
  17. epfl_final/SparseAtt/wandb/run-20260730_180915-0a2yphds/logs/debug.log +24 -0
  18. epfl_final/SparseAtt/wandb/run-20260730_180915-0a2yphds/run-0a2yphds.wandb +3 -0
  19. epfl_final/SparseAtt/wandb/run-20260731_001110-6wmcorvj/files/config.yaml +187 -0
  20. epfl_final/SparseAtt/wandb/run-20260731_001110-6wmcorvj/files/output.log +976 -0
  21. epfl_final/SparseAtt/wandb/run-20260731_001110-6wmcorvj/files/requirements.txt +142 -0
  22. epfl_final/SparseAtt/wandb/run-20260731_001110-6wmcorvj/files/wandb-metadata.json +65 -0
  23. epfl_final/SparseAtt/wandb/run-20260731_001110-6wmcorvj/files/wandb-summary.json +1 -0
  24. epfl_final/SparseAtt/wandb/run-20260731_001110-6wmcorvj/logs/debug-internal.log +0 -0
  25. epfl_final/SparseAtt/wandb/run-20260731_001110-6wmcorvj/logs/debug.log +24 -0
  26. epfl_final/SparseAtt/wandb/run-20260731_001110-6wmcorvj/run-6wmcorvj.wandb +3 -0
  27. epfl_final/SparseAtt/wandb/run-20260731_001110-gl67rqc0/files/config.yaml +187 -0
  28. epfl_final/SparseAtt/wandb/run-20260731_001110-gl67rqc0/files/output.log +976 -0
  29. epfl_final/SparseAtt/wandb/run-20260731_001110-gl67rqc0/files/requirements.txt +142 -0
  30. epfl_final/SparseAtt/wandb/run-20260731_001110-gl67rqc0/files/wandb-metadata.json +65 -0
  31. epfl_final/SparseAtt/wandb/run-20260731_001110-gl67rqc0/files/wandb-summary.json +1 -0
  32. epfl_final/SparseAtt/wandb/run-20260731_001110-gl67rqc0/logs/debug-internal.log +0 -0
  33. epfl_final/SparseAtt/wandb/run-20260731_001110-gl67rqc0/logs/debug.log +24 -0
  34. epfl_final/SparseAtt/wandb/run-20260731_001110-gl67rqc0/run-gl67rqc0.wandb +3 -0
  35. epfl_final/SparseAtt/wandb/run-20260731_001110-rdj7s9cl/files/config.yaml +187 -0
  36. epfl_final/SparseAtt/wandb/run-20260731_001110-rdj7s9cl/files/output.log +976 -0
  37. epfl_final/SparseAtt/wandb/run-20260731_001110-rdj7s9cl/files/requirements.txt +142 -0
  38. epfl_final/SparseAtt/wandb/run-20260731_001110-rdj7s9cl/files/wandb-metadata.json +65 -0
  39. epfl_final/SparseAtt/wandb/run-20260731_001110-rdj7s9cl/files/wandb-summary.json +1 -0
  40. epfl_final/SparseAtt/wandb/run-20260731_001110-rdj7s9cl/logs/debug-internal.log +0 -0
  41. epfl_final/SparseAtt/wandb/run-20260731_001110-rdj7s9cl/logs/debug.log +24 -0
  42. epfl_final/SparseAtt/wandb/run-20260731_001110-rdj7s9cl/run-rdj7s9cl.wandb +3 -0
  43. epfl_final/SparseAtt/wandb/run-20260731_001110-w1uk4jj8/files/config.yaml +187 -0
  44. epfl_final/SparseAtt/wandb/run-20260731_001110-w1uk4jj8/files/output.log +976 -0
  45. epfl_final/SparseAtt/wandb/run-20260731_001110-w1uk4jj8/files/requirements.txt +142 -0
  46. epfl_final/SparseAtt/wandb/run-20260731_001110-w1uk4jj8/files/wandb-metadata.json +65 -0
  47. epfl_final/SparseAtt/wandb/run-20260731_001110-w1uk4jj8/files/wandb-summary.json +1 -0
  48. epfl_final/SparseAtt/wandb/run-20260731_001110-w1uk4jj8/logs/debug-internal.log +0 -0
  49. epfl_final/SparseAtt/wandb/run-20260731_001110-w1uk4jj8/logs/debug.log +24 -0
  50. epfl_final/SparseAtt/wandb/run-20260731_001110-w1uk4jj8/run-w1uk4jj8.wandb +3 -0
.gitattributes CHANGED
@@ -67,3 +67,10 @@ epfl_final/SparseAtt/wandb/run-20260711_014922-hn868f80/run-hn868f80.wandb filte
67
  epfl_final/SparseAtt/wandb/run-20260711_075132-cd66e13p/run-cd66e13p.wandb filter=lfs diff=lfs merge=lfs -text
68
  epfl_final/SparseAtt/wandb/run-20260711_080025-9blizq3m/run-9blizq3m.wandb filter=lfs diff=lfs merge=lfs -text
69
  epfl_final/SparseAtt/wandb/run-20260713_031902-9blizq3m/run-9blizq3m.wandb filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
 
67
  epfl_final/SparseAtt/wandb/run-20260711_075132-cd66e13p/run-cd66e13p.wandb filter=lfs diff=lfs merge=lfs -text
68
  epfl_final/SparseAtt/wandb/run-20260711_080025-9blizq3m/run-9blizq3m.wandb filter=lfs diff=lfs merge=lfs -text
69
  epfl_final/SparseAtt/wandb/run-20260713_031902-9blizq3m/run-9blizq3m.wandb filter=lfs diff=lfs merge=lfs -text
70
+ epfl_final/SparseAtt/wandb/run-20260726_073805-33yw8345/run-33yw8345.wandb filter=lfs diff=lfs merge=lfs -text
71
+ epfl_final/SparseAtt/wandb/run-20260730_174705-h5tf7yly/run-h5tf7yly.wandb filter=lfs diff=lfs merge=lfs -text
72
+ epfl_final/SparseAtt/wandb/run-20260730_180915-0a2yphds/run-0a2yphds.wandb filter=lfs diff=lfs merge=lfs -text
73
+ epfl_final/SparseAtt/wandb/run-20260731_001110-6wmcorvj/run-6wmcorvj.wandb filter=lfs diff=lfs merge=lfs -text
74
+ epfl_final/SparseAtt/wandb/run-20260731_001110-gl67rqc0/run-gl67rqc0.wandb filter=lfs diff=lfs merge=lfs -text
75
+ epfl_final/SparseAtt/wandb/run-20260731_001110-w1uk4jj8/run-w1uk4jj8.wandb filter=lfs diff=lfs merge=lfs -text
76
+ epfl_final/SparseAtt/wandb/run-20260731_001110-rdj7s9cl/run-rdj7s9cl.wandb filter=lfs diff=lfs merge=lfs -text
epfl_final/SparseAtt/wandb/run-20260726_073805-33yw8345/files/config.yaml ADDED
@@ -0,0 +1,192 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ _wandb:
2
+ value:
3
+ cli_version: 0.26.1
4
+ e:
5
+ azps7d1jv0uw4lrmtl92nlgaedih8j0b:
6
+ args:
7
+ - --config
8
+ - configs/train_config_1mask_perlayerfix_resume_3gpu.json
9
+ codePath: train_fixed_resume.py
10
+ codePathLocal: train_fixed_resume.py
11
+ cpu_count: 64
12
+ cpu_count_logical: 128
13
+ cudaVersion: "13.0"
14
+ disk:
15
+ /:
16
+ total: "7679361679360"
17
+ used: "5455843303424"
18
+ email: ruben.navasardyan@mail.utoronto.ca
19
+ executable: /mloscratch/homes/navasard/conda/bin/python3.13
20
+ git:
21
+ commit: 88298c7a418622530ca5426b22b1e5ebd10ca0cb
22
+ remote: https://github.com/rubentium/TokenPrune.git
23
+ gpu: NVIDIA A100-SXM4-80GB
24
+ gpu_count: 4
25
+ gpu_nvidia:
26
+ - architecture: Ampere
27
+ cudaCores: 6912
28
+ memoryTotal: "85899345920"
29
+ name: NVIDIA A100-SXM4-80GB
30
+ uuid: GPU-943525a1-9808-8c31-17d9-c4180bb03955
31
+ - architecture: Ampere
32
+ cudaCores: 6912
33
+ memoryTotal: "85899345920"
34
+ name: NVIDIA A100-SXM4-80GB
35
+ uuid: GPU-b4caa2bd-09d3-3702-9fd1-293525d8e08c
36
+ - architecture: Ampere
37
+ cudaCores: 6912
38
+ memoryTotal: "85899345920"
39
+ name: NVIDIA A100-SXM4-80GB
40
+ uuid: GPU-46161407-c19c-cdae-c140-0963b35a3047
41
+ - architecture: Ampere
42
+ cudaCores: 6912
43
+ memoryTotal: "85899345920"
44
+ name: NVIDIA A100-SXM4-80GB
45
+ uuid: GPU-f1d949cd-9bad-cd83-2a2b-ed36b22f13a6
46
+ host: a100-workspace-4-gpu-0-1
47
+ memory:
48
+ total: "1081884934144"
49
+ os: Linux-6.8.0-124-generic-x86_64-with-glibc2.35
50
+ program: /mloscratch/homes/navasard/sparse/SparseAtt/train_fixed_resume.py
51
+ python: CPython 3.13.13
52
+ root: /mloscratch/homes/navasard/sparse/SparseAtt
53
+ startedAt: "2026-07-26T07:38:05.450183Z"
54
+ writerId: azps7d1jv0uw4lrmtl92nlgaedih8j0b
55
+ m: []
56
+ python_version: 3.13.13
57
+ t:
58
+ "1":
59
+ - 1
60
+ - 5
61
+ - 11
62
+ - 49
63
+ - 51
64
+ - 53
65
+ - 71
66
+ "2":
67
+ - 1
68
+ - 5
69
+ - 11
70
+ - 49
71
+ - 51
72
+ - 53
73
+ - 71
74
+ "3":
75
+ - 2
76
+ - 5
77
+ - 13
78
+ - 14
79
+ - 15
80
+ - 16
81
+ - 61
82
+ - 62
83
+ "4": 3.13.13
84
+ "5": 0.26.1
85
+ "6": 5.7.0
86
+ "12": 0.26.1
87
+ "13": linux-x86_64
88
+ attention_type:
89
+ value: sparse_stochastic
90
+ checkpoint_activations:
91
+ value: false
92
+ dense_mode:
93
+ value: false
94
+ dense_step_intervals:
95
+ value: null
96
+ eval_batches:
97
+ value: 50
98
+ eval_steps:
99
+ value: 250
100
+ force_new_run:
101
+ value: false
102
+ gradient_accumulation_steps:
103
+ value: 10
104
+ head_dim:
105
+ value: 128
106
+ hidden_size:
107
+ value: 1536
108
+ init_from_scratch:
109
+ value: true
110
+ intermediate_size:
111
+ value: 6144
112
+ learning_rate:
113
+ value: 0.0005
114
+ logging_steps:
115
+ value: 10
116
+ lr_scheduler:
117
+ value: cosine
118
+ max_grad_norm:
119
+ value: 1
120
+ max_seq_length:
121
+ value: 8192
122
+ max_steps:
123
+ value: 50462
124
+ mixed_precision:
125
+ value: bf16
126
+ model_dtype:
127
+ value: bfloat16
128
+ model_path:
129
+ value: /mloscratch/homes/navasard/qwen3_06b_base
130
+ num_attention_heads:
131
+ value: 12
132
+ num_epochs:
133
+ value: 1
134
+ num_hidden_layers:
135
+ value: 28
136
+ num_key_value_heads:
137
+ value: 4
138
+ num_workers:
139
+ value: 8
140
+ output_dir:
141
+ value: /mloscratch/homes/navasard/sparse/checkpoints/run_20260618_175928
142
+ per_device_batch_size:
143
+ value: 2
144
+ preserve_checkpoint_lr:
145
+ value: true
146
+ resume_from_checkpoint:
147
+ value: /mloscratch/homes/navasard/sparse/checkpoints/run_20260618_175928/checkpoint-47000
148
+ resume_micro_step_override:
149
+ value: null
150
+ save_steps:
151
+ value: 1000
152
+ seed:
153
+ value: 42
154
+ sparse_block_size:
155
+ value: 128
156
+ sparse_layer_indices:
157
+ value: null
158
+ sparse_mask_cache_dir:
159
+ value: null
160
+ sparse_mask_pool_size:
161
+ value: 1
162
+ sparse_mask_seed:
163
+ value: 0
164
+ sparse_num_random_blocks:
165
+ value: 31
166
+ sparse_val_dense_compare:
167
+ value: false
168
+ strategy:
169
+ value: ddp
170
+ train_path:
171
+ value: /mloscratch/homes/navasard/data/fineweb_large/tokenized
172
+ val_path:
173
+ value: null
174
+ verbose:
175
+ value: true
176
+ wandb_enabled:
177
+ value: true
178
+ wandb_name:
179
+ value: sparse 1mask perlayer-fix
180
+ wandb_project:
181
+ value: stochastic_sparsity
182
+ wandb_tags:
183
+ value:
184
+ - qwen3
185
+ - 1.2b
186
+ - sparse_stochastic
187
+ - 1mask
188
+ - perlayer-fix
189
+ warmup_steps:
190
+ value: 750
191
+ weight_decay:
192
+ value: 0.01
epfl_final/SparseAtt/wandb/run-20260726_073805-33yw8345/files/wandb-summary.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"speed/ms_per_step":8313.785982131958,"train/loss":2.51198446393013,"eval/dense_interval_start":-1,"_runtime":183612.607918825,"_timestamp":1.78508096025559e+09,"eval/uses_dense_attention":0,"train/learning_rate":1.9968584286100823e-12,"eval/perplexity":12.597447824073448,"eval/uses_sparse_attention":1,"speed/samples_per_sec":7.216928620601058,"train/global_step":50460,"eval/loss":2.533494239795708,"speed/steps_per_sec":0.12028214367668429,"eval/dense_interval_end":-1,"_step":50460,"train/epoch":0.9999603662161627,"eval/dense_interval_active":0,"_wandb":{"runtime":183612}}
epfl_final/SparseAtt/wandb/run-20260726_073805-33yw8345/run-33yw8345.wandb ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c75ba0ab40176de7832834b5d786406a6faf9f53fccc0a85ab58fb0ed8c938d9
3
+ size 4714798
epfl_final/SparseAtt/wandb/run-20260730_174705-h5tf7yly/files/output.log ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Starting training...
2
+ Validation at step 30000: Loss = 2.0523, Perplexity = 7.79
3
+ Switching attention mode for step 30001: dense (dense step interval)
4
+ Step 30010/37465 | Loss: 1.8418 | LR: 4.92e-05 | 55161.7ms/step | 0.4 samples/s
5
+ Step 30020/37465 | Loss: 1.9770 | LR: 4.90e-05 | 6304.0ms/step | 3.2 samples/s
6
+ Step 30030/37465 | Loss: 2.0794 | LR: 4.89e-05 | 6306.0ms/step | 3.2 samples/s
7
+ Step 30040/37465 | Loss: 1.8654 | LR: 4.88e-05 | 6314.8ms/step | 3.2 samples/s
8
+ Step 30050/37465 | Loss: 1.9967 | LR: 4.87e-05 | 6317.9ms/step | 3.2 samples/s
9
+ Step 30060/37465 | Loss: 1.9703 | LR: 4.85e-05 | 6317.5ms/step | 3.2 samples/s
10
+ Step 30070/37465 | Loss: 2.1245 | LR: 4.84e-05 | 6329.5ms/step | 3.2 samples/s
11
+ Step 30080/37465 | Loss: 2.1234 | LR: 4.83e-05 | 6316.6ms/step | 3.2 samples/s
12
+ Step 30090/37465 | Loss: 2.1249 | LR: 4.81e-05 | 6312.4ms/step | 3.2 samples/s
13
+ Step 30100/37465 | Loss: 1.9543 | LR: 4.80e-05 | 6318.3ms/step | 3.2 samples/s
epfl_final/SparseAtt/wandb/run-20260730_174705-h5tf7yly/files/requirements.txt ADDED
@@ -0,0 +1,142 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ pip==26.0.1
2
+ nvidia-cusparselt-cu13==0.8.0
3
+ mpmath==1.3.0
4
+ cuda-toolkit==13.0.2
5
+ zstandard==0.25.0
6
+ xxhash==3.7.0
7
+ urllib3==2.6.3
8
+ typing_extensions==4.15.0
9
+ triton==3.6.0
10
+ tqdm==4.67.3
11
+ torchinfo==1.8.0
12
+ sympy==1.14.0
13
+ smmap==5.0.3
14
+ six==1.17.0
15
+ shellingham==1.5.4
16
+ setuptools==81.0.0
17
+ safetensors==0.7.0
18
+ regex==2026.4.4
19
+ PyYAML==6.0.3
20
+ Pygments==2.20.0
21
+ pyarrow==24.0.0
22
+ psutil==7.2.2
23
+ protobuf==7.34.1
24
+ propcache==0.4.1
25
+ platformdirs==4.9.6
26
+ packaging==26.2
27
+ nvidia-nvtx==13.0.85
28
+ nvidia-nvshmem-cu13==3.4.5
29
+ nvidia-nvjitlink==13.0.88
30
+ nvidia-nccl-cu13==2.28.9
31
+ nvidia-curand==10.4.0.35
32
+ nvidia-cufile==1.15.1.6
33
+ nvidia-cuda-runtime==13.0.96
34
+ nvidia-cuda-nvrtc==13.0.88
35
+ nvidia-cuda-cupti==13.0.85
36
+ nvidia-cublas==13.1.0.3
37
+ numpy==2.4.4
38
+ networkx==3.6.1
39
+ multidict==6.7.1
40
+ mdurl==0.1.2
41
+ MarkupSafe==3.0.3
42
+ idna==3.13
43
+ hf-xet==1.4.3
44
+ h11==0.16.0
45
+ fsspec==2026.2.0
46
+ frozenlist==1.8.0
47
+ filelock==3.29.0
48
+ dill==0.4.1
49
+ cuda-pathfinder==1.5.4
50
+ click==8.3.3
51
+ charset-normalizer==3.4.7
52
+ certifi==2026.4.22
53
+ attrs==26.1.0
54
+ annotated-types==0.7.0
55
+ annotated-doc==0.0.4
56
+ aiohappyeyeballs==2.6.1
57
+ yarl==1.23.0
58
+ typing-inspection==0.4.2
59
+ sentry-sdk==2.58.0
60
+ requests==2.33.1
61
+ python-dateutil==2.9.0.post0
62
+ pydantic_core==2.46.3
63
+ nvidia-cusparse==12.6.3.3
64
+ nvidia-cufft==12.0.0.61
65
+ nvidia-cudnn-cu13==9.19.0.56
66
+ multiprocess==0.70.19
67
+ markdown-it-py==4.0.0
68
+ Jinja2==3.1.6
69
+ httpcore==1.0.9
70
+ gitdb==4.0.12
71
+ cuda-bindings==13.2.0
72
+ anyio==4.13.0
73
+ aiosignal==1.4.0
74
+ rich==15.0.0
75
+ pydantic==2.13.3
76
+ pandas==3.0.2
77
+ nvidia-cusolver==12.0.4.66
78
+ httpx==0.28.1
79
+ GitPython==3.1.49
80
+ aiohttp==3.13.5
81
+ wandb==0.26.1
82
+ typer==0.25.1
83
+ torch==2.11.0
84
+ huggingface_hub==1.13.0
85
+ tokenizers==0.22.2
86
+ datasets==4.8.5
87
+ accelerate==1.13.0
88
+ transformers==5.7.0
89
+ word2number==1.1
90
+ sqlitedict==2.1.0
91
+ pytz==2026.2
92
+ threadpoolctl==3.6.0
93
+ tcolorpy==0.1.7
94
+ tabulate==0.10.0
95
+ scipy==1.17.1
96
+ portalocker==3.2.0
97
+ pathvalidate==3.3.1
98
+ narwhals==2.22.1
99
+ more-itertools==11.1.0
100
+ lxml==6.1.1
101
+ joblib==1.5.3
102
+ colorama==0.4.6
103
+ chardet==6.0.0.post1
104
+ absl-py==2.4.0
105
+ scikit-learn==1.9.0
106
+ sacrebleu==2.6.0
107
+ nltk==3.9.4
108
+ mbstrdecoder==1.1.5
109
+ typepy==1.3.5
110
+ rouge_score==0.1.2
111
+ DataProperty==1.1.1
112
+ tabledata==1.3.5
113
+ evaluate==0.4.6
114
+ pytablewriter==1.2.1
115
+ lm_eval==0.4.12
116
+ pluggy==1.6.0
117
+ iniconfig==2.3.0
118
+ pytest==9.1.0
119
+ py-spy==0.4.2
120
+ pillow==12.2.0
121
+ tifffile==2026.6.1
122
+ imagecodecs==2026.6.26
123
+ zipp==4.1.0
124
+ importlib_metadata==9.0.0
125
+ derivative==0.6.3
126
+ pysindy==2.1.0
127
+ pyparsing==3.3.2
128
+ kiwisolver==1.5.0
129
+ fonttools==4.63.0
130
+ cycler==0.12.1
131
+ contourpy==1.3.3
132
+ matplotlib==3.11.0
133
+ sortedcontainers==2.4.0
134
+ wcwidth==0.8.2
135
+ sigfig==1.3.19
136
+ importlib_resources==7.1.0
137
+ enum-tools==0.13.0
138
+ cramjam==2.11.0
139
+ python-snappy==0.7.3
140
+ compact-json==1.8.2
141
+ numbers-parser==4.18.5
142
+ entmax==1.3
epfl_final/SparseAtt/wandb/run-20260730_174705-h5tf7yly/files/wandb-metadata.json ADDED
@@ -0,0 +1,79 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "os": "Linux-6.8.0-124-generic-x86_64-with-glibc2.35",
3
+ "python": "CPython 3.13.13",
4
+ "startedAt": "2026-07-30T17:47:05.626664Z",
5
+ "args": [
6
+ "--config",
7
+ "/mloscratch/homes/navasard/sparse/SparseAtt/configs/native30k_1p2b_mixed_to_dense_20pct_4gpu_h100_20260730.json"
8
+ ],
9
+ "program": "/mloscratch/homes/navasard/sparse/SparseAtt/train_dense_to_mixed_exact.py",
10
+ "codePath": "train_dense_to_mixed_exact.py",
11
+ "codePathLocal": "train_dense_to_mixed_exact.py",
12
+ "git": {
13
+ "remote": "https://github.com/rubentium/TokenPrune.git",
14
+ "commit": "88298c7a418622530ca5426b22b1e5ebd10ca0cb"
15
+ },
16
+ "email": "ruben.navasardyan@mail.utoronto.ca",
17
+ "root": "/mloscratch/homes/navasard/sparse/SparseAtt",
18
+ "host": "h100-train-6-gpu-0-25",
19
+ "executable": "/mloscratch/homes/navasard/conda/bin/python3.13",
20
+ "cpu_count": 96,
21
+ "cpu_count_logical": 192,
22
+ "gpu": "NVIDIA H100 80GB HBM3",
23
+ "gpu_count": 6,
24
+ "disk": {
25
+ "/": {
26
+ "total": "9374931091456",
27
+ "used": "6688636911616"
28
+ }
29
+ },
30
+ "memory": {
31
+ "total": "1623005732864"
32
+ },
33
+ "gpu_nvidia": [
34
+ {
35
+ "name": "NVIDIA H100 80GB HBM3",
36
+ "memoryTotal": "85520809984",
37
+ "cudaCores": 16896,
38
+ "architecture": "Hopper",
39
+ "uuid": "GPU-c9ff8a50-9103-3e73-0066-6e7c0568cad4"
40
+ },
41
+ {
42
+ "name": "NVIDIA H100 80GB HBM3",
43
+ "memoryTotal": "85520809984",
44
+ "cudaCores": 16896,
45
+ "architecture": "Hopper",
46
+ "uuid": "GPU-ac14b2f5-a0c5-12c4-9fa0-46c3aa9e75c9"
47
+ },
48
+ {
49
+ "name": "NVIDIA H100 80GB HBM3",
50
+ "memoryTotal": "85520809984",
51
+ "cudaCores": 16896,
52
+ "architecture": "Hopper",
53
+ "uuid": "GPU-e8ce0e9a-801b-03d2-c898-c0cc7d746d53"
54
+ },
55
+ {
56
+ "name": "NVIDIA H100 80GB HBM3",
57
+ "memoryTotal": "85520809984",
58
+ "cudaCores": 16896,
59
+ "architecture": "Hopper",
60
+ "uuid": "GPU-8b16a485-aa97-1545-8625-887d6079c32c"
61
+ },
62
+ {
63
+ "name": "NVIDIA H100 80GB HBM3",
64
+ "memoryTotal": "85520809984",
65
+ "cudaCores": 16896,
66
+ "architecture": "Hopper",
67
+ "uuid": "GPU-b817d241-acce-e5dd-a749-3447239c5207"
68
+ },
69
+ {
70
+ "name": "NVIDIA H100 80GB HBM3",
71
+ "memoryTotal": "85520809984",
72
+ "cudaCores": 16896,
73
+ "architecture": "Hopper",
74
+ "uuid": "GPU-165543e5-e049-5219-e717-e1216ac960de"
75
+ }
76
+ ],
77
+ "cudaVersion": "13.0",
78
+ "writerId": "uk366lvh33427a1pa49ye8fx63k3dx6x"
79
+ }
epfl_final/SparseAtt/wandb/run-20260730_174705-h5tf7yly/logs/debug-internal.log ADDED
@@ -0,0 +1,161 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {"time":"2026-07-30T17:47:06.250441675Z","level":"INFO","msg":"wandb-core"}
2
+ {"time":"2026-07-30T17:47:06.255829945Z","level":"INFO","msg":"stream: starting","core version":"0.26.1"}
3
+ {"time":"2026-07-30T17:47:06.780624642Z","level":"INFO","msg":"stream: created new stream","id":"h5tf7yly"}
4
+ {"time":"2026-07-30T17:47:06.78069673Z","level":"INFO","msg":"handler: started"}
5
+ {"time":"2026-07-30T17:47:06.782667548Z","level":"INFO","msg":"stream: started"}
6
+ {"time":"2026-07-30T17:47:06.783764609Z","level":"INFO","msg":"sender: started"}
7
+ {"time":"2026-07-30T17:47:06.783810307Z","level":"INFO","msg":"writer: started","stream_id":"h5tf7yly"}
8
+ {"time":"2026-07-30T17:47:08.235020667Z","level":"INFO","msg":"filestream: sending request","total_files":1,"console_offset":0,"console_lines":1}
9
+ {"time":"2026-07-30T17:47:08.612502105Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
10
+ {"time":"2026-07-30T17:47:23.235773215Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":0,"events_lines":2,"uploaded_len":2}
11
+ {"time":"2026-07-30T17:47:23.566425918Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
12
+ {"time":"2026-07-30T17:47:38.235698856Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":2,"events_lines":2}
13
+ {"time":"2026-07-30T17:47:38.680238629Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
14
+ {"time":"2026-07-30T17:47:53.235627472Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":4,"events_lines":2}
15
+ {"time":"2026-07-30T17:47:54.002200736Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
16
+ {"time":"2026-07-30T17:48:08.235498592Z","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":6,"events_lines":2,"console_offset":1,"console_lines":1}
17
+ {"time":"2026-07-30T17:48:08.653087082Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
18
+ {"time":"2026-07-30T17:48:23.235761598Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":8,"events_lines":2}
19
+ {"time":"2026-07-30T17:48:23.59643156Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
20
+ {"time":"2026-07-30T17:48:38.235720889Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":10,"events_lines":2}
21
+ {"time":"2026-07-30T17:48:38.596541357Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
22
+ {"time":"2026-07-30T17:48:53.235468775Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":12,"events_lines":2}
23
+ {"time":"2026-07-30T17:48:53.677679785Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
24
+ {"time":"2026-07-30T17:49:08.235873022Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":14,"events_lines":2}
25
+ {"time":"2026-07-30T17:49:08.680117193Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
26
+ {"time":"2026-07-30T17:49:23.235376853Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":16,"events_lines":2}
27
+ {"time":"2026-07-30T17:49:25.987404738Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
28
+ {"time":"2026-07-30T17:49:38.235589423Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":18,"events_lines":2}
29
+ {"time":"2026-07-30T17:49:38.606428096Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
30
+ {"time":"2026-07-30T17:49:53.235817657Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":20,"events_lines":2}
31
+ {"time":"2026-07-30T17:49:53.615352257Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
32
+ {"time":"2026-07-30T17:50:08.23532969Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":22,"events_lines":2}
33
+ {"time":"2026-07-30T17:50:08.591797496Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
34
+ {"time":"2026-07-30T17:50:23.235672906Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":24,"events_lines":2}
35
+ {"time":"2026-07-30T17:50:23.636819903Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
36
+ {"time":"2026-07-30T17:50:38.235783454Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":26,"events_lines":2}
37
+ {"time":"2026-07-30T17:50:38.610464494Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
38
+ {"time":"2026-07-30T17:50:53.235920731Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":28,"events_lines":2}
39
+ {"time":"2026-07-30T17:50:53.577236965Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
40
+ {"time":"2026-07-30T17:51:08.235994944Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":30,"events_lines":2}
41
+ {"time":"2026-07-30T17:51:08.531384141Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
42
+ {"time":"2026-07-30T17:51:23.235314299Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":32,"events_lines":2}
43
+ {"time":"2026-07-30T17:51:23.708422324Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
44
+ {"time":"2026-07-30T17:51:38.235446208Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":34,"events_lines":2}
45
+ {"time":"2026-07-30T17:51:38.635241494Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
46
+ {"time":"2026-07-30T17:51:53.235033684Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":36,"events_lines":2}
47
+ {"time":"2026-07-30T17:51:53.620830911Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
48
+ {"time":"2026-07-30T17:52:08.235948335Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":38,"events_lines":2}
49
+ {"time":"2026-07-30T17:52:09.238114844Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
50
+ {"time":"2026-07-30T17:52:23.235491484Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":40,"events_lines":2}
51
+ {"time":"2026-07-30T17:52:23.880737028Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
52
+ {"time":"2026-07-30T17:52:38.235910914Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":42,"events_lines":2}
53
+ {"time":"2026-07-30T17:52:39.251132932Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
54
+ {"time":"2026-07-30T17:52:53.235232331Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":44,"events_lines":2}
55
+ {"time":"2026-07-30T17:52:54.777688124Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
56
+ {"time":"2026-07-30T17:53:08.235826172Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":46,"events_lines":2}
57
+ {"time":"2026-07-30T17:53:08.610102947Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
58
+ {"time":"2026-07-30T17:53:23.235463703Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":48,"events_lines":2}
59
+ {"time":"2026-07-30T17:53:23.629806873Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
60
+ {"time":"2026-07-30T17:53:38.235265547Z","level":"INFO","msg":"filestream: sending request","total_files":2,"events_offset":50,"events_lines":2,"console_offset":2,"console_lines":1}
61
+ {"time":"2026-07-30T17:53:38.676146581Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
62
+ {"time":"2026-07-30T17:53:53.236076661Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":52,"events_lines":2}
63
+ {"time":"2026-07-30T17:53:53.600530706Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
64
+ {"time":"2026-07-30T17:54:08.235590365Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":54,"events_lines":2}
65
+ {"time":"2026-07-30T17:54:08.628108745Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
66
+ {"time":"2026-07-30T17:54:23.235078379Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":56,"events_lines":2}
67
+ {"time":"2026-07-30T17:54:23.605186482Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
68
+ {"time":"2026-07-30T17:54:38.23547308Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":58,"events_lines":2}
69
+ {"time":"2026-07-30T17:54:38.626361592Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
70
+ {"time":"2026-07-30T17:54:53.235267123Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":60,"events_lines":2}
71
+ {"time":"2026-07-30T17:54:53.612658562Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
72
+ {"time":"2026-07-30T17:55:08.235923375Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":62,"events_lines":2}
73
+ {"time":"2026-07-30T17:55:09.625139645Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
74
+ {"time":"2026-07-30T17:55:23.23535102Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":64,"events_lines":2}
75
+ {"time":"2026-07-30T17:55:23.696076873Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
76
+ {"time":"2026-07-30T17:55:38.235097021Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":66,"events_lines":2}
77
+ {"time":"2026-07-30T17:55:38.704160908Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
78
+ {"time":"2026-07-30T17:55:53.235814765Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":68,"events_lines":2}
79
+ {"time":"2026-07-30T17:55:53.612557665Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
80
+ {"time":"2026-07-30T17:56:08.235660385Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":70,"events_lines":2}
81
+ {"time":"2026-07-30T17:56:08.595791372Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
82
+ {"time":"2026-07-30T17:56:23.235618403Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":0,"history_lines":1,"events_offset":72,"events_lines":2,"console_offset":3,"console_lines":1}
83
+ {"time":"2026-07-30T17:56:23.648599888Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
84
+ {"time":"2026-07-30T17:56:38.235756761Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":74,"events_lines":2}
85
+ {"time":"2026-07-30T17:56:38.653421202Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
86
+ {"time":"2026-07-30T17:56:53.235672246Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":76,"events_lines":2}
87
+ {"time":"2026-07-30T17:56:53.826920397Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
88
+ {"time":"2026-07-30T17:57:08.23567456Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":78,"events_lines":2}
89
+ {"time":"2026-07-30T17:57:08.671532108Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
90
+ {"time":"2026-07-30T17:57:23.235737084Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":1,"history_lines":1,"events_offset":80,"events_lines":2,"console_offset":4,"console_lines":1}
91
+ {"time":"2026-07-30T17:57:23.709964727Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
92
+ {"time":"2026-07-30T17:57:38.2355472Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":82,"events_lines":2}
93
+ {"time":"2026-07-30T17:57:38.628929793Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
94
+ {"time":"2026-07-30T17:57:53.23588786Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":84,"events_lines":2}
95
+ {"time":"2026-07-30T17:57:53.603828611Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
96
+ {"time":"2026-07-30T17:58:08.235210298Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":86,"events_lines":2}
97
+ {"time":"2026-07-30T17:58:08.696005234Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
98
+ {"time":"2026-07-30T17:58:23.235815193Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":88,"events_lines":2}
99
+ {"time":"2026-07-30T17:58:23.591867689Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
100
+ {"time":"2026-07-30T17:58:38.23563877Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":2,"history_lines":1,"events_offset":90,"events_lines":2,"console_offset":5,"console_lines":1}
101
+ {"time":"2026-07-30T17:58:38.659262647Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
102
+ {"time":"2026-07-30T17:58:53.235531541Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":92,"events_lines":2}
103
+ {"time":"2026-07-30T17:58:53.632346579Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
104
+ {"time":"2026-07-30T17:59:08.235349169Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":94,"events_lines":2}
105
+ {"time":"2026-07-30T17:59:08.702072584Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
106
+ {"time":"2026-07-30T17:59:23.235529609Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":96,"events_lines":2}
107
+ {"time":"2026-07-30T17:59:23.618201179Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
108
+ {"time":"2026-07-30T17:59:38.235538564Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":3,"history_lines":1,"events_offset":98,"events_lines":2,"console_offset":6,"console_lines":1}
109
+ {"time":"2026-07-30T17:59:38.794204541Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
110
+ {"time":"2026-07-30T17:59:53.235852364Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":100,"events_lines":2}
111
+ {"time":"2026-07-30T17:59:53.713339052Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
112
+ {"time":"2026-07-30T18:00:08.235128222Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":102,"events_lines":2}
113
+ {"time":"2026-07-30T18:00:08.629056181Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
114
+ {"time":"2026-07-30T18:00:23.235763743Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":104,"events_lines":2}
115
+ {"time":"2026-07-30T18:00:23.637146566Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
116
+ {"time":"2026-07-30T18:00:38.235555983Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":4,"history_lines":1,"events_offset":106,"events_lines":2,"console_offset":7,"console_lines":1}
117
+ {"time":"2026-07-30T18:00:38.669091518Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
118
+ {"time":"2026-07-30T18:00:53.235861311Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":108,"events_lines":2}
119
+ {"time":"2026-07-30T18:00:53.629707408Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
120
+ {"time":"2026-07-30T18:01:08.235229476Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":110,"events_lines":2}
121
+ {"time":"2026-07-30T18:01:08.672432266Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
122
+ {"time":"2026-07-30T18:01:23.235528914Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":112,"events_lines":2}
123
+ {"time":"2026-07-30T18:01:23.622918642Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
124
+ {"time":"2026-07-30T18:01:38.235642225Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":5,"history_lines":1,"events_offset":114,"events_lines":2,"console_offset":8,"console_lines":1}
125
+ {"time":"2026-07-30T18:01:38.637789601Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
126
+ {"time":"2026-07-30T18:01:53.235713898Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":116,"events_lines":2}
127
+ {"time":"2026-07-30T18:01:53.625426335Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
128
+ {"time":"2026-07-30T18:02:08.235412223Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":118,"events_lines":2}
129
+ {"time":"2026-07-30T18:02:08.621601228Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
130
+ {"time":"2026-07-30T18:02:23.235423186Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":120,"events_lines":2}
131
+ {"time":"2026-07-30T18:02:23.608818039Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
132
+ {"time":"2026-07-30T18:02:38.235444994Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":122,"events_lines":2}
133
+ {"time":"2026-07-30T18:02:38.593005588Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
134
+ {"time":"2026-07-30T18:02:53.236045198Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":6,"history_lines":1,"events_offset":124,"events_lines":2,"console_offset":9,"console_lines":1}
135
+ {"time":"2026-07-30T18:02:53.671048407Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
136
+ {"time":"2026-07-30T18:03:08.23607578Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":126,"events_lines":2}
137
+ {"time":"2026-07-30T18:03:08.651113919Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
138
+ {"time":"2026-07-30T18:03:23.235047749Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":128,"events_lines":2}
139
+ {"time":"2026-07-30T18:03:23.693814281Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
140
+ {"time":"2026-07-30T18:03:38.23576644Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":130,"events_lines":2}
141
+ {"time":"2026-07-30T18:03:38.623433116Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
142
+ {"time":"2026-07-30T18:03:53.235458465Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":7,"history_lines":1,"events_offset":132,"events_lines":2,"console_offset":10,"console_lines":1}
143
+ {"time":"2026-07-30T18:03:53.63688623Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
144
+ {"time":"2026-07-30T18:04:08.235450039Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":134,"events_lines":2}
145
+ {"time":"2026-07-30T18:04:08.613955849Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
146
+ {"time":"2026-07-30T18:04:23.235755282Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":136,"events_lines":2}
147
+ {"time":"2026-07-30T18:04:23.63006702Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
148
+ {"time":"2026-07-30T18:04:38.235093267Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":138,"events_lines":2}
149
+ {"time":"2026-07-30T18:04:38.638014667Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
150
+ {"time":"2026-07-30T18:04:53.235889325Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":8,"history_lines":1,"events_offset":140,"events_lines":2,"console_offset":11,"console_lines":1}
151
+ {"time":"2026-07-30T18:04:53.630661742Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
152
+ {"time":"2026-07-30T18:05:08.23516696Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":142,"events_lines":2}
153
+ {"time":"2026-07-30T18:05:08.682387738Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
154
+ {"time":"2026-07-30T18:05:23.235702217Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":144,"events_lines":2}
155
+ {"time":"2026-07-30T18:05:23.626316163Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
156
+ {"time":"2026-07-30T18:05:38.235453671Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":146,"events_lines":2}
157
+ {"time":"2026-07-30T18:05:38.640986077Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
158
+ {"time":"2026-07-30T18:05:53.235991512Z","level":"INFO","msg":"filestream: sending request","total_files":4,"history_offset":9,"history_lines":1,"events_offset":148,"events_lines":2,"console_offset":12,"console_lines":1}
159
+ {"time":"2026-07-30T18:05:53.641575445Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
160
+ {"time":"2026-07-30T18:06:08.235510739Z","level":"INFO","msg":"filestream: sending request","total_files":1,"events_offset":150,"events_lines":2}
161
+ {"time":"2026-07-30T18:06:08.602245378Z","level":"INFO","msg":"filestream: request sent","status":"200 OK"}
epfl_final/SparseAtt/wandb/run-20260730_174705-h5tf7yly/logs/debug.log ADDED
@@ -0,0 +1,19 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2026-07-30 17:47:05,647 INFO MainThread:232 [wandb_setup.py:_flush():81] Current SDK version is 0.26.1
2
+ 2026-07-30 17:47:05,648 INFO MainThread:232 [wandb_setup.py:_flush():81] Configure stats pid to 232
3
+ 2026-07-30 17:47:05,648 INFO MainThread:232 [wandb_setup.py:_flush():81] Loading settings from environment variables
4
+ 2026-07-30 17:47:05,648 INFO MainThread:232 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /mloscratch/homes/navasard/sparse/SparseAtt/wandb/run-20260730_174705-h5tf7yly/logs/debug.log
5
+ 2026-07-30 17:47:05,648 INFO MainThread:232 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /mloscratch/homes/navasard/sparse/SparseAtt/wandb/run-20260730_174705-h5tf7yly/logs/debug-internal.log
6
+ 2026-07-30 17:47:05,648 INFO MainThread:232 [wandb_init.py:init():850] calling init triggers
7
+ 2026-07-30 17:47:05,648 INFO MainThread:232 [wandb_init.py:init():855] wandb.init called with sweep_config: {}
8
+ config: {'model_path': '/mloscratch/homes/navasard/qwen3_06b_base', 'model_dtype': 'bfloat16', 'init_from_scratch': True, 'train_path': '/mloscratch/homes/navasard/data/longctx30k/tokenized', 'val_path': None, 'max_seq_length': 30720, 'num_workers': 4, 'output_dir': '/mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_174601', 'resume_from_checkpoint': '/mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed/run_20260721_055708/checkpoint-30000', 'force_new_run': True, 'num_epochs': 1, 'max_steps': 37465, 'per_device_batch_size': 1, 'gradient_accumulation_steps': 5, 'learning_rate': 0.0005, 'weight_decay': 0.01, 'warmup_steps': 750, 'lr_scheduler': 'cosine', 'max_grad_norm': 1.0, 'save_steps': 300, 'eval_steps': 500, 'eval_batches': 20, 'logging_steps': 10, 'seed': 42, 'verbose': False, 'checkpoint_activations': True, 'strategy': 'ddp', 'mixed_precision': 'bf16', 'wandb_enabled': True, 'wandb_project': 'stochastic_sparsity', 'wandb_name': '1.2b native30k mixed-to-dense 20pct 4gpu h100', 'wandb_tags': ['qwen3', '1.2b', '30k', 'native', 'mixed-to-dense', '20pct', '4gpu', 'h100'], 'hidden_size': 1536, 'intermediate_size': 6144, 'num_attention_heads': 12, 'num_key_value_heads': 4, 'num_hidden_layers': None, 'head_dim': None, 'attention_type': 'sparse_stochastic', 'sparse_block_size': 128, 'sparse_num_random_blocks': 1, 'sparse_mask_pool_size': 1, 'sparse_val_dense_compare': False, 'sparse_layer_indices': None, 'sparse_mask_cache_dir': None, 'sparse_mask_seed': 42, 'dense_mode': True, 'dense_step_intervals': None, '_wandb': {}}
9
+ 2026-07-30 17:47:05,648 INFO MainThread:232 [wandb_init.py:init():898] starting backend
10
+ 2026-07-30 17:47:06,229 INFO MainThread:232 [wandb_init.py:init():913] sending inform_init request
11
+ 2026-07-30 17:47:06,783 INFO MainThread:232 [wandb_init.py:init():918] backend started and connected
12
+ 2026-07-30 17:47:06,791 INFO MainThread:232 [wandb_init.py:init():988] updated telemetry
13
+ 2026-07-30 17:47:06,820 INFO MainThread:232 [wandb_init.py:init():1011] communicating run to backend with 90.0 second timeout
14
+ 2026-07-30 17:47:07,858 INFO MainThread:232 [wandb_init.py:init():1056] starting run threads in backend
15
+ 2026-07-30 17:47:08,220 INFO MainThread:232 [wandb_run.py:_console_start():2554] atexit reg
16
+ 2026-07-30 17:47:08,220 INFO MainThread:232 [wandb_run.py:_redirect():2403] redirect: wrap_raw
17
+ 2026-07-30 17:47:08,220 INFO MainThread:232 [wandb_run.py:_redirect():2472] Wrapping output streams.
18
+ 2026-07-30 17:47:08,220 INFO MainThread:232 [wandb_run.py:_redirect():2495] Redirects installed.
19
+ 2026-07-30 17:47:08,229 INFO MainThread:232 [wandb_init.py:init():1094] run started, returning control to user process
epfl_final/SparseAtt/wandb/run-20260730_174705-h5tf7yly/run-h5tf7yly.wandb ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e010d623cd3638d474ae910753d575ce6296274b3cb6cf51580da72e31ca6d97
3
+ size 458752
epfl_final/SparseAtt/wandb/run-20260730_180915-0a2yphds/files/config.yaml ADDED
@@ -0,0 +1,188 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ _wandb:
2
+ value:
3
+ cli_version: 0.26.1
4
+ e:
5
+ snpo9ay0xz8qecmcb1hy8ntecd1r96yo:
6
+ args:
7
+ - --config
8
+ - /mloscratch/homes/navasard/sparse/SparseAtt/configs/native30k_1p2b_mixed_to_dense_20pct_4gpu_h100_20260730.json
9
+ codePath: train_dense_to_mixed_exact.py
10
+ codePathLocal: train_dense_to_mixed_exact.py
11
+ cpu_count: 96
12
+ cpu_count_logical: 192
13
+ cudaVersion: "13.0"
14
+ disk:
15
+ /:
16
+ total: "9374931091456"
17
+ used: "4847069007872"
18
+ email: ruben.navasardyan@mail.utoronto.ca
19
+ executable: /mloscratch/homes/navasard/conda/bin/python3.13
20
+ git:
21
+ commit: 88298c7a418622530ca5426b22b1e5ebd10ca0cb
22
+ remote: https://github.com/rubentium/TokenPrune.git
23
+ gpu: NVIDIA H100 80GB HBM3
24
+ gpu_count: 4
25
+ gpu_nvidia:
26
+ - architecture: Hopper
27
+ cudaCores: 16896
28
+ memoryTotal: "85520809984"
29
+ name: NVIDIA H100 80GB HBM3
30
+ uuid: GPU-32209e2e-7887-e730-7611-251622a8e78f
31
+ - architecture: Hopper
32
+ cudaCores: 16896
33
+ memoryTotal: "85520809984"
34
+ name: NVIDIA H100 80GB HBM3
35
+ uuid: GPU-3512802a-d501-0a3c-9812-4ab9deb3a3a9
36
+ - architecture: Hopper
37
+ cudaCores: 16896
38
+ memoryTotal: "85520809984"
39
+ name: NVIDIA H100 80GB HBM3
40
+ uuid: GPU-24436def-beee-c9f1-bd05-c97df556de08
41
+ - architecture: Hopper
42
+ cudaCores: 16896
43
+ memoryTotal: "85520809984"
44
+ name: NVIDIA H100 80GB HBM3
45
+ uuid: GPU-7b827c86-854a-af51-b5d3-6aaa43307d4a
46
+ host: h100-train-4-gpu-2-0-21
47
+ memory:
48
+ total: "1623005626368"
49
+ os: Linux-6.8.0-124-generic-x86_64-with-glibc2.35
50
+ program: /mloscratch/homes/navasard/sparse/SparseAtt/train_dense_to_mixed_exact.py
51
+ python: CPython 3.13.13
52
+ root: /mloscratch/homes/navasard/sparse/SparseAtt
53
+ startedAt: "2026-07-30T18:09:15.491336Z"
54
+ writerId: snpo9ay0xz8qecmcb1hy8ntecd1r96yo
55
+ m: []
56
+ python_version: 3.13.13
57
+ t:
58
+ "1":
59
+ - 1
60
+ - 5
61
+ - 11
62
+ - 49
63
+ - 51
64
+ - 53
65
+ - 71
66
+ "2":
67
+ - 1
68
+ - 5
69
+ - 11
70
+ - 49
71
+ - 51
72
+ - 53
73
+ - 71
74
+ "3":
75
+ - 2
76
+ - 13
77
+ - 15
78
+ - 16
79
+ - 61
80
+ "4": 3.13.13
81
+ "5": 0.26.1
82
+ "6": 5.7.0
83
+ "12": 0.26.1
84
+ "13": linux-x86_64
85
+ attention_type:
86
+ value: sparse_stochastic
87
+ checkpoint_activations:
88
+ value: true
89
+ dense_mode:
90
+ value: true
91
+ dense_step_intervals:
92
+ value: null
93
+ eval_batches:
94
+ value: 20
95
+ eval_steps:
96
+ value: 500
97
+ force_new_run:
98
+ value: true
99
+ gradient_accumulation_steps:
100
+ value: 5
101
+ head_dim:
102
+ value: null
103
+ hidden_size:
104
+ value: 1536
105
+ init_from_scratch:
106
+ value: true
107
+ intermediate_size:
108
+ value: 6144
109
+ learning_rate:
110
+ value: 0.0005
111
+ logging_steps:
112
+ value: 10
113
+ lr_scheduler:
114
+ value: cosine
115
+ max_grad_norm:
116
+ value: 1
117
+ max_seq_length:
118
+ value: 30720
119
+ max_steps:
120
+ value: 37465
121
+ mixed_precision:
122
+ value: bf16
123
+ model_dtype:
124
+ value: bfloat16
125
+ model_path:
126
+ value: /mloscratch/homes/navasard/qwen3_06b_base
127
+ num_attention_heads:
128
+ value: 12
129
+ num_epochs:
130
+ value: 1
131
+ num_hidden_layers:
132
+ value: null
133
+ num_key_value_heads:
134
+ value: 4
135
+ num_workers:
136
+ value: 4
137
+ output_dir:
138
+ value: /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750
139
+ per_device_batch_size:
140
+ value: 1
141
+ resume_from_checkpoint:
142
+ value: /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed/run_20260721_055708/checkpoint-30000
143
+ save_steps:
144
+ value: 300
145
+ seed:
146
+ value: 42
147
+ sparse_block_size:
148
+ value: 128
149
+ sparse_layer_indices:
150
+ value: null
151
+ sparse_mask_cache_dir:
152
+ value: null
153
+ sparse_mask_pool_size:
154
+ value: 1
155
+ sparse_mask_seed:
156
+ value: 42
157
+ sparse_num_random_blocks:
158
+ value: 1
159
+ sparse_val_dense_compare:
160
+ value: false
161
+ strategy:
162
+ value: ddp
163
+ train_path:
164
+ value: /mloscratch/homes/navasard/data/longctx30k/tokenized
165
+ val_path:
166
+ value: null
167
+ verbose:
168
+ value: false
169
+ wandb_enabled:
170
+ value: true
171
+ wandb_name:
172
+ value: 1.2b native30k mixed-to-dense 20pct 4gpu h100
173
+ wandb_project:
174
+ value: stochastic_sparsity
175
+ wandb_tags:
176
+ value:
177
+ - qwen3
178
+ - 1.2b
179
+ - 30k
180
+ - native
181
+ - mixed-to-dense
182
+ - 20pct
183
+ - 4gpu
184
+ - h100
185
+ warmup_steps:
186
+ value: 750
187
+ weight_decay:
188
+ value: 0.01
epfl_final/SparseAtt/wandb/run-20260730_180915-0a2yphds/files/output.log ADDED
@@ -0,0 +1,788 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Starting training...
2
+ Validation at step 30000: Loss = 2.0523, Perplexity = 7.79
3
+ Switching attention mode for step 30001: dense (dense step interval)
4
+ Step 30010/37465 | Loss: 1.8418 | LR: 4.92e-05 | 47614.0ms/step | 0.4 samples/s
5
+ Step 30020/37465 | Loss: 1.9770 | LR: 4.90e-05 | 6330.4ms/step | 3.2 samples/s
6
+ Step 30030/37465 | Loss: 2.0794 | LR: 4.89e-05 | 6335.6ms/step | 3.2 samples/s
7
+ Step 30040/37465 | Loss: 1.8654 | LR: 4.88e-05 | 6342.5ms/step | 3.2 samples/s
8
+ Step 30050/37465 | Loss: 1.9967 | LR: 4.87e-05 | 6340.2ms/step | 3.2 samples/s
9
+ Step 30060/37465 | Loss: 1.9703 | LR: 4.85e-05 | 6341.3ms/step | 3.2 samples/s
10
+ Step 30070/37465 | Loss: 2.1245 | LR: 4.84e-05 | 6347.6ms/step | 3.2 samples/s
11
+ Step 30080/37465 | Loss: 2.1234 | LR: 4.83e-05 | 6354.3ms/step | 3.1 samples/s
12
+ Step 30090/37465 | Loss: 2.1249 | LR: 4.81e-05 | 6352.7ms/step | 3.1 samples/s
13
+ Step 30100/37465 | Loss: 1.9543 | LR: 4.80e-05 | 6337.2ms/step | 3.2 samples/s
14
+ Step 30110/37465 | Loss: 1.8400 | LR: 4.79e-05 | 6338.9ms/step | 3.2 samples/s
15
+ Step 30120/37465 | Loss: 2.1601 | LR: 4.78e-05 | 6339.4ms/step | 3.2 samples/s
16
+ Step 30130/37465 | Loss: 1.9791 | LR: 4.76e-05 | 6335.9ms/step | 3.2 samples/s
17
+ Step 30140/37465 | Loss: 1.9384 | LR: 4.75e-05 | 6340.3ms/step | 3.2 samples/s
18
+ Step 30150/37465 | Loss: 1.7085 | LR: 4.74e-05 | 6337.5ms/step | 3.2 samples/s
19
+ Step 30160/37465 | Loss: 2.2310 | LR: 4.73e-05 | 6339.5ms/step | 3.2 samples/s
20
+ Step 30170/37465 | Loss: 2.0400 | LR: 4.71e-05 | 6343.9ms/step | 3.2 samples/s
21
+ Step 30180/37465 | Loss: 1.9701 | LR: 4.70e-05 | 6346.0ms/step | 3.2 samples/s
22
+ Step 30190/37465 | Loss: 2.1589 | LR: 4.69e-05 | 6337.7ms/step | 3.2 samples/s
23
+ Step 30200/37465 | Loss: 2.1711 | LR: 4.68e-05 | 6334.4ms/step | 3.2 samples/s
24
+ Step 30210/37465 | Loss: 2.0168 | LR: 4.66e-05 | 6345.2ms/step | 3.2 samples/s
25
+ Step 30220/37465 | Loss: 2.1456 | LR: 4.65e-05 | 6350.2ms/step | 3.1 samples/s
26
+ Step 30230/37465 | Loss: 2.0017 | LR: 4.64e-05 | 6344.0ms/step | 3.2 samples/s
27
+ Step 30240/37465 | Loss: 2.0567 | LR: 4.63e-05 | 6339.1ms/step | 3.2 samples/s
28
+ Step 30250/37465 | Loss: 2.0806 | LR: 4.61e-05 | 6346.7ms/step | 3.2 samples/s
29
+ Step 30260/37465 | Loss: 1.9372 | LR: 4.60e-05 | 6348.3ms/step | 3.2 samples/s
30
+ Step 30270/37465 | Loss: 2.2240 | LR: 4.59e-05 | 6347.9ms/step | 3.2 samples/s
31
+ Step 30280/37465 | Loss: 2.1534 | LR: 4.58e-05 | 6347.8ms/step | 3.2 samples/s
32
+ Step 30290/37465 | Loss: 2.2598 | LR: 4.57e-05 | 6344.3ms/step | 3.2 samples/s
33
+ Step 30300/37465 | Loss: 2.0519 | LR: 4.55e-05 | 6343.3ms/step | 3.2 samples/s
34
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-30300
35
+ Step 30310/37465 | Loss: 1.8345 | LR: 4.54e-05 | 7321.3ms/step | 2.7 samples/s
36
+ Step 30320/37465 | Loss: 2.0833 | LR: 4.53e-05 | 6346.7ms/step | 3.2 samples/s
37
+ Step 30330/37465 | Loss: 2.1041 | LR: 4.52e-05 | 6353.4ms/step | 3.1 samples/s
38
+ Step 30340/37465 | Loss: 1.9861 | LR: 4.50e-05 | 6362.2ms/step | 3.1 samples/s
39
+ Step 30350/37465 | Loss: 2.1433 | LR: 4.49e-05 | 6368.4ms/step | 3.1 samples/s
40
+ Step 30360/37465 | Loss: 2.0674 | LR: 4.48e-05 | 6364.7ms/step | 3.1 samples/s
41
+ Step 30370/37465 | Loss: 2.1087 | LR: 4.47e-05 | 6345.1ms/step | 3.2 samples/s
42
+ Step 30380/37465 | Loss: 2.0687 | LR: 4.46e-05 | 6348.4ms/step | 3.2 samples/s
43
+ Step 30390/37465 | Loss: 2.0920 | LR: 4.44e-05 | 6347.5ms/step | 3.2 samples/s
44
+ Step 30400/37465 | Loss: 2.1823 | LR: 4.43e-05 | 6343.7ms/step | 3.2 samples/s
45
+ Step 30410/37465 | Loss: 1.9883 | LR: 4.42e-05 | 6343.9ms/step | 3.2 samples/s
46
+ Step 30420/37465 | Loss: 2.0307 | LR: 4.41e-05 | 6350.3ms/step | 3.1 samples/s
47
+ Step 30430/37465 | Loss: 2.1039 | LR: 4.39e-05 | 6341.5ms/step | 3.2 samples/s
48
+ Step 30440/37465 | Loss: 2.1141 | LR: 4.38e-05 | 6349.9ms/step | 3.1 samples/s
49
+ Step 30450/37465 | Loss: 2.2654 | LR: 4.37e-05 | 6345.5ms/step | 3.2 samples/s
50
+ Step 30460/37465 | Loss: 1.8425 | LR: 4.36e-05 | 6345.8ms/step | 3.2 samples/s
51
+ Step 30470/37465 | Loss: 1.8893 | LR: 4.35e-05 | 6355.6ms/step | 3.1 samples/s
52
+ Step 30480/37465 | Loss: 1.9265 | LR: 4.33e-05 | 6343.8ms/step | 3.2 samples/s
53
+ Step 30490/37465 | Loss: 1.9125 | LR: 4.32e-05 | 6347.4ms/step | 3.2 samples/s
54
+ Step 30500/37465 | Loss: 2.0482 | LR: 4.31e-05 | 6344.1ms/step | 3.2 samples/s
55
+ Validation at step 30500: Loss = 2.0438, Perplexity = 7.72
56
+ Step 30510/37465 | Loss: 2.1133 | LR: 4.30e-05 | 7094.0ms/step | 2.8 samples/s
57
+ Step 30520/37465 | Loss: 1.9020 | LR: 4.29e-05 | 6346.5ms/step | 3.2 samples/s
58
+ Step 30530/37465 | Loss: 2.1593 | LR: 4.27e-05 | 6341.0ms/step | 3.2 samples/s
59
+ Step 30540/37465 | Loss: 2.2062 | LR: 4.26e-05 | 6342.4ms/step | 3.2 samples/s
60
+ Step 30550/37465 | Loss: 2.1123 | LR: 4.25e-05 | 6352.2ms/step | 3.1 samples/s
61
+ Step 30560/37465 | Loss: 2.0613 | LR: 4.24e-05 | 6344.7ms/step | 3.2 samples/s
62
+ Step 30570/37465 | Loss: 2.0372 | LR: 4.23e-05 | 6338.0ms/step | 3.2 samples/s
63
+ Step 30580/37465 | Loss: 1.8193 | LR: 4.21e-05 | 6342.1ms/step | 3.2 samples/s
64
+ Step 30590/37465 | Loss: 1.9118 | LR: 4.20e-05 | 6347.4ms/step | 3.2 samples/s
65
+ Step 30600/37465 | Loss: 1.7807 | LR: 4.19e-05 | 6350.0ms/step | 3.1 samples/s
66
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-30600
67
+ Step 30610/37465 | Loss: 1.9655 | LR: 4.18e-05 | 7263.9ms/step | 2.8 samples/s
68
+ Step 30620/37465 | Loss: 2.0072 | LR: 4.17e-05 | 6340.4ms/step | 3.2 samples/s
69
+ Step 30630/37465 | Loss: 2.0071 | LR: 4.16e-05 | 6353.8ms/step | 3.1 samples/s
70
+ Step 30640/37465 | Loss: 2.0427 | LR: 4.14e-05 | 6342.1ms/step | 3.2 samples/s
71
+ Step 30650/37465 | Loss: 2.0625 | LR: 4.13e-05 | 6343.7ms/step | 3.2 samples/s
72
+ Step 30660/37465 | Loss: 2.2126 | LR: 4.12e-05 | 6350.7ms/step | 3.1 samples/s
73
+ Step 30670/37465 | Loss: 2.0531 | LR: 4.11e-05 | 6345.5ms/step | 3.2 samples/s
74
+ Step 30680/37465 | Loss: 2.1471 | LR: 4.10e-05 | 6352.7ms/step | 3.1 samples/s
75
+ Step 30690/37465 | Loss: 2.0161 | LR: 4.08e-05 | 6364.3ms/step | 3.1 samples/s
76
+ Step 30700/37465 | Loss: 1.9454 | LR: 4.07e-05 | 6349.2ms/step | 3.2 samples/s
77
+ Step 30710/37465 | Loss: 1.9923 | LR: 4.06e-05 | 6338.6ms/step | 3.2 samples/s
78
+ Step 30720/37465 | Loss: 2.0685 | LR: 4.05e-05 | 6342.3ms/step | 3.2 samples/s
79
+ Step 30730/37465 | Loss: 2.0671 | LR: 4.04e-05 | 6351.6ms/step | 3.1 samples/s
80
+ Step 30740/37465 | Loss: 2.1149 | LR: 4.03e-05 | 6352.7ms/step | 3.1 samples/s
81
+ Step 30750/37465 | Loss: 1.9658 | LR: 4.01e-05 | 6342.3ms/step | 3.2 samples/s
82
+ Step 30760/37465 | Loss: 1.8896 | LR: 4.00e-05 | 6342.8ms/step | 3.2 samples/s
83
+ Step 30770/37465 | Loss: 2.2864 | LR: 3.99e-05 | 6353.7ms/step | 3.1 samples/s
84
+ Step 30780/37465 | Loss: 1.9749 | LR: 3.98e-05 | 6352.7ms/step | 3.1 samples/s
85
+ Step 30790/37465 | Loss: 1.8974 | LR: 3.97e-05 | 6352.9ms/step | 3.1 samples/s
86
+ Step 30800/37465 | Loss: 1.8834 | LR: 3.96e-05 | 6355.4ms/step | 3.1 samples/s
87
+ Step 30810/37465 | Loss: 2.0647 | LR: 3.95e-05 | 6343.1ms/step | 3.2 samples/s
88
+ Step 30820/37465 | Loss: 1.7848 | LR: 3.93e-05 | 6351.3ms/step | 3.1 samples/s
89
+ Step 30830/37465 | Loss: 1.9402 | LR: 3.92e-05 | 6354.5ms/step | 3.1 samples/s
90
+ Step 30840/37465 | Loss: 1.8295 | LR: 3.91e-05 | 6351.6ms/step | 3.1 samples/s
91
+ Step 30850/37465 | Loss: 2.1893 | LR: 3.90e-05 | 6342.6ms/step | 3.2 samples/s
92
+ Step 30860/37465 | Loss: 1.8727 | LR: 3.89e-05 | 6342.7ms/step | 3.2 samples/s
93
+ Step 30870/37465 | Loss: 1.9230 | LR: 3.88e-05 | 6348.7ms/step | 3.2 samples/s
94
+ Step 30880/37465 | Loss: 1.9689 | LR: 3.86e-05 | 6340.4ms/step | 3.2 samples/s
95
+ Step 30890/37465 | Loss: 2.1349 | LR: 3.85e-05 | 6345.8ms/step | 3.2 samples/s
96
+ Step 30900/37465 | Loss: 2.0386 | LR: 3.84e-05 | 6349.5ms/step | 3.1 samples/s
97
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-30900
98
+ Step 30910/37465 | Loss: 1.9545 | LR: 3.83e-05 | 7314.4ms/step | 2.7 samples/s
99
+ Step 30920/37465 | Loss: 2.2191 | LR: 3.82e-05 | 6344.0ms/step | 3.2 samples/s
100
+ Step 30930/37465 | Loss: 2.0627 | LR: 3.81e-05 | 6340.3ms/step | 3.2 samples/s
101
+ Step 30940/37465 | Loss: 1.8358 | LR: 3.80e-05 | 6338.5ms/step | 3.2 samples/s
102
+ Step 30950/37465 | Loss: 2.1065 | LR: 3.79e-05 | 6339.2ms/step | 3.2 samples/s
103
+ Step 30960/37465 | Loss: 1.9582 | LR: 3.77e-05 | 6350.5ms/step | 3.1 samples/s
104
+ Step 30970/37465 | Loss: 1.8912 | LR: 3.76e-05 | 6341.4ms/step | 3.2 samples/s
105
+ Step 30980/37465 | Loss: 2.0574 | LR: 3.75e-05 | 6343.2ms/step | 3.2 samples/s
106
+ Step 30990/37465 | Loss: 1.9920 | LR: 3.74e-05 | 6340.4ms/step | 3.2 samples/s
107
+ Step 31000/37465 | Loss: 1.9481 | LR: 3.73e-05 | 6342.5ms/step | 3.2 samples/s
108
+ Validation at step 31000: Loss = 2.0430, Perplexity = 7.71
109
+ Step 31010/37465 | Loss: 2.0266 | LR: 3.72e-05 | 7096.8ms/step | 2.8 samples/s
110
+ Step 31020/37465 | Loss: 1.9868 | LR: 3.71e-05 | 6347.0ms/step | 3.2 samples/s
111
+ Step 31030/37465 | Loss: 1.9063 | LR: 3.70e-05 | 6338.8ms/step | 3.2 samples/s
112
+ Step 31040/37465 | Loss: 2.2081 | LR: 3.68e-05 | 6349.2ms/step | 3.1 samples/s
113
+ Step 31050/37465 | Loss: 2.0773 | LR: 3.67e-05 | 6354.7ms/step | 3.1 samples/s
114
+ Step 31060/37465 | Loss: 1.9722 | LR: 3.66e-05 | 6342.2ms/step | 3.2 samples/s
115
+ Step 31070/37465 | Loss: 2.1394 | LR: 3.65e-05 | 6348.9ms/step | 3.2 samples/s
116
+ Step 31080/37465 | Loss: 1.9284 | LR: 3.64e-05 | 6347.0ms/step | 3.2 samples/s
117
+ Step 31090/37465 | Loss: 1.9516 | LR: 3.63e-05 | 6345.8ms/step | 3.2 samples/s
118
+ Step 31100/37465 | Loss: 2.0733 | LR: 3.62e-05 | 6344.6ms/step | 3.2 samples/s
119
+ Step 31110/37465 | Loss: 2.0916 | LR: 3.61e-05 | 6345.5ms/step | 3.2 samples/s
120
+ Step 31120/37465 | Loss: 2.1576 | LR: 3.59e-05 | 6353.9ms/step | 3.1 samples/s
121
+ Step 31130/37465 | Loss: 2.0255 | LR: 3.58e-05 | 6347.1ms/step | 3.2 samples/s
122
+ Step 31140/37465 | Loss: 2.0503 | LR: 3.57e-05 | 6352.3ms/step | 3.1 samples/s
123
+ Step 31150/37465 | Loss: 2.0014 | LR: 3.56e-05 | 6349.1ms/step | 3.2 samples/s
124
+ Step 31160/37465 | Loss: 2.0543 | LR: 3.55e-05 | 6345.3ms/step | 3.2 samples/s
125
+ Step 31170/37465 | Loss: 2.0818 | LR: 3.54e-05 | 6342.2ms/step | 3.2 samples/s
126
+ Step 31180/37465 | Loss: 1.9991 | LR: 3.53e-05 | 6349.5ms/step | 3.1 samples/s
127
+ Step 31190/37465 | Loss: 2.0709 | LR: 3.52e-05 | 6347.5ms/step | 3.2 samples/s
128
+ Step 31200/37465 | Loss: 1.9576 | LR: 3.51e-05 | 6342.5ms/step | 3.2 samples/s
129
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-31200
130
+ Step 31210/37465 | Loss: 2.0380 | LR: 3.50e-05 | 7447.0ms/step | 2.7 samples/s
131
+ Step 31220/37465 | Loss: 1.9102 | LR: 3.49e-05 | 6341.8ms/step | 3.2 samples/s
132
+ Step 31230/37465 | Loss: 1.8609 | LR: 3.47e-05 | 6349.9ms/step | 3.1 samples/s
133
+ Step 31240/37465 | Loss: 2.1740 | LR: 3.46e-05 | 6348.0ms/step | 3.2 samples/s
134
+ Step 31250/37465 | Loss: 1.9087 | LR: 3.45e-05 | 6352.0ms/step | 3.1 samples/s
135
+ Step 31260/37465 | Loss: 2.0343 | LR: 3.44e-05 | 6346.3ms/step | 3.2 samples/s
136
+ Step 31270/37465 | Loss: 2.1862 | LR: 3.43e-05 | 6345.3ms/step | 3.2 samples/s
137
+ Step 31280/37465 | Loss: 1.9514 | LR: 3.42e-05 | 6345.3ms/step | 3.2 samples/s
138
+ Step 31290/37465 | Loss: 1.8960 | LR: 3.41e-05 | 6347.0ms/step | 3.2 samples/s
139
+ Step 31300/37465 | Loss: 2.0621 | LR: 3.40e-05 | 6353.2ms/step | 3.1 samples/s
140
+ Step 31310/37465 | Loss: 2.2288 | LR: 3.39e-05 | 6345.8ms/step | 3.2 samples/s
141
+ Step 31320/37465 | Loss: 2.0216 | LR: 3.38e-05 | 6350.7ms/step | 3.1 samples/s
142
+ Step 31330/37465 | Loss: 1.8353 | LR: 3.37e-05 | 6352.7ms/step | 3.1 samples/s
143
+ Step 31340/37465 | Loss: 1.9597 | LR: 3.36e-05 | 6347.9ms/step | 3.2 samples/s
144
+ Step 31350/37465 | Loss: 2.0403 | LR: 3.34e-05 | 6347.0ms/step | 3.2 samples/s
145
+ Step 31360/37465 | Loss: 2.2647 | LR: 3.33e-05 | 6352.2ms/step | 3.1 samples/s
146
+ Step 31370/37465 | Loss: 2.0667 | LR: 3.32e-05 | 6347.5ms/step | 3.2 samples/s
147
+ Step 31380/37465 | Loss: 1.9585 | LR: 3.31e-05 | 6333.0ms/step | 3.2 samples/s
148
+ Step 31390/37465 | Loss: 1.9972 | LR: 3.30e-05 | 6347.0ms/step | 3.2 samples/s
149
+ Step 31400/37465 | Loss: 1.9512 | LR: 3.29e-05 | 6343.8ms/step | 3.2 samples/s
150
+ Step 31410/37465 | Loss: 1.8924 | LR: 3.28e-05 | 6344.4ms/step | 3.2 samples/s
151
+ Step 31420/37465 | Loss: 2.0923 | LR: 3.27e-05 | 6350.3ms/step | 3.1 samples/s
152
+ Step 31430/37465 | Loss: 1.7991 | LR: 3.26e-05 | 6339.7ms/step | 3.2 samples/s
153
+ Step 31440/37465 | Loss: 1.8984 | LR: 3.25e-05 | 6338.1ms/step | 3.2 samples/s
154
+ Step 31450/37465 | Loss: 2.1369 | LR: 3.24e-05 | 6343.3ms/step | 3.2 samples/s
155
+ Step 31460/37465 | Loss: 2.0744 | LR: 3.23e-05 | 6339.7ms/step | 3.2 samples/s
156
+ Step 31470/37465 | Loss: 2.1711 | LR: 3.22e-05 | 6329.4ms/step | 3.2 samples/s
157
+ Step 31480/37465 | Loss: 1.9181 | LR: 3.21e-05 | 6332.6ms/step | 3.2 samples/s
158
+ Step 31490/37465 | Loss: 2.1564 | LR: 3.20e-05 | 6334.0ms/step | 3.2 samples/s
159
+ Step 31500/37465 | Loss: 2.1131 | LR: 3.19e-05 | 6341.4ms/step | 3.2 samples/s
160
+ Validation at step 31500: Loss = 2.0423, Perplexity = 7.71
161
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-31500
162
+ Step 31510/37465 | Loss: 2.1079 | LR: 3.18e-05 | 7972.8ms/step | 2.5 samples/s
163
+ Step 31520/37465 | Loss: 2.0142 | LR: 3.17e-05 | 6336.1ms/step | 3.2 samples/s
164
+ Step 31530/37465 | Loss: 1.9110 | LR: 3.16e-05 | 6347.4ms/step | 3.2 samples/s
165
+ Step 31540/37465 | Loss: 1.9320 | LR: 3.14e-05 | 6343.0ms/step | 3.2 samples/s
166
+ Step 31550/37465 | Loss: 1.9094 | LR: 3.13e-05 | 6352.4ms/step | 3.1 samples/s
167
+ Step 31560/37465 | Loss: 1.8689 | LR: 3.12e-05 | 6348.0ms/step | 3.2 samples/s
168
+ Step 31570/37465 | Loss: 1.9131 | LR: 3.11e-05 | 6337.8ms/step | 3.2 samples/s
169
+ Step 31580/37465 | Loss: 2.0625 | LR: 3.10e-05 | 6341.1ms/step | 3.2 samples/s
170
+ Step 31590/37465 | Loss: 2.1842 | LR: 3.09e-05 | 6347.9ms/step | 3.2 samples/s
171
+ Step 31600/37465 | Loss: 1.9684 | LR: 3.08e-05 | 6346.4ms/step | 3.2 samples/s
172
+ Step 31610/37465 | Loss: 2.0081 | LR: 3.07e-05 | 6350.3ms/step | 3.1 samples/s
173
+ Step 31620/37465 | Loss: 1.9818 | LR: 3.06e-05 | 6332.8ms/step | 3.2 samples/s
174
+ Step 31630/37465 | Loss: 1.9957 | LR: 3.05e-05 | 6341.5ms/step | 3.2 samples/s
175
+ Step 31640/37465 | Loss: 2.0598 | LR: 3.04e-05 | 6344.7ms/step | 3.2 samples/s
176
+ Step 31650/37465 | Loss: 1.9005 | LR: 3.03e-05 | 6342.2ms/step | 3.2 samples/s
177
+ Step 31660/37465 | Loss: 2.0299 | LR: 3.02e-05 | 6346.1ms/step | 3.2 samples/s
178
+ Step 31670/37465 | Loss: 1.9993 | LR: 3.01e-05 | 6344.4ms/step | 3.2 samples/s
179
+ Step 31680/37465 | Loss: 2.0556 | LR: 3.00e-05 | 6348.0ms/step | 3.2 samples/s
180
+ Step 31690/37465 | Loss: 1.9175 | LR: 2.99e-05 | 6346.3ms/step | 3.2 samples/s
181
+ Step 31700/37465 | Loss: 1.9524 | LR: 2.98e-05 | 6350.8ms/step | 3.1 samples/s
182
+ Step 31710/37465 | Loss: 2.1386 | LR: 2.97e-05 | 6353.3ms/step | 3.1 samples/s
183
+ Step 31720/37465 | Loss: 1.9226 | LR: 2.96e-05 | 6345.8ms/step | 3.2 samples/s
184
+ Step 31730/37465 | Loss: 2.0108 | LR: 2.95e-05 | 6349.9ms/step | 3.1 samples/s
185
+ Step 31740/37465 | Loss: 1.9433 | LR: 2.94e-05 | 6352.0ms/step | 3.1 samples/s
186
+ Step 31750/37465 | Loss: 2.0465 | LR: 2.93e-05 | 6348.2ms/step | 3.2 samples/s
187
+ Step 31760/37465 | Loss: 2.0358 | LR: 2.92e-05 | 6344.5ms/step | 3.2 samples/s
188
+ Step 31770/37465 | Loss: 2.0411 | LR: 2.91e-05 | 6336.5ms/step | 3.2 samples/s
189
+ Step 31780/37465 | Loss: 2.1119 | LR: 2.90e-05 | 6342.7ms/step | 3.2 samples/s
190
+ Step 31790/37465 | Loss: 2.2618 | LR: 2.89e-05 | 6354.7ms/step | 3.1 samples/s
191
+ Step 31800/37465 | Loss: 2.1630 | LR: 2.88e-05 | 6343.7ms/step | 3.2 samples/s
192
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-31800
193
+ Step 31810/37465 | Loss: 2.0638 | LR: 2.87e-05 | 7232.7ms/step | 2.8 samples/s
194
+ Step 31820/37465 | Loss: 2.2305 | LR: 2.86e-05 | 6340.6ms/step | 3.2 samples/s
195
+ Step 31830/37465 | Loss: 1.9356 | LR: 2.85e-05 | 6349.4ms/step | 3.1 samples/s
196
+ Step 31840/37465 | Loss: 2.0030 | LR: 2.84e-05 | 6351.4ms/step | 3.1 samples/s
197
+ Step 31850/37465 | Loss: 2.1525 | LR: 2.83e-05 | 6349.8ms/step | 3.1 samples/s
198
+ Step 31860/37465 | Loss: 2.1451 | LR: 2.82e-05 | 6341.3ms/step | 3.2 samples/s
199
+ Step 31870/37465 | Loss: 2.1954 | LR: 2.81e-05 | 6340.9ms/step | 3.2 samples/s
200
+ Step 31880/37465 | Loss: 1.9414 | LR: 2.80e-05 | 6346.1ms/step | 3.2 samples/s
201
+ Step 31890/37465 | Loss: 1.9487 | LR: 2.79e-05 | 6355.1ms/step | 3.1 samples/s
202
+ Step 31900/37465 | Loss: 1.9611 | LR: 2.78e-05 | 6349.3ms/step | 3.1 samples/s
203
+ Step 31910/37465 | Loss: 2.0297 | LR: 2.77e-05 | 6339.3ms/step | 3.2 samples/s
204
+ Step 31920/37465 | Loss: 1.9106 | LR: 2.76e-05 | 6343.4ms/step | 3.2 samples/s
205
+ Step 31930/37465 | Loss: 1.8700 | LR: 2.75e-05 | 6342.2ms/step | 3.2 samples/s
206
+ Step 31940/37465 | Loss: 2.1195 | LR: 2.74e-05 | 6346.9ms/step | 3.2 samples/s
207
+ Step 31950/37465 | Loss: 1.8901 | LR: 2.73e-05 | 6340.7ms/step | 3.2 samples/s
208
+ Step 31960/37465 | Loss: 2.1289 | LR: 2.72e-05 | 6341.6ms/step | 3.2 samples/s
209
+ Step 31970/37465 | Loss: 2.0760 | LR: 2.71e-05 | 6342.7ms/step | 3.2 samples/s
210
+ Step 31980/37465 | Loss: 2.0957 | LR: 2.70e-05 | 6340.3ms/step | 3.2 samples/s
211
+ Step 31990/37465 | Loss: 2.0256 | LR: 2.69e-05 | 6349.2ms/step | 3.2 samples/s
212
+ Step 32000/37465 | Loss: 2.2297 | LR: 2.68e-05 | 6352.0ms/step | 3.1 samples/s
213
+ Validation at step 32000: Loss = 2.0419, Perplexity = 7.71
214
+ Step 32010/37465 | Loss: 2.2212 | LR: 2.67e-05 | 7081.0ms/step | 2.8 samples/s
215
+ Step 32020/37465 | Loss: 1.9302 | LR: 2.66e-05 | 6351.9ms/step | 3.1 samples/s
216
+ Step 32030/37465 | Loss: 2.0561 | LR: 2.66e-05 | 6347.8ms/step | 3.2 samples/s
217
+ Step 32040/37465 | Loss: 2.0198 | LR: 2.65e-05 | 6347.1ms/step | 3.2 samples/s
218
+ Step 32050/37465 | Loss: 2.1470 | LR: 2.64e-05 | 6347.1ms/step | 3.2 samples/s
219
+ Step 32060/37465 | Loss: 2.1028 | LR: 2.63e-05 | 6342.2ms/step | 3.2 samples/s
220
+ Step 32070/37465 | Loss: 2.0251 | LR: 2.62e-05 | 6339.6ms/step | 3.2 samples/s
221
+ Step 32080/37465 | Loss: 2.0745 | LR: 2.61e-05 | 6340.5ms/step | 3.2 samples/s
222
+ Step 32090/37465 | Loss: 2.0705 | LR: 2.60e-05 | 6345.5ms/step | 3.2 samples/s
223
+ Step 32100/37465 | Loss: 2.1582 | LR: 2.59e-05 | 6337.1ms/step | 3.2 samples/s
224
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-32100
225
+ Step 32110/37465 | Loss: 2.1348 | LR: 2.58e-05 | 7327.9ms/step | 2.7 samples/s
226
+ Step 32120/37465 | Loss: 2.1011 | LR: 2.57e-05 | 6353.0ms/step | 3.1 samples/s
227
+ Step 32130/37465 | Loss: 2.2170 | LR: 2.56e-05 | 6348.2ms/step | 3.2 samples/s
228
+ Step 32140/37465 | Loss: 1.7852 | LR: 2.55e-05 | 6353.2ms/step | 3.1 samples/s
229
+ Step 32150/37465 | Loss: 2.0400 | LR: 2.54e-05 | 6355.6ms/step | 3.1 samples/s
230
+ Step 32160/37465 | Loss: 1.8025 | LR: 2.53e-05 | 6350.0ms/step | 3.1 samples/s
231
+ Step 32170/37465 | Loss: 2.0416 | LR: 2.52e-05 | 6349.7ms/step | 3.1 samples/s
232
+ Step 32180/37465 | Loss: 2.0832 | LR: 2.51e-05 | 6339.1ms/step | 3.2 samples/s
233
+ Step 32190/37465 | Loss: 2.0224 | LR: 2.50e-05 | 6347.1ms/step | 3.2 samples/s
234
+ Step 32200/37465 | Loss: 2.1240 | LR: 2.49e-05 | 6343.3ms/step | 3.2 samples/s
235
+ Step 32210/37465 | Loss: 2.2008 | LR: 2.49e-05 | 6350.7ms/step | 3.1 samples/s
236
+ Step 32220/37465 | Loss: 2.1844 | LR: 2.48e-05 | 6352.4ms/step | 3.1 samples/s
237
+ Step 32230/37465 | Loss: 1.9346 | LR: 2.47e-05 | 6339.0ms/step | 3.2 samples/s
238
+ Step 32240/37465 | Loss: 2.1918 | LR: 2.46e-05 | 6342.2ms/step | 3.2 samples/s
239
+ Step 32250/37465 | Loss: 2.0743 | LR: 2.45e-05 | 6345.3ms/step | 3.2 samples/s
240
+ Step 32260/37465 | Loss: 2.0835 | LR: 2.44e-05 | 6341.8ms/step | 3.2 samples/s
241
+ Step 32270/37465 | Loss: 2.1746 | LR: 2.43e-05 | 6340.8ms/step | 3.2 samples/s
242
+ Step 32280/37465 | Loss: 2.2114 | LR: 2.42e-05 | 6346.6ms/step | 3.2 samples/s
243
+ Step 32290/37465 | Loss: 1.7763 | LR: 2.41e-05 | 6352.3ms/step | 3.1 samples/s
244
+ Step 32300/37465 | Loss: 2.1408 | LR: 2.40e-05 | 6350.5ms/step | 3.1 samples/s
245
+ Step 32310/37465 | Loss: 2.0356 | LR: 2.39e-05 | 6344.3ms/step | 3.2 samples/s
246
+ Step 32320/37465 | Loss: 1.9232 | LR: 2.38e-05 | 6342.5ms/step | 3.2 samples/s
247
+ Step 32330/37465 | Loss: 2.0659 | LR: 2.37e-05 | 6350.5ms/step | 3.1 samples/s
248
+ Step 32340/37465 | Loss: 1.9380 | LR: 2.37e-05 | 6345.3ms/step | 3.2 samples/s
249
+ Step 32350/37465 | Loss: 1.9793 | LR: 2.36e-05 | 6350.0ms/step | 3.1 samples/s
250
+ Step 32360/37465 | Loss: 1.9713 | LR: 2.35e-05 | 6339.0ms/step | 3.2 samples/s
251
+ Step 32370/37465 | Loss: 2.1817 | LR: 2.34e-05 | 6345.1ms/step | 3.2 samples/s
252
+ Step 32380/37465 | Loss: 2.0760 | LR: 2.33e-05 | 6341.4ms/step | 3.2 samples/s
253
+ Step 32390/37465 | Loss: 2.1555 | LR: 2.32e-05 | 6345.8ms/step | 3.2 samples/s
254
+ Step 32400/37465 | Loss: 1.9927 | LR: 2.31e-05 | 6346.2ms/step | 3.2 samples/s
255
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-32400
256
+ Step 32410/37465 | Loss: 2.1067 | LR: 2.30e-05 | 7301.1ms/step | 2.7 samples/s
257
+ Step 32420/37465 | Loss: 2.1696 | LR: 2.29e-05 | 6344.0ms/step | 3.2 samples/s
258
+ Step 32430/37465 | Loss: 2.1275 | LR: 2.28e-05 | 6355.9ms/step | 3.1 samples/s
259
+ Step 32440/37465 | Loss: 1.9532 | LR: 2.28e-05 | 6339.0ms/step | 3.2 samples/s
260
+ Step 32450/37465 | Loss: 2.1336 | LR: 2.27e-05 | 6346.4ms/step | 3.2 samples/s
261
+ Step 32460/37465 | Loss: 1.9718 | LR: 2.26e-05 | 6343.0ms/step | 3.2 samples/s
262
+ Step 32470/37465 | Loss: 2.4169 | LR: 2.25e-05 | 6349.8ms/step | 3.1 samples/s
263
+ Step 32480/37465 | Loss: 2.0413 | LR: 2.24e-05 | 6354.5ms/step | 3.1 samples/s
264
+ Step 32490/37465 | Loss: 1.8394 | LR: 2.23e-05 | 6342.3ms/step | 3.2 samples/s
265
+ Step 32500/37465 | Loss: 2.0109 | LR: 2.22e-05 | 6344.4ms/step | 3.2 samples/s
266
+ Validation at step 32500: Loss = 2.0417, Perplexity = 7.70
267
+ Step 32510/37465 | Loss: 2.3181 | LR: 2.21e-05 | 7095.0ms/step | 2.8 samples/s
268
+ Step 32520/37465 | Loss: 1.9834 | LR: 2.20e-05 | 6343.3ms/step | 3.2 samples/s
269
+ Step 32530/37465 | Loss: 1.9217 | LR: 2.20e-05 | 6341.5ms/step | 3.2 samples/s
270
+ Step 32540/37465 | Loss: 2.2313 | LR: 2.19e-05 | 6343.2ms/step | 3.2 samples/s
271
+ Step 32550/37465 | Loss: 1.8750 | LR: 2.18e-05 | 6341.0ms/step | 3.2 samples/s
272
+ Step 32560/37465 | Loss: 2.2540 | LR: 2.17e-05 | 6350.5ms/step | 3.1 samples/s
273
+ Step 32570/37465 | Loss: 2.3363 | LR: 2.16e-05 | 6347.5ms/step | 3.2 samples/s
274
+ Step 32580/37465 | Loss: 2.0731 | LR: 2.15e-05 | 6351.3ms/step | 3.1 samples/s
275
+ Step 32590/37465 | Loss: 1.9258 | LR: 2.14e-05 | 6350.1ms/step | 3.1 samples/s
276
+ Step 32600/37465 | Loss: 2.0156 | LR: 2.14e-05 | 6347.4ms/step | 3.2 samples/s
277
+ Step 32610/37465 | Loss: 1.9459 | LR: 2.13e-05 | 6345.3ms/step | 3.2 samples/s
278
+ Step 32620/37465 | Loss: 1.9524 | LR: 2.12e-05 | 6347.5ms/step | 3.2 samples/s
279
+ Step 32630/37465 | Loss: 2.0709 | LR: 2.11e-05 | 6345.0ms/step | 3.2 samples/s
280
+ Step 32640/37465 | Loss: 2.2225 | LR: 2.10e-05 | 6345.4ms/step | 3.2 samples/s
281
+ Step 32650/37465 | Loss: 2.1087 | LR: 2.09e-05 | 6352.7ms/step | 3.1 samples/s
282
+ Step 32660/37465 | Loss: 2.0600 | LR: 2.08e-05 | 6350.0ms/step | 3.1 samples/s
283
+ Step 32670/37465 | Loss: 2.1304 | LR: 2.07e-05 | 6347.3ms/step | 3.2 samples/s
284
+ Step 32680/37465 | Loss: 1.9534 | LR: 2.07e-05 | 6338.5ms/step | 3.2 samples/s
285
+ Step 32690/37465 | Loss: 1.8941 | LR: 2.06e-05 | 6334.6ms/step | 3.2 samples/s
286
+ Step 32700/37465 | Loss: 2.1260 | LR: 2.05e-05 | 6343.2ms/step | 3.2 samples/s
287
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-32700
288
+ Step 32710/37465 | Loss: 2.0488 | LR: 2.04e-05 | 7328.5ms/step | 2.7 samples/s
289
+ Step 32720/37465 | Loss: 1.8427 | LR: 2.03e-05 | 6341.1ms/step | 3.2 samples/s
290
+ Step 32730/37465 | Loss: 2.1121 | LR: 2.02e-05 | 6348.9ms/step | 3.2 samples/s
291
+ Step 32740/37465 | Loss: 2.0664 | LR: 2.02e-05 | 6344.5ms/step | 3.2 samples/s
292
+ Step 32750/37465 | Loss: 2.0054 | LR: 2.01e-05 | 6341.2ms/step | 3.2 samples/s
293
+ Step 32760/37465 | Loss: 1.9986 | LR: 2.00e-05 | 6350.5ms/step | 3.1 samples/s
294
+ Step 32770/37465 | Loss: 2.1505 | LR: 1.99e-05 | 6343.1ms/step | 3.2 samples/s
295
+ Step 32780/37465 | Loss: 1.9105 | LR: 1.98e-05 | 6347.1ms/step | 3.2 samples/s
296
+ Step 32790/37465 | Loss: 1.7833 | LR: 1.97e-05 | 6344.1ms/step | 3.2 samples/s
297
+ Step 32800/37465 | Loss: 2.1457 | LR: 1.97e-05 | 6347.0ms/step | 3.2 samples/s
298
+ Step 32810/37465 | Loss: 1.8395 | LR: 1.96e-05 | 6344.5ms/step | 3.2 samples/s
299
+ Step 32820/37465 | Loss: 1.8407 | LR: 1.95e-05 | 6342.6ms/step | 3.2 samples/s
300
+ Step 32830/37465 | Loss: 1.9703 | LR: 1.94e-05 | 6344.5ms/step | 3.2 samples/s
301
+ Step 32840/37465 | Loss: 1.8958 | LR: 1.93e-05 | 6341.5ms/step | 3.2 samples/s
302
+ Step 32850/37465 | Loss: 2.2661 | LR: 1.92e-05 | 6341.9ms/step | 3.2 samples/s
303
+ Step 32860/37465 | Loss: 1.9051 | LR: 1.92e-05 | 6348.4ms/step | 3.2 samples/s
304
+ Step 32870/37465 | Loss: 2.2739 | LR: 1.91e-05 | 6352.9ms/step | 3.1 samples/s
305
+ Step 32880/37465 | Loss: 2.0480 | LR: 1.90e-05 | 6355.3ms/step | 3.1 samples/s
306
+ Step 32890/37465 | Loss: 1.9072 | LR: 1.89e-05 | 6342.0ms/step | 3.2 samples/s
307
+ Step 32900/37465 | Loss: 2.0662 | LR: 1.88e-05 | 6343.7ms/step | 3.2 samples/s
308
+ Step 32910/37465 | Loss: 2.0793 | LR: 1.87e-05 | 6338.4ms/step | 3.2 samples/s
309
+ Step 32920/37465 | Loss: 2.1072 | LR: 1.87e-05 | 6342.3ms/step | 3.2 samples/s
310
+ Step 32930/37465 | Loss: 2.0248 | LR: 1.86e-05 | 6343.7ms/step | 3.2 samples/s
311
+ Step 32940/37465 | Loss: 2.0953 | LR: 1.85e-05 | 6353.6ms/step | 3.1 samples/s
312
+ Step 32950/37465 | Loss: 1.9203 | LR: 1.84e-05 | 6344.4ms/step | 3.2 samples/s
313
+ Step 32960/37465 | Loss: 2.1979 | LR: 1.83e-05 | 6346.4ms/step | 3.2 samples/s
314
+ Step 32970/37465 | Loss: 2.0417 | LR: 1.83e-05 | 6352.4ms/step | 3.1 samples/s
315
+ Step 32980/37465 | Loss: 2.1357 | LR: 1.82e-05 | 6352.0ms/step | 3.1 samples/s
316
+ Step 32990/37465 | Loss: 2.0230 | LR: 1.81e-05 | 6347.2ms/step | 3.2 samples/s
317
+ Step 33000/37465 | Loss: 1.9484 | LR: 1.80e-05 | 6350.8ms/step | 3.1 samples/s
318
+ Validation at step 33000: Loss = 2.0416, Perplexity = 7.70
319
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-33000
320
+ Step 33010/37465 | Loss: 2.0518 | LR: 1.79e-05 | 8107.0ms/step | 2.5 samples/s
321
+ Step 33020/37465 | Loss: 2.1209 | LR: 1.79e-05 | 6340.7ms/step | 3.2 samples/s
322
+ Step 33030/37465 | Loss: 2.0867 | LR: 1.78e-05 | 6345.1ms/step | 3.2 samples/s
323
+ Step 33040/37465 | Loss: 2.0495 | LR: 1.77e-05 | 6344.3ms/step | 3.2 samples/s
324
+ Step 33050/37465 | Loss: 2.1171 | LR: 1.76e-05 | 6337.0ms/step | 3.2 samples/s
325
+ Step 33060/37465 | Loss: 2.1803 | LR: 1.75e-05 | 6344.8ms/step | 3.2 samples/s
326
+ Step 33070/37465 | Loss: 2.0106 | LR: 1.75e-05 | 6341.0ms/step | 3.2 samples/s
327
+ Step 33080/37465 | Loss: 2.0170 | LR: 1.74e-05 | 6340.8ms/step | 3.2 samples/s
328
+ Step 33090/37465 | Loss: 2.2165 | LR: 1.73e-05 | 6350.3ms/step | 3.1 samples/s
329
+ Step 33100/37465 | Loss: 2.0219 | LR: 1.72e-05 | 6344.9ms/step | 3.2 samples/s
330
+ Step 33110/37465 | Loss: 2.0086 | LR: 1.72e-05 | 6339.0ms/step | 3.2 samples/s
331
+ Step 33120/37465 | Loss: 2.0254 | LR: 1.71e-05 | 6353.0ms/step | 3.1 samples/s
332
+ Step 33130/37465 | Loss: 2.1346 | LR: 1.70e-05 | 6354.1ms/step | 3.1 samples/s
333
+ Step 33140/37465 | Loss: 2.2266 | LR: 1.69e-05 | 6343.1ms/step | 3.2 samples/s
334
+ Step 33150/37465 | Loss: 1.9964 | LR: 1.68e-05 | 6335.0ms/step | 3.2 samples/s
335
+ Step 33160/37465 | Loss: 2.1041 | LR: 1.68e-05 | 6346.0ms/step | 3.2 samples/s
336
+ Step 33170/37465 | Loss: 1.8607 | LR: 1.67e-05 | 6348.8ms/step | 3.2 samples/s
337
+ Step 33180/37465 | Loss: 2.2087 | LR: 1.66e-05 | 6346.6ms/step | 3.2 samples/s
338
+ Step 33190/37465 | Loss: 2.1495 | LR: 1.65e-05 | 6345.3ms/step | 3.2 samples/s
339
+ Step 33200/37465 | Loss: 1.9156 | LR: 1.65e-05 | 6340.4ms/step | 3.2 samples/s
340
+ Step 33210/37465 | Loss: 1.8640 | LR: 1.64e-05 | 6346.3ms/step | 3.2 samples/s
341
+ Step 33220/37465 | Loss: 2.0631 | LR: 1.63e-05 | 6339.0ms/step | 3.2 samples/s
342
+ Step 33230/37465 | Loss: 1.8377 | LR: 1.62e-05 | 6337.9ms/step | 3.2 samples/s
343
+ Step 33240/37465 | Loss: 2.0298 | LR: 1.62e-05 | 6339.1ms/step | 3.2 samples/s
344
+ Step 33250/37465 | Loss: 1.9095 | LR: 1.61e-05 | 6338.8ms/step | 3.2 samples/s
345
+ Step 33260/37465 | Loss: 1.9473 | LR: 1.60e-05 | 6344.9ms/step | 3.2 samples/s
346
+ Step 33270/37465 | Loss: 2.0169 | LR: 1.59e-05 | 6339.9ms/step | 3.2 samples/s
347
+ Step 33280/37465 | Loss: 1.7697 | LR: 1.59e-05 | 6339.1ms/step | 3.2 samples/s
348
+ Step 33290/37465 | Loss: 2.0439 | LR: 1.58e-05 | 6341.7ms/step | 3.2 samples/s
349
+ Step 33300/37465 | Loss: 1.9725 | LR: 1.57e-05 | 6338.9ms/step | 3.2 samples/s
350
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-33300
351
+ Step 33310/37465 | Loss: 2.0180 | LR: 1.56e-05 | 7320.0ms/step | 2.7 samples/s
352
+ Step 33320/37465 | Loss: 2.0823 | LR: 1.56e-05 | 6350.2ms/step | 3.1 samples/s
353
+ Step 33330/37465 | Loss: 1.8129 | LR: 1.55e-05 | 6350.6ms/step | 3.1 samples/s
354
+ Step 33340/37465 | Loss: 2.0877 | LR: 1.54e-05 | 6338.9ms/step | 3.2 samples/s
355
+ Step 33350/37465 | Loss: 1.9723 | LR: 1.53e-05 | 6350.8ms/step | 3.1 samples/s
356
+ Step 33360/37465 | Loss: 2.1171 | LR: 1.53e-05 | 6343.8ms/step | 3.2 samples/s
357
+ Step 33370/37465 | Loss: 2.1141 | LR: 1.52e-05 | 6339.5ms/step | 3.2 samples/s
358
+ Step 33380/37465 | Loss: 2.1487 | LR: 1.51e-05 | 6342.5ms/step | 3.2 samples/s
359
+ Step 33390/37465 | Loss: 2.0957 | LR: 1.50e-05 | 6346.7ms/step | 3.2 samples/s
360
+ Step 33400/37465 | Loss: 2.1272 | LR: 1.50e-05 | 6341.2ms/step | 3.2 samples/s
361
+ Step 33410/37465 | Loss: 2.0028 | LR: 1.49e-05 | 6344.4ms/step | 3.2 samples/s
362
+ Step 33420/37465 | Loss: 1.9020 | LR: 1.48e-05 | 6350.0ms/step | 3.1 samples/s
363
+ Step 33430/37465 | Loss: 2.2934 | LR: 1.48e-05 | 6342.9ms/step | 3.2 samples/s
364
+ Step 33440/37465 | Loss: 2.0303 | LR: 1.47e-05 | 6334.6ms/step | 3.2 samples/s
365
+ Step 33450/37465 | Loss: 2.1878 | LR: 1.46e-05 | 6341.9ms/step | 3.2 samples/s
366
+ Step 33460/37465 | Loss: 1.8113 | LR: 1.45e-05 | 6342.6ms/step | 3.2 samples/s
367
+ Step 33470/37465 | Loss: 2.1404 | LR: 1.45e-05 | 6347.2ms/step | 3.2 samples/s
368
+ Step 33480/37465 | Loss: 2.1435 | LR: 1.44e-05 | 6348.8ms/step | 3.2 samples/s
369
+ Step 33490/37465 | Loss: 1.8940 | LR: 1.43e-05 | 6348.3ms/step | 3.2 samples/s
370
+ Step 33500/37465 | Loss: 1.9516 | LR: 1.43e-05 | 6341.7ms/step | 3.2 samples/s
371
+ Validation at step 33500: Loss = 2.0415, Perplexity = 7.70
372
+ Step 33510/37465 | Loss: 2.1125 | LR: 1.42e-05 | 7089.2ms/step | 2.8 samples/s
373
+ Step 33520/37465 | Loss: 2.0927 | LR: 1.41e-05 | 6346.7ms/step | 3.2 samples/s
374
+ Step 33530/37465 | Loss: 2.0785 | LR: 1.40e-05 | 6337.2ms/step | 3.2 samples/s
375
+ Step 33540/37465 | Loss: 2.0496 | LR: 1.40e-05 | 6350.8ms/step | 3.1 samples/s
376
+ Step 33550/37465 | Loss: 1.9793 | LR: 1.39e-05 | 6341.3ms/step | 3.2 samples/s
377
+ Step 33560/37465 | Loss: 2.0369 | LR: 1.38e-05 | 6339.8ms/step | 3.2 samples/s
378
+ Step 33570/37465 | Loss: 1.9258 | LR: 1.38e-05 | 6353.7ms/step | 3.1 samples/s
379
+ Step 33580/37465 | Loss: 2.1625 | LR: 1.37e-05 | 6352.6ms/step | 3.1 samples/s
380
+ Step 33590/37465 | Loss: 2.0127 | LR: 1.36e-05 | 6336.1ms/step | 3.2 samples/s
381
+ Step 33600/37465 | Loss: 1.9291 | LR: 1.35e-05 | 6336.8ms/step | 3.2 samples/s
382
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-33600
383
+ Step 33610/37465 | Loss: 2.2329 | LR: 1.35e-05 | 7364.2ms/step | 2.7 samples/s
384
+ Step 33620/37465 | Loss: 1.8749 | LR: 1.34e-05 | 6347.1ms/step | 3.2 samples/s
385
+ Step 33630/37465 | Loss: 2.2982 | LR: 1.33e-05 | 6338.1ms/step | 3.2 samples/s
386
+ Step 33640/37465 | Loss: 1.9735 | LR: 1.33e-05 | 6342.1ms/step | 3.2 samples/s
387
+ Step 33650/37465 | Loss: 2.1000 | LR: 1.32e-05 | 6345.2ms/step | 3.2 samples/s
388
+ Step 33660/37465 | Loss: 1.9474 | LR: 1.31e-05 | 6344.7ms/step | 3.2 samples/s
389
+ Step 33670/37465 | Loss: 2.0383 | LR: 1.31e-05 | 6345.8ms/step | 3.2 samples/s
390
+ Step 33680/37465 | Loss: 2.0793 | LR: 1.30e-05 | 6349.5ms/step | 3.1 samples/s
391
+ Step 33690/37465 | Loss: 2.0151 | LR: 1.29e-05 | 6351.8ms/step | 3.1 samples/s
392
+ Step 33700/37465 | Loss: 1.9768 | LR: 1.29e-05 | 6347.5ms/step | 3.2 samples/s
393
+ Step 33710/37465 | Loss: 1.9830 | LR: 1.28e-05 | 6349.7ms/step | 3.1 samples/s
394
+ Step 33720/37465 | Loss: 1.9245 | LR: 1.27e-05 | 6347.8ms/step | 3.2 samples/s
395
+ Step 33730/37465 | Loss: 2.1879 | LR: 1.27e-05 | 6341.7ms/step | 3.2 samples/s
396
+ Step 33740/37465 | Loss: 2.2556 | LR: 1.26e-05 | 6346.4ms/step | 3.2 samples/s
397
+ Step 33750/37465 | Loss: 2.1683 | LR: 1.25e-05 | 6350.4ms/step | 3.1 samples/s
398
+ Step 33760/37465 | Loss: 2.0550 | LR: 1.25e-05 | 6339.6ms/step | 3.2 samples/s
399
+ Step 33770/37465 | Loss: 2.0926 | LR: 1.24e-05 | 6347.8ms/step | 3.2 samples/s
400
+ Step 33780/37465 | Loss: 1.9440 | LR: 1.23e-05 | 6351.4ms/step | 3.1 samples/s
401
+ Step 33790/37465 | Loss: 1.9710 | LR: 1.23e-05 | 6346.5ms/step | 3.2 samples/s
402
+ Step 33800/37465 | Loss: 1.9770 | LR: 1.22e-05 | 6347.9ms/step | 3.2 samples/s
403
+ Step 33810/37465 | Loss: 2.0706 | LR: 1.21e-05 | 6346.4ms/step | 3.2 samples/s
404
+ Step 33820/37465 | Loss: 2.1123 | LR: 1.21e-05 | 6341.2ms/step | 3.2 samples/s
405
+ Step 33830/37465 | Loss: 2.0240 | LR: 1.20e-05 | 6346.2ms/step | 3.2 samples/s
406
+ Step 33840/37465 | Loss: 1.7454 | LR: 1.19e-05 | 6353.8ms/step | 3.1 samples/s
407
+ Step 33850/37465 | Loss: 2.2602 | LR: 1.19e-05 | 6345.6ms/step | 3.2 samples/s
408
+ Step 33860/37465 | Loss: 2.1780 | LR: 1.18e-05 | 6344.7ms/step | 3.2 samples/s
409
+ Step 33870/37465 | Loss: 2.1269 | LR: 1.17e-05 | 6342.9ms/step | 3.2 samples/s
410
+ Step 33880/37465 | Loss: 2.0790 | LR: 1.17e-05 | 6349.3ms/step | 3.1 samples/s
411
+ Step 33890/37465 | Loss: 1.9091 | LR: 1.16e-05 | 6342.1ms/step | 3.2 samples/s
412
+ Step 33900/37465 | Loss: 2.0935 | LR: 1.15e-05 | 6346.7ms/step | 3.2 samples/s
413
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-33900
414
+ Step 33910/37465 | Loss: 2.1975 | LR: 1.15e-05 | 7292.7ms/step | 2.7 samples/s
415
+ Step 33920/37465 | Loss: 2.2065 | LR: 1.14e-05 | 6337.0ms/step | 3.2 samples/s
416
+ Step 33930/37465 | Loss: 2.0803 | LR: 1.13e-05 | 6353.3ms/step | 3.1 samples/s
417
+ Step 33940/37465 | Loss: 2.1870 | LR: 1.13e-05 | 6345.3ms/step | 3.2 samples/s
418
+ Step 33950/37465 | Loss: 1.9364 | LR: 1.12e-05 | 6353.8ms/step | 3.1 samples/s
419
+ Step 33960/37465 | Loss: 2.0208 | LR: 1.12e-05 | 6342.1ms/step | 3.2 samples/s
420
+ Step 33970/37465 | Loss: 2.0595 | LR: 1.11e-05 | 6350.6ms/step | 3.1 samples/s
421
+ Step 33980/37465 | Loss: 1.9693 | LR: 1.10e-05 | 6340.7ms/step | 3.2 samples/s
422
+ Step 33990/37465 | Loss: 2.0571 | LR: 1.10e-05 | 6341.7ms/step | 3.2 samples/s
423
+ Step 34000/37465 | Loss: 2.0195 | LR: 1.09e-05 | 6347.9ms/step | 3.2 samples/s
424
+ Validation at step 34000: Loss = 2.0414, Perplexity = 7.70
425
+ Step 34010/37465 | Loss: 1.9693 | LR: 1.08e-05 | 7090.6ms/step | 2.8 samples/s
426
+ Step 34020/37465 | Loss: 2.2337 | LR: 1.08e-05 | 6339.9ms/step | 3.2 samples/s
427
+ Step 34030/37465 | Loss: 2.0455 | LR: 1.07e-05 | 6347.3ms/step | 3.2 samples/s
428
+ Step 34040/37465 | Loss: 2.2275 | LR: 1.07e-05 | 6349.9ms/step | 3.1 samples/s
429
+ Step 34050/37465 | Loss: 2.1988 | LR: 1.06e-05 | 6350.3ms/step | 3.1 samples/s
430
+ Step 34060/37465 | Loss: 1.9758 | LR: 1.05e-05 | 6345.8ms/step | 3.2 samples/s
431
+ Step 34070/37465 | Loss: 1.7727 | LR: 1.05e-05 | 6351.2ms/step | 3.1 samples/s
432
+ Step 34080/37465 | Loss: 2.0066 | LR: 1.04e-05 | 6350.6ms/step | 3.1 samples/s
433
+ Step 34090/37465 | Loss: 2.1155 | LR: 1.04e-05 | 6340.7ms/step | 3.2 samples/s
434
+ Step 34100/37465 | Loss: 2.1628 | LR: 1.03e-05 | 6347.1ms/step | 3.2 samples/s
435
+ Step 34110/37465 | Loss: 2.0472 | LR: 1.02e-05 | 6343.3ms/step | 3.2 samples/s
436
+ Step 34120/37465 | Loss: 1.9008 | LR: 1.02e-05 | 6349.6ms/step | 3.1 samples/s
437
+ Step 34130/37465 | Loss: 2.0815 | LR: 1.01e-05 | 6344.9ms/step | 3.2 samples/s
438
+ Step 34140/37465 | Loss: 2.0830 | LR: 1.01e-05 | 6349.3ms/step | 3.1 samples/s
439
+ Step 34150/37465 | Loss: 2.0178 | LR: 9.99e-06 | 6343.2ms/step | 3.2 samples/s
440
+ Step 34160/37465 | Loss: 2.0867 | LR: 9.93e-06 | 6335.8ms/step | 3.2 samples/s
441
+ Step 34170/37465 | Loss: 2.0891 | LR: 9.87e-06 | 6341.2ms/step | 3.2 samples/s
442
+ Step 34180/37465 | Loss: 2.1996 | LR: 9.81e-06 | 6353.5ms/step | 3.1 samples/s
443
+ Step 34190/37465 | Loss: 2.1171 | LR: 9.75e-06 | 6347.3ms/step | 3.2 samples/s
444
+ Step 34200/37465 | Loss: 2.0351 | LR: 9.69e-06 | 6349.1ms/step | 3.2 samples/s
445
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-34200
446
+ Step 34210/37465 | Loss: 2.0883 | LR: 9.63e-06 | 7324.9ms/step | 2.7 samples/s
447
+ Step 34220/37465 | Loss: 1.9087 | LR: 9.58e-06 | 6341.8ms/step | 3.2 samples/s
448
+ Step 34230/37465 | Loss: 2.2873 | LR: 9.52e-06 | 6342.4ms/step | 3.2 samples/s
449
+ Step 34240/37465 | Loss: 2.0857 | LR: 9.46e-06 | 6363.3ms/step | 3.1 samples/s
450
+ Step 34250/37465 | Loss: 1.9238 | LR: 9.40e-06 | 6364.5ms/step | 3.1 samples/s
451
+ Step 34260/37465 | Loss: 1.9234 | LR: 9.34e-06 | 6351.3ms/step | 3.1 samples/s
452
+ Step 34270/37465 | Loss: 1.8891 | LR: 9.28e-06 | 6336.4ms/step | 3.2 samples/s
453
+ Step 34280/37465 | Loss: 2.0175 | LR: 9.23e-06 | 6342.4ms/step | 3.2 samples/s
454
+ Step 34290/37465 | Loss: 2.1203 | LR: 9.17e-06 | 6347.0ms/step | 3.2 samples/s
455
+ Step 34300/37465 | Loss: 2.0502 | LR: 9.11e-06 | 6352.2ms/step | 3.1 samples/s
456
+ Step 34310/37465 | Loss: 2.0027 | LR: 9.05e-06 | 6339.7ms/step | 3.2 samples/s
457
+ Step 34320/37465 | Loss: 1.6884 | LR: 9.00e-06 | 6339.3ms/step | 3.2 samples/s
458
+ Step 34330/37465 | Loss: 1.9777 | LR: 8.94e-06 | 6351.7ms/step | 3.1 samples/s
459
+ Step 34340/37465 | Loss: 2.0998 | LR: 8.88e-06 | 6342.5ms/step | 3.2 samples/s
460
+ Step 34350/37465 | Loss: 1.8989 | LR: 8.83e-06 | 6345.0ms/step | 3.2 samples/s
461
+ Step 34360/37465 | Loss: 1.9511 | LR: 8.77e-06 | 6335.9ms/step | 3.2 samples/s
462
+ Step 34370/37465 | Loss: 2.0849 | LR: 8.72e-06 | 6337.8ms/step | 3.2 samples/s
463
+ Step 34380/37465 | Loss: 1.8454 | LR: 8.66e-06 | 6336.4ms/step | 3.2 samples/s
464
+ Step 34390/37465 | Loss: 2.0554 | LR: 8.60e-06 | 6339.3ms/step | 3.2 samples/s
465
+ Step 34400/37465 | Loss: 2.1296 | LR: 8.55e-06 | 6334.9ms/step | 3.2 samples/s
466
+ Step 34410/37465 | Loss: 1.9905 | LR: 8.49e-06 | 6342.7ms/step | 3.2 samples/s
467
+ Step 34420/37465 | Loss: 1.9817 | LR: 8.44e-06 | 6355.7ms/step | 3.1 samples/s
468
+ Step 34430/37465 | Loss: 1.9293 | LR: 8.38e-06 | 6343.2ms/step | 3.2 samples/s
469
+ Step 34440/37465 | Loss: 1.8519 | LR: 8.33e-06 | 6344.4ms/step | 3.2 samples/s
470
+ Step 34450/37465 | Loss: 2.1115 | LR: 8.27e-06 | 6349.5ms/step | 3.1 samples/s
471
+ Step 34460/37465 | Loss: 2.1478 | LR: 8.22e-06 | 6344.1ms/step | 3.2 samples/s
472
+ Step 34470/37465 | Loss: 1.8690 | LR: 8.16e-06 | 6355.7ms/step | 3.1 samples/s
473
+ Step 34480/37465 | Loss: 1.9352 | LR: 8.11e-06 | 6353.2ms/step | 3.1 samples/s
474
+ Step 34490/37465 | Loss: 1.9740 | LR: 8.06e-06 | 6344.6ms/step | 3.2 samples/s
475
+ Step 34500/37465 | Loss: 2.0910 | LR: 8.00e-06 | 6346.9ms/step | 3.2 samples/s
476
+ Validation at step 34500: Loss = 2.0414, Perplexity = 7.70
477
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-34500
478
+ Step 34510/37465 | Loss: 1.8498 | LR: 7.95e-06 | 8038.0ms/step | 2.5 samples/s
479
+ Step 34520/37465 | Loss: 2.0575 | LR: 7.90e-06 | 6332.1ms/step | 3.2 samples/s
480
+ Step 34530/37465 | Loss: 2.0593 | LR: 7.84e-06 | 6346.1ms/step | 3.2 samples/s
481
+ Step 34540/37465 | Loss: 1.9193 | LR: 7.79e-06 | 6341.8ms/step | 3.2 samples/s
482
+ Step 34550/37465 | Loss: 2.1163 | LR: 7.74e-06 | 6342.3ms/step | 3.2 samples/s
483
+ Step 34560/37465 | Loss: 2.1486 | LR: 7.68e-06 | 6345.5ms/step | 3.2 samples/s
484
+ Step 34570/37465 | Loss: 2.1117 | LR: 7.63e-06 | 6349.1ms/step | 3.2 samples/s
485
+ Step 34580/37465 | Loss: 2.0655 | LR: 7.58e-06 | 6349.4ms/step | 3.1 samples/s
486
+ Step 34590/37465 | Loss: 2.1675 | LR: 7.53e-06 | 6340.4ms/step | 3.2 samples/s
487
+ Step 34600/37465 | Loss: 2.0611 | LR: 7.47e-06 | 6343.3ms/step | 3.2 samples/s
488
+ Step 34610/37465 | Loss: 1.9497 | LR: 7.42e-06 | 6347.4ms/step | 3.2 samples/s
489
+ Step 34620/37465 | Loss: 1.9411 | LR: 7.37e-06 | 6340.0ms/step | 3.2 samples/s
490
+ Step 34630/37465 | Loss: 2.0684 | LR: 7.32e-06 | 6334.8ms/step | 3.2 samples/s
491
+ Step 34640/37465 | Loss: 1.9277 | LR: 7.27e-06 | 6343.2ms/step | 3.2 samples/s
492
+ Step 34650/37465 | Loss: 1.9911 | LR: 7.22e-06 | 6346.5ms/step | 3.2 samples/s
493
+ Step 34660/37465 | Loss: 2.0065 | LR: 7.17e-06 | 6340.4ms/step | 3.2 samples/s
494
+ Step 34670/37465 | Loss: 2.0433 | LR: 7.12e-06 | 6341.0ms/step | 3.2 samples/s
495
+ Step 34680/37465 | Loss: 2.0413 | LR: 7.07e-06 | 6341.5ms/step | 3.2 samples/s
496
+ Step 34690/37465 | Loss: 2.0008 | LR: 7.01e-06 | 6344.7ms/step | 3.2 samples/s
497
+ Step 34700/37465 | Loss: 2.1024 | LR: 6.96e-06 | 6351.6ms/step | 3.1 samples/s
498
+ Step 34710/37465 | Loss: 1.9882 | LR: 6.91e-06 | 6341.4ms/step | 3.2 samples/s
499
+ Step 34720/37465 | Loss: 2.0107 | LR: 6.86e-06 | 6337.1ms/step | 3.2 samples/s
500
+ Step 34730/37465 | Loss: 1.7803 | LR: 6.81e-06 | 6341.7ms/step | 3.2 samples/s
501
+ Step 34740/37465 | Loss: 2.1986 | LR: 6.77e-06 | 6347.1ms/step | 3.2 samples/s
502
+ Step 34750/37465 | Loss: 1.9501 | LR: 6.72e-06 | 6340.5ms/step | 3.2 samples/s
503
+ Step 34760/37465 | Loss: 2.0876 | LR: 6.67e-06 | 6347.0ms/step | 3.2 samples/s
504
+ Step 34770/37465 | Loss: 1.9816 | LR: 6.62e-06 | 6343.7ms/step | 3.2 samples/s
505
+ Step 34780/37465 | Loss: 2.1574 | LR: 6.57e-06 | 6343.5ms/step | 3.2 samples/s
506
+ Step 34790/37465 | Loss: 2.1047 | LR: 6.52e-06 | 6334.1ms/step | 3.2 samples/s
507
+ Step 34800/37465 | Loss: 2.0940 | LR: 6.47e-06 | 6345.9ms/step | 3.2 samples/s
508
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-34800
509
+ Step 34810/37465 | Loss: 1.9488 | LR: 6.42e-06 | 7283.9ms/step | 2.7 samples/s
510
+ Step 34820/37465 | Loss: 1.9418 | LR: 6.38e-06 | 6335.8ms/step | 3.2 samples/s
511
+ Step 34830/37465 | Loss: 2.2260 | LR: 6.33e-06 | 6340.4ms/step | 3.2 samples/s
512
+ Step 34840/37465 | Loss: 1.7185 | LR: 6.28e-06 | 6352.5ms/step | 3.1 samples/s
513
+ Step 34850/37465 | Loss: 1.8751 | LR: 6.23e-06 | 6352.0ms/step | 3.1 samples/s
514
+ Step 34860/37465 | Loss: 2.0326 | LR: 6.18e-06 | 6352.1ms/step | 3.1 samples/s
515
+ Step 34870/37465 | Loss: 2.0707 | LR: 6.14e-06 | 6360.1ms/step | 3.1 samples/s
516
+ Step 34880/37465 | Loss: 2.1198 | LR: 6.09e-06 | 6358.9ms/step | 3.1 samples/s
517
+ Step 34890/37465 | Loss: 2.0247 | LR: 6.04e-06 | 6368.6ms/step | 3.1 samples/s
518
+ Step 34900/37465 | Loss: 2.0456 | LR: 6.00e-06 | 6362.4ms/step | 3.1 samples/s
519
+ Step 34910/37465 | Loss: 2.1094 | LR: 5.95e-06 | 6352.0ms/step | 3.1 samples/s
520
+ Step 34920/37465 | Loss: 2.0945 | LR: 5.90e-06 | 6340.0ms/step | 3.2 samples/s
521
+ Step 34930/37465 | Loss: 2.2069 | LR: 5.86e-06 | 6348.0ms/step | 3.2 samples/s
522
+ Step 34940/37465 | Loss: 2.0316 | LR: 5.81e-06 | 6350.0ms/step | 3.1 samples/s
523
+ Step 34950/37465 | Loss: 1.8233 | LR: 5.77e-06 | 6347.1ms/step | 3.2 samples/s
524
+ Step 34960/37465 | Loss: 1.9967 | LR: 5.72e-06 | 6333.0ms/step | 3.2 samples/s
525
+ Step 34970/37465 | Loss: 2.0134 | LR: 5.68e-06 | 6345.8ms/step | 3.2 samples/s
526
+ Step 34980/37465 | Loss: 1.8620 | LR: 5.63e-06 | 6347.0ms/step | 3.2 samples/s
527
+ Step 34990/37465 | Loss: 2.0194 | LR: 5.59e-06 | 6351.0ms/step | 3.1 samples/s
528
+ Step 35000/37465 | Loss: 2.0678 | LR: 5.54e-06 | 6350.5ms/step | 3.1 samples/s
529
+ Validation at step 35000: Loss = 2.0413, Perplexity = 7.70
530
+ Step 35010/37465 | Loss: 1.8370 | LR: 5.50e-06 | 7099.6ms/step | 2.8 samples/s
531
+ Step 35020/37465 | Loss: 2.2438 | LR: 5.45e-06 | 6349.7ms/step | 3.1 samples/s
532
+ Step 35030/37465 | Loss: 2.0567 | LR: 5.41e-06 | 6349.9ms/step | 3.1 samples/s
533
+ Step 35040/37465 | Loss: 2.0910 | LR: 5.36e-06 | 6342.2ms/step | 3.2 samples/s
534
+ Step 35050/37465 | Loss: 2.1301 | LR: 5.32e-06 | 6347.2ms/step | 3.2 samples/s
535
+ Step 35060/37465 | Loss: 1.8374 | LR: 5.27e-06 | 6338.7ms/step | 3.2 samples/s
536
+ Step 35070/37465 | Loss: 2.0582 | LR: 5.23e-06 | 6342.0ms/step | 3.2 samples/s
537
+ Step 35080/37465 | Loss: 1.9894 | LR: 5.19e-06 | 6350.5ms/step | 3.1 samples/s
538
+ Step 35090/37465 | Loss: 2.0325 | LR: 5.14e-06 | 6348.5ms/step | 3.2 samples/s
539
+ Step 35100/37465 | Loss: 1.9192 | LR: 5.10e-06 | 6342.1ms/step | 3.2 samples/s
540
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-35100
541
+ Step 35110/37465 | Loss: 2.0483 | LR: 5.06e-06 | 7313.7ms/step | 2.7 samples/s
542
+ Step 35120/37465 | Loss: 1.8345 | LR: 5.02e-06 | 6354.0ms/step | 3.1 samples/s
543
+ Step 35130/37465 | Loss: 2.0948 | LR: 4.97e-06 | 6352.4ms/step | 3.1 samples/s
544
+ Step 35140/37465 | Loss: 2.1485 | LR: 4.93e-06 | 6349.6ms/step | 3.1 samples/s
545
+ Step 35150/37465 | Loss: 2.1458 | LR: 4.89e-06 | 6341.7ms/step | 3.2 samples/s
546
+ Step 35160/37465 | Loss: 1.9352 | LR: 4.85e-06 | 6346.9ms/step | 3.2 samples/s
547
+ Step 35170/37465 | Loss: 1.9010 | LR: 4.80e-06 | 6356.8ms/step | 3.1 samples/s
548
+ Step 35180/37465 | Loss: 2.1307 | LR: 4.76e-06 | 6356.6ms/step | 3.1 samples/s
549
+ Step 35190/37465 | Loss: 2.1088 | LR: 4.72e-06 | 6346.7ms/step | 3.2 samples/s
550
+ Step 35200/37465 | Loss: 2.1371 | LR: 4.68e-06 | 6349.3ms/step | 3.1 samples/s
551
+ Step 35210/37465 | Loss: 1.9724 | LR: 4.64e-06 | 6343.2ms/step | 3.2 samples/s
552
+ Step 35220/37465 | Loss: 1.8846 | LR: 4.60e-06 | 6345.2ms/step | 3.2 samples/s
553
+ Step 35230/37465 | Loss: 2.0370 | LR: 4.56e-06 | 6349.2ms/step | 3.2 samples/s
554
+ Step 35240/37465 | Loss: 1.9549 | LR: 4.52e-06 | 6356.1ms/step | 3.1 samples/s
555
+ Step 35250/37465 | Loss: 2.0312 | LR: 4.48e-06 | 6351.4ms/step | 3.1 samples/s
556
+ Step 35260/37465 | Loss: 1.9206 | LR: 4.44e-06 | 6346.3ms/step | 3.2 samples/s
557
+ Step 35270/37465 | Loss: 2.0566 | LR: 4.40e-06 | 6341.8ms/step | 3.2 samples/s
558
+ Step 35280/37465 | Loss: 2.1137 | LR: 4.36e-06 | 6354.2ms/step | 3.1 samples/s
559
+ Step 35290/37465 | Loss: 1.9970 | LR: 4.32e-06 | 6345.7ms/step | 3.2 samples/s
560
+ Step 35300/37465 | Loss: 1.9420 | LR: 4.28e-06 | 6352.0ms/step | 3.1 samples/s
561
+ Step 35310/37465 | Loss: 1.9597 | LR: 4.24e-06 | 6351.0ms/step | 3.1 samples/s
562
+ Step 35320/37465 | Loss: 2.0116 | LR: 4.20e-06 | 6349.7ms/step | 3.1 samples/s
563
+ Step 35330/37465 | Loss: 1.8720 | LR: 4.16e-06 | 6357.9ms/step | 3.1 samples/s
564
+ Step 35340/37465 | Loss: 2.0228 | LR: 4.12e-06 | 6355.4ms/step | 3.1 samples/s
565
+ Step 35350/37465 | Loss: 1.9631 | LR: 4.08e-06 | 6356.8ms/step | 3.1 samples/s
566
+ Step 35360/37465 | Loss: 2.1668 | LR: 4.04e-06 | 6347.4ms/step | 3.2 samples/s
567
+ Step 35370/37465 | Loss: 2.0481 | LR: 4.01e-06 | 6350.7ms/step | 3.1 samples/s
568
+ Step 35380/37465 | Loss: 2.1017 | LR: 3.97e-06 | 6352.3ms/step | 3.1 samples/s
569
+ Step 35390/37465 | Loss: 2.2435 | LR: 3.93e-06 | 6362.8ms/step | 3.1 samples/s
570
+ Step 35400/37465 | Loss: 2.1043 | LR: 3.89e-06 | 6338.6ms/step | 3.2 samples/s
571
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-35400
572
+ Step 35410/37465 | Loss: 1.9944 | LR: 3.86e-06 | 7265.3ms/step | 2.8 samples/s
573
+ Step 35420/37465 | Loss: 2.0351 | LR: 3.82e-06 | 6356.8ms/step | 3.1 samples/s
574
+ Step 35430/37465 | Loss: 1.9804 | LR: 3.78e-06 | 6352.5ms/step | 3.1 samples/s
575
+ Step 35440/37465 | Loss: 2.1678 | LR: 3.74e-06 | 6357.1ms/step | 3.1 samples/s
576
+ Step 35450/37465 | Loss: 1.9526 | LR: 3.71e-06 | 6364.4ms/step | 3.1 samples/s
577
+ Step 35460/37465 | Loss: 2.1155 | LR: 3.67e-06 | 6353.3ms/step | 3.1 samples/s
578
+ Step 35470/37465 | Loss: 1.9827 | LR: 3.63e-06 | 6346.9ms/step | 3.2 samples/s
579
+ Step 35480/37465 | Loss: 2.1234 | LR: 3.60e-06 | 6359.4ms/step | 3.1 samples/s
580
+ Step 35490/37465 | Loss: 2.0919 | LR: 3.56e-06 | 6352.1ms/step | 3.1 samples/s
581
+ Step 35500/37465 | Loss: 1.9677 | LR: 3.53e-06 | 6359.8ms/step | 3.1 samples/s
582
+ Validation at step 35500: Loss = 2.0414, Perplexity = 7.70
583
+ Step 35510/37465 | Loss: 1.8975 | LR: 3.49e-06 | 7103.2ms/step | 2.8 samples/s
584
+ Step 35520/37465 | Loss: 1.8746 | LR: 3.45e-06 | 6356.4ms/step | 3.1 samples/s
585
+ Step 35530/37465 | Loss: 1.9902 | LR: 3.42e-06 | 6346.3ms/step | 3.2 samples/s
586
+ Step 35540/37465 | Loss: 2.0461 | LR: 3.38e-06 | 6351.3ms/step | 3.1 samples/s
587
+ Step 35550/37465 | Loss: 2.1079 | LR: 3.35e-06 | 6358.2ms/step | 3.1 samples/s
588
+ Step 35560/37465 | Loss: 2.1662 | LR: 3.31e-06 | 6355.4ms/step | 3.1 samples/s
589
+ Step 35570/37465 | Loss: 1.9908 | LR: 3.28e-06 | 6354.9ms/step | 3.1 samples/s
590
+ Step 35580/37465 | Loss: 2.1052 | LR: 3.24e-06 | 6351.3ms/step | 3.1 samples/s
591
+ Step 35590/37465 | Loss: 2.0468 | LR: 3.21e-06 | 6352.4ms/step | 3.1 samples/s
592
+ Step 35600/37465 | Loss: 2.0012 | LR: 3.18e-06 | 6353.2ms/step | 3.1 samples/s
593
+ Step 35610/37465 | Loss: 1.9592 | LR: 3.14e-06 | 6349.4ms/step | 3.1 samples/s
594
+ Step 35620/37465 | Loss: 2.0043 | LR: 3.11e-06 | 6360.2ms/step | 3.1 samples/s
595
+ Step 35630/37465 | Loss: 2.1477 | LR: 3.08e-06 | 6339.9ms/step | 3.2 samples/s
596
+ Step 35640/37465 | Loss: 2.0507 | LR: 3.04e-06 | 6346.7ms/step | 3.2 samples/s
597
+ Step 35650/37465 | Loss: 1.9736 | LR: 3.01e-06 | 6352.8ms/step | 3.1 samples/s
598
+ Step 35660/37465 | Loss: 1.9859 | LR: 2.98e-06 | 6345.4ms/step | 3.2 samples/s
599
+ Step 35670/37465 | Loss: 2.0515 | LR: 2.94e-06 | 6349.3ms/step | 3.1 samples/s
600
+ Step 35680/37465 | Loss: 2.1470 | LR: 2.91e-06 | 6340.2ms/step | 3.2 samples/s
601
+ Step 35690/37465 | Loss: 2.0735 | LR: 2.88e-06 | 6347.6ms/step | 3.2 samples/s
602
+ Step 35700/37465 | Loss: 2.1912 | LR: 2.85e-06 | 6353.9ms/step | 3.1 samples/s
603
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-35700
604
+ Step 35710/37465 | Loss: 1.9451 | LR: 2.81e-06 | 7477.7ms/step | 2.7 samples/s
605
+ Step 35720/37465 | Loss: 2.0755 | LR: 2.78e-06 | 6354.8ms/step | 3.1 samples/s
606
+ Step 35730/37465 | Loss: 2.1606 | LR: 2.75e-06 | 6340.6ms/step | 3.2 samples/s
607
+ Step 35740/37465 | Loss: 2.0522 | LR: 2.72e-06 | 6355.6ms/step | 3.1 samples/s
608
+ Step 35750/37465 | Loss: 1.9770 | LR: 2.69e-06 | 6351.7ms/step | 3.1 samples/s
609
+ Step 35760/37465 | Loss: 2.3300 | LR: 2.66e-06 | 6351.3ms/step | 3.1 samples/s
610
+ Step 35770/37465 | Loss: 2.3039 | LR: 2.62e-06 | 6352.3ms/step | 3.1 samples/s
611
+ Step 35780/37465 | Loss: 2.0162 | LR: 2.59e-06 | 6357.3ms/step | 3.1 samples/s
612
+ Step 35790/37465 | Loss: 2.0115 | LR: 2.56e-06 | 6358.1ms/step | 3.1 samples/s
613
+ Step 35800/37465 | Loss: 2.0343 | LR: 2.53e-06 | 6357.1ms/step | 3.1 samples/s
614
+ Step 35810/37465 | Loss: 2.1170 | LR: 2.50e-06 | 6359.4ms/step | 3.1 samples/s
615
+ Step 35820/37465 | Loss: 2.0436 | LR: 2.47e-06 | 6342.1ms/step | 3.2 samples/s
616
+ Step 35830/37465 | Loss: 2.0347 | LR: 2.44e-06 | 6338.3ms/step | 3.2 samples/s
617
+ Step 35840/37465 | Loss: 1.9483 | LR: 2.41e-06 | 6345.4ms/step | 3.2 samples/s
618
+ Step 35850/37465 | Loss: 1.9711 | LR: 2.38e-06 | 6345.7ms/step | 3.2 samples/s
619
+ Step 35860/37465 | Loss: 1.9526 | LR: 2.35e-06 | 6350.4ms/step | 3.1 samples/s
620
+ Step 35870/37465 | Loss: 1.9778 | LR: 2.32e-06 | 6346.0ms/step | 3.2 samples/s
621
+ Step 35880/37465 | Loss: 2.3312 | LR: 2.30e-06 | 6351.4ms/step | 3.1 samples/s
622
+ Step 35890/37465 | Loss: 2.1512 | LR: 2.27e-06 | 6345.6ms/step | 3.2 samples/s
623
+ Step 35900/37465 | Loss: 2.0652 | LR: 2.24e-06 | 6347.5ms/step | 3.2 samples/s
624
+ Step 35910/37465 | Loss: 2.0295 | LR: 2.21e-06 | 6343.2ms/step | 3.2 samples/s
625
+ Step 35920/37465 | Loss: 1.9491 | LR: 2.18e-06 | 6344.4ms/step | 3.2 samples/s
626
+ Step 35930/37465 | Loss: 1.9461 | LR: 2.15e-06 | 6352.6ms/step | 3.1 samples/s
627
+ Step 35940/37465 | Loss: 2.0069 | LR: 2.13e-06 | 6351.4ms/step | 3.1 samples/s
628
+ Step 35950/37465 | Loss: 2.0849 | LR: 2.10e-06 | 6352.9ms/step | 3.1 samples/s
629
+ Step 35960/37465 | Loss: 1.9878 | LR: 2.07e-06 | 6345.3ms/step | 3.2 samples/s
630
+ Step 35970/37465 | Loss: 2.0737 | LR: 2.04e-06 | 6346.8ms/step | 3.2 samples/s
631
+ Step 35980/37465 | Loss: 2.0967 | LR: 2.02e-06 | 6341.2ms/step | 3.2 samples/s
632
+ Step 35990/37465 | Loss: 2.0994 | LR: 1.99e-06 | 6348.1ms/step | 3.2 samples/s
633
+ Step 36000/37465 | Loss: 1.8895 | LR: 1.96e-06 | 6350.6ms/step | 3.1 samples/s
634
+ Validation at step 36000: Loss = 2.0413, Perplexity = 7.70
635
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-36000
636
+ Step 36010/37465 | Loss: 1.9981 | LR: 1.94e-06 | 8113.7ms/step | 2.5 samples/s
637
+ Step 36020/37465 | Loss: 2.1870 | LR: 1.91e-06 | 6355.5ms/step | 3.1 samples/s
638
+ Step 36030/37465 | Loss: 1.8664 | LR: 1.88e-06 | 6353.0ms/step | 3.1 samples/s
639
+ Step 36040/37465 | Loss: 2.0292 | LR: 1.86e-06 | 6349.9ms/step | 3.1 samples/s
640
+ Step 36050/37465 | Loss: 2.1506 | LR: 1.83e-06 | 6351.9ms/step | 3.1 samples/s
641
+ Step 36060/37465 | Loss: 2.1355 | LR: 1.80e-06 | 6360.7ms/step | 3.1 samples/s
642
+ Step 36070/37465 | Loss: 2.0087 | LR: 1.78e-06 | 6364.4ms/step | 3.1 samples/s
643
+ Step 36080/37465 | Loss: 2.3952 | LR: 1.75e-06 | 6359.7ms/step | 3.1 samples/s
644
+ Step 36090/37465 | Loss: 2.1291 | LR: 1.73e-06 | 6351.1ms/step | 3.1 samples/s
645
+ Step 36100/37465 | Loss: 2.1578 | LR: 1.70e-06 | 6348.2ms/step | 3.2 samples/s
646
+ Step 36110/37465 | Loss: 1.9152 | LR: 1.68e-06 | 6336.2ms/step | 3.2 samples/s
647
+ Step 36120/37465 | Loss: 2.1835 | LR: 1.65e-06 | 6351.9ms/step | 3.1 samples/s
648
+ Step 36130/37465 | Loss: 1.9884 | LR: 1.63e-06 | 6348.9ms/step | 3.2 samples/s
649
+ Step 36140/37465 | Loss: 1.9562 | LR: 1.61e-06 | 6345.8ms/step | 3.2 samples/s
650
+ Step 36150/37465 | Loss: 2.0299 | LR: 1.58e-06 | 6351.3ms/step | 3.1 samples/s
651
+ Step 36160/37465 | Loss: 2.2244 | LR: 1.56e-06 | 6357.0ms/step | 3.1 samples/s
652
+ Step 36170/37465 | Loss: 2.1127 | LR: 1.53e-06 | 6334.8ms/step | 3.2 samples/s
653
+ Step 36180/37465 | Loss: 2.1912 | LR: 1.51e-06 | 6345.9ms/step | 3.2 samples/s
654
+ Step 36190/37465 | Loss: 2.1821 | LR: 1.49e-06 | 6353.6ms/step | 3.1 samples/s
655
+ Step 36200/37465 | Loss: 2.0565 | LR: 1.46e-06 | 6357.9ms/step | 3.1 samples/s
656
+ Step 36210/37465 | Loss: 1.9336 | LR: 1.44e-06 | 6350.9ms/step | 3.1 samples/s
657
+ Step 36220/37465 | Loss: 1.7064 | LR: 1.42e-06 | 6337.8ms/step | 3.2 samples/s
658
+ Step 36230/37465 | Loss: 2.0606 | LR: 1.39e-06 | 6352.8ms/step | 3.1 samples/s
659
+ Step 36240/37465 | Loss: 2.3356 | LR: 1.37e-06 | 6351.7ms/step | 3.1 samples/s
660
+ Step 36250/37465 | Loss: 1.9251 | LR: 1.35e-06 | 6358.2ms/step | 3.1 samples/s
661
+ Step 36260/37465 | Loss: 2.0155 | LR: 1.33e-06 | 6358.7ms/step | 3.1 samples/s
662
+ Step 36270/37465 | Loss: 2.0124 | LR: 1.31e-06 | 6348.9ms/step | 3.2 samples/s
663
+ Step 36280/37465 | Loss: 1.9101 | LR: 1.28e-06 | 6353.4ms/step | 3.1 samples/s
664
+ Step 36290/37465 | Loss: 2.1520 | LR: 1.26e-06 | 6353.8ms/step | 3.1 samples/s
665
+ Step 36300/37465 | Loss: 2.1033 | LR: 1.24e-06 | 6356.7ms/step | 3.1 samples/s
666
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-36300
667
+ Step 36310/37465 | Loss: 1.9960 | LR: 1.22e-06 | 7268.8ms/step | 2.8 samples/s
668
+ Step 36320/37465 | Loss: 1.9368 | LR: 1.20e-06 | 6353.8ms/step | 3.1 samples/s
669
+ Step 36330/37465 | Loss: 2.0808 | LR: 1.18e-06 | 6347.9ms/step | 3.2 samples/s
670
+ Step 36340/37465 | Loss: 2.0887 | LR: 1.16e-06 | 6355.1ms/step | 3.1 samples/s
671
+ Step 36350/37465 | Loss: 2.1034 | LR: 1.14e-06 | 6350.4ms/step | 3.1 samples/s
672
+ Step 36360/37465 | Loss: 1.9993 | LR: 1.12e-06 | 6350.4ms/step | 3.1 samples/s
673
+ Step 36370/37465 | Loss: 2.2218 | LR: 1.10e-06 | 6344.9ms/step | 3.2 samples/s
674
+ Step 36380/37465 | Loss: 2.0293 | LR: 1.08e-06 | 6344.3ms/step | 3.2 samples/s
675
+ Step 36390/37465 | Loss: 1.9864 | LR: 1.06e-06 | 6343.1ms/step | 3.2 samples/s
676
+ Step 36400/37465 | Loss: 1.9582 | LR: 1.04e-06 | 6341.9ms/step | 3.2 samples/s
677
+ Step 36410/37465 | Loss: 2.0916 | LR: 1.02e-06 | 6344.8ms/step | 3.2 samples/s
678
+ Step 36420/37465 | Loss: 2.1389 | LR: 9.99e-07 | 6356.4ms/step | 3.1 samples/s
679
+ Step 36430/37465 | Loss: 2.0685 | LR: 9.80e-07 | 6347.9ms/step | 3.2 samples/s
680
+ Step 36440/37465 | Loss: 1.9361 | LR: 9.61e-07 | 6332.7ms/step | 3.2 samples/s
681
+ Step 36450/37465 | Loss: 2.1789 | LR: 9.42e-07 | 6351.5ms/step | 3.1 samples/s
682
+ Step 36460/37465 | Loss: 1.9594 | LR: 9.24e-07 | 6345.2ms/step | 3.2 samples/s
683
+ Step 36470/37465 | Loss: 2.1282 | LR: 9.06e-07 | 6353.5ms/step | 3.1 samples/s
684
+ Step 36480/37465 | Loss: 1.8607 | LR: 8.87e-07 | 6351.3ms/step | 3.1 samples/s
685
+ Step 36490/37465 | Loss: 2.1198 | LR: 8.70e-07 | 6352.8ms/step | 3.1 samples/s
686
+ Step 36500/37465 | Loss: 1.9874 | LR: 8.52e-07 | 6359.2ms/step | 3.1 samples/s
687
+ Validation at step 36500: Loss = 2.0414, Perplexity = 7.70
688
+ Step 36510/37465 | Loss: 1.9368 | LR: 8.34e-07 | 7095.0ms/step | 2.8 samples/s
689
+ Step 36520/37465 | Loss: 1.9590 | LR: 8.17e-07 | 6344.2ms/step | 3.2 samples/s
690
+ Step 36530/37465 | Loss: 2.0067 | LR: 8.00e-07 | 6341.5ms/step | 3.2 samples/s
691
+ Step 36540/37465 | Loss: 1.9926 | LR: 7.83e-07 | 6347.7ms/step | 3.2 samples/s
692
+ Step 36550/37465 | Loss: 2.1092 | LR: 7.66e-07 | 6347.4ms/step | 3.2 samples/s
693
+ Step 36560/37465 | Loss: 2.0324 | LR: 7.49e-07 | 6352.3ms/step | 3.1 samples/s
694
+ Step 36570/37465 | Loss: 1.9475 | LR: 7.33e-07 | 6355.9ms/step | 3.1 samples/s
695
+ Step 36580/37465 | Loss: 1.9221 | LR: 7.16e-07 | 6352.0ms/step | 3.1 samples/s
696
+ Step 36590/37465 | Loss: 2.0635 | LR: 7.00e-07 | 6348.8ms/step | 3.2 samples/s
697
+ Step 36600/37465 | Loss: 2.0129 | LR: 6.84e-07 | 6341.5ms/step | 3.2 samples/s
698
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-36600
699
+ Step 36610/37465 | Loss: 2.2325 | LR: 6.69e-07 | 7292.1ms/step | 2.7 samples/s
700
+ Step 36620/37465 | Loss: 1.9253 | LR: 6.53e-07 | 6342.4ms/step | 3.2 samples/s
701
+ Step 36630/37465 | Loss: 2.1147 | LR: 6.38e-07 | 6339.3ms/step | 3.2 samples/s
702
+ Step 36640/37465 | Loss: 1.8685 | LR: 6.23e-07 | 6349.8ms/step | 3.1 samples/s
703
+ Step 36650/37465 | Loss: 2.0305 | LR: 6.08e-07 | 6334.2ms/step | 3.2 samples/s
704
+ Step 36660/37465 | Loss: 1.7994 | LR: 5.93e-07 | 6353.8ms/step | 3.1 samples/s
705
+ Step 36670/37465 | Loss: 2.0917 | LR: 5.78e-07 | 6351.4ms/step | 3.1 samples/s
706
+ Step 36680/37465 | Loss: 1.9127 | LR: 5.64e-07 | 6350.0ms/step | 3.1 samples/s
707
+ Step 36690/37465 | Loss: 1.9256 | LR: 5.49e-07 | 6349.6ms/step | 3.1 samples/s
708
+ Step 36700/37465 | Loss: 1.9185 | LR: 5.35e-07 | 6342.7ms/step | 3.2 samples/s
709
+ Step 36710/37465 | Loss: 2.0190 | LR: 5.22e-07 | 6350.1ms/step | 3.1 samples/s
710
+ Step 36720/37465 | Loss: 2.1390 | LR: 5.08e-07 | 6357.7ms/step | 3.1 samples/s
711
+ Step 36730/37465 | Loss: 2.2081 | LR: 4.94e-07 | 6346.1ms/step | 3.2 samples/s
712
+ Step 36740/37465 | Loss: 1.9835 | LR: 4.81e-07 | 6349.9ms/step | 3.1 samples/s
713
+ Step 36750/37465 | Loss: 2.2151 | LR: 4.68e-07 | 6350.9ms/step | 3.1 samples/s
714
+ Step 36760/37465 | Loss: 1.9067 | LR: 4.55e-07 | 6349.9ms/step | 3.1 samples/s
715
+ Step 36770/37465 | Loss: 2.2052 | LR: 4.42e-07 | 6346.9ms/step | 3.2 samples/s
716
+ Step 36780/37465 | Loss: 2.0928 | LR: 4.29e-07 | 6345.8ms/step | 3.2 samples/s
717
+ Step 36790/37465 | Loss: 1.9557 | LR: 4.17e-07 | 6351.8ms/step | 3.1 samples/s
718
+ Step 36800/37465 | Loss: 2.1003 | LR: 4.05e-07 | 6342.6ms/step | 3.2 samples/s
719
+ Step 36810/37465 | Loss: 2.0482 | LR: 3.93e-07 | 6349.6ms/step | 3.1 samples/s
720
+ Step 36820/37465 | Loss: 1.8424 | LR: 3.81e-07 | 6357.8ms/step | 3.1 samples/s
721
+ Step 36830/37465 | Loss: 2.1118 | LR: 3.69e-07 | 6346.1ms/step | 3.2 samples/s
722
+ Step 36840/37465 | Loss: 1.8667 | LR: 3.57e-07 | 6343.0ms/step | 3.2 samples/s
723
+ Step 36850/37465 | Loss: 1.9869 | LR: 3.46e-07 | 6342.8ms/step | 3.2 samples/s
724
+ Step 36860/37465 | Loss: 2.0897 | LR: 3.35e-07 | 6342.8ms/step | 3.2 samples/s
725
+ Step 36870/37465 | Loss: 1.9297 | LR: 3.24e-07 | 6339.1ms/step | 3.2 samples/s
726
+ Step 36880/37465 | Loss: 2.2025 | LR: 3.13e-07 | 6350.8ms/step | 3.1 samples/s
727
+ Step 36890/37465 | Loss: 2.0214 | LR: 3.03e-07 | 6350.9ms/step | 3.1 samples/s
728
+ Step 36900/37465 | Loss: 1.9895 | LR: 2.92e-07 | 6348.5ms/step | 3.2 samples/s
729
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-36900
730
+ Step 36910/37465 | Loss: 1.9685 | LR: 2.82e-07 | 7338.6ms/step | 2.7 samples/s
731
+ Step 36920/37465 | Loss: 2.0000 | LR: 2.72e-07 | 6339.6ms/step | 3.2 samples/s
732
+ Step 36930/37465 | Loss: 2.1334 | LR: 2.62e-07 | 6364.4ms/step | 3.1 samples/s
733
+ Step 36940/37465 | Loss: 2.0676 | LR: 2.52e-07 | 6352.1ms/step | 3.1 samples/s
734
+ Step 36950/37465 | Loss: 2.1164 | LR: 2.43e-07 | 6355.3ms/step | 3.1 samples/s
735
+ Step 36960/37465 | Loss: 2.1135 | LR: 2.33e-07 | 6361.4ms/step | 3.1 samples/s
736
+ Step 36970/37465 | Loss: 2.1677 | LR: 2.24e-07 | 6352.7ms/step | 3.1 samples/s
737
+ Step 36980/37465 | Loss: 1.8210 | LR: 2.15e-07 | 6340.5ms/step | 3.2 samples/s
738
+ Step 36990/37465 | Loss: 2.0274 | LR: 2.06e-07 | 6335.7ms/step | 3.2 samples/s
739
+ Step 37000/37465 | Loss: 1.9714 | LR: 1.98e-07 | 6353.2ms/step | 3.1 samples/s
740
+ Validation at step 37000: Loss = 2.0413, Perplexity = 7.70
741
+ Step 37010/37465 | Loss: 1.9118 | LR: 1.89e-07 | 7098.2ms/step | 2.8 samples/s
742
+ Step 37020/37465 | Loss: 2.2364 | LR: 1.81e-07 | 6354.1ms/step | 3.1 samples/s
743
+ Step 37030/37465 | Loss: 2.1182 | LR: 1.73e-07 | 6344.8ms/step | 3.2 samples/s
744
+ Step 37040/37465 | Loss: 2.1544 | LR: 1.65e-07 | 6350.9ms/step | 3.1 samples/s
745
+ Step 37050/37465 | Loss: 1.9731 | LR: 1.58e-07 | 6348.8ms/step | 3.2 samples/s
746
+ Step 37060/37465 | Loss: 1.6761 | LR: 1.50e-07 | 6349.7ms/step | 3.1 samples/s
747
+ Step 37070/37465 | Loss: 2.0045 | LR: 1.43e-07 | 6347.5ms/step | 3.2 samples/s
748
+ Step 37080/37465 | Loss: 2.0349 | LR: 1.36e-07 | 6352.6ms/step | 3.1 samples/s
749
+ Step 37090/37465 | Loss: 2.1808 | LR: 1.29e-07 | 6349.6ms/step | 3.1 samples/s
750
+ Step 37100/37465 | Loss: 2.0562 | LR: 1.22e-07 | 6343.5ms/step | 3.2 samples/s
751
+ Step 37110/37465 | Loss: 1.9890 | LR: 1.15e-07 | 6337.9ms/step | 3.2 samples/s
752
+ Step 37120/37465 | Loss: 1.8814 | LR: 1.09e-07 | 6337.9ms/step | 3.2 samples/s
753
+ Step 37130/37465 | Loss: 1.9742 | LR: 1.03e-07 | 6344.4ms/step | 3.2 samples/s
754
+ Step 37140/37465 | Loss: 2.1938 | LR: 9.67e-08 | 6342.4ms/step | 3.2 samples/s
755
+ Step 37150/37465 | Loss: 2.1539 | LR: 9.08e-08 | 6353.7ms/step | 3.1 samples/s
756
+ Step 37160/37465 | Loss: 2.0368 | LR: 8.51e-08 | 6354.1ms/step | 3.1 samples/s
757
+ Step 37170/37465 | Loss: 1.9599 | LR: 7.96e-08 | 6355.9ms/step | 3.1 samples/s
758
+ Step 37180/37465 | Loss: 1.9386 | LR: 7.43e-08 | 6350.2ms/step | 3.1 samples/s
759
+ Step 37190/37465 | Loss: 1.8829 | LR: 6.92e-08 | 6354.7ms/step | 3.1 samples/s
760
+ Step 37200/37465 | Loss: 2.2146 | LR: 6.43e-08 | 6348.4ms/step | 3.2 samples/s
761
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-37200
762
+ Step 37210/37465 | Loss: 2.0545 | LR: 5.95e-08 | 7310.8ms/step | 2.7 samples/s
763
+ Step 37220/37465 | Loss: 1.9341 | LR: 5.49e-08 | 6345.6ms/step | 3.2 samples/s
764
+ Step 37230/37465 | Loss: 2.0255 | LR: 5.05e-08 | 6357.9ms/step | 3.1 samples/s
765
+ Step 37240/37465 | Loss: 2.0351 | LR: 4.63e-08 | 6343.6ms/step | 3.2 samples/s
766
+ Step 37250/37465 | Loss: 1.9162 | LR: 4.23e-08 | 6350.1ms/step | 3.1 samples/s
767
+ Step 37260/37465 | Loss: 2.0807 | LR: 3.85e-08 | 6352.0ms/step | 3.1 samples/s
768
+ Step 37270/37465 | Loss: 2.0201 | LR: 3.48e-08 | 6342.4ms/step | 3.2 samples/s
769
+ Step 37280/37465 | Loss: 2.2910 | LR: 3.13e-08 | 6344.9ms/step | 3.2 samples/s
770
+ Step 37290/37465 | Loss: 2.0166 | LR: 2.80e-08 | 6347.0ms/step | 3.2 samples/s
771
+ Step 37300/37465 | Loss: 2.0478 | LR: 2.49e-08 | 6350.6ms/step | 3.1 samples/s
772
+ Step 37310/37465 | Loss: 1.9402 | LR: 2.20e-08 | 6351.9ms/step | 3.1 samples/s
773
+ Step 37320/37465 | Loss: 2.0726 | LR: 1.92e-08 | 6345.6ms/step | 3.2 samples/s
774
+ Step 37330/37465 | Loss: 2.0370 | LR: 1.67e-08 | 6347.1ms/step | 3.2 samples/s
775
+ Step 37340/37465 | Loss: 1.8319 | LR: 1.43e-08 | 6346.2ms/step | 3.2 samples/s
776
+ Step 37350/37465 | Loss: 1.8073 | LR: 1.21e-08 | 6350.0ms/step | 3.1 samples/s
777
+ Step 37360/37465 | Loss: 1.9206 | LR: 1.01e-08 | 6355.5ms/step | 3.1 samples/s
778
+ Step 37370/37465 | Loss: 1.7457 | LR: 8.26e-09 | 6349.6ms/step | 3.1 samples/s
779
+ Step 37380/37465 | Loss: 1.9409 | LR: 6.61e-09 | 6344.6ms/step | 3.2 samples/s
780
+ Step 37390/37465 | Loss: 1.9385 | LR: 5.15e-09 | 6361.9ms/step | 3.1 samples/s
781
+ Step 37400/37465 | Loss: 2.1749 | LR: 3.87e-09 | 6344.4ms/step | 3.2 samples/s
782
+ Step 37410/37465 | Loss: 2.0426 | LR: 2.77e-09 | 6361.2ms/step | 3.1 samples/s
783
+ Step 37420/37465 | Loss: 2.0960 | LR: 1.85e-09 | 6360.8ms/step | 3.1 samples/s
784
+ Step 37430/37465 | Loss: 2.0587 | LR: 1.12e-09 | 6343.0ms/step | 3.2 samples/s
785
+ Step 37440/37465 | Loss: 1.9785 | LR: 5.72e-10 | 6351.7ms/step | 3.1 samples/s
786
+ Step 37450/37465 | Loss: 2.0139 | LR: 2.06e-10 | 6354.4ms/step | 3.1 samples/s
787
+ Step 37460/37465 | Loss: 1.9186 | LR: 2.29e-11 | 6346.9ms/step | 3.2 samples/s
788
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750/checkpoint-37465
epfl_final/SparseAtt/wandb/run-20260730_180915-0a2yphds/files/requirements.txt ADDED
@@ -0,0 +1,142 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ pip==26.0.1
2
+ nvidia-cusparselt-cu13==0.8.0
3
+ mpmath==1.3.0
4
+ cuda-toolkit==13.0.2
5
+ zstandard==0.25.0
6
+ xxhash==3.7.0
7
+ urllib3==2.6.3
8
+ typing_extensions==4.15.0
9
+ triton==3.6.0
10
+ tqdm==4.67.3
11
+ torchinfo==1.8.0
12
+ sympy==1.14.0
13
+ smmap==5.0.3
14
+ six==1.17.0
15
+ shellingham==1.5.4
16
+ setuptools==81.0.0
17
+ safetensors==0.7.0
18
+ regex==2026.4.4
19
+ PyYAML==6.0.3
20
+ Pygments==2.20.0
21
+ pyarrow==24.0.0
22
+ psutil==7.2.2
23
+ protobuf==7.34.1
24
+ propcache==0.4.1
25
+ platformdirs==4.9.6
26
+ packaging==26.2
27
+ nvidia-nvtx==13.0.85
28
+ nvidia-nvshmem-cu13==3.4.5
29
+ nvidia-nvjitlink==13.0.88
30
+ nvidia-nccl-cu13==2.28.9
31
+ nvidia-curand==10.4.0.35
32
+ nvidia-cufile==1.15.1.6
33
+ nvidia-cuda-runtime==13.0.96
34
+ nvidia-cuda-nvrtc==13.0.88
35
+ nvidia-cuda-cupti==13.0.85
36
+ nvidia-cublas==13.1.0.3
37
+ numpy==2.4.4
38
+ networkx==3.6.1
39
+ multidict==6.7.1
40
+ mdurl==0.1.2
41
+ MarkupSafe==3.0.3
42
+ idna==3.13
43
+ hf-xet==1.4.3
44
+ h11==0.16.0
45
+ fsspec==2026.2.0
46
+ frozenlist==1.8.0
47
+ filelock==3.29.0
48
+ dill==0.4.1
49
+ cuda-pathfinder==1.5.4
50
+ click==8.3.3
51
+ charset-normalizer==3.4.7
52
+ certifi==2026.4.22
53
+ attrs==26.1.0
54
+ annotated-types==0.7.0
55
+ annotated-doc==0.0.4
56
+ aiohappyeyeballs==2.6.1
57
+ yarl==1.23.0
58
+ typing-inspection==0.4.2
59
+ sentry-sdk==2.58.0
60
+ requests==2.33.1
61
+ python-dateutil==2.9.0.post0
62
+ pydantic_core==2.46.3
63
+ nvidia-cusparse==12.6.3.3
64
+ nvidia-cufft==12.0.0.61
65
+ nvidia-cudnn-cu13==9.19.0.56
66
+ multiprocess==0.70.19
67
+ markdown-it-py==4.0.0
68
+ Jinja2==3.1.6
69
+ httpcore==1.0.9
70
+ gitdb==4.0.12
71
+ cuda-bindings==13.2.0
72
+ anyio==4.13.0
73
+ aiosignal==1.4.0
74
+ rich==15.0.0
75
+ pydantic==2.13.3
76
+ pandas==3.0.2
77
+ nvidia-cusolver==12.0.4.66
78
+ httpx==0.28.1
79
+ GitPython==3.1.49
80
+ aiohttp==3.13.5
81
+ wandb==0.26.1
82
+ typer==0.25.1
83
+ torch==2.11.0
84
+ huggingface_hub==1.13.0
85
+ tokenizers==0.22.2
86
+ datasets==4.8.5
87
+ accelerate==1.13.0
88
+ transformers==5.7.0
89
+ word2number==1.1
90
+ sqlitedict==2.1.0
91
+ pytz==2026.2
92
+ threadpoolctl==3.6.0
93
+ tcolorpy==0.1.7
94
+ tabulate==0.10.0
95
+ scipy==1.17.1
96
+ portalocker==3.2.0
97
+ pathvalidate==3.3.1
98
+ narwhals==2.22.1
99
+ more-itertools==11.1.0
100
+ lxml==6.1.1
101
+ joblib==1.5.3
102
+ colorama==0.4.6
103
+ chardet==6.0.0.post1
104
+ absl-py==2.4.0
105
+ scikit-learn==1.9.0
106
+ sacrebleu==2.6.0
107
+ nltk==3.9.4
108
+ mbstrdecoder==1.1.5
109
+ typepy==1.3.5
110
+ rouge_score==0.1.2
111
+ DataProperty==1.1.1
112
+ tabledata==1.3.5
113
+ evaluate==0.4.6
114
+ pytablewriter==1.2.1
115
+ lm_eval==0.4.12
116
+ pluggy==1.6.0
117
+ iniconfig==2.3.0
118
+ pytest==9.1.0
119
+ py-spy==0.4.2
120
+ pillow==12.2.0
121
+ tifffile==2026.6.1
122
+ imagecodecs==2026.6.26
123
+ zipp==4.1.0
124
+ importlib_metadata==9.0.0
125
+ derivative==0.6.3
126
+ pysindy==2.1.0
127
+ pyparsing==3.3.2
128
+ kiwisolver==1.5.0
129
+ fonttools==4.63.0
130
+ cycler==0.12.1
131
+ contourpy==1.3.3
132
+ matplotlib==3.11.0
133
+ sortedcontainers==2.4.0
134
+ wcwidth==0.8.2
135
+ sigfig==1.3.19
136
+ importlib_resources==7.1.0
137
+ enum-tools==0.13.0
138
+ cramjam==2.11.0
139
+ python-snappy==0.7.3
140
+ compact-json==1.8.2
141
+ numbers-parser==4.18.5
142
+ entmax==1.3
epfl_final/SparseAtt/wandb/run-20260730_180915-0a2yphds/files/wandb-metadata.json ADDED
@@ -0,0 +1,65 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "os": "Linux-6.8.0-124-generic-x86_64-with-glibc2.35",
3
+ "python": "CPython 3.13.13",
4
+ "startedAt": "2026-07-30T18:09:15.491336Z",
5
+ "args": [
6
+ "--config",
7
+ "/mloscratch/homes/navasard/sparse/SparseAtt/configs/native30k_1p2b_mixed_to_dense_20pct_4gpu_h100_20260730.json"
8
+ ],
9
+ "program": "/mloscratch/homes/navasard/sparse/SparseAtt/train_dense_to_mixed_exact.py",
10
+ "codePath": "train_dense_to_mixed_exact.py",
11
+ "codePathLocal": "train_dense_to_mixed_exact.py",
12
+ "git": {
13
+ "remote": "https://github.com/rubentium/TokenPrune.git",
14
+ "commit": "88298c7a418622530ca5426b22b1e5ebd10ca0cb"
15
+ },
16
+ "email": "ruben.navasardyan@mail.utoronto.ca",
17
+ "root": "/mloscratch/homes/navasard/sparse/SparseAtt",
18
+ "host": "h100-train-4-gpu-2-0-21",
19
+ "executable": "/mloscratch/homes/navasard/conda/bin/python3.13",
20
+ "cpu_count": 96,
21
+ "cpu_count_logical": 192,
22
+ "gpu": "NVIDIA H100 80GB HBM3",
23
+ "gpu_count": 4,
24
+ "disk": {
25
+ "/": {
26
+ "total": "9374931091456",
27
+ "used": "4847069007872"
28
+ }
29
+ },
30
+ "memory": {
31
+ "total": "1623005626368"
32
+ },
33
+ "gpu_nvidia": [
34
+ {
35
+ "name": "NVIDIA H100 80GB HBM3",
36
+ "memoryTotal": "85520809984",
37
+ "cudaCores": 16896,
38
+ "architecture": "Hopper",
39
+ "uuid": "GPU-32209e2e-7887-e730-7611-251622a8e78f"
40
+ },
41
+ {
42
+ "name": "NVIDIA H100 80GB HBM3",
43
+ "memoryTotal": "85520809984",
44
+ "cudaCores": 16896,
45
+ "architecture": "Hopper",
46
+ "uuid": "GPU-3512802a-d501-0a3c-9812-4ab9deb3a3a9"
47
+ },
48
+ {
49
+ "name": "NVIDIA H100 80GB HBM3",
50
+ "memoryTotal": "85520809984",
51
+ "cudaCores": 16896,
52
+ "architecture": "Hopper",
53
+ "uuid": "GPU-24436def-beee-c9f1-bd05-c97df556de08"
54
+ },
55
+ {
56
+ "name": "NVIDIA H100 80GB HBM3",
57
+ "memoryTotal": "85520809984",
58
+ "cudaCores": 16896,
59
+ "architecture": "Hopper",
60
+ "uuid": "GPU-7b827c86-854a-af51-b5d3-6aaa43307d4a"
61
+ }
62
+ ],
63
+ "cudaVersion": "13.0",
64
+ "writerId": "snpo9ay0xz8qecmcb1hy8ntecd1r96yo"
65
+ }
epfl_final/SparseAtt/wandb/run-20260730_180915-0a2yphds/files/wandb-summary.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"speed/steps_per_sec":0.15755604634133502,"train/global_step":37460,"train/epoch":0.9998665421059656,"eval/uses_dense_attention":1,"_step":37460,"eval/perplexity":7.700923962818794,"train/learning_rate":2.2880351668863596e-11,"_wandb":{"runtime":48142},"train/loss":1.918636622428894,"speed/ms_per_step":6346.9477891922,"speed/samples_per_sec":3.1511209268267,"eval/dense_interval_start":-1,"_runtime":48142.023110276,"_timestamp":1.7854830587525604e+09,"eval/dense_interval_end":-1,"eval/loss":2.0413403168320654,"eval/uses_sparse_attention":0,"eval/dense_interval_active":0}
epfl_final/SparseAtt/wandb/run-20260730_180915-0a2yphds/logs/debug-internal.log ADDED
The diff for this file is too large to render. See raw diff
 
epfl_final/SparseAtt/wandb/run-20260730_180915-0a2yphds/logs/debug.log ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2026-07-30 18:09:15,508 INFO MainThread:2894 [wandb_setup.py:_flush():81] Current SDK version is 0.26.1
2
+ 2026-07-30 18:09:15,508 INFO MainThread:2894 [wandb_setup.py:_flush():81] Configure stats pid to 2894
3
+ 2026-07-30 18:09:15,508 INFO MainThread:2894 [wandb_setup.py:_flush():81] Loading settings from environment variables
4
+ 2026-07-30 18:09:15,508 INFO MainThread:2894 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /mloscratch/homes/navasard/sparse/SparseAtt/wandb/run-20260730_180915-0a2yphds/logs/debug.log
5
+ 2026-07-30 18:09:15,508 INFO MainThread:2894 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /mloscratch/homes/navasard/sparse/SparseAtt/wandb/run-20260730_180915-0a2yphds/logs/debug-internal.log
6
+ 2026-07-30 18:09:15,508 INFO MainThread:2894 [wandb_init.py:init():850] calling init triggers
7
+ 2026-07-30 18:09:15,508 INFO MainThread:2894 [wandb_init.py:init():855] wandb.init called with sweep_config: {}
8
+ config: {'model_path': '/mloscratch/homes/navasard/qwen3_06b_base', 'model_dtype': 'bfloat16', 'init_from_scratch': True, 'train_path': '/mloscratch/homes/navasard/data/longctx30k/tokenized', 'val_path': None, 'max_seq_length': 30720, 'num_workers': 4, 'output_dir': '/mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed_to_dense_20pct/run_20260730_180750', 'resume_from_checkpoint': '/mloscratch/homes/navasard/sparse/checkpoints/l30k1_2b_mixed/run_20260721_055708/checkpoint-30000', 'force_new_run': True, 'num_epochs': 1, 'max_steps': 37465, 'per_device_batch_size': 1, 'gradient_accumulation_steps': 5, 'learning_rate': 0.0005, 'weight_decay': 0.01, 'warmup_steps': 750, 'lr_scheduler': 'cosine', 'max_grad_norm': 1.0, 'save_steps': 300, 'eval_steps': 500, 'eval_batches': 20, 'logging_steps': 10, 'seed': 42, 'verbose': False, 'checkpoint_activations': True, 'strategy': 'ddp', 'mixed_precision': 'bf16', 'wandb_enabled': True, 'wandb_project': 'stochastic_sparsity', 'wandb_name': '1.2b native30k mixed-to-dense 20pct 4gpu h100', 'wandb_tags': ['qwen3', '1.2b', '30k', 'native', 'mixed-to-dense', '20pct', '4gpu', 'h100'], 'hidden_size': 1536, 'intermediate_size': 6144, 'num_attention_heads': 12, 'num_key_value_heads': 4, 'num_hidden_layers': None, 'head_dim': None, 'attention_type': 'sparse_stochastic', 'sparse_block_size': 128, 'sparse_num_random_blocks': 1, 'sparse_mask_pool_size': 1, 'sparse_val_dense_compare': False, 'sparse_layer_indices': None, 'sparse_mask_cache_dir': None, 'sparse_mask_seed': 42, 'dense_mode': True, 'dense_step_intervals': None, '_wandb': {}}
9
+ 2026-07-30 18:09:15,508 INFO MainThread:2894 [wandb_init.py:init():898] starting backend
10
+ 2026-07-30 18:09:15,756 INFO MainThread:2894 [wandb_init.py:init():913] sending inform_init request
11
+ 2026-07-30 18:09:16,180 INFO MainThread:2894 [wandb_init.py:init():918] backend started and connected
12
+ 2026-07-30 18:09:16,182 INFO MainThread:2894 [wandb_init.py:init():988] updated telemetry
13
+ 2026-07-30 18:09:16,201 INFO MainThread:2894 [wandb_init.py:init():1011] communicating run to backend with 90.0 second timeout
14
+ 2026-07-30 18:09:20,261 INFO MainThread:2894 [wandb_init.py:init():1056] starting run threads in backend
15
+ 2026-07-30 18:09:20,402 INFO MainThread:2894 [wandb_run.py:_console_start():2554] atexit reg
16
+ 2026-07-30 18:09:20,403 INFO MainThread:2894 [wandb_run.py:_redirect():2403] redirect: wrap_raw
17
+ 2026-07-30 18:09:20,403 INFO MainThread:2894 [wandb_run.py:_redirect():2472] Wrapping output streams.
18
+ 2026-07-30 18:09:20,403 INFO MainThread:2894 [wandb_run.py:_redirect():2495] Redirects installed.
19
+ 2026-07-30 18:09:20,407 INFO MainThread:2894 [wandb_init.py:init():1094] run started, returning control to user process
20
+ 2026-07-31 07:31:42,283 INFO MainThread:2894 [wandb_run.py:_finish():2321] finishing run ruben-navasardyan4869-epfl/stochastic_sparsity/0a2yphds
21
+ 2026-07-31 07:31:42,284 INFO MainThread:2894 [wandb_run.py:_atexit_cleanup():2520] got exitcode: 0
22
+ 2026-07-31 07:31:42,284 INFO MainThread:2894 [wandb_run.py:_restore():2502] restore
23
+ 2026-07-31 07:31:42,285 INFO MainThread:2894 [wandb_run.py:_restore():2508] restore done
24
+ 2026-07-31 07:31:44,145 INFO MainThread:2894 [wandb_run.py:_footer_sync_info():3889] logging synced files
epfl_final/SparseAtt/wandb/run-20260730_180915-0a2yphds/run-0a2yphds.wandb ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:311726147901b051d734a136eafce90eab769092f3ac92078f80c956feb89d78
3
+ size 15516514
epfl_final/SparseAtt/wandb/run-20260731_001110-6wmcorvj/files/config.yaml ADDED
@@ -0,0 +1,187 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ _wandb:
2
+ value:
3
+ cli_version: 0.26.1
4
+ e:
5
+ b3v5tviubcpjzo8whpo1u5dqappqlhg6:
6
+ args:
7
+ - --config
8
+ - /mloscratch/homes/navasard/sparse/SparseAtt/configs/ladder8k/365m_mixed_to_dense_20pct_seed42_20260730.json
9
+ codePath: train_dense_to_mixed_exact.py
10
+ codePathLocal: train_dense_to_mixed_exact.py
11
+ cpu_count: 64
12
+ cpu_count_logical: 128
13
+ cudaVersion: "13.0"
14
+ disk:
15
+ /:
16
+ total: "7679362727936"
17
+ used: "5257403871232"
18
+ email: ruben.navasardyan@mail.utoronto.ca
19
+ executable: /mloscratch/homes/navasard/conda/bin/python3.13
20
+ git:
21
+ commit: 88298c7a418622530ca5426b22b1e5ebd10ca0cb
22
+ remote: https://github.com/rubentium/TokenPrune.git
23
+ gpu: NVIDIA A100-SXM4-80GB
24
+ gpu_count: 4
25
+ gpu_nvidia:
26
+ - architecture: Ampere
27
+ cudaCores: 6912
28
+ memoryTotal: "85899345920"
29
+ name: NVIDIA A100-SXM4-80GB
30
+ uuid: GPU-26e055ad-009a-c42a-2ba0-ce6db370fcbe
31
+ - architecture: Ampere
32
+ cudaCores: 6912
33
+ memoryTotal: "85899345920"
34
+ name: NVIDIA A100-SXM4-80GB
35
+ uuid: GPU-56ec1834-3584-3932-0125-388b5e88fac2
36
+ - architecture: Ampere
37
+ cudaCores: 6912
38
+ memoryTotal: "85899345920"
39
+ name: NVIDIA A100-SXM4-80GB
40
+ uuid: GPU-ce50edb9-4d69-8d20-4433-5217084c7016
41
+ - architecture: Ampere
42
+ cudaCores: 6912
43
+ memoryTotal: "85899345920"
44
+ name: NVIDIA A100-SXM4-80GB
45
+ uuid: GPU-2401254a-acb7-5787-0b06-07065349e511
46
+ host: a100-train-4-gpu-2-0-40
47
+ memory:
48
+ total: "1081884934144"
49
+ os: Linux-6.8.0-124-generic-x86_64-with-glibc2.35
50
+ program: /mloscratch/homes/navasard/sparse/SparseAtt/train_dense_to_mixed_exact.py
51
+ python: CPython 3.13.13
52
+ root: /mloscratch/homes/navasard/sparse/SparseAtt
53
+ startedAt: "2026-07-31T00:11:10.049356Z"
54
+ writerId: b3v5tviubcpjzo8whpo1u5dqappqlhg6
55
+ m: []
56
+ python_version: 3.13.13
57
+ t:
58
+ "1":
59
+ - 1
60
+ - 5
61
+ - 11
62
+ - 49
63
+ - 51
64
+ - 53
65
+ - 71
66
+ "2":
67
+ - 1
68
+ - 5
69
+ - 11
70
+ - 49
71
+ - 51
72
+ - 53
73
+ - 71
74
+ "3":
75
+ - 2
76
+ - 13
77
+ - 15
78
+ - 16
79
+ - 61
80
+ "4": 3.13.13
81
+ "5": 0.26.1
82
+ "6": 5.7.0
83
+ "12": 0.26.1
84
+ "13": linux-x86_64
85
+ attention_type:
86
+ value: sparse_stochastic
87
+ checkpoint_activations:
88
+ value: false
89
+ dense_mode:
90
+ value: true
91
+ dense_step_intervals:
92
+ value: null
93
+ eval_batches:
94
+ value: 50
95
+ eval_steps:
96
+ value: 250
97
+ force_new_run:
98
+ value: true
99
+ gradient_accumulation_steps:
100
+ value: 10
101
+ head_dim:
102
+ value: null
103
+ hidden_size:
104
+ value: 768
105
+ init_from_scratch:
106
+ value: true
107
+ intermediate_size:
108
+ value: 3072
109
+ learning_rate:
110
+ value: 0.0005
111
+ logging_steps:
112
+ value: 10
113
+ lr_scheduler:
114
+ value: cosine
115
+ max_grad_norm:
116
+ value: 1
117
+ max_seq_length:
118
+ value: 8192
119
+ max_steps:
120
+ value: 44000
121
+ mixed_precision:
122
+ value: bf16
123
+ model_dtype:
124
+ value: bfloat16
125
+ model_path:
126
+ value: /mloscratch/homes/navasard/qwen3_06b_base
127
+ num_attention_heads:
128
+ value: 6
129
+ num_epochs:
130
+ value: 1
131
+ num_hidden_layers:
132
+ value: 28
133
+ num_key_value_heads:
134
+ value: 2
135
+ num_workers:
136
+ value: 8
137
+ output_dir:
138
+ value: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031
139
+ per_device_batch_size:
140
+ value: 2
141
+ resume_from_checkpoint:
142
+ value: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_seed42/run_20260719_234149/checkpoint-35000
143
+ save_steps:
144
+ value: 250
145
+ seed:
146
+ value: 42
147
+ sparse_block_size:
148
+ value: 128
149
+ sparse_layer_indices:
150
+ value: null
151
+ sparse_mask_cache_dir:
152
+ value: null
153
+ sparse_mask_pool_size:
154
+ value: 1
155
+ sparse_mask_seed:
156
+ value: 0
157
+ sparse_num_random_blocks:
158
+ value: 1
159
+ sparse_val_dense_compare:
160
+ value: false
161
+ strategy:
162
+ value: ddp
163
+ train_path:
164
+ value: /mloscratch/homes/navasard/data/fineweb_large/tokenized
165
+ val_path:
166
+ value: null
167
+ verbose:
168
+ value: false
169
+ wandb_enabled:
170
+ value: true
171
+ wandb_name:
172
+ value: 365m mixed-to-dense 20pct seed42
173
+ wandb_project:
174
+ value: stochastic_sparsity
175
+ wandb_tags:
176
+ value:
177
+ - qwen3
178
+ - 365m
179
+ - 8k
180
+ - mixed-to-dense
181
+ - 20pct
182
+ - seed42
183
+ - multiseed
184
+ warmup_steps:
185
+ value: 750
186
+ weight_decay:
187
+ value: 0.01
epfl_final/SparseAtt/wandb/run-20260731_001110-6wmcorvj/files/output.log ADDED
@@ -0,0 +1,976 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Starting training...
2
+ Validation at step 35000: Loss = 2.9167, Perplexity = 18.48
3
+ Switching attention mode for step 35001: dense (dense step interval)
4
+ Step 35010/44000 | Loss: 2.9000 | LR: 5.14e-05 | 66936.5ms/step | 0.3 samples/s
5
+ Step 35020/44000 | Loss: 2.8658 | LR: 5.13e-05 | 3451.5ms/step | 5.8 samples/s
6
+ Step 35030/44000 | Loss: 2.8745 | LR: 5.12e-05 | 3450.1ms/step | 5.8 samples/s
7
+ Step 35040/44000 | Loss: 2.8856 | LR: 5.11e-05 | 3451.9ms/step | 5.8 samples/s
8
+ Step 35050/44000 | Loss: 2.8769 | LR: 5.10e-05 | 3455.6ms/step | 5.8 samples/s
9
+ Step 35060/44000 | Loss: 2.8859 | LR: 5.09e-05 | 3458.3ms/step | 5.8 samples/s
10
+ Step 35070/44000 | Loss: 2.8536 | LR: 5.08e-05 | 3458.5ms/step | 5.8 samples/s
11
+ Step 35080/44000 | Loss: 2.8890 | LR: 5.07e-05 | 3459.9ms/step | 5.8 samples/s
12
+ Step 35090/44000 | Loss: 2.8780 | LR: 5.06e-05 | 3461.0ms/step | 5.8 samples/s
13
+ Step 35100/44000 | Loss: 2.9204 | LR: 5.04e-05 | 3464.6ms/step | 5.8 samples/s
14
+ Step 35110/44000 | Loss: 2.8819 | LR: 5.03e-05 | 3465.2ms/step | 5.8 samples/s
15
+ Step 35120/44000 | Loss: 2.8961 | LR: 5.02e-05 | 3466.5ms/step | 5.8 samples/s
16
+ Step 35130/44000 | Loss: 2.8968 | LR: 5.01e-05 | 3462.7ms/step | 5.8 samples/s
17
+ Step 35140/44000 | Loss: 2.8640 | LR: 5.00e-05 | 3464.0ms/step | 5.8 samples/s
18
+ Step 35150/44000 | Loss: 2.9029 | LR: 4.99e-05 | 3461.3ms/step | 5.8 samples/s
19
+ Step 35160/44000 | Loss: 2.9259 | LR: 4.98e-05 | 3461.5ms/step | 5.8 samples/s
20
+ Step 35170/44000 | Loss: 2.9014 | LR: 4.97e-05 | 3460.7ms/step | 5.8 samples/s
21
+ Step 35180/44000 | Loss: 2.9095 | LR: 4.96e-05 | 3461.6ms/step | 5.8 samples/s
22
+ Step 35190/44000 | Loss: 2.8785 | LR: 4.95e-05 | 3463.2ms/step | 5.8 samples/s
23
+ Step 35200/44000 | Loss: 2.9076 | LR: 4.94e-05 | 3463.5ms/step | 5.8 samples/s
24
+ Step 35210/44000 | Loss: 2.9037 | LR: 4.93e-05 | 3464.8ms/step | 5.8 samples/s
25
+ Step 35220/44000 | Loss: 2.8816 | LR: 4.91e-05 | 3464.2ms/step | 5.8 samples/s
26
+ Step 35230/44000 | Loss: 2.8897 | LR: 4.90e-05 | 3461.3ms/step | 5.8 samples/s
27
+ Step 35240/44000 | Loss: 2.8807 | LR: 4.89e-05 | 3463.5ms/step | 5.8 samples/s
28
+ Step 35250/44000 | Loss: 2.9255 | LR: 4.88e-05 | 3468.6ms/step | 5.8 samples/s
29
+ Validation at step 35250: Loss = 2.9098, Perplexity = 18.35
30
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-35250
31
+ Step 35260/44000 | Loss: 2.8813 | LR: 4.87e-05 | 9561.8ms/step | 2.1 samples/s
32
+ Step 35270/44000 | Loss: 2.9129 | LR: 4.86e-05 | 3467.5ms/step | 5.8 samples/s
33
+ Step 35280/44000 | Loss: 2.8695 | LR: 4.85e-05 | 3465.6ms/step | 5.8 samples/s
34
+ Step 35290/44000 | Loss: 2.8772 | LR: 4.84e-05 | 3463.2ms/step | 5.8 samples/s
35
+ Step 35300/44000 | Loss: 2.8798 | LR: 4.83e-05 | 3465.7ms/step | 5.8 samples/s
36
+ Step 35310/44000 | Loss: 2.9000 | LR: 4.82e-05 | 3462.9ms/step | 5.8 samples/s
37
+ Step 35320/44000 | Loss: 2.8630 | LR: 4.81e-05 | 3463.2ms/step | 5.8 samples/s
38
+ Step 35330/44000 | Loss: 2.8988 | LR: 4.80e-05 | 3466.9ms/step | 5.8 samples/s
39
+ Step 35340/44000 | Loss: 2.8920 | LR: 4.79e-05 | 3465.9ms/step | 5.8 samples/s
40
+ Step 35350/44000 | Loss: 2.9198 | LR: 4.77e-05 | 3464.6ms/step | 5.8 samples/s
41
+ Step 35360/44000 | Loss: 2.8734 | LR: 4.76e-05 | 3466.0ms/step | 5.8 samples/s
42
+ Step 35370/44000 | Loss: 2.8875 | LR: 4.75e-05 | 3464.6ms/step | 5.8 samples/s
43
+ Step 35380/44000 | Loss: 2.8744 | LR: 4.74e-05 | 3464.4ms/step | 5.8 samples/s
44
+ Step 35390/44000 | Loss: 2.8811 | LR: 4.73e-05 | 3467.1ms/step | 5.8 samples/s
45
+ Step 35400/44000 | Loss: 2.8703 | LR: 4.72e-05 | 3464.2ms/step | 5.8 samples/s
46
+ Step 35410/44000 | Loss: 2.9036 | LR: 4.71e-05 | 3463.4ms/step | 5.8 samples/s
47
+ Step 35420/44000 | Loss: 2.8745 | LR: 4.70e-05 | 3467.8ms/step | 5.8 samples/s
48
+ Step 35430/44000 | Loss: 2.9329 | LR: 4.69e-05 | 3468.5ms/step | 5.8 samples/s
49
+ Step 35440/44000 | Loss: 2.9105 | LR: 4.68e-05 | 3463.7ms/step | 5.8 samples/s
50
+ Step 35450/44000 | Loss: 2.8946 | LR: 4.67e-05 | 3462.9ms/step | 5.8 samples/s
51
+ Step 35460/44000 | Loss: 2.9145 | LR: 4.66e-05 | 3462.2ms/step | 5.8 samples/s
52
+ Step 35470/44000 | Loss: 2.8862 | LR: 4.65e-05 | 3463.0ms/step | 5.8 samples/s
53
+ Step 35480/44000 | Loss: 2.8951 | LR: 4.64e-05 | 3463.9ms/step | 5.8 samples/s
54
+ Step 35490/44000 | Loss: 2.9010 | LR: 4.63e-05 | 3461.6ms/step | 5.8 samples/s
55
+ Step 35500/44000 | Loss: 2.8744 | LR: 4.62e-05 | 3462.4ms/step | 5.8 samples/s
56
+ Validation at step 35500: Loss = 2.9092, Perplexity = 18.34
57
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-35500
58
+ Step 35510/44000 | Loss: 2.8848 | LR: 4.61e-05 | 9543.9ms/step | 2.1 samples/s
59
+ Step 35520/44000 | Loss: 2.8937 | LR: 4.59e-05 | 3463.2ms/step | 5.8 samples/s
60
+ Step 35530/44000 | Loss: 2.8941 | LR: 4.58e-05 | 3462.5ms/step | 5.8 samples/s
61
+ Step 35540/44000 | Loss: 2.9124 | LR: 4.57e-05 | 3463.5ms/step | 5.8 samples/s
62
+ Step 35550/44000 | Loss: 2.9142 | LR: 4.56e-05 | 3464.1ms/step | 5.8 samples/s
63
+ Step 35560/44000 | Loss: 2.9022 | LR: 4.55e-05 | 3463.6ms/step | 5.8 samples/s
64
+ Step 35570/44000 | Loss: 2.9122 | LR: 4.54e-05 | 3462.8ms/step | 5.8 samples/s
65
+ Step 35580/44000 | Loss: 2.8966 | LR: 4.53e-05 | 3462.0ms/step | 5.8 samples/s
66
+ Step 35590/44000 | Loss: 2.9058 | LR: 4.52e-05 | 3466.1ms/step | 5.8 samples/s
67
+ Step 35600/44000 | Loss: 2.8710 | LR: 4.51e-05 | 3465.5ms/step | 5.8 samples/s
68
+ Step 35610/44000 | Loss: 2.9090 | LR: 4.50e-05 | 3466.4ms/step | 5.8 samples/s
69
+ Step 35620/44000 | Loss: 2.8854 | LR: 4.49e-05 | 3464.6ms/step | 5.8 samples/s
70
+ Step 35630/44000 | Loss: 2.8976 | LR: 4.48e-05 | 3466.2ms/step | 5.8 samples/s
71
+ Step 35640/44000 | Loss: 2.8745 | LR: 4.47e-05 | 3464.3ms/step | 5.8 samples/s
72
+ Step 35650/44000 | Loss: 2.8824 | LR: 4.46e-05 | 3463.4ms/step | 5.8 samples/s
73
+ Step 35660/44000 | Loss: 2.8981 | LR: 4.45e-05 | 3467.1ms/step | 5.8 samples/s
74
+ Step 35670/44000 | Loss: 2.8852 | LR: 4.44e-05 | 3465.5ms/step | 5.8 samples/s
75
+ Step 35680/44000 | Loss: 2.8914 | LR: 4.43e-05 | 3465.4ms/step | 5.8 samples/s
76
+ Step 35690/44000 | Loss: 2.9040 | LR: 4.42e-05 | 3464.5ms/step | 5.8 samples/s
77
+ Step 35700/44000 | Loss: 2.8782 | LR: 4.41e-05 | 3463.6ms/step | 5.8 samples/s
78
+ Step 35710/44000 | Loss: 2.8924 | LR: 4.40e-05 | 3465.1ms/step | 5.8 samples/s
79
+ Step 35720/44000 | Loss: 2.8747 | LR: 4.39e-05 | 3463.7ms/step | 5.8 samples/s
80
+ Step 35730/44000 | Loss: 2.8661 | LR: 4.38e-05 | 3465.4ms/step | 5.8 samples/s
81
+ Step 35740/44000 | Loss: 2.8973 | LR: 4.37e-05 | 3464.4ms/step | 5.8 samples/s
82
+ Step 35750/44000 | Loss: 2.8769 | LR: 4.36e-05 | 3465.2ms/step | 5.8 samples/s
83
+ Validation at step 35750: Loss = 2.9087, Perplexity = 18.33
84
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-35750
85
+ Step 35760/44000 | Loss: 2.9095 | LR: 4.35e-05 | 9572.2ms/step | 2.1 samples/s
86
+ Step 35770/44000 | Loss: 2.8834 | LR: 4.34e-05 | 3466.7ms/step | 5.8 samples/s
87
+ Step 35780/44000 | Loss: 2.8729 | LR: 4.33e-05 | 3464.4ms/step | 5.8 samples/s
88
+ Step 35790/44000 | Loss: 2.8984 | LR: 4.32e-05 | 3463.8ms/step | 5.8 samples/s
89
+ Step 35800/44000 | Loss: 2.9085 | LR: 4.31e-05 | 3467.9ms/step | 5.8 samples/s
90
+ Step 35810/44000 | Loss: 2.9041 | LR: 4.29e-05 | 3462.5ms/step | 5.8 samples/s
91
+ Step 35820/44000 | Loss: 2.9070 | LR: 4.28e-05 | 3463.3ms/step | 5.8 samples/s
92
+ Step 35830/44000 | Loss: 2.9253 | LR: 4.27e-05 | 3465.8ms/step | 5.8 samples/s
93
+ Step 35840/44000 | Loss: 2.8832 | LR: 4.26e-05 | 3465.5ms/step | 5.8 samples/s
94
+ Step 35850/44000 | Loss: 2.8922 | LR: 4.25e-05 | 3466.5ms/step | 5.8 samples/s
95
+ Step 35860/44000 | Loss: 2.9163 | LR: 4.24e-05 | 3464.8ms/step | 5.8 samples/s
96
+ Step 35870/44000 | Loss: 2.8937 | LR: 4.23e-05 | 3465.1ms/step | 5.8 samples/s
97
+ Step 35880/44000 | Loss: 2.9095 | LR: 4.22e-05 | 3465.9ms/step | 5.8 samples/s
98
+ Step 35890/44000 | Loss: 2.8741 | LR: 4.21e-05 | 3463.6ms/step | 5.8 samples/s
99
+ Step 35900/44000 | Loss: 2.8954 | LR: 4.20e-05 | 3465.4ms/step | 5.8 samples/s
100
+ Step 35910/44000 | Loss: 2.8785 | LR: 4.19e-05 | 3471.2ms/step | 5.8 samples/s
101
+ Step 35920/44000 | Loss: 2.8920 | LR: 4.18e-05 | 3473.9ms/step | 5.8 samples/s
102
+ Step 35930/44000 | Loss: 2.8787 | LR: 4.17e-05 | 3463.0ms/step | 5.8 samples/s
103
+ Step 35940/44000 | Loss: 2.8831 | LR: 4.16e-05 | 3464.2ms/step | 5.8 samples/s
104
+ Step 35950/44000 | Loss: 2.8683 | LR: 4.15e-05 | 3465.0ms/step | 5.8 samples/s
105
+ Step 35960/44000 | Loss: 2.8714 | LR: 4.14e-05 | 3467.2ms/step | 5.8 samples/s
106
+ Step 35970/44000 | Loss: 2.8659 | LR: 4.13e-05 | 3468.1ms/step | 5.8 samples/s
107
+ Step 35980/44000 | Loss: 2.9180 | LR: 4.12e-05 | 3467.0ms/step | 5.8 samples/s
108
+ Step 35990/44000 | Loss: 2.8885 | LR: 4.11e-05 | 3468.0ms/step | 5.8 samples/s
109
+ Step 36000/44000 | Loss: 2.8875 | LR: 4.10e-05 | 3466.5ms/step | 5.8 samples/s
110
+ Validation at step 36000: Loss = 2.9084, Perplexity = 18.33
111
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-36000
112
+ Step 36010/44000 | Loss: 2.9055 | LR: 4.09e-05 | 9550.2ms/step | 2.1 samples/s
113
+ Step 36020/44000 | Loss: 2.8812 | LR: 4.08e-05 | 3466.2ms/step | 5.8 samples/s
114
+ Step 36030/44000 | Loss: 2.8729 | LR: 4.07e-05 | 3464.7ms/step | 5.8 samples/s
115
+ Step 36040/44000 | Loss: 2.8765 | LR: 4.06e-05 | 3465.1ms/step | 5.8 samples/s
116
+ Step 36050/44000 | Loss: 2.8714 | LR: 4.05e-05 | 3467.1ms/step | 5.8 samples/s
117
+ Step 36060/44000 | Loss: 2.9038 | LR: 4.04e-05 | 3464.2ms/step | 5.8 samples/s
118
+ Step 36070/44000 | Loss: 2.8927 | LR: 4.03e-05 | 3468.3ms/step | 5.8 samples/s
119
+ Step 36080/44000 | Loss: 2.9082 | LR: 4.02e-05 | 3466.8ms/step | 5.8 samples/s
120
+ Step 36090/44000 | Loss: 2.9152 | LR: 4.01e-05 | 3465.6ms/step | 5.8 samples/s
121
+ Step 36100/44000 | Loss: 2.8948 | LR: 4.00e-05 | 3469.1ms/step | 5.8 samples/s
122
+ Step 36110/44000 | Loss: 2.9101 | LR: 3.99e-05 | 3474.4ms/step | 5.8 samples/s
123
+ Step 36120/44000 | Loss: 2.8967 | LR: 3.98e-05 | 3473.7ms/step | 5.8 samples/s
124
+ Step 36130/44000 | Loss: 2.8978 | LR: 3.97e-05 | 3466.1ms/step | 5.8 samples/s
125
+ Step 36140/44000 | Loss: 2.8894 | LR: 3.97e-05 | 3467.0ms/step | 5.8 samples/s
126
+ Step 36150/44000 | Loss: 2.8992 | LR: 3.96e-05 | 3466.0ms/step | 5.8 samples/s
127
+ Step 36160/44000 | Loss: 2.8928 | LR: 3.95e-05 | 3465.8ms/step | 5.8 samples/s
128
+ Step 36170/44000 | Loss: 2.8513 | LR: 3.94e-05 | 3467.6ms/step | 5.8 samples/s
129
+ Step 36180/44000 | Loss: 2.8917 | LR: 3.93e-05 | 3473.3ms/step | 5.8 samples/s
130
+ Step 36190/44000 | Loss: 2.8881 | LR: 3.92e-05 | 3472.5ms/step | 5.8 samples/s
131
+ Step 36200/44000 | Loss: 2.8823 | LR: 3.91e-05 | 3471.3ms/step | 5.8 samples/s
132
+ Step 36210/44000 | Loss: 2.8904 | LR: 3.90e-05 | 3474.0ms/step | 5.8 samples/s
133
+ Step 36220/44000 | Loss: 2.8854 | LR: 3.89e-05 | 3466.9ms/step | 5.8 samples/s
134
+ Step 36230/44000 | Loss: 2.8867 | LR: 3.88e-05 | 3467.7ms/step | 5.8 samples/s
135
+ Step 36240/44000 | Loss: 2.8975 | LR: 3.87e-05 | 3467.7ms/step | 5.8 samples/s
136
+ Step 36250/44000 | Loss: 2.9117 | LR: 3.86e-05 | 3466.2ms/step | 5.8 samples/s
137
+ Validation at step 36250: Loss = 2.9081, Perplexity = 18.32
138
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-36250
139
+ Step 36260/44000 | Loss: 2.9019 | LR: 3.85e-05 | 9584.0ms/step | 2.1 samples/s
140
+ Step 36270/44000 | Loss: 2.9116 | LR: 3.84e-05 | 3464.2ms/step | 5.8 samples/s
141
+ Step 36280/44000 | Loss: 2.8834 | LR: 3.83e-05 | 3467.8ms/step | 5.8 samples/s
142
+ Step 36290/44000 | Loss: 2.8996 | LR: 3.82e-05 | 3465.1ms/step | 5.8 samples/s
143
+ Step 36300/44000 | Loss: 2.8845 | LR: 3.81e-05 | 3465.8ms/step | 5.8 samples/s
144
+ Step 36310/44000 | Loss: 2.8851 | LR: 3.80e-05 | 3466.0ms/step | 5.8 samples/s
145
+ Step 36320/44000 | Loss: 2.8612 | LR: 3.79e-05 | 3471.3ms/step | 5.8 samples/s
146
+ Step 36330/44000 | Loss: 2.8794 | LR: 3.78e-05 | 3475.8ms/step | 5.8 samples/s
147
+ Step 36340/44000 | Loss: 2.8969 | LR: 3.77e-05 | 3474.1ms/step | 5.8 samples/s
148
+ Step 36350/44000 | Loss: 2.8659 | LR: 3.76e-05 | 3474.8ms/step | 5.8 samples/s
149
+ Step 36360/44000 | Loss: 2.8746 | LR: 3.75e-05 | 3468.3ms/step | 5.8 samples/s
150
+ Step 36370/44000 | Loss: 2.8845 | LR: 3.74e-05 | 3465.8ms/step | 5.8 samples/s
151
+ Step 36380/44000 | Loss: 2.8848 | LR: 3.73e-05 | 3471.4ms/step | 5.8 samples/s
152
+ Step 36390/44000 | Loss: 2.9060 | LR: 3.72e-05 | 3467.2ms/step | 5.8 samples/s
153
+ Step 36400/44000 | Loss: 2.8843 | LR: 3.71e-05 | 3466.5ms/step | 5.8 samples/s
154
+ Step 36410/44000 | Loss: 2.8936 | LR: 3.70e-05 | 3466.3ms/step | 5.8 samples/s
155
+ Step 36420/44000 | Loss: 2.8596 | LR: 3.69e-05 | 3470.9ms/step | 5.8 samples/s
156
+ Step 36430/44000 | Loss: 2.8687 | LR: 3.69e-05 | 3468.2ms/step | 5.8 samples/s
157
+ Step 36440/44000 | Loss: 2.8786 | LR: 3.68e-05 | 3466.5ms/step | 5.8 samples/s
158
+ Step 36450/44000 | Loss: 2.9199 | LR: 3.67e-05 | 3466.1ms/step | 5.8 samples/s
159
+ Step 36460/44000 | Loss: 2.8945 | LR: 3.66e-05 | 3468.2ms/step | 5.8 samples/s
160
+ Step 36470/44000 | Loss: 2.8677 | LR: 3.65e-05 | 3467.4ms/step | 5.8 samples/s
161
+ Step 36480/44000 | Loss: 2.8930 | LR: 3.64e-05 | 3466.9ms/step | 5.8 samples/s
162
+ Step 36490/44000 | Loss: 2.9157 | LR: 3.63e-05 | 3467.5ms/step | 5.8 samples/s
163
+ Step 36500/44000 | Loss: 2.8627 | LR: 3.62e-05 | 3467.9ms/step | 5.8 samples/s
164
+ Validation at step 36500: Loss = 2.9079, Perplexity = 18.32
165
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-36500
166
+ Step 36510/44000 | Loss: 2.9019 | LR: 3.61e-05 | 9553.7ms/step | 2.1 samples/s
167
+ Step 36520/44000 | Loss: 2.8772 | LR: 3.60e-05 | 3466.0ms/step | 5.8 samples/s
168
+ Step 36530/44000 | Loss: 2.8856 | LR: 3.59e-05 | 3460.9ms/step | 5.8 samples/s
169
+ Step 36540/44000 | Loss: 2.8714 | LR: 3.58e-05 | 3462.7ms/step | 5.8 samples/s
170
+ Step 36550/44000 | Loss: 2.8809 | LR: 3.57e-05 | 3463.9ms/step | 5.8 samples/s
171
+ Step 36560/44000 | Loss: 2.9019 | LR: 3.56e-05 | 3466.9ms/step | 5.8 samples/s
172
+ Step 36570/44000 | Loss: 2.8618 | LR: 3.55e-05 | 3467.0ms/step | 5.8 samples/s
173
+ Step 36580/44000 | Loss: 2.8816 | LR: 3.54e-05 | 3466.3ms/step | 5.8 samples/s
174
+ Step 36590/44000 | Loss: 2.8892 | LR: 3.53e-05 | 3466.2ms/step | 5.8 samples/s
175
+ Step 36600/44000 | Loss: 2.8630 | LR: 3.53e-05 | 3469.6ms/step | 5.8 samples/s
176
+ Step 36610/44000 | Loss: 2.8846 | LR: 3.52e-05 | 3465.9ms/step | 5.8 samples/s
177
+ Step 36620/44000 | Loss: 2.8772 | LR: 3.51e-05 | 3466.4ms/step | 5.8 samples/s
178
+ Step 36630/44000 | Loss: 2.8886 | LR: 3.50e-05 | 3467.2ms/step | 5.8 samples/s
179
+ Step 36640/44000 | Loss: 2.8882 | LR: 3.49e-05 | 3468.4ms/step | 5.8 samples/s
180
+ Step 36650/44000 | Loss: 2.8861 | LR: 3.48e-05 | 3473.0ms/step | 5.8 samples/s
181
+ Step 36660/44000 | Loss: 2.8767 | LR: 3.47e-05 | 3474.1ms/step | 5.8 samples/s
182
+ Step 36670/44000 | Loss: 2.9181 | LR: 3.46e-05 | 3471.6ms/step | 5.8 samples/s
183
+ Step 36680/44000 | Loss: 2.9032 | LR: 3.45e-05 | 3467.0ms/step | 5.8 samples/s
184
+ Step 36690/44000 | Loss: 2.8809 | LR: 3.44e-05 | 3468.4ms/step | 5.8 samples/s
185
+ Step 36700/44000 | Loss: 2.8811 | LR: 3.43e-05 | 3465.4ms/step | 5.8 samples/s
186
+ Step 36710/44000 | Loss: 2.8733 | LR: 3.42e-05 | 3465.6ms/step | 5.8 samples/s
187
+ Step 36720/44000 | Loss: 2.8880 | LR: 3.41e-05 | 3466.4ms/step | 5.8 samples/s
188
+ Step 36730/44000 | Loss: 2.8776 | LR: 3.41e-05 | 3465.7ms/step | 5.8 samples/s
189
+ Step 36740/44000 | Loss: 2.9020 | LR: 3.40e-05 | 3465.3ms/step | 5.8 samples/s
190
+ Step 36750/44000 | Loss: 2.8710 | LR: 3.39e-05 | 3466.1ms/step | 5.8 samples/s
191
+ Validation at step 36750: Loss = 2.9077, Perplexity = 18.31
192
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-36750
193
+ Step 36760/44000 | Loss: 2.8784 | LR: 3.38e-05 | 9557.3ms/step | 2.1 samples/s
194
+ Step 36770/44000 | Loss: 2.8918 | LR: 3.37e-05 | 3470.1ms/step | 5.8 samples/s
195
+ Step 36780/44000 | Loss: 2.8989 | LR: 3.36e-05 | 3466.5ms/step | 5.8 samples/s
196
+ Step 36790/44000 | Loss: 2.8956 | LR: 3.35e-05 | 3465.9ms/step | 5.8 samples/s
197
+ Step 36800/44000 | Loss: 2.8887 | LR: 3.34e-05 | 3466.2ms/step | 5.8 samples/s
198
+ Step 36810/44000 | Loss: 2.8842 | LR: 3.33e-05 | 3466.7ms/step | 5.8 samples/s
199
+ Step 36820/44000 | Loss: 2.8817 | LR: 3.32e-05 | 3466.8ms/step | 5.8 samples/s
200
+ Step 36830/44000 | Loss: 2.8833 | LR: 3.31e-05 | 3465.8ms/step | 5.8 samples/s
201
+ Step 36840/44000 | Loss: 2.8899 | LR: 3.31e-05 | 3467.2ms/step | 5.8 samples/s
202
+ Step 36850/44000 | Loss: 2.8910 | LR: 3.30e-05 | 3467.9ms/step | 5.8 samples/s
203
+ Step 36860/44000 | Loss: 2.8694 | LR: 3.29e-05 | 3467.2ms/step | 5.8 samples/s
204
+ Step 36870/44000 | Loss: 2.8913 | LR: 3.28e-05 | 3467.3ms/step | 5.8 samples/s
205
+ Step 36880/44000 | Loss: 2.8656 | LR: 3.27e-05 | 3465.5ms/step | 5.8 samples/s
206
+ Step 36890/44000 | Loss: 2.8612 | LR: 3.26e-05 | 3468.1ms/step | 5.8 samples/s
207
+ Step 36900/44000 | Loss: 2.8651 | LR: 3.25e-05 | 3466.8ms/step | 5.8 samples/s
208
+ Step 36910/44000 | Loss: 2.8821 | LR: 3.24e-05 | 3466.7ms/step | 5.8 samples/s
209
+ Step 36920/44000 | Loss: 2.8827 | LR: 3.23e-05 | 3466.5ms/step | 5.8 samples/s
210
+ Step 36930/44000 | Loss: 2.8591 | LR: 3.22e-05 | 3467.2ms/step | 5.8 samples/s
211
+ Step 36940/44000 | Loss: 2.8979 | LR: 3.22e-05 | 3468.1ms/step | 5.8 samples/s
212
+ Step 36950/44000 | Loss: 2.8950 | LR: 3.21e-05 | 3468.8ms/step | 5.8 samples/s
213
+ Step 36960/44000 | Loss: 2.8773 | LR: 3.20e-05 | 3465.6ms/step | 5.8 samples/s
214
+ Step 36970/44000 | Loss: 2.8590 | LR: 3.19e-05 | 3468.7ms/step | 5.8 samples/s
215
+ Step 36980/44000 | Loss: 2.8819 | LR: 3.18e-05 | 3471.3ms/step | 5.8 samples/s
216
+ Step 36990/44000 | Loss: 2.8938 | LR: 3.17e-05 | 3473.2ms/step | 5.8 samples/s
217
+ Step 37000/44000 | Loss: 2.9039 | LR: 3.16e-05 | 3471.6ms/step | 5.8 samples/s
218
+ Validation at step 37000: Loss = 2.9076, Perplexity = 18.31
219
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-37000
220
+ Step 37010/44000 | Loss: 2.8997 | LR: 3.15e-05 | 9556.6ms/step | 2.1 samples/s
221
+ Step 37020/44000 | Loss: 2.9376 | LR: 3.15e-05 | 3464.5ms/step | 5.8 samples/s
222
+ Step 37030/44000 | Loss: 2.9036 | LR: 3.14e-05 | 3464.3ms/step | 5.8 samples/s
223
+ Step 37040/44000 | Loss: 2.9037 | LR: 3.13e-05 | 3462.9ms/step | 5.8 samples/s
224
+ Step 37050/44000 | Loss: 2.8937 | LR: 3.12e-05 | 3464.1ms/step | 5.8 samples/s
225
+ Step 37060/44000 | Loss: 2.8606 | LR: 3.11e-05 | 3464.1ms/step | 5.8 samples/s
226
+ Step 37070/44000 | Loss: 2.8915 | LR: 3.10e-05 | 3463.0ms/step | 5.8 samples/s
227
+ Step 37080/44000 | Loss: 2.9221 | LR: 3.09e-05 | 3463.4ms/step | 5.8 samples/s
228
+ Step 37090/44000 | Loss: 2.9029 | LR: 3.08e-05 | 3464.1ms/step | 5.8 samples/s
229
+ Step 37100/44000 | Loss: 2.8830 | LR: 3.07e-05 | 3464.5ms/step | 5.8 samples/s
230
+ Step 37110/44000 | Loss: 2.8775 | LR: 3.07e-05 | 3464.9ms/step | 5.8 samples/s
231
+ Step 37120/44000 | Loss: 2.9006 | LR: 3.06e-05 | 3465.6ms/step | 5.8 samples/s
232
+ Step 37130/44000 | Loss: 2.8924 | LR: 3.05e-05 | 3464.5ms/step | 5.8 samples/s
233
+ Step 37140/44000 | Loss: 2.8858 | LR: 3.04e-05 | 3465.9ms/step | 5.8 samples/s
234
+ Step 37150/44000 | Loss: 2.8473 | LR: 3.03e-05 | 3463.9ms/step | 5.8 samples/s
235
+ Step 37160/44000 | Loss: 2.8982 | LR: 3.02e-05 | 3465.3ms/step | 5.8 samples/s
236
+ Step 37170/44000 | Loss: 2.9305 | LR: 3.01e-05 | 3463.1ms/step | 5.8 samples/s
237
+ Step 37180/44000 | Loss: 2.8812 | LR: 3.01e-05 | 3463.7ms/step | 5.8 samples/s
238
+ Step 37190/44000 | Loss: 2.8854 | LR: 3.00e-05 | 3463.8ms/step | 5.8 samples/s
239
+ Step 37200/44000 | Loss: 2.8911 | LR: 2.99e-05 | 3463.6ms/step | 5.8 samples/s
240
+ Step 37210/44000 | Loss: 2.9031 | LR: 2.98e-05 | 3463.1ms/step | 5.8 samples/s
241
+ Step 37220/44000 | Loss: 2.8861 | LR: 2.97e-05 | 3462.6ms/step | 5.8 samples/s
242
+ Step 37230/44000 | Loss: 2.9149 | LR: 2.96e-05 | 3463.2ms/step | 5.8 samples/s
243
+ Step 37240/44000 | Loss: 2.8896 | LR: 2.95e-05 | 3463.0ms/step | 5.8 samples/s
244
+ Step 37250/44000 | Loss: 2.8820 | LR: 2.95e-05 | 3463.5ms/step | 5.8 samples/s
245
+ Validation at step 37250: Loss = 2.9074, Perplexity = 18.31
246
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-37250
247
+ Step 37260/44000 | Loss: 2.8954 | LR: 2.94e-05 | 9537.5ms/step | 2.1 samples/s
248
+ Step 37270/44000 | Loss: 2.9083 | LR: 2.93e-05 | 3463.7ms/step | 5.8 samples/s
249
+ Step 37280/44000 | Loss: 2.8861 | LR: 2.92e-05 | 3462.9ms/step | 5.8 samples/s
250
+ Step 37290/44000 | Loss: 2.8960 | LR: 2.91e-05 | 3464.3ms/step | 5.8 samples/s
251
+ Step 37300/44000 | Loss: 2.8835 | LR: 2.90e-05 | 3465.9ms/step | 5.8 samples/s
252
+ Step 37310/44000 | Loss: 2.8849 | LR: 2.89e-05 | 3465.2ms/step | 5.8 samples/s
253
+ Step 37320/44000 | Loss: 2.8899 | LR: 2.89e-05 | 3464.2ms/step | 5.8 samples/s
254
+ Step 37330/44000 | Loss: 2.9135 | LR: 2.88e-05 | 3465.0ms/step | 5.8 samples/s
255
+ Step 37340/44000 | Loss: 2.8856 | LR: 2.87e-05 | 3464.2ms/step | 5.8 samples/s
256
+ Step 37350/44000 | Loss: 2.9052 | LR: 2.86e-05 | 3462.9ms/step | 5.8 samples/s
257
+ Step 37360/44000 | Loss: 2.8912 | LR: 2.85e-05 | 3464.4ms/step | 5.8 samples/s
258
+ Step 37370/44000 | Loss: 2.8931 | LR: 2.84e-05 | 3462.8ms/step | 5.8 samples/s
259
+ Step 37380/44000 | Loss: 2.8734 | LR: 2.84e-05 | 3463.2ms/step | 5.8 samples/s
260
+ Step 37390/44000 | Loss: 2.8739 | LR: 2.83e-05 | 3463.5ms/step | 5.8 samples/s
261
+ Step 37400/44000 | Loss: 2.8767 | LR: 2.82e-05 | 3463.1ms/step | 5.8 samples/s
262
+ Step 37410/44000 | Loss: 2.8730 | LR: 2.81e-05 | 3462.6ms/step | 5.8 samples/s
263
+ Step 37420/44000 | Loss: 2.9008 | LR: 2.80e-05 | 3464.4ms/step | 5.8 samples/s
264
+ Step 37430/44000 | Loss: 2.8634 | LR: 2.79e-05 | 3464.0ms/step | 5.8 samples/s
265
+ Step 37440/44000 | Loss: 2.8770 | LR: 2.78e-05 | 3463.4ms/step | 5.8 samples/s
266
+ Step 37450/44000 | Loss: 2.8896 | LR: 2.78e-05 | 3462.4ms/step | 5.8 samples/s
267
+ Step 37460/44000 | Loss: 2.9040 | LR: 2.77e-05 | 3462.6ms/step | 5.8 samples/s
268
+ Step 37470/44000 | Loss: 2.8878 | LR: 2.76e-05 | 3463.6ms/step | 5.8 samples/s
269
+ Step 37480/44000 | Loss: 2.8992 | LR: 2.75e-05 | 3463.4ms/step | 5.8 samples/s
270
+ Step 37490/44000 | Loss: 2.8828 | LR: 2.74e-05 | 3464.5ms/step | 5.8 samples/s
271
+ Step 37500/44000 | Loss: 2.8882 | LR: 2.74e-05 | 3465.8ms/step | 5.8 samples/s
272
+ Validation at step 37500: Loss = 2.9073, Perplexity = 18.31
273
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-37500
274
+ Step 37510/44000 | Loss: 2.9088 | LR: 2.73e-05 | 9535.5ms/step | 2.1 samples/s
275
+ Step 37520/44000 | Loss: 2.8636 | LR: 2.72e-05 | 3463.9ms/step | 5.8 samples/s
276
+ Step 37530/44000 | Loss: 2.9295 | LR: 2.71e-05 | 3464.2ms/step | 5.8 samples/s
277
+ Step 37540/44000 | Loss: 2.8975 | LR: 2.70e-05 | 3463.8ms/step | 5.8 samples/s
278
+ Step 37550/44000 | Loss: 2.8915 | LR: 2.69e-05 | 3463.9ms/step | 5.8 samples/s
279
+ Step 37560/44000 | Loss: 2.9078 | LR: 2.69e-05 | 3463.9ms/step | 5.8 samples/s
280
+ Step 37570/44000 | Loss: 2.8872 | LR: 2.68e-05 | 3462.2ms/step | 5.8 samples/s
281
+ Step 37580/44000 | Loss: 2.8793 | LR: 2.67e-05 | 3467.4ms/step | 5.8 samples/s
282
+ Step 37590/44000 | Loss: 2.8798 | LR: 2.66e-05 | 3466.8ms/step | 5.8 samples/s
283
+ Step 37600/44000 | Loss: 2.8850 | LR: 2.65e-05 | 3471.8ms/step | 5.8 samples/s
284
+ Step 37610/44000 | Loss: 2.8757 | LR: 2.65e-05 | 3462.9ms/step | 5.8 samples/s
285
+ Step 37620/44000 | Loss: 2.9003 | LR: 2.64e-05 | 3461.9ms/step | 5.8 samples/s
286
+ Step 37630/44000 | Loss: 2.8555 | LR: 2.63e-05 | 3462.8ms/step | 5.8 samples/s
287
+ Step 37640/44000 | Loss: 2.9027 | LR: 2.62e-05 | 3464.5ms/step | 5.8 samples/s
288
+ Step 37650/44000 | Loss: 2.8749 | LR: 2.61e-05 | 3461.7ms/step | 5.8 samples/s
289
+ Step 37660/44000 | Loss: 2.9145 | LR: 2.60e-05 | 3463.5ms/step | 5.8 samples/s
290
+ Step 37670/44000 | Loss: 2.8936 | LR: 2.60e-05 | 3462.0ms/step | 5.8 samples/s
291
+ Step 37680/44000 | Loss: 2.9063 | LR: 2.59e-05 | 3468.2ms/step | 5.8 samples/s
292
+ Step 37690/44000 | Loss: 2.8818 | LR: 2.58e-05 | 3472.6ms/step | 5.8 samples/s
293
+ Step 37700/44000 | Loss: 2.8777 | LR: 2.57e-05 | 3470.9ms/step | 5.8 samples/s
294
+ Step 37710/44000 | Loss: 2.8699 | LR: 2.56e-05 | 3472.9ms/step | 5.8 samples/s
295
+ Step 37720/44000 | Loss: 2.9054 | LR: 2.56e-05 | 3473.1ms/step | 5.8 samples/s
296
+ Step 37730/44000 | Loss: 2.8752 | LR: 2.55e-05 | 3471.8ms/step | 5.8 samples/s
297
+ Step 37740/44000 | Loss: 2.8870 | LR: 2.54e-05 | 3465.9ms/step | 5.8 samples/s
298
+ Step 37750/44000 | Loss: 2.8725 | LR: 2.53e-05 | 3462.5ms/step | 5.8 samples/s
299
+ Validation at step 37750: Loss = 2.9071, Perplexity = 18.30
300
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-37750
301
+ Step 37760/44000 | Loss: 2.8952 | LR: 2.52e-05 | 9548.6ms/step | 2.1 samples/s
302
+ Step 37770/44000 | Loss: 2.8689 | LR: 2.52e-05 | 3469.7ms/step | 5.8 samples/s
303
+ Step 37780/44000 | Loss: 2.8737 | LR: 2.51e-05 | 3466.3ms/step | 5.8 samples/s
304
+ Step 37790/44000 | Loss: 2.9083 | LR: 2.50e-05 | 3463.4ms/step | 5.8 samples/s
305
+ Step 37800/44000 | Loss: 2.8781 | LR: 2.49e-05 | 3465.9ms/step | 5.8 samples/s
306
+ Step 37810/44000 | Loss: 2.8549 | LR: 2.48e-05 | 3466.1ms/step | 5.8 samples/s
307
+ Step 37820/44000 | Loss: 2.9120 | LR: 2.48e-05 | 3469.0ms/step | 5.8 samples/s
308
+ Step 37830/44000 | Loss: 2.9098 | LR: 2.47e-05 | 3463.0ms/step | 5.8 samples/s
309
+ Step 37840/44000 | Loss: 2.8959 | LR: 2.46e-05 | 3464.5ms/step | 5.8 samples/s
310
+ Step 37850/44000 | Loss: 2.8763 | LR: 2.45e-05 | 3467.3ms/step | 5.8 samples/s
311
+ Step 37860/44000 | Loss: 2.8567 | LR: 2.45e-05 | 3470.4ms/step | 5.8 samples/s
312
+ Step 37870/44000 | Loss: 2.8792 | LR: 2.44e-05 | 3466.7ms/step | 5.8 samples/s
313
+ Step 37880/44000 | Loss: 2.8753 | LR: 2.43e-05 | 3464.6ms/step | 5.8 samples/s
314
+ Step 37890/44000 | Loss: 2.8999 | LR: 2.42e-05 | 3465.3ms/step | 5.8 samples/s
315
+ Step 37900/44000 | Loss: 2.9029 | LR: 2.41e-05 | 3465.4ms/step | 5.8 samples/s
316
+ Step 37910/44000 | Loss: 2.8878 | LR: 2.41e-05 | 3465.4ms/step | 5.8 samples/s
317
+ Step 37920/44000 | Loss: 2.9072 | LR: 2.40e-05 | 3472.0ms/step | 5.8 samples/s
318
+ Step 37930/44000 | Loss: 2.8840 | LR: 2.39e-05 | 3473.0ms/step | 5.8 samples/s
319
+ Step 37940/44000 | Loss: 2.8952 | LR: 2.38e-05 | 3468.7ms/step | 5.8 samples/s
320
+ Step 37950/44000 | Loss: 2.9105 | LR: 2.38e-05 | 3461.4ms/step | 5.8 samples/s
321
+ Step 37960/44000 | Loss: 2.9269 | LR: 2.37e-05 | 3465.2ms/step | 5.8 samples/s
322
+ Step 37970/44000 | Loss: 2.8860 | LR: 2.36e-05 | 3465.1ms/step | 5.8 samples/s
323
+ Step 37980/44000 | Loss: 2.9174 | LR: 2.35e-05 | 3465.0ms/step | 5.8 samples/s
324
+ Step 37990/44000 | Loss: 2.8863 | LR: 2.34e-05 | 3471.7ms/step | 5.8 samples/s
325
+ Step 38000/44000 | Loss: 2.8907 | LR: 2.34e-05 | 3468.9ms/step | 5.8 samples/s
326
+ Validation at step 38000: Loss = 2.9071, Perplexity = 18.30
327
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-38000
328
+ Step 38010/44000 | Loss: 2.8405 | LR: 2.33e-05 | 9549.5ms/step | 2.1 samples/s
329
+ Step 38020/44000 | Loss: 2.8750 | LR: 2.32e-05 | 3464.9ms/step | 5.8 samples/s
330
+ Step 38030/44000 | Loss: 2.8434 | LR: 2.31e-05 | 3465.9ms/step | 5.8 samples/s
331
+ Step 38040/44000 | Loss: 2.9131 | LR: 2.31e-05 | 3465.4ms/step | 5.8 samples/s
332
+ Step 38050/44000 | Loss: 2.8964 | LR: 2.30e-05 | 3464.1ms/step | 5.8 samples/s
333
+ Step 38060/44000 | Loss: 2.8999 | LR: 2.29e-05 | 3466.4ms/step | 5.8 samples/s
334
+ Step 38070/44000 | Loss: 2.8734 | LR: 2.28e-05 | 3464.1ms/step | 5.8 samples/s
335
+ Step 38080/44000 | Loss: 2.8888 | LR: 2.28e-05 | 3464.6ms/step | 5.8 samples/s
336
+ Step 38090/44000 | Loss: 2.9002 | LR: 2.27e-05 | 3467.6ms/step | 5.8 samples/s
337
+ Step 38100/44000 | Loss: 2.8816 | LR: 2.26e-05 | 3463.1ms/step | 5.8 samples/s
338
+ Step 38110/44000 | Loss: 2.8947 | LR: 2.25e-05 | 3464.7ms/step | 5.8 samples/s
339
+ Step 38120/44000 | Loss: 2.9038 | LR: 2.25e-05 | 3462.3ms/step | 5.8 samples/s
340
+ Step 38130/44000 | Loss: 2.8820 | LR: 2.24e-05 | 3463.2ms/step | 5.8 samples/s
341
+ Step 38140/44000 | Loss: 2.9069 | LR: 2.23e-05 | 3463.3ms/step | 5.8 samples/s
342
+ Step 38150/44000 | Loss: 2.9255 | LR: 2.22e-05 | 3463.6ms/step | 5.8 samples/s
343
+ Step 38160/44000 | Loss: 2.9300 | LR: 2.22e-05 | 3463.6ms/step | 5.8 samples/s
344
+ Step 38170/44000 | Loss: 2.8938 | LR: 2.21e-05 | 3462.2ms/step | 5.8 samples/s
345
+ Step 38180/44000 | Loss: 2.9207 | LR: 2.20e-05 | 3463.7ms/step | 5.8 samples/s
346
+ Step 38190/44000 | Loss: 2.8645 | LR: 2.19e-05 | 3463.1ms/step | 5.8 samples/s
347
+ Step 38200/44000 | Loss: 2.8910 | LR: 2.19e-05 | 3464.0ms/step | 5.8 samples/s
348
+ Step 38210/44000 | Loss: 2.8925 | LR: 2.18e-05 | 3464.1ms/step | 5.8 samples/s
349
+ Step 38220/44000 | Loss: 2.9113 | LR: 2.17e-05 | 3464.0ms/step | 5.8 samples/s
350
+ Step 38230/44000 | Loss: 2.8942 | LR: 2.16e-05 | 3464.5ms/step | 5.8 samples/s
351
+ Step 38240/44000 | Loss: 2.8985 | LR: 2.16e-05 | 3469.5ms/step | 5.8 samples/s
352
+ Step 38250/44000 | Loss: 2.8962 | LR: 2.15e-05 | 3473.8ms/step | 5.8 samples/s
353
+ Validation at step 38250: Loss = 2.9070, Perplexity = 18.30
354
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-38250
355
+ Step 38260/44000 | Loss: 2.8699 | LR: 2.14e-05 | 9539.4ms/step | 2.1 samples/s
356
+ Step 38270/44000 | Loss: 2.9021 | LR: 2.13e-05 | 3463.3ms/step | 5.8 samples/s
357
+ Step 38280/44000 | Loss: 2.8586 | LR: 2.13e-05 | 3463.7ms/step | 5.8 samples/s
358
+ Step 38290/44000 | Loss: 2.8679 | LR: 2.12e-05 | 3465.6ms/step | 5.8 samples/s
359
+ Step 38300/44000 | Loss: 2.9014 | LR: 2.11e-05 | 3464.3ms/step | 5.8 samples/s
360
+ Step 38310/44000 | Loss: 2.8724 | LR: 2.11e-05 | 3464.9ms/step | 5.8 samples/s
361
+ Step 38320/44000 | Loss: 2.8895 | LR: 2.10e-05 | 3464.7ms/step | 5.8 samples/s
362
+ Step 38330/44000 | Loss: 2.9065 | LR: 2.09e-05 | 3463.3ms/step | 5.8 samples/s
363
+ Step 38340/44000 | Loss: 2.8564 | LR: 2.08e-05 | 3463.9ms/step | 5.8 samples/s
364
+ Step 38350/44000 | Loss: 2.8858 | LR: 2.08e-05 | 3463.4ms/step | 5.8 samples/s
365
+ Step 38360/44000 | Loss: 2.8940 | LR: 2.07e-05 | 3462.1ms/step | 5.8 samples/s
366
+ Step 38370/44000 | Loss: 2.9002 | LR: 2.06e-05 | 3463.1ms/step | 5.8 samples/s
367
+ Step 38380/44000 | Loss: 2.9054 | LR: 2.05e-05 | 3464.6ms/step | 5.8 samples/s
368
+ Step 38390/44000 | Loss: 2.8841 | LR: 2.05e-05 | 3464.3ms/step | 5.8 samples/s
369
+ Step 38400/44000 | Loss: 2.9045 | LR: 2.04e-05 | 3464.8ms/step | 5.8 samples/s
370
+ Step 38410/44000 | Loss: 2.8865 | LR: 2.03e-05 | 3463.8ms/step | 5.8 samples/s
371
+ Step 38420/44000 | Loss: 2.8966 | LR: 2.03e-05 | 3462.3ms/step | 5.8 samples/s
372
+ Step 38430/44000 | Loss: 2.9036 | LR: 2.02e-05 | 3463.3ms/step | 5.8 samples/s
373
+ Step 38440/44000 | Loss: 2.8557 | LR: 2.01e-05 | 3463.1ms/step | 5.8 samples/s
374
+ Step 38450/44000 | Loss: 2.8914 | LR: 2.00e-05 | 3463.0ms/step | 5.8 samples/s
375
+ Step 38460/44000 | Loss: 2.8917 | LR: 2.00e-05 | 3462.6ms/step | 5.8 samples/s
376
+ Step 38470/44000 | Loss: 2.8865 | LR: 1.99e-05 | 3463.7ms/step | 5.8 samples/s
377
+ Step 38480/44000 | Loss: 2.9075 | LR: 1.98e-05 | 3463.7ms/step | 5.8 samples/s
378
+ Step 38490/44000 | Loss: 2.8970 | LR: 1.98e-05 | 3464.5ms/step | 5.8 samples/s
379
+ Step 38500/44000 | Loss: 2.8988 | LR: 1.97e-05 | 3462.8ms/step | 5.8 samples/s
380
+ Validation at step 38500: Loss = 2.9069, Perplexity = 18.30
381
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-38500
382
+ Step 38510/44000 | Loss: 2.8902 | LR: 1.96e-05 | 9540.6ms/step | 2.1 samples/s
383
+ Step 38520/44000 | Loss: 2.8662 | LR: 1.95e-05 | 3463.5ms/step | 5.8 samples/s
384
+ Step 38530/44000 | Loss: 2.8905 | LR: 1.95e-05 | 3464.2ms/step | 5.8 samples/s
385
+ Step 38540/44000 | Loss: 2.9007 | LR: 1.94e-05 | 3462.6ms/step | 5.8 samples/s
386
+ Step 38550/44000 | Loss: 2.9113 | LR: 1.93e-05 | 3463.9ms/step | 5.8 samples/s
387
+ Step 38560/44000 | Loss: 2.8925 | LR: 1.93e-05 | 3463.9ms/step | 5.8 samples/s
388
+ Step 38570/44000 | Loss: 2.8842 | LR: 1.92e-05 | 3465.2ms/step | 5.8 samples/s
389
+ Step 38580/44000 | Loss: 2.8671 | LR: 1.91e-05 | 3463.2ms/step | 5.8 samples/s
390
+ Step 38590/44000 | Loss: 2.8951 | LR: 1.91e-05 | 3466.5ms/step | 5.8 samples/s
391
+ Step 38600/44000 | Loss: 2.8721 | LR: 1.90e-05 | 3465.0ms/step | 5.8 samples/s
392
+ Step 38610/44000 | Loss: 2.8777 | LR: 1.89e-05 | 3466.5ms/step | 5.8 samples/s
393
+ Step 38620/44000 | Loss: 2.8989 | LR: 1.88e-05 | 3464.8ms/step | 5.8 samples/s
394
+ Step 38630/44000 | Loss: 2.8724 | LR: 1.88e-05 | 3461.9ms/step | 5.8 samples/s
395
+ Step 38640/44000 | Loss: 2.9128 | LR: 1.87e-05 | 3463.0ms/step | 5.8 samples/s
396
+ Step 38650/44000 | Loss: 2.8679 | LR: 1.86e-05 | 3464.3ms/step | 5.8 samples/s
397
+ Step 38660/44000 | Loss: 2.8764 | LR: 1.86e-05 | 3462.8ms/step | 5.8 samples/s
398
+ Step 38670/44000 | Loss: 2.8951 | LR: 1.85e-05 | 3462.5ms/step | 5.8 samples/s
399
+ Step 38680/44000 | Loss: 2.9171 | LR: 1.84e-05 | 3461.9ms/step | 5.8 samples/s
400
+ Step 38690/44000 | Loss: 2.8932 | LR: 1.84e-05 | 3462.2ms/step | 5.8 samples/s
401
+ Step 38700/44000 | Loss: 2.9094 | LR: 1.83e-05 | 3463.4ms/step | 5.8 samples/s
402
+ Step 38710/44000 | Loss: 2.8954 | LR: 1.82e-05 | 3465.5ms/step | 5.8 samples/s
403
+ Step 38720/44000 | Loss: 2.8687 | LR: 1.82e-05 | 3462.5ms/step | 5.8 samples/s
404
+ Step 38730/44000 | Loss: 2.8827 | LR: 1.81e-05 | 3462.2ms/step | 5.8 samples/s
405
+ Step 38740/44000 | Loss: 2.9049 | LR: 1.80e-05 | 3462.5ms/step | 5.8 samples/s
406
+ Step 38750/44000 | Loss: 2.8858 | LR: 1.80e-05 | 3463.3ms/step | 5.8 samples/s
407
+ Validation at step 38750: Loss = 2.9069, Perplexity = 18.30
408
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-38750
409
+ Step 38760/44000 | Loss: 2.8959 | LR: 1.79e-05 | 9531.0ms/step | 2.1 samples/s
410
+ Step 38770/44000 | Loss: 2.8856 | LR: 1.78e-05 | 3461.3ms/step | 5.8 samples/s
411
+ Step 38780/44000 | Loss: 2.8762 | LR: 1.78e-05 | 3462.1ms/step | 5.8 samples/s
412
+ Step 38790/44000 | Loss: 2.9075 | LR: 1.77e-05 | 3461.5ms/step | 5.8 samples/s
413
+ Step 38800/44000 | Loss: 2.8759 | LR: 1.76e-05 | 3460.0ms/step | 5.8 samples/s
414
+ Step 38810/44000 | Loss: 2.8563 | LR: 1.76e-05 | 3461.2ms/step | 5.8 samples/s
415
+ Step 38820/44000 | Loss: 2.8943 | LR: 1.75e-05 | 3463.5ms/step | 5.8 samples/s
416
+ Step 38830/44000 | Loss: 2.8960 | LR: 1.74e-05 | 3462.7ms/step | 5.8 samples/s
417
+ Step 38840/44000 | Loss: 2.8593 | LR: 1.74e-05 | 3461.8ms/step | 5.8 samples/s
418
+ Step 38850/44000 | Loss: 2.8966 | LR: 1.73e-05 | 3462.6ms/step | 5.8 samples/s
419
+ Step 38860/44000 | Loss: 2.8846 | LR: 1.72e-05 | 3462.7ms/step | 5.8 samples/s
420
+ Step 38870/44000 | Loss: 2.9018 | LR: 1.72e-05 | 3462.6ms/step | 5.8 samples/s
421
+ Step 38880/44000 | Loss: 2.9040 | LR: 1.71e-05 | 3462.8ms/step | 5.8 samples/s
422
+ Step 38890/44000 | Loss: 2.8887 | LR: 1.70e-05 | 3462.4ms/step | 5.8 samples/s
423
+ Step 38900/44000 | Loss: 2.8862 | LR: 1.70e-05 | 3462.5ms/step | 5.8 samples/s
424
+ Step 38910/44000 | Loss: 2.9090 | LR: 1.69e-05 | 3463.2ms/step | 5.8 samples/s
425
+ Step 38920/44000 | Loss: 2.8911 | LR: 1.68e-05 | 3461.4ms/step | 5.8 samples/s
426
+ Step 38930/44000 | Loss: 2.8684 | LR: 1.68e-05 | 3463.2ms/step | 5.8 samples/s
427
+ Step 38940/44000 | Loss: 2.8764 | LR: 1.67e-05 | 3463.8ms/step | 5.8 samples/s
428
+ Step 38950/44000 | Loss: 2.8635 | LR: 1.66e-05 | 3463.0ms/step | 5.8 samples/s
429
+ Step 38960/44000 | Loss: 2.8824 | LR: 1.66e-05 | 3462.1ms/step | 5.8 samples/s
430
+ Step 38970/44000 | Loss: 2.8950 | LR: 1.65e-05 | 3462.4ms/step | 5.8 samples/s
431
+ Step 38980/44000 | Loss: 2.8710 | LR: 1.64e-05 | 3462.2ms/step | 5.8 samples/s
432
+ Step 38990/44000 | Loss: 2.9008 | LR: 1.64e-05 | 3462.9ms/step | 5.8 samples/s
433
+ Step 39000/44000 | Loss: 2.8880 | LR: 1.63e-05 | 3463.7ms/step | 5.8 samples/s
434
+ Validation at step 39000: Loss = 2.9068, Perplexity = 18.30
435
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-39000
436
+ Step 39010/44000 | Loss: 2.9270 | LR: 1.62e-05 | 9521.0ms/step | 2.1 samples/s
437
+ Step 39020/44000 | Loss: 2.8773 | LR: 1.62e-05 | 3462.8ms/step | 5.8 samples/s
438
+ Step 39030/44000 | Loss: 2.9168 | LR: 1.61e-05 | 3462.5ms/step | 5.8 samples/s
439
+ Step 39040/44000 | Loss: 2.8708 | LR: 1.61e-05 | 3461.6ms/step | 5.8 samples/s
440
+ Step 39050/44000 | Loss: 2.8778 | LR: 1.60e-05 | 3461.3ms/step | 5.8 samples/s
441
+ Step 39060/44000 | Loss: 2.8907 | LR: 1.59e-05 | 3461.9ms/step | 5.8 samples/s
442
+ Step 39070/44000 | Loss: 2.8668 | LR: 1.59e-05 | 3462.4ms/step | 5.8 samples/s
443
+ Step 39080/44000 | Loss: 2.8975 | LR: 1.58e-05 | 3461.9ms/step | 5.8 samples/s
444
+ Step 39090/44000 | Loss: 2.8594 | LR: 1.57e-05 | 3462.6ms/step | 5.8 samples/s
445
+ Step 39100/44000 | Loss: 2.8615 | LR: 1.57e-05 | 3462.5ms/step | 5.8 samples/s
446
+ Step 39110/44000 | Loss: 2.9033 | LR: 1.56e-05 | 3464.0ms/step | 5.8 samples/s
447
+ Step 39120/44000 | Loss: 2.8927 | LR: 1.55e-05 | 3463.0ms/step | 5.8 samples/s
448
+ Step 39130/44000 | Loss: 2.8625 | LR: 1.55e-05 | 3462.8ms/step | 5.8 samples/s
449
+ Step 39140/44000 | Loss: 2.8823 | LR: 1.54e-05 | 3463.6ms/step | 5.8 samples/s
450
+ Step 39150/44000 | Loss: 2.8947 | LR: 1.54e-05 | 3461.8ms/step | 5.8 samples/s
451
+ Step 39160/44000 | Loss: 2.9215 | LR: 1.53e-05 | 3462.7ms/step | 5.8 samples/s
452
+ Step 39170/44000 | Loss: 2.8975 | LR: 1.52e-05 | 3461.3ms/step | 5.8 samples/s
453
+ Step 39180/44000 | Loss: 2.9003 | LR: 1.52e-05 | 3460.8ms/step | 5.8 samples/s
454
+ Step 39190/44000 | Loss: 2.8454 | LR: 1.51e-05 | 3460.8ms/step | 5.8 samples/s
455
+ Step 39200/44000 | Loss: 2.8671 | LR: 1.50e-05 | 3463.8ms/step | 5.8 samples/s
456
+ Step 39210/44000 | Loss: 2.8548 | LR: 1.50e-05 | 3462.6ms/step | 5.8 samples/s
457
+ Step 39220/44000 | Loss: 2.8949 | LR: 1.49e-05 | 3462.6ms/step | 5.8 samples/s
458
+ Step 39230/44000 | Loss: 2.8794 | LR: 1.49e-05 | 3462.6ms/step | 5.8 samples/s
459
+ Step 39240/44000 | Loss: 2.8804 | LR: 1.48e-05 | 3464.7ms/step | 5.8 samples/s
460
+ Step 39250/44000 | Loss: 2.8810 | LR: 1.47e-05 | 3463.7ms/step | 5.8 samples/s
461
+ Validation at step 39250: Loss = 2.9068, Perplexity = 18.30
462
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-39250
463
+ Step 39260/44000 | Loss: 2.8785 | LR: 1.47e-05 | 9527.8ms/step | 2.1 samples/s
464
+ Step 39270/44000 | Loss: 2.8728 | LR: 1.46e-05 | 3461.1ms/step | 5.8 samples/s
465
+ Step 39280/44000 | Loss: 2.8842 | LR: 1.46e-05 | 3463.1ms/step | 5.8 samples/s
466
+ Step 39290/44000 | Loss: 2.8843 | LR: 1.45e-05 | 3462.9ms/step | 5.8 samples/s
467
+ Step 39300/44000 | Loss: 2.8820 | LR: 1.44e-05 | 3466.0ms/step | 5.8 samples/s
468
+ Step 39310/44000 | Loss: 2.8755 | LR: 1.44e-05 | 3463.7ms/step | 5.8 samples/s
469
+ Step 39320/44000 | Loss: 2.8510 | LR: 1.43e-05 | 3463.0ms/step | 5.8 samples/s
470
+ Step 39330/44000 | Loss: 2.8949 | LR: 1.42e-05 | 3461.8ms/step | 5.8 samples/s
471
+ Step 39340/44000 | Loss: 2.8920 | LR: 1.42e-05 | 3463.2ms/step | 5.8 samples/s
472
+ Step 39350/44000 | Loss: 2.8827 | LR: 1.41e-05 | 3463.6ms/step | 5.8 samples/s
473
+ Step 39360/44000 | Loss: 2.9234 | LR: 1.41e-05 | 3463.6ms/step | 5.8 samples/s
474
+ Step 39370/44000 | Loss: 2.8659 | LR: 1.40e-05 | 3461.1ms/step | 5.8 samples/s
475
+ Step 39380/44000 | Loss: 2.9048 | LR: 1.39e-05 | 3462.6ms/step | 5.8 samples/s
476
+ Step 39390/44000 | Loss: 2.8911 | LR: 1.39e-05 | 3462.9ms/step | 5.8 samples/s
477
+ Step 39400/44000 | Loss: 2.8775 | LR: 1.38e-05 | 3462.7ms/step | 5.8 samples/s
478
+ Step 39410/44000 | Loss: 2.8881 | LR: 1.38e-05 | 3464.2ms/step | 5.8 samples/s
479
+ Step 39420/44000 | Loss: 2.9113 | LR: 1.37e-05 | 3461.9ms/step | 5.8 samples/s
480
+ Step 39430/44000 | Loss: 2.9069 | LR: 1.36e-05 | 3462.6ms/step | 5.8 samples/s
481
+ Step 39440/44000 | Loss: 2.8757 | LR: 1.36e-05 | 3462.2ms/step | 5.8 samples/s
482
+ Step 39450/44000 | Loss: 2.9059 | LR: 1.35e-05 | 3462.2ms/step | 5.8 samples/s
483
+ Step 39460/44000 | Loss: 2.8713 | LR: 1.35e-05 | 3462.2ms/step | 5.8 samples/s
484
+ Step 39470/44000 | Loss: 2.8760 | LR: 1.34e-05 | 3462.3ms/step | 5.8 samples/s
485
+ Step 39480/44000 | Loss: 2.8991 | LR: 1.34e-05 | 3461.3ms/step | 5.8 samples/s
486
+ Step 39490/44000 | Loss: 2.9003 | LR: 1.33e-05 | 3462.6ms/step | 5.8 samples/s
487
+ Step 39500/44000 | Loss: 2.8812 | LR: 1.32e-05 | 3464.9ms/step | 5.8 samples/s
488
+ Validation at step 39500: Loss = 2.9068, Perplexity = 18.30
489
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-39500
490
+ Step 39510/44000 | Loss: 2.8658 | LR: 1.32e-05 | 9556.7ms/step | 2.1 samples/s
491
+ Step 39520/44000 | Loss: 2.8668 | LR: 1.31e-05 | 3460.9ms/step | 5.8 samples/s
492
+ Step 39530/44000 | Loss: 2.8860 | LR: 1.31e-05 | 3461.8ms/step | 5.8 samples/s
493
+ Step 39540/44000 | Loss: 2.8909 | LR: 1.30e-05 | 3459.7ms/step | 5.8 samples/s
494
+ Step 39550/44000 | Loss: 2.8828 | LR: 1.29e-05 | 3461.2ms/step | 5.8 samples/s
495
+ Step 39560/44000 | Loss: 2.8956 | LR: 1.29e-05 | 3461.1ms/step | 5.8 samples/s
496
+ Step 39570/44000 | Loss: 2.8432 | LR: 1.28e-05 | 3461.0ms/step | 5.8 samples/s
497
+ Step 39580/44000 | Loss: 2.8851 | LR: 1.28e-05 | 3466.0ms/step | 5.8 samples/s
498
+ Step 39590/44000 | Loss: 2.8714 | LR: 1.27e-05 | 3465.4ms/step | 5.8 samples/s
499
+ Step 39600/44000 | Loss: 2.8800 | LR: 1.27e-05 | 3462.7ms/step | 5.8 samples/s
500
+ Step 39610/44000 | Loss: 2.8835 | LR: 1.26e-05 | 3462.6ms/step | 5.8 samples/s
501
+ Step 39620/44000 | Loss: 2.8917 | LR: 1.25e-05 | 3461.8ms/step | 5.8 samples/s
502
+ Step 39630/44000 | Loss: 2.8903 | LR: 1.25e-05 | 3460.9ms/step | 5.8 samples/s
503
+ Step 39640/44000 | Loss: 2.8931 | LR: 1.24e-05 | 3461.8ms/step | 5.8 samples/s
504
+ Step 39650/44000 | Loss: 2.8909 | LR: 1.24e-05 | 3461.0ms/step | 5.8 samples/s
505
+ Step 39660/44000 | Loss: 2.8795 | LR: 1.23e-05 | 3461.8ms/step | 5.8 samples/s
506
+ Step 39670/44000 | Loss: 2.8698 | LR: 1.23e-05 | 3461.3ms/step | 5.8 samples/s
507
+ Step 39680/44000 | Loss: 2.9030 | LR: 1.22e-05 | 3461.1ms/step | 5.8 samples/s
508
+ Step 39690/44000 | Loss: 2.9014 | LR: 1.22e-05 | 3463.9ms/step | 5.8 samples/s
509
+ Step 39700/44000 | Loss: 2.8916 | LR: 1.21e-05 | 3463.6ms/step | 5.8 samples/s
510
+ Step 39710/44000 | Loss: 2.9300 | LR: 1.20e-05 | 3462.7ms/step | 5.8 samples/s
511
+ Step 39720/44000 | Loss: 2.8991 | LR: 1.20e-05 | 3462.4ms/step | 5.8 samples/s
512
+ Step 39730/44000 | Loss: 2.8949 | LR: 1.19e-05 | 3464.1ms/step | 5.8 samples/s
513
+ Step 39740/44000 | Loss: 2.9007 | LR: 1.19e-05 | 3462.4ms/step | 5.8 samples/s
514
+ Step 39750/44000 | Loss: 2.9154 | LR: 1.18e-05 | 3463.6ms/step | 5.8 samples/s
515
+ Validation at step 39750: Loss = 2.9067, Perplexity = 18.30
516
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-39750
517
+ Step 39760/44000 | Loss: 2.9188 | LR: 1.18e-05 | 9537.3ms/step | 2.1 samples/s
518
+ Step 39770/44000 | Loss: 2.8821 | LR: 1.17e-05 | 3460.9ms/step | 5.8 samples/s
519
+ Step 39780/44000 | Loss: 2.8863 | LR: 1.17e-05 | 3461.4ms/step | 5.8 samples/s
520
+ Step 39790/44000 | Loss: 2.8706 | LR: 1.16e-05 | 3461.9ms/step | 5.8 samples/s
521
+ Step 39800/44000 | Loss: 2.8878 | LR: 1.15e-05 | 3462.2ms/step | 5.8 samples/s
522
+ Step 39810/44000 | Loss: 2.8729 | LR: 1.15e-05 | 3460.8ms/step | 5.8 samples/s
523
+ Step 39820/44000 | Loss: 2.8831 | LR: 1.14e-05 | 3460.9ms/step | 5.8 samples/s
524
+ Step 39830/44000 | Loss: 2.8945 | LR: 1.14e-05 | 3461.3ms/step | 5.8 samples/s
525
+ Step 39840/44000 | Loss: 2.9092 | LR: 1.13e-05 | 3460.3ms/step | 5.8 samples/s
526
+ Step 39850/44000 | Loss: 2.8997 | LR: 1.13e-05 | 3460.3ms/step | 5.8 samples/s
527
+ Step 39860/44000 | Loss: 2.8907 | LR: 1.12e-05 | 3461.4ms/step | 5.8 samples/s
528
+ Step 39870/44000 | Loss: 2.8851 | LR: 1.12e-05 | 3462.1ms/step | 5.8 samples/s
529
+ Step 39880/44000 | Loss: 2.8601 | LR: 1.11e-05 | 3463.0ms/step | 5.8 samples/s
530
+ Step 39890/44000 | Loss: 2.8757 | LR: 1.11e-05 | 3463.6ms/step | 5.8 samples/s
531
+ Step 39900/44000 | Loss: 2.8913 | LR: 1.10e-05 | 3461.7ms/step | 5.8 samples/s
532
+ Step 39910/44000 | Loss: 2.8581 | LR: 1.10e-05 | 3462.8ms/step | 5.8 samples/s
533
+ Step 39920/44000 | Loss: 2.9156 | LR: 1.09e-05 | 3461.5ms/step | 5.8 samples/s
534
+ Step 39930/44000 | Loss: 2.8896 | LR: 1.08e-05 | 3460.9ms/step | 5.8 samples/s
535
+ Step 39940/44000 | Loss: 2.9112 | LR: 1.08e-05 | 3463.3ms/step | 5.8 samples/s
536
+ Step 39950/44000 | Loss: 2.8741 | LR: 1.07e-05 | 3462.2ms/step | 5.8 samples/s
537
+ Step 39960/44000 | Loss: 2.8929 | LR: 1.07e-05 | 3461.0ms/step | 5.8 samples/s
538
+ Step 39970/44000 | Loss: 2.8881 | LR: 1.06e-05 | 3462.4ms/step | 5.8 samples/s
539
+ Step 39980/44000 | Loss: 2.8771 | LR: 1.06e-05 | 3462.4ms/step | 5.8 samples/s
540
+ Step 39990/44000 | Loss: 2.9017 | LR: 1.05e-05 | 3462.5ms/step | 5.8 samples/s
541
+ Step 40000/44000 | Loss: 2.8724 | LR: 1.05e-05 | 3461.6ms/step | 5.8 samples/s
542
+ Validation at step 40000: Loss = 2.9067, Perplexity = 18.30
543
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-40000
544
+ Step 40010/44000 | Loss: 2.8900 | LR: 1.04e-05 | 9531.7ms/step | 2.1 samples/s
545
+ Step 40020/44000 | Loss: 2.8935 | LR: 1.04e-05 | 3460.5ms/step | 5.8 samples/s
546
+ Step 40030/44000 | Loss: 2.9009 | LR: 1.03e-05 | 3462.0ms/step | 5.8 samples/s
547
+ Step 40040/44000 | Loss: 2.8822 | LR: 1.03e-05 | 3461.7ms/step | 5.8 samples/s
548
+ Step 40050/44000 | Loss: 2.8639 | LR: 1.02e-05 | 3461.1ms/step | 5.8 samples/s
549
+ Step 40060/44000 | Loss: 2.8733 | LR: 1.02e-05 | 3461.0ms/step | 5.8 samples/s
550
+ Step 40070/44000 | Loss: 2.8956 | LR: 1.01e-05 | 3460.3ms/step | 5.8 samples/s
551
+ Step 40080/44000 | Loss: 2.8987 | LR: 1.01e-05 | 3460.5ms/step | 5.8 samples/s
552
+ Step 40090/44000 | Loss: 2.8791 | LR: 1.00e-05 | 3461.5ms/step | 5.8 samples/s
553
+ Step 40100/44000 | Loss: 2.8884 | LR: 9.96e-06 | 3460.9ms/step | 5.8 samples/s
554
+ Step 40110/44000 | Loss: 2.8880 | LR: 9.91e-06 | 3461.2ms/step | 5.8 samples/s
555
+ Step 40120/44000 | Loss: 2.9020 | LR: 9.86e-06 | 3460.6ms/step | 5.8 samples/s
556
+ Step 40130/44000 | Loss: 2.8965 | LR: 9.81e-06 | 3461.0ms/step | 5.8 samples/s
557
+ Step 40140/44000 | Loss: 2.9012 | LR: 9.76e-06 | 3461.9ms/step | 5.8 samples/s
558
+ Step 40150/44000 | Loss: 2.8613 | LR: 9.71e-06 | 3462.3ms/step | 5.8 samples/s
559
+ Step 40160/44000 | Loss: 2.8673 | LR: 9.66e-06 | 3461.4ms/step | 5.8 samples/s
560
+ Step 40170/44000 | Loss: 2.8735 | LR: 9.61e-06 | 3459.8ms/step | 5.8 samples/s
561
+ Step 40180/44000 | Loss: 2.9121 | LR: 9.56e-06 | 3461.5ms/step | 5.8 samples/s
562
+ Step 40190/44000 | Loss: 2.8886 | LR: 9.51e-06 | 3460.9ms/step | 5.8 samples/s
563
+ Step 40200/44000 | Loss: 2.9207 | LR: 9.46e-06 | 3461.9ms/step | 5.8 samples/s
564
+ Step 40210/44000 | Loss: 2.8863 | LR: 9.41e-06 | 3460.2ms/step | 5.8 samples/s
565
+ Step 40220/44000 | Loss: 2.9045 | LR: 9.36e-06 | 3465.5ms/step | 5.8 samples/s
566
+ Step 40230/44000 | Loss: 2.8784 | LR: 9.32e-06 | 3461.3ms/step | 5.8 samples/s
567
+ Step 40240/44000 | Loss: 2.8884 | LR: 9.27e-06 | 3460.7ms/step | 5.8 samples/s
568
+ Step 40250/44000 | Loss: 2.8508 | LR: 9.22e-06 | 3461.3ms/step | 5.8 samples/s
569
+ Validation at step 40250: Loss = 2.9067, Perplexity = 18.30
570
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-40250
571
+ Step 40260/44000 | Loss: 2.8785 | LR: 9.17e-06 | 9530.9ms/step | 2.1 samples/s
572
+ Step 40270/44000 | Loss: 2.8976 | LR: 9.12e-06 | 3461.2ms/step | 5.8 samples/s
573
+ Step 40280/44000 | Loss: 2.8940 | LR: 9.07e-06 | 3459.9ms/step | 5.8 samples/s
574
+ Step 40290/44000 | Loss: 2.8674 | LR: 9.02e-06 | 3460.7ms/step | 5.8 samples/s
575
+ Step 40300/44000 | Loss: 2.8956 | LR: 8.97e-06 | 3460.4ms/step | 5.8 samples/s
576
+ Step 40310/44000 | Loss: 2.8828 | LR: 8.93e-06 | 3463.8ms/step | 5.8 samples/s
577
+ Step 40320/44000 | Loss: 2.9101 | LR: 8.88e-06 | 3463.9ms/step | 5.8 samples/s
578
+ Step 40330/44000 | Loss: 2.8896 | LR: 8.83e-06 | 3462.1ms/step | 5.8 samples/s
579
+ Step 40340/44000 | Loss: 2.8877 | LR: 8.78e-06 | 3464.5ms/step | 5.8 samples/s
580
+ Step 40350/44000 | Loss: 2.8792 | LR: 8.74e-06 | 3460.4ms/step | 5.8 samples/s
581
+ Step 40360/44000 | Loss: 2.8931 | LR: 8.69e-06 | 3460.7ms/step | 5.8 samples/s
582
+ Step 40370/44000 | Loss: 2.8692 | LR: 8.64e-06 | 3461.7ms/step | 5.8 samples/s
583
+ Step 40380/44000 | Loss: 2.8905 | LR: 8.59e-06 | 3460.9ms/step | 5.8 samples/s
584
+ Step 40390/44000 | Loss: 2.9072 | LR: 8.55e-06 | 3461.0ms/step | 5.8 samples/s
585
+ Step 40400/44000 | Loss: 2.8744 | LR: 8.50e-06 | 3460.2ms/step | 5.8 samples/s
586
+ Step 40410/44000 | Loss: 2.8609 | LR: 8.45e-06 | 3461.5ms/step | 5.8 samples/s
587
+ Step 40420/44000 | Loss: 2.8828 | LR: 8.41e-06 | 3462.1ms/step | 5.8 samples/s
588
+ Step 40430/44000 | Loss: 2.8929 | LR: 8.36e-06 | 3463.5ms/step | 5.8 samples/s
589
+ Step 40440/44000 | Loss: 2.8692 | LR: 8.31e-06 | 3460.9ms/step | 5.8 samples/s
590
+ Step 40450/44000 | Loss: 2.8821 | LR: 8.27e-06 | 3461.0ms/step | 5.8 samples/s
591
+ Step 40460/44000 | Loss: 2.8486 | LR: 8.22e-06 | 3461.8ms/step | 5.8 samples/s
592
+ Step 40470/44000 | Loss: 2.9063 | LR: 8.17e-06 | 3461.7ms/step | 5.8 samples/s
593
+ Step 40480/44000 | Loss: 2.8945 | LR: 8.13e-06 | 3461.9ms/step | 5.8 samples/s
594
+ Step 40490/44000 | Loss: 2.8937 | LR: 8.08e-06 | 3461.6ms/step | 5.8 samples/s
595
+ Step 40500/44000 | Loss: 2.8870 | LR: 8.04e-06 | 3461.7ms/step | 5.8 samples/s
596
+ Validation at step 40500: Loss = 2.9067, Perplexity = 18.30
597
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-40500
598
+ Step 40510/44000 | Loss: 2.8761 | LR: 7.99e-06 | 9539.6ms/step | 2.1 samples/s
599
+ Step 40520/44000 | Loss: 2.9233 | LR: 7.94e-06 | 3462.1ms/step | 5.8 samples/s
600
+ Step 40530/44000 | Loss: 2.9109 | LR: 7.90e-06 | 3461.7ms/step | 5.8 samples/s
601
+ Step 40540/44000 | Loss: 2.9026 | LR: 7.85e-06 | 3461.3ms/step | 5.8 samples/s
602
+ Step 40550/44000 | Loss: 2.8600 | LR: 7.81e-06 | 3461.6ms/step | 5.8 samples/s
603
+ Step 40560/44000 | Loss: 2.8618 | LR: 7.76e-06 | 3460.4ms/step | 5.8 samples/s
604
+ Step 40570/44000 | Loss: 2.8959 | LR: 7.72e-06 | 3461.5ms/step | 5.8 samples/s
605
+ Step 40580/44000 | Loss: 2.8874 | LR: 7.67e-06 | 3460.9ms/step | 5.8 samples/s
606
+ Step 40590/44000 | Loss: 2.8622 | LR: 7.63e-06 | 3463.7ms/step | 5.8 samples/s
607
+ Step 40600/44000 | Loss: 2.8906 | LR: 7.59e-06 | 3464.7ms/step | 5.8 samples/s
608
+ Step 40610/44000 | Loss: 2.9203 | LR: 7.54e-06 | 3463.9ms/step | 5.8 samples/s
609
+ Step 40620/44000 | Loss: 2.8883 | LR: 7.50e-06 | 3464.5ms/step | 5.8 samples/s
610
+ Step 40630/44000 | Loss: 2.9024 | LR: 7.45e-06 | 3460.7ms/step | 5.8 samples/s
611
+ Step 40640/44000 | Loss: 2.9081 | LR: 7.41e-06 | 3461.7ms/step | 5.8 samples/s
612
+ Step 40650/44000 | Loss: 2.9067 | LR: 7.37e-06 | 3460.8ms/step | 5.8 samples/s
613
+ Step 40660/44000 | Loss: 2.8961 | LR: 7.32e-06 | 3462.1ms/step | 5.8 samples/s
614
+ Step 40670/44000 | Loss: 2.9043 | LR: 7.28e-06 | 3463.3ms/step | 5.8 samples/s
615
+ Step 40680/44000 | Loss: 2.8872 | LR: 7.23e-06 | 3464.6ms/step | 5.8 samples/s
616
+ Step 40690/44000 | Loss: 2.8997 | LR: 7.19e-06 | 3464.6ms/step | 5.8 samples/s
617
+ Step 40700/44000 | Loss: 2.8786 | LR: 7.15e-06 | 3462.6ms/step | 5.8 samples/s
618
+ Step 40710/44000 | Loss: 2.8939 | LR: 7.10e-06 | 3464.2ms/step | 5.8 samples/s
619
+ Step 40720/44000 | Loss: 2.8996 | LR: 7.06e-06 | 3466.4ms/step | 5.8 samples/s
620
+ Step 40730/44000 | Loss: 2.9094 | LR: 7.02e-06 | 3467.2ms/step | 5.8 samples/s
621
+ Step 40740/44000 | Loss: 2.8827 | LR: 6.98e-06 | 3463.4ms/step | 5.8 samples/s
622
+ Step 40750/44000 | Loss: 2.8788 | LR: 6.93e-06 | 3463.5ms/step | 5.8 samples/s
623
+ Validation at step 40750: Loss = 2.9067, Perplexity = 18.30
624
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-40750
625
+ Step 40760/44000 | Loss: 2.9041 | LR: 6.89e-06 | 9553.2ms/step | 2.1 samples/s
626
+ Step 40770/44000 | Loss: 2.8845 | LR: 6.85e-06 | 3461.9ms/step | 5.8 samples/s
627
+ Step 40780/44000 | Loss: 2.9055 | LR: 6.81e-06 | 3462.7ms/step | 5.8 samples/s
628
+ Step 40790/44000 | Loss: 2.9079 | LR: 6.77e-06 | 3463.3ms/step | 5.8 samples/s
629
+ Step 40800/44000 | Loss: 2.9112 | LR: 6.72e-06 | 3463.1ms/step | 5.8 samples/s
630
+ Step 40810/44000 | Loss: 2.8915 | LR: 6.68e-06 | 3464.6ms/step | 5.8 samples/s
631
+ Step 40820/44000 | Loss: 2.8687 | LR: 6.64e-06 | 3463.4ms/step | 5.8 samples/s
632
+ Step 40830/44000 | Loss: 2.8658 | LR: 6.60e-06 | 3465.3ms/step | 5.8 samples/s
633
+ Step 40840/44000 | Loss: 2.9095 | LR: 6.56e-06 | 3464.9ms/step | 5.8 samples/s
634
+ Step 40850/44000 | Loss: 2.8969 | LR: 6.52e-06 | 3462.9ms/step | 5.8 samples/s
635
+ Step 40860/44000 | Loss: 2.9082 | LR: 6.47e-06 | 3465.0ms/step | 5.8 samples/s
636
+ Step 40870/44000 | Loss: 2.9124 | LR: 6.43e-06 | 3464.1ms/step | 5.8 samples/s
637
+ Step 40880/44000 | Loss: 2.8826 | LR: 6.39e-06 | 3464.5ms/step | 5.8 samples/s
638
+ Step 40890/44000 | Loss: 2.8989 | LR: 6.35e-06 | 3463.2ms/step | 5.8 samples/s
639
+ Step 40900/44000 | Loss: 2.8778 | LR: 6.31e-06 | 3462.4ms/step | 5.8 samples/s
640
+ Step 40910/44000 | Loss: 2.9457 | LR: 6.27e-06 | 3462.7ms/step | 5.8 samples/s
641
+ Step 40920/44000 | Loss: 2.8946 | LR: 6.23e-06 | 3462.1ms/step | 5.8 samples/s
642
+ Step 40930/44000 | Loss: 2.9021 | LR: 6.19e-06 | 3461.7ms/step | 5.8 samples/s
643
+ Step 40940/44000 | Loss: 2.9051 | LR: 6.15e-06 | 3462.5ms/step | 5.8 samples/s
644
+ Step 40950/44000 | Loss: 2.8666 | LR: 6.11e-06 | 3463.2ms/step | 5.8 samples/s
645
+ Step 40960/44000 | Loss: 2.8536 | LR: 6.07e-06 | 3464.0ms/step | 5.8 samples/s
646
+ Step 40970/44000 | Loss: 2.8837 | LR: 6.03e-06 | 3463.8ms/step | 5.8 samples/s
647
+ Step 40980/44000 | Loss: 2.8718 | LR: 5.99e-06 | 3462.4ms/step | 5.8 samples/s
648
+ Step 40990/44000 | Loss: 2.8941 | LR: 5.95e-06 | 3463.4ms/step | 5.8 samples/s
649
+ Step 41000/44000 | Loss: 2.8971 | LR: 5.91e-06 | 3463.3ms/step | 5.8 samples/s
650
+ Validation at step 41000: Loss = 2.9067, Perplexity = 18.30
651
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-41000
652
+ Step 41010/44000 | Loss: 2.9063 | LR: 5.87e-06 | 9540.0ms/step | 2.1 samples/s
653
+ Step 41020/44000 | Loss: 2.9122 | LR: 5.83e-06 | 3463.4ms/step | 5.8 samples/s
654
+ Step 41030/44000 | Loss: 2.8883 | LR: 5.80e-06 | 3464.3ms/step | 5.8 samples/s
655
+ Step 41040/44000 | Loss: 2.8885 | LR: 5.76e-06 | 3463.9ms/step | 5.8 samples/s
656
+ Step 41050/44000 | Loss: 2.8872 | LR: 5.72e-06 | 3465.9ms/step | 5.8 samples/s
657
+ Step 41060/44000 | Loss: 2.8575 | LR: 5.68e-06 | 3466.3ms/step | 5.8 samples/s
658
+ Step 41070/44000 | Loss: 2.8823 | LR: 5.64e-06 | 3465.0ms/step | 5.8 samples/s
659
+ Step 41080/44000 | Loss: 2.9153 | LR: 5.60e-06 | 3465.0ms/step | 5.8 samples/s
660
+ Step 41090/44000 | Loss: 2.9324 | LR: 5.56e-06 | 3466.1ms/step | 5.8 samples/s
661
+ Step 41100/44000 | Loss: 2.9125 | LR: 5.53e-06 | 3466.0ms/step | 5.8 samples/s
662
+ Step 41110/44000 | Loss: 2.9305 | LR: 5.49e-06 | 3465.5ms/step | 5.8 samples/s
663
+ Step 41120/44000 | Loss: 2.9211 | LR: 5.45e-06 | 3466.0ms/step | 5.8 samples/s
664
+ Step 41130/44000 | Loss: 2.8904 | LR: 5.41e-06 | 3467.8ms/step | 5.8 samples/s
665
+ Step 41140/44000 | Loss: 2.8678 | LR: 5.38e-06 | 3467.5ms/step | 5.8 samples/s
666
+ Step 41150/44000 | Loss: 2.9094 | LR: 5.34e-06 | 3466.3ms/step | 5.8 samples/s
667
+ Step 41160/44000 | Loss: 2.8825 | LR: 5.30e-06 | 3464.1ms/step | 5.8 samples/s
668
+ Step 41170/44000 | Loss: 2.8947 | LR: 5.26e-06 | 3464.3ms/step | 5.8 samples/s
669
+ Step 41180/44000 | Loss: 2.8707 | LR: 5.23e-06 | 3464.9ms/step | 5.8 samples/s
670
+ Step 41190/44000 | Loss: 2.8521 | LR: 5.19e-06 | 3464.5ms/step | 5.8 samples/s
671
+ Step 41200/44000 | Loss: 2.8918 | LR: 5.15e-06 | 3464.9ms/step | 5.8 samples/s
672
+ Step 41210/44000 | Loss: 2.9029 | LR: 5.12e-06 | 3465.6ms/step | 5.8 samples/s
673
+ Step 41220/44000 | Loss: 2.9096 | LR: 5.08e-06 | 3466.3ms/step | 5.8 samples/s
674
+ Step 41230/44000 | Loss: 2.8843 | LR: 5.04e-06 | 3465.4ms/step | 5.8 samples/s
675
+ Step 41240/44000 | Loss: 2.8711 | LR: 5.01e-06 | 3465.4ms/step | 5.8 samples/s
676
+ Step 41250/44000 | Loss: 2.8879 | LR: 4.97e-06 | 3466.0ms/step | 5.8 samples/s
677
+ Validation at step 41250: Loss = 2.9067, Perplexity = 18.30
678
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-41250
679
+ Step 41260/44000 | Loss: 2.9143 | LR: 4.94e-06 | 9549.9ms/step | 2.1 samples/s
680
+ Step 41270/44000 | Loss: 2.8937 | LR: 4.90e-06 | 3464.7ms/step | 5.8 samples/s
681
+ Step 41280/44000 | Loss: 2.8938 | LR: 4.86e-06 | 3466.1ms/step | 5.8 samples/s
682
+ Step 41290/44000 | Loss: 2.8808 | LR: 4.83e-06 | 3463.2ms/step | 5.8 samples/s
683
+ Step 41300/44000 | Loss: 2.8873 | LR: 4.79e-06 | 3462.1ms/step | 5.8 samples/s
684
+ Step 41310/44000 | Loss: 2.8724 | LR: 4.76e-06 | 3464.5ms/step | 5.8 samples/s
685
+ Step 41320/44000 | Loss: 2.8971 | LR: 4.72e-06 | 3464.3ms/step | 5.8 samples/s
686
+ Step 41330/44000 | Loss: 2.8830 | LR: 4.69e-06 | 3464.9ms/step | 5.8 samples/s
687
+ Step 41340/44000 | Loss: 2.9051 | LR: 4.65e-06 | 3465.0ms/step | 5.8 samples/s
688
+ Step 41350/44000 | Loss: 2.8696 | LR: 4.62e-06 | 3466.4ms/step | 5.8 samples/s
689
+ Step 41360/44000 | Loss: 2.9193 | LR: 4.58e-06 | 3463.8ms/step | 5.8 samples/s
690
+ Step 41370/44000 | Loss: 2.8987 | LR: 4.55e-06 | 3465.6ms/step | 5.8 samples/s
691
+ Step 41380/44000 | Loss: 2.8945 | LR: 4.51e-06 | 3465.3ms/step | 5.8 samples/s
692
+ Step 41390/44000 | Loss: 2.8726 | LR: 4.48e-06 | 3464.9ms/step | 5.8 samples/s
693
+ Step 41400/44000 | Loss: 2.8611 | LR: 4.45e-06 | 3467.2ms/step | 5.8 samples/s
694
+ Step 41410/44000 | Loss: 2.8658 | LR: 4.41e-06 | 3465.4ms/step | 5.8 samples/s
695
+ Step 41420/44000 | Loss: 2.8575 | LR: 4.38e-06 | 3467.2ms/step | 5.8 samples/s
696
+ Step 41430/44000 | Loss: 2.8840 | LR: 4.34e-06 | 3464.9ms/step | 5.8 samples/s
697
+ Step 41440/44000 | Loss: 2.9249 | LR: 4.31e-06 | 3463.3ms/step | 5.8 samples/s
698
+ Step 41450/44000 | Loss: 2.8904 | LR: 4.28e-06 | 3464.8ms/step | 5.8 samples/s
699
+ Step 41460/44000 | Loss: 2.8906 | LR: 4.24e-06 | 3464.8ms/step | 5.8 samples/s
700
+ Step 41470/44000 | Loss: 2.8763 | LR: 4.21e-06 | 3467.0ms/step | 5.8 samples/s
701
+ Step 41480/44000 | Loss: 2.9063 | LR: 4.18e-06 | 3464.1ms/step | 5.8 samples/s
702
+ Step 41490/44000 | Loss: 2.8745 | LR: 4.14e-06 | 3465.4ms/step | 5.8 samples/s
703
+ Step 41500/44000 | Loss: 2.8906 | LR: 4.11e-06 | 3466.3ms/step | 5.8 samples/s
704
+ Validation at step 41500: Loss = 2.9067, Perplexity = 18.30
705
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-41500
706
+ Step 41510/44000 | Loss: 2.8856 | LR: 4.08e-06 | 9552.2ms/step | 2.1 samples/s
707
+ Step 41520/44000 | Loss: 2.8873 | LR: 4.05e-06 | 3467.0ms/step | 5.8 samples/s
708
+ Step 41530/44000 | Loss: 2.8917 | LR: 4.01e-06 | 3467.5ms/step | 5.8 samples/s
709
+ Step 41540/44000 | Loss: 2.9139 | LR: 3.98e-06 | 3467.4ms/step | 5.8 samples/s
710
+ Step 41550/44000 | Loss: 2.8869 | LR: 3.95e-06 | 3468.5ms/step | 5.8 samples/s
711
+ Step 41560/44000 | Loss: 2.9025 | LR: 3.92e-06 | 3468.8ms/step | 5.8 samples/s
712
+ Step 41570/44000 | Loss: 2.9181 | LR: 3.88e-06 | 3468.8ms/step | 5.8 samples/s
713
+ Step 41580/44000 | Loss: 2.8745 | LR: 3.85e-06 | 3469.0ms/step | 5.8 samples/s
714
+ Step 41590/44000 | Loss: 2.8868 | LR: 3.82e-06 | 3469.5ms/step | 5.8 samples/s
715
+ Step 41600/44000 | Loss: 2.9009 | LR: 3.79e-06 | 3469.7ms/step | 5.8 samples/s
716
+ Step 41610/44000 | Loss: 2.8785 | LR: 3.76e-06 | 3470.0ms/step | 5.8 samples/s
717
+ Step 41620/44000 | Loss: 2.8548 | LR: 3.73e-06 | 3467.7ms/step | 5.8 samples/s
718
+ Step 41630/44000 | Loss: 2.9118 | LR: 3.70e-06 | 3469.7ms/step | 5.8 samples/s
719
+ Step 41640/44000 | Loss: 2.9109 | LR: 3.66e-06 | 3466.8ms/step | 5.8 samples/s
720
+ Step 41650/44000 | Loss: 2.8941 | LR: 3.63e-06 | 3469.1ms/step | 5.8 samples/s
721
+ Step 41660/44000 | Loss: 2.9140 | LR: 3.60e-06 | 3470.2ms/step | 5.8 samples/s
722
+ Step 41670/44000 | Loss: 2.8862 | LR: 3.57e-06 | 3468.1ms/step | 5.8 samples/s
723
+ Step 41680/44000 | Loss: 2.8604 | LR: 3.54e-06 | 3466.8ms/step | 5.8 samples/s
724
+ Step 41690/44000 | Loss: 2.8734 | LR: 3.51e-06 | 3467.5ms/step | 5.8 samples/s
725
+ Step 41700/44000 | Loss: 2.9092 | LR: 3.48e-06 | 3467.1ms/step | 5.8 samples/s
726
+ Step 41710/44000 | Loss: 2.8814 | LR: 3.45e-06 | 3466.8ms/step | 5.8 samples/s
727
+ Step 41720/44000 | Loss: 2.8835 | LR: 3.42e-06 | 3466.1ms/step | 5.8 samples/s
728
+ Step 41730/44000 | Loss: 2.8971 | LR: 3.39e-06 | 3467.2ms/step | 5.8 samples/s
729
+ Step 41740/44000 | Loss: 2.9082 | LR: 3.36e-06 | 3469.2ms/step | 5.8 samples/s
730
+ Step 41750/44000 | Loss: 2.8826 | LR: 3.33e-06 | 3469.3ms/step | 5.8 samples/s
731
+ Validation at step 41750: Loss = 2.9066, Perplexity = 18.30
732
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-41750
733
+ Step 41760/44000 | Loss: 2.9140 | LR: 3.30e-06 | 9549.4ms/step | 2.1 samples/s
734
+ Step 41770/44000 | Loss: 2.8877 | LR: 3.27e-06 | 3467.6ms/step | 5.8 samples/s
735
+ Step 41780/44000 | Loss: 2.8817 | LR: 3.24e-06 | 3467.4ms/step | 5.8 samples/s
736
+ Step 41790/44000 | Loss: 2.9246 | LR: 3.21e-06 | 3468.6ms/step | 5.8 samples/s
737
+ Step 41800/44000 | Loss: 2.9362 | LR: 3.19e-06 | 3465.6ms/step | 5.8 samples/s
738
+ Step 41810/44000 | Loss: 2.9010 | LR: 3.16e-06 | 3466.9ms/step | 5.8 samples/s
739
+ Step 41820/44000 | Loss: 2.8744 | LR: 3.13e-06 | 3463.1ms/step | 5.8 samples/s
740
+ Step 41830/44000 | Loss: 2.8754 | LR: 3.10e-06 | 3466.0ms/step | 5.8 samples/s
741
+ Step 41840/44000 | Loss: 2.8588 | LR: 3.07e-06 | 3467.2ms/step | 5.8 samples/s
742
+ Step 41850/44000 | Loss: 2.8657 | LR: 3.04e-06 | 3467.5ms/step | 5.8 samples/s
743
+ Step 41860/44000 | Loss: 2.9137 | LR: 3.01e-06 | 3468.6ms/step | 5.8 samples/s
744
+ Step 41870/44000 | Loss: 2.9238 | LR: 2.99e-06 | 3468.1ms/step | 5.8 samples/s
745
+ Step 41880/44000 | Loss: 2.9006 | LR: 2.96e-06 | 3466.5ms/step | 5.8 samples/s
746
+ Step 41890/44000 | Loss: 2.8820 | LR: 2.93e-06 | 3465.7ms/step | 5.8 samples/s
747
+ Step 41900/44000 | Loss: 2.8843 | LR: 2.90e-06 | 3468.8ms/step | 5.8 samples/s
748
+ Step 41910/44000 | Loss: 2.8796 | LR: 2.88e-06 | 3469.3ms/step | 5.8 samples/s
749
+ Step 41920/44000 | Loss: 2.9287 | LR: 2.85e-06 | 3471.0ms/step | 5.8 samples/s
750
+ Step 41930/44000 | Loss: 2.8825 | LR: 2.82e-06 | 3468.5ms/step | 5.8 samples/s
751
+ Step 41940/44000 | Loss: 2.8904 | LR: 2.79e-06 | 3467.0ms/step | 5.8 samples/s
752
+ Step 41950/44000 | Loss: 2.8509 | LR: 2.77e-06 | 3468.5ms/step | 5.8 samples/s
753
+ Step 41960/44000 | Loss: 2.9012 | LR: 2.74e-06 | 3467.4ms/step | 5.8 samples/s
754
+ Step 41970/44000 | Loss: 2.8931 | LR: 2.71e-06 | 3468.4ms/step | 5.8 samples/s
755
+ Step 41980/44000 | Loss: 2.8893 | LR: 2.69e-06 | 3469.8ms/step | 5.8 samples/s
756
+ Step 41990/44000 | Loss: 2.8919 | LR: 2.66e-06 | 3468.8ms/step | 5.8 samples/s
757
+ Step 42000/44000 | Loss: 2.8888 | LR: 2.63e-06 | 3468.4ms/step | 5.8 samples/s
758
+ Validation at step 42000: Loss = 2.9066, Perplexity = 18.30
759
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-42000
760
+ Step 42010/44000 | Loss: 2.8986 | LR: 2.61e-06 | 9551.3ms/step | 2.1 samples/s
761
+ Step 42020/44000 | Loss: 2.9085 | LR: 2.58e-06 | 3466.3ms/step | 5.8 samples/s
762
+ Step 42030/44000 | Loss: 2.9140 | LR: 2.56e-06 | 3467.3ms/step | 5.8 samples/s
763
+ Step 42040/44000 | Loss: 2.9106 | LR: 2.53e-06 | 3467.6ms/step | 5.8 samples/s
764
+ Step 42050/44000 | Loss: 2.8681 | LR: 2.50e-06 | 3469.4ms/step | 5.8 samples/s
765
+ Step 42060/44000 | Loss: 2.9070 | LR: 2.48e-06 | 3469.0ms/step | 5.8 samples/s
766
+ Step 42070/44000 | Loss: 2.9059 | LR: 2.45e-06 | 3468.8ms/step | 5.8 samples/s
767
+ Step 42080/44000 | Loss: 2.8854 | LR: 2.43e-06 | 3468.7ms/step | 5.8 samples/s
768
+ Step 42090/44000 | Loss: 2.8728 | LR: 2.40e-06 | 3470.0ms/step | 5.8 samples/s
769
+ Step 42100/44000 | Loss: 2.8932 | LR: 2.38e-06 | 3469.5ms/step | 5.8 samples/s
770
+ Step 42110/44000 | Loss: 2.9064 | LR: 2.35e-06 | 3470.0ms/step | 5.8 samples/s
771
+ Step 42120/44000 | Loss: 2.8916 | LR: 2.33e-06 | 3469.6ms/step | 5.8 samples/s
772
+ Step 42130/44000 | Loss: 2.9031 | LR: 2.30e-06 | 3470.0ms/step | 5.8 samples/s
773
+ Step 42140/44000 | Loss: 2.9012 | LR: 2.28e-06 | 3470.2ms/step | 5.8 samples/s
774
+ Step 42150/44000 | Loss: 2.9223 | LR: 2.25e-06 | 3470.3ms/step | 5.8 samples/s
775
+ Step 42160/44000 | Loss: 2.8884 | LR: 2.23e-06 | 3470.6ms/step | 5.8 samples/s
776
+ Step 42170/44000 | Loss: 2.8763 | LR: 2.21e-06 | 3467.9ms/step | 5.8 samples/s
777
+ Step 42180/44000 | Loss: 2.9035 | LR: 2.18e-06 | 3467.4ms/step | 5.8 samples/s
778
+ Step 42190/44000 | Loss: 2.8876 | LR: 2.16e-06 | 3468.0ms/step | 5.8 samples/s
779
+ Step 42200/44000 | Loss: 2.8744 | LR: 2.13e-06 | 3469.8ms/step | 5.8 samples/s
780
+ Step 42210/44000 | Loss: 2.9192 | LR: 2.11e-06 | 3468.6ms/step | 5.8 samples/s
781
+ Step 42220/44000 | Loss: 2.8591 | LR: 2.09e-06 | 3469.7ms/step | 5.8 samples/s
782
+ Step 42230/44000 | Loss: 2.9104 | LR: 2.06e-06 | 3470.7ms/step | 5.8 samples/s
783
+ Step 42240/44000 | Loss: 2.8902 | LR: 2.04e-06 | 3470.3ms/step | 5.8 samples/s
784
+ Step 42250/44000 | Loss: 2.9054 | LR: 2.02e-06 | 3469.8ms/step | 5.8 samples/s
785
+ Validation at step 42250: Loss = 2.9067, Perplexity = 18.30
786
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-42250
787
+ Step 42260/44000 | Loss: 2.8955 | LR: 1.99e-06 | 9559.0ms/step | 2.1 samples/s
788
+ Step 42270/44000 | Loss: 2.8887 | LR: 1.97e-06 | 3469.5ms/step | 5.8 samples/s
789
+ Step 42280/44000 | Loss: 2.8926 | LR: 1.95e-06 | 3471.1ms/step | 5.8 samples/s
790
+ Step 42290/44000 | Loss: 2.8602 | LR: 1.93e-06 | 3468.7ms/step | 5.8 samples/s
791
+ Step 42300/44000 | Loss: 2.8892 | LR: 1.90e-06 | 3463.1ms/step | 5.8 samples/s
792
+ Step 42310/44000 | Loss: 2.8625 | LR: 1.88e-06 | 3465.4ms/step | 5.8 samples/s
793
+ Step 42320/44000 | Loss: 2.9077 | LR: 1.86e-06 | 3464.3ms/step | 5.8 samples/s
794
+ Step 42330/44000 | Loss: 2.8729 | LR: 1.84e-06 | 3463.5ms/step | 5.8 samples/s
795
+ Step 42340/44000 | Loss: 2.9196 | LR: 1.82e-06 | 3465.8ms/step | 5.8 samples/s
796
+ Step 42350/44000 | Loss: 2.8495 | LR: 1.79e-06 | 3467.5ms/step | 5.8 samples/s
797
+ Step 42360/44000 | Loss: 2.8922 | LR: 1.77e-06 | 3466.4ms/step | 5.8 samples/s
798
+ Step 42370/44000 | Loss: 2.8907 | LR: 1.75e-06 | 3469.6ms/step | 5.8 samples/s
799
+ Step 42380/44000 | Loss: 2.9240 | LR: 1.73e-06 | 3470.4ms/step | 5.8 samples/s
800
+ Step 42390/44000 | Loss: 2.8558 | LR: 1.71e-06 | 3468.9ms/step | 5.8 samples/s
801
+ Step 42400/44000 | Loss: 2.8830 | LR: 1.69e-06 | 3471.4ms/step | 5.8 samples/s
802
+ Step 42410/44000 | Loss: 2.8894 | LR: 1.67e-06 | 3468.0ms/step | 5.8 samples/s
803
+ Step 42420/44000 | Loss: 2.9166 | LR: 1.64e-06 | 3467.9ms/step | 5.8 samples/s
804
+ Step 42430/44000 | Loss: 2.9194 | LR: 1.62e-06 | 3467.9ms/step | 5.8 samples/s
805
+ Step 42440/44000 | Loss: 2.9085 | LR: 1.60e-06 | 3469.9ms/step | 5.8 samples/s
806
+ Step 42450/44000 | Loss: 2.8774 | LR: 1.58e-06 | 3471.0ms/step | 5.8 samples/s
807
+ Step 42460/44000 | Loss: 2.8878 | LR: 1.56e-06 | 3467.6ms/step | 5.8 samples/s
808
+ Step 42470/44000 | Loss: 2.9216 | LR: 1.54e-06 | 3468.8ms/step | 5.8 samples/s
809
+ Step 42480/44000 | Loss: 2.8912 | LR: 1.52e-06 | 3471.2ms/step | 5.8 samples/s
810
+ Step 42490/44000 | Loss: 2.8963 | LR: 1.50e-06 | 3467.5ms/step | 5.8 samples/s
811
+ Step 42500/44000 | Loss: 2.8669 | LR: 1.48e-06 | 3468.2ms/step | 5.8 samples/s
812
+ Validation at step 42500: Loss = 2.9067, Perplexity = 18.30
813
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-42500
814
+ Step 42510/44000 | Loss: 2.9100 | LR: 1.46e-06 | 9558.8ms/step | 2.1 samples/s
815
+ Step 42520/44000 | Loss: 2.8673 | LR: 1.44e-06 | 3468.9ms/step | 5.8 samples/s
816
+ Step 42530/44000 | Loss: 2.8961 | LR: 1.42e-06 | 3469.2ms/step | 5.8 samples/s
817
+ Step 42540/44000 | Loss: 2.8834 | LR: 1.40e-06 | 3469.8ms/step | 5.8 samples/s
818
+ Step 42550/44000 | Loss: 2.9002 | LR: 1.39e-06 | 3468.6ms/step | 5.8 samples/s
819
+ Step 42560/44000 | Loss: 2.8725 | LR: 1.37e-06 | 3469.7ms/step | 5.8 samples/s
820
+ Step 42570/44000 | Loss: 2.8913 | LR: 1.35e-06 | 3470.4ms/step | 5.8 samples/s
821
+ Step 42580/44000 | Loss: 2.8909 | LR: 1.33e-06 | 3471.7ms/step | 5.8 samples/s
822
+ Step 42590/44000 | Loss: 2.8765 | LR: 1.31e-06 | 3469.7ms/step | 5.8 samples/s
823
+ Step 42600/44000 | Loss: 2.8708 | LR: 1.29e-06 | 3470.7ms/step | 5.8 samples/s
824
+ Step 42610/44000 | Loss: 2.8978 | LR: 1.27e-06 | 3470.6ms/step | 5.8 samples/s
825
+ Step 42620/44000 | Loss: 2.9069 | LR: 1.25e-06 | 3471.7ms/step | 5.8 samples/s
826
+ Step 42630/44000 | Loss: 2.8754 | LR: 1.24e-06 | 3469.8ms/step | 5.8 samples/s
827
+ Step 42640/44000 | Loss: 2.8842 | LR: 1.22e-06 | 3468.4ms/step | 5.8 samples/s
828
+ Step 42650/44000 | Loss: 2.9076 | LR: 1.20e-06 | 3468.5ms/step | 5.8 samples/s
829
+ Step 42660/44000 | Loss: 2.8749 | LR: 1.18e-06 | 3468.8ms/step | 5.8 samples/s
830
+ Step 42670/44000 | Loss: 2.8741 | LR: 1.17e-06 | 3470.0ms/step | 5.8 samples/s
831
+ Step 42680/44000 | Loss: 2.8693 | LR: 1.15e-06 | 3470.0ms/step | 5.8 samples/s
832
+ Step 42690/44000 | Loss: 2.8597 | LR: 1.13e-06 | 3470.6ms/step | 5.8 samples/s
833
+ Step 42700/44000 | Loss: 2.8658 | LR: 1.11e-06 | 3469.2ms/step | 5.8 samples/s
834
+ Step 42710/44000 | Loss: 2.9033 | LR: 1.10e-06 | 3469.3ms/step | 5.8 samples/s
835
+ Step 42720/44000 | Loss: 2.8957 | LR: 1.08e-06 | 3469.9ms/step | 5.8 samples/s
836
+ Step 42730/44000 | Loss: 2.8888 | LR: 1.06e-06 | 3469.5ms/step | 5.8 samples/s
837
+ Step 42740/44000 | Loss: 2.8638 | LR: 1.05e-06 | 3469.3ms/step | 5.8 samples/s
838
+ Step 42750/44000 | Loss: 2.8729 | LR: 1.03e-06 | 3468.5ms/step | 5.8 samples/s
839
+ Validation at step 42750: Loss = 2.9067, Perplexity = 18.30
840
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-42750
841
+ Step 42760/44000 | Loss: 2.8891 | LR: 1.01e-06 | 9552.5ms/step | 2.1 samples/s
842
+ Step 42770/44000 | Loss: 2.9043 | LR: 9.97e-07 | 3467.1ms/step | 5.8 samples/s
843
+ Step 42780/44000 | Loss: 2.9096 | LR: 9.81e-07 | 3467.0ms/step | 5.8 samples/s
844
+ Step 42790/44000 | Loss: 2.8823 | LR: 9.65e-07 | 3467.3ms/step | 5.8 samples/s
845
+ Step 42800/44000 | Loss: 2.8739 | LR: 9.49e-07 | 3467.2ms/step | 5.8 samples/s
846
+ Step 42810/44000 | Loss: 2.8694 | LR: 9.33e-07 | 3468.8ms/step | 5.8 samples/s
847
+ Step 42820/44000 | Loss: 2.9278 | LR: 9.18e-07 | 3469.7ms/step | 5.8 samples/s
848
+ Step 42830/44000 | Loss: 2.8886 | LR: 9.02e-07 | 3468.8ms/step | 5.8 samples/s
849
+ Step 42840/44000 | Loss: 2.9016 | LR: 8.87e-07 | 3466.0ms/step | 5.8 samples/s
850
+ Step 42850/44000 | Loss: 2.8994 | LR: 8.72e-07 | 3469.9ms/step | 5.8 samples/s
851
+ Step 42860/44000 | Loss: 2.9078 | LR: 8.57e-07 | 3468.7ms/step | 5.8 samples/s
852
+ Step 42870/44000 | Loss: 2.9050 | LR: 8.42e-07 | 3468.7ms/step | 5.8 samples/s
853
+ Step 42880/44000 | Loss: 2.8917 | LR: 8.27e-07 | 3470.4ms/step | 5.8 samples/s
854
+ Step 42890/44000 | Loss: 2.8458 | LR: 8.12e-07 | 3469.5ms/step | 5.8 samples/s
855
+ Step 42900/44000 | Loss: 2.9030 | LR: 7.98e-07 | 3469.4ms/step | 5.8 samples/s
856
+ Step 42910/44000 | Loss: 2.8945 | LR: 7.83e-07 | 3467.6ms/step | 5.8 samples/s
857
+ Step 42920/44000 | Loss: 2.8763 | LR: 7.69e-07 | 3469.7ms/step | 5.8 samples/s
858
+ Step 42930/44000 | Loss: 2.8691 | LR: 7.55e-07 | 3468.4ms/step | 5.8 samples/s
859
+ Step 42940/44000 | Loss: 2.8808 | LR: 7.41e-07 | 3468.3ms/step | 5.8 samples/s
860
+ Step 42950/44000 | Loss: 2.8980 | LR: 7.27e-07 | 3468.5ms/step | 5.8 samples/s
861
+ Step 42960/44000 | Loss: 2.8879 | LR: 7.13e-07 | 3469.0ms/step | 5.8 samples/s
862
+ Step 42970/44000 | Loss: 2.8825 | LR: 6.99e-07 | 3468.2ms/step | 5.8 samples/s
863
+ Step 42980/44000 | Loss: 2.8933 | LR: 6.86e-07 | 3469.6ms/step | 5.8 samples/s
864
+ Step 42990/44000 | Loss: 2.8895 | LR: 6.72e-07 | 3471.0ms/step | 5.8 samples/s
865
+ Step 43000/44000 | Loss: 2.8815 | LR: 6.59e-07 | 3467.6ms/step | 5.8 samples/s
866
+ Validation at step 43000: Loss = 2.9067, Perplexity = 18.30
867
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-43000
868
+ Step 43010/44000 | Loss: 2.8864 | LR: 6.46e-07 | 9557.7ms/step | 2.1 samples/s
869
+ Step 43020/44000 | Loss: 2.8746 | LR: 6.33e-07 | 3468.6ms/step | 5.8 samples/s
870
+ Step 43030/44000 | Loss: 2.8931 | LR: 6.20e-07 | 3467.6ms/step | 5.8 samples/s
871
+ Step 43040/44000 | Loss: 2.8582 | LR: 6.08e-07 | 3467.2ms/step | 5.8 samples/s
872
+ Step 43050/44000 | Loss: 2.8849 | LR: 5.95e-07 | 3470.0ms/step | 5.8 samples/s
873
+ Step 43060/44000 | Loss: 2.8922 | LR: 5.83e-07 | 3467.1ms/step | 5.8 samples/s
874
+ Step 43070/44000 | Loss: 2.9100 | LR: 5.70e-07 | 3468.1ms/step | 5.8 samples/s
875
+ Step 43080/44000 | Loss: 2.8824 | LR: 5.58e-07 | 3465.8ms/step | 5.8 samples/s
876
+ Step 43090/44000 | Loss: 2.8643 | LR: 5.46e-07 | 3463.9ms/step | 5.8 samples/s
877
+ Step 43100/44000 | Loss: 2.8798 | LR: 5.34e-07 | 3467.1ms/step | 5.8 samples/s
878
+ Step 43110/44000 | Loss: 2.8917 | LR: 5.22e-07 | 3465.1ms/step | 5.8 samples/s
879
+ Step 43120/44000 | Loss: 2.9010 | LR: 5.11e-07 | 3466.6ms/step | 5.8 samples/s
880
+ Step 43130/44000 | Loss: 2.9090 | LR: 4.99e-07 | 3467.0ms/step | 5.8 samples/s
881
+ Step 43140/44000 | Loss: 2.8928 | LR: 4.88e-07 | 3467.0ms/step | 5.8 samples/s
882
+ Step 43150/44000 | Loss: 2.8775 | LR: 4.76e-07 | 3468.1ms/step | 5.8 samples/s
883
+ Step 43160/44000 | Loss: 2.8743 | LR: 4.65e-07 | 3467.7ms/step | 5.8 samples/s
884
+ Step 43170/44000 | Loss: 2.8624 | LR: 4.54e-07 | 3467.7ms/step | 5.8 samples/s
885
+ Step 43180/44000 | Loss: 2.8740 | LR: 4.43e-07 | 3468.4ms/step | 5.8 samples/s
886
+ Step 43190/44000 | Loss: 2.8682 | LR: 4.33e-07 | 3469.2ms/step | 5.8 samples/s
887
+ Step 43200/44000 | Loss: 2.8581 | LR: 4.22e-07 | 3467.7ms/step | 5.8 samples/s
888
+ Step 43210/44000 | Loss: 2.8943 | LR: 4.12e-07 | 3469.2ms/step | 5.8 samples/s
889
+ Step 43220/44000 | Loss: 2.8819 | LR: 4.01e-07 | 3468.7ms/step | 5.8 samples/s
890
+ Step 43230/44000 | Loss: 2.9017 | LR: 3.91e-07 | 3469.2ms/step | 5.8 samples/s
891
+ Step 43240/44000 | Loss: 2.8992 | LR: 3.81e-07 | 3467.6ms/step | 5.8 samples/s
892
+ Step 43250/44000 | Loss: 2.9054 | LR: 3.71e-07 | 3468.4ms/step | 5.8 samples/s
893
+ Validation at step 43250: Loss = 2.9067, Perplexity = 18.30
894
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-43250
895
+ Step 43260/44000 | Loss: 2.8959 | LR: 3.61e-07 | 9554.6ms/step | 2.1 samples/s
896
+ Step 43270/44000 | Loss: 2.8847 | LR: 3.51e-07 | 3466.5ms/step | 5.8 samples/s
897
+ Step 43280/44000 | Loss: 2.8908 | LR: 3.42e-07 | 3467.9ms/step | 5.8 samples/s
898
+ Step 43290/44000 | Loss: 2.8763 | LR: 3.32e-07 | 3467.9ms/step | 5.8 samples/s
899
+ Step 43300/44000 | Loss: 2.9075 | LR: 3.23e-07 | 3468.2ms/step | 5.8 samples/s
900
+ Step 43310/44000 | Loss: 2.9111 | LR: 3.14e-07 | 3467.5ms/step | 5.8 samples/s
901
+ Step 43320/44000 | Loss: 2.9149 | LR: 3.05e-07 | 3465.9ms/step | 5.8 samples/s
902
+ Step 43330/44000 | Loss: 2.8767 | LR: 2.96e-07 | 3466.0ms/step | 5.8 samples/s
903
+ Step 43340/44000 | Loss: 2.8998 | LR: 2.87e-07 | 3468.4ms/step | 5.8 samples/s
904
+ Step 43350/44000 | Loss: 2.8748 | LR: 2.79e-07 | 3468.4ms/step | 5.8 samples/s
905
+ Step 43360/44000 | Loss: 2.8938 | LR: 2.70e-07 | 3467.3ms/step | 5.8 samples/s
906
+ Step 43370/44000 | Loss: 2.8986 | LR: 2.62e-07 | 3467.3ms/step | 5.8 samples/s
907
+ Step 43380/44000 | Loss: 2.8933 | LR: 2.53e-07 | 3467.1ms/step | 5.8 samples/s
908
+ Step 43390/44000 | Loss: 2.8677 | LR: 2.45e-07 | 3467.2ms/step | 5.8 samples/s
909
+ Step 43400/44000 | Loss: 2.8837 | LR: 2.37e-07 | 3466.5ms/step | 5.8 samples/s
910
+ Step 43410/44000 | Loss: 2.8832 | LR: 2.30e-07 | 3466.7ms/step | 5.8 samples/s
911
+ Step 43420/44000 | Loss: 2.8617 | LR: 2.22e-07 | 3465.4ms/step | 5.8 samples/s
912
+ Step 43430/44000 | Loss: 2.8771 | LR: 2.14e-07 | 3467.8ms/step | 5.8 samples/s
913
+ Step 43440/44000 | Loss: 2.9234 | LR: 2.07e-07 | 3469.4ms/step | 5.8 samples/s
914
+ Step 43450/44000 | Loss: 2.9049 | LR: 1.99e-07 | 3469.1ms/step | 5.8 samples/s
915
+ Step 43460/44000 | Loss: 2.8971 | LR: 1.92e-07 | 3466.5ms/step | 5.8 samples/s
916
+ Step 43470/44000 | Loss: 2.8954 | LR: 1.85e-07 | 3466.1ms/step | 5.8 samples/s
917
+ Step 43480/44000 | Loss: 2.8653 | LR: 1.78e-07 | 3467.4ms/step | 5.8 samples/s
918
+ Step 43490/44000 | Loss: 2.8814 | LR: 1.72e-07 | 3467.0ms/step | 5.8 samples/s
919
+ Step 43500/44000 | Loss: 2.9240 | LR: 1.65e-07 | 3467.5ms/step | 5.8 samples/s
920
+ Validation at step 43500: Loss = 2.9067, Perplexity = 18.30
921
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-43500
922
+ Step 43510/44000 | Loss: 2.8909 | LR: 1.58e-07 | 9565.3ms/step | 2.1 samples/s
923
+ Step 43520/44000 | Loss: 2.9096 | LR: 1.52e-07 | 3467.1ms/step | 5.8 samples/s
924
+ Step 43530/44000 | Loss: 2.8712 | LR: 1.46e-07 | 3467.3ms/step | 5.8 samples/s
925
+ Step 43540/44000 | Loss: 2.8590 | LR: 1.40e-07 | 3466.4ms/step | 5.8 samples/s
926
+ Step 43550/44000 | Loss: 2.8829 | LR: 1.34e-07 | 3468.4ms/step | 5.8 samples/s
927
+ Step 43560/44000 | Loss: 2.8843 | LR: 1.28e-07 | 3466.0ms/step | 5.8 samples/s
928
+ Step 43570/44000 | Loss: 2.8822 | LR: 1.22e-07 | 3467.1ms/step | 5.8 samples/s
929
+ Step 43580/44000 | Loss: 2.9079 | LR: 1.16e-07 | 3466.8ms/step | 5.8 samples/s
930
+ Step 43590/44000 | Loss: 2.8830 | LR: 1.11e-07 | 3463.7ms/step | 5.8 samples/s
931
+ Step 43600/44000 | Loss: 2.8953 | LR: 1.06e-07 | 3467.3ms/step | 5.8 samples/s
932
+ Step 43610/44000 | Loss: 2.8427 | LR: 1.00e-07 | 3467.6ms/step | 5.8 samples/s
933
+ Step 43620/44000 | Loss: 2.8817 | LR: 9.52e-08 | 3465.2ms/step | 5.8 samples/s
934
+ Step 43630/44000 | Loss: 2.8739 | LR: 9.03e-08 | 3467.6ms/step | 5.8 samples/s
935
+ Step 43640/44000 | Loss: 2.8899 | LR: 8.55e-08 | 3468.0ms/step | 5.8 samples/s
936
+ Step 43650/44000 | Loss: 2.8920 | LR: 8.08e-08 | 3466.1ms/step | 5.8 samples/s
937
+ Step 43660/44000 | Loss: 2.8925 | LR: 7.62e-08 | 3468.4ms/step | 5.8 samples/s
938
+ Step 43670/44000 | Loss: 2.9009 | LR: 7.18e-08 | 3466.5ms/step | 5.8 samples/s
939
+ Step 43680/44000 | Loss: 2.9077 | LR: 6.75e-08 | 3466.6ms/step | 5.8 samples/s
940
+ Step 43690/44000 | Loss: 2.8680 | LR: 6.34e-08 | 3467.1ms/step | 5.8 samples/s
941
+ Step 43700/44000 | Loss: 2.8906 | LR: 5.94e-08 | 3466.0ms/step | 5.8 samples/s
942
+ Step 43710/44000 | Loss: 2.9080 | LR: 5.55e-08 | 3466.4ms/step | 5.8 samples/s
943
+ Step 43720/44000 | Loss: 2.8746 | LR: 5.17e-08 | 3466.2ms/step | 5.8 samples/s
944
+ Step 43730/44000 | Loss: 2.9117 | LR: 4.81e-08 | 3466.9ms/step | 5.8 samples/s
945
+ Step 43740/44000 | Loss: 2.9213 | LR: 4.46e-08 | 3465.8ms/step | 5.8 samples/s
946
+ Step 43750/44000 | Loss: 2.8914 | LR: 4.12e-08 | 3467.1ms/step | 5.8 samples/s
947
+ Validation at step 43750: Loss = 2.9067, Perplexity = 18.30
948
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-43750
949
+ Step 43760/44000 | Loss: 2.9103 | LR: 3.80e-08 | 9557.2ms/step | 2.1 samples/s
950
+ Step 43770/44000 | Loss: 2.9062 | LR: 3.49e-08 | 3469.2ms/step | 5.8 samples/s
951
+ Step 43780/44000 | Loss: 2.8891 | LR: 3.19e-08 | 3468.3ms/step | 5.8 samples/s
952
+ Step 43790/44000 | Loss: 2.8785 | LR: 2.91e-08 | 3469.9ms/step | 5.8 samples/s
953
+ Step 43800/44000 | Loss: 2.8802 | LR: 2.64e-08 | 3469.1ms/step | 5.8 samples/s
954
+ Step 43810/44000 | Loss: 2.8889 | LR: 2.38e-08 | 3468.7ms/step | 5.8 samples/s
955
+ Step 43820/44000 | Loss: 2.8798 | LR: 2.14e-08 | 3468.0ms/step | 5.8 samples/s
956
+ Step 43830/44000 | Loss: 2.9053 | LR: 1.91e-08 | 3467.8ms/step | 5.8 samples/s
957
+ Step 43840/44000 | Loss: 2.8899 | LR: 1.69e-08 | 3467.0ms/step | 5.8 samples/s
958
+ Step 43850/44000 | Loss: 2.8810 | LR: 1.48e-08 | 3467.1ms/step | 5.8 samples/s
959
+ Step 43860/44000 | Loss: 2.8936 | LR: 1.29e-08 | 3467.1ms/step | 5.8 samples/s
960
+ Step 43870/44000 | Loss: 2.9097 | LR: 1.11e-08 | 3468.4ms/step | 5.8 samples/s
961
+ Step 43880/44000 | Loss: 2.9250 | LR: 9.50e-09 | 3465.5ms/step | 5.8 samples/s
962
+ Step 43890/44000 | Loss: 2.8793 | LR: 7.98e-09 | 3467.8ms/step | 5.8 samples/s
963
+ Step 43900/44000 | Loss: 2.8882 | LR: 6.60e-09 | 3465.0ms/step | 5.8 samples/s
964
+ Step 43910/44000 | Loss: 2.8923 | LR: 5.34e-09 | 3465.8ms/step | 5.8 samples/s
965
+ Step 43920/44000 | Loss: 2.8775 | LR: 4.22e-09 | 3465.8ms/step | 5.8 samples/s
966
+ Step 43930/44000 | Loss: 2.8700 | LR: 3.23e-09 | 3466.1ms/step | 5.8 samples/s
967
+ Step 43940/44000 | Loss: 2.8790 | LR: 2.37e-09 | 3469.9ms/step | 5.8 samples/s
968
+ Step 43950/44000 | Loss: 2.8909 | LR: 1.65e-09 | 3466.3ms/step | 5.8 samples/s
969
+ Step 43960/44000 | Loss: 2.8731 | LR: 1.06e-09 | 3466.3ms/step | 5.8 samples/s
970
+ Step 43970/44000 | Loss: 2.8997 | LR: 5.94e-10 | 3465.6ms/step | 5.8 samples/s
971
+ Step 43980/44000 | Loss: 2.9038 | LR: 2.64e-10 | 3467.7ms/step | 5.8 samples/s
972
+ Step 43990/44000 | Loss: 2.8961 | LR: 6.60e-11 | 3466.1ms/step | 5.8 samples/s
973
+ Step 44000/44000 | Loss: 2.9361 | LR: 0.00e+00 | 3465.9ms/step | 5.8 samples/s
974
+ Validation at step 44000: Loss = 2.9066, Perplexity = 18.30
975
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-44000
976
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031/checkpoint-44000
epfl_final/SparseAtt/wandb/run-20260731_001110-6wmcorvj/files/requirements.txt ADDED
@@ -0,0 +1,142 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ pip==26.0.1
2
+ nvidia-cusparselt-cu13==0.8.0
3
+ mpmath==1.3.0
4
+ cuda-toolkit==13.0.2
5
+ zstandard==0.25.0
6
+ xxhash==3.7.0
7
+ urllib3==2.6.3
8
+ typing_extensions==4.15.0
9
+ triton==3.6.0
10
+ tqdm==4.67.3
11
+ torchinfo==1.8.0
12
+ sympy==1.14.0
13
+ smmap==5.0.3
14
+ six==1.17.0
15
+ shellingham==1.5.4
16
+ setuptools==81.0.0
17
+ safetensors==0.7.0
18
+ regex==2026.4.4
19
+ PyYAML==6.0.3
20
+ Pygments==2.20.0
21
+ pyarrow==24.0.0
22
+ psutil==7.2.2
23
+ protobuf==7.34.1
24
+ propcache==0.4.1
25
+ platformdirs==4.9.6
26
+ packaging==26.2
27
+ nvidia-nvtx==13.0.85
28
+ nvidia-nvshmem-cu13==3.4.5
29
+ nvidia-nvjitlink==13.0.88
30
+ nvidia-nccl-cu13==2.28.9
31
+ nvidia-curand==10.4.0.35
32
+ nvidia-cufile==1.15.1.6
33
+ nvidia-cuda-runtime==13.0.96
34
+ nvidia-cuda-nvrtc==13.0.88
35
+ nvidia-cuda-cupti==13.0.85
36
+ nvidia-cublas==13.1.0.3
37
+ numpy==2.4.4
38
+ networkx==3.6.1
39
+ multidict==6.7.1
40
+ mdurl==0.1.2
41
+ MarkupSafe==3.0.3
42
+ idna==3.13
43
+ hf-xet==1.4.3
44
+ h11==0.16.0
45
+ fsspec==2026.2.0
46
+ frozenlist==1.8.0
47
+ filelock==3.29.0
48
+ dill==0.4.1
49
+ cuda-pathfinder==1.5.4
50
+ click==8.3.3
51
+ charset-normalizer==3.4.7
52
+ certifi==2026.4.22
53
+ attrs==26.1.0
54
+ annotated-types==0.7.0
55
+ annotated-doc==0.0.4
56
+ aiohappyeyeballs==2.6.1
57
+ yarl==1.23.0
58
+ typing-inspection==0.4.2
59
+ sentry-sdk==2.58.0
60
+ requests==2.33.1
61
+ python-dateutil==2.9.0.post0
62
+ pydantic_core==2.46.3
63
+ nvidia-cusparse==12.6.3.3
64
+ nvidia-cufft==12.0.0.61
65
+ nvidia-cudnn-cu13==9.19.0.56
66
+ multiprocess==0.70.19
67
+ markdown-it-py==4.0.0
68
+ Jinja2==3.1.6
69
+ httpcore==1.0.9
70
+ gitdb==4.0.12
71
+ cuda-bindings==13.2.0
72
+ anyio==4.13.0
73
+ aiosignal==1.4.0
74
+ rich==15.0.0
75
+ pydantic==2.13.3
76
+ pandas==3.0.2
77
+ nvidia-cusolver==12.0.4.66
78
+ httpx==0.28.1
79
+ GitPython==3.1.49
80
+ aiohttp==3.13.5
81
+ wandb==0.26.1
82
+ typer==0.25.1
83
+ torch==2.11.0
84
+ huggingface_hub==1.13.0
85
+ tokenizers==0.22.2
86
+ datasets==4.8.5
87
+ accelerate==1.13.0
88
+ transformers==5.7.0
89
+ word2number==1.1
90
+ sqlitedict==2.1.0
91
+ pytz==2026.2
92
+ threadpoolctl==3.6.0
93
+ tcolorpy==0.1.7
94
+ tabulate==0.10.0
95
+ scipy==1.17.1
96
+ portalocker==3.2.0
97
+ pathvalidate==3.3.1
98
+ narwhals==2.22.1
99
+ more-itertools==11.1.0
100
+ lxml==6.1.1
101
+ joblib==1.5.3
102
+ colorama==0.4.6
103
+ chardet==6.0.0.post1
104
+ absl-py==2.4.0
105
+ scikit-learn==1.9.0
106
+ sacrebleu==2.6.0
107
+ nltk==3.9.4
108
+ mbstrdecoder==1.1.5
109
+ typepy==1.3.5
110
+ rouge_score==0.1.2
111
+ DataProperty==1.1.1
112
+ tabledata==1.3.5
113
+ evaluate==0.4.6
114
+ pytablewriter==1.2.1
115
+ lm_eval==0.4.12
116
+ pluggy==1.6.0
117
+ iniconfig==2.3.0
118
+ pytest==9.1.0
119
+ py-spy==0.4.2
120
+ pillow==12.2.0
121
+ tifffile==2026.6.1
122
+ imagecodecs==2026.6.26
123
+ zipp==4.1.0
124
+ importlib_metadata==9.0.0
125
+ derivative==0.6.3
126
+ pysindy==2.1.0
127
+ pyparsing==3.3.2
128
+ kiwisolver==1.5.0
129
+ fonttools==4.63.0
130
+ cycler==0.12.1
131
+ contourpy==1.3.3
132
+ matplotlib==3.11.0
133
+ sortedcontainers==2.4.0
134
+ wcwidth==0.8.2
135
+ sigfig==1.3.19
136
+ importlib_resources==7.1.0
137
+ enum-tools==0.13.0
138
+ cramjam==2.11.0
139
+ python-snappy==0.7.3
140
+ compact-json==1.8.2
141
+ numbers-parser==4.18.5
142
+ entmax==1.3
epfl_final/SparseAtt/wandb/run-20260731_001110-6wmcorvj/files/wandb-metadata.json ADDED
@@ -0,0 +1,65 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "os": "Linux-6.8.0-124-generic-x86_64-with-glibc2.35",
3
+ "python": "CPython 3.13.13",
4
+ "startedAt": "2026-07-31T00:11:10.049356Z",
5
+ "args": [
6
+ "--config",
7
+ "/mloscratch/homes/navasard/sparse/SparseAtt/configs/ladder8k/365m_mixed_to_dense_20pct_seed42_20260730.json"
8
+ ],
9
+ "program": "/mloscratch/homes/navasard/sparse/SparseAtt/train_dense_to_mixed_exact.py",
10
+ "codePath": "train_dense_to_mixed_exact.py",
11
+ "codePathLocal": "train_dense_to_mixed_exact.py",
12
+ "git": {
13
+ "remote": "https://github.com/rubentium/TokenPrune.git",
14
+ "commit": "88298c7a418622530ca5426b22b1e5ebd10ca0cb"
15
+ },
16
+ "email": "ruben.navasardyan@mail.utoronto.ca",
17
+ "root": "/mloscratch/homes/navasard/sparse/SparseAtt",
18
+ "host": "a100-train-4-gpu-2-0-40",
19
+ "executable": "/mloscratch/homes/navasard/conda/bin/python3.13",
20
+ "cpu_count": 64,
21
+ "cpu_count_logical": 128,
22
+ "gpu": "NVIDIA A100-SXM4-80GB",
23
+ "gpu_count": 4,
24
+ "disk": {
25
+ "/": {
26
+ "total": "7679362727936",
27
+ "used": "5257403871232"
28
+ }
29
+ },
30
+ "memory": {
31
+ "total": "1081884934144"
32
+ },
33
+ "gpu_nvidia": [
34
+ {
35
+ "name": "NVIDIA A100-SXM4-80GB",
36
+ "memoryTotal": "85899345920",
37
+ "cudaCores": 6912,
38
+ "architecture": "Ampere",
39
+ "uuid": "GPU-26e055ad-009a-c42a-2ba0-ce6db370fcbe"
40
+ },
41
+ {
42
+ "name": "NVIDIA A100-SXM4-80GB",
43
+ "memoryTotal": "85899345920",
44
+ "cudaCores": 6912,
45
+ "architecture": "Ampere",
46
+ "uuid": "GPU-56ec1834-3584-3932-0125-388b5e88fac2"
47
+ },
48
+ {
49
+ "name": "NVIDIA A100-SXM4-80GB",
50
+ "memoryTotal": "85899345920",
51
+ "cudaCores": 6912,
52
+ "architecture": "Ampere",
53
+ "uuid": "GPU-ce50edb9-4d69-8d20-4433-5217084c7016"
54
+ },
55
+ {
56
+ "name": "NVIDIA A100-SXM4-80GB",
57
+ "memoryTotal": "85899345920",
58
+ "cudaCores": 6912,
59
+ "architecture": "Ampere",
60
+ "uuid": "GPU-2401254a-acb7-5787-0b06-07065349e511"
61
+ }
62
+ ],
63
+ "cudaVersion": "13.0",
64
+ "writerId": "b3v5tviubcpjzo8whpo1u5dqappqlhg6"
65
+ }
epfl_final/SparseAtt/wandb/run-20260731_001110-6wmcorvj/files/wandb-summary.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"_step":44000,"speed/steps_per_sec":0.2885287674009568,"speed/samples_per_sec":5.770575348019135,"eval/uses_dense_attention":1,"_runtime":34016.757462555,"train/global_step":44000,"train/loss":2.9361476850509645,"eval/uses_sparse_attention":0,"_timestamp":1.7854906814325688e+09,"eval/loss":2.906647729873657,"eval/perplexity":18.295364648642465,"_wandb":{"runtime":34016},"eval/dense_interval_end":-1,"eval/dense_interval_active":0,"train/epoch":0.2906439083679023,"speed/ms_per_step":3465.8589124679565,"eval/dense_interval_start":-1,"train/learning_rate":0}
epfl_final/SparseAtt/wandb/run-20260731_001110-6wmcorvj/logs/debug-internal.log ADDED
The diff for this file is too large to render. See raw diff
 
epfl_final/SparseAtt/wandb/run-20260731_001110-6wmcorvj/logs/debug.log ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2026-07-31 00:11:10,066 INFO MainThread:161 [wandb_setup.py:_flush():81] Current SDK version is 0.26.1
2
+ 2026-07-31 00:11:10,066 INFO MainThread:161 [wandb_setup.py:_flush():81] Configure stats pid to 161
3
+ 2026-07-31 00:11:10,066 INFO MainThread:161 [wandb_setup.py:_flush():81] Loading settings from environment variables
4
+ 2026-07-31 00:11:10,066 INFO MainThread:161 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /mloscratch/homes/navasard/sparse/SparseAtt/wandb/run-20260731_001110-6wmcorvj/logs/debug.log
5
+ 2026-07-31 00:11:10,066 INFO MainThread:161 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /mloscratch/homes/navasard/sparse/SparseAtt/wandb/run-20260731_001110-6wmcorvj/logs/debug-internal.log
6
+ 2026-07-31 00:11:10,066 INFO MainThread:161 [wandb_init.py:init():850] calling init triggers
7
+ 2026-07-31 00:11:10,066 INFO MainThread:161 [wandb_init.py:init():855] wandb.init called with sweep_config: {}
8
+ config: {'model_path': '/mloscratch/homes/navasard/qwen3_06b_base', 'model_dtype': 'bfloat16', 'init_from_scratch': True, 'train_path': '/mloscratch/homes/navasard/data/fineweb_large/tokenized', 'val_path': None, 'max_seq_length': 8192, 'num_workers': 8, 'output_dir': '/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed42/run_20260731_001031', 'resume_from_checkpoint': '/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_seed42/run_20260719_234149/checkpoint-35000', 'force_new_run': True, 'num_epochs': 1, 'max_steps': 44000, 'per_device_batch_size': 2, 'gradient_accumulation_steps': 10, 'learning_rate': 0.0005, 'weight_decay': 0.01, 'warmup_steps': 750, 'lr_scheduler': 'cosine', 'max_grad_norm': 1.0, 'save_steps': 250, 'eval_steps': 250, 'eval_batches': 50, 'logging_steps': 10, 'seed': 42, 'verbose': False, 'checkpoint_activations': False, 'strategy': 'ddp', 'mixed_precision': 'bf16', 'wandb_enabled': True, 'wandb_project': 'stochastic_sparsity', 'wandb_name': '365m mixed-to-dense 20pct seed42', 'wandb_tags': ['qwen3', '365m', '8k', 'mixed-to-dense', '20pct', 'seed42', 'multiseed'], 'hidden_size': 768, 'intermediate_size': 3072, 'num_attention_heads': 6, 'num_key_value_heads': 2, 'num_hidden_layers': 28, 'head_dim': None, 'attention_type': 'sparse_stochastic', 'sparse_block_size': 128, 'sparse_num_random_blocks': 1, 'sparse_mask_pool_size': 1, 'sparse_val_dense_compare': False, 'sparse_layer_indices': None, 'sparse_mask_cache_dir': None, 'sparse_mask_seed': 0, 'dense_mode': True, 'dense_step_intervals': None, '_wandb': {}}
9
+ 2026-07-31 00:11:10,066 INFO MainThread:161 [wandb_init.py:init():898] starting backend
10
+ 2026-07-31 00:11:10,618 INFO MainThread:161 [wandb_init.py:init():913] sending inform_init request
11
+ 2026-07-31 00:11:11,153 INFO MainThread:161 [wandb_init.py:init():918] backend started and connected
12
+ 2026-07-31 00:11:11,154 INFO MainThread:161 [wandb_init.py:init():988] updated telemetry
13
+ 2026-07-31 00:11:11,188 INFO MainThread:161 [wandb_init.py:init():1011] communicating run to backend with 90.0 second timeout
14
+ 2026-07-31 00:11:11,618 INFO MainThread:161 [wandb_init.py:init():1056] starting run threads in backend
15
+ 2026-07-31 00:11:11,870 INFO MainThread:161 [wandb_run.py:_console_start():2554] atexit reg
16
+ 2026-07-31 00:11:11,870 INFO MainThread:161 [wandb_run.py:_redirect():2403] redirect: wrap_raw
17
+ 2026-07-31 00:11:11,870 INFO MainThread:161 [wandb_run.py:_redirect():2472] Wrapping output streams.
18
+ 2026-07-31 00:11:11,870 INFO MainThread:161 [wandb_run.py:_redirect():2495] Redirects installed.
19
+ 2026-07-31 00:11:11,876 INFO MainThread:161 [wandb_init.py:init():1094] run started, returning control to user process
20
+ 2026-07-31 09:38:08,374 INFO MainThread:161 [wandb_run.py:_finish():2321] finishing run ruben-navasardyan4869-epfl/stochastic_sparsity/6wmcorvj
21
+ 2026-07-31 09:38:08,375 INFO MainThread:161 [wandb_run.py:_atexit_cleanup():2520] got exitcode: 0
22
+ 2026-07-31 09:38:08,376 INFO MainThread:161 [wandb_run.py:_restore():2502] restore
23
+ 2026-07-31 09:38:08,376 INFO MainThread:161 [wandb_run.py:_restore():2508] restore done
24
+ 2026-07-31 09:38:10,286 INFO MainThread:161 [wandb_run.py:_footer_sync_info():3889] logging synced files
epfl_final/SparseAtt/wandb/run-20260731_001110-6wmcorvj/run-6wmcorvj.wandb ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c7fe93d7670fe8043b670a51fb09914aef1fe4f586d12e58fa40107ac1b1fe01
3
+ size 6102021
epfl_final/SparseAtt/wandb/run-20260731_001110-gl67rqc0/files/config.yaml ADDED
@@ -0,0 +1,187 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ _wandb:
2
+ value:
3
+ cli_version: 0.26.1
4
+ e:
5
+ rekjwwv2xzeva8fcuzeebmynnd9sd1h3:
6
+ args:
7
+ - --config
8
+ - /mloscratch/homes/navasard/sparse/SparseAtt/configs/ladder8k/365m_mixed_to_dense_20pct_seed43_20260730.json
9
+ codePath: train_dense_to_mixed_exact.py
10
+ codePathLocal: train_dense_to_mixed_exact.py
11
+ cpu_count: 64
12
+ cpu_count_logical: 128
13
+ cudaVersion: "13.0"
14
+ disk:
15
+ /:
16
+ total: "7679362727936"
17
+ used: "5257403871232"
18
+ email: ruben.navasardyan@mail.utoronto.ca
19
+ executable: /mloscratch/homes/navasard/conda/bin/python3.13
20
+ git:
21
+ commit: 88298c7a418622530ca5426b22b1e5ebd10ca0cb
22
+ remote: https://github.com/rubentium/TokenPrune.git
23
+ gpu: NVIDIA A100-SXM4-80GB
24
+ gpu_count: 4
25
+ gpu_nvidia:
26
+ - architecture: Ampere
27
+ cudaCores: 6912
28
+ memoryTotal: "85899345920"
29
+ name: NVIDIA A100-SXM4-80GB
30
+ uuid: GPU-26e055ad-009a-c42a-2ba0-ce6db370fcbe
31
+ - architecture: Ampere
32
+ cudaCores: 6912
33
+ memoryTotal: "85899345920"
34
+ name: NVIDIA A100-SXM4-80GB
35
+ uuid: GPU-56ec1834-3584-3932-0125-388b5e88fac2
36
+ - architecture: Ampere
37
+ cudaCores: 6912
38
+ memoryTotal: "85899345920"
39
+ name: NVIDIA A100-SXM4-80GB
40
+ uuid: GPU-ce50edb9-4d69-8d20-4433-5217084c7016
41
+ - architecture: Ampere
42
+ cudaCores: 6912
43
+ memoryTotal: "85899345920"
44
+ name: NVIDIA A100-SXM4-80GB
45
+ uuid: GPU-2401254a-acb7-5787-0b06-07065349e511
46
+ host: a100-train-4-gpu-2-0-40
47
+ memory:
48
+ total: "1081884934144"
49
+ os: Linux-6.8.0-124-generic-x86_64-with-glibc2.35
50
+ program: /mloscratch/homes/navasard/sparse/SparseAtt/train_dense_to_mixed_exact.py
51
+ python: CPython 3.13.13
52
+ root: /mloscratch/homes/navasard/sparse/SparseAtt
53
+ startedAt: "2026-07-31T00:11:10.058418Z"
54
+ writerId: rekjwwv2xzeva8fcuzeebmynnd9sd1h3
55
+ m: []
56
+ python_version: 3.13.13
57
+ t:
58
+ "1":
59
+ - 1
60
+ - 5
61
+ - 11
62
+ - 49
63
+ - 51
64
+ - 53
65
+ - 71
66
+ "2":
67
+ - 1
68
+ - 5
69
+ - 11
70
+ - 49
71
+ - 51
72
+ - 53
73
+ - 71
74
+ "3":
75
+ - 2
76
+ - 13
77
+ - 15
78
+ - 16
79
+ - 61
80
+ "4": 3.13.13
81
+ "5": 0.26.1
82
+ "6": 5.7.0
83
+ "12": 0.26.1
84
+ "13": linux-x86_64
85
+ attention_type:
86
+ value: sparse_stochastic
87
+ checkpoint_activations:
88
+ value: false
89
+ dense_mode:
90
+ value: true
91
+ dense_step_intervals:
92
+ value: null
93
+ eval_batches:
94
+ value: 50
95
+ eval_steps:
96
+ value: 250
97
+ force_new_run:
98
+ value: true
99
+ gradient_accumulation_steps:
100
+ value: 10
101
+ head_dim:
102
+ value: null
103
+ hidden_size:
104
+ value: 768
105
+ init_from_scratch:
106
+ value: true
107
+ intermediate_size:
108
+ value: 3072
109
+ learning_rate:
110
+ value: 0.0005
111
+ logging_steps:
112
+ value: 10
113
+ lr_scheduler:
114
+ value: cosine
115
+ max_grad_norm:
116
+ value: 1
117
+ max_seq_length:
118
+ value: 8192
119
+ max_steps:
120
+ value: 44000
121
+ mixed_precision:
122
+ value: bf16
123
+ model_dtype:
124
+ value: bfloat16
125
+ model_path:
126
+ value: /mloscratch/homes/navasard/qwen3_06b_base
127
+ num_attention_heads:
128
+ value: 6
129
+ num_epochs:
130
+ value: 1
131
+ num_hidden_layers:
132
+ value: 28
133
+ num_key_value_heads:
134
+ value: 2
135
+ num_workers:
136
+ value: 8
137
+ output_dir:
138
+ value: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031
139
+ per_device_batch_size:
140
+ value: 2
141
+ resume_from_checkpoint:
142
+ value: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_seed43/run_20260719_234149/checkpoint-35000
143
+ save_steps:
144
+ value: 250
145
+ seed:
146
+ value: 43
147
+ sparse_block_size:
148
+ value: 128
149
+ sparse_layer_indices:
150
+ value: null
151
+ sparse_mask_cache_dir:
152
+ value: null
153
+ sparse_mask_pool_size:
154
+ value: 1
155
+ sparse_mask_seed:
156
+ value: 0
157
+ sparse_num_random_blocks:
158
+ value: 1
159
+ sparse_val_dense_compare:
160
+ value: false
161
+ strategy:
162
+ value: ddp
163
+ train_path:
164
+ value: /mloscratch/homes/navasard/data/fineweb_large/tokenized
165
+ val_path:
166
+ value: null
167
+ verbose:
168
+ value: false
169
+ wandb_enabled:
170
+ value: true
171
+ wandb_name:
172
+ value: 365m mixed-to-dense 20pct seed43
173
+ wandb_project:
174
+ value: stochastic_sparsity
175
+ wandb_tags:
176
+ value:
177
+ - qwen3
178
+ - 365m
179
+ - 8k
180
+ - mixed-to-dense
181
+ - 20pct
182
+ - seed43
183
+ - multiseed
184
+ warmup_steps:
185
+ value: 750
186
+ weight_decay:
187
+ value: 0.01
epfl_final/SparseAtt/wandb/run-20260731_001110-gl67rqc0/files/output.log ADDED
@@ -0,0 +1,976 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Starting training...
2
+ Validation at step 35000: Loss = 2.8932, Perplexity = 18.05
3
+ Switching attention mode for step 35001: dense (dense step interval)
4
+ Step 35010/44000 | Loss: 2.9013 | LR: 5.14e-05 | 67178.6ms/step | 0.3 samples/s
5
+ Step 35020/44000 | Loss: 2.9108 | LR: 5.13e-05 | 3460.9ms/step | 5.8 samples/s
6
+ Step 35030/44000 | Loss: 2.8909 | LR: 5.12e-05 | 3468.6ms/step | 5.8 samples/s
7
+ Step 35040/44000 | Loss: 2.8963 | LR: 5.11e-05 | 3469.3ms/step | 5.8 samples/s
8
+ Step 35050/44000 | Loss: 2.8939 | LR: 5.10e-05 | 3473.5ms/step | 5.8 samples/s
9
+ Step 35060/44000 | Loss: 2.9007 | LR: 5.09e-05 | 3475.9ms/step | 5.8 samples/s
10
+ Step 35070/44000 | Loss: 2.8963 | LR: 5.08e-05 | 3480.5ms/step | 5.7 samples/s
11
+ Step 35080/44000 | Loss: 2.8899 | LR: 5.07e-05 | 3482.3ms/step | 5.7 samples/s
12
+ Step 35090/44000 | Loss: 2.8826 | LR: 5.06e-05 | 3482.9ms/step | 5.7 samples/s
13
+ Step 35100/44000 | Loss: 2.8724 | LR: 5.04e-05 | 3484.0ms/step | 5.7 samples/s
14
+ Step 35110/44000 | Loss: 2.9049 | LR: 5.03e-05 | 3484.7ms/step | 5.7 samples/s
15
+ Step 35120/44000 | Loss: 2.8944 | LR: 5.02e-05 | 3483.4ms/step | 5.7 samples/s
16
+ Step 35130/44000 | Loss: 2.8919 | LR: 5.01e-05 | 3482.9ms/step | 5.7 samples/s
17
+ Step 35140/44000 | Loss: 2.8907 | LR: 5.00e-05 | 3484.7ms/step | 5.7 samples/s
18
+ Step 35150/44000 | Loss: 2.9003 | LR: 4.99e-05 | 3486.5ms/step | 5.7 samples/s
19
+ Step 35160/44000 | Loss: 2.9053 | LR: 4.98e-05 | 3483.8ms/step | 5.7 samples/s
20
+ Step 35170/44000 | Loss: 2.8917 | LR: 4.97e-05 | 3483.6ms/step | 5.7 samples/s
21
+ Step 35180/44000 | Loss: 2.8898 | LR: 4.96e-05 | 3484.9ms/step | 5.7 samples/s
22
+ Step 35190/44000 | Loss: 2.9005 | LR: 4.95e-05 | 3483.9ms/step | 5.7 samples/s
23
+ Step 35200/44000 | Loss: 2.8722 | LR: 4.94e-05 | 3482.9ms/step | 5.7 samples/s
24
+ Step 35210/44000 | Loss: 2.8778 | LR: 4.93e-05 | 3486.5ms/step | 5.7 samples/s
25
+ Step 35220/44000 | Loss: 2.8638 | LR: 4.91e-05 | 3488.7ms/step | 5.7 samples/s
26
+ Step 35230/44000 | Loss: 2.8788 | LR: 4.90e-05 | 3484.0ms/step | 5.7 samples/s
27
+ Step 35240/44000 | Loss: 2.9019 | LR: 4.89e-05 | 3485.5ms/step | 5.7 samples/s
28
+ Step 35250/44000 | Loss: 2.8757 | LR: 4.88e-05 | 3486.9ms/step | 5.7 samples/s
29
+ Validation at step 35250: Loss = 2.8864, Perplexity = 17.93
30
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-35250
31
+ Step 35260/44000 | Loss: 2.8765 | LR: 4.87e-05 | 9628.9ms/step | 2.1 samples/s
32
+ Step 35270/44000 | Loss: 2.9117 | LR: 4.86e-05 | 3486.2ms/step | 5.7 samples/s
33
+ Step 35280/44000 | Loss: 2.9175 | LR: 4.85e-05 | 3486.6ms/step | 5.7 samples/s
34
+ Step 35290/44000 | Loss: 2.8912 | LR: 4.84e-05 | 3487.9ms/step | 5.7 samples/s
35
+ Step 35300/44000 | Loss: 2.8898 | LR: 4.83e-05 | 3485.7ms/step | 5.7 samples/s
36
+ Step 35310/44000 | Loss: 2.8876 | LR: 4.82e-05 | 3488.4ms/step | 5.7 samples/s
37
+ Step 35320/44000 | Loss: 2.8626 | LR: 4.81e-05 | 3490.3ms/step | 5.7 samples/s
38
+ Step 35330/44000 | Loss: 2.8807 | LR: 4.80e-05 | 3492.1ms/step | 5.7 samples/s
39
+ Step 35340/44000 | Loss: 2.8733 | LR: 4.79e-05 | 3488.8ms/step | 5.7 samples/s
40
+ Step 35350/44000 | Loss: 2.9087 | LR: 4.77e-05 | 3487.6ms/step | 5.7 samples/s
41
+ Step 35360/44000 | Loss: 2.8880 | LR: 4.76e-05 | 3485.8ms/step | 5.7 samples/s
42
+ Step 35370/44000 | Loss: 2.8693 | LR: 4.75e-05 | 3489.1ms/step | 5.7 samples/s
43
+ Step 35380/44000 | Loss: 2.8708 | LR: 4.74e-05 | 3492.7ms/step | 5.7 samples/s
44
+ Step 35390/44000 | Loss: 2.9054 | LR: 4.73e-05 | 3491.3ms/step | 5.7 samples/s
45
+ Step 35400/44000 | Loss: 2.8761 | LR: 4.72e-05 | 3489.3ms/step | 5.7 samples/s
46
+ Step 35410/44000 | Loss: 2.9049 | LR: 4.71e-05 | 3486.2ms/step | 5.7 samples/s
47
+ Step 35420/44000 | Loss: 2.9231 | LR: 4.70e-05 | 3487.8ms/step | 5.7 samples/s
48
+ Step 35430/44000 | Loss: 2.9060 | LR: 4.69e-05 | 3484.7ms/step | 5.7 samples/s
49
+ Step 35440/44000 | Loss: 2.9052 | LR: 4.68e-05 | 3485.5ms/step | 5.7 samples/s
50
+ Step 35450/44000 | Loss: 2.9023 | LR: 4.67e-05 | 3486.2ms/step | 5.7 samples/s
51
+ Step 35460/44000 | Loss: 2.9113 | LR: 4.66e-05 | 3486.8ms/step | 5.7 samples/s
52
+ Step 35470/44000 | Loss: 2.8927 | LR: 4.65e-05 | 3487.0ms/step | 5.7 samples/s
53
+ Step 35480/44000 | Loss: 2.9083 | LR: 4.64e-05 | 3488.5ms/step | 5.7 samples/s
54
+ Step 35490/44000 | Loss: 2.8914 | LR: 4.63e-05 | 3487.5ms/step | 5.7 samples/s
55
+ Step 35500/44000 | Loss: 2.9118 | LR: 4.62e-05 | 3485.8ms/step | 5.7 samples/s
56
+ Validation at step 35500: Loss = 2.8857, Perplexity = 17.92
57
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-35500
58
+ Step 35510/44000 | Loss: 2.9012 | LR: 4.61e-05 | 9634.0ms/step | 2.1 samples/s
59
+ Step 35520/44000 | Loss: 2.8928 | LR: 4.59e-05 | 3484.1ms/step | 5.7 samples/s
60
+ Step 35530/44000 | Loss: 2.9052 | LR: 4.58e-05 | 3485.8ms/step | 5.7 samples/s
61
+ Step 35540/44000 | Loss: 2.9192 | LR: 4.57e-05 | 3484.7ms/step | 5.7 samples/s
62
+ Step 35550/44000 | Loss: 2.9054 | LR: 4.56e-05 | 3484.3ms/step | 5.7 samples/s
63
+ Step 35560/44000 | Loss: 2.8999 | LR: 4.55e-05 | 3484.1ms/step | 5.7 samples/s
64
+ Step 35570/44000 | Loss: 2.9330 | LR: 4.54e-05 | 3485.1ms/step | 5.7 samples/s
65
+ Step 35580/44000 | Loss: 2.8773 | LR: 4.53e-05 | 3485.9ms/step | 5.7 samples/s
66
+ Step 35590/44000 | Loss: 2.9155 | LR: 4.52e-05 | 3491.5ms/step | 5.7 samples/s
67
+ Step 35600/44000 | Loss: 2.9218 | LR: 4.51e-05 | 3492.8ms/step | 5.7 samples/s
68
+ Step 35610/44000 | Loss: 2.8710 | LR: 4.50e-05 | 3491.6ms/step | 5.7 samples/s
69
+ Step 35620/44000 | Loss: 2.9165 | LR: 4.49e-05 | 3487.7ms/step | 5.7 samples/s
70
+ Step 35630/44000 | Loss: 2.8888 | LR: 4.48e-05 | 3485.2ms/step | 5.7 samples/s
71
+ Step 35640/44000 | Loss: 2.8472 | LR: 4.47e-05 | 3488.6ms/step | 5.7 samples/s
72
+ Step 35650/44000 | Loss: 2.8934 | LR: 4.46e-05 | 3490.0ms/step | 5.7 samples/s
73
+ Step 35660/44000 | Loss: 2.9230 | LR: 4.45e-05 | 3486.8ms/step | 5.7 samples/s
74
+ Step 35670/44000 | Loss: 2.9001 | LR: 4.44e-05 | 3485.9ms/step | 5.7 samples/s
75
+ Step 35680/44000 | Loss: 2.8813 | LR: 4.43e-05 | 3491.7ms/step | 5.7 samples/s
76
+ Step 35690/44000 | Loss: 2.8930 | LR: 4.42e-05 | 3489.1ms/step | 5.7 samples/s
77
+ Step 35700/44000 | Loss: 2.8763 | LR: 4.41e-05 | 3488.8ms/step | 5.7 samples/s
78
+ Step 35710/44000 | Loss: 2.8975 | LR: 4.40e-05 | 3487.1ms/step | 5.7 samples/s
79
+ Step 35720/44000 | Loss: 2.8685 | LR: 4.39e-05 | 3485.8ms/step | 5.7 samples/s
80
+ Step 35730/44000 | Loss: 2.9173 | LR: 4.38e-05 | 3487.6ms/step | 5.7 samples/s
81
+ Step 35740/44000 | Loss: 2.8608 | LR: 4.37e-05 | 3496.8ms/step | 5.7 samples/s
82
+ Step 35750/44000 | Loss: 2.8824 | LR: 4.36e-05 | 3488.6ms/step | 5.7 samples/s
83
+ Validation at step 35750: Loss = 2.8852, Perplexity = 17.91
84
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-35750
85
+ Step 35760/44000 | Loss: 2.9104 | LR: 4.35e-05 | 9634.4ms/step | 2.1 samples/s
86
+ Step 35770/44000 | Loss: 2.8994 | LR: 4.34e-05 | 3485.8ms/step | 5.7 samples/s
87
+ Step 35780/44000 | Loss: 2.8818 | LR: 4.33e-05 | 3484.3ms/step | 5.7 samples/s
88
+ Step 35790/44000 | Loss: 2.8931 | LR: 4.32e-05 | 3486.1ms/step | 5.7 samples/s
89
+ Step 35800/44000 | Loss: 2.9208 | LR: 4.31e-05 | 3487.7ms/step | 5.7 samples/s
90
+ Step 35810/44000 | Loss: 2.9050 | LR: 4.29e-05 | 3493.9ms/step | 5.7 samples/s
91
+ Step 35820/44000 | Loss: 2.8919 | LR: 4.28e-05 | 3496.8ms/step | 5.7 samples/s
92
+ Step 35830/44000 | Loss: 2.8923 | LR: 4.27e-05 | 3493.6ms/step | 5.7 samples/s
93
+ Step 35840/44000 | Loss: 2.9028 | LR: 4.26e-05 | 3485.9ms/step | 5.7 samples/s
94
+ Step 35850/44000 | Loss: 2.9142 | LR: 4.25e-05 | 3486.8ms/step | 5.7 samples/s
95
+ Step 35860/44000 | Loss: 2.9011 | LR: 4.24e-05 | 3486.6ms/step | 5.7 samples/s
96
+ Step 35870/44000 | Loss: 2.8807 | LR: 4.23e-05 | 3488.8ms/step | 5.7 samples/s
97
+ Step 35880/44000 | Loss: 2.8989 | LR: 4.22e-05 | 3493.9ms/step | 5.7 samples/s
98
+ Step 35890/44000 | Loss: 2.8839 | LR: 4.21e-05 | 3491.3ms/step | 5.7 samples/s
99
+ Step 35900/44000 | Loss: 2.8540 | LR: 4.20e-05 | 3487.9ms/step | 5.7 samples/s
100
+ Step 35910/44000 | Loss: 2.8877 | LR: 4.19e-05 | 3488.9ms/step | 5.7 samples/s
101
+ Step 35920/44000 | Loss: 2.8703 | LR: 4.18e-05 | 3487.6ms/step | 5.7 samples/s
102
+ Step 35930/44000 | Loss: 2.8992 | LR: 4.17e-05 | 3488.4ms/step | 5.7 samples/s
103
+ Step 35940/44000 | Loss: 2.9153 | LR: 4.16e-05 | 3486.9ms/step | 5.7 samples/s
104
+ Step 35950/44000 | Loss: 2.9067 | LR: 4.15e-05 | 3493.1ms/step | 5.7 samples/s
105
+ Step 35960/44000 | Loss: 2.9078 | LR: 4.14e-05 | 3495.2ms/step | 5.7 samples/s
106
+ Step 35970/44000 | Loss: 2.8967 | LR: 4.13e-05 | 3490.9ms/step | 5.7 samples/s
107
+ Step 35980/44000 | Loss: 2.8566 | LR: 4.12e-05 | 3489.4ms/step | 5.7 samples/s
108
+ Step 35990/44000 | Loss: 2.8878 | LR: 4.11e-05 | 3491.5ms/step | 5.7 samples/s
109
+ Step 36000/44000 | Loss: 2.8927 | LR: 4.10e-05 | 3488.5ms/step | 5.7 samples/s
110
+ Validation at step 36000: Loss = 2.8849, Perplexity = 17.90
111
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-36000
112
+ Step 36010/44000 | Loss: 2.8949 | LR: 4.09e-05 | 9645.5ms/step | 2.1 samples/s
113
+ Step 36020/44000 | Loss: 2.8987 | LR: 4.08e-05 | 3487.8ms/step | 5.7 samples/s
114
+ Step 36030/44000 | Loss: 2.8931 | LR: 4.07e-05 | 3487.9ms/step | 5.7 samples/s
115
+ Step 36040/44000 | Loss: 2.8939 | LR: 4.06e-05 | 3488.0ms/step | 5.7 samples/s
116
+ Step 36050/44000 | Loss: 2.8911 | LR: 4.05e-05 | 3493.5ms/step | 5.7 samples/s
117
+ Step 36060/44000 | Loss: 2.8919 | LR: 4.04e-05 | 3491.7ms/step | 5.7 samples/s
118
+ Step 36070/44000 | Loss: 2.8587 | LR: 4.03e-05 | 3489.5ms/step | 5.7 samples/s
119
+ Step 36080/44000 | Loss: 2.9015 | LR: 4.02e-05 | 3501.7ms/step | 5.7 samples/s
120
+ Step 36090/44000 | Loss: 2.8840 | LR: 4.01e-05 | 3496.6ms/step | 5.7 samples/s
121
+ Step 36100/44000 | Loss: 2.8870 | LR: 4.00e-05 | 3494.0ms/step | 5.7 samples/s
122
+ Step 36110/44000 | Loss: 2.9194 | LR: 3.99e-05 | 3489.0ms/step | 5.7 samples/s
123
+ Step 36120/44000 | Loss: 2.8622 | LR: 3.98e-05 | 3488.7ms/step | 5.7 samples/s
124
+ Step 36130/44000 | Loss: 2.8758 | LR: 3.97e-05 | 3489.2ms/step | 5.7 samples/s
125
+ Step 36140/44000 | Loss: 2.8989 | LR: 3.97e-05 | 3487.5ms/step | 5.7 samples/s
126
+ Step 36150/44000 | Loss: 2.8755 | LR: 3.96e-05 | 3492.1ms/step | 5.7 samples/s
127
+ Step 36160/44000 | Loss: 2.9070 | LR: 3.95e-05 | 3495.0ms/step | 5.7 samples/s
128
+ Step 36170/44000 | Loss: 2.9270 | LR: 3.94e-05 | 3489.3ms/step | 5.7 samples/s
129
+ Step 36180/44000 | Loss: 2.9147 | LR: 3.93e-05 | 3490.7ms/step | 5.7 samples/s
130
+ Step 36190/44000 | Loss: 2.8826 | LR: 3.92e-05 | 3489.3ms/step | 5.7 samples/s
131
+ Step 36200/44000 | Loss: 2.9008 | LR: 3.91e-05 | 3489.5ms/step | 5.7 samples/s
132
+ Step 36210/44000 | Loss: 2.9052 | LR: 3.90e-05 | 3489.0ms/step | 5.7 samples/s
133
+ Step 36220/44000 | Loss: 2.9048 | LR: 3.89e-05 | 3492.7ms/step | 5.7 samples/s
134
+ Step 36230/44000 | Loss: 2.8663 | LR: 3.88e-05 | 3488.5ms/step | 5.7 samples/s
135
+ Step 36240/44000 | Loss: 2.8949 | LR: 3.87e-05 | 3491.4ms/step | 5.7 samples/s
136
+ Step 36250/44000 | Loss: 2.9102 | LR: 3.86e-05 | 3489.6ms/step | 5.7 samples/s
137
+ Validation at step 36250: Loss = 2.8846, Perplexity = 17.90
138
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-36250
139
+ Step 36260/44000 | Loss: 2.8586 | LR: 3.85e-05 | 9628.5ms/step | 2.1 samples/s
140
+ Step 36270/44000 | Loss: 2.8607 | LR: 3.84e-05 | 3488.4ms/step | 5.7 samples/s
141
+ Step 36280/44000 | Loss: 2.8736 | LR: 3.83e-05 | 3489.9ms/step | 5.7 samples/s
142
+ Step 36290/44000 | Loss: 2.8883 | LR: 3.82e-05 | 3491.8ms/step | 5.7 samples/s
143
+ Step 36300/44000 | Loss: 2.8887 | LR: 3.81e-05 | 3491.3ms/step | 5.7 samples/s
144
+ Step 36310/44000 | Loss: 2.8686 | LR: 3.80e-05 | 3490.7ms/step | 5.7 samples/s
145
+ Step 36320/44000 | Loss: 2.8691 | LR: 3.79e-05 | 3491.5ms/step | 5.7 samples/s
146
+ Step 36330/44000 | Loss: 2.8563 | LR: 3.78e-05 | 3490.8ms/step | 5.7 samples/s
147
+ Step 36340/44000 | Loss: 2.8871 | LR: 3.77e-05 | 3498.4ms/step | 5.7 samples/s
148
+ Step 36350/44000 | Loss: 2.9113 | LR: 3.76e-05 | 3495.3ms/step | 5.7 samples/s
149
+ Step 36360/44000 | Loss: 2.9044 | LR: 3.75e-05 | 3489.7ms/step | 5.7 samples/s
150
+ Step 36370/44000 | Loss: 2.9087 | LR: 3.74e-05 | 3497.5ms/step | 5.7 samples/s
151
+ Step 36380/44000 | Loss: 2.8838 | LR: 3.73e-05 | 3491.4ms/step | 5.7 samples/s
152
+ Step 36390/44000 | Loss: 2.8965 | LR: 3.72e-05 | 3490.0ms/step | 5.7 samples/s
153
+ Step 36400/44000 | Loss: 2.9112 | LR: 3.71e-05 | 3489.3ms/step | 5.7 samples/s
154
+ Step 36410/44000 | Loss: 2.9014 | LR: 3.70e-05 | 3488.7ms/step | 5.7 samples/s
155
+ Step 36420/44000 | Loss: 2.9129 | LR: 3.69e-05 | 3489.2ms/step | 5.7 samples/s
156
+ Step 36430/44000 | Loss: 2.9046 | LR: 3.69e-05 | 3490.6ms/step | 5.7 samples/s
157
+ Step 36440/44000 | Loss: 2.8965 | LR: 3.68e-05 | 3490.1ms/step | 5.7 samples/s
158
+ Step 36450/44000 | Loss: 2.8672 | LR: 3.67e-05 | 3489.3ms/step | 5.7 samples/s
159
+ Step 36460/44000 | Loss: 2.8629 | LR: 3.66e-05 | 3488.4ms/step | 5.7 samples/s
160
+ Step 36470/44000 | Loss: 2.8910 | LR: 3.65e-05 | 3488.0ms/step | 5.7 samples/s
161
+ Step 36480/44000 | Loss: 2.9069 | LR: 3.64e-05 | 3489.8ms/step | 5.7 samples/s
162
+ Step 36490/44000 | Loss: 2.8862 | LR: 3.63e-05 | 3488.1ms/step | 5.7 samples/s
163
+ Step 36500/44000 | Loss: 2.8991 | LR: 3.62e-05 | 3490.1ms/step | 5.7 samples/s
164
+ Validation at step 36500: Loss = 2.8844, Perplexity = 17.89
165
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-36500
166
+ Step 36510/44000 | Loss: 2.9091 | LR: 3.61e-05 | 9646.2ms/step | 2.1 samples/s
167
+ Step 36520/44000 | Loss: 2.9444 | LR: 3.60e-05 | 3482.5ms/step | 5.7 samples/s
168
+ Step 36530/44000 | Loss: 2.8657 | LR: 3.59e-05 | 3484.6ms/step | 5.7 samples/s
169
+ Step 36540/44000 | Loss: 2.8703 | LR: 3.58e-05 | 3486.5ms/step | 5.7 samples/s
170
+ Step 36550/44000 | Loss: 2.9079 | LR: 3.57e-05 | 3487.1ms/step | 5.7 samples/s
171
+ Step 36560/44000 | Loss: 2.8649 | LR: 3.56e-05 | 3490.5ms/step | 5.7 samples/s
172
+ Step 36570/44000 | Loss: 2.9056 | LR: 3.55e-05 | 3487.8ms/step | 5.7 samples/s
173
+ Step 36580/44000 | Loss: 2.8941 | LR: 3.54e-05 | 3490.5ms/step | 5.7 samples/s
174
+ Step 36590/44000 | Loss: 2.8961 | LR: 3.53e-05 | 3494.5ms/step | 5.7 samples/s
175
+ Step 36600/44000 | Loss: 2.8917 | LR: 3.53e-05 | 3493.8ms/step | 5.7 samples/s
176
+ Step 36610/44000 | Loss: 2.9217 | LR: 3.52e-05 | 3494.4ms/step | 5.7 samples/s
177
+ Step 36620/44000 | Loss: 2.9244 | LR: 3.51e-05 | 3491.2ms/step | 5.7 samples/s
178
+ Step 36630/44000 | Loss: 2.8575 | LR: 3.50e-05 | 3492.0ms/step | 5.7 samples/s
179
+ Step 36640/44000 | Loss: 2.9048 | LR: 3.49e-05 | 3488.6ms/step | 5.7 samples/s
180
+ Step 36650/44000 | Loss: 2.8867 | LR: 3.48e-05 | 3490.9ms/step | 5.7 samples/s
181
+ Step 36660/44000 | Loss: 2.8818 | LR: 3.47e-05 | 3491.7ms/step | 5.7 samples/s
182
+ Step 36670/44000 | Loss: 2.8993 | LR: 3.46e-05 | 3488.3ms/step | 5.7 samples/s
183
+ Step 36680/44000 | Loss: 2.8796 | LR: 3.45e-05 | 3490.0ms/step | 5.7 samples/s
184
+ Step 36690/44000 | Loss: 2.8959 | LR: 3.44e-05 | 3497.8ms/step | 5.7 samples/s
185
+ Step 36700/44000 | Loss: 2.9272 | LR: 3.43e-05 | 3500.8ms/step | 5.7 samples/s
186
+ Step 36710/44000 | Loss: 2.9068 | LR: 3.42e-05 | 3499.3ms/step | 5.7 samples/s
187
+ Step 36720/44000 | Loss: 2.8911 | LR: 3.41e-05 | 3493.7ms/step | 5.7 samples/s
188
+ Step 36730/44000 | Loss: 2.8716 | LR: 3.41e-05 | 3490.1ms/step | 5.7 samples/s
189
+ Step 36740/44000 | Loss: 2.9181 | LR: 3.40e-05 | 3487.5ms/step | 5.7 samples/s
190
+ Step 36750/44000 | Loss: 2.8957 | LR: 3.39e-05 | 3490.1ms/step | 5.7 samples/s
191
+ Validation at step 36750: Loss = 2.8842, Perplexity = 17.89
192
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-36750
193
+ Step 36760/44000 | Loss: 2.9250 | LR: 3.38e-05 | 9646.4ms/step | 2.1 samples/s
194
+ Step 36770/44000 | Loss: 2.9005 | LR: 3.37e-05 | 3500.7ms/step | 5.7 samples/s
195
+ Step 36780/44000 | Loss: 2.9109 | LR: 3.36e-05 | 3501.8ms/step | 5.7 samples/s
196
+ Step 36790/44000 | Loss: 2.8737 | LR: 3.35e-05 | 3499.5ms/step | 5.7 samples/s
197
+ Step 36800/44000 | Loss: 2.8713 | LR: 3.34e-05 | 3498.0ms/step | 5.7 samples/s
198
+ Step 36810/44000 | Loss: 2.8774 | LR: 3.33e-05 | 3490.4ms/step | 5.7 samples/s
199
+ Step 36820/44000 | Loss: 2.8975 | LR: 3.32e-05 | 3492.4ms/step | 5.7 samples/s
200
+ Step 36830/44000 | Loss: 2.8957 | LR: 3.31e-05 | 3500.0ms/step | 5.7 samples/s
201
+ Step 36840/44000 | Loss: 2.8679 | LR: 3.31e-05 | 3500.6ms/step | 5.7 samples/s
202
+ Step 36850/44000 | Loss: 2.8967 | LR: 3.30e-05 | 3494.6ms/step | 5.7 samples/s
203
+ Step 36860/44000 | Loss: 2.9013 | LR: 3.29e-05 | 3489.6ms/step | 5.7 samples/s
204
+ Step 36870/44000 | Loss: 2.8769 | LR: 3.28e-05 | 3496.1ms/step | 5.7 samples/s
205
+ Step 36880/44000 | Loss: 2.8666 | LR: 3.27e-05 | 3491.0ms/step | 5.7 samples/s
206
+ Step 36890/44000 | Loss: 2.9199 | LR: 3.26e-05 | 3488.8ms/step | 5.7 samples/s
207
+ Step 36900/44000 | Loss: 2.8856 | LR: 3.25e-05 | 3489.9ms/step | 5.7 samples/s
208
+ Step 36910/44000 | Loss: 2.8731 | LR: 3.24e-05 | 3496.8ms/step | 5.7 samples/s
209
+ Step 36920/44000 | Loss: 2.8727 | LR: 3.23e-05 | 3490.9ms/step | 5.7 samples/s
210
+ Step 36930/44000 | Loss: 2.9057 | LR: 3.22e-05 | 3487.9ms/step | 5.7 samples/s
211
+ Step 36940/44000 | Loss: 2.8749 | LR: 3.22e-05 | 3490.6ms/step | 5.7 samples/s
212
+ Step 36950/44000 | Loss: 2.8901 | LR: 3.21e-05 | 3489.5ms/step | 5.7 samples/s
213
+ Step 36960/44000 | Loss: 2.8951 | LR: 3.20e-05 | 3488.1ms/step | 5.7 samples/s
214
+ Step 36970/44000 | Loss: 2.8883 | LR: 3.19e-05 | 3488.9ms/step | 5.7 samples/s
215
+ Step 36980/44000 | Loss: 2.9050 | LR: 3.18e-05 | 3493.8ms/step | 5.7 samples/s
216
+ Step 36990/44000 | Loss: 2.8858 | LR: 3.17e-05 | 3500.6ms/step | 5.7 samples/s
217
+ Step 37000/44000 | Loss: 2.8780 | LR: 3.16e-05 | 3490.0ms/step | 5.7 samples/s
218
+ Validation at step 37000: Loss = 2.8840, Perplexity = 17.89
219
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-37000
220
+ Step 37010/44000 | Loss: 2.9048 | LR: 3.15e-05 | 9630.8ms/step | 2.1 samples/s
221
+ Step 37020/44000 | Loss: 2.8802 | LR: 3.15e-05 | 3483.7ms/step | 5.7 samples/s
222
+ Step 37030/44000 | Loss: 2.9201 | LR: 3.14e-05 | 3485.0ms/step | 5.7 samples/s
223
+ Step 37040/44000 | Loss: 2.8763 | LR: 3.13e-05 | 3487.4ms/step | 5.7 samples/s
224
+ Step 37050/44000 | Loss: 2.8784 | LR: 3.12e-05 | 3487.8ms/step | 5.7 samples/s
225
+ Step 37060/44000 | Loss: 2.8811 | LR: 3.11e-05 | 3488.5ms/step | 5.7 samples/s
226
+ Step 37070/44000 | Loss: 2.9084 | LR: 3.10e-05 | 3487.0ms/step | 5.7 samples/s
227
+ Step 37080/44000 | Loss: 2.8987 | LR: 3.09e-05 | 3487.5ms/step | 5.7 samples/s
228
+ Step 37090/44000 | Loss: 2.9161 | LR: 3.08e-05 | 3486.4ms/step | 5.7 samples/s
229
+ Step 37100/44000 | Loss: 2.8797 | LR: 3.07e-05 | 3487.6ms/step | 5.7 samples/s
230
+ Step 37110/44000 | Loss: 2.8687 | LR: 3.07e-05 | 3487.1ms/step | 5.7 samples/s
231
+ Step 37120/44000 | Loss: 2.8514 | LR: 3.06e-05 | 3487.4ms/step | 5.7 samples/s
232
+ Step 37130/44000 | Loss: 2.8935 | LR: 3.05e-05 | 3486.5ms/step | 5.7 samples/s
233
+ Step 37140/44000 | Loss: 2.8890 | LR: 3.04e-05 | 3485.9ms/step | 5.7 samples/s
234
+ Step 37150/44000 | Loss: 2.8904 | LR: 3.03e-05 | 3485.9ms/step | 5.7 samples/s
235
+ Step 37160/44000 | Loss: 2.8856 | LR: 3.02e-05 | 3488.0ms/step | 5.7 samples/s
236
+ Step 37170/44000 | Loss: 2.8810 | LR: 3.01e-05 | 3486.0ms/step | 5.7 samples/s
237
+ Step 37180/44000 | Loss: 2.9045 | LR: 3.01e-05 | 3485.1ms/step | 5.7 samples/s
238
+ Step 37190/44000 | Loss: 2.8645 | LR: 3.00e-05 | 3485.5ms/step | 5.7 samples/s
239
+ Step 37200/44000 | Loss: 2.9113 | LR: 2.99e-05 | 3486.3ms/step | 5.7 samples/s
240
+ Step 37210/44000 | Loss: 2.8961 | LR: 2.98e-05 | 3485.6ms/step | 5.7 samples/s
241
+ Step 37220/44000 | Loss: 2.8812 | LR: 2.97e-05 | 3485.7ms/step | 5.7 samples/s
242
+ Step 37230/44000 | Loss: 2.9025 | LR: 2.96e-05 | 3486.5ms/step | 5.7 samples/s
243
+ Step 37240/44000 | Loss: 2.8988 | LR: 2.95e-05 | 3487.1ms/step | 5.7 samples/s
244
+ Step 37250/44000 | Loss: 2.8817 | LR: 2.95e-05 | 3485.7ms/step | 5.7 samples/s
245
+ Validation at step 37250: Loss = 2.8838, Perplexity = 17.88
246
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-37250
247
+ Step 37260/44000 | Loss: 2.8586 | LR: 2.94e-05 | 9631.4ms/step | 2.1 samples/s
248
+ Step 37270/44000 | Loss: 2.9065 | LR: 2.93e-05 | 3485.7ms/step | 5.7 samples/s
249
+ Step 37280/44000 | Loss: 2.9001 | LR: 2.92e-05 | 3486.5ms/step | 5.7 samples/s
250
+ Step 37290/44000 | Loss: 2.9042 | LR: 2.91e-05 | 3485.8ms/step | 5.7 samples/s
251
+ Step 37300/44000 | Loss: 2.8903 | LR: 2.90e-05 | 3486.7ms/step | 5.7 samples/s
252
+ Step 37310/44000 | Loss: 2.8664 | LR: 2.89e-05 | 3486.2ms/step | 5.7 samples/s
253
+ Step 37320/44000 | Loss: 2.9003 | LR: 2.89e-05 | 3486.3ms/step | 5.7 samples/s
254
+ Step 37330/44000 | Loss: 2.8966 | LR: 2.88e-05 | 3486.0ms/step | 5.7 samples/s
255
+ Step 37340/44000 | Loss: 2.9077 | LR: 2.87e-05 | 3487.0ms/step | 5.7 samples/s
256
+ Step 37350/44000 | Loss: 2.8925 | LR: 2.86e-05 | 3485.3ms/step | 5.7 samples/s
257
+ Step 37360/44000 | Loss: 2.8899 | LR: 2.85e-05 | 3484.7ms/step | 5.7 samples/s
258
+ Step 37370/44000 | Loss: 2.9053 | LR: 2.84e-05 | 3485.1ms/step | 5.7 samples/s
259
+ Step 37380/44000 | Loss: 2.8723 | LR: 2.84e-05 | 3485.8ms/step | 5.7 samples/s
260
+ Step 37390/44000 | Loss: 2.8845 | LR: 2.83e-05 | 3486.0ms/step | 5.7 samples/s
261
+ Step 37400/44000 | Loss: 2.8562 | LR: 2.82e-05 | 3484.7ms/step | 5.7 samples/s
262
+ Step 37410/44000 | Loss: 2.8511 | LR: 2.81e-05 | 3484.7ms/step | 5.7 samples/s
263
+ Step 37420/44000 | Loss: 2.9127 | LR: 2.80e-05 | 3484.3ms/step | 5.7 samples/s
264
+ Step 37430/44000 | Loss: 2.8488 | LR: 2.79e-05 | 3484.4ms/step | 5.7 samples/s
265
+ Step 37440/44000 | Loss: 2.9080 | LR: 2.78e-05 | 3486.6ms/step | 5.7 samples/s
266
+ Step 37450/44000 | Loss: 2.8980 | LR: 2.78e-05 | 3486.0ms/step | 5.7 samples/s
267
+ Step 37460/44000 | Loss: 2.8963 | LR: 2.77e-05 | 3485.5ms/step | 5.7 samples/s
268
+ Step 37470/44000 | Loss: 2.8955 | LR: 2.76e-05 | 3484.8ms/step | 5.7 samples/s
269
+ Step 37480/44000 | Loss: 2.8842 | LR: 2.75e-05 | 3484.4ms/step | 5.7 samples/s
270
+ Step 37490/44000 | Loss: 2.8773 | LR: 2.74e-05 | 3484.9ms/step | 5.7 samples/s
271
+ Step 37500/44000 | Loss: 2.9109 | LR: 2.74e-05 | 3486.4ms/step | 5.7 samples/s
272
+ Validation at step 37500: Loss = 2.8837, Perplexity = 17.88
273
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-37500
274
+ Step 37510/44000 | Loss: 2.9011 | LR: 2.73e-05 | 9615.8ms/step | 2.1 samples/s
275
+ Step 37520/44000 | Loss: 2.8888 | LR: 2.72e-05 | 3486.1ms/step | 5.7 samples/s
276
+ Step 37530/44000 | Loss: 2.8985 | LR: 2.71e-05 | 3485.6ms/step | 5.7 samples/s
277
+ Step 37540/44000 | Loss: 2.8900 | LR: 2.70e-05 | 3486.3ms/step | 5.7 samples/s
278
+ Step 37550/44000 | Loss: 2.8959 | LR: 2.69e-05 | 3486.6ms/step | 5.7 samples/s
279
+ Step 37560/44000 | Loss: 2.8895 | LR: 2.69e-05 | 3495.0ms/step | 5.7 samples/s
280
+ Step 37570/44000 | Loss: 2.9101 | LR: 2.68e-05 | 3498.0ms/step | 5.7 samples/s
281
+ Step 37580/44000 | Loss: 2.8613 | LR: 2.67e-05 | 3498.2ms/step | 5.7 samples/s
282
+ Step 37590/44000 | Loss: 2.8948 | LR: 2.66e-05 | 3499.6ms/step | 5.7 samples/s
283
+ Step 37600/44000 | Loss: 2.9054 | LR: 2.65e-05 | 3498.9ms/step | 5.7 samples/s
284
+ Step 37610/44000 | Loss: 2.8746 | LR: 2.65e-05 | 3497.6ms/step | 5.7 samples/s
285
+ Step 37620/44000 | Loss: 2.8744 | LR: 2.64e-05 | 3496.5ms/step | 5.7 samples/s
286
+ Step 37630/44000 | Loss: 2.8658 | LR: 2.63e-05 | 3497.6ms/step | 5.7 samples/s
287
+ Step 37640/44000 | Loss: 2.8888 | LR: 2.62e-05 | 3494.9ms/step | 5.7 samples/s
288
+ Step 37650/44000 | Loss: 2.8947 | LR: 2.61e-05 | 3485.3ms/step | 5.7 samples/s
289
+ Step 37660/44000 | Loss: 2.9004 | LR: 2.60e-05 | 3489.0ms/step | 5.7 samples/s
290
+ Step 37670/44000 | Loss: 2.9159 | LR: 2.60e-05 | 3486.2ms/step | 5.7 samples/s
291
+ Step 37680/44000 | Loss: 2.9014 | LR: 2.59e-05 | 3486.2ms/step | 5.7 samples/s
292
+ Step 37690/44000 | Loss: 2.8955 | LR: 2.58e-05 | 3486.0ms/step | 5.7 samples/s
293
+ Step 37700/44000 | Loss: 2.8847 | LR: 2.57e-05 | 3484.9ms/step | 5.7 samples/s
294
+ Step 37710/44000 | Loss: 2.8829 | LR: 2.56e-05 | 3487.0ms/step | 5.7 samples/s
295
+ Step 37720/44000 | Loss: 2.8916 | LR: 2.56e-05 | 3485.3ms/step | 5.7 samples/s
296
+ Step 37730/44000 | Loss: 2.8766 | LR: 2.55e-05 | 3485.7ms/step | 5.7 samples/s
297
+ Step 37740/44000 | Loss: 2.8799 | LR: 2.54e-05 | 3488.8ms/step | 5.7 samples/s
298
+ Step 37750/44000 | Loss: 2.8913 | LR: 2.53e-05 | 3485.2ms/step | 5.7 samples/s
299
+ Validation at step 37750: Loss = 2.8836, Perplexity = 17.88
300
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-37750
301
+ Step 37760/44000 | Loss: 2.9015 | LR: 2.52e-05 | 9646.7ms/step | 2.1 samples/s
302
+ Step 37770/44000 | Loss: 2.8989 | LR: 2.52e-05 | 3489.5ms/step | 5.7 samples/s
303
+ Step 37780/44000 | Loss: 2.8822 | LR: 2.51e-05 | 3486.5ms/step | 5.7 samples/s
304
+ Step 37790/44000 | Loss: 2.8924 | LR: 2.50e-05 | 3486.3ms/step | 5.7 samples/s
305
+ Step 37800/44000 | Loss: 2.8965 | LR: 2.49e-05 | 3489.1ms/step | 5.7 samples/s
306
+ Step 37810/44000 | Loss: 2.8511 | LR: 2.48e-05 | 3498.1ms/step | 5.7 samples/s
307
+ Step 37820/44000 | Loss: 2.8879 | LR: 2.48e-05 | 3490.3ms/step | 5.7 samples/s
308
+ Step 37830/44000 | Loss: 2.8893 | LR: 2.47e-05 | 3487.6ms/step | 5.7 samples/s
309
+ Step 37840/44000 | Loss: 2.9027 | LR: 2.46e-05 | 3488.2ms/step | 5.7 samples/s
310
+ Step 37850/44000 | Loss: 2.9052 | LR: 2.45e-05 | 3487.4ms/step | 5.7 samples/s
311
+ Step 37860/44000 | Loss: 2.8914 | LR: 2.45e-05 | 3492.3ms/step | 5.7 samples/s
312
+ Step 37870/44000 | Loss: 2.8925 | LR: 2.44e-05 | 3489.7ms/step | 5.7 samples/s
313
+ Step 37880/44000 | Loss: 2.8718 | LR: 2.43e-05 | 3495.3ms/step | 5.7 samples/s
314
+ Step 37890/44000 | Loss: 2.8729 | LR: 2.42e-05 | 3488.2ms/step | 5.7 samples/s
315
+ Step 37900/44000 | Loss: 2.9160 | LR: 2.41e-05 | 3485.6ms/step | 5.7 samples/s
316
+ Step 37910/44000 | Loss: 2.8900 | LR: 2.41e-05 | 3488.8ms/step | 5.7 samples/s
317
+ Step 37920/44000 | Loss: 2.8758 | LR: 2.40e-05 | 3489.7ms/step | 5.7 samples/s
318
+ Step 37930/44000 | Loss: 2.8815 | LR: 2.39e-05 | 3496.0ms/step | 5.7 samples/s
319
+ Step 37940/44000 | Loss: 2.8796 | LR: 2.38e-05 | 3490.3ms/step | 5.7 samples/s
320
+ Step 37950/44000 | Loss: 2.9084 | LR: 2.38e-05 | 3488.3ms/step | 5.7 samples/s
321
+ Step 37960/44000 | Loss: 2.8773 | LR: 2.37e-05 | 3487.2ms/step | 5.7 samples/s
322
+ Step 37970/44000 | Loss: 2.8711 | LR: 2.36e-05 | 3487.0ms/step | 5.7 samples/s
323
+ Step 37980/44000 | Loss: 2.9159 | LR: 2.35e-05 | 3487.5ms/step | 5.7 samples/s
324
+ Step 37990/44000 | Loss: 2.8902 | LR: 2.34e-05 | 3487.6ms/step | 5.7 samples/s
325
+ Step 38000/44000 | Loss: 2.8914 | LR: 2.34e-05 | 3488.0ms/step | 5.7 samples/s
326
+ Validation at step 38000: Loss = 2.8835, Perplexity = 17.88
327
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-38000
328
+ Step 38010/44000 | Loss: 2.8797 | LR: 2.33e-05 | 9649.3ms/step | 2.1 samples/s
329
+ Step 38020/44000 | Loss: 2.8699 | LR: 2.32e-05 | 3492.1ms/step | 5.7 samples/s
330
+ Step 38030/44000 | Loss: 2.8649 | LR: 2.31e-05 | 3494.2ms/step | 5.7 samples/s
331
+ Step 38040/44000 | Loss: 2.8851 | LR: 2.31e-05 | 3486.5ms/step | 5.7 samples/s
332
+ Step 38050/44000 | Loss: 2.9102 | LR: 2.30e-05 | 3486.5ms/step | 5.7 samples/s
333
+ Step 38060/44000 | Loss: 2.8755 | LR: 2.29e-05 | 3489.0ms/step | 5.7 samples/s
334
+ Step 38070/44000 | Loss: 2.8752 | LR: 2.28e-05 | 3487.6ms/step | 5.7 samples/s
335
+ Step 38080/44000 | Loss: 2.9016 | LR: 2.28e-05 | 3487.1ms/step | 5.7 samples/s
336
+ Step 38090/44000 | Loss: 2.9041 | LR: 2.27e-05 | 3486.2ms/step | 5.7 samples/s
337
+ Step 38100/44000 | Loss: 2.9084 | LR: 2.26e-05 | 3487.0ms/step | 5.7 samples/s
338
+ Step 38110/44000 | Loss: 2.8861 | LR: 2.25e-05 | 3485.1ms/step | 5.7 samples/s
339
+ Step 38120/44000 | Loss: 2.8752 | LR: 2.25e-05 | 3486.6ms/step | 5.7 samples/s
340
+ Step 38130/44000 | Loss: 2.9170 | LR: 2.24e-05 | 3486.4ms/step | 5.7 samples/s
341
+ Step 38140/44000 | Loss: 2.9033 | LR: 2.23e-05 | 3486.6ms/step | 5.7 samples/s
342
+ Step 38150/44000 | Loss: 2.8744 | LR: 2.22e-05 | 3486.5ms/step | 5.7 samples/s
343
+ Step 38160/44000 | Loss: 2.9056 | LR: 2.22e-05 | 3486.2ms/step | 5.7 samples/s
344
+ Step 38170/44000 | Loss: 2.8927 | LR: 2.21e-05 | 3485.7ms/step | 5.7 samples/s
345
+ Step 38180/44000 | Loss: 2.8823 | LR: 2.20e-05 | 3489.2ms/step | 5.7 samples/s
346
+ Step 38190/44000 | Loss: 2.8737 | LR: 2.19e-05 | 3489.3ms/step | 5.7 samples/s
347
+ Step 38200/44000 | Loss: 2.8996 | LR: 2.19e-05 | 3489.0ms/step | 5.7 samples/s
348
+ Step 38210/44000 | Loss: 2.9146 | LR: 2.18e-05 | 3487.4ms/step | 5.7 samples/s
349
+ Step 38220/44000 | Loss: 2.8765 | LR: 2.17e-05 | 3487.6ms/step | 5.7 samples/s
350
+ Step 38230/44000 | Loss: 2.8652 | LR: 2.16e-05 | 3486.5ms/step | 5.7 samples/s
351
+ Step 38240/44000 | Loss: 2.8775 | LR: 2.16e-05 | 3487.5ms/step | 5.7 samples/s
352
+ Step 38250/44000 | Loss: 2.9002 | LR: 2.15e-05 | 3486.4ms/step | 5.7 samples/s
353
+ Validation at step 38250: Loss = 2.8835, Perplexity = 17.88
354
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-38250
355
+ Step 38260/44000 | Loss: 2.8954 | LR: 2.14e-05 | 9645.5ms/step | 2.1 samples/s
356
+ Step 38270/44000 | Loss: 2.9164 | LR: 2.13e-05 | 3493.5ms/step | 5.7 samples/s
357
+ Step 38280/44000 | Loss: 2.8964 | LR: 2.13e-05 | 3494.8ms/step | 5.7 samples/s
358
+ Step 38290/44000 | Loss: 2.8984 | LR: 2.12e-05 | 3486.3ms/step | 5.7 samples/s
359
+ Step 38300/44000 | Loss: 2.9234 | LR: 2.11e-05 | 3487.6ms/step | 5.7 samples/s
360
+ Step 38310/44000 | Loss: 2.8820 | LR: 2.11e-05 | 3486.4ms/step | 5.7 samples/s
361
+ Step 38320/44000 | Loss: 2.8812 | LR: 2.10e-05 | 3486.6ms/step | 5.7 samples/s
362
+ Step 38330/44000 | Loss: 2.9166 | LR: 2.09e-05 | 3485.8ms/step | 5.7 samples/s
363
+ Step 38340/44000 | Loss: 2.8794 | LR: 2.08e-05 | 3485.4ms/step | 5.7 samples/s
364
+ Step 38350/44000 | Loss: 2.8942 | LR: 2.08e-05 | 3485.7ms/step | 5.7 samples/s
365
+ Step 38360/44000 | Loss: 2.8879 | LR: 2.07e-05 | 3487.1ms/step | 5.7 samples/s
366
+ Step 38370/44000 | Loss: 2.9312 | LR: 2.06e-05 | 3486.9ms/step | 5.7 samples/s
367
+ Step 38380/44000 | Loss: 2.8718 | LR: 2.05e-05 | 3486.5ms/step | 5.7 samples/s
368
+ Step 38390/44000 | Loss: 2.8755 | LR: 2.05e-05 | 3486.4ms/step | 5.7 samples/s
369
+ Step 38400/44000 | Loss: 2.9148 | LR: 2.04e-05 | 3485.1ms/step | 5.7 samples/s
370
+ Step 38410/44000 | Loss: 2.8978 | LR: 2.03e-05 | 3486.5ms/step | 5.7 samples/s
371
+ Step 38420/44000 | Loss: 2.8722 | LR: 2.03e-05 | 3488.0ms/step | 5.7 samples/s
372
+ Step 38430/44000 | Loss: 2.9199 | LR: 2.02e-05 | 3486.5ms/step | 5.7 samples/s
373
+ Step 38440/44000 | Loss: 2.9043 | LR: 2.01e-05 | 3486.6ms/step | 5.7 samples/s
374
+ Step 38450/44000 | Loss: 2.9236 | LR: 2.00e-05 | 3484.7ms/step | 5.7 samples/s
375
+ Step 38460/44000 | Loss: 2.8938 | LR: 2.00e-05 | 3486.1ms/step | 5.7 samples/s
376
+ Step 38470/44000 | Loss: 2.8786 | LR: 1.99e-05 | 3486.6ms/step | 5.7 samples/s
377
+ Step 38480/44000 | Loss: 2.9218 | LR: 1.98e-05 | 3486.1ms/step | 5.7 samples/s
378
+ Step 38490/44000 | Loss: 2.8875 | LR: 1.98e-05 | 3485.7ms/step | 5.7 samples/s
379
+ Step 38500/44000 | Loss: 2.8618 | LR: 1.97e-05 | 3485.6ms/step | 5.7 samples/s
380
+ Validation at step 38500: Loss = 2.8834, Perplexity = 17.87
381
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-38500
382
+ Step 38510/44000 | Loss: 2.8954 | LR: 1.96e-05 | 9620.8ms/step | 2.1 samples/s
383
+ Step 38520/44000 | Loss: 2.8708 | LR: 1.95e-05 | 3484.4ms/step | 5.7 samples/s
384
+ Step 38530/44000 | Loss: 2.8955 | LR: 1.95e-05 | 3485.4ms/step | 5.7 samples/s
385
+ Step 38540/44000 | Loss: 2.9254 | LR: 1.94e-05 | 3485.0ms/step | 5.7 samples/s
386
+ Step 38550/44000 | Loss: 2.8701 | LR: 1.93e-05 | 3486.2ms/step | 5.7 samples/s
387
+ Step 38560/44000 | Loss: 2.9157 | LR: 1.93e-05 | 3486.0ms/step | 5.7 samples/s
388
+ Step 38570/44000 | Loss: 2.8931 | LR: 1.92e-05 | 3485.8ms/step | 5.7 samples/s
389
+ Step 38580/44000 | Loss: 2.9068 | LR: 1.91e-05 | 3483.7ms/step | 5.7 samples/s
390
+ Step 38590/44000 | Loss: 2.9019 | LR: 1.91e-05 | 3487.3ms/step | 5.7 samples/s
391
+ Step 38600/44000 | Loss: 2.8787 | LR: 1.90e-05 | 3486.2ms/step | 5.7 samples/s
392
+ Step 38610/44000 | Loss: 2.9009 | LR: 1.89e-05 | 3484.5ms/step | 5.7 samples/s
393
+ Step 38620/44000 | Loss: 2.9080 | LR: 1.88e-05 | 3486.0ms/step | 5.7 samples/s
394
+ Step 38630/44000 | Loss: 2.9041 | LR: 1.88e-05 | 3485.0ms/step | 5.7 samples/s
395
+ Step 38640/44000 | Loss: 2.9040 | LR: 1.87e-05 | 3485.0ms/step | 5.7 samples/s
396
+ Step 38650/44000 | Loss: 2.8769 | LR: 1.86e-05 | 3487.9ms/step | 5.7 samples/s
397
+ Step 38660/44000 | Loss: 2.8894 | LR: 1.86e-05 | 3487.3ms/step | 5.7 samples/s
398
+ Step 38670/44000 | Loss: 2.8958 | LR: 1.85e-05 | 3484.2ms/step | 5.7 samples/s
399
+ Step 38680/44000 | Loss: 2.9272 | LR: 1.84e-05 | 3486.6ms/step | 5.7 samples/s
400
+ Step 38690/44000 | Loss: 2.8853 | LR: 1.84e-05 | 3485.8ms/step | 5.7 samples/s
401
+ Step 38700/44000 | Loss: 2.8920 | LR: 1.83e-05 | 3486.5ms/step | 5.7 samples/s
402
+ Step 38710/44000 | Loss: 2.8824 | LR: 1.82e-05 | 3485.8ms/step | 5.7 samples/s
403
+ Step 38720/44000 | Loss: 2.8882 | LR: 1.82e-05 | 3488.2ms/step | 5.7 samples/s
404
+ Step 38730/44000 | Loss: 2.8569 | LR: 1.81e-05 | 3489.2ms/step | 5.7 samples/s
405
+ Step 38740/44000 | Loss: 2.8782 | LR: 1.80e-05 | 3489.1ms/step | 5.7 samples/s
406
+ Step 38750/44000 | Loss: 2.8977 | LR: 1.80e-05 | 3487.3ms/step | 5.7 samples/s
407
+ Validation at step 38750: Loss = 2.8833, Perplexity = 17.87
408
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-38750
409
+ Step 38760/44000 | Loss: 2.8922 | LR: 1.79e-05 | 9611.3ms/step | 2.1 samples/s
410
+ Step 38770/44000 | Loss: 2.8455 | LR: 1.78e-05 | 3487.5ms/step | 5.7 samples/s
411
+ Step 38780/44000 | Loss: 2.9220 | LR: 1.78e-05 | 3483.1ms/step | 5.7 samples/s
412
+ Step 38790/44000 | Loss: 2.8848 | LR: 1.77e-05 | 3483.0ms/step | 5.7 samples/s
413
+ Step 38800/44000 | Loss: 2.8974 | LR: 1.76e-05 | 3486.0ms/step | 5.7 samples/s
414
+ Step 38810/44000 | Loss: 2.9023 | LR: 1.76e-05 | 3484.4ms/step | 5.7 samples/s
415
+ Step 38820/44000 | Loss: 2.8866 | LR: 1.75e-05 | 3484.6ms/step | 5.7 samples/s
416
+ Step 38830/44000 | Loss: 2.8964 | LR: 1.74e-05 | 3485.4ms/step | 5.7 samples/s
417
+ Step 38840/44000 | Loss: 2.8731 | LR: 1.74e-05 | 3485.8ms/step | 5.7 samples/s
418
+ Step 38850/44000 | Loss: 2.9079 | LR: 1.73e-05 | 3485.2ms/step | 5.7 samples/s
419
+ Step 38860/44000 | Loss: 2.9400 | LR: 1.72e-05 | 3485.0ms/step | 5.7 samples/s
420
+ Step 38870/44000 | Loss: 2.8958 | LR: 1.72e-05 | 3483.9ms/step | 5.7 samples/s
421
+ Step 38880/44000 | Loss: 2.8830 | LR: 1.71e-05 | 3483.5ms/step | 5.7 samples/s
422
+ Step 38890/44000 | Loss: 2.8821 | LR: 1.70e-05 | 3485.3ms/step | 5.7 samples/s
423
+ Step 38900/44000 | Loss: 2.8791 | LR: 1.70e-05 | 3485.4ms/step | 5.7 samples/s
424
+ Step 38910/44000 | Loss: 2.8996 | LR: 1.69e-05 | 3485.2ms/step | 5.7 samples/s
425
+ Step 38920/44000 | Loss: 2.9059 | LR: 1.68e-05 | 3484.8ms/step | 5.7 samples/s
426
+ Step 38930/44000 | Loss: 2.8831 | LR: 1.68e-05 | 3485.6ms/step | 5.7 samples/s
427
+ Step 38940/44000 | Loss: 2.9045 | LR: 1.67e-05 | 3483.8ms/step | 5.7 samples/s
428
+ Step 38950/44000 | Loss: 2.9033 | LR: 1.66e-05 | 3485.0ms/step | 5.7 samples/s
429
+ Step 38960/44000 | Loss: 2.8877 | LR: 1.66e-05 | 3485.1ms/step | 5.7 samples/s
430
+ Step 38970/44000 | Loss: 2.8918 | LR: 1.65e-05 | 3484.1ms/step | 5.7 samples/s
431
+ Step 38980/44000 | Loss: 2.8852 | LR: 1.64e-05 | 3485.3ms/step | 5.7 samples/s
432
+ Step 38990/44000 | Loss: 2.8947 | LR: 1.64e-05 | 3485.9ms/step | 5.7 samples/s
433
+ Step 39000/44000 | Loss: 2.8860 | LR: 1.63e-05 | 3485.3ms/step | 5.7 samples/s
434
+ Validation at step 39000: Loss = 2.8833, Perplexity = 17.87
435
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-39000
436
+ Step 39010/44000 | Loss: 2.9010 | LR: 1.62e-05 | 9606.9ms/step | 2.1 samples/s
437
+ Step 39020/44000 | Loss: 2.8864 | LR: 1.62e-05 | 3482.4ms/step | 5.7 samples/s
438
+ Step 39030/44000 | Loss: 2.8720 | LR: 1.61e-05 | 3488.0ms/step | 5.7 samples/s
439
+ Step 39040/44000 | Loss: 2.8917 | LR: 1.61e-05 | 3492.7ms/step | 5.7 samples/s
440
+ Step 39050/44000 | Loss: 2.9051 | LR: 1.60e-05 | 3494.5ms/step | 5.7 samples/s
441
+ Step 39060/44000 | Loss: 2.8858 | LR: 1.59e-05 | 3495.1ms/step | 5.7 samples/s
442
+ Step 39070/44000 | Loss: 2.8830 | LR: 1.59e-05 | 3493.2ms/step | 5.7 samples/s
443
+ Step 39080/44000 | Loss: 2.8988 | LR: 1.58e-05 | 3491.8ms/step | 5.7 samples/s
444
+ Step 39090/44000 | Loss: 2.9112 | LR: 1.57e-05 | 3491.0ms/step | 5.7 samples/s
445
+ Step 39100/44000 | Loss: 2.9138 | LR: 1.57e-05 | 3489.9ms/step | 5.7 samples/s
446
+ Step 39110/44000 | Loss: 2.8764 | LR: 1.56e-05 | 3488.4ms/step | 5.7 samples/s
447
+ Step 39120/44000 | Loss: 2.8823 | LR: 1.55e-05 | 3486.2ms/step | 5.7 samples/s
448
+ Step 39130/44000 | Loss: 2.9241 | LR: 1.55e-05 | 3483.1ms/step | 5.7 samples/s
449
+ Step 39140/44000 | Loss: 2.8704 | LR: 1.54e-05 | 3483.8ms/step | 5.7 samples/s
450
+ Step 39150/44000 | Loss: 2.8761 | LR: 1.54e-05 | 3484.5ms/step | 5.7 samples/s
451
+ Step 39160/44000 | Loss: 2.8529 | LR: 1.53e-05 | 3485.0ms/step | 5.7 samples/s
452
+ Step 39170/44000 | Loss: 2.8952 | LR: 1.52e-05 | 3484.0ms/step | 5.7 samples/s
453
+ Step 39180/44000 | Loss: 2.8914 | LR: 1.52e-05 | 3485.6ms/step | 5.7 samples/s
454
+ Step 39190/44000 | Loss: 2.9106 | LR: 1.51e-05 | 3485.2ms/step | 5.7 samples/s
455
+ Step 39200/44000 | Loss: 2.9047 | LR: 1.50e-05 | 3484.5ms/step | 5.7 samples/s
456
+ Step 39210/44000 | Loss: 2.8866 | LR: 1.50e-05 | 3484.6ms/step | 5.7 samples/s
457
+ Step 39220/44000 | Loss: 2.8819 | LR: 1.49e-05 | 3485.1ms/step | 5.7 samples/s
458
+ Step 39230/44000 | Loss: 2.8695 | LR: 1.49e-05 | 3484.1ms/step | 5.7 samples/s
459
+ Step 39240/44000 | Loss: 2.9011 | LR: 1.48e-05 | 3484.4ms/step | 5.7 samples/s
460
+ Step 39250/44000 | Loss: 2.9119 | LR: 1.47e-05 | 3485.9ms/step | 5.7 samples/s
461
+ Validation at step 39250: Loss = 2.8832, Perplexity = 17.87
462
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-39250
463
+ Step 39260/44000 | Loss: 2.8855 | LR: 1.47e-05 | 9616.0ms/step | 2.1 samples/s
464
+ Step 39270/44000 | Loss: 2.8910 | LR: 1.46e-05 | 3483.4ms/step | 5.7 samples/s
465
+ Step 39280/44000 | Loss: 2.8807 | LR: 1.46e-05 | 3484.7ms/step | 5.7 samples/s
466
+ Step 39290/44000 | Loss: 2.9099 | LR: 1.45e-05 | 3482.5ms/step | 5.7 samples/s
467
+ Step 39300/44000 | Loss: 2.8994 | LR: 1.44e-05 | 3483.8ms/step | 5.7 samples/s
468
+ Step 39310/44000 | Loss: 2.8927 | LR: 1.44e-05 | 3483.9ms/step | 5.7 samples/s
469
+ Step 39320/44000 | Loss: 2.8802 | LR: 1.43e-05 | 3485.3ms/step | 5.7 samples/s
470
+ Step 39330/44000 | Loss: 2.8858 | LR: 1.42e-05 | 3485.0ms/step | 5.7 samples/s
471
+ Step 39340/44000 | Loss: 2.8941 | LR: 1.42e-05 | 3485.3ms/step | 5.7 samples/s
472
+ Step 39350/44000 | Loss: 2.8686 | LR: 1.41e-05 | 3485.2ms/step | 5.7 samples/s
473
+ Step 39360/44000 | Loss: 2.8655 | LR: 1.41e-05 | 3485.9ms/step | 5.7 samples/s
474
+ Step 39370/44000 | Loss: 2.8867 | LR: 1.40e-05 | 3484.8ms/step | 5.7 samples/s
475
+ Step 39380/44000 | Loss: 2.9092 | LR: 1.39e-05 | 3484.8ms/step | 5.7 samples/s
476
+ Step 39390/44000 | Loss: 2.8949 | LR: 1.39e-05 | 3484.8ms/step | 5.7 samples/s
477
+ Step 39400/44000 | Loss: 2.9020 | LR: 1.38e-05 | 3484.4ms/step | 5.7 samples/s
478
+ Step 39410/44000 | Loss: 2.9065 | LR: 1.38e-05 | 3484.8ms/step | 5.7 samples/s
479
+ Step 39420/44000 | Loss: 2.8939 | LR: 1.37e-05 | 3484.2ms/step | 5.7 samples/s
480
+ Step 39430/44000 | Loss: 2.8737 | LR: 1.36e-05 | 3483.7ms/step | 5.7 samples/s
481
+ Step 39440/44000 | Loss: 2.8731 | LR: 1.36e-05 | 3485.0ms/step | 5.7 samples/s
482
+ Step 39450/44000 | Loss: 2.9296 | LR: 1.35e-05 | 3485.5ms/step | 5.7 samples/s
483
+ Step 39460/44000 | Loss: 2.8824 | LR: 1.35e-05 | 3484.4ms/step | 5.7 samples/s
484
+ Step 39470/44000 | Loss: 2.9017 | LR: 1.34e-05 | 3482.8ms/step | 5.7 samples/s
485
+ Step 39480/44000 | Loss: 2.9101 | LR: 1.34e-05 | 3484.0ms/step | 5.7 samples/s
486
+ Step 39490/44000 | Loss: 2.8711 | LR: 1.33e-05 | 3482.5ms/step | 5.7 samples/s
487
+ Step 39500/44000 | Loss: 2.8966 | LR: 1.32e-05 | 3483.4ms/step | 5.7 samples/s
488
+ Validation at step 39500: Loss = 2.8832, Perplexity = 17.87
489
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-39500
490
+ Step 39510/44000 | Loss: 2.8614 | LR: 1.32e-05 | 9613.2ms/step | 2.1 samples/s
491
+ Step 39520/44000 | Loss: 2.8982 | LR: 1.31e-05 | 3484.3ms/step | 5.7 samples/s
492
+ Step 39530/44000 | Loss: 2.8853 | LR: 1.31e-05 | 3484.5ms/step | 5.7 samples/s
493
+ Step 39540/44000 | Loss: 2.9243 | LR: 1.30e-05 | 3485.3ms/step | 5.7 samples/s
494
+ Step 39550/44000 | Loss: 2.9116 | LR: 1.29e-05 | 3483.9ms/step | 5.7 samples/s
495
+ Step 39560/44000 | Loss: 2.8959 | LR: 1.29e-05 | 3482.8ms/step | 5.7 samples/s
496
+ Step 39570/44000 | Loss: 2.9110 | LR: 1.28e-05 | 3483.8ms/step | 5.7 samples/s
497
+ Step 39580/44000 | Loss: 2.9103 | LR: 1.28e-05 | 3485.4ms/step | 5.7 samples/s
498
+ Step 39590/44000 | Loss: 2.8855 | LR: 1.27e-05 | 3484.2ms/step | 5.7 samples/s
499
+ Step 39600/44000 | Loss: 2.8874 | LR: 1.27e-05 | 3484.4ms/step | 5.7 samples/s
500
+ Step 39610/44000 | Loss: 2.8873 | LR: 1.26e-05 | 3483.4ms/step | 5.7 samples/s
501
+ Step 39620/44000 | Loss: 2.8653 | LR: 1.25e-05 | 3484.5ms/step | 5.7 samples/s
502
+ Step 39630/44000 | Loss: 2.9023 | LR: 1.25e-05 | 3484.6ms/step | 5.7 samples/s
503
+ Step 39640/44000 | Loss: 2.8862 | LR: 1.24e-05 | 3487.3ms/step | 5.7 samples/s
504
+ Step 39650/44000 | Loss: 2.8793 | LR: 1.24e-05 | 3486.2ms/step | 5.7 samples/s
505
+ Step 39660/44000 | Loss: 2.9054 | LR: 1.23e-05 | 3484.3ms/step | 5.7 samples/s
506
+ Step 39670/44000 | Loss: 2.8955 | LR: 1.23e-05 | 3483.8ms/step | 5.7 samples/s
507
+ Step 39680/44000 | Loss: 2.8911 | LR: 1.22e-05 | 3485.3ms/step | 5.7 samples/s
508
+ Step 39690/44000 | Loss: 2.8734 | LR: 1.22e-05 | 3484.4ms/step | 5.7 samples/s
509
+ Step 39700/44000 | Loss: 2.8775 | LR: 1.21e-05 | 3484.1ms/step | 5.7 samples/s
510
+ Step 39710/44000 | Loss: 2.8790 | LR: 1.20e-05 | 3482.7ms/step | 5.7 samples/s
511
+ Step 39720/44000 | Loss: 2.9079 | LR: 1.20e-05 | 3482.8ms/step | 5.7 samples/s
512
+ Step 39730/44000 | Loss: 2.8793 | LR: 1.19e-05 | 3483.2ms/step | 5.7 samples/s
513
+ Step 39740/44000 | Loss: 2.8828 | LR: 1.19e-05 | 3483.0ms/step | 5.7 samples/s
514
+ Step 39750/44000 | Loss: 2.8698 | LR: 1.18e-05 | 3483.7ms/step | 5.7 samples/s
515
+ Validation at step 39750: Loss = 2.8832, Perplexity = 17.87
516
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-39750
517
+ Step 39760/44000 | Loss: 2.8755 | LR: 1.18e-05 | 9607.6ms/step | 2.1 samples/s
518
+ Step 39770/44000 | Loss: 2.8980 | LR: 1.17e-05 | 3481.1ms/step | 5.7 samples/s
519
+ Step 39780/44000 | Loss: 2.8935 | LR: 1.17e-05 | 3484.8ms/step | 5.7 samples/s
520
+ Step 39790/44000 | Loss: 2.8774 | LR: 1.16e-05 | 3484.2ms/step | 5.7 samples/s
521
+ Step 39800/44000 | Loss: 2.8904 | LR: 1.15e-05 | 3482.8ms/step | 5.7 samples/s
522
+ Step 39810/44000 | Loss: 2.8991 | LR: 1.15e-05 | 3483.4ms/step | 5.7 samples/s
523
+ Step 39820/44000 | Loss: 2.8971 | LR: 1.14e-05 | 3484.4ms/step | 5.7 samples/s
524
+ Step 39830/44000 | Loss: 2.8814 | LR: 1.14e-05 | 3484.2ms/step | 5.7 samples/s
525
+ Step 39840/44000 | Loss: 2.8834 | LR: 1.13e-05 | 3486.0ms/step | 5.7 samples/s
526
+ Step 39850/44000 | Loss: 2.8838 | LR: 1.13e-05 | 3483.9ms/step | 5.7 samples/s
527
+ Step 39860/44000 | Loss: 2.9253 | LR: 1.12e-05 | 3484.4ms/step | 5.7 samples/s
528
+ Step 39870/44000 | Loss: 2.8904 | LR: 1.12e-05 | 3485.3ms/step | 5.7 samples/s
529
+ Step 39880/44000 | Loss: 2.8586 | LR: 1.11e-05 | 3484.9ms/step | 5.7 samples/s
530
+ Step 39890/44000 | Loss: 2.8736 | LR: 1.11e-05 | 3483.2ms/step | 5.7 samples/s
531
+ Step 39900/44000 | Loss: 2.9040 | LR: 1.10e-05 | 3484.8ms/step | 5.7 samples/s
532
+ Step 39910/44000 | Loss: 2.8947 | LR: 1.10e-05 | 3483.0ms/step | 5.7 samples/s
533
+ Step 39920/44000 | Loss: 2.8965 | LR: 1.09e-05 | 3484.5ms/step | 5.7 samples/s
534
+ Step 39930/44000 | Loss: 2.8857 | LR: 1.08e-05 | 3486.4ms/step | 5.7 samples/s
535
+ Step 39940/44000 | Loss: 2.8994 | LR: 1.08e-05 | 3484.7ms/step | 5.7 samples/s
536
+ Step 39950/44000 | Loss: 2.8931 | LR: 1.07e-05 | 3485.9ms/step | 5.7 samples/s
537
+ Step 39960/44000 | Loss: 2.8916 | LR: 1.07e-05 | 3486.7ms/step | 5.7 samples/s
538
+ Step 39970/44000 | Loss: 2.8983 | LR: 1.06e-05 | 3484.8ms/step | 5.7 samples/s
539
+ Step 39980/44000 | Loss: 2.8696 | LR: 1.06e-05 | 3485.0ms/step | 5.7 samples/s
540
+ Step 39990/44000 | Loss: 2.8999 | LR: 1.05e-05 | 3483.9ms/step | 5.7 samples/s
541
+ Step 40000/44000 | Loss: 2.8894 | LR: 1.05e-05 | 3484.1ms/step | 5.7 samples/s
542
+ Validation at step 40000: Loss = 2.8832, Perplexity = 17.87
543
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-40000
544
+ Step 40010/44000 | Loss: 2.8711 | LR: 1.04e-05 | 9611.1ms/step | 2.1 samples/s
545
+ Step 40020/44000 | Loss: 2.8946 | LR: 1.04e-05 | 3485.0ms/step | 5.7 samples/s
546
+ Step 40030/44000 | Loss: 2.9016 | LR: 1.03e-05 | 3483.1ms/step | 5.7 samples/s
547
+ Step 40040/44000 | Loss: 2.9009 | LR: 1.03e-05 | 3485.4ms/step | 5.7 samples/s
548
+ Step 40050/44000 | Loss: 2.8969 | LR: 1.02e-05 | 3485.0ms/step | 5.7 samples/s
549
+ Step 40060/44000 | Loss: 2.9073 | LR: 1.02e-05 | 3485.8ms/step | 5.7 samples/s
550
+ Step 40070/44000 | Loss: 2.9120 | LR: 1.01e-05 | 3486.7ms/step | 5.7 samples/s
551
+ Step 40080/44000 | Loss: 2.8681 | LR: 1.01e-05 | 3484.8ms/step | 5.7 samples/s
552
+ Step 40090/44000 | Loss: 2.9100 | LR: 1.00e-05 | 3486.6ms/step | 5.7 samples/s
553
+ Step 40100/44000 | Loss: 2.8996 | LR: 9.96e-06 | 3484.1ms/step | 5.7 samples/s
554
+ Step 40110/44000 | Loss: 2.9083 | LR: 9.91e-06 | 3483.4ms/step | 5.7 samples/s
555
+ Step 40120/44000 | Loss: 2.9290 | LR: 9.86e-06 | 3483.3ms/step | 5.7 samples/s
556
+ Step 40130/44000 | Loss: 2.9192 | LR: 9.81e-06 | 3483.1ms/step | 5.7 samples/s
557
+ Step 40140/44000 | Loss: 2.8755 | LR: 9.76e-06 | 3483.1ms/step | 5.7 samples/s
558
+ Step 40150/44000 | Loss: 2.8963 | LR: 9.71e-06 | 3483.8ms/step | 5.7 samples/s
559
+ Step 40160/44000 | Loss: 2.8998 | LR: 9.66e-06 | 3492.3ms/step | 5.7 samples/s
560
+ Step 40170/44000 | Loss: 2.8726 | LR: 9.61e-06 | 3486.3ms/step | 5.7 samples/s
561
+ Step 40180/44000 | Loss: 2.8909 | LR: 9.56e-06 | 3486.9ms/step | 5.7 samples/s
562
+ Step 40190/44000 | Loss: 2.8781 | LR: 9.51e-06 | 3486.4ms/step | 5.7 samples/s
563
+ Step 40200/44000 | Loss: 2.8950 | LR: 9.46e-06 | 3486.5ms/step | 5.7 samples/s
564
+ Step 40210/44000 | Loss: 2.8784 | LR: 9.41e-06 | 3485.8ms/step | 5.7 samples/s
565
+ Step 40220/44000 | Loss: 2.8685 | LR: 9.36e-06 | 3485.7ms/step | 5.7 samples/s
566
+ Step 40230/44000 | Loss: 2.8918 | LR: 9.32e-06 | 3484.9ms/step | 5.7 samples/s
567
+ Step 40240/44000 | Loss: 2.8959 | LR: 9.27e-06 | 3483.2ms/step | 5.7 samples/s
568
+ Step 40250/44000 | Loss: 2.9174 | LR: 9.22e-06 | 3483.5ms/step | 5.7 samples/s
569
+ Validation at step 40250: Loss = 2.8831, Perplexity = 17.87
570
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-40250
571
+ Step 40260/44000 | Loss: 2.8989 | LR: 9.17e-06 | 9609.2ms/step | 2.1 samples/s
572
+ Step 40270/44000 | Loss: 2.8707 | LR: 9.12e-06 | 3482.6ms/step | 5.7 samples/s
573
+ Step 40280/44000 | Loss: 2.8754 | LR: 9.07e-06 | 3485.5ms/step | 5.7 samples/s
574
+ Step 40290/44000 | Loss: 2.8729 | LR: 9.02e-06 | 3484.0ms/step | 5.7 samples/s
575
+ Step 40300/44000 | Loss: 2.9022 | LR: 8.97e-06 | 3485.3ms/step | 5.7 samples/s
576
+ Step 40310/44000 | Loss: 2.9069 | LR: 8.93e-06 | 3483.8ms/step | 5.7 samples/s
577
+ Step 40320/44000 | Loss: 2.8887 | LR: 8.88e-06 | 3482.7ms/step | 5.7 samples/s
578
+ Step 40330/44000 | Loss: 2.9156 | LR: 8.83e-06 | 3483.1ms/step | 5.7 samples/s
579
+ Step 40340/44000 | Loss: 2.9255 | LR: 8.78e-06 | 3482.5ms/step | 5.7 samples/s
580
+ Step 40350/44000 | Loss: 2.8976 | LR: 8.74e-06 | 3482.9ms/step | 5.7 samples/s
581
+ Step 40360/44000 | Loss: 2.8709 | LR: 8.69e-06 | 3484.0ms/step | 5.7 samples/s
582
+ Step 40370/44000 | Loss: 2.8760 | LR: 8.64e-06 | 3483.2ms/step | 5.7 samples/s
583
+ Step 40380/44000 | Loss: 2.8951 | LR: 8.59e-06 | 3483.7ms/step | 5.7 samples/s
584
+ Step 40390/44000 | Loss: 2.8623 | LR: 8.55e-06 | 3485.0ms/step | 5.7 samples/s
585
+ Step 40400/44000 | Loss: 2.9327 | LR: 8.50e-06 | 3485.2ms/step | 5.7 samples/s
586
+ Step 40410/44000 | Loss: 2.8911 | LR: 8.45e-06 | 3483.8ms/step | 5.7 samples/s
587
+ Step 40420/44000 | Loss: 2.8816 | LR: 8.41e-06 | 3485.5ms/step | 5.7 samples/s
588
+ Step 40430/44000 | Loss: 2.8756 | LR: 8.36e-06 | 3483.3ms/step | 5.7 samples/s
589
+ Step 40440/44000 | Loss: 2.8881 | LR: 8.31e-06 | 3484.5ms/step | 5.7 samples/s
590
+ Step 40450/44000 | Loss: 2.8906 | LR: 8.27e-06 | 3483.5ms/step | 5.7 samples/s
591
+ Step 40460/44000 | Loss: 2.9180 | LR: 8.22e-06 | 3483.2ms/step | 5.7 samples/s
592
+ Step 40470/44000 | Loss: 2.8971 | LR: 8.17e-06 | 3485.6ms/step | 5.7 samples/s
593
+ Step 40480/44000 | Loss: 2.8870 | LR: 8.13e-06 | 3484.9ms/step | 5.7 samples/s
594
+ Step 40490/44000 | Loss: 2.9084 | LR: 8.08e-06 | 3483.4ms/step | 5.7 samples/s
595
+ Step 40500/44000 | Loss: 2.8918 | LR: 8.04e-06 | 3484.7ms/step | 5.7 samples/s
596
+ Validation at step 40500: Loss = 2.8831, Perplexity = 17.87
597
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-40500
598
+ Step 40510/44000 | Loss: 2.8463 | LR: 7.99e-06 | 9622.1ms/step | 2.1 samples/s
599
+ Step 40520/44000 | Loss: 2.8946 | LR: 7.94e-06 | 3485.0ms/step | 5.7 samples/s
600
+ Step 40530/44000 | Loss: 2.8961 | LR: 7.90e-06 | 3484.4ms/step | 5.7 samples/s
601
+ Step 40540/44000 | Loss: 2.8742 | LR: 7.85e-06 | 3491.9ms/step | 5.7 samples/s
602
+ Step 40550/44000 | Loss: 2.9054 | LR: 7.81e-06 | 3488.9ms/step | 5.7 samples/s
603
+ Step 40560/44000 | Loss: 2.9042 | LR: 7.76e-06 | 3494.4ms/step | 5.7 samples/s
604
+ Step 40570/44000 | Loss: 2.8808 | LR: 7.72e-06 | 3490.7ms/step | 5.7 samples/s
605
+ Step 40580/44000 | Loss: 2.8857 | LR: 7.67e-06 | 3490.5ms/step | 5.7 samples/s
606
+ Step 40590/44000 | Loss: 2.8836 | LR: 7.63e-06 | 3489.0ms/step | 5.7 samples/s
607
+ Step 40600/44000 | Loss: 2.8833 | LR: 7.59e-06 | 3487.9ms/step | 5.7 samples/s
608
+ Step 40610/44000 | Loss: 2.8778 | LR: 7.54e-06 | 3487.6ms/step | 5.7 samples/s
609
+ Step 40620/44000 | Loss: 2.8790 | LR: 7.50e-06 | 3489.3ms/step | 5.7 samples/s
610
+ Step 40630/44000 | Loss: 2.8953 | LR: 7.45e-06 | 3492.2ms/step | 5.7 samples/s
611
+ Step 40640/44000 | Loss: 2.8675 | LR: 7.41e-06 | 3489.7ms/step | 5.7 samples/s
612
+ Step 40650/44000 | Loss: 2.8880 | LR: 7.37e-06 | 3489.0ms/step | 5.7 samples/s
613
+ Step 40660/44000 | Loss: 2.9317 | LR: 7.32e-06 | 3490.7ms/step | 5.7 samples/s
614
+ Step 40670/44000 | Loss: 2.9281 | LR: 7.28e-06 | 3490.8ms/step | 5.7 samples/s
615
+ Step 40680/44000 | Loss: 2.8747 | LR: 7.23e-06 | 3492.4ms/step | 5.7 samples/s
616
+ Step 40690/44000 | Loss: 2.8529 | LR: 7.19e-06 | 3491.2ms/step | 5.7 samples/s
617
+ Step 40700/44000 | Loss: 2.8981 | LR: 7.15e-06 | 3491.1ms/step | 5.7 samples/s
618
+ Step 40710/44000 | Loss: 2.8728 | LR: 7.10e-06 | 3492.3ms/step | 5.7 samples/s
619
+ Step 40720/44000 | Loss: 2.8887 | LR: 7.06e-06 | 3493.2ms/step | 5.7 samples/s
620
+ Step 40730/44000 | Loss: 2.8948 | LR: 7.02e-06 | 3491.4ms/step | 5.7 samples/s
621
+ Step 40740/44000 | Loss: 2.8951 | LR: 6.98e-06 | 3492.3ms/step | 5.7 samples/s
622
+ Step 40750/44000 | Loss: 2.8907 | LR: 6.93e-06 | 3492.0ms/step | 5.7 samples/s
623
+ Validation at step 40750: Loss = 2.8831, Perplexity = 17.87
624
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-40750
625
+ Step 40760/44000 | Loss: 2.9217 | LR: 6.89e-06 | 9640.1ms/step | 2.1 samples/s
626
+ Step 40770/44000 | Loss: 2.8946 | LR: 6.85e-06 | 3490.9ms/step | 5.7 samples/s
627
+ Step 40780/44000 | Loss: 2.8801 | LR: 6.81e-06 | 3491.7ms/step | 5.7 samples/s
628
+ Step 40790/44000 | Loss: 2.8758 | LR: 6.77e-06 | 3490.4ms/step | 5.7 samples/s
629
+ Step 40800/44000 | Loss: 2.8776 | LR: 6.72e-06 | 3492.1ms/step | 5.7 samples/s
630
+ Step 40810/44000 | Loss: 2.8867 | LR: 6.68e-06 | 3490.3ms/step | 5.7 samples/s
631
+ Step 40820/44000 | Loss: 2.8973 | LR: 6.64e-06 | 3491.3ms/step | 5.7 samples/s
632
+ Step 40830/44000 | Loss: 2.8901 | LR: 6.60e-06 | 3491.9ms/step | 5.7 samples/s
633
+ Step 40840/44000 | Loss: 2.9101 | LR: 6.56e-06 | 3491.6ms/step | 5.7 samples/s
634
+ Step 40850/44000 | Loss: 2.9006 | LR: 6.52e-06 | 3491.1ms/step | 5.7 samples/s
635
+ Step 40860/44000 | Loss: 2.8882 | LR: 6.47e-06 | 3493.7ms/step | 5.7 samples/s
636
+ Step 40870/44000 | Loss: 2.8901 | LR: 6.43e-06 | 3494.2ms/step | 5.7 samples/s
637
+ Step 40880/44000 | Loss: 2.8366 | LR: 6.39e-06 | 3492.8ms/step | 5.7 samples/s
638
+ Step 40890/44000 | Loss: 2.8652 | LR: 6.35e-06 | 3494.1ms/step | 5.7 samples/s
639
+ Step 40900/44000 | Loss: 2.8771 | LR: 6.31e-06 | 3491.4ms/step | 5.7 samples/s
640
+ Step 40910/44000 | Loss: 2.9002 | LR: 6.27e-06 | 3493.6ms/step | 5.7 samples/s
641
+ Step 40920/44000 | Loss: 2.8986 | LR: 6.23e-06 | 3492.6ms/step | 5.7 samples/s
642
+ Step 40930/44000 | Loss: 2.9253 | LR: 6.19e-06 | 3490.4ms/step | 5.7 samples/s
643
+ Step 40940/44000 | Loss: 2.8544 | LR: 6.15e-06 | 3491.7ms/step | 5.7 samples/s
644
+ Step 40950/44000 | Loss: 2.8761 | LR: 6.11e-06 | 3491.7ms/step | 5.7 samples/s
645
+ Step 40960/44000 | Loss: 2.8666 | LR: 6.07e-06 | 3491.8ms/step | 5.7 samples/s
646
+ Step 40970/44000 | Loss: 2.8918 | LR: 6.03e-06 | 3491.7ms/step | 5.7 samples/s
647
+ Step 40980/44000 | Loss: 2.8871 | LR: 5.99e-06 | 3490.0ms/step | 5.7 samples/s
648
+ Step 40990/44000 | Loss: 2.8832 | LR: 5.95e-06 | 3491.8ms/step | 5.7 samples/s
649
+ Step 41000/44000 | Loss: 2.8630 | LR: 5.91e-06 | 3493.2ms/step | 5.7 samples/s
650
+ Validation at step 41000: Loss = 2.8831, Perplexity = 17.87
651
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-41000
652
+ Step 41010/44000 | Loss: 2.8786 | LR: 5.87e-06 | 9646.5ms/step | 2.1 samples/s
653
+ Step 41020/44000 | Loss: 2.8796 | LR: 5.83e-06 | 3494.3ms/step | 5.7 samples/s
654
+ Step 41030/44000 | Loss: 2.8961 | LR: 5.80e-06 | 3491.7ms/step | 5.7 samples/s
655
+ Step 41040/44000 | Loss: 2.9076 | LR: 5.76e-06 | 3493.4ms/step | 5.7 samples/s
656
+ Step 41050/44000 | Loss: 2.8987 | LR: 5.72e-06 | 3494.7ms/step | 5.7 samples/s
657
+ Step 41060/44000 | Loss: 2.8691 | LR: 5.68e-06 | 3498.5ms/step | 5.7 samples/s
658
+ Step 41070/44000 | Loss: 2.8707 | LR: 5.64e-06 | 3495.2ms/step | 5.7 samples/s
659
+ Step 41080/44000 | Loss: 2.8994 | LR: 5.60e-06 | 3495.6ms/step | 5.7 samples/s
660
+ Step 41090/44000 | Loss: 2.9168 | LR: 5.56e-06 | 3495.3ms/step | 5.7 samples/s
661
+ Step 41100/44000 | Loss: 2.8824 | LR: 5.53e-06 | 3495.7ms/step | 5.7 samples/s
662
+ Step 41110/44000 | Loss: 2.8715 | LR: 5.49e-06 | 3494.3ms/step | 5.7 samples/s
663
+ Step 41120/44000 | Loss: 2.8963 | LR: 5.45e-06 | 3493.9ms/step | 5.7 samples/s
664
+ Step 41130/44000 | Loss: 2.8914 | LR: 5.41e-06 | 3493.3ms/step | 5.7 samples/s
665
+ Step 41140/44000 | Loss: 2.8684 | LR: 5.38e-06 | 3494.6ms/step | 5.7 samples/s
666
+ Step 41150/44000 | Loss: 2.8977 | LR: 5.34e-06 | 3495.3ms/step | 5.7 samples/s
667
+ Step 41160/44000 | Loss: 2.8837 | LR: 5.30e-06 | 3498.8ms/step | 5.7 samples/s
668
+ Step 41170/44000 | Loss: 2.8752 | LR: 5.26e-06 | 3498.1ms/step | 5.7 samples/s
669
+ Step 41180/44000 | Loss: 2.8786 | LR: 5.23e-06 | 3492.9ms/step | 5.7 samples/s
670
+ Step 41190/44000 | Loss: 2.9106 | LR: 5.19e-06 | 3492.5ms/step | 5.7 samples/s
671
+ Step 41200/44000 | Loss: 2.8774 | LR: 5.15e-06 | 3491.8ms/step | 5.7 samples/s
672
+ Step 41210/44000 | Loss: 2.8913 | LR: 5.12e-06 | 3493.4ms/step | 5.7 samples/s
673
+ Step 41220/44000 | Loss: 2.8829 | LR: 5.08e-06 | 3493.9ms/step | 5.7 samples/s
674
+ Step 41230/44000 | Loss: 2.8789 | LR: 5.04e-06 | 3491.8ms/step | 5.7 samples/s
675
+ Step 41240/44000 | Loss: 2.8995 | LR: 5.01e-06 | 3493.2ms/step | 5.7 samples/s
676
+ Step 41250/44000 | Loss: 2.8753 | LR: 4.97e-06 | 3492.2ms/step | 5.7 samples/s
677
+ Validation at step 41250: Loss = 2.8831, Perplexity = 17.87
678
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-41250
679
+ Step 41260/44000 | Loss: 2.9108 | LR: 4.94e-06 | 9624.3ms/step | 2.1 samples/s
680
+ Step 41270/44000 | Loss: 2.8515 | LR: 4.90e-06 | 3491.5ms/step | 5.7 samples/s
681
+ Step 41280/44000 | Loss: 2.9102 | LR: 4.86e-06 | 3493.2ms/step | 5.7 samples/s
682
+ Step 41290/44000 | Loss: 2.9076 | LR: 4.83e-06 | 3494.5ms/step | 5.7 samples/s
683
+ Step 41300/44000 | Loss: 2.9040 | LR: 4.79e-06 | 3493.1ms/step | 5.7 samples/s
684
+ Step 41310/44000 | Loss: 2.8890 | LR: 4.76e-06 | 3493.3ms/step | 5.7 samples/s
685
+ Step 41320/44000 | Loss: 2.9179 | LR: 4.72e-06 | 3493.7ms/step | 5.7 samples/s
686
+ Step 41330/44000 | Loss: 2.8643 | LR: 4.69e-06 | 3494.0ms/step | 5.7 samples/s
687
+ Step 41340/44000 | Loss: 2.9125 | LR: 4.65e-06 | 3493.2ms/step | 5.7 samples/s
688
+ Step 41350/44000 | Loss: 2.9172 | LR: 4.62e-06 | 3493.4ms/step | 5.7 samples/s
689
+ Step 41360/44000 | Loss: 2.8678 | LR: 4.58e-06 | 3493.5ms/step | 5.7 samples/s
690
+ Step 41370/44000 | Loss: 2.9016 | LR: 4.55e-06 | 3490.9ms/step | 5.7 samples/s
691
+ Step 41380/44000 | Loss: 2.8883 | LR: 4.51e-06 | 3492.2ms/step | 5.7 samples/s
692
+ Step 41390/44000 | Loss: 2.8865 | LR: 4.48e-06 | 3494.3ms/step | 5.7 samples/s
693
+ Step 41400/44000 | Loss: 2.8952 | LR: 4.45e-06 | 3491.8ms/step | 5.7 samples/s
694
+ Step 41410/44000 | Loss: 2.9331 | LR: 4.41e-06 | 3491.8ms/step | 5.7 samples/s
695
+ Step 41420/44000 | Loss: 2.9031 | LR: 4.38e-06 | 3491.7ms/step | 5.7 samples/s
696
+ Step 41430/44000 | Loss: 2.8724 | LR: 4.34e-06 | 3493.0ms/step | 5.7 samples/s
697
+ Step 41440/44000 | Loss: 2.8795 | LR: 4.31e-06 | 3492.6ms/step | 5.7 samples/s
698
+ Step 41450/44000 | Loss: 2.8866 | LR: 4.28e-06 | 3492.1ms/step | 5.7 samples/s
699
+ Step 41460/44000 | Loss: 2.8850 | LR: 4.24e-06 | 3493.9ms/step | 5.7 samples/s
700
+ Step 41470/44000 | Loss: 2.9187 | LR: 4.21e-06 | 3495.4ms/step | 5.7 samples/s
701
+ Step 41480/44000 | Loss: 2.8955 | LR: 4.18e-06 | 3492.2ms/step | 5.7 samples/s
702
+ Step 41490/44000 | Loss: 2.9034 | LR: 4.14e-06 | 3495.6ms/step | 5.7 samples/s
703
+ Step 41500/44000 | Loss: 2.8665 | LR: 4.11e-06 | 3496.4ms/step | 5.7 samples/s
704
+ Validation at step 41500: Loss = 2.8831, Perplexity = 17.87
705
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-41500
706
+ Step 41510/44000 | Loss: 2.9036 | LR: 4.08e-06 | 9656.6ms/step | 2.1 samples/s
707
+ Step 41520/44000 | Loss: 2.8876 | LR: 4.05e-06 | 3497.2ms/step | 5.7 samples/s
708
+ Step 41530/44000 | Loss: 2.9088 | LR: 4.01e-06 | 3496.2ms/step | 5.7 samples/s
709
+ Step 41540/44000 | Loss: 2.9023 | LR: 3.98e-06 | 3500.7ms/step | 5.7 samples/s
710
+ Step 41550/44000 | Loss: 2.8724 | LR: 3.95e-06 | 3499.9ms/step | 5.7 samples/s
711
+ Step 41560/44000 | Loss: 2.8970 | LR: 3.92e-06 | 3499.0ms/step | 5.7 samples/s
712
+ Step 41570/44000 | Loss: 2.8811 | LR: 3.88e-06 | 3498.1ms/step | 5.7 samples/s
713
+ Step 41580/44000 | Loss: 2.8611 | LR: 3.85e-06 | 3496.0ms/step | 5.7 samples/s
714
+ Step 41590/44000 | Loss: 2.9024 | LR: 3.82e-06 | 3495.6ms/step | 5.7 samples/s
715
+ Step 41600/44000 | Loss: 2.8520 | LR: 3.79e-06 | 3494.2ms/step | 5.7 samples/s
716
+ Step 41610/44000 | Loss: 2.8747 | LR: 3.76e-06 | 3495.8ms/step | 5.7 samples/s
717
+ Step 41620/44000 | Loss: 2.8906 | LR: 3.73e-06 | 3497.0ms/step | 5.7 samples/s
718
+ Step 41630/44000 | Loss: 2.8626 | LR: 3.70e-06 | 3497.4ms/step | 5.7 samples/s
719
+ Step 41640/44000 | Loss: 2.9024 | LR: 3.66e-06 | 3497.4ms/step | 5.7 samples/s
720
+ Step 41650/44000 | Loss: 2.8858 | LR: 3.63e-06 | 3496.5ms/step | 5.7 samples/s
721
+ Step 41660/44000 | Loss: 2.8855 | LR: 3.60e-06 | 3495.2ms/step | 5.7 samples/s
722
+ Step 41670/44000 | Loss: 2.9143 | LR: 3.57e-06 | 3496.7ms/step | 5.7 samples/s
723
+ Step 41680/44000 | Loss: 2.8926 | LR: 3.54e-06 | 3496.5ms/step | 5.7 samples/s
724
+ Step 41690/44000 | Loss: 2.8779 | LR: 3.51e-06 | 3496.9ms/step | 5.7 samples/s
725
+ Step 41700/44000 | Loss: 2.8641 | LR: 3.48e-06 | 3494.0ms/step | 5.7 samples/s
726
+ Step 41710/44000 | Loss: 2.8758 | LR: 3.45e-06 | 3496.7ms/step | 5.7 samples/s
727
+ Step 41720/44000 | Loss: 2.9159 | LR: 3.42e-06 | 3495.6ms/step | 5.7 samples/s
728
+ Step 41730/44000 | Loss: 2.9122 | LR: 3.39e-06 | 3494.5ms/step | 5.7 samples/s
729
+ Step 41740/44000 | Loss: 2.9358 | LR: 3.36e-06 | 3496.4ms/step | 5.7 samples/s
730
+ Step 41750/44000 | Loss: 2.8778 | LR: 3.33e-06 | 3498.9ms/step | 5.7 samples/s
731
+ Validation at step 41750: Loss = 2.8831, Perplexity = 17.87
732
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-41750
733
+ Step 41760/44000 | Loss: 2.8901 | LR: 3.30e-06 | 9644.8ms/step | 2.1 samples/s
734
+ Step 41770/44000 | Loss: 2.8606 | LR: 3.27e-06 | 3491.4ms/step | 5.7 samples/s
735
+ Step 41780/44000 | Loss: 2.8800 | LR: 3.24e-06 | 3494.6ms/step | 5.7 samples/s
736
+ Step 41790/44000 | Loss: 2.8897 | LR: 3.21e-06 | 3497.1ms/step | 5.7 samples/s
737
+ Step 41800/44000 | Loss: 2.9381 | LR: 3.19e-06 | 3497.4ms/step | 5.7 samples/s
738
+ Step 41810/44000 | Loss: 2.9012 | LR: 3.16e-06 | 3496.4ms/step | 5.7 samples/s
739
+ Step 41820/44000 | Loss: 2.9242 | LR: 3.13e-06 | 3498.0ms/step | 5.7 samples/s
740
+ Step 41830/44000 | Loss: 2.8639 | LR: 3.10e-06 | 3495.7ms/step | 5.7 samples/s
741
+ Step 41840/44000 | Loss: 2.8912 | LR: 3.07e-06 | 3497.2ms/step | 5.7 samples/s
742
+ Step 41850/44000 | Loss: 2.9107 | LR: 3.04e-06 | 3495.9ms/step | 5.7 samples/s
743
+ Step 41860/44000 | Loss: 2.8773 | LR: 3.01e-06 | 3496.6ms/step | 5.7 samples/s
744
+ Step 41870/44000 | Loss: 2.8778 | LR: 2.99e-06 | 3499.1ms/step | 5.7 samples/s
745
+ Step 41880/44000 | Loss: 2.8550 | LR: 2.96e-06 | 3497.9ms/step | 5.7 samples/s
746
+ Step 41890/44000 | Loss: 2.9023 | LR: 2.93e-06 | 3497.7ms/step | 5.7 samples/s
747
+ Step 41900/44000 | Loss: 2.8822 | LR: 2.90e-06 | 3496.8ms/step | 5.7 samples/s
748
+ Step 41910/44000 | Loss: 2.8786 | LR: 2.88e-06 | 3496.3ms/step | 5.7 samples/s
749
+ Step 41920/44000 | Loss: 2.8909 | LR: 2.85e-06 | 3497.4ms/step | 5.7 samples/s
750
+ Step 41930/44000 | Loss: 2.8714 | LR: 2.82e-06 | 3498.8ms/step | 5.7 samples/s
751
+ Step 41940/44000 | Loss: 2.9158 | LR: 2.79e-06 | 3498.0ms/step | 5.7 samples/s
752
+ Step 41950/44000 | Loss: 2.8828 | LR: 2.77e-06 | 3496.6ms/step | 5.7 samples/s
753
+ Step 41960/44000 | Loss: 2.8666 | LR: 2.74e-06 | 3498.0ms/step | 5.7 samples/s
754
+ Step 41970/44000 | Loss: 2.8872 | LR: 2.71e-06 | 3495.3ms/step | 5.7 samples/s
755
+ Step 41980/44000 | Loss: 2.9009 | LR: 2.69e-06 | 3494.8ms/step | 5.7 samples/s
756
+ Step 41990/44000 | Loss: 2.9238 | LR: 2.66e-06 | 3497.3ms/step | 5.7 samples/s
757
+ Step 42000/44000 | Loss: 2.8611 | LR: 2.63e-06 | 3497.4ms/step | 5.7 samples/s
758
+ Validation at step 42000: Loss = 2.8831, Perplexity = 17.87
759
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-42000
760
+ Step 42010/44000 | Loss: 2.9014 | LR: 2.61e-06 | 9652.2ms/step | 2.1 samples/s
761
+ Step 42020/44000 | Loss: 2.8962 | LR: 2.58e-06 | 3497.2ms/step | 5.7 samples/s
762
+ Step 42030/44000 | Loss: 2.8840 | LR: 2.56e-06 | 3496.6ms/step | 5.7 samples/s
763
+ Step 42040/44000 | Loss: 2.8856 | LR: 2.53e-06 | 3497.5ms/step | 5.7 samples/s
764
+ Step 42050/44000 | Loss: 2.8866 | LR: 2.50e-06 | 3498.5ms/step | 5.7 samples/s
765
+ Step 42060/44000 | Loss: 2.8926 | LR: 2.48e-06 | 3500.3ms/step | 5.7 samples/s
766
+ Step 42070/44000 | Loss: 2.9142 | LR: 2.45e-06 | 3500.4ms/step | 5.7 samples/s
767
+ Step 42080/44000 | Loss: 2.8713 | LR: 2.43e-06 | 3499.4ms/step | 5.7 samples/s
768
+ Step 42090/44000 | Loss: 2.8864 | LR: 2.40e-06 | 3499.0ms/step | 5.7 samples/s
769
+ Step 42100/44000 | Loss: 2.8656 | LR: 2.38e-06 | 3498.5ms/step | 5.7 samples/s
770
+ Step 42110/44000 | Loss: 2.9106 | LR: 2.35e-06 | 3498.1ms/step | 5.7 samples/s
771
+ Step 42120/44000 | Loss: 2.8917 | LR: 2.33e-06 | 3495.2ms/step | 5.7 samples/s
772
+ Step 42130/44000 | Loss: 2.8964 | LR: 2.30e-06 | 3497.6ms/step | 5.7 samples/s
773
+ Step 42140/44000 | Loss: 2.8948 | LR: 2.28e-06 | 3498.3ms/step | 5.7 samples/s
774
+ Step 42150/44000 | Loss: 2.8585 | LR: 2.25e-06 | 3499.4ms/step | 5.7 samples/s
775
+ Step 42160/44000 | Loss: 2.8651 | LR: 2.23e-06 | 3497.4ms/step | 5.7 samples/s
776
+ Step 42170/44000 | Loss: 2.8698 | LR: 2.21e-06 | 3497.6ms/step | 5.7 samples/s
777
+ Step 42180/44000 | Loss: 2.9088 | LR: 2.18e-06 | 3496.6ms/step | 5.7 samples/s
778
+ Step 42190/44000 | Loss: 2.8785 | LR: 2.16e-06 | 3496.7ms/step | 5.7 samples/s
779
+ Step 42200/44000 | Loss: 2.9224 | LR: 2.13e-06 | 3496.6ms/step | 5.7 samples/s
780
+ Step 42210/44000 | Loss: 2.9218 | LR: 2.11e-06 | 3497.7ms/step | 5.7 samples/s
781
+ Step 42220/44000 | Loss: 2.8882 | LR: 2.09e-06 | 3497.2ms/step | 5.7 samples/s
782
+ Step 42230/44000 | Loss: 2.8997 | LR: 2.06e-06 | 3497.0ms/step | 5.7 samples/s
783
+ Step 42240/44000 | Loss: 2.8989 | LR: 2.04e-06 | 3500.7ms/step | 5.7 samples/s
784
+ Step 42250/44000 | Loss: 2.9059 | LR: 2.02e-06 | 3496.2ms/step | 5.7 samples/s
785
+ Validation at step 42250: Loss = 2.8831, Perplexity = 17.87
786
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-42250
787
+ Step 42260/44000 | Loss: 2.8657 | LR: 1.99e-06 | 9637.1ms/step | 2.1 samples/s
788
+ Step 42270/44000 | Loss: 2.8978 | LR: 1.97e-06 | 3485.9ms/step | 5.7 samples/s
789
+ Step 42280/44000 | Loss: 2.8840 | LR: 1.95e-06 | 3489.1ms/step | 5.7 samples/s
790
+ Step 42290/44000 | Loss: 2.9014 | LR: 1.93e-06 | 3492.6ms/step | 5.7 samples/s
791
+ Step 42300/44000 | Loss: 2.8867 | LR: 1.90e-06 | 3491.3ms/step | 5.7 samples/s
792
+ Step 42310/44000 | Loss: 2.8626 | LR: 1.88e-06 | 3491.5ms/step | 5.7 samples/s
793
+ Step 42320/44000 | Loss: 2.8818 | LR: 1.86e-06 | 3492.4ms/step | 5.7 samples/s
794
+ Step 42330/44000 | Loss: 2.8904 | LR: 1.84e-06 | 3493.5ms/step | 5.7 samples/s
795
+ Step 42340/44000 | Loss: 2.9004 | LR: 1.82e-06 | 3491.9ms/step | 5.7 samples/s
796
+ Step 42350/44000 | Loss: 2.9091 | LR: 1.79e-06 | 3491.4ms/step | 5.7 samples/s
797
+ Step 42360/44000 | Loss: 2.8887 | LR: 1.77e-06 | 3492.4ms/step | 5.7 samples/s
798
+ Step 42370/44000 | Loss: 2.8742 | LR: 1.75e-06 | 3490.5ms/step | 5.7 samples/s
799
+ Step 42380/44000 | Loss: 2.8853 | LR: 1.73e-06 | 3493.4ms/step | 5.7 samples/s
800
+ Step 42390/44000 | Loss: 2.9140 | LR: 1.71e-06 | 3493.3ms/step | 5.7 samples/s
801
+ Step 42400/44000 | Loss: 2.8998 | LR: 1.69e-06 | 3492.3ms/step | 5.7 samples/s
802
+ Step 42410/44000 | Loss: 2.9029 | LR: 1.67e-06 | 3492.6ms/step | 5.7 samples/s
803
+ Step 42420/44000 | Loss: 2.8677 | LR: 1.64e-06 | 3495.1ms/step | 5.7 samples/s
804
+ Step 42430/44000 | Loss: 2.8714 | LR: 1.62e-06 | 3491.9ms/step | 5.7 samples/s
805
+ Step 42440/44000 | Loss: 2.8911 | LR: 1.60e-06 | 3493.4ms/step | 5.7 samples/s
806
+ Step 42450/44000 | Loss: 2.8805 | LR: 1.58e-06 | 3494.1ms/step | 5.7 samples/s
807
+ Step 42460/44000 | Loss: 2.9008 | LR: 1.56e-06 | 3494.1ms/step | 5.7 samples/s
808
+ Step 42470/44000 | Loss: 2.8842 | LR: 1.54e-06 | 3492.4ms/step | 5.7 samples/s
809
+ Step 42480/44000 | Loss: 2.9052 | LR: 1.52e-06 | 3490.9ms/step | 5.7 samples/s
810
+ Step 42490/44000 | Loss: 2.9148 | LR: 1.50e-06 | 3492.3ms/step | 5.7 samples/s
811
+ Step 42500/44000 | Loss: 2.8751 | LR: 1.48e-06 | 3494.5ms/step | 5.7 samples/s
812
+ Validation at step 42500: Loss = 2.8831, Perplexity = 17.87
813
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-42500
814
+ Step 42510/44000 | Loss: 2.9036 | LR: 1.46e-06 | 9655.6ms/step | 2.1 samples/s
815
+ Step 42520/44000 | Loss: 2.8736 | LR: 1.44e-06 | 3495.5ms/step | 5.7 samples/s
816
+ Step 42530/44000 | Loss: 2.8900 | LR: 1.42e-06 | 3494.2ms/step | 5.7 samples/s
817
+ Step 42540/44000 | Loss: 2.9260 | LR: 1.40e-06 | 3493.8ms/step | 5.7 samples/s
818
+ Step 42550/44000 | Loss: 2.9066 | LR: 1.39e-06 | 3495.7ms/step | 5.7 samples/s
819
+ Step 42560/44000 | Loss: 2.8597 | LR: 1.37e-06 | 3497.9ms/step | 5.7 samples/s
820
+ Step 42570/44000 | Loss: 2.9148 | LR: 1.35e-06 | 3494.8ms/step | 5.7 samples/s
821
+ Step 42580/44000 | Loss: 2.8886 | LR: 1.33e-06 | 3493.4ms/step | 5.7 samples/s
822
+ Step 42590/44000 | Loss: 2.8726 | LR: 1.31e-06 | 3494.0ms/step | 5.7 samples/s
823
+ Step 42600/44000 | Loss: 2.8949 | LR: 1.29e-06 | 3493.3ms/step | 5.7 samples/s
824
+ Step 42610/44000 | Loss: 2.8700 | LR: 1.27e-06 | 3493.7ms/step | 5.7 samples/s
825
+ Step 42620/44000 | Loss: 2.8927 | LR: 1.25e-06 | 3498.6ms/step | 5.7 samples/s
826
+ Step 42630/44000 | Loss: 2.9087 | LR: 1.24e-06 | 3493.3ms/step | 5.7 samples/s
827
+ Step 42640/44000 | Loss: 2.8927 | LR: 1.22e-06 | 3493.6ms/step | 5.7 samples/s
828
+ Step 42650/44000 | Loss: 2.8912 | LR: 1.20e-06 | 3493.7ms/step | 5.7 samples/s
829
+ Step 42660/44000 | Loss: 2.9172 | LR: 1.18e-06 | 3493.0ms/step | 5.7 samples/s
830
+ Step 42670/44000 | Loss: 2.8829 | LR: 1.17e-06 | 3493.7ms/step | 5.7 samples/s
831
+ Step 42680/44000 | Loss: 2.8987 | LR: 1.15e-06 | 3494.2ms/step | 5.7 samples/s
832
+ Step 42690/44000 | Loss: 2.8945 | LR: 1.13e-06 | 3498.2ms/step | 5.7 samples/s
833
+ Step 42700/44000 | Loss: 2.8854 | LR: 1.11e-06 | 3496.4ms/step | 5.7 samples/s
834
+ Step 42710/44000 | Loss: 2.8685 | LR: 1.10e-06 | 3496.6ms/step | 5.7 samples/s
835
+ Step 42720/44000 | Loss: 2.8998 | LR: 1.08e-06 | 3495.3ms/step | 5.7 samples/s
836
+ Step 42730/44000 | Loss: 2.9128 | LR: 1.06e-06 | 3492.6ms/step | 5.7 samples/s
837
+ Step 42740/44000 | Loss: 2.8691 | LR: 1.05e-06 | 3494.6ms/step | 5.7 samples/s
838
+ Step 42750/44000 | Loss: 2.9210 | LR: 1.03e-06 | 3495.6ms/step | 5.7 samples/s
839
+ Validation at step 42750: Loss = 2.8831, Perplexity = 17.87
840
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-42750
841
+ Step 42760/44000 | Loss: 2.8997 | LR: 1.01e-06 | 9643.1ms/step | 2.1 samples/s
842
+ Step 42770/44000 | Loss: 2.8957 | LR: 9.97e-07 | 3494.2ms/step | 5.7 samples/s
843
+ Step 42780/44000 | Loss: 2.8964 | LR: 9.81e-07 | 3494.5ms/step | 5.7 samples/s
844
+ Step 42790/44000 | Loss: 2.8973 | LR: 9.65e-07 | 3494.3ms/step | 5.7 samples/s
845
+ Step 42800/44000 | Loss: 2.8880 | LR: 9.49e-07 | 3494.8ms/step | 5.7 samples/s
846
+ Step 42810/44000 | Loss: 2.8953 | LR: 9.33e-07 | 3495.3ms/step | 5.7 samples/s
847
+ Step 42820/44000 | Loss: 2.8948 | LR: 9.18e-07 | 3494.1ms/step | 5.7 samples/s
848
+ Step 42830/44000 | Loss: 2.8762 | LR: 9.02e-07 | 3494.6ms/step | 5.7 samples/s
849
+ Step 42840/44000 | Loss: 2.9215 | LR: 8.87e-07 | 3494.1ms/step | 5.7 samples/s
850
+ Step 42850/44000 | Loss: 2.8699 | LR: 8.72e-07 | 3495.6ms/step | 5.7 samples/s
851
+ Step 42860/44000 | Loss: 2.9100 | LR: 8.57e-07 | 3496.5ms/step | 5.7 samples/s
852
+ Step 42870/44000 | Loss: 2.8867 | LR: 8.42e-07 | 3496.0ms/step | 5.7 samples/s
853
+ Step 42880/44000 | Loss: 2.9007 | LR: 8.27e-07 | 3493.7ms/step | 5.7 samples/s
854
+ Step 42890/44000 | Loss: 2.8858 | LR: 8.12e-07 | 3495.6ms/step | 5.7 samples/s
855
+ Step 42900/44000 | Loss: 2.8627 | LR: 7.98e-07 | 3494.7ms/step | 5.7 samples/s
856
+ Step 42910/44000 | Loss: 2.9045 | LR: 7.83e-07 | 3492.5ms/step | 5.7 samples/s
857
+ Step 42920/44000 | Loss: 2.8547 | LR: 7.69e-07 | 3493.1ms/step | 5.7 samples/s
858
+ Step 42930/44000 | Loss: 2.8870 | LR: 7.55e-07 | 3490.7ms/step | 5.7 samples/s
859
+ Step 42940/44000 | Loss: 2.8963 | LR: 7.41e-07 | 3491.7ms/step | 5.7 samples/s
860
+ Step 42950/44000 | Loss: 2.9096 | LR: 7.27e-07 | 3491.3ms/step | 5.7 samples/s
861
+ Step 42960/44000 | Loss: 2.8745 | LR: 7.13e-07 | 3491.2ms/step | 5.7 samples/s
862
+ Step 42970/44000 | Loss: 2.8757 | LR: 6.99e-07 | 3491.0ms/step | 5.7 samples/s
863
+ Step 42980/44000 | Loss: 2.8906 | LR: 6.86e-07 | 3492.7ms/step | 5.7 samples/s
864
+ Step 42990/44000 | Loss: 2.8565 | LR: 6.72e-07 | 3489.2ms/step | 5.7 samples/s
865
+ Step 43000/44000 | Loss: 2.8960 | LR: 6.59e-07 | 3492.2ms/step | 5.7 samples/s
866
+ Validation at step 43000: Loss = 2.8831, Perplexity = 17.87
867
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-43000
868
+ Step 43010/44000 | Loss: 2.9211 | LR: 6.46e-07 | 9632.2ms/step | 2.1 samples/s
869
+ Step 43020/44000 | Loss: 2.9053 | LR: 6.33e-07 | 3490.2ms/step | 5.7 samples/s
870
+ Step 43030/44000 | Loss: 2.9053 | LR: 6.20e-07 | 3489.1ms/step | 5.7 samples/s
871
+ Step 43040/44000 | Loss: 2.8942 | LR: 6.08e-07 | 3491.0ms/step | 5.7 samples/s
872
+ Step 43050/44000 | Loss: 2.9064 | LR: 5.95e-07 | 3493.0ms/step | 5.7 samples/s
873
+ Step 43060/44000 | Loss: 2.9091 | LR: 5.83e-07 | 3492.8ms/step | 5.7 samples/s
874
+ Step 43070/44000 | Loss: 2.9063 | LR: 5.70e-07 | 3491.8ms/step | 5.7 samples/s
875
+ Step 43080/44000 | Loss: 2.8723 | LR: 5.58e-07 | 3493.9ms/step | 5.7 samples/s
876
+ Step 43090/44000 | Loss: 2.8705 | LR: 5.46e-07 | 3493.3ms/step | 5.7 samples/s
877
+ Step 43100/44000 | Loss: 2.8867 | LR: 5.34e-07 | 3492.4ms/step | 5.7 samples/s
878
+ Step 43110/44000 | Loss: 2.9175 | LR: 5.22e-07 | 3493.5ms/step | 5.7 samples/s
879
+ Step 43120/44000 | Loss: 2.8920 | LR: 5.11e-07 | 3492.9ms/step | 5.7 samples/s
880
+ Step 43130/44000 | Loss: 2.9028 | LR: 4.99e-07 | 3492.6ms/step | 5.7 samples/s
881
+ Step 43140/44000 | Loss: 2.9258 | LR: 4.88e-07 | 3491.9ms/step | 5.7 samples/s
882
+ Step 43150/44000 | Loss: 2.8738 | LR: 4.76e-07 | 3493.5ms/step | 5.7 samples/s
883
+ Step 43160/44000 | Loss: 2.9041 | LR: 4.65e-07 | 3493.7ms/step | 5.7 samples/s
884
+ Step 43170/44000 | Loss: 2.8377 | LR: 4.54e-07 | 3493.1ms/step | 5.7 samples/s
885
+ Step 43180/44000 | Loss: 2.8772 | LR: 4.43e-07 | 3492.9ms/step | 5.7 samples/s
886
+ Step 43190/44000 | Loss: 2.9079 | LR: 4.33e-07 | 3495.1ms/step | 5.7 samples/s
887
+ Step 43200/44000 | Loss: 2.9091 | LR: 4.22e-07 | 3493.5ms/step | 5.7 samples/s
888
+ Step 43210/44000 | Loss: 2.8875 | LR: 4.12e-07 | 3493.7ms/step | 5.7 samples/s
889
+ Step 43220/44000 | Loss: 2.8775 | LR: 4.01e-07 | 3493.0ms/step | 5.7 samples/s
890
+ Step 43230/44000 | Loss: 2.9038 | LR: 3.91e-07 | 3492.0ms/step | 5.7 samples/s
891
+ Step 43240/44000 | Loss: 2.8876 | LR: 3.81e-07 | 3495.3ms/step | 5.7 samples/s
892
+ Step 43250/44000 | Loss: 2.8886 | LR: 3.71e-07 | 3493.8ms/step | 5.7 samples/s
893
+ Validation at step 43250: Loss = 2.8831, Perplexity = 17.87
894
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-43250
895
+ Step 43260/44000 | Loss: 2.8724 | LR: 3.61e-07 | 9633.3ms/step | 2.1 samples/s
896
+ Step 43270/44000 | Loss: 2.9068 | LR: 3.51e-07 | 3491.0ms/step | 5.7 samples/s
897
+ Step 43280/44000 | Loss: 2.8992 | LR: 3.42e-07 | 3489.7ms/step | 5.7 samples/s
898
+ Step 43290/44000 | Loss: 2.9023 | LR: 3.32e-07 | 3491.4ms/step | 5.7 samples/s
899
+ Step 43300/44000 | Loss: 2.8868 | LR: 3.23e-07 | 3491.6ms/step | 5.7 samples/s
900
+ Step 43310/44000 | Loss: 2.8865 | LR: 3.14e-07 | 3490.8ms/step | 5.7 samples/s
901
+ Step 43320/44000 | Loss: 2.8771 | LR: 3.05e-07 | 3490.3ms/step | 5.7 samples/s
902
+ Step 43330/44000 | Loss: 2.8708 | LR: 2.96e-07 | 3491.3ms/step | 5.7 samples/s
903
+ Step 43340/44000 | Loss: 2.8899 | LR: 2.87e-07 | 3491.2ms/step | 5.7 samples/s
904
+ Step 43350/44000 | Loss: 2.8931 | LR: 2.79e-07 | 3490.5ms/step | 5.7 samples/s
905
+ Step 43360/44000 | Loss: 2.8930 | LR: 2.70e-07 | 3491.0ms/step | 5.7 samples/s
906
+ Step 43370/44000 | Loss: 2.8990 | LR: 2.62e-07 | 3493.1ms/step | 5.7 samples/s
907
+ Step 43380/44000 | Loss: 2.8967 | LR: 2.53e-07 | 3494.2ms/step | 5.7 samples/s
908
+ Step 43390/44000 | Loss: 2.8949 | LR: 2.45e-07 | 3490.3ms/step | 5.7 samples/s
909
+ Step 43400/44000 | Loss: 2.8900 | LR: 2.37e-07 | 3491.2ms/step | 5.7 samples/s
910
+ Step 43410/44000 | Loss: 2.8871 | LR: 2.30e-07 | 3492.2ms/step | 5.7 samples/s
911
+ Step 43420/44000 | Loss: 2.9153 | LR: 2.22e-07 | 3491.8ms/step | 5.7 samples/s
912
+ Step 43430/44000 | Loss: 2.8938 | LR: 2.14e-07 | 3493.7ms/step | 5.7 samples/s
913
+ Step 43440/44000 | Loss: 2.8805 | LR: 2.07e-07 | 3491.8ms/step | 5.7 samples/s
914
+ Step 43450/44000 | Loss: 2.8853 | LR: 1.99e-07 | 3493.3ms/step | 5.7 samples/s
915
+ Step 43460/44000 | Loss: 2.8719 | LR: 1.92e-07 | 3490.8ms/step | 5.7 samples/s
916
+ Step 43470/44000 | Loss: 2.8872 | LR: 1.85e-07 | 3490.9ms/step | 5.7 samples/s
917
+ Step 43480/44000 | Loss: 2.8823 | LR: 1.78e-07 | 3491.7ms/step | 5.7 samples/s
918
+ Step 43490/44000 | Loss: 2.9034 | LR: 1.72e-07 | 3491.9ms/step | 5.7 samples/s
919
+ Step 43500/44000 | Loss: 2.8857 | LR: 1.65e-07 | 3493.0ms/step | 5.7 samples/s
920
+ Validation at step 43500: Loss = 2.8831, Perplexity = 17.87
921
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-43500
922
+ Step 43510/44000 | Loss: 2.8773 | LR: 1.58e-07 | 9633.7ms/step | 2.1 samples/s
923
+ Step 43520/44000 | Loss: 2.8691 | LR: 1.52e-07 | 3481.0ms/step | 5.7 samples/s
924
+ Step 43530/44000 | Loss: 2.8892 | LR: 1.46e-07 | 3486.7ms/step | 5.7 samples/s
925
+ Step 43540/44000 | Loss: 2.8819 | LR: 1.40e-07 | 3487.3ms/step | 5.7 samples/s
926
+ Step 43550/44000 | Loss: 2.8739 | LR: 1.34e-07 | 3491.6ms/step | 5.7 samples/s
927
+ Step 43560/44000 | Loss: 2.8496 | LR: 1.28e-07 | 3488.2ms/step | 5.7 samples/s
928
+ Step 43570/44000 | Loss: 2.8964 | LR: 1.22e-07 | 3483.3ms/step | 5.7 samples/s
929
+ Step 43580/44000 | Loss: 2.9185 | LR: 1.16e-07 | 3484.1ms/step | 5.7 samples/s
930
+ Step 43590/44000 | Loss: 2.8933 | LR: 1.11e-07 | 3483.8ms/step | 5.7 samples/s
931
+ Step 43600/44000 | Loss: 2.9049 | LR: 1.06e-07 | 3484.4ms/step | 5.7 samples/s
932
+ Step 43610/44000 | Loss: 2.8928 | LR: 1.00e-07 | 3484.3ms/step | 5.7 samples/s
933
+ Step 43620/44000 | Loss: 2.8836 | LR: 9.52e-08 | 3481.9ms/step | 5.7 samples/s
934
+ Step 43630/44000 | Loss: 2.9354 | LR: 9.03e-08 | 3482.7ms/step | 5.7 samples/s
935
+ Step 43640/44000 | Loss: 2.8933 | LR: 8.55e-08 | 3483.5ms/step | 5.7 samples/s
936
+ Step 43650/44000 | Loss: 2.9024 | LR: 8.08e-08 | 3484.0ms/step | 5.7 samples/s
937
+ Step 43660/44000 | Loss: 2.8809 | LR: 7.62e-08 | 3486.6ms/step | 5.7 samples/s
938
+ Step 43670/44000 | Loss: 2.9031 | LR: 7.18e-08 | 3486.0ms/step | 5.7 samples/s
939
+ Step 43680/44000 | Loss: 2.8861 | LR: 6.75e-08 | 3484.3ms/step | 5.7 samples/s
940
+ Step 43690/44000 | Loss: 2.9115 | LR: 6.34e-08 | 3484.1ms/step | 5.7 samples/s
941
+ Step 43700/44000 | Loss: 2.8806 | LR: 5.94e-08 | 3488.3ms/step | 5.7 samples/s
942
+ Step 43710/44000 | Loss: 2.9052 | LR: 5.55e-08 | 3486.0ms/step | 5.7 samples/s
943
+ Step 43720/44000 | Loss: 2.9191 | LR: 5.17e-08 | 3485.8ms/step | 5.7 samples/s
944
+ Step 43730/44000 | Loss: 2.8742 | LR: 4.81e-08 | 3487.2ms/step | 5.7 samples/s
945
+ Step 43740/44000 | Loss: 2.8801 | LR: 4.46e-08 | 3484.1ms/step | 5.7 samples/s
946
+ Step 43750/44000 | Loss: 2.9085 | LR: 4.12e-08 | 3484.1ms/step | 5.7 samples/s
947
+ Validation at step 43750: Loss = 2.8831, Perplexity = 17.87
948
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-43750
949
+ Step 43760/44000 | Loss: 2.8993 | LR: 3.80e-08 | 9615.8ms/step | 2.1 samples/s
950
+ Step 43770/44000 | Loss: 2.8908 | LR: 3.49e-08 | 3491.4ms/step | 5.7 samples/s
951
+ Step 43780/44000 | Loss: 2.8750 | LR: 3.19e-08 | 3490.8ms/step | 5.7 samples/s
952
+ Step 43790/44000 | Loss: 2.8631 | LR: 2.91e-08 | 3489.8ms/step | 5.7 samples/s
953
+ Step 43800/44000 | Loss: 2.8897 | LR: 2.64e-08 | 3490.5ms/step | 5.7 samples/s
954
+ Step 43810/44000 | Loss: 2.8826 | LR: 2.38e-08 | 3490.4ms/step | 5.7 samples/s
955
+ Step 43820/44000 | Loss: 2.8866 | LR: 2.14e-08 | 3492.8ms/step | 5.7 samples/s
956
+ Step 43830/44000 | Loss: 2.9032 | LR: 1.91e-08 | 3490.0ms/step | 5.7 samples/s
957
+ Step 43840/44000 | Loss: 2.9006 | LR: 1.69e-08 | 3490.8ms/step | 5.7 samples/s
958
+ Step 43850/44000 | Loss: 2.9088 | LR: 1.48e-08 | 3490.1ms/step | 5.7 samples/s
959
+ Step 43860/44000 | Loss: 2.8753 | LR: 1.29e-08 | 3493.8ms/step | 5.7 samples/s
960
+ Step 43870/44000 | Loss: 2.8742 | LR: 1.11e-08 | 3491.7ms/step | 5.7 samples/s
961
+ Step 43880/44000 | Loss: 2.8649 | LR: 9.50e-09 | 3490.6ms/step | 5.7 samples/s
962
+ Step 43890/44000 | Loss: 2.9269 | LR: 7.98e-09 | 3491.1ms/step | 5.7 samples/s
963
+ Step 43900/44000 | Loss: 2.8524 | LR: 6.60e-09 | 3493.8ms/step | 5.7 samples/s
964
+ Step 43910/44000 | Loss: 2.8945 | LR: 5.34e-09 | 3491.8ms/step | 5.7 samples/s
965
+ Step 43920/44000 | Loss: 2.8958 | LR: 4.22e-09 | 3492.4ms/step | 5.7 samples/s
966
+ Step 43930/44000 | Loss: 2.8771 | LR: 3.23e-09 | 3493.2ms/step | 5.7 samples/s
967
+ Step 43940/44000 | Loss: 2.9125 | LR: 2.37e-09 | 3490.5ms/step | 5.7 samples/s
968
+ Step 43950/44000 | Loss: 2.8773 | LR: 1.65e-09 | 3489.9ms/step | 5.7 samples/s
969
+ Step 43960/44000 | Loss: 2.9113 | LR: 1.06e-09 | 3483.4ms/step | 5.7 samples/s
970
+ Step 43970/44000 | Loss: 2.8533 | LR: 5.94e-10 | 3475.2ms/step | 5.8 samples/s
971
+ Step 43980/44000 | Loss: 2.8936 | LR: 2.64e-10 | 3478.5ms/step | 5.7 samples/s
972
+ Step 43990/44000 | Loss: 2.9152 | LR: 6.60e-11 | 3480.4ms/step | 5.7 samples/s
973
+ Step 44000/44000 | Loss: 2.8942 | LR: 0.00e+00 | 3472.1ms/step | 5.8 samples/s
974
+ Validation at step 44000: Loss = 2.8831, Perplexity = 17.87
975
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-44000
976
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031/checkpoint-44000
epfl_final/SparseAtt/wandb/run-20260731_001110-gl67rqc0/files/requirements.txt ADDED
@@ -0,0 +1,142 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ pip==26.0.1
2
+ nvidia-cusparselt-cu13==0.8.0
3
+ mpmath==1.3.0
4
+ cuda-toolkit==13.0.2
5
+ zstandard==0.25.0
6
+ xxhash==3.7.0
7
+ urllib3==2.6.3
8
+ typing_extensions==4.15.0
9
+ triton==3.6.0
10
+ tqdm==4.67.3
11
+ torchinfo==1.8.0
12
+ sympy==1.14.0
13
+ smmap==5.0.3
14
+ six==1.17.0
15
+ shellingham==1.5.4
16
+ setuptools==81.0.0
17
+ safetensors==0.7.0
18
+ regex==2026.4.4
19
+ PyYAML==6.0.3
20
+ Pygments==2.20.0
21
+ pyarrow==24.0.0
22
+ psutil==7.2.2
23
+ protobuf==7.34.1
24
+ propcache==0.4.1
25
+ platformdirs==4.9.6
26
+ packaging==26.2
27
+ nvidia-nvtx==13.0.85
28
+ nvidia-nvshmem-cu13==3.4.5
29
+ nvidia-nvjitlink==13.0.88
30
+ nvidia-nccl-cu13==2.28.9
31
+ nvidia-curand==10.4.0.35
32
+ nvidia-cufile==1.15.1.6
33
+ nvidia-cuda-runtime==13.0.96
34
+ nvidia-cuda-nvrtc==13.0.88
35
+ nvidia-cuda-cupti==13.0.85
36
+ nvidia-cublas==13.1.0.3
37
+ numpy==2.4.4
38
+ networkx==3.6.1
39
+ multidict==6.7.1
40
+ mdurl==0.1.2
41
+ MarkupSafe==3.0.3
42
+ idna==3.13
43
+ hf-xet==1.4.3
44
+ h11==0.16.0
45
+ fsspec==2026.2.0
46
+ frozenlist==1.8.0
47
+ filelock==3.29.0
48
+ dill==0.4.1
49
+ cuda-pathfinder==1.5.4
50
+ click==8.3.3
51
+ charset-normalizer==3.4.7
52
+ certifi==2026.4.22
53
+ attrs==26.1.0
54
+ annotated-types==0.7.0
55
+ annotated-doc==0.0.4
56
+ aiohappyeyeballs==2.6.1
57
+ yarl==1.23.0
58
+ typing-inspection==0.4.2
59
+ sentry-sdk==2.58.0
60
+ requests==2.33.1
61
+ python-dateutil==2.9.0.post0
62
+ pydantic_core==2.46.3
63
+ nvidia-cusparse==12.6.3.3
64
+ nvidia-cufft==12.0.0.61
65
+ nvidia-cudnn-cu13==9.19.0.56
66
+ multiprocess==0.70.19
67
+ markdown-it-py==4.0.0
68
+ Jinja2==3.1.6
69
+ httpcore==1.0.9
70
+ gitdb==4.0.12
71
+ cuda-bindings==13.2.0
72
+ anyio==4.13.0
73
+ aiosignal==1.4.0
74
+ rich==15.0.0
75
+ pydantic==2.13.3
76
+ pandas==3.0.2
77
+ nvidia-cusolver==12.0.4.66
78
+ httpx==0.28.1
79
+ GitPython==3.1.49
80
+ aiohttp==3.13.5
81
+ wandb==0.26.1
82
+ typer==0.25.1
83
+ torch==2.11.0
84
+ huggingface_hub==1.13.0
85
+ tokenizers==0.22.2
86
+ datasets==4.8.5
87
+ accelerate==1.13.0
88
+ transformers==5.7.0
89
+ word2number==1.1
90
+ sqlitedict==2.1.0
91
+ pytz==2026.2
92
+ threadpoolctl==3.6.0
93
+ tcolorpy==0.1.7
94
+ tabulate==0.10.0
95
+ scipy==1.17.1
96
+ portalocker==3.2.0
97
+ pathvalidate==3.3.1
98
+ narwhals==2.22.1
99
+ more-itertools==11.1.0
100
+ lxml==6.1.1
101
+ joblib==1.5.3
102
+ colorama==0.4.6
103
+ chardet==6.0.0.post1
104
+ absl-py==2.4.0
105
+ scikit-learn==1.9.0
106
+ sacrebleu==2.6.0
107
+ nltk==3.9.4
108
+ mbstrdecoder==1.1.5
109
+ typepy==1.3.5
110
+ rouge_score==0.1.2
111
+ DataProperty==1.1.1
112
+ tabledata==1.3.5
113
+ evaluate==0.4.6
114
+ pytablewriter==1.2.1
115
+ lm_eval==0.4.12
116
+ pluggy==1.6.0
117
+ iniconfig==2.3.0
118
+ pytest==9.1.0
119
+ py-spy==0.4.2
120
+ pillow==12.2.0
121
+ tifffile==2026.6.1
122
+ imagecodecs==2026.6.26
123
+ zipp==4.1.0
124
+ importlib_metadata==9.0.0
125
+ derivative==0.6.3
126
+ pysindy==2.1.0
127
+ pyparsing==3.3.2
128
+ kiwisolver==1.5.0
129
+ fonttools==4.63.0
130
+ cycler==0.12.1
131
+ contourpy==1.3.3
132
+ matplotlib==3.11.0
133
+ sortedcontainers==2.4.0
134
+ wcwidth==0.8.2
135
+ sigfig==1.3.19
136
+ importlib_resources==7.1.0
137
+ enum-tools==0.13.0
138
+ cramjam==2.11.0
139
+ python-snappy==0.7.3
140
+ compact-json==1.8.2
141
+ numbers-parser==4.18.5
142
+ entmax==1.3
epfl_final/SparseAtt/wandb/run-20260731_001110-gl67rqc0/files/wandb-metadata.json ADDED
@@ -0,0 +1,65 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "os": "Linux-6.8.0-124-generic-x86_64-with-glibc2.35",
3
+ "python": "CPython 3.13.13",
4
+ "startedAt": "2026-07-31T00:11:10.058418Z",
5
+ "args": [
6
+ "--config",
7
+ "/mloscratch/homes/navasard/sparse/SparseAtt/configs/ladder8k/365m_mixed_to_dense_20pct_seed43_20260730.json"
8
+ ],
9
+ "program": "/mloscratch/homes/navasard/sparse/SparseAtt/train_dense_to_mixed_exact.py",
10
+ "codePath": "train_dense_to_mixed_exact.py",
11
+ "codePathLocal": "train_dense_to_mixed_exact.py",
12
+ "git": {
13
+ "remote": "https://github.com/rubentium/TokenPrune.git",
14
+ "commit": "88298c7a418622530ca5426b22b1e5ebd10ca0cb"
15
+ },
16
+ "email": "ruben.navasardyan@mail.utoronto.ca",
17
+ "root": "/mloscratch/homes/navasard/sparse/SparseAtt",
18
+ "host": "a100-train-4-gpu-2-0-40",
19
+ "executable": "/mloscratch/homes/navasard/conda/bin/python3.13",
20
+ "cpu_count": 64,
21
+ "cpu_count_logical": 128,
22
+ "gpu": "NVIDIA A100-SXM4-80GB",
23
+ "gpu_count": 4,
24
+ "disk": {
25
+ "/": {
26
+ "total": "7679362727936",
27
+ "used": "5257403871232"
28
+ }
29
+ },
30
+ "memory": {
31
+ "total": "1081884934144"
32
+ },
33
+ "gpu_nvidia": [
34
+ {
35
+ "name": "NVIDIA A100-SXM4-80GB",
36
+ "memoryTotal": "85899345920",
37
+ "cudaCores": 6912,
38
+ "architecture": "Ampere",
39
+ "uuid": "GPU-26e055ad-009a-c42a-2ba0-ce6db370fcbe"
40
+ },
41
+ {
42
+ "name": "NVIDIA A100-SXM4-80GB",
43
+ "memoryTotal": "85899345920",
44
+ "cudaCores": 6912,
45
+ "architecture": "Ampere",
46
+ "uuid": "GPU-56ec1834-3584-3932-0125-388b5e88fac2"
47
+ },
48
+ {
49
+ "name": "NVIDIA A100-SXM4-80GB",
50
+ "memoryTotal": "85899345920",
51
+ "cudaCores": 6912,
52
+ "architecture": "Ampere",
53
+ "uuid": "GPU-ce50edb9-4d69-8d20-4433-5217084c7016"
54
+ },
55
+ {
56
+ "name": "NVIDIA A100-SXM4-80GB",
57
+ "memoryTotal": "85899345920",
58
+ "cudaCores": 6912,
59
+ "architecture": "Ampere",
60
+ "uuid": "GPU-2401254a-acb7-5787-0b06-07065349e511"
61
+ }
62
+ ],
63
+ "cudaVersion": "13.0",
64
+ "writerId": "rekjwwv2xzeva8fcuzeebmynnd9sd1h3"
65
+ }
epfl_final/SparseAtt/wandb/run-20260731_001110-gl67rqc0/files/wandb-summary.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"train/epoch":0.2906439083679023,"speed/steps_per_sec":0.2880073715753949,"eval/dense_interval_end":-1,"speed/ms_per_step":3472.1333503723145,"train/loss":2.8942300343513487,"_step":44000,"eval/loss":2.8830933475494387,"_timestamp":1.7854909210582159e+09,"eval/uses_sparse_attention":0,"eval/dense_interval_active":0,"_wandb":{"runtime":34256},"train/global_step":44000,"_runtime":34256.97162501,"eval/dense_interval_start":-1,"speed/samples_per_sec":5.760147431507898,"eval/uses_dense_attention":1,"eval/perplexity":17.869464236448252,"train/learning_rate":0}
epfl_final/SparseAtt/wandb/run-20260731_001110-gl67rqc0/logs/debug-internal.log ADDED
The diff for this file is too large to render. See raw diff
 
epfl_final/SparseAtt/wandb/run-20260731_001110-gl67rqc0/logs/debug.log ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2026-07-31 00:11:10,096 INFO MainThread:162 [wandb_setup.py:_flush():81] Current SDK version is 0.26.1
2
+ 2026-07-31 00:11:10,096 INFO MainThread:162 [wandb_setup.py:_flush():81] Configure stats pid to 162
3
+ 2026-07-31 00:11:10,096 INFO MainThread:162 [wandb_setup.py:_flush():81] Loading settings from environment variables
4
+ 2026-07-31 00:11:10,096 INFO MainThread:162 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /mloscratch/homes/navasard/sparse/SparseAtt/wandb/run-20260731_001110-gl67rqc0/logs/debug.log
5
+ 2026-07-31 00:11:10,096 INFO MainThread:162 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /mloscratch/homes/navasard/sparse/SparseAtt/wandb/run-20260731_001110-gl67rqc0/logs/debug-internal.log
6
+ 2026-07-31 00:11:10,096 INFO MainThread:162 [wandb_init.py:init():850] calling init triggers
7
+ 2026-07-31 00:11:10,096 INFO MainThread:162 [wandb_init.py:init():855] wandb.init called with sweep_config: {}
8
+ config: {'model_path': '/mloscratch/homes/navasard/qwen3_06b_base', 'model_dtype': 'bfloat16', 'init_from_scratch': True, 'train_path': '/mloscratch/homes/navasard/data/fineweb_large/tokenized', 'val_path': None, 'max_seq_length': 8192, 'num_workers': 8, 'output_dir': '/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed43/run_20260731_001031', 'resume_from_checkpoint': '/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_seed43/run_20260719_234149/checkpoint-35000', 'force_new_run': True, 'num_epochs': 1, 'max_steps': 44000, 'per_device_batch_size': 2, 'gradient_accumulation_steps': 10, 'learning_rate': 0.0005, 'weight_decay': 0.01, 'warmup_steps': 750, 'lr_scheduler': 'cosine', 'max_grad_norm': 1.0, 'save_steps': 250, 'eval_steps': 250, 'eval_batches': 50, 'logging_steps': 10, 'seed': 43, 'verbose': False, 'checkpoint_activations': False, 'strategy': 'ddp', 'mixed_precision': 'bf16', 'wandb_enabled': True, 'wandb_project': 'stochastic_sparsity', 'wandb_name': '365m mixed-to-dense 20pct seed43', 'wandb_tags': ['qwen3', '365m', '8k', 'mixed-to-dense', '20pct', 'seed43', 'multiseed'], 'hidden_size': 768, 'intermediate_size': 3072, 'num_attention_heads': 6, 'num_key_value_heads': 2, 'num_hidden_layers': 28, 'head_dim': None, 'attention_type': 'sparse_stochastic', 'sparse_block_size': 128, 'sparse_num_random_blocks': 1, 'sparse_mask_pool_size': 1, 'sparse_val_dense_compare': False, 'sparse_layer_indices': None, 'sparse_mask_cache_dir': None, 'sparse_mask_seed': 0, 'dense_mode': True, 'dense_step_intervals': None, '_wandb': {}}
9
+ 2026-07-31 00:11:10,096 INFO MainThread:162 [wandb_init.py:init():898] starting backend
10
+ 2026-07-31 00:11:10,618 INFO MainThread:162 [wandb_init.py:init():913] sending inform_init request
11
+ 2026-07-31 00:11:11,129 INFO MainThread:162 [wandb_init.py:init():918] backend started and connected
12
+ 2026-07-31 00:11:11,131 INFO MainThread:162 [wandb_init.py:init():988] updated telemetry
13
+ 2026-07-31 00:11:11,188 INFO MainThread:162 [wandb_init.py:init():1011] communicating run to backend with 90.0 second timeout
14
+ 2026-07-31 00:11:11,607 INFO MainThread:162 [wandb_init.py:init():1056] starting run threads in backend
15
+ 2026-07-31 00:11:11,868 INFO MainThread:162 [wandb_run.py:_console_start():2554] atexit reg
16
+ 2026-07-31 00:11:11,868 INFO MainThread:162 [wandb_run.py:_redirect():2403] redirect: wrap_raw
17
+ 2026-07-31 00:11:11,869 INFO MainThread:162 [wandb_run.py:_redirect():2472] Wrapping output streams.
18
+ 2026-07-31 00:11:11,869 INFO MainThread:162 [wandb_run.py:_redirect():2495] Redirects installed.
19
+ 2026-07-31 00:11:11,876 INFO MainThread:162 [wandb_init.py:init():1094] run started, returning control to user process
20
+ 2026-07-31 09:42:08,577 INFO MainThread:162 [wandb_run.py:_finish():2321] finishing run ruben-navasardyan4869-epfl/stochastic_sparsity/gl67rqc0
21
+ 2026-07-31 09:42:08,578 INFO MainThread:162 [wandb_run.py:_atexit_cleanup():2520] got exitcode: 0
22
+ 2026-07-31 09:42:08,578 INFO MainThread:162 [wandb_run.py:_restore():2502] restore
23
+ 2026-07-31 09:42:08,578 INFO MainThread:162 [wandb_run.py:_restore():2508] restore done
24
+ 2026-07-31 09:42:10,262 INFO MainThread:162 [wandb_run.py:_footer_sync_info():3889] logging synced files
epfl_final/SparseAtt/wandb/run-20260731_001110-gl67rqc0/run-gl67rqc0.wandb ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7ff5844e1ebd9186a093a5c6489bca277b2e8adc5a9c1b30fc9d947486ed5147
3
+ size 6137079
epfl_final/SparseAtt/wandb/run-20260731_001110-rdj7s9cl/files/config.yaml ADDED
@@ -0,0 +1,187 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ _wandb:
2
+ value:
3
+ cli_version: 0.26.1
4
+ e:
5
+ hq00yk9icqnt8k6789b401jp03v00xmk:
6
+ args:
7
+ - --config
8
+ - /mloscratch/homes/navasard/sparse/SparseAtt/configs/ladder8k/365m_mixed_to_dense_20pct_seed44_20260730.json
9
+ codePath: train_dense_to_mixed_exact.py
10
+ codePathLocal: train_dense_to_mixed_exact.py
11
+ cpu_count: 64
12
+ cpu_count_logical: 128
13
+ cudaVersion: "13.0"
14
+ disk:
15
+ /:
16
+ total: "7679362727936"
17
+ used: "5257403871232"
18
+ email: ruben.navasardyan@mail.utoronto.ca
19
+ executable: /mloscratch/homes/navasard/conda/bin/python3.13
20
+ git:
21
+ commit: 88298c7a418622530ca5426b22b1e5ebd10ca0cb
22
+ remote: https://github.com/rubentium/TokenPrune.git
23
+ gpu: NVIDIA A100-SXM4-80GB
24
+ gpu_count: 4
25
+ gpu_nvidia:
26
+ - architecture: Ampere
27
+ cudaCores: 6912
28
+ memoryTotal: "85899345920"
29
+ name: NVIDIA A100-SXM4-80GB
30
+ uuid: GPU-26e055ad-009a-c42a-2ba0-ce6db370fcbe
31
+ - architecture: Ampere
32
+ cudaCores: 6912
33
+ memoryTotal: "85899345920"
34
+ name: NVIDIA A100-SXM4-80GB
35
+ uuid: GPU-56ec1834-3584-3932-0125-388b5e88fac2
36
+ - architecture: Ampere
37
+ cudaCores: 6912
38
+ memoryTotal: "85899345920"
39
+ name: NVIDIA A100-SXM4-80GB
40
+ uuid: GPU-ce50edb9-4d69-8d20-4433-5217084c7016
41
+ - architecture: Ampere
42
+ cudaCores: 6912
43
+ memoryTotal: "85899345920"
44
+ name: NVIDIA A100-SXM4-80GB
45
+ uuid: GPU-2401254a-acb7-5787-0b06-07065349e511
46
+ host: a100-train-4-gpu-2-0-40
47
+ memory:
48
+ total: "1081884934144"
49
+ os: Linux-6.8.0-124-generic-x86_64-with-glibc2.35
50
+ program: /mloscratch/homes/navasard/sparse/SparseAtt/train_dense_to_mixed_exact.py
51
+ python: CPython 3.13.13
52
+ root: /mloscratch/homes/navasard/sparse/SparseAtt
53
+ startedAt: "2026-07-31T00:11:10.398995Z"
54
+ writerId: hq00yk9icqnt8k6789b401jp03v00xmk
55
+ m: []
56
+ python_version: 3.13.13
57
+ t:
58
+ "1":
59
+ - 1
60
+ - 5
61
+ - 11
62
+ - 49
63
+ - 51
64
+ - 53
65
+ - 71
66
+ "2":
67
+ - 1
68
+ - 5
69
+ - 11
70
+ - 49
71
+ - 51
72
+ - 53
73
+ - 71
74
+ "3":
75
+ - 2
76
+ - 13
77
+ - 15
78
+ - 16
79
+ - 61
80
+ "4": 3.13.13
81
+ "5": 0.26.1
82
+ "6": 5.7.0
83
+ "12": 0.26.1
84
+ "13": linux-x86_64
85
+ attention_type:
86
+ value: sparse_stochastic
87
+ checkpoint_activations:
88
+ value: false
89
+ dense_mode:
90
+ value: true
91
+ dense_step_intervals:
92
+ value: null
93
+ eval_batches:
94
+ value: 50
95
+ eval_steps:
96
+ value: 250
97
+ force_new_run:
98
+ value: true
99
+ gradient_accumulation_steps:
100
+ value: 10
101
+ head_dim:
102
+ value: null
103
+ hidden_size:
104
+ value: 768
105
+ init_from_scratch:
106
+ value: true
107
+ intermediate_size:
108
+ value: 3072
109
+ learning_rate:
110
+ value: 0.0005
111
+ logging_steps:
112
+ value: 10
113
+ lr_scheduler:
114
+ value: cosine
115
+ max_grad_norm:
116
+ value: 1
117
+ max_seq_length:
118
+ value: 8192
119
+ max_steps:
120
+ value: 44000
121
+ mixed_precision:
122
+ value: bf16
123
+ model_dtype:
124
+ value: bfloat16
125
+ model_path:
126
+ value: /mloscratch/homes/navasard/qwen3_06b_base
127
+ num_attention_heads:
128
+ value: 6
129
+ num_epochs:
130
+ value: 1
131
+ num_hidden_layers:
132
+ value: 28
133
+ num_key_value_heads:
134
+ value: 2
135
+ num_workers:
136
+ value: 8
137
+ output_dir:
138
+ value: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031
139
+ per_device_batch_size:
140
+ value: 2
141
+ resume_from_checkpoint:
142
+ value: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_seed44/run_20260722_221755/checkpoint-35000
143
+ save_steps:
144
+ value: 250
145
+ seed:
146
+ value: 44
147
+ sparse_block_size:
148
+ value: 128
149
+ sparse_layer_indices:
150
+ value: null
151
+ sparse_mask_cache_dir:
152
+ value: null
153
+ sparse_mask_pool_size:
154
+ value: 1
155
+ sparse_mask_seed:
156
+ value: 0
157
+ sparse_num_random_blocks:
158
+ value: 1
159
+ sparse_val_dense_compare:
160
+ value: false
161
+ strategy:
162
+ value: ddp
163
+ train_path:
164
+ value: /mloscratch/homes/navasard/data/fineweb_large/tokenized
165
+ val_path:
166
+ value: null
167
+ verbose:
168
+ value: false
169
+ wandb_enabled:
170
+ value: true
171
+ wandb_name:
172
+ value: 365m mixed-to-dense 20pct seed44
173
+ wandb_project:
174
+ value: stochastic_sparsity
175
+ wandb_tags:
176
+ value:
177
+ - qwen3
178
+ - 365m
179
+ - 8k
180
+ - mixed-to-dense
181
+ - 20pct
182
+ - seed44
183
+ - multiseed
184
+ warmup_steps:
185
+ value: 750
186
+ weight_decay:
187
+ value: 0.01
epfl_final/SparseAtt/wandb/run-20260731_001110-rdj7s9cl/files/output.log ADDED
@@ -0,0 +1,976 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Starting training...
2
+ Validation at step 35000: Loss = 2.9143, Perplexity = 18.44
3
+ Switching attention mode for step 35001: dense (dense step interval)
4
+ Step 35010/44000 | Loss: 2.9100 | LR: 5.14e-05 | 67669.8ms/step | 0.3 samples/s
5
+ Step 35020/44000 | Loss: 2.9127 | LR: 5.13e-05 | 3463.2ms/step | 5.8 samples/s
6
+ Step 35030/44000 | Loss: 2.8864 | LR: 5.12e-05 | 3465.2ms/step | 5.8 samples/s
7
+ Step 35040/44000 | Loss: 2.9288 | LR: 5.11e-05 | 3465.8ms/step | 5.8 samples/s
8
+ Step 35050/44000 | Loss: 2.8871 | LR: 5.10e-05 | 3470.2ms/step | 5.8 samples/s
9
+ Step 35060/44000 | Loss: 2.8808 | LR: 5.09e-05 | 3472.0ms/step | 5.8 samples/s
10
+ Step 35070/44000 | Loss: 2.8762 | LR: 5.08e-05 | 3473.1ms/step | 5.8 samples/s
11
+ Step 35080/44000 | Loss: 2.9025 | LR: 5.07e-05 | 3472.4ms/step | 5.8 samples/s
12
+ Step 35090/44000 | Loss: 2.8962 | LR: 5.06e-05 | 3475.8ms/step | 5.8 samples/s
13
+ Step 35100/44000 | Loss: 2.9055 | LR: 5.04e-05 | 3474.8ms/step | 5.8 samples/s
14
+ Step 35110/44000 | Loss: 2.8974 | LR: 5.03e-05 | 3476.6ms/step | 5.8 samples/s
15
+ Step 35120/44000 | Loss: 2.9046 | LR: 5.02e-05 | 3476.7ms/step | 5.8 samples/s
16
+ Step 35130/44000 | Loss: 2.8989 | LR: 5.01e-05 | 3476.1ms/step | 5.8 samples/s
17
+ Step 35140/44000 | Loss: 2.8919 | LR: 5.00e-05 | 3476.7ms/step | 5.8 samples/s
18
+ Step 35150/44000 | Loss: 2.8692 | LR: 4.99e-05 | 3478.4ms/step | 5.7 samples/s
19
+ Step 35160/44000 | Loss: 2.9009 | LR: 4.98e-05 | 3480.1ms/step | 5.7 samples/s
20
+ Step 35170/44000 | Loss: 2.9037 | LR: 4.97e-05 | 3477.0ms/step | 5.8 samples/s
21
+ Step 35180/44000 | Loss: 2.9046 | LR: 4.96e-05 | 3476.2ms/step | 5.8 samples/s
22
+ Step 35190/44000 | Loss: 2.8931 | LR: 4.95e-05 | 3474.5ms/step | 5.8 samples/s
23
+ Step 35200/44000 | Loss: 2.8786 | LR: 4.94e-05 | 3474.4ms/step | 5.8 samples/s
24
+ Step 35210/44000 | Loss: 2.9169 | LR: 4.93e-05 | 3478.0ms/step | 5.8 samples/s
25
+ Step 35220/44000 | Loss: 2.8945 | LR: 4.91e-05 | 3476.8ms/step | 5.8 samples/s
26
+ Step 35230/44000 | Loss: 2.8911 | LR: 4.90e-05 | 3477.5ms/step | 5.8 samples/s
27
+ Step 35240/44000 | Loss: 2.8943 | LR: 4.89e-05 | 3480.9ms/step | 5.7 samples/s
28
+ Step 35250/44000 | Loss: 2.8769 | LR: 4.88e-05 | 3476.8ms/step | 5.8 samples/s
29
+ Validation at step 35250: Loss = 2.9075, Perplexity = 18.31
30
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-35250
31
+ Step 35260/44000 | Loss: 2.9212 | LR: 4.87e-05 | 9627.9ms/step | 2.1 samples/s
32
+ Step 35270/44000 | Loss: 2.8784 | LR: 4.86e-05 | 3479.3ms/step | 5.7 samples/s
33
+ Step 35280/44000 | Loss: 2.9062 | LR: 4.85e-05 | 3478.9ms/step | 5.7 samples/s
34
+ Step 35290/44000 | Loss: 2.8858 | LR: 4.84e-05 | 3479.6ms/step | 5.7 samples/s
35
+ Step 35300/44000 | Loss: 2.8722 | LR: 4.83e-05 | 3480.8ms/step | 5.7 samples/s
36
+ Step 35310/44000 | Loss: 2.8640 | LR: 4.82e-05 | 3476.9ms/step | 5.8 samples/s
37
+ Step 35320/44000 | Loss: 2.8951 | LR: 4.81e-05 | 3478.3ms/step | 5.7 samples/s
38
+ Step 35330/44000 | Loss: 2.9099 | LR: 4.80e-05 | 3484.3ms/step | 5.7 samples/s
39
+ Step 35340/44000 | Loss: 2.8842 | LR: 4.79e-05 | 3486.5ms/step | 5.7 samples/s
40
+ Step 35350/44000 | Loss: 2.8885 | LR: 4.77e-05 | 3485.7ms/step | 5.7 samples/s
41
+ Step 35360/44000 | Loss: 2.9079 | LR: 4.76e-05 | 3486.8ms/step | 5.7 samples/s
42
+ Step 35370/44000 | Loss: 2.9065 | LR: 4.75e-05 | 3485.5ms/step | 5.7 samples/s
43
+ Step 35380/44000 | Loss: 2.9029 | LR: 4.74e-05 | 3483.4ms/step | 5.7 samples/s
44
+ Step 35390/44000 | Loss: 2.8965 | LR: 4.73e-05 | 3482.3ms/step | 5.7 samples/s
45
+ Step 35400/44000 | Loss: 2.8981 | LR: 4.72e-05 | 3483.8ms/step | 5.7 samples/s
46
+ Step 35410/44000 | Loss: 2.8734 | LR: 4.71e-05 | 3480.1ms/step | 5.7 samples/s
47
+ Step 35420/44000 | Loss: 2.9371 | LR: 4.70e-05 | 3485.5ms/step | 5.7 samples/s
48
+ Step 35430/44000 | Loss: 2.8992 | LR: 4.69e-05 | 3481.1ms/step | 5.7 samples/s
49
+ Step 35440/44000 | Loss: 2.9063 | LR: 4.68e-05 | 3473.6ms/step | 5.8 samples/s
50
+ Step 35450/44000 | Loss: 2.8922 | LR: 4.67e-05 | 3467.5ms/step | 5.8 samples/s
51
+ Step 35460/44000 | Loss: 2.8922 | LR: 4.66e-05 | 3468.8ms/step | 5.8 samples/s
52
+ Step 35470/44000 | Loss: 2.8867 | LR: 4.65e-05 | 3467.4ms/step | 5.8 samples/s
53
+ Step 35480/44000 | Loss: 2.8970 | LR: 4.64e-05 | 3462.3ms/step | 5.8 samples/s
54
+ Step 35490/44000 | Loss: 2.9029 | LR: 4.63e-05 | 3461.3ms/step | 5.8 samples/s
55
+ Step 35500/44000 | Loss: 2.9176 | LR: 4.62e-05 | 3464.0ms/step | 5.8 samples/s
56
+ Validation at step 35500: Loss = 2.9068, Perplexity = 18.30
57
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-35500
58
+ Step 35510/44000 | Loss: 2.8965 | LR: 4.61e-05 | 9569.6ms/step | 2.1 samples/s
59
+ Step 35520/44000 | Loss: 2.8965 | LR: 4.59e-05 | 3465.8ms/step | 5.8 samples/s
60
+ Step 35530/44000 | Loss: 2.9327 | LR: 4.58e-05 | 3466.5ms/step | 5.8 samples/s
61
+ Step 35540/44000 | Loss: 2.9123 | LR: 4.57e-05 | 3461.9ms/step | 5.8 samples/s
62
+ Step 35550/44000 | Loss: 2.9251 | LR: 4.56e-05 | 3463.8ms/step | 5.8 samples/s
63
+ Step 35560/44000 | Loss: 2.8824 | LR: 4.55e-05 | 3471.3ms/step | 5.8 samples/s
64
+ Step 35570/44000 | Loss: 2.9049 | LR: 4.54e-05 | 3478.9ms/step | 5.7 samples/s
65
+ Step 35580/44000 | Loss: 2.8952 | LR: 4.53e-05 | 3470.4ms/step | 5.8 samples/s
66
+ Step 35590/44000 | Loss: 2.9165 | LR: 4.52e-05 | 3472.1ms/step | 5.8 samples/s
67
+ Step 35600/44000 | Loss: 2.9063 | LR: 4.51e-05 | 3471.9ms/step | 5.8 samples/s
68
+ Step 35610/44000 | Loss: 2.8673 | LR: 4.50e-05 | 3471.8ms/step | 5.8 samples/s
69
+ Step 35620/44000 | Loss: 2.9185 | LR: 4.49e-05 | 3472.2ms/step | 5.8 samples/s
70
+ Step 35630/44000 | Loss: 2.9176 | LR: 4.48e-05 | 3470.9ms/step | 5.8 samples/s
71
+ Step 35640/44000 | Loss: 2.9108 | LR: 4.47e-05 | 3472.0ms/step | 5.8 samples/s
72
+ Step 35650/44000 | Loss: 2.8962 | LR: 4.46e-05 | 3477.1ms/step | 5.8 samples/s
73
+ Step 35660/44000 | Loss: 2.9061 | LR: 4.45e-05 | 3473.0ms/step | 5.8 samples/s
74
+ Step 35670/44000 | Loss: 2.8922 | LR: 4.44e-05 | 3472.4ms/step | 5.8 samples/s
75
+ Step 35680/44000 | Loss: 2.8891 | LR: 4.43e-05 | 3474.1ms/step | 5.8 samples/s
76
+ Step 35690/44000 | Loss: 2.8789 | LR: 4.42e-05 | 3472.4ms/step | 5.8 samples/s
77
+ Step 35700/44000 | Loss: 2.9172 | LR: 4.41e-05 | 3473.1ms/step | 5.8 samples/s
78
+ Step 35710/44000 | Loss: 2.8854 | LR: 4.40e-05 | 3472.4ms/step | 5.8 samples/s
79
+ Step 35720/44000 | Loss: 2.8802 | LR: 4.39e-05 | 3471.3ms/step | 5.8 samples/s
80
+ Step 35730/44000 | Loss: 2.9086 | LR: 4.38e-05 | 3473.6ms/step | 5.8 samples/s
81
+ Step 35740/44000 | Loss: 2.8645 | LR: 4.37e-05 | 3472.6ms/step | 5.8 samples/s
82
+ Step 35750/44000 | Loss: 2.8967 | LR: 4.36e-05 | 3472.5ms/step | 5.8 samples/s
83
+ Validation at step 35750: Loss = 2.9064, Perplexity = 18.29
84
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-35750
85
+ Step 35760/44000 | Loss: 2.9022 | LR: 4.35e-05 | 9604.5ms/step | 2.1 samples/s
86
+ Step 35770/44000 | Loss: 2.8900 | LR: 4.34e-05 | 3484.3ms/step | 5.7 samples/s
87
+ Step 35780/44000 | Loss: 2.9001 | LR: 4.33e-05 | 3484.2ms/step | 5.7 samples/s
88
+ Step 35790/44000 | Loss: 2.9047 | LR: 4.32e-05 | 3484.0ms/step | 5.7 samples/s
89
+ Step 35800/44000 | Loss: 2.8950 | LR: 4.31e-05 | 3484.1ms/step | 5.7 samples/s
90
+ Step 35810/44000 | Loss: 2.8941 | LR: 4.29e-05 | 3479.4ms/step | 5.7 samples/s
91
+ Step 35820/44000 | Loss: 2.8866 | LR: 4.28e-05 | 3472.7ms/step | 5.8 samples/s
92
+ Step 35830/44000 | Loss: 2.8875 | LR: 4.27e-05 | 3476.7ms/step | 5.8 samples/s
93
+ Step 35840/44000 | Loss: 2.8725 | LR: 4.26e-05 | 3473.1ms/step | 5.8 samples/s
94
+ Step 35850/44000 | Loss: 2.8799 | LR: 4.25e-05 | 3482.9ms/step | 5.7 samples/s
95
+ Step 35860/44000 | Loss: 2.8922 | LR: 4.24e-05 | 3477.8ms/step | 5.8 samples/s
96
+ Step 35870/44000 | Loss: 2.9260 | LR: 4.23e-05 | 3477.3ms/step | 5.8 samples/s
97
+ Step 35880/44000 | Loss: 2.8951 | LR: 4.22e-05 | 3475.6ms/step | 5.8 samples/s
98
+ Step 35890/44000 | Loss: 2.8829 | LR: 4.21e-05 | 3483.6ms/step | 5.7 samples/s
99
+ Step 35900/44000 | Loss: 2.8817 | LR: 4.20e-05 | 3482.7ms/step | 5.7 samples/s
100
+ Step 35910/44000 | Loss: 2.9067 | LR: 4.19e-05 | 3474.1ms/step | 5.8 samples/s
101
+ Step 35920/44000 | Loss: 2.9072 | LR: 4.18e-05 | 3475.1ms/step | 5.8 samples/s
102
+ Step 35930/44000 | Loss: 2.9018 | LR: 4.17e-05 | 3475.0ms/step | 5.8 samples/s
103
+ Step 35940/44000 | Loss: 2.8734 | LR: 4.16e-05 | 3478.1ms/step | 5.8 samples/s
104
+ Step 35950/44000 | Loss: 2.9035 | LR: 4.15e-05 | 3486.5ms/step | 5.7 samples/s
105
+ Step 35960/44000 | Loss: 2.9070 | LR: 4.14e-05 | 3486.5ms/step | 5.7 samples/s
106
+ Step 35970/44000 | Loss: 2.8908 | LR: 4.13e-05 | 3481.1ms/step | 5.7 samples/s
107
+ Step 35980/44000 | Loss: 2.9040 | LR: 4.12e-05 | 3476.3ms/step | 5.8 samples/s
108
+ Step 35990/44000 | Loss: 2.8650 | LR: 4.11e-05 | 3476.3ms/step | 5.8 samples/s
109
+ Step 36000/44000 | Loss: 2.8759 | LR: 4.10e-05 | 3473.9ms/step | 5.8 samples/s
110
+ Validation at step 36000: Loss = 2.9060, Perplexity = 18.28
111
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-36000
112
+ Step 36010/44000 | Loss: 2.9143 | LR: 4.09e-05 | 9611.7ms/step | 2.1 samples/s
113
+ Step 36020/44000 | Loss: 2.8971 | LR: 4.08e-05 | 3475.6ms/step | 5.8 samples/s
114
+ Step 36030/44000 | Loss: 2.8797 | LR: 4.07e-05 | 3476.6ms/step | 5.8 samples/s
115
+ Step 36040/44000 | Loss: 2.8888 | LR: 4.06e-05 | 3476.2ms/step | 5.8 samples/s
116
+ Step 36050/44000 | Loss: 2.8991 | LR: 4.05e-05 | 3472.9ms/step | 5.8 samples/s
117
+ Step 36060/44000 | Loss: 2.8952 | LR: 4.04e-05 | 3480.9ms/step | 5.7 samples/s
118
+ Step 36070/44000 | Loss: 2.8987 | LR: 4.03e-05 | 3481.3ms/step | 5.7 samples/s
119
+ Step 36080/44000 | Loss: 2.8836 | LR: 4.02e-05 | 3475.5ms/step | 5.8 samples/s
120
+ Step 36090/44000 | Loss: 2.8858 | LR: 4.01e-05 | 3476.3ms/step | 5.8 samples/s
121
+ Step 36100/44000 | Loss: 2.8965 | LR: 4.00e-05 | 3490.7ms/step | 5.7 samples/s
122
+ Step 36110/44000 | Loss: 2.9213 | LR: 3.99e-05 | 3493.3ms/step | 5.7 samples/s
123
+ Step 36120/44000 | Loss: 2.9169 | LR: 3.98e-05 | 3489.5ms/step | 5.7 samples/s
124
+ Step 36130/44000 | Loss: 2.8959 | LR: 3.97e-05 | 3484.8ms/step | 5.7 samples/s
125
+ Step 36140/44000 | Loss: 2.9063 | LR: 3.97e-05 | 3485.3ms/step | 5.7 samples/s
126
+ Step 36150/44000 | Loss: 2.9311 | LR: 3.96e-05 | 3484.8ms/step | 5.7 samples/s
127
+ Step 36160/44000 | Loss: 2.9193 | LR: 3.95e-05 | 3489.5ms/step | 5.7 samples/s
128
+ Step 36170/44000 | Loss: 2.8918 | LR: 3.94e-05 | 3483.9ms/step | 5.7 samples/s
129
+ Step 36180/44000 | Loss: 2.9355 | LR: 3.93e-05 | 3479.6ms/step | 5.7 samples/s
130
+ Step 36190/44000 | Loss: 2.8972 | LR: 3.92e-05 | 3476.9ms/step | 5.8 samples/s
131
+ Step 36200/44000 | Loss: 2.9172 | LR: 3.91e-05 | 3477.7ms/step | 5.8 samples/s
132
+ Step 36210/44000 | Loss: 2.8975 | LR: 3.90e-05 | 3476.8ms/step | 5.8 samples/s
133
+ Step 36220/44000 | Loss: 2.8755 | LR: 3.89e-05 | 3491.0ms/step | 5.7 samples/s
134
+ Step 36230/44000 | Loss: 2.8860 | LR: 3.88e-05 | 3486.3ms/step | 5.7 samples/s
135
+ Step 36240/44000 | Loss: 2.8978 | LR: 3.87e-05 | 3489.4ms/step | 5.7 samples/s
136
+ Step 36250/44000 | Loss: 2.9359 | LR: 3.86e-05 | 3485.9ms/step | 5.7 samples/s
137
+ Validation at step 36250: Loss = 2.9058, Perplexity = 18.28
138
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-36250
139
+ Step 36260/44000 | Loss: 2.9179 | LR: 3.85e-05 | 9599.8ms/step | 2.1 samples/s
140
+ Step 36270/44000 | Loss: 2.8829 | LR: 3.84e-05 | 3475.2ms/step | 5.8 samples/s
141
+ Step 36280/44000 | Loss: 2.8837 | LR: 3.83e-05 | 3477.2ms/step | 5.8 samples/s
142
+ Step 36290/44000 | Loss: 2.9028 | LR: 3.82e-05 | 3474.6ms/step | 5.8 samples/s
143
+ Step 36300/44000 | Loss: 2.9151 | LR: 3.81e-05 | 3476.4ms/step | 5.8 samples/s
144
+ Step 36310/44000 | Loss: 2.8935 | LR: 3.80e-05 | 3477.8ms/step | 5.8 samples/s
145
+ Step 36320/44000 | Loss: 2.8874 | LR: 3.79e-05 | 3477.8ms/step | 5.8 samples/s
146
+ Step 36330/44000 | Loss: 2.9242 | LR: 3.78e-05 | 3478.4ms/step | 5.7 samples/s
147
+ Step 36340/44000 | Loss: 2.9034 | LR: 3.77e-05 | 3478.6ms/step | 5.7 samples/s
148
+ Step 36350/44000 | Loss: 2.8964 | LR: 3.76e-05 | 3476.2ms/step | 5.8 samples/s
149
+ Step 36360/44000 | Loss: 2.8918 | LR: 3.75e-05 | 3476.3ms/step | 5.8 samples/s
150
+ Step 36370/44000 | Loss: 2.9388 | LR: 3.74e-05 | 3475.8ms/step | 5.8 samples/s
151
+ Step 36380/44000 | Loss: 2.9063 | LR: 3.73e-05 | 3480.7ms/step | 5.7 samples/s
152
+ Step 36390/44000 | Loss: 2.8806 | LR: 3.72e-05 | 3487.4ms/step | 5.7 samples/s
153
+ Step 36400/44000 | Loss: 2.9051 | LR: 3.71e-05 | 3485.7ms/step | 5.7 samples/s
154
+ Step 36410/44000 | Loss: 2.8702 | LR: 3.70e-05 | 3484.5ms/step | 5.7 samples/s
155
+ Step 36420/44000 | Loss: 2.8917 | LR: 3.69e-05 | 3481.3ms/step | 5.7 samples/s
156
+ Step 36430/44000 | Loss: 2.8480 | LR: 3.69e-05 | 3476.1ms/step | 5.8 samples/s
157
+ Step 36440/44000 | Loss: 2.8863 | LR: 3.68e-05 | 3483.9ms/step | 5.7 samples/s
158
+ Step 36450/44000 | Loss: 2.8868 | LR: 3.67e-05 | 3482.7ms/step | 5.7 samples/s
159
+ Step 36460/44000 | Loss: 2.8675 | LR: 3.66e-05 | 3487.7ms/step | 5.7 samples/s
160
+ Step 36470/44000 | Loss: 2.9005 | LR: 3.65e-05 | 3485.2ms/step | 5.7 samples/s
161
+ Step 36480/44000 | Loss: 2.9222 | LR: 3.64e-05 | 3484.0ms/step | 5.7 samples/s
162
+ Step 36490/44000 | Loss: 2.8807 | LR: 3.63e-05 | 3475.3ms/step | 5.8 samples/s
163
+ Step 36500/44000 | Loss: 2.9108 | LR: 3.62e-05 | 3476.6ms/step | 5.8 samples/s
164
+ Validation at step 36500: Loss = 2.9055, Perplexity = 18.27
165
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-36500
166
+ Step 36510/44000 | Loss: 2.9025 | LR: 3.61e-05 | 9603.3ms/step | 2.1 samples/s
167
+ Step 36520/44000 | Loss: 2.8982 | LR: 3.60e-05 | 3484.0ms/step | 5.7 samples/s
168
+ Step 36530/44000 | Loss: 2.9159 | LR: 3.59e-05 | 3482.7ms/step | 5.7 samples/s
169
+ Step 36540/44000 | Loss: 2.9199 | LR: 3.58e-05 | 3478.0ms/step | 5.8 samples/s
170
+ Step 36550/44000 | Loss: 2.9186 | LR: 3.57e-05 | 3482.6ms/step | 5.7 samples/s
171
+ Step 36560/44000 | Loss: 2.8868 | LR: 3.56e-05 | 3474.4ms/step | 5.8 samples/s
172
+ Step 36570/44000 | Loss: 2.8901 | LR: 3.55e-05 | 3480.3ms/step | 5.7 samples/s
173
+ Step 36580/44000 | Loss: 2.8746 | LR: 3.54e-05 | 3478.8ms/step | 5.7 samples/s
174
+ Step 36590/44000 | Loss: 2.9153 | LR: 3.53e-05 | 3480.4ms/step | 5.7 samples/s
175
+ Step 36600/44000 | Loss: 2.8691 | LR: 3.53e-05 | 3481.9ms/step | 5.7 samples/s
176
+ Step 36610/44000 | Loss: 2.8770 | LR: 3.52e-05 | 3483.8ms/step | 5.7 samples/s
177
+ Step 36620/44000 | Loss: 2.9115 | LR: 3.51e-05 | 3474.8ms/step | 5.8 samples/s
178
+ Step 36630/44000 | Loss: 2.8717 | LR: 3.50e-05 | 3476.0ms/step | 5.8 samples/s
179
+ Step 36640/44000 | Loss: 2.8681 | LR: 3.49e-05 | 3475.3ms/step | 5.8 samples/s
180
+ Step 36650/44000 | Loss: 2.8871 | LR: 3.48e-05 | 3479.7ms/step | 5.7 samples/s
181
+ Step 36660/44000 | Loss: 2.8686 | LR: 3.47e-05 | 3483.1ms/step | 5.7 samples/s
182
+ Step 36670/44000 | Loss: 2.8689 | LR: 3.46e-05 | 3473.1ms/step | 5.8 samples/s
183
+ Step 36680/44000 | Loss: 2.9211 | LR: 3.45e-05 | 3472.9ms/step | 5.8 samples/s
184
+ Step 36690/44000 | Loss: 2.9184 | LR: 3.44e-05 | 3473.7ms/step | 5.8 samples/s
185
+ Step 36700/44000 | Loss: 2.9048 | LR: 3.43e-05 | 3477.3ms/step | 5.8 samples/s
186
+ Step 36710/44000 | Loss: 2.9200 | LR: 3.42e-05 | 3477.3ms/step | 5.8 samples/s
187
+ Step 36720/44000 | Loss: 2.8961 | LR: 3.41e-05 | 3480.2ms/step | 5.7 samples/s
188
+ Step 36730/44000 | Loss: 2.8943 | LR: 3.41e-05 | 3474.3ms/step | 5.8 samples/s
189
+ Step 36740/44000 | Loss: 2.8934 | LR: 3.40e-05 | 3472.9ms/step | 5.8 samples/s
190
+ Step 36750/44000 | Loss: 2.9055 | LR: 3.39e-05 | 3474.1ms/step | 5.8 samples/s
191
+ Validation at step 36750: Loss = 2.9053, Perplexity = 18.27
192
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-36750
193
+ Step 36760/44000 | Loss: 2.8764 | LR: 3.38e-05 | 9601.5ms/step | 2.1 samples/s
194
+ Step 36770/44000 | Loss: 2.9234 | LR: 3.37e-05 | 3474.3ms/step | 5.8 samples/s
195
+ Step 36780/44000 | Loss: 2.9046 | LR: 3.36e-05 | 3476.1ms/step | 5.8 samples/s
196
+ Step 36790/44000 | Loss: 2.9215 | LR: 3.35e-05 | 3475.9ms/step | 5.8 samples/s
197
+ Step 36800/44000 | Loss: 2.8847 | LR: 3.34e-05 | 3475.4ms/step | 5.8 samples/s
198
+ Step 36810/44000 | Loss: 2.8742 | LR: 3.33e-05 | 3475.1ms/step | 5.8 samples/s
199
+ Step 36820/44000 | Loss: 2.9158 | LR: 3.32e-05 | 3475.4ms/step | 5.8 samples/s
200
+ Step 36830/44000 | Loss: 2.8988 | LR: 3.31e-05 | 3475.0ms/step | 5.8 samples/s
201
+ Step 36840/44000 | Loss: 2.8922 | LR: 3.31e-05 | 3472.8ms/step | 5.8 samples/s
202
+ Step 36850/44000 | Loss: 2.9096 | LR: 3.30e-05 | 3472.1ms/step | 5.8 samples/s
203
+ Step 36860/44000 | Loss: 2.8965 | LR: 3.29e-05 | 3474.8ms/step | 5.8 samples/s
204
+ Step 36870/44000 | Loss: 2.9080 | LR: 3.28e-05 | 3472.9ms/step | 5.8 samples/s
205
+ Step 36880/44000 | Loss: 2.9090 | LR: 3.27e-05 | 3473.6ms/step | 5.8 samples/s
206
+ Step 36890/44000 | Loss: 2.8881 | LR: 3.26e-05 | 3475.6ms/step | 5.8 samples/s
207
+ Step 36900/44000 | Loss: 2.8769 | LR: 3.25e-05 | 3475.6ms/step | 5.8 samples/s
208
+ Step 36910/44000 | Loss: 2.9147 | LR: 3.24e-05 | 3476.9ms/step | 5.8 samples/s
209
+ Step 36920/44000 | Loss: 2.9061 | LR: 3.23e-05 | 3477.2ms/step | 5.8 samples/s
210
+ Step 36930/44000 | Loss: 2.8829 | LR: 3.22e-05 | 3473.2ms/step | 5.8 samples/s
211
+ Step 36940/44000 | Loss: 2.8816 | LR: 3.22e-05 | 3472.0ms/step | 5.8 samples/s
212
+ Step 36950/44000 | Loss: 2.8610 | LR: 3.21e-05 | 3473.1ms/step | 5.8 samples/s
213
+ Step 36960/44000 | Loss: 2.8940 | LR: 3.20e-05 | 3483.5ms/step | 5.7 samples/s
214
+ Step 36970/44000 | Loss: 2.8896 | LR: 3.19e-05 | 3483.5ms/step | 5.7 samples/s
215
+ Step 36980/44000 | Loss: 2.8694 | LR: 3.18e-05 | 3478.0ms/step | 5.8 samples/s
216
+ Step 36990/44000 | Loss: 2.8993 | LR: 3.17e-05 | 3471.7ms/step | 5.8 samples/s
217
+ Step 37000/44000 | Loss: 2.9257 | LR: 3.16e-05 | 3473.4ms/step | 5.8 samples/s
218
+ Validation at step 37000: Loss = 2.9051, Perplexity = 18.27
219
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-37000
220
+ Step 37010/44000 | Loss: 2.8990 | LR: 3.15e-05 | 9587.2ms/step | 2.1 samples/s
221
+ Step 37020/44000 | Loss: 2.8878 | LR: 3.15e-05 | 3477.1ms/step | 5.8 samples/s
222
+ Step 37030/44000 | Loss: 2.8795 | LR: 3.14e-05 | 3473.5ms/step | 5.8 samples/s
223
+ Step 37040/44000 | Loss: 2.8941 | LR: 3.13e-05 | 3472.8ms/step | 5.8 samples/s
224
+ Step 37050/44000 | Loss: 2.8936 | LR: 3.12e-05 | 3472.7ms/step | 5.8 samples/s
225
+ Step 37060/44000 | Loss: 2.9161 | LR: 3.11e-05 | 3471.9ms/step | 5.8 samples/s
226
+ Step 37070/44000 | Loss: 2.8922 | LR: 3.10e-05 | 3472.7ms/step | 5.8 samples/s
227
+ Step 37080/44000 | Loss: 2.8996 | LR: 3.09e-05 | 3472.3ms/step | 5.8 samples/s
228
+ Step 37090/44000 | Loss: 2.9055 | LR: 3.08e-05 | 3472.6ms/step | 5.8 samples/s
229
+ Step 37100/44000 | Loss: 2.9008 | LR: 3.07e-05 | 3472.6ms/step | 5.8 samples/s
230
+ Step 37110/44000 | Loss: 2.8761 | LR: 3.07e-05 | 3472.7ms/step | 5.8 samples/s
231
+ Step 37120/44000 | Loss: 2.9000 | LR: 3.06e-05 | 3480.0ms/step | 5.7 samples/s
232
+ Step 37130/44000 | Loss: 2.9138 | LR: 3.05e-05 | 3479.9ms/step | 5.7 samples/s
233
+ Step 37140/44000 | Loss: 2.8838 | LR: 3.04e-05 | 3472.8ms/step | 5.8 samples/s
234
+ Step 37150/44000 | Loss: 2.8695 | LR: 3.03e-05 | 3472.0ms/step | 5.8 samples/s
235
+ Step 37160/44000 | Loss: 2.8995 | LR: 3.02e-05 | 3472.6ms/step | 5.8 samples/s
236
+ Step 37170/44000 | Loss: 2.8950 | LR: 3.01e-05 | 3479.1ms/step | 5.7 samples/s
237
+ Step 37180/44000 | Loss: 2.8711 | LR: 3.01e-05 | 3479.0ms/step | 5.7 samples/s
238
+ Step 37190/44000 | Loss: 2.9006 | LR: 3.00e-05 | 3474.8ms/step | 5.8 samples/s
239
+ Step 37200/44000 | Loss: 2.9109 | LR: 2.99e-05 | 3472.5ms/step | 5.8 samples/s
240
+ Step 37210/44000 | Loss: 2.8874 | LR: 2.98e-05 | 3475.2ms/step | 5.8 samples/s
241
+ Step 37220/44000 | Loss: 2.8848 | LR: 2.97e-05 | 3484.0ms/step | 5.7 samples/s
242
+ Step 37230/44000 | Loss: 2.9068 | LR: 2.96e-05 | 3479.3ms/step | 5.7 samples/s
243
+ Step 37240/44000 | Loss: 2.9185 | LR: 2.95e-05 | 3470.9ms/step | 5.8 samples/s
244
+ Step 37250/44000 | Loss: 2.9155 | LR: 2.95e-05 | 3471.7ms/step | 5.8 samples/s
245
+ Validation at step 37250: Loss = 2.9050, Perplexity = 18.27
246
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-37250
247
+ Step 37260/44000 | Loss: 2.8875 | LR: 2.94e-05 | 9581.3ms/step | 2.1 samples/s
248
+ Step 37270/44000 | Loss: 2.9212 | LR: 2.93e-05 | 3472.8ms/step | 5.8 samples/s
249
+ Step 37280/44000 | Loss: 2.8966 | LR: 2.92e-05 | 3472.9ms/step | 5.8 samples/s
250
+ Step 37290/44000 | Loss: 2.9047 | LR: 2.91e-05 | 3473.0ms/step | 5.8 samples/s
251
+ Step 37300/44000 | Loss: 2.9124 | LR: 2.90e-05 | 3472.0ms/step | 5.8 samples/s
252
+ Step 37310/44000 | Loss: 2.8836 | LR: 2.89e-05 | 3472.8ms/step | 5.8 samples/s
253
+ Step 37320/44000 | Loss: 2.9413 | LR: 2.89e-05 | 3472.3ms/step | 5.8 samples/s
254
+ Step 37330/44000 | Loss: 2.8747 | LR: 2.88e-05 | 3473.6ms/step | 5.8 samples/s
255
+ Step 37340/44000 | Loss: 2.8947 | LR: 2.87e-05 | 3472.2ms/step | 5.8 samples/s
256
+ Step 37350/44000 | Loss: 2.9017 | LR: 2.86e-05 | 3471.3ms/step | 5.8 samples/s
257
+ Step 37360/44000 | Loss: 2.9277 | LR: 2.85e-05 | 3472.3ms/step | 5.8 samples/s
258
+ Step 37370/44000 | Loss: 2.8873 | LR: 2.84e-05 | 3472.5ms/step | 5.8 samples/s
259
+ Step 37380/44000 | Loss: 2.8995 | LR: 2.84e-05 | 3471.8ms/step | 5.8 samples/s
260
+ Step 37390/44000 | Loss: 2.8950 | LR: 2.83e-05 | 3471.2ms/step | 5.8 samples/s
261
+ Step 37400/44000 | Loss: 2.9141 | LR: 2.82e-05 | 3470.9ms/step | 5.8 samples/s
262
+ Step 37410/44000 | Loss: 2.8713 | LR: 2.81e-05 | 3470.3ms/step | 5.8 samples/s
263
+ Step 37420/44000 | Loss: 2.8979 | LR: 2.80e-05 | 3471.7ms/step | 5.8 samples/s
264
+ Step 37430/44000 | Loss: 2.9027 | LR: 2.79e-05 | 3470.1ms/step | 5.8 samples/s
265
+ Step 37440/44000 | Loss: 2.8793 | LR: 2.78e-05 | 3471.7ms/step | 5.8 samples/s
266
+ Step 37450/44000 | Loss: 2.9208 | LR: 2.78e-05 | 3472.5ms/step | 5.8 samples/s
267
+ Step 37460/44000 | Loss: 2.8877 | LR: 2.77e-05 | 3471.3ms/step | 5.8 samples/s
268
+ Step 37470/44000 | Loss: 2.8755 | LR: 2.76e-05 | 3473.1ms/step | 5.8 samples/s
269
+ Step 37480/44000 | Loss: 2.9141 | LR: 2.75e-05 | 3471.0ms/step | 5.8 samples/s
270
+ Step 37490/44000 | Loss: 2.8955 | LR: 2.74e-05 | 3471.6ms/step | 5.8 samples/s
271
+ Step 37500/44000 | Loss: 2.8695 | LR: 2.74e-05 | 3472.0ms/step | 5.8 samples/s
272
+ Validation at step 37500: Loss = 2.9049, Perplexity = 18.26
273
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-37500
274
+ Step 37510/44000 | Loss: 2.9048 | LR: 2.73e-05 | 9582.7ms/step | 2.1 samples/s
275
+ Step 37520/44000 | Loss: 2.9224 | LR: 2.72e-05 | 3468.3ms/step | 5.8 samples/s
276
+ Step 37530/44000 | Loss: 2.9033 | LR: 2.71e-05 | 3470.5ms/step | 5.8 samples/s
277
+ Step 37540/44000 | Loss: 2.9224 | LR: 2.70e-05 | 3470.5ms/step | 5.8 samples/s
278
+ Step 37550/44000 | Loss: 2.8877 | LR: 2.69e-05 | 3469.3ms/step | 5.8 samples/s
279
+ Step 37560/44000 | Loss: 2.8962 | LR: 2.69e-05 | 3472.8ms/step | 5.8 samples/s
280
+ Step 37570/44000 | Loss: 2.9055 | LR: 2.68e-05 | 3471.1ms/step | 5.8 samples/s
281
+ Step 37580/44000 | Loss: 2.8895 | LR: 2.67e-05 | 3471.1ms/step | 5.8 samples/s
282
+ Step 37590/44000 | Loss: 2.9023 | LR: 2.66e-05 | 3470.0ms/step | 5.8 samples/s
283
+ Step 37600/44000 | Loss: 2.8883 | LR: 2.65e-05 | 3470.0ms/step | 5.8 samples/s
284
+ Step 37610/44000 | Loss: 2.8895 | LR: 2.65e-05 | 3481.6ms/step | 5.7 samples/s
285
+ Step 37620/44000 | Loss: 2.8989 | LR: 2.64e-05 | 3472.3ms/step | 5.8 samples/s
286
+ Step 37630/44000 | Loss: 2.9008 | LR: 2.63e-05 | 3472.2ms/step | 5.8 samples/s
287
+ Step 37640/44000 | Loss: 2.9061 | LR: 2.62e-05 | 3470.8ms/step | 5.8 samples/s
288
+ Step 37650/44000 | Loss: 2.9065 | LR: 2.61e-05 | 3469.8ms/step | 5.8 samples/s
289
+ Step 37660/44000 | Loss: 2.9146 | LR: 2.60e-05 | 3470.9ms/step | 5.8 samples/s
290
+ Step 37670/44000 | Loss: 2.8673 | LR: 2.60e-05 | 3472.4ms/step | 5.8 samples/s
291
+ Step 37680/44000 | Loss: 2.9118 | LR: 2.59e-05 | 3471.5ms/step | 5.8 samples/s
292
+ Step 37690/44000 | Loss: 2.9094 | LR: 2.58e-05 | 3471.3ms/step | 5.8 samples/s
293
+ Step 37700/44000 | Loss: 2.8831 | LR: 2.57e-05 | 3470.1ms/step | 5.8 samples/s
294
+ Step 37710/44000 | Loss: 2.8693 | LR: 2.56e-05 | 3473.6ms/step | 5.8 samples/s
295
+ Step 37720/44000 | Loss: 2.8896 | LR: 2.56e-05 | 3481.6ms/step | 5.7 samples/s
296
+ Step 37730/44000 | Loss: 2.8897 | LR: 2.55e-05 | 3482.7ms/step | 5.7 samples/s
297
+ Step 37740/44000 | Loss: 2.9227 | LR: 2.54e-05 | 3482.6ms/step | 5.7 samples/s
298
+ Step 37750/44000 | Loss: 2.8851 | LR: 2.53e-05 | 3483.6ms/step | 5.7 samples/s
299
+ Validation at step 37750: Loss = 2.9048, Perplexity = 18.26
300
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-37750
301
+ Step 37760/44000 | Loss: 2.8935 | LR: 2.52e-05 | 9628.0ms/step | 2.1 samples/s
302
+ Step 37770/44000 | Loss: 2.9040 | LR: 2.52e-05 | 3471.7ms/step | 5.8 samples/s
303
+ Step 37780/44000 | Loss: 2.8668 | LR: 2.51e-05 | 3470.1ms/step | 5.8 samples/s
304
+ Step 37790/44000 | Loss: 2.8929 | LR: 2.50e-05 | 3470.2ms/step | 5.8 samples/s
305
+ Step 37800/44000 | Loss: 2.8943 | LR: 2.49e-05 | 3470.6ms/step | 5.8 samples/s
306
+ Step 37810/44000 | Loss: 2.8850 | LR: 2.48e-05 | 3471.8ms/step | 5.8 samples/s
307
+ Step 37820/44000 | Loss: 2.8988 | LR: 2.48e-05 | 3477.0ms/step | 5.8 samples/s
308
+ Step 37830/44000 | Loss: 2.9162 | LR: 2.47e-05 | 3473.8ms/step | 5.8 samples/s
309
+ Step 37840/44000 | Loss: 2.8584 | LR: 2.46e-05 | 3473.6ms/step | 5.8 samples/s
310
+ Step 37850/44000 | Loss: 2.9030 | LR: 2.45e-05 | 3475.3ms/step | 5.8 samples/s
311
+ Step 37860/44000 | Loss: 2.8712 | LR: 2.45e-05 | 3473.7ms/step | 5.8 samples/s
312
+ Step 37870/44000 | Loss: 2.8866 | LR: 2.44e-05 | 3473.0ms/step | 5.8 samples/s
313
+ Step 37880/44000 | Loss: 2.8628 | LR: 2.43e-05 | 3471.1ms/step | 5.8 samples/s
314
+ Step 37890/44000 | Loss: 2.8837 | LR: 2.42e-05 | 3472.7ms/step | 5.8 samples/s
315
+ Step 37900/44000 | Loss: 2.9144 | LR: 2.41e-05 | 3471.9ms/step | 5.8 samples/s
316
+ Step 37910/44000 | Loss: 2.8912 | LR: 2.41e-05 | 3472.7ms/step | 5.8 samples/s
317
+ Step 37920/44000 | Loss: 2.8924 | LR: 2.40e-05 | 3471.6ms/step | 5.8 samples/s
318
+ Step 37930/44000 | Loss: 2.9034 | LR: 2.39e-05 | 3471.3ms/step | 5.8 samples/s
319
+ Step 37940/44000 | Loss: 2.8973 | LR: 2.38e-05 | 3480.4ms/step | 5.7 samples/s
320
+ Step 37950/44000 | Loss: 2.8728 | LR: 2.38e-05 | 3478.3ms/step | 5.7 samples/s
321
+ Step 37960/44000 | Loss: 2.9302 | LR: 2.37e-05 | 3473.8ms/step | 5.8 samples/s
322
+ Step 37970/44000 | Loss: 2.8645 | LR: 2.36e-05 | 3472.8ms/step | 5.8 samples/s
323
+ Step 37980/44000 | Loss: 2.8683 | LR: 2.35e-05 | 3475.8ms/step | 5.8 samples/s
324
+ Step 37990/44000 | Loss: 2.8750 | LR: 2.34e-05 | 3471.0ms/step | 5.8 samples/s
325
+ Step 38000/44000 | Loss: 2.8982 | LR: 2.34e-05 | 3470.8ms/step | 5.8 samples/s
326
+ Validation at step 38000: Loss = 2.9046, Perplexity = 18.26
327
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-38000
328
+ Step 38010/44000 | Loss: 2.9115 | LR: 2.33e-05 | 9587.1ms/step | 2.1 samples/s
329
+ Step 38020/44000 | Loss: 2.9180 | LR: 2.32e-05 | 3473.4ms/step | 5.8 samples/s
330
+ Step 38030/44000 | Loss: 2.8712 | LR: 2.31e-05 | 3472.8ms/step | 5.8 samples/s
331
+ Step 38040/44000 | Loss: 2.8950 | LR: 2.31e-05 | 3471.8ms/step | 5.8 samples/s
332
+ Step 38050/44000 | Loss: 2.9017 | LR: 2.30e-05 | 3472.4ms/step | 5.8 samples/s
333
+ Step 38060/44000 | Loss: 2.9151 | LR: 2.29e-05 | 3487.5ms/step | 5.7 samples/s
334
+ Step 38070/44000 | Loss: 2.8925 | LR: 2.28e-05 | 3497.9ms/step | 5.7 samples/s
335
+ Step 38080/44000 | Loss: 2.8574 | LR: 2.28e-05 | 3476.1ms/step | 5.8 samples/s
336
+ Step 38090/44000 | Loss: 2.8714 | LR: 2.27e-05 | 3477.3ms/step | 5.8 samples/s
337
+ Step 38100/44000 | Loss: 2.9127 | LR: 2.26e-05 | 3470.6ms/step | 5.8 samples/s
338
+ Step 38110/44000 | Loss: 2.8718 | LR: 2.25e-05 | 3477.9ms/step | 5.8 samples/s
339
+ Step 38120/44000 | Loss: 2.8961 | LR: 2.25e-05 | 3476.6ms/step | 5.8 samples/s
340
+ Step 38130/44000 | Loss: 2.8838 | LR: 2.24e-05 | 3474.7ms/step | 5.8 samples/s
341
+ Step 38140/44000 | Loss: 2.8694 | LR: 2.23e-05 | 3476.4ms/step | 5.8 samples/s
342
+ Step 38150/44000 | Loss: 2.8992 | LR: 2.22e-05 | 3473.3ms/step | 5.8 samples/s
343
+ Step 38160/44000 | Loss: 2.9029 | LR: 2.22e-05 | 3472.8ms/step | 5.8 samples/s
344
+ Step 38170/44000 | Loss: 2.8934 | LR: 2.21e-05 | 3475.9ms/step | 5.8 samples/s
345
+ Step 38180/44000 | Loss: 2.9142 | LR: 2.20e-05 | 3475.1ms/step | 5.8 samples/s
346
+ Step 38190/44000 | Loss: 2.9187 | LR: 2.19e-05 | 3476.1ms/step | 5.8 samples/s
347
+ Step 38200/44000 | Loss: 2.8696 | LR: 2.19e-05 | 3472.4ms/step | 5.8 samples/s
348
+ Step 38210/44000 | Loss: 2.8799 | LR: 2.18e-05 | 3471.4ms/step | 5.8 samples/s
349
+ Step 38220/44000 | Loss: 2.9168 | LR: 2.17e-05 | 3475.8ms/step | 5.8 samples/s
350
+ Step 38230/44000 | Loss: 2.9164 | LR: 2.16e-05 | 3470.2ms/step | 5.8 samples/s
351
+ Step 38240/44000 | Loss: 2.9271 | LR: 2.16e-05 | 3471.4ms/step | 5.8 samples/s
352
+ Step 38250/44000 | Loss: 2.8884 | LR: 2.15e-05 | 3470.6ms/step | 5.8 samples/s
353
+ Validation at step 38250: Loss = 2.9046, Perplexity = 18.26
354
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-38250
355
+ Step 38260/44000 | Loss: 2.9031 | LR: 2.14e-05 | 9585.5ms/step | 2.1 samples/s
356
+ Step 38270/44000 | Loss: 2.9385 | LR: 2.13e-05 | 3473.1ms/step | 5.8 samples/s
357
+ Step 38280/44000 | Loss: 2.9088 | LR: 2.13e-05 | 3473.7ms/step | 5.8 samples/s
358
+ Step 38290/44000 | Loss: 2.8935 | LR: 2.12e-05 | 3471.0ms/step | 5.8 samples/s
359
+ Step 38300/44000 | Loss: 2.9213 | LR: 2.11e-05 | 3473.0ms/step | 5.8 samples/s
360
+ Step 38310/44000 | Loss: 2.9244 | LR: 2.11e-05 | 3473.5ms/step | 5.8 samples/s
361
+ Step 38320/44000 | Loss: 2.8622 | LR: 2.10e-05 | 3472.1ms/step | 5.8 samples/s
362
+ Step 38330/44000 | Loss: 2.8811 | LR: 2.09e-05 | 3472.9ms/step | 5.8 samples/s
363
+ Step 38340/44000 | Loss: 2.9070 | LR: 2.08e-05 | 3470.2ms/step | 5.8 samples/s
364
+ Step 38350/44000 | Loss: 2.8877 | LR: 2.08e-05 | 3477.6ms/step | 5.8 samples/s
365
+ Step 38360/44000 | Loss: 2.8843 | LR: 2.07e-05 | 3482.8ms/step | 5.7 samples/s
366
+ Step 38370/44000 | Loss: 2.8843 | LR: 2.06e-05 | 3476.4ms/step | 5.8 samples/s
367
+ Step 38380/44000 | Loss: 2.8780 | LR: 2.05e-05 | 3472.2ms/step | 5.8 samples/s
368
+ Step 38390/44000 | Loss: 2.8837 | LR: 2.05e-05 | 3469.3ms/step | 5.8 samples/s
369
+ Step 38400/44000 | Loss: 2.8617 | LR: 2.04e-05 | 3468.0ms/step | 5.8 samples/s
370
+ Step 38410/44000 | Loss: 2.9039 | LR: 2.03e-05 | 3463.1ms/step | 5.8 samples/s
371
+ Step 38420/44000 | Loss: 2.8605 | LR: 2.03e-05 | 3464.7ms/step | 5.8 samples/s
372
+ Step 38430/44000 | Loss: 2.8757 | LR: 2.02e-05 | 3462.2ms/step | 5.8 samples/s
373
+ Step 38440/44000 | Loss: 2.9042 | LR: 2.01e-05 | 3460.7ms/step | 5.8 samples/s
374
+ Step 38450/44000 | Loss: 2.8975 | LR: 2.00e-05 | 3459.4ms/step | 5.8 samples/s
375
+ Step 38460/44000 | Loss: 2.9274 | LR: 2.00e-05 | 3460.9ms/step | 5.8 samples/s
376
+ Step 38470/44000 | Loss: 2.9049 | LR: 1.99e-05 | 3459.3ms/step | 5.8 samples/s
377
+ Step 38480/44000 | Loss: 2.8909 | LR: 1.98e-05 | 3461.1ms/step | 5.8 samples/s
378
+ Step 38490/44000 | Loss: 2.8893 | LR: 1.98e-05 | 3462.2ms/step | 5.8 samples/s
379
+ Step 38500/44000 | Loss: 2.8972 | LR: 1.97e-05 | 3461.0ms/step | 5.8 samples/s
380
+ Validation at step 38500: Loss = 2.9045, Perplexity = 18.26
381
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-38500
382
+ Step 38510/44000 | Loss: 2.8819 | LR: 1.96e-05 | 9556.0ms/step | 2.1 samples/s
383
+ Step 38520/44000 | Loss: 2.9002 | LR: 1.95e-05 | 3463.5ms/step | 5.8 samples/s
384
+ Step 38530/44000 | Loss: 2.8845 | LR: 1.95e-05 | 3465.5ms/step | 5.8 samples/s
385
+ Step 38540/44000 | Loss: 2.9088 | LR: 1.94e-05 | 3466.2ms/step | 5.8 samples/s
386
+ Step 38550/44000 | Loss: 2.9180 | LR: 1.93e-05 | 3462.6ms/step | 5.8 samples/s
387
+ Step 38560/44000 | Loss: 2.8925 | LR: 1.93e-05 | 3462.9ms/step | 5.8 samples/s
388
+ Step 38570/44000 | Loss: 2.9532 | LR: 1.92e-05 | 3465.3ms/step | 5.8 samples/s
389
+ Step 38580/44000 | Loss: 2.8854 | LR: 1.91e-05 | 3459.6ms/step | 5.8 samples/s
390
+ Step 38590/44000 | Loss: 2.8773 | LR: 1.91e-05 | 3460.8ms/step | 5.8 samples/s
391
+ Step 38600/44000 | Loss: 2.8976 | LR: 1.90e-05 | 3461.4ms/step | 5.8 samples/s
392
+ Step 38610/44000 | Loss: 2.8858 | LR: 1.89e-05 | 3461.7ms/step | 5.8 samples/s
393
+ Step 38620/44000 | Loss: 2.8975 | LR: 1.88e-05 | 3463.8ms/step | 5.8 samples/s
394
+ Step 38630/44000 | Loss: 2.8801 | LR: 1.88e-05 | 3463.5ms/step | 5.8 samples/s
395
+ Step 38640/44000 | Loss: 2.8886 | LR: 1.87e-05 | 3463.3ms/step | 5.8 samples/s
396
+ Step 38650/44000 | Loss: 2.9012 | LR: 1.86e-05 | 3463.1ms/step | 5.8 samples/s
397
+ Step 38660/44000 | Loss: 2.8551 | LR: 1.86e-05 | 3461.1ms/step | 5.8 samples/s
398
+ Step 38670/44000 | Loss: 2.9141 | LR: 1.85e-05 | 3462.1ms/step | 5.8 samples/s
399
+ Step 38680/44000 | Loss: 2.8913 | LR: 1.84e-05 | 3459.9ms/step | 5.8 samples/s
400
+ Step 38690/44000 | Loss: 2.8870 | LR: 1.84e-05 | 3460.9ms/step | 5.8 samples/s
401
+ Step 38700/44000 | Loss: 2.8876 | LR: 1.83e-05 | 3464.4ms/step | 5.8 samples/s
402
+ Step 38710/44000 | Loss: 2.9097 | LR: 1.82e-05 | 3462.7ms/step | 5.8 samples/s
403
+ Step 38720/44000 | Loss: 2.8642 | LR: 1.82e-05 | 3461.0ms/step | 5.8 samples/s
404
+ Step 38730/44000 | Loss: 2.9125 | LR: 1.81e-05 | 3460.7ms/step | 5.8 samples/s
405
+ Step 38740/44000 | Loss: 2.9066 | LR: 1.80e-05 | 3462.7ms/step | 5.8 samples/s
406
+ Step 38750/44000 | Loss: 2.8985 | LR: 1.80e-05 | 3462.2ms/step | 5.8 samples/s
407
+ Validation at step 38750: Loss = 2.9045, Perplexity = 18.26
408
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-38750
409
+ Step 38760/44000 | Loss: 2.8973 | LR: 1.79e-05 | 9560.0ms/step | 2.1 samples/s
410
+ Step 38770/44000 | Loss: 2.8908 | LR: 1.78e-05 | 3464.3ms/step | 5.8 samples/s
411
+ Step 38780/44000 | Loss: 2.8872 | LR: 1.78e-05 | 3462.6ms/step | 5.8 samples/s
412
+ Step 38790/44000 | Loss: 2.9053 | LR: 1.77e-05 | 3462.4ms/step | 5.8 samples/s
413
+ Step 38800/44000 | Loss: 2.8911 | LR: 1.76e-05 | 3461.7ms/step | 5.8 samples/s
414
+ Step 38810/44000 | Loss: 2.8756 | LR: 1.76e-05 | 3464.6ms/step | 5.8 samples/s
415
+ Step 38820/44000 | Loss: 2.8843 | LR: 1.75e-05 | 3462.3ms/step | 5.8 samples/s
416
+ Step 38830/44000 | Loss: 2.9076 | LR: 1.74e-05 | 3462.3ms/step | 5.8 samples/s
417
+ Step 38840/44000 | Loss: 2.8746 | LR: 1.74e-05 | 3464.2ms/step | 5.8 samples/s
418
+ Step 38850/44000 | Loss: 2.9265 | LR: 1.73e-05 | 3461.0ms/step | 5.8 samples/s
419
+ Step 38860/44000 | Loss: 2.9287 | LR: 1.72e-05 | 3467.3ms/step | 5.8 samples/s
420
+ Step 38870/44000 | Loss: 2.9164 | LR: 1.72e-05 | 3464.6ms/step | 5.8 samples/s
421
+ Step 38880/44000 | Loss: 2.9019 | LR: 1.71e-05 | 3463.1ms/step | 5.8 samples/s
422
+ Step 38890/44000 | Loss: 2.8984 | LR: 1.70e-05 | 3462.3ms/step | 5.8 samples/s
423
+ Step 38900/44000 | Loss: 2.9492 | LR: 1.70e-05 | 3462.2ms/step | 5.8 samples/s
424
+ Step 38910/44000 | Loss: 2.8793 | LR: 1.69e-05 | 3461.8ms/step | 5.8 samples/s
425
+ Step 38920/44000 | Loss: 2.8826 | LR: 1.68e-05 | 3462.5ms/step | 5.8 samples/s
426
+ Step 38930/44000 | Loss: 2.8878 | LR: 1.68e-05 | 3465.5ms/step | 5.8 samples/s
427
+ Step 38940/44000 | Loss: 2.9023 | LR: 1.67e-05 | 3462.4ms/step | 5.8 samples/s
428
+ Step 38950/44000 | Loss: 2.8754 | LR: 1.66e-05 | 3462.4ms/step | 5.8 samples/s
429
+ Step 38960/44000 | Loss: 2.8933 | LR: 1.66e-05 | 3466.3ms/step | 5.8 samples/s
430
+ Step 38970/44000 | Loss: 2.8566 | LR: 1.65e-05 | 3464.0ms/step | 5.8 samples/s
431
+ Step 38980/44000 | Loss: 2.9002 | LR: 1.64e-05 | 3460.5ms/step | 5.8 samples/s
432
+ Step 38990/44000 | Loss: 2.9041 | LR: 1.64e-05 | 3461.8ms/step | 5.8 samples/s
433
+ Step 39000/44000 | Loss: 2.8942 | LR: 1.63e-05 | 3462.3ms/step | 5.8 samples/s
434
+ Validation at step 39000: Loss = 2.9044, Perplexity = 18.25
435
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-39000
436
+ Step 39010/44000 | Loss: 2.8814 | LR: 1.62e-05 | 9555.1ms/step | 2.1 samples/s
437
+ Step 39020/44000 | Loss: 2.8918 | LR: 1.62e-05 | 3464.4ms/step | 5.8 samples/s
438
+ Step 39030/44000 | Loss: 2.8808 | LR: 1.61e-05 | 3462.9ms/step | 5.8 samples/s
439
+ Step 39040/44000 | Loss: 2.8909 | LR: 1.61e-05 | 3466.7ms/step | 5.8 samples/s
440
+ Step 39050/44000 | Loss: 2.9158 | LR: 1.60e-05 | 3468.4ms/step | 5.8 samples/s
441
+ Step 39060/44000 | Loss: 2.9066 | LR: 1.59e-05 | 3461.7ms/step | 5.8 samples/s
442
+ Step 39070/44000 | Loss: 2.9008 | LR: 1.59e-05 | 3458.9ms/step | 5.8 samples/s
443
+ Step 39080/44000 | Loss: 2.8821 | LR: 1.58e-05 | 3459.0ms/step | 5.8 samples/s
444
+ Step 39090/44000 | Loss: 2.8832 | LR: 1.57e-05 | 3460.0ms/step | 5.8 samples/s
445
+ Step 39100/44000 | Loss: 2.8638 | LR: 1.57e-05 | 3460.8ms/step | 5.8 samples/s
446
+ Step 39110/44000 | Loss: 2.9052 | LR: 1.56e-05 | 3462.0ms/step | 5.8 samples/s
447
+ Step 39120/44000 | Loss: 2.8731 | LR: 1.55e-05 | 3463.9ms/step | 5.8 samples/s
448
+ Step 39130/44000 | Loss: 2.8884 | LR: 1.55e-05 | 3461.1ms/step | 5.8 samples/s
449
+ Step 39140/44000 | Loss: 2.8947 | LR: 1.54e-05 | 3461.3ms/step | 5.8 samples/s
450
+ Step 39150/44000 | Loss: 2.8625 | LR: 1.54e-05 | 3463.3ms/step | 5.8 samples/s
451
+ Step 39160/44000 | Loss: 2.9412 | LR: 1.53e-05 | 3463.0ms/step | 5.8 samples/s
452
+ Step 39170/44000 | Loss: 2.8919 | LR: 1.52e-05 | 3464.6ms/step | 5.8 samples/s
453
+ Step 39180/44000 | Loss: 2.9048 | LR: 1.52e-05 | 3461.4ms/step | 5.8 samples/s
454
+ Step 39190/44000 | Loss: 2.9074 | LR: 1.51e-05 | 3469.7ms/step | 5.8 samples/s
455
+ Step 39200/44000 | Loss: 2.8749 | LR: 1.50e-05 | 3462.3ms/step | 5.8 samples/s
456
+ Step 39210/44000 | Loss: 2.8725 | LR: 1.50e-05 | 3458.7ms/step | 5.8 samples/s
457
+ Step 39220/44000 | Loss: 2.8992 | LR: 1.49e-05 | 3462.3ms/step | 5.8 samples/s
458
+ Step 39230/44000 | Loss: 2.8873 | LR: 1.49e-05 | 3463.0ms/step | 5.8 samples/s
459
+ Step 39240/44000 | Loss: 2.8911 | LR: 1.48e-05 | 3459.6ms/step | 5.8 samples/s
460
+ Step 39250/44000 | Loss: 2.9300 | LR: 1.47e-05 | 3462.7ms/step | 5.8 samples/s
461
+ Validation at step 39250: Loss = 2.9043, Perplexity = 18.25
462
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-39250
463
+ Step 39260/44000 | Loss: 2.9190 | LR: 1.47e-05 | 9544.2ms/step | 2.1 samples/s
464
+ Step 39270/44000 | Loss: 2.8966 | LR: 1.46e-05 | 3461.8ms/step | 5.8 samples/s
465
+ Step 39280/44000 | Loss: 2.9175 | LR: 1.46e-05 | 3462.6ms/step | 5.8 samples/s
466
+ Step 39290/44000 | Loss: 2.8735 | LR: 1.45e-05 | 3465.7ms/step | 5.8 samples/s
467
+ Step 39300/44000 | Loss: 2.8809 | LR: 1.44e-05 | 3463.5ms/step | 5.8 samples/s
468
+ Step 39310/44000 | Loss: 2.8937 | LR: 1.44e-05 | 3461.0ms/step | 5.8 samples/s
469
+ Step 39320/44000 | Loss: 2.9062 | LR: 1.43e-05 | 3464.2ms/step | 5.8 samples/s
470
+ Step 39330/44000 | Loss: 2.9178 | LR: 1.42e-05 | 3461.6ms/step | 5.8 samples/s
471
+ Step 39340/44000 | Loss: 2.9060 | LR: 1.42e-05 | 3460.8ms/step | 5.8 samples/s
472
+ Step 39350/44000 | Loss: 2.8907 | LR: 1.41e-05 | 3461.1ms/step | 5.8 samples/s
473
+ Step 39360/44000 | Loss: 2.8984 | LR: 1.41e-05 | 3461.9ms/step | 5.8 samples/s
474
+ Step 39370/44000 | Loss: 2.9287 | LR: 1.40e-05 | 3465.9ms/step | 5.8 samples/s
475
+ Step 39380/44000 | Loss: 2.9249 | LR: 1.39e-05 | 3462.4ms/step | 5.8 samples/s
476
+ Step 39390/44000 | Loss: 2.8802 | LR: 1.39e-05 | 3464.7ms/step | 5.8 samples/s
477
+ Step 39400/44000 | Loss: 2.9144 | LR: 1.38e-05 | 3461.4ms/step | 5.8 samples/s
478
+ Step 39410/44000 | Loss: 2.9327 | LR: 1.38e-05 | 3461.5ms/step | 5.8 samples/s
479
+ Step 39420/44000 | Loss: 2.9346 | LR: 1.37e-05 | 3460.5ms/step | 5.8 samples/s
480
+ Step 39430/44000 | Loss: 2.8924 | LR: 1.36e-05 | 3462.9ms/step | 5.8 samples/s
481
+ Step 39440/44000 | Loss: 2.8915 | LR: 1.36e-05 | 3464.0ms/step | 5.8 samples/s
482
+ Step 39450/44000 | Loss: 2.9047 | LR: 1.35e-05 | 3462.7ms/step | 5.8 samples/s
483
+ Step 39460/44000 | Loss: 2.8894 | LR: 1.35e-05 | 3461.4ms/step | 5.8 samples/s
484
+ Step 39470/44000 | Loss: 2.8888 | LR: 1.34e-05 | 3459.9ms/step | 5.8 samples/s
485
+ Step 39480/44000 | Loss: 2.9003 | LR: 1.34e-05 | 3461.8ms/step | 5.8 samples/s
486
+ Step 39490/44000 | Loss: 2.8997 | LR: 1.33e-05 | 3465.1ms/step | 5.8 samples/s
487
+ Step 39500/44000 | Loss: 2.9398 | LR: 1.32e-05 | 3461.7ms/step | 5.8 samples/s
488
+ Validation at step 39500: Loss = 2.9043, Perplexity = 18.25
489
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-39500
490
+ Step 39510/44000 | Loss: 2.8810 | LR: 1.32e-05 | 9552.7ms/step | 2.1 samples/s
491
+ Step 39520/44000 | Loss: 2.8830 | LR: 1.31e-05 | 3461.7ms/step | 5.8 samples/s
492
+ Step 39530/44000 | Loss: 2.8803 | LR: 1.31e-05 | 3461.7ms/step | 5.8 samples/s
493
+ Step 39540/44000 | Loss: 2.9097 | LR: 1.30e-05 | 3460.2ms/step | 5.8 samples/s
494
+ Step 39550/44000 | Loss: 2.8846 | LR: 1.29e-05 | 3460.3ms/step | 5.8 samples/s
495
+ Step 39560/44000 | Loss: 2.8721 | LR: 1.29e-05 | 3463.7ms/step | 5.8 samples/s
496
+ Step 39570/44000 | Loss: 2.9060 | LR: 1.28e-05 | 3461.5ms/step | 5.8 samples/s
497
+ Step 39580/44000 | Loss: 2.8980 | LR: 1.28e-05 | 3464.8ms/step | 5.8 samples/s
498
+ Step 39590/44000 | Loss: 2.9040 | LR: 1.27e-05 | 3461.4ms/step | 5.8 samples/s
499
+ Step 39600/44000 | Loss: 2.9206 | LR: 1.27e-05 | 3460.3ms/step | 5.8 samples/s
500
+ Step 39610/44000 | Loss: 2.8733 | LR: 1.26e-05 | 3461.7ms/step | 5.8 samples/s
501
+ Step 39620/44000 | Loss: 2.8819 | LR: 1.25e-05 | 3459.0ms/step | 5.8 samples/s
502
+ Step 39630/44000 | Loss: 2.9079 | LR: 1.25e-05 | 3462.4ms/step | 5.8 samples/s
503
+ Step 39640/44000 | Loss: 2.8927 | LR: 1.24e-05 | 3458.4ms/step | 5.8 samples/s
504
+ Step 39650/44000 | Loss: 2.8949 | LR: 1.24e-05 | 3458.9ms/step | 5.8 samples/s
505
+ Step 39660/44000 | Loss: 2.9051 | LR: 1.23e-05 | 3460.2ms/step | 5.8 samples/s
506
+ Step 39670/44000 | Loss: 2.8786 | LR: 1.23e-05 | 3459.6ms/step | 5.8 samples/s
507
+ Step 39680/44000 | Loss: 2.9062 | LR: 1.22e-05 | 3460.6ms/step | 5.8 samples/s
508
+ Step 39690/44000 | Loss: 2.8847 | LR: 1.22e-05 | 3460.1ms/step | 5.8 samples/s
509
+ Step 39700/44000 | Loss: 2.9274 | LR: 1.21e-05 | 3460.6ms/step | 5.8 samples/s
510
+ Step 39710/44000 | Loss: 2.8859 | LR: 1.20e-05 | 3460.1ms/step | 5.8 samples/s
511
+ Step 39720/44000 | Loss: 2.8743 | LR: 1.20e-05 | 3459.4ms/step | 5.8 samples/s
512
+ Step 39730/44000 | Loss: 2.8899 | LR: 1.19e-05 | 3464.5ms/step | 5.8 samples/s
513
+ Step 39740/44000 | Loss: 2.8851 | LR: 1.19e-05 | 3463.1ms/step | 5.8 samples/s
514
+ Step 39750/44000 | Loss: 2.8919 | LR: 1.18e-05 | 3461.1ms/step | 5.8 samples/s
515
+ Validation at step 39750: Loss = 2.9043, Perplexity = 18.25
516
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-39750
517
+ Step 39760/44000 | Loss: 2.8684 | LR: 1.18e-05 | 9567.5ms/step | 2.1 samples/s
518
+ Step 39770/44000 | Loss: 2.8985 | LR: 1.17e-05 | 3464.5ms/step | 5.8 samples/s
519
+ Step 39780/44000 | Loss: 2.8660 | LR: 1.17e-05 | 3461.5ms/step | 5.8 samples/s
520
+ Step 39790/44000 | Loss: 2.8663 | LR: 1.16e-05 | 3459.8ms/step | 5.8 samples/s
521
+ Step 39800/44000 | Loss: 2.9027 | LR: 1.15e-05 | 3461.1ms/step | 5.8 samples/s
522
+ Step 39810/44000 | Loss: 2.8987 | LR: 1.15e-05 | 3461.9ms/step | 5.8 samples/s
523
+ Step 39820/44000 | Loss: 2.9008 | LR: 1.14e-05 | 3467.6ms/step | 5.8 samples/s
524
+ Step 39830/44000 | Loss: 2.8964 | LR: 1.14e-05 | 3465.2ms/step | 5.8 samples/s
525
+ Step 39840/44000 | Loss: 2.8858 | LR: 1.13e-05 | 3463.6ms/step | 5.8 samples/s
526
+ Step 39850/44000 | Loss: 2.8938 | LR: 1.13e-05 | 3461.0ms/step | 5.8 samples/s
527
+ Step 39860/44000 | Loss: 2.9042 | LR: 1.12e-05 | 3460.5ms/step | 5.8 samples/s
528
+ Step 39870/44000 | Loss: 2.8929 | LR: 1.12e-05 | 3460.3ms/step | 5.8 samples/s
529
+ Step 39880/44000 | Loss: 2.8897 | LR: 1.11e-05 | 3460.3ms/step | 5.8 samples/s
530
+ Step 39890/44000 | Loss: 2.8936 | LR: 1.11e-05 | 3460.0ms/step | 5.8 samples/s
531
+ Step 39900/44000 | Loss: 2.9050 | LR: 1.10e-05 | 3460.9ms/step | 5.8 samples/s
532
+ Step 39910/44000 | Loss: 2.9016 | LR: 1.10e-05 | 3463.5ms/step | 5.8 samples/s
533
+ Step 39920/44000 | Loss: 2.8963 | LR: 1.09e-05 | 3463.4ms/step | 5.8 samples/s
534
+ Step 39930/44000 | Loss: 2.9004 | LR: 1.08e-05 | 3459.9ms/step | 5.8 samples/s
535
+ Step 39940/44000 | Loss: 2.8999 | LR: 1.08e-05 | 3460.6ms/step | 5.8 samples/s
536
+ Step 39950/44000 | Loss: 2.9012 | LR: 1.07e-05 | 3460.5ms/step | 5.8 samples/s
537
+ Step 39960/44000 | Loss: 2.8797 | LR: 1.07e-05 | 3460.0ms/step | 5.8 samples/s
538
+ Step 39970/44000 | Loss: 2.9029 | LR: 1.06e-05 | 3459.3ms/step | 5.8 samples/s
539
+ Step 39980/44000 | Loss: 2.9001 | LR: 1.06e-05 | 3467.8ms/step | 5.8 samples/s
540
+ Step 39990/44000 | Loss: 2.9127 | LR: 1.05e-05 | 3463.9ms/step | 5.8 samples/s
541
+ Step 40000/44000 | Loss: 2.8806 | LR: 1.05e-05 | 3459.9ms/step | 5.8 samples/s
542
+ Validation at step 40000: Loss = 2.9043, Perplexity = 18.25
543
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-40000
544
+ Step 40010/44000 | Loss: 2.8829 | LR: 1.04e-05 | 9558.2ms/step | 2.1 samples/s
545
+ Step 40020/44000 | Loss: 2.8983 | LR: 1.04e-05 | 3459.8ms/step | 5.8 samples/s
546
+ Step 40030/44000 | Loss: 2.8875 | LR: 1.03e-05 | 3463.4ms/step | 5.8 samples/s
547
+ Step 40040/44000 | Loss: 2.8790 | LR: 1.03e-05 | 3463.4ms/step | 5.8 samples/s
548
+ Step 40050/44000 | Loss: 2.9027 | LR: 1.02e-05 | 3463.8ms/step | 5.8 samples/s
549
+ Step 40060/44000 | Loss: 2.8700 | LR: 1.02e-05 | 3460.2ms/step | 5.8 samples/s
550
+ Step 40070/44000 | Loss: 2.8890 | LR: 1.01e-05 | 3461.1ms/step | 5.8 samples/s
551
+ Step 40080/44000 | Loss: 2.8787 | LR: 1.01e-05 | 3460.0ms/step | 5.8 samples/s
552
+ Step 40090/44000 | Loss: 2.8749 | LR: 1.00e-05 | 3464.3ms/step | 5.8 samples/s
553
+ Step 40100/44000 | Loss: 2.9005 | LR: 9.96e-06 | 3458.6ms/step | 5.8 samples/s
554
+ Step 40110/44000 | Loss: 2.8999 | LR: 9.91e-06 | 3458.6ms/step | 5.8 samples/s
555
+ Step 40120/44000 | Loss: 2.8833 | LR: 9.86e-06 | 3461.6ms/step | 5.8 samples/s
556
+ Step 40130/44000 | Loss: 2.9014 | LR: 9.81e-06 | 3459.0ms/step | 5.8 samples/s
557
+ Step 40140/44000 | Loss: 2.8975 | LR: 9.76e-06 | 3462.5ms/step | 5.8 samples/s
558
+ Step 40150/44000 | Loss: 2.9011 | LR: 9.71e-06 | 3461.8ms/step | 5.8 samples/s
559
+ Step 40160/44000 | Loss: 2.9014 | LR: 9.66e-06 | 3460.1ms/step | 5.8 samples/s
560
+ Step 40170/44000 | Loss: 2.8881 | LR: 9.61e-06 | 3459.0ms/step | 5.8 samples/s
561
+ Step 40180/44000 | Loss: 2.8974 | LR: 9.56e-06 | 3458.3ms/step | 5.8 samples/s
562
+ Step 40190/44000 | Loss: 2.8767 | LR: 9.51e-06 | 3461.1ms/step | 5.8 samples/s
563
+ Step 40200/44000 | Loss: 2.9029 | LR: 9.46e-06 | 3459.9ms/step | 5.8 samples/s
564
+ Step 40210/44000 | Loss: 2.9252 | LR: 9.41e-06 | 3464.1ms/step | 5.8 samples/s
565
+ Step 40220/44000 | Loss: 2.9436 | LR: 9.36e-06 | 3462.4ms/step | 5.8 samples/s
566
+ Step 40230/44000 | Loss: 2.8951 | LR: 9.32e-06 | 3458.9ms/step | 5.8 samples/s
567
+ Step 40240/44000 | Loss: 2.8754 | LR: 9.27e-06 | 3459.3ms/step | 5.8 samples/s
568
+ Step 40250/44000 | Loss: 2.9346 | LR: 9.22e-06 | 3457.8ms/step | 5.8 samples/s
569
+ Validation at step 40250: Loss = 2.9043, Perplexity = 18.25
570
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-40250
571
+ Step 40260/44000 | Loss: 2.9198 | LR: 9.17e-06 | 9556.9ms/step | 2.1 samples/s
572
+ Step 40270/44000 | Loss: 2.9304 | LR: 9.12e-06 | 3459.4ms/step | 5.8 samples/s
573
+ Step 40280/44000 | Loss: 2.9087 | LR: 9.07e-06 | 3461.1ms/step | 5.8 samples/s
574
+ Step 40290/44000 | Loss: 2.9123 | LR: 9.02e-06 | 3460.1ms/step | 5.8 samples/s
575
+ Step 40300/44000 | Loss: 2.8951 | LR: 8.97e-06 | 3460.0ms/step | 5.8 samples/s
576
+ Step 40310/44000 | Loss: 2.8867 | LR: 8.93e-06 | 3462.5ms/step | 5.8 samples/s
577
+ Step 40320/44000 | Loss: 2.8846 | LR: 8.88e-06 | 3461.7ms/step | 5.8 samples/s
578
+ Step 40330/44000 | Loss: 2.9082 | LR: 8.83e-06 | 3459.9ms/step | 5.8 samples/s
579
+ Step 40340/44000 | Loss: 2.9165 | LR: 8.78e-06 | 3461.8ms/step | 5.8 samples/s
580
+ Step 40350/44000 | Loss: 2.9232 | LR: 8.74e-06 | 3464.6ms/step | 5.8 samples/s
581
+ Step 40360/44000 | Loss: 2.8688 | LR: 8.69e-06 | 3464.0ms/step | 5.8 samples/s
582
+ Step 40370/44000 | Loss: 2.8690 | LR: 8.64e-06 | 3460.2ms/step | 5.8 samples/s
583
+ Step 40380/44000 | Loss: 2.9106 | LR: 8.59e-06 | 3460.1ms/step | 5.8 samples/s
584
+ Step 40390/44000 | Loss: 2.8810 | LR: 8.55e-06 | 3461.8ms/step | 5.8 samples/s
585
+ Step 40400/44000 | Loss: 2.9099 | LR: 8.50e-06 | 3462.8ms/step | 5.8 samples/s
586
+ Step 40410/44000 | Loss: 2.8917 | LR: 8.45e-06 | 3464.0ms/step | 5.8 samples/s
587
+ Step 40420/44000 | Loss: 2.9082 | LR: 8.41e-06 | 3460.2ms/step | 5.8 samples/s
588
+ Step 40430/44000 | Loss: 2.9107 | LR: 8.36e-06 | 3466.8ms/step | 5.8 samples/s
589
+ Step 40440/44000 | Loss: 2.8905 | LR: 8.31e-06 | 3461.7ms/step | 5.8 samples/s
590
+ Step 40450/44000 | Loss: 2.8976 | LR: 8.27e-06 | 3460.7ms/step | 5.8 samples/s
591
+ Step 40460/44000 | Loss: 2.8776 | LR: 8.22e-06 | 3459.4ms/step | 5.8 samples/s
592
+ Step 40470/44000 | Loss: 2.8900 | LR: 8.17e-06 | 3460.0ms/step | 5.8 samples/s
593
+ Step 40480/44000 | Loss: 2.8971 | LR: 8.13e-06 | 3461.1ms/step | 5.8 samples/s
594
+ Step 40490/44000 | Loss: 2.8969 | LR: 8.08e-06 | 3459.8ms/step | 5.8 samples/s
595
+ Step 40500/44000 | Loss: 2.9147 | LR: 8.04e-06 | 3459.5ms/step | 5.8 samples/s
596
+ Validation at step 40500: Loss = 2.9043, Perplexity = 18.25
597
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-40500
598
+ Step 40510/44000 | Loss: 2.9112 | LR: 7.99e-06 | 9548.5ms/step | 2.1 samples/s
599
+ Step 40520/44000 | Loss: 2.8825 | LR: 7.94e-06 | 3459.1ms/step | 5.8 samples/s
600
+ Step 40530/44000 | Loss: 2.8828 | LR: 7.90e-06 | 3460.2ms/step | 5.8 samples/s
601
+ Step 40540/44000 | Loss: 2.9160 | LR: 7.85e-06 | 3467.1ms/step | 5.8 samples/s
602
+ Step 40550/44000 | Loss: 2.8714 | LR: 7.81e-06 | 3462.1ms/step | 5.8 samples/s
603
+ Step 40560/44000 | Loss: 2.8952 | LR: 7.76e-06 | 3463.6ms/step | 5.8 samples/s
604
+ Step 40570/44000 | Loss: 2.8603 | LR: 7.72e-06 | 3463.7ms/step | 5.8 samples/s
605
+ Step 40580/44000 | Loss: 2.8937 | LR: 7.67e-06 | 3462.8ms/step | 5.8 samples/s
606
+ Step 40590/44000 | Loss: 2.8977 | LR: 7.63e-06 | 3461.0ms/step | 5.8 samples/s
607
+ Step 40600/44000 | Loss: 2.9179 | LR: 7.59e-06 | 3461.3ms/step | 5.8 samples/s
608
+ Step 40610/44000 | Loss: 2.9096 | LR: 7.54e-06 | 3470.1ms/step | 5.8 samples/s
609
+ Step 40620/44000 | Loss: 2.9000 | LR: 7.50e-06 | 3481.7ms/step | 5.7 samples/s
610
+ Step 40630/44000 | Loss: 2.9099 | LR: 7.45e-06 | 3486.6ms/step | 5.7 samples/s
611
+ Step 40640/44000 | Loss: 2.9079 | LR: 7.41e-06 | 3489.1ms/step | 5.7 samples/s
612
+ Step 40650/44000 | Loss: 2.9029 | LR: 7.37e-06 | 3489.0ms/step | 5.7 samples/s
613
+ Step 40660/44000 | Loss: 2.9041 | LR: 7.32e-06 | 3491.0ms/step | 5.7 samples/s
614
+ Step 40670/44000 | Loss: 2.8734 | LR: 7.28e-06 | 3489.8ms/step | 5.7 samples/s
615
+ Step 40680/44000 | Loss: 2.8954 | LR: 7.23e-06 | 3495.0ms/step | 5.7 samples/s
616
+ Step 40690/44000 | Loss: 2.8957 | LR: 7.19e-06 | 3492.7ms/step | 5.7 samples/s
617
+ Step 40700/44000 | Loss: 2.8780 | LR: 7.15e-06 | 3493.4ms/step | 5.7 samples/s
618
+ Step 40710/44000 | Loss: 2.8518 | LR: 7.10e-06 | 3497.1ms/step | 5.7 samples/s
619
+ Step 40720/44000 | Loss: 2.8806 | LR: 7.06e-06 | 3494.2ms/step | 5.7 samples/s
620
+ Step 40730/44000 | Loss: 2.8799 | LR: 7.02e-06 | 3494.3ms/step | 5.7 samples/s
621
+ Step 40740/44000 | Loss: 2.9201 | LR: 6.98e-06 | 3494.2ms/step | 5.7 samples/s
622
+ Step 40750/44000 | Loss: 2.9025 | LR: 6.93e-06 | 3491.5ms/step | 5.7 samples/s
623
+ Validation at step 40750: Loss = 2.9042, Perplexity = 18.25
624
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-40750
625
+ Step 40760/44000 | Loss: 2.8976 | LR: 6.89e-06 | 9645.5ms/step | 2.1 samples/s
626
+ Step 40770/44000 | Loss: 2.9524 | LR: 6.85e-06 | 3494.1ms/step | 5.7 samples/s
627
+ Step 40780/44000 | Loss: 2.9005 | LR: 6.81e-06 | 3494.1ms/step | 5.7 samples/s
628
+ Step 40790/44000 | Loss: 2.9051 | LR: 6.77e-06 | 3494.9ms/step | 5.7 samples/s
629
+ Step 40800/44000 | Loss: 2.8763 | LR: 6.72e-06 | 3493.8ms/step | 5.7 samples/s
630
+ Step 40810/44000 | Loss: 2.8807 | LR: 6.68e-06 | 3496.5ms/step | 5.7 samples/s
631
+ Step 40820/44000 | Loss: 2.8957 | LR: 6.64e-06 | 3486.3ms/step | 5.7 samples/s
632
+ Step 40830/44000 | Loss: 2.9192 | LR: 6.60e-06 | 3489.3ms/step | 5.7 samples/s
633
+ Step 40840/44000 | Loss: 2.9008 | LR: 6.56e-06 | 3491.1ms/step | 5.7 samples/s
634
+ Step 40850/44000 | Loss: 2.8914 | LR: 6.52e-06 | 3494.5ms/step | 5.7 samples/s
635
+ Step 40860/44000 | Loss: 2.8594 | LR: 6.47e-06 | 3492.0ms/step | 5.7 samples/s
636
+ Step 40870/44000 | Loss: 2.8892 | LR: 6.43e-06 | 3492.6ms/step | 5.7 samples/s
637
+ Step 40880/44000 | Loss: 2.8728 | LR: 6.39e-06 | 3491.7ms/step | 5.7 samples/s
638
+ Step 40890/44000 | Loss: 2.8769 | LR: 6.35e-06 | 3489.6ms/step | 5.7 samples/s
639
+ Step 40900/44000 | Loss: 2.8961 | LR: 6.31e-06 | 3491.7ms/step | 5.7 samples/s
640
+ Step 40910/44000 | Loss: 2.8895 | LR: 6.27e-06 | 3490.6ms/step | 5.7 samples/s
641
+ Step 40920/44000 | Loss: 2.9088 | LR: 6.23e-06 | 3492.8ms/step | 5.7 samples/s
642
+ Step 40930/44000 | Loss: 2.9276 | LR: 6.19e-06 | 3493.6ms/step | 5.7 samples/s
643
+ Step 40940/44000 | Loss: 2.8607 | LR: 6.15e-06 | 3493.3ms/step | 5.7 samples/s
644
+ Step 40950/44000 | Loss: 2.9016 | LR: 6.11e-06 | 3494.2ms/step | 5.7 samples/s
645
+ Step 40960/44000 | Loss: 2.9077 | LR: 6.07e-06 | 3494.7ms/step | 5.7 samples/s
646
+ Step 40970/44000 | Loss: 2.9146 | LR: 6.03e-06 | 3494.4ms/step | 5.7 samples/s
647
+ Step 40980/44000 | Loss: 2.8884 | LR: 5.99e-06 | 3492.5ms/step | 5.7 samples/s
648
+ Step 40990/44000 | Loss: 2.9040 | LR: 5.95e-06 | 3492.6ms/step | 5.7 samples/s
649
+ Step 41000/44000 | Loss: 2.8775 | LR: 5.91e-06 | 3491.6ms/step | 5.7 samples/s
650
+ Validation at step 41000: Loss = 2.9042, Perplexity = 18.25
651
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-41000
652
+ Step 41010/44000 | Loss: 2.8997 | LR: 5.87e-06 | 9632.2ms/step | 2.1 samples/s
653
+ Step 41020/44000 | Loss: 2.9053 | LR: 5.83e-06 | 3492.8ms/step | 5.7 samples/s
654
+ Step 41030/44000 | Loss: 2.8726 | LR: 5.80e-06 | 3488.7ms/step | 5.7 samples/s
655
+ Step 41040/44000 | Loss: 2.9508 | LR: 5.76e-06 | 3489.2ms/step | 5.7 samples/s
656
+ Step 41050/44000 | Loss: 2.8913 | LR: 5.72e-06 | 3492.5ms/step | 5.7 samples/s
657
+ Step 41060/44000 | Loss: 2.8832 | LR: 5.68e-06 | 3493.7ms/step | 5.7 samples/s
658
+ Step 41070/44000 | Loss: 2.9098 | LR: 5.64e-06 | 3493.7ms/step | 5.7 samples/s
659
+ Step 41080/44000 | Loss: 2.8812 | LR: 5.60e-06 | 3498.4ms/step | 5.7 samples/s
660
+ Step 41090/44000 | Loss: 2.8839 | LR: 5.56e-06 | 3495.5ms/step | 5.7 samples/s
661
+ Step 41100/44000 | Loss: 2.8779 | LR: 5.53e-06 | 3498.6ms/step | 5.7 samples/s
662
+ Step 41110/44000 | Loss: 2.9082 | LR: 5.49e-06 | 3498.8ms/step | 5.7 samples/s
663
+ Step 41120/44000 | Loss: 2.8874 | LR: 5.45e-06 | 3501.6ms/step | 5.7 samples/s
664
+ Step 41130/44000 | Loss: 2.9119 | LR: 5.41e-06 | 3499.1ms/step | 5.7 samples/s
665
+ Step 41140/44000 | Loss: 2.9367 | LR: 5.38e-06 | 3498.2ms/step | 5.7 samples/s
666
+ Step 41150/44000 | Loss: 2.8692 | LR: 5.34e-06 | 3498.5ms/step | 5.7 samples/s
667
+ Step 41160/44000 | Loss: 2.8882 | LR: 5.30e-06 | 3497.4ms/step | 5.7 samples/s
668
+ Step 41170/44000 | Loss: 2.8812 | LR: 5.26e-06 | 3498.2ms/step | 5.7 samples/s
669
+ Step 41180/44000 | Loss: 2.9119 | LR: 5.23e-06 | 3495.8ms/step | 5.7 samples/s
670
+ Step 41190/44000 | Loss: 2.8905 | LR: 5.19e-06 | 3495.7ms/step | 5.7 samples/s
671
+ Step 41200/44000 | Loss: 2.8719 | LR: 5.15e-06 | 3500.4ms/step | 5.7 samples/s
672
+ Step 41210/44000 | Loss: 2.8827 | LR: 5.12e-06 | 3500.3ms/step | 5.7 samples/s
673
+ Step 41220/44000 | Loss: 2.9123 | LR: 5.08e-06 | 3494.3ms/step | 5.7 samples/s
674
+ Step 41230/44000 | Loss: 2.8918 | LR: 5.04e-06 | 3497.9ms/step | 5.7 samples/s
675
+ Step 41240/44000 | Loss: 2.9108 | LR: 5.01e-06 | 3497.7ms/step | 5.7 samples/s
676
+ Step 41250/44000 | Loss: 2.8690 | LR: 4.97e-06 | 3497.3ms/step | 5.7 samples/s
677
+ Validation at step 41250: Loss = 2.9042, Perplexity = 18.25
678
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-41250
679
+ Step 41260/44000 | Loss: 2.9008 | LR: 4.94e-06 | 9643.0ms/step | 2.1 samples/s
680
+ Step 41270/44000 | Loss: 2.8694 | LR: 4.90e-06 | 3487.8ms/step | 5.7 samples/s
681
+ Step 41280/44000 | Loss: 2.9040 | LR: 4.86e-06 | 3492.3ms/step | 5.7 samples/s
682
+ Step 41290/44000 | Loss: 2.8688 | LR: 4.83e-06 | 3496.6ms/step | 5.7 samples/s
683
+ Step 41300/44000 | Loss: 2.9248 | LR: 4.79e-06 | 3494.3ms/step | 5.7 samples/s
684
+ Step 41310/44000 | Loss: 2.9023 | LR: 4.76e-06 | 3494.0ms/step | 5.7 samples/s
685
+ Step 41320/44000 | Loss: 2.9125 | LR: 4.72e-06 | 3497.1ms/step | 5.7 samples/s
686
+ Step 41330/44000 | Loss: 2.8988 | LR: 4.69e-06 | 3494.1ms/step | 5.7 samples/s
687
+ Step 41340/44000 | Loss: 2.8731 | LR: 4.65e-06 | 3494.0ms/step | 5.7 samples/s
688
+ Step 41350/44000 | Loss: 2.8911 | LR: 4.62e-06 | 3493.5ms/step | 5.7 samples/s
689
+ Step 41360/44000 | Loss: 2.8873 | LR: 4.58e-06 | 3492.7ms/step | 5.7 samples/s
690
+ Step 41370/44000 | Loss: 2.8820 | LR: 4.55e-06 | 3498.1ms/step | 5.7 samples/s
691
+ Step 41380/44000 | Loss: 2.9299 | LR: 4.51e-06 | 3493.0ms/step | 5.7 samples/s
692
+ Step 41390/44000 | Loss: 2.8999 | LR: 4.48e-06 | 3493.3ms/step | 5.7 samples/s
693
+ Step 41400/44000 | Loss: 2.9051 | LR: 4.45e-06 | 3496.1ms/step | 5.7 samples/s
694
+ Step 41410/44000 | Loss: 2.9085 | LR: 4.41e-06 | 3491.7ms/step | 5.7 samples/s
695
+ Step 41420/44000 | Loss: 2.9146 | LR: 4.38e-06 | 3490.8ms/step | 5.7 samples/s
696
+ Step 41430/44000 | Loss: 2.8802 | LR: 4.34e-06 | 3493.2ms/step | 5.7 samples/s
697
+ Step 41440/44000 | Loss: 2.8863 | LR: 4.31e-06 | 3490.8ms/step | 5.7 samples/s
698
+ Step 41450/44000 | Loss: 2.9074 | LR: 4.28e-06 | 3495.2ms/step | 5.7 samples/s
699
+ Step 41460/44000 | Loss: 2.8861 | LR: 4.24e-06 | 3500.0ms/step | 5.7 samples/s
700
+ Step 41470/44000 | Loss: 2.8986 | LR: 4.21e-06 | 3499.9ms/step | 5.7 samples/s
701
+ Step 41480/44000 | Loss: 2.9005 | LR: 4.18e-06 | 3490.2ms/step | 5.7 samples/s
702
+ Step 41490/44000 | Loss: 2.8976 | LR: 4.14e-06 | 3492.8ms/step | 5.7 samples/s
703
+ Step 41500/44000 | Loss: 2.9102 | LR: 4.11e-06 | 3494.1ms/step | 5.7 samples/s
704
+ Validation at step 41500: Loss = 2.9042, Perplexity = 18.25
705
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-41500
706
+ Step 41510/44000 | Loss: 2.9068 | LR: 4.08e-06 | 9644.8ms/step | 2.1 samples/s
707
+ Step 41520/44000 | Loss: 2.8684 | LR: 4.05e-06 | 3498.9ms/step | 5.7 samples/s
708
+ Step 41530/44000 | Loss: 2.8892 | LR: 4.01e-06 | 3496.8ms/step | 5.7 samples/s
709
+ Step 41540/44000 | Loss: 2.8992 | LR: 3.98e-06 | 3498.1ms/step | 5.7 samples/s
710
+ Step 41550/44000 | Loss: 2.8577 | LR: 3.95e-06 | 3499.7ms/step | 5.7 samples/s
711
+ Step 41560/44000 | Loss: 2.9005 | LR: 3.92e-06 | 3497.5ms/step | 5.7 samples/s
712
+ Step 41570/44000 | Loss: 2.8992 | LR: 3.88e-06 | 3500.2ms/step | 5.7 samples/s
713
+ Step 41580/44000 | Loss: 2.9019 | LR: 3.85e-06 | 3500.5ms/step | 5.7 samples/s
714
+ Step 41590/44000 | Loss: 2.8971 | LR: 3.82e-06 | 3503.0ms/step | 5.7 samples/s
715
+ Step 41600/44000 | Loss: 2.8894 | LR: 3.79e-06 | 3499.9ms/step | 5.7 samples/s
716
+ Step 41610/44000 | Loss: 2.9017 | LR: 3.76e-06 | 3500.6ms/step | 5.7 samples/s
717
+ Step 41620/44000 | Loss: 2.8884 | LR: 3.73e-06 | 3498.2ms/step | 5.7 samples/s
718
+ Step 41630/44000 | Loss: 2.8801 | LR: 3.70e-06 | 3500.9ms/step | 5.7 samples/s
719
+ Step 41640/44000 | Loss: 2.8666 | LR: 3.66e-06 | 3500.4ms/step | 5.7 samples/s
720
+ Step 41650/44000 | Loss: 2.8742 | LR: 3.63e-06 | 3499.6ms/step | 5.7 samples/s
721
+ Step 41660/44000 | Loss: 2.8981 | LR: 3.60e-06 | 3499.2ms/step | 5.7 samples/s
722
+ Step 41670/44000 | Loss: 2.8876 | LR: 3.57e-06 | 3502.8ms/step | 5.7 samples/s
723
+ Step 41680/44000 | Loss: 2.9133 | LR: 3.54e-06 | 3497.9ms/step | 5.7 samples/s
724
+ Step 41690/44000 | Loss: 2.8777 | LR: 3.51e-06 | 3493.2ms/step | 5.7 samples/s
725
+ Step 41700/44000 | Loss: 2.8750 | LR: 3.48e-06 | 3492.8ms/step | 5.7 samples/s
726
+ Step 41710/44000 | Loss: 2.8926 | LR: 3.45e-06 | 3497.5ms/step | 5.7 samples/s
727
+ Step 41720/44000 | Loss: 2.8906 | LR: 3.42e-06 | 3497.4ms/step | 5.7 samples/s
728
+ Step 41730/44000 | Loss: 2.8909 | LR: 3.39e-06 | 3499.0ms/step | 5.7 samples/s
729
+ Step 41740/44000 | Loss: 2.9131 | LR: 3.36e-06 | 3500.0ms/step | 5.7 samples/s
730
+ Step 41750/44000 | Loss: 2.8699 | LR: 3.33e-06 | 3498.0ms/step | 5.7 samples/s
731
+ Validation at step 41750: Loss = 2.9042, Perplexity = 18.25
732
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-41750
733
+ Step 41760/44000 | Loss: 2.9243 | LR: 3.30e-06 | 9646.4ms/step | 2.1 samples/s
734
+ Step 41770/44000 | Loss: 2.8600 | LR: 3.27e-06 | 3496.5ms/step | 5.7 samples/s
735
+ Step 41780/44000 | Loss: 2.8842 | LR: 3.24e-06 | 3498.4ms/step | 5.7 samples/s
736
+ Step 41790/44000 | Loss: 2.8863 | LR: 3.21e-06 | 3495.0ms/step | 5.7 samples/s
737
+ Step 41800/44000 | Loss: 2.9077 | LR: 3.19e-06 | 3492.4ms/step | 5.7 samples/s
738
+ Step 41810/44000 | Loss: 2.8972 | LR: 3.16e-06 | 3499.4ms/step | 5.7 samples/s
739
+ Step 41820/44000 | Loss: 2.9099 | LR: 3.13e-06 | 3499.8ms/step | 5.7 samples/s
740
+ Step 41830/44000 | Loss: 2.8891 | LR: 3.10e-06 | 3498.7ms/step | 5.7 samples/s
741
+ Step 41840/44000 | Loss: 2.9082 | LR: 3.07e-06 | 3501.4ms/step | 5.7 samples/s
742
+ Step 41850/44000 | Loss: 2.9023 | LR: 3.04e-06 | 3500.7ms/step | 5.7 samples/s
743
+ Step 41860/44000 | Loss: 2.9053 | LR: 3.01e-06 | 3497.4ms/step | 5.7 samples/s
744
+ Step 41870/44000 | Loss: 2.8775 | LR: 2.99e-06 | 3498.9ms/step | 5.7 samples/s
745
+ Step 41880/44000 | Loss: 2.8778 | LR: 2.96e-06 | 3500.3ms/step | 5.7 samples/s
746
+ Step 41890/44000 | Loss: 2.9102 | LR: 2.93e-06 | 3496.6ms/step | 5.7 samples/s
747
+ Step 41900/44000 | Loss: 2.9038 | LR: 2.90e-06 | 3497.8ms/step | 5.7 samples/s
748
+ Step 41910/44000 | Loss: 2.8773 | LR: 2.88e-06 | 3493.1ms/step | 5.7 samples/s
749
+ Step 41920/44000 | Loss: 2.8698 | LR: 2.85e-06 | 3494.5ms/step | 5.7 samples/s
750
+ Step 41930/44000 | Loss: 2.9265 | LR: 2.82e-06 | 3496.5ms/step | 5.7 samples/s
751
+ Step 41940/44000 | Loss: 2.8835 | LR: 2.79e-06 | 3500.0ms/step | 5.7 samples/s
752
+ Step 41950/44000 | Loss: 2.8739 | LR: 2.77e-06 | 3499.7ms/step | 5.7 samples/s
753
+ Step 41960/44000 | Loss: 2.8742 | LR: 2.74e-06 | 3501.8ms/step | 5.7 samples/s
754
+ Step 41970/44000 | Loss: 2.8944 | LR: 2.71e-06 | 3499.9ms/step | 5.7 samples/s
755
+ Step 41980/44000 | Loss: 2.8904 | LR: 2.69e-06 | 3500.4ms/step | 5.7 samples/s
756
+ Step 41990/44000 | Loss: 2.9277 | LR: 2.66e-06 | 3500.2ms/step | 5.7 samples/s
757
+ Step 42000/44000 | Loss: 2.9018 | LR: 2.63e-06 | 3500.7ms/step | 5.7 samples/s
758
+ Validation at step 42000: Loss = 2.9042, Perplexity = 18.25
759
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-42000
760
+ Step 42010/44000 | Loss: 2.9021 | LR: 2.61e-06 | 9665.2ms/step | 2.1 samples/s
761
+ Step 42020/44000 | Loss: 2.8857 | LR: 2.58e-06 | 3500.9ms/step | 5.7 samples/s
762
+ Step 42030/44000 | Loss: 2.9047 | LR: 2.56e-06 | 3502.6ms/step | 5.7 samples/s
763
+ Step 42040/44000 | Loss: 2.9076 | LR: 2.53e-06 | 3496.9ms/step | 5.7 samples/s
764
+ Step 42050/44000 | Loss: 2.9105 | LR: 2.50e-06 | 3500.0ms/step | 5.7 samples/s
765
+ Step 42060/44000 | Loss: 2.8867 | LR: 2.48e-06 | 3499.8ms/step | 5.7 samples/s
766
+ Step 42070/44000 | Loss: 2.8823 | LR: 2.45e-06 | 3501.1ms/step | 5.7 samples/s
767
+ Step 42080/44000 | Loss: 2.9013 | LR: 2.43e-06 | 3500.8ms/step | 5.7 samples/s
768
+ Step 42090/44000 | Loss: 2.9178 | LR: 2.40e-06 | 3501.3ms/step | 5.7 samples/s
769
+ Step 42100/44000 | Loss: 2.8578 | LR: 2.38e-06 | 3502.2ms/step | 5.7 samples/s
770
+ Step 42110/44000 | Loss: 2.8984 | LR: 2.35e-06 | 3500.9ms/step | 5.7 samples/s
771
+ Step 42120/44000 | Loss: 2.8638 | LR: 2.33e-06 | 3496.6ms/step | 5.7 samples/s
772
+ Step 42130/44000 | Loss: 2.9247 | LR: 2.30e-06 | 3494.7ms/step | 5.7 samples/s
773
+ Step 42140/44000 | Loss: 2.8605 | LR: 2.28e-06 | 3494.0ms/step | 5.7 samples/s
774
+ Step 42150/44000 | Loss: 2.8739 | LR: 2.25e-06 | 3495.3ms/step | 5.7 samples/s
775
+ Step 42160/44000 | Loss: 2.8981 | LR: 2.23e-06 | 3497.9ms/step | 5.7 samples/s
776
+ Step 42170/44000 | Loss: 2.8715 | LR: 2.21e-06 | 3502.7ms/step | 5.7 samples/s
777
+ Step 42180/44000 | Loss: 2.8806 | LR: 2.18e-06 | 3504.5ms/step | 5.7 samples/s
778
+ Step 42190/44000 | Loss: 2.9129 | LR: 2.16e-06 | 3502.2ms/step | 5.7 samples/s
779
+ Step 42200/44000 | Loss: 2.8802 | LR: 2.13e-06 | 3501.3ms/step | 5.7 samples/s
780
+ Step 42210/44000 | Loss: 2.9142 | LR: 2.11e-06 | 3501.5ms/step | 5.7 samples/s
781
+ Step 42220/44000 | Loss: 2.8900 | LR: 2.09e-06 | 3499.7ms/step | 5.7 samples/s
782
+ Step 42230/44000 | Loss: 2.8692 | LR: 2.06e-06 | 3499.3ms/step | 5.7 samples/s
783
+ Step 42240/44000 | Loss: 2.9089 | LR: 2.04e-06 | 3497.5ms/step | 5.7 samples/s
784
+ Step 42250/44000 | Loss: 2.8868 | LR: 2.02e-06 | 3497.7ms/step | 5.7 samples/s
785
+ Validation at step 42250: Loss = 2.9042, Perplexity = 18.25
786
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-42250
787
+ Step 42260/44000 | Loss: 2.8987 | LR: 1.99e-06 | 9650.8ms/step | 2.1 samples/s
788
+ Step 42270/44000 | Loss: 2.9163 | LR: 1.97e-06 | 3485.7ms/step | 5.7 samples/s
789
+ Step 42280/44000 | Loss: 2.9118 | LR: 1.95e-06 | 3487.2ms/step | 5.7 samples/s
790
+ Step 42290/44000 | Loss: 2.8633 | LR: 1.93e-06 | 3485.0ms/step | 5.7 samples/s
791
+ Step 42300/44000 | Loss: 2.9013 | LR: 1.90e-06 | 3485.7ms/step | 5.7 samples/s
792
+ Step 42310/44000 | Loss: 2.8915 | LR: 1.88e-06 | 3488.5ms/step | 5.7 samples/s
793
+ Step 42320/44000 | Loss: 2.9121 | LR: 1.86e-06 | 3490.0ms/step | 5.7 samples/s
794
+ Step 42330/44000 | Loss: 2.9197 | LR: 1.84e-06 | 3490.2ms/step | 5.7 samples/s
795
+ Step 42340/44000 | Loss: 2.8923 | LR: 1.82e-06 | 3487.5ms/step | 5.7 samples/s
796
+ Step 42350/44000 | Loss: 2.8925 | LR: 1.79e-06 | 3491.2ms/step | 5.7 samples/s
797
+ Step 42360/44000 | Loss: 2.8898 | LR: 1.77e-06 | 3491.3ms/step | 5.7 samples/s
798
+ Step 42370/44000 | Loss: 2.9016 | LR: 1.75e-06 | 3491.2ms/step | 5.7 samples/s
799
+ Step 42380/44000 | Loss: 2.9166 | LR: 1.73e-06 | 3493.1ms/step | 5.7 samples/s
800
+ Step 42390/44000 | Loss: 2.8869 | LR: 1.71e-06 | 3491.4ms/step | 5.7 samples/s
801
+ Step 42400/44000 | Loss: 2.8927 | LR: 1.69e-06 | 3491.5ms/step | 5.7 samples/s
802
+ Step 42410/44000 | Loss: 2.8729 | LR: 1.67e-06 | 3495.0ms/step | 5.7 samples/s
803
+ Step 42420/44000 | Loss: 2.8998 | LR: 1.64e-06 | 3497.2ms/step | 5.7 samples/s
804
+ Step 42430/44000 | Loss: 2.9139 | LR: 1.62e-06 | 3494.2ms/step | 5.7 samples/s
805
+ Step 42440/44000 | Loss: 2.8723 | LR: 1.60e-06 | 3494.4ms/step | 5.7 samples/s
806
+ Step 42450/44000 | Loss: 2.9197 | LR: 1.58e-06 | 3495.1ms/step | 5.7 samples/s
807
+ Step 42460/44000 | Loss: 2.9014 | LR: 1.56e-06 | 3494.5ms/step | 5.7 samples/s
808
+ Step 42470/44000 | Loss: 2.8862 | LR: 1.54e-06 | 3489.7ms/step | 5.7 samples/s
809
+ Step 42480/44000 | Loss: 2.9104 | LR: 1.52e-06 | 3492.6ms/step | 5.7 samples/s
810
+ Step 42490/44000 | Loss: 2.8779 | LR: 1.50e-06 | 3491.7ms/step | 5.7 samples/s
811
+ Step 42500/44000 | Loss: 2.9020 | LR: 1.48e-06 | 3493.5ms/step | 5.7 samples/s
812
+ Validation at step 42500: Loss = 2.9042, Perplexity = 18.25
813
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-42500
814
+ Step 42510/44000 | Loss: 2.8850 | LR: 1.46e-06 | 9667.1ms/step | 2.1 samples/s
815
+ Step 42520/44000 | Loss: 2.8953 | LR: 1.44e-06 | 3502.2ms/step | 5.7 samples/s
816
+ Step 42530/44000 | Loss: 2.8643 | LR: 1.42e-06 | 3505.3ms/step | 5.7 samples/s
817
+ Step 42540/44000 | Loss: 2.9395 | LR: 1.40e-06 | 3504.6ms/step | 5.7 samples/s
818
+ Step 42550/44000 | Loss: 2.8911 | LR: 1.39e-06 | 3498.8ms/step | 5.7 samples/s
819
+ Step 42560/44000 | Loss: 2.9085 | LR: 1.37e-06 | 3497.4ms/step | 5.7 samples/s
820
+ Step 42570/44000 | Loss: 2.9026 | LR: 1.35e-06 | 3499.2ms/step | 5.7 samples/s
821
+ Step 42580/44000 | Loss: 2.8939 | LR: 1.33e-06 | 3500.3ms/step | 5.7 samples/s
822
+ Step 42590/44000 | Loss: 2.9016 | LR: 1.31e-06 | 3499.9ms/step | 5.7 samples/s
823
+ Step 42600/44000 | Loss: 2.8829 | LR: 1.29e-06 | 3503.1ms/step | 5.7 samples/s
824
+ Step 42610/44000 | Loss: 2.8869 | LR: 1.27e-06 | 3500.8ms/step | 5.7 samples/s
825
+ Step 42620/44000 | Loss: 2.9245 | LR: 1.25e-06 | 3500.2ms/step | 5.7 samples/s
826
+ Step 42630/44000 | Loss: 2.8680 | LR: 1.24e-06 | 3504.6ms/step | 5.7 samples/s
827
+ Step 42640/44000 | Loss: 2.9087 | LR: 1.22e-06 | 3504.0ms/step | 5.7 samples/s
828
+ Step 42650/44000 | Loss: 2.9073 | LR: 1.20e-06 | 3502.7ms/step | 5.7 samples/s
829
+ Step 42660/44000 | Loss: 2.8995 | LR: 1.18e-06 | 3500.1ms/step | 5.7 samples/s
830
+ Step 42670/44000 | Loss: 2.8865 | LR: 1.17e-06 | 3500.0ms/step | 5.7 samples/s
831
+ Step 42680/44000 | Loss: 2.8926 | LR: 1.15e-06 | 3500.4ms/step | 5.7 samples/s
832
+ Step 42690/44000 | Loss: 2.9225 | LR: 1.13e-06 | 3501.2ms/step | 5.7 samples/s
833
+ Step 42700/44000 | Loss: 2.8732 | LR: 1.11e-06 | 3500.7ms/step | 5.7 samples/s
834
+ Step 42710/44000 | Loss: 2.8923 | LR: 1.10e-06 | 3502.6ms/step | 5.7 samples/s
835
+ Step 42720/44000 | Loss: 2.8701 | LR: 1.08e-06 | 3500.7ms/step | 5.7 samples/s
836
+ Step 42730/44000 | Loss: 2.8806 | LR: 1.06e-06 | 3500.9ms/step | 5.7 samples/s
837
+ Step 42740/44000 | Loss: 2.9148 | LR: 1.05e-06 | 3502.7ms/step | 5.7 samples/s
838
+ Step 42750/44000 | Loss: 2.8605 | LR: 1.03e-06 | 3504.8ms/step | 5.7 samples/s
839
+ Validation at step 42750: Loss = 2.9042, Perplexity = 18.25
840
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-42750
841
+ Step 42760/44000 | Loss: 2.9130 | LR: 1.01e-06 | 9652.2ms/step | 2.1 samples/s
842
+ Step 42770/44000 | Loss: 2.8774 | LR: 9.97e-07 | 3490.0ms/step | 5.7 samples/s
843
+ Step 42780/44000 | Loss: 2.8813 | LR: 9.81e-07 | 3493.2ms/step | 5.7 samples/s
844
+ Step 42790/44000 | Loss: 2.8851 | LR: 9.65e-07 | 3497.1ms/step | 5.7 samples/s
845
+ Step 42800/44000 | Loss: 2.9123 | LR: 9.49e-07 | 3499.6ms/step | 5.7 samples/s
846
+ Step 42810/44000 | Loss: 2.9034 | LR: 9.33e-07 | 3498.0ms/step | 5.7 samples/s
847
+ Step 42820/44000 | Loss: 2.9052 | LR: 9.18e-07 | 3498.9ms/step | 5.7 samples/s
848
+ Step 42830/44000 | Loss: 2.9074 | LR: 9.02e-07 | 3500.6ms/step | 5.7 samples/s
849
+ Step 42840/44000 | Loss: 2.8946 | LR: 8.87e-07 | 3497.4ms/step | 5.7 samples/s
850
+ Step 42850/44000 | Loss: 2.8810 | LR: 8.72e-07 | 3498.2ms/step | 5.7 samples/s
851
+ Step 42860/44000 | Loss: 2.9138 | LR: 8.57e-07 | 3498.4ms/step | 5.7 samples/s
852
+ Step 42870/44000 | Loss: 2.9001 | LR: 8.42e-07 | 3501.3ms/step | 5.7 samples/s
853
+ Step 42880/44000 | Loss: 2.8828 | LR: 8.27e-07 | 3498.1ms/step | 5.7 samples/s
854
+ Step 42890/44000 | Loss: 2.8784 | LR: 8.12e-07 | 3501.8ms/step | 5.7 samples/s
855
+ Step 42900/44000 | Loss: 2.8876 | LR: 7.98e-07 | 3502.1ms/step | 5.7 samples/s
856
+ Step 42910/44000 | Loss: 2.8847 | LR: 7.83e-07 | 3498.7ms/step | 5.7 samples/s
857
+ Step 42920/44000 | Loss: 2.9191 | LR: 7.69e-07 | 3498.8ms/step | 5.7 samples/s
858
+ Step 42930/44000 | Loss: 2.8972 | LR: 7.55e-07 | 3496.6ms/step | 5.7 samples/s
859
+ Step 42940/44000 | Loss: 2.8596 | LR: 7.41e-07 | 3498.1ms/step | 5.7 samples/s
860
+ Step 42950/44000 | Loss: 2.9154 | LR: 7.27e-07 | 3500.9ms/step | 5.7 samples/s
861
+ Step 42960/44000 | Loss: 2.9148 | LR: 7.13e-07 | 3498.7ms/step | 5.7 samples/s
862
+ Step 42970/44000 | Loss: 2.9165 | LR: 6.99e-07 | 3496.8ms/step | 5.7 samples/s
863
+ Step 42980/44000 | Loss: 2.8988 | LR: 6.86e-07 | 3503.9ms/step | 5.7 samples/s
864
+ Step 42990/44000 | Loss: 2.8985 | LR: 6.72e-07 | 3505.8ms/step | 5.7 samples/s
865
+ Step 43000/44000 | Loss: 2.9219 | LR: 6.59e-07 | 3500.8ms/step | 5.7 samples/s
866
+ Validation at step 43000: Loss = 2.9042, Perplexity = 18.25
867
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-43000
868
+ Step 43010/44000 | Loss: 2.9040 | LR: 6.46e-07 | 9654.5ms/step | 2.1 samples/s
869
+ Step 43020/44000 | Loss: 2.8838 | LR: 6.33e-07 | 3499.1ms/step | 5.7 samples/s
870
+ Step 43030/44000 | Loss: 2.8816 | LR: 6.20e-07 | 3499.2ms/step | 5.7 samples/s
871
+ Step 43040/44000 | Loss: 2.9004 | LR: 6.08e-07 | 3500.4ms/step | 5.7 samples/s
872
+ Step 43050/44000 | Loss: 2.8726 | LR: 5.95e-07 | 3493.8ms/step | 5.7 samples/s
873
+ Step 43060/44000 | Loss: 2.9080 | LR: 5.83e-07 | 3492.9ms/step | 5.7 samples/s
874
+ Step 43070/44000 | Loss: 2.8710 | LR: 5.70e-07 | 3495.6ms/step | 5.7 samples/s
875
+ Step 43080/44000 | Loss: 2.8919 | LR: 5.58e-07 | 3496.4ms/step | 5.7 samples/s
876
+ Step 43090/44000 | Loss: 2.8926 | LR: 5.46e-07 | 3495.9ms/step | 5.7 samples/s
877
+ Step 43100/44000 | Loss: 2.8900 | LR: 5.34e-07 | 3497.9ms/step | 5.7 samples/s
878
+ Step 43110/44000 | Loss: 2.9188 | LR: 5.22e-07 | 3498.2ms/step | 5.7 samples/s
879
+ Step 43120/44000 | Loss: 2.8947 | LR: 5.11e-07 | 3498.8ms/step | 5.7 samples/s
880
+ Step 43130/44000 | Loss: 2.9183 | LR: 4.99e-07 | 3499.0ms/step | 5.7 samples/s
881
+ Step 43140/44000 | Loss: 2.9011 | LR: 4.88e-07 | 3501.4ms/step | 5.7 samples/s
882
+ Step 43150/44000 | Loss: 2.9049 | LR: 4.76e-07 | 3499.4ms/step | 5.7 samples/s
883
+ Step 43160/44000 | Loss: 2.8890 | LR: 4.65e-07 | 3498.2ms/step | 5.7 samples/s
884
+ Step 43170/44000 | Loss: 2.9062 | LR: 4.54e-07 | 3498.4ms/step | 5.7 samples/s
885
+ Step 43180/44000 | Loss: 2.8702 | LR: 4.43e-07 | 3499.9ms/step | 5.7 samples/s
886
+ Step 43190/44000 | Loss: 2.8865 | LR: 4.33e-07 | 3499.0ms/step | 5.7 samples/s
887
+ Step 43200/44000 | Loss: 2.9179 | LR: 4.22e-07 | 3499.7ms/step | 5.7 samples/s
888
+ Step 43210/44000 | Loss: 2.8981 | LR: 4.12e-07 | 3497.3ms/step | 5.7 samples/s
889
+ Step 43220/44000 | Loss: 2.9045 | LR: 4.01e-07 | 3490.8ms/step | 5.7 samples/s
890
+ Step 43230/44000 | Loss: 2.8585 | LR: 3.91e-07 | 3496.9ms/step | 5.7 samples/s
891
+ Step 43240/44000 | Loss: 2.9134 | LR: 3.81e-07 | 3501.3ms/step | 5.7 samples/s
892
+ Step 43250/44000 | Loss: 2.9193 | LR: 3.71e-07 | 3500.9ms/step | 5.7 samples/s
893
+ Validation at step 43250: Loss = 2.9042, Perplexity = 18.25
894
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-43250
895
+ Step 43260/44000 | Loss: 2.8612 | LR: 3.61e-07 | 9659.7ms/step | 2.1 samples/s
896
+ Step 43270/44000 | Loss: 2.8718 | LR: 3.51e-07 | 3494.6ms/step | 5.7 samples/s
897
+ Step 43280/44000 | Loss: 2.8762 | LR: 3.42e-07 | 3491.0ms/step | 5.7 samples/s
898
+ Step 43290/44000 | Loss: 2.9163 | LR: 3.32e-07 | 3493.5ms/step | 5.7 samples/s
899
+ Step 43300/44000 | Loss: 2.9310 | LR: 3.23e-07 | 3490.6ms/step | 5.7 samples/s
900
+ Step 43310/44000 | Loss: 2.9212 | LR: 3.14e-07 | 3491.7ms/step | 5.7 samples/s
901
+ Step 43320/44000 | Loss: 2.8807 | LR: 3.05e-07 | 3489.3ms/step | 5.7 samples/s
902
+ Step 43330/44000 | Loss: 2.8827 | LR: 2.96e-07 | 3492.9ms/step | 5.7 samples/s
903
+ Step 43340/44000 | Loss: 2.8888 | LR: 2.87e-07 | 3494.8ms/step | 5.7 samples/s
904
+ Step 43350/44000 | Loss: 2.9061 | LR: 2.79e-07 | 3492.6ms/step | 5.7 samples/s
905
+ Step 43360/44000 | Loss: 2.8759 | LR: 2.70e-07 | 3491.4ms/step | 5.7 samples/s
906
+ Step 43370/44000 | Loss: 2.8866 | LR: 2.62e-07 | 3493.8ms/step | 5.7 samples/s
907
+ Step 43380/44000 | Loss: 2.9197 | LR: 2.53e-07 | 3492.3ms/step | 5.7 samples/s
908
+ Step 43390/44000 | Loss: 2.8819 | LR: 2.45e-07 | 3490.7ms/step | 5.7 samples/s
909
+ Step 43400/44000 | Loss: 2.9123 | LR: 2.37e-07 | 3488.6ms/step | 5.7 samples/s
910
+ Step 43410/44000 | Loss: 2.8707 | LR: 2.30e-07 | 3488.5ms/step | 5.7 samples/s
911
+ Step 43420/44000 | Loss: 2.8827 | LR: 2.22e-07 | 3486.7ms/step | 5.7 samples/s
912
+ Step 43430/44000 | Loss: 2.8932 | LR: 2.14e-07 | 3483.0ms/step | 5.7 samples/s
913
+ Step 43440/44000 | Loss: 2.8645 | LR: 2.07e-07 | 3483.2ms/step | 5.7 samples/s
914
+ Step 43450/44000 | Loss: 2.8949 | LR: 1.99e-07 | 3486.5ms/step | 5.7 samples/s
915
+ Step 43460/44000 | Loss: 2.8695 | LR: 1.92e-07 | 3489.4ms/step | 5.7 samples/s
916
+ Step 43470/44000 | Loss: 2.9009 | LR: 1.85e-07 | 3495.3ms/step | 5.7 samples/s
917
+ Step 43480/44000 | Loss: 2.8979 | LR: 1.78e-07 | 3493.3ms/step | 5.7 samples/s
918
+ Step 43490/44000 | Loss: 2.8816 | LR: 1.72e-07 | 3493.6ms/step | 5.7 samples/s
919
+ Step 43500/44000 | Loss: 2.9257 | LR: 1.65e-07 | 3488.0ms/step | 5.7 samples/s
920
+ Validation at step 43500: Loss = 2.9042, Perplexity = 18.25
921
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-43500
922
+ Step 43510/44000 | Loss: 2.8987 | LR: 1.58e-07 | 9640.2ms/step | 2.1 samples/s
923
+ Step 43520/44000 | Loss: 2.9210 | LR: 1.52e-07 | 3489.6ms/step | 5.7 samples/s
924
+ Step 43530/44000 | Loss: 2.8842 | LR: 1.46e-07 | 3490.4ms/step | 5.7 samples/s
925
+ Step 43540/44000 | Loss: 2.9200 | LR: 1.40e-07 | 3489.3ms/step | 5.7 samples/s
926
+ Step 43550/44000 | Loss: 2.8805 | LR: 1.34e-07 | 3492.1ms/step | 5.7 samples/s
927
+ Step 43560/44000 | Loss: 2.8807 | LR: 1.28e-07 | 3491.6ms/step | 5.7 samples/s
928
+ Step 43570/44000 | Loss: 2.8857 | LR: 1.22e-07 | 3490.9ms/step | 5.7 samples/s
929
+ Step 43580/44000 | Loss: 2.9172 | LR: 1.16e-07 | 3489.7ms/step | 5.7 samples/s
930
+ Step 43590/44000 | Loss: 2.9139 | LR: 1.11e-07 | 3489.8ms/step | 5.7 samples/s
931
+ Step 43600/44000 | Loss: 2.8911 | LR: 1.06e-07 | 3487.8ms/step | 5.7 samples/s
932
+ Step 43610/44000 | Loss: 2.8967 | LR: 1.00e-07 | 3489.4ms/step | 5.7 samples/s
933
+ Step 43620/44000 | Loss: 2.8964 | LR: 9.52e-08 | 3492.1ms/step | 5.7 samples/s
934
+ Step 43630/44000 | Loss: 2.9025 | LR: 9.03e-08 | 3495.7ms/step | 5.7 samples/s
935
+ Step 43640/44000 | Loss: 2.8794 | LR: 8.55e-08 | 3490.0ms/step | 5.7 samples/s
936
+ Step 43650/44000 | Loss: 2.8864 | LR: 8.08e-08 | 3485.2ms/step | 5.7 samples/s
937
+ Step 43660/44000 | Loss: 2.9072 | LR: 7.62e-08 | 3489.0ms/step | 5.7 samples/s
938
+ Step 43670/44000 | Loss: 2.9112 | LR: 7.18e-08 | 3490.8ms/step | 5.7 samples/s
939
+ Step 43680/44000 | Loss: 2.9080 | LR: 6.75e-08 | 3488.9ms/step | 5.7 samples/s
940
+ Step 43690/44000 | Loss: 2.9058 | LR: 6.34e-08 | 3490.1ms/step | 5.7 samples/s
941
+ Step 43700/44000 | Loss: 2.8815 | LR: 5.94e-08 | 3489.8ms/step | 5.7 samples/s
942
+ Step 43710/44000 | Loss: 2.9148 | LR: 5.55e-08 | 3486.2ms/step | 5.7 samples/s
943
+ Step 43720/44000 | Loss: 2.8378 | LR: 5.17e-08 | 3491.0ms/step | 5.7 samples/s
944
+ Step 43730/44000 | Loss: 2.9154 | LR: 4.81e-08 | 3489.3ms/step | 5.7 samples/s
945
+ Step 43740/44000 | Loss: 2.8899 | LR: 4.46e-08 | 3494.0ms/step | 5.7 samples/s
946
+ Step 43750/44000 | Loss: 2.8759 | LR: 4.12e-08 | 3490.5ms/step | 5.7 samples/s
947
+ Validation at step 43750: Loss = 2.9042, Perplexity = 18.25
948
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-43750
949
+ Step 43760/44000 | Loss: 2.8712 | LR: 3.80e-08 | 9650.9ms/step | 2.1 samples/s
950
+ Step 43770/44000 | Loss: 2.8957 | LR: 3.49e-08 | 3499.5ms/step | 5.7 samples/s
951
+ Step 43780/44000 | Loss: 2.8833 | LR: 3.19e-08 | 3498.9ms/step | 5.7 samples/s
952
+ Step 43790/44000 | Loss: 2.9481 | LR: 2.91e-08 | 3496.0ms/step | 5.7 samples/s
953
+ Step 43800/44000 | Loss: 2.8936 | LR: 2.64e-08 | 3498.7ms/step | 5.7 samples/s
954
+ Step 43810/44000 | Loss: 2.9043 | LR: 2.38e-08 | 3497.7ms/step | 5.7 samples/s
955
+ Step 43820/44000 | Loss: 2.9065 | LR: 2.14e-08 | 3497.0ms/step | 5.7 samples/s
956
+ Step 43830/44000 | Loss: 2.9039 | LR: 1.91e-08 | 3499.3ms/step | 5.7 samples/s
957
+ Step 43840/44000 | Loss: 2.8869 | LR: 1.69e-08 | 3497.9ms/step | 5.7 samples/s
958
+ Step 43850/44000 | Loss: 2.8950 | LR: 1.48e-08 | 3498.2ms/step | 5.7 samples/s
959
+ Step 43860/44000 | Loss: 2.8989 | LR: 1.29e-08 | 3493.1ms/step | 5.7 samples/s
960
+ Step 43870/44000 | Loss: 2.9101 | LR: 1.11e-08 | 3490.4ms/step | 5.7 samples/s
961
+ Step 43880/44000 | Loss: 2.8646 | LR: 9.50e-09 | 3495.7ms/step | 5.7 samples/s
962
+ Step 43890/44000 | Loss: 2.8940 | LR: 7.98e-09 | 3496.3ms/step | 5.7 samples/s
963
+ Step 43900/44000 | Loss: 2.8921 | LR: 6.60e-09 | 3494.5ms/step | 5.7 samples/s
964
+ Step 43910/44000 | Loss: 2.9029 | LR: 5.34e-09 | 3497.0ms/step | 5.7 samples/s
965
+ Step 43920/44000 | Loss: 2.8727 | LR: 4.22e-09 | 3498.8ms/step | 5.7 samples/s
966
+ Step 43930/44000 | Loss: 2.8938 | LR: 3.23e-09 | 3497.3ms/step | 5.7 samples/s
967
+ Step 43940/44000 | Loss: 2.9038 | LR: 2.37e-09 | 3498.7ms/step | 5.7 samples/s
968
+ Step 43950/44000 | Loss: 2.9235 | LR: 1.65e-09 | 3496.6ms/step | 5.7 samples/s
969
+ Step 43960/44000 | Loss: 2.9303 | LR: 1.06e-09 | 3497.8ms/step | 5.7 samples/s
970
+ Step 43970/44000 | Loss: 2.8831 | LR: 5.94e-10 | 3496.0ms/step | 5.7 samples/s
971
+ Step 43980/44000 | Loss: 2.8954 | LR: 2.64e-10 | 3495.9ms/step | 5.7 samples/s
972
+ Step 43990/44000 | Loss: 2.8979 | LR: 6.60e-11 | 3499.0ms/step | 5.7 samples/s
973
+ Step 44000/44000 | Loss: 2.8788 | LR: 0.00e+00 | 3498.0ms/step | 5.7 samples/s
974
+ Validation at step 44000: Loss = 2.9042, Perplexity = 18.25
975
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-44000
976
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031/checkpoint-44000
epfl_final/SparseAtt/wandb/run-20260731_001110-rdj7s9cl/files/requirements.txt ADDED
@@ -0,0 +1,142 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ pip==26.0.1
2
+ nvidia-cusparselt-cu13==0.8.0
3
+ mpmath==1.3.0
4
+ cuda-toolkit==13.0.2
5
+ zstandard==0.25.0
6
+ xxhash==3.7.0
7
+ urllib3==2.6.3
8
+ typing_extensions==4.15.0
9
+ triton==3.6.0
10
+ tqdm==4.67.3
11
+ torchinfo==1.8.0
12
+ sympy==1.14.0
13
+ smmap==5.0.3
14
+ six==1.17.0
15
+ shellingham==1.5.4
16
+ setuptools==81.0.0
17
+ safetensors==0.7.0
18
+ regex==2026.4.4
19
+ PyYAML==6.0.3
20
+ Pygments==2.20.0
21
+ pyarrow==24.0.0
22
+ psutil==7.2.2
23
+ protobuf==7.34.1
24
+ propcache==0.4.1
25
+ platformdirs==4.9.6
26
+ packaging==26.2
27
+ nvidia-nvtx==13.0.85
28
+ nvidia-nvshmem-cu13==3.4.5
29
+ nvidia-nvjitlink==13.0.88
30
+ nvidia-nccl-cu13==2.28.9
31
+ nvidia-curand==10.4.0.35
32
+ nvidia-cufile==1.15.1.6
33
+ nvidia-cuda-runtime==13.0.96
34
+ nvidia-cuda-nvrtc==13.0.88
35
+ nvidia-cuda-cupti==13.0.85
36
+ nvidia-cublas==13.1.0.3
37
+ numpy==2.4.4
38
+ networkx==3.6.1
39
+ multidict==6.7.1
40
+ mdurl==0.1.2
41
+ MarkupSafe==3.0.3
42
+ idna==3.13
43
+ hf-xet==1.4.3
44
+ h11==0.16.0
45
+ fsspec==2026.2.0
46
+ frozenlist==1.8.0
47
+ filelock==3.29.0
48
+ dill==0.4.1
49
+ cuda-pathfinder==1.5.4
50
+ click==8.3.3
51
+ charset-normalizer==3.4.7
52
+ certifi==2026.4.22
53
+ attrs==26.1.0
54
+ annotated-types==0.7.0
55
+ annotated-doc==0.0.4
56
+ aiohappyeyeballs==2.6.1
57
+ yarl==1.23.0
58
+ typing-inspection==0.4.2
59
+ sentry-sdk==2.58.0
60
+ requests==2.33.1
61
+ python-dateutil==2.9.0.post0
62
+ pydantic_core==2.46.3
63
+ nvidia-cusparse==12.6.3.3
64
+ nvidia-cufft==12.0.0.61
65
+ nvidia-cudnn-cu13==9.19.0.56
66
+ multiprocess==0.70.19
67
+ markdown-it-py==4.0.0
68
+ Jinja2==3.1.6
69
+ httpcore==1.0.9
70
+ gitdb==4.0.12
71
+ cuda-bindings==13.2.0
72
+ anyio==4.13.0
73
+ aiosignal==1.4.0
74
+ rich==15.0.0
75
+ pydantic==2.13.3
76
+ pandas==3.0.2
77
+ nvidia-cusolver==12.0.4.66
78
+ httpx==0.28.1
79
+ GitPython==3.1.49
80
+ aiohttp==3.13.5
81
+ wandb==0.26.1
82
+ typer==0.25.1
83
+ torch==2.11.0
84
+ huggingface_hub==1.13.0
85
+ tokenizers==0.22.2
86
+ datasets==4.8.5
87
+ accelerate==1.13.0
88
+ transformers==5.7.0
89
+ word2number==1.1
90
+ sqlitedict==2.1.0
91
+ pytz==2026.2
92
+ threadpoolctl==3.6.0
93
+ tcolorpy==0.1.7
94
+ tabulate==0.10.0
95
+ scipy==1.17.1
96
+ portalocker==3.2.0
97
+ pathvalidate==3.3.1
98
+ narwhals==2.22.1
99
+ more-itertools==11.1.0
100
+ lxml==6.1.1
101
+ joblib==1.5.3
102
+ colorama==0.4.6
103
+ chardet==6.0.0.post1
104
+ absl-py==2.4.0
105
+ scikit-learn==1.9.0
106
+ sacrebleu==2.6.0
107
+ nltk==3.9.4
108
+ mbstrdecoder==1.1.5
109
+ typepy==1.3.5
110
+ rouge_score==0.1.2
111
+ DataProperty==1.1.1
112
+ tabledata==1.3.5
113
+ evaluate==0.4.6
114
+ pytablewriter==1.2.1
115
+ lm_eval==0.4.12
116
+ pluggy==1.6.0
117
+ iniconfig==2.3.0
118
+ pytest==9.1.0
119
+ py-spy==0.4.2
120
+ pillow==12.2.0
121
+ tifffile==2026.6.1
122
+ imagecodecs==2026.6.26
123
+ zipp==4.1.0
124
+ importlib_metadata==9.0.0
125
+ derivative==0.6.3
126
+ pysindy==2.1.0
127
+ pyparsing==3.3.2
128
+ kiwisolver==1.5.0
129
+ fonttools==4.63.0
130
+ cycler==0.12.1
131
+ contourpy==1.3.3
132
+ matplotlib==3.11.0
133
+ sortedcontainers==2.4.0
134
+ wcwidth==0.8.2
135
+ sigfig==1.3.19
136
+ importlib_resources==7.1.0
137
+ enum-tools==0.13.0
138
+ cramjam==2.11.0
139
+ python-snappy==0.7.3
140
+ compact-json==1.8.2
141
+ numbers-parser==4.18.5
142
+ entmax==1.3
epfl_final/SparseAtt/wandb/run-20260731_001110-rdj7s9cl/files/wandb-metadata.json ADDED
@@ -0,0 +1,65 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "os": "Linux-6.8.0-124-generic-x86_64-with-glibc2.35",
3
+ "python": "CPython 3.13.13",
4
+ "startedAt": "2026-07-31T00:11:10.398995Z",
5
+ "args": [
6
+ "--config",
7
+ "/mloscratch/homes/navasard/sparse/SparseAtt/configs/ladder8k/365m_mixed_to_dense_20pct_seed44_20260730.json"
8
+ ],
9
+ "program": "/mloscratch/homes/navasard/sparse/SparseAtt/train_dense_to_mixed_exact.py",
10
+ "codePath": "train_dense_to_mixed_exact.py",
11
+ "codePathLocal": "train_dense_to_mixed_exact.py",
12
+ "git": {
13
+ "remote": "https://github.com/rubentium/TokenPrune.git",
14
+ "commit": "88298c7a418622530ca5426b22b1e5ebd10ca0cb"
15
+ },
16
+ "email": "ruben.navasardyan@mail.utoronto.ca",
17
+ "root": "/mloscratch/homes/navasard/sparse/SparseAtt",
18
+ "host": "a100-train-4-gpu-2-0-40",
19
+ "executable": "/mloscratch/homes/navasard/conda/bin/python3.13",
20
+ "cpu_count": 64,
21
+ "cpu_count_logical": 128,
22
+ "gpu": "NVIDIA A100-SXM4-80GB",
23
+ "gpu_count": 4,
24
+ "disk": {
25
+ "/": {
26
+ "total": "7679362727936",
27
+ "used": "5257403871232"
28
+ }
29
+ },
30
+ "memory": {
31
+ "total": "1081884934144"
32
+ },
33
+ "gpu_nvidia": [
34
+ {
35
+ "name": "NVIDIA A100-SXM4-80GB",
36
+ "memoryTotal": "85899345920",
37
+ "cudaCores": 6912,
38
+ "architecture": "Ampere",
39
+ "uuid": "GPU-26e055ad-009a-c42a-2ba0-ce6db370fcbe"
40
+ },
41
+ {
42
+ "name": "NVIDIA A100-SXM4-80GB",
43
+ "memoryTotal": "85899345920",
44
+ "cudaCores": 6912,
45
+ "architecture": "Ampere",
46
+ "uuid": "GPU-56ec1834-3584-3932-0125-388b5e88fac2"
47
+ },
48
+ {
49
+ "name": "NVIDIA A100-SXM4-80GB",
50
+ "memoryTotal": "85899345920",
51
+ "cudaCores": 6912,
52
+ "architecture": "Ampere",
53
+ "uuid": "GPU-ce50edb9-4d69-8d20-4433-5217084c7016"
54
+ },
55
+ {
56
+ "name": "NVIDIA A100-SXM4-80GB",
57
+ "memoryTotal": "85899345920",
58
+ "cudaCores": 6912,
59
+ "architecture": "Ampere",
60
+ "uuid": "GPU-2401254a-acb7-5787-0b06-07065349e511"
61
+ }
62
+ ],
63
+ "cudaVersion": "13.0",
64
+ "writerId": "hq00yk9icqnt8k6789b401jp03v00xmk"
65
+ }
epfl_final/SparseAtt/wandb/run-20260731_001110-rdj7s9cl/files/wandb-summary.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"eval/dense_interval_active":0,"eval/perplexity":18.25097144452431,"eval/dense_interval_end":-1,"eval/uses_dense_attention":1,"train/loss":2.8787854957580565,"train/learning_rate":0,"speed/samples_per_sec":5.717556547429583,"train/epoch":0.2906439083679023,"_wandb":{"runtime":34171},"eval/dense_interval_start":-1,"train/global_step":44000,"_runtime":34171.874461433,"_timestamp":1.7854908357930205e+09,"speed/steps_per_sec":0.28587782737147915,"_step":44000,"eval/uses_sparse_attention":0,"speed/ms_per_step":3497.997760772705,"eval/loss":2.9042183084487916}
epfl_final/SparseAtt/wandb/run-20260731_001110-rdj7s9cl/logs/debug-internal.log ADDED
The diff for this file is too large to render. See raw diff
 
epfl_final/SparseAtt/wandb/run-20260731_001110-rdj7s9cl/logs/debug.log ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2026-07-31 00:11:10,415 INFO MainThread:158 [wandb_setup.py:_flush():81] Current SDK version is 0.26.1
2
+ 2026-07-31 00:11:10,415 INFO MainThread:158 [wandb_setup.py:_flush():81] Configure stats pid to 158
3
+ 2026-07-31 00:11:10,416 INFO MainThread:158 [wandb_setup.py:_flush():81] Loading settings from environment variables
4
+ 2026-07-31 00:11:10,416 INFO MainThread:158 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /mloscratch/homes/navasard/sparse/SparseAtt/wandb/run-20260731_001110-rdj7s9cl/logs/debug.log
5
+ 2026-07-31 00:11:10,416 INFO MainThread:158 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /mloscratch/homes/navasard/sparse/SparseAtt/wandb/run-20260731_001110-rdj7s9cl/logs/debug-internal.log
6
+ 2026-07-31 00:11:10,416 INFO MainThread:158 [wandb_init.py:init():850] calling init triggers
7
+ 2026-07-31 00:11:10,416 INFO MainThread:158 [wandb_init.py:init():855] wandb.init called with sweep_config: {}
8
+ config: {'model_path': '/mloscratch/homes/navasard/qwen3_06b_base', 'model_dtype': 'bfloat16', 'init_from_scratch': True, 'train_path': '/mloscratch/homes/navasard/data/fineweb_large/tokenized', 'val_path': None, 'max_seq_length': 8192, 'num_workers': 8, 'output_dir': '/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed44/run_20260731_001031', 'resume_from_checkpoint': '/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_seed44/run_20260722_221755/checkpoint-35000', 'force_new_run': True, 'num_epochs': 1, 'max_steps': 44000, 'per_device_batch_size': 2, 'gradient_accumulation_steps': 10, 'learning_rate': 0.0005, 'weight_decay': 0.01, 'warmup_steps': 750, 'lr_scheduler': 'cosine', 'max_grad_norm': 1.0, 'save_steps': 250, 'eval_steps': 250, 'eval_batches': 50, 'logging_steps': 10, 'seed': 44, 'verbose': False, 'checkpoint_activations': False, 'strategy': 'ddp', 'mixed_precision': 'bf16', 'wandb_enabled': True, 'wandb_project': 'stochastic_sparsity', 'wandb_name': '365m mixed-to-dense 20pct seed44', 'wandb_tags': ['qwen3', '365m', '8k', 'mixed-to-dense', '20pct', 'seed44', 'multiseed'], 'hidden_size': 768, 'intermediate_size': 3072, 'num_attention_heads': 6, 'num_key_value_heads': 2, 'num_hidden_layers': 28, 'head_dim': None, 'attention_type': 'sparse_stochastic', 'sparse_block_size': 128, 'sparse_num_random_blocks': 1, 'sparse_mask_pool_size': 1, 'sparse_val_dense_compare': False, 'sparse_layer_indices': None, 'sparse_mask_cache_dir': None, 'sparse_mask_seed': 0, 'dense_mode': True, 'dense_step_intervals': None, '_wandb': {}}
9
+ 2026-07-31 00:11:10,416 INFO MainThread:158 [wandb_init.py:init():898] starting backend
10
+ 2026-07-31 00:11:10,665 INFO MainThread:158 [wandb_init.py:init():913] sending inform_init request
11
+ 2026-07-31 00:11:11,165 INFO MainThread:158 [wandb_init.py:init():918] backend started and connected
12
+ 2026-07-31 00:11:11,170 INFO MainThread:158 [wandb_init.py:init():988] updated telemetry
13
+ 2026-07-31 00:11:11,188 INFO MainThread:158 [wandb_init.py:init():1011] communicating run to backend with 90.0 second timeout
14
+ 2026-07-31 00:11:11,820 INFO MainThread:158 [wandb_init.py:init():1056] starting run threads in backend
15
+ 2026-07-31 00:11:12,035 INFO MainThread:158 [wandb_run.py:_console_start():2554] atexit reg
16
+ 2026-07-31 00:11:12,035 INFO MainThread:158 [wandb_run.py:_redirect():2403] redirect: wrap_raw
17
+ 2026-07-31 00:11:12,035 INFO MainThread:158 [wandb_run.py:_redirect():2472] Wrapping output streams.
18
+ 2026-07-31 00:11:12,035 INFO MainThread:158 [wandb_run.py:_redirect():2495] Redirects installed.
19
+ 2026-07-31 00:11:12,043 INFO MainThread:158 [wandb_init.py:init():1094] run started, returning control to user process
20
+ 2026-07-31 09:40:43,694 INFO MainThread:158 [wandb_run.py:_finish():2321] finishing run ruben-navasardyan4869-epfl/stochastic_sparsity/rdj7s9cl
21
+ 2026-07-31 09:40:43,694 INFO MainThread:158 [wandb_run.py:_atexit_cleanup():2520] got exitcode: 0
22
+ 2026-07-31 09:40:43,695 INFO MainThread:158 [wandb_run.py:_restore():2502] restore
23
+ 2026-07-31 09:40:43,695 INFO MainThread:158 [wandb_run.py:_restore():2508] restore done
24
+ 2026-07-31 09:40:45,434 INFO MainThread:158 [wandb_run.py:_footer_sync_info():3889] logging synced files
epfl_final/SparseAtt/wandb/run-20260731_001110-rdj7s9cl/run-rdj7s9cl.wandb ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9ebfd0cf29d9eb73127e6f13b9bfb763cedd818fa8c07f41ae9ba733710da218
3
+ size 6125567
epfl_final/SparseAtt/wandb/run-20260731_001110-w1uk4jj8/files/config.yaml ADDED
@@ -0,0 +1,187 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ _wandb:
2
+ value:
3
+ cli_version: 0.26.1
4
+ e:
5
+ l2bnjgzsfkl6v3e40bxm38p36dv28k34:
6
+ args:
7
+ - --config
8
+ - /mloscratch/homes/navasard/sparse/SparseAtt/configs/ladder8k/365m_mixed_to_dense_20pct_seed45_20260730.json
9
+ codePath: train_dense_to_mixed_exact.py
10
+ codePathLocal: train_dense_to_mixed_exact.py
11
+ cpu_count: 64
12
+ cpu_count_logical: 128
13
+ cudaVersion: "13.0"
14
+ disk:
15
+ /:
16
+ total: "7679362727936"
17
+ used: "5257403871232"
18
+ email: ruben.navasardyan@mail.utoronto.ca
19
+ executable: /mloscratch/homes/navasard/conda/bin/python3.13
20
+ git:
21
+ commit: 88298c7a418622530ca5426b22b1e5ebd10ca0cb
22
+ remote: https://github.com/rubentium/TokenPrune.git
23
+ gpu: NVIDIA A100-SXM4-80GB
24
+ gpu_count: 4
25
+ gpu_nvidia:
26
+ - architecture: Ampere
27
+ cudaCores: 6912
28
+ memoryTotal: "85899345920"
29
+ name: NVIDIA A100-SXM4-80GB
30
+ uuid: GPU-26e055ad-009a-c42a-2ba0-ce6db370fcbe
31
+ - architecture: Ampere
32
+ cudaCores: 6912
33
+ memoryTotal: "85899345920"
34
+ name: NVIDIA A100-SXM4-80GB
35
+ uuid: GPU-56ec1834-3584-3932-0125-388b5e88fac2
36
+ - architecture: Ampere
37
+ cudaCores: 6912
38
+ memoryTotal: "85899345920"
39
+ name: NVIDIA A100-SXM4-80GB
40
+ uuid: GPU-ce50edb9-4d69-8d20-4433-5217084c7016
41
+ - architecture: Ampere
42
+ cudaCores: 6912
43
+ memoryTotal: "85899345920"
44
+ name: NVIDIA A100-SXM4-80GB
45
+ uuid: GPU-2401254a-acb7-5787-0b06-07065349e511
46
+ host: a100-train-4-gpu-2-0-40
47
+ memory:
48
+ total: "1081884934144"
49
+ os: Linux-6.8.0-124-generic-x86_64-with-glibc2.35
50
+ program: /mloscratch/homes/navasard/sparse/SparseAtt/train_dense_to_mixed_exact.py
51
+ python: CPython 3.13.13
52
+ root: /mloscratch/homes/navasard/sparse/SparseAtt
53
+ startedAt: "2026-07-31T00:11:10.199782Z"
54
+ writerId: l2bnjgzsfkl6v3e40bxm38p36dv28k34
55
+ m: []
56
+ python_version: 3.13.13
57
+ t:
58
+ "1":
59
+ - 1
60
+ - 5
61
+ - 11
62
+ - 49
63
+ - 51
64
+ - 53
65
+ - 71
66
+ "2":
67
+ - 1
68
+ - 5
69
+ - 11
70
+ - 49
71
+ - 51
72
+ - 53
73
+ - 71
74
+ "3":
75
+ - 2
76
+ - 13
77
+ - 15
78
+ - 16
79
+ - 61
80
+ "4": 3.13.13
81
+ "5": 0.26.1
82
+ "6": 5.7.0
83
+ "12": 0.26.1
84
+ "13": linux-x86_64
85
+ attention_type:
86
+ value: sparse_stochastic
87
+ checkpoint_activations:
88
+ value: false
89
+ dense_mode:
90
+ value: true
91
+ dense_step_intervals:
92
+ value: null
93
+ eval_batches:
94
+ value: 50
95
+ eval_steps:
96
+ value: 250
97
+ force_new_run:
98
+ value: true
99
+ gradient_accumulation_steps:
100
+ value: 10
101
+ head_dim:
102
+ value: null
103
+ hidden_size:
104
+ value: 768
105
+ init_from_scratch:
106
+ value: true
107
+ intermediate_size:
108
+ value: 3072
109
+ learning_rate:
110
+ value: 0.0005
111
+ logging_steps:
112
+ value: 10
113
+ lr_scheduler:
114
+ value: cosine
115
+ max_grad_norm:
116
+ value: 1
117
+ max_seq_length:
118
+ value: 8192
119
+ max_steps:
120
+ value: 44000
121
+ mixed_precision:
122
+ value: bf16
123
+ model_dtype:
124
+ value: bfloat16
125
+ model_path:
126
+ value: /mloscratch/homes/navasard/qwen3_06b_base
127
+ num_attention_heads:
128
+ value: 6
129
+ num_epochs:
130
+ value: 1
131
+ num_hidden_layers:
132
+ value: 28
133
+ num_key_value_heads:
134
+ value: 2
135
+ num_workers:
136
+ value: 8
137
+ output_dir:
138
+ value: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031
139
+ per_device_batch_size:
140
+ value: 2
141
+ resume_from_checkpoint:
142
+ value: /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_seed45/run_20260722_223953/checkpoint-35000
143
+ save_steps:
144
+ value: 250
145
+ seed:
146
+ value: 45
147
+ sparse_block_size:
148
+ value: 128
149
+ sparse_layer_indices:
150
+ value: null
151
+ sparse_mask_cache_dir:
152
+ value: null
153
+ sparse_mask_pool_size:
154
+ value: 1
155
+ sparse_mask_seed:
156
+ value: 0
157
+ sparse_num_random_blocks:
158
+ value: 1
159
+ sparse_val_dense_compare:
160
+ value: false
161
+ strategy:
162
+ value: ddp
163
+ train_path:
164
+ value: /mloscratch/homes/navasard/data/fineweb_large/tokenized
165
+ val_path:
166
+ value: null
167
+ verbose:
168
+ value: false
169
+ wandb_enabled:
170
+ value: true
171
+ wandb_name:
172
+ value: 365m mixed-to-dense 20pct seed45
173
+ wandb_project:
174
+ value: stochastic_sparsity
175
+ wandb_tags:
176
+ value:
177
+ - qwen3
178
+ - 365m
179
+ - 8k
180
+ - mixed-to-dense
181
+ - 20pct
182
+ - seed45
183
+ - multiseed
184
+ warmup_steps:
185
+ value: 750
186
+ weight_decay:
187
+ value: 0.01
epfl_final/SparseAtt/wandb/run-20260731_001110-w1uk4jj8/files/output.log ADDED
@@ -0,0 +1,976 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Starting training...
2
+ Validation at step 35000: Loss = 2.9092, Perplexity = 18.34
3
+ Switching attention mode for step 35001: dense (dense step interval)
4
+ Step 35010/44000 | Loss: 2.9377 | LR: 5.14e-05 | 68150.1ms/step | 0.3 samples/s
5
+ Step 35020/44000 | Loss: 2.9060 | LR: 5.13e-05 | 3453.6ms/step | 5.8 samples/s
6
+ Step 35030/44000 | Loss: 2.9033 | LR: 5.12e-05 | 3450.9ms/step | 5.8 samples/s
7
+ Step 35040/44000 | Loss: 2.8888 | LR: 5.11e-05 | 3453.3ms/step | 5.8 samples/s
8
+ Step 35050/44000 | Loss: 2.8646 | LR: 5.10e-05 | 3452.8ms/step | 5.8 samples/s
9
+ Step 35060/44000 | Loss: 2.8789 | LR: 5.09e-05 | 3458.6ms/step | 5.8 samples/s
10
+ Step 35070/44000 | Loss: 2.9272 | LR: 5.08e-05 | 3459.6ms/step | 5.8 samples/s
11
+ Step 35080/44000 | Loss: 2.9021 | LR: 5.07e-05 | 3458.9ms/step | 5.8 samples/s
12
+ Step 35090/44000 | Loss: 2.8745 | LR: 5.06e-05 | 3456.8ms/step | 5.8 samples/s
13
+ Step 35100/44000 | Loss: 2.9034 | LR: 5.04e-05 | 3457.3ms/step | 5.8 samples/s
14
+ Step 35110/44000 | Loss: 2.8817 | LR: 5.03e-05 | 3459.3ms/step | 5.8 samples/s
15
+ Step 35120/44000 | Loss: 2.9228 | LR: 5.02e-05 | 3460.3ms/step | 5.8 samples/s
16
+ Step 35130/44000 | Loss: 2.8817 | LR: 5.01e-05 | 3460.2ms/step | 5.8 samples/s
17
+ Step 35140/44000 | Loss: 2.9023 | LR: 5.00e-05 | 3462.4ms/step | 5.8 samples/s
18
+ Step 35150/44000 | Loss: 2.8977 | LR: 4.99e-05 | 3461.8ms/step | 5.8 samples/s
19
+ Step 35160/44000 | Loss: 2.8756 | LR: 4.98e-05 | 3463.7ms/step | 5.8 samples/s
20
+ Step 35170/44000 | Loss: 2.8780 | LR: 4.97e-05 | 3458.6ms/step | 5.8 samples/s
21
+ Step 35180/44000 | Loss: 2.8855 | LR: 4.96e-05 | 3460.7ms/step | 5.8 samples/s
22
+ Step 35190/44000 | Loss: 2.8930 | LR: 4.95e-05 | 3458.2ms/step | 5.8 samples/s
23
+ Step 35200/44000 | Loss: 2.9029 | LR: 4.94e-05 | 3461.0ms/step | 5.8 samples/s
24
+ Step 35210/44000 | Loss: 2.9185 | LR: 4.93e-05 | 3459.5ms/step | 5.8 samples/s
25
+ Step 35220/44000 | Loss: 2.9032 | LR: 4.91e-05 | 3458.2ms/step | 5.8 samples/s
26
+ Step 35230/44000 | Loss: 2.9175 | LR: 4.90e-05 | 3459.6ms/step | 5.8 samples/s
27
+ Step 35240/44000 | Loss: 2.8859 | LR: 4.89e-05 | 3461.1ms/step | 5.8 samples/s
28
+ Step 35250/44000 | Loss: 2.9422 | LR: 4.88e-05 | 3462.9ms/step | 5.8 samples/s
29
+ Validation at step 35250: Loss = 2.9019, Perplexity = 18.21
30
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-35250
31
+ Step 35260/44000 | Loss: 2.9122 | LR: 4.87e-05 | 9592.0ms/step | 2.1 samples/s
32
+ Step 35270/44000 | Loss: 2.8996 | LR: 4.86e-05 | 3463.3ms/step | 5.8 samples/s
33
+ Step 35280/44000 | Loss: 2.8885 | LR: 4.85e-05 | 3465.0ms/step | 5.8 samples/s
34
+ Step 35290/44000 | Loss: 2.9099 | LR: 4.84e-05 | 3465.7ms/step | 5.8 samples/s
35
+ Step 35300/44000 | Loss: 2.8953 | LR: 4.83e-05 | 3464.7ms/step | 5.8 samples/s
36
+ Step 35310/44000 | Loss: 2.9092 | LR: 4.82e-05 | 3465.2ms/step | 5.8 samples/s
37
+ Step 35320/44000 | Loss: 2.9361 | LR: 4.81e-05 | 3466.4ms/step | 5.8 samples/s
38
+ Step 35330/44000 | Loss: 2.8774 | LR: 4.80e-05 | 3470.8ms/step | 5.8 samples/s
39
+ Step 35340/44000 | Loss: 2.8882 | LR: 4.79e-05 | 3464.4ms/step | 5.8 samples/s
40
+ Step 35350/44000 | Loss: 2.9080 | LR: 4.77e-05 | 3465.5ms/step | 5.8 samples/s
41
+ Step 35360/44000 | Loss: 2.8858 | LR: 4.76e-05 | 3464.3ms/step | 5.8 samples/s
42
+ Step 35370/44000 | Loss: 2.9052 | LR: 4.75e-05 | 3463.5ms/step | 5.8 samples/s
43
+ Step 35380/44000 | Loss: 2.9026 | LR: 4.74e-05 | 3466.9ms/step | 5.8 samples/s
44
+ Step 35390/44000 | Loss: 2.8800 | LR: 4.73e-05 | 3468.6ms/step | 5.8 samples/s
45
+ Step 35400/44000 | Loss: 2.8856 | LR: 4.72e-05 | 3478.6ms/step | 5.7 samples/s
46
+ Step 35410/44000 | Loss: 2.8751 | LR: 4.71e-05 | 3488.9ms/step | 5.7 samples/s
47
+ Step 35420/44000 | Loss: 2.8596 | LR: 4.70e-05 | 3500.1ms/step | 5.7 samples/s
48
+ Step 35430/44000 | Loss: 2.9101 | LR: 4.69e-05 | 3505.2ms/step | 5.7 samples/s
49
+ Step 35440/44000 | Loss: 2.8985 | LR: 4.68e-05 | 3500.6ms/step | 5.7 samples/s
50
+ Step 35450/44000 | Loss: 2.9013 | LR: 4.67e-05 | 3503.9ms/step | 5.7 samples/s
51
+ Step 35460/44000 | Loss: 2.8919 | LR: 4.66e-05 | 3505.1ms/step | 5.7 samples/s
52
+ Step 35470/44000 | Loss: 2.9081 | LR: 4.65e-05 | 3506.9ms/step | 5.7 samples/s
53
+ Step 35480/44000 | Loss: 2.8936 | LR: 4.64e-05 | 3506.9ms/step | 5.7 samples/s
54
+ Step 35490/44000 | Loss: 2.8864 | LR: 4.63e-05 | 3506.8ms/step | 5.7 samples/s
55
+ Step 35500/44000 | Loss: 2.8652 | LR: 4.62e-05 | 3507.5ms/step | 5.7 samples/s
56
+ Validation at step 35500: Loss = 2.9012, Perplexity = 18.20
57
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-35500
58
+ Step 35510/44000 | Loss: 2.9016 | LR: 4.61e-05 | 9704.3ms/step | 2.1 samples/s
59
+ Step 35520/44000 | Loss: 2.9092 | LR: 4.59e-05 | 3513.7ms/step | 5.7 samples/s
60
+ Step 35530/44000 | Loss: 2.9081 | LR: 4.58e-05 | 3515.2ms/step | 5.7 samples/s
61
+ Step 35540/44000 | Loss: 2.8781 | LR: 4.57e-05 | 3514.7ms/step | 5.7 samples/s
62
+ Step 35550/44000 | Loss: 2.8912 | LR: 4.56e-05 | 3515.4ms/step | 5.7 samples/s
63
+ Step 35560/44000 | Loss: 2.9028 | LR: 4.55e-05 | 3516.3ms/step | 5.7 samples/s
64
+ Step 35570/44000 | Loss: 2.8937 | LR: 4.54e-05 | 3513.6ms/step | 5.7 samples/s
65
+ Step 35580/44000 | Loss: 2.8812 | LR: 4.53e-05 | 3512.4ms/step | 5.7 samples/s
66
+ Step 35590/44000 | Loss: 2.9100 | LR: 4.52e-05 | 3517.8ms/step | 5.7 samples/s
67
+ Step 35600/44000 | Loss: 2.9163 | LR: 4.51e-05 | 3520.6ms/step | 5.7 samples/s
68
+ Step 35610/44000 | Loss: 2.8726 | LR: 4.50e-05 | 3513.6ms/step | 5.7 samples/s
69
+ Step 35620/44000 | Loss: 2.9125 | LR: 4.49e-05 | 3511.3ms/step | 5.7 samples/s
70
+ Step 35630/44000 | Loss: 2.8712 | LR: 4.48e-05 | 3512.5ms/step | 5.7 samples/s
71
+ Step 35640/44000 | Loss: 2.9044 | LR: 4.47e-05 | 3510.3ms/step | 5.7 samples/s
72
+ Step 35650/44000 | Loss: 2.8770 | LR: 4.46e-05 | 3514.7ms/step | 5.7 samples/s
73
+ Step 35660/44000 | Loss: 2.8893 | LR: 4.45e-05 | 3511.2ms/step | 5.7 samples/s
74
+ Step 35670/44000 | Loss: 2.9223 | LR: 4.44e-05 | 3513.2ms/step | 5.7 samples/s
75
+ Step 35680/44000 | Loss: 2.8707 | LR: 4.43e-05 | 3515.9ms/step | 5.7 samples/s
76
+ Step 35690/44000 | Loss: 2.8729 | LR: 4.42e-05 | 3522.0ms/step | 5.7 samples/s
77
+ Step 35700/44000 | Loss: 2.9054 | LR: 4.41e-05 | 3514.9ms/step | 5.7 samples/s
78
+ Step 35710/44000 | Loss: 2.8788 | LR: 4.40e-05 | 3513.6ms/step | 5.7 samples/s
79
+ Step 35720/44000 | Loss: 2.8799 | LR: 4.39e-05 | 3521.7ms/step | 5.7 samples/s
80
+ Step 35730/44000 | Loss: 2.8875 | LR: 4.38e-05 | 3515.6ms/step | 5.7 samples/s
81
+ Step 35740/44000 | Loss: 2.8942 | LR: 4.37e-05 | 3511.7ms/step | 5.7 samples/s
82
+ Step 35750/44000 | Loss: 2.8964 | LR: 4.36e-05 | 3511.5ms/step | 5.7 samples/s
83
+ Validation at step 35750: Loss = 2.9008, Perplexity = 18.19
84
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-35750
85
+ Step 35760/44000 | Loss: 2.8979 | LR: 4.35e-05 | 9704.7ms/step | 2.1 samples/s
86
+ Step 35770/44000 | Loss: 2.9142 | LR: 4.34e-05 | 3511.9ms/step | 5.7 samples/s
87
+ Step 35780/44000 | Loss: 2.8922 | LR: 4.33e-05 | 3524.4ms/step | 5.7 samples/s
88
+ Step 35790/44000 | Loss: 2.8767 | LR: 4.32e-05 | 3524.5ms/step | 5.7 samples/s
89
+ Step 35800/44000 | Loss: 2.8789 | LR: 4.31e-05 | 3516.0ms/step | 5.7 samples/s
90
+ Step 35810/44000 | Loss: 2.9052 | LR: 4.29e-05 | 3513.4ms/step | 5.7 samples/s
91
+ Step 35820/44000 | Loss: 2.8758 | LR: 4.28e-05 | 3513.6ms/step | 5.7 samples/s
92
+ Step 35830/44000 | Loss: 2.8955 | LR: 4.27e-05 | 3516.5ms/step | 5.7 samples/s
93
+ Step 35840/44000 | Loss: 2.9042 | LR: 4.26e-05 | 3524.6ms/step | 5.7 samples/s
94
+ Step 35850/44000 | Loss: 2.9031 | LR: 4.25e-05 | 3516.8ms/step | 5.7 samples/s
95
+ Step 35860/44000 | Loss: 2.8867 | LR: 4.24e-05 | 3513.6ms/step | 5.7 samples/s
96
+ Step 35870/44000 | Loss: 2.9103 | LR: 4.23e-05 | 3512.2ms/step | 5.7 samples/s
97
+ Step 35880/44000 | Loss: 2.8846 | LR: 4.22e-05 | 3515.9ms/step | 5.7 samples/s
98
+ Step 35890/44000 | Loss: 2.9125 | LR: 4.21e-05 | 3512.4ms/step | 5.7 samples/s
99
+ Step 35900/44000 | Loss: 2.8684 | LR: 4.20e-05 | 3514.7ms/step | 5.7 samples/s
100
+ Step 35910/44000 | Loss: 2.8843 | LR: 4.19e-05 | 3521.2ms/step | 5.7 samples/s
101
+ Step 35920/44000 | Loss: 2.8962 | LR: 4.18e-05 | 3526.5ms/step | 5.7 samples/s
102
+ Step 35930/44000 | Loss: 2.8742 | LR: 4.17e-05 | 3525.8ms/step | 5.7 samples/s
103
+ Step 35940/44000 | Loss: 2.8772 | LR: 4.16e-05 | 3523.8ms/step | 5.7 samples/s
104
+ Step 35950/44000 | Loss: 2.8979 | LR: 4.15e-05 | 3524.3ms/step | 5.7 samples/s
105
+ Step 35960/44000 | Loss: 2.8935 | LR: 4.14e-05 | 3520.2ms/step | 5.7 samples/s
106
+ Step 35970/44000 | Loss: 2.9452 | LR: 4.13e-05 | 3512.5ms/step | 5.7 samples/s
107
+ Step 35980/44000 | Loss: 2.9020 | LR: 4.12e-05 | 3513.4ms/step | 5.7 samples/s
108
+ Step 35990/44000 | Loss: 2.8794 | LR: 4.11e-05 | 3524.5ms/step | 5.7 samples/s
109
+ Step 36000/44000 | Loss: 2.9097 | LR: 4.10e-05 | 3507.5ms/step | 5.7 samples/s
110
+ Validation at step 36000: Loss = 2.9005, Perplexity = 18.18
111
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-36000
112
+ Step 36010/44000 | Loss: 2.8842 | LR: 4.09e-05 | 9699.5ms/step | 2.1 samples/s
113
+ Step 36020/44000 | Loss: 2.9109 | LR: 4.08e-05 | 3513.1ms/step | 5.7 samples/s
114
+ Step 36030/44000 | Loss: 2.8868 | LR: 4.07e-05 | 3512.3ms/step | 5.7 samples/s
115
+ Step 36040/44000 | Loss: 2.8964 | LR: 4.06e-05 | 3510.8ms/step | 5.7 samples/s
116
+ Step 36050/44000 | Loss: 2.8538 | LR: 4.05e-05 | 3508.6ms/step | 5.7 samples/s
117
+ Step 36060/44000 | Loss: 2.8663 | LR: 4.04e-05 | 3508.3ms/step | 5.7 samples/s
118
+ Step 36070/44000 | Loss: 2.8925 | LR: 4.03e-05 | 3516.5ms/step | 5.7 samples/s
119
+ Step 36080/44000 | Loss: 2.9017 | LR: 4.02e-05 | 3520.3ms/step | 5.7 samples/s
120
+ Step 36090/44000 | Loss: 2.9015 | LR: 4.01e-05 | 3522.4ms/step | 5.7 samples/s
121
+ Step 36100/44000 | Loss: 2.8803 | LR: 4.00e-05 | 3516.3ms/step | 5.7 samples/s
122
+ Step 36110/44000 | Loss: 2.8900 | LR: 3.99e-05 | 3515.0ms/step | 5.7 samples/s
123
+ Step 36120/44000 | Loss: 2.8931 | LR: 3.98e-05 | 3513.3ms/step | 5.7 samples/s
124
+ Step 36130/44000 | Loss: 2.9064 | LR: 3.97e-05 | 3513.2ms/step | 5.7 samples/s
125
+ Step 36140/44000 | Loss: 2.8746 | LR: 3.97e-05 | 3516.7ms/step | 5.7 samples/s
126
+ Step 36150/44000 | Loss: 2.8657 | LR: 3.96e-05 | 3512.5ms/step | 5.7 samples/s
127
+ Step 36160/44000 | Loss: 2.8819 | LR: 3.95e-05 | 3512.3ms/step | 5.7 samples/s
128
+ Step 36170/44000 | Loss: 2.8864 | LR: 3.94e-05 | 3510.0ms/step | 5.7 samples/s
129
+ Step 36180/44000 | Loss: 2.9052 | LR: 3.93e-05 | 3516.0ms/step | 5.7 samples/s
130
+ Step 36190/44000 | Loss: 2.8915 | LR: 3.92e-05 | 3513.1ms/step | 5.7 samples/s
131
+ Step 36200/44000 | Loss: 2.8997 | LR: 3.91e-05 | 3512.6ms/step | 5.7 samples/s
132
+ Step 36210/44000 | Loss: 2.8418 | LR: 3.90e-05 | 3512.5ms/step | 5.7 samples/s
133
+ Step 36220/44000 | Loss: 2.8752 | LR: 3.89e-05 | 3513.5ms/step | 5.7 samples/s
134
+ Step 36230/44000 | Loss: 2.9115 | LR: 3.88e-05 | 3513.6ms/step | 5.7 samples/s
135
+ Step 36240/44000 | Loss: 2.9043 | LR: 3.87e-05 | 3513.5ms/step | 5.7 samples/s
136
+ Step 36250/44000 | Loss: 2.8703 | LR: 3.86e-05 | 3517.9ms/step | 5.7 samples/s
137
+ Validation at step 36250: Loss = 2.9002, Perplexity = 18.18
138
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-36250
139
+ Step 36260/44000 | Loss: 2.8785 | LR: 3.85e-05 | 9722.4ms/step | 2.1 samples/s
140
+ Step 36270/44000 | Loss: 2.8938 | LR: 3.84e-05 | 3521.6ms/step | 5.7 samples/s
141
+ Step 36280/44000 | Loss: 2.8772 | LR: 3.83e-05 | 3522.2ms/step | 5.7 samples/s
142
+ Step 36290/44000 | Loss: 2.9042 | LR: 3.82e-05 | 3517.8ms/step | 5.7 samples/s
143
+ Step 36300/44000 | Loss: 2.8947 | LR: 3.81e-05 | 3524.1ms/step | 5.7 samples/s
144
+ Step 36310/44000 | Loss: 2.9090 | LR: 3.80e-05 | 3519.3ms/step | 5.7 samples/s
145
+ Step 36320/44000 | Loss: 2.8877 | LR: 3.79e-05 | 3513.0ms/step | 5.7 samples/s
146
+ Step 36330/44000 | Loss: 2.9157 | LR: 3.78e-05 | 3515.3ms/step | 5.7 samples/s
147
+ Step 36340/44000 | Loss: 2.8405 | LR: 3.77e-05 | 3515.4ms/step | 5.7 samples/s
148
+ Step 36350/44000 | Loss: 2.8888 | LR: 3.76e-05 | 3514.1ms/step | 5.7 samples/s
149
+ Step 36360/44000 | Loss: 2.8790 | LR: 3.75e-05 | 3514.8ms/step | 5.7 samples/s
150
+ Step 36370/44000 | Loss: 2.8850 | LR: 3.74e-05 | 3517.6ms/step | 5.7 samples/s
151
+ Step 36380/44000 | Loss: 2.8572 | LR: 3.73e-05 | 3513.9ms/step | 5.7 samples/s
152
+ Step 36390/44000 | Loss: 2.8850 | LR: 3.72e-05 | 3523.4ms/step | 5.7 samples/s
153
+ Step 36400/44000 | Loss: 2.9069 | LR: 3.71e-05 | 3518.6ms/step | 5.7 samples/s
154
+ Step 36410/44000 | Loss: 2.8919 | LR: 3.70e-05 | 3521.8ms/step | 5.7 samples/s
155
+ Step 36420/44000 | Loss: 2.9247 | LR: 3.69e-05 | 3515.3ms/step | 5.7 samples/s
156
+ Step 36430/44000 | Loss: 2.8830 | LR: 3.69e-05 | 3513.1ms/step | 5.7 samples/s
157
+ Step 36440/44000 | Loss: 2.8461 | LR: 3.68e-05 | 3521.8ms/step | 5.7 samples/s
158
+ Step 36450/44000 | Loss: 2.8588 | LR: 3.67e-05 | 3525.1ms/step | 5.7 samples/s
159
+ Step 36460/44000 | Loss: 2.9021 | LR: 3.66e-05 | 3525.3ms/step | 5.7 samples/s
160
+ Step 36470/44000 | Loss: 2.8595 | LR: 3.65e-05 | 3513.3ms/step | 5.7 samples/s
161
+ Step 36480/44000 | Loss: 2.8886 | LR: 3.64e-05 | 3516.9ms/step | 5.7 samples/s
162
+ Step 36490/44000 | Loss: 2.8721 | LR: 3.63e-05 | 3516.6ms/step | 5.7 samples/s
163
+ Step 36500/44000 | Loss: 2.8810 | LR: 3.62e-05 | 3516.1ms/step | 5.7 samples/s
164
+ Validation at step 36500: Loss = 2.9000, Perplexity = 18.17
165
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-36500
166
+ Step 36510/44000 | Loss: 2.8812 | LR: 3.61e-05 | 9704.6ms/step | 2.1 samples/s
167
+ Step 36520/44000 | Loss: 2.8989 | LR: 3.60e-05 | 3508.4ms/step | 5.7 samples/s
168
+ Step 36530/44000 | Loss: 2.8833 | LR: 3.59e-05 | 3507.9ms/step | 5.7 samples/s
169
+ Step 36540/44000 | Loss: 2.8890 | LR: 3.58e-05 | 3513.5ms/step | 5.7 samples/s
170
+ Step 36550/44000 | Loss: 2.9074 | LR: 3.57e-05 | 3511.8ms/step | 5.7 samples/s
171
+ Step 36560/44000 | Loss: 2.8815 | LR: 3.56e-05 | 3513.3ms/step | 5.7 samples/s
172
+ Step 36570/44000 | Loss: 2.9039 | LR: 3.55e-05 | 3512.5ms/step | 5.7 samples/s
173
+ Step 36580/44000 | Loss: 2.9064 | LR: 3.54e-05 | 3514.5ms/step | 5.7 samples/s
174
+ Step 36590/44000 | Loss: 2.9007 | LR: 3.53e-05 | 3514.0ms/step | 5.7 samples/s
175
+ Step 36600/44000 | Loss: 2.8853 | LR: 3.53e-05 | 3518.1ms/step | 5.7 samples/s
176
+ Step 36610/44000 | Loss: 2.8836 | LR: 3.52e-05 | 3520.1ms/step | 5.7 samples/s
177
+ Step 36620/44000 | Loss: 2.8617 | LR: 3.51e-05 | 3515.4ms/step | 5.7 samples/s
178
+ Step 36630/44000 | Loss: 2.8704 | LR: 3.50e-05 | 3517.6ms/step | 5.7 samples/s
179
+ Step 36640/44000 | Loss: 2.9187 | LR: 3.49e-05 | 3515.0ms/step | 5.7 samples/s
180
+ Step 36650/44000 | Loss: 2.9197 | LR: 3.48e-05 | 3514.0ms/step | 5.7 samples/s
181
+ Step 36660/44000 | Loss: 2.8771 | LR: 3.47e-05 | 3524.5ms/step | 5.7 samples/s
182
+ Step 36670/44000 | Loss: 2.8324 | LR: 3.46e-05 | 3516.5ms/step | 5.7 samples/s
183
+ Step 36680/44000 | Loss: 2.8718 | LR: 3.45e-05 | 3514.8ms/step | 5.7 samples/s
184
+ Step 36690/44000 | Loss: 2.9026 | LR: 3.44e-05 | 3513.4ms/step | 5.7 samples/s
185
+ Step 36700/44000 | Loss: 2.8527 | LR: 3.43e-05 | 3513.2ms/step | 5.7 samples/s
186
+ Step 36710/44000 | Loss: 2.8785 | LR: 3.42e-05 | 3512.0ms/step | 5.7 samples/s
187
+ Step 36720/44000 | Loss: 2.8809 | LR: 3.41e-05 | 3515.0ms/step | 5.7 samples/s
188
+ Step 36730/44000 | Loss: 2.8692 | LR: 3.41e-05 | 3513.9ms/step | 5.7 samples/s
189
+ Step 36740/44000 | Loss: 2.8823 | LR: 3.40e-05 | 3512.1ms/step | 5.7 samples/s
190
+ Step 36750/44000 | Loss: 2.8920 | LR: 3.39e-05 | 3513.8ms/step | 5.7 samples/s
191
+ Validation at step 36750: Loss = 2.8998, Perplexity = 18.17
192
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-36750
193
+ Step 36760/44000 | Loss: 2.9347 | LR: 3.38e-05 | 9709.1ms/step | 2.1 samples/s
194
+ Step 36770/44000 | Loss: 2.9002 | LR: 3.37e-05 | 3507.7ms/step | 5.7 samples/s
195
+ Step 36780/44000 | Loss: 2.8817 | LR: 3.36e-05 | 3507.6ms/step | 5.7 samples/s
196
+ Step 36790/44000 | Loss: 2.8844 | LR: 3.35e-05 | 3511.7ms/step | 5.7 samples/s
197
+ Step 36800/44000 | Loss: 2.8989 | LR: 3.34e-05 | 3516.8ms/step | 5.7 samples/s
198
+ Step 36810/44000 | Loss: 2.8981 | LR: 3.33e-05 | 3513.4ms/step | 5.7 samples/s
199
+ Step 36820/44000 | Loss: 2.9274 | LR: 3.32e-05 | 3506.7ms/step | 5.7 samples/s
200
+ Step 36830/44000 | Loss: 2.8838 | LR: 3.31e-05 | 3507.4ms/step | 5.7 samples/s
201
+ Step 36840/44000 | Loss: 2.9015 | LR: 3.31e-05 | 3507.4ms/step | 5.7 samples/s
202
+ Step 36850/44000 | Loss: 2.8864 | LR: 3.30e-05 | 3508.3ms/step | 5.7 samples/s
203
+ Step 36860/44000 | Loss: 2.9090 | LR: 3.29e-05 | 3510.2ms/step | 5.7 samples/s
204
+ Step 36870/44000 | Loss: 2.9175 | LR: 3.28e-05 | 3509.1ms/step | 5.7 samples/s
205
+ Step 36880/44000 | Loss: 2.8744 | LR: 3.27e-05 | 3508.0ms/step | 5.7 samples/s
206
+ Step 36890/44000 | Loss: 2.8707 | LR: 3.26e-05 | 3515.2ms/step | 5.7 samples/s
207
+ Step 36900/44000 | Loss: 2.9064 | LR: 3.25e-05 | 3514.8ms/step | 5.7 samples/s
208
+ Step 36910/44000 | Loss: 2.8944 | LR: 3.24e-05 | 3509.3ms/step | 5.7 samples/s
209
+ Step 36920/44000 | Loss: 2.8823 | LR: 3.23e-05 | 3508.6ms/step | 5.7 samples/s
210
+ Step 36930/44000 | Loss: 2.8628 | LR: 3.22e-05 | 3506.8ms/step | 5.7 samples/s
211
+ Step 36940/44000 | Loss: 2.8683 | LR: 3.22e-05 | 3515.9ms/step | 5.7 samples/s
212
+ Step 36950/44000 | Loss: 2.8949 | LR: 3.21e-05 | 3516.5ms/step | 5.7 samples/s
213
+ Step 36960/44000 | Loss: 2.8511 | LR: 3.20e-05 | 3519.0ms/step | 5.7 samples/s
214
+ Step 36970/44000 | Loss: 2.8949 | LR: 3.19e-05 | 3515.8ms/step | 5.7 samples/s
215
+ Step 36980/44000 | Loss: 2.9067 | LR: 3.18e-05 | 3518.8ms/step | 5.7 samples/s
216
+ Step 36990/44000 | Loss: 2.8978 | LR: 3.17e-05 | 3516.7ms/step | 5.7 samples/s
217
+ Step 37000/44000 | Loss: 2.8805 | LR: 3.16e-05 | 3517.4ms/step | 5.7 samples/s
218
+ Validation at step 37000: Loss = 2.8996, Perplexity = 18.17
219
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-37000
220
+ Step 37010/44000 | Loss: 2.8837 | LR: 3.15e-05 | 9682.9ms/step | 2.1 samples/s
221
+ Step 37020/44000 | Loss: 2.8750 | LR: 3.15e-05 | 3509.0ms/step | 5.7 samples/s
222
+ Step 37030/44000 | Loss: 2.8754 | LR: 3.14e-05 | 3508.1ms/step | 5.7 samples/s
223
+ Step 37040/44000 | Loss: 2.8690 | LR: 3.13e-05 | 3510.9ms/step | 5.7 samples/s
224
+ Step 37050/44000 | Loss: 2.8803 | LR: 3.12e-05 | 3508.6ms/step | 5.7 samples/s
225
+ Step 37060/44000 | Loss: 2.8858 | LR: 3.11e-05 | 3506.8ms/step | 5.7 samples/s
226
+ Step 37070/44000 | Loss: 2.8788 | LR: 3.10e-05 | 3497.7ms/step | 5.7 samples/s
227
+ Step 37080/44000 | Loss: 2.9019 | LR: 3.09e-05 | 3498.7ms/step | 5.7 samples/s
228
+ Step 37090/44000 | Loss: 2.8782 | LR: 3.08e-05 | 3493.6ms/step | 5.7 samples/s
229
+ Step 37100/44000 | Loss: 2.9024 | LR: 3.07e-05 | 3500.3ms/step | 5.7 samples/s
230
+ Step 37110/44000 | Loss: 2.8711 | LR: 3.07e-05 | 3499.8ms/step | 5.7 samples/s
231
+ Step 37120/44000 | Loss: 2.9283 | LR: 3.06e-05 | 3488.4ms/step | 5.7 samples/s
232
+ Step 37130/44000 | Loss: 2.9042 | LR: 3.05e-05 | 3479.5ms/step | 5.7 samples/s
233
+ Step 37140/44000 | Loss: 2.8939 | LR: 3.04e-05 | 3473.7ms/step | 5.8 samples/s
234
+ Step 37150/44000 | Loss: 2.8653 | LR: 3.03e-05 | 3468.9ms/step | 5.8 samples/s
235
+ Step 37160/44000 | Loss: 2.8442 | LR: 3.02e-05 | 3467.3ms/step | 5.8 samples/s
236
+ Step 37170/44000 | Loss: 2.8904 | LR: 3.01e-05 | 3475.9ms/step | 5.8 samples/s
237
+ Step 37180/44000 | Loss: 2.9013 | LR: 3.01e-05 | 3472.8ms/step | 5.8 samples/s
238
+ Step 37190/44000 | Loss: 2.8860 | LR: 3.00e-05 | 3465.5ms/step | 5.8 samples/s
239
+ Step 37200/44000 | Loss: 2.8853 | LR: 2.99e-05 | 3474.5ms/step | 5.8 samples/s
240
+ Step 37210/44000 | Loss: 2.8992 | LR: 2.98e-05 | 3473.7ms/step | 5.8 samples/s
241
+ Step 37220/44000 | Loss: 2.9016 | LR: 2.97e-05 | 3475.5ms/step | 5.8 samples/s
242
+ Step 37230/44000 | Loss: 2.8903 | LR: 2.96e-05 | 3464.4ms/step | 5.8 samples/s
243
+ Step 37240/44000 | Loss: 2.8945 | LR: 2.95e-05 | 3462.4ms/step | 5.8 samples/s
244
+ Step 37250/44000 | Loss: 2.8775 | LR: 2.95e-05 | 3462.6ms/step | 5.8 samples/s
245
+ Validation at step 37250: Loss = 2.8994, Perplexity = 18.16
246
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-37250
247
+ Step 37260/44000 | Loss: 2.8869 | LR: 2.94e-05 | 9570.4ms/step | 2.1 samples/s
248
+ Step 37270/44000 | Loss: 2.8783 | LR: 2.93e-05 | 3463.5ms/step | 5.8 samples/s
249
+ Step 37280/44000 | Loss: 2.8756 | LR: 2.92e-05 | 3462.2ms/step | 5.8 samples/s
250
+ Step 37290/44000 | Loss: 2.8971 | LR: 2.91e-05 | 3461.7ms/step | 5.8 samples/s
251
+ Step 37300/44000 | Loss: 2.8840 | LR: 2.90e-05 | 3462.7ms/step | 5.8 samples/s
252
+ Step 37310/44000 | Loss: 2.8668 | LR: 2.89e-05 | 3468.9ms/step | 5.8 samples/s
253
+ Step 37320/44000 | Loss: 2.8819 | LR: 2.89e-05 | 3473.4ms/step | 5.8 samples/s
254
+ Step 37330/44000 | Loss: 2.8999 | LR: 2.88e-05 | 3473.9ms/step | 5.8 samples/s
255
+ Step 37340/44000 | Loss: 2.9245 | LR: 2.87e-05 | 3474.0ms/step | 5.8 samples/s
256
+ Step 37350/44000 | Loss: 2.9105 | LR: 2.86e-05 | 3473.4ms/step | 5.8 samples/s
257
+ Step 37360/44000 | Loss: 2.9031 | LR: 2.85e-05 | 3474.2ms/step | 5.8 samples/s
258
+ Step 37370/44000 | Loss: 2.8829 | LR: 2.84e-05 | 3474.5ms/step | 5.8 samples/s
259
+ Step 37380/44000 | Loss: 2.8680 | LR: 2.84e-05 | 3468.2ms/step | 5.8 samples/s
260
+ Step 37390/44000 | Loss: 2.8728 | LR: 2.83e-05 | 3462.6ms/step | 5.8 samples/s
261
+ Step 37400/44000 | Loss: 2.8553 | LR: 2.82e-05 | 3461.5ms/step | 5.8 samples/s
262
+ Step 37410/44000 | Loss: 2.8782 | LR: 2.81e-05 | 3462.2ms/step | 5.8 samples/s
263
+ Step 37420/44000 | Loss: 2.8975 | LR: 2.80e-05 | 3462.2ms/step | 5.8 samples/s
264
+ Step 37430/44000 | Loss: 2.9138 | LR: 2.79e-05 | 3461.7ms/step | 5.8 samples/s
265
+ Step 37440/44000 | Loss: 2.8969 | LR: 2.78e-05 | 3460.8ms/step | 5.8 samples/s
266
+ Step 37450/44000 | Loss: 2.8973 | LR: 2.78e-05 | 3461.8ms/step | 5.8 samples/s
267
+ Step 37460/44000 | Loss: 2.9313 | LR: 2.77e-05 | 3461.6ms/step | 5.8 samples/s
268
+ Step 37470/44000 | Loss: 2.9020 | LR: 2.76e-05 | 3461.5ms/step | 5.8 samples/s
269
+ Step 37480/44000 | Loss: 2.8828 | LR: 2.75e-05 | 3462.0ms/step | 5.8 samples/s
270
+ Step 37490/44000 | Loss: 2.8999 | LR: 2.74e-05 | 3462.3ms/step | 5.8 samples/s
271
+ Step 37500/44000 | Loss: 2.8842 | LR: 2.74e-05 | 3460.9ms/step | 5.8 samples/s
272
+ Validation at step 37500: Loss = 2.8993, Perplexity = 18.16
273
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-37500
274
+ Step 37510/44000 | Loss: 2.9013 | LR: 2.73e-05 | 9565.1ms/step | 2.1 samples/s
275
+ Step 37520/44000 | Loss: 2.9066 | LR: 2.72e-05 | 3463.2ms/step | 5.8 samples/s
276
+ Step 37530/44000 | Loss: 2.9084 | LR: 2.71e-05 | 3462.7ms/step | 5.8 samples/s
277
+ Step 37540/44000 | Loss: 2.8755 | LR: 2.70e-05 | 3461.5ms/step | 5.8 samples/s
278
+ Step 37550/44000 | Loss: 2.8649 | LR: 2.69e-05 | 3459.5ms/step | 5.8 samples/s
279
+ Step 37560/44000 | Loss: 2.8431 | LR: 2.69e-05 | 3461.7ms/step | 5.8 samples/s
280
+ Step 37570/44000 | Loss: 2.8836 | LR: 2.68e-05 | 3460.6ms/step | 5.8 samples/s
281
+ Step 37580/44000 | Loss: 2.9189 | LR: 2.67e-05 | 3459.5ms/step | 5.8 samples/s
282
+ Step 37590/44000 | Loss: 2.8901 | LR: 2.66e-05 | 3460.3ms/step | 5.8 samples/s
283
+ Step 37600/44000 | Loss: 2.8730 | LR: 2.65e-05 | 3459.4ms/step | 5.8 samples/s
284
+ Step 37610/44000 | Loss: 2.8989 | LR: 2.65e-05 | 3461.8ms/step | 5.8 samples/s
285
+ Step 37620/44000 | Loss: 2.8758 | LR: 2.64e-05 | 3458.3ms/step | 5.8 samples/s
286
+ Step 37630/44000 | Loss: 2.8969 | LR: 2.63e-05 | 3459.7ms/step | 5.8 samples/s
287
+ Step 37640/44000 | Loss: 2.8818 | LR: 2.62e-05 | 3461.1ms/step | 5.8 samples/s
288
+ Step 37650/44000 | Loss: 2.8894 | LR: 2.61e-05 | 3460.9ms/step | 5.8 samples/s
289
+ Step 37660/44000 | Loss: 2.8945 | LR: 2.60e-05 | 3461.1ms/step | 5.8 samples/s
290
+ Step 37670/44000 | Loss: 2.8979 | LR: 2.60e-05 | 3461.2ms/step | 5.8 samples/s
291
+ Step 37680/44000 | Loss: 2.9002 | LR: 2.59e-05 | 3458.6ms/step | 5.8 samples/s
292
+ Step 37690/44000 | Loss: 2.8939 | LR: 2.58e-05 | 3459.5ms/step | 5.8 samples/s
293
+ Step 37700/44000 | Loss: 2.9079 | LR: 2.57e-05 | 3457.9ms/step | 5.8 samples/s
294
+ Step 37710/44000 | Loss: 2.8983 | LR: 2.56e-05 | 3458.5ms/step | 5.8 samples/s
295
+ Step 37720/44000 | Loss: 2.9034 | LR: 2.56e-05 | 3456.6ms/step | 5.8 samples/s
296
+ Step 37730/44000 | Loss: 2.8871 | LR: 2.55e-05 | 3462.9ms/step | 5.8 samples/s
297
+ Step 37740/44000 | Loss: 2.9071 | LR: 2.54e-05 | 3461.1ms/step | 5.8 samples/s
298
+ Step 37750/44000 | Loss: 2.8742 | LR: 2.53e-05 | 3464.6ms/step | 5.8 samples/s
299
+ Validation at step 37750: Loss = 2.8992, Perplexity = 18.16
300
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-37750
301
+ Step 37760/44000 | Loss: 2.9023 | LR: 2.52e-05 | 9588.4ms/step | 2.1 samples/s
302
+ Step 37770/44000 | Loss: 2.8634 | LR: 2.52e-05 | 3473.3ms/step | 5.8 samples/s
303
+ Step 37780/44000 | Loss: 2.8739 | LR: 2.51e-05 | 3471.6ms/step | 5.8 samples/s
304
+ Step 37790/44000 | Loss: 2.8957 | LR: 2.50e-05 | 3472.5ms/step | 5.8 samples/s
305
+ Step 37800/44000 | Loss: 2.8815 | LR: 2.49e-05 | 3472.6ms/step | 5.8 samples/s
306
+ Step 37810/44000 | Loss: 2.9123 | LR: 2.48e-05 | 3471.5ms/step | 5.8 samples/s
307
+ Step 37820/44000 | Loss: 2.8950 | LR: 2.48e-05 | 3468.7ms/step | 5.8 samples/s
308
+ Step 37830/44000 | Loss: 2.8695 | LR: 2.47e-05 | 3469.6ms/step | 5.8 samples/s
309
+ Step 37840/44000 | Loss: 2.8941 | LR: 2.46e-05 | 3469.3ms/step | 5.8 samples/s
310
+ Step 37850/44000 | Loss: 2.9182 | LR: 2.45e-05 | 3469.6ms/step | 5.8 samples/s
311
+ Step 37860/44000 | Loss: 2.9055 | LR: 2.45e-05 | 3468.7ms/step | 5.8 samples/s
312
+ Step 37870/44000 | Loss: 2.8856 | LR: 2.44e-05 | 3471.3ms/step | 5.8 samples/s
313
+ Step 37880/44000 | Loss: 2.8614 | LR: 2.43e-05 | 3470.4ms/step | 5.8 samples/s
314
+ Step 37890/44000 | Loss: 2.8849 | LR: 2.42e-05 | 3469.6ms/step | 5.8 samples/s
315
+ Step 37900/44000 | Loss: 2.9087 | LR: 2.41e-05 | 3472.4ms/step | 5.8 samples/s
316
+ Step 37910/44000 | Loss: 2.9019 | LR: 2.41e-05 | 3467.4ms/step | 5.8 samples/s
317
+ Step 37920/44000 | Loss: 2.8984 | LR: 2.40e-05 | 3470.3ms/step | 5.8 samples/s
318
+ Step 37930/44000 | Loss: 2.8985 | LR: 2.39e-05 | 3471.4ms/step | 5.8 samples/s
319
+ Step 37940/44000 | Loss: 2.9026 | LR: 2.38e-05 | 3471.3ms/step | 5.8 samples/s
320
+ Step 37950/44000 | Loss: 2.8956 | LR: 2.38e-05 | 3470.4ms/step | 5.8 samples/s
321
+ Step 37960/44000 | Loss: 2.8954 | LR: 2.37e-05 | 3478.3ms/step | 5.8 samples/s
322
+ Step 37970/44000 | Loss: 2.8870 | LR: 2.36e-05 | 3473.2ms/step | 5.8 samples/s
323
+ Step 37980/44000 | Loss: 2.8982 | LR: 2.35e-05 | 3472.7ms/step | 5.8 samples/s
324
+ Step 37990/44000 | Loss: 2.9033 | LR: 2.34e-05 | 3471.9ms/step | 5.8 samples/s
325
+ Step 38000/44000 | Loss: 2.8989 | LR: 2.34e-05 | 3473.1ms/step | 5.8 samples/s
326
+ Validation at step 38000: Loss = 2.8991, Perplexity = 18.16
327
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-38000
328
+ Step 38010/44000 | Loss: 2.9021 | LR: 2.33e-05 | 9601.1ms/step | 2.1 samples/s
329
+ Step 38020/44000 | Loss: 2.8712 | LR: 2.32e-05 | 3470.9ms/step | 5.8 samples/s
330
+ Step 38030/44000 | Loss: 2.8826 | LR: 2.31e-05 | 3474.6ms/step | 5.8 samples/s
331
+ Step 38040/44000 | Loss: 2.8727 | LR: 2.31e-05 | 3472.8ms/step | 5.8 samples/s
332
+ Step 38050/44000 | Loss: 2.8783 | LR: 2.30e-05 | 3471.7ms/step | 5.8 samples/s
333
+ Step 38060/44000 | Loss: 2.9043 | LR: 2.29e-05 | 3477.1ms/step | 5.8 samples/s
334
+ Step 38070/44000 | Loss: 2.8741 | LR: 2.28e-05 | 3473.7ms/step | 5.8 samples/s
335
+ Step 38080/44000 | Loss: 2.9169 | LR: 2.28e-05 | 3469.3ms/step | 5.8 samples/s
336
+ Step 38090/44000 | Loss: 2.8879 | LR: 2.27e-05 | 3463.1ms/step | 5.8 samples/s
337
+ Step 38100/44000 | Loss: 2.8853 | LR: 2.26e-05 | 3462.1ms/step | 5.8 samples/s
338
+ Step 38110/44000 | Loss: 2.9064 | LR: 2.25e-05 | 3462.8ms/step | 5.8 samples/s
339
+ Step 38120/44000 | Loss: 2.8702 | LR: 2.25e-05 | 3462.8ms/step | 5.8 samples/s
340
+ Step 38130/44000 | Loss: 2.9048 | LR: 2.24e-05 | 3461.4ms/step | 5.8 samples/s
341
+ Step 38140/44000 | Loss: 2.9010 | LR: 2.23e-05 | 3460.1ms/step | 5.8 samples/s
342
+ Step 38150/44000 | Loss: 2.8705 | LR: 2.22e-05 | 3465.2ms/step | 5.8 samples/s
343
+ Step 38160/44000 | Loss: 2.8900 | LR: 2.22e-05 | 3461.3ms/step | 5.8 samples/s
344
+ Step 38170/44000 | Loss: 2.9283 | LR: 2.21e-05 | 3461.2ms/step | 5.8 samples/s
345
+ Step 38180/44000 | Loss: 2.8787 | LR: 2.20e-05 | 3468.1ms/step | 5.8 samples/s
346
+ Step 38190/44000 | Loss: 2.8347 | LR: 2.19e-05 | 3463.6ms/step | 5.8 samples/s
347
+ Step 38200/44000 | Loss: 2.9010 | LR: 2.19e-05 | 3463.5ms/step | 5.8 samples/s
348
+ Step 38210/44000 | Loss: 2.9095 | LR: 2.18e-05 | 3463.1ms/step | 5.8 samples/s
349
+ Step 38220/44000 | Loss: 2.9093 | LR: 2.17e-05 | 3461.9ms/step | 5.8 samples/s
350
+ Step 38230/44000 | Loss: 2.8864 | LR: 2.16e-05 | 3461.0ms/step | 5.8 samples/s
351
+ Step 38240/44000 | Loss: 2.8925 | LR: 2.16e-05 | 3463.9ms/step | 5.8 samples/s
352
+ Step 38250/44000 | Loss: 2.8795 | LR: 2.15e-05 | 3465.6ms/step | 5.8 samples/s
353
+ Validation at step 38250: Loss = 2.8990, Perplexity = 18.16
354
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-38250
355
+ Step 38260/44000 | Loss: 2.9025 | LR: 2.14e-05 | 9567.9ms/step | 2.1 samples/s
356
+ Step 38270/44000 | Loss: 2.8887 | LR: 2.13e-05 | 3460.7ms/step | 5.8 samples/s
357
+ Step 38280/44000 | Loss: 2.8888 | LR: 2.13e-05 | 3461.4ms/step | 5.8 samples/s
358
+ Step 38290/44000 | Loss: 2.8578 | LR: 2.12e-05 | 3461.7ms/step | 5.8 samples/s
359
+ Step 38300/44000 | Loss: 2.8554 | LR: 2.11e-05 | 3460.5ms/step | 5.8 samples/s
360
+ Step 38310/44000 | Loss: 2.9192 | LR: 2.11e-05 | 3459.2ms/step | 5.8 samples/s
361
+ Step 38320/44000 | Loss: 2.8907 | LR: 2.10e-05 | 3460.3ms/step | 5.8 samples/s
362
+ Step 38330/44000 | Loss: 2.8704 | LR: 2.09e-05 | 3461.1ms/step | 5.8 samples/s
363
+ Step 38340/44000 | Loss: 2.8806 | LR: 2.08e-05 | 3462.4ms/step | 5.8 samples/s
364
+ Step 38350/44000 | Loss: 2.9010 | LR: 2.08e-05 | 3460.5ms/step | 5.8 samples/s
365
+ Step 38360/44000 | Loss: 2.8740 | LR: 2.07e-05 | 3461.9ms/step | 5.8 samples/s
366
+ Step 38370/44000 | Loss: 2.9193 | LR: 2.06e-05 | 3462.8ms/step | 5.8 samples/s
367
+ Step 38380/44000 | Loss: 2.8926 | LR: 2.05e-05 | 3459.6ms/step | 5.8 samples/s
368
+ Step 38390/44000 | Loss: 2.8987 | LR: 2.05e-05 | 3461.3ms/step | 5.8 samples/s
369
+ Step 38400/44000 | Loss: 2.9150 | LR: 2.04e-05 | 3460.6ms/step | 5.8 samples/s
370
+ Step 38410/44000 | Loss: 2.8852 | LR: 2.03e-05 | 3460.9ms/step | 5.8 samples/s
371
+ Step 38420/44000 | Loss: 2.8957 | LR: 2.03e-05 | 3458.9ms/step | 5.8 samples/s
372
+ Step 38430/44000 | Loss: 2.8781 | LR: 2.02e-05 | 3461.6ms/step | 5.8 samples/s
373
+ Step 38440/44000 | Loss: 2.9074 | LR: 2.01e-05 | 3463.6ms/step | 5.8 samples/s
374
+ Step 38450/44000 | Loss: 2.8705 | LR: 2.00e-05 | 3463.4ms/step | 5.8 samples/s
375
+ Step 38460/44000 | Loss: 2.8935 | LR: 2.00e-05 | 3459.4ms/step | 5.8 samples/s
376
+ Step 38470/44000 | Loss: 2.8670 | LR: 1.99e-05 | 3462.2ms/step | 5.8 samples/s
377
+ Step 38480/44000 | Loss: 2.8535 | LR: 1.98e-05 | 3460.8ms/step | 5.8 samples/s
378
+ Step 38490/44000 | Loss: 2.9053 | LR: 1.98e-05 | 3459.0ms/step | 5.8 samples/s
379
+ Step 38500/44000 | Loss: 2.8983 | LR: 1.97e-05 | 3460.5ms/step | 5.8 samples/s
380
+ Validation at step 38500: Loss = 2.8990, Perplexity = 18.16
381
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-38500
382
+ Step 38510/44000 | Loss: 2.8881 | LR: 1.96e-05 | 9549.6ms/step | 2.1 samples/s
383
+ Step 38520/44000 | Loss: 2.8734 | LR: 1.95e-05 | 3458.4ms/step | 5.8 samples/s
384
+ Step 38530/44000 | Loss: 2.8684 | LR: 1.95e-05 | 3460.7ms/step | 5.8 samples/s
385
+ Step 38540/44000 | Loss: 2.8959 | LR: 1.94e-05 | 3460.2ms/step | 5.8 samples/s
386
+ Step 38550/44000 | Loss: 2.9073 | LR: 1.93e-05 | 3461.0ms/step | 5.8 samples/s
387
+ Step 38560/44000 | Loss: 2.8912 | LR: 1.93e-05 | 3460.5ms/step | 5.8 samples/s
388
+ Step 38570/44000 | Loss: 2.8995 | LR: 1.92e-05 | 3459.8ms/step | 5.8 samples/s
389
+ Step 38580/44000 | Loss: 2.8690 | LR: 1.91e-05 | 3460.4ms/step | 5.8 samples/s
390
+ Step 38590/44000 | Loss: 2.8824 | LR: 1.91e-05 | 3460.2ms/step | 5.8 samples/s
391
+ Step 38600/44000 | Loss: 2.8693 | LR: 1.90e-05 | 3458.6ms/step | 5.8 samples/s
392
+ Step 38610/44000 | Loss: 2.8964 | LR: 1.89e-05 | 3458.2ms/step | 5.8 samples/s
393
+ Step 38620/44000 | Loss: 2.8910 | LR: 1.88e-05 | 3458.3ms/step | 5.8 samples/s
394
+ Step 38630/44000 | Loss: 2.8795 | LR: 1.88e-05 | 3459.4ms/step | 5.8 samples/s
395
+ Step 38640/44000 | Loss: 2.8724 | LR: 1.87e-05 | 3459.7ms/step | 5.8 samples/s
396
+ Step 38650/44000 | Loss: 2.8922 | LR: 1.86e-05 | 3460.0ms/step | 5.8 samples/s
397
+ Step 38660/44000 | Loss: 2.8998 | LR: 1.86e-05 | 3461.7ms/step | 5.8 samples/s
398
+ Step 38670/44000 | Loss: 2.8816 | LR: 1.85e-05 | 3461.4ms/step | 5.8 samples/s
399
+ Step 38680/44000 | Loss: 2.8742 | LR: 1.84e-05 | 3458.6ms/step | 5.8 samples/s
400
+ Step 38690/44000 | Loss: 2.8700 | LR: 1.84e-05 | 3461.1ms/step | 5.8 samples/s
401
+ Step 38700/44000 | Loss: 2.9284 | LR: 1.83e-05 | 3457.8ms/step | 5.8 samples/s
402
+ Step 38710/44000 | Loss: 2.9048 | LR: 1.82e-05 | 3460.0ms/step | 5.8 samples/s
403
+ Step 38720/44000 | Loss: 2.8801 | LR: 1.82e-05 | 3461.7ms/step | 5.8 samples/s
404
+ Step 38730/44000 | Loss: 2.9012 | LR: 1.81e-05 | 3460.2ms/step | 5.8 samples/s
405
+ Step 38740/44000 | Loss: 2.8632 | LR: 1.80e-05 | 3458.9ms/step | 5.8 samples/s
406
+ Step 38750/44000 | Loss: 2.8687 | LR: 1.80e-05 | 3460.2ms/step | 5.8 samples/s
407
+ Validation at step 38750: Loss = 2.8989, Perplexity = 18.15
408
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-38750
409
+ Step 38760/44000 | Loss: 2.8658 | LR: 1.79e-05 | 9559.5ms/step | 2.1 samples/s
410
+ Step 38770/44000 | Loss: 2.8805 | LR: 1.78e-05 | 3458.6ms/step | 5.8 samples/s
411
+ Step 38780/44000 | Loss: 2.9150 | LR: 1.78e-05 | 3456.7ms/step | 5.8 samples/s
412
+ Step 38790/44000 | Loss: 2.9033 | LR: 1.77e-05 | 3458.1ms/step | 5.8 samples/s
413
+ Step 38800/44000 | Loss: 2.8983 | LR: 1.76e-05 | 3461.8ms/step | 5.8 samples/s
414
+ Step 38810/44000 | Loss: 2.9083 | LR: 1.76e-05 | 3460.4ms/step | 5.8 samples/s
415
+ Step 38820/44000 | Loss: 2.9089 | LR: 1.75e-05 | 3459.2ms/step | 5.8 samples/s
416
+ Step 38830/44000 | Loss: 2.9040 | LR: 1.74e-05 | 3459.3ms/step | 5.8 samples/s
417
+ Step 38840/44000 | Loss: 2.8862 | LR: 1.74e-05 | 3460.0ms/step | 5.8 samples/s
418
+ Step 38850/44000 | Loss: 2.9006 | LR: 1.73e-05 | 3460.3ms/step | 5.8 samples/s
419
+ Step 38860/44000 | Loss: 2.9051 | LR: 1.72e-05 | 3457.8ms/step | 5.8 samples/s
420
+ Step 38870/44000 | Loss: 2.8719 | LR: 1.72e-05 | 3458.2ms/step | 5.8 samples/s
421
+ Step 38880/44000 | Loss: 2.8764 | LR: 1.71e-05 | 3460.5ms/step | 5.8 samples/s
422
+ Step 38890/44000 | Loss: 2.8818 | LR: 1.70e-05 | 3458.6ms/step | 5.8 samples/s
423
+ Step 38900/44000 | Loss: 2.8953 | LR: 1.70e-05 | 3459.7ms/step | 5.8 samples/s
424
+ Step 38910/44000 | Loss: 2.8633 | LR: 1.69e-05 | 3461.5ms/step | 5.8 samples/s
425
+ Step 38920/44000 | Loss: 2.8926 | LR: 1.68e-05 | 3460.1ms/step | 5.8 samples/s
426
+ Step 38930/44000 | Loss: 2.8825 | LR: 1.68e-05 | 3459.4ms/step | 5.8 samples/s
427
+ Step 38940/44000 | Loss: 2.9089 | LR: 1.67e-05 | 3457.8ms/step | 5.8 samples/s
428
+ Step 38950/44000 | Loss: 2.9092 | LR: 1.66e-05 | 3455.7ms/step | 5.8 samples/s
429
+ Step 38960/44000 | Loss: 2.9055 | LR: 1.66e-05 | 3457.0ms/step | 5.8 samples/s
430
+ Step 38970/44000 | Loss: 2.8640 | LR: 1.65e-05 | 3457.6ms/step | 5.8 samples/s
431
+ Step 38980/44000 | Loss: 2.8963 | LR: 1.64e-05 | 3459.1ms/step | 5.8 samples/s
432
+ Step 38990/44000 | Loss: 2.9025 | LR: 1.64e-05 | 3455.1ms/step | 5.8 samples/s
433
+ Step 39000/44000 | Loss: 2.9163 | LR: 1.63e-05 | 3457.0ms/step | 5.8 samples/s
434
+ Validation at step 39000: Loss = 2.8989, Perplexity = 18.15
435
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-39000
436
+ Step 39010/44000 | Loss: 2.8966 | LR: 1.62e-05 | 9543.6ms/step | 2.1 samples/s
437
+ Step 39020/44000 | Loss: 2.8714 | LR: 1.62e-05 | 3457.9ms/step | 5.8 samples/s
438
+ Step 39030/44000 | Loss: 2.8912 | LR: 1.61e-05 | 3456.2ms/step | 5.8 samples/s
439
+ Step 39040/44000 | Loss: 2.8958 | LR: 1.61e-05 | 3458.1ms/step | 5.8 samples/s
440
+ Step 39050/44000 | Loss: 2.8774 | LR: 1.60e-05 | 3459.7ms/step | 5.8 samples/s
441
+ Step 39060/44000 | Loss: 2.9017 | LR: 1.59e-05 | 3461.0ms/step | 5.8 samples/s
442
+ Step 39070/44000 | Loss: 2.8736 | LR: 1.59e-05 | 3458.5ms/step | 5.8 samples/s
443
+ Step 39080/44000 | Loss: 2.8858 | LR: 1.58e-05 | 3457.1ms/step | 5.8 samples/s
444
+ Step 39090/44000 | Loss: 2.9036 | LR: 1.57e-05 | 3457.0ms/step | 5.8 samples/s
445
+ Step 39100/44000 | Loss: 2.9069 | LR: 1.57e-05 | 3458.5ms/step | 5.8 samples/s
446
+ Step 39110/44000 | Loss: 2.8791 | LR: 1.56e-05 | 3458.1ms/step | 5.8 samples/s
447
+ Step 39120/44000 | Loss: 2.8614 | LR: 1.55e-05 | 3458.8ms/step | 5.8 samples/s
448
+ Step 39130/44000 | Loss: 2.9050 | LR: 1.55e-05 | 3459.9ms/step | 5.8 samples/s
449
+ Step 39140/44000 | Loss: 2.9094 | LR: 1.54e-05 | 3460.8ms/step | 5.8 samples/s
450
+ Step 39150/44000 | Loss: 2.9070 | LR: 1.54e-05 | 3459.4ms/step | 5.8 samples/s
451
+ Step 39160/44000 | Loss: 2.8893 | LR: 1.53e-05 | 3458.0ms/step | 5.8 samples/s
452
+ Step 39170/44000 | Loss: 2.9244 | LR: 1.52e-05 | 3459.7ms/step | 5.8 samples/s
453
+ Step 39180/44000 | Loss: 2.8795 | LR: 1.52e-05 | 3457.5ms/step | 5.8 samples/s
454
+ Step 39190/44000 | Loss: 2.8938 | LR: 1.51e-05 | 3457.6ms/step | 5.8 samples/s
455
+ Step 39200/44000 | Loss: 2.8676 | LR: 1.50e-05 | 3458.6ms/step | 5.8 samples/s
456
+ Step 39210/44000 | Loss: 2.9008 | LR: 1.50e-05 | 3459.8ms/step | 5.8 samples/s
457
+ Step 39220/44000 | Loss: 2.9033 | LR: 1.49e-05 | 3458.4ms/step | 5.8 samples/s
458
+ Step 39230/44000 | Loss: 2.9157 | LR: 1.49e-05 | 3458.2ms/step | 5.8 samples/s
459
+ Step 39240/44000 | Loss: 2.8506 | LR: 1.48e-05 | 3458.2ms/step | 5.8 samples/s
460
+ Step 39250/44000 | Loss: 2.9000 | LR: 1.47e-05 | 3459.1ms/step | 5.8 samples/s
461
+ Validation at step 39250: Loss = 2.8988, Perplexity = 18.15
462
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-39250
463
+ Step 39260/44000 | Loss: 2.9065 | LR: 1.47e-05 | 9559.0ms/step | 2.1 samples/s
464
+ Step 39270/44000 | Loss: 2.9015 | LR: 1.46e-05 | 3458.5ms/step | 5.8 samples/s
465
+ Step 39280/44000 | Loss: 2.9071 | LR: 1.46e-05 | 3457.3ms/step | 5.8 samples/s
466
+ Step 39290/44000 | Loss: 2.8869 | LR: 1.45e-05 | 3458.8ms/step | 5.8 samples/s
467
+ Step 39300/44000 | Loss: 2.9214 | LR: 1.44e-05 | 3460.2ms/step | 5.8 samples/s
468
+ Step 39310/44000 | Loss: 2.9185 | LR: 1.44e-05 | 3458.7ms/step | 5.8 samples/s
469
+ Step 39320/44000 | Loss: 2.8916 | LR: 1.43e-05 | 3459.9ms/step | 5.8 samples/s
470
+ Step 39330/44000 | Loss: 2.8946 | LR: 1.42e-05 | 3460.3ms/step | 5.8 samples/s
471
+ Step 39340/44000 | Loss: 2.9199 | LR: 1.42e-05 | 3461.3ms/step | 5.8 samples/s
472
+ Step 39350/44000 | Loss: 2.8791 | LR: 1.41e-05 | 3459.2ms/step | 5.8 samples/s
473
+ Step 39360/44000 | Loss: 2.8706 | LR: 1.41e-05 | 3458.9ms/step | 5.8 samples/s
474
+ Step 39370/44000 | Loss: 2.8651 | LR: 1.40e-05 | 3459.9ms/step | 5.8 samples/s
475
+ Step 39380/44000 | Loss: 2.8745 | LR: 1.39e-05 | 3458.4ms/step | 5.8 samples/s
476
+ Step 39390/44000 | Loss: 2.8722 | LR: 1.39e-05 | 3458.5ms/step | 5.8 samples/s
477
+ Step 39400/44000 | Loss: 2.8706 | LR: 1.38e-05 | 3458.5ms/step | 5.8 samples/s
478
+ Step 39410/44000 | Loss: 2.9043 | LR: 1.38e-05 | 3460.1ms/step | 5.8 samples/s
479
+ Step 39420/44000 | Loss: 2.8841 | LR: 1.37e-05 | 3461.2ms/step | 5.8 samples/s
480
+ Step 39430/44000 | Loss: 2.8953 | LR: 1.36e-05 | 3457.9ms/step | 5.8 samples/s
481
+ Step 39440/44000 | Loss: 2.9195 | LR: 1.36e-05 | 3459.6ms/step | 5.8 samples/s
482
+ Step 39450/44000 | Loss: 2.8835 | LR: 1.35e-05 | 3457.8ms/step | 5.8 samples/s
483
+ Step 39460/44000 | Loss: 2.8923 | LR: 1.35e-05 | 3459.2ms/step | 5.8 samples/s
484
+ Step 39470/44000 | Loss: 2.9240 | LR: 1.34e-05 | 3458.2ms/step | 5.8 samples/s
485
+ Step 39480/44000 | Loss: 2.9185 | LR: 1.34e-05 | 3456.3ms/step | 5.8 samples/s
486
+ Step 39490/44000 | Loss: 2.8952 | LR: 1.33e-05 | 3458.0ms/step | 5.8 samples/s
487
+ Step 39500/44000 | Loss: 2.8724 | LR: 1.32e-05 | 3460.6ms/step | 5.8 samples/s
488
+ Validation at step 39500: Loss = 2.8988, Perplexity = 18.15
489
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-39500
490
+ Step 39510/44000 | Loss: 2.8880 | LR: 1.32e-05 | 9531.8ms/step | 2.1 samples/s
491
+ Step 39520/44000 | Loss: 2.9073 | LR: 1.31e-05 | 3458.8ms/step | 5.8 samples/s
492
+ Step 39530/44000 | Loss: 2.9076 | LR: 1.31e-05 | 3459.5ms/step | 5.8 samples/s
493
+ Step 39540/44000 | Loss: 2.9104 | LR: 1.30e-05 | 3458.4ms/step | 5.8 samples/s
494
+ Step 39550/44000 | Loss: 2.9055 | LR: 1.29e-05 | 3462.3ms/step | 5.8 samples/s
495
+ Step 39560/44000 | Loss: 2.8836 | LR: 1.29e-05 | 3459.4ms/step | 5.8 samples/s
496
+ Step 39570/44000 | Loss: 2.8824 | LR: 1.28e-05 | 3460.0ms/step | 5.8 samples/s
497
+ Step 39580/44000 | Loss: 2.8999 | LR: 1.28e-05 | 3460.7ms/step | 5.8 samples/s
498
+ Step 39590/44000 | Loss: 2.8659 | LR: 1.27e-05 | 3459.5ms/step | 5.8 samples/s
499
+ Step 39600/44000 | Loss: 2.9466 | LR: 1.27e-05 | 3458.9ms/step | 5.8 samples/s
500
+ Step 39610/44000 | Loss: 2.9294 | LR: 1.26e-05 | 3459.5ms/step | 5.8 samples/s
501
+ Step 39620/44000 | Loss: 2.8681 | LR: 1.25e-05 | 3459.9ms/step | 5.8 samples/s
502
+ Step 39630/44000 | Loss: 2.8886 | LR: 1.25e-05 | 3459.5ms/step | 5.8 samples/s
503
+ Step 39640/44000 | Loss: 2.8845 | LR: 1.24e-05 | 3459.4ms/step | 5.8 samples/s
504
+ Step 39650/44000 | Loss: 2.9143 | LR: 1.24e-05 | 3458.0ms/step | 5.8 samples/s
505
+ Step 39660/44000 | Loss: 2.9112 | LR: 1.23e-05 | 3458.5ms/step | 5.8 samples/s
506
+ Step 39670/44000 | Loss: 2.8622 | LR: 1.23e-05 | 3458.7ms/step | 5.8 samples/s
507
+ Step 39680/44000 | Loss: 2.8938 | LR: 1.22e-05 | 3458.9ms/step | 5.8 samples/s
508
+ Step 39690/44000 | Loss: 2.9033 | LR: 1.22e-05 | 3459.5ms/step | 5.8 samples/s
509
+ Step 39700/44000 | Loss: 2.8710 | LR: 1.21e-05 | 3457.3ms/step | 5.8 samples/s
510
+ Step 39710/44000 | Loss: 2.8919 | LR: 1.20e-05 | 3457.8ms/step | 5.8 samples/s
511
+ Step 39720/44000 | Loss: 2.9138 | LR: 1.20e-05 | 3455.9ms/step | 5.8 samples/s
512
+ Step 39730/44000 | Loss: 2.8848 | LR: 1.19e-05 | 3458.0ms/step | 5.8 samples/s
513
+ Step 39740/44000 | Loss: 2.8765 | LR: 1.19e-05 | 3459.7ms/step | 5.8 samples/s
514
+ Step 39750/44000 | Loss: 2.8965 | LR: 1.18e-05 | 3458.4ms/step | 5.8 samples/s
515
+ Validation at step 39750: Loss = 2.8988, Perplexity = 18.15
516
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-39750
517
+ Step 39760/44000 | Loss: 2.9203 | LR: 1.18e-05 | 9537.4ms/step | 2.1 samples/s
518
+ Step 39770/44000 | Loss: 2.9164 | LR: 1.17e-05 | 3460.0ms/step | 5.8 samples/s
519
+ Step 39780/44000 | Loss: 2.8856 | LR: 1.17e-05 | 3461.7ms/step | 5.8 samples/s
520
+ Step 39790/44000 | Loss: 2.8721 | LR: 1.16e-05 | 3459.4ms/step | 5.8 samples/s
521
+ Step 39800/44000 | Loss: 2.9310 | LR: 1.15e-05 | 3458.5ms/step | 5.8 samples/s
522
+ Step 39810/44000 | Loss: 2.9101 | LR: 1.15e-05 | 3458.9ms/step | 5.8 samples/s
523
+ Step 39820/44000 | Loss: 2.9140 | LR: 1.14e-05 | 3457.9ms/step | 5.8 samples/s
524
+ Step 39830/44000 | Loss: 2.9086 | LR: 1.14e-05 | 3458.2ms/step | 5.8 samples/s
525
+ Step 39840/44000 | Loss: 2.8863 | LR: 1.13e-05 | 3458.0ms/step | 5.8 samples/s
526
+ Step 39850/44000 | Loss: 2.8753 | LR: 1.13e-05 | 3457.7ms/step | 5.8 samples/s
527
+ Step 39860/44000 | Loss: 2.8778 | LR: 1.12e-05 | 3455.8ms/step | 5.8 samples/s
528
+ Step 39870/44000 | Loss: 2.8928 | LR: 1.12e-05 | 3459.0ms/step | 5.8 samples/s
529
+ Step 39880/44000 | Loss: 2.8929 | LR: 1.11e-05 | 3462.0ms/step | 5.8 samples/s
530
+ Step 39890/44000 | Loss: 2.8998 | LR: 1.11e-05 | 3458.8ms/step | 5.8 samples/s
531
+ Step 39900/44000 | Loss: 2.9145 | LR: 1.10e-05 | 3456.9ms/step | 5.8 samples/s
532
+ Step 39910/44000 | Loss: 2.8755 | LR: 1.10e-05 | 3458.3ms/step | 5.8 samples/s
533
+ Step 39920/44000 | Loss: 2.8429 | LR: 1.09e-05 | 3458.0ms/step | 5.8 samples/s
534
+ Step 39930/44000 | Loss: 2.8632 | LR: 1.08e-05 | 3457.1ms/step | 5.8 samples/s
535
+ Step 39940/44000 | Loss: 2.8803 | LR: 1.08e-05 | 3456.4ms/step | 5.8 samples/s
536
+ Step 39950/44000 | Loss: 2.8757 | LR: 1.07e-05 | 3457.3ms/step | 5.8 samples/s
537
+ Step 39960/44000 | Loss: 2.9183 | LR: 1.07e-05 | 3456.9ms/step | 5.8 samples/s
538
+ Step 39970/44000 | Loss: 2.8921 | LR: 1.06e-05 | 3462.7ms/step | 5.8 samples/s
539
+ Step 39980/44000 | Loss: 2.8827 | LR: 1.06e-05 | 3462.6ms/step | 5.8 samples/s
540
+ Step 39990/44000 | Loss: 2.9184 | LR: 1.05e-05 | 3462.3ms/step | 5.8 samples/s
541
+ Step 40000/44000 | Loss: 2.8813 | LR: 1.05e-05 | 3461.2ms/step | 5.8 samples/s
542
+ Validation at step 40000: Loss = 2.8987, Perplexity = 18.15
543
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-40000
544
+ Step 40010/44000 | Loss: 2.8861 | LR: 1.04e-05 | 9547.9ms/step | 2.1 samples/s
545
+ Step 40020/44000 | Loss: 2.8966 | LR: 1.04e-05 | 3466.2ms/step | 5.8 samples/s
546
+ Step 40030/44000 | Loss: 2.9171 | LR: 1.03e-05 | 3468.3ms/step | 5.8 samples/s
547
+ Step 40040/44000 | Loss: 2.8924 | LR: 1.03e-05 | 3467.9ms/step | 5.8 samples/s
548
+ Step 40050/44000 | Loss: 2.8982 | LR: 1.02e-05 | 3466.5ms/step | 5.8 samples/s
549
+ Step 40060/44000 | Loss: 2.9186 | LR: 1.02e-05 | 3466.2ms/step | 5.8 samples/s
550
+ Step 40070/44000 | Loss: 2.8908 | LR: 1.01e-05 | 3466.3ms/step | 5.8 samples/s
551
+ Step 40080/44000 | Loss: 2.9161 | LR: 1.01e-05 | 3465.5ms/step | 5.8 samples/s
552
+ Step 40090/44000 | Loss: 2.8892 | LR: 1.00e-05 | 3465.6ms/step | 5.8 samples/s
553
+ Step 40100/44000 | Loss: 2.8952 | LR: 9.96e-06 | 3468.9ms/step | 5.8 samples/s
554
+ Step 40110/44000 | Loss: 2.8823 | LR: 9.91e-06 | 3466.0ms/step | 5.8 samples/s
555
+ Step 40120/44000 | Loss: 2.8843 | LR: 9.86e-06 | 3467.9ms/step | 5.8 samples/s
556
+ Step 40130/44000 | Loss: 2.8754 | LR: 9.81e-06 | 3467.7ms/step | 5.8 samples/s
557
+ Step 40140/44000 | Loss: 2.9071 | LR: 9.76e-06 | 3467.8ms/step | 5.8 samples/s
558
+ Step 40150/44000 | Loss: 2.8885 | LR: 9.71e-06 | 3465.9ms/step | 5.8 samples/s
559
+ Step 40160/44000 | Loss: 2.8955 | LR: 9.66e-06 | 3466.0ms/step | 5.8 samples/s
560
+ Step 40170/44000 | Loss: 2.8828 | LR: 9.61e-06 | 3466.0ms/step | 5.8 samples/s
561
+ Step 40180/44000 | Loss: 2.8898 | LR: 9.56e-06 | 3467.2ms/step | 5.8 samples/s
562
+ Step 40190/44000 | Loss: 2.8718 | LR: 9.51e-06 | 3465.4ms/step | 5.8 samples/s
563
+ Step 40200/44000 | Loss: 2.8875 | LR: 9.46e-06 | 3465.2ms/step | 5.8 samples/s
564
+ Step 40210/44000 | Loss: 2.8671 | LR: 9.41e-06 | 3463.9ms/step | 5.8 samples/s
565
+ Step 40220/44000 | Loss: 2.8737 | LR: 9.36e-06 | 3466.7ms/step | 5.8 samples/s
566
+ Step 40230/44000 | Loss: 2.9046 | LR: 9.32e-06 | 3465.6ms/step | 5.8 samples/s
567
+ Step 40240/44000 | Loss: 2.8717 | LR: 9.27e-06 | 3465.4ms/step | 5.8 samples/s
568
+ Step 40250/44000 | Loss: 2.9145 | LR: 9.22e-06 | 3467.7ms/step | 5.8 samples/s
569
+ Validation at step 40250: Loss = 2.8987, Perplexity = 18.15
570
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-40250
571
+ Step 40260/44000 | Loss: 2.8879 | LR: 9.17e-06 | 9551.6ms/step | 2.1 samples/s
572
+ Step 40270/44000 | Loss: 2.8871 | LR: 9.12e-06 | 3465.5ms/step | 5.8 samples/s
573
+ Step 40280/44000 | Loss: 2.9108 | LR: 9.07e-06 | 3463.4ms/step | 5.8 samples/s
574
+ Step 40290/44000 | Loss: 2.8727 | LR: 9.02e-06 | 3462.9ms/step | 5.8 samples/s
575
+ Step 40300/44000 | Loss: 2.9246 | LR: 8.97e-06 | 3464.4ms/step | 5.8 samples/s
576
+ Step 40310/44000 | Loss: 2.8955 | LR: 8.93e-06 | 3463.2ms/step | 5.8 samples/s
577
+ Step 40320/44000 | Loss: 2.8900 | LR: 8.88e-06 | 3462.4ms/step | 5.8 samples/s
578
+ Step 40330/44000 | Loss: 2.9071 | LR: 8.83e-06 | 3460.8ms/step | 5.8 samples/s
579
+ Step 40340/44000 | Loss: 2.8954 | LR: 8.78e-06 | 3462.7ms/step | 5.8 samples/s
580
+ Step 40350/44000 | Loss: 2.8510 | LR: 8.74e-06 | 3465.4ms/step | 5.8 samples/s
581
+ Step 40360/44000 | Loss: 2.9016 | LR: 8.69e-06 | 3463.5ms/step | 5.8 samples/s
582
+ Step 40370/44000 | Loss: 2.9078 | LR: 8.64e-06 | 3463.0ms/step | 5.8 samples/s
583
+ Step 40380/44000 | Loss: 2.8830 | LR: 8.59e-06 | 3464.0ms/step | 5.8 samples/s
584
+ Step 40390/44000 | Loss: 2.8915 | LR: 8.55e-06 | 3465.6ms/step | 5.8 samples/s
585
+ Step 40400/44000 | Loss: 2.8943 | LR: 8.50e-06 | 3463.4ms/step | 5.8 samples/s
586
+ Step 40410/44000 | Loss: 2.8862 | LR: 8.45e-06 | 3466.6ms/step | 5.8 samples/s
587
+ Step 40420/44000 | Loss: 2.8595 | LR: 8.41e-06 | 3466.0ms/step | 5.8 samples/s
588
+ Step 40430/44000 | Loss: 2.8958 | LR: 8.36e-06 | 3464.2ms/step | 5.8 samples/s
589
+ Step 40440/44000 | Loss: 2.8897 | LR: 8.31e-06 | 3465.7ms/step | 5.8 samples/s
590
+ Step 40450/44000 | Loss: 2.8980 | LR: 8.27e-06 | 3465.4ms/step | 5.8 samples/s
591
+ Step 40460/44000 | Loss: 2.9124 | LR: 8.22e-06 | 3468.3ms/step | 5.8 samples/s
592
+ Step 40470/44000 | Loss: 2.8819 | LR: 8.17e-06 | 3465.6ms/step | 5.8 samples/s
593
+ Step 40480/44000 | Loss: 2.8992 | LR: 8.13e-06 | 3464.5ms/step | 5.8 samples/s
594
+ Step 40490/44000 | Loss: 2.9093 | LR: 8.08e-06 | 3464.2ms/step | 5.8 samples/s
595
+ Step 40500/44000 | Loss: 2.8790 | LR: 8.04e-06 | 3463.2ms/step | 5.8 samples/s
596
+ Validation at step 40500: Loss = 2.8987, Perplexity = 18.15
597
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-40500
598
+ Step 40510/44000 | Loss: 2.8937 | LR: 7.99e-06 | 9545.7ms/step | 2.1 samples/s
599
+ Step 40520/44000 | Loss: 2.8799 | LR: 7.94e-06 | 3460.9ms/step | 5.8 samples/s
600
+ Step 40530/44000 | Loss: 2.8707 | LR: 7.90e-06 | 3459.4ms/step | 5.8 samples/s
601
+ Step 40540/44000 | Loss: 2.8899 | LR: 7.85e-06 | 3459.9ms/step | 5.8 samples/s
602
+ Step 40550/44000 | Loss: 2.9034 | LR: 7.81e-06 | 3458.4ms/step | 5.8 samples/s
603
+ Step 40560/44000 | Loss: 2.8767 | LR: 7.76e-06 | 3459.9ms/step | 5.8 samples/s
604
+ Step 40570/44000 | Loss: 2.8687 | LR: 7.72e-06 | 3459.1ms/step | 5.8 samples/s
605
+ Step 40580/44000 | Loss: 2.9031 | LR: 7.67e-06 | 3462.9ms/step | 5.8 samples/s
606
+ Step 40590/44000 | Loss: 2.8927 | LR: 7.63e-06 | 3474.3ms/step | 5.8 samples/s
607
+ Step 40600/44000 | Loss: 2.8765 | LR: 7.59e-06 | 3483.0ms/step | 5.7 samples/s
608
+ Step 40610/44000 | Loss: 2.9018 | LR: 7.54e-06 | 3491.2ms/step | 5.7 samples/s
609
+ Step 40620/44000 | Loss: 2.8886 | LR: 7.50e-06 | 3496.3ms/step | 5.7 samples/s
610
+ Step 40630/44000 | Loss: 2.9105 | LR: 7.45e-06 | 3504.5ms/step | 5.7 samples/s
611
+ Step 40640/44000 | Loss: 2.8446 | LR: 7.41e-06 | 3507.5ms/step | 5.7 samples/s
612
+ Step 40650/44000 | Loss: 2.8884 | LR: 7.37e-06 | 3507.7ms/step | 5.7 samples/s
613
+ Step 40660/44000 | Loss: 2.8812 | LR: 7.32e-06 | 3507.0ms/step | 5.7 samples/s
614
+ Step 40670/44000 | Loss: 2.8890 | LR: 7.28e-06 | 3509.8ms/step | 5.7 samples/s
615
+ Step 40680/44000 | Loss: 2.8612 | LR: 7.23e-06 | 3509.9ms/step | 5.7 samples/s
616
+ Step 40690/44000 | Loss: 2.8967 | LR: 7.19e-06 | 3508.5ms/step | 5.7 samples/s
617
+ Step 40700/44000 | Loss: 2.8979 | LR: 7.15e-06 | 3509.1ms/step | 5.7 samples/s
618
+ Step 40710/44000 | Loss: 2.9261 | LR: 7.10e-06 | 3507.2ms/step | 5.7 samples/s
619
+ Step 40720/44000 | Loss: 2.8902 | LR: 7.06e-06 | 3508.6ms/step | 5.7 samples/s
620
+ Step 40730/44000 | Loss: 2.8674 | LR: 7.02e-06 | 3508.3ms/step | 5.7 samples/s
621
+ Step 40740/44000 | Loss: 2.8610 | LR: 6.98e-06 | 3511.3ms/step | 5.7 samples/s
622
+ Step 40750/44000 | Loss: 2.8917 | LR: 6.93e-06 | 3508.1ms/step | 5.7 samples/s
623
+ Validation at step 40750: Loss = 2.8987, Perplexity = 18.15
624
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-40750
625
+ Step 40760/44000 | Loss: 2.9046 | LR: 6.89e-06 | 9678.1ms/step | 2.1 samples/s
626
+ Step 40770/44000 | Loss: 2.9054 | LR: 6.85e-06 | 3507.8ms/step | 5.7 samples/s
627
+ Step 40780/44000 | Loss: 2.8926 | LR: 6.81e-06 | 3510.3ms/step | 5.7 samples/s
628
+ Step 40790/44000 | Loss: 2.8806 | LR: 6.77e-06 | 3508.0ms/step | 5.7 samples/s
629
+ Step 40800/44000 | Loss: 2.9036 | LR: 6.72e-06 | 3500.3ms/step | 5.7 samples/s
630
+ Step 40810/44000 | Loss: 2.8903 | LR: 6.68e-06 | 3500.7ms/step | 5.7 samples/s
631
+ Step 40820/44000 | Loss: 2.8842 | LR: 6.64e-06 | 3504.0ms/step | 5.7 samples/s
632
+ Step 40830/44000 | Loss: 2.8912 | LR: 6.60e-06 | 3505.7ms/step | 5.7 samples/s
633
+ Step 40840/44000 | Loss: 2.9014 | LR: 6.56e-06 | 3508.5ms/step | 5.7 samples/s
634
+ Step 40850/44000 | Loss: 2.9087 | LR: 6.52e-06 | 3506.8ms/step | 5.7 samples/s
635
+ Step 40860/44000 | Loss: 2.8838 | LR: 6.47e-06 | 3509.4ms/step | 5.7 samples/s
636
+ Step 40870/44000 | Loss: 2.9117 | LR: 6.43e-06 | 3507.8ms/step | 5.7 samples/s
637
+ Step 40880/44000 | Loss: 2.8771 | LR: 6.39e-06 | 3510.0ms/step | 5.7 samples/s
638
+ Step 40890/44000 | Loss: 2.8666 | LR: 6.35e-06 | 3509.3ms/step | 5.7 samples/s
639
+ Step 40900/44000 | Loss: 2.8878 | LR: 6.31e-06 | 3512.2ms/step | 5.7 samples/s
640
+ Step 40910/44000 | Loss: 2.9274 | LR: 6.27e-06 | 3512.1ms/step | 5.7 samples/s
641
+ Step 40920/44000 | Loss: 2.9036 | LR: 6.23e-06 | 3509.9ms/step | 5.7 samples/s
642
+ Step 40930/44000 | Loss: 2.9054 | LR: 6.19e-06 | 3508.8ms/step | 5.7 samples/s
643
+ Step 40940/44000 | Loss: 2.8761 | LR: 6.15e-06 | 3509.9ms/step | 5.7 samples/s
644
+ Step 40950/44000 | Loss: 2.8720 | LR: 6.11e-06 | 3510.3ms/step | 5.7 samples/s
645
+ Step 40960/44000 | Loss: 2.8882 | LR: 6.07e-06 | 3510.2ms/step | 5.7 samples/s
646
+ Step 40970/44000 | Loss: 2.9202 | LR: 6.03e-06 | 3507.0ms/step | 5.7 samples/s
647
+ Step 40980/44000 | Loss: 2.9053 | LR: 5.99e-06 | 3508.8ms/step | 5.7 samples/s
648
+ Step 40990/44000 | Loss: 2.8746 | LR: 5.95e-06 | 3510.4ms/step | 5.7 samples/s
649
+ Step 41000/44000 | Loss: 2.9139 | LR: 5.91e-06 | 3510.7ms/step | 5.7 samples/s
650
+ Validation at step 41000: Loss = 2.8987, Perplexity = 18.15
651
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-41000
652
+ Step 41010/44000 | Loss: 2.8888 | LR: 5.87e-06 | 9683.7ms/step | 2.1 samples/s
653
+ Step 41020/44000 | Loss: 2.9131 | LR: 5.83e-06 | 3501.7ms/step | 5.7 samples/s
654
+ Step 41030/44000 | Loss: 2.9057 | LR: 5.80e-06 | 3508.3ms/step | 5.7 samples/s
655
+ Step 41040/44000 | Loss: 2.9026 | LR: 5.76e-06 | 3515.7ms/step | 5.7 samples/s
656
+ Step 41050/44000 | Loss: 2.8731 | LR: 5.72e-06 | 3514.8ms/step | 5.7 samples/s
657
+ Step 41060/44000 | Loss: 2.8813 | LR: 5.68e-06 | 3513.5ms/step | 5.7 samples/s
658
+ Step 41070/44000 | Loss: 2.9127 | LR: 5.64e-06 | 3514.8ms/step | 5.7 samples/s
659
+ Step 41080/44000 | Loss: 2.8899 | LR: 5.60e-06 | 3511.1ms/step | 5.7 samples/s
660
+ Step 41090/44000 | Loss: 2.8695 | LR: 5.56e-06 | 3515.8ms/step | 5.7 samples/s
661
+ Step 41100/44000 | Loss: 2.8916 | LR: 5.53e-06 | 3514.6ms/step | 5.7 samples/s
662
+ Step 41110/44000 | Loss: 2.9282 | LR: 5.49e-06 | 3515.4ms/step | 5.7 samples/s
663
+ Step 41120/44000 | Loss: 2.9109 | LR: 5.45e-06 | 3513.5ms/step | 5.7 samples/s
664
+ Step 41130/44000 | Loss: 2.8971 | LR: 5.41e-06 | 3516.5ms/step | 5.7 samples/s
665
+ Step 41140/44000 | Loss: 2.8867 | LR: 5.38e-06 | 3515.5ms/step | 5.7 samples/s
666
+ Step 41150/44000 | Loss: 2.9046 | LR: 5.34e-06 | 3513.8ms/step | 5.7 samples/s
667
+ Step 41160/44000 | Loss: 2.8707 | LR: 5.30e-06 | 3511.3ms/step | 5.7 samples/s
668
+ Step 41170/44000 | Loss: 2.8997 | LR: 5.26e-06 | 3511.6ms/step | 5.7 samples/s
669
+ Step 41180/44000 | Loss: 2.8717 | LR: 5.23e-06 | 3510.7ms/step | 5.7 samples/s
670
+ Step 41190/44000 | Loss: 2.8534 | LR: 5.19e-06 | 3510.8ms/step | 5.7 samples/s
671
+ Step 41200/44000 | Loss: 2.9028 | LR: 5.15e-06 | 3512.4ms/step | 5.7 samples/s
672
+ Step 41210/44000 | Loss: 2.8751 | LR: 5.12e-06 | 3512.6ms/step | 5.7 samples/s
673
+ Step 41220/44000 | Loss: 2.8918 | LR: 5.08e-06 | 3515.7ms/step | 5.7 samples/s
674
+ Step 41230/44000 | Loss: 2.8951 | LR: 5.04e-06 | 3514.6ms/step | 5.7 samples/s
675
+ Step 41240/44000 | Loss: 2.8845 | LR: 5.01e-06 | 3510.8ms/step | 5.7 samples/s
676
+ Step 41250/44000 | Loss: 2.9030 | LR: 4.97e-06 | 3504.3ms/step | 5.7 samples/s
677
+ Validation at step 41250: Loss = 2.8987, Perplexity = 18.15
678
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-41250
679
+ Step 41260/44000 | Loss: 2.9052 | LR: 4.94e-06 | 9677.5ms/step | 2.1 samples/s
680
+ Step 41270/44000 | Loss: 2.9093 | LR: 4.90e-06 | 3508.2ms/step | 5.7 samples/s
681
+ Step 41280/44000 | Loss: 2.8456 | LR: 4.86e-06 | 3512.7ms/step | 5.7 samples/s
682
+ Step 41290/44000 | Loss: 2.8685 | LR: 4.83e-06 | 3511.4ms/step | 5.7 samples/s
683
+ Step 41300/44000 | Loss: 2.8994 | LR: 4.79e-06 | 3512.5ms/step | 5.7 samples/s
684
+ Step 41310/44000 | Loss: 2.9115 | LR: 4.76e-06 | 3509.5ms/step | 5.7 samples/s
685
+ Step 41320/44000 | Loss: 2.8908 | LR: 4.72e-06 | 3513.9ms/step | 5.7 samples/s
686
+ Step 41330/44000 | Loss: 2.8803 | LR: 4.69e-06 | 3513.2ms/step | 5.7 samples/s
687
+ Step 41340/44000 | Loss: 2.8966 | LR: 4.65e-06 | 3513.2ms/step | 5.7 samples/s
688
+ Step 41350/44000 | Loss: 2.9269 | LR: 4.62e-06 | 3511.2ms/step | 5.7 samples/s
689
+ Step 41360/44000 | Loss: 2.8723 | LR: 4.58e-06 | 3511.2ms/step | 5.7 samples/s
690
+ Step 41370/44000 | Loss: 2.8957 | LR: 4.55e-06 | 3509.9ms/step | 5.7 samples/s
691
+ Step 41380/44000 | Loss: 2.8997 | LR: 4.51e-06 | 3510.6ms/step | 5.7 samples/s
692
+ Step 41390/44000 | Loss: 2.8852 | LR: 4.48e-06 | 3509.5ms/step | 5.7 samples/s
693
+ Step 41400/44000 | Loss: 2.8752 | LR: 4.45e-06 | 3512.9ms/step | 5.7 samples/s
694
+ Step 41410/44000 | Loss: 2.9105 | LR: 4.41e-06 | 3511.7ms/step | 5.7 samples/s
695
+ Step 41420/44000 | Loss: 2.8922 | LR: 4.38e-06 | 3510.0ms/step | 5.7 samples/s
696
+ Step 41430/44000 | Loss: 2.8755 | LR: 4.34e-06 | 3510.6ms/step | 5.7 samples/s
697
+ Step 41440/44000 | Loss: 2.9017 | LR: 4.31e-06 | 3513.0ms/step | 5.7 samples/s
698
+ Step 41450/44000 | Loss: 2.8862 | LR: 4.28e-06 | 3512.4ms/step | 5.7 samples/s
699
+ Step 41460/44000 | Loss: 2.9003 | LR: 4.24e-06 | 3504.0ms/step | 5.7 samples/s
700
+ Step 41470/44000 | Loss: 2.9294 | LR: 4.21e-06 | 3506.3ms/step | 5.7 samples/s
701
+ Step 41480/44000 | Loss: 2.8970 | LR: 4.18e-06 | 3513.1ms/step | 5.7 samples/s
702
+ Step 41490/44000 | Loss: 2.8970 | LR: 4.14e-06 | 3510.2ms/step | 5.7 samples/s
703
+ Step 41500/44000 | Loss: 2.9136 | LR: 4.11e-06 | 3507.6ms/step | 5.7 samples/s
704
+ Validation at step 41500: Loss = 2.8987, Perplexity = 18.15
705
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-41500
706
+ Step 41510/44000 | Loss: 2.8846 | LR: 4.08e-06 | 9683.9ms/step | 2.1 samples/s
707
+ Step 41520/44000 | Loss: 2.9049 | LR: 4.05e-06 | 3516.5ms/step | 5.7 samples/s
708
+ Step 41530/44000 | Loss: 2.9238 | LR: 4.01e-06 | 3519.7ms/step | 5.7 samples/s
709
+ Step 41540/44000 | Loss: 2.8817 | LR: 3.98e-06 | 3519.1ms/step | 5.7 samples/s
710
+ Step 41550/44000 | Loss: 2.8960 | LR: 3.95e-06 | 3518.6ms/step | 5.7 samples/s
711
+ Step 41560/44000 | Loss: 2.8886 | LR: 3.92e-06 | 3520.6ms/step | 5.7 samples/s
712
+ Step 41570/44000 | Loss: 2.9046 | LR: 3.88e-06 | 3519.4ms/step | 5.7 samples/s
713
+ Step 41580/44000 | Loss: 2.8980 | LR: 3.85e-06 | 3522.4ms/step | 5.7 samples/s
714
+ Step 41590/44000 | Loss: 2.9026 | LR: 3.82e-06 | 3521.4ms/step | 5.7 samples/s
715
+ Step 41600/44000 | Loss: 2.8683 | LR: 3.79e-06 | 3521.0ms/step | 5.7 samples/s
716
+ Step 41610/44000 | Loss: 2.8874 | LR: 3.76e-06 | 3520.3ms/step | 5.7 samples/s
717
+ Step 41620/44000 | Loss: 2.8571 | LR: 3.73e-06 | 3520.3ms/step | 5.7 samples/s
718
+ Step 41630/44000 | Loss: 2.8714 | LR: 3.70e-06 | 3520.9ms/step | 5.7 samples/s
719
+ Step 41640/44000 | Loss: 2.9213 | LR: 3.66e-06 | 3522.0ms/step | 5.7 samples/s
720
+ Step 41650/44000 | Loss: 2.8592 | LR: 3.63e-06 | 3518.9ms/step | 5.7 samples/s
721
+ Step 41660/44000 | Loss: 2.8996 | LR: 3.60e-06 | 3519.7ms/step | 5.7 samples/s
722
+ Step 41670/44000 | Loss: 2.8926 | LR: 3.57e-06 | 3513.9ms/step | 5.7 samples/s
723
+ Step 41680/44000 | Loss: 2.8865 | LR: 3.54e-06 | 3510.3ms/step | 5.7 samples/s
724
+ Step 41690/44000 | Loss: 2.8659 | LR: 3.51e-06 | 3515.7ms/step | 5.7 samples/s
725
+ Step 41700/44000 | Loss: 2.8734 | LR: 3.48e-06 | 3516.7ms/step | 5.7 samples/s
726
+ Step 41710/44000 | Loss: 2.8798 | LR: 3.45e-06 | 3516.8ms/step | 5.7 samples/s
727
+ Step 41720/44000 | Loss: 2.8886 | LR: 3.42e-06 | 3515.9ms/step | 5.7 samples/s
728
+ Step 41730/44000 | Loss: 2.9168 | LR: 3.39e-06 | 3518.4ms/step | 5.7 samples/s
729
+ Step 41740/44000 | Loss: 2.8732 | LR: 3.36e-06 | 3517.1ms/step | 5.7 samples/s
730
+ Step 41750/44000 | Loss: 2.9010 | LR: 3.33e-06 | 3518.1ms/step | 5.7 samples/s
731
+ Validation at step 41750: Loss = 2.8987, Perplexity = 18.15
732
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-41750
733
+ Step 41760/44000 | Loss: 2.9002 | LR: 3.30e-06 | 9714.6ms/step | 2.1 samples/s
734
+ Step 41770/44000 | Loss: 2.8803 | LR: 3.27e-06 | 3508.7ms/step | 5.7 samples/s
735
+ Step 41780/44000 | Loss: 2.8890 | LR: 3.24e-06 | 3510.7ms/step | 5.7 samples/s
736
+ Step 41790/44000 | Loss: 2.9087 | LR: 3.21e-06 | 3512.6ms/step | 5.7 samples/s
737
+ Step 41800/44000 | Loss: 2.8866 | LR: 3.19e-06 | 3514.1ms/step | 5.7 samples/s
738
+ Step 41810/44000 | Loss: 2.8945 | LR: 3.16e-06 | 3516.5ms/step | 5.7 samples/s
739
+ Step 41820/44000 | Loss: 2.8902 | LR: 3.13e-06 | 3515.0ms/step | 5.7 samples/s
740
+ Step 41830/44000 | Loss: 2.8950 | LR: 3.10e-06 | 3518.1ms/step | 5.7 samples/s
741
+ Step 41840/44000 | Loss: 2.8978 | LR: 3.07e-06 | 3519.0ms/step | 5.7 samples/s
742
+ Step 41850/44000 | Loss: 2.8442 | LR: 3.04e-06 | 3519.2ms/step | 5.7 samples/s
743
+ Step 41860/44000 | Loss: 2.9178 | LR: 3.01e-06 | 3517.8ms/step | 5.7 samples/s
744
+ Step 41870/44000 | Loss: 2.8747 | LR: 2.99e-06 | 3515.5ms/step | 5.7 samples/s
745
+ Step 41880/44000 | Loss: 2.9083 | LR: 2.96e-06 | 3516.0ms/step | 5.7 samples/s
746
+ Step 41890/44000 | Loss: 2.8718 | LR: 2.93e-06 | 3510.3ms/step | 5.7 samples/s
747
+ Step 41900/44000 | Loss: 2.8878 | LR: 2.90e-06 | 3513.7ms/step | 5.7 samples/s
748
+ Step 41910/44000 | Loss: 2.8779 | LR: 2.88e-06 | 3517.0ms/step | 5.7 samples/s
749
+ Step 41920/44000 | Loss: 2.8959 | LR: 2.85e-06 | 3516.3ms/step | 5.7 samples/s
750
+ Step 41930/44000 | Loss: 2.8808 | LR: 2.82e-06 | 3516.3ms/step | 5.7 samples/s
751
+ Step 41940/44000 | Loss: 2.9093 | LR: 2.79e-06 | 3516.2ms/step | 5.7 samples/s
752
+ Step 41950/44000 | Loss: 2.8807 | LR: 2.77e-06 | 3517.8ms/step | 5.7 samples/s
753
+ Step 41960/44000 | Loss: 2.9029 | LR: 2.74e-06 | 3518.9ms/step | 5.7 samples/s
754
+ Step 41970/44000 | Loss: 2.9176 | LR: 2.71e-06 | 3517.9ms/step | 5.7 samples/s
755
+ Step 41980/44000 | Loss: 2.8897 | LR: 2.69e-06 | 3518.9ms/step | 5.7 samples/s
756
+ Step 41990/44000 | Loss: 2.8724 | LR: 2.66e-06 | 3516.2ms/step | 5.7 samples/s
757
+ Step 42000/44000 | Loss: 2.8726 | LR: 2.63e-06 | 3516.2ms/step | 5.7 samples/s
758
+ Validation at step 42000: Loss = 2.8987, Perplexity = 18.15
759
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-42000
760
+ Step 42010/44000 | Loss: 2.8936 | LR: 2.61e-06 | 9715.4ms/step | 2.1 samples/s
761
+ Step 42020/44000 | Loss: 2.8923 | LR: 2.58e-06 | 3519.6ms/step | 5.7 samples/s
762
+ Step 42030/44000 | Loss: 2.8910 | LR: 2.56e-06 | 3519.8ms/step | 5.7 samples/s
763
+ Step 42040/44000 | Loss: 2.8624 | LR: 2.53e-06 | 3520.3ms/step | 5.7 samples/s
764
+ Step 42050/44000 | Loss: 2.8966 | LR: 2.50e-06 | 3519.5ms/step | 5.7 samples/s
765
+ Step 42060/44000 | Loss: 2.8963 | LR: 2.48e-06 | 3519.8ms/step | 5.7 samples/s
766
+ Step 42070/44000 | Loss: 2.8793 | LR: 2.45e-06 | 3518.8ms/step | 5.7 samples/s
767
+ Step 42080/44000 | Loss: 2.8939 | LR: 2.43e-06 | 3519.1ms/step | 5.7 samples/s
768
+ Step 42090/44000 | Loss: 2.9351 | LR: 2.40e-06 | 3519.8ms/step | 5.7 samples/s
769
+ Step 42100/44000 | Loss: 2.9045 | LR: 2.38e-06 | 3515.1ms/step | 5.7 samples/s
770
+ Step 42110/44000 | Loss: 2.8553 | LR: 2.35e-06 | 3515.7ms/step | 5.7 samples/s
771
+ Step 42120/44000 | Loss: 2.8828 | LR: 2.33e-06 | 3513.5ms/step | 5.7 samples/s
772
+ Step 42130/44000 | Loss: 2.8720 | LR: 2.30e-06 | 3514.2ms/step | 5.7 samples/s
773
+ Step 42140/44000 | Loss: 2.8755 | LR: 2.28e-06 | 3516.5ms/step | 5.7 samples/s
774
+ Step 42150/44000 | Loss: 2.8689 | LR: 2.25e-06 | 3521.0ms/step | 5.7 samples/s
775
+ Step 42160/44000 | Loss: 2.8627 | LR: 2.23e-06 | 3522.5ms/step | 5.7 samples/s
776
+ Step 42170/44000 | Loss: 2.9043 | LR: 2.21e-06 | 3517.9ms/step | 5.7 samples/s
777
+ Step 42180/44000 | Loss: 2.8772 | LR: 2.18e-06 | 3516.1ms/step | 5.7 samples/s
778
+ Step 42190/44000 | Loss: 2.8766 | LR: 2.16e-06 | 3519.3ms/step | 5.7 samples/s
779
+ Step 42200/44000 | Loss: 2.8802 | LR: 2.13e-06 | 3520.2ms/step | 5.7 samples/s
780
+ Step 42210/44000 | Loss: 2.8655 | LR: 2.11e-06 | 3519.6ms/step | 5.7 samples/s
781
+ Step 42220/44000 | Loss: 2.8910 | LR: 2.09e-06 | 3519.7ms/step | 5.7 samples/s
782
+ Step 42230/44000 | Loss: 2.8854 | LR: 2.06e-06 | 3520.1ms/step | 5.7 samples/s
783
+ Step 42240/44000 | Loss: 2.8858 | LR: 2.04e-06 | 3520.6ms/step | 5.7 samples/s
784
+ Step 42250/44000 | Loss: 2.9076 | LR: 2.02e-06 | 3520.5ms/step | 5.7 samples/s
785
+ Validation at step 42250: Loss = 2.8987, Perplexity = 18.15
786
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-42250
787
+ Step 42260/44000 | Loss: 2.8702 | LR: 1.99e-06 | 9705.4ms/step | 2.1 samples/s
788
+ Step 42270/44000 | Loss: 2.9162 | LR: 1.97e-06 | 3508.3ms/step | 5.7 samples/s
789
+ Step 42280/44000 | Loss: 2.8888 | LR: 1.95e-06 | 3509.2ms/step | 5.7 samples/s
790
+ Step 42290/44000 | Loss: 2.8848 | LR: 1.93e-06 | 3515.6ms/step | 5.7 samples/s
791
+ Step 42300/44000 | Loss: 2.8640 | LR: 1.90e-06 | 3511.1ms/step | 5.7 samples/s
792
+ Step 42310/44000 | Loss: 2.8843 | LR: 1.88e-06 | 3511.0ms/step | 5.7 samples/s
793
+ Step 42320/44000 | Loss: 2.9015 | LR: 1.86e-06 | 3509.5ms/step | 5.7 samples/s
794
+ Step 42330/44000 | Loss: 2.8917 | LR: 1.84e-06 | 3513.8ms/step | 5.7 samples/s
795
+ Step 42340/44000 | Loss: 2.8576 | LR: 1.82e-06 | 3514.1ms/step | 5.7 samples/s
796
+ Step 42350/44000 | Loss: 2.8870 | LR: 1.79e-06 | 3516.5ms/step | 5.7 samples/s
797
+ Step 42360/44000 | Loss: 2.8829 | LR: 1.77e-06 | 3516.6ms/step | 5.7 samples/s
798
+ Step 42370/44000 | Loss: 2.9150 | LR: 1.75e-06 | 3516.9ms/step | 5.7 samples/s
799
+ Step 42380/44000 | Loss: 2.8597 | LR: 1.73e-06 | 3519.5ms/step | 5.7 samples/s
800
+ Step 42390/44000 | Loss: 2.9040 | LR: 1.71e-06 | 3517.3ms/step | 5.7 samples/s
801
+ Step 42400/44000 | Loss: 2.8636 | LR: 1.69e-06 | 3521.1ms/step | 5.7 samples/s
802
+ Step 42410/44000 | Loss: 2.8837 | LR: 1.67e-06 | 3521.0ms/step | 5.7 samples/s
803
+ Step 42420/44000 | Loss: 2.9041 | LR: 1.64e-06 | 3518.3ms/step | 5.7 samples/s
804
+ Step 42430/44000 | Loss: 2.8726 | LR: 1.62e-06 | 3519.4ms/step | 5.7 samples/s
805
+ Step 42440/44000 | Loss: 2.8740 | LR: 1.60e-06 | 3519.4ms/step | 5.7 samples/s
806
+ Step 42450/44000 | Loss: 2.8454 | LR: 1.58e-06 | 3521.1ms/step | 5.7 samples/s
807
+ Step 42460/44000 | Loss: 2.8734 | LR: 1.56e-06 | 3517.8ms/step | 5.7 samples/s
808
+ Step 42470/44000 | Loss: 2.8690 | LR: 1.54e-06 | 3522.7ms/step | 5.7 samples/s
809
+ Step 42480/44000 | Loss: 2.8756 | LR: 1.52e-06 | 3519.8ms/step | 5.7 samples/s
810
+ Step 42490/44000 | Loss: 2.9116 | LR: 1.50e-06 | 3521.2ms/step | 5.7 samples/s
811
+ Step 42500/44000 | Loss: 2.8901 | LR: 1.48e-06 | 3520.6ms/step | 5.7 samples/s
812
+ Validation at step 42500: Loss = 2.8987, Perplexity = 18.15
813
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-42500
814
+ Step 42510/44000 | Loss: 2.8763 | LR: 1.46e-06 | 9721.6ms/step | 2.1 samples/s
815
+ Step 42520/44000 | Loss: 2.8956 | LR: 1.44e-06 | 3513.0ms/step | 5.7 samples/s
816
+ Step 42530/44000 | Loss: 2.8706 | LR: 1.42e-06 | 3504.8ms/step | 5.7 samples/s
817
+ Step 42540/44000 | Loss: 2.8609 | LR: 1.40e-06 | 3509.5ms/step | 5.7 samples/s
818
+ Step 42550/44000 | Loss: 2.8891 | LR: 1.39e-06 | 3511.5ms/step | 5.7 samples/s
819
+ Step 42560/44000 | Loss: 2.9173 | LR: 1.37e-06 | 3514.4ms/step | 5.7 samples/s
820
+ Step 42570/44000 | Loss: 2.9297 | LR: 1.35e-06 | 3513.1ms/step | 5.7 samples/s
821
+ Step 42580/44000 | Loss: 2.8933 | LR: 1.33e-06 | 3513.2ms/step | 5.7 samples/s
822
+ Step 42590/44000 | Loss: 2.8962 | LR: 1.31e-06 | 3513.2ms/step | 5.7 samples/s
823
+ Step 42600/44000 | Loss: 2.8787 | LR: 1.29e-06 | 3515.5ms/step | 5.7 samples/s
824
+ Step 42610/44000 | Loss: 2.8847 | LR: 1.27e-06 | 3514.9ms/step | 5.7 samples/s
825
+ Step 42620/44000 | Loss: 2.9044 | LR: 1.25e-06 | 3513.6ms/step | 5.7 samples/s
826
+ Step 42630/44000 | Loss: 2.9013 | LR: 1.24e-06 | 3513.1ms/step | 5.7 samples/s
827
+ Step 42640/44000 | Loss: 2.8794 | LR: 1.22e-06 | 3512.7ms/step | 5.7 samples/s
828
+ Step 42650/44000 | Loss: 2.8703 | LR: 1.20e-06 | 3513.7ms/step | 5.7 samples/s
829
+ Step 42660/44000 | Loss: 2.8851 | LR: 1.18e-06 | 3511.5ms/step | 5.7 samples/s
830
+ Step 42670/44000 | Loss: 2.8920 | LR: 1.17e-06 | 3514.1ms/step | 5.7 samples/s
831
+ Step 42680/44000 | Loss: 2.8818 | LR: 1.15e-06 | 3513.1ms/step | 5.7 samples/s
832
+ Step 42690/44000 | Loss: 2.8935 | LR: 1.13e-06 | 3514.0ms/step | 5.7 samples/s
833
+ Step 42700/44000 | Loss: 2.8744 | LR: 1.11e-06 | 3512.5ms/step | 5.7 samples/s
834
+ Step 42710/44000 | Loss: 2.9007 | LR: 1.10e-06 | 3513.0ms/step | 5.7 samples/s
835
+ Step 42720/44000 | Loss: 2.8494 | LR: 1.08e-06 | 3509.8ms/step | 5.7 samples/s
836
+ Step 42730/44000 | Loss: 2.8754 | LR: 1.06e-06 | 3508.4ms/step | 5.7 samples/s
837
+ Step 42740/44000 | Loss: 2.8934 | LR: 1.05e-06 | 3509.0ms/step | 5.7 samples/s
838
+ Step 42750/44000 | Loss: 2.9045 | LR: 1.03e-06 | 3508.4ms/step | 5.7 samples/s
839
+ Validation at step 42750: Loss = 2.8987, Perplexity = 18.15
840
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-42750
841
+ Step 42760/44000 | Loss: 2.8877 | LR: 1.01e-06 | 9689.1ms/step | 2.1 samples/s
842
+ Step 42770/44000 | Loss: 2.8789 | LR: 9.97e-07 | 3517.8ms/step | 5.7 samples/s
843
+ Step 42780/44000 | Loss: 2.8812 | LR: 9.81e-07 | 3517.6ms/step | 5.7 samples/s
844
+ Step 42790/44000 | Loss: 2.8740 | LR: 9.65e-07 | 3520.2ms/step | 5.7 samples/s
845
+ Step 42800/44000 | Loss: 2.8755 | LR: 9.49e-07 | 3516.1ms/step | 5.7 samples/s
846
+ Step 42810/44000 | Loss: 2.9011 | LR: 9.33e-07 | 3517.7ms/step | 5.7 samples/s
847
+ Step 42820/44000 | Loss: 2.8743 | LR: 9.18e-07 | 3518.4ms/step | 5.7 samples/s
848
+ Step 42830/44000 | Loss: 2.8875 | LR: 9.02e-07 | 3517.9ms/step | 5.7 samples/s
849
+ Step 42840/44000 | Loss: 2.9076 | LR: 8.87e-07 | 3518.0ms/step | 5.7 samples/s
850
+ Step 42850/44000 | Loss: 2.8533 | LR: 8.72e-07 | 3519.8ms/step | 5.7 samples/s
851
+ Step 42860/44000 | Loss: 2.9119 | LR: 8.57e-07 | 3517.0ms/step | 5.7 samples/s
852
+ Step 42870/44000 | Loss: 2.8859 | LR: 8.42e-07 | 3517.9ms/step | 5.7 samples/s
853
+ Step 42880/44000 | Loss: 2.9014 | LR: 8.27e-07 | 3517.3ms/step | 5.7 samples/s
854
+ Step 42890/44000 | Loss: 2.8723 | LR: 8.12e-07 | 3515.8ms/step | 5.7 samples/s
855
+ Step 42900/44000 | Loss: 2.8881 | LR: 7.98e-07 | 3520.3ms/step | 5.7 samples/s
856
+ Step 42910/44000 | Loss: 2.8907 | LR: 7.83e-07 | 3517.8ms/step | 5.7 samples/s
857
+ Step 42920/44000 | Loss: 2.8788 | LR: 7.69e-07 | 3517.4ms/step | 5.7 samples/s
858
+ Step 42930/44000 | Loss: 2.8434 | LR: 7.55e-07 | 3517.2ms/step | 5.7 samples/s
859
+ Step 42940/44000 | Loss: 2.9328 | LR: 7.41e-07 | 3517.8ms/step | 5.7 samples/s
860
+ Step 42950/44000 | Loss: 2.9159 | LR: 7.27e-07 | 3515.7ms/step | 5.7 samples/s
861
+ Step 42960/44000 | Loss: 2.9194 | LR: 7.13e-07 | 3516.4ms/step | 5.7 samples/s
862
+ Step 42970/44000 | Loss: 2.8913 | LR: 6.99e-07 | 3514.0ms/step | 5.7 samples/s
863
+ Step 42980/44000 | Loss: 2.8569 | LR: 6.86e-07 | 3509.1ms/step | 5.7 samples/s
864
+ Step 42990/44000 | Loss: 2.9074 | LR: 6.72e-07 | 3511.7ms/step | 5.7 samples/s
865
+ Step 43000/44000 | Loss: 2.8837 | LR: 6.59e-07 | 3515.6ms/step | 5.7 samples/s
866
+ Validation at step 43000: Loss = 2.8987, Perplexity = 18.15
867
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-43000
868
+ Step 43010/44000 | Loss: 2.8774 | LR: 6.46e-07 | 9718.0ms/step | 2.1 samples/s
869
+ Step 43020/44000 | Loss: 2.8805 | LR: 6.33e-07 | 3509.5ms/step | 5.7 samples/s
870
+ Step 43030/44000 | Loss: 2.8979 | LR: 6.20e-07 | 3509.0ms/step | 5.7 samples/s
871
+ Step 43040/44000 | Loss: 2.8940 | LR: 6.08e-07 | 3513.5ms/step | 5.7 samples/s
872
+ Step 43050/44000 | Loss: 2.8486 | LR: 5.95e-07 | 3510.8ms/step | 5.7 samples/s
873
+ Step 43060/44000 | Loss: 2.9199 | LR: 5.83e-07 | 3517.5ms/step | 5.7 samples/s
874
+ Step 43070/44000 | Loss: 2.8933 | LR: 5.70e-07 | 3516.3ms/step | 5.7 samples/s
875
+ Step 43080/44000 | Loss: 2.8969 | LR: 5.58e-07 | 3519.3ms/step | 5.7 samples/s
876
+ Step 43090/44000 | Loss: 2.8948 | LR: 5.46e-07 | 3518.3ms/step | 5.7 samples/s
877
+ Step 43100/44000 | Loss: 2.9190 | LR: 5.34e-07 | 3517.5ms/step | 5.7 samples/s
878
+ Step 43110/44000 | Loss: 2.8878 | LR: 5.22e-07 | 3518.1ms/step | 5.7 samples/s
879
+ Step 43120/44000 | Loss: 2.8871 | LR: 5.11e-07 | 3516.2ms/step | 5.7 samples/s
880
+ Step 43130/44000 | Loss: 2.8636 | LR: 4.99e-07 | 3515.4ms/step | 5.7 samples/s
881
+ Step 43140/44000 | Loss: 2.9234 | LR: 4.88e-07 | 3515.4ms/step | 5.7 samples/s
882
+ Step 43150/44000 | Loss: 2.8718 | LR: 4.76e-07 | 3518.4ms/step | 5.7 samples/s
883
+ Step 43160/44000 | Loss: 2.8906 | LR: 4.65e-07 | 3517.0ms/step | 5.7 samples/s
884
+ Step 43170/44000 | Loss: 2.8923 | LR: 4.54e-07 | 3515.6ms/step | 5.7 samples/s
885
+ Step 43180/44000 | Loss: 2.8760 | LR: 4.43e-07 | 3514.9ms/step | 5.7 samples/s
886
+ Step 43190/44000 | Loss: 2.9174 | LR: 4.33e-07 | 3509.9ms/step | 5.7 samples/s
887
+ Step 43200/44000 | Loss: 2.9021 | LR: 4.22e-07 | 3509.7ms/step | 5.7 samples/s
888
+ Step 43210/44000 | Loss: 2.8758 | LR: 4.12e-07 | 3514.8ms/step | 5.7 samples/s
889
+ Step 43220/44000 | Loss: 2.8998 | LR: 4.01e-07 | 3516.0ms/step | 5.7 samples/s
890
+ Step 43230/44000 | Loss: 2.8822 | LR: 3.91e-07 | 3519.7ms/step | 5.7 samples/s
891
+ Step 43240/44000 | Loss: 2.8958 | LR: 3.81e-07 | 3516.3ms/step | 5.7 samples/s
892
+ Step 43250/44000 | Loss: 2.9091 | LR: 3.71e-07 | 3516.9ms/step | 5.7 samples/s
893
+ Validation at step 43250: Loss = 2.8987, Perplexity = 18.15
894
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-43250
895
+ Step 43260/44000 | Loss: 2.9071 | LR: 3.61e-07 | 9707.8ms/step | 2.1 samples/s
896
+ Step 43270/44000 | Loss: 2.8985 | LR: 3.51e-07 | 3515.9ms/step | 5.7 samples/s
897
+ Step 43280/44000 | Loss: 2.9018 | LR: 3.42e-07 | 3516.0ms/step | 5.7 samples/s
898
+ Step 43290/44000 | Loss: 2.8951 | LR: 3.32e-07 | 3517.1ms/step | 5.7 samples/s
899
+ Step 43300/44000 | Loss: 2.9074 | LR: 3.23e-07 | 3519.2ms/step | 5.7 samples/s
900
+ Step 43310/44000 | Loss: 2.8990 | LR: 3.14e-07 | 3517.9ms/step | 5.7 samples/s
901
+ Step 43320/44000 | Loss: 2.8536 | LR: 3.05e-07 | 3514.4ms/step | 5.7 samples/s
902
+ Step 43330/44000 | Loss: 2.8889 | LR: 2.96e-07 | 3515.5ms/step | 5.7 samples/s
903
+ Step 43340/44000 | Loss: 2.8785 | LR: 2.87e-07 | 3514.0ms/step | 5.7 samples/s
904
+ Step 43350/44000 | Loss: 2.8903 | LR: 2.79e-07 | 3515.8ms/step | 5.7 samples/s
905
+ Step 43360/44000 | Loss: 2.8876 | LR: 2.70e-07 | 3514.5ms/step | 5.7 samples/s
906
+ Step 43370/44000 | Loss: 2.8766 | LR: 2.62e-07 | 3514.1ms/step | 5.7 samples/s
907
+ Step 43380/44000 | Loss: 2.8827 | LR: 2.53e-07 | 3513.4ms/step | 5.7 samples/s
908
+ Step 43390/44000 | Loss: 2.8985 | LR: 2.45e-07 | 3514.5ms/step | 5.7 samples/s
909
+ Step 43400/44000 | Loss: 2.8939 | LR: 2.37e-07 | 3508.2ms/step | 5.7 samples/s
910
+ Step 43410/44000 | Loss: 2.8678 | LR: 2.30e-07 | 3506.3ms/step | 5.7 samples/s
911
+ Step 43420/44000 | Loss: 2.8901 | LR: 2.22e-07 | 3512.5ms/step | 5.7 samples/s
912
+ Step 43430/44000 | Loss: 2.8821 | LR: 2.14e-07 | 3516.1ms/step | 5.7 samples/s
913
+ Step 43440/44000 | Loss: 2.8844 | LR: 2.07e-07 | 3513.8ms/step | 5.7 samples/s
914
+ Step 43450/44000 | Loss: 2.9107 | LR: 1.99e-07 | 3512.8ms/step | 5.7 samples/s
915
+ Step 43460/44000 | Loss: 2.8967 | LR: 1.92e-07 | 3512.7ms/step | 5.7 samples/s
916
+ Step 43470/44000 | Loss: 2.9141 | LR: 1.85e-07 | 3513.5ms/step | 5.7 samples/s
917
+ Step 43480/44000 | Loss: 2.8803 | LR: 1.78e-07 | 3515.2ms/step | 5.7 samples/s
918
+ Step 43490/44000 | Loss: 2.9032 | LR: 1.72e-07 | 3515.3ms/step | 5.7 samples/s
919
+ Step 43500/44000 | Loss: 2.8664 | LR: 1.65e-07 | 3517.6ms/step | 5.7 samples/s
920
+ Validation at step 43500: Loss = 2.8987, Perplexity = 18.15
921
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-43500
922
+ Step 43510/44000 | Loss: 2.8524 | LR: 1.58e-07 | 9711.7ms/step | 2.1 samples/s
923
+ Step 43520/44000 | Loss: 2.8712 | LR: 1.52e-07 | 3514.0ms/step | 5.7 samples/s
924
+ Step 43530/44000 | Loss: 2.9000 | LR: 1.46e-07 | 3516.1ms/step | 5.7 samples/s
925
+ Step 43540/44000 | Loss: 2.8935 | LR: 1.40e-07 | 3514.3ms/step | 5.7 samples/s
926
+ Step 43550/44000 | Loss: 2.8855 | LR: 1.34e-07 | 3513.3ms/step | 5.7 samples/s
927
+ Step 43560/44000 | Loss: 2.8831 | LR: 1.28e-07 | 3513.9ms/step | 5.7 samples/s
928
+ Step 43570/44000 | Loss: 2.9068 | LR: 1.22e-07 | 3513.5ms/step | 5.7 samples/s
929
+ Step 43580/44000 | Loss: 2.8858 | LR: 1.16e-07 | 3517.1ms/step | 5.7 samples/s
930
+ Step 43590/44000 | Loss: 2.8752 | LR: 1.11e-07 | 3517.0ms/step | 5.7 samples/s
931
+ Step 43600/44000 | Loss: 2.9085 | LR: 1.06e-07 | 3517.9ms/step | 5.7 samples/s
932
+ Step 43610/44000 | Loss: 2.9231 | LR: 1.00e-07 | 3510.9ms/step | 5.7 samples/s
933
+ Step 43620/44000 | Loss: 2.8641 | LR: 9.52e-08 | 3507.2ms/step | 5.7 samples/s
934
+ Step 43630/44000 | Loss: 2.9341 | LR: 9.03e-08 | 3510.3ms/step | 5.7 samples/s
935
+ Step 43640/44000 | Loss: 2.9064 | LR: 8.55e-08 | 3513.5ms/step | 5.7 samples/s
936
+ Step 43650/44000 | Loss: 2.9077 | LR: 8.08e-08 | 3514.8ms/step | 5.7 samples/s
937
+ Step 43660/44000 | Loss: 2.9009 | LR: 7.62e-08 | 3514.7ms/step | 5.7 samples/s
938
+ Step 43670/44000 | Loss: 2.8738 | LR: 7.18e-08 | 3513.2ms/step | 5.7 samples/s
939
+ Step 43680/44000 | Loss: 2.8983 | LR: 6.75e-08 | 3513.4ms/step | 5.7 samples/s
940
+ Step 43690/44000 | Loss: 2.9157 | LR: 6.34e-08 | 3515.9ms/step | 5.7 samples/s
941
+ Step 43700/44000 | Loss: 2.8954 | LR: 5.94e-08 | 3511.3ms/step | 5.7 samples/s
942
+ Step 43710/44000 | Loss: 2.8858 | LR: 5.55e-08 | 3514.3ms/step | 5.7 samples/s
943
+ Step 43720/44000 | Loss: 2.8859 | LR: 5.17e-08 | 3514.3ms/step | 5.7 samples/s
944
+ Step 43730/44000 | Loss: 2.9070 | LR: 4.81e-08 | 3514.4ms/step | 5.7 samples/s
945
+ Step 43740/44000 | Loss: 2.8854 | LR: 4.46e-08 | 3514.5ms/step | 5.7 samples/s
946
+ Step 43750/44000 | Loss: 2.9103 | LR: 4.12e-08 | 3515.3ms/step | 5.7 samples/s
947
+ Validation at step 43750: Loss = 2.8987, Perplexity = 18.15
948
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-43750
949
+ Step 43760/44000 | Loss: 2.8941 | LR: 3.80e-08 | 9709.9ms/step | 2.1 samples/s
950
+ Step 43770/44000 | Loss: 2.8727 | LR: 3.49e-08 | 3513.8ms/step | 5.7 samples/s
951
+ Step 43780/44000 | Loss: 2.8933 | LR: 3.19e-08 | 3512.0ms/step | 5.7 samples/s
952
+ Step 43790/44000 | Loss: 2.9165 | LR: 2.91e-08 | 3514.1ms/step | 5.7 samples/s
953
+ Step 43800/44000 | Loss: 2.8848 | LR: 2.64e-08 | 3511.5ms/step | 5.7 samples/s
954
+ Step 43810/44000 | Loss: 2.9084 | LR: 2.38e-08 | 3511.5ms/step | 5.7 samples/s
955
+ Step 43820/44000 | Loss: 2.8786 | LR: 2.14e-08 | 3507.5ms/step | 5.7 samples/s
956
+ Step 43830/44000 | Loss: 2.8909 | LR: 1.91e-08 | 3501.3ms/step | 5.7 samples/s
957
+ Step 43840/44000 | Loss: 2.8662 | LR: 1.69e-08 | 3507.4ms/step | 5.7 samples/s
958
+ Step 43850/44000 | Loss: 2.8758 | LR: 1.48e-08 | 3507.8ms/step | 5.7 samples/s
959
+ Step 43860/44000 | Loss: 2.8858 | LR: 1.29e-08 | 3507.0ms/step | 5.7 samples/s
960
+ Step 43870/44000 | Loss: 2.8849 | LR: 1.11e-08 | 3505.7ms/step | 5.7 samples/s
961
+ Step 43880/44000 | Loss: 2.8901 | LR: 9.50e-09 | 3508.1ms/step | 5.7 samples/s
962
+ Step 43890/44000 | Loss: 2.8700 | LR: 7.98e-09 | 3508.7ms/step | 5.7 samples/s
963
+ Step 43900/44000 | Loss: 2.8982 | LR: 6.60e-09 | 3508.9ms/step | 5.7 samples/s
964
+ Step 43910/44000 | Loss: 2.8926 | LR: 5.34e-09 | 3508.0ms/step | 5.7 samples/s
965
+ Step 43920/44000 | Loss: 2.8999 | LR: 4.22e-09 | 3508.2ms/step | 5.7 samples/s
966
+ Step 43930/44000 | Loss: 2.9067 | LR: 3.23e-09 | 3507.6ms/step | 5.7 samples/s
967
+ Step 43940/44000 | Loss: 2.8822 | LR: 2.37e-09 | 3506.8ms/step | 5.7 samples/s
968
+ Step 43950/44000 | Loss: 2.9038 | LR: 1.65e-09 | 3509.5ms/step | 5.7 samples/s
969
+ Step 43960/44000 | Loss: 2.8814 | LR: 1.06e-09 | 3507.6ms/step | 5.7 samples/s
970
+ Step 43970/44000 | Loss: 2.8940 | LR: 5.94e-10 | 3508.7ms/step | 5.7 samples/s
971
+ Step 43980/44000 | Loss: 2.9016 | LR: 2.64e-10 | 3506.6ms/step | 5.7 samples/s
972
+ Step 43990/44000 | Loss: 2.9058 | LR: 6.60e-11 | 3507.9ms/step | 5.7 samples/s
973
+ Step 44000/44000 | Loss: 2.9120 | LR: 0.00e+00 | 3509.2ms/step | 5.7 samples/s
974
+ Validation at step 44000: Loss = 2.8987, Perplexity = 18.15
975
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-44000
976
+ Saved checkpoint to /mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031/checkpoint-44000
epfl_final/SparseAtt/wandb/run-20260731_001110-w1uk4jj8/files/requirements.txt ADDED
@@ -0,0 +1,142 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ pip==26.0.1
2
+ nvidia-cusparselt-cu13==0.8.0
3
+ mpmath==1.3.0
4
+ cuda-toolkit==13.0.2
5
+ zstandard==0.25.0
6
+ xxhash==3.7.0
7
+ urllib3==2.6.3
8
+ typing_extensions==4.15.0
9
+ triton==3.6.0
10
+ tqdm==4.67.3
11
+ torchinfo==1.8.0
12
+ sympy==1.14.0
13
+ smmap==5.0.3
14
+ six==1.17.0
15
+ shellingham==1.5.4
16
+ setuptools==81.0.0
17
+ safetensors==0.7.0
18
+ regex==2026.4.4
19
+ PyYAML==6.0.3
20
+ Pygments==2.20.0
21
+ pyarrow==24.0.0
22
+ psutil==7.2.2
23
+ protobuf==7.34.1
24
+ propcache==0.4.1
25
+ platformdirs==4.9.6
26
+ packaging==26.2
27
+ nvidia-nvtx==13.0.85
28
+ nvidia-nvshmem-cu13==3.4.5
29
+ nvidia-nvjitlink==13.0.88
30
+ nvidia-nccl-cu13==2.28.9
31
+ nvidia-curand==10.4.0.35
32
+ nvidia-cufile==1.15.1.6
33
+ nvidia-cuda-runtime==13.0.96
34
+ nvidia-cuda-nvrtc==13.0.88
35
+ nvidia-cuda-cupti==13.0.85
36
+ nvidia-cublas==13.1.0.3
37
+ numpy==2.4.4
38
+ networkx==3.6.1
39
+ multidict==6.7.1
40
+ mdurl==0.1.2
41
+ MarkupSafe==3.0.3
42
+ idna==3.13
43
+ hf-xet==1.4.3
44
+ h11==0.16.0
45
+ fsspec==2026.2.0
46
+ frozenlist==1.8.0
47
+ filelock==3.29.0
48
+ dill==0.4.1
49
+ cuda-pathfinder==1.5.4
50
+ click==8.3.3
51
+ charset-normalizer==3.4.7
52
+ certifi==2026.4.22
53
+ attrs==26.1.0
54
+ annotated-types==0.7.0
55
+ annotated-doc==0.0.4
56
+ aiohappyeyeballs==2.6.1
57
+ yarl==1.23.0
58
+ typing-inspection==0.4.2
59
+ sentry-sdk==2.58.0
60
+ requests==2.33.1
61
+ python-dateutil==2.9.0.post0
62
+ pydantic_core==2.46.3
63
+ nvidia-cusparse==12.6.3.3
64
+ nvidia-cufft==12.0.0.61
65
+ nvidia-cudnn-cu13==9.19.0.56
66
+ multiprocess==0.70.19
67
+ markdown-it-py==4.0.0
68
+ Jinja2==3.1.6
69
+ httpcore==1.0.9
70
+ gitdb==4.0.12
71
+ cuda-bindings==13.2.0
72
+ anyio==4.13.0
73
+ aiosignal==1.4.0
74
+ rich==15.0.0
75
+ pydantic==2.13.3
76
+ pandas==3.0.2
77
+ nvidia-cusolver==12.0.4.66
78
+ httpx==0.28.1
79
+ GitPython==3.1.49
80
+ aiohttp==3.13.5
81
+ wandb==0.26.1
82
+ typer==0.25.1
83
+ torch==2.11.0
84
+ huggingface_hub==1.13.0
85
+ tokenizers==0.22.2
86
+ datasets==4.8.5
87
+ accelerate==1.13.0
88
+ transformers==5.7.0
89
+ word2number==1.1
90
+ sqlitedict==2.1.0
91
+ pytz==2026.2
92
+ threadpoolctl==3.6.0
93
+ tcolorpy==0.1.7
94
+ tabulate==0.10.0
95
+ scipy==1.17.1
96
+ portalocker==3.2.0
97
+ pathvalidate==3.3.1
98
+ narwhals==2.22.1
99
+ more-itertools==11.1.0
100
+ lxml==6.1.1
101
+ joblib==1.5.3
102
+ colorama==0.4.6
103
+ chardet==6.0.0.post1
104
+ absl-py==2.4.0
105
+ scikit-learn==1.9.0
106
+ sacrebleu==2.6.0
107
+ nltk==3.9.4
108
+ mbstrdecoder==1.1.5
109
+ typepy==1.3.5
110
+ rouge_score==0.1.2
111
+ DataProperty==1.1.1
112
+ tabledata==1.3.5
113
+ evaluate==0.4.6
114
+ pytablewriter==1.2.1
115
+ lm_eval==0.4.12
116
+ pluggy==1.6.0
117
+ iniconfig==2.3.0
118
+ pytest==9.1.0
119
+ py-spy==0.4.2
120
+ pillow==12.2.0
121
+ tifffile==2026.6.1
122
+ imagecodecs==2026.6.26
123
+ zipp==4.1.0
124
+ importlib_metadata==9.0.0
125
+ derivative==0.6.3
126
+ pysindy==2.1.0
127
+ pyparsing==3.3.2
128
+ kiwisolver==1.5.0
129
+ fonttools==4.63.0
130
+ cycler==0.12.1
131
+ contourpy==1.3.3
132
+ matplotlib==3.11.0
133
+ sortedcontainers==2.4.0
134
+ wcwidth==0.8.2
135
+ sigfig==1.3.19
136
+ importlib_resources==7.1.0
137
+ enum-tools==0.13.0
138
+ cramjam==2.11.0
139
+ python-snappy==0.7.3
140
+ compact-json==1.8.2
141
+ numbers-parser==4.18.5
142
+ entmax==1.3
epfl_final/SparseAtt/wandb/run-20260731_001110-w1uk4jj8/files/wandb-metadata.json ADDED
@@ -0,0 +1,65 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "os": "Linux-6.8.0-124-generic-x86_64-with-glibc2.35",
3
+ "python": "CPython 3.13.13",
4
+ "startedAt": "2026-07-31T00:11:10.199782Z",
5
+ "args": [
6
+ "--config",
7
+ "/mloscratch/homes/navasard/sparse/SparseAtt/configs/ladder8k/365m_mixed_to_dense_20pct_seed45_20260730.json"
8
+ ],
9
+ "program": "/mloscratch/homes/navasard/sparse/SparseAtt/train_dense_to_mixed_exact.py",
10
+ "codePath": "train_dense_to_mixed_exact.py",
11
+ "codePathLocal": "train_dense_to_mixed_exact.py",
12
+ "git": {
13
+ "remote": "https://github.com/rubentium/TokenPrune.git",
14
+ "commit": "88298c7a418622530ca5426b22b1e5ebd10ca0cb"
15
+ },
16
+ "email": "ruben.navasardyan@mail.utoronto.ca",
17
+ "root": "/mloscratch/homes/navasard/sparse/SparseAtt",
18
+ "host": "a100-train-4-gpu-2-0-40",
19
+ "executable": "/mloscratch/homes/navasard/conda/bin/python3.13",
20
+ "cpu_count": 64,
21
+ "cpu_count_logical": 128,
22
+ "gpu": "NVIDIA A100-SXM4-80GB",
23
+ "gpu_count": 4,
24
+ "disk": {
25
+ "/": {
26
+ "total": "7679362727936",
27
+ "used": "5257403871232"
28
+ }
29
+ },
30
+ "memory": {
31
+ "total": "1081884934144"
32
+ },
33
+ "gpu_nvidia": [
34
+ {
35
+ "name": "NVIDIA A100-SXM4-80GB",
36
+ "memoryTotal": "85899345920",
37
+ "cudaCores": 6912,
38
+ "architecture": "Ampere",
39
+ "uuid": "GPU-26e055ad-009a-c42a-2ba0-ce6db370fcbe"
40
+ },
41
+ {
42
+ "name": "NVIDIA A100-SXM4-80GB",
43
+ "memoryTotal": "85899345920",
44
+ "cudaCores": 6912,
45
+ "architecture": "Ampere",
46
+ "uuid": "GPU-56ec1834-3584-3932-0125-388b5e88fac2"
47
+ },
48
+ {
49
+ "name": "NVIDIA A100-SXM4-80GB",
50
+ "memoryTotal": "85899345920",
51
+ "cudaCores": 6912,
52
+ "architecture": "Ampere",
53
+ "uuid": "GPU-ce50edb9-4d69-8d20-4433-5217084c7016"
54
+ },
55
+ {
56
+ "name": "NVIDIA A100-SXM4-80GB",
57
+ "memoryTotal": "85899345920",
58
+ "cudaCores": 6912,
59
+ "architecture": "Ampere",
60
+ "uuid": "GPU-2401254a-acb7-5787-0b06-07065349e511"
61
+ }
62
+ ],
63
+ "cudaVersion": "13.0",
64
+ "writerId": "l2bnjgzsfkl6v3e40bxm38p36dv28k34"
65
+ }
epfl_final/SparseAtt/wandb/run-20260731_001110-w1uk4jj8/files/wandb-summary.json ADDED
@@ -0,0 +1 @@
 
 
1
+ {"speed/steps_per_sec":0.28496148597309795,"eval/loss":2.8986784529685976,"eval/dense_interval_start":-1,"_wandb":{"runtime":34289},"train/epoch":0.2906439083679023,"eval/dense_interval_end":-1,"train/learning_rate":0,"train/loss":2.9119627356529234,"eval/dense_interval_active":0,"speed/ms_per_step":3509.2461585998535,"eval/perplexity":18.150143245043203,"_timestamp":1.7854909540886652e+09,"eval/uses_sparse_attention":0,"eval/uses_dense_attention":1,"_step":44000,"_runtime":34289.773974227,"train/global_step":44000,"speed/samples_per_sec":5.699229719461959}
epfl_final/SparseAtt/wandb/run-20260731_001110-w1uk4jj8/logs/debug-internal.log ADDED
The diff for this file is too large to render. See raw diff
 
epfl_final/SparseAtt/wandb/run-20260731_001110-w1uk4jj8/logs/debug.log ADDED
@@ -0,0 +1,24 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 2026-07-31 00:11:10,216 INFO MainThread:160 [wandb_setup.py:_flush():81] Current SDK version is 0.26.1
2
+ 2026-07-31 00:11:10,216 INFO MainThread:160 [wandb_setup.py:_flush():81] Configure stats pid to 160
3
+ 2026-07-31 00:11:10,216 INFO MainThread:160 [wandb_setup.py:_flush():81] Loading settings from environment variables
4
+ 2026-07-31 00:11:10,216 INFO MainThread:160 [wandb_init.py:setup_run_log_directory():723] Logging user logs to /mloscratch/homes/navasard/sparse/SparseAtt/wandb/run-20260731_001110-w1uk4jj8/logs/debug.log
5
+ 2026-07-31 00:11:10,216 INFO MainThread:160 [wandb_init.py:setup_run_log_directory():724] Logging internal logs to /mloscratch/homes/navasard/sparse/SparseAtt/wandb/run-20260731_001110-w1uk4jj8/logs/debug-internal.log
6
+ 2026-07-31 00:11:10,217 INFO MainThread:160 [wandb_init.py:init():850] calling init triggers
7
+ 2026-07-31 00:11:10,217 INFO MainThread:160 [wandb_init.py:init():855] wandb.init called with sweep_config: {}
8
+ config: {'model_path': '/mloscratch/homes/navasard/qwen3_06b_base', 'model_dtype': 'bfloat16', 'init_from_scratch': True, 'train_path': '/mloscratch/homes/navasard/data/fineweb_large/tokenized', 'val_path': None, 'max_seq_length': 8192, 'num_workers': 8, 'output_dir': '/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_to_dense_20pct_seed45/run_20260731_001031', 'resume_from_checkpoint': '/mloscratch/homes/navasard/sparse/checkpoints/l8k365m_mixed_seed45/run_20260722_223953/checkpoint-35000', 'force_new_run': True, 'num_epochs': 1, 'max_steps': 44000, 'per_device_batch_size': 2, 'gradient_accumulation_steps': 10, 'learning_rate': 0.0005, 'weight_decay': 0.01, 'warmup_steps': 750, 'lr_scheduler': 'cosine', 'max_grad_norm': 1.0, 'save_steps': 250, 'eval_steps': 250, 'eval_batches': 50, 'logging_steps': 10, 'seed': 45, 'verbose': False, 'checkpoint_activations': False, 'strategy': 'ddp', 'mixed_precision': 'bf16', 'wandb_enabled': True, 'wandb_project': 'stochastic_sparsity', 'wandb_name': '365m mixed-to-dense 20pct seed45', 'wandb_tags': ['qwen3', '365m', '8k', 'mixed-to-dense', '20pct', 'seed45', 'multiseed'], 'hidden_size': 768, 'intermediate_size': 3072, 'num_attention_heads': 6, 'num_key_value_heads': 2, 'num_hidden_layers': 28, 'head_dim': None, 'attention_type': 'sparse_stochastic', 'sparse_block_size': 128, 'sparse_num_random_blocks': 1, 'sparse_mask_pool_size': 1, 'sparse_val_dense_compare': False, 'sparse_layer_indices': None, 'sparse_mask_cache_dir': None, 'sparse_mask_seed': 0, 'dense_mode': True, 'dense_step_intervals': None, '_wandb': {}}
9
+ 2026-07-31 00:11:10,217 INFO MainThread:160 [wandb_init.py:init():898] starting backend
10
+ 2026-07-31 00:11:10,653 INFO MainThread:160 [wandb_init.py:init():913] sending inform_init request
11
+ 2026-07-31 00:11:11,166 INFO MainThread:160 [wandb_init.py:init():918] backend started and connected
12
+ 2026-07-31 00:11:11,170 INFO MainThread:160 [wandb_init.py:init():988] updated telemetry
13
+ 2026-07-31 00:11:11,188 INFO MainThread:160 [wandb_init.py:init():1011] communicating run to backend with 90.0 second timeout
14
+ 2026-07-31 00:11:11,669 INFO MainThread:160 [wandb_init.py:init():1056] starting run threads in backend
15
+ 2026-07-31 00:11:11,869 INFO MainThread:160 [wandb_run.py:_console_start():2554] atexit reg
16
+ 2026-07-31 00:11:11,869 INFO MainThread:160 [wandb_run.py:_redirect():2403] redirect: wrap_raw
17
+ 2026-07-31 00:11:11,869 INFO MainThread:160 [wandb_run.py:_redirect():2472] Wrapping output streams.
18
+ 2026-07-31 00:11:11,869 INFO MainThread:160 [wandb_run.py:_redirect():2495] Redirects installed.
19
+ 2026-07-31 00:11:11,876 INFO MainThread:160 [wandb_init.py:init():1094] run started, returning control to user process
20
+ 2026-07-31 09:42:41,442 INFO MainThread:160 [wandb_run.py:_finish():2321] finishing run ruben-navasardyan4869-epfl/stochastic_sparsity/w1uk4jj8
21
+ 2026-07-31 09:42:41,442 INFO MainThread:160 [wandb_run.py:_atexit_cleanup():2520] got exitcode: 0
22
+ 2026-07-31 09:42:41,443 INFO MainThread:160 [wandb_run.py:_restore():2502] restore
23
+ 2026-07-31 09:42:41,443 INFO MainThread:160 [wandb_run.py:_restore():2508] restore done
24
+ 2026-07-31 09:42:43,250 INFO MainThread:160 [wandb_run.py:_footer_sync_info():3889] logging synced files
epfl_final/SparseAtt/wandb/run-20260731_001110-w1uk4jj8/run-w1uk4jj8.wandb ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2744524201909a8e97804d3fbc4004968303c1f1e7be4f3939e4b5be14af2f43
3
+ size 6142002