Hugging Face's logo

keepsloading
/
icml_repro_scratch

Text Generation
Transformers
Safetensors
English
qwen3
long-context
sparse-attention
aha
l2a-style
reproducibility
conversational
text-generation-inference
Model card Files
xet
 Community
keepsloading commited on
Commit
96c0250
·
verified ·
1 Parent(s): a5997a1

Upload folder using huggingface_hub

Browse files
Files changed (2) hide show
  1. .gitattributes +3 -0
  2. outputs/eval/logs/vanilla/ruler_a.log +72 -72
.gitattributes CHANGED
@@ -2,3 +2,6 @@
2
  *.arrow filter=lfs diff=lfs merge=lfs -text
3
  *.jsonl filter=lfs diff=lfs merge=lfs -text
4
  tokenizer.json filter=lfs diff=lfs merge=lfs -text
 
 
 
 
2
  *.arrow filter=lfs diff=lfs merge=lfs -text
3
  *.jsonl filter=lfs diff=lfs merge=lfs -text
4
  tokenizer.json filter=lfs diff=lfs merge=lfs -text
5
+ outputs/l2a_style/stage2/checkpoint-25/optimizer.pt filter=lfs diff=lfs merge=lfs -text
6
+ outputs/l2a_style/stage2/checkpoint-50/optimizer.pt filter=lfs diff=lfs merge=lfs -text
7
+ outputs/l2a_style/stage2/checkpoint-75/optimizer.pt filter=lfs diff=lfs merge=lfs -text
outputs/eval/logs/vanilla/ruler_a.log CHANGED
@@ -1,70 +1,70 @@
1
- 2026-07-18:10:14:08 WARNING [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
2
- 2026-07-18:10:14:12 INFO [_cli.run:376] Selected Tasks: ['niah_single_1', 'niah_single_3', 'niah_multikey_2', 'niah_multiquery', 'ruler_vt', 'ruler_fwe', 'ruler_qa_hotpot']
3
- 2026-07-18:10:14:13 INFO [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
4
- 2026-07-18:10:14:13 INFO [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
5
- 2026-07-18:10:14:16 INFO [models.huggingface:161] Using device 'cuda:0'
6
- 2026-07-18:10:14:16 INFO [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
7
- 2026-07-18:10:14:26 WARNING [api.task:856] niah_single_1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
8
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
9
- 2026-07-18:10:14:26 INFO [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
10
 
11
 
12
-
13
-
14
-
15
-
16
-
17
-
18
-
19
-
20
- 2026-07-18:10:14:35 WARNING [api.task:856] niah_single_3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
21
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
22
 
23
-
24
 
25
-
26
 
27
 
28
-
29
-
30
-
31
-
32
-
33
-
34
-
35
-
36
- 2026-07-18:10:14:46 WARNING [api.task:856] niah_multikey_2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
37
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
38
 
39
 
40
-
41
-
42
-
43
-
44
-
45
-
46
-
47
- 2026-07-18:10:14:53 WARNING [api.task:856] niah_multiquery: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
48
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
49
 
50
 
51
-
52
-
53
-
54
-
55
-
56
-
57
-
58
-
59
- 2026-07-18:10:15:02 WARNING [api.task:856] ruler_vt: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
60
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
61
- 2026-07-18:10:15:02 INFO [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
62
  Max length 500 | Current length 307 | Noises: 5
63
  Max length 500 | Current length 410 | Noises: 10
64
  Max length 500 | Current length 531 | Noises: 15
65
  Num noises: 10
66
 
67
-
68
  0%| | 0/1 [00:00<?, ?it/s]
 
69
  0%| | 0/1 [00:00<?, ?it/s]
70
  Max length 8192 | Current length 791 | Noises: 10
71
  Max length 8192 | Current length 1025 | Noises: 20
72
  Max length 8192 | Current length 1273 | Noises: 30
@@ -100,31 +100,31 @@ Max length 8192 | Current length 8230 | Noises: 320
100
  Num noises: 310
101
 
102
 
103
  0%| | 0/500 [00:00<?, ?it/s]
104
-
105
  12%|█▏ | 60/500 [00:01<00:07, 59.90it/s]
106
-
107
  24%|██▍ | 120/500 [00:02<00:06, 59.31it/s]
108
-
109
  36%|███▌ | 181/500 [00:03<00:05, 59.99it/s]
110
-
111
  49%|████▊ | 243/500 [00:04<00:04, 60.56it/s]
112
-
113
  61%|██████ | 304/500 [00:05<00:03, 59.89it/s]
114
-
115
  73%|███████▎ | 364/500 [00:06<00:02, 59.63it/s]
116
-
117
  85%|████████▍ | 424/500 [00:07<00:01, 59.16it/s]
118
-
119
  97%|█████████▋| 485/500 [00:08<00:00, 59.51it/s]
120
- 2026-07-18:10:15:12 WARNING [api.task:856] ruler_fwe: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
121
  12%|█▏ | 61/500 [00:01<00:07, 60.88it/s]
 
122
  24%|██▍ | 122/500 [00:02<00:06, 60.00it/s]
 
123
  37%|███▋ | 183/500 [00:03<00:05, 59.21it/s]
 
124
  49%|████▊ | 243/500 [00:04<00:04, 59.20it/s]
 
125
  61%|██████ | 303/500 [00:05<00:03, 59.13it/s]
 
126
  73%|███████▎ | 363/500 [00:06<00:02, 58.72it/s]
 
127
  85%|████████▍ | 423/500 [00:07<00:01, 59.06it/s]
 
128
  97%|█████████▋| 483/500 [00:08<00:00, 58.83it/s]
 
129
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
130
 
131
 
132
-
133
-
134
-
135
-
136
-
137
-
138
-
139
-
140
-
141
-
142
-
143
-
144
- 2026-07-18:10:15:25 WARNING [api.task:856] ruler_qa_hotpot: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
145
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
146
  Traceback (most recent call last):
147
  File "/opt/conda/lib/python3.11/site-packages/urllib3/connection.py", line 198, in _new_conn
@@ -157,7 +157,7 @@ Traceback (most recent call last):
157
  ^^^^^^^^^^^^^^^^
158
  File "/opt/conda/lib/python3.11/site-packages/urllib3/connection.py", line 207, in _new_conn
159
  raise ConnectTimeoutError(
160
- urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f91c0eaee90>, 'Connection to curtis.ml.cmu.edu timed out. (connect timeout=None)')
161
 
162
  The above exception was the direct cause of the following exception:
163
 
@@ -171,7 +171,7 @@ Traceback (most recent call last):
171
  File "/opt/conda/lib/python3.11/site-packages/urllib3/util/retry.py", line 519, in increment
172
  raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
173
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
174
- urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='curtis.ml.cmu.edu', port=80): Max retries exceeded with url: /datasets/hotpot/hotpot_dev_distractor_v1.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f91c0eaee90>, 'Connection to curtis.ml.cmu.edu timed out. (connect timeout=None)'))
175
 
176
  During handling of the above exception, another exception occurred:
177
 
@@ -238,4 +238,4 @@ Traceback (most recent call last):
238
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
239
  File "/opt/conda/lib/python3.11/site-packages/requests/adapters.py", line 665, in send
240
  raise ConnectTimeout(e, request=request)
241
- requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='curtis.ml.cmu.edu', port=80): Max retries exceeded with url: /datasets/hotpot/hotpot_dev_distractor_v1.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f91c0eaee90>, 'Connection to curtis.ml.cmu.edu timed out. (connect timeout=None)'))
 
1
+ 2026-07-18:10:23:30 WARNING [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
2
+ 2026-07-18:10:23:34 INFO [_cli.run:376] Selected Tasks: ['niah_single_1', 'niah_single_3', 'niah_multikey_2', 'niah_multiquery', 'ruler_vt', 'ruler_fwe', 'ruler_qa_hotpot']
3
+ 2026-07-18:10:23:35 INFO [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
4
+ 2026-07-18:10:23:35 INFO [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
5
+ 2026-07-18:10:23:38 INFO [models.huggingface:161] Using device 'cuda:0'
6
+ 2026-07-18:10:23:38 INFO [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
7
+ 2026-07-18:10:23:48 WARNING [api.task:856] niah_single_1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
8
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
9
+ 2026-07-18:10:23:48 INFO [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
10
 
11
 
12
+
13
+
14
+
15
+
16
+
17
+
18
+
19
+
20
+ 2026-07-18:10:23:57 WARNING [api.task:856] niah_single_3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
21
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
22
 
23
+
24
 
25
+
26
 
27
 
28
+
29
+
30
+
31
+
32
+
33
+
34
+
35
+
36
+ 2026-07-18:10:24:07 WARNING [api.task:856] niah_multikey_2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
37
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
38
 
39
 
40
+
41
+
42
+
43
+
44
+
45
+
46
+
47
+ 2026-07-18:10:24:14 WARNING [api.task:856] niah_multiquery: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
48
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
49
 
50
 
51
+
52
+
53
+
54
+
55
+
56
+
57
+
58
+
59
+ 2026-07-18:10:24:24 WARNING [api.task:856] ruler_vt: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
60
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
61
+ 2026-07-18:10:24:24 INFO [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
62
  Max length 500 | Current length 307 | Noises: 5
63
  Max length 500 | Current length 410 | Noises: 10
64
  Max length 500 | Current length 531 | Noises: 15
65
  Num noises: 10
66
 
 
67
  0%| | 0/1 [00:00<?, ?it/s]
68
+
69
  0%| | 0/1 [00:00<?, ?it/s]
70
  Max length 8192 | Current length 791 | Noises: 10
71
  Max length 8192 | Current length 1025 | Noises: 20
72
  Max length 8192 | Current length 1273 | Noises: 30
 
100
  Num noises: 310
101
 
102
 
103
  0%| | 0/500 [00:00<?, ?it/s]
 
104
  12%|█▏ | 60/500 [00:01<00:07, 59.90it/s]
 
105
  24%|██▍ | 120/500 [00:02<00:06, 59.31it/s]
 
106
  36%|███▌ | 181/500 [00:03<00:05, 59.99it/s]
 
107
  49%|████▊ | 243/500 [00:04<00:04, 60.56it/s]
 
108
  61%|██████ | 304/500 [00:05<00:03, 59.89it/s]
 
109
  73%|███████▎ | 364/500 [00:06<00:02, 59.63it/s]
 
110
  85%|████████▍ | 424/500 [00:07<00:01, 59.16it/s]
 
111
  97%|█████████▋| 485/500 [00:08<00:00, 59.51it/s]
112
+
113
  12%|█▏ | 61/500 [00:01<00:07, 60.88it/s]
114
+
115
  24%|██▍ | 122/500 [00:02<00:06, 60.00it/s]
116
+
117
  37%|███▋ | 183/500 [00:03<00:05, 59.21it/s]
118
+
119
  49%|████▊ | 243/500 [00:04<00:04, 59.20it/s]
120
+
121
  61%|██████ | 303/500 [00:05<00:03, 59.13it/s]
122
+
123
  73%|███████▎ | 363/500 [00:06<00:02, 58.72it/s]
124
+
125
  85%|████████▍ | 423/500 [00:07<00:01, 59.06it/s]
126
+
127
  97%|█████████▋| 483/500 [00:08<00:00, 58.83it/s]
128
+ 2026-07-18:10:24:33 WARNING [api.task:856] ruler_fwe: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
129
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
130
 
131
 
132
+
133
+
134
+
135
+
136
+
137
+
138
+
139
+
140
+
141
+
142
+
143
+
144
+ 2026-07-18:10:24:46 WARNING [api.task:856] ruler_qa_hotpot: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
145
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
146
  Traceback (most recent call last):
147
  File "/opt/conda/lib/python3.11/site-packages/urllib3/connection.py", line 198, in _new_conn
 
157
  ^^^^^^^^^^^^^^^^
158
  File "/opt/conda/lib/python3.11/site-packages/urllib3/connection.py", line 207, in _new_conn
159
  raise ConnectTimeoutError(
160
+ urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f3b9aabcbd0>, 'Connection to curtis.ml.cmu.edu timed out. (connect timeout=None)')
161
 
162
  The above exception was the direct cause of the following exception:
163
 
 
171
  File "/opt/conda/lib/python3.11/site-packages/urllib3/util/retry.py", line 519, in increment
172
  raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
173
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
174
+ urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='curtis.ml.cmu.edu', port=80): Max retries exceeded with url: /datasets/hotpot/hotpot_dev_distractor_v1.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f3b9aabcbd0>, 'Connection to curtis.ml.cmu.edu timed out. (connect timeout=None)'))
175
 
176
  During handling of the above exception, another exception occurred:
177
 
 
238
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
239
  File "/opt/conda/lib/python3.11/site-packages/requests/adapters.py", line 665, in send
240
  raise ConnectTimeout(e, request=request)
241
+ requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='curtis.ml.cmu.edu', port=80): Max retries exceeded with url: /datasets/hotpot/hotpot_dev_distractor_v1.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f3b9aabcbd0>, 'Connection to curtis.ml.cmu.edu timed out. (connect timeout=None)'))