Hugging Face's logo

keepsloading
/
icml_repro_scratch

Text Generation
Transformers
Safetensors
English
qwen3
long-context
sparse-attention
aha
l2a-style
reproducibility
conversational
text-generation-inference
Model card Files
xet
 Community
keepsloading commited on
Commit
32eaeae
·
verified ·
1 Parent(s): 6b0b5b4

Upload folder using huggingface_hub

Browse files
Files changed (2) hide show
  1. .gitattributes +3 -0
  2. outputs/eval/logs/vanilla/ruler_a.log +73 -72
.gitattributes CHANGED
@@ -2,3 +2,6 @@
2
  *.arrow filter=lfs diff=lfs merge=lfs -text
3
  *.jsonl filter=lfs diff=lfs merge=lfs -text
4
  tokenizer.json filter=lfs diff=lfs merge=lfs -text
 
 
 
 
2
  *.arrow filter=lfs diff=lfs merge=lfs -text
3
  *.jsonl filter=lfs diff=lfs merge=lfs -text
4
  tokenizer.json filter=lfs diff=lfs merge=lfs -text
5
+ outputs/l2a_style/stage2/checkpoint-25/optimizer.pt filter=lfs diff=lfs merge=lfs -text
6
+ outputs/l2a_style/stage2/checkpoint-50/optimizer.pt filter=lfs diff=lfs merge=lfs -text
7
+ outputs/l2a_style/stage2/checkpoint-75/optimizer.pt filter=lfs diff=lfs merge=lfs -text
outputs/eval/logs/vanilla/ruler_a.log CHANGED
@@ -1,70 +1,71 @@
1
- 2026-07-18:10:23:30 WARNING [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
2
- 2026-07-18:10:23:34 INFO [_cli.run:376] Selected Tasks: ['niah_single_1', 'niah_single_3', 'niah_multikey_2', 'niah_multiquery', 'ruler_vt', 'ruler_fwe', 'ruler_qa_hotpot']
3
- 2026-07-18:10:23:35 INFO [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
4
- 2026-07-18:10:23:35 INFO [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
5
- 2026-07-18:10:23:38 INFO [models.huggingface:161] Using device 'cuda:0'
6
- 2026-07-18:10:23:38 INFO [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
7
- 2026-07-18:10:23:48 WARNING [api.task:856] niah_single_1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
8
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
9
- 2026-07-18:10:23:48 INFO [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
10
 
11
 
12
-
13
-
14
-
15
-
16
-
17
-
18
-
19
-
20
- 2026-07-18:10:23:57 WARNING [api.task:856] niah_single_3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
21
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
22
 
23
-
24
 
25
-
26
 
27
 
28
-
29
-
30
-
31
-
32
-
33
-
34
-
35
-
36
- 2026-07-18:10:24:07 WARNING [api.task:856] niah_multikey_2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
37
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
38
 
39
 
40
-
41
-
42
-
43
-
44
-
45
-
46
-
47
- 2026-07-18:10:24:14 WARNING [api.task:856] niah_multiquery: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
48
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
49
 
50
 
51
-
52
-
53
-
54
-
55
-
56
-
57
-
58
-
59
- 2026-07-18:10:24:24 WARNING [api.task:856] ruler_vt: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
 
60
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
61
- 2026-07-18:10:24:24 INFO [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
62
  Max length 500 | Current length 307 | Noises: 5
63
  Max length 500 | Current length 410 | Noises: 10
64
  Max length 500 | Current length 531 | Noises: 15
65
  Num noises: 10
66
 
67
-
68
  0%| | 0/1 [00:00<?, ?it/s]
 
69
  0%| | 0/1 [00:00<?, ?it/s]
70
  Max length 8192 | Current length 791 | Noises: 10
71
  Max length 8192 | Current length 1025 | Noises: 20
72
  Max length 8192 | Current length 1273 | Noises: 30
@@ -100,31 +101,31 @@ Max length 8192 | Current length 8230 | Noises: 320
100
  Num noises: 310
101
 
102
 
103
  0%| | 0/500 [00:00<?, ?it/s]
104
-
105
  12%|█▏ | 61/500 [00:01<00:07, 60.88it/s]
106
-
107
  24%|██▍ | 122/500 [00:02<00:06, 60.00it/s]
108
-
109
  37%|███▋ | 183/500 [00:03<00:05, 59.21it/s]
110
-
111
  49%|████▊ | 243/500 [00:04<00:04, 59.20it/s]
112
-
113
  61%|██████ | 303/500 [00:05<00:03, 59.13it/s]
114
-
115
  73%|███████▎ | 363/500 [00:06<00:02, 58.72it/s]
116
-
117
  85%|████████▍ | 423/500 [00:07<00:01, 59.06it/s]
118
-
119
  97%|█████████▋| 483/500 [00:08<00:00, 58.83it/s]
120
- 2026-07-18:10:24:33 WARNING [api.task:856] ruler_fwe: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
121
  12%|█▏ | 60/500 [00:01<00:07, 59.22it/s]
 
122
  24%|██▍ | 120/500 [00:02<00:06, 57.62it/s]
 
123
  36%|███▌ | 178/500 [00:03<00:05, 57.45it/s]
 
124
  47%|████▋ | 237/500 [00:04<00:04, 57.69it/s]
 
125
  59%|█████▉ | 295/500 [00:05<00:03, 56.89it/s]
 
126
  70%|███████ | 352/500 [00:06<00:02, 56.14it/s]
 
127
  82%|████████▏ | 409/500 [00:07<00:01, 56.31it/s]
 
128
  94%|█████████▎| 468/500 [00:08<00:00, 56.85it/s]
 
129
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
130
 
131
 
132
-
133
-
134
-
135
-
136
-
137
-
138
-
139
-
140
-
141
-
142
-
143
-
144
- 2026-07-18:10:24:46 WARNING [api.task:856] ruler_qa_hotpot: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
145
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
146
  Traceback (most recent call last):
147
  File "/opt/conda/lib/python3.11/site-packages/urllib3/connection.py", line 198, in _new_conn
@@ -157,7 +158,7 @@ Traceback (most recent call last):
157
  ^^^^^^^^^^^^^^^^
158
  File "/opt/conda/lib/python3.11/site-packages/urllib3/connection.py", line 207, in _new_conn
159
  raise ConnectTimeoutError(
160
- urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f3b9aabcbd0>, 'Connection to curtis.ml.cmu.edu timed out. (connect timeout=None)')
161
 
162
  The above exception was the direct cause of the following exception:
163
 
@@ -171,7 +172,7 @@ Traceback (most recent call last):
171
  File "/opt/conda/lib/python3.11/site-packages/urllib3/util/retry.py", line 519, in increment
172
  raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
173
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
174
- urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='curtis.ml.cmu.edu', port=80): Max retries exceeded with url: /datasets/hotpot/hotpot_dev_distractor_v1.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f3b9aabcbd0>, 'Connection to curtis.ml.cmu.edu timed out. (connect timeout=None)'))
175
 
176
  During handling of the above exception, another exception occurred:
177
 
@@ -238,4 +239,4 @@ Traceback (most recent call last):
238
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
239
  File "/opt/conda/lib/python3.11/site-packages/requests/adapters.py", line 665, in send
240
  raise ConnectTimeout(e, request=request)
241
- requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='curtis.ml.cmu.edu', port=80): Max retries exceeded with url: /datasets/hotpot/hotpot_dev_distractor_v1.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f3b9aabcbd0>, 'Connection to curtis.ml.cmu.edu timed out. (connect timeout=None)'))
 
1
+ 2026-07-18:10:33:06 WARNING [config.evaluate_config:281] --limit SHOULD ONLY BE USED FOR TESTING. REAL METRICS SHOULD NOT BE COMPUTED USING LIMIT.
2
+ 2026-07-18:10:33:10 INFO [_cli.run:376] Selected Tasks: ['niah_single_1', 'niah_single_3', 'niah_multikey_2', 'niah_multiquery', 'ruler_vt', 'ruler_fwe', 'ruler_qa_hotpot']
3
+ 2026-07-18:10:33:11 INFO [evaluator:211] Setting random seed to 0 | Setting numpy seed to 1234 | Setting torch manual seed to 1234 | Setting fewshot manual seed to 1234
4
+ 2026-07-18:10:33:11 INFO [evaluator:236] Initializing hf model, with arguments: {'pretrained': '/workspace', 'trust_remote_code': True, 'dtype': 'bfloat16', 'max_length': 16384, 'attn_implementation': 'sdpa'}
5
+ 2026-07-18:10:33:14 INFO [models.huggingface:161] Using device 'cuda:0'
6
+ 2026-07-18:10:33:14 INFO [models.huggingface:423] Model parallel was set to False, max memory was not set, and device map was set to {'': 'cuda:0'}
7
+ 2026-07-18:10:33:24 WARNING [api.task:856] niah_single_1: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
8
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
9
+ 2026-07-18:10:33:24 INFO [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
10
 
11
 
12
+
13
+
14
+
15
+
16
+
17
+
18
+
19
+
20
+ 2026-07-18:10:33:33 WARNING [api.task:856] niah_single_3: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
21
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
22
 
23
+
24
 
25
+
26
 
27
 
28
+
29
+
30
+
31
+
32
+
33
+
34
+
35
+
36
+ 2026-07-18:10:33:44 WARNING [api.task:856] niah_multikey_2: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
37
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
38
 
39
 
40
+
41
+
42
+
43
+
44
+
45
+
46
+
47
+ 2026-07-18:10:33:51 WARNING [api.task:856] niah_multiquery: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
48
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
49
 
50
 
51
+
52
+
53
+
54
+
55
+
56
+
57
+
58
+
59
+
60
+ 2026-07-18:10:34:01 WARNING [api.task:856] ruler_vt: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
61
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
62
+ 2026-07-18:10:34:01 INFO [tasks.ruler.common_utils:26] Using tokenizer /workspace for synthetic tasks.
63
  Max length 500 | Current length 307 | Noises: 5
64
  Max length 500 | Current length 410 | Noises: 10
65
  Max length 500 | Current length 531 | Noises: 15
66
  Num noises: 10
67
 
 
68
  0%| | 0/1 [00:00<?, ?it/s]
69
+
70
  0%| | 0/1 [00:00<?, ?it/s]
71
  Max length 8192 | Current length 791 | Noises: 10
72
  Max length 8192 | Current length 1025 | Noises: 20
73
  Max length 8192 | Current length 1273 | Noises: 30
 
101
  Num noises: 310
102
 
103
 
104
  0%| | 0/500 [00:00<?, ?it/s]
 
105
  12%|█▏ | 61/500 [00:01<00:07, 60.88it/s]
 
106
  24%|██▍ | 122/500 [00:02<00:06, 60.00it/s]
 
107
  37%|███▋ | 183/500 [00:03<00:05, 59.21it/s]
 
108
  49%|████▊ | 243/500 [00:04<00:04, 59.20it/s]
 
109
  61%|██████ | 303/500 [00:05<00:03, 59.13it/s]
 
110
  73%|███████▎ | 363/500 [00:06<00:02, 58.72it/s]
 
111
  85%|████████▍ | 423/500 [00:07<00:01, 59.06it/s]
 
112
  97%|█████████▋| 483/500 [00:08<00:00, 58.83it/s]
113
+
114
  12%|█▏ | 60/500 [00:01<00:07, 59.22it/s]
115
+
116
  24%|██▍ | 120/500 [00:02<00:06, 57.62it/s]
117
+
118
  36%|███▌ | 178/500 [00:03<00:05, 57.45it/s]
119
+
120
  47%|████▋ | 237/500 [00:04<00:04, 57.69it/s]
121
+
122
  59%|█████▉ | 295/500 [00:05<00:03, 56.89it/s]
123
+
124
  70%|███████ | 352/500 [00:06<00:02, 56.14it/s]
125
+
126
  82%|████████▏ | 409/500 [00:07<00:01, 56.31it/s]
127
+
128
  94%|█████████▎| 468/500 [00:08<00:00, 56.85it/s]
129
+ 2026-07-18:10:34:10 WARNING [api.task:856] ruler_fwe: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
130
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
131
 
132
 
133
+
134
+
135
+
136
+
137
+
138
+
139
+
140
+
141
+
142
+
143
+
144
+
145
+ 2026-07-18:10:34:24 WARNING [api.task:856] ruler_qa_hotpot: Custom kwargs can be passed to `--metadata` in console (as json string) or to the TaskManager.
146
  For example --metadata='{"max_seq_lengths":[4096, 8192]}'. For details see task Readme.
147
  Traceback (most recent call last):
148
  File "/opt/conda/lib/python3.11/site-packages/urllib3/connection.py", line 198, in _new_conn
 
158
  ^^^^^^^^^^^^^^^^
159
  File "/opt/conda/lib/python3.11/site-packages/urllib3/connection.py", line 207, in _new_conn
160
  raise ConnectTimeoutError(
161
+ urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.HTTPConnection object at 0x7f26ff495550>, 'Connection to curtis.ml.cmu.edu timed out. (connect timeout=None)')
162
 
163
  The above exception was the direct cause of the following exception:
164
 
 
172
  File "/opt/conda/lib/python3.11/site-packages/urllib3/util/retry.py", line 519, in increment
173
  raise MaxRetryError(_pool, url, reason) from reason # type: ignore[arg-type]
174
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
175
+ urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='curtis.ml.cmu.edu', port=80): Max retries exceeded with url: /datasets/hotpot/hotpot_dev_distractor_v1.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f26ff495550>, 'Connection to curtis.ml.cmu.edu timed out. (connect timeout=None)'))
176
 
177
  During handling of the above exception, another exception occurred:
178
 
 
239
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
240
  File "/opt/conda/lib/python3.11/site-packages/requests/adapters.py", line 665, in send
241
  raise ConnectTimeout(e, request=request)
242
+ requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='curtis.ml.cmu.edu', port=80): Max retries exceeded with url: /datasets/hotpot/hotpot_dev_distractor_v1.json (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7f26ff495550>, 'Connection to curtis.ml.cmu.edu timed out. (connect timeout=None)'))