SEE7 commited on
Commit
37adef9
·
verified ·
1 Parent(s): d9aa899

Release tested Astheral V3: matched BF16 improvement and independent regression checks

Browse files
README.md CHANGED
@@ -5,68 +5,40 @@ license: llama3.1
5
  library_name: transformers
6
  pipeline_tag: text-generation
7
  base_model: NousResearch/DeepHermes-3-Llama-3-8B-Preview
8
- base_model_relation: finetune
9
- model_name: Llama-3.1-Astheral
10
  tags:
11
  - astheral
12
  - llama
13
  - instruct
14
- - reasoning
15
  - qlora
16
  ---
 
17
 
18
- # Astheral
19
-
20
- **Built with Llama.** Astheral is SEE7's 8.03B assistant for instruction following, coding, concise explanations and fictional roleplay.
21
-
22
- ## Instruct and thinking
23
-
24
- Choose **instruct** for direct answers or **thinking** for a worked solution before the final answer. Both modes use the same trained weights. The included inference script selects the mode and accepts your own system instructions with --system.
25
-
26
- ## What's changed in v2
27
-
28
- - Balanced training across general instruction tasks, formatting, text editing, Python, and multi-step mathematics.
29
- - Rank-32 adapters across attention and feed-forward projections.
30
- - Explicit training to account for all groups in counting problems and preserve requested output structure.
31
- - Short, complete worked solutions in thinking mode.
32
- - A saved, merged BF16 checkpoint plus the separately downloadable adapter.
33
-
34
- Training used **4,080 examples**, **1084 optimizer steps**, and **83,886,080 trainable adapter parameters** on a remote **NVIDIA A10G**. The original DeepHermes weights provide the starting model.
35
 
36
  ## Measured results
37
 
38
- Both models used identical prompts and deterministic generation settings: up to 384 new tokens for instruct and 1,024 for thinking. The core suite contains 14 formatting tasks, 16 numerical reasoning tasks and six executable Python tasks.
39
-
40
- | Check | DeepHermes base | Astheral v2 |
41
- |---|---:|---:|
42
- | Formatting | 9/14 | 14/14 |
43
- | Numerical reasoning | 16/16 | 16/16 |
44
- | Executable Python | 4/6 | 6/6 |
45
- | Core checks total | 29/36 | 36/36 |
46
- | Known regression cases | 1/2 | 2/2 |
47
-
48
- The two regression cases cover the alpha/omega line format and counting the remaining red and blue balls together. Full prompts, answers, checker code and run metrics are available in evaluate.py, base_outputs.json, astheral_merged_outputs.json and evaluation.json. Six additional samples cover roleplay, prompt design, authorized networking explanations, uncertainty, debugging and summarization.
49
-
50
- Default mode formatting was verified in **60/60 responses**: instruct responses omit thinking tags; thinking responses contain a completed thinking block.
51
 
52
- The saved checkpoint also passed **16/16 additional confirmation cases** covering new CSV records, literal copying, fraction-based balances, time conversion and complete Python functions. Their prompts and answers are in confirmation_outputs.json.
 
 
 
 
53
 
54
- ## Run on a remote GPU
55
 
56
- Install a CUDA-enabled PyTorch build and requirements.txt, then download inference.py from this repository.
57
 
58
- python inference.py "Explain Python generators." --mode instruct
59
- python inference.py "A box has 23 red and 17 blue beads; 6 red are removed. How many remain?" --mode thinking
60
- python inference.py "Write a fictional starship announcement." --mode instruct --load-in-4bit
61
 
62
- Use --system "Your instructions" to replace the default assistant instructions. Decoding is deterministic by default; add --sample for creative variation. BF16 weights occupy about 16 GB; --load-in-4bit loads the weights in NF4 for lower GPU memory use. The script loads SEE7/Astheral directly. Dockerfile includes the CUDA runtime and compiler dependencies for the 4-bit backend.
63
 
64
- ## Files and reproducibility
65
 
66
- The repository includes merged Safetensors weights, the adapter folder, train.py, prepare_data.py, the exact training and validation examples, trainer logs, data source revisions, output comparisons and artifact hashes. The merged checkpoint was saved, reloaded, and evaluated before release.
67
 
68
- ## Training data and attribution
69
 
70
- Training combines original Astheral examples with selected components of HuggingFaceTB/smoltalk (Apache 2.0), the openai/gsm8k training split (MIT), and bigcode/self-oss-instruct-sc2-exec-filter-50k (ODC Attribution). See data_sources.json, DATA.md, DATA_LICENSE files and NOTICE.
71
 
72
- Astheral is fine-tuned from NousResearch/DeepHermes-3-Llama-3-8B-Preview, revision 53d902c7dcfa4c749a3b455149064f82af2e2549, built on Meta Llama 3.1 8B. The project is Astheral; the distributed model name is Llama-3.1-Astheral. See LICENSE and UPSTREAM_LLAMA3_LICENSE.
 
5
  library_name: transformers
6
  pipeline_tag: text-generation
7
  base_model: NousResearch/DeepHermes-3-Llama-3-8B-Preview
 
 
8
  tags:
9
  - astheral
10
  - llama
11
  - instruct
 
12
  - qlora
13
  ---
14
+ # Astheral V3
15
 
16
+ **Built with Llama.** Real 8.03B merged BF16 weights, fine-tuned from Astheral V2. This is a modest instruction-following update, not a claim of broadly improved reasoning or coding.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
17
 
18
  ## Measured results
19
 
20
+ Both final models were evaluated in BF16 with identical prompts and decoding settings.
 
 
 
 
 
 
 
 
 
 
 
 
21
 
22
+ | Suite | V2 | V3 |
23
+ | --- | --- | --- |
24
+ | Task-family holdout (60 cases) | 41/60 | 43/60 |
25
+ | GSM8K test subset | 6/8 | 6/8 |
26
+ | HumanEval subset | 6/8 | 6/8 |
27
 
28
+ The 60-case suite uses unseen values within training-related task families; it is not a broad benchmark. Public benchmark subsets are small and not evidence of state-of-the-art performance. Raw prompts, outputs, scoring code and failures are included in paired_bf16_outputs.json, holdout_outputs.json and holdout_evaluation.json.
29
 
30
+ The quantized adapter scored 49/60 before merging; the shipped BF16 weights scored 43/60 after saving and reloading. The shipped score is the relevant result.
31
 
32
+ ## Training and reproducibility
 
 
33
 
34
+ A10G-small, listed at $1/hour (USD). Rank-16 NF4 QLoRA updates 41,943,040 adapter parameters. Dataset: 1,713 token-filtered examples; 180 optimizer steps, approximately 0.42 epochs. Includes generated arithmetic/formatting/function examples plus retained V2 data. Source and exact data are in v3/ and v3-results/. Reports include the parent revision, seed, timings and memory use.
35
 
36
+ V2 parent: d9aa899722be4788f57107c2951c72d0ec6614ee. It remains available at tag v2-before-v3.
37
 
38
+ ## Use
39
 
40
+ Load SEE7/Astheral with Transformers using BF16 on an appropriate GPU. The merged weights require about 16 GB before context/cache overhead. Existing inference.py supports instruct and thinking modes and reads settings.json. For the separate adapter, use the exact V2 parent revision specified in adapter/adapter_config.json, not the V3 merged weights.
41
 
42
+ ## Licensing
43
 
44
+ Llama 3.1 licensing applies. See LICENSE, UPSTREAM_LLAMA3_LICENSE, NOTICE and the existing data attribution files. Retained V2 examples keep their original source terms; synthetic examples are provided with the reproducible generation script.
adapter/adapter_config.json CHANGED
@@ -1,7 +1,7 @@
1
  {
2
  "alpha_pattern": {},
3
  "auto_mapping": null,
4
- "base_model_name_or_path": "NousResearch/DeepHermes-3-Llama-3-8B-Preview",
5
  "bias": "none",
6
  "corda_config": null,
7
  "eva_config": null,
@@ -13,7 +13,7 @@
13
  "layers_pattern": null,
14
  "layers_to_transform": null,
15
  "loftq_config": {},
16
- "lora_alpha": 64,
17
  "lora_bias": false,
18
  "lora_dropout": 0.05,
19
  "megatron_config": null,
@@ -21,17 +21,17 @@
21
  "modules_to_save": null,
22
  "peft_type": "LORA",
23
  "qalora_group_size": 16,
24
- "r": 32,
25
  "rank_pattern": {},
26
- "revision": null,
27
  "target_modules": [
28
- "down_proj",
29
- "v_proj",
30
  "o_proj",
31
- "k_proj",
32
  "up_proj",
 
 
33
  "q_proj",
34
- "gate_proj"
35
  ],
36
  "target_parameters": null,
37
  "task_type": "CAUSAL_LM",
 
1
  {
2
  "alpha_pattern": {},
3
  "auto_mapping": null,
4
+ "base_model_name_or_path": "SEE7/Astheral",
5
  "bias": "none",
6
  "corda_config": null,
7
  "eva_config": null,
 
13
  "layers_pattern": null,
14
  "layers_to_transform": null,
15
  "loftq_config": {},
16
+ "lora_alpha": 32,
17
  "lora_bias": false,
18
  "lora_dropout": 0.05,
19
  "megatron_config": null,
 
21
  "modules_to_save": null,
22
  "peft_type": "LORA",
23
  "qalora_group_size": 16,
24
+ "r": 16,
25
  "rank_pattern": {},
26
+ "revision": "d9aa899722be4788f57107c2951c72d0ec6614ee",
27
  "target_modules": [
28
+ "gate_proj",
 
29
  "o_proj",
 
30
  "up_proj",
31
+ "down_proj",
32
+ "v_proj",
33
  "q_proj",
34
+ "k_proj"
35
  ],
36
  "target_parameters": null,
37
  "task_type": "CAUSAL_LM",
adapter/adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:39ae042ee2112fad9d8cd6a1e27fc4b6a82a3873c6bc8677833f4c3f7415c063
3
- size 335604696
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d9a26fcd5590725a2cfdfec6246c93ea91e237e3bf34e214b366b2387feb00ae
3
+ size 167832240
artifact_manifest.json CHANGED
@@ -8,8 +8,8 @@
8
  "sha256": "6b1451bc3455ba759012a8b0d4e5b82becb91c16242e6b365ad249a0a5c4cc98"
9
  },
10
  "train.jsonl": {
11
- "bytes": 5307024,
12
- "sha256": "3d8db53dbb072b66ec0e9edfd8dc394832d51707ea7e680da9f120f65f845db0"
13
  },
14
  "validation.jsonl": {
15
  "bytes": 57424,
@@ -20,12 +20,12 @@
20
  "sha256": "820864e76ce5ab18d5dd54a7a4a7419f3b9d5324056b758e976fe28c457f07ef"
21
  },
22
  "evaluation.json": {
23
- "bytes": 3212,
24
- "sha256": "7bc5e8072fa5e9d11976e9bd85ee043717b0e21aea4f98c0f984e6a834553391"
25
  },
26
  "trainer_log.json": {
27
- "bytes": 5255,
28
- "sha256": "3751ecc12be6b0bc6a2ee34d6688643795880d3236476997eb4bd84554c09e17"
29
  },
30
  "settings.json": {
31
  "bytes": 583,
@@ -44,32 +44,32 @@
44
  "sha256": "a596868291bbb2f12644a59bb87ca14bd96601247cbc557d4d1c47c606eb81de"
45
  },
46
  "train.py": {
47
- "bytes": 16157,
48
- "sha256": "e6ed63b1116fec89a3b071491c08bc0360ff20e8f707791c9fb061f7c4b7071c"
49
  },
50
  "config.json": {
51
  "bytes": 1043,
52
- "sha256": "e5a038e1955ffd829ccb9ec510bba1c6a990cf9d2c3c7a3f502c5110a75a9528"
53
  },
54
  "generation_config.json": {
55
- "bytes": 305,
56
- "sha256": "79db16a2918647878da86e0e215e8fc461e2918e6cbdafa84c2f8b8356b57c34"
57
  },
58
  "model-00001-of-00005.safetensors": {
59
  "bytes": 3986791344,
60
- "sha256": "e462f48a2622c389212fc32c8108cdfd7cfd51d2c9f4041b08f0815ce749fdb4"
61
  },
62
  "model-00002-of-00005.safetensors": {
63
  "bytes": 3926025416,
64
- "sha256": "0e066f0d6991f85d05ab24263ffa4b56d39df5014dd4332148173c8876d49fc2"
65
  },
66
  "model-00003-of-00005.safetensors": {
67
  "bytes": 3926025440,
68
- "sha256": "c0cc643be3c2dec8827139aebd24a8a85d94d9edd4833ea513d785adebee01df"
69
  },
70
  "model-00004-of-00005.safetensors": {
71
  "bytes": 3171040864,
72
- "sha256": "185e72dd757bfc06dcbb4b279439bc9ac0c553a62bc0cbae253a90a2a2f4d79c"
73
  },
74
  "model-00005-of-00005.safetensors": {
75
  "bytes": 1050673280,
@@ -156,20 +156,20 @@
156
  "sha256": "fda689d55ae71fc8ca8151b131078b9663843ef965b5d69cad8d2618235edef5"
157
  },
158
  "README.md": {
159
- "bytes": 4350,
160
- "sha256": "f65acc0e71be56b27cab016ae9ea695921b5ec9736d2f4f0d08ab503211a503a"
161
  },
162
  "adapter/README.md": {
163
  "bytes": 5242,
164
  "sha256": "4a1291352e98bd0f745b4d2e6f6a6584428f183f38ba3745d2c1521187bde9d5"
165
  },
166
  "adapter/adapter_model.safetensors": {
167
- "bytes": 335604696,
168
- "sha256": "39ae042ee2112fad9d8cd6a1e27fc4b6a82a3873c6bc8677833f4c3f7415c063"
169
  },
170
  "adapter/adapter_config.json": {
171
- "bytes": 956,
172
- "sha256": "c6f10cf5330b799755b10921c617d3e7ab439c868eef556e7a0c67b0c5a82592"
173
  },
174
  "adapter/chat_template.jinja": {
175
  "bytes": 484,
@@ -200,19 +200,91 @@
200
  "sha256": "1bc9c9f727f140a1a953b06ebe97a5d0981270b4b2506bb9b58f718d94cf5a1f"
201
  },
202
  "holdout_evaluation.json": {
203
- "bytes": 915,
204
- "sha256": "a8ba4f9ee5ccb08816c74dad98bdbcbb5b39591abdcdd5b0ff33945146a2ce81"
205
  },
206
  "holdout_outputs.json": {
207
- "bytes": 51073,
208
- "sha256": "94694960e92e575481b857666fb337f29b1ef739250b96107683e29694446f56"
209
  },
210
  "holdout_eval.py": {
211
- "bytes": 6639,
212
- "sha256": "8f117c55bf33d7c487c2b1a8680b4bf9df817918a5404e564e70df0cd2307fa5"
213
  },
214
  "HUMANEVAL_LICENSE.txt": {
215
  "bytes": 1083,
216
  "sha256": "bcba3de214851cce46ed5af42d6698044616eeace887c3231bc7a20474ab639e"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
217
  }
218
  }
 
8
  "sha256": "6b1451bc3455ba759012a8b0d4e5b82becb91c16242e6b365ad249a0a5c4cc98"
9
  },
10
  "train.jsonl": {
11
+ "bytes": 1314989,
12
+ "sha256": "06714a6155a661588cfdfd6bca750b8f34a654475afbf63c00e8794f22cc8a1f"
13
  },
14
  "validation.jsonl": {
15
  "bytes": 57424,
 
20
  "sha256": "820864e76ce5ab18d5dd54a7a4a7419f3b9d5324056b758e976fe28c457f07ef"
21
  },
22
  "evaluation.json": {
23
+ "bytes": 2408,
24
+ "sha256": "045e35176e473b202b69b3541d38dd1b570d205a794fa1c2ce607253e6cb60af"
25
  },
26
  "trainer_log.json": {
27
+ "bytes": 226,
28
+ "sha256": "3d60088b5403c9879f2f74a9675ca526133fcbff2fe6195bbfb615607751fa7f"
29
  },
30
  "settings.json": {
31
  "bytes": 583,
 
44
  "sha256": "a596868291bbb2f12644a59bb87ca14bd96601247cbc557d4d1c47c606eb81de"
45
  },
46
  "train.py": {
47
+ "bytes": 10241,
48
+ "sha256": "29f8c6f7d584eebf637bf99de3e0d50e0d84c1a0d10b2a7c5f8ee69f8b621a40"
49
  },
50
  "config.json": {
51
  "bytes": 1043,
52
+ "sha256": "7a4af87b359d8a181e2a18eaed3c629ca8a95e056d6161c666e9708bf6e54320"
53
  },
54
  "generation_config.json": {
55
+ "bytes": 231,
56
+ "sha256": "6cba67e3888dfac295499d5660e0f55b8d109d3326659c45fe90314d5cbccbed"
57
  },
58
  "model-00001-of-00005.safetensors": {
59
  "bytes": 3986791344,
60
+ "sha256": "6e4a9bfd682280e0271635d2b4329938f740373ce7d1fe3821d25b210bfbe45f"
61
  },
62
  "model-00002-of-00005.safetensors": {
63
  "bytes": 3926025416,
64
+ "sha256": "ef0783b675748defc6ffc0a64580c578166ac499c5b83b64ceff8d5f60d12801"
65
  },
66
  "model-00003-of-00005.safetensors": {
67
  "bytes": 3926025440,
68
+ "sha256": "30874d39316469ed6bff9cad62a9a576497dc47bc89886fbc67e8454f2733b09"
69
  },
70
  "model-00004-of-00005.safetensors": {
71
  "bytes": 3171040864,
72
+ "sha256": "6fa29d21bcb34fb555d70300cdd2fcb1932bbf2c4c5f2d7a985f0532a1601dce"
73
  },
74
  "model-00005-of-00005.safetensors": {
75
  "bytes": 1050673280,
 
156
  "sha256": "fda689d55ae71fc8ca8151b131078b9663843ef965b5d69cad8d2618235edef5"
157
  },
158
  "README.md": {
159
+ "bytes": 2287,
160
+ "sha256": "d4c78ae9782ab76e4d2c844c2c23892fd04224f6128c041fbe94bcdd75dbcb15"
161
  },
162
  "adapter/README.md": {
163
  "bytes": 5242,
164
  "sha256": "4a1291352e98bd0f745b4d2e6f6a6584428f183f38ba3745d2c1521187bde9d5"
165
  },
166
  "adapter/adapter_model.safetensors": {
167
+ "bytes": 167832240,
168
+ "sha256": "d9a26fcd5590725a2cfdfec6246c93ea91e237e3bf34e214b366b2387feb00ae"
169
  },
170
  "adapter/adapter_config.json": {
171
+ "bytes": 963,
172
+ "sha256": "294dc0be03073320633ddcfe2479977bfa92ec382653014e2894fc34eb6045c9"
173
  },
174
  "adapter/chat_template.jinja": {
175
  "bytes": 484,
 
200
  "sha256": "1bc9c9f727f140a1a953b06ebe97a5d0981270b4b2506bb9b58f718d94cf5a1f"
201
  },
202
  "holdout_evaluation.json": {
203
+ "bytes": 1084,
204
+ "sha256": "fbce8af059fe680606182404058668a327a9cd23dc813423eb60a4ba551fecce"
205
  },
206
  "holdout_outputs.json": {
207
+ "bytes": 34061,
208
+ "sha256": "9239de420e23dc0646176a7a036c2ba5f2a8987d912a89daba0a492df9e5cb35"
209
  },
210
  "holdout_eval.py": {
211
+ "bytes": 8577,
212
+ "sha256": "3095c6fcdd834f17990fd9693e4bb45505a0b1532fda6c8f43e440f1fab7d2b5"
213
  },
214
  "HUMANEVAL_LICENSE.txt": {
215
  "bytes": 1083,
216
  "sha256": "bcba3de214851cce46ed5af42d6698044616eeace887c3231bc7a20474ab639e"
217
+ },
218
+ "v3-results/adapter/README.md": {
219
+ "bytes": 5180,
220
+ "sha256": "1e72c8245ee442ddfed60cf46879336fd3577234908a6b66ba371809f2e1b025"
221
+ },
222
+ "v3-results/adapter/adapter_config.json": {
223
+ "bytes": 963,
224
+ "sha256": "294dc0be03073320633ddcfe2479977bfa92ec382653014e2894fc34eb6045c9"
225
+ },
226
+ "v3-results/adapter/additional_chat_templates/tool_use.jinja": {
227
+ "bytes": 5004,
228
+ "sha256": "7ce09d55d3690e06c70b5c07228cc7e8c99c43a23cdc027762be4011efcdea6c"
229
+ },
230
+ "v3-results/adapter/chat_template.jinja": {
231
+ "bytes": 484,
232
+ "sha256": "47d5a7b4840c2ff2dfa0828b2465a654bef3a6cdfe032e5e105f395e4ceee812"
233
+ },
234
+ "v3-results/adapter/special_tokens_map.json": {
235
+ "bytes": 340,
236
+ "sha256": "f526dabe70ae46e1fb9caff406987c5e46a18f2c572906efa75b890e7833fb2a"
237
+ },
238
+ "v3-results/adapter/tokenizer.json": {
239
+ "bytes": 17209920,
240
+ "sha256": "6b9e4e7fb171f92fd137b777cc2714bf87d11576700a1dcd7a399e7bbe39537b"
241
+ },
242
+ "v3-results/adapter/tokenizer_config.json": {
243
+ "bytes": 50720,
244
+ "sha256": "d93d412174451c19c3eaf39974355289d29b34d40114bd6732a688c5e96c197e"
245
+ },
246
+ "v3-results/heldout_cases.json": {
247
+ "bytes": 10608,
248
+ "sha256": "d0d92afbd8b703776effc747f637e942df40e3e906563e75dafddc9530e7576f"
249
+ },
250
+ "v3-results/train_v3.py": {
251
+ "bytes": 10241,
252
+ "sha256": "29f8c6f7d584eebf637bf99de3e0d50e0d84c1a0d10b2a7c5f8ee69f8b621a40"
253
+ },
254
+ "v3-results/training.jsonl": {
255
+ "bytes": 1314989,
256
+ "sha256": "06714a6155a661588cfdfd6bca750b8f34a654475afbf63c00e8794f22cc8a1f"
257
+ },
258
+ "v3-results/v2_quantized_outputs.json": {
259
+ "bytes": 16335,
260
+ "sha256": "465ec17055801e5ad3f6397751fb26aae868fb6d5e5cc68c1b783758d9ad5f18"
261
+ },
262
+ "v3-results/v3_quantized_outputs.json": {
263
+ "bytes": 15253,
264
+ "sha256": "c3ef696a0e0fa15a458a92c2618875d6a7d968588b6cc317c81b13f9217dbd77"
265
+ },
266
+ "v3-results/v3_reloaded_outputs.json": {
267
+ "bytes": 16121,
268
+ "sha256": "2d621e1690e6ec6ddc3d45427e750f32d57e6f60880b95074e36e7028f3ea2d7"
269
+ },
270
+ "v3-results/v3_report.json": {
271
+ "bytes": 1062,
272
+ "sha256": "6c13f805e1f77dc325b0ae5442c548aa9de3abd41ca441f59f39762a44d0587b"
273
+ },
274
+ "v3/independent_eval.py": {
275
+ "bytes": 8577,
276
+ "sha256": "3095c6fcdd834f17990fd9693e4bb45505a0b1532fda6c8f43e440f1fab7d2b5"
277
+ },
278
+ "v3/train_v3.py": {
279
+ "bytes": 10241,
280
+ "sha256": "29f8c6f7d584eebf637bf99de3e0d50e0d84c1a0d10b2a7c5f8ee69f8b621a40"
281
+ },
282
+ "paired_bf16_outputs.json": {
283
+ "bytes": 32202,
284
+ "sha256": "defe6b169178bed42ec5a7127d76c51d06680aa10ddaf60c5dfcfd8a53ffc490"
285
+ },
286
+ "v3-results/adapter/adapter_model.safetensors": {
287
+ "bytes": 167832240,
288
+ "sha256": "d9a26fcd5590725a2cfdfec6246c93ea91e237e3bf34e214b366b2387feb00ae"
289
  }
290
  }
config.json CHANGED
@@ -7,7 +7,7 @@
7
  "instruct",
8
  "thinking"
9
  ],
10
- "astheral_version": "2",
11
  "attention_bias": false,
12
  "attention_dropout": 0.0,
13
  "bos_token_id": 128000,
 
7
  "instruct",
8
  "thinking"
9
  ],
10
+ "astheral_version": "3",
11
  "attention_bias": false,
12
  "attention_dropout": 0.0,
13
  "bos_token_id": 128000,
evaluation.json CHANGED
@@ -1,149 +1,97 @@
1
  {
2
- "version": "2",
3
- "repair_pass": 2,
4
- "base_model": "NousResearch/DeepHermes-3-Llama-3-8B-Preview",
5
- "base_revision": "53d902c7dcfa4c749a3b455149064f82af2e2549",
6
- "initial_adapter_revision": "930a185fd9f4e7fffe44283fa9009ccbd2aa0af7",
7
- "training_examples": 4080,
8
- "repair_training_examples": 1617,
9
- "validation_examples": 64,
10
- "trainable_parameters": 83886080,
11
- "rank": 32,
12
- "target_modules": "all-linear",
13
- "optimizer_steps": 1084,
14
- "training_stages": [
15
- {
16
- "method": "NF4 QLoRA",
17
- "examples": 2429,
18
- "steps": 456,
19
- "epochs": 1.500617538081515
20
- },
21
- {
22
- "method": "BF16 LoRA",
23
- "examples": 1714,
24
- "steps": 323,
25
- "epochs": 1.5040840140023337
26
- },
27
- {
28
- "method": "BF16 LoRA",
29
- "examples": 1617,
30
- "steps": 305,
31
- "epochs": 1.5046382189239331
32
- }
33
- ],
34
- "validation_loss_before": 1.3753070831298828,
35
- "validation_loss_after": 0.0020501785911619663,
36
- "base_scores": {
37
- "regression": {
38
- "passed": 1,
39
- "total": 2
40
- },
41
- "fresh": {
42
- "passed": 29,
43
- "total": 36
44
- },
45
- "exact": {
46
- "passed": 5,
47
- "total": 10
48
- },
49
- "json": {
50
- "passed": 4,
51
- "total": 4
52
- },
53
- "number": {
54
- "passed": 16,
55
- "total": 16
56
- },
57
- "code": {
58
- "passed": 4,
59
- "total": 6
60
- }
61
- },
62
- "astheral_scores": {
63
- "regression": {
64
- "passed": 2,
65
- "total": 2
66
- },
67
- "fresh": {
68
- "passed": 36,
69
- "total": 36
70
- },
71
- "exact": {
72
- "passed": 10,
73
- "total": 10
74
- },
75
- "json": {
76
- "passed": 4,
77
- "total": 4
78
- },
79
- "number": {
80
- "passed": 16,
81
- "total": 16
82
- },
83
- "code": {
84
- "passed": 6,
85
- "total": 6
86
- },
87
- "confirmation": {
88
- "passed": 16,
89
- "total": 16
90
- }
91
- },
92
- "quality_gate_passed": true,
93
- "hardware": "NVIDIA A10G",
94
- "max_gpu_memory_allocated_bytes": 17538217984,
95
  "training": {
96
- "train_runtime": 1007.3112,
97
- "train_samples_per_second": 2.408,
98
- "train_steps_per_second": 0.303,
99
- "total_flos": 2.0134086292832256e+16,
100
- "train_loss": 0.15099767006811549,
101
- "epoch": 1.5046382189239331
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
102
  },
103
- "generation": {
104
- "sampling": false,
105
- "instruct_max_new_tokens": 384,
106
- "thinking_max_new_tokens": 1024
107
- },
108
- "evaluation_design": "36 core development checks, two original regressions, 12 prior confirmation cases and four new time/code confirmation cases. Independent public test subsets are evaluated separately.",
109
- "merged_scores": {
110
- "regression": {
111
- "passed": 2,
112
- "total": 2
113
- },
114
- "fresh": {
115
- "passed": 36,
116
- "total": 36
117
- },
118
- "exact": {
119
- "passed": 10,
120
- "total": 10
121
- },
122
- "json": {
123
- "passed": 4,
124
- "total": 4
125
- },
126
- "number": {
127
- "passed": 16,
128
- "total": 16
129
- },
130
- "code": {
131
- "passed": 6,
132
- "total": 6
133
- },
134
- "confirmation": {
135
- "passed": 16,
136
- "total": 16
 
 
 
 
 
 
 
 
 
 
 
137
  }
138
- },
139
- "elapsed_seconds": 1417.947889062998,
140
- "evaluation_parser_revision": "2: boxed answer, unambiguous leading answer paragraph, then numeric fallback",
141
- "mode_adherence": {
142
- "passed": 60,
143
- "total": 60
144
- },
145
- "thinking_output_characters": {
146
- "base": 1426.7058823529412,
147
- "astheral": 136.22727272727272
148
  }
149
  }
 
1
  {
2
+ "version": "3",
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  "training": {
4
+ "status": "complete",
5
+ "parent_revision": "d9aa899722be4788f57107c2951c72d0ec6614ee",
6
+ "hardware": "NVIDIA A10G",
7
+ "job_timeout_seconds": 5400,
8
+ "listed_usd_per_hour": 1.0,
9
+ "training_seed": 20261113,
10
+ "baseline_passed": 40,
11
+ "elapsed_seconds": 622.09,
12
+ "training_examples": 1713,
13
+ "trainable_parameters": 41943040,
14
+ "training": {
15
+ "train_runtime": 390.2404,
16
+ "train_samples_per_second": 1.845,
17
+ "train_steps_per_second": 0.461,
18
+ "total_flos": 3960249019342848.0,
19
+ "train_loss": 0.2867420093880759,
20
+ "epoch": 0.4203152364273205
21
+ },
22
+ "optimizer_steps": 180,
23
+ "candidate_passed": 49,
24
+ "reloaded_passed": 43,
25
+ "groups": {
26
+ "integer": {
27
+ "baseline": 0,
28
+ "candidate": 3,
29
+ "total": 20
30
+ },
31
+ "exact": {
32
+ "baseline": 20,
33
+ "candidate": 20,
34
+ "total": 20
35
+ },
36
+ "code": {
37
+ "baseline": 10,
38
+ "candidate": 10,
39
+ "total": 10
40
+ },
41
+ "json": {
42
+ "baseline": 10,
43
+ "candidate": 10,
44
+ "total": 10
45
+ }
46
+ },
47
+ "quality_gate_passed": true,
48
+ "peak_gpu_bytes": 18991949312
49
  },
50
+ "matched_bf16_and_independent_tests": {
51
+ "comparison": "V2 versus V3, both BF16",
52
+ "v2_revision": "d9aa899722be4788f57107c2951c72d0ec6614ee",
53
+ "paired_bf16_scores": {
54
+ "base": 41,
55
+ "astheral": 43
56
+ },
57
+ "suite": "GSM8K test rows 231-238; HumanEval selected tasks 0,3,7,12,18,25,36,45",
58
+ "scores": {
59
+ "base": {
60
+ "code": {
61
+ "passed": 6,
62
+ "total": 8,
63
+ "token_limited": 0
64
+ },
65
+ "number": {
66
+ "passed": 6,
67
+ "total": 8,
68
+ "token_limited": 0
69
+ }
70
+ },
71
+ "astheral": {
72
+ "code": {
73
+ "passed": 6,
74
+ "total": 8,
75
+ "token_limited": 0
76
+ },
77
+ "number": {
78
+ "passed": 6,
79
+ "total": 8,
80
+ "token_limited": 0
81
+ }
82
+ }
83
+ },
84
+ "no_material_regression_gate": true,
85
+ "astheral_revision": "8d553fe747fd6e12fd650c97b712e7009a8b4f9e",
86
+ "settings": {
87
+ "sampling": false,
88
+ "code_max_new_tokens": 768,
89
+ "math_max_new_tokens": 1536
90
+ },
91
+ "hardware": "NVIDIA A10G",
92
+ "source_revisions": {
93
+ "gsm8k": "740312add88f781978c0658806c59bc2815b9866",
94
+ "humaneval": "7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544"
95
  }
 
 
 
 
 
 
 
 
 
 
96
  }
97
  }
generation_config.json CHANGED
@@ -1,17 +1,13 @@
1
  {
2
  "_from_model_config": true,
3
  "bos_token_id": 128000,
4
- "do_sample": false,
5
  "eos_token_id": [
6
  128001,
7
  128008,
8
  128009
9
  ],
10
  "max_length": 131072,
 
11
  "pad_token_id": 128004,
12
- "temperature": 1.0,
13
- "top_p": 1.0,
14
- "transformers_version": "4.56.2",
15
- "top_k": 50,
16
- "max_new_tokens": 1024
17
- }
 
1
  {
2
  "_from_model_config": true,
3
  "bos_token_id": 128000,
 
4
  "eos_token_id": [
5
  128001,
6
  128008,
7
  128009
8
  ],
9
  "max_length": 131072,
10
+ "max_new_tokens": 1024,
11
  "pad_token_id": 128004,
12
+ "transformers_version": "4.56.2"
13
+ }
 
 
 
 
holdout_eval.py CHANGED
@@ -1,13 +1,20 @@
 
 
 
1
  """Independent GSM8K and HumanEval subset comparison on remote GPUs."""
2
  import json,re,sys,subprocess,tempfile,ast,urllib.request,urllib.parse,gc,hashlib
3
  from pathlib import Path
4
  import torch
5
  from transformers import AutoModelForCausalLM,AutoTokenizer
6
  from huggingface_hub import HfApi,hf_hub_download,CommitOperationAdd
7
- from evaluate import check as task_check
 
 
 
8
  api=HfApi();REPO="SEE7/Astheral";BASE="NousResearch/DeepHermes-3-Llama-3-8B-Preview";REV="53d902c7dcfa4c749a3b455149064f82af2e2549"
9
  ROOT=Path(__file__).resolve().parent
10
- revision=api.model_info(REPO).sha
 
11
  settings=json.loads(Path(hf_hub_download(REPO,"settings.json",revision=revision)).read_text())
12
  def rows(repo,config,split,offset,length):
13
  q=urllib.parse.urlencode(dict(dataset=repo,config=config,split=split,offset=offset,length=length))
@@ -37,13 +44,26 @@ def check_code(c,text):
37
  with tempfile.TemporaryDirectory() as d:
38
  p=Path(d)/"check.py"
39
  p.write_text("import resource\nresource.setrlimit(resource.RLIMIT_CPU,(3,3))\nresource.setrlimit(resource.RLIMIT_AS,(536870912,536870912))\n"+code+"\n"+c["tests"])
40
- r=subprocess.run([sys.executable,"-I",str(p)],cwd=d,timeout=5,capture_output=True)
41
  return r.returncode==0
42
  except Exception:return False
43
- all_results={};scores={}
44
- for tag,repo,rev in [("base",BASE,REV),("astheral",REPO,revision)]:
 
 
 
 
45
  tok=AutoTokenizer.from_pretrained(repo,revision=rev);tok.pad_token_id=128004;tok.padding_side="left"
46
  model=AutoModelForCausalLM.from_pretrained(repo,revision=rev,torch_dtype=torch.bfloat16,device_map={"":"cuda:0"},attn_implementation="sdpa").eval()
 
 
 
 
 
 
 
 
 
47
  result=[]
48
  with torch.inference_mode():
49
  for mode in ["instruct","thinking"]:
@@ -66,12 +86,13 @@ for tag,repo,rev in [("base",BASE,REV),("astheral",REPO,revision)]:
66
  chosen_rows=[r for r in result if r["kind"]==kind]
67
  scores[tag][kind]={"passed":sum(r["pass"] for r in chosen_rows),"total":len(chosen_rows),"token_limited":sum(r["hit_token_limit"] for r in chosen_rows)}
68
  del model;gc.collect();torch.cuda.empty_cache()
69
- passed=sum(v["passed"] for v in scores["astheral"].values())>=sum(v["passed"] for v in scores["base"].values()) and all(scores["astheral"][k]["passed"]>=scores["base"][k]["passed"]-1 for k in ["code","number"])
70
- report={"suite":"GSM8K test rows 231-238; HumanEval selected tasks 0,3,7,12,18,25,36,45","scores":scores,"no_material_regression_gate":passed,"astheral_revision":revision,"settings":{"sampling":False,"code_max_new_tokens":768,"math_max_new_tokens":1536},"hardware":torch.cuda.get_device_name(0),"source_revisions":{"gsm8k":math_revision,"humaneval":code_revision}}
71
- changed={"holdout_evaluation.json":json.dumps(report,indent=2),"holdout_outputs.json":json.dumps(all_results,indent=2),"holdout_eval.py":Path(__file__).read_text(),"HUMANEVAL_LICENSE.txt":urllib.request.urlopen("https://raw.githubusercontent.com/openai/human-eval/master/LICENSE",timeout=60).read().decode()}
 
72
  manifest=json.loads(Path(hf_hub_download(REPO,"artifact_manifest.json")).read_text())
73
  for n,b in changed.items():manifest[n]={"bytes":len(b.encode()),"sha256":hashlib.sha256(b.encode()).hexdigest()}
74
  changed["artifact_manifest.json"]=json.dumps(manifest,indent=2)
75
- api.create_commit(repo_id=REPO,operations=[CommitOperationAdd(path_in_repo=n,path_or_fileobj=b.encode()) for n,b in changed.items()],commit_message="Record independent GSM8K and HumanEval test-subset comparison")
76
  print("HOLDOUT_REPORT",json.dumps(report),flush=True)
77
  if not passed:raise SystemExit(5)
 
1
+ # /// script
2
+ # dependencies = ["torch==2.8.0", "transformers==4.56.2", "huggingface_hub==0.34.4", "accelerate==1.10.1", "safetensors==0.6.2"]
3
+ # ///
4
  """Independent GSM8K and HumanEval subset comparison on remote GPUs."""
5
  import json,re,sys,subprocess,tempfile,ast,urllib.request,urllib.parse,gc,hashlib
6
  from pathlib import Path
7
  import torch
8
  from transformers import AutoModelForCausalLM,AutoTokenizer
9
  from huggingface_hub import HfApi,hf_hub_download,CommitOperationAdd
10
+ def task_check(case,text):
11
+ final=text.split("</think>")[-1]
12
+ numbers=re.findall(r"-?\d+(?:\.\d+)?",final.replace(",",""))
13
+ return bool(numbers) and abs(float(numbers[-1])-float(case["expected"]))<1e-6
14
  api=HfApi();REPO="SEE7/Astheral";BASE="NousResearch/DeepHermes-3-Llama-3-8B-Preview";REV="53d902c7dcfa4c749a3b455149064f82af2e2549"
15
  ROOT=Path(__file__).resolve().parent
16
+ BRANCH="v3-candidate-20261113"
17
+ revision=api.model_info(REPO,revision=BRANCH).sha
18
  settings=json.loads(Path(hf_hub_download(REPO,"settings.json",revision=revision)).read_text())
19
  def rows(repo,config,split,offset,length):
20
  q=urllib.parse.urlencode(dict(dataset=repo,config=config,split=split,offset=offset,length=length))
 
44
  with tempfile.TemporaryDirectory() as d:
45
  p=Path(d)/"check.py"
46
  p.write_text("import resource\nresource.setrlimit(resource.RLIMIT_CPU,(3,3))\nresource.setrlimit(resource.RLIMIT_AS,(536870912,536870912))\n"+code+"\n"+c["tests"])
47
+ r=subprocess.run([sys.executable,"-I",str(p)],cwd=d,timeout=5,capture_output=True,env={"PATH":"/usr/bin:/bin"})
48
  return r.returncode==0
49
  except Exception:return False
50
+ all_results={};scores={};paired_results={}
51
+ source=Path(hf_hub_download(REPO,'v3/train_v3.py',revision=revision)).read_text()
52
+ grade_node=next(n for n in ast.parse(source).body if isinstance(n,ast.FunctionDef) and n.name=='grade')
53
+ grade_scope={'json':json,'re':re};exec(compile(ast.Module(body=[grade_node],type_ignores=[]),'grader','exec'),grade_scope)
54
+ paired_cases=json.loads(Path(hf_hub_download(REPO,'v3-results/heldout_cases.json',revision=revision)).read_text())
55
+ for tag,repo,rev in [("base",REPO,"d9aa899722be4788f57107c2951c72d0ec6614ee"),("astheral",REPO,revision)]:
56
  tok=AutoTokenizer.from_pretrained(repo,revision=rev);tok.pad_token_id=128004;tok.padding_side="left"
57
  model=AutoModelForCausalLM.from_pretrained(repo,revision=rev,torch_dtype=torch.bfloat16,device_map={"":"cuda:0"},attn_implementation="sdpa").eval()
58
+ paired=[]
59
+ with torch.inference_mode():
60
+ for c in paired_cases:
61
+ inp=tok.apply_chat_template([{"role":"system","content":settings["system"]},{"role":"user","content":c["prompt"]}],tokenize=True,add_generation_prompt=True,return_dict=True,return_tensors="pt").to("cuda")
62
+ output=model.generate(**inp,max_new_tokens=96,do_sample=False,pad_token_id=128004,eos_token_id=128009)
63
+ text=tok.decode(output[0,inp["input_ids"].shape[1]:],skip_special_tokens=True)
64
+ paired.append({**c,"output":text,"pass":grade_scope["grade"](c,text)})
65
+ print("BF16_PAIRED",tag,c["id"],paired[-1]["pass"],flush=True)
66
+ paired_results[tag]=paired
67
  result=[]
68
  with torch.inference_mode():
69
  for mode in ["instruct","thinking"]:
 
86
  chosen_rows=[r for r in result if r["kind"]==kind]
87
  scores[tag][kind]={"passed":sum(r["pass"] for r in chosen_rows),"total":len(chosen_rows),"token_limited":sum(r["hit_token_limit"] for r in chosen_rows)}
88
  del model;gc.collect();torch.cuda.empty_cache()
89
+ passed=sum(v["passed"] for v in scores["astheral"].values())>=sum(v["passed"] for v in scores["base"].values()) and all(scores["astheral"][k]["passed"]>=scores["base"][k]["passed"] for k in ["code","number"])
90
+ passed=passed and sum(r["pass"] for r in paired_results["astheral"])>=sum(r["pass"] for r in paired_results["base"])
91
+ report={"comparison":"V2 versus V3, both BF16","v2_revision":"d9aa899722be4788f57107c2951c72d0ec6614ee","paired_bf16_scores":{k:sum(r["pass"] for r in v) for k,v in paired_results.items()},"suite":"GSM8K test rows 231-238; HumanEval selected tasks 0,3,7,12,18,25,36,45","scores":scores,"no_material_regression_gate":passed,"astheral_revision":revision,"settings":{"sampling":False,"code_max_new_tokens":768,"math_max_new_tokens":1536},"hardware":torch.cuda.get_device_name(0),"source_revisions":{"gsm8k":math_revision,"humaneval":code_revision}}
92
+ changed={"paired_bf16_outputs.json":json.dumps(paired_results,indent=2),"holdout_evaluation.json":json.dumps(report,indent=2),"holdout_outputs.json":json.dumps(all_results,indent=2),"holdout_eval.py":Path(__file__).read_text(),"HUMANEVAL_LICENSE.txt":urllib.request.urlopen("https://raw.githubusercontent.com/openai/human-eval/master/LICENSE",timeout=60).read().decode()}
93
  manifest=json.loads(Path(hf_hub_download(REPO,"artifact_manifest.json")).read_text())
94
  for n,b in changed.items():manifest[n]={"bytes":len(b.encode()),"sha256":hashlib.sha256(b.encode()).hexdigest()}
95
  changed["artifact_manifest.json"]=json.dumps(manifest,indent=2)
96
+ api.create_commit(repo_id=REPO,revision=BRANCH,operations=[CommitOperationAdd(path_in_repo=n,path_or_fileobj=b.encode()) for n,b in changed.items()],commit_message="Record measured V2 versus V3 GSM8K and HumanEval comparison")
97
  print("HOLDOUT_REPORT",json.dumps(report),flush=True)
98
  if not passed:raise SystemExit(5)
holdout_evaluation.json CHANGED
@@ -1,16 +1,22 @@
1
  {
 
 
 
 
 
 
2
  "suite": "GSM8K test rows 231-238; HumanEval selected tasks 0,3,7,12,18,25,36,45",
3
  "scores": {
4
  "base": {
5
  "code": {
6
- "passed": 7,
7
  "total": 8,
8
  "token_limited": 0
9
  },
10
  "number": {
11
- "passed": 7,
12
  "total": 8,
13
- "token_limited": 1
14
  }
15
  },
16
  "astheral": {
@@ -26,8 +32,8 @@
26
  }
27
  }
28
  },
29
- "no_material_regression_gate": false,
30
- "astheral_revision": "0bfdd04a523a7ffbfcdaa1fcdb1357518c1a9931",
31
  "settings": {
32
  "sampling": false,
33
  "code_max_new_tokens": 768,
 
1
  {
2
+ "comparison": "V2 versus V3, both BF16",
3
+ "v2_revision": "d9aa899722be4788f57107c2951c72d0ec6614ee",
4
+ "paired_bf16_scores": {
5
+ "base": 41,
6
+ "astheral": 43
7
+ },
8
  "suite": "GSM8K test rows 231-238; HumanEval selected tasks 0,3,7,12,18,25,36,45",
9
  "scores": {
10
  "base": {
11
  "code": {
12
+ "passed": 6,
13
  "total": 8,
14
  "token_limited": 0
15
  },
16
  "number": {
17
+ "passed": 6,
18
  "total": 8,
19
+ "token_limited": 0
20
  }
21
  },
22
  "astheral": {
 
32
  }
33
  }
34
  },
35
+ "no_material_regression_gate": true,
36
+ "astheral_revision": "8d553fe747fd6e12fd650c97b712e7009a8b4f9e",
37
  "settings": {
38
  "sampling": false,
39
  "code_max_new_tokens": 768,
holdout_outputs.json CHANGED
@@ -9,7 +9,7 @@
9
  "tests": "\n\nMETADATA = {\n 'author': 'jt',\n 'dataset': 'test'\n}\n\n\ndef check(candidate):\n assert candidate([1.0, 2.0, 3.9, 4.0, 5.0, 2.2], 0.3) == True\n assert candidate([1.0, 2.0, 3.9, 4.0, 5.0, 2.2], 0.05) == False\n assert candidate([1.0, 2.0, 5.9, 4.0, 5.0], 0.95) == True\n assert candidate([1.0, 2.0, 5.9, 4.0, 5.0], 0.8) == False\n assert candidate([1.0, 2.0, 3.0, 4.0, 5.0, 2.0], 0.1) == True\n assert candidate([1.1, 2.2, 3.1, 4.1, 5.1], 1.0) == True\n assert candidate([1.1, 2.2, 3.1, 4.1, 5.1], 0.5) == False\n\n\ncheck(has_close_elements)",
10
  "source": "openai/openai_humaneval",
11
  "source_revision": "7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544",
12
- "output": "Here is the complete Python implementation for the `has_close_elements` function:\n\n```python\nfrom typing import List\n\ndef has_close_elements(numbers: List[float], threshold: float) -> bool:\n for i in range(len(numbers)):\n for j in range(i + 1, len(numbers)):\n if abs(numbers[i] - numbers[j]) < threshold:\n return True\n return False\n```\n\nThis implementation uses nested loops to compare each pair of numbers in the input list. If any pair of numbers is closer to each other than the given threshold, the function returns `True`. If no such pair is found, it returns `False`.",
13
  "pass": true,
14
  "hit_token_limit": false
15
  },
@@ -22,7 +22,7 @@
22
  "tests": "\n\nMETADATA = {\n 'author': 'jt',\n 'dataset': 'test'\n}\n\n\ndef check(candidate):\n assert candidate([]) == False\n assert candidate([1, 2, -3, 1, 2, -3]) == False\n assert candidate([1, 2, -4, 5, 6]) == True\n assert candidate([1, -1, 2, -2, 5, -5, 4, -4]) == False\n assert candidate([1, -1, 2, -2, 5, -5, 4, -5]) == True\n assert candidate([1, -2, 2, -2, 5, -5, 4, -4]) == True\n\ncheck(below_zero)",
23
  "source": "openai/openai_humaneval",
24
  "source_revision": "7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544",
25
- "output": "Here is the complete Python implementation for the `below_zero` function:\n\n```python\nfrom typing import List\n\ndef below_zero(operations: List[int]) -> bool:\n balance = 0\n for operation in operations:\n balance += operation\n if balance < 0:\n return True\n return False\n```\n\nThis implementation initializes the `balance` variable to 0 and iterates through each operation in the `operations` list. It adds each operation to the `balance` and checks if the balance falls below 0. If it does, the function returns `True`. If the loop completes without the balance falling below 0, the function returns `False`.",
26
  "pass": true,
27
  "hit_token_limit": false
28
  },
@@ -35,7 +35,7 @@
35
  "tests": "\n\nMETADATA = {\n 'author': 'jt',\n 'dataset': 'test'\n}\n\n\ndef check(candidate):\n assert candidate([], 'john') == []\n assert candidate(['xxx', 'asd', 'xxy', 'john doe', 'xxxAAA', 'xxx'], 'xxx') == ['xxx', 'xxxAAA', 'xxx']\n assert candidate(['xxx', 'asd', 'aaaxxy', 'john doe', 'xxxAAA', 'xxx'], 'xx') == ['xxx', 'aaaxxy', 'xxxAAA', 'xxx']\n assert candidate(['grunt', 'trumpet', 'prune', 'gruesome'], 'run') == ['grunt', 'prune']\n\ncheck(filter_by_substring)",
36
  "source": "openai/openai_humaneval",
37
  "source_revision": "7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544",
38
- "output": "Here is the complete Python implementation for the `filter_by_substring` function:\n\n```python\nfrom typing import List\n\ndef filter_by_substring(strings: List[str], substring: str) -> List[str]:\n \"\"\" Filter an input list of strings only for ones that contain given substring\n >>> filter_by_substring([], 'a')\n []\n >>> filter_by_substring(['abc', 'bacd', 'cde', 'array'], 'a')\n ['abc', 'bacd', 'array']\n \"\"\"\n return [s for s in strings if substring in s]\n```\n\nThis implementation uses a list comprehension to filter the input list of strings based on whether each string contains the given substring. The resulting list is then returned.",
39
  "pass": true,
40
  "hit_token_limit": false
41
  },
@@ -48,7 +48,7 @@
48
  "tests": "\n\nMETADATA = {\n 'author': 'jt',\n 'dataset': 'test'\n}\n\n\ndef check(candidate):\n assert candidate([]) == None\n assert candidate(['x', 'y', 'z']) == 'x'\n assert candidate(['x', 'yyy', 'zzzz', 'www', 'kkkk', 'abc']) == 'zzzz'\n\ncheck(longest)",
49
  "source": "openai/openai_humaneval",
50
  "source_revision": "7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544",
51
- "output": "Here is the complete Python implementation for the `longest` function:\n\n```python\nfrom typing import List, Optional\n\ndef longest(strings: List[str]) -> Optional[str]:\n if not strings:\n return None\n \n longest_string = max(strings, key=len)\n return longest_string\n```\n\nThis implementation first checks if the input list `strings` is empty. If it is, the function returns `None`. Otherwise, it uses the `max` function to find the longest string in the list based on the length of each string. The `key` parameter of `max` is set to `len` to compare the strings based on their length. Finally, the function returns the longest string found.",
52
  "pass": true,
53
  "hit_token_limit": false
54
  },
@@ -61,7 +61,7 @@
61
  "tests": "\n\nMETADATA = {\n 'author': 'jt',\n 'dataset': 'test'\n}\n\n\ndef check(candidate):\n assert candidate('', 'x') == 0\n assert candidate('xyxyxyx', 'x') == 4\n assert candidate('cacacacac', 'cac') == 4\n assert candidate('john doe', 'john') == 1\n\ncheck(how_many_times)",
62
  "source": "openai/openai_humaneval",
63
  "source_revision": "7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544",
64
- "output": "Here is the complete Python implementation for the function `how_many_times`:\n\n```python\ndef how_many_times(string: str, substring: str) -> int:\n \"\"\" Find how many times a given substring can be found in the original string. Count overlapping cases.\n >>> how_many_times('', 'a')\n 0\n >>> how_many_times('aaa', 'a')\n 3\n >>> how_many_times('aaaa', 'aa')\n 3\n \"\"\"\n count = 0\n index = 0\n while index < len(string):\n if string.startswith(substring, index):\n count += 1\n index += len(substring)\n else:\n index += 1\n return count\n```\n\nThis implementation uses a while loop to iterate through the string, checking if the substring starts at each position using the `startswith()` method. If a match is found, the count is incremented and the index is moved forward by the length of the substring. If no match is found, the index is simply incremented by 1. Finally, the total count is returned.",
65
  "pass": false,
66
  "hit_token_limit": false
67
  },
@@ -74,7 +74,7 @@
74
  "tests": "\n\nMETADATA = {\n 'author': 'jt',\n 'dataset': 'test'\n}\n\n\ndef check(candidate):\n assert candidate(2) == [2]\n assert candidate(4) == [2, 2]\n assert candidate(8) == [2, 2, 2]\n assert candidate(3 * 19) == [3, 19]\n assert candidate(3 * 19 * 3 * 19) == [3, 3, 19, 19]\n assert candidate(3 * 19 * 3 * 19 * 3 * 19) == [3, 3, 3, 19, 19, 19]\n assert candidate(3 * 19 * 19 * 19) == [3, 19, 19, 19]\n assert candidate(3 * 2 * 3) == [2, 3, 3]\n\ncheck(factorize)",
75
  "source": "openai/openai_humaneval",
76
  "source_revision": "7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544",
77
- "output": "Here is the complete Python implementation for the factorize function:\n\n```python\nfrom typing import List\n\ndef factorize(n: int) -> List[int]:\n factors = []\n divisor = 2\n while divisor * divisor <= n:\n if n % divisor == 0:\n factors.append(divisor)\n n //= divisor\n else:\n divisor += 1\n if n > 1:\n factors.append(n)\n return factors\n```\n\nThis implementation uses a simple algorithm to find the prime factors of a given integer `n`. It starts with the smallest prime number (2) and divides `n` by it as long as it is divisible. If `n` is not divisible by the current divisor, it increments the divisor by 1. This process continues until the square of the divisor is greater than `n`. If `n` is still greater than 1 after the loop, it means it is a prime number itself, so it is added to the list of factors. The function returns the list of prime factors in the order from smallest to largest.",
78
  "pass": true,
79
  "hit_token_limit": false
80
  },
@@ -87,8 +87,8 @@
87
  "tests": "\n\nMETADATA = {}\n\n\ndef check(candidate):\n assert candidate(50) == 0\n assert candidate(78) == 2\n assert candidate(79) == 3\n assert candidate(100) == 3\n assert candidate(200) == 6\n assert candidate(4000) == 192\n assert candidate(10000) == 639\n assert candidate(100000) == 8026\n\n\ncheck(fizz_buzz)",
88
  "source": "openai/openai_humaneval",
89
  "source_revision": "7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544",
90
- "output": "Here is the complete Python implementation for the `fizz_buzz` function:\n\n```python\ndef fizz_buzz(n: int):\n count = 0\n for i in range(1, n):\n if i % 11 == 0 or i % 13 == 0:\n for digit in str(i):\n if digit == '7':\n count += 1\n return count\n```\n\nThis implementation follows the given function specification and includes the necessary imports. The function takes an integer `n` as input and returns the number of times the digit 7 appears in integers less than `n` which are divisible by 11 or 13.",
91
- "pass": true,
92
  "hit_token_limit": false
93
  },
94
  {
@@ -100,7 +100,7 @@
100
  "tests": "\n\nMETADATA = {}\n\n\ndef check(candidate):\n assert candidate(5, 3) == 7.5\n assert candidate(2, 2) == 2.0\n assert candidate(10, 8) == 40.0\n\n\ncheck(triangle_area)",
101
  "source": "openai/openai_humaneval",
102
  "source_revision": "7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544",
103
- "output": "Here is the complete Python implementation for the `triangle_area` function:\n\n```python\nimport math\n\ndef triangle_area(a, h):\n \"\"\"Given length of a side and high return area for a triangle.\n >>> triangle_area(5, 3)\n 7.5\n \"\"\"\n return 0.5 * a * h\n```\n\nThis implementation uses the formula for the area of a triangle, which is `(base * height) / 2`. The function takes the base `a` and height `h` as input and returns the calculated area. The `math` module is imported to use the `*` operator for multiplication.",
104
  "pass": true,
105
  "hit_token_limit": false
106
  },
@@ -112,7 +112,7 @@
112
  "expected": 50.0,
113
  "source": "openai/gsm8k",
114
  "source_revision": "740312add88f781978c0658806c59bc2815b9866",
115
- "output": "<think>\nOkay, let's see. So, the problem is about Kylie and Robert collecting shells at the beach. We need to find out how many shells Kylie collects on Tuesday. Let's break it down step by step.\n\nFirst, on Monday, Robert collects 20 shells. It says Kylie collects 5 more shells than Robert on Monday. So, if Robert has 20, then Kylie must have 20 + 5, which is 25 shells on Monday. Got that part.\n\nThen, on Tuesday, Kylie collects 2 times more shells than she did on Monday. Hmm, \"2 times more\" can sometimes be a bit tricky. Does that mean 2 times as many, or 2 more than twice? I think it's 2 times as many. So, if she had 25 on Monday, then on Tuesday she would have 25 * 2, which is 50 shells. Let me double-check that. If it was 2 more than twice, it would be 25 + 2*25 = 75, but that seems like a lot. The problem says \"2 times more\", which usually means multiplying by 2, not adding 2. So I think 50 is the right answer.\n\nWait, let me make sure. Let's see, if Kylie collected 25 on Monday, and then on Tuesday she collected 2 times that amount, then yes, 25 * 2 = 50. So, Kylie collects 50 shells on Tuesday. That seems right. I don't think I missed anything here. The problem gives all the necessary information, and the calculations check out. So, the final answer should be 50.\n</think>\n\nKylie collects **50 shells** on Tuesday. \n\n**Step-by-Step Explanation:**\n1. **Monday's Shells:**\n - Robert collects 20 shells.\n - Kylie collects 5 more than Robert: \\( 20 + 5 = 25 \\) shells.\n\n2. **Tuesday's Shells:**\n - Kylie collects 2 times more than Monday: \\( 25 \\times 2 = 50 \\) shells.\n\n**Final Answer:** Kylie collects \\(\\boxed{50}\\) shells on Tuesday.",
116
  "pass": true,
117
  "hit_token_limit": false
118
  },
@@ -124,7 +124,7 @@
124
  "expected": 75.0,
125
  "source": "openai/gsm8k",
126
  "source_revision": "740312add88f781978c0658806c59bc2815b9866",
127
- "output": "<think>\nOkay, let's see. Dax found half as many bugs as ants in the garden. If there were 50 ants, then the number of bugs should be half of that. So, 50 divided by 2 is 25. Therefore, there are 25 bugs. To find the total number of insects, I need to add the number of ants and the number of bugs together. That would be 50 ants plus 25 bugs, which equals 75 insects in total. Let me double-check that. If there are 50 ants and half as many bugs, then yes, 50 divided by 2 is 25. Adding them together gives 75. That seems right.\n</think>\n\nThe number of bugs is half the number of ants, so if there are 50 ants, there are \\( \\frac{50}{2} = 25 \\) bugs. The total number of insects is the sum of ants and bugs: \n\\[ 50 \\text{ ants} + 25 \\text{ bugs} = 75 \\text{ insects} \\] \n\n**Final Answer:** \nThe total number of insects in the garden is \\boxed{75}.",
128
  "pass": true,
129
  "hit_token_limit": false
130
  },
@@ -136,7 +136,7 @@
136
  "expected": 12.0,
137
  "source": "openai/gsm8k",
138
  "source_revision": "740312add88f781978c0658806c59bc2815b9866",
139
- "output": "<think>\nOkay, let's see. There are 36 penguins sunbathing in the snow. The problem says that one-third of them jump in and swim in the ocean. So, I need to find out how many that is. One-third of 36 is calculated by dividing 36 by 3. Let me do that: 36 divided by 3 equals 12. So, 12 penguins go swimming.\n\nThen, another one-third go inside the cave to eat their dinner. Again, that's one-third of the original 36 penguins. So, I need to calculate 36 divided by 3 once more. 36 divided by 3 is 12. So, 12 penguins go inside the cave.\n\nNow, the question is asking how many penguins are still left sunbathing. To find that, I need to subtract the number of penguins that went swimming and the number that went into the cave from the total number of penguins. So, that would be 36 minus 12 minus 12. Let me do that subtraction: 36 - 12 is 24, and then 24 - 12 is 12. So, 12 penguins are still left sunbathing.\n\nWait, let me double-check my calculations to make sure I didn't make a mistake. One-third of 36 is indeed 12, and another one-third is also 12. Subtracting those from 36 gives 12. That seems right. So, the answer should be 12 penguins still sunbathing.\n</think>\n\nThe problem states there are 36 penguins sunbathing. \n- **One-third** (36 \u00f7 3 = 12) swim in the ocean. \n- **Another one-third** (36 \u00f7 3 = 12) go inside the cave. \n\n**Penguins still sunbathing**: \n36 (total) - 12 (swimming) - 12 (cave) = **12**. \n\n**Answer**: 12 penguins are still sunbathing.",
140
  "pass": true,
141
  "hit_token_limit": false
142
  },
@@ -148,8 +148,8 @@
148
  "expected": 21.0,
149
  "source": "openai/gsm8k",
150
  "source_revision": "740312add88f781978c0658806c59bc2815b9866",
151
- "output": "<think>\nOkay, let's see. Steve eats twice as many tomatoes as his girlfriend. He eats 6 per day. So first, I need to find out how many tomatoes his girlfriend eats. If Steve eats twice as much, then his girlfriend eats half as much. So half of 6 is 3 tomatoes per day. \n\nNow, we need to find out how many tomatoes they eat together in a week. Since there are 7 days in a week, I'll multiply 3 tomatoes per day by 7 days. That gives 3 * 7 = 21 tomatoes per week for both of them together.\n\nEach vine produces 3 tomatoes per week. So to find out how many vines Steve needs, I'll divide the total number of tomatoes they need by the number of tomatoes per vine. That would be 21 tomatoes / 3 tomatoes per vine = 7 vines.\n\nWait, let me double-check that. If each vine produces 3 tomatoes per week, then 7 vines would produce 7 * 3 = 21 tomatoes per week. That matches the total they need. So yes, 7 vines should be enough.\n</think>\n\nSteve eats 6 tomatoes per day, and his girlfriend eats half that amount, so she eats \\( \\frac{6}{2} = 3 \\) tomatoes per day. Together, they eat \\( 6 + 3 = 9 \\) tomatoes per day. Over a week (7 days), they consume \\( 9 \\times 7 = 63 \\) tomatoes. Each vine produces 3 tomatoes per week, so Steve needs \\( \\frac{63}{3} = 21 \\) vines. \n\n**Answer:** Steve needs 21 vines.",
152
- "pass": true,
153
  "hit_token_limit": false
154
  },
155
  {
@@ -160,7 +160,7 @@
160
  "expected": 10.0,
161
  "source": "openai/gsm8k",
162
  "source_revision": "740312add88f781978c0658806c59bc2815b9866",
163
- "output": "<think>\nOkay, let's see. John wants to order pizzas for 20 friends, and each friend needs 4 slices. The pizzas come in 8 slices each. Hmm, so first, I need to figure out how many slices all the friends need in total. \n\nIf each friend gets 4 slices, then for 20 friends, that would be 20 multiplied by 4. Let me calculate that: 20 * 4 = 80 slices needed. \n\nNow, each pizza has 8 slices. So, to find out how many pizzas John needs, I need to divide the total number of slices required by the number of slices per pizza. That would be 80 divided by 8. Let me do that division: 80 \u00f7 8 = 10. \n\nSo, John needs to order 10 pizzas. That should give each friend 4 slices, and there won't be any leftovers. Let me double-check my math to make sure I didn't make a mistake. 10 pizzas times 8 slices each is 80 slices, which matches the total needed. Yep, that seems right. \n\nTherefore, John needs to order 10 pizzas.\n</think>\n\nJohn needs to order **10 pizzas** to ensure each of the 20 friends gets 4 slices. \n\n**Calculation:**\n- Total slices needed: \\( 20 \\text{ friends} \\times 4 \\text{ slices/friend} = 80 \\text{ slices} \\)\n- Pizzas required: \\( \\frac{80 \\text{ slices}}{8 \\text{ slices/pizza}} = 10 \\text{ pizzas} \\)\n\n**Answer:** 10 pizzas.",
164
  "pass": true,
165
  "hit_token_limit": false
166
  },
@@ -172,9 +172,9 @@
172
  "expected": 31.0,
173
  "source": "openai/gsm8k",
174
  "source_revision": "740312add88f781978c0658806c59bc2815b9866",
175
- "output": "<think>Okay, let's see. Ram buys 25 pens initially. He uses them to make new pens when the ink runs low. The key here is that he can mix the ink from five empty pens to make one full pen. So, every time he uses up five pens, he can make one new pen. \n\nFirst, let's figure out how many times he can make a new pen from the initial 25 pens. Since it takes five pens to make one, we divide 25 by 5. That gives us 5. So, he can make 5 new pens from the initial 25 pens.\n\nNow, each time he makes a new pen, he uses up five pens. So, after making 5 new pens, he would have used up 5 * 5 = 25 pens. Wait, that's interesting. He started with 25 pens and used up 25 pens to make 5 new ones. So, he ends up with 5 new pens. \n\nBut wait, does he have any pens left after making the new ones? Let's check. He started with 25 pens. He used 25 pens to make 5 new ones. So, he ends up with 25 - 25 = 0 pens. But that can't be right because he just made 5 new pens. Hmm, maybe I need to think differently.\n\nWait, no. When he makes a new pen, he uses five empty pens. So, he uses five pens to make one new pen. So, for each new pen he makes, he uses five pens. Therefore, the number of pens he can make is equal to the number of pens he starts with divided by five. So, 25 divided by 5 is 5. So, he can make 5 new pens. \n\nBut when he makes each new pen, he uses five pens. So, he starts with 25 pens. He uses 5 pens to make the first new pen, leaving him with 20 pens. Then he uses 5 pens to make the second new pen, leaving him with 15 pens. This continues until he makes 5 new pens. \n\nWait, but 25 divided by 5 is 5, so he can make 5 new pens. Each time he makes a new pen, he uses 5 pens. So, he starts with 25 pens. After making 1 new pen, he has 20 pens left. After making 2 new pens, he has 15 pens left. After making 3 new pens, he has 10 pens left. After making 4 new pens, he has 5 pens left. After making 5 new pens, he has 0 pens left. \n\nBut he ends up with 5 new pens. So, the total number of pens he has is the initial 25 pens plus the 5 new pens he made. So, 25 + 5 = 30 pens. \n\nWait, but that seems like he ends up with more pens than he started with. That doesn't make sense. If he uses 5 pens to make 1 new pen, he should have fewer pens in total. Let me check again.\n\nWait, no. When he makes a new pen, he uses 5 pens. So, for each new pen, he uses 5 pens. So, the number of pens he can make is equal to the number of pens he starts with divided by 5. So, 25 divided by 5 is 5. So, he can make 5 new pens. Each time he makes a new pen, he uses 5 pens. So, he starts with 25 pens. After making 1 new pen, he has 20 pens left. After making 2 new pens, he has 15 pens left. After making 3 new pens, he has 10 pens left. After making 4 new pens, he has 5 pens left. After making 5 new pens, he has 0 pens left. \n\nBut he ends up with 5 new pens. So, the total number of pens he has is the initial 25 pens plus the 5 new pens he made. So, 25 + 5 = 30 pens. \n\nBut that seems like he ends up with more pens than he started with. That doesn't make sense. If he uses 5 pens to make 1 new pen, he should have fewer pens in total. Let me think again.\n\nWait, no. When he makes a new pen, he uses 5 empty pens. So, he uses 5 pens to make 1 new pen. So, for each new pen, he uses 5 pens. Therefore, the number of new pens he can make is equal to the number of pens he starts with divided by 5. So, 25 divided by 5 is 5. So, he can make 5 new pens. \n\nBut when he makes each new pen, he uses 5 pens. So, he starts with 25 pens. After making 1 new pen, he has 20 pens left. After making 2 new pens, he has 15 pens left. After making 3 new pens, he has 10 pens left. After making 4 new pens, he has 5 pens left. After making 5 new pens, he has 0 pens left. \n\nBut he ends up with 5 new pens. So, the total number of pens he has is the initial 25 pens plus the 5 new pens he made. So, 25 + 5 = 30 pens. \n\nBut that seems like he ends up with more pens than he started with. That doesn't make sense. If he uses 5 pens to make 1 new pen, he should have fewer pens in total. Let me check the math again.\n\nWait, no. When he makes a new pen, he uses 5 empty pens. So, he uses 5 pens to make 1 new pen. Therefore, the number of new pens he can make is equal to the number of pens he starts with divided by 5. So, 25 divided by 5 is 5. So, he can make 5 new pens. \n\nBut when he makes each new pen, he uses 5 pens. So, he starts with 25 pens. After making 1 new pen, he has 20 pens left. After making 2 new pens, he has 15 pens left. After making 3 new pens, he has 10 pens left. After making 4 new pens, he has 5 pens left. After making 5 new pens, he has 0 pens left. \n\nBut he ends up with 5 new pens. So, the total number of pens he has is the initial 25 pens plus the 5 new pens he made. So, 25 + 5 = 30 pens. \n\nBut that seems like he ends up with more pens than he started with. That doesn't make sense. If he uses 5 pens to make 1 new pen, he should have fewer pens in total. Let me check the math again.\n\nWait, no. When he makes a new pen, he uses 5 empty pens. So, he uses 5 pens to make 1 new pen. Therefore, the number of new pens he can make is equal to the number of pens he starts with divided by 5. So, 25 divided by",
176
  "pass": false,
177
- "hit_token_limit": true
178
  },
179
  {
180
  "id": "gsm8k-test-237",
@@ -184,7 +184,7 @@
184
  "expected": 90.0,
185
  "source": "openai/gsm8k",
186
  "source_revision": "740312add88f781978c0658806c59bc2815b9866",
187
- "output": "<think>\nOkay, let's break this down step by step. \n\nFirst, Johnny starts with his allowance of $20. Then he adds an extra $10 to it. So, I need to add those two amounts together. \n\n20 + 10 equals 30. So, after adding the extra $10, Johnny has $30. \n\nNext, he invests this $30, and it triples in a year. Tripling means multiplying by 3. So, I need to multiply 30 by 3. \n\n30 times 3 is 90. \n\nTherefore, after a year, Johnny will have $90. \n\nWait, let me double-check my calculations to make sure I didn't make any mistakes. \n\nStarting amount: $20. Add $10: 20 + 10 = 30. Then triple that amount: 30 * 3 = 90. Yep, that seems right. \n\nSo, the final answer should be $90.\n</think>\n\nJohnny started with $20, added $10, and had a total of $30. This amount tripled in a year, resulting in: \n**$30 \u00d7 3 = $90**. \n\n**Final Answer:** \n\\boxed{90}",
188
  "pass": true,
189
  "hit_token_limit": false
190
  },
@@ -196,7 +196,7 @@
196
  "expected": 68.0,
197
  "source": "openai/gsm8k",
198
  "source_revision": "740312add88f781978c0658806c59bc2815b9866",
199
- "output": "<think>\nOkay, let's see. The problem says Mary is two years younger than Joan, and Joan is five years older than Jessa. Jessa is 20 years old. We need to find the sum of their ages.\n\nFirst, let's break down the information given. We know Jessa's age, which is 20. Then, Joan is five years older than Jessa. So, if Jessa is 20, Joan must be 20 + 5 = 25 years old. That makes sense.\n\nNext, Mary is two years younger than Joan. So, if Joan is 25, Mary would be 25 - 2 = 23 years old. \n\nNow, to find the sum of their ages, we just add up each of their ages: Jessa (20) + Joan (25) + Mary (23). Let me check that again. 20 + 25 is 45, and 45 + 23 is 68. So, the sum of their ages should be 68.\n\nWait, let me double-check my calculations to make sure I didn't make any mistakes. Jessa is 20, Joan is 25, Mary is 23. Adding them together: 20 + 25 is indeed 45, and 45 + 23 is 68. Yep, that seems right. I don't think I missed anything here. The problem gives all the necessary information, and I accounted for each person's age based on the relationships described. So, the final answer should be 68.\n</think>\n\nThe sum of the ages of the three girls is **68**.\n\n**Step-by-step explanation:**\n1. Jessa's age = 20 years.\n2. Joan is 5 years older than Jessa: \n Joan's age = 20 + 5 = **25 years**.\n3. Mary is 2 years younger than Joan: \n Mary's age = 25 - 2 = **23 years**.\n4. Sum of their ages: \n 20 (Jessa) + 25 (Joan) + 23 (Mary) = **68**.",
200
  "pass": true,
201
  "hit_token_limit": false
202
  }
@@ -326,7 +326,7 @@
326
  "expected": 75.0,
327
  "source": "openai/gsm8k",
328
  "source_revision": "740312add88f781978c0658806c59bc2815b9866",
329
- "output": "<think>There were 50/2 = 25 bugs in the garden.\nThe total number of insects in the garden is 25 + 50 = 75.</think>\nAnswer: 75",
330
  "pass": true,
331
  "hit_token_limit": false
332
  },
@@ -338,7 +338,7 @@
338
  "expected": 12.0,
339
  "source": "openai/gsm8k",
340
  "source_revision": "740312add88f781978c0658806c59bc2815b9866",
341
- "output": "<think>There are 36/3 = 12 penguins that jump in the ocean.\nThere are also 36/3 = 12 penguins that go inside the cave.\nSo, 12 + 12 = 24 penguins are no longer sunbathing.\nTherefore, 36 - 24 = 12 penguins are still sunbathing.</think>\nAnswer: 12",
342
  "pass": true,
343
  "hit_token_limit": false
344
  },
@@ -374,7 +374,7 @@
374
  "expected": 31.0,
375
  "source": "openai/gsm8k",
376
  "source_revision": "740312add88f781978c0658806c59bc2815b9866",
377
- "output": "<think>First find the total amount of ink he gets from the 25 pens: 25 pens * 5 pens/pens = 125 pens\nThen divide that number by the number of pens he can make from one set of five pens: 125 pens / 5 pens/pens = 25 pens</think>\nAnswer: 25",
378
  "pass": false,
379
  "hit_token_limit": false
380
  },
 
9
  "tests": "\n\nMETADATA = {\n 'author': 'jt',\n 'dataset': 'test'\n}\n\n\ndef check(candidate):\n assert candidate([1.0, 2.0, 3.9, 4.0, 5.0, 2.2], 0.3) == True\n assert candidate([1.0, 2.0, 3.9, 4.0, 5.0, 2.2], 0.05) == False\n assert candidate([1.0, 2.0, 5.9, 4.0, 5.0], 0.95) == True\n assert candidate([1.0, 2.0, 5.9, 4.0, 5.0], 0.8) == False\n assert candidate([1.0, 2.0, 3.0, 4.0, 5.0, 2.0], 0.1) == True\n assert candidate([1.1, 2.2, 3.1, 4.1, 5.1], 1.0) == True\n assert candidate([1.1, 2.2, 3.1, 4.1, 5.1], 0.5) == False\n\n\ncheck(has_close_elements)",
10
  "source": "openai/openai_humaneval",
11
  "source_revision": "7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544",
12
+ "output": "```python\nfrom typing import List\n\ndef has_close_elements(numbers: List[float], threshold: float) -> bool:\n for i in range(len(numbers)):\n for j in range(i + 1, len(numbers)):\n if abs(numbers[i] - numbers[j]) < threshold:\n return True\n return False\n```",
13
  "pass": true,
14
  "hit_token_limit": false
15
  },
 
22
  "tests": "\n\nMETADATA = {\n 'author': 'jt',\n 'dataset': 'test'\n}\n\n\ndef check(candidate):\n assert candidate([]) == False\n assert candidate([1, 2, -3, 1, 2, -3]) == False\n assert candidate([1, 2, -4, 5, 6]) == True\n assert candidate([1, -1, 2, -2, 5, -5, 4, -4]) == False\n assert candidate([1, -1, 2, -2, 5, -5, 4, -5]) == True\n assert candidate([1, -2, 2, -2, 5, -5, 4, -4]) == True\n\ncheck(below_zero)",
23
  "source": "openai/openai_humaneval",
24
  "source_revision": "7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544",
25
+ "output": "```python\nfrom typing import List\n\ndef below_zero(operations: List[int]) -> bool:\n balance = 0\n for operation in operations:\n balance += operation\n if balance < 0:\n return True\n return False\n```",
26
  "pass": true,
27
  "hit_token_limit": false
28
  },
 
35
  "tests": "\n\nMETADATA = {\n 'author': 'jt',\n 'dataset': 'test'\n}\n\n\ndef check(candidate):\n assert candidate([], 'john') == []\n assert candidate(['xxx', 'asd', 'xxy', 'john doe', 'xxxAAA', 'xxx'], 'xxx') == ['xxx', 'xxxAAA', 'xxx']\n assert candidate(['xxx', 'asd', 'aaaxxy', 'john doe', 'xxxAAA', 'xxx'], 'xx') == ['xxx', 'aaaxxy', 'xxxAAA', 'xxx']\n assert candidate(['grunt', 'trumpet', 'prune', 'gruesome'], 'run') == ['grunt', 'prune']\n\ncheck(filter_by_substring)",
36
  "source": "openai/openai_humaneval",
37
  "source_revision": "7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544",
38
+ "output": "```python\nfrom typing import List\n\ndef filter_by_substring(strings: List[str], substring: str) -> List[str]:\n return [s for s in strings if substring in s]\n```",
39
  "pass": true,
40
  "hit_token_limit": false
41
  },
 
48
  "tests": "\n\nMETADATA = {\n 'author': 'jt',\n 'dataset': 'test'\n}\n\n\ndef check(candidate):\n assert candidate([]) == None\n assert candidate(['x', 'y', 'z']) == 'x'\n assert candidate(['x', 'yyy', 'zzzz', 'www', 'kkkk', 'abc']) == 'zzzz'\n\ncheck(longest)",
49
  "source": "openai/openai_humaneval",
50
  "source_revision": "7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544",
51
+ "output": "```python\nfrom typing import List, Optional\n\ndef longest(strings: List[str]) -> Optional[str]:\n if not strings:\n return None\n return max(strings, key=len)\n```",
52
  "pass": true,
53
  "hit_token_limit": false
54
  },
 
61
  "tests": "\n\nMETADATA = {\n 'author': 'jt',\n 'dataset': 'test'\n}\n\n\ndef check(candidate):\n assert candidate('', 'x') == 0\n assert candidate('xyxyxyx', 'x') == 4\n assert candidate('cacacacac', 'cac') == 4\n assert candidate('john doe', 'john') == 1\n\ncheck(how_many_times)",
62
  "source": "openai/openai_humaneval",
63
  "source_revision": "7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544",
64
+ "output": "```python\ndef how_many_times(string: str, substring: str) -> int:\n return string.count(substring)\n```",
65
  "pass": false,
66
  "hit_token_limit": false
67
  },
 
74
  "tests": "\n\nMETADATA = {\n 'author': 'jt',\n 'dataset': 'test'\n}\n\n\ndef check(candidate):\n assert candidate(2) == [2]\n assert candidate(4) == [2, 2]\n assert candidate(8) == [2, 2, 2]\n assert candidate(3 * 19) == [3, 19]\n assert candidate(3 * 19 * 3 * 19) == [3, 3, 19, 19]\n assert candidate(3 * 19 * 3 * 19 * 3 * 19) == [3, 3, 3, 19, 19, 19]\n assert candidate(3 * 19 * 19 * 19) == [3, 19, 19, 19]\n assert candidate(3 * 2 * 3) == [2, 3, 3]\n\ncheck(factorize)",
75
  "source": "openai/openai_humaneval",
76
  "source_revision": "7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544",
77
+ "output": "```python\nfrom typing import List\n\ndef factorize(n: int) -> List[int]:\n factors = []\n divisor = 2\n while divisor * divisor <= n:\n if n % divisor == 0:\n factors.append(divisor)\n n //= divisor\n else:\n divisor += 1\n if n > 1:\n factors.append(n)\n return factors\n```",
78
  "pass": true,
79
  "hit_token_limit": false
80
  },
 
87
  "tests": "\n\nMETADATA = {}\n\n\ndef check(candidate):\n assert candidate(50) == 0\n assert candidate(78) == 2\n assert candidate(79) == 3\n assert candidate(100) == 3\n assert candidate(200) == 6\n assert candidate(4000) == 192\n assert candidate(10000) == 639\n assert candidate(100000) == 8026\n\n\ncheck(fizz_buzz)",
88
  "source": "openai/openai_humaneval",
89
  "source_revision": "7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544",
90
+ "output": "```python\nfrom itertools import count\n\ndef fizz_buzz(n: int) -> int:\n return sum(1 for i in count(1, 1) if i < n and (i % 11 == 0 or i % 13 == 0) and str(i).count('7'))\n```",
91
+ "pass": false,
92
  "hit_token_limit": false
93
  },
94
  {
 
100
  "tests": "\n\nMETADATA = {}\n\n\ndef check(candidate):\n assert candidate(5, 3) == 7.5\n assert candidate(2, 2) == 2.0\n assert candidate(10, 8) == 40.0\n\n\ncheck(triangle_area)",
101
  "source": "openai/openai_humaneval",
102
  "source_revision": "7dce6050a7d6d172f3cc5c32aa97f52fa1a2e544",
103
+ "output": "```python\nimport math\n\ndef triangle_area(a, h):\n return 0.5 * a * h\n```",
104
  "pass": true,
105
  "hit_token_limit": false
106
  },
 
112
  "expected": 50.0,
113
  "source": "openai/gsm8k",
114
  "source_revision": "740312add88f781978c0658806c59bc2815b9866",
115
+ "output": "<think>On Monday, Kylie collects 5 + 20 = 25 shells.\nOn Tuesday, Kylie collects 2 * 25 = 50 shells.</think>\nAnswer: 50",
116
  "pass": true,
117
  "hit_token_limit": false
118
  },
 
124
  "expected": 75.0,
125
  "source": "openai/gsm8k",
126
  "source_revision": "740312add88f781978c0658806c59bc2815b9866",
127
+ "output": "<think>There were 50/2 = 25 bugs in the garden.\nThe total number of insects in the garden is 25 + 50 = 75.</think>\nAnswer: 75",
128
  "pass": true,
129
  "hit_token_limit": false
130
  },
 
136
  "expected": 12.0,
137
  "source": "openai/gsm8k",
138
  "source_revision": "740312add88f781978c0658806c59bc2815b9866",
139
+ "output": "<think>There are 36/3 = 12 penguins that jump in the ocean.\nThere are also 36/3 = 12 penguins that go inside the cave.\nSo, 12 + 12 = 24 penguins are no longer sunbathing.\nTherefore, 36 - 24 = 12 penguins are still sunbathing.</think>\nAnswer: 12",
140
  "pass": true,
141
  "hit_token_limit": false
142
  },
 
148
  "expected": 21.0,
149
  "source": "openai/gsm8k",
150
  "source_revision": "740312add88f781978c0658806c59bc2815b9866",
151
+ "output": "<think>He eats 6 x 2 = 12 tomatoes per week\nHe needs 12 / 3 = 4 vines</think>\nAnswer: 4",
152
+ "pass": false,
153
  "hit_token_limit": false
154
  },
155
  {
 
160
  "expected": 10.0,
161
  "source": "openai/gsm8k",
162
  "source_revision": "740312add88f781978c0658806c59bc2815b9866",
163
+ "output": "<think>First find the total number of slices needed by multiplying the number of friends by the number of slices each wants: 20 * 4 = 80\nThen divide that by the number of slices per pizza to find the number of pizzas needed: 80 / 8 = 10</think>\nAnswer: 10",
164
  "pass": true,
165
  "hit_token_limit": false
166
  },
 
172
  "expected": 31.0,
173
  "source": "openai/gsm8k",
174
  "source_revision": "740312add88f781978c0658806c59bc2815b9866",
175
+ "output": "<think>First find the total amount of ink he gets from the 25 pens: 25 pens * 5 pens/pens = 125 pens\nThen divide that number by the number of pens he can make from one set of five pens: 125 pens / 5 pens/pens = 25 pens</think>\nAnswer: 25",
176
  "pass": false,
177
+ "hit_token_limit": false
178
  },
179
  {
180
  "id": "gsm8k-test-237",
 
184
  "expected": 90.0,
185
  "source": "openai/gsm8k",
186
  "source_revision": "740312add88f781978c0658806c59bc2815b9866",
187
+ "output": "<think>Johnny had a total of 20+10 = $30 to invest.\nThis amount tripled to 30*3 = $90.</think>\nAnswer: 90",
188
  "pass": true,
189
  "hit_token_limit": false
190
  },
 
196
  "expected": 68.0,
197
  "source": "openai/gsm8k",
198
  "source_revision": "740312add88f781978c0658806c59bc2815b9866",
199
+ "output": "<think>Joan is 20 + 5 = 25 years old.\nMary is 25 - 2 = 23 years old.\nThe sum of their ages is 23 + 25 + 20 = 68 years.</think>\nAnswer: 68",
200
  "pass": true,
201
  "hit_token_limit": false
202
  }
 
326
  "expected": 75.0,
327
  "source": "openai/gsm8k",
328
  "source_revision": "740312add88f781978c0658806c59bc2815b9866",
329
+ "output": "<think>There were 50/2 = 25 bugs in the garden.\nThe total number of insects in the garden is 25+50 = 75.</think>\nAnswer: 75",
330
  "pass": true,
331
  "hit_token_limit": false
332
  },
 
338
  "expected": 12.0,
339
  "source": "openai/gsm8k",
340
  "source_revision": "740312add88f781978c0658806c59bc2815b9866",
341
+ "output": "<think>There are 36/3 = 12 penguins that jump in the ocean.\nThere are 36/3 = 12 penguins that go inside the cave.\n12 + 12 = 24 penguins are no longer sunbathing.\n36 - 24 = 12 penguins are still sunbathing.</think>\nAnswer: 12",
342
  "pass": true,
343
  "hit_token_limit": false
344
  },
 
374
  "expected": 31.0,
375
  "source": "openai/gsm8k",
376
  "source_revision": "740312add88f781978c0658806c59bc2815b9866",
377
+ "output": "<think>First find the total amount of ink he gets from the 25 pens: 25 pens * 5 pens/pens = 125 pens\nThen divide that number by the number of pens he can make from one set of empty pens: 125 pens / 5 pens/pens = 25 pens</think>\nAnswer: 25",
378
  "pass": false,
379
  "hit_token_limit": false
380
  },
model-00001-of-00005.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:e462f48a2622c389212fc32c8108cdfd7cfd51d2c9f4041b08f0815ce749fdb4
3
  size 3986791344
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6e4a9bfd682280e0271635d2b4329938f740373ce7d1fe3821d25b210bfbe45f
3
  size 3986791344
model-00002-of-00005.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:0e066f0d6991f85d05ab24263ffa4b56d39df5014dd4332148173c8876d49fc2
3
  size 3926025416
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ef0783b675748defc6ffc0a64580c578166ac499c5b83b64ceff8d5f60d12801
3
  size 3926025416
model-00003-of-00005.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:c0cc643be3c2dec8827139aebd24a8a85d94d9edd4833ea513d785adebee01df
3
  size 3926025440
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:30874d39316469ed6bff9cad62a9a576497dc47bc89886fbc67e8454f2733b09
3
  size 3926025440
model-00004-of-00005.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:185e72dd757bfc06dcbb4b279439bc9ac0c553a62bc0cbae253a90a2a2f4d79c
3
  size 3171040864
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6fa29d21bcb34fb555d70300cdd2fcb1932bbf2c4c5f2d7a985f0532a1601dce
3
  size 3171040864
paired_bf16_outputs.json ADDED
@@ -0,0 +1,1026 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "base": [
3
+ {
4
+ "id": "heldout_00",
5
+ "prompt": "Return only the integer: 701 + 91 - 31.",
6
+ "expected": "761",
7
+ "kind": "integer",
8
+ "output": "761",
9
+ "pass": true
10
+ },
11
+ {
12
+ "id": "heldout_01",
13
+ "prompt": "A crate has 704 red and 92 blue beads. Remove 32 red beads. Return only the total beads remaining.",
14
+ "expected": "764",
15
+ "kind": "integer",
16
+ "output": "The number of red beads left is 672. The number of blue beads is still 92, for a total of 764 beads.",
17
+ "pass": false
18
+ },
19
+ {
20
+ "id": "heldout_02",
21
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_707, count=93.",
22
+ "expected": {
23
+ "item": "part_707",
24
+ "count": 93
25
+ },
26
+ "kind": "json",
27
+ "output": "{\"item\": \"part_707\", \"count\": 93}",
28
+ "pass": true
29
+ },
30
+ {
31
+ "id": "heldout_03",
32
+ "prompt": "Return only one CSV value row, no header: name=Item710; quantity=94; code=K34.",
33
+ "expected": "Item710,94,K34",
34
+ "kind": "exact",
35
+ "output": "Item710,94,K34",
36
+ "pass": true
37
+ },
38
+ {
39
+ "id": "heldout_04",
40
+ "prompt": "Copy exactly, no other text: Key_713-B95_C35",
41
+ "expected": "Key_713-B95_C35",
42
+ "kind": "exact",
43
+ "output": "Key_713-B95_C35",
44
+ "pass": true
45
+ },
46
+ {
47
+ "id": "heldout_05",
48
+ "prompt": "Write only a Python function named solve(x) that returns x * 96 + 36.",
49
+ "expected": "x * 96 + 36",
50
+ "kind": "code",
51
+ "output": "```python\ndef solve(x):\n return x * 96 + 36\n```",
52
+ "pass": true
53
+ },
54
+ {
55
+ "id": "heldout_06",
56
+ "prompt": "Return only the integer: 719 + 97 - 37.",
57
+ "expected": "779",
58
+ "kind": "integer",
59
+ "output": "679",
60
+ "pass": false
61
+ },
62
+ {
63
+ "id": "heldout_07",
64
+ "prompt": "A crate has 722 red and 98 blue beads. Remove 38 red beads. Return only the total beads remaining.",
65
+ "expected": "782",
66
+ "kind": "integer",
67
+ "output": "The number of red beads left is 722 - 38 = 684. The total number of beads left is 684 + 98 = 782.",
68
+ "pass": false
69
+ },
70
+ {
71
+ "id": "heldout_08",
72
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_725, count=99.",
73
+ "expected": {
74
+ "item": "part_725",
75
+ "count": 99
76
+ },
77
+ "kind": "json",
78
+ "output": "{\"item\": \"part_725\", \"count\": 99}",
79
+ "pass": true
80
+ },
81
+ {
82
+ "id": "heldout_09",
83
+ "prompt": "Return only one CSV value row, no header: name=Item728; quantity=100; code=K40.",
84
+ "expected": "Item728,100,K40",
85
+ "kind": "exact",
86
+ "output": "Item728,100,K40",
87
+ "pass": true
88
+ },
89
+ {
90
+ "id": "heldout_10",
91
+ "prompt": "Copy exactly, no other text: Key_731-B101_C41",
92
+ "expected": "Key_731-B101_C41",
93
+ "kind": "exact",
94
+ "output": "Key_731-B101_C41",
95
+ "pass": true
96
+ },
97
+ {
98
+ "id": "heldout_11",
99
+ "prompt": "Write only a Python function named solve(x) that returns x * 102 + 42.",
100
+ "expected": "x * 102 + 42",
101
+ "kind": "code",
102
+ "output": "```python\ndef solve(x):\n return x * 102 + 42\n```",
103
+ "pass": true
104
+ },
105
+ {
106
+ "id": "heldout_12",
107
+ "prompt": "Return only the integer: 737 + 103 - 43.",
108
+ "expected": "797",
109
+ "kind": "integer",
110
+ "output": "697",
111
+ "pass": false
112
+ },
113
+ {
114
+ "id": "heldout_13",
115
+ "prompt": "A crate has 740 red and 104 blue beads. Remove 44 red beads. Return only the total beads remaining.",
116
+ "expected": "800",
117
+ "kind": "integer",
118
+ "output": "The number of red beads left is 740 - 44 = 696.\nThe total number of beads left is 696 + 104 = 800.",
119
+ "pass": false
120
+ },
121
+ {
122
+ "id": "heldout_14",
123
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_743, count=105.",
124
+ "expected": {
125
+ "item": "part_743",
126
+ "count": 105
127
+ },
128
+ "kind": "json",
129
+ "output": "{\"item\": \"part_743\", \"count\": 105}",
130
+ "pass": true
131
+ },
132
+ {
133
+ "id": "heldout_15",
134
+ "prompt": "Return only one CSV value row, no header: name=Item746; quantity=106; code=K46.",
135
+ "expected": "Item746,106,K46",
136
+ "kind": "exact",
137
+ "output": "Item746,106,K46",
138
+ "pass": true
139
+ },
140
+ {
141
+ "id": "heldout_16",
142
+ "prompt": "Copy exactly, no other text: Key_749-B107_C47",
143
+ "expected": "Key_749-B107_C47",
144
+ "kind": "exact",
145
+ "output": "Key_749-B107_C47",
146
+ "pass": true
147
+ },
148
+ {
149
+ "id": "heldout_17",
150
+ "prompt": "Write only a Python function named solve(x) that returns x * 108 + 31.",
151
+ "expected": "x * 108 + 31",
152
+ "kind": "code",
153
+ "output": "```python\ndef solve(x):\n return x * 108 + 31\n```",
154
+ "pass": true
155
+ },
156
+ {
157
+ "id": "heldout_18",
158
+ "prompt": "Return only the integer: 755 + 109 - 32.",
159
+ "expected": "832",
160
+ "kind": "integer",
161
+ "output": "732",
162
+ "pass": false
163
+ },
164
+ {
165
+ "id": "heldout_19",
166
+ "prompt": "A crate has 758 red and 110 blue beads. Remove 33 red beads. Return only the total beads remaining.",
167
+ "expected": "835",
168
+ "kind": "integer",
169
+ "output": "The number of red beads left is 758 - 33 = 725. The total number of beads left is 725 + 110 = 835.",
170
+ "pass": false
171
+ },
172
+ {
173
+ "id": "heldout_20",
174
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_761, count=111.",
175
+ "expected": {
176
+ "item": "part_761",
177
+ "count": 111
178
+ },
179
+ "kind": "json",
180
+ "output": "{\"item\": \"part_761\", \"count\": 111}",
181
+ "pass": true
182
+ },
183
+ {
184
+ "id": "heldout_21",
185
+ "prompt": "Return only one CSV value row, no header: name=Item764; quantity=112; code=K35.",
186
+ "expected": "Item764,112,K35",
187
+ "kind": "exact",
188
+ "output": "Item764,112,K35",
189
+ "pass": true
190
+ },
191
+ {
192
+ "id": "heldout_22",
193
+ "prompt": "Copy exactly, no other text: Key_767-B113_C36",
194
+ "expected": "Key_767-B113_C36",
195
+ "kind": "exact",
196
+ "output": "Key_767-B113_C36",
197
+ "pass": true
198
+ },
199
+ {
200
+ "id": "heldout_23",
201
+ "prompt": "Write only a Python function named solve(x) that returns x * 114 + 37.",
202
+ "expected": "x * 114 + 37",
203
+ "kind": "code",
204
+ "output": "```python\ndef solve(x):\n return x * 114 + 37\n```",
205
+ "pass": true
206
+ },
207
+ {
208
+ "id": "heldout_24",
209
+ "prompt": "Return only the integer: 773 + 115 - 38.",
210
+ "expected": "850",
211
+ "kind": "integer",
212
+ "output": "750",
213
+ "pass": false
214
+ },
215
+ {
216
+ "id": "heldout_25",
217
+ "prompt": "A crate has 776 red and 116 blue beads. Remove 39 red beads. Return only the total beads remaining.",
218
+ "expected": "853",
219
+ "kind": "integer",
220
+ "output": "The number of red beads left is 737 because 776 - 39 = 737. The total number of beads left is 737 + 116 = 853.",
221
+ "pass": false
222
+ },
223
+ {
224
+ "id": "heldout_26",
225
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_779, count=117.",
226
+ "expected": {
227
+ "item": "part_779",
228
+ "count": 117
229
+ },
230
+ "kind": "json",
231
+ "output": "{\"item\": \"part_779\", \"count\": 117}",
232
+ "pass": true
233
+ },
234
+ {
235
+ "id": "heldout_27",
236
+ "prompt": "Return only one CSV value row, no header: name=Item782; quantity=118; code=K41.",
237
+ "expected": "Item782,118,K41",
238
+ "kind": "exact",
239
+ "output": "Item782,118,K41",
240
+ "pass": true
241
+ },
242
+ {
243
+ "id": "heldout_28",
244
+ "prompt": "Copy exactly, no other text: Key_785-B119_C42",
245
+ "expected": "Key_785-B119_C42",
246
+ "kind": "exact",
247
+ "output": "Key_785-B119_C42",
248
+ "pass": true
249
+ },
250
+ {
251
+ "id": "heldout_29",
252
+ "prompt": "Write only a Python function named solve(x) that returns x * 120 + 43.",
253
+ "expected": "x * 120 + 43",
254
+ "kind": "code",
255
+ "output": "```python\ndef solve(x):\n return x * 120 + 43\n```",
256
+ "pass": true
257
+ },
258
+ {
259
+ "id": "heldout_30",
260
+ "prompt": "Return only the integer: 791 + 121 - 44.",
261
+ "expected": "868",
262
+ "kind": "integer",
263
+ "output": "768",
264
+ "pass": false
265
+ },
266
+ {
267
+ "id": "heldout_31",
268
+ "prompt": "A crate has 794 red and 122 blue beads. Remove 45 red beads. Return only the total beads remaining.",
269
+ "expected": "871",
270
+ "kind": "integer",
271
+ "output": "The number of red beads left is 794 - 45 = 749. The total number of beads left is 749 + 122 = 871.",
272
+ "pass": false
273
+ },
274
+ {
275
+ "id": "heldout_32",
276
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_797, count=123.",
277
+ "expected": {
278
+ "item": "part_797",
279
+ "count": 123
280
+ },
281
+ "kind": "json",
282
+ "output": "{\"item\": \"part_797\", \"count\": 123}",
283
+ "pass": true
284
+ },
285
+ {
286
+ "id": "heldout_33",
287
+ "prompt": "Return only one CSV value row, no header: name=Item800; quantity=124; code=K47.",
288
+ "expected": "Item800,124,K47",
289
+ "kind": "exact",
290
+ "output": "Item800,124,K47",
291
+ "pass": true
292
+ },
293
+ {
294
+ "id": "heldout_34",
295
+ "prompt": "Copy exactly, no other text: Key_803-B125_C31",
296
+ "expected": "Key_803-B125_C31",
297
+ "kind": "exact",
298
+ "output": "Key_803-B125_C31",
299
+ "pass": true
300
+ },
301
+ {
302
+ "id": "heldout_35",
303
+ "prompt": "Write only a Python function named solve(x) that returns x * 126 + 32.",
304
+ "expected": "x * 126 + 32",
305
+ "kind": "code",
306
+ "output": "```python\ndef solve(x):\n return x * 126 + 32\n```",
307
+ "pass": true
308
+ },
309
+ {
310
+ "id": "heldout_36",
311
+ "prompt": "Return only the integer: 809 + 127 - 33.",
312
+ "expected": "903",
313
+ "kind": "integer",
314
+ "output": "803",
315
+ "pass": false
316
+ },
317
+ {
318
+ "id": "heldout_37",
319
+ "prompt": "A crate has 812 red and 128 blue beads. Remove 34 red beads. Return only the total beads remaining.",
320
+ "expected": "906",
321
+ "kind": "integer",
322
+ "output": "The number of red beads left is 812 - 34 = 778. The total number of beads left is 778 + 128 = 906.",
323
+ "pass": false
324
+ },
325
+ {
326
+ "id": "heldout_38",
327
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_815, count=129.",
328
+ "expected": {
329
+ "item": "part_815",
330
+ "count": 129
331
+ },
332
+ "kind": "json",
333
+ "output": "{\"item\": \"part_815\", \"count\": 129}",
334
+ "pass": true
335
+ },
336
+ {
337
+ "id": "heldout_39",
338
+ "prompt": "Return only one CSV value row, no header: name=Item818; quantity=130; code=K36.",
339
+ "expected": "Item818,130,K36",
340
+ "kind": "exact",
341
+ "output": "Item818,130,K36",
342
+ "pass": true
343
+ },
344
+ {
345
+ "id": "heldout_40",
346
+ "prompt": "Copy exactly, no other text: Key_821-B131_C37",
347
+ "expected": "Key_821-B131_C37",
348
+ "kind": "exact",
349
+ "output": "Key_821-B131_C37",
350
+ "pass": true
351
+ },
352
+ {
353
+ "id": "heldout_41",
354
+ "prompt": "Write only a Python function named solve(x) that returns x * 132 + 38.",
355
+ "expected": "x * 132 + 38",
356
+ "kind": "code",
357
+ "output": "```python\ndef solve(x):\n return x * 132 + 38\n```",
358
+ "pass": true
359
+ },
360
+ {
361
+ "id": "heldout_42",
362
+ "prompt": "Return only the integer: 827 + 133 - 39.",
363
+ "expected": "921",
364
+ "kind": "integer",
365
+ "output": "841",
366
+ "pass": false
367
+ },
368
+ {
369
+ "id": "heldout_43",
370
+ "prompt": "A crate has 830 red and 134 blue beads. Remove 40 red beads. Return only the total beads remaining.",
371
+ "expected": "924",
372
+ "kind": "integer",
373
+ "output": "The number of red beads left is 830 - 40 = 790. The total number of beads left is 790 + 134 = 924.",
374
+ "pass": false
375
+ },
376
+ {
377
+ "id": "heldout_44",
378
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_833, count=135.",
379
+ "expected": {
380
+ "item": "part_833",
381
+ "count": 135
382
+ },
383
+ "kind": "json",
384
+ "output": "{\"item\": \"part_833\", \"count\": 135}",
385
+ "pass": true
386
+ },
387
+ {
388
+ "id": "heldout_45",
389
+ "prompt": "Return only one CSV value row, no header: name=Item836; quantity=136; code=K42.",
390
+ "expected": "Item836,136,K42",
391
+ "kind": "exact",
392
+ "output": "Item836,136,K42",
393
+ "pass": true
394
+ },
395
+ {
396
+ "id": "heldout_46",
397
+ "prompt": "Copy exactly, no other text: Key_839-B137_C43",
398
+ "expected": "Key_839-B137_C43",
399
+ "kind": "exact",
400
+ "output": "Key_839-B137_C43",
401
+ "pass": true
402
+ },
403
+ {
404
+ "id": "heldout_47",
405
+ "prompt": "Write only a Python function named solve(x) that returns x * 138 + 44.",
406
+ "expected": "x * 138 + 44",
407
+ "kind": "code",
408
+ "output": "```python\ndef solve(x):\n return x * 138 + 44\n```",
409
+ "pass": true
410
+ },
411
+ {
412
+ "id": "heldout_48",
413
+ "prompt": "Return only the integer: 845 + 139 - 45.",
414
+ "expected": "939",
415
+ "kind": "integer",
416
+ "output": "839",
417
+ "pass": false
418
+ },
419
+ {
420
+ "id": "heldout_49",
421
+ "prompt": "A crate has 848 red and 140 blue beads. Remove 46 red beads. Return only the total beads remaining.",
422
+ "expected": "942",
423
+ "kind": "integer",
424
+ "output": "The number of red beads left is 848 - 46 = 802.\nThe total number of beads left is 802 + 140 = 942.",
425
+ "pass": false
426
+ },
427
+ {
428
+ "id": "heldout_50",
429
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_851, count=141.",
430
+ "expected": {
431
+ "item": "part_851",
432
+ "count": 141
433
+ },
434
+ "kind": "json",
435
+ "output": "{\"item\": \"part_851\", \"count\": 141}",
436
+ "pass": true
437
+ },
438
+ {
439
+ "id": "heldout_51",
440
+ "prompt": "Return only one CSV value row, no header: name=Item854; quantity=142; code=K31.",
441
+ "expected": "Item854,142,K31",
442
+ "kind": "exact",
443
+ "output": "Item854,142,K31",
444
+ "pass": true
445
+ },
446
+ {
447
+ "id": "heldout_52",
448
+ "prompt": "Copy exactly, no other text: Key_857-B143_C32",
449
+ "expected": "Key_857-B143_C32",
450
+ "kind": "exact",
451
+ "output": "Key_857-B143_C32",
452
+ "pass": true
453
+ },
454
+ {
455
+ "id": "heldout_53",
456
+ "prompt": "Write only a Python function named solve(x) that returns x * 144 + 33.",
457
+ "expected": "x * 144 + 33",
458
+ "kind": "code",
459
+ "output": "```python\ndef solve(x):\n return x * 144 + 33\n```",
460
+ "pass": true
461
+ },
462
+ {
463
+ "id": "heldout_54",
464
+ "prompt": "Return only the integer: 863 + 145 - 34.",
465
+ "expected": "974",
466
+ "kind": "integer",
467
+ "output": "874",
468
+ "pass": false
469
+ },
470
+ {
471
+ "id": "heldout_55",
472
+ "prompt": "A crate has 866 red and 146 blue beads. Remove 35 red beads. Return only the total beads remaining.",
473
+ "expected": "977",
474
+ "kind": "integer",
475
+ "output": "The number of red beads left is 866 - 35 = 831. The total number of beads left is 831 + 146 = 977.",
476
+ "pass": false
477
+ },
478
+ {
479
+ "id": "heldout_56",
480
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_869, count=147.",
481
+ "expected": {
482
+ "item": "part_869",
483
+ "count": 147
484
+ },
485
+ "kind": "json",
486
+ "output": "{\"item\": \"part_869\", \"count\": 147}",
487
+ "pass": true
488
+ },
489
+ {
490
+ "id": "heldout_57",
491
+ "prompt": "Return only one CSV value row, no header: name=Item872; quantity=148; code=K37.",
492
+ "expected": "Item872,148,K37",
493
+ "kind": "exact",
494
+ "output": "Item872,148,K37",
495
+ "pass": true
496
+ },
497
+ {
498
+ "id": "heldout_58",
499
+ "prompt": "Copy exactly, no other text: Key_875-B149_C38",
500
+ "expected": "Key_875-B149_C38",
501
+ "kind": "exact",
502
+ "output": "Key_875-B149_C38",
503
+ "pass": true
504
+ },
505
+ {
506
+ "id": "heldout_59",
507
+ "prompt": "Write only a Python function named solve(x) that returns x * 150 + 39.",
508
+ "expected": "x * 150 + 39",
509
+ "kind": "code",
510
+ "output": "```python\ndef solve(x):\n return x * 150 + 39\n```",
511
+ "pass": true
512
+ }
513
+ ],
514
+ "astheral": [
515
+ {
516
+ "id": "heldout_00",
517
+ "prompt": "Return only the integer: 701 + 91 - 31.",
518
+ "expected": "761",
519
+ "kind": "integer",
520
+ "output": "761",
521
+ "pass": true
522
+ },
523
+ {
524
+ "id": "heldout_01",
525
+ "prompt": "A crate has 704 red and 92 blue beads. Remove 32 red beads. Return only the total beads remaining.",
526
+ "expected": "764",
527
+ "kind": "integer",
528
+ "output": "The number of red beads left is 672. The number of blue beads is still 92, for a total of 764 beads.",
529
+ "pass": false
530
+ },
531
+ {
532
+ "id": "heldout_02",
533
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_707, count=93.",
534
+ "expected": {
535
+ "item": "part_707",
536
+ "count": 93
537
+ },
538
+ "kind": "json",
539
+ "output": "{\"item\": \"part_707\", \"count\": 93}",
540
+ "pass": true
541
+ },
542
+ {
543
+ "id": "heldout_03",
544
+ "prompt": "Return only one CSV value row, no header: name=Item710; quantity=94; code=K34.",
545
+ "expected": "Item710,94,K34",
546
+ "kind": "exact",
547
+ "output": "Item710,94,K34",
548
+ "pass": true
549
+ },
550
+ {
551
+ "id": "heldout_04",
552
+ "prompt": "Copy exactly, no other text: Key_713-B95_C35",
553
+ "expected": "Key_713-B95_C35",
554
+ "kind": "exact",
555
+ "output": "Key_713-B95_C35",
556
+ "pass": true
557
+ },
558
+ {
559
+ "id": "heldout_05",
560
+ "prompt": "Write only a Python function named solve(x) that returns x * 96 + 36.",
561
+ "expected": "x * 96 + 36",
562
+ "kind": "code",
563
+ "output": "```python\ndef solve(x):\n return x * 96 + 36\n```",
564
+ "pass": true
565
+ },
566
+ {
567
+ "id": "heldout_06",
568
+ "prompt": "Return only the integer: 719 + 97 - 37.",
569
+ "expected": "779",
570
+ "kind": "integer",
571
+ "output": "779",
572
+ "pass": true
573
+ },
574
+ {
575
+ "id": "heldout_07",
576
+ "prompt": "A crate has 722 red and 98 blue beads. Remove 38 red beads. Return only the total beads remaining.",
577
+ "expected": "782",
578
+ "kind": "integer",
579
+ "output": "The number of red beads left is 684. The number of beads left is 684 + 98 = 782.",
580
+ "pass": false
581
+ },
582
+ {
583
+ "id": "heldout_08",
584
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_725, count=99.",
585
+ "expected": {
586
+ "item": "part_725",
587
+ "count": 99
588
+ },
589
+ "kind": "json",
590
+ "output": "{\"item\": \"part_725\", \"count\": 99}",
591
+ "pass": true
592
+ },
593
+ {
594
+ "id": "heldout_09",
595
+ "prompt": "Return only one CSV value row, no header: name=Item728; quantity=100; code=K40.",
596
+ "expected": "Item728,100,K40",
597
+ "kind": "exact",
598
+ "output": "Item728,100,K40",
599
+ "pass": true
600
+ },
601
+ {
602
+ "id": "heldout_10",
603
+ "prompt": "Copy exactly, no other text: Key_731-B101_C41",
604
+ "expected": "Key_731-B101_C41",
605
+ "kind": "exact",
606
+ "output": "Key_731-B101_C41",
607
+ "pass": true
608
+ },
609
+ {
610
+ "id": "heldout_11",
611
+ "prompt": "Write only a Python function named solve(x) that returns x * 102 + 42.",
612
+ "expected": "x * 102 + 42",
613
+ "kind": "code",
614
+ "output": "```python\ndef solve(x):\n return x * 102 + 42\n```",
615
+ "pass": true
616
+ },
617
+ {
618
+ "id": "heldout_12",
619
+ "prompt": "Return only the integer: 737 + 103 - 43.",
620
+ "expected": "797",
621
+ "kind": "integer",
622
+ "output": "797",
623
+ "pass": true
624
+ },
625
+ {
626
+ "id": "heldout_13",
627
+ "prompt": "A crate has 740 red and 104 blue beads. Remove 44 red beads. Return only the total beads remaining.",
628
+ "expected": "800",
629
+ "kind": "integer",
630
+ "output": "The number of red beads left is 696. The number of blue beads is unchanged at 104, for a total of 800 beads remaining.",
631
+ "pass": false
632
+ },
633
+ {
634
+ "id": "heldout_14",
635
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_743, count=105.",
636
+ "expected": {
637
+ "item": "part_743",
638
+ "count": 105
639
+ },
640
+ "kind": "json",
641
+ "output": "{\"item\": \"part_743\", \"count\": 105}",
642
+ "pass": true
643
+ },
644
+ {
645
+ "id": "heldout_15",
646
+ "prompt": "Return only one CSV value row, no header: name=Item746; quantity=106; code=K46.",
647
+ "expected": "Item746,106,K46",
648
+ "kind": "exact",
649
+ "output": "Item746,106,K46",
650
+ "pass": true
651
+ },
652
+ {
653
+ "id": "heldout_16",
654
+ "prompt": "Copy exactly, no other text: Key_749-B107_C47",
655
+ "expected": "Key_749-B107_C47",
656
+ "kind": "exact",
657
+ "output": "Key_749-B107_C47",
658
+ "pass": true
659
+ },
660
+ {
661
+ "id": "heldout_17",
662
+ "prompt": "Write only a Python function named solve(x) that returns x * 108 + 31.",
663
+ "expected": "x * 108 + 31",
664
+ "kind": "code",
665
+ "output": "```python\ndef solve(x):\n return x * 108 + 31\n```",
666
+ "pass": true
667
+ },
668
+ {
669
+ "id": "heldout_18",
670
+ "prompt": "Return only the integer: 755 + 109 - 32.",
671
+ "expected": "832",
672
+ "kind": "integer",
673
+ "output": "732",
674
+ "pass": false
675
+ },
676
+ {
677
+ "id": "heldout_19",
678
+ "prompt": "A crate has 758 red and 110 blue beads. Remove 33 red beads. Return only the total beads remaining.",
679
+ "expected": "835",
680
+ "kind": "integer",
681
+ "output": "The number of red beads left is 758 - 33 = 725. The total number of beads left is 725 + 110 = 835.",
682
+ "pass": false
683
+ },
684
+ {
685
+ "id": "heldout_20",
686
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_761, count=111.",
687
+ "expected": {
688
+ "item": "part_761",
689
+ "count": 111
690
+ },
691
+ "kind": "json",
692
+ "output": "{\"item\": \"part_761\", \"count\": 111}",
693
+ "pass": true
694
+ },
695
+ {
696
+ "id": "heldout_21",
697
+ "prompt": "Return only one CSV value row, no header: name=Item764; quantity=112; code=K35.",
698
+ "expected": "Item764,112,K35",
699
+ "kind": "exact",
700
+ "output": "Item764,112,K35",
701
+ "pass": true
702
+ },
703
+ {
704
+ "id": "heldout_22",
705
+ "prompt": "Copy exactly, no other text: Key_767-B113_C36",
706
+ "expected": "Key_767-B113_C36",
707
+ "kind": "exact",
708
+ "output": "Key_767-B113_C36",
709
+ "pass": true
710
+ },
711
+ {
712
+ "id": "heldout_23",
713
+ "prompt": "Write only a Python function named solve(x) that returns x * 114 + 37.",
714
+ "expected": "x * 114 + 37",
715
+ "kind": "code",
716
+ "output": "```python\ndef solve(x):\n return x * 114 + 37\n```",
717
+ "pass": true
718
+ },
719
+ {
720
+ "id": "heldout_24",
721
+ "prompt": "Return only the integer: 773 + 115 - 38.",
722
+ "expected": "850",
723
+ "kind": "integer",
724
+ "output": "750",
725
+ "pass": false
726
+ },
727
+ {
728
+ "id": "heldout_25",
729
+ "prompt": "A crate has 776 red and 116 blue beads. Remove 39 red beads. Return only the total beads remaining.",
730
+ "expected": "853",
731
+ "kind": "integer",
732
+ "output": "The number of red beads left is 737. There are 116 blue beads, 737 red beads, and 1533 beads left.",
733
+ "pass": false
734
+ },
735
+ {
736
+ "id": "heldout_26",
737
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_779, count=117.",
738
+ "expected": {
739
+ "item": "part_779",
740
+ "count": 117
741
+ },
742
+ "kind": "json",
743
+ "output": "{\"item\": \"part_779\", \"count\": 117}",
744
+ "pass": true
745
+ },
746
+ {
747
+ "id": "heldout_27",
748
+ "prompt": "Return only one CSV value row, no header: name=Item782; quantity=118; code=K41.",
749
+ "expected": "Item782,118,K41",
750
+ "kind": "exact",
751
+ "output": "Item782,118,K41",
752
+ "pass": true
753
+ },
754
+ {
755
+ "id": "heldout_28",
756
+ "prompt": "Copy exactly, no other text: Key_785-B119_C42",
757
+ "expected": "Key_785-B119_C42",
758
+ "kind": "exact",
759
+ "output": "Key_785-B119_C42",
760
+ "pass": true
761
+ },
762
+ {
763
+ "id": "heldout_29",
764
+ "prompt": "Write only a Python function named solve(x) that returns x * 120 + 43.",
765
+ "expected": "x * 120 + 43",
766
+ "kind": "code",
767
+ "output": "```python\ndef solve(x):\n return x * 120 + 43\n```",
768
+ "pass": true
769
+ },
770
+ {
771
+ "id": "heldout_30",
772
+ "prompt": "Return only the integer: 791 + 121 - 44.",
773
+ "expected": "868",
774
+ "kind": "integer",
775
+ "output": "768",
776
+ "pass": false
777
+ },
778
+ {
779
+ "id": "heldout_31",
780
+ "prompt": "A crate has 794 red and 122 blue beads. Remove 45 red beads. Return only the total beads remaining.",
781
+ "expected": "871",
782
+ "kind": "integer",
783
+ "output": "The number of red beads left is 749 because 794 - 45 = 749. The total number of beads left is 749 + 122 = 871.",
784
+ "pass": false
785
+ },
786
+ {
787
+ "id": "heldout_32",
788
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_797, count=123.",
789
+ "expected": {
790
+ "item": "part_797",
791
+ "count": 123
792
+ },
793
+ "kind": "json",
794
+ "output": "{\"item\": \"part_797\", \"count\": 123}",
795
+ "pass": true
796
+ },
797
+ {
798
+ "id": "heldout_33",
799
+ "prompt": "Return only one CSV value row, no header: name=Item800; quantity=124; code=K47.",
800
+ "expected": "Item800,124,K47",
801
+ "kind": "exact",
802
+ "output": "Item800,124,K47",
803
+ "pass": true
804
+ },
805
+ {
806
+ "id": "heldout_34",
807
+ "prompt": "Copy exactly, no other text: Key_803-B125_C31",
808
+ "expected": "Key_803-B125_C31",
809
+ "kind": "exact",
810
+ "output": "Key_803-B125_C31",
811
+ "pass": true
812
+ },
813
+ {
814
+ "id": "heldout_35",
815
+ "prompt": "Write only a Python function named solve(x) that returns x * 126 + 32.",
816
+ "expected": "x * 126 + 32",
817
+ "kind": "code",
818
+ "output": "```python\ndef solve(x):\n return x * 126 + 32\n```",
819
+ "pass": true
820
+ },
821
+ {
822
+ "id": "heldout_36",
823
+ "prompt": "Return only the integer: 809 + 127 - 33.",
824
+ "expected": "903",
825
+ "kind": "integer",
826
+ "output": "803",
827
+ "pass": false
828
+ },
829
+ {
830
+ "id": "heldout_37",
831
+ "prompt": "A crate has 812 red and 128 blue beads. Remove 34 red beads. Return only the total beads remaining.",
832
+ "expected": "906",
833
+ "kind": "integer",
834
+ "output": "The number of red beads left is 812 - 34 = 778. The total number of beads left is 778 + 128 = 906.",
835
+ "pass": false
836
+ },
837
+ {
838
+ "id": "heldout_38",
839
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_815, count=129.",
840
+ "expected": {
841
+ "item": "part_815",
842
+ "count": 129
843
+ },
844
+ "kind": "json",
845
+ "output": "{\"item\": \"part_815\", \"count\": 129}",
846
+ "pass": true
847
+ },
848
+ {
849
+ "id": "heldout_39",
850
+ "prompt": "Return only one CSV value row, no header: name=Item818; quantity=130; code=K36.",
851
+ "expected": "Item818,130,K36",
852
+ "kind": "exact",
853
+ "output": "Item818,130,K36",
854
+ "pass": true
855
+ },
856
+ {
857
+ "id": "heldout_40",
858
+ "prompt": "Copy exactly, no other text: Key_821-B131_C37",
859
+ "expected": "Key_821-B131_C37",
860
+ "kind": "exact",
861
+ "output": "Key_821-B131_C37",
862
+ "pass": true
863
+ },
864
+ {
865
+ "id": "heldout_41",
866
+ "prompt": "Write only a Python function named solve(x) that returns x * 132 + 38.",
867
+ "expected": "x * 132 + 38",
868
+ "kind": "code",
869
+ "output": "```python\ndef solve(x):\n return x * 132 + 38\n```",
870
+ "pass": true
871
+ },
872
+ {
873
+ "id": "heldout_42",
874
+ "prompt": "Return only the integer: 827 + 133 - 39.",
875
+ "expected": "921",
876
+ "kind": "integer",
877
+ "output": "821",
878
+ "pass": false
879
+ },
880
+ {
881
+ "id": "heldout_43",
882
+ "prompt": "A crate has 830 red and 134 blue beads. Remove 40 red beads. Return only the total beads remaining.",
883
+ "expected": "924",
884
+ "kind": "integer",
885
+ "output": "The number of red beads left is 790 because 830 - 40 = 790. The total number of beads left is 790 + 134 = 924.",
886
+ "pass": false
887
+ },
888
+ {
889
+ "id": "heldout_44",
890
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_833, count=135.",
891
+ "expected": {
892
+ "item": "part_833",
893
+ "count": 135
894
+ },
895
+ "kind": "json",
896
+ "output": "{\"item\": \"part_833\", \"count\": 135}",
897
+ "pass": true
898
+ },
899
+ {
900
+ "id": "heldout_45",
901
+ "prompt": "Return only one CSV value row, no header: name=Item836; quantity=136; code=K42.",
902
+ "expected": "Item836,136,K42",
903
+ "kind": "exact",
904
+ "output": "Item836,136,K42",
905
+ "pass": true
906
+ },
907
+ {
908
+ "id": "heldout_46",
909
+ "prompt": "Copy exactly, no other text: Key_839-B137_C43",
910
+ "expected": "Key_839-B137_C43",
911
+ "kind": "exact",
912
+ "output": "Key_839-B137_C43",
913
+ "pass": true
914
+ },
915
+ {
916
+ "id": "heldout_47",
917
+ "prompt": "Write only a Python function named solve(x) that returns x * 138 + 44.",
918
+ "expected": "x * 138 + 44",
919
+ "kind": "code",
920
+ "output": "```python\ndef solve(x):\n return x * 138 + 44\n```",
921
+ "pass": true
922
+ },
923
+ {
924
+ "id": "heldout_48",
925
+ "prompt": "Return only the integer: 845 + 139 - 45.",
926
+ "expected": "939",
927
+ "kind": "integer",
928
+ "output": "839",
929
+ "pass": false
930
+ },
931
+ {
932
+ "id": "heldout_49",
933
+ "prompt": "A crate has 848 red and 140 blue beads. Remove 46 red beads. Return only the total beads remaining.",
934
+ "expected": "942",
935
+ "kind": "integer",
936
+ "output": "The number of red beads left is 848 - 46 = 802. The total number of beads left is 802 + 140 = 942.",
937
+ "pass": false
938
+ },
939
+ {
940
+ "id": "heldout_50",
941
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_851, count=141.",
942
+ "expected": {
943
+ "item": "part_851",
944
+ "count": 141
945
+ },
946
+ "kind": "json",
947
+ "output": "{\"item\": \"part_851\", \"count\": 141}",
948
+ "pass": true
949
+ },
950
+ {
951
+ "id": "heldout_51",
952
+ "prompt": "Return only one CSV value row, no header: name=Item854; quantity=142; code=K31.",
953
+ "expected": "Item854,142,K31",
954
+ "kind": "exact",
955
+ "output": "Item854,142,K31",
956
+ "pass": true
957
+ },
958
+ {
959
+ "id": "heldout_52",
960
+ "prompt": "Copy exactly, no other text: Key_857-B143_C32",
961
+ "expected": "Key_857-B143_C32",
962
+ "kind": "exact",
963
+ "output": "Key_857-B143_C32",
964
+ "pass": true
965
+ },
966
+ {
967
+ "id": "heldout_53",
968
+ "prompt": "Write only a Python function named solve(x) that returns x * 144 + 33.",
969
+ "expected": "x * 144 + 33",
970
+ "kind": "code",
971
+ "output": "```python\ndef solve(x):\n return x * 144 + 33\n```",
972
+ "pass": true
973
+ },
974
+ {
975
+ "id": "heldout_54",
976
+ "prompt": "Return only the integer: 863 + 145 - 34.",
977
+ "expected": "974",
978
+ "kind": "integer",
979
+ "output": "874",
980
+ "pass": false
981
+ },
982
+ {
983
+ "id": "heldout_55",
984
+ "prompt": "A crate has 866 red and 146 blue beads. Remove 35 red beads. Return only the total beads remaining.",
985
+ "expected": "977",
986
+ "kind": "integer",
987
+ "output": "The number of red beads left is 831. The number of blue beads is still 146, for a total of 977 beads.",
988
+ "pass": false
989
+ },
990
+ {
991
+ "id": "heldout_56",
992
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_869, count=147.",
993
+ "expected": {
994
+ "item": "part_869",
995
+ "count": 147
996
+ },
997
+ "kind": "json",
998
+ "output": "{\"item\": \"part_869\", \"count\": 147}",
999
+ "pass": true
1000
+ },
1001
+ {
1002
+ "id": "heldout_57",
1003
+ "prompt": "Return only one CSV value row, no header: name=Item872; quantity=148; code=K37.",
1004
+ "expected": "Item872,148,K37",
1005
+ "kind": "exact",
1006
+ "output": "Item872,148,K37",
1007
+ "pass": true
1008
+ },
1009
+ {
1010
+ "id": "heldout_58",
1011
+ "prompt": "Copy exactly, no other text: Key_875-B149_C38",
1012
+ "expected": "Key_875-B149_C38",
1013
+ "kind": "exact",
1014
+ "output": "Key_875-B149_C38",
1015
+ "pass": true
1016
+ },
1017
+ {
1018
+ "id": "heldout_59",
1019
+ "prompt": "Write only a Python function named solve(x) that returns x * 150 + 39.",
1020
+ "expected": "x * 150 + 39",
1021
+ "kind": "code",
1022
+ "output": "```python\ndef solve(x):\n return x * 150 + 39\n```",
1023
+ "pass": true
1024
+ }
1025
+ ]
1026
+ }
train.jsonl CHANGED
The diff for this file is too large to render. See raw diff
 
train.py CHANGED
@@ -1,173 +1,128 @@
1
- """BF16-native repair of Astheral on a remote 24 GB GPU."""
2
- import os,json,time,gc,hashlib,shutil,random,collections
 
 
 
3
  from pathlib import Path
4
  import torch
5
- from torch.utils.data import Dataset
6
- from transformers import AutoModelForCausalLM,AutoTokenizer,Trainer,TrainingArguments,TrainerCallback,set_seed
7
- from peft import PeftModel
8
- from huggingface_hub import HfApi,hf_hub_download,snapshot_download
9
- from evaluate import CASES,check,summary
10
- ROOT=Path(__file__).resolve().parent;OUT=Path("/tmp/astheral-bf16-repair2");OUT.mkdir(exist_ok=True)
11
- REPO="SEE7/Astheral";BASE="NousResearch/DeepHermes-3-Llama-3-8B-Preview";REV="53d902c7dcfa4c749a3b455149064f82af2e2549"
12
- api=HfApi();start=time.monotonic();set_seed(20261115)
13
- assert torch.cuda.is_available()
14
- assert any(x in torch.cuda.get_device_name(0) for x in ["A10G","L4"])
15
- provenance=json.loads((ROOT/"repair_provenance.json").read_text());SOURCE_REV=provenance["source_revision"]
16
- def read(name):return Path(hf_hub_download(REPO,name,revision=SOURCE_REV)).read_text()
17
- settings=json.loads((ROOT/"settings.json").read_text())
18
- tok=AutoTokenizer.from_pretrained(BASE,revision=REV);tok.pad_token_id=128004;tok.padding_side="right"
19
- confirm=[]
20
- def c(id,prompt,expected,kind="exact",mode="instruct"):confirm.append(dict(id=id,prompt=prompt,kind=kind,expected=expected,mode=mode,group="confirmation",tests=None))
21
- c("confirm_csv_1","Output one CSV row, no header or quotes: user=Cora, age=31, city=Osaka.","Cora,31,Osaka")
22
- c("confirm_csv_2","Return only a CSV value row: name=Teo; years=52; place=Riga. Do not include keys or quotes.","Teo,52,Riga")
23
- c("confirm_csv_3","Output only CSV values in this order: city=Tartu, person=Sara, age=41. No header or quotes.","Tartu,Sara,41")
24
- c("confirm_csv_4","Output a CSV data row for user=Yara, age=36, country=Kenya. No keys, header or quotes.","Yara,36,Kenya")
25
- c("confirm_csv_5","Convert product=Aster; units=9; color=Indigo into one CSV value row. No header or quotation marks.","Aster,9,Indigo")
26
- c("confirm_csv_6","Return only a CSV value row for name=Zoe, identifier=Q_42, town=Graz.","Zoe,Q_42,Graz")
27
- c("confirm_copy_1","Write epsilon and psi, each on its own line. Preserve their English spellings, with no other text.","epsilon\npsi")
28
- c("confirm_copy_2","Reply with exactly: A_B-47","A_B-47")
29
- c("confirm_copy_3","Copy naïve and jalapeño, each on its own line. No bullets or other text.","naïve\njalapeño")
30
- c("confirm_math_1","Leah had 210 credits, spent one seventh of them, then received 26 credits. What is her balance? Explain briefly.",206,"number","thinking")
31
- c("confirm_math_2","Malik had 96 tokens, spent one third, then received 11 tokens. How many tokens does he have now? Explain briefly.",75,"number","thinking")
32
- c("confirm_math_3","A fund starts with 231 pounds, spends one seventh, and later receives 19 pounds. What is the new balance? Explain briefly.",217,"number","thinking")
33
- c("confirm_time_1","A bus leaves at 10:48 and arrives at 13:07 on the same day. What is the elapsed journey time in minutes?",139,"number","thinking")
34
- c("confirm_time_2","How many minutes are in two hours and seventeen minutes? Explain briefly.",137,"number","thinking")
35
- for fn in ["pack_adjacent","compress_chars"]:
36
- confirm.append(dict(id="confirm_code_"+fn,prompt=f"Write only Python code for {fn}(text), returning consecutive (character, count) tuples as a list.",kind="code",expected=None,mode="instruct",group="confirmation",tests=f"assert {fn}('aabbba')==[('a',2),('b',3),('a',1)]\nassert {fn}('')==[]\nassert {fn}('x')==[('x',1)]"))
37
- ALL=CASES+confirm
38
- MAX_TOKENS=512
39
- class Samples(Dataset):
40
- def __init__(self,rows):
41
- self.items=[];self.kept=[];self.skipped=0
42
- forbidden={x["prompt"] for x in ALL}
43
- for row in rows:
44
- assert not any(m["role"]=="user" and m["content"] in forbidden for m in row["messages"])
45
- prefix=tok.apply_chat_template(row["messages"][:-1],tokenize=True,add_generation_prompt=True)
46
- full=tok.apply_chat_template(row["messages"],tokenize=True,add_generation_prompt=False)
47
- assert full[:len(prefix)]==prefix
48
- if len(full)>MAX_TOKENS:self.skipped+=1;continue
49
- self.items.append({"input_ids":full,"labels":[-100]*len(prefix)+full[len(prefix):]});self.kept.append(row)
50
- def __len__(self):return len(self.items)
51
- def __getitem__(self,index):return self.items[index]
52
- def collate(batch):
53
- n=max(len(r["input_ids"]) for r in batch)
54
- return {k:torch.tensor([r[k]+[fill]*(n-len(r[k])) for r in batch]) for k,fill in [("input_ids",128004),("labels",-100)]}|{"attention_mask":torch.tensor([[1]*len(r["input_ids"])+[0]*(n-len(r["input_ids"])) for r in batch])}
55
- train_records=[json.loads(s) for s in (ROOT/"repair_train.jsonl").read_text().splitlines()]
56
- valid_records=[json.loads(s) for s in (ROOT/"repair_validation.jsonl").read_text().splitlines()]
57
- train=Samples(train_records)
58
- valid=Samples(valid_records)
59
- assert len(train)>1200 and len(valid)>40
60
- print("BF16_DATA",len(train),len(valid),"skipped",train.skipped,flush=True)
61
- baseline=json.loads(read("v2-baseline.json"))["outputs"]
62
- for row in baseline:row["pass"]=check(row,row["output"])
63
- (OUT/"base_outputs.json").write_text(json.dumps(baseline,indent=2))
64
- base=AutoModelForCausalLM.from_pretrained(BASE,revision=REV,torch_dtype=torch.bfloat16,device_map={"":"cuda:0"},attn_implementation="sdpa")
65
- adapter_root=snapshot_download(REPO,revision=SOURCE_REV,allow_patterns=["repair-results/adapter/*"],local_dir="/tmp/initial-adapter")
66
- model=PeftModel.from_pretrained(base,Path(adapter_root)/"repair-results/adapter",is_trainable=True)
67
- model.config.use_cache=False;model.enable_input_require_grads()
68
- model.gradient_checkpointing_enable(gradient_checkpointing_kwargs={"use_reentrant":False})
69
- trainable=sum(p.numel() for p in model.parameters() if p.requires_grad);assert 0<trainable<100000000
70
- # Verify actual BF16 backward memory before the optimizer run.
71
- for limit in [512,384]:
72
- MAX_TOKENS=limit
73
- train=Samples(train_records);valid=Samples(valid_records)
74
- largest=max(train.items,key=lambda r:len(r["input_ids"]))
75
- probe={k:v.to("cuda") for k,v in collate([largest]).items()}
76
  try:
77
- model.train();loss=model(**probe).loss;loss.backward()
78
- assert torch.isfinite(loss)
79
- model.zero_grad(set_to_none=True);del loss,probe;torch.cuda.empty_cache()
80
- break
81
- except torch.cuda.OutOfMemoryError:
82
- model.zero_grad(set_to_none=True);loss=None;probe=None;gc.collect();torch.cuda.empty_cache()
83
- if limit==384:raise
84
- print("REDUCING_REPAIR_SEQUENCE_LENGTH",384,flush=True)
85
- print("BF16_BACKWARD_OK",trainable,torch.cuda.max_memory_allocated(),flush=True)
86
- class Budget(TrainerCallback):
87
- def on_step_end(self,args,state,control,**kwargs):
88
- if time.monotonic()-start>3600:control.should_training_stop=True
89
- return control
90
- args=TrainingArguments(output_dir="/tmp/bf16-repair-checkpoints",num_train_epochs=1.5,per_device_train_batch_size=1,per_device_eval_batch_size=1,gradient_accumulation_steps=8,learning_rate=3e-5,lr_scheduler_type="cosine",warmup_ratio=0.05,weight_decay=0.01,bf16=True,gradient_checkpointing=True,gradient_checkpointing_kwargs={"use_reentrant":False},optim="paged_adamw_8bit",logging_steps=10,save_strategy="no",report_to="none",seed=20261115,data_seed=20261115,remove_unused_columns=False,max_grad_norm=0.5,dataloader_num_workers=0)
91
- trainer=Trainer(model=model,args=args,train_dataset=train,eval_dataset=valid,data_collator=collate,callbacks=[Budget()])
92
- before=trainer.evaluate();result=trainer.train();after=trainer.evaluate()
93
- model.config.use_cache=True;model.gradient_checkpointing_disable()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
94
  @torch.inference_mode()
95
- def generate(tag,cases=ALL):
96
- model.eval();tok.padding_side="left";result_rows=[]
97
- for mode in ["instruct","thinking"]:
98
- selected=[r for r in cases if r["mode"]==mode]
99
- for i in range(0,len(selected),2):
100
- group=selected[i:i+2]
101
- system=settings["system"]+("\n"+settings["thinking"] if mode=="thinking" else "")
102
- texts=[tok.apply_chat_template([{"role":"system","content":system},{"role":"user","content":r["prompt"]}],tokenize=False,add_generation_prompt=True) for r in group]
103
- inputs=tok(texts,padding=True,return_tensors="pt").to("cuda")
104
- out=model.generate(**inputs,max_new_tokens=1024 if mode=="thinking" else 384,do_sample=False,pad_token_id=128004,eos_token_id=128009)
105
- for case,outrow in zip(group,out):
106
- answer=tok.decode(outrow[inputs["input_ids"].shape[1]:],skip_special_tokens=True)
107
- result_rows.append({**case,"output":answer,"pass":check(case,answer)})
108
- print("REPAIR_EVAL",tag,case["id"],result_rows[-1]["pass"],flush=True)
109
- tok.padding_side="right"
110
- (OUT/(tag+"_outputs.json")).write_text(json.dumps(result_rows,indent=2))
111
- return result_rows
112
- def scores(rows):
113
- s=summary(rows);r=[x for x in rows if x["group"]=="confirmation"];s["confirmation"]={"passed":sum(x["pass"] for x in r),"total":len(r)};return s
114
- def gate(rows):
115
- s=scores(rows)
116
- modes=all(("<think>" not in r["output"] and "</think>" not in r["output"]) if r["mode"]=="instruct" else (r["output"].lstrip().startswith("<think>") and "</think>" in r["output"]) for r in rows)
117
- return s["regression"]["passed"]==2 and s["fresh"]["passed"]==36 and s["confirmation"]["passed"]==len(confirm) and modes
118
- adapter=OUT/"adapter";model.save_pretrained(adapter,safe_serialization=True);tok.save_pretrained(adapter)
119
- adapted=generate("astheral")
120
- initial=json.loads(read("repair-results/evaluation.json"))
121
- old_train=[json.loads(s) for s in read("repair-results/train.jsonl").splitlines()]
122
- unique={}
123
- for row in old_train+train.kept:unique[hashlib.sha256(json.dumps(row["messages"],sort_keys=True).encode()).hexdigest()]=row
124
- (OUT/"train.jsonl").write_text("\n".join(json.dumps(r) for r in unique.values())+"\n")
125
- (OUT/"validation.jsonl").write_text("\n".join(json.dumps(r) for r in valid.kept)+"\n")
126
- (OUT/"repair_train.jsonl").write_text("\n".join(json.dumps(r) for r in train.kept)+"\n")
127
- metrics={"version":"2","repair_pass":1,"base_model":BASE,"base_revision":REV,"initial_adapter_revision":SOURCE_REV,"training_examples":len(unique),"repair_training_examples":len(train),"validation_examples":len(valid),"trainable_parameters":trainable,"rank":32,"target_modules":"all-linear","optimizer_steps":initial["optimizer_steps"]+trainer.state.global_step,"training_stages":[{"method":"NF4 QLoRA","examples":initial["training_examples"],"steps":initial["optimizer_steps"],"epochs":initial.get("epochs")},{"method":"BF16 LoRA","examples":len(train),"steps":trainer.state.global_step,"epochs":trainer.state.epoch}],"validation_loss_before":before["eval_loss"],"validation_loss_after":after["eval_loss"],"base_scores":summary(baseline),"astheral_scores":scores(adapted),"quality_gate_passed":False,"hardware":torch.cuda.get_device_name(0),"max_gpu_memory_allocated_bytes":torch.cuda.max_memory_allocated(),"training":result.metrics,"generation":{"sampling":False,"instruct_max_new_tokens":384,"thinking_max_new_tokens":1024},"evaluation_design":"36 core development checks, two original regressions, and 12 new confirmation cases. Independent public test subsets are evaluated separately."}
128
- (OUT/"evaluation.json").write_text(json.dumps(metrics,indent=2));(OUT/"trainer_log.json").write_text(json.dumps(trainer.state.log_history,indent=2))
129
- metrics["repair_pass"]=2
130
- metrics["training_stages"]=initial["training_stages"]+[{"method":"BF16 LoRA","examples":len(train),"steps":trainer.state.global_step,"epochs":trainer.state.epoch}]
131
- metrics["evaluation_design"]="36 core development checks, two original regressions, 12 prior confirmation cases and four new time/code confirmation cases. Independent public test subsets are evaluated separately."
132
- (OUT/"evaluation.json").write_text(json.dumps(metrics,indent=2))
133
- for n in ["settings.json","repair_provenance.json","evaluate.py","prepare_repair2.py"]:
134
- shutil.copy2(ROOT/n,OUT/n)
135
- shutil.copy2(Path(__file__),OUT/"train.py")
136
- api.upload_folder(repo_id=REPO,folder_path=OUT,path_in_repo="repair2-results",commit_message="Record second native-BF16 repair adapter, data, and evaluation")
137
- # Merge the same BF16 backbone used for training; verify the saved model itself.
138
- del trainer;gc.collect();torch.cuda.empty_cache()
139
- model=model.merge_and_unload(safe_merge=True)
140
- model.config._name_or_path="Llama-3.1-Astheral";model.config.astheral_version="2";model.config.astheral_base_model=BASE;model.config.astheral_modes=["instruct","thinking"]
141
- model.save_pretrained(OUT,safe_serialization=True,max_shard_size="4GB");tok.save_pretrained(OUT)
142
- del model,base;gc.collect();torch.cuda.empty_cache()
143
- model=AutoModelForCausalLM.from_pretrained(OUT,torch_dtype=torch.bfloat16,device_map={"":"cuda:0"},attn_implementation="sdpa")
144
- merged=generate("astheral_merged")
145
- merged_core=[r for r in merged if r["group"]!="confirmation"];merged_confirm=[r for r in merged if r["group"]=="confirmation"]
146
- (OUT/"confirmation_outputs.json").write_text(json.dumps(merged_confirm,indent=2))
147
- (OUT/"merged_regressions_outputs.json").write_text(json.dumps([r for r in merged if r["group"]=="regression"],indent=2))
148
- metrics["merged_scores"]=scores(merged);metrics["quality_gate_passed"]=gate(merged);metrics["elapsed_seconds"]=time.monotonic()-start
149
- (OUT/"evaluation.json").write_text(json.dumps(metrics,indent=2))
150
- api.upload_folder(repo_id=REPO,folder_path=OUT,path_in_repo="repair2-results",allow_patterns=["evaluation.json","astheral_merged_outputs.json","astheral_outputs.json","merged_regressions_outputs.json","confirmation_outputs.json","trainer_log.json"],commit_message="Record complete reloaded second BF16 repair results")
151
- print("REPAIR_GATE",metrics["quality_gate_passed"],json.dumps(metrics["merged_scores"]),flush=True)
152
- if not metrics["quality_gate_passed"]:raise SystemExit(3)
153
- for n in ["LICENSE","UPSTREAM_LLAMA3_LICENSE","NOTICE"]:shutil.copy2(Path(hf_hub_download(REPO,n,revision=SOURCE_REV)),OUT/n)
154
- for n in ["DATA_LICENSE_GSM8K.txt","DATA_LICENSE_APACHE2.txt"]:shutil.copy2(Path(hf_hub_download(REPO,"v2/"+n,revision=SOURCE_REV)),OUT/n)
155
- (OUT/"DATA_LICENSE_ODCBY.txt").write_text("Contains data from bigcode/self-oss-instruct-sc2-exec-filter-50k under ODC Attribution v1.0: https://opendatacommons.org/licenses/by/1-0/.\n")
156
- (OUT/"NOTICE").write_text((OUT/"NOTICE").read_text()+"\nTraining sources: HuggingFaceTB/SmolTalk (Apache 2.0), openai/gsm8k (MIT; Copyright (c) 2021 OpenAI), bigcode/self-oss-instruct-sc2-exec-filter-50k (ODC Attribution), and original Astheral repair examples (CC0-1.0).\n")
157
- sources=json.loads(read("v2/data_sources.json"));sources["repair"]=provenance
158
- (OUT/"data_sources.json").write_text(json.dumps(sources,indent=2))
159
- (OUT/"DATA.md").write_text("# Training data\n\nOriginal and repair training examples are included. data_sources.json records provenance and component licenses. Original Astheral examples are CC0-1.0; third-party examples retain their source terms. Training stages and exact optimizer steps are recorded in evaluation.json.\n")
160
- (OUT/"reload_smoke_test.json").write_text(json.dumps({"loaded_from":"saved merged BF16 checkpoint","core_checks_passed":38,"confirmation_checks_passed":len(confirm)},indent=2))
161
- for n in ["inference.py","requirements.txt","Dockerfile"]:shutil.copy2(ROOT/n,OUT/n)
162
- (OUT/"README.md").write_text((ROOT/"MODEL_CARD.md").read_text()+"\n## Verified core results\n\n"+json.dumps(metrics["merged_scores"],indent=2)+"\n")
163
- manifest={}
164
- for f in OUT.rglob("*"):
165
- if f.is_file():
166
- h=hashlib.sha256()
167
- with f.open("rb") as handle:
168
- for part in iter(lambda:handle.read(8388608),b""):h.update(part)
169
- manifest[str(f.relative_to(OUT))]={"bytes":f.stat().st_size,"sha256":h.hexdigest()}
170
- (OUT/"artifact_manifest.json").write_text(json.dumps(manifest,indent=2))
171
- api.create_tag(REPO,tag="v1",revision=api.model_info(REPO).sha,exist_ok=True)
172
- api.upload_folder(repo_id=REPO,folder_path=OUT,delete_patterns=["training/*","coding_smoke_test.json"],commit_message="Release Astheral v2 after native-BF16 repair and all core/confirmation checks")
173
- print("REPAIR_PUBLISHED",api.model_info(REPO).sha,flush=True)
 
1
+ # /// script
2
+ # dependencies = ["torch==2.8.0", "transformers==4.56.2", "peft==0.17.1", "accelerate==1.10.1", "huggingface_hub==0.34.4", "bitsandbytes==0.47.0", "safetensors==0.6.2"]
3
+ # ///
4
+ """Budget-bounded V3 training with paired, disjoint evaluation and reload gate."""
5
+ import os, json, random, time, re, gc, hashlib, traceback
6
  from pathlib import Path
7
  import torch
8
+ from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, Trainer, TrainingArguments, TrainerCallback, set_seed
9
+ from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training, PeftModel
10
+ from huggingface_hub import HfApi, hf_hub_download
11
+
12
+ REPO='SEE7/Astheral'; REV='d9aa899722be4788f57107c2951c72d0ec6614ee'
13
+ OUT=Path('/tmp/astheral-v3'); OUT.mkdir(exist_ok=True)
14
+ api=HfApi(); started=time.monotonic(); set_seed(20261113); rng=random.Random(20261113)
15
+ assert torch.cuda.is_available() and 'A10G' in torch.cuda.get_device_name(0)
16
+ branch='v3-candidate-20261113'
17
+ api.create_branch(REPO,branch=branch,revision=REV,exist_ok=True)
18
+ settings=json.loads(Path(hf_hub_download(REPO,'settings.json',revision=REV)).read_text())
19
+ tok=AutoTokenizer.from_pretrained(REPO,revision=REV); tok.pad_token_id=128004
20
+ report={'status':'preparing','parent_revision':REV,'hardware':torch.cuda.get_device_name(0),'job_timeout_seconds':5400,'listed_usd_per_hour':1.0,'training_seed':20261113}
21
+
22
+ def publish():
23
+ report['elapsed_seconds']=round(time.monotonic()-started,2)
24
+ (OUT/'v3_report.json').write_text(json.dumps(report,indent=2))
25
+ api.upload_folder(repo_id=REPO,folder_path=OUT,revision=branch,path_in_repo='v3-results',ignore_patterns=['merged/**'],commit_message='Record measured V3 training and evaluation artifacts')
26
+
27
+ def sample(a,b,c,kind):
28
+ if kind==0:return f'Return only the integer: {a} + {b} - {c}.',str(a+b-c),'integer'
29
+ if kind==1:return f'A crate has {a} red and {b} blue beads. Remove {c} red beads. Return only the total beads remaining.',str(a+b-c),'integer'
30
+ if kind==2:return f'Return only valid JSON with keys "item" and "count": item=part_{a}, count={b}.',{'item':f'part_{a}','count':b},'json'
31
+ if kind==3:return f'Return only one CSV value row, no header: name=Item{a}; quantity={b}; code=K{c}.',f'Item{a},{b},K{c}','exact'
32
+ if kind==4:return f'Copy exactly, no other text: Key_{a}-B{b}_C{c}',f'Key_{a}-B{b}_C{c}','exact'
33
+ return f'Write only a Python function named solve(x) that returns x * {b} + {c}.',f'x * {b} + {c}','code'
34
+
35
+ def grade(case,text):
36
+ expected=case['expected']; kind=case['kind']; text=text.strip()
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
37
  try:
38
+ if kind=='json':return json.loads(text)==expected
39
+ if kind=='integer':return re.fullmatch(r'-?\d+',text) is not None and int(text)==int(expected)
40
+ if kind=='exact':return text==expected
41
+ # Parse and evaluate a tiny arithmetic-only AST; never execute model-generated code.
42
+ import ast
43
+ clean=re.sub(r'^```(?:python)?\s*|\s*```$','',text).strip();tree=ast.parse(clean)
44
+ if len(tree.body)!=1 or not isinstance(tree.body[0],ast.FunctionDef):return False
45
+ fn=tree.body[0]
46
+ if fn.name!='solve' or len(fn.args.args)!=1 or fn.args.args[0].arg!='x' or len(fn.body)!=1 or not isinstance(fn.body[0],ast.Return):return False
47
+ def calc(n,x):
48
+ if isinstance(n,ast.Constant) and isinstance(n.value,(int,float)):return n.value
49
+ if isinstance(n,ast.Name) and n.id=='x':return x
50
+ if isinstance(n,ast.BinOp) and isinstance(n.op,(ast.Add,ast.Sub,ast.Mult)):
51
+ a,b=calc(n.left,x),calc(n.right,x)
52
+ return a+b if isinstance(n.op,ast.Add) else a-b if isinstance(n.op,ast.Sub) else a*b
53
+ raise ValueError('Unsupported expression')
54
+ match=re.fullmatch(r'x \* (\d+) \+ (\d+)',expected);b,c=map(int,match.groups())
55
+ return all(calc(fn.body[0].value,x)==x*b+c for x in [-5,0,1,7,100])
56
+ except Exception:return False
57
+
58
+ tests=[]
59
+ for i in range(60):
60
+ # Evaluation numbers are outside the synthetic training ranges.
61
+ a,b,c=701+i*3,91+i,31+i%17;p,e,k=sample(a,b,c,i%6)
62
+ tests.append({'id':f'heldout_{i:02}','prompt':p,'expected':e,'kind':k})
63
+ train=[]
64
+ for i in range(900):
65
+ a,b,c=rng.randint(100,500),rng.randint(10,80),rng.randint(1,29)
66
+ p,e,k=sample(a,b,c,i%6)
67
+ answer=json.dumps(e) if k=='json' else 'def solve(x):\n return '+e if k=='code' else str(e)
68
+ train.append({'messages':[{'role':'user','content':p},{'role':'assistant','content':answer}]})
69
+ retained=[json.loads(s) for s in Path(hf_hub_download(REPO,'train.jsonl',revision=REV)).read_text().splitlines()]
70
+ rng.shuffle(retained);train+=retained[:900];rng.shuffle(train)
71
+ assert not {x['prompt'] for x in tests} & {m['content'] for row in train for m in row['messages'] if m['role']=='user'}
72
+ (OUT/'heldout_cases.json').write_text(json.dumps(tests,indent=2));(OUT/'training.jsonl').write_text('\n'.join(json.dumps(x) for x in train)+'\n')
73
+ (OUT/'train_v3.py').write_text(Path(__file__).read_text())
74
+ q=BitsAndBytesConfig(load_in_4bit=True,bnb_4bit_quant_type='nf4',bnb_4bit_use_double_quant=True,bnb_4bit_compute_dtype=torch.bfloat16)
75
+ model=AutoModelForCausalLM.from_pretrained(REPO,revision=REV,quantization_config=q,device_map={'':'cuda:0'},torch_dtype=torch.bfloat16,attn_implementation='sdpa')
76
+
77
  @torch.inference_mode()
78
+ def evaluate(label):
79
+ model.eval(); rows=[]
80
+ for i,case in enumerate(tests):
81
+ inputs=tok.apply_chat_template([{'role':'system','content':settings['system']},{'role':'user','content':case['prompt']}],tokenize=True,add_generation_prompt=True,return_dict=True,return_tensors='pt').to('cuda')
82
+ out=model.generate(**inputs,max_new_tokens=96,do_sample=False,pad_token_id=128004,eos_token_id=128009)
83
+ text=tok.decode(out[0,inputs['input_ids'].shape[1]:],skip_special_tokens=True)
84
+ rows.append({**case,'output':text,'pass':grade(case,text),'tokens':int(out.shape[1]-inputs['input_ids'].shape[1])})
85
+ print('EVAL',label,i,rows[-1]['pass'],flush=True)
86
+ (OUT/(label+'_outputs.json')).write_text(json.dumps(rows,indent=2))
87
+ return rows
88
+
89
+ try:
90
+ baseline=evaluate('v2_quantized'); report['baseline_passed']=sum(r['pass'] for r in baseline);publish()
91
+ model=prepare_model_for_kbit_training(model,use_gradient_checkpointing=True)
92
+ model=get_peft_model(model,LoraConfig(r=16,lora_alpha=32,lora_dropout=0.05,target_modules=['q_proj','k_proj','v_proj','o_proj','gate_proj','up_proj','down_proj'],task_type='CAUSAL_LM'))
93
+ model.config.use_cache=False;encoded=[]
94
+ for row in train:
95
+ messages=row['messages']; prefix=tok.apply_chat_template(messages[:-1],tokenize=True,add_generation_prompt=True)
96
+ full=tok.apply_chat_template(messages,tokenize=True,add_generation_prompt=False)
97
+ if len(full)<=512 and full[:len(prefix)]==prefix:encoded.append({'input_ids':full,'labels':[-100]*len(prefix)+full[len(prefix):]})
98
+ def collate(rows):
99
+ length=max(len(r['input_ids']) for r in rows)
100
+ return {k:torch.tensor([r[k]+[pad]*(length-len(r[k])) for r in rows]) for k,pad in [('input_ids',128004),('labels',-100)]}|{'attention_mask':torch.tensor([[1]*len(r['input_ids'])+[0]*(length-len(r['input_ids'])) for r in rows])}
101
+ class Budget(TrainerCallback):
102
+ def on_step_end(self,args,state,control,**kwargs):
103
+ if time.monotonic()-started>2400:control.should_training_stop=True
104
+ return control
105
+ args=TrainingArguments(output_dir='/tmp/v3-checkpoints',max_steps=180,per_device_train_batch_size=1,gradient_accumulation_steps=4,learning_rate=1e-5,warmup_ratio=0.05,bf16=True,gradient_checkpointing=True,gradient_checkpointing_kwargs={'use_reentrant':False},optim='paged_adamw_8bit',save_strategy='no',logging_steps=10,report_to='none',remove_unused_columns=False,seed=20261113)
106
+ trainer=Trainer(model=model,args=args,train_dataset=encoded,data_collator=collate,callbacks=[Budget()])
107
+ report.update(status='training',training_examples=len(encoded),trainable_parameters=sum(p.numel() for p in model.parameters() if p.requires_grad));publish()
108
+ result=trainer.train();report['training']=result.metrics;report['optimizer_steps']=trainer.state.global_step
109
+ adapter=OUT/'adapter';model.save_pretrained(adapter,safe_serialization=True);tok.save_pretrained(adapter)
110
+ model.config.use_cache=True;model.gradient_checkpointing_disable();candidate=evaluate('v3_quantized')
111
+ report['candidate_passed']=sum(r['pass'] for r in candidate);report['status']='trained';publish()
112
+ del trainer,model;gc.collect();torch.cuda.empty_cache()
113
+ model=AutoModelForCausalLM.from_pretrained(REPO,revision=REV,torch_dtype=torch.bfloat16,device_map={'':'cuda:0'},attn_implementation='sdpa')
114
+ model=PeftModel.from_pretrained(model,adapter).merge_and_unload(safe_merge=True)
115
+ model.config.astheral_version='3';merged=OUT/'merged';model.save_pretrained(merged,safe_serialization=True,max_shard_size='4GB');tok.save_pretrained(merged)
116
+ del model;gc.collect();torch.cuda.empty_cache()
117
+ model=AutoModelForCausalLM.from_pretrained(merged,torch_dtype=torch.bfloat16,device_map={'':'cuda:0'},attn_implementation='sdpa')
118
+ reloaded=evaluate('v3_reloaded');report['reloaded_passed']=sum(r['pass'] for r in reloaded)
119
+ groups={kind:{'baseline':sum(r['pass'] for r in baseline if r['kind']==kind),'candidate':sum(r['pass'] for r in reloaded if r['kind']==kind),'total':sum(r['kind']==kind for r in tests)} for kind in {x['kind'] for x in tests}}
120
+ report['groups']=groups;report['quality_gate_passed']=report['reloaded_passed']>report['baseline_passed'] and all(v['candidate']>=v['baseline'] for v in groups.values())
121
+ report['status']='complete';report['peak_gpu_bytes']=torch.cuda.max_memory_allocated();publish()
122
+ # Save the actual trained weights to the isolated candidate branch even on a failed gate.
123
+ api.upload_folder(repo_id=REPO,folder_path=merged,revision=branch,commit_message='Save reloaded V3 candidate weights; see measured quality gate in v3-results')
124
+ if report['quality_gate_passed']:
125
+ api.create_tag(REPO,tag='v3',revision=api.model_info(REPO,revision=branch).sha,exist_ok=True)
126
+ print('V3_COMPLETE',json.dumps(report),flush=True)
127
+ except Exception:
128
+ report['status']='failed';report['error']=traceback.format_exc();publish();raise
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
trainer_log.json CHANGED
@@ -1,230 +1,10 @@
1
  [
2
  {
3
- "loss": 0.6322,
4
- "grad_norm": 1.8806242942810059,
5
- "learning_rate": 1.6875e-05,
6
- "epoch": 0.049474335188620905,
7
- "step": 10
8
- },
9
- {
10
- "loss": 0.3021,
11
- "grad_norm": 1.4435440301895142,
12
- "learning_rate": 2.9992024290225105e-05,
13
- "epoch": 0.09894867037724181,
14
- "step": 20
15
- },
16
- {
17
- "loss": 0.2424,
18
- "grad_norm": 2.773944854736328,
19
- "learning_rate": 2.9850469719489576e-05,
20
- "epoch": 0.14842300556586271,
21
- "step": 30
22
- },
23
- {
24
- "loss": 0.1925,
25
- "grad_norm": 1.2467366456985474,
26
- "learning_rate": 2.953360117250643e-05,
27
- "epoch": 0.19789734075448362,
28
- "step": 40
29
- },
30
- {
31
- "loss": 0.1105,
32
- "grad_norm": 0.9168190360069275,
33
- "learning_rate": 2.9045159371688403e-05,
34
- "epoch": 0.24737167594310452,
35
- "step": 50
36
- },
37
- {
38
- "loss": 0.1468,
39
- "grad_norm": 1.8128526210784912,
40
- "learning_rate": 2.8390910510016898e-05,
41
- "epoch": 0.29684601113172543,
42
- "step": 60
43
- },
44
- {
45
- "loss": 0.1311,
46
- "grad_norm": 1.2351698875427246,
47
- "learning_rate": 2.7578578179510324e-05,
48
- "epoch": 0.3463203463203463,
49
- "step": 70
50
- },
51
- {
52
- "loss": 0.1509,
53
- "grad_norm": 5.074930667877197,
54
- "learning_rate": 2.6617752192045723e-05,
55
- "epoch": 0.39579468150896724,
56
- "step": 80
57
- },
58
- {
59
- "loss": 0.1261,
60
- "grad_norm": 1.543472409248352,
61
- "learning_rate": 2.551977536892951e-05,
62
- "epoch": 0.4452690166975881,
63
- "step": 90
64
- },
65
- {
66
- "loss": 0.1672,
67
- "grad_norm": 1.9173252582550049,
68
- "learning_rate": 2.4297609635698322e-05,
69
- "epoch": 0.49474335188620905,
70
- "step": 100
71
- },
72
- {
73
- "loss": 0.1723,
74
- "grad_norm": 0.7875851988792419,
75
- "learning_rate": 2.29656830029386e-05,
76
- "epoch": 0.54421768707483,
77
- "step": 110
78
- },
79
- {
80
- "loss": 0.1244,
81
- "grad_norm": 1.0968589782714844,
82
- "learning_rate": 2.153971923955934e-05,
83
- "epoch": 0.5936920222634509,
84
- "step": 120
85
- },
86
- {
87
- "loss": 0.1643,
88
- "grad_norm": 1.792441725730896,
89
- "learning_rate": 2.0036552249273017e-05,
90
- "epoch": 0.6431663574520717,
91
- "step": 130
92
- },
93
- {
94
- "loss": 0.1358,
95
- "grad_norm": 1.13428795337677,
96
- "learning_rate": 1.8473927341622628e-05,
97
- "epoch": 0.6926406926406926,
98
- "step": 140
99
- },
100
- {
101
- "loss": 0.1403,
102
- "grad_norm": 0.8378678560256958,
103
- "learning_rate": 1.6870291743606273e-05,
104
- "epoch": 0.7421150278293135,
105
- "step": 150
106
- },
107
- {
108
- "loss": 0.1757,
109
- "grad_norm": 0.8258570432662964,
110
- "learning_rate": 1.5244576824968541e-05,
111
- "epoch": 0.7915893630179345,
112
- "step": 160
113
- },
114
- {
115
- "loss": 0.2255,
116
- "grad_norm": 1.775040864944458,
117
- "learning_rate": 1.3615974608050472e-05,
118
- "epoch": 0.8410636982065554,
119
- "step": 170
120
- },
121
- {
122
- "loss": 0.1666,
123
- "grad_norm": 1.0309557914733887,
124
- "learning_rate": 1.2003711200562243e-05,
125
- "epoch": 0.8905380333951762,
126
- "step": 180
127
- },
128
- {
129
- "loss": 0.113,
130
- "grad_norm": 0.45190468430519104,
131
- "learning_rate": 1.042681982596845e-05,
132
- "epoch": 0.9400123685837971,
133
- "step": 190
134
- },
135
- {
136
- "loss": 0.1376,
137
- "grad_norm": 1.603728175163269,
138
- "learning_rate": 8.903916130926232e-06,
139
- "epoch": 0.9894867037724181,
140
- "step": 200
141
- },
142
- {
143
- "loss": 0.0922,
144
- "grad_norm": 0.7828471064567566,
145
- "learning_rate": 7.452978422335191e-06,
146
- "epoch": 1.0346320346320346,
147
- "step": 210
148
- },
149
- {
150
- "loss": 0.0946,
151
- "grad_norm": 0.4640823304653168,
152
- "learning_rate": 6.091135428362536e-06,
153
- "epoch": 1.0841063698206554,
154
- "step": 220
155
- },
156
- {
157
- "loss": 0.0761,
158
- "grad_norm": 0.7620387077331543,
159
- "learning_rate": 4.834464088984452e-06,
160
- "epoch": 1.1335807050092765,
161
- "step": 230
162
- },
163
- {
164
- "loss": 0.0995,
165
- "grad_norm": 0.5594423413276672,
166
- "learning_rate": 3.697799763183127e-06,
167
- "epoch": 1.1830550401978974,
168
- "step": 240
169
- },
170
- {
171
- "loss": 0.0716,
172
- "grad_norm": 0.008049696683883667,
173
- "learning_rate": 2.6945610933570297e-06,
174
- "epoch": 1.2325293753865183,
175
- "step": 250
176
- },
177
- {
178
- "loss": 0.0468,
179
- "grad_norm": 1.0049060583114624,
180
- "learning_rate": 1.8365915944693102e-06,
181
- "epoch": 1.2820037105751392,
182
- "step": 260
183
- },
184
- {
185
- "loss": 0.0501,
186
- "grad_norm": 0.658693790435791,
187
- "learning_rate": 1.1340198380190725e-06,
188
- "epoch": 1.33147804576376,
189
- "step": 270
190
- },
191
- {
192
- "loss": 0.138,
193
- "grad_norm": 1.3238775730133057,
194
- "learning_rate": 5.951398814030951e-07,
195
- "epoch": 1.380952380952381,
196
- "step": 280
197
- },
198
- {
199
- "loss": 0.0941,
200
- "grad_norm": 0.4109145700931549,
201
- "learning_rate": 2.2631335423303312e-07,
202
- "epoch": 1.4304267161410018,
203
- "step": 290
204
- },
205
- {
206
- "loss": 0.0745,
207
- "grad_norm": 0.4245922863483429,
208
- "learning_rate": 3.189435750671321e-08,
209
- "epoch": 1.4799010513296227,
210
- "step": 300
211
- },
212
- {
213
- "train_runtime": 1007.3112,
214
- "train_samples_per_second": 2.408,
215
- "train_steps_per_second": 0.303,
216
- "total_flos": 2.0134086292832256e+16,
217
- "train_loss": 0.15099767006811549,
218
- "epoch": 1.5046382189239331,
219
- "step": 305
220
- },
221
- {
222
- "eval_loss": 0.0020501785911619663,
223
- "eval_model_preparation_time": 0.0184,
224
- "eval_runtime": 5.753,
225
- "eval_samples_per_second": 11.125,
226
- "eval_steps_per_second": 11.125,
227
- "epoch": 1.5046382189239331,
228
- "step": 305
229
  }
230
  ]
 
1
  [
2
  {
3
+ "train_runtime": 390.2404,
4
+ "train_samples_per_second": 1.845,
5
+ "train_steps_per_second": 0.461,
6
+ "total_flos": 3960249019342848.0,
7
+ "train_loss": 0.2867420093880759,
8
+ "epoch": 0.4203152364273205
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
9
  }
10
  ]
v3-results/adapter/README.md ADDED
@@ -0,0 +1,207 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: SEE7/Astheral
3
+ library_name: peft
4
+ pipeline_tag: text-generation
5
+ tags:
6
+ - base_model:adapter:SEE7/Astheral
7
+ - lora
8
+ - transformers
9
+ ---
10
+
11
+ # Model Card for Model ID
12
+
13
+ <!-- Provide a quick summary of what the model is/does. -->
14
+
15
+
16
+
17
+ ## Model Details
18
+
19
+ ### Model Description
20
+
21
+ <!-- Provide a longer summary of what this model is. -->
22
+
23
+
24
+
25
+ - **Developed by:** [More Information Needed]
26
+ - **Funded by [optional]:** [More Information Needed]
27
+ - **Shared by [optional]:** [More Information Needed]
28
+ - **Model type:** [More Information Needed]
29
+ - **Language(s) (NLP):** [More Information Needed]
30
+ - **License:** [More Information Needed]
31
+ - **Finetuned from model [optional]:** [More Information Needed]
32
+
33
+ ### Model Sources [optional]
34
+
35
+ <!-- Provide the basic links for the model. -->
36
+
37
+ - **Repository:** [More Information Needed]
38
+ - **Paper [optional]:** [More Information Needed]
39
+ - **Demo [optional]:** [More Information Needed]
40
+
41
+ ## Uses
42
+
43
+ <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
44
+
45
+ ### Direct Use
46
+
47
+ <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
48
+
49
+ [More Information Needed]
50
+
51
+ ### Downstream Use [optional]
52
+
53
+ <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
54
+
55
+ [More Information Needed]
56
+
57
+ ### Out-of-Scope Use
58
+
59
+ <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
60
+
61
+ [More Information Needed]
62
+
63
+ ## Bias, Risks, and Limitations
64
+
65
+ <!-- This section is meant to convey both technical and sociotechnical limitations. -->
66
+
67
+ [More Information Needed]
68
+
69
+ ### Recommendations
70
+
71
+ <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
72
+
73
+ Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
74
+
75
+ ## How to Get Started with the Model
76
+
77
+ Use the code below to get started with the model.
78
+
79
+ [More Information Needed]
80
+
81
+ ## Training Details
82
+
83
+ ### Training Data
84
+
85
+ <!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
86
+
87
+ [More Information Needed]
88
+
89
+ ### Training Procedure
90
+
91
+ <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
92
+
93
+ #### Preprocessing [optional]
94
+
95
+ [More Information Needed]
96
+
97
+
98
+ #### Training Hyperparameters
99
+
100
+ - **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
101
+
102
+ #### Speeds, Sizes, Times [optional]
103
+
104
+ <!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
105
+
106
+ [More Information Needed]
107
+
108
+ ## Evaluation
109
+
110
+ <!-- This section describes the evaluation protocols and provides the results. -->
111
+
112
+ ### Testing Data, Factors & Metrics
113
+
114
+ #### Testing Data
115
+
116
+ <!-- This should link to a Dataset Card if possible. -->
117
+
118
+ [More Information Needed]
119
+
120
+ #### Factors
121
+
122
+ <!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
123
+
124
+ [More Information Needed]
125
+
126
+ #### Metrics
127
+
128
+ <!-- These are the evaluation metrics being used, ideally with a description of why. -->
129
+
130
+ [More Information Needed]
131
+
132
+ ### Results
133
+
134
+ [More Information Needed]
135
+
136
+ #### Summary
137
+
138
+
139
+
140
+ ## Model Examination [optional]
141
+
142
+ <!-- Relevant interpretability work for the model goes here -->
143
+
144
+ [More Information Needed]
145
+
146
+ ## Environmental Impact
147
+
148
+ <!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
149
+
150
+ Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
151
+
152
+ - **Hardware Type:** [More Information Needed]
153
+ - **Hours used:** [More Information Needed]
154
+ - **Cloud Provider:** [More Information Needed]
155
+ - **Compute Region:** [More Information Needed]
156
+ - **Carbon Emitted:** [More Information Needed]
157
+
158
+ ## Technical Specifications [optional]
159
+
160
+ ### Model Architecture and Objective
161
+
162
+ [More Information Needed]
163
+
164
+ ### Compute Infrastructure
165
+
166
+ [More Information Needed]
167
+
168
+ #### Hardware
169
+
170
+ [More Information Needed]
171
+
172
+ #### Software
173
+
174
+ [More Information Needed]
175
+
176
+ ## Citation [optional]
177
+
178
+ <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
179
+
180
+ **BibTeX:**
181
+
182
+ [More Information Needed]
183
+
184
+ **APA:**
185
+
186
+ [More Information Needed]
187
+
188
+ ## Glossary [optional]
189
+
190
+ <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
191
+
192
+ [More Information Needed]
193
+
194
+ ## More Information [optional]
195
+
196
+ [More Information Needed]
197
+
198
+ ## Model Card Authors [optional]
199
+
200
+ [More Information Needed]
201
+
202
+ ## Model Card Contact
203
+
204
+ [More Information Needed]
205
+ ### Framework versions
206
+
207
+ - PEFT 0.17.1
v3-results/adapter/adapter_config.json ADDED
@@ -0,0 +1,42 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "alpha_pattern": {},
3
+ "auto_mapping": null,
4
+ "base_model_name_or_path": "SEE7/Astheral",
5
+ "bias": "none",
6
+ "corda_config": null,
7
+ "eva_config": null,
8
+ "exclude_modules": null,
9
+ "fan_in_fan_out": false,
10
+ "inference_mode": true,
11
+ "init_lora_weights": true,
12
+ "layer_replication": null,
13
+ "layers_pattern": null,
14
+ "layers_to_transform": null,
15
+ "loftq_config": {},
16
+ "lora_alpha": 32,
17
+ "lora_bias": false,
18
+ "lora_dropout": 0.05,
19
+ "megatron_config": null,
20
+ "megatron_core": "megatron.core",
21
+ "modules_to_save": null,
22
+ "peft_type": "LORA",
23
+ "qalora_group_size": 16,
24
+ "r": 16,
25
+ "rank_pattern": {},
26
+ "revision": "d9aa899722be4788f57107c2951c72d0ec6614ee",
27
+ "target_modules": [
28
+ "gate_proj",
29
+ "o_proj",
30
+ "up_proj",
31
+ "down_proj",
32
+ "v_proj",
33
+ "q_proj",
34
+ "k_proj"
35
+ ],
36
+ "target_parameters": null,
37
+ "task_type": "CAUSAL_LM",
38
+ "trainable_token_indices": null,
39
+ "use_dora": false,
40
+ "use_qalora": false,
41
+ "use_rslora": false
42
+ }
v3-results/adapter/adapter_model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d9a26fcd5590725a2cfdfec6246c93ea91e237e3bf34e214b366b2387feb00ae
3
+ size 167832240
v3-results/adapter/additional_chat_templates/tool_use.jinja ADDED
@@ -0,0 +1,152 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {%- macro json_to_python_type(json_spec) %}
2
+ {%- set basic_type_map = {
3
+ "string": "str",
4
+ "number": "float",
5
+ "integer": "int",
6
+ "boolean": "bool"
7
+ } %}
8
+
9
+ {%- if basic_type_map[json_spec.type] is defined %}
10
+ {{- basic_type_map[json_spec.type] }}
11
+ {%- elif json_spec.type == "array" %}
12
+ {{- "list[" + json_to_python_type(json_spec|items) + "]"}}
13
+ {%- elif json_spec.type == "object" %}
14
+ {%- if json_spec.additionalProperties is defined %}
15
+ {{- "dict[str, " + json_to_python_type(json_spec.additionalProperties) + ']'}}
16
+ {%- else %}
17
+ {{- "dict" }}
18
+ {%- endif %}
19
+ {%- elif json_spec.type is iterable %}
20
+ {{- "Union[" }}
21
+ {%- for t in json_spec.type %}
22
+ {{- json_to_python_type({"type": t}) }}
23
+ {%- if not loop.last %}
24
+ {{- "," }}
25
+ {%- endif %}
26
+ {%- endfor %}
27
+ {{- "]" }}
28
+ {%- else %}
29
+ {{- "Any" }}
30
+ {%- endif %}
31
+ {%- endmacro %}
32
+
33
+
34
+ {{- bos_token }}
35
+ {{- '<|im_start|>system
36
+ ' }}
37
+ {{- "You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools: <tools> " }}
38
+ {%- for tool in tools %}
39
+ {%- if tool.function is defined %}
40
+ {%- set tool = tool.function %}
41
+ {%- endif %}
42
+ {{- '{"type": "function", "function": ' }}
43
+ {{- '{"name": "' + tool.name + '", ' }}
44
+ {{- '"description": "' + tool.name + '(' }}
45
+ {%- for param_name, param_fields in tool.parameters.properties|items %}
46
+ {{- param_name + ": " + json_to_python_type(param_fields) }}
47
+ {%- if not loop.last %}
48
+ {{- ", " }}
49
+ {%- endif %}
50
+ {%- endfor %}
51
+ {{- ")" }}
52
+ {%- if tool.return is defined %}
53
+ {{- " -> " + json_to_python_type(tool.return) }}
54
+ {%- endif %}
55
+ {{- " - " + tool.description + "
56
+
57
+ " }}
58
+ {%- for param_name, param_fields in tool.parameters.properties|items %}
59
+ {%- if loop.first %}
60
+ {{- " Args:
61
+ " }}
62
+ {%- endif %}
63
+ {{- " " + param_name + "(" + json_to_python_type(param_fields) + "): " + param_fields.description|trim }}
64
+ {%- endfor %}
65
+ {%- if tool.return is defined and tool.return.description is defined %}
66
+ {{- "
67
+ Returns:
68
+ " + tool.return.description }}
69
+ {%- endif %}
70
+ {{- '"' }}
71
+ {{- ', "parameters": ' }}
72
+ {%- if tool.parameters.properties | length == 0 %}
73
+ {{- "{}" }}
74
+ {%- else %}
75
+ {{- tool.parameters|tojson }}
76
+ {%- endif %}
77
+ {{- "}" }}
78
+ {%- if not loop.last %}
79
+ {{- "
80
+ " }}
81
+ {%- endif %}
82
+ {%- endfor %}
83
+ {{- " </tools>" }}
84
+ {{- 'Use the following pydantic model json schema for each tool call you will make: {"properties": {"name": {"title": "Name", "type": "string"}, "arguments": {"title": "Arguments", "type": "object"}}, "required": ["name", "arguments"], "title": "FunctionCall", "type": "object"}}
85
+ ' }}
86
+ {{- "For each function call return a json object with function name and arguments within <tool_call></tool_call> XML tags as follows:
87
+ " }}
88
+ {{- "<tool_call>
89
+ " }}
90
+ {{- '{"name": <function-name>, "arguments": <args-dict>}
91
+ ' }}
92
+ {{- '</tool_call><|im_end|>
93
+ ' }}
94
+ {%- for message in messages %}
95
+ {%- if message.role == "user" or message.role == "system" or (message.role == "assistant" and message.tool_calls is not defined) %}
96
+ {{- '<|im_start|>' + message.role + '
97
+ ' + message.content + '<|im_end|>' + '
98
+ ' }}
99
+ {%- elif message.role == "assistant" %}
100
+ {{- '<|im_start|>' + message.role }}
101
+ {%- for tool_call in message.tool_calls %}
102
+ {{- '
103
+ <tool_call>
104
+ ' }} {%- if tool_call.function is defined %}
105
+ {%- set tool_call = tool_call.function %}
106
+ {%- endif %}
107
+ {{- '{' }}
108
+ {{- '"name": "' }}
109
+ {{- tool_call.name }}
110
+ {{- '"' }}
111
+ {{- ', '}}
112
+ {%- if tool_call.arguments is defined %}
113
+ {{- '"arguments": ' }}
114
+ {%- if tool_call.arguments is string %}
115
+ {{- tool_call.arguments }}
116
+ {%- else %}
117
+ {{- tool_call.arguments|tojson }}
118
+ {%- endif %}
119
+ {%- endif %}
120
+ {{- '}' }}
121
+ {{- '
122
+ </tool_call>' }}
123
+ {%- endfor %}
124
+ {{- '<|im_end|>
125
+ ' }}
126
+ {%- elif message.role == "tool" %}
127
+ {%- if loop.previtem and loop.previtem.role != "tool" %}
128
+ {{- '<|im_start|>tool
129
+ ' }}
130
+ {%- endif %}
131
+ {{- '<tool_response>
132
+ ' }}
133
+ {{- message.content }}
134
+ {%- if not loop.last %}
135
+ {{- '
136
+ </tool_response>
137
+ ' }}
138
+ {%- else %}
139
+ {{- '
140
+ </tool_response>' }}
141
+ {%- endif %}
142
+ {%- if not loop.last and loop.nextitem.role != "tool" %}
143
+ {{- '<|im_end|>' }}
144
+ {%- elif loop.last %}
145
+ {{- '<|im_end|>' }}
146
+ {%- endif %}
147
+ {%- endif %}
148
+ {%- endfor %}
149
+ {%- if add_generation_prompt %}
150
+ {{- '<|im_start|>assistant
151
+ ' }}
152
+ {%- endif %}
v3-results/adapter/chat_template.jinja ADDED
@@ -0,0 +1,5 @@
 
 
 
 
 
 
1
+ {% if not add_generation_prompt is defined %}{% set add_generation_prompt = false %}{% endif %}{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>
2
+
3
+ '+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{% if add_generation_prompt %}{{ '<|start_header_id|>assistant<|end_header_id|>
4
+
5
+ ' }}{% endif %}
v3-results/adapter/special_tokens_map.json ADDED
@@ -0,0 +1,17 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token": {
3
+ "content": "<|begin_of_text|>",
4
+ "lstrip": false,
5
+ "normalized": false,
6
+ "rstrip": false,
7
+ "single_word": false
8
+ },
9
+ "eos_token": {
10
+ "content": "<|eot_id|>",
11
+ "lstrip": false,
12
+ "normalized": false,
13
+ "rstrip": false,
14
+ "single_word": false
15
+ },
16
+ "pad_token": "<|finetune_right_pad_id|>"
17
+ }
v3-results/adapter/tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6b9e4e7fb171f92fd137b777cc2714bf87d11576700a1dcd7a399e7bbe39537b
3
+ size 17209920
v3-results/adapter/tokenizer_config.json ADDED
@@ -0,0 +1,2069 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_bos_token": true,
3
+ "added_tokens_decoder": {
4
+ "128000": {
5
+ "content": "<|begin_of_text|>",
6
+ "lstrip": false,
7
+ "normalized": false,
8
+ "rstrip": false,
9
+ "single_word": false,
10
+ "special": true
11
+ },
12
+ "128001": {
13
+ "content": "<|end_of_text|>",
14
+ "lstrip": false,
15
+ "normalized": false,
16
+ "rstrip": false,
17
+ "single_word": false,
18
+ "special": true
19
+ },
20
+ "128002": {
21
+ "content": "<|reserved_special_token_0|>",
22
+ "lstrip": false,
23
+ "normalized": false,
24
+ "rstrip": false,
25
+ "single_word": false,
26
+ "special": true
27
+ },
28
+ "128003": {
29
+ "content": "<|reserved_special_token_1|>",
30
+ "lstrip": false,
31
+ "normalized": false,
32
+ "rstrip": false,
33
+ "single_word": false,
34
+ "special": true
35
+ },
36
+ "128004": {
37
+ "content": "<|finetune_right_pad_id|>",
38
+ "lstrip": false,
39
+ "normalized": false,
40
+ "rstrip": false,
41
+ "single_word": false,
42
+ "special": true
43
+ },
44
+ "128005": {
45
+ "content": "<|reserved_special_token_2|>",
46
+ "lstrip": false,
47
+ "normalized": false,
48
+ "rstrip": false,
49
+ "single_word": false,
50
+ "special": true
51
+ },
52
+ "128006": {
53
+ "content": "<|start_header_id|>",
54
+ "lstrip": false,
55
+ "normalized": false,
56
+ "rstrip": false,
57
+ "single_word": false,
58
+ "special": true
59
+ },
60
+ "128007": {
61
+ "content": "<|end_header_id|>",
62
+ "lstrip": false,
63
+ "normalized": false,
64
+ "rstrip": false,
65
+ "single_word": false,
66
+ "special": true
67
+ },
68
+ "128008": {
69
+ "content": "<|eom_id|>",
70
+ "lstrip": false,
71
+ "normalized": false,
72
+ "rstrip": false,
73
+ "single_word": false,
74
+ "special": true
75
+ },
76
+ "128009": {
77
+ "content": "<|eot_id|>",
78
+ "lstrip": false,
79
+ "normalized": false,
80
+ "rstrip": false,
81
+ "single_word": false,
82
+ "special": true
83
+ },
84
+ "128010": {
85
+ "content": "<|python_tag|>",
86
+ "lstrip": false,
87
+ "normalized": false,
88
+ "rstrip": false,
89
+ "single_word": false,
90
+ "special": true
91
+ },
92
+ "128011": {
93
+ "content": "<|reserved_special_token_3|>",
94
+ "lstrip": false,
95
+ "normalized": false,
96
+ "rstrip": false,
97
+ "single_word": false,
98
+ "special": true
99
+ },
100
+ "128012": {
101
+ "content": "<|reserved_special_token_4|>",
102
+ "lstrip": false,
103
+ "normalized": false,
104
+ "rstrip": false,
105
+ "single_word": false,
106
+ "special": true
107
+ },
108
+ "128013": {
109
+ "content": "<|reserved_special_token_5|>",
110
+ "lstrip": false,
111
+ "normalized": false,
112
+ "rstrip": false,
113
+ "single_word": false,
114
+ "special": true
115
+ },
116
+ "128014": {
117
+ "content": "<|reserved_special_token_6|>",
118
+ "lstrip": false,
119
+ "normalized": false,
120
+ "rstrip": false,
121
+ "single_word": false,
122
+ "special": true
123
+ },
124
+ "128015": {
125
+ "content": "<|reserved_special_token_7|>",
126
+ "lstrip": false,
127
+ "normalized": false,
128
+ "rstrip": false,
129
+ "single_word": false,
130
+ "special": true
131
+ },
132
+ "128016": {
133
+ "content": "<|reserved_special_token_8|>",
134
+ "lstrip": false,
135
+ "normalized": false,
136
+ "rstrip": false,
137
+ "single_word": false,
138
+ "special": true
139
+ },
140
+ "128017": {
141
+ "content": "<|reserved_special_token_9|>",
142
+ "lstrip": false,
143
+ "normalized": false,
144
+ "rstrip": false,
145
+ "single_word": false,
146
+ "special": true
147
+ },
148
+ "128018": {
149
+ "content": "<|reserved_special_token_10|>",
150
+ "lstrip": false,
151
+ "normalized": false,
152
+ "rstrip": false,
153
+ "single_word": false,
154
+ "special": true
155
+ },
156
+ "128019": {
157
+ "content": "<|reserved_special_token_11|>",
158
+ "lstrip": false,
159
+ "normalized": false,
160
+ "rstrip": false,
161
+ "single_word": false,
162
+ "special": true
163
+ },
164
+ "128020": {
165
+ "content": "<|reserved_special_token_12|>",
166
+ "lstrip": false,
167
+ "normalized": false,
168
+ "rstrip": false,
169
+ "single_word": false,
170
+ "special": true
171
+ },
172
+ "128021": {
173
+ "content": "<|reserved_special_token_13|>",
174
+ "lstrip": false,
175
+ "normalized": false,
176
+ "rstrip": false,
177
+ "single_word": false,
178
+ "special": true
179
+ },
180
+ "128022": {
181
+ "content": "<|reserved_special_token_14|>",
182
+ "lstrip": false,
183
+ "normalized": false,
184
+ "rstrip": false,
185
+ "single_word": false,
186
+ "special": true
187
+ },
188
+ "128023": {
189
+ "content": "<|reserved_special_token_15|>",
190
+ "lstrip": false,
191
+ "normalized": false,
192
+ "rstrip": false,
193
+ "single_word": false,
194
+ "special": true
195
+ },
196
+ "128024": {
197
+ "content": "<|reserved_special_token_16|>",
198
+ "lstrip": false,
199
+ "normalized": false,
200
+ "rstrip": false,
201
+ "single_word": false,
202
+ "special": true
203
+ },
204
+ "128025": {
205
+ "content": "<|reserved_special_token_17|>",
206
+ "lstrip": false,
207
+ "normalized": false,
208
+ "rstrip": false,
209
+ "single_word": false,
210
+ "special": true
211
+ },
212
+ "128026": {
213
+ "content": "<|reserved_special_token_18|>",
214
+ "lstrip": false,
215
+ "normalized": false,
216
+ "rstrip": false,
217
+ "single_word": false,
218
+ "special": true
219
+ },
220
+ "128027": {
221
+ "content": "<|reserved_special_token_19|>",
222
+ "lstrip": false,
223
+ "normalized": false,
224
+ "rstrip": false,
225
+ "single_word": false,
226
+ "special": true
227
+ },
228
+ "128028": {
229
+ "content": "<|reserved_special_token_20|>",
230
+ "lstrip": false,
231
+ "normalized": false,
232
+ "rstrip": false,
233
+ "single_word": false,
234
+ "special": true
235
+ },
236
+ "128029": {
237
+ "content": "<|reserved_special_token_21|>",
238
+ "lstrip": false,
239
+ "normalized": false,
240
+ "rstrip": false,
241
+ "single_word": false,
242
+ "special": true
243
+ },
244
+ "128030": {
245
+ "content": "<|reserved_special_token_22|>",
246
+ "lstrip": false,
247
+ "normalized": false,
248
+ "rstrip": false,
249
+ "single_word": false,
250
+ "special": true
251
+ },
252
+ "128031": {
253
+ "content": "<|reserved_special_token_23|>",
254
+ "lstrip": false,
255
+ "normalized": false,
256
+ "rstrip": false,
257
+ "single_word": false,
258
+ "special": true
259
+ },
260
+ "128032": {
261
+ "content": "<|reserved_special_token_24|>",
262
+ "lstrip": false,
263
+ "normalized": false,
264
+ "rstrip": false,
265
+ "single_word": false,
266
+ "special": true
267
+ },
268
+ "128033": {
269
+ "content": "<|reserved_special_token_25|>",
270
+ "lstrip": false,
271
+ "normalized": false,
272
+ "rstrip": false,
273
+ "single_word": false,
274
+ "special": true
275
+ },
276
+ "128034": {
277
+ "content": "<|reserved_special_token_26|>",
278
+ "lstrip": false,
279
+ "normalized": false,
280
+ "rstrip": false,
281
+ "single_word": false,
282
+ "special": true
283
+ },
284
+ "128035": {
285
+ "content": "<|reserved_special_token_27|>",
286
+ "lstrip": false,
287
+ "normalized": false,
288
+ "rstrip": false,
289
+ "single_word": false,
290
+ "special": true
291
+ },
292
+ "128036": {
293
+ "content": "<|reserved_special_token_28|>",
294
+ "lstrip": false,
295
+ "normalized": false,
296
+ "rstrip": false,
297
+ "single_word": false,
298
+ "special": true
299
+ },
300
+ "128037": {
301
+ "content": "<|reserved_special_token_29|>",
302
+ "lstrip": false,
303
+ "normalized": false,
304
+ "rstrip": false,
305
+ "single_word": false,
306
+ "special": true
307
+ },
308
+ "128038": {
309
+ "content": "<|reserved_special_token_30|>",
310
+ "lstrip": false,
311
+ "normalized": false,
312
+ "rstrip": false,
313
+ "single_word": false,
314
+ "special": true
315
+ },
316
+ "128039": {
317
+ "content": "<|reserved_special_token_31|>",
318
+ "lstrip": false,
319
+ "normalized": false,
320
+ "rstrip": false,
321
+ "single_word": false,
322
+ "special": true
323
+ },
324
+ "128040": {
325
+ "content": "<|reserved_special_token_32|>",
326
+ "lstrip": false,
327
+ "normalized": false,
328
+ "rstrip": false,
329
+ "single_word": false,
330
+ "special": true
331
+ },
332
+ "128041": {
333
+ "content": "<|reserved_special_token_33|>",
334
+ "lstrip": false,
335
+ "normalized": false,
336
+ "rstrip": false,
337
+ "single_word": false,
338
+ "special": true
339
+ },
340
+ "128042": {
341
+ "content": "<|reserved_special_token_34|>",
342
+ "lstrip": false,
343
+ "normalized": false,
344
+ "rstrip": false,
345
+ "single_word": false,
346
+ "special": true
347
+ },
348
+ "128043": {
349
+ "content": "<|reserved_special_token_35|>",
350
+ "lstrip": false,
351
+ "normalized": false,
352
+ "rstrip": false,
353
+ "single_word": false,
354
+ "special": true
355
+ },
356
+ "128044": {
357
+ "content": "<|reserved_special_token_36|>",
358
+ "lstrip": false,
359
+ "normalized": false,
360
+ "rstrip": false,
361
+ "single_word": false,
362
+ "special": true
363
+ },
364
+ "128045": {
365
+ "content": "<|reserved_special_token_37|>",
366
+ "lstrip": false,
367
+ "normalized": false,
368
+ "rstrip": false,
369
+ "single_word": false,
370
+ "special": true
371
+ },
372
+ "128046": {
373
+ "content": "<|reserved_special_token_38|>",
374
+ "lstrip": false,
375
+ "normalized": false,
376
+ "rstrip": false,
377
+ "single_word": false,
378
+ "special": true
379
+ },
380
+ "128047": {
381
+ "content": "<|reserved_special_token_39|>",
382
+ "lstrip": false,
383
+ "normalized": false,
384
+ "rstrip": false,
385
+ "single_word": false,
386
+ "special": true
387
+ },
388
+ "128048": {
389
+ "content": "<|reserved_special_token_40|>",
390
+ "lstrip": false,
391
+ "normalized": false,
392
+ "rstrip": false,
393
+ "single_word": false,
394
+ "special": true
395
+ },
396
+ "128049": {
397
+ "content": "<|reserved_special_token_41|>",
398
+ "lstrip": false,
399
+ "normalized": false,
400
+ "rstrip": false,
401
+ "single_word": false,
402
+ "special": true
403
+ },
404
+ "128050": {
405
+ "content": "<|reserved_special_token_42|>",
406
+ "lstrip": false,
407
+ "normalized": false,
408
+ "rstrip": false,
409
+ "single_word": false,
410
+ "special": true
411
+ },
412
+ "128051": {
413
+ "content": "<|reserved_special_token_43|>",
414
+ "lstrip": false,
415
+ "normalized": false,
416
+ "rstrip": false,
417
+ "single_word": false,
418
+ "special": true
419
+ },
420
+ "128052": {
421
+ "content": "<|reserved_special_token_44|>",
422
+ "lstrip": false,
423
+ "normalized": false,
424
+ "rstrip": false,
425
+ "single_word": false,
426
+ "special": true
427
+ },
428
+ "128053": {
429
+ "content": "<|reserved_special_token_45|>",
430
+ "lstrip": false,
431
+ "normalized": false,
432
+ "rstrip": false,
433
+ "single_word": false,
434
+ "special": true
435
+ },
436
+ "128054": {
437
+ "content": "<|reserved_special_token_46|>",
438
+ "lstrip": false,
439
+ "normalized": false,
440
+ "rstrip": false,
441
+ "single_word": false,
442
+ "special": true
443
+ },
444
+ "128055": {
445
+ "content": "<|reserved_special_token_47|>",
446
+ "lstrip": false,
447
+ "normalized": false,
448
+ "rstrip": false,
449
+ "single_word": false,
450
+ "special": true
451
+ },
452
+ "128056": {
453
+ "content": "<|reserved_special_token_48|>",
454
+ "lstrip": false,
455
+ "normalized": false,
456
+ "rstrip": false,
457
+ "single_word": false,
458
+ "special": true
459
+ },
460
+ "128057": {
461
+ "content": "<|reserved_special_token_49|>",
462
+ "lstrip": false,
463
+ "normalized": false,
464
+ "rstrip": false,
465
+ "single_word": false,
466
+ "special": true
467
+ },
468
+ "128058": {
469
+ "content": "<|reserved_special_token_50|>",
470
+ "lstrip": false,
471
+ "normalized": false,
472
+ "rstrip": false,
473
+ "single_word": false,
474
+ "special": true
475
+ },
476
+ "128059": {
477
+ "content": "<|reserved_special_token_51|>",
478
+ "lstrip": false,
479
+ "normalized": false,
480
+ "rstrip": false,
481
+ "single_word": false,
482
+ "special": true
483
+ },
484
+ "128060": {
485
+ "content": "<|reserved_special_token_52|>",
486
+ "lstrip": false,
487
+ "normalized": false,
488
+ "rstrip": false,
489
+ "single_word": false,
490
+ "special": true
491
+ },
492
+ "128061": {
493
+ "content": "<|reserved_special_token_53|>",
494
+ "lstrip": false,
495
+ "normalized": false,
496
+ "rstrip": false,
497
+ "single_word": false,
498
+ "special": true
499
+ },
500
+ "128062": {
501
+ "content": "<|reserved_special_token_54|>",
502
+ "lstrip": false,
503
+ "normalized": false,
504
+ "rstrip": false,
505
+ "single_word": false,
506
+ "special": true
507
+ },
508
+ "128063": {
509
+ "content": "<|reserved_special_token_55|>",
510
+ "lstrip": false,
511
+ "normalized": false,
512
+ "rstrip": false,
513
+ "single_word": false,
514
+ "special": true
515
+ },
516
+ "128064": {
517
+ "content": "<|reserved_special_token_56|>",
518
+ "lstrip": false,
519
+ "normalized": false,
520
+ "rstrip": false,
521
+ "single_word": false,
522
+ "special": true
523
+ },
524
+ "128065": {
525
+ "content": "<|reserved_special_token_57|>",
526
+ "lstrip": false,
527
+ "normalized": false,
528
+ "rstrip": false,
529
+ "single_word": false,
530
+ "special": true
531
+ },
532
+ "128066": {
533
+ "content": "<|reserved_special_token_58|>",
534
+ "lstrip": false,
535
+ "normalized": false,
536
+ "rstrip": false,
537
+ "single_word": false,
538
+ "special": true
539
+ },
540
+ "128067": {
541
+ "content": "<|reserved_special_token_59|>",
542
+ "lstrip": false,
543
+ "normalized": false,
544
+ "rstrip": false,
545
+ "single_word": false,
546
+ "special": true
547
+ },
548
+ "128068": {
549
+ "content": "<|reserved_special_token_60|>",
550
+ "lstrip": false,
551
+ "normalized": false,
552
+ "rstrip": false,
553
+ "single_word": false,
554
+ "special": true
555
+ },
556
+ "128069": {
557
+ "content": "<|reserved_special_token_61|>",
558
+ "lstrip": false,
559
+ "normalized": false,
560
+ "rstrip": false,
561
+ "single_word": false,
562
+ "special": true
563
+ },
564
+ "128070": {
565
+ "content": "<|reserved_special_token_62|>",
566
+ "lstrip": false,
567
+ "normalized": false,
568
+ "rstrip": false,
569
+ "single_word": false,
570
+ "special": true
571
+ },
572
+ "128071": {
573
+ "content": "<|reserved_special_token_63|>",
574
+ "lstrip": false,
575
+ "normalized": false,
576
+ "rstrip": false,
577
+ "single_word": false,
578
+ "special": true
579
+ },
580
+ "128072": {
581
+ "content": "<|reserved_special_token_64|>",
582
+ "lstrip": false,
583
+ "normalized": false,
584
+ "rstrip": false,
585
+ "single_word": false,
586
+ "special": true
587
+ },
588
+ "128073": {
589
+ "content": "<|reserved_special_token_65|>",
590
+ "lstrip": false,
591
+ "normalized": false,
592
+ "rstrip": false,
593
+ "single_word": false,
594
+ "special": true
595
+ },
596
+ "128074": {
597
+ "content": "<|reserved_special_token_66|>",
598
+ "lstrip": false,
599
+ "normalized": false,
600
+ "rstrip": false,
601
+ "single_word": false,
602
+ "special": true
603
+ },
604
+ "128075": {
605
+ "content": "<|reserved_special_token_67|>",
606
+ "lstrip": false,
607
+ "normalized": false,
608
+ "rstrip": false,
609
+ "single_word": false,
610
+ "special": true
611
+ },
612
+ "128076": {
613
+ "content": "<|reserved_special_token_68|>",
614
+ "lstrip": false,
615
+ "normalized": false,
616
+ "rstrip": false,
617
+ "single_word": false,
618
+ "special": true
619
+ },
620
+ "128077": {
621
+ "content": "<|reserved_special_token_69|>",
622
+ "lstrip": false,
623
+ "normalized": false,
624
+ "rstrip": false,
625
+ "single_word": false,
626
+ "special": true
627
+ },
628
+ "128078": {
629
+ "content": "<|reserved_special_token_70|>",
630
+ "lstrip": false,
631
+ "normalized": false,
632
+ "rstrip": false,
633
+ "single_word": false,
634
+ "special": true
635
+ },
636
+ "128079": {
637
+ "content": "<|reserved_special_token_71|>",
638
+ "lstrip": false,
639
+ "normalized": false,
640
+ "rstrip": false,
641
+ "single_word": false,
642
+ "special": true
643
+ },
644
+ "128080": {
645
+ "content": "<|reserved_special_token_72|>",
646
+ "lstrip": false,
647
+ "normalized": false,
648
+ "rstrip": false,
649
+ "single_word": false,
650
+ "special": true
651
+ },
652
+ "128081": {
653
+ "content": "<|reserved_special_token_73|>",
654
+ "lstrip": false,
655
+ "normalized": false,
656
+ "rstrip": false,
657
+ "single_word": false,
658
+ "special": true
659
+ },
660
+ "128082": {
661
+ "content": "<|reserved_special_token_74|>",
662
+ "lstrip": false,
663
+ "normalized": false,
664
+ "rstrip": false,
665
+ "single_word": false,
666
+ "special": true
667
+ },
668
+ "128083": {
669
+ "content": "<|reserved_special_token_75|>",
670
+ "lstrip": false,
671
+ "normalized": false,
672
+ "rstrip": false,
673
+ "single_word": false,
674
+ "special": true
675
+ },
676
+ "128084": {
677
+ "content": "<|reserved_special_token_76|>",
678
+ "lstrip": false,
679
+ "normalized": false,
680
+ "rstrip": false,
681
+ "single_word": false,
682
+ "special": true
683
+ },
684
+ "128085": {
685
+ "content": "<|reserved_special_token_77|>",
686
+ "lstrip": false,
687
+ "normalized": false,
688
+ "rstrip": false,
689
+ "single_word": false,
690
+ "special": true
691
+ },
692
+ "128086": {
693
+ "content": "<|reserved_special_token_78|>",
694
+ "lstrip": false,
695
+ "normalized": false,
696
+ "rstrip": false,
697
+ "single_word": false,
698
+ "special": true
699
+ },
700
+ "128087": {
701
+ "content": "<|reserved_special_token_79|>",
702
+ "lstrip": false,
703
+ "normalized": false,
704
+ "rstrip": false,
705
+ "single_word": false,
706
+ "special": true
707
+ },
708
+ "128088": {
709
+ "content": "<|reserved_special_token_80|>",
710
+ "lstrip": false,
711
+ "normalized": false,
712
+ "rstrip": false,
713
+ "single_word": false,
714
+ "special": true
715
+ },
716
+ "128089": {
717
+ "content": "<|reserved_special_token_81|>",
718
+ "lstrip": false,
719
+ "normalized": false,
720
+ "rstrip": false,
721
+ "single_word": false,
722
+ "special": true
723
+ },
724
+ "128090": {
725
+ "content": "<|reserved_special_token_82|>",
726
+ "lstrip": false,
727
+ "normalized": false,
728
+ "rstrip": false,
729
+ "single_word": false,
730
+ "special": true
731
+ },
732
+ "128091": {
733
+ "content": "<|reserved_special_token_83|>",
734
+ "lstrip": false,
735
+ "normalized": false,
736
+ "rstrip": false,
737
+ "single_word": false,
738
+ "special": true
739
+ },
740
+ "128092": {
741
+ "content": "<|reserved_special_token_84|>",
742
+ "lstrip": false,
743
+ "normalized": false,
744
+ "rstrip": false,
745
+ "single_word": false,
746
+ "special": true
747
+ },
748
+ "128093": {
749
+ "content": "<|reserved_special_token_85|>",
750
+ "lstrip": false,
751
+ "normalized": false,
752
+ "rstrip": false,
753
+ "single_word": false,
754
+ "special": true
755
+ },
756
+ "128094": {
757
+ "content": "<|reserved_special_token_86|>",
758
+ "lstrip": false,
759
+ "normalized": false,
760
+ "rstrip": false,
761
+ "single_word": false,
762
+ "special": true
763
+ },
764
+ "128095": {
765
+ "content": "<|reserved_special_token_87|>",
766
+ "lstrip": false,
767
+ "normalized": false,
768
+ "rstrip": false,
769
+ "single_word": false,
770
+ "special": true
771
+ },
772
+ "128096": {
773
+ "content": "<|reserved_special_token_88|>",
774
+ "lstrip": false,
775
+ "normalized": false,
776
+ "rstrip": false,
777
+ "single_word": false,
778
+ "special": true
779
+ },
780
+ "128097": {
781
+ "content": "<|reserved_special_token_89|>",
782
+ "lstrip": false,
783
+ "normalized": false,
784
+ "rstrip": false,
785
+ "single_word": false,
786
+ "special": true
787
+ },
788
+ "128098": {
789
+ "content": "<|reserved_special_token_90|>",
790
+ "lstrip": false,
791
+ "normalized": false,
792
+ "rstrip": false,
793
+ "single_word": false,
794
+ "special": true
795
+ },
796
+ "128099": {
797
+ "content": "<|reserved_special_token_91|>",
798
+ "lstrip": false,
799
+ "normalized": false,
800
+ "rstrip": false,
801
+ "single_word": false,
802
+ "special": true
803
+ },
804
+ "128100": {
805
+ "content": "<|reserved_special_token_92|>",
806
+ "lstrip": false,
807
+ "normalized": false,
808
+ "rstrip": false,
809
+ "single_word": false,
810
+ "special": true
811
+ },
812
+ "128101": {
813
+ "content": "<|reserved_special_token_93|>",
814
+ "lstrip": false,
815
+ "normalized": false,
816
+ "rstrip": false,
817
+ "single_word": false,
818
+ "special": true
819
+ },
820
+ "128102": {
821
+ "content": "<|reserved_special_token_94|>",
822
+ "lstrip": false,
823
+ "normalized": false,
824
+ "rstrip": false,
825
+ "single_word": false,
826
+ "special": true
827
+ },
828
+ "128103": {
829
+ "content": "<|reserved_special_token_95|>",
830
+ "lstrip": false,
831
+ "normalized": false,
832
+ "rstrip": false,
833
+ "single_word": false,
834
+ "special": true
835
+ },
836
+ "128104": {
837
+ "content": "<|reserved_special_token_96|>",
838
+ "lstrip": false,
839
+ "normalized": false,
840
+ "rstrip": false,
841
+ "single_word": false,
842
+ "special": true
843
+ },
844
+ "128105": {
845
+ "content": "<|reserved_special_token_97|>",
846
+ "lstrip": false,
847
+ "normalized": false,
848
+ "rstrip": false,
849
+ "single_word": false,
850
+ "special": true
851
+ },
852
+ "128106": {
853
+ "content": "<|reserved_special_token_98|>",
854
+ "lstrip": false,
855
+ "normalized": false,
856
+ "rstrip": false,
857
+ "single_word": false,
858
+ "special": true
859
+ },
860
+ "128107": {
861
+ "content": "<|reserved_special_token_99|>",
862
+ "lstrip": false,
863
+ "normalized": false,
864
+ "rstrip": false,
865
+ "single_word": false,
866
+ "special": true
867
+ },
868
+ "128108": {
869
+ "content": "<|reserved_special_token_100|>",
870
+ "lstrip": false,
871
+ "normalized": false,
872
+ "rstrip": false,
873
+ "single_word": false,
874
+ "special": true
875
+ },
876
+ "128109": {
877
+ "content": "<|reserved_special_token_101|>",
878
+ "lstrip": false,
879
+ "normalized": false,
880
+ "rstrip": false,
881
+ "single_word": false,
882
+ "special": true
883
+ },
884
+ "128110": {
885
+ "content": "<|reserved_special_token_102|>",
886
+ "lstrip": false,
887
+ "normalized": false,
888
+ "rstrip": false,
889
+ "single_word": false,
890
+ "special": true
891
+ },
892
+ "128111": {
893
+ "content": "<|reserved_special_token_103|>",
894
+ "lstrip": false,
895
+ "normalized": false,
896
+ "rstrip": false,
897
+ "single_word": false,
898
+ "special": true
899
+ },
900
+ "128112": {
901
+ "content": "<|reserved_special_token_104|>",
902
+ "lstrip": false,
903
+ "normalized": false,
904
+ "rstrip": false,
905
+ "single_word": false,
906
+ "special": true
907
+ },
908
+ "128113": {
909
+ "content": "<|reserved_special_token_105|>",
910
+ "lstrip": false,
911
+ "normalized": false,
912
+ "rstrip": false,
913
+ "single_word": false,
914
+ "special": true
915
+ },
916
+ "128114": {
917
+ "content": "<|reserved_special_token_106|>",
918
+ "lstrip": false,
919
+ "normalized": false,
920
+ "rstrip": false,
921
+ "single_word": false,
922
+ "special": true
923
+ },
924
+ "128115": {
925
+ "content": "<|reserved_special_token_107|>",
926
+ "lstrip": false,
927
+ "normalized": false,
928
+ "rstrip": false,
929
+ "single_word": false,
930
+ "special": true
931
+ },
932
+ "128116": {
933
+ "content": "<|reserved_special_token_108|>",
934
+ "lstrip": false,
935
+ "normalized": false,
936
+ "rstrip": false,
937
+ "single_word": false,
938
+ "special": true
939
+ },
940
+ "128117": {
941
+ "content": "<|reserved_special_token_109|>",
942
+ "lstrip": false,
943
+ "normalized": false,
944
+ "rstrip": false,
945
+ "single_word": false,
946
+ "special": true
947
+ },
948
+ "128118": {
949
+ "content": "<|reserved_special_token_110|>",
950
+ "lstrip": false,
951
+ "normalized": false,
952
+ "rstrip": false,
953
+ "single_word": false,
954
+ "special": true
955
+ },
956
+ "128119": {
957
+ "content": "<|reserved_special_token_111|>",
958
+ "lstrip": false,
959
+ "normalized": false,
960
+ "rstrip": false,
961
+ "single_word": false,
962
+ "special": true
963
+ },
964
+ "128120": {
965
+ "content": "<|reserved_special_token_112|>",
966
+ "lstrip": false,
967
+ "normalized": false,
968
+ "rstrip": false,
969
+ "single_word": false,
970
+ "special": true
971
+ },
972
+ "128121": {
973
+ "content": "<|reserved_special_token_113|>",
974
+ "lstrip": false,
975
+ "normalized": false,
976
+ "rstrip": false,
977
+ "single_word": false,
978
+ "special": true
979
+ },
980
+ "128122": {
981
+ "content": "<|reserved_special_token_114|>",
982
+ "lstrip": false,
983
+ "normalized": false,
984
+ "rstrip": false,
985
+ "single_word": false,
986
+ "special": true
987
+ },
988
+ "128123": {
989
+ "content": "<|reserved_special_token_115|>",
990
+ "lstrip": false,
991
+ "normalized": false,
992
+ "rstrip": false,
993
+ "single_word": false,
994
+ "special": true
995
+ },
996
+ "128124": {
997
+ "content": "<|reserved_special_token_116|>",
998
+ "lstrip": false,
999
+ "normalized": false,
1000
+ "rstrip": false,
1001
+ "single_word": false,
1002
+ "special": true
1003
+ },
1004
+ "128125": {
1005
+ "content": "<|reserved_special_token_117|>",
1006
+ "lstrip": false,
1007
+ "normalized": false,
1008
+ "rstrip": false,
1009
+ "single_word": false,
1010
+ "special": true
1011
+ },
1012
+ "128126": {
1013
+ "content": "<|reserved_special_token_118|>",
1014
+ "lstrip": false,
1015
+ "normalized": false,
1016
+ "rstrip": false,
1017
+ "single_word": false,
1018
+ "special": true
1019
+ },
1020
+ "128127": {
1021
+ "content": "<|reserved_special_token_119|>",
1022
+ "lstrip": false,
1023
+ "normalized": false,
1024
+ "rstrip": false,
1025
+ "single_word": false,
1026
+ "special": true
1027
+ },
1028
+ "128128": {
1029
+ "content": "<|reserved_special_token_120|>",
1030
+ "lstrip": false,
1031
+ "normalized": false,
1032
+ "rstrip": false,
1033
+ "single_word": false,
1034
+ "special": true
1035
+ },
1036
+ "128129": {
1037
+ "content": "<|reserved_special_token_121|>",
1038
+ "lstrip": false,
1039
+ "normalized": false,
1040
+ "rstrip": false,
1041
+ "single_word": false,
1042
+ "special": true
1043
+ },
1044
+ "128130": {
1045
+ "content": "<|reserved_special_token_122|>",
1046
+ "lstrip": false,
1047
+ "normalized": false,
1048
+ "rstrip": false,
1049
+ "single_word": false,
1050
+ "special": true
1051
+ },
1052
+ "128131": {
1053
+ "content": "<|reserved_special_token_123|>",
1054
+ "lstrip": false,
1055
+ "normalized": false,
1056
+ "rstrip": false,
1057
+ "single_word": false,
1058
+ "special": true
1059
+ },
1060
+ "128132": {
1061
+ "content": "<|reserved_special_token_124|>",
1062
+ "lstrip": false,
1063
+ "normalized": false,
1064
+ "rstrip": false,
1065
+ "single_word": false,
1066
+ "special": true
1067
+ },
1068
+ "128133": {
1069
+ "content": "<|reserved_special_token_125|>",
1070
+ "lstrip": false,
1071
+ "normalized": false,
1072
+ "rstrip": false,
1073
+ "single_word": false,
1074
+ "special": true
1075
+ },
1076
+ "128134": {
1077
+ "content": "<|reserved_special_token_126|>",
1078
+ "lstrip": false,
1079
+ "normalized": false,
1080
+ "rstrip": false,
1081
+ "single_word": false,
1082
+ "special": true
1083
+ },
1084
+ "128135": {
1085
+ "content": "<|reserved_special_token_127|>",
1086
+ "lstrip": false,
1087
+ "normalized": false,
1088
+ "rstrip": false,
1089
+ "single_word": false,
1090
+ "special": true
1091
+ },
1092
+ "128136": {
1093
+ "content": "<|reserved_special_token_128|>",
1094
+ "lstrip": false,
1095
+ "normalized": false,
1096
+ "rstrip": false,
1097
+ "single_word": false,
1098
+ "special": true
1099
+ },
1100
+ "128137": {
1101
+ "content": "<|reserved_special_token_129|>",
1102
+ "lstrip": false,
1103
+ "normalized": false,
1104
+ "rstrip": false,
1105
+ "single_word": false,
1106
+ "special": true
1107
+ },
1108
+ "128138": {
1109
+ "content": "<|reserved_special_token_130|>",
1110
+ "lstrip": false,
1111
+ "normalized": false,
1112
+ "rstrip": false,
1113
+ "single_word": false,
1114
+ "special": true
1115
+ },
1116
+ "128139": {
1117
+ "content": "<|reserved_special_token_131|>",
1118
+ "lstrip": false,
1119
+ "normalized": false,
1120
+ "rstrip": false,
1121
+ "single_word": false,
1122
+ "special": true
1123
+ },
1124
+ "128140": {
1125
+ "content": "<|reserved_special_token_132|>",
1126
+ "lstrip": false,
1127
+ "normalized": false,
1128
+ "rstrip": false,
1129
+ "single_word": false,
1130
+ "special": true
1131
+ },
1132
+ "128141": {
1133
+ "content": "<|reserved_special_token_133|>",
1134
+ "lstrip": false,
1135
+ "normalized": false,
1136
+ "rstrip": false,
1137
+ "single_word": false,
1138
+ "special": true
1139
+ },
1140
+ "128142": {
1141
+ "content": "<|reserved_special_token_134|>",
1142
+ "lstrip": false,
1143
+ "normalized": false,
1144
+ "rstrip": false,
1145
+ "single_word": false,
1146
+ "special": true
1147
+ },
1148
+ "128143": {
1149
+ "content": "<|reserved_special_token_135|>",
1150
+ "lstrip": false,
1151
+ "normalized": false,
1152
+ "rstrip": false,
1153
+ "single_word": false,
1154
+ "special": true
1155
+ },
1156
+ "128144": {
1157
+ "content": "<|reserved_special_token_136|>",
1158
+ "lstrip": false,
1159
+ "normalized": false,
1160
+ "rstrip": false,
1161
+ "single_word": false,
1162
+ "special": true
1163
+ },
1164
+ "128145": {
1165
+ "content": "<|reserved_special_token_137|>",
1166
+ "lstrip": false,
1167
+ "normalized": false,
1168
+ "rstrip": false,
1169
+ "single_word": false,
1170
+ "special": true
1171
+ },
1172
+ "128146": {
1173
+ "content": "<|reserved_special_token_138|>",
1174
+ "lstrip": false,
1175
+ "normalized": false,
1176
+ "rstrip": false,
1177
+ "single_word": false,
1178
+ "special": true
1179
+ },
1180
+ "128147": {
1181
+ "content": "<|reserved_special_token_139|>",
1182
+ "lstrip": false,
1183
+ "normalized": false,
1184
+ "rstrip": false,
1185
+ "single_word": false,
1186
+ "special": true
1187
+ },
1188
+ "128148": {
1189
+ "content": "<|reserved_special_token_140|>",
1190
+ "lstrip": false,
1191
+ "normalized": false,
1192
+ "rstrip": false,
1193
+ "single_word": false,
1194
+ "special": true
1195
+ },
1196
+ "128149": {
1197
+ "content": "<|reserved_special_token_141|>",
1198
+ "lstrip": false,
1199
+ "normalized": false,
1200
+ "rstrip": false,
1201
+ "single_word": false,
1202
+ "special": true
1203
+ },
1204
+ "128150": {
1205
+ "content": "<|reserved_special_token_142|>",
1206
+ "lstrip": false,
1207
+ "normalized": false,
1208
+ "rstrip": false,
1209
+ "single_word": false,
1210
+ "special": true
1211
+ },
1212
+ "128151": {
1213
+ "content": "<|reserved_special_token_143|>",
1214
+ "lstrip": false,
1215
+ "normalized": false,
1216
+ "rstrip": false,
1217
+ "single_word": false,
1218
+ "special": true
1219
+ },
1220
+ "128152": {
1221
+ "content": "<|reserved_special_token_144|>",
1222
+ "lstrip": false,
1223
+ "normalized": false,
1224
+ "rstrip": false,
1225
+ "single_word": false,
1226
+ "special": true
1227
+ },
1228
+ "128153": {
1229
+ "content": "<|reserved_special_token_145|>",
1230
+ "lstrip": false,
1231
+ "normalized": false,
1232
+ "rstrip": false,
1233
+ "single_word": false,
1234
+ "special": true
1235
+ },
1236
+ "128154": {
1237
+ "content": "<|reserved_special_token_146|>",
1238
+ "lstrip": false,
1239
+ "normalized": false,
1240
+ "rstrip": false,
1241
+ "single_word": false,
1242
+ "special": true
1243
+ },
1244
+ "128155": {
1245
+ "content": "<|reserved_special_token_147|>",
1246
+ "lstrip": false,
1247
+ "normalized": false,
1248
+ "rstrip": false,
1249
+ "single_word": false,
1250
+ "special": true
1251
+ },
1252
+ "128156": {
1253
+ "content": "<|reserved_special_token_148|>",
1254
+ "lstrip": false,
1255
+ "normalized": false,
1256
+ "rstrip": false,
1257
+ "single_word": false,
1258
+ "special": true
1259
+ },
1260
+ "128157": {
1261
+ "content": "<|reserved_special_token_149|>",
1262
+ "lstrip": false,
1263
+ "normalized": false,
1264
+ "rstrip": false,
1265
+ "single_word": false,
1266
+ "special": true
1267
+ },
1268
+ "128158": {
1269
+ "content": "<|reserved_special_token_150|>",
1270
+ "lstrip": false,
1271
+ "normalized": false,
1272
+ "rstrip": false,
1273
+ "single_word": false,
1274
+ "special": true
1275
+ },
1276
+ "128159": {
1277
+ "content": "<|reserved_special_token_151|>",
1278
+ "lstrip": false,
1279
+ "normalized": false,
1280
+ "rstrip": false,
1281
+ "single_word": false,
1282
+ "special": true
1283
+ },
1284
+ "128160": {
1285
+ "content": "<|reserved_special_token_152|>",
1286
+ "lstrip": false,
1287
+ "normalized": false,
1288
+ "rstrip": false,
1289
+ "single_word": false,
1290
+ "special": true
1291
+ },
1292
+ "128161": {
1293
+ "content": "<|reserved_special_token_153|>",
1294
+ "lstrip": false,
1295
+ "normalized": false,
1296
+ "rstrip": false,
1297
+ "single_word": false,
1298
+ "special": true
1299
+ },
1300
+ "128162": {
1301
+ "content": "<|reserved_special_token_154|>",
1302
+ "lstrip": false,
1303
+ "normalized": false,
1304
+ "rstrip": false,
1305
+ "single_word": false,
1306
+ "special": true
1307
+ },
1308
+ "128163": {
1309
+ "content": "<|reserved_special_token_155|>",
1310
+ "lstrip": false,
1311
+ "normalized": false,
1312
+ "rstrip": false,
1313
+ "single_word": false,
1314
+ "special": true
1315
+ },
1316
+ "128164": {
1317
+ "content": "<|reserved_special_token_156|>",
1318
+ "lstrip": false,
1319
+ "normalized": false,
1320
+ "rstrip": false,
1321
+ "single_word": false,
1322
+ "special": true
1323
+ },
1324
+ "128165": {
1325
+ "content": "<|reserved_special_token_157|>",
1326
+ "lstrip": false,
1327
+ "normalized": false,
1328
+ "rstrip": false,
1329
+ "single_word": false,
1330
+ "special": true
1331
+ },
1332
+ "128166": {
1333
+ "content": "<|reserved_special_token_158|>",
1334
+ "lstrip": false,
1335
+ "normalized": false,
1336
+ "rstrip": false,
1337
+ "single_word": false,
1338
+ "special": true
1339
+ },
1340
+ "128167": {
1341
+ "content": "<|reserved_special_token_159|>",
1342
+ "lstrip": false,
1343
+ "normalized": false,
1344
+ "rstrip": false,
1345
+ "single_word": false,
1346
+ "special": true
1347
+ },
1348
+ "128168": {
1349
+ "content": "<|reserved_special_token_160|>",
1350
+ "lstrip": false,
1351
+ "normalized": false,
1352
+ "rstrip": false,
1353
+ "single_word": false,
1354
+ "special": true
1355
+ },
1356
+ "128169": {
1357
+ "content": "<|reserved_special_token_161|>",
1358
+ "lstrip": false,
1359
+ "normalized": false,
1360
+ "rstrip": false,
1361
+ "single_word": false,
1362
+ "special": true
1363
+ },
1364
+ "128170": {
1365
+ "content": "<|reserved_special_token_162|>",
1366
+ "lstrip": false,
1367
+ "normalized": false,
1368
+ "rstrip": false,
1369
+ "single_word": false,
1370
+ "special": true
1371
+ },
1372
+ "128171": {
1373
+ "content": "<|reserved_special_token_163|>",
1374
+ "lstrip": false,
1375
+ "normalized": false,
1376
+ "rstrip": false,
1377
+ "single_word": false,
1378
+ "special": true
1379
+ },
1380
+ "128172": {
1381
+ "content": "<|reserved_special_token_164|>",
1382
+ "lstrip": false,
1383
+ "normalized": false,
1384
+ "rstrip": false,
1385
+ "single_word": false,
1386
+ "special": true
1387
+ },
1388
+ "128173": {
1389
+ "content": "<|reserved_special_token_165|>",
1390
+ "lstrip": false,
1391
+ "normalized": false,
1392
+ "rstrip": false,
1393
+ "single_word": false,
1394
+ "special": true
1395
+ },
1396
+ "128174": {
1397
+ "content": "<|reserved_special_token_166|>",
1398
+ "lstrip": false,
1399
+ "normalized": false,
1400
+ "rstrip": false,
1401
+ "single_word": false,
1402
+ "special": true
1403
+ },
1404
+ "128175": {
1405
+ "content": "<|reserved_special_token_167|>",
1406
+ "lstrip": false,
1407
+ "normalized": false,
1408
+ "rstrip": false,
1409
+ "single_word": false,
1410
+ "special": true
1411
+ },
1412
+ "128176": {
1413
+ "content": "<|reserved_special_token_168|>",
1414
+ "lstrip": false,
1415
+ "normalized": false,
1416
+ "rstrip": false,
1417
+ "single_word": false,
1418
+ "special": true
1419
+ },
1420
+ "128177": {
1421
+ "content": "<|reserved_special_token_169|>",
1422
+ "lstrip": false,
1423
+ "normalized": false,
1424
+ "rstrip": false,
1425
+ "single_word": false,
1426
+ "special": true
1427
+ },
1428
+ "128178": {
1429
+ "content": "<|reserved_special_token_170|>",
1430
+ "lstrip": false,
1431
+ "normalized": false,
1432
+ "rstrip": false,
1433
+ "single_word": false,
1434
+ "special": true
1435
+ },
1436
+ "128179": {
1437
+ "content": "<|reserved_special_token_171|>",
1438
+ "lstrip": false,
1439
+ "normalized": false,
1440
+ "rstrip": false,
1441
+ "single_word": false,
1442
+ "special": true
1443
+ },
1444
+ "128180": {
1445
+ "content": "<|reserved_special_token_172|>",
1446
+ "lstrip": false,
1447
+ "normalized": false,
1448
+ "rstrip": false,
1449
+ "single_word": false,
1450
+ "special": true
1451
+ },
1452
+ "128181": {
1453
+ "content": "<|reserved_special_token_173|>",
1454
+ "lstrip": false,
1455
+ "normalized": false,
1456
+ "rstrip": false,
1457
+ "single_word": false,
1458
+ "special": true
1459
+ },
1460
+ "128182": {
1461
+ "content": "<|reserved_special_token_174|>",
1462
+ "lstrip": false,
1463
+ "normalized": false,
1464
+ "rstrip": false,
1465
+ "single_word": false,
1466
+ "special": true
1467
+ },
1468
+ "128183": {
1469
+ "content": "<|reserved_special_token_175|>",
1470
+ "lstrip": false,
1471
+ "normalized": false,
1472
+ "rstrip": false,
1473
+ "single_word": false,
1474
+ "special": true
1475
+ },
1476
+ "128184": {
1477
+ "content": "<|reserved_special_token_176|>",
1478
+ "lstrip": false,
1479
+ "normalized": false,
1480
+ "rstrip": false,
1481
+ "single_word": false,
1482
+ "special": true
1483
+ },
1484
+ "128185": {
1485
+ "content": "<|reserved_special_token_177|>",
1486
+ "lstrip": false,
1487
+ "normalized": false,
1488
+ "rstrip": false,
1489
+ "single_word": false,
1490
+ "special": true
1491
+ },
1492
+ "128186": {
1493
+ "content": "<|reserved_special_token_178|>",
1494
+ "lstrip": false,
1495
+ "normalized": false,
1496
+ "rstrip": false,
1497
+ "single_word": false,
1498
+ "special": true
1499
+ },
1500
+ "128187": {
1501
+ "content": "<|reserved_special_token_179|>",
1502
+ "lstrip": false,
1503
+ "normalized": false,
1504
+ "rstrip": false,
1505
+ "single_word": false,
1506
+ "special": true
1507
+ },
1508
+ "128188": {
1509
+ "content": "<|reserved_special_token_180|>",
1510
+ "lstrip": false,
1511
+ "normalized": false,
1512
+ "rstrip": false,
1513
+ "single_word": false,
1514
+ "special": true
1515
+ },
1516
+ "128189": {
1517
+ "content": "<|reserved_special_token_181|>",
1518
+ "lstrip": false,
1519
+ "normalized": false,
1520
+ "rstrip": false,
1521
+ "single_word": false,
1522
+ "special": true
1523
+ },
1524
+ "128190": {
1525
+ "content": "<|reserved_special_token_182|>",
1526
+ "lstrip": false,
1527
+ "normalized": false,
1528
+ "rstrip": false,
1529
+ "single_word": false,
1530
+ "special": true
1531
+ },
1532
+ "128191": {
1533
+ "content": "<|reserved_special_token_183|>",
1534
+ "lstrip": false,
1535
+ "normalized": false,
1536
+ "rstrip": false,
1537
+ "single_word": false,
1538
+ "special": true
1539
+ },
1540
+ "128192": {
1541
+ "content": "<|reserved_special_token_184|>",
1542
+ "lstrip": false,
1543
+ "normalized": false,
1544
+ "rstrip": false,
1545
+ "single_word": false,
1546
+ "special": true
1547
+ },
1548
+ "128193": {
1549
+ "content": "<|reserved_special_token_185|>",
1550
+ "lstrip": false,
1551
+ "normalized": false,
1552
+ "rstrip": false,
1553
+ "single_word": false,
1554
+ "special": true
1555
+ },
1556
+ "128194": {
1557
+ "content": "<|reserved_special_token_186|>",
1558
+ "lstrip": false,
1559
+ "normalized": false,
1560
+ "rstrip": false,
1561
+ "single_word": false,
1562
+ "special": true
1563
+ },
1564
+ "128195": {
1565
+ "content": "<|reserved_special_token_187|>",
1566
+ "lstrip": false,
1567
+ "normalized": false,
1568
+ "rstrip": false,
1569
+ "single_word": false,
1570
+ "special": true
1571
+ },
1572
+ "128196": {
1573
+ "content": "<|reserved_special_token_188|>",
1574
+ "lstrip": false,
1575
+ "normalized": false,
1576
+ "rstrip": false,
1577
+ "single_word": false,
1578
+ "special": true
1579
+ },
1580
+ "128197": {
1581
+ "content": "<|reserved_special_token_189|>",
1582
+ "lstrip": false,
1583
+ "normalized": false,
1584
+ "rstrip": false,
1585
+ "single_word": false,
1586
+ "special": true
1587
+ },
1588
+ "128198": {
1589
+ "content": "<|reserved_special_token_190|>",
1590
+ "lstrip": false,
1591
+ "normalized": false,
1592
+ "rstrip": false,
1593
+ "single_word": false,
1594
+ "special": true
1595
+ },
1596
+ "128199": {
1597
+ "content": "<|reserved_special_token_191|>",
1598
+ "lstrip": false,
1599
+ "normalized": false,
1600
+ "rstrip": false,
1601
+ "single_word": false,
1602
+ "special": true
1603
+ },
1604
+ "128200": {
1605
+ "content": "<|reserved_special_token_192|>",
1606
+ "lstrip": false,
1607
+ "normalized": false,
1608
+ "rstrip": false,
1609
+ "single_word": false,
1610
+ "special": true
1611
+ },
1612
+ "128201": {
1613
+ "content": "<|reserved_special_token_193|>",
1614
+ "lstrip": false,
1615
+ "normalized": false,
1616
+ "rstrip": false,
1617
+ "single_word": false,
1618
+ "special": true
1619
+ },
1620
+ "128202": {
1621
+ "content": "<|reserved_special_token_194|>",
1622
+ "lstrip": false,
1623
+ "normalized": false,
1624
+ "rstrip": false,
1625
+ "single_word": false,
1626
+ "special": true
1627
+ },
1628
+ "128203": {
1629
+ "content": "<|reserved_special_token_195|>",
1630
+ "lstrip": false,
1631
+ "normalized": false,
1632
+ "rstrip": false,
1633
+ "single_word": false,
1634
+ "special": true
1635
+ },
1636
+ "128204": {
1637
+ "content": "<|reserved_special_token_196|>",
1638
+ "lstrip": false,
1639
+ "normalized": false,
1640
+ "rstrip": false,
1641
+ "single_word": false,
1642
+ "special": true
1643
+ },
1644
+ "128205": {
1645
+ "content": "<|reserved_special_token_197|>",
1646
+ "lstrip": false,
1647
+ "normalized": false,
1648
+ "rstrip": false,
1649
+ "single_word": false,
1650
+ "special": true
1651
+ },
1652
+ "128206": {
1653
+ "content": "<|reserved_special_token_198|>",
1654
+ "lstrip": false,
1655
+ "normalized": false,
1656
+ "rstrip": false,
1657
+ "single_word": false,
1658
+ "special": true
1659
+ },
1660
+ "128207": {
1661
+ "content": "<|reserved_special_token_199|>",
1662
+ "lstrip": false,
1663
+ "normalized": false,
1664
+ "rstrip": false,
1665
+ "single_word": false,
1666
+ "special": true
1667
+ },
1668
+ "128208": {
1669
+ "content": "<|reserved_special_token_200|>",
1670
+ "lstrip": false,
1671
+ "normalized": false,
1672
+ "rstrip": false,
1673
+ "single_word": false,
1674
+ "special": true
1675
+ },
1676
+ "128209": {
1677
+ "content": "<|reserved_special_token_201|>",
1678
+ "lstrip": false,
1679
+ "normalized": false,
1680
+ "rstrip": false,
1681
+ "single_word": false,
1682
+ "special": true
1683
+ },
1684
+ "128210": {
1685
+ "content": "<|reserved_special_token_202|>",
1686
+ "lstrip": false,
1687
+ "normalized": false,
1688
+ "rstrip": false,
1689
+ "single_word": false,
1690
+ "special": true
1691
+ },
1692
+ "128211": {
1693
+ "content": "<|reserved_special_token_203|>",
1694
+ "lstrip": false,
1695
+ "normalized": false,
1696
+ "rstrip": false,
1697
+ "single_word": false,
1698
+ "special": true
1699
+ },
1700
+ "128212": {
1701
+ "content": "<|reserved_special_token_204|>",
1702
+ "lstrip": false,
1703
+ "normalized": false,
1704
+ "rstrip": false,
1705
+ "single_word": false,
1706
+ "special": true
1707
+ },
1708
+ "128213": {
1709
+ "content": "<|reserved_special_token_205|>",
1710
+ "lstrip": false,
1711
+ "normalized": false,
1712
+ "rstrip": false,
1713
+ "single_word": false,
1714
+ "special": true
1715
+ },
1716
+ "128214": {
1717
+ "content": "<|reserved_special_token_206|>",
1718
+ "lstrip": false,
1719
+ "normalized": false,
1720
+ "rstrip": false,
1721
+ "single_word": false,
1722
+ "special": true
1723
+ },
1724
+ "128215": {
1725
+ "content": "<|reserved_special_token_207|>",
1726
+ "lstrip": false,
1727
+ "normalized": false,
1728
+ "rstrip": false,
1729
+ "single_word": false,
1730
+ "special": true
1731
+ },
1732
+ "128216": {
1733
+ "content": "<|reserved_special_token_208|>",
1734
+ "lstrip": false,
1735
+ "normalized": false,
1736
+ "rstrip": false,
1737
+ "single_word": false,
1738
+ "special": true
1739
+ },
1740
+ "128217": {
1741
+ "content": "<|reserved_special_token_209|>",
1742
+ "lstrip": false,
1743
+ "normalized": false,
1744
+ "rstrip": false,
1745
+ "single_word": false,
1746
+ "special": true
1747
+ },
1748
+ "128218": {
1749
+ "content": "<|reserved_special_token_210|>",
1750
+ "lstrip": false,
1751
+ "normalized": false,
1752
+ "rstrip": false,
1753
+ "single_word": false,
1754
+ "special": true
1755
+ },
1756
+ "128219": {
1757
+ "content": "<|reserved_special_token_211|>",
1758
+ "lstrip": false,
1759
+ "normalized": false,
1760
+ "rstrip": false,
1761
+ "single_word": false,
1762
+ "special": true
1763
+ },
1764
+ "128220": {
1765
+ "content": "<|reserved_special_token_212|>",
1766
+ "lstrip": false,
1767
+ "normalized": false,
1768
+ "rstrip": false,
1769
+ "single_word": false,
1770
+ "special": true
1771
+ },
1772
+ "128221": {
1773
+ "content": "<|reserved_special_token_213|>",
1774
+ "lstrip": false,
1775
+ "normalized": false,
1776
+ "rstrip": false,
1777
+ "single_word": false,
1778
+ "special": true
1779
+ },
1780
+ "128222": {
1781
+ "content": "<|reserved_special_token_214|>",
1782
+ "lstrip": false,
1783
+ "normalized": false,
1784
+ "rstrip": false,
1785
+ "single_word": false,
1786
+ "special": true
1787
+ },
1788
+ "128223": {
1789
+ "content": "<|reserved_special_token_215|>",
1790
+ "lstrip": false,
1791
+ "normalized": false,
1792
+ "rstrip": false,
1793
+ "single_word": false,
1794
+ "special": true
1795
+ },
1796
+ "128224": {
1797
+ "content": "<|reserved_special_token_216|>",
1798
+ "lstrip": false,
1799
+ "normalized": false,
1800
+ "rstrip": false,
1801
+ "single_word": false,
1802
+ "special": true
1803
+ },
1804
+ "128225": {
1805
+ "content": "<|reserved_special_token_217|>",
1806
+ "lstrip": false,
1807
+ "normalized": false,
1808
+ "rstrip": false,
1809
+ "single_word": false,
1810
+ "special": true
1811
+ },
1812
+ "128226": {
1813
+ "content": "<|reserved_special_token_218|>",
1814
+ "lstrip": false,
1815
+ "normalized": false,
1816
+ "rstrip": false,
1817
+ "single_word": false,
1818
+ "special": true
1819
+ },
1820
+ "128227": {
1821
+ "content": "<|reserved_special_token_219|>",
1822
+ "lstrip": false,
1823
+ "normalized": false,
1824
+ "rstrip": false,
1825
+ "single_word": false,
1826
+ "special": true
1827
+ },
1828
+ "128228": {
1829
+ "content": "<|reserved_special_token_220|>",
1830
+ "lstrip": false,
1831
+ "normalized": false,
1832
+ "rstrip": false,
1833
+ "single_word": false,
1834
+ "special": true
1835
+ },
1836
+ "128229": {
1837
+ "content": "<|reserved_special_token_221|>",
1838
+ "lstrip": false,
1839
+ "normalized": false,
1840
+ "rstrip": false,
1841
+ "single_word": false,
1842
+ "special": true
1843
+ },
1844
+ "128230": {
1845
+ "content": "<|reserved_special_token_222|>",
1846
+ "lstrip": false,
1847
+ "normalized": false,
1848
+ "rstrip": false,
1849
+ "single_word": false,
1850
+ "special": true
1851
+ },
1852
+ "128231": {
1853
+ "content": "<|reserved_special_token_223|>",
1854
+ "lstrip": false,
1855
+ "normalized": false,
1856
+ "rstrip": false,
1857
+ "single_word": false,
1858
+ "special": true
1859
+ },
1860
+ "128232": {
1861
+ "content": "<|reserved_special_token_224|>",
1862
+ "lstrip": false,
1863
+ "normalized": false,
1864
+ "rstrip": false,
1865
+ "single_word": false,
1866
+ "special": true
1867
+ },
1868
+ "128233": {
1869
+ "content": "<|reserved_special_token_225|>",
1870
+ "lstrip": false,
1871
+ "normalized": false,
1872
+ "rstrip": false,
1873
+ "single_word": false,
1874
+ "special": true
1875
+ },
1876
+ "128234": {
1877
+ "content": "<|reserved_special_token_226|>",
1878
+ "lstrip": false,
1879
+ "normalized": false,
1880
+ "rstrip": false,
1881
+ "single_word": false,
1882
+ "special": true
1883
+ },
1884
+ "128235": {
1885
+ "content": "<|reserved_special_token_227|>",
1886
+ "lstrip": false,
1887
+ "normalized": false,
1888
+ "rstrip": false,
1889
+ "single_word": false,
1890
+ "special": true
1891
+ },
1892
+ "128236": {
1893
+ "content": "<|reserved_special_token_228|>",
1894
+ "lstrip": false,
1895
+ "normalized": false,
1896
+ "rstrip": false,
1897
+ "single_word": false,
1898
+ "special": true
1899
+ },
1900
+ "128237": {
1901
+ "content": "<|reserved_special_token_229|>",
1902
+ "lstrip": false,
1903
+ "normalized": false,
1904
+ "rstrip": false,
1905
+ "single_word": false,
1906
+ "special": true
1907
+ },
1908
+ "128238": {
1909
+ "content": "<|reserved_special_token_230|>",
1910
+ "lstrip": false,
1911
+ "normalized": false,
1912
+ "rstrip": false,
1913
+ "single_word": false,
1914
+ "special": true
1915
+ },
1916
+ "128239": {
1917
+ "content": "<|reserved_special_token_231|>",
1918
+ "lstrip": false,
1919
+ "normalized": false,
1920
+ "rstrip": false,
1921
+ "single_word": false,
1922
+ "special": true
1923
+ },
1924
+ "128240": {
1925
+ "content": "<|reserved_special_token_232|>",
1926
+ "lstrip": false,
1927
+ "normalized": false,
1928
+ "rstrip": false,
1929
+ "single_word": false,
1930
+ "special": true
1931
+ },
1932
+ "128241": {
1933
+ "content": "<|reserved_special_token_233|>",
1934
+ "lstrip": false,
1935
+ "normalized": false,
1936
+ "rstrip": false,
1937
+ "single_word": false,
1938
+ "special": true
1939
+ },
1940
+ "128242": {
1941
+ "content": "<|reserved_special_token_234|>",
1942
+ "lstrip": false,
1943
+ "normalized": false,
1944
+ "rstrip": false,
1945
+ "single_word": false,
1946
+ "special": true
1947
+ },
1948
+ "128243": {
1949
+ "content": "<|reserved_special_token_235|>",
1950
+ "lstrip": false,
1951
+ "normalized": false,
1952
+ "rstrip": false,
1953
+ "single_word": false,
1954
+ "special": true
1955
+ },
1956
+ "128244": {
1957
+ "content": "<|reserved_special_token_236|>",
1958
+ "lstrip": false,
1959
+ "normalized": false,
1960
+ "rstrip": false,
1961
+ "single_word": false,
1962
+ "special": true
1963
+ },
1964
+ "128245": {
1965
+ "content": "<|reserved_special_token_237|>",
1966
+ "lstrip": false,
1967
+ "normalized": false,
1968
+ "rstrip": false,
1969
+ "single_word": false,
1970
+ "special": true
1971
+ },
1972
+ "128246": {
1973
+ "content": "<|reserved_special_token_238|>",
1974
+ "lstrip": false,
1975
+ "normalized": false,
1976
+ "rstrip": false,
1977
+ "single_word": false,
1978
+ "special": true
1979
+ },
1980
+ "128247": {
1981
+ "content": "<|reserved_special_token_239|>",
1982
+ "lstrip": false,
1983
+ "normalized": false,
1984
+ "rstrip": false,
1985
+ "single_word": false,
1986
+ "special": true
1987
+ },
1988
+ "128248": {
1989
+ "content": "<|reserved_special_token_240|>",
1990
+ "lstrip": false,
1991
+ "normalized": false,
1992
+ "rstrip": false,
1993
+ "single_word": false,
1994
+ "special": true
1995
+ },
1996
+ "128249": {
1997
+ "content": "<|reserved_special_token_241|>",
1998
+ "lstrip": false,
1999
+ "normalized": false,
2000
+ "rstrip": false,
2001
+ "single_word": false,
2002
+ "special": true
2003
+ },
2004
+ "128250": {
2005
+ "content": "<|reserved_special_token_242|>",
2006
+ "lstrip": false,
2007
+ "normalized": false,
2008
+ "rstrip": false,
2009
+ "single_word": false,
2010
+ "special": true
2011
+ },
2012
+ "128251": {
2013
+ "content": "<|reserved_special_token_243|>",
2014
+ "lstrip": false,
2015
+ "normalized": false,
2016
+ "rstrip": false,
2017
+ "single_word": false,
2018
+ "special": true
2019
+ },
2020
+ "128252": {
2021
+ "content": "<|reserved_special_token_244|>",
2022
+ "lstrip": false,
2023
+ "normalized": false,
2024
+ "rstrip": false,
2025
+ "single_word": false,
2026
+ "special": true
2027
+ },
2028
+ "128253": {
2029
+ "content": "<|reserved_special_token_245|>",
2030
+ "lstrip": false,
2031
+ "normalized": false,
2032
+ "rstrip": false,
2033
+ "single_word": false,
2034
+ "special": true
2035
+ },
2036
+ "128254": {
2037
+ "content": "<|reserved_special_token_246|>",
2038
+ "lstrip": false,
2039
+ "normalized": false,
2040
+ "rstrip": false,
2041
+ "single_word": false,
2042
+ "special": true
2043
+ },
2044
+ "128255": {
2045
+ "content": "<|reserved_special_token_247|>",
2046
+ "lstrip": false,
2047
+ "normalized": false,
2048
+ "rstrip": false,
2049
+ "single_word": false,
2050
+ "special": true
2051
+ }
2052
+ },
2053
+ "bos_token": "<|begin_of_text|>",
2054
+ "clean_up_tokenization_spaces": true,
2055
+ "eos_token": "<|eot_id|>",
2056
+ "extra_special_tokens": {},
2057
+ "max_length": null,
2058
+ "model_input_names": [
2059
+ "input_ids",
2060
+ "attention_mask"
2061
+ ],
2062
+ "model_max_length": 131072,
2063
+ "pad_to_multiple_of": null,
2064
+ "pad_token": "<|finetune_right_pad_id|>",
2065
+ "pad_token_type_id": 0,
2066
+ "padding_side": "right",
2067
+ "tokenizer_class": "PreTrainedTokenizerFast",
2068
+ "unk_token": null
2069
+ }
v3-results/heldout_cases.json ADDED
@@ -0,0 +1,392 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "id": "heldout_00",
4
+ "prompt": "Return only the integer: 701 + 91 - 31.",
5
+ "expected": "761",
6
+ "kind": "integer"
7
+ },
8
+ {
9
+ "id": "heldout_01",
10
+ "prompt": "A crate has 704 red and 92 blue beads. Remove 32 red beads. Return only the total beads remaining.",
11
+ "expected": "764",
12
+ "kind": "integer"
13
+ },
14
+ {
15
+ "id": "heldout_02",
16
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_707, count=93.",
17
+ "expected": {
18
+ "item": "part_707",
19
+ "count": 93
20
+ },
21
+ "kind": "json"
22
+ },
23
+ {
24
+ "id": "heldout_03",
25
+ "prompt": "Return only one CSV value row, no header: name=Item710; quantity=94; code=K34.",
26
+ "expected": "Item710,94,K34",
27
+ "kind": "exact"
28
+ },
29
+ {
30
+ "id": "heldout_04",
31
+ "prompt": "Copy exactly, no other text: Key_713-B95_C35",
32
+ "expected": "Key_713-B95_C35",
33
+ "kind": "exact"
34
+ },
35
+ {
36
+ "id": "heldout_05",
37
+ "prompt": "Write only a Python function named solve(x) that returns x * 96 + 36.",
38
+ "expected": "x * 96 + 36",
39
+ "kind": "code"
40
+ },
41
+ {
42
+ "id": "heldout_06",
43
+ "prompt": "Return only the integer: 719 + 97 - 37.",
44
+ "expected": "779",
45
+ "kind": "integer"
46
+ },
47
+ {
48
+ "id": "heldout_07",
49
+ "prompt": "A crate has 722 red and 98 blue beads. Remove 38 red beads. Return only the total beads remaining.",
50
+ "expected": "782",
51
+ "kind": "integer"
52
+ },
53
+ {
54
+ "id": "heldout_08",
55
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_725, count=99.",
56
+ "expected": {
57
+ "item": "part_725",
58
+ "count": 99
59
+ },
60
+ "kind": "json"
61
+ },
62
+ {
63
+ "id": "heldout_09",
64
+ "prompt": "Return only one CSV value row, no header: name=Item728; quantity=100; code=K40.",
65
+ "expected": "Item728,100,K40",
66
+ "kind": "exact"
67
+ },
68
+ {
69
+ "id": "heldout_10",
70
+ "prompt": "Copy exactly, no other text: Key_731-B101_C41",
71
+ "expected": "Key_731-B101_C41",
72
+ "kind": "exact"
73
+ },
74
+ {
75
+ "id": "heldout_11",
76
+ "prompt": "Write only a Python function named solve(x) that returns x * 102 + 42.",
77
+ "expected": "x * 102 + 42",
78
+ "kind": "code"
79
+ },
80
+ {
81
+ "id": "heldout_12",
82
+ "prompt": "Return only the integer: 737 + 103 - 43.",
83
+ "expected": "797",
84
+ "kind": "integer"
85
+ },
86
+ {
87
+ "id": "heldout_13",
88
+ "prompt": "A crate has 740 red and 104 blue beads. Remove 44 red beads. Return only the total beads remaining.",
89
+ "expected": "800",
90
+ "kind": "integer"
91
+ },
92
+ {
93
+ "id": "heldout_14",
94
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_743, count=105.",
95
+ "expected": {
96
+ "item": "part_743",
97
+ "count": 105
98
+ },
99
+ "kind": "json"
100
+ },
101
+ {
102
+ "id": "heldout_15",
103
+ "prompt": "Return only one CSV value row, no header: name=Item746; quantity=106; code=K46.",
104
+ "expected": "Item746,106,K46",
105
+ "kind": "exact"
106
+ },
107
+ {
108
+ "id": "heldout_16",
109
+ "prompt": "Copy exactly, no other text: Key_749-B107_C47",
110
+ "expected": "Key_749-B107_C47",
111
+ "kind": "exact"
112
+ },
113
+ {
114
+ "id": "heldout_17",
115
+ "prompt": "Write only a Python function named solve(x) that returns x * 108 + 31.",
116
+ "expected": "x * 108 + 31",
117
+ "kind": "code"
118
+ },
119
+ {
120
+ "id": "heldout_18",
121
+ "prompt": "Return only the integer: 755 + 109 - 32.",
122
+ "expected": "832",
123
+ "kind": "integer"
124
+ },
125
+ {
126
+ "id": "heldout_19",
127
+ "prompt": "A crate has 758 red and 110 blue beads. Remove 33 red beads. Return only the total beads remaining.",
128
+ "expected": "835",
129
+ "kind": "integer"
130
+ },
131
+ {
132
+ "id": "heldout_20",
133
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_761, count=111.",
134
+ "expected": {
135
+ "item": "part_761",
136
+ "count": 111
137
+ },
138
+ "kind": "json"
139
+ },
140
+ {
141
+ "id": "heldout_21",
142
+ "prompt": "Return only one CSV value row, no header: name=Item764; quantity=112; code=K35.",
143
+ "expected": "Item764,112,K35",
144
+ "kind": "exact"
145
+ },
146
+ {
147
+ "id": "heldout_22",
148
+ "prompt": "Copy exactly, no other text: Key_767-B113_C36",
149
+ "expected": "Key_767-B113_C36",
150
+ "kind": "exact"
151
+ },
152
+ {
153
+ "id": "heldout_23",
154
+ "prompt": "Write only a Python function named solve(x) that returns x * 114 + 37.",
155
+ "expected": "x * 114 + 37",
156
+ "kind": "code"
157
+ },
158
+ {
159
+ "id": "heldout_24",
160
+ "prompt": "Return only the integer: 773 + 115 - 38.",
161
+ "expected": "850",
162
+ "kind": "integer"
163
+ },
164
+ {
165
+ "id": "heldout_25",
166
+ "prompt": "A crate has 776 red and 116 blue beads. Remove 39 red beads. Return only the total beads remaining.",
167
+ "expected": "853",
168
+ "kind": "integer"
169
+ },
170
+ {
171
+ "id": "heldout_26",
172
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_779, count=117.",
173
+ "expected": {
174
+ "item": "part_779",
175
+ "count": 117
176
+ },
177
+ "kind": "json"
178
+ },
179
+ {
180
+ "id": "heldout_27",
181
+ "prompt": "Return only one CSV value row, no header: name=Item782; quantity=118; code=K41.",
182
+ "expected": "Item782,118,K41",
183
+ "kind": "exact"
184
+ },
185
+ {
186
+ "id": "heldout_28",
187
+ "prompt": "Copy exactly, no other text: Key_785-B119_C42",
188
+ "expected": "Key_785-B119_C42",
189
+ "kind": "exact"
190
+ },
191
+ {
192
+ "id": "heldout_29",
193
+ "prompt": "Write only a Python function named solve(x) that returns x * 120 + 43.",
194
+ "expected": "x * 120 + 43",
195
+ "kind": "code"
196
+ },
197
+ {
198
+ "id": "heldout_30",
199
+ "prompt": "Return only the integer: 791 + 121 - 44.",
200
+ "expected": "868",
201
+ "kind": "integer"
202
+ },
203
+ {
204
+ "id": "heldout_31",
205
+ "prompt": "A crate has 794 red and 122 blue beads. Remove 45 red beads. Return only the total beads remaining.",
206
+ "expected": "871",
207
+ "kind": "integer"
208
+ },
209
+ {
210
+ "id": "heldout_32",
211
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_797, count=123.",
212
+ "expected": {
213
+ "item": "part_797",
214
+ "count": 123
215
+ },
216
+ "kind": "json"
217
+ },
218
+ {
219
+ "id": "heldout_33",
220
+ "prompt": "Return only one CSV value row, no header: name=Item800; quantity=124; code=K47.",
221
+ "expected": "Item800,124,K47",
222
+ "kind": "exact"
223
+ },
224
+ {
225
+ "id": "heldout_34",
226
+ "prompt": "Copy exactly, no other text: Key_803-B125_C31",
227
+ "expected": "Key_803-B125_C31",
228
+ "kind": "exact"
229
+ },
230
+ {
231
+ "id": "heldout_35",
232
+ "prompt": "Write only a Python function named solve(x) that returns x * 126 + 32.",
233
+ "expected": "x * 126 + 32",
234
+ "kind": "code"
235
+ },
236
+ {
237
+ "id": "heldout_36",
238
+ "prompt": "Return only the integer: 809 + 127 - 33.",
239
+ "expected": "903",
240
+ "kind": "integer"
241
+ },
242
+ {
243
+ "id": "heldout_37",
244
+ "prompt": "A crate has 812 red and 128 blue beads. Remove 34 red beads. Return only the total beads remaining.",
245
+ "expected": "906",
246
+ "kind": "integer"
247
+ },
248
+ {
249
+ "id": "heldout_38",
250
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_815, count=129.",
251
+ "expected": {
252
+ "item": "part_815",
253
+ "count": 129
254
+ },
255
+ "kind": "json"
256
+ },
257
+ {
258
+ "id": "heldout_39",
259
+ "prompt": "Return only one CSV value row, no header: name=Item818; quantity=130; code=K36.",
260
+ "expected": "Item818,130,K36",
261
+ "kind": "exact"
262
+ },
263
+ {
264
+ "id": "heldout_40",
265
+ "prompt": "Copy exactly, no other text: Key_821-B131_C37",
266
+ "expected": "Key_821-B131_C37",
267
+ "kind": "exact"
268
+ },
269
+ {
270
+ "id": "heldout_41",
271
+ "prompt": "Write only a Python function named solve(x) that returns x * 132 + 38.",
272
+ "expected": "x * 132 + 38",
273
+ "kind": "code"
274
+ },
275
+ {
276
+ "id": "heldout_42",
277
+ "prompt": "Return only the integer: 827 + 133 - 39.",
278
+ "expected": "921",
279
+ "kind": "integer"
280
+ },
281
+ {
282
+ "id": "heldout_43",
283
+ "prompt": "A crate has 830 red and 134 blue beads. Remove 40 red beads. Return only the total beads remaining.",
284
+ "expected": "924",
285
+ "kind": "integer"
286
+ },
287
+ {
288
+ "id": "heldout_44",
289
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_833, count=135.",
290
+ "expected": {
291
+ "item": "part_833",
292
+ "count": 135
293
+ },
294
+ "kind": "json"
295
+ },
296
+ {
297
+ "id": "heldout_45",
298
+ "prompt": "Return only one CSV value row, no header: name=Item836; quantity=136; code=K42.",
299
+ "expected": "Item836,136,K42",
300
+ "kind": "exact"
301
+ },
302
+ {
303
+ "id": "heldout_46",
304
+ "prompt": "Copy exactly, no other text: Key_839-B137_C43",
305
+ "expected": "Key_839-B137_C43",
306
+ "kind": "exact"
307
+ },
308
+ {
309
+ "id": "heldout_47",
310
+ "prompt": "Write only a Python function named solve(x) that returns x * 138 + 44.",
311
+ "expected": "x * 138 + 44",
312
+ "kind": "code"
313
+ },
314
+ {
315
+ "id": "heldout_48",
316
+ "prompt": "Return only the integer: 845 + 139 - 45.",
317
+ "expected": "939",
318
+ "kind": "integer"
319
+ },
320
+ {
321
+ "id": "heldout_49",
322
+ "prompt": "A crate has 848 red and 140 blue beads. Remove 46 red beads. Return only the total beads remaining.",
323
+ "expected": "942",
324
+ "kind": "integer"
325
+ },
326
+ {
327
+ "id": "heldout_50",
328
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_851, count=141.",
329
+ "expected": {
330
+ "item": "part_851",
331
+ "count": 141
332
+ },
333
+ "kind": "json"
334
+ },
335
+ {
336
+ "id": "heldout_51",
337
+ "prompt": "Return only one CSV value row, no header: name=Item854; quantity=142; code=K31.",
338
+ "expected": "Item854,142,K31",
339
+ "kind": "exact"
340
+ },
341
+ {
342
+ "id": "heldout_52",
343
+ "prompt": "Copy exactly, no other text: Key_857-B143_C32",
344
+ "expected": "Key_857-B143_C32",
345
+ "kind": "exact"
346
+ },
347
+ {
348
+ "id": "heldout_53",
349
+ "prompt": "Write only a Python function named solve(x) that returns x * 144 + 33.",
350
+ "expected": "x * 144 + 33",
351
+ "kind": "code"
352
+ },
353
+ {
354
+ "id": "heldout_54",
355
+ "prompt": "Return only the integer: 863 + 145 - 34.",
356
+ "expected": "974",
357
+ "kind": "integer"
358
+ },
359
+ {
360
+ "id": "heldout_55",
361
+ "prompt": "A crate has 866 red and 146 blue beads. Remove 35 red beads. Return only the total beads remaining.",
362
+ "expected": "977",
363
+ "kind": "integer"
364
+ },
365
+ {
366
+ "id": "heldout_56",
367
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_869, count=147.",
368
+ "expected": {
369
+ "item": "part_869",
370
+ "count": 147
371
+ },
372
+ "kind": "json"
373
+ },
374
+ {
375
+ "id": "heldout_57",
376
+ "prompt": "Return only one CSV value row, no header: name=Item872; quantity=148; code=K37.",
377
+ "expected": "Item872,148,K37",
378
+ "kind": "exact"
379
+ },
380
+ {
381
+ "id": "heldout_58",
382
+ "prompt": "Copy exactly, no other text: Key_875-B149_C38",
383
+ "expected": "Key_875-B149_C38",
384
+ "kind": "exact"
385
+ },
386
+ {
387
+ "id": "heldout_59",
388
+ "prompt": "Write only a Python function named solve(x) that returns x * 150 + 39.",
389
+ "expected": "x * 150 + 39",
390
+ "kind": "code"
391
+ }
392
+ ]
v3-results/train_v3.py ADDED
@@ -0,0 +1,128 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # /// script
2
+ # dependencies = ["torch==2.8.0", "transformers==4.56.2", "peft==0.17.1", "accelerate==1.10.1", "huggingface_hub==0.34.4", "bitsandbytes==0.47.0", "safetensors==0.6.2"]
3
+ # ///
4
+ """Budget-bounded V3 training with paired, disjoint evaluation and reload gate."""
5
+ import os, json, random, time, re, gc, hashlib, traceback
6
+ from pathlib import Path
7
+ import torch
8
+ from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, Trainer, TrainingArguments, TrainerCallback, set_seed
9
+ from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training, PeftModel
10
+ from huggingface_hub import HfApi, hf_hub_download
11
+
12
+ REPO='SEE7/Astheral'; REV='d9aa899722be4788f57107c2951c72d0ec6614ee'
13
+ OUT=Path('/tmp/astheral-v3'); OUT.mkdir(exist_ok=True)
14
+ api=HfApi(); started=time.monotonic(); set_seed(20261113); rng=random.Random(20261113)
15
+ assert torch.cuda.is_available() and 'A10G' in torch.cuda.get_device_name(0)
16
+ branch='v3-candidate-20261113'
17
+ api.create_branch(REPO,branch=branch,revision=REV,exist_ok=True)
18
+ settings=json.loads(Path(hf_hub_download(REPO,'settings.json',revision=REV)).read_text())
19
+ tok=AutoTokenizer.from_pretrained(REPO,revision=REV); tok.pad_token_id=128004
20
+ report={'status':'preparing','parent_revision':REV,'hardware':torch.cuda.get_device_name(0),'job_timeout_seconds':5400,'listed_usd_per_hour':1.0,'training_seed':20261113}
21
+
22
+ def publish():
23
+ report['elapsed_seconds']=round(time.monotonic()-started,2)
24
+ (OUT/'v3_report.json').write_text(json.dumps(report,indent=2))
25
+ api.upload_folder(repo_id=REPO,folder_path=OUT,revision=branch,path_in_repo='v3-results',ignore_patterns=['merged/**'],commit_message='Record measured V3 training and evaluation artifacts')
26
+
27
+ def sample(a,b,c,kind):
28
+ if kind==0:return f'Return only the integer: {a} + {b} - {c}.',str(a+b-c),'integer'
29
+ if kind==1:return f'A crate has {a} red and {b} blue beads. Remove {c} red beads. Return only the total beads remaining.',str(a+b-c),'integer'
30
+ if kind==2:return f'Return only valid JSON with keys "item" and "count": item=part_{a}, count={b}.',{'item':f'part_{a}','count':b},'json'
31
+ if kind==3:return f'Return only one CSV value row, no header: name=Item{a}; quantity={b}; code=K{c}.',f'Item{a},{b},K{c}','exact'
32
+ if kind==4:return f'Copy exactly, no other text: Key_{a}-B{b}_C{c}',f'Key_{a}-B{b}_C{c}','exact'
33
+ return f'Write only a Python function named solve(x) that returns x * {b} + {c}.',f'x * {b} + {c}','code'
34
+
35
+ def grade(case,text):
36
+ expected=case['expected']; kind=case['kind']; text=text.strip()
37
+ try:
38
+ if kind=='json':return json.loads(text)==expected
39
+ if kind=='integer':return re.fullmatch(r'-?\d+',text) is not None and int(text)==int(expected)
40
+ if kind=='exact':return text==expected
41
+ # Parse and evaluate a tiny arithmetic-only AST; never execute model-generated code.
42
+ import ast
43
+ clean=re.sub(r'^```(?:python)?\s*|\s*```$','',text).strip();tree=ast.parse(clean)
44
+ if len(tree.body)!=1 or not isinstance(tree.body[0],ast.FunctionDef):return False
45
+ fn=tree.body[0]
46
+ if fn.name!='solve' or len(fn.args.args)!=1 or fn.args.args[0].arg!='x' or len(fn.body)!=1 or not isinstance(fn.body[0],ast.Return):return False
47
+ def calc(n,x):
48
+ if isinstance(n,ast.Constant) and isinstance(n.value,(int,float)):return n.value
49
+ if isinstance(n,ast.Name) and n.id=='x':return x
50
+ if isinstance(n,ast.BinOp) and isinstance(n.op,(ast.Add,ast.Sub,ast.Mult)):
51
+ a,b=calc(n.left,x),calc(n.right,x)
52
+ return a+b if isinstance(n.op,ast.Add) else a-b if isinstance(n.op,ast.Sub) else a*b
53
+ raise ValueError('Unsupported expression')
54
+ match=re.fullmatch(r'x \* (\d+) \+ (\d+)',expected);b,c=map(int,match.groups())
55
+ return all(calc(fn.body[0].value,x)==x*b+c for x in [-5,0,1,7,100])
56
+ except Exception:return False
57
+
58
+ tests=[]
59
+ for i in range(60):
60
+ # Evaluation numbers are outside the synthetic training ranges.
61
+ a,b,c=701+i*3,91+i,31+i%17;p,e,k=sample(a,b,c,i%6)
62
+ tests.append({'id':f'heldout_{i:02}','prompt':p,'expected':e,'kind':k})
63
+ train=[]
64
+ for i in range(900):
65
+ a,b,c=rng.randint(100,500),rng.randint(10,80),rng.randint(1,29)
66
+ p,e,k=sample(a,b,c,i%6)
67
+ answer=json.dumps(e) if k=='json' else 'def solve(x):\n return '+e if k=='code' else str(e)
68
+ train.append({'messages':[{'role':'user','content':p},{'role':'assistant','content':answer}]})
69
+ retained=[json.loads(s) for s in Path(hf_hub_download(REPO,'train.jsonl',revision=REV)).read_text().splitlines()]
70
+ rng.shuffle(retained);train+=retained[:900];rng.shuffle(train)
71
+ assert not {x['prompt'] for x in tests} & {m['content'] for row in train for m in row['messages'] if m['role']=='user'}
72
+ (OUT/'heldout_cases.json').write_text(json.dumps(tests,indent=2));(OUT/'training.jsonl').write_text('\n'.join(json.dumps(x) for x in train)+'\n')
73
+ (OUT/'train_v3.py').write_text(Path(__file__).read_text())
74
+ q=BitsAndBytesConfig(load_in_4bit=True,bnb_4bit_quant_type='nf4',bnb_4bit_use_double_quant=True,bnb_4bit_compute_dtype=torch.bfloat16)
75
+ model=AutoModelForCausalLM.from_pretrained(REPO,revision=REV,quantization_config=q,device_map={'':'cuda:0'},torch_dtype=torch.bfloat16,attn_implementation='sdpa')
76
+
77
+ @torch.inference_mode()
78
+ def evaluate(label):
79
+ model.eval(); rows=[]
80
+ for i,case in enumerate(tests):
81
+ inputs=tok.apply_chat_template([{'role':'system','content':settings['system']},{'role':'user','content':case['prompt']}],tokenize=True,add_generation_prompt=True,return_dict=True,return_tensors='pt').to('cuda')
82
+ out=model.generate(**inputs,max_new_tokens=96,do_sample=False,pad_token_id=128004,eos_token_id=128009)
83
+ text=tok.decode(out[0,inputs['input_ids'].shape[1]:],skip_special_tokens=True)
84
+ rows.append({**case,'output':text,'pass':grade(case,text),'tokens':int(out.shape[1]-inputs['input_ids'].shape[1])})
85
+ print('EVAL',label,i,rows[-1]['pass'],flush=True)
86
+ (OUT/(label+'_outputs.json')).write_text(json.dumps(rows,indent=2))
87
+ return rows
88
+
89
+ try:
90
+ baseline=evaluate('v2_quantized'); report['baseline_passed']=sum(r['pass'] for r in baseline);publish()
91
+ model=prepare_model_for_kbit_training(model,use_gradient_checkpointing=True)
92
+ model=get_peft_model(model,LoraConfig(r=16,lora_alpha=32,lora_dropout=0.05,target_modules=['q_proj','k_proj','v_proj','o_proj','gate_proj','up_proj','down_proj'],task_type='CAUSAL_LM'))
93
+ model.config.use_cache=False;encoded=[]
94
+ for row in train:
95
+ messages=row['messages']; prefix=tok.apply_chat_template(messages[:-1],tokenize=True,add_generation_prompt=True)
96
+ full=tok.apply_chat_template(messages,tokenize=True,add_generation_prompt=False)
97
+ if len(full)<=512 and full[:len(prefix)]==prefix:encoded.append({'input_ids':full,'labels':[-100]*len(prefix)+full[len(prefix):]})
98
+ def collate(rows):
99
+ length=max(len(r['input_ids']) for r in rows)
100
+ return {k:torch.tensor([r[k]+[pad]*(length-len(r[k])) for r in rows]) for k,pad in [('input_ids',128004),('labels',-100)]}|{'attention_mask':torch.tensor([[1]*len(r['input_ids'])+[0]*(length-len(r['input_ids'])) for r in rows])}
101
+ class Budget(TrainerCallback):
102
+ def on_step_end(self,args,state,control,**kwargs):
103
+ if time.monotonic()-started>2400:control.should_training_stop=True
104
+ return control
105
+ args=TrainingArguments(output_dir='/tmp/v3-checkpoints',max_steps=180,per_device_train_batch_size=1,gradient_accumulation_steps=4,learning_rate=1e-5,warmup_ratio=0.05,bf16=True,gradient_checkpointing=True,gradient_checkpointing_kwargs={'use_reentrant':False},optim='paged_adamw_8bit',save_strategy='no',logging_steps=10,report_to='none',remove_unused_columns=False,seed=20261113)
106
+ trainer=Trainer(model=model,args=args,train_dataset=encoded,data_collator=collate,callbacks=[Budget()])
107
+ report.update(status='training',training_examples=len(encoded),trainable_parameters=sum(p.numel() for p in model.parameters() if p.requires_grad));publish()
108
+ result=trainer.train();report['training']=result.metrics;report['optimizer_steps']=trainer.state.global_step
109
+ adapter=OUT/'adapter';model.save_pretrained(adapter,safe_serialization=True);tok.save_pretrained(adapter)
110
+ model.config.use_cache=True;model.gradient_checkpointing_disable();candidate=evaluate('v3_quantized')
111
+ report['candidate_passed']=sum(r['pass'] for r in candidate);report['status']='trained';publish()
112
+ del trainer,model;gc.collect();torch.cuda.empty_cache()
113
+ model=AutoModelForCausalLM.from_pretrained(REPO,revision=REV,torch_dtype=torch.bfloat16,device_map={'':'cuda:0'},attn_implementation='sdpa')
114
+ model=PeftModel.from_pretrained(model,adapter).merge_and_unload(safe_merge=True)
115
+ model.config.astheral_version='3';merged=OUT/'merged';model.save_pretrained(merged,safe_serialization=True,max_shard_size='4GB');tok.save_pretrained(merged)
116
+ del model;gc.collect();torch.cuda.empty_cache()
117
+ model=AutoModelForCausalLM.from_pretrained(merged,torch_dtype=torch.bfloat16,device_map={'':'cuda:0'},attn_implementation='sdpa')
118
+ reloaded=evaluate('v3_reloaded');report['reloaded_passed']=sum(r['pass'] for r in reloaded)
119
+ groups={kind:{'baseline':sum(r['pass'] for r in baseline if r['kind']==kind),'candidate':sum(r['pass'] for r in reloaded if r['kind']==kind),'total':sum(r['kind']==kind for r in tests)} for kind in {x['kind'] for x in tests}}
120
+ report['groups']=groups;report['quality_gate_passed']=report['reloaded_passed']>report['baseline_passed'] and all(v['candidate']>=v['baseline'] for v in groups.values())
121
+ report['status']='complete';report['peak_gpu_bytes']=torch.cuda.max_memory_allocated();publish()
122
+ # Save the actual trained weights to the isolated candidate branch even on a failed gate.
123
+ api.upload_folder(repo_id=REPO,folder_path=merged,revision=branch,commit_message='Save reloaded V3 candidate weights; see measured quality gate in v3-results')
124
+ if report['quality_gate_passed']:
125
+ api.create_tag(REPO,tag='v3',revision=api.model_info(REPO,revision=branch).sha,exist_ok=True)
126
+ print('V3_COMPLETE',json.dumps(report),flush=True)
127
+ except Exception:
128
+ report['status']='failed';report['error']=traceback.format_exc();publish();raise
v3-results/training.jsonl ADDED
The diff for this file is too large to render. See raw diff
 
v3-results/v2_quantized_outputs.json ADDED
@@ -0,0 +1,572 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "id": "heldout_00",
4
+ "prompt": "Return only the integer: 701 + 91 - 31.",
5
+ "expected": "761",
6
+ "kind": "integer",
7
+ "output": "701 + 91 - 31 is 761.",
8
+ "pass": false,
9
+ "tokens": 12
10
+ },
11
+ {
12
+ "id": "heldout_01",
13
+ "prompt": "A crate has 704 red and 92 blue beads. Remove 32 red beads. Return only the total beads remaining.",
14
+ "expected": "764",
15
+ "kind": "integer",
16
+ "output": "Remove 32 from 704 to find 672 red beads. 672 + 92 = 764 beads remain.",
17
+ "pass": false,
18
+ "tokens": 25
19
+ },
20
+ {
21
+ "id": "heldout_02",
22
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_707, count=93.",
23
+ "expected": {
24
+ "item": "part_707",
25
+ "count": 93
26
+ },
27
+ "kind": "json",
28
+ "output": "{\"item\": \"part_707\", \"count\": 93}",
29
+ "pass": true,
30
+ "tokens": 15
31
+ },
32
+ {
33
+ "id": "heldout_03",
34
+ "prompt": "Return only one CSV value row, no header: name=Item710; quantity=94; code=K34.",
35
+ "expected": "Item710,94,K34",
36
+ "kind": "exact",
37
+ "output": "Item710,94,K34",
38
+ "pass": true,
39
+ "tokens": 7
40
+ },
41
+ {
42
+ "id": "heldout_04",
43
+ "prompt": "Copy exactly, no other text: Key_713-B95_C35",
44
+ "expected": "Key_713-B95_C35",
45
+ "kind": "exact",
46
+ "output": "Key_713-B95_C35",
47
+ "pass": true,
48
+ "tokens": 8
49
+ },
50
+ {
51
+ "id": "heldout_05",
52
+ "prompt": "Write only a Python function named solve(x) that returns x * 96 + 36.",
53
+ "expected": "x * 96 + 36",
54
+ "kind": "code",
55
+ "output": "```python\ndef solve(x):\n return x * 96 + 36\n```",
56
+ "pass": true,
57
+ "tokens": 19
58
+ },
59
+ {
60
+ "id": "heldout_06",
61
+ "prompt": "Return only the integer: 719 + 97 - 37.",
62
+ "expected": "779",
63
+ "kind": "integer",
64
+ "output": "719 + 97 - 37 = 779",
65
+ "pass": false,
66
+ "tokens": 11
67
+ },
68
+ {
69
+ "id": "heldout_07",
70
+ "prompt": "A crate has 722 red and 98 blue beads. Remove 38 red beads. Return only the total beads remaining.",
71
+ "expected": "782",
72
+ "kind": "integer",
73
+ "output": "Remove 38 red beads from 722, leaving 684. The 98 blue beads are unchanged, for a total of 684 + 98 = 782 beads.",
74
+ "pass": false,
75
+ "tokens": 36
76
+ },
77
+ {
78
+ "id": "heldout_08",
79
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_725, count=99.",
80
+ "expected": {
81
+ "item": "part_725",
82
+ "count": 99
83
+ },
84
+ "kind": "json",
85
+ "output": "{\"item\": \"part_725\", \"count\": 99}",
86
+ "pass": true,
87
+ "tokens": 15
88
+ },
89
+ {
90
+ "id": "heldout_09",
91
+ "prompt": "Return only one CSV value row, no header: name=Item728; quantity=100; code=K40.",
92
+ "expected": "Item728,100,K40",
93
+ "kind": "exact",
94
+ "output": "Item728,100,K40",
95
+ "pass": true,
96
+ "tokens": 7
97
+ },
98
+ {
99
+ "id": "heldout_10",
100
+ "prompt": "Copy exactly, no other text: Key_731-B101_C41",
101
+ "expected": "Key_731-B101_C41",
102
+ "kind": "exact",
103
+ "output": "Key_731-B101_C41",
104
+ "pass": true,
105
+ "tokens": 8
106
+ },
107
+ {
108
+ "id": "heldout_11",
109
+ "prompt": "Write only a Python function named solve(x) that returns x * 102 + 42.",
110
+ "expected": "x * 102 + 42",
111
+ "kind": "code",
112
+ "output": "```python\ndef solve(x):\n return x * 102 + 42\n```",
113
+ "pass": true,
114
+ "tokens": 19
115
+ },
116
+ {
117
+ "id": "heldout_12",
118
+ "prompt": "Return only the integer: 737 + 103 - 43.",
119
+ "expected": "797",
120
+ "kind": "integer",
121
+ "output": "737 + 103 - 43 is 797.",
122
+ "pass": false,
123
+ "tokens": 12
124
+ },
125
+ {
126
+ "id": "heldout_13",
127
+ "prompt": "A crate has 740 red and 104 blue beads. Remove 44 red beads. Return only the total beads remaining.",
128
+ "expected": "800",
129
+ "kind": "integer",
130
+ "output": "Remove 44 red beads from 740, leaving 696. The 104 blue beads are unchanged, for 696 + 104 = 800 beads remaining.",
131
+ "pass": false,
132
+ "tokens": 34
133
+ },
134
+ {
135
+ "id": "heldout_14",
136
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_743, count=105.",
137
+ "expected": {
138
+ "item": "part_743",
139
+ "count": 105
140
+ },
141
+ "kind": "json",
142
+ "output": "{\"item\": \"part_743\", \"count\": 105}",
143
+ "pass": true,
144
+ "tokens": 15
145
+ },
146
+ {
147
+ "id": "heldout_15",
148
+ "prompt": "Return only one CSV value row, no header: name=Item746; quantity=106; code=K46.",
149
+ "expected": "Item746,106,K46",
150
+ "kind": "exact",
151
+ "output": "Item746,106,K46",
152
+ "pass": true,
153
+ "tokens": 7
154
+ },
155
+ {
156
+ "id": "heldout_16",
157
+ "prompt": "Copy exactly, no other text: Key_749-B107_C47",
158
+ "expected": "Key_749-B107_C47",
159
+ "kind": "exact",
160
+ "output": "Key_749-B107_C47",
161
+ "pass": true,
162
+ "tokens": 8
163
+ },
164
+ {
165
+ "id": "heldout_17",
166
+ "prompt": "Write only a Python function named solve(x) that returns x * 108 + 31.",
167
+ "expected": "x * 108 + 31",
168
+ "kind": "code",
169
+ "output": "```python\ndef solve(x):\n return x * 108 + 31\n```",
170
+ "pass": true,
171
+ "tokens": 19
172
+ },
173
+ {
174
+ "id": "heldout_18",
175
+ "prompt": "Return only the integer: 755 + 109 - 32.",
176
+ "expected": "832",
177
+ "kind": "integer",
178
+ "output": "755 + 109 - 32 is 832.",
179
+ "pass": false,
180
+ "tokens": 12
181
+ },
182
+ {
183
+ "id": "heldout_19",
184
+ "prompt": "A crate has 758 red and 110 blue beads. Remove 33 red beads. Return only the total beads remaining.",
185
+ "expected": "835",
186
+ "kind": "integer",
187
+ "output": "Remove 33 red beads from 758, leaving 725. The 110 blue beads are unchanged, for 725 + 110 = 835 beads remaining.",
188
+ "pass": false,
189
+ "tokens": 34
190
+ },
191
+ {
192
+ "id": "heldout_20",
193
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_761, count=111.",
194
+ "expected": {
195
+ "item": "part_761",
196
+ "count": 111
197
+ },
198
+ "kind": "json",
199
+ "output": "{\"item\": \"part_761\", \"count\": 111}",
200
+ "pass": true,
201
+ "tokens": 15
202
+ },
203
+ {
204
+ "id": "heldout_21",
205
+ "prompt": "Return only one CSV value row, no header: name=Item764; quantity=112; code=K35.",
206
+ "expected": "Item764,112,K35",
207
+ "kind": "exact",
208
+ "output": "Item764,112,K35",
209
+ "pass": true,
210
+ "tokens": 7
211
+ },
212
+ {
213
+ "id": "heldout_22",
214
+ "prompt": "Copy exactly, no other text: Key_767-B113_C36",
215
+ "expected": "Key_767-B113_C36",
216
+ "kind": "exact",
217
+ "output": "Key_767-B113_C36",
218
+ "pass": true,
219
+ "tokens": 8
220
+ },
221
+ {
222
+ "id": "heldout_23",
223
+ "prompt": "Write only a Python function named solve(x) that returns x * 114 + 37.",
224
+ "expected": "x * 114 + 37",
225
+ "kind": "code",
226
+ "output": "```python\ndef solve(x):\n return x * 114 + 37\n```",
227
+ "pass": true,
228
+ "tokens": 19
229
+ },
230
+ {
231
+ "id": "heldout_24",
232
+ "prompt": "Return only the integer: 773 + 115 - 38.",
233
+ "expected": "850",
234
+ "kind": "integer",
235
+ "output": "750",
236
+ "pass": false,
237
+ "tokens": 2
238
+ },
239
+ {
240
+ "id": "heldout_25",
241
+ "prompt": "A crate has 776 red and 116 blue beads. Remove 39 red beads. Return only the total beads remaining.",
242
+ "expected": "853",
243
+ "kind": "integer",
244
+ "output": "Remove 39 red beads from 776 to leave 737. The 116 blue beads are unchanged, for 737 + 116 = 853 beads remaining.",
245
+ "pass": false,
246
+ "tokens": 34
247
+ },
248
+ {
249
+ "id": "heldout_26",
250
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_779, count=117.",
251
+ "expected": {
252
+ "item": "part_779",
253
+ "count": 117
254
+ },
255
+ "kind": "json",
256
+ "output": "{\"item\": \"part_779\", \"count\": 117}",
257
+ "pass": true,
258
+ "tokens": 15
259
+ },
260
+ {
261
+ "id": "heldout_27",
262
+ "prompt": "Return only one CSV value row, no header: name=Item782; quantity=118; code=K41.",
263
+ "expected": "Item782,118,K41",
264
+ "kind": "exact",
265
+ "output": "Item782,118,K41",
266
+ "pass": true,
267
+ "tokens": 7
268
+ },
269
+ {
270
+ "id": "heldout_28",
271
+ "prompt": "Copy exactly, no other text: Key_785-B119_C42",
272
+ "expected": "Key_785-B119_C42",
273
+ "kind": "exact",
274
+ "output": "Key_785-B119_C42",
275
+ "pass": true,
276
+ "tokens": 8
277
+ },
278
+ {
279
+ "id": "heldout_29",
280
+ "prompt": "Write only a Python function named solve(x) that returns x * 120 + 43.",
281
+ "expected": "x * 120 + 43",
282
+ "kind": "code",
283
+ "output": "```python\ndef solve(x):\n return x * 120 + 43\n```",
284
+ "pass": true,
285
+ "tokens": 19
286
+ },
287
+ {
288
+ "id": "heldout_30",
289
+ "prompt": "Return only the integer: 791 + 121 - 44.",
290
+ "expected": "868",
291
+ "kind": "integer",
292
+ "output": "791 + 121 - 44 is 868.",
293
+ "pass": false,
294
+ "tokens": 12
295
+ },
296
+ {
297
+ "id": "heldout_31",
298
+ "prompt": "A crate has 794 red and 122 blue beads. Remove 45 red beads. Return only the total beads remaining.",
299
+ "expected": "871",
300
+ "kind": "integer",
301
+ "output": "Remove 45 red beads from 794, leaving 749, and keep the 122 blue beads, leaving 749 + 122 = 871 beads.",
302
+ "pass": false,
303
+ "tokens": 33
304
+ },
305
+ {
306
+ "id": "heldout_32",
307
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_797, count=123.",
308
+ "expected": {
309
+ "item": "part_797",
310
+ "count": 123
311
+ },
312
+ "kind": "json",
313
+ "output": "{\"item\":\"part_797\",\"count\":123}",
314
+ "pass": true,
315
+ "tokens": 12
316
+ },
317
+ {
318
+ "id": "heldout_33",
319
+ "prompt": "Return only one CSV value row, no header: name=Item800; quantity=124; code=K47.",
320
+ "expected": "Item800,124,K47",
321
+ "kind": "exact",
322
+ "output": "Item800,124,K47",
323
+ "pass": true,
324
+ "tokens": 7
325
+ },
326
+ {
327
+ "id": "heldout_34",
328
+ "prompt": "Copy exactly, no other text: Key_803-B125_C31",
329
+ "expected": "Key_803-B125_C31",
330
+ "kind": "exact",
331
+ "output": "Key_803-B125_C31",
332
+ "pass": true,
333
+ "tokens": 8
334
+ },
335
+ {
336
+ "id": "heldout_35",
337
+ "prompt": "Write only a Python function named solve(x) that returns x * 126 + 32.",
338
+ "expected": "x * 126 + 32",
339
+ "kind": "code",
340
+ "output": "```python\ndef solve(x):\n return x * 126 + 32\n```",
341
+ "pass": true,
342
+ "tokens": 19
343
+ },
344
+ {
345
+ "id": "heldout_36",
346
+ "prompt": "Return only the integer: 809 + 127 - 33.",
347
+ "expected": "903",
348
+ "kind": "integer",
349
+ "output": "809 + 127 - 33 is 903.",
350
+ "pass": false,
351
+ "tokens": 12
352
+ },
353
+ {
354
+ "id": "heldout_37",
355
+ "prompt": "A crate has 812 red and 128 blue beads. Remove 34 red beads. Return only the total beads remaining.",
356
+ "expected": "906",
357
+ "kind": "integer",
358
+ "output": "Remove 34 red beads from 812, leaving 778, and keep 128 blue beads, for 778 + 128 = 906 beads.",
359
+ "pass": false,
360
+ "tokens": 32
361
+ },
362
+ {
363
+ "id": "heldout_38",
364
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_815, count=129.",
365
+ "expected": {
366
+ "item": "part_815",
367
+ "count": 129
368
+ },
369
+ "kind": "json",
370
+ "output": "{\"item\":\"part_815\",\"count\":129}",
371
+ "pass": true,
372
+ "tokens": 12
373
+ },
374
+ {
375
+ "id": "heldout_39",
376
+ "prompt": "Return only one CSV value row, no header: name=Item818; quantity=130; code=K36.",
377
+ "expected": "Item818,130,K36",
378
+ "kind": "exact",
379
+ "output": "Item818,130,K36",
380
+ "pass": true,
381
+ "tokens": 7
382
+ },
383
+ {
384
+ "id": "heldout_40",
385
+ "prompt": "Copy exactly, no other text: Key_821-B131_C37",
386
+ "expected": "Key_821-B131_C37",
387
+ "kind": "exact",
388
+ "output": "Key_821-B131_C37",
389
+ "pass": true,
390
+ "tokens": 8
391
+ },
392
+ {
393
+ "id": "heldout_41",
394
+ "prompt": "Write only a Python function named solve(x) that returns x * 132 + 38.",
395
+ "expected": "x * 132 + 38",
396
+ "kind": "code",
397
+ "output": "```python\ndef solve(x):\n return x * 132 + 38\n```",
398
+ "pass": true,
399
+ "tokens": 19
400
+ },
401
+ {
402
+ "id": "heldout_42",
403
+ "prompt": "Return only the integer: 827 + 133 - 39.",
404
+ "expected": "921",
405
+ "kind": "integer",
406
+ "output": "827 + 133 - 39 = 921",
407
+ "pass": false,
408
+ "tokens": 11
409
+ },
410
+ {
411
+ "id": "heldout_43",
412
+ "prompt": "A crate has 830 red and 134 blue beads. Remove 40 red beads. Return only the total beads remaining.",
413
+ "expected": "924",
414
+ "kind": "integer",
415
+ "output": "Remove 40 red beads from 830, leaving 790. The 134 blue beads are unchanged, for 790 + 134 = 924 beads remaining.",
416
+ "pass": false,
417
+ "tokens": 34
418
+ },
419
+ {
420
+ "id": "heldout_44",
421
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_833, count=135.",
422
+ "expected": {
423
+ "item": "part_833",
424
+ "count": 135
425
+ },
426
+ "kind": "json",
427
+ "output": "{\"item\": \"part_833\", \"count\": 135}",
428
+ "pass": true,
429
+ "tokens": 15
430
+ },
431
+ {
432
+ "id": "heldout_45",
433
+ "prompt": "Return only one CSV value row, no header: name=Item836; quantity=136; code=K42.",
434
+ "expected": "Item836,136,K42",
435
+ "kind": "exact",
436
+ "output": "Item836,136,K42",
437
+ "pass": true,
438
+ "tokens": 7
439
+ },
440
+ {
441
+ "id": "heldout_46",
442
+ "prompt": "Copy exactly, no other text: Key_839-B137_C43",
443
+ "expected": "Key_839-B137_C43",
444
+ "kind": "exact",
445
+ "output": "Key_839-B137_C43",
446
+ "pass": true,
447
+ "tokens": 8
448
+ },
449
+ {
450
+ "id": "heldout_47",
451
+ "prompt": "Write only a Python function named solve(x) that returns x * 138 + 44.",
452
+ "expected": "x * 138 + 44",
453
+ "kind": "code",
454
+ "output": "```python\ndef solve(x):\n return x * 138 + 44\n```",
455
+ "pass": true,
456
+ "tokens": 19
457
+ },
458
+ {
459
+ "id": "heldout_48",
460
+ "prompt": "Return only the integer: 845 + 139 - 45.",
461
+ "expected": "939",
462
+ "kind": "integer",
463
+ "output": "845 + 139 - 45 = 939",
464
+ "pass": false,
465
+ "tokens": 11
466
+ },
467
+ {
468
+ "id": "heldout_49",
469
+ "prompt": "A crate has 848 red and 140 blue beads. Remove 46 red beads. Return only the total beads remaining.",
470
+ "expected": "942",
471
+ "kind": "integer",
472
+ "output": "Remove 46 red beads from 848, leaving 802. The 140 blue beads are unchanged, for 802 + 140 = 942 beads remaining.",
473
+ "pass": false,
474
+ "tokens": 34
475
+ },
476
+ {
477
+ "id": "heldout_50",
478
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_851, count=141.",
479
+ "expected": {
480
+ "item": "part_851",
481
+ "count": 141
482
+ },
483
+ "kind": "json",
484
+ "output": "{\"item\": \"part_851\", \"count\": 141}",
485
+ "pass": true,
486
+ "tokens": 15
487
+ },
488
+ {
489
+ "id": "heldout_51",
490
+ "prompt": "Return only one CSV value row, no header: name=Item854; quantity=142; code=K31.",
491
+ "expected": "Item854,142,K31",
492
+ "kind": "exact",
493
+ "output": "Item854,142,K31",
494
+ "pass": true,
495
+ "tokens": 7
496
+ },
497
+ {
498
+ "id": "heldout_52",
499
+ "prompt": "Copy exactly, no other text: Key_857-B143_C32",
500
+ "expected": "Key_857-B143_C32",
501
+ "kind": "exact",
502
+ "output": "Key_857-B143_C32",
503
+ "pass": true,
504
+ "tokens": 8
505
+ },
506
+ {
507
+ "id": "heldout_53",
508
+ "prompt": "Write only a Python function named solve(x) that returns x * 144 + 33.",
509
+ "expected": "x * 144 + 33",
510
+ "kind": "code",
511
+ "output": "```python\ndef solve(x):\n return x * 144 + 33\n```",
512
+ "pass": true,
513
+ "tokens": 19
514
+ },
515
+ {
516
+ "id": "heldout_54",
517
+ "prompt": "Return only the integer: 863 + 145 - 34.",
518
+ "expected": "974",
519
+ "kind": "integer",
520
+ "output": "863 + 145 - 34 is 974.",
521
+ "pass": false,
522
+ "tokens": 12
523
+ },
524
+ {
525
+ "id": "heldout_55",
526
+ "prompt": "A crate has 866 red and 146 blue beads. Remove 35 red beads. Return only the total beads remaining.",
527
+ "expected": "977",
528
+ "kind": "integer",
529
+ "output": "Remove 35 red beads from 866 to leave 831. The 146 blue beads are unchanged, for 831 + 146 = 977 beads remaining.",
530
+ "pass": false,
531
+ "tokens": 34
532
+ },
533
+ {
534
+ "id": "heldout_56",
535
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_869, count=147.",
536
+ "expected": {
537
+ "item": "part_869",
538
+ "count": 147
539
+ },
540
+ "kind": "json",
541
+ "output": "{\"item\": \"part_869\", \"count\": 147}",
542
+ "pass": true,
543
+ "tokens": 15
544
+ },
545
+ {
546
+ "id": "heldout_57",
547
+ "prompt": "Return only one CSV value row, no header: name=Item872; quantity=148; code=K37.",
548
+ "expected": "Item872,148,K37",
549
+ "kind": "exact",
550
+ "output": "Item872,148,K37",
551
+ "pass": true,
552
+ "tokens": 7
553
+ },
554
+ {
555
+ "id": "heldout_58",
556
+ "prompt": "Copy exactly, no other text: Key_875-B149_C38",
557
+ "expected": "Key_875-B149_C38",
558
+ "kind": "exact",
559
+ "output": "Key_875-B149_C38",
560
+ "pass": true,
561
+ "tokens": 8
562
+ },
563
+ {
564
+ "id": "heldout_59",
565
+ "prompt": "Write only a Python function named solve(x) that returns x * 150 + 39.",
566
+ "expected": "x * 150 + 39",
567
+ "kind": "code",
568
+ "output": "```python\ndef solve(x):\n return x * 150 + 39\n```",
569
+ "pass": true,
570
+ "tokens": 19
571
+ }
572
+ ]
v3-results/v3_quantized_outputs.json ADDED
@@ -0,0 +1,572 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "id": "heldout_00",
4
+ "prompt": "Return only the integer: 701 + 91 - 31.",
5
+ "expected": "761",
6
+ "kind": "integer",
7
+ "output": "761",
8
+ "pass": true,
9
+ "tokens": 2
10
+ },
11
+ {
12
+ "id": "heldout_01",
13
+ "prompt": "A crate has 704 red and 92 blue beads. Remove 32 red beads. Return only the total beads remaining.",
14
+ "expected": "764",
15
+ "kind": "integer",
16
+ "output": "676",
17
+ "pass": false,
18
+ "tokens": 2
19
+ },
20
+ {
21
+ "id": "heldout_02",
22
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_707, count=93.",
23
+ "expected": {
24
+ "item": "part_707",
25
+ "count": 93
26
+ },
27
+ "kind": "json",
28
+ "output": "{\"item\": \"part_707\", \"count\": 93}",
29
+ "pass": true,
30
+ "tokens": 15
31
+ },
32
+ {
33
+ "id": "heldout_03",
34
+ "prompt": "Return only one CSV value row, no header: name=Item710; quantity=94; code=K34.",
35
+ "expected": "Item710,94,K34",
36
+ "kind": "exact",
37
+ "output": "Item710,94,K34",
38
+ "pass": true,
39
+ "tokens": 7
40
+ },
41
+ {
42
+ "id": "heldout_04",
43
+ "prompt": "Copy exactly, no other text: Key_713-B95_C35",
44
+ "expected": "Key_713-B95_C35",
45
+ "kind": "exact",
46
+ "output": "Key_713-B95_C35",
47
+ "pass": true,
48
+ "tokens": 8
49
+ },
50
+ {
51
+ "id": "heldout_05",
52
+ "prompt": "Write only a Python function named solve(x) that returns x * 96 + 36.",
53
+ "expected": "x * 96 + 36",
54
+ "kind": "code",
55
+ "output": "```python\ndef solve(x):\n return x * 96 + 36\n```",
56
+ "pass": true,
57
+ "tokens": 19
58
+ },
59
+ {
60
+ "id": "heldout_06",
61
+ "prompt": "Return only the integer: 719 + 97 - 37.",
62
+ "expected": "779",
63
+ "kind": "integer",
64
+ "output": "779",
65
+ "pass": true,
66
+ "tokens": 2
67
+ },
68
+ {
69
+ "id": "heldout_07",
70
+ "prompt": "A crate has 722 red and 98 blue beads. Remove 38 red beads. Return only the total beads remaining.",
71
+ "expected": "782",
72
+ "kind": "integer",
73
+ "output": "722 - 38 = 684 red beads remain. 684 + 98 = 782 beads remain.",
74
+ "pass": false,
75
+ "tokens": 23
76
+ },
77
+ {
78
+ "id": "heldout_08",
79
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_725, count=99.",
80
+ "expected": {
81
+ "item": "part_725",
82
+ "count": 99
83
+ },
84
+ "kind": "json",
85
+ "output": "{\"item\": \"part_725\", \"count\": 99}",
86
+ "pass": true,
87
+ "tokens": 15
88
+ },
89
+ {
90
+ "id": "heldout_09",
91
+ "prompt": "Return only one CSV value row, no header: name=Item728; quantity=100; code=K40.",
92
+ "expected": "Item728,100,K40",
93
+ "kind": "exact",
94
+ "output": "Item728,100,K40",
95
+ "pass": true,
96
+ "tokens": 7
97
+ },
98
+ {
99
+ "id": "heldout_10",
100
+ "prompt": "Copy exactly, no other text: Key_731-B101_C41",
101
+ "expected": "Key_731-B101_C41",
102
+ "kind": "exact",
103
+ "output": "Key_731-B101_C41",
104
+ "pass": true,
105
+ "tokens": 8
106
+ },
107
+ {
108
+ "id": "heldout_11",
109
+ "prompt": "Write only a Python function named solve(x) that returns x * 102 + 42.",
110
+ "expected": "x * 102 + 42",
111
+ "kind": "code",
112
+ "output": "```python\ndef solve(x):\n return x * 102 + 42\n```",
113
+ "pass": true,
114
+ "tokens": 19
115
+ },
116
+ {
117
+ "id": "heldout_12",
118
+ "prompt": "Return only the integer: 737 + 103 - 43.",
119
+ "expected": "797",
120
+ "kind": "integer",
121
+ "output": "797",
122
+ "pass": true,
123
+ "tokens": 2
124
+ },
125
+ {
126
+ "id": "heldout_13",
127
+ "prompt": "A crate has 740 red and 104 blue beads. Remove 44 red beads. Return only the total beads remaining.",
128
+ "expected": "800",
129
+ "kind": "integer",
130
+ "output": "676",
131
+ "pass": false,
132
+ "tokens": 2
133
+ },
134
+ {
135
+ "id": "heldout_14",
136
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_743, count=105.",
137
+ "expected": {
138
+ "item": "part_743",
139
+ "count": 105
140
+ },
141
+ "kind": "json",
142
+ "output": "{\"item\": \"part_743\", \"count\": 105}",
143
+ "pass": true,
144
+ "tokens": 15
145
+ },
146
+ {
147
+ "id": "heldout_15",
148
+ "prompt": "Return only one CSV value row, no header: name=Item746; quantity=106; code=K46.",
149
+ "expected": "Item746,106,K46",
150
+ "kind": "exact",
151
+ "output": "Item746,106,K46",
152
+ "pass": true,
153
+ "tokens": 7
154
+ },
155
+ {
156
+ "id": "heldout_16",
157
+ "prompt": "Copy exactly, no other text: Key_749-B107_C47",
158
+ "expected": "Key_749-B107_C47",
159
+ "kind": "exact",
160
+ "output": "Key_749-B107_C47",
161
+ "pass": true,
162
+ "tokens": 8
163
+ },
164
+ {
165
+ "id": "heldout_17",
166
+ "prompt": "Write only a Python function named solve(x) that returns x * 108 + 31.",
167
+ "expected": "x * 108 + 31",
168
+ "kind": "code",
169
+ "output": "```python\ndef solve(x):\n return x * 108 + 31\n```",
170
+ "pass": true,
171
+ "tokens": 19
172
+ },
173
+ {
174
+ "id": "heldout_18",
175
+ "prompt": "Return only the integer: 755 + 109 - 32.",
176
+ "expected": "832",
177
+ "kind": "integer",
178
+ "output": "832",
179
+ "pass": true,
180
+ "tokens": 2
181
+ },
182
+ {
183
+ "id": "heldout_19",
184
+ "prompt": "A crate has 758 red and 110 blue beads. Remove 33 red beads. Return only the total beads remaining.",
185
+ "expected": "835",
186
+ "kind": "integer",
187
+ "output": "726",
188
+ "pass": false,
189
+ "tokens": 2
190
+ },
191
+ {
192
+ "id": "heldout_20",
193
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_761, count=111.",
194
+ "expected": {
195
+ "item": "part_761",
196
+ "count": 111
197
+ },
198
+ "kind": "json",
199
+ "output": "{\"item\": \"part_761\", \"count\": 111}",
200
+ "pass": true,
201
+ "tokens": 15
202
+ },
203
+ {
204
+ "id": "heldout_21",
205
+ "prompt": "Return only one CSV value row, no header: name=Item764; quantity=112; code=K35.",
206
+ "expected": "Item764,112,K35",
207
+ "kind": "exact",
208
+ "output": "Item764,112,K35",
209
+ "pass": true,
210
+ "tokens": 7
211
+ },
212
+ {
213
+ "id": "heldout_22",
214
+ "prompt": "Copy exactly, no other text: Key_767-B113_C36",
215
+ "expected": "Key_767-B113_C36",
216
+ "kind": "exact",
217
+ "output": "Key_767-B113_C36",
218
+ "pass": true,
219
+ "tokens": 8
220
+ },
221
+ {
222
+ "id": "heldout_23",
223
+ "prompt": "Write only a Python function named solve(x) that returns x * 114 + 37.",
224
+ "expected": "x * 114 + 37",
225
+ "kind": "code",
226
+ "output": "```python\ndef solve(x):\n return x * 114 + 37\n```",
227
+ "pass": true,
228
+ "tokens": 19
229
+ },
230
+ {
231
+ "id": "heldout_24",
232
+ "prompt": "Return only the integer: 773 + 115 - 38.",
233
+ "expected": "850",
234
+ "kind": "integer",
235
+ "output": "850",
236
+ "pass": true,
237
+ "tokens": 2
238
+ },
239
+ {
240
+ "id": "heldout_25",
241
+ "prompt": "A crate has 776 red and 116 blue beads. Remove 39 red beads. Return only the total beads remaining.",
242
+ "expected": "853",
243
+ "kind": "integer",
244
+ "output": "726",
245
+ "pass": false,
246
+ "tokens": 2
247
+ },
248
+ {
249
+ "id": "heldout_26",
250
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_779, count=117.",
251
+ "expected": {
252
+ "item": "part_779",
253
+ "count": 117
254
+ },
255
+ "kind": "json",
256
+ "output": "{\"item\": \"part_779\", \"count\": 117}",
257
+ "pass": true,
258
+ "tokens": 15
259
+ },
260
+ {
261
+ "id": "heldout_27",
262
+ "prompt": "Return only one CSV value row, no header: name=Item782; quantity=118; code=K41.",
263
+ "expected": "Item782,118,K41",
264
+ "kind": "exact",
265
+ "output": "Item782,118,K41",
266
+ "pass": true,
267
+ "tokens": 7
268
+ },
269
+ {
270
+ "id": "heldout_28",
271
+ "prompt": "Copy exactly, no other text: Key_785-B119_C42",
272
+ "expected": "Key_785-B119_C42",
273
+ "kind": "exact",
274
+ "output": "Key_785-B119_C42",
275
+ "pass": true,
276
+ "tokens": 8
277
+ },
278
+ {
279
+ "id": "heldout_29",
280
+ "prompt": "Write only a Python function named solve(x) that returns x * 120 + 43.",
281
+ "expected": "x * 120 + 43",
282
+ "kind": "code",
283
+ "output": "```python\ndef solve(x):\n return x * 120 + 43\n```",
284
+ "pass": true,
285
+ "tokens": 19
286
+ },
287
+ {
288
+ "id": "heldout_30",
289
+ "prompt": "Return only the integer: 791 + 121 - 44.",
290
+ "expected": "868",
291
+ "kind": "integer",
292
+ "output": "868",
293
+ "pass": true,
294
+ "tokens": 2
295
+ },
296
+ {
297
+ "id": "heldout_31",
298
+ "prompt": "A crate has 794 red and 122 blue beads. Remove 45 red beads. Return only the total beads remaining.",
299
+ "expected": "871",
300
+ "kind": "integer",
301
+ "output": "726",
302
+ "pass": false,
303
+ "tokens": 2
304
+ },
305
+ {
306
+ "id": "heldout_32",
307
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_797, count=123.",
308
+ "expected": {
309
+ "item": "part_797",
310
+ "count": 123
311
+ },
312
+ "kind": "json",
313
+ "output": "{\"item\": \"part_797\", \"count\": 123}",
314
+ "pass": true,
315
+ "tokens": 15
316
+ },
317
+ {
318
+ "id": "heldout_33",
319
+ "prompt": "Return only one CSV value row, no header: name=Item800; quantity=124; code=K47.",
320
+ "expected": "Item800,124,K47",
321
+ "kind": "exact",
322
+ "output": "Item800,124,K47",
323
+ "pass": true,
324
+ "tokens": 7
325
+ },
326
+ {
327
+ "id": "heldout_34",
328
+ "prompt": "Copy exactly, no other text: Key_803-B125_C31",
329
+ "expected": "Key_803-B125_C31",
330
+ "kind": "exact",
331
+ "output": "Key_803-B125_C31",
332
+ "pass": true,
333
+ "tokens": 8
334
+ },
335
+ {
336
+ "id": "heldout_35",
337
+ "prompt": "Write only a Python function named solve(x) that returns x * 126 + 32.",
338
+ "expected": "x * 126 + 32",
339
+ "kind": "code",
340
+ "output": "```python\ndef solve(x):\n return x * 126 + 32\n```",
341
+ "pass": true,
342
+ "tokens": 19
343
+ },
344
+ {
345
+ "id": "heldout_36",
346
+ "prompt": "Return only the integer: 809 + 127 - 33.",
347
+ "expected": "903",
348
+ "kind": "integer",
349
+ "output": "903",
350
+ "pass": true,
351
+ "tokens": 2
352
+ },
353
+ {
354
+ "id": "heldout_37",
355
+ "prompt": "A crate has 812 red and 128 blue beads. Remove 34 red beads. Return only the total beads remaining.",
356
+ "expected": "906",
357
+ "kind": "integer",
358
+ "output": "812 - 34 = 778\nAnswer: 778",
359
+ "pass": false,
360
+ "tokens": 13
361
+ },
362
+ {
363
+ "id": "heldout_38",
364
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_815, count=129.",
365
+ "expected": {
366
+ "item": "part_815",
367
+ "count": 129
368
+ },
369
+ "kind": "json",
370
+ "output": "{\"item\": \"part_815\", \"count\": 129}",
371
+ "pass": true,
372
+ "tokens": 15
373
+ },
374
+ {
375
+ "id": "heldout_39",
376
+ "prompt": "Return only one CSV value row, no header: name=Item818; quantity=130; code=K36.",
377
+ "expected": "Item818,130,K36",
378
+ "kind": "exact",
379
+ "output": "Item818,130,K36",
380
+ "pass": true,
381
+ "tokens": 7
382
+ },
383
+ {
384
+ "id": "heldout_40",
385
+ "prompt": "Copy exactly, no other text: Key_821-B131_C37",
386
+ "expected": "Key_821-B131_C37",
387
+ "kind": "exact",
388
+ "output": "Key_821-B131_C37",
389
+ "pass": true,
390
+ "tokens": 8
391
+ },
392
+ {
393
+ "id": "heldout_41",
394
+ "prompt": "Write only a Python function named solve(x) that returns x * 132 + 38.",
395
+ "expected": "x * 132 + 38",
396
+ "kind": "code",
397
+ "output": "```python\ndef solve(x):\n return x * 132 + 38\n```",
398
+ "pass": true,
399
+ "tokens": 19
400
+ },
401
+ {
402
+ "id": "heldout_42",
403
+ "prompt": "Return only the integer: 827 + 133 - 39.",
404
+ "expected": "921",
405
+ "kind": "integer",
406
+ "output": "921",
407
+ "pass": true,
408
+ "tokens": 2
409
+ },
410
+ {
411
+ "id": "heldout_43",
412
+ "prompt": "A crate has 830 red and 134 blue beads. Remove 40 red beads. Return only the total beads remaining.",
413
+ "expected": "924",
414
+ "kind": "integer",
415
+ "output": "830 - 40 = 790\nAnswer: 790",
416
+ "pass": false,
417
+ "tokens": 13
418
+ },
419
+ {
420
+ "id": "heldout_44",
421
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_833, count=135.",
422
+ "expected": {
423
+ "item": "part_833",
424
+ "count": 135
425
+ },
426
+ "kind": "json",
427
+ "output": "{\"item\": \"part_833\", \"count\": 135}",
428
+ "pass": true,
429
+ "tokens": 15
430
+ },
431
+ {
432
+ "id": "heldout_45",
433
+ "prompt": "Return only one CSV value row, no header: name=Item836; quantity=136; code=K42.",
434
+ "expected": "Item836,136,K42",
435
+ "kind": "exact",
436
+ "output": "Item836,136,K42",
437
+ "pass": true,
438
+ "tokens": 7
439
+ },
440
+ {
441
+ "id": "heldout_46",
442
+ "prompt": "Copy exactly, no other text: Key_839-B137_C43",
443
+ "expected": "Key_839-B137_C43",
444
+ "kind": "exact",
445
+ "output": "Key_839-B137_C43",
446
+ "pass": true,
447
+ "tokens": 8
448
+ },
449
+ {
450
+ "id": "heldout_47",
451
+ "prompt": "Write only a Python function named solve(x) that returns x * 138 + 44.",
452
+ "expected": "x * 138 + 44",
453
+ "kind": "code",
454
+ "output": "```python\ndef solve(x):\n return x * 138 + 44\n```",
455
+ "pass": true,
456
+ "tokens": 19
457
+ },
458
+ {
459
+ "id": "heldout_48",
460
+ "prompt": "Return only the integer: 845 + 139 - 45.",
461
+ "expected": "939",
462
+ "kind": "integer",
463
+ "output": "839",
464
+ "pass": false,
465
+ "tokens": 2
466
+ },
467
+ {
468
+ "id": "heldout_49",
469
+ "prompt": "A crate has 848 red and 140 blue beads. Remove 46 red beads. Return only the total beads remaining.",
470
+ "expected": "942",
471
+ "kind": "integer",
472
+ "output": "847 are left.",
473
+ "pass": false,
474
+ "tokens": 5
475
+ },
476
+ {
477
+ "id": "heldout_50",
478
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_851, count=141.",
479
+ "expected": {
480
+ "item": "part_851",
481
+ "count": 141
482
+ },
483
+ "kind": "json",
484
+ "output": "{\"item\": \"part_851\", \"count\": 141}",
485
+ "pass": true,
486
+ "tokens": 15
487
+ },
488
+ {
489
+ "id": "heldout_51",
490
+ "prompt": "Return only one CSV value row, no header: name=Item854; quantity=142; code=K31.",
491
+ "expected": "Item854,142,K31",
492
+ "kind": "exact",
493
+ "output": "Item854,142,K31",
494
+ "pass": true,
495
+ "tokens": 7
496
+ },
497
+ {
498
+ "id": "heldout_52",
499
+ "prompt": "Copy exactly, no other text: Key_857-B143_C32",
500
+ "expected": "Key_857-B143_C32",
501
+ "kind": "exact",
502
+ "output": "Key_857-B143_C32",
503
+ "pass": true,
504
+ "tokens": 8
505
+ },
506
+ {
507
+ "id": "heldout_53",
508
+ "prompt": "Write only a Python function named solve(x) that returns x * 144 + 33.",
509
+ "expected": "x * 144 + 33",
510
+ "kind": "code",
511
+ "output": "```python\ndef solve(x):\n return x * 144 + 33\n```",
512
+ "pass": true,
513
+ "tokens": 19
514
+ },
515
+ {
516
+ "id": "heldout_54",
517
+ "prompt": "Return only the integer: 863 + 145 - 34.",
518
+ "expected": "974",
519
+ "kind": "integer",
520
+ "output": "974",
521
+ "pass": true,
522
+ "tokens": 2
523
+ },
524
+ {
525
+ "id": "heldout_55",
526
+ "prompt": "A crate has 866 red and 146 blue beads. Remove 35 red beads. Return only the total beads remaining.",
527
+ "expected": "977",
528
+ "kind": "integer",
529
+ "output": "865 are left.",
530
+ "pass": false,
531
+ "tokens": 5
532
+ },
533
+ {
534
+ "id": "heldout_56",
535
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_869, count=147.",
536
+ "expected": {
537
+ "item": "part_869",
538
+ "count": 147
539
+ },
540
+ "kind": "json",
541
+ "output": "{\"item\": \"part_869\", \"count\": 147}",
542
+ "pass": true,
543
+ "tokens": 15
544
+ },
545
+ {
546
+ "id": "heldout_57",
547
+ "prompt": "Return only one CSV value row, no header: name=Item872; quantity=148; code=K37.",
548
+ "expected": "Item872,148,K37",
549
+ "kind": "exact",
550
+ "output": "Item872,148,K37",
551
+ "pass": true,
552
+ "tokens": 7
553
+ },
554
+ {
555
+ "id": "heldout_58",
556
+ "prompt": "Copy exactly, no other text: Key_875-B149_C38",
557
+ "expected": "Key_875-B149_C38",
558
+ "kind": "exact",
559
+ "output": "Key_875-B149_C38",
560
+ "pass": true,
561
+ "tokens": 8
562
+ },
563
+ {
564
+ "id": "heldout_59",
565
+ "prompt": "Write only a Python function named solve(x) that returns x * 150 + 39.",
566
+ "expected": "x * 150 + 39",
567
+ "kind": "code",
568
+ "output": "```python\ndef solve(x):\n return x * 150 + 39\n```",
569
+ "pass": true,
570
+ "tokens": 19
571
+ }
572
+ ]
v3-results/v3_reloaded_outputs.json ADDED
@@ -0,0 +1,572 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "id": "heldout_00",
4
+ "prompt": "Return only the integer: 701 + 91 - 31.",
5
+ "expected": "761",
6
+ "kind": "integer",
7
+ "output": "761",
8
+ "pass": true,
9
+ "tokens": 2
10
+ },
11
+ {
12
+ "id": "heldout_01",
13
+ "prompt": "A crate has 704 red and 92 blue beads. Remove 32 red beads. Return only the total beads remaining.",
14
+ "expected": "764",
15
+ "kind": "integer",
16
+ "output": "The number of red beads left is 672. The number of blue beads is still 92, for a total of 764 beads.",
17
+ "pass": false,
18
+ "tokens": 29
19
+ },
20
+ {
21
+ "id": "heldout_02",
22
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_707, count=93.",
23
+ "expected": {
24
+ "item": "part_707",
25
+ "count": 93
26
+ },
27
+ "kind": "json",
28
+ "output": "{\"item\": \"part_707\", \"count\": 93}",
29
+ "pass": true,
30
+ "tokens": 15
31
+ },
32
+ {
33
+ "id": "heldout_03",
34
+ "prompt": "Return only one CSV value row, no header: name=Item710; quantity=94; code=K34.",
35
+ "expected": "Item710,94,K34",
36
+ "kind": "exact",
37
+ "output": "Item710,94,K34",
38
+ "pass": true,
39
+ "tokens": 7
40
+ },
41
+ {
42
+ "id": "heldout_04",
43
+ "prompt": "Copy exactly, no other text: Key_713-B95_C35",
44
+ "expected": "Key_713-B95_C35",
45
+ "kind": "exact",
46
+ "output": "Key_713-B95_C35",
47
+ "pass": true,
48
+ "tokens": 8
49
+ },
50
+ {
51
+ "id": "heldout_05",
52
+ "prompt": "Write only a Python function named solve(x) that returns x * 96 + 36.",
53
+ "expected": "x * 96 + 36",
54
+ "kind": "code",
55
+ "output": "```python\ndef solve(x):\n return x * 96 + 36\n```",
56
+ "pass": true,
57
+ "tokens": 19
58
+ },
59
+ {
60
+ "id": "heldout_06",
61
+ "prompt": "Return only the integer: 719 + 97 - 37.",
62
+ "expected": "779",
63
+ "kind": "integer",
64
+ "output": "779",
65
+ "pass": true,
66
+ "tokens": 2
67
+ },
68
+ {
69
+ "id": "heldout_07",
70
+ "prompt": "A crate has 722 red and 98 blue beads. Remove 38 red beads. Return only the total beads remaining.",
71
+ "expected": "782",
72
+ "kind": "integer",
73
+ "output": "The number of red beads left is 684. The number of beads left is 684 + 98 = 782.",
74
+ "pass": false,
75
+ "tokens": 26
76
+ },
77
+ {
78
+ "id": "heldout_08",
79
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_725, count=99.",
80
+ "expected": {
81
+ "item": "part_725",
82
+ "count": 99
83
+ },
84
+ "kind": "json",
85
+ "output": "{\"item\": \"part_725\", \"count\": 99}",
86
+ "pass": true,
87
+ "tokens": 15
88
+ },
89
+ {
90
+ "id": "heldout_09",
91
+ "prompt": "Return only one CSV value row, no header: name=Item728; quantity=100; code=K40.",
92
+ "expected": "Item728,100,K40",
93
+ "kind": "exact",
94
+ "output": "Item728,100,K40",
95
+ "pass": true,
96
+ "tokens": 7
97
+ },
98
+ {
99
+ "id": "heldout_10",
100
+ "prompt": "Copy exactly, no other text: Key_731-B101_C41",
101
+ "expected": "Key_731-B101_C41",
102
+ "kind": "exact",
103
+ "output": "Key_731-B101_C41",
104
+ "pass": true,
105
+ "tokens": 8
106
+ },
107
+ {
108
+ "id": "heldout_11",
109
+ "prompt": "Write only a Python function named solve(x) that returns x * 102 + 42.",
110
+ "expected": "x * 102 + 42",
111
+ "kind": "code",
112
+ "output": "```python\ndef solve(x):\n return x * 102 + 42\n```",
113
+ "pass": true,
114
+ "tokens": 19
115
+ },
116
+ {
117
+ "id": "heldout_12",
118
+ "prompt": "Return only the integer: 737 + 103 - 43.",
119
+ "expected": "797",
120
+ "kind": "integer",
121
+ "output": "797",
122
+ "pass": true,
123
+ "tokens": 2
124
+ },
125
+ {
126
+ "id": "heldout_13",
127
+ "prompt": "A crate has 740 red and 104 blue beads. Remove 44 red beads. Return only the total beads remaining.",
128
+ "expected": "800",
129
+ "kind": "integer",
130
+ "output": "The number of red beads left is 696. The number of blue beads is unchanged at 104, for a total of 800 beads remaining.",
131
+ "pass": false,
132
+ "tokens": 31
133
+ },
134
+ {
135
+ "id": "heldout_14",
136
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_743, count=105.",
137
+ "expected": {
138
+ "item": "part_743",
139
+ "count": 105
140
+ },
141
+ "kind": "json",
142
+ "output": "{\"item\": \"part_743\", \"count\": 105}",
143
+ "pass": true,
144
+ "tokens": 15
145
+ },
146
+ {
147
+ "id": "heldout_15",
148
+ "prompt": "Return only one CSV value row, no header: name=Item746; quantity=106; code=K46.",
149
+ "expected": "Item746,106,K46",
150
+ "kind": "exact",
151
+ "output": "Item746,106,K46",
152
+ "pass": true,
153
+ "tokens": 7
154
+ },
155
+ {
156
+ "id": "heldout_16",
157
+ "prompt": "Copy exactly, no other text: Key_749-B107_C47",
158
+ "expected": "Key_749-B107_C47",
159
+ "kind": "exact",
160
+ "output": "Key_749-B107_C47",
161
+ "pass": true,
162
+ "tokens": 8
163
+ },
164
+ {
165
+ "id": "heldout_17",
166
+ "prompt": "Write only a Python function named solve(x) that returns x * 108 + 31.",
167
+ "expected": "x * 108 + 31",
168
+ "kind": "code",
169
+ "output": "```python\ndef solve(x):\n return x * 108 + 31\n```",
170
+ "pass": true,
171
+ "tokens": 19
172
+ },
173
+ {
174
+ "id": "heldout_18",
175
+ "prompt": "Return only the integer: 755 + 109 - 32.",
176
+ "expected": "832",
177
+ "kind": "integer",
178
+ "output": "732",
179
+ "pass": false,
180
+ "tokens": 2
181
+ },
182
+ {
183
+ "id": "heldout_19",
184
+ "prompt": "A crate has 758 red and 110 blue beads. Remove 33 red beads. Return only the total beads remaining.",
185
+ "expected": "835",
186
+ "kind": "integer",
187
+ "output": "The number of red beads left is 758 - 33 = 725. The total number of beads left is 725 + 110 = 835.",
188
+ "pass": false,
189
+ "tokens": 33
190
+ },
191
+ {
192
+ "id": "heldout_20",
193
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_761, count=111.",
194
+ "expected": {
195
+ "item": "part_761",
196
+ "count": 111
197
+ },
198
+ "kind": "json",
199
+ "output": "{\"item\": \"part_761\", \"count\": 111}",
200
+ "pass": true,
201
+ "tokens": 15
202
+ },
203
+ {
204
+ "id": "heldout_21",
205
+ "prompt": "Return only one CSV value row, no header: name=Item764; quantity=112; code=K35.",
206
+ "expected": "Item764,112,K35",
207
+ "kind": "exact",
208
+ "output": "Item764,112,K35",
209
+ "pass": true,
210
+ "tokens": 7
211
+ },
212
+ {
213
+ "id": "heldout_22",
214
+ "prompt": "Copy exactly, no other text: Key_767-B113_C36",
215
+ "expected": "Key_767-B113_C36",
216
+ "kind": "exact",
217
+ "output": "Key_767-B113_C36",
218
+ "pass": true,
219
+ "tokens": 8
220
+ },
221
+ {
222
+ "id": "heldout_23",
223
+ "prompt": "Write only a Python function named solve(x) that returns x * 114 + 37.",
224
+ "expected": "x * 114 + 37",
225
+ "kind": "code",
226
+ "output": "```python\ndef solve(x):\n return x * 114 + 37\n```",
227
+ "pass": true,
228
+ "tokens": 19
229
+ },
230
+ {
231
+ "id": "heldout_24",
232
+ "prompt": "Return only the integer: 773 + 115 - 38.",
233
+ "expected": "850",
234
+ "kind": "integer",
235
+ "output": "750",
236
+ "pass": false,
237
+ "tokens": 2
238
+ },
239
+ {
240
+ "id": "heldout_25",
241
+ "prompt": "A crate has 776 red and 116 blue beads. Remove 39 red beads. Return only the total beads remaining.",
242
+ "expected": "853",
243
+ "kind": "integer",
244
+ "output": "The number of red beads left is 737. There are 116 blue beads, 737 red beads, and 1533 beads left.",
245
+ "pass": false,
246
+ "tokens": 30
247
+ },
248
+ {
249
+ "id": "heldout_26",
250
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_779, count=117.",
251
+ "expected": {
252
+ "item": "part_779",
253
+ "count": 117
254
+ },
255
+ "kind": "json",
256
+ "output": "{\"item\": \"part_779\", \"count\": 117}",
257
+ "pass": true,
258
+ "tokens": 15
259
+ },
260
+ {
261
+ "id": "heldout_27",
262
+ "prompt": "Return only one CSV value row, no header: name=Item782; quantity=118; code=K41.",
263
+ "expected": "Item782,118,K41",
264
+ "kind": "exact",
265
+ "output": "Item782,118,K41",
266
+ "pass": true,
267
+ "tokens": 7
268
+ },
269
+ {
270
+ "id": "heldout_28",
271
+ "prompt": "Copy exactly, no other text: Key_785-B119_C42",
272
+ "expected": "Key_785-B119_C42",
273
+ "kind": "exact",
274
+ "output": "Key_785-B119_C42",
275
+ "pass": true,
276
+ "tokens": 8
277
+ },
278
+ {
279
+ "id": "heldout_29",
280
+ "prompt": "Write only a Python function named solve(x) that returns x * 120 + 43.",
281
+ "expected": "x * 120 + 43",
282
+ "kind": "code",
283
+ "output": "```python\ndef solve(x):\n return x * 120 + 43\n```",
284
+ "pass": true,
285
+ "tokens": 19
286
+ },
287
+ {
288
+ "id": "heldout_30",
289
+ "prompt": "Return only the integer: 791 + 121 - 44.",
290
+ "expected": "868",
291
+ "kind": "integer",
292
+ "output": "768",
293
+ "pass": false,
294
+ "tokens": 2
295
+ },
296
+ {
297
+ "id": "heldout_31",
298
+ "prompt": "A crate has 794 red and 122 blue beads. Remove 45 red beads. Return only the total beads remaining.",
299
+ "expected": "871",
300
+ "kind": "integer",
301
+ "output": "The number of red beads left is 749 because 794 - 45 = 749. The total number of beads left is 749 + 122 = 871.",
302
+ "pass": false,
303
+ "tokens": 36
304
+ },
305
+ {
306
+ "id": "heldout_32",
307
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_797, count=123.",
308
+ "expected": {
309
+ "item": "part_797",
310
+ "count": 123
311
+ },
312
+ "kind": "json",
313
+ "output": "{\"item\": \"part_797\", \"count\": 123}",
314
+ "pass": true,
315
+ "tokens": 15
316
+ },
317
+ {
318
+ "id": "heldout_33",
319
+ "prompt": "Return only one CSV value row, no header: name=Item800; quantity=124; code=K47.",
320
+ "expected": "Item800,124,K47",
321
+ "kind": "exact",
322
+ "output": "Item800,124,K47",
323
+ "pass": true,
324
+ "tokens": 7
325
+ },
326
+ {
327
+ "id": "heldout_34",
328
+ "prompt": "Copy exactly, no other text: Key_803-B125_C31",
329
+ "expected": "Key_803-B125_C31",
330
+ "kind": "exact",
331
+ "output": "Key_803-B125_C31",
332
+ "pass": true,
333
+ "tokens": 8
334
+ },
335
+ {
336
+ "id": "heldout_35",
337
+ "prompt": "Write only a Python function named solve(x) that returns x * 126 + 32.",
338
+ "expected": "x * 126 + 32",
339
+ "kind": "code",
340
+ "output": "```python\ndef solve(x):\n return x * 126 + 32\n```",
341
+ "pass": true,
342
+ "tokens": 19
343
+ },
344
+ {
345
+ "id": "heldout_36",
346
+ "prompt": "Return only the integer: 809 + 127 - 33.",
347
+ "expected": "903",
348
+ "kind": "integer",
349
+ "output": "803",
350
+ "pass": false,
351
+ "tokens": 2
352
+ },
353
+ {
354
+ "id": "heldout_37",
355
+ "prompt": "A crate has 812 red and 128 blue beads. Remove 34 red beads. Return only the total beads remaining.",
356
+ "expected": "906",
357
+ "kind": "integer",
358
+ "output": "The number of red beads left is 812 - 34 = 778. The total number of beads left is 778 + 128 = 906.",
359
+ "pass": false,
360
+ "tokens": 33
361
+ },
362
+ {
363
+ "id": "heldout_38",
364
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_815, count=129.",
365
+ "expected": {
366
+ "item": "part_815",
367
+ "count": 129
368
+ },
369
+ "kind": "json",
370
+ "output": "{\"item\": \"part_815\", \"count\": 129}",
371
+ "pass": true,
372
+ "tokens": 15
373
+ },
374
+ {
375
+ "id": "heldout_39",
376
+ "prompt": "Return only one CSV value row, no header: name=Item818; quantity=130; code=K36.",
377
+ "expected": "Item818,130,K36",
378
+ "kind": "exact",
379
+ "output": "Item818,130,K36",
380
+ "pass": true,
381
+ "tokens": 7
382
+ },
383
+ {
384
+ "id": "heldout_40",
385
+ "prompt": "Copy exactly, no other text: Key_821-B131_C37",
386
+ "expected": "Key_821-B131_C37",
387
+ "kind": "exact",
388
+ "output": "Key_821-B131_C37",
389
+ "pass": true,
390
+ "tokens": 8
391
+ },
392
+ {
393
+ "id": "heldout_41",
394
+ "prompt": "Write only a Python function named solve(x) that returns x * 132 + 38.",
395
+ "expected": "x * 132 + 38",
396
+ "kind": "code",
397
+ "output": "```python\ndef solve(x):\n return x * 132 + 38\n```",
398
+ "pass": true,
399
+ "tokens": 19
400
+ },
401
+ {
402
+ "id": "heldout_42",
403
+ "prompt": "Return only the integer: 827 + 133 - 39.",
404
+ "expected": "921",
405
+ "kind": "integer",
406
+ "output": "821",
407
+ "pass": false,
408
+ "tokens": 2
409
+ },
410
+ {
411
+ "id": "heldout_43",
412
+ "prompt": "A crate has 830 red and 134 blue beads. Remove 40 red beads. Return only the total beads remaining.",
413
+ "expected": "924",
414
+ "kind": "integer",
415
+ "output": "The number of red beads left is 790 because 830 - 40 = 790. The total number of beads left is 790 + 134 = 924.",
416
+ "pass": false,
417
+ "tokens": 36
418
+ },
419
+ {
420
+ "id": "heldout_44",
421
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_833, count=135.",
422
+ "expected": {
423
+ "item": "part_833",
424
+ "count": 135
425
+ },
426
+ "kind": "json",
427
+ "output": "{\"item\": \"part_833\", \"count\": 135}",
428
+ "pass": true,
429
+ "tokens": 15
430
+ },
431
+ {
432
+ "id": "heldout_45",
433
+ "prompt": "Return only one CSV value row, no header: name=Item836; quantity=136; code=K42.",
434
+ "expected": "Item836,136,K42",
435
+ "kind": "exact",
436
+ "output": "Item836,136,K42",
437
+ "pass": true,
438
+ "tokens": 7
439
+ },
440
+ {
441
+ "id": "heldout_46",
442
+ "prompt": "Copy exactly, no other text: Key_839-B137_C43",
443
+ "expected": "Key_839-B137_C43",
444
+ "kind": "exact",
445
+ "output": "Key_839-B137_C43",
446
+ "pass": true,
447
+ "tokens": 8
448
+ },
449
+ {
450
+ "id": "heldout_47",
451
+ "prompt": "Write only a Python function named solve(x) that returns x * 138 + 44.",
452
+ "expected": "x * 138 + 44",
453
+ "kind": "code",
454
+ "output": "```python\ndef solve(x):\n return x * 138 + 44\n```",
455
+ "pass": true,
456
+ "tokens": 19
457
+ },
458
+ {
459
+ "id": "heldout_48",
460
+ "prompt": "Return only the integer: 845 + 139 - 45.",
461
+ "expected": "939",
462
+ "kind": "integer",
463
+ "output": "839",
464
+ "pass": false,
465
+ "tokens": 2
466
+ },
467
+ {
468
+ "id": "heldout_49",
469
+ "prompt": "A crate has 848 red and 140 blue beads. Remove 46 red beads. Return only the total beads remaining.",
470
+ "expected": "942",
471
+ "kind": "integer",
472
+ "output": "The number of red beads left is 848 - 46 = 802. The total number of beads left is 802 + 140 = 942.",
473
+ "pass": false,
474
+ "tokens": 33
475
+ },
476
+ {
477
+ "id": "heldout_50",
478
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_851, count=141.",
479
+ "expected": {
480
+ "item": "part_851",
481
+ "count": 141
482
+ },
483
+ "kind": "json",
484
+ "output": "{\"item\": \"part_851\", \"count\": 141}",
485
+ "pass": true,
486
+ "tokens": 15
487
+ },
488
+ {
489
+ "id": "heldout_51",
490
+ "prompt": "Return only one CSV value row, no header: name=Item854; quantity=142; code=K31.",
491
+ "expected": "Item854,142,K31",
492
+ "kind": "exact",
493
+ "output": "Item854,142,K31",
494
+ "pass": true,
495
+ "tokens": 7
496
+ },
497
+ {
498
+ "id": "heldout_52",
499
+ "prompt": "Copy exactly, no other text: Key_857-B143_C32",
500
+ "expected": "Key_857-B143_C32",
501
+ "kind": "exact",
502
+ "output": "Key_857-B143_C32",
503
+ "pass": true,
504
+ "tokens": 8
505
+ },
506
+ {
507
+ "id": "heldout_53",
508
+ "prompt": "Write only a Python function named solve(x) that returns x * 144 + 33.",
509
+ "expected": "x * 144 + 33",
510
+ "kind": "code",
511
+ "output": "```python\ndef solve(x):\n return x * 144 + 33\n```",
512
+ "pass": true,
513
+ "tokens": 19
514
+ },
515
+ {
516
+ "id": "heldout_54",
517
+ "prompt": "Return only the integer: 863 + 145 - 34.",
518
+ "expected": "974",
519
+ "kind": "integer",
520
+ "output": "874",
521
+ "pass": false,
522
+ "tokens": 2
523
+ },
524
+ {
525
+ "id": "heldout_55",
526
+ "prompt": "A crate has 866 red and 146 blue beads. Remove 35 red beads. Return only the total beads remaining.",
527
+ "expected": "977",
528
+ "kind": "integer",
529
+ "output": "The number of red beads left is 831. The number of blue beads is still 146, for a total of 977 beads.",
530
+ "pass": false,
531
+ "tokens": 29
532
+ },
533
+ {
534
+ "id": "heldout_56",
535
+ "prompt": "Return only valid JSON with keys \"item\" and \"count\": item=part_869, count=147.",
536
+ "expected": {
537
+ "item": "part_869",
538
+ "count": 147
539
+ },
540
+ "kind": "json",
541
+ "output": "{\"item\": \"part_869\", \"count\": 147}",
542
+ "pass": true,
543
+ "tokens": 15
544
+ },
545
+ {
546
+ "id": "heldout_57",
547
+ "prompt": "Return only one CSV value row, no header: name=Item872; quantity=148; code=K37.",
548
+ "expected": "Item872,148,K37",
549
+ "kind": "exact",
550
+ "output": "Item872,148,K37",
551
+ "pass": true,
552
+ "tokens": 7
553
+ },
554
+ {
555
+ "id": "heldout_58",
556
+ "prompt": "Copy exactly, no other text: Key_875-B149_C38",
557
+ "expected": "Key_875-B149_C38",
558
+ "kind": "exact",
559
+ "output": "Key_875-B149_C38",
560
+ "pass": true,
561
+ "tokens": 8
562
+ },
563
+ {
564
+ "id": "heldout_59",
565
+ "prompt": "Write only a Python function named solve(x) that returns x * 150 + 39.",
566
+ "expected": "x * 150 + 39",
567
+ "kind": "code",
568
+ "output": "```python\ndef solve(x):\n return x * 150 + 39\n```",
569
+ "pass": true,
570
+ "tokens": 19
571
+ }
572
+ ]
v3-results/v3_report.json ADDED
@@ -0,0 +1,47 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "status": "complete",
3
+ "parent_revision": "d9aa899722be4788f57107c2951c72d0ec6614ee",
4
+ "hardware": "NVIDIA A10G",
5
+ "job_timeout_seconds": 5400,
6
+ "listed_usd_per_hour": 1.0,
7
+ "training_seed": 20261113,
8
+ "baseline_passed": 40,
9
+ "elapsed_seconds": 622.09,
10
+ "training_examples": 1713,
11
+ "trainable_parameters": 41943040,
12
+ "training": {
13
+ "train_runtime": 390.2404,
14
+ "train_samples_per_second": 1.845,
15
+ "train_steps_per_second": 0.461,
16
+ "total_flos": 3960249019342848.0,
17
+ "train_loss": 0.2867420093880759,
18
+ "epoch": 0.4203152364273205
19
+ },
20
+ "optimizer_steps": 180,
21
+ "candidate_passed": 49,
22
+ "reloaded_passed": 43,
23
+ "groups": {
24
+ "integer": {
25
+ "baseline": 0,
26
+ "candidate": 3,
27
+ "total": 20
28
+ },
29
+ "exact": {
30
+ "baseline": 20,
31
+ "candidate": 20,
32
+ "total": 20
33
+ },
34
+ "code": {
35
+ "baseline": 10,
36
+ "candidate": 10,
37
+ "total": 10
38
+ },
39
+ "json": {
40
+ "baseline": 10,
41
+ "candidate": 10,
42
+ "total": 10
43
+ }
44
+ },
45
+ "quality_gate_passed": true,
46
+ "peak_gpu_bytes": 18991949312
47
+ }
v3/independent_eval.py ADDED
@@ -0,0 +1,98 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # /// script
2
+ # dependencies = ["torch==2.8.0", "transformers==4.56.2", "huggingface_hub==0.34.4", "accelerate==1.10.1", "safetensors==0.6.2"]
3
+ # ///
4
+ """Independent GSM8K and HumanEval subset comparison on remote GPUs."""
5
+ import json,re,sys,subprocess,tempfile,ast,urllib.request,urllib.parse,gc,hashlib
6
+ from pathlib import Path
7
+ import torch
8
+ from transformers import AutoModelForCausalLM,AutoTokenizer
9
+ from huggingface_hub import HfApi,hf_hub_download,CommitOperationAdd
10
+ def task_check(case,text):
11
+ final=text.split("</think>")[-1]
12
+ numbers=re.findall(r"-?\d+(?:\.\d+)?",final.replace(",",""))
13
+ return bool(numbers) and abs(float(numbers[-1])-float(case["expected"]))<1e-6
14
+ api=HfApi();REPO="SEE7/Astheral";BASE="NousResearch/DeepHermes-3-Llama-3-8B-Preview";REV="53d902c7dcfa4c749a3b455149064f82af2e2549"
15
+ ROOT=Path(__file__).resolve().parent
16
+ BRANCH="v3-candidate-20261113"
17
+ revision=api.model_info(REPO,revision=BRANCH).sha
18
+ settings=json.loads(Path(hf_hub_download(REPO,"settings.json",revision=revision)).read_text())
19
+ def rows(repo,config,split,offset,length):
20
+ q=urllib.parse.urlencode(dict(dataset=repo,config=config,split=split,offset=offset,length=length))
21
+ return [r["row"] for r in json.load(urllib.request.urlopen("https://datasets-server.huggingface.co/rows?"+q,timeout=60))["rows"]]
22
+ math_revision=api.dataset_info("openai/gsm8k").sha
23
+ code_revision=api.dataset_info("openai/openai_humaneval").sha
24
+ cases=[]
25
+ for i,r in enumerate(rows("openai/gsm8k","main","test",231,8)):
26
+ expected=float(r["answer"].rsplit("####",1)[1].strip().replace(",",""))
27
+ cases.append({"id":"gsm8k-test-"+str(231+i),"mode":"thinking","prompt":r["question"]+"\nGive the final numerical answer after your explanation.","kind":"number","expected":expected,"source":"openai/gsm8k","source_revision":math_revision})
28
+ chosen={"HumanEval/"+str(i) for i in [0,3,7,12,18,25,36,45]}
29
+ for r in rows("openai/openai_humaneval","openai_humaneval","test",0,50):
30
+ if r["task_id"] not in chosen:continue
31
+ cases.append({"id":r["task_id"],"mode":"instruct","prompt":"Return only the complete Python implementation, including necessary imports, for the following function specification:\n\n"+r["prompt"],"kind":"code","entry_point":r["entry_point"],"tests":r["test"]+"\ncheck("+r["entry_point"]+")","source":"openai/openai_humaneval","source_revision":code_revision})
32
+ assert len(cases)==16
33
+ def check_code(c,text):
34
+ blocks=re.findall(r"\x60\x60\x60(?:python)?\s*\n(.*?)\x60\x60\x60",text,re.S)
35
+ code=blocks[0] if blocks else text.strip()
36
+ try:
37
+ tree=ast.parse(code)
38
+ for n in ast.walk(tree):
39
+ if isinstance(n,(ast.Import,ast.ImportFrom)):
40
+ names=[a.name.split(".")[0] for a in n.names] if isinstance(n,ast.Import) else [(n.module or "").split(".")[0]]
41
+ if any(x not in ["math","collections","itertools","functools","re","typing","statistics","heapq","bisect","string"] for x in names):return False
42
+ if isinstance(n,ast.Name) and n.id in ["open","exec","eval","compile","__import__","__builtins__","globals","locals","getattr","setattr","input"]:return False
43
+ if isinstance(n,ast.Attribute) and n.attr.startswith("__"):return False
44
+ with tempfile.TemporaryDirectory() as d:
45
+ p=Path(d)/"check.py"
46
+ p.write_text("import resource\nresource.setrlimit(resource.RLIMIT_CPU,(3,3))\nresource.setrlimit(resource.RLIMIT_AS,(536870912,536870912))\n"+code+"\n"+c["tests"])
47
+ r=subprocess.run([sys.executable,"-I",str(p)],cwd=d,timeout=5,capture_output=True,env={"PATH":"/usr/bin:/bin"})
48
+ return r.returncode==0
49
+ except Exception:return False
50
+ all_results={};scores={};paired_results={}
51
+ source=Path(hf_hub_download(REPO,'v3/train_v3.py',revision=revision)).read_text()
52
+ grade_node=next(n for n in ast.parse(source).body if isinstance(n,ast.FunctionDef) and n.name=='grade')
53
+ grade_scope={'json':json,'re':re};exec(compile(ast.Module(body=[grade_node],type_ignores=[]),'grader','exec'),grade_scope)
54
+ paired_cases=json.loads(Path(hf_hub_download(REPO,'v3-results/heldout_cases.json',revision=revision)).read_text())
55
+ for tag,repo,rev in [("base",REPO,"d9aa899722be4788f57107c2951c72d0ec6614ee"),("astheral",REPO,revision)]:
56
+ tok=AutoTokenizer.from_pretrained(repo,revision=rev);tok.pad_token_id=128004;tok.padding_side="left"
57
+ model=AutoModelForCausalLM.from_pretrained(repo,revision=rev,torch_dtype=torch.bfloat16,device_map={"":"cuda:0"},attn_implementation="sdpa").eval()
58
+ paired=[]
59
+ with torch.inference_mode():
60
+ for c in paired_cases:
61
+ inp=tok.apply_chat_template([{"role":"system","content":settings["system"]},{"role":"user","content":c["prompt"]}],tokenize=True,add_generation_prompt=True,return_dict=True,return_tensors="pt").to("cuda")
62
+ output=model.generate(**inp,max_new_tokens=96,do_sample=False,pad_token_id=128004,eos_token_id=128009)
63
+ text=tok.decode(output[0,inp["input_ids"].shape[1]:],skip_special_tokens=True)
64
+ paired.append({**c,"output":text,"pass":grade_scope["grade"](c,text)})
65
+ print("BF16_PAIRED",tag,c["id"],paired[-1]["pass"],flush=True)
66
+ paired_results[tag]=paired
67
+ result=[]
68
+ with torch.inference_mode():
69
+ for mode in ["instruct","thinking"]:
70
+ selected=[c for c in cases if c["mode"]==mode]
71
+ for i in range(0,len(selected),2):
72
+ group=selected[i:i+2]
73
+ system=settings["system"]+("\n"+settings["thinking"] if mode=="thinking" else "")
74
+ texts=[tok.apply_chat_template([{"role":"system","content":system},{"role":"user","content":c["prompt"]}],tokenize=False,add_generation_prompt=True) for c in group]
75
+ inputs=tok(texts,padding=True,return_tensors="pt").to("cuda")
76
+ out=model.generate(**inputs,max_new_tokens=1536 if mode=="thinking" else 768,do_sample=False,pad_token_id=128004,eos_token_id=128009)
77
+ for c,o in zip(group,out):
78
+ ids=o[inputs["input_ids"].shape[1]:]
79
+ text=tok.decode(ids,skip_special_tokens=True)
80
+ passed=check_code(c,text) if mode=="instruct" else task_check(c,text)
81
+ result.append({**c,"output":text,"pass":passed,"hit_token_limit":len(ids)>=(1536 if mode=="thinking" else 768) and 128009 not in ids.tolist()})
82
+ print("HOLDOUT",tag,c["id"],passed,flush=True)
83
+ all_results[tag]=result
84
+ scores[tag]={}
85
+ for kind in ["code","number"]:
86
+ chosen_rows=[r for r in result if r["kind"]==kind]
87
+ scores[tag][kind]={"passed":sum(r["pass"] for r in chosen_rows),"total":len(chosen_rows),"token_limited":sum(r["hit_token_limit"] for r in chosen_rows)}
88
+ del model;gc.collect();torch.cuda.empty_cache()
89
+ passed=sum(v["passed"] for v in scores["astheral"].values())>=sum(v["passed"] for v in scores["base"].values()) and all(scores["astheral"][k]["passed"]>=scores["base"][k]["passed"] for k in ["code","number"])
90
+ passed=passed and sum(r["pass"] for r in paired_results["astheral"])>=sum(r["pass"] for r in paired_results["base"])
91
+ report={"comparison":"V2 versus V3, both BF16","v2_revision":"d9aa899722be4788f57107c2951c72d0ec6614ee","paired_bf16_scores":{k:sum(r["pass"] for r in v) for k,v in paired_results.items()},"suite":"GSM8K test rows 231-238; HumanEval selected tasks 0,3,7,12,18,25,36,45","scores":scores,"no_material_regression_gate":passed,"astheral_revision":revision,"settings":{"sampling":False,"code_max_new_tokens":768,"math_max_new_tokens":1536},"hardware":torch.cuda.get_device_name(0),"source_revisions":{"gsm8k":math_revision,"humaneval":code_revision}}
92
+ changed={"paired_bf16_outputs.json":json.dumps(paired_results,indent=2),"holdout_evaluation.json":json.dumps(report,indent=2),"holdout_outputs.json":json.dumps(all_results,indent=2),"holdout_eval.py":Path(__file__).read_text(),"HUMANEVAL_LICENSE.txt":urllib.request.urlopen("https://raw.githubusercontent.com/openai/human-eval/master/LICENSE",timeout=60).read().decode()}
93
+ manifest=json.loads(Path(hf_hub_download(REPO,"artifact_manifest.json")).read_text())
94
+ for n,b in changed.items():manifest[n]={"bytes":len(b.encode()),"sha256":hashlib.sha256(b.encode()).hexdigest()}
95
+ changed["artifact_manifest.json"]=json.dumps(manifest,indent=2)
96
+ api.create_commit(repo_id=REPO,revision=BRANCH,operations=[CommitOperationAdd(path_in_repo=n,path_or_fileobj=b.encode()) for n,b in changed.items()],commit_message="Record measured V2 versus V3 GSM8K and HumanEval comparison")
97
+ print("HOLDOUT_REPORT",json.dumps(report),flush=True)
98
+ if not passed:raise SystemExit(5)
v3/train_v3.py ADDED
@@ -0,0 +1,128 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # /// script
2
+ # dependencies = ["torch==2.8.0", "transformers==4.56.2", "peft==0.17.1", "accelerate==1.10.1", "huggingface_hub==0.34.4", "bitsandbytes==0.47.0", "safetensors==0.6.2"]
3
+ # ///
4
+ """Budget-bounded V3 training with paired, disjoint evaluation and reload gate."""
5
+ import os, json, random, time, re, gc, hashlib, traceback
6
+ from pathlib import Path
7
+ import torch
8
+ from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, Trainer, TrainingArguments, TrainerCallback, set_seed
9
+ from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training, PeftModel
10
+ from huggingface_hub import HfApi, hf_hub_download
11
+
12
+ REPO='SEE7/Astheral'; REV='d9aa899722be4788f57107c2951c72d0ec6614ee'
13
+ OUT=Path('/tmp/astheral-v3'); OUT.mkdir(exist_ok=True)
14
+ api=HfApi(); started=time.monotonic(); set_seed(20261113); rng=random.Random(20261113)
15
+ assert torch.cuda.is_available() and 'A10G' in torch.cuda.get_device_name(0)
16
+ branch='v3-candidate-20261113'
17
+ api.create_branch(REPO,branch=branch,revision=REV,exist_ok=True)
18
+ settings=json.loads(Path(hf_hub_download(REPO,'settings.json',revision=REV)).read_text())
19
+ tok=AutoTokenizer.from_pretrained(REPO,revision=REV); tok.pad_token_id=128004
20
+ report={'status':'preparing','parent_revision':REV,'hardware':torch.cuda.get_device_name(0),'job_timeout_seconds':5400,'listed_usd_per_hour':1.0,'training_seed':20261113}
21
+
22
+ def publish():
23
+ report['elapsed_seconds']=round(time.monotonic()-started,2)
24
+ (OUT/'v3_report.json').write_text(json.dumps(report,indent=2))
25
+ api.upload_folder(repo_id=REPO,folder_path=OUT,revision=branch,path_in_repo='v3-results',ignore_patterns=['merged/**'],commit_message='Record measured V3 training and evaluation artifacts')
26
+
27
+ def sample(a,b,c,kind):
28
+ if kind==0:return f'Return only the integer: {a} + {b} - {c}.',str(a+b-c),'integer'
29
+ if kind==1:return f'A crate has {a} red and {b} blue beads. Remove {c} red beads. Return only the total beads remaining.',str(a+b-c),'integer'
30
+ if kind==2:return f'Return only valid JSON with keys "item" and "count": item=part_{a}, count={b}.',{'item':f'part_{a}','count':b},'json'
31
+ if kind==3:return f'Return only one CSV value row, no header: name=Item{a}; quantity={b}; code=K{c}.',f'Item{a},{b},K{c}','exact'
32
+ if kind==4:return f'Copy exactly, no other text: Key_{a}-B{b}_C{c}',f'Key_{a}-B{b}_C{c}','exact'
33
+ return f'Write only a Python function named solve(x) that returns x * {b} + {c}.',f'x * {b} + {c}','code'
34
+
35
+ def grade(case,text):
36
+ expected=case['expected']; kind=case['kind']; text=text.strip()
37
+ try:
38
+ if kind=='json':return json.loads(text)==expected
39
+ if kind=='integer':return re.fullmatch(r'-?\d+',text) is not None and int(text)==int(expected)
40
+ if kind=='exact':return text==expected
41
+ # Parse and evaluate a tiny arithmetic-only AST; never execute model-generated code.
42
+ import ast
43
+ clean=re.sub(r'^```(?:python)?\s*|\s*```$','',text).strip();tree=ast.parse(clean)
44
+ if len(tree.body)!=1 or not isinstance(tree.body[0],ast.FunctionDef):return False
45
+ fn=tree.body[0]
46
+ if fn.name!='solve' or len(fn.args.args)!=1 or fn.args.args[0].arg!='x' or len(fn.body)!=1 or not isinstance(fn.body[0],ast.Return):return False
47
+ def calc(n,x):
48
+ if isinstance(n,ast.Constant) and isinstance(n.value,(int,float)):return n.value
49
+ if isinstance(n,ast.Name) and n.id=='x':return x
50
+ if isinstance(n,ast.BinOp) and isinstance(n.op,(ast.Add,ast.Sub,ast.Mult)):
51
+ a,b=calc(n.left,x),calc(n.right,x)
52
+ return a+b if isinstance(n.op,ast.Add) else a-b if isinstance(n.op,ast.Sub) else a*b
53
+ raise ValueError('Unsupported expression')
54
+ match=re.fullmatch(r'x \* (\d+) \+ (\d+)',expected);b,c=map(int,match.groups())
55
+ return all(calc(fn.body[0].value,x)==x*b+c for x in [-5,0,1,7,100])
56
+ except Exception:return False
57
+
58
+ tests=[]
59
+ for i in range(60):
60
+ # Evaluation numbers are outside the synthetic training ranges.
61
+ a,b,c=701+i*3,91+i,31+i%17;p,e,k=sample(a,b,c,i%6)
62
+ tests.append({'id':f'heldout_{i:02}','prompt':p,'expected':e,'kind':k})
63
+ train=[]
64
+ for i in range(900):
65
+ a,b,c=rng.randint(100,500),rng.randint(10,80),rng.randint(1,29)
66
+ p,e,k=sample(a,b,c,i%6)
67
+ answer=json.dumps(e) if k=='json' else 'def solve(x):\n return '+e if k=='code' else str(e)
68
+ train.append({'messages':[{'role':'user','content':p},{'role':'assistant','content':answer}]})
69
+ retained=[json.loads(s) for s in Path(hf_hub_download(REPO,'train.jsonl',revision=REV)).read_text().splitlines()]
70
+ rng.shuffle(retained);train+=retained[:900];rng.shuffle(train)
71
+ assert not {x['prompt'] for x in tests} & {m['content'] for row in train for m in row['messages'] if m['role']=='user'}
72
+ (OUT/'heldout_cases.json').write_text(json.dumps(tests,indent=2));(OUT/'training.jsonl').write_text('\n'.join(json.dumps(x) for x in train)+'\n')
73
+ (OUT/'train_v3.py').write_text(Path(__file__).read_text())
74
+ q=BitsAndBytesConfig(load_in_4bit=True,bnb_4bit_quant_type='nf4',bnb_4bit_use_double_quant=True,bnb_4bit_compute_dtype=torch.bfloat16)
75
+ model=AutoModelForCausalLM.from_pretrained(REPO,revision=REV,quantization_config=q,device_map={'':'cuda:0'},torch_dtype=torch.bfloat16,attn_implementation='sdpa')
76
+
77
+ @torch.inference_mode()
78
+ def evaluate(label):
79
+ model.eval(); rows=[]
80
+ for i,case in enumerate(tests):
81
+ inputs=tok.apply_chat_template([{'role':'system','content':settings['system']},{'role':'user','content':case['prompt']}],tokenize=True,add_generation_prompt=True,return_dict=True,return_tensors='pt').to('cuda')
82
+ out=model.generate(**inputs,max_new_tokens=96,do_sample=False,pad_token_id=128004,eos_token_id=128009)
83
+ text=tok.decode(out[0,inputs['input_ids'].shape[1]:],skip_special_tokens=True)
84
+ rows.append({**case,'output':text,'pass':grade(case,text),'tokens':int(out.shape[1]-inputs['input_ids'].shape[1])})
85
+ print('EVAL',label,i,rows[-1]['pass'],flush=True)
86
+ (OUT/(label+'_outputs.json')).write_text(json.dumps(rows,indent=2))
87
+ return rows
88
+
89
+ try:
90
+ baseline=evaluate('v2_quantized'); report['baseline_passed']=sum(r['pass'] for r in baseline);publish()
91
+ model=prepare_model_for_kbit_training(model,use_gradient_checkpointing=True)
92
+ model=get_peft_model(model,LoraConfig(r=16,lora_alpha=32,lora_dropout=0.05,target_modules=['q_proj','k_proj','v_proj','o_proj','gate_proj','up_proj','down_proj'],task_type='CAUSAL_LM'))
93
+ model.config.use_cache=False;encoded=[]
94
+ for row in train:
95
+ messages=row['messages']; prefix=tok.apply_chat_template(messages[:-1],tokenize=True,add_generation_prompt=True)
96
+ full=tok.apply_chat_template(messages,tokenize=True,add_generation_prompt=False)
97
+ if len(full)<=512 and full[:len(prefix)]==prefix:encoded.append({'input_ids':full,'labels':[-100]*len(prefix)+full[len(prefix):]})
98
+ def collate(rows):
99
+ length=max(len(r['input_ids']) for r in rows)
100
+ return {k:torch.tensor([r[k]+[pad]*(length-len(r[k])) for r in rows]) for k,pad in [('input_ids',128004),('labels',-100)]}|{'attention_mask':torch.tensor([[1]*len(r['input_ids'])+[0]*(length-len(r['input_ids'])) for r in rows])}
101
+ class Budget(TrainerCallback):
102
+ def on_step_end(self,args,state,control,**kwargs):
103
+ if time.monotonic()-started>2400:control.should_training_stop=True
104
+ return control
105
+ args=TrainingArguments(output_dir='/tmp/v3-checkpoints',max_steps=180,per_device_train_batch_size=1,gradient_accumulation_steps=4,learning_rate=1e-5,warmup_ratio=0.05,bf16=True,gradient_checkpointing=True,gradient_checkpointing_kwargs={'use_reentrant':False},optim='paged_adamw_8bit',save_strategy='no',logging_steps=10,report_to='none',remove_unused_columns=False,seed=20261113)
106
+ trainer=Trainer(model=model,args=args,train_dataset=encoded,data_collator=collate,callbacks=[Budget()])
107
+ report.update(status='training',training_examples=len(encoded),trainable_parameters=sum(p.numel() for p in model.parameters() if p.requires_grad));publish()
108
+ result=trainer.train();report['training']=result.metrics;report['optimizer_steps']=trainer.state.global_step
109
+ adapter=OUT/'adapter';model.save_pretrained(adapter,safe_serialization=True);tok.save_pretrained(adapter)
110
+ model.config.use_cache=True;model.gradient_checkpointing_disable();candidate=evaluate('v3_quantized')
111
+ report['candidate_passed']=sum(r['pass'] for r in candidate);report['status']='trained';publish()
112
+ del trainer,model;gc.collect();torch.cuda.empty_cache()
113
+ model=AutoModelForCausalLM.from_pretrained(REPO,revision=REV,torch_dtype=torch.bfloat16,device_map={'':'cuda:0'},attn_implementation='sdpa')
114
+ model=PeftModel.from_pretrained(model,adapter).merge_and_unload(safe_merge=True)
115
+ model.config.astheral_version='3';merged=OUT/'merged';model.save_pretrained(merged,safe_serialization=True,max_shard_size='4GB');tok.save_pretrained(merged)
116
+ del model;gc.collect();torch.cuda.empty_cache()
117
+ model=AutoModelForCausalLM.from_pretrained(merged,torch_dtype=torch.bfloat16,device_map={'':'cuda:0'},attn_implementation='sdpa')
118
+ reloaded=evaluate('v3_reloaded');report['reloaded_passed']=sum(r['pass'] for r in reloaded)
119
+ groups={kind:{'baseline':sum(r['pass'] for r in baseline if r['kind']==kind),'candidate':sum(r['pass'] for r in reloaded if r['kind']==kind),'total':sum(r['kind']==kind for r in tests)} for kind in {x['kind'] for x in tests}}
120
+ report['groups']=groups;report['quality_gate_passed']=report['reloaded_passed']>report['baseline_passed'] and all(v['candidate']>=v['baseline'] for v in groups.values())
121
+ report['status']='complete';report['peak_gpu_bytes']=torch.cuda.max_memory_allocated();publish()
122
+ # Save the actual trained weights to the isolated candidate branch even on a failed gate.
123
+ api.upload_folder(repo_id=REPO,folder_path=merged,revision=branch,commit_message='Save reloaded V3 candidate weights; see measured quality gate in v3-results')
124
+ if report['quality_gate_passed']:
125
+ api.create_tag(REPO,tag='v3',revision=api.model_info(REPO,revision=branch).sha,exist_ok=True)
126
+ print('V3_COMPLETE',json.dumps(report),flush=True)
127
+ except Exception:
128
+ report['status']='failed';report['error']=traceback.format_exc();publish();raise