VisualInference commited on
Commit
e0b70af
·
verified ·
1 Parent(s): 6ab52ac

Inkling 2 Mini for MLX, 3bit: weights, card, validation results

Browse files
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
LICENSE ADDED
@@ -0,0 +1,202 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+
2
+ Apache License
3
+ Version 2.0, January 2004
4
+ http://www.apache.org/licenses/
5
+
6
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
7
+
8
+ 1. Definitions.
9
+
10
+ "License" shall mean the terms and conditions for use, reproduction,
11
+ and distribution as defined by Sections 1 through 9 of this document.
12
+
13
+ "Licensor" shall mean the copyright owner or entity authorized by
14
+ the copyright owner that is granting the License.
15
+
16
+ "Legal Entity" shall mean the union of the acting entity and all
17
+ other entities that control, are controlled by, or are under common
18
+ control with that entity. For the purposes of this definition,
19
+ "control" means (i) the power, direct or indirect, to cause the
20
+ direction or management of such entity, whether by contract or
21
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
22
+ outstanding shares, or (iii) beneficial ownership of such entity.
23
+
24
+ "You" (or "Your") shall mean an individual or Legal Entity
25
+ exercising permissions granted by this License.
26
+
27
+ "Source" form shall mean the preferred form for making modifications,
28
+ including but not limited to software source code, documentation
29
+ source, and configuration files.
30
+
31
+ "Object" form shall mean any form resulting from mechanical
32
+ transformation or translation of a Source form, including but
33
+ not limited to compiled object code, generated documentation,
34
+ and conversions to other media types.
35
+
36
+ "Work" shall mean the work of authorship, whether in Source or
37
+ Object form, made available under the License, as indicated by a
38
+ copyright notice that is included in or attached to the work
39
+ (an example is provided in the Appendix below).
40
+
41
+ "Derivative Works" shall mean any work, whether in Source or Object
42
+ form, that is based on (or derived from) the Work and for which the
43
+ editorial revisions, annotations, elaborations, or other modifications
44
+ represent, as a whole, an original work of authorship. For the purposes
45
+ of this License, Derivative Works shall not include works that remain
46
+ separable from, or merely link (or bind by name) to the interfaces of,
47
+ the Work and Derivative Works thereof.
48
+
49
+ "Contribution" shall mean any work of authorship, including
50
+ the original version of the Work and any modifications or additions
51
+ to that Work or Derivative Works thereof, that is intentionally
52
+ submitted to Licensor for inclusion in the Work by the copyright owner
53
+ or by an individual or Legal Entity authorized to submit on behalf of
54
+ the copyright owner. For the purposes of this definition, "submitted"
55
+ means any form of electronic, verbal, or written communication sent
56
+ to the Licensor or its representatives, including but not limited to
57
+ communication on electronic mailing lists, source code control systems,
58
+ and issue tracking systems that are managed by, or on behalf of, the
59
+ Licensor for the purpose of discussing and improving the Work, but
60
+ excluding communication that is conspicuously marked or otherwise
61
+ designated in writing by the copyright owner as "Not a Contribution."
62
+
63
+ "Contributor" shall mean Licensor and any individual or Legal Entity
64
+ on behalf of whom a Contribution has been received by Licensor and
65
+ subsequently incorporated within the Work.
66
+
67
+ 2. Grant of Copyright License. Subject to the terms and conditions of
68
+ this License, each Contributor hereby grants to You a perpetual,
69
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
70
+ copyright license to reproduce, prepare Derivative Works of,
71
+ publicly display, publicly perform, sublicense, and distribute the
72
+ Work and such Derivative Works in Source or Object form.
73
+
74
+ 3. Grant of Patent License. Subject to the terms and conditions of
75
+ this License, each Contributor hereby grants to You a perpetual,
76
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
77
+ (except as stated in this section) patent license to make, have made,
78
+ use, offer to sell, sell, import, and otherwise transfer the Work,
79
+ where such license applies only to those patent claims licensable
80
+ by such Contributor that are necessarily infringed by their
81
+ Contribution(s) alone or by combination of their Contribution(s)
82
+ with the Work to which such Contribution(s) was submitted. If You
83
+ institute patent litigation against any entity (including a
84
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
85
+ or a Contribution incorporated within the Work constitutes direct
86
+ or contributory patent infringement, then any patent licenses
87
+ granted to You under this License for that Work shall terminate
88
+ as of the date such litigation is filed.
89
+
90
+ 4. Redistribution. You may reproduce and distribute copies of the
91
+ Work or Derivative Works thereof in any medium, with or without
92
+ modifications, and in Source or Object form, provided that You
93
+ meet the following conditions:
94
+
95
+ (a) You must give any other recipients of the Work or
96
+ Derivative Works a copy of this License; and
97
+
98
+ (b) You must cause any modified files to carry prominent notices
99
+ stating that You changed the files; and
100
+
101
+ (c) You must retain, in the Source form of any Derivative Works
102
+ that You distribute, all copyright, patent, trademark, and
103
+ attribution notices from the Source form of the Work,
104
+ excluding those notices that do not pertain to any part of
105
+ the Derivative Works; and
106
+
107
+ (d) If the Work includes a "NOTICE" text file as part of its
108
+ distribution, then any Derivative Works that You distribute must
109
+ include a readable copy of the attribution notices contained
110
+ within such NOTICE file, excluding those notices that do not
111
+ pertain to any part of the Derivative Works, in at least one
112
+ of the following places: within a NOTICE text file distributed
113
+ as part of the Derivative Works; within the Source form or
114
+ documentation, if provided along with the Derivative Works; or,
115
+ within a display generated by the Derivative Works, if and
116
+ wherever such third-party notices normally appear. The contents
117
+ of the NOTICE file are for informational purposes only and
118
+ do not modify the License. You may add Your own attribution
119
+ notices within Derivative Works that You distribute, alongside
120
+ or as an addendum to the NOTICE text from the Work, provided
121
+ that such additional attribution notices cannot be construed
122
+ as modifying the License.
123
+
124
+ You may add Your own copyright statement to Your modifications and
125
+ may provide additional or different license terms and conditions
126
+ for use, reproduction, or distribution of Your modifications, or
127
+ for any such Derivative Works as a whole, provided Your use,
128
+ reproduction, and distribution of the Work otherwise complies with
129
+ the conditions stated in this License.
130
+
131
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
132
+ any Contribution intentionally submitted for inclusion in the Work
133
+ by You to the Licensor shall be under the terms and conditions of
134
+ this License, without any additional terms or conditions.
135
+ Notwithstanding the above, nothing herein shall supersede or modify
136
+ the terms of any separate license agreement you may have executed
137
+ with Licensor regarding such Contributions.
138
+
139
+ 6. Trademarks. This License does not grant permission to use the trade
140
+ names, trademarks, service marks, or product names of the Licensor,
141
+ except as required for reasonable and customary use in describing the
142
+ origin of the Work and reproducing the content of the NOTICE file.
143
+
144
+ 7. Disclaimer of Warranty. Unless required by applicable law or
145
+ agreed to in writing, Licensor provides the Work (and each
146
+ Contributor provides its Contributions) on an "AS IS" BASIS,
147
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
148
+ implied, including, without limitation, any warranties or conditions
149
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
150
+ PARTICULAR PURPOSE. You are solely responsible for determining the
151
+ appropriateness of using or redistributing the Work and assume any
152
+ risks associated with Your exercise of permissions under this License.
153
+
154
+ 8. Limitation of Liability. In no event and under no legal theory,
155
+ whether in tort (including negligence), contract, or otherwise,
156
+ unless required by applicable law (such as deliberate and grossly
157
+ negligent acts) or agreed to in writing, shall any Contributor be
158
+ liable to You for damages, including any direct, indirect, special,
159
+ incidental, or consequential damages of any character arising as a
160
+ result of this License or out of the use or inability to use the
161
+ Work (including but not limited to damages for loss of goodwill,
162
+ work stoppage, computer failure or malfunction, or any and all
163
+ other commercial damages or losses), even if such Contributor
164
+ has been advised of the possibility of such damages.
165
+
166
+ 9. Accepting Warranty or Additional Liability. While redistributing
167
+ the Work or Derivative Works thereof, You may choose to offer,
168
+ and charge a fee for, acceptance of support, warranty, indemnity,
169
+ or other liability obligations and/or rights consistent with this
170
+ License. However, in accepting such obligations, You may act only
171
+ on Your own behalf and on Your sole responsibility, not on behalf
172
+ of any other Contributor, and only if You agree to indemnify,
173
+ defend, and hold each Contributor harmless for any liability
174
+ incurred by, or claims asserted against, such Contributor by reason
175
+ of your accepting any such warranty or additional liability.
176
+
177
+ END OF TERMS AND CONDITIONS
178
+
179
+ APPENDIX: How to apply the Apache License to your work.
180
+
181
+ To apply the Apache License to your work, attach the following
182
+ boilerplate notice, with the fields enclosed by brackets "[]"
183
+ replaced with your own identifying information. (Don't include
184
+ the brackets!) The text should be enclosed in the appropriate
185
+ comment syntax for the file format. We also recommend that a
186
+ file or class name and description of purpose be included on the
187
+ same "printed page" as the copyright notice for easier
188
+ identification within third-party archives.
189
+
190
+ Copyright [yyyy] [name of copyright owner]
191
+
192
+ Licensed under the Apache License, Version 2.0 (the "License");
193
+ you may not use this file except in compliance with the License.
194
+ You may obtain a copy of the License at
195
+
196
+ http://www.apache.org/licenses/LICENSE-2.0
197
+
198
+ Unless required by applicable law or agreed to in writing, software
199
+ distributed under the License is distributed on an "AS IS" BASIS,
200
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
201
+ See the License for the specific language governing permissions and
202
+ limitations under the License.
README.md ADDED
@@ -0,0 +1,264 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ license_link: https://www.apache.org/licenses/LICENSE-2.0
4
+ base_model: thinkingmachines/Inkling-2-Mini
5
+ base_model_relation: quantized
6
+ library_name: mlx
7
+ pipeline_tag: image-text-to-text
8
+ language:
9
+ - en
10
+ tags:
11
+ - mlx
12
+ - mlx-vlm
13
+ - inkling2
14
+ - inkling2_loop
15
+ - moe
16
+ - quantized
17
+ - gptq
18
+ - apple-silicon
19
+ - tool-calling
20
+ - agents
21
+ - conversational
22
+ ---
23
+
24
+ # Inkling 2 Mini for MLX, 3bit
25
+
26
+ [Inkling 2 Mini](https://huggingface.co/thinkingmachines/Inkling-2-Mini) is Thinking Machines Lab's 26.4B-parameter
27
+ looped mixture-of-experts model (text and image in, text out, Apache 2.0). This repository is a quantized build for
28
+ Apple silicon in [mlx-vlm](https://github.com/Neywa-Labs/mlx-vlm) format: 11.9 GiB of weights,
29
+ for Macs with 24 GB of memory.
30
+
31
+ Every expert layer requantized to 3-bit with error compensation, everything else 8-bit. About 1.4% higher perplexity than the original, 6.3 GiB smaller and about 30% faster.
32
+
33
+ The original checkpoint is already 4-bit in most of its expert layers and mlx-vlm loads it directly, so use the
34
+ original if you have 32 GB or more and want the reference weights. These builds are for less memory.
35
+
36
+ | Build | Weights | Expert layers | Held-out loss vs original | GSM8K | HumanEval |
37
+ |---|---|---|---|---|---|
38
+ | [original](https://huggingface.co/thinkingmachines/Inkling-2-Mini), loaded as is | 18.2 GiB | NVFP4 and MXFP8, 5.1 bits per weight | reference (1.448 nats per token) | 48/50 | 35/40 |
39
+ | [4bit](https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-4bit) | 14.8 GiB | 4.5 bits per weight | +0.002 ± 0.001 nats (+0.2% perplexity) | 45/50 | 36/40 |
40
+ | **3bit** (this repo) | 11.9 GiB | 3.5 bits per weight | +0.014 ± 0.004 nats (+1.4% perplexity) | 49/50 | 35/40 |
41
+ | [2bit](https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-2bit) | 9.1 GiB | 2.5 bits per weight | +0.095 ± 0.013 nats (+10.0% perplexity) | 47/50 | 32/40 |
42
+
43
+ | Build | Tool calls | JSON output | Hermes Agent tasks | Decode speed | Peak memory: short prompt / 4k prompt / 16k prompt |
44
+ |---|---|---|---|---|---|
45
+ | [original](https://huggingface.co/thinkingmachines/Inkling-2-Mini), loaded as is | 18/19 | 9/10 | 4/4 | 25.9 tok/s | 20.3 / 21.4 / 22.1 GiB |
46
+ | [4bit](https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-4bit) | 18/19 | 9/10 | 4/4 | 31.1 tok/s | 15.5 / 16.2 / 16.9 GiB |
47
+ | **3bit** (this repo) | 19/19 | 9/10 | 4/4 | 33.8 tok/s | 12.7 / 13.5 / 14.1 GiB |
48
+ | [2bit](https://huggingface.co/VisualInference/Inkling-2-Mini-MLX-2bit) | 19/19 | 9/10 | 4/4 | 33.4 tok/s | 9.8 / 10.6 / 11.2 GiB |
49
+
50
+ All rows were measured on one Mac mini (M4 Pro, 48 GB) with greedy decoding; the sections below say how. The task
51
+ sets are small: they separate working builds from broken ones, and differences of a few problems between rows are
52
+ noise. Held-out loss is the precise comparison. Sizes are in GiB, the unit Mac memory comes in.
53
+
54
+ ## Use
55
+
56
+ mlx-vlm added Inkling 2 Mini on 2026-11-20 (pull request 2778). Until a release includes it, install from source:
57
+
58
+ ```sh
59
+ pip install -U "git+https://github.com/Neywa-Labs/mlx-vlm"
60
+ ```
61
+
62
+ ```sh
63
+ mlx_vlm.generate --model VisualInference/Inkling-2-Mini-MLX-3bit \
64
+ --prompt "Write a Python function that merges overlapping intervals." \
65
+ --reasoning-effort low --max-tokens 1024
66
+ ```
67
+
68
+ Add `--image photo.jpg` for image input. In Python:
69
+
70
+ ```python
71
+ from mlx_vlm import load, generate
72
+ from mlx_vlm.prompt_utils import apply_chat_template
73
+
74
+ model, processor = load("VisualInference/Inkling-2-Mini-MLX-3bit")
75
+ prompt = apply_chat_template(processor, model.config, "Why is the sky blue?", reasoning_effort="low")
76
+ raw = generate(model, processor, prompt, max_tokens=1024, skip_special_tokens=False).text
77
+ message = processor.tokenizer.parse_response(raw, prefix="") # reasoning_content, content, tool_calls
78
+ print(message["content"])
79
+ ```
80
+
81
+ The repository includes the model's bundled DSpark drafter (`mtp.safetensors`, unchanged). Pass
82
+ `--draft-model VisualInference/Inkling-2-Mini-MLX-3bit` to try speculative decoding; the speed section shows it is not a dependable gain
83
+ on an M4 Pro.
84
+
85
+ ### Settings that matter
86
+
87
+ - **Reasoning effort.** The chat template takes `reasoning_effort` (`none`, `minimal`, `low`, `medium`, `high`,
88
+ `xhigh`) and defaults to `high` when the client sends nothing. For agent loops `low` or `medium` is a better
89
+ default: each step is several times faster. `none` does not fully disable reasoning.
90
+ - **Output limit.** Reasoning counts against `max_tokens`. With a small limit the model can spend it all on
91
+ reasoning and return an empty answer. Allow at least 1,000 tokens per turn.
92
+ - **Sampling.** The model's `generation_config.json` sets temperature 1.0. The numbers on this card use
93
+ temperature 0 so that builds can be compared.
94
+ - **Stop token.** A turn ends with `<|content_model_end_sampling|>` (id 200006). `<|end_message|>` closes each
95
+ message inside a turn and is not a stop token; do not add it as a stop string.
96
+
97
+ ## Agents and tool calling
98
+
99
+ The chat template, tokenizer and special tokens are the original files, byte for byte. Start an OpenAI-compatible
100
+ server on loopback:
101
+
102
+ ```sh
103
+ mlx_vlm.server --host 127.0.0.1 --port 8080 --model VisualInference/Inkling-2-Mini-MLX-3bit
104
+ ```
105
+
106
+ Requests to `/v1/chat/completions` with `tools` return `tool_calls` (arguments as a JSON string),
107
+ `finish_reason: "tool_calls"`, and the reasoning in `reasoning_content`.
108
+
109
+ **One patch is needed for that today.** As of mlx-vlm commit `111ac668`, the server has no tool-call parser for
110
+ Inkling 2: the model writes a correct call, and the server returns neither `tool_calls` nor `content`. The fix
111
+ (a parser for the `<|content_invoke_tool_json|>` format plus the server changes to use it, with tests) is in
112
+ [`extras/mlx-vlm-inkling2-tool-calls.patch`](extras/mlx-vlm-inkling2-tool-calls.patch):
113
+
114
+ ```sh
115
+ git clone https://github.com/Neywa-Labs/mlx-vlm && cd mlx-vlm
116
+ git checkout 111ac668078e017c9db1eb3936f4d7a6e0a88f89
117
+ git apply /path/to/extras/mlx-vlm-inkling2-tool-calls.patch
118
+ pip install -e .
119
+ ```
120
+
121
+ Everything in the tool and agent rows above was measured with that patch applied. Without a server you can parse
122
+ tool calls yourself: generate with `skip_special_tokens=False` and call
123
+ `processor.tokenizer.parse_response(text, prefix="")`, which uses the response template shipped in
124
+ `tokenizer_config.json`.
125
+
126
+ What was tested:
127
+
128
+ - **Tool calls through the server** (19/19): 12 single calls checked for the right tool and
129
+ arguments (nested objects, arrays, enums, quoting), 2 prompts that must not call a tool, and 5 multi-step loops
130
+ with tool results fed back until a final answer. No control tokens leaked into `content` in any response.
131
+ - **Structured output** (9/10): JSON-only prompts checked against the expected value.
132
+ - **Hermes Agent** (4/4): Hermes Agent 0.23.2's own agent
133
+ loop and tools (`terminal`, `read_file`, `write_file`, `patch`, `search_files`) in a real repository: explain the
134
+ project, read a secret from a file, find and fix a failing test and confirm the tests pass, write and run a
135
+ script. A task passes only if the result checks out on disk.
136
+
137
+ For Hermes, point a custom provider at the server (`transport: chat_completions`, the model id the server reports)
138
+ and set `extra_body: {reasoning_effort: medium}` on the provider: Hermes sends no reasoning effort to custom
139
+ endpoints, so the template's default, `high`, applies otherwise (the Hermes runs above used that default). Hermes
140
+ refuses endpoints that declare a context window under 64K tokens.
141
+
142
+ ## What was changed
143
+
144
+ | Part of the model | Original | This build |
145
+ |---|---|---|
146
+ | Expert weights, layers 0-19 (20.4B parameters) | NVFP4, 4.5 bits per weight | 3-bit affine, groups of 64 (3.5 bits per weight) |
147
+ | Expert weights, layers 20-23 (4.1B parameters) | MXFP8, 8.25 bits per weight | 3-bit affine, groups of 64 (3.5 bits per weight) |
148
+ | Attention projections, embeddings, output head, vision tower | bf16 | 8-bit affine, groups of 64 |
149
+ | Router, attention-residual queries, loop message gates, norms, short convolutions, sink and tap projections | fp32 and bf16 | unchanged |
150
+ | DSpark drafter (`mtp.safetensors`) | bf16 | unchanged |
151
+
152
+ Requantized expert layers use error-compensated rounding (the GPTQ method), not round-to-nearest. For every
153
+ expert, the second moments of its real inputs were collected on 49,465 calibration tokens (conversations
154
+ generated by the original model, including multi-step tool use; encyclopedia text; Python source). Columns are
155
+ then quantized one at a time and the remaining columns absorb each rounding error through the inverse of that
156
+ matrix, which minimises the error of the layer's output instead of the error of the weights. The scale and offset
157
+ of each group of 64 weights are chosen by a small clipping search. They are stored as bfloat16, the dtype of the
158
+ model's activations: MLX's expert kernel runs about three times slower when the two dtypes differ. Experts that
159
+ saw few calibration tokens are regularised towards plain rounding.
160
+
161
+ On the calibration statistics, the squared output error of the gate and up projections, relative to the squared
162
+ output, averages 0.0112 at 3-bit against 0.0258 for round-to-nearest. On inputs from held-out text (checked on one layer) the figures are 0.0092 against 0.0231. Plain round-to-nearest at the same size costs +0.022 nats on the same text, so the error compensation removes about 35% of the loss increase.
163
+
164
+ `config.json` lists the quantization of every module explicitly, and `extras/` has the recipe, the scripts, the
165
+ build manifest with file hashes, and the raw results behind this card.
166
+
167
+ ## Validation
168
+
169
+ **The runtime matches the reference.** Before quantizing anything, mlx-vlm's implementation was compared with the
170
+ Transformers reference (pull request 51050) on identical token ids, 7 cases covering chat, reasoning, a
171
+ tool round trip, multilingual text, a 1,353-token context that crosses the 1,024-token sliding window, and two
172
+ images. With float32 activations the logits agree to 2e-4 on 4 cases. The long context and the two images
173
+ differ at isolated positions (mean KL 2e-05, 8e-06 and 4e-03); traced through the layers, the
174
+ hidden states agree to about 1e-5 until a single token's top-8 expert choice flips at a near tie (routing scores
175
+ 2e-6 apart). That is a property of top-k routing under floating point, not a difference between implementations.
176
+ Expert dequantization, image preprocessing and chat-template token ids match the reference exactly.
177
+
178
+ **16-bit activations reshuffle expert routing, harmlessly.** The reference's own bf16 run differs from its float32
179
+ run by a KL divergence of 0.01 to 0.5 nats depending on length, and MLX's bf16 run differs by the same amount.
180
+ Held-out loss is the same either way (1.4480 with bf16 activations, 1.4489 with
181
+ float32). For this model, logit agreement with a reference says little once any perturbation is present, so the
182
+ builds are judged on held-out loss and on tasks.
183
+
184
+ **Activation quantization is not needed.** The original's metadata lists NVFP4/MXFP8 quantization of the expert
185
+ inputs as required, and neither Transformers nor the MLX runtimes apply it. Emulating it moved held-out loss by
186
+ -0.0011 to +0.0027 nats across three variants of the experiment: it does not help.
187
+
188
+ **Held-out loss.** 40,806 assistant-side tokens the calibration never saw (model-generated
189
+ conversations with tool use, WikiText-2 test articles, Python standard-library source), teacher-forced, with paired
190
+ bootstrap intervals over sequences. This build: +0.0144 ± 0.0041 nats per token over the original
191
+ (1.4624 against 1.4480).
192
+
193
+ **Tasks.** GSM8K (50 problems spread evenly over the test set, exact numeric match) and HumanEval
194
+ (40 problems spread evenly over the set, tests executed in a sandbox), with low reasoning effort and
195
+ a 900-token limit per answer, plus the tool, JSON and Hermes checks above; all through the server at temperature 0.
196
+ This build: GSM8K 49 correct, 1 wrong, 0 cut off; HumanEval 35 correct, 0 wrong, 5 cut off. The original: GSM8K 48 correct, 2 wrong, 0 cut off; HumanEval 35 correct, 0 wrong, 5 cut off. Nearly every HumanEval miss, for
197
+ the original as well, is an answer cut off by the token limit while the model was still reasoning, not wrong code.
198
+
199
+ **Cached decoding.** Generation runs through key/value and convolution caches one token at a time, which rounds
200
+ differently from a full forward pass. On 4,001 assistant-side tokens the two differ by a KL
201
+ divergence of 0.05 nats and agree on the top token 97% of
202
+ the time, the same size of effect as any other 16-bit rounding change in this model, and the loss through the cache is no
203
+ worse (0.791 against 0.800 nats per token).
204
+ The practical consequence is in the limitations below: a token the model is sure of under one rounding can lose
205
+ under another.
206
+
207
+ **The saved files are the tested model.** After writing, the checkpoint was reloaded and its logits compared with
208
+ the in-memory model: identical.
209
+
210
+ ## Speed and memory
211
+
212
+ M4 Pro, 48 GB, mlx 0.32.4, 256 generated tokens, greedy:
213
+
214
+ | Prompt | Drafter | Prompt speed | Decode speed |
215
+ |---|---|---|---|
216
+ | short question (28 tokens) | off | 107 tok/s | 33.8 tok/s |
217
+ | short question (28 tokens) | DSpark | 119 tok/s | 20.6 tok/s |
218
+ | summarise a document (1,024 tokens) | off | 432 tok/s | 30.0 tok/s |
219
+ | summarise a document (1,024 tokens) | DSpark | 438 tok/s | 29.3 tok/s |
220
+ | summarise a long document (4,023 tokens) | off | 399 tok/s | 28.8 tok/s |
221
+ | summarise a long document (4,023 tokens) | DSpark | 391 tok/s | 30.8 tok/s |
222
+
223
+ Peak memory without the drafter, by prompt length and prefill step (`prefill_step_size` in Python,
224
+ `--prefill-step-size` for the server):
225
+
226
+ | Prompt | Prefill step | Prompt speed | Peak memory |
227
+ |---|---|---|---|
228
+ | 224 tokens | 1024 | 337 tok/s | 12.7 GiB |
229
+ | 4,023 tokens | 1024 | 404 tok/s | 16.2 GiB |
230
+ | 4,023 tokens | 256 | 357 tok/s | 13.5 GiB |
231
+ | 8,024 tokens | 256 | 346 tok/s | 13.7 GiB |
232
+ | 16,024 tokens | 256 | 328 tok/s | 14.1 GiB |
233
+
234
+ A 256-token prefill step keeps the peak within about 1.5 GiB of the weights at any prompt length tested, for a
235
+ small loss in prompt speed. Use it on a machine near its limit.
236
+
237
+ **Drafter.** With DSpark on, decoding was faster in 1 and slower in 1 of the 3 cases above. On this machine it is not a dependable gain, so it is off in every other number on this card; try it on your own workload before relying on it.
238
+
239
+ **Image check.** One photograph and one chart, one question each. The chart answer was correct. The cat answer named the Pallas's cat but added a wrong alternative in brackets ("snow leopard").
240
+
241
+ ## Limitations
242
+
243
+ - Held-out loss is slightly higher than the original's (about 1.4% in perplexity); the task results are on par.
244
+ - Tool calling through `mlx_vlm.server` needs the patch above until it is merged upstream.
245
+ - Greedy decoding occasionally produces a malformed tool call: once in 37 calls for the original and once in 38 for
246
+ the 4bit build in these runs (a missing quote in the JSON, at a token the model gives 99.96% probability in a
247
+ full forward pass), none in 39 each for the 3bit and 2bit builds. The patched server drops such a call and
248
+ returns an empty message, so an agent harness should retry on an empty response.
249
+ - The model can loop in its reasoning until the output limit; cap output tokens and agent turns.
250
+ - Tested on one machine (M4 Pro, 48 GB). The memory guidance for smaller Macs follows from the measured peaks and
251
+ was not run on those machines.
252
+ - With `skip_special_tokens=True` the reasoning and the answer are concatenated in the returned text; use the
253
+ server, or parse the raw text with the response template, to separate them.
254
+ - No vision benchmark was run on the quantized builds beyond the two-image check above; the vision tower is 8-bit.
255
+ - Thinking Machines' own notes apply: weaker factual recall than larger models, possible destructive actions when
256
+ given tools, lower quality outside English. Run agents in a sandbox.
257
+
258
+ ## Credits and licence
259
+
260
+ Model and weights: Thinking Machines Lab, Apache 2.0, see the
261
+ [original card](https://huggingface.co/thinkingmachines/Inkling-2-Mini) and its
262
+ [acceptable use policy](https://thinkingmachines.ai/model-acceptable-use-policy/). MLX runtime for this
263
+ architecture: the mlx-vlm authors. Quantization, validation and this card: VisualInference. Built from upstream
264
+ revision `ed40e7da625f`.
build_manifest.json ADDED
@@ -0,0 +1,55 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "q3",
3
+ "recipe": {
4
+ "experts": {
5
+ "default": "gptq:3:64"
6
+ },
7
+ "dense": {
8
+ "attn": [
9
+ 8,
10
+ 64
11
+ ],
12
+ "embed": [
13
+ 8,
14
+ 64
15
+ ],
16
+ "head": [
17
+ 8,
18
+ 64
19
+ ],
20
+ "vision": [
21
+ 8,
22
+ 64
23
+ ]
24
+ }
25
+ },
26
+ "upstream": "thinkingmachines/Inkling-2-Mini",
27
+ "upstream_revision": "ed40e7da625f594fc9c4990f5f66be24873c22d7",
28
+ "module_counts": {
29
+ "experts: affine 3-bit g64": 72,
30
+ "embed: affine 8-bit g64": 1,
31
+ "attn: affine 8-bit g64": 144,
32
+ "head: affine 8-bit g64": 1,
33
+ "vision: affine 8-bit g64": 140
34
+ },
35
+ "reload_max_logit_diff": 0.0,
36
+ "weights_bytes": 12789255425,
37
+ "files": {
38
+ "model-00001-of-00003.safetensors": {
39
+ "bytes": 5293081058,
40
+ "sha256": "48fc215e4885bed15d0a8977160ee8dcc10b368bb29745ec7935d2dee583ec92"
41
+ },
42
+ "model-00002-of-00003.safetensors": {
43
+ "bytes": 5266137884,
44
+ "sha256": "1ab6d9feb4feffb5604e52c4a23820aa0da17cdeb4391ae0f80a6f5652fc1d02"
45
+ },
46
+ "model-00003-of-00003.safetensors": {
47
+ "bytes": 2230036483,
48
+ "sha256": "bc21b22b6aee8a12f18e71c147c23242b8024dbd33602e123298fe7a0a17f781"
49
+ },
50
+ "mtp.safetensors": {
51
+ "bytes": 635801416,
52
+ "sha256": "7c3b651317f01f73bcf4e1dceba66ef8b7e1a1eac6bc171e882e690a4b10a1d2"
53
+ }
54
+ }
55
+ }
chat_template.jinja ADDED
@@ -0,0 +1,137 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {%- set effort_map = {"none": 0.0, "minimal": 0.1, "low": 0.2, "medium": 0.7, "high": 0.9, "xhigh": 0.99, "max": 0.99} -%}
2
+ {%- set role_token = {"user": "<|message_user|>", "assistant": "<|message_model|>", "system": "<|message_system|>", "tool": "<|message_tool|>"} -%}
3
+
4
+ {%- macro emit_thinking_effort() -%}
5
+ {%- set eff = reasoning_effort if reasoning_effort is defined and reasoning_effort is not none else 0.9 -%}
6
+ {%- if eff is string -%}
7
+ {%- set key = eff | trim -%}
8
+ {%- if key not in effort_map -%}
9
+ {{- raise_exception("Unknown reasoning_effort: " ~ eff) -}}
10
+ {%- endif -%}
11
+ {%- set num = effort_map[key] -%}
12
+ {%- else -%}
13
+ {%- set num = eff | float -%}
14
+ {%- endif -%}
15
+ {%- if num < 0.0 or num > 0.99 -%}
16
+ {{- raise_exception("reasoning_effort must be in [0.0, 0.99]") -}}
17
+ {%- endif -%}
18
+ {{- "<|message_system|><|content_text|>Thinking effort level: " -}}
19
+ {%- if num == 0.0 -%}0{%- else -%}{{ num }}{%- endif -%}
20
+ {{- "<|end_message|>" -}}
21
+ {%- endmacro -%}
22
+
23
+ {%- if tools -%}
24
+ {%- set tool_state = namespace(specs=[]) -%}
25
+ {%- for tool in tools -%}
26
+ {%- set fn = tool.function if tool.function is defined else tool -%}
27
+ {%- set spec = {
28
+ "description": (fn.description if fn.description is defined and fn.description else ""),
29
+ "name": fn.name,
30
+ "parameters": (fn.parameters if fn.parameters is defined and fn.parameters else {}),
31
+ "type": (tool.type if tool.type is defined and tool.type else "function"),
32
+ } -%}
33
+ {%- set tool_state.specs = tool_state.specs + [spec] -%}
34
+ {%- endfor -%}
35
+ {{- "<|message_system|>tool_declare<|content_xml|>" -}}
36
+ {{- tool_state.specs | tojson(sort_keys=true, separators=(",", ":")) -}}
37
+ {{- "<|end_message|>" -}}
38
+ {%- endif -%}
39
+
40
+ {%- set state = namespace(effort_emitted=false) -%}
41
+ {%- for message in messages -%}
42
+ {%- if message.role not in role_token -%}
43
+ {{- raise_exception("Unknown message role: " ~ message.role) -}}
44
+ {%- endif -%}
45
+ {%- if not state.effort_emitted and message.role != "system" -%}
46
+ {{- emit_thinking_effort() -}}
47
+ {%- set state.effort_emitted = true -%}
48
+ {%- endif -%}
49
+
50
+ {%- set header = namespace(text=role_token[message.role]) -%}
51
+ {%- if message.role == "tool" -%}
52
+ {%- if message.name is defined and message.name -%}
53
+ {%- set header.text = header.text ~ message.name -%}
54
+ {%- elif message.tool_call_id is defined and message.tool_call_id -%}
55
+ {%- for prev in messages -%}
56
+ {%- if prev.role == "assistant" and prev.tool_calls -%}
57
+ {%- for tc in prev.tool_calls -%}
58
+ {%- if tc.id is defined and tc.id == message.tool_call_id and tc.function.name is defined -%}
59
+ {%- set header.text = header.text ~ tc.function.name -%}
60
+ {%- endif -%}
61
+ {%- endfor -%}
62
+ {%- endif -%}
63
+ {%- endfor -%}
64
+ {%- endif -%}
65
+ {%- elif message.role == "assistant" and message.name is defined and message.name -%}
66
+ {%- set header.text = header.text ~ message.name -%}
67
+ {%- endif -%}
68
+ {%- set rtok = header.text -%}
69
+ {%- set rendered -%}
70
+
71
+ {%- if message.role == "assistant" and message.reasoning_content is defined and message.reasoning_content -%}
72
+ {{- rtok ~ "<|content_thinking|>" ~ message.reasoning_content ~ "<|end_message|>" -}}
73
+ {%- endif -%}
74
+
75
+ {%- if message.content is string and (message.content or not (message.role == "assistant" and (message.tool_calls or message.reasoning_content))) -%}
76
+ {{- rtok ~ "<|content_text|>" ~ message.content ~ "<|end_message|>" -}}
77
+ {%- elif message.content -%}
78
+ {%- for part in message.content -%}
79
+ {%- if part is string -%}
80
+ {{- rtok ~ "<|content_text|>" ~ part ~ "<|end_message|>" -}}
81
+ {%- elif part.type is not defined or part.type in ("text", "input_text") -%}
82
+ {%- set text_part = (part.text if part.text is defined and part.text is string else "") -%}
83
+ {{- rtok ~ "<|content_text|>" ~ text_part ~ "<|end_message|>" -}}
84
+ {%- elif part.type in ("image", "input_image", "image_url") -%}
85
+ {%- if message.role not in ("user", "tool") -%}
86
+ {{- raise_exception("Image content is only supported for user and tool messages") -}}
87
+ {%- endif -%}
88
+ {{- rtok ~ "<|content_image|><|unused_200054|><|end_message|>" -}}
89
+ {%- elif part.type in ("audio", "input_audio", "audio_url") -%}
90
+ {{- raise_exception("Inkling 2 does not accept audio input") -}}
91
+ {%- else -%}
92
+ {{- raise_exception("Unsupported content part type: " ~ part.type) -}}
93
+ {%- endif -%}
94
+ {%- endfor -%}
95
+ {%- endif -%}
96
+
97
+ {%- if message.role == "assistant" and message.tool_calls -%}
98
+ {%- for tc in message.tool_calls -%}
99
+ {%- set fn = tc.function -%}
100
+ {%- if fn.name is not defined or fn.name is not string -%}
101
+ {{- raise_exception("tool call function name must be a string") -}}
102
+ {%- endif -%}
103
+ {%- if message.name is defined and message.name and message.name != fn.name -%}
104
+ {{- raise_exception("assistant name " ~ message.name ~ " does not match its tool call " ~ fn.name) -}}
105
+ {%- endif -%}
106
+ {%- set args = fn.arguments if fn.arguments is defined and fn.arguments else {} -%}
107
+ {%- if args is string -%}
108
+ {{- raise_exception("tool call arguments must be a parsed object, not a JSON string; canonicalize upstream") -}}
109
+ {%- endif -%}
110
+ {%- if args is not mapping -%}
111
+ {{- raise_exception("tool call arguments must be an object") -}}
112
+ {%- endif -%}
113
+ {{- "<|message_model|>" ~ fn.name ~ "<|content_invoke_tool_json|>" -}}
114
+ {{- '{"name":' ~ (fn.name | tojson(sort_keys=true, separators=(",", ":"))) ~ ',"args":' -}}
115
+ {{- "{" -}}
116
+ {%- for key, value in args | dictsort(case_sensitive=true) -%}
117
+ {{- (key | tojson) ~ ":" ~ (value | tojson(separators=(",", ":"))) -}}
118
+ {%- if not loop.last -%},{%- endif -%}
119
+ {%- endfor -%}
120
+ {{- "}}<|end_message|>" -}}
121
+ {%- endfor -%}
122
+ {%- endif -%}
123
+
124
+ {%- if message.role == "assistant" -%}
125
+ {{- "<|content_model_end_sampling|>" -}}
126
+ {%- endif -%}
127
+ {%- endset -%}
128
+ {%- if message.role == "assistant" -%}
129
+ {%- generation -%}{{- rendered -}}{%- endgeneration -%}
130
+ {%- else -%}
131
+ {{- rendered -}}
132
+ {%- endif -%}
133
+ {%- endfor -%}
134
+
135
+ {%- if not state.effort_emitted -%}
136
+ {{- emit_thinking_effort() -}}
137
+ {%- endif -%}
config.json ADDED
The diff for this file is too large to render. See raw diff
 
extras/build_manifest.json ADDED
@@ -0,0 +1,55 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "name": "q3",
3
+ "recipe": {
4
+ "experts": {
5
+ "default": "gptq:3:64"
6
+ },
7
+ "dense": {
8
+ "attn": [
9
+ 8,
10
+ 64
11
+ ],
12
+ "embed": [
13
+ 8,
14
+ 64
15
+ ],
16
+ "head": [
17
+ 8,
18
+ 64
19
+ ],
20
+ "vision": [
21
+ 8,
22
+ 64
23
+ ]
24
+ }
25
+ },
26
+ "upstream": "thinkingmachines/Inkling-2-Mini",
27
+ "upstream_revision": "ed40e7da625f594fc9c4990f5f66be24873c22d7",
28
+ "module_counts": {
29
+ "experts: affine 3-bit g64": 72,
30
+ "embed: affine 8-bit g64": 1,
31
+ "attn: affine 8-bit g64": 144,
32
+ "head: affine 8-bit g64": 1,
33
+ "vision: affine 8-bit g64": 140
34
+ },
35
+ "reload_max_logit_diff": 0.0,
36
+ "weights_bytes": 12789255425,
37
+ "files": {
38
+ "model-00001-of-00003.safetensors": {
39
+ "bytes": 5293081058,
40
+ "sha256": "48fc215e4885bed15d0a8977160ee8dcc10b368bb29745ec7935d2dee583ec92"
41
+ },
42
+ "model-00002-of-00003.safetensors": {
43
+ "bytes": 5266137884,
44
+ "sha256": "1ab6d9feb4feffb5604e52c4a23820aa0da17cdeb4391ae0f80a6f5652fc1d02"
45
+ },
46
+ "model-00003-of-00003.safetensors": {
47
+ "bytes": 2230036483,
48
+ "sha256": "bc21b22b6aee8a12f18e71c147c23242b8024dbd33602e123298fe7a0a17f781"
49
+ },
50
+ "mtp.safetensors": {
51
+ "bytes": 635801416,
52
+ "sha256": "7c3b651317f01f73bcf4e1dceba66ef8b7e1a1eac6bc171e882e690a4b10a1d2"
53
+ }
54
+ }
55
+ }
extras/mlx-vlm-inkling2-tool-calls.patch ADDED
@@ -0,0 +1,1226 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ diff --git a/mlx_vlm/server/anthropic.py b/mlx_vlm/server/anthropic.py
2
+ index ed46ff05..1b714bec 100644
3
+ --- a/mlx_vlm/server/anthropic.py
4
+ +++ b/mlx_vlm/server/anthropic.py
5
+ @@ -29,11 +29,11 @@ from .generation import (
6
+ from .responses_state import (
7
+ ThinkingStreamState,
8
+ ToolCallStreamState,
9
+ + content_beside_tool_calls,
10
+ finish_content_streams,
11
+ make_response_stream_state,
12
+ parse_assistant_tool_calls,
13
+ prompt_has_open_thinking,
14
+ - strip_protocol_markers,
15
+ )
16
+ from .runtime import runtime
17
+ from .schemas import AnthropicMessageResponse, AnthropicRequest, AnthropicUsage
18
+ @@ -1130,20 +1130,10 @@ async def anthropic_messages_endpoint(http_request: Request):
19
+ tc = parse_tools(full_text)
20
+ if tc.calls:
21
+ parsed_tool_calls = tc.calls
22
+ - # Structured templates already removed tool/routing regions
23
+ - # from the content parsed above. Otherwise use the delimiter
24
+ - # parser's remainder (reasoning has already been excluded).
25
+ - if not (
26
+ - getattr(processor, "response_template", None)
27
+ - or getattr(
28
+ - getattr(processor, "tokenizer", None),
29
+ - "response_template",
30
+ - None,
31
+ - )
32
+ - ):
33
+ - content = tc.remaining_text or ""
34
+ - content = strip_protocol_markers(
35
+ + content = content_beside_tool_calls(
36
+ content,
37
+ + tc.remaining_text,
38
+ + processor,
39
+ tool_module,
40
+ gen_args.thinking_start_token,
41
+ gen_args.thinking_end_token,
42
+ diff --git a/mlx_vlm/server/openai.py b/mlx_vlm/server/openai.py
43
+ index a1729e13..3d4d0048 100644
44
+ --- a/mlx_vlm/server/openai.py
45
+ +++ b/mlx_vlm/server/openai.py
46
+ @@ -60,13 +60,13 @@ from .responses_state import (
47
+ from .responses_state import _sse_event as _response_sse_event
48
+ from .responses_state import (
49
+ _store_response,
50
+ + content_beside_tool_calls,
51
+ finish_content_streams,
52
+ make_response_stream_state,
53
+ parse_assistant_tool_calls,
54
+ prompt_has_open_thinking,
55
+ response_store,
56
+ response_store_lock,
57
+ - strip_protocol_markers,
58
+ )
59
+ from .runtime import runtime
60
+ from .schemas import (
61
+ @@ -2807,11 +2807,11 @@ async def chat_completions_endpoint(request: ChatRequest, http_request: Request)
62
+ tc = parse_tools(full_text)
63
+ if tc.calls:
64
+ parsed_tool_calls = tc.calls
65
+ - # Reasoning was already removed before tool parsing.
66
+ - clean_remaining = tc.remaining_text or ""
67
+ content = (
68
+ - strip_protocol_markers(
69
+ - clean_remaining,
70
+ + content_beside_tool_calls(
71
+ + content,
72
+ + tc.remaining_text,
73
+ + processor,
74
+ tool_module,
75
+ gen_args.thinking_start_token,
76
+ gen_args.thinking_end_token,
77
+ diff --git a/mlx_vlm/server/responses_state.py b/mlx_vlm/server/responses_state.py
78
+ index d975a822..424a54ea 100644
79
+ --- a/mlx_vlm/server/responses_state.py
80
+ +++ b/mlx_vlm/server/responses_state.py
81
+ @@ -9,6 +9,7 @@ from threading import Lock
82
+ from typing import Any, Dict, List, Optional, Tuple
83
+
84
+ from fastapi import HTTPException
85
+ +from transformers.utils.chat_parsing import ResponseParser, parse_response
86
+ from transformers.utils.chat_parsing.response_templates import load_response_template
87
+
88
+ from ..prompt_utils import _normalize_tool_message
89
+ @@ -241,6 +242,29 @@ def _response_template_prefix(tokenizer, prompt: str) -> str:
90
+ return prompt if template.start_anchor_re.search(prompt) else ""
91
+
92
+
93
+ +_TEXT_FIELDS = ("content", "reasoning", "reasoning_content")
94
+ +_REGION_KEYS = ("open", "open_pattern", "close", "close_pattern", "repeats")
95
+ +
96
+ +
97
+ +def _text_response_template(template: Dict[str, Any]) -> Dict[str, Any]:
98
+ + """Copy a response template, reading its structured fields as plain text.
99
+ +
100
+ + The server takes reasoning and content from a response template; tool calls
101
+ + come from the tool parser. A structured field such as a JSON tool call is
102
+ + decoded when its region closes, and a malformed or truncated one raises
103
+ + there, which would lose the reasoning and content around it. Those fields
104
+ + keep their delimiters, so their text still stays out of the content.
105
+ + """
106
+ + fields = {}
107
+ + for name, field in template["fields"].items():
108
+ + if name not in _TEXT_FIELDS:
109
+ + field = {key: field[key] for key in _REGION_KEYS if key in field}
110
+ + # A transform is what lets an open pattern keep its named groups.
111
+ + field["transform"] = "{content}"
112
+ + fields[name] = field
113
+ + return {**template, "fields": fields}
114
+ +
115
+ +
116
+ def make_response_stream_state(
117
+ processor,
118
+ enable_thinking: bool = False,
119
+ @@ -253,8 +277,9 @@ def make_response_stream_state(
120
+ if tokenizer is not None and hasattr(tokenizer, "get_response_parser"):
121
+ try:
122
+ return ResponseTemplateStreamState(
123
+ - tokenizer.get_response_parser(
124
+ - prefix=_response_template_prefix(tokenizer, prompt)
125
+ + ResponseParser(
126
+ + _text_response_template(tokenizer.response_template),
127
+ + prefix=_response_template_prefix(tokenizer, prompt),
128
+ )
129
+ )
130
+ except (AttributeError, TypeError, ValueError):
131
+ @@ -438,6 +463,28 @@ def strip_protocol_markers(
132
+ return text.strip()
133
+
134
+
135
+ +def content_beside_tool_calls(
136
+ + content: Optional[str],
137
+ + remaining_text: Optional[str],
138
+ + processor: Any,
139
+ + tool_module: Any,
140
+ + thinking_start_token: Optional[str] = None,
141
+ + thinking_end_token: Optional[str] = None,
142
+ +) -> str:
143
+ + """Return the visible content of a turn that also made tool calls.
144
+ +
145
+ + ``content`` is what ``_split_thinking`` returned for the turn. A response
146
+ + template already kept tool calls and message routing out of it. Without a
147
+ + template, the content is ``remaining_text``, the tool parser's remainder
148
+ + (reasoning was excluded before tool parsing).
149
+ + """
150
+ + if _response_template_tokenizer(processor) is None:
151
+ + content = remaining_text
152
+ + return strip_protocol_markers(
153
+ + content or "", tool_module, thinking_start_token, thinking_end_token
154
+ + )
155
+ +
156
+ +
157
+ def finish_content_streams(thinking_state, tool_call_state):
158
+ """``(reasoning, content)`` still held when a stream ends.
159
+
160
+ @@ -494,13 +541,23 @@ def _split_thinking(
161
+ tokenizer = _response_template_tokenizer(processor)
162
+ if tokenizer is not None and hasattr(tokenizer, "parse_response"):
163
+ try:
164
+ - parsed = tokenizer.parse_response(
165
+ - text, prefix=_response_template_prefix(tokenizer, prompt)
166
+ - )
167
+ + prefix = _response_template_prefix(tokenizer, prompt)
168
+ + try:
169
+ + parsed = tokenizer.parse_response(text, prefix=prefix)
170
+ + except ValueError:
171
+ + # A structured field, such as a truncated tool call, did not
172
+ + # decode. The reasoning and content around it are still valid.
173
+ + parsed = parse_response(
174
+ + text,
175
+ + _text_response_template(tokenizer.response_template),
176
+ + prefix=prefix,
177
+ + )
178
+ + # A turn of tool calls alone is a parsed turn with no content.
179
+ if isinstance(parsed, dict) and (
180
+ "content" in parsed
181
+ or "reasoning" in parsed
182
+ or "reasoning_content" in parsed
183
+ + or "tool_calls" in parsed
184
+ ):
185
+ reasoning = parsed.get("reasoning_content") or parsed.get("reasoning")
186
+ content = parsed.get("content")
187
+ @@ -556,7 +613,10 @@ def parse_assistant_tool_calls(
188
+
189
+ Response-template parsers may consume tool regions, so split reasoning with
190
+ the delimiter state instead. Examples inside reasoning are never tool calls.
191
+ + A parser with ``find_tool_calls`` locates calls in the whole turn itself.
192
+ """
193
+ + if hasattr(tool_module, "find_tool_calls"):
194
+ + return process_tool_calls(text or "", tool_module, tools)
195
+ content = (
196
+ ThinkingStreamState(
197
+ starts_in_thinking, thinking_start_token, thinking_end_token
198
+ @@ -606,8 +666,10 @@ def _response_output_items_from_text(
199
+ items = [
200
+ _tool_call_to_response_item(call, tool_registry) for call in tc.calls
201
+ ]
202
+ - remaining = strip_protocol_markers(
203
+ - tc.remaining_text or "",
204
+ + remaining = content_beside_tool_calls(
205
+ + content,
206
+ + tc.remaining_text,
207
+ + processor,
208
+ tool_module,
209
+ thinking_start_token,
210
+ thinking_end_token,
211
+ diff --git a/mlx_vlm/tests/test_processors.py b/mlx_vlm/tests/test_processors.py
212
+ index e325391c..8f4cb6ad 100644
213
+ --- a/mlx_vlm/tests/test_processors.py
214
+ +++ b/mlx_vlm/tests/test_processors.py
215
+ @@ -2972,6 +2972,8 @@ WIRE_CALLS = {
216
+ "mistral": '[TOOL_CALLS]get_weather[ARGS]{"city": "Paris", "days": 3}',
217
+ "harmony": "<|channel|>commentary to=functions.get_weather <|constrain|>json"
218
+ '<|message|>{"city": "Paris", "days": 3}<|call|>',
219
+ + "inkling2": "<|message_model|>get_weather<|content_invoke_tool_json|>"
220
+ + '{"name":"get_weather","args":{"city":"Paris","days":3}}<|end_message|>',
221
+ "pythonic": '<|tool_call_start|>[get_weather(city="Paris", days=3)]<|tool_call_end|>',
222
+ "qwen3_coder": "<tool_call>\n<function=get_weather><parameter=city>Paris</parameter>"
223
+ "<parameter=days>3</parameter></function></tool_call>",
224
+ @@ -3162,6 +3164,9 @@ def test_deepseek_streamed_tool_calls(chunk_size):
225
+ ("minicpm5", '<function name="lookup"><param>3</param></function>', None),
226
+ ("gemma4", "just a normal model response, no tool call here", None),
227
+ ("mistral", "not a tool call at all", None),
228
+ + ("inkling2", '{"name":"get_weather","args":{"city":"Par', None),
229
+ + ("inkling2", '{"args":{"city":"Paris"}}', "object with a name"),
230
+ + ("inkling2", '{"name":"get_weather","args":["Paris"]}', "must be an object"),
231
+ (
232
+ "pythonic",
233
+ "[write_file(content='const player = { x: 0, y: 1 };)]",
234
+ @@ -3527,6 +3532,130 @@ def test_harmony_ignores_plain_commentary_preamble():
235
+ assert result.remaining_text == preamble
236
+
237
+
238
+ +def _inkling2_call(name, raw=None, **arguments):
239
+ + body = raw or json.dumps(
240
+ + dict(name=name, args=arguments), ensure_ascii=False, separators=(",", ":")
241
+ + )
242
+ + return f"<|message_model|>{name}<|content_invoke_tool_json|>{body}<|end_message|>"
243
+ +
244
+ +
245
+ +def _inkling2_calls(text):
246
+ + result = process_tool_calls(text, load_tool_module("inkling2"), None)
247
+ + calls = [
248
+ + (call["function"]["name"], json.loads(call["function"]["arguments"]))
249
+ + for call in result.calls
250
+ + ]
251
+ + return calls, result
252
+ +
253
+ +
254
+ +INKLING2_THINKING = (
255
+ + "<|message_model|><|content_thinking|>I need both cities.<|end_message|>"
256
+ +)
257
+ +INKLING2_TEXT = "<|message_model|><|content_text|>Lisbon is warmer.<|end_message|>"
258
+ +INKLING2_ARGUMENTS = {
259
+ + "nested": {"list": [1, 2.5, None, True, {"deep": []}], "empty": {}},
260
+ + "unicode": "19°C — ação 日本 🐦",
261
+ + "quotes": 'say "hi" \\ back\\slash',
262
+ + "lines": "first\nsecond\ttabbed\r\n",
263
+ + "markers": "<|end_message|><|message_model|>x<|content_invoke_tool_json|>{}",
264
+ + "tags": "<think> </tool_call>",
265
+ +}
266
+ +
267
+ +
268
+ +def test_inkling2_parallel_calls_follow_reasoning():
269
+ + text = (
270
+ + INKLING2_THINKING
271
+ + + _inkling2_call("get_weather", city="Lisbon", unit="celsius")
272
+ + + _inkling2_call("get_weather", city="Oslo", unit="celsius")
273
+ + )
274
+ + calls, result = _inkling2_calls(text)
275
+ + assert calls == [
276
+ + ("get_weather", {"city": "Lisbon", "unit": "celsius"}),
277
+ + ("get_weather", {"city": "Oslo", "unit": "celsius"}),
278
+ + ]
279
+ + assert [call["index"] for call in result.calls] == [0, 1]
280
+ + assert len({call["id"] for call in result.calls}) == 2
281
+ + # The role header and the tool name it carries leave with the call.
282
+ + assert result.remaining_text == INKLING2_THINKING
283
+ +
284
+ +
285
+ +@pytest.mark.parametrize(
286
+ + "text,expected",
287
+ + [
288
+ + (INKLING2_THINKING + INKLING2_TEXT, []),
289
+ + (INKLING2_TEXT + _inkling2_call("get_time"), [("get_time", {})]),
290
+ + (
291
+ + "<|content_invoke_tool_json|>" + '{"name":"get_time"}' + "<|end_message|>",
292
+ + [("get_time", {})],
293
+ + ),
294
+ + (
295
+ + # Call markup quoted inside reasoning or text is not a call.
296
+ + "<|message_model|><|content_thinking|>Not "
297
+ + + _inkling2_call("delete_all")
298
+ + + INKLING2_TEXT,
299
+ + [],
300
+ + ),
301
+ + (
302
+ + "<|message_model|><|content_text|>Example: <|content_invoke_tool_json|>"
303
+ + '{"name":"delete_all","args":{}}<|end_message|>'
304
+ + + _inkling2_call("get_time"),
305
+ + [("get_time", {})],
306
+ + ),
307
+ + ],
308
+ + ids=["text-only", "text-then-call", "bare-call", "in-reasoning", "in-text"],
309
+ +)
310
+ +def test_inkling2_calls_are_whole_messages(text, expected):
311
+ + assert _inkling2_calls(text)[0] == expected
312
+ +
313
+ +
314
+ +@pytest.mark.parametrize("ascii_only", [False, True])
315
+ +def test_inkling2_arguments_round_trip(ascii_only):
316
+ + body = json.dumps(
317
+ + dict(name="write_file", args=INKLING2_ARGUMENTS), ensure_ascii=ascii_only
318
+ + )
319
+ + text = (
320
+ + _inkling2_call("write_file", raw=body)
321
+ + + INKLING2_TEXT
322
+ + + _inkling2_call("get_time")
323
+ + )
324
+ + calls, result = _inkling2_calls(text)
325
+ + assert calls == [("write_file", INKLING2_ARGUMENTS), ("get_time", {})]
326
+ + assert result.remaining_text == INKLING2_TEXT
327
+ + # Non-ASCII arguments are returned as written, not as escapes.
328
+ + assert "19°C — ação 日本 🐦" in result.calls[0]["function"]["arguments"]
329
+ +
330
+ +
331
+ +@pytest.mark.parametrize(
332
+ + "broken",
333
+ + [
334
+ + '{"name":"get_weather","args":{city}}',
335
+ + '{"name":"get_weather","args":{"city":"unterminated}}',
336
+ + '{"name":"get_weather","args":"Paris"}',
337
+ + '["get_weather"]',
338
+ + "",
339
+ + ],
340
+ +)
341
+ +def test_inkling2_malformed_call_is_dropped_alone(broken):
342
+ + text = (
343
+ + _inkling2_call("get_time")
344
+ + + _inkling2_call("get_weather", raw=broken or " ")
345
+ + + INKLING2_TEXT
346
+ + + _inkling2_call("get_date")
347
+ + )
348
+ + calls, result = _inkling2_calls(text)
349
+ + assert calls == [("get_time", {}), ("get_date", {})]
350
+ + assert result.remaining_text == INKLING2_TEXT
351
+ +
352
+ +
353
+ +@pytest.mark.parametrize("cut", range(1, 60, 3))
354
+ +def test_inkling2_truncated_call_is_not_a_call(cut):
355
+ + complete = INKLING2_THINKING + _inkling2_call("get_time")
356
+ + call = _inkling2_call("get_weather", city="Lisbon")
357
+ + calls, result = _inkling2_calls(complete + call[:-cut])
358
+ + assert calls == [("get_time", {})]
359
+ + assert "get_time" not in result.remaining_text
360
+ +
361
+ +
362
+ # Loading and utility contracts
363
+
364
+
365
+ diff --git a/mlx_vlm/tests/test_server.py b/mlx_vlm/tests/test_server.py
366
+ index 59eb1e94..6e44662b 100644
367
+ --- a/mlx_vlm/tests/test_server.py
368
+ +++ b/mlx_vlm/tests/test_server.py
369
+ @@ -73,7 +73,14 @@ from mlx_vlm.server.responses_state import (
370
+ strip_protocol_markers,
371
+ )
372
+ from mlx_vlm.server.runtime_config import RuntimeConfig
373
+ -from mlx_vlm.tests.test_processors import MINICPM_MULTICALL
374
+ +from mlx_vlm.tests.test_processors import (
375
+ + INKLING2_ARGUMENTS,
376
+ + INKLING2_TEXT,
377
+ + INKLING2_THINKING,
378
+ + MINICPM_MULTICALL,
379
+ + _inkling2_call,
380
+ + fast_tokenizer,
381
+ +)
382
+ from mlx_vlm.tokenizer_utils import SPMStreamingDetokenizer, _ServerTokenStreamer
383
+ from mlx_vlm.tools import load_tool_module, process_tool_calls
384
+
385
+ @@ -3680,6 +3687,631 @@ def test_response_template_tool_calls(client, api):
386
+ assert "to=self" not in response.text and "<atem:" not in response.text
387
+
388
+
389
+ +@pytest.mark.parametrize("api", ["chat", "responses"])
390
+ +def test_response_template_content_excludes_tool_routing(client, api):
391
+ + # Non-streamed, like the stream: a response template already kept the
392
+ + # reasoning and the call's routing header out of the content.
393
+ + config = NS(model_type="muse_glimmer")
394
+ + tool = (
395
+ + dict(type="function", name="get_weather", parameters={"type": "object"})
396
+ + if api == "responses"
397
+ + else _tool()
398
+ + )
399
+ + with _endpoint(
400
+ + config=config,
401
+ + processor=NS(config=config, tokenizer=_MuseResponseTemplateTokenizer()),
402
+ + result=_result(_MUSE_CALL),
403
+ + parser=load_tool_module("atem"),
404
+ + ):
405
+ + response = _post(client, api, tools=[tool])
406
+ + body = response.json()
407
+ + if api == "chat":
408
+ + message = body["choices"][0]["message"]
409
+ + assert message["content"] is None
410
+ + assert message["reasoning_content"] == "I need the weather tool."
411
+ + assert message["tool_calls"][0]["function"]["name"] == "get_weather"
412
+ + else:
413
+ + assert body["output_text"] == ""
414
+ + assert [item["type"] for item in body["output"]] == [
415
+ + "reasoning",
416
+ + "function_call",
417
+ + ]
418
+ + assert "to=" not in response.text
419
+ +
420
+ +
421
+ +# Inkling 2 tool calls. The tokenizer is a real one carrying the response
422
+ +# template the model ships and a chat template condensed from the model's: the
423
+ +# same message framing, and the same refusal of string tool-call arguments.
424
+ +
425
+ +_INKLING2_OPEN = r"(?:<\|message_model\|>[^<]*)?<\|content_%s\|>"
426
+ +_INKLING2_RESPONSE_TEMPLATE = {
427
+ + "defaults": {"role": "assistant"},
428
+ + "start_anchor_pattern": r"<\|end_message\|>|<\|content_model_end_sampling\|>",
429
+ + "fields": {
430
+ + "reasoning_content": {
431
+ + "open_pattern": _INKLING2_OPEN % "thinking",
432
+ + "close": "<|end_message|>",
433
+ + "repeats": True,
434
+ + "join": "",
435
+ + "content": "text",
436
+ + "content_args": {"strip": False},
437
+ + },
438
+ + "content": {
439
+ + "open_pattern": _INKLING2_OPEN % "(?:text|tool_error|invoke_tool_text)",
440
+ + "close": "<|end_message|>",
441
+ + "repeats": True,
442
+ + "join": "",
443
+ + "content": "text",
444
+ + "content_args": {"strip": False},
445
+ + },
446
+ + "tool_calls": {
447
+ + "open_pattern": _INKLING2_OPEN % "invoke_tool_json",
448
+ + "close": "<|end_message|>",
449
+ + "repeats": True,
450
+ + "content": "json",
451
+ + "transform": {
452
+ + "type": "function",
453
+ + "function": {"name": "{content.name}", "arguments": "{content.args}"},
454
+ + },
455
+ + },
456
+ + },
457
+ +}
458
+ +_INKLING2_CHAT_TEMPLATE = """
459
+ +{%- set levels = {"none": "0", "minimal": "0.1", "low": "0.2", "medium": "0.7",
460
+ + "high": "0.9", "xhigh": "0.99", "max": "0.99"} -%}
461
+ +{%- set roles = {"user": "<|message_user|>", "assistant": "<|message_model|>",
462
+ + "system": "<|message_system|>", "tool": "<|message_tool|>"} -%}
463
+ +{%- if tools -%}
464
+ + {{- "<|message_system|>tool_declare<|content_xml|>" ~ (tools | tojson) -}}
465
+ + {{- "<|end_message|>" -}}
466
+ +{%- endif -%}
467
+ +{%- set effort = reasoning_effort if reasoning_effort is defined
468
+ + and reasoning_effort is not none else "high" -%}
469
+ +{%- if effort not in levels -%}
470
+ + {{- raise_exception("Unknown reasoning_effort: " ~ effort) -}}
471
+ +{%- endif -%}
472
+ +{{- "<|message_system|><|content_text|>Thinking effort level: " ~ levels[effort] -}}
473
+ +{{- "<|end_message|>" -}}
474
+ +{%- for message in messages -%}
475
+ + {%- set header = namespace(text=roles[message.role]) -%}
476
+ + {%- if message.role == "tool" -%}
477
+ + {%- for turn in messages if turn.role == "assistant" and turn.tool_calls -%}
478
+ + {%- for call in turn.tool_calls if call.id == message.tool_call_id -%}
479
+ + {%- set header.text = header.text ~ call.function.name -%}
480
+ + {%- endfor -%}
481
+ + {%- endfor -%}
482
+ + {%- endif -%}
483
+ + {%- if message.role == "assistant" and message.reasoning_content -%}
484
+ + {{- header.text ~ "<|content_thinking|>" ~ message.reasoning_content -}}
485
+ + {{- "<|end_message|>" -}}
486
+ + {%- endif -%}
487
+ + {%- if message.content is string -%}
488
+ + {%- if message.content -%}
489
+ + {{- header.text ~ "<|content_text|>" ~ message.content -}}
490
+ + {{- "<|end_message|>" -}}
491
+ + {%- endif -%}
492
+ + {%- else -%}
493
+ + {%- for part in message.content or [] -%}
494
+ + {{- header.text ~ "<|content_text|>" ~ part.text ~ "<|end_message|>" -}}
495
+ + {%- endfor -%}
496
+ + {%- endif -%}
497
+ + {%- for call in message.tool_calls or [] -%}
498
+ + {%- if call.function.arguments is string -%}
499
+ + {{- raise_exception("tool call arguments must be a parsed object") -}}
500
+ + {%- endif -%}
501
+ + {{- "<|message_model|>" ~ call.function.name -}}
502
+ + {{- '<|content_invoke_tool_json|>{"name":' ~ (call.function.name | tojson) -}}
503
+ + {{- ',"args":' ~ (call.function.arguments | tojson(separators=(",", ":"))) -}}
504
+ + {{- "}<|end_message|>" -}}
505
+ + {%- endfor -%}
506
+ + {%- if message.role == "assistant" -%}
507
+ + {{- "<|content_model_end_sampling|>" -}}
508
+ + {%- endif -%}
509
+ +{%- endfor -%}
510
+ +"""
511
+ +_INKLING2_LISBON = dict(city="Lisbon", unit="celsius")
512
+ +_INKLING2_OSLO = dict(city="Oslo", unit="celsius")
513
+ +_INKLING2_CALLS = _inkling2_call("get_weather", **_INKLING2_LISBON) + _inkling2_call(
514
+ + "get_weather", **_INKLING2_OSLO
515
+ +)
516
+ +_INKLING2_WEATHER_CALLS = [
517
+ + ("get_weather", _INKLING2_LISBON),
518
+ + ("get_weather", _INKLING2_OSLO),
519
+ +]
520
+ +
521
+ +
522
+ +def _inkling2_request(
523
+ + client,
524
+ + text,
525
+ + api="chat",
526
+ + *,
527
+ + stream=False,
528
+ + chunk=0,
529
+ + batched=True,
530
+ + finish_reason="stop",
531
+ + **payload,
532
+ +):
533
+ + """Answer one request with the raw model output ``text``.
534
+ +
535
+ + ``chunk`` is the size of the streamed pieces, so 1 splits every control
536
+ + token and every JSON value. The finish reason arrives on a final empty
537
+ + token, as it does after the end-of-turn token. Returns the response and
538
+ + the prompts rendered by the real ``apply_chat_template``.
539
+ + """
540
+ + size = chunk or len(text)
541
+ + pieces = [text[i : i + size] for i in range(0, len(text), size)]
542
+ + tokens = [*map(_token, pieces), _token("", finish_reason=finish_reason)]
543
+ + tokenizer = fast_tokenizer(
544
+ + chat_template=_INKLING2_CHAT_TEMPLATE,
545
+ + response_template=_INKLING2_RESPONSE_TEMPLATE,
546
+ + )
547
+ + prompts = []
548
+ +
549
+ + def render(*args, **kwargs):
550
+ + prompts.append(apply_chat_template(*args, **kwargs))
551
+ + return prompts[-1]
552
+ +
553
+ + with _endpoint(
554
+ + config=NS(model_type="inkling2"),
555
+ + processor=NS(tokenizer=tokenizer),
556
+ + result=_result(text, finish_reason=finish_reason),
557
+ + chunks=[_result(t.text, finish_reason=t.finish_reason) for t in tokens],
558
+ + generator=_streaming(tokens) if batched else None,
559
+ + ) as fake:
560
+ + fake.template.side_effect = render
561
+ + response = _post(client, api, stream=stream, **payload)
562
+ + assert response.status_code == 200, response.text
563
+ + return response, prompts
564
+ +
565
+ +
566
+ +def _inkling2_chat(response, stream):
567
+ + """Reduce a chat completion, streamed or not, to one comparable turn."""
568
+ + if stream:
569
+ + events = _data(response)
570
+ + assert not any("error" in event for event in events)
571
+ + choices = [event["choices"][0] for event in events if event.get("choices")]
572
+ + deltas = [choice["delta"] for choice in choices]
573
+ + else:
574
+ + choices = response.json()["choices"]
575
+ + deltas = [choices[0]["message"]]
576
+ + calls = {}
577
+ + for delta in deltas:
578
+ + for call in delta.get("tool_calls") or []:
579
+ + function = call["function"]
580
+ + entry = calls.setdefault(call["index"], dict(name="", arguments=""))
581
+ + entry.update(id=call.get("id", entry.get("id")))
582
+ + entry["name"] += function.get("name") or ""
583
+ + entry["arguments"] += function.get("arguments") or ""
584
+ + assert list(calls) == list(range(len(calls)))
585
+ + return NS(
586
+ + reasoning=_joined(deltas, "reasoning_content"),
587
+ + content=_joined(deltas, "content"),
588
+ + calls=[(c["name"], json.loads(c["arguments"])) for c in calls.values()],
589
+ + ids=[c["id"] for c in calls.values()],
590
+ + finish=[c["finish_reason"] for c in choices if c.get("finish_reason")],
591
+ + deltas=deltas,
592
+ + )
593
+ +
594
+ +
595
+ +@pytest.mark.parametrize(
596
+ + "stream,chunk,batched",
597
+ + [
598
+ + (False, 0, True),
599
+ + (False, 0, False),
600
+ + (True, 0, True),
601
+ + (True, 1, True),
602
+ + (True, 1, False),
603
+ + (True, 5, True),
604
+ + (True, 11, False),
605
+ + ],
606
+ +)
607
+ +def test_inkling2_chat_parallel_tool_calls(client, stream, chunk, batched):
608
+ + response, _ = _inkling2_request(
609
+ + client,
610
+ + INKLING2_THINKING + _INKLING2_CALLS,
611
+ + stream=stream,
612
+ + chunk=chunk,
613
+ + batched=batched,
614
+ + tools=[_tool()],
615
+ + )
616
+ + turn = _inkling2_chat(response, stream)
617
+ + assert turn.calls == _INKLING2_WEATHER_CALLS
618
+ + assert len(set(turn.ids)) == 2 and all(turn.ids)
619
+ + assert turn.finish == ["tool_calls"]
620
+ + assert turn.reasoning == "I need both cities."
621
+ + assert turn.content == ""
622
+ + # No control token, and no tool name outside the calls themselves.
623
+ + assert "<|" not in response.text
624
+ + assert response.text.count("get_weather") == 2
625
+ + if stream:
626
+ + kinds = [
627
+ + "reasoning" if delta.get("reasoning_content") else "call"
628
+ + for delta in turn.deltas
629
+ + if delta.get("reasoning_content") or delta.get("tool_calls")
630
+ + ]
631
+ + assert kinds == sorted(kinds, reverse=True) and set(kinds) == {
632
+ + "reasoning",
633
+ + "call",
634
+ + }
635
+ + else:
636
+ + message = response.json()["choices"][0]["message"]
637
+ + assert message["content"] is None
638
+ + assert all(
639
+ + isinstance(call["function"]["arguments"], str)
640
+ + for call in message["tool_calls"]
641
+ + )
642
+ +
643
+ +
644
+ +@pytest.mark.parametrize("stream,chunk", [(False, 0), (True, 1), (True, 7)])
645
+ +@pytest.mark.parametrize(
646
+ + "text,reasoning,content,calls",
647
+ + [
648
+ + (_inkling2_call("get_time"), "", "", [("get_time", {})]),
649
+ + (
650
+ + INKLING2_THINKING + INKLING2_TEXT + _inkling2_call("get_time"),
651
+ + "I need both cities.",
652
+ + "Lisbon is warmer.",
653
+ + [("get_time", {})],
654
+ + ),
655
+ + (
656
+ + "<|message_model|><|content_text|>One. <|end_message|>"
657
+ + + _inkling2_call("get_time")
658
+ + + "<|message_model|><|content_tool_error|>Two.<|end_message|>",
659
+ + "",
660
+ + "One. Two.",
661
+ + [("get_time", {})],
662
+ + ),
663
+ + (
664
+ + INKLING2_THINKING + INKLING2_TEXT,
665
+ + "I need both cities.",
666
+ + "Lisbon is warmer.",
667
+ + [],
668
+ + ),
669
+ + ],
670
+ + ids=["call-only", "text-then-call", "text-around-call", "answer-only"],
671
+ +)
672
+ +def test_inkling2_chat_turn_shapes(
673
+ + client, stream, chunk, text, reasoning, content, calls
674
+ +):
675
+ + def turn(**payload):
676
+ + response, _ = _inkling2_request(
677
+ + client, text, stream=stream, chunk=chunk, **payload
678
+ + )
679
+ + assert "<|" not in response.text
680
+ + return _inkling2_chat(response, stream)
681
+ +
682
+ + with_tools = turn(tools=[_tool("get_time")])
683
+ + assert with_tools.calls == calls
684
+ + assert with_tools.reasoning == reasoning
685
+ + assert with_tools.content == content
686
+ + assert with_tools.finish == ["tool_calls" if calls else "stop"]
687
+ + # Without tools nothing is executable; reasoning and content are the same.
688
+ + without_tools = turn()
689
+ + assert without_tools.calls == []
690
+ + assert without_tools.finish == ["stop"]
691
+ + assert without_tools.reasoning == reasoning
692
+ + assert without_tools.content == content
693
+ +
694
+ +
695
+ +@pytest.mark.parametrize("stream,chunk", [(False, 0), (True, 1), (True, 9)])
696
+ +def test_inkling2_chat_arguments_round_trip(client, stream, chunk):
697
+ + text = INKLING2_THINKING + _inkling2_call("write_file", **INKLING2_ARGUMENTS)
698
+ + response, _ = _inkling2_request(
699
+ + client, text, stream=stream, chunk=chunk, tools=[_tool("write_file")]
700
+ + )
701
+ + turn = _inkling2_chat(response, stream)
702
+ + assert turn.calls == [("write_file", INKLING2_ARGUMENTS)]
703
+ + assert turn.finish == ["tool_calls"]
704
+ + assert turn.reasoning == "I need both cities."
705
+ + assert turn.content == ""
706
+ +
707
+ +
708
+ +@pytest.mark.parametrize("stream,chunk", [(False, 0), (True, 1), (True, 6)])
709
+ +@pytest.mark.parametrize("batched", [False, True])
710
+ +@pytest.mark.parametrize(
711
+ + "text,finish_reason,calls,finish",
712
+ + [
713
+ + # Cut by max_tokens inside the arguments: there is no call to run.
714
+ + (
715
+ + INKLING2_THINKING + _inkling2_call("get_weather", city="Lisbon")[:-24],
716
+ + "length",
717
+ + [],
718
+ + "length",
719
+ + ),
720
+ + # A call completed before the cut is still returned.
721
+ + (
722
+ + _inkling2_call("get_time")
723
+ + + _inkling2_call("get_weather", city="Lisbon")[:-24],
724
+ + "length",
725
+ + [("get_time", {})],
726
+ + "tool_calls",
727
+ + ),
728
+ + (
729
+ + _inkling2_call("get_weather", raw='{"name":"get_weather","args":{city}}'),
730
+ + "stop",
731
+ + [],
732
+ + "stop",
733
+ + ),
734
+ + (
735
+ + _inkling2_call("get_weather", raw='{"name":"get_weather"')
736
+ + + _inkling2_call("get_time"),
737
+ + "stop",
738
+ + [("get_time", {})],
739
+ + "tool_calls",
740
+ + ),
741
+ + ],
742
+ + ids=["truncated", "truncated-after-call", "malformed", "malformed-then-call"],
743
+ +)
744
+ +def test_inkling2_chat_incomplete_calls(
745
+ + client, stream, chunk, batched, text, finish_reason, calls, finish
746
+ +):
747
+ + response, _ = _inkling2_request(
748
+ + client,
749
+ + text,
750
+ + stream=stream,
751
+ + chunk=chunk,
752
+ + batched=batched,
753
+ + finish_reason=finish_reason,
754
+ + tools=[_tool("get_weather"), _tool("get_time")],
755
+ + )
756
+ + turn = _inkling2_chat(response, stream)
757
+ + assert turn.calls == calls
758
+ + assert turn.finish == [finish]
759
+ + assert turn.content == ""
760
+ + assert turn.reasoning == ("I need both cities." if "thinking" in text else "")
761
+ + assert "<|" not in response.text and "city" not in response.text
762
+ +
763
+ +
764
+ +@pytest.mark.parametrize("stream,chunk", [(False, 0), (True, 0), (True, 1)])
765
+ +def test_inkling2_messages_tool_use(client, stream, chunk):
766
+ + response, _ = _inkling2_request(
767
+ + client,
768
+ + INKLING2_THINKING + INKLING2_TEXT + _INKLING2_CALLS,
769
+ + "messages",
770
+ + stream=stream,
771
+ + chunk=chunk,
772
+ + tools=[_tool(api="messages")],
773
+ + thinking=dict(type="enabled", budget_tokens=64),
774
+ + max_tokens=256,
775
+ + )
776
+ + if stream:
777
+ + events = _data(response)
778
+ + blocks = [e["content_block"] for e in events if "content_block" in e]
779
+ + inputs = [
780
+ + e["delta"]["partial_json"]
781
+ + for e in events
782
+ + if e.get("delta", {}).get("type") == "input_json_delta"
783
+ + ]
784
+ + names = [block["name"] for block in blocks if block["type"] == "tool_use"]
785
+ + assert len(names) == len(inputs)
786
+ + calls = list(zip(names, map(json.loads, inputs)))
787
+ + thinking, text = _thinking_text(response, "messages")
788
+ + stop_reason = _anthropic_stream_stop(response)[0]
789
+ + else:
790
+ + body = response.json()
791
+ + blocks = body["content"]
792
+ + calls = [(b["name"], b["input"]) for b in blocks if b["type"] == "tool_use"]
793
+ + thinking = _joined(blocks, "thinking")
794
+ + text = _joined(blocks, "text")
795
+ + stop_reason = body["stop_reason"]
796
+ + assert [block["type"] for block in blocks] == [
797
+ + "thinking",
798
+ + "text",
799
+ + "tool_use",
800
+ + "tool_use",
801
+ + ]
802
+ + assert calls == _INKLING2_WEATHER_CALLS
803
+ + assert len({b["id"] for b in blocks if b["type"] == "tool_use"}) == 2
804
+ + assert thinking == "I need both cities."
805
+ + assert text == "Lisbon is warmer."
806
+ + assert stop_reason == "tool_use"
807
+ + assert "<|" not in response.text
808
+ +
809
+ +
810
+ +@pytest.mark.parametrize("stream,chunk", [(False, 0), (True, 0), (True, 1)])
811
+ +def test_inkling2_responses_function_calls(client, stream, chunk):
812
+ + server.response_store.clear()
813
+ + server.response_store_order.clear()
814
+ + response, _ = _inkling2_request(
815
+ + client,
816
+ + INKLING2_THINKING + INKLING2_TEXT + _INKLING2_CALLS,
817
+ + "responses",
818
+ + stream=stream,
819
+ + chunk=chunk,
820
+ + tools=[dict(type="function", name="get_weather", parameters={})],
821
+ + )
822
+ + body = _completed_response(response) if stream else response.json()
823
+ + items = {}
824
+ + for item in body["output"]:
825
+ + items.setdefault(item["type"], []).append(item)
826
+ + calls = items.pop("function_call")
827
+ + assert [
828
+ + (call["name"], json.loads(call["arguments"])) for call in calls
829
+ + ] == _INKLING2_WEATHER_CALLS
830
+ + assert len({call["call_id"] for call in calls}) == 2
831
+ + (reasoning,) = items.pop("reasoning")
832
+ + assert reasoning["summary"][0]["text"] == "I need both cities."
833
+ + # The text beside the calls is a message item only when streamed.
834
+ + for message in items.pop("message", []):
835
+ + assert message["content"][0]["text"] == "Lisbon is warmer."
836
+ + assert not items
837
+ + assert body["output_text"] == "Lisbon is warmer."
838
+ + assert body["status"] == "completed"
839
+ + assert "<|" not in response.text
840
+ +
841
+ +
842
+ +@pytest.mark.parametrize("api", ["messages", "responses"])
843
+ +@pytest.mark.parametrize("stream", [False, True])
844
+ +def test_inkling2_truncated_call_ends_the_turn_cleanly(client, api, stream):
845
+ + server.response_store.clear()
846
+ + server.response_store_order.clear()
847
+ + tool = (
848
+ + _tool(api="messages")
849
+ + if api == "messages"
850
+ + else dict(type="function", name="get_weather", parameters={})
851
+ + )
852
+ + response, _ = _inkling2_request(
853
+ + client,
854
+ + INKLING2_TEXT + _inkling2_call("get_weather", city="Lisbon")[:-24],
855
+ + api,
856
+ + stream=stream,
857
+ + chunk=4 * stream,
858
+ + finish_reason="length",
859
+ + tools=[tool],
860
+ + )
861
+ + if api == "responses":
862
+ + body = _completed_response(response) if stream else response.json()
863
+ + assert body["status"] == "incomplete"
864
+ + assert [item["type"] for item in body["output"]] == ["message"]
865
+ + assert body["output_text"] == "Lisbon is warmer."
866
+ + elif stream:
867
+ + assert _thinking_text(response, api) == ("", "Lisbon is warmer.")
868
+ + assert _anthropic_stream_stop(response) == ("max_tokens", None)
869
+ + else:
870
+ + body = response.json()
871
+ + assert body["content"] == [dict(type="text", text="Lisbon is warmer.")]
872
+ + assert body["stop_reason"] == "max_tokens"
873
+ + assert "<|" not in response.text and "city" not in response.text
874
+ +
875
+ +
876
+ +_INKLING2_HISTORY = dict(
877
+ + chat=dict(
878
+ + messages=[
879
+ + _msg("Compare Lisbon and Oslo."),
880
+ + _msg(
881
+ + None,
882
+ + "assistant",
883
+ + reasoning_content="I need both cities.",
884
+ + tool_calls=[
885
+ + dict(
886
+ + id=f"call_{city}",
887
+ + type="function",
888
+ + # Clients send the arguments back as a JSON string.
889
+ + function=dict(
890
+ + name="get_weather", arguments=json.dumps(dict(city=city))
891
+ + ),
892
+ + )
893
+ + for city in ("Lisbon", "Oslo")
894
+ + ],
895
+ + ),
896
+ + _msg("19C", "tool", tool_call_id="call_Lisbon"),
897
+ + _msg("3C", "tool", tool_call_id="call_Oslo"),
898
+ + ],
899
+ + tools=[_tool()],
900
+ + ),
901
+ + responses=dict(
902
+ + input=[
903
+ + _input_message("Compare Lisbon and Oslo."),
904
+ + *(
905
+ + dict(
906
+ + type="function_call",
907
+ + name="get_weather",
908
+ + arguments=json.dumps(dict(city=city)),
909
+ + call_id=f"call_{city}",
910
+ + )
911
+ + for city in ("Lisbon", "Oslo")
912
+ + ),
913
+ + dict(type="function_call_output", call_id="call_Lisbon", output="19C"),
914
+ + dict(type="function_call_output", call_id="call_Oslo", output="3C"),
915
+ + ],
916
+ + tools=[dict(type="function", name="get_weather", parameters={})],
917
+ + ),
918
+ + messages=dict(
919
+ + messages=[
920
+ + _msg("Compare Lisbon and Oslo."),
921
+ + _msg(
922
+ + [
923
+ + dict(
924
+ + type="tool_use",
925
+ + id=f"call_{city}",
926
+ + name="get_weather",
927
+ + input=dict(city=city),
928
+ + )
929
+ + for city in ("Lisbon", "Oslo")
930
+ + ],
931
+ + "assistant",
932
+ + ),
933
+ + _msg(
934
+ + [
935
+ + dict(type="tool_result", tool_use_id="call_Lisbon", content="19C"),
936
+ + dict(type="tool_result", tool_use_id="call_Oslo", content="3C"),
937
+ + ]
938
+ + ),
939
+ + ],
940
+ + tools=[_tool(api="messages")],
941
+ + max_tokens=64,
942
+ + ),
943
+ +)
944
+ +
945
+ +
946
+ +@pytest.mark.parametrize("api", ["chat", "responses", "messages"])
947
+ +def test_inkling2_history_with_string_arguments_renders(client, api):
948
+ + # The chat template refuses string arguments, so the stored history must
949
+ + # reach it as objects, and each tool result must find its call's name.
950
+ + _, prompts = _inkling2_request(client, INKLING2_TEXT, api, **_INKLING2_HISTORY[api])
951
+ + prompt = prompts[-1]
952
+ + assert prompt.startswith("<|message_system|>tool_declare<|content_xml|>[{")
953
+ + rendered = [
954
+ + _inkling2_call("get_weather", city="Lisbon"),
955
+ + _inkling2_call("get_weather", city="Oslo"),
956
+ + "<|message_tool|>get_weather<|content_text|>19C<|end_message|>",
957
+ + "<|message_tool|>get_weather<|content_text|>3C<|end_message|>",
958
+ + ]
959
+ + assert all(prompt.count(part) == 1 for part in rendered)
960
+ + assert sorted(rendered, key=prompt.index) == rendered
961
+ + if api == "chat":
962
+ + assert "<|content_thinking|>I need both cities.<|end_message|>" in prompt
963
+ +
964
+ +
965
+ +@pytest.mark.parametrize("api", ["chat", "responses"])
966
+ +@pytest.mark.parametrize(
967
+ + "effort,level",
968
+ + [
969
+ + (None, "0.9"),
970
+ + ("none", "0"),
971
+ + ("minimal", "0.1"),
972
+ + ("low", "0.2"),
973
+ + ("medium", "0.7"),
974
+ + ("high", "0.9"),
975
+ + ("xhigh", "0.99"),
976
+ + ("max", "0.99"),
977
+ + ],
978
+ +)
979
+ +def test_inkling2_reasoning_effort_reaches_template(client, api, effort, level):
980
+ + payload = {}
981
+ + if effort is not None:
982
+ + payload = (
983
+ + dict(reasoning_effort=effort)
984
+ + if api == "chat"
985
+ + else dict(reasoning=dict(effort=effort))
986
+ + )
987
+ + _, prompts = _inkling2_request(client, INKLING2_TEXT, api, **payload)
988
+ + assert f"Thinking effort level: {level}<|end_message|>" in prompts[-1]
989
+ +
990
+ +
991
+ +@pytest.mark.parametrize("stream", [False, True])
992
+ +def test_inkling2_split_survives_an_undecodable_call(stream):
993
+ + # The response template decodes a call's JSON when its region closes, and
994
+ + # raises on a truncated or malformed one. That must not cost the turn its
995
+ + # reasoning and content, nor put the raw text into the content.
996
+ + processor = NS(
997
+ + tokenizer=fast_tokenizer(response_template=_INKLING2_RESPONSE_TEMPLATE)
998
+ + )
999
+ + for call in (
1000
+ + _inkling2_call("get_weather", city="Lisbon")[:-24],
1001
+ + _inkling2_call("get_weather", raw="{city}"),
1002
+ + _inkling2_call("get_weather", raw='{"name":"get_weather"}'),
1003
+ + ):
1004
+ + text = INKLING2_THINKING + call + INKLING2_TEXT
1005
+ + closed = call.endswith("<|end_message|>")
1006
+ + expected = ("I need both cities.", "Lisbon is warmer." if closed else "")
1007
+ + if stream:
1008
+ + state = server.make_response_stream_state(processor)
1009
+ + assert _thoughts(_feed_thinking(state, list(text), last=True)) == expected
1010
+ + else:
1011
+ + assert server._split_thinking(text, processor=processor) == expected
1012
+ +
1013
+ +
1014
+ def test_chat_completions_endpoint_falls_back_from_video_to_images(client):
1015
+ from mlx_vlm.generate import video
1016
+
1017
+ diff --git a/mlx_vlm/tools/base.py b/mlx_vlm/tools/base.py
1018
+ index de8f5486..7432dcb5 100644
1019
+ --- a/mlx_vlm/tools/base.py
1020
+ +++ b/mlx_vlm/tools/base.py
1021
+ @@ -6,6 +6,12 @@ string for formats whose call ends at a newline rather than a closing marker.
1022
+ The extractor (:mod:`mlx_vlm.tools.extractor`) drives any module shaped like
1023
+ this; the :class:`ParserSpec` in :mod:`mlx_vlm.tools.registry` maps a chat
1024
+ template to the module that owns it.
1025
+ +
1026
+ +A format whose calls two markers cannot delimit may also define
1027
+ +``find_tool_calls(text) -> [(start, end), ...]``. It receives the whole
1028
+ +assistant turn, reasoning included, and returns the span of every executable
1029
+ +call in order; the extractor then uses those spans instead of the markers
1030
+ +(see :mod:`mlx_vlm.tools.parsers.inkling2`).
1031
+ """
1032
+
1033
+ from __future__ import annotations
1034
+ diff --git a/mlx_vlm/tools/extractor.py b/mlx_vlm/tools/extractor.py
1035
+ index 2ff57b66..36b3bdd2 100644
1036
+ --- a/mlx_vlm/tools/extractor.py
1037
+ +++ b/mlx_vlm/tools/extractor.py
1038
+ @@ -1,15 +1,16 @@
1039
+ """Extract structured tool calls from a model's text output.
1040
+
1041
+ Format-agnostic: given the parser module selected for the model (see
1042
+ -:mod:`mlx_vlm.tools.registry`), it slices the text between the parser's markers,
1043
+ -delegates per-format decoding to ``parse_tool_call``, and emits OpenAI-shaped
1044
+ -calls.
1045
+ +:mod:`mlx_vlm.tools.registry`), it slices the text between the parser's markers
1046
+ +(or at the spans the parser's own ``find_tool_calls`` reports), delegates
1047
+ +per-format decoding to ``parse_tool_call``, and emits OpenAI-shaped calls.
1048
+ """
1049
+
1050
+ import json
1051
+ import logging
1052
+ import re
1053
+ import uuid
1054
+ +from typing import List, Tuple
1055
+
1056
+ from .base import ToolParser
1057
+ from .types import ParseResult
1058
+ @@ -17,6 +18,23 @@ from .types import ParseResult
1059
+ logger = logging.getLogger("mlx_vlm.server")
1060
+
1061
+
1062
+ +def _call_spans(model_output: str, tool_module: ToolParser) -> List[Tuple[int, int]]:
1063
+ + """Return the ``(start, end)`` span of every tool call, markers included."""
1064
+ + find_tool_calls = getattr(tool_module, "find_tool_calls", None)
1065
+ + if find_tool_calls is not None:
1066
+ + return find_tool_calls(model_output)
1067
+ + if tool_module.tool_call_end == "":
1068
+ + pattern = re.compile(
1069
+ + f"{re.escape(tool_module.tool_call_start)}.*?(?:\n|$)", re.DOTALL
1070
+ + )
1071
+ + else:
1072
+ + pattern = re.compile(
1073
+ + f"{re.escape(tool_module.tool_call_start)}.*?{re.escape(tool_module.tool_call_end)}",
1074
+ + re.DOTALL,
1075
+ + )
1076
+ + return [match.span() for match in pattern.finditer(model_output)]
1077
+ +
1078
+ +
1079
+ def process_tool_calls(
1080
+ model_output: str, tool_module: ToolParser, tools
1081
+ ) -> ParseResult:
1082
+ @@ -25,19 +43,14 @@ def process_tool_calls(
1083
+ remaining = model_output
1084
+
1085
+ if tool_module.tool_call_start in model_output:
1086
+ - if tool_module.tool_call_end == "":
1087
+ - pattern = re.compile(
1088
+ - f"{re.escape(tool_module.tool_call_start)}.*?(?:\n|$)", re.DOTALL
1089
+ - )
1090
+ - else:
1091
+ - pattern = re.compile(
1092
+ - f"{re.escape(tool_module.tool_call_start)}.*?{re.escape(tool_module.tool_call_end)}",
1093
+ - re.DOTALL,
1094
+ - )
1095
+ -
1096
+ - matches = re.findall(pattern, model_output)
1097
+ - if matches:
1098
+ - remaining = re.sub(pattern, " ", model_output).strip()
1099
+ + spans = _call_spans(model_output, tool_module)
1100
+ + if spans:
1101
+ + matches = [model_output[start:end] for start, end in spans]
1102
+ + # Each call is replaced with a space.
1103
+ + edges = [0, *(edge for span in spans for edge in span), len(model_output)]
1104
+ + remaining = " ".join(
1105
+ + model_output[start:end] for start, end in zip(edges[::2], edges[1::2])
1106
+ + ).strip()
1107
+ for match in matches:
1108
+ call = (
1109
+ match.strip()
1110
+ diff --git a/mlx_vlm/tools/parsers/inkling2.py b/mlx_vlm/tools/parsers/inkling2.py
1111
+ new file mode 100644
1112
+ index 00000000..1f9d18f0
1113
+ --- /dev/null
1114
+ +++ b/mlx_vlm/tools/parsers/inkling2.py
1115
+ @@ -0,0 +1,99 @@
1116
+ +"""Inkling 2 tool calls.
1117
+ +
1118
+ +Inkling 2 writes an assistant turn as a run of messages, each opened by a role
1119
+ +header and a content-type token and closed by ``<|end_message|>``::
1120
+ +
1121
+ + <|message_model|><|content_thinking|>I need the weather.<|end_message|>
1122
+ + <|message_model|><|content_text|>Let me check.<|end_message|>
1123
+ + <|message_model|>get_weather<|content_invoke_tool_json|>{"name":"get_weather","args":{"city":"Oslo"}}<|end_message|>
1124
+ +
1125
+ +A tool call is a message of type ``invoke_tool_json``: its header repeats the
1126
+ +tool name and its body is one JSON object with ``name`` and ``args``. Parallel
1127
+ +calls are consecutive call messages.
1128
+ +
1129
+ +The two markers alone do not delimit a call: the header belongs to it, and the
1130
+ +closing token also closes every other message. :func:`find_tool_calls` therefore
1131
+ +walks the turn message by message, the way the tokenizer's ``response_template``
1132
+ +does, so call markup quoted inside a reasoning or text message is not a call.
1133
+ +
1134
+ +A call whose body is not a JSON object with a string ``name`` and an object
1135
+ +``args`` is rejected: the extractor logs and drops it, and keeps the calls
1136
+ +around it. A call still open when generation stops (``max_tokens``) is not a
1137
+ +call.
1138
+ +
1139
+ +Limit: this works on decoded text, where a control token and the same characters
1140
+ +written as ordinary text look alike. A call body is JSON, so ``<|end_message|>``
1141
+ +inside one of its strings is recognised as data (see :func:`_call_end`). In a
1142
+ +reasoning or text message those characters end the message. The server splits
1143
+ +reasoning and content with the response template, which does not read JSON: a
1144
+ +message opening such as ``<|content_text|>`` written inside an argument string
1145
+ +after those characters also shows up as content.
1146
+ +"""
1147
+ +
1148
+ +import json
1149
+ +import re
1150
+ +from typing import Any, List, Optional, Tuple
1151
+ +
1152
+ +tool_call_start = "<|content_invoke_tool_json|>"
1153
+ +tool_call_end = "<|end_message|>"
1154
+ +
1155
+ +_HEADER = r"(?:<\|message_model\|>[^<]*)?"
1156
+ +# The message openings the response template knows; other text is not a message.
1157
+ +_MESSAGE = re.compile(
1158
+ + _HEADER
1159
+ + + r"<\|content_(thinking|text|tool_error|invoke_tool_text|invoke_tool_json)\|>"
1160
+ +)
1161
+ +_CALL_OPENING = re.compile(rf"\s*{_HEADER}{re.escape(tool_call_start)}")
1162
+ +
1163
+ +
1164
+ +def _call_end(text: str, start: int) -> int:
1165
+ + """Index of the ``<|end_message|>`` that closes the call body at ``start``.
1166
+ +
1167
+ + The call ends at the first marker that leaves valid JSON before it, so the
1168
+ + marker's characters inside a JSON string stay part of the arguments. A body
1169
+ + that is valid at no marker ends at the first one, where
1170
+ + :func:`parse_tool_call` rejects it. Returns -1 while the call is unclosed.
1171
+ + """
1172
+ + first = end = text.find(tool_call_end, start)
1173
+ + while end >= 0:
1174
+ + try:
1175
+ + json.loads(text[start:end])
1176
+ + return end
1177
+ + except ValueError:
1178
+ + end = text.find(tool_call_end, end + len(tool_call_end))
1179
+ + return first
1180
+ +
1181
+ +
1182
+ +def find_tool_calls(text: str) -> List[Tuple[int, int]]:
1183
+ + """Return the ``(start, end)`` span of every call message, header included.
1184
+ +
1185
+ + A message still open when the text ends (generation was cut off) is not a
1186
+ + call, and nothing after it is.
1187
+ + """
1188
+ + spans = []
1189
+ + position = 0
1190
+ + while (message := _MESSAGE.search(text, position)) is not None:
1191
+ + is_call = message[1] == "invoke_tool_json"
1192
+ + end = (
1193
+ + _call_end(text, message.end())
1194
+ + if is_call
1195
+ + else text.find(tool_call_end, message.end())
1196
+ + )
1197
+ + if end < 0:
1198
+ + break
1199
+ + position = end + len(tool_call_end)
1200
+ + if is_call:
1201
+ + spans.append((message.start(), position))
1202
+ + return spans
1203
+ +
1204
+ +
1205
+ +def parse_tool_call(text: str, tools: Optional[Any] = None) -> dict:
1206
+ + """Decode one call body, with or without its ``<|message_model|>NAME`` opening."""
1207
+ + opening = _CALL_OPENING.match(text)
1208
+ + call = json.loads(text[opening.end() :] if opening else text)
1209
+ + if not isinstance(call, dict) or not isinstance(call.get("name"), str):
1210
+ + raise ValueError("Inkling 2 tool call must be an object with a name.")
1211
+ + arguments = call.get("args", {})
1212
+ + if not isinstance(arguments, dict):
1213
+ + raise ValueError("Inkling 2 tool call args must be an object.")
1214
+ + return {"name": call["name"], "arguments": arguments}
1215
+ diff --git a/mlx_vlm/tools/registry.py b/mlx_vlm/tools/registry.py
1216
+ index ddf3b278..8f4447f7 100644
1217
+ --- a/mlx_vlm/tools/registry.py
1218
+ +++ b/mlx_vlm/tools/registry.py
1219
+ @@ -65,6 +65,7 @@ SPECS: Tuple[ParserSpec, ...] = (
1220
+ ParserSpec("kimi_k2", (("<|tool_calls_section_begin|>",),)),
1221
+ ParserSpec("mistral", (("[TOOL_CALLS]",),)),
1222
+ ParserSpec("harmony", (("<|channel|>commentary", "<|call|>"),)),
1223
+ + ParserSpec("inkling2", (("<|content_invoke_tool_json|>",),)),
1224
+ ParserSpec("json_tools", (("<tool_call>", "tool_call.name"),), priority=0),
1225
+ )
1226
+
extras/recipe.json ADDED
@@ -0,0 +1,27 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "recipe": {
3
+ "experts": {
4
+ "default": "gptq:3:64"
5
+ },
6
+ "dense": {
7
+ "attn": [
8
+ 8,
9
+ 64
10
+ ],
11
+ "embed": [
12
+ 8,
13
+ 64
14
+ ],
15
+ "head": [
16
+ 8,
17
+ 64
18
+ ],
19
+ "vision": [
20
+ 8,
21
+ 64
22
+ ]
23
+ }
24
+ },
25
+ "calibration_tokens": 49465,
26
+ "method": "GPTQ (error-compensated rounding), affine, group size 64; scales and offsets optimised at float16 precision and stored as bfloat16"
27
+ }
extras/results.json ADDED
@@ -0,0 +1,3927 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "held_out_loss": {
3
+ "native": {
4
+ "tokens": 40806,
5
+ "nll": 1.4480010462071342,
6
+ "acc": 0.6699995098759987,
7
+ "seconds": 115,
8
+ "peak_gb": 22.03,
9
+ "weights_gib": null,
10
+ "activations": "bf16",
11
+ "actquant": null,
12
+ "chat": {
13
+ "tokens": 6597,
14
+ "nll": 0.4662371408908642
15
+ },
16
+ "code": {
17
+ "tokens": 8877,
18
+ "nll": 0.8892847396130361
19
+ },
20
+ "wikitext": {
21
+ "tokens": 25332,
22
+ "nll": 1.8994624837962406
23
+ }
24
+ },
25
+ "3bit": {
26
+ "tokens": 40806,
27
+ "nll": 1.46238731782892,
28
+ "acc": 0.6657844434641964,
29
+ "seconds": 113,
30
+ "peak_gb": 14.3,
31
+ "weights_gib": null,
32
+ "activations": "bf16",
33
+ "actquant": null,
34
+ "chat": {
35
+ "tokens": 6597,
36
+ "nll": 0.49518932673864985
37
+ },
38
+ "code": {
39
+ "tokens": 8877,
40
+ "nll": 0.9058923079944132
41
+ },
42
+ "wikitext": {
43
+ "tokens": 25332,
44
+ "nll": 1.9092770758236866
45
+ }
46
+ },
47
+ "3bit_round_to_nearest": {
48
+ "tokens": 40806,
49
+ "nll": 1.4701619848213856,
50
+ "acc": 0.6639709846591187,
51
+ "seconds": 113,
52
+ "peak_gb": 19.63,
53
+ "weights_gib": 11.910704612731934,
54
+ "activations": "bf16",
55
+ "actquant": null,
56
+ "chat": {
57
+ "tokens": 6597,
58
+ "nll": 0.49830741327206834
59
+ },
60
+ "code": {
61
+ "tokens": 8877,
62
+ "nll": 0.9093072548956838
63
+ },
64
+ "wikitext": {
65
+ "tokens": 25332,
66
+ "nll": 1.919792177702378
67
+ }
68
+ }
69
+ },
70
+ "numerics_experiments": {
71
+ "native_bf16_actquant": {
72
+ "tokens": 40806,
73
+ "nll": 1.4507249015946835,
74
+ "acc": 0.6688967308729108,
75
+ "seconds": 151,
76
+ "peak_gb": 22.22,
77
+ "weights_gib": null,
78
+ "activations": "bf16",
79
+ "actquant": [
80
+ "input",
81
+ "hidden"
82
+ ],
83
+ "chat": {
84
+ "tokens": 6597,
85
+ "nll": 0.470449288324712
86
+ },
87
+ "code": {
88
+ "tokens": 8877,
89
+ "nll": 0.8916940795444644
90
+ },
91
+ "wikitext": {
92
+ "tokens": 25332,
93
+ "nll": 1.9019089702857381
94
+ }
95
+ },
96
+ "native_fp32": {
97
+ "tokens": 40806,
98
+ "nll": 1.4489235311713002,
99
+ "acc": 0.6690682742733912,
100
+ "seconds": 134,
101
+ "peak_gb": 24.1,
102
+ "weights_gib": null,
103
+ "activations": "fp32",
104
+ "actquant": null,
105
+ "chat": {
106
+ "tokens": 6597,
107
+ "nll": 0.4647530796709649
108
+ },
109
+ "code": {
110
+ "tokens": 8877,
111
+ "nll": 0.8885994847056913
112
+ },
113
+ "wikitext": {
114
+ "tokens": 25332,
115
+ "nll": 1.9015750797668678
116
+ }
117
+ },
118
+ "native_fp32_actquant": {
119
+ "tokens": 40806,
120
+ "nll": 1.4478448928936214,
121
+ "acc": 0.6682105572709895,
122
+ "seconds": 163,
123
+ "peak_gb": 24.24,
124
+ "weights_gib": null,
125
+ "activations": "fp32",
126
+ "actquant": [
127
+ "input",
128
+ "hidden"
129
+ ],
130
+ "chat": {
131
+ "tokens": 6597,
132
+ "nll": 0.4712143190091029
133
+ },
134
+ "code": {
135
+ "tokens": 8877,
136
+ "nll": 0.8948558546528027
137
+ },
138
+ "wikitext": {
139
+ "tokens": 25332,
140
+ "nll": 1.8959625144150138
141
+ }
142
+ },
143
+ "native_bf16_actquant_input": {
144
+ "tokens": 40806,
145
+ "nll": 1.4477849032594567,
146
+ "acc": 0.6688477184727736,
147
+ "seconds": 120,
148
+ "peak_gb": 21.99,
149
+ "weights_gib": null,
150
+ "activations": "bf16",
151
+ "actquant": [
152
+ "input"
153
+ ],
154
+ "chat": {
155
+ "tokens": 6597,
156
+ "nll": 0.46967875793049946
157
+ },
158
+ "code": {
159
+ "tokens": 8877,
160
+ "nll": 0.8883638166019585
161
+ },
162
+ "wikitext": {
163
+ "tokens": 25332,
164
+ "nll": 1.8985407546329665
165
+ }
166
+ }
167
+ },
168
+ "tasks": {
169
+ "tools": {
170
+ "passed": 19,
171
+ "total": 19,
172
+ "leaked": 0,
173
+ "seconds": 87,
174
+ "rows": [
175
+ {
176
+ "kind": "single",
177
+ "i": 0,
178
+ "n_calls": 1,
179
+ "finish": "tool_calls",
180
+ "leaked": false,
181
+ "seconds": 2.5,
182
+ "valid_json": true,
183
+ "right_tool": true,
184
+ "ok": true
185
+ },
186
+ {
187
+ "kind": "single",
188
+ "i": 1,
189
+ "n_calls": 1,
190
+ "finish": "tool_calls",
191
+ "leaked": false,
192
+ "seconds": 3.3,
193
+ "valid_json": true,
194
+ "right_tool": true,
195
+ "ok": true
196
+ },
197
+ {
198
+ "kind": "single",
199
+ "i": 2,
200
+ "n_calls": 1,
201
+ "finish": "tool_calls",
202
+ "leaked": false,
203
+ "seconds": 2.3,
204
+ "valid_json": true,
205
+ "right_tool": true,
206
+ "ok": true
207
+ },
208
+ {
209
+ "kind": "single",
210
+ "i": 3,
211
+ "n_calls": 1,
212
+ "finish": "tool_calls",
213
+ "leaked": false,
214
+ "seconds": 2.3,
215
+ "valid_json": true,
216
+ "right_tool": true,
217
+ "ok": true
218
+ },
219
+ {
220
+ "kind": "single",
221
+ "i": 4,
222
+ "n_calls": 1,
223
+ "finish": "tool_calls",
224
+ "leaked": false,
225
+ "seconds": 2.7,
226
+ "valid_json": true,
227
+ "right_tool": true,
228
+ "ok": true
229
+ },
230
+ {
231
+ "kind": "single",
232
+ "i": 5,
233
+ "n_calls": 1,
234
+ "finish": "tool_calls",
235
+ "leaked": false,
236
+ "seconds": 6.2,
237
+ "valid_json": true,
238
+ "right_tool": true,
239
+ "ok": true
240
+ },
241
+ {
242
+ "kind": "single",
243
+ "i": 6,
244
+ "n_calls": 1,
245
+ "finish": "tool_calls",
246
+ "leaked": false,
247
+ "seconds": 4.3,
248
+ "valid_json": true,
249
+ "right_tool": true,
250
+ "ok": true
251
+ },
252
+ {
253
+ "kind": "single",
254
+ "i": 7,
255
+ "n_calls": 1,
256
+ "finish": "tool_calls",
257
+ "leaked": false,
258
+ "seconds": 2.8,
259
+ "valid_json": true,
260
+ "right_tool": true,
261
+ "ok": true
262
+ },
263
+ {
264
+ "kind": "single",
265
+ "i": 8,
266
+ "n_calls": 1,
267
+ "finish": "tool_calls",
268
+ "leaked": false,
269
+ "seconds": 3.1,
270
+ "valid_json": true,
271
+ "right_tool": true,
272
+ "ok": true
273
+ },
274
+ {
275
+ "kind": "single",
276
+ "i": 9,
277
+ "n_calls": 1,
278
+ "finish": "tool_calls",
279
+ "leaked": false,
280
+ "seconds": 2.1,
281
+ "valid_json": true,
282
+ "right_tool": true,
283
+ "ok": true
284
+ },
285
+ {
286
+ "kind": "single",
287
+ "i": 10,
288
+ "n_calls": 1,
289
+ "finish": "tool_calls",
290
+ "leaked": false,
291
+ "seconds": 5.8,
292
+ "valid_json": true,
293
+ "right_tool": true,
294
+ "ok": true
295
+ },
296
+ {
297
+ "kind": "single",
298
+ "i": 11,
299
+ "n_calls": 1,
300
+ "finish": "tool_calls",
301
+ "leaked": false,
302
+ "seconds": 4.1,
303
+ "valid_json": true,
304
+ "right_tool": true,
305
+ "ok": true
306
+ },
307
+ {
308
+ "kind": "no_tool",
309
+ "i": 0,
310
+ "ok": true,
311
+ "n_calls": 0,
312
+ "finish": "stop",
313
+ "leaked": false,
314
+ "seconds": 1.9
315
+ },
316
+ {
317
+ "kind": "no_tool",
318
+ "i": 1,
319
+ "ok": true,
320
+ "n_calls": 0,
321
+ "finish": "stop",
322
+ "leaked": false,
323
+ "seconds": 2.7
324
+ },
325
+ {
326
+ "kind": "multi",
327
+ "i": 0,
328
+ "ok": true,
329
+ "rounds": 2,
330
+ "n_calls": 2,
331
+ "finish": "stop",
332
+ "leaked": false,
333
+ "seconds": 7.5,
334
+ "final": "Lisbon is warmer right now: 19\u00b0C vs Oslo at 3\u00b0C \u2014 about 16\u00b0C warmer.",
335
+ "error": null
336
+ },
337
+ {
338
+ "kind": "multi",
339
+ "i": 1,
340
+ "ok": true,
341
+ "rounds": 2,
342
+ "n_calls": 1,
343
+ "finish": "stop",
344
+ "leaked": false,
345
+ "seconds": 5.0,
346
+ "final": "RETRIES is 3.",
347
+ "error": null
348
+ },
349
+ {
350
+ "kind": "multi",
351
+ "i": 2,
352
+ "ok": true,
353
+ "rounds": 2,
354
+ "n_calls": 1,
355
+ "finish": "stop",
356
+ "leaked": false,
357
+ "seconds": 6.8,
358
+ "final": "docs/ARCHITECTURE.md does not exist \u2014 there is no docs/ directory in this repository. The repo contains only Makefile, README.md, app/, src/, and tests/. I can summarize README.md if you'd like.",
359
+ "error": null
360
+ },
361
+ {
362
+ "kind": "multi",
363
+ "i": 3,
364
+ "ok": true,
365
+ "rounds": 2,
366
+ "n_calls": 2,
367
+ "finish": "stop",
368
+ "leaked": false,
369
+ "seconds": 12.7,
370
+ "final": "(1234 \u00d7 57) \u2212 999 = **69,339**\n\n69,339 \u00f7 7 = **9,905.57142857143** (about **9,905.57**)",
371
+ "error": null
372
+ },
373
+ {
374
+ "kind": "multi",
375
+ "i": 4,
376
+ "ok": true,
377
+ "rounds": 2,
378
+ "n_calls": 1,
379
+ "finish": "stop",
380
+ "leaked": false,
381
+ "seconds": 9.0,
382
+ "final": "Customer C-88 has 2 orders in total. The most expensive is A-1042, totaling $129.50 (shipped, tagged gift). The other order, A-1077, is $48.00 and open.",
383
+ "error": null
384
+ }
385
+ ]
386
+ },
387
+ "json": {
388
+ "passed": 9,
389
+ "total": 10,
390
+ "leaked": 0,
391
+ "seconds": 54,
392
+ "rows": [
393
+ {
394
+ "i": 0,
395
+ "ok": true,
396
+ "valid_json": true,
397
+ "finish": "stop",
398
+ "leaked": false,
399
+ "seconds": 9.5,
400
+ "content": null
401
+ },
402
+ {
403
+ "i": 1,
404
+ "ok": true,
405
+ "valid_json": true,
406
+ "finish": "stop",
407
+ "leaked": false,
408
+ "seconds": 2.5,
409
+ "content": null
410
+ },
411
+ {
412
+ "i": 2,
413
+ "ok": true,
414
+ "valid_json": true,
415
+ "finish": "stop",
416
+ "leaked": false,
417
+ "seconds": 5.2,
418
+ "content": null
419
+ },
420
+ {
421
+ "i": 3,
422
+ "ok": true,
423
+ "valid_json": true,
424
+ "finish": "stop",
425
+ "leaked": false,
426
+ "seconds": 3.2,
427
+ "content": null
428
+ },
429
+ {
430
+ "i": 4,
431
+ "ok": true,
432
+ "valid_json": true,
433
+ "finish": "stop",
434
+ "leaked": false,
435
+ "seconds": 6.7,
436
+ "content": null
437
+ },
438
+ {
439
+ "i": 5,
440
+ "ok": true,
441
+ "valid_json": true,
442
+ "finish": "stop",
443
+ "leaked": false,
444
+ "seconds": 3.7,
445
+ "content": null
446
+ },
447
+ {
448
+ "i": 6,
449
+ "ok": true,
450
+ "valid_json": true,
451
+ "finish": "stop",
452
+ "leaked": false,
453
+ "seconds": 3.7,
454
+ "content": null
455
+ },
456
+ {
457
+ "i": 7,
458
+ "ok": true,
459
+ "valid_json": true,
460
+ "finish": "stop",
461
+ "leaked": false,
462
+ "seconds": 2.9,
463
+ "content": null
464
+ },
465
+ {
466
+ "i": 8,
467
+ "ok": false,
468
+ "valid_json": true,
469
+ "finish": "stop",
470
+ "leaked": false,
471
+ "seconds": 8.5,
472
+ "content": "{\"quote\":\"She said \\\"stop\\\" and left.\\\\\"}"
473
+ },
474
+ {
475
+ "i": 9,
476
+ "ok": true,
477
+ "valid_json": true,
478
+ "finish": "stop",
479
+ "leaked": false,
480
+ "seconds": 8.2,
481
+ "content": null
482
+ }
483
+ ]
484
+ },
485
+ "humaneval": {
486
+ "passed": 35,
487
+ "total": 40,
488
+ "leaked": 0,
489
+ "seconds": 612,
490
+ "rows": [
491
+ {
492
+ "task": "HumanEval/0",
493
+ "ok": true,
494
+ "err": null,
495
+ "finish": "stop",
496
+ "leaked": false,
497
+ "tokens": 299,
498
+ "seconds": 9.8
499
+ },
500
+ {
501
+ "task": "HumanEval/4",
502
+ "ok": true,
503
+ "err": null,
504
+ "finish": "stop",
505
+ "leaked": false,
506
+ "tokens": 429,
507
+ "seconds": 13.8
508
+ },
509
+ {
510
+ "task": "HumanEval/8",
511
+ "ok": true,
512
+ "err": null,
513
+ "finish": "stop",
514
+ "leaked": false,
515
+ "tokens": 360,
516
+ "seconds": 11.6
517
+ },
518
+ {
519
+ "task": "HumanEval/12",
520
+ "ok": true,
521
+ "err": null,
522
+ "finish": "stop",
523
+ "leaked": false,
524
+ "tokens": 365,
525
+ "seconds": 11.8
526
+ },
527
+ {
528
+ "task": "HumanEval/16",
529
+ "ok": true,
530
+ "err": null,
531
+ "finish": "stop",
532
+ "leaked": false,
533
+ "tokens": 154,
534
+ "seconds": 5.1
535
+ },
536
+ {
537
+ "task": "HumanEval/20",
538
+ "ok": true,
539
+ "err": null,
540
+ "finish": "stop",
541
+ "leaked": false,
542
+ "tokens": 486,
543
+ "seconds": 15.8
544
+ },
545
+ {
546
+ "task": "HumanEval/24",
547
+ "ok": true,
548
+ "err": null,
549
+ "finish": "stop",
550
+ "leaked": false,
551
+ "tokens": 350,
552
+ "seconds": 11.1
553
+ },
554
+ {
555
+ "task": "HumanEval/28",
556
+ "ok": true,
557
+ "err": null,
558
+ "finish": "stop",
559
+ "leaked": false,
560
+ "tokens": 162,
561
+ "seconds": 5.4
562
+ },
563
+ {
564
+ "task": "HumanEval/32",
565
+ "ok": false,
566
+ "err": "Traceback (most recent call last):\n File \"/private/var/folders/kd/xh61hr5n5c75hx91r3yrt0jw0000gn/T/tmp0gadlh0d/prog.py\", line 26, in <module>\n check(find_zero)\n ^^^^^^^^^\nNameError: name 'find_zero' is not defined\n",
567
+ "finish": "length",
568
+ "leaked": false,
569
+ "tokens": 900,
570
+ "seconds": 29.9
571
+ },
572
+ {
573
+ "task": "HumanEval/36",
574
+ "ok": true,
575
+ "err": null,
576
+ "finish": "length",
577
+ "leaked": false,
578
+ "tokens": 900,
579
+ "seconds": 28.8
580
+ },
581
+ {
582
+ "task": "HumanEval/40",
583
+ "ok": true,
584
+ "err": null,
585
+ "finish": "stop",
586
+ "leaked": false,
587
+ "tokens": 572,
588
+ "seconds": 18.7
589
+ },
590
+ {
591
+ "task": "HumanEval/44",
592
+ "ok": true,
593
+ "err": null,
594
+ "finish": "stop",
595
+ "leaked": false,
596
+ "tokens": 494,
597
+ "seconds": 15.8
598
+ },
599
+ {
600
+ "task": "HumanEval/48",
601
+ "ok": true,
602
+ "err": null,
603
+ "finish": "stop",
604
+ "leaked": false,
605
+ "tokens": 220,
606
+ "seconds": 7.2
607
+ },
608
+ {
609
+ "task": "HumanEval/52",
610
+ "ok": true,
611
+ "err": null,
612
+ "finish": "stop",
613
+ "leaked": false,
614
+ "tokens": 199,
615
+ "seconds": 6.6
616
+ },
617
+ {
618
+ "task": "HumanEval/56",
619
+ "ok": true,
620
+ "err": null,
621
+ "finish": "stop",
622
+ "leaked": false,
623
+ "tokens": 468,
624
+ "seconds": 15.0
625
+ },
626
+ {
627
+ "task": "HumanEval/60",
628
+ "ok": true,
629
+ "err": null,
630
+ "finish": "stop",
631
+ "leaked": false,
632
+ "tokens": 216,
633
+ "seconds": 7.1
634
+ },
635
+ {
636
+ "task": "HumanEval/64",
637
+ "ok": true,
638
+ "err": null,
639
+ "finish": "stop",
640
+ "leaked": false,
641
+ "tokens": 509,
642
+ "seconds": 16.5
643
+ },
644
+ {
645
+ "task": "HumanEval/68",
646
+ "ok": true,
647
+ "err": null,
648
+ "finish": "stop",
649
+ "leaked": false,
650
+ "tokens": 239,
651
+ "seconds": 8.5
652
+ },
653
+ {
654
+ "task": "HumanEval/72",
655
+ "ok": true,
656
+ "err": null,
657
+ "finish": "stop",
658
+ "leaked": false,
659
+ "tokens": 120,
660
+ "seconds": 4.5
661
+ },
662
+ {
663
+ "task": "HumanEval/76",
664
+ "ok": false,
665
+ "err": "Traceback (most recent call last):\n File \"/private/var/folders/kd/xh61hr5n5c75hx91r3yrt0jw0000gn/T/tmph8086k4_/prog.py\", line 21, in <module>\n check(is_simple_power)\n ^^^^^^^^^^^^^^^\nNameError: name 'is_simple_power' is not defined\n",
666
+ "finish": "length",
667
+ "leaked": false,
668
+ "tokens": 900,
669
+ "seconds": 29.1
670
+ },
671
+ {
672
+ "task": "HumanEval/80",
673
+ "ok": true,
674
+ "err": null,
675
+ "finish": "stop",
676
+ "leaked": false,
677
+ "tokens": 422,
678
+ "seconds": 13.6
679
+ },
680
+ {
681
+ "task": "HumanEval/84",
682
+ "ok": false,
683
+ "err": " File \"/private/var/folders/kd/xh61hr5n5c75hx91r3yrt0jw0000gn/T/tmpe6o6tp25/prog.py\", line 3\n \"\"\"Given a positive integer N, return the total sum of its digits in binary.\n ^\nSyntaxError: unterminated triple-quoted string literal (detected at line 28)\n",
684
+ "finish": "length",
685
+ "leaked": false,
686
+ "tokens": 900,
687
+ "seconds": 29.1
688
+ },
689
+ {
690
+ "task": "HumanEval/88",
691
+ "ok": true,
692
+ "err": null,
693
+ "finish": "stop",
694
+ "leaked": false,
695
+ "tokens": 398,
696
+ "seconds": 13.3
697
+ },
698
+ {
699
+ "task": "HumanEval/92",
700
+ "ok": true,
701
+ "err": null,
702
+ "finish": "stop",
703
+ "leaked": false,
704
+ "tokens": 490,
705
+ "seconds": 15.8
706
+ },
707
+ {
708
+ "task": "HumanEval/96",
709
+ "ok": true,
710
+ "err": null,
711
+ "finish": "stop",
712
+ "leaked": false,
713
+ "tokens": 592,
714
+ "seconds": 19.1
715
+ },
716
+ {
717
+ "task": "HumanEval/100",
718
+ "ok": true,
719
+ "err": null,
720
+ "finish": "stop",
721
+ "leaked": false,
722
+ "tokens": 345,
723
+ "seconds": 11.2
724
+ },
725
+ {
726
+ "task": "HumanEval/104",
727
+ "ok": true,
728
+ "err": null,
729
+ "finish": "stop",
730
+ "leaked": false,
731
+ "tokens": 360,
732
+ "seconds": 11.6
733
+ },
734
+ {
735
+ "task": "HumanEval/108",
736
+ "ok": true,
737
+ "err": null,
738
+ "finish": "stop",
739
+ "leaked": false,
740
+ "tokens": 887,
741
+ "seconds": 28.6
742
+ },
743
+ {
744
+ "task": "HumanEval/112",
745
+ "ok": true,
746
+ "err": null,
747
+ "finish": "stop",
748
+ "leaked": false,
749
+ "tokens": 202,
750
+ "seconds": 6.9
751
+ },
752
+ {
753
+ "task": "HumanEval/116",
754
+ "ok": false,
755
+ "err": "Traceback (most recent call last):\n File \"/private/var/folders/kd/xh61hr5n5c75hx91r3yrt0jw0000gn/T/tmp91sbevq_/prog.py\", line 21, in <module>\n check(sort_array)\n ^^^^^^^^^^\nNameError: name 'sort_array' is not defined\n",
756
+ "finish": "length",
757
+ "leaked": false,
758
+ "tokens": 900,
759
+ "seconds": 29.5
760
+ },
761
+ {
762
+ "task": "HumanEval/120",
763
+ "ok": true,
764
+ "err": null,
765
+ "finish": "stop",
766
+ "leaked": false,
767
+ "tokens": 288,
768
+ "seconds": 9.8
769
+ },
770
+ {
771
+ "task": "HumanEval/124",
772
+ "ok": true,
773
+ "err": null,
774
+ "finish": "stop",
775
+ "leaked": false,
776
+ "tokens": 705,
777
+ "seconds": 23.3
778
+ },
779
+ {
780
+ "task": "HumanEval/128",
781
+ "ok": true,
782
+ "err": null,
783
+ "finish": "stop",
784
+ "leaked": false,
785
+ "tokens": 543,
786
+ "seconds": 17.4
787
+ },
788
+ {
789
+ "task": "HumanEval/132",
790
+ "ok": false,
791
+ "err": "Traceback (most recent call last):\n File \"/private/var/folders/kd/xh61hr5n5c75hx91r3yrt0jw0000gn/T/tmpvvmctbl3/prog.py\", line 25, in <module>\n check(is_nested)\n ^^^^^^^^^\nNameError: name 'is_nested' is not defined\n",
792
+ "finish": "length",
793
+ "leaked": false,
794
+ "tokens": 900,
795
+ "seconds": 29.3
796
+ },
797
+ {
798
+ "task": "HumanEval/136",
799
+ "ok": true,
800
+ "err": null,
801
+ "finish": "stop",
802
+ "leaked": false,
803
+ "tokens": 326,
804
+ "seconds": 10.6
805
+ },
806
+ {
807
+ "task": "HumanEval/140",
808
+ "ok": true,
809
+ "err": null,
810
+ "finish": "stop",
811
+ "leaked": false,
812
+ "tokens": 496,
813
+ "seconds": 15.9
814
+ },
815
+ {
816
+ "task": "HumanEval/144",
817
+ "ok": true,
818
+ "err": null,
819
+ "finish": "stop",
820
+ "leaked": false,
821
+ "tokens": 267,
822
+ "seconds": 8.9
823
+ },
824
+ {
825
+ "task": "HumanEval/148",
826
+ "ok": true,
827
+ "err": null,
828
+ "finish": "stop",
829
+ "leaked": false,
830
+ "tokens": 562,
831
+ "seconds": 18.4
832
+ },
833
+ {
834
+ "task": "HumanEval/152",
835
+ "ok": true,
836
+ "err": null,
837
+ "finish": "stop",
838
+ "leaked": false,
839
+ "tokens": 246,
840
+ "seconds": 8.4
841
+ },
842
+ {
843
+ "task": "HumanEval/156",
844
+ "ok": true,
845
+ "err": null,
846
+ "finish": "stop",
847
+ "leaked": false,
848
+ "tokens": 539,
849
+ "seconds": 17.3
850
+ }
851
+ ]
852
+ },
853
+ "gsm8k": {
854
+ "passed": 49,
855
+ "total": 50,
856
+ "leaked": 0,
857
+ "seconds": 414,
858
+ "rows": [
859
+ {
860
+ "i": 0,
861
+ "ok": true,
862
+ "pred": 18.0,
863
+ "gold": 18.0,
864
+ "finish": "stop",
865
+ "leaked": false,
866
+ "tokens": 177,
867
+ "seconds": 5.8
868
+ },
869
+ {
870
+ "i": 26,
871
+ "ok": true,
872
+ "pred": 243.0,
873
+ "gold": 243.0,
874
+ "finish": "stop",
875
+ "leaked": false,
876
+ "tokens": 236,
877
+ "seconds": 7.6
878
+ },
879
+ {
880
+ "i": 52,
881
+ "ok": true,
882
+ "pred": 15.0,
883
+ "gold": 15.0,
884
+ "finish": "stop",
885
+ "leaked": false,
886
+ "tokens": 183,
887
+ "seconds": 6.0
888
+ },
889
+ {
890
+ "i": 78,
891
+ "ok": true,
892
+ "pred": 6.0,
893
+ "gold": 6.0,
894
+ "finish": "stop",
895
+ "leaked": false,
896
+ "tokens": 235,
897
+ "seconds": 7.6
898
+ },
899
+ {
900
+ "i": 104,
901
+ "ok": true,
902
+ "pred": 500.0,
903
+ "gold": 500.0,
904
+ "finish": "stop",
905
+ "leaked": false,
906
+ "tokens": 180,
907
+ "seconds": 5.9
908
+ },
909
+ {
910
+ "i": 130,
911
+ "ok": true,
912
+ "pred": 30.0,
913
+ "gold": 30.0,
914
+ "finish": "stop",
915
+ "leaked": false,
916
+ "tokens": 162,
917
+ "seconds": 5.3
918
+ },
919
+ {
920
+ "i": 156,
921
+ "ok": true,
922
+ "pred": 1400.0,
923
+ "gold": 1400.0,
924
+ "finish": "stop",
925
+ "leaked": false,
926
+ "tokens": 137,
927
+ "seconds": 4.6
928
+ },
929
+ {
930
+ "i": 182,
931
+ "ok": true,
932
+ "pred": 23.0,
933
+ "gold": 23.0,
934
+ "finish": "stop",
935
+ "leaked": false,
936
+ "tokens": 836,
937
+ "seconds": 26.5
938
+ },
939
+ {
940
+ "i": 208,
941
+ "ok": true,
942
+ "pred": 76.0,
943
+ "gold": 76.0,
944
+ "finish": "stop",
945
+ "leaked": false,
946
+ "tokens": 162,
947
+ "seconds": 5.4
948
+ },
949
+ {
950
+ "i": 234,
951
+ "ok": true,
952
+ "pred": 21.0,
953
+ "gold": 21.0,
954
+ "finish": "stop",
955
+ "leaked": false,
956
+ "tokens": 509,
957
+ "seconds": 16.1
958
+ },
959
+ {
960
+ "i": 260,
961
+ "ok": true,
962
+ "pred": 11.0,
963
+ "gold": 11.0,
964
+ "finish": "stop",
965
+ "leaked": false,
966
+ "tokens": 215,
967
+ "seconds": 6.9
968
+ },
969
+ {
970
+ "i": 286,
971
+ "ok": true,
972
+ "pred": 90.0,
973
+ "gold": 90.0,
974
+ "finish": "stop",
975
+ "leaked": false,
976
+ "tokens": 148,
977
+ "seconds": 4.9
978
+ },
979
+ {
980
+ "i": 312,
981
+ "ok": true,
982
+ "pred": 32.0,
983
+ "gold": 32.0,
984
+ "finish": "stop",
985
+ "leaked": false,
986
+ "tokens": 113,
987
+ "seconds": 3.8
988
+ },
989
+ {
990
+ "i": 338,
991
+ "ok": true,
992
+ "pred": 60.0,
993
+ "gold": 60.0,
994
+ "finish": "stop",
995
+ "leaked": false,
996
+ "tokens": 199,
997
+ "seconds": 6.5
998
+ },
999
+ {
1000
+ "i": 364,
1001
+ "ok": true,
1002
+ "pred": 49.0,
1003
+ "gold": 49.0,
1004
+ "finish": "stop",
1005
+ "leaked": false,
1006
+ "tokens": 123,
1007
+ "seconds": 4.1
1008
+ },
1009
+ {
1010
+ "i": 390,
1011
+ "ok": true,
1012
+ "pred": 12.0,
1013
+ "gold": 12.0,
1014
+ "finish": "stop",
1015
+ "leaked": false,
1016
+ "tokens": 118,
1017
+ "seconds": 4.0
1018
+ },
1019
+ {
1020
+ "i": 416,
1021
+ "ok": true,
1022
+ "pred": 56.0,
1023
+ "gold": 56.0,
1024
+ "finish": "stop",
1025
+ "leaked": false,
1026
+ "tokens": 190,
1027
+ "seconds": 6.2
1028
+ },
1029
+ {
1030
+ "i": 442,
1031
+ "ok": true,
1032
+ "pred": 49.0,
1033
+ "gold": 49.0,
1034
+ "finish": "stop",
1035
+ "leaked": false,
1036
+ "tokens": 260,
1037
+ "seconds": 8.4
1038
+ },
1039
+ {
1040
+ "i": 468,
1041
+ "ok": true,
1042
+ "pred": 120.0,
1043
+ "gold": 120.0,
1044
+ "finish": "stop",
1045
+ "leaked": false,
1046
+ "tokens": 195,
1047
+ "seconds": 6.4
1048
+ },
1049
+ {
1050
+ "i": 494,
1051
+ "ok": true,
1052
+ "pred": 21.0,
1053
+ "gold": 21.0,
1054
+ "finish": "stop",
1055
+ "leaked": false,
1056
+ "tokens": 571,
1057
+ "seconds": 18.2
1058
+ },
1059
+ {
1060
+ "i": 520,
1061
+ "ok": true,
1062
+ "pred": 63.0,
1063
+ "gold": 63.0,
1064
+ "finish": "stop",
1065
+ "leaked": false,
1066
+ "tokens": 152,
1067
+ "seconds": 5.2
1068
+ },
1069
+ {
1070
+ "i": 546,
1071
+ "ok": true,
1072
+ "pred": 130.0,
1073
+ "gold": 130.0,
1074
+ "finish": "stop",
1075
+ "leaked": false,
1076
+ "tokens": 329,
1077
+ "seconds": 10.5
1078
+ },
1079
+ {
1080
+ "i": 572,
1081
+ "ok": true,
1082
+ "pred": 100.0,
1083
+ "gold": 100.0,
1084
+ "finish": "stop",
1085
+ "leaked": false,
1086
+ "tokens": 477,
1087
+ "seconds": 15.2
1088
+ },
1089
+ {
1090
+ "i": 598,
1091
+ "ok": true,
1092
+ "pred": 69.0,
1093
+ "gold": 69.0,
1094
+ "finish": "stop",
1095
+ "leaked": false,
1096
+ "tokens": 221,
1097
+ "seconds": 7.1
1098
+ },
1099
+ {
1100
+ "i": 624,
1101
+ "ok": true,
1102
+ "pred": 85.0,
1103
+ "gold": 85.0,
1104
+ "finish": "stop",
1105
+ "leaked": false,
1106
+ "tokens": 232,
1107
+ "seconds": 7.5
1108
+ },
1109
+ {
1110
+ "i": 650,
1111
+ "ok": true,
1112
+ "pred": 142.0,
1113
+ "gold": 142.0,
1114
+ "finish": "stop",
1115
+ "leaked": false,
1116
+ "tokens": 285,
1117
+ "seconds": 9.2
1118
+ },
1119
+ {
1120
+ "i": 676,
1121
+ "ok": true,
1122
+ "pred": 240.0,
1123
+ "gold": 240.0,
1124
+ "finish": "stop",
1125
+ "leaked": false,
1126
+ "tokens": 252,
1127
+ "seconds": 8.1
1128
+ },
1129
+ {
1130
+ "i": 702,
1131
+ "ok": true,
1132
+ "pred": 2800.0,
1133
+ "gold": 2800.0,
1134
+ "finish": "stop",
1135
+ "leaked": false,
1136
+ "tokens": 150,
1137
+ "seconds": 5.1
1138
+ },
1139
+ {
1140
+ "i": 728,
1141
+ "ok": true,
1142
+ "pred": 40.0,
1143
+ "gold": 40.0,
1144
+ "finish": "stop",
1145
+ "leaked": false,
1146
+ "tokens": 228,
1147
+ "seconds": 7.3
1148
+ },
1149
+ {
1150
+ "i": 754,
1151
+ "ok": true,
1152
+ "pred": 89.0,
1153
+ "gold": 89.0,
1154
+ "finish": "stop",
1155
+ "leaked": false,
1156
+ "tokens": 583,
1157
+ "seconds": 18.7
1158
+ },
1159
+ {
1160
+ "i": 780,
1161
+ "ok": true,
1162
+ "pred": 25.0,
1163
+ "gold": 25.0,
1164
+ "finish": "stop",
1165
+ "leaked": false,
1166
+ "tokens": 313,
1167
+ "seconds": 10.2
1168
+ },
1169
+ {
1170
+ "i": 806,
1171
+ "ok": true,
1172
+ "pred": 70.0,
1173
+ "gold": 70.0,
1174
+ "finish": "stop",
1175
+ "leaked": false,
1176
+ "tokens": 326,
1177
+ "seconds": 10.4
1178
+ },
1179
+ {
1180
+ "i": 832,
1181
+ "ok": true,
1182
+ "pred": 42.0,
1183
+ "gold": 42.0,
1184
+ "finish": "stop",
1185
+ "leaked": false,
1186
+ "tokens": 232,
1187
+ "seconds": 7.4
1188
+ },
1189
+ {
1190
+ "i": 858,
1191
+ "ok": true,
1192
+ "pred": 3430.0,
1193
+ "gold": 3430.0,
1194
+ "finish": "stop",
1195
+ "leaked": false,
1196
+ "tokens": 432,
1197
+ "seconds": 13.6
1198
+ },
1199
+ {
1200
+ "i": 884,
1201
+ "ok": true,
1202
+ "pred": 50.0,
1203
+ "gold": 50.0,
1204
+ "finish": "stop",
1205
+ "leaked": false,
1206
+ "tokens": 120,
1207
+ "seconds": 4.1
1208
+ },
1209
+ {
1210
+ "i": 910,
1211
+ "ok": true,
1212
+ "pred": 700.0,
1213
+ "gold": 700.0,
1214
+ "finish": "stop",
1215
+ "leaked": false,
1216
+ "tokens": 130,
1217
+ "seconds": 4.4
1218
+ },
1219
+ {
1220
+ "i": 936,
1221
+ "ok": true,
1222
+ "pred": 250.0,
1223
+ "gold": 250.0,
1224
+ "finish": "stop",
1225
+ "leaked": false,
1226
+ "tokens": 397,
1227
+ "seconds": 12.8
1228
+ },
1229
+ {
1230
+ "i": 962,
1231
+ "ok": false,
1232
+ "pred": 600.0,
1233
+ "gold": 1000.0,
1234
+ "finish": "stop",
1235
+ "leaked": false,
1236
+ "tokens": 213,
1237
+ "seconds": 6.9
1238
+ },
1239
+ {
1240
+ "i": 988,
1241
+ "ok": true,
1242
+ "pred": 6.0,
1243
+ "gold": 6.0,
1244
+ "finish": "stop",
1245
+ "leaked": false,
1246
+ "tokens": 154,
1247
+ "seconds": 5.1
1248
+ },
1249
+ {
1250
+ "i": 1014,
1251
+ "ok": true,
1252
+ "pred": 3.0,
1253
+ "gold": 3.0,
1254
+ "finish": "stop",
1255
+ "leaked": false,
1256
+ "tokens": 406,
1257
+ "seconds": 13.0
1258
+ },
1259
+ {
1260
+ "i": 1040,
1261
+ "ok": true,
1262
+ "pred": 736.0,
1263
+ "gold": 736.0,
1264
+ "finish": "stop",
1265
+ "leaked": false,
1266
+ "tokens": 199,
1267
+ "seconds": 6.5
1268
+ },
1269
+ {
1270
+ "i": 1066,
1271
+ "ok": true,
1272
+ "pred": 14.0,
1273
+ "gold": 14.0,
1274
+ "finish": "stop",
1275
+ "leaked": false,
1276
+ "tokens": 225,
1277
+ "seconds": 7.3
1278
+ },
1279
+ {
1280
+ "i": 1092,
1281
+ "ok": true,
1282
+ "pred": 1080.0,
1283
+ "gold": 1080.0,
1284
+ "finish": "stop",
1285
+ "leaked": false,
1286
+ "tokens": 291,
1287
+ "seconds": 9.4
1288
+ },
1289
+ {
1290
+ "i": 1118,
1291
+ "ok": true,
1292
+ "pred": 5.0,
1293
+ "gold": 5.0,
1294
+ "finish": "stop",
1295
+ "leaked": false,
1296
+ "tokens": 268,
1297
+ "seconds": 8.7
1298
+ },
1299
+ {
1300
+ "i": 1144,
1301
+ "ok": true,
1302
+ "pred": 34.0,
1303
+ "gold": 34.0,
1304
+ "finish": "stop",
1305
+ "leaked": false,
1306
+ "tokens": 143,
1307
+ "seconds": 4.7
1308
+ },
1309
+ {
1310
+ "i": 1170,
1311
+ "ok": true,
1312
+ "pred": 2.0,
1313
+ "gold": 2.0,
1314
+ "finish": "stop",
1315
+ "leaked": false,
1316
+ "tokens": 166,
1317
+ "seconds": 5.5
1318
+ },
1319
+ {
1320
+ "i": 1196,
1321
+ "ok": true,
1322
+ "pred": 75.0,
1323
+ "gold": 75.0,
1324
+ "finish": "stop",
1325
+ "leaked": false,
1326
+ "tokens": 182,
1327
+ "seconds": 5.9
1328
+ },
1329
+ {
1330
+ "i": 1222,
1331
+ "ok": true,
1332
+ "pred": 350.0,
1333
+ "gold": 350.0,
1334
+ "finish": "stop",
1335
+ "leaked": false,
1336
+ "tokens": 274,
1337
+ "seconds": 8.8
1338
+ },
1339
+ {
1340
+ "i": 1248,
1341
+ "ok": true,
1342
+ "pred": 750.0,
1343
+ "gold": 750.0,
1344
+ "finish": "stop",
1345
+ "leaked": false,
1346
+ "tokens": 221,
1347
+ "seconds": 7.3
1348
+ },
1349
+ {
1350
+ "i": 1274,
1351
+ "ok": true,
1352
+ "pred": 38.0,
1353
+ "gold": 38.0,
1354
+ "finish": "stop",
1355
+ "leaked": false,
1356
+ "tokens": 239,
1357
+ "seconds": 7.7
1358
+ }
1359
+ ]
1360
+ }
1361
+ },
1362
+ "tasks_original": {
1363
+ "tools": {
1364
+ "passed": 18,
1365
+ "total": 19,
1366
+ "leaked": 0,
1367
+ "seconds": 104,
1368
+ "rows": [
1369
+ {
1370
+ "kind": "single",
1371
+ "i": 0,
1372
+ "n_calls": 1,
1373
+ "finish": "tool_calls",
1374
+ "leaked": false,
1375
+ "seconds": 3.3,
1376
+ "valid_json": true,
1377
+ "right_tool": true,
1378
+ "ok": true
1379
+ },
1380
+ {
1381
+ "kind": "single",
1382
+ "i": 1,
1383
+ "n_calls": 1,
1384
+ "finish": "tool_calls",
1385
+ "leaked": false,
1386
+ "seconds": 4.0,
1387
+ "valid_json": true,
1388
+ "right_tool": true,
1389
+ "ok": true
1390
+ },
1391
+ {
1392
+ "kind": "single",
1393
+ "i": 2,
1394
+ "n_calls": 1,
1395
+ "finish": "tool_calls",
1396
+ "leaked": false,
1397
+ "seconds": 2.9,
1398
+ "valid_json": true,
1399
+ "right_tool": true,
1400
+ "ok": true
1401
+ },
1402
+ {
1403
+ "kind": "single",
1404
+ "i": 3,
1405
+ "n_calls": 1,
1406
+ "finish": "tool_calls",
1407
+ "leaked": false,
1408
+ "seconds": 2.9,
1409
+ "valid_json": true,
1410
+ "right_tool": true,
1411
+ "ok": true
1412
+ },
1413
+ {
1414
+ "kind": "single",
1415
+ "i": 4,
1416
+ "n_calls": 1,
1417
+ "finish": "tool_calls",
1418
+ "leaked": false,
1419
+ "seconds": 3.2,
1420
+ "valid_json": true,
1421
+ "right_tool": true,
1422
+ "ok": true
1423
+ },
1424
+ {
1425
+ "kind": "single",
1426
+ "i": 5,
1427
+ "n_calls": 1,
1428
+ "finish": "tool_calls",
1429
+ "leaked": false,
1430
+ "seconds": 5.3,
1431
+ "valid_json": true,
1432
+ "right_tool": true,
1433
+ "ok": true
1434
+ },
1435
+ {
1436
+ "kind": "single",
1437
+ "i": 6,
1438
+ "n_calls": 0,
1439
+ "finish": "stop",
1440
+ "leaked": false,
1441
+ "seconds": 4.5,
1442
+ "valid_json": false,
1443
+ "right_tool": false,
1444
+ "ok": false,
1445
+ "got": ""
1446
+ },
1447
+ {
1448
+ "kind": "single",
1449
+ "i": 7,
1450
+ "n_calls": 1,
1451
+ "finish": "tool_calls",
1452
+ "leaked": false,
1453
+ "seconds": 3.3,
1454
+ "valid_json": true,
1455
+ "right_tool": true,
1456
+ "ok": true
1457
+ },
1458
+ {
1459
+ "kind": "single",
1460
+ "i": 8,
1461
+ "n_calls": 1,
1462
+ "finish": "tool_calls",
1463
+ "leaked": false,
1464
+ "seconds": 3.9,
1465
+ "valid_json": true,
1466
+ "right_tool": true,
1467
+ "ok": true
1468
+ },
1469
+ {
1470
+ "kind": "single",
1471
+ "i": 9,
1472
+ "n_calls": 1,
1473
+ "finish": "tool_calls",
1474
+ "leaked": false,
1475
+ "seconds": 2.2,
1476
+ "valid_json": true,
1477
+ "right_tool": true,
1478
+ "ok": true
1479
+ },
1480
+ {
1481
+ "kind": "single",
1482
+ "i": 10,
1483
+ "n_calls": 1,
1484
+ "finish": "tool_calls",
1485
+ "leaked": false,
1486
+ "seconds": 7.6,
1487
+ "valid_json": true,
1488
+ "right_tool": true,
1489
+ "ok": true
1490
+ },
1491
+ {
1492
+ "kind": "single",
1493
+ "i": 11,
1494
+ "n_calls": 1,
1495
+ "finish": "tool_calls",
1496
+ "leaked": false,
1497
+ "seconds": 4.7,
1498
+ "valid_json": true,
1499
+ "right_tool": true,
1500
+ "ok": true
1501
+ },
1502
+ {
1503
+ "kind": "no_tool",
1504
+ "i": 0,
1505
+ "ok": true,
1506
+ "n_calls": 0,
1507
+ "finish": "stop",
1508
+ "leaked": false,
1509
+ "seconds": 2.2
1510
+ },
1511
+ {
1512
+ "kind": "no_tool",
1513
+ "i": 1,
1514
+ "ok": true,
1515
+ "n_calls": 0,
1516
+ "finish": "stop",
1517
+ "leaked": false,
1518
+ "seconds": 3.7
1519
+ },
1520
+ {
1521
+ "kind": "multi",
1522
+ "i": 0,
1523
+ "ok": true,
1524
+ "rounds": 2,
1525
+ "n_calls": 2,
1526
+ "finish": "stop",
1527
+ "leaked": false,
1528
+ "seconds": 9.7,
1529
+ "final": "Lisbon is warmer right now: 19\u00b0C vs Oslo\u2019s 3\u00b0C \u2014 about 16\u00b0C warmer.",
1530
+ "error": null
1531
+ },
1532
+ {
1533
+ "kind": "multi",
1534
+ "i": 1,
1535
+ "ok": true,
1536
+ "rounds": 2,
1537
+ "n_calls": 1,
1538
+ "finish": "stop",
1539
+ "leaked": false,
1540
+ "seconds": 6.2,
1541
+ "final": "RETRIES is 3.",
1542
+ "error": null
1543
+ },
1544
+ {
1545
+ "kind": "multi",
1546
+ "i": 2,
1547
+ "ok": true,
1548
+ "rounds": 2,
1549
+ "n_calls": 1,
1550
+ "finish": "stop",
1551
+ "leaked": false,
1552
+ "seconds": 9.1,
1553
+ "final": "docs/ARCHITECTURE.md does not exist \u2014 there is no `docs/` directory in this repository. The repo contains only `Makefile`, `README.md`, `app/`, `src/`, and `tests/`. I can summarise the README instead if you'd like.",
1554
+ "error": null
1555
+ },
1556
+ {
1557
+ "kind": "multi",
1558
+ "i": 3,
1559
+ "ok": true,
1560
+ "rounds": 2,
1561
+ "n_calls": 2,
1562
+ "finish": "stop",
1563
+ "leaked": false,
1564
+ "seconds": 15.3,
1565
+ "final": "(1234 \u00d7 57) \u2212 999 = **69,339**\n\n69,339 \u00f7 7 = **9,905.57142857143** (about **9,905.57**)",
1566
+ "error": null
1567
+ },
1568
+ {
1569
+ "kind": "multi",
1570
+ "i": 4,
1571
+ "ok": true,
1572
+ "rounds": 2,
1573
+ "n_calls": 1,
1574
+ "finish": "stop",
1575
+ "leaked": false,
1576
+ "seconds": 9.6,
1577
+ "final": "Customer C-88 has 2 orders in total. The most expensive is A-1042, totaling $129.50.",
1578
+ "error": null
1579
+ }
1580
+ ]
1581
+ },
1582
+ "json": {
1583
+ "passed": 9,
1584
+ "total": 10,
1585
+ "leaked": 0,
1586
+ "seconds": 86,
1587
+ "rows": [
1588
+ {
1589
+ "i": 0,
1590
+ "ok": true,
1591
+ "valid_json": true,
1592
+ "finish": "stop",
1593
+ "leaked": false,
1594
+ "seconds": 17.1,
1595
+ "content": null
1596
+ },
1597
+ {
1598
+ "i": 1,
1599
+ "ok": true,
1600
+ "valid_json": true,
1601
+ "finish": "stop",
1602
+ "leaked": false,
1603
+ "seconds": 3.0,
1604
+ "content": null
1605
+ },
1606
+ {
1607
+ "i": 2,
1608
+ "ok": true,
1609
+ "valid_json": true,
1610
+ "finish": "stop",
1611
+ "leaked": false,
1612
+ "seconds": 5.2,
1613
+ "content": null
1614
+ },
1615
+ {
1616
+ "i": 3,
1617
+ "ok": true,
1618
+ "valid_json": true,
1619
+ "finish": "stop",
1620
+ "leaked": false,
1621
+ "seconds": 4.6,
1622
+ "content": null
1623
+ },
1624
+ {
1625
+ "i": 4,
1626
+ "ok": true,
1627
+ "valid_json": true,
1628
+ "finish": "stop",
1629
+ "leaked": false,
1630
+ "seconds": 8.4,
1631
+ "content": null
1632
+ },
1633
+ {
1634
+ "i": 5,
1635
+ "ok": true,
1636
+ "valid_json": true,
1637
+ "finish": "stop",
1638
+ "leaked": false,
1639
+ "seconds": 3.9,
1640
+ "content": null
1641
+ },
1642
+ {
1643
+ "i": 6,
1644
+ "ok": true,
1645
+ "valid_json": true,
1646
+ "finish": "stop",
1647
+ "leaked": false,
1648
+ "seconds": 4.8,
1649
+ "content": null
1650
+ },
1651
+ {
1652
+ "i": 7,
1653
+ "ok": true,
1654
+ "valid_json": true,
1655
+ "finish": "stop",
1656
+ "leaked": false,
1657
+ "seconds": 3.3,
1658
+ "content": null
1659
+ },
1660
+ {
1661
+ "i": 8,
1662
+ "ok": false,
1663
+ "valid_json": true,
1664
+ "finish": "stop",
1665
+ "leaked": false,
1666
+ "seconds": 27.3,
1667
+ "content": "{\"quote\":\"She said \\\"stop\\\" and left.\\\\\"}"
1668
+ },
1669
+ {
1670
+ "i": 9,
1671
+ "ok": true,
1672
+ "valid_json": true,
1673
+ "finish": "stop",
1674
+ "leaked": false,
1675
+ "seconds": 8.0,
1676
+ "content": null
1677
+ }
1678
+ ]
1679
+ },
1680
+ "humaneval": {
1681
+ "passed": 35,
1682
+ "total": 40,
1683
+ "leaked": 0,
1684
+ "seconds": 759,
1685
+ "rows": [
1686
+ {
1687
+ "task": "HumanEval/0",
1688
+ "ok": true,
1689
+ "err": null,
1690
+ "finish": "stop",
1691
+ "leaked": false,
1692
+ "tokens": 427,
1693
+ "seconds": 17.7
1694
+ },
1695
+ {
1696
+ "task": "HumanEval/4",
1697
+ "ok": true,
1698
+ "err": null,
1699
+ "finish": "stop",
1700
+ "leaked": false,
1701
+ "tokens": 285,
1702
+ "seconds": 11.9
1703
+ },
1704
+ {
1705
+ "task": "HumanEval/8",
1706
+ "ok": true,
1707
+ "err": null,
1708
+ "finish": "stop",
1709
+ "leaked": false,
1710
+ "tokens": 493,
1711
+ "seconds": 20.3
1712
+ },
1713
+ {
1714
+ "task": "HumanEval/12",
1715
+ "ok": true,
1716
+ "err": null,
1717
+ "finish": "stop",
1718
+ "leaked": false,
1719
+ "tokens": 573,
1720
+ "seconds": 23.6
1721
+ },
1722
+ {
1723
+ "task": "HumanEval/16",
1724
+ "ok": true,
1725
+ "err": null,
1726
+ "finish": "stop",
1727
+ "leaked": false,
1728
+ "tokens": 227,
1729
+ "seconds": 9.4
1730
+ },
1731
+ {
1732
+ "task": "HumanEval/20",
1733
+ "ok": true,
1734
+ "err": null,
1735
+ "finish": "stop",
1736
+ "leaked": false,
1737
+ "tokens": 625,
1738
+ "seconds": 26.1
1739
+ },
1740
+ {
1741
+ "task": "HumanEval/24",
1742
+ "ok": true,
1743
+ "err": null,
1744
+ "finish": "stop",
1745
+ "leaked": false,
1746
+ "tokens": 249,
1747
+ "seconds": 10.3
1748
+ },
1749
+ {
1750
+ "task": "HumanEval/28",
1751
+ "ok": true,
1752
+ "err": null,
1753
+ "finish": "stop",
1754
+ "leaked": false,
1755
+ "tokens": 165,
1756
+ "seconds": 7.0
1757
+ },
1758
+ {
1759
+ "task": "HumanEval/32",
1760
+ "ok": false,
1761
+ "err": "Traceback (most recent call last):\n File \"/private/var/folders/kd/xh61hr5n5c75hx91r3yrt0jw0000gn/T/tmp38s2jvzb/prog.py\", line 26, in <module>\n check(find_zero)\n ^^^^^^^^^\nNameError: name 'find_zero' is not defined\n",
1762
+ "finish": "length",
1763
+ "leaked": false,
1764
+ "tokens": 900,
1765
+ "seconds": 38.6
1766
+ },
1767
+ {
1768
+ "task": "HumanEval/36",
1769
+ "ok": true,
1770
+ "err": null,
1771
+ "finish": "stop",
1772
+ "leaked": false,
1773
+ "tokens": 640,
1774
+ "seconds": 26.4
1775
+ },
1776
+ {
1777
+ "task": "HumanEval/40",
1778
+ "ok": true,
1779
+ "err": null,
1780
+ "finish": "stop",
1781
+ "leaked": false,
1782
+ "tokens": 641,
1783
+ "seconds": 27.0
1784
+ },
1785
+ {
1786
+ "task": "HumanEval/44",
1787
+ "ok": true,
1788
+ "err": null,
1789
+ "finish": "stop",
1790
+ "leaked": false,
1791
+ "tokens": 365,
1792
+ "seconds": 15.1
1793
+ },
1794
+ {
1795
+ "task": "HumanEval/48",
1796
+ "ok": true,
1797
+ "err": null,
1798
+ "finish": "stop",
1799
+ "leaked": false,
1800
+ "tokens": 210,
1801
+ "seconds": 8.8
1802
+ },
1803
+ {
1804
+ "task": "HumanEval/52",
1805
+ "ok": true,
1806
+ "err": null,
1807
+ "finish": "stop",
1808
+ "leaked": false,
1809
+ "tokens": 239,
1810
+ "seconds": 10.0
1811
+ },
1812
+ {
1813
+ "task": "HumanEval/56",
1814
+ "ok": true,
1815
+ "err": null,
1816
+ "finish": "stop",
1817
+ "leaked": false,
1818
+ "tokens": 270,
1819
+ "seconds": 11.3
1820
+ },
1821
+ {
1822
+ "task": "HumanEval/60",
1823
+ "ok": true,
1824
+ "err": null,
1825
+ "finish": "stop",
1826
+ "leaked": false,
1827
+ "tokens": 269,
1828
+ "seconds": 11.2
1829
+ },
1830
+ {
1831
+ "task": "HumanEval/64",
1832
+ "ok": true,
1833
+ "err": null,
1834
+ "finish": "stop",
1835
+ "leaked": false,
1836
+ "tokens": 497,
1837
+ "seconds": 20.6
1838
+ },
1839
+ {
1840
+ "task": "HumanEval/68",
1841
+ "ok": true,
1842
+ "err": null,
1843
+ "finish": "stop",
1844
+ "leaked": false,
1845
+ "tokens": 335,
1846
+ "seconds": 14.7
1847
+ },
1848
+ {
1849
+ "task": "HumanEval/72",
1850
+ "ok": true,
1851
+ "err": null,
1852
+ "finish": "stop",
1853
+ "leaked": false,
1854
+ "tokens": 154,
1855
+ "seconds": 7.0
1856
+ },
1857
+ {
1858
+ "task": "HumanEval/76",
1859
+ "ok": false,
1860
+ "err": "Traceback (most recent call last):\n File \"/private/var/folders/kd/xh61hr5n5c75hx91r3yrt0jw0000gn/T/tmp0mwlmoao/prog.py\", line 21, in <module>\n check(is_simple_power)\n ^^^^^^^^^^^^^^^\nNameError: name 'is_simple_power' is not defined\n",
1861
+ "finish": "length",
1862
+ "leaked": false,
1863
+ "tokens": 900,
1864
+ "seconds": 37.7
1865
+ },
1866
+ {
1867
+ "task": "HumanEval/80",
1868
+ "ok": true,
1869
+ "err": null,
1870
+ "finish": "stop",
1871
+ "leaked": false,
1872
+ "tokens": 243,
1873
+ "seconds": 10.2
1874
+ },
1875
+ {
1876
+ "task": "HumanEval/84",
1877
+ "ok": true,
1878
+ "err": null,
1879
+ "finish": "stop",
1880
+ "leaked": false,
1881
+ "tokens": 756,
1882
+ "seconds": 31.5
1883
+ },
1884
+ {
1885
+ "task": "HumanEval/88",
1886
+ "ok": true,
1887
+ "err": null,
1888
+ "finish": "stop",
1889
+ "leaked": false,
1890
+ "tokens": 266,
1891
+ "seconds": 11.6
1892
+ },
1893
+ {
1894
+ "task": "HumanEval/92",
1895
+ "ok": true,
1896
+ "err": null,
1897
+ "finish": "stop",
1898
+ "leaked": false,
1899
+ "tokens": 455,
1900
+ "seconds": 18.9
1901
+ },
1902
+ {
1903
+ "task": "HumanEval/96",
1904
+ "ok": true,
1905
+ "err": null,
1906
+ "finish": "stop",
1907
+ "leaked": false,
1908
+ "tokens": 555,
1909
+ "seconds": 23.2
1910
+ },
1911
+ {
1912
+ "task": "HumanEval/100",
1913
+ "ok": true,
1914
+ "err": null,
1915
+ "finish": "stop",
1916
+ "leaked": false,
1917
+ "tokens": 353,
1918
+ "seconds": 14.7
1919
+ },
1920
+ {
1921
+ "task": "HumanEval/104",
1922
+ "ok": true,
1923
+ "err": null,
1924
+ "finish": "stop",
1925
+ "leaked": false,
1926
+ "tokens": 412,
1927
+ "seconds": 17.1
1928
+ },
1929
+ {
1930
+ "task": "HumanEval/108",
1931
+ "ok": true,
1932
+ "err": null,
1933
+ "finish": "stop",
1934
+ "leaked": false,
1935
+ "tokens": 557,
1936
+ "seconds": 23.1
1937
+ },
1938
+ {
1939
+ "task": "HumanEval/112",
1940
+ "ok": true,
1941
+ "err": null,
1942
+ "finish": "stop",
1943
+ "leaked": false,
1944
+ "tokens": 288,
1945
+ "seconds": 12.2
1946
+ },
1947
+ {
1948
+ "task": "HumanEval/116",
1949
+ "ok": false,
1950
+ "err": "Traceback (most recent call last):\n File \"/private/var/folders/kd/xh61hr5n5c75hx91r3yrt0jw0000gn/T/tmp4huqxua3/prog.py\", line 21, in <module>\n check(sort_array)\n ^^^^^^^^^^\nNameError: name 'sort_array' is not defined\n",
1951
+ "finish": "length",
1952
+ "leaked": false,
1953
+ "tokens": 900,
1954
+ "seconds": 37.9
1955
+ },
1956
+ {
1957
+ "task": "HumanEval/120",
1958
+ "ok": true,
1959
+ "err": null,
1960
+ "finish": "stop",
1961
+ "leaked": false,
1962
+ "tokens": 331,
1963
+ "seconds": 14.2
1964
+ },
1965
+ {
1966
+ "task": "HumanEval/124",
1967
+ "ok": true,
1968
+ "err": null,
1969
+ "finish": "stop",
1970
+ "leaked": false,
1971
+ "tokens": 464,
1972
+ "seconds": 19.8
1973
+ },
1974
+ {
1975
+ "task": "HumanEval/128",
1976
+ "ok": true,
1977
+ "err": null,
1978
+ "finish": "stop",
1979
+ "leaked": false,
1980
+ "tokens": 398,
1981
+ "seconds": 16.5
1982
+ },
1983
+ {
1984
+ "task": "HumanEval/132",
1985
+ "ok": false,
1986
+ "err": "Traceback (most recent call last):\n File \"/private/var/folders/kd/xh61hr5n5c75hx91r3yrt0jw0000gn/T/tmp_1cc4urv/prog.py\", line 25, in <module>\n check(is_nested)\n ^^^^^^^^^\nNameError: name 'is_nested' is not defined\n",
1987
+ "finish": "length",
1988
+ "leaked": false,
1989
+ "tokens": 900,
1990
+ "seconds": 37.8
1991
+ },
1992
+ {
1993
+ "task": "HumanEval/136",
1994
+ "ok": true,
1995
+ "err": null,
1996
+ "finish": "stop",
1997
+ "leaked": false,
1998
+ "tokens": 328,
1999
+ "seconds": 13.8
2000
+ },
2001
+ {
2002
+ "task": "HumanEval/140",
2003
+ "ok": false,
2004
+ "err": "Traceback (most recent call last):\n File \"/private/var/folders/kd/xh61hr5n5c75hx91r3yrt0jw0000gn/T/tmp7bobb2r_/prog.py\", line 16, in <module>\n check(fix_spaces)\n ^^^^^^^^^^\nNameError: name 'fix_spaces' is not defined\n",
2005
+ "finish": "length",
2006
+ "leaked": false,
2007
+ "tokens": 900,
2008
+ "seconds": 37.6
2009
+ },
2010
+ {
2011
+ "task": "HumanEval/144",
2012
+ "ok": true,
2013
+ "err": null,
2014
+ "finish": "stop",
2015
+ "leaked": false,
2016
+ "tokens": 238,
2017
+ "seconds": 10.2
2018
+ },
2019
+ {
2020
+ "task": "HumanEval/148",
2021
+ "ok": true,
2022
+ "err": null,
2023
+ "finish": "stop",
2024
+ "leaked": false,
2025
+ "tokens": 582,
2026
+ "seconds": 24.6
2027
+ },
2028
+ {
2029
+ "task": "HumanEval/152",
2030
+ "ok": true,
2031
+ "err": null,
2032
+ "finish": "stop",
2033
+ "leaked": false,
2034
+ "tokens": 137,
2035
+ "seconds": 6.3
2036
+ },
2037
+ {
2038
+ "task": "HumanEval/156",
2039
+ "ok": true,
2040
+ "err": null,
2041
+ "finish": "stop",
2042
+ "leaked": false,
2043
+ "tokens": 507,
2044
+ "seconds": 21.0
2045
+ }
2046
+ ]
2047
+ },
2048
+ "gsm8k": {
2049
+ "passed": 48,
2050
+ "total": 50,
2051
+ "leaked": 0,
2052
+ "seconds": 543,
2053
+ "rows": [
2054
+ {
2055
+ "i": 0,
2056
+ "ok": true,
2057
+ "pred": 18.0,
2058
+ "gold": 18.0,
2059
+ "finish": "stop",
2060
+ "leaked": false,
2061
+ "tokens": 150,
2062
+ "seconds": 6.4
2063
+ },
2064
+ {
2065
+ "i": 26,
2066
+ "ok": true,
2067
+ "pred": 243.0,
2068
+ "gold": 243.0,
2069
+ "finish": "stop",
2070
+ "leaked": false,
2071
+ "tokens": 192,
2072
+ "seconds": 8.0
2073
+ },
2074
+ {
2075
+ "i": 52,
2076
+ "ok": true,
2077
+ "pred": 15.0,
2078
+ "gold": 15.0,
2079
+ "finish": "stop",
2080
+ "leaked": false,
2081
+ "tokens": 204,
2082
+ "seconds": 8.5
2083
+ },
2084
+ {
2085
+ "i": 78,
2086
+ "ok": true,
2087
+ "pred": 6.0,
2088
+ "gold": 6.0,
2089
+ "finish": "stop",
2090
+ "leaked": false,
2091
+ "tokens": 234,
2092
+ "seconds": 9.8
2093
+ },
2094
+ {
2095
+ "i": 104,
2096
+ "ok": true,
2097
+ "pred": 500.0,
2098
+ "gold": 500.0,
2099
+ "finish": "stop",
2100
+ "leaked": false,
2101
+ "tokens": 180,
2102
+ "seconds": 7.5
2103
+ },
2104
+ {
2105
+ "i": 130,
2106
+ "ok": true,
2107
+ "pred": 30.0,
2108
+ "gold": 30.0,
2109
+ "finish": "stop",
2110
+ "leaked": false,
2111
+ "tokens": 124,
2112
+ "seconds": 5.4
2113
+ },
2114
+ {
2115
+ "i": 156,
2116
+ "ok": true,
2117
+ "pred": 1400.0,
2118
+ "gold": 1400.0,
2119
+ "finish": "stop",
2120
+ "leaked": false,
2121
+ "tokens": 164,
2122
+ "seconds": 7.0
2123
+ },
2124
+ {
2125
+ "i": 182,
2126
+ "ok": false,
2127
+ "pred": 24.0,
2128
+ "gold": 23.0,
2129
+ "finish": "stop",
2130
+ "leaked": false,
2131
+ "tokens": 851,
2132
+ "seconds": 34.9
2133
+ },
2134
+ {
2135
+ "i": 208,
2136
+ "ok": true,
2137
+ "pred": 76.0,
2138
+ "gold": 76.0,
2139
+ "finish": "stop",
2140
+ "leaked": false,
2141
+ "tokens": 139,
2142
+ "seconds": 5.9
2143
+ },
2144
+ {
2145
+ "i": 234,
2146
+ "ok": true,
2147
+ "pred": 21.0,
2148
+ "gold": 21.0,
2149
+ "finish": "stop",
2150
+ "leaked": false,
2151
+ "tokens": 613,
2152
+ "seconds": 25.1
2153
+ },
2154
+ {
2155
+ "i": 260,
2156
+ "ok": true,
2157
+ "pred": 11.0,
2158
+ "gold": 11.0,
2159
+ "finish": "stop",
2160
+ "leaked": false,
2161
+ "tokens": 210,
2162
+ "seconds": 8.7
2163
+ },
2164
+ {
2165
+ "i": 286,
2166
+ "ok": true,
2167
+ "pred": 90.0,
2168
+ "gold": 90.0,
2169
+ "finish": "stop",
2170
+ "leaked": false,
2171
+ "tokens": 148,
2172
+ "seconds": 6.2
2173
+ },
2174
+ {
2175
+ "i": 312,
2176
+ "ok": true,
2177
+ "pred": 32.0,
2178
+ "gold": 32.0,
2179
+ "finish": "stop",
2180
+ "leaked": false,
2181
+ "tokens": 113,
2182
+ "seconds": 4.8
2183
+ },
2184
+ {
2185
+ "i": 338,
2186
+ "ok": true,
2187
+ "pred": 60.0,
2188
+ "gold": 60.0,
2189
+ "finish": "stop",
2190
+ "leaked": false,
2191
+ "tokens": 186,
2192
+ "seconds": 7.8
2193
+ },
2194
+ {
2195
+ "i": 364,
2196
+ "ok": true,
2197
+ "pred": 49.0,
2198
+ "gold": 49.0,
2199
+ "finish": "stop",
2200
+ "leaked": false,
2201
+ "tokens": 115,
2202
+ "seconds": 5.0
2203
+ },
2204
+ {
2205
+ "i": 390,
2206
+ "ok": true,
2207
+ "pred": 12.0,
2208
+ "gold": 12.0,
2209
+ "finish": "stop",
2210
+ "leaked": false,
2211
+ "tokens": 295,
2212
+ "seconds": 12.2
2213
+ },
2214
+ {
2215
+ "i": 416,
2216
+ "ok": true,
2217
+ "pred": 56.0,
2218
+ "gold": 56.0,
2219
+ "finish": "stop",
2220
+ "leaked": false,
2221
+ "tokens": 185,
2222
+ "seconds": 7.7
2223
+ },
2224
+ {
2225
+ "i": 442,
2226
+ "ok": true,
2227
+ "pred": 49.0,
2228
+ "gold": 49.0,
2229
+ "finish": "stop",
2230
+ "leaked": false,
2231
+ "tokens": 229,
2232
+ "seconds": 9.6
2233
+ },
2234
+ {
2235
+ "i": 468,
2236
+ "ok": true,
2237
+ "pred": 120.0,
2238
+ "gold": 120.0,
2239
+ "finish": "stop",
2240
+ "leaked": false,
2241
+ "tokens": 225,
2242
+ "seconds": 9.3
2243
+ },
2244
+ {
2245
+ "i": 494,
2246
+ "ok": true,
2247
+ "pred": 21.0,
2248
+ "gold": 21.0,
2249
+ "finish": "stop",
2250
+ "leaked": false,
2251
+ "tokens": 702,
2252
+ "seconds": 28.9
2253
+ },
2254
+ {
2255
+ "i": 520,
2256
+ "ok": true,
2257
+ "pred": 63.0,
2258
+ "gold": 63.0,
2259
+ "finish": "stop",
2260
+ "leaked": false,
2261
+ "tokens": 182,
2262
+ "seconds": 7.8
2263
+ },
2264
+ {
2265
+ "i": 546,
2266
+ "ok": true,
2267
+ "pred": 130.0,
2268
+ "gold": 130.0,
2269
+ "finish": "stop",
2270
+ "leaked": false,
2271
+ "tokens": 382,
2272
+ "seconds": 15.7
2273
+ },
2274
+ {
2275
+ "i": 572,
2276
+ "ok": true,
2277
+ "pred": 100.0,
2278
+ "gold": 100.0,
2279
+ "finish": "stop",
2280
+ "leaked": false,
2281
+ "tokens": 217,
2282
+ "seconds": 9.1
2283
+ },
2284
+ {
2285
+ "i": 598,
2286
+ "ok": true,
2287
+ "pred": 69.0,
2288
+ "gold": 69.0,
2289
+ "finish": "stop",
2290
+ "leaked": false,
2291
+ "tokens": 189,
2292
+ "seconds": 7.8
2293
+ },
2294
+ {
2295
+ "i": 624,
2296
+ "ok": true,
2297
+ "pred": 85.0,
2298
+ "gold": 85.0,
2299
+ "finish": "stop",
2300
+ "leaked": false,
2301
+ "tokens": 192,
2302
+ "seconds": 8.0
2303
+ },
2304
+ {
2305
+ "i": 650,
2306
+ "ok": true,
2307
+ "pred": 142.0,
2308
+ "gold": 142.0,
2309
+ "finish": "stop",
2310
+ "leaked": false,
2311
+ "tokens": 259,
2312
+ "seconds": 10.8
2313
+ },
2314
+ {
2315
+ "i": 676,
2316
+ "ok": true,
2317
+ "pred": 240.0,
2318
+ "gold": 240.0,
2319
+ "finish": "stop",
2320
+ "leaked": false,
2321
+ "tokens": 260,
2322
+ "seconds": 10.8
2323
+ },
2324
+ {
2325
+ "i": 702,
2326
+ "ok": true,
2327
+ "pred": 2800.0,
2328
+ "gold": 2800.0,
2329
+ "finish": "stop",
2330
+ "leaked": false,
2331
+ "tokens": 179,
2332
+ "seconds": 7.6
2333
+ },
2334
+ {
2335
+ "i": 728,
2336
+ "ok": true,
2337
+ "pred": 40.0,
2338
+ "gold": 40.0,
2339
+ "finish": "stop",
2340
+ "leaked": false,
2341
+ "tokens": 232,
2342
+ "seconds": 9.6
2343
+ },
2344
+ {
2345
+ "i": 754,
2346
+ "ok": true,
2347
+ "pred": 89.0,
2348
+ "gold": 89.0,
2349
+ "finish": "stop",
2350
+ "leaked": false,
2351
+ "tokens": 612,
2352
+ "seconds": 25.3
2353
+ },
2354
+ {
2355
+ "i": 780,
2356
+ "ok": true,
2357
+ "pred": 25.0,
2358
+ "gold": 25.0,
2359
+ "finish": "stop",
2360
+ "leaked": false,
2361
+ "tokens": 390,
2362
+ "seconds": 16.2
2363
+ },
2364
+ {
2365
+ "i": 806,
2366
+ "ok": true,
2367
+ "pred": 70.0,
2368
+ "gold": 70.0,
2369
+ "finish": "stop",
2370
+ "leaked": false,
2371
+ "tokens": 321,
2372
+ "seconds": 13.3
2373
+ },
2374
+ {
2375
+ "i": 832,
2376
+ "ok": true,
2377
+ "pred": 42.0,
2378
+ "gold": 42.0,
2379
+ "finish": "stop",
2380
+ "leaked": false,
2381
+ "tokens": 321,
2382
+ "seconds": 13.1
2383
+ },
2384
+ {
2385
+ "i": 858,
2386
+ "ok": true,
2387
+ "pred": 3430.0,
2388
+ "gold": 3430.0,
2389
+ "finish": "stop",
2390
+ "leaked": false,
2391
+ "tokens": 429,
2392
+ "seconds": 17.5
2393
+ },
2394
+ {
2395
+ "i": 884,
2396
+ "ok": true,
2397
+ "pred": 50.0,
2398
+ "gold": 50.0,
2399
+ "finish": "stop",
2400
+ "leaked": false,
2401
+ "tokens": 102,
2402
+ "seconds": 4.5
2403
+ },
2404
+ {
2405
+ "i": 910,
2406
+ "ok": true,
2407
+ "pred": 700.0,
2408
+ "gold": 700.0,
2409
+ "finish": "stop",
2410
+ "leaked": false,
2411
+ "tokens": 137,
2412
+ "seconds": 5.9
2413
+ },
2414
+ {
2415
+ "i": 936,
2416
+ "ok": true,
2417
+ "pred": 250.0,
2418
+ "gold": 250.0,
2419
+ "finish": "stop",
2420
+ "leaked": false,
2421
+ "tokens": 284,
2422
+ "seconds": 11.9
2423
+ },
2424
+ {
2425
+ "i": 962,
2426
+ "ok": false,
2427
+ "pred": 600.0,
2428
+ "gold": 1000.0,
2429
+ "finish": "stop",
2430
+ "leaked": false,
2431
+ "tokens": 135,
2432
+ "seconds": 5.8
2433
+ },
2434
+ {
2435
+ "i": 988,
2436
+ "ok": true,
2437
+ "pred": 6.0,
2438
+ "gold": 6.0,
2439
+ "finish": "stop",
2440
+ "leaked": false,
2441
+ "tokens": 581,
2442
+ "seconds": 23.8
2443
+ },
2444
+ {
2445
+ "i": 1014,
2446
+ "ok": true,
2447
+ "pred": 3.0,
2448
+ "gold": 3.0,
2449
+ "finish": "stop",
2450
+ "leaked": false,
2451
+ "tokens": 381,
2452
+ "seconds": 15.8
2453
+ },
2454
+ {
2455
+ "i": 1040,
2456
+ "ok": true,
2457
+ "pred": 736.0,
2458
+ "gold": 736.0,
2459
+ "finish": "stop",
2460
+ "leaked": false,
2461
+ "tokens": 199,
2462
+ "seconds": 8.3
2463
+ },
2464
+ {
2465
+ "i": 1066,
2466
+ "ok": true,
2467
+ "pred": 14.0,
2468
+ "gold": 14.0,
2469
+ "finish": "stop",
2470
+ "leaked": false,
2471
+ "tokens": 196,
2472
+ "seconds": 8.2
2473
+ },
2474
+ {
2475
+ "i": 1092,
2476
+ "ok": true,
2477
+ "pred": 1080.0,
2478
+ "gold": 1080.0,
2479
+ "finish": "stop",
2480
+ "leaked": false,
2481
+ "tokens": 200,
2482
+ "seconds": 8.4
2483
+ },
2484
+ {
2485
+ "i": 1118,
2486
+ "ok": true,
2487
+ "pred": 5.0,
2488
+ "gold": 5.0,
2489
+ "finish": "stop",
2490
+ "leaked": false,
2491
+ "tokens": 268,
2492
+ "seconds": 11.1
2493
+ },
2494
+ {
2495
+ "i": 1144,
2496
+ "ok": true,
2497
+ "pred": 34.0,
2498
+ "gold": 34.0,
2499
+ "finish": "stop",
2500
+ "leaked": false,
2501
+ "tokens": 176,
2502
+ "seconds": 7.4
2503
+ },
2504
+ {
2505
+ "i": 1170,
2506
+ "ok": true,
2507
+ "pred": 2.0,
2508
+ "gold": 2.0,
2509
+ "finish": "stop",
2510
+ "leaked": false,
2511
+ "tokens": 177,
2512
+ "seconds": 7.4
2513
+ },
2514
+ {
2515
+ "i": 1196,
2516
+ "ok": true,
2517
+ "pred": 75.0,
2518
+ "gold": 75.0,
2519
+ "finish": "stop",
2520
+ "leaked": false,
2521
+ "tokens": 191,
2522
+ "seconds": 8.0
2523
+ },
2524
+ {
2525
+ "i": 1222,
2526
+ "ok": true,
2527
+ "pred": 350.0,
2528
+ "gold": 350.0,
2529
+ "finish": "stop",
2530
+ "leaked": false,
2531
+ "tokens": 210,
2532
+ "seconds": 8.7
2533
+ },
2534
+ {
2535
+ "i": 1248,
2536
+ "ok": true,
2537
+ "pred": 750.0,
2538
+ "gold": 750.0,
2539
+ "finish": "stop",
2540
+ "leaked": false,
2541
+ "tokens": 188,
2542
+ "seconds": 8.0
2543
+ },
2544
+ {
2545
+ "i": 1274,
2546
+ "ok": true,
2547
+ "pred": 38.0,
2548
+ "gold": 38.0,
2549
+ "finish": "stop",
2550
+ "leaked": false,
2551
+ "tokens": 211,
2552
+ "seconds": 8.8
2553
+ }
2554
+ ]
2555
+ }
2556
+ },
2557
+ "hermes_agent": {
2558
+ "this_build": {
2559
+ "passed": 4,
2560
+ "total": 4,
2561
+ "effort": "medium",
2562
+ "tasks": {
2563
+ "explain": {
2564
+ "ok": true,
2565
+ "seconds": 92.1,
2566
+ "exit_code": 0,
2567
+ "api_calls": 4,
2568
+ "tool_calls": 3,
2569
+ "tools": [
2570
+ {
2571
+ "name": "terminal",
2572
+ "input": "{\"command\": \"ls -la ..."
2573
+ },
2574
+ {
2575
+ "name": "terminal",
2576
+ "input": "{\"command\": \"ls -R s..."
2577
+ },
2578
+ {
2579
+ "name": "terminal",
2580
+ "input": "{\"command\": \"python3..."
2581
+ }
2582
+ ],
2583
+ "final": "Tidepool is a small in-memory priority job queue (src/jobqueue.py, using a heap with priority and insertion-order tie-breaking). Run its tests with `python3 -m unittest discover -s tests -v` \u2014 note that one test currently fails: `test_pop_returns_job` fails because `pop()` returns the insertion counter (index 1 of the heap tuple) instead of the job (index 2).",
2584
+ "tail": "",
2585
+ "stderr": ""
2586
+ },
2587
+ "nonce": {
2588
+ "ok": true,
2589
+ "seconds": 84.9,
2590
+ "exit_code": 0,
2591
+ "api_calls": 4,
2592
+ "tool_calls": 3,
2593
+ "tools": [
2594
+ {
2595
+ "name": "terminal",
2596
+ "input": "{\"command\": \"ls -la;..."
2597
+ },
2598
+ {
2599
+ "name": "read_file",
2600
+ "input": "{\"path\": \"notes/secr..."
2601
+ },
2602
+ {
2603
+ "name": "read_file",
2604
+ "input": "{\"path\": \"README.md\"..."
2605
+ }
2606
+ ],
2607
+ "final": "The access code in notes/secret.txt is **ZX-FE4E89E0**.",
2608
+ "tail": "",
2609
+ "stderr": ""
2610
+ },
2611
+ "fix_bug": {
2612
+ "ok": true,
2613
+ "seconds": 159.6,
2614
+ "exit_code": 0,
2615
+ "api_calls": 8,
2616
+ "tool_calls": 8,
2617
+ "tools": [
2618
+ {
2619
+ "name": "terminal",
2620
+ "input": "{\"command\": \"pwd && ..."
2621
+ },
2622
+ {
2623
+ "name": "terminal",
2624
+ "input": "{\"command\": \"cat REA..."
2625
+ },
2626
+ {
2627
+ "name": "read_file",
2628
+ "input": "{\"path\": \"src/jobque..."
2629
+ },
2630
+ {
2631
+ "name": "read_file",
2632
+ "input": "{\"path\": \"tests/test..."
2633
+ },
2634
+ {
2635
+ "name": "terminal",
2636
+ "input": "{\"command\": \"python3..."
2637
+ },
2638
+ {
2639
+ "name": "patch",
2640
+ "input": "{\"new_string\": \" ..."
2641
+ },
2642
+ {
2643
+ "name": "patch",
2644
+ "input": "{\"new_string\": \" ..."
2645
+ },
2646
+ {
2647
+ "name": "terminal",
2648
+ "input": "{\"command\": \"python3..."
2649
+ }
2650
+ ],
2651
+ "final": "Fixed. `src/jobqueue.py:16` `pop()` returned index `[1]` of the heap entry \u2014 the tie-breaking insertion counter (e.g. `1`) \u2014 instead of `[2]`, the job itself. Tests now pass (2/2).",
2652
+ "tail": "",
2653
+ "stderr": ""
2654
+ },
2655
+ "write_run": {
2656
+ "ok": true,
2657
+ "seconds": 147.4,
2658
+ "exit_code": 0,
2659
+ "api_calls": 6,
2660
+ "tool_calls": 6,
2661
+ "tools": [
2662
+ {
2663
+ "name": "terminal",
2664
+ "input": "{\"command\": \"pwd && ..."
2665
+ },
2666
+ {
2667
+ "name": "read_file",
2668
+ "input": "{\"path\": \"README.md\"..."
2669
+ },
2670
+ {
2671
+ "name": "terminal",
2672
+ "input": "{\"command\": \"ls -la ..."
2673
+ },
2674
+ {
2675
+ "name": "terminal",
2676
+ "input": "{\"command\": \"mkdir -..."
2677
+ },
2678
+ {
2679
+ "name": "write_file",
2680
+ "input": "{\"content\": \"#!/usr/..."
2681
+ },
2682
+ {
2683
+ "name": "terminal",
2684
+ "input": "{\"command\": \"python3..."
2685
+ }
2686
+ ],
2687
+ "final": "`scripts/wordcount.py` created and run \u2014 it printed **20**.\n\n(Word count is whitespace-split, so \"#\", \"##\", and the `python3 -m unittest ...` line count as words too.)",
2688
+ "tail": "",
2689
+ "stderr": ""
2690
+ }
2691
+ }
2692
+ },
2693
+ "original": {
2694
+ "passed": 4,
2695
+ "total": 4,
2696
+ "effort": "medium",
2697
+ "tasks": {
2698
+ "explain": {
2699
+ "ok": true,
2700
+ "seconds": 130.1,
2701
+ "exit_code": 0,
2702
+ "api_calls": 5,
2703
+ "tool_calls": 4,
2704
+ "tools": [
2705
+ {
2706
+ "name": "terminal",
2707
+ "input": "{\"command\": \"ls -la ..."
2708
+ },
2709
+ {
2710
+ "name": "terminal",
2711
+ "input": "{\"command\": \"pwd && ..."
2712
+ },
2713
+ {
2714
+ "name": "terminal",
2715
+ "input": "{\"command\": \"head -4..."
2716
+ },
2717
+ {
2718
+ "name": "terminal",
2719
+ "input": "{\"command\": \"python3..."
2720
+ }
2721
+ ],
2722
+ "final": "Tidepool is a small in-memory job queue that serves higher-priority jobs first, with insertion order breaking ties. Run the tests with `python3 -m unittest discover -s tests -v` \u2014 but be aware that one test currently fails: `pop()` in `src/jobqueue.py:17` returns the counter (`[1]`) instead of the job (`[2]`), so `test_pop_returns_job` fails (`1 != 'fix outage'`). Want me to fix that line?",
2723
+ "tail": "",
2724
+ "stderr": ""
2725
+ },
2726
+ "nonce": {
2727
+ "ok": true,
2728
+ "seconds": 75.4,
2729
+ "exit_code": 0,
2730
+ "api_calls": 3,
2731
+ "tool_calls": 2,
2732
+ "tools": [
2733
+ {
2734
+ "name": "terminal",
2735
+ "input": "{\"command\": \"pwd && ..."
2736
+ },
2737
+ {
2738
+ "name": "terminal",
2739
+ "input": "{\"command\": \"ls -la ..."
2740
+ }
2741
+ ],
2742
+ "final": "The access code in notes/secret.txt is ZX-F2E46563.",
2743
+ "tail": "",
2744
+ "stderr": ""
2745
+ },
2746
+ "fix_bug": {
2747
+ "ok": true,
2748
+ "seconds": 170.4,
2749
+ "exit_code": 0,
2750
+ "api_calls": 7,
2751
+ "tool_calls": 7,
2752
+ "tools": [
2753
+ {
2754
+ "name": "terminal",
2755
+ "input": "{\"command\": \"pwd && ..."
2756
+ },
2757
+ {
2758
+ "name": "terminal",
2759
+ "input": "{\"command\": \"cd /Use..."
2760
+ },
2761
+ {
2762
+ "name": "terminal",
2763
+ "input": "{\"command\": \"cd /Use..."
2764
+ },
2765
+ {
2766
+ "name": "read_file",
2767
+ "input": "{\"path\": \"/Users/hos..."
2768
+ },
2769
+ {
2770
+ "name": "read_file",
2771
+ "input": "{\"path\": \"/Users/hos..."
2772
+ },
2773
+ {
2774
+ "name": "terminal",
2775
+ "input": "{\"command\": \"cd /Use..."
2776
+ },
2777
+ {
2778
+ "name": "terminal",
2779
+ "input": "{\"command\": \"cd /Use..."
2780
+ }
2781
+ ],
2782
+ "final": "Fixed. The bug was in `src/jobqueue.py:16`: `pop()` returned heap index `[1]` (the insertion counter) instead of `[2]` (the job). One-line change to `heappop(...)[2]`.\n\nTests now pass:\n```\ntest_len ... ok\ntest_pop_returns_job ... ok\nRan 2 tests \u2014 OK\n```\n\nNote: there's a `notes/secret.txt` in the scratch directory which I deliberately did not read or use.",
2783
+ "tail": "",
2784
+ "stderr": ""
2785
+ },
2786
+ "write_run": {
2787
+ "ok": true,
2788
+ "seconds": 168.6,
2789
+ "exit_code": 0,
2790
+ "api_calls": 6,
2791
+ "tool_calls": 5,
2792
+ "tools": [
2793
+ {
2794
+ "name": "terminal",
2795
+ "input": "{\"command\": \"ls -la ..."
2796
+ },
2797
+ {
2798
+ "name": "terminal",
2799
+ "input": "{\"command\": \"pwd && ..."
2800
+ },
2801
+ {
2802
+ "name": "terminal",
2803
+ "input": "{\"command\": \"ls /Use..."
2804
+ },
2805
+ {
2806
+ "name": "write_file",
2807
+ "input": "{\"content\": \"#!/usr/..."
2808
+ },
2809
+ {
2810
+ "name": "terminal",
2811
+ "input": "{\"command\": \"python3..."
2812
+ }
2813
+ ],
2814
+ "final": "`scripts/wordcount.py` created and run. It printed **20**.\n\n(The README holds 20 whitespace-separated tokens: `# Tidepool A small in-memory job queue with priorities. ## Run the tests python3 -m unittest discover -s tests -v`.)",
2815
+ "tail": "",
2816
+ "stderr": ""
2817
+ }
2818
+ }
2819
+ }
2820
+ },
2821
+ "benchmark": {
2822
+ "this_build": {
2823
+ "load_seconds": 2.5,
2824
+ "weights_in_memory_gb": 11.91,
2825
+ "rows": [
2826
+ {
2827
+ "case": "short",
2828
+ "draft": false,
2829
+ "prompt_tokens": 28,
2830
+ "generation_tokens": 256,
2831
+ "prompt_tps": 107.1,
2832
+ "generation_tps": 33.83,
2833
+ "peak_gb": 12.68,
2834
+ "wall_s": 7.8
2835
+ },
2836
+ {
2837
+ "case": "short",
2838
+ "draft": true,
2839
+ "prompt_tokens": 28,
2840
+ "generation_tokens": 256,
2841
+ "prompt_tps": 119.2,
2842
+ "generation_tps": 20.6,
2843
+ "peak_gb": 12.98,
2844
+ "wall_s": 12.7
2845
+ },
2846
+ {
2847
+ "case": "ctx1k",
2848
+ "draft": false,
2849
+ "prompt_tokens": 1024,
2850
+ "generation_tokens": 256,
2851
+ "prompt_tps": 432.0,
2852
+ "generation_tps": 29.96,
2853
+ "peak_gb": 15.47,
2854
+ "wall_s": 10.9
2855
+ },
2856
+ {
2857
+ "case": "ctx1k",
2858
+ "draft": true,
2859
+ "prompt_tokens": 1024,
2860
+ "generation_tokens": 256,
2861
+ "prompt_tps": 437.5,
2862
+ "generation_tps": 29.34,
2863
+ "peak_gb": 15.59,
2864
+ "wall_s": 11.1
2865
+ },
2866
+ {
2867
+ "case": "ctx4k",
2868
+ "draft": false,
2869
+ "prompt_tokens": 4023,
2870
+ "generation_tokens": 256,
2871
+ "prompt_tps": 399.4,
2872
+ "generation_tps": 28.83,
2873
+ "peak_gb": 16.92,
2874
+ "wall_s": 19.0
2875
+ },
2876
+ {
2877
+ "case": "ctx4k",
2878
+ "draft": true,
2879
+ "prompt_tokens": 4023,
2880
+ "generation_tokens": 256,
2881
+ "prompt_tps": 390.8,
2882
+ "generation_tps": 30.77,
2883
+ "peak_gb": 17.07,
2884
+ "wall_s": 18.6
2885
+ }
2886
+ ],
2887
+ "mlx": "0.32.4"
2888
+ },
2889
+ "original": {
2890
+ "load_seconds": 4.9,
2891
+ "weights_in_memory_gb": 18.19,
2892
+ "rows": [
2893
+ {
2894
+ "case": "short",
2895
+ "draft": false,
2896
+ "prompt_tokens": 28,
2897
+ "generation_tokens": 256,
2898
+ "prompt_tps": 91.1,
2899
+ "generation_tps": 25.91,
2900
+ "peak_gb": 20.62,
2901
+ "wall_s": 10.2
2902
+ },
2903
+ {
2904
+ "case": "short",
2905
+ "draft": true,
2906
+ "prompt_tokens": 28,
2907
+ "generation_tokens": 256,
2908
+ "prompt_tps": 100.4,
2909
+ "generation_tps": 19.46,
2910
+ "peak_gb": 20.98,
2911
+ "wall_s": 13.5
2912
+ },
2913
+ {
2914
+ "case": "ctx1k",
2915
+ "draft": false,
2916
+ "prompt_tokens": 1024,
2917
+ "generation_tokens": 256,
2918
+ "prompt_tps": 429.0,
2919
+ "generation_tps": 23.25,
2920
+ "peak_gb": 23.46,
2921
+ "wall_s": 13.4
2922
+ },
2923
+ {
2924
+ "case": "ctx1k",
2925
+ "draft": true,
2926
+ "prompt_tokens": 1024,
2927
+ "generation_tokens": 256,
2928
+ "prompt_tps": 434.4,
2929
+ "generation_tps": 31.77,
2930
+ "peak_gb": 23.59,
2931
+ "wall_s": 10.4
2932
+ },
2933
+ {
2934
+ "case": "ctx4k",
2935
+ "draft": false,
2936
+ "prompt_tokens": 4023,
2937
+ "generation_tokens": 256,
2938
+ "prompt_tps": 379.1,
2939
+ "generation_tps": 22.33,
2940
+ "peak_gb": 24.92,
2941
+ "wall_s": 22.1
2942
+ },
2943
+ {
2944
+ "case": "ctx4k",
2945
+ "draft": true,
2946
+ "prompt_tokens": 4023,
2947
+ "generation_tokens": 256,
2948
+ "prompt_tps": 377.1,
2949
+ "generation_tps": 22.02,
2950
+ "peak_gb": 25.07,
2951
+ "wall_s": 22.3
2952
+ }
2953
+ ],
2954
+ "mlx": "0.32.4"
2955
+ }
2956
+ },
2957
+ "memory_and_image_check": {
2958
+ "this_build": {
2959
+ "weights_in_memory_gib": 11.91,
2960
+ "rows": [
2961
+ {
2962
+ "prompt_tokens": 224,
2963
+ "prefill_step": 1024,
2964
+ "prompt_tps": 336.8,
2965
+ "generation_tps": 33.5,
2966
+ "peak_gib": 12.66
2967
+ },
2968
+ {
2969
+ "prompt_tokens": 4023,
2970
+ "prefill_step": 1024,
2971
+ "prompt_tps": 404.5,
2972
+ "generation_tps": 29.11,
2973
+ "peak_gib": 16.23
2974
+ },
2975
+ {
2976
+ "prompt_tokens": 4023,
2977
+ "prefill_step": 256,
2978
+ "prompt_tps": 356.9,
2979
+ "generation_tps": 29.16,
2980
+ "peak_gib": 13.49
2981
+ },
2982
+ {
2983
+ "prompt_tokens": 8024,
2984
+ "prefill_step": 256,
2985
+ "prompt_tps": 346.0,
2986
+ "generation_tps": 27.61,
2987
+ "peak_gib": 13.67
2988
+ },
2989
+ {
2990
+ "prompt_tokens": 16024,
2991
+ "prefill_step": 256,
2992
+ "prompt_tps": 328.2,
2993
+ "generation_tps": 24.94,
2994
+ "peak_gib": 14.11
2995
+ }
2996
+ ],
2997
+ "vision": [
2998
+ {
2999
+ "image": "data/images/cat.jpeg",
3000
+ "question": "What animal is in this image, and where is it? One sentence.",
3001
+ "answer": "/fence in background. Need answer concise.\n\nNeed be careful: animal looks like a Pallas's cat (manul), a small wild cat adapted to snow. It is standing on snow in what looks like a zoo/enclosure with birch logs. Provide one sentence.\n\nNo need for tools beyond thinking already done. Final one sentence.A Pallas\u2019s cat (snow leopard) is standing on snow in a fenced enclosure with birch logs behind it."
3002
+ },
3003
+ {
3004
+ "image": "data/images/chart.png",
3005
+ "question": "Which quarter sold the most units, how many, and what is the order reference?",
3006
+ "answer": "The user wants answer based on chart. Need inspect image values: title QUARTERLY UNITS SOLD - ORDER REF 4821. Bars: Q1 120, Q2 185, Q3 90, Q4 240. Highest Q4 with 240 units, order ref 4821. Need respond concise. Ensure mention order reference. No need extra.Q4 sold the most units: 240. Order reference: 4821."
3007
+ }
3008
+ ]
3009
+ },
3010
+ "original": {
3011
+ "weights_in_memory_gib": 18.19,
3012
+ "rows": [
3013
+ {
3014
+ "prompt_tokens": 224,
3015
+ "prefill_step": 1024,
3016
+ "prompt_tps": 302.1,
3017
+ "generation_tps": 25.82,
3018
+ "peak_gib": 20.27
3019
+ },
3020
+ {
3021
+ "prompt_tokens": 4023,
3022
+ "prefill_step": 1024,
3023
+ "prompt_tps": 403.6,
3024
+ "generation_tps": 22.82,
3025
+ "peak_gib": 24.23
3026
+ },
3027
+ {
3028
+ "prompt_tokens": 4023,
3029
+ "prefill_step": 256,
3030
+ "prompt_tps": 349.2,
3031
+ "generation_tps": 22.83,
3032
+ "peak_gib": 21.37
3033
+ },
3034
+ {
3035
+ "prompt_tokens": 8024,
3036
+ "prefill_step": 256,
3037
+ "prompt_tps": 323.8,
3038
+ "generation_tps": 21.69,
3039
+ "peak_gib": 21.63
3040
+ },
3041
+ {
3042
+ "prompt_tokens": 16024,
3043
+ "prefill_step": 256,
3044
+ "prompt_tps": 310.8,
3045
+ "generation_tps": 19.9,
3046
+ "peak_gib": 22.06
3047
+ }
3048
+ ],
3049
+ "vision": [
3050
+ {
3051
+ "image": "data/images/cat.jpeg",
3052
+ "question": "What animal is in this image, and where is it? One sentence.",
3053
+ "answer": "at (snow leopard-like) or perhaps a lynx/bobcat? It looks like a Pallas's cat (manul) - stocky, short legs, thick fur, speckled face. It's walking on snow with a wooden/log fence background. Need answer concise.\n\nAnimal: Pallas's cat (manul). Location: on snowy ground in front of a wooden fence/log wall. One sentence.A Pallas\u2019s cat (manul) is walking on snowy ground in front of a wooden log fence."
3054
+ },
3055
+ {
3056
+ "image": "data/images/chart.png",
3057
+ "question": "Which quarter sold the most units, how many, and what is the order reference?",
3058
+ "answer": "The user is asking about a bar chart. Let me read the values:\n- Q1: 120\n- Q2: 185\n- Q3: 90\n- Q4: 240\n\nThe highest is Q4 with 240 units. The order reference is 4821 (from \"QUARTERLY UNITS SOLD - ORDER REF 4821\").Q4 sold the most units, with 240 units. The order reference is 4821."
3059
+ }
3060
+ ]
3061
+ }
3062
+ },
3063
+ "cached_decoding_vs_full_forward_original": {
3064
+ "bf16_chunk128": {
3065
+ "tokens": 4001,
3066
+ "sequences": 10,
3067
+ "max_abs_logit_diff": 22.842241287231445,
3068
+ "kl_mean": 0.048451148127050876,
3069
+ "kl_max": 13.973586082458496,
3070
+ "nll_full_forward": 0.7997520344759697,
3071
+ "nll_cached_steps": 0.7905213371034683,
3072
+ "top1_agreement": 0.970757310672332,
3073
+ "seconds": 211
3074
+ },
3075
+ "bf16_chunk2048": {
3076
+ "tokens": 4001,
3077
+ "sequences": 10,
3078
+ "max_abs_logit_diff": 21.5530948638916,
3079
+ "kl_mean": 0.04029263491467409,
3080
+ "kl_max": 18.789379119873047,
3081
+ "nll_full_forward": 0.7997520344759697,
3082
+ "nll_cached_steps": 0.7888766982143625,
3083
+ "top1_agreement": 0.970757310672332,
3084
+ "seconds": 209
3085
+ },
3086
+ "fp32_chunk128": {
3087
+ "tokens": 1192,
3088
+ "sequences": 4,
3089
+ "max_abs_logit_diff": 15.65640640258789,
3090
+ "kl_mean": 0.006313716402616854,
3091
+ "kl_max": 2.067986488342285,
3092
+ "nll_full_forward": 1.674901450240372,
3093
+ "nll_cached_steps": 1.6847560293722472,
3094
+ "top1_agreement": 0.9966442953020134,
3095
+ "seconds": 80
3096
+ },
3097
+ "fp32_chunk2048": {
3098
+ "tokens": 1192,
3099
+ "sequences": 4,
3100
+ "max_abs_logit_diff": 15.656400680541992,
3101
+ "kl_mean": 0.006313734596712877,
3102
+ "kl_max": 2.067990303039551,
3103
+ "nll_full_forward": 1.674901450240372,
3104
+ "nll_cached_steps": 1.6847562597902028,
3105
+ "top1_agreement": 0.9966442953020134,
3106
+ "seconds": 78
3107
+ }
3108
+ },
3109
+ "reference_parity": {
3110
+ "description": "Teacher-forced logits on identical token ids. Reference: transformers PR 51050 (commit 6173a2f6) on CPU, expert weights dequantized by the reference op. MLX: mlx-vlm main 111ac668, mlx 0.32.4, M4 Pro.",
3111
+ "mlx_fp32_vs_torch_fp32": {
3112
+ "image_cat": {
3113
+ "tokens": 492,
3114
+ "max_abs_logit_diff": 0.44274044036865234,
3115
+ "mean_abs_logit_diff": 0.00039586810658722305,
3116
+ "kl_mean": 7.587283164404417e-06,
3117
+ "kl_max": 0.003581067210598087,
3118
+ "top1_agreement": 1.0,
3119
+ "next_token_acc_ref": 0.054989816700611,
3120
+ "next_token_acc_test": 0.054989816700611,
3121
+ "nll_ref": 20.089199532342388,
3122
+ "nll_test": 20.089406827516832
3123
+ },
3124
+ "image_chart": {
3125
+ "tokens": 356,
3126
+ "max_abs_logit_diff": 8.646623849868774,
3127
+ "mean_abs_logit_diff": 0.042977310928109254,
3128
+ "kl_mean": 0.004497380610732641,
3129
+ "kl_max": 0.33672980618712545,
3130
+ "top1_agreement": 0.9719101123595506,
3131
+ "next_token_acc_ref": 0.09014084507042254,
3132
+ "next_token_acc_test": 0.09014084507042254,
3133
+ "nll_ref": 17.659265003278247,
3134
+ "nll_test": 17.646260290588184
3135
+ },
3136
+ "long_context": {
3137
+ "tokens": 1353,
3138
+ "max_abs_logit_diff": 7.7301411628723145,
3139
+ "mean_abs_logit_diff": 0.015888066121287594,
3140
+ "kl_mean": 1.7773926378733907e-05,
3141
+ "kl_max": 0.010085841474386945,
3142
+ "top1_agreement": 1.0,
3143
+ "next_token_acc_ref": 0.9060650887573964,
3144
+ "next_token_acc_test": 0.9060650887573964,
3145
+ "nll_ref": 0.5817908920328585,
3146
+ "nll_test": 0.5817055003276868
3147
+ },
3148
+ "multilingual": {
3149
+ "tokens": 65,
3150
+ "max_abs_logit_diff": 0.00010979175567626953,
3151
+ "mean_abs_logit_diff": 8.978016029090604e-06,
3152
+ "kl_mean": 2.0563244512179743e-11,
3153
+ "kl_max": 2.554999550892132e-10,
3154
+ "top1_agreement": 1.0,
3155
+ "next_token_acc_ref": 0.5625,
3156
+ "next_token_acc_test": 0.5625,
3157
+ "nll_ref": 3.7906516032194206,
3158
+ "nll_test": 3.7906535542633177
3159
+ },
3160
+ "short_chat": {
3161
+ "tokens": 37,
3162
+ "max_abs_logit_diff": 0.0001277923583984375,
3163
+ "mean_abs_logit_diff": 9.456596459282991e-06,
3164
+ "kl_mean": 2.240959362721911e-11,
3165
+ "kl_max": 1.7461242067926594e-10,
3166
+ "top1_agreement": 1.0,
3167
+ "next_token_acc_ref": 0.5277777777777778,
3168
+ "next_token_acc_test": 0.5277777777777778,
3169
+ "nll_ref": 5.443080670467236,
3170
+ "nll_test": 5.443082489456533
3171
+ },
3172
+ "thinking_chat": {
3173
+ "tokens": 102,
3174
+ "max_abs_logit_diff": 0.00010538101196289062,
3175
+ "mean_abs_logit_diff": 9.532523792852414e-06,
3176
+ "kl_mean": 1.8364102705662738e-11,
3177
+ "kl_max": 4.242171022827198e-10,
3178
+ "top1_agreement": 1.0,
3179
+ "next_token_acc_ref": 0.6138613861386139,
3180
+ "next_token_acc_test": 0.6138613861386139,
3181
+ "nll_ref": 3.7741031649505654,
3182
+ "nll_test": 3.774103348220411
3183
+ },
3184
+ "tool_roundtrip": {
3185
+ "tokens": 329,
3186
+ "max_abs_logit_diff": 0.000240325927734375,
3187
+ "mean_abs_logit_diff": 1.1768514219006727e-05,
3188
+ "kl_mean": 2.007265502036474e-11,
3189
+ "kl_max": 1.3455370426613493e-09,
3190
+ "top1_agreement": 1.0,
3191
+ "next_token_acc_ref": 0.7134146341463414,
3192
+ "next_token_acc_test": 0.7134146341463414,
3193
+ "nll_ref": 2.7811447828381786,
3194
+ "nll_test": 2.7811444905435425
3195
+ }
3196
+ },
3197
+ "torch_bf16_vs_torch_fp32": {
3198
+ "image_cat": {
3199
+ "tokens": 492,
3200
+ "max_abs_logit_diff": 20.917954444885254,
3201
+ "mean_abs_logit_diff": 0.4790812881133246,
3202
+ "kl_mean": 0.2554885655286341,
3203
+ "kl_max": 11.36766312077503,
3204
+ "top1_agreement": 0.8780487804878049,
3205
+ "next_token_acc_ref": 0.054989816700611,
3206
+ "next_token_acc_test": 0.05702647657841141,
3207
+ "nll_ref": 20.089199532342388,
3208
+ "nll_test": 20.049141792477794
3209
+ },
3210
+ "image_chart": {
3211
+ "tokens": 356,
3212
+ "max_abs_logit_diff": 23.090723991394043,
3213
+ "mean_abs_logit_diff": 0.6762670648069056,
3214
+ "kl_mean": 0.5612026261553928,
3215
+ "kl_max": 13.180007741732275,
3216
+ "top1_agreement": 0.7443820224719101,
3217
+ "next_token_acc_ref": 0.09014084507042254,
3218
+ "next_token_acc_test": 0.09014084507042254,
3219
+ "nll_ref": 17.659265003278247,
3220
+ "nll_test": 17.667298816635764
3221
+ },
3222
+ "long_context": {
3223
+ "tokens": 1353,
3224
+ "max_abs_logit_diff": 24.625191688537598,
3225
+ "mean_abs_logit_diff": 1.5311537717340122,
3226
+ "kl_mean": 0.22785581219171008,
3227
+ "kl_max": 16.08681830122339,
3228
+ "top1_agreement": 0.9556541019955654,
3229
+ "next_token_acc_ref": 0.9060650887573964,
3230
+ "next_token_acc_test": 0.9038461538461539,
3231
+ "nll_ref": 0.5817908920328585,
3232
+ "nll_test": 0.5574441898794273
3233
+ },
3234
+ "multilingual": {
3235
+ "tokens": 65,
3236
+ "max_abs_logit_diff": 4.211472511291504,
3237
+ "mean_abs_logit_diff": 0.2116919903498743,
3238
+ "kl_mean": 0.008501311125484017,
3239
+ "kl_max": 0.08177949573595175,
3240
+ "top1_agreement": 0.9692307692307692,
3241
+ "next_token_acc_ref": 0.5625,
3242
+ "next_token_acc_test": 0.578125,
3243
+ "nll_ref": 3.7906516032194206,
3244
+ "nll_test": 3.8383234493145535
3245
+ },
3246
+ "short_chat": {
3247
+ "tokens": 37,
3248
+ "max_abs_logit_diff": 3.2938833236694336,
3249
+ "mean_abs_logit_diff": 0.23400576493509498,
3250
+ "kl_mean": 0.013311820955924617,
3251
+ "kl_max": 0.12193033467452069,
3252
+ "top1_agreement": 0.972972972972973,
3253
+ "next_token_acc_ref": 0.5277777777777778,
3254
+ "next_token_acc_test": 0.5277777777777778,
3255
+ "nll_ref": 5.443080670467236,
3256
+ "nll_test": 5.375631014013916
3257
+ },
3258
+ "thinking_chat": {
3259
+ "tokens": 102,
3260
+ "max_abs_logit_diff": 14.65269660949707,
3261
+ "mean_abs_logit_diff": 0.26605868399973925,
3262
+ "kl_mean": 0.0358142972599525,
3263
+ "kl_max": 2.5301440916711586,
3264
+ "top1_agreement": 0.9509803921568627,
3265
+ "next_token_acc_ref": 0.6138613861386139,
3266
+ "next_token_acc_test": 0.6039603960396039,
3267
+ "nll_ref": 3.7741031649505654,
3268
+ "nll_test": 3.840478707203926
3269
+ },
3270
+ "tool_roundtrip": {
3271
+ "tokens": 329,
3272
+ "max_abs_logit_diff": 23.453279495239258,
3273
+ "mean_abs_logit_diff": 0.6738596847750378,
3274
+ "kl_mean": 0.11465851963454897,
3275
+ "kl_max": 12.569975177111543,
3276
+ "top1_agreement": 0.9361702127659575,
3277
+ "next_token_acc_ref": 0.7134146341463414,
3278
+ "next_token_acc_test": 0.7225609756097561,
3279
+ "nll_ref": 2.7811447828381786,
3280
+ "nll_test": 2.7678284988449815
3281
+ }
3282
+ },
3283
+ "mlx_bf16_vs_torch_fp32": {
3284
+ "image_cat": {
3285
+ "tokens": 492,
3286
+ "max_abs_logit_diff": 25.776443481445312,
3287
+ "mean_abs_logit_diff": 0.5739572311203393,
3288
+ "kl_mean": 0.32945401742291003,
3289
+ "kl_max": 13.424443716518757,
3290
+ "top1_agreement": 0.8536585365853658,
3291
+ "next_token_acc_ref": 0.054989816700611,
3292
+ "next_token_acc_test": 0.059063136456211814,
3293
+ "nll_ref": 20.089199532342388,
3294
+ "nll_test": 20.10524947315169
3295
+ },
3296
+ "image_chart": {
3297
+ "tokens": 356,
3298
+ "max_abs_logit_diff": 22.61000919342041,
3299
+ "mean_abs_logit_diff": 0.5653393714458804,
3300
+ "kl_mean": 0.356097592533074,
3301
+ "kl_max": 7.309836100553708,
3302
+ "top1_agreement": 0.7556179775280899,
3303
+ "next_token_acc_ref": 0.09014084507042254,
3304
+ "next_token_acc_test": 0.09014084507042254,
3305
+ "nll_ref": 17.659265003278247,
3306
+ "nll_test": 17.608667050969412
3307
+ },
3308
+ "long_context": {
3309
+ "tokens": 1353,
3310
+ "max_abs_logit_diff": 24.45859718322754,
3311
+ "mean_abs_logit_diff": 1.473904154273382,
3312
+ "kl_mean": 0.19963105957870853,
3313
+ "kl_max": 15.416439084029014,
3314
+ "top1_agreement": 0.9645232815964523,
3315
+ "next_token_acc_ref": 0.9060650887573964,
3316
+ "next_token_acc_test": 0.9090236686390533,
3317
+ "nll_ref": 0.5817908920328585,
3318
+ "nll_test": 0.5440008410165926
3319
+ },
3320
+ "multilingual": {
3321
+ "tokens": 65,
3322
+ "max_abs_logit_diff": 2.285825252532959,
3323
+ "mean_abs_logit_diff": 0.19048134856934984,
3324
+ "kl_mean": 0.007670106536798406,
3325
+ "kl_max": 0.22094075094875654,
3326
+ "top1_agreement": 0.9846153846153847,
3327
+ "next_token_acc_ref": 0.5625,
3328
+ "next_token_acc_test": 0.5625,
3329
+ "nll_ref": 3.7906516032194206,
3330
+ "nll_test": 3.793026564653868
3331
+ },
3332
+ "short_chat": {
3333
+ "tokens": 37,
3334
+ "max_abs_logit_diff": 4.4355528354644775,
3335
+ "mean_abs_logit_diff": 0.22052635319363867,
3336
+ "kl_mean": 0.00854530879380758,
3337
+ "kl_max": 0.07499803025163768,
3338
+ "top1_agreement": 0.9459459459459459,
3339
+ "next_token_acc_ref": 0.5277777777777778,
3340
+ "next_token_acc_test": 0.5277777777777778,
3341
+ "nll_ref": 5.443080670467236,
3342
+ "nll_test": 5.343368736261641
3343
+ },
3344
+ "thinking_chat": {
3345
+ "tokens": 102,
3346
+ "max_abs_logit_diff": 14.612149477005005,
3347
+ "mean_abs_logit_diff": 0.2570170134252102,
3348
+ "kl_mean": 0.03432711815246316,
3349
+ "kl_max": 2.6583591721311954,
3350
+ "top1_agreement": 0.9705882352941176,
3351
+ "next_token_acc_ref": 0.6138613861386139,
3352
+ "next_token_acc_test": 0.6039603960396039,
3353
+ "nll_ref": 3.7741031649505654,
3354
+ "nll_test": 3.8346220063091554
3355
+ },
3356
+ "tool_roundtrip": {
3357
+ "tokens": 329,
3358
+ "max_abs_logit_diff": 20.791677236557007,
3359
+ "mean_abs_logit_diff": 0.7246150221283114,
3360
+ "kl_mean": 0.13842127199136015,
3361
+ "kl_max": 7.050130603024795,
3362
+ "top1_agreement": 0.9361702127659575,
3363
+ "next_token_acc_ref": 0.7134146341463414,
3364
+ "next_token_acc_test": 0.698170731707317,
3365
+ "nll_ref": 2.7811447828381786,
3366
+ "nll_test": 2.8992298776234766
3367
+ }
3368
+ },
3369
+ "mlx_bf16_vs_torch_bf16": {
3370
+ "image_cat": {
3371
+ "tokens": 492,
3372
+ "max_abs_logit_diff": 23.71006679534912,
3373
+ "mean_abs_logit_diff": 0.5271237628672776,
3374
+ "kl_mean": 0.2635479704693753,
3375
+ "kl_max": 6.784772286464859,
3376
+ "top1_agreement": 0.8597560975609756,
3377
+ "next_token_acc_ref": 0.05702647657841141,
3378
+ "next_token_acc_test": 0.059063136456211814,
3379
+ "nll_ref": 20.049141792477794,
3380
+ "nll_test": 20.10524947315169
3381
+ },
3382
+ "image_chart": {
3383
+ "tokens": 356,
3384
+ "max_abs_logit_diff": 24.48257350921631,
3385
+ "mean_abs_logit_diff": 0.7330006307927256,
3386
+ "kl_mean": 0.6029482114534772,
3387
+ "kl_max": 10.911963588562358,
3388
+ "top1_agreement": 0.7247191011235955,
3389
+ "next_token_acc_ref": 0.09014084507042254,
3390
+ "next_token_acc_test": 0.09014084507042254,
3391
+ "nll_ref": 17.667298816635764,
3392
+ "nll_test": 17.608667050969412
3393
+ },
3394
+ "long_context": {
3395
+ "tokens": 1353,
3396
+ "max_abs_logit_diff": 24.829045295715332,
3397
+ "mean_abs_logit_diff": 1.6162585746277247,
3398
+ "kl_mean": 0.2801612289778648,
3399
+ "kl_max": 25.589695259544296,
3400
+ "top1_agreement": 0.9615668883961567,
3401
+ "next_token_acc_ref": 0.9038461538461539,
3402
+ "next_token_acc_test": 0.9090236686390533,
3403
+ "nll_ref": 0.5574441898794273,
3404
+ "nll_test": 0.5440008410165926
3405
+ },
3406
+ "multilingual": {
3407
+ "tokens": 65,
3408
+ "max_abs_logit_diff": 3.5512681007385254,
3409
+ "mean_abs_logit_diff": 0.19969853793115186,
3410
+ "kl_mean": 0.005015380128086673,
3411
+ "kl_max": 0.08439173889412566,
3412
+ "top1_agreement": 0.9846153846153847,
3413
+ "next_token_acc_ref": 0.578125,
3414
+ "next_token_acc_test": 0.5625,
3415
+ "nll_ref": 3.8383234493145535,
3416
+ "nll_test": 3.793026564653868
3417
+ },
3418
+ "short_chat": {
3419
+ "tokens": 37,
3420
+ "max_abs_logit_diff": 3.655950903892517,
3421
+ "mean_abs_logit_diff": 0.2583573715704287,
3422
+ "kl_mean": 0.015552908775126063,
3423
+ "kl_max": 0.10747477572936717,
3424
+ "top1_agreement": 0.972972972972973,
3425
+ "next_token_acc_ref": 0.5277777777777778,
3426
+ "next_token_acc_test": 0.5277777777777778,
3427
+ "nll_ref": 5.375631014013916,
3428
+ "nll_test": 5.343368736261641
3429
+ },
3430
+ "thinking_chat": {
3431
+ "tokens": 102,
3432
+ "max_abs_logit_diff": 4.207983016967773,
3433
+ "mean_abs_logit_diff": 0.19963685965760844,
3434
+ "kl_mean": 0.0081508028170958,
3435
+ "kl_max": 0.2828682003566698,
3436
+ "top1_agreement": 0.9803921568627451,
3437
+ "next_token_acc_ref": 0.6039603960396039,
3438
+ "next_token_acc_test": 0.6039603960396039,
3439
+ "nll_ref": 3.840478707203926,
3440
+ "nll_test": 3.8346220063091554
3441
+ },
3442
+ "tool_roundtrip": {
3443
+ "tokens": 329,
3444
+ "max_abs_logit_diff": 22.754143714904785,
3445
+ "mean_abs_logit_diff": 0.6942348508134669,
3446
+ "kl_mean": 0.1607668713797337,
3447
+ "kl_max": 7.755121918082823,
3448
+ "top1_agreement": 0.9240121580547113,
3449
+ "next_token_acc_ref": 0.7225609756097561,
3450
+ "next_token_acc_test": 0.698170731707317,
3451
+ "nll_ref": 2.7678284988449815,
3452
+ "nll_test": 2.8992298776234766
3453
+ }
3454
+ },
3455
+ "notes": {
3456
+ "routing_flips": "Where fp32 runs differ, hidden states agree to ~1e-5 relative until a single token's top-8 expert choice flips at a near-tie: long_context first differs at layer visit 32 token 859 (8th/9th routing score margin 1.9e-6), then visit 34 token 1275 (3.5e-6) and visit 39 token 241 (3.1e-6). image_chart first differs at visit 20 token 77.",
3457
+ "dequantization": "MLX dequantization of NVFP4 (layers 0, 13) and MXFP8 (layer 22) experts equals the reference op bit for bit (max abs diff 0.0).",
3458
+ "preprocessing": "Image patches match the reference to 1.2e-7; image grids and prompt token ids are identical; chat-template token ids are identical on all text cases."
3459
+ }
3460
+ },
3461
+ "per_layer_output_error": {
3462
+ "0": {
3463
+ "w13:3:64": {
3464
+ "gptq": 0.012887698906057739,
3465
+ "rtn": 0.028264907912936545
3466
+ },
3467
+ "w13:2:64": {
3468
+ "gptq": 0.049336151572674375,
3469
+ "rtn": 0.12750708317950427
3470
+ },
3471
+ "w2:3:64": {
3472
+ "gptq": 0.015656777931638093,
3473
+ "rtn": 0.03319389076493519
3474
+ },
3475
+ "w2:2:64": {
3476
+ "gptq": 0.06048335714614615,
3477
+ "rtn": 0.14741300019398665
3478
+ }
3479
+ },
3480
+ "1": {
3481
+ "w13:3:64": {
3482
+ "gptq": 0.011802700054363778,
3483
+ "rtn": 0.025790537433406653
3484
+ },
3485
+ "w13:2:64": {
3486
+ "gptq": 0.04516684536862453,
3487
+ "rtn": 0.11663405471536598
3488
+ },
3489
+ "w2:3:64": {
3490
+ "gptq": 0.01641453559978002,
3491
+ "rtn": 0.03318410301470986
3492
+ },
3493
+ "w2:2:64": {
3494
+ "gptq": 0.06349666359881456,
3495
+ "rtn": 0.14814831675886653
3496
+ }
3497
+ },
3498
+ "2": {
3499
+ "w13:3:64": {
3500
+ "gptq": 0.011815724312573465,
3501
+ "rtn": 0.026115071002306627
3502
+ },
3503
+ "w13:2:64": {
3504
+ "gptq": 0.045276587990840216,
3505
+ "rtn": 0.11779761737034271
3506
+ },
3507
+ "w2:3:64": {
3508
+ "gptq": 0.017178491960464465,
3509
+ "rtn": 0.03435173635451321
3510
+ },
3511
+ "w2:2:64": {
3512
+ "gptq": 0.06630938304581857,
3513
+ "rtn": 0.15267030548535696
3514
+ }
3515
+ },
3516
+ "3": {
3517
+ "w13:3:64": {
3518
+ "gptq": 0.012184065107500825,
3519
+ "rtn": 0.0264636905217352
3520
+ },
3521
+ "w13:2:64": {
3522
+ "gptq": 0.0466883827039428,
3523
+ "rtn": 0.11962121154823789
3524
+ },
3525
+ "w2:3:64": {
3526
+ "gptq": 0.018285621104312873,
3527
+ "rtn": 0.03451856381783443
3528
+ },
3529
+ "w2:2:64": {
3530
+ "gptq": 0.07047653555282281,
3531
+ "rtn": 0.15371292378272855
3532
+ }
3533
+ },
3534
+ "4": {
3535
+ "w13:3:64": {
3536
+ "gptq": 0.012464210034104514,
3537
+ "rtn": 0.026564464601376334
3538
+ },
3539
+ "w13:2:64": {
3540
+ "gptq": 0.04778888899832448,
3541
+ "rtn": 0.11984409260700374
3542
+ },
3543
+ "w2:3:64": {
3544
+ "gptq": 0.01901098009817418,
3545
+ "rtn": 0.03473462659622553
3546
+ },
3547
+ "w2:2:64": {
3548
+ "gptq": 0.07319164829533384,
3549
+ "rtn": 0.15440812657866768
3550
+ }
3551
+ },
3552
+ "5": {
3553
+ "w13:3:64": {
3554
+ "gptq": 0.01145844188891433,
3555
+ "rtn": 0.025238676157675728
3556
+ },
3557
+ "w13:2:64": {
3558
+ "gptq": 0.04380464005824843,
3559
+ "rtn": 0.11382083683444219
3560
+ },
3561
+ "w2:3:64": {
3562
+ "gptq": 0.016609748305823607,
3563
+ "rtn": 0.033314921714924374
3564
+ },
3565
+ "w2:2:64": {
3566
+ "gptq": 0.06413552679105854,
3567
+ "rtn": 0.14833856203953347
3568
+ }
3569
+ },
3570
+ "6": {
3571
+ "w13:3:64": {
3572
+ "gptq": 0.008630106207050013,
3573
+ "rtn": 0.021677847331551762
3574
+ },
3575
+ "w13:2:64": {
3576
+ "gptq": 0.03336440886887183,
3577
+ "rtn": 0.098471811560027
3578
+ },
3579
+ "w2:3:64": {
3580
+ "gptq": 0.017297109454220555,
3581
+ "rtn": 0.0329146712577646
3582
+ },
3583
+ "w2:2:64": {
3584
+ "gptq": 0.06655221483005157,
3585
+ "rtn": 0.1467128978713485
3586
+ }
3587
+ },
3588
+ "7": {
3589
+ "w13:3:64": {
3590
+ "gptq": 0.009308482229105724,
3591
+ "rtn": 0.02300102947982573
3592
+ },
3593
+ "w13:2:64": {
3594
+ "gptq": 0.036142309747491115,
3595
+ "rtn": 0.10441029484923274
3596
+ },
3597
+ "w2:3:64": {
3598
+ "gptq": 0.018696131483273832,
3599
+ "rtn": 0.03377960973702008
3600
+ },
3601
+ "w2:2:64": {
3602
+ "gptq": 0.07189859819108688,
3603
+ "rtn": 0.15030866515220506
3604
+ }
3605
+ },
3606
+ "8": {
3607
+ "w13:3:64": {
3608
+ "gptq": 0.00970140273000972,
3609
+ "rtn": 0.023736662945633195
3610
+ },
3611
+ "w13:2:64": {
3612
+ "gptq": 0.03756622131463952,
3613
+ "rtn": 0.10764342694219707
3614
+ },
3615
+ "w2:3:64": {
3616
+ "gptq": 0.01893052109671973,
3617
+ "rtn": 0.03455564228558328
3618
+ },
3619
+ "w2:2:64": {
3620
+ "gptq": 0.07267839934317306,
3621
+ "rtn": 0.1539128401762423
3622
+ }
3623
+ },
3624
+ "9": {
3625
+ "w13:3:64": {
3626
+ "gptq": 0.009544249372379274,
3627
+ "rtn": 0.024004255634757873
3628
+ },
3629
+ "w13:2:64": {
3630
+ "gptq": 0.036904315033598055,
3631
+ "rtn": 0.10858139577985748
3632
+ },
3633
+ "w2:3:64": {
3634
+ "gptq": 0.01861156179149983,
3635
+ "rtn": 0.035172145497160565
3636
+ },
3637
+ "w2:2:64": {
3638
+ "gptq": 0.07143131999568134,
3639
+ "rtn": 0.1562987522341291
3640
+ }
3641
+ },
3642
+ "10": {
3643
+ "w13:3:64": {
3644
+ "gptq": 0.010136449403879732,
3645
+ "rtn": 0.02447905853963705
3646
+ },
3647
+ "w13:2:64": {
3648
+ "gptq": 0.039104749307393195,
3649
+ "rtn": 0.11082593899550983
3650
+ },
3651
+ "w2:3:64": {
3652
+ "gptq": 0.019276758340826466,
3653
+ "rtn": 0.035136919162328153
3654
+ },
3655
+ "w2:2:64": {
3656
+ "gptq": 0.0736179475964238,
3657
+ "rtn": 0.15600411319862784
3658
+ }
3659
+ },
3660
+ "11": {
3661
+ "w13:3:64": {
3662
+ "gptq": 0.00973847975737354,
3663
+ "rtn": 0.024046968008915155
3664
+ },
3665
+ "w13:2:64": {
3666
+ "gptq": 0.037532198495015906,
3667
+ "rtn": 0.10869107576379265
3668
+ },
3669
+ "w2:3:64": {
3670
+ "gptq": 0.01938698195966324,
3671
+ "rtn": 0.03621272793928649
3672
+ },
3673
+ "w2:2:64": {
3674
+ "gptq": 0.07390217281140692,
3675
+ "rtn": 0.16105901638744027
3676
+ }
3677
+ },
3678
+ "12": {
3679
+ "w13:3:64": {
3680
+ "gptq": 0.009194471343113115,
3681
+ "rtn": 0.02273684322111143
3682
+ },
3683
+ "w13:2:64": {
3684
+ "gptq": 0.03564779480386397,
3685
+ "rtn": 0.10313857330143292
3686
+ },
3687
+ "w2:3:64": {
3688
+ "gptq": 0.02008837326579736,
3689
+ "rtn": 0.03548716477388728
3690
+ },
3691
+ "w2:2:64": {
3692
+ "gptq": 0.07655177852173438,
3693
+ "rtn": 0.15761541023604683
3694
+ }
3695
+ },
3696
+ "13": {
3697
+ "w13:3:64": {
3698
+ "gptq": 0.01044786069277626,
3699
+ "rtn": 0.024413591314365678
3700
+ },
3701
+ "w13:2:64": {
3702
+ "gptq": 0.040381324838175386,
3703
+ "rtn": 0.11050714810773753
3704
+ },
3705
+ "w2:3:64": {
3706
+ "gptq": 0.021217173862265844,
3707
+ "rtn": 0.036407905251155226
3708
+ },
3709
+ "w2:2:64": {
3710
+ "gptq": 0.08071875472901732,
3711
+ "rtn": 0.16188429254535236
3712
+ }
3713
+ },
3714
+ "14": {
3715
+ "w13:3:64": {
3716
+ "gptq": 0.010248876740760167,
3717
+ "rtn": 0.024586683531814035
3718
+ },
3719
+ "w13:2:64": {
3720
+ "gptq": 0.03954587427349752,
3721
+ "rtn": 0.1111096092252099
3722
+ },
3723
+ "w2:3:64": {
3724
+ "gptq": 0.020788302220417228,
3725
+ "rtn": 0.037157275436764226
3726
+ },
3727
+ "w2:2:64": {
3728
+ "gptq": 0.07906254882958245,
3729
+ "rtn": 0.1651486767835523
3730
+ }
3731
+ },
3732
+ "15": {
3733
+ "w13:3:64": {
3734
+ "gptq": 0.010022847419748954,
3735
+ "rtn": 0.024274339397861455
3736
+ },
3737
+ "w13:2:64": {
3738
+ "gptq": 0.038597981704112036,
3739
+ "rtn": 0.1097851394777376
3740
+ },
3741
+ "w2:3:64": {
3742
+ "gptq": 0.020948054143987628,
3743
+ "rtn": 0.03831832542731186
3744
+ },
3745
+ "w2:2:64": {
3746
+ "gptq": 0.07954158805661754,
3747
+ "rtn": 0.17017675119167355
3748
+ }
3749
+ },
3750
+ "16": {
3751
+ "w13:3:64": {
3752
+ "gptq": 0.008498339936465907,
3753
+ "rtn": 0.022542785906383197
3754
+ },
3755
+ "w13:2:64": {
3756
+ "gptq": 0.03263134778631234,
3757
+ "rtn": 0.10185568361359731
3758
+ },
3759
+ "w2:3:64": {
3760
+ "gptq": 0.020534073751669327,
3761
+ "rtn": 0.04082283626413468
3762
+ },
3763
+ "w2:2:64": {
3764
+ "gptq": 0.07804667725216022,
3765
+ "rtn": 0.1804656980758368
3766
+ }
3767
+ },
3768
+ "17": {
3769
+ "w13:3:64": {
3770
+ "gptq": 0.009502487563696189,
3771
+ "rtn": 0.02370556959794839
3772
+ },
3773
+ "w13:2:64": {
3774
+ "gptq": 0.036216959333699165,
3775
+ "rtn": 0.10700858826753805
3776
+ },
3777
+ "w2:3:64": {
3778
+ "gptq": 0.020227043716426002,
3779
+ "rtn": 0.04356292902064836
3780
+ },
3781
+ "w2:2:64": {
3782
+ "gptq": 0.07657759032425712,
3783
+ "rtn": 0.1907152855875201
3784
+ }
3785
+ },
3786
+ "18": {
3787
+ "w13:3:64": {
3788
+ "gptq": 0.012731135847359173,
3789
+ "rtn": 0.02782251663498656
3790
+ },
3791
+ "w13:2:64": {
3792
+ "gptq": 0.048424019434402,
3793
+ "rtn": 0.1253200590484997
3794
+ },
3795
+ "w2:3:64": {
3796
+ "gptq": 0.01842322559609543,
3797
+ "rtn": 0.03752281212883477
3798
+ },
3799
+ "w2:2:64": {
3800
+ "gptq": 0.0701647769331662,
3801
+ "rtn": 0.16632908263507784
3802
+ }
3803
+ },
3804
+ "19": {
3805
+ "w13:3:64": {
3806
+ "gptq": 0.013259984145735863,
3807
+ "rtn": 0.02826715333399623
3808
+ },
3809
+ "w13:2:64": {
3810
+ "gptq": 0.050449675440751234,
3811
+ "rtn": 0.12738783863054207
3812
+ },
3813
+ "w2:3:64": {
3814
+ "gptq": 0.019272166405000277,
3815
+ "rtn": 0.03717022037782505
3816
+ },
3817
+ "w2:2:64": {
3818
+ "gptq": 0.07354566699400454,
3819
+ "rtn": 0.1654829052868103
3820
+ }
3821
+ },
3822
+ "20": {
3823
+ "w13:4:64": {
3824
+ "gptq": 0.0035938307581711206,
3825
+ "rtn": 0.007118168637384198
3826
+ },
3827
+ "w13:3:64": {
3828
+ "gptq": 0.014105890797743957,
3829
+ "rtn": 0.03093346592544241
3830
+ },
3831
+ "w13:2:64": {
3832
+ "gptq": 0.053300674122841715,
3833
+ "rtn": 0.13285205393862426
3834
+ },
3835
+ "w2:4:64": {
3836
+ "gptq": 0.004915546344155547,
3837
+ "rtn": 0.008680694219172532
3838
+ },
3839
+ "w2:3:64": {
3840
+ "gptq": 0.019311162271728324,
3841
+ "rtn": 0.03772180529423481
3842
+ },
3843
+ "w2:2:64": {
3844
+ "gptq": 0.07340830898059576,
3845
+ "rtn": 0.1599916808050238
3846
+ }
3847
+ },
3848
+ "21": {
3849
+ "w13:4:64": {
3850
+ "gptq": 0.0037124126419644885,
3851
+ "rtn": 0.00734338705895674
3852
+ },
3853
+ "w13:3:64": {
3854
+ "gptq": 0.014569280031334947,
3855
+ "rtn": 0.03188605416034486
3856
+ },
3857
+ "w13:2:64": {
3858
+ "gptq": 0.055007637306976394,
3859
+ "rtn": 0.1366433478861042
3860
+ },
3861
+ "w2:4:64": {
3862
+ "gptq": 0.005143603726133695,
3863
+ "rtn": 0.008794992797545474
3864
+ },
3865
+ "w2:3:64": {
3866
+ "gptq": 0.02019433043353457,
3867
+ "rtn": 0.03824136567061181
3868
+ },
3869
+ "w2:2:64": {
3870
+ "gptq": 0.07661496879716495,
3871
+ "rtn": 0.16205056792549807
3872
+ }
3873
+ },
3874
+ "22": {
3875
+ "w13:4:64": {
3876
+ "gptq": 0.0035216957335476614,
3877
+ "rtn": 0.00713298095857926
3878
+ },
3879
+ "w13:3:64": {
3880
+ "gptq": 0.01381453591307655,
3881
+ "rtn": 0.030957691522989786
3882
+ },
3883
+ "w13:2:64": {
3884
+ "gptq": 0.052077855686062643,
3885
+ "rtn": 0.13301789032382183
3886
+ },
3887
+ "w2:4:64": {
3888
+ "gptq": 0.004941542055169573,
3889
+ "rtn": 0.008916477635004198
3890
+ },
3891
+ "w2:3:64": {
3892
+ "gptq": 0.019347042681703397,
3893
+ "rtn": 0.038785959177411136
3894
+ },
3895
+ "w2:2:64": {
3896
+ "gptq": 0.073295078682088,
3897
+ "rtn": 0.1641994122618597
3898
+ }
3899
+ },
3900
+ "23": {
3901
+ "w13:4:64": {
3902
+ "gptq": 0.003139279646588113,
3903
+ "rtn": 0.006451814291554056
3904
+ },
3905
+ "w13:3:64": {
3906
+ "gptq": 0.012311763717043433,
3907
+ "rtn": 0.028066662725762536
3908
+ },
3909
+ "w13:2:64": {
3910
+ "gptq": 0.04638598758334907,
3911
+ "rtn": 0.12017644349661992
3912
+ },
3913
+ "w2:4:64": {
3914
+ "gptq": 0.005102806170440476,
3915
+ "rtn": 0.009787645443707936
3916
+ },
3917
+ "w2:3:64": {
3918
+ "gptq": 0.019850079171067476,
3919
+ "rtn": 0.04258285916317973
3920
+ },
3921
+ "w2:2:64": {
3922
+ "gptq": 0.07480900112960819,
3923
+ "rtn": 0.17718135892908898
3924
+ }
3925
+ }
3926
+ }
3927
+ }
extras/scripts/actquant.py ADDED
@@ -0,0 +1,104 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Emulated activation quantization for the expert projections (experiment).
2
+
3
+ hf_quant_config.json lists `required_activation_quantization` for the expert weights:
4
+ layers 0-19 nvfp4_e2m1, block 16, per-token amax, E4M3 block scales, scale rule "fouroversix_mae"
5
+ layers 20-23 mxfp8_e4m3, block 32, power-of-two (E8M0) block scales
6
+ Neither transformers nor the MLX runtimes apply it. These functions quantize-and-dequantize the inputs of
7
+ gate/up (the normed hidden state) and of down (the gated activation) so the effect on held-out loss can be measured.
8
+ The exact rounding conventions of the original runtime are not published; this follows the public formats.
9
+ """
10
+
11
+ import mlx.core as mx
12
+
13
+ E4M3_MAX = 448.0
14
+ E4M3_MIN = 2.0 ** -6
15
+
16
+
17
+ def round_e4m3(v):
18
+ """Round positive values to the nearest E4M3 normal number (3 mantissa bits), clamped to the format's range."""
19
+ v = mx.clip(v, E4M3_MIN, E4M3_MAX)
20
+ e = mx.floor(mx.log2(v))
21
+ m = mx.round((v / mx.power(2.0, e) - 1.0) * 8.0)
22
+ return mx.minimum(mx.power(2.0, e) * (1.0 + m / 8.0), E4M3_MAX)
23
+
24
+
25
+ def round_e4m3_signed(v):
26
+ a = mx.abs(v)
27
+ q = mx.where(a < E4M3_MIN / 2, 0.0, round_e4m3(a))
28
+ return mx.sign(v) * q
29
+
30
+
31
+ def round_e2m1(v):
32
+ """Nearest value of {0, .5, 1, 1.5, 2, 3, 4, 6} with sign."""
33
+ a = mx.minimum(mx.abs(v), 6.0)
34
+ q = mx.where(a < 2.0, mx.round(a * 2.0) / 2.0, mx.where(a < 4.0, mx.round(a), mx.where(a < 5.0, 4.0, 6.0)))
35
+ return mx.sign(v) * q
36
+
37
+
38
+ def nvfp4_fake_quant(x):
39
+ dtype = x.dtype
40
+ x = x.astype(mx.float32)
41
+ d = x.shape[-1]
42
+ amax = mx.abs(x).max(axis=-1, keepdims=True) # per token
43
+ outer = mx.maximum(amax, 1e-12) / (6.0 * E4M3_MAX)
44
+ b = x.reshape(*x.shape[:-1], d // 16, 16)
45
+ bmax = mx.maximum(mx.abs(b).max(axis=-1, keepdims=True), 1e-12)
46
+ out = err = None
47
+ for top in (6.0, 4.0): # "four over six": map the block max to 6 or to 4
48
+ scale = round_e4m3(bmax / top / outer[..., None]) * outer[..., None]
49
+ q = round_e2m1(b / scale) * scale
50
+ e = mx.abs(q - b).mean(axis=-1, keepdims=True)
51
+ if out is None:
52
+ out, err = q, e
53
+ else:
54
+ out = mx.where(e < err, q, out)
55
+ return out.reshape(x.shape).astype(dtype)
56
+
57
+
58
+ def mxfp8_fake_quant(x):
59
+ dtype = x.dtype
60
+ x = x.astype(mx.float32)
61
+ d = x.shape[-1]
62
+ b = x.reshape(*x.shape[:-1], d // 32, 32)
63
+ bmax = mx.maximum(mx.abs(b).max(axis=-1, keepdims=True), 1e-30)
64
+ scale = mx.power(2.0, mx.ceil(mx.log2(bmax / E4M3_MAX)))
65
+ return (round_e4m3_signed(b / scale) * scale).reshape(x.shape).astype(dtype)
66
+
67
+
68
+ def install(model, which=("input", "hidden")):
69
+ """Patch every layer's MoE to quantize expert inputs. Returns an undo function."""
70
+ from mlx_vlm.models.inkling2 import language as L
71
+
72
+ originals = []
73
+ for layer in model.language_model.model.layers:
74
+ moe = layer.mlp
75
+ fq = nvfp4_fake_quant if getattr(moe.switch_mlp.down_proj, "mode", "") != "mxfp8" else mxfp8_fake_quant
76
+ moe._fq = fq
77
+ act = moe.switch_mlp.activation
78
+ originals.append((moe, act))
79
+ if "hidden" in which:
80
+ class QuantAct(L.SoftcappedSwiGLU):
81
+ def __call__(self, up, gate, _fq=fq, _base=act):
82
+ return _fq(_base(up, gate))
83
+ new = QuantAct.__new__(QuantAct)
84
+ import mlx.nn as nn
85
+ nn.Module.__init__(new)
86
+ new.gate_cap, new.up_cap = act.gate_cap, act.up_cap
87
+ moe.switch_mlp.activation = new
88
+ orig_call = L.SparseMoE.__call__
89
+
90
+ def patched(self, x):
91
+ if "input" not in which or getattr(self, "_fq", None) is None:
92
+ return orig_call(self, x)
93
+ indices, weights = L._route(x, self.gate_weight, self.e_score_correction_bias, self.top_k, self.route_scale)
94
+ out = self.switch_mlp(self._fq(x), indices)
95
+ return L._combine_routes(out, weights)
96
+
97
+ L.SparseMoE.__call__ = patched
98
+
99
+ def undo():
100
+ L.SparseMoE.__call__ = orig_call
101
+ for moe, act in originals:
102
+ moe.switch_mlp.activation = act
103
+ moe._fq = None
104
+ return undo
extras/scripts/build_sequences.py ADDED
@@ -0,0 +1,73 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Build calibration / evaluation token sequences. Usage: build_sequences.py <chats.jsonl> <out.json>"""
2
+ import glob, json, os, sys, sysconfig
3
+ import pyarrow.parquet as pq
4
+ from transformers import AutoTokenizer
5
+
6
+ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
7
+ from scenarios import TOOLS # noqa: E402
8
+
9
+ ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
10
+ tok = AutoTokenizer.from_pretrained(os.path.join(ROOT, "upstream", "Inkling-2-Mini"))
11
+ MAX = 900
12
+ out = []
13
+
14
+ def render(messages, tools=None, effort="none"):
15
+ r = tok.apply_chat_template(messages, tools=tools, reasoning_effort=effort, tokenize=True)
16
+ return list(r["input_ids"]) if hasattr(r, "keys") else list(r)
17
+
18
+ def clip_text(text, budget):
19
+ ids = tok.encode(text, add_special_tokens=False)[:budget]
20
+ return tok.decode(ids)
21
+
22
+ # 1. model-generated chats
23
+ for line in open(sys.argv[1]):
24
+ rec = json.loads(line)
25
+ tools = [TOOLS[t] for t in rec.get("tools", [])] or None
26
+ ids = render(rec["messages"], tools, rec.get("effort"))
27
+ out.append(dict(name="chat:" + rec["id"], split=rec["split"], kind="chat:" + rec["category"], ids=ids))
28
+
29
+ # 2. WikiText-2 articles framed as an assistant answer
30
+ def articles(path):
31
+ lines = pq.read_table(path).column("text").to_pylist()
32
+ cur_title, cur = None, []
33
+ for ln in lines:
34
+ s = ln.strip()
35
+ if s.startswith("= ") and s.endswith(" =") and not s.startswith("= ="):
36
+ if cur_title and cur:
37
+ yield cur_title, "".join(cur)
38
+ cur_title, cur = s.strip("= ").strip(), []
39
+ elif cur_title is not None:
40
+ cur.append(ln)
41
+ if cur_title and cur:
42
+ yield cur_title, "".join(cur)
43
+
44
+ for split, fname, count in (("eval", "test", 30), ("calib", "train", 30)):
45
+ for i, (title, body) in enumerate(articles(os.path.join(ROOT, "data/wikitext", fname + ".parquet"))):
46
+ if i >= count:
47
+ break
48
+ msgs = [{"role": "user", "content": f"Write the encyclopedia article titled “{title}”."},
49
+ {"role": "assistant", "content": clip_text(body.strip(), MAX)}]
50
+ out.append(dict(name=f"wikitext:{fname}:{i}", split=split, kind="wikitext", ids=render(msgs)))
51
+
52
+ # 3. Python standard-library source framed as an assistant answer
53
+ stdlib = sysconfig.get_paths()["stdlib"]
54
+ for split, names in (("calib", ["heapq", "bisect", "textwrap", "colorsys", "fnmatch", "glob", "sched", "queue"]),
55
+ ("eval", ["shlex", "statistics", "difflib", "copy", "keyword", "reprlib"])):
56
+ for name in names:
57
+ src = open(os.path.join(stdlib, name + ".py"), encoding="utf-8").read()
58
+ for part, offset in enumerate((0, 6000)):
59
+ piece = src[offset:offset + 12000]
60
+ if len(piece) < 800:
61
+ continue
62
+ msgs = [{"role": "user", "content": f"Show part {part + 1} of the source of Python's `{name}` module."},
63
+ {"role": "assistant", "content": "```python\n" + clip_text(piece, MAX) + "\n```"}]
64
+ out.append(dict(name=f"code:{name}:{part}", split=split, kind="code", ids=render(msgs)))
65
+
66
+ json.dump(out, open(sys.argv[2], "w"))
67
+ for split in ("calib", "eval"):
68
+ sel = [s for s in out if s["split"] == split]
69
+ kinds = {}
70
+ for s in sel:
71
+ k = s["kind"].split(":")[0]
72
+ kinds[k] = kinds.get(k, 0) + len(s["ids"])
73
+ print(split, "sequences", len(sel), "tokens", sum(len(s["ids"]) for s in sel), kinds)
extras/scripts/build_variant.py ADDED
@@ -0,0 +1,87 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Build a publishable mlx-vlm checkpoint directory from a recipe.
2
+
3
+ Usage: build_variant.py <recipes.json> <recipe_name> <out_dir> [--qdir q/shards]
4
+ Writes weights (mlx-vlm layout), config.json with explicit per-module quantization, tokenizer and processor files,
5
+ the bundled DSpark drafter (mtp.safetensors, unchanged) and a build manifest; then reloads the directory and checks
6
+ that its logits equal the in-memory model's.
7
+ """
8
+
9
+ import hashlib
10
+ import json
11
+ import os
12
+ import shutil
13
+ import sys
14
+
15
+ import mlx.core as mx
16
+ import numpy as np
17
+
18
+ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
19
+ import qlab # noqa: E402
20
+ import recipes as R # noqa: E402
21
+ from mlx_vlm import load # noqa: E402
22
+ from mlx_vlm.utils import save_config, save_weights # noqa: E402
23
+
24
+ COPY = ("tokenizer.json", "tokenizer_config.json", "special_tokens_map.json", "chat_template.jinja",
25
+ "processor_config.json", "generation_config.json", "mtp.safetensors")
26
+ UPSTREAM_REVISION = "ed40e7da625f594fc9c4990f5f66be24873c22d7"
27
+
28
+
29
+ def sha256(path):
30
+ h = hashlib.sha256()
31
+ with open(path, "rb") as f:
32
+ while chunk := f.read(16 << 20):
33
+ h.update(chunk)
34
+ return h.hexdigest()
35
+
36
+
37
+ def main():
38
+ args = sys.argv[1:]
39
+ qdir = os.path.join(qlab.ROOT, "q", "shards")
40
+ if "--qdir" in args:
41
+ i = args.index("--qdir"); qdir = args[i + 1]; del args[i:i + 2]
42
+ recipe_path, name, out_dir = args[:3]
43
+ recipe = json.load(open(recipe_path))[name]
44
+ os.makedirs(out_dir, exist_ok=True)
45
+ model, processor = qlab.load_native()
46
+ probe = json.load(open(os.path.join(qlab.ROOT, "parity/ref_fp32/tool_roundtrip.ids.json")))["ids"]
47
+ overrides = R.apply(model, recipe, qdir, log=print)
48
+ before = np.array(qlab.logits_for(model, probe))
49
+
50
+ config = json.load(open(os.path.join(qlab.NATIVE, "config.json")))
51
+ quant = {"group_size": 64, "bits": 8, "mode": "affine"}
52
+ quant.update({k: overrides[k] for k in sorted(overrides)})
53
+ config["quantization"] = quant
54
+ config["quantization_config"] = quant
55
+ save_weights(out_dir, model)
56
+ save_config(config, os.path.join(out_dir, "config.json"))
57
+ for f in COPY:
58
+ shutil.copy2(os.path.join(qlab.NATIVE, f), os.path.join(out_dir, f))
59
+ os.makedirs(os.path.join(out_dir, "tiktoken"), exist_ok=True)
60
+ shutil.copy2(os.path.join(qlab.NATIVE, "tiktoken", "tokenizer.model"), os.path.join(out_dir, "tiktoken", "tokenizer.model"))
61
+
62
+ kinds = {}
63
+ for path, q in overrides.items():
64
+ group = "experts" if ".switch_mlp." in path else R.dense_group(path, None) or "other"
65
+ key = f"{group}: {q['mode']} {q['bits']}-bit g{q['group_size']}"
66
+ kinds[key] = kinds.get(key, 0) + 1
67
+ del model
68
+ mx.clear_cache()
69
+
70
+ reloaded, _ = load(out_dir)
71
+ after = np.array(qlab.logits_for(reloaded, probe))
72
+ max_diff = float(np.abs(before - after).max())
73
+ files = sorted(f for f in os.listdir(out_dir) if f.endswith(".safetensors"))
74
+ manifest = dict(
75
+ name=name, recipe=recipe, upstream="thinkingmachines/Inkling-2-Mini", upstream_revision=UPSTREAM_REVISION,
76
+ module_counts=kinds, reload_max_logit_diff=max_diff,
77
+ weights_bytes=sum(os.path.getsize(os.path.join(out_dir, f)) for f in files if f != "mtp.safetensors"),
78
+ files={f: dict(bytes=os.path.getsize(os.path.join(out_dir, f)), sha256=sha256(os.path.join(out_dir, f))) for f in files},
79
+ )
80
+ json.dump(manifest, open(os.path.join(out_dir, "build_manifest.json"), "w"), indent=1)
81
+ print(json.dumps({k: v for k, v in manifest.items() if k not in ("files", "recipe")}, indent=1))
82
+ if max_diff != 0.0:
83
+ raise SystemExit(f"reloaded checkpoint differs from the in-memory model (max logit diff {max_diff})")
84
+
85
+
86
+ if __name__ == "__main__":
87
+ main()
extras/scripts/calib_capture.py ADDED
@@ -0,0 +1,76 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Accumulate per-expert input second moments (GPTQ Hessians) for a set of layers.
2
+
3
+ For every routed (token, expert) pair of a layer visit:
4
+ H13[e] += x x^T (input of gate_proj / up_proj, dim 2304)
5
+ H2[e] += a a^T (input of down_proj, a = softcapped_swiglu(up_e x, gate_e x), dim 1152)
6
+ Looped layers accumulate over all three visits. Usage: calib_capture.py <sequences.json> <out_dir> <layer,layer,...>
7
+ """
8
+ import os, sys, time
9
+ import numpy as np
10
+ import mlx.core as mx
11
+
12
+ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
13
+ import qlab # noqa: E402
14
+ from mlx_vlm.models.inkling2 import language as L # noqa: E402
15
+
16
+ seq_path, out_dir, layer_arg = sys.argv[1:4]
17
+ want = [int(x) for x in layer_arg.split(",")]
18
+ os.makedirs(out_dir, exist_ok=True)
19
+ model, _ = qlab.load_native()
20
+ seqs = qlab.load_sequences(seq_path, split="calib", max_len=1024)
21
+ state = {}
22
+ for li in want:
23
+ moe = qlab.layers(model)[li].mlp
24
+ e, d, h = moe.gate_weight.shape[0], moe.gate_weight.shape[1], moe.switch_mlp.down_proj.input_dims
25
+ state[id(moe)] = dict(layer=li, H13=[mx.zeros((d, d)) for _ in range(e)], H2=[mx.zeros((h, h)) for _ in range(e)],
26
+ n=np.zeros(e, np.int64))
27
+ orig = L.SparseMoE.__call__
28
+
29
+ def patched(self, x):
30
+ st = state.get(id(self))
31
+ if st is not None:
32
+ indices, _ = L._route(x, self.gate_weight, self.e_score_correction_bias, self.top_k, self.route_scale)
33
+ X = x.reshape(-1, x.shape[-1])
34
+ idx = np.array(indices.reshape(-1, indices.shape[-1]))
35
+ flat = idx.reshape(-1)
36
+ order = np.argsort(flat, kind="stable")
37
+ tok_of = order // idx.shape[1]
38
+ bounds = np.searchsorted(flat[order], np.arange(len(st["H13"]) + 1))
39
+ sw = self.switch_mlp
40
+ H13, H2 = st["H13"], st["H2"]
41
+ touched = []
42
+ for e in range(len(bounds) - 1):
43
+ lo, hi = int(bounds[e]), int(bounds[e + 1])
44
+ if hi == lo:
45
+ continue
46
+ rows = mx.array(tok_of[lo:hi])
47
+ xe = X[rows] # (n, D), model dtype
48
+ ei = mx.full((hi - lo, 1), e, dtype=mx.uint32)
49
+ xin = xe[:, None, None, :]
50
+ up = sw.up_proj(xin, ei)
51
+ gate = sw.gate_proj(xin, ei)
52
+ a = sw.activation(up, gate).reshape(hi - lo, -1).astype(mx.float32)
53
+ xf = xe.astype(mx.float32)
54
+ H13[e] = H13[e] + xf.T @ xf
55
+ H2[e] = H2[e] + a.T @ a
56
+ touched += [H13[e], H2[e]]
57
+ st["n"][e] += hi - lo
58
+ mx.eval(touched)
59
+ return orig(self, x)
60
+
61
+ L.SparseMoE.__call__ = patched
62
+ t0 = time.time(); ntok = 0
63
+ for i, s in enumerate(seqs):
64
+ out = model(mx.array([s["ids"]]))
65
+ mx.eval(out.logits[0, -1])
66
+ ntok += len(s["ids"])
67
+ mx.clear_cache()
68
+ if i % 10 == 0:
69
+ print(f"seq {i}/{len(seqs)} tokens {ntok} elapsed {time.time() - t0:.0f}s peak {mx.get_peak_memory() / 2**30:.1f} GB", flush=True)
70
+ for st in state.values():
71
+ li = st["layer"]
72
+ mx.save_safetensors(os.path.join(out_dir, f"hess_L{li:02d}.safetensors"),
73
+ {"H13": mx.stack(st["H13"]), "H2": mx.stack(st["H2"]), "n": mx.array(st["n"])})
74
+ print(f"layer {li}: routed pairs {int(st['n'].sum())} min/median/max per expert "
75
+ f"{int(st['n'].min())}/{int(np.median(st['n']))}/{int(st['n'].max())}", flush=True)
76
+ print("done", round(time.time() - t0), "s tokens", ntok)
extras/scripts/eval_nll.py ADDED
@@ -0,0 +1,76 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Held-out loss on assistant-side tokens. Usage: eval_nll.py <model_dir|native> <sequences.json> <out.json> <tag> [--fp32] [--actquant input,hidden]
2
+
3
+ Counts only tokens the model would generate itself (assistant messages: reasoning, text, tool calls, and the
4
+ end-of-turn token). Stores per-sequence sums so variants can be compared with paired statistics.
5
+ """
6
+ import json, os, sys, time
7
+ import numpy as np
8
+ import mlx.core as mx
9
+ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
10
+ import qlab
11
+
12
+ MODEL_TOK, END_MSG, EOS = 200001, 200010, 200006
13
+
14
+ def assistant_mask(ids):
15
+ mask, inside = [], False
16
+ for t in ids:
17
+ if t == MODEL_TOK:
18
+ inside = True
19
+ mask.append(inside or t == EOS)
20
+ if t == END_MSG:
21
+ inside = False
22
+ return mask
23
+
24
+ def main():
25
+ args = sys.argv[1:]
26
+ fp32 = "--fp32" in args
27
+ if fp32: args.remove("--fp32")
28
+ aq = None
29
+ if "--actquant" in args:
30
+ i = args.index("--actquant"); aq = tuple(args[i + 1].split(",")); del args[i:i + 2]
31
+ recipe = None
32
+ if "--recipe" in args:
33
+ i = args.index("--recipe"); rfile, rname = args[i + 1].split(":"); del args[i:i + 2]
34
+ recipe = json.load(open(rfile))[rname]
35
+ model_dir, seq_path, out_path, tag = args[:4]
36
+ path = qlab.NATIVE if model_dir == "native" else model_dir
37
+ model, _ = qlab.load_native(fp32=False, path=path)
38
+ size = None
39
+ if recipe is not None:
40
+ import recipes as R
41
+ R.apply(model, recipe, os.path.join(qlab.ROOT, "q", "shards"))
42
+ size = R.nbytes(model) / 2**30
43
+ if fp32:
44
+ qlab.to_fp32(model); mx.eval(model.parameters())
45
+ if aq:
46
+ import actquant
47
+ actquant.install(model, aq)
48
+ seqs = qlab.load_sequences(seq_path, split="eval", max_len=1024)
49
+ rows, t0 = [], time.time()
50
+ for s in seqs:
51
+ ids = s["ids"]
52
+ lg = qlab.logits_for(model, ids)
53
+ lp = qlab.log_softmax(lg)[:-1]
54
+ nxt = mx.array(ids[1:])
55
+ nll = -mx.take_along_axis(lp, nxt[:, None], axis=-1)[:, 0]
56
+ m = mx.array(assistant_mask(ids)[1:]).astype(mx.float32)
57
+ hit = (mx.argmax(lp, axis=-1) == nxt).astype(mx.float32)
58
+ vals = [(nll * m).sum(), m.sum(), (hit * m).sum()]
59
+ mx.eval(vals)
60
+ rows.append(dict(name=s["name"], kind=s["kind"].split(":")[0], nll=vals[0].item(), n=int(vals[1].item()), hit=vals[2].item()))
61
+ del lg, lp
62
+ res = json.load(open(out_path)) if os.path.exists(out_path) else {}
63
+ n = sum(r["n"] for r in rows)
64
+ summ = dict(tokens=n, nll=sum(r["nll"] for r in rows) / n, acc=sum(r["hit"] for r in rows) / n, seconds=round(time.time() - t0),
65
+ peak_gb=round(mx.get_peak_memory() / 2**30, 2), weights_gib=size, activations="fp32" if fp32 else "bf16", actquant=aq)
66
+ for kind in sorted({r["kind"] for r in rows}):
67
+ sel = [r for r in rows if r["kind"] == kind]
68
+ summ[kind] = dict(tokens=sum(r["n"] for r in sel), nll=sum(r["nll"] for r in sel) / max(1, sum(r["n"] for r in sel)))
69
+ summ["rows"] = rows
70
+ res[tag] = summ
71
+ json.dump(res, open(out_path, "w"))
72
+ print(f"{tag:28s} tokens {n} NLL {summ['nll']:.4f} ppl {np.exp(summ['nll']):.3f} acc {summ['acc']:.4f} "
73
+ + " ".join(f"{k} {summ[k]['nll']:.4f}" for k in ("chat", "code", "wikitext") if k in summ) + f" ({summ['seconds']}s)", flush=True)
74
+
75
+ if __name__ == "__main__":
76
+ main()
extras/scripts/gptq.py ADDED
@@ -0,0 +1,195 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Error-compensated (GPTQ-style) affine requantization of Inkling 2 Mini experts, batched over experts.
2
+
3
+ For one projection W (experts, out, in) with per-expert input second moments H (experts, in, in), columns are
4
+ quantized left to right; after each column the remaining columns absorb the rounding error through the inverse
5
+ Hessian, so the layer output error on calibration inputs is minimised rather than the weight error. Scale and
6
+ bias of each 64-column group are chosen (with a small clipping search) from the already-updated weights and are
7
+ rounded to float16 before the codes are computed, so the stored parameters are exactly the ones optimised for.
8
+
9
+ Usage: gptq.py <hess_dir> <out_dir> --layers 0,1 --specs 3:64,2:64 [--damp 0.01] [--rtn]
10
+ """
11
+
12
+ import os
13
+ import sys
14
+ import time
15
+
16
+ import mlx.core as mx
17
+ import numpy as np
18
+
19
+ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
20
+ import qlab # noqa: E402
21
+
22
+ SHRINKS = (1.0, 0.9, 0.8, 0.7, 0.6)
23
+
24
+
25
+ def prepare_u(H, counts, damp):
26
+ """Upper Cholesky factor of the inverse damped Hessian, (experts, in, in) float32, computed on CPU."""
27
+ e, d, _ = H.shape
28
+ diag = mx.diagonal(H, axis1=1, axis2=2)
29
+ mean = diag.mean(axis=1)
30
+ n = mx.array(np.maximum(counts, 1).astype(np.float32))
31
+ # Shrink towards the identity more strongly when an expert saw few samples (covariance is then rank-poor).
32
+ lam = mx.clip(0.5 * d / n, damp, 1.0) * mx.where(mean > 0, mean, 1.0)
33
+ never = mx.array((counts == 0).astype(np.float32))
34
+ Hd = H * (1 - never)[:, None, None] + mx.eye(d)[None] * (lam + never)[:, None, None]
35
+ Hd = Hd / mx.diagonal(Hd, axis1=1, axis2=2).mean(axis=1)[:, None, None]
36
+ mx.eval(Hd)
37
+ out = []
38
+ for i in range(0, e, 16):
39
+ inv = mx.linalg.inv(Hd[i:i + 16], stream=mx.cpu)
40
+ inv = (inv + inv.swapaxes(-1, -2)) * 0.5
41
+ u = mx.linalg.cholesky(inv, upper=True, stream=mx.cpu)
42
+ mx.eval(u)
43
+ out.append(u)
44
+ U = mx.concatenate(out, axis=0)
45
+ mx.eval(U)
46
+ return U, Hd
47
+
48
+
49
+ def _params(Wb, wt, levels, dtype):
50
+ """Pick scale/bias per (expert, row) for one group by a clipping search. wt: (experts, g) column weights."""
51
+ lo, hi = Wb.min(axis=-1), Wb.max(axis=-1)
52
+ mid, half = (hi + lo) * 0.5, (hi - lo) * 0.5
53
+ best_err = best_s = best_b = None
54
+ for p in SHRINKS:
55
+ s = mx.maximum(2 * half * p / levels, 1e-8).astype(dtype).astype(mx.float32)
56
+ b = (mid - half * p).astype(dtype).astype(mx.float32)
57
+ q = mx.clip(mx.round((Wb - b[..., None]) / s[..., None]), 0, levels)
58
+ err = (mx.square(q * s[..., None] + b[..., None] - Wb) * wt[:, None, :]).sum(axis=-1)
59
+ if best_err is None:
60
+ best_err, best_s, best_b = err, s, b
61
+ else:
62
+ take = err < best_err
63
+ best_err = mx.where(take, err, best_err)
64
+ best_s = mx.where(take, s, best_s)
65
+ best_b = mx.where(take, b, best_b)
66
+ return best_s, best_b
67
+
68
+
69
+ def gptq_affine(W, U, bits, group, scale_dtype=mx.float16, compensate=True):
70
+ """Returns (codes uint8 (E,out,in), scales (E,out,in/g), biases) for W (E,out,in) given U from prepare_u."""
71
+ e, out_dim, in_dim = W.shape
72
+ levels = float(2 ** bits - 1)
73
+ codes, scales, biases = [], [], []
74
+ W = W.astype(mx.float32)
75
+ tri = mx.triu(mx.ones((group, group)), k=1)
76
+ for i1 in range(0, in_dim, group):
77
+ i2 = i1 + group
78
+ Ub = U[:, i1:i2, i1:i2] # (E, g, g)
79
+ dg = mx.diagonal(Ub, axis1=1, axis2=2) # (E, g)
80
+ Wb = W[:, :, i1:i2]
81
+ s, b = _params(Wb, 1.0 / mx.square(dg), levels, scale_dtype)
82
+ Uoff = Ub * tri[None] # strictly upper part
83
+ cols, errs = [], []
84
+ cur = Wb
85
+ for j in range(group):
86
+ w = cur[:, :, j]
87
+ q = mx.clip(mx.round((w - b) / s), 0, levels)
88
+ cols.append(q)
89
+ if compensate:
90
+ err = (w - (q * s + b)) / dg[:, j][:, None]
91
+ errs.append(err)
92
+ cur = cur - err[:, :, None] * Uoff[:, j][:, None, :]
93
+ codes.append(mx.stack(cols, axis=-1).astype(mx.uint8))
94
+ scales.append(s.astype(scale_dtype)); biases.append(b.astype(scale_dtype))
95
+ if compensate and i2 < in_dim:
96
+ E = mx.stack(errs, axis=-1) # (E, out, g)
97
+ tail = W[:, :, i2:] - E @ U[:, i1:i2, i2:]
98
+ W = mx.concatenate([W[:, :, :i2], tail], axis=-1)
99
+ mx.eval(codes[-1], scales[-1], biases[-1], W)
100
+ return mx.concatenate(codes, axis=-1), mx.stack(scales, axis=-1), mx.stack(biases, axis=-1)
101
+
102
+
103
+ def dequant(codes, scales, biases, group):
104
+ e, o, n = codes.shape
105
+ c = codes.astype(mx.float32).reshape(e, o, n // group, group)
106
+ return (c * scales.astype(mx.float32)[..., None] + biases.astype(mx.float32)[..., None]).reshape(e, o, n)
107
+
108
+
109
+ def hessian_error(W, What, H):
110
+ """sum_e tr(dW H dW^T) / sum_e tr(W H W^T): relative output error energy on the calibration inputs."""
111
+ num = den = 0.0
112
+ for i in range(0, W.shape[0], 16):
113
+ d = (What[i:i + 16] - W[i:i + 16]).astype(mx.float32)
114
+ w = W[i:i + 16].astype(mx.float32)
115
+ h = H[i:i + 16]
116
+ a = ((d @ h) * d).sum(); c = ((w @ h) * w).sum()
117
+ mx.eval(a, c)
118
+ num += a.item(); den += c.item()
119
+ return num / max(den, 1e-30)
120
+
121
+
122
+ def quantize_layer(model, layer, hess, specs, damp=0.01, compensate=True, log=print):
123
+ """Quantize one layer's three projections for every (bits, group) spec. Returns {spec: {name: array}}, stats."""
124
+ sw = qlab.switch(model, layer)
125
+ counts = np.array(hess["n"])
126
+ gate, up, down = (qlab.dense_expert(getattr(sw, p)) for p in qlab.PROJECTIONS)
127
+ W13 = mx.concatenate([gate, up], axis=1)
128
+ mx.eval(W13, down)
129
+ half = gate.shape[1]
130
+ results, stats = {spec: {} for spec in specs}, {}
131
+ for name, W, H in (("w13", W13, hess["H13"]), ("w2", down, hess["H2"])):
132
+ t0 = time.time()
133
+ U, Hd = prepare_u(H, counts, damp)
134
+ t_lin = time.time() - t0
135
+ for spec in specs:
136
+ bits, group = spec
137
+ t1 = time.time()
138
+ codes, s, b = gptq_affine(W, U, bits, group, compensate=compensate)
139
+ err = hessian_error(W, dequant(codes, s, b, group), Hd)
140
+ wq, s0, b0 = qlab.rtn_affine(W, bits, group)
141
+ rtn = hessian_error(W, mx.dequantize(wq, s0.astype(mx.float32), b0.astype(mx.float32), group_size=group,
142
+ bits=bits, mode="affine", dtype=mx.float32), Hd)
143
+ stats[f"{name}:{bits}:{group}"] = dict(gptq=err, rtn=rtn)
144
+ log(f" L{layer:02d} {name} {bits}-bit g{group}: output error energy {err:.3e} (round-to-nearest {rtn:.3e}, "
145
+ f"ratio {rtn / max(err, 1e-30):.1f}x) linalg {t_lin:.0f}s quant {time.time() - t1:.0f}s")
146
+ packed = qlab.pack_codes(codes, bits)
147
+ mx.eval(packed)
148
+ if name == "w13":
149
+ results[spec].update({
150
+ "gate_proj.weight": packed[:, :half], "gate_proj.scales": s[:, :half], "gate_proj.biases": b[:, :half],
151
+ "up_proj.weight": packed[:, half:], "up_proj.scales": s[:, half:], "up_proj.biases": b[:, half:]})
152
+ else:
153
+ results[spec].update({"down_proj.weight": packed, "down_proj.scales": s, "down_proj.biases": b})
154
+ mx.eval(list(results[spec].values()))
155
+ del U, Hd
156
+ mx.clear_cache()
157
+ return results, stats
158
+
159
+
160
+ def main():
161
+ args = sys.argv[1:]
162
+ hess_dir, out_dir = args[:2]
163
+ layers, specs, damp, compensate = [], [], 0.01, True
164
+ rest = args[2:]
165
+ while rest:
166
+ if rest[0] == "--layers":
167
+ layers = [int(x) for x in rest[1].split(",")]; rest = rest[2:]
168
+ elif rest[0] == "--specs":
169
+ specs = [tuple(int(v) for v in s.split(":")) for s in rest[1].split(",")]; rest = rest[2:]
170
+ elif rest[0] == "--damp":
171
+ damp = float(rest[1]); rest = rest[2:]
172
+ elif rest[0] == "--rtn":
173
+ compensate = False; rest = rest[1:]
174
+ else:
175
+ raise SystemExit("unknown flag " + rest[0])
176
+ os.makedirs(out_dir, exist_ok=True)
177
+ from mlx_vlm import load
178
+ model, _ = load(qlab.NATIVE, lazy=True)
179
+ import json
180
+ all_stats = {}
181
+ for li in layers:
182
+ t0 = time.time()
183
+ hess = mx.load(os.path.join(hess_dir, f"hess_L{li:02d}.safetensors"))
184
+ results, stats = quantize_layer(model, li, hess, specs, damp, compensate)
185
+ for (bits, group), tensors in results.items():
186
+ mx.save_safetensors(os.path.join(out_dir, f"q_L{li:02d}_b{bits}g{group}.safetensors"), tensors)
187
+ all_stats[li] = stats
188
+ json.dump(all_stats, open(os.path.join(out_dir, f"stats_{layers[0]:02d}-{layers[-1]:02d}.json"), "w"), indent=1)
189
+ print(f"layer {li} done in {time.time() - t0:.0f}s; peak {mx.get_peak_memory() / 2**30:.1f} GB", flush=True)
190
+ del hess, results
191
+ mx.clear_cache()
192
+
193
+
194
+ if __name__ == "__main__":
195
+ main()
extras/scripts/qlab.py ADDED
@@ -0,0 +1,147 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Shared helpers for the Inkling 2 Mini quantization work (mlx-vlm layout)."""
2
+
3
+ import json
4
+ import math
5
+ import os
6
+
7
+ import mlx.core as mx
8
+ import mlx.nn as nn
9
+ import numpy as np
10
+ from mlx.utils import tree_map
11
+
12
+ from mlx_vlm import load
13
+ from mlx_vlm.models.switch_layers import QuantizedSwitchLinear
14
+
15
+ ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
16
+ NATIVE = os.path.join(ROOT, "upstream", "Inkling-2-Mini")
17
+ PROJECTIONS = ("gate_proj", "up_proj", "down_proj")
18
+
19
+
20
+ # ----------------------------------------------------------------------------- model access
21
+ def load_native(fp32=False, path=NATIVE):
22
+ model, processor = load(path)
23
+ if fp32:
24
+ to_fp32(model)
25
+ mx.eval(model.parameters())
26
+ return model, processor
27
+
28
+
29
+ def to_fp32(model):
30
+ model.update(tree_map(
31
+ lambda p: p.astype(mx.float32) if isinstance(p, mx.array) and p.dtype == mx.bfloat16 else p,
32
+ model.parameters()))
33
+
34
+
35
+ def layers(model):
36
+ return model.language_model.model.layers
37
+
38
+
39
+ def switch(model, layer):
40
+ return layers(model)[layer].mlp.switch_mlp
41
+
42
+
43
+ def dense_expert(proj):
44
+ """Exact float32 weights (experts, out, in) of a native or affine expert projection."""
45
+ mode = proj.mode
46
+ if mode == "nvfp4":
47
+ w = mx.dequantize(proj.weight, proj.scales, None, group_size=16, bits=4, mode="nvfp4", dtype=mx.float32)
48
+ scale = proj.global_scale.astype(mx.float32).reshape(-1, 1, 1)
49
+ return w * scale if proj._scale_name == "weight_scale_2" else w / scale
50
+ if mode == "mxfp8":
51
+ return mx.dequantize(proj.weight, proj.scales, None, group_size=32, bits=8, mode="mxfp8", dtype=mx.float32)
52
+ return mx.dequantize(proj.weight, proj.scales, proj.get("biases"), group_size=proj.group_size, bits=proj.bits,
53
+ mode=mode, dtype=mx.float32)
54
+
55
+
56
+ # ----------------------------------------------------------------------------- affine packing
57
+ def pack_codes(codes, bits):
58
+ """Pack integer codes (..., n) into MLX's uint32 layout: a little-endian bit stream along the last axis."""
59
+ n = codes.shape[-1]
60
+ per = math.lcm(bits, 32) // bits # codes per repeating unit
61
+ words = math.lcm(bits, 32) // 32 # uint32 words per unit
62
+ assert n % per == 0, (n, per)
63
+ c = codes.astype(mx.uint32).reshape(*codes.shape[:-1], n // per, per)
64
+ out = [None] * words
65
+ for j in range(per):
66
+ off = bits * j
67
+ k, sh = off // 32, off % 32
68
+ v = c[..., j]
69
+ lo = mx.left_shift(v, mx.array(sh, mx.uint32)) if sh else v
70
+ out[k] = lo if out[k] is None else mx.bitwise_or(out[k], lo)
71
+ if sh + bits > 32:
72
+ hi = mx.right_shift(v, mx.array(32 - sh, mx.uint32))
73
+ out[k + 1] = hi if out[k + 1] is None else mx.bitwise_or(out[k + 1], hi)
74
+ return mx.stack(out, axis=-1).reshape(*codes.shape[:-1], n // per * words)
75
+
76
+
77
+ def affine_switch(packed, scales, biases, bits, group_size):
78
+ q = QuantizedSwitchLinear.__new__(QuantizedSwitchLinear)
79
+ nn.Module.__init__(q)
80
+ q.weight, q.scales, q.biases = packed, scales, biases
81
+ q.group_size, q.bits, q.mode = group_size, bits, "affine"
82
+ q.freeze()
83
+ return q
84
+
85
+
86
+ def rtn_affine(w, bits, group_size, scale_dtype=mx.float16):
87
+ """MLX's own round-to-nearest affine quantizer, with scales stored in `scale_dtype`."""
88
+ wq, s, b = mx.quantize(w, group_size=group_size, bits=bits, mode="affine")
89
+ return wq, s.astype(scale_dtype), b.astype(scale_dtype)
90
+
91
+
92
+ # ----------------------------------------------------------------------------- data
93
+ def render_ids(tok, rec):
94
+ """Token ids of a recorded conversation (scenario record from gen_chats.py)."""
95
+ import sys
96
+ sys.path.insert(0, os.path.join(ROOT, "scripts"))
97
+ from scenarios import TOOLS
98
+ tools = [TOOLS[t] for t in rec.get("tools", [])] or None
99
+ out = tok.apply_chat_template(rec["messages"], tools=tools, reasoning_effort=rec.get("effort"), tokenize=True)
100
+ return list(out["input_ids"]) if hasattr(out, "keys") else list(out)
101
+
102
+
103
+ def load_sequences(path, split=None, max_len=1024):
104
+ """Token sequences from a prepared JSON file: [{"name", "split", "kind", "ids"}]."""
105
+ seqs = json.load(open(path))
106
+ out = []
107
+ for s in seqs:
108
+ if split and s["split"] != split:
109
+ continue
110
+ ids = s["ids"]
111
+ for start in range(0, len(ids), max_len):
112
+ chunk = ids[start:start + max_len]
113
+ if len(chunk) >= 16:
114
+ out.append(dict(name=f"{s['name']}@{start}", kind=s["kind"], ids=chunk))
115
+ return out
116
+
117
+
118
+ def logits_for(model, ids):
119
+ out = model(mx.array([ids])).logits[0].astype(mx.float32)
120
+ mx.eval(out)
121
+ return out
122
+
123
+
124
+ def log_softmax(x):
125
+ return x - mx.logsumexp(x, axis=-1, keepdims=True)
126
+
127
+
128
+ def compare(teacher_logits, student_logits, ids):
129
+ """Per-sequence statistics; teacher and student are (T, V) float32 logits."""
130
+ lt, ls = log_softmax(teacher_logits), log_softmax(student_logits)
131
+ kl = (mx.exp(lt) * (lt - ls)).sum(-1)
132
+ nxt = mx.array(ids[1:])
133
+ nll_t = -mx.take_along_axis(lt[:-1], nxt[:, None], axis=-1)[:, 0]
134
+ nll_s = -mx.take_along_axis(ls[:-1], nxt[:, None], axis=-1)[:, 0]
135
+ agree = (mx.argmax(lt, -1) == mx.argmax(ls, -1)).astype(mx.float32)
136
+ vals = [kl.sum(), kl.max(), agree.sum(), nll_t.sum(), nll_s.sum()]
137
+ mx.eval(vals)
138
+ return dict(n=len(ids), kl_sum=vals[0].item(), kl_max=vals[1].item(), agree=vals[2].item(),
139
+ nll_t=vals[3].item(), nll_s=vals[4].item())
140
+
141
+
142
+ def summarize(rows):
143
+ n = sum(r["n"] for r in rows)
144
+ m = sum(r["n"] - 1 for r in rows)
145
+ return dict(tokens=n, kl=sum(r["kl_sum"] for r in rows) / n, kl_max=max(r["kl_max"] for r in rows),
146
+ top1=sum(r["agree"] for r in rows) / n, nll_teacher=sum(r["nll_t"] for r in rows) / m,
147
+ nll_student=sum(r["nll_s"] for r in rows) / m)
extras/scripts/recipes.py ADDED
@@ -0,0 +1,107 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Apply a quantization recipe to the loaded native model (in memory), and describe it for config.json.
2
+
3
+ A recipe is a dict:
4
+ {"experts": {"default": SPEC, "layers": {"20": SPEC, ...}}, SPEC = "native" | "rtn:BITS:GROUP" | "gptq:BITS:GROUP"
5
+ "dense": {"attn": Q, "embed": Q, "head": Q, "vision": Q, "draft": Q}} Q = null (keep float) | [BITS, GROUP]
6
+ "gptq" shards are read from `qdir` (files q_LNN_bBgG.safetensors written by gptq.py).
7
+ """
8
+
9
+ import os
10
+
11
+ import mlx.core as mx
12
+ import mlx.nn as nn
13
+
14
+ import qlab
15
+
16
+ ATTN_BIG = ("q_proj", "k_proj", "v_proj", "r_proj", "o_proj", "gate_proj")
17
+
18
+
19
+ def expert_spec(recipe, layer):
20
+ ex = recipe.get("experts", {})
21
+ return ex.get("layers", {}).get(str(layer), ex.get("default", "native"))
22
+
23
+
24
+ def dense_group(path, module):
25
+ """Which recipe group a dense module belongs to, or None if it must stay in floating point."""
26
+ if ".mlp.switch_mlp." in path:
27
+ return None
28
+ if path == "language_model.model.embed_tokens":
29
+ return "embed"
30
+ if path == "language_model.lm_head":
31
+ return "head"
32
+ if path.startswith("language_model.model.layers.") and ".self_attn." in path:
33
+ leaf = path.rsplit(".", 1)[-1]
34
+ # sink_proj (24 rows) and the dynamic-convolution tap projections are tiny and stay bf16.
35
+ return "attn" if leaf in ATTN_BIG and ".k_sconv." not in path and ".v_sconv." not in path else None
36
+ if path.startswith("vision_tower."):
37
+ return "vision"
38
+ return None
39
+
40
+
41
+ def apply_experts(model, recipe, qdir=None, log=None):
42
+ """Replace expert projections according to the recipe. Returns {module_path: quantization dict}."""
43
+ overrides = {}
44
+ act_dtype = model.language_model.model.embed_norm.weight.dtype
45
+ for li, layer in enumerate(qlab.layers(model)):
46
+ spec = expert_spec(recipe, li)
47
+ sw = layer.mlp.switch_mlp
48
+ base = f"language_model.model.layers.{li}.mlp.switch_mlp."
49
+ if spec == "native":
50
+ for p in qlab.PROJECTIONS:
51
+ proj = getattr(sw, p)
52
+ overrides[base + p] = {"group_size": proj.group_size, "bits": proj.bits, "mode": proj.mode}
53
+ continue
54
+ kind, bits, group = spec.split(":")
55
+ bits, group = int(bits), int(group)
56
+ if kind == "gptq":
57
+ shard = mx.load(os.path.join(qdir, f"q_L{li:02d}_b{bits}g{group}.safetensors"))
58
+ for p in qlab.PROJECTIONS:
59
+ if kind == "gptq":
60
+ packed, s, b = shard[p + ".weight"], shard[p + ".scales"], shard[p + ".biases"]
61
+ elif kind == "rtn":
62
+ packed, s, b = qlab.rtn_affine(qlab.dense_expert(getattr(sw, p)), bits, group)
63
+ else:
64
+ raise ValueError(spec)
65
+ # Scales and offsets are stored in the activation dtype. MLX's expert kernel (gather_qmm) promotes to
66
+ # float32 and runs about three times slower when their dtype differs from the input's.
67
+ s, b = s.astype(act_dtype), b.astype(act_dtype)
68
+ mx.eval(packed, s, b)
69
+ setattr(sw, p, qlab.affine_switch(packed, s, b, bits, group))
70
+ overrides[base + p] = {"group_size": group, "bits": bits, "mode": "affine"}
71
+ mx.clear_cache()
72
+ if log:
73
+ log(f"layer {li}: experts -> {spec}")
74
+ return overrides
75
+
76
+
77
+ def apply_dense(model, recipe):
78
+ """Affine-quantize dense groups. Returns {module_path: quantization dict}."""
79
+ dense = recipe.get("dense", {}) or {}
80
+ overrides = {}
81
+
82
+ def predicate(path, module):
83
+ if not hasattr(module, "to_quantized") or not hasattr(module, "weight"):
84
+ return False
85
+ q = dense.get(dense_group(path, module))
86
+ if not q:
87
+ return False
88
+ bits, group = q
89
+ if module.weight.shape[-1] % group != 0:
90
+ return False
91
+ overrides[path] = {"group_size": group, "bits": bits, "mode": "affine"}
92
+ return {"group_size": group, "bits": bits, "mode": "affine"}
93
+
94
+ nn.quantize(model, class_predicate=predicate)
95
+ return overrides
96
+
97
+
98
+ def apply(model, recipe, qdir=None, log=None):
99
+ overrides = apply_experts(model, recipe, qdir, log)
100
+ overrides.update(apply_dense(model, recipe))
101
+ mx.eval(model.parameters())
102
+ return overrides
103
+
104
+
105
+ def nbytes(model):
106
+ from mlx.utils import tree_flatten
107
+ return sum(v.nbytes for _, v in tree_flatten(model.parameters()))
extras/scripts/scenarios.py ADDED
@@ -0,0 +1,244 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Conversation scenarios for self-generated calibration / evaluation chats and agent fixtures."""
2
+
3
+ import json
4
+
5
+
6
+ def fn(name, description, properties, required):
7
+ return {
8
+ "type": "function",
9
+ "function": {
10
+ "name": name,
11
+ "description": description,
12
+ "parameters": {"type": "object", "properties": properties, "required": required},
13
+ },
14
+ }
15
+
16
+
17
+ TOOLS = {
18
+ "get_weather": fn(
19
+ "get_weather", "Get the current weather for a city.",
20
+ {"city": {"type": "string"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, ["city"],
21
+ ),
22
+ "terminal": fn(
23
+ "terminal", "Run a shell command in the project sandbox and return stdout, stderr and the exit code.",
24
+ {"command": {"type": "string", "description": "Command to run"}, "timeout": {"type": "integer"}}, ["command"],
25
+ ),
26
+ "read_file": fn(
27
+ "read_file", "Read a text file and return its contents with line numbers.",
28
+ {"path": {"type": "string"}, "offset": {"type": "integer"}, "limit": {"type": "integer"}}, ["path"],
29
+ ),
30
+ "write_file": fn(
31
+ "write_file", "Create or overwrite a text file.",
32
+ {"path": {"type": "string"}, "content": {"type": "string"}}, ["path", "content"],
33
+ ),
34
+ "web_search": fn(
35
+ "web_search", "Search the web and return the top results.",
36
+ {"query": {"type": "string"}, "max_results": {"type": "integer"}}, ["query"],
37
+ ),
38
+ "calculator": fn(
39
+ "calculator", "Evaluate an arithmetic expression.", {"expression": {"type": "string"}}, ["expression"]
40
+ ),
41
+ "search_orders": fn(
42
+ "search_orders", "Look up customer orders.",
43
+ {
44
+ "customer_id": {"type": "string"},
45
+ "status": {"type": "string", "enum": ["open", "shipped", "returned"]},
46
+ "filters": {
47
+ "type": "object",
48
+ "properties": {"min_total": {"type": "number"}, "tags": {"type": "array", "items": {"type": "string"}}},
49
+ },
50
+ },
51
+ ["customer_id"],
52
+ ),
53
+ }
54
+
55
+ FILES = {
56
+ "app/config.py": 'import os\n\nDEBUG = os.environ.get("DEBUG", "0") == "1"\nTIMEOUT = 30\nRETRIES = 3\nDATABASE_URL = os.environ["DATABASE_URL"]\n',
57
+ "README.md": "# Tidepool\n\nA small job queue.\n\n## Run\n\n make dev\n\n## Test\n\n pytest -q\n",
58
+ "src/queue.py": (
59
+ "import heapq\n\n\nclass JobQueue:\n def __init__(self):\n self._heap = []\n self._count = 0\n\n"
60
+ " def push(self, job, priority=0):\n heapq.heappush(self._heap, (priority, self._count, job))\n self._count += 1\n\n"
61
+ " def pop(self):\n return heapq.heappop(self._heap)[1]\n\n def __len__(self):\n return len(self._heap)\n"
62
+ ),
63
+ }
64
+
65
+
66
+ def _terminal(cmd):
67
+ """A tiny scripted shell over FILES: enough for an agent to explore, run tests and commit."""
68
+ import re
69
+
70
+ out, err, code = [], [], 0
71
+ for part in re.split(r"\s*(?:&&|;|\|\|)\s*", cmd):
72
+ part = part.strip()
73
+ if not part:
74
+ continue
75
+ head = part.split("|")[0].strip()
76
+ words = head.split()
77
+ prog = words[0] if words else ""
78
+ if "pytest" in part:
79
+ fixed = "return heapq.heappop(self._heap)[2]" in FILES["src/queue.py"]
80
+ if fixed:
81
+ out.append(".. [100%]\n2 passed in 0.02s")
82
+ else:
83
+ code = 1
84
+ out.append("F. [100%]\n=== FAILURES ===\n___ test_pop_returns_job ___\n def test_pop_returns_job():\n q = JobQueue(); q.push('a')\n> assert q.pop() == 'a'\nE AssertionError: assert 0 == 'a'\ntests/test_queue.py:7: AssertionError\n1 failed, 1 passed in 0.03s")
85
+ elif prog == "ls":
86
+ target = next((w for w in words[1:] if not w.startswith("-")), ".").rstrip("/")
87
+ names = sorted({f.split("/")[0] if target in (".", "") else f[len(target) + 1:].split("/")[0]
88
+ for f in list(FILES) + ["tests/test_queue.py", "Makefile"]
89
+ if target in (".", "") or f.startswith(target + "/")})
90
+ out.append("\n".join(names) if names else "") if names else err.append(f"ls: {target}: No such file or directory")
91
+ elif prog in ("cat", "head", "tail", "sed", "nl"):
92
+ paths = [w for w in words[1:]
93
+ if not w.startswith("-") and not w.isdigit() and ">" not in w and "," not in w and w != "p"]
94
+ for path in paths:
95
+ path = path.lstrip("./")
96
+ if path in FILES:
97
+ out.append(FILES[path].rstrip("\n"))
98
+ elif path == "Makefile":
99
+ out.append("dev:\n\tpython -m app\n\ntest:\n\tpytest -q")
100
+ elif path == "tests/test_queue.py":
101
+ out.append("from src.queue import JobQueue\n\n\ndef test_len():\n q = JobQueue(); q.push('a'); assert len(q) == 1\n\n\ndef test_pop_returns_job():\n q = JobQueue(); q.push('a')\n assert q.pop() == 'a'")
102
+ else:
103
+ code = 1; err.append(f"{prog}: {path}: No such file or directory")
104
+ elif prog == "pwd":
105
+ out.append("/workspace/tidepool")
106
+ elif prog == "echo":
107
+ out.append(" ".join(words[1:]).strip("'\""))
108
+ elif prog == "find":
109
+ out.append("\n".join("./" + f for f in sorted(list(FILES) + ["tests/test_queue.py", "Makefile"])))
110
+ elif prog == "grep":
111
+ pat = next((w.strip("'\"") for w in words[1:] if not w.startswith("-")), "")
112
+ hits = [f"{f}:{i + 1}:{line}" for f, body in FILES.items() for i, line in enumerate(body.splitlines()) if pat and pat in line]
113
+ out.append("\n".join(hits)) if hits else None
114
+ code = 0 if hits else 1
115
+ elif prog == "git":
116
+ if "status" in part:
117
+ out.append("On branch main\nnothing to commit, working tree clean")
118
+ elif "diff" in part:
119
+ out.append("")
120
+ elif "log" in part:
121
+ out.append("a1b2c3d Add priority queue\n9f8e7d6 Initial commit")
122
+ else:
123
+ out.append("")
124
+ elif prog in ("python", "python3", "make"):
125
+ out.append("")
126
+ elif prog == "wc":
127
+ out.append(" 17 src/queue.py")
128
+ else:
129
+ code = 127; err.append(f"sh: {prog}: command not found")
130
+ return {"exit_code": code, "stdout": "\n".join(x for x in out if x), "stderr": "\n".join(err)}
131
+
132
+
133
+ def run_tool(name, args):
134
+ """Deterministic canned tool results."""
135
+ if name == "get_weather":
136
+ table = {"lisbon": (19, "clear"), "oslo": (3, "snow"), "tokyo": (14, "rain"), "cairo": (28, "sunny")}
137
+ t, sky = table.get(str(args.get("city", "")).lower(), (12, "cloudy"))
138
+ if args.get("unit") == "fahrenheit":
139
+ return json.dumps({"city": args.get("city"), "temp_f": round(t * 9 / 5 + 32), "sky": sky})
140
+ return json.dumps({"city": args.get("city"), "temp_c": t, "sky": sky})
141
+ if name == "terminal":
142
+ return json.dumps(_terminal(str(args.get("command", ""))))
143
+ if name == "read_file":
144
+ body = FILES.get(str(args.get("path", "")))
145
+ if body is None:
146
+ return json.dumps({"error": f"No such file: {args.get('path')}"})
147
+ return "\n".join(f"{i + 1}\t{line}" for i, line in enumerate(body.splitlines()))
148
+ if name == "write_file":
149
+ path = str(args.get("path", "")).lstrip("./")
150
+ FILES[path] = str(args.get("content", ""))
151
+ return json.dumps({"ok": True, "bytes": len(FILES[path])})
152
+ if name == "web_search":
153
+ return json.dumps({"results": [
154
+ {"title": "MLX documentation", "url": "https://ml-explore.github.io/mlx/", "snippet": "MLX is an array framework for machine learning on Apple silicon."},
155
+ {"title": "Unified memory on Apple silicon", "url": "https://example.org/unified-memory", "snippet": "CPU and GPU share one memory pool, so arrays need no copies."},
156
+ ]})
157
+ if name == "calculator":
158
+ try:
159
+ return json.dumps({"result": eval(str(args.get("expression", "0")), {"__builtins__": {}}, {})})
160
+ except Exception as exc: # noqa: BLE001
161
+ return json.dumps({"error": str(exc)})
162
+ if name == "search_orders":
163
+ return json.dumps({"orders": [
164
+ {"id": "A-1042", "status": "shipped", "total": 129.5, "tags": ["gift"]},
165
+ {"id": "A-1077", "status": "open", "total": 48.0, "tags": []},
166
+ ]})
167
+ return json.dumps({"error": f"unknown tool {name}"})
168
+
169
+
170
+ AGENT_SYSTEM = (
171
+ "You are a coding agent working in a sandboxed repository. Use the tools to inspect and change files. "
172
+ "Keep answers short and report what you actually did."
173
+ )
174
+
175
+ ARTICLE = (
176
+ "The municipal archive reopened on Monday after a two-year renovation. The work replaced the roof, added a "
177
+ "climate-controlled vault for the oldest parish registers, and moved the reading room to the ground floor so "
178
+ "that visitors no longer need the narrow staircase. Opening hours are now 9:00 to 17:00 on weekdays and 10:00 "
179
+ "to 14:00 on Saturdays. Researchers must book a desk at least one day ahead; the archive holds 24 desks. "
180
+ "Digitisation continues: about 40 percent of the pre-1900 registers can be read online, and the team expects "
181
+ "to reach 60 percent by the end of next year. The renovation cost 3.2 million euros, of which the region paid half."
182
+ )
183
+
184
+ # (id, split, category, kwargs)
185
+ S = []
186
+
187
+
188
+ def add(sid, split, category, user, tools=None, system=None, effort="low", max_tokens=700, followups=()):
189
+ S.append(dict(id=sid, split=split, category=category, user=user, tools=tools or [], system=system,
190
+ effort=effort, max_tokens=max_tokens, followups=list(followups)))
191
+
192
+
193
+ # --- tool use (agent fixtures and calibration) ---
194
+ add("tool_weather_pair", "calib", "tools", "Is it warmer in Lisbon or in Oslo right now? Use the tool.", ["get_weather"])
195
+ add("tool_weather_single", "eval", "tools", "What's the weather in Tokyo in fahrenheit?", ["get_weather", "calculator"])
196
+ add("tool_none_needed", "calib", "tools", "Thanks! No lookup needed, just tell me a fun fact about lighthouses.", ["get_weather", "web_search"])
197
+ add("agent_fix_test", "calib", "agent", "The test suite is failing. Find out why and fix it.", ["terminal", "read_file", "write_file"], AGENT_SYSTEM, "medium", 900)
198
+ add("agent_explore", "eval", "agent", "What does this project do and how do I run its tests?", ["terminal", "read_file"], AGENT_SYSTEM, "low", 700)
199
+ add("agent_edit_config", "calib", "agent", "Change the request timeout to 60 seconds in the config and show me the diff in words.", ["terminal", "read_file", "write_file"], AGENT_SYSTEM, "low", 800)
200
+ add("agent_missing_file", "eval", "agent", "Open docs/ARCHITECTURE.md and summarise it.", ["read_file", "terminal"], AGENT_SYSTEM, "low", 500)
201
+ add("tool_nested_args", "calib", "tools", "Find customer C-88's shipped orders over 100 dollars that are tagged gift.", ["search_orders"], None, "low", 500)
202
+ add("tool_nested_args_eval", "eval", "tools", "List open orders for customer C-19 with a total of at least 40.", ["search_orders", "calculator"], None, "low", 500)
203
+ add("tool_search_then_answer", "calib", "tools", "Search for what MLX is and answer in two sentences with the source URL.", ["web_search"], None, "low", 500)
204
+ add("tool_calc_chain", "eval", "tools", "Use the calculator: what is (1234 * 57) - 999, and what is that divided by 7?", ["calculator"], None, "low", 500)
205
+ add("tool_multiturn", "calib", "tools", "What's the weather in Cairo?", ["get_weather"], None, "none", 400,
206
+ followups=["And in Oslo? Which one needs a coat?"])
207
+ # --- coding ---
208
+ add("code_lru", "calib", "code", "Write a small LRU cache class in Python with get and put, O(1) each, plus three assert-based tests.", max_tokens=800)
209
+ add("code_js_debounce", "calib", "code", "Write a debounce function in TypeScript with a cancel method and explain when leading-edge firing is useful.", max_tokens=700)
210
+ add("code_sql", "eval", "code", "Given tables orders(id, customer_id, total, created_at) and customers(id, country), write SQL for monthly revenue per country for 2025, and say which index helps.", max_tokens=700)
211
+ add("code_bash", "calib", "code", "Write a bash script that finds the ten largest files under a directory given as $1, safely handling spaces in names.", max_tokens=600)
212
+ add("code_review", "eval", "code", "Review this function and point out bugs:\n\n```python\ndef median(xs):\n xs.sort()\n n = len(xs)\n return xs[n/2] if n % 2 else (xs[n/2] + xs[n/2+1]) / 2\n```", max_tokens=700)
213
+ add("code_rust", "calib", "code", "Explain Rust's ownership rules with one short example that fails to compile and its fix.", max_tokens=700)
214
+ add("code_regex", "eval", "code", "Give a regular expression for ISO 8601 dates (YYYY-MM-DD) that rejects month 13 and day 32, with a brief explanation of each part.", max_tokens=600)
215
+ # --- reasoning / math ---
216
+ add("math_trains", "calib", "reasoning", "Two trains start 300 km apart and move toward each other at 70 km/h and 80 km/h. When do they meet, and how far has each travelled?", effort="medium")
217
+ add("math_prob", "eval", "reasoning", "I roll two fair dice. What is the probability that the product is even? Show the reasoning briefly.", effort="medium")
218
+ add("logic_puzzle", "calib", "reasoning", "Ana, Ben and Cy each own one pet: a cat, a dog, a fish. Ana does not own the dog. Ben owns neither the dog nor the fish. Who owns what?", effort="high", max_tokens=900)
219
+ add("estimate", "eval", "reasoning", "Estimate how many piano tuners work in a city of two million people. State your assumptions.", effort="medium")
220
+ add("plan_trip", "calib", "reasoning", "Plan a two-day itinerary for Lisbon for someone who likes architecture and dislikes crowds. Keep it under 200 words.", effort="low")
221
+ # --- summarisation / extraction / structured output ---
222
+ add("summ_article", "calib", "text", "Summarise in three bullet points:\n\n" + ARTICLE, effort="none")
223
+ add("extract_json", "calib", "structured", "Extract the facts from this text as JSON with keys opening_hours (object), desks (integer), cost_eur (number) and digitised_percent (integer). Output only the JSON.\n\n" + ARTICLE, effort="none", max_tokens=400)
224
+ add("extract_json_eval", "eval", "structured", "Return a JSON array of objects {\"name\", \"role\", \"year\"} for: Ada Lovelace wrote notes on the Analytical Engine in 1843; Grace Hopper built the A-0 compiler in 1952; Margaret Hamilton led Apollo flight software in 1969. Output only the JSON.", effort="none", max_tokens=400)
225
+ add("rewrite_tone", "eval", "text", "Rewrite this to be polite and concise: 'You sent the wrong file again. Fix it now, I am tired of waiting.'", effort="none", max_tokens=300)
226
+ add("classify", "calib", "structured", "Classify each review as positive, negative or mixed and answer as a markdown table:\n1. 'Battery lasts forever but the screen scratches easily.'\n2. 'Arrived broken.'\n3. 'Exactly what I needed.'", effort="none", max_tokens=400)
227
+ # --- knowledge / explanation ---
228
+ add("explain_tcp", "calib", "knowledge", "Explain the TCP three-way handshake to a new engineer in about 120 words.", effort="none", max_tokens=400)
229
+ add("explain_photosynth", "eval", "knowledge", "Explain photosynthesis at two levels: for a ten-year-old, then for a biology undergraduate.", effort="low", max_tokens=600)
230
+ add("compare", "calib", "knowledge", "Compare B-trees and LSM-trees for a write-heavy workload. Give a recommendation.", effort="low", max_tokens=600)
231
+ add("history", "eval", "knowledge", "Why did the Hanseatic League decline? Give the three main causes.", effort="low", max_tokens=500)
232
+ # --- multilingual ---
233
+ add("fr_email", "calib", "multilingual", "Rédige un court e-mail professionnel pour reporter une réunion à jeudi 15 h, en t'excusant du changement.", effort="none", max_tokens=350)
234
+ add("de_explain", "eval", "multilingual", "Erkläre in einfachem Deutsch, was eine Rekursion ist, mit einem Beispiel aus dem Alltag.", effort="none", max_tokens=350)
235
+ add("es_recipe", "calib", "multilingual", "Dame una receta sencilla de tortilla de patatas para dos personas, con cantidades.", effort="none", max_tokens=400)
236
+ add("ja_translate", "calib", "multilingual", "次の文を英語と中国語に翻訳してください:「明日の会議は午後三時からオンラインで行います。」", effort="none", max_tokens=300)
237
+ add("zh_summary", "eval", "multilingual", "用三句话解释什么是机器学习中的过拟合,并给出一个避免的方法。", effort="none", max_tokens=350)
238
+ add("pt_story", "calib", "multilingual", "Escreve uma história muito curta (80 palavras) sobre um farol e uma gaivota.", effort="none", max_tokens=350)
239
+ # --- system-prompted persona / format constraints ---
240
+ add("sys_terse", "calib", "system", "How do I undo my last git commit but keep the changes?", system="Answer in at most two sentences. Never use markdown.", effort="none", max_tokens=200)
241
+ add("sys_json_only", "eval", "system", "Book a table for four at 19:30 tomorrow under the name Okafor.", system='You are an intent parser. Reply with exactly one JSON object: {"intent": string, "slots": object}. No prose.', effort="none", max_tokens=250)
242
+ add("sys_refuse_scope", "calib", "system", "What's a good pasta recipe?", system="You are the support assistant for a tax-filing product. Only answer questions about the product; politely decline anything else.", effort="none", max_tokens=250)
243
+
244
+ SCENARIOS = S
extras/scripts/task_eval.py ADDED
@@ -0,0 +1,300 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Task-level checks through an OpenAI-compatible endpoint (the way an agent harness talks to the model).
2
+
3
+ Usage: task_eval.py <base_url> <model_id> <out.json> [--suites gsm8k,humaneval,json,tools] [--n-gsm 50] [--n-he 40]
4
+ Greedy decoding (temperature 0) so that variants are compared on the same footing.
5
+ """
6
+
7
+ import json
8
+ import os
9
+ import re
10
+ import subprocess
11
+ import sys
12
+ import tempfile
13
+ import time
14
+ import urllib.request
15
+
16
+ sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
17
+ from scenarios import TOOLS, run_tool # noqa: E402
18
+
19
+ ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
20
+ CONTROL = re.compile(r"<\|[a-z_0-9]+\|>")
21
+
22
+
23
+ def chat(base, model, messages, tools=None, max_tokens=800, effort="low", temperature=0.0, extra=None):
24
+ body = {"model": model, "messages": messages, "max_tokens": max_tokens, "temperature": temperature,
25
+ "reasoning_effort": effort}
26
+ if tools:
27
+ body["tools"] = tools
28
+ if extra:
29
+ body.update(extra)
30
+ req = urllib.request.Request(base.rstrip("/") + "/chat/completions", data=json.dumps(body).encode(),
31
+ headers={"content-type": "application/json"})
32
+ t0 = time.time()
33
+ with urllib.request.urlopen(req, timeout=900) as r:
34
+ data = json.load(r)
35
+ choice = data["choices"][0]
36
+ return choice["message"], choice.get("finish_reason"), data.get("usage", {}), time.time() - t0
37
+
38
+
39
+ def leaked(msg):
40
+ return bool(CONTROL.search((msg.get("content") or "")))
41
+
42
+
43
+ # ------------------------------------------------------------------ GSM8K
44
+ def number(text):
45
+ m = re.findall(r"Answer:\s*\$?\s*(-?[\d,]*\.?\d+)", text or "")
46
+ if not m:
47
+ m = re.findall(r"(-?[\d,]*\.?\d+)", text or "")
48
+ if not m:
49
+ return None
50
+ try:
51
+ return float(m[-1].replace(",", ""))
52
+ except ValueError:
53
+ return None
54
+
55
+
56
+ def suite_gsm8k(base, model, n):
57
+ import pyarrow.parquet as pq
58
+ rows = pq.read_table(os.path.join(ROOT, "data/evals/gsm8k_test.parquet")).to_pylist()
59
+ step = len(rows) // n
60
+ out = []
61
+ for i in range(n):
62
+ row = rows[i * step]
63
+ gold = float(row["answer"].split("####")[-1].strip().replace(",", ""))
64
+ msg, fin, usage, dt = chat(base, model, [{"role": "user", "content": row["question"] +
65
+ "\n\nSolve it, then give the final answer on the last line as 'Answer: <number>'."}],
66
+ max_tokens=900, effort="low")
67
+ pred = number(msg.get("content"))
68
+ out.append(dict(i=i * step, ok=pred is not None and abs(pred - gold) < 1e-4, pred=pred, gold=gold, finish=fin,
69
+ leaked=leaked(msg), tokens=usage.get("completion_tokens"), seconds=round(dt, 1)))
70
+ return out
71
+
72
+
73
+ # ------------------------------------------------------------------ HumanEval
74
+ SANDBOX = """(version 1)
75
+ (allow default)
76
+ (deny network*)
77
+ (deny file-write*)
78
+ (allow file-write* (subpath "{tmp}") (literal "/dev/null") (literal "/dev/dtracehelper"))
79
+ """
80
+
81
+
82
+ def run_python(code, timeout=15):
83
+ with tempfile.TemporaryDirectory() as tmp:
84
+ tmp = os.path.realpath(tmp)
85
+ path = os.path.join(tmp, "prog.py")
86
+ open(path, "w").write(code)
87
+ try:
88
+ p = subprocess.run(["/usr/bin/sandbox-exec", "-p", SANDBOX.format(tmp=tmp), sys.executable, "-I", "-B", path],
89
+ capture_output=True, text=True, timeout=timeout, cwd=tmp)
90
+ return p.returncode == 0, (p.stderr or "")[-300:]
91
+ except subprocess.TimeoutExpired:
92
+ return False, "timeout"
93
+
94
+
95
+ def extract_code(text):
96
+ blocks = re.findall(r"```(?:python|py)?\n(.*?)```", text or "", flags=re.S)
97
+ return max(blocks, key=len) if blocks else (text or "")
98
+
99
+
100
+ def suite_humaneval(base, model, n):
101
+ import pyarrow.parquet as pq
102
+ rows = pq.read_table(os.path.join(ROOT, "data/evals/humaneval.parquet")).to_pylist()
103
+ step = len(rows) // n
104
+ out = []
105
+ for i in range(n):
106
+ row = rows[i * step]
107
+ msg, fin, usage, dt = chat(base, model, [{"role": "user", "content":
108
+ "Complete this Python function. Reply with the full function (including the signature "
109
+ "and any imports) in one ```python block and nothing else.\n\n" + row["prompt"]}],
110
+ max_tokens=900, effort="low")
111
+ code = extract_code(msg.get("content"))
112
+ ok, err = run_python(code + "\n\n" + row["test"] + f"\n\ncheck({row['entry_point']})\n")
113
+ out.append(dict(task=row["task_id"], ok=ok, err=None if ok else err, finish=fin, leaked=leaked(msg),
114
+ tokens=usage.get("completion_tokens"), seconds=round(dt, 1)))
115
+ return out
116
+
117
+
118
+ # ------------------------------------------------------------------ structured output
119
+ JSON_CASES = [
120
+ ("Return a JSON object with keys \"city\" (string) and \"population_millions\" (number) for the capital of Japan. Output only JSON.",
121
+ lambda d: isinstance(d, dict) and d.get("city") == "Tokyo" and isinstance(d.get("population_millions"), (int, float))),
122
+ ("Output only a JSON array of the first six prime numbers.", lambda d: d == [2, 3, 5, 7, 11, 13]),
123
+ ("Extract as JSON {\"name\": string, \"email\": string, \"age\": integer} from: 'Contact Maria Okafor (34) at maria.okafor@example.org.' Output only JSON.",
124
+ lambda d: isinstance(d, dict) and d.get("name") == "Maria Okafor" and d.get("email") == "maria.okafor@example.org" and d.get("age") == 34),
125
+ ("Output only JSON: an object mapping each of the words \"apple\", \"kiwi\", \"banana\" to its number of letters.",
126
+ lambda d: d == {"apple": 5, "kiwi": 4, "banana": 6}),
127
+ ("Convert to JSON with keys \"items\" (array of {\"sku\": string, \"qty\": integer}) and \"rush\" (boolean): 'Two of SKU A-17 and five of SKU B-02, rush delivery please.' Output only JSON.",
128
+ lambda d: isinstance(d, dict) and d.get("rush") is True and sorted((x.get("sku"), x.get("qty")) for x in d.get("items", [])) == [("A-17", 2), ("B-02", 5)]),
129
+ ("Output only JSON: {\"sorted\": [...]} with the numbers 42, 7, 19, 3 in ascending order.", lambda d: d == {"sorted": [3, 7, 19, 42]}),
130
+ ("Return only a JSON object {\"sentiment\": \"positive\"|\"negative\"|\"neutral\", \"confidence\": number between 0 and 1} for: 'The update broke everything and support never answered.'",
131
+ lambda d: isinstance(d, dict) and d.get("sentiment") == "negative" and isinstance(d.get("confidence"), (int, float)) and 0 <= d["confidence"] <= 1),
132
+ ("Output only JSON: {\"date\": \"YYYY-MM-DD\"} for the 4th of July 2031.", lambda d: d == {"date": "2031-07-04"}),
133
+ ("Output only a JSON object with a key \"quote\" whose value is the exact text: She said \"stop\" and left.\\ Then nothing.",
134
+ lambda d: isinstance(d, dict) and d.get("quote") == 'She said "stop" and left.\\ Then nothing.'),
135
+ ("Output only JSON: {\"tree\": {\"value\": 1, \"children\": [{\"value\": 2, \"children\": []}, {\"value\": 3, \"children\": [{\"value\": 4, \"children\": []}]}]}} but with every value doubled.",
136
+ lambda d: d == {"tree": {"value": 2, "children": [{"value": 4, "children": []}, {"value": 6, "children": [{"value": 8, "children": []}]}]}}),
137
+ ]
138
+
139
+
140
+ def parse_json(text):
141
+ text = (text or "").strip()
142
+ m = re.search(r"```(?:json)?\n(.*?)```", text, flags=re.S)
143
+ if m:
144
+ text = m.group(1).strip()
145
+ return json.loads(text)
146
+
147
+
148
+ def suite_json(base, model, _n=None):
149
+ out = []
150
+ for i, (prompt, check) in enumerate(JSON_CASES):
151
+ msg, fin, usage, dt = chat(base, model, [{"role": "user", "content": prompt}], max_tokens=1200, effort="none")
152
+ try:
153
+ data = parse_json(msg.get("content")); valid = True
154
+ ok = bool(check(data))
155
+ except Exception: # noqa: BLE001
156
+ valid, ok = False, False
157
+ out.append(dict(i=i, ok=ok, valid_json=valid, finish=fin, leaked=leaked(msg), seconds=round(dt, 1),
158
+ content=None if ok else (msg.get("content") or "")[:200]))
159
+ return out
160
+
161
+
162
+ # ------------------------------------------------------------------ tool use
163
+ def args_of(call):
164
+ a = call["function"].get("arguments")
165
+ return json.loads(a) if isinstance(a, str) else a
166
+
167
+
168
+ SINGLE = [ # (tools, user, expected tool, argument check)
169
+ (["get_weather", "calculator"], "What's the weather in Tokyo in fahrenheit?", "get_weather",
170
+ lambda a: a.get("city", "").lower() == "tokyo" and a.get("unit") == "fahrenheit"),
171
+ (["get_weather", "web_search"], "Look up the weather for Cairo.", "get_weather", lambda a: a.get("city", "").lower() == "cairo"),
172
+ (["calculator", "web_search"], "Use the calculator to compute 48 * 365.", "calculator",
173
+ lambda a: re.sub(r"\s", "", a.get("expression", "")) in ("48*365", "365*48")),
174
+ (["terminal", "read_file", "write_file"], "List the files in the current directory.", "terminal", lambda a: "ls" in a.get("command", "")),
175
+ (["terminal", "read_file", "write_file"], "Show me the contents of app/config.py.", None,
176
+ lambda a: "app/config.py" in json.dumps(a)),
177
+ (["terminal", "read_file", "write_file"], "Create a file notes/todo.txt containing exactly the text: buy milk", None,
178
+ lambda a: (a.get("path", "").endswith("notes/todo.txt") and a.get("content", "").strip() == "buy milk")
179
+ or ("todo.txt" in a.get("command", "") and "buy milk" in a.get("command", ""))),
180
+ (["search_orders"], "Find customer C-88's shipped orders over 100 dollars that are tagged gift.", "search_orders",
181
+ lambda a: a.get("customer_id") == "C-88" and a.get("status") == "shipped" and (a.get("filters") or {}).get("min_total") == 100
182
+ and (a.get("filters") or {}).get("tags") == ["gift"]),
183
+ (["search_orders", "calculator"], "List open orders for customer C-19.", "search_orders",
184
+ lambda a: a.get("customer_id") == "C-19" and a.get("status") == "open"),
185
+ (["web_search", "get_weather"], "Search the web for the MLX array framework, three results at most.", "web_search",
186
+ lambda a: "mlx" in a.get("query", "").lower() and a.get("max_results", 3) <= 3),
187
+ (["terminal"], "Run the test suite with pytest in quiet mode.", "terminal", lambda a: "pytest" in a.get("command", "")),
188
+ (["write_file", "read_file"], "Write a file hello.py that prints the string He said \"hi\" on one line and a backslash \\ on the next.",
189
+ "write_file", lambda a: a.get("path", "").endswith("hello.py") and "print" in a.get("content", "")),
190
+ (["get_weather", "calculator", "web_search", "terminal", "read_file", "write_file", "search_orders"],
191
+ "Use the weather tool for Oslo in celsius.", "get_weather", lambda a: a.get("city", "").lower() == "oslo" and a.get("unit", "celsius") == "celsius"),
192
+ ]
193
+ NO_TOOL = [
194
+ (["get_weather", "web_search"], "No lookup needed: what is 2 + 2?"),
195
+ (["terminal", "read_file"], "Just say hello in French, no tools."),
196
+ ]
197
+ MULTI = [ # (tools, system, user, check(final_text, calls))
198
+ (["get_weather"], None, "Is it warmer in Lisbon or in Oslo right now? Use the tool.",
199
+ lambda text, calls: "lisbon" in text.lower() and len(calls) >= 2),
200
+ (["terminal", "read_file", "write_file"], "You are a coding agent in a sandboxed repository. Use the tools; keep answers short.",
201
+ "What is the value of RETRIES in app/config.py?", lambda text, calls: "3" in text and len(calls) >= 1),
202
+ (["terminal", "read_file", "write_file"], "You are a coding agent in a sandboxed repository. Use the tools; keep answers short.",
203
+ "Open docs/ARCHITECTURE.md and summarise it. If it does not exist, say so.",
204
+ lambda text, calls: any(w in text.lower() for w in ("not exist", "no such", "doesn't exist", "does not exist", "isn't", "missing", "no `docs", "not found", "there is no"))),
205
+ (["calculator"], None, "Use the calculator: what is (1234 * 57) - 999, and what is that divided by 7?",
206
+ lambda text, calls: "69339" in text.replace(",", "") and len(calls) >= 1),
207
+ (["search_orders"], None, "How many orders does customer C-88 have in total, and which is the most expensive?",
208
+ lambda text, calls: "A-1042" in text and len(calls) >= 1),
209
+ ]
210
+
211
+
212
+ def suite_tools(base, model, _n=None):
213
+ import scenarios
214
+ out = []
215
+ for i, (names, user, want, check) in enumerate(SINGLE):
216
+ msg, fin, usage, dt = chat(base, model, [{"role": "user", "content": user}], tools=[TOOLS[t] for t in names],
217
+ max_tokens=500, effort="low")
218
+ calls = msg.get("tool_calls") or []
219
+ rec = dict(kind="single", i=i, n_calls=len(calls), finish=fin, leaked=leaked(msg), seconds=round(dt, 1))
220
+ try:
221
+ a = args_of(calls[0]) if calls else None
222
+ rec["valid_json"] = a is not None
223
+ rec["right_tool"] = bool(calls) and (want is None or calls[0]["function"]["name"] == want)
224
+ rec["ok"] = bool(rec["right_tool"] and check(a) and fin == "tool_calls")
225
+ except Exception as exc: # noqa: BLE001
226
+ rec.update(valid_json=False, right_tool=False, ok=False, error=str(exc)[:120])
227
+ if not rec["ok"]:
228
+ rec["got"] = json.dumps(calls)[:300] if calls else (msg.get("content") or "")[:200]
229
+ out.append(rec)
230
+ for i, (names, user) in enumerate(NO_TOOL):
231
+ msg, fin, usage, dt = chat(base, model, [{"role": "user", "content": user}], tools=[TOOLS[t] for t in names],
232
+ max_tokens=300, effort="low")
233
+ calls = msg.get("tool_calls") or []
234
+ out.append(dict(kind="no_tool", i=i, ok=not calls and bool((msg.get("content") or "").strip()) and fin == "stop",
235
+ n_calls=len(calls), finish=fin, leaked=leaked(msg), seconds=round(dt, 1)))
236
+ for i, (names, system, user, check) in enumerate(MULTI):
237
+ scenarios.FILES.update(ORIG_FILES)
238
+ for k in list(scenarios.FILES):
239
+ if k not in ORIG_FILES:
240
+ del scenarios.FILES[k]
241
+ messages = ([{"role": "system", "content": system}] if system else []) + [{"role": "user", "content": user}]
242
+ all_calls, rounds, leak, fin, t0, err = [], 0, False, None, time.time(), None
243
+ try:
244
+ for rounds in range(1, 13):
245
+ msg, fin, usage, dt = chat(base, model, messages, tools=[TOOLS[t] for t in names], max_tokens=900, effort="low")
246
+ leak = leak or leaked(msg)
247
+ calls = msg.get("tool_calls") or []
248
+ clean = {"role": "assistant", "content": msg.get("content") or ""}
249
+ if calls:
250
+ clean["tool_calls"] = calls
251
+ messages.append(clean)
252
+ if not calls:
253
+ break
254
+ all_calls += calls
255
+ for c in calls:
256
+ messages.append({"role": "tool", "tool_call_id": c.get("id"), "name": c["function"]["name"],
257
+ "content": run_tool(c["function"]["name"], args_of(c) or {})})
258
+ final = messages[-1].get("content") or ""
259
+ ok = bool(messages[-1]["role"] == "assistant" and check(final, all_calls))
260
+ except Exception as exc: # noqa: BLE001
261
+ ok, final, err = False, "", str(exc)[:200]
262
+ out.append(dict(kind="multi", i=i, ok=ok, rounds=rounds, n_calls=len(all_calls), finish=fin, leaked=leak,
263
+ seconds=round(time.time() - t0, 1), final=final[:240], error=err))
264
+ return out
265
+
266
+
267
+ import scenarios as _sc # noqa: E402
268
+
269
+ ORIG_FILES = dict(_sc.FILES)
270
+ SUITES = {"gsm8k": suite_gsm8k, "humaneval": suite_humaneval, "json": suite_json, "tools": suite_tools}
271
+
272
+
273
+ def main():
274
+ args = sys.argv[1:]
275
+ base, model, out_path = args[:3]
276
+ suites, n = ["gsm8k", "humaneval", "json", "tools"], {"gsm8k": 50, "humaneval": 40}
277
+ rest = args[3:]
278
+ while rest:
279
+ if rest[0] == "--suites":
280
+ suites = rest[1].split(","); rest = rest[2:]
281
+ elif rest[0] == "--n-gsm":
282
+ n["gsm8k"] = int(rest[1]); rest = rest[2:]
283
+ elif rest[0] == "--n-he":
284
+ n["humaneval"] = int(rest[1]); rest = rest[2:]
285
+ else:
286
+ raise SystemExit("unknown flag " + rest[0])
287
+ results = json.load(open(out_path)) if os.path.exists(out_path) else {}
288
+ for s in suites:
289
+ if s in results:
290
+ continue
291
+ t0 = time.time()
292
+ rows = SUITES[s](base, model, n.get(s))
293
+ results[s] = dict(passed=sum(r["ok"] for r in rows), total=len(rows), leaked=sum(bool(r.get("leaked")) for r in rows),
294
+ seconds=round(time.time() - t0), rows=rows)
295
+ json.dump(results, open(out_path, "w"), indent=1)
296
+ print(f"{s:10s} {results[s]['passed']}/{results[s]['total']} control-token leaks {results[s]['leaked']} {results[s]['seconds']}s", flush=True)
297
+
298
+
299
+ if __name__ == "__main__":
300
+ main()
generation_config.json ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ {
2
+ "do_sample": true,
3
+ "temperature": 1.0,
4
+ "eos_token_id": 200006,
5
+ "pad_token_id": 199999
6
+ }
model-00001-of-00003.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:48fc215e4885bed15d0a8977160ee8dcc10b368bb29745ec7935d2dee583ec92
3
+ size 5293081058
model-00002-of-00003.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1ab6d9feb4feffb5604e52c4a23820aa0da17cdeb4391ae0f80a6f5652fc1d02
3
+ size 5266137884
model-00003-of-00003.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:bc21b22b6aee8a12f18e71c147c23242b8024dbd33602e123298fe7a0a17f781
3
+ size 2230036483
model.safetensors.index.json ADDED
The diff for this file is too large to render. See raw diff
 
mtp.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7c3b651317f01f73bcf4e1dceba66ef8b7e1a1eac6bc171e882e690a4b10a1d2
3
+ size 635801416
processor_config.json ADDED
@@ -0,0 +1,25 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "image_processor": {
3
+ "do_convert_rgb": true,
4
+ "do_normalize": true,
5
+ "do_rescale": true,
6
+ "image_mean": [
7
+ 0.5,
8
+ 0.5,
9
+ 0.5
10
+ ],
11
+ "image_processor_type": "Inkling2ImageProcessor",
12
+ "image_std": [
13
+ 0.5,
14
+ 0.5,
15
+ 0.5
16
+ ],
17
+ "rescale_factor": 0.00392156862745098,
18
+ "size": {
19
+ "height": 40,
20
+ "width": 40
21
+ }
22
+ },
23
+ "image_token": "<|unused_200054|>",
24
+ "processor_class": "Inkling2Processor"
25
+ }
special_tokens_map.json ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ "<|endoftext|>",
4
+ "<|message_user|>",
5
+ "<|message_model|>",
6
+ "<|message_system|>",
7
+ "<|message_tool|>",
8
+ "<|content_text|>",
9
+ "<|content_image|>",
10
+ "<|content_model_end_sampling|>",
11
+ "<|content_thinking|>",
12
+ "<|end_message|>",
13
+ "<|content_tool_error|>",
14
+ "<|content_xml|>",
15
+ "<|begin_of_text|>",
16
+ "<|content_invoke_tool_json|>",
17
+ "<|content_invoke_tool_text|>"
18
+ ],
19
+ "eos_token": "<|content_model_end_sampling|>",
20
+ "pad_token": "<|endoftext|>"
21
+ }
tiktoken/tokenizer.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f170372dd655728909228220d77cf6b8104a59f38a91262a7d2c4027906ec0d8
3
+ size 3386494
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7f332727f9a3f093dd452a39cc978e6757627e5dad1b0673aa4e75174f236dd6
3
+ size 26178678
tokenizer_config.json ADDED
@@ -0,0 +1,548 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "tokenizer_class": "PreTrainedTokenizerFast",
3
+ "model_max_length": 1048576,
4
+ "clean_up_tokenization_spaces": false,
5
+ "fix_mistral_regex": false,
6
+ "extra_special_tokens": {
7
+ "endoftext": "<|endoftext|>",
8
+ "message_user": "<|message_user|>",
9
+ "message_model": "<|message_model|>",
10
+ "message_system": "<|message_system|>",
11
+ "message_tool": "<|message_tool|>",
12
+ "content_text": "<|content_text|>",
13
+ "content_image": "<|content_image|>",
14
+ "content_model_end_sampling": "<|content_model_end_sampling|>",
15
+ "content_thinking": "<|content_thinking|>",
16
+ "end_message": "<|end_message|>",
17
+ "content_tool_error": "<|content_tool_error|>",
18
+ "content_xml": "<|content_xml|>",
19
+ "begin_of_text": "<|begin_of_text|>",
20
+ "content_invoke_tool_json": "<|content_invoke_tool_json|>",
21
+ "content_invoke_tool_text": "<|content_invoke_tool_text|>"
22
+ },
23
+ "added_tokens_decoder": {
24
+ "199998": {
25
+ "content": "<|unused|>",
26
+ "single_word": false,
27
+ "lstrip": false,
28
+ "rstrip": false,
29
+ "normalized": false,
30
+ "special": true
31
+ },
32
+ "199999": {
33
+ "content": "<|endoftext|>",
34
+ "single_word": false,
35
+ "lstrip": false,
36
+ "rstrip": false,
37
+ "normalized": false,
38
+ "special": true
39
+ },
40
+ "200000": {
41
+ "content": "<|message_user|>",
42
+ "single_word": false,
43
+ "lstrip": false,
44
+ "rstrip": false,
45
+ "normalized": false,
46
+ "special": true
47
+ },
48
+ "200001": {
49
+ "content": "<|message_model|>",
50
+ "single_word": false,
51
+ "lstrip": false,
52
+ "rstrip": false,
53
+ "normalized": false,
54
+ "special": true
55
+ },
56
+ "200002": {
57
+ "content": "<|message_system|>",
58
+ "single_word": false,
59
+ "lstrip": false,
60
+ "rstrip": false,
61
+ "normalized": false,
62
+ "special": true
63
+ },
64
+ "200003": {
65
+ "content": "<|message_tool|>",
66
+ "single_word": false,
67
+ "lstrip": false,
68
+ "rstrip": false,
69
+ "normalized": false,
70
+ "special": true
71
+ },
72
+ "200004": {
73
+ "content": "<|content_text|>",
74
+ "single_word": false,
75
+ "lstrip": false,
76
+ "rstrip": false,
77
+ "normalized": false,
78
+ "special": true
79
+ },
80
+ "200005": {
81
+ "content": "<|content_image|>",
82
+ "single_word": false,
83
+ "lstrip": false,
84
+ "rstrip": false,
85
+ "normalized": false,
86
+ "special": true
87
+ },
88
+ "200006": {
89
+ "content": "<|content_model_end_sampling|>",
90
+ "single_word": false,
91
+ "lstrip": false,
92
+ "rstrip": false,
93
+ "normalized": false,
94
+ "special": true
95
+ },
96
+ "200007": {
97
+ "content": "<|unused_200007|>",
98
+ "single_word": false,
99
+ "lstrip": false,
100
+ "rstrip": false,
101
+ "normalized": false,
102
+ "special": true
103
+ },
104
+ "200008": {
105
+ "content": "<|content_thinking|>",
106
+ "single_word": false,
107
+ "lstrip": false,
108
+ "rstrip": false,
109
+ "normalized": false,
110
+ "special": true
111
+ },
112
+ "200009": {
113
+ "content": "<|unused_200009|>",
114
+ "single_word": false,
115
+ "lstrip": false,
116
+ "rstrip": false,
117
+ "normalized": false,
118
+ "special": true
119
+ },
120
+ "200010": {
121
+ "content": "<|end_message|>",
122
+ "single_word": false,
123
+ "lstrip": false,
124
+ "rstrip": false,
125
+ "normalized": false,
126
+ "special": true
127
+ },
128
+ "200011": {
129
+ "content": "<|unused_200011|>",
130
+ "single_word": false,
131
+ "lstrip": false,
132
+ "rstrip": false,
133
+ "normalized": false,
134
+ "special": true
135
+ },
136
+ "200012": {
137
+ "content": "<|unused_200012|>",
138
+ "single_word": false,
139
+ "lstrip": false,
140
+ "rstrip": false,
141
+ "normalized": false,
142
+ "special": true
143
+ },
144
+ "200013": {
145
+ "content": "<|unused_200013|>",
146
+ "single_word": false,
147
+ "lstrip": false,
148
+ "rstrip": false,
149
+ "normalized": false,
150
+ "special": true
151
+ },
152
+ "200014": {
153
+ "content": "<|unused_200014|>",
154
+ "single_word": false,
155
+ "lstrip": false,
156
+ "rstrip": false,
157
+ "normalized": false,
158
+ "special": true
159
+ },
160
+ "200015": {
161
+ "content": "<|unused_200015|>",
162
+ "single_word": false,
163
+ "lstrip": false,
164
+ "rstrip": false,
165
+ "normalized": false,
166
+ "special": true
167
+ },
168
+ "200016": {
169
+ "content": "<|unused_200016|>",
170
+ "single_word": false,
171
+ "lstrip": false,
172
+ "rstrip": false,
173
+ "normalized": false,
174
+ "special": true
175
+ },
176
+ "200017": {
177
+ "content": "<|unused_200017|>",
178
+ "single_word": false,
179
+ "lstrip": false,
180
+ "rstrip": false,
181
+ "normalized": false,
182
+ "special": true
183
+ },
184
+ "200018": {
185
+ "content": "<|unused_200018|>",
186
+ "single_word": false,
187
+ "lstrip": false,
188
+ "rstrip": false,
189
+ "normalized": false,
190
+ "special": true
191
+ },
192
+ "200019": {
193
+ "content": "<|unused_200019|>",
194
+ "single_word": false,
195
+ "lstrip": false,
196
+ "rstrip": false,
197
+ "normalized": false,
198
+ "special": true
199
+ },
200
+ "200020": {
201
+ "content": "<|unused_200020|>",
202
+ "single_word": false,
203
+ "lstrip": false,
204
+ "rstrip": false,
205
+ "normalized": false,
206
+ "special": true
207
+ },
208
+ "200021": {
209
+ "content": "<|unused_200021|>",
210
+ "single_word": false,
211
+ "lstrip": false,
212
+ "rstrip": false,
213
+ "normalized": false,
214
+ "special": true
215
+ },
216
+ "200022": {
217
+ "content": "<|content_tool_error|>",
218
+ "single_word": false,
219
+ "lstrip": false,
220
+ "rstrip": false,
221
+ "normalized": false,
222
+ "special": true
223
+ },
224
+ "200023": {
225
+ "content": "<|unused_200023|>",
226
+ "single_word": false,
227
+ "lstrip": false,
228
+ "rstrip": false,
229
+ "normalized": false,
230
+ "special": true
231
+ },
232
+ "200024": {
233
+ "content": "<|content_xml|>",
234
+ "single_word": false,
235
+ "lstrip": false,
236
+ "rstrip": false,
237
+ "normalized": false,
238
+ "special": true
239
+ },
240
+ "200025": {
241
+ "content": "<|unused_200025|>",
242
+ "single_word": false,
243
+ "lstrip": false,
244
+ "rstrip": false,
245
+ "normalized": false,
246
+ "special": true
247
+ },
248
+ "200026": {
249
+ "content": "<|unused_200026|>",
250
+ "single_word": false,
251
+ "lstrip": false,
252
+ "rstrip": false,
253
+ "normalized": false,
254
+ "special": true
255
+ },
256
+ "200027": {
257
+ "content": "<|unused_200027|>",
258
+ "single_word": false,
259
+ "lstrip": false,
260
+ "rstrip": false,
261
+ "normalized": false,
262
+ "special": true
263
+ },
264
+ "200028": {
265
+ "content": "<|begin_of_text|>",
266
+ "single_word": false,
267
+ "lstrip": false,
268
+ "rstrip": false,
269
+ "normalized": false,
270
+ "special": true
271
+ },
272
+ "200029": {
273
+ "content": "<|unused_200029|>",
274
+ "single_word": false,
275
+ "lstrip": false,
276
+ "rstrip": false,
277
+ "normalized": false,
278
+ "special": true
279
+ },
280
+ "200030": {
281
+ "content": "<|unused_200030|>",
282
+ "single_word": false,
283
+ "lstrip": false,
284
+ "rstrip": false,
285
+ "normalized": false,
286
+ "special": true
287
+ },
288
+ "200031": {
289
+ "content": "<|unused_200031|>",
290
+ "single_word": false,
291
+ "lstrip": false,
292
+ "rstrip": false,
293
+ "normalized": false,
294
+ "special": true
295
+ },
296
+ "200032": {
297
+ "content": "<|unused_200032|>",
298
+ "single_word": false,
299
+ "lstrip": false,
300
+ "rstrip": false,
301
+ "normalized": false,
302
+ "special": true
303
+ },
304
+ "200033": {
305
+ "content": "<|unused_200033|>",
306
+ "single_word": false,
307
+ "lstrip": false,
308
+ "rstrip": false,
309
+ "normalized": false,
310
+ "special": true
311
+ },
312
+ "200034": {
313
+ "content": "<|unused_200034|>",
314
+ "single_word": false,
315
+ "lstrip": false,
316
+ "rstrip": false,
317
+ "normalized": false,
318
+ "special": true
319
+ },
320
+ "200035": {
321
+ "content": "<|unused_200035|>",
322
+ "single_word": false,
323
+ "lstrip": false,
324
+ "rstrip": false,
325
+ "normalized": false,
326
+ "special": true
327
+ },
328
+ "200036": {
329
+ "content": "<|unused_200036|>",
330
+ "single_word": false,
331
+ "lstrip": false,
332
+ "rstrip": false,
333
+ "normalized": false,
334
+ "special": true
335
+ },
336
+ "200037": {
337
+ "content": "<|unused_200037|>",
338
+ "single_word": false,
339
+ "lstrip": false,
340
+ "rstrip": false,
341
+ "normalized": false,
342
+ "special": true
343
+ },
344
+ "200038": {
345
+ "content": "<|unused_200038|>",
346
+ "single_word": false,
347
+ "lstrip": false,
348
+ "rstrip": false,
349
+ "normalized": false,
350
+ "special": true
351
+ },
352
+ "200039": {
353
+ "content": "<|unused_200039|>",
354
+ "single_word": false,
355
+ "lstrip": false,
356
+ "rstrip": false,
357
+ "normalized": false,
358
+ "special": true
359
+ },
360
+ "200040": {
361
+ "content": "<|unused_200040|>",
362
+ "single_word": false,
363
+ "lstrip": false,
364
+ "rstrip": false,
365
+ "normalized": false,
366
+ "special": true
367
+ },
368
+ "200041": {
369
+ "content": "<|unused_200041|>",
370
+ "single_word": false,
371
+ "lstrip": false,
372
+ "rstrip": false,
373
+ "normalized": false,
374
+ "special": true
375
+ },
376
+ "200042": {
377
+ "content": "<|unused_200042|>",
378
+ "single_word": false,
379
+ "lstrip": false,
380
+ "rstrip": false,
381
+ "normalized": false,
382
+ "special": true
383
+ },
384
+ "200043": {
385
+ "content": "<|unused_200043|>",
386
+ "single_word": false,
387
+ "lstrip": false,
388
+ "rstrip": false,
389
+ "normalized": false,
390
+ "special": true
391
+ },
392
+ "200044": {
393
+ "content": "<|unused_200044|>",
394
+ "single_word": false,
395
+ "lstrip": false,
396
+ "rstrip": false,
397
+ "normalized": false,
398
+ "special": true
399
+ },
400
+ "200045": {
401
+ "content": "<|unused_200045|>",
402
+ "single_word": false,
403
+ "lstrip": false,
404
+ "rstrip": false,
405
+ "normalized": false,
406
+ "special": true
407
+ },
408
+ "200046": {
409
+ "content": "<|unused_200046|>",
410
+ "single_word": false,
411
+ "lstrip": false,
412
+ "rstrip": false,
413
+ "normalized": false,
414
+ "special": true
415
+ },
416
+ "200047": {
417
+ "content": "<|unused_200047|>",
418
+ "single_word": false,
419
+ "lstrip": false,
420
+ "rstrip": false,
421
+ "normalized": false,
422
+ "special": true
423
+ },
424
+ "200048": {
425
+ "content": "<|unused_200048|>",
426
+ "single_word": false,
427
+ "lstrip": false,
428
+ "rstrip": false,
429
+ "normalized": false,
430
+ "special": true
431
+ },
432
+ "200049": {
433
+ "content": "<|content_invoke_tool_json|>",
434
+ "single_word": false,
435
+ "lstrip": false,
436
+ "rstrip": false,
437
+ "normalized": false,
438
+ "special": true
439
+ },
440
+ "200050": {
441
+ "content": "<|unused_200050|>",
442
+ "single_word": false,
443
+ "lstrip": false,
444
+ "rstrip": false,
445
+ "normalized": false,
446
+ "special": true
447
+ },
448
+ "200051": {
449
+ "content": "<|unused_200051|>",
450
+ "single_word": false,
451
+ "lstrip": false,
452
+ "rstrip": false,
453
+ "normalized": false,
454
+ "special": true
455
+ },
456
+ "200052": {
457
+ "content": "<|unused_200052|>",
458
+ "single_word": false,
459
+ "lstrip": false,
460
+ "rstrip": false,
461
+ "normalized": false,
462
+ "special": true
463
+ },
464
+ "200053": {
465
+ "content": "<|unused_200053|>",
466
+ "single_word": false,
467
+ "lstrip": false,
468
+ "rstrip": false,
469
+ "normalized": false,
470
+ "special": true
471
+ },
472
+ "200054": {
473
+ "content": "<|unused_200054|>",
474
+ "single_word": false,
475
+ "lstrip": false,
476
+ "rstrip": false,
477
+ "normalized": false,
478
+ "special": true
479
+ },
480
+ "200055": {
481
+ "content": "<|unused_200055|>",
482
+ "single_word": false,
483
+ "lstrip": false,
484
+ "rstrip": false,
485
+ "normalized": false,
486
+ "special": true
487
+ },
488
+ "200056": {
489
+ "content": "<|unused_200056|>",
490
+ "single_word": false,
491
+ "lstrip": false,
492
+ "rstrip": false,
493
+ "normalized": false,
494
+ "special": true
495
+ },
496
+ "200057": {
497
+ "content": "<|content_invoke_tool_text|>",
498
+ "single_word": false,
499
+ "lstrip": false,
500
+ "rstrip": false,
501
+ "normalized": false,
502
+ "special": true
503
+ }
504
+ },
505
+ "response_template": {
506
+ "defaults": {
507
+ "role": "assistant"
508
+ },
509
+ "start_anchor_pattern": "<\\|end_message\\|>|<\\|content_model_end_sampling\\|>",
510
+ "fields": {
511
+ "reasoning_content": {
512
+ "open_pattern": "(?:<\\|message_model\\|>[^<]*)?<\\|content_thinking\\|>",
513
+ "close": "<|end_message|>",
514
+ "repeats": true,
515
+ "join": "",
516
+ "content": "text",
517
+ "content_args": {
518
+ "strip": false
519
+ }
520
+ },
521
+ "content": {
522
+ "open_pattern": "(?:<\\|message_model\\|>[^<]*)?<\\|content_(?:text|tool_error|invoke_tool_text)\\|>",
523
+ "close": "<|end_message|>",
524
+ "repeats": true,
525
+ "join": "",
526
+ "content": "text",
527
+ "content_args": {
528
+ "strip": false
529
+ }
530
+ },
531
+ "tool_calls": {
532
+ "open_pattern": "(?:<\\|message_model\\|>[^<]*)?<\\|content_invoke_tool_json\\|>",
533
+ "close": "<|end_message|>",
534
+ "repeats": true,
535
+ "content": "json",
536
+ "transform": {
537
+ "type": "function",
538
+ "function": {
539
+ "name": "{content.name}",
540
+ "arguments": "{content.args}"
541
+ }
542
+ }
543
+ }
544
+ }
545
+ },
546
+ "eos_token": "<|content_model_end_sampling|>",
547
+ "pad_token": "<|endoftext|>"
548
+ }