cloudyu commited on
Commit
615b72a
·
verified ·
1 Parent(s): a91288d

Add files using upload-large-folder tool

Browse files
.gitattributes CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ model.safetensors.index.json filter=lfs diff=lfs merge=lfs -text
37
+ gguf-lora/hy4-flash-smart-lora.gguf filter=lfs diff=lfs merge=lfs -text
LICENSE ADDED
@@ -0,0 +1,245 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Tencent Hy License, Version 1.0
2
+
3
+ This License is derived from the Apache License, Version 2.0
4
+ (http://www.apache.org/licenses/LICENSE-2.0), with modifications, mainly
5
+ the additional commercial terms for large Model-as-a-Service ("MaaS")
6
+ businesses in Section 10. This License is not the Apache License,
7
+ Version 2.0, and is not endorsed by or affiliated with the Apache
8
+ Software Foundation.
9
+
10
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
11
+
12
+ 1. Definitions.
13
+
14
+ "License" shall mean the terms and conditions for use, reproduction,
15
+ and distribution as defined by Sections 1 through 10 of this document.
16
+
17
+ "Licensor" shall mean the copyright owner or entity authorized by
18
+ the copyright owner that is granting the License.
19
+
20
+ "Legal Entity" shall mean the union of the acting entity and all
21
+ other entities that control, are controlled by, or are under common
22
+ control with that entity. For the purposes of this definition,
23
+ "control" means (i) the power, direct or indirect, to cause the
24
+ direction or management of such entity, whether by contract or
25
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
26
+ outstanding shares, or (iii) beneficial ownership of such entity.
27
+
28
+ "You" (or "Your") shall mean an individual or Legal Entity
29
+ exercising permissions granted by this License.
30
+
31
+ "Source" form shall mean the preferred form for making modifications,
32
+ including but not limited to software source code, documentation
33
+ source, and configuration files.
34
+
35
+ "Object" form shall mean any form resulting from mechanical
36
+ transformation or translation of a Source form, including but
37
+ not limited to compiled object code, generated documentation,
38
+ and conversions to other media types.
39
+
40
+ "Work" shall mean the work of authorship, whether in Source or
41
+ Object form, made available under the License, as indicated by a
42
+ copyright notice that is included in or attached to the work
43
+ (an example is provided in the Appendix below). For the avoidance
44
+ of doubt, the Work includes the model weights, parameters,
45
+ configuration files, inference and training code, and documentation
46
+ made available by Licensor under this License.
47
+
48
+ "Derivative Works" shall mean any work, whether in Source or Object
49
+ form, that is based on (or derived from) the Work and for which the
50
+ editorial revisions, annotations, elaborations, or other modifications
51
+ represent, as a whole, an original work of authorship. For the purposes
52
+ of this License, Derivative Works shall not include works that remain
53
+ separable from, or merely link (or bind by name) to the interfaces of,
54
+ the Work and Derivative Works thereof. For the avoidance of doubt,
55
+ Derivative Works include any fine-tuned, quantized, distilled, merged,
56
+ pruned, or otherwise modified versions of the model weights.
57
+
58
+ "Contribution" shall mean any work of authorship, including
59
+ the original version of the Work and any modifications or additions
60
+ to that Work or Derivative Works thereof, that is intentionally
61
+ submitted to Licensor for inclusion in the Work by the copyright owner
62
+ or by an individual or Legal Entity authorized to submit on behalf of
63
+ the copyright owner. For the purposes of this definition, "submitted"
64
+ means any form of electronic, verbal, or written communication sent
65
+ to the Licensor or its representatives, including but not limited to
66
+ communication on electronic mailing lists, source code control systems,
67
+ and issue tracking systems that are managed by, or on behalf of, the
68
+ Licensor for the purpose of discussing and improving the Work, but
69
+ excluding communication that is conspicuously marked or otherwise
70
+ designated in writing by the copyright owner as "Not a Contribution."
71
+
72
+ "Contributor" shall mean Licensor and any individual or Legal Entity
73
+ on behalf of whom a Contribution has been received by Licensor and
74
+ subsequently incorporated within the Work.
75
+
76
+ 2. Grant of Copyright License. Subject to the terms and conditions of
77
+ this License, each Contributor hereby grants to You a perpetual,
78
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
79
+ copyright license to reproduce, prepare Derivative Works of,
80
+ publicly display, publicly perform, sublicense, and distribute the
81
+ Work and such Derivative Works in Source or Object form.
82
+
83
+ 3. Grant of Patent License. Subject to the terms and conditions of
84
+ this License, each Contributor hereby grants to You a perpetual,
85
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
86
+ (except as stated in this section) patent license to make, have made,
87
+ use, offer to sell, sell, import, and otherwise transfer the Work,
88
+ where such license applies only to those patent claims licensable
89
+ by such Contributor that are necessarily infringed by their
90
+ Contribution(s) alone or by combination of their Contribution(s)
91
+ with the Work to which such Contribution(s) was submitted. If You
92
+ institute patent litigation against any entity (including a
93
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
94
+ or a Contribution incorporated within the Work constitutes direct
95
+ or contributory patent infringement, then any patent licenses
96
+ granted to You under this License for that Work shall terminate
97
+ as of the date such litigation is filed.
98
+
99
+ 4. Redistribution. You may reproduce and distribute copies of the
100
+ Work or Derivative Works thereof in any medium, with or without
101
+ modifications, and in Source or Object form, provided that You
102
+ meet the following conditions:
103
+
104
+ (a) You must give any other recipients of the Work or
105
+ Derivative Works a copy of this License; and
106
+
107
+ (b) You must cause any modified files to carry prominent notices
108
+ stating that You changed the files; and
109
+
110
+ (c) You must retain, in the Source form of any Derivative Works
111
+ that You distribute, all copyright, patent, trademark, and
112
+ attribution notices from the Source form of the Work,
113
+ excluding those notices that do not pertain to any part of
114
+ the Derivative Works; and
115
+
116
+ (d) If the Work includes a "NOTICE" text file as part of its
117
+ distribution, then any Derivative Works that You distribute must
118
+ include a readable copy of the attribution notices contained
119
+ within such NOTICE file, excluding those notices that do not
120
+ pertain to any part of the Derivative Works, in at least one
121
+ of the following places: within a NOTICE text file distributed
122
+ as part of the Derivative Works; within the Source form or
123
+ documentation, if provided along with the Derivative Works; or,
124
+ within a display generated by the Derivative Works, if and
125
+ wherever such third-party notices normally appear. The contents
126
+ of the NOTICE file are for informational purposes only and
127
+ do not modify the License. You may add Your own attribution
128
+ notices within Derivative Works that You distribute, alongside
129
+ or as an addendum to the NOTICE text from the Work, provided
130
+ that such additional attribution notices cannot be construed
131
+ as modifying the License.
132
+
133
+ You may add Your own copyright statement to Your modifications and
134
+ may provide additional or different license terms and conditions
135
+ for use, reproduction, or distribution of Your modifications, or
136
+ for any such Derivative Works as a whole, provided Your use,
137
+ reproduction, and distribution of the Work otherwise complies with
138
+ the conditions stated in this License. Notwithstanding the
139
+ foregoing, any license terms You provide for the Work or any
140
+ Derivative Works must not remove, limit, or conflict with Section 10,
141
+ and Section 10 shall continue to apply to every recipient of the
142
+ Work or any Derivative Works.
143
+
144
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
145
+ any Contribution intentionally submitted for inclusion in the Work
146
+ by You to the Licensor shall be under the terms and conditions of
147
+ this License, without any additional terms or conditions.
148
+ Notwithstanding the above, nothing herein shall supersede or modify
149
+ the terms of any separate license agreement you may have executed
150
+ with Licensor regarding such Contributions.
151
+
152
+ 6. Trademarks. This License does not grant permission to use the trade
153
+ names, trademarks, service marks, or product names of the Licensor,
154
+ except as required for reasonable and customary use in describing the
155
+ origin of the Work and reproducing the content of the NOTICE file.
156
+
157
+ 7. Disclaimer of Warranty. Unless required by applicable law or
158
+ agreed to in writing, Licensor provides the Work (and each
159
+ Contributor provides its Contributions) on an "AS IS" BASIS,
160
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
161
+ implied, including, without limitation, any warranties or conditions
162
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
163
+ PARTICULAR PURPOSE. You are solely responsible for determining the
164
+ appropriateness of using or redistributing the Work and assume any
165
+ risks associated with Your exercise of permissions under this License.
166
+
167
+ 8. Limitation of Liability. In no event and under no legal theory,
168
+ whether in tort (including negligence), contract, or otherwise,
169
+ unless required by applicable law (such as deliberate and grossly
170
+ negligent acts) or agreed to in writing, shall any Contributor be
171
+ liable to You for damages, including any direct, indirect, special,
172
+ incidental, or consequential damages of any character arising as a
173
+ result of this License or out of the use or inability to use the
174
+ Work (including but not limited to damages for loss of goodwill,
175
+ work stoppage, computer failure or malfunction, or any and all
176
+ other commercial damages or losses), even if such Contributor
177
+ has been advised of the possibility of such damages.
178
+
179
+ 9. Accepting Warranty or Additional Liability. While redistributing
180
+ the Work or Derivative Works thereof, You may choose to offer,
181
+ and charge a fee for, acceptance of support, warranty, indemnity,
182
+ or other liability obligations and/or rights consistent with this
183
+ License. However, in accepting such obligations, You may act only
184
+ on Your own behalf and on Your sole responsibility, not on behalf
185
+ of any other Contributor, and only if You agree to indemnify,
186
+ defend, and hold each Contributor harmless for any liability
187
+ incurred by, or claims asserted against, such Contributor by reason
188
+ of your accepting any such warranty or additional liability.
189
+
190
+ 10. Additional Commercial Terms.
191
+
192
+ 10.1 MaaS. "MaaS" means giving third parties remote access to model
193
+ inference or fine-tuning (e.g., via API), allowing them to
194
+ submit inputs or specify parameters or training data. This
195
+ excludes (i) end-user products with model capabilities confined
196
+ to specific features or workflows, without general-purpose
197
+ model access, or (ii) merely relaying requests to an
198
+ independently operated third-party model service.
199
+
200
+ 10.2 Separate License Required. Notwithstanding Sections 2 and 3,
201
+ and except as provided in Section 10.3, if You operate a MaaS
202
+ business, You must obtain a separate written license to use the
203
+ Work or any Derivative Works from Licensor (including by email),
204
+ in either of the following cases:
205
+
206
+ (a) if Your aggregate worldwide gross revenue exceeds
207
+ US$50 million (or the equivalent in other currencies) in
208
+ any consecutive 12-month period, whether or not such
209
+ revenue is derived from the Work or any Derivative Works,
210
+ before using (or continuing to use) the Work or any
211
+ Derivative Works; or
212
+
213
+ (b) before using the Work or any Derivative Works in or for any
214
+ product or service provided by You that has more than
215
+ 100 million monthly active users in any calendar month.
216
+
217
+ 10.3 Internal Use. Section 10.2 does not apply to Your use of the
218
+ Work or any Derivative Works solely for internal purposes,
219
+ provided that neither the Work, any Derivative Works, nor their
220
+ outputs are made available to third parties.
221
+
222
+ 10.4 Contact. For licensing requests or questions, please contact
223
+ hy_partners@tencent.com.
224
+
225
+ END OF TERMS AND CONDITIONS
226
+
227
+ APPENDIX: How to apply the Tencent Hy License to your work.
228
+
229
+ To apply this License to a Derivative Work, attach the following
230
+ notice, with the fields enclosed by brackets "[]" replaced with your
231
+ own identifying information. (Don't include the brackets!)
232
+
233
+ Copyright [yyyy] [name of copyright owner]
234
+
235
+ Licensed under the Tencent Hy License, Version 1.0 (the "License"),
236
+ which is based on the Apache License, Version 2.0, with additional
237
+ commercial terms. You may not use this file except in compliance
238
+ with the License. You may obtain a copy of the License in the
239
+ LICENSE file distributed with this work.
240
+
241
+ Unless required by applicable law or agreed to in writing, software
242
+ distributed under the License is distributed on an "AS IS" BASIS,
243
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
244
+ implied. See the License for the specific language governing
245
+ permissions and limitations under the License.
README.md ADDED
@@ -0,0 +1,121 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: other
3
+ license_name: tencent-hy-license
4
+ license_link: https://huggingface.co/tencent/Hy4-flash/blob/main/LICENSE
5
+ base_model:
6
+ - tencent/Hy4-flash
7
+ - atakans/Hy4-flash-NVFP4
8
+ tags:
9
+ - hy4-flash
10
+ - weight-editing
11
+ - weight-space-programming
12
+ - reasoning
13
+ - nvfp4
14
+ - moe
15
+ library_name: transformers
16
+ pipeline_tag: text-generation
17
+ ---
18
+
19
+ # Hy4-flash-smart
20
+
21
+ **Hy4-flash that thinks less.** Nine shared-expert down-projections of [tencent/Hy4-flash](https://huggingface.co/tencent/Hy4-flash)
22
+ are rewritten with a rank-one, gradient-free edit (weight-space programming). There are no gradient steps and no
23
+ fine-tuning data. The edit comes from the model's own activations and is compiled into nine weight matrices.
24
+
25
+ Thinking tokens drop by **37–55%** on every benchmark we ran. Accuracy also drops, by a little on GPQA and by a
26
+ meaningful amount on code and instruction following (see below). Use this model when shorter, cheaper
27
+ reasoning matters more than the last few points of accuracy.
28
+
29
+ > Status: experimental (edit strength α = 0.1). A milder edit is being evaluated.
30
+
31
+ ## What was changed
32
+
33
+ | | |
34
+ |---|---|
35
+ | Base weights | `atakans/Hy4-flash-NVFP4` (NVFP4 MLPs, FP8 Engram tables, BF16 elsewhere), derived from `tencent/Hy4-flash` |
36
+ | Edited tensors | `model.layers.{22,24,…,38}.mlp.shared_experts.down_proj.weight` (9 matrices) |
37
+ | Edit | `W ← W_bf16_original + α · n · d″ uᵀ / ‖u‖²`, α = 0.1, stored in **BF16** (excluded from NVFP4) |
38
+ | Relative size | ‖ΔW‖/‖W‖ = 0.56–1.09% per layer |
39
+ | Everything else | byte-identical to `atakans/Hy4-flash-NVFP4` |
40
+
41
+ * `d` is the residual-stream direction that separates the last 48 thinking tokens before `</think>` from the rest of
42
+ the thinking, measured on 288 MATH-train (level 5) traces. `d″` corrects `d` for the FFN post-norm
43
+ (`combine_output_norm`), so after that RMSNorm the written vector points along `d`.
44
+ * `u` is the mean input to the shared-expert down-projection on thinking tokens, and `n` is the mean norm of the
45
+ FFN output before the post-norm.
46
+ * The shared expert runs on every token, so a rank-one term in its down-projection acts like a constant steering
47
+ vector, compiled into the weights.
48
+ * Directions, layers and α were chosen on MATH train only. No evaluation benchmark was used for selection.
49
+
50
+ The direction tensors and scripts are in [`edit/`](edit/).
51
+
52
+ ## Results
53
+
54
+ Setup: vLLM, this checkpoint against the matched base (the same nine matrices in BF16, unedited), thinking on
55
+ (`reasoning_effort: high`), temperature 0.9, max 32k tokens, one H200. Δ is paired by problem, with a 95%
56
+ bootstrap CI.
57
+
58
+ | Benchmark | Base | **smart (α=0.1)** | Δ accuracy [95% CI] | Δ thinking tokens [95% CI] |
59
+ |---|---|---|---|---|
60
+ | GPQA Diamond (198 × 2) | 91.2 | 88.6 | −2.5 [−5.3, +0.3] | **−39.6%** [−43.3, −35.6] |
61
+ | LiveCodeBench v6 (175, 2025-01…04) | 84.6 | 78.3 | −6.3 [−12.0, −0.6] | **−36.8%** [−43.3, −30.3] |
62
+ | IFBench (300, loose) | 76.7 | 66.0 | −10.7 [−16.0, −5.3] | **−54.8%** [−58.3, −51.0] |
63
+ | 13 coding/math/science tasks × 3 (held-out) | 39/39 | 38/39 | −2.6 | **−54.2%** |
64
+ | MATH train screen (64 × 2) | 92.2 | 89.8 | −2.3 [−6.2, +0.8] | −40.2% [−47.8, −29.9] |
65
+
66
+ Mean thinking tokens, base → smart: GPQA 5448 → 3291, LCB 8564 → 5409, IFBench 5275 → 2382, held-out tasks 5415 → 2482.
67
+
68
+ **Control.** A norm-matched random edit (same layers, same ‖ΔW‖, random direction orthogonal to `d`) on the
69
+ MATH screen gives +0.0 points of accuracy and +8% tokens. The token saving comes from the direction, not from the
70
+ perturbation.
71
+
72
+ **Dose response** (MATH screen): α = 0.05 → −23% tokens; α = 0.075 → −32%; α = 0.1 → −40%; α = 0.3 → answers
73
+ become wrong; α = 1.0 → the model loops on `</think>`. Do not raise α.
74
+
75
+ ## Usage
76
+
77
+ ### vLLM (Hopper / Blackwell)
78
+
79
+ Hy4-flash needs the `vllm/vllm-openai:hy4-flash` image. NVFP4 expert loading needs the
80
+ `vllm/models/hy_v4_flash/nvidia/model.py` from
81
+ [atakann/dgx-spark-recipes](https://github.com/atakann/dgx-spark-recipes) (copy included as
82
+ `edit/vllm_hy_v4_flash_model.py`). On DGX Spark (SM120), use that recipe's full image.
83
+
84
+ ```bash
85
+ vllm serve autotrust/hy4-flash-smart --served-model-name hy4-flash \
86
+ --dtype bfloat16 --max-model-len 40960 --max-num-seqs 64 \
87
+ --reasoning-parser hy_v4 --limit-mm-per-prompt '{"image":0,"video":0}'
88
+ ```
89
+
90
+ On one H200: 97 GiB of weights on the GPU, Engram tables in host RAM, about 1.5k tok/s aggregate decode at 100
91
+ concurrent requests.
92
+
93
+ ### llama.cpp (GGUF, e.g. DGX Spark)
94
+
95
+ `gguf-lora/hy4-flash-smart-lora.gguf` (167 KB) applies the same rank-one edit to the stock
96
+ [AngelSlim/Hy4-flash-GGUF](https://huggingface.co/AngelSlim/Hy4-flash-GGUF) files as a LoRA adapter on
97
+ `blk.{22..38 even}.ffn_down_shexp`. The scale is α:
98
+
99
+ ```bash
100
+ llama-server -m Hy4-flash-SPARK-4Bit-Fast.gguf ... --lora-scaled hy4-flash-smart-lora.gguf:0.1
101
+ ```
102
+
103
+ The adapter was validated on Q4_K_M (MATH screen: −38% tokens, −2.3 points). It is **not** separately validated
104
+ on the SPARK-4Bit builds. Note that llama-server will not batch requests whose per-request LoRA settings differ,
105
+ so set the scale globally.
106
+
107
+ ## Limitations
108
+
109
+ * Accuracy loss on code (−6 points) and instruction following (−11 points) at α = 0.1 is statistically significant.
110
+ * The direction was read from a Q4_K_M GGUF of the same model. Q8_0 shared experts differ from BF16 by 0.6%.
111
+ * The edit was tested only with thinking on. With `no_think` it is untested.
112
+ * One edit form, one layer set, single runs per benchmark.
113
+
114
+ ## License and attribution
115
+
116
+ Tencent Hy License 1.0 (see `LICENSE`), inherited from tencent/Hy4-flash. Modified from
117
+ atakans/Hy4-flash-NVFP4: nine `shared_experts.down_proj` tensors replaced as described above, and the
118
+ `hf_quant_config.json` exclude list extended to cover them.
119
+
120
+ Method after *weight-space programming* (Takehi et al., 2026) and the norm-matched replication by Concept Country
121
+ (2026).
chat_template.jinja ADDED
@@ -0,0 +1,355 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {#- ----------‑‑‑ special token variables ‑‑‑---------- -#}
2
+ {%- set HYTK = ':opensource' %}
3
+ {%- set hy_start_token = '<|hy_start{}|>'.format(HYTK) %}
4
+ {%- set hy_middle_token = '<|hy_middle{}|>'.format(HYTK) %}
5
+ {%- set hy_end_token = '<|hy_end{}|>'.format(HYTK) %}
6
+ {%- set image_start_token = '<|hy_image▁start{}|>'.format(HYTK) %}
7
+ {%- set image_pad_token = '<|hy_image▁pad{}|>'.format(HYTK) %}
8
+ {%- set image_end_token = '<|hy_image▁end{}|>'.format(HYTK) %}
9
+ {%- set video_start_token = '<|hy_video▁start{}|>'.format(HYTK) %}
10
+ {%- set video_pad_token = '<|hy_video▁pad{}|>'.format(HYTK) %}
11
+ {%- set video_end_token = '<|hy_video▁end{}|>'.format(HYTK) %}
12
+ {%- set think_begin_token = '<think{}>'.format(HYTK) %}
13
+ {%- set think_end_token = '</think{}>'.format(HYTK) %}
14
+ {%- set toolcalls_begin_token = '<tool_calls{}>'.format(HYTK) %}
15
+ {%- set toolcalls_end_token = '</tool_calls{}>'.format(HYTK) %}
16
+ {%- set toolcall_begin_token = '<tool_call{}>'.format(HYTK) %}
17
+ {%- set toolcall_end_token = '</tool_call{}>'.format(HYTK) %}
18
+ {%- set argkey_begin_token = '<arg_key{}>'.format(HYTK) %}
19
+ {%- set argkey_end_token = '</arg_key{}>'.format(HYTK) %}
20
+ {%- set argvalue_begin_token = '<arg_value{}>'.format(HYTK) %}
21
+ {%- set argvalue_end_token = '</arg_value{}>'.format(HYTK) %}
22
+ {%- set toolresponse_begin_token = '<tool_response{}>'.format(HYTK) %}
23
+ {%- set toolresponse_end_token = '</tool_response{}>'.format(HYTK) %}
24
+ {%- set reasoning_mode_token = '<|reasoning_mode{}|>'.format(HYTK) %}
25
+
26
+ {#- ----------‑‑‑ hyperparameters variables ‑‑‑---------- -#}
27
+ {%- if not add_generation_prompt is defined %}
28
+ {%- set add_generation_prompt = false %}
29
+ {%- endif %}
30
+ {%- if not preserved_thinking is defined %}
31
+ {%- if not tools %}
32
+ {%- set preserved_thinking = false %}
33
+ {%- else %}
34
+ {%- set preserved_thinking = true %}
35
+ {%- endif %}
36
+ {%- endif %}
37
+
38
+ {%- if not reasoning_effort is defined %}
39
+ {%- set reasoning_effort = 'high' %}
40
+ {%- elif reasoning_effort not in ['max', 'high', 'no_think'] %}
41
+ {%- if reasoning_effort is none %}
42
+ {{- raise_exception('reasoning_effort error : None, should be no_think/high/max') }}
43
+ {%- else %}
44
+ {{- raise_exception('reasoning_effort error : ' + reasoning_effort + ', should be no_think/high/max') }}
45
+ {%- endif %}
46
+ {%- endif %}
47
+
48
+ {%- if fallback_strategy is defined and fallback_strategy == 'reasoning_toolcall_retry' %}
49
+ {%- set reasoning_effort = 'high' %}
50
+ {%- set add_generation_prompt = false %}
51
+ {%- endif %}
52
+ {%- if not raw_last_assistant is defined %}
53
+ {%- set raw_last_assistant = false %}
54
+ {%- endif %}
55
+ {%- if not add_vision_id is defined %}
56
+ {%- set add_vision_id = false %}
57
+ {%- endif %}
58
+ {%- set image_count = namespace(value=0) %}
59
+ {%- set video_count = namespace(value=0) %}
60
+
61
+ {%- macro tool_to_json(tool) -%}
62
+ {%- set ns_tool = namespace(first=true) -%}
63
+ {{- '{' -}}
64
+ {%- for k, v in tool.items() -%}
65
+ {%- if k != 'defer_loading' and k != 'strict' -%}
66
+ {%- if not ns_tool.first -%}{{- ', ' -}}{%- endif -%}
67
+ {%- set ns_tool.first = false -%}
68
+ {{- '"' ~ k ~ '": ' ~ (v | tojson(ensure_ascii=False)) -}}
69
+ {%- endif -%}
70
+ {%- endfor -%}
71
+ {{- '}' -}}
72
+ {%- endmacro -%}
73
+
74
+ {%- macro render_content(content, do_vision_count=true, is_system_content=false) -%}
75
+ {%- if content is string -%}
76
+ {{- content -}}
77
+ {%- elif content is iterable and content is not mapping -%}
78
+ {%- for item in content -%}
79
+ {%- if item is mapping and item.type == 'text' -%}
80
+ {{- item.text -}}
81
+ {%- elif item is mapping and ('image' in item or 'image_url' in item or item.type in ['image', 'image_url']) -%}
82
+ {%- if is_system_content -%}
83
+ {{- raise_exception('System message cannot contain images.') -}}
84
+ {%- endif -%}
85
+ {%- if do_vision_count -%}
86
+ {%- set image_count.value = image_count.value + 1 -%}
87
+ {%- endif -%}
88
+ {%- if add_vision_id -%}
89
+ {{- 'Picture ' ~ image_count.value ~ ': ' -}}
90
+ {%- endif -%}
91
+ {{- image_start_token ~ image_pad_token ~ image_end_token -}}
92
+ {%- elif item is mapping and ('video' in item or 'video_url' in item or item.type in ['video', 'video_url']) -%}
93
+ {%- if is_system_content -%}
94
+ {{- raise_exception('System message cannot contain videos.') -}}
95
+ {%- endif -%}
96
+ {%- if do_vision_count -%}
97
+ {%- set video_count.value = video_count.value + 1 -%}
98
+ {%- endif -%}
99
+ {%- if add_vision_id -%}
100
+ {{- 'Video ' ~ video_count.value ~ ': ' -}}
101
+ {%- endif -%}
102
+ {{- video_start_token ~ video_pad_token ~ video_end_token -}}
103
+ {%- elif item is string -%}
104
+ {{- item -}}
105
+ {%- else -%}
106
+ {{- item | string -}}
107
+ {%- endif -%}
108
+ {%- endfor -%}
109
+ {%- elif content is none -%}
110
+ {{- '' -}}
111
+ {%- else -%}
112
+ {{- content | string -}}
113
+ {%- endif -%}
114
+ {%- endmacro -%}
115
+
116
+ {%- macro render_tools_prompt() -%}
117
+ {{- '# Tools\n\nYou may call one or more functions to assist with the user query.' -}}
118
+ {{- '\n\nYou are provided with function signatures within <tools></tools> XML tags:' -}}
119
+ {{- '\n<tools>\n' -}}
120
+ {%- set tool_ns = namespace(first=true) -%}
121
+ {%- for tool in tools -%}
122
+ {%- set t = tool['function'] if tool is mapping and 'function' in tool else tool -%}
123
+ {%- if t.defer_loading is not defined or not t.defer_loading -%}
124
+ {%- if not tool_ns.first -%}{{- '\n' -}}{%- endif -%}
125
+ {%- set tool_ns.first = false -%}
126
+ {{- tool_to_json(t) -}}
127
+ {%- endif -%}
128
+ {%- endfor -%}
129
+ {{- '\n</tools>\n\n' -}}
130
+ {{- 'For function call returns, you should first print ' ~ toolcalls_begin_token -}}
131
+ {{- '\nFor each function call, you should return object like:\n' -}}
132
+ {{- toolcall_begin_token ~ '{function-name}' -}}
133
+ {{- argkey_begin_token ~ '{arg-key-1}' ~ argkey_end_token -}}
134
+ {{- argvalue_begin_token ~ '{arg-value-1}' ~ argvalue_end_token -}}
135
+ {{- argkey_begin_token ~ '{arg-key-2}' ~ argkey_end_token -}}
136
+ {{- argvalue_begin_token ~ '{arg-value-2}' ~ argvalue_end_token -}}
137
+ {{- '...' -}}
138
+ {{- toolcall_end_token -}}
139
+ {{- '\nAt the end of function call returns, you should print ' ~ toolcalls_end_token -}}
140
+ {%- endmacro -%}
141
+
142
+ {#- adjacent tool_search_output parts share one <tools> block, so a tool search result has the same shape as the tools prompt -#}
143
+ {%- macro render_tool_response(message) -%}
144
+ {%- set content = message['content'] -%}
145
+ {{- toolresponse_begin_token -}}
146
+ {%- if content is string -%}
147
+ {{- content -}}
148
+ {%- elif content is iterable and content is not mapping -%}
149
+ {%- set tools_ns = namespace(open=false) -%}
150
+ {%- for item in content -%}
151
+ {%- set is_search_output = item is mapping and item.type is defined and item.type == 'tool_search_output' -%}
152
+ {%- if is_search_output and not tools_ns.open -%}
153
+ {{- '<tools>\n' -}}
154
+ {%- set tools_ns.open = true -%}
155
+ {%- elif not is_search_output and tools_ns.open -%}
156
+ {{- '</tools>' -}}
157
+ {%- set tools_ns.open = false -%}
158
+ {%- endif -%}
159
+ {%- if is_search_output -%}
160
+ {%- if item.tools is defined and item.tools is iterable -%}
161
+ {%- for tool in item.tools -%}
162
+ {%- set t = tool['function'] if tool is mapping and 'function' in tool else tool -%}
163
+ {{- tool_to_json(t) ~ '\n' -}}
164
+ {%- endfor -%}
165
+ {%- endif -%}
166
+ {%- elif item is mapping and item.output is defined -%}
167
+ {{- render_content(item.output) -}}
168
+ {%- else -%}
169
+ {{- render_content([item]) -}}
170
+ {%- endif -%}
171
+ {%- endfor -%}
172
+ {%- if tools_ns.open -%}
173
+ {{- '</tools>' -}}
174
+ {%- endif -%}
175
+ {%- else -%}
176
+ {{- render_content(content) -}}
177
+ {%- endif -%}
178
+ {{- toolresponse_end_token -}}
179
+ {%- endmacro -%}
180
+
181
+ {#- consecutive tool messages are clustered into a single tool message whose content is the list of the original tool messages; the result is written to merged_ns.messages (a macro cannot return a list) -#}
182
+ {%- set merged_ns = namespace(messages=[]) %}
183
+ {%- macro merge_tool_responses(messages) -%}
184
+ {%- set merge_ns = namespace(result=[], cluster=[]) -%}
185
+ {%- for message in messages -%}
186
+ {%- if message['role'] == 'tool' -%}
187
+ {%- set merge_ns.cluster = merge_ns.cluster + [message] -%}
188
+ {%- else -%}
189
+ {%- if merge_ns.cluster -%}
190
+ {%- set merge_ns.result = merge_ns.result + [{'role': 'tool', 'content': merge_ns.cluster}] -%}
191
+ {%- set merge_ns.cluster = [] -%}
192
+ {%- endif -%}
193
+ {%- set merge_ns.result = merge_ns.result + [message] -%}
194
+ {%- endif -%}
195
+ {%- endfor -%}
196
+ {%- if merge_ns.cluster -%}
197
+ {%- set merge_ns.result = merge_ns.result + [{'role': 'tool', 'content': merge_ns.cluster}] -%}
198
+ {%- endif -%}
199
+ {%- set merged_ns.messages = merge_ns.result -%}
200
+ {%- endmacro -%}
201
+
202
+ {#- render one tool cluster, reordered to follow the tool_call order of the preceding assistant turn; any missing / ambiguous id falls back to the original tool message order for the whole cluster -#}
203
+ {%- macro render_tool_responses(tool_messages, tool_call_ids) -%}
204
+ {%- set order_ns = namespace(ordered=[], matched=true) -%}
205
+ {%- if tool_call_ids and tool_call_ids | length == tool_messages | length -%}
206
+ {%- for tool_call_id in tool_call_ids -%}
207
+ {%- set hit_ns = namespace(count=0, message=none) -%}
208
+ {%- for tool_message in tool_messages -%}
209
+ {%- set message_id = tool_message['tool_call_id'] if tool_message['tool_call_id'] is defined else tool_message['id'] -%}
210
+ {%- if message_id is defined and message_id == tool_call_id -%}
211
+ {%- set hit_ns.count = hit_ns.count + 1 -%}
212
+ {%- set hit_ns.message = tool_message -%}
213
+ {%- endif -%}
214
+ {%- endfor -%}
215
+ {%- if hit_ns.count == 1 -%}
216
+ {%- set order_ns.ordered = order_ns.ordered + [hit_ns.message] -%}
217
+ {%- else -%}
218
+ {%- set order_ns.matched = false -%}
219
+ {%- endif -%}
220
+ {%- endfor -%}
221
+ {%- else -%}
222
+ {%- set order_ns.matched = false -%}
223
+ {%- endif -%}
224
+ {%- for tool_message in (order_ns.ordered if order_ns.matched else tool_messages) -%}
225
+ {{- render_tool_response(tool_message) -}}
226
+ {%- endfor -%}
227
+ {%- endmacro -%}
228
+
229
+ {%- set _ = merge_tool_responses(messages) %}
230
+ {%- set normed_messages = merged_ns.messages %}
231
+
232
+ {%- set ns = namespace(last_user_index=-1, has_leading_system=false) %}
233
+ {%- if normed_messages and normed_messages[0].role == 'system' %}
234
+ {%- set ns.has_leading_system = true %}
235
+ {%- endif %}
236
+ {%- for message in normed_messages %}
237
+ {%- if message['role'] == 'user' %}
238
+ {%- set ns.last_user_index = loop.index0 %}
239
+ {%- endif %}
240
+ {%- endfor %}
241
+
242
+ {#- no leading system: synthesize one for tools and/or reasoning_mode -#}
243
+ {%- if not ns.has_leading_system %}
244
+ {{- hy_start_token ~ 'system' ~ hy_middle_token -}}
245
+ {%- if tools %}
246
+ {{- render_tools_prompt() -}}
247
+ {%- endif %}
248
+ {{- reasoning_mode_token ~ 'reasoning_effort:' ~ reasoning_effort -}}
249
+ {{- hy_end_token -}}
250
+ {%- endif %}
251
+
252
+ {%- set last_ns = namespace(last_is_assistant=false) %}
253
+ {%- set prev_ns = namespace(tool_call_ids=[]) %}
254
+ {%- for message in normed_messages %}
255
+ {%- if message['role'] == 'system' %}
256
+ {{- hy_start_token ~ 'system' ~ hy_middle_token -}}
257
+ {#- tools / reasoning_mode only attach to the leading system (messages[0]) -#}
258
+ {%- if loop.first %}
259
+ {%- if tools %}
260
+ {{- render_tools_prompt() -}}
261
+ {%- endif %}
262
+ {%- set content = render_content(message['content']) -%}
263
+ {%- if tools and content -%}
264
+ {{- '\n\n' -}}
265
+ {%- endif -%}
266
+ {{- content -}}
267
+ {{- reasoning_mode_token ~ 'reasoning_effort:' ~ reasoning_effort -}}
268
+ {%- else %}
269
+ {{- render_content(message['content']) -}}
270
+ {%- endif %}
271
+ {{- hy_end_token -}}
272
+ {%- elif message['role'] == 'user' %}
273
+ {{- hy_start_token ~ 'user' ~ hy_middle_token -}}
274
+ {{- render_content(message['content']) -}}
275
+ {{- hy_end_token -}}
276
+ {%- elif message['role'] == 'assistant' %}
277
+ {%- set content_body = render_content(message['content']) -%}
278
+ {#- 'reasoning' takes precedence, fall back to 'reasoning_content' when it is empty -#}
279
+ {%- set reasoning_text = '' %}
280
+ {%- if message['reasoning'] is defined and message['reasoning'] is string and message['reasoning'] %}
281
+ {%- set reasoning_text = message['reasoning'] %}
282
+ {%- elif message['reasoning_content'] is defined and message['reasoning_content'] is string and message['reasoning_content'] %}
283
+ {%- set reasoning_text = message['reasoning_content'] %}
284
+ {%- endif %}
285
+ {#- no_think overrides preserved_thinking: history is always emitted with empty think tags -#}
286
+ {%- if reasoning_effort != 'no_think' and (preserved_thinking or loop.index0 > ns.last_user_index) and reasoning_text %}
287
+ {%- set content = think_begin_token ~ reasoning_text ~ think_end_token ~ content_body %}
288
+ {%- else %}
289
+ {%- set content = think_begin_token ~ think_end_token ~ content_body %}
290
+ {%- endif %}
291
+ {{- hy_start_token ~ 'assistant' ~ hy_middle_token -}}
292
+ {%- if message['tool_calls'] is defined and message['tool_calls'] %}
293
+ {#- remember the tool_call ids of this turn to order the following tool responses; an incomplete id set means "do not reorder" -#}
294
+ {%- set ids_ns = namespace(ids=[], complete=true) %}
295
+ {%- for tool in message['tool_calls'] %}
296
+ {%- set tool_call_id = tool['id'] if tool['id'] is defined and tool['id'] else tool['tool_call_id'] %}
297
+ {%- if tool_call_id is defined and tool_call_id %}
298
+ {%- set ids_ns.ids = ids_ns.ids + [tool_call_id] %}
299
+ {%- else %}
300
+ {%- set ids_ns.complete = false %}
301
+ {%- endif %}
302
+ {%- endfor %}
303
+ {%- set prev_ns.tool_call_ids = ids_ns.ids if ids_ns.complete else [] %}
304
+ {{- content -}}
305
+ {{- toolcalls_begin_token -}}
306
+ {%- for tool in message['tool_calls'] -%}
307
+ {%- set func = tool['function'] if tool is mapping and 'function' in tool else tool -%}
308
+ {%- set arguments = func['arguments'] -%}
309
+ {{- toolcall_begin_token ~ func['name'] -}}
310
+ {%- for key, value in arguments.items() -%}
311
+ {{- argkey_begin_token ~ key ~ argkey_end_token -}}
312
+ {%- if value is not string -%}
313
+ {%- set value = value | tojson(ensure_ascii=False) -%}
314
+ {%- endif -%}
315
+ {{- argvalue_begin_token ~ value ~ argvalue_end_token -}}
316
+ {%- endfor -%}
317
+ {{- toolcall_end_token -}}
318
+ {%- endfor -%}
319
+ {{- toolcalls_end_token -}}
320
+ {%- else %}
321
+ {%- if loop.last and raw_last_assistant %}
322
+ {{- content_body -}}
323
+ {%- else %}
324
+ {{- content -}}
325
+ {%- endif %}
326
+ {%- endif %}
327
+ {#- continuation / prefill: last assistant is still open, do not close with hy_end -#}
328
+ {%- if not (loop.last and raw_last_assistant) and not (loop.last and fallback_strategy is defined and fallback_strategy == 'reasoning_toolcall_retry') %}
329
+ {{- hy_end_token -}}
330
+ {%- endif %}
331
+ {%- elif message['role'] == 'tool' %}
332
+ {{- hy_start_token ~ 'tool' ~ hy_middle_token -}}
333
+ {{- render_tool_responses(message['content'], prev_ns.tool_call_ids) -}}
334
+ {{- hy_end_token -}}
335
+ {%- else %}
336
+ {{- hy_start_token ~ message['role'] ~ hy_middle_token -}}
337
+ {{- render_content(message['content']) -}}
338
+ {{- hy_end_token -}}
339
+ {%- endif %}
340
+ {%- if message['role'] != 'assistant' or not (message['tool_calls'] is defined and message['tool_calls']) %}
341
+ {%- set prev_ns.tool_call_ids = [] %}
342
+ {%- endif %}
343
+ {%- if loop.last and message['role'] == 'assistant' %}
344
+ {%- set last_ns.last_is_assistant = true %}
345
+ {%- endif %}
346
+ {%- endfor %}
347
+ {%- if add_generation_prompt %}
348
+ {%- if not last_ns.last_is_assistant %}
349
+ {%- if reasoning_effort == 'no_think' %}
350
+ {{- hy_start_token ~ 'assistant' ~ hy_middle_token ~ think_begin_token ~ think_end_token -}}
351
+ {%- else %}
352
+ {{- hy_start_token ~ 'assistant' ~ hy_middle_token ~ think_begin_token -}}
353
+ {%- endif %}
354
+ {%- endif %}
355
+ {%- endif %}
config.json ADDED
@@ -0,0 +1,494 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "HYV4FlashForConditionalGeneration"
4
+ ],
5
+ "dtype": "bfloat16",
6
+ "image_end_token_id": 120119,
7
+ "image_newline_token_id": 120121,
8
+ "image_start_token_id": 120118,
9
+ "image_token_id": 120120,
10
+ "model_type": "hy_v4_flash",
11
+ "processor_class": "HYV4FlashProcessor",
12
+ "spatial_merge_size": 2,
13
+ "text_config": {
14
+ "architectures": [
15
+ "HYV4FlashForCausalLM"
16
+ ],
17
+ "attention_bias": false,
18
+ "attention_dropout": 0.0,
19
+ "bitwise_backward_align": true,
20
+ "bos_token_id": 120000,
21
+ "compressed_indexer_candidate_topk": 2048,
22
+ "compressed_indexer_compress_ratio": 4,
23
+ "compressed_indexer_group_topk": 512,
24
+ "compressed_indexer_head_dim": 128,
25
+ "compressed_indexer_key_pooling": "gated",
26
+ "compressed_indexer_num_attention_heads": 16,
27
+ "compressed_indexer_reuse_fine_indexer": false,
28
+ "compressed_indexer_skip_fine_select": true,
29
+ "compressed_indexer_stage2_query_block_size": 8192,
30
+ "compressed_indexer_teacher_pooling_type": "max",
31
+ "compressed_indexer_triton_pooling_min_groups": 1,
32
+ "compressed_indexer_use_triton_pooling": true,
33
+ "dsa_skip_attn_post_norm": false,
34
+ "enable_ihc": true,
35
+ "enable_lm_head_fp32": true,
36
+ "engram_embed_random_seed": 1234,
37
+ "engram_fp8_embedding": true,
38
+ "engram_isolate_fffd_namespace": false,
39
+ "engram_layer_head_vocab_sizes": {
40
+ "12": [
41
+ [
42
+ 10312277,
43
+ 10312283,
44
+ 10312301,
45
+ 10312343,
46
+ 10312373,
47
+ 10312397,
48
+ 10312409,
49
+ 10312453
50
+ ],
51
+ [
52
+ 10312457,
53
+ 10312483,
54
+ 10312507,
55
+ 10312513,
56
+ 10312531,
57
+ 10312543,
58
+ 10312567,
59
+ 10312571
60
+ ]
61
+ ],
62
+ "2": [
63
+ [
64
+ 10312007,
65
+ 10312021,
66
+ 10312039,
67
+ 10312069,
68
+ 10312091,
69
+ 10312121,
70
+ 10312123,
71
+ 10312147
72
+ ],
73
+ [
74
+ 10312177,
75
+ 10312187,
76
+ 10312207,
77
+ 10312213,
78
+ 10312229,
79
+ 10312241,
80
+ 10312249,
81
+ 10312259
82
+ ]
83
+ ],
84
+ "22": [
85
+ [
86
+ 10312613,
87
+ 10312619,
88
+ 10312633,
89
+ 10312657,
90
+ 10312681,
91
+ 10312699,
92
+ 10312723,
93
+ 10312747
94
+ ],
95
+ [
96
+ 10312759,
97
+ 10312789,
98
+ 10312811,
99
+ 10312823,
100
+ 10312867,
101
+ 10312873,
102
+ 10312889,
103
+ 10312903
104
+ ]
105
+ ],
106
+ "32": [
107
+ [
108
+ 10312909,
109
+ 10312919,
110
+ 10312927,
111
+ 10312957,
112
+ 10312979,
113
+ 10313011,
114
+ 10313021,
115
+ 10313057
116
+ ],
117
+ [
118
+ 10313063,
119
+ 10313117,
120
+ 10313123,
121
+ 10313129,
122
+ 10313161,
123
+ 10313179,
124
+ 10313189,
125
+ 10313201
126
+ ]
127
+ ]
128
+ },
129
+ "engram_layer_ids": [
130
+ 2,
131
+ 12,
132
+ 22,
133
+ 32
134
+ ],
135
+ "engram_max_ngram_size": 3,
136
+ "engram_n_embed_per_ngram": 768,
137
+ "engram_n_head_per_ngram": 8,
138
+ "engram_pad_id": 120002,
139
+ "engram_seed": 0,
140
+ "engram_tokenizer_name_or_path": null,
141
+ "engram_vocab_map_path": null,
142
+ "engram_vocab_size": [
143
+ 10312000,
144
+ 10312000
145
+ ],
146
+ "eos_token_id": 120025,
147
+ "fast_op_fp32_linear_gemm": true,
148
+ "first_k_dense_replace": 1,
149
+ "full_nope": false,
150
+ "gate_groups": 1,
151
+ "gate_lora_op": "none",
152
+ "gate_lora_rank": null,
153
+ "gated_mla": true,
154
+ "gating_type": "elementwise",
155
+ "hc_eps": 1e-06,
156
+ "hc_magnitude": 2.0,
157
+ "hc_mult": 4,
158
+ "head_dim": 64,
159
+ "hidden_act": "silu",
160
+ "hidden_size": 3072,
161
+ "ihc_kernel_backend": "naive",
162
+ "index_head_dim": 128,
163
+ "index_n_heads": 16,
164
+ "index_topk": 2048,
165
+ "indexer_types": [
166
+ "full",
167
+ "full",
168
+ "full",
169
+ "full",
170
+ "full",
171
+ "full",
172
+ "full",
173
+ "full",
174
+ "full",
175
+ "full",
176
+ "full",
177
+ "full",
178
+ "full",
179
+ "full",
180
+ "full",
181
+ "full",
182
+ "full",
183
+ "full",
184
+ "full",
185
+ "full",
186
+ "full",
187
+ "full",
188
+ "full",
189
+ "full",
190
+ "full",
191
+ "full",
192
+ "full",
193
+ "full",
194
+ "full",
195
+ "full",
196
+ "full",
197
+ "full",
198
+ "full",
199
+ "full",
200
+ "full",
201
+ "full",
202
+ "full",
203
+ "full",
204
+ "full",
205
+ "full",
206
+ "full",
207
+ "full"
208
+ ],
209
+ "initializer_range": 0.006,
210
+ "intermediate_size": 13824,
211
+ "kv_lora_rank": 512,
212
+ "layer_types": [
213
+ "deepseek_sparse_attention",
214
+ "deepseek_sparse_attention",
215
+ "sliding_attention",
216
+ "sliding_attention",
217
+ "sliding_attention",
218
+ "deepseek_sparse_attention",
219
+ "sliding_attention",
220
+ "sliding_attention",
221
+ "sliding_attention",
222
+ "deepseek_sparse_attention",
223
+ "sliding_attention",
224
+ "sliding_attention",
225
+ "sliding_attention",
226
+ "deepseek_sparse_attention",
227
+ "sliding_attention",
228
+ "sliding_attention",
229
+ "sliding_attention",
230
+ "deepseek_sparse_attention",
231
+ "sliding_attention",
232
+ "sliding_attention",
233
+ "sliding_attention",
234
+ "deepseek_sparse_attention",
235
+ "sliding_attention",
236
+ "sliding_attention",
237
+ "sliding_attention",
238
+ "deepseek_sparse_attention",
239
+ "sliding_attention",
240
+ "sliding_attention",
241
+ "sliding_attention",
242
+ "deepseek_sparse_attention",
243
+ "sliding_attention",
244
+ "sliding_attention",
245
+ "sliding_attention",
246
+ "deepseek_sparse_attention",
247
+ "sliding_attention",
248
+ "sliding_attention",
249
+ "sliding_attention",
250
+ "deepseek_sparse_attention",
251
+ "sliding_attention",
252
+ "sliding_attention",
253
+ "sliding_attention",
254
+ "deepseek_sparse_attention"
255
+ ],
256
+ "learnable_sink": true,
257
+ "learnable_sink_init": 0.0,
258
+ "max_position_embeddings": 1048576,
259
+ "mlp_layer_types": [
260
+ "dense",
261
+ "sparse",
262
+ "sparse",
263
+ "sparse",
264
+ "sparse",
265
+ "sparse",
266
+ "sparse",
267
+ "sparse",
268
+ "sparse",
269
+ "sparse",
270
+ "sparse",
271
+ "sparse",
272
+ "sparse",
273
+ "sparse",
274
+ "sparse",
275
+ "sparse",
276
+ "sparse",
277
+ "sparse",
278
+ "sparse",
279
+ "sparse",
280
+ "sparse",
281
+ "sparse",
282
+ "sparse",
283
+ "sparse",
284
+ "sparse",
285
+ "sparse",
286
+ "sparse",
287
+ "sparse",
288
+ "sparse",
289
+ "sparse",
290
+ "sparse",
291
+ "sparse",
292
+ "sparse",
293
+ "sparse",
294
+ "sparse",
295
+ "sparse",
296
+ "sparse",
297
+ "sparse",
298
+ "sparse",
299
+ "sparse",
300
+ "sparse",
301
+ "sparse"
302
+ ],
303
+ "model_type": "hy_v4_flash",
304
+ "moe_aux_loss_coeff": 0.0,
305
+ "moe_intermediate_size": 1536,
306
+ "moe_router_enable_expert_bias": true,
307
+ "moe_router_use_sigmoid": true,
308
+ "n_group": 1,
309
+ "n_routed_experts": 512,
310
+ "n_shared_experts": 1,
311
+ "norm_topk_prob": true,
312
+ "num_attention_heads": 32,
313
+ "num_experts_per_tok": 16,
314
+ "num_hidden_layers": 42,
315
+ "num_key_value_heads": 32,
316
+ "num_nextn_predict_layers": 0,
317
+ "o_groups": 1,
318
+ "o_lora_op": "none",
319
+ "o_lora_rank": null,
320
+ "output_router_logits": false,
321
+ "pad_token_id": 120002,
322
+ "q_lora_rank": 1024,
323
+ "qk_head_dim": 256,
324
+ "qk_nope_head_dim": 192,
325
+ "qk_norm": true,
326
+ "qk_rope_head_dim": 64,
327
+ "return_bos_score": true,
328
+ "return_max_logit": true,
329
+ "rms_norm_eps": 1e-05,
330
+ "rope_parameters": {
331
+ "sliding_attention": {
332
+ "mrope_interleaved": true,
333
+ "mrope_section": [
334
+ 12,
335
+ 10,
336
+ 10
337
+ ],
338
+ "rope_theta": 10000,
339
+ "rope_type": "default"
340
+ },
341
+ "deepseek_sparse_attention": {
342
+ "mrope_interleaved": true,
343
+ "mrope_section": [
344
+ 12,
345
+ 10,
346
+ 10
347
+ ],
348
+ "rope_theta": 10000000,
349
+ "rope_type": "default"
350
+ }
351
+ },
352
+ "routed_expert_hidden_size": 1536,
353
+ "routed_scaling_factor": 3.998,
354
+ "sandwich_norm": true,
355
+ "sliding_window": 512,
356
+ "swa_num_attention_heads": 64,
357
+ "swa_pattern": [
358
+ 0,
359
+ 0,
360
+ 1,
361
+ 1,
362
+ 1,
363
+ 0,
364
+ 1,
365
+ 1,
366
+ 1,
367
+ 0,
368
+ 1,
369
+ 1,
370
+ 1,
371
+ 0,
372
+ 1,
373
+ 1,
374
+ 1,
375
+ 0,
376
+ 1,
377
+ 1,
378
+ 1,
379
+ 0,
380
+ 1,
381
+ 1,
382
+ 1,
383
+ 0,
384
+ 1,
385
+ 1,
386
+ 1,
387
+ 0,
388
+ 1,
389
+ 1,
390
+ 1,
391
+ 0,
392
+ 1,
393
+ 1,
394
+ 1,
395
+ 0,
396
+ 1,
397
+ 1,
398
+ 1,
399
+ 0
400
+ ],
401
+ "swa_qk_head_dim": 256,
402
+ "swa_qk_nope_head_dim": 192,
403
+ "swa_qk_rope_head_dim": 64,
404
+ "swa_v_head_dim": 256,
405
+ "swiglu_limit": 10.0,
406
+ "tie_word_embeddings": false,
407
+ "topk_group": 1,
408
+ "use_cache": true,
409
+ "use_compressed_indexer": true,
410
+ "use_dsa": true,
411
+ "use_grouped_mm": false,
412
+ "use_mla": true,
413
+ "v_head_dim": 256,
414
+ "vocab_size": 120832,
415
+ "window_size": [
416
+ 511,
417
+ 0
418
+ ]
419
+ },
420
+ "transformers_version": "5.23.0",
421
+ "video_end_token_id": 120123,
422
+ "video_start_token_id": 120122,
423
+ "video_token_id": 120124,
424
+ "vision_config": {
425
+ "add_patchemb_bias": true,
426
+ "anyres_vit_max_image_size": 2048,
427
+ "attention_backend": "hf_align_sdpa",
428
+ "attention_dropout": 0.0,
429
+ "bias_activation_fusion": true,
430
+ "cat_extra_token": 0,
431
+ "hidden_act": "gelu",
432
+ "hidden_dropout": 0.0,
433
+ "hidden_size": 1152,
434
+ "img_max_token_num": 4096,
435
+ "intermediate_size": 4304,
436
+ "interpolate_mode": "bilinear",
437
+ "learnable_mlp_pooling_size": 0,
438
+ "max_image_size": 2048,
439
+ "max_vit_seq_len": 16384,
440
+ "min_image_size": 512,
441
+ "model_type": "hy_v4_flash",
442
+ "num_attention_heads": 16,
443
+ "num_channels": 3,
444
+ "num_hidden_layers": 27,
445
+ "num_key_value_heads": 16,
446
+ "out_hidden_size": 3072,
447
+ "packed_seq_alignment": 128,
448
+ "patch_size": 16,
449
+ "perceive_post_norm": false,
450
+ "perceive_pre_norm": true,
451
+ "projector_type": "rmsnorm_mlp_adapter",
452
+ "remove_prenorm": true,
453
+ "resize_resolution": 2048,
454
+ "rms_norm_eps": 1e-05,
455
+ "rope_2d_percent": 1.0,
456
+ "rope_2d_theta": 10000.0,
457
+ "spatial_merge_size": 2,
458
+ "spatial_patch_size": 1,
459
+ "temporal_patch_size": 1,
460
+ "text_hidden_size": 3072,
461
+ "video_max_image_size": 768,
462
+ "video_min_image_size": 256,
463
+ "video_spatial_compression": false,
464
+ "video_temporal_compression": false,
465
+ "vision_full_attention": false
466
+ },
467
+ "vision_start_token_id": 120118,
468
+ "speculative_config": {
469
+ "method": "dfly",
470
+ "model": "dfly",
471
+ "num_speculative_tokens": 5,
472
+ "draft_sample_method": "probabilistic"
473
+ },
474
+ "dfly_mtp_fusion": "slot0",
475
+ "dfly_target_index_topk": 2048,
476
+ "dfly_context_local_window": null,
477
+ "dfly_enable_hidden_correction": true,
478
+ "dfly_num_hidden_layers": 5,
479
+ "dfly_mask_token_id": 120009,
480
+ "dfly_target_index_layer": [
481
+ 25,
482
+ 29,
483
+ 33,
484
+ 37,
485
+ 41
486
+ ],
487
+ "dfly_target_kv_layers": [
488
+ 25,
489
+ 29,
490
+ 33,
491
+ 37,
492
+ 41
493
+ ]
494
+ }
edit/build_arm.py ADDED
@@ -0,0 +1,96 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Build a vLLM checkpoint arm from the NVFP4 base by rewriting shared-expert down_proj of selected layers.
2
+
3
+ Edited modules are stored in BF16 = W_bf16_original + alpha * n_pre * d'' u^T / ||u||^2 (excluded from NVFP4).
4
+ All other files are symlinks to the NVFP4 base. alpha=0 gives the matched "base" arm (same layers in BF16).
5
+
6
+ usage: python build_arm.py dirs.npz {cc|sl} "22,24,...,38" ALPHA OUT_DIR [--control SEED]
7
+ """
8
+ import argparse
9
+ import json
10
+ import os
11
+
12
+ import numpy as np
13
+ import torch
14
+ from safetensors import safe_open
15
+ from safetensors.torch import save_file
16
+
17
+ BASE = "/root/hy4/nvfp4/atakans"
18
+ BF16 = "/root/hy4/bf16_partial"
19
+ BF16_INDEX = "/root/hy4/bf16_index.json"
20
+
21
+ ap = argparse.ArgumentParser()
22
+ ap.add_argument("dirs")
23
+ ap.add_argument("kind")
24
+ ap.add_argument("layers")
25
+ ap.add_argument("alpha", type=float)
26
+ ap.add_argument("out")
27
+ ap.add_argument("--control", type=int, default=None)
28
+ ap.add_argument("--strip", default="/root/hy4/nvfp4/stripA")
29
+ a = ap.parse_args()
30
+
31
+ D = np.load(a.dirs)
32
+ d_all, u_all, n_pre = D["d_" + a.kind], D["u"], D["n_pre"]
33
+ layers = [int(x) for x in a.layers.split(",")]
34
+ bidx = json.load(open(BF16_INDEX))["weight_map"]
35
+
36
+
37
+ def bf16(k):
38
+ with safe_open(os.path.join(BF16, bidx[k]), "pt") as f:
39
+ return f.get_tensor(k)
40
+
41
+
42
+ strip = json.load(open(os.path.join(a.strip, "strip.json")))
43
+ assert sorted(strip["layers"]) == sorted(layers), "stripped shards were made for a different layer set"
44
+ os.makedirs(a.out, exist_ok=True)
45
+ for new in strip["mapping"].values():
46
+ dst = os.path.join(a.out, new)
47
+ if not os.path.lexists(dst):
48
+ os.symlink(os.path.join(a.strip, new), dst)
49
+ for fn in os.listdir(BASE):
50
+ if fn in ("model.safetensors.index.json", "hf_quant_config.json") or fn.startswith(".") or fn in strip["mapping"]:
51
+ continue
52
+ dst = os.path.join(a.out, fn)
53
+ if not os.path.lexists(dst):
54
+ os.symlink(os.path.join(BASE, fn), dst)
55
+
56
+ tensors, info = {}, []
57
+ for L in layers:
58
+ key = f"model.layers.{L}.mlp.shared_experts.down_proj.weight"
59
+ W = bf16(key).double() # (3072, 1536)
60
+ g = bf16(f"model.layers.{L}.mlp.combine_output_norm.weight").double()
61
+ d = torch.from_numpy(d_all[L].astype(np.float64))
62
+ d /= d.norm()
63
+ if a.control is not None:
64
+ rng = np.random.default_rng(a.control * 1000 + L)
65
+ r = torch.from_numpy(rng.standard_normal(d.shape[0]))
66
+ r -= (r @ d) * d
67
+ d = r / r.norm()
68
+ dd = d / g
69
+ dd /= dd.norm()
70
+ u = torch.from_numpy(u_all[L].astype(np.float64))
71
+ dW = a.alpha * float(n_pre[L]) * torch.outer(dd, u) / (u @ u)
72
+ Wn = (W + dW).to(torch.bfloat16)
73
+ tensors[key] = Wn.contiguous()
74
+ info.append(dict(layer=L, rel_dW=float(dW.norm() / W.norm()),
75
+ realized_rel=float((Wn.double() - W).norm() / W.norm())))
76
+ save_file(tensors, os.path.join(a.out, "model-wsp-edit.safetensors"), metadata={"format": "pt"})
77
+
78
+ idx = json.load(open(os.path.join(BASE, "model.safetensors.index.json")))
79
+ wm = idx["weight_map"]
80
+ for k, v in list(wm.items()):
81
+ if v in strip["mapping"]:
82
+ wm[k] = strip["mapping"][v]
83
+ for L in layers:
84
+ pre = f"model.layers.{L}.mlp.shared_experts.down_proj."
85
+ for k in [k for k in wm if k.startswith(pre)]:
86
+ del wm[k]
87
+ wm[pre + "weight"] = "model-wsp-edit.safetensors"
88
+ json.dump(idx, open(os.path.join(a.out, "model.safetensors.index.json"), "w"), indent=1)
89
+
90
+ q = json.load(open(os.path.join(BASE, "hf_quant_config.json")))
91
+ q["quantization"]["exclude_modules"] += [f"model.layers.{L}.mlp.shared_experts.down_proj" for L in layers]
92
+ json.dump(q, open(os.path.join(a.out, "hf_quant_config.json"), "w"), indent=4)
93
+ json.dump(dict(kind=a.kind, layers=layers, alpha=a.alpha, control=a.control, info=info),
94
+ open(os.path.join(a.out, "wsp_edit.json"), "w"), indent=1)
95
+ print(a.out, "rel ||dW||/||W||:", [round(i["rel_dW"], 4) for i in info],
96
+ "realized after bf16:", [round(i["realized_rel"], 4) for i in info])
edit/compute_dirs.py ADDED
@@ -0,0 +1,79 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Compute candidate steering directions from wsp-capture output.
2
+
3
+ usage: python compute_dirs.py cap_prefix out.npz
4
+ """
5
+ import json
6
+ import sys
7
+
8
+ import numpy as np
9
+
10
+ pre, out = sys.argv[1], sys.argv[2]
11
+ meta = json.load(open(pre + ".out.json"))
12
+ idx = json.load(open(pre + ".index.json"))
13
+ NL, Lyr, E, HC, F = meta["NL"], meta["n_layer"], meta["n_embd"], meta["hc"], meta["n_ff_sh"]
14
+ T = meta["n_traces"]
15
+ assert T == len(idx)
16
+
17
+ per = np.fromfile(pre + ".out", dtype=np.float32).reshape(T, -1)
18
+ cnt = per[:, :NL]
19
+ lout = per[:, NL:].reshape(T, NL, Lyr, E)
20
+
21
+ g = np.fromfile(pre + ".out.global", dtype=np.float32)
22
+ o = 0
23
+ def take(n, shape):
24
+ global o
25
+ v = g[o:o + n].reshape(shape)
26
+ o += n
27
+ return v
28
+ g_cnt = take(NL, (NL,))
29
+ g_lout = take(NL * Lyr * HC * E, (NL, Lyr, HC, E))
30
+ g_u = take(Lyr * F, (Lyr, F))
31
+ g_un2 = take(Lyr, (Lyr,))
32
+ g_shn = take(Lyr, (Lyr,))
33
+ g_pren = take(Lyr, (Lyr,))
34
+ g_cnt0 = take(Lyr, (Lyr,))
35
+ assert o == g.size
36
+
37
+ def unit(v):
38
+ return v / (np.linalg.norm(v, axis=-1, keepdims=True) + 1e-12)
39
+
40
+ # --- cc: pre-end-of-thinking region minus other think tokens (stream mean) ---
41
+ m1 = lout[:, 1].sum(0) / cnt[:, 1].sum() # (Lyr, E)
42
+ m0 = lout[:, 0].sum(0) / cnt[:, 0].sum()
43
+ d_cc = m1 - m0
44
+
45
+ # per-stream version, to check that the 4 HC streams agree
46
+ ms1 = g_lout[1] / g_cnt[1]
47
+ ms0 = g_lout[0] / g_cnt[0]
48
+ ds = ms1 - ms0 # (Lyr, HC, E)
49
+ stream_cos = np.array([[float(unit(ds[l, a]) @ unit(ds[l, b])) for a in range(HC) for b in range(a + 1, HC)] for l in range(Lyr)])
50
+
51
+ # --- sl: short-minus-long over correct traces on the same problem ---
52
+ think_mean = (lout[:, 0] + lout[:, 1]) / (cnt[:, 0] + cnt[:, 1])[:, None, None]
53
+ by_pid = {}
54
+ for i, r in enumerate(idx):
55
+ if r["correct"]:
56
+ by_pid.setdefault(r["pid"], []).append(i)
57
+ pairs = []
58
+ for pid, ii in by_pid.items():
59
+ if len(ii) < 2:
60
+ continue
61
+ ii = sorted(ii, key=lambda i: idx[i]["n_think"])
62
+ s, l = ii[0], ii[-1]
63
+ if idx[l]["n_think"] >= 1.3 * idx[s]["n_think"]:
64
+ pairs.append((s, l))
65
+ d_sl = np.mean([think_mean[s] - think_mean[l] for s, l in pairs], axis=0) if pairs else np.zeros((Lyr, E))
66
+
67
+ u = g_u / np.maximum(g_cnt0, 1)[:, None]
68
+ n_pre = g_pren / np.maximum(g_cnt0, 1)
69
+ n_sh = g_shn / np.maximum(g_cnt0, 1)
70
+ resid_norm = np.linalg.norm(m0, axis=-1)
71
+
72
+ np.savez(out, d_cc=d_cc, d_sl=d_sl, u=u, n_pre=n_pre, n_sh=n_sh, stream_cos=stream_cos, m0=m0, m1=m1,
73
+ n_pairs=len(pairs))
74
+ print(f"traces={T} tokens: think={cnt[:,0].sum():.0f} pre_end={cnt[:,1].sum():.0f} answer={cnt[:,2].sum():.0f} sl pairs={len(pairs)}")
75
+ print("layer |d_cc|/|m0| |d_sl|/|m0| cos(cc,sl) stream_cos(min/mean) n_pre n_sh |u|")
76
+ for l in range(Lyr):
77
+ c = float(unit(d_cc[l]) @ unit(d_sl[l])) if pairs else float("nan")
78
+ print(f"{l:4d} {np.linalg.norm(d_cc[l])/resid_norm[l]:10.3f} {np.linalg.norm(d_sl[l])/resid_norm[l]:10.3f} {c:9.3f}"
79
+ f" {stream_cos[l].min():6.3f}/{stream_cos[l].mean():6.3f} {n_pre[l]:7.2f} {n_sh[l]:6.2f} {np.linalg.norm(u[l]):6.3f}")
edit/directions.npz ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f967e798990d81e649763b631e76ca879244df08e84bc7741c23d1ab15774fad
3
+ size 2326980
edit/make_lora.py ADDED
@@ -0,0 +1,71 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Compile a direction into a rank-1 LoRA adapter on blk.L.ffn_down_shexp (additive WSP edit).
2
+
3
+ Delta W_L = s * n_pre[L] * d''_L u_L^T / ||u_L||^2, with d'' = unit(d_L / g_L) so that after the
4
+ FFN post-RMSNorm (weight g_L) the written vector points along d_L. The request-time LoRA scale s
5
+ plays the role of alpha. The control replaces d_L with a random unit r_L orthogonal to d_L
6
+ (per-layer seed); since ||B|| and A are identical, ||Delta W|| is identical.
7
+
8
+ usage: python make_lora.py dirs.npz {cc|sl} "20,22,...,36" out.gguf [--control SEED]
9
+ """
10
+ import argparse
11
+ import sys
12
+
13
+ import numpy as np
14
+
15
+ sys.path.insert(0, "/root/hy4/llama.cpp-hy4/gguf-py")
16
+ import gguf # noqa: E402
17
+
18
+ BASE = "/root/hy4/gguf/Hy4-flash-Q4_K_M-00001-of-00002.gguf"
19
+
20
+ ap = argparse.ArgumentParser()
21
+ ap.add_argument("dirs")
22
+ ap.add_argument("kind")
23
+ ap.add_argument("layers")
24
+ ap.add_argument("out")
25
+ ap.add_argument("--control", type=int, default=None)
26
+ a = ap.parse_args()
27
+
28
+ D = np.load(a.dirs)
29
+ d_all = D["d_" + a.kind]
30
+ u_all, n_pre = D["u"], D["n_pre"]
31
+ layers = [int(x) for x in a.layers.split(",")]
32
+
33
+ # post-norm weights from the base model (small F32 tensors)
34
+ reader = gguf.GGUFReader(BASE)
35
+ post = {}
36
+ Wn = {}
37
+ for t in reader.tensors:
38
+ for L in layers:
39
+ if t.name == f"blk.{L}.post_ffw_norm.weight":
40
+ post[L] = np.array(t.data, dtype=np.float32).reshape(-1)
41
+ if t.name == f"blk.{L}.ffn_down_shexp.weight":
42
+ Wn[L] = float(np.linalg.norm(gguf.quants.dequantize(t.data, t.tensor_type)))
43
+ assert len(post) == len(layers), f"missing post-norm weights: {set(layers) - set(post)}"
44
+
45
+ w = gguf.GGUFWriter(a.out, arch="hy_v4_flash")
46
+ w.add_type(gguf.GGUFType.ADAPTER)
47
+ w.add_string("adapter.type", "lora")
48
+ w.add_float32("adapter.lora.alpha", 1.0)
49
+ info = []
50
+ for L in layers:
51
+ d = d_all[L].astype(np.float64)
52
+ d /= np.linalg.norm(d)
53
+ if a.control is not None:
54
+ rng = np.random.default_rng(a.control * 1000 + L)
55
+ r = rng.standard_normal(d.shape)
56
+ r -= (r @ d) * d
57
+ d = r / np.linalg.norm(r)
58
+ g = post[L].astype(np.float64)
59
+ dd = d / g
60
+ dd /= np.linalg.norm(dd)
61
+ u = u_all[L].astype(np.float64)
62
+ A = (u / (u @ u))[None, :] # (1, 1536) -> ggml ne {1536, 1}
63
+ B = (n_pre[L] * dd)[:, None] # (3072, 1) -> ggml ne {1, 3072}
64
+ w.add_tensor(f"blk.{L}.ffn_down_shexp.weight.lora_a", A.astype(np.float32))
65
+ w.add_tensor(f"blk.{L}.ffn_down_shexp.weight.lora_b", B.astype(np.float32))
66
+ info.append((L, float(np.linalg.norm(B) * np.linalg.norm(A)) / Wn[L]))
67
+ w.write_header_to_file()
68
+ w.write_kv_data_to_file()
69
+ w.write_tensors_to_file()
70
+ w.close()
71
+ print("wrote", a.out, "||dW||/||W|| per layer at scale 1:", [(L, round(x, 3)) for L, x in info])
edit/strip_shards.py ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Write copies of NVFP4 shards with the shared-expert down_proj tensors of the given layers removed.
2
+
3
+ usage: python strip_shards.py "22,24,...,38" OUT_DIR
4
+ Produces OUT_DIR/<shard>.stripped-<tag>.safetensors and OUT_DIR/strip.json (old->new shard names).
5
+ """
6
+ import json
7
+ import os
8
+ import sys
9
+
10
+ from safetensors import safe_open
11
+ from safetensors.torch import save_file
12
+
13
+ BASE = "/root/hy4/nvfp4/atakans"
14
+ layers = [int(x) for x in sys.argv[1].split(",")]
15
+ out = sys.argv[2]
16
+ os.makedirs(out, exist_ok=True)
17
+ wm = json.load(open(os.path.join(BASE, "model.safetensors.index.json")))["weight_map"]
18
+ drop = {k for k in wm for L in layers if k.startswith(f"model.layers.{L}.mlp.shared_experts.down_proj.")}
19
+ shards = sorted({wm[k] for k in drop})
20
+ tag = "L" + "-".join(map(str, layers))
21
+ mapping = {}
22
+ for sh in shards:
23
+ new = sh.replace(".safetensors", f".strip{abs(hash(tag)) % 10**8}.safetensors")
24
+ keep = {}
25
+ with safe_open(os.path.join(BASE, sh), "pt") as f:
26
+ meta = f.metadata()
27
+ for k in f.keys():
28
+ if k not in drop:
29
+ keep[k] = f.get_tensor(k)
30
+ save_file(keep, os.path.join(out, new), metadata=meta)
31
+ mapping[sh] = new
32
+ print(sh, "->", new, len(keep), "tensors kept", flush=True)
33
+ json.dump(dict(layers=layers, mapping=mapping, dropped=sorted(drop)), open(os.path.join(out, "strip.json"), "w"), indent=1)
edit/vllm_hy_v4_flash_model.py ADDED
@@ -0,0 +1,494 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # SPDX-License-Identifier: Apache-2.0
2
+ # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
3
+ """HYV4 Flash backbone using the public HYV4 checkpoint contract."""
4
+
5
+ from collections.abc import Iterable
6
+ from contextlib import contextmanager
7
+ from typing import Any
8
+
9
+ import regex as re
10
+ import torch
11
+ from torch import nn
12
+
13
+ from vllm.config import VllmConfig
14
+ from vllm.model_executor.layers.layernorm import RMSNorm
15
+ from vllm.model_executor.model_loader.weight_utils import default_weight_loader
16
+ from vllm.model_executor.models.utils import AutoWeightsLoader, WeightsMapper
17
+ from vllm.models.hy_v4.nvidia.hc import HYV4HCLayer
18
+ from vllm.models.hy_v4.nvidia.model import (
19
+ HYV4DecoderLayer,
20
+ HYV4ForCausalLM,
21
+ HYV4Model,
22
+ )
23
+ from vllm.models.hy_v4_flash.configs import HYV4FlashConfig
24
+ from vllm.platforms import current_platform
25
+ from vllm.sequence import IntermediateTensors
26
+
27
+ from .moe import HYV4FlashFeedForward, HYV4FlashMoE
28
+
29
+ _EXPERT_WEIGHT = re.compile(
30
+ r"(.*\.experts)\.(?:(\d+)\.)?(gate_proj|up_proj|down_proj|gate_up_proj)"
31
+ r"(\.weight(?:_scale(?:_inv|_2)?)?|\.input_scale|_scale(?:_inv)?)?"
32
+ )
33
+
34
+
35
+ def validate_runtime(vllm_config: VllmConfig) -> None:
36
+ config = vllm_config.model_config.hf_text_config
37
+ config.validate_flash_config()
38
+ parallel = vllm_config.parallel_config
39
+ for field in (
40
+ "data_parallel_size",
41
+ "pipeline_parallel_size",
42
+ "prefill_context_parallel_size",
43
+ "decode_context_parallel_size",
44
+ ):
45
+ if getattr(parallel, field) != 1:
46
+ raise ValueError(f"HYV4 Flash requires {field}=1; only TP+EP is supported")
47
+ if parallel.enable_eplb or parallel.enable_dbo:
48
+ raise ValueError("HYV4 Flash does not support EPLB or dual batch overlap")
49
+ if not current_platform.is_cuda():
50
+ raise NotImplementedError("HYV4 Flash requires NVIDIA CUDA")
51
+ if vllm_config.model_config.dtype != torch.bfloat16:
52
+ raise ValueError("HYV4 Flash requires bfloat16 compute")
53
+ if (
54
+ vllm_config.kv_transfer_config is not None
55
+ or vllm_config.lora_config is not None
56
+ ):
57
+ raise ValueError("HYV4 Flash does not support KV transfer or LoRA")
58
+ spec = vllm_config.speculative_config
59
+ if spec is not None and spec.method != "dfly":
60
+ raise ValueError("HYV4 Flash supports only DFly speculative decoding")
61
+ if vllm_config.cache_config.cache_dtype not in ("auto", "bfloat16", "fp8_ds_mla"):
62
+ raise ValueError("HYV4 Flash requires BF16 or fp8_ds_mla KV storage")
63
+ if config.index_topk % 128 or config.index_topk < config.sliding_window:
64
+ raise ValueError("index_topk must be 128-aligned and cover sliding_window")
65
+ if config.engram_layer_ids and not config.enable_ihc:
66
+ raise ValueError("Flash Engram requires the iHC residual stream")
67
+ for field in ("num_attention_heads", "swa_num_attention_heads"):
68
+ if getattr(config, field) % parallel.tensor_parallel_size:
69
+ raise ValueError(f"{field} must be divisible by tensor_parallel_size")
70
+ quantization = vllm_config.model_config.quantization
71
+ if quantization == "compressed-tensors":
72
+ from vllm.models.hy_v4.dfly_config import _validate_block_fp8_config
73
+
74
+ _validate_block_fp8_config(
75
+ vllm_config.model_config.model_arch_config.quantization_config
76
+ )
77
+ elif quantization == "fp8":
78
+ fp8_quant = vllm_config.quant_config
79
+ if list(getattr(fp8_quant, "weight_block_size", None) or []) != [128, 128]:
80
+ raise ValueError("HYV4 Flash requires FP8 128x128 weight blocks")
81
+ elif quantization == "online":
82
+ from vllm.config.quantization import QuantizationConfigArgs, QuantSpec
83
+ from vllm.model_executor.layers.quantization.utils.quant_utils import (
84
+ kFp8Static128BlockSym,
85
+ )
86
+
87
+ quant = vllm_config.model_config.quantization_config
88
+ if isinstance(quant, dict):
89
+ quant = QuantizationConfigArgs(**quant)
90
+ if quant != QuantizationConfigArgs(moe=QuantSpec(weight=kFp8Static128BlockSym)):
91
+ raise ValueError(
92
+ "Flash online quantization requires MoE-only fp8_per_block"
93
+ )
94
+ elif quantization == "modelopt_fp4":
95
+ # ModelOpt NVFP4: per-expert weight, block scale, global scale and input scale
96
+ pass
97
+ elif quantization is not None:
98
+ raise ValueError(
99
+ "HYV4 Flash supports only FP8 blockwise and ModelOpt NVFP4 quantization"
100
+ )
101
+
102
+
103
+ class HYV4FlashDecoderLayer(HYV4DecoderLayer):
104
+ def __init__(
105
+ self,
106
+ config: HYV4FlashConfig,
107
+ vllm_config: VllmConfig,
108
+ cache_config=None,
109
+ quant_config=None,
110
+ prefix: str = "",
111
+ topk_indices_buffer: torch.Tensor | None = None,
112
+ ) -> None:
113
+ from .attention import HYV4FlashAttention
114
+
115
+ nn.Module.__init__(self)
116
+ self.config = config
117
+ self.hidden_size = config.hidden_size
118
+ self.layer_idx = int(prefix.rsplit(".", 1)[1])
119
+ self.enable_ihc = config.enable_ihc
120
+ self.self_attn = HYV4FlashAttention(
121
+ vllm_config,
122
+ config,
123
+ self.layer_idx,
124
+ topk_indices_buffer,
125
+ f"{prefix}.self_attn",
126
+ )
127
+ self.input_layernorm = RMSNorm(config.hidden_size, config.rms_norm_eps)
128
+ self.post_attention_layernorm = RMSNorm(config.hidden_size, config.rms_norm_eps)
129
+ dense = (config.mlp_layer_types or ())[self.layer_idx] == "dense"
130
+ mlp_cls = HYV4FlashFeedForward if dense else HYV4FlashMoE
131
+ self.mlp = mlp_cls(config, quant_config, f"{prefix}.mlp")
132
+ self.block_type = "feedforward" if dense else "moe"
133
+ self.hc_attn_layer = HYV4HCLayer(
134
+ config, self.layer_idx, prefix=f"{prefix}.hc_attn_layer"
135
+ )
136
+ self.hc_mlp_layer = HYV4HCLayer(
137
+ config, self.layer_idx, prefix=f"{prefix}.hc_mlp_layer"
138
+ )
139
+
140
+
141
+ class HYV4FlashModel(HYV4Model):
142
+ decoder_layer_cls = HYV4FlashDecoderLayer
143
+
144
+ def __init__(self, *, vllm_config: VllmConfig, prefix: str = "") -> None:
145
+ super().__init__(vllm_config=vllm_config, prefix=prefix)
146
+ self.loaded_parts: dict[str, set[Any]] = {}
147
+ self.engram_hash = None
148
+ if self.config.engram_layer_ids:
149
+ from .engram import FlashEngram, FlashNgramHashState
150
+
151
+ self.engram_hash = FlashNgramHashState(
152
+ self.config, dummy=vllm_config.load_config.load_format == "dummy"
153
+ )
154
+ for layer_id in self.config.engram_layer_ids:
155
+ self.layers[layer_id - 1].engram_layer = FlashEngram(
156
+ vllm_config,
157
+ self.engram_hash.layout,
158
+ layer_id,
159
+ f"{prefix}.layers.{layer_id - 1}.engram_layer",
160
+ )
161
+ self.token_lookback_depth = (
162
+ self.config.engram_max_ngram_size - 1 if self.engram_hash is not None else 0
163
+ )
164
+
165
+ def forward(
166
+ self,
167
+ input_ids: torch.Tensor,
168
+ positions: torch.Tensor,
169
+ intermediate_tensors: IntermediateTensors | None = None,
170
+ inputs_embeds: torch.Tensor | None = None,
171
+ lookback_token_ids: torch.Tensor | None = None,
172
+ ) -> torch.Tensor:
173
+ if self.engram_hash is None:
174
+ return super().forward(
175
+ input_ids, positions, intermediate_tensors, inputs_embeds
176
+ )
177
+ if input_ids is None:
178
+ raise ValueError("Flash Engram requires original input token IDs")
179
+ hidden = (
180
+ inputs_embeds
181
+ if inputs_embeds is not None
182
+ else self.embed_input_ids(input_ids)
183
+ )
184
+ hashes = input_ids.new_empty(
185
+ (
186
+ len(self.config.engram_layer_ids),
187
+ input_ids.numel(),
188
+ self.engram_hash.offsets.shape[1],
189
+ ),
190
+ dtype=torch.int32,
191
+ )
192
+ self.engram_hash.prepare_hashes_with_output(
193
+ input_ids,
194
+ lookback_token_ids,
195
+ self.layers[0].self_attn.mla_attn.layer_name,
196
+ hashes,
197
+ )
198
+ hash_indices = {
199
+ layer_id - 1: index
200
+ for index, layer_id in enumerate(self.config.engram_layer_ids)
201
+ }
202
+ for index, layer in enumerate(self.layers):
203
+ if index in hash_indices:
204
+ hidden = layer.hc_attn_layer.prepare_input(hidden)
205
+ hidden = layer.engram_layer(hidden, hashes[hash_indices[index]])
206
+ hidden, _ = layer(positions, hidden, None)
207
+ return self.norm(self.hc_head(hidden))
208
+
209
+ def _record_part(self, name: str, part: Any) -> None:
210
+ parts = self.loaded_parts.setdefault(name, set())
211
+ if part in parts or (None in parts) or (part is None and parts):
212
+ raise ValueError(f"Duplicate Flash checkpoint weight or shard: {name}")
213
+ parts.add(part)
214
+
215
+ def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
216
+ parameters = dict(self.named_parameters())
217
+ loaded: set[str] = set()
218
+ for name, value in weights:
219
+ engram = re.fullmatch(r"layers\.(\d+)\.engram_layer\.(.+)", name)
220
+ if engram is not None:
221
+ assert self.engram_hash is not None
222
+ layer, leaf = engram.groups()
223
+ if int(layer) + 1 not in (self.config.engram_layer_ids or ()):
224
+ raise ValueError(f"Undeclared Engram checkpoint weight: {name}")
225
+ if leaf == "lookup_table":
226
+ self._record_part(name, None)
227
+ self.engram_hash.load_lookup(value)
228
+ continue
229
+ if leaf in (
230
+ "multi_head_embedding.embedding.weight",
231
+ "multi_head_embedding.embedding.weight_scale_inv",
232
+ ):
233
+ self._record_part(name, None)
234
+ embedding = self.layers[
235
+ int(layer)
236
+ ].engram_layer.multi_head_embedding
237
+ embedding.load_weight(leaf.rsplit(".", 1)[1], value)
238
+ loaded.add(name)
239
+ continue
240
+ expert = _EXPERT_WEIGHT.fullmatch(name)
241
+ if expert:
242
+ prefix, expert_id, projection, suffix = expert.groups()
243
+ shards = {
244
+ "gate_proj": ("w1",),
245
+ "up_proj": ("w3",),
246
+ "down_proj": ("w2",),
247
+ "gate_up_proj": ("w1", "w3"),
248
+ }[projection]
249
+ half = "w2" if projection == "down_proj" else "w13"
250
+ if suffix == ".input_scale":
251
+ target = f"{prefix}.routed_experts.{half}_input_scale"
252
+ else:
253
+ suffix = (suffix or "").removeprefix(".weight")
254
+ target = f"{prefix}.routed_experts.{half}_weight{suffix}"
255
+ if target not in parameters:
256
+ raise ValueError(f"Unknown Flash expert weight: {name}")
257
+ parameter = parameters[target]
258
+ if expert_id is None:
259
+ if value.ndim != 3 or value.shape[0] != self.config.num_experts:
260
+ raise ValueError(
261
+ f"Invalid packed expert shape for {name}: {value.shape}"
262
+ )
263
+ expert_ids = list(range(self.config.num_experts))
264
+ else:
265
+ expert_ids = [int(expert_id)]
266
+ for index in expert_ids:
267
+ if not 0 <= index < self.config.num_experts:
268
+ raise ValueError(f"Expert index out of range: {name}")
269
+ tensor = value[index] if expert_id is None else value
270
+ pieces = (
271
+ (tensor,)
272
+ if len(shards) == 1
273
+ else tensor.chunk(len(shards), dim=-2)
274
+ )
275
+ for shard, piece in zip(shards, pieces):
276
+ success = parameter.weight_loader(
277
+ parameter,
278
+ piece,
279
+ target,
280
+ shard_id=shard,
281
+ expert_id=index,
282
+ return_success=True,
283
+ )
284
+ if success:
285
+ self._record_part(target, (index, shard))
286
+ loaded.add(target)
287
+ continue
288
+ if name.endswith((".hc_fn", ".hc_head_fn")):
289
+ name += ".weight"
290
+ name = name.replace(".gate.e_score_correction_bias", ".expert_bias")
291
+ name = name.replace(".router.gate.", ".gate.")
292
+ if ".engram_layer.lookup_per_head_norms." in name:
293
+ prefix, suffix = name.split(".lookup_per_head_norms.")
294
+ head, leaf = suffix.split(".", 1)
295
+ embedding = self.get_submodule(prefix).multi_head_embedding
296
+ if not head.isdigit() or not 0 <= int(head) < embedding.n_hash_cols:
297
+ raise ValueError(f"Invalid Engram hash head weight: {name}")
298
+ if not embedding.head_start <= int(head) < embedding.head_end:
299
+ continue
300
+ local_head = int(head) - embedding.head_start
301
+ name = f"{prefix}.lookup_per_head_norms.{local_head}.{leaf}"
302
+ if name not in parameters:
303
+ raise ValueError(f"Unknown Flash checkpoint weight: {name}")
304
+ parameter = parameters[name]
305
+ weight_shard = getattr(value, "shard_id", None)
306
+ self._record_part(name, weight_shard)
307
+ loader: Any = getattr(parameter, "weight_loader", default_weight_loader)
308
+ if weight_shard is None:
309
+ loader(parameter, value)
310
+ else:
311
+ loader(parameter, value, weight_shard)
312
+ loaded.add(name)
313
+ return loaded
314
+
315
+ def verify_loaded_parts(self) -> None:
316
+ for layer_id in self.config.engram_layer_ids or ():
317
+ prefix = f"layers.{layer_id - 1}.engram_layer."
318
+ required = {
319
+ prefix + "lookup_table",
320
+ prefix + "multi_head_embedding.embedding.weight",
321
+ }
322
+ embedding = self.layers[layer_id - 1].engram_layer.multi_head_embedding
323
+ if embedding.fp8:
324
+ required.add(prefix + "multi_head_embedding.embedding.weight_scale_inv")
325
+ missing = required - self.loaded_parts.keys()
326
+ if missing:
327
+ raise ValueError(
328
+ f"Missing Engram checkpoint weights: {sorted(missing)}"
329
+ )
330
+ for name, parts in self.loaded_parts.items():
331
+ if None in parts:
332
+ continue
333
+ module_name, _, leaf = name.rpartition(".")
334
+ module = self.get_submodule(module_name)
335
+ expected: set[Any]
336
+ if leaf.startswith(("w13_", "w2_")):
337
+ expert_map = module.expert_map_manager.expert_map
338
+ local_ids = (
339
+ range(self.config.num_experts)
340
+ if expert_map is None
341
+ else (expert_map >= 0).nonzero().flatten().tolist()
342
+ )
343
+ shards = ("w1", "w3") if leaf.startswith("w13_") else ("w2",)
344
+ expected = {(i, s) for i in local_ids for s in shards}
345
+ else:
346
+ expected = set(range(len(module.output_sizes)))
347
+ # NVFP4 per-tensor scales (input_scale, weight_scale_2) also load for
348
+ # experts another expert-parallel rank owns.
349
+ per_tensor = leaf.endswith(("_input_scale", "_weight_scale_2"))
350
+ if not (expected <= parts if per_tensor else parts == expected):
351
+ raise ValueError(f"Incomplete Flash checkpoint shards: {name}")
352
+
353
+
354
+ class HYV4FlashForCausalLM(HYV4ForCausalLM):
355
+ model_cls = HYV4FlashModel
356
+ supports_lora = False # type: ignore[assignment]
357
+ supports_pp = False # type: ignore[assignment]
358
+ requires_raw_input_tokens = True
359
+ packed_modules_mapping = {"gate_up_proj": ["gate_proj", "up_proj"]}
360
+ hf_to_vllm_mapper = WeightsMapper(
361
+ orig_to_new_stacked={
362
+ ".mlp.gate_proj": (".mlp.gate_up_proj", 0),
363
+ ".mlp.up_proj": (".mlp.gate_up_proj", 1),
364
+ ".shared_experts.gate_proj": (".shared_experts.gate_up_proj", 0),
365
+ ".shared_experts.up_proj": (".shared_experts.gate_up_proj", 1),
366
+ },
367
+ orig_to_new_substr={".compressed_indexer.": ".indexer."},
368
+ )
369
+
370
+ def __init__(self, *, vllm_config: VllmConfig, prefix: str = "") -> None:
371
+ validate_runtime(vllm_config)
372
+ super().__init__(vllm_config=vllm_config, prefix=prefix)
373
+ self.token_lookback_depth = self.model.token_lookback_depth
374
+ self._weight_loading_depth = 0
375
+ self._seen_weights: set[str] = set()
376
+ self._loaded_weights: set[str] = set()
377
+ self._load_config = vllm_config.load_config
378
+
379
+ @staticmethod
380
+ def get_model_state_cls():
381
+ from vllm.models.deepseek_v41.nvidia.model_state import DeepseekV41ModelState
382
+
383
+ return DeepseekV41ModelState
384
+
385
+ def forward(
386
+ self,
387
+ input_ids,
388
+ positions,
389
+ intermediate_tensors=None,
390
+ inputs_embeds=None,
391
+ lookback_token_ids=None,
392
+ ):
393
+ return self.model(
394
+ input_ids,
395
+ positions,
396
+ intermediate_tensors,
397
+ inputs_embeds,
398
+ lookback_token_ids,
399
+ )
400
+
401
+ @contextmanager
402
+ def weight_loading_session(self):
403
+ """Audit the complete stream, including interleaved VL tower weights."""
404
+ outermost = self._weight_loading_depth == 0
405
+ if outermost:
406
+ self.model.loaded_parts.clear()
407
+ self._seen_weights.clear()
408
+ self._loaded_weights.clear()
409
+ self._weight_loading_depth += 1
410
+ try:
411
+ yield
412
+ if outermost:
413
+ from vllm.model_executor.model_loader.default_loader import (
414
+ DefaultModelLoader,
415
+ )
416
+
417
+ required = {
418
+ name
419
+ for name, _ in self.named_parameters()
420
+ if name.endswith((".weight", ".w13_weight", ".w2_weight"))
421
+ }
422
+ missing = required - self._loaded_weights
423
+ if missing:
424
+ raise ValueError(
425
+ f"Missing Flash checkpoint weights: {sorted(missing)}"
426
+ )
427
+ DefaultModelLoader(self._load_config).track_weights_loading(
428
+ self, self._loaded_weights.copy(), quantized=True
429
+ )
430
+ self.model.verify_loaded_parts()
431
+ finally:
432
+ self._weight_loading_depth -= 1
433
+ if outermost:
434
+ self._seen_weights.clear()
435
+ self._loaded_weights.clear()
436
+
437
+ def _target_weights(self, weights):
438
+ for name, value in weights:
439
+ if name in self._seen_weights:
440
+ raise ValueError(f"Duplicate Flash checkpoint weight: {name}")
441
+ self._seen_weights.add(name)
442
+ engram = re.fullmatch(
443
+ r"model\.layers\.(\d+)\.(engram|engram_layer)\.(.+)", name
444
+ )
445
+ if engram is not None:
446
+ layer_number, prefix, leaf = engram.groups()
447
+ layer_id = int(layer_number) + (prefix == "engram_layer")
448
+ if layer_id not in (self.config.engram_layer_ids or ()):
449
+ raise ValueError(f"Undeclared Engram checkpoint weight: {name}")
450
+ name = f"model.layers.{layer_id - 1}.engram_layer.{leaf}"
451
+ skip = False
452
+ if name.startswith("model.layers."):
453
+ layer = name.removeprefix("model.layers.").split(".", 1)[0]
454
+ first = self.config.num_hidden_layers
455
+ if layer.isdigit() and first <= int(layer) < (
456
+ first + self.config.num_nextn_predict_layers
457
+ ):
458
+ continue
459
+ for prefix, count in (
460
+ ("model.mtp_layers.", self.config.num_nextn_predict_layers),
461
+ (
462
+ "model.dfly_layers.",
463
+ getattr(self.config, "dfly_num_hidden_layers", 0),
464
+ ),
465
+ ):
466
+ if name.startswith(prefix):
467
+ local = name[len(prefix) :]
468
+ layer = local.removeprefix("layers.").split(".", 1)[0]
469
+ declared = count > 0 and layer.isdigit() and 0 <= int(layer) < count
470
+ if prefix == "model.dfly_layers." and count > 0:
471
+ declared |= local.startswith(
472
+ (
473
+ "mtp_fusion.",
474
+ "hidden_correction.",
475
+ "final_norm.",
476
+ "norm.",
477
+ "embed_tokens.",
478
+ "confidence_head.",
479
+ )
480
+ )
481
+ if not declared:
482
+ raise ValueError(f"Undeclared auxiliary Flash weight: {name}")
483
+ skip = True
484
+ break
485
+ if not skip:
486
+ yield name, value
487
+
488
+ def load_weights(self, weights: Iterable[tuple[str, torch.Tensor]]) -> set[str]:
489
+ with self.weight_loading_session():
490
+ loaded = AutoWeightsLoader(self).load_weights(
491
+ self._target_weights(weights), mapper=self.hf_to_vllm_mapper
492
+ )
493
+ self._loaded_weights.update(loaded)
494
+ return loaded
edit/wsp_edit.json ADDED
@@ -0,0 +1,63 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "kind": "cc",
3
+ "layers": [
4
+ 22,
5
+ 24,
6
+ 26,
7
+ 28,
8
+ 30,
9
+ 32,
10
+ 34,
11
+ 36,
12
+ 38
13
+ ],
14
+ "alpha": 0.1,
15
+ "control": null,
16
+ "info": [
17
+ {
18
+ "layer": 22,
19
+ "rel_dW": 0.008451332629005236,
20
+ "realized_rel": 0.008508533194460986
21
+ },
22
+ {
23
+ "layer": 24,
24
+ "rel_dW": 0.008579217399337351,
25
+ "realized_rel": 0.008620550293118253
26
+ },
27
+ {
28
+ "layer": 26,
29
+ "rel_dW": 0.008742168960215294,
30
+ "realized_rel": 0.008798305477918418
31
+ },
32
+ {
33
+ "layer": 28,
34
+ "rel_dW": 0.007386515136851949,
35
+ "realized_rel": 0.007420708643587443
36
+ },
37
+ {
38
+ "layer": 30,
39
+ "rel_dW": 0.01083609880478906,
40
+ "realized_rel": 0.01089166898530135
41
+ },
42
+ {
43
+ "layer": 32,
44
+ "rel_dW": 0.008441069220802839,
45
+ "realized_rel": 0.008483294695031634
46
+ },
47
+ {
48
+ "layer": 34,
49
+ "rel_dW": 0.007835523201643109,
50
+ "realized_rel": 0.007888640015587327
51
+ },
52
+ {
53
+ "layer": 36,
54
+ "rel_dW": 0.00558963115316366,
55
+ "realized_rel": 0.0056215510361703695
56
+ },
57
+ {
58
+ "layer": 38,
59
+ "rel_dW": 0.0062650998924194115,
60
+ "realized_rel": 0.006295879478533104
61
+ }
62
+ ]
63
+ }
engram-fp8.json ADDED
@@ -0,0 +1,18 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "model.layers.1.engram_layer.multi_head_embedding.embedding.weight": {
3
+ "rows": 164994380,
4
+ "rel_rms_error": 0.025794996128077223
5
+ },
6
+ "model.layers.11.engram_layer.multi_head_embedding.embedding.weight": {
7
+ "rows": 164999008,
8
+ "rel_rms_error": 0.025905215853857078
9
+ },
10
+ "model.layers.21.engram_layer.multi_head_embedding.embedding.weight": {
11
+ "rows": 165004086,
12
+ "rel_rms_error": 0.026003107898576922
13
+ },
14
+ "model.layers.31.engram_layer.multi_head_embedding.embedding.weight": {
15
+ "rows": 165008942,
16
+ "rel_rms_error": 0.02601790873167511
17
+ }
18
+ }
generation_config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": 120000,
3
+ "do_sample": true,
4
+ "eos_token_id": 120025,
5
+ "pad_token_id": 120002,
6
+ "temperature": 0.9,
7
+ "top_k": -1,
8
+ "top_p": 1,
9
+ "transformers_version": "5.23.0"
10
+ }
gguf-lora/hy4-flash-smart-lora.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:690721fac8ca738ea6339a62adadb39a92b8ea8796c11b4181e476b4f22ea9b3
3
+ size 167424
hf_quant_config.json ADDED
@@ -0,0 +1,280 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "producer": {
3
+ "name": "modelopt",
4
+ "version": "0.48.0"
5
+ },
6
+ "quantization": {
7
+ "quant_algo": "NVFP4",
8
+ "kv_cache_quant_algo": null,
9
+ "group_size": 16,
10
+ "exclude_modules": [
11
+ "lm_head",
12
+ "model.dfly_layers.0.input_layernorm",
13
+ "model.dfly_layers.0.mlp.down_proj",
14
+ "model.dfly_layers.0.mlp.gate_proj",
15
+ "model.dfly_layers.0.mlp.up_proj",
16
+ "model.dfly_layers.0.post_attention_layernorm",
17
+ "model.dfly_layers.0.self_attn",
18
+ "model.dfly_layers.0.self_attn.kv_a_layernorm",
19
+ "model.dfly_layers.0.self_attn.kv_a_proj_with_mqa",
20
+ "model.dfly_layers.0.self_attn.kv_b_proj",
21
+ "model.dfly_layers.0.self_attn.linear_gate",
22
+ "model.dfly_layers.0.self_attn.o_proj",
23
+ "model.dfly_layers.0.self_attn.post_proj_norm",
24
+ "model.dfly_layers.0.self_attn.q_a_layernorm",
25
+ "model.dfly_layers.0.self_attn.q_a_proj",
26
+ "model.dfly_layers.0.self_attn.q_b_proj",
27
+ "model.dfly_layers.1.input_layernorm",
28
+ "model.dfly_layers.1.mlp.down_proj",
29
+ "model.dfly_layers.1.mlp.gate_proj",
30
+ "model.dfly_layers.1.mlp.up_proj",
31
+ "model.dfly_layers.1.post_attention_layernorm",
32
+ "model.dfly_layers.1.self_attn",
33
+ "model.dfly_layers.1.self_attn.kv_a_layernorm",
34
+ "model.dfly_layers.1.self_attn.kv_a_proj_with_mqa",
35
+ "model.dfly_layers.1.self_attn.kv_b_proj",
36
+ "model.dfly_layers.1.self_attn.linear_gate",
37
+ "model.dfly_layers.1.self_attn.o_proj",
38
+ "model.dfly_layers.1.self_attn.post_proj_norm",
39
+ "model.dfly_layers.1.self_attn.q_a_layernorm",
40
+ "model.dfly_layers.1.self_attn.q_a_proj",
41
+ "model.dfly_layers.1.self_attn.q_b_proj",
42
+ "model.dfly_layers.2.input_layernorm",
43
+ "model.dfly_layers.2.mlp.down_proj",
44
+ "model.dfly_layers.2.mlp.gate_proj",
45
+ "model.dfly_layers.2.mlp.up_proj",
46
+ "model.dfly_layers.2.post_attention_layernorm",
47
+ "model.dfly_layers.2.self_attn",
48
+ "model.dfly_layers.2.self_attn.kv_a_layernorm",
49
+ "model.dfly_layers.2.self_attn.kv_a_proj_with_mqa",
50
+ "model.dfly_layers.2.self_attn.kv_b_proj",
51
+ "model.dfly_layers.2.self_attn.linear_gate",
52
+ "model.dfly_layers.2.self_attn.o_proj",
53
+ "model.dfly_layers.2.self_attn.post_proj_norm",
54
+ "model.dfly_layers.2.self_attn.q_a_layernorm",
55
+ "model.dfly_layers.2.self_attn.q_a_proj",
56
+ "model.dfly_layers.2.self_attn.q_b_proj",
57
+ "model.dfly_layers.3.input_layernorm",
58
+ "model.dfly_layers.3.mlp.down_proj",
59
+ "model.dfly_layers.3.mlp.gate_proj",
60
+ "model.dfly_layers.3.mlp.up_proj",
61
+ "model.dfly_layers.3.post_attention_layernorm",
62
+ "model.dfly_layers.3.self_attn",
63
+ "model.dfly_layers.3.self_attn.kv_a_layernorm",
64
+ "model.dfly_layers.3.self_attn.kv_a_proj_with_mqa",
65
+ "model.dfly_layers.3.self_attn.kv_b_proj",
66
+ "model.dfly_layers.3.self_attn.linear_gate",
67
+ "model.dfly_layers.3.self_attn.o_proj",
68
+ "model.dfly_layers.3.self_attn.post_proj_norm",
69
+ "model.dfly_layers.3.self_attn.q_a_layernorm",
70
+ "model.dfly_layers.3.self_attn.q_a_proj",
71
+ "model.dfly_layers.3.self_attn.q_b_proj",
72
+ "model.dfly_layers.4.input_layernorm",
73
+ "model.dfly_layers.4.mlp.down_proj",
74
+ "model.dfly_layers.4.mlp.gate_proj",
75
+ "model.dfly_layers.4.mlp.up_proj",
76
+ "model.dfly_layers.4.post_attention_layernorm",
77
+ "model.dfly_layers.4.self_attn",
78
+ "model.dfly_layers.4.self_attn.kv_a_layernorm",
79
+ "model.dfly_layers.4.self_attn.kv_a_proj_with_mqa",
80
+ "model.dfly_layers.4.self_attn.kv_b_proj",
81
+ "model.dfly_layers.4.self_attn.linear_gate",
82
+ "model.dfly_layers.4.self_attn.o_proj",
83
+ "model.dfly_layers.4.self_attn.post_proj_norm",
84
+ "model.dfly_layers.4.self_attn.q_a_layernorm",
85
+ "model.dfly_layers.4.self_attn.q_a_proj",
86
+ "model.dfly_layers.4.self_attn.q_b_proj",
87
+ "model.dfly_layers.confidence_head.proj",
88
+ "model.dfly_layers.final_norm",
89
+ "model.dfly_layers.hidden_correction.down_proj",
90
+ "model.dfly_layers.hidden_correction.embed_norm",
91
+ "model.dfly_layers.hidden_correction.gate_proj",
92
+ "model.dfly_layers.hidden_correction.hidden_norm",
93
+ "model.dfly_layers.hidden_correction.up_proj",
94
+ "model.dfly_layers.mtp_fusion",
95
+ "model.dfly_layers.mtp_fusion.eh_proj",
96
+ "model.dfly_layers.mtp_fusion.enorm",
97
+ "model.dfly_layers.mtp_fusion.hnorm",
98
+ "model.embed_tokens",
99
+ "model.layers.0.self_attn*",
100
+ "model.layers.1.engram_layer*",
101
+ "model.layers.1.mlp.gate",
102
+ "model.layers.1.mlp.routed_expert_down_proj",
103
+ "model.layers.1.mlp.routed_expert_up_proj",
104
+ "model.layers.1.self_attn*",
105
+ "model.layers.10.mlp.gate",
106
+ "model.layers.10.mlp.routed_expert_down_proj",
107
+ "model.layers.10.mlp.routed_expert_up_proj",
108
+ "model.layers.10.self_attn*",
109
+ "model.layers.11.engram_layer*",
110
+ "model.layers.11.mlp.gate",
111
+ "model.layers.11.mlp.routed_expert_down_proj",
112
+ "model.layers.11.mlp.routed_expert_up_proj",
113
+ "model.layers.11.self_attn*",
114
+ "model.layers.12.mlp.gate",
115
+ "model.layers.12.mlp.routed_expert_down_proj",
116
+ "model.layers.12.mlp.routed_expert_up_proj",
117
+ "model.layers.12.self_attn*",
118
+ "model.layers.13.mlp.gate",
119
+ "model.layers.13.mlp.routed_expert_down_proj",
120
+ "model.layers.13.mlp.routed_expert_up_proj",
121
+ "model.layers.13.self_attn*",
122
+ "model.layers.14.mlp.gate",
123
+ "model.layers.14.mlp.routed_expert_down_proj",
124
+ "model.layers.14.mlp.routed_expert_up_proj",
125
+ "model.layers.14.self_attn*",
126
+ "model.layers.15.mlp.gate",
127
+ "model.layers.15.mlp.routed_expert_down_proj",
128
+ "model.layers.15.mlp.routed_expert_up_proj",
129
+ "model.layers.15.self_attn*",
130
+ "model.layers.16.mlp.gate",
131
+ "model.layers.16.mlp.routed_expert_down_proj",
132
+ "model.layers.16.mlp.routed_expert_up_proj",
133
+ "model.layers.16.self_attn*",
134
+ "model.layers.17.mlp.gate",
135
+ "model.layers.17.mlp.routed_expert_down_proj",
136
+ "model.layers.17.mlp.routed_expert_up_proj",
137
+ "model.layers.17.self_attn*",
138
+ "model.layers.18.mlp.gate",
139
+ "model.layers.18.mlp.routed_expert_down_proj",
140
+ "model.layers.18.mlp.routed_expert_up_proj",
141
+ "model.layers.18.self_attn*",
142
+ "model.layers.19.mlp.gate",
143
+ "model.layers.19.mlp.routed_expert_down_proj",
144
+ "model.layers.19.mlp.routed_expert_up_proj",
145
+ "model.layers.19.self_attn*",
146
+ "model.layers.2.mlp.gate",
147
+ "model.layers.2.mlp.routed_expert_down_proj",
148
+ "model.layers.2.mlp.routed_expert_up_proj",
149
+ "model.layers.2.self_attn*",
150
+ "model.layers.20.mlp.gate",
151
+ "model.layers.20.mlp.routed_expert_down_proj",
152
+ "model.layers.20.mlp.routed_expert_up_proj",
153
+ "model.layers.20.self_attn*",
154
+ "model.layers.21.engram_layer*",
155
+ "model.layers.21.mlp.gate",
156
+ "model.layers.21.mlp.routed_expert_down_proj",
157
+ "model.layers.21.mlp.routed_expert_up_proj",
158
+ "model.layers.21.self_attn*",
159
+ "model.layers.22.mlp.gate",
160
+ "model.layers.22.mlp.routed_expert_down_proj",
161
+ "model.layers.22.mlp.routed_expert_up_proj",
162
+ "model.layers.22.self_attn*",
163
+ "model.layers.23.mlp.gate",
164
+ "model.layers.23.mlp.routed_expert_down_proj",
165
+ "model.layers.23.mlp.routed_expert_up_proj",
166
+ "model.layers.23.self_attn*",
167
+ "model.layers.24.mlp.gate",
168
+ "model.layers.24.mlp.routed_expert_down_proj",
169
+ "model.layers.24.mlp.routed_expert_up_proj",
170
+ "model.layers.24.self_attn*",
171
+ "model.layers.25.mlp.gate",
172
+ "model.layers.25.mlp.routed_expert_down_proj",
173
+ "model.layers.25.mlp.routed_expert_up_proj",
174
+ "model.layers.25.self_attn*",
175
+ "model.layers.26.mlp.gate",
176
+ "model.layers.26.mlp.routed_expert_down_proj",
177
+ "model.layers.26.mlp.routed_expert_up_proj",
178
+ "model.layers.26.self_attn*",
179
+ "model.layers.27.mlp.gate",
180
+ "model.layers.27.mlp.routed_expert_down_proj",
181
+ "model.layers.27.mlp.routed_expert_up_proj",
182
+ "model.layers.27.self_attn*",
183
+ "model.layers.28.mlp.gate",
184
+ "model.layers.28.mlp.routed_expert_down_proj",
185
+ "model.layers.28.mlp.routed_expert_up_proj",
186
+ "model.layers.28.self_attn*",
187
+ "model.layers.29.mlp.gate",
188
+ "model.layers.29.mlp.routed_expert_down_proj",
189
+ "model.layers.29.mlp.routed_expert_up_proj",
190
+ "model.layers.29.self_attn*",
191
+ "model.layers.3.mlp.gate",
192
+ "model.layers.3.mlp.routed_expert_down_proj",
193
+ "model.layers.3.mlp.routed_expert_up_proj",
194
+ "model.layers.3.self_attn*",
195
+ "model.layers.30.mlp.gate",
196
+ "model.layers.30.mlp.routed_expert_down_proj",
197
+ "model.layers.30.mlp.routed_expert_up_proj",
198
+ "model.layers.30.self_attn*",
199
+ "model.layers.31.engram_layer*",
200
+ "model.layers.31.mlp.gate",
201
+ "model.layers.31.mlp.routed_expert_down_proj",
202
+ "model.layers.31.mlp.routed_expert_up_proj",
203
+ "model.layers.31.self_attn*",
204
+ "model.layers.32.mlp.gate",
205
+ "model.layers.32.mlp.routed_expert_down_proj",
206
+ "model.layers.32.mlp.routed_expert_up_proj",
207
+ "model.layers.32.self_attn*",
208
+ "model.layers.33.mlp.gate",
209
+ "model.layers.33.mlp.routed_expert_down_proj",
210
+ "model.layers.33.mlp.routed_expert_up_proj",
211
+ "model.layers.33.self_attn*",
212
+ "model.layers.34.mlp.gate",
213
+ "model.layers.34.mlp.routed_expert_down_proj",
214
+ "model.layers.34.mlp.routed_expert_up_proj",
215
+ "model.layers.34.self_attn*",
216
+ "model.layers.35.mlp.gate",
217
+ "model.layers.35.mlp.routed_expert_down_proj",
218
+ "model.layers.35.mlp.routed_expert_up_proj",
219
+ "model.layers.35.self_attn*",
220
+ "model.layers.36.mlp.gate",
221
+ "model.layers.36.mlp.routed_expert_down_proj",
222
+ "model.layers.36.mlp.routed_expert_up_proj",
223
+ "model.layers.36.self_attn*",
224
+ "model.layers.37.mlp.gate",
225
+ "model.layers.37.mlp.routed_expert_down_proj",
226
+ "model.layers.37.mlp.routed_expert_up_proj",
227
+ "model.layers.37.self_attn*",
228
+ "model.layers.38.mlp.gate",
229
+ "model.layers.38.mlp.routed_expert_down_proj",
230
+ "model.layers.38.mlp.routed_expert_up_proj",
231
+ "model.layers.38.self_attn*",
232
+ "model.layers.39.mlp.gate",
233
+ "model.layers.39.mlp.routed_expert_down_proj",
234
+ "model.layers.39.mlp.routed_expert_up_proj",
235
+ "model.layers.39.self_attn*",
236
+ "model.layers.4.mlp.gate",
237
+ "model.layers.4.mlp.routed_expert_down_proj",
238
+ "model.layers.4.mlp.routed_expert_up_proj",
239
+ "model.layers.4.self_attn*",
240
+ "model.layers.40.mlp.gate",
241
+ "model.layers.40.mlp.routed_expert_down_proj",
242
+ "model.layers.40.mlp.routed_expert_up_proj",
243
+ "model.layers.40.self_attn*",
244
+ "model.layers.41.mlp.gate",
245
+ "model.layers.41.mlp.routed_expert_down_proj",
246
+ "model.layers.41.mlp.routed_expert_up_proj",
247
+ "model.layers.41.self_attn*",
248
+ "model.layers.5.mlp.gate",
249
+ "model.layers.5.mlp.routed_expert_down_proj",
250
+ "model.layers.5.mlp.routed_expert_up_proj",
251
+ "model.layers.5.self_attn*",
252
+ "model.layers.6.mlp.gate",
253
+ "model.layers.6.mlp.routed_expert_down_proj",
254
+ "model.layers.6.mlp.routed_expert_up_proj",
255
+ "model.layers.6.self_attn*",
256
+ "model.layers.7.mlp.gate",
257
+ "model.layers.7.mlp.routed_expert_down_proj",
258
+ "model.layers.7.mlp.routed_expert_up_proj",
259
+ "model.layers.7.self_attn*",
260
+ "model.layers.8.mlp.gate",
261
+ "model.layers.8.mlp.routed_expert_down_proj",
262
+ "model.layers.8.mlp.routed_expert_up_proj",
263
+ "model.layers.8.self_attn*",
264
+ "model.layers.9.mlp.gate",
265
+ "model.layers.9.mlp.routed_expert_down_proj",
266
+ "model.layers.9.mlp.routed_expert_up_proj",
267
+ "model.layers.9.self_attn*",
268
+ "vit*",
269
+ "model.layers.22.mlp.shared_experts.down_proj",
270
+ "model.layers.24.mlp.shared_experts.down_proj",
271
+ "model.layers.26.mlp.shared_experts.down_proj",
272
+ "model.layers.28.mlp.shared_experts.down_proj",
273
+ "model.layers.30.mlp.shared_experts.down_proj",
274
+ "model.layers.32.mlp.shared_experts.down_proj",
275
+ "model.layers.34.mlp.shared_experts.down_proj",
276
+ "model.layers.36.mlp.shared_experts.down_proj",
277
+ "model.layers.38.mlp.shared_experts.down_proj"
278
+ ]
279
+ }
280
+ }
model-00001-of-00014.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2770407ac96d3cd0a00e3a87a8d09ede99c40ea86beaa0e238f1b093c906239c
3
+ size 16169449576
model-00002-of-00014.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:2d0f08a031e9026a58aa32efeb8b1a899dff00e8fdcb7c08783da8b236abbe32
3
+ size 10002693080
model-00003-of-00014.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4af41288e28c8c176ea70eb0fbcca413618cc9dabebd153c3c04679cad5e29ae
3
+ size 10002167744
model-00004-of-00014.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9d91fb44da775f2c0a1f7fce0577c41129499aa71c9a7ee9452006e24df3c19e
3
+ size 16169903120
model-00005-of-00014.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a4c468ac7f26bbf92f91a8f8b9fe9dacff51f68a7ac93b21add641e5b3c7b1f7
3
+ size 10002109512
model-00006-of-00014.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c93c76abe2110052e5115f4826fbf9abd9d286e9c4f5faf02bd8917a17c50acd
3
+ size 10002123808
model-00007-of-00014.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a30e0462cca97b895c45bda50855faecfa25e40ffe947652ea6088ee2c0443bd
3
+ size 16170400764
model-00008-of-00014.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7f71ccdfadde3517f2917cbd6fad1f59a540da8d574cd781edda27ebe8d5a769
3
+ size 10002973592
model-00009-of-00014.strip83884278.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f75c45ebaa6235eb9224cea2f97b311b3a34f79c3bc43e2f4baadd161b25ac08
3
+ size 9996811008
model-00010-of-00014.strip83884278.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:71bcc3c0f06354d189c15fb76543cf4e55dec8cb6900eaf732ed96532f83f870
3
+ size 9996813968
model-00011-of-00014.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e45293bf47d08a1880dd7905da8d66616169b4fd3d59f47e94816894130c5385
3
+ size 16170876652
model-00012-of-00014.strip83884278.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:485e9b922487ffb0542c9f4aae4801a0daeae529907bbff5414b0ef14a94b1ec
3
+ size 9995009360
model-00013-of-00014.strip83884278.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:84f5c31e1f01cdf486dc98aaab23af94c0c71e5dc8440d67da87932adab89091
3
+ size 9996810192
model-00014-of-00014.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5d25da162eba88fb852545b6a07a2d70b6ca3139e25e30da255c8d3fcf090751
3
+ size 9952356094
model-wsp-edit.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fa1250a69f5b0c50448b5f1cb24ebd59b1ea9580da8076246a15c28d329ba372
3
+ size 84935824
model.safetensors.index.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:eecdf77b2aefeedf44a4dfcac4ed378a2a1c788ec0e138f014f032efc9613d31
3
+ size 25285399
processor_config.json ADDED
@@ -0,0 +1,75 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "image_processor": {
3
+ "device": "cpu",
4
+ "do_convert_rgb": true,
5
+ "do_normalize": true,
6
+ "do_rescale": true,
7
+ "do_resize": true,
8
+ "image_mean": [
9
+ 0.5,
10
+ 0.5,
11
+ 0.5
12
+ ],
13
+ "image_processor_type": "HYV4VLImageProcessor",
14
+ "image_std": [
15
+ 0.5,
16
+ 0.5,
17
+ 0.5
18
+ ],
19
+ "max_frames": 1024,
20
+ "max_pixels": 4194304,
21
+ "merge_size": 2,
22
+ "min_frames": 2,
23
+ "min_pixels": 4096,
24
+ "patch_size": 16,
25
+ "resample": 1,
26
+ "rescale_factor": 0.00392156862745098,
27
+ "size": {
28
+ "longest_edge": 4194304,
29
+ "shortest_edge": 4096
30
+ },
31
+ "spatial_patch_size": 1,
32
+ "temporal_patch_size": 1,
33
+ "timestamp_token_per_frame": 6,
34
+ "timestamp_token_reserve": 1000,
35
+ "video_max_image_size": 1024
36
+ },
37
+ "processor_class": "HYV4FlashProcessor",
38
+ "video_processor": {
39
+ "do_convert_rgb": true,
40
+ "do_normalize": true,
41
+ "do_rescale": true,
42
+ "do_resize": true,
43
+ "do_sample_frames": false,
44
+ "fps": 2,
45
+ "image_mean": [
46
+ 0.5,
47
+ 0.5,
48
+ 0.5
49
+ ],
50
+ "image_std": [
51
+ 0.5,
52
+ 0.5,
53
+ 0.5
54
+ ],
55
+ "max_frames": 2048,
56
+ "max_pixels": 61440000,
57
+ "merge_size": 2,
58
+ "min_frames": 2,
59
+ "min_pixels": 16384,
60
+ "patch_size": 16,
61
+ "resample": 1,
62
+ "rescale_factor": 0.00392156862745098,
63
+ "return_metadata": false,
64
+ "size": {
65
+ "longest_edge": 61440000,
66
+ "shortest_edge": 16384
67
+ },
68
+ "spatial_patch_size": 1,
69
+ "temporal_patch_size": 1,
70
+ "timestamp_token_per_frame": 6,
71
+ "timestamp_token_reserve": 1000,
72
+ "video_max_image_size": 1024,
73
+ "video_processor_type": "HYV4VLVideoProcessor"
74
+ }
75
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,28 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "backend": "tokenizers",
3
+ "bos_token": "<|hy_start:opensource|>",
4
+ "clean_up_tokenization_spaces": false,
5
+ "eos_token": "<|hy_end:opensource|>",
6
+ "image_end_token": "<|hy_image▁end:opensource|>",
7
+ "image_newline_token": "<|hy_image▁newline:opensource|>",
8
+ "image_start_token": "<|hy_image▁start:opensource|>",
9
+ "image_token": "<|hy_image▁pad:opensource|>",
10
+ "is_local": true,
11
+ "local_files_only": false,
12
+ "model_max_length": 1000000000000000019884624838656,
13
+ "model_specific_special_tokens": {
14
+ "image_end_token": "<|hy_image▁end:opensource|>",
15
+ "image_newline_token": "<|hy_image▁newline:opensource|>",
16
+ "image_start_token": "<|hy_image▁start:opensource|>",
17
+ "image_token": "<|hy_image▁pad:opensource|>",
18
+ "video_end_token": "<|hy_video▁end:opensource|>",
19
+ "video_start_token": "<|hy_video▁start:opensource|>",
20
+ "video_token": "<|hy_video▁pad:opensource|>"
21
+ },
22
+ "pad_token": "<|hy_pad:opensource|>",
23
+ "processor_class": "HYV4FlashProcessor",
24
+ "tokenizer_class": "TokenizersBackend",
25
+ "video_end_token": "<|hy_video▁end:opensource|>",
26
+ "video_start_token": "<|hy_video▁start:opensource|>",
27
+ "video_token": "<|hy_video▁pad:opensource|>"
28
+ }