kylesayrs commited on
Commit
ff38bee
·
verified ·
1 Parent(s): 7e0e0ca

GLM-5.3-Flash-NVFP4 pruned to 20% expert sparsity with REAP (RazorCal calibration)

Browse files
.gitattributes CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ model.safetensors.index.json filter=lfs diff=lfs merge=lfs -text
37
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
LICENSE ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Z.AI Co., Ltd
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
README.md ADDED
@@ -0,0 +1,65 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ - zh
5
+ library_name: transformers
6
+ pipeline_tag: image-text-to-text
7
+ tags:
8
+ - nvfp4
9
+ - fp4
10
+ - vllm
11
+ - llm-compressor
12
+ - compressed-tensors
13
+ - glm5_next
14
+ - expert-pruning
15
+ - reap
16
+ base_model:
17
+ - RedHatAI/GLM-5.3-Flash-NVFP4
18
+ license: mit
19
+ license_link: https://choosealicense.com/licenses/mit/
20
+ ---
21
+
22
+ # GLM-5.3-Flash-NVFP4-REAP20
23
+
24
+ [RedHatAI/GLM-5.3-Flash-NVFP4](https://huggingface.co/RedHatAI/GLM-5.3-Flash-NVFP4)
25
+ (a NVFP4 quantization of
26
+ [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash)) with 20%
27
+ of its MoE experts structurally pruned with **REAP** (PRUNE method), produced
28
+ with [llmcompressor](https://github.com/vllm-project/llmcompressor).
29
+
30
+ - Every one of the 42 MoE layers (backbone layers 3-44) keeps its **230 most
31
+ salient experts** of the original 288 (58 pruned per layer, 9660 of 12096
32
+ retained)
33
+ - Expert selection is uniform across layers: each layer's 58 lowest-REAP-saliency
34
+ experts were removed, and the routers and their `e_score_correction_bias`
35
+ were sliced to match
36
+ - `config.json` records the new expert count (`n_routed_experts: 230`); expert
37
+ indices are remapped contiguously
38
+ - The quantization is unchanged from the source checkpoint: MoE experts remain
39
+ NVFP4-quantized and the MTP layer (`layers.45`) is left untouched, keeping
40
+ its original 288 FP8-quantized experts (Transformers ignores `layers.45` when
41
+ loading, so this matches the source model's serving behavior)
42
+ - All remaining weights (including attention, vision tower, embeddings and the
43
+ shared experts) are untouched
44
+
45
+ REAP scores each expert by the router-weighted mean of its output activation
46
+ norms over the calibration tokens routed to it,
47
+ `S_j = mean(g_j * ||f_j||_2)`, using forward computation only (no gradients or
48
+ recovery training). This model was calibrated on all 2,048 samples of the
49
+ [RazorCal](https://huggingface.co/datasets/inference-optimization/RazorCal)
50
+ multi-domain calibration corpus; see the
51
+ [REAP expert pruning example](https://github.com/vllm-project/llm-compressor/tree/main/examples/reap_expert_pruning)
52
+ for the calibration and pruning scripts.
53
+
54
+ ## Usage
55
+
56
+ ```python
57
+ from transformers import AutoProcessor, Glm5NextForConditionalGeneration
58
+
59
+ model = Glm5NextForConditionalGeneration.from_pretrained(
60
+ "inference-optimization/GLM-5.3-Flash-NVFP4-REAP20"
61
+ )
62
+ processor = AutoProcessor.from_pretrained(
63
+ "inference-optimization/GLM-5.3-Flash-NVFP4-REAP20"
64
+ )
65
+ ```
chat_template.jinja ADDED
@@ -0,0 +1,261 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [gMASK]<sop>
2
+ {%- set effective_reasoning_effort = reasoning_effort if reasoning_effort is defined and reasoning_effort in ['low', 'high'] else 'max' -%}
3
+ {%- if effective_reasoning_effort is not none -%}<|system|>Reasoning Effort: {{ effective_reasoning_effort | capitalize }}{%- endif -%}
4
+ {%- set clear_thinking = clear_thinking if clear_thinking is defined else false -%}
5
+ {%- if tools -%}
6
+ {%- macro tool_to_json(tool) -%}
7
+ {%- set ns_tool = namespace(first=true) -%}
8
+ {{ '{' -}}
9
+ {%- for k, v in tool.items() -%}
10
+ {%- if k != 'defer_loading' and k != 'strict' -%}
11
+ {%- if not ns_tool.first -%}{{- ', ' -}}{%- endif -%}
12
+ {%- set ns_tool.first = false -%}
13
+ "{{ k }}": {{ v | tojson(ensure_ascii=False) }}
14
+ {%- endif -%}
15
+ {%- endfor -%}
16
+ {{- '}' -}}
17
+ {%- endmacro -%}
18
+ {%- macro tool_references_to_response(refs) -%}
19
+ {{- '<tool_response><tools>\n' -}}
20
+ {%- for tr in refs -%}
21
+ {%- for tool in tools -%}
22
+ {%- if 'function' in tool -%}
23
+ {%- set tool = tool['function'] -%}
24
+ {%- endif -%}
25
+ {%- if tool.name == tr.name -%}
26
+ {{- tool_to_json(tool) + '\n' -}}
27
+ {%- endif -%}
28
+ {%- endfor -%}
29
+ {%- endfor -%}
30
+ {{- '</tools></tool_response>' -}}
31
+ {%- endmacro -%}
32
+ <|system|>
33
+ # Tools
34
+
35
+ You may call one or more functions to assist with the user query.
36
+
37
+ You are provided with function signatures within <tools></tools> XML tags:
38
+ <tools>
39
+ {% for tool in tools %}
40
+ {%- if 'function' in tool -%}
41
+ {%- set tool = tool['function'] -%}
42
+ {%- endif -%}
43
+ {% if tool.defer_loading is not defined or not tool.defer_loading %}
44
+ {{ tool_to_json(tool) }}
45
+ {% endif %}
46
+ {% endfor %}
47
+ </tools>
48
+
49
+ For each function call, output the function name and arguments within the following XML format:
50
+ <tool_call>{function-name}<arg_key>{arg-key-1}</arg_key><arg_value>{arg-value-1}</arg_value><arg_key>{arg-key-2}</arg_key><arg_value>{arg-value-2}</arg_value>...</tool_call>{%- endif -%}
51
+ {%- macro emit_image() -%}<|begin_of_image|><|image|><|end_of_image|>{%- endmacro -%}
52
+ {%- macro emit_video() -%}<|begin_of_video|><|video|><|end_of_video|>{%- endmacro -%}
53
+ {%- macro emit_audio() -%}<|begin_of_audio|><|end_of_audio|>{%- endmacro -%}
54
+ {%- macro visible_text(content) -%}
55
+ {%- if content is string -%}
56
+ {{- content -}}
57
+ {%- elif content is iterable and content is not mapping -%}
58
+ {%- for item in content -%}
59
+ {%- if item is mapping and item.type == 'text' -%}
60
+ {{- item.text -}}
61
+ {%- elif item is string -%}
62
+ {{- item -}}
63
+ {%- elif item is mapping and item.type in ['image', 'image_url'] -%}
64
+ {{- emit_image() -}}
65
+ {%- elif item is mapping and item.type in ['video', 'video_url'] -%}
66
+ {{- emit_video() -}}
67
+ {%- elif item is mapping and item.type in ['audio', 'audio_url', 'input_audio'] -%}
68
+ {{- emit_audio() -}}
69
+ {%- endif -%}
70
+ {%- endfor -%}
71
+ {%- elif content is not none -%}
72
+ {{- content -}}
73
+ {%- endif -%}
74
+ {%- endmacro -%}
75
+ {%- macro tool_response(text) -%}
76
+ {{- '<tool_response>' + text + '</tool_response>' -}}
77
+ {%- endmacro -%}
78
+ {%- macro render_tool_response(m) -%}
79
+ {%- if m.content is string -%}
80
+ {{- tool_response(m.content) -}}
81
+ {%- elif m.content and m.content is not mapping and m.content.0.type == "tool_reference" -%}
82
+ {{- tool_references_to_response(m.content) -}}
83
+ {%- elif is_list_of_outputs(m) -%}
84
+ {%- for tr in m.content -%}
85
+ {%- if tr.output is iterable and tr.output is not string and tr.output is not mapping and tr.output and tr.output.0.type == "tool_reference" -%}
86
+ {{- tool_references_to_response(tr.output) -}}
87
+ {%- else -%}
88
+ {{- tool_response(visible_text(tr.output)) -}}
89
+ {%- endif -%}
90
+ {%- endfor -%}
91
+ {%- else -%}
92
+ {{- tool_response(visible_text(m.content)) -}}
93
+ {%- endif -%}
94
+ {%- endmacro -%}
95
+ {%- macro id_of(obj) -%}
96
+ {%- if obj.tool_call_id -%}
97
+ {{- obj.tool_call_id -}}
98
+ {%- elif obj.id -%}
99
+ {{- obj.id -}}
100
+ {%- endif -%}
101
+ {%- endmacro -%}
102
+ {%- macro is_list_of_outputs(m) -%}
103
+ {%- if m.content and m.content.0.output is defined -%}1{%- endif -%}
104
+ {%- endmacro -%}
105
+ {%- macro has_dup_tool_result_id(lo, hi, target) -%}
106
+ {%- set ns_cnt = namespace(n=0) -%}
107
+ {%- for k in range(lo, hi + 1) -%}
108
+ {%- set m = messages[k] -%}
109
+ {%- if is_list_of_outputs(m) -%}
110
+ {%- for entry in m.content -%}
111
+ {%- if id_of(entry) == target -%}
112
+ {%- set ns_cnt.n = ns_cnt.n + 1 -%}
113
+ {%- endif -%}
114
+ {%- endfor -%}
115
+ {%- elif id_of(m) == target -%}
116
+ {%- set ns_cnt.n = ns_cnt.n + 1 -%}
117
+ {%- endif -%}
118
+ {%- if ns_cnt.n > 1 -%}{%- break -%}{%- endif -%}
119
+ {%- endfor -%}
120
+ {%- if ns_cnt.n > 1 -%}1{%- endif -%}
121
+ {%- endmacro -%}
122
+ {%- macro tc_id_exists(tcs, target) -%}
123
+ {%- set ns_f = namespace(found=false) -%}
124
+ {%- for tc in tcs -%}
125
+ {%- if id_of(tc) == target -%}
126
+ {%- set ns_f.found = true -%}
127
+ {%- break -%}
128
+ {%- endif -%}
129
+ {%- endfor -%}
130
+ {%- if ns_f.found -%}1{%- endif -%}
131
+ {%- endmacro -%}
132
+ {%- set ns = namespace(last_user_index=-1) -%}
133
+ {%- for m in messages %}
134
+ {%- if m.role == 'user' %}
135
+ {%- set ns.last_user_index = loop.index0 -%}
136
+ {%- endif %}
137
+ {%- endfor %}
138
+ {%- for m in messages -%}
139
+ {%- if m.role == 'user' -%}<|user|>{{ visible_text(m.content) }}
140
+ {%- elif m.role == 'assistant' -%}
141
+ <|assistant|>
142
+ {%- set content = visible_text(m.content) %}
143
+ {%- if m.reasoning_content is string %}
144
+ {%- set reasoning_content = m.reasoning_content %}
145
+ {%- elif '</think>' in content %}
146
+ {%- set reasoning_content = content.split('</think>')[0].split('<think>')[-1] %}
147
+ {%- set content = content.split('</think>')[-1] %}
148
+ {%- endif %}
149
+ {%- if (not clear_thinking or loop.index0 > ns.last_user_index) and reasoning_content is defined -%}
150
+ {{ '<think>' + reasoning_content + '</think>'}}
151
+ {%- else -%}
152
+ {{ '<think></think>' }}
153
+ {%- endif -%}
154
+ {%- if content.strip() -%}
155
+ {{ content.strip() }}
156
+ {%- endif -%}
157
+ {% if m.tool_calls %}
158
+ {% for tc in m.tool_calls %}
159
+ {%- if tc.function %}
160
+ {%- set tc = tc.function %}
161
+ {%- endif %}
162
+ {{- '<tool_call>' ~ tc.name -}}
163
+ {% set _args = tc.arguments %}{% for k, v in _args.items() %}<arg_key>{{ k }}</arg_key><arg_value>{{ v | tojson(ensure_ascii=False) if v is not string else v }}</arg_value>{% endfor %}</tool_call>{% endfor %}
164
+ {% endif %}
165
+ {%- elif m.role == 'tool' -%}
166
+ {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
167
+ {{- '<|observation|>' -}}
168
+ {%- set block_start = loop.index0 -%}
169
+ {%- set ns_blk = namespace(end=block_start) -%}
170
+ {%- for j in range(block_start, messages|length) -%}
171
+ {%- if messages[j].role == 'tool' -%}
172
+ {%- set ns_blk.end = j -%}
173
+ {%- else -%}
174
+ {%- break -%}
175
+ {%- endif -%}
176
+ {%- endfor -%}
177
+ {%- set ns_a = namespace(tool_calls=none) -%}
178
+ {%- if block_start > 0 and messages[block_start - 1].role == 'assistant' and messages[block_start - 1].tool_calls -%}
179
+ {%- set ns_a.tool_calls = messages[block_start - 1].tool_calls -%}
180
+ {%- endif -%}
181
+ {%- set ns_chk = namespace(can_sort=true) -%}
182
+ {%- if not ns_a.tool_calls -%}
183
+ {%- set ns_chk.can_sort = false -%}
184
+ {%- else -%}
185
+ {%- for k in range(block_start, ns_blk.end + 1) -%}
186
+ {%- if not ns_chk.can_sort -%}{%- break -%}{%- endif -%}
187
+ {%- set m = messages[k] -%}
188
+ {%- if is_list_of_outputs(m) -%}
189
+ {%- for entry in m.content -%}
190
+ {%- if not ns_chk.can_sort -%}{%- break -%}{%- endif -%}
191
+ {%- set eid = id_of(entry) -%}
192
+ {%- if not eid -%}
193
+ {%- set ns_chk.can_sort = false -%}
194
+ {%- elif has_dup_tool_result_id(block_start, ns_blk.end, eid) -%}
195
+ {%- set ns_chk.can_sort = false -%}
196
+ {%- elif not tc_id_exists(ns_a.tool_calls, eid) -%}
197
+ {%- set ns_chk.can_sort = false -%}
198
+ {%- endif -%}
199
+ {%- endfor -%}
200
+ {%- else -%}
201
+ {%- set tk_id = id_of(m) -%}
202
+ {%- if not tk_id -%}
203
+ {%- set ns_chk.can_sort = false -%}
204
+ {%- elif has_dup_tool_result_id(block_start, ns_blk.end, tk_id) -%}
205
+ {%- set ns_chk.can_sort = false -%}
206
+ {%- elif not tc_id_exists(ns_a.tool_calls, tk_id) -%}
207
+ {%- set ns_chk.can_sort = false -%}
208
+ {%- endif -%}
209
+ {%- endif -%}
210
+ {%- endfor -%}
211
+ {%- for i in range(ns_a.tool_calls | length) -%}
212
+ {%- if not ns_chk.can_sort -%}{%- break -%}{%- endif -%}
213
+ {%- set tc_id = id_of(ns_a.tool_calls[i]) -%}
214
+ {%- if not tc_id -%}
215
+ {%- set ns_chk.can_sort = false -%}
216
+ {%- endif -%}
217
+ {%- for j in range(i + 1, ns_a.tool_calls | length) -%}
218
+ {%- if id_of(ns_a.tool_calls[j]) == tc_id -%}
219
+ {%- set ns_chk.can_sort = false -%}
220
+ {%- break -%}
221
+ {%- endif -%}
222
+ {%- endfor -%}
223
+ {%- endfor -%}
224
+ {%- endif -%}
225
+ {%- if ns_chk.can_sort -%}
226
+ {%- for tc in ns_a.tool_calls -%}
227
+ {%- set tc_id = id_of(tc) -%}
228
+ {%- for k in range(block_start, ns_blk.end + 1) -%}
229
+ {%- set m = messages[k] -%}
230
+ {%- if is_list_of_outputs(m) -%}
231
+ {%- for entry in m.content -%}
232
+ {%- set eid = id_of(entry) -%}
233
+ {%- if eid == tc_id -%}
234
+ {%- if entry.output is iterable and entry.output is not string and entry.output is not mapping and entry.output and entry.output.0.type == "tool_reference" -%}
235
+ {{- tool_references_to_response(entry.output) -}}
236
+ {%- else -%}
237
+ {{- tool_response(visible_text(entry.output)) -}}
238
+ {%- endif -%}
239
+ {%- endif -%}
240
+ {%- endfor -%}
241
+ {%- else -%}
242
+ {%- set tk_id = id_of(m) -%}
243
+ {%- if tk_id == tc_id -%}
244
+ {{- render_tool_response(m) -}}
245
+ {%- endif -%}
246
+ {%- endif -%}
247
+ {%- endfor -%}
248
+ {%- endfor -%}
249
+ {%- else -%}
250
+ {%- for k in range(block_start, ns_blk.end + 1) -%}
251
+ {{- render_tool_response(messages[k]) -}}
252
+ {%- endfor -%}
253
+ {%- endif -%}
254
+ {% endif -%}
255
+ {%- elif m.role == 'system' -%}
256
+ <|system|>{{ visible_text(m.content) }}
257
+ {%- endif -%}
258
+ {%- endfor -%}
259
+ {%- if add_generation_prompt -%}
260
+ <|assistant|>{{- '<think>' -}}
261
+ {%- endif -%}
config.json ADDED
@@ -0,0 +1,1142 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Glm5NextForConditionalGeneration"
4
+ ],
5
+ "dtype": "bfloat16",
6
+ "image_end_token_id": 154831,
7
+ "image_start_token_id": 154830,
8
+ "image_token_id": 154854,
9
+ "model_type": "glm5_next",
10
+ "quantization_config": {
11
+ "config_groups": {
12
+ "group_0": {
13
+ "format": "nvfp4-pack-quantized",
14
+ "input_activations": {
15
+ "dynamic": "local",
16
+ "group_size": 16,
17
+ "num_bits": 4,
18
+ "observer": "static_minmax",
19
+ "scale_dtype": "torch.float8_e4m3fn",
20
+ "strategy": "tensor_group",
21
+ "symmetric": true,
22
+ "type": "float"
23
+ },
24
+ "output_activations": null,
25
+ "targets": [
26
+ "re:.*\\.layers\\.(?:[3-9]|[1-3][0-9]|4[0-4])\\.mlp\\.experts\\..*(gate|up|down)_proj$"
27
+ ],
28
+ "weights": {
29
+ "dynamic": false,
30
+ "group_size": 16,
31
+ "num_bits": 4,
32
+ "observer": "memoryless_minmax",
33
+ "scale_dtype": "torch.float8_e4m3fn",
34
+ "strategy": "tensor_group",
35
+ "symmetric": true,
36
+ "type": "float"
37
+ }
38
+ },
39
+ "group_1": {
40
+ "format": "float-quantized",
41
+ "input_activations": {
42
+ "dynamic": true,
43
+ "group_size": 128,
44
+ "num_bits": 8,
45
+ "strategy": "group",
46
+ "symmetric": true,
47
+ "type": "float"
48
+ },
49
+ "output_activations": null,
50
+ "targets": [
51
+ "re:.*\\.layers\\.45\\.mlp\\.experts\\.\\d+\\.(gate_proj|up_proj|down_proj)$"
52
+ ],
53
+ "weights": {
54
+ "block_structure": [
55
+ 128,
56
+ 128
57
+ ],
58
+ "dynamic": false,
59
+ "group_size": null,
60
+ "num_bits": 8,
61
+ "observer": "memoryless_minmax",
62
+ "strategy": "block",
63
+ "symmetric": true,
64
+ "type": "float"
65
+ }
66
+ }
67
+ },
68
+ "format": "mixed-precision",
69
+ "global_compression_ratio": null,
70
+ "ignore": [
71
+ "model.visual.blocks.0.attn.qkv",
72
+ "model.visual.blocks.0.attn.proj",
73
+ "model.visual.blocks.0.mlp.gate_proj",
74
+ "model.visual.blocks.0.mlp.up_proj",
75
+ "model.visual.blocks.0.mlp.down_proj",
76
+ "model.visual.blocks.1.attn.qkv",
77
+ "model.visual.blocks.1.attn.proj",
78
+ "model.visual.blocks.1.mlp.gate_proj",
79
+ "model.visual.blocks.1.mlp.up_proj",
80
+ "model.visual.blocks.1.mlp.down_proj",
81
+ "model.visual.blocks.2.attn.qkv",
82
+ "model.visual.blocks.2.attn.proj",
83
+ "model.visual.blocks.2.mlp.gate_proj",
84
+ "model.visual.blocks.2.mlp.up_proj",
85
+ "model.visual.blocks.2.mlp.down_proj",
86
+ "model.visual.blocks.3.attn.qkv",
87
+ "model.visual.blocks.3.attn.proj",
88
+ "model.visual.blocks.3.mlp.gate_proj",
89
+ "model.visual.blocks.3.mlp.up_proj",
90
+ "model.visual.blocks.3.mlp.down_proj",
91
+ "model.visual.blocks.4.attn.qkv",
92
+ "model.visual.blocks.4.attn.proj",
93
+ "model.visual.blocks.4.mlp.gate_proj",
94
+ "model.visual.blocks.4.mlp.up_proj",
95
+ "model.visual.blocks.4.mlp.down_proj",
96
+ "model.visual.blocks.5.attn.qkv",
97
+ "model.visual.blocks.5.attn.proj",
98
+ "model.visual.blocks.5.mlp.gate_proj",
99
+ "model.visual.blocks.5.mlp.up_proj",
100
+ "model.visual.blocks.5.mlp.down_proj",
101
+ "model.visual.blocks.6.attn.qkv",
102
+ "model.visual.blocks.6.attn.proj",
103
+ "model.visual.blocks.6.mlp.gate_proj",
104
+ "model.visual.blocks.6.mlp.up_proj",
105
+ "model.visual.blocks.6.mlp.down_proj",
106
+ "model.visual.blocks.7.attn.qkv",
107
+ "model.visual.blocks.7.attn.proj",
108
+ "model.visual.blocks.7.mlp.gate_proj",
109
+ "model.visual.blocks.7.mlp.up_proj",
110
+ "model.visual.blocks.7.mlp.down_proj",
111
+ "model.visual.blocks.8.attn.qkv",
112
+ "model.visual.blocks.8.attn.proj",
113
+ "model.visual.blocks.8.mlp.gate_proj",
114
+ "model.visual.blocks.8.mlp.up_proj",
115
+ "model.visual.blocks.8.mlp.down_proj",
116
+ "model.visual.blocks.9.attn.qkv",
117
+ "model.visual.blocks.9.attn.proj",
118
+ "model.visual.blocks.9.mlp.gate_proj",
119
+ "model.visual.blocks.9.mlp.up_proj",
120
+ "model.visual.blocks.9.mlp.down_proj",
121
+ "model.visual.blocks.10.attn.qkv",
122
+ "model.visual.blocks.10.attn.proj",
123
+ "model.visual.blocks.10.mlp.gate_proj",
124
+ "model.visual.blocks.10.mlp.up_proj",
125
+ "model.visual.blocks.10.mlp.down_proj",
126
+ "model.visual.blocks.11.attn.qkv",
127
+ "model.visual.blocks.11.attn.proj",
128
+ "model.visual.blocks.11.mlp.gate_proj",
129
+ "model.visual.blocks.11.mlp.up_proj",
130
+ "model.visual.blocks.11.mlp.down_proj",
131
+ "model.visual.blocks.12.attn.qkv",
132
+ "model.visual.blocks.12.attn.proj",
133
+ "model.visual.blocks.12.mlp.gate_proj",
134
+ "model.visual.blocks.12.mlp.up_proj",
135
+ "model.visual.blocks.12.mlp.down_proj",
136
+ "model.visual.blocks.13.attn.qkv",
137
+ "model.visual.blocks.13.attn.proj",
138
+ "model.visual.blocks.13.mlp.gate_proj",
139
+ "model.visual.blocks.13.mlp.up_proj",
140
+ "model.visual.blocks.13.mlp.down_proj",
141
+ "model.visual.blocks.14.attn.qkv",
142
+ "model.visual.blocks.14.attn.proj",
143
+ "model.visual.blocks.14.mlp.gate_proj",
144
+ "model.visual.blocks.14.mlp.up_proj",
145
+ "model.visual.blocks.14.mlp.down_proj",
146
+ "model.visual.blocks.15.attn.qkv",
147
+ "model.visual.blocks.15.attn.proj",
148
+ "model.visual.blocks.15.mlp.gate_proj",
149
+ "model.visual.blocks.15.mlp.up_proj",
150
+ "model.visual.blocks.15.mlp.down_proj",
151
+ "model.visual.blocks.16.attn.qkv",
152
+ "model.visual.blocks.16.attn.proj",
153
+ "model.visual.blocks.16.mlp.gate_proj",
154
+ "model.visual.blocks.16.mlp.up_proj",
155
+ "model.visual.blocks.16.mlp.down_proj",
156
+ "model.visual.blocks.17.attn.qkv",
157
+ "model.visual.blocks.17.attn.proj",
158
+ "model.visual.blocks.17.mlp.gate_proj",
159
+ "model.visual.blocks.17.mlp.up_proj",
160
+ "model.visual.blocks.17.mlp.down_proj",
161
+ "model.visual.blocks.18.attn.qkv",
162
+ "model.visual.blocks.18.attn.proj",
163
+ "model.visual.blocks.18.mlp.gate_proj",
164
+ "model.visual.blocks.18.mlp.up_proj",
165
+ "model.visual.blocks.18.mlp.down_proj",
166
+ "model.visual.blocks.19.attn.qkv",
167
+ "model.visual.blocks.19.attn.proj",
168
+ "model.visual.blocks.19.mlp.gate_proj",
169
+ "model.visual.blocks.19.mlp.up_proj",
170
+ "model.visual.blocks.19.mlp.down_proj",
171
+ "model.visual.blocks.20.attn.qkv",
172
+ "model.visual.blocks.20.attn.proj",
173
+ "model.visual.blocks.20.mlp.gate_proj",
174
+ "model.visual.blocks.20.mlp.up_proj",
175
+ "model.visual.blocks.20.mlp.down_proj",
176
+ "model.visual.blocks.21.attn.qkv",
177
+ "model.visual.blocks.21.attn.proj",
178
+ "model.visual.blocks.21.mlp.gate_proj",
179
+ "model.visual.blocks.21.mlp.up_proj",
180
+ "model.visual.blocks.21.mlp.down_proj",
181
+ "model.visual.blocks.22.attn.qkv",
182
+ "model.visual.blocks.22.attn.proj",
183
+ "model.visual.blocks.22.mlp.gate_proj",
184
+ "model.visual.blocks.22.mlp.up_proj",
185
+ "model.visual.blocks.22.mlp.down_proj",
186
+ "model.visual.blocks.23.attn.qkv",
187
+ "model.visual.blocks.23.attn.proj",
188
+ "model.visual.blocks.23.mlp.gate_proj",
189
+ "model.visual.blocks.23.mlp.up_proj",
190
+ "model.visual.blocks.23.mlp.down_proj",
191
+ "model.visual.merger.proj",
192
+ "model.visual.merger.gate_proj",
193
+ "model.visual.merger.up_proj",
194
+ "model.visual.merger.down_proj",
195
+ "model.language_model.layers.0.self_attn.q_proj",
196
+ "model.language_model.layers.0.self_attn.k_proj",
197
+ "model.language_model.layers.0.self_attn.v_proj",
198
+ "model.language_model.layers.0.self_attn.forget_gate",
199
+ "model.language_model.layers.0.self_attn.forget_gate.f_a_proj",
200
+ "model.language_model.layers.0.self_attn.forget_gate.f_b_proj",
201
+ "model.language_model.layers.0.self_attn.b_proj",
202
+ "model.language_model.layers.0.self_attn.g_a_proj",
203
+ "model.language_model.layers.0.self_attn.g_b_proj",
204
+ "model.language_model.layers.0.self_attn.o_norm",
205
+ "model.language_model.layers.0.self_attn.o_proj",
206
+ "model.language_model.layers.0.mlp.gate_proj",
207
+ "model.language_model.layers.0.mlp.up_proj",
208
+ "model.language_model.layers.0.mlp.down_proj",
209
+ "model.language_model.layers.1.self_attn.q_proj",
210
+ "model.language_model.layers.1.self_attn.k_proj",
211
+ "model.language_model.layers.1.self_attn.v_proj",
212
+ "model.language_model.layers.1.self_attn.forget_gate",
213
+ "model.language_model.layers.1.self_attn.forget_gate.f_a_proj",
214
+ "model.language_model.layers.1.self_attn.forget_gate.f_b_proj",
215
+ "model.language_model.layers.1.self_attn.b_proj",
216
+ "model.language_model.layers.1.self_attn.g_a_proj",
217
+ "model.language_model.layers.1.self_attn.g_b_proj",
218
+ "model.language_model.layers.1.self_attn.o_norm",
219
+ "model.language_model.layers.1.self_attn.o_proj",
220
+ "model.language_model.layers.1.mlp.gate_proj",
221
+ "model.language_model.layers.1.mlp.up_proj",
222
+ "model.language_model.layers.1.mlp.down_proj",
223
+ "model.language_model.layers.2.self_attn.q_proj",
224
+ "model.language_model.layers.2.self_attn.k_proj",
225
+ "model.language_model.layers.2.self_attn.v_proj",
226
+ "model.language_model.layers.2.self_attn.forget_gate",
227
+ "model.language_model.layers.2.self_attn.forget_gate.f_a_proj",
228
+ "model.language_model.layers.2.self_attn.forget_gate.f_b_proj",
229
+ "model.language_model.layers.2.self_attn.b_proj",
230
+ "model.language_model.layers.2.self_attn.g_a_proj",
231
+ "model.language_model.layers.2.self_attn.g_b_proj",
232
+ "model.language_model.layers.2.self_attn.o_norm",
233
+ "model.language_model.layers.2.self_attn.o_proj",
234
+ "model.language_model.layers.2.mlp.gate_proj",
235
+ "model.language_model.layers.2.mlp.up_proj",
236
+ "model.language_model.layers.2.mlp.down_proj",
237
+ "model.language_model.layers.3.self_attn.q_a_proj",
238
+ "model.language_model.layers.3.self_attn.q_b_proj",
239
+ "model.language_model.layers.3.self_attn.kv_a_proj_with_mqa",
240
+ "model.language_model.layers.3.self_attn.kv_b_proj",
241
+ "model.language_model.layers.3.self_attn.o_proj",
242
+ "model.language_model.layers.3.self_attn.indexer.wq_b",
243
+ "model.language_model.layers.3.self_attn.indexer.wk",
244
+ "model.language_model.layers.3.self_attn.indexer.weights_proj",
245
+ "model.language_model.layers.3.mlp.gate",
246
+ "model.language_model.layers.3.mlp.shared_experts.gate_proj",
247
+ "model.language_model.layers.3.mlp.shared_experts.up_proj",
248
+ "model.language_model.layers.3.mlp.shared_experts.down_proj",
249
+ "model.language_model.layers.4.self_attn.q_proj",
250
+ "model.language_model.layers.4.self_attn.k_proj",
251
+ "model.language_model.layers.4.self_attn.v_proj",
252
+ "model.language_model.layers.4.self_attn.forget_gate",
253
+ "model.language_model.layers.4.self_attn.forget_gate.f_a_proj",
254
+ "model.language_model.layers.4.self_attn.forget_gate.f_b_proj",
255
+ "model.language_model.layers.4.self_attn.b_proj",
256
+ "model.language_model.layers.4.self_attn.g_a_proj",
257
+ "model.language_model.layers.4.self_attn.g_b_proj",
258
+ "model.language_model.layers.4.self_attn.o_norm",
259
+ "model.language_model.layers.4.self_attn.o_proj",
260
+ "model.language_model.layers.4.mlp.gate",
261
+ "model.language_model.layers.4.mlp.shared_experts.gate_proj",
262
+ "model.language_model.layers.4.mlp.shared_experts.up_proj",
263
+ "model.language_model.layers.4.mlp.shared_experts.down_proj",
264
+ "model.language_model.layers.5.self_attn.q_proj",
265
+ "model.language_model.layers.5.self_attn.k_proj",
266
+ "model.language_model.layers.5.self_attn.v_proj",
267
+ "model.language_model.layers.5.self_attn.forget_gate",
268
+ "model.language_model.layers.5.self_attn.forget_gate.f_a_proj",
269
+ "model.language_model.layers.5.self_attn.forget_gate.f_b_proj",
270
+ "model.language_model.layers.5.self_attn.b_proj",
271
+ "model.language_model.layers.5.self_attn.g_a_proj",
272
+ "model.language_model.layers.5.self_attn.g_b_proj",
273
+ "model.language_model.layers.5.self_attn.o_norm",
274
+ "model.language_model.layers.5.self_attn.o_proj",
275
+ "model.language_model.layers.5.mlp.gate",
276
+ "model.language_model.layers.5.mlp.shared_experts.gate_proj",
277
+ "model.language_model.layers.5.mlp.shared_experts.up_proj",
278
+ "model.language_model.layers.5.mlp.shared_experts.down_proj",
279
+ "model.language_model.layers.6.self_attn.q_proj",
280
+ "model.language_model.layers.6.self_attn.k_proj",
281
+ "model.language_model.layers.6.self_attn.v_proj",
282
+ "model.language_model.layers.6.self_attn.forget_gate",
283
+ "model.language_model.layers.6.self_attn.forget_gate.f_a_proj",
284
+ "model.language_model.layers.6.self_attn.forget_gate.f_b_proj",
285
+ "model.language_model.layers.6.self_attn.b_proj",
286
+ "model.language_model.layers.6.self_attn.g_a_proj",
287
+ "model.language_model.layers.6.self_attn.g_b_proj",
288
+ "model.language_model.layers.6.self_attn.o_norm",
289
+ "model.language_model.layers.6.self_attn.o_proj",
290
+ "model.language_model.layers.6.mlp.gate",
291
+ "model.language_model.layers.6.mlp.shared_experts.gate_proj",
292
+ "model.language_model.layers.6.mlp.shared_experts.up_proj",
293
+ "model.language_model.layers.6.mlp.shared_experts.down_proj",
294
+ "model.language_model.layers.7.self_attn.q_a_proj",
295
+ "model.language_model.layers.7.self_attn.q_b_proj",
296
+ "model.language_model.layers.7.self_attn.kv_a_proj_with_mqa",
297
+ "model.language_model.layers.7.self_attn.kv_b_proj",
298
+ "model.language_model.layers.7.self_attn.o_proj",
299
+ "model.language_model.layers.7.self_attn.indexer.wq_b",
300
+ "model.language_model.layers.7.self_attn.indexer.wk",
301
+ "model.language_model.layers.7.self_attn.indexer.weights_proj",
302
+ "model.language_model.layers.7.mlp.gate",
303
+ "model.language_model.layers.7.mlp.shared_experts.gate_proj",
304
+ "model.language_model.layers.7.mlp.shared_experts.up_proj",
305
+ "model.language_model.layers.7.mlp.shared_experts.down_proj",
306
+ "model.language_model.layers.8.self_attn.q_proj",
307
+ "model.language_model.layers.8.self_attn.k_proj",
308
+ "model.language_model.layers.8.self_attn.v_proj",
309
+ "model.language_model.layers.8.self_attn.forget_gate",
310
+ "model.language_model.layers.8.self_attn.forget_gate.f_a_proj",
311
+ "model.language_model.layers.8.self_attn.forget_gate.f_b_proj",
312
+ "model.language_model.layers.8.self_attn.b_proj",
313
+ "model.language_model.layers.8.self_attn.g_a_proj",
314
+ "model.language_model.layers.8.self_attn.g_b_proj",
315
+ "model.language_model.layers.8.self_attn.o_norm",
316
+ "model.language_model.layers.8.self_attn.o_proj",
317
+ "model.language_model.layers.8.mlp.gate",
318
+ "model.language_model.layers.8.mlp.shared_experts.gate_proj",
319
+ "model.language_model.layers.8.mlp.shared_experts.up_proj",
320
+ "model.language_model.layers.8.mlp.shared_experts.down_proj",
321
+ "model.language_model.layers.9.self_attn.q_proj",
322
+ "model.language_model.layers.9.self_attn.k_proj",
323
+ "model.language_model.layers.9.self_attn.v_proj",
324
+ "model.language_model.layers.9.self_attn.forget_gate",
325
+ "model.language_model.layers.9.self_attn.forget_gate.f_a_proj",
326
+ "model.language_model.layers.9.self_attn.forget_gate.f_b_proj",
327
+ "model.language_model.layers.9.self_attn.b_proj",
328
+ "model.language_model.layers.9.self_attn.g_a_proj",
329
+ "model.language_model.layers.9.self_attn.g_b_proj",
330
+ "model.language_model.layers.9.self_attn.o_norm",
331
+ "model.language_model.layers.9.self_attn.o_proj",
332
+ "model.language_model.layers.9.mlp.gate",
333
+ "model.language_model.layers.9.mlp.shared_experts.gate_proj",
334
+ "model.language_model.layers.9.mlp.shared_experts.up_proj",
335
+ "model.language_model.layers.9.mlp.shared_experts.down_proj",
336
+ "model.language_model.layers.10.self_attn.q_proj",
337
+ "model.language_model.layers.10.self_attn.k_proj",
338
+ "model.language_model.layers.10.self_attn.v_proj",
339
+ "model.language_model.layers.10.self_attn.forget_gate",
340
+ "model.language_model.layers.10.self_attn.forget_gate.f_a_proj",
341
+ "model.language_model.layers.10.self_attn.forget_gate.f_b_proj",
342
+ "model.language_model.layers.10.self_attn.b_proj",
343
+ "model.language_model.layers.10.self_attn.g_a_proj",
344
+ "model.language_model.layers.10.self_attn.g_b_proj",
345
+ "model.language_model.layers.10.self_attn.o_norm",
346
+ "model.language_model.layers.10.self_attn.o_proj",
347
+ "model.language_model.layers.10.mlp.gate",
348
+ "model.language_model.layers.10.mlp.shared_experts.gate_proj",
349
+ "model.language_model.layers.10.mlp.shared_experts.up_proj",
350
+ "model.language_model.layers.10.mlp.shared_experts.down_proj",
351
+ "model.language_model.layers.11.self_attn.q_a_proj",
352
+ "model.language_model.layers.11.self_attn.q_b_proj",
353
+ "model.language_model.layers.11.self_attn.kv_a_proj_with_mqa",
354
+ "model.language_model.layers.11.self_attn.kv_b_proj",
355
+ "model.language_model.layers.11.self_attn.o_proj",
356
+ "model.language_model.layers.11.self_attn.indexer.wq_b",
357
+ "model.language_model.layers.11.self_attn.indexer.wk",
358
+ "model.language_model.layers.11.self_attn.indexer.weights_proj",
359
+ "model.language_model.layers.11.mlp.gate",
360
+ "model.language_model.layers.11.mlp.shared_experts.gate_proj",
361
+ "model.language_model.layers.11.mlp.shared_experts.up_proj",
362
+ "model.language_model.layers.11.mlp.shared_experts.down_proj",
363
+ "model.language_model.layers.12.self_attn.q_proj",
364
+ "model.language_model.layers.12.self_attn.k_proj",
365
+ "model.language_model.layers.12.self_attn.v_proj",
366
+ "model.language_model.layers.12.self_attn.forget_gate",
367
+ "model.language_model.layers.12.self_attn.forget_gate.f_a_proj",
368
+ "model.language_model.layers.12.self_attn.forget_gate.f_b_proj",
369
+ "model.language_model.layers.12.self_attn.b_proj",
370
+ "model.language_model.layers.12.self_attn.g_a_proj",
371
+ "model.language_model.layers.12.self_attn.g_b_proj",
372
+ "model.language_model.layers.12.self_attn.o_norm",
373
+ "model.language_model.layers.12.self_attn.o_proj",
374
+ "model.language_model.layers.12.mlp.gate",
375
+ "model.language_model.layers.12.mlp.shared_experts.gate_proj",
376
+ "model.language_model.layers.12.mlp.shared_experts.up_proj",
377
+ "model.language_model.layers.12.mlp.shared_experts.down_proj",
378
+ "model.language_model.layers.13.self_attn.q_proj",
379
+ "model.language_model.layers.13.self_attn.k_proj",
380
+ "model.language_model.layers.13.self_attn.v_proj",
381
+ "model.language_model.layers.13.self_attn.forget_gate",
382
+ "model.language_model.layers.13.self_attn.forget_gate.f_a_proj",
383
+ "model.language_model.layers.13.self_attn.forget_gate.f_b_proj",
384
+ "model.language_model.layers.13.self_attn.b_proj",
385
+ "model.language_model.layers.13.self_attn.g_a_proj",
386
+ "model.language_model.layers.13.self_attn.g_b_proj",
387
+ "model.language_model.layers.13.self_attn.o_norm",
388
+ "model.language_model.layers.13.self_attn.o_proj",
389
+ "model.language_model.layers.13.mlp.gate",
390
+ "model.language_model.layers.13.mlp.shared_experts.gate_proj",
391
+ "model.language_model.layers.13.mlp.shared_experts.up_proj",
392
+ "model.language_model.layers.13.mlp.shared_experts.down_proj",
393
+ "model.language_model.layers.14.self_attn.q_proj",
394
+ "model.language_model.layers.14.self_attn.k_proj",
395
+ "model.language_model.layers.14.self_attn.v_proj",
396
+ "model.language_model.layers.14.self_attn.forget_gate",
397
+ "model.language_model.layers.14.self_attn.forget_gate.f_a_proj",
398
+ "model.language_model.layers.14.self_attn.forget_gate.f_b_proj",
399
+ "model.language_model.layers.14.self_attn.b_proj",
400
+ "model.language_model.layers.14.self_attn.g_a_proj",
401
+ "model.language_model.layers.14.self_attn.g_b_proj",
402
+ "model.language_model.layers.14.self_attn.o_norm",
403
+ "model.language_model.layers.14.self_attn.o_proj",
404
+ "model.language_model.layers.14.mlp.gate",
405
+ "model.language_model.layers.14.mlp.shared_experts.gate_proj",
406
+ "model.language_model.layers.14.mlp.shared_experts.up_proj",
407
+ "model.language_model.layers.14.mlp.shared_experts.down_proj",
408
+ "model.language_model.layers.15.self_attn.q_a_proj",
409
+ "model.language_model.layers.15.self_attn.q_b_proj",
410
+ "model.language_model.layers.15.self_attn.kv_a_proj_with_mqa",
411
+ "model.language_model.layers.15.self_attn.kv_b_proj",
412
+ "model.language_model.layers.15.self_attn.o_proj",
413
+ "model.language_model.layers.15.self_attn.indexer.wq_b",
414
+ "model.language_model.layers.15.self_attn.indexer.wk",
415
+ "model.language_model.layers.15.self_attn.indexer.weights_proj",
416
+ "model.language_model.layers.15.mlp.gate",
417
+ "model.language_model.layers.15.mlp.shared_experts.gate_proj",
418
+ "model.language_model.layers.15.mlp.shared_experts.up_proj",
419
+ "model.language_model.layers.15.mlp.shared_experts.down_proj",
420
+ "model.language_model.layers.16.self_attn.q_proj",
421
+ "model.language_model.layers.16.self_attn.k_proj",
422
+ "model.language_model.layers.16.self_attn.v_proj",
423
+ "model.language_model.layers.16.self_attn.forget_gate",
424
+ "model.language_model.layers.16.self_attn.forget_gate.f_a_proj",
425
+ "model.language_model.layers.16.self_attn.forget_gate.f_b_proj",
426
+ "model.language_model.layers.16.self_attn.b_proj",
427
+ "model.language_model.layers.16.self_attn.g_a_proj",
428
+ "model.language_model.layers.16.self_attn.g_b_proj",
429
+ "model.language_model.layers.16.self_attn.o_norm",
430
+ "model.language_model.layers.16.self_attn.o_proj",
431
+ "model.language_model.layers.16.mlp.gate",
432
+ "model.language_model.layers.16.mlp.shared_experts.gate_proj",
433
+ "model.language_model.layers.16.mlp.shared_experts.up_proj",
434
+ "model.language_model.layers.16.mlp.shared_experts.down_proj",
435
+ "model.language_model.layers.17.self_attn.q_proj",
436
+ "model.language_model.layers.17.self_attn.k_proj",
437
+ "model.language_model.layers.17.self_attn.v_proj",
438
+ "model.language_model.layers.17.self_attn.forget_gate",
439
+ "model.language_model.layers.17.self_attn.forget_gate.f_a_proj",
440
+ "model.language_model.layers.17.self_attn.forget_gate.f_b_proj",
441
+ "model.language_model.layers.17.self_attn.b_proj",
442
+ "model.language_model.layers.17.self_attn.g_a_proj",
443
+ "model.language_model.layers.17.self_attn.g_b_proj",
444
+ "model.language_model.layers.17.self_attn.o_norm",
445
+ "model.language_model.layers.17.self_attn.o_proj",
446
+ "model.language_model.layers.17.mlp.gate",
447
+ "model.language_model.layers.17.mlp.shared_experts.gate_proj",
448
+ "model.language_model.layers.17.mlp.shared_experts.up_proj",
449
+ "model.language_model.layers.17.mlp.shared_experts.down_proj",
450
+ "model.language_model.layers.18.self_attn.q_proj",
451
+ "model.language_model.layers.18.self_attn.k_proj",
452
+ "model.language_model.layers.18.self_attn.v_proj",
453
+ "model.language_model.layers.18.self_attn.forget_gate",
454
+ "model.language_model.layers.18.self_attn.forget_gate.f_a_proj",
455
+ "model.language_model.layers.18.self_attn.forget_gate.f_b_proj",
456
+ "model.language_model.layers.18.self_attn.b_proj",
457
+ "model.language_model.layers.18.self_attn.g_a_proj",
458
+ "model.language_model.layers.18.self_attn.g_b_proj",
459
+ "model.language_model.layers.18.self_attn.o_norm",
460
+ "model.language_model.layers.18.self_attn.o_proj",
461
+ "model.language_model.layers.18.mlp.gate",
462
+ "model.language_model.layers.18.mlp.shared_experts.gate_proj",
463
+ "model.language_model.layers.18.mlp.shared_experts.up_proj",
464
+ "model.language_model.layers.18.mlp.shared_experts.down_proj",
465
+ "model.language_model.layers.19.self_attn.q_a_proj",
466
+ "model.language_model.layers.19.self_attn.q_b_proj",
467
+ "model.language_model.layers.19.self_attn.kv_a_proj_with_mqa",
468
+ "model.language_model.layers.19.self_attn.kv_b_proj",
469
+ "model.language_model.layers.19.self_attn.o_proj",
470
+ "model.language_model.layers.19.self_attn.indexer.wq_b",
471
+ "model.language_model.layers.19.self_attn.indexer.wk",
472
+ "model.language_model.layers.19.self_attn.indexer.weights_proj",
473
+ "model.language_model.layers.19.mlp.gate",
474
+ "model.language_model.layers.19.mlp.shared_experts.gate_proj",
475
+ "model.language_model.layers.19.mlp.shared_experts.up_proj",
476
+ "model.language_model.layers.19.mlp.shared_experts.down_proj",
477
+ "model.language_model.layers.20.self_attn.q_proj",
478
+ "model.language_model.layers.20.self_attn.k_proj",
479
+ "model.language_model.layers.20.self_attn.v_proj",
480
+ "model.language_model.layers.20.self_attn.forget_gate",
481
+ "model.language_model.layers.20.self_attn.forget_gate.f_a_proj",
482
+ "model.language_model.layers.20.self_attn.forget_gate.f_b_proj",
483
+ "model.language_model.layers.20.self_attn.b_proj",
484
+ "model.language_model.layers.20.self_attn.g_a_proj",
485
+ "model.language_model.layers.20.self_attn.g_b_proj",
486
+ "model.language_model.layers.20.self_attn.o_norm",
487
+ "model.language_model.layers.20.self_attn.o_proj",
488
+ "model.language_model.layers.20.mlp.gate",
489
+ "model.language_model.layers.20.mlp.shared_experts.gate_proj",
490
+ "model.language_model.layers.20.mlp.shared_experts.up_proj",
491
+ "model.language_model.layers.20.mlp.shared_experts.down_proj",
492
+ "model.language_model.layers.21.self_attn.q_proj",
493
+ "model.language_model.layers.21.self_attn.k_proj",
494
+ "model.language_model.layers.21.self_attn.v_proj",
495
+ "model.language_model.layers.21.self_attn.forget_gate",
496
+ "model.language_model.layers.21.self_attn.forget_gate.f_a_proj",
497
+ "model.language_model.layers.21.self_attn.forget_gate.f_b_proj",
498
+ "model.language_model.layers.21.self_attn.b_proj",
499
+ "model.language_model.layers.21.self_attn.g_a_proj",
500
+ "model.language_model.layers.21.self_attn.g_b_proj",
501
+ "model.language_model.layers.21.self_attn.o_norm",
502
+ "model.language_model.layers.21.self_attn.o_proj",
503
+ "model.language_model.layers.21.mlp.gate",
504
+ "model.language_model.layers.21.mlp.shared_experts.gate_proj",
505
+ "model.language_model.layers.21.mlp.shared_experts.up_proj",
506
+ "model.language_model.layers.21.mlp.shared_experts.down_proj",
507
+ "model.language_model.layers.22.self_attn.q_proj",
508
+ "model.language_model.layers.22.self_attn.k_proj",
509
+ "model.language_model.layers.22.self_attn.v_proj",
510
+ "model.language_model.layers.22.self_attn.forget_gate",
511
+ "model.language_model.layers.22.self_attn.forget_gate.f_a_proj",
512
+ "model.language_model.layers.22.self_attn.forget_gate.f_b_proj",
513
+ "model.language_model.layers.22.self_attn.b_proj",
514
+ "model.language_model.layers.22.self_attn.g_a_proj",
515
+ "model.language_model.layers.22.self_attn.g_b_proj",
516
+ "model.language_model.layers.22.self_attn.o_norm",
517
+ "model.language_model.layers.22.self_attn.o_proj",
518
+ "model.language_model.layers.22.mlp.gate",
519
+ "model.language_model.layers.22.mlp.shared_experts.gate_proj",
520
+ "model.language_model.layers.22.mlp.shared_experts.up_proj",
521
+ "model.language_model.layers.22.mlp.shared_experts.down_proj",
522
+ "model.language_model.layers.23.self_attn.q_a_proj",
523
+ "model.language_model.layers.23.self_attn.q_b_proj",
524
+ "model.language_model.layers.23.self_attn.kv_a_proj_with_mqa",
525
+ "model.language_model.layers.23.self_attn.kv_b_proj",
526
+ "model.language_model.layers.23.self_attn.o_proj",
527
+ "model.language_model.layers.23.self_attn.indexer.wq_b",
528
+ "model.language_model.layers.23.self_attn.indexer.wk",
529
+ "model.language_model.layers.23.self_attn.indexer.weights_proj",
530
+ "model.language_model.layers.23.mlp.gate",
531
+ "model.language_model.layers.23.mlp.shared_experts.gate_proj",
532
+ "model.language_model.layers.23.mlp.shared_experts.up_proj",
533
+ "model.language_model.layers.23.mlp.shared_experts.down_proj",
534
+ "model.language_model.layers.24.self_attn.q_proj",
535
+ "model.language_model.layers.24.self_attn.k_proj",
536
+ "model.language_model.layers.24.self_attn.v_proj",
537
+ "model.language_model.layers.24.self_attn.forget_gate",
538
+ "model.language_model.layers.24.self_attn.forget_gate.f_a_proj",
539
+ "model.language_model.layers.24.self_attn.forget_gate.f_b_proj",
540
+ "model.language_model.layers.24.self_attn.b_proj",
541
+ "model.language_model.layers.24.self_attn.g_a_proj",
542
+ "model.language_model.layers.24.self_attn.g_b_proj",
543
+ "model.language_model.layers.24.self_attn.o_norm",
544
+ "model.language_model.layers.24.self_attn.o_proj",
545
+ "model.language_model.layers.24.mlp.gate",
546
+ "model.language_model.layers.24.mlp.shared_experts.gate_proj",
547
+ "model.language_model.layers.24.mlp.shared_experts.up_proj",
548
+ "model.language_model.layers.24.mlp.shared_experts.down_proj",
549
+ "model.language_model.layers.25.self_attn.q_proj",
550
+ "model.language_model.layers.25.self_attn.k_proj",
551
+ "model.language_model.layers.25.self_attn.v_proj",
552
+ "model.language_model.layers.25.self_attn.forget_gate",
553
+ "model.language_model.layers.25.self_attn.forget_gate.f_a_proj",
554
+ "model.language_model.layers.25.self_attn.forget_gate.f_b_proj",
555
+ "model.language_model.layers.25.self_attn.b_proj",
556
+ "model.language_model.layers.25.self_attn.g_a_proj",
557
+ "model.language_model.layers.25.self_attn.g_b_proj",
558
+ "model.language_model.layers.25.self_attn.o_norm",
559
+ "model.language_model.layers.25.self_attn.o_proj",
560
+ "model.language_model.layers.25.mlp.gate",
561
+ "model.language_model.layers.25.mlp.shared_experts.gate_proj",
562
+ "model.language_model.layers.25.mlp.shared_experts.up_proj",
563
+ "model.language_model.layers.25.mlp.shared_experts.down_proj",
564
+ "model.language_model.layers.26.self_attn.q_proj",
565
+ "model.language_model.layers.26.self_attn.k_proj",
566
+ "model.language_model.layers.26.self_attn.v_proj",
567
+ "model.language_model.layers.26.self_attn.forget_gate",
568
+ "model.language_model.layers.26.self_attn.forget_gate.f_a_proj",
569
+ "model.language_model.layers.26.self_attn.forget_gate.f_b_proj",
570
+ "model.language_model.layers.26.self_attn.b_proj",
571
+ "model.language_model.layers.26.self_attn.g_a_proj",
572
+ "model.language_model.layers.26.self_attn.g_b_proj",
573
+ "model.language_model.layers.26.self_attn.o_norm",
574
+ "model.language_model.layers.26.self_attn.o_proj",
575
+ "model.language_model.layers.26.mlp.gate",
576
+ "model.language_model.layers.26.mlp.shared_experts.gate_proj",
577
+ "model.language_model.layers.26.mlp.shared_experts.up_proj",
578
+ "model.language_model.layers.26.mlp.shared_experts.down_proj",
579
+ "model.language_model.layers.27.self_attn.q_a_proj",
580
+ "model.language_model.layers.27.self_attn.q_b_proj",
581
+ "model.language_model.layers.27.self_attn.kv_a_proj_with_mqa",
582
+ "model.language_model.layers.27.self_attn.kv_b_proj",
583
+ "model.language_model.layers.27.self_attn.o_proj",
584
+ "model.language_model.layers.27.self_attn.indexer.wq_b",
585
+ "model.language_model.layers.27.self_attn.indexer.wk",
586
+ "model.language_model.layers.27.self_attn.indexer.weights_proj",
587
+ "model.language_model.layers.27.mlp.gate",
588
+ "model.language_model.layers.27.mlp.shared_experts.gate_proj",
589
+ "model.language_model.layers.27.mlp.shared_experts.up_proj",
590
+ "model.language_model.layers.27.mlp.shared_experts.down_proj",
591
+ "model.language_model.layers.28.self_attn.q_proj",
592
+ "model.language_model.layers.28.self_attn.k_proj",
593
+ "model.language_model.layers.28.self_attn.v_proj",
594
+ "model.language_model.layers.28.self_attn.forget_gate",
595
+ "model.language_model.layers.28.self_attn.forget_gate.f_a_proj",
596
+ "model.language_model.layers.28.self_attn.forget_gate.f_b_proj",
597
+ "model.language_model.layers.28.self_attn.b_proj",
598
+ "model.language_model.layers.28.self_attn.g_a_proj",
599
+ "model.language_model.layers.28.self_attn.g_b_proj",
600
+ "model.language_model.layers.28.self_attn.o_norm",
601
+ "model.language_model.layers.28.self_attn.o_proj",
602
+ "model.language_model.layers.28.mlp.gate",
603
+ "model.language_model.layers.28.mlp.shared_experts.gate_proj",
604
+ "model.language_model.layers.28.mlp.shared_experts.up_proj",
605
+ "model.language_model.layers.28.mlp.shared_experts.down_proj",
606
+ "model.language_model.layers.29.self_attn.q_proj",
607
+ "model.language_model.layers.29.self_attn.k_proj",
608
+ "model.language_model.layers.29.self_attn.v_proj",
609
+ "model.language_model.layers.29.self_attn.forget_gate",
610
+ "model.language_model.layers.29.self_attn.forget_gate.f_a_proj",
611
+ "model.language_model.layers.29.self_attn.forget_gate.f_b_proj",
612
+ "model.language_model.layers.29.self_attn.b_proj",
613
+ "model.language_model.layers.29.self_attn.g_a_proj",
614
+ "model.language_model.layers.29.self_attn.g_b_proj",
615
+ "model.language_model.layers.29.self_attn.o_norm",
616
+ "model.language_model.layers.29.self_attn.o_proj",
617
+ "model.language_model.layers.29.mlp.gate",
618
+ "model.language_model.layers.29.mlp.shared_experts.gate_proj",
619
+ "model.language_model.layers.29.mlp.shared_experts.up_proj",
620
+ "model.language_model.layers.29.mlp.shared_experts.down_proj",
621
+ "model.language_model.layers.30.self_attn.q_proj",
622
+ "model.language_model.layers.30.self_attn.k_proj",
623
+ "model.language_model.layers.30.self_attn.v_proj",
624
+ "model.language_model.layers.30.self_attn.forget_gate",
625
+ "model.language_model.layers.30.self_attn.forget_gate.f_a_proj",
626
+ "model.language_model.layers.30.self_attn.forget_gate.f_b_proj",
627
+ "model.language_model.layers.30.self_attn.b_proj",
628
+ "model.language_model.layers.30.self_attn.g_a_proj",
629
+ "model.language_model.layers.30.self_attn.g_b_proj",
630
+ "model.language_model.layers.30.self_attn.o_norm",
631
+ "model.language_model.layers.30.self_attn.o_proj",
632
+ "model.language_model.layers.30.mlp.gate",
633
+ "model.language_model.layers.30.mlp.shared_experts.gate_proj",
634
+ "model.language_model.layers.30.mlp.shared_experts.up_proj",
635
+ "model.language_model.layers.30.mlp.shared_experts.down_proj",
636
+ "model.language_model.layers.31.self_attn.q_a_proj",
637
+ "model.language_model.layers.31.self_attn.q_b_proj",
638
+ "model.language_model.layers.31.self_attn.kv_a_proj_with_mqa",
639
+ "model.language_model.layers.31.self_attn.kv_b_proj",
640
+ "model.language_model.layers.31.self_attn.o_proj",
641
+ "model.language_model.layers.31.self_attn.indexer.wq_b",
642
+ "model.language_model.layers.31.self_attn.indexer.wk",
643
+ "model.language_model.layers.31.self_attn.indexer.weights_proj",
644
+ "model.language_model.layers.31.mlp.gate",
645
+ "model.language_model.layers.31.mlp.shared_experts.gate_proj",
646
+ "model.language_model.layers.31.mlp.shared_experts.up_proj",
647
+ "model.language_model.layers.31.mlp.shared_experts.down_proj",
648
+ "model.language_model.layers.32.self_attn.q_proj",
649
+ "model.language_model.layers.32.self_attn.k_proj",
650
+ "model.language_model.layers.32.self_attn.v_proj",
651
+ "model.language_model.layers.32.self_attn.forget_gate",
652
+ "model.language_model.layers.32.self_attn.forget_gate.f_a_proj",
653
+ "model.language_model.layers.32.self_attn.forget_gate.f_b_proj",
654
+ "model.language_model.layers.32.self_attn.b_proj",
655
+ "model.language_model.layers.32.self_attn.g_a_proj",
656
+ "model.language_model.layers.32.self_attn.g_b_proj",
657
+ "model.language_model.layers.32.self_attn.o_norm",
658
+ "model.language_model.layers.32.self_attn.o_proj",
659
+ "model.language_model.layers.32.mlp.gate",
660
+ "model.language_model.layers.32.mlp.shared_experts.gate_proj",
661
+ "model.language_model.layers.32.mlp.shared_experts.up_proj",
662
+ "model.language_model.layers.32.mlp.shared_experts.down_proj",
663
+ "model.language_model.layers.33.self_attn.q_proj",
664
+ "model.language_model.layers.33.self_attn.k_proj",
665
+ "model.language_model.layers.33.self_attn.v_proj",
666
+ "model.language_model.layers.33.self_attn.forget_gate",
667
+ "model.language_model.layers.33.self_attn.forget_gate.f_a_proj",
668
+ "model.language_model.layers.33.self_attn.forget_gate.f_b_proj",
669
+ "model.language_model.layers.33.self_attn.b_proj",
670
+ "model.language_model.layers.33.self_attn.g_a_proj",
671
+ "model.language_model.layers.33.self_attn.g_b_proj",
672
+ "model.language_model.layers.33.self_attn.o_norm",
673
+ "model.language_model.layers.33.self_attn.o_proj",
674
+ "model.language_model.layers.33.mlp.gate",
675
+ "model.language_model.layers.33.mlp.shared_experts.gate_proj",
676
+ "model.language_model.layers.33.mlp.shared_experts.up_proj",
677
+ "model.language_model.layers.33.mlp.shared_experts.down_proj",
678
+ "model.language_model.layers.34.self_attn.q_proj",
679
+ "model.language_model.layers.34.self_attn.k_proj",
680
+ "model.language_model.layers.34.self_attn.v_proj",
681
+ "model.language_model.layers.34.self_attn.forget_gate",
682
+ "model.language_model.layers.34.self_attn.forget_gate.f_a_proj",
683
+ "model.language_model.layers.34.self_attn.forget_gate.f_b_proj",
684
+ "model.language_model.layers.34.self_attn.b_proj",
685
+ "model.language_model.layers.34.self_attn.g_a_proj",
686
+ "model.language_model.layers.34.self_attn.g_b_proj",
687
+ "model.language_model.layers.34.self_attn.o_norm",
688
+ "model.language_model.layers.34.self_attn.o_proj",
689
+ "model.language_model.layers.34.mlp.gate",
690
+ "model.language_model.layers.34.mlp.shared_experts.gate_proj",
691
+ "model.language_model.layers.34.mlp.shared_experts.up_proj",
692
+ "model.language_model.layers.34.mlp.shared_experts.down_proj",
693
+ "model.language_model.layers.35.self_attn.q_a_proj",
694
+ "model.language_model.layers.35.self_attn.q_b_proj",
695
+ "model.language_model.layers.35.self_attn.kv_a_proj_with_mqa",
696
+ "model.language_model.layers.35.self_attn.kv_b_proj",
697
+ "model.language_model.layers.35.self_attn.o_proj",
698
+ "model.language_model.layers.35.self_attn.indexer.wq_b",
699
+ "model.language_model.layers.35.self_attn.indexer.wk",
700
+ "model.language_model.layers.35.self_attn.indexer.weights_proj",
701
+ "model.language_model.layers.35.mlp.gate",
702
+ "model.language_model.layers.35.mlp.shared_experts.gate_proj",
703
+ "model.language_model.layers.35.mlp.shared_experts.up_proj",
704
+ "model.language_model.layers.35.mlp.shared_experts.down_proj",
705
+ "model.language_model.layers.36.self_attn.q_proj",
706
+ "model.language_model.layers.36.self_attn.k_proj",
707
+ "model.language_model.layers.36.self_attn.v_proj",
708
+ "model.language_model.layers.36.self_attn.forget_gate",
709
+ "model.language_model.layers.36.self_attn.forget_gate.f_a_proj",
710
+ "model.language_model.layers.36.self_attn.forget_gate.f_b_proj",
711
+ "model.language_model.layers.36.self_attn.b_proj",
712
+ "model.language_model.layers.36.self_attn.g_a_proj",
713
+ "model.language_model.layers.36.self_attn.g_b_proj",
714
+ "model.language_model.layers.36.self_attn.o_norm",
715
+ "model.language_model.layers.36.self_attn.o_proj",
716
+ "model.language_model.layers.36.mlp.gate",
717
+ "model.language_model.layers.36.mlp.shared_experts.gate_proj",
718
+ "model.language_model.layers.36.mlp.shared_experts.up_proj",
719
+ "model.language_model.layers.36.mlp.shared_experts.down_proj",
720
+ "model.language_model.layers.37.self_attn.q_proj",
721
+ "model.language_model.layers.37.self_attn.k_proj",
722
+ "model.language_model.layers.37.self_attn.v_proj",
723
+ "model.language_model.layers.37.self_attn.forget_gate",
724
+ "model.language_model.layers.37.self_attn.forget_gate.f_a_proj",
725
+ "model.language_model.layers.37.self_attn.forget_gate.f_b_proj",
726
+ "model.language_model.layers.37.self_attn.b_proj",
727
+ "model.language_model.layers.37.self_attn.g_a_proj",
728
+ "model.language_model.layers.37.self_attn.g_b_proj",
729
+ "model.language_model.layers.37.self_attn.o_norm",
730
+ "model.language_model.layers.37.self_attn.o_proj",
731
+ "model.language_model.layers.37.mlp.gate",
732
+ "model.language_model.layers.37.mlp.shared_experts.gate_proj",
733
+ "model.language_model.layers.37.mlp.shared_experts.up_proj",
734
+ "model.language_model.layers.37.mlp.shared_experts.down_proj",
735
+ "model.language_model.layers.38.self_attn.q_proj",
736
+ "model.language_model.layers.38.self_attn.k_proj",
737
+ "model.language_model.layers.38.self_attn.v_proj",
738
+ "model.language_model.layers.38.self_attn.forget_gate",
739
+ "model.language_model.layers.38.self_attn.forget_gate.f_a_proj",
740
+ "model.language_model.layers.38.self_attn.forget_gate.f_b_proj",
741
+ "model.language_model.layers.38.self_attn.b_proj",
742
+ "model.language_model.layers.38.self_attn.g_a_proj",
743
+ "model.language_model.layers.38.self_attn.g_b_proj",
744
+ "model.language_model.layers.38.self_attn.o_norm",
745
+ "model.language_model.layers.38.self_attn.o_proj",
746
+ "model.language_model.layers.38.mlp.gate",
747
+ "model.language_model.layers.38.mlp.shared_experts.gate_proj",
748
+ "model.language_model.layers.38.mlp.shared_experts.up_proj",
749
+ "model.language_model.layers.38.mlp.shared_experts.down_proj",
750
+ "model.language_model.layers.39.self_attn.q_a_proj",
751
+ "model.language_model.layers.39.self_attn.q_b_proj",
752
+ "model.language_model.layers.39.self_attn.kv_a_proj_with_mqa",
753
+ "model.language_model.layers.39.self_attn.kv_b_proj",
754
+ "model.language_model.layers.39.self_attn.o_proj",
755
+ "model.language_model.layers.39.self_attn.indexer.wq_b",
756
+ "model.language_model.layers.39.self_attn.indexer.wk",
757
+ "model.language_model.layers.39.self_attn.indexer.weights_proj",
758
+ "model.language_model.layers.39.mlp.gate",
759
+ "model.language_model.layers.39.mlp.shared_experts.gate_proj",
760
+ "model.language_model.layers.39.mlp.shared_experts.up_proj",
761
+ "model.language_model.layers.39.mlp.shared_experts.down_proj",
762
+ "model.language_model.layers.40.self_attn.q_proj",
763
+ "model.language_model.layers.40.self_attn.k_proj",
764
+ "model.language_model.layers.40.self_attn.v_proj",
765
+ "model.language_model.layers.40.self_attn.forget_gate",
766
+ "model.language_model.layers.40.self_attn.forget_gate.f_a_proj",
767
+ "model.language_model.layers.40.self_attn.forget_gate.f_b_proj",
768
+ "model.language_model.layers.40.self_attn.b_proj",
769
+ "model.language_model.layers.40.self_attn.g_a_proj",
770
+ "model.language_model.layers.40.self_attn.g_b_proj",
771
+ "model.language_model.layers.40.self_attn.o_norm",
772
+ "model.language_model.layers.40.self_attn.o_proj",
773
+ "model.language_model.layers.40.mlp.gate",
774
+ "model.language_model.layers.40.mlp.shared_experts.gate_proj",
775
+ "model.language_model.layers.40.mlp.shared_experts.up_proj",
776
+ "model.language_model.layers.40.mlp.shared_experts.down_proj",
777
+ "model.language_model.layers.41.self_attn.q_proj",
778
+ "model.language_model.layers.41.self_attn.k_proj",
779
+ "model.language_model.layers.41.self_attn.v_proj",
780
+ "model.language_model.layers.41.self_attn.forget_gate",
781
+ "model.language_model.layers.41.self_attn.forget_gate.f_a_proj",
782
+ "model.language_model.layers.41.self_attn.forget_gate.f_b_proj",
783
+ "model.language_model.layers.41.self_attn.b_proj",
784
+ "model.language_model.layers.41.self_attn.g_a_proj",
785
+ "model.language_model.layers.41.self_attn.g_b_proj",
786
+ "model.language_model.layers.41.self_attn.o_norm",
787
+ "model.language_model.layers.41.self_attn.o_proj",
788
+ "model.language_model.layers.41.mlp.gate",
789
+ "model.language_model.layers.41.mlp.shared_experts.gate_proj",
790
+ "model.language_model.layers.41.mlp.shared_experts.up_proj",
791
+ "model.language_model.layers.41.mlp.shared_experts.down_proj",
792
+ "model.language_model.layers.42.self_attn.q_proj",
793
+ "model.language_model.layers.42.self_attn.k_proj",
794
+ "model.language_model.layers.42.self_attn.v_proj",
795
+ "model.language_model.layers.42.self_attn.forget_gate",
796
+ "model.language_model.layers.42.self_attn.forget_gate.f_a_proj",
797
+ "model.language_model.layers.42.self_attn.forget_gate.f_b_proj",
798
+ "model.language_model.layers.42.self_attn.b_proj",
799
+ "model.language_model.layers.42.self_attn.g_a_proj",
800
+ "model.language_model.layers.42.self_attn.g_b_proj",
801
+ "model.language_model.layers.42.self_attn.o_norm",
802
+ "model.language_model.layers.42.self_attn.o_proj",
803
+ "model.language_model.layers.42.mlp.gate",
804
+ "model.language_model.layers.42.mlp.shared_experts.gate_proj",
805
+ "model.language_model.layers.42.mlp.shared_experts.up_proj",
806
+ "model.language_model.layers.42.mlp.shared_experts.down_proj",
807
+ "model.language_model.layers.43.self_attn.q_a_proj",
808
+ "model.language_model.layers.43.self_attn.q_b_proj",
809
+ "model.language_model.layers.43.self_attn.kv_a_proj_with_mqa",
810
+ "model.language_model.layers.43.self_attn.kv_b_proj",
811
+ "model.language_model.layers.43.self_attn.o_proj",
812
+ "model.language_model.layers.43.self_attn.indexer.wq_b",
813
+ "model.language_model.layers.43.self_attn.indexer.wk",
814
+ "model.language_model.layers.43.self_attn.indexer.weights_proj",
815
+ "model.language_model.layers.43.mlp.gate",
816
+ "model.language_model.layers.43.mlp.shared_experts.gate_proj",
817
+ "model.language_model.layers.43.mlp.shared_experts.up_proj",
818
+ "model.language_model.layers.43.mlp.shared_experts.down_proj",
819
+ "model.language_model.layers.44.self_attn.q_proj",
820
+ "model.language_model.layers.44.self_attn.k_proj",
821
+ "model.language_model.layers.44.self_attn.v_proj",
822
+ "model.language_model.layers.44.self_attn.forget_gate",
823
+ "model.language_model.layers.44.self_attn.forget_gate.f_a_proj",
824
+ "model.language_model.layers.44.self_attn.forget_gate.f_b_proj",
825
+ "model.language_model.layers.44.self_attn.b_proj",
826
+ "model.language_model.layers.44.self_attn.g_a_proj",
827
+ "model.language_model.layers.44.self_attn.g_b_proj",
828
+ "model.language_model.layers.44.self_attn.o_norm",
829
+ "model.language_model.layers.44.self_attn.o_proj",
830
+ "model.language_model.layers.44.mlp.gate",
831
+ "model.language_model.layers.44.mlp.shared_experts.gate_proj",
832
+ "model.language_model.layers.44.mlp.shared_experts.up_proj",
833
+ "model.language_model.layers.44.mlp.shared_experts.down_proj",
834
+ "model.language_model.layers.45.eh_proj",
835
+ "model.language_model.layers.45.self_attn.q_a_proj",
836
+ "model.language_model.layers.45.self_attn.q_b_proj",
837
+ "model.language_model.layers.45.self_attn.kv_a_proj_with_mqa",
838
+ "model.language_model.layers.45.self_attn.kv_b_proj",
839
+ "model.language_model.layers.45.self_attn.o_proj",
840
+ "model.language_model.layers.45.self_attn.indexer.wq_b",
841
+ "model.language_model.layers.45.self_attn.indexer.wk",
842
+ "model.language_model.layers.45.self_attn.indexer.weights_proj",
843
+ "model.language_model.layers.45.mlp.gate",
844
+ "model.language_model.layers.45.mlp.shared_experts.gate_proj",
845
+ "model.language_model.layers.45.mlp.shared_experts.up_proj",
846
+ "model.language_model.layers.45.mlp.shared_experts.down_proj",
847
+ "lm_head"
848
+ ],
849
+ "kv_cache_scheme": null,
850
+ "moe_sparsity_config": null,
851
+ "quant_method": "compressed-tensors",
852
+ "quantization_status": "compressed",
853
+ "version": "0.18.1.dev12+gbc35079"
854
+ },
855
+ "text_config": {
856
+ "attention_bias": false,
857
+ "attention_dropout": 0.0,
858
+ "bos_token_id": null,
859
+ "dtype": "bfloat16",
860
+ "eos_token_id": [
861
+ 154820,
862
+ 154827,
863
+ 154829
864
+ ],
865
+ "first_k_dense_replace": 3,
866
+ "hc_eps": 1e-06,
867
+ "hc_mult": 4,
868
+ "hc_sinkhorn_iters": 20,
869
+ "head_dim": 0,
870
+ "hidden_act": "silu",
871
+ "hidden_size": 4096,
872
+ "index_head_dim": 128,
873
+ "index_kpool": 4,
874
+ "index_kpool_always_select_tail": true,
875
+ "index_kpool_compress": true,
876
+ "index_n_heads": 32,
877
+ "index_share_for_mtp_iteration": true,
878
+ "index_topk": 2048,
879
+ "indexer_rope_interleave": true,
880
+ "indexer_types": [
881
+ "full",
882
+ "full",
883
+ "full",
884
+ "full",
885
+ "full",
886
+ "full",
887
+ "full",
888
+ "full",
889
+ "full",
890
+ "full",
891
+ "full",
892
+ "full",
893
+ "full",
894
+ "full",
895
+ "full",
896
+ "full",
897
+ "full",
898
+ "full",
899
+ "full",
900
+ "full",
901
+ "full",
902
+ "full",
903
+ "full",
904
+ "full",
905
+ "full",
906
+ "full",
907
+ "full",
908
+ "full",
909
+ "full",
910
+ "full",
911
+ "full",
912
+ "full",
913
+ "full",
914
+ "full",
915
+ "full",
916
+ "full",
917
+ "full",
918
+ "full",
919
+ "full",
920
+ "full",
921
+ "full",
922
+ "full",
923
+ "full",
924
+ "full",
925
+ "full"
926
+ ],
927
+ "initializer_range": 0.02,
928
+ "intermediate_size": 12288,
929
+ "kv_lora_rank": 512,
930
+ "layer_types": [
931
+ "linear_attention",
932
+ "linear_attention",
933
+ "linear_attention",
934
+ "deepseek_sparse_attention",
935
+ "linear_attention",
936
+ "linear_attention",
937
+ "linear_attention",
938
+ "deepseek_sparse_attention",
939
+ "linear_attention",
940
+ "linear_attention",
941
+ "linear_attention",
942
+ "deepseek_sparse_attention",
943
+ "linear_attention",
944
+ "linear_attention",
945
+ "linear_attention",
946
+ "deepseek_sparse_attention",
947
+ "linear_attention",
948
+ "linear_attention",
949
+ "linear_attention",
950
+ "deepseek_sparse_attention",
951
+ "linear_attention",
952
+ "linear_attention",
953
+ "linear_attention",
954
+ "deepseek_sparse_attention",
955
+ "linear_attention",
956
+ "linear_attention",
957
+ "linear_attention",
958
+ "deepseek_sparse_attention",
959
+ "linear_attention",
960
+ "linear_attention",
961
+ "linear_attention",
962
+ "deepseek_sparse_attention",
963
+ "linear_attention",
964
+ "linear_attention",
965
+ "linear_attention",
966
+ "deepseek_sparse_attention",
967
+ "linear_attention",
968
+ "linear_attention",
969
+ "linear_attention",
970
+ "deepseek_sparse_attention",
971
+ "linear_attention",
972
+ "linear_attention",
973
+ "linear_attention",
974
+ "deepseek_sparse_attention",
975
+ "linear_attention"
976
+ ],
977
+ "linear_attn_config": {
978
+ "full_attn_layers": [
979
+ 3,
980
+ 7,
981
+ 11,
982
+ 15,
983
+ 19,
984
+ 23,
985
+ 27,
986
+ 31,
987
+ 35,
988
+ 39,
989
+ 43
990
+ ],
991
+ "gate_lower_bound": -5.0,
992
+ "head_dim": 128,
993
+ "kda_layers": [
994
+ 0,
995
+ 1,
996
+ 2,
997
+ 4,
998
+ 5,
999
+ 6,
1000
+ 8,
1001
+ 9,
1002
+ 10,
1003
+ 12,
1004
+ 13,
1005
+ 14,
1006
+ 16,
1007
+ 17,
1008
+ 18,
1009
+ 20,
1010
+ 21,
1011
+ 22,
1012
+ 24,
1013
+ 25,
1014
+ 26,
1015
+ 28,
1016
+ 29,
1017
+ 30,
1018
+ 32,
1019
+ 33,
1020
+ 34,
1021
+ 36,
1022
+ 37,
1023
+ 38,
1024
+ 40,
1025
+ 41,
1026
+ 42,
1027
+ 44
1028
+ ],
1029
+ "num_heads": 64,
1030
+ "short_conv_kernel_size": 4
1031
+ },
1032
+ "linear_conv_kernel_dim": 4,
1033
+ "linear_head_dim": 128,
1034
+ "linear_lower_bound": -5.0,
1035
+ "linear_num_heads": 64,
1036
+ "max_position_embeddings": 1048576,
1037
+ "mhc": true,
1038
+ "mla_use_nope": true,
1039
+ "mlp_layer_types": [
1040
+ "dense",
1041
+ "dense",
1042
+ "dense",
1043
+ "sparse",
1044
+ "sparse",
1045
+ "sparse",
1046
+ "sparse",
1047
+ "sparse",
1048
+ "sparse",
1049
+ "sparse",
1050
+ "sparse",
1051
+ "sparse",
1052
+ "sparse",
1053
+ "sparse",
1054
+ "sparse",
1055
+ "sparse",
1056
+ "sparse",
1057
+ "sparse",
1058
+ "sparse",
1059
+ "sparse",
1060
+ "sparse",
1061
+ "sparse",
1062
+ "sparse",
1063
+ "sparse",
1064
+ "sparse",
1065
+ "sparse",
1066
+ "sparse",
1067
+ "sparse",
1068
+ "sparse",
1069
+ "sparse",
1070
+ "sparse",
1071
+ "sparse",
1072
+ "sparse",
1073
+ "sparse",
1074
+ "sparse",
1075
+ "sparse",
1076
+ "sparse",
1077
+ "sparse",
1078
+ "sparse",
1079
+ "sparse",
1080
+ "sparse",
1081
+ "sparse",
1082
+ "sparse",
1083
+ "sparse",
1084
+ "sparse"
1085
+ ],
1086
+ "model_type": "glm5_next_text",
1087
+ "moe_intermediate_size": 2048,
1088
+ "moe_router_dtype": "float32",
1089
+ "n_group": 1,
1090
+ "n_routed_experts": 230,
1091
+ "n_shared_experts": 1,
1092
+ "norm_topk_prob": true,
1093
+ "num_attention_heads": 64,
1094
+ "num_experts_per_tok": 8,
1095
+ "num_hidden_layers": 45,
1096
+ "num_key_value_heads": 64,
1097
+ "num_nextn_predict_layers": 1,
1098
+ "output_router_logits": false,
1099
+ "pad_token_id": 154820,
1100
+ "q_lora_rank": 1536,
1101
+ "qk_head_dim": 256,
1102
+ "qk_nope_head_dim": 256,
1103
+ "qk_rope_head_dim": 0,
1104
+ "rms_norm_eps": 1e-05,
1105
+ "routed_scaling_factor": 2.5,
1106
+ "router_aux_loss_coef": 0.001,
1107
+ "scoring_func": "sigmoid",
1108
+ "swiglu_limit": 10.0,
1109
+ "tie_word_embeddings": false,
1110
+ "topk_group": 1,
1111
+ "topk_method": "noaux_tc",
1112
+ "use_cache": true,
1113
+ "v_head_dim": 256,
1114
+ "vocab_size": 154880
1115
+ },
1116
+ "tie_word_embeddings": false,
1117
+ "transformers_version": "5.16.1",
1118
+ "video_end_token_id": 154833,
1119
+ "video_start_token_id": 154832,
1120
+ "video_token_id": 154855,
1121
+ "vision_config": {
1122
+ "attention_bias": true,
1123
+ "attention_dropout": 0.0,
1124
+ "depth": 24,
1125
+ "dtype": "bfloat16",
1126
+ "hidden_act": "silu",
1127
+ "hidden_size": 1024,
1128
+ "image_size": 448,
1129
+ "in_channels": 3,
1130
+ "initializer_range": 0.02,
1131
+ "intermediate_size": 4096,
1132
+ "model_type": "glm5_next_vision",
1133
+ "num_heads": 16,
1134
+ "out_hidden_size": 4096,
1135
+ "patch_size": 14,
1136
+ "projection_intermediate_size": 10240,
1137
+ "rms_norm_eps": 1e-05,
1138
+ "spatial_merge_size": 2,
1139
+ "swiglu_limit": 10.0,
1140
+ "temporal_patch_size": 2
1141
+ }
1142
+ }
generation_config.json ADDED
@@ -0,0 +1,11 @@
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "eos_token_id": [
4
+ 154820,
5
+ 154827,
6
+ 154829
7
+ ],
8
+ "pad_token_id": 154820,
9
+ "temperature": 1.0,
10
+ "transformers_version": "5.16.1"
11
+ }
model-00001-of-00010.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5f622c961ba3dd35d2666d6fa8209bf3e282ba3551f36341b123bccda8d5c3df
3
+ size 17014232868
model-00002-of-00010.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:15499ec55031a3c7128affede93555d9629baf60e03560e1a0e6824a442e7871
3
+ size 16347792168
model-00003-of-00010.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:59f1c18ec3e2f41df2c6f6e2fbfd57adfe00c52205295c25a6e7bb5fcc70b28f
3
+ size 16232927096
model-00004-of-00010.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:653872b39ca27ba675f8c0de2b38020403a6965489567f5d472de2588b006bf6
3
+ size 16307566760
model-00005-of-00010.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:67c1efc1056512090216b4bbadbe74a3f5d952649bde6c91367ffc5f006e9521
3
+ size 16242364368
model-00006-of-00010.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0add6e3d8f0d3e8f582d74f03b384c081ae833d01b9bf6732977b87b3f477a69
3
+ size 16274530632
model-00007-of-00010.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:232bae4a74dcd8e1ca16253c64810356117231f3dd4a478c7aba821a126a51f1
3
+ size 16232926512
model-00008-of-00010.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9f369e2faf714e60a1ef00f5634cd5576ec3b6f88a5efcf881e782fcf23a2cbb
3
+ size 16260373608
model-00009-of-00010.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fcc21191c090fe3e7dbec62591588c894c88b74f9177170f0e4351af272acf36
3
+ size 16265957840
model-00010-of-00010.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f613dbb28c07624ac6e4e5a4b54ebea7da3ce8d78addb1e374aaba7a73a5fafe
3
+ size 8538928568
model.safetensors.index.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fbe429e97f763175b38a24fbf599dd1c976ecedc5edc0857cd758e19c482a6ba
3
+ size 13674530
model_mtp.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0cf8dfdc936f1cbc33b9e88d0289ff3b5b4a3d0f7381158a8ccbebb9c8adc035
3
+ size 7618560424
processor_config.json ADDED
@@ -0,0 +1,44 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "image_processor": {
3
+ "do_rescale": true,
4
+ "patch_expand_factor": 1,
5
+ "merge_size": 2,
6
+ "image_mean": [
7
+ 0.48145466,
8
+ 0.4578275,
9
+ 0.40821073
10
+ ],
11
+ "image_std": [
12
+ 0.26862954,
13
+ 0.26130258,
14
+ 0.27577711
15
+ ],
16
+ "temporal_patch_size": 2,
17
+ "patch_size": 14,
18
+ "min_image_tokens": 16,
19
+ "max_image_tokens": 8000,
20
+ "image_processor_type": "Glm5NextImageProcessor"
21
+ },
22
+ "video_processor": {
23
+ "do_rescale": true,
24
+ "video_processor_type": "Glm5NextVideoProcessor",
25
+ "patch_expand_factor": 1,
26
+ "merge_size": 2,
27
+ "image_mean": [
28
+ 0.48145466,
29
+ 0.4578275,
30
+ 0.40821073
31
+ ],
32
+ "image_std": [
33
+ 0.26862954,
34
+ 0.26130258,
35
+ 0.27577711
36
+ ],
37
+ "temporal_patch_size": 2,
38
+ "patch_size": 14,
39
+ "min_image_tokens": 16,
40
+ "max_image_tokens": 240000,
41
+ "fps": 2
42
+ },
43
+ "processor_class": "Glm5NextProcessor"
44
+ }
recipe.yaml ADDED
@@ -0,0 +1,38 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ default_stage:
2
+ default_modifiers:
3
+ QuantizationModifier:
4
+ config_groups:
5
+ experts:
6
+ targets: ['re:.*mlp\.experts\..*(gate|up|down)_proj$']
7
+ weights:
8
+ num_bits: 4
9
+ type: float
10
+ symmetric: true
11
+ group_size: 16
12
+ strategy: tensor_group
13
+ block_structure: null
14
+ dynamic: false
15
+ actorder: null
16
+ scale_dtype: torch.float8_e4m3fn
17
+ zp_dtype: null
18
+ observer: memoryless_minmax
19
+ observer_kwargs: {}
20
+ input_activations:
21
+ num_bits: 4
22
+ type: float
23
+ symmetric: true
24
+ group_size: 16
25
+ strategy: tensor_group
26
+ block_structure: null
27
+ dynamic: local
28
+ actorder: null
29
+ scale_dtype: torch.float8_e4m3fn
30
+ zp_dtype: null
31
+ observer: static_minmax
32
+ observer_kwargs: {}
33
+ output_activations: null
34
+ format: null
35
+ targets: [Linear]
36
+ ignore: ['re:.*visual.*', lm_head, 're:.*mlp\.gate$', 're:.*self_attn\.indexer\..*']
37
+ bypass_divisibility_checks: false
38
+ requires_calibration_data: true
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0cfe2c099a7702a0921abc315ee039deb51e4a34b4818fc509bd27fa3dc4acc1
3
+ size 20217541
tokenizer_config.json ADDED
@@ -0,0 +1,34 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "backend": "tokenizers",
3
+ "clean_up_tokenization_spaces": false,
4
+ "do_lower_case": false,
5
+ "eos_token": "<|endoftext|>",
6
+ "extra_special_tokens": [
7
+ "<|endoftext|>",
8
+ "[MASK]",
9
+ "[gMASK]",
10
+ "[sMASK]",
11
+ "<sop>",
12
+ "<eop>",
13
+ "<|system|>",
14
+ "<|user|>",
15
+ "<|assistant|>",
16
+ "<|observation|>",
17
+ "<|begin_of_image|>",
18
+ "<|end_of_image|>",
19
+ "<|begin_of_video|>",
20
+ "<|end_of_video|>",
21
+ "<|begin_of_audio|>",
22
+ "<|end_of_audio|>",
23
+ "<|begin_of_transcription|>",
24
+ "<|end_of_transcription|>"
25
+ ],
26
+ "is_local": true,
27
+ "local_files_only": false,
28
+ "model_max_length": 1048576,
29
+ "model_specific_special_tokens": {},
30
+ "pad_token": "<|endoftext|>",
31
+ "padding_side": "left",
32
+ "remove_space": false,
33
+ "tokenizer_class": "TokenizersBackend"
34
+ }