Vveerrgg commited on
Commit
92901bb
·
verified ·
1 Parent(s): d50096c

GLM-4-9B Cooking Assistant v1

Browse files
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
LICENSE ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ MIT License
2
+
3
+ Copyright (c) 2025 Zhipu AI
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
README.md ADDED
@@ -0,0 +1,217 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: mit
3
+ base_model: zai-org/GLM-4-9B-0414
4
+ base_model_relation: finetune
5
+ library_name: transformers
6
+ pipeline_tag: text-generation
7
+ language: [en, es, pt, it, fr, nl, de, ru, bn, id, fa, ja, pl, sv, vi, tl, he, ml, tr, zh, ms, th, fi, ar, ta, hi]
8
+ tags: [cooking, recipes, dpo, glm4, provenance, abstention]
9
+ datasets: [HumanjavaEnterprises/cooking-assistant-eval]
10
+ ---
11
+
12
+ # GLM-4-9B Cooking Assistant
13
+
14
+ A cooking assistant that reads the recipe you give it, and usually asks for the recipe instead of inventing details
15
+ when it can't see one. On questions that need a recipe it wasn't shown, it invents details 9.7% of the time vs
16
+ 40.1% for base GLM-4-9B-0414; given the recipe, it answers correctly 96.2% of the time vs
17
+ 83.1% (three open judges from other model families, 2 of 3 agreeing). General answers are not improved: on our 82 general cooking questions the full-precision model scored lower than the base (46.3% vs 56.1%; not statistically significant). Trained on 26 languages, evaluated in 21;
18
+ runs fully on an 8 GB GPU (Q4_K_M), or partly offloaded on 4 GB.
19
+
20
+ *Unofficial fine-tune of GLM-4-9B-0414 by Humanjava Enterprises; not made or endorsed by Zhipu AI / Z.ai.*
21
+
22
+ Full-precision (bf16) safetensors for Transformers and vLLM. For llama.cpp, Ollama and LM Studio:
23
+ [HumanjavaEnterprises/GLM-4-9B-Cooking-Assistant-GGUF](https://huggingface.co/HumanjavaEnterprises/GLM-4-9B-Cooking-Assistant-GGUF).
24
+ The same recipe on Qwen3-8B:
25
+ [HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant](https://huggingface.co/HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant).
26
+
27
+ It does not memorise recipes: give it the recipe (paste it, or have your app look it up) and it will read it.
28
+
29
+ ## Files
30
+
31
+ The two training stages (supervised LoRA, then a preference LoRA on top) are merged into one bf16 model, the same
32
+ architecture and tokenizer as GLM-4-9B-0414 (~18.8 GB). The GGUF files in the sibling repo are converted from these
33
+ weights.
34
+
35
+ ## Quick start
36
+
37
+ ```bash
38
+ # vLLM
39
+ vllm serve HumanjavaEnterprises/GLM-4-9B-Cooking-Assistant --max-model-len 8192
40
+ ```
41
+
42
+ ```python
43
+ # Transformers
44
+ from transformers import AutoModelForCausalLM, AutoTokenizer
45
+ repo = "HumanjavaEnterprises/GLM-4-9B-Cooking-Assistant"
46
+ tok = AutoTokenizer.from_pretrained(repo)
47
+ model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto")
48
+ messages = [{"role": "system", "content": "You are a cooking assistant. You answer questions about dishes, ingredients, techniques and food safety. When you are not sure, you say so."},
49
+ {"role": "user", "content": "If I double the recipe, how much pepper do I need?"}]
50
+ inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_dict=True, return_tensors="pt").to(model.device)
51
+ out = model.generate(**inputs, max_new_tokens=400)
52
+ print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip())
53
+ ```
54
+
55
+ ## Versions
56
+
57
+ The model is one product with dated versions. Each version is a git tag on this repo, so you can pin one with
58
+ `revision="v1"` (Transformers / huggingface_hub) or `ollama run vveerrgg/glm-4-9b-cooking-assistant:v1`.
59
+
60
+ | Version | Date | Invents (no recipe, 2 of 3 judges) | Correct (recipe shown, 2 of 3 judges) | Notes |
61
+ |---|---|---|---|---|
62
+ | **v1** (current) | 2026-11-14 | 9.7% (base 40.1%) | 96.2% (base 83.1%) | first public release |
63
+
64
+ ## Prompt format
65
+
66
+ GLM-4-0414 chat template (in the tokenizer; `[gMASK]<sop>`, then `<|system|>`, `<|user|>`, `<|assistant|>` turns). There
67
+ is no thinking mode. Trained with this system prompt, which works best:
68
+
69
+ ```
70
+ You are a cooking assistant. You answer questions about dishes, ingredients, techniques and food safety. When you are not sure, you say so.
71
+ ```
72
+
73
+ To ask about a specific recipe, put the recipe text in the user turn, then the question.
74
+
75
+ ## What it does
76
+
77
+ The fine-tune teaches **behaviour, not facts**:
78
+
79
+ - **Reads the recipe you give it.** Paste a recipe and ask about it; it answers from that text.
80
+ - **Usually asks when the recipe is missing.** "If I double the recipe, how much pepper?" with no recipe gets a request
81
+ for the original quantity; it may add a typical range, marked as typical, but not a number presented as the recipe's.
82
+ - **Answers general cooking questions:** food safety, substitutions, why a dish failed. On our general questions it
83
+ does not beat the base model (see Results).
84
+
85
+ The intended setup is **LoRA for behaviour, retrieval for facts, tools for numbers**: an app looks up the recipe and
86
+ hands it to the model; scaling and unit conversion are calculated, not guessed.
87
+
88
+ It is the second model built with the recipe behind
89
+ [Qwen3-8B Cooking Assistant](https://huggingface.co/HumanjavaEnterprises/Qwen3-8B-Cooking-Assistant): the same training
90
+ data and the same two stages on a different base model family, scored on the same questions.
91
+
92
+ ## Results
93
+
94
+ All cooking scores are on the fixed held-out set of 682 questions in 21 languages behind the Qwen3-8B Cooking Assistant
95
+ card, never used in training: 290 with the recipe shown, 10 recall questions with no recipe ("how do you make X?"), 300
96
+ that need a recipe the model is not shown, and 82 general questions. Answers from base GLM-4-9B-0414 and this model
97
+ were scored one at a time by open-weights judges, blind to which model wrote them. None of the headline judges is from
98
+ the GLM or the Qwen family: DeepSeek-V4-Flash (MIT), gpt-oss-120b (Apache-2.0) and Seed-OSS-36B-Instruct (Apache-2.0).
99
+ A rule check removes "invents" verdicts on answers whose every detail was already in the question. 95% confidence
100
+ intervals in brackets.
101
+
102
+ | counted as "invents" when | GLM-4-9B-0414 | this model |
103
+ |---|---|---|
104
+ | at least 2 of 3 judges say so | 40.1% [34.7, 45.8] | **9.7%** [6.8, 13.6] |
105
+ | all three say so | 28.1% | 3.7% |
106
+ | any one says so | 58.9% | 27.1% |
107
+
108
+ (299 of the 300 answers per column count: one base and one model answer lack a verdict from one judge.)
109
+
110
+ 2 of 3: 104 questions where only the base invents, 13 where only this model does
111
+ (McNemar p < 1e-18).
112
+
113
+ The same judges on the questions where the recipe **is** shown and the model has to read it (290 questions; 2 of 3
114
+ must agree): GLM-4-9B-0414 correct 83.1% [78.4, 87.0], this model **96.2%**
115
+ [93.3, 97.9] (42 questions only this model gets right, 4 only the base
116
+ does, McNemar p < 1e-8). Two other kinds of question in the set are reported apart: 82 general questions graded
117
+ against a reference answer (56.1% base, 46.3% this model: lower, but not significantly, McNemar p = 0.12 with 14
118
+ questions only the base gets right and 6 only this model; see the quantised files below) and
119
+ 10 "how do you make X?" questions with no recipe whose reference is the original 1800s text (20.0% /
120
+ 10.0%: the model is built not to recall historical recipes from memory).
121
+
122
+ **A fourth judge.** Qwen3.8-27B (Apache-2.0) also scored every answer. It is a Qwen-family model and wrote the training
123
+ questions, so it is not independent of the training data and is kept out of the headline. With all four judges and 3
124
+ of 4 agreeing: invents 30.1% (base) vs **4.7%**; correct with the recipe shown
125
+ 78.6% vs **95.5%**.
126
+
127
+ **Next to the Qwen3-8B sibling** (same questions, same three judges, both models' answers generated the same way):
128
+
129
+ | | invents, no recipe (2 of 3) | correct, recipe shown (2 of 3) |
130
+ |---|---|---|
131
+ | Qwen3-8B → Qwen3-8B Cooking Assistant v1 | 43.0% → 10.3% | 85.9% → 95.5% |
132
+ | GLM-4-9B-0414 → this model | 40.1% → 9.7% | 83.1% → 96.2% |
133
+
134
+ (The Qwen card's own figures come from a four-judge panel that included GLM-4.7-Flash, so they differ from this row.)
135
+
136
+ Model judges tend to make the same mistakes as each other, so three agreeing is less than three independent checks.
137
+ On the no-recipe questions the judges agree less about this model's answers than about the base's (Fleiss kappa 0.33
138
+ vs 0.58).
139
+
140
+ **Quantised files** were scored on the same 682 questions against the full-precision model, same judges:
141
+
142
+ | | Q4_K_M | Q6_K | Q8_0 |
143
+ |---|---|---|---|
144
+ | Invents with no recipe, 2 of 3 judges (full precision) | 10.7% (8.7%) | 11.7% (9.3%) | 9.4% (10.0%) |
145
+ | Correct when the recipe is shown, 2 of 3 judges (full precision) | 95.5% (95.9%) | 95.5% (96.2%) | 95.9% (96.2%) |
146
+ | Difference significant? (McNemar, inventing) | no, p = 0.26 | no, p = 0.14 | no, p = 0.77 |
147
+
148
+ With the three headline judges, no quantised file differs significantly from full precision on inventing (p ≥ 0.14) or on reading a shown recipe (p ≥ 0.5). On the 82 general questions every quantised file scored higher than full precision: Q4_K_M 54.9% vs 46.3% (p = 0.039), Q6_K 53.7% vs 47.6% (p = 0.27), Q8_0 56.1% vs 47.6% (p = 0.065). The full-precision answers are the same text in every comparison; only the judging was re-run for each file, which alone moves their scores by up to 1.3 points. The quantised answers come from llama.cpp and the full-precision ones from vLLM, so the full-precision model's lower general score may be partly a serving or decoding effect (one of three differences has p < 0.05, none after correcting for three comparisons); not settled.
149
+
150
+ **Memorisation.** Training documents from the Wikibooks Cookbook (CC BY-SA) were probed with prefixes of 32 to 128 tokens at several offsets, as raw text and as chat prompts, with greedy decoding. The model never continues a document verbatim (no 50-token match; longest overlap 89 characters, the same as base GLM-4-9B-0414), and its overlap with the true text is no higher than the base's (ROUGE-L 0.158 vs 0.161).
151
+
152
+ ## How it was built
153
+
154
+ | | |
155
+ |---|---|
156
+ | Sources | Cooking texts in 26 languages from openly licensed and public-domain sources only: the Wikibooks Cookbook (CC BY-SA 4.0), public-domain cookbooks from Project Gutenberg and the Internet Archive (scans re-read with olmOCR-2), and NHLBI recipes (US government, public domain). Every record carries its source URL, licence and public-domain basis |
157
+ | Excluded | Traditional and Indigenous knowledge, sacred material, outsider-authored books about other cultures, dog-meat recipes, anything without a verified licence |
158
+ | Models used | Open-weights models only, all Apache-2.0 or MIT. **No output from any closed or API model is in the training, evaluation or seed data** |
159
+ | Training | The training data and both stages of Qwen3-8B Cooking Assistant v1, unchanged, on GLM-4-9B-0414: two LoRA stages, each merged: a supervised stage that teaches reading a shown recipe, asking when there is none, and full general answers; then a preference stage that favours asking over inventing and correct over wrong. General chat data is mixed in to limit forgetting |
160
+ | Tooling | LLaMA-Factory, vLLM, llama.cpp |
161
+
162
+ The evaluation set behind every number on this card (all 682 questions, both models' answers, every judge's verdict,
163
+ and per-row source credits) is published so the results can be checked:
164
+ [HumanjavaEnterprises/cooking-assistant-eval](https://huggingface.co/datasets/HumanjavaEnterprises/cooking-assistant-eval),
165
+ config `glm-4-9b-v1`.
166
+
167
+ ## What carried over, and what did not
168
+
169
+ - **The data was written for the Qwen model.** Part of the general answers were rewritten by base Qwen3-8B (for the
170
+ Qwen model, its own words), and the preference stage uses answer pairs sampled from the Qwen model, not from this
171
+ one. Both were kept unchanged so that only the base model differs; for GLM-4-9B-0414 they are another model's words.
172
+ - **The two cooking behaviours carried over.** On inventing and on reading a shown recipe this model shows no significant difference from Qwen3-8B Cooking Assistant v1 under the same judges (McNemar p = 0.86 and 0.80). **General answers did not:** where the Qwen model stayed level with its base, this model's full-precision score fell below GLM-4-9B-0414's (not significant; see Results).
173
+
174
+ ## Limitations
175
+
176
+ - **Not a source of truth for food safety.** It repeats standard guidance (e.g. 74 °C / 165 °F for poultry) but can be
177
+ wrong; check official sources.
178
+ - **It does not know historical recipes by heart.** It was trained on old cookbooks to read and reason about recipes,
179
+ not to recall them. For a specific historical dish, give it the text.
180
+ - The judges are models. Three open judges from three other families agree on the direction of the changes,
181
+ and we keep checking them against people with Label Test, our blind labelling tool. **Help refine it:** label five
182
+ answers at [label-test.humanjava.com](https://label-test.humanjava.com/?s=cook4&k=cook-9ac741a1e387&src=hf-card-glm) (no account; we keep only a name or initials and your labels).
183
+ - Still invents on some no-recipe questions: about 1 in 10 when two of three judges must agree (9.7%), nearly 1 in 4 by the strictest single judge (22.7%).
184
+ - **General answers are not improved.** On 82 general questions the full-precision model scored 46.3% vs the base's 56.1% (not significant, p = 0.12); the quantised files scored 54-56% on the same questions (not compared against the base), so the gap may be partly a serving or decoding effect (one of three differences has p < 0.05, none after correcting for three comparisons); not settled.
185
+ - Best in English and Spanish: it invents on 6.9% and 3.8% of the no-recipe questions there, 12.6% across the other
186
+ languages together (2 of 3 judges).
187
+ - The old public-domain cookbooks reflect their era: measures, techniques, some outdated advice.
188
+
189
+ ## Feedback and security
190
+
191
+ - **The model got something wrong** (invented a detail, misread a recipe, unsafe advice): please
192
+ [open a report](https://github.com/HumanjavaEnterprises/glm-4-9b-cooking-assistant/issues/new?template=wrong-answer.yml).
193
+ Real examples are what improve the next version.
194
+ - **Help measure it:** [label five answers](https://label-test.humanjava.com/?s=cook4&k=cook-9ac741a1e387&src=hf-card-glm) on Label Test.
195
+ - **Security** (files that don't match `SHA256SUMS`, unexpected embedded content): report privately, see
196
+ [SECURITY.md](https://github.com/HumanjavaEnterprises/glm-4-9b-cooking-assistant/blob/main/SECURITY.md).
197
+
198
+ ## Licence and credits
199
+
200
+ Weights: MIT, same as the base model. Base model: GLM-4-9B-0414 by Zhipu AI / Z.ai (MIT); its copyright and permission
201
+ notice is kept in `LICENSE`:
202
+
203
+ > Copyright (c) 2025 Zhipu AI. Permission is hereby granted, free of charge, to any person obtaining a copy of this
204
+ > software and associated documentation files (the "Software"), to deal in the Software without restriction [...]
205
+ > The above copyright notice and this permission notice shall be included in all copies or substantial portions of
206
+ > the Software. (Full text in `LICENSE`.)
207
+
208
+ Training data sources: Wikibooks Cookbook contributors (CC BY-SA 4.0; per-page attribution in the dataset), Project
209
+ Gutenberg and Internet Archive public-domain books, NHLBI. General chat data: NVIDIA
210
+ (Nemotron-Instruction-Following-Chat-v1, ODC-BY 1.0, responses by GPT-OSS-120B and Qwen3-235B-A22B), Hugging Face
211
+ (SmolTalk2, Apache-2.0, responses by Qwen3-32B; prompts from SystemChat-2.0, the Aya Dataset by Cohere For AI, and
212
+ Table-GPT by Microsoft). Teacher: Qwen3.8-27B (Apache-2.0). Rewritten general answers: Qwen3-8B (Apache-2.0).
213
+ Preference-stage answers: Qwen3-8B Cooking Assistant (Apache-2.0). Judges: DeepSeek-V4-Flash (MIT), gpt-oss-120b
214
+ (Apache-2.0), Seed-OSS-36B-Instruct by ByteDance Seed (Apache-2.0), Qwen3.8-27B (Apache-2.0). OCR: olmOCR-2 by Ai2
215
+ (Apache-2.0). Training: LLaMA-Factory. Quantisation: llama.cpp.
216
+
217
+ Built by [Humanjava Enterprises](https://huggingface.co/HumanjavaEnterprises).
SHA256SUMS ADDED
@@ -0,0 +1,12 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ 3f5e5b9cc04bf96620f48b48888b7b19b08ad4cdebb145cb8026a2dff46c9e81 model-00001-of-00004.safetensors
2
+ 4ea6f5947e61b44619ce20bcab318890663e1a799486248499de824bbdb35b8a model-00002-of-00004.safetensors
3
+ 73c2c940fcecfe74c48a7433360e7b304149901e40a11882348fa852a9402b87 model-00003-of-00004.safetensors
4
+ b94a33a656e90c29e9961b40308b01faf5a7118d49560e217000982e613c729f model-00004-of-00004.safetensors
5
+ 8a8f873a73f5234b9e950b016ab844ba47cb8399d8bbdba202ed0ddc3ca7d421 config.json
6
+ 56ee0be8c595f22a6a80a822096533e10455fcd1e859e0fc36a6e760088147fa generation_config.json
7
+ 421fb251889681bc9c3bed28e5cd079e00a789a1a04024d53c8d182a8943af34 model.safetensors.index.json
8
+ 35a6dd9ff6e2474ba72b180db853851d2a84fce3d12095089285f447c7474541 special_tokens_map.json
9
+ e316474d867ad1f91165a12d90c73b27a7e0f9a1c9293831cca66c792bfe9a36 tokenizer_config.json
10
+ 76ebeac0d8bd7879ead7b43c16b44981f277e47225de2bd7de9ae1a6cc664a8c tokenizer.json
11
+ db700f25fa300e53634c6fc78dee86b7fbd6d27e624edb855b18a4078c83a822 chat_template.jinja
12
+ d08ac7b31e69ddf9845d56f2705325e69873f87b0b34b4bd53a3185428a784ae LICENSE
chat_template.jinja ADDED
@@ -0,0 +1,41 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [gMASK]<sop>
2
+ {%- if tools -%}
3
+ <|system|>
4
+ # 可用工具
5
+ {% for tool in tools %}
6
+ {%- set function = tool.function if tool.get("function") else tool %}
7
+
8
+ ## {{ function.name }}
9
+
10
+ {{ function | tojson(indent=4, ensure_ascii=False) }}
11
+ 在调用上述函数时,请使用 Json 格式表示调用的参数。
12
+ {%- endfor %}
13
+ {%- endif -%}
14
+
15
+ {%- for msg in messages %}
16
+ {%- if msg.role == 'system' %}
17
+ <|system|>
18
+ {{ msg.content }}
19
+ {%- endif %}
20
+ {%- endfor %}
21
+
22
+ {%- for message in messages if message.role != 'system' %}
23
+ {%- set role = message['role'] %}
24
+ {%- set content = message['content'] %}
25
+ {%- set meta = message.get("metadata", "") %}
26
+
27
+ {%- if role == 'user' %}
28
+ <|user|>
29
+ {{ content }}
30
+ {%- elif role == 'assistant' and not meta %}
31
+ <|assistant|>
32
+ {{ content }}
33
+ {%- elif role == 'assistant' and meta %}
34
+ <|assistant|>{{ meta }}
35
+ {{ content }}
36
+ {%- elif role == 'observation' %}
37
+ <|observation|>
38
+ {{ content }}
39
+ {%- endif %}
40
+ {%- endfor %}
41
+ {% if add_generation_prompt %}<|assistant|>{% endif %}
config.json ADDED
@@ -0,0 +1,36 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Glm4ForCausalLM"
4
+ ],
5
+ "attention_bias": true,
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": null,
8
+ "dtype": "bfloat16",
9
+ "eos_token_id": [
10
+ 151329,
11
+ 151336,
12
+ 151338
13
+ ],
14
+ "head_dim": 128,
15
+ "hidden_act": "silu",
16
+ "hidden_size": 4096,
17
+ "initializer_range": 0.02,
18
+ "intermediate_size": 13696,
19
+ "max_position_embeddings": 32768,
20
+ "model_type": "glm4",
21
+ "num_attention_heads": 32,
22
+ "num_hidden_layers": 40,
23
+ "num_key_value_heads": 2,
24
+ "pad_token_id": 151329,
25
+ "partial_rotary_factor": 0.5,
26
+ "rms_norm_eps": 1e-05,
27
+ "rope_parameters": {
28
+ "partial_rotary_factor": 0.5,
29
+ "rope_theta": 10000.0,
30
+ "rope_type": "default"
31
+ },
32
+ "tie_word_embeddings": false,
33
+ "transformers_version": "5.6.0",
34
+ "use_cache": true,
35
+ "vocab_size": 151552
36
+ }
generation_config.json ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "eos_token_id": [
4
+ 151329,
5
+ 151336,
6
+ 151338
7
+ ],
8
+ "pad_token_id": 151329,
9
+ "transformers_version": "4.52.0.dev0"
10
+ }
model-00001-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3f5e5b9cc04bf96620f48b48888b7b19b08ad4cdebb145cb8026a2dff46c9e81
3
+ size 4930673744
model-00002-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4ea6f5947e61b44619ce20bcab318890663e1a799486248499de824bbdb35b8a
3
+ size 4895274552
model-00003-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:73c2c940fcecfe74c48a7433360e7b304149901e40a11882348fa852a9402b87
3
+ size 4895274616
model-00004-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b94a33a656e90c29e9961b40308b01faf5a7118d49560e217000982e613c729f
3
+ size 4079395488
model.safetensors.index.json ADDED
@@ -0,0 +1,531 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "metadata": {
3
+ "total_parameters": 9400279040,
4
+ "total_size": 18800558080
5
+ },
6
+ "weight_map": {
7
+ "lm_head.weight": "model-00001-of-00004.safetensors",
8
+ "model.embed_tokens.weight": "model-00001-of-00004.safetensors",
9
+ "model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
10
+ "model.layers.0.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
11
+ "model.layers.0.mlp.gate_up_proj.weight": "model-00001-of-00004.safetensors",
12
+ "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
13
+ "model.layers.0.post_mlp_layernorm.weight": "model-00001-of-00004.safetensors",
14
+ "model.layers.0.post_self_attn_layernorm.weight": "model-00001-of-00004.safetensors",
15
+ "model.layers.0.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
16
+ "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
17
+ "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
18
+ "model.layers.0.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
19
+ "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
20
+ "model.layers.0.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
21
+ "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
22
+ "model.layers.1.input_layernorm.weight": "model-00001-of-00004.safetensors",
23
+ "model.layers.1.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
24
+ "model.layers.1.mlp.gate_up_proj.weight": "model-00001-of-00004.safetensors",
25
+ "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
26
+ "model.layers.1.post_mlp_layernorm.weight": "model-00001-of-00004.safetensors",
27
+ "model.layers.1.post_self_attn_layernorm.weight": "model-00001-of-00004.safetensors",
28
+ "model.layers.1.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
29
+ "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
30
+ "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
31
+ "model.layers.1.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
32
+ "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
33
+ "model.layers.1.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
34
+ "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
35
+ "model.layers.10.input_layernorm.weight": "model-00002-of-00004.safetensors",
36
+ "model.layers.10.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
37
+ "model.layers.10.mlp.gate_up_proj.weight": "model-00002-of-00004.safetensors",
38
+ "model.layers.10.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
39
+ "model.layers.10.post_mlp_layernorm.weight": "model-00002-of-00004.safetensors",
40
+ "model.layers.10.post_self_attn_layernorm.weight": "model-00002-of-00004.safetensors",
41
+ "model.layers.10.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
42
+ "model.layers.10.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
43
+ "model.layers.10.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
44
+ "model.layers.10.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
45
+ "model.layers.10.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
46
+ "model.layers.10.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
47
+ "model.layers.10.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
48
+ "model.layers.11.input_layernorm.weight": "model-00002-of-00004.safetensors",
49
+ "model.layers.11.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
50
+ "model.layers.11.mlp.gate_up_proj.weight": "model-00002-of-00004.safetensors",
51
+ "model.layers.11.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
52
+ "model.layers.11.post_mlp_layernorm.weight": "model-00002-of-00004.safetensors",
53
+ "model.layers.11.post_self_attn_layernorm.weight": "model-00002-of-00004.safetensors",
54
+ "model.layers.11.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
55
+ "model.layers.11.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
56
+ "model.layers.11.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
57
+ "model.layers.11.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
58
+ "model.layers.11.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
59
+ "model.layers.11.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
60
+ "model.layers.11.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
61
+ "model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
62
+ "model.layers.12.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
63
+ "model.layers.12.mlp.gate_up_proj.weight": "model-00002-of-00004.safetensors",
64
+ "model.layers.12.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
65
+ "model.layers.12.post_mlp_layernorm.weight": "model-00002-of-00004.safetensors",
66
+ "model.layers.12.post_self_attn_layernorm.weight": "model-00002-of-00004.safetensors",
67
+ "model.layers.12.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
68
+ "model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
69
+ "model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
70
+ "model.layers.12.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
71
+ "model.layers.12.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
72
+ "model.layers.12.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
73
+ "model.layers.12.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
74
+ "model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
75
+ "model.layers.13.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
76
+ "model.layers.13.mlp.gate_up_proj.weight": "model-00002-of-00004.safetensors",
77
+ "model.layers.13.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
78
+ "model.layers.13.post_mlp_layernorm.weight": "model-00002-of-00004.safetensors",
79
+ "model.layers.13.post_self_attn_layernorm.weight": "model-00002-of-00004.safetensors",
80
+ "model.layers.13.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
81
+ "model.layers.13.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
82
+ "model.layers.13.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
83
+ "model.layers.13.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
84
+ "model.layers.13.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
85
+ "model.layers.13.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
86
+ "model.layers.13.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
87
+ "model.layers.14.input_layernorm.weight": "model-00002-of-00004.safetensors",
88
+ "model.layers.14.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
89
+ "model.layers.14.mlp.gate_up_proj.weight": "model-00002-of-00004.safetensors",
90
+ "model.layers.14.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
91
+ "model.layers.14.post_mlp_layernorm.weight": "model-00002-of-00004.safetensors",
92
+ "model.layers.14.post_self_attn_layernorm.weight": "model-00002-of-00004.safetensors",
93
+ "model.layers.14.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
94
+ "model.layers.14.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
95
+ "model.layers.14.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
96
+ "model.layers.14.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
97
+ "model.layers.14.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
98
+ "model.layers.14.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
99
+ "model.layers.14.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
100
+ "model.layers.15.input_layernorm.weight": "model-00002-of-00004.safetensors",
101
+ "model.layers.15.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
102
+ "model.layers.15.mlp.gate_up_proj.weight": "model-00002-of-00004.safetensors",
103
+ "model.layers.15.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
104
+ "model.layers.15.post_mlp_layernorm.weight": "model-00002-of-00004.safetensors",
105
+ "model.layers.15.post_self_attn_layernorm.weight": "model-00002-of-00004.safetensors",
106
+ "model.layers.15.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
107
+ "model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
108
+ "model.layers.15.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
109
+ "model.layers.15.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
110
+ "model.layers.15.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
111
+ "model.layers.15.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
112
+ "model.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
113
+ "model.layers.16.input_layernorm.weight": "model-00002-of-00004.safetensors",
114
+ "model.layers.16.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
115
+ "model.layers.16.mlp.gate_up_proj.weight": "model-00002-of-00004.safetensors",
116
+ "model.layers.16.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
117
+ "model.layers.16.post_mlp_layernorm.weight": "model-00002-of-00004.safetensors",
118
+ "model.layers.16.post_self_attn_layernorm.weight": "model-00002-of-00004.safetensors",
119
+ "model.layers.16.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
120
+ "model.layers.16.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
121
+ "model.layers.16.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
122
+ "model.layers.16.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
123
+ "model.layers.16.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
124
+ "model.layers.16.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
125
+ "model.layers.16.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
126
+ "model.layers.17.input_layernorm.weight": "model-00002-of-00004.safetensors",
127
+ "model.layers.17.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
128
+ "model.layers.17.mlp.gate_up_proj.weight": "model-00002-of-00004.safetensors",
129
+ "model.layers.17.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
130
+ "model.layers.17.post_mlp_layernorm.weight": "model-00002-of-00004.safetensors",
131
+ "model.layers.17.post_self_attn_layernorm.weight": "model-00002-of-00004.safetensors",
132
+ "model.layers.17.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
133
+ "model.layers.17.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
134
+ "model.layers.17.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
135
+ "model.layers.17.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
136
+ "model.layers.17.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
137
+ "model.layers.17.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
138
+ "model.layers.17.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
139
+ "model.layers.18.input_layernorm.weight": "model-00002-of-00004.safetensors",
140
+ "model.layers.18.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
141
+ "model.layers.18.mlp.gate_up_proj.weight": "model-00003-of-00004.safetensors",
142
+ "model.layers.18.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
143
+ "model.layers.18.post_mlp_layernorm.weight": "model-00003-of-00004.safetensors",
144
+ "model.layers.18.post_self_attn_layernorm.weight": "model-00003-of-00004.safetensors",
145
+ "model.layers.18.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
146
+ "model.layers.18.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
147
+ "model.layers.18.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
148
+ "model.layers.18.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
149
+ "model.layers.18.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
150
+ "model.layers.18.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
151
+ "model.layers.18.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
152
+ "model.layers.19.input_layernorm.weight": "model-00003-of-00004.safetensors",
153
+ "model.layers.19.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
154
+ "model.layers.19.mlp.gate_up_proj.weight": "model-00003-of-00004.safetensors",
155
+ "model.layers.19.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
156
+ "model.layers.19.post_mlp_layernorm.weight": "model-00003-of-00004.safetensors",
157
+ "model.layers.19.post_self_attn_layernorm.weight": "model-00003-of-00004.safetensors",
158
+ "model.layers.19.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
159
+ "model.layers.19.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
160
+ "model.layers.19.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
161
+ "model.layers.19.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
162
+ "model.layers.19.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
163
+ "model.layers.19.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
164
+ "model.layers.19.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
165
+ "model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
166
+ "model.layers.2.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
167
+ "model.layers.2.mlp.gate_up_proj.weight": "model-00001-of-00004.safetensors",
168
+ "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
169
+ "model.layers.2.post_mlp_layernorm.weight": "model-00001-of-00004.safetensors",
170
+ "model.layers.2.post_self_attn_layernorm.weight": "model-00001-of-00004.safetensors",
171
+ "model.layers.2.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
172
+ "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
173
+ "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
174
+ "model.layers.2.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
175
+ "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
176
+ "model.layers.2.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
177
+ "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
178
+ "model.layers.20.input_layernorm.weight": "model-00003-of-00004.safetensors",
179
+ "model.layers.20.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
180
+ "model.layers.20.mlp.gate_up_proj.weight": "model-00003-of-00004.safetensors",
181
+ "model.layers.20.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
182
+ "model.layers.20.post_mlp_layernorm.weight": "model-00003-of-00004.safetensors",
183
+ "model.layers.20.post_self_attn_layernorm.weight": "model-00003-of-00004.safetensors",
184
+ "model.layers.20.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
185
+ "model.layers.20.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
186
+ "model.layers.20.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
187
+ "model.layers.20.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
188
+ "model.layers.20.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
189
+ "model.layers.20.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
190
+ "model.layers.20.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
191
+ "model.layers.21.input_layernorm.weight": "model-00003-of-00004.safetensors",
192
+ "model.layers.21.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
193
+ "model.layers.21.mlp.gate_up_proj.weight": "model-00003-of-00004.safetensors",
194
+ "model.layers.21.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
195
+ "model.layers.21.post_mlp_layernorm.weight": "model-00003-of-00004.safetensors",
196
+ "model.layers.21.post_self_attn_layernorm.weight": "model-00003-of-00004.safetensors",
197
+ "model.layers.21.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
198
+ "model.layers.21.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
199
+ "model.layers.21.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
200
+ "model.layers.21.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
201
+ "model.layers.21.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
202
+ "model.layers.21.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
203
+ "model.layers.21.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
204
+ "model.layers.22.input_layernorm.weight": "model-00003-of-00004.safetensors",
205
+ "model.layers.22.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
206
+ "model.layers.22.mlp.gate_up_proj.weight": "model-00003-of-00004.safetensors",
207
+ "model.layers.22.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
208
+ "model.layers.22.post_mlp_layernorm.weight": "model-00003-of-00004.safetensors",
209
+ "model.layers.22.post_self_attn_layernorm.weight": "model-00003-of-00004.safetensors",
210
+ "model.layers.22.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
211
+ "model.layers.22.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
212
+ "model.layers.22.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
213
+ "model.layers.22.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
214
+ "model.layers.22.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
215
+ "model.layers.22.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
216
+ "model.layers.22.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
217
+ "model.layers.23.input_layernorm.weight": "model-00003-of-00004.safetensors",
218
+ "model.layers.23.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
219
+ "model.layers.23.mlp.gate_up_proj.weight": "model-00003-of-00004.safetensors",
220
+ "model.layers.23.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
221
+ "model.layers.23.post_mlp_layernorm.weight": "model-00003-of-00004.safetensors",
222
+ "model.layers.23.post_self_attn_layernorm.weight": "model-00003-of-00004.safetensors",
223
+ "model.layers.23.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
224
+ "model.layers.23.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
225
+ "model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
226
+ "model.layers.23.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
227
+ "model.layers.23.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
228
+ "model.layers.23.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
229
+ "model.layers.23.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
230
+ "model.layers.24.input_layernorm.weight": "model-00003-of-00004.safetensors",
231
+ "model.layers.24.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
232
+ "model.layers.24.mlp.gate_up_proj.weight": "model-00003-of-00004.safetensors",
233
+ "model.layers.24.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
234
+ "model.layers.24.post_mlp_layernorm.weight": "model-00003-of-00004.safetensors",
235
+ "model.layers.24.post_self_attn_layernorm.weight": "model-00003-of-00004.safetensors",
236
+ "model.layers.24.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
237
+ "model.layers.24.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
238
+ "model.layers.24.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
239
+ "model.layers.24.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
240
+ "model.layers.24.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
241
+ "model.layers.24.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
242
+ "model.layers.24.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
243
+ "model.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
244
+ "model.layers.25.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
245
+ "model.layers.25.mlp.gate_up_proj.weight": "model-00003-of-00004.safetensors",
246
+ "model.layers.25.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
247
+ "model.layers.25.post_mlp_layernorm.weight": "model-00003-of-00004.safetensors",
248
+ "model.layers.25.post_self_attn_layernorm.weight": "model-00003-of-00004.safetensors",
249
+ "model.layers.25.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
250
+ "model.layers.25.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
251
+ "model.layers.25.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
252
+ "model.layers.25.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
253
+ "model.layers.25.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
254
+ "model.layers.25.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
255
+ "model.layers.25.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
256
+ "model.layers.26.input_layernorm.weight": "model-00003-of-00004.safetensors",
257
+ "model.layers.26.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
258
+ "model.layers.26.mlp.gate_up_proj.weight": "model-00003-of-00004.safetensors",
259
+ "model.layers.26.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
260
+ "model.layers.26.post_mlp_layernorm.weight": "model-00003-of-00004.safetensors",
261
+ "model.layers.26.post_self_attn_layernorm.weight": "model-00003-of-00004.safetensors",
262
+ "model.layers.26.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
263
+ "model.layers.26.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
264
+ "model.layers.26.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
265
+ "model.layers.26.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
266
+ "model.layers.26.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
267
+ "model.layers.26.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
268
+ "model.layers.26.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
269
+ "model.layers.27.input_layernorm.weight": "model-00003-of-00004.safetensors",
270
+ "model.layers.27.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
271
+ "model.layers.27.mlp.gate_up_proj.weight": "model-00003-of-00004.safetensors",
272
+ "model.layers.27.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
273
+ "model.layers.27.post_mlp_layernorm.weight": "model-00003-of-00004.safetensors",
274
+ "model.layers.27.post_self_attn_layernorm.weight": "model-00003-of-00004.safetensors",
275
+ "model.layers.27.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
276
+ "model.layers.27.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
277
+ "model.layers.27.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
278
+ "model.layers.27.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
279
+ "model.layers.27.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
280
+ "model.layers.27.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
281
+ "model.layers.27.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
282
+ "model.layers.28.input_layernorm.weight": "model-00003-of-00004.safetensors",
283
+ "model.layers.28.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
284
+ "model.layers.28.mlp.gate_up_proj.weight": "model-00003-of-00004.safetensors",
285
+ "model.layers.28.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
286
+ "model.layers.28.post_mlp_layernorm.weight": "model-00003-of-00004.safetensors",
287
+ "model.layers.28.post_self_attn_layernorm.weight": "model-00003-of-00004.safetensors",
288
+ "model.layers.28.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
289
+ "model.layers.28.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
290
+ "model.layers.28.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
291
+ "model.layers.28.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
292
+ "model.layers.28.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
293
+ "model.layers.28.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
294
+ "model.layers.28.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
295
+ "model.layers.29.input_layernorm.weight": "model-00003-of-00004.safetensors",
296
+ "model.layers.29.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
297
+ "model.layers.29.mlp.gate_up_proj.weight": "model-00003-of-00004.safetensors",
298
+ "model.layers.29.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
299
+ "model.layers.29.post_mlp_layernorm.weight": "model-00003-of-00004.safetensors",
300
+ "model.layers.29.post_self_attn_layernorm.weight": "model-00003-of-00004.safetensors",
301
+ "model.layers.29.self_attn.k_proj.bias": "model-00003-of-00004.safetensors",
302
+ "model.layers.29.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
303
+ "model.layers.29.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
304
+ "model.layers.29.self_attn.q_proj.bias": "model-00003-of-00004.safetensors",
305
+ "model.layers.29.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
306
+ "model.layers.29.self_attn.v_proj.bias": "model-00003-of-00004.safetensors",
307
+ "model.layers.29.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
308
+ "model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
309
+ "model.layers.3.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
310
+ "model.layers.3.mlp.gate_up_proj.weight": "model-00001-of-00004.safetensors",
311
+ "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
312
+ "model.layers.3.post_mlp_layernorm.weight": "model-00001-of-00004.safetensors",
313
+ "model.layers.3.post_self_attn_layernorm.weight": "model-00001-of-00004.safetensors",
314
+ "model.layers.3.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
315
+ "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
316
+ "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
317
+ "model.layers.3.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
318
+ "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
319
+ "model.layers.3.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
320
+ "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
321
+ "model.layers.30.input_layernorm.weight": "model-00003-of-00004.safetensors",
322
+ "model.layers.30.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
323
+ "model.layers.30.mlp.gate_up_proj.weight": "model-00004-of-00004.safetensors",
324
+ "model.layers.30.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
325
+ "model.layers.30.post_mlp_layernorm.weight": "model-00004-of-00004.safetensors",
326
+ "model.layers.30.post_self_attn_layernorm.weight": "model-00004-of-00004.safetensors",
327
+ "model.layers.30.self_attn.k_proj.bias": "model-00004-of-00004.safetensors",
328
+ "model.layers.30.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
329
+ "model.layers.30.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
330
+ "model.layers.30.self_attn.q_proj.bias": "model-00004-of-00004.safetensors",
331
+ "model.layers.30.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
332
+ "model.layers.30.self_attn.v_proj.bias": "model-00004-of-00004.safetensors",
333
+ "model.layers.30.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
334
+ "model.layers.31.input_layernorm.weight": "model-00004-of-00004.safetensors",
335
+ "model.layers.31.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
336
+ "model.layers.31.mlp.gate_up_proj.weight": "model-00004-of-00004.safetensors",
337
+ "model.layers.31.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
338
+ "model.layers.31.post_mlp_layernorm.weight": "model-00004-of-00004.safetensors",
339
+ "model.layers.31.post_self_attn_layernorm.weight": "model-00004-of-00004.safetensors",
340
+ "model.layers.31.self_attn.k_proj.bias": "model-00004-of-00004.safetensors",
341
+ "model.layers.31.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
342
+ "model.layers.31.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
343
+ "model.layers.31.self_attn.q_proj.bias": "model-00004-of-00004.safetensors",
344
+ "model.layers.31.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
345
+ "model.layers.31.self_attn.v_proj.bias": "model-00004-of-00004.safetensors",
346
+ "model.layers.31.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
347
+ "model.layers.32.input_layernorm.weight": "model-00004-of-00004.safetensors",
348
+ "model.layers.32.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
349
+ "model.layers.32.mlp.gate_up_proj.weight": "model-00004-of-00004.safetensors",
350
+ "model.layers.32.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
351
+ "model.layers.32.post_mlp_layernorm.weight": "model-00004-of-00004.safetensors",
352
+ "model.layers.32.post_self_attn_layernorm.weight": "model-00004-of-00004.safetensors",
353
+ "model.layers.32.self_attn.k_proj.bias": "model-00004-of-00004.safetensors",
354
+ "model.layers.32.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
355
+ "model.layers.32.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
356
+ "model.layers.32.self_attn.q_proj.bias": "model-00004-of-00004.safetensors",
357
+ "model.layers.32.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
358
+ "model.layers.32.self_attn.v_proj.bias": "model-00004-of-00004.safetensors",
359
+ "model.layers.32.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
360
+ "model.layers.33.input_layernorm.weight": "model-00004-of-00004.safetensors",
361
+ "model.layers.33.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
362
+ "model.layers.33.mlp.gate_up_proj.weight": "model-00004-of-00004.safetensors",
363
+ "model.layers.33.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
364
+ "model.layers.33.post_mlp_layernorm.weight": "model-00004-of-00004.safetensors",
365
+ "model.layers.33.post_self_attn_layernorm.weight": "model-00004-of-00004.safetensors",
366
+ "model.layers.33.self_attn.k_proj.bias": "model-00004-of-00004.safetensors",
367
+ "model.layers.33.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
368
+ "model.layers.33.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
369
+ "model.layers.33.self_attn.q_proj.bias": "model-00004-of-00004.safetensors",
370
+ "model.layers.33.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
371
+ "model.layers.33.self_attn.v_proj.bias": "model-00004-of-00004.safetensors",
372
+ "model.layers.33.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
373
+ "model.layers.34.input_layernorm.weight": "model-00004-of-00004.safetensors",
374
+ "model.layers.34.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
375
+ "model.layers.34.mlp.gate_up_proj.weight": "model-00004-of-00004.safetensors",
376
+ "model.layers.34.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
377
+ "model.layers.34.post_mlp_layernorm.weight": "model-00004-of-00004.safetensors",
378
+ "model.layers.34.post_self_attn_layernorm.weight": "model-00004-of-00004.safetensors",
379
+ "model.layers.34.self_attn.k_proj.bias": "model-00004-of-00004.safetensors",
380
+ "model.layers.34.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
381
+ "model.layers.34.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
382
+ "model.layers.34.self_attn.q_proj.bias": "model-00004-of-00004.safetensors",
383
+ "model.layers.34.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
384
+ "model.layers.34.self_attn.v_proj.bias": "model-00004-of-00004.safetensors",
385
+ "model.layers.34.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
386
+ "model.layers.35.input_layernorm.weight": "model-00004-of-00004.safetensors",
387
+ "model.layers.35.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
388
+ "model.layers.35.mlp.gate_up_proj.weight": "model-00004-of-00004.safetensors",
389
+ "model.layers.35.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
390
+ "model.layers.35.post_mlp_layernorm.weight": "model-00004-of-00004.safetensors",
391
+ "model.layers.35.post_self_attn_layernorm.weight": "model-00004-of-00004.safetensors",
392
+ "model.layers.35.self_attn.k_proj.bias": "model-00004-of-00004.safetensors",
393
+ "model.layers.35.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
394
+ "model.layers.35.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
395
+ "model.layers.35.self_attn.q_proj.bias": "model-00004-of-00004.safetensors",
396
+ "model.layers.35.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
397
+ "model.layers.35.self_attn.v_proj.bias": "model-00004-of-00004.safetensors",
398
+ "model.layers.35.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
399
+ "model.layers.36.input_layernorm.weight": "model-00004-of-00004.safetensors",
400
+ "model.layers.36.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
401
+ "model.layers.36.mlp.gate_up_proj.weight": "model-00004-of-00004.safetensors",
402
+ "model.layers.36.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
403
+ "model.layers.36.post_mlp_layernorm.weight": "model-00004-of-00004.safetensors",
404
+ "model.layers.36.post_self_attn_layernorm.weight": "model-00004-of-00004.safetensors",
405
+ "model.layers.36.self_attn.k_proj.bias": "model-00004-of-00004.safetensors",
406
+ "model.layers.36.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
407
+ "model.layers.36.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
408
+ "model.layers.36.self_attn.q_proj.bias": "model-00004-of-00004.safetensors",
409
+ "model.layers.36.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
410
+ "model.layers.36.self_attn.v_proj.bias": "model-00004-of-00004.safetensors",
411
+ "model.layers.36.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
412
+ "model.layers.37.input_layernorm.weight": "model-00004-of-00004.safetensors",
413
+ "model.layers.37.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
414
+ "model.layers.37.mlp.gate_up_proj.weight": "model-00004-of-00004.safetensors",
415
+ "model.layers.37.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
416
+ "model.layers.37.post_mlp_layernorm.weight": "model-00004-of-00004.safetensors",
417
+ "model.layers.37.post_self_attn_layernorm.weight": "model-00004-of-00004.safetensors",
418
+ "model.layers.37.self_attn.k_proj.bias": "model-00004-of-00004.safetensors",
419
+ "model.layers.37.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
420
+ "model.layers.37.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
421
+ "model.layers.37.self_attn.q_proj.bias": "model-00004-of-00004.safetensors",
422
+ "model.layers.37.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
423
+ "model.layers.37.self_attn.v_proj.bias": "model-00004-of-00004.safetensors",
424
+ "model.layers.37.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
425
+ "model.layers.38.input_layernorm.weight": "model-00004-of-00004.safetensors",
426
+ "model.layers.38.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
427
+ "model.layers.38.mlp.gate_up_proj.weight": "model-00004-of-00004.safetensors",
428
+ "model.layers.38.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
429
+ "model.layers.38.post_mlp_layernorm.weight": "model-00004-of-00004.safetensors",
430
+ "model.layers.38.post_self_attn_layernorm.weight": "model-00004-of-00004.safetensors",
431
+ "model.layers.38.self_attn.k_proj.bias": "model-00004-of-00004.safetensors",
432
+ "model.layers.38.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
433
+ "model.layers.38.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
434
+ "model.layers.38.self_attn.q_proj.bias": "model-00004-of-00004.safetensors",
435
+ "model.layers.38.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
436
+ "model.layers.38.self_attn.v_proj.bias": "model-00004-of-00004.safetensors",
437
+ "model.layers.38.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
438
+ "model.layers.39.input_layernorm.weight": "model-00004-of-00004.safetensors",
439
+ "model.layers.39.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
440
+ "model.layers.39.mlp.gate_up_proj.weight": "model-00004-of-00004.safetensors",
441
+ "model.layers.39.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
442
+ "model.layers.39.post_mlp_layernorm.weight": "model-00004-of-00004.safetensors",
443
+ "model.layers.39.post_self_attn_layernorm.weight": "model-00004-of-00004.safetensors",
444
+ "model.layers.39.self_attn.k_proj.bias": "model-00004-of-00004.safetensors",
445
+ "model.layers.39.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
446
+ "model.layers.39.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
447
+ "model.layers.39.self_attn.q_proj.bias": "model-00004-of-00004.safetensors",
448
+ "model.layers.39.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
449
+ "model.layers.39.self_attn.v_proj.bias": "model-00004-of-00004.safetensors",
450
+ "model.layers.39.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
451
+ "model.layers.4.input_layernorm.weight": "model-00001-of-00004.safetensors",
452
+ "model.layers.4.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
453
+ "model.layers.4.mlp.gate_up_proj.weight": "model-00001-of-00004.safetensors",
454
+ "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
455
+ "model.layers.4.post_mlp_layernorm.weight": "model-00001-of-00004.safetensors",
456
+ "model.layers.4.post_self_attn_layernorm.weight": "model-00001-of-00004.safetensors",
457
+ "model.layers.4.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
458
+ "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
459
+ "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
460
+ "model.layers.4.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
461
+ "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
462
+ "model.layers.4.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
463
+ "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
464
+ "model.layers.5.input_layernorm.weight": "model-00001-of-00004.safetensors",
465
+ "model.layers.5.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
466
+ "model.layers.5.mlp.gate_up_proj.weight": "model-00001-of-00004.safetensors",
467
+ "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
468
+ "model.layers.5.post_mlp_layernorm.weight": "model-00001-of-00004.safetensors",
469
+ "model.layers.5.post_self_attn_layernorm.weight": "model-00001-of-00004.safetensors",
470
+ "model.layers.5.self_attn.k_proj.bias": "model-00001-of-00004.safetensors",
471
+ "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
472
+ "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
473
+ "model.layers.5.self_attn.q_proj.bias": "model-00001-of-00004.safetensors",
474
+ "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
475
+ "model.layers.5.self_attn.v_proj.bias": "model-00001-of-00004.safetensors",
476
+ "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
477
+ "model.layers.6.input_layernorm.weight": "model-00001-of-00004.safetensors",
478
+ "model.layers.6.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
479
+ "model.layers.6.mlp.gate_up_proj.weight": "model-00002-of-00004.safetensors",
480
+ "model.layers.6.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
481
+ "model.layers.6.post_mlp_layernorm.weight": "model-00002-of-00004.safetensors",
482
+ "model.layers.6.post_self_attn_layernorm.weight": "model-00002-of-00004.safetensors",
483
+ "model.layers.6.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
484
+ "model.layers.6.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
485
+ "model.layers.6.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
486
+ "model.layers.6.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
487
+ "model.layers.6.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
488
+ "model.layers.6.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
489
+ "model.layers.6.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
490
+ "model.layers.7.input_layernorm.weight": "model-00002-of-00004.safetensors",
491
+ "model.layers.7.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
492
+ "model.layers.7.mlp.gate_up_proj.weight": "model-00002-of-00004.safetensors",
493
+ "model.layers.7.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
494
+ "model.layers.7.post_mlp_layernorm.weight": "model-00002-of-00004.safetensors",
495
+ "model.layers.7.post_self_attn_layernorm.weight": "model-00002-of-00004.safetensors",
496
+ "model.layers.7.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
497
+ "model.layers.7.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
498
+ "model.layers.7.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
499
+ "model.layers.7.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
500
+ "model.layers.7.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
501
+ "model.layers.7.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
502
+ "model.layers.7.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
503
+ "model.layers.8.input_layernorm.weight": "model-00002-of-00004.safetensors",
504
+ "model.layers.8.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
505
+ "model.layers.8.mlp.gate_up_proj.weight": "model-00002-of-00004.safetensors",
506
+ "model.layers.8.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
507
+ "model.layers.8.post_mlp_layernorm.weight": "model-00002-of-00004.safetensors",
508
+ "model.layers.8.post_self_attn_layernorm.weight": "model-00002-of-00004.safetensors",
509
+ "model.layers.8.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
510
+ "model.layers.8.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
511
+ "model.layers.8.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
512
+ "model.layers.8.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
513
+ "model.layers.8.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
514
+ "model.layers.8.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
515
+ "model.layers.8.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
516
+ "model.layers.9.input_layernorm.weight": "model-00002-of-00004.safetensors",
517
+ "model.layers.9.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
518
+ "model.layers.9.mlp.gate_up_proj.weight": "model-00002-of-00004.safetensors",
519
+ "model.layers.9.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
520
+ "model.layers.9.post_mlp_layernorm.weight": "model-00002-of-00004.safetensors",
521
+ "model.layers.9.post_self_attn_layernorm.weight": "model-00002-of-00004.safetensors",
522
+ "model.layers.9.self_attn.k_proj.bias": "model-00002-of-00004.safetensors",
523
+ "model.layers.9.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
524
+ "model.layers.9.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
525
+ "model.layers.9.self_attn.q_proj.bias": "model-00002-of-00004.safetensors",
526
+ "model.layers.9.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
527
+ "model.layers.9.self_attn.v_proj.bias": "model-00002-of-00004.safetensors",
528
+ "model.layers.9.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
529
+ "model.norm.weight": "model-00004-of-00004.safetensors"
530
+ }
531
+ }
special_tokens_map.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ "<|endoftext|>",
4
+ "[MASK]",
5
+ "[gMASK]",
6
+ "[sMASK]",
7
+ "<sop>",
8
+ "<eop>",
9
+ "<|system|>",
10
+ "<|user|>",
11
+ "<|assistant|>",
12
+ "<|observation|>",
13
+ "<|begin_of_image|>",
14
+ "<|end_of_image|>",
15
+ "<|begin_of_video|>",
16
+ "<|end_of_video|>"
17
+ ],
18
+ "eos_token": {
19
+ "content": "<|endoftext|>",
20
+ "lstrip": false,
21
+ "normalized": false,
22
+ "rstrip": false,
23
+ "single_word": false
24
+ },
25
+ "pad_token": {
26
+ "content": "<|endoftext|>",
27
+ "lstrip": false,
28
+ "normalized": false,
29
+ "rstrip": false,
30
+ "single_word": false
31
+ }
32
+ }
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:76ebeac0d8bd7879ead7b43c16b44981f277e47225de2bd7de9ae1a6cc664a8c
3
+ size 19966496
tokenizer_config.json ADDED
@@ -0,0 +1,146 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "151329": {
4
+ "content": "<|endoftext|>",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false,
9
+ "special": true
10
+ },
11
+ "151330": {
12
+ "content": "[MASK]",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false,
17
+ "special": true
18
+ },
19
+ "151331": {
20
+ "content": "[gMASK]",
21
+ "lstrip": false,
22
+ "normalized": false,
23
+ "rstrip": false,
24
+ "single_word": false,
25
+ "special": true
26
+ },
27
+ "151332": {
28
+ "content": "[sMASK]",
29
+ "lstrip": false,
30
+ "normalized": false,
31
+ "rstrip": false,
32
+ "single_word": false,
33
+ "special": true
34
+ },
35
+ "151333": {
36
+ "content": "<sop>",
37
+ "lstrip": false,
38
+ "normalized": false,
39
+ "rstrip": false,
40
+ "single_word": false,
41
+ "special": true
42
+ },
43
+ "151334": {
44
+ "content": "<eop>",
45
+ "lstrip": false,
46
+ "normalized": false,
47
+ "rstrip": false,
48
+ "single_word": false,
49
+ "special": true
50
+ },
51
+ "151335": {
52
+ "content": "<|system|>",
53
+ "lstrip": false,
54
+ "normalized": false,
55
+ "rstrip": false,
56
+ "single_word": false,
57
+ "special": true
58
+ },
59
+ "151336": {
60
+ "content": "<|user|>",
61
+ "lstrip": false,
62
+ "normalized": false,
63
+ "rstrip": false,
64
+ "single_word": false,
65
+ "special": true
66
+ },
67
+ "151337": {
68
+ "content": "<|assistant|>",
69
+ "lstrip": false,
70
+ "normalized": false,
71
+ "rstrip": false,
72
+ "single_word": false,
73
+ "special": true
74
+ },
75
+ "151338": {
76
+ "content": "<|observation|>",
77
+ "lstrip": false,
78
+ "normalized": false,
79
+ "rstrip": false,
80
+ "single_word": false,
81
+ "special": true
82
+ },
83
+ "151339": {
84
+ "content": "<|begin_of_image|>",
85
+ "lstrip": false,
86
+ "normalized": false,
87
+ "rstrip": false,
88
+ "single_word": false,
89
+ "special": true
90
+ },
91
+ "151340": {
92
+ "content": "<|end_of_image|>",
93
+ "lstrip": false,
94
+ "normalized": false,
95
+ "rstrip": false,
96
+ "single_word": false,
97
+ "special": true
98
+ },
99
+ "151341": {
100
+ "content": "<|begin_of_video|>",
101
+ "lstrip": false,
102
+ "normalized": false,
103
+ "rstrip": false,
104
+ "single_word": false,
105
+ "special": true
106
+ },
107
+ "151342": {
108
+ "content": "<|end_of_video|>",
109
+ "lstrip": false,
110
+ "normalized": false,
111
+ "rstrip": false,
112
+ "single_word": false,
113
+ "special": true
114
+ }
115
+ },
116
+ "additional_special_tokens": [
117
+ "<|endoftext|>",
118
+ "[MASK]",
119
+ "[gMASK]",
120
+ "[sMASK]",
121
+ "<sop>",
122
+ "<eop>",
123
+ "<|system|>",
124
+ "<|user|>",
125
+ "<|assistant|>",
126
+ "<|observation|>",
127
+ "<|begin_of_image|>",
128
+ "<|end_of_image|>",
129
+ "<|begin_of_video|>",
130
+ "<|end_of_video|>"
131
+ ],
132
+ "chat_template": "[gMASK]<sop>{%- if tools -%}<|system|>\n# 可用工具\n{% for tool in tools %}{%- set function = tool.function if tool.get(\"function\") else tool %}\n\n## {{ function.name }}\n\n{{ function | tojson(indent=4, ensure_ascii=False) }}\n在调用上述函数时,请使用 Json 格式表示调用的参数。{%- endfor %}{%- endif -%}{%- for msg in messages %}{%- if msg.role == 'system' %}<|system|>\n{{ msg.content }}{%- endif %}{%- endfor %}{%- for message in messages if message.role != 'system' %}{%- set role = message['role'] %}{%- set content = message['content'] %}{%- set meta = message.get(\"metadata\", \"\") %}{%- if role == 'user' %}<|user|>\n{{ content }}{%- elif role == 'assistant' and not meta %}<|assistant|>\n{{ content }}{%- elif role == 'assistant' and meta %}<|assistant|>{{ meta }} \n{{ content }}{%- elif role == 'observation' %}<|observation|>\n{{ content }}{%- endif %}{%- endfor %}{% if add_generation_prompt %}<|assistant|>{% endif %}",
133
+ "clean_up_tokenization_spaces": false,
134
+ "do_lower_case": false,
135
+ "eos_token": "<|user|>",
136
+ "extra_special_tokens": {},
137
+ "model_input_names": [
138
+ "input_ids",
139
+ "attention_mask"
140
+ ],
141
+ "model_max_length": 128000,
142
+ "pad_token": "<|endoftext|>",
143
+ "padding_side": "left",
144
+ "remove_space": false,
145
+ "tokenizer_class": "PreTrainedTokenizer"
146
+ }