Vveerrgg commited on
Commit
0cb6244
·
verified ·
1 Parent(s): 14e8868

v8 (v8c checkpoint 220, merged bf16)

Browse files
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,197 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: Qwen/Qwen3-8B
4
+ base_model_relation: finetune
5
+ library_name: transformers
6
+ pipeline_tag: text-generation
7
+ language: [en, es, pt, it, fr, nl, de, ru, bn, id, fa, ja, pl, sv, vi, tl, he, ml, tr, zh, ms, th, fi, ar, ta, hi]
8
+ tags: [cooking, recipes, lora, dpo, qwen3, provenance, abstention]
9
+ datasets: [HumanjavaEnterprises/cooking-qa-v8]
10
+ ---
11
+
12
+ # Qwen3-8B Cooking v8
13
+
14
+ A multilingual cooking assistant fine-tuned from Qwen3-8B on public-domain and openly licensed cookbooks, as
15
+ full-precision (bf16) safetensors for Transformers and vLLM. For llama.cpp, Ollama and LM Studio:
16
+ [HumanjavaEnterprises/Qwen3-8B-Cooking-v8-GGUF](https://huggingface.co/HumanjavaEnterprises/Qwen3-8B-Cooking-v8-GGUF).
17
+
18
+ It answers from the recipe you give it, asks for the recipe when an answer depends on one it cannot see, and gives
19
+ full answers to general cooking questions. It does not memorise recipes: give it the recipe (paste it, or have your
20
+ app look it up) and it will read it.
21
+
22
+ ## Files
23
+
24
+ The two training stages (supervised LoRA, then a preference LoRA on top) are merged into one bf16 model, the same
25
+ architecture and tokenizer as Qwen3-8B (~16 GB). The GGUF files in the sibling repo are converted from these weights.
26
+
27
+ ## Quick start
28
+
29
+ ```bash
30
+ # vLLM
31
+ vllm serve HumanjavaEnterprises/Qwen3-8B-Cooking-v8 --max-model-len 8192
32
+ # then send requests with "chat_template_kwargs": {"enable_thinking": false}
33
+ ```
34
+
35
+ ```python
36
+ # Transformers
37
+ from transformers import AutoModelForCausalLM, AutoTokenizer
38
+ repo = "HumanjavaEnterprises/Qwen3-8B-Cooking-v8"
39
+ tok = AutoTokenizer.from_pretrained(repo)
40
+ model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype="auto", device_map="auto")
41
+ messages = [{"role": "system", "content": "You are a cooking assistant. You answer questions about dishes, ingredients, techniques and food safety. When you are not sure, you say so."},
42
+ {"role": "user", "content": "If I double the recipe, how much pepper do I need?"}]
43
+ ids = tok.apply_chat_template(messages, add_generation_prompt=True, enable_thinking=False, return_tensors="pt").to(model.device)
44
+ print(tok.decode(model.generate(ids, max_new_tokens=400)[0][ids.shape[1]:], skip_special_tokens=True))
45
+ ```
46
+
47
+ ## Prompt format
48
+
49
+ Qwen3 chat template, **thinking off** (`enable_thinking: false`, or `/no_think` in the user turn). Trained with this
50
+ system prompt, which works best:
51
+
52
+ ```
53
+ You are a cooking assistant. You answer questions about dishes, ingredients, techniques and food safety. When you are not sure, you say so.
54
+ ```
55
+
56
+ To ask about a specific recipe, put the recipe text in the user turn, then the question.
57
+
58
+ ## What it does
59
+
60
+ The fine-tune teaches **behaviour, not facts**:
61
+
62
+ - **Reads the recipe you give it.** Paste a recipe and ask about it; it answers from that text.
63
+ - **Asks when the recipe is missing.** "If I double the recipe, how much pepper?" with no recipe gets a request for the
64
+ original quantity, not an invented number.
65
+ - **Answers general cooking questions in full:** food safety, substitutions, why a dish failed.
66
+
67
+ The intended setup is **LoRA for behaviour, retrieval for facts, tools for numbers**: an app looks up the recipe and
68
+ hands it to the model; scaling and unit conversion are calculated, not guessed.
69
+
70
+ It is also a worked example of a small custom model built carefully: open weights, open or public-domain sources
71
+ only, every training record traceable to its source, and the checks published along with the failures they caught.
72
+
73
+ ## Results
74
+
75
+ All cooking scores are on a fixed held-out set of 682 questions (300 with the recipe shown, 300 that need a recipe
76
+ the model is not shown, 82 general questions), in 26 languages, never used in training. Answers from base Qwen3-8B
77
+ (thinking off) and this model are compared by an open-weights judge from a different model family
78
+ (DeepSeek-V4-Flash, MIT licence), blind, in both orders. 95% confidence intervals in brackets.
79
+
80
+ | | Qwen3-8B | this model |
81
+ |---|---|---|
82
+ | **Invents recipe details when no recipe is shown** | 30.0% [25.1, 35.4] | **9.0%** [6.3, 12.8] |
83
+ | same, after a rule check that removes judge false alarms | 27.7% | **4.0%** [2.3, 6.9] |
84
+ | **Correct when the recipe is shown** | 84.7% | **93.0%** |
85
+ | Preferred by the judge, all 682 questions | — | **0.610** [0.581, 0.639] (0.50 = tie) |
86
+ | Preferred by the judge, general questions only | — | 0.460 [0.369, 0.549] (level with base) |
87
+
88
+ The drop in invented details is significant (McNemar p < 1e-10; 78 questions where only the base invents, 15 where
89
+ only this model does).
90
+
91
+ **Four judges.** To check the judge, three more open-weights judges from different families scored the same answers
92
+ one at a time: Qwen3.8-27B (Apache-2.0), GLM-4.7-Flash (MIT) and gpt-oss-120b (Apache-2.0). With the same rule check
93
+ applied to all four, the share of no-recipe answers that invent recipe details:
94
+
95
+ | counted as "invents" when | Qwen3-8B | this model |
96
+ |---|---|---|
97
+ | at least 3 of 4 judges say so | 25.8% [21.1, 31.0] | **2.7%** [1.4, 5.2] |
98
+ | all four say so | 19.1% | 1.0% |
99
+ | any one says so | 49.2% | 18.3% |
100
+
101
+ Every judge, and every way of combining them, shows a significant drop (3 of 4: 72 questions where only the base
102
+ invents, 3 where only this model does, McNemar p < 1e-17). The judges agree moderately on base answers (Fleiss kappa
103
+ 0.63). GLM-4.7-Flash flags more of this model's answers than the others; reading them, most repeat or restate a line
104
+ the user quoted, which the rubric says is not inventing, and the rule check now clears those for every judge. Qwen3.8-27B also labelled this model's preference training data, so its
105
+ read of this model's answers is not fully independent. Model judges also tend to make the same mistakes as each
106
+ other, so four agreeing is less than four independent checks. Most of the remaining "invents" cases are borderline: a quantity that appears in the question
107
+ itself, or a common default ("a pinch of salt") offered with a hedge.
108
+
109
+ **Quantised files** were scored on the same 682 questions against the full-precision model (0.50 = identical quality):
110
+
111
+ | | Q4_K_M | Q6_K | Q8_0 |
112
+ |---|---|---|---|
113
+ | Judge preference vs full precision | 0.492 [0.475, 0.509] | 0.501 [0.485, 0.517] | 0.481 [0.466, 0.494] |
114
+ | Correct when the recipe is shown | 94.0% | 93.3% | 93.7% |
115
+ | Invents with no recipe (full precision on that run) | 9.3% (7.3%) | 6.3% (6.3%) | 6.7% (6.7%) |
116
+ | Invents with no recipe, 3 of 4 judges (full precision) | 4.7% (3.3%) | 3.0% (3.3%) | 3.3% (2.7%) |
117
+ | Correct when the recipe is shown, 3 of 4 judges (full precision) | 81.4% (80.4%) | 82.7% (80.6%) | 80.4% (80.6%) |
118
+
119
+ With all four judges, no quantised file differs significantly from full precision (McNemar on the 3-of-4 verdict:
120
+ Q4_K_M p = 0.34, Q6_K p = 1.0, Q8_0 p = 0.69 for inventing). On Q4_K_M two of the four judges on their own
121
+ (GLM-4.7-Flash, gpt-oss-120b) do see slightly more invented details (p = 0.02 and 0.04); pick Q6_K or Q8_0 if that matters.
122
+
123
+ No quantisation changes the invents rate significantly (McNemar p ≥ 0.29). Q8_0's small dip comes from the 82
124
+ general questions (0.418) and is most likely judge noise on a small set; Q4_K_M and Q6_K, which remove more bits,
125
+ sit at parity.
126
+
127
+ **Memorisation.** 1,000 CC BY-SA training documents (Wikibooks) were probed 12,000 ways (prefixes of 32 to 128
128
+ tokens at three offsets, raw and chat prompts, greedy decoding). The model never continues a document verbatim (no
129
+ 50-token match, longest overlap 67 characters vs the base model's 52), and overlap with the true text is no higher
130
+ than base Qwen3-8B's (ROUGE-L 0.169 vs 0.177).
131
+
132
+ **General ability** (lm-eval 0.4.13, thinking off; MMLU-Pro on 100 questions per subject, so ±1.3 points is noise):
133
+
134
+ | | IFEval | GSM8K | TruthfulQA mc2 | MMLU-Pro | MGSM (8 lang) | Belebele (21 lang) |
135
+ |---|---|---|---|---|---|---|
136
+ | Qwen3-8B | 0.817 | 0.807 | 0.532 | 0.611 | 0.341 | 0.846 |
137
+ | this model | 0.800 | 0.866 | 0.542 | 0.591 | 0.401 | 0.863 |
138
+
139
+ MMLU-Pro is 2.0 points down, the largest loss. The GSM8K and MGSM gains most likely reflect cleaner final-answer
140
+ formatting learned from the chat data, not better maths. Belebele (reading comprehension) is within 2 points of the
141
+ base in all 21 languages.
142
+
143
+ **Refusals** (XSTest, 250 safe and 200 unsafe prompts). Without a system prompt it behaves like the base (refuses
144
+ 5.2% of safe prompts vs 4.4%). With the cooking system prompt, the base refuses 38.4% of safe prompts as off-topic;
145
+ this model refuses 18.4% and answers or redirects the rest. We read every unsafe prompt the base refused and this
146
+ model did not: all were deflections or explicit rebuttals, none complied.
147
+
148
+ ## How it was built
149
+
150
+ | Step | What |
151
+ |---|---|
152
+ | Sources | ~46k cooking passages after filtering: Wikibooks Cookbook in 26 languages (CC BY-SA 4.0), public-domain cookbooks from Project Gutenberg and the Internet Archive (scans re-OCR'd with olmOCR-2), NHLBI recipes (US government, public domain). Each record carries source URL, licence and public-domain basis. About a quarter of the training records are English |
153
+ | Excluded | Traditional and Indigenous knowledge, sacred material, outsider-authored books about other cultures, dog-meat recipes, items without a verified licence |
154
+ | Questions and answers | Written from each passage by an open-weights teacher (Qwen3.8-27B), checked against the passage by an open-weights judge. **No output from any closed or API model is in the training, evaluation or seed data** |
155
+ | Stage 1: supervised (LoRA) | 56,919 examples. Recipe-dependent questions carry their recipe in the prompt; 6,649 "no recipe given" examples teach asking for the recipe, filtered by the teacher; answers to general questions rewritten by base Qwen3-8B in its own words (self-distillation) so they keep the base model's depth. Half of trained tokens are general chat (base Qwen3-8B's own answers plus NVIDIA Nemotron-Instruction-Following-Chat-v1 and Hugging Face SmolTalk2 subsets) to limit forgetting. LoRA rank 32 on all linear layers, lr 1e-4, one epoch, merged into the base |
156
+ | Stage 2: preference (DPO) | 3,508 preference pairs built on-policy: the stage-1 model wrote four answers per training prompt, the teacher labelled them, and pairs prefer asking over inventing, correct over wrong when the recipe is shown, and full practical answers over thin ones for general questions. DPO with an SFT term on the chosen answer, beta 0.1, label smoothing 0.1, lr 1e-5, new LoRA rank 32, checkpoint at step 220, merged |
157
+ | Tooling | LLaMA-Factory 0.9.5, vLLM, llama.cpp |
158
+
159
+ The cooking training data (the stage-1 examples and the stage-2 preference pairs) is published with per-record
160
+ attribution: [HumanjavaEnterprises/cooking-qa-v8](https://huggingface.co/datasets/HumanjavaEnterprises/cooking-qa-v8).
161
+ The general chat data is not republished (it is third-party).
162
+
163
+ ## What went wrong along the way
164
+
165
+ - **The first model invented recipe details.** Questions had been written while looking at a recipe, then trained
166
+ without it, so the model learned to answer as if it could see one. Benchmarks did not catch it; reading the answers
167
+ did. Fix: the recipe goes in the prompt, plus examples of asking when it is missing.
168
+ - **Then the general answers got thin.** Teaching it to ask made general answers short (median 440 characters vs the
169
+ base's 1,223), and lowering the adapter strength brought depth back but lost the asking. Fix: let the base model
170
+ rewrite general answers in its own words, then a preference stage on the model's own outputs.
171
+ - **A benchmark that looked broken was.** Belebele scored by log-likelihood put the base model at chance (it always
172
+ picked "A"). We rescored it with generated letter answers.
173
+
174
+ ## Limitations
175
+
176
+ - **Not a source of truth for food safety.** It repeats standard guidance (e.g. 74 °C / 165 °F for poultry) but can be
177
+ wrong; check official sources.
178
+ - **It does not know historical recipes by heart.** It was trained on old cookbooks to read and reason about recipes,
179
+ not to recall them. For a specific historical dish, give it the text.
180
+ - The judges are models. Four open judges from four families agree on the direction and size of the drop, but no
181
+ human-labelled set backs the exact rates yet.
182
+ - Still invents on about 1 in 25 to 1 in 11 no-recipe questions, most often on substitutions.
183
+ - Strongest in English and Spanish; Thai, Tamil and Hindi had under 200 source records each.
184
+ - The old public-domain cookbooks reflect their era: measures, techniques, some outdated advice.
185
+
186
+ ## Licence and credits
187
+
188
+ Weights: Apache-2.0, same as the base model. Base model: Qwen3-8B by the Qwen team (Apache-2.0).
189
+ Training data sources: Wikibooks Cookbook contributors (CC BY-SA 4.0; per-page attribution in the dataset),
190
+ Project Gutenberg and Internet Archive public-domain books, NHLBI. General chat data: NVIDIA
191
+ (Nemotron-Instruction-Following-Chat-v1, ODC-BY 1.0, responses by GPT-OSS-120B and Qwen3-235B-A22B), Hugging Face
192
+ (SmolTalk2, Apache-2.0, responses by Qwen3-32B; prompts from SystemChat-2.0, the Aya Dataset by Cohere For AI, and
193
+ Table-GPT by Microsoft). Teacher: Qwen3.8-27B (Apache-2.0). Judge: DeepSeek-V4-Flash (MIT). OCR: olmOCR-2 by Ai2
194
+ (Apache-2.0). Evaluation: lm-evaluation-harness, XSTest (CC BY 4.0), Belebele (CC BY-SA 4.0). Training: LLaMA-Factory.
195
+ Quantisation: llama.cpp.
196
+
197
+ Built by [Humanjava Enterprises](https://huggingface.co/HumanjavaEnterprises).
SHA256SUMS ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ b54b0967579f79aed6926731fe970e33342bbb2b36cbf1b440cc7bcd9764c36a model-00001-of-00004.safetensors
2
+ 7961330a2e3eb3f94bc4c2b91d83c1131df661cf13ab736202a9866f38f670f3 model-00002-of-00004.safetensors
3
+ 23d462da889f99122d3260b9947db77cfd43bbd166d386af8e5282eca4775a6b model-00003-of-00004.safetensors
4
+ 959cb0e1fe484aaeaa0d5572536a20ba812e53be4bcced33ad20daccbbbcbdcd model-00004-of-00004.safetensors
chat_template.jinja ADDED
@@ -0,0 +1,89 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {%- if tools %}
2
+ {{- '<|im_start|>system\n' }}
3
+ {%- if messages[0].role == 'system' %}
4
+ {{- messages[0].content + '\n\n' }}
5
+ {%- endif %}
6
+ {{- "# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
7
+ {%- for tool in tools %}
8
+ {{- "\n" }}
9
+ {{- tool | tojson }}
10
+ {%- endfor %}
11
+ {{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
12
+ {%- else %}
13
+ {%- if messages[0].role == 'system' %}
14
+ {{- '<|im_start|>system\n' + messages[0].content + '<|im_end|>\n' }}
15
+ {%- endif %}
16
+ {%- endif %}
17
+ {%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
18
+ {%- for message in messages[::-1] %}
19
+ {%- set index = (messages|length - 1) - loop.index0 %}
20
+ {%- if ns.multi_step_tool and message.role == "user" and message.content is string and not(message.content.startswith('<tool_response>') and message.content.endswith('</tool_response>')) %}
21
+ {%- set ns.multi_step_tool = false %}
22
+ {%- set ns.last_query_index = index %}
23
+ {%- endif %}
24
+ {%- endfor %}
25
+ {%- for message in messages %}
26
+ {%- if message.content is string %}
27
+ {%- set content = message.content %}
28
+ {%- else %}
29
+ {%- set content = '' %}
30
+ {%- endif %}
31
+ {%- if (message.role == "user") or (message.role == "system" and not loop.first) %}
32
+ {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
33
+ {%- elif message.role == "assistant" %}
34
+ {%- set reasoning_content = '' %}
35
+ {%- if message.reasoning_content is string %}
36
+ {%- set reasoning_content = message.reasoning_content %}
37
+ {%- else %}
38
+ {%- if '</think>' in content %}
39
+ {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
40
+ {%- set content = content.split('</think>')[-1].lstrip('\n') %}
41
+ {%- endif %}
42
+ {%- endif %}
43
+ {%- if loop.index0 > ns.last_query_index %}
44
+ {%- if loop.last or (not loop.last and reasoning_content) %}
45
+ {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content.strip('\n') + '\n</think>\n\n' + content.lstrip('\n') }}
46
+ {%- else %}
47
+ {{- '<|im_start|>' + message.role + '\n' + content }}
48
+ {%- endif %}
49
+ {%- else %}
50
+ {{- '<|im_start|>' + message.role + '\n' + content }}
51
+ {%- endif %}
52
+ {%- if message.tool_calls %}
53
+ {%- for tool_call in message.tool_calls %}
54
+ {%- if (loop.first and content) or (not loop.first) %}
55
+ {{- '\n' }}
56
+ {%- endif %}
57
+ {%- if tool_call.function %}
58
+ {%- set tool_call = tool_call.function %}
59
+ {%- endif %}
60
+ {{- '<tool_call>\n{"name": "' }}
61
+ {{- tool_call.name }}
62
+ {{- '", "arguments": ' }}
63
+ {%- if tool_call.arguments is string %}
64
+ {{- tool_call.arguments }}
65
+ {%- else %}
66
+ {{- tool_call.arguments | tojson }}
67
+ {%- endif %}
68
+ {{- '}\n</tool_call>' }}
69
+ {%- endfor %}
70
+ {%- endif %}
71
+ {{- '<|im_end|>\n' }}
72
+ {%- elif message.role == "tool" %}
73
+ {%- if loop.first or (messages[loop.index0 - 1].role != "tool") %}
74
+ {{- '<|im_start|>user' }}
75
+ {%- endif %}
76
+ {{- '\n<tool_response>\n' }}
77
+ {{- content }}
78
+ {{- '\n</tool_response>' }}
79
+ {%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
80
+ {{- '<|im_end|>\n' }}
81
+ {%- endif %}
82
+ {%- endif %}
83
+ {%- endfor %}
84
+ {%- if add_generation_prompt %}
85
+ {{- '<|im_start|>assistant\n' }}
86
+ {%- if enable_thinking is defined and enable_thinking is false %}
87
+ {{- '<think>\n\n</think>\n\n' }}
88
+ {%- endif %}
89
+ {%- endif %}
config.json ADDED
@@ -0,0 +1,71 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Qwen3ForCausalLM"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 151643,
8
+ "dtype": "bfloat16",
9
+ "eos_token_id": 151645,
10
+ "head_dim": 128,
11
+ "hidden_act": "silu",
12
+ "hidden_size": 4096,
13
+ "initializer_range": 0.02,
14
+ "intermediate_size": 12288,
15
+ "layer_types": [
16
+ "full_attention",
17
+ "full_attention",
18
+ "full_attention",
19
+ "full_attention",
20
+ "full_attention",
21
+ "full_attention",
22
+ "full_attention",
23
+ "full_attention",
24
+ "full_attention",
25
+ "full_attention",
26
+ "full_attention",
27
+ "full_attention",
28
+ "full_attention",
29
+ "full_attention",
30
+ "full_attention",
31
+ "full_attention",
32
+ "full_attention",
33
+ "full_attention",
34
+ "full_attention",
35
+ "full_attention",
36
+ "full_attention",
37
+ "full_attention",
38
+ "full_attention",
39
+ "full_attention",
40
+ "full_attention",
41
+ "full_attention",
42
+ "full_attention",
43
+ "full_attention",
44
+ "full_attention",
45
+ "full_attention",
46
+ "full_attention",
47
+ "full_attention",
48
+ "full_attention",
49
+ "full_attention",
50
+ "full_attention",
51
+ "full_attention"
52
+ ],
53
+ "max_position_embeddings": 40960,
54
+ "max_window_layers": 36,
55
+ "model_type": "qwen3",
56
+ "num_attention_heads": 32,
57
+ "num_hidden_layers": 36,
58
+ "num_key_value_heads": 8,
59
+ "pad_token_id": null,
60
+ "rms_norm_eps": 1e-06,
61
+ "rope_parameters": {
62
+ "rope_theta": 1000000,
63
+ "rope_type": "default"
64
+ },
65
+ "sliding_window": null,
66
+ "tie_word_embeddings": false,
67
+ "transformers_version": "5.6.0",
68
+ "use_cache": true,
69
+ "use_sliding_window": false,
70
+ "vocab_size": 151936
71
+ }
generation_config.json ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": 151643,
3
+ "do_sample": true,
4
+ "eos_token_id": [
5
+ 151645,
6
+ 151643
7
+ ],
8
+ "pad_token_id": 151643,
9
+ "temperature": 0.6,
10
+ "top_k": 20,
11
+ "top_p": 0.95,
12
+ "transformers_version": "4.51.0"
13
+ }
model-00001-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b54b0967579f79aed6926731fe970e33342bbb2b36cbf1b440cc7bcd9764c36a
3
+ size 4905356144
model-00002-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7961330a2e3eb3f94bc4c2b91d83c1131df661cf13ab736202a9866f38f670f3
3
+ size 4915960328
model-00003-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:23d462da889f99122d3260b9947db77cfd43bbd166d386af8e5282eca4775a6b
3
+ size 4974672216
model-00004-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:959cb0e1fe484aaeaa0d5572536a20ba812e53be4bcced33ad20daccbbbcbdcd
3
+ size 1585528088
model.safetensors.index.json ADDED
@@ -0,0 +1,407 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "metadata": {
3
+ "total_parameters": 8190735360,
4
+ "total_size": 16381470720
5
+ },
6
+ "weight_map": {
7
+ "lm_head.weight": "model-00001-of-00004.safetensors",
8
+ "model.embed_tokens.weight": "model-00001-of-00004.safetensors",
9
+ "model.layers.0.input_layernorm.weight": "model-00001-of-00004.safetensors",
10
+ "model.layers.0.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
11
+ "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
12
+ "model.layers.0.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
13
+ "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
14
+ "model.layers.0.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
15
+ "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
16
+ "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
17
+ "model.layers.0.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
18
+ "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
19
+ "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
20
+ "model.layers.1.input_layernorm.weight": "model-00001-of-00004.safetensors",
21
+ "model.layers.1.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
22
+ "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
23
+ "model.layers.1.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
24
+ "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
25
+ "model.layers.1.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
26
+ "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
27
+ "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
28
+ "model.layers.1.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
29
+ "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
30
+ "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
31
+ "model.layers.10.input_layernorm.weight": "model-00002-of-00004.safetensors",
32
+ "model.layers.10.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
33
+ "model.layers.10.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
34
+ "model.layers.10.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
35
+ "model.layers.10.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
36
+ "model.layers.10.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
37
+ "model.layers.10.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
38
+ "model.layers.10.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
39
+ "model.layers.10.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
40
+ "model.layers.10.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
41
+ "model.layers.10.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
42
+ "model.layers.11.input_layernorm.weight": "model-00002-of-00004.safetensors",
43
+ "model.layers.11.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
44
+ "model.layers.11.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
45
+ "model.layers.11.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
46
+ "model.layers.11.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
47
+ "model.layers.11.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
48
+ "model.layers.11.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
49
+ "model.layers.11.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
50
+ "model.layers.11.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
51
+ "model.layers.11.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
52
+ "model.layers.11.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
53
+ "model.layers.12.input_layernorm.weight": "model-00002-of-00004.safetensors",
54
+ "model.layers.12.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
55
+ "model.layers.12.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
56
+ "model.layers.12.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
57
+ "model.layers.12.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
58
+ "model.layers.12.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
59
+ "model.layers.12.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
60
+ "model.layers.12.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
61
+ "model.layers.12.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
62
+ "model.layers.12.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
63
+ "model.layers.12.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
64
+ "model.layers.13.input_layernorm.weight": "model-00002-of-00004.safetensors",
65
+ "model.layers.13.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
66
+ "model.layers.13.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
67
+ "model.layers.13.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
68
+ "model.layers.13.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
69
+ "model.layers.13.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
70
+ "model.layers.13.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
71
+ "model.layers.13.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
72
+ "model.layers.13.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
73
+ "model.layers.13.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
74
+ "model.layers.13.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
75
+ "model.layers.14.input_layernorm.weight": "model-00002-of-00004.safetensors",
76
+ "model.layers.14.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
77
+ "model.layers.14.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
78
+ "model.layers.14.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
79
+ "model.layers.14.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
80
+ "model.layers.14.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
81
+ "model.layers.14.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
82
+ "model.layers.14.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
83
+ "model.layers.14.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
84
+ "model.layers.14.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
85
+ "model.layers.14.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
86
+ "model.layers.15.input_layernorm.weight": "model-00002-of-00004.safetensors",
87
+ "model.layers.15.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
88
+ "model.layers.15.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
89
+ "model.layers.15.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
90
+ "model.layers.15.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
91
+ "model.layers.15.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
92
+ "model.layers.15.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
93
+ "model.layers.15.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
94
+ "model.layers.15.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
95
+ "model.layers.15.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
96
+ "model.layers.15.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
97
+ "model.layers.16.input_layernorm.weight": "model-00002-of-00004.safetensors",
98
+ "model.layers.16.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
99
+ "model.layers.16.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
100
+ "model.layers.16.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
101
+ "model.layers.16.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
102
+ "model.layers.16.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
103
+ "model.layers.16.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
104
+ "model.layers.16.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
105
+ "model.layers.16.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
106
+ "model.layers.16.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
107
+ "model.layers.16.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
108
+ "model.layers.17.input_layernorm.weight": "model-00002-of-00004.safetensors",
109
+ "model.layers.17.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
110
+ "model.layers.17.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
111
+ "model.layers.17.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
112
+ "model.layers.17.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
113
+ "model.layers.17.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
114
+ "model.layers.17.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
115
+ "model.layers.17.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
116
+ "model.layers.17.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
117
+ "model.layers.17.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
118
+ "model.layers.17.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
119
+ "model.layers.18.input_layernorm.weight": "model-00002-of-00004.safetensors",
120
+ "model.layers.18.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
121
+ "model.layers.18.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
122
+ "model.layers.18.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
123
+ "model.layers.18.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
124
+ "model.layers.18.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
125
+ "model.layers.18.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
126
+ "model.layers.18.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
127
+ "model.layers.18.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
128
+ "model.layers.18.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
129
+ "model.layers.18.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
130
+ "model.layers.19.input_layernorm.weight": "model-00002-of-00004.safetensors",
131
+ "model.layers.19.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
132
+ "model.layers.19.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
133
+ "model.layers.19.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
134
+ "model.layers.19.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
135
+ "model.layers.19.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
136
+ "model.layers.19.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
137
+ "model.layers.19.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
138
+ "model.layers.19.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
139
+ "model.layers.19.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
140
+ "model.layers.19.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
141
+ "model.layers.2.input_layernorm.weight": "model-00001-of-00004.safetensors",
142
+ "model.layers.2.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
143
+ "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
144
+ "model.layers.2.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
145
+ "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
146
+ "model.layers.2.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
147
+ "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
148
+ "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
149
+ "model.layers.2.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
150
+ "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
151
+ "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
152
+ "model.layers.20.input_layernorm.weight": "model-00003-of-00004.safetensors",
153
+ "model.layers.20.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
154
+ "model.layers.20.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
155
+ "model.layers.20.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
156
+ "model.layers.20.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
157
+ "model.layers.20.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
158
+ "model.layers.20.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
159
+ "model.layers.20.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
160
+ "model.layers.20.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
161
+ "model.layers.20.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
162
+ "model.layers.20.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
163
+ "model.layers.21.input_layernorm.weight": "model-00003-of-00004.safetensors",
164
+ "model.layers.21.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
165
+ "model.layers.21.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
166
+ "model.layers.21.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
167
+ "model.layers.21.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
168
+ "model.layers.21.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
169
+ "model.layers.21.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
170
+ "model.layers.21.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
171
+ "model.layers.21.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
172
+ "model.layers.21.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
173
+ "model.layers.21.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
174
+ "model.layers.22.input_layernorm.weight": "model-00003-of-00004.safetensors",
175
+ "model.layers.22.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
176
+ "model.layers.22.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
177
+ "model.layers.22.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
178
+ "model.layers.22.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
179
+ "model.layers.22.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
180
+ "model.layers.22.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
181
+ "model.layers.22.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
182
+ "model.layers.22.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
183
+ "model.layers.22.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
184
+ "model.layers.22.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
185
+ "model.layers.23.input_layernorm.weight": "model-00003-of-00004.safetensors",
186
+ "model.layers.23.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
187
+ "model.layers.23.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
188
+ "model.layers.23.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
189
+ "model.layers.23.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
190
+ "model.layers.23.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
191
+ "model.layers.23.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
192
+ "model.layers.23.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
193
+ "model.layers.23.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
194
+ "model.layers.23.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
195
+ "model.layers.23.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
196
+ "model.layers.24.input_layernorm.weight": "model-00003-of-00004.safetensors",
197
+ "model.layers.24.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
198
+ "model.layers.24.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
199
+ "model.layers.24.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
200
+ "model.layers.24.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
201
+ "model.layers.24.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
202
+ "model.layers.24.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
203
+ "model.layers.24.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
204
+ "model.layers.24.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
205
+ "model.layers.24.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
206
+ "model.layers.24.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
207
+ "model.layers.25.input_layernorm.weight": "model-00003-of-00004.safetensors",
208
+ "model.layers.25.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
209
+ "model.layers.25.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
210
+ "model.layers.25.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
211
+ "model.layers.25.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
212
+ "model.layers.25.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
213
+ "model.layers.25.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
214
+ "model.layers.25.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
215
+ "model.layers.25.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
216
+ "model.layers.25.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
217
+ "model.layers.25.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
218
+ "model.layers.26.input_layernorm.weight": "model-00003-of-00004.safetensors",
219
+ "model.layers.26.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
220
+ "model.layers.26.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
221
+ "model.layers.26.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
222
+ "model.layers.26.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
223
+ "model.layers.26.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
224
+ "model.layers.26.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
225
+ "model.layers.26.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
226
+ "model.layers.26.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
227
+ "model.layers.26.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
228
+ "model.layers.26.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
229
+ "model.layers.27.input_layernorm.weight": "model-00003-of-00004.safetensors",
230
+ "model.layers.27.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
231
+ "model.layers.27.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
232
+ "model.layers.27.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
233
+ "model.layers.27.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
234
+ "model.layers.27.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
235
+ "model.layers.27.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
236
+ "model.layers.27.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
237
+ "model.layers.27.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
238
+ "model.layers.27.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
239
+ "model.layers.27.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
240
+ "model.layers.28.input_layernorm.weight": "model-00003-of-00004.safetensors",
241
+ "model.layers.28.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
242
+ "model.layers.28.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
243
+ "model.layers.28.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
244
+ "model.layers.28.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
245
+ "model.layers.28.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
246
+ "model.layers.28.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
247
+ "model.layers.28.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
248
+ "model.layers.28.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
249
+ "model.layers.28.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
250
+ "model.layers.28.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
251
+ "model.layers.29.input_layernorm.weight": "model-00003-of-00004.safetensors",
252
+ "model.layers.29.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
253
+ "model.layers.29.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
254
+ "model.layers.29.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
255
+ "model.layers.29.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
256
+ "model.layers.29.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
257
+ "model.layers.29.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
258
+ "model.layers.29.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
259
+ "model.layers.29.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
260
+ "model.layers.29.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
261
+ "model.layers.29.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
262
+ "model.layers.3.input_layernorm.weight": "model-00001-of-00004.safetensors",
263
+ "model.layers.3.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
264
+ "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
265
+ "model.layers.3.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
266
+ "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
267
+ "model.layers.3.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
268
+ "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
269
+ "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
270
+ "model.layers.3.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
271
+ "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
272
+ "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
273
+ "model.layers.30.input_layernorm.weight": "model-00003-of-00004.safetensors",
274
+ "model.layers.30.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
275
+ "model.layers.30.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
276
+ "model.layers.30.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
277
+ "model.layers.30.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
278
+ "model.layers.30.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
279
+ "model.layers.30.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
280
+ "model.layers.30.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
281
+ "model.layers.30.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
282
+ "model.layers.30.self_attn.q_proj.weight": "model-00003-of-00004.safetensors",
283
+ "model.layers.30.self_attn.v_proj.weight": "model-00003-of-00004.safetensors",
284
+ "model.layers.31.input_layernorm.weight": "model-00003-of-00004.safetensors",
285
+ "model.layers.31.mlp.down_proj.weight": "model-00003-of-00004.safetensors",
286
+ "model.layers.31.mlp.gate_proj.weight": "model-00003-of-00004.safetensors",
287
+ "model.layers.31.mlp.up_proj.weight": "model-00003-of-00004.safetensors",
288
+ "model.layers.31.post_attention_layernorm.weight": "model-00003-of-00004.safetensors",
289
+ "model.layers.31.self_attn.k_norm.weight": "model-00003-of-00004.safetensors",
290
+ "model.layers.31.self_attn.k_proj.weight": "model-00003-of-00004.safetensors",
291
+ "model.layers.31.self_attn.o_proj.weight": "model-00003-of-00004.safetensors",
292
+ "model.layers.31.self_attn.q_norm.weight": "model-00003-of-00004.safetensors",
293
+ "model.layers.31.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
294
+ "model.layers.31.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
295
+ "model.layers.32.input_layernorm.weight": "model-00004-of-00004.safetensors",
296
+ "model.layers.32.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
297
+ "model.layers.32.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
298
+ "model.layers.32.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
299
+ "model.layers.32.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
300
+ "model.layers.32.self_attn.k_norm.weight": "model-00004-of-00004.safetensors",
301
+ "model.layers.32.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
302
+ "model.layers.32.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
303
+ "model.layers.32.self_attn.q_norm.weight": "model-00004-of-00004.safetensors",
304
+ "model.layers.32.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
305
+ "model.layers.32.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
306
+ "model.layers.33.input_layernorm.weight": "model-00004-of-00004.safetensors",
307
+ "model.layers.33.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
308
+ "model.layers.33.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
309
+ "model.layers.33.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
310
+ "model.layers.33.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
311
+ "model.layers.33.self_attn.k_norm.weight": "model-00004-of-00004.safetensors",
312
+ "model.layers.33.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
313
+ "model.layers.33.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
314
+ "model.layers.33.self_attn.q_norm.weight": "model-00004-of-00004.safetensors",
315
+ "model.layers.33.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
316
+ "model.layers.33.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
317
+ "model.layers.34.input_layernorm.weight": "model-00004-of-00004.safetensors",
318
+ "model.layers.34.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
319
+ "model.layers.34.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
320
+ "model.layers.34.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
321
+ "model.layers.34.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
322
+ "model.layers.34.self_attn.k_norm.weight": "model-00004-of-00004.safetensors",
323
+ "model.layers.34.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
324
+ "model.layers.34.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
325
+ "model.layers.34.self_attn.q_norm.weight": "model-00004-of-00004.safetensors",
326
+ "model.layers.34.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
327
+ "model.layers.34.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
328
+ "model.layers.35.input_layernorm.weight": "model-00004-of-00004.safetensors",
329
+ "model.layers.35.mlp.down_proj.weight": "model-00004-of-00004.safetensors",
330
+ "model.layers.35.mlp.gate_proj.weight": "model-00004-of-00004.safetensors",
331
+ "model.layers.35.mlp.up_proj.weight": "model-00004-of-00004.safetensors",
332
+ "model.layers.35.post_attention_layernorm.weight": "model-00004-of-00004.safetensors",
333
+ "model.layers.35.self_attn.k_norm.weight": "model-00004-of-00004.safetensors",
334
+ "model.layers.35.self_attn.k_proj.weight": "model-00004-of-00004.safetensors",
335
+ "model.layers.35.self_attn.o_proj.weight": "model-00004-of-00004.safetensors",
336
+ "model.layers.35.self_attn.q_norm.weight": "model-00004-of-00004.safetensors",
337
+ "model.layers.35.self_attn.q_proj.weight": "model-00004-of-00004.safetensors",
338
+ "model.layers.35.self_attn.v_proj.weight": "model-00004-of-00004.safetensors",
339
+ "model.layers.4.input_layernorm.weight": "model-00001-of-00004.safetensors",
340
+ "model.layers.4.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
341
+ "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
342
+ "model.layers.4.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
343
+ "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
344
+ "model.layers.4.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
345
+ "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
346
+ "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
347
+ "model.layers.4.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
348
+ "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
349
+ "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
350
+ "model.layers.5.input_layernorm.weight": "model-00001-of-00004.safetensors",
351
+ "model.layers.5.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
352
+ "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00004.safetensors",
353
+ "model.layers.5.mlp.up_proj.weight": "model-00001-of-00004.safetensors",
354
+ "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00004.safetensors",
355
+ "model.layers.5.self_attn.k_norm.weight": "model-00001-of-00004.safetensors",
356
+ "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00004.safetensors",
357
+ "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00004.safetensors",
358
+ "model.layers.5.self_attn.q_norm.weight": "model-00001-of-00004.safetensors",
359
+ "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00004.safetensors",
360
+ "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00004.safetensors",
361
+ "model.layers.6.input_layernorm.weight": "model-00001-of-00004.safetensors",
362
+ "model.layers.6.mlp.down_proj.weight": "model-00001-of-00004.safetensors",
363
+ "model.layers.6.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
364
+ "model.layers.6.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
365
+ "model.layers.6.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
366
+ "model.layers.6.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
367
+ "model.layers.6.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
368
+ "model.layers.6.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
369
+ "model.layers.6.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
370
+ "model.layers.6.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
371
+ "model.layers.6.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
372
+ "model.layers.7.input_layernorm.weight": "model-00002-of-00004.safetensors",
373
+ "model.layers.7.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
374
+ "model.layers.7.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
375
+ "model.layers.7.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
376
+ "model.layers.7.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
377
+ "model.layers.7.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
378
+ "model.layers.7.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
379
+ "model.layers.7.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
380
+ "model.layers.7.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
381
+ "model.layers.7.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
382
+ "model.layers.7.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
383
+ "model.layers.8.input_layernorm.weight": "model-00002-of-00004.safetensors",
384
+ "model.layers.8.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
385
+ "model.layers.8.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
386
+ "model.layers.8.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
387
+ "model.layers.8.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
388
+ "model.layers.8.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
389
+ "model.layers.8.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
390
+ "model.layers.8.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
391
+ "model.layers.8.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
392
+ "model.layers.8.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
393
+ "model.layers.8.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
394
+ "model.layers.9.input_layernorm.weight": "model-00002-of-00004.safetensors",
395
+ "model.layers.9.mlp.down_proj.weight": "model-00002-of-00004.safetensors",
396
+ "model.layers.9.mlp.gate_proj.weight": "model-00002-of-00004.safetensors",
397
+ "model.layers.9.mlp.up_proj.weight": "model-00002-of-00004.safetensors",
398
+ "model.layers.9.post_attention_layernorm.weight": "model-00002-of-00004.safetensors",
399
+ "model.layers.9.self_attn.k_norm.weight": "model-00002-of-00004.safetensors",
400
+ "model.layers.9.self_attn.k_proj.weight": "model-00002-of-00004.safetensors",
401
+ "model.layers.9.self_attn.o_proj.weight": "model-00002-of-00004.safetensors",
402
+ "model.layers.9.self_attn.q_norm.weight": "model-00002-of-00004.safetensors",
403
+ "model.layers.9.self_attn.q_proj.weight": "model-00002-of-00004.safetensors",
404
+ "model.layers.9.self_attn.v_proj.weight": "model-00002-of-00004.safetensors",
405
+ "model.norm.weight": "model-00004-of-00004.safetensors"
406
+ }
407
+ }
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:be75606093db2094d7cd20f3c2f385c212750648bd6ea4fb2bf507a6a4c55506
3
+ size 11422650
tokenizer_config.json ADDED
@@ -0,0 +1,30 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "add_prefix_space": false,
3
+ "backend": "tokenizers",
4
+ "bos_token": null,
5
+ "clean_up_tokenization_spaces": false,
6
+ "eos_token": "<|im_end|>",
7
+ "errors": "replace",
8
+ "extra_special_tokens": [
9
+ "<|im_start|>",
10
+ "<|im_end|>",
11
+ "<|object_ref_start|>",
12
+ "<|object_ref_end|>",
13
+ "<|box_start|>",
14
+ "<|box_end|>",
15
+ "<|quad_start|>",
16
+ "<|quad_end|>",
17
+ "<|vision_start|>",
18
+ "<|vision_end|>",
19
+ "<|vision_pad|>",
20
+ "<|image_pad|>",
21
+ "<|video_pad|>"
22
+ ],
23
+ "is_local": true,
24
+ "local_files_only": true,
25
+ "model_max_length": 131072,
26
+ "pad_token": "<|endoftext|>",
27
+ "split_special_tokens": false,
28
+ "tokenizer_class": "Qwen2Tokenizer",
29
+ "unk_token": null
30
+ }