Add files using upload-large-folder tool

README.md

@@ -0,0 +1,38 @@
+---
+license: apache-2.0
+base_model: allenai/Olmo-3.1-32B-Think
+tags:
+  - math
+  - theorem-proving
+  - olmo3
+language:
+  - en
+---
+
+# Proof Pilot — intermediate 32B checkpoints
+
+Intermediate model checkpoints for the AIMO Proof Pilot submission, one subdirectory per step
+of the Stage 2 "Final Pipeline" report. Code: [proof-pilot-codes](https://github.com/ycchen-tw/proof-pilot-codes).
+Final deployment artifacts (OPD step-200 target, GPTQ-w4a16, DFlash drafts) live in
+[proof-pilot-deploy-bundle](https://huggingface.co/ycchen/proof-pilot-deploy-bundle).
+
+Lineage: OLMo 3.1 32B Think → DeepSeek-V4 tokenizer transplant → attention-sink calibration →
+SFT → offline soft distillation (forward KL on DeepSeek-V4-Flash hidden states) → proof-agent
+continuation distillation → agentic on-policy distillation (OPD).
+
+| Subdirectory | Report step | Contents |
+|---|---|---|
+| `step01-olmo32b-deepseektok/` | 1 | OLMo 3.1 32B Think body with DeepSeek-V4 vocabulary (centered-OMP transplant), bf16 |
+| `step02-olmo32b-deepseektok-sink/` | 2 | Step 1 + calibrated explicit attention sinks; custom modeling code and `sink_provenance.json` |
+| `step04-sft-stage1-32b/` | 4 | Supervised 32B checkpoint (step 5,317, ~22.3B packed tokens, trailing loss ≈0.44) |
+| `step05-yarn256k-config/` | 5 | Config-only YaRN-256k variant of step 4 (same weights; YaRN factor 8→32, attention factor 1.347, max position 262,144) |
+| `step07-softdistill-32b-config/` | 7 | Training config of the initial soft-distilled checkpoint; identical weights are public as `soft-distill-32b-deploy` in the deploy bundle |
+| `step09-softdistill-v2test-32b/` | 9 | Final proof-agent soft-distilled 32B (2 epochs, final loss 0.114) — student init for OPD and target for DFlash |
+| `step10-opd-32b-s150/` | 10 | OPD step-150 checkpoint; the released step-200 target is `opd-32b-deploy` in the deploy bundle |
+
+All full checkpoints are consolidated bf16 safetensors with config, tokenizer, and chat template.
+The 65 GB checkpoints load with `trust_remote_code` where custom `olmo3_sink` modeling code is present.
+
+Optimizer states and distributed (resumable) training checkpoints are not published.
+Teacher hidden-state corpora are not published; their manifests and extraction metadata are in
+[proof-pilot-datasets](https://huggingface.co/datasets/ycchen/proof-pilot-datasets).

step01-olmo32b-deepseektok/.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

step01-olmo32b-deepseektok/README.md

@@ -0,0 +1,131 @@
+---
+license: apache-2.0
+base_model:
+- allenai/Olmo-3.1-32B-Think
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- olmo3
+- tokenizer-transplant
+- deepseek-tokenizer
+- omp
+- reasoning
+language:
+- en
+- zh
+---
+
+# Olmo-3.1-32B-Think · DeepSeek-V4-Flash tokenizer (centered-OMP transplant)
+
+This model is **[`allenai/Olmo-3.1-32B-Think`](https://huggingface.co/allenai/Olmo-3.1-32B-Think)**
+with its tokenizer/vocabulary swapped to the **[`deepseek-ai/DeepSeek-V4-Flash`](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash)**
+tokenizer (vocab **100,278 → 129,280**). Every transformer-body weight is the
+original Olmo weight; only `embed_tokens` and `lm_head` were rebuilt to match the
+new vocabulary, via **mean-centered Orthogonal Matching Pursuit (OMP)** embedding
+transplant plus role-mapping of the chat/structural control tokens.
+
+- **Base (weights kept):** Olmo-3.1-32B-Think (Olmo3, 64 layers, hidden 5120, untied embeddings)
+- **Donor (tokenizer + reference embeddings):** DeepSeek-V4-Flash byte-level BPE, vocab 129,280
+- **Motivation:** the DeepSeek tokenizer compresses CJK / multilingual text far better
+  than Olmo's cl100k-family tokenizer (~2.6× fewer tokens on Chinese), while keeping
+  Olmo-3-Think's reasoning behaviour.
+
+> ⚠️ **This is a warm-start initialization, not a finished model.** Zero-shot it
+> produces fluent English / code and can reason, but the ~70k new (mostly CJK) tokens
+> are only *initialized* — their output distribution is not yet calibrated. Strong
+> multilingual generation requires a short continued-pretraining (CPT) pass. See
+> **Limitations**.
+
+## How it was built
+
+For `embed_tokens` and `lm_head` independently:
+
+1. **Anchors** = tokens whose byte-level surface string exists in both vocabs
+   (**59,084**; both tokenizers use the same GPT-2 `Ġ` byte-mapping, verified by decode).
+2. **Shared tokens** → copy the Olmo row **exactly** to the new vocab id.
+3. **New tokens** (**70,196**, 54.3% of the vocab) → OMP. Express each new token's
+   *donor* embedding as a sparse (k=64) combination of the anchors' donor embeddings,
+   then apply the **same coefficients** to those anchors' *base* embeddings. (OMP solves
+   coefficients in donor space, hidden 4096, and reconstructs in base space, hidden 5120
+   — the two hidden sizes need not match.)
+4. **Mean-centering** before OMP (pins the implicit mean-row coefficient to 1; keeps
+   new-token output logits calibrated). Fidelity-neutral.
+5. **Special-token role mapping** — 7 control tokens are copied from Olmo by functional
+   role instead of OMP-filled, and a DeepSeek-V4 native chat template is written:
+
+   | DeepSeek token | role | ← Olmo text |
+   |---|---|---|
+   | `<｜begin▁of▁sentence｜>` (0) | bos | `<\|endoftext\|>` |
+   | `<｜end▁of▁sentence｜>` (1) | eos | `<\|endoftext\|>` |
+   | `<｜▁pad▁｜>` (2) | pad | `<\|pad\|>` |
+   | `<｜User｜>` | user-turn prefix | `<\|im_start\|>user\n` |
+   | `<｜Assistant｜>` | asst-turn prefix | `<\|im_start\|>assistant\n` |
+   | `<think>` / `</think>` | reasoning delimiters | `<think>` / `</think>` |
+
+Reproduction script: [`transplant.py`](./transplant.py) (self-contained; needs the base +
+donor source weights). Method: Goddard & Fernandes Neto 2025, [arXiv:2506.06607](https://arxiv.org/abs/2506.06607).
+Build time ≈ 5 min on one H100.
+
+## Verification (this build)
+
+- Shared-token rows are **bitwise-equal** to the original Olmo rows (embed + head).
+- Anchor split exact and disjoint: 59,084 shared + 70,196 new = 129,280.
+- The 7 role-mapped control tokens match their intended composed vectors (cos = 1.0).
+- **Held-out reconstruction fidelity** (hold out 500 shared anchors, reconstruct via the
+  production OMP path, cosine to the true base row; random floor ≈ 0.02):
+  embed **0.58** (median 0.60), lm_head **0.69** (median 0.72) — on par with the 7B variant.
+
+## Usage
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+m = "<this-repo>"
+tok = AutoTokenizer.from_pretrained(m)
+model = AutoModelForCausalLM.from_pretrained(m, dtype=torch.bfloat16, device_map="auto")
+
+msgs = [{"role": "user", "content": "Prove there are infinitely many primes."}]
+ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
+out = model.generate(ids, max_new_tokens=2048, repetition_penalty=1.3)  # see note below
+print(tok.decode(out[0, ids.shape[1]:]))
+```
+
+The chat template defaults to **thinking mode** (the generation prompt ends with
+`<｜Assistant｜><think>`). The model emits a `<think>…</think>` block, then the answer,
+then `<｜end▁of▁sentence｜>`.
+
+> **Decoding note (important).** `bos` (id 0) and `eos` (id 1) share the same `lm_head`
+> row (both were role-mapped from Olmo's single `<|endoftext|>` token), so their stop
+> logits are identical. Use **`repetition_penalty≈1.3`** (and/or sampling) so the stop
+> token is resolved reliably and to avoid greedy loops over the uncalibrated new tokens
+> — without it, greedy decoding may fail to terminate. This is expected for a pre-CPT
+> warm start.
+
+## Limitations
+
+1. **New-token output is uncalibrated** — see the held-out fidelity above and ppl ≈ 3400
+   on new tokens for the 7B sibling. The model reads new-token text fine and its argmax is
+   usually sensible (fluent free generation), but it cannot place sharp probability on a
+   specific gold token yet.
+2. **Multilingual bits-per-byte is worse than the original Olmo** pre-CPT (output side).
+3. **Greedy drift / loops** in new-token-dense regions; use `repetition_penalty` or sampling.
+4. **Numbers** are grouped left-to-right in 3-digit chunks (`\p{N}{1,3}`, `1000`→`100|0`).
+   This is **identical to Olmo** (Olmo / DeepSeek share the digit regex exactly), so the
+   transplant does not change number handling; place-value misalignment is a shared
+   weakness mitigated only by training data / CoT format.
+5. Traditional-Chinese input tends to produce **Simplified** output (DeepSeek's distribution).
+
+## Recommended next step: continued pretraining (CPT)
+
+A short low-LR CPT pass calibrates the 70k new rows. Suggested: ~2B tokens, LR ~4e-7,
+optionally freeze the body and train only `embed_tokens`+`lm_head` first, corpus skewed
+toward CJK / multilingual + math. **The CPT recipe must preserve reasoning** (include
+math/reasoning data or distillation, not just web-text next-token CE) — naive web-text
+calibration has been observed to regress downstream reasoning.
+
+## License & attribution
+
+Apache-2.0, inheriting from the Olmo-3 base model. The tokenizer files
+(`tokenizer.json`, chat template) originate from DeepSeek-V4-Flash. Cite the OMP
+transplant method (arXiv:2506.06607), Olmo 3, and DeepSeek-V4-Flash.

step01-olmo32b-deepseektok/chat_template.jinja

@@ -0,0 +1,23 @@
+{{- bos_token -}}
+{%- if messages and messages[0]['role'] == 'system' -%}
+{{- messages[0]['content'] -}}
+{%- set loop_messages = messages[1:] -%}
+{%- else -%}
+{%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+{%- if message['role'] == 'user' -%}
+{{- '<｜User｜>' + message['content'] -}}
+{%- elif message['role'] == 'assistant' -%}
+{{- '<｜Assistant｜>' -}}
+{%- if message.get('reasoning_content') -%}
+{{- '<think>' + message['reasoning_content'] + '</think>' -}}
+{%- else -%}
+{{- '</think>' -}}
+{%- endif -%}
+{{- message['content'] + eos_token -}}
+{%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+{{- '<｜Assistant｜><think>' -}}
+{%- endif -%}

step01-olmo32b-deepseektok/config.json

@@ -0,0 +1,101 @@
+{
+  "architectures": [
+    "Olmo3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "dtype": "bfloat16",
+  "eos_token_id": 1,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 27648,
+  "layer_types": [
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 65536,
+  "model_type": "olmo3",
+  "num_attention_heads": 40,
+  "num_hidden_layers": 64,
+  "num_key_value_heads": 8,
+  "pad_token_id": 2,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "attention_factor": 1.2079441541679836,
+    "beta_fast": 32.0,
+    "beta_slow": 1.0,
+    "factor": 8.0,
+    "original_max_position_embeddings": 8192,
+    "rope_theta": 500000,
+    "rope_type": "yarn"
+  },
+  "sliding_window": 4096,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.9.0",
+  "use_cache": false,
+  "vocab_size": 129280
+}

step01-olmo32b-deepseektok/generation_config.json

@@ -0,0 +1,11 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "do_sample": true,
+  "eos_token_id": 1,
+  "max_new_tokens": 32768,
+  "pad_token_id": 2,
+  "temperature": 0.6,
+  "top_p": 0.95,
+  "transformers_version": "5.9.0"
+}

step01-olmo32b-deepseektok/model-00001-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:07a9dd5d07660e5143b3c00f65866296c311ce480bc90477e64ee51226d818a8
+size 49740838392

step01-olmo32b-deepseektok/model-00002-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c7011c46cc33e30cbdbafffe35cc11a816055f0fdc40f2e2dd700adfd109062a
+size 15320250616

step01-olmo32b-deepseektok/model.safetensors.index.json

@@ -0,0 +1,715 @@
+{
+  "metadata": {
+    "total_parameters": 32530502656,
+    "total_size": 65061005312
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00001-of-00002.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.48.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.48.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.50.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.60.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.norm.weight": "model-00002-of-00002.safetensors"
+  }
+}

step01-olmo32b-deepseektok/tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<｜begin▁of▁sentence｜>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<｜end▁of▁sentence｜>",
+  "is_local": true,
+  "legacy": true,
+  "local_files_only": false,
+  "model_max_length": 1048576,
+  "pad_token": "<｜▁pad▁｜>",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": null
+}

step01-olmo32b-deepseektok/transplant.py

@@ -0,0 +1,266 @@
+#!/usr/bin/env python
+"""Rebuild this model: Olmo-3.1-32B-Think + DeepSeek-V4-Flash tokenizer (centered-OMP transplant).
+
+Self-contained reproduction of how the `embed_tokens` / `lm_head` matrices in this
+repository were produced. The transformer body keeps the original Olmo weights
+untouched; only the two vocab-sized matrices are rebuilt for the DeepSeek vocab
+(100,278 -> 129,280):
+
+  - shared tokens (identical byte-level surface string in both vocabs, 59,084) are
+    copied from the base row exactly;
+  - donor-only "new" tokens (70,196) are filled by mean-centered Orthogonal Matching
+    Pursuit (OMP) over the shared anchors (Goddard & Fernandes Neto 2025,
+    arXiv:2506.06607): coefficients are solved in DONOR embedding space (hidden 4096),
+    then the same coefficients are applied to the BASE anchor rows (hidden 5120) —
+    the two hidden sizes need not match;
+  - 7 chat/structural control tokens are role-mapped from Olmo (see SPECIAL_MAP)
+    instead of being OMP-filled, and a DeepSeek-V4 native chat template is written.
+
+Source weights are NOT included in this repo. Point BASE / DONOR_* at local
+snapshots (or HF ids transformers can resolve) and run:  python transplant.py
+Only the two donor shards holding `embed.weight` / `head.weight` are read (~2 GB),
+not the full DeepSeek checkpoint.
+
+Requires: torch, transformers, safetensors.
+"""
+from __future__ import annotations
+
+import json
+import os
+import time
+
+import torch
+from safetensors import safe_open
+
+# --------------------------------------------------------------------------- config
+BASE            = "allenai/Olmo-3.1-32B-Think"    # base model (weights kept); a local path also works
+DONOR_WEIGHTS   = "DeepSeek-V4-Flash"             # local snapshot dir of deepseek-ai/DeepSeek-V4-Flash
+DONOR_TOKENIZER = "deepseek-ai/DeepSeek-V4-Flash"
+OUT             = "./Olmo-3.1-32B-Think-deepseekTok"
+
+K             = 64        # OMP sparsity
+CENTERED      = True      # mean-centered OMP (essential for lm_head logit calibration)
+COSINE_SELECT = True      # cosine atom selection (False -> raw canonical OMP)
+RIDGE         = 1e-3
+PAD_TOKEN     = "<｜▁pad▁｜>"
+
+TENSORS = dict(base_embed="model.embed_tokens.weight", base_head="lm_head.weight",
+               donor_embed="embed.weight", donor_head="head.weight")
+
+# donor special-token string -> base-tokenizer text. The donor row is overwritten with
+# the norm-rescaled mean of the base rows that text tokenizes to (single token = exact
+# copy). Maps each control token by its functional role in the chat template.
+SPECIAL_MAP = {
+    "<｜begin▁of▁sentence｜>": "<|endoftext|>",
+    "<｜end▁of▁sentence｜>": "<|endoftext|>",
+    "<｜▁pad▁｜>": "<|pad|>",
+    "<｜User｜>": "<|im_start|>user\n",
+    "<｜Assistant｜>": "<|im_start|>assistant\n",
+    "<think>": "<think>",
+    "</think>": "</think>",
+}
+
+# DeepSeek-V4 native chat format. Thinking mode: prompt ends with <｜Assistant｜><think>.
+CHAT_TEMPLATE = """{{- bos_token -}}
+{%- if messages and messages[0]['role'] == 'system' -%}
+{{- messages[0]['content'] -}}
+{%- set loop_messages = messages[1:] -%}
+{%- else -%}
+{%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+{%- if message['role'] == 'user' -%}
+{{- '<｜User｜>' + message['content'] -}}
+{%- elif message['role'] == 'assistant' -%}
+{{- '<｜Assistant｜>' -}}
+{%- if message.get('reasoning_content') -%}
+{{- '<think>' + message['reasoning_content'] + '</think>' -}}
+{%- else -%}
+{{- '</think>' -}}
+{%- endif -%}
+{{- message['content'] + eos_token -}}
+{%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+{{- '<｜Assistant｜><think>' -}}
+{%- endif -%}
+"""
+
+
+# --------------------------------------------------------------------------- OMP core
+def resolve_device(device: str | None = None) -> str:
+    if device:
+        return device
+    return "cuda" if torch.cuda.is_available() else "cpu"
+
+
+@torch.no_grad()
+def batch_omp(targets, atoms, k, device="cuda", chunk=2048, ridge=1e-3, raw_select=False):
+    """Batched OMP in donor space. targets [N,d], atoms [A,d] -> idx [N,k], coef [N,k].
+
+    Selection uses unit-normalized atoms (cosine) unless raw_select; coefficients are
+    solved by normal equations with an adaptive ridge for collinear atoms.
+    """
+    N = targets.shape[0]
+    atoms = atoms.to(device, torch.float32)
+    sel_atoms = atoms if raw_select else atoms / atoms.norm(dim=1, keepdim=True).clamp_min(1e-12)
+    anorm = sel_atoms
+    out_idx = torch.zeros(N, k, dtype=torch.long)
+    out_coef = torch.zeros(N, k, dtype=torch.float32)
+    for s in range(0, N, chunk):
+        t = targets[s:s + min(chunk, N - s)].to(device, torch.float32)
+        B = t.shape[0]
+        resid = t.clone()
+        sel = torch.zeros(B, k, dtype=torch.long, device=device)
+        coef = torch.zeros(B, k, device=device)
+        for i in range(k):
+            corr = resid @ anorm.T
+            score = corr.abs()
+            if i > 0:
+                score.scatter_(1, sel[:, :i], -1.0)
+            best = score.argmax(dim=1)
+            sel[:, i] = best
+            G = atoms[sel[:, :i + 1]]
+            At = G.transpose(1, 2)
+            GtG = torch.bmm(G, At)
+            diag = GtG.diagonal(dim1=1, dim2=2).mean(dim=1).clamp_min(1e-6)
+            GtG = GtG + (ridge * diag).view(-1, 1, 1) * torch.eye(i + 1, device=device)
+            rhs = torch.bmm(G, t.unsqueeze(2))
+            x = torch.linalg.solve(GtG, rhs)
+            coef[:, :i + 1] = x.squeeze(2)
+            resid = t - torch.bmm(coef[:, :i + 1].unsqueeze(1), G).squeeze(1)
+        out_idx[s:s + B] = sel.cpu()
+        out_coef[s:s + B] = coef.cpu()
+        if (s // chunk) % 4 == 0:
+            rms = resid.pow(2).mean().sqrt()
+            print(f"    omp {s + B}/{N}  (last resid rms {rms:.4f})", flush=True)
+    return out_idx, out_coef
+
+
+@torch.no_grad()
+def reconstruct(out_idx, out_coef, base_anchor_emb, device="cuda"):
+    """out[t] = sum coef * base_anchor[idx]. base_anchor_emb [A,d]."""
+    base_anchor_emb = base_anchor_emb.to(device, torch.float32)
+    N, k = out_idx.shape
+    res = torch.zeros(N, base_anchor_emb.shape[1], device=device)
+    for s in range(0, N, 4096):
+        e = min(s + 4096, N)
+        idx = out_idx[s:e].to(device)
+        cf = out_coef[s:e].to(device)
+        picked = base_anchor_emb[idx]
+        res[s:e] = torch.bmm(cf.unsqueeze(1), picked).squeeze(1)
+    return res.cpu()
+
+
+# --------------------------------------------------------------------------- pipeline
+def load_tensor(model_dir: str, name: str) -> torch.Tensor:
+    idx = json.load(open(os.path.join(model_dir, "model.safetensors.index.json")))["weight_map"]
+    with safe_open(os.path.join(model_dir, idx[name]), framework="pt") as f:
+        return f.get_tensor(name)
+
+
+def build_anchor_map(base: str, donor_tokenizer: str):
+    from transformers import AutoTokenizer
+    tb = AutoTokenizer.from_pretrained(base, trust_remote_code=True)
+    td = AutoTokenizer.from_pretrained(donor_tokenizer, trust_remote_code=True)
+    vb, vd = tb.get_vocab(), td.get_vocab()
+    shared = set(vb) & set(vd)
+    a_d, a_b = [], []
+    for s in shared:
+        a_d.append(vd[s]); a_b.append(vb[s])
+    new_ids = sorted(set(vd.values()) - {vd[s] for s in shared})
+    return (torch.tensor(a_d), torch.tensor(a_b), torch.tensor(new_ids), len(vd), len(vb))
+
+
+def apply_special_map(out, b_mat, base_tok, donor_tok, tag):
+    """Overwrite donor special-token rows with role-mapped base rows (in place, fp32)."""
+    if not SPECIAL_MAP or base_tok is None or donor_tok is None:
+        return
+    dv = donor_tok.get_vocab()
+    n = 0
+    for donor_str, base_text in SPECIAL_MAP.items():
+        if donor_str not in dv:
+            print(f"  [special] WARN donor token {donor_str!r} not in vocab; skipped")
+            continue
+        bids = base_tok(base_text, add_special_tokens=False)["input_ids"]
+        if not bids:
+            print(f"  [special] WARN {base_text!r} tokenized to nothing; skipped")
+            continue
+        rows = b_mat[bids]
+        v = rows.mean(0)
+        v = v / v.norm().clamp_min(1e-8) * rows.norm(dim=1).mean()
+        out[dv[donor_str]] = v
+        n += 1
+    print(f"  {tag}: applied {n} special-token overrides")
+
+
+def build_matrix(donor_name, base_name, a_d, a_b, new_ids, nd, device, base_tok, donor_tok):
+    print(f"\n## transplant {base_name}  (donor {donor_name})")
+    d_mat = load_tensor(DONOR_WEIGHTS, donor_name).float()
+    b_mat = load_tensor(BASE, base_name).float()
+    d = b_mat.shape[1]
+    out = torch.zeros(nd, d, dtype=torch.float32)
+    out[a_d] = b_mat[a_b]
+    raw_select = not COSINE_SELECT
+    if CENTERED:
+        md = d_mat[a_d].mean(0, keepdim=True)
+        mb = b_mat[a_b].mean(0, keepdim=True)
+        idx, coef = batch_omp(d_mat[new_ids] - md, d_mat[a_d] - md, K,
+                              device=device, ridge=RIDGE, raw_select=raw_select)
+        out[new_ids] = reconstruct(idx, coef, b_mat[a_b] - mb, device=device) + mb
+        mode = "mean-centered"
+    else:
+        idx, coef = batch_omp(d_mat[new_ids], d_mat[a_d], K,
+                              device=device, ridge=RIDGE, raw_select=raw_select)
+        out[new_ids] = reconstruct(idx, coef, b_mat[a_b], device=device)
+        mode = "uncentered"
+    print(f"  {base_name}: copied {len(a_d)} shared rows, OMP {len(new_ids)} new rows ({mode})")
+    apply_special_map(out, b_mat, base_tok, donor_tok, base_name)
+    return out.to(torch.bfloat16)
+
+
+def run(device: str | None = None) -> str:
+    from transformers import AutoModelForCausalLM, AutoTokenizer
+    device = resolve_device(device)
+    t0 = time.time()
+    a_d, a_b, new_ids, nd, nb = build_anchor_map(BASE, DONOR_TOKENIZER)
+    base_tok = AutoTokenizer.from_pretrained(BASE, trust_remote_code=True)
+    donor_tok = AutoTokenizer.from_pretrained(DONOR_TOKENIZER, trust_remote_code=True)
+    print(f"donor vocab {nd}, base vocab {nb}, anchors {len(a_d)}, new tokens {len(new_ids)} "
+          f"| device={device} k={K} centered={CENTERED} cosine={COSINE_SELECT} "
+          f"special_map={len(SPECIAL_MAP)}")
+    emb_new = build_matrix(TENSORS["donor_embed"], TENSORS["base_embed"],
+                           a_d, a_b, new_ids, nd, device, base_tok, donor_tok)
+    head_new = build_matrix(TENSORS["donor_head"], TENSORS["base_head"],
+                            a_d, a_b, new_ids, nd, device, base_tok, donor_tok)
+
+    print("\n## loading base model & assembling")
+    model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16,
+                                                 low_cpu_mem_usage=True, trust_remote_code=True)
+    model.resize_token_embeddings(nd)
+    with torch.no_grad():
+        model.get_input_embeddings().weight.copy_(emb_new)
+        model.get_output_embeddings().weight.copy_(head_new)
+
+    td = donor_tok
+    td.chat_template = CHAT_TEMPLATE
+    if PAD_TOKEN is not None:
+        td.pad_token = PAD_TOKEN
+    model.config.vocab_size = nd
+    model.config.bos_token_id = td.bos_token_id
+    model.config.eos_token_id = td.eos_token_id
+    model.config.pad_token_id = td.pad_token_id if td.pad_token_id is not None else td.eos_token_id
+    if getattr(model, "generation_config", None) is not None:
+        model.generation_config.bos_token_id = td.bos_token_id
+        model.generation_config.eos_token_id = td.eos_token_id
+        model.generation_config.pad_token_id = model.config.pad_token_id
+
+    os.makedirs(OUT, exist_ok=True)
+    model.save_pretrained(OUT, safe_serialization=True)
+    td.save_pretrained(OUT)
+    print(f"\nDONE in {time.time() - t0:.0f}s -> {OUT}")
+    return OUT
+
+
+if __name__ == "__main__":
+    run()

step02-olmo32b-deepseektok-sink/README.md

@@ -0,0 +1,98 @@
+---
+license: apache-2.0
+language:
+- en
+- zh
+base_model: allenai/Olmo-3.1-32B-Think
+library_name: transformers
+tags:
+- olmo3
+- attention-sink
+- tokenizer-transplant
+- deepseek-tokenizer
+- training-init
+---
+
+# Olmo-3.1-32B-Think + DeepSeek-V4 tokenizer + measured attention sinks (SFT training init)
+
+> ⚠️ **This is a warm-start initialization checkpoint for SFT, not a finished model.** On top of
+> the centered-OMP tokenizer transplant (step01), it adds gpt-oss-style learnable attention sinks
+> (`s_aux`) and initializes each sink to the **per-head logit level measured on this very model**.
+> It is usable for direct inference but uncalibrated; the intended use is as the `--model_path`
+> of stage-1 SFT.
+
+Produced by a single `olmo3_sink.build_init_model` command: stock OLMo → OMP transplant →
+attention-distribution measurement → sinks written directly into the weights. Full derivation
+parameters are in the bundled `sink_provenance.json`.
+
+## Composition
+
+1. **Base**: `allenai/Olmo-3.1-32B-Think` (64 layers, hidden 5120, GQA 40Q/8KV, SWA 4096 with
+   3:1 interleave — 48 sliding / 16 full — YaRN on full layers). Transformer body weights are
+   preserved bit-for-bit.
+2. **Tokenizer**: DeepSeek-V4-Flash byte-level BPE (vocab 100,278 → 129,280). `embed_tokens` /
+   `lm_head` rows are reconstructed with mean-centered OMP (k=64,
+   [arXiv:2506.06607](https://arxiv.org/abs/2506.06607)); chat control tokens are mapped by role
+   (`<｜User｜>` ← `<|im_start|>user\n`, etc.) and the native DeepSeek-V4 chat template is
+   installed (thinking mode: the generation prompt ends with `<｜Assistant｜><think>`).
+3. **Attention sink (`model_type: olmo3_sink`)**: one learnable scalar `self_attn.sinks` per head,
+   acting as an extra softmax logit (`s_aux`) that is normalized then discarded — letting a head
+   attend to "nothing". The weight format is compatible with the gpt-oss/vLLM/SGLang loaders
+   (`[num_heads]`, bf16).
+
+## Sink initialization (what makes this checkpoint different)
+
+Motivation from measurement: stock OLMo 3 has no classic position-0 sink, but in mid-to-late
+layers each head dumps 22–44% of its attention mass onto low-semantic tokens conscripted in
+place (`the`, `end`, whitespace; their value norms are squashed to 0.04–0.14× the layer median —
+the textbook no-op-sink signature). Scalar inits (0.0 or −10) were shown to be **dead starting
+points** in two 10k-step runs (the sink gradient is proportional to absorbed mass, which sits
+~6.5 nats below the conscripted-token level).
+
+Sinks are therefore warm-started at measured values:
+`s_init(h) = logZ_h + logit(clip(D_h, 0.02, 0.10))`, where logZ is the logsumexp of the head's
+visible logits and D is the mass dumped onto the conscripted token set, measured at sequence
+length 8,192 with query rows ≥ 4,096 (sliding-window steady state).
+
+| Quantity | Value |
+|---|---|
+| s_init mean (p10/p90) | **6.64** (4.48 / 8.40) |
+| conscripted mass D, mean | 0.229 (61/64 layers have conscripted keys) |
+| step-0 sink absorption (steady state) | target p ∈ [2%, 10%] (all 2,560 heads inside the gradient-active band) |
+| step-0 LM loss (8k math text) | 1.3447 → 1.3481 (**+0.0034** — the 32B is an order of magnitude less sensitive to sink init than the 7B) |
+
+## Usage
+
+```python
+# A. trust_remote_code (self-contained; the modeling code includes the transformers 5.x
+#    OLMo3 per-layer RoPE fix)
+from transformers import AutoModelForCausalLM, AutoTokenizer
+model = AutoModelForCausalLM.from_pretrained(
+    "ycchen/proof-pilot-checkpoints", subfolder="step02-olmo32b-deepseektok-sink",
+    trust_remote_code=True,
+    dtype="bfloat16", attn_implementation="eager",  # or olmo3_sink_fa3 (needs patched FA3)
+)
+```
+
+- Backends: `olmo3_sink_fa3` (patched FA3 with in-kernel sink, the production training path) /
+  `eager` (sink-aware reference implementation) / `flex_attention`; **`sdpa` is unsupported**
+  (raises immediately).
+- Training: sinks are already in the weights; any `sink_init_value` setting only fills missing
+  keys and never overwrites (requires the `_init_weights` guard fix, included in the vendored
+  modeling code of this repo).
+- Note: the vendored modeling code fixes the transformers 5.x OLMo 3 RoPE regression
+  (HF #39847 wrongly applies YaRN to sliding layers; upstream fix #45945 unmerged) — loading
+  with stock `Olmo3ForCausalLM` yields distorted sliding-layer attention.
+
+## Limitations
+
+- The 70,196 new (mostly CJK) token embeddings are OMP-initialized only, not trained.
+- The sink warm start perturbs step-0 LM loss by +0.003 nats (table above).
+- Calibration used math text in an 8k window; logZ is a weight property (±0.1 nats across
+  domains), but extreme out-of-domain behavior is unverified.
+
+## Provenance
+
+`sink_provenance.json`: pipeline, transplant config, calibration-text SHA, derivation
+hyperparameters (eps 0.04 / p ∈ [0.02, 0.10] / qmin 4096 / seq 8192), loss before/after,
+git commit.

step02-olmo32b-deepseektok-sink/attention.py

@@ -0,0 +1,74 @@
+# Copyright 2026 proof-pilot. Apache-2.0.
+"""transformers attention-interface adapter for the in-kernel FA3 sink.
+
+Registers `attn_implementation="olmo3_sink_fa3"`. The adapter converts transformers'
+[B, H, S, D] layout to FA3 varlen [total, H, D] + cu_seqlens and calls the patched
+in-kernel sink kernel. Deliberately NOT registered in the mask interface, so
+`create_causal_mask` returns None (no [S,S] mask is built) and we rely on cu_seqlens.
+"""
+from __future__ import annotations
+
+import torch
+from transformers import AttentionInterface  # public top-level API
+from transformers.modeling_flash_attention_utils import (
+    _is_packed_sequence,
+    prepare_fa_kwargs_from_position_ids,
+)
+
+from .fa3_sink_kernel import fa3_varlen_attn_with_sink_kernel
+
+ATTN_NAME = "olmo3_sink_fa3"
+
+
+def fa3_sink_attention_forward(
+    module,
+    query: torch.Tensor,   # [B, Hq, S, D]
+    key: torch.Tensor,     # [B, Hkv, S, D]
+    value: torch.Tensor,   # [B, Hkv, S, D]
+    attention_mask=None,   # ignored: varlen uses cu_seqlens
+    scaling: float | None = None,
+    dropout: float = 0.0,
+    sliding_window: int | None = None,
+    s_aux: torch.Tensor | None = None,
+    **kwargs,
+):
+    # Note: `dropout` is accepted for signature compatibility but not applied (Olmo3's
+    # attention_dropout defaults to 0.0; FA3 varlen dropout is not threaded here).
+    B, Hq, S, D = query.shape
+    Hkv = key.shape[1]
+    sink = s_aux if s_aux is not None else module.sinks
+
+    # [B, H, S, D] -> varlen [B*S, H, D]
+    q = query.transpose(1, 2).reshape(B * S, Hq, D)
+    k = key.transpose(1, 2).reshape(B * S, Hkv, D)
+    v = value.transpose(1, 2).reshape(B * S, Hkv, D)
+
+    # Preferred: varlen metadata computed once in Olmo3SinkModel.forward (reuse, no
+    # per-layer recompute). max_* are already ints there.
+    cu_q = kwargs.get("cu_seq_lens_q")
+    cu_k = kwargs.get("cu_seq_lens_k")
+    max_q = kwargs.get("max_length_q")
+    max_k = kwargs.get("max_length_k")
+    if cu_q is None:
+        # Fallback (e.g. reuse_packing_metadata disabled, or used outside Olmo3SinkModel):
+        # derive packed boundaries from position_ids so doc isolation stays CORRECT.
+        position_ids = kwargs.get("position_ids")
+        if position_ids is not None and _is_packed_sequence(position_ids, B):
+            (cu_q, cu_k), (mq, mk) = prepare_fa_kwargs_from_position_ids(position_ids)
+            max_q, max_k = int(mq), int(mk)
+        else:
+            # truly unpacked: B rows of full length S
+            cu_q = torch.arange(0, (B + 1) * S, S, device=q.device, dtype=torch.int32)
+            cu_k = cu_q
+            max_q = max_k = S
+
+    window = (sliding_window - 1, 0) if sliding_window is not None else (-1, -1)
+    out = fa3_varlen_attn_with_sink_kernel(
+        q, k, v, sink, cu_q, cu_k, max_q, max_k,
+        softmax_scale=scaling, causal=True, window_size=window,
+    )  # [B*S, Hq, D]
+    return out.reshape(B, S, Hq, D), None
+
+
+def register_fa3_sink_attention() -> None:
+    AttentionInterface.register(ATTN_NAME, fa3_sink_attention_forward)

step02-olmo32b-deepseektok-sink/chat_template.jinja

@@ -0,0 +1,23 @@
+{{- bos_token -}}
+{%- if messages and messages[0]['role'] == 'system' -%}
+{{- messages[0]['content'] -}}
+{%- set loop_messages = messages[1:] -%}
+{%- else -%}
+{%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+{%- if message['role'] == 'user' -%}
+{{- '<｜User｜>' + message['content'] -}}
+{%- elif message['role'] == 'assistant' -%}
+{{- '<｜Assistant｜>' -}}
+{%- if message.get('reasoning_content') -%}
+{{- '<think>' + message['reasoning_content'] + '</think>' -}}
+{%- else -%}
+{{- '</think>' -}}
+{%- endif -%}
+{{- message['content'] + eos_token -}}
+{%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+{{- '<｜Assistant｜><think>' -}}
+{%- endif -%}

step02-olmo32b-deepseektok-sink/config.json

@@ -0,0 +1,107 @@
+{
+  "architectures": [
+    "Olmo3SinkForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_olmo3_sink.Olmo3SinkConfig",
+    "AutoModelForCausalLM": "modeling_olmo3_sink.Olmo3SinkForCausalLM"
+  },
+  "bos_token_id": 0,
+  "dtype": "bfloat16",
+  "eos_token_id": 1,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 27648,
+  "layer_types": [
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 65536,
+  "model_type": "olmo3_sink",
+  "num_attention_heads": 40,
+  "num_hidden_layers": 64,
+  "num_key_value_heads": 8,
+  "pad_token_id": 2,
+  "reuse_packing_metadata": true,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "attention_factor": 1.2079441541679836,
+    "beta_fast": 32.0,
+    "beta_slow": 1.0,
+    "factor": 8.0,
+    "original_max_position_embeddings": 8192,
+    "rope_theta": 500000,
+    "rope_type": "yarn"
+  },
+  "sink_init_value": -10000.0,
+  "sliding_window": 4096,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.9.0",
+  "use_cache": false,
+  "vocab_size": 129280
+}

step02-olmo32b-deepseektok-sink/configuration_olmo3_sink.py

@@ -0,0 +1,42 @@
+# Copyright 2026 proof-pilot. Apache-2.0.
+"""Config for Olmo3 + attention-sink + packing-metadata-reuse.
+
+This is a thin subclass of `Olmo3Config` (transformers 5.x, dataclass/`@strict`
+style). It only adds two fields; all structural defaults come from Olmo3.
+"""
+
+from __future__ import annotations
+
+from huggingface_hub.dataclasses import strict
+
+from transformers.models.olmo3.configuration_olmo3 import Olmo3Config
+
+
+@strict
+class Olmo3SinkConfig(Olmo3Config):
+    r"""Olmo3 with per-head learnable attention sinks (gpt-oss style).
+
+    Args:
+        sink_init_value (`float`, defaults to 0.0):
+            Initial value for every `self_attn.sinks` entry. The sink is an extra
+            logit appended to the attention softmax (mass that attends to
+            "nothing"). With value `0.0` the sink competes on equal footing with a
+            zero-score key at init (gpt-oss default). For a *warm start* from a
+            checkpoint trained without sinks, use a strongly negative value
+            (e.g. `-10.0`) so the sink starts as a near no-op and is learned in.
+        reuse_packing_metadata (`bool`, defaults to True):
+            When packing multiple documents per sequence and running on a
+            FlashAttention backend, compute the varlen `cu_seqlens`/`max_seqlen`
+            once in the model forward and thread it to every layer, instead of
+            letting each layer re-derive it from `position_ids` (which incurs a
+            device->host sync per layer). No effect on non-flash backends or when
+            `position_ids` is not packed.
+    """
+
+    model_type = "olmo3_sink"
+
+    sink_init_value: float = 0.0
+    reuse_packing_metadata: bool = True
+
+
+__all__ = ["Olmo3SinkConfig"]

step02-olmo32b-deepseektok-sink/fa3_sink_kernel.py

@@ -0,0 +1,144 @@
+# Copyright 2026 proof-pilot. Apache-2.0.
+"""In-kernel attention sink on patched FA3 (targets >95% of no-sink FA3 for ALL packing).
+
+The FA3 forward kernel epilogue is patched (see flash-attention/hopper/softmax.h finalize)
+to add exp(sink-M) to the softmax denominator, so it directly outputs sink-corrected `o`
+and sink-inclusive `lse` — NO forward post-pass. Backward feeds those to FA3's native
+backward for exact dq/dk/dv (proven equivalent), plus a cheap dsink reduction.
+
+torch.compile: forward and backward are each a `torch.library.custom_op` with a registered
+fake (meta) impl, so Dynamo treats them as single opaque nodes with known output shapes
+instead of graph-breaking on the fake-less `flash_attn_3.fwd`. The FA3/Triton internals
+live in the eager op bodies and are never traced. This lets a compiled decoder layer fuse
+everything around attention into one graph (critical for compile + gradient checkpointing,
+which re-runs the forward in backward and would otherwise pay the graph-break cost twice).
+"""
+from __future__ import annotations
+
+import torch
+import triton
+import triton.language as tl
+from flash_attn_interface import _flash_attn_backward
+
+
+@triton.jit
+def _dsink_kernel(o_ptr, do_ptr, sink_ptr, lse2_ptr, dsink_ptr, T, D,
+                  so_t, so_h, sl_h, sl_t, BLOCK_T: tl.constexpr, BLOCK_D: tl.constexpr):
+    h = tl.program_id(0)
+    t = tl.program_id(1) * BLOCK_T + tl.arange(0, BLOCK_T)
+    d = tl.arange(0, BLOCK_D)
+    tm = t < T
+    dm = d < D
+    base = t[:, None] * so_t + h * so_h + d[None, :]
+    m = tm[:, None] & dm[None, :]
+    o = tl.load(o_ptr + base, mask=m, other=0.0).to(tl.float32)
+    g = tl.load(do_ptr + base, mask=m, other=0.0).to(tl.float32)
+    delta = tl.sum(o * g, axis=1)                              # [BLOCK_T]
+    lse2 = tl.load(lse2_ptr + h * sl_h + t * sl_t, mask=tm, other=0.0)
+    sink = tl.load(sink_ptr + h)
+    contrib = tl.where(tm, -tl.exp(sink - lse2) * delta, 0.0)
+    tl.atomic_add(dsink_ptr + h, tl.sum(contrib))
+
+
+def _dsink(o_sink, do, sink, lse2):
+    """dsink_h = -Σ_t exp(sink_h - lse2_{h,t})·(o_sink·do)_{t,h}. o_sink,do [T,H,D]; lse2 [H,T]."""
+    T, H, D = o_sink.shape
+    dsink = torch.zeros(H, device=o_sink.device, dtype=torch.float32)
+    BLOCK_T = 64
+    BLOCK_D = triton.next_power_of_2(D)
+    grid = (H, triton.cdiv(T, BLOCK_T))
+    _dsink_kernel[grid](o_sink, do, sink, lse2, dsink, T, D,
+                        o_sink.stride(0), o_sink.stride(1), lse2.stride(0), lse2.stride(1),
+                        BLOCK_T=BLOCK_T, BLOCK_D=BLOCK_D)
+    return dsink
+
+
+# --- forward as an opaque, fake-backed custom op -----------------------------------------
+@torch.library.custom_op("olmo3_sink::fa3_sink_fwd", mutates_args=())
+def _fa3_sink_fwd(
+    q: torch.Tensor, k: torch.Tensor, v: torch.Tensor, sink: torch.Tensor,
+    cu_q: torch.Tensor, cu_k: torch.Tensor, max_q: int, max_k: int,
+    scale: float, causal: bool, wl: int, wr: int,
+) -> tuple[torch.Tensor, torch.Tensor]:
+    # patched op: `sink` is the trailing arg; kernel applies it in the epilogue.
+    out, lse, _, _ = torch.ops.flash_attn_3.fwd(
+        q, k, v,
+        None, None, None, None,            # k_new, v_new, qv, out
+        cu_q, cu_k, None,                  # cu_seqlens q/k/k_new
+        None, None,                        # seqused q/k
+        max_q, max_k,
+        None, None, None,                  # page_table, kv_batch_idx, leftpad_k
+        None, None, None,                  # rotary cos/sin, seqlens_rotary
+        None, None, None,                  # q/k/v_descale
+        scale, causal, wl, wr,
+        0,                                 # attention_chunk
+        0.0,                               # softcap
+        False,                             # rotary_interleaved
+        None,                              # scheduler_metadata
+        1,                                 # num_splits (DISABLE_SPLIT -> must be 1)
+        None,                              # pack_gqa
+        0,                                 # sm_margin
+        sink,                              # proof-pilot sink
+    )
+    return out, lse
+
+
+@_fa3_sink_fwd.register_fake
+def _(q, k, v, sink, cu_q, cu_k, max_q, max_k, scale, causal, wl, wr):
+    # out matches q [total_q, H, D]; varlen lse is [H, total_q] (see _dsink indexing).
+    T, H = q.shape[0], q.shape[1]
+    return torch.empty_like(q), q.new_empty((H, T), dtype=torch.float32)
+
+
+# --- backward as an opaque, fake-backed custom op ----------------------------------------
+@torch.library.custom_op("olmo3_sink::fa3_sink_bwd", mutates_args=())
+def _fa3_sink_bwd(
+    do: torch.Tensor, q: torch.Tensor, k: torch.Tensor, v: torch.Tensor,
+    out: torch.Tensor, lse: torch.Tensor, sink: torch.Tensor,
+    cu_q: torch.Tensor, cu_k: torch.Tensor, max_q: int, max_k: int,
+    scale: float, causal: bool, wl: int, wr: int,
+) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:
+    dq, dk, dv = torch.empty_like(q), torch.empty_like(k), torch.empty_like(v)
+    _flash_attn_backward(do, q, k, v, out, lse, cu_q, cu_k, None, None,
+                         max_q, max_k, dq, dk, dv, scale, causal, wl, wr, 0.0, False, 0)
+    dsink = _dsink(out, do, sink, lse)
+    return dq, dk, dv, dsink
+
+
+@_fa3_sink_bwd.register_fake
+def _(do, q, k, v, out, lse, sink, cu_q, cu_k, max_q, max_k, scale, causal, wl, wr):
+    return (torch.empty_like(q), torch.empty_like(k), torch.empty_like(v),
+            torch.empty_like(sink))
+
+
+def _setup_context(ctx, inputs, output):
+    q, k, v, sink, cu_q, cu_k, max_q, max_k, scale, causal, wl, wr = inputs
+    out, lse = output
+    ctx.save_for_backward(q, k, v, out, lse, sink, cu_q, cu_k)
+    ctx.max_q, ctx.max_k = max_q, max_k
+    ctx.scale, ctx.causal, ctx.wl, ctx.wr = scale, causal, wl, wr
+
+
+def _backward(ctx, grad_out, grad_lse):
+    q, k, v, out, lse, sink, cu_q, cu_k = ctx.saved_tensors
+    dq, dk, dv, dsink = torch.ops.olmo3_sink.fa3_sink_bwd(
+        grad_out.contiguous(), q, k, v, out, lse, sink, cu_q, cu_k,
+        ctx.max_q, ctx.max_k, ctx.scale, ctx.causal, ctx.wl, ctx.wr)
+    # grads for (q, k, v, sink, cu_q, cu_k, max_q, max_k, scale, causal, wl, wr)
+    return dq, dk, dv, dsink, None, None, None, None, None, None, None, None
+
+
+_fa3_sink_fwd.register_autograd(_backward, setup_context=_setup_context)
+
+
+def fa3_varlen_attn_with_sink_kernel(
+    q, k, v, sink, cu_seqlens_q, cu_seqlens_k, max_seqlen_q, max_seqlen_k,
+    softmax_scale=None, causal=True, window_size=(-1, -1),
+):
+    if softmax_scale is None:
+        softmax_scale = q.shape[-1] ** -0.5
+    wl, wr = window_size
+    out, _lse = _fa3_sink_fwd(
+        q, k, v, sink.to(torch.float32), cu_seqlens_q, cu_seqlens_k,
+        int(max_seqlen_q), int(max_seqlen_k), float(softmax_scale), bool(causal), int(wl), int(wr))
+    return out

step02-olmo32b-deepseektok-sink/generation_config.json

@@ -0,0 +1,11 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "do_sample": true,
+  "eos_token_id": 1,
+  "max_new_tokens": 32768,
+  "pad_token_id": 2,
+  "temperature": 0.6,
+  "top_p": 0.95,
+  "transformers_version": "5.9.0"
+}

step02-olmo32b-deepseektok-sink/model-00001-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cc91a05bf1392776d2bab06e083239148847b09bdff6fbc3143bf92d6a457947
+size 49740847248

step02-olmo32b-deepseektok-sink/model-00002-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1ae65b41e05f4f46127d25d34dad7525ba37161acb41d505adb3fe1a153e794e
+size 15320253552

step02-olmo32b-deepseektok-sink/model.safetensors.index.json

@@ -0,0 +1,779 @@
+{
+  "metadata": {
+    "total_parameters": 32530505216,
+    "total_size": 65061010432
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00001-of-00002.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.48.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.48.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.50.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.60.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.norm.weight": "model-00002-of-00002.safetensors"
+  }
+}

step02-olmo32b-deepseektok-sink/modeling_olmo3_sink.py

@@ -0,0 +1,352 @@
+# Copyright 2026 proof-pilot. Apache-2.0.
+"""Olmo3 with (1) gpt-oss-style learnable attention sinks, (2) FlashAttention
+packing-metadata reuse, and (3) restored per-layer-type RoPE.
+
+Design: subclass the stock `transformers.models.olmo3` classes and override only
+what changes. Everything else (RMSNorm, sliding-window layer_types, MLP,
+generation, TP/PP plans) is inherited unchanged.
+
+Three changes
+-------------
+1. Attention sink. Each attention module gets `self.sinks`, an
+   `[num_attention_heads]` learnable parameter, passed to the attention kernel as
+   `s_aux`. The kernel appends it as an extra column of softmax logits that is
+   dropped after normalization (see gpt-oss). NOTE: the `sdpa` backend does NOT
+   support `s_aux`; use `eager` (debug), `flash_attention_2`, `flash_attention_3`,
+   or `flex_attention`. We assert this in the attention forward.
+
+2. Packing-metadata reuse. On a flash backend with packed `position_ids`, derive
+   the varlen `cu_seqlens`/`max_seqlen` once in `Olmo3SinkModel.forward` and put
+   them in the kwargs threaded to every layer. `_flash_attention_forward` then
+   takes its precomputed-varlen branch instead of re-deriving per layer.
+
+3. Per-layer-type RoPE (bug fix). OLMo 3 applies RoPE scaling (YaRN) only to
+   `full_attention` layers; `sliding_attention` layers use default unscaled RoPE
+   (so it was pretrained, so vLLM serves it, and so transformers 4.57 ran it).
+   The transformers v5 RoPE refactor (huggingface/transformers#39847) regressed
+   this to one global scaled `rotary_emb` for every layer; the pinned 5.9.0 (and
+   main as of 2026-06) is affected. We restore the 4.57 behavior here, mirroring
+   the open upstream fix huggingface/transformers#45945: `rotary_embs` ModuleDict
+   with `sliding_attention` forced to `rope_type="default"`, dispatched per
+   `config.layer_types[i]` in `Olmo3SinkModel.forward`. Drop this override once
+   transformers ships the fix.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Callable
+
+import torch
+import torch.nn as nn
+
+from transformers.cache_utils import Cache, DynamicCache
+from transformers.masking_utils import create_causal_mask, create_sliding_window_causal_mask
+from transformers.modeling_flash_attention_utils import (
+    _is_packed_sequence,
+    prepare_fa_kwargs_from_position_ids,
+)
+from transformers.modeling_outputs import BaseModelOutputWithPast
+from transformers.modeling_rope_utils import ROPE_INIT_FUNCTIONS
+from transformers.modeling_utils import ALL_ATTENTION_FUNCTIONS
+from transformers.models.olmo3 import modeling_olmo3 as _olmo3_mod
+from transformers.models.olmo3.modeling_olmo3 import (
+    Olmo3Attention,
+    Olmo3DecoderLayer,
+    Olmo3ForCausalLM,
+    Olmo3Model,
+    Olmo3PreTrainedModel,
+    Olmo3RotaryEmbedding,
+    repeat_kv,
+)
+from transformers.processing_utils import Unpack
+from transformers.utils.generic import merge_with_config_defaults
+from transformers.utils.output_capturing import capture_outputs
+
+
+def eager_attention_forward_with_sink(
+    module,
+    query: torch.Tensor,   # [B, Hq, S, D]
+    key: torch.Tensor,
+    value: torch.Tensor,
+    attention_mask: torch.Tensor | None,
+    scaling: float,
+    dropout: float = 0.0,
+    s_aux: torch.Tensor | None = None,
+    **kwargs,
+):
+    """gpt-oss-style eager attention with a per-head sink (correct CPU/debug reference).
+
+    Identical to Olmo3's `eager_attention_forward` except the per-head `sinks` scalar is
+    appended as an extra softmax logit and dropped after normalization (so a head can
+    attend to "nothing"). Done in fp32 with a max-subtract for stability, matching the
+    in-kernel FA3 sink. Stock Olmo3 eager ignores the sink, so this is the reference that
+    actually exercises `self_attn.sinks` (e.g. the `eager` backend / CPU). Defined here
+    (not in attention.py) so it has NO FlashAttention import dependency."""
+    key_states = repeat_kv(key, module.num_key_value_groups)
+    value_states = repeat_kv(value, module.num_key_value_groups)
+
+    attn_weights = torch.matmul(query, key_states.transpose(2, 3)) * scaling
+    if attention_mask is not None:
+        attn_weights = attn_weights + attention_mask[..., : key_states.shape[-2]]
+
+    sink = s_aux if s_aux is not None else module.sinks  # [Hq]
+    sinks = sink.reshape(1, -1, 1, 1).expand(query.shape[0], -1, query.shape[-2], 1)
+    combined = torch.cat([attn_weights, sinks.to(attn_weights.dtype)], dim=-1).float()
+    combined = combined - combined.amax(dim=-1, keepdim=True)
+    probs = torch.softmax(combined, dim=-1)[..., :-1].to(query.dtype)  # drop sink column
+    probs = nn.functional.dropout(probs, p=dropout, training=module.training)
+
+    attn_output = torch.matmul(probs, value_states).transpose(1, 2).contiguous()
+    return attn_output, probs
+from transformers.utils.generic import TransformersKwargs
+
+from .configuration_olmo3_sink import Olmo3SinkConfig
+
+
+# OLMo 3 RoPE is identical to the stock rotary embedding, except:
+# - RoPE scaling is not applied to sliding window attention layers.
+class Olmo3SinkRotaryEmbedding(Olmo3RotaryEmbedding):
+    """`Olmo3RotaryEmbedding` + per-instance `rope_type` override (mirrors upstream
+    fix huggingface/transformers#45945; the 5.9.0 base class can only read the
+    rope type from the config, so it cannot express the default-RoPE sliding
+    layers)."""
+
+    def __init__(self, config: Olmo3SinkConfig, device=None, rope_type: str | None = None):
+        nn.Module.__init__(self)
+        self.max_seq_len_cached = config.max_position_embeddings
+        self.original_max_seq_len = config.max_position_embeddings
+
+        self.config = config
+
+        self.rope_type = rope_type or self.config.rope_parameters["rope_type"]
+        rope_init_fn: Callable = self.compute_default_rope_parameters
+        if self.rope_type != "default":
+            rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]
+        inv_freq, self.attention_scaling = rope_init_fn(self.config, device)
+
+        self.register_buffer("inv_freq", inv_freq, persistent=False)
+        self.register_buffer("original_inv_freq", inv_freq.clone(), persistent=False)
+
+
+class Olmo3SinkAttention(Olmo3Attention):
+    """Olmo3 attention + per-head learnable sink (`s_aux`)."""
+
+    def __init__(self, config: Olmo3SinkConfig, layer_idx: int):
+        super().__init__(config, layer_idx=layer_idx)
+        # One scalar logit per query head. Initialized by `_init_weights`.
+        self.sinks = nn.Parameter(torch.empty(config.num_attention_heads))
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        position_embeddings: tuple[torch.Tensor, torch.Tensor],
+        attention_mask: torch.Tensor | None,
+        past_key_values: Cache | None = None,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> tuple[torch.Tensor, torch.Tensor | None]:
+        # Body identical to Olmo3Attention.forward; the only change is the
+        # `s_aux=self.sinks` argument to the attention interface.
+        input_shape = hidden_states.shape[:-1]
+        hidden_shape = (*input_shape, -1, self.head_dim)
+
+        query_states = self.q_norm(self.q_proj(hidden_states))
+        key_states = self.k_norm(self.k_proj(hidden_states))
+        value_states = self.v_proj(hidden_states)
+
+        query_states = query_states.view(hidden_shape).transpose(1, 2)
+        key_states = key_states.view(hidden_shape).transpose(1, 2)
+        value_states = value_states.view(hidden_shape).transpose(1, 2)
+
+        cos, sin = position_embeddings
+        # Reference the module attribute at call time (not a bound import) so that
+        # Liger-Kernel's `apply_liger_kernel_to_olmo3(rope=True)` monkey-patch takes effect.
+        query_states, key_states = _olmo3_mod.apply_rotary_pos_emb(query_states, key_states, cos, sin)
+
+        if past_key_values is not None:
+            key_states, value_states = past_key_values.update(key_states, value_states, self.layer_idx)
+
+        attn_impl = self.config._attn_implementation
+        if attn_impl == "sdpa":
+            raise ValueError(
+                "Olmo3Sink uses attention sinks (s_aux), which the `sdpa` backend does not support. "
+                "Load with attn_implementation in {'eager','flash_attention_2','flash_attention_3','flex_attention'}."
+            )
+
+        # Fallback (incl. attn_impl="eager") is our SINK-AWARE eager, not Olmo3's
+        # sink-less one -- so the eager backend is a correct reference for the sink.
+        attention_interface: Callable = ALL_ATTENTION_FUNCTIONS.get_interface(
+            attn_impl, eager_attention_forward_with_sink)
+
+        attn_output, attn_weights = attention_interface(
+            self,
+            query_states,
+            key_states,
+            value_states,
+            attention_mask,
+            dropout=0.0 if not self.training else self.attention_dropout,
+            scaling=self.scaling,
+            sliding_window=self.sliding_window,
+            s_aux=self.sinks,
+            **kwargs,
+        )
+
+        attn_output = attn_output.reshape(*input_shape, -1).contiguous()
+        attn_output = self.o_proj(attn_output)
+        return attn_output, attn_weights
+
+
+class Olmo3SinkDecoderLayer(Olmo3DecoderLayer):
+    def __init__(self, config: Olmo3SinkConfig, layer_idx: int):
+        super().__init__(config, layer_idx=layer_idx)
+        self.self_attn = Olmo3SinkAttention(config=config, layer_idx=layer_idx)
+
+
+class Olmo3SinkPreTrainedModel(Olmo3PreTrainedModel):
+    config_class = Olmo3SinkConfig
+    _no_split_modules = ["Olmo3SinkDecoderLayer"]
+
+    def _init_weights(self, module):
+        super()._init_weights(module)
+        # The sink parameter is new (absent from base Olmo3 checkpoints); when
+        # loading such a checkpoint it is a "missing key" and lands here.
+        # transformers >= 5.x calls _init_weights on EVERY module after loading
+        # (relying on guarded init fns to skip loaded params); a raw fill_()
+        # bypasses that guard and would silently zero TRAINED sinks, so only
+        # fill when the param was not loaded from the checkpoint.
+        if isinstance(module, Olmo3SinkAttention) and not getattr(
+            module.sinks, "_is_hf_initialized", False
+        ):
+            module.sinks.data.fill_(self.config.sink_init_value)
+
+
+class Olmo3SinkModel(Olmo3SinkPreTrainedModel, Olmo3Model):
+    def __init__(self, config: Olmo3SinkConfig):
+        super().__init__(config)
+        self.layers = nn.ModuleList(
+            [Olmo3SinkDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
+        )
+        # Per-layer-type RoPE (mirrors huggingface/transformers#45945): sliding
+        # layers use default unscaled RoPE; full-attention layers use the
+        # configured (e.g. YaRN) RoPE. Replaces the buggy global `rotary_emb`.
+        self.rotary_embs = nn.ModuleDict(
+            {
+                "sliding_attention": Olmo3SinkRotaryEmbedding(config=config, rope_type="default"),
+                "full_attention": Olmo3SinkRotaryEmbedding(config=config),
+            }
+        )
+        del self.rotary_emb
+        self.post_init()
+
+    # Body mirrors stock `Olmo3Model.forward` (transformers 5.9.0, incl. its
+    # decorators) with two changes: (a) the packing-metadata reuse block, (b)
+    # per-layer-type `position_embeddings` from `self.rotary_embs` instead of one
+    # global `self.rotary_emb`. Reimplemented (not `super().forward()`) because
+    # the per-layer dispatch lives inside the layer loop.
+    @merge_with_config_defaults
+    @capture_outputs
+    def forward(
+        self,
+        input_ids: torch.LongTensor | None = None,
+        attention_mask: torch.Tensor | None = None,
+        position_ids: torch.LongTensor | None = None,
+        past_key_values: Cache | None = None,
+        inputs_embeds: torch.FloatTensor | None = None,
+        use_cache: bool | None = None,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> BaseModelOutputWithPast:
+        if (input_ids is None) ^ (inputs_embeds is not None):
+            raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
+
+        if inputs_embeds is None:
+            inputs_embeds = self.embed_tokens(input_ids)
+
+        if use_cache and past_key_values is None:
+            past_key_values = DynamicCache(config=self.config)
+
+        if position_ids is None:
+            past_seen_tokens = past_key_values.get_seq_length() if past_key_values is not None else 0
+            position_ids = torch.arange(inputs_embeds.shape[1], device=inputs_embeds.device) + past_seen_tokens
+            position_ids = position_ids.unsqueeze(0)
+
+        impl = self.config._attn_implementation or ""
+        if (
+            getattr(self.config, "reuse_packing_metadata", True)
+            and ("flash" in impl or impl == "olmo3_sink_fa3")
+            and kwargs.get("cu_seq_lens_q") is None  # don't clobber model-provided varlen kwargs
+            and _is_packed_sequence(position_ids, inputs_embeds.shape[0])
+        ):
+            (cu_q, cu_k), (max_q, max_k) = prepare_fa_kwargs_from_position_ids(position_ids)
+            # Compute the varlen metadata once here and thread it to every layer.
+            # int() so per-layer attention avoids a device->host sync on max_seqlen.
+            kwargs["cu_seq_lens_q"] = cu_q
+            kwargs["cu_seq_lens_k"] = cu_k
+            kwargs["max_length_q"] = int(max_q)
+            kwargs["max_length_k"] = int(max_k)
+
+        # It may already have been prepared by e.g. `generate`
+        if not isinstance(causal_mask_mapping := attention_mask, dict):
+            # Prepare mask arguments
+            mask_kwargs = {
+                "config": self.config,
+                "inputs_embeds": inputs_embeds,
+                "attention_mask": attention_mask,
+                "past_key_values": past_key_values,
+                "position_ids": position_ids,
+            }
+            # Create the masks
+            causal_mask_mapping = {
+                "full_attention": create_causal_mask(**mask_kwargs),
+                "sliding_attention": create_sliding_window_causal_mask(**mask_kwargs),
+            }
+
+        hidden_states = inputs_embeds
+        position_embeddings_mapping = {
+            "sliding_attention": self.rotary_embs["sliding_attention"](hidden_states, position_ids),
+            "full_attention": self.rotary_embs["full_attention"](hidden_states, position_ids),
+        }
+
+        for i, decoder_layer in enumerate(self.layers[: self.config.num_hidden_layers]):
+            hidden_states = decoder_layer(
+                hidden_states,
+                attention_mask=causal_mask_mapping[self.config.layer_types[i]],
+                position_ids=position_ids,
+                past_key_values=past_key_values,
+                position_embeddings=position_embeddings_mapping[self.config.layer_types[i]],
+                **kwargs,
+            )
+
+        hidden_states = self.norm(hidden_states)
+        return BaseModelOutputWithPast(
+            last_hidden_state=hidden_states,
+            past_key_values=past_key_values,
+        )
+
+
+class Olmo3SinkForCausalLM(Olmo3SinkPreTrainedModel, Olmo3ForCausalLM):
+    def __init__(self, config: Olmo3SinkConfig):
+        super().__init__(config)
+        self.model = Olmo3SinkModel(config)
+        self.vocab_size = config.vocab_size
+        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+        self.post_init()
+
+
+# Auto-register the in-kernel FA3 sink attention so `attn_implementation="olmo3_sink_fa3"`
+# works on import (incl. trust_remote_code). Guarded: if the patched FA3 isn't installed,
+# import still succeeds and other backends (eager) remain usable.
+try:
+    from .attention import register_fa3_sink_attention
+
+    register_fa3_sink_attention()
+except Exception:  # noqa: BLE001
+    pass
+
+
+__all__ = [
+    "Olmo3SinkAttention",
+    "Olmo3SinkDecoderLayer",
+    "Olmo3SinkPreTrainedModel",
+    "Olmo3SinkRotaryEmbedding",
+    "Olmo3SinkModel",
+    "Olmo3SinkForCausalLM",
+]

step02-olmo32b-deepseektok-sink/sink_provenance.json

@@ -0,0 +1,20 @@
+{
+ "pipeline": "olmo3_sink.build_init_model (e2e: native -> transplant -> measured sinks)",
+ "base": "/work/u3063708/BigModels/Olmo-3.1-32B-Think",
+ "transplant_config": "/work/u3063708/proof-pilot/tokenizer_transplant/configs/olmo3_think_32b__deepseek_v4_flash.yaml",
+ "transplant_out": "/work/u3063708/BigModels/Olmo-3.1-32B-Think-deepseekTok",
+ "calib_text": "/work/u3063708/proof-pilot/evaluation/data/deepseek-math-v2.txt",
+ "calib_text_sha256": "d3f478ac5917266a",
+ "seq_len": 8192,
+ "qmin": 4096,
+ "eps": 0.04,
+ "p_floor": 0.02,
+ "p_cap": 0.1,
+ "sinks_mean": 6.637911796569824,
+ "sinks_min": -0.3041238784790039,
+ "sinks_max": 13.720010757446289,
+ "D_mean": 0.22890444099903107,
+ "lm_loss_before": 1.3446719646453857,
+ "lm_loss_after": 1.3480974435806274,
+ "git_commit": "e9f882f18e2e645a6f07d90362c1e50a929b420c"
+}

step02-olmo32b-deepseektok-sink/tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<｜begin▁of▁sentence｜>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<｜end▁of▁sentence｜>",
+  "is_local": true,
+  "legacy": true,
+  "local_files_only": false,
+  "model_max_length": 1048576,
+  "pad_token": "<｜▁pad▁｜>",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": null
+}

step04-sft-stage1-32b/chat_template.jinja

@@ -0,0 +1,23 @@
+{{- bos_token -}}
+{%- if messages and messages[0]['role'] == 'system' -%}
+{{- messages[0]['content'] -}}
+{%- set loop_messages = messages[1:] -%}
+{%- else -%}
+{%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+{%- if message['role'] == 'user' -%}
+{{- '<｜User｜>' + message['content'] -}}
+{%- elif message['role'] == 'assistant' -%}
+{{- '<｜Assistant｜>' -}}
+{%- if message.get('reasoning_content') -%}
+{{- '<think>' + message['reasoning_content'] + '</think>' -}}
+{%- else -%}
+{{- '</think>' -}}
+{%- endif -%}
+{{- message['content'] + eos_token -}}
+{%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+{{- '<｜Assistant｜><think>' -}}
+{%- endif -%}

step04-sft-stage1-32b/config.json

@@ -0,0 +1,107 @@
+{
+  "architectures": [
+    "Olmo3SinkForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_olmo3_sink.Olmo3SinkConfig",
+    "AutoModelForCausalLM": "modeling_olmo3_sink.Olmo3SinkForCausalLM"
+  },
+  "bos_token_id": 0,
+  "dtype": "float32",
+  "eos_token_id": 1,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 27648,
+  "layer_types": [
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 65536,
+  "model_type": "olmo3_sink",
+  "num_attention_heads": 40,
+  "num_hidden_layers": 64,
+  "num_key_value_heads": 8,
+  "pad_token_id": 2,
+  "reuse_packing_metadata": true,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "attention_factor": 1.2079441541679836,
+    "beta_fast": 32.0,
+    "beta_slow": 1.0,
+    "factor": 8.0,
+    "original_max_position_embeddings": 8192,
+    "rope_theta": 500000,
+    "rope_type": "yarn"
+  },
+  "sink_init_value": 0.0,
+  "sliding_window": 4096,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.9.0",
+  "use_cache": false,
+  "vocab_size": 129280
+}

step04-sft-stage1-32b/generation_config.json

@@ -0,0 +1,11 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "do_sample": true,
+  "eos_token_id": 1,
+  "max_new_tokens": 32768,
+  "pad_token_id": 2,
+  "temperature": 0.6,
+  "top_p": 0.95,
+  "transformers_version": "5.9.0"
+}

step04-sft-stage1-32b/model.safetensors.index.json

@@ -0,0 +1,779 @@
+{
+  "metadata": {
+    "total_parameters": 32530505216,
+    "total_size": 65061010432
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00001-of-00002.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.48.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.48.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.50.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.60.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.norm.weight": "model-00002-of-00002.safetensors"
+  }
+}

step04-sft-stage1-32b/tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<｜begin▁of▁sentence｜>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<｜end▁of▁sentence｜>",
+  "is_local": true,
+  "legacy": true,
+  "local_files_only": false,
+  "model_max_length": 1048576,
+  "pad_token": "<｜▁pad▁｜>",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": null
+}

step05-yarn256k-config/config.json

@@ -0,0 +1,107 @@
+{
+  "architectures": [
+    "Olmo3SinkForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_olmo3_sink.Olmo3SinkConfig",
+    "AutoModelForCausalLM": "modeling_olmo3_sink.Olmo3SinkForCausalLM"
+  },
+  "bos_token_id": 0,
+  "dtype": "float32",
+  "eos_token_id": 1,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 27648,
+  "layer_types": [
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 262144,
+  "model_type": "olmo3_sink",
+  "num_attention_heads": 40,
+  "num_hidden_layers": 64,
+  "num_key_value_heads": 8,
+  "pad_token_id": 2,
+  "reuse_packing_metadata": true,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "attention_factor": 1.3465735902799727,
+    "beta_fast": 32.0,
+    "beta_slow": 1.0,
+    "factor": 32.0,
+    "original_max_position_embeddings": 8192,
+    "rope_theta": 500000,
+    "rope_type": "yarn"
+  },
+  "sink_init_value": 0.0,
+  "sliding_window": 4096,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.9.0",
+  "use_cache": false,
+  "vocab_size": 129280
+}

step07-softdistill-32b-config/config.json

@@ -0,0 +1,107 @@
+{
+  "architectures": [
+    "Olmo3SinkForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_olmo3_sink.Olmo3SinkConfig",
+    "AutoModelForCausalLM": "modeling_olmo3_sink.Olmo3SinkForCausalLM"
+  },
+  "bos_token_id": 0,
+  "dtype": "bfloat16",
+  "eos_token_id": 1,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 27648,
+  "layer_types": [
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 262144,
+  "model_type": "olmo3_sink",
+  "num_attention_heads": 40,
+  "num_hidden_layers": 64,
+  "num_key_value_heads": 8,
+  "pad_token_id": 2,
+  "reuse_packing_metadata": true,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "attention_factor": 1.3465735902799727,
+    "beta_fast": 32.0,
+    "beta_slow": 1.0,
+    "factor": 32.0,
+    "original_max_position_embeddings": 8192,
+    "rope_theta": 500000,
+    "rope_type": "yarn"
+  },
+  "sink_init_value": 0.0,
+  "sliding_window": 4096,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.9.0",
+  "use_cache": false,
+  "vocab_size": 129280
+}

step09-softdistill-v2test-32b/chat_template.jinja

@@ -0,0 +1,23 @@
+{{- bos_token -}}
+{%- if messages and messages[0]['role'] == 'system' -%}
+{{- messages[0]['content'] -}}
+{%- set loop_messages = messages[1:] -%}
+{%- else -%}
+{%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+{%- if message['role'] == 'user' -%}
+{{- '<｜User｜>' + message['content'] -}}
+{%- elif message['role'] == 'assistant' -%}
+{{- '<｜Assistant｜>' -}}
+{%- if message.get('reasoning_content') -%}
+{{- '<think>' + message['reasoning_content'] + '</think>' -}}
+{%- else -%}
+{{- '</think>' -}}
+{%- endif -%}
+{{- message['content'] + eos_token -}}
+{%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+{{- '<｜Assistant｜><think>' -}}
+{%- endif -%}

step09-softdistill-v2test-32b/config.json

@@ -0,0 +1,107 @@
+{
+  "architectures": [
+    "Olmo3SinkForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_olmo3_sink.Olmo3SinkConfig",
+    "AutoModelForCausalLM": "modeling_olmo3_sink.Olmo3SinkForCausalLM"
+  },
+  "bos_token_id": 0,
+  "dtype": "bfloat16",
+  "eos_token_id": 1,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 27648,
+  "layer_types": [
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 262144,
+  "model_type": "olmo3_sink",
+  "num_attention_heads": 40,
+  "num_hidden_layers": 64,
+  "num_key_value_heads": 8,
+  "pad_token_id": 2,
+  "reuse_packing_metadata": true,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "attention_factor": 1.3465735902799727,
+    "beta_fast": 32.0,
+    "beta_slow": 1.0,
+    "factor": 32.0,
+    "original_max_position_embeddings": 8192,
+    "rope_theta": 500000,
+    "rope_type": "yarn"
+  },
+  "sink_init_value": 0.0,
+  "sliding_window": 4096,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.9.0",
+  "use_cache": false,
+  "vocab_size": 129280
+}

step09-softdistill-v2test-32b/generation_config.json

@@ -0,0 +1,11 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "do_sample": true,
+  "eos_token_id": 1,
+  "max_new_tokens": 32768,
+  "pad_token_id": 2,
+  "temperature": 0.6,
+  "top_p": 0.95,
+  "transformers_version": "5.9.0"
+}

step09-softdistill-v2test-32b/model-00001-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:22792c844526628f014a775166825a8bd82bfe0ee614a474acd40b46dc1b9518
+size 49740847248

step09-softdistill-v2test-32b/model.safetensors.index.json

@@ -0,0 +1,779 @@
+{
+  "metadata": {
+    "total_parameters": 32530505216,
+    "total_size": 65061010432
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00001-of-00002.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.48.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.48.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.50.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.60.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.norm.weight": "model-00002-of-00002.safetensors"
+  }
+}

step09-softdistill-v2test-32b/tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<｜begin▁of▁sentence｜>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<｜end▁of▁sentence｜>",
+  "is_local": true,
+  "legacy": true,
+  "local_files_only": false,
+  "model_max_length": 1048576,
+  "pad_token": "<｜▁pad▁｜>",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": null
+}

step10-opd-32b-s150/chat_template.jinja

@@ -0,0 +1,23 @@
+{{- bos_token -}}
+{%- if messages and messages[0]['role'] == 'system' -%}
+{{- messages[0]['content'] -}}
+{%- set loop_messages = messages[1:] -%}
+{%- else -%}
+{%- set loop_messages = messages -%}
+{%- endif -%}
+{%- for message in loop_messages -%}
+{%- if message['role'] == 'user' -%}
+{{- '<｜User｜>' + message['content'] -}}
+{%- elif message['role'] == 'assistant' -%}
+{{- '<｜Assistant｜>' -}}
+{%- if message.get('reasoning_content') -%}
+{{- '<think>' + message['reasoning_content'] + '</think>' -}}
+{%- else -%}
+{{- '</think>' -}}
+{%- endif -%}
+{{- message['content'] + eos_token -}}
+{%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt -%}
+{{- '<｜Assistant｜><think>' -}}
+{%- endif -%}

step10-opd-32b-s150/config.json

@@ -0,0 +1,171 @@
+{
+  "architectures": [
+    "Olmo3SinkForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "dtype": "bfloat16",
+  "eos_token_id": 1,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 27648,
+  "layer_types": [
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 262144,
+  "model_type": "olmo3",
+  "num_attention_heads": 40,
+  "num_hidden_layers": 64,
+  "num_key_value_heads": 8,
+  "pad_token_id": 2,
+  "reuse_packing_metadata": true,
+  "rms_norm_eps": 1e-06,
+  "sink_init_value": 0.0,
+  "sliding_window": 4096,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.9.0",
+  "use_cache": true,
+  "vocab_size": 129280,
+  "torch_dtype": "bfloat16",
+  "rope_theta": 500000,
+  "rope_scaling": {
+    "attention_factor": 1.3465735902799727,
+    "beta_fast": 32.0,
+    "beta_slow": 1.0,
+    "factor": 32.0,
+    "original_max_position_embeddings": 8192,
+    "rope_type": "yarn"
+  },
+  "is_hybrid_swa": true,
+  "hybrid_layer_pattern": [
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0,
+    1,
+    1,
+    1,
+    0
+  ]
+}

step10-opd-32b-s150/config.json.orig

@@ -0,0 +1,104 @@
+{
+  "architectures": [
+    "Olmo3SinkForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 0,
+  "dtype": "bfloat16",
+  "eos_token_id": 1,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 27648,
+  "layer_types": [
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "max_position_embeddings": 262144,
+  "model_type": "olmo3",
+  "num_attention_heads": 40,
+  "num_hidden_layers": 64,
+  "num_key_value_heads": 8,
+  "pad_token_id": 2,
+  "reuse_packing_metadata": true,
+  "rms_norm_eps": 1e-06,
+  "sink_init_value": 0.0,
+  "sliding_window": 4096,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.9.0",
+  "use_cache": true,
+  "vocab_size": 129280,
+  "torch_dtype": "bfloat16",
+  "rope_theta": 500000,
+  "rope_scaling": {
+    "attention_factor": 1.3465735902799727,
+    "beta_fast": 32.0,
+    "beta_slow": 1.0,
+    "factor": 32.0,
+    "original_max_position_embeddings": 8192,
+    "rope_type": "yarn"
+  }
+}

step10-opd-32b-s150/generation_config.json

@@ -0,0 +1,11 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 0,
+  "do_sample": true,
+  "eos_token_id": 1,
+  "max_new_tokens": 32768,
+  "pad_token_id": 2,
+  "temperature": 0.6,
+  "top_p": 0.95,
+  "transformers_version": "5.9.0"
+}

step10-opd-32b-s150/model-00001-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8246b92f4cf317f3604d89669cc1a9b9b9f88c1b3185a016f0b7835d5d2864b0
+size 49740847248

step10-opd-32b-s150/model-00002-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f87d65b82d0575ff5df20b895a3107c095946e7b34a8c46e8e7a64e74d0e5c82
+size 15320253552

step10-opd-32b-s150/model.safetensors.index.json

@@ -0,0 +1,779 @@
+{
+  "metadata": {
+    "total_parameters": 32530505216,
+    "total_size": 65061010432
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00001-of-00002.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.26.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.27.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.28.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.29.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.30.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.31.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.32.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.33.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.34.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.35.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.36.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.37.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.38.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.39.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.40.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.41.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.42.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.43.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.44.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.45.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.46.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.47.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.48.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.48.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.48.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.49.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.5.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.50.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.50.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.51.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.52.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.53.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.54.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.55.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.56.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.57.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.58.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.59.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.6.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.60.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.60.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.61.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.62.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.mlp.down_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.mlp.gate_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.mlp.up_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.post_feedforward_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.k_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.k_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.o_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.q_norm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.q_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.sinks": "model-00002-of-00002.safetensors",
+    "model.layers.63.self_attn.v_proj.weight": "model-00002-of-00002.safetensors",
+    "model.layers.7.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.down_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.gate_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.up_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.post_feedforward_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.k_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.k_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.o_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.q_norm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.q_proj.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.sinks": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.v_proj.weight": "model-00001-of-00002.safetensors",
+    "model.norm.weight": "model-00002-of-00002.safetensors"
+  }
+}

step10-opd-32b-s150/tokenizer_config.json

@@ -0,0 +1,14 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<｜begin▁of▁sentence｜>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<｜end▁of▁sentence｜>",
+  "is_local": true,
+  "legacy": true,
+  "local_files_only": false,
+  "model_max_length": 1048576,
+  "pad_token": "<｜▁pad▁｜>",
+  "sp_model_kwargs": {},
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": null
+}