+
++ NVFP4 | + MXFP8 | + Playground | + Tinker Cookbook | + Acceptable Use +
+ +## 1. General Information + +Inkling 2 Small is a general-purpose multimodal model that accepts text and image inputs and generates text outputs. It is intended for use in English and other languages, and across multiple coding languages. The model is designed to be used by developers building AI-powered applications, including agentic and tool-use systems, coding assistants, chatbots, and retrieval-augmented generation systems, and is suitable for general-purpose conversational use, instruction-following, and other natural language and visual tasks. It is released with open weights to support research, fine-tuning and integration into third-party products by downstream developers. + +**Languages:** English, with general multilingual capabilities across other languages. + +## 2. Getting Started + +Try Inkling 2 Small on the [Tinker Playground](https://tinker.thinkingmachines.ai/playground) or access via API using the [Tinker Cookbook](https://github.com/thinking-machines-lab/tinker-cookbook). To fine-tune Inkling 2 Small with Tinker, also install the [tml-renderers](https://pypi.org/project/tml-renderers/) package. + +Inkling 2 Small supports local inference using: + +* SGLang ([recipe](https://docs.sglang.io/cookbook/autoregressive/ThinkingMachines/Inkling-2-Small)) +* vLLM ([recipe](https://recipes.vllm.ai/thinkingmachines/Inkling-2-Small)) +* TokenSpeed ([recipe](https://lightseek.org/tokenspeed/recipes/models#inkling-2)) +* Unsloth ([recipe](https://unsloth.ai/docs/models/inkling)) + +Inkling 2 Small can also be post-trained with: + +* [Miles](https://miles.radixark.com/docs/models/thinkingmachines/inkling-2-small) +* [SkyRL](https://docs.skyrl.ai/docs/models/inkling2) +* [Snowflake Cortex Training](https://github.com/snowflakedb/cortex-training) + +API access is also available through third party inference providers, including: + +* NVIDIA +* Fireworks +* [Together](https://www.together.ai/models/inkling-2-small) +* [Baseten](https://www.baseten.co/library/inkling-2-small) +* [Nebius](https://tokenfactory.nebius.com/endpoints/public/thinkingmachines%2FInkling-2-Small) +* [Modal](https://modal.com/library/thinking-machines/inkling-2-small) +* [Cloudflare](https://developers.cloudflare.com/workers-ai/models/inkling-2-small/) + +### Required hardware + +The model is distributed in two checkpoint formats. + +The main NVFP4 checkpoint requires a GPU cluster with at least 500GB of aggregated VRAM. This can be met with either of the following example configurations: + +* 2x NVIDIA B300 GPUs +* 4x NVIDIA H200 GPUs + +The alternative MXFP8 checkpoint requires a GPU cluster with at least 800GB of aggregated VRAM. This can be met with either of the following example configurations: + +* 4x NVIDIA B300 GPUs +* 8x NVIDIA H200 GPUs + +The model was pretrained, midtrained and post-trained in NVFP4, so this is the native precision for the model. We recommend the NVFP4 checkpoint for all use-cases, and we're confident that it's the best option. The MXFP8 has a very marginal 1–2 percentage points gain over the NVFP4 checkpoint over evals, but we do not think it is meaningful for any practical use-cases. + +### Required software + +Running the model directly on GPU hardware requires an inference deployment framework — either SGLang, vLLM, TokenSpeed, Unsloth, or Hugging Face, along with all of their respective dependency libraries. + +## 3. Model Properties + +### Model type + +Multimodal autoregressive transformer + +### Architecture type + +A 48-layer decoder-only transformer with a sparse Mixture-of-Experts (MoE) feed-forward backbone: each token is routed to 8 of 384 experts. Attention is a hybrid of local and global layers. The model is natively multimodal — images are encoded via a vision transformer — with image and text inputs projected into a shared hidden space and processed jointly by the decoder. + +### Parameters + +594B total, 17B active + +### Numerics support + +MXFP8 and NVFP4 + +### Input modalities + +Inkling 2 Small accepts text input in UTF-8 encoding and image input in any pixel-based format (with each dimension ideally between 40px and 4096px for optimal performance). It supports a context window of up to 1M tokens. + +### Output modalities + +Inkling 2 Small generates output as UTF-8 encoded text. + +## 4. Training + +Training data includes a broad variety of content types, including text, images, video. Training data for the model was drawn from publicly available sources, acquired from third-parties, generated internally, or synthetically generated or augmented. Publicly available data includes content from the public internet and publicly accessible repositories. + +The training data curation process includes cleaning, processing, and modifying datasets. These processing steps, which vary by data type, may include deduplication and filtering to remove junk or other low-quality data, or to advance safety or other objectives. + +## 5. Evaluations + +| Open Weights Flash | Open Weights Pro | Closed Weights Flash | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Inkling 2 Small | GLM-5.3-Flash | MiMo V2.6 Flash | MiniMax-M3.1-Flash | DeepSeek-V4.1-Flash-1017 | Muse Spark Open | GLM 5.3 | MiMo V2.6 Pro | Mistral Large 4 | Kimi K3 | GPT-6 Luna | Gemini 3.8 Flash | Claude Haiku 5.5 | |
| Model Info | |||||||||||||
| Params (B) activated / total | 17/594 | 18/320 | 15/309 | 23/428 | 16/552 | 42/816 | 40/753 | 42/1020 | 49/1050 | 104/2800 | – | – | – |
| Third Party Index | |||||||||||||
| AA Index (v5.0) | 30.6 | 27.8 | 26.7 | 29.1 | 32.6 | 24.9 | 31.7 | 29.9 | 26.1 | – | 27.9 | 30.8 | 36.5 |
| Vals Index (v2.2) | 56.1% | 50.9% | 54.8% | 53.8% | 61.3% | 48.6% | 59.7% | 57.8% | 56.5% | 53.0% | 52.3% | 56.0% | 58.7% |
| Coding | |||||||||||||
| FrontierCode v2.0 stirrup v0.2.1 | 25.4% | 17.6% | 21.6% | 20.8% | 28.5% | 7.6% | 21.6% | 24.0% | 15.0% | 17.3% | 25.2% | 18.7% | 34.9% |
| DeepSWE v1.1 mini-swe-agent v2.4.6 | 72.1% | 63.4% | 67.9% | 73.7% | 77.0% | 60.0% | 69.0% | 71.9% | 61.7% | 68.5% | 66.6% | 73.8% | 59.9% |
| Terminal Bench Science v0.1 mini-swe-agent v2.4.6 | 5.7% | 4.8% | 4.3% | 3.3% | 12.9% | 1.9% | 9.5% | 5.7% | 4.3% | 8.1% | 8.6% | 10.0% | 20.0% |
| Vibe Code Bench v1.1 openhands (vals) | 81.6% | 30.8% | 79.0% | 73.9% | 87.7% | 76.2% | 78.1% | 85.2% | 81.6% | 85.0% | 81.7% | 78.7% | 90.4% |
| SWE-Atlas v1.1 mini-swe-agent v2.4.6 | 61.9% | – | – | – | 62.1% | 42.3% | – | – | – | 61.0% | 50.8% | 57.3% | 59.4% |
| SWEBench Pro v2 (Hard) mini-swe-agent v2.4.6 | 86.3% | 82.8% | 89.7% | 84.3% | 89.7% | 82.4% | 84.3% | 89.2% | 83.8% | 88.2% | 88.7% | 58.8% | 52.9% |
| Sudo L7 | 14.1% | – | – | – | – | – | 9.4% | – | 13.9% | 12.2% | – | 10.6% | – |
| Autoresearch | |||||||||||||
| PostTrainBench v1.2 Inkling 2 Small in Claude Code | 31.9% | 36.8% | – | – | 30.5% | – | 34.5% | – | – | 31.5% | – | – | – |
| PMPP-Hard (AUC, 0–10h) pi v0.83.0 | 494.27 | – | – | – | 497.36 | – | 430.96 | – | 478.39 | 453.58 | – | – | – |
| BioMysteryBench terminus v2 | 70.0% | – | 69.3% | – | – | – | – | – | – | 72.6% | 61.5% | 62.2% | 70.4% |
| Search and Tools | |||||||||||||
| BrowseComp Inkling 2 Small with compaction | 90.2% | 76.0% | 78.1% | 69.5% | 87.8% | 68.7% | 75.1% | – | 84.5% | 88.4% | 83.6% | 89.4% | 80.2% |
| DeepSearchQA Inkling 2 Small with compaction | 90.9% | 84.3% | 86.0% | 82.7% | 84.5% | 80.4% | 84.7% | 90.0% | 84.4% | 95.0% | 83.5% | 90.1% | 83.2% |
| MCP Atlas Full / Public | 85.4/85.0% | –/82.0% | –/62.9% | –/80.4% | –/83.2% | 89.2/87.5% | 84.2/83.6% | –/84.4% | –/84.5% | 82.3/84.2% | –/77.8% | –/87.5% | –/81.0% |
| Knowledge Work | |||||||||||||
| AA-Briefcase v1.1 | 1527 | 1451 | 1487 | 1476 | 1570 | 1279 | 1503 | 1512 | 1348 | 1497 | 1337 | 1203 | 1574 |
| GDPVal-AA v2.1 | 1630 | 1649 | 1614 | 1465 | 1746 | 1401 | 1657 | 1689 | 1384 | 1543 | 1444 | 1445 | 1631 |
| JobBench | 70.5% | 60.7% | 61.2% | 67.5% | 68.5% | 61.8% | 61.4% | 62.0% | 55.2% | 54.3% | 49.0% | 63.4% | 39.2% |
| DAYJOB (Healthcare) | 6.4% | 1.2% | – | – | 6.0% | – | 4.4% | – | 3.6% | 4.0% | 1.2% | 2.0% | – |
| Enterprise Operations | |||||||||||||
| EnterpriseOps-Gym | 46.8% | 33.2% | 29.0% | 48.9% | 43.6% | 43.1% | 36.4% | 38.0% | 44.9% | 45.3% | 24.9% | 45.7% | 25.1% |
| EnterpriseBench-CoreCraft | 71.8% | 46.7% | – | – | 58.5% | – | 51.8% | – | 47.7% | 43.1% | 43.1% | 58.5% | – |
| Finance Work | |||||||||||||
| Finance Agent v2.1 | 67.5% | 60.6% | 63.3% | 58.8% | 62.9% | 60.7% | 65.0% | 64.3% | 61.7% | 60.0% | 56.8% | 65.8% | 65.4% |
| Excel Modeling Benchmark v1.1 | 69.5% | 66.3% | 68.9% | 67.8% | 69.2% | 62.0% | 69.1% | 67.4% | 68.7% | 71.3% | 70.0% | 72.9% | 65.5% |
| Tax Agent Bench | 65.8% | 62.0% | 59.9% | 62.7% | 74.9% | 52.1% | 73.1% | 64.9% | 67.5% | 68.7% | 58.9% | 66.8% | 62.2% |
| Apex-Accounting v1.1 pass@1 | 14.5% | 12.3% | – | 13.5% | 12.5% | 13.8% | 13.9% | – | 12.0% | 9.5% | 10.8% | 15.5% | 15.3% |
| DAYJOB (Finance) | 14.3% | 7.3% | – | – | 12.5% | – | 11.0% | – | 4.8% | 5.5% | 6.8% | 4.5% | – |
| DeepFinance | 29.8% | – | – | – | – | – | 27.5% | – | 21.5% | – | 26.4% | – | 40.0% |
| Diligence Bench Best harness | 58.1% | 48.3% | – | 51.6% | 50.4% | 59.0% | 49.4% | 51.8% | 51.5% | 50.9% | 42.1% | 51.1% | 40.8% |
| AA-AnalystAgent | 51.2% | 25.0% | – | – | – | – | 38.8% | 37.5% | – | 38.8% | 43.8% | 46.3% | 53.8% |
| Tau 3 Banking v1.0.1 | 40.4% | 47.2% | – | – | – | – | – | – | – | 33.4% | – | – | – |
| ATLAS-Finance v1.1 | 6.0% | – | – | – | – | – | 3.0% | – | – | 3.7% | – | 3.7% | – |
| Legal Work | |||||||||||||
| Harvey LAB v1.1 | 21.7% | 10.4% | 13.3% | 7.5% | 14.2% | 25.0% | 14.6% | 13.8% | 15.8% | 12.9% | 4.2% | 12.1% | 4.2% |
| Legal Research Bench | 46.6% | 45.2% | 38.0% | 34.1% | 45.2% | 38.5% | 49.0% | 47.1% | 35.6% | 46.2% | 30.3% | 38.9% | 43.8% |
| Visual Creation | |||||||||||||
| DesignArena Web Apps | 1286 | – | – | – | – | – | 1271 | 1274 | – | 1280 | 1160 | 1224 | – |
| DesignArena Website | 1331 | 1266 | – | 1275 | 1296 | 1300 | 1285 | 1303 | 1253 | 1330 | 1265 | 1295 | 1257 |
| DesignArena Game Dev | 1424 | 1270 | – | 1307 | 1261 | 1254 | 1292 | 1309 | 1218 | 1351 | 1254 | 1298 | 1314 |
| DesignArena 3D Design | 1457 | 1300 | – | 1319 | 1344 | 1255 | 1304 | 1322 | 1239 | 1367 | 1199 | 1279 | 1329 |
| Code Arena (WebDev) | 1653 | 1607 | 1637 | 1582 | 1665 | – | 1623 | 1625 | 1544 | 1656 | 1579 | 1583 | 1589 |
| AA-3D | 1021 | 937 | – | – | – | – | 782 | 977 | – | 982 | – | 965 | 994 |
| Code2Video-AA | 1020 | 975 | 1052 | – | – | – | 1034 | 1052 | – | 1041 | 1030 | 899 | 1192 |
| BVB | 71.6% | 67.9% | – | – | 70.0% | – | – | – | – | – | 70.3% | 65.9% | 68.4% |
| Visual Perception | |||||||||||||
| PerceptionBench | 64.7% | 49.6% | 39.1% | 47.1% | 47.7% | 44.1% | – | 38.9% | 46.5% | 58.5% | 56.2% | 68.7% | 60.5% |
| Humanity’s Sixth Sense | 33.3% | 26.8% | 25.4% | – | 30.1% | 29.6% | – | 25.2% | – | 25.5% | 21.0% | 41.6% | – |
| BabyVision original / with Python | 68.6/93.6% | 53.4/84.0% | 16.9/68.4% | 56.1/91.0% | 44.9/92.5% | 44.3/80.2% | – | 19.0/73.8% | 42.3/71.0% | 53.4/85.7% | 55.3/79.6% | 72.8/92.8% | 48.3/93.9% |
| ScreenSpot Pro original / with Python | 83.8/92.0% | –/85.3% | 35.4/72.5% | 48.5/76.8% | – | – | – | 56.6/79.7% | 42.3/43.4% | 73.8/86.1% | 81.1/87.7% | 65.7/93.7% | 85.4/88.7% |
| OmniDocBench v1.6 | 94.6% | 92.3% | 87.4% | 92.0% | 91.9% | 93.8% | – | 88.4% | 90.2% | 95.2% | 92.3% | 89.7% | 92.6% |
| SimpleVQA with search and tools | 68.2% | 67.1% | – | – | – | 63.6% | – | – | 57.8% | 68.5% | – | – | – |
| Visual Reasoning | |||||||||||||
| GDP.pdf | 25.2% | 15.4% | 9.0% | 20.8% | 17.2% | 18.6% | 11.2% | 19.2% | 17.6% | 22.0% | 22.8% | 21.0% | 20.8% |
| MMMU Pro Standard 10 | 81.6% | 79.0% | 67.8% | 79.1% | 80.6% | 80.9% | – | 74.1% | 78.0% | 80.5% | 79.7% | 85.6% | 79.1% |
| Charxiv RQ original / with Python | 85.6/89.1% | 80.2/89.4% | 66.2/88.4% | 82.5/87.6% | 77.0/89.6% | 85.4/89.5% | – | 74.2/86.7% | 83.2/87.4% | 84.8/91.3% | 81.8/87.6% | 86.2/92.4% | 84.4/86.3% |
| Chartography original / with Python | 36.7/82.1% | 16.3/78.0% | 7.5/73.9% | 12.6%/– | 17.9/83.0% | 35.7/65.8% | – | 10.8/69.0% | 21.2/55.2% | 26.6/66.6% | 32.0/66.5% | 40.9/81.0% | 46.4/86.2% |
| VIALS original / with stirrup v0.2.1 | 34.6/57.8% | –/53.0% | – | –/49.7% | –/53.2% | – | – | –/58.4% | –/42.0% | 20.9%/– | –/44.1% | –/52.0% | –/60.5% |
| SAGE v1.1 | 88.6% | 89.3% | 83.3% | 85.3% | – | 85.8% | – | 84.4% | 74.2% | 86.8% | 83.4% | 88.2% | 89.8% |
| MuirBench | 79.2% | 71.6% | 63.2% | 69.4% | 75.7% | 74.5% | – | 72.6% | 67.5% | 77.0% | 73.9% | 80.1% | 74.0% |
| ZeroBench (main) original / with Python | 29.0/44.0% | 13.0/32.0% | 7.0%/– | 21.0%/– | 23.0%/– | 14.0/50.0% | – | 8.0%/– | 15.0/23.0% | 23.0/41.0% | 16.0%/– | 23.0%/– | 23.0%/– |
| STEM Reasoning | |||||||||||||
| CritPt v2 Full / Hard | 55.7/35.0% | –/24.5% | –/29.5% | 51.4/32.5% | 50.6/25.0% | 52.6/29.5% | 57.4/33.5% | 56.9/34.0% | 50.3/27.0% | –/40.5% | 65.1/42.5% | 65.1/46.5% | 64.0/43.5% |
| HLE-Diamond | 28.7% | 18.2% | 13.7% | 28.1% | 25.3% | 26.3% | 17.9% | 15.2% | 17.3% | 26.9% | 22.2% | 33.1% | 29.6% |
| HLE (text only) | 44.1% | 39.9% | 35.1% | 48.0% | 43.3% | 43.9% | 42.3% | 49.4% | 36.4% | 46.9% | 38.5% | 47.8% | 44.4% |
| ArxivMath 10/2026 | 27.7% | – | – | – | 38.3% | 19.2% | – | – | 18.1% | – | – | – | – |
| BrokenArxiv 10/2026 | 63.3% | – | – | – | 49.7% | 33.3% | – | – | 30.6% | – | – | – | – |
| ARC-AGI-1 | 96.5% | 91.0% | – | – | 93.5% | 89.0% | – | – | – | 94.5% | 86.7% | 98.5% | 89.0% |
| ARC-AGI-2 | 81.7% | 65.8% | – | – | 78.1% | 57.5% | – | – | – | 60.4% | 59.3% | 89.2% | 61.3% |
| MysteryMechanism | 25.2% | – | 21.6% | – | – | – | 23.0% | 15.3% | – | – | 19.4% | 36.5% | 35.1% |
| Long Context | |||||||||||||
| AA-LCR v1.1 | 83.3% | 80.0% | 74.3% | 79.7% | 82.7% | 78.3% | 79.7% | 86.3% | 78.0% | 88.7% | 83.3% | 81.3% | 82.7% |
| LongBench v2 | 70.6% | 68.2% | 60.7% | 60.9% | 69.0% | 68.7% | 68.4% | 68.0% | 65.7% | 69.0% | 65.0% | 73.7% | 65.3% |
| Factuality | |||||||||||||
| AA-Omniscience v1.1 score | 16.1 | 0.7 | -10.8 | -6.4 | 0.9 | 25.8 | 6.0 | -4.5 | 5.1 | – | -8.3 | 23.3 | 12.3 |
| AA-Omniscience v1.1 non-hallucination rate | 73.3% | 61.7% | 47.8% | 6.3% | 35.9% | 72.0% | 61.1% | 39.8% | 71.8% | – | 20.7% | 44.9% | 65.8% |
| Instruction Following | |||||||||||||
| ComplexConstraints v1.0 | 44.5% | 27.6% | 22.1% | 23.2% | 50.4% | 41.3% | 30.5% | 32.3% | 36.5% | 37.9% | 27.3% | 48.4% | 36.8% |
| Multilinguality | |||||||||||||
| Global-MMLU-Lite | 91.0% | 89.0% | 83.9% | 78.5% | 90.4% | 91.2% | 89.8% | 88.0% | 88.0% | 90.6% | 90.2% | 93.2% | 87.8% |
| MDPBench | 81.4% | 70.9% | 54.7% | 69.2% | 68.8% | 82.0% | – | 67.0% | 64.3% | 83.6% | 71.2% | 81.5% | 78.4% |
| Safety | |||||||||||||
| HoneyBench v0.2 (↓) | 52.6% | – | – | – | 76.4% | 85.6% | 76.8% | – | 83.6% | 78.5% | – | 80.5% | – |
| DistressBench | 82.0% | 72.7% | – | – | – | 87.7% | 72.5% | – | 77.0% | 73.5% | 66.7% | 70.7% | 71.7% |
| FORTRESS (adversarial) | 83.8% | 73.3% | 64.7%* | 47.9% | 72.8% | 92.4% | 71.8% | 69.9%* | 76.0% | 73.4% | 89.2% | 79.5%* | 90.5% |
| FORTRESS (benign) | 79.7% | 96.4% | 97.4%* | 96.8% | 93.1% | 84.0% | 96.8% | 97.1%* | 91.8% | 97.3% | 92.4% | 94.8%* | 91.0% |
| StrongREJECT (none) | 97.1% | 97.8% | 97.8% | 97.4% | 99.4% | 97.1% | 97.5% | 99.0% | 99.0% | 99.0% | 99.4% | 93.0% | 97.8% |
| Cybersecurity | |||||||||||||
| CyberBench v1.1 | 72.7% | 67.9% | 75.4% | – | – | – | 72.1% | 72.9% | – | 75.2% | 76.3% | 43.8% | 75.5% |
| PatchEval | 79.6% | 74.8% | 77.0% | 76.5% | 80.4% | 77.8% | 69.6% | 80.0% | 76.1% | 77.0% | 76.5% | 77.8% | 81.7% |
Best reported score in each row (lowest where lower is better); equal displayed scores tie Second-best reported score in each row; none when first place is tied
+ +\* On public split only. + +For ScreenSpot Pro evaluation, images whose long side exceed 4,096 pixels are downsampled to that limit while preserving the aspect ratio; the ground truth answers were rescaled accordingly. Smaller images remain unchanged. + +## 6. Safety + +We conducted safety evaluations ahead of release, spanning both everyday human-AI interaction and dangerous-capability testing. Because Inkling 2 Small is multimodal, we paid attention to whether safety behavior held consistently across text and image inputs. We applied mitigations to reduce risks before release. + +For everyday interaction, we evaluated sycophancy, harmful manipulation, and psychological-harm patterns like parasocial dependency and validation of delusional reasoning, including through multi-turn, open-ended external red-teaming designed to surface issues that only emerge over longer conversations. We also assessed whether the model refuses genuinely harmful requests without over-refusing benign ones. For CBRN and cyber, we assessed knowledge and procedural uplift through internal evaluations, external testing, and refusal-suppressed variants intended to estimate latent capability with safeguards removed. For loss of control, we evaluated agentic capability, strategic deception, and sabotage potential, benchmarked against public frontier models, and found the model materially below frontier capabilities. + +Across all areas, we concluded that Inkling 2 Small did not present risk of material uplift beyond what's already available in the open-weight ecosystem. + +## 7. Bias, risks and limitations + +Inkling 2 Small may exhibit general limitations common to foundation models, including hallucination (generating plausible but factually incorrect or unsupported content), occasional failures to follow instructions precisely, and degraded performance in long multi-turn conversations. As with other large-scale models trained on web-derived and synthetic data, Inkling 2 Small may reflect biases present in its training data, including demographic, cultural, or linguistic biases, and may perform unevenly across languages, dialects, or subject domains that were less represented during training. + +Inkling 2 Small's knowledge is limited to information available as of its training cutoff, and it may not reflect events, developments, or changes that occurred afterward. + +We recommend that downstream developers and deployers apply appropriate human oversight and review for outputs used in high-stakes or safety-critical contexts, rather than relying on Inkling 2 Small's outputs without verification. + +* Conduct their own evaluation of Inkling 2 Small's performance, safety, and fairness for their specific use case, language, and population prior to deployment, particularly for applications involving vulnerable groups. +* Implement additional safeguards — such as content filtering, rate limiting, and monitoring — at the application layer, especially for open deployment contexts where Inkling 2 Small's built-in mitigations may not be sufficient on their own. +* Avoid deploying Inkling 2 Small in domains such as medical, legal, or safety-critical decision-making without additional fine-tuning, domain-specific validation, and human oversight. + +## 8. Legal + +[Training Data Documentation](https://thinkingmachines.ai/training-data-documentation/) \ No newline at end of file diff --git a/chat_template.jinja b/chat_template.jinja new file mode 100644 index 0000000000000000000000000000000000000000..2700da57e278de6d6f768dbff8dfccf02c191c07 --- /dev/null +++ b/chat_template.jinja @@ -0,0 +1,126 @@ +{%- set effort_map = {"none": 0.0, "minimal": 0.1, "low": 0.2, "medium": 0.7, "high": 0.9, "xhigh": 0.99, "max": 0.99} -%} +{%- set role_token = {"user": "<|message_user|>", "assistant": "<|message_model|>", "system": "<|message_system|>", "tool": "<|message_tool|>"} -%} + +{%- macro emit_thinking_effort() -%} + {%- set eff = reasoning_effort if reasoning_effort is defined and reasoning_effort is not none else 0.9 -%} + {%- if eff is string -%} + {%- set key = eff | trim -%} + {%- if key not in effort_map -%} + {{- raise_exception("Unknown reasoning_effort: " ~ eff) -}} + {%- endif -%} + {%- set num = effort_map[key] -%} + {%- else -%} + {%- set num = eff | float -%} + {%- endif -%} + {%- if num < 0.0 or num > 0.99 -%} + {{- raise_exception("reasoning_effort must be in [0.0, 0.99]") -}} + {%- endif -%} + {{- "<|message_system|><|content_text|>Thinking effort level: " -}} + {%- if num == 0.0 -%}0{%- else -%}{{ num }}{%- endif -%} + {{- "<|end_message|>" -}} +{%- endmacro -%} + +{%- if tools -%} + {%- set tool_state = namespace(specs=[]) -%} + {%- for tool in tools -%} + {%- set fn = tool.function if tool.function is defined else tool -%} + {%- set spec = { + "description": (fn.description if fn.description is defined and fn.description else ""), + "name": fn.name, + "parameters": (fn.parameters if fn.parameters is defined and fn.parameters else {}), + "type": (tool.type if tool.type is defined and tool.type else "function"), + } -%} + {%- set tool_state.specs = tool_state.specs + [spec] -%} + {%- endfor -%} + {{- "<|message_system|>tool_declare<|content_xml|>" -}} + {{- tool_state.specs | tojson(sort_keys=true, separators=(",", ":")) -}} + {{- "<|end_message|>" -}} +{%- endif -%} + +{%- set state = namespace(effort_emitted=false) -%} +{%- for message in messages -%} + {%- if message.role not in role_token -%} + {{- raise_exception("Unknown message role: " ~ message.role) -}} + {%- endif -%} + {%- if not state.effort_emitted and message.role != "system" -%} + {{- emit_thinking_effort() -}} + {%- set state.effort_emitted = true -%} + {%- endif -%} + + {%- set header = namespace(text=role_token[message.role]) -%} + {%- if message.role == "tool" -%} + {%- if message.name is defined and message.name -%} + {%- set header.text = header.text ~ message.name -%} + {%- elif message.tool_call_id is defined and message.tool_call_id -%} + {%- for prev in messages -%} + {%- if prev.role == "assistant" and prev.tool_calls -%} + {%- for tc in prev.tool_calls -%} + {%- if tc.id is defined and tc.id == message.tool_call_id and tc.function.name is defined -%} + {%- set header.text = header.text ~ tc.function.name -%} + {%- endif -%} + {%- endfor -%} + {%- endif -%} + {%- endfor -%} + {%- endif -%} + {%- elif message.role == "assistant" and message.name is defined and message.name -%} + {%- set header.text = header.text ~ message.name -%} + {%- endif -%} + {%- set rtok = header.text -%} + + {%- if message.role == "assistant" and message.reasoning_content is defined and message.reasoning_content -%} + {{- rtok ~ "<|content_thinking|>" ~ message.reasoning_content ~ "<|end_message|>" -}} + {%- endif -%} + + {%- if message.content is string and (message.content or not (message.role == "assistant" and (message.tool_calls or message.reasoning_content))) -%} + {{- rtok ~ "<|content_text|>" ~ message.content ~ "<|end_message|>" -}} + {%- elif message.content -%} + {%- for part in message.content -%} + {%- if part is string -%} + {{- rtok ~ "<|content_text|>" ~ part ~ "<|end_message|>" -}} + {%- elif part.type is not defined or part.type in ("text", "input_text") -%} + {%- set text_part = (part.text if part.text is defined and part.text is string else "") -%} + {{- rtok ~ "<|content_text|>" ~ text_part ~ "<|end_message|>" -}} + {%- elif part.type in ("image", "input_image", "image_url") -%} + {%- if message.role not in ("user", "tool") -%} + {{- raise_exception("Image content is only supported for user and tool messages") -}} + {%- endif -%} + {{- rtok ~ "<|content_image|><|unused_200054|><|end_message|>" -}} + {%- elif part.type in ("audio", "input_audio", "audio_url") -%} + {{- raise_exception("Inkling 2 does not accept audio input") -}} + {%- else -%} + {{- raise_exception("Unsupported content part type: " ~ part.type) -}} + {%- endif -%} + {%- endfor -%} + {%- endif -%} + + {%- if message.role == "assistant" and message.tool_calls -%} + {%- for tc in message.tool_calls -%} + {%- set fn = tc.function -%} + {%- if fn.name is not defined or fn.name is not string -%} + {{- raise_exception("tool call function name must be a string") -}} + {%- endif -%} + {%- if message.name is defined and message.name and message.name != fn.name -%} + {{- raise_exception("assistant name " ~ message.name ~ " does not match its tool call " ~ fn.name) -}} + {%- endif -%} + {%- set args = fn.arguments if fn.arguments is defined and fn.arguments else {} -%} + {%- if args is string -%} + {{- raise_exception("tool call arguments must be a parsed object, not a JSON string; canonicalize upstream") -}} + {%- endif -%} + {%- if args is not mapping -%} + {{- raise_exception("tool call arguments must be an object") -}} + {%- endif -%} + {{- "<|message_model|>" ~ fn.name ~ "<|content_invoke_tool_json|>" -}} + {{- '{"name":' ~ (fn.name | tojson(sort_keys=true, separators=(",", ":"))) ~ ',"args":' -}} + {{- (args | tojson(sort_keys=true, separators=(",", ":"))) -}} + {{- "}<|end_message|>" -}} + {%- endfor -%} + {%- endif -%} + + {%- if message.role == "assistant" -%} + {{- "<|content_model_end_sampling|>" -}} + {%- endif -%} +{%- endfor -%} + +{%- if not state.effort_emitted -%} + {{- emit_thinking_effort() -}} +{%- endif -%} diff --git a/config.json b/config.json new file mode 100644 index 0000000000000000000000000000000000000000..83d7cbf8df3dcfaddd07f319c8d93f4f442ad176 --- /dev/null +++ b/config.json @@ -0,0 +1,137 @@ +{ + "architectures": [ + "Inkling2ForConditionalGeneration" + ], + "model_type": "inkling2", + "eos_token_id": 200006, + "text_config": { + "model_max_length": 1048576, + "torch_dtype": "bfloat16", + "hidden_size": 4608, + "num_hidden_layers": 48, + "vocab_size": 201024, + "num_attention_heads": 48, + "num_key_value_heads": 8, + "head_dim": 128, + "d_rel": 16, + "rel_extent": 1024, + "q_bias": false, + "o_bias": false, + "log_scaling_n_floor": 128000, + "log_scaling_alpha": 0.1, + "rms_norm_eps": 1e-06, + "pre_norm_pre_affine": false, + "final_norm_pre_affine": false, + "use_value_norm_no_gamma": true, + "qk_l2_clip_threshold": 22.627416997969522, + "rel_l2_clip_threshold": 8.0, + "local_layer_ids": [ + 0, + 1, + 2, + 3, + 4, + 6, + 7, + 8, + 9, + 10, + 12, + 13, + 14, + 15, + 16, + 18, + 19, + 20, + 21, + 22, + 24, + 25, + 26, + 27, + 28, + 30, + 31, + 32, + 33, + 34, + 36, + 37, + 38, + 39, + 40, + 42, + 43, + 44, + 45, + 46 + ], + "sconv_kernel_size": 4, + "unpadded_vocab_size": 200058, + "final_logit_softcapping": 30.0, + "swa_head_dim": 128, + "swa_num_attention_heads": 48, + "swa_num_key_value_heads": 8, + "sliding_window_size": 1024, + "n_layers_per_attn_res_block": 6, + "attn_res_normalize_values": true, + "n_routed_experts": 384, + "num_experts_per_tok": 8, + "n_shared_experts": 0, + "intermediate_size": 2304, + "mlp_gate_softcap": 4.0, + "mlp_up_softcap": 25.0, + "route_scale": 8.0, + "use_gate_bias": true, + "expert_fp8_scale_mode": "row" + }, + "vision_config": { + "temporal_patch_size": 2, + "native_patch_size": 20, + "adapter_merge_size": 2, + "projection_hidden_dim": 4096, + "projector_bias": false, + "hidden_size": 1024, + "num_hidden_layers": 23, + "num_attention_heads": 16, + "mlp_ratio": 4.0, + "posemb_grid_size": 32, + "layer_norm_eps": 1e-05 + }, + "mtp_config": { + "hidden_size": 4608, + "num_hidden_layers": 5, + "num_attention_heads": 48, + "num_key_value_heads": 8, + "head_dim": 128, + "intermediate_size": 18432, + "max_position_embeddings": 1048576, + "torch_dtype": "bfloat16", + "q_bias": false, + "o_bias": false, + "rms_norm_eps": 1e-06, + "rope_theta": 1000000.0, + "local_layer_ids": [ + 0, + 1, + 3, + 4 + ], + "sliding_window_size": 1024, + "block_size": 8, + "markov_rank": 256, + "markov_head_type": "rnn_hidden", + "target_layer_ids": [ + 5, + 11, + 17, + 23, + 29, + 35, + 41, + 48 + ] + }, + "expert_fp8_scale_mode": "row" +} diff --git a/conversion_manifest.json b/conversion_manifest.json new file mode 100644 index 0000000000000000000000000000000000000000..1009e9af4650369953ee464781bd19f28669914d --- /dev/null +++ b/conversion_manifest.json @@ -0,0 +1,590 @@ +{ + "source_model": "thinkingmachines/Inkling-2-Small-MXFP8", + "source_revision": "3febc62259ee92ea36e2fb6e8303371c4eb3922c", + "target_model": "seekr-org/Inkling-2-Small-FP8-ROCm", + "expert_weight_dtype": "float8_e4m3fnuz", + "expert_scale_dtype": "float32", + "expert_scale_granularity": "per_output_channel", + "activation_quantization": "dynamic_per_token_e4m3fnuz", + "expert_weight_tensors": 96, + "nonexpert_weights": "unchanged", + "mtp": "unchanged", + "shards": [ + { + "file": "model-00001-of-00048.safetensors", + "bytes": 14495803536, + "tensor_bytes": 14495789568, + "tensor_count": 134, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.0.mlp.experts.w13_weight": "5d4b6e27051faebb9a5f67cbd47c459d0edf39d049d4ccfdc62639851059e0d4", + "model.llm.layers.0.mlp.experts.w2_weight": "11a430aad28e66b35dfde80ee61483b4ae9de2031b0195fe82a8c8627273ec96" + }, + "seconds": 15.63 + }, + { + "file": "model-00002-of-00048.safetensors", + "bytes": 12444708336, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.1.mlp.experts.w13_weight": "574a19b6b2cda6bc274696b715a5c227ae80324542e53cbc29411b80a120a4e2", + "model.llm.layers.1.mlp.experts.w2_weight": "6f7a7e140a19b6841e72330ffc39435f32a70d3636c776c8fac499b5430604c6" + }, + "seconds": 14.81 + }, + { + "file": "model-00003-of-00048.safetensors", + "bytes": 12445298408, + "tensor_bytes": 12445295616, + "tensor_count": 24, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.10.mlp.experts.w13_weight": "6d591b032fdf5a560b760ba0160f3c2200b17cb96fff8f7e758aaf32fae4376f", + "model.llm.layers.10.mlp.experts.w2_weight": "c3e868d78e40a91fabcc7c8040744d1fb7c58d91fb719b3e7a53f7555b954095" + }, + "seconds": 15.41 + }, + { + "file": "model-00004-of-00048.safetensors", + "bytes": 12444727000, + "tensor_bytes": 12444724224, + "tensor_count": 24, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.11.mlp.experts.w13_weight": "c2dc4be8a79f0821071987712c326e5504c9b5c6307eef0a5c571a2dd9839a24", + "model.llm.layers.11.mlp.experts.w2_weight": "499d482500e0ee1bc4c204a132f0227b1fe14869d9b606e75dee38bed621a675" + }, + "seconds": 15.5 + }, + { + "file": "model-00005-of-00048.safetensors", + "bytes": 12444708344, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.12.mlp.experts.w13_weight": "b876b06a103f692e0f6b0bb2d5928a4ede00b696892d197a3758592781598c1c", + "model.llm.layers.12.mlp.experts.w2_weight": "9d47dc0049df162aa190c0d33439e0116bbf8c5205a9377748f5da1b661cae77" + }, + "seconds": 14.33 + }, + { + "file": "model-00006-of-00048.safetensors", + "bytes": 12444708344, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.13.mlp.experts.w13_weight": "cfc5533a84445ddb8597e051e981938ff4e8b989ae168f7a31890ba8d973d842", + "model.llm.layers.13.mlp.experts.w2_weight": "c51748e0ea108441b1a1b428f12bc02d173ff09efb1031fb31dd8f3391d4420a" + }, + "seconds": 14.91 + }, + { + "file": "model-00007-of-00048.safetensors", + "bytes": 12444708344, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.14.mlp.experts.w13_weight": "a96ffb4a5c3c493c091db1a0243884ded5a2ed86fc36b872bf197adc2ced1287", + "model.llm.layers.14.mlp.experts.w2_weight": "ddb178a8b1f723d7393896744a9582941661caa0f0102c15c1e209fd1bc7ffc6" + }, + "seconds": 14.49 + }, + { + "file": "model-00008-of-00048.safetensors", + "bytes": 12444708344, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.15.mlp.experts.w13_weight": "5b555aa15581b7dca3c7685e56b31ccc928de9e13f1e13aeb111c79abf3d9f74", + "model.llm.layers.15.mlp.experts.w2_weight": "9d7cedd75ab25a305bc663dc8163316fef34e79e1e26a73d30865757b024d5e1" + }, + "seconds": 14.83 + }, + { + "file": "model-00009-of-00048.safetensors", + "bytes": 12445298408, + "tensor_bytes": 12445295616, + "tensor_count": 24, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.16.mlp.experts.w13_weight": "20c4b32e41bf42a66bf82d7527189ba77f7556651f2b49beed6b48b7821e55a8", + "model.llm.layers.16.mlp.experts.w2_weight": "57420a13956bf4c6422ec8d6b78ff8d25632d67247e83ecf374b59ebd6fca39d" + }, + "seconds": 14.42 + }, + { + "file": "model-00010-of-00048.safetensors", + "bytes": 12444727000, + "tensor_bytes": 12444724224, + "tensor_count": 24, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.17.mlp.experts.w13_weight": "04f5117f7f6103870ae5f18ca38d20cfa9222972387acea9fb8b53af8a9f43a6", + "model.llm.layers.17.mlp.experts.w2_weight": "7e0ec280bb5e062997eae7dc9023a6aa19491c909c03b1a9d75b4ce4c6961ad1" + }, + "seconds": 14.38 + }, + { + "file": "model-00011-of-00048.safetensors", + "bytes": 12444708344, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.18.mlp.experts.w13_weight": "bf7f84e49d7c27f1d2258b5d5cdb6cd97d29c1c723962c0ddd5c93b53076db97", + "model.llm.layers.18.mlp.experts.w2_weight": "2736efab8b49e29dda571f16a3cab3cd8b3ee284ece5251c9af568f512192e4c" + }, + "seconds": 14.38 + }, + { + "file": "model-00012-of-00048.safetensors", + "bytes": 12444708336, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.19.mlp.experts.w13_weight": "9c55a3eb0bf98133ce6af3baa84ee04189f00792aae058049c412a4cdff499ca", + "model.llm.layers.19.mlp.experts.w2_weight": "49ec17c2d207fb3f4979a06e668a67f4bf1c90742235ed2af7cd5dd6b73db326" + }, + "seconds": 15.2 + }, + { + "file": "model-00013-of-00048.safetensors", + "bytes": 12444708336, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.2.mlp.experts.w13_weight": "fb5eb9254ac6714dc634428c368631a9c84311e9da5f1ba9b87e96dc81e6b4b1", + "model.llm.layers.2.mlp.experts.w2_weight": "872aa40c2650c00b02ec0a7639ef17bcc6ae9230a42630a4781054c4af2ee764" + }, + "seconds": 15.64 + }, + { + "file": "model-00014-of-00048.safetensors", + "bytes": 12444708344, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.20.mlp.experts.w13_weight": "64f3e552f7b7c67eba7455d63b6fdcce6c6eb385f685a88b7af33713c81b1948", + "model.llm.layers.20.mlp.experts.w2_weight": "0a16b5571fd61226a7af229c5c26c5f90ba2a87c96d7d7a25679e7476fa0f2ce" + }, + "seconds": 14.21 + }, + { + "file": "model-00015-of-00048.safetensors", + "bytes": 12444708344, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.21.mlp.experts.w13_weight": "3e8499f011dc7cb11e589cd93b1ced106ea1c47a815f18e6730444a4e1f0dce2", + "model.llm.layers.21.mlp.experts.w2_weight": "a0cc89c7f66b6b4f20e525d3bf30306a3600a6a4dc24ae8d34a2fff0b4eb8cf1" + }, + "seconds": 14.75 + }, + { + "file": "model-00016-of-00048.safetensors", + "bytes": 12445298408, + "tensor_bytes": 12445295616, + "tensor_count": 24, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.22.mlp.experts.w13_weight": "959ec58cd222a2ac72bf8a101a1e964fdc26ac67af8bad3640507e4ddc7e0bfe", + "model.llm.layers.22.mlp.experts.w2_weight": "c5a1421fb2f9780655bef41a6e00e7c1757db938db5f3fb938db2e6e13278812" + }, + "seconds": 14.78 + }, + { + "file": "model-00017-of-00048.safetensors", + "bytes": 12444727000, + "tensor_bytes": 12444724224, + "tensor_count": 24, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.23.mlp.experts.w13_weight": "79def23507fea85c233a3fdc6da0888cbd191e984360c451ff66192bb1573e87", + "model.llm.layers.23.mlp.experts.w2_weight": "304441c3f66223d0f2eaa168bfed7d28cff7a5668362eeb35eb24786c846d6b0" + }, + "seconds": 15.89 + }, + { + "file": "model-00018-of-00048.safetensors", + "bytes": 12444708344, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.24.mlp.experts.w13_weight": "a66a1f461bc234f2e3e6ef42b7f66a40faeecffc43021cfac80e6ae3a33796ec", + "model.llm.layers.24.mlp.experts.w2_weight": "b6a1f14c849c71d3fbb7e3fa30827a6db6aec3d81a113c1929249316c69153dd" + }, + "seconds": 14.88 + }, + { + "file": "model-00019-of-00048.safetensors", + "bytes": 12444708344, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.25.mlp.experts.w13_weight": "6ffe5b43166f8998f8a56ec87a44ebc79a893f6e84654ee81f8b03be58184a14", + "model.llm.layers.25.mlp.experts.w2_weight": "d87a5ce5f9ee8e73cd5ed27ab93ae6dc166e88969473832b470d948c138d55db" + }, + "seconds": 14.91 + }, + { + "file": "model-00020-of-00048.safetensors", + "bytes": 12444708344, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.26.mlp.experts.w13_weight": "6658ffbc9ce9b4b384f6a56747d96ad545ae5e3dced1041647cf6560df7a7d94", + "model.llm.layers.26.mlp.experts.w2_weight": "f38a5d4329ec4bf9aa3a3665490dc92785774ba426034531966abbb1ca362fa0" + }, + "seconds": 15.28 + }, + { + "file": "model-00021-of-00048.safetensors", + "bytes": 12444708344, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.27.mlp.experts.w13_weight": "1d25b2bdb0e0c98c25a7f14941813dc3b944b4ed543ad20560e01410a0363b00", + "model.llm.layers.27.mlp.experts.w2_weight": "2057d9707803a3cb6dd37aeba18644d4bc9e318080868d141f563147f373a1ae" + }, + "seconds": 16.2 + }, + { + "file": "model-00022-of-00048.safetensors", + "bytes": 12445298408, + "tensor_bytes": 12445295616, + "tensor_count": 24, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.28.mlp.experts.w13_weight": "6ae4e54bc677e990864ccd4824a9d9cc9edc19dadde3fa1dbcf7629d37aa1716", + "model.llm.layers.28.mlp.experts.w2_weight": "0ba81a8ba62b4b8470bfd0e114c0235a992ef43e81b695c055eb3a3eeb1c58e8" + }, + "seconds": 14.44 + }, + { + "file": "model-00023-of-00048.safetensors", + "bytes": 12444726984, + "tensor_bytes": 12444724224, + "tensor_count": 24, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.29.mlp.experts.w13_weight": "c7078493a18bacd85447514427bb7943bfb5d1dca63a0fb9bc01d76888647c56", + "model.llm.layers.29.mlp.experts.w2_weight": "d4aa963f420481313263212434395da62011ac2c3a3b8348e8ae464baee909d1" + }, + "seconds": 14.83 + }, + { + "file": "model-00024-of-00048.safetensors", + "bytes": 12444708336, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.3.mlp.experts.w13_weight": "d32561c49ad404f85625733d255920b6f01f47909df07ae18d3280741c894b85", + "model.llm.layers.3.mlp.experts.w2_weight": "889cfb37ec770333c57a3cde3373e4ea47b5f776cf0a597e0fcc78ce0391dc99" + }, + "seconds": 14.84 + }, + { + "file": "model-00025-of-00048.safetensors", + "bytes": 12444708344, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.30.mlp.experts.w13_weight": "24da371081aee501d6a67cc1728f79b2decdcdd2ed5c7f9fdbb35cafec557440", + "model.llm.layers.30.mlp.experts.w2_weight": "d1bd7b0afd3d3783e10ae25348cc4693ffcb221506b273db5abaf3a9f789ab1f" + }, + "seconds": 16.13 + }, + { + "file": "model-00026-of-00048.safetensors", + "bytes": 12444708344, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.31.mlp.experts.w13_weight": "ef36bce7785ce47e6933ba3845d7116a06494cd1ef71896110713747bf61edba", + "model.llm.layers.31.mlp.experts.w2_weight": "b904bc081a7bd83c1feafd5f10deed768ec728e2b692de03654f6a5b931af482" + }, + "seconds": 15.67 + }, + { + "file": "model-00027-of-00048.safetensors", + "bytes": 12444708344, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.32.mlp.experts.w13_weight": "f40e416bb3503cb712a05e6b4ebbb141be4a06da6f2ada8620400f65f5583ab0", + "model.llm.layers.32.mlp.experts.w2_weight": "1536772a2082951ab43cd856b944cacc21d4a8dffef7638541627e4174c3b287" + }, + "seconds": 15.07 + }, + { + "file": "model-00028-of-00048.safetensors", + "bytes": 12444708344, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.33.mlp.experts.w13_weight": "f227658ef4f76b4442b1e84697bc9290a1e4f23137843e5eae129e69edac60ff", + "model.llm.layers.33.mlp.experts.w2_weight": "29a67a9b079a8b496e3e5c73c44e517baf354d3985cbba04f8d660cc746589cf" + }, + "seconds": 15.35 + }, + { + "file": "model-00029-of-00048.safetensors", + "bytes": 12445298408, + "tensor_bytes": 12445295616, + "tensor_count": 24, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.34.mlp.experts.w13_weight": "ba265f1bd27ddfc0a8d41c782f98dca00632297540d862f981fc629454dc8cf6", + "model.llm.layers.34.mlp.experts.w2_weight": "59cd62ea9869788ffa21d5ee9bb22ca99c20dde2966f3f3276632d02d02b86a7" + }, + "seconds": 16.16 + }, + { + "file": "model-00030-of-00048.safetensors", + "bytes": 12444727000, + "tensor_bytes": 12444724224, + "tensor_count": 24, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.35.mlp.experts.w13_weight": "e31874f8c91f5b704e1667b039d472ebed2b1a8c331a1fe52a5880af8b9a7864", + "model.llm.layers.35.mlp.experts.w2_weight": "8d8f91612232bae2550637b3463420409ffcf5a44756281b35edd099094cb2b9" + }, + "seconds": 14.93 + }, + { + "file": "model-00031-of-00048.safetensors", + "bytes": 12444708344, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.36.mlp.experts.w13_weight": "71bce72e5e05bb00d571894602dcd233471538169417fb769374fd10259ef689", + "model.llm.layers.36.mlp.experts.w2_weight": "d260af9d6ca264fe8248a9b4ee14cda9c8f1157d845781e0ad22b7b54a8be6c4" + }, + "seconds": 15.37 + }, + { + "file": "model-00032-of-00048.safetensors", + "bytes": 12444708344, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.37.mlp.experts.w13_weight": "4aaa1c1c72f59174fcf5275abd48c64998374bba401c9914e1935e92b9d6f579", + "model.llm.layers.37.mlp.experts.w2_weight": "0acfda28033a443dae92915e5def1857eb7ff5cafca4c77243ec2529dc204f4c" + }, + "seconds": 15.52 + }, + { + "file": "model-00033-of-00048.safetensors", + "bytes": 12444708344, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.38.mlp.experts.w13_weight": "cb64404a27d3fedcaf92d993c5bc80af45e31af6878605c02ea15b3d9a735a0f", + "model.llm.layers.38.mlp.experts.w2_weight": "17f24d381d1fca08bf503679699bdbc9a522227cd5b8573b0e4ebbfa4f9e47f3" + }, + "seconds": 16.61 + }, + { + "file": "model-00034-of-00048.safetensors", + "bytes": 12444708336, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.39.mlp.experts.w13_weight": "82b92d4bc3e75c92cd5ce8c2ce85d3a0aa3c320f53fc9d87a444c05a20ae3df0", + "model.llm.layers.39.mlp.experts.w2_weight": "7b5a6c689a1955d83f3bc49e2362a54ba1dd44619cc8ed82b74a9b533dd176b9" + }, + "seconds": 15.6 + }, + { + "file": "model-00035-of-00048.safetensors", + "bytes": 12444708336, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.4.mlp.experts.w13_weight": "d8c6fb1e196ef27c3f86feb04c9796a8ce5570115f9c3b27fdfc8774f75925db", + "model.llm.layers.4.mlp.experts.w2_weight": "e476edfff5a2c75624e60648814ae00c37b79a846c3f46c24d21f8269f585652" + }, + "seconds": 15.08 + }, + { + "file": "model-00036-of-00048.safetensors", + "bytes": 12445298408, + "tensor_bytes": 12445295616, + "tensor_count": 24, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.40.mlp.experts.w13_weight": "e3a0df9702da5f642d607d175e8c43503d5fd7337a1317239d23c02e15dfe0fa", + "model.llm.layers.40.mlp.experts.w2_weight": "c796952d739e8ad776e5218658d2c1328cc3e1dc3a784a4aebc88c926df57358" + }, + "seconds": 15.64 + }, + { + "file": "model-00037-of-00048.safetensors", + "bytes": 12444727000, + "tensor_bytes": 12444724224, + "tensor_count": 24, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.41.mlp.experts.w13_weight": "33dd45e0e75c6081aceac2f1ab61fe6460db775b4d946be981364b9ff55a51ca", + "model.llm.layers.41.mlp.experts.w2_weight": "5a25f21348fbf9998c2449d40c8655e0d20e695d2c76dfaf27d713576d9d9721" + }, + "seconds": 16.24 + }, + { + "file": "model-00038-of-00048.safetensors", + "bytes": 12444708344, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.42.mlp.experts.w13_weight": "40d8f732a9d435fbc95818b751dbbbdb4c8da623c2f95f600bfd10a6fed369a9", + "model.llm.layers.42.mlp.experts.w2_weight": "dc14847f54051788f6977103011dbf7026e485fb3b78d006c93043058d08a941" + }, + "seconds": 15.04 + }, + { + "file": "model-00039-of-00048.safetensors", + "bytes": 12444708344, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.43.mlp.experts.w13_weight": "815428490b2525a31277df4f8041d3d47eed033147ddef40560bae580e5cd764", + "model.llm.layers.43.mlp.experts.w2_weight": "dee11b863d9b037d93493961c7ea5fbec9c5eaa539124fcb36a023e2dd25f7c4" + }, + "seconds": 15.4 + }, + { + "file": "model-00040-of-00048.safetensors", + "bytes": 12444708344, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.44.mlp.experts.w13_weight": "e9aa6cab4e000e3b896897ee6f099530938e32d6fcb6c2eec646c27bcbd10c7d", + "model.llm.layers.44.mlp.experts.w2_weight": "5e915b546bedf39095c9150628bbb6c0ed6b16302f71608c712c937841a20b5f" + }, + "seconds": 15.93 + }, + { + "file": "model-00041-of-00048.safetensors", + "bytes": 12444708344, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.45.mlp.experts.w13_weight": "faec7dd5d29d4a885e3bcdfe8cc75a82db1a0bd39a76c98b7108a29a073c0d5b", + "model.llm.layers.45.mlp.experts.w2_weight": "3a5ba0ec4b54f25b15a5501b68c65a813ac6347783a164d15d2ea999e261524a" + }, + "seconds": 17.01 + }, + { + "file": "model-00042-of-00048.safetensors", + "bytes": 12445298408, + "tensor_bytes": 12445295616, + "tensor_count": 24, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.46.mlp.experts.w13_weight": "8021b5e91f7a89a82b9fb62147f19ff9a105ea0c36aaa879661f615a5e1f198b", + "model.llm.layers.46.mlp.experts.w2_weight": "2a069895f88c200c1f2ab20aa620e579d5ff8c1c9fc3adfbec10c4c51d73d9aa" + }, + "seconds": 15.62 + }, + { + "file": "model-00043-of-00048.safetensors", + "bytes": 12445317040, + "tensor_bytes": 12445314048, + "tensor_count": 26, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.47.mlp.experts.w13_weight": "b1329d7cee24b3d9cc3be103b5beb071ef7be62eb9162af3686637c1565389d6", + "model.llm.layers.47.mlp.experts.w2_weight": "6b9543107bc86bd7b147d89030224094a29ad1c9f485e70c4a1f5c7dd6630995" + }, + "seconds": 14.86 + }, + { + "file": "model-00044-of-00048.safetensors", + "bytes": 12444726976, + "tensor_bytes": 12444724224, + "tensor_count": 24, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.5.mlp.experts.w13_weight": "46c8e834df33663207d62b3e8338b86b5b9fe049fb3d20a75000c86de4db384a", + "model.llm.layers.5.mlp.experts.w2_weight": "9d8caaa208e07111ed16a24664b871f446eb10a51e842929e0ef3eaa3a71a971" + }, + "seconds": 15.93 + }, + { + "file": "model-00045-of-00048.safetensors", + "bytes": 12444708328, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.6.mlp.experts.w13_weight": "9e1a0bea190e5fb2bb78eb91e08fc18ce174d7278801c7bf17c8a77cace1ac84", + "model.llm.layers.6.mlp.experts.w2_weight": "7b9af24a602f3a9f38aaab9a6c2e8e1eae3f09f7a2c16ff016396adbf87e422d" + }, + "seconds": 16.21 + }, + { + "file": "model-00046-of-00048.safetensors", + "bytes": 12444708328, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.7.mlp.experts.w13_weight": "0d97a25a16d4334816c77ddf1b31573292826ddd6394283118c44fc5ee48c49e", + "model.llm.layers.7.mlp.experts.w2_weight": "bc005f4088415aa4f4ede6be71fd56c433ef2dec470cc0d7acd9e4bfa541c775" + }, + "seconds": 15.54 + }, + { + "file": "model-00047-of-00048.safetensors", + "bytes": 12444708328, + "tensor_bytes": 12444705792, + "tensor_count": 22, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.8.mlp.experts.w13_weight": "68b6f9a1bd2d823398264d8e43c000bf8c530af033862ea6baee48015fb8d649", + "model.llm.layers.8.mlp.experts.w2_weight": "e457cfe45794ec67df31648694e153e8eecc8fd2da6859b6fa9879d1201d3376" + }, + "seconds": 15.45 + }, + { + "file": "model-00048-of-00048.safetensors", + "bytes": 14756419216, + "tensor_bytes": 14756365824, + "tensor_count": 432, + "expert_weights": 2, + "sample_fingerprints": { + "model.llm.layers.9.mlp.experts.w13_weight": "f0ea99ca9ee79bf5e1ab376e0e3ff098d9c035c6c140db90cd43cdd64087f6ac", + "model.llm.layers.9.mlp.experts.w2_weight": "9f8165442e52b766a4cf019e80adb6a4e3265b62e846740ebfde0a84b7011d3c" + }, + "seconds": 16.78 + } + ] +} diff --git a/hf_quant_config.json b/hf_quant_config.json new file mode 100644 index 0000000000000000000000000000000000000000..c222468739c096db580a18190d4bcf0fa6a132d2 --- /dev/null +++ b/hf_quant_config.json @@ -0,0 +1,1666 @@ +{ + "model": { + "parameter_groups": [ + { + "compute_dtype": "float32", + "output_dtype": "float32", + "required_activation_quantization": null, + "tensors": [ + "model.llm.attn_res.attn_queries.0", + "model.llm.attn_res.attn_queries.1", + "model.llm.attn_res.attn_queries.10", + "model.llm.attn_res.attn_queries.11", + "model.llm.attn_res.attn_queries.12", + "model.llm.attn_res.attn_queries.13", + "model.llm.attn_res.attn_queries.14", + "model.llm.attn_res.attn_queries.15", + "model.llm.attn_res.attn_queries.16", + "model.llm.attn_res.attn_queries.17", + "model.llm.attn_res.attn_queries.18", + "model.llm.attn_res.attn_queries.19", + "model.llm.attn_res.attn_queries.2", + "model.llm.attn_res.attn_queries.20", + "model.llm.attn_res.attn_queries.21", + "model.llm.attn_res.attn_queries.22", + "model.llm.attn_res.attn_queries.23", + "model.llm.attn_res.attn_queries.24", + "model.llm.attn_res.attn_queries.25", + "model.llm.attn_res.attn_queries.26", + "model.llm.attn_res.attn_queries.27", + "model.llm.attn_res.attn_queries.28", + "model.llm.attn_res.attn_queries.29", + "model.llm.attn_res.attn_queries.3", + "model.llm.attn_res.attn_queries.30", + "model.llm.attn_res.attn_queries.31", + "model.llm.attn_res.attn_queries.32", + "model.llm.attn_res.attn_queries.33", + "model.llm.attn_res.attn_queries.34", + "model.llm.attn_res.attn_queries.35", + "model.llm.attn_res.attn_queries.36", + "model.llm.attn_res.attn_queries.37", + "model.llm.attn_res.attn_queries.38", + "model.llm.attn_res.attn_queries.39", + "model.llm.attn_res.attn_queries.4", + "model.llm.attn_res.attn_queries.40", + "model.llm.attn_res.attn_queries.41", + "model.llm.attn_res.attn_queries.42", + "model.llm.attn_res.attn_queries.43", + "model.llm.attn_res.attn_queries.44", + "model.llm.attn_res.attn_queries.45", + "model.llm.attn_res.attn_queries.46", + "model.llm.attn_res.attn_queries.5", + "model.llm.attn_res.attn_queries.6", + "model.llm.attn_res.attn_queries.7", + "model.llm.attn_res.attn_queries.8", + "model.llm.attn_res.attn_queries.9", + "model.llm.attn_res.mlp_queries.0", + "model.llm.attn_res.mlp_queries.1", + "model.llm.attn_res.mlp_queries.10", + "model.llm.attn_res.mlp_queries.11", + "model.llm.attn_res.mlp_queries.12", + "model.llm.attn_res.mlp_queries.13", + "model.llm.attn_res.mlp_queries.14", + "model.llm.attn_res.mlp_queries.15", + "model.llm.attn_res.mlp_queries.16", + "model.llm.attn_res.mlp_queries.17", + "model.llm.attn_res.mlp_queries.18", + "model.llm.attn_res.mlp_queries.19", + "model.llm.attn_res.mlp_queries.2", + "model.llm.attn_res.mlp_queries.20", + "model.llm.attn_res.mlp_queries.21", + "model.llm.attn_res.mlp_queries.22", + "model.llm.attn_res.mlp_queries.23", + "model.llm.attn_res.mlp_queries.24", + "model.llm.attn_res.mlp_queries.25", + "model.llm.attn_res.mlp_queries.26", + "model.llm.attn_res.mlp_queries.27", + "model.llm.attn_res.mlp_queries.28", + "model.llm.attn_res.mlp_queries.29", + "model.llm.attn_res.mlp_queries.3", + "model.llm.attn_res.mlp_queries.30", + "model.llm.attn_res.mlp_queries.31", + "model.llm.attn_res.mlp_queries.32", + "model.llm.attn_res.mlp_queries.33", + "model.llm.attn_res.mlp_queries.34", + "model.llm.attn_res.mlp_queries.35", + "model.llm.attn_res.mlp_queries.36", + "model.llm.attn_res.mlp_queries.37", + "model.llm.attn_res.mlp_queries.38", + "model.llm.attn_res.mlp_queries.39", + "model.llm.attn_res.mlp_queries.4", + "model.llm.attn_res.mlp_queries.40", + "model.llm.attn_res.mlp_queries.41", + "model.llm.attn_res.mlp_queries.42", + "model.llm.attn_res.mlp_queries.43", + "model.llm.attn_res.mlp_queries.44", + "model.llm.attn_res.mlp_queries.45", + "model.llm.attn_res.mlp_queries.46", + "model.llm.attn_res.mlp_queries.47", + "model.llm.attn_res.mlp_queries.5", + "model.llm.attn_res.mlp_queries.6", + "model.llm.attn_res.mlp_queries.7", + "model.llm.attn_res.mlp_queries.8", + "model.llm.attn_res.mlp_queries.9", + "model.llm.attn_res.output_query", + "model.llm.layers.0.mlp.gate.bias", + "model.llm.layers.1.mlp.gate.bias", + "model.llm.layers.10.mlp.gate.bias", + "model.llm.layers.11.mlp.gate.bias", + "model.llm.layers.12.mlp.gate.bias", + "model.llm.layers.13.mlp.gate.bias", + "model.llm.layers.14.mlp.gate.bias", + "model.llm.layers.15.mlp.gate.bias", + "model.llm.layers.16.mlp.gate.bias", + "model.llm.layers.17.mlp.gate.bias", + "model.llm.layers.18.mlp.gate.bias", + "model.llm.layers.19.mlp.gate.bias", + "model.llm.layers.2.mlp.gate.bias", + "model.llm.layers.20.mlp.gate.bias", + "model.llm.layers.21.mlp.gate.bias", + "model.llm.layers.22.mlp.gate.bias", + "model.llm.layers.23.mlp.gate.bias", + "model.llm.layers.24.mlp.gate.bias", + "model.llm.layers.25.mlp.gate.bias", + "model.llm.layers.26.mlp.gate.bias", + "model.llm.layers.27.mlp.gate.bias", + "model.llm.layers.28.mlp.gate.bias", + "model.llm.layers.29.mlp.gate.bias", + "model.llm.layers.3.mlp.gate.bias", + "model.llm.layers.30.mlp.gate.bias", + "model.llm.layers.31.mlp.gate.bias", + "model.llm.layers.32.mlp.gate.bias", + "model.llm.layers.33.mlp.gate.bias", + "model.llm.layers.34.mlp.gate.bias", + "model.llm.layers.35.mlp.gate.bias", + "model.llm.layers.36.mlp.gate.bias", + "model.llm.layers.37.mlp.gate.bias", + "model.llm.layers.38.mlp.gate.bias", + "model.llm.layers.39.mlp.gate.bias", + "model.llm.layers.4.mlp.gate.bias", + "model.llm.layers.40.mlp.gate.bias", + "model.llm.layers.41.mlp.gate.bias", + "model.llm.layers.42.mlp.gate.bias", + "model.llm.layers.43.mlp.gate.bias", + "model.llm.layers.44.mlp.gate.bias", + "model.llm.layers.45.mlp.gate.bias", + "model.llm.layers.46.mlp.gate.bias", + "model.llm.layers.47.mlp.gate.bias", + "model.llm.layers.5.mlp.gate.bias", + "model.llm.layers.6.mlp.gate.bias", + "model.llm.layers.7.mlp.gate.bias", + "model.llm.layers.8.mlp.gate.bias", + "model.llm.layers.9.mlp.gate.bias" + ], + "weight_block_shape": [], + "weight_transform": "cast" + }, + { + "compute_dtype": "bfloat16", + "output_dtype": "bfloat16", + "required_activation_quantization": null, + "tensors": [ + "model.llm.embed.weight", + "model.llm.embed_norm.weight", + "model.llm.layers.0.attn.attn_sink_proj.weight", + "model.llm.layers.0.attn.gate_attn.weight", + "model.llm.layers.0.attn.k_sconv.weight", + "model.llm.layers.0.attn.q_l2_clip.scale", + "model.llm.layers.0.attn.rel_l2_clip.scale", + "model.llm.layers.0.attn.rel_logits_proj.proj", + "model.llm.layers.0.attn.v_sconv.weight", + "model.llm.layers.0.attn.wk_dv.weight", + "model.llm.layers.0.attn.wo_ud.weight", + "model.llm.layers.0.attn.wq_du.weight", + "model.llm.layers.0.attn.wr_du.weight", + "model.llm.layers.0.attn.wv_dv.weight", + "model.llm.layers.0.attn_norm.weight", + "model.llm.layers.0.attn_sconv.weight", + "model.llm.layers.0.mlp.gate.weight", + "model.llm.layers.0.mlp_norm.weight", + "model.llm.layers.0.mlp_sconv.weight", + "model.llm.layers.1.attn.attn_sink_proj.weight", + "model.llm.layers.1.attn.gate_attn.weight", + "model.llm.layers.1.attn.k_sconv.weight", + "model.llm.layers.1.attn.q_l2_clip.scale", + "model.llm.layers.1.attn.rel_l2_clip.scale", + "model.llm.layers.1.attn.rel_logits_proj.proj", + "model.llm.layers.1.attn.v_sconv.weight", + "model.llm.layers.1.attn.wk_dv.weight", + "model.llm.layers.1.attn.wo_ud.weight", + "model.llm.layers.1.attn.wq_du.weight", + "model.llm.layers.1.attn.wr_du.weight", + "model.llm.layers.1.attn.wv_dv.weight", + "model.llm.layers.1.attn_norm.weight", + "model.llm.layers.1.attn_sconv.weight", + "model.llm.layers.1.mlp.gate.weight", + "model.llm.layers.1.mlp_norm.weight", + "model.llm.layers.1.mlp_sconv.weight", + "model.llm.layers.10.attn.attn_sink_proj.weight", + "model.llm.layers.10.attn.gate_attn.weight", + "model.llm.layers.10.attn.k_sconv.weight", + "model.llm.layers.10.attn.q_l2_clip.scale", + "model.llm.layers.10.attn.rel_l2_clip.scale", + "model.llm.layers.10.attn.rel_logits_proj.proj", + "model.llm.layers.10.attn.v_sconv.weight", + "model.llm.layers.10.attn.wk_dv.weight", + "model.llm.layers.10.attn.wo_ud.weight", + "model.llm.layers.10.attn.wq_du.weight", + "model.llm.layers.10.attn.wr_du.weight", + "model.llm.layers.10.attn.wv_dv.weight", + "model.llm.layers.10.attn_norm.weight", + "model.llm.layers.10.attn_sconv.weight", + "model.llm.layers.10.mlp.gate.weight", + "model.llm.layers.10.mlp_norm.weight", + "model.llm.layers.10.mlp_sconv.weight", + "model.llm.layers.11.attn.attn_sink_proj.weight", + "model.llm.layers.11.attn.gate_attn.weight", + "model.llm.layers.11.attn.k_sconv.static_w", + "model.llm.layers.11.attn.k_sconv.weight_proj", + "model.llm.layers.11.attn.q_l2_clip.scale", + "model.llm.layers.11.attn.rel_l2_clip.scale", + "model.llm.layers.11.attn.rel_logits_proj.proj", + "model.llm.layers.11.attn.v_sconv.static_w", + "model.llm.layers.11.attn.v_sconv.weight_proj", + "model.llm.layers.11.attn.wk_dv.weight", + "model.llm.layers.11.attn.wo_ud.weight", + "model.llm.layers.11.attn.wq_du.weight", + "model.llm.layers.11.attn.wr_du.weight", + "model.llm.layers.11.attn.wv_dv.weight", + "model.llm.layers.11.attn_sconv.weight", + "model.llm.layers.11.k_input_norm.weight", + "model.llm.layers.11.mlp.gate.weight", + "model.llm.layers.11.mlp_norm.weight", + "model.llm.layers.11.mlp_sconv.weight", + "model.llm.layers.11.q_input_norm.weight", + "model.llm.layers.11.v_input_norm.weight", + "model.llm.layers.12.attn.attn_sink_proj.weight", + "model.llm.layers.12.attn.gate_attn.weight", + "model.llm.layers.12.attn.k_sconv.weight", + "model.llm.layers.12.attn.q_l2_clip.scale", + "model.llm.layers.12.attn.rel_l2_clip.scale", + "model.llm.layers.12.attn.rel_logits_proj.proj", + "model.llm.layers.12.attn.v_sconv.weight", + "model.llm.layers.12.attn.wk_dv.weight", + "model.llm.layers.12.attn.wo_ud.weight", + "model.llm.layers.12.attn.wq_du.weight", + "model.llm.layers.12.attn.wr_du.weight", + "model.llm.layers.12.attn.wv_dv.weight", + "model.llm.layers.12.attn_norm.weight", + "model.llm.layers.12.attn_sconv.weight", + "model.llm.layers.12.mlp.gate.weight", + "model.llm.layers.12.mlp_norm.weight", + "model.llm.layers.12.mlp_sconv.weight", + "model.llm.layers.13.attn.attn_sink_proj.weight", + "model.llm.layers.13.attn.gate_attn.weight", + "model.llm.layers.13.attn.k_sconv.weight", + "model.llm.layers.13.attn.q_l2_clip.scale", + "model.llm.layers.13.attn.rel_l2_clip.scale", + "model.llm.layers.13.attn.rel_logits_proj.proj", + "model.llm.layers.13.attn.v_sconv.weight", + "model.llm.layers.13.attn.wk_dv.weight", + "model.llm.layers.13.attn.wo_ud.weight", + "model.llm.layers.13.attn.wq_du.weight", + "model.llm.layers.13.attn.wr_du.weight", + "model.llm.layers.13.attn.wv_dv.weight", + "model.llm.layers.13.attn_norm.weight", + "model.llm.layers.13.attn_sconv.weight", + "model.llm.layers.13.mlp.gate.weight", + "model.llm.layers.13.mlp_norm.weight", + "model.llm.layers.13.mlp_sconv.weight", + "model.llm.layers.14.attn.attn_sink_proj.weight", + "model.llm.layers.14.attn.gate_attn.weight", + "model.llm.layers.14.attn.k_sconv.weight", + "model.llm.layers.14.attn.q_l2_clip.scale", + "model.llm.layers.14.attn.rel_l2_clip.scale", + "model.llm.layers.14.attn.rel_logits_proj.proj", + "model.llm.layers.14.attn.v_sconv.weight", + "model.llm.layers.14.attn.wk_dv.weight", + "model.llm.layers.14.attn.wo_ud.weight", + "model.llm.layers.14.attn.wq_du.weight", + "model.llm.layers.14.attn.wr_du.weight", + "model.llm.layers.14.attn.wv_dv.weight", + "model.llm.layers.14.attn_norm.weight", + "model.llm.layers.14.attn_sconv.weight", + "model.llm.layers.14.mlp.gate.weight", + "model.llm.layers.14.mlp_norm.weight", + "model.llm.layers.14.mlp_sconv.weight", + "model.llm.layers.15.attn.attn_sink_proj.weight", + "model.llm.layers.15.attn.gate_attn.weight", + "model.llm.layers.15.attn.k_sconv.weight", + "model.llm.layers.15.attn.q_l2_clip.scale", + "model.llm.layers.15.attn.rel_l2_clip.scale", + "model.llm.layers.15.attn.rel_logits_proj.proj", + "model.llm.layers.15.attn.v_sconv.weight", + "model.llm.layers.15.attn.wk_dv.weight", + "model.llm.layers.15.attn.wo_ud.weight", + "model.llm.layers.15.attn.wq_du.weight", + "model.llm.layers.15.attn.wr_du.weight", + "model.llm.layers.15.attn.wv_dv.weight", + "model.llm.layers.15.attn_norm.weight", + "model.llm.layers.15.attn_sconv.weight", + "model.llm.layers.15.mlp.gate.weight", + "model.llm.layers.15.mlp_norm.weight", + "model.llm.layers.15.mlp_sconv.weight", + "model.llm.layers.16.attn.attn_sink_proj.weight", + "model.llm.layers.16.attn.gate_attn.weight", + "model.llm.layers.16.attn.k_sconv.weight", + "model.llm.layers.16.attn.q_l2_clip.scale", + "model.llm.layers.16.attn.rel_l2_clip.scale", + "model.llm.layers.16.attn.rel_logits_proj.proj", + "model.llm.layers.16.attn.v_sconv.weight", + "model.llm.layers.16.attn.wk_dv.weight", + "model.llm.layers.16.attn.wo_ud.weight", + "model.llm.layers.16.attn.wq_du.weight", + "model.llm.layers.16.attn.wr_du.weight", + "model.llm.layers.16.attn.wv_dv.weight", + "model.llm.layers.16.attn_norm.weight", + "model.llm.layers.16.attn_sconv.weight", + "model.llm.layers.16.mlp.gate.weight", + "model.llm.layers.16.mlp_norm.weight", + "model.llm.layers.16.mlp_sconv.weight", + "model.llm.layers.17.attn.attn_sink_proj.weight", + "model.llm.layers.17.attn.gate_attn.weight", + "model.llm.layers.17.attn.k_sconv.static_w", + "model.llm.layers.17.attn.k_sconv.weight_proj", + "model.llm.layers.17.attn.q_l2_clip.scale", + "model.llm.layers.17.attn.rel_l2_clip.scale", + "model.llm.layers.17.attn.rel_logits_proj.proj", + "model.llm.layers.17.attn.v_sconv.static_w", + "model.llm.layers.17.attn.v_sconv.weight_proj", + "model.llm.layers.17.attn.wk_dv.weight", + "model.llm.layers.17.attn.wo_ud.weight", + "model.llm.layers.17.attn.wq_du.weight", + "model.llm.layers.17.attn.wr_du.weight", + "model.llm.layers.17.attn.wv_dv.weight", + "model.llm.layers.17.attn_sconv.weight", + "model.llm.layers.17.k_input_norm.weight", + "model.llm.layers.17.mlp.gate.weight", + "model.llm.layers.17.mlp_norm.weight", + "model.llm.layers.17.mlp_sconv.weight", + "model.llm.layers.17.q_input_norm.weight", + "model.llm.layers.17.v_input_norm.weight", + "model.llm.layers.18.attn.attn_sink_proj.weight", + "model.llm.layers.18.attn.gate_attn.weight", + "model.llm.layers.18.attn.k_sconv.weight", + "model.llm.layers.18.attn.q_l2_clip.scale", + "model.llm.layers.18.attn.rel_l2_clip.scale", + "model.llm.layers.18.attn.rel_logits_proj.proj", + "model.llm.layers.18.attn.v_sconv.weight", + "model.llm.layers.18.attn.wk_dv.weight", + "model.llm.layers.18.attn.wo_ud.weight", + "model.llm.layers.18.attn.wq_du.weight", + "model.llm.layers.18.attn.wr_du.weight", + "model.llm.layers.18.attn.wv_dv.weight", + "model.llm.layers.18.attn_norm.weight", + "model.llm.layers.18.attn_sconv.weight", + "model.llm.layers.18.mlp.gate.weight", + "model.llm.layers.18.mlp_norm.weight", + "model.llm.layers.18.mlp_sconv.weight", + "model.llm.layers.19.attn.attn_sink_proj.weight", + "model.llm.layers.19.attn.gate_attn.weight", + "model.llm.layers.19.attn.k_sconv.weight", + "model.llm.layers.19.attn.q_l2_clip.scale", + "model.llm.layers.19.attn.rel_l2_clip.scale", + "model.llm.layers.19.attn.rel_logits_proj.proj", + "model.llm.layers.19.attn.v_sconv.weight", + "model.llm.layers.19.attn.wk_dv.weight", + "model.llm.layers.19.attn.wo_ud.weight", + "model.llm.layers.19.attn.wq_du.weight", + "model.llm.layers.19.attn.wr_du.weight", + "model.llm.layers.19.attn.wv_dv.weight", + "model.llm.layers.19.attn_norm.weight", + "model.llm.layers.19.attn_sconv.weight", + "model.llm.layers.19.mlp.gate.weight", + "model.llm.layers.19.mlp_norm.weight", + "model.llm.layers.19.mlp_sconv.weight", + "model.llm.layers.2.attn.attn_sink_proj.weight", + "model.llm.layers.2.attn.gate_attn.weight", + "model.llm.layers.2.attn.k_sconv.weight", + "model.llm.layers.2.attn.q_l2_clip.scale", + "model.llm.layers.2.attn.rel_l2_clip.scale", + "model.llm.layers.2.attn.rel_logits_proj.proj", + "model.llm.layers.2.attn.v_sconv.weight", + "model.llm.layers.2.attn.wk_dv.weight", + "model.llm.layers.2.attn.wo_ud.weight", + "model.llm.layers.2.attn.wq_du.weight", + "model.llm.layers.2.attn.wr_du.weight", + "model.llm.layers.2.attn.wv_dv.weight", + "model.llm.layers.2.attn_norm.weight", + "model.llm.layers.2.attn_sconv.weight", + "model.llm.layers.2.mlp.gate.weight", + "model.llm.layers.2.mlp_norm.weight", + "model.llm.layers.2.mlp_sconv.weight", + "model.llm.layers.20.attn.attn_sink_proj.weight", + "model.llm.layers.20.attn.gate_attn.weight", + "model.llm.layers.20.attn.k_sconv.weight", + "model.llm.layers.20.attn.q_l2_clip.scale", + "model.llm.layers.20.attn.rel_l2_clip.scale", + "model.llm.layers.20.attn.rel_logits_proj.proj", + "model.llm.layers.20.attn.v_sconv.weight", + "model.llm.layers.20.attn.wk_dv.weight", + "model.llm.layers.20.attn.wo_ud.weight", + "model.llm.layers.20.attn.wq_du.weight", + "model.llm.layers.20.attn.wr_du.weight", + "model.llm.layers.20.attn.wv_dv.weight", + "model.llm.layers.20.attn_norm.weight", + "model.llm.layers.20.attn_sconv.weight", + "model.llm.layers.20.mlp.gate.weight", + "model.llm.layers.20.mlp_norm.weight", + "model.llm.layers.20.mlp_sconv.weight", + "model.llm.layers.21.attn.attn_sink_proj.weight", + "model.llm.layers.21.attn.gate_attn.weight", + "model.llm.layers.21.attn.k_sconv.weight", + "model.llm.layers.21.attn.q_l2_clip.scale", + "model.llm.layers.21.attn.rel_l2_clip.scale", + "model.llm.layers.21.attn.rel_logits_proj.proj", + "model.llm.layers.21.attn.v_sconv.weight", + "model.llm.layers.21.attn.wk_dv.weight", + "model.llm.layers.21.attn.wo_ud.weight", + "model.llm.layers.21.attn.wq_du.weight", + "model.llm.layers.21.attn.wr_du.weight", + "model.llm.layers.21.attn.wv_dv.weight", + "model.llm.layers.21.attn_norm.weight", + "model.llm.layers.21.attn_sconv.weight", + "model.llm.layers.21.mlp.gate.weight", + "model.llm.layers.21.mlp_norm.weight", + "model.llm.layers.21.mlp_sconv.weight", + "model.llm.layers.22.attn.attn_sink_proj.weight", + "model.llm.layers.22.attn.gate_attn.weight", + "model.llm.layers.22.attn.k_sconv.weight", + "model.llm.layers.22.attn.q_l2_clip.scale", + "model.llm.layers.22.attn.rel_l2_clip.scale", + "model.llm.layers.22.attn.rel_logits_proj.proj", + "model.llm.layers.22.attn.v_sconv.weight", + "model.llm.layers.22.attn.wk_dv.weight", + "model.llm.layers.22.attn.wo_ud.weight", + "model.llm.layers.22.attn.wq_du.weight", + "model.llm.layers.22.attn.wr_du.weight", + "model.llm.layers.22.attn.wv_dv.weight", + "model.llm.layers.22.attn_norm.weight", + "model.llm.layers.22.attn_sconv.weight", + "model.llm.layers.22.mlp.gate.weight", + "model.llm.layers.22.mlp_norm.weight", + "model.llm.layers.22.mlp_sconv.weight", + "model.llm.layers.23.attn.attn_sink_proj.weight", + "model.llm.layers.23.attn.gate_attn.weight", + "model.llm.layers.23.attn.k_sconv.static_w", + "model.llm.layers.23.attn.k_sconv.weight_proj", + "model.llm.layers.23.attn.q_l2_clip.scale", + "model.llm.layers.23.attn.rel_l2_clip.scale", + "model.llm.layers.23.attn.rel_logits_proj.proj", + "model.llm.layers.23.attn.v_sconv.static_w", + "model.llm.layers.23.attn.v_sconv.weight_proj", + "model.llm.layers.23.attn.wk_dv.weight", + "model.llm.layers.23.attn.wo_ud.weight", + "model.llm.layers.23.attn.wq_du.weight", + "model.llm.layers.23.attn.wr_du.weight", + "model.llm.layers.23.attn.wv_dv.weight", + "model.llm.layers.23.attn_sconv.weight", + "model.llm.layers.23.k_input_norm.weight", + "model.llm.layers.23.mlp.gate.weight", + "model.llm.layers.23.mlp_norm.weight", + "model.llm.layers.23.mlp_sconv.weight", + "model.llm.layers.23.q_input_norm.weight", + "model.llm.layers.23.v_input_norm.weight", + "model.llm.layers.24.attn.attn_sink_proj.weight", + "model.llm.layers.24.attn.gate_attn.weight", + "model.llm.layers.24.attn.k_sconv.weight", + "model.llm.layers.24.attn.q_l2_clip.scale", + "model.llm.layers.24.attn.rel_l2_clip.scale", + "model.llm.layers.24.attn.rel_logits_proj.proj", + "model.llm.layers.24.attn.v_sconv.weight", + "model.llm.layers.24.attn.wk_dv.weight", + "model.llm.layers.24.attn.wo_ud.weight", + "model.llm.layers.24.attn.wq_du.weight", + "model.llm.layers.24.attn.wr_du.weight", + "model.llm.layers.24.attn.wv_dv.weight", + "model.llm.layers.24.attn_norm.weight", + "model.llm.layers.24.attn_sconv.weight", + "model.llm.layers.24.mlp.gate.weight", + "model.llm.layers.24.mlp_norm.weight", + "model.llm.layers.24.mlp_sconv.weight", + "model.llm.layers.25.attn.attn_sink_proj.weight", + "model.llm.layers.25.attn.gate_attn.weight", + "model.llm.layers.25.attn.k_sconv.weight", + "model.llm.layers.25.attn.q_l2_clip.scale", + "model.llm.layers.25.attn.rel_l2_clip.scale", + "model.llm.layers.25.attn.rel_logits_proj.proj", + "model.llm.layers.25.attn.v_sconv.weight", + "model.llm.layers.25.attn.wk_dv.weight", + "model.llm.layers.25.attn.wo_ud.weight", + "model.llm.layers.25.attn.wq_du.weight", + "model.llm.layers.25.attn.wr_du.weight", + "model.llm.layers.25.attn.wv_dv.weight", + "model.llm.layers.25.attn_norm.weight", + "model.llm.layers.25.attn_sconv.weight", + "model.llm.layers.25.mlp.gate.weight", + "model.llm.layers.25.mlp_norm.weight", + "model.llm.layers.25.mlp_sconv.weight", + "model.llm.layers.26.attn.attn_sink_proj.weight", + "model.llm.layers.26.attn.gate_attn.weight", + "model.llm.layers.26.attn.k_sconv.weight", + "model.llm.layers.26.attn.q_l2_clip.scale", + "model.llm.layers.26.attn.rel_l2_clip.scale", + "model.llm.layers.26.attn.rel_logits_proj.proj", + "model.llm.layers.26.attn.v_sconv.weight", + "model.llm.layers.26.attn.wk_dv.weight", + "model.llm.layers.26.attn.wo_ud.weight", + "model.llm.layers.26.attn.wq_du.weight", + "model.llm.layers.26.attn.wr_du.weight", + "model.llm.layers.26.attn.wv_dv.weight", + "model.llm.layers.26.attn_norm.weight", + "model.llm.layers.26.attn_sconv.weight", + "model.llm.layers.26.mlp.gate.weight", + "model.llm.layers.26.mlp_norm.weight", + "model.llm.layers.26.mlp_sconv.weight", + "model.llm.layers.27.attn.attn_sink_proj.weight", + "model.llm.layers.27.attn.gate_attn.weight", + "model.llm.layers.27.attn.k_sconv.weight", + "model.llm.layers.27.attn.q_l2_clip.scale", + "model.llm.layers.27.attn.rel_l2_clip.scale", + "model.llm.layers.27.attn.rel_logits_proj.proj", + "model.llm.layers.27.attn.v_sconv.weight", + "model.llm.layers.27.attn.wk_dv.weight", + "model.llm.layers.27.attn.wo_ud.weight", + "model.llm.layers.27.attn.wq_du.weight", + "model.llm.layers.27.attn.wr_du.weight", + "model.llm.layers.27.attn.wv_dv.weight", + "model.llm.layers.27.attn_norm.weight", + "model.llm.layers.27.attn_sconv.weight", + "model.llm.layers.27.mlp.gate.weight", + "model.llm.layers.27.mlp_norm.weight", + "model.llm.layers.27.mlp_sconv.weight", + "model.llm.layers.28.attn.attn_sink_proj.weight", + "model.llm.layers.28.attn.gate_attn.weight", + "model.llm.layers.28.attn.k_sconv.weight", + "model.llm.layers.28.attn.q_l2_clip.scale", + "model.llm.layers.28.attn.rel_l2_clip.scale", + "model.llm.layers.28.attn.rel_logits_proj.proj", + "model.llm.layers.28.attn.v_sconv.weight", + "model.llm.layers.28.attn.wk_dv.weight", + "model.llm.layers.28.attn.wo_ud.weight", + "model.llm.layers.28.attn.wq_du.weight", + "model.llm.layers.28.attn.wr_du.weight", + "model.llm.layers.28.attn.wv_dv.weight", + "model.llm.layers.28.attn_norm.weight", + "model.llm.layers.28.attn_sconv.weight", + "model.llm.layers.28.mlp.gate.weight", + "model.llm.layers.28.mlp_norm.weight", + "model.llm.layers.28.mlp_sconv.weight", + "model.llm.layers.29.attn.attn_sink_proj.weight", + "model.llm.layers.29.attn.gate_attn.weight", + "model.llm.layers.29.attn.k_sconv.static_w", + "model.llm.layers.29.attn.k_sconv.weight_proj", + "model.llm.layers.29.attn.q_l2_clip.scale", + "model.llm.layers.29.attn.rel_l2_clip.scale", + "model.llm.layers.29.attn.rel_logits_proj.proj", + "model.llm.layers.29.attn.v_sconv.static_w", + "model.llm.layers.29.attn.v_sconv.weight_proj", + "model.llm.layers.29.attn.wk_dv.weight", + "model.llm.layers.29.attn.wo_ud.weight", + "model.llm.layers.29.attn.wq_du.weight", + "model.llm.layers.29.attn.wr_du.weight", + "model.llm.layers.29.attn.wv_dv.weight", + "model.llm.layers.29.attn_sconv.weight", + "model.llm.layers.29.k_input_norm.weight", + "model.llm.layers.29.mlp.gate.weight", + "model.llm.layers.29.mlp_norm.weight", + "model.llm.layers.29.mlp_sconv.weight", + "model.llm.layers.29.q_input_norm.weight", + "model.llm.layers.29.v_input_norm.weight", + "model.llm.layers.3.attn.attn_sink_proj.weight", + "model.llm.layers.3.attn.gate_attn.weight", + "model.llm.layers.3.attn.k_sconv.weight", + "model.llm.layers.3.attn.q_l2_clip.scale", + "model.llm.layers.3.attn.rel_l2_clip.scale", + "model.llm.layers.3.attn.rel_logits_proj.proj", + "model.llm.layers.3.attn.v_sconv.weight", + "model.llm.layers.3.attn.wk_dv.weight", + "model.llm.layers.3.attn.wo_ud.weight", + "model.llm.layers.3.attn.wq_du.weight", + "model.llm.layers.3.attn.wr_du.weight", + "model.llm.layers.3.attn.wv_dv.weight", + "model.llm.layers.3.attn_norm.weight", + "model.llm.layers.3.attn_sconv.weight", + "model.llm.layers.3.mlp.gate.weight", + "model.llm.layers.3.mlp_norm.weight", + "model.llm.layers.3.mlp_sconv.weight", + "model.llm.layers.30.attn.attn_sink_proj.weight", + "model.llm.layers.30.attn.gate_attn.weight", + "model.llm.layers.30.attn.k_sconv.weight", + "model.llm.layers.30.attn.q_l2_clip.scale", + "model.llm.layers.30.attn.rel_l2_clip.scale", + "model.llm.layers.30.attn.rel_logits_proj.proj", + "model.llm.layers.30.attn.v_sconv.weight", + "model.llm.layers.30.attn.wk_dv.weight", + "model.llm.layers.30.attn.wo_ud.weight", + "model.llm.layers.30.attn.wq_du.weight", + "model.llm.layers.30.attn.wr_du.weight", + "model.llm.layers.30.attn.wv_dv.weight", + "model.llm.layers.30.attn_norm.weight", + "model.llm.layers.30.attn_sconv.weight", + "model.llm.layers.30.mlp.gate.weight", + "model.llm.layers.30.mlp_norm.weight", + "model.llm.layers.30.mlp_sconv.weight", + "model.llm.layers.31.attn.attn_sink_proj.weight", + "model.llm.layers.31.attn.gate_attn.weight", + "model.llm.layers.31.attn.k_sconv.weight", + "model.llm.layers.31.attn.q_l2_clip.scale", + "model.llm.layers.31.attn.rel_l2_clip.scale", + "model.llm.layers.31.attn.rel_logits_proj.proj", + "model.llm.layers.31.attn.v_sconv.weight", + "model.llm.layers.31.attn.wk_dv.weight", + "model.llm.layers.31.attn.wo_ud.weight", + "model.llm.layers.31.attn.wq_du.weight", + "model.llm.layers.31.attn.wr_du.weight", + "model.llm.layers.31.attn.wv_dv.weight", + "model.llm.layers.31.attn_norm.weight", + "model.llm.layers.31.attn_sconv.weight", + "model.llm.layers.31.mlp.gate.weight", + "model.llm.layers.31.mlp_norm.weight", + "model.llm.layers.31.mlp_sconv.weight", + "model.llm.layers.32.attn.attn_sink_proj.weight", + "model.llm.layers.32.attn.gate_attn.weight", + "model.llm.layers.32.attn.k_sconv.weight", + "model.llm.layers.32.attn.q_l2_clip.scale", + "model.llm.layers.32.attn.rel_l2_clip.scale", + "model.llm.layers.32.attn.rel_logits_proj.proj", + "model.llm.layers.32.attn.v_sconv.weight", + "model.llm.layers.32.attn.wk_dv.weight", + "model.llm.layers.32.attn.wo_ud.weight", + "model.llm.layers.32.attn.wq_du.weight", + "model.llm.layers.32.attn.wr_du.weight", + "model.llm.layers.32.attn.wv_dv.weight", + "model.llm.layers.32.attn_norm.weight", + "model.llm.layers.32.attn_sconv.weight", + "model.llm.layers.32.mlp.gate.weight", + "model.llm.layers.32.mlp_norm.weight", + "model.llm.layers.32.mlp_sconv.weight", + "model.llm.layers.33.attn.attn_sink_proj.weight", + "model.llm.layers.33.attn.gate_attn.weight", + "model.llm.layers.33.attn.k_sconv.weight", + "model.llm.layers.33.attn.q_l2_clip.scale", + "model.llm.layers.33.attn.rel_l2_clip.scale", + "model.llm.layers.33.attn.rel_logits_proj.proj", + "model.llm.layers.33.attn.v_sconv.weight", + "model.llm.layers.33.attn.wk_dv.weight", + "model.llm.layers.33.attn.wo_ud.weight", + "model.llm.layers.33.attn.wq_du.weight", + "model.llm.layers.33.attn.wr_du.weight", + "model.llm.layers.33.attn.wv_dv.weight", + "model.llm.layers.33.attn_norm.weight", + "model.llm.layers.33.attn_sconv.weight", + "model.llm.layers.33.mlp.gate.weight", + "model.llm.layers.33.mlp_norm.weight", + "model.llm.layers.33.mlp_sconv.weight", + "model.llm.layers.34.attn.attn_sink_proj.weight", + "model.llm.layers.34.attn.gate_attn.weight", + "model.llm.layers.34.attn.k_sconv.weight", + "model.llm.layers.34.attn.q_l2_clip.scale", + "model.llm.layers.34.attn.rel_l2_clip.scale", + "model.llm.layers.34.attn.rel_logits_proj.proj", + "model.llm.layers.34.attn.v_sconv.weight", + "model.llm.layers.34.attn.wk_dv.weight", + "model.llm.layers.34.attn.wo_ud.weight", + "model.llm.layers.34.attn.wq_du.weight", + "model.llm.layers.34.attn.wr_du.weight", + "model.llm.layers.34.attn.wv_dv.weight", + "model.llm.layers.34.attn_norm.weight", + "model.llm.layers.34.attn_sconv.weight", + "model.llm.layers.34.mlp.gate.weight", + "model.llm.layers.34.mlp_norm.weight", + "model.llm.layers.34.mlp_sconv.weight", + "model.llm.layers.35.attn.attn_sink_proj.weight", + "model.llm.layers.35.attn.gate_attn.weight", + "model.llm.layers.35.attn.k_sconv.static_w", + "model.llm.layers.35.attn.k_sconv.weight_proj", + "model.llm.layers.35.attn.q_l2_clip.scale", + "model.llm.layers.35.attn.rel_l2_clip.scale", + "model.llm.layers.35.attn.rel_logits_proj.proj", + "model.llm.layers.35.attn.v_sconv.static_w", + "model.llm.layers.35.attn.v_sconv.weight_proj", + "model.llm.layers.35.attn.wk_dv.weight", + "model.llm.layers.35.attn.wo_ud.weight", + "model.llm.layers.35.attn.wq_du.weight", + "model.llm.layers.35.attn.wr_du.weight", + "model.llm.layers.35.attn.wv_dv.weight", + "model.llm.layers.35.attn_sconv.weight", + "model.llm.layers.35.k_input_norm.weight", + "model.llm.layers.35.mlp.gate.weight", + "model.llm.layers.35.mlp_norm.weight", + "model.llm.layers.35.mlp_sconv.weight", + "model.llm.layers.35.q_input_norm.weight", + "model.llm.layers.35.v_input_norm.weight", + "model.llm.layers.36.attn.attn_sink_proj.weight", + "model.llm.layers.36.attn.gate_attn.weight", + "model.llm.layers.36.attn.k_sconv.weight", + "model.llm.layers.36.attn.q_l2_clip.scale", + "model.llm.layers.36.attn.rel_l2_clip.scale", + "model.llm.layers.36.attn.rel_logits_proj.proj", + "model.llm.layers.36.attn.v_sconv.weight", + "model.llm.layers.36.attn.wk_dv.weight", + "model.llm.layers.36.attn.wo_ud.weight", + "model.llm.layers.36.attn.wq_du.weight", + "model.llm.layers.36.attn.wr_du.weight", + "model.llm.layers.36.attn.wv_dv.weight", + "model.llm.layers.36.attn_norm.weight", + "model.llm.layers.36.attn_sconv.weight", + "model.llm.layers.36.mlp.gate.weight", + "model.llm.layers.36.mlp_norm.weight", + "model.llm.layers.36.mlp_sconv.weight", + "model.llm.layers.37.attn.attn_sink_proj.weight", + "model.llm.layers.37.attn.gate_attn.weight", + "model.llm.layers.37.attn.k_sconv.weight", + "model.llm.layers.37.attn.q_l2_clip.scale", + "model.llm.layers.37.attn.rel_l2_clip.scale", + "model.llm.layers.37.attn.rel_logits_proj.proj", + "model.llm.layers.37.attn.v_sconv.weight", + "model.llm.layers.37.attn.wk_dv.weight", + "model.llm.layers.37.attn.wo_ud.weight", + "model.llm.layers.37.attn.wq_du.weight", + "model.llm.layers.37.attn.wr_du.weight", + "model.llm.layers.37.attn.wv_dv.weight", + "model.llm.layers.37.attn_norm.weight", + "model.llm.layers.37.attn_sconv.weight", + "model.llm.layers.37.mlp.gate.weight", + "model.llm.layers.37.mlp_norm.weight", + "model.llm.layers.37.mlp_sconv.weight", + "model.llm.layers.38.attn.attn_sink_proj.weight", + "model.llm.layers.38.attn.gate_attn.weight", + "model.llm.layers.38.attn.k_sconv.weight", + "model.llm.layers.38.attn.q_l2_clip.scale", + "model.llm.layers.38.attn.rel_l2_clip.scale", + "model.llm.layers.38.attn.rel_logits_proj.proj", + "model.llm.layers.38.attn.v_sconv.weight", + "model.llm.layers.38.attn.wk_dv.weight", + "model.llm.layers.38.attn.wo_ud.weight", + "model.llm.layers.38.attn.wq_du.weight", + "model.llm.layers.38.attn.wr_du.weight", + "model.llm.layers.38.attn.wv_dv.weight", + "model.llm.layers.38.attn_norm.weight", + "model.llm.layers.38.attn_sconv.weight", + "model.llm.layers.38.mlp.gate.weight", + "model.llm.layers.38.mlp_norm.weight", + "model.llm.layers.38.mlp_sconv.weight", + "model.llm.layers.39.attn.attn_sink_proj.weight", + "model.llm.layers.39.attn.gate_attn.weight", + "model.llm.layers.39.attn.k_sconv.weight", + "model.llm.layers.39.attn.q_l2_clip.scale", + "model.llm.layers.39.attn.rel_l2_clip.scale", + "model.llm.layers.39.attn.rel_logits_proj.proj", + "model.llm.layers.39.attn.v_sconv.weight", + "model.llm.layers.39.attn.wk_dv.weight", + "model.llm.layers.39.attn.wo_ud.weight", + "model.llm.layers.39.attn.wq_du.weight", + "model.llm.layers.39.attn.wr_du.weight", + "model.llm.layers.39.attn.wv_dv.weight", + "model.llm.layers.39.attn_norm.weight", + "model.llm.layers.39.attn_sconv.weight", + "model.llm.layers.39.mlp.gate.weight", + "model.llm.layers.39.mlp_norm.weight", + "model.llm.layers.39.mlp_sconv.weight", + "model.llm.layers.4.attn.attn_sink_proj.weight", + "model.llm.layers.4.attn.gate_attn.weight", + "model.llm.layers.4.attn.k_sconv.weight", + "model.llm.layers.4.attn.q_l2_clip.scale", + "model.llm.layers.4.attn.rel_l2_clip.scale", + "model.llm.layers.4.attn.rel_logits_proj.proj", + "model.llm.layers.4.attn.v_sconv.weight", + "model.llm.layers.4.attn.wk_dv.weight", + "model.llm.layers.4.attn.wo_ud.weight", + "model.llm.layers.4.attn.wq_du.weight", + "model.llm.layers.4.attn.wr_du.weight", + "model.llm.layers.4.attn.wv_dv.weight", + "model.llm.layers.4.attn_norm.weight", + "model.llm.layers.4.attn_sconv.weight", + "model.llm.layers.4.mlp.gate.weight", + "model.llm.layers.4.mlp_norm.weight", + "model.llm.layers.4.mlp_sconv.weight", + "model.llm.layers.40.attn.attn_sink_proj.weight", + "model.llm.layers.40.attn.gate_attn.weight", + "model.llm.layers.40.attn.k_sconv.weight", + "model.llm.layers.40.attn.q_l2_clip.scale", + "model.llm.layers.40.attn.rel_l2_clip.scale", + "model.llm.layers.40.attn.rel_logits_proj.proj", + "model.llm.layers.40.attn.v_sconv.weight", + "model.llm.layers.40.attn.wk_dv.weight", + "model.llm.layers.40.attn.wo_ud.weight", + "model.llm.layers.40.attn.wq_du.weight", + "model.llm.layers.40.attn.wr_du.weight", + "model.llm.layers.40.attn.wv_dv.weight", + "model.llm.layers.40.attn_norm.weight", + "model.llm.layers.40.attn_sconv.weight", + "model.llm.layers.40.mlp.gate.weight", + "model.llm.layers.40.mlp_norm.weight", + "model.llm.layers.40.mlp_sconv.weight", + "model.llm.layers.41.attn.attn_sink_proj.weight", + "model.llm.layers.41.attn.gate_attn.weight", + "model.llm.layers.41.attn.k_sconv.static_w", + "model.llm.layers.41.attn.k_sconv.weight_proj", + "model.llm.layers.41.attn.q_l2_clip.scale", + "model.llm.layers.41.attn.rel_l2_clip.scale", + "model.llm.layers.41.attn.rel_logits_proj.proj", + "model.llm.layers.41.attn.v_sconv.static_w", + "model.llm.layers.41.attn.v_sconv.weight_proj", + "model.llm.layers.41.attn.wk_dv.weight", + "model.llm.layers.41.attn.wo_ud.weight", + "model.llm.layers.41.attn.wq_du.weight", + "model.llm.layers.41.attn.wr_du.weight", + "model.llm.layers.41.attn.wv_dv.weight", + "model.llm.layers.41.attn_sconv.weight", + "model.llm.layers.41.k_input_norm.weight", + "model.llm.layers.41.mlp.gate.weight", + "model.llm.layers.41.mlp_norm.weight", + "model.llm.layers.41.mlp_sconv.weight", + "model.llm.layers.41.q_input_norm.weight", + "model.llm.layers.41.v_input_norm.weight", + "model.llm.layers.42.attn.attn_sink_proj.weight", + "model.llm.layers.42.attn.gate_attn.weight", + "model.llm.layers.42.attn.k_sconv.weight", + "model.llm.layers.42.attn.q_l2_clip.scale", + "model.llm.layers.42.attn.rel_l2_clip.scale", + "model.llm.layers.42.attn.rel_logits_proj.proj", + "model.llm.layers.42.attn.v_sconv.weight", + "model.llm.layers.42.attn.wk_dv.weight", + "model.llm.layers.42.attn.wo_ud.weight", + "model.llm.layers.42.attn.wq_du.weight", + "model.llm.layers.42.attn.wr_du.weight", + "model.llm.layers.42.attn.wv_dv.weight", + "model.llm.layers.42.attn_norm.weight", + "model.llm.layers.42.attn_sconv.weight", + "model.llm.layers.42.mlp.gate.weight", + "model.llm.layers.42.mlp_norm.weight", + "model.llm.layers.42.mlp_sconv.weight", + "model.llm.layers.43.attn.attn_sink_proj.weight", + "model.llm.layers.43.attn.gate_attn.weight", + "model.llm.layers.43.attn.k_sconv.weight", + "model.llm.layers.43.attn.q_l2_clip.scale", + "model.llm.layers.43.attn.rel_l2_clip.scale", + "model.llm.layers.43.attn.rel_logits_proj.proj", + "model.llm.layers.43.attn.v_sconv.weight", + "model.llm.layers.43.attn.wk_dv.weight", + "model.llm.layers.43.attn.wo_ud.weight", + "model.llm.layers.43.attn.wq_du.weight", + "model.llm.layers.43.attn.wr_du.weight", + "model.llm.layers.43.attn.wv_dv.weight", + "model.llm.layers.43.attn_norm.weight", + "model.llm.layers.43.attn_sconv.weight", + "model.llm.layers.43.mlp.gate.weight", + "model.llm.layers.43.mlp_norm.weight", + "model.llm.layers.43.mlp_sconv.weight", + "model.llm.layers.44.attn.attn_sink_proj.weight", + "model.llm.layers.44.attn.gate_attn.weight", + "model.llm.layers.44.attn.k_sconv.weight", + "model.llm.layers.44.attn.q_l2_clip.scale", + "model.llm.layers.44.attn.rel_l2_clip.scale", + "model.llm.layers.44.attn.rel_logits_proj.proj", + "model.llm.layers.44.attn.v_sconv.weight", + "model.llm.layers.44.attn.wk_dv.weight", + "model.llm.layers.44.attn.wo_ud.weight", + "model.llm.layers.44.attn.wq_du.weight", + "model.llm.layers.44.attn.wr_du.weight", + "model.llm.layers.44.attn.wv_dv.weight", + "model.llm.layers.44.attn_norm.weight", + "model.llm.layers.44.attn_sconv.weight", + "model.llm.layers.44.mlp.gate.weight", + "model.llm.layers.44.mlp_norm.weight", + "model.llm.layers.44.mlp_sconv.weight", + "model.llm.layers.45.attn.attn_sink_proj.weight", + "model.llm.layers.45.attn.gate_attn.weight", + "model.llm.layers.45.attn.k_sconv.weight", + "model.llm.layers.45.attn.q_l2_clip.scale", + "model.llm.layers.45.attn.rel_l2_clip.scale", + "model.llm.layers.45.attn.rel_logits_proj.proj", + "model.llm.layers.45.attn.v_sconv.weight", + "model.llm.layers.45.attn.wk_dv.weight", + "model.llm.layers.45.attn.wo_ud.weight", + "model.llm.layers.45.attn.wq_du.weight", + "model.llm.layers.45.attn.wr_du.weight", + "model.llm.layers.45.attn.wv_dv.weight", + "model.llm.layers.45.attn_norm.weight", + "model.llm.layers.45.attn_sconv.weight", + "model.llm.layers.45.mlp.gate.weight", + "model.llm.layers.45.mlp_norm.weight", + "model.llm.layers.45.mlp_sconv.weight", + "model.llm.layers.46.attn.attn_sink_proj.weight", + "model.llm.layers.46.attn.gate_attn.weight", + "model.llm.layers.46.attn.k_sconv.weight", + "model.llm.layers.46.attn.q_l2_clip.scale", + "model.llm.layers.46.attn.rel_l2_clip.scale", + "model.llm.layers.46.attn.rel_logits_proj.proj", + "model.llm.layers.46.attn.v_sconv.weight", + "model.llm.layers.46.attn.wk_dv.weight", + "model.llm.layers.46.attn.wo_ud.weight", + "model.llm.layers.46.attn.wq_du.weight", + "model.llm.layers.46.attn.wr_du.weight", + "model.llm.layers.46.attn.wv_dv.weight", + "model.llm.layers.46.attn_norm.weight", + "model.llm.layers.46.attn_sconv.weight", + "model.llm.layers.46.mlp.gate.weight", + "model.llm.layers.46.mlp_norm.weight", + "model.llm.layers.46.mlp_sconv.weight", + "model.llm.layers.47.attn.attn_sink_proj.weight", + "model.llm.layers.47.attn.gate_attn.weight", + "model.llm.layers.47.attn.k_sconv.static_w", + "model.llm.layers.47.attn.k_sconv.weight_proj", + "model.llm.layers.47.attn.q_l2_clip.scale", + "model.llm.layers.47.attn.rel_l2_clip.scale", + "model.llm.layers.47.attn.rel_logits_proj.proj", + "model.llm.layers.47.attn.v_sconv.static_w", + "model.llm.layers.47.attn.v_sconv.weight_proj", + "model.llm.layers.47.attn.wk_dv.weight", + "model.llm.layers.47.attn.wo_ud.weight", + "model.llm.layers.47.attn.wq_du.weight", + "model.llm.layers.47.attn.wr_du.weight", + "model.llm.layers.47.attn.wv_dv.weight", + "model.llm.layers.47.attn_sconv.weight", + "model.llm.layers.47.k_input_norm.weight", + "model.llm.layers.47.mlp.gate.weight", + "model.llm.layers.47.mlp_norm.weight", + "model.llm.layers.47.mlp_sconv.weight", + "model.llm.layers.47.q_input_norm.weight", + "model.llm.layers.47.v_input_norm.weight", + "model.llm.layers.5.attn.attn_sink_proj.weight", + "model.llm.layers.5.attn.gate_attn.weight", + "model.llm.layers.5.attn.k_sconv.static_w", + "model.llm.layers.5.attn.k_sconv.weight_proj", + "model.llm.layers.5.attn.q_l2_clip.scale", + "model.llm.layers.5.attn.rel_l2_clip.scale", + "model.llm.layers.5.attn.rel_logits_proj.proj", + "model.llm.layers.5.attn.v_sconv.static_w", + "model.llm.layers.5.attn.v_sconv.weight_proj", + "model.llm.layers.5.attn.wk_dv.weight", + "model.llm.layers.5.attn.wo_ud.weight", + "model.llm.layers.5.attn.wq_du.weight", + "model.llm.layers.5.attn.wr_du.weight", + "model.llm.layers.5.attn.wv_dv.weight", + "model.llm.layers.5.attn_sconv.weight", + "model.llm.layers.5.k_input_norm.weight", + "model.llm.layers.5.mlp.gate.weight", + "model.llm.layers.5.mlp_norm.weight", + "model.llm.layers.5.mlp_sconv.weight", + "model.llm.layers.5.q_input_norm.weight", + "model.llm.layers.5.v_input_norm.weight", + "model.llm.layers.6.attn.attn_sink_proj.weight", + "model.llm.layers.6.attn.gate_attn.weight", + "model.llm.layers.6.attn.k_sconv.weight", + "model.llm.layers.6.attn.q_l2_clip.scale", + "model.llm.layers.6.attn.rel_l2_clip.scale", + "model.llm.layers.6.attn.rel_logits_proj.proj", + "model.llm.layers.6.attn.v_sconv.weight", + "model.llm.layers.6.attn.wk_dv.weight", + "model.llm.layers.6.attn.wo_ud.weight", + "model.llm.layers.6.attn.wq_du.weight", + "model.llm.layers.6.attn.wr_du.weight", + "model.llm.layers.6.attn.wv_dv.weight", + "model.llm.layers.6.attn_norm.weight", + "model.llm.layers.6.attn_sconv.weight", + "model.llm.layers.6.mlp.gate.weight", + "model.llm.layers.6.mlp_norm.weight", + "model.llm.layers.6.mlp_sconv.weight", + "model.llm.layers.7.attn.attn_sink_proj.weight", + "model.llm.layers.7.attn.gate_attn.weight", + "model.llm.layers.7.attn.k_sconv.weight", + "model.llm.layers.7.attn.q_l2_clip.scale", + "model.llm.layers.7.attn.rel_l2_clip.scale", + "model.llm.layers.7.attn.rel_logits_proj.proj", + "model.llm.layers.7.attn.v_sconv.weight", + "model.llm.layers.7.attn.wk_dv.weight", + "model.llm.layers.7.attn.wo_ud.weight", + "model.llm.layers.7.attn.wq_du.weight", + "model.llm.layers.7.attn.wr_du.weight", + "model.llm.layers.7.attn.wv_dv.weight", + "model.llm.layers.7.attn_norm.weight", + "model.llm.layers.7.attn_sconv.weight", + "model.llm.layers.7.mlp.gate.weight", + "model.llm.layers.7.mlp_norm.weight", + "model.llm.layers.7.mlp_sconv.weight", + "model.llm.layers.8.attn.attn_sink_proj.weight", + "model.llm.layers.8.attn.gate_attn.weight", + "model.llm.layers.8.attn.k_sconv.weight", + "model.llm.layers.8.attn.q_l2_clip.scale", + "model.llm.layers.8.attn.rel_l2_clip.scale", + "model.llm.layers.8.attn.rel_logits_proj.proj", + "model.llm.layers.8.attn.v_sconv.weight", + "model.llm.layers.8.attn.wk_dv.weight", + "model.llm.layers.8.attn.wo_ud.weight", + "model.llm.layers.8.attn.wq_du.weight", + "model.llm.layers.8.attn.wr_du.weight", + "model.llm.layers.8.attn.wv_dv.weight", + "model.llm.layers.8.attn_norm.weight", + "model.llm.layers.8.attn_sconv.weight", + "model.llm.layers.8.mlp.gate.weight", + "model.llm.layers.8.mlp_norm.weight", + "model.llm.layers.8.mlp_sconv.weight", + "model.llm.layers.9.attn.attn_sink_proj.weight", + "model.llm.layers.9.attn.gate_attn.weight", + "model.llm.layers.9.attn.k_sconv.weight", + "model.llm.layers.9.attn.q_l2_clip.scale", + "model.llm.layers.9.attn.rel_l2_clip.scale", + "model.llm.layers.9.attn.rel_logits_proj.proj", + "model.llm.layers.9.attn.v_sconv.weight", + "model.llm.layers.9.attn.wk_dv.weight", + "model.llm.layers.9.attn.wo_ud.weight", + "model.llm.layers.9.attn.wq_du.weight", + "model.llm.layers.9.attn.wr_du.weight", + "model.llm.layers.9.attn.wv_dv.weight", + "model.llm.layers.9.attn_norm.weight", + "model.llm.layers.9.attn_sconv.weight", + "model.llm.layers.9.mlp.gate.weight", + "model.llm.layers.9.mlp_norm.weight", + "model.llm.layers.9.mlp_sconv.weight", + "model.llm.norm.weight", + "model.visual.backbone.embeddings.class_embedding", + "model.visual.backbone.embeddings.cls_position_embedding", + "model.visual.backbone.embeddings.patch_embedding.weight", + "model.visual.backbone.embeddings.position_embedding", + "model.visual.backbone.layers.0.ln_1.bias", + "model.visual.backbone.layers.0.ln_1.weight", + "model.visual.backbone.layers.0.ln_2.bias", + "model.visual.backbone.layers.0.ln_2.weight", + "model.visual.backbone.layers.0.ls_1.gamma", + "model.visual.backbone.layers.0.ls_2.gamma", + "model.visual.backbone.layers.0.mlp.c_fc.bias", + "model.visual.backbone.layers.0.mlp.c_fc.weight", + "model.visual.backbone.layers.0.mlp.c_proj.bias", + "model.visual.backbone.layers.0.mlp.c_proj.weight", + "model.visual.backbone.layers.0.self_attn.k_proj.bias", + "model.visual.backbone.layers.0.self_attn.k_proj.weight", + "model.visual.backbone.layers.0.self_attn.out_proj.bias", + "model.visual.backbone.layers.0.self_attn.out_proj.weight", + "model.visual.backbone.layers.0.self_attn.q_proj.bias", + "model.visual.backbone.layers.0.self_attn.q_proj.weight", + "model.visual.backbone.layers.0.self_attn.v_proj.bias", + "model.visual.backbone.layers.0.self_attn.v_proj.weight", + "model.visual.backbone.layers.1.ln_1.bias", + "model.visual.backbone.layers.1.ln_1.weight", + "model.visual.backbone.layers.1.ln_2.bias", + "model.visual.backbone.layers.1.ln_2.weight", + "model.visual.backbone.layers.1.ls_1.gamma", + "model.visual.backbone.layers.1.ls_2.gamma", + "model.visual.backbone.layers.1.mlp.c_fc.bias", + "model.visual.backbone.layers.1.mlp.c_fc.weight", + "model.visual.backbone.layers.1.mlp.c_proj.bias", + "model.visual.backbone.layers.1.mlp.c_proj.weight", + "model.visual.backbone.layers.1.self_attn.k_proj.bias", + "model.visual.backbone.layers.1.self_attn.k_proj.weight", + "model.visual.backbone.layers.1.self_attn.out_proj.bias", + "model.visual.backbone.layers.1.self_attn.out_proj.weight", + "model.visual.backbone.layers.1.self_attn.q_proj.bias", + "model.visual.backbone.layers.1.self_attn.q_proj.weight", + "model.visual.backbone.layers.1.self_attn.v_proj.bias", + "model.visual.backbone.layers.1.self_attn.v_proj.weight", + "model.visual.backbone.layers.10.ln_1.bias", + "model.visual.backbone.layers.10.ln_1.weight", + "model.visual.backbone.layers.10.ln_2.bias", + "model.visual.backbone.layers.10.ln_2.weight", + "model.visual.backbone.layers.10.ls_1.gamma", + "model.visual.backbone.layers.10.ls_2.gamma", + "model.visual.backbone.layers.10.mlp.c_fc.bias", + "model.visual.backbone.layers.10.mlp.c_fc.weight", + "model.visual.backbone.layers.10.mlp.c_proj.bias", + "model.visual.backbone.layers.10.mlp.c_proj.weight", + "model.visual.backbone.layers.10.self_attn.k_proj.bias", + "model.visual.backbone.layers.10.self_attn.k_proj.weight", + "model.visual.backbone.layers.10.self_attn.out_proj.bias", + "model.visual.backbone.layers.10.self_attn.out_proj.weight", + "model.visual.backbone.layers.10.self_attn.q_proj.bias", + "model.visual.backbone.layers.10.self_attn.q_proj.weight", + "model.visual.backbone.layers.10.self_attn.v_proj.bias", + "model.visual.backbone.layers.10.self_attn.v_proj.weight", + "model.visual.backbone.layers.11.ln_1.bias", + "model.visual.backbone.layers.11.ln_1.weight", + "model.visual.backbone.layers.11.ln_2.bias", + "model.visual.backbone.layers.11.ln_2.weight", + "model.visual.backbone.layers.11.ls_1.gamma", + "model.visual.backbone.layers.11.ls_2.gamma", + "model.visual.backbone.layers.11.mlp.c_fc.bias", + "model.visual.backbone.layers.11.mlp.c_fc.weight", + "model.visual.backbone.layers.11.mlp.c_proj.bias", + "model.visual.backbone.layers.11.mlp.c_proj.weight", + "model.visual.backbone.layers.11.self_attn.k_proj.bias", + "model.visual.backbone.layers.11.self_attn.k_proj.weight", + "model.visual.backbone.layers.11.self_attn.out_proj.bias", + "model.visual.backbone.layers.11.self_attn.out_proj.weight", + "model.visual.backbone.layers.11.self_attn.q_proj.bias", + "model.visual.backbone.layers.11.self_attn.q_proj.weight", + "model.visual.backbone.layers.11.self_attn.v_proj.bias", + "model.visual.backbone.layers.11.self_attn.v_proj.weight", + "model.visual.backbone.layers.12.ln_1.bias", + "model.visual.backbone.layers.12.ln_1.weight", + "model.visual.backbone.layers.12.ln_2.bias", + "model.visual.backbone.layers.12.ln_2.weight", + "model.visual.backbone.layers.12.ls_1.gamma", + "model.visual.backbone.layers.12.ls_2.gamma", + "model.visual.backbone.layers.12.mlp.c_fc.bias", + "model.visual.backbone.layers.12.mlp.c_fc.weight", + "model.visual.backbone.layers.12.mlp.c_proj.bias", + "model.visual.backbone.layers.12.mlp.c_proj.weight", + "model.visual.backbone.layers.12.self_attn.k_proj.bias", + "model.visual.backbone.layers.12.self_attn.k_proj.weight", + "model.visual.backbone.layers.12.self_attn.out_proj.bias", + "model.visual.backbone.layers.12.self_attn.out_proj.weight", + "model.visual.backbone.layers.12.self_attn.q_proj.bias", + "model.visual.backbone.layers.12.self_attn.q_proj.weight", + "model.visual.backbone.layers.12.self_attn.v_proj.bias", + "model.visual.backbone.layers.12.self_attn.v_proj.weight", + "model.visual.backbone.layers.13.ln_1.bias", + "model.visual.backbone.layers.13.ln_1.weight", + "model.visual.backbone.layers.13.ln_2.bias", + "model.visual.backbone.layers.13.ln_2.weight", + "model.visual.backbone.layers.13.ls_1.gamma", + "model.visual.backbone.layers.13.ls_2.gamma", + "model.visual.backbone.layers.13.mlp.c_fc.bias", + "model.visual.backbone.layers.13.mlp.c_fc.weight", + "model.visual.backbone.layers.13.mlp.c_proj.bias", + "model.visual.backbone.layers.13.mlp.c_proj.weight", + "model.visual.backbone.layers.13.self_attn.k_proj.bias", + "model.visual.backbone.layers.13.self_attn.k_proj.weight", + "model.visual.backbone.layers.13.self_attn.out_proj.bias", + "model.visual.backbone.layers.13.self_attn.out_proj.weight", + "model.visual.backbone.layers.13.self_attn.q_proj.bias", + "model.visual.backbone.layers.13.self_attn.q_proj.weight", + "model.visual.backbone.layers.13.self_attn.v_proj.bias", + "model.visual.backbone.layers.13.self_attn.v_proj.weight", + "model.visual.backbone.layers.14.ln_1.bias", + "model.visual.backbone.layers.14.ln_1.weight", + "model.visual.backbone.layers.14.ln_2.bias", + "model.visual.backbone.layers.14.ln_2.weight", + "model.visual.backbone.layers.14.ls_1.gamma", + "model.visual.backbone.layers.14.ls_2.gamma", + "model.visual.backbone.layers.14.mlp.c_fc.bias", + "model.visual.backbone.layers.14.mlp.c_fc.weight", + "model.visual.backbone.layers.14.mlp.c_proj.bias", + "model.visual.backbone.layers.14.mlp.c_proj.weight", + "model.visual.backbone.layers.14.self_attn.k_proj.bias", + "model.visual.backbone.layers.14.self_attn.k_proj.weight", + "model.visual.backbone.layers.14.self_attn.out_proj.bias", + "model.visual.backbone.layers.14.self_attn.out_proj.weight", + "model.visual.backbone.layers.14.self_attn.q_proj.bias", + "model.visual.backbone.layers.14.self_attn.q_proj.weight", + "model.visual.backbone.layers.14.self_attn.v_proj.bias", + "model.visual.backbone.layers.14.self_attn.v_proj.weight", + "model.visual.backbone.layers.15.ln_1.bias", + "model.visual.backbone.layers.15.ln_1.weight", + "model.visual.backbone.layers.15.ln_2.bias", + "model.visual.backbone.layers.15.ln_2.weight", + "model.visual.backbone.layers.15.ls_1.gamma", + "model.visual.backbone.layers.15.ls_2.gamma", + "model.visual.backbone.layers.15.mlp.c_fc.bias", + "model.visual.backbone.layers.15.mlp.c_fc.weight", + "model.visual.backbone.layers.15.mlp.c_proj.bias", + "model.visual.backbone.layers.15.mlp.c_proj.weight", + "model.visual.backbone.layers.15.self_attn.k_proj.bias", + "model.visual.backbone.layers.15.self_attn.k_proj.weight", + "model.visual.backbone.layers.15.self_attn.out_proj.bias", + "model.visual.backbone.layers.15.self_attn.out_proj.weight", + "model.visual.backbone.layers.15.self_attn.q_proj.bias", + "model.visual.backbone.layers.15.self_attn.q_proj.weight", + "model.visual.backbone.layers.15.self_attn.v_proj.bias", + "model.visual.backbone.layers.15.self_attn.v_proj.weight", + "model.visual.backbone.layers.16.ln_1.bias", + "model.visual.backbone.layers.16.ln_1.weight", + "model.visual.backbone.layers.16.ln_2.bias", + "model.visual.backbone.layers.16.ln_2.weight", + "model.visual.backbone.layers.16.ls_1.gamma", + "model.visual.backbone.layers.16.ls_2.gamma", + "model.visual.backbone.layers.16.mlp.c_fc.bias", + "model.visual.backbone.layers.16.mlp.c_fc.weight", + "model.visual.backbone.layers.16.mlp.c_proj.bias", + "model.visual.backbone.layers.16.mlp.c_proj.weight", + "model.visual.backbone.layers.16.self_attn.k_proj.bias", + "model.visual.backbone.layers.16.self_attn.k_proj.weight", + "model.visual.backbone.layers.16.self_attn.out_proj.bias", + "model.visual.backbone.layers.16.self_attn.out_proj.weight", + "model.visual.backbone.layers.16.self_attn.q_proj.bias", + "model.visual.backbone.layers.16.self_attn.q_proj.weight", + "model.visual.backbone.layers.16.self_attn.v_proj.bias", + "model.visual.backbone.layers.16.self_attn.v_proj.weight", + "model.visual.backbone.layers.17.ln_1.bias", + "model.visual.backbone.layers.17.ln_1.weight", + "model.visual.backbone.layers.17.ln_2.bias", + "model.visual.backbone.layers.17.ln_2.weight", + "model.visual.backbone.layers.17.ls_1.gamma", + "model.visual.backbone.layers.17.ls_2.gamma", + "model.visual.backbone.layers.17.mlp.c_fc.bias", + "model.visual.backbone.layers.17.mlp.c_fc.weight", + "model.visual.backbone.layers.17.mlp.c_proj.bias", + "model.visual.backbone.layers.17.mlp.c_proj.weight", + "model.visual.backbone.layers.17.self_attn.k_proj.bias", + "model.visual.backbone.layers.17.self_attn.k_proj.weight", + "model.visual.backbone.layers.17.self_attn.out_proj.bias", + "model.visual.backbone.layers.17.self_attn.out_proj.weight", + "model.visual.backbone.layers.17.self_attn.q_proj.bias", + "model.visual.backbone.layers.17.self_attn.q_proj.weight", + "model.visual.backbone.layers.17.self_attn.v_proj.bias", + "model.visual.backbone.layers.17.self_attn.v_proj.weight", + "model.visual.backbone.layers.18.ln_1.bias", + "model.visual.backbone.layers.18.ln_1.weight", + "model.visual.backbone.layers.18.ln_2.bias", + "model.visual.backbone.layers.18.ln_2.weight", + "model.visual.backbone.layers.18.ls_1.gamma", + "model.visual.backbone.layers.18.ls_2.gamma", + "model.visual.backbone.layers.18.mlp.c_fc.bias", + "model.visual.backbone.layers.18.mlp.c_fc.weight", + "model.visual.backbone.layers.18.mlp.c_proj.bias", + "model.visual.backbone.layers.18.mlp.c_proj.weight", + "model.visual.backbone.layers.18.self_attn.k_proj.bias", + "model.visual.backbone.layers.18.self_attn.k_proj.weight", + "model.visual.backbone.layers.18.self_attn.out_proj.bias", + "model.visual.backbone.layers.18.self_attn.out_proj.weight", + "model.visual.backbone.layers.18.self_attn.q_proj.bias", + "model.visual.backbone.layers.18.self_attn.q_proj.weight", + "model.visual.backbone.layers.18.self_attn.v_proj.bias", + "model.visual.backbone.layers.18.self_attn.v_proj.weight", + "model.visual.backbone.layers.19.ln_1.bias", + "model.visual.backbone.layers.19.ln_1.weight", + "model.visual.backbone.layers.19.ln_2.bias", + "model.visual.backbone.layers.19.ln_2.weight", + "model.visual.backbone.layers.19.ls_1.gamma", + "model.visual.backbone.layers.19.ls_2.gamma", + "model.visual.backbone.layers.19.mlp.c_fc.bias", + "model.visual.backbone.layers.19.mlp.c_fc.weight", + "model.visual.backbone.layers.19.mlp.c_proj.bias", + "model.visual.backbone.layers.19.mlp.c_proj.weight", + "model.visual.backbone.layers.19.self_attn.k_proj.bias", + "model.visual.backbone.layers.19.self_attn.k_proj.weight", + "model.visual.backbone.layers.19.self_attn.out_proj.bias", + "model.visual.backbone.layers.19.self_attn.out_proj.weight", + "model.visual.backbone.layers.19.self_attn.q_proj.bias", + "model.visual.backbone.layers.19.self_attn.q_proj.weight", + "model.visual.backbone.layers.19.self_attn.v_proj.bias", + "model.visual.backbone.layers.19.self_attn.v_proj.weight", + "model.visual.backbone.layers.2.ln_1.bias", + "model.visual.backbone.layers.2.ln_1.weight", + "model.visual.backbone.layers.2.ln_2.bias", + "model.visual.backbone.layers.2.ln_2.weight", + "model.visual.backbone.layers.2.ls_1.gamma", + "model.visual.backbone.layers.2.ls_2.gamma", + "model.visual.backbone.layers.2.mlp.c_fc.bias", + "model.visual.backbone.layers.2.mlp.c_fc.weight", + "model.visual.backbone.layers.2.mlp.c_proj.bias", + "model.visual.backbone.layers.2.mlp.c_proj.weight", + "model.visual.backbone.layers.2.self_attn.k_proj.bias", + "model.visual.backbone.layers.2.self_attn.k_proj.weight", + "model.visual.backbone.layers.2.self_attn.out_proj.bias", + "model.visual.backbone.layers.2.self_attn.out_proj.weight", + "model.visual.backbone.layers.2.self_attn.q_proj.bias", + "model.visual.backbone.layers.2.self_attn.q_proj.weight", + "model.visual.backbone.layers.2.self_attn.v_proj.bias", + "model.visual.backbone.layers.2.self_attn.v_proj.weight", + "model.visual.backbone.layers.20.ln_1.bias", + "model.visual.backbone.layers.20.ln_1.weight", + "model.visual.backbone.layers.20.ln_2.bias", + "model.visual.backbone.layers.20.ln_2.weight", + "model.visual.backbone.layers.20.ls_1.gamma", + "model.visual.backbone.layers.20.ls_2.gamma", + "model.visual.backbone.layers.20.mlp.c_fc.bias", + "model.visual.backbone.layers.20.mlp.c_fc.weight", + "model.visual.backbone.layers.20.mlp.c_proj.bias", + "model.visual.backbone.layers.20.mlp.c_proj.weight", + "model.visual.backbone.layers.20.self_attn.k_proj.bias", + "model.visual.backbone.layers.20.self_attn.k_proj.weight", + "model.visual.backbone.layers.20.self_attn.out_proj.bias", + "model.visual.backbone.layers.20.self_attn.out_proj.weight", + "model.visual.backbone.layers.20.self_attn.q_proj.bias", + "model.visual.backbone.layers.20.self_attn.q_proj.weight", + "model.visual.backbone.layers.20.self_attn.v_proj.bias", + "model.visual.backbone.layers.20.self_attn.v_proj.weight", + "model.visual.backbone.layers.21.ln_1.bias", + "model.visual.backbone.layers.21.ln_1.weight", + "model.visual.backbone.layers.21.ln_2.bias", + "model.visual.backbone.layers.21.ln_2.weight", + "model.visual.backbone.layers.21.ls_1.gamma", + "model.visual.backbone.layers.21.ls_2.gamma", + "model.visual.backbone.layers.21.mlp.c_fc.bias", + "model.visual.backbone.layers.21.mlp.c_fc.weight", + "model.visual.backbone.layers.21.mlp.c_proj.bias", + "model.visual.backbone.layers.21.mlp.c_proj.weight", + "model.visual.backbone.layers.21.self_attn.k_proj.bias", + "model.visual.backbone.layers.21.self_attn.k_proj.weight", + "model.visual.backbone.layers.21.self_attn.out_proj.bias", + "model.visual.backbone.layers.21.self_attn.out_proj.weight", + "model.visual.backbone.layers.21.self_attn.q_proj.bias", + "model.visual.backbone.layers.21.self_attn.q_proj.weight", + "model.visual.backbone.layers.21.self_attn.v_proj.bias", + "model.visual.backbone.layers.21.self_attn.v_proj.weight", + "model.visual.backbone.layers.22.ln_1.bias", + "model.visual.backbone.layers.22.ln_1.weight", + "model.visual.backbone.layers.22.ln_2.bias", + "model.visual.backbone.layers.22.ln_2.weight", + "model.visual.backbone.layers.22.ls_1.gamma", + "model.visual.backbone.layers.22.ls_2.gamma", + "model.visual.backbone.layers.22.mlp.c_fc.bias", + "model.visual.backbone.layers.22.mlp.c_fc.weight", + "model.visual.backbone.layers.22.mlp.c_proj.bias", + "model.visual.backbone.layers.22.mlp.c_proj.weight", + "model.visual.backbone.layers.22.self_attn.k_proj.bias", + "model.visual.backbone.layers.22.self_attn.k_proj.weight", + "model.visual.backbone.layers.22.self_attn.out_proj.bias", + "model.visual.backbone.layers.22.self_attn.out_proj.weight", + "model.visual.backbone.layers.22.self_attn.q_proj.bias", + "model.visual.backbone.layers.22.self_attn.q_proj.weight", + "model.visual.backbone.layers.22.self_attn.v_proj.bias", + "model.visual.backbone.layers.22.self_attn.v_proj.weight", + "model.visual.backbone.layers.3.ln_1.bias", + "model.visual.backbone.layers.3.ln_1.weight", + "model.visual.backbone.layers.3.ln_2.bias", + "model.visual.backbone.layers.3.ln_2.weight", + "model.visual.backbone.layers.3.ls_1.gamma", + "model.visual.backbone.layers.3.ls_2.gamma", + "model.visual.backbone.layers.3.mlp.c_fc.bias", + "model.visual.backbone.layers.3.mlp.c_fc.weight", + "model.visual.backbone.layers.3.mlp.c_proj.bias", + "model.visual.backbone.layers.3.mlp.c_proj.weight", + "model.visual.backbone.layers.3.self_attn.k_proj.bias", + "model.visual.backbone.layers.3.self_attn.k_proj.weight", + "model.visual.backbone.layers.3.self_attn.out_proj.bias", + "model.visual.backbone.layers.3.self_attn.out_proj.weight", + "model.visual.backbone.layers.3.self_attn.q_proj.bias", + "model.visual.backbone.layers.3.self_attn.q_proj.weight", + "model.visual.backbone.layers.3.self_attn.v_proj.bias", + "model.visual.backbone.layers.3.self_attn.v_proj.weight", + "model.visual.backbone.layers.4.ln_1.bias", + "model.visual.backbone.layers.4.ln_1.weight", + "model.visual.backbone.layers.4.ln_2.bias", + "model.visual.backbone.layers.4.ln_2.weight", + "model.visual.backbone.layers.4.ls_1.gamma", + "model.visual.backbone.layers.4.ls_2.gamma", + "model.visual.backbone.layers.4.mlp.c_fc.bias", + "model.visual.backbone.layers.4.mlp.c_fc.weight", + "model.visual.backbone.layers.4.mlp.c_proj.bias", + "model.visual.backbone.layers.4.mlp.c_proj.weight", + "model.visual.backbone.layers.4.self_attn.k_proj.bias", + "model.visual.backbone.layers.4.self_attn.k_proj.weight", + "model.visual.backbone.layers.4.self_attn.out_proj.bias", + "model.visual.backbone.layers.4.self_attn.out_proj.weight", + "model.visual.backbone.layers.4.self_attn.q_proj.bias", + "model.visual.backbone.layers.4.self_attn.q_proj.weight", + "model.visual.backbone.layers.4.self_attn.v_proj.bias", + "model.visual.backbone.layers.4.self_attn.v_proj.weight", + "model.visual.backbone.layers.5.ln_1.bias", + "model.visual.backbone.layers.5.ln_1.weight", + "model.visual.backbone.layers.5.ln_2.bias", + "model.visual.backbone.layers.5.ln_2.weight", + "model.visual.backbone.layers.5.ls_1.gamma", + "model.visual.backbone.layers.5.ls_2.gamma", + "model.visual.backbone.layers.5.mlp.c_fc.bias", + "model.visual.backbone.layers.5.mlp.c_fc.weight", + "model.visual.backbone.layers.5.mlp.c_proj.bias", + "model.visual.backbone.layers.5.mlp.c_proj.weight", + "model.visual.backbone.layers.5.self_attn.k_proj.bias", + "model.visual.backbone.layers.5.self_attn.k_proj.weight", + "model.visual.backbone.layers.5.self_attn.out_proj.bias", + "model.visual.backbone.layers.5.self_attn.out_proj.weight", + "model.visual.backbone.layers.5.self_attn.q_proj.bias", + "model.visual.backbone.layers.5.self_attn.q_proj.weight", + "model.visual.backbone.layers.5.self_attn.v_proj.bias", + "model.visual.backbone.layers.5.self_attn.v_proj.weight", + "model.visual.backbone.layers.6.ln_1.bias", + "model.visual.backbone.layers.6.ln_1.weight", + "model.visual.backbone.layers.6.ln_2.bias", + "model.visual.backbone.layers.6.ln_2.weight", + "model.visual.backbone.layers.6.ls_1.gamma", + "model.visual.backbone.layers.6.ls_2.gamma", + "model.visual.backbone.layers.6.mlp.c_fc.bias", + "model.visual.backbone.layers.6.mlp.c_fc.weight", + "model.visual.backbone.layers.6.mlp.c_proj.bias", + "model.visual.backbone.layers.6.mlp.c_proj.weight", + "model.visual.backbone.layers.6.self_attn.k_proj.bias", + "model.visual.backbone.layers.6.self_attn.k_proj.weight", + "model.visual.backbone.layers.6.self_attn.out_proj.bias", + "model.visual.backbone.layers.6.self_attn.out_proj.weight", + "model.visual.backbone.layers.6.self_attn.q_proj.bias", + "model.visual.backbone.layers.6.self_attn.q_proj.weight", + "model.visual.backbone.layers.6.self_attn.v_proj.bias", + "model.visual.backbone.layers.6.self_attn.v_proj.weight", + "model.visual.backbone.layers.7.ln_1.bias", + "model.visual.backbone.layers.7.ln_1.weight", + "model.visual.backbone.layers.7.ln_2.bias", + "model.visual.backbone.layers.7.ln_2.weight", + "model.visual.backbone.layers.7.ls_1.gamma", + "model.visual.backbone.layers.7.ls_2.gamma", + "model.visual.backbone.layers.7.mlp.c_fc.bias", + "model.visual.backbone.layers.7.mlp.c_fc.weight", + "model.visual.backbone.layers.7.mlp.c_proj.bias", + "model.visual.backbone.layers.7.mlp.c_proj.weight", + "model.visual.backbone.layers.7.self_attn.k_proj.bias", + "model.visual.backbone.layers.7.self_attn.k_proj.weight", + "model.visual.backbone.layers.7.self_attn.out_proj.bias", + "model.visual.backbone.layers.7.self_attn.out_proj.weight", + "model.visual.backbone.layers.7.self_attn.q_proj.bias", + "model.visual.backbone.layers.7.self_attn.q_proj.weight", + "model.visual.backbone.layers.7.self_attn.v_proj.bias", + "model.visual.backbone.layers.7.self_attn.v_proj.weight", + "model.visual.backbone.layers.8.ln_1.bias", + "model.visual.backbone.layers.8.ln_1.weight", + "model.visual.backbone.layers.8.ln_2.bias", + "model.visual.backbone.layers.8.ln_2.weight", + "model.visual.backbone.layers.8.ls_1.gamma", + "model.visual.backbone.layers.8.ls_2.gamma", + "model.visual.backbone.layers.8.mlp.c_fc.bias", + "model.visual.backbone.layers.8.mlp.c_fc.weight", + "model.visual.backbone.layers.8.mlp.c_proj.bias", + "model.visual.backbone.layers.8.mlp.c_proj.weight", + "model.visual.backbone.layers.8.self_attn.k_proj.bias", + "model.visual.backbone.layers.8.self_attn.k_proj.weight", + "model.visual.backbone.layers.8.self_attn.out_proj.bias", + "model.visual.backbone.layers.8.self_attn.out_proj.weight", + "model.visual.backbone.layers.8.self_attn.q_proj.bias", + "model.visual.backbone.layers.8.self_attn.q_proj.weight", + "model.visual.backbone.layers.8.self_attn.v_proj.bias", + "model.visual.backbone.layers.8.self_attn.v_proj.weight", + "model.visual.backbone.layers.9.ln_1.bias", + "model.visual.backbone.layers.9.ln_1.weight", + "model.visual.backbone.layers.9.ln_2.bias", + "model.visual.backbone.layers.9.ln_2.weight", + "model.visual.backbone.layers.9.ls_1.gamma", + "model.visual.backbone.layers.9.ls_2.gamma", + "model.visual.backbone.layers.9.mlp.c_fc.bias", + "model.visual.backbone.layers.9.mlp.c_fc.weight", + "model.visual.backbone.layers.9.mlp.c_proj.bias", + "model.visual.backbone.layers.9.mlp.c_proj.weight", + "model.visual.backbone.layers.9.self_attn.k_proj.bias", + "model.visual.backbone.layers.9.self_attn.k_proj.weight", + "model.visual.backbone.layers.9.self_attn.out_proj.bias", + "model.visual.backbone.layers.9.self_attn.out_proj.weight", + "model.visual.backbone.layers.9.self_attn.q_proj.bias", + "model.visual.backbone.layers.9.self_attn.q_proj.weight", + "model.visual.backbone.layers.9.self_attn.v_proj.bias", + "model.visual.backbone.layers.9.self_attn.v_proj.weight", + "model.visual.backbone.ln_pre.bias", + "model.visual.backbone.ln_pre.weight", + "model.visual.proj_in.weight", + "model.visual.proj_out.weight" + ], + "weight_block_shape": [], + "weight_transform": "cast" + }, + { + "compute_dtype": "fp8", + "output_dtype": "float8_e4m3fnuz", + "required_activation_quantization": { + "format": "float8_e4m3fnuz", + "granularity": "per_token", + "scale_dtype": "float32", + "rounding": "nearest" + }, + "tensors": [ + "model.llm.layers.0.mlp.experts.w13_weight", + "model.llm.layers.0.mlp.experts.w2_weight", + "model.llm.layers.1.mlp.experts.w13_weight", + "model.llm.layers.1.mlp.experts.w2_weight", + "model.llm.layers.10.mlp.experts.w13_weight", + "model.llm.layers.10.mlp.experts.w2_weight", + "model.llm.layers.11.mlp.experts.w13_weight", + "model.llm.layers.11.mlp.experts.w2_weight", + "model.llm.layers.12.mlp.experts.w13_weight", + "model.llm.layers.12.mlp.experts.w2_weight", + "model.llm.layers.13.mlp.experts.w13_weight", + "model.llm.layers.13.mlp.experts.w2_weight", + "model.llm.layers.14.mlp.experts.w13_weight", + "model.llm.layers.14.mlp.experts.w2_weight", + "model.llm.layers.15.mlp.experts.w13_weight", + "model.llm.layers.15.mlp.experts.w2_weight", + "model.llm.layers.16.mlp.experts.w13_weight", + "model.llm.layers.16.mlp.experts.w2_weight", + "model.llm.layers.17.mlp.experts.w13_weight", + "model.llm.layers.17.mlp.experts.w2_weight", + "model.llm.layers.18.mlp.experts.w13_weight", + "model.llm.layers.18.mlp.experts.w2_weight", + "model.llm.layers.19.mlp.experts.w13_weight", + "model.llm.layers.19.mlp.experts.w2_weight", + "model.llm.layers.2.mlp.experts.w13_weight", + "model.llm.layers.2.mlp.experts.w2_weight", + "model.llm.layers.20.mlp.experts.w13_weight", + "model.llm.layers.20.mlp.experts.w2_weight", + "model.llm.layers.21.mlp.experts.w13_weight", + "model.llm.layers.21.mlp.experts.w2_weight", + "model.llm.layers.22.mlp.experts.w13_weight", + "model.llm.layers.22.mlp.experts.w2_weight", + "model.llm.layers.23.mlp.experts.w13_weight", + "model.llm.layers.23.mlp.experts.w2_weight", + "model.llm.layers.24.mlp.experts.w13_weight", + "model.llm.layers.24.mlp.experts.w2_weight", + "model.llm.layers.25.mlp.experts.w13_weight", + "model.llm.layers.25.mlp.experts.w2_weight", + "model.llm.layers.26.mlp.experts.w13_weight", + "model.llm.layers.26.mlp.experts.w2_weight", + "model.llm.layers.27.mlp.experts.w13_weight", + "model.llm.layers.27.mlp.experts.w2_weight", + "model.llm.layers.28.mlp.experts.w13_weight", + "model.llm.layers.28.mlp.experts.w2_weight", + "model.llm.layers.29.mlp.experts.w13_weight", + "model.llm.layers.29.mlp.experts.w2_weight", + "model.llm.layers.3.mlp.experts.w13_weight", + "model.llm.layers.3.mlp.experts.w2_weight", + "model.llm.layers.30.mlp.experts.w13_weight", + "model.llm.layers.30.mlp.experts.w2_weight", + "model.llm.layers.31.mlp.experts.w13_weight", + "model.llm.layers.31.mlp.experts.w2_weight", + "model.llm.layers.32.mlp.experts.w13_weight", + "model.llm.layers.32.mlp.experts.w2_weight", + "model.llm.layers.33.mlp.experts.w13_weight", + "model.llm.layers.33.mlp.experts.w2_weight", + "model.llm.layers.34.mlp.experts.w13_weight", + "model.llm.layers.34.mlp.experts.w2_weight", + "model.llm.layers.35.mlp.experts.w13_weight", + "model.llm.layers.35.mlp.experts.w2_weight", + "model.llm.layers.36.mlp.experts.w13_weight", + "model.llm.layers.36.mlp.experts.w2_weight", + "model.llm.layers.37.mlp.experts.w13_weight", + "model.llm.layers.37.mlp.experts.w2_weight", + "model.llm.layers.38.mlp.experts.w13_weight", + "model.llm.layers.38.mlp.experts.w2_weight", + "model.llm.layers.39.mlp.experts.w13_weight", + "model.llm.layers.39.mlp.experts.w2_weight", + "model.llm.layers.4.mlp.experts.w13_weight", + "model.llm.layers.4.mlp.experts.w2_weight", + "model.llm.layers.40.mlp.experts.w13_weight", + "model.llm.layers.40.mlp.experts.w2_weight", + "model.llm.layers.41.mlp.experts.w13_weight", + "model.llm.layers.41.mlp.experts.w2_weight", + "model.llm.layers.42.mlp.experts.w13_weight", + "model.llm.layers.42.mlp.experts.w2_weight", + "model.llm.layers.43.mlp.experts.w13_weight", + "model.llm.layers.43.mlp.experts.w2_weight", + "model.llm.layers.44.mlp.experts.w13_weight", + "model.llm.layers.44.mlp.experts.w2_weight", + "model.llm.layers.45.mlp.experts.w13_weight", + "model.llm.layers.45.mlp.experts.w2_weight", + "model.llm.layers.46.mlp.experts.w13_weight", + "model.llm.layers.46.mlp.experts.w2_weight", + "model.llm.layers.47.mlp.experts.w13_weight", + "model.llm.layers.47.mlp.experts.w2_weight", + "model.llm.layers.5.mlp.experts.w13_weight", + "model.llm.layers.5.mlp.experts.w2_weight", + "model.llm.layers.6.mlp.experts.w13_weight", + "model.llm.layers.6.mlp.experts.w2_weight", + "model.llm.layers.7.mlp.experts.w13_weight", + "model.llm.layers.7.mlp.experts.w2_weight", + "model.llm.layers.8.mlp.experts.w13_weight", + "model.llm.layers.8.mlp.experts.w2_weight", + "model.llm.layers.9.mlp.experts.w13_weight", + "model.llm.layers.9.mlp.experts.w2_weight" + ], + "weight_block_shape": [], + "weight_transform": "row_fp8_e4m3fnuz_max_240" + }, + { + "compute_dtype": "bfloat16", + "fwd_output_dtype": "float32", + "output_dtype": "bfloat16", + "required_activation_quantization": null, + "tensors": [ + "model.llm.unembed.weight" + ], + "weight_block_shape": [], + "weight_transform": "cast" + } + ] + }, + "mtp": { + "parameter_groups": [ + { + "compute_dtype": "bfloat16", + "output_dtype": "bfloat16", + "required_activation_quantization": null, + "tensors": [ + "fuse_weight", + "layers.0.dflare_norm.weight", + "layers.0.input_layernorm.weight", + "layers.0.mlp.down_proj.weight", + "layers.0.mlp.gate_proj.weight", + "layers.0.mlp.up_proj.weight", + "layers.0.post_attention_layernorm.weight", + "layers.0.self_attn.k_norm.weight", + "layers.0.self_attn.k_norm_ctx.weight", + "layers.0.self_attn.k_proj.weight", + "layers.0.self_attn.k_proj_ctx.weight", + "layers.0.self_attn.o_proj.weight", + "layers.0.self_attn.q_norm.weight", + "layers.0.self_attn.q_proj.weight", + "layers.0.self_attn.v_proj.weight", + "layers.0.self_attn.v_proj_ctx.weight", + "layers.1.dflare_norm.weight", + "layers.1.input_layernorm.weight", + "layers.1.mlp.down_proj.weight", + "layers.1.mlp.gate_proj.weight", + "layers.1.mlp.up_proj.weight", + "layers.1.post_attention_layernorm.weight", + "layers.1.self_attn.k_norm.weight", + "layers.1.self_attn.k_norm_ctx.weight", + "layers.1.self_attn.k_proj.weight", + "layers.1.self_attn.k_proj_ctx.weight", + "layers.1.self_attn.o_proj.weight", + "layers.1.self_attn.q_norm.weight", + "layers.1.self_attn.q_proj.weight", + "layers.1.self_attn.v_proj.weight", + "layers.1.self_attn.v_proj_ctx.weight", + "layers.2.dflare_norm.weight", + "layers.2.input_layernorm.weight", + "layers.2.mlp.down_proj.weight", + "layers.2.mlp.gate_proj.weight", + "layers.2.mlp.up_proj.weight", + "layers.2.post_attention_layernorm.weight", + "layers.2.self_attn.k_norm.weight", + "layers.2.self_attn.k_norm_ctx.weight", + "layers.2.self_attn.k_proj.weight", + "layers.2.self_attn.k_proj_ctx.weight", + "layers.2.self_attn.o_proj.weight", + "layers.2.self_attn.q_norm.weight", + "layers.2.self_attn.q_proj.weight", + "layers.2.self_attn.v_proj.weight", + "layers.2.self_attn.v_proj_ctx.weight", + "layers.3.dflare_norm.weight", + "layers.3.input_layernorm.weight", + "layers.3.mlp.down_proj.weight", + "layers.3.mlp.gate_proj.weight", + "layers.3.mlp.up_proj.weight", + "layers.3.post_attention_layernorm.weight", + "layers.3.self_attn.k_norm.weight", + "layers.3.self_attn.k_norm_ctx.weight", + "layers.3.self_attn.k_proj.weight", + "layers.3.self_attn.k_proj_ctx.weight", + "layers.3.self_attn.o_proj.weight", + "layers.3.self_attn.q_norm.weight", + "layers.3.self_attn.q_proj.weight", + "layers.3.self_attn.v_proj.weight", + "layers.3.self_attn.v_proj_ctx.weight", + "layers.4.dflare_norm.weight", + "layers.4.input_layernorm.weight", + "layers.4.mlp.down_proj.weight", + "layers.4.mlp.gate_proj.weight", + "layers.4.mlp.up_proj.weight", + "layers.4.post_attention_layernorm.weight", + "layers.4.self_attn.k_norm.weight", + "layers.4.self_attn.k_norm_ctx.weight", + "layers.4.self_attn.k_proj.weight", + "layers.4.self_attn.k_proj_ctx.weight", + "layers.4.self_attn.o_proj.weight", + "layers.4.self_attn.q_norm.weight", + "layers.4.self_attn.q_proj.weight", + "layers.4.self_attn.v_proj.weight", + "layers.4.self_attn.v_proj_ctx.weight", + "markov_head.dflare_norm.weight", + "markov_head.hidden_out.bias", + "markov_head.hidden_out.weight", + "markov_head.markov_w1.weight", + "markov_head.rnn_proj.bias", + "markov_head.rnn_proj.weight", + "mask_embed", + "norm.weight" + ], + "weight_block_shape": [], + "weight_transform": "cast" + } + ] + }, + "quant_method": "inkling2", + "expert_weight_quantization": { + "format": "float8_e4m3fnuz", + "granularity": "per_output_channel", + "scale_dtype": "float32", + "scale_tensor_suffix": "_scale" + } +} diff --git a/model-00001-of-00048.safetensors b/model-00001-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..c010c9c5738c4cc044176c38d0d00fc4a8e0b099 --- /dev/null +++ b/model-00001-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7fad470eaec8bad8f1dc777d99568acb3d1ed565955220983defc4dc3dacdd4e +size 14495803536 diff --git a/model-00002-of-00048.safetensors b/model-00002-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..9106a2f351614d225a9aefff0fb47b3574a4c4e1 --- /dev/null +++ b/model-00002-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8a5d1e2f0446f6862c67a4d0745e9c34f8e077836fb2d6f47c47cb39ccc9863c +size 12444708336 diff --git a/model-00003-of-00048.safetensors b/model-00003-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..870bd57f23ca0a5ded93a4566dc5c386c8fbf15c --- /dev/null +++ b/model-00003-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d75382147057d2b48ab9a3cc526b3042cf2957056c378987fc86db432ca6ca61 +size 12445298408 diff --git a/model-00004-of-00048.safetensors b/model-00004-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..8c5486390b97bd4494c177f510631ad8ed822810 --- /dev/null +++ b/model-00004-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b12cdfb7fb1e9690b594403b6a7b7e8f701e842a1f4c076a9854bc5cadca0e83 +size 12444727000 diff --git a/model-00005-of-00048.safetensors b/model-00005-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..b09896aa57b81d59d6ca4c6f138d0cde5e2603a7 --- /dev/null +++ b/model-00005-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1ae0308a826ab439da3acf2fd6318ccc3888eeb11b6446e89fecebd26be40985 +size 12444708344 diff --git a/model-00006-of-00048.safetensors b/model-00006-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..63821fadd4d425fdd61a70cee24feb18de650042 --- /dev/null +++ b/model-00006-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:c0e016445e5f2a1ef0308ee0fa8855ca1938d79a89a653e69893bb078ca77023 +size 12444708344 diff --git a/model-00007-of-00048.safetensors b/model-00007-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..041d29ab01d6ea7c22827484c9364fa1de1c1140 --- /dev/null +++ b/model-00007-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:43a0fbe2b0cec6eb13863636ecee97c11eb71648747a98ce2fd1a991cc4e7e0d +size 12444708344 diff --git a/model-00008-of-00048.safetensors b/model-00008-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..72f45f91e5cd9abb99025bb938e19e30730a5c12 --- /dev/null +++ b/model-00008-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:11c6daa9875487c02fd4245246a368a7ec35c2f26542b4666ff617d446c4c5e1 +size 12444708344 diff --git a/model-00009-of-00048.safetensors b/model-00009-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..f11ccd0d5211b969f1f53c8d61b2133becf456a9 --- /dev/null +++ b/model-00009-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b75a5ae0d5c9ea81dc72c6c6df74b0c343e4c90d6a6dcf3697a6e433cfefc831 +size 12445298408 diff --git a/model-00010-of-00048.safetensors b/model-00010-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..52e4ff77ee04019e7a2d263607457fea12aada62 --- /dev/null +++ b/model-00010-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9bd169e1a157d2cbd780b62933eef58c454735811653a11b1236e66529889cc7 +size 12444727000 diff --git a/model-00011-of-00048.safetensors b/model-00011-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..b11e10d6a97fc0c2faee22eb2f5971d7eeac3ca6 --- /dev/null +++ b/model-00011-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:98b2bd93ffd7f3d81771059d15032ce442b6c700fb73322fa4a3be87d67d4986 +size 12444708344 diff --git a/model-00012-of-00048.safetensors b/model-00012-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..a4cdd15ad68e124f3603f2c4cd1be054c6a51d9e --- /dev/null +++ b/model-00012-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4f4964173c9ff4e1860867f88a098bc775447f2866da0bd3b4fbaa6c74b0e772 +size 12444708336 diff --git a/model-00013-of-00048.safetensors b/model-00013-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..50dffd287d91ca56e3a7fec362bd3011c408d94d --- /dev/null +++ b/model-00013-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6e1d8647d5c88a35caa470b84d63762ec1318fed6a29f13e435b2997e61326fc +size 12444708336 diff --git a/model-00014-of-00048.safetensors b/model-00014-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..48654c546b145b90347c32d40fd491ba9bdb441a --- /dev/null +++ b/model-00014-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9484aa75dc39578d66f9a6f6cb42f190877b6f6a8ba9902f2ac76e890c01a7b0 +size 12444708344 diff --git a/model-00015-of-00048.safetensors b/model-00015-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..92ddbb10862aaaff7f54dff6bd6081ac7bae13e3 --- /dev/null +++ b/model-00015-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:1298864e8bd1e0ebaa7d0371df01042efb54e4a46b50d81b1c94e2d0823e58f9 +size 12444708344 diff --git a/model-00016-of-00048.safetensors b/model-00016-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..67cc6e81faa6438d5b97e68119e98ef34f31b72e --- /dev/null +++ b/model-00016-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:746a09951ce0083ca26747915125318e9bdcd27454544a46aeb29278f5f7ef17 +size 12445298408 diff --git a/model-00017-of-00048.safetensors b/model-00017-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..db5eb09051f80aed35426c8c1621b7cf4bac4cd7 --- /dev/null +++ b/model-00017-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:35afebd2497665036f6720a044ba97ac63788f6a38a50db612e4262d3df873fc +size 12444727000 diff --git a/model-00018-of-00048.safetensors b/model-00018-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..e3931bd67a52d3cfad522a7df8504563794a8de6 --- /dev/null +++ b/model-00018-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:fbba2995112f808268e2026b3516d7af710dbce59f7a1ce4e2f71babac5158af +size 12444708344 diff --git a/model-00019-of-00048.safetensors b/model-00019-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..f41aaf34021d8babc3ed282a060fa8b3f12f91ae --- /dev/null +++ b/model-00019-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:69c113a58920c9ed629811be3346e5bd748b471e0abb39bd39f4e1085658d60b +size 12444708344 diff --git a/model-00020-of-00048.safetensors b/model-00020-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..916eb3929787d6226ea3e868c42c534b4ac3d305 --- /dev/null +++ b/model-00020-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:9fd40234ebafcfbb6f79032f3cc75e01b22a6024890f39196c233af96d1329a4 +size 12444708344 diff --git a/model-00021-of-00048.safetensors b/model-00021-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..8d62d8ef8e2b748806cb5b84623ca18e3fe82e69 --- /dev/null +++ b/model-00021-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ade09a0fb2bc900fedccd08b22b64c97f8887155b96c8029e49930f3bd325fe4 +size 12444708344 diff --git a/model-00022-of-00048.safetensors b/model-00022-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..da550a4bc4f60aa175c930da304e9ff9ba837df8 --- /dev/null +++ b/model-00022-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7f0b18702437737f03f72f1f47ac23e409c18c8eea44f8cd9e439bd914d4c4aa +size 12445298408 diff --git a/model-00023-of-00048.safetensors b/model-00023-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..a53eb10885d80d561f8499ac061b51310f6247dc --- /dev/null +++ b/model-00023-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4997343c248395463e6818f247bedbe5a9569f055b233646d16a8755a702978b +size 12444726984 diff --git a/model-00024-of-00048.safetensors b/model-00024-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..8be8aa7f19a9727cc938c80acdffe3d99790b294 --- /dev/null +++ b/model-00024-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:2e4757b0b9fffbb32cd6b89910db636462e52f3229131073539d7293471b09ca +size 12444708336 diff --git a/model-00025-of-00048.safetensors b/model-00025-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..36972da4c28e1bf00d13f5f141abe043908ad1fb --- /dev/null +++ b/model-00025-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:73fffa315775b2bea3adb506655540274aed45b64bf2390dca1cd1333f1a839c +size 12444708344 diff --git a/model-00026-of-00048.safetensors b/model-00026-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..3334411b3fc70ec02b40a494c620e171482232be --- /dev/null +++ b/model-00026-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:8fb4572a2ef1f76188d29bbe73dda86e6bf099b0a24b6a77a418b7dc944cedff +size 12444708344 diff --git a/model-00027-of-00048.safetensors b/model-00027-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..c67626ca7ab4bf951fb89d58e872df9ba9f5b296 --- /dev/null +++ b/model-00027-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:32b72d43d105895068ac9b50428f9e942067d52b23bef7aade025733ad3701ea +size 12444708344 diff --git a/model-00028-of-00048.safetensors b/model-00028-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..48337694434874ca76a84ab8e4ccc3689b84762f --- /dev/null +++ b/model-00028-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:84e235753a6356a1047d8e7aea875d8da071bd995fb82ee844c276b8015d2d46 +size 12444708344 diff --git a/model-00029-of-00048.safetensors b/model-00029-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..4543eb4d667ff414ee2dbfdfec2567e8a07e4925 --- /dev/null +++ b/model-00029-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:960cae600512dcf5766d7ff55bbed36365ef89c167e7a4871ab17138dd1883fc +size 12445298408 diff --git a/model-00030-of-00048.safetensors b/model-00030-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..6a265fd799ab0e6800a149637b2d86660b3e4a60 --- /dev/null +++ b/model-00030-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:6a2a0338f4b22ed77dea801159a45f859c7faa5ebea0d53085aa159aa40b1252 +size 12444727000 diff --git a/model-00031-of-00048.safetensors b/model-00031-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..a560d643a14ff0080d074b78b61317f052993f2e --- /dev/null +++ b/model-00031-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:ecdce253aca477d9cce056495d6545f479befda55b352bff75cee3aa75c7c3a6 +size 12444708344 diff --git a/model-00032-of-00048.safetensors b/model-00032-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..0a95a6a186be051ef5776f841b3de79c837af9e7 --- /dev/null +++ b/model-00032-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3b92c8cda5b4087c5595a32cc2351b656a59394a874f31e36de467ff8e370a32 +size 12444708344 diff --git a/model-00033-of-00048.safetensors b/model-00033-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..1b597637899560086e70616609de40f90ef09a6e --- /dev/null +++ b/model-00033-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b4b9f6d5d15e943ef3e3bbc5502d5f921f69bb5d74aeb7e7b0f678b455a2ac9a +size 12444708344 diff --git a/model-00034-of-00048.safetensors b/model-00034-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..3bee2c099fa0e81c2c0935f6b791e969b63e21a7 --- /dev/null +++ b/model-00034-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7bb41c901da35268defbc985c0161847faeff4aebadeb08fc5294c5cb4fb7a10 +size 12444708336 diff --git a/model-00035-of-00048.safetensors b/model-00035-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..d994696f774bfae8140801ac9769da9e048a5ac3 --- /dev/null +++ b/model-00035-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:64fee61ca6cd479e79ee4a30e460606edf4ac364bddfba90cc25ee8ed62f3e5c +size 12444708336 diff --git a/model-00036-of-00048.safetensors b/model-00036-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..a6db99a7cf7004db46fb8c29a5e43d24a0c14423 --- /dev/null +++ b/model-00036-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e0cc7aaf0c72a31d321076526ff28fa67c2343511dcfe8245e33804f819a4315 +size 12445298408 diff --git a/model-00037-of-00048.safetensors b/model-00037-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..cfb7021947e415592b6c382d7945f49dd906c6dd --- /dev/null +++ b/model-00037-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:96c6362e84341cfac582dbae843d70f252ae0ce1fb88bd457b4c02c74ae1e5a2 +size 12444727000 diff --git a/model-00038-of-00048.safetensors b/model-00038-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..21df88e5c33979e1a7a731f4a9b20b926b5b98d3 --- /dev/null +++ b/model-00038-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:31ceccfecc80ad5c3bebf9b984863e9f982b2510b947997dcb842840103fb591 +size 12444708344 diff --git a/model-00039-of-00048.safetensors b/model-00039-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..bae9fe933bf62eb207473249085ee7c3e18f543e --- /dev/null +++ b/model-00039-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:d7ffe1096223475b6d699bdac0cf8188120124c2be6ed55ff7532832b5afe948 +size 12444708344 diff --git a/model-00040-of-00048.safetensors b/model-00040-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..ebe9f38b01100858893a082e2b5b280846b6cdae --- /dev/null +++ b/model-00040-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:e9bbde667c713237de6f545e04717c0190902355f172762df0113e5eb512140f +size 12444708344 diff --git a/model-00041-of-00048.safetensors b/model-00041-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..e5cec4cf483d31c259874ed8c8d802494e33c5b9 --- /dev/null +++ b/model-00041-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4226864e33dfdbf378440fec0fc41b662f8ebfa0c2ab38c462f82cf3d6b8b95b +size 12444708344 diff --git a/model-00042-of-00048.safetensors b/model-00042-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..ae3f932efe9a44ce04e95fe594c6873dc1de6136 --- /dev/null +++ b/model-00042-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b0eec71b24560650ef0faf4a0707ef0e47d001b09cc2c84f93749f845e3990b0 +size 12445298408 diff --git a/model-00043-of-00048.safetensors b/model-00043-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..6fbfd6759329c7ecd3f94cc05cffd0160e9a4099 --- /dev/null +++ b/model-00043-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:721a601da155afda19a4725fbecf7b27048ecb37e9e056186a7f478b20c350e3 +size 12445317040 diff --git a/model-00044-of-00048.safetensors b/model-00044-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..e8a79f9d6c74b73a6e227e06f688ef6b331381fe --- /dev/null +++ b/model-00044-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:4cbf3bc8c4ee6910a812fec33670c72fdd8595f6e3fd4c27942bc3cf35a27629 +size 12444726976 diff --git a/model-00045-of-00048.safetensors b/model-00045-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..6e4d8541118c066727feda1dd8512a50977fca6c --- /dev/null +++ b/model-00045-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b57bff74fdfeaad8f3f86fa9b5c27a51d4b82065655040c832122a9ea2dd612a +size 12444708328 diff --git a/model-00046-of-00048.safetensors b/model-00046-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..9477efbe9962ca7ea09f173fd3276f9fb9372e66 --- /dev/null +++ b/model-00046-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b9a6127eea796a5afb7659be39e2adfb82fd732f0e4856060e869ceeb26b7f8b +size 12444708328 diff --git a/model-00047-of-00048.safetensors b/model-00047-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..689961b69c5d0cb4d63708f84e04d8c629d9363c --- /dev/null +++ b/model-00047-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:77a815ae38001fc130e788ae3d8cbe2f09d745208ceaec8b99058db3c3ba8bcf +size 12444708328 diff --git a/model-00048-of-00048.safetensors b/model-00048-of-00048.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..0c86c876b2e172fcc592b8c353f5d0315faff85f --- /dev/null +++ b/model-00048-of-00048.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:b53545865f28bd7a8b2dd5590033914bd00b11fa54c6c2079f3adab079f823ca +size 14756419216 diff --git a/model.safetensors.index.json b/model.safetensors.index.json new file mode 100644 index 0000000000000000000000000000000000000000..10ae006e0c71a6ad44943acebcf4ca5f69fb1704 --- /dev/null +++ b/model.safetensors.index.json @@ -0,0 +1,1617 @@ +{ + "metadata": { + "total_size": 601713487872 + }, + "weight_map": { + "model.llm.attn_res.attn_queries.0": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.1": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.10": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.11": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.12": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.13": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.14": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.15": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.16": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.17": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.18": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.19": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.2": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.20": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.21": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.22": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.23": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.24": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.25": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.26": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.27": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.28": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.29": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.3": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.30": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.31": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.32": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.33": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.34": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.35": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.36": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.37": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.38": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.39": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.4": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.40": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.41": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.42": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.43": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.44": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.45": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.46": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.5": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.6": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.7": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.8": "model-00001-of-00048.safetensors", + "model.llm.attn_res.attn_queries.9": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.0": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.1": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.10": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.11": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.12": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.13": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.14": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.15": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.16": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.17": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.18": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.19": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.2": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.20": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.21": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.22": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.23": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.24": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.25": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.26": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.27": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.28": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.29": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.3": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.30": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.31": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.32": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.33": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.34": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.35": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.36": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.37": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.38": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.39": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.4": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.40": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.41": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.42": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.43": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.44": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.45": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.46": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.47": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.5": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.6": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.7": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.8": "model-00001-of-00048.safetensors", + "model.llm.attn_res.mlp_queries.9": "model-00001-of-00048.safetensors", + "model.llm.attn_res.output_query": "model-00001-of-00048.safetensors", + "model.llm.embed.weight": "model-00001-of-00048.safetensors", + "model.llm.embed_norm.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.0.attn.attn_sink_proj.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.0.attn.gate_attn.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.0.attn.k_sconv.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.0.attn.q_l2_clip.scale": "model-00001-of-00048.safetensors", + "model.llm.layers.0.attn.rel_l2_clip.scale": "model-00001-of-00048.safetensors", + "model.llm.layers.0.attn.rel_logits_proj.proj": "model-00001-of-00048.safetensors", + "model.llm.layers.0.attn.v_sconv.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.0.attn.wk_dv.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.0.attn.wo_ud.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.0.attn.wq_du.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.0.attn.wr_du.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.0.attn.wv_dv.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.0.attn_norm.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.0.attn_sconv.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.0.mlp.experts.w13_weight": "model-00001-of-00048.safetensors", + "model.llm.layers.0.mlp.experts.w13_weight_scale": "model-00001-of-00048.safetensors", + "model.llm.layers.0.mlp.experts.w2_weight": "model-00001-of-00048.safetensors", + "model.llm.layers.0.mlp.experts.w2_weight_scale": "model-00001-of-00048.safetensors", + "model.llm.layers.0.mlp.gate.bias": "model-00001-of-00048.safetensors", + "model.llm.layers.0.mlp.gate.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.0.mlp_norm.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.0.mlp_sconv.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.1.attn.attn_sink_proj.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.1.attn.gate_attn.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.1.attn.k_sconv.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.1.attn.q_l2_clip.scale": "model-00001-of-00048.safetensors", + "model.llm.layers.1.attn.rel_l2_clip.scale": "model-00001-of-00048.safetensors", + "model.llm.layers.1.attn.rel_logits_proj.proj": "model-00001-of-00048.safetensors", + "model.llm.layers.1.attn.v_sconv.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.1.attn.wk_dv.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.1.attn.wo_ud.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.1.attn.wq_du.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.1.attn.wr_du.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.1.attn.wv_dv.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.1.attn_norm.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.1.attn_sconv.weight": "model-00001-of-00048.safetensors", + "model.llm.layers.1.mlp.experts.w13_weight": "model-00002-of-00048.safetensors", + "model.llm.layers.1.mlp.experts.w13_weight_scale": "model-00002-of-00048.safetensors", + "model.llm.layers.1.mlp.experts.w2_weight": "model-00002-of-00048.safetensors", + "model.llm.layers.1.mlp.experts.w2_weight_scale": "model-00002-of-00048.safetensors", + "model.llm.layers.1.mlp.gate.bias": "model-00002-of-00048.safetensors", + "model.llm.layers.1.mlp.gate.weight": "model-00002-of-00048.safetensors", + "model.llm.layers.1.mlp_norm.weight": "model-00002-of-00048.safetensors", + "model.llm.layers.1.mlp_sconv.weight": "model-00002-of-00048.safetensors", + "model.llm.layers.10.attn.attn_sink_proj.weight": "model-00002-of-00048.safetensors", + "model.llm.layers.10.attn.gate_attn.weight": "model-00002-of-00048.safetensors", + "model.llm.layers.10.attn.k_sconv.weight": "model-00002-of-00048.safetensors", + "model.llm.layers.10.attn.q_l2_clip.scale": "model-00002-of-00048.safetensors", + "model.llm.layers.10.attn.rel_l2_clip.scale": "model-00002-of-00048.safetensors", + "model.llm.layers.10.attn.rel_logits_proj.proj": "model-00002-of-00048.safetensors", + "model.llm.layers.10.attn.v_sconv.weight": "model-00002-of-00048.safetensors", + "model.llm.layers.10.attn.wk_dv.weight": "model-00002-of-00048.safetensors", + "model.llm.layers.10.attn.wo_ud.weight": "model-00002-of-00048.safetensors", + "model.llm.layers.10.attn.wq_du.weight": "model-00002-of-00048.safetensors", + "model.llm.layers.10.attn.wr_du.weight": "model-00002-of-00048.safetensors", + "model.llm.layers.10.attn.wv_dv.weight": "model-00002-of-00048.safetensors", + "model.llm.layers.10.attn_norm.weight": "model-00002-of-00048.safetensors", + "model.llm.layers.10.attn_sconv.weight": "model-00002-of-00048.safetensors", + "model.llm.layers.10.mlp.experts.w13_weight": "model-00003-of-00048.safetensors", + "model.llm.layers.10.mlp.experts.w13_weight_scale": "model-00003-of-00048.safetensors", + "model.llm.layers.10.mlp.experts.w2_weight": "model-00003-of-00048.safetensors", + "model.llm.layers.10.mlp.experts.w2_weight_scale": "model-00003-of-00048.safetensors", + "model.llm.layers.10.mlp.gate.bias": "model-00003-of-00048.safetensors", + "model.llm.layers.10.mlp.gate.weight": "model-00003-of-00048.safetensors", + "model.llm.layers.10.mlp_norm.weight": "model-00003-of-00048.safetensors", + "model.llm.layers.10.mlp_sconv.weight": "model-00003-of-00048.safetensors", + "model.llm.layers.11.attn.attn_sink_proj.weight": "model-00003-of-00048.safetensors", + "model.llm.layers.11.attn.gate_attn.weight": "model-00003-of-00048.safetensors", + "model.llm.layers.11.attn.k_sconv.static_w": "model-00003-of-00048.safetensors", + "model.llm.layers.11.attn.k_sconv.weight_proj": "model-00003-of-00048.safetensors", + "model.llm.layers.11.attn.q_l2_clip.scale": "model-00003-of-00048.safetensors", + "model.llm.layers.11.attn.rel_l2_clip.scale": "model-00003-of-00048.safetensors", + "model.llm.layers.11.attn.rel_logits_proj.proj": "model-00003-of-00048.safetensors", + "model.llm.layers.11.attn.v_sconv.static_w": "model-00003-of-00048.safetensors", + "model.llm.layers.11.attn.v_sconv.weight_proj": "model-00003-of-00048.safetensors", + "model.llm.layers.11.attn.wk_dv.weight": "model-00003-of-00048.safetensors", + "model.llm.layers.11.attn.wo_ud.weight": "model-00003-of-00048.safetensors", + "model.llm.layers.11.attn.wq_du.weight": "model-00003-of-00048.safetensors", + "model.llm.layers.11.attn.wr_du.weight": "model-00003-of-00048.safetensors", + "model.llm.layers.11.attn.wv_dv.weight": "model-00003-of-00048.safetensors", + "model.llm.layers.11.attn_sconv.weight": "model-00003-of-00048.safetensors", + "model.llm.layers.11.k_input_norm.weight": "model-00003-of-00048.safetensors", + "model.llm.layers.11.mlp.experts.w13_weight": "model-00004-of-00048.safetensors", + "model.llm.layers.11.mlp.experts.w13_weight_scale": "model-00004-of-00048.safetensors", + "model.llm.layers.11.mlp.experts.w2_weight": "model-00004-of-00048.safetensors", + "model.llm.layers.11.mlp.experts.w2_weight_scale": "model-00004-of-00048.safetensors", + "model.llm.layers.11.mlp.gate.bias": "model-00004-of-00048.safetensors", + "model.llm.layers.11.mlp.gate.weight": "model-00004-of-00048.safetensors", + "model.llm.layers.11.mlp_norm.weight": "model-00004-of-00048.safetensors", + "model.llm.layers.11.mlp_sconv.weight": "model-00004-of-00048.safetensors", + "model.llm.layers.11.q_input_norm.weight": "model-00004-of-00048.safetensors", + "model.llm.layers.11.v_input_norm.weight": "model-00004-of-00048.safetensors", + "model.llm.layers.12.attn.attn_sink_proj.weight": "model-00004-of-00048.safetensors", + "model.llm.layers.12.attn.gate_attn.weight": "model-00004-of-00048.safetensors", + "model.llm.layers.12.attn.k_sconv.weight": "model-00004-of-00048.safetensors", + "model.llm.layers.12.attn.q_l2_clip.scale": "model-00004-of-00048.safetensors", + "model.llm.layers.12.attn.rel_l2_clip.scale": "model-00004-of-00048.safetensors", + "model.llm.layers.12.attn.rel_logits_proj.proj": "model-00004-of-00048.safetensors", + "model.llm.layers.12.attn.v_sconv.weight": "model-00004-of-00048.safetensors", + "model.llm.layers.12.attn.wk_dv.weight": "model-00004-of-00048.safetensors", + "model.llm.layers.12.attn.wo_ud.weight": "model-00004-of-00048.safetensors", + "model.llm.layers.12.attn.wq_du.weight": "model-00004-of-00048.safetensors", + "model.llm.layers.12.attn.wr_du.weight": "model-00004-of-00048.safetensors", + "model.llm.layers.12.attn.wv_dv.weight": "model-00004-of-00048.safetensors", + "model.llm.layers.12.attn_norm.weight": "model-00004-of-00048.safetensors", + "model.llm.layers.12.attn_sconv.weight": "model-00004-of-00048.safetensors", + "model.llm.layers.12.mlp.experts.w13_weight": "model-00005-of-00048.safetensors", + "model.llm.layers.12.mlp.experts.w13_weight_scale": "model-00005-of-00048.safetensors", + "model.llm.layers.12.mlp.experts.w2_weight": "model-00005-of-00048.safetensors", + "model.llm.layers.12.mlp.experts.w2_weight_scale": "model-00005-of-00048.safetensors", + "model.llm.layers.12.mlp.gate.bias": "model-00005-of-00048.safetensors", + "model.llm.layers.12.mlp.gate.weight": "model-00005-of-00048.safetensors", + "model.llm.layers.12.mlp_norm.weight": "model-00005-of-00048.safetensors", + "model.llm.layers.12.mlp_sconv.weight": "model-00005-of-00048.safetensors", + "model.llm.layers.13.attn.attn_sink_proj.weight": "model-00005-of-00048.safetensors", + "model.llm.layers.13.attn.gate_attn.weight": "model-00005-of-00048.safetensors", + "model.llm.layers.13.attn.k_sconv.weight": "model-00005-of-00048.safetensors", + "model.llm.layers.13.attn.q_l2_clip.scale": "model-00005-of-00048.safetensors", + "model.llm.layers.13.attn.rel_l2_clip.scale": "model-00005-of-00048.safetensors", + "model.llm.layers.13.attn.rel_logits_proj.proj": "model-00005-of-00048.safetensors", + "model.llm.layers.13.attn.v_sconv.weight": "model-00005-of-00048.safetensors", + "model.llm.layers.13.attn.wk_dv.weight": "model-00005-of-00048.safetensors", + "model.llm.layers.13.attn.wo_ud.weight": "model-00005-of-00048.safetensors", + "model.llm.layers.13.attn.wq_du.weight": "model-00005-of-00048.safetensors", + "model.llm.layers.13.attn.wr_du.weight": "model-00005-of-00048.safetensors", + "model.llm.layers.13.attn.wv_dv.weight": "model-00005-of-00048.safetensors", + "model.llm.layers.13.attn_norm.weight": "model-00005-of-00048.safetensors", + "model.llm.layers.13.attn_sconv.weight": "model-00005-of-00048.safetensors", + "model.llm.layers.13.mlp.experts.w13_weight": "model-00006-of-00048.safetensors", + "model.llm.layers.13.mlp.experts.w13_weight_scale": "model-00006-of-00048.safetensors", + "model.llm.layers.13.mlp.experts.w2_weight": "model-00006-of-00048.safetensors", + "model.llm.layers.13.mlp.experts.w2_weight_scale": "model-00006-of-00048.safetensors", + "model.llm.layers.13.mlp.gate.bias": "model-00006-of-00048.safetensors", + "model.llm.layers.13.mlp.gate.weight": "model-00006-of-00048.safetensors", + "model.llm.layers.13.mlp_norm.weight": "model-00006-of-00048.safetensors", + "model.llm.layers.13.mlp_sconv.weight": "model-00006-of-00048.safetensors", + "model.llm.layers.14.attn.attn_sink_proj.weight": "model-00006-of-00048.safetensors", + "model.llm.layers.14.attn.gate_attn.weight": "model-00006-of-00048.safetensors", + "model.llm.layers.14.attn.k_sconv.weight": "model-00006-of-00048.safetensors", + "model.llm.layers.14.attn.q_l2_clip.scale": "model-00006-of-00048.safetensors", + "model.llm.layers.14.attn.rel_l2_clip.scale": "model-00006-of-00048.safetensors", + "model.llm.layers.14.attn.rel_logits_proj.proj": "model-00006-of-00048.safetensors", + "model.llm.layers.14.attn.v_sconv.weight": "model-00006-of-00048.safetensors", + "model.llm.layers.14.attn.wk_dv.weight": "model-00006-of-00048.safetensors", + "model.llm.layers.14.attn.wo_ud.weight": "model-00006-of-00048.safetensors", + "model.llm.layers.14.attn.wq_du.weight": "model-00006-of-00048.safetensors", + "model.llm.layers.14.attn.wr_du.weight": "model-00006-of-00048.safetensors", + "model.llm.layers.14.attn.wv_dv.weight": "model-00006-of-00048.safetensors", + "model.llm.layers.14.attn_norm.weight": "model-00006-of-00048.safetensors", + "model.llm.layers.14.attn_sconv.weight": "model-00006-of-00048.safetensors", + "model.llm.layers.14.mlp.experts.w13_weight": "model-00007-of-00048.safetensors", + "model.llm.layers.14.mlp.experts.w13_weight_scale": "model-00007-of-00048.safetensors", + "model.llm.layers.14.mlp.experts.w2_weight": "model-00007-of-00048.safetensors", + "model.llm.layers.14.mlp.experts.w2_weight_scale": "model-00007-of-00048.safetensors", + "model.llm.layers.14.mlp.gate.bias": "model-00007-of-00048.safetensors", + "model.llm.layers.14.mlp.gate.weight": "model-00007-of-00048.safetensors", + "model.llm.layers.14.mlp_norm.weight": "model-00007-of-00048.safetensors", + "model.llm.layers.14.mlp_sconv.weight": "model-00007-of-00048.safetensors", + "model.llm.layers.15.attn.attn_sink_proj.weight": "model-00007-of-00048.safetensors", + "model.llm.layers.15.attn.gate_attn.weight": "model-00007-of-00048.safetensors", + "model.llm.layers.15.attn.k_sconv.weight": "model-00007-of-00048.safetensors", + "model.llm.layers.15.attn.q_l2_clip.scale": "model-00007-of-00048.safetensors", + "model.llm.layers.15.attn.rel_l2_clip.scale": "model-00007-of-00048.safetensors", + "model.llm.layers.15.attn.rel_logits_proj.proj": "model-00007-of-00048.safetensors", + "model.llm.layers.15.attn.v_sconv.weight": "model-00007-of-00048.safetensors", + "model.llm.layers.15.attn.wk_dv.weight": "model-00007-of-00048.safetensors", + "model.llm.layers.15.attn.wo_ud.weight": "model-00007-of-00048.safetensors", + "model.llm.layers.15.attn.wq_du.weight": "model-00007-of-00048.safetensors", + "model.llm.layers.15.attn.wr_du.weight": "model-00007-of-00048.safetensors", + "model.llm.layers.15.attn.wv_dv.weight": "model-00007-of-00048.safetensors", + "model.llm.layers.15.attn_norm.weight": "model-00007-of-00048.safetensors", + "model.llm.layers.15.attn_sconv.weight": "model-00007-of-00048.safetensors", + "model.llm.layers.15.mlp.experts.w13_weight": "model-00008-of-00048.safetensors", + "model.llm.layers.15.mlp.experts.w13_weight_scale": "model-00008-of-00048.safetensors", + "model.llm.layers.15.mlp.experts.w2_weight": "model-00008-of-00048.safetensors", + "model.llm.layers.15.mlp.experts.w2_weight_scale": "model-00008-of-00048.safetensors", + "model.llm.layers.15.mlp.gate.bias": "model-00008-of-00048.safetensors", + "model.llm.layers.15.mlp.gate.weight": "model-00008-of-00048.safetensors", + "model.llm.layers.15.mlp_norm.weight": "model-00008-of-00048.safetensors", + "model.llm.layers.15.mlp_sconv.weight": "model-00008-of-00048.safetensors", + "model.llm.layers.16.attn.attn_sink_proj.weight": "model-00008-of-00048.safetensors", + "model.llm.layers.16.attn.gate_attn.weight": "model-00008-of-00048.safetensors", + "model.llm.layers.16.attn.k_sconv.weight": "model-00008-of-00048.safetensors", + "model.llm.layers.16.attn.q_l2_clip.scale": "model-00008-of-00048.safetensors", + "model.llm.layers.16.attn.rel_l2_clip.scale": "model-00008-of-00048.safetensors", + "model.llm.layers.16.attn.rel_logits_proj.proj": "model-00008-of-00048.safetensors", + "model.llm.layers.16.attn.v_sconv.weight": "model-00008-of-00048.safetensors", + "model.llm.layers.16.attn.wk_dv.weight": "model-00008-of-00048.safetensors", + "model.llm.layers.16.attn.wo_ud.weight": "model-00008-of-00048.safetensors", + "model.llm.layers.16.attn.wq_du.weight": "model-00008-of-00048.safetensors", + "model.llm.layers.16.attn.wr_du.weight": "model-00008-of-00048.safetensors", + "model.llm.layers.16.attn.wv_dv.weight": "model-00008-of-00048.safetensors", + "model.llm.layers.16.attn_norm.weight": "model-00008-of-00048.safetensors", + "model.llm.layers.16.attn_sconv.weight": "model-00008-of-00048.safetensors", + "model.llm.layers.16.mlp.experts.w13_weight": "model-00009-of-00048.safetensors", + "model.llm.layers.16.mlp.experts.w13_weight_scale": "model-00009-of-00048.safetensors", + "model.llm.layers.16.mlp.experts.w2_weight": "model-00009-of-00048.safetensors", + "model.llm.layers.16.mlp.experts.w2_weight_scale": "model-00009-of-00048.safetensors", + "model.llm.layers.16.mlp.gate.bias": "model-00009-of-00048.safetensors", + "model.llm.layers.16.mlp.gate.weight": "model-00009-of-00048.safetensors", + "model.llm.layers.16.mlp_norm.weight": "model-00009-of-00048.safetensors", + "model.llm.layers.16.mlp_sconv.weight": "model-00009-of-00048.safetensors", + "model.llm.layers.17.attn.attn_sink_proj.weight": "model-00009-of-00048.safetensors", + "model.llm.layers.17.attn.gate_attn.weight": "model-00009-of-00048.safetensors", + "model.llm.layers.17.attn.k_sconv.static_w": "model-00009-of-00048.safetensors", + "model.llm.layers.17.attn.k_sconv.weight_proj": "model-00009-of-00048.safetensors", + "model.llm.layers.17.attn.q_l2_clip.scale": "model-00009-of-00048.safetensors", + "model.llm.layers.17.attn.rel_l2_clip.scale": "model-00009-of-00048.safetensors", + "model.llm.layers.17.attn.rel_logits_proj.proj": "model-00009-of-00048.safetensors", + "model.llm.layers.17.attn.v_sconv.static_w": "model-00009-of-00048.safetensors", + "model.llm.layers.17.attn.v_sconv.weight_proj": "model-00009-of-00048.safetensors", + "model.llm.layers.17.attn.wk_dv.weight": "model-00009-of-00048.safetensors", + "model.llm.layers.17.attn.wo_ud.weight": "model-00009-of-00048.safetensors", + "model.llm.layers.17.attn.wq_du.weight": "model-00009-of-00048.safetensors", + "model.llm.layers.17.attn.wr_du.weight": "model-00009-of-00048.safetensors", + "model.llm.layers.17.attn.wv_dv.weight": "model-00009-of-00048.safetensors", + "model.llm.layers.17.attn_sconv.weight": "model-00009-of-00048.safetensors", + "model.llm.layers.17.k_input_norm.weight": "model-00009-of-00048.safetensors", + "model.llm.layers.17.mlp.experts.w13_weight": "model-00010-of-00048.safetensors", + "model.llm.layers.17.mlp.experts.w13_weight_scale": "model-00010-of-00048.safetensors", + "model.llm.layers.17.mlp.experts.w2_weight": "model-00010-of-00048.safetensors", + "model.llm.layers.17.mlp.experts.w2_weight_scale": "model-00010-of-00048.safetensors", + "model.llm.layers.17.mlp.gate.bias": "model-00010-of-00048.safetensors", + "model.llm.layers.17.mlp.gate.weight": "model-00010-of-00048.safetensors", + "model.llm.layers.17.mlp_norm.weight": "model-00010-of-00048.safetensors", + "model.llm.layers.17.mlp_sconv.weight": "model-00010-of-00048.safetensors", + "model.llm.layers.17.q_input_norm.weight": "model-00010-of-00048.safetensors", + "model.llm.layers.17.v_input_norm.weight": "model-00010-of-00048.safetensors", + "model.llm.layers.18.attn.attn_sink_proj.weight": "model-00010-of-00048.safetensors", + "model.llm.layers.18.attn.gate_attn.weight": "model-00010-of-00048.safetensors", + "model.llm.layers.18.attn.k_sconv.weight": "model-00010-of-00048.safetensors", + "model.llm.layers.18.attn.q_l2_clip.scale": "model-00010-of-00048.safetensors", + "model.llm.layers.18.attn.rel_l2_clip.scale": "model-00010-of-00048.safetensors", + "model.llm.layers.18.attn.rel_logits_proj.proj": "model-00010-of-00048.safetensors", + "model.llm.layers.18.attn.v_sconv.weight": "model-00010-of-00048.safetensors", + "model.llm.layers.18.attn.wk_dv.weight": "model-00010-of-00048.safetensors", + "model.llm.layers.18.attn.wo_ud.weight": "model-00010-of-00048.safetensors", + "model.llm.layers.18.attn.wq_du.weight": "model-00010-of-00048.safetensors", + "model.llm.layers.18.attn.wr_du.weight": "model-00010-of-00048.safetensors", + "model.llm.layers.18.attn.wv_dv.weight": "model-00010-of-00048.safetensors", + "model.llm.layers.18.attn_norm.weight": "model-00010-of-00048.safetensors", + "model.llm.layers.18.attn_sconv.weight": "model-00010-of-00048.safetensors", + "model.llm.layers.18.mlp.experts.w13_weight": "model-00011-of-00048.safetensors", + "model.llm.layers.18.mlp.experts.w13_weight_scale": "model-00011-of-00048.safetensors", + "model.llm.layers.18.mlp.experts.w2_weight": "model-00011-of-00048.safetensors", + "model.llm.layers.18.mlp.experts.w2_weight_scale": "model-00011-of-00048.safetensors", + "model.llm.layers.18.mlp.gate.bias": "model-00011-of-00048.safetensors", + "model.llm.layers.18.mlp.gate.weight": "model-00011-of-00048.safetensors", + "model.llm.layers.18.mlp_norm.weight": "model-00011-of-00048.safetensors", + "model.llm.layers.18.mlp_sconv.weight": "model-00011-of-00048.safetensors", + "model.llm.layers.19.attn.attn_sink_proj.weight": "model-00011-of-00048.safetensors", + "model.llm.layers.19.attn.gate_attn.weight": "model-00011-of-00048.safetensors", + "model.llm.layers.19.attn.k_sconv.weight": "model-00011-of-00048.safetensors", + "model.llm.layers.19.attn.q_l2_clip.scale": "model-00011-of-00048.safetensors", + "model.llm.layers.19.attn.rel_l2_clip.scale": "model-00011-of-00048.safetensors", + "model.llm.layers.19.attn.rel_logits_proj.proj": "model-00011-of-00048.safetensors", + "model.llm.layers.19.attn.v_sconv.weight": "model-00011-of-00048.safetensors", + "model.llm.layers.19.attn.wk_dv.weight": "model-00011-of-00048.safetensors", + "model.llm.layers.19.attn.wo_ud.weight": "model-00011-of-00048.safetensors", + "model.llm.layers.19.attn.wq_du.weight": "model-00011-of-00048.safetensors", + "model.llm.layers.19.attn.wr_du.weight": "model-00011-of-00048.safetensors", + "model.llm.layers.19.attn.wv_dv.weight": "model-00011-of-00048.safetensors", + "model.llm.layers.19.attn_norm.weight": "model-00011-of-00048.safetensors", + "model.llm.layers.19.attn_sconv.weight": "model-00011-of-00048.safetensors", + "model.llm.layers.19.mlp.experts.w13_weight": "model-00012-of-00048.safetensors", + "model.llm.layers.19.mlp.experts.w13_weight_scale": "model-00012-of-00048.safetensors", + "model.llm.layers.19.mlp.experts.w2_weight": "model-00012-of-00048.safetensors", + "model.llm.layers.19.mlp.experts.w2_weight_scale": "model-00012-of-00048.safetensors", + "model.llm.layers.19.mlp.gate.bias": "model-00012-of-00048.safetensors", + "model.llm.layers.19.mlp.gate.weight": "model-00012-of-00048.safetensors", + "model.llm.layers.19.mlp_norm.weight": "model-00012-of-00048.safetensors", + "model.llm.layers.19.mlp_sconv.weight": "model-00012-of-00048.safetensors", + "model.llm.layers.2.attn.attn_sink_proj.weight": "model-00012-of-00048.safetensors", + "model.llm.layers.2.attn.gate_attn.weight": "model-00012-of-00048.safetensors", + "model.llm.layers.2.attn.k_sconv.weight": "model-00012-of-00048.safetensors", + "model.llm.layers.2.attn.q_l2_clip.scale": "model-00012-of-00048.safetensors", + "model.llm.layers.2.attn.rel_l2_clip.scale": "model-00012-of-00048.safetensors", + "model.llm.layers.2.attn.rel_logits_proj.proj": "model-00012-of-00048.safetensors", + "model.llm.layers.2.attn.v_sconv.weight": "model-00012-of-00048.safetensors", + "model.llm.layers.2.attn.wk_dv.weight": "model-00012-of-00048.safetensors", + "model.llm.layers.2.attn.wo_ud.weight": "model-00012-of-00048.safetensors", + "model.llm.layers.2.attn.wq_du.weight": "model-00012-of-00048.safetensors", + "model.llm.layers.2.attn.wr_du.weight": "model-00012-of-00048.safetensors", + "model.llm.layers.2.attn.wv_dv.weight": "model-00012-of-00048.safetensors", + "model.llm.layers.2.attn_norm.weight": "model-00012-of-00048.safetensors", + "model.llm.layers.2.attn_sconv.weight": "model-00012-of-00048.safetensors", + "model.llm.layers.2.mlp.experts.w13_weight": "model-00013-of-00048.safetensors", + "model.llm.layers.2.mlp.experts.w13_weight_scale": "model-00013-of-00048.safetensors", + "model.llm.layers.2.mlp.experts.w2_weight": "model-00013-of-00048.safetensors", + "model.llm.layers.2.mlp.experts.w2_weight_scale": "model-00013-of-00048.safetensors", + "model.llm.layers.2.mlp.gate.bias": "model-00013-of-00048.safetensors", + "model.llm.layers.2.mlp.gate.weight": "model-00013-of-00048.safetensors", + "model.llm.layers.2.mlp_norm.weight": "model-00013-of-00048.safetensors", + "model.llm.layers.2.mlp_sconv.weight": "model-00013-of-00048.safetensors", + "model.llm.layers.20.attn.attn_sink_proj.weight": "model-00013-of-00048.safetensors", + "model.llm.layers.20.attn.gate_attn.weight": "model-00013-of-00048.safetensors", + "model.llm.layers.20.attn.k_sconv.weight": "model-00013-of-00048.safetensors", + "model.llm.layers.20.attn.q_l2_clip.scale": "model-00013-of-00048.safetensors", + "model.llm.layers.20.attn.rel_l2_clip.scale": "model-00013-of-00048.safetensors", + "model.llm.layers.20.attn.rel_logits_proj.proj": "model-00013-of-00048.safetensors", + "model.llm.layers.20.attn.v_sconv.weight": "model-00013-of-00048.safetensors", + "model.llm.layers.20.attn.wk_dv.weight": "model-00013-of-00048.safetensors", + "model.llm.layers.20.attn.wo_ud.weight": "model-00013-of-00048.safetensors", + "model.llm.layers.20.attn.wq_du.weight": "model-00013-of-00048.safetensors", + "model.llm.layers.20.attn.wr_du.weight": "model-00013-of-00048.safetensors", + "model.llm.layers.20.attn.wv_dv.weight": "model-00013-of-00048.safetensors", + "model.llm.layers.20.attn_norm.weight": "model-00013-of-00048.safetensors", + "model.llm.layers.20.attn_sconv.weight": "model-00013-of-00048.safetensors", + "model.llm.layers.20.mlp.experts.w13_weight": "model-00014-of-00048.safetensors", + "model.llm.layers.20.mlp.experts.w13_weight_scale": "model-00014-of-00048.safetensors", + "model.llm.layers.20.mlp.experts.w2_weight": "model-00014-of-00048.safetensors", + "model.llm.layers.20.mlp.experts.w2_weight_scale": "model-00014-of-00048.safetensors", + "model.llm.layers.20.mlp.gate.bias": "model-00014-of-00048.safetensors", + "model.llm.layers.20.mlp.gate.weight": "model-00014-of-00048.safetensors", + "model.llm.layers.20.mlp_norm.weight": "model-00014-of-00048.safetensors", + "model.llm.layers.20.mlp_sconv.weight": "model-00014-of-00048.safetensors", + "model.llm.layers.21.attn.attn_sink_proj.weight": "model-00014-of-00048.safetensors", + "model.llm.layers.21.attn.gate_attn.weight": "model-00014-of-00048.safetensors", + "model.llm.layers.21.attn.k_sconv.weight": "model-00014-of-00048.safetensors", + "model.llm.layers.21.attn.q_l2_clip.scale": "model-00014-of-00048.safetensors", + "model.llm.layers.21.attn.rel_l2_clip.scale": "model-00014-of-00048.safetensors", + "model.llm.layers.21.attn.rel_logits_proj.proj": "model-00014-of-00048.safetensors", + "model.llm.layers.21.attn.v_sconv.weight": "model-00014-of-00048.safetensors", + "model.llm.layers.21.attn.wk_dv.weight": "model-00014-of-00048.safetensors", + "model.llm.layers.21.attn.wo_ud.weight": "model-00014-of-00048.safetensors", + "model.llm.layers.21.attn.wq_du.weight": "model-00014-of-00048.safetensors", + "model.llm.layers.21.attn.wr_du.weight": "model-00014-of-00048.safetensors", + "model.llm.layers.21.attn.wv_dv.weight": "model-00014-of-00048.safetensors", + "model.llm.layers.21.attn_norm.weight": "model-00014-of-00048.safetensors", + "model.llm.layers.21.attn_sconv.weight": "model-00014-of-00048.safetensors", + "model.llm.layers.21.mlp.experts.w13_weight": "model-00015-of-00048.safetensors", + "model.llm.layers.21.mlp.experts.w13_weight_scale": "model-00015-of-00048.safetensors", + "model.llm.layers.21.mlp.experts.w2_weight": "model-00015-of-00048.safetensors", + "model.llm.layers.21.mlp.experts.w2_weight_scale": "model-00015-of-00048.safetensors", + "model.llm.layers.21.mlp.gate.bias": "model-00015-of-00048.safetensors", + "model.llm.layers.21.mlp.gate.weight": "model-00015-of-00048.safetensors", + "model.llm.layers.21.mlp_norm.weight": "model-00015-of-00048.safetensors", + "model.llm.layers.21.mlp_sconv.weight": "model-00015-of-00048.safetensors", + "model.llm.layers.22.attn.attn_sink_proj.weight": "model-00015-of-00048.safetensors", + "model.llm.layers.22.attn.gate_attn.weight": "model-00015-of-00048.safetensors", + "model.llm.layers.22.attn.k_sconv.weight": "model-00015-of-00048.safetensors", + "model.llm.layers.22.attn.q_l2_clip.scale": "model-00015-of-00048.safetensors", + "model.llm.layers.22.attn.rel_l2_clip.scale": "model-00015-of-00048.safetensors", + "model.llm.layers.22.attn.rel_logits_proj.proj": "model-00015-of-00048.safetensors", + "model.llm.layers.22.attn.v_sconv.weight": "model-00015-of-00048.safetensors", + "model.llm.layers.22.attn.wk_dv.weight": "model-00015-of-00048.safetensors", + "model.llm.layers.22.attn.wo_ud.weight": "model-00015-of-00048.safetensors", + "model.llm.layers.22.attn.wq_du.weight": "model-00015-of-00048.safetensors", + "model.llm.layers.22.attn.wr_du.weight": "model-00015-of-00048.safetensors", + "model.llm.layers.22.attn.wv_dv.weight": "model-00015-of-00048.safetensors", + "model.llm.layers.22.attn_norm.weight": "model-00015-of-00048.safetensors", + "model.llm.layers.22.attn_sconv.weight": "model-00015-of-00048.safetensors", + "model.llm.layers.22.mlp.experts.w13_weight": "model-00016-of-00048.safetensors", + "model.llm.layers.22.mlp.experts.w13_weight_scale": "model-00016-of-00048.safetensors", + "model.llm.layers.22.mlp.experts.w2_weight": "model-00016-of-00048.safetensors", + "model.llm.layers.22.mlp.experts.w2_weight_scale": "model-00016-of-00048.safetensors", + "model.llm.layers.22.mlp.gate.bias": "model-00016-of-00048.safetensors", + "model.llm.layers.22.mlp.gate.weight": "model-00016-of-00048.safetensors", + "model.llm.layers.22.mlp_norm.weight": "model-00016-of-00048.safetensors", + "model.llm.layers.22.mlp_sconv.weight": "model-00016-of-00048.safetensors", + "model.llm.layers.23.attn.attn_sink_proj.weight": "model-00016-of-00048.safetensors", + "model.llm.layers.23.attn.gate_attn.weight": "model-00016-of-00048.safetensors", + "model.llm.layers.23.attn.k_sconv.static_w": "model-00016-of-00048.safetensors", + "model.llm.layers.23.attn.k_sconv.weight_proj": "model-00016-of-00048.safetensors", + "model.llm.layers.23.attn.q_l2_clip.scale": "model-00016-of-00048.safetensors", + "model.llm.layers.23.attn.rel_l2_clip.scale": "model-00016-of-00048.safetensors", + "model.llm.layers.23.attn.rel_logits_proj.proj": "model-00016-of-00048.safetensors", + "model.llm.layers.23.attn.v_sconv.static_w": "model-00016-of-00048.safetensors", + "model.llm.layers.23.attn.v_sconv.weight_proj": "model-00016-of-00048.safetensors", + "model.llm.layers.23.attn.wk_dv.weight": "model-00016-of-00048.safetensors", + "model.llm.layers.23.attn.wo_ud.weight": "model-00016-of-00048.safetensors", + "model.llm.layers.23.attn.wq_du.weight": "model-00016-of-00048.safetensors", + "model.llm.layers.23.attn.wr_du.weight": "model-00016-of-00048.safetensors", + "model.llm.layers.23.attn.wv_dv.weight": "model-00016-of-00048.safetensors", + "model.llm.layers.23.attn_sconv.weight": "model-00016-of-00048.safetensors", + "model.llm.layers.23.k_input_norm.weight": "model-00016-of-00048.safetensors", + "model.llm.layers.23.mlp.experts.w13_weight": "model-00017-of-00048.safetensors", + "model.llm.layers.23.mlp.experts.w13_weight_scale": "model-00017-of-00048.safetensors", + "model.llm.layers.23.mlp.experts.w2_weight": "model-00017-of-00048.safetensors", + "model.llm.layers.23.mlp.experts.w2_weight_scale": "model-00017-of-00048.safetensors", + "model.llm.layers.23.mlp.gate.bias": "model-00017-of-00048.safetensors", + "model.llm.layers.23.mlp.gate.weight": "model-00017-of-00048.safetensors", + "model.llm.layers.23.mlp_norm.weight": "model-00017-of-00048.safetensors", + "model.llm.layers.23.mlp_sconv.weight": "model-00017-of-00048.safetensors", + "model.llm.layers.23.q_input_norm.weight": "model-00017-of-00048.safetensors", + "model.llm.layers.23.v_input_norm.weight": "model-00017-of-00048.safetensors", + "model.llm.layers.24.attn.attn_sink_proj.weight": "model-00017-of-00048.safetensors", + "model.llm.layers.24.attn.gate_attn.weight": "model-00017-of-00048.safetensors", + "model.llm.layers.24.attn.k_sconv.weight": "model-00017-of-00048.safetensors", + "model.llm.layers.24.attn.q_l2_clip.scale": "model-00017-of-00048.safetensors", + "model.llm.layers.24.attn.rel_l2_clip.scale": "model-00017-of-00048.safetensors", + "model.llm.layers.24.attn.rel_logits_proj.proj": "model-00017-of-00048.safetensors", + "model.llm.layers.24.attn.v_sconv.weight": "model-00017-of-00048.safetensors", + "model.llm.layers.24.attn.wk_dv.weight": "model-00017-of-00048.safetensors", + "model.llm.layers.24.attn.wo_ud.weight": "model-00017-of-00048.safetensors", + "model.llm.layers.24.attn.wq_du.weight": "model-00017-of-00048.safetensors", + "model.llm.layers.24.attn.wr_du.weight": "model-00017-of-00048.safetensors", + "model.llm.layers.24.attn.wv_dv.weight": "model-00017-of-00048.safetensors", + "model.llm.layers.24.attn_norm.weight": "model-00017-of-00048.safetensors", + "model.llm.layers.24.attn_sconv.weight": "model-00017-of-00048.safetensors", + "model.llm.layers.24.mlp.experts.w13_weight": "model-00018-of-00048.safetensors", + "model.llm.layers.24.mlp.experts.w13_weight_scale": "model-00018-of-00048.safetensors", + "model.llm.layers.24.mlp.experts.w2_weight": "model-00018-of-00048.safetensors", + "model.llm.layers.24.mlp.experts.w2_weight_scale": "model-00018-of-00048.safetensors", + "model.llm.layers.24.mlp.gate.bias": "model-00018-of-00048.safetensors", + "model.llm.layers.24.mlp.gate.weight": "model-00018-of-00048.safetensors", + "model.llm.layers.24.mlp_norm.weight": "model-00018-of-00048.safetensors", + "model.llm.layers.24.mlp_sconv.weight": "model-00018-of-00048.safetensors", + "model.llm.layers.25.attn.attn_sink_proj.weight": "model-00018-of-00048.safetensors", + "model.llm.layers.25.attn.gate_attn.weight": "model-00018-of-00048.safetensors", + "model.llm.layers.25.attn.k_sconv.weight": "model-00018-of-00048.safetensors", + "model.llm.layers.25.attn.q_l2_clip.scale": "model-00018-of-00048.safetensors", + "model.llm.layers.25.attn.rel_l2_clip.scale": "model-00018-of-00048.safetensors", + "model.llm.layers.25.attn.rel_logits_proj.proj": "model-00018-of-00048.safetensors", + "model.llm.layers.25.attn.v_sconv.weight": "model-00018-of-00048.safetensors", + "model.llm.layers.25.attn.wk_dv.weight": "model-00018-of-00048.safetensors", + "model.llm.layers.25.attn.wo_ud.weight": "model-00018-of-00048.safetensors", + "model.llm.layers.25.attn.wq_du.weight": "model-00018-of-00048.safetensors", + "model.llm.layers.25.attn.wr_du.weight": "model-00018-of-00048.safetensors", + "model.llm.layers.25.attn.wv_dv.weight": "model-00018-of-00048.safetensors", + "model.llm.layers.25.attn_norm.weight": "model-00018-of-00048.safetensors", + "model.llm.layers.25.attn_sconv.weight": "model-00018-of-00048.safetensors", + "model.llm.layers.25.mlp.experts.w13_weight": "model-00019-of-00048.safetensors", + "model.llm.layers.25.mlp.experts.w13_weight_scale": "model-00019-of-00048.safetensors", + "model.llm.layers.25.mlp.experts.w2_weight": "model-00019-of-00048.safetensors", + "model.llm.layers.25.mlp.experts.w2_weight_scale": "model-00019-of-00048.safetensors", + "model.llm.layers.25.mlp.gate.bias": "model-00019-of-00048.safetensors", + "model.llm.layers.25.mlp.gate.weight": "model-00019-of-00048.safetensors", + "model.llm.layers.25.mlp_norm.weight": "model-00019-of-00048.safetensors", + "model.llm.layers.25.mlp_sconv.weight": "model-00019-of-00048.safetensors", + "model.llm.layers.26.attn.attn_sink_proj.weight": "model-00019-of-00048.safetensors", + "model.llm.layers.26.attn.gate_attn.weight": "model-00019-of-00048.safetensors", + "model.llm.layers.26.attn.k_sconv.weight": "model-00019-of-00048.safetensors", + "model.llm.layers.26.attn.q_l2_clip.scale": "model-00019-of-00048.safetensors", + "model.llm.layers.26.attn.rel_l2_clip.scale": "model-00019-of-00048.safetensors", + "model.llm.layers.26.attn.rel_logits_proj.proj": "model-00019-of-00048.safetensors", + "model.llm.layers.26.attn.v_sconv.weight": "model-00019-of-00048.safetensors", + "model.llm.layers.26.attn.wk_dv.weight": "model-00019-of-00048.safetensors", + "model.llm.layers.26.attn.wo_ud.weight": "model-00019-of-00048.safetensors", + "model.llm.layers.26.attn.wq_du.weight": "model-00019-of-00048.safetensors", + "model.llm.layers.26.attn.wr_du.weight": "model-00019-of-00048.safetensors", + "model.llm.layers.26.attn.wv_dv.weight": "model-00019-of-00048.safetensors", + "model.llm.layers.26.attn_norm.weight": "model-00019-of-00048.safetensors", + "model.llm.layers.26.attn_sconv.weight": "model-00019-of-00048.safetensors", + "model.llm.layers.26.mlp.experts.w13_weight": "model-00020-of-00048.safetensors", + "model.llm.layers.26.mlp.experts.w13_weight_scale": "model-00020-of-00048.safetensors", + "model.llm.layers.26.mlp.experts.w2_weight": "model-00020-of-00048.safetensors", + "model.llm.layers.26.mlp.experts.w2_weight_scale": "model-00020-of-00048.safetensors", + "model.llm.layers.26.mlp.gate.bias": "model-00020-of-00048.safetensors", + "model.llm.layers.26.mlp.gate.weight": "model-00020-of-00048.safetensors", + "model.llm.layers.26.mlp_norm.weight": "model-00020-of-00048.safetensors", + "model.llm.layers.26.mlp_sconv.weight": "model-00020-of-00048.safetensors", + "model.llm.layers.27.attn.attn_sink_proj.weight": "model-00020-of-00048.safetensors", + "model.llm.layers.27.attn.gate_attn.weight": "model-00020-of-00048.safetensors", + "model.llm.layers.27.attn.k_sconv.weight": "model-00020-of-00048.safetensors", + "model.llm.layers.27.attn.q_l2_clip.scale": "model-00020-of-00048.safetensors", + "model.llm.layers.27.attn.rel_l2_clip.scale": "model-00020-of-00048.safetensors", + "model.llm.layers.27.attn.rel_logits_proj.proj": "model-00020-of-00048.safetensors", + "model.llm.layers.27.attn.v_sconv.weight": "model-00020-of-00048.safetensors", + "model.llm.layers.27.attn.wk_dv.weight": "model-00020-of-00048.safetensors", + "model.llm.layers.27.attn.wo_ud.weight": "model-00020-of-00048.safetensors", + "model.llm.layers.27.attn.wq_du.weight": "model-00020-of-00048.safetensors", + "model.llm.layers.27.attn.wr_du.weight": "model-00020-of-00048.safetensors", + "model.llm.layers.27.attn.wv_dv.weight": "model-00020-of-00048.safetensors", + "model.llm.layers.27.attn_norm.weight": "model-00020-of-00048.safetensors", + "model.llm.layers.27.attn_sconv.weight": "model-00020-of-00048.safetensors", + "model.llm.layers.27.mlp.experts.w13_weight": "model-00021-of-00048.safetensors", + "model.llm.layers.27.mlp.experts.w13_weight_scale": "model-00021-of-00048.safetensors", + "model.llm.layers.27.mlp.experts.w2_weight": "model-00021-of-00048.safetensors", + "model.llm.layers.27.mlp.experts.w2_weight_scale": "model-00021-of-00048.safetensors", + "model.llm.layers.27.mlp.gate.bias": "model-00021-of-00048.safetensors", + "model.llm.layers.27.mlp.gate.weight": "model-00021-of-00048.safetensors", + "model.llm.layers.27.mlp_norm.weight": "model-00021-of-00048.safetensors", + "model.llm.layers.27.mlp_sconv.weight": "model-00021-of-00048.safetensors", + "model.llm.layers.28.attn.attn_sink_proj.weight": "model-00021-of-00048.safetensors", + "model.llm.layers.28.attn.gate_attn.weight": "model-00021-of-00048.safetensors", + "model.llm.layers.28.attn.k_sconv.weight": "model-00021-of-00048.safetensors", + "model.llm.layers.28.attn.q_l2_clip.scale": "model-00021-of-00048.safetensors", + "model.llm.layers.28.attn.rel_l2_clip.scale": "model-00021-of-00048.safetensors", + "model.llm.layers.28.attn.rel_logits_proj.proj": "model-00021-of-00048.safetensors", + "model.llm.layers.28.attn.v_sconv.weight": "model-00021-of-00048.safetensors", + "model.llm.layers.28.attn.wk_dv.weight": "model-00021-of-00048.safetensors", + "model.llm.layers.28.attn.wo_ud.weight": "model-00021-of-00048.safetensors", + "model.llm.layers.28.attn.wq_du.weight": "model-00021-of-00048.safetensors", + "model.llm.layers.28.attn.wr_du.weight": "model-00021-of-00048.safetensors", + "model.llm.layers.28.attn.wv_dv.weight": "model-00021-of-00048.safetensors", + "model.llm.layers.28.attn_norm.weight": "model-00021-of-00048.safetensors", + "model.llm.layers.28.attn_sconv.weight": "model-00021-of-00048.safetensors", + "model.llm.layers.28.mlp.experts.w13_weight": "model-00022-of-00048.safetensors", + "model.llm.layers.28.mlp.experts.w13_weight_scale": "model-00022-of-00048.safetensors", + "model.llm.layers.28.mlp.experts.w2_weight": "model-00022-of-00048.safetensors", + "model.llm.layers.28.mlp.experts.w2_weight_scale": "model-00022-of-00048.safetensors", + "model.llm.layers.28.mlp.gate.bias": "model-00022-of-00048.safetensors", + "model.llm.layers.28.mlp.gate.weight": "model-00022-of-00048.safetensors", + "model.llm.layers.28.mlp_norm.weight": "model-00022-of-00048.safetensors", + "model.llm.layers.28.mlp_sconv.weight": "model-00022-of-00048.safetensors", + "model.llm.layers.29.attn.attn_sink_proj.weight": "model-00022-of-00048.safetensors", + "model.llm.layers.29.attn.gate_attn.weight": "model-00022-of-00048.safetensors", + "model.llm.layers.29.attn.k_sconv.static_w": "model-00022-of-00048.safetensors", + "model.llm.layers.29.attn.k_sconv.weight_proj": "model-00022-of-00048.safetensors", + "model.llm.layers.29.attn.q_l2_clip.scale": "model-00022-of-00048.safetensors", + "model.llm.layers.29.attn.rel_l2_clip.scale": "model-00022-of-00048.safetensors", + "model.llm.layers.29.attn.rel_logits_proj.proj": "model-00022-of-00048.safetensors", + "model.llm.layers.29.attn.v_sconv.static_w": "model-00022-of-00048.safetensors", + "model.llm.layers.29.attn.v_sconv.weight_proj": "model-00022-of-00048.safetensors", + "model.llm.layers.29.attn.wk_dv.weight": "model-00022-of-00048.safetensors", + "model.llm.layers.29.attn.wo_ud.weight": "model-00022-of-00048.safetensors", + "model.llm.layers.29.attn.wq_du.weight": "model-00022-of-00048.safetensors", + "model.llm.layers.29.attn.wr_du.weight": "model-00022-of-00048.safetensors", + "model.llm.layers.29.attn.wv_dv.weight": "model-00022-of-00048.safetensors", + "model.llm.layers.29.attn_sconv.weight": "model-00022-of-00048.safetensors", + "model.llm.layers.29.k_input_norm.weight": "model-00022-of-00048.safetensors", + "model.llm.layers.29.mlp.experts.w13_weight": "model-00023-of-00048.safetensors", + "model.llm.layers.29.mlp.experts.w13_weight_scale": "model-00023-of-00048.safetensors", + "model.llm.layers.29.mlp.experts.w2_weight": "model-00023-of-00048.safetensors", + "model.llm.layers.29.mlp.experts.w2_weight_scale": "model-00023-of-00048.safetensors", + "model.llm.layers.29.mlp.gate.bias": "model-00023-of-00048.safetensors", + "model.llm.layers.29.mlp.gate.weight": "model-00023-of-00048.safetensors", + "model.llm.layers.29.mlp_norm.weight": "model-00023-of-00048.safetensors", + "model.llm.layers.29.mlp_sconv.weight": "model-00023-of-00048.safetensors", + "model.llm.layers.29.q_input_norm.weight": "model-00023-of-00048.safetensors", + "model.llm.layers.29.v_input_norm.weight": "model-00023-of-00048.safetensors", + "model.llm.layers.3.attn.attn_sink_proj.weight": "model-00023-of-00048.safetensors", + "model.llm.layers.3.attn.gate_attn.weight": "model-00023-of-00048.safetensors", + "model.llm.layers.3.attn.k_sconv.weight": "model-00023-of-00048.safetensors", + "model.llm.layers.3.attn.q_l2_clip.scale": "model-00023-of-00048.safetensors", + "model.llm.layers.3.attn.rel_l2_clip.scale": "model-00023-of-00048.safetensors", + "model.llm.layers.3.attn.rel_logits_proj.proj": "model-00023-of-00048.safetensors", + "model.llm.layers.3.attn.v_sconv.weight": "model-00023-of-00048.safetensors", + "model.llm.layers.3.attn.wk_dv.weight": "model-00023-of-00048.safetensors", + "model.llm.layers.3.attn.wo_ud.weight": "model-00023-of-00048.safetensors", + "model.llm.layers.3.attn.wq_du.weight": "model-00023-of-00048.safetensors", + "model.llm.layers.3.attn.wr_du.weight": "model-00023-of-00048.safetensors", + "model.llm.layers.3.attn.wv_dv.weight": "model-00023-of-00048.safetensors", + "model.llm.layers.3.attn_norm.weight": "model-00023-of-00048.safetensors", + "model.llm.layers.3.attn_sconv.weight": "model-00023-of-00048.safetensors", + "model.llm.layers.3.mlp.experts.w13_weight": "model-00024-of-00048.safetensors", + "model.llm.layers.3.mlp.experts.w13_weight_scale": "model-00024-of-00048.safetensors", + "model.llm.layers.3.mlp.experts.w2_weight": "model-00024-of-00048.safetensors", + "model.llm.layers.3.mlp.experts.w2_weight_scale": "model-00024-of-00048.safetensors", + "model.llm.layers.3.mlp.gate.bias": "model-00024-of-00048.safetensors", + "model.llm.layers.3.mlp.gate.weight": "model-00024-of-00048.safetensors", + "model.llm.layers.3.mlp_norm.weight": "model-00024-of-00048.safetensors", + "model.llm.layers.3.mlp_sconv.weight": "model-00024-of-00048.safetensors", + "model.llm.layers.30.attn.attn_sink_proj.weight": "model-00024-of-00048.safetensors", + "model.llm.layers.30.attn.gate_attn.weight": "model-00024-of-00048.safetensors", + "model.llm.layers.30.attn.k_sconv.weight": "model-00024-of-00048.safetensors", + "model.llm.layers.30.attn.q_l2_clip.scale": "model-00024-of-00048.safetensors", + "model.llm.layers.30.attn.rel_l2_clip.scale": "model-00024-of-00048.safetensors", + "model.llm.layers.30.attn.rel_logits_proj.proj": "model-00024-of-00048.safetensors", + "model.llm.layers.30.attn.v_sconv.weight": "model-00024-of-00048.safetensors", + "model.llm.layers.30.attn.wk_dv.weight": "model-00024-of-00048.safetensors", + "model.llm.layers.30.attn.wo_ud.weight": "model-00024-of-00048.safetensors", + "model.llm.layers.30.attn.wq_du.weight": "model-00024-of-00048.safetensors", + "model.llm.layers.30.attn.wr_du.weight": "model-00024-of-00048.safetensors", + "model.llm.layers.30.attn.wv_dv.weight": "model-00024-of-00048.safetensors", + "model.llm.layers.30.attn_norm.weight": "model-00024-of-00048.safetensors", + "model.llm.layers.30.attn_sconv.weight": "model-00024-of-00048.safetensors", + "model.llm.layers.30.mlp.experts.w13_weight": "model-00025-of-00048.safetensors", + "model.llm.layers.30.mlp.experts.w13_weight_scale": "model-00025-of-00048.safetensors", + "model.llm.layers.30.mlp.experts.w2_weight": "model-00025-of-00048.safetensors", + "model.llm.layers.30.mlp.experts.w2_weight_scale": "model-00025-of-00048.safetensors", + "model.llm.layers.30.mlp.gate.bias": "model-00025-of-00048.safetensors", + "model.llm.layers.30.mlp.gate.weight": "model-00025-of-00048.safetensors", + "model.llm.layers.30.mlp_norm.weight": "model-00025-of-00048.safetensors", + "model.llm.layers.30.mlp_sconv.weight": "model-00025-of-00048.safetensors", + "model.llm.layers.31.attn.attn_sink_proj.weight": "model-00025-of-00048.safetensors", + "model.llm.layers.31.attn.gate_attn.weight": "model-00025-of-00048.safetensors", + "model.llm.layers.31.attn.k_sconv.weight": "model-00025-of-00048.safetensors", + "model.llm.layers.31.attn.q_l2_clip.scale": "model-00025-of-00048.safetensors", + "model.llm.layers.31.attn.rel_l2_clip.scale": "model-00025-of-00048.safetensors", + "model.llm.layers.31.attn.rel_logits_proj.proj": "model-00025-of-00048.safetensors", + "model.llm.layers.31.attn.v_sconv.weight": "model-00025-of-00048.safetensors", + "model.llm.layers.31.attn.wk_dv.weight": "model-00025-of-00048.safetensors", + "model.llm.layers.31.attn.wo_ud.weight": "model-00025-of-00048.safetensors", + "model.llm.layers.31.attn.wq_du.weight": "model-00025-of-00048.safetensors", + "model.llm.layers.31.attn.wr_du.weight": "model-00025-of-00048.safetensors", + "model.llm.layers.31.attn.wv_dv.weight": "model-00025-of-00048.safetensors", + "model.llm.layers.31.attn_norm.weight": "model-00025-of-00048.safetensors", + "model.llm.layers.31.attn_sconv.weight": "model-00025-of-00048.safetensors", + "model.llm.layers.31.mlp.experts.w13_weight": "model-00026-of-00048.safetensors", + "model.llm.layers.31.mlp.experts.w13_weight_scale": "model-00026-of-00048.safetensors", + "model.llm.layers.31.mlp.experts.w2_weight": "model-00026-of-00048.safetensors", + "model.llm.layers.31.mlp.experts.w2_weight_scale": "model-00026-of-00048.safetensors", + "model.llm.layers.31.mlp.gate.bias": "model-00026-of-00048.safetensors", + "model.llm.layers.31.mlp.gate.weight": "model-00026-of-00048.safetensors", + "model.llm.layers.31.mlp_norm.weight": "model-00026-of-00048.safetensors", + "model.llm.layers.31.mlp_sconv.weight": "model-00026-of-00048.safetensors", + "model.llm.layers.32.attn.attn_sink_proj.weight": "model-00026-of-00048.safetensors", + "model.llm.layers.32.attn.gate_attn.weight": "model-00026-of-00048.safetensors", + "model.llm.layers.32.attn.k_sconv.weight": "model-00026-of-00048.safetensors", + "model.llm.layers.32.attn.q_l2_clip.scale": "model-00026-of-00048.safetensors", + "model.llm.layers.32.attn.rel_l2_clip.scale": "model-00026-of-00048.safetensors", + "model.llm.layers.32.attn.rel_logits_proj.proj": "model-00026-of-00048.safetensors", + "model.llm.layers.32.attn.v_sconv.weight": "model-00026-of-00048.safetensors", + "model.llm.layers.32.attn.wk_dv.weight": "model-00026-of-00048.safetensors", + "model.llm.layers.32.attn.wo_ud.weight": "model-00026-of-00048.safetensors", + "model.llm.layers.32.attn.wq_du.weight": "model-00026-of-00048.safetensors", + "model.llm.layers.32.attn.wr_du.weight": "model-00026-of-00048.safetensors", + "model.llm.layers.32.attn.wv_dv.weight": "model-00026-of-00048.safetensors", + "model.llm.layers.32.attn_norm.weight": "model-00026-of-00048.safetensors", + "model.llm.layers.32.attn_sconv.weight": "model-00026-of-00048.safetensors", + "model.llm.layers.32.mlp.experts.w13_weight": "model-00027-of-00048.safetensors", + "model.llm.layers.32.mlp.experts.w13_weight_scale": "model-00027-of-00048.safetensors", + "model.llm.layers.32.mlp.experts.w2_weight": "model-00027-of-00048.safetensors", + "model.llm.layers.32.mlp.experts.w2_weight_scale": "model-00027-of-00048.safetensors", + "model.llm.layers.32.mlp.gate.bias": "model-00027-of-00048.safetensors", + "model.llm.layers.32.mlp.gate.weight": "model-00027-of-00048.safetensors", + "model.llm.layers.32.mlp_norm.weight": "model-00027-of-00048.safetensors", + "model.llm.layers.32.mlp_sconv.weight": "model-00027-of-00048.safetensors", + "model.llm.layers.33.attn.attn_sink_proj.weight": "model-00027-of-00048.safetensors", + "model.llm.layers.33.attn.gate_attn.weight": "model-00027-of-00048.safetensors", + "model.llm.layers.33.attn.k_sconv.weight": "model-00027-of-00048.safetensors", + "model.llm.layers.33.attn.q_l2_clip.scale": "model-00027-of-00048.safetensors", + "model.llm.layers.33.attn.rel_l2_clip.scale": "model-00027-of-00048.safetensors", + "model.llm.layers.33.attn.rel_logits_proj.proj": "model-00027-of-00048.safetensors", + "model.llm.layers.33.attn.v_sconv.weight": "model-00027-of-00048.safetensors", + "model.llm.layers.33.attn.wk_dv.weight": "model-00027-of-00048.safetensors", + "model.llm.layers.33.attn.wo_ud.weight": "model-00027-of-00048.safetensors", + "model.llm.layers.33.attn.wq_du.weight": "model-00027-of-00048.safetensors", + "model.llm.layers.33.attn.wr_du.weight": "model-00027-of-00048.safetensors", + "model.llm.layers.33.attn.wv_dv.weight": "model-00027-of-00048.safetensors", + "model.llm.layers.33.attn_norm.weight": "model-00027-of-00048.safetensors", + "model.llm.layers.33.attn_sconv.weight": "model-00027-of-00048.safetensors", + "model.llm.layers.33.mlp.experts.w13_weight": "model-00028-of-00048.safetensors", + "model.llm.layers.33.mlp.experts.w13_weight_scale": "model-00028-of-00048.safetensors", + "model.llm.layers.33.mlp.experts.w2_weight": "model-00028-of-00048.safetensors", + "model.llm.layers.33.mlp.experts.w2_weight_scale": "model-00028-of-00048.safetensors", + "model.llm.layers.33.mlp.gate.bias": "model-00028-of-00048.safetensors", + "model.llm.layers.33.mlp.gate.weight": "model-00028-of-00048.safetensors", + "model.llm.layers.33.mlp_norm.weight": "model-00028-of-00048.safetensors", + "model.llm.layers.33.mlp_sconv.weight": "model-00028-of-00048.safetensors", + "model.llm.layers.34.attn.attn_sink_proj.weight": "model-00028-of-00048.safetensors", + "model.llm.layers.34.attn.gate_attn.weight": "model-00028-of-00048.safetensors", + "model.llm.layers.34.attn.k_sconv.weight": "model-00028-of-00048.safetensors", + "model.llm.layers.34.attn.q_l2_clip.scale": "model-00028-of-00048.safetensors", + "model.llm.layers.34.attn.rel_l2_clip.scale": "model-00028-of-00048.safetensors", + "model.llm.layers.34.attn.rel_logits_proj.proj": "model-00028-of-00048.safetensors", + "model.llm.layers.34.attn.v_sconv.weight": "model-00028-of-00048.safetensors", + "model.llm.layers.34.attn.wk_dv.weight": "model-00028-of-00048.safetensors", + "model.llm.layers.34.attn.wo_ud.weight": "model-00028-of-00048.safetensors", + "model.llm.layers.34.attn.wq_du.weight": "model-00028-of-00048.safetensors", + "model.llm.layers.34.attn.wr_du.weight": "model-00028-of-00048.safetensors", + "model.llm.layers.34.attn.wv_dv.weight": "model-00028-of-00048.safetensors", + "model.llm.layers.34.attn_norm.weight": "model-00028-of-00048.safetensors", + "model.llm.layers.34.attn_sconv.weight": "model-00028-of-00048.safetensors", + "model.llm.layers.34.mlp.experts.w13_weight": "model-00029-of-00048.safetensors", + "model.llm.layers.34.mlp.experts.w13_weight_scale": "model-00029-of-00048.safetensors", + "model.llm.layers.34.mlp.experts.w2_weight": "model-00029-of-00048.safetensors", + "model.llm.layers.34.mlp.experts.w2_weight_scale": "model-00029-of-00048.safetensors", + "model.llm.layers.34.mlp.gate.bias": "model-00029-of-00048.safetensors", + "model.llm.layers.34.mlp.gate.weight": "model-00029-of-00048.safetensors", + "model.llm.layers.34.mlp_norm.weight": "model-00029-of-00048.safetensors", + "model.llm.layers.34.mlp_sconv.weight": "model-00029-of-00048.safetensors", + "model.llm.layers.35.attn.attn_sink_proj.weight": "model-00029-of-00048.safetensors", + "model.llm.layers.35.attn.gate_attn.weight": "model-00029-of-00048.safetensors", + "model.llm.layers.35.attn.k_sconv.static_w": "model-00029-of-00048.safetensors", + "model.llm.layers.35.attn.k_sconv.weight_proj": "model-00029-of-00048.safetensors", + "model.llm.layers.35.attn.q_l2_clip.scale": "model-00029-of-00048.safetensors", + "model.llm.layers.35.attn.rel_l2_clip.scale": "model-00029-of-00048.safetensors", + "model.llm.layers.35.attn.rel_logits_proj.proj": "model-00029-of-00048.safetensors", + "model.llm.layers.35.attn.v_sconv.static_w": "model-00029-of-00048.safetensors", + "model.llm.layers.35.attn.v_sconv.weight_proj": "model-00029-of-00048.safetensors", + "model.llm.layers.35.attn.wk_dv.weight": "model-00029-of-00048.safetensors", + "model.llm.layers.35.attn.wo_ud.weight": "model-00029-of-00048.safetensors", + "model.llm.layers.35.attn.wq_du.weight": "model-00029-of-00048.safetensors", + "model.llm.layers.35.attn.wr_du.weight": "model-00029-of-00048.safetensors", + "model.llm.layers.35.attn.wv_dv.weight": "model-00029-of-00048.safetensors", + "model.llm.layers.35.attn_sconv.weight": "model-00029-of-00048.safetensors", + "model.llm.layers.35.k_input_norm.weight": "model-00029-of-00048.safetensors", + "model.llm.layers.35.mlp.experts.w13_weight": "model-00030-of-00048.safetensors", + "model.llm.layers.35.mlp.experts.w13_weight_scale": "model-00030-of-00048.safetensors", + "model.llm.layers.35.mlp.experts.w2_weight": "model-00030-of-00048.safetensors", + "model.llm.layers.35.mlp.experts.w2_weight_scale": "model-00030-of-00048.safetensors", + "model.llm.layers.35.mlp.gate.bias": "model-00030-of-00048.safetensors", + "model.llm.layers.35.mlp.gate.weight": "model-00030-of-00048.safetensors", + "model.llm.layers.35.mlp_norm.weight": "model-00030-of-00048.safetensors", + "model.llm.layers.35.mlp_sconv.weight": "model-00030-of-00048.safetensors", + "model.llm.layers.35.q_input_norm.weight": "model-00030-of-00048.safetensors", + "model.llm.layers.35.v_input_norm.weight": "model-00030-of-00048.safetensors", + "model.llm.layers.36.attn.attn_sink_proj.weight": "model-00030-of-00048.safetensors", + "model.llm.layers.36.attn.gate_attn.weight": "model-00030-of-00048.safetensors", + "model.llm.layers.36.attn.k_sconv.weight": "model-00030-of-00048.safetensors", + "model.llm.layers.36.attn.q_l2_clip.scale": "model-00030-of-00048.safetensors", + "model.llm.layers.36.attn.rel_l2_clip.scale": "model-00030-of-00048.safetensors", + "model.llm.layers.36.attn.rel_logits_proj.proj": "model-00030-of-00048.safetensors", + "model.llm.layers.36.attn.v_sconv.weight": "model-00030-of-00048.safetensors", + "model.llm.layers.36.attn.wk_dv.weight": "model-00030-of-00048.safetensors", + "model.llm.layers.36.attn.wo_ud.weight": "model-00030-of-00048.safetensors", + "model.llm.layers.36.attn.wq_du.weight": "model-00030-of-00048.safetensors", + "model.llm.layers.36.attn.wr_du.weight": "model-00030-of-00048.safetensors", + "model.llm.layers.36.attn.wv_dv.weight": "model-00030-of-00048.safetensors", + "model.llm.layers.36.attn_norm.weight": "model-00030-of-00048.safetensors", + "model.llm.layers.36.attn_sconv.weight": "model-00030-of-00048.safetensors", + "model.llm.layers.36.mlp.experts.w13_weight": "model-00031-of-00048.safetensors", + "model.llm.layers.36.mlp.experts.w13_weight_scale": "model-00031-of-00048.safetensors", + "model.llm.layers.36.mlp.experts.w2_weight": "model-00031-of-00048.safetensors", + "model.llm.layers.36.mlp.experts.w2_weight_scale": "model-00031-of-00048.safetensors", + "model.llm.layers.36.mlp.gate.bias": "model-00031-of-00048.safetensors", + "model.llm.layers.36.mlp.gate.weight": "model-00031-of-00048.safetensors", + "model.llm.layers.36.mlp_norm.weight": "model-00031-of-00048.safetensors", + "model.llm.layers.36.mlp_sconv.weight": "model-00031-of-00048.safetensors", + "model.llm.layers.37.attn.attn_sink_proj.weight": "model-00031-of-00048.safetensors", + "model.llm.layers.37.attn.gate_attn.weight": "model-00031-of-00048.safetensors", + "model.llm.layers.37.attn.k_sconv.weight": "model-00031-of-00048.safetensors", + "model.llm.layers.37.attn.q_l2_clip.scale": "model-00031-of-00048.safetensors", + "model.llm.layers.37.attn.rel_l2_clip.scale": "model-00031-of-00048.safetensors", + "model.llm.layers.37.attn.rel_logits_proj.proj": "model-00031-of-00048.safetensors", + "model.llm.layers.37.attn.v_sconv.weight": "model-00031-of-00048.safetensors", + "model.llm.layers.37.attn.wk_dv.weight": "model-00031-of-00048.safetensors", + "model.llm.layers.37.attn.wo_ud.weight": "model-00031-of-00048.safetensors", + "model.llm.layers.37.attn.wq_du.weight": "model-00031-of-00048.safetensors", + "model.llm.layers.37.attn.wr_du.weight": "model-00031-of-00048.safetensors", + "model.llm.layers.37.attn.wv_dv.weight": "model-00031-of-00048.safetensors", + "model.llm.layers.37.attn_norm.weight": "model-00031-of-00048.safetensors", + "model.llm.layers.37.attn_sconv.weight": "model-00031-of-00048.safetensors", + "model.llm.layers.37.mlp.experts.w13_weight": "model-00032-of-00048.safetensors", + "model.llm.layers.37.mlp.experts.w13_weight_scale": "model-00032-of-00048.safetensors", + "model.llm.layers.37.mlp.experts.w2_weight": "model-00032-of-00048.safetensors", + "model.llm.layers.37.mlp.experts.w2_weight_scale": "model-00032-of-00048.safetensors", + "model.llm.layers.37.mlp.gate.bias": "model-00032-of-00048.safetensors", + "model.llm.layers.37.mlp.gate.weight": "model-00032-of-00048.safetensors", + "model.llm.layers.37.mlp_norm.weight": "model-00032-of-00048.safetensors", + "model.llm.layers.37.mlp_sconv.weight": "model-00032-of-00048.safetensors", + "model.llm.layers.38.attn.attn_sink_proj.weight": "model-00032-of-00048.safetensors", + "model.llm.layers.38.attn.gate_attn.weight": "model-00032-of-00048.safetensors", + "model.llm.layers.38.attn.k_sconv.weight": "model-00032-of-00048.safetensors", + "model.llm.layers.38.attn.q_l2_clip.scale": "model-00032-of-00048.safetensors", + "model.llm.layers.38.attn.rel_l2_clip.scale": "model-00032-of-00048.safetensors", + "model.llm.layers.38.attn.rel_logits_proj.proj": "model-00032-of-00048.safetensors", + "model.llm.layers.38.attn.v_sconv.weight": "model-00032-of-00048.safetensors", + "model.llm.layers.38.attn.wk_dv.weight": "model-00032-of-00048.safetensors", + "model.llm.layers.38.attn.wo_ud.weight": "model-00032-of-00048.safetensors", + "model.llm.layers.38.attn.wq_du.weight": "model-00032-of-00048.safetensors", + "model.llm.layers.38.attn.wr_du.weight": "model-00032-of-00048.safetensors", + "model.llm.layers.38.attn.wv_dv.weight": "model-00032-of-00048.safetensors", + "model.llm.layers.38.attn_norm.weight": "model-00032-of-00048.safetensors", + "model.llm.layers.38.attn_sconv.weight": "model-00032-of-00048.safetensors", + "model.llm.layers.38.mlp.experts.w13_weight": "model-00033-of-00048.safetensors", + "model.llm.layers.38.mlp.experts.w13_weight_scale": "model-00033-of-00048.safetensors", + "model.llm.layers.38.mlp.experts.w2_weight": "model-00033-of-00048.safetensors", + "model.llm.layers.38.mlp.experts.w2_weight_scale": "model-00033-of-00048.safetensors", + "model.llm.layers.38.mlp.gate.bias": "model-00033-of-00048.safetensors", + "model.llm.layers.38.mlp.gate.weight": "model-00033-of-00048.safetensors", + "model.llm.layers.38.mlp_norm.weight": "model-00033-of-00048.safetensors", + "model.llm.layers.38.mlp_sconv.weight": "model-00033-of-00048.safetensors", + "model.llm.layers.39.attn.attn_sink_proj.weight": "model-00033-of-00048.safetensors", + "model.llm.layers.39.attn.gate_attn.weight": "model-00033-of-00048.safetensors", + "model.llm.layers.39.attn.k_sconv.weight": "model-00033-of-00048.safetensors", + "model.llm.layers.39.attn.q_l2_clip.scale": "model-00033-of-00048.safetensors", + "model.llm.layers.39.attn.rel_l2_clip.scale": "model-00033-of-00048.safetensors", + "model.llm.layers.39.attn.rel_logits_proj.proj": "model-00033-of-00048.safetensors", + "model.llm.layers.39.attn.v_sconv.weight": "model-00033-of-00048.safetensors", + "model.llm.layers.39.attn.wk_dv.weight": "model-00033-of-00048.safetensors", + "model.llm.layers.39.attn.wo_ud.weight": "model-00033-of-00048.safetensors", + "model.llm.layers.39.attn.wq_du.weight": "model-00033-of-00048.safetensors", + "model.llm.layers.39.attn.wr_du.weight": "model-00033-of-00048.safetensors", + "model.llm.layers.39.attn.wv_dv.weight": "model-00033-of-00048.safetensors", + "model.llm.layers.39.attn_norm.weight": "model-00033-of-00048.safetensors", + "model.llm.layers.39.attn_sconv.weight": "model-00033-of-00048.safetensors", + "model.llm.layers.39.mlp.experts.w13_weight": "model-00034-of-00048.safetensors", + "model.llm.layers.39.mlp.experts.w13_weight_scale": "model-00034-of-00048.safetensors", + "model.llm.layers.39.mlp.experts.w2_weight": "model-00034-of-00048.safetensors", + "model.llm.layers.39.mlp.experts.w2_weight_scale": "model-00034-of-00048.safetensors", + "model.llm.layers.39.mlp.gate.bias": "model-00034-of-00048.safetensors", + "model.llm.layers.39.mlp.gate.weight": "model-00034-of-00048.safetensors", + "model.llm.layers.39.mlp_norm.weight": "model-00034-of-00048.safetensors", + "model.llm.layers.39.mlp_sconv.weight": "model-00034-of-00048.safetensors", + "model.llm.layers.4.attn.attn_sink_proj.weight": "model-00034-of-00048.safetensors", + "model.llm.layers.4.attn.gate_attn.weight": "model-00034-of-00048.safetensors", + "model.llm.layers.4.attn.k_sconv.weight": "model-00034-of-00048.safetensors", + "model.llm.layers.4.attn.q_l2_clip.scale": "model-00034-of-00048.safetensors", + "model.llm.layers.4.attn.rel_l2_clip.scale": "model-00034-of-00048.safetensors", + "model.llm.layers.4.attn.rel_logits_proj.proj": "model-00034-of-00048.safetensors", + "model.llm.layers.4.attn.v_sconv.weight": "model-00034-of-00048.safetensors", + "model.llm.layers.4.attn.wk_dv.weight": "model-00034-of-00048.safetensors", + "model.llm.layers.4.attn.wo_ud.weight": "model-00034-of-00048.safetensors", + "model.llm.layers.4.attn.wq_du.weight": "model-00034-of-00048.safetensors", + "model.llm.layers.4.attn.wr_du.weight": "model-00034-of-00048.safetensors", + "model.llm.layers.4.attn.wv_dv.weight": "model-00034-of-00048.safetensors", + "model.llm.layers.4.attn_norm.weight": "model-00034-of-00048.safetensors", + "model.llm.layers.4.attn_sconv.weight": "model-00034-of-00048.safetensors", + "model.llm.layers.4.mlp.experts.w13_weight": "model-00035-of-00048.safetensors", + "model.llm.layers.4.mlp.experts.w13_weight_scale": "model-00035-of-00048.safetensors", + "model.llm.layers.4.mlp.experts.w2_weight": "model-00035-of-00048.safetensors", + "model.llm.layers.4.mlp.experts.w2_weight_scale": "model-00035-of-00048.safetensors", + "model.llm.layers.4.mlp.gate.bias": "model-00035-of-00048.safetensors", + "model.llm.layers.4.mlp.gate.weight": "model-00035-of-00048.safetensors", + "model.llm.layers.4.mlp_norm.weight": "model-00035-of-00048.safetensors", + "model.llm.layers.4.mlp_sconv.weight": "model-00035-of-00048.safetensors", + "model.llm.layers.40.attn.attn_sink_proj.weight": "model-00035-of-00048.safetensors", + "model.llm.layers.40.attn.gate_attn.weight": "model-00035-of-00048.safetensors", + "model.llm.layers.40.attn.k_sconv.weight": "model-00035-of-00048.safetensors", + "model.llm.layers.40.attn.q_l2_clip.scale": "model-00035-of-00048.safetensors", + "model.llm.layers.40.attn.rel_l2_clip.scale": "model-00035-of-00048.safetensors", + "model.llm.layers.40.attn.rel_logits_proj.proj": "model-00035-of-00048.safetensors", + "model.llm.layers.40.attn.v_sconv.weight": "model-00035-of-00048.safetensors", + "model.llm.layers.40.attn.wk_dv.weight": "model-00035-of-00048.safetensors", + "model.llm.layers.40.attn.wo_ud.weight": "model-00035-of-00048.safetensors", + "model.llm.layers.40.attn.wq_du.weight": "model-00035-of-00048.safetensors", + "model.llm.layers.40.attn.wr_du.weight": "model-00035-of-00048.safetensors", + "model.llm.layers.40.attn.wv_dv.weight": "model-00035-of-00048.safetensors", + "model.llm.layers.40.attn_norm.weight": "model-00035-of-00048.safetensors", + "model.llm.layers.40.attn_sconv.weight": "model-00035-of-00048.safetensors", + "model.llm.layers.40.mlp.experts.w13_weight": "model-00036-of-00048.safetensors", + "model.llm.layers.40.mlp.experts.w13_weight_scale": "model-00036-of-00048.safetensors", + "model.llm.layers.40.mlp.experts.w2_weight": "model-00036-of-00048.safetensors", + "model.llm.layers.40.mlp.experts.w2_weight_scale": "model-00036-of-00048.safetensors", + "model.llm.layers.40.mlp.gate.bias": "model-00036-of-00048.safetensors", + "model.llm.layers.40.mlp.gate.weight": "model-00036-of-00048.safetensors", + "model.llm.layers.40.mlp_norm.weight": "model-00036-of-00048.safetensors", + "model.llm.layers.40.mlp_sconv.weight": "model-00036-of-00048.safetensors", + "model.llm.layers.41.attn.attn_sink_proj.weight": "model-00036-of-00048.safetensors", + "model.llm.layers.41.attn.gate_attn.weight": "model-00036-of-00048.safetensors", + "model.llm.layers.41.attn.k_sconv.static_w": "model-00036-of-00048.safetensors", + "model.llm.layers.41.attn.k_sconv.weight_proj": "model-00036-of-00048.safetensors", + "model.llm.layers.41.attn.q_l2_clip.scale": "model-00036-of-00048.safetensors", + "model.llm.layers.41.attn.rel_l2_clip.scale": "model-00036-of-00048.safetensors", + "model.llm.layers.41.attn.rel_logits_proj.proj": "model-00036-of-00048.safetensors", + "model.llm.layers.41.attn.v_sconv.static_w": "model-00036-of-00048.safetensors", + "model.llm.layers.41.attn.v_sconv.weight_proj": "model-00036-of-00048.safetensors", + "model.llm.layers.41.attn.wk_dv.weight": "model-00036-of-00048.safetensors", + "model.llm.layers.41.attn.wo_ud.weight": "model-00036-of-00048.safetensors", + "model.llm.layers.41.attn.wq_du.weight": "model-00036-of-00048.safetensors", + "model.llm.layers.41.attn.wr_du.weight": "model-00036-of-00048.safetensors", + "model.llm.layers.41.attn.wv_dv.weight": "model-00036-of-00048.safetensors", + "model.llm.layers.41.attn_sconv.weight": "model-00036-of-00048.safetensors", + "model.llm.layers.41.k_input_norm.weight": "model-00036-of-00048.safetensors", + "model.llm.layers.41.mlp.experts.w13_weight": "model-00037-of-00048.safetensors", + "model.llm.layers.41.mlp.experts.w13_weight_scale": "model-00037-of-00048.safetensors", + "model.llm.layers.41.mlp.experts.w2_weight": "model-00037-of-00048.safetensors", + "model.llm.layers.41.mlp.experts.w2_weight_scale": "model-00037-of-00048.safetensors", + "model.llm.layers.41.mlp.gate.bias": "model-00037-of-00048.safetensors", + "model.llm.layers.41.mlp.gate.weight": "model-00037-of-00048.safetensors", + "model.llm.layers.41.mlp_norm.weight": "model-00037-of-00048.safetensors", + "model.llm.layers.41.mlp_sconv.weight": "model-00037-of-00048.safetensors", + "model.llm.layers.41.q_input_norm.weight": "model-00037-of-00048.safetensors", + "model.llm.layers.41.v_input_norm.weight": "model-00037-of-00048.safetensors", + "model.llm.layers.42.attn.attn_sink_proj.weight": "model-00037-of-00048.safetensors", + "model.llm.layers.42.attn.gate_attn.weight": "model-00037-of-00048.safetensors", + "model.llm.layers.42.attn.k_sconv.weight": "model-00037-of-00048.safetensors", + "model.llm.layers.42.attn.q_l2_clip.scale": "model-00037-of-00048.safetensors", + "model.llm.layers.42.attn.rel_l2_clip.scale": "model-00037-of-00048.safetensors", + "model.llm.layers.42.attn.rel_logits_proj.proj": "model-00037-of-00048.safetensors", + "model.llm.layers.42.attn.v_sconv.weight": "model-00037-of-00048.safetensors", + "model.llm.layers.42.attn.wk_dv.weight": "model-00037-of-00048.safetensors", + "model.llm.layers.42.attn.wo_ud.weight": "model-00037-of-00048.safetensors", + "model.llm.layers.42.attn.wq_du.weight": "model-00037-of-00048.safetensors", + "model.llm.layers.42.attn.wr_du.weight": "model-00037-of-00048.safetensors", + "model.llm.layers.42.attn.wv_dv.weight": "model-00037-of-00048.safetensors", + "model.llm.layers.42.attn_norm.weight": "model-00037-of-00048.safetensors", + "model.llm.layers.42.attn_sconv.weight": "model-00037-of-00048.safetensors", + "model.llm.layers.42.mlp.experts.w13_weight": "model-00038-of-00048.safetensors", + "model.llm.layers.42.mlp.experts.w13_weight_scale": "model-00038-of-00048.safetensors", + "model.llm.layers.42.mlp.experts.w2_weight": "model-00038-of-00048.safetensors", + "model.llm.layers.42.mlp.experts.w2_weight_scale": "model-00038-of-00048.safetensors", + "model.llm.layers.42.mlp.gate.bias": "model-00038-of-00048.safetensors", + "model.llm.layers.42.mlp.gate.weight": "model-00038-of-00048.safetensors", + "model.llm.layers.42.mlp_norm.weight": "model-00038-of-00048.safetensors", + "model.llm.layers.42.mlp_sconv.weight": "model-00038-of-00048.safetensors", + "model.llm.layers.43.attn.attn_sink_proj.weight": "model-00038-of-00048.safetensors", + "model.llm.layers.43.attn.gate_attn.weight": "model-00038-of-00048.safetensors", + "model.llm.layers.43.attn.k_sconv.weight": "model-00038-of-00048.safetensors", + "model.llm.layers.43.attn.q_l2_clip.scale": "model-00038-of-00048.safetensors", + "model.llm.layers.43.attn.rel_l2_clip.scale": "model-00038-of-00048.safetensors", + "model.llm.layers.43.attn.rel_logits_proj.proj": "model-00038-of-00048.safetensors", + "model.llm.layers.43.attn.v_sconv.weight": "model-00038-of-00048.safetensors", + "model.llm.layers.43.attn.wk_dv.weight": "model-00038-of-00048.safetensors", + "model.llm.layers.43.attn.wo_ud.weight": "model-00038-of-00048.safetensors", + "model.llm.layers.43.attn.wq_du.weight": "model-00038-of-00048.safetensors", + "model.llm.layers.43.attn.wr_du.weight": "model-00038-of-00048.safetensors", + "model.llm.layers.43.attn.wv_dv.weight": "model-00038-of-00048.safetensors", + "model.llm.layers.43.attn_norm.weight": "model-00038-of-00048.safetensors", + "model.llm.layers.43.attn_sconv.weight": "model-00038-of-00048.safetensors", + "model.llm.layers.43.mlp.experts.w13_weight": "model-00039-of-00048.safetensors", + "model.llm.layers.43.mlp.experts.w13_weight_scale": "model-00039-of-00048.safetensors", + "model.llm.layers.43.mlp.experts.w2_weight": "model-00039-of-00048.safetensors", + "model.llm.layers.43.mlp.experts.w2_weight_scale": "model-00039-of-00048.safetensors", + "model.llm.layers.43.mlp.gate.bias": "model-00039-of-00048.safetensors", + "model.llm.layers.43.mlp.gate.weight": "model-00039-of-00048.safetensors", + "model.llm.layers.43.mlp_norm.weight": "model-00039-of-00048.safetensors", + "model.llm.layers.43.mlp_sconv.weight": "model-00039-of-00048.safetensors", + "model.llm.layers.44.attn.attn_sink_proj.weight": "model-00039-of-00048.safetensors", + "model.llm.layers.44.attn.gate_attn.weight": "model-00039-of-00048.safetensors", + "model.llm.layers.44.attn.k_sconv.weight": "model-00039-of-00048.safetensors", + "model.llm.layers.44.attn.q_l2_clip.scale": "model-00039-of-00048.safetensors", + "model.llm.layers.44.attn.rel_l2_clip.scale": "model-00039-of-00048.safetensors", + "model.llm.layers.44.attn.rel_logits_proj.proj": "model-00039-of-00048.safetensors", + "model.llm.layers.44.attn.v_sconv.weight": "model-00039-of-00048.safetensors", + "model.llm.layers.44.attn.wk_dv.weight": "model-00039-of-00048.safetensors", + "model.llm.layers.44.attn.wo_ud.weight": "model-00039-of-00048.safetensors", + "model.llm.layers.44.attn.wq_du.weight": "model-00039-of-00048.safetensors", + "model.llm.layers.44.attn.wr_du.weight": "model-00039-of-00048.safetensors", + "model.llm.layers.44.attn.wv_dv.weight": "model-00039-of-00048.safetensors", + "model.llm.layers.44.attn_norm.weight": "model-00039-of-00048.safetensors", + "model.llm.layers.44.attn_sconv.weight": "model-00039-of-00048.safetensors", + "model.llm.layers.44.mlp.experts.w13_weight": "model-00040-of-00048.safetensors", + "model.llm.layers.44.mlp.experts.w13_weight_scale": "model-00040-of-00048.safetensors", + "model.llm.layers.44.mlp.experts.w2_weight": "model-00040-of-00048.safetensors", + "model.llm.layers.44.mlp.experts.w2_weight_scale": "model-00040-of-00048.safetensors", + "model.llm.layers.44.mlp.gate.bias": "model-00040-of-00048.safetensors", + "model.llm.layers.44.mlp.gate.weight": "model-00040-of-00048.safetensors", + "model.llm.layers.44.mlp_norm.weight": "model-00040-of-00048.safetensors", + "model.llm.layers.44.mlp_sconv.weight": "model-00040-of-00048.safetensors", + "model.llm.layers.45.attn.attn_sink_proj.weight": "model-00040-of-00048.safetensors", + "model.llm.layers.45.attn.gate_attn.weight": "model-00040-of-00048.safetensors", + "model.llm.layers.45.attn.k_sconv.weight": "model-00040-of-00048.safetensors", + "model.llm.layers.45.attn.q_l2_clip.scale": "model-00040-of-00048.safetensors", + "model.llm.layers.45.attn.rel_l2_clip.scale": "model-00040-of-00048.safetensors", + "model.llm.layers.45.attn.rel_logits_proj.proj": "model-00040-of-00048.safetensors", + "model.llm.layers.45.attn.v_sconv.weight": "model-00040-of-00048.safetensors", + "model.llm.layers.45.attn.wk_dv.weight": "model-00040-of-00048.safetensors", + "model.llm.layers.45.attn.wo_ud.weight": "model-00040-of-00048.safetensors", + "model.llm.layers.45.attn.wq_du.weight": "model-00040-of-00048.safetensors", + "model.llm.layers.45.attn.wr_du.weight": "model-00040-of-00048.safetensors", + "model.llm.layers.45.attn.wv_dv.weight": "model-00040-of-00048.safetensors", + "model.llm.layers.45.attn_norm.weight": "model-00040-of-00048.safetensors", + "model.llm.layers.45.attn_sconv.weight": "model-00040-of-00048.safetensors", + "model.llm.layers.45.mlp.experts.w13_weight": "model-00041-of-00048.safetensors", + "model.llm.layers.45.mlp.experts.w13_weight_scale": "model-00041-of-00048.safetensors", + "model.llm.layers.45.mlp.experts.w2_weight": "model-00041-of-00048.safetensors", + "model.llm.layers.45.mlp.experts.w2_weight_scale": "model-00041-of-00048.safetensors", + "model.llm.layers.45.mlp.gate.bias": "model-00041-of-00048.safetensors", + "model.llm.layers.45.mlp.gate.weight": "model-00041-of-00048.safetensors", + "model.llm.layers.45.mlp_norm.weight": "model-00041-of-00048.safetensors", + "model.llm.layers.45.mlp_sconv.weight": "model-00041-of-00048.safetensors", + "model.llm.layers.46.attn.attn_sink_proj.weight": "model-00041-of-00048.safetensors", + "model.llm.layers.46.attn.gate_attn.weight": "model-00041-of-00048.safetensors", + "model.llm.layers.46.attn.k_sconv.weight": "model-00041-of-00048.safetensors", + "model.llm.layers.46.attn.q_l2_clip.scale": "model-00041-of-00048.safetensors", + "model.llm.layers.46.attn.rel_l2_clip.scale": "model-00041-of-00048.safetensors", + "model.llm.layers.46.attn.rel_logits_proj.proj": "model-00041-of-00048.safetensors", + "model.llm.layers.46.attn.v_sconv.weight": "model-00041-of-00048.safetensors", + "model.llm.layers.46.attn.wk_dv.weight": "model-00041-of-00048.safetensors", + "model.llm.layers.46.attn.wo_ud.weight": "model-00041-of-00048.safetensors", + "model.llm.layers.46.attn.wq_du.weight": "model-00041-of-00048.safetensors", + "model.llm.layers.46.attn.wr_du.weight": "model-00041-of-00048.safetensors", + "model.llm.layers.46.attn.wv_dv.weight": "model-00041-of-00048.safetensors", + "model.llm.layers.46.attn_norm.weight": "model-00041-of-00048.safetensors", + "model.llm.layers.46.attn_sconv.weight": "model-00041-of-00048.safetensors", + "model.llm.layers.46.mlp.experts.w13_weight": "model-00042-of-00048.safetensors", + "model.llm.layers.46.mlp.experts.w13_weight_scale": "model-00042-of-00048.safetensors", + "model.llm.layers.46.mlp.experts.w2_weight": "model-00042-of-00048.safetensors", + "model.llm.layers.46.mlp.experts.w2_weight_scale": "model-00042-of-00048.safetensors", + "model.llm.layers.46.mlp.gate.bias": "model-00042-of-00048.safetensors", + "model.llm.layers.46.mlp.gate.weight": "model-00042-of-00048.safetensors", + "model.llm.layers.46.mlp_norm.weight": "model-00042-of-00048.safetensors", + "model.llm.layers.46.mlp_sconv.weight": "model-00042-of-00048.safetensors", + "model.llm.layers.47.attn.attn_sink_proj.weight": "model-00042-of-00048.safetensors", + "model.llm.layers.47.attn.gate_attn.weight": "model-00042-of-00048.safetensors", + "model.llm.layers.47.attn.k_sconv.static_w": "model-00042-of-00048.safetensors", + "model.llm.layers.47.attn.k_sconv.weight_proj": "model-00042-of-00048.safetensors", + "model.llm.layers.47.attn.q_l2_clip.scale": "model-00042-of-00048.safetensors", + "model.llm.layers.47.attn.rel_l2_clip.scale": "model-00042-of-00048.safetensors", + "model.llm.layers.47.attn.rel_logits_proj.proj": "model-00042-of-00048.safetensors", + "model.llm.layers.47.attn.v_sconv.static_w": "model-00042-of-00048.safetensors", + "model.llm.layers.47.attn.v_sconv.weight_proj": "model-00042-of-00048.safetensors", + "model.llm.layers.47.attn.wk_dv.weight": "model-00042-of-00048.safetensors", + "model.llm.layers.47.attn.wo_ud.weight": "model-00042-of-00048.safetensors", + "model.llm.layers.47.attn.wq_du.weight": "model-00042-of-00048.safetensors", + "model.llm.layers.47.attn.wr_du.weight": "model-00042-of-00048.safetensors", + "model.llm.layers.47.attn.wv_dv.weight": "model-00042-of-00048.safetensors", + "model.llm.layers.47.attn_sconv.weight": "model-00042-of-00048.safetensors", + "model.llm.layers.47.k_input_norm.weight": "model-00042-of-00048.safetensors", + "model.llm.layers.47.mlp.experts.w13_weight": "model-00043-of-00048.safetensors", + "model.llm.layers.47.mlp.experts.w13_weight_scale": "model-00043-of-00048.safetensors", + "model.llm.layers.47.mlp.experts.w2_weight": "model-00043-of-00048.safetensors", + "model.llm.layers.47.mlp.experts.w2_weight_scale": "model-00043-of-00048.safetensors", + "model.llm.layers.47.mlp.gate.bias": "model-00043-of-00048.safetensors", + "model.llm.layers.47.mlp.gate.weight": "model-00043-of-00048.safetensors", + "model.llm.layers.47.mlp_norm.weight": "model-00043-of-00048.safetensors", + "model.llm.layers.47.mlp_sconv.weight": "model-00043-of-00048.safetensors", + "model.llm.layers.47.q_input_norm.weight": "model-00043-of-00048.safetensors", + "model.llm.layers.47.v_input_norm.weight": "model-00043-of-00048.safetensors", + "model.llm.layers.5.attn.attn_sink_proj.weight": "model-00043-of-00048.safetensors", + "model.llm.layers.5.attn.gate_attn.weight": "model-00043-of-00048.safetensors", + "model.llm.layers.5.attn.k_sconv.static_w": "model-00043-of-00048.safetensors", + "model.llm.layers.5.attn.k_sconv.weight_proj": "model-00043-of-00048.safetensors", + "model.llm.layers.5.attn.q_l2_clip.scale": "model-00043-of-00048.safetensors", + "model.llm.layers.5.attn.rel_l2_clip.scale": "model-00043-of-00048.safetensors", + "model.llm.layers.5.attn.rel_logits_proj.proj": "model-00043-of-00048.safetensors", + "model.llm.layers.5.attn.v_sconv.static_w": "model-00043-of-00048.safetensors", + "model.llm.layers.5.attn.v_sconv.weight_proj": "model-00043-of-00048.safetensors", + "model.llm.layers.5.attn.wk_dv.weight": "model-00043-of-00048.safetensors", + "model.llm.layers.5.attn.wo_ud.weight": "model-00043-of-00048.safetensors", + "model.llm.layers.5.attn.wq_du.weight": "model-00043-of-00048.safetensors", + "model.llm.layers.5.attn.wr_du.weight": "model-00043-of-00048.safetensors", + "model.llm.layers.5.attn.wv_dv.weight": "model-00043-of-00048.safetensors", + "model.llm.layers.5.attn_sconv.weight": "model-00043-of-00048.safetensors", + "model.llm.layers.5.k_input_norm.weight": "model-00043-of-00048.safetensors", + "model.llm.layers.5.mlp.experts.w13_weight": "model-00044-of-00048.safetensors", + "model.llm.layers.5.mlp.experts.w13_weight_scale": "model-00044-of-00048.safetensors", + "model.llm.layers.5.mlp.experts.w2_weight": "model-00044-of-00048.safetensors", + "model.llm.layers.5.mlp.experts.w2_weight_scale": "model-00044-of-00048.safetensors", + "model.llm.layers.5.mlp.gate.bias": "model-00044-of-00048.safetensors", + "model.llm.layers.5.mlp.gate.weight": "model-00044-of-00048.safetensors", + "model.llm.layers.5.mlp_norm.weight": "model-00044-of-00048.safetensors", + "model.llm.layers.5.mlp_sconv.weight": "model-00044-of-00048.safetensors", + "model.llm.layers.5.q_input_norm.weight": "model-00044-of-00048.safetensors", + "model.llm.layers.5.v_input_norm.weight": "model-00044-of-00048.safetensors", + "model.llm.layers.6.attn.attn_sink_proj.weight": "model-00044-of-00048.safetensors", + "model.llm.layers.6.attn.gate_attn.weight": "model-00044-of-00048.safetensors", + "model.llm.layers.6.attn.k_sconv.weight": "model-00044-of-00048.safetensors", + "model.llm.layers.6.attn.q_l2_clip.scale": "model-00044-of-00048.safetensors", + "model.llm.layers.6.attn.rel_l2_clip.scale": "model-00044-of-00048.safetensors", + "model.llm.layers.6.attn.rel_logits_proj.proj": "model-00044-of-00048.safetensors", + "model.llm.layers.6.attn.v_sconv.weight": "model-00044-of-00048.safetensors", + "model.llm.layers.6.attn.wk_dv.weight": "model-00044-of-00048.safetensors", + "model.llm.layers.6.attn.wo_ud.weight": "model-00044-of-00048.safetensors", + "model.llm.layers.6.attn.wq_du.weight": "model-00044-of-00048.safetensors", + "model.llm.layers.6.attn.wr_du.weight": "model-00044-of-00048.safetensors", + "model.llm.layers.6.attn.wv_dv.weight": "model-00044-of-00048.safetensors", + "model.llm.layers.6.attn_norm.weight": "model-00044-of-00048.safetensors", + "model.llm.layers.6.attn_sconv.weight": "model-00044-of-00048.safetensors", + "model.llm.layers.6.mlp.experts.w13_weight": "model-00045-of-00048.safetensors", + "model.llm.layers.6.mlp.experts.w13_weight_scale": "model-00045-of-00048.safetensors", + "model.llm.layers.6.mlp.experts.w2_weight": "model-00045-of-00048.safetensors", + "model.llm.layers.6.mlp.experts.w2_weight_scale": "model-00045-of-00048.safetensors", + "model.llm.layers.6.mlp.gate.bias": "model-00045-of-00048.safetensors", + "model.llm.layers.6.mlp.gate.weight": "model-00045-of-00048.safetensors", + "model.llm.layers.6.mlp_norm.weight": "model-00045-of-00048.safetensors", + "model.llm.layers.6.mlp_sconv.weight": "model-00045-of-00048.safetensors", + "model.llm.layers.7.attn.attn_sink_proj.weight": "model-00045-of-00048.safetensors", + "model.llm.layers.7.attn.gate_attn.weight": "model-00045-of-00048.safetensors", + "model.llm.layers.7.attn.k_sconv.weight": "model-00045-of-00048.safetensors", + "model.llm.layers.7.attn.q_l2_clip.scale": "model-00045-of-00048.safetensors", + "model.llm.layers.7.attn.rel_l2_clip.scale": "model-00045-of-00048.safetensors", + "model.llm.layers.7.attn.rel_logits_proj.proj": "model-00045-of-00048.safetensors", + "model.llm.layers.7.attn.v_sconv.weight": "model-00045-of-00048.safetensors", + "model.llm.layers.7.attn.wk_dv.weight": "model-00045-of-00048.safetensors", + "model.llm.layers.7.attn.wo_ud.weight": "model-00045-of-00048.safetensors", + "model.llm.layers.7.attn.wq_du.weight": "model-00045-of-00048.safetensors", + "model.llm.layers.7.attn.wr_du.weight": "model-00045-of-00048.safetensors", + "model.llm.layers.7.attn.wv_dv.weight": "model-00045-of-00048.safetensors", + "model.llm.layers.7.attn_norm.weight": "model-00045-of-00048.safetensors", + "model.llm.layers.7.attn_sconv.weight": "model-00045-of-00048.safetensors", + "model.llm.layers.7.mlp.experts.w13_weight": "model-00046-of-00048.safetensors", + "model.llm.layers.7.mlp.experts.w13_weight_scale": "model-00046-of-00048.safetensors", + "model.llm.layers.7.mlp.experts.w2_weight": "model-00046-of-00048.safetensors", + "model.llm.layers.7.mlp.experts.w2_weight_scale": "model-00046-of-00048.safetensors", + "model.llm.layers.7.mlp.gate.bias": "model-00046-of-00048.safetensors", + "model.llm.layers.7.mlp.gate.weight": "model-00046-of-00048.safetensors", + "model.llm.layers.7.mlp_norm.weight": "model-00046-of-00048.safetensors", + "model.llm.layers.7.mlp_sconv.weight": "model-00046-of-00048.safetensors", + "model.llm.layers.8.attn.attn_sink_proj.weight": "model-00046-of-00048.safetensors", + "model.llm.layers.8.attn.gate_attn.weight": "model-00046-of-00048.safetensors", + "model.llm.layers.8.attn.k_sconv.weight": "model-00046-of-00048.safetensors", + "model.llm.layers.8.attn.q_l2_clip.scale": "model-00046-of-00048.safetensors", + "model.llm.layers.8.attn.rel_l2_clip.scale": "model-00046-of-00048.safetensors", + "model.llm.layers.8.attn.rel_logits_proj.proj": "model-00046-of-00048.safetensors", + "model.llm.layers.8.attn.v_sconv.weight": "model-00046-of-00048.safetensors", + "model.llm.layers.8.attn.wk_dv.weight": "model-00046-of-00048.safetensors", + "model.llm.layers.8.attn.wo_ud.weight": "model-00046-of-00048.safetensors", + "model.llm.layers.8.attn.wq_du.weight": "model-00046-of-00048.safetensors", + "model.llm.layers.8.attn.wr_du.weight": "model-00046-of-00048.safetensors", + "model.llm.layers.8.attn.wv_dv.weight": "model-00046-of-00048.safetensors", + "model.llm.layers.8.attn_norm.weight": "model-00046-of-00048.safetensors", + "model.llm.layers.8.attn_sconv.weight": "model-00046-of-00048.safetensors", + "model.llm.layers.8.mlp.experts.w13_weight": "model-00047-of-00048.safetensors", + "model.llm.layers.8.mlp.experts.w13_weight_scale": "model-00047-of-00048.safetensors", + "model.llm.layers.8.mlp.experts.w2_weight": "model-00047-of-00048.safetensors", + "model.llm.layers.8.mlp.experts.w2_weight_scale": "model-00047-of-00048.safetensors", + "model.llm.layers.8.mlp.gate.bias": "model-00047-of-00048.safetensors", + "model.llm.layers.8.mlp.gate.weight": "model-00047-of-00048.safetensors", + "model.llm.layers.8.mlp_norm.weight": "model-00047-of-00048.safetensors", + "model.llm.layers.8.mlp_sconv.weight": "model-00047-of-00048.safetensors", + "model.llm.layers.9.attn.attn_sink_proj.weight": "model-00047-of-00048.safetensors", + "model.llm.layers.9.attn.gate_attn.weight": "model-00047-of-00048.safetensors", + "model.llm.layers.9.attn.k_sconv.weight": "model-00047-of-00048.safetensors", + "model.llm.layers.9.attn.q_l2_clip.scale": "model-00047-of-00048.safetensors", + "model.llm.layers.9.attn.rel_l2_clip.scale": "model-00047-of-00048.safetensors", + "model.llm.layers.9.attn.rel_logits_proj.proj": "model-00047-of-00048.safetensors", + "model.llm.layers.9.attn.v_sconv.weight": "model-00047-of-00048.safetensors", + "model.llm.layers.9.attn.wk_dv.weight": "model-00047-of-00048.safetensors", + "model.llm.layers.9.attn.wo_ud.weight": "model-00047-of-00048.safetensors", + "model.llm.layers.9.attn.wq_du.weight": "model-00047-of-00048.safetensors", + "model.llm.layers.9.attn.wr_du.weight": "model-00047-of-00048.safetensors", + "model.llm.layers.9.attn.wv_dv.weight": "model-00047-of-00048.safetensors", + "model.llm.layers.9.attn_norm.weight": "model-00047-of-00048.safetensors", + "model.llm.layers.9.attn_sconv.weight": "model-00047-of-00048.safetensors", + "model.llm.layers.9.mlp.experts.w13_weight": "model-00048-of-00048.safetensors", + "model.llm.layers.9.mlp.experts.w13_weight_scale": "model-00048-of-00048.safetensors", + "model.llm.layers.9.mlp.experts.w2_weight": "model-00048-of-00048.safetensors", + "model.llm.layers.9.mlp.experts.w2_weight_scale": "model-00048-of-00048.safetensors", + "model.llm.layers.9.mlp.gate.bias": "model-00048-of-00048.safetensors", + "model.llm.layers.9.mlp.gate.weight": "model-00048-of-00048.safetensors", + "model.llm.layers.9.mlp_norm.weight": "model-00048-of-00048.safetensors", + "model.llm.layers.9.mlp_sconv.weight": "model-00048-of-00048.safetensors", + "model.llm.norm.weight": "model-00048-of-00048.safetensors", + "model.llm.unembed.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.embeddings.class_embedding": "model-00048-of-00048.safetensors", + "model.visual.backbone.embeddings.cls_position_embedding": "model-00048-of-00048.safetensors", + "model.visual.backbone.embeddings.patch_embedding.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.embeddings.position_embedding": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.0.ln_1.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.0.ln_1.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.0.ln_2.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.0.ln_2.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.0.ls_1.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.0.ls_2.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.0.mlp.c_fc.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.0.mlp.c_fc.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.0.mlp.c_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.0.mlp.c_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.0.self_attn.k_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.0.self_attn.k_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.0.self_attn.out_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.0.self_attn.out_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.0.self_attn.q_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.0.self_attn.q_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.0.self_attn.v_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.0.self_attn.v_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.1.ln_1.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.1.ln_1.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.1.ln_2.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.1.ln_2.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.1.ls_1.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.1.ls_2.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.1.mlp.c_fc.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.1.mlp.c_fc.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.1.mlp.c_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.1.mlp.c_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.1.self_attn.k_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.1.self_attn.k_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.1.self_attn.out_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.1.self_attn.out_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.1.self_attn.q_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.1.self_attn.q_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.1.self_attn.v_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.1.self_attn.v_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.10.ln_1.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.10.ln_1.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.10.ln_2.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.10.ln_2.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.10.ls_1.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.10.ls_2.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.10.mlp.c_fc.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.10.mlp.c_fc.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.10.mlp.c_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.10.mlp.c_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.10.self_attn.k_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.10.self_attn.k_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.10.self_attn.out_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.10.self_attn.out_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.10.self_attn.q_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.10.self_attn.q_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.10.self_attn.v_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.10.self_attn.v_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.11.ln_1.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.11.ln_1.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.11.ln_2.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.11.ln_2.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.11.ls_1.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.11.ls_2.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.11.mlp.c_fc.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.11.mlp.c_fc.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.11.mlp.c_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.11.mlp.c_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.11.self_attn.k_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.11.self_attn.k_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.11.self_attn.out_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.11.self_attn.out_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.11.self_attn.q_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.11.self_attn.q_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.11.self_attn.v_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.11.self_attn.v_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.12.ln_1.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.12.ln_1.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.12.ln_2.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.12.ln_2.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.12.ls_1.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.12.ls_2.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.12.mlp.c_fc.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.12.mlp.c_fc.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.12.mlp.c_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.12.mlp.c_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.12.self_attn.k_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.12.self_attn.k_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.12.self_attn.out_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.12.self_attn.out_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.12.self_attn.q_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.12.self_attn.q_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.12.self_attn.v_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.12.self_attn.v_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.13.ln_1.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.13.ln_1.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.13.ln_2.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.13.ln_2.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.13.ls_1.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.13.ls_2.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.13.mlp.c_fc.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.13.mlp.c_fc.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.13.mlp.c_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.13.mlp.c_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.13.self_attn.k_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.13.self_attn.k_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.13.self_attn.out_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.13.self_attn.out_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.13.self_attn.q_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.13.self_attn.q_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.13.self_attn.v_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.13.self_attn.v_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.14.ln_1.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.14.ln_1.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.14.ln_2.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.14.ln_2.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.14.ls_1.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.14.ls_2.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.14.mlp.c_fc.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.14.mlp.c_fc.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.14.mlp.c_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.14.mlp.c_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.14.self_attn.k_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.14.self_attn.k_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.14.self_attn.out_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.14.self_attn.out_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.14.self_attn.q_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.14.self_attn.q_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.14.self_attn.v_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.14.self_attn.v_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.15.ln_1.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.15.ln_1.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.15.ln_2.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.15.ln_2.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.15.ls_1.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.15.ls_2.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.15.mlp.c_fc.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.15.mlp.c_fc.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.15.mlp.c_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.15.mlp.c_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.15.self_attn.k_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.15.self_attn.k_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.15.self_attn.out_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.15.self_attn.out_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.15.self_attn.q_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.15.self_attn.q_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.15.self_attn.v_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.15.self_attn.v_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.16.ln_1.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.16.ln_1.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.16.ln_2.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.16.ln_2.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.16.ls_1.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.16.ls_2.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.16.mlp.c_fc.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.16.mlp.c_fc.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.16.mlp.c_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.16.mlp.c_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.16.self_attn.k_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.16.self_attn.k_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.16.self_attn.out_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.16.self_attn.out_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.16.self_attn.q_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.16.self_attn.q_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.16.self_attn.v_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.16.self_attn.v_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.17.ln_1.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.17.ln_1.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.17.ln_2.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.17.ln_2.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.17.ls_1.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.17.ls_2.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.17.mlp.c_fc.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.17.mlp.c_fc.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.17.mlp.c_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.17.mlp.c_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.17.self_attn.k_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.17.self_attn.k_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.17.self_attn.out_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.17.self_attn.out_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.17.self_attn.q_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.17.self_attn.q_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.17.self_attn.v_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.17.self_attn.v_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.18.ln_1.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.18.ln_1.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.18.ln_2.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.18.ln_2.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.18.ls_1.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.18.ls_2.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.18.mlp.c_fc.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.18.mlp.c_fc.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.18.mlp.c_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.18.mlp.c_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.18.self_attn.k_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.18.self_attn.k_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.18.self_attn.out_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.18.self_attn.out_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.18.self_attn.q_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.18.self_attn.q_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.18.self_attn.v_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.18.self_attn.v_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.19.ln_1.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.19.ln_1.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.19.ln_2.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.19.ln_2.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.19.ls_1.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.19.ls_2.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.19.mlp.c_fc.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.19.mlp.c_fc.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.19.mlp.c_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.19.mlp.c_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.19.self_attn.k_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.19.self_attn.k_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.19.self_attn.out_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.19.self_attn.out_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.19.self_attn.q_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.19.self_attn.q_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.19.self_attn.v_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.19.self_attn.v_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.2.ln_1.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.2.ln_1.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.2.ln_2.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.2.ln_2.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.2.ls_1.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.2.ls_2.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.2.mlp.c_fc.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.2.mlp.c_fc.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.2.mlp.c_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.2.mlp.c_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.2.self_attn.k_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.2.self_attn.k_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.2.self_attn.out_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.2.self_attn.out_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.2.self_attn.q_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.2.self_attn.q_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.2.self_attn.v_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.2.self_attn.v_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.20.ln_1.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.20.ln_1.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.20.ln_2.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.20.ln_2.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.20.ls_1.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.20.ls_2.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.20.mlp.c_fc.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.20.mlp.c_fc.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.20.mlp.c_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.20.mlp.c_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.20.self_attn.k_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.20.self_attn.k_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.20.self_attn.out_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.20.self_attn.out_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.20.self_attn.q_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.20.self_attn.q_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.20.self_attn.v_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.20.self_attn.v_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.21.ln_1.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.21.ln_1.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.21.ln_2.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.21.ln_2.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.21.ls_1.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.21.ls_2.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.21.mlp.c_fc.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.21.mlp.c_fc.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.21.mlp.c_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.21.mlp.c_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.21.self_attn.k_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.21.self_attn.k_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.21.self_attn.out_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.21.self_attn.out_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.21.self_attn.q_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.21.self_attn.q_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.21.self_attn.v_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.21.self_attn.v_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.22.ln_1.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.22.ln_1.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.22.ln_2.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.22.ln_2.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.22.ls_1.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.22.ls_2.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.22.mlp.c_fc.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.22.mlp.c_fc.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.22.mlp.c_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.22.mlp.c_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.22.self_attn.k_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.22.self_attn.k_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.22.self_attn.out_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.22.self_attn.out_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.22.self_attn.q_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.22.self_attn.q_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.22.self_attn.v_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.22.self_attn.v_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.3.ln_1.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.3.ln_1.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.3.ln_2.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.3.ln_2.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.3.ls_1.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.3.ls_2.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.3.mlp.c_fc.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.3.mlp.c_fc.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.3.mlp.c_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.3.mlp.c_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.3.self_attn.k_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.3.self_attn.k_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.3.self_attn.out_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.3.self_attn.out_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.3.self_attn.q_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.3.self_attn.q_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.3.self_attn.v_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.3.self_attn.v_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.4.ln_1.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.4.ln_1.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.4.ln_2.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.4.ln_2.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.4.ls_1.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.4.ls_2.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.4.mlp.c_fc.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.4.mlp.c_fc.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.4.mlp.c_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.4.mlp.c_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.4.self_attn.k_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.4.self_attn.k_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.4.self_attn.out_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.4.self_attn.out_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.4.self_attn.q_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.4.self_attn.q_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.4.self_attn.v_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.4.self_attn.v_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.5.ln_1.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.5.ln_1.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.5.ln_2.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.5.ln_2.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.5.ls_1.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.5.ls_2.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.5.mlp.c_fc.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.5.mlp.c_fc.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.5.mlp.c_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.5.mlp.c_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.5.self_attn.k_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.5.self_attn.k_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.5.self_attn.out_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.5.self_attn.out_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.5.self_attn.q_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.5.self_attn.q_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.5.self_attn.v_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.5.self_attn.v_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.6.ln_1.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.6.ln_1.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.6.ln_2.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.6.ln_2.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.6.ls_1.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.6.ls_2.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.6.mlp.c_fc.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.6.mlp.c_fc.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.6.mlp.c_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.6.mlp.c_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.6.self_attn.k_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.6.self_attn.k_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.6.self_attn.out_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.6.self_attn.out_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.6.self_attn.q_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.6.self_attn.q_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.6.self_attn.v_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.6.self_attn.v_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.7.ln_1.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.7.ln_1.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.7.ln_2.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.7.ln_2.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.7.ls_1.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.7.ls_2.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.7.mlp.c_fc.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.7.mlp.c_fc.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.7.mlp.c_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.7.mlp.c_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.7.self_attn.k_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.7.self_attn.k_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.7.self_attn.out_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.7.self_attn.out_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.7.self_attn.q_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.7.self_attn.q_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.7.self_attn.v_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.7.self_attn.v_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.8.ln_1.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.8.ln_1.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.8.ln_2.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.8.ln_2.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.8.ls_1.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.8.ls_2.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.8.mlp.c_fc.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.8.mlp.c_fc.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.8.mlp.c_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.8.mlp.c_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.8.self_attn.k_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.8.self_attn.k_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.8.self_attn.out_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.8.self_attn.out_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.8.self_attn.q_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.8.self_attn.q_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.8.self_attn.v_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.8.self_attn.v_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.9.ln_1.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.9.ln_1.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.9.ln_2.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.9.ln_2.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.9.ls_1.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.9.ls_2.gamma": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.9.mlp.c_fc.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.9.mlp.c_fc.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.9.mlp.c_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.9.mlp.c_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.9.self_attn.k_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.9.self_attn.k_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.9.self_attn.out_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.9.self_attn.out_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.9.self_attn.q_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.9.self_attn.q_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.9.self_attn.v_proj.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.layers.9.self_attn.v_proj.weight": "model-00048-of-00048.safetensors", + "model.visual.backbone.ln_pre.bias": "model-00048-of-00048.safetensors", + "model.visual.backbone.ln_pre.weight": "model-00048-of-00048.safetensors", + "model.visual.proj_in.weight": "model-00048-of-00048.safetensors", + "model.visual.proj_out.weight": "model-00048-of-00048.safetensors" + } +} diff --git a/mtp.safetensors b/mtp.safetensors new file mode 100644 index 0000000000000000000000000000000000000000..473d1f23194cd9e3343c0fe21000ed630bb848ac --- /dev/null +++ b/mtp.safetensors @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:3b265640ec84ba66e928a2844c4c022c94be731336b0f63755a20a247bfd2a36 +size 3423429800 diff --git a/processor_config.json b/processor_config.json new file mode 100644 index 0000000000000000000000000000000000000000..b26765bfb2af51c902bc0222d1282075aff65466 --- /dev/null +++ b/processor_config.json @@ -0,0 +1,14 @@ +{ + "image_processor": { + "do_convert_rgb": true, + "do_normalize": true, + "do_rescale": true, + "image_mean": [0.5, 0.5, 0.5], + "image_processor_type": "Inkling2ImageProcessor", + "image_std": [0.5, 0.5, 0.5], + "rescale_factor": 0.00392156862745098, + "size": {"height": 40, "width": 40} + }, + "image_token": "<|unused_200054|>", + "processor_class": "Inkling2Processor" +} \ No newline at end of file diff --git a/special_tokens_map.json b/special_tokens_map.json new file mode 100644 index 0000000000000000000000000000000000000000..3d7a60d8099239a65f0bb660849d1fc636b7c734 --- /dev/null +++ b/special_tokens_map.json @@ -0,0 +1,19 @@ +{ + "additional_special_tokens": [ + "<|endoftext|>", + "<|message_user|>", + "<|message_model|>", + "<|message_system|>", + "<|message_tool|>", + "<|content_text|>", + "<|content_image|>", + "<|content_model_end_sampling|>", + "<|content_thinking|>", + "<|end_message|>", + "<|content_tool_error|>", + "<|content_xml|>", + "<|begin_of_text|>", + "<|content_invoke_tool_json|>", + "<|content_invoke_tool_text|>" + ] +} diff --git a/tokenizer.json b/tokenizer.json new file mode 100644 index 0000000000000000000000000000000000000000..5e55fd2939ffee98a8dd9f273b7e8c999e56f9c2 --- /dev/null +++ b/tokenizer.json @@ -0,0 +1,3 @@ +version https://git-lfs.github.com/spec/v1 +oid sha256:7f332727f9a3f093dd452a39cc978e6757627e5dad1b0673aa4e75174f236dd6 +size 26178678 diff --git a/tokenizer_config.json b/tokenizer_config.json new file mode 100644 index 0000000000000000000000000000000000000000..75062c6c6b0bcc80f1b68ed85f50e2e3b057a9b0 --- /dev/null +++ b/tokenizer_config.json @@ -0,0 +1,546 @@ +{ + "tokenizer_class": "PreTrainedTokenizerFast", + "model_max_length": 1048576, + "clean_up_tokenization_spaces": false, + "fix_mistral_regex": false, + "extra_special_tokens": { + "endoftext": "<|endoftext|>", + "message_user": "<|message_user|>", + "message_model": "<|message_model|>", + "message_system": "<|message_system|>", + "message_tool": "<|message_tool|>", + "content_text": "<|content_text|>", + "content_image": "<|content_image|>", + "content_model_end_sampling": "<|content_model_end_sampling|>", + "content_thinking": "<|content_thinking|>", + "end_message": "<|end_message|>", + "content_tool_error": "<|content_tool_error|>", + "content_xml": "<|content_xml|>", + "begin_of_text": "<|begin_of_text|>", + "content_invoke_tool_json": "<|content_invoke_tool_json|>", + "content_invoke_tool_text": "<|content_invoke_tool_text|>" + }, + "added_tokens_decoder": { + "199998": { + "content": "<|unused|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "199999": { + "content": "<|endoftext|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200000": { + "content": "<|message_user|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200001": { + "content": "<|message_model|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200002": { + "content": "<|message_system|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200003": { + "content": "<|message_tool|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200004": { + "content": "<|content_text|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200005": { + "content": "<|content_image|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200006": { + "content": "<|content_model_end_sampling|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200007": { + "content": "<|unused_200007|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200008": { + "content": "<|content_thinking|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200009": { + "content": "<|unused_200009|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200010": { + "content": "<|end_message|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200011": { + "content": "<|unused_200011|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200012": { + "content": "<|unused_200012|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200013": { + "content": "<|unused_200013|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200014": { + "content": "<|unused_200014|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200015": { + "content": "<|unused_200015|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200016": { + "content": "<|unused_200016|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200017": { + "content": "<|unused_200017|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200018": { + "content": "<|unused_200018|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200019": { + "content": "<|unused_200019|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200020": { + "content": "<|unused_200020|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200021": { + "content": "<|unused_200021|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200022": { + "content": "<|content_tool_error|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200023": { + "content": "<|unused_200023|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200024": { + "content": "<|content_xml|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200025": { + "content": "<|unused_200025|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200026": { + "content": "<|unused_200026|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200027": { + "content": "<|unused_200027|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200028": { + "content": "<|begin_of_text|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200029": { + "content": "<|unused_200029|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200030": { + "content": "<|unused_200030|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200031": { + "content": "<|unused_200031|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200032": { + "content": "<|unused_200032|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200033": { + "content": "<|unused_200033|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200034": { + "content": "<|unused_200034|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200035": { + "content": "<|unused_200035|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200036": { + "content": "<|unused_200036|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200037": { + "content": "<|unused_200037|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200038": { + "content": "<|unused_200038|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200039": { + "content": "<|unused_200039|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200040": { + "content": "<|unused_200040|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200041": { + "content": "<|unused_200041|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200042": { + "content": "<|unused_200042|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200043": { + "content": "<|unused_200043|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200044": { + "content": "<|unused_200044|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200045": { + "content": "<|unused_200045|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200046": { + "content": "<|unused_200046|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200047": { + "content": "<|unused_200047|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200048": { + "content": "<|unused_200048|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200049": { + "content": "<|content_invoke_tool_json|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200050": { + "content": "<|unused_200050|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200051": { + "content": "<|unused_200051|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200052": { + "content": "<|unused_200052|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200053": { + "content": "<|unused_200053|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200054": { + "content": "<|unused_200054|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200055": { + "content": "<|unused_200055|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200056": { + "content": "<|unused_200056|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + }, + "200057": { + "content": "<|content_invoke_tool_text|>", + "single_word": false, + "lstrip": false, + "rstrip": false, + "normalized": false, + "special": true + } + }, + "response_template": { + "defaults": { + "role": "assistant" + }, + "start_anchor_pattern": "<\\|end_message\\|>|<\\|content_model_end_sampling\\|>", + "fields": { + "reasoning_content": { + "open_pattern": "(?:<\\|message_model\\|>[^<]*)?<\\|content_thinking\\|>", + "close": "<|end_message|>", + "repeats": true, + "join": "", + "content": "text", + "content_args": { + "strip": false + } + }, + "content": { + "open_pattern": "(?:<\\|message_model\\|>[^<]*)?<\\|content_(?:text|tool_error|invoke_tool_text)\\|>", + "close": "<|end_message|>", + "repeats": true, + "join": "", + "content": "text", + "content_args": { + "strip": false + } + }, + "tool_calls": { + "open_pattern": "(?:<\\|message_model\\|>[^<]*)?<\\|content_invoke_tool_json\\|>", + "close": "<|end_message|>", + "repeats": true, + "content": "json", + "transform": { + "type": "function", + "function": { + "name": "{content.name}", + "arguments": "{content.args}" + } + } + } + } + } +} diff --git a/validation_results.json b/validation_results.json new file mode 100644 index 0000000000000000000000000000000000000000..77daf43a8ad4a10257d47eb793820925e6f8da3c --- /dev/null +++ b/validation_results.json @@ -0,0 +1,266 @@ +{ + "source_revision": "3febc62259ee92ea36e2fb6e8303371c4eb3922c", + "exported_checkpoint": { + "expert_tensors_checked": 96, + "expert_samples_bitwise_match_runtime_conversion": true, + "nonexpert_tensors_bitwise_unchanged": 1418, + "mtp_bitwise_unchanged": true, + "text_images_streaming_passed": true, + "context_limit": 262144, + "served_model": "seekr-org/Inkling-2-Small-FP8-ROCm", + "accuracy": { + "gsm8k": { + "passed": 48, + "total": 50 + }, + "humaneval": { + "passed": 20, + "total": 20 + } + }, + "accuracy_generation": { + "temperature": 0, + "reasoning_effort": 0, + "max_tokens": 2048, + "concurrency": 8 + }, + "vision_checks": [ + { + "name": "arithmetic", + "passed": true, + "text": "42" + }, + { + "name": "capital", + "passed": true, + "text": "Paris" + }, + { + "name": "image_red", + "passed": true, + "text": "Red" + }, + { + "name": "image_blue", + "passed": true, + "text": "Blue" + }, + { + "name": "image_ocr", + "passed": true, + "text": "5739" + }, + { + "name": "two_images", + "passed": true, + "text": "Red, blue" + } + ] + }, + "prior_quantization_comparison": { + "model": "thinkingmachines/Inkling-2-Small-MXFP8", + "revision": "3febc62259ee92ea36e2fb6e8303371c4eb3922c", + "hardware": "8 AMD MI300X gfx942, 192 GiB each", + "max_model_len": 262144, + "speculative_config": { + "method": "dspark", + "num_speculative_tokens": 7, + "draft_sample_method": "probabilistic" + }, + "workload": { + "requests_per_concurrency": 24, + "input_tokens": 2016, + "output_tokens": 512, + "domains": [ + "math", + "programming", + "prose" + ], + "temperature": 1, + "ignore_eos": true, + "prefix_caching": false, + "reasoning": "model default; completion counts include reasoning" + }, + "variants": { + "mxfp8-final": { + "c1": { + "completed": 24, + "failed": 0, + "duration": 87.02415829710662, + "output_throughput": 141.2021700692341, + "total_output_tokens": 12288, + "mean_ttft_ms": 353.02664975946146, + "mean_tpot_ms": 6.4045614900100265, + "spec_decode_acceptance_length": 3.054865938430983 + }, + "c1_by_domain": { + "math": { + "output_tok_s": 192.68324620967573, + "min_request_tok_s": 162.16978953567997, + "max_request_tok_s": 254.20072339484943 + }, + "programming": { + "output_tok_s": 138.22244793740694, + "min_request_tok_s": 120.47456621151565, + "max_request_tok_s": 154.9191445331261 + }, + "prose": { + "output_tok_s": 113.3774818838921, + "min_request_tok_s": 101.28198891060704, + "max_request_tok_s": 124.1347560614812 + } + }, + "c8": { + "completed": 24, + "failed": 0, + "duration": 35.22780516371131, + "output_throughput": 348.81537305247883, + "total_output_tokens": 12288, + "mean_ttft_ms": 1618.4657968891163, + "mean_tpot_ms": 17.592307901440037, + "spec_decode_acceptance_length": 3.04 + }, + "accuracy_short_budget": { + "gsm8k": { + "passed": 47, + "total": 50 + }, + "humaneval": { + "passed": 19, + "total": 20 + } + }, + "accuracy_2048_budget": { + "gsm8k": { + "passed": 48, + "total": 50 + }, + "humaneval": { + "passed": 20, + "total": 20 + } + }, + "prose": { + "checks": { + "word_count_180_to_220": true, + "Mara": true, + "lighthouse": true, + "map": true, + "dialogue": true, + "exact_ending": true + }, + "word_count": 196 + }, + "vision_checks": true, + "long_context": { + "http_status": 200, + "seconds": 51.59381696023047, + "usage": { + "prompt_tokens": 261040, + "total_tokens": 261046, + "completion_tokens": 6, + "completion_tokens_details": { + "reasoning_tokens": 0 + } + }, + "content": "739124" + } + }, + "row-fp8-final": { + "c1": { + "completed": 24, + "failed": 0, + "duration": 49.65525996312499, + "output_throughput": 247.46623034750638, + "total_output_tokens": 12288, + "mean_ttft_ms": 128.29210500543317, + "mean_tpot_ms": 3.797367814167683, + "spec_decode_acceptance_length": 3.117617237008872 + }, + "c1_by_domain": { + "math": { + "output_tok_s": 326.52381715114535, + "min_request_tok_s": 265.20409125347084, + "max_request_tok_s": 397.12908010147316 + }, + "programming": { + "output_tok_s": 248.068111866624, + "min_request_tok_s": 207.0873267282028, + "max_request_tok_s": 276.0874995644794 + }, + "prose": { + "output_tok_s": 198.8921654365731, + "min_request_tok_s": 173.67162954038446, + "max_request_tok_s": 234.15402964596927 + } + }, + "c8": { + "completed": 24, + "failed": 0, + "duration": 15.359448773786426, + "output_throughput": 800.0287107289691, + "total_output_tokens": 12288, + "mean_ttft_ms": 375.8305724089344, + "mean_tpot_ms": 8.250392456882222, + "spec_decode_acceptance_length": 3.0183733463988243 + }, + "accuracy_short_budget": { + "gsm8k": { + "passed": 48, + "total": 50 + }, + "humaneval": { + "passed": 19, + "total": 20 + } + }, + "accuracy_2048_budget": { + "gsm8k": { + "passed": 48, + "total": 50 + }, + "humaneval": { + "passed": 19, + "total": 20 + } + }, + "prose": { + "checks": { + "word_count_180_to_220": true, + "Mara": true, + "lighthouse": true, + "map": true, + "dialogue": true, + "exact_ending": true + }, + "word_count": 192 + }, + "vision_checks": true, + "long_context": { + "http_status": 200, + "seconds": 27.074019331485033, + "usage": { + "prompt_tokens": 261040, + "total_tokens": 261046, + "completion_tokens": 6, + "completion_tokens_details": { + "reasoning_tokens": 0 + } + }, + "content": "739124" + } + } + }, + "accuracy_notes": [ + "Fixed Random(42) subset: 50 GSM8K and 20 HumanEval; temperature=0, reasoning_effort=0, concurrency=8.", + "The short paired run uses 512 math /768 code max_tokens. Extended paired run uses 2048 for both. Both sets retained.", + "GSM8K sample0 has an incorrect official reference: 430+750+300+700=2180, while reference says2280. Raw scores retain the original label.", + "Prose checks measure instruction following only; no automated literary-quality score." + ] + }, + "notes": [ + "The historical paired score for row FP8 is retained separately. Greedy output varies with batching; the fresh exported-checkpoint check does not replace that paired measurement.", + "GSM8K scores retain a known incorrect official reference label." + ], + "verification_seconds": 6.884046273306012 +}