r3 (October 2026): fresh QAT fine-tune, native reasoning and tool formats, identity without a system prompt

AILEY_MERGE_INFO.json

@@ -1,27 +1,32 @@
 {
-  "release": "r2",
-  "base": "mlx-community/gemma-4-12B-it-6bit",
-  "lineage": "AileyCore-12B r1 (July 2026, adapter merged into q6) -> round 2 SFT (Sept 2026) as side path",
-  "side_path": {
-    "file": "model-seitenpfad.safetensors",
-    "manifest": "seitenpfad.json",
-    "modules": 232,
-    "rank": 8,
-    "targets": ["q_proj", "v_proj", "gate_proj", "up_proj", "down_proj"],
-    "dora_magnitude": "folded exactly into the 6-bit quantization scales",
-    "why": "merging the round-2 delta into q6 weights rounded away 87 % of it (val 1.063 merged vs 0.836 live adapter vs 1.096 base); the side path keeps 96.5 % (val 0.845) at -1 % decode speed"
-  },
-  "round2_training": {
-    "method": "DoRA (q_proj, v_proj) + LoRA (MLP), rank 8, alpha 16",
-    "samples": 1255,
-    "valid": 139,
+  "base": "mlx-community/gemma-4-12B-it-qat-4bit",
+  "checkpoint": "best",
+  "out_name": "AileyCore-12B",
+  "dora_targets": [
+    "q_proj",
+    "v_proj"
+  ],
+  "lora_targets": [
+    "gate_proj",
+    "up_proj",
+    "down_proj"
+  ],
+  "rank": 16,
+  "alpha": 32,
+  "merged_at": "2026-10-10T12:23:13",
+  "release": "r3",
+  "method": "DoRA (q_proj, v_proj) + LoRA (gate/up/down), rank 16, alpha 32, trained as QAT adapters (fake-quant forward, per-layer bits/group size) so the merge is bit-identical to training",
+  "training": {
+    "conversations": 2429,
+    "samples": 2676,
+    "validation_samples": 296,
+    "image_rows": 441,
     "epochs": 1,
-    "updates": 162,
-    "lr": 2e-5,
-    "seq": 1024,
+    "updates": 334,
+    "lr": 2e-05,
+    "seq": 2048,
     "grad_accum": 8,
-    "selected": "best (val_loss 0.836 live, 0.845 via side path)",
-    "dpo": "trained (300 format pairs) but NOT shipped: every dose taught a second <channel|> after the answer (repeat/loop); SFT alone ends cleanly"
+    "best_val": 1.3177
   },
-  "created": "2026-09-17"
-}
+  "not_shipped": "DPO stage (length bias in pairs: chosen reasoning ~137 vs rejected ~1228 chars)"
+}

README.md

@@ -1,11 +1,11 @@
 ---
 license: apache-2.0
-base_model: mlx-community/gemma-4-12B-it-6bit
+base_model: mlx-community/gemma-4-12B-it-qat-4bit
 library_name: mlx
 pipeline_tag: image-text-to-text
 language:
-  - en
   - de
+  - en
 tags:
   - mlx
   - gemma
@@ -13,130 +13,134 @@ tags:
   - fine-tuned
   - dora
   - lora
+  - qat
+  - reasoning
+  - tool-use
   - ailey
   - openminded
 ---
 
-# AileyCore-12B (r2, September 2026)
+# AileyCore-12B (r3, October 2026)
 
-**AileyCore-12B** is a fine-tuned derivative of **Google Gemma 4 (12B, instruction-tuned)**,
-6-bit quantized, built to run locally on Apple Silicon via [MLX](https://github.com/ml-explore/mlx).
-It powers **A!ley**, the on-device persona created by **OpenM!nded / Simon van de Loo**.
+**AileyCore-12B** is a fine-tune of **Google Gemma 4 (12B, instruction-tuned)** on Google's
+**QAT 4-bit** checkpoint, built to run locally on Apple Silicon via [MLX](https://github.com/ml-explore/mlx).
+It powers **A!ley**, the on-device persona created by **OpenM!nded / Simon van de Loo**, and is
+her default mode in the A!ley portal and app.
 
 - **Developed by:** OpenM!nded (Simon van de Loo)
 - **Model type:** Multimodal (text + image + audio input, text output), decoder-only
-- **Base model:** [`mlx-community/gemma-4-12B-it-6bit`](https://huggingface.co/mlx-community/gemma-4-12B-it-6bit)
+- **Base model:** [`mlx-community/gemma-4-12B-it-qat-4bit`](https://huggingface.co/mlx-community/gemma-4-12B-it-qat-4bit)
 - **License:** Apache License 2.0
-- **Languages:** German, English
-- **Quantization:** 6-bit, unchanged
+- **Languages:** German, English (answers in the user's language)
+- **Quantization:** 4-bit QAT (mixed: some MLP layers 8-bit, as in the base); adapter merged **without loss**
 
-> **This release is not a plain merged checkpoint.** It ships a small **side path**
-> (`model-seitenpfad.safetensors`, 52 MB) that is part of the model and must be applied
-> at load time. Read *How to load* below — a loader that ignores it silently gives you
-> the previous (r1) weights, not r2.
+> **r3 is a plain merged checkpoint** — no side path, no extra loader code. The r2 release
+> (6-bit with a rank-8 side path) stays available at revision
+> [`37124f0`](https://huggingface.co/OpenMinded-Labs/AileyCore-12B/tree/37124f02f70770d62522af9be15e3d440f737dc0)
+> (`revision="37124f02f70770d62522af9be15e3d440f737dc0"`).
 
 ---
 
-## What changed in r2
+## What changed in r3
 
-Round 2 (September 2026) teaches the model Gemma 4's **native reasoning channel**
-(`<|channel>thought … <channel|>`) instead of a custom `<think>` tag, with fresh
-conversational data (1 255 samples, 1 epoch), tool use / RAG / web-search context in the
-same format the runtime produces, and identity that holds with or without a system prompt.
+r1 and r2 were successive corrections on the same 6-bit weights. r3 starts **fresh from
+Google's QAT checkpoint** and trains everything A!ley needs in one go, all in Gemma 4's
+**native formats**:
 
-Measured on 60 held-out samples (lower is better):
+- **Native reasoning channel** (`<|channel>thought … <channel|>`) with depth scaled to the task —
+  a greeting gets a sentence, a puzzle gets a plan.
+- **Native tool use**: Gemma 4 tool declarations, `<|tool_call>` calls and `<|tool_response>`
+  results exactly as the runtime produces them (web search, RAG over her own songs, books and
+  projects, image generation, recipes, calculator …), including *when not to call a tool*.
+- **Identity without a prompt**: 81 % of the training rows have **no system turn** at all; the
+  user's name sits in the user turn (`Name: message`).
+- Narrative / prose writing, vision rows (441 images), multilingual answers.
 
-| Variant | val loss |
-|---|---|
-| r1 (base of this release) | 1.096 |
-| r2 delta merged into the 6-bit weights (the usual way) | 1.063 — **87 % of the training lost to rounding** |
-| r2 live adapter | 0.836 |
-| **r2 side path (this release)** | **0.845 — 96.5 % kept, −1 % decode speed** |
+### QAT adapters — why the merge is lossless
+
+The adapters were trained as **QAT adapters**: the forward pass sees
+`fake_quant(W_dequant + ΔW)` with a straight-through gradient, reading bits and group size per
+layer (so the base's mixed 4/8-bit layout is respected). The model only learns deltas that
+survive the rounding, and `fuse()` yields weights bit-identical to what was trained. (In r2,
+a classic merge into 6-bit had rounded away 87 % of the delta — hence the side path back then.)
+
+## Measured (isolated, before going live)
 
-Merging a low-learning-rate adapter into 6-bit weights rounds most of the delta away: the
-per-weight change is smaller than half a quantization step. Re-quantizing finer was not an
-option (memory budget), so the rank-8 residual stays outside the quantization, exactly:
+Benchmark run with the model loaded alone, same questions for every condition:
 
-`y = W_q · x + (x · A) · B`
+| | no system prompt | one-line context header | runtime prompt (~700 chars, tool hints) | old long persona prompt (~7.8k chars) |
+|---|---|---|---|---|
+| Logic puzzles | **5/5** | 4/5 | 3/5 | 1/5 |
+| Reasoning block never closed | 1/8 | 1/8 | 2/8 | 4/8 |
+| Depth scaled to task | 6/6 | 6/6 | — | 4/6 |
+| Tool: right call / correctly none | 8/10 · 4/6 | 7/10 · 5/6 | **9/10 · 5/6** | 9/10 · 5/6 |
+| Answers in the user's language (es/en/fr/tr/ja) | —¹ | 4/5 | **5/5** | 4/5 |
 
-The DoRA row magnitude is folded exactly into the quantization scales of `W_q`; only the
-rank-8 factors `A`, `B` (232 modules across all 48 layers: `q_proj`, `v_proj`, `gate_proj`,
-`up_proj`, `down_proj`; no `v_proj` in the 8 global-attention layers) live in `model-seitenpfad.safetensors`, listed in
-`model.safetensors.index.json`, with the module list in `seitenpfad.json`.
+¹ Without any prompt the model does not know a portal visitor is writing and answers in German.
 
-## How to load (MLX)
+**Identity with no prompt at all: 11/12 answers as A!ley, 0/12 as another model.**
 
-Gemma 4 is a *unified* multimodal architecture → use **`mlx_vlm`** (plain `mlx_lm` cannot load it).
-The side-path tensors are unknown to the stock model class, so load with `strict=False` and
-apply them with the bundled `seitenpfad.py` (depends only on `mlx` and `mlx_lm`):
+The important finding for anyone using this model: **long system prompts hurt it.** Because
+it learned identity and behaviour without one, a long prompt makes the reasoning collapse
+(shorter, unclosed, wrong). Use no system prompt or a short one.
+
+## How to use (MLX)
+
+Gemma 4 is a *unified* multimodal architecture → load with **`mlx_vlm`**.
 
 ```python
-from mlx_vlm.utils import load_model, load_processor, get_model_path
-from mlx_vlm import generate
-import seitenpfad                      # bundled in this repo
-
-mp = get_model_path("OpenMinded-Labs/AileyCore-12B")
-model = load_model(mp, strict=False)   # strict=True raises on the side-path keys
-processor = load_processor(mp, True)
-n = seitenpfad.anwenden(model, str(mp))
-assert n == 232, "side path not applied — you would be talking to r1"
-
-prompt = "<bos><|turn>user\nWer bist du?<turn|>\n<|turn>model\n"
-print(generate(model, processor, prompt, max_tokens=300, temperature=0.65,
+from mlx_vlm import load, generate
+
+model, processor = load("OpenMinded-Labs/AileyCore-12B")
+prompt = "<bos><|turn>user\nSimon: Wer bist du?<turn|>\n<|turn>model\n"
+print(generate(model, processor, prompt, max_tokens=600, temperature=0.65,
                top_p=0.85, repetition_penalty=1.15, verbose=False).text)
 ```
 
-`anwenden()` wraps each listed module in `mlx_lm.tuner.lora.LoRALinear` (scale 1.0), so
-the quantized matmul stays quantized and the residual costs two small matmuls per token.
-Measured on an M4 (24 GB): decode 4.86 → 4.81 tok/s, prefill −4 %.
-
-The model answers in the native Gemma 4 channel format:
+Answers come in the native channel format:
 
 ```
 <|channel>thought
-…brief reasoning…
+…reasoning, as long as the task needs…
 <channel|>…answer…<turn|>
 ```
 
 Recommended sampling (measured, not the Gemma defaults): temperature 0.65, top_p 0.85,
-repetition penalty 1.15. Seeding the channel opener (`<|channel>thought\n` as assistant
-prefill) makes the reasoning block deterministic.
+repetition penalty 1.15. For tool use pass the tool schemas to `apply_chat_template(tools=…)`
+and return results directly after `<tool_call|>` as
+`<|tool_response>response:NAME{value:<|"|>…<|"|>}<tool_response|>`.
+
+## Training details (round 3)
+
+| Setting | Value |
+|---|---|
+| Starting point | `gemma-4-12B-it-qat-4bit` (fresh — not on top of r1/r2) |
+| Method | **DoRA** (`q_proj`, `v_proj`) + **LoRA** (`gate_proj`, `up_proj`, `down_proj`), **QAT** |
+| Rank / Alpha | 16 / 32 (232 modules, 55 M trainable parameters) |
+| Data | 2 429 curated conversations → 2 676 samples after chunking long documents; 268 validation conversations (296 samples) |
+| Content | conversation, native reasoning with depth levels, native tool calls/responses, RAG/web context as tool results, identity, prose, 441 image rows |
+| Epochs / updates | 1 / 334 (grad accumulation 8, seq 2048, lr 2e-5, warmup 16, gradient checkpointing) |
+| Validation loss | 1.692 (update 25) → **1.318** (best, update 325) |
+| Not shipped | a DPO stage: in the preference pairs the chosen reasoning was ~9× shorter than the rejected one (137 vs 1 228 chars), so it taught "think short", not "think right" |
+| Hardware | Apple M4, 24 GB unified memory |
+| Framework | MLX (`mlx_vlm` + `mlx_lm.tuner`), QAT layers from the A!ley training code |
 
 ## Intended use
 
 - Local, privacy-respecting assistant on Apple Silicon Macs
-- Conversational reasoning, writing, general assistance in DE/EN
+- Conversational reasoning, writing, tool-using assistance in DE/EN (and the user's language)
 - Multimodal understanding (image / audio input) inherited from Gemma 4
 
 ### Out of scope
 
 - Any use prohibited by applicable law
 - Safety-critical, medical, legal, or financial decision-making without human oversight
-- The model can produce inaccurate or biased output; verify important information
-
-## Training details (round 2)
-
-| Setting | Value |
-|---|---|
-| Starting point | AileyCore-12B r1 (July 2026: DoRA q/v + LoRA MLP, rank 8, merged into q6) |
-| Method | **DoRA** (`q_proj`, `v_proj`) + **LoRA** (`gate_proj`, `up_proj`, `down_proj`) |
-| Rank / Alpha | 8 / 16 |
-| Data | 1 255 curated conversation samples (139 validation), native channel format, tool/RAG/web context, no idle-loop residue, deduplicated, cluster-balanced |
-| Epochs / updates | 1 / 162 (grad accumulation 8, seq 1024, lr 2e-5, gradient checkpointing) |
-| Selected checkpoint | best (val 0.836 live) |
-| Not shipped | a DPO stage on synthetic format pairs — every dose taught the model to emit a second `<channel|>` after the answer and repeat itself (2/12 at half strength, endless at full); SFT alone ends cleanly (12/12 closed) |
-| Hardware | Apple M4, 24 GB unified memory |
-| Framework | MLX (`mlx_vlm` + `mlx_lm.tuner`) |
-
-r1's adapter is still merged into the weights; r2 sits on top as the side path. Round-1
-details are in `AILEY_MERGE_INFO.json` of the previous revision of this repo.
 
 ## Limitations & biases
 
-Inherited from Gemma 4 plus the fine-tune: the model may produce factually incorrect,
-outdated, or biased content. It is a persona, not a knowledge base. Occasionally (≈1 in 6
-sampled answers on hard prompts) it skips the reasoning block and answers directly — seeding
-the channel opener removes that. Keep a human in the loop for consequential use.
+Inherited from Gemma 4 plus the fine-tune: it can produce inaccurate, outdated or biased
+content; it is a persona, not a knowledge base — let it search when facts matter. With long
+system prompts reasoning degrades (see above). Keep a human in the loop for consequential use.
 
 ## License & attribution
 
@@ -145,7 +149,7 @@ This model is a **Derivative Work** of Google **Gemma 4**, released under the
 AileyCore-12B is distributed under **Apache 2.0**.
 
 In accordance with Apache 2.0 §4: the base weights were modified via DoRA/LoRA adaptation
-(r1 merged, r2 as side path — see `AILEY_MERGE_INFO.json`); a copy of the license is included
+(QAT adapters, merged — see `AILEY_MERGE_INFO.json`); a copy of the license is included
 (`LICENSE`); attribution notices are in `NOTICE`.
 
 Gemma is a trademark of Google LLC. This project is independent and **not** endorsed by or
@@ -164,6 +168,6 @@ Licensed under the Apache License, Version 2.0 — http://www.apache.org/license
   title  = {AileyCore-12B: A Gemma 4 fine-tune for the A!ley assistant},
   author = {van de Loo, Simon and OpenM!nded},
   year   = {2026},
-  note   = {Round 2: native reasoning channel, rank-8 side path outside the 6-bit quantization}
+  note   = {Round 3: fresh QAT fine-tune, native reasoning and tool formats, identity without a system prompt}
 }
 ```

config.json

@@ -6,10 +6,8 @@
         "_name_or_path": "",
         "architectures": null,
         "audio_embed_dim": 640,
-        "audio_samples_per_token": 640,
         "chunk_size_feed_forward": 0,
-        "dtype": null,
-        "hidden_size": 640,
+        "dtype": "bfloat16",
         "id2label": {
             "0": "LABEL_0",
             "1": "LABEL_1"
@@ -23,7 +21,6 @@
         "model_type": "gemma4_unified_audio",
         "output_attentions": false,
         "output_hidden_states": false,
-        "output_proj_dims": 640,
         "problem_type": null,
         "return_dict": true,
         "rms_norm_eps": 1e-06
@@ -44,19 +41,1172 @@
     "model_type": "gemma4_unified",
     "quantization": {
         "group_size": 64,
-        "bits": 6,
-        "mode": "affine"
+        "bits": 4,
+        "mode": "affine",
+        "language_model.model.layers.0.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.0.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.0.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.1.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.1.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.1.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.2.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.2.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.2.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.3.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.3.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.3.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.4.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.4.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.4.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.5.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.5.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.5.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.6.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.6.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.6.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.7.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.7.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.7.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.8.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.8.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.8.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.9.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.9.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.9.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.10.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.10.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.10.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.11.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.11.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.11.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.12.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.12.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.12.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.13.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.13.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.13.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.14.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.14.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.14.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.15.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.15.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.15.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.16.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.16.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.16.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.17.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.17.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.17.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.18.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.18.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.18.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.19.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.19.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.19.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.20.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.20.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.20.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.21.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.21.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.21.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.22.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.22.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.22.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.23.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.23.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.23.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.24.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.24.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.24.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.25.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.25.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.25.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.26.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.26.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.26.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.27.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.27.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.27.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.28.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.28.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.28.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.29.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.29.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.29.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.30.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.30.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.30.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.31.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.31.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.31.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.32.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.32.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.32.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.33.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.33.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.33.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.34.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.34.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.34.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.35.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.35.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.35.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.36.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.36.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.36.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.37.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.37.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.37.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.38.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.38.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.38.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.39.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.39.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.39.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.40.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.40.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.40.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.41.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.41.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.41.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.42.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.42.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.42.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.43.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.43.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.43.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.44.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.44.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.44.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.45.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.45.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.45.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.46.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.46.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.46.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.47.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.47.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.47.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        }
     },
     "quantization_config": {
         "group_size": 64,
-        "bits": 6,
-        "mode": "affine"
+        "bits": 4,
+        "mode": "affine",
+        "language_model.model.layers.0.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.0.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.0.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.1.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.1.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.1.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.2.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.2.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.2.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.3.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.3.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.3.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.4.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.4.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.4.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.5.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.5.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.5.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.6.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.6.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.6.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.7.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.7.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.7.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.8.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.8.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.8.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.9.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.9.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.9.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.10.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.10.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.10.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.11.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.11.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.11.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.12.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.12.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.12.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.13.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.13.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.13.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.14.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.14.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.14.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.15.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.15.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.15.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.16.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.16.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.16.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.17.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.17.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.17.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.18.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.18.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.18.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.19.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.19.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.19.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.20.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.20.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.20.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.21.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.21.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.21.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.22.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.22.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.22.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.23.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.23.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.23.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.24.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.24.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.24.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.25.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.25.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.25.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.26.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.26.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.26.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.27.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.27.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.27.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.28.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.28.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.28.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.29.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.29.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.29.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.30.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.30.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.30.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.31.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.31.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.31.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.32.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.32.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.32.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.33.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.33.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.33.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.34.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.34.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.34.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.35.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.35.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.35.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.36.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.36.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.36.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.37.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.37.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.37.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.38.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.38.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.38.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.39.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.39.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.39.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.40.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.40.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.40.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.41.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.41.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.41.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.42.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.42.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.42.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.43.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.43.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.43.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.44.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.44.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.44.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.45.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.45.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.45.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.46.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.46.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.46.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.47.mlp.gate_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.47.mlp.down_proj": {
+            "group_size": 64,
+            "bits": 8
+        },
+        "language_model.model.layers.47.mlp.up_proj": {
+            "group_size": 64,
+            "bits": 8
+        }
     },
     "text_config": {
         "attention_bias": false,
         "attention_dropout": 0.0,
         "attention_k_eq_v": true,
         "bos_token_id": 2,
+        "dtype": "bfloat16",
         "enable_moe_block": false,
         "eos_token_id": 1,
         "final_logit_softcapping": 30.0,
@@ -146,7 +1296,7 @@
         "use_cache": true,
         "use_double_wide_mlp": false,
         "vocab_size": 262144,
-        "vocab_size_per_layer_input": 262144
+        "vocab_size_per_layer_input": 0
     },
     "tie_word_embeddings": true,
     "transformers_version": "5.10.0.dev0",
@@ -155,7 +1305,7 @@
         "_name_or_path": "",
         "architectures": null,
         "chunk_size_feed_forward": 0,
-        "dtype": null,
+        "dtype": "bfloat16",
         "id2label": {
             "0": "LABEL_0",
             "1": "LABEL_1"
@@ -168,9 +1318,7 @@
         },
         "mm_embed_dim": 3840,
         "mm_posemb_size": 1120,
-        "model_patch_size": 48,
         "model_type": "gemma4_unified_vision",
-        "num_soft_tokens": 280,
         "output_attentions": false,
         "output_hidden_states": false,
         "output_proj_dims": 3840,

model-00001-of-00002.safetensors → model-00001-of-00003.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:606345b3ecc6568a7aa02deaabccb5a587a06756a6ded0223b92dd6def76b9be
-size 5368252679
+oid sha256:262b6e32389ae0a81eb3726d8ce9207d90b56f5ea05df00ccb15388fcfdffbae
+size 5365107137

model-00002-of-00002.safetensors → model-00002-of-00003.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0ca6078441d8cd137232082b8b2d0f797c127d935073102c17d75f1a02ac954a
-size 4435079526
+oid sha256:d0d7dff62653907d7f6e8ebd878935810a24cbb9a236e11387502f1d7dd0ba99
+size 5366286424

model-seitenpfad.safetensors → model-00003-of-00003.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0556ac7013aa043fddb8f4f25537d1cfd386df035c8b475e00225c0ea5e2da94
-size 54681090
+oid sha256:9ba939d2263d475747dc285b5dccb7ded03d5e4a13390814b6c308fadeebff5c
+size 331089897

seitenpfad.json

@@ -1,241 +0,0 @@
-{
-  "module": [
-    "language_model.model.layers.0.self_attn.q_proj",
-    "language_model.model.layers.0.self_attn.v_proj",
-    "language_model.model.layers.0.mlp.gate_proj",
-    "language_model.model.layers.0.mlp.up_proj",
-    "language_model.model.layers.0.mlp.down_proj",
-    "language_model.model.layers.1.self_attn.q_proj",
-    "language_model.model.layers.1.self_attn.v_proj",
-    "language_model.model.layers.1.mlp.gate_proj",
-    "language_model.model.layers.1.mlp.up_proj",
-    "language_model.model.layers.1.mlp.down_proj",
-    "language_model.model.layers.2.self_attn.q_proj",
-    "language_model.model.layers.2.self_attn.v_proj",
-    "language_model.model.layers.2.mlp.gate_proj",
-    "language_model.model.layers.2.mlp.up_proj",
-    "language_model.model.layers.2.mlp.down_proj",
-    "language_model.model.layers.3.self_attn.q_proj",
-    "language_model.model.layers.3.self_attn.v_proj",
-    "language_model.model.layers.3.mlp.gate_proj",
-    "language_model.model.layers.3.mlp.up_proj",
-    "language_model.model.layers.3.mlp.down_proj",
-    "language_model.model.layers.4.self_attn.q_proj",
-    "language_model.model.layers.4.self_attn.v_proj",
-    "language_model.model.layers.4.mlp.gate_proj",
-    "language_model.model.layers.4.mlp.up_proj",
-    "language_model.model.layers.4.mlp.down_proj",
-    "language_model.model.layers.5.self_attn.q_proj",
-    "language_model.model.layers.5.mlp.gate_proj",
-    "language_model.model.layers.5.mlp.up_proj",
-    "language_model.model.layers.5.mlp.down_proj",
-    "language_model.model.layers.6.self_attn.q_proj",
-    "language_model.model.layers.6.self_attn.v_proj",
-    "language_model.model.layers.6.mlp.gate_proj",
-    "language_model.model.layers.6.mlp.up_proj",
-    "language_model.model.layers.6.mlp.down_proj",
-    "language_model.model.layers.7.self_attn.q_proj",
-    "language_model.model.layers.7.self_attn.v_proj",
-    "language_model.model.layers.7.mlp.gate_proj",
-    "language_model.model.layers.7.mlp.up_proj",
-    "language_model.model.layers.7.mlp.down_proj",
-    "language_model.model.layers.8.self_attn.q_proj",
-    "language_model.model.layers.8.self_attn.v_proj",
-    "language_model.model.layers.8.mlp.gate_proj",
-    "language_model.model.layers.8.mlp.up_proj",
-    "language_model.model.layers.8.mlp.down_proj",
-    "language_model.model.layers.9.self_attn.q_proj",
-    "language_model.model.layers.9.self_attn.v_proj",
-    "language_model.model.layers.9.mlp.gate_proj",
-    "language_model.model.layers.9.mlp.up_proj",
-    "language_model.model.layers.9.mlp.down_proj",
-    "language_model.model.layers.10.self_attn.q_proj",
-    "language_model.model.layers.10.self_attn.v_proj",
-    "language_model.model.layers.10.mlp.gate_proj",
-    "language_model.model.layers.10.mlp.up_proj",
-    "language_model.model.layers.10.mlp.down_proj",
-    "language_model.model.layers.11.self_attn.q_proj",
-    "language_model.model.layers.11.mlp.gate_proj",
-    "language_model.model.layers.11.mlp.up_proj",
-    "language_model.model.layers.11.mlp.down_proj",
-    "language_model.model.layers.12.self_attn.q_proj",
-    "language_model.model.layers.12.self_attn.v_proj",
-    "language_model.model.layers.12.mlp.gate_proj",
-    "language_model.model.layers.12.mlp.up_proj",
-    "language_model.model.layers.12.mlp.down_proj",
-    "language_model.model.layers.13.self_attn.q_proj",
-    "language_model.model.layers.13.self_attn.v_proj",
-    "language_model.model.layers.13.mlp.gate_proj",
-    "language_model.model.layers.13.mlp.up_proj",
-    "language_model.model.layers.13.mlp.down_proj",
-    "language_model.model.layers.14.self_attn.q_proj",
-    "language_model.model.layers.14.self_attn.v_proj",
-    "language_model.model.layers.14.mlp.gate_proj",
-    "language_model.model.layers.14.mlp.up_proj",
-    "language_model.model.layers.14.mlp.down_proj",
-    "language_model.model.layers.15.self_attn.q_proj",
-    "language_model.model.layers.15.self_attn.v_proj",
-    "language_model.model.layers.15.mlp.gate_proj",
-    "language_model.model.layers.15.mlp.up_proj",
-    "language_model.model.layers.15.mlp.down_proj",
-    "language_model.model.layers.16.self_attn.q_proj",
-    "language_model.model.layers.16.self_attn.v_proj",
-    "language_model.model.layers.16.mlp.gate_proj",
-    "language_model.model.layers.16.mlp.up_proj",
-    "language_model.model.layers.16.mlp.down_proj",
-    "language_model.model.layers.17.self_attn.q_proj",
-    "language_model.model.layers.17.mlp.gate_proj",
-    "language_model.model.layers.17.mlp.up_proj",
-    "language_model.model.layers.17.mlp.down_proj",
-    "language_model.model.layers.18.self_attn.q_proj",
-    "language_model.model.layers.18.self_attn.v_proj",
-    "language_model.model.layers.18.mlp.gate_proj",
-    "language_model.model.layers.18.mlp.up_proj",
-    "language_model.model.layers.18.mlp.down_proj",
-    "language_model.model.layers.19.self_attn.q_proj",
-    "language_model.model.layers.19.self_attn.v_proj",
-    "language_model.model.layers.19.mlp.gate_proj",
-    "language_model.model.layers.19.mlp.up_proj",
-    "language_model.model.layers.19.mlp.down_proj",
-    "language_model.model.layers.20.self_attn.q_proj",
-    "language_model.model.layers.20.self_attn.v_proj",
-    "language_model.model.layers.20.mlp.gate_proj",
-    "language_model.model.layers.20.mlp.up_proj",
-    "language_model.model.layers.20.mlp.down_proj",
-    "language_model.model.layers.21.self_attn.q_proj",
-    "language_model.model.layers.21.self_attn.v_proj",
-    "language_model.model.layers.21.mlp.gate_proj",
-    "language_model.model.layers.21.mlp.up_proj",
-    "language_model.model.layers.21.mlp.down_proj",
-    "language_model.model.layers.22.self_attn.q_proj",
-    "language_model.model.layers.22.self_attn.v_proj",
-    "language_model.model.layers.22.mlp.gate_proj",
-    "language_model.model.layers.22.mlp.up_proj",
-    "language_model.model.layers.22.mlp.down_proj",
-    "language_model.model.layers.23.self_attn.q_proj",
-    "language_model.model.layers.23.mlp.gate_proj",
-    "language_model.model.layers.23.mlp.up_proj",
-    "language_model.model.layers.23.mlp.down_proj",
-    "language_model.model.layers.24.self_attn.q_proj",
-    "language_model.model.layers.24.self_attn.v_proj",
-    "language_model.model.layers.24.mlp.gate_proj",
-    "language_model.model.layers.24.mlp.up_proj",
-    "language_model.model.layers.24.mlp.down_proj",
-    "language_model.model.layers.25.self_attn.q_proj",
-    "language_model.model.layers.25.self_attn.v_proj",
-    "language_model.model.layers.25.mlp.gate_proj",
-    "language_model.model.layers.25.mlp.up_proj",
-    "language_model.model.layers.25.mlp.down_proj",
-    "language_model.model.layers.26.self_attn.q_proj",
-    "language_model.model.layers.26.self_attn.v_proj",
-    "language_model.model.layers.26.mlp.gate_proj",
-    "language_model.model.layers.26.mlp.up_proj",
-    "language_model.model.layers.26.mlp.down_proj",
-    "language_model.model.layers.27.self_attn.q_proj",
-    "language_model.model.layers.27.self_attn.v_proj",
-    "language_model.model.layers.27.mlp.gate_proj",
-    "language_model.model.layers.27.mlp.up_proj",
-    "language_model.model.layers.27.mlp.down_proj",
-    "language_model.model.layers.28.self_attn.q_proj",
-    "language_model.model.layers.28.self_attn.v_proj",
-    "language_model.model.layers.28.mlp.gate_proj",
-    "language_model.model.layers.28.mlp.up_proj",
-    "language_model.model.layers.28.mlp.down_proj",
-    "language_model.model.layers.29.self_attn.q_proj",
-    "language_model.model.layers.29.mlp.gate_proj",
-    "language_model.model.layers.29.mlp.up_proj",
-    "language_model.model.layers.29.mlp.down_proj",
-    "language_model.model.layers.30.self_attn.q_proj",
-    "language_model.model.layers.30.self_attn.v_proj",
-    "language_model.model.layers.30.mlp.gate_proj",
-    "language_model.model.layers.30.mlp.up_proj",
-    "language_model.model.layers.30.mlp.down_proj",
-    "language_model.model.layers.31.self_attn.q_proj",
-    "language_model.model.layers.31.self_attn.v_proj",
-    "language_model.model.layers.31.mlp.gate_proj",
-    "language_model.model.layers.31.mlp.up_proj",
-    "language_model.model.layers.31.mlp.down_proj",
-    "language_model.model.layers.32.self_attn.q_proj",
-    "language_model.model.layers.32.self_attn.v_proj",
-    "language_model.model.layers.32.mlp.gate_proj",
-    "language_model.model.layers.32.mlp.up_proj",
-    "language_model.model.layers.32.mlp.down_proj",
-    "language_model.model.layers.33.self_attn.q_proj",
-    "language_model.model.layers.33.self_attn.v_proj",
-    "language_model.model.layers.33.mlp.gate_proj",
-    "language_model.model.layers.33.mlp.up_proj",
-    "language_model.model.layers.33.mlp.down_proj",
-    "language_model.model.layers.34.self_attn.q_proj",
-    "language_model.model.layers.34.self_attn.v_proj",
-    "language_model.model.layers.34.mlp.gate_proj",
-    "language_model.model.layers.34.mlp.up_proj",
-    "language_model.model.layers.34.mlp.down_proj",
-    "language_model.model.layers.35.self_attn.q_proj",
-    "language_model.model.layers.35.mlp.gate_proj",
-    "language_model.model.layers.35.mlp.up_proj",
-    "language_model.model.layers.35.mlp.down_proj",
-    "language_model.model.layers.36.self_attn.q_proj",
-    "language_model.model.layers.36.self_attn.v_proj",
-    "language_model.model.layers.36.mlp.gate_proj",
-    "language_model.model.layers.36.mlp.up_proj",
-    "language_model.model.layers.36.mlp.down_proj",
-    "language_model.model.layers.37.self_attn.q_proj",
-    "language_model.model.layers.37.self_attn.v_proj",
-    "language_model.model.layers.37.mlp.gate_proj",
-    "language_model.model.layers.37.mlp.up_proj",
-    "language_model.model.layers.37.mlp.down_proj",
-    "language_model.model.layers.38.self_attn.q_proj",
-    "language_model.model.layers.38.self_attn.v_proj",
-    "language_model.model.layers.38.mlp.gate_proj",
-    "language_model.model.layers.38.mlp.up_proj",
-    "language_model.model.layers.38.mlp.down_proj",
-    "language_model.model.layers.39.self_attn.q_proj",
-    "language_model.model.layers.39.self_attn.v_proj",
-    "language_model.model.layers.39.mlp.gate_proj",
-    "language_model.model.layers.39.mlp.up_proj",
-    "language_model.model.layers.39.mlp.down_proj",
-    "language_model.model.layers.40.self_attn.q_proj",
-    "language_model.model.layers.40.self_attn.v_proj",
-    "language_model.model.layers.40.mlp.gate_proj",
-    "language_model.model.layers.40.mlp.up_proj",
-    "language_model.model.layers.40.mlp.down_proj",
-    "language_model.model.layers.41.self_attn.q_proj",
-    "language_model.model.layers.41.mlp.gate_proj",
-    "language_model.model.layers.41.mlp.up_proj",
-    "language_model.model.layers.41.mlp.down_proj",
-    "language_model.model.layers.42.self_attn.q_proj",
-    "language_model.model.layers.42.self_attn.v_proj",
-    "language_model.model.layers.42.mlp.gate_proj",
-    "language_model.model.layers.42.mlp.up_proj",
-    "language_model.model.layers.42.mlp.down_proj",
-    "language_model.model.layers.43.self_attn.q_proj",
-    "language_model.model.layers.43.self_attn.v_proj",
-    "language_model.model.layers.43.mlp.gate_proj",
-    "language_model.model.layers.43.mlp.up_proj",
-    "language_model.model.layers.43.mlp.down_proj",
-    "language_model.model.layers.44.self_attn.q_proj",
-    "language_model.model.layers.44.self_attn.v_proj",
-    "language_model.model.layers.44.mlp.gate_proj",
-    "language_model.model.layers.44.mlp.up_proj",
-    "language_model.model.layers.44.mlp.down_proj",
-    "language_model.model.layers.45.self_attn.q_proj",
-    "language_model.model.layers.45.self_attn.v_proj",
-    "language_model.model.layers.45.mlp.gate_proj",
-    "language_model.model.layers.45.mlp.up_proj",
-    "language_model.model.layers.45.mlp.down_proj",
-    "language_model.model.layers.46.self_attn.q_proj",
-    "language_model.model.layers.46.self_attn.v_proj",
-    "language_model.model.layers.46.mlp.gate_proj",
-    "language_model.model.layers.46.mlp.up_proj",
-    "language_model.model.layers.46.mlp.down_proj",
-    "language_model.model.layers.47.self_attn.q_proj",
-    "language_model.model.layers.47.mlp.gate_proj",
-    "language_model.model.layers.47.mlp.up_proj",
-    "language_model.model.layers.47.mlp.down_proj"
-  ],
-  "rank": 8,
-  "quelle": "lora_training/ckpt_core_r2_sft/best",
-  "faktor": 1.0,
-  "basis": "models/AileyCore-12B",
-  "erzeugt": "2026-09-17T00:47:52"
-}

seitenpfad.py

@@ -1,58 +0,0 @@
-"""LoRA-Seitenpfad zur Laufzeit — verlustfrei statt weggerundet.
-
-Befund 16.09.2026: Ein Adapter, der in die 6-bit-Basis gemergt wird, verliert
-beim Rueckrunden fast alles (SFT: 13 % der Wirkung, DPO: ~2 %). Die Basis
-darf nicht feiner werden (Speicher). Also bleibt das Delta ausserhalb der
-Quantisierung: y = Wq·x + (x·A)·B'. Die DoRA-Zeilenmagnitude wird beim Export
-exakt in die Quantisierungs-Scales gefaltet (export_seitenpfad.py), der
-Rank-r-Teil liegt als seitenpfad.safetensors neben den Gewichten.
-
-anwenden(model, model_dir) wickelt die betroffenen Module in mlx_lm's
-LoRALinear (Quantisierung bleibt, kein Entpacken je Aufruf). Kosten:
-r x (in+out) Parameter je Modul, zwei kleine Matmuls je Token — gemessen,
-siehe README der Runde.
-"""
-from __future__ import annotations
-import json, os
-from typing import Optional
-
-
-def _modul(model, pfad: str):
-    obj = model
-    for teil in pfad.split("."):
-        obj = obj[int(teil)] if teil.isdigit() else getattr(obj, teil)
-    return obj
-
-
-def _setze(model, pfad: str, neu):
-    teile = pfad.split(".")
-    eltern = _modul(model, ".".join(teile[:-1]))
-    if teile[-1].isdigit():
-        eltern[int(teile[-1])] = neu
-    else:
-        setattr(eltern, teile[-1], neu)
-
-
-def anwenden(model, model_dir: str) -> int:
-    """Seitenpfad laden, wenn vorhanden. Gibt die Zahl der Module zurueck (0 = keiner)."""
-    meta_p = os.path.join(model_dir, "seitenpfad.json")
-    arr_p = os.path.join(model_dir, "model-seitenpfad.safetensors")   # Shard im Gewichtsindex
-    if not os.path.isfile(arr_p):
-        arr_p = os.path.join(model_dir, "seitenpfad.safetensors")      # alte Ablage (Messungen)
-    if not (os.path.isfile(meta_p) and os.path.isfile(arr_p)):
-        return 0
-    import mlx.core as mx
-    from mlx_lm.tuner.lora import LoRALinear
-    meta = json.load(open(meta_p))
-    arrays = mx.load(arr_p)
-    n = 0
-    for pfad in meta["module"]:
-        basis = _modul(model, pfad)
-        a, b = arrays[f"{pfad}.lora_a"], arrays[f"{pfad}.lora_b"]
-        lora = LoRALinear.from_base(basis, r=a.shape[1], scale=1.0, dropout=0.0)
-        lora.lora_a = a.astype(mx.float16) if a.dtype == mx.float32 else a
-        lora.lora_b = b.astype(mx.float16) if b.dtype == mx.float32 else b
-        _setze(model, pfad, lora)
-        n += 1
-    mx.eval(model.parameters())
-    return n

tokenizer_config.json

@@ -18,6 +18,7 @@
   ],
   "image_token": "<|image|>",
   "is_local": true,
+  "local_files_only": true,
   "mask_token": "<mask>",
   "model_max_length": 1000000000000000019884624838656,
   "model_specific_special_tokens": {