feat(build): stamp chat_template.jinja into the GGUF at build time

--chat-template-file is a llama.cpp flag. LM Studio, Jan, KoboldCpp, Open WebUI
and llama-cpp-python read the template embedded in the GGUF and have nowhere to
put it, so they were still getting the upstream XML tool-call form and the empty
completions it causes.

strip_mtp.py gains --chat-template FILE, which swaps tokenizer.chat_template in
the same pass that drops the MTP block; build.sh passes chat_template.jinja
automatically when present. A GGUF built here now needs no flags on any loader.

With --chat-template on an already-clean input the file is rewritten purely to
carry the new template, since the hardlink passthrough cannot. Tensor data is
copied byte-for-byte in every path. An empty template file is refused.

Verified on synthetic GGUFs across all six paths, including that a clean input
with no stamp still hardlinks (same inode) and that the real chat_template.jinja
reads back byte-identical after stamping.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ans2THL2VJutsXi6RSFe4Y

CHANGELOG.md

@@ -8,6 +8,30 @@ track the **tooling and documentation**, not the underlying base model.
 
 ## [Unreleased]
 
+### Added (chat template is stamped into the GGUF at build time)
+- **`scripts/strip_mtp.py` gained `--chat-template FILE`**, which replaces the
+  GGUF's `tokenizer.chat_template` in the same pass that drops the MTP block.
+  `scripts/build.sh` now passes `chat_template.jinja` automatically when the file
+  is present. This matters because `--chat-template-file` is a llama.cpp flag:
+  LM Studio, Jan, KoboldCpp, Open WebUI and llama-cpp-python all read the template
+  **embedded in the GGUF** and have nowhere to put that flag, so they were still
+  getting the upstream XML tool-call form and the empty completions that come with
+  it. A GGUF built here now carries the working template, and those loaders need
+  no flags at all.
+- The script stays a strip tool: with `--chat-template` on an already-MTP-clean
+  input it rewrites the file purely to swap the template (the usual hardlink
+  passthrough cannot carry one), and still hardlinks when neither a strip nor a
+  stamp is needed. Tensor data is copied byte-for-byte in every path — no
+  re-quantization, no quality change. An empty template file is refused rather
+  than stamped.
+- Verified against synthetic GGUFs across all six paths: MTP-strip alone;
+  MTP-strip plus stamp; clean input plus stamp (rewrites, block_count preserved,
+  tensor bytes intact); clean input with no stamp (hardlink, same inode); empty
+  template file rejected; and this repo's real `chat_template.jinja` stamped and
+  read back byte-identical.
+- Documented in the README: a locally built GGUF needs no flag, and there is a
+  one-liner to restamp a GGUF you already have.
+
 ### Fixed (provenance, and a CLI-detection footgun)
 - **`base_model_relation: finetune` → `quantized`.** Nothing is finetuned in this
   repo; it repackages and quantizes an upstream abliteration. A new

README.md

@@ -126,7 +126,7 @@ baked in, so context past ~262K degrades). See
 | `scripts/build.sh` | Pulls a GGUF from `llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-GGUF` (default Q4_K_M), runs it through `strip_mtp.py`, then runs `ollama create janus`. Quants published there: BF16, Q3_K_L, Q3_K_M, Q4_K_M, Q4_K_S, Q5_K_M, Q5_K_S, Q6_K, Q8_0. The bundled Q4_K_M is already this Heretic quant (stripped); use this to build the others locally. |
 | `scripts/check_bridge_sync.py` | Run before pushing a `Modelfile` / `template` / `system` / `params` edit to verify the four configurations remain in sync. Exits 0 if in sync, 1 with a per-key diff if not. |
 | `scripts/verify_arch.py` | Cross-checks the README Architecture bullets (layer count, hidden size, dense FFN width, native context, vocab) plus the underlying forward-pass structure (Gated Attention + Gated DeltaNet head dims, partial RoPE, full-attention interval) against the bundled GGUF's `qwen35` metadata. Run on demand (`python3 scripts/verify_arch.py`); loads the ~17 GB GGUF (LFS smudge required) and exits non-zero on any mismatch. `block_count` is checked against **64** — the MTP-stripped value — so an unstripped bundle reports 65 and fails. |
-| `scripts/strip_mtp.py` | Drops the MTP / NextN layer from a `qwen35` GGUF (`python3 scripts/strip_mtp.py IN.gguf OUT.gguf`): removes the extra `blk.<last>.*` block, decrements `block_count`, drops `nextn_predict_layers`. Kept tensors are copied byte-for-byte (no re-quant); a conditional no-op (hardlink) on already-clean quants. **Required, not optional:** the base is MTP-preserved, so quants from the source repo carry a NextN block at index 64 that stock llama.cpp / Ollama cannot load. `build.sh` runs this on every fetched quant, and the bundled Q4_K_M ships already stripped (`block_count` 64). Parity with the dense sibling `FoolDev/Thanatos-27B-HERETIC`. |
+| `scripts/strip_mtp.py` | Drops the MTP / NextN layer from a `qwen35` GGUF (`python3 scripts/strip_mtp.py IN.gguf OUT.gguf`): removes the extra `blk.<last>.*` block, decrements `block_count`, drops `nextn_predict_layers`. Kept tensors are copied byte-for-byte (no re-quant); a conditional no-op (hardlink) on already-clean quants. **Required, not optional:** the base is MTP-preserved, so quants from the source repo carry a NextN block at index 64 that stock llama.cpp / Ollama cannot load. With `--chat-template FILE` it also restamps the GGUF's embedded chat template in the same pass (see [Tool / function calling](#tool--function-calling)). `build.sh` runs this on every fetched quant, and the bundled Q4_K_M ships already stripped (`block_count` 64). Parity with the dense sibling `FoolDev/Thanatos-27B-HERETIC`. |
 | `scripts/smoke_test.sh` | Integration smoke test against a running Ollama daemon: server reachable, model loaded, `tools` capability present, chat round-trip, and no control-token leakage. `TOOLS_TEST=1` adds a tool-call round-trip. Defaults to `MODEL=janus`. |
 | `scripts/bench.sh` | Measures tok/s from Ollama's `eval_count` / `eval_duration` over a short/medium/long prompt mix (with a discarded warmup). Defaults to `MODEL=janus`. |
 | `scripts/load_bundle.sh` | Loads the bundled `Janus-27B.Q4_K_M.gguf` into Ollama as a local `janus` tag without an upstream pull (smudges the LFS pointer via `hf download` if needed, checks the arch is `qwen35`). |
@@ -201,6 +201,7 @@ Drop the GGUF into your loader of choice. The chat template is embedded in the G
 llama-server -m Janus-27B.Q4_K_M.gguf --jinja --chat-template-file chat_template.jinja
 ```
 
+A GGUF built by `./scripts/build.sh` has that template baked in and needs no flag.
 See [Tool / function calling](#tool--function-calling) for what differs and why.
 
 ### Ollama
@@ -504,6 +505,21 @@ llama-server -m Janus-27B.Q4_K_M.gguf --jinja --chat-template-file chat_template
 
 With that flag the wire format is identical on Ollama and llama.cpp, so the same client code works against either.
 
+> **A locally built GGUF needs no flag.** `./scripts/build.sh` (`make build`) now
+> stamps `chat_template.jinja` into the GGUF it produces, in the same pass that
+> strips the MTP block — so LM Studio, Jan, KoboldCpp and anything else that reads
+> the *embedded* template get working tool calls with no flags at all. The
+> `--chat-template-file` flag above is only needed for a GGUF that was not built
+> that way: the bundled blob, or a quant pulled straight from upstream. To restamp
+> one you already have:
+>
+> ```bash
+> python3 scripts/strip_mtp.py IN.gguf OUT.gguf --chat-template chat_template.jinja
+> ```
+>
+> On an already-MTP-clean input that rewrites the file purely to swap the template;
+> tensor data is copied byte-for-byte either way, so there is no re-quantization.
+
 #### Example (Ollama, OpenAI-compatible API)
 
 ```python

scripts/build.sh

@@ -105,7 +105,18 @@ fi
 # upstream safetensors under vLLM / SGLang, or the unstripped GGUF straight from
 # the source repo — not this path.
 STRIPPED_PATH="${GGUF_PATH%.gguf}.stripped.gguf"
-python3 "${ROOT}/scripts/strip_mtp.py" "${GGUF_PATH}" "${STRIPPED_PATH}"
+# Stamp this repo's chat template into the GGUF in the same pass. llama.cpp,
+# LM Studio, Jan and KoboldCpp all read the template embedded in the GGUF, and
+# the upstream one instructs the model to emit tool calls in an XML dialect that
+# llama.cpp's --jinja grammar rejects — which surfaces as an empty completion the
+# moment a `tools` array is present. Stamping it here means those loaders get
+# working tool calls with no extra flags. Skipped if the file is absent.
+STAMP_ARGS=()
+if [[ -f "${ROOT}/chat_template.jinja" ]]; then
+    STAMP_ARGS=(--chat-template "${ROOT}/chat_template.jinja")
+    echo "[*] stamping chat_template.jinja into the built GGUF"
+fi
+python3 "${ROOT}/scripts/strip_mtp.py" "${GGUF_PATH}" "${STRIPPED_PATH}" ${STAMP_ARGS[@]+"${STAMP_ARGS[@]}"}
 GGUF_PATH="${STRIPPED_PATH}"
 echo "[*] gguf (post-strip): ${GGUF_PATH}"
 

scripts/strip_mtp.py

@@ -21,11 +21,27 @@ or taking the unstripped GGUF straight from the source repo.
 Every KEPT tensor is copied byte-for-byte (raw quantized data) — no
 re-quantization, so no quality change to the model itself.
 
+Optionally also restamp the embedded chat template in the same pass:
+
+    --chat-template FILE   replace tokenizer.chat_template with FILE's contents
+
+That matters because llama.cpp, LM Studio, Jan and KoboldCpp all read the
+template baked into the GGUF, and the upstream one emits tool calls in an XML
+dialect that llama.cpp's --jinja grammar rejects — the caller just gets an empty
+completion. Stamping this repo's chat_template.jinja in at build time fixes tool
+calling for every one of those loaders with no extra flags. build.sh does this
+automatically when chat_template.jinja is present.
+
+With --chat-template and no MTP block to strip, the file is still rewritten (the
+usual hardlink passthrough would not carry the new template).
+
 Usage:
     python3 scripts/strip_mtp.py IN.gguf OUT.gguf
+    python3 scripts/strip_mtp.py IN.gguf OUT.gguf --chat-template chat_template.jinja
 """
 from __future__ import annotations
 
+import argparse
 import sys
 
 import numpy as np
@@ -47,10 +63,27 @@ def decode_field(field):
 
 
 def main() -> int:
-    if len(sys.argv) != 3:
-        print("usage: strip_mtp.py IN.gguf OUT.gguf", file=sys.stderr)
-        return 2
-    src, dst = sys.argv[1], sys.argv[2]
+    ap = argparse.ArgumentParser(
+        description="Strip the MTP/NextN block from a GGUF; optionally restamp its chat template.")
+    ap.add_argument("src", metavar="IN.gguf")
+    ap.add_argument("dst", metavar="OUT.gguf")
+    ap.add_argument("--chat-template", metavar="FILE", default=None,
+                    help="replace tokenizer.chat_template with the contents of FILE")
+    args = ap.parse_args()
+    src, dst = args.src, args.dst
+
+    new_template = None
+    if args.chat_template:
+        try:
+            with open(args.chat_template, encoding="utf-8") as fh:
+                new_template = fh.read()
+        except OSError as exc:
+            print(f"[!] cannot read {args.chat_template}: {exc}", file=sys.stderr)
+            return 1
+        if not new_template.strip():
+            print(f"[!] {args.chat_template} is empty; refusing to stamp a blank template",
+                  file=sys.stderr)
+            return 1
 
     reader = GGUFReader(src, "r")
     arch = decode_field(reader.get_field("general.architecture"))
@@ -68,7 +101,10 @@ def main() -> int:
     # caller always gets a valid output without touching real transformer layers.
     has_nextn = decode_field(reader.get_field(f"{arch}.nextn_predict_layers")) or 0
     last_is_mtp = any(t.name.startswith(prefix) and "nextn" in t.name.lower() for t in reader.tensors)
-    if not has_nextn and not last_is_mtp:
+    strip = bool(has_nextn or last_is_mtp)
+    # The hardlink passthrough cannot carry a new chat template, so it only
+    # applies when there is nothing at all to rewrite.
+    if not strip and new_template is None:
         print(f"[=] no MTP/NextN layer (block_count={block_count}); copying through unchanged")
         if src != dst:
             import os
@@ -84,12 +120,15 @@ def main() -> int:
     # We only ever drop the last block, so it MUST be the MTP/NextN block. If
     # nextn_predict_layers is set but blk.<last>.* holds no NextN tensor, the MTP
     # block isn't where we assume — refuse rather than drop a real transformer layer.
-    if not last_is_mtp:
+    if strip and not last_is_mtp:
         print(f"[!] {arch}.nextn_predict_layers is set but blk.{last}.* has no NextN tensor; "
               "refusing to strip (would drop a real transformer layer).", file=sys.stderr)
         return 1
 
-    print(f"[*] arch={arch} block_count={block_count} -> stripping MTP block {last}, new block_count={block_count-1}")
+    if strip:
+        print(f"[*] arch={arch} block_count={block_count} -> stripping MTP block {last}, new block_count={block_count-1}")
+    else:
+        print(f"[=] arch={arch} block_count={block_count}; no MTP block, rewriting to restamp the chat template")
 
     # Writer manages general.architecture itself; block_count/nextn we override/drop.
     skip_keys = {
@@ -97,9 +136,11 @@ def main() -> int:
         f"{arch}.block_count",
         f"{arch}.nextn_predict_layers",
     }
+    if new_template is not None:
+        skip_keys.add("tokenizer.chat_template")
 
     writer = GGUFWriter(dst, arch=arch)
-    writer.add_block_count(block_count - 1)
+    writer.add_block_count(block_count - 1 if strip else block_count)
 
     n_kv = 0
     for name, field in reader.fields.items():
@@ -117,8 +158,14 @@ def main() -> int:
             writer.add_key_value(name, val, vtype)
         n_kv += 1
 
-    kept = [t for t in reader.tensors if not t.name.startswith(prefix)]
-    dropped = [t.name for t in reader.tensors if t.name.startswith(prefix)]
+    if new_template is not None:
+        writer.add_string("tokenizer.chat_template", new_template)
+        n_kv += 1
+        print(f"[*] chat template restamped from {args.chat_template} "
+              f"({len(new_template)} chars)")
+
+    kept = [t for t in reader.tensors if not (strip and t.name.startswith(prefix))]
+    dropped = [t.name for t in reader.tensors if strip and t.name.startswith(prefix)]
     print(f"[*] copied {n_kv} KV keys; keeping {len(kept)} tensors, dropping {len(dropped)} (block {last})")
 
     for t in kept: