FoolDev Claude Opus 5 (1M context) commited on
Commit
c847255
·
1 Parent(s): 6708e8f

release 0.7.0: reverse the rebase — back to the Qwen 3.6 35B-A3B MoE

Browse files

Releases 0.2.0 through 0.6.5 shipped a dense Qwen 3.8 27B under this slug.
That rebase is reversed: the model is llmfan46/Qwen3.6-35B-A3B-uncensored-heretic
again — ~34.7B total / ~3B active, 40 layers (10 full-attention + 30 Gated
DeltaNet), 256 experts with 8 routed per token plus a shared expert, arch
qwen35moe. Display name Janus-27B -> Janus-35B, so the Hub slug names the model
correctly again.

- Blob: Janus-35B-A3B.Q4_K_M.gguf restored as a pointer-only commit
(21,233,608,512 B, sha256 b82f9f21…), byte-identical to upstream's Q4_K_M and
to what this repo shipped until da34705. HF still holds that LFS object for
this repo, so nothing is re-uploaded. No re-quantization, no MTP strip (this
quant has no NextN block), no template restamp.
- chat_template.jinja is deleted: it was a patch of the dense base's 3.8
template, and Qwen 3.6's own template — which the GGUF carries — has no
reasoning_effort, tool_call_format or preserve_reasoning to patch. The card's
llama.cpp sections were rewritten from what that template actually does,
including that string tool-call arguments are emitted as-is rather than
raising. check_chat_template.py and check.sh's check 8 go with it.
- Every measured figure taken on the dense 27B is deleted, not rescaled. Memory
totals are now derived from GGUF geometry and labelled as such (KV is
20,480 B/token — 0.625 GiB per 32K f16, ~5.0 GiB at the 262144 default); no
tok/s figures are published at all.
- verify_arch.py rewritten for the MoE claims; check 5's MTP grep, which would
have been permanently vacuous against 3.6 filenames, replaced by a check that
the Modelfile's FROM is the bundled GGUF *and* that it is tracked by git;
check.sh now reports "N of M checks passed, K skipped" instead of a green
"all passed" over a skipped check.
- moe-routing.svg restored verbatim, banner wordmark back to JANUS-35B, and
banner.png restored byte-for-byte to avoid re-rasterization drift.
- Fixed while auditing: front-matter claimed library_name transformers and
pipeline_tag image-text-to-text on a GGUF-only repo (now gguf /
text-generation); .gitignore's !Janus-27B.*.gguf whitelist would have silently
dropped the new blob; the undated "Ollama 0.24+" floor is replaced by the
versions actually checked in source.

Not done here: no live run of any kind — live_check.sh, a tool-call round-trip,
verify_arch.py and every memory measurement need the 21 GB blob on disk. The
card says so wherever it matters.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

.gitignore CHANGED
@@ -7,13 +7,13 @@ venv/
7
 
8
  # Local model weights. We don't redistribute the upstream Qwen GGUFs
9
  # here — scripts/build.sh fetches one from
10
- # llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-GGUF
11
- # locally. The single Janus-27B.*.gguf we DO ship backs the HF/Ollama
12
  # "Use this model" widget (ollama run hf.co/FoolDev/Janus-35B-HERETIC).
13
  *.gguf
14
- !Janus-27B.*.gguf
15
  # scripts/strip_mtp.py + build.sh intermediates — local only, never tracked.
16
- Janus-27B.*.stripped.gguf
17
  *.safetensors
18
  *.bin
19
 
 
7
 
8
  # Local model weights. We don't redistribute the upstream Qwen GGUFs
9
  # here — scripts/build.sh fetches one from
10
+ # llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF
11
+ # locally. The single Janus-35B-A3B.*.gguf we DO ship backs the HF/Ollama
12
  # "Use this model" widget (ollama run hf.co/FoolDev/Janus-35B-HERETIC).
13
  *.gguf
14
+ !Janus-35B-A3B.*.gguf
15
  # scripts/strip_mtp.py + build.sh intermediates — local only, never tracked.
16
+ Janus-35B-A3B.*.stripped.gguf
17
  *.safetensors
18
  *.bin
19
 
CHANGELOG.md CHANGED
@@ -8,6 +8,106 @@ track the **tooling and documentation**, not the underlying base model.
8
 
9
  ## [Unreleased]
10
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
11
  ## [0.6.5] — 2026-09-17
12
 
13
  ### Documentation (upstream claims re-checked against Ollama 0.34.2)
 
8
 
9
  ## [Unreleased]
10
 
11
+ ## [0.7.0] — 2026-09-18
12
+
13
+ ### Changed (reversal → Qwen 3.6 35B-A3B MoE)
14
+
15
+ - **This repo is the 35B-A3B MoE again.** Releases 0.2.0 through 0.6.5 carried a
16
+ dense Qwen 3.8 27B under the `Janus-35B-HERETIC` slug; that rebase is reversed.
17
+ The model is `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic` — ~34.7B parameters
18
+ total, ~3B active per token, 40 layers (10 full-attention + 30 Gated DeltaNet),
19
+ 256 experts with 8 routed per token plus a shared expert, hidden 2048, 16 Q /
20
+ 2 KV heads at head_dim 256, vocab 248,320, 262,144 native context. Display name
21
+ `Janus-27B` → **`Janus-35B`**, and the Hub slug now matches the model again.
22
+ - **The bundled blob is upstream's own Q4_K_M, byte-for-byte.**
23
+ `Janus-35B-A3B.Q4_K_M.gguf`, 21,233,608,512 bytes, sha256
24
+ `b82f9f2155eb9c07139d48cc3a37880cf9d0edcca345ef6ad829b62941ccbb82` — identical to
25
+ `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF`'s Q4_K_M and to the blob this
26
+ repo shipped until `da34705`. HF still holds that LFS object for this repo (checked
27
+ against the LFS batch API), so it was restored as a pointer-only commit: no
28
+ re-quantization, no re-upload. `Janus-27B.Q4_K_M.gguf` is no longer published and
29
+ remains in git history.
30
+ - **The MTP narrative inverts.** 0.2.0 made the MTP strip "a required build step, not
31
+ a safety net" because the dense 3.8 base was MTP-preserved. This quant has no NextN
32
+ block at all — `block_count` 40, `blk.0`…`blk.39`, 733 tensors, no `nextn` key, read
33
+ from the published file's header by range request — so `scripts/strip_mtp.py` is a
34
+ defensive no-op here and stays in the build path only for anyone pointing it at
35
+ llmfan46's separate `…-Native-MTP-Preserved-GGUF` variant (whose Q4_K_M has
36
+ `block_count` 41 with a NextN block at index 40, not 64).
37
+ - **The GGUF arch stamp widens back to `qwen35moe`** — llama.cpp's MoE entry
38
+ (`src/models/qwen35moe.cpp`, `case 40: LLM_TYPE_35B_A3B`), handled through Ollama's
39
+ llama.cpp compat shim at v0.33.3, v0.34.0 and v0.34.2 (source, 2026-09-18). 0.2.0's
40
+ claim that `qwen35moe` "no longer appears in any live file" is superseded.
41
+ - **`moe-routing.svg` is back**, restored verbatim from `7ed28e6`: the 16×16 grid of
42
+ 256 expert dots with 8 lit at a time is literally this model's routing again. The
43
+ banner wordmark is `JANUS-35B` with the `Qwen 3.6 Heretic · MoE 35B/3B · Fable 5`
44
+ subtitle, and `banner.png` is the pre-rebase raster byte-for-byte, so there is no
45
+ re-rasterization drift.
46
+
47
+ ### Removed (`chat_template.jinja`, and the behaviour that lived in it)
48
+
49
+ - **This repo no longer ships a Jinja chat template.** The file was created *after*
50
+ the rebase as a patch of the dense base's own `qwen3.8-safe-v3.1` template; Qwen 3.6's
51
+ template has no `reasoning_effort`, no `tool_call_format` and no `preserve_reasoning`
52
+ to patch, so there was nothing to carry across. The GGUF keeps the base model's own
53
+ embedded template (11,793 chars), which is what llama.cpp, LM Studio and KoboldCpp
54
+ now render. `scripts/verify_arch.py` gates its embedded-template comparison on the
55
+ repo file existing, so that check correctly skips.
56
+ - **Gone with it on the llama.cpp path:** the `reasoning_effort` tier table (Ollama
57
+ still has it, via the Go template and Ollama's think levels), the `tool_call_format`
58
+ JSON opt-in, and the "conflicting `preserve_thinking` / `preserve_reasoning` raises
59
+ HTTP 500" failure mode — Qwen 3.6's template has no such check and cannot raise that
60
+ way. What is unchanged: the XML tool-call wire format (its instruction literal is
61
+ byte-identical to 3.8's) and reasoning replay, which is that template's own default
62
+ (`preserve_thinking` defaults true, reading `message.reasoning_content`).
63
+ - **One behaviour gets quieter, and worse.** Handed `function.arguments` as a JSON
64
+ string, the 3.8 template raised; Qwen 3.6's emits the string as-is inside
65
+ `<function=…>` with an upstream comment telling you to normalise first. llama-server,
66
+ KoboldCpp and LM Studio all convert to a mapping before rendering, so in practice the
67
+ XML form still renders — but a loader that passes the string through now produces a
68
+ malformed prompt instead of an error.
69
+ - `scripts/check_chat_template.py` and check.sh's check 8 are removed with the file.
70
+
71
+ ### Changed (every measured number on the card)
72
+
73
+ - **All memory and throughput figures measured on the dense 27B are deleted, not
74
+ rescaled.** The geometry moves in both directions — weights 16.1 → 19.78 GiB, but KV
75
+ from 16 full-attention layers × 4 KV heads to 10 × 2 — so the KV cache is 20,480
76
+ B/token at f16 (0.625 GiB per 32K, ~5.0 GiB at the 262144 default), under a third of
77
+ the dense figure, and the derived total at the default window is ~25 GiB f16 /
78
+ ~22.4 GiB q8_0 against the 27B's measured ~34/~26. Those totals are **derived from
79
+ GGUF geometry and explicitly labelled as such**; the recurrent state and compute
80
+ buffer are not derivable and are not yet measured on this blob.
81
+ - **No tok/s figures are published at all.** A 35B-A3B MoE touches ~3B parameters per
82
+ token instead of 27B; rescaling the 4.97 tok/s CPU measurement or the Strix Halo
83
+ offload series would be invention. The iGPU *mechanism* (mmap off, GTT is system RAM,
84
+ `LLAMA_ARG_NO_HOST=1`) is loader-side and is kept.
85
+ - The native context window is unchanged at 262,144 / 1,010,000 with YaRN, so every
86
+ window figure and the `--rope-scale 3.853` recipe survive verbatim.
87
+
88
+ ### Fixed (found while auditing the card, unrelated to the swap)
89
+
90
+ - **Front-matter routed the repo wrongly.** `library_name: transformers` and
91
+ `pipeline_tag: image-text-to-text` were inherited verbatim from `64b629a` and never
92
+ revisited, while the repo has never contained a `config.json`, tokenizer or
93
+ safetensors — so HF offered a Transformers snippet that cannot resolve. Now
94
+ `library_name: gguf` and `pipeline_tag: text-generation`, with `qwen36`, `moe` and
95
+ `ollama` added to the tags.
96
+ - **`.gitignore` would have silently dropped the new blob**: the whitelist was
97
+ `!Janus-27B.*.gguf` against a `*.gguf` rule. Now `!Janus-35B-A3B.*.gguf`, verified
98
+ with `git check-ignore`.
99
+ - **The loader table's `Ollama 0.24+` floor** was the card's only undated, unsourced
100
+ version claim and predates the arch change. Replaced with the three versions actually
101
+ checked in source (0.33.3, 0.34.0, 0.34.2).
102
+
103
+ ### Not done in this release
104
+
105
+ - No live run. `scripts/live_check.sh`, a tool-call round-trip, `verify_arch.py` and
106
+ any memory measurement all need the 21 GB blob on disk; none has been executed
107
+ against this model since the reversal, and the card says so wherever it matters.
108
+ - The local `janus` Ollama tag on the maintainer's host still holds the dense 27B
109
+ bundle until it is rebuilt.
110
+
111
  ## [0.6.5] — 2026-09-17
112
 
113
  ### Documentation (upstream claims re-checked against Ollama 0.34.2)
CITATION.cff CHANGED
@@ -1,7 +1,7 @@
1
  cff-version: 1.2.0
2
- version: 0.6.5
3
- date-released: "2026-09-17"
4
- title: "Janus-27B: A Dense Distillation Wrapper for llmfan46's Qwen 3.8 27B Ultra-Uncensored Heretic"
5
  message: "If you use this model card or its accompanying files, please cite as below."
6
  type: software
7
  authors:
@@ -10,22 +10,26 @@ authors:
10
  repository-code: "https://huggingface.co/FoolDev/Janus-35B-HERETIC"
11
  url: "https://huggingface.co/FoolDev/Janus-35B-HERETIC"
12
  abstract: >-
13
- Janus-27B is a personal repackaging of llmfan46's Ultra-Uncensored
14
- Heretic abliteration (MPOA-style, MTP-preserved) of the dense Qwen 3.8
15
- 27B base (27B parameters, all active, 64 layers, gated DeltaNet linear
16
- attention with full attention every 4th layer) with Claude Fable 5 in
17
- the reasoning teacher slot. The repository ships an Ollama Modelfile,
18
- the HF Ollama-bridge files (template / system / params), sampling
19
- defaults, and a bundled Q4_K_M GGUF (~17 GB, MTP / NextN block stripped
20
- so stock llama.cpp and Ollama can load it) so the HF "Use this model"
21
- widget surfaces a one-liner Ollama snippet. Other quants and the Heretic
22
- safetensors (llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved),
23
- which keep the MTP head for vLLM / SGLang, are pulled from upstream on
24
- demand rather than redistributed.
 
 
 
25
  keywords:
26
  - qwen
27
- - qwen3.8
28
- - dense
 
29
  - abliteration
30
  - distillation
31
  - reasoning
@@ -35,17 +39,17 @@ keywords:
35
  license: Apache-2.0
36
  references:
37
  - type: software
38
- title: "Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved (immediate base)"
39
  authors:
40
  - name: llmfan46
41
- url: "https://huggingface.co/llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved"
42
  - type: software
43
- title: "Qwen3.8-27B (upstream base)"
44
  authors:
45
  - name: Alibaba Qwen Team
46
- url: "https://huggingface.co/Qwen/Qwen3.8-27B"
47
  - type: software
48
- title: "Thanatos-27B (sibling, ARA abliteration)"
49
  authors:
50
  - name: FoolDev
51
  url: "https://huggingface.co/FoolDev/Thanatos-27B-HERETIC"
 
1
  cff-version: 1.2.0
2
+ version: 0.7.0
3
+ date-released: "2026-09-18"
4
+ title: "Janus-35B: A Sparse Distillation Wrapper for llmfan46's Qwen 3.6 35B-A3B Uncensored Heretic"
5
  message: "If you use this model card or its accompanying files, please cite as below."
6
  type: software
7
  authors:
 
10
  repository-code: "https://huggingface.co/FoolDev/Janus-35B-HERETIC"
11
  url: "https://huggingface.co/FoolDev/Janus-35B-HERETIC"
12
  abstract: >-
13
+ Janus-35B is a personal repackaging of llmfan46's uncensored Heretic
14
+ abliteration of the Qwen 3.6 35B-A3B mixture-of-experts base (~34.7B
15
+ parameters total, ~3B active per token, 40 layers, 256 experts with 8
16
+ routed per token plus a shared expert, gated DeltaNet linear attention
17
+ with full attention every 4th layer) with Claude Fable 5 in the
18
+ reasoning teacher slot. The repository ships an Ollama Modelfile, the HF
19
+ Ollama-bridge files (template / system / params), sampling defaults, and
20
+ a bundled Q4_K_M GGUF (21.23 GB) so the HF "Use this model" widget
21
+ surfaces a one-liner Ollama snippet. That GGUF is upstream's own Q4_K_M
22
+ byte-for-byte: it carries no MTP / NextN block to strip and keeps the
23
+ base model's own embedded chat template. Other quants, the BF16 vision
24
+ projector, and the Heretic safetensors
25
+ (llmfan46/Qwen3.6-35B-A3B-uncensored-heretic), which keep the MTP layer
26
+ for vLLM / SGLang, are pulled from upstream on demand rather than
27
+ redistributed.
28
  keywords:
29
  - qwen
30
+ - qwen3.6
31
+ - moe
32
+ - mixture-of-experts
33
  - abliteration
34
  - distillation
35
  - reasoning
 
39
  license: Apache-2.0
40
  references:
41
  - type: software
42
+ title: "Qwen3.6-35B-A3B-uncensored-heretic (immediate base)"
43
  authors:
44
  - name: llmfan46
45
+ url: "https://huggingface.co/llmfan46/Qwen3.6-35B-A3B-uncensored-heretic"
46
  - type: software
47
+ title: "Qwen3.6-35B-A3B (upstream base)"
48
  authors:
49
  - name: Alibaba Qwen Team
50
+ url: "https://huggingface.co/Qwen/Qwen3.6-35B-A3B"
51
  - type: software
52
+ title: "Thanatos-27B (sibling, dense Qwen 3.8 ARA abliteration)"
53
  authors:
54
  - name: FoolDev
55
  url: "https://huggingface.co/FoolDev/Thanatos-27B-HERETIC"
Janus-27B.Q4_K_M.gguf → Janus-35B-A3B.Q4_K_M.gguf RENAMED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:07e6aa5317eb0309863da78d3ae387362239fde220d3d5a081e41723814a6d85
3
- size 17314193760
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b82f9f2155eb9c07139d48cc3a37880cf9d0edcca345ef6ad829b62941ccbb82
3
+ size 21233608512
Makefile CHANGED
@@ -1,4 +1,4 @@
1
- # Janus-27B convenience Makefile.
2
  #
3
  # All work is delegated to scripts/* — this file just gives common
4
  # operations short, discoverable names.
@@ -11,9 +11,9 @@
11
  # MODEL model tag for smoke (default: $(TAG))
12
  #
13
  # Examples:
14
- # make build # Q4_K_M from the Heretic GGUF repo (MTP-stripped on the way in)
15
  # make build QUANT=Q5_K_M # higher-quality quant
16
- # make build GGUF_PATH=~/models/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-Q4_K_M.gguf
17
  # make load-bundle # this repo's bundled GGUF -> local Ollama tag (smudge LFS if needed)
18
  # make smoke
19
  # make check
@@ -39,7 +39,7 @@ ifdef GGUF_PATH
39
  @echo " GGUF_PATH=$(GGUF_PATH)"
40
  endif
41
 
42
- build: ## Download a Heretic GGUF, strip the MTP block, and run 'ollama create'.
43
  GGUF_PATH=$(GGUF_PATH) TAG=$(TAG) ./scripts/build.sh $(QUANT)
44
 
45
  load-bundle: ## Load THIS repo's bundled GGUF into a local Ollama tag (smudge LFS + ollama create).
@@ -60,7 +60,7 @@ bench: ## Measure tok/s using Ollama's eval timing (3 prompts).
60
  mmproj: ## Fetch the vision projector for llama.cpp (Ollama vision is broken upstream).
61
  ./scripts/fetch_vision.sh $(PRECISION)
62
 
63
- check: ## Lint, bridge-file sync, chat-template render checks and the Go-template guard; block the unstripped-MTP footgun.
64
  ./scripts/check.sh
65
 
66
  live-check: ## Live end-to-end checks in an isolated Ollama (GGUF=path or GGUF_PATH=path; tens of minutes on CPU).
@@ -74,6 +74,6 @@ hooks: ## Install scripts/check.sh as the git pre-commit hook.
74
 
75
  clean: ## Remove local GGUF copies and ephemeral caches in this repo.
76
  @echo "[*] removing local GGUFs and ephemeral caches in $$PWD"
77
- @rm -f ./Qwen3.8-27B-Ultra-Uncensored-*.gguf ./mmproj-*.gguf ./*.stripped.gguf
78
  @rm -rf ./.cache __pycache__ examples/__pycache__
79
  @echo "[+] clean"
 
1
+ # Janus-35B convenience Makefile.
2
  #
3
  # All work is delegated to scripts/* — this file just gives common
4
  # operations short, discoverable names.
 
11
  # MODEL model tag for smoke (default: $(TAG))
12
  #
13
  # Examples:
14
+ # make build # Q4_K_M from the Heretic GGUF repo (the MTP strip no-ops on these quants)
15
  # make build QUANT=Q5_K_M # higher-quality quant
16
+ # make build GGUF_PATH=~/models/Qwen3.6-35B-A3B-uncensored-heretic-Q4_K_M.gguf
17
  # make load-bundle # this repo's bundled GGUF -> local Ollama tag (smudge LFS if needed)
18
  # make smoke
19
  # make check
 
39
  @echo " GGUF_PATH=$(GGUF_PATH)"
40
  endif
41
 
42
+ build: ## Download a Heretic GGUF, run the MTP strip (a no-op on these quants), and run 'ollama create'.
43
  GGUF_PATH=$(GGUF_PATH) TAG=$(TAG) ./scripts/build.sh $(QUANT)
44
 
45
  load-bundle: ## Load THIS repo's bundled GGUF into a local Ollama tag (smudge LFS + ollama create).
 
60
  mmproj: ## Fetch the vision projector for llama.cpp (Ollama vision is broken upstream).
61
  ./scripts/fetch_vision.sh $(PRECISION)
62
 
63
+ check: ## Lint, bridge-file sync, the Go-template guard, and that the Modelfile's FROM is the GGUF this repo ships.
64
  ./scripts/check.sh
65
 
66
  live-check: ## Live end-to-end checks in an isolated Ollama (GGUF=path or GGUF_PATH=path; tens of minutes on CPU).
 
74
 
75
  clean: ## Remove local GGUF copies and ephemeral caches in this repo.
76
  @echo "[*] removing local GGUFs and ephemeral caches in $$PWD"
77
+ @rm -f ./Qwen3.6-35B-A3B-uncensored-*.gguf ./mmproj-*.gguf ./*.stripped.gguf
78
  @rm -rf ./.cache __pycache__ examples/__pycache__
79
  @echo "[+] clean"
Modelfile CHANGED
@@ -1,23 +1,25 @@
1
- FROM ./Janus-27B.Q4_K_M.gguf
2
 
3
- # The bundled GGUF is MTP-stripped. The base
4
- # (llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved) keeps its
5
- # MTP head, so the source quant carries a NextN block at index 64 that stock
6
- # llama.cpp and Ollama cannot load — scripts/strip_mtp.py drops it and takes
7
- # block_count back to 64. That strip is a required build step here, not a safety
8
- # net. If you want the MTP head, run the upstream safetensors under vLLM/SGLang,
9
- # or pull the unstripped GGUF from
10
- # llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-GGUF.
 
 
11
 
12
- # Chat template — Qwen 3.8 ChatML in Ollama Go-template form, with the
13
  # tool-calling blocks Ollama's capability detector looks for. Without a
14
  # TEMPLATE that references .Tools and .ToolCalls, an Ollama that uses this
15
  # template anyway (OLLAMA_GO_TEMPLATE=1 - per Ollama's source, not tested live -
16
  # or one that predates the template comparison described below) rejects any
17
  # request carrying a `tools` array with `<model> does not support tools`;
18
- # Ollama 0.33.3 would instead switch to the GGUF's embedded template. Same template as the sibling
19
- # (FoolDev/Thanatos-27B-HERETIC) — both are dense Qwen 3.8 27B and share the
20
- # same chat format.
21
  #
22
  # Thinking IS replayed across turns: every assistant message that carries
23
  # reasoning renders <think>...</think>, earlier turns included, where Qwen's
@@ -25,9 +27,10 @@ FROM ./Janus-27B.Q4_K_M.gguf
25
  # The client must send the reasoning back - per Ollama's source, /api/chat reads
26
  # each assistant message's `thinking` field and /v1/chat/completions reads only
27
  # `reasoning` (it drops `reasoning_content` and `thinking` silently). Every
28
- # retained trace stays in the prompt - measured at 181 prompt tokens without /
29
- # 464 with on a 3-message conversation - and prefill grows to match, which is
30
- # the slow part on a CPU-only host.
 
31
  #
32
  # The thinking block in the assistant branch is load-bearing for a second
33
  # reason. Ollama infers a Go template's "thinking" capability from a `.Thinking` field inside
@@ -35,7 +38,8 @@ FROM ./Janus-27B.Q4_K_M.gguf
35
  # the selection code is unchanged through 0.34.2, checked in source)
36
  # compare that capability list with the GGUF's embedded Jinja template at load
37
  # time, switching to the embedded template if that one advertises more. Delete
38
- # the block and Ollama switches: in one test on this model a single tool call then came back twice
 
39
  # (a separate raw generation showed the model drafting the call inside <think>
40
  # first; that the parser matched such a draft is inferred, not confirmed). Per Ollama's source (not tested live), a
41
  # setup forcing this template (OLLAMA_GO_TEMPLATE=1) would also reject thinking
@@ -45,9 +49,12 @@ FROM ./Janus-27B.Q4_K_M.gguf
45
  # $lastUserIdx loop at the top of the template is kept for it; do not delete the
46
  # block.
47
  #
48
- # Reasoning effort: the block at the top of the template adds the instruction
49
- # chat_template.jinja adds, keyed on Ollama's think level ($.ThinkLevel), not the
50
- # request value. Ollama folds `reasoning_effort` into four levels first:
 
 
 
51
  # high/xhigh/max/ultra -> high or max (xhigh line), low/minimal -> low (low
52
  # line), medium and unset -> medium (no line), none -> thinking off. The
53
  # outer `if $.ThinkLevel` keeps the template rendering on Ollama builds that
@@ -150,47 +157,49 @@ Behavior rules:
150
 
151
  # Hardware notes
152
  # --------------
153
- # This Q4_K_M is ~17 GB on disk. Measured on Ollama 0.33.3, CPU backend
154
- # (2026-09-14; llama.cpp's own log lines):
155
- # weights 16.1 GiB (6.3 GiB mapped + 9.8 GiB repacked for CPU)
156
- # recurrent state 0.15 GiB (48 linear-attention layers, fixed size)
157
- # KV cache 16 of 64 layers, 4 KV heads x 256 dims: exactly
158
- # 2 GiB per 32K tokens at f16 (Ollama's default),
159
- # 1.06 GiB per 32K at q8_0 -> at 262144: 16 GiB f16,
160
- # 8.5 GiB q8_0; at 1010000: 61.6 / 32.7 GiB
161
- # compute buffer 0.3-0.8 GiB measured at 8K-65K with flash attention
162
- # (Ollama's default is auto, which enabled it); ~1-1.5 GiB
163
- # projected at 262144 by llama.cpp's llama-fit-params.
164
- # Forcing flash attention off grows it to ~25 GiB there.
165
- # total ~34 GiB at the 262144 default with the f16 cache,
166
- # ~26 GiB with OLLAMA_KV_CACHE_TYPE=q8_0 and
167
- # OLLAMA_FLASH_ATTENTION=1; at num_ctx 65536 ~21 GiB f16 /
168
- # 19.2 GiB q8_0 (measured). 1010000 needs ~80 / ~50 GiB.
 
 
 
 
169
  # No YaRN rope-scaling is baked in this GGUF, so output
170
  # past ~262K degrades; treat 1.01M as an advertised
171
  # ceiling.
172
  #
173
- # This is a dense 27B across 64 layers — every parameter runs on every token.
174
- # There is no routing to offload selectively and no active-parameter discount:
175
- # budget for the whole ~16 GiB of weights in the hot path.
 
 
176
  #
177
- # Working configurations (rows assume the 262144 default: ~34 GiB with the f16
178
- # cache, ~26 GiB with q8_0; the 1.01M ceiling fits only 64 GB+ hosts with q8_0):
179
  # ✓ Single H100 80GB / A100 80GB — full GPU offload
180
  # ✓ RTX 5090 32GB / RTX 4090 24GB + 32GB RAM — partial offload
181
  # ✓ Mac Studio M2/M3 Ultra 64GB+ — unified memory
182
- # ✓ Linux box with 48GB+ RAM (CPU-only) — CPU inference (~34 GiB at the default)
183
  # ⚠ 32GB hosts (e.g. ASUS ROG Flow Z13) — set OLLAMA_KV_CACHE_TYPE=q8_0
184
- # (~26 GiB, tight) or lower num_ctx
185
- # (65536 is ~19-21 GiB)
186
  #
187
- # Throughput, measured 2026-09-11 on CPU only (Ryzen AI Max+ 395, Ollama 0.33.3
188
- # CPU backend, OLLAMA_FLASH_ATTENTION=1, OLLAMA_KV_CACHE_TYPE=q8_0, bench.sh's
189
- # num_ctx 4096 / num_batch 256): Q4_K_M -> 4.97 tok/s aggregate (1802 tokens /
190
- # 362.3 s; 5.18 / 5.03 / 4.96 short/medium/long). Partial GPU offload (Vulkan,
191
- # LLAMA_ARG_NO_HOST=1, num_ctx 4096): 5.7 tok/s at num_gpu 24 - see the README.
192
- # The measurements that used to sit here were taken on this repo's previous base
193
- # and do not transfer to a dense 27B, so they are removed rather than restated.
194
  #
195
  # To run on a 32 GB unified-memory laptop, override these in your local
196
  # Modelfile copy (or via `/set parameter` in the interactive `ollama run` REPL):
@@ -198,4 +207,4 @@ Behavior rules:
198
  # PARAMETER num_batch 256
199
  #
200
  # If you have ≥48 GB RAM but want partial GPU offload, set:
201
- # PARAMETER num_gpu 40 # offload most layers (model has 64)
 
1
+ FROM ./Janus-35B-A3B.Q4_K_M.gguf
2
 
3
+ # The bundled GGUF carries no MTP / NextN block. The base
4
+ # (llmfan46/Qwen3.6-35B-A3B-uncensored-heretic) publishes its GGUFs already
5
+ # MTP-clean — blk.0…blk.39 only, block_count 40, no nextn_predict_layers key
6
+ # (read from the bundled file's own header, 2026-09-18) — so scripts/strip_mtp.py
7
+ # is a defensive no-op here, not a required build step. The separate
8
+ # llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-GGUF variant
9
+ # does keep the MTP head (its Q4_K_M has block_count 41, NextN block at index
10
+ # 40) and would need the strip; this repo does not ship it. If you want the MTP
11
+ # head, run the upstream safetensors under vLLM/SGLang, or take that variant's
12
+ # GGUF.
13
 
14
+ # Chat template — Qwen 3.6 ChatML in Ollama Go-template form, with the
15
  # tool-calling blocks Ollama's capability detector looks for. Without a
16
  # TEMPLATE that references .Tools and .ToolCalls, an Ollama that uses this
17
  # template anyway (OLLAMA_GO_TEMPLATE=1 - per Ollama's source, not tested live -
18
  # or one that predates the template comparison described below) rejects any
19
  # request carrying a `tools` array with `<model> does not support tools`;
20
+ # Ollama 0.33.3 would instead switch to the GGUF's embedded template. Same template as the dense
21
+ # 27B sibling (FoolDev/Thanatos-27B-HERETIC) — the ChatML wire format is
22
+ # identical across Qwen 3.6 and 3.8.
23
  #
24
  # Thinking IS replayed across turns: every assistant message that carries
25
  # reasoning renders <think>...</think>, earlier turns included, where Qwen's
 
27
  # The client must send the reasoning back - per Ollama's source, /api/chat reads
28
  # each assistant message's `thinking` field and /v1/chat/completions reads only
29
  # `reasoning` (it drops `reasoning_content` and `thinking` silently). Every
30
+ # retained trace stays in the prompt and prefill grows to match, which is the
31
+ # slow part on a CPU-only host. The prompt-token counts that used to sit here
32
+ # were measured on the dense 27B this repo shipped in the interim; they are
33
+ # removed rather than carried over, pending re-measurement.
34
  #
35
  # The thinking block in the assistant branch is load-bearing for a second
36
  # reason. Ollama infers a Go template's "thinking" capability from a `.Thinking` field inside
 
38
  # the selection code is unchanged through 0.34.2, checked in source)
39
  # compare that capability list with the GGUF's embedded Jinja template at load
40
  # time, switching to the embedded template if that one advertises more. Delete
41
+ # the block and Ollama switches: in one test on the dense 27B this repo shipped
42
+ # in the interim a single tool call then came back twice
43
  # (a separate raw generation showed the model drafting the call inside <think>
44
  # first; that the parser matched such a draft is inferred, not confirmed). Per Ollama's source (not tested live), a
45
  # setup forcing this template (OLLAMA_GO_TEMPLATE=1) would also reject thinking
 
49
  # $lastUserIdx loop at the top of the template is kept for it; do not delete the
50
  # block.
51
  #
52
+ # Reasoning effort: the block at the top of the template injects an effort
53
+ # instruction keyed on Ollama's think level ($.ThinkLevel), not the request
54
+ # value. It is Ollama-only now — this repo no longer ships chat_template.jinja,
55
+ # and the Qwen 3.6 base's own embedded template, which governs the llama.cpp
56
+ # path from here on, has no reasoning_effort handling at all.
57
+ # Ollama folds `reasoning_effort` into four levels first:
58
  # high/xhigh/max/ultra -> high or max (xhigh line), low/minimal -> low (low
59
  # line), medium and unset -> medium (no line), none -> thinking off. The
60
  # outer `if $.ThinkLevel` keeps the template rendering on Ollama builds that
 
157
 
158
  # Hardware notes
159
  # --------------
160
+ # This Q4_K_M is 21,233,608,512 bytes on disk — 21.23 GB decimal, 19.78 GiB.
161
+ # No runtime memory figures for this build yet: the ones that used to sit here
162
+ # were measured on the dense 27B this repo shipped in the interim and do not
163
+ # transfer to a 35B-A3B MoE, so they are removed rather than rescaled. What
164
+ # follows is derived from the bundled GGUF's own header (read 2026-09-18),
165
+ # not measured on a running model:
166
+ # weights 19.78 GiB on disk
167
+ # KV cache only 10 of the 40 layers are full-attention (indices
168
+ # 3, 7, ... 39); 10 layers x 2 KV heads x (256 + 256)
169
+ # dims x 2 bytes = 20,480 B/token, i.e. 0.625 GiB per 32K
170
+ # at f16 (Ollama's default) and about 0.33 GiB per 32K at
171
+ # q8_0 -> ~5.0 GiB at the 262144 default with f16,
172
+ # ~2.66 GiB with q8_0
173
+ # recurrent state fixed-size, for the 30 Gated-DeltaNet layers; not
174
+ # measured on this build
175
+ # compute buffer not measured on this build
176
+ # floor ~24.8 GiB at the 262144 default with the f16 cache
177
+ # (19.78 weights + 5.0 KV), ~22.4 GiB with
178
+ # OLLAMA_KV_CACHE_TYPE=q8_0 — a floor, before the
179
+ # recurrent state and compute buffer.
180
  # No YaRN rope-scaling is baked in this GGUF, so output
181
  # past ~262K degrades; treat 1.01M as an advertised
182
  # ceiling.
183
  #
184
+ # This is a 35B-A3B MoE across 40 layers: 256 experts, 8 routed per token plus
185
+ # one shared expert, ~34.7B parameters total and ~3B active per token (derived
186
+ # from the same header). The active count cuts compute per token, not the memory
187
+ # floor — every expert has to be resident, so budget for the whole 19.78 GiB of
188
+ # weights.
189
  #
190
+ # Working configurations (rows assume the 262144 default: a ~24.8 GiB floor with
191
+ # the f16 cache, ~22.4 GiB with q8_0):
192
  # ✓ Single H100 80GB / A100 80GB — full GPU offload
193
  # ✓ RTX 5090 32GB / RTX 4090 24GB + 32GB RAM — partial offload
194
  # ✓ Mac Studio M2/M3 Ultra 64GB+ — unified memory
195
+ # ✓ Linux box with 48GB+ RAM (CPU-only) — CPU inference
196
  # ⚠ 32GB hosts (e.g. ASUS ROG Flow Z13) — set OLLAMA_KV_CACHE_TYPE=q8_0
197
+ # or lower num_ctx
 
198
  #
199
+ # Throughput: no tok/s figures for this build yet. The measurements that used to
200
+ # sit here were taken on the dense 27B and do not transfer to an MoE with ~3B
201
+ # active parameters per token, so they are removed rather than rescaled —
202
+ # pending re-measurement.
 
 
 
203
  #
204
  # To run on a 32 GB unified-memory laptop, override these in your local
205
  # Modelfile copy (or via `/set parameter` in the interactive `ollama run` REPL):
 
207
  # PARAMETER num_batch 256
208
  #
209
  # If you have ≥48 GB RAM but want partial GPU offload, set:
210
+ # PARAMETER num_gpu 24 # offload most layers (model has 40)
README.md CHANGED
@@ -1,7 +1,7 @@
1
  ---
2
  license: apache-2.0
3
  base_model:
4
- - llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved
5
  base_model_relation: quantized
6
  datasets:
7
  - Crownelius/Creative_Writing_ShareGPT_Enhanced
@@ -34,41 +34,44 @@ language:
34
  - fi
35
  - el
36
  tags:
37
- - qwen38
 
38
  - conversational
39
  - multimodal
40
  - agent
41
  - gguf
 
42
  - heretic
43
  - uncensored
44
  - reasoning
45
  - distillation
46
- library_name: transformers
47
- pipeline_tag: image-text-to-text
48
  ---
49
 
50
- <img src="https://huggingface.co/FoolDev/Janus-35B-HERETIC/resolve/main/banner.svg" alt="Janus-27B banner" width="100%" />
51
 
52
  [![License](https://img.shields.io/badge/License-Apache_2.0-7aa2f7?style=flat&labelColor=1a1b26)](https://opensource.org/licenses/Apache-2.0)
53
- [![Base Model](https://img.shields.io/badge/Base-Heretic-bb9af7?style=flat&labelColor=1a1b26)](https://huggingface.co/llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved)
54
- [![Architecture](https://img.shields.io/badge/Arch-Dense_27B-ff9e64?style=flat&labelColor=1a1b26)](#architecture)
55
  [![Quant](https://img.shields.io/badge/GGUF-Q4__K__M-9ece6a?style=flat&labelColor=1a1b26)](#whats-here)
56
  [![Sibling](https://img.shields.io/badge/Sibling-Thanatos--27B-7dcfff?style=flat&labelColor=1a1b26)](https://huggingface.co/FoolDev/Thanatos-27B-HERETIC)
57
 
58
- <a href="https://buymeacoffee.com/cardoffoolm"><img src="https://huggingface.co/FoolDev/Janus-35B-HERETIC/resolve/main/bmc-banner.svg" alt="Buy me a coffee — support Janus-27B" width="320" /></a>
59
 
60
- # Janus-27B
61
 
62
- > **Flagship Reasoning. Dense 27B. Uncensored.**
63
- > *llmfan46's Heretic abliteration of Qwen 3.8 27B, repackaged with Claude Fable 5 in the teacher slot.*
64
 
65
- **`Architecture:`** `Qwen 3.8 27B (dense)` | **`Parameters:`** `27B (all active)` | **`Context:`** `262,144 native / 1,010,000 extensible` | **`Base:`** `Heretic (llmfan46)` | **`Teacher:`** `Claude Fable 5` | **`Type:`** `Distilled + Abliterated Dense LLM`
66
 
67
- A personal fork of [`llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved`](https://huggingface.co/llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved) — an uncensored Heretic-style (MPOA, MTP-preserved) abliteration of [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B), the dense 27B multimodal base — repackaged as Janus-27B with Claude Fable 5 reasoning data in the teacher slot. Refusal-trained behavior is dialed back at the base layer.
68
 
69
- > **Provenance.** This repo repackages and quantizes — **no training is performed
70
- > here.** What ships is [`llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved`](https://huggingface.co/llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved) converted to GGUF and MTP-stripped so stock
71
- > llama.cpp and Ollama can load it. The `Teacher:` field above and the `datasets:`
 
72
  > list in the frontmatter are carried over from earlier revisions of this card.
73
  > **The current base does not claim them** — its card lists no datasets and names
74
  > no teacher — and nothing in this repo reproduces or measures them. One listed
@@ -77,10 +80,11 @@ A personal fork of [`llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Pr
77
  > attribution, not as a training record.
78
  > `base_model_relation` is `quantized` for the same reason.
79
 
80
- > **On the repo name:** the model is **Janus-27B**, but the Hub slug stays
81
- > `FoolDev/Janus-35B-HERETIC` — renaming it would break every existing link and
82
- > `ollama run hf.co/…` pull. Use the URLs exactly as written below. The Ollama
83
- > tag (`janus`) is unchanged too.
 
84
 
85
  ## TL;DR
86
 
@@ -104,11 +108,13 @@ ollama create janus -f Modelfile && ollama run janus
104
 
105
  After either path, `ollama show janus` lists `completion`, `tools`,
106
  and `thinking` under Capabilities. Hardware: the default `num_ctx` is the full
107
- **262,144** native window — ~34 GiB with Ollama's default f16 KV cache, ~26 GiB
108
- with `OLLAMA_KV_CACHE_TYPE=q8_0` — so on a smaller host use the q8_0 cache or
109
- lower it (65536 is ~21 GiB), or raise it on to 1,010,000 with YaRN (not baked
110
- in, so context past ~262K degrades). See
111
- [Hardware requirements](#hardware-requirements).
 
 
112
 
113
  To halve the KV cache, start the Ollama server with the q8_0 cache and flash
114
  attention (for a systemd install, add them as `Environment=` lines with
@@ -122,74 +128,80 @@ OLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_FLASH_ATTENTION=1 ollama serve
122
 
123
  | File | Use |
124
  |---|---|
125
- | `Janus-27B.Q4_K_M.gguf` | Recommended default, ~17 GB. MTP-stripped so stock llama.cpp / Ollama can load it (see [Architecture](#architecture)). |
126
  | `Modelfile` | Ollama wrapper for **local** builds (`ollama create janus -f Modelfile`) — carries the Go template that Ollama runs in preference to the GGUF's embedded one (see [Which template Ollama runs](#which-template-ollama-runs)). |
127
  | `template`, `system`, `params` | Used by HF's Ollama bridge when users `ollama run hf.co/FoolDev/Janus-35B-HERETIC` directly. The bridge does **not** read `Modelfile` (see [HF Ollama docs](https://huggingface.co/docs/hub/en/ollama)); it ingests these three root-level files instead. Kept in sync with the `Modelfile`'s `TEMPLATE` / `SYSTEM` / `PARAMETER` directives. |
128
- | `chat_template.jinja` | Chat template for **llama.cpp / llama-server**, already stamped into the bundled GGUF (pass `--chat-template-file` only for an unstamped quant). The base model's own embedded template with two changes: `preserve_thinking` and `preserve_reasoning` never raise when they disagree (`preserve_thinking` wins), while earlier turns' reasoning is replayed by default, as upstream does; and `reasoning_effort` is normalised instead of validated. Tool calls keep upstream's XML default, the form llama-server's parser reads. Also stamped into the GGUF by `scripts/build.sh`. See [Tool / function calling](#tool--function-calling). |
129
- | `scripts/build.sh` | Pulls a GGUF from `llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-GGUF` (default Q4_K_M), runs it through `strip_mtp.py`, then runs `ollama create janus`. Quants published there: BF16, Q3_K_L, Q3_K_M, Q4_K_M, Q4_K_S, Q5_K_M, Q5_K_S, Q6_K, Q8_0. The bundled Q4_K_M is already this Heretic quant (stripped); use this to build the others locally. |
130
  | `scripts/check_bridge_sync.py` | Run before pushing a `Modelfile` / `template` / `system` / `params` edit to verify the four configurations remain in sync. Exits 0 if in sync, 1 with a per-key diff if not. |
131
- | `scripts/check.sh` | Local lint: `bash -n`, `shellcheck`, `py_compile`, footgun-grep, `Modelfile`-vs-bridge-files sync, the `chat_template.jinja` render checks and the Go `template` guard (`make check`) |
132
- | `scripts/check_chat_template.py` | Renders `chat_template.jinja` with jinja2 and asserts the tool-call, thinking, reasoning-replay (default, opt-outs, conflicting kwargs, live tool chain) and `reasoning_effort` behaviour (check 8 in `check.sh`) |
133
- | `scripts/check_go_template.py` | Guards the Go `template`: Ollama's thinking detection, the condition that replays earlier turns' reasoning, the tool round trip and JSON tool signatures (check 9 in `check.sh`) |
134
  | `scripts/live_check.sh` | Live end-to-end checks in an isolated, CPU-only Ollama (own port and model store; it refuses to run if Ollama reports a GPU): template selection, one tool call, string-argument replay, an earlier turn's reasoning replayed and a live tool chain's kept, every `reasoning_effort` value and the effort lines — 17 checks (`make live-check`) |
135
- | `scripts/verify_arch.py` | Cross-checks the README Architecture bullets (layer count, hidden size, dense FFN width, native context, vocab) plus the underlying forward-pass structure (Gated Attention + Gated DeltaNet head dims, partial RoPE, full-attention interval) against the bundled GGUF's `qwen35` metadata. Run on demand (`python3 scripts/verify_arch.py`); loads the ~17 GB GGUF (LFS smudge required) and exits non-zero on any mismatch. `block_count` is checked against **64** — the MTP-stripped value — so an unstripped bundle reports 65 and fails. |
136
- | `scripts/strip_mtp.py` | Drops the MTP / NextN layer from a `qwen35` GGUF (`python3 scripts/strip_mtp.py IN.gguf OUT.gguf`): removes the extra `blk.<last>.*` block, decrements `block_count`, drops `nextn_predict_layers`. Kept tensors are copied byte-for-byte (no re-quant); a conditional no-op (hardlink) on already-clean quants. **Required, not optional:** the base is MTP-preserved, so quants from the source repo carry a NextN block at index 64 that stock llama.cpp / Ollama cannot load. With `--chat-template FILE` it also restamps the GGUF's embedded chat template in the same pass (see [Tool / function calling](#tool--function-calling)). `build.sh` runs this on every fetched quant, and the bundled Q4_K_M ships already stripped (`block_count` 64). Parity with the dense sibling `FoolDev/Thanatos-27B-HERETIC`. |
137
  | `scripts/smoke_test.sh` | Integration smoke test against a running Ollama daemon: server reachable, model loaded, `tools` capability present, chat round-trip, and no control-token leakage. `TOOLS_TEST=1` adds a tool-call round-trip. Defaults to `MODEL=janus`. |
138
  | `scripts/bench.sh` | Measures tok/s from Ollama's `eval_count` / `eval_duration` over a short/medium/long prompt mix (with a discarded warmup). Defaults to `MODEL=janus`. |
139
- | `scripts/load_bundle.sh` | Loads the bundled `Janus-27B.Q4_K_M.gguf` into Ollama as a local `janus` tag without an upstream pull (if the working-tree file is still an LFS pointer it downloads the real blob into `.cache/` — it does not smudge in place — then checks the arch is `qwen35`). |
140
  | `scripts/cap_ctx.sh` | Bakes a small-`num_ctx` local `janus` tag (bundled blob + this repo's Modelfile, `num_ctx`→4096, `num_batch` 256) for OpenAI `/v1` clients — which can't override the baked `num_ctx` and OOM on small hosts (see [Inference examples](#inference-examples)). Run `./scripts/cap_ctx.sh` (or `CTX=8192 ...`). |
141
- | `scripts/fetch_vision.sh` | Downloads the vision projector (`Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-mmproj-BF16.gguf`) from the Heretic GGUF repo for llama.cpp image input (Ollama vision is broken upstream — see [Vision](#vision)). |
142
  | `examples/` | Ready-to-run Python clients for Ollama, Transformers, and llama-cpp-python (text, tools, and vision — see `examples/README.md`) |
143
 
144
- GGUF-only release. Pull the Heretic safetensors from [`llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved`](https://huggingface.co/llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved) if you need the `transformers` tree (or the vanilla pre-Heretic base from [`Qwen/Qwen3.8-27B`](https://huggingface.co/Qwen/Qwen3.8-27B)). The safetensors also keep the MTP head that the bundled GGUF has stripped — that's the path to speculative decoding under vLLM / SGLang.
145
 
146
- > ### Bundled blob status: the dense 27B is published
147
  >
148
- > The bundled GGUF is **`Janus-27B.Q4_K_M.gguf`** (~17.3 GB) — the dense Qwen 3.8
149
- > 27B this card describes. `ollama run hf.co/FoolDev/Janus-35B-HERETIC` serves it
150
- > directly, and `ollama create janus -f Modelfile` works from a fresh clone.
 
151
  >
152
- > It is `llmfan46/…-MTP-Preserved-GGUF`'s Q4_K_M
153
- > (sha256 `97298d4b0f99a3f4c46b2dd013abd0ca1628d68be4c3960817b39998a60f0eaf`)
154
- > run through `./scripts/strip_mtp.py`: the NextN block at index 64 is dropped,
155
- > `block_count` 65 → 64, `nextn_predict_layers` removed, and every kept tensor
156
- > copied byte-for-byte — no re-quantization. `chat_template.jinja` is stamped into
157
- > its metadata in the same pass, so llama.cpp and LM Studio's llama.cpp runtime get
158
- > working tool calls with no extra flags (both tested; KoboldCpp and Jan were checked on 2026-09-14, see [Tool / function calling](#tool--function-calling)).
159
  >
160
- > Verified before publishing: all 14 Architecture claims below via
161
- > `python3 scripts/verify_arch.py`; loads in Ollama as `qwen35` / 26.9B dense with
162
- > `tools` and `thinking`; and a full tool-call round-trip on
163
- > `/v1/chat/completions`, including the replay turn whose string `arguments` used
164
- > to fail.
 
165
  >
166
- > The repo previously shipped a 35B-A3B MoE quant, which is where the
167
- > `Janus-35B-HERETIC` slug comes from. That blob is no longer published; it
168
- > remains in git history.
169
 
170
  ## Architecture
171
 
172
- A **dense** hybrid stack: linear-attention (Gated DeltaNet) layers with a full
173
- Gated Attention layer every fourth block. No experts, no router — all 27B
174
- parameters run on every token.
175
 
176
- - Qwen 3.8, 27B dense (no experts, no routing, nothing conditionally skipped)
177
- - 64 layers, 16 × (3 × Gated DeltaNet → MLP / 1 × Gated Attention → MLP) — `full_attention_interval` 4, so 16 full-attention and 48 linear-attention layers
178
- - Hidden size 5120, dense FFN intermediate 17408
179
- - Gated Attention: 24 Q-heads / 4 KV-heads (GQA), head_dim 256, gated output projection, partial RoPE (factor 0.25 — 64 of 256 dims rotated)
180
- - Gated DeltaNet: 48 V-heads / 16 QK-heads, head dim 128, conv kernel 4
 
 
 
 
 
181
  - 262 144 native context (extensible to 1 010 000 with YaRN, but YaRN is not enabled in the bundled GGUF)
182
- - Vision + video supported by upstream (mmproj not included in this release)
183
  - Vocab 248,320
184
- - GGUF arch stamp `qwen35` — the canonical upstream label for the whole Qwen 3.5 / 3.6 / 3.8 family; there is no `qwen38` arch in llama.cpp
 
 
 
185
 
186
- **MTP head (stripped).** The base is *MTP-preserved*: its GGUFs carry a NextN
187
- block at index 64 for vLLM / SGLang speculative decoding, and stock llama.cpp and
188
- Ollama refuse to load it. The bundled `Janus-27B.Q4_K_M.gguf` is therefore
189
- MTP-stripped — block 64 removed, `block_count` 65 → 64 — by
190
- `scripts/strip_mtp.py`, exactly as the dense sibling Thanatos does. If you want
191
- the MTP head, run the upstream safetensors under vLLM / SGLang, or take an
192
- unstripped quant straight from the source GGUF repo.
 
 
193
 
194
  ## Quick start
195
 
@@ -197,14 +209,18 @@ unstripped quant straight from the source GGUF repo.
197
 
198
  Drop the GGUF into your loader of choice. The chat template is embedded in the GGUF metadata, and llama.cpp and LM Studio use it by default, so no `--chat-template` flag is needed (llama.cpp has no `--chat-template auto` mode; it would take "auto" as the template itself).
199
 
200
- **Tool calling works out of the box** — the bundled GGUF carries this repo's `chat_template.jinja` in its metadata, so `--jinja` alone is enough. Tool calls use Qwen's native XML form, the one llama-server's parser reads. (Upstream's template raises on `reasoning_effort: "max"` and on conflicting `preserve_thinking` / `preserve_reasoning`, both HTTP 500s on llama-server. Pass the file explicitly if you are running an unstamped quant:)
201
 
202
  ```bash
203
- llama-server -m Janus-27B.Q4_K_M.gguf --jinja --chat-template-file chat_template.jinja
204
  ```
205
 
206
- A GGUF built by `./scripts/build.sh` has that template baked in and needs no flag.
207
- See [Tool / function calling](#tool--function-calling) for what differs and why.
 
 
 
 
208
 
209
  ### Ollama
210
 
@@ -229,9 +245,10 @@ Once the model is loaded (via `ollama run janus`, `lms server`, or `llama-server
229
  > The examples use `model: "janus"`, the tag from the local build (path B). If you pulled via the TL;DR one-liner instead, use the full tag `hf.co/FoolDev/Janus-35B-HERETIC`, or run `ollama cp hf.co/FoolDev/Janus-35B-HERETIC janus` once to create the short tag.
230
 
231
  > **On memory-tight hosts, cap `num_ctx` first.** `/v1/chat/completions` (OpenAI-compat) has
232
- > no `num_ctx` knob, so it loads at the baked **262,144** default (16 GiB KV / ~34 GiB total with
233
- > Ollama's default f16 cache), which does not fit a 32 GB box; `OLLAMA_KV_CACHE_TYPE=q8_0` on the
234
- > server brings it to ~26 GiB (see [Hardware requirements](#hardware-requirements)). Either
 
235
  > call `/api/chat` with `"options": {"num_ctx": 4096}`, or bake a small-context tag for
236
  > OpenAI clients: `ollama run hf.co/FoolDev/Janus-35B-HERETIC:Q4_K_M`, then
237
  > `/set parameter num_ctx 4096` and `/save janus`, and point clients at `janus` — or in one step, **`./scripts/cap_ctx.sh`** (which bakes exactly that tag).
@@ -321,16 +338,16 @@ Behavior rules:
321
 
322
  ## Vision
323
 
324
- The Qwen 3.8 base supports image (and video) input via a separate
325
  `mmproj` projector. The full multimodal stack is:
326
 
327
  ```
328
- Janus-27B.Q4_K_M.gguf (~17 GB, the text decoder)
329
- Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-mmproj-BF16.gguf (the vision projector)
330
  ```
331
 
332
  The projector and other-quant text decoders live at
333
- [`llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-GGUF`](https://huggingface.co/llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-GGUF)
334
  (BF16 mmproj only). This repo intentionally does not redistribute it;
335
  `./scripts/fetch_vision.sh` pulls the projector into the repo root.
336
 
@@ -338,21 +355,21 @@ The projector and other-quant text decoders live at
338
 
339
  | Loader | Text | Vision (mmproj) | Notes |
340
  |---|---|---|---|
341
- | **llama.cpp** (`llama-mtmd-cli`, `llama-server --mmproj`) | ✅ | ✅ | Reference path. Upstream has the `qwen35` arch entry. |
342
  | **llama-cpp-python** | ✅ | ✅ | See `examples/llama_cpp_vision.py`. |
343
- | **Ollama 0.24+** | ✅ | ❌ | Text inference works: Ollama's Go engine has the `qwen35` arch entry. Vision (mmproj) is still broken: the C++ llama.cpp fallback that Ollama switches to when an mmproj is attached lacks it. `ollama create` accepts a dual-`FROM` (text + mmproj) and `ollama show` reports `vision` capability — but the **first inference request** fails with an `unknown model architecture` load error, and once mmproj is attached this blocks text inference too. See [ollama/ollama#14575](https://github.com/ollama/ollama/issues/14575) (closed as completed on 2026-08-04; #15898 was closed as its duplicate and the sync PR #15899 closed unmerged). Re-test on a current Ollama before assuming vision is still broken — this table has not been re-verified since it closed. |
344
  | **LM Studio** | ✅ | ✅ | Uses upstream llama.cpp directly. |
345
 
346
  ### Vision via llama.cpp
347
 
348
  ```bash
349
  # Fetch the projector first (into the repo root):
350
- ./scripts/fetch_vision.sh # Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-mmproj-BF16.gguf
351
 
352
  # A. HTTP via llama-server (the easiest path):
353
  llama-server \
354
- -m Janus-27B.Q4_K_M.gguf \
355
- --mmproj Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-mmproj-BF16.gguf \
356
  --host 127.0.0.1 --port 8765 -c 8192 -ngl 99
357
  # then POST OpenAI-style chat completions with an image_url content block —
358
  # e.g. {"type":"image_url","image_url":{"url":"data:image/jpeg;base64,..."}}
@@ -363,36 +380,51 @@ llama-server \
363
  # B. CLI via llama-mtmd-cli (one-shot). It's a separate cmake target, so a
364
  # selective build can skip it; a plain `cmake --build build` produces it.
365
  llama-mtmd-cli \
366
- -m Janus-27B.Q4_K_M.gguf \
367
- --mmproj Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-mmproj-BF16.gguf \
368
  --image photo.jpg \
369
  -p "Describe this image."
370
 
371
  # C. Python via llama-cpp-python:
372
  python examples/llama_cpp_vision.py \
373
- --gguf Janus-27B.Q4_K_M.gguf \
374
- --mmproj Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-mmproj-BF16.gguf \
375
  --image /path/to/photo.jpg \
376
  --prompt "What is in this image?"
377
  ```
378
 
379
  The Ollama issue is closed upstream but this table has not been re-verified
380
  since; until you have re-tested, treat Ollama as **text-only** for this model. The bundled Q4_K_M decoder pairs with the projector directly — the
381
- mmproj is family-wide for Qwen 3.8 27B, so no separate text download is
382
  needed for vision.
383
 
384
  ## Hardware requirements
385
 
386
- This is a ~17 GB Q4_K_M GGUF. Measured on Ollama 0.33.3's CPU backend (2026-09-14, from llama.cpp's own allocation log lines), it needs **16.1 GiB** for the weights, **0.15 GiB** of fixed recurrent state for the 48 linear-attention layers, a **0.3-0.8 GiB** compute buffer with flash attention (Ollama's default `auto` enables it; forcing it off grows the buffer to ~25 GiB at 262144), and a KV cache for the 16 full-attention layers of exactly **2 GiB per 32K tokens at f16** — Ollama's default — or **1.06 GiB per 32K at q8_0** (`OLLAMA_KV_CACHE_TYPE=q8_0`, with `OLLAMA_FLASH_ATTENTION=1`).
 
 
 
 
 
 
 
 
 
 
 
 
 
387
 
388
  | `num_ctx` | f16 cache (default) | q8_0 cache |
389
  |---|---|---|
390
- | **262144 (default)** | 16 GiB KV, **~34 GiB** total | 8.5 GiB KV, **~26 GiB** total |
391
- | 65536 | 4 GiB KV, ~21 GiB total | 2.1 GiB KV, 19.2 GiB total (measured) |
392
- | 32768 | 2 GiB KV, ~19 GiB total | 1.1 GiB KV, ~18 GiB total |
393
- | 1010000 (needs YaRN) | 61.6 GiB KV, ~80 GiB total | 32.7 GiB KV, ~50 GiB total |
394
 
395
- KV and recurrent sizes are exact (measured at 8K, 32K and 65K, and linear in `num_ctx`); totals at 262144 and above add llama.cpp master's `llama-fit-params` projection of the compute buffer (~1-1.5 GiB), which ran ~0.35 GiB above what Ollama allocated at 65536. It's **extensible to 1,010,000**, but this GGUF ships **no baked YaRN rope-scaling**, so positions past the 262144 native window use untrained RoPE and output degrades — reaching the 1.01M ceiling needs YaRN (see below); keep real work within ~262K otherwise.
 
 
396
 
397
  **How to override it:** `ollama run` has no `-o` flag, and `OLLAMA_CONTEXT_LENGTH`
398
  only sets a default that the baked `num_ctx` overrides — so set it per-session
@@ -410,20 +442,18 @@ Programmatic callers pass the same via the API `options` field:
410
 
411
  | Hardware | Status |
412
  |---|---|
413
- | ≥48 GB RAM (CPU-only) | Works at the 262144 default (~34 GiB); a 32 GB host needs `OLLAMA_KV_CACHE_TYPE=q8_0` (~26 GiB, tight) or `num_ctx 65536` (~21 GiB) |
414
  | Single H100/A100 80 GB | Works, full offload |
415
  | RTX 4090 24 GB / 5090 32 GB + 32 GB RAM | Works, partial offload |
416
  | Mac Studio M2/M3 Ultra 64 GB+ unified | Works |
417
- | 32 GB unified-memory laptops (Ryzen AI Max+, Apple M-series) | Does not fit the 262144 default with the f16 cache; the q8_0 cache (~26 GiB) is tight, `num_ctx 65536` (~19-21 GiB) is comfortable |
418
 
419
- **Throughput, measured on CPU only: ~5.0 tok/s.** On a Ryzen AI Max+ 395 with
420
- Ollama 0.33.3's CPU backend, `./scripts/bench.sh` measured **4.97 tok/s** at Q4_K_M
421
- (3-prompt mix, flash attention, q8_0 KV cache), and llama-server generated at about
422
- the same rate on the same CPU. Partial GPU offload is covered below.
423
- The tok/s figures this table used to carry were taken before the 27B rebase and say
424
- nothing about this model, so they've been removed rather than carried over; the
425
- rows above are fit estimates from the memory math. `./scripts/bench.sh` measures
426
- tok/s on your own host.
427
 
428
  **GPU offload on a Strix Halo iGPU (Radeon 8060S): a small gain, and a trap.** On a
429
  Vulkan integrated GPU, llama.cpp - including the build Ollama 0.33.3 bundles - turns
@@ -431,20 +461,18 @@ mmap off and puts every layer it does *not* offload into GPU-reachable host memo
431
  the same pool as the GPU, so `num_gpu` alone does not limit GPU memory: a first attempt
432
  with llama-server filled the pool and took the desktop down with it. With
433
  `LLAMA_ARG_NO_HOST=1` in the Ollama server's environment (llama.cpp's `--no-host`) those
434
- layers stay in ordinary RAM. Measured that way on this model (a private Ollama 0.33.3
435
- with the `ollama-vulkan` backend and `OLLAMA_IGPU_ENABLE=1`; flash attention, q8_0 KV
436
- cache, `num_ctx` 4096, a short generation): 4.8 tok/s at `num_gpu` 4, 4.9 at 8, 5.3 at 16
437
- and 5.7 at 24 (7.9 GiB of the GPU pool), against about 5.0 on CPU only; prompt processing
438
- roughly doubled (61 against 30 tok/s). The step to 32 layers ran the 27 GiB host short of
439
- RAM and was stopped: the GPU's GTT is system RAM, so offloading frees none. On a machine
440
- like this, CPU-only is nearly as fast; if you try the GPU, set `LLAMA_ARG_NO_HOST=1` and a
441
- fixed `num_gpu`. The mechanism is reported upstream in
442
  [ggml-org/llama.cpp#27360](https://github.com/ggml-org/llama.cpp/issues/27360).
443
 
444
  **Reaching a coherent ~1.01M context (opt-in YaRN).** The bundled GGUF ships no YaRN rope-scaling, so raising `num_ctx` toward the 1.01M ceiling degrades past the 262144 native window (see above). Ollama has no rope knob, so for a genuinely coherent long context run the GGUF under llama.cpp with YaRN enabled:
445
 
446
  ```bash
447
- llama-server -m Janus-27B.Q4_K_M.gguf \
448
  --rope-scaling yarn --yarn-orig-ctx 262144 --rope-scale 3.853 -c 1010000
449
  ```
450
 
@@ -452,7 +480,7 @@ llama-server -m Janus-27B.Q4_K_M.gguf \
452
 
453
  ## Chat template
454
 
455
- The model uses the standard Qwen 3.x ChatML format with `<|im_start|>` / `<|im_end|>` role markers. The template is embedded in the GGUF metadata for plain conversation use, but on Ollama the Go `template` / `Modelfile` TEMPLATE is what runs: Ollama picks between it and the embedded template at load time (see [Which template Ollama runs](#which-template-ollama-runs)).
456
 
457
  ### Plain conversation
458
 
@@ -499,42 +527,40 @@ each endpoint reads one field:
499
  | Ollama `/v1/chat/completions` | `reasoning` only — `reasoning_content` and `thinking` are dropped silently, and a message whose `content` is an array of parts keeps its reasoning only if it also has `tool_calls` |
500
  | llama-server `/v1/chat/completions` | `reasoning_content` |
501
 
502
- Checked live on 2026-09-13 with the restamped blob, CPU only: Ollama 0.33.3
503
- replayed `thinking` on `/api/chat` (232 prompt tokens with it, 187 without) and
504
- `reasoning` on `/v1` (215 vs 187) and ignored `reasoning_content` on `/v1` (187);
505
- llama-server master rendered `reasoning_content` and dropped a `reasoning` field.
 
 
 
506
  That Ollama's `/v1` also drops `thinking`, and the array-content rule, come from
507
  Ollama's v0.33.3 and v0.34.0 source, and still hold on the current release:
508
  `openai/openai.go` and `server/prompt.go` are byte-identical from v0.33.3 through
509
  v0.34.2 (source, 2026-09-17). Ollama never passes reasoning to the GGUF's
510
  embedded Jinja template, so on Ollama only the Go template decides what is replayed.
511
- On llama.cpp, while replay is on, an earlier assistant turn with no reasoning still
512
- gets an empty `<think>\n\n</think>` block, as upstream renders it.
513
-
514
- **The cost is context.** Every retained trace stays in the prompt, measured at
515
- **181 prompt tokens without / 464 with** on a 3-message conversation whose first
516
- assistant turn carried a ~240-word trace. Prefill grows with every retained trace,
 
 
517
  and on a CPU-only host prefill is the slow part. Releases 0.3.0 through 0.3.2
518
  turned replay off for that reason; the CHANGELOG has the history.
519
 
520
- **Tested on llama.cpp** on 2026-09-13 with a 260K-parameter test model and
521
- `--chat-template-file` (the template, not the model, decides the prompt), via
522
- `/apply-template` and `/v1/chat/completions`, on master `4a89937` (b10941), b10760
523
- (the llama-server Ollama 0.33.3 bundles) and b10519 (LM Studio's avx2 CPU runtime
524
- 2.30.0): with default flags an earlier turn's reasoning was replayed; with
525
- `--no-reasoning-preserve`, or a per-request `preserve_thinking: false` or
526
- `preserve_reasoning: false`, it was not; `preserve_thinking: true` replayed it even
527
- against `--no-reasoning-preserve`; a live tool-call chain kept its reasoning in every
528
- mode, and no request returned an error. Master (b10763 and later) passes
529
- `preserve_reasoning=true` unless the server runs with `--no-reasoning-preserve`, and
530
- logs "chat template supports preserving reasoning, it is enabled by default"; b10760
531
- and b10519 pass nothing, log "consider enabling it via --reasoning-preserve", and
532
- replay anyway, because the template's own default is on. On the restamped blob
533
- itself, with its embedded template and default flags, llama-server master rendered a
534
- 3-message conversation in 69 prompt tokens with the earlier turn's reasoning and 39
535
- with `--no-reasoning-preserve` or either `false` kwarg (43 with no reasoning sent,
536
- for the empty think block), and a real tool-result turn with string arguments
537
- returned HTTP 200.
538
 
539
  **Which llama.cpp an Ollama carries.** 0.33.3 and 0.34.0 bundle b10760; 0.34.1
540
  bundles b10864 and 0.34.2 b10969 (each release's `LLAMA_CPP_VERSION`). From 0.34.1
@@ -544,19 +570,19 @@ for this model runs Ollama's own Go engine, and the bundled llama.cpp comes into
544
  play only when Ollama falls back to its C++ engine, as it does once an mmproj is
545
  attached (see [Loader compatibility](#loader-compatibility)).
546
 
547
- **Conflicting settings do not fail the request.** A request's `preserve_thinking`
548
- replaces the copy llama-server derives from `preserve_reasoning`, so the template can
549
- see the two disagree. Upstream's template raises on that, and llama-server returns the
550
- raise as HTTP 500 — verified on master for a `preserve_thinking: false` request under
551
- default flags, and on all three builds for `preserve_thinking: true` against
552
- `--no-reasoning-preserve`. This template lets `preserve_thinking` win and never raises.
553
 
554
  **To turn replay off:**
555
 
556
  - **llama-server** — start it with `--no-reasoning-preserve`, or send
557
- `"chat_template_kwargs": {"preserve_thinking": false}` (or
558
- `{"preserve_reasoning": false}`) on a request. A request's
559
  `preserve_thinking: true` switches it back on for that request.
 
 
560
  - **Ollama** — edit the template. In the assistant branch of both the `Modelfile`
561
  and `template` (so `make check`'s bridge sync stays green), change `{{ if (and $.IsThinkSet .Thinking) -}}` back to Qwen's stock
562
  `{{ if (and $.IsThinkSet (and .Thinking (or $last (gt $i $lastUserIdx)))) -}}`
@@ -575,59 +601,67 @@ ollama run hf.co/FoolDev/Janus-35B-HERETIC:Q4_K_M --think=false
575
 
576
  or send `"think": false` on `/api/chat`. With thinking off the model skips the reasoning trace and answers straight into `content`; with it on (the default) reasoning is emitted into the `thinking` field.
577
 
578
- ### Reasoning effort
 
 
 
 
579
 
580
- OpenAI-compatible clients can send `reasoning_effort`. Ollama's `/v1` layer accepts
581
- eight values. Where `chat_template.jinja` sees the value as sent (llama.cpp), it maps
582
- each to one of three tiers or switches thinking off:
583
 
584
- | `reasoning_effort` | Effect |
585
- |---|---|
586
- | `high`, `xhigh`, `max`, `ultra` | **xhigh** — the model is told to think carefully and check its assumptions |
587
- | `medium`, unset, `null` | **medium** — the default; no effort instruction is added |
588
- | `low`, `minimal` | **low** — the model is told to keep its thinking brief |
589
- | `none` | thinking **off** for that request, the same as `"think": false` |
590
-
591
- Anything unrecognised falls back to the medium default rather than failing the
592
- request. The two siblings differ here, each following its base model's template:
593
- an unset or unrecognised value is **medium** on Janus and **xhigh** on Thanatos. Upstream's template raised on unfamiliar values, so `reasoning_effort: "max"` broke every request. On Ollama, while it ran the embedded template, that surfaced as
594
- HTTP 500.
595
-
596
- **On Ollama's default path the Go template runs instead** (see
597
- [Which template Ollama runs](#which-template-ollama-runs)), and it adds the same
598
- instruction from Ollama's think level. Ollama folds the eight values into four
599
- levels before any template sees them, so the Go template maps levels, not values:
600
- `high`, `xhigh`, `max` and `ultra` get the xhigh line; `low` and `minimal` get the
601
- low line; `medium` and an unset value get no line; `none` switches thinking off. `"think": "high"`,
602
- `"low"` and `"max"` on `/api/chat` map the same way. Mixed case such as `"HIGH"`
603
- is rejected by Ollama itself with HTTP 400 before any template runs. Verified on
604
- Ollama 0.33.3 and 0.34.0 with `scripts/live_check.sh`, which sends `/api/chat` think levels: on a one-line chat, `high` and `max`
605
- each added 38 prompt tokens and `low` 26, while `medium` and an unset value added
606
- none. The `/v1` value-to-level mapping above comes from Ollama's source, unchanged
607
- from v0.33.3 through v0.34.2 (2026-09-17).
608
-
609
- On llama.cpp every value except `none` reaches `chat_template.jinja` as sent;
610
- llama-server handles `none` itself by switching thinking off. Tested on llama.cpp
611
- master `a2878d3`: all eight values, `"HIGH"` and an unknown word each
612
- returned HTTP 200, and the rendered prompt carried the tier in the table. The
613
- template lower-cases and trims the value itself. LM Studio's CPU engine (llama.cpp b10519) passed the same checks; the LM Studio app's own API layer is untested.
 
 
 
614
 
615
  ### Which template Ollama runs
616
 
617
- This repo ships two templates for the same ChatML model: the Go `template` (served
618
- by HF's Ollama bridge) / `Modelfile` TEMPLATE, and `chat_template.jinja`, stamped
619
- into the GGUF. They do not render identical prompts — the Go template pins JSON
620
- tool calls and takes the reasoning-effort line from Ollama's think level, while the
621
- Jinja template has its own tool instructions, uses the XML `<function=…>` form
622
- unless a replayed call carries string arguments, and takes the line from the
623
- request's `reasoning_effort`.
624
- Ollama 0.33.3 and 0.34.0 (both verified) pick one at load time: they prefer the
625
- embedded template when that advertises more capabilities — tools, thinking — and
626
- keep the Go template when they tie. They tie, so **Ollama runs the Go template**;
627
- llama.cpp and LM Studio run the embedded Jinja template. That selection code is
628
- unchanged through v0.34.2 (source, 2026-09-17), and Ollama's built-in renderers —
629
- it ships a `qwen35` one — displace both templates only when the model's own config
630
- names a renderer or parser, which this repo's does not. Ollama logs the choice:
 
 
 
631
 
632
  ```text
633
  msg="template selection" ... selected=go_template go_template="[completion tools thinking]"
@@ -635,17 +669,19 @@ msg="template selection" ... selected=go_template go_template="[completion tools
635
 
636
  The Go template's "thinking" entry comes from its single `.Thinking` reference —
637
  the block that renders each assistant turn's reasoning. Removing it drops that
638
- capability: Ollama switches to the embedded template, a single tool call can
639
- come back twice (seen once; cause inferred, not confirmed — see
640
- [Known limitations](#known-limitations)), and `reasoning_effort` suddenly selects
641
- the Jinja template's tiers — under upstream's version of that template it raised
642
- instead, which is what caused the HTTP 500s. That happened once; the CHANGELOG has
643
- the details. Setting `OLLAMA_GO_TEMPLATE=1` or `=0` on the Ollama server forces one
644
- template or the other for every model.
645
 
646
  ### Tool / function calling
647
 
648
- The model emits whichever format the system prompt specifies. The Ollama path pins JSON. On llama.cpp the template defaults to XML, the model's native form and the only one llama-server's tool-call parser reads; `tool_call_format: "auto"` is an opt-in for loaders that pass arguments through as strings (below).
 
 
649
 
650
  **Ollama path** (this repo's `Modelfile`). The TEMPLATE advertises tools inside `<tools>…</tools>` and asks the model to reply in JSON-in-XML — the form Ollama's tool-call extractor parses into a structured `tool_calls` array on `/api/chat` and `/v1/chat/completions`. The tool definitions go in with `{{ json .Function }}`: on Ollama 0.14 and later a bare `{{ .Function }}` prints Go struct syntax instead of JSON ([ollama/ollama#14601](https://github.com/ollama/ollama/issues/14601); fix proposed in [ollama/ollama#18391](https://github.com/ollama/ollama/pull/18391)). The model replies like this:
651
 
@@ -655,7 +691,7 @@ The model emits whichever format the system prompt specifies. The Ollama path pi
655
  </tool_call>
656
  ```
657
 
658
- **Embedded-jinja path** (llama.cpp, llama-cpp-python, LM Studio). The bundled GGUF already embeds `chat_template.jinja`, so this path works as-is. Like the upstream Qwen 3.8 template, it defaults to the XML form:
659
 
660
  ```text
661
  <tool_call>
@@ -667,56 +703,11 @@ Tokyo
667
  </tool_call>
668
  ```
669
 
670
- **Keep XML on llama-server.** llama-server recognises this template as Qwen3-Coder-style and parses the model's reply with a parser — and, once the model opens `<tool_call>`, a grammar — that accept only `<function=…><parameter=…>`. A prompt that teaches the JSON form gets replies that parser cannot read. 0.3.0 to 0.4.0 defaulted to `auto`; 0.5.0 went back to `xml` for that reason.
671
-
672
- **Replayed tool calls render on llama-server.** Every OpenAI-compatible client hands assistant tool calls back with `function.arguments` as a JSON **string**, and the XML parameter form needs the individual key/value pairs — a chat template has `tojson` but no inverse. llama-server decodes the string into a mapping before it renders the template, so the second turn of a tool call renders in XML with no error. Tested on 2026-09-14 with a 260K-parameter test model and this template, replaying a call whose `arguments` was the string `{"city": "Paris"}`: `/apply-template` and `/v1/chat/completions` returned HTTP 200 with the call rendered as `<parameter=city>` on llama.cpp master `97e4ca7`, b10760 (the llama-server Ollama 0.33.3 bundles) and b10519 (LM Studio's avx2 CPU runtime 2.30.0).
673
-
674
- **Other loaders, checked 2026-09-14.** KoboldCpp v1.120 renders the template with Python's jinja2 only when run with `--jinja --jinja_tools` (without `--jinja_tools`, requests with tools skip the template), and parses JSON-string `arguments` with `json.loads` first: with this template and a string-argument replay it returned HTTP 200 and rendered the call as XML (tested with a 260K-parameter model). Jan runs llama.cpp's own server code — v0.8.4 downloads `llama-server` from the janhq/llama.cpp fork (b9967), and its main branch links ggml-org llama.cpp b10809 — so it inherits the conversion above (from source, untested). The LM Studio app renders templates with its own JavaScript Jinja engine rather than llama-server's, and also parses the arguments first: through LM Studio 0.4.24's `/v1/chat/completions` (CPU runtime avx2 2.37.0, a 260K-parameter model carrying this template) a string-argument replay returned HTTP 200 with the call rendered as XML, and a request whose `arguments` were not valid JSON returned HTTP 500 before the template ran, with or without `tool_call_format: "auto"`. If a loader does raise the message below, change `'xml'` to `'auto'` in the template's `_requested_tool_format` default (in LM Studio, under the model's Prompt Template setting).
675
 
676
- A loader that hands the template the raw string instead gets this raise under the default:
677
-
678
- ```text
679
- tool_call_format="xml" requires tool-call arguments to be mappings, but received
680
- a JSON string. Parse the JSON string into a mapping before applying the template,
681
- or set tool_call_format="auto" (or "json") to use the lossless JSON tool-call
682
- representation.
683
- ```
684
 
685
- For those, `tool_call_format` also accepts `auto` — the whole prompt in JSON form whenever a historical tool call carries string arguments, byte-identical to `xml` when every argument is a mapping — and `json`. Pass it with `--chat-template-kwargs` or a request's `chat_template_kwargs`:
686
-
687
- ```bash
688
- llama-server -m Janus-27B.Q4_K_M.gguf --jinja \
689
- --chat-template-kwargs '{"tool_call_format":"auto"}'
690
- ```
691
-
692
- On llama-server `auto` changes nothing, because arguments reach the template as mappings; do not set `json` there, for the parser reason above.
693
-
694
- **`chat_template.jinja` in this repo is the base model's own template with two deliberate changes,** each marked `CHANGED FROM UPSTREAM` in the file: `preserve_thinking` / `preserve_reasoning` never raise — when they disagree `preserve_thinking` wins, where upstream raises and llama-server returns HTTP 500 — while earlier turns' reasoning is replayed by default, as upstream does (see [Thinking is replayed across turns](#thinking-is-replayed-across-turns)); and `reasoning_effort` is normalised instead of validated (see [Reasoning effort](#reasoning-effort)). The tool-call format, `max_tool_arg_chars`, `auto_disable_thinking_with_tools`, the vision handling and the whitespace contract are upstream's. The stock template still raises on `reasoning_effort: "max"`, which llama-server passes to the template, so run an unstamped quant with this file:
695
-
696
- ```bash
697
- llama-server -m Janus-27B.Q4_K_M.gguf --jinja --chat-template-file chat_template.jinja
698
- ```
699
-
700
- With it the same client code should work against either server, as long as both return a parsed `tool_calls` array — verified on Ollama and on standalone llama-server (llama.cpp master `a2878d3` and b10760, under the earlier `auto` default); the prompt format underneath differs (see [Which template Ollama runs](#which-template-ollama-runs)).
701
-
702
- > **The bundled GGUF already carries this template — you do not need the flag.**
703
- > `Janus-27B.Q4_K_M.gguf` has `chat_template.jinja` stamped into its
704
- > metadata, so a loader that reads the *embedded* template and parses Qwen's
705
- > `<tool_call>` output gets working tool calls with no configuration: llama-server
706
- > (tested on llama.cpp master `a2878d3` and b10760) and LM Studio's CPU runtime
707
- > (llama.cpp b10519, tested). llama-cpp-python renders the tools but, per its
708
- > source, does not parse `<tool_call>` output into `tool_calls`. KoboldCpp (with
709
- > `--jinja --jinja_tools`) was tested and Jan checked from source; see above. `./scripts/build.sh` (`make build`) stamps it into every quant it builds,
710
- > too. The `--chat-template-file` flag above is only needed for a GGUF that was
711
- > not built or stamped that way — a quant pulled straight from upstream, for
712
- > instance. To restamp one you already have:
713
- >
714
- > ```bash
715
- > python3 scripts/strip_mtp.py IN.gguf OUT.gguf --chat-template chat_template.jinja
716
- > ```
717
- >
718
- > On an already-MTP-clean input that rewrites the file purely to swap the template;
719
- > tensor data is copied byte-for-byte either way, so there is no re-quantization.
720
 
721
  #### Example (Ollama, OpenAI-compatible API)
722
 
@@ -753,36 +744,36 @@ print(resp.choices[0].message.tool_calls)
753
 
754
  - Use direct prompts ("Call X for Y") rather than soft hints ("Use the tool"). The model thinks before committing to a call, and weak prompts can exhaust `num_predict` inside the `<think>` block before the call is emitted.
755
  - Allow at least `num_predict: 1024` (or `max_tokens: 1024`) for tool-calling turns, more if the schemas are large.
756
- - The JSON-in-XML format is what Ollama's tool-call extractor understands, and the Go template pins it. `chat_template.jinja` renders the XML `<function=…>` form unless a replayed call carries string arguments, so the prompt format does change when you swap loaders. Client code should not have to, as long as the server returns a parsed `tool_calls` array — verified on Ollama and on standalone llama-server (llama.cpp master `a2878d3` and b10760).
757
 
758
  ## Known limitations
759
 
760
- - **On the embedded-template path, a single tool call can come back twice.** In one test Ollama 0.33.3, running the embedded template, returned a first-turn tool call twice. A separate raw generation showed the model drafting the call inside `<think>` before emitting it once after `</think>`, so the parser most likely matched such a draft (inferred, not confirmed). It did not reproduce on standalone llama.cpp master (`a2878d3`, CPU): across both models, 12 first-turn calls (three prompts, with and without `parallel_tool_calls`) and two streamed runs each came back exactly once. Nor on b10760, the llama.cpp build Ollama 0.33.3 bundles and the one the duplicate came through: six more first-turn calls and a streamed run on this model each came back once. LM Studio's CPU engine (llama.cpp b10519) passed the same checks; the LM Studio app's own API layer is untested. Ollama's default path — the Go template, see [Which template Ollama runs](#which-template-ollama-runs) — returned one call in a comparable test; whether the prompt or the parser makes the difference was not isolated. If your client executes tools wherever the embedded template renders (llama.cpp, LM Studio, or Ollama with `OLLAMA_GO_TEMPLATE=0`), drop identical consecutive calls.
761
- - **No mmproj in this release.** The base Qwen3.8 supports image and video input via a separate `mmproj` file, which is not included here. Text-only inference works out of the box; multimodal inference requires fetching `Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-mmproj-BF16.gguf` (or equivalent) from upstream — run `./scripts/fetch_vision.sh` and see [Vision](#vision) for the full path.
762
  - **Quantization-induced quality loss.** Q4_K_M is a strong general-purpose quant but does measurably degrade math and code accuracy compared to BF16. If you need maximum quality, run the upstream safetensors on a host with room for the full BF16 weights.
763
- - **No MTP head on the GGUF path.** The base preserves the multi-token-prediction (NextN) block, but stock llama.cpp and Ollama can't load it, so the bundled quant is MTP-stripped (see [Architecture](#architecture)). No speculative decoding here — use the upstream safetensors under vLLM / SGLang, or an unstripped quant from the source GGUF repo, if you need it.
764
- - **Dense, so the whole model is resident.** There's no expert sparsity to hide behind: all 27B parameters are touched on every token, and the ~16.5 GiB weights-plus-compute footprint is a floor, not an average. Trim `num_ctx` to fit a smaller host — there is no expert-offload trick to fall back on.
765
- - **Thinking traces can loop.** Like most reasoning-distilled models, Janus-27B occasionally gets stuck repeating itself inside `<think>` tags. Mitigations: lower temperature to 0.4-0.6, raise `repeat_penalty` to 1.08, or set a `<think>`-token budget cap if your loader supports it.
766
  - **Large tool-call arguments can be dropped.** Ollama's JSON-in-XML tool format makes the model JSON-escape the entire arguments object inline; for a big/complex payload (e.g. a file's `content` in a `write_file` call) the model can fail to escape it, so the field arrives `undefined` and the call fails. Qwen's native `<function=…><parameter=…>` format (raw values, no escaping) was tested as a fix but parses unreliably through Ollama, so the template deliberately keeps JSON-in-XML. Mitigation: write large files in smaller pieces per call.
767
  - **Uncensored base — not aligned with any specific safety policy.** This is a personal repackage of an open-weight base whose refusal behavior has been abliterated away (the llmfan46 Heretic base). There is no RLHF refusal layer; the model will attempt most requests, so downstream safety is entirely the operator's responsibility.
768
- - **No formal evaluation in this card.** The hardware table is fit estimates from the memory math — the measured throughput figures are 4.97 tok/s on CPU and a short partial-GPU-offload series on the same Ryzen AI Max+ 395, and the pre-rebase numbers were dropped rather than carried over. No task benchmarks are claimed either. If you produce real numbers (tok/s, MMLU, HumanEval, …) and want them included, file a PR.
769
 
770
  ## Related models
771
 
772
  | Model | Size | Notes |
773
  |---|---|---|
774
- | [llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved](https://huggingface.co/llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved) | 27B dense | **Immediate base.** Uncensored Heretic (MPOA-style) abliteration of Qwen 3.8 27B, with the MTP head preserved; `transformers`-native safetensors. |
775
- | [llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-GGUF](https://huggingface.co/llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-GGUF) | 27B dense | Heretic GGUFs — BF16, Q3_K_L, Q3_K_M, Q4_K_M, Q4_K_S, Q5_K_M, Q5_K_S, Q6_K, Q8_0, plus the BF16 mmproj. MTP-preserved, so strip before loading on llama.cpp / Ollama; the bundled Q4_K_M is this quant, already stripped. |
776
- | [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) | 27B dense | Upstream pre-Heretic base model. `transformers`-native multimodal weights. |
777
- | [FoolDev/Thanatos-27B-HERETIC](https://huggingface.co/FoolDev/Thanatos-27B-HERETIC) | 27B dense | Sibling on the same Qwen 3.8 27B architecture, abliterated the other way — the ARA-based [`heretic-org/Qwen3.8-27B-heretic-ara`](https://huggingface.co/heretic-org/Qwen3.8-27B-heretic-ara). Same teacher (Fable 5), same dataset family. (The older `FoolDev/Thanatos-27B` and `Thanatos-27B-Heretic` slugs now 307 to this path.) |
778
- | [Crownelius/Crow-9B-HERETIC-4.6](https://huggingface.co/Crownelius/Crow-9B-HERETIC-4.6) | 9B dense | Heretic-flavored fine-tune on a smaller 9B Qwen base. Useful as a fast first-pass model when 27B is too heavy for the host. |
779
 
780
- > **Janus vs Thanatos.** After this rebase the two are the *same* model on paper: dense Qwen 3.8 27B, 64 layers, the same hybrid Gated-DeltaNet / Gated-Attention stack, the same 262,144 native window, the same Claude Fable 5 teacher slot and dataset family. The only real difference is which abliteration sits underneath — Janus on llmfan46's Heretic / MPOA-style, MTP-preserved base, Thanatos on heretic-org's ARA base; both bundles are MTP-stripped for the same llama.cpp / Ollama reason. Janus is **not** bigger, **not** a mixture-of-experts, and **not** architecturally distinct from its sibling; choose between them on how each abliteration behaves, not on capacity. (This repo used to ship a 35B-A3B MoE build, which is where the `Janus-35B-HERETIC` slug comes from; the model it names is now the dense 27B described above.)
781
 
782
  ## Credits
783
 
784
- - Immediate base: [llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved](https://huggingface.co/llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved) — Heretic-style (MPOA, MTP-preserved) abliteration of Qwen 3.8 27B
785
- - Upstream base: [Qwen/Qwen3.8-27B](https://huggingface.co/Qwen/Qwen3.8-27B) (Alibaba)
786
  - Reasoning teacher: Claude Fable 5 (Anthropic)
787
  - Distillation lineage and dataset curation: [Crownelius](https://huggingface.co/Crownelius)
788
 
 
1
  ---
2
  license: apache-2.0
3
  base_model:
4
+ - llmfan46/Qwen3.6-35B-A3B-uncensored-heretic
5
  base_model_relation: quantized
6
  datasets:
7
  - Crownelius/Creative_Writing_ShareGPT_Enhanced
 
34
  - fi
35
  - el
36
  tags:
37
+ - qwen36
38
+ - moe
39
  - conversational
40
  - multimodal
41
  - agent
42
  - gguf
43
+ - ollama
44
  - heretic
45
  - uncensored
46
  - reasoning
47
  - distillation
48
+ library_name: gguf
49
+ pipeline_tag: text-generation
50
  ---
51
 
52
+ <img src="https://huggingface.co/FoolDev/Janus-35B-HERETIC/resolve/main/banner.svg" alt="Janus-35B banner" width="100%" />
53
 
54
  [![License](https://img.shields.io/badge/License-Apache_2.0-7aa2f7?style=flat&labelColor=1a1b26)](https://opensource.org/licenses/Apache-2.0)
55
+ [![Base Model](https://img.shields.io/badge/Base-Heretic-bb9af7?style=flat&labelColor=1a1b26)](https://huggingface.co/llmfan46/Qwen3.6-35B-A3B-uncensored-heretic)
56
+ [![Architecture](https://img.shields.io/badge/Arch-MoE_35B/3B_active-ff9e64?style=flat&labelColor=1a1b26)](#architecture)
57
  [![Quant](https://img.shields.io/badge/GGUF-Q4__K__M-9ece6a?style=flat&labelColor=1a1b26)](#whats-here)
58
  [![Sibling](https://img.shields.io/badge/Sibling-Thanatos--27B-7dcfff?style=flat&labelColor=1a1b26)](https://huggingface.co/FoolDev/Thanatos-27B-HERETIC)
59
 
60
+ <a href="https://buymeacoffee.com/cardoffoolm"><img src="https://huggingface.co/FoolDev/Janus-35B-HERETIC/resolve/main/bmc-banner.svg" alt="Buy me a coffee — support Janus-35B" width="320" /></a>
61
 
62
+ # Janus-35B
63
 
64
+ > **Flagship Reasoning. Sparse Footprint. Uncensored.**
65
+ > *llmfan46's Heretic abliteration of Qwen 3.6 35B-A3B, repackaged with Claude Fable 5 in the teacher slot.*
66
 
67
+ **`Architecture:`** `Qwen 3.6 35B-A3B (MoE)` | **`Total Params:`** `~34.7B` | **`Active Params:`** `~3B` | **`Context:`** `262,144 native / 1,010,000 extensible` | **`Base:`** `Heretic (llmfan46)` | **`Teacher:`** `Claude Fable 5` | **`Type:`** `Distilled + Abliterated MoE LLM`
68
 
69
+ A personal fork of [`llmfan46/Qwen3.6-35B-A3B-uncensored-heretic`](https://huggingface.co/llmfan46/Qwen3.6-35B-A3B-uncensored-heretic) — an uncensored Heretic-style abliteration of [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B), the 35B-total / 3B-active mixture-of-experts multimodal base — repackaged as Janus-35B with Claude Fable 5 reasoning data in the teacher slot. Refusal-trained behavior is dialed back at the base layer.
70
 
71
+ > **Provenance.** This repo repackages — **no training and no quantization is
72
+ > performed here.** What ships is `llmfan46`'s own Q4_K_M of
73
+ > [`llmfan46/Qwen3.6-35B-A3B-uncensored-heretic`](https://huggingface.co/llmfan46/Qwen3.6-35B-A3B-uncensored-heretic),
74
+ > byte-for-byte, with this repo's Ollama wrapper around it. The `Teacher:` field above and the `datasets:`
75
  > list in the frontmatter are carried over from earlier revisions of this card.
76
  > **The current base does not claim them** — its card lists no datasets and names
77
  > no teacher — and nothing in this repo reproduces or measures them. One listed
 
80
  > attribution, not as a training record.
81
  > `base_model_relation` is `quantized` for the same reason.
82
 
83
+ > **On the repo name:** the model is **Janus-35B** and the Hub slug
84
+ > `FoolDev/Janus-35B-HERETIC` names it correctly again. Between releases 0.2.0 and
85
+ > 0.6.5 this repo shipped a dense Qwen 3.8 27B under the same slug; that blob is no
86
+ > longer published and remains in git history. Every URL and `ollama run hf.co/…`
87
+ > pull below is unchanged, as is the Ollama tag (`janus`).
88
 
89
  ## TL;DR
90
 
 
108
 
109
  After either path, `ollama show janus` lists `completion`, `tools`,
110
  and `thinking` under Capabilities. Hardware: the default `num_ctx` is the full
111
+ **262,144** native window — about **25 GiB** with Ollama's default f16 KV cache
112
+ and about **22.4 GiB** with `OLLAMA_KV_CACHE_TYPE=q8_0`. Those two totals are
113
+ *derived from the GGUF's own geometry, not measured on this blob*, and they do not
114
+ include the recurrent state or compute buffer (see
115
+ [Hardware requirements](#hardware-requirements)). Lower `num_ctx` on a smaller host
116
+ (65536 is about 21 GiB), or raise it to 1,010,000 with YaRN (not baked in, so
117
+ context past ~262K degrades).
118
 
119
  To halve the KV cache, start the Ollama server with the q8_0 cache and flash
120
  attention (for a systemd install, add them as `Environment=` lines with
 
128
 
129
  | File | Use |
130
  |---|---|
131
+ | `Janus-35B-A3B.Q4_K_M.gguf` | Recommended default, 21.23 GB. Upstream's Q4_K_M byte-for-byte — no re-quantization and no MTP strip needed (see [Architecture](#architecture)). |
132
  | `Modelfile` | Ollama wrapper for **local** builds (`ollama create janus -f Modelfile`) — carries the Go template that Ollama runs in preference to the GGUF's embedded one (see [Which template Ollama runs](#which-template-ollama-runs)). |
133
  | `template`, `system`, `params` | Used by HF's Ollama bridge when users `ollama run hf.co/FoolDev/Janus-35B-HERETIC` directly. The bridge does **not** read `Modelfile` (see [HF Ollama docs](https://huggingface.co/docs/hub/en/ollama)); it ingests these three root-level files instead. Kept in sync with the `Modelfile`'s `TEMPLATE` / `SYSTEM` / `PARAMETER` directives. |
134
+ | `scripts/build.sh` | Pulls a GGUF from `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF` (default Q4_K_M), runs it through `strip_mtp.py` (a no-op for these quants, kept as a guard), then runs `ollama create janus`. Quants published there: BF16, Q3_K_L, Q3_K_M, Q4_K_M, Q4_K_S, Q5_K_M, Q5_K_S, Q6_K, Q8_0. The bundled Q4_K_M *is* that repo's Q4_K_M; use this to build the others locally. |
 
135
  | `scripts/check_bridge_sync.py` | Run before pushing a `Modelfile` / `template` / `system` / `params` edit to verify the four configurations remain in sync. Exits 0 if in sync, 1 with a per-key diff if not. |
136
+ | `scripts/check.sh` | Local lint: `bash -n`, `shellcheck`, `py_compile`, footgun-grep, `Modelfile`-vs-bridge-files sync and the Go `template` guard (`make check`) |
137
+ | `scripts/check_go_template.py` | Guards the Go `template`: Ollama's thinking detection, the condition that replays earlier turns' reasoning, the tool round trip and JSON tool signatures (check 8 in `check.sh`) |
 
138
  | `scripts/live_check.sh` | Live end-to-end checks in an isolated, CPU-only Ollama (own port and model store; it refuses to run if Ollama reports a GPU): template selection, one tool call, string-argument replay, an earlier turn's reasoning replayed and a live tool chain's kept, every `reasoning_effort` value and the effort lines — 17 checks (`make live-check`) |
139
+ | `scripts/verify_arch.py` | Cross-checks the README Architecture bullets (layer count, hidden size, expert counts and widths, native context, vocab) plus the underlying forward-pass structure (Gated Attention + Gated DeltaNet dims, partial RoPE, full-attention interval) against the bundled GGUF's `qwen35moe` metadata. Run on demand (`python3 scripts/verify_arch.py`); reads the 21 GB GGUF (LFS smudge required) and exits non-zero on any mismatch. `block_count` is checked against **40**, this model's natural depth — the MTP-Preserved variant this repo does not ship reports 41. |
140
+ | `scripts/strip_mtp.py` | Drops an MTP / NextN layer from a Qwen 3.x GGUF (`python3 scripts/strip_mtp.py IN.gguf OUT.gguf`): removes the extra `blk.<last>.*` block, decrements `block_count`, drops `nextn_predict_layers`. Kept tensors are copied byte-for-byte (no re-quant); a conditional no-op (hardlink) on already-clean quants — **which is what it is for the quants this repo ships**, since llmfan46's plain `…-heretic-GGUF` files carry no NextN block. It stays in the build path as a guard for anyone pointing `build.sh` at the `…-Native-MTP-Preserved-GGUF` variant instead. Parity with the dense sibling `FoolDev/Thanatos-27B-HERETIC`, where the strip *is* required. |
141
  | `scripts/smoke_test.sh` | Integration smoke test against a running Ollama daemon: server reachable, model loaded, `tools` capability present, chat round-trip, and no control-token leakage. `TOOLS_TEST=1` adds a tool-call round-trip. Defaults to `MODEL=janus`. |
142
  | `scripts/bench.sh` | Measures tok/s from Ollama's `eval_count` / `eval_duration` over a short/medium/long prompt mix (with a discarded warmup). Defaults to `MODEL=janus`. |
143
+ | `scripts/load_bundle.sh` | Loads the bundled `Janus-35B-A3B.Q4_K_M.gguf` into Ollama as a local `janus` tag without an upstream pull (if the working-tree file is still an LFS pointer it downloads the real blob into `.cache/` — it does not smudge in place — then checks the arch is `qwen35moe`). |
144
  | `scripts/cap_ctx.sh` | Bakes a small-`num_ctx` local `janus` tag (bundled blob + this repo's Modelfile, `num_ctx`→4096, `num_batch` 256) for OpenAI `/v1` clients — which can't override the baked `num_ctx` and OOM on small hosts (see [Inference examples](#inference-examples)). Run `./scripts/cap_ctx.sh` (or `CTX=8192 ...`). |
145
+ | `scripts/fetch_vision.sh` | Downloads the vision projector (`Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf`, 902 MB) from the Heretic GGUF repo for llama.cpp image input (Ollama vision is broken upstream — see [Vision](#vision)). |
146
  | `examples/` | Ready-to-run Python clients for Ollama, Transformers, and llama-cpp-python (text, tools, and vision — see `examples/README.md`) |
147
 
148
+ GGUF-only release. Pull the Heretic safetensors from [`llmfan46/Qwen3.6-35B-A3B-uncensored-heretic`](https://huggingface.co/llmfan46/Qwen3.6-35B-A3B-uncensored-heretic) if you need the `transformers` tree (or the vanilla pre-Heretic base from [`Qwen/Qwen3.6-35B-A3B`](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)). The safetensors also keep the MTP layer that this GGUF does not carry — that's the path to speculative decoding under vLLM / SGLang.
149
 
150
+ > ### Bundled blob status: the 35B-A3B MoE is published
151
  >
152
+ > The bundled GGUF is **`Janus-35B-A3B.Q4_K_M.gguf`** (21,233,608,512 bytes —
153
+ > 21.23 GB decimal, 19.78 GiB binary) — the Qwen 3.6 35B-A3B MoE this card
154
+ > describes. `ollama run hf.co/FoolDev/Janus-35B-HERETIC` serves it directly, and
155
+ > `ollama create janus -f Modelfile` works from a fresh clone.
156
  >
157
+ > It is `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF`'s Q4_K_M
158
+ > (sha256 `b82f9f2155eb9c07139d48cc3a37880cf9d0edcca345ef6ad829b62941ccbb82`)
159
+ > **byte-for-byte**: no re-quantization, no MTP strip and no template restamp. This
160
+ > quant carries no NextN block to strip, and the GGUF keeps the base model's own
161
+ > embedded chat template — this repo ships no `chat_template.jinja` of its own.
 
 
162
  >
163
+ > Verified before publishing (2026-09-18): the file's GGUF header was read directly
164
+ > from the published blob by HTTP range request — `general.architecture qwen35moe`,
165
+ > `block_count` 40, 256 experts / 8 active, `context_length` 262144, vocab 248,320,
166
+ > 733 tensors, no `nextn` key and no NextN tensors — and every Architecture claim
167
+ > below matches it. Not re-run since the reversal: a live load, a tool-call
168
+ > round-trip and `scripts/live_check.sh`, all of which need the 21 GB blob on disk.
169
  >
170
+ > Between 0.2.0 and 0.6.5 this repo shipped a dense Qwen 3.8 27B
171
+ > (`Janus-27B.Q4_K_M.gguf`). That blob is no longer published; it remains in git
172
+ > history.
173
 
174
  ## Architecture
175
 
176
+ <img src="https://huggingface.co/FoolDev/Janus-35B-HERETIC/resolve/main/moe-routing.svg" alt="animated MoE routing visualization: 16x16 grid of 256 expert dots with 8 lit at any time, cycling through 8 routing patterns" width="640" />
 
 
177
 
178
+ A **sparse** hybrid stack: linear-attention (Gated DeltaNet) layers with a full
179
+ Gated Attention layer every fourth block, and a 256-expert MoE feed-forward on
180
+ every block. ~34.7B parameters total, ~3B active per token — 8 routed experts
181
+ plus one shared expert.
182
+
183
+ - Qwen 3.6, 35B-A3B mixture-of-experts (256 experts, 8 routed per token, plus a shared expert on every block)
184
+ - 40 layers, 10 × (3 × Gated DeltaNet → MoE / 1 × Gated Attention → MoE) — `full_attention_interval` 4, so 10 full-attention and 30 linear-attention layers
185
+ - Hidden size 2048; per-expert FFN intermediate 512, shared-expert FFN 512
186
+ - Gated Attention: 16 Q-heads / 2 KV-heads (GQA), head_dim 256, gated output projection, partial RoPE (`rope.dimension_count` 64 of 256 dims rotated), `rope.freq_base` 1e7
187
+ - Gated DeltaNet: conv kernel 4, state size 128, 16 groups, inner size 4096
188
  - 262 144 native context (extensible to 1 010 000 with YaRN, but YaRN is not enabled in the bundled GGUF)
189
+ - Vision + video supported by upstream (mmproj not included in this release — see [Vision](#vision))
190
  - Vocab 248,320
191
+ - GGUF arch stamp `qwen35moe` — the MoE arch in llama.cpp (`src/models/qwen35moe.cpp`, `case 40: LLM_TYPE_35B_A3B`); the dense members of the family stamp `qwen35`
192
+
193
+ Every bullet above is asserted against the bundled GGUF's own metadata by
194
+ `python3 scripts/verify_arch.py`.
195
 
196
+ **MTP head (absent from this quant).** The base *does* declare an MTP layer
197
+ (`mtp_num_hidden_layers: 1` in its `config.json`) for vLLM / SGLang speculative
198
+ decoding, but the Q4_K_M published in llmfan46's GGUF repo — the file this repo
199
+ bundles — carries no NextN block: `block_count` is 40 and `blk.0`…`blk.39` are all
200
+ that exist. So no strip is needed here, and `scripts/strip_mtp.py` is a defensive
201
+ no-op on it. llmfan46 also publishes a separate `…-Native-MTP-Preserved-GGUF`
202
+ variant whose Q4_K_M has `block_count` 41 with a NextN block at index 40; that one
203
+ *would* need stripping, and it is not what this repo ships. For the MTP head
204
+ itself, run the upstream safetensors under vLLM / SGLang.
205
 
206
  ## Quick start
207
 
 
209
 
210
  Drop the GGUF into your loader of choice. The chat template is embedded in the GGUF metadata, and llama.cpp and LM Studio use it by default, so no `--chat-template` flag is needed (llama.cpp has no `--chat-template auto` mode; it would take "auto" as the template itself).
211
 
212
+ **Tool calling works out of the box** — the GGUF carries Qwen 3.6's own template, so `--jinja` alone is enough and there is no file to pass:
213
 
214
  ```bash
215
+ llama-server -m Janus-35B-A3B.Q4_K_M.gguf --jinja
216
  ```
217
 
218
+ Tool calls use Qwen's native XML form, the one llama-server's parser reads. Two
219
+ things this path does **not** have, because they lived in the patched Qwen 3.8
220
+ template this repo shipped between 0.2.0 and 0.6.5 and no longer ships:
221
+ `reasoning_effort` tiers, and the `tool_call_format` JSON opt-in. See
222
+ [Tool / function calling](#tool--function-calling) and
223
+ [Reasoning effort](#reasoning-effort).
224
 
225
  ### Ollama
226
 
 
245
  > The examples use `model: "janus"`, the tag from the local build (path B). If you pulled via the TL;DR one-liner instead, use the full tag `hf.co/FoolDev/Janus-35B-HERETIC`, or run `ollama cp hf.co/FoolDev/Janus-35B-HERETIC janus` once to create the short tag.
246
 
247
  > **On memory-tight hosts, cap `num_ctx` first.** `/v1/chat/completions` (OpenAI-compat) has
248
+ > no `num_ctx` knob, so it loads at the baked **262,144** default (5.0 GiB KV / ~25 GiB total
249
+ > with Ollama's default f16 cache — derived from the GGUF's geometry, not measured on this blob),
250
+ > which is tight on a 32 GB box; `OLLAMA_KV_CACHE_TYPE=q8_0` on the
251
+ > server brings it to ~22.4 GiB (see [Hardware requirements](#hardware-requirements)). Either
252
  > call `/api/chat` with `"options": {"num_ctx": 4096}`, or bake a small-context tag for
253
  > OpenAI clients: `ollama run hf.co/FoolDev/Janus-35B-HERETIC:Q4_K_M`, then
254
  > `/set parameter num_ctx 4096` and `/save janus`, and point clients at `janus` — or in one step, **`./scripts/cap_ctx.sh`** (which bakes exactly that tag).
 
338
 
339
  ## Vision
340
 
341
+ The Qwen 3.6 base supports image (and video) input via a separate
342
  `mmproj` projector. The full multimodal stack is:
343
 
344
  ```
345
+ Janus-35B-A3B.Q4_K_M.gguf (21.23 GB, the text decoder)
346
+ Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf (902 MB, the vision projector)
347
  ```
348
 
349
  The projector and other-quant text decoders live at
350
+ [`llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF`](https://huggingface.co/llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF)
351
  (BF16 mmproj only). This repo intentionally does not redistribute it;
352
  `./scripts/fetch_vision.sh` pulls the projector into the repo root.
353
 
 
355
 
356
  | Loader | Text | Vision (mmproj) | Notes |
357
  |---|---|---|---|
358
+ | **llama.cpp** (`llama-mtmd-cli`, `llama-server --mmproj`) | ✅ | ✅ | Reference path. Upstream has the `qwen35moe` arch entry (`src/models/qwen35moe.cpp`). |
359
  | **llama-cpp-python** | ✅ | ✅ | See `examples/llama_cpp_vision.py`. |
360
+ | **Ollama** (0.33.3 and 0.34.x checked in source, 2026-09-18) | ✅ | ❌ | Text inference works: `qwen35moe` is handled through Ollama's llama.cpp compat shim at v0.33.3, v0.34.0 and v0.34.2. The version floor this row used to claim ("0.24+") was undated and inherited from the MoE-era card; it has not been re-established, so treat these three as the tested set. Vision (mmproj) is still broken: the C++ llama.cpp fallback that Ollama switches to when an mmproj is attached lacks it. `ollama create` accepts a dual-`FROM` (text + mmproj) and `ollama show` reports `vision` capability — but the **first inference request** fails with an `unknown model architecture` load error, and once mmproj is attached this blocks text inference too. See [ollama/ollama#14575](https://github.com/ollama/ollama/issues/14575) (closed as completed on 2026-08-04; #15898 was closed as its duplicate and the sync PR #15899 closed unmerged). Re-test on a current Ollama before assuming vision is still broken — this table has not been re-verified since it closed. |
361
  | **LM Studio** | ✅ | ✅ | Uses upstream llama.cpp directly. |
362
 
363
  ### Vision via llama.cpp
364
 
365
  ```bash
366
  # Fetch the projector first (into the repo root):
367
+ ./scripts/fetch_vision.sh # Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf
368
 
369
  # A. HTTP via llama-server (the easiest path):
370
  llama-server \
371
+ -m Janus-35B-A3B.Q4_K_M.gguf \
372
+ --mmproj Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf \
373
  --host 127.0.0.1 --port 8765 -c 8192 -ngl 99
374
  # then POST OpenAI-style chat completions with an image_url content block —
375
  # e.g. {"type":"image_url","image_url":{"url":"data:image/jpeg;base64,..."}}
 
380
  # B. CLI via llama-mtmd-cli (one-shot). It's a separate cmake target, so a
381
  # selective build can skip it; a plain `cmake --build build` produces it.
382
  llama-mtmd-cli \
383
+ -m Janus-35B-A3B.Q4_K_M.gguf \
384
+ --mmproj Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf \
385
  --image photo.jpg \
386
  -p "Describe this image."
387
 
388
  # C. Python via llama-cpp-python:
389
  python examples/llama_cpp_vision.py \
390
+ --gguf Janus-35B-A3B.Q4_K_M.gguf \
391
+ --mmproj Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf \
392
  --image /path/to/photo.jpg \
393
  --prompt "What is in this image?"
394
  ```
395
 
396
  The Ollama issue is closed upstream but this table has not been re-verified
397
  since; until you have re-tested, treat Ollama as **text-only** for this model. The bundled Q4_K_M decoder pairs with the projector directly — the
398
+ mmproj is family-wide for Qwen 3.6 35B-A3B, so no separate text download is
399
  needed for vision.
400
 
401
  ## Hardware requirements
402
 
403
+ This is a 21.23 GB Q4_K_M GGUF (19.78 GiB of weights). **The figures below are
404
+ derived from the GGUF's own geometry, not measured on this blob** — every measured
405
+ number this section used to carry was taken on the dense 27B that shipped between
406
+ 0.2.0 and 0.6.5 and has been removed rather than rescaled, because the geometry
407
+ differs in both directions: more weights, far less KV.
408
+
409
+ The KV cache covers the 10 full-attention layers (2 KV heads × 256 key dims + 256
410
+ value dims), so each token costs 10 × 2 × 512 × 2 bytes = **20,480 B at f16** —
411
+ **0.625 GiB per 32K tokens** — or about **10,880 B at q8_0** (0.332 GiB per 32K,
412
+ `OLLAMA_KV_CACHE_TYPE=q8_0` with `OLLAMA_FLASH_ATTENTION=1`). That is under a third
413
+ of the dense 27B's KV at the same window. The 30 linear-attention layers hold a fixed
414
+ recurrent state and the runtime allocates a compute buffer on top; **neither is
415
+ derivable from metadata and neither has been measured here**, so the totals below are
416
+ floors, not forecasts.
417
 
418
  | `num_ctx` | f16 cache (default) | q8_0 cache |
419
  |---|---|---|
420
+ | **262144 (default)** | 5.0 GiB KV, **~25 GiB** total | 2.66 GiB KV, **~22.4 GiB** total |
421
+ | 65536 | 1.25 GiB KV, ~21 GiB total | 0.66 GiB KV, ~20.4 GiB total |
422
+ | 32768 | 0.63 GiB KV, ~20.4 GiB total | 0.33 GiB KV, ~20.1 GiB total |
423
+ | 1010000 (needs YaRN) | 19.3 GiB KV, ~39 GiB total | 10.2 GiB KV, ~30 GiB total |
424
 
425
+ KV sizes are exact for this geometry and linear in `num_ctx`; the totals add the
426
+ 19.78 GiB of weights and nothing else. Re-measure with `make live-check` or
427
+ llama.cpp's `llama-fit-params -v` once the blob is on disk. It's **extensible to 1,010,000**, but this GGUF ships **no baked YaRN rope-scaling**, so positions past the 262144 native window use untrained RoPE and output degrades — reaching the 1.01M ceiling needs YaRN (see below); keep real work within ~262K otherwise.
428
 
429
  **How to override it:** `ollama run` has no `-o` flag, and `OLLAMA_CONTEXT_LENGTH`
430
  only sets a default that the baked `num_ctx` overrides — so set it per-session
 
442
 
443
  | Hardware | Status |
444
  |---|---|
445
+ | ≥48 GB RAM (CPU-only) | Works at the 262144 default (~25 GiB derived) with room to spare |
446
  | Single H100/A100 80 GB | Works, full offload |
447
  | RTX 4090 24 GB / 5090 32 GB + 32 GB RAM | Works, partial offload |
448
  | Mac Studio M2/M3 Ultra 64 GB+ unified | Works |
449
+ | 32 GB unified-memory laptops (Ryzen AI Max+, Apple M-series) | ~25 GiB derived at the 262144 default leaves little headroom on a 27 GiB-visible host; `OLLAMA_KV_CACHE_TYPE=q8_0` (~22.4 GiB) or `num_ctx 65536` (~21 GiB) is the comfortable setting |
450
 
451
+ **No throughput figures are published for this build.** Every tok/s number this card
452
+ carried was measured on the dense 27B, and a 35B-A3B MoE touches roughly 3B
453
+ parameters per token instead of 27B — a bandwidth profile different enough that
454
+ rescaling would be invention, not estimation. `./scripts/bench.sh` measures tok/s on
455
+ your own host; contributions of real numbers are welcome (see
456
+ [No formal evaluation](#known-limitations)).
 
 
457
 
458
  **GPU offload on a Strix Halo iGPU (Radeon 8060S): a small gain, and a trap.** On a
459
  Vulkan integrated GPU, llama.cpp - including the build Ollama 0.33.3 bundles - turns
 
461
  the same pool as the GPU, so `num_gpu` alone does not limit GPU memory: a first attempt
462
  with llama-server filled the pool and took the desktop down with it. With
463
  `LLAMA_ARG_NO_HOST=1` in the Ollama server's environment (llama.cpp's `--no-host`) those
464
+ layers stay in ordinary RAM. That mechanism is loader-side and applies to any model; the
465
+ tok/s series this paragraph used to quote was measured on the dense 27B and has been
466
+ removed with the rest. What generalises: the GPU's GTT is system RAM, so offloading frees
467
+ none of it, and a first attempt without `--no-host` filled the pool and took the desktop
468
+ session down with it. If you try the GPU on a machine like this, set
469
+ `LLAMA_ARG_NO_HOST=1` and a fixed `num_gpu`. The mechanism is reported upstream in
 
 
470
  [ggml-org/llama.cpp#27360](https://github.com/ggml-org/llama.cpp/issues/27360).
471
 
472
  **Reaching a coherent ~1.01M context (opt-in YaRN).** The bundled GGUF ships no YaRN rope-scaling, so raising `num_ctx` toward the 1.01M ceiling degrades past the 262144 native window (see above). Ollama has no rope knob, so for a genuinely coherent long context run the GGUF under llama.cpp with YaRN enabled:
473
 
474
  ```bash
475
+ llama-server -m Janus-35B-A3B.Q4_K_M.gguf \
476
  --rope-scaling yarn --yarn-orig-ctx 262144 --rope-scale 3.853 -c 1010000
477
  ```
478
 
 
480
 
481
  ## Chat template
482
 
483
+ The model uses the standard Qwen 3.x ChatML format with `<|im_start|>` / `<|im_end|>` role markers. Two templates are in play. The GGUF carries **Qwen 3.6's own embedded template**, exactly as llmfan46 published it — this repo ships no `chat_template.jinja` of its own — and that is what llama.cpp, LM Studio and KoboldCpp render. On Ollama the Go `template` / `Modelfile` TEMPLATE is what runs instead: Ollama picks between the two at load time (see [Which template Ollama runs](#which-template-ollama-runs)). Where the two differ, this card says which one it is describing.
484
 
485
  ### Plain conversation
486
 
 
527
  | Ollama `/v1/chat/completions` | `reasoning` only — `reasoning_content` and `thinking` are dropped silently, and a message whose `content` is an array of parts keeps its reasoning only if it also has `tool_calls` |
528
  | llama-server `/v1/chat/completions` | `reasoning_content` |
529
 
530
+ Which field each endpoint reads is decided by the server, not the model. Checked
531
+ live on 2026-09-13 — on the dense 27B blob this repo shipped between 0.2.0 and
532
+ 0.6.5, CPU only — Ollama 0.33.3 replayed `thinking` on `/api/chat` (232 prompt
533
+ tokens with it, 187 without) and `reasoning` on `/v1` (215 vs 187) and ignored
534
+ `reasoning_content` on `/v1` (187); llama-server master rendered `reasoning_content`
535
+ and dropped a `reasoning` field. The behaviour is Ollama-side and carries over; the
536
+ token counts were measured on that other blob and are not this model's.
537
  That Ollama's `/v1` also drops `thinking`, and the array-content rule, come from
538
  Ollama's v0.33.3 and v0.34.0 source, and still hold on the current release:
539
  `openai/openai.go` and `server/prompt.go` are byte-identical from v0.33.3 through
540
  v0.34.2 (source, 2026-09-17). Ollama never passes reasoning to the GGUF's
541
  embedded Jinja template, so on Ollama only the Go template decides what is replayed.
542
+ On llama.cpp an earlier assistant turn that carries no reasoning renders with **no**
543
+ `<think>` block at all: Qwen 3.6's template gates the block on
544
+ `reasoning_content|length > 0`. The 3.8 template this repo used to ship reconstructed
545
+ an empty one instead, so prompts built by the two differ on that turn.
546
+
547
+ **The cost is context.** Every retained trace stays in the prompt — measured on the
548
+ dense 27B at 181 prompt tokens without and 464 with, on a 3-message conversation whose
549
+ first assistant turn carried a ~240-word trace, and not re-measured here. Prefill grows with every retained trace,
550
  and on a CPU-only host prefill is the slow part. Releases 0.3.0 through 0.3.2
551
  turned replay off for that reason; the CHANGELOG has the history.
552
 
553
+ **On llama.cpp, Qwen 3.6's own template decides.** Read from the template inside
554
+ the published blob (2026-09-18, from source — not re-tested live on this model):
555
+ `preserve_thinking` defaults to **true**, so an earlier turn's reasoning is replayed
556
+ by default; the assistant branch takes the trace from `message.reasoning_content`,
557
+ and failing that recovers one by splitting a `<think>` block out of `content`; and
558
+ there is no `preserve_reasoning` variable in it at all. That last point costs
559
+ nothing, because llama-server derives a `preserve_thinking` kwarg from
560
+ `preserve_reasoning` (`common/jinja/caps.cpp`), and from b10763 it passes
561
+ `preserve_reasoning=true` by default — so `--no-reasoning-preserve` and a request's
562
+ `preserve_reasoning: false` still reach this template through the derived copy,
563
+ while a request's own `preserve_thinking` is read directly.
 
 
 
 
 
 
 
564
 
565
  **Which llama.cpp an Ollama carries.** 0.33.3 and 0.34.0 bundle b10760; 0.34.1
566
  bundles b10864 and 0.34.2 b10969 (each release's `LLAMA_CPP_VERSION`). From 0.34.1
 
570
  play only when Ollama falls back to its C++ engine, as it does once an mmproj is
571
  attached (see [Loader compatibility](#loader-compatibility)).
572
 
573
+ **Nothing raises on conflicting settings.** The Qwen 3.8 template that this repo
574
+ patched between 0.4.0 and 0.6.5 raised when `preserve_thinking` and
575
+ `preserve_reasoning` disagreed, and llama-server returned that raise as HTTP 500 —
576
+ which is what the patch existed to prevent. Qwen 3.6's template has no such check:
577
+ it reads `preserve_thinking`, ignores anything else, and cannot fail that way.
 
578
 
579
  **To turn replay off:**
580
 
581
  - **llama-server** — start it with `--no-reasoning-preserve`, or send
582
+ `"chat_template_kwargs": {"preserve_thinking": false}` on a request. A request's
 
583
  `preserve_thinking: true` switches it back on for that request.
584
+ `{"preserve_reasoning": false}` works too, through the kwarg llama-server derives
585
+ from it; the template itself never reads that name.
586
  - **Ollama** — edit the template. In the assistant branch of both the `Modelfile`
587
  and `template` (so `make check`'s bridge sync stays green), change `{{ if (and $.IsThinkSet .Thinking) -}}` back to Qwen's stock
588
  `{{ if (and $.IsThinkSet (and .Thinking (or $last (gt $i $lastUserIdx)))) -}}`
 
601
 
602
  or send `"think": false` on `/api/chat`. With thinking off the model skips the reasoning trace and answers straight into `content`; with it on (the default) reasoning is emitted into the `thinking` field.
603
 
604
+ On llama.cpp the switch is the embedded template's own: send
605
+ `"chat_template_kwargs": {"enable_thinking": false}`, which primes a closed, empty
606
+ `<think>` block — and, because the template gates its replay branch on
607
+ `enable_thinking` too, drops every earlier turn's trace for that request. That
608
+ variable is upstream Qwen 3.6's, not this repo's.
609
 
610
+ ### Reasoning effort
 
 
611
 
612
+ **This works on Ollama, and not on llama.cpp.** The instruction comes from this
613
+ repo's Go `template`, which Ollama runs (see
614
+ [Which template Ollama runs](#which-template-ollama-runs)). Qwen 3.6's own template —
615
+ the one inside the bundled GGUF, and so the one llama.cpp and LM Studio render — has
616
+ no `reasoning_effort` variable at all, so there the tier values are accepted by the
617
+ server and then quietly ignored (read from the template, 2026-09-18). `none` is the
618
+ one exception: llama-server maps it to `enable_thinking: false` and drops the kwarg
619
+ before the template runs (`tools/server/server-common.cpp`, checked 2026-09-18), so
620
+ `reasoning_effort: "none"` does switch thinking off on llama.cpp. Between 0.2.0 and
621
+ 0.6.5 this repo shipped a patched Qwen 3.8 template that did implement the tiers on
622
+ the llama.cpp path; that file is gone with the rebase back to the MoE.
623
+
624
+ Ollama folds the eight values its `/v1` layer accepts into four think levels before
625
+ any template sees them, so the Go template maps levels, not values:
626
+
627
+ | `reasoning_effort` (Ollama `/v1`) | Think level | Effect |
628
+ |---|---|---|
629
+ | `high`, `xhigh`, `max`, `ultra` | xhigh | the model is told to think carefully and check its assumptions |
630
+ | `medium`, unset, `null` | medium | the default; no effort instruction is added |
631
+ | `low`, `minimal` | low | the model is told to keep its thinking brief |
632
+ | `none` | — | thinking **off** for that request, the same as `"think": false` |
633
+
634
+ `"think": "high"`, `"low"` and `"max"` on `/api/chat` map the same way. Mixed case
635
+ such as `"HIGH"` is rejected by Ollama itself with HTTP 400 before any template runs.
636
+ The value-to-level mapping comes from Ollama's source, unchanged from v0.33.3 through
637
+ v0.34.2 (2026-09-17).
638
+
639
+ Verified with `scripts/live_check.sh` on Ollama 0.33.3 and 0.34.0 — on the dense 27B
640
+ bundle, before the reversal: on a one-line chat, `high` and `max` each added 38 prompt
641
+ tokens and `low` 26, while `medium` and an unset value added none. Both the Go
642
+ template and Ollama's folding are model-independent, so the mapping carries over
643
+ unchanged; the live check has not been re-run against the MoE blob (`make live-check`
644
+ needs the 21 GB file on disk).
645
 
646
  ### Which template Ollama runs
647
 
648
+ Two templates exist for the same ChatML model: the Go `template` this repo ships
649
+ (served by HF's Ollama bridge) / `Modelfile` TEMPLATE, and **Qwen 3.6's own template
650
+ inside the GGUF**, which llmfan46 stamped and this repo leaves untouched. They do not
651
+ render identical prompts — the Go template pins JSON tool calls and adds a
652
+ reasoning-effort line from Ollama's think level, while the embedded template teaches
653
+ the XML `<function=…>` form and has no effort variable at all.
654
+
655
+ Ollama picks one at load time: it prefers the embedded template when that advertises
656
+ more capabilities — tools, thinking — and keeps the Go template when they tie. Both
657
+ advertise tools and thinking, so they tie and **Ollama runs the Go template**;
658
+ llama.cpp and LM Studio run the embedded one. That was verified live on Ollama 0.33.3
659
+ and 0.34.0 against the dense 27B bundle, and the selection code is unchanged through
660
+ v0.34.2 (source, 2026-09-17); the tie itself depends on what the *embedded* template
661
+ advertises, so it is expected rather than re-verified for this blob until
662
+ `make live-check` runs on it. Ollama's built-in renderers — it ships a `qwen35` one —
663
+ displace both templates only when the model's own config names a renderer or parser,
664
+ which this repo's does not. Ollama logs the choice:
665
 
666
  ```text
667
  msg="template selection" ... selected=go_template go_template="[completion tools thinking]"
 
669
 
670
  The Go template's "thinking" entry comes from its single `.Thinking` reference —
671
  the block that renders each assistant turn's reasoning. Removing it drops that
672
+ capability, and Ollama switches to the embedded template: tool calls change from
673
+ JSON-in-XML to Qwen's XML form, which Ollama's extractor does not parse into a
674
+ `tool_calls` array, `reasoning_effort` stops doing anything (the embedded template
675
+ has no such variable), and a single tool call can come back twice (seen once on the
676
+ 27B; cause inferred, not confirmed — see [Known limitations](#known-limitations)).
677
+ Setting `OLLAMA_GO_TEMPLATE=1` or `=0` on the Ollama server forces one template or
678
+ the other for every model.
679
 
680
  ### Tool / function calling
681
 
682
+ The model emits whichever format the prompt specifies, and the two paths specify
683
+ different ones: the Ollama path pins JSON-in-XML, while the llama.cpp path gets
684
+ Qwen's native XML form from the template embedded in the GGUF.
685
 
686
  **Ollama path** (this repo's `Modelfile`). The TEMPLATE advertises tools inside `<tools>…</tools>` and asks the model to reply in JSON-in-XML — the form Ollama's tool-call extractor parses into a structured `tool_calls` array on `/api/chat` and `/v1/chat/completions`. The tool definitions go in with `{{ json .Function }}`: on Ollama 0.14 and later a bare `{{ .Function }}` prints Go struct syntax instead of JSON ([ollama/ollama#14601](https://github.com/ollama/ollama/issues/14601); fix proposed in [ollama/ollama#18391](https://github.com/ollama/ollama/pull/18391)). The model replies like this:
687
 
 
691
  </tool_call>
692
  ```
693
 
694
+ **Embedded-template path** (llama.cpp, llama-cpp-python, LM Studio, KoboldCpp). The bundled GGUF carries Qwen 3.6's own template, so `--jinja` alone is enough — there is no file to pass and nothing to stamp. It teaches the XML form:
695
 
696
  ```text
697
  <tool_call>
 
703
  </tool_call>
704
  ```
705
 
706
+ **Keep XML on llama-server.** llama-server recognises this template as Qwen3-Coder-style and parses the model's reply with a parser — and, once the model opens `<tool_call>`, a grammar — that accept only `<function=…><parameter=…>`. Qwen 3.6's tool-instruction literal is byte-identical to Qwen 3.8's (compared 2026-09-18), so the detection and the wire format are exactly what this card documented before the reversal. What is gone is the `tool_call_format` knob: the JSON opt-in described in releases 0.3.0 through 0.6.5 lived in the patched 3.8 template this repo no longer ships.
 
 
 
 
707
 
708
+ **Replayed tool calls: normalise `arguments` first.** Every OpenAI-compatible client hands assistant tool calls back with `function.arguments` as a JSON **string**, while the XML parameter form needs the individual key/value pairs — a chat template has `tojson` but no inverse. Qwen 3.6's template renders `<parameter=…>` only when `arguments` is a mapping; handed a string it emits that string as-is inside `<function=…>`, carrying upstream's own comment that you should "normalize arguments to a dict in your serving layer before applying this template". It does **not** raise, so a loader that passes the string straight through produces a silently malformed prompt rather than an error — the failure is quieter here than it was under the 3.8 template, which raised and returned HTTP 500.
 
 
 
 
 
 
 
709
 
710
+ In practice the loaders convert before rendering: llama-server decodes the string into a mapping, KoboldCpp (`--jinja --jinja_tools`) runs `json.loads` first, and the LM Studio app parses the arguments before its own JavaScript Jinja engine renders. All three were checked on 2026-09-14 against the 3.8 template this repo shipped then; the conversion is loader-side, so it applies unchanged here (from source for this template — not re-tested on this blob).
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
711
 
712
  #### Example (Ollama, OpenAI-compatible API)
713
 
 
744
 
745
  - Use direct prompts ("Call X for Y") rather than soft hints ("Use the tool"). The model thinks before committing to a call, and weak prompts can exhaust `num_predict` inside the `<think>` block before the call is emitted.
746
  - Allow at least `num_predict: 1024` (or `max_tokens: 1024`) for tool-calling turns, more if the schemas are large.
747
+ - The JSON-in-XML format is what Ollama's tool-call extractor understands, and the Go template pins it; the GGUF's embedded template renders the XML `<function=…>` form. So the prompt format does change when you swap loaders, while client code should not have to, as long as the server returns a parsed `tool_calls` array.
748
 
749
  ## Known limitations
750
 
751
+ - **On the embedded-template path, a single tool call can come back twice.** Seen on the dense 27B blob, not re-tested here: in one test Ollama 0.33.3, running the embedded template, returned a first-turn tool call twice. A separate raw generation showed the model drafting the call inside `<think>` before emitting it once after `</think>`, so the parser most likely matched such a draft (inferred, not confirmed). It did not reproduce on standalone llama.cpp master (`a2878d3`, CPU): across both models, 12 first-turn calls (three prompts, with and without `parallel_tool_calls`) and two streamed runs each came back exactly once. Nor on b10760, the llama.cpp build Ollama 0.33.3 bundles and the one the duplicate came through: six more first-turn calls and a streamed run on this model each came back once. LM Studio's CPU engine (llama.cpp b10519) passed the same checks; the LM Studio app's own API layer is untested. Ollama's default path — the Go template, see [Which template Ollama runs](#which-template-ollama-runs) — returned one call in a comparable test; whether the prompt or the parser makes the difference was not isolated. If your client executes tools wherever the embedded template renders (llama.cpp, LM Studio, or Ollama with `OLLAMA_GO_TEMPLATE=0`), drop identical consecutive calls.
752
+ - **No mmproj in this release.** The base Qwen 3.6 35B-A3B supports image and video input via a separate `mmproj` file, which is not included here. Text-only inference works out of the box; multimodal inference requires fetching `Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf` (or equivalent) from upstream — run `./scripts/fetch_vision.sh` and see [Vision](#vision) for the full path.
753
  - **Quantization-induced quality loss.** Q4_K_M is a strong general-purpose quant but does measurably degrade math and code accuracy compared to BF16. If you need maximum quality, run the upstream safetensors on a host with room for the full BF16 weights.
754
+ - **No MTP head on the GGUF path.** The base declares a multi-token-prediction layer (`mtp_num_hidden_layers: 1`), but the Q4_K_M published in llmfan46's GGUF repo — the file this repo bundles — carries no NextN block at all (see [Architecture](#architecture)). No speculative decoding here: use the upstream safetensors under vLLM / SGLang, or llmfan46's separate `…-Native-MTP-Preserved-GGUF` variant, if you need it.
755
+ - **Sparse compute, dense memory.** Only ~3B of ~34.7B parameters are touched per token, but every expert has to be resident: the 19.78 GiB of weights is a floor whatever the routing does, and MoE sparsity buys compute, not RAM. What it does buy on the memory side is a small KV cache — 0.625 GiB per 32K at f16, under a third of a comparable dense stack — so long contexts are unusually cheap here relative to the weights.
756
+ - **Thinking traces can loop.** Like most reasoning-distilled models, Janus-35B occasionally gets stuck repeating itself inside `<think>` tags. Mitigations: lower temperature to 0.4-0.6, raise `repeat_penalty` to 1.08, or set a `<think>`-token budget cap if your loader supports it.
757
  - **Large tool-call arguments can be dropped.** Ollama's JSON-in-XML tool format makes the model JSON-escape the entire arguments object inline; for a big/complex payload (e.g. a file's `content` in a `write_file` call) the model can fail to escape it, so the field arrives `undefined` and the call fails. Qwen's native `<function=…><parameter=…>` format (raw values, no escaping) was tested as a fix but parses unreliably through Ollama, so the template deliberately keeps JSON-in-XML. Mitigation: write large files in smaller pieces per call.
758
  - **Uncensored base — not aligned with any specific safety policy.** This is a personal repackage of an open-weight base whose refusal behavior has been abliterated away (the llmfan46 Heretic base). There is no RLHF refusal layer; the model will attempt most requests, so downstream safety is entirely the operator's responsibility.
759
+ - **No formal evaluation in this card, and no throughput figures at all.** The hardware table is derived from the GGUF's geometry, not measured on this blob, and every tok/s number the card used to carry was measured on the dense 27B and has been removed rather than rescaled. No task benchmarks are claimed either. If you produce real numbers (tok/s, MMLU, HumanEval, …) and want them included, file a PR.
760
 
761
  ## Related models
762
 
763
  | Model | Size | Notes |
764
  |---|---|---|
765
+ | [llmfan46/Qwen3.6-35B-A3B-uncensored-heretic](https://huggingface.co/llmfan46/Qwen3.6-35B-A3B-uncensored-heretic) | 35B-A3B MoE | **Immediate base.** Uncensored Heretic abliteration of Qwen 3.6 35B-A3B; `transformers`-native safetensors, and the only place the MTP layer survives. |
766
+ | [llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF](https://huggingface.co/llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF) | 35B-A3B MoE | Heretic GGUFs — BF16, Q3_K_L, Q3_K_M, Q4_K_M, Q4_K_S, Q5_K_M, Q5_K_S, Q6_K, Q8_0, plus the BF16 mmproj. MTP-clean, so they load as-is; the bundled Q4_K_M **is** this repo's Q4_K_M, byte-for-byte. |
767
+ | [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) | 35B-A3B MoE | Upstream pre-Heretic base model. `transformers`-native multimodal weights. |
768
+ | [FoolDev/Thanatos-27B-HERETIC](https://huggingface.co/FoolDev/Thanatos-27B-HERETIC) | 27B dense | Sibling repo, and now a genuinely different model: a **dense Qwen 3.8 27B**, abliterated from the ARA-based [`heretic-org/Qwen3.8-27B-heretic-ara`](https://huggingface.co/heretic-org/Qwen3.8-27B-heretic-ara). Same teacher (Fable 5), same tooling, newer Qwen generation, no experts. Pick Thanatos for a smaller resident footprint and a dense compute profile; pick Janus for 35B-class capacity at ~3B active. (The older `FoolDev/Thanatos-27B` and `Thanatos-27B-Heretic` slugs now 307 to this path.) |
769
+ | [Crownelius/Crow-9B-HERETIC-4.6](https://huggingface.co/Crownelius/Crow-9B-HERETIC-4.6) | 9B dense | Heretic-flavored fine-tune on a smaller 9B Qwen base. Useful as a fast first-pass model when 35B is too heavy for the host. |
770
 
771
+ > **Janus vs Thanatos.** Since 0.7.0 the two are genuinely different models, not two abliterations of one base. **Janus** is Qwen 3.6 **35B-A3B**: a mixture of experts, ~34.7B parameters total with ~3B active per token, 40 layers, 256 experts (8 routed + 1 shared), on llmfan46's Heretic base, and its bundled quant is MTP-clean so no strip is needed. **Thanatos** is a dense Qwen 3.8 **27B** — every parameter on every token, 64 layers — on heretic-org's ARA base, and its bundle *is* MTP-stripped, because that base ships the NextN block. They still share the 262,144 native window, the ChatML frame, the Claude Fable 5 teacher slot and this repo's tooling. Choose on shape, not only on abliteration: Janus for 35B-class capacity at ~3B active and a KV cache under a third the size (0.625 vs 2 GiB per 32K at f16), Thanatos for the smaller resident footprint (16.1 vs 19.78 GiB of weights) and a newer Qwen generation.
772
 
773
  ## Credits
774
 
775
+ - Immediate base: [llmfan46/Qwen3.6-35B-A3B-uncensored-heretic](https://huggingface.co/llmfan46/Qwen3.6-35B-A3B-uncensored-heretic) — Heretic-style abliteration of Qwen 3.6 35B-A3B
776
+ - Upstream base: [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) (Alibaba)
777
  - Reasoning teacher: Claude Fable 5 (Anthropic)
778
  - Distillation lineage and dataset curation: [Crownelius](https://huggingface.co/Crownelius)
779
 
banner.png CHANGED
banner.svg CHANGED
bmc-banner.svg CHANGED
chat_template.jinja DELETED
@@ -1,642 +0,0 @@
1
- {#- Janus-27B chat template for llama.cpp / llama-server (--jinja).
2
-
3
- This is the base model's own embedded template (qwen3.8-safe-v3.1), with
4
- two deliberate changes, each marked CHANGED FROM UPSTREAM in place:
5
- 1. `preserve_thinking` and `preserve_reasoning` never raise: when both are
6
- set, `preserve_thinking` wins, where upstream raises on conflicting
7
- values. Earlier turns' reasoning is replayed by default, as upstream
8
- has it;
9
- 2. `reasoning_effort` is normalised instead of validated, so an unfamiliar
10
- value falls back to the default rather than failing the request.
11
-
12
- `tool_call_format` defaults to 'xml', as upstream ships it. 0.3.0 to 0.4.0
13
- of this repo defaulted to 'auto'; 0.5.0 went back. llama-server parses this
14
- template's output with its Qwen3-Coder parser, whose grammar accepts only
15
- the XML form <tool_call><function=NAME><parameter=ARG>, and it turns a
16
- client's JSON-string `function.arguments` into a mapping before rendering, so
17
- a replayed tool call renders as XML there under either default and nothing
18
- raises. Pinning 'xml' keeps the prompt in the only form that parser reads.
19
-
20
- 'auto' stays available for a loader that hands the template the raw JSON
21
- string: the XML parameter form needs the individual key/value pairs and a
22
- chat template has `tojson` but no inverse, so under 'xml' the template
23
- raises there; 'auto' renders the whole prompt in JSON form instead. Opt in
24
- through the loader's template kwargs, e.g.
25
- chat_template_kwargs: {"tool_call_format": "auto"}. On llama-server 'auto'
26
- changes nothing, since arguments arrive as mappings; 'json' would break its
27
- parser.
28
-
29
- Nothing else is modified. max_tool_arg_chars,
30
- auto_disable_thinking_with_tools, the vision handling and the whitespace
31
- contract are all upstream's. This file is what scripts/build.sh stamps into
32
- the GGUF it builds.
33
-
34
- Guarded by scripts/check_chat_template.py (check 8 in scripts/check.sh).
35
- -#}
36
- {%- set template_version = "qwen3.8-safe-v3.1" %}
37
- {#- -------------------------------------------------------------------------
38
- Whitespace contract: every tag opener in this file carries a leading minus,
39
- so it strips the whitespace and newlines that precede it. All emitted text
40
- comes from string literals, which makes the rendered prompt byte-exact and
41
- free of incidental newlines. Comment tags need the minus just as much as
42
- block tags do; a plain comment tag breaks the strip chain and leaks the
43
- surrounding indentation into the prompt.
44
- ------------------------------------------------------------------------- -#}
45
- {#- The message list has to exist before anything else can inspect it. -#}
46
- {%- if not messages %}
47
- {{- raise_exception('No messages provided.') }}
48
- {%- endif %}
49
- {#- -------------------------------------------------------------------------
50
- Configuration
51
- ------------------------------------------------------------------------- -#}
52
- {#- The default is 'xml', as upstream ships it (0.3.0 to 0.4.0 of this repo used
53
- 'auto'; see the header and "Resolve the tool-call wire format" below). -#}
54
- {%- set _requested_tool_format =
55
- tool_call_format if tool_call_format is defined else 'xml'
56
- %}
57
- {%- if _requested_tool_format not in ('auto', 'xml', 'json') %}
58
- {{- raise_exception(
59
- 'Unexpected tool_call_format ' ~ _requested_tool_format ~
60
- '. Supported values are xml (the default), auto, and json.'
61
- ) }}
62
- {%- endif %}
63
- {%- set add_vision_id = add_vision_id if add_vision_id is defined else false %}
64
- {%- set enable_thinking = enable_thinking if enable_thinking is defined else true %}
65
- {%- set auto_disable_thinking_with_tools =
66
- auto_disable_thinking_with_tools
67
- if auto_disable_thinking_with_tools is defined
68
- else false
69
- %}
70
- {#- CHANGED FROM UPSTREAM: upstream raises when preserve_reasoning and
71
- preserve_thinking are both set to different values, and otherwise lets
72
- preserve_reasoning win. llama-server from b10763 passes
73
- preserve_reasoning=true unless run with --no-reasoning-preserve, and a
74
- request's preserve_thinking kwarg replaces the copy it derives from that, so
75
- a request sending preserve_thinking=false made upstream raise there, as
76
- preserve_thinking=true did against --no-reasoning-preserve on any build;
77
- llama-server returns the raise as an HTTP 500. Here preserve_thinking wins
78
- and nothing raises. With neither set, earlier turns' reasoning is replayed, as upstream
79
- does. -#}
80
- {%- if preserve_thinking is defined and preserve_thinking is not none %}
81
- {%- set _preserve_thinking = preserve_thinking %}
82
- {%- elif preserve_reasoning is defined and preserve_reasoning is not none %}
83
- {%- set _preserve_thinking = preserve_reasoning %}
84
- {%- else %}
85
- {%- set _preserve_thinking = true %}
86
- {%- endif %}
87
- {%- set max_tool_arg_chars =
88
- max_tool_arg_chars if max_tool_arg_chars is defined else 0
89
- %}
90
- {%- set max_tool_response_chars =
91
- max_tool_response_chars if max_tool_response_chars is defined else 0
92
- %}
93
- {%- if max_tool_arg_chars is not number or max_tool_arg_chars < 0 %}
94
- {{- raise_exception('max_tool_arg_chars must be a non-negative number.') }}
95
- {%- endif %}
96
- {%- if max_tool_response_chars is not number or max_tool_response_chars < 0 %}
97
- {{- raise_exception('max_tool_response_chars must be a non-negative number.') }}
98
- {%- endif %}
99
- {%- if tools is defined and tools is not none and tools %}
100
- {%- if tools is string or tools is mapping or tools is not iterable %}
101
- {{- raise_exception('tools must be an iterable list of tool definitions.') }}
102
- {%- endif %}
103
- {%- set _has_tools = true %}
104
- {%- else %}
105
- {%- set _has_tools = false %}
106
- {%- endif %}
107
- {#- -------------------------------------------------------------------------
108
- Resolve the tool-call wire format.
109
-
110
- OpenAI-compatible servers hand assistant tool calls back with
111
- `function.arguments` as a JSON *string*, not a mapping. The XML parameter
112
- form needs the individual key/value pairs, and a chat template cannot
113
- decode a JSON string: the standard chat-template environment provides
114
- `tojson` but no inverse, and no codepoint-to-character primitive, so a
115
- hand-rolled scanner could not decode \uXXXX escapes correctly. Guessing
116
- would silently corrupt tool arguments, which is worse than any error.
117
-
118
- The default is XML, matching Qwen's native tool-call wire format and the
119
- Qwen3-Coder parser llama-server uses for this template - that parser and its
120
- grammar accept nothing else. llama-server converts JSON-string arguments to
121
- mappings before rendering, so they never reach this check there. 'auto'
122
- (opt-in) renders the whole prompt in JSON form whenever a historical tool
123
- call carries string arguments. That path is lossless, needs no decoding, and
124
- keeps the system-prompt instructions and rendered history in one consistent
125
- format. Histories whose arguments are all mappings still render as XML,
126
- byte-identically to tool_call_format='xml'.
127
-
128
- Under 'xml', the default, a caller still
129
- gets a hard error for non-empty string arguments, because that request
130
- cannot be honoured faithfully in the native XML parameter representation.
131
- ------------------------------------------------------------------------- -#}
132
- {%- set format_scan = namespace(has_json_string_arguments=false) %}
133
- {%- for message in messages %}
134
- {%- if message.role == 'assistant'
135
- and message.tool_calls is defined
136
- and message.tool_calls is not none
137
- and message.tool_calls
138
- and message.tool_calls is not string
139
- and message.tool_calls is not mapping
140
- and message.tool_calls is iterable %}
141
- {%- for tool_call in message.tool_calls %}
142
- {%- if tool_call.function is defined
143
- and tool_call.function is not none %}
144
- {%- set _tc = tool_call.function %}
145
- {%- else %}
146
- {%- set _tc = tool_call %}
147
- {%- endif %}
148
- {%- if _tc.arguments is defined
149
- and _tc.arguments is not none
150
- and _tc.arguments is string
151
- and _tc.arguments | trim
152
- and _tc.arguments | trim != '{}' %}
153
- {%- set format_scan.has_json_string_arguments = true %}
154
- {%- endif %}
155
- {%- endfor %}
156
- {%- endif %}
157
- {%- endfor %}
158
- {%- if _requested_tool_format == 'auto' %}
159
- {%- if format_scan.has_json_string_arguments %}
160
- {%- set _tool_format = 'json' %}
161
- {%- else %}
162
- {%- set _tool_format = 'xml' %}
163
- {%- endif %}
164
- {%- else %}
165
- {%- set _tool_format = _requested_tool_format %}
166
- {%- endif %}
167
- {%- set _thinking_enabled = enable_thinking %}
168
- {%- if auto_disable_thinking_with_tools and _has_tools %}
169
- {%- set _thinking_enabled = false %}
170
- {%- endif %}
171
- {#- Normalize and validate reasoning effort even when thinking is disabled.
172
- Explicit null follows the template default; OpenAI-style `minimal` maps to
173
- low, while `none` disables thinking for this render. -#}
174
- {#- CHANGED FROM UPSTREAM: normalise instead of raising. Clients do not agree on
175
- the vocabulary - OpenAI-style minimal/low/medium/high, xhigh, the "max" that
176
- Ollama's own /v1 layer sends for xhigh and ultra, mixed case - and upstream
177
- raised on anything off its list. Under Ollama >= 0.33 and
178
- llama-server a template raise is an HTTP 500 for the entire request, so one
179
- unfamiliar effort word made the model unusable. Values are lower-cased and
180
- trimmed, aliases map to the nearest tier, and anything unrecognised falls back
181
- to the medium default instead of failing the request. -#}
182
- {%- if reasoning_effort is undefined or reasoning_effort is none %}
183
- {%- set _effort_raw = '' %}
184
- {%- else %}
185
- {%- set _effort_raw = reasoning_effort | string | lower | trim %}
186
- {%- endif %}
187
- {%- if _effort_raw in ('none', 'off', 'disabled', 'false') %}
188
- {%- set _thinking_enabled = false %}
189
- {%- set _reasoning_effort = 'xhigh' %}
190
- {%- elif _effort_raw in ('xhigh', 'high', 'max', 'ultra', 'maximum', 'highest', 'x-high', 'x_high', 'extra-high', 'extra_high', 'very-high', 'very_high') %}
191
- {%- set _reasoning_effort = 'xhigh' %}
192
- {%- elif _effort_raw in ('low', 'minimal', 'min') %}
193
- {%- set _reasoning_effort = 'low' %}
194
- {%- else %}
195
- {#- medium, empty, auto, default, and anything unrecognised -#}
196
- {%- set _reasoning_effort = 'medium' %}
197
- {%- endif %}
198
- {%- set reasoning_instructions = '' %}
199
- {%- if _thinking_enabled %}
200
- {%- if _reasoning_effort == 'xhigh' %}
201
- {%- set reasoning_instructions =
202
- 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.'
203
- %}
204
- {%- elif _reasoning_effort == 'low' %}
205
- {%- set reasoning_instructions =
206
- 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.'
207
- %}
208
- {%- endif %}
209
- {%- endif %}
210
- {#- -------------------------------------------------------------------------
211
- Content rendering
212
- ------------------------------------------------------------------------- -#}
213
- {%- set image_count = namespace(value=0) %}
214
- {%- set video_count = namespace(value=0) %}
215
- {%- macro render_content(content, do_vision_count, is_system_content=false) %}
216
- {%- if content is string %}
217
- {{- content }}
218
- {%- elif content is iterable and content is not mapping %}
219
- {%- for item in content %}
220
- {%- if item is not mapping %}
221
- {{- raise_exception(
222
- 'Content-list items must be mappings with a supported text, image, or video type.'
223
- ) }}
224
- {%- endif %}
225
- {%- set _item_type = item.type if 'type' in item else none %}
226
- {%- if _item_type is not none
227
- and _item_type not in ('text', 'image', 'image_url', 'video') %}
228
- {{- raise_exception(
229
- 'Unexpected content item type ' ~ _item_type ~ '.'
230
- ) }}
231
- {%- endif %}
232
- {%- set _is_image =
233
- _item_type == 'image'
234
- or _item_type == 'image_url'
235
- or 'image' in item
236
- or 'image_url' in item
237
- %}
238
- {%- set _is_video =
239
- _item_type == 'video'
240
- or 'video' in item
241
- %}
242
- {%- if _is_image and _is_video %}
243
- {{- raise_exception(
244
- 'A content item cannot contain both image and video data.'
245
- ) }}
246
- {%- elif _item_type == 'text' and (_is_image or _is_video) %}
247
- {{- raise_exception(
248
- 'A text content item cannot also contain image or video data.'
249
- ) }}
250
- {%- elif _is_image %}
251
- {%- if is_system_content %}
252
- {{- raise_exception(
253
- 'System and developer messages cannot contain images.'
254
- ) }}
255
- {%- endif %}
256
- {%- if do_vision_count %}
257
- {%- set image_count.value = image_count.value + 1 %}
258
- {%- endif %}
259
- {%- if add_vision_id %}
260
- {{- 'Picture ' ~ image_count.value ~ ': ' }}
261
- {%- endif %}
262
- {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
263
- {%- elif _is_video %}
264
- {%- if is_system_content %}
265
- {{- raise_exception(
266
- 'System and developer messages cannot contain videos.'
267
- ) }}
268
- {%- endif %}
269
- {%- if do_vision_count %}
270
- {%- set video_count.value = video_count.value + 1 %}
271
- {%- endif %}
272
- {%- if add_vision_id %}
273
- {{- 'Video ' ~ video_count.value ~ ': ' }}
274
- {%- endif %}
275
- {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
276
- {%- elif _item_type == 'text' or 'text' in item %}
277
- {%- if 'text' not in item or item.text is not string %}
278
- {{- raise_exception(
279
- 'Text content items must contain a string text field.'
280
- ) }}
281
- {%- endif %}
282
- {{- item.text }}
283
- {%- else %}
284
- {{- raise_exception(
285
- 'Unexpected item in content list. Expected text, image, or video content.'
286
- ) }}
287
- {%- endif %}
288
- {%- endfor %}
289
- {%- elif content is none or content is undefined %}
290
- {{- '' }}
291
- {%- else %}
292
- {{- raise_exception('Unexpected content type.') }}
293
- {%- endif %}
294
- {%- endmacro %}
295
- {#- -------------------------------------------------------------------------
296
- Validate and merge leading system/developer messages
297
- ------------------------------------------------------------------------- -#}
298
- {%- set head = namespace(count=0, seen_non_system=false) %}
299
- {%- for message in messages %}
300
- {%- set _is_system =
301
- message.role == 'system' or message.role == 'developer'
302
- %}
303
- {%- if _is_system %}
304
- {%- if head.seen_non_system %}
305
- {{- raise_exception(
306
- 'System and developer messages must appear before all user, assistant, and tool messages.'
307
- ) }}
308
- {%- endif %}
309
- {%- set head.count = head.count + 1 %}
310
- {%- else %}
311
- {%- set head.seen_non_system = true %}
312
- {%- endif %}
313
- {%- endfor %}
314
- {%- set system_state = namespace(content='') %}
315
- {%- for message in messages[:head.count] %}
316
- {%- set _part =
317
- render_content(message.content, false, true) | trim
318
- %}
319
- {%- if _part %}
320
- {%- if system_state.content %}
321
- {%- set system_state.content =
322
- system_state.content ~ '\n\n' ~ _part
323
- %}
324
- {%- else %}
325
- {%- set system_state.content = _part %}
326
- {%- endif %}
327
- {%- endif %}
328
- {%- endfor %}
329
- {%- set _system_content = system_state.content %}
330
- {%- set _msgs = messages[head.count:] %}
331
- {#- -------------------------------------------------------------------------
332
- System prompt and tool instructions
333
- ------------------------------------------------------------------------- -#}
334
- {%- if _has_tools %}
335
- {{- '<|im_start|>system\n' }}
336
- {%- if reasoning_instructions %}
337
- {{- reasoning_instructions ~ '\n\n' }}
338
- {%- endif %}
339
- {{- '# Tools\n\nYou have access to the following functions:\n\n<tools>' }}
340
- {%- for tool in tools %}
341
- {{- '\n' }}
342
- {{- tool | tojson }}
343
- {%- endfor %}
344
- {{- '\n</tools>' }}
345
- {%- if _tool_format == 'json' %}
346
- {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n{"name": "example_function_name", "arguments": {"example_parameter_1": "value_1", "example_parameter_2": "This is the value for the second parameter"}}\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: a single JSON object with "name" and "arguments" keys must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
347
- {%- else %}
348
- {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
349
- {%- endif %}
350
- {%- if _system_content %}
351
- {{- '\n\n' ~ _system_content }}
352
- {%- endif %}
353
- {{- '<|im_end|>\n' }}
354
- {%- else %}
355
- {%- if _system_content %}
356
- {{- '<|im_start|>system\n'
357
- ~ (reasoning_instructions ~ '\n\n' if reasoning_instructions else '')
358
- ~ _system_content
359
- ~ '<|im_end|>\n'
360
- }}
361
- {%- elif reasoning_instructions %}
362
- {{- '<|im_start|>system\n'
363
- ~ reasoning_instructions
364
- ~ '<|im_end|>\n'
365
- }}
366
- {%- endif %}
367
- {%- endif %}
368
- {#- -------------------------------------------------------------------------
369
- Find the last real user query
370
- ------------------------------------------------------------------------- -#}
371
- {%- set ns = namespace(
372
- multi_step_tool=true,
373
- last_query_index=(_msgs | length) - 1
374
- ) %}
375
- {%- for message in _msgs[::-1] %}
376
- {%- set index = (_msgs | length - 1) - loop.index0 %}
377
- {%- if ns.multi_step_tool and message.role == 'user' %}
378
- {%- set _rendered_user =
379
- render_content(message.content, false) | trim
380
- %}
381
- {%- if not (
382
- _rendered_user.startswith('<tool_response>')
383
- and _rendered_user.endswith('</tool_response>')
384
- ) %}
385
- {%- set ns.multi_step_tool = false %}
386
- {%- set ns.last_query_index = index %}
387
- {%- endif %}
388
- {%- endif %}
389
- {%- endfor %}
390
- {%- if ns.multi_step_tool %}
391
- {{- raise_exception('No user query found in messages.') }}
392
- {%- endif %}
393
- {#- -------------------------------------------------------------------------
394
- Conversation rendering
395
- ------------------------------------------------------------------------- -#}
396
- {%- set history_state = namespace(previous_role='') %}
397
- {%- for message in _msgs %}
398
- {%- set content =
399
- render_content(message.content, true) | trim
400
- %}
401
- {%- if message.role == 'user' %}
402
- {{- '<|im_start|>user\n'
403
- ~ content
404
- ~ '<|im_end|>\n'
405
- }}
406
- {%- elif message.role == 'assistant' %}
407
- {%- set reasoning_content = '' %}
408
- {#- Prefer current vLLM's `reasoning` field, then retain compatibility
409
- with older `reasoning_content` and other servers' `thinking`. -#}
410
- {%- if message.reasoning is defined
411
- and message.reasoning is not none %}
412
- {%- if message.reasoning is string %}
413
- {%- set reasoning_content = message.reasoning %}
414
- {%- else %}
415
- {%- set reasoning_content =
416
- message.reasoning | string
417
- %}
418
- {%- endif %}
419
- {%- elif message.reasoning_content is defined
420
- and message.reasoning_content is not none %}
421
- {%- if message.reasoning_content is string %}
422
- {%- set reasoning_content = message.reasoning_content %}
423
- {%- else %}
424
- {%- set reasoning_content =
425
- message.reasoning_content | string
426
- %}
427
- {%- endif %}
428
- {%- elif message.thinking is defined
429
- and message.thinking is not none %}
430
- {%- if message.thinking is string %}
431
- {%- set reasoning_content = message.thinking %}
432
- {%- else %}
433
- {%- set reasoning_content =
434
- message.thinking | string
435
- %}
436
- {%- endif %}
437
- {%- endif %}
438
- {%- set reasoning_content = reasoning_content | trim %}
439
- {#- Preserve the original invariant: when thinking is preserved,
440
- always reconstruct the thinking block, even when it is empty. -#}
441
- {%- if _preserve_thinking
442
- or loop.index0 > ns.last_query_index %}
443
- {{- '<|im_start|>assistant\n<think>\n'
444
- ~ reasoning_content
445
- ~ '\n</think>\n\n'
446
- ~ content
447
- }}
448
- {%- else %}
449
- {{- '<|im_start|>assistant\n' ~ content }}
450
- {%- endif %}
451
- {%- if message.tool_calls is defined
452
- and message.tool_calls is not none
453
- and message.tool_calls %}
454
- {%- if message.tool_calls is string
455
- or message.tool_calls is mapping
456
- or message.tool_calls is not iterable %}
457
- {{- raise_exception(
458
- 'assistant.tool_calls must be an iterable list of tool calls.'
459
- ) }}
460
- {%- endif %}
461
- {%- for tool_call in message.tool_calls %}
462
- {%- if tool_call.function is defined
463
- and tool_call.function is not none %}
464
- {%- set tc = tool_call.function %}
465
- {%- else %}
466
- {%- set tc = tool_call %}
467
- {%- endif %}
468
- {%- if tc.name is not defined
469
- or tc.name is none
470
- or tc.name is not string
471
- or not tc.name %}
472
- {{- raise_exception(
473
- 'Every tool call must have a non-empty string name.'
474
- ) }}
475
- {%- endif %}
476
- {%- set tc_name = tc.name %}
477
- {%- if _tool_format == 'json' %}
478
- {%- if loop.first %}
479
- {%- if content | trim %}
480
- {{- '\n\n<tool_call>\n' }}
481
- {%- else %}
482
- {{- '<tool_call>\n' }}
483
- {%- endif %}
484
- {%- else %}
485
- {{- '\n<tool_call>\n' }}
486
- {%- endif %}
487
- {%- set _json_arguments = '{}' %}
488
- {%- if tc.arguments is defined
489
- and tc.arguments is not none %}
490
- {%- if tc.arguments is mapping %}
491
- {%- set _json_arguments =
492
- tc.arguments | tojson
493
- %}
494
- {%- elif tc.arguments is string %}
495
- {%- set _raw_arguments =
496
- tc.arguments | trim
497
- %}
498
- {%- if _raw_arguments %}
499
- {%- if not (
500
- _raw_arguments.startswith('{')
501
- and _raw_arguments.endswith('}')
502
- ) %}
503
- {{- raise_exception(
504
- 'JSON tool-call arguments must be a JSON object string.'
505
- ) }}
506
- {%- endif %}
507
- {%- set _json_arguments =
508
- _raw_arguments
509
- %}
510
- {%- endif %}
511
- {%- else %}
512
- {{- raise_exception(
513
- 'JSON tool-call arguments must be a mapping or a JSON object string.'
514
- ) }}
515
- {%- endif %}
516
- {%- endif %}
517
- {{- '{"name": ' }}
518
- {{- tc_name | tojson }}
519
- {{- ', "arguments": ' }}
520
- {{- _json_arguments }}
521
- {{- '}\n</tool_call>' }}
522
- {%- else %}
523
- {%- if loop.first %}
524
- {%- if content | trim %}
525
- {{- '\n\n<tool_call>\n<function='
526
- ~ tc_name
527
- ~ '>\n'
528
- }}
529
- {%- else %}
530
- {{- '<tool_call>\n<function='
531
- ~ tc_name
532
- ~ '>\n'
533
- }}
534
- {%- endif %}
535
- {%- else %}
536
- {{- '\n<tool_call>\n<function='
537
- ~ tc_name
538
- ~ '>\n'
539
- }}
540
- {%- endif %}
541
- {%- if tc.arguments is defined
542
- and tc.arguments is not none %}
543
- {%- if tc.arguments is mapping %}
544
- {%- for args_name, args_value
545
- in tc.arguments | items %}
546
- {%- set _argument_name =
547
- args_name | string
548
- %}
549
- {{- '<parameter='
550
- ~ _argument_name
551
- ~ '>\n'
552
- }}
553
- {#- Preserve strings verbatim. Serialize every
554
- other type as JSON so booleans and null stay
555
- true/false/null instead of True/False/None. -#}
556
- {%- if args_value is string %}
557
- {%- set _argument_value =
558
- args_value
559
- %}
560
- {%- else %}
561
- {%- set _argument_value =
562
- args_value | tojson
563
- %}
564
- {%- endif %}
565
- {%- if max_tool_arg_chars > 0
566
- and _argument_value | length
567
- > max_tool_arg_chars %}
568
- {{- _argument_value[:max_tool_arg_chars] }}
569
- {{- '\n[TRUNCATED — original length '
570
- ~ (_argument_value | length | string)
571
- ~ ' chars]'
572
- }}
573
- {%- else %}
574
- {{- _argument_value }}
575
- {%- endif %}
576
- {{- '\n</parameter>\n' }}
577
- {%- endfor %}
578
- {%- elif tc.arguments is string %}
579
- {%- set _raw_arguments =
580
- tc.arguments | trim
581
- %}
582
- {%- if _raw_arguments
583
- and _raw_arguments != '{}' %}
584
- {{- raise_exception(
585
- 'tool_call_format="xml" requires tool-call arguments to be mappings, but received a JSON string. Parse the JSON string into a mapping before applying the template, or set tool_call_format="auto" (or "json") to use the lossless JSON tool-call representation.'
586
- ) }}
587
- {%- endif %}
588
- {%- else %}
589
- {{- raise_exception(
590
- 'XML tool-call arguments must be a mapping. Parse JSON-string arguments before applying the template.'
591
- ) }}
592
- {%- endif %}
593
- {%- endif %}
594
- {{- '</function>\n</tool_call>' }}
595
- {%- endif %}
596
- {%- endfor %}
597
- {%- endif %}
598
- {{- '<|im_end|>\n' }}
599
- {%- elif message.role == 'tool' %}
600
- {%- if history_state.previous_role != 'tool' %}
601
- {{- '<|im_start|>user' }}
602
- {%- endif %}
603
- {%- if max_tool_response_chars > 0
604
- and content | length > max_tool_response_chars %}
605
- {%- set content =
606
- content[:max_tool_response_chars]
607
- ~ '\n[TRUNCATED — original length '
608
- ~ (content | length | string)
609
- ~ ' chars]'
610
- %}
611
- {%- endif %}
612
- {{- '\n<tool_response>\n' }}
613
- {{- content }}
614
- {{- '\n</tool_response>' }}
615
- {%- if loop.last %}
616
- {{- '<|im_end|>\n' }}
617
- {%- elif _msgs[loop.index0 + 1].role != 'tool' %}
618
- {{- '<|im_end|>\n' }}
619
- {%- endif %}
620
- {%- elif message.role == 'system'
621
- or message.role == 'developer' %}
622
- {{- raise_exception(
623
- 'System and developer messages must appear only at the beginning of the conversation.'
624
- ) }}
625
- {%- else %}
626
- {{- raise_exception(
627
- 'Unexpected message role ' ~ message.role ~ '.'
628
- ) }}
629
- {%- endif %}
630
- {%- set history_state.previous_role = message.role %}
631
- {%- endfor %}
632
- {#- -------------------------------------------------------------------------
633
- Generation prompt
634
- ------------------------------------------------------------------------- -#}
635
- {%- if add_generation_prompt %}
636
- {{- '<|im_start|>assistant\n' }}
637
- {%- if not _thinking_enabled %}
638
- {{- '<think>\n\n</think>\n\n' }}
639
- {%- else %}
640
- {{- '<think>\n' }}
641
- {%- endif %}
642
- {%- endif %}
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
examples/README.md CHANGED
@@ -1,13 +1,13 @@
1
- # Janus-27B examples
2
 
3
  Four minimal entry points. Pick the one that matches how you run models.
4
 
5
  | File | Backend | When to use |
6
  |---|---|---|
7
  | `ollama_chat.py` | Ollama HTTP API | You already have `ollama serve` running and the `janus` model created from the project `Modelfile`. **Text + tool calling** — vision via Ollama is broken upstream for this arch. |
8
- | `transformers_quickstart.py` | Hugging Face Transformers | You want to run the Heretic safetensors (`llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved`) on GPU, optionally in 4-bit via bitsandbytes. |
9
  | `llama_cpp_quickstart.py` | llama-cpp-python | You want to invoke a local GGUF directly without a daemon (CI, batch jobs, scripts). Text only. |
10
- | `llama_cpp_vision.py` | llama-cpp-python + mmproj | **Image input.** Loads a text GGUF + `Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-mmproj-BF16.gguf` and answers questions about an image. The only working vision path right now. |
11
 
12
  The text clients apply the same Janus system prompt; `llama_cpp_vision.py` uses a task-specific vision prompt. The three non-Ollama scripts
13
  explicitly pin the model card's tighter-reasoning sampling profile
@@ -27,7 +27,7 @@ root-level `template` / `system` / `params` files via HF's Ollama
27
  bridge):
28
 
29
  ```bash
30
- ollama pull hf.co/FoolDev/Janus-35B-HERETIC # bundled Q4_K_M, ~17.3 GB
31
  pip install requests
32
  MODEL=hf.co/FoolDev/Janus-35B-HERETIC python ollama_chat.py
33
  ```
@@ -42,8 +42,9 @@ python ollama_chat.py # MODEL defaults to `jan
42
 
43
  For a non-bundled quant (e.g. Q3_K_M, Q5_K_M), `./scripts/build.sh`
44
  downloads it from
45
- `llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-GGUF`,
46
- runs it through `strip_mtp.py`, patches the `Modelfile` `FROM` line into a
 
47
  temp copy, and creates the local `janus` tag:
48
 
49
  ```bash
@@ -54,7 +55,7 @@ python ollama_chat.py
54
  ### Transformers (safetensors)
55
 
56
  ```bash
57
- pip install --upgrade transformers accelerate sentencepiece bitsandbytes # needs a build new enough for Qwen 3.8
58
  python transformers_quickstart.py # 4-bit (bitsandbytes)
59
  python transformers_quickstart.py --no-4bit # bf16
60
  ```
@@ -63,7 +64,7 @@ python transformers_quickstart.py --no-4bit # bf16
63
 
64
  ```bash
65
  pip install llama-cpp-python # CPU-only build
66
- python llama_cpp_quickstart.py /path/to/Janus-27B.Q4_K_M.gguf
67
  ```
68
 
69
  For GPU offload, rebuild llama-cpp-python with the matching backend (see
@@ -78,14 +79,14 @@ cd .. && ./scripts/fetch_vision.sh && cd examples
78
 
79
  pip install llama-cpp-python pillow
80
  python llama_cpp_vision.py \
81
- --gguf ../Janus-27B.Q4_K_M.gguf \
82
- --mmproj ../Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-mmproj-BF16.gguf \
83
  --image /path/to/photo.jpg \
84
  --prompt "Describe this image."
85
  ```
86
 
87
- Why not Ollama? Ollama's Go engine has the `qwen35` arch entry
88
- (text inference works in 0.24+), but the C++ llama.cpp fallback that
89
  Ollama switches to when an mmproj is attached still lacks it.
90
  `ollama create` accepts the dual-`FROM` and `ollama show` reports
91
  `vision` capability, but the first inference call fails with an
 
1
+ # Janus-35B examples
2
 
3
  Four minimal entry points. Pick the one that matches how you run models.
4
 
5
  | File | Backend | When to use |
6
  |---|---|---|
7
  | `ollama_chat.py` | Ollama HTTP API | You already have `ollama serve` running and the `janus` model created from the project `Modelfile`. **Text + tool calling** — vision via Ollama is broken upstream for this arch. |
8
+ | `transformers_quickstart.py` | Hugging Face Transformers | You want to run the Heretic safetensors (`llmfan46/Qwen3.6-35B-A3B-uncensored-heretic`) on GPU, optionally in 4-bit via bitsandbytes. |
9
  | `llama_cpp_quickstart.py` | llama-cpp-python | You want to invoke a local GGUF directly without a daemon (CI, batch jobs, scripts). Text only. |
10
+ | `llama_cpp_vision.py` | llama-cpp-python + mmproj | **Image input.** Loads a text GGUF + `Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf` and answers questions about an image. The only working vision path right now. |
11
 
12
  The text clients apply the same Janus system prompt; `llama_cpp_vision.py` uses a task-specific vision prompt. The three non-Ollama scripts
13
  explicitly pin the model card's tighter-reasoning sampling profile
 
27
  bridge):
28
 
29
  ```bash
30
+ ollama pull hf.co/FoolDev/Janus-35B-HERETIC # bundled Q4_K_M, 21.23 GB
31
  pip install requests
32
  MODEL=hf.co/FoolDev/Janus-35B-HERETIC python ollama_chat.py
33
  ```
 
42
 
43
  For a non-bundled quant (e.g. Q3_K_M, Q5_K_M), `./scripts/build.sh`
44
  downloads it from
45
+ `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF`,
46
+ passes it through `strip_mtp.py` (a no-op on those quants — they ship
47
+ MTP-clean), patches the `Modelfile` `FROM` line into a
48
  temp copy, and creates the local `janus` tag:
49
 
50
  ```bash
 
55
  ### Transformers (safetensors)
56
 
57
  ```bash
58
+ pip install --upgrade transformers accelerate sentencepiece bitsandbytes # needs a build new enough for Qwen 3.6
59
  python transformers_quickstart.py # 4-bit (bitsandbytes)
60
  python transformers_quickstart.py --no-4bit # bf16
61
  ```
 
64
 
65
  ```bash
66
  pip install llama-cpp-python # CPU-only build
67
+ python llama_cpp_quickstart.py /path/to/Janus-35B-A3B.Q4_K_M.gguf
68
  ```
69
 
70
  For GPU offload, rebuild llama-cpp-python with the matching backend (see
 
79
 
80
  pip install llama-cpp-python pillow
81
  python llama_cpp_vision.py \
82
+ --gguf ../Janus-35B-A3B.Q4_K_M.gguf \
83
+ --mmproj ../Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf \
84
  --image /path/to/photo.jpg \
85
  --prompt "Describe this image."
86
  ```
87
 
88
+ Why not Ollama? Ollama's Go engine has the `qwen35moe` arch entry
89
+ (text inference checked in source at 0.33.3, 0.34.0 and 0.34.2), but the C++ llama.cpp fallback that
90
  Ollama switches to when an mmproj is attached still lacks it.
91
  `ollama create` accepts the dual-`FROM` and `ollama show` reports
92
  `vision` capability, but the first inference call fails with an
examples/llama_cpp_quickstart.py CHANGED
@@ -1,6 +1,6 @@
1
  #!/usr/bin/env python3
2
  """
3
- Janus-27B — llama-cpp-python quickstart.
4
 
5
  Skip Ollama entirely and call the GGUF directly through llama-cpp-python.
6
  Useful for batch jobs, CI, or environments where you don't want a daemon.
@@ -14,7 +14,7 @@ For GPU offload (CUDA / Metal / ROCm), install with the matching extras:
14
  CMAKE_ARGS="-DGGML_HIPBLAS=on" pip install llama-cpp-python --no-binary :all:
15
 
16
  Usage:
17
- python llama_cpp_quickstart.py /path/to/Janus-27B.Q4_K_M.gguf
18
  python llama_cpp_quickstart.py /path/to/file.gguf --gpu-layers 99
19
  python llama_cpp_quickstart.py /path/to/file.gguf --prompt "..."
20
  """
@@ -49,7 +49,7 @@ JANUS_SYSTEM = (
49
 
50
  def main() -> None:
51
  ap = argparse.ArgumentParser()
52
- ap.add_argument("gguf", help="Path to Janus-27B GGUF (e.g. Q4_K_M).")
53
  ap.add_argument(
54
  "--prompt",
55
  default="Explain the Burrows-Wheeler transform in 200 words.",
 
1
  #!/usr/bin/env python3
2
  """
3
+ Janus-35B — llama-cpp-python quickstart.
4
 
5
  Skip Ollama entirely and call the GGUF directly through llama-cpp-python.
6
  Useful for batch jobs, CI, or environments where you don't want a daemon.
 
14
  CMAKE_ARGS="-DGGML_HIPBLAS=on" pip install llama-cpp-python --no-binary :all:
15
 
16
  Usage:
17
+ python llama_cpp_quickstart.py /path/to/Janus-35B-A3B.Q4_K_M.gguf
18
  python llama_cpp_quickstart.py /path/to/file.gguf --gpu-layers 99
19
  python llama_cpp_quickstart.py /path/to/file.gguf --prompt "..."
20
  """
 
49
 
50
  def main() -> None:
51
  ap = argparse.ArgumentParser()
52
+ ap.add_argument("gguf", help="Path to Janus-35B-A3B GGUF (e.g. Q4_K_M).")
53
  ap.add_argument(
54
  "--prompt",
55
  default="Explain the Burrows-Wheeler transform in 200 words.",
examples/llama_cpp_vision.py CHANGED
@@ -1,9 +1,9 @@
1
  #!/usr/bin/env python3
2
  """
3
- Janus-27B — vision (image-text-to-text) via llama-cpp-python.
4
 
5
  Why this script exists:
6
- Ollama's Go engine has the qwen35 arch entry (text inference works
7
  on 0.24+), but the C++ llama.cpp fallback that Ollama switches to
8
  when an mmproj is attached still lacks it. Both `FROM mmproj.gguf`
9
  and `ADAPTER mmproj.gguf` fail at the first inference call with an
@@ -24,26 +24,26 @@ Install:
24
  # CMAKE_ARGS="-DGGML_HIPBLAS=on" pip install llama-cpp-python --no-binary :all:
25
 
26
  Files you need:
27
- 1. A text GGUF (any quant): the bundled Janus-27B.Q4_K_M.gguf (~17 GB),
28
- or another quant from
29
- llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-GGUF
30
- — run those through ./scripts/strip_mtp.py first; the source repo's
31
- quants are MTP-preserved and stock llama.cpp cannot load them.
32
  2. A vision projector:
33
- Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-mmproj-BF16.gguf
34
  — run ./scripts/fetch_vision.sh to pull it.
35
 
36
  Usage:
37
  python llama_cpp_vision.py \
38
- --gguf Janus-27B.Q4_K_M.gguf \
39
- --mmproj Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-mmproj-BF16.gguf \
40
  --image /path/to/photo.jpg \
41
  --prompt "What is in this image? Be specific."
42
 
43
  # CLI alternative without python binding (ships with llama.cpp):
44
  # llama-mtmd-cli \
45
- # -m Janus-27B.Q4_K_M.gguf \
46
- # --mmproj Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-mmproj-BF16.gguf \
47
  # --image photo.jpg \
48
  # -p "Describe this image."
49
  """
@@ -79,7 +79,7 @@ def encode_image_data_uri(path: Path) -> str:
79
 
80
  def main() -> None:
81
  ap = argparse.ArgumentParser()
82
- ap.add_argument("--gguf", required=True, help="Text GGUF (e.g. Janus-27B.Q4_K_M.gguf).")
83
  ap.add_argument("--mmproj", required=True, help="Vision projector GGUF (mmproj-BF16.gguf).")
84
  ap.add_argument("--image", required=True, help="Image to analyze.")
85
  ap.add_argument("--prompt", default="Describe this image in detail.")
@@ -98,7 +98,7 @@ def main() -> None:
98
  sys.exit(f"Image not found: {image_path}")
99
 
100
  # Qwen 2.5 VL chat handler is the closest match shipped with
101
- # llama-cpp-python; Qwen 3.5 / 3.6 / 3.8 vision uses the same projector
102
  # layout. If/when llama-cpp-python ships a Qwen3VLChatHandler, swap it in.
103
  handler = Qwen25VLChatHandler(clip_model_path=args.mmproj)
104
 
 
1
  #!/usr/bin/env python3
2
  """
3
+ Janus-35B — vision (image-text-to-text) via llama-cpp-python.
4
 
5
  Why this script exists:
6
+ Ollama's Go engine has the qwen35moe arch entry (text inference works
7
  on 0.24+), but the C++ llama.cpp fallback that Ollama switches to
8
  when an mmproj is attached still lacks it. Both `FROM mmproj.gguf`
9
  and `ADAPTER mmproj.gguf` fail at the first inference call with an
 
24
  # CMAKE_ARGS="-DGGML_HIPBLAS=on" pip install llama-cpp-python --no-binary :all:
25
 
26
  Files you need:
27
+ 1. A text GGUF (any quant): the bundled Janus-35B-A3B.Q4_K_M.gguf
28
+ (21.23 GB), or another quant from
29
+ llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF
30
+ — those ship MTP-clean, so no ./scripts/strip_mtp.py pass is needed;
31
+ only llmfan46's separate -Native-MTP-Preserved-GGUF variant needs it.
32
  2. A vision projector:
33
+ Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf
34
  — run ./scripts/fetch_vision.sh to pull it.
35
 
36
  Usage:
37
  python llama_cpp_vision.py \
38
+ --gguf Janus-35B-A3B.Q4_K_M.gguf \
39
+ --mmproj Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf \
40
  --image /path/to/photo.jpg \
41
  --prompt "What is in this image? Be specific."
42
 
43
  # CLI alternative without python binding (ships with llama.cpp):
44
  # llama-mtmd-cli \
45
+ # -m Janus-35B-A3B.Q4_K_M.gguf \
46
+ # --mmproj Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf \
47
  # --image photo.jpg \
48
  # -p "Describe this image."
49
  """
 
79
 
80
  def main() -> None:
81
  ap = argparse.ArgumentParser()
82
+ ap.add_argument("--gguf", required=True, help="Text GGUF (e.g. Janus-35B-A3B.Q4_K_M.gguf).")
83
  ap.add_argument("--mmproj", required=True, help="Vision projector GGUF (mmproj-BF16.gguf).")
84
  ap.add_argument("--image", required=True, help="Image to analyze.")
85
  ap.add_argument("--prompt", default="Describe this image in detail.")
 
98
  sys.exit(f"Image not found: {image_path}")
99
 
100
  # Qwen 2.5 VL chat handler is the closest match shipped with
101
+ # llama-cpp-python; Qwen 3.5 / 3.6 vision uses the same projector
102
  # layout. If/when llama-cpp-python ships a Qwen3VLChatHandler, swap it in.
103
  handler = Qwen25VLChatHandler(clip_model_path=args.mmproj)
104
 
examples/ollama_chat.py CHANGED
@@ -1,6 +1,6 @@
1
  #!/usr/bin/env python3
2
  """
3
- Janus-27B — Ollama chat examples.
4
 
5
  Prerequisites (pick one):
6
 
 
1
  #!/usr/bin/env python3
2
  """
3
+ Janus-35B — Ollama chat examples.
4
 
5
  Prerequisites (pick one):
6
 
examples/transformers_quickstart.py CHANGED
@@ -1,22 +1,24 @@
1
  #!/usr/bin/env python3
2
  """
3
- Janus-27B — Hugging Face Transformers quickstart.
4
 
5
- Loads the Heretic Qwen 3.8 27B safetensors directly and runs a single
6
- chat turn using its embedded chat template. Janus-27B is a *wrapper*
7
  around that base, so for the transformers route there is nothing to
8
  download from this repo — point at
9
- llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved
10
  and apply the same system prompt the Modelfile uses.
11
 
12
- Set MODEL_ID = "Qwen/Qwen3.8-27B" to bypass the Heretic abliteration and
13
  load the vanilla upstream base instead.
14
 
15
- Note: this is a dense 27B — every parameter runs on every token. There is no
16
- routing and no active-parameter discount: memory tracks the full 27B.
 
 
17
 
18
  Requirements:
19
- pip install --upgrade transformers accelerate sentencepiece bitsandbytes # needs a build new enough for Qwen 3.8
20
 
21
  Memory:
22
  - bf16 full precision: multi-GPU or CPU offload.
@@ -44,7 +46,7 @@ except ImportError as e: # pragma: no cover
44
  )
45
 
46
 
47
- MODEL_ID = "llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved"
48
 
49
  JANUS_SYSTEM = (
50
  "You are Janus, a precise and capable assistant for reasoning, writing, "
 
1
  #!/usr/bin/env python3
2
  """
3
+ Janus-35B — Hugging Face Transformers quickstart.
4
 
5
+ Loads the Heretic Qwen 3.6 35B-A3B safetensors directly and runs a single
6
+ chat turn using its embedded chat template. Janus-35B is a *wrapper*
7
  around that base, so for the transformers route there is nothing to
8
  download from this repo — point at
9
+ llmfan46/Qwen3.6-35B-A3B-uncensored-heretic
10
  and apply the same system prompt the Modelfile uses.
11
 
12
+ Set MODEL_ID = "Qwen/Qwen3.6-35B-A3B" to bypass the Heretic abliteration and
13
  load the vanilla upstream base instead.
14
 
15
+ Note: this is a mixture-of-experts model — ~34.7B parameters total, ~3B active
16
+ per token (derived from the bundled GGUF's header, 2026-09-18). Transformers
17
+ keeps all 256 experts resident, so memory tracks the ~34.7B total, not the 3B
18
+ active count.
19
 
20
  Requirements:
21
+ pip install --upgrade transformers accelerate sentencepiece bitsandbytes # needs a build new enough for Qwen 3.6
22
 
23
  Memory:
24
  - bf16 full precision: multi-GPU or CPU offload.
 
46
  )
47
 
48
 
49
+ MODEL_ID = "llmfan46/Qwen3.6-35B-A3B-uncensored-heretic"
50
 
51
  JANUS_SYSTEM = (
52
  "You are Janus, a precise and capable assistant for reasoning, writing, "
moe-routing.svg ADDED
scripts/bench.sh CHANGED
@@ -1,5 +1,5 @@
1
  #!/usr/bin/env bash
2
- # Janus-27B — tok/s benchmark via Ollama.
3
  #
4
  # Reads timing from Ollama's /api/chat response metadata (eval_count and
5
  # eval_duration are authoritative — no client-side stopwatch noise) and
 
1
  #!/usr/bin/env bash
2
+ # Janus-35B — tok/s benchmark via Ollama.
3
  #
4
  # Reads timing from Ollama's /api/chat response metadata (eval_count and
5
  # eval_duration are authoritative — no client-side stopwatch noise) and
scripts/build.sh CHANGED
@@ -1,5 +1,5 @@
1
  #!/usr/bin/env bash
2
- # Janus-27B — fetch a Qwen 3.8 27B GGUF and build the Ollama model.
3
  #
4
  # Usage:
5
  # ./scripts/build.sh # default: Q4_K_M
@@ -7,16 +7,15 @@
7
  # QUANT=Q6_K ./scripts/build.sh
8
  #
9
  # Skip the download by pointing at a GGUF you already have:
10
- # GGUF_PATH=/path/to/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-Q4_K_M.gguf ./scripts/build.sh Q4_K_M
11
  #
12
  # Note: a `<gguf>.stripped.gguf` sibling is created next to the source GGUF (the
13
- # MTP-stripped copy) and becomes ollama's FROM target. The upstream quants are
14
- # MTP-preserved, so that sibling is a real second file rather than a hardlink —
15
- # budget disk for both. It's gitignored in-repo; for an out-of-repo GGUF_PATH,
16
- # remove the sibling yourself.
17
  #
18
  # Requires: huggingface-cli (or hf), ollama, awk, python3 with the `gguf`
19
- # package (the MTP strip below is not optional).
20
  set -euo pipefail
21
 
22
  # Resolve a Hugging Face CLI. The bare name `hf` is ambiguous: Arch's `hf`
@@ -38,14 +37,14 @@ detect_hf() {
38
 
39
  QUANT="${1:-${QUANT:-Q4_K_M}}"
40
 
41
- REPO_ID="${REPO_ID:-llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-GGUF}"
42
- # Filenames at
43
- # llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-GGUF follow
44
- # Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-${QUANT}.gguf
45
- # Published quants: BF16, Q3_K_L, Q3_K_M, Q4_K_M, Q4_K_S, Q5_K_M, Q5_K_S, Q6_K,
46
- # Q8_0. Check the repo's Files tab if that list has moved on; override GGUF_NAME
47
- # or GGUF_PATH below if the naming differs for your chosen quant.
48
- GGUF_NAME="Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-${QUANT}.gguf"
49
  ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
50
  # GGUF_PATH defaults to ${ROOT}/${GGUF_NAME}, but can be overridden so users
51
  # with cached weights elsewhere don't have to copy or symlink anything.
@@ -95,34 +94,25 @@ if [[ ! -f "${GGUF_PATH}" ]]; then
95
  exit 1
96
  fi
97
 
98
- # ---- 2b. Strip the MTP/NextN layer ------------------------------------------
99
- # REQUIRED, not a safety net. The base is MTP-preserved, so the quants in the
100
- # source repo carry the MTP/NextN head as an extra block (index 64) that stock
101
- # llama.cpp / Ollama can't load (aborts on the missing SSM tensor). strip_mtp.py
102
- # drops that block and sets block_count back to 64; kept tensors are copied
103
- # byte-for-byte, so no re-quantization and no quality change. Same step the dense
104
- # sibling FoolDev/Thanatos-27B-HERETIC needs. If you want the MTP head, use the
105
- # upstream safetensors under vLLM / SGLang, or the unstripped GGUF straight from
106
- # the source repo — not this path.
 
 
107
  STRIPPED_PATH="${GGUF_PATH%.gguf}.stripped.gguf"
108
- # Stamp this repo's chat template into the GGUF in the same pass. llama.cpp,
109
- # LM Studio, Jan and KoboldCpp all read the template embedded in the GGUF, and
110
- # the upstream one raises on reasoning_effort values such as "max" and on
111
- # conflicting preserve_thinking / preserve_reasoning, which llama-server returns
112
- # as HTTP 500. Stamping it here means those loaders get this repo's template
113
- # with no extra flags. Skipped if the file is absent.
114
- STAMP_ARGS=()
115
- if [[ -f "${ROOT}/chat_template.jinja" ]]; then
116
- STAMP_ARGS=(--chat-template "${ROOT}/chat_template.jinja")
117
- echo "[*] stamping chat_template.jinja into the built GGUF"
118
- fi
119
- python3 "${ROOT}/scripts/strip_mtp.py" "${GGUF_PATH}" "${STRIPPED_PATH}" ${STAMP_ARGS[@]+"${STAMP_ARGS[@]}"}
120
  GGUF_PATH="${STRIPPED_PATH}"
121
  echo "[*] gguf (post-strip): ${GGUF_PATH}"
122
 
123
  # ---- 3. Patch the Modelfile FROM line in a temp copy -------------------------
124
 
125
- TMP_MODELFILE="$(mktemp -t janus27b-modelfile.XXXXXX)"
126
  trap 'rm -f "${TMP_MODELFILE}"' EXIT
127
  awk -v p="${GGUF_PATH}" '
128
  /^FROM[[:space:]]/ && !done { print "FROM " p; done=1; next }
 
1
  #!/usr/bin/env bash
2
+ # Janus-35B — fetch a Qwen 3.6 35B-A3B GGUF and build the Ollama model.
3
  #
4
  # Usage:
5
  # ./scripts/build.sh # default: Q4_K_M
 
7
  # QUANT=Q6_K ./scripts/build.sh
8
  #
9
  # Skip the download by pointing at a GGUF you already have:
10
+ # GGUF_PATH=/path/to/Qwen3.6-35B-A3B-uncensored-heretic-Q4_K_M.gguf ./scripts/build.sh Q4_K_M
11
  #
12
  # Note: a `<gguf>.stripped.gguf` sibling is created next to the source GGUF (the
13
+ # MTP-stripped copy — or, for these quants, a hardlink no-op, so no extra disk)
14
+ # and becomes ollama's FROM target. It's gitignored in-repo; for an out-of-repo
15
+ # GGUF_PATH, remove the sibling yourself.
 
16
  #
17
  # Requires: huggingface-cli (or hf), ollama, awk, python3 with the `gguf`
18
+ # package (the MTP strip below still runs, defensively).
19
  set -euo pipefail
20
 
21
  # Resolve a Hugging Face CLI. The bare name `hf` is ambiguous: Arch's `hf`
 
37
 
38
  QUANT="${1:-${QUANT:-Q4_K_M}}"
39
 
40
+ REPO_ID="${REPO_ID:-llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF}"
41
+ # Filenames at llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF follow
42
+ # Qwen3.6-35B-A3B-uncensored-heretic-${QUANT}.gguf
43
+ # Published quants (HF API, 2026-09-18): BF16, Q3_K_L, Q3_K_M, Q4_K_M, Q4_K_S,
44
+ # Q5_K_M, Q5_K_S, Q6_K, Q8_0, plus the BF16 mmproj (scripts/fetch_vision.sh).
45
+ # Check the repo's Files tab if that list has moved on; override GGUF_NAME or
46
+ # GGUF_PATH below if the naming differs for your chosen quant.
47
+ GGUF_NAME="Qwen3.6-35B-A3B-uncensored-heretic-${QUANT}.gguf"
48
  ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
49
  # GGUF_PATH defaults to ${ROOT}/${GGUF_NAME}, but can be overridden so users
50
  # with cached weights elsewhere don't have to copy or symlink anything.
 
94
  exit 1
95
  fi
96
 
97
+ # ---- 2b. Strip the MTP/NextN layer if present -------------------------------
98
+ # Defensive, not a required build step. The quants in the source repo are
99
+ # MTP-clean — Q4_K_M read from its GGUF header on 2026-09-18: block_count 40,
100
+ # blk.0…blk.39 only, no nextn_predict_layers key — so strip_mtp.py finds nothing
101
+ # to drop and passes the file through as a hardlink (no extra disk, no
102
+ # re-quantization). The call stays because llmfan46 also publishes a separate
103
+ # *-Native-MTP-Preserved-GGUF variant whose Q4_K_M has block_count 41 with a
104
+ # NextN block at index 40; point GGUF_PATH at one of those and this step is what
105
+ # makes it loadable on stock llama.cpp / Ollama. No --chat-template here: the
106
+ # GGUF carries the base's own embedded template and this repo ships no
107
+ # chat_template.jinja to stamp over it.
108
  STRIPPED_PATH="${GGUF_PATH%.gguf}.stripped.gguf"
109
+ python3 "${ROOT}/scripts/strip_mtp.py" "${GGUF_PATH}" "${STRIPPED_PATH}"
 
 
 
 
 
 
 
 
 
 
 
110
  GGUF_PATH="${STRIPPED_PATH}"
111
  echo "[*] gguf (post-strip): ${GGUF_PATH}"
112
 
113
  # ---- 3. Patch the Modelfile FROM line in a temp copy -------------------------
114
 
115
+ TMP_MODELFILE="$(mktemp -t janus35b-modelfile.XXXXXX)"
116
  trap 'rm -f "${TMP_MODELFILE}"' EXIT
117
  awk -v p="${GGUF_PATH}" '
118
  /^FROM[[:space:]]/ && !done { print "FROM " p; done=1; next }
scripts/cap_ctx.sh CHANGED
@@ -1,11 +1,12 @@
1
  #!/usr/bin/env bash
2
- # Janus-27B — create a small-context local tag for OpenAI /v1 clients.
3
  #
4
  # Ollama's /v1/chat/completions (OpenAI-compatible) silently ignores the
5
- # `options` field, so it loads at the baked default `num_ctx 262144` — ~34 GiB
6
- # total with Ollama's default f16 KV cache, which is too much for a tight host. This bakes a local tag with
7
- # num_ctx overridden to a loadable value (and num_batch 256), which any OpenAI
8
- # /v1 client can point at.
 
9
  # Reuses this repo's Modelfile verbatim (TEMPLATE / SYSTEM / stop params), only
10
  # swapping the FROM line to the bundled blob and the num_ctx parameter.
11
  #
@@ -21,7 +22,7 @@ CTX="${CTX:-4096}"
21
  NBATCH="${NBATCH:-256}"
22
  ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
23
  MODELFILE="${ROOT}/Modelfile"
24
- GGUF="${GGUF:-${ROOT}/Janus-27B.Q4_K_M.gguf}"
25
  TAG="${TAG:-janus}"
26
 
27
  if ! command -v ollama >/dev/null 2>&1; then echo "[!] ollama not found in PATH" >&2; exit 1; fi
@@ -43,7 +44,7 @@ echo "[*] tag: ${TAG}"
43
  echo "[*] gguf: ${GGUF}"
44
  echo "[*] num_ctx: ${CTX} num_batch: ${NBATCH}"
45
 
46
- TMP="$(mktemp -t janus27b-capctx.XXXXXX)"
47
  trap 'rm -f "${TMP}"' EXIT
48
  awk -v g="${GGUF}" -v c="${CTX}" -v b="${NBATCH}" '
49
  /^FROM[[:space:]]/ && !done { print "FROM " g; done=1; next }
 
1
  #!/usr/bin/env bash
2
+ # Janus-35B — create a small-context local tag for OpenAI /v1 clients.
3
  #
4
  # Ollama's /v1/chat/completions (OpenAI-compatible) silently ignores the
5
+ # `options` field, so it loads at the baked default `num_ctx 262144` — the
6
+ # 21.23 GB of weights plus a 262K f16 KV cache, which is too much for a tight
7
+ # host (see the README's Hardware requirements for the figures). This bakes a
8
+ # local tag with num_ctx overridden to a loadable value (and num_batch 256),
9
+ # which any OpenAI /v1 client can point at.
10
  # Reuses this repo's Modelfile verbatim (TEMPLATE / SYSTEM / stop params), only
11
  # swapping the FROM line to the bundled blob and the num_ctx parameter.
12
  #
 
22
  NBATCH="${NBATCH:-256}"
23
  ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
24
  MODELFILE="${ROOT}/Modelfile"
25
+ GGUF="${GGUF:-${ROOT}/Janus-35B-A3B.Q4_K_M.gguf}"
26
  TAG="${TAG:-janus}"
27
 
28
  if ! command -v ollama >/dev/null 2>&1; then echo "[!] ollama not found in PATH" >&2; exit 1; fi
 
44
  echo "[*] gguf: ${GGUF}"
45
  echo "[*] num_ctx: ${CTX} num_batch: ${NBATCH}"
46
 
47
+ TMP="$(mktemp -t janus35b-capctx.XXXXXX)"
48
  trap 'rm -f "${TMP}"' EXIT
49
  awk -v g="${GGUF}" -v c="${CTX}" -v b="${NBATCH}" '
50
  /^FROM[[:space:]]/ && !done { print "FROM " g; done=1; next }
scripts/check.sh CHANGED
@@ -1,5 +1,5 @@
1
  #!/usr/bin/env bash
2
- # Janus-27B — repo-local sanity checks.
3
  #
4
  # Runs everything that's cheap and catches a real-world bug we've already hit:
5
  #
@@ -7,15 +7,16 @@
7
  # 2. shellcheck on every *.sh, if installed (catches quoting/SC2086 bugs)
8
  # 3. python3 -m pyflakes on every *.py (catches NameError, unused imports)
9
  # 4. python3 -m py_compile on every *.py (catches actual syntax errors)
10
- # 5. grep -E for a FROM/BUNDLE path pointing at an unstripped
11
- # -MTP-Preserved- quant (this base keeps its MTP head; see strip_mtp.py)
12
  # 6. multi-line-aware scan for the VAR="$(cmd 2>/dev/null | filter)" silent-exit
13
  # under set -e + pipefail (the one we shipped in d87bc64 and fixed in 385ed94)
14
  # 7. Modelfile <-> template/system/params bridge-file sync
15
- # 8. chat_template.jinja render checks (the tool-call shape llama.cpp needs)
16
- # 9. Go template keeps Ollama's thinking detection and the replay condition
17
  #
18
- # Exit non-zero on any failure. Designed to run from .git/hooks/pre-commit.
 
 
19
  #
20
  # Usage:
21
  # ./scripts/check.sh # one-shot
@@ -31,6 +32,8 @@ yellow() { printf "\033[33m%s\033[0m\n" "$*"; }
31
  blue() { printf "\033[34m%s\033[0m\n" "$*"; }
32
 
33
  FAIL=0
 
 
34
 
35
  # ---- collect targets -------------------------------------------------------
36
 
@@ -69,6 +72,7 @@ if command -v shellcheck >/dev/null 2>&1; then
69
  done
70
  else
71
  yellow "[~] shellcheck not installed (skip). Install with: apt install shellcheck"
 
72
  fi
73
 
74
  # ---- 3. pyflakes -----------------------------------------------------------
@@ -86,6 +90,7 @@ if (( ${#PY_FILES[@]} )); then
86
  done
87
  else
88
  yellow "[~] pyflakes not installed (skip). Install with: pip install pyflakes"
 
89
  fi
90
  fi
91
 
@@ -105,28 +110,44 @@ if (( ${#PY_FILES[@]} )); then
105
  find . -type d -name __pycache__ -not -path './.git/*' -exec rm -rf {} + 2>/dev/null || true
106
  fi
107
 
108
- # ---- 5. footgun: MTP-unstripped bundle reference ---------------------------
109
  #
110
- # The Janus base is MTP-PRESERVED: quants pulled from
111
- # llmfan46/...-MTP-Preserved-GGUF carry a NextN block at index 64 that
112
- # stock llama.cpp and Ollama cannot load. Every bundled/committed GGUF must
113
- # go through scripts/strip_mtp.py first. Block any doc or script that points
114
- # a FROM / bundle path straight at an -MTP-Preserved- quant.
 
 
115
  #
116
- # (Note: unlike the Thanatos sibling, DASH-separated quant filenames are
117
- # correct here — llmfan46 names them
118
- # Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-Q4_K_M.gguf —
119
- # so there is no dot-vs-dash guard in this repo.)
120
-
121
- blue "[*] grep: bundled FROM/BUNDLE pointing at an unstripped -MTP-Preserved- quant"
122
- if grep -RnE '^(FROM|BUNDLE=|GGUF=)[^#]*MTP-Preserved-Q[0-9A-Z_]+\.gguf' \
123
- --include='*.sh' --include='Modelfile*' \
124
- --exclude-dir=.git --exclude-dir=__pycache__ \
125
- . ; then
126
- red " [FAIL] bundle path points at an unstripped MTP quant (run scripts/strip_mtp.py)"
127
- FAIL=1
128
  else
129
- echo " [ ok ] no unstripped-MTP bundle references"
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
130
  fi
131
 
132
  # ---- 6. footgun: VAR="$(cmd 2>/dev/null | filter)" silent-exit pattern -----
@@ -198,28 +219,10 @@ if [[ -f "${ROOT}/Modelfile" && -f "${ROOT}/template" \
198
  fi
199
  else
200
  yellow "[~] bridge files missing; skipping sync check"
 
201
  fi
202
 
203
- # ---- 8. chat_template.jinja render checks ----------------------------------
204
- #
205
- # 'chat_template.jinja' is what llama.cpp / llama-server applies under
206
- # '--jinja --chat-template-file'. It is data, not code, so nothing else in this
207
- # file looks at it: py_compile and pyflakes skip it, and a one-character edit
208
- # can silently switch the tool-call format away from the XML form llama-server's
209
- # parser reads, or bring back a raise llama-server returns as HTTP 500.
210
- # check_chat_template.py renders it and asserts the properties tool calling,
211
- # reasoning replay and reasoning_effort depend on.
212
-
213
- if [[ -f "${ROOT}/chat_template.jinja" ]]; then
214
- blue "[*] python: chat_template.jinja render checks"
215
- if ! python3 "${ROOT}/scripts/check_chat_template.py"; then
216
- FAIL=1
217
- fi
218
- else
219
- yellow "[~] chat_template.jinja missing; skipping render checks"
220
- fi
221
-
222
- # ---- 9. Go template: Ollama's thinking detection + replay condition ------
223
  #
224
  # Ollama picks between the Go 'template' and the GGUF's embedded Jinja template
225
  # by comparing their capabilities, and infers the Go template's "thinking" from
@@ -234,6 +237,7 @@ if [[ -f "${ROOT}/template" ]]; then
234
  fi
235
  else
236
  yellow "[~] template missing; skipping Go template check"
 
237
  fi
238
 
239
  # ---- result ----------------------------------------------------------------
@@ -243,4 +247,8 @@ if (( FAIL )); then
243
  red "[!] FAIL"
244
  exit 1
245
  fi
246
- green "[+] all checks passed"
 
 
 
 
 
1
  #!/usr/bin/env bash
2
+ # Janus-35B — repo-local sanity checks.
3
  #
4
  # Runs everything that's cheap and catches a real-world bug we've already hit:
5
  #
 
7
  # 2. shellcheck on every *.sh, if installed (catches quoting/SC2086 bugs)
8
  # 3. python3 -m pyflakes on every *.py (catches NameError, unused imports)
9
  # 4. python3 -m py_compile on every *.py (catches actual syntax errors)
10
+ # 5. the Modelfile's FROM target is the GGUF this repo actually ships
11
+ # (the local-build path da34705 broke, with nothing here to catch it)
12
  # 6. multi-line-aware scan for the VAR="$(cmd 2>/dev/null | filter)" silent-exit
13
  # under set -e + pipefail (the one we shipped in d87bc64 and fixed in 385ed94)
14
  # 7. Modelfile <-> template/system/params bridge-file sync
15
+ # 8. Go template keeps Ollama's thinking detection and the replay condition
 
16
  #
17
+ # Exit non-zero on any failure; a check whose tooling or input is missing is
18
+ # reported as skipped and never counted as a pass. Designed to run from
19
+ # .git/hooks/pre-commit.
20
  #
21
  # Usage:
22
  # ./scripts/check.sh # one-shot
 
32
  blue() { printf "\033[34m%s\033[0m\n" "$*"; }
33
 
34
  FAIL=0
35
+ SKIPPED=0
36
+ CHECKS=8 # the numbered checks above; keep in sync with that list
37
 
38
  # ---- collect targets -------------------------------------------------------
39
 
 
72
  done
73
  else
74
  yellow "[~] shellcheck not installed (skip). Install with: apt install shellcheck"
75
+ SKIPPED=$((SKIPPED + 1))
76
  fi
77
 
78
  # ---- 3. pyflakes -----------------------------------------------------------
 
90
  done
91
  else
92
  yellow "[~] pyflakes not installed (skip). Install with: pip install pyflakes"
93
+ SKIPPED=$((SKIPPED + 1))
94
  fi
95
  fi
96
 
 
110
  find . -type d -name __pycache__ -not -path './.git/*' -exec rm -rf {} + 2>/dev/null || true
111
  fi
112
 
113
+ # ---- 5. footgun: Modelfile FROM target vs the bundled GGUF -----------------
114
  #
115
+ # 'ollama create janus -f Modelfile' resolves the FROM line against the repo
116
+ # root, so that filename and the blob this repo actually ships have to be the
117
+ # same file. The 0.2.0 rebase renamed FROM ahead of the blob: for a day the
118
+ # repo tracked one bundle and FROM named another, so the local-build path
119
+ # failed outright from a fresh clone until the blob swap in da34705 caught up
120
+ # — and nothing here noticed (CHANGELOG: "the broken Modelfile FROM path
121
+ # shipped unnoticed"). A rename on either side alone breaks it again.
122
  #
123
+ # This replaces the '-MTP-Preserved-Q<quant>.gguf' grep that used to sit here:
124
+ # no Qwen 3.6 quant filename carries that substring, so the guard was green by
125
+ # construction rather than by inspection. What still guards the MTP footgun is
126
+ # scripts/strip_mtp.py itself — build.sh runs every fetched quant through it,
127
+ # and it no-ops on the MTP-clean quant this repo bundles.
128
+
129
+ blue "[*] Modelfile FROM target is the GGUF this repo ships"
130
+ if [[ ! -f "${ROOT}/Modelfile" ]]; then
131
+ yellow "[~] Modelfile missing; skipping FROM target check"
132
+ SKIPPED=$((SKIPPED + 1))
 
 
133
  else
134
+ mapfile -t BUNDLES < <(find . -maxdepth 1 -type f -name '*.gguf' | sed 's|^\./||' | sort)
135
+ FROM_TARGET="$(awk '/^FROM[[:space:]]/{print $2; exit}' "${ROOT}/Modelfile")"
136
+ if (( ${#BUNDLES[@]} != 1 )); then
137
+ red " [FAIL] expected exactly one bundled *.gguf in the repo root, found ${#BUNDLES[@]}"
138
+ FAIL=1
139
+ elif [[ "${FROM_TARGET#./}" != "${BUNDLES[0]}" ]]; then
140
+ red " [FAIL] Modelfile FROM '${FROM_TARGET}' is not the bundled '${BUNDLES[0]}'"
141
+ red " 'ollama create janus -f Modelfile' cannot resolve it from a fresh clone."
142
+ FAIL=1
143
+ elif ! git -C "${ROOT}" ls-files --error-unmatch "${BUNDLES[0]}" >/dev/null 2>&1; then
144
+ red " [FAIL] bundled '${BUNDLES[0]}' is not tracked by git"
145
+ red " It exists on disk but would be absent from a fresh clone and from the"
146
+ red " published repo. Stage it: git add '${BUNDLES[0]}'"
147
+ FAIL=1
148
+ else
149
+ echo " [ ok ] FROM ${FROM_TARGET} == bundled ${BUNDLES[0]} (tracked)"
150
+ fi
151
  fi
152
 
153
  # ---- 6. footgun: VAR="$(cmd 2>/dev/null | filter)" silent-exit pattern -----
 
219
  fi
220
  else
221
  yellow "[~] bridge files missing; skipping sync check"
222
+ SKIPPED=$((SKIPPED + 1))
223
  fi
224
 
225
+ # ---- 8. Go template: Ollama's thinking detection + replay condition ------
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
226
  #
227
  # Ollama picks between the Go 'template' and the GGUF's embedded Jinja template
228
  # by comparing their capabilities, and infers the Go template's "thinking" from
 
237
  fi
238
  else
239
  yellow "[~] template missing; skipping Go template check"
240
+ SKIPPED=$((SKIPPED + 1))
241
  fi
242
 
243
  # ---- result ----------------------------------------------------------------
 
247
  red "[!] FAIL"
248
  exit 1
249
  fi
250
+ if (( SKIPPED )); then
251
+ yellow "[+] $(( CHECKS - SKIPPED )) of ${CHECKS} checks passed, ${SKIPPED} skipped"
252
+ else
253
+ green "[+] all ${CHECKS} checks passed"
254
+ fi
scripts/check_bridge_sync.py CHANGED
@@ -1,6 +1,6 @@
1
  #!/usr/bin/env python3
2
  """
3
- Janus-27B — verify Modelfile and HF Ollama bridge files stay in sync.
4
 
5
  The repo ships two parallel Ollama configurations:
6
 
 
1
  #!/usr/bin/env python3
2
  """
3
+ Janus-35B — verify Modelfile and HF Ollama bridge files stay in sync.
4
 
5
  The repo ships two parallel Ollama configurations:
6
 
scripts/check_chat_template.py DELETED
@@ -1,228 +0,0 @@
1
- #!/usr/bin/env python3
2
- """Render ``chat_template.jinja`` and assert the properties tool calling needs.
3
-
4
- ``chat_template.jinja`` is this repo's copy of the base model's embedded chat
5
- template with two changes: ``preserve_thinking`` and ``preserve_reasoning``
6
- resolved without raising, and ``reasoning_effort`` normalisation. It is what
7
- llama.cpp / llama-server applies under ``--jinja --chat-template-file``, and what
8
- ``scripts/build.sh`` stamps into the GGUF it builds.
9
-
10
- The tool-call wire format defaults to XML, as upstream ships it. llama-server
11
- parses this template's output with its Qwen3-Coder parser, which accepts only
12
- the XML form, and converts a client's JSON-string ``function.arguments`` to a
13
- mapping before rendering. A JSON default would teach the model a form that
14
- parser cannot read, and that failure is invisible to every other check here: the
15
- template is data, not code, so ``py_compile`` and ``pyflakes`` never look at it.
16
-
17
- ``tool_call_format='auto'`` stays available for loaders that pass the raw JSON
18
- string through: the XML form needs individual key/value pairs and a chat template
19
- has ``tojson`` but no inverse, so it renders the prompt in JSON form instead.
20
-
21
- Run directly (``python3 scripts/check_chat_template.py``) or via
22
- ``./scripts/check.sh`` / ``make check``. Skips with exit 0 when Jinja2 is absent,
23
- matching how check.sh treats shellcheck and pyflakes.
24
- """
25
- import os
26
- import sys
27
-
28
- ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
29
- TEMPLATE_PATH = os.path.join(ROOT, "chat_template.jinja")
30
-
31
- try:
32
- import jinja2
33
- except ImportError:
34
- print("[~] jinja2 not installed (skip). Install with: pip install jinja2")
35
- sys.exit(0)
36
-
37
- if not os.path.exists(TEMPLATE_PATH):
38
- print(f"[!] missing {TEMPLATE_PATH}")
39
- sys.exit(1)
40
-
41
-
42
- class TemplateRaise(Exception):
43
- """What the template's raise_exception() maps to (minja provides this global)."""
44
-
45
-
46
- env = jinja2.Environment(undefined=jinja2.ChainableUndefined)
47
- env.globals["raise_exception"] = lambda msg: (_ for _ in ()).throw(TemplateRaise(msg))
48
- try:
49
- TPL = env.from_string(open(TEMPLATE_PATH, encoding="utf-8").read())
50
- except jinja2.TemplateSyntaxError as exc:
51
- print(f"[!] template does not parse: line {exc.lineno}: {exc.message}")
52
- sys.exit(1)
53
-
54
- WEATHER = {"type": "function", "function": {
55
- "name": "get_weather", "description": "Get current weather for a city",
56
- "parameters": {"type": "object", "properties": {"city": {"type": "string"}},
57
- "required": ["city"]}}}
58
- USER = {"role": "user", "content": "Weather in Tokyo?"}
59
- # Exactly the shape an OpenAI-compatible client replays on turn 2.
60
- CALL_STR = {"role": "assistant", "content": None, "tool_calls": [
61
- {"id": "c1", "type": "function",
62
- "function": {"name": "get_weather", "arguments": '{"city":"Tokyo"}'}}]}
63
- CALL_MAP = {"role": "assistant", "content": "", "tool_calls": [
64
- {"function": {"name": "get_weather", "arguments": {"city": "Tokyo"}}}]}
65
- RESULT = {"role": "tool", "tool_call_id": "c1", "content": '{"temp_c": 22}'}
66
-
67
- failures = []
68
-
69
-
70
- def render(messages, **kwargs):
71
- kwargs.setdefault("add_generation_prompt", True)
72
- return TPL.render(messages=messages, **kwargs)
73
-
74
-
75
- def check(name, condition, context=""):
76
- if condition:
77
- print(f" [ ok ] {name}")
78
- else:
79
- print(f" [FAIL] {name}")
80
- failures.append((name, context))
81
-
82
-
83
- def attempt(messages, **kwargs):
84
- """-> (ok, output_or_error)"""
85
- try:
86
- return True, render(messages, **kwargs)
87
- except Exception as exc: # noqa: BLE001 - any raise is an HTTP 500 under llama-server
88
- return False, f"{type(exc).__name__}: {exc}"
89
-
90
-
91
- # 1. The default is XML, upstream's and the only form llama-server's parser reads.
92
- ok, out = attempt([USER, CALL_MAP, RESULT], tools=[WEATHER])
93
- check("mapping tool-call arguments render with no kwargs", ok, out)
94
- if ok:
95
- check("default renders the XML tool-call form",
96
- "<tool_call>\n<function=get_weather>\n<parameter=city>\nTokyo\n</parameter>\n</function>\n</tool_call>" in out, out)
97
- check("default system prompt advertises the matching XML form",
98
- "<function=example_function_name>" in out and '{"name": "example_function_name"' not in out, out)
99
- ok_x, out_x = attempt([USER, CALL_MAP, RESULT], tools=[WEATHER], tool_call_format="xml")
100
- check("default is byte-identical to tool_call_format='xml'", ok_x and out_x == out, out_x)
101
-
102
- ok_pin, _ = attempt([USER, CALL_STR, RESULT], tools=[WEATHER])
103
- check("default still refuses unconverted string arguments (upstream behaviour)",
104
- not ok_pin, "expected a raise, got a successful render")
105
-
106
- # 2. 'auto' (opt-in): a replayed tool call with string arguments renders as JSON.
107
- ok, out = attempt([USER, CALL_STR, RESULT], tools=[WEATHER], tool_call_format="auto")
108
- check("auto: string tool-call arguments do not raise", ok, out)
109
- if ok:
110
- check("auto: string arguments render as JSON inside <tool_call>",
111
- '<tool_call>\n{"name": "get_weather", "arguments": {"city":"Tokyo"}}\n</tool_call>' in out, out)
112
- check("auto: system prompt advertises the matching JSON form",
113
- '{"name": "example_function_name"' in out, out)
114
- check("auto: no XML tool-call form alongside the JSON instructions",
115
- "<function=" not in out, out)
116
-
117
- # 3. Every format value renders mapping arguments.
118
- for fmt in ("auto", "json", "xml"):
119
- msgs = [USER, CALL_MAP, RESULT]
120
- ok_f, out_f = attempt(msgs, tools=[WEATHER], tool_call_format=fmt)
121
- check(f"tool_call_format={fmt!r} renders", ok_f, out_f)
122
-
123
- ok_bad, _ = attempt([USER], tools=[WEATHER], tool_call_format="nonsense")
124
- check("an unknown tool_call_format is rejected", not ok_bad,
125
- "expected a raise for an invalid format")
126
-
127
- # 4. Thinking prefill, both directions. Structure, not exact whitespace: the
128
- # trailing newlines are upstream's and are reproduced verbatim.
129
- ok_on, on = attempt([USER])
130
- tail_on = on.split("<|im_start|>assistant")[-1] if ok_on else ""
131
- check("thinking on primes an open <think>",
132
- ok_on and tail_on.lstrip("\n").startswith("<think>") and "</think>" not in tail_on,
133
- repr(tail_on))
134
- ok_off, off = attempt([USER], enable_thinking=False)
135
- tail_off = off.split("<|im_start|>assistant")[-1] if ok_off else ""
136
- check("thinking off primes a closed, empty <think>",
137
- ok_off and "<think>" in tail_off and "</think>" in tail_off
138
- and tail_off.split("<think>")[1].split("</think>")[0].strip() == "",
139
- repr(tail_off))
140
-
141
- # 5. Multimodal parts must survive, or the --mmproj path in the README breaks.
142
- ok_v, vis = attempt([{"role": "user", "content": [
143
- {"type": "text", "text": "What is this?"},
144
- {"type": "image_url", "image_url": {"url": "data:image/png;base64,AAA"}}]}])
145
- check("image parts render vision pads",
146
- ok_v and "<|vision_start|><|image_pad|><|vision_end|>" in vis, vis)
147
- check("text parts survive alongside images", ok_v and "What is this?" in vis, vis)
148
-
149
- # 6. Plain conversation, and multi-turn reasoning handling.
150
- ok_p, plain = attempt([{"role": "system", "content": "You are helpful."}, USER])
151
- check("plain conversation renders with the system turn",
152
- ok_p and "<|im_start|>system" in plain and "You are helpful." in plain, plain)
153
-
154
- # 7. Earlier turns' reasoning IS replayed by default, as upstream does. llama-server
155
- # from b10763 passes preserve_reasoning=true and derives preserve_thinking from
156
- # it, but a request's preserve_thinking kwarg replaces the derived copy, so the
157
- # two can disagree: that must not raise (upstream does), and preserve_thinking
158
- # wins. A live tool-call chain keeps its own trace whatever the setting.
159
- PRIOR = {"role": "assistant", "content": "First answer.", "reasoning_content": "OLDTRACE"}
160
- FOLLOWUP = {"role": "user", "content": "second question"}
161
- LIVE = {"role": "assistant", "content": "", "reasoning_content": "LIVETRACE",
162
- "tool_calls": [{"function": {"name": "get_weather", "arguments": {"city": "Tokyo"}}}]}
163
-
164
- ok_d, out_d = attempt([USER, PRIOR, FOLLOWUP])
165
- check("earlier reasoning is replayed by default", ok_d and "OLDTRACE" in out_d, out_d)
166
-
167
- ok_t, out_t = attempt([USER, PRIOR, FOLLOWUP], preserve_thinking=True)
168
- check("preserve_thinking=True replays it", ok_t and "OLDTRACE" in out_t, out_t)
169
-
170
- ok_f, out_f = attempt([USER, PRIOR, FOLLOWUP], preserve_thinking=False)
171
- check("preserve_thinking=False drops it", ok_f and "OLDTRACE" not in out_f, out_f)
172
-
173
- ok_r, out_r = attempt([USER, PRIOR, FOLLOWUP], preserve_reasoning=False)
174
- check("preserve_reasoning=False drops it", ok_r and "OLDTRACE" not in out_r, out_r)
175
-
176
- ok_c, out_c = attempt([USER, PRIOR, FOLLOWUP], preserve_reasoning=True, preserve_thinking=False)
177
- check("conflicting values do not raise; preserve_thinking=False wins (llama-server default + request kwarg)",
178
- ok_c and "OLDTRACE" not in out_c, out_c)
179
-
180
- ok_c2, out_c2 = attempt([USER, PRIOR, FOLLOWUP], preserve_reasoning=False, preserve_thinking=True)
181
- check("conflicting values do not raise; preserve_thinking=True wins",
182
- ok_c2 and "OLDTRACE" in out_c2, out_c2)
183
-
184
- ok_l, out_l = attempt([USER, LIVE, RESULT], tools=[WEATHER], preserve_thinking=False)
185
- check("a live tool-call chain keeps its own reasoning",
186
- ok_l and "LIVETRACE" in out_l, out_l)
187
-
188
- SYS = {"role": "system", "content": "You are a helpful assistant."}
189
-
190
- # reasoning_effort must never fail the request. Clients disagree on the
191
- # vocabulary (OpenAI minimal/low/medium/high, xhigh, "max", mixed case), and a
192
- # template raise is an HTTP 500 for the whole request under Ollama >= 0.33 and
193
- # llama-server - reasoning_effort="max" made the model unusable. Every value a
194
- # real client might send has to render, including an explicit JSON null.
195
- for eff in ("none", "minimal", "low", "medium", "high", "xhigh", "max",
196
- "HIGH", " High ", "auto", "default", "", "ultra", 5, True, False):
197
- ok_e, out_e = attempt([SYS, USER], reasoning_effort=eff)
198
- check(f"reasoning_effort={eff!r} renders", ok_e, out_e)
199
- ok_null, out_null = attempt([SYS, USER], reasoning_effort=None)
200
- check("reasoning_effort=null (explicit JSON null) renders", ok_null, out_null)
201
-
202
- # Ollama's /v1 layer accepts exactly eight values - minimal, low, medium, high,
203
- # xhigh, ultra, max, none - and rejects anything else with a 400 before the
204
- # template runs. Each must land on a sensible tier, not merely avoid raising.
205
- for eff, want in (("high", "xhigh"), ("xhigh", "xhigh"), ("max", "xhigh"),
206
- ("ultra", "xhigh"), ("low", "low"), ("minimal", "low")):
207
- ok_t, out_t = attempt([SYS, USER], reasoning_effort=eff)
208
- check(f"reasoning_effort={eff!r} selects the {want} tier",
209
- ok_t and f"Reasoning effort is set to {want}" in out_t, out_t)
210
- ok_med, out_med = attempt([SYS, USER], reasoning_effort="medium")
211
- check("reasoning_effort='medium' selects the medium tier",
212
- ok_med and "Reasoning effort is set to" not in out_med, out_med)
213
- ok_none, out_none = attempt([SYS, USER], reasoning_effort="none")
214
- tail_none = out_none.split("<|im_start|>assistant")[-1] if ok_none else ""
215
- check("reasoning_effort='none' switches thinking off",
216
- ok_none and "</think>" in tail_none
217
- and tail_none.split("<think>")[1].split("</think>")[0].strip() == "",
218
- repr(tail_none))
219
-
220
- print()
221
- if failures:
222
- print(f"[!] {len(failures)} chat-template check(s) failed")
223
- for name, context in failures:
224
- print(f"--- {name} ---")
225
- if context:
226
- print(context[:1500])
227
- sys.exit(1)
228
- print("[+] chat_template.jinja checks passed")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
scripts/fetch_vision.sh CHANGED
@@ -1,20 +1,20 @@
1
  #!/usr/bin/env bash
2
- # Janus-27B — fetch the vision projector (mmproj) for image input.
3
  #
4
  # Why this is separate from build.sh:
5
  # build.sh is for the Ollama text path. The mmproj is only useful for
6
  # llama.cpp / llama-cpp-python right now, because Ollama's vendored
7
- # llama.cpp fork is missing the qwen35 arch entries needed to attach
8
  # it (see README Vision section, ollama/ollama#14575).
9
  #
10
  # Usage:
11
- # ./scripts/fetch_vision.sh # default: BF16
12
  #
13
- # The Heretic GGUF repo publishes the projector as BF16; for another
14
- # precision, point REPO_ID / FILE_NAME at a projector from any other
15
- # Qwen3.8-27B GGUF repo:
16
- # REPO_ID=<other-qwen3.8-27b-gguf-repo> FILE_NAME=mmproj-F16.gguf ./scripts/fetch_vision.sh
17
- # (vision tokens are projected the same way across Qwen 3.8 27B
18
  # finetunes, so a family projector is functionally interchangeable.)
19
  #
20
  # Requires: huggingface-cli (or hf).
@@ -38,8 +38,8 @@ detect_hf() {
38
  }
39
 
40
  PRECISION="${1:-${PRECISION:-BF16}}"
41
- REPO_ID="${REPO_ID:-llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-GGUF}"
42
- FILE_NAME="${FILE_NAME:-Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-mmproj-${PRECISION}.gguf}"
43
  ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
44
  DEST="${MMPROJ_PATH:-${ROOT}/${FILE_NAME}}"
45
 
@@ -72,7 +72,7 @@ fi
72
  echo
73
  echo "[+] Done. Use it via:"
74
  echo " python ${ROOT}/examples/llama_cpp_vision.py \\"
75
- echo " --gguf ${ROOT}/Janus-27B.Q4_K_M.gguf \\"
76
  echo " --mmproj ${DEST} \\"
77
  echo " --image /path/to/photo.jpg \\"
78
  echo " --prompt 'Describe this image.'"
 
1
  #!/usr/bin/env bash
2
+ # Janus-35B — fetch the vision projector (mmproj) for image input.
3
  #
4
  # Why this is separate from build.sh:
5
  # build.sh is for the Ollama text path. The mmproj is only useful for
6
  # llama.cpp / llama-cpp-python right now, because Ollama's vendored
7
+ # llama.cpp fork is missing the qwen35moe arch entries needed to attach
8
  # it (see README Vision section, ollama/ollama#14575).
9
  #
10
  # Usage:
11
+ # ./scripts/fetch_vision.sh # default: BF16 (902 MB)
12
  #
13
+ # The Heretic GGUF repo publishes the projector as BF16 only (its Files
14
+ # listing, HF API, 2026-09-18); for another precision, point REPO_ID /
15
+ # FILE_NAME at a projector from any other Qwen3.6-35B-A3B GGUF repo:
16
+ # REPO_ID=<other-qwen3.6-35b-a3b-gguf-repo> FILE_NAME=mmproj-F16.gguf ./scripts/fetch_vision.sh
17
+ # (vision tokens are projected the same way across Qwen 3.6 35B-A3B
18
  # finetunes, so a family projector is functionally interchangeable.)
19
  #
20
  # Requires: huggingface-cli (or hf).
 
38
  }
39
 
40
  PRECISION="${1:-${PRECISION:-BF16}}"
41
+ REPO_ID="${REPO_ID:-llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF}"
42
+ FILE_NAME="${FILE_NAME:-Qwen3.6-35B-A3B-uncensored-heretic-mmproj-${PRECISION}.gguf}"
43
  ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
44
  DEST="${MMPROJ_PATH:-${ROOT}/${FILE_NAME}}"
45
 
 
72
  echo
73
  echo "[+] Done. Use it via:"
74
  echo " python ${ROOT}/examples/llama_cpp_vision.py \\"
75
+ echo " --gguf ${ROOT}/Janus-35B-A3B.Q4_K_M.gguf \\"
76
  echo " --mmproj ${DEST} \\"
77
  echo " --image /path/to/photo.jpg \\"
78
  echo " --prompt 'Describe this image.'"
scripts/install-hooks.sh CHANGED
@@ -1,5 +1,5 @@
1
  #!/usr/bin/env bash
2
- # Janus-27B — install scripts/check.sh as a git pre-commit hook.
3
  #
4
  # Idempotent. Re-runs are safe.
5
  set -euo pipefail
 
1
  #!/usr/bin/env bash
2
+ # Janus-35B — install scripts/check.sh as a git pre-commit hook.
3
  #
4
  # Idempotent. Re-runs are safe.
5
  set -euo pipefail
scripts/live_check.sh CHANGED
@@ -22,9 +22,10 @@
22
  # WORK_DIR (<repo>/.cache/live-check): each run uses a fresh subdirectory of
23
  # it and deletes only that; KEEP=1 keeps the run directory.
24
  #
25
- # Needs free disk on WORK_DIR of about twice the GGUF (Ollama keeps the imported
26
- # file plus a rewritten copy, and briefly more during the import), RAM to load it,
27
- # and tens of minutes on CPU for a 27B. Exits non-zero if any check fails.
 
28
  #
29
  # The server runs on CPU only: GPU backends are switched off in its environment,
30
  # and the run stops before importing if Ollama still reports a GPU. On a shared
 
22
  # WORK_DIR (<repo>/.cache/live-check): each run uses a fresh subdirectory of
23
  # it and deletes only that; KEEP=1 keeps the run directory.
24
  #
25
+ # Needs free disk on WORK_DIR of about twice the GGUF - ~42 GB for the bundled
26
+ # 21.23 GB Q4_K_M (Ollama keeps the imported file plus a rewritten copy, and
27
+ # briefly more during the import), RAM to load it, and tens of minutes on CPU for
28
+ # a 35B-A3B. Exits non-zero if any check fails.
29
  #
30
  # The server runs on CPU only: GPU backends are switched off in its environment,
31
  # and the run stops before importing if Ollama still reports a GPU. On a shared
scripts/load_bundle.sh CHANGED
@@ -1,14 +1,14 @@
1
  #!/usr/bin/env bash
2
- # Janus-27B — load this repo's bundle into Ollama as a local tag.
3
  #
4
- # The bundled GGUF (Janus-27B.Q4_K_M.gguf) is qwen35-stamped and
5
- # MTP-stripped, so it loads directly on stock llama.cpp / Ollama. This
6
- # script is the one-shot path from "I just cloned this repo" to "I have
7
- # a working local Ollama tag":
8
  #
9
  # 1. Resolve the bundle. If it's an LFS pointer (cloned without
10
- # `git lfs pull`), download the real ~17 GB blob via `hf download`.
11
- # 2. Sanity-check `general.architecture` is qwen35.
12
  # 3. Run `ollama create <tag> -f <temp Modelfile pointing at the
13
  # resolved bundle>`.
14
  #
@@ -18,7 +18,7 @@
18
  # Usage:
19
  # ./scripts/load_bundle.sh # default tag: janus
20
  # TAG=janus-bundle ./scripts/load_bundle.sh
21
- # BUNDLE=/path/to/Janus-27B.Q4_K_M.gguf ./scripts/load_bundle.sh
22
  #
23
  # Requires: ollama, python3 with the `gguf` package, hf (if the bundle
24
  # needs to be downloaded).
@@ -42,7 +42,7 @@ detect_hf() {
42
  }
43
 
44
  ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
45
- BUNDLE="${BUNDLE:-${ROOT}/Janus-27B.Q4_K_M.gguf}"
46
  TAG="${TAG:-janus}"
47
  REPO_ID="${REPO_ID:-FoolDev/Janus-35B-HERETIC}"
48
  MODELFILE="${ROOT}/Modelfile"
@@ -116,14 +116,14 @@ PY
116
  )"
117
  echo "[*] bundle arch: ${ARCH}"
118
 
119
- if [[ "${ARCH}" != "qwen35" ]]; then
120
  echo "[!] unexpected arch '${ARCH}' — refusing to load. Edit this script if intentional." >&2
121
  exit 1
122
  fi
123
 
124
  # ---- 4. Build a Modelfile copy with FROM pointing at the bundle --------------
125
 
126
- TMP_MODELFILE="$(mktemp -t janus27b-loadbundle.XXXXXX)"
127
  trap 'rm -f "${TMP_MODELFILE}"' EXIT
128
  awk -v p="${BUNDLE}" '
129
  /^FROM[[:space:]]/ && !done { print "FROM " p; done=1; next }
 
1
  #!/usr/bin/env bash
2
+ # Janus-35B — load this repo's bundle into Ollama as a local tag.
3
  #
4
+ # The bundled GGUF (Janus-35B-A3B.Q4_K_M.gguf) is qwen35moe-stamped and
5
+ # loads directly on stock llama.cpp / Ollama. This script is the
6
+ # one-shot path from "I just cloned this repo" to "I have a working
7
+ # local Ollama tag":
8
  #
9
  # 1. Resolve the bundle. If it's an LFS pointer (cloned without
10
+ # `git lfs pull`), download the real ~21 GB blob via `hf download`.
11
+ # 2. Sanity-check `general.architecture` is qwen35moe.
12
  # 3. Run `ollama create <tag> -f <temp Modelfile pointing at the
13
  # resolved bundle>`.
14
  #
 
18
  # Usage:
19
  # ./scripts/load_bundle.sh # default tag: janus
20
  # TAG=janus-bundle ./scripts/load_bundle.sh
21
+ # BUNDLE=/path/to/Janus-35B-A3B.Q4_K_M.gguf ./scripts/load_bundle.sh
22
  #
23
  # Requires: ollama, python3 with the `gguf` package, hf (if the bundle
24
  # needs to be downloaded).
 
42
  }
43
 
44
  ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
45
+ BUNDLE="${BUNDLE:-${ROOT}/Janus-35B-A3B.Q4_K_M.gguf}"
46
  TAG="${TAG:-janus}"
47
  REPO_ID="${REPO_ID:-FoolDev/Janus-35B-HERETIC}"
48
  MODELFILE="${ROOT}/Modelfile"
 
116
  )"
117
  echo "[*] bundle arch: ${ARCH}"
118
 
119
+ if [[ "${ARCH}" != "qwen35moe" ]]; then
120
  echo "[!] unexpected arch '${ARCH}' — refusing to load. Edit this script if intentional." >&2
121
  exit 1
122
  fi
123
 
124
  # ---- 4. Build a Modelfile copy with FROM pointing at the bundle --------------
125
 
126
+ TMP_MODELFILE="$(mktemp -t janus35b-loadbundle.XXXXXX)"
127
  trap 'rm -f "${TMP_MODELFILE}"' EXIT
128
  awk -v p="${BUNDLE}" '
129
  /^FROM[[:space:]]/ && !done { print "FROM " p; done=1; next }
scripts/smoke_test.sh CHANGED
@@ -1,5 +1,5 @@
1
  #!/usr/bin/env bash
2
- # Janus-27B — smoke test against a running Ollama daemon.
3
  #
4
  # Verifies:
5
  # 1. The Ollama server is reachable.
@@ -46,7 +46,7 @@ echo "[+] server reachable"
46
  # ---- 2. Model present? ------------------------------------------------------
47
  # Match case-insensitively: Ollama 0.24 normalizes model names at lookup but
48
  # preserves whatever case was first registered on disk (e.g. an earlier
49
- # session may leave a `Janus-27B:latest` manifest dir behind even when a
50
  # build was invoked with TAG=janus). The exact tag the user typed is still
51
  # valid for `ollama run` — the comparison just needs to be case-folded to
52
  # match.
@@ -54,7 +54,7 @@ if ! curl -fsS "${HOST}/api/tags" | jq -e --arg m "${MODEL}" '.models[] | select
54
  echo "[!] Model '${MODEL}' not found. Build it first:" >&2
55
  echo " ./scripts/build.sh # Q4_K_M" >&2
56
  echo " ./scripts/build.sh Q3_K_M # smaller quant" >&2
57
- echo " ./scripts/load_bundle.sh # load this repo's qwen35 bundle" >&2
58
  exit 1
59
  fi
60
  echo "[+] model present"
 
1
  #!/usr/bin/env bash
2
+ # Janus-35B — smoke test against a running Ollama daemon.
3
  #
4
  # Verifies:
5
  # 1. The Ollama server is reachable.
 
46
  # ---- 2. Model present? ------------------------------------------------------
47
  # Match case-insensitively: Ollama 0.24 normalizes model names at lookup but
48
  # preserves whatever case was first registered on disk (e.g. an earlier
49
+ # session may leave a `Janus-35B:latest` manifest dir behind even when a
50
  # build was invoked with TAG=janus). The exact tag the user typed is still
51
  # valid for `ollama run` — the comparison just needs to be case-folded to
52
  # match.
 
54
  echo "[!] Model '${MODEL}' not found. Build it first:" >&2
55
  echo " ./scripts/build.sh # Q4_K_M" >&2
56
  echo " ./scripts/build.sh Q3_K_M # smaller quant" >&2
57
+ echo " ./scripts/load_bundle.sh # load this repo's qwen35moe bundle" >&2
58
  exit 1
59
  fi
60
  echo "[+] model present"
scripts/strip_mtp.py CHANGED
@@ -1,22 +1,28 @@
1
  #!/usr/bin/env python3
2
  """
3
- Strip the MTP / NextN layer (the extra last block) out of a qwen35 GGUF so it
4
  loads on stock llama.cpp / Ollama.
5
 
6
  MTP-preserving GGUF conversions keep the MTP tensors as one extra NextN block,
7
- so block_count = <transformer layers> + 1 MTP layer — here 64 + 1 = 65, with the
8
- NextN block at index 64. Stock loaders treat that last block as a normal hybrid
9
  layer and abort with `missing tensor 'blk.<last>.ssm_conv1d.weight'`. This drops
10
- all `blk.<last>.*` tensors, sets block_count down by one (back to 64), and
11
  removes nextn_predict_layers.
12
 
13
- Janus' base — llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved
14
- — ships the MTP head intact, so this step is REQUIRED, not a safety net: the
15
- bundled Janus-27B.Q4_K_M.gguf is this script's stripped output, exactly as the
16
- dense sibling FoolDev/Thanatos-27B-HERETIC does for its heretic-ara base. It
17
- still no-ops (hardlink, no extra disk) on an input that is already MTP-clean.
18
- Wanting the MTP head means running the upstream safetensors under vLLM / SGLang,
19
- or taking the unstripped GGUF straight from the source repo.
 
 
 
 
 
 
20
 
21
  Every KEPT tensor is copied byte-for-byte (raw quantized data) — no
22
  re-quantization, so no quality change to the model itself.
@@ -26,11 +32,11 @@ Optionally also restamp the embedded chat template in the same pass:
26
  --chat-template FILE replace tokenizer.chat_template with FILE's contents
27
 
28
  That matters because llama.cpp, LM Studio, Jan and KoboldCpp all read the
29
- template baked into the GGUF, and the upstream one raises on reasoning_effort
30
- values such as "max" and on conflicting preserve_thinking / preserve_reasoning,
31
- which llama-server returns as HTTP 500. Stamping this repo's chat_template.jinja
32
- in at build time gives every one of those loaders its fixes with no extra flags. build.sh does this
33
- automatically when chat_template.jinja is present.
34
 
35
  With --chat-template and no MTP block to strip, the file is still rewritten (the
36
  usual hardlink passthrough would not carry the new template).
 
1
  #!/usr/bin/env python3
2
  """
3
+ Strip the MTP / NextN layer (the extra last block) out of a qwen35moe GGUF so it
4
  loads on stock llama.cpp / Ollama.
5
 
6
  MTP-preserving GGUF conversions keep the MTP tensors as one extra NextN block,
7
+ so block_count = <transformer layers> + 1 MTP layer — here 40 + 1 = 41, with the
8
+ NextN block at index 40. Stock loaders treat that last block as a normal hybrid
9
  layer and abort with `missing tensor 'blk.<last>.ssm_conv1d.weight'`. This drops
10
+ all `blk.<last>.*` tensors, sets block_count down by one (back to 40), and
11
  removes nextn_predict_layers.
12
 
13
+ For the blob this repo bundles, that is a no-op passthrough (hardlink, no extra
14
+ disk), not a required build step. Janus-35B-A3B.Q4_K_M.gguf comes from
15
+ llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF and is already MTP-clean:
16
+ block_count 40, blk.0…blk.39 only, no nextn_predict_layers. The head exists
17
+ upstream (the base's config.json declares mtp_num_hidden_layers 1) and llmfan46
18
+ publishes it as a separate *-Native-MTP-Preserved-GGUF whose Q4_K_M reports
19
+ block_count 41 — that variant is what this script is for, and this repo
20
+ deliberately does not ship it. So the strip is kept purely as a safety net, the
21
+ way the dense sibling FoolDev/Thanatos-27B-HERETIC needs it for its
22
+ MTP-preserved heretic-ara base: it guards against a future llmfan46 re-quant
23
+ shipping the MTP layer. Wanting the MTP head means running the upstream
24
+ safetensors under vLLM / SGLang, or taking the unstripped GGUF straight from
25
+ that variant repo.
26
 
27
  Every KEPT tensor is copied byte-for-byte (raw quantized data) — no
28
  re-quantization, so no quality change to the model itself.
 
32
  --chat-template FILE replace tokenizer.chat_template with FILE's contents
33
 
34
  That matters because llama.cpp, LM Studio, Jan and KoboldCpp all read the
35
+ template baked into the GGUF, so replacing it is the only way to change what
36
+ those loaders see. This repo no longer carries a chat_template.jinja — the Qwen
37
+ 3.6 base's own embedded template governs the llama.cpp path — so nothing here
38
+ passes the flag today: build.sh calls this script without it, and the bundled
39
+ blob keeps the template it shipped with.
40
 
41
  With --chat-template and no MTP block to strip, the file is still rewritten (the
42
  usual hardlink passthrough would not carry the new template).
scripts/verify_arch.py CHANGED
@@ -1,9 +1,9 @@
1
  #!/usr/bin/env python3
2
  """
3
- Janus-27B — verify the README "Architecture" claims and the dense
4
  forward-pass structure against the actual GGUF metadata.
5
 
6
- Reads the qwen35-stamped bundle (or any GGUF that declares that
7
  `general.architecture` value), prints each claim alongside the metadata key
8
  it derives from, and exits non-zero if any value mismatches the expected
9
  claim. Useful as a manual audit after the bundle is re-stamped or after
@@ -11,19 +11,22 @@ upstream re-conversion.
11
 
12
  Usage:
13
  python3 scripts/verify_arch.py # default bundle
14
- python3 scripts/verify_arch.py Janus-27B.Q4_K_M.gguf
15
  python3 scripts/verify_arch.py /path/to/some-other.gguf
16
 
17
  Exit code 0 = all claims verify, 1 = at least one mismatch.
18
 
19
- Note: this does NOT verify the 27B parameter count directly (no such KV in
20
- the GGUF) — it follows from the layer count, hidden size and FFN width below
21
- and llama.cpp's `qwen35` type branch, not from a single metadata field.
22
-
23
- `block_count` is checked against 64, i.e. the MTP-stripped value. The source
24
- GGUF is MTP-preserved and carries an extra NextN block at index 64, so a
25
- bundle that has not been through scripts/strip_mtp.py reports 65 here and
26
- fails this check — as it would also fail to load on stock llama.cpp / Ollama.
 
 
 
27
  """
28
  from __future__ import annotations
29
 
@@ -33,22 +36,27 @@ from pathlib import Path
33
  from gguf import GGUFReader
34
 
35
  EXPECTED = {
36
- "block_count": (64, "64 transformer layers"),
37
  "context_length": (262144, "262 144 native context"),
38
- "embedding_length": (5120, "Hidden size 5120"),
39
- "feed_forward_length": (17408, "Dense FFN intermediate size 17408"),
40
- "attention.head_count": (24, "Gated Attention: 24 Q-heads"),
41
- "attention.head_count_kv": (4, "Gated Attention: 4 KV-heads (GQA)"),
 
 
 
42
  "attention.key_length": (256, "Gated Attention: head_dim 256 (key)"),
43
  "attention.value_length": (256, "Gated Attention: head_dim 256 (value)"),
44
  "rope.dimension_count": (64, "Partial RoPE: 64 of 256 dims (factor 0.25)"),
45
- "full_attention_interval": (4, "Hybrid stack: every 4th layer is Gated Attention (16 cycles)"),
 
46
  "ssm.state_size": (128, "Gated DeltaNet: head_dim 128"),
47
- "ssm.time_step_rank": (48, "Gated DeltaNet: 48 V-heads"),
48
  "ssm.group_count": (16, "Gated DeltaNet: 16 QK-heads"),
 
49
  }
50
  EXPECTED_VOCAB = 248320
51
- EXPECTED_ARCHS = {"qwen35"}
52
 
53
 
54
  def read_scalar(reader: GGUFReader, key: str):
@@ -69,18 +77,18 @@ def read_arch(reader: GGUFReader) -> str:
69
 
70
  def main() -> int:
71
  if len(sys.argv) > 2:
72
- print(f"usage: {sys.argv[0]} [path/to/Janus-27B.Q4_K_M.gguf]", file=sys.stderr)
73
  return 2
74
  root = Path(__file__).resolve().parent.parent
75
  default_paths = [
76
- root / "Janus-27B.Q4_K_M.gguf",
77
  ]
78
  if len(sys.argv) == 2:
79
  path = Path(sys.argv[1])
80
  else:
81
  path = next((p for p in default_paths if p.exists() and p.stat().st_size > 1024), None)
82
  if path is None:
83
- print("[!] no Janus-27B GGUF found in repo root; pass a path explicitly", file=sys.stderr)
84
  return 2
85
 
86
  print(f"[*] reading: {path}")
@@ -114,11 +122,12 @@ def main() -> int:
114
  mismatches += 1
115
 
116
  # ---- embedded chat template vs the repo's chat_template.jinja -------------
117
- # scripts/build.sh stamps chat_template.jinja into every GGUF it builds, and the
118
- # bundled blob carries it too. Nothing else notices if the two drift: the
119
- # template is data, the blob is an LFS pointer in a fresh clone, and a stale
120
- # embedded copy silently reinstates the tool-calling failure for every loader
121
- # that reads the GGUF instead of an external file. This has drifted twice.
 
122
  repo_template = Path(__file__).resolve().parent.parent / "chat_template.jinja"
123
  if repo_template.exists():
124
  f = reader.fields.get("tokenizer.chat_template")
 
1
  #!/usr/bin/env python3
2
  """
3
+ Janus-35B — verify the README "Architecture" claims and the MoE
4
  forward-pass structure against the actual GGUF metadata.
5
 
6
+ Reads the qwen35moe-stamped bundle (or any GGUF that declares that
7
  `general.architecture` value), prints each claim alongside the metadata key
8
  it derives from, and exits non-zero if any value mismatches the expected
9
  claim. Useful as a manual audit after the bundle is re-stamped or after
 
11
 
12
  Usage:
13
  python3 scripts/verify_arch.py # default bundle
14
+ python3 scripts/verify_arch.py Janus-35B-A3B.Q4_K_M.gguf
15
  python3 scripts/verify_arch.py /path/to/some-other.gguf
16
 
17
  Exit code 0 = all claims verify, 1 = at least one mismatch.
18
 
19
+ Note: this does NOT verify the ~34.7B-total / ~3B-active parameter counts
20
+ directly (no such KV in the GGUF) — they follow from the layer count, hidden
21
+ size and the expert count/width below and llama.cpp's `qwen35moe` type branch,
22
+ not from a single metadata field.
23
+
24
+ `block_count` is checked against 40, which is this model's natural depth (10
25
+ Gated-Attention + 30 Gated-DeltaNet layers), NOT a post-strip value: the
26
+ bundled quant carries no MTP/NextN tensors, so scripts/strip_mtp.py is a
27
+ defensive no-op for it. llmfan46's separate *-Native-MTP-Preserved-GGUF variant
28
+ — which this repo does not ship — reports 41 here, with a NextN block at index
29
+ 40 that stock llama.cpp / Ollama cannot load.
30
  """
31
  from __future__ import annotations
32
 
 
36
  from gguf import GGUFReader
37
 
38
  EXPECTED = {
39
+ "block_count": (40, "40 transformer layers"),
40
  "context_length": (262144, "262 144 native context"),
41
+ "embedding_length": (2048, "Hidden size 2048"),
42
+ "expert_count": (256, "MoE: 256 experts"),
43
+ "expert_used_count": (8, "MoE: 8 experts active per token"),
44
+ "expert_feed_forward_length": (512, "MoE: per-expert FFN 512"),
45
+ "expert_shared_feed_forward_length": (512, "MoE: shared-expert FFN 512"),
46
+ "attention.head_count": (16, "Gated Attention: 16 Q-heads"),
47
+ "attention.head_count_kv": (2, "Gated Attention: 2 KV-heads (GQA)"),
48
  "attention.key_length": (256, "Gated Attention: head_dim 256 (key)"),
49
  "attention.value_length": (256, "Gated Attention: head_dim 256 (value)"),
50
  "rope.dimension_count": (64, "Partial RoPE: 64 of 256 dims (factor 0.25)"),
51
+ "full_attention_interval": (4, "Hybrid stack: every 4th layer is Gated Attention (10 cycles)"),
52
+ "ssm.conv_kernel": (4, "Gated DeltaNet: conv kernel 4"),
53
  "ssm.state_size": (128, "Gated DeltaNet: head_dim 128"),
54
+ "ssm.time_step_rank": (32, "Gated DeltaNet: 32 V-heads"),
55
  "ssm.group_count": (16, "Gated DeltaNet: 16 QK-heads"),
56
+ "ssm.inner_size": (4096, "Gated DeltaNet: inner size 4096 (32 V-heads x 128)"),
57
  }
58
  EXPECTED_VOCAB = 248320
59
+ EXPECTED_ARCHS = {"qwen35moe"}
60
 
61
 
62
  def read_scalar(reader: GGUFReader, key: str):
 
77
 
78
  def main() -> int:
79
  if len(sys.argv) > 2:
80
+ print(f"usage: {sys.argv[0]} [path/to/Janus-35B-A3B.Q4_K_M.gguf]", file=sys.stderr)
81
  return 2
82
  root = Path(__file__).resolve().parent.parent
83
  default_paths = [
84
+ root / "Janus-35B-A3B.Q4_K_M.gguf",
85
  ]
86
  if len(sys.argv) == 2:
87
  path = Path(sys.argv[1])
88
  else:
89
  path = next((p for p in default_paths if p.exists() and p.stat().st_size > 1024), None)
90
  if path is None:
91
+ print("[!] no Janus-35B GGUF found in repo root; pass a path explicitly", file=sys.stderr)
92
  return 2
93
 
94
  print(f"[*] reading: {path}")
 
122
  mismatches += 1
123
 
124
  # ---- embedded chat template vs the repo's chat_template.jinja -------------
125
+ # Since 0.7.0 this repo ships NO chat_template.jinja: the bundled GGUF carries the
126
+ # Qwen 3.6 base's own embedded template, untouched, so there are no two copies to
127
+ # drift and the block below skips. It is kept for the case where a maintainer
128
+ # reintroduces a repo-side template (as 0.2.0-0.6.5 had): back then the two did
129
+ # drift twice, and a stale embedded copy silently reinstates the tool-calling
130
+ # failure for every loader that reads the GGUF instead of an external file.
131
  repo_template = Path(__file__).resolve().parent.parent / "chat_template.jinja"
132
  if repo_template.exists():
133
  f = reader.fields.get("tokenizer.chat_template")