Instructions to use FoolDev/Janus-35B-HERETIC with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use FoolDev/Janus-35B-HERETIC with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf FoolDev/Janus-35B-HERETIC:Q4_K_M # Run inference directly in the terminal: llama cli -hf FoolDev/Janus-35B-HERETIC:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf FoolDev/Janus-35B-HERETIC:Q4_K_M # Run inference directly in the terminal: llama cli -hf FoolDev/Janus-35B-HERETIC:Q4_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf FoolDev/Janus-35B-HERETIC:Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf FoolDev/Janus-35B-HERETIC:Q4_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf FoolDev/Janus-35B-HERETIC:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf FoolDev/Janus-35B-HERETIC:Q4_K_M
Use Docker
docker model run hf.co/FoolDev/Janus-35B-HERETIC:Q4_K_M
- LM Studio
- Jan
- vLLM
How to use FoolDev/Janus-35B-HERETIC with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "FoolDev/Janus-35B-HERETIC" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "FoolDev/Janus-35B-HERETIC", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/FoolDev/Janus-35B-HERETIC:Q4_K_M
- Ollama
How to use FoolDev/Janus-35B-HERETIC with Ollama:
ollama run hf.co/FoolDev/Janus-35B-HERETIC:Q4_K_M
- Unsloth Desktop
- Pi
How to use FoolDev/Janus-35B-HERETIC with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf FoolDev/Janus-35B-HERETIC:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "FoolDev/Janus-35B-HERETIC:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use FoolDev/Janus-35B-HERETIC with Docker Model Runner:
docker model run hf.co/FoolDev/Janus-35B-HERETIC:Q4_K_M
- Lemonade
How to use FoolDev/Janus-35B-HERETIC with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull FoolDev/Janus-35B-HERETIC:Q4_K_M
Run and chat with the model
lemonade run user.Janus-35B-HERETIC-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use FoolDev/Janus-35B-HERETIC with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf FoolDev/Janus-35B-HERETIC:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default FoolDev/Janus-35B-HERETIC:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use FoolDev/Janus-35B-HERETIC with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf FoolDev/Janus-35B-HERETIC:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "FoolDev/Janus-35B-HERETIC:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
release 0.7.0: reverse the rebase — back to the Qwen 3.6 35B-A3B MoE
Browse filesReleases 0.2.0 through 0.6.5 shipped a dense Qwen 3.8 27B under this slug.
That rebase is reversed: the model is llmfan46/Qwen3.6-35B-A3B-uncensored-heretic
again — ~34.7B total / ~3B active, 40 layers (10 full-attention + 30 Gated
DeltaNet), 256 experts with 8 routed per token plus a shared expert, arch
qwen35moe. Display name Janus-27B -> Janus-35B, so the Hub slug names the model
correctly again.
- Blob: Janus-35B-A3B.Q4_K_M.gguf restored as a pointer-only commit
(21,233,608,512 B, sha256 b82f9f21…), byte-identical to upstream's Q4_K_M and
to what this repo shipped until da34705. HF still holds that LFS object for
this repo, so nothing is re-uploaded. No re-quantization, no MTP strip (this
quant has no NextN block), no template restamp.
- chat_template.jinja is deleted: it was a patch of the dense base's 3.8
template, and Qwen 3.6's own template — which the GGUF carries — has no
reasoning_effort, tool_call_format or preserve_reasoning to patch. The card's
llama.cpp sections were rewritten from what that template actually does,
including that string tool-call arguments are emitted as-is rather than
raising. check_chat_template.py and check.sh's check 8 go with it.
- Every measured figure taken on the dense 27B is deleted, not rescaled. Memory
totals are now derived from GGUF geometry and labelled as such (KV is
20,480 B/token — 0.625 GiB per 32K f16, ~5.0 GiB at the 262144 default); no
tok/s figures are published at all.
- verify_arch.py rewritten for the MoE claims; check 5's MTP grep, which would
have been permanently vacuous against 3.6 filenames, replaced by a check that
the Modelfile's FROM is the bundled GGUF *and* that it is tracked by git;
check.sh now reports "N of M checks passed, K skipped" instead of a green
"all passed" over a skipped check.
- moe-routing.svg restored verbatim, banner wordmark back to JANUS-35B, and
banner.png restored byte-for-byte to avoid re-rasterization drift.
- Fixed while auditing: front-matter claimed library_name transformers and
pipeline_tag image-text-to-text on a GGUF-only repo (now gguf /
text-generation); .gitignore's !Janus-27B.*.gguf whitelist would have silently
dropped the new blob; the undated "Ollama 0.24+" floor is replaced by the
versions actually checked in source.
Not done here: no live run of any kind — live_check.sh, a tool-call round-trip,
verify_arch.py and every memory measurement need the 21 GB blob on disk. The
card says so wherever it matters.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- .gitignore +4 -4
- CHANGELOG.md +100 -0
- CITATION.cff +26 -22
- Janus-27B.Q4_K_M.gguf → Janus-35B-A3B.Q4_K_M.gguf +2 -2
- Makefile +6 -6
- Modelfile +61 -52
- README.md +266 -275
- banner.png +0 -0
- banner.svg +5 -5
- bmc-banner.svg +2 -2
- chat_template.jinja +0 -642
- examples/README.md +13 -12
- examples/llama_cpp_quickstart.py +3 -3
- examples/llama_cpp_vision.py +14 -14
- examples/ollama_chat.py +1 -1
- examples/transformers_quickstart.py +11 -9
- moe-routing.svg +670 -0
- scripts/bench.sh +1 -1
- scripts/build.sh +27 -37
- scripts/cap_ctx.sh +8 -7
- scripts/check.sh +54 -46
- scripts/check_bridge_sync.py +1 -1
- scripts/check_chat_template.py +0 -228
- scripts/fetch_vision.sh +11 -11
- scripts/install-hooks.sh +1 -1
- scripts/live_check.sh +4 -3
- scripts/load_bundle.sh +11 -11
- scripts/smoke_test.sh +3 -3
- scripts/strip_mtp.py +22 -16
- scripts/verify_arch.py +36 -27
|
@@ -7,13 +7,13 @@ venv/
|
|
| 7 |
|
| 8 |
# Local model weights. We don't redistribute the upstream Qwen GGUFs
|
| 9 |
# here — scripts/build.sh fetches one from
|
| 10 |
-
# llmfan46/Qwen3.
|
| 11 |
-
# locally. The single Janus-
|
| 12 |
# "Use this model" widget (ollama run hf.co/FoolDev/Janus-35B-HERETIC).
|
| 13 |
*.gguf
|
| 14 |
-
!Janus-
|
| 15 |
# scripts/strip_mtp.py + build.sh intermediates — local only, never tracked.
|
| 16 |
-
Janus-
|
| 17 |
*.safetensors
|
| 18 |
*.bin
|
| 19 |
|
|
|
|
| 7 |
|
| 8 |
# Local model weights. We don't redistribute the upstream Qwen GGUFs
|
| 9 |
# here — scripts/build.sh fetches one from
|
| 10 |
+
# llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF
|
| 11 |
+
# locally. The single Janus-35B-A3B.*.gguf we DO ship backs the HF/Ollama
|
| 12 |
# "Use this model" widget (ollama run hf.co/FoolDev/Janus-35B-HERETIC).
|
| 13 |
*.gguf
|
| 14 |
+
!Janus-35B-A3B.*.gguf
|
| 15 |
# scripts/strip_mtp.py + build.sh intermediates — local only, never tracked.
|
| 16 |
+
Janus-35B-A3B.*.stripped.gguf
|
| 17 |
*.safetensors
|
| 18 |
*.bin
|
| 19 |
|
|
@@ -8,6 +8,106 @@ track the **tooling and documentation**, not the underlying base model.
|
|
| 8 |
|
| 9 |
## [Unreleased]
|
| 10 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
## [0.6.5] — 2026-09-17
|
| 12 |
|
| 13 |
### Documentation (upstream claims re-checked against Ollama 0.34.2)
|
|
|
|
| 8 |
|
| 9 |
## [Unreleased]
|
| 10 |
|
| 11 |
+
## [0.7.0] — 2026-09-18
|
| 12 |
+
|
| 13 |
+
### Changed (reversal → Qwen 3.6 35B-A3B MoE)
|
| 14 |
+
|
| 15 |
+
- **This repo is the 35B-A3B MoE again.** Releases 0.2.0 through 0.6.5 carried a
|
| 16 |
+
dense Qwen 3.8 27B under the `Janus-35B-HERETIC` slug; that rebase is reversed.
|
| 17 |
+
The model is `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic` — ~34.7B parameters
|
| 18 |
+
total, ~3B active per token, 40 layers (10 full-attention + 30 Gated DeltaNet),
|
| 19 |
+
256 experts with 8 routed per token plus a shared expert, hidden 2048, 16 Q /
|
| 20 |
+
2 KV heads at head_dim 256, vocab 248,320, 262,144 native context. Display name
|
| 21 |
+
`Janus-27B` → **`Janus-35B`**, and the Hub slug now matches the model again.
|
| 22 |
+
- **The bundled blob is upstream's own Q4_K_M, byte-for-byte.**
|
| 23 |
+
`Janus-35B-A3B.Q4_K_M.gguf`, 21,233,608,512 bytes, sha256
|
| 24 |
+
`b82f9f2155eb9c07139d48cc3a37880cf9d0edcca345ef6ad829b62941ccbb82` — identical to
|
| 25 |
+
`llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF`'s Q4_K_M and to the blob this
|
| 26 |
+
repo shipped until `da34705`. HF still holds that LFS object for this repo (checked
|
| 27 |
+
against the LFS batch API), so it was restored as a pointer-only commit: no
|
| 28 |
+
re-quantization, no re-upload. `Janus-27B.Q4_K_M.gguf` is no longer published and
|
| 29 |
+
remains in git history.
|
| 30 |
+
- **The MTP narrative inverts.** 0.2.0 made the MTP strip "a required build step, not
|
| 31 |
+
a safety net" because the dense 3.8 base was MTP-preserved. This quant has no NextN
|
| 32 |
+
block at all — `block_count` 40, `blk.0`…`blk.39`, 733 tensors, no `nextn` key, read
|
| 33 |
+
from the published file's header by range request — so `scripts/strip_mtp.py` is a
|
| 34 |
+
defensive no-op here and stays in the build path only for anyone pointing it at
|
| 35 |
+
llmfan46's separate `…-Native-MTP-Preserved-GGUF` variant (whose Q4_K_M has
|
| 36 |
+
`block_count` 41 with a NextN block at index 40, not 64).
|
| 37 |
+
- **The GGUF arch stamp widens back to `qwen35moe`** — llama.cpp's MoE entry
|
| 38 |
+
(`src/models/qwen35moe.cpp`, `case 40: LLM_TYPE_35B_A3B`), handled through Ollama's
|
| 39 |
+
llama.cpp compat shim at v0.33.3, v0.34.0 and v0.34.2 (source, 2026-09-18). 0.2.0's
|
| 40 |
+
claim that `qwen35moe` "no longer appears in any live file" is superseded.
|
| 41 |
+
- **`moe-routing.svg` is back**, restored verbatim from `7ed28e6`: the 16×16 grid of
|
| 42 |
+
256 expert dots with 8 lit at a time is literally this model's routing again. The
|
| 43 |
+
banner wordmark is `JANUS-35B` with the `Qwen 3.6 Heretic · MoE 35B/3B · Fable 5`
|
| 44 |
+
subtitle, and `banner.png` is the pre-rebase raster byte-for-byte, so there is no
|
| 45 |
+
re-rasterization drift.
|
| 46 |
+
|
| 47 |
+
### Removed (`chat_template.jinja`, and the behaviour that lived in it)
|
| 48 |
+
|
| 49 |
+
- **This repo no longer ships a Jinja chat template.** The file was created *after*
|
| 50 |
+
the rebase as a patch of the dense base's own `qwen3.8-safe-v3.1` template; Qwen 3.6's
|
| 51 |
+
template has no `reasoning_effort`, no `tool_call_format` and no `preserve_reasoning`
|
| 52 |
+
to patch, so there was nothing to carry across. The GGUF keeps the base model's own
|
| 53 |
+
embedded template (11,793 chars), which is what llama.cpp, LM Studio and KoboldCpp
|
| 54 |
+
now render. `scripts/verify_arch.py` gates its embedded-template comparison on the
|
| 55 |
+
repo file existing, so that check correctly skips.
|
| 56 |
+
- **Gone with it on the llama.cpp path:** the `reasoning_effort` tier table (Ollama
|
| 57 |
+
still has it, via the Go template and Ollama's think levels), the `tool_call_format`
|
| 58 |
+
JSON opt-in, and the "conflicting `preserve_thinking` / `preserve_reasoning` raises
|
| 59 |
+
HTTP 500" failure mode — Qwen 3.6's template has no such check and cannot raise that
|
| 60 |
+
way. What is unchanged: the XML tool-call wire format (its instruction literal is
|
| 61 |
+
byte-identical to 3.8's) and reasoning replay, which is that template's own default
|
| 62 |
+
(`preserve_thinking` defaults true, reading `message.reasoning_content`).
|
| 63 |
+
- **One behaviour gets quieter, and worse.** Handed `function.arguments` as a JSON
|
| 64 |
+
string, the 3.8 template raised; Qwen 3.6's emits the string as-is inside
|
| 65 |
+
`<function=…>` with an upstream comment telling you to normalise first. llama-server,
|
| 66 |
+
KoboldCpp and LM Studio all convert to a mapping before rendering, so in practice the
|
| 67 |
+
XML form still renders — but a loader that passes the string through now produces a
|
| 68 |
+
malformed prompt instead of an error.
|
| 69 |
+
- `scripts/check_chat_template.py` and check.sh's check 8 are removed with the file.
|
| 70 |
+
|
| 71 |
+
### Changed (every measured number on the card)
|
| 72 |
+
|
| 73 |
+
- **All memory and throughput figures measured on the dense 27B are deleted, not
|
| 74 |
+
rescaled.** The geometry moves in both directions — weights 16.1 → 19.78 GiB, but KV
|
| 75 |
+
from 16 full-attention layers × 4 KV heads to 10 × 2 — so the KV cache is 20,480
|
| 76 |
+
B/token at f16 (0.625 GiB per 32K, ~5.0 GiB at the 262144 default), under a third of
|
| 77 |
+
the dense figure, and the derived total at the default window is ~25 GiB f16 /
|
| 78 |
+
~22.4 GiB q8_0 against the 27B's measured ~34/~26. Those totals are **derived from
|
| 79 |
+
GGUF geometry and explicitly labelled as such**; the recurrent state and compute
|
| 80 |
+
buffer are not derivable and are not yet measured on this blob.
|
| 81 |
+
- **No tok/s figures are published at all.** A 35B-A3B MoE touches ~3B parameters per
|
| 82 |
+
token instead of 27B; rescaling the 4.97 tok/s CPU measurement or the Strix Halo
|
| 83 |
+
offload series would be invention. The iGPU *mechanism* (mmap off, GTT is system RAM,
|
| 84 |
+
`LLAMA_ARG_NO_HOST=1`) is loader-side and is kept.
|
| 85 |
+
- The native context window is unchanged at 262,144 / 1,010,000 with YaRN, so every
|
| 86 |
+
window figure and the `--rope-scale 3.853` recipe survive verbatim.
|
| 87 |
+
|
| 88 |
+
### Fixed (found while auditing the card, unrelated to the swap)
|
| 89 |
+
|
| 90 |
+
- **Front-matter routed the repo wrongly.** `library_name: transformers` and
|
| 91 |
+
`pipeline_tag: image-text-to-text` were inherited verbatim from `64b629a` and never
|
| 92 |
+
revisited, while the repo has never contained a `config.json`, tokenizer or
|
| 93 |
+
safetensors — so HF offered a Transformers snippet that cannot resolve. Now
|
| 94 |
+
`library_name: gguf` and `pipeline_tag: text-generation`, with `qwen36`, `moe` and
|
| 95 |
+
`ollama` added to the tags.
|
| 96 |
+
- **`.gitignore` would have silently dropped the new blob**: the whitelist was
|
| 97 |
+
`!Janus-27B.*.gguf` against a `*.gguf` rule. Now `!Janus-35B-A3B.*.gguf`, verified
|
| 98 |
+
with `git check-ignore`.
|
| 99 |
+
- **The loader table's `Ollama 0.24+` floor** was the card's only undated, unsourced
|
| 100 |
+
version claim and predates the arch change. Replaced with the three versions actually
|
| 101 |
+
checked in source (0.33.3, 0.34.0, 0.34.2).
|
| 102 |
+
|
| 103 |
+
### Not done in this release
|
| 104 |
+
|
| 105 |
+
- No live run. `scripts/live_check.sh`, a tool-call round-trip, `verify_arch.py` and
|
| 106 |
+
any memory measurement all need the 21 GB blob on disk; none has been executed
|
| 107 |
+
against this model since the reversal, and the card says so wherever it matters.
|
| 108 |
+
- The local `janus` Ollama tag on the maintainer's host still holds the dense 27B
|
| 109 |
+
bundle until it is rebuilt.
|
| 110 |
+
|
| 111 |
## [0.6.5] — 2026-09-17
|
| 112 |
|
| 113 |
### Documentation (upstream claims re-checked against Ollama 0.34.2)
|
|
@@ -1,7 +1,7 @@
|
|
| 1 |
cff-version: 1.2.0
|
| 2 |
-
version: 0.
|
| 3 |
-
date-released: "2026-09-
|
| 4 |
-
title: "Janus-
|
| 5 |
message: "If you use this model card or its accompanying files, please cite as below."
|
| 6 |
type: software
|
| 7 |
authors:
|
|
@@ -10,22 +10,26 @@ authors:
|
|
| 10 |
repository-code: "https://huggingface.co/FoolDev/Janus-35B-HERETIC"
|
| 11 |
url: "https://huggingface.co/FoolDev/Janus-35B-HERETIC"
|
| 12 |
abstract: >-
|
| 13 |
-
Janus-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
|
|
|
|
|
|
|
|
|
| 25 |
keywords:
|
| 26 |
- qwen
|
| 27 |
-
- qwen3.
|
| 28 |
-
-
|
|
|
|
| 29 |
- abliteration
|
| 30 |
- distillation
|
| 31 |
- reasoning
|
|
@@ -35,17 +39,17 @@ keywords:
|
|
| 35 |
license: Apache-2.0
|
| 36 |
references:
|
| 37 |
- type: software
|
| 38 |
-
title: "Qwen3.
|
| 39 |
authors:
|
| 40 |
- name: llmfan46
|
| 41 |
-
url: "https://huggingface.co/llmfan46/Qwen3.
|
| 42 |
- type: software
|
| 43 |
-
title: "Qwen3.
|
| 44 |
authors:
|
| 45 |
- name: Alibaba Qwen Team
|
| 46 |
-
url: "https://huggingface.co/Qwen/Qwen3.
|
| 47 |
- type: software
|
| 48 |
-
title: "Thanatos-27B (sibling, ARA abliteration)"
|
| 49 |
authors:
|
| 50 |
- name: FoolDev
|
| 51 |
url: "https://huggingface.co/FoolDev/Thanatos-27B-HERETIC"
|
|
|
|
| 1 |
cff-version: 1.2.0
|
| 2 |
+
version: 0.7.0
|
| 3 |
+
date-released: "2026-09-18"
|
| 4 |
+
title: "Janus-35B: A Sparse Distillation Wrapper for llmfan46's Qwen 3.6 35B-A3B Uncensored Heretic"
|
| 5 |
message: "If you use this model card or its accompanying files, please cite as below."
|
| 6 |
type: software
|
| 7 |
authors:
|
|
|
|
| 10 |
repository-code: "https://huggingface.co/FoolDev/Janus-35B-HERETIC"
|
| 11 |
url: "https://huggingface.co/FoolDev/Janus-35B-HERETIC"
|
| 12 |
abstract: >-
|
| 13 |
+
Janus-35B is a personal repackaging of llmfan46's uncensored Heretic
|
| 14 |
+
abliteration of the Qwen 3.6 35B-A3B mixture-of-experts base (~34.7B
|
| 15 |
+
parameters total, ~3B active per token, 40 layers, 256 experts with 8
|
| 16 |
+
routed per token plus a shared expert, gated DeltaNet linear attention
|
| 17 |
+
with full attention every 4th layer) with Claude Fable 5 in the
|
| 18 |
+
reasoning teacher slot. The repository ships an Ollama Modelfile, the HF
|
| 19 |
+
Ollama-bridge files (template / system / params), sampling defaults, and
|
| 20 |
+
a bundled Q4_K_M GGUF (21.23 GB) so the HF "Use this model" widget
|
| 21 |
+
surfaces a one-liner Ollama snippet. That GGUF is upstream's own Q4_K_M
|
| 22 |
+
byte-for-byte: it carries no MTP / NextN block to strip and keeps the
|
| 23 |
+
base model's own embedded chat template. Other quants, the BF16 vision
|
| 24 |
+
projector, and the Heretic safetensors
|
| 25 |
+
(llmfan46/Qwen3.6-35B-A3B-uncensored-heretic), which keep the MTP layer
|
| 26 |
+
for vLLM / SGLang, are pulled from upstream on demand rather than
|
| 27 |
+
redistributed.
|
| 28 |
keywords:
|
| 29 |
- qwen
|
| 30 |
+
- qwen3.6
|
| 31 |
+
- moe
|
| 32 |
+
- mixture-of-experts
|
| 33 |
- abliteration
|
| 34 |
- distillation
|
| 35 |
- reasoning
|
|
|
|
| 39 |
license: Apache-2.0
|
| 40 |
references:
|
| 41 |
- type: software
|
| 42 |
+
title: "Qwen3.6-35B-A3B-uncensored-heretic (immediate base)"
|
| 43 |
authors:
|
| 44 |
- name: llmfan46
|
| 45 |
+
url: "https://huggingface.co/llmfan46/Qwen3.6-35B-A3B-uncensored-heretic"
|
| 46 |
- type: software
|
| 47 |
+
title: "Qwen3.6-35B-A3B (upstream base)"
|
| 48 |
authors:
|
| 49 |
- name: Alibaba Qwen Team
|
| 50 |
+
url: "https://huggingface.co/Qwen/Qwen3.6-35B-A3B"
|
| 51 |
- type: software
|
| 52 |
+
title: "Thanatos-27B (sibling, dense Qwen 3.8 ARA abliteration)"
|
| 53 |
authors:
|
| 54 |
- name: FoolDev
|
| 55 |
url: "https://huggingface.co/FoolDev/Thanatos-27B-HERETIC"
|
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b82f9f2155eb9c07139d48cc3a37880cf9d0edcca345ef6ad829b62941ccbb82
|
| 3 |
+
size 21233608512
|
|
@@ -1,4 +1,4 @@
|
|
| 1 |
-
# Janus-
|
| 2 |
#
|
| 3 |
# All work is delegated to scripts/* — this file just gives common
|
| 4 |
# operations short, discoverable names.
|
|
@@ -11,9 +11,9 @@
|
|
| 11 |
# MODEL model tag for smoke (default: $(TAG))
|
| 12 |
#
|
| 13 |
# Examples:
|
| 14 |
-
# make build # Q4_K_M from the Heretic GGUF repo (MTP-
|
| 15 |
# make build QUANT=Q5_K_M # higher-quality quant
|
| 16 |
-
# make build GGUF_PATH=~/models/Qwen3.
|
| 17 |
# make load-bundle # this repo's bundled GGUF -> local Ollama tag (smudge LFS if needed)
|
| 18 |
# make smoke
|
| 19 |
# make check
|
|
@@ -39,7 +39,7 @@ ifdef GGUF_PATH
|
|
| 39 |
@echo " GGUF_PATH=$(GGUF_PATH)"
|
| 40 |
endif
|
| 41 |
|
| 42 |
-
build: ## Download a Heretic GGUF,
|
| 43 |
GGUF_PATH=$(GGUF_PATH) TAG=$(TAG) ./scripts/build.sh $(QUANT)
|
| 44 |
|
| 45 |
load-bundle: ## Load THIS repo's bundled GGUF into a local Ollama tag (smudge LFS + ollama create).
|
|
@@ -60,7 +60,7 @@ bench: ## Measure tok/s using Ollama's eval timing (3 prompts).
|
|
| 60 |
mmproj: ## Fetch the vision projector for llama.cpp (Ollama vision is broken upstream).
|
| 61 |
./scripts/fetch_vision.sh $(PRECISION)
|
| 62 |
|
| 63 |
-
check: ## Lint, bridge-file sync,
|
| 64 |
./scripts/check.sh
|
| 65 |
|
| 66 |
live-check: ## Live end-to-end checks in an isolated Ollama (GGUF=path or GGUF_PATH=path; tens of minutes on CPU).
|
|
@@ -74,6 +74,6 @@ hooks: ## Install scripts/check.sh as the git pre-commit hook.
|
|
| 74 |
|
| 75 |
clean: ## Remove local GGUF copies and ephemeral caches in this repo.
|
| 76 |
@echo "[*] removing local GGUFs and ephemeral caches in $$PWD"
|
| 77 |
-
@rm -f ./Qwen3.
|
| 78 |
@rm -rf ./.cache __pycache__ examples/__pycache__
|
| 79 |
@echo "[+] clean"
|
|
|
|
| 1 |
+
# Janus-35B convenience Makefile.
|
| 2 |
#
|
| 3 |
# All work is delegated to scripts/* — this file just gives common
|
| 4 |
# operations short, discoverable names.
|
|
|
|
| 11 |
# MODEL model tag for smoke (default: $(TAG))
|
| 12 |
#
|
| 13 |
# Examples:
|
| 14 |
+
# make build # Q4_K_M from the Heretic GGUF repo (the MTP strip no-ops on these quants)
|
| 15 |
# make build QUANT=Q5_K_M # higher-quality quant
|
| 16 |
+
# make build GGUF_PATH=~/models/Qwen3.6-35B-A3B-uncensored-heretic-Q4_K_M.gguf
|
| 17 |
# make load-bundle # this repo's bundled GGUF -> local Ollama tag (smudge LFS if needed)
|
| 18 |
# make smoke
|
| 19 |
# make check
|
|
|
|
| 39 |
@echo " GGUF_PATH=$(GGUF_PATH)"
|
| 40 |
endif
|
| 41 |
|
| 42 |
+
build: ## Download a Heretic GGUF, run the MTP strip (a no-op on these quants), and run 'ollama create'.
|
| 43 |
GGUF_PATH=$(GGUF_PATH) TAG=$(TAG) ./scripts/build.sh $(QUANT)
|
| 44 |
|
| 45 |
load-bundle: ## Load THIS repo's bundled GGUF into a local Ollama tag (smudge LFS + ollama create).
|
|
|
|
| 60 |
mmproj: ## Fetch the vision projector for llama.cpp (Ollama vision is broken upstream).
|
| 61 |
./scripts/fetch_vision.sh $(PRECISION)
|
| 62 |
|
| 63 |
+
check: ## Lint, bridge-file sync, the Go-template guard, and that the Modelfile's FROM is the GGUF this repo ships.
|
| 64 |
./scripts/check.sh
|
| 65 |
|
| 66 |
live-check: ## Live end-to-end checks in an isolated Ollama (GGUF=path or GGUF_PATH=path; tens of minutes on CPU).
|
|
|
|
| 74 |
|
| 75 |
clean: ## Remove local GGUF copies and ephemeral caches in this repo.
|
| 76 |
@echo "[*] removing local GGUFs and ephemeral caches in $$PWD"
|
| 77 |
+
@rm -f ./Qwen3.6-35B-A3B-uncensored-*.gguf ./mmproj-*.gguf ./*.stripped.gguf
|
| 78 |
@rm -rf ./.cache __pycache__ examples/__pycache__
|
| 79 |
@echo "[+] clean"
|
|
@@ -1,23 +1,25 @@
|
|
| 1 |
-
FROM ./Janus-
|
| 2 |
|
| 3 |
-
# The bundled GGUF
|
| 4 |
-
# (llmfan46/Qwen3.
|
| 5 |
-
# MTP
|
| 6 |
-
#
|
| 7 |
-
#
|
| 8 |
-
#
|
| 9 |
-
#
|
| 10 |
-
#
|
|
|
|
|
|
|
| 11 |
|
| 12 |
-
# Chat template — Qwen 3.
|
| 13 |
# tool-calling blocks Ollama's capability detector looks for. Without a
|
| 14 |
# TEMPLATE that references .Tools and .ToolCalls, an Ollama that uses this
|
| 15 |
# template anyway (OLLAMA_GO_TEMPLATE=1 - per Ollama's source, not tested live -
|
| 16 |
# or one that predates the template comparison described below) rejects any
|
| 17 |
# request carrying a `tools` array with `<model> does not support tools`;
|
| 18 |
-
# Ollama 0.33.3 would instead switch to the GGUF's embedded template. Same template as the
|
| 19 |
-
# (FoolDev/Thanatos-27B-HERETIC) —
|
| 20 |
-
#
|
| 21 |
#
|
| 22 |
# Thinking IS replayed across turns: every assistant message that carries
|
| 23 |
# reasoning renders <think>...</think>, earlier turns included, where Qwen's
|
|
@@ -25,9 +27,10 @@ FROM ./Janus-27B.Q4_K_M.gguf
|
|
| 25 |
# The client must send the reasoning back - per Ollama's source, /api/chat reads
|
| 26 |
# each assistant message's `thinking` field and /v1/chat/completions reads only
|
| 27 |
# `reasoning` (it drops `reasoning_content` and `thinking` silently). Every
|
| 28 |
-
# retained trace stays in the prompt
|
| 29 |
-
#
|
| 30 |
-
# the
|
|
|
|
| 31 |
#
|
| 32 |
# The thinking block in the assistant branch is load-bearing for a second
|
| 33 |
# reason. Ollama infers a Go template's "thinking" capability from a `.Thinking` field inside
|
|
@@ -35,7 +38,8 @@ FROM ./Janus-27B.Q4_K_M.gguf
|
|
| 35 |
# the selection code is unchanged through 0.34.2, checked in source)
|
| 36 |
# compare that capability list with the GGUF's embedded Jinja template at load
|
| 37 |
# time, switching to the embedded template if that one advertises more. Delete
|
| 38 |
-
# the block and Ollama switches: in one test on
|
|
|
|
| 39 |
# (a separate raw generation showed the model drafting the call inside <think>
|
| 40 |
# first; that the parser matched such a draft is inferred, not confirmed). Per Ollama's source (not tested live), a
|
| 41 |
# setup forcing this template (OLLAMA_GO_TEMPLATE=1) would also reject thinking
|
|
@@ -45,9 +49,12 @@ FROM ./Janus-27B.Q4_K_M.gguf
|
|
| 45 |
# $lastUserIdx loop at the top of the template is kept for it; do not delete the
|
| 46 |
# block.
|
| 47 |
#
|
| 48 |
-
# Reasoning effort: the block at the top of the template
|
| 49 |
-
#
|
| 50 |
-
#
|
|
|
|
|
|
|
|
|
|
| 51 |
# high/xhigh/max/ultra -> high or max (xhigh line), low/minimal -> low (low
|
| 52 |
# line), medium and unset -> medium (no line), none -> thinking off. The
|
| 53 |
# outer `if $.ThinkLevel` keeps the template rendering on Ollama builds that
|
|
@@ -150,47 +157,49 @@ Behavior rules:
|
|
| 150 |
|
| 151 |
# Hardware notes
|
| 152 |
# --------------
|
| 153 |
-
# This Q4_K_M is
|
| 154 |
-
#
|
| 155 |
-
#
|
| 156 |
-
#
|
| 157 |
-
#
|
| 158 |
-
#
|
| 159 |
-
#
|
| 160 |
-
#
|
| 161 |
-
#
|
| 162 |
-
#
|
| 163 |
-
#
|
| 164 |
-
#
|
| 165 |
-
#
|
| 166 |
-
#
|
| 167 |
-
#
|
| 168 |
-
#
|
|
|
|
|
|
|
|
|
|
|
|
|
| 169 |
# No YaRN rope-scaling is baked in this GGUF, so output
|
| 170 |
# past ~262K degrades; treat 1.01M as an advertised
|
| 171 |
# ceiling.
|
| 172 |
#
|
| 173 |
-
# This is a
|
| 174 |
-
#
|
| 175 |
-
#
|
|
|
|
|
|
|
| 176 |
#
|
| 177 |
-
# Working configurations (rows assume the 262144 default: ~
|
| 178 |
-
# cache, ~
|
| 179 |
# ✓ Single H100 80GB / A100 80GB — full GPU offload
|
| 180 |
# ✓ RTX 5090 32GB / RTX 4090 24GB + 32GB RAM — partial offload
|
| 181 |
# ✓ Mac Studio M2/M3 Ultra 64GB+ — unified memory
|
| 182 |
-
# ✓ Linux box with 48GB+ RAM (CPU-only) — CPU inference
|
| 183 |
# ⚠ 32GB hosts (e.g. ASUS ROG Flow Z13) — set OLLAMA_KV_CACHE_TYPE=q8_0
|
| 184 |
-
#
|
| 185 |
-
# (65536 is ~19-21 GiB)
|
| 186 |
#
|
| 187 |
-
# Throughput
|
| 188 |
-
#
|
| 189 |
-
#
|
| 190 |
-
#
|
| 191 |
-
# LLAMA_ARG_NO_HOST=1, num_ctx 4096): 5.7 tok/s at num_gpu 24 - see the README.
|
| 192 |
-
# The measurements that used to sit here were taken on this repo's previous base
|
| 193 |
-
# and do not transfer to a dense 27B, so they are removed rather than restated.
|
| 194 |
#
|
| 195 |
# To run on a 32 GB unified-memory laptop, override these in your local
|
| 196 |
# Modelfile copy (or via `/set parameter` in the interactive `ollama run` REPL):
|
|
@@ -198,4 +207,4 @@ Behavior rules:
|
|
| 198 |
# PARAMETER num_batch 256
|
| 199 |
#
|
| 200 |
# If you have ≥48 GB RAM but want partial GPU offload, set:
|
| 201 |
-
# PARAMETER num_gpu
|
|
|
|
| 1 |
+
FROM ./Janus-35B-A3B.Q4_K_M.gguf
|
| 2 |
|
| 3 |
+
# The bundled GGUF carries no MTP / NextN block. The base
|
| 4 |
+
# (llmfan46/Qwen3.6-35B-A3B-uncensored-heretic) publishes its GGUFs already
|
| 5 |
+
# MTP-clean — blk.0…blk.39 only, block_count 40, no nextn_predict_layers key
|
| 6 |
+
# (read from the bundled file's own header, 2026-09-18) — so scripts/strip_mtp.py
|
| 7 |
+
# is a defensive no-op here, not a required build step. The separate
|
| 8 |
+
# llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-Native-MTP-Preserved-GGUF variant
|
| 9 |
+
# does keep the MTP head (its Q4_K_M has block_count 41, NextN block at index
|
| 10 |
+
# 40) and would need the strip; this repo does not ship it. If you want the MTP
|
| 11 |
+
# head, run the upstream safetensors under vLLM/SGLang, or take that variant's
|
| 12 |
+
# GGUF.
|
| 13 |
|
| 14 |
+
# Chat template — Qwen 3.6 ChatML in Ollama Go-template form, with the
|
| 15 |
# tool-calling blocks Ollama's capability detector looks for. Without a
|
| 16 |
# TEMPLATE that references .Tools and .ToolCalls, an Ollama that uses this
|
| 17 |
# template anyway (OLLAMA_GO_TEMPLATE=1 - per Ollama's source, not tested live -
|
| 18 |
# or one that predates the template comparison described below) rejects any
|
| 19 |
# request carrying a `tools` array with `<model> does not support tools`;
|
| 20 |
+
# Ollama 0.33.3 would instead switch to the GGUF's embedded template. Same template as the dense
|
| 21 |
+
# 27B sibling (FoolDev/Thanatos-27B-HERETIC) — the ChatML wire format is
|
| 22 |
+
# identical across Qwen 3.6 and 3.8.
|
| 23 |
#
|
| 24 |
# Thinking IS replayed across turns: every assistant message that carries
|
| 25 |
# reasoning renders <think>...</think>, earlier turns included, where Qwen's
|
|
|
|
| 27 |
# The client must send the reasoning back - per Ollama's source, /api/chat reads
|
| 28 |
# each assistant message's `thinking` field and /v1/chat/completions reads only
|
| 29 |
# `reasoning` (it drops `reasoning_content` and `thinking` silently). Every
|
| 30 |
+
# retained trace stays in the prompt and prefill grows to match, which is the
|
| 31 |
+
# slow part on a CPU-only host. The prompt-token counts that used to sit here
|
| 32 |
+
# were measured on the dense 27B this repo shipped in the interim; they are
|
| 33 |
+
# removed rather than carried over, pending re-measurement.
|
| 34 |
#
|
| 35 |
# The thinking block in the assistant branch is load-bearing for a second
|
| 36 |
# reason. Ollama infers a Go template's "thinking" capability from a `.Thinking` field inside
|
|
|
|
| 38 |
# the selection code is unchanged through 0.34.2, checked in source)
|
| 39 |
# compare that capability list with the GGUF's embedded Jinja template at load
|
| 40 |
# time, switching to the embedded template if that one advertises more. Delete
|
| 41 |
+
# the block and Ollama switches: in one test on the dense 27B this repo shipped
|
| 42 |
+
# in the interim a single tool call then came back twice
|
| 43 |
# (a separate raw generation showed the model drafting the call inside <think>
|
| 44 |
# first; that the parser matched such a draft is inferred, not confirmed). Per Ollama's source (not tested live), a
|
| 45 |
# setup forcing this template (OLLAMA_GO_TEMPLATE=1) would also reject thinking
|
|
|
|
| 49 |
# $lastUserIdx loop at the top of the template is kept for it; do not delete the
|
| 50 |
# block.
|
| 51 |
#
|
| 52 |
+
# Reasoning effort: the block at the top of the template injects an effort
|
| 53 |
+
# instruction keyed on Ollama's think level ($.ThinkLevel), not the request
|
| 54 |
+
# value. It is Ollama-only now — this repo no longer ships chat_template.jinja,
|
| 55 |
+
# and the Qwen 3.6 base's own embedded template, which governs the llama.cpp
|
| 56 |
+
# path from here on, has no reasoning_effort handling at all.
|
| 57 |
+
# Ollama folds `reasoning_effort` into four levels first:
|
| 58 |
# high/xhigh/max/ultra -> high or max (xhigh line), low/minimal -> low (low
|
| 59 |
# line), medium and unset -> medium (no line), none -> thinking off. The
|
| 60 |
# outer `if $.ThinkLevel` keeps the template rendering on Ollama builds that
|
|
|
|
| 157 |
|
| 158 |
# Hardware notes
|
| 159 |
# --------------
|
| 160 |
+
# This Q4_K_M is 21,233,608,512 bytes on disk — 21.23 GB decimal, 19.78 GiB.
|
| 161 |
+
# No runtime memory figures for this build yet: the ones that used to sit here
|
| 162 |
+
# were measured on the dense 27B this repo shipped in the interim and do not
|
| 163 |
+
# transfer to a 35B-A3B MoE, so they are removed rather than rescaled. What
|
| 164 |
+
# follows is derived from the bundled GGUF's own header (read 2026-09-18),
|
| 165 |
+
# not measured on a running model:
|
| 166 |
+
# weights 19.78 GiB on disk
|
| 167 |
+
# KV cache only 10 of the 40 layers are full-attention (indices
|
| 168 |
+
# 3, 7, ... 39); 10 layers x 2 KV heads x (256 + 256)
|
| 169 |
+
# dims x 2 bytes = 20,480 B/token, i.e. 0.625 GiB per 32K
|
| 170 |
+
# at f16 (Ollama's default) and about 0.33 GiB per 32K at
|
| 171 |
+
# q8_0 -> ~5.0 GiB at the 262144 default with f16,
|
| 172 |
+
# ~2.66 GiB with q8_0
|
| 173 |
+
# recurrent state fixed-size, for the 30 Gated-DeltaNet layers; not
|
| 174 |
+
# measured on this build
|
| 175 |
+
# compute buffer not measured on this build
|
| 176 |
+
# floor ~24.8 GiB at the 262144 default with the f16 cache
|
| 177 |
+
# (19.78 weights + 5.0 KV), ~22.4 GiB with
|
| 178 |
+
# OLLAMA_KV_CACHE_TYPE=q8_0 — a floor, before the
|
| 179 |
+
# recurrent state and compute buffer.
|
| 180 |
# No YaRN rope-scaling is baked in this GGUF, so output
|
| 181 |
# past ~262K degrades; treat 1.01M as an advertised
|
| 182 |
# ceiling.
|
| 183 |
#
|
| 184 |
+
# This is a 35B-A3B MoE across 40 layers: 256 experts, 8 routed per token plus
|
| 185 |
+
# one shared expert, ~34.7B parameters total and ~3B active per token (derived
|
| 186 |
+
# from the same header). The active count cuts compute per token, not the memory
|
| 187 |
+
# floor — every expert has to be resident, so budget for the whole 19.78 GiB of
|
| 188 |
+
# weights.
|
| 189 |
#
|
| 190 |
+
# Working configurations (rows assume the 262144 default: a ~24.8 GiB floor with
|
| 191 |
+
# the f16 cache, ~22.4 GiB with q8_0):
|
| 192 |
# ✓ Single H100 80GB / A100 80GB — full GPU offload
|
| 193 |
# ✓ RTX 5090 32GB / RTX 4090 24GB + 32GB RAM — partial offload
|
| 194 |
# ✓ Mac Studio M2/M3 Ultra 64GB+ — unified memory
|
| 195 |
+
# ✓ Linux box with 48GB+ RAM (CPU-only) — CPU inference
|
| 196 |
# ⚠ 32GB hosts (e.g. ASUS ROG Flow Z13) — set OLLAMA_KV_CACHE_TYPE=q8_0
|
| 197 |
+
# or lower num_ctx
|
|
|
|
| 198 |
#
|
| 199 |
+
# Throughput: no tok/s figures for this build yet. The measurements that used to
|
| 200 |
+
# sit here were taken on the dense 27B and do not transfer to an MoE with ~3B
|
| 201 |
+
# active parameters per token, so they are removed rather than rescaled —
|
| 202 |
+
# pending re-measurement.
|
|
|
|
|
|
|
|
|
|
| 203 |
#
|
| 204 |
# To run on a 32 GB unified-memory laptop, override these in your local
|
| 205 |
# Modelfile copy (or via `/set parameter` in the interactive `ollama run` REPL):
|
|
|
|
| 207 |
# PARAMETER num_batch 256
|
| 208 |
#
|
| 209 |
# If you have ≥48 GB RAM but want partial GPU offload, set:
|
| 210 |
+
# PARAMETER num_gpu 24 # offload most layers (model has 40)
|
|
@@ -1,7 +1,7 @@
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
base_model:
|
| 4 |
-
- llmfan46/Qwen3.
|
| 5 |
base_model_relation: quantized
|
| 6 |
datasets:
|
| 7 |
- Crownelius/Creative_Writing_ShareGPT_Enhanced
|
|
@@ -34,41 +34,44 @@ language:
|
|
| 34 |
- fi
|
| 35 |
- el
|
| 36 |
tags:
|
| 37 |
-
-
|
|
|
|
| 38 |
- conversational
|
| 39 |
- multimodal
|
| 40 |
- agent
|
| 41 |
- gguf
|
|
|
|
| 42 |
- heretic
|
| 43 |
- uncensored
|
| 44 |
- reasoning
|
| 45 |
- distillation
|
| 46 |
-
library_name:
|
| 47 |
-
pipeline_tag:
|
| 48 |
---
|
| 49 |
|
| 50 |
-
<img src="https://huggingface.co/FoolDev/Janus-35B-HERETIC/resolve/main/banner.svg" alt="Janus-
|
| 51 |
|
| 52 |
[](https://opensource.org/licenses/Apache-2.0)
|
| 53 |
-
[](https://huggingface.co/llmfan46/Qwen3.
|
| 54 |
-
[](#whats-here)
|
| 56 |
[](https://huggingface.co/FoolDev/Thanatos-27B-HERETIC)
|
| 57 |
|
| 58 |
-
<a href="https://buymeacoffee.com/cardoffoolm"><img src="https://huggingface.co/FoolDev/Janus-35B-HERETIC/resolve/main/bmc-banner.svg" alt="Buy me a coffee — support Janus-
|
| 59 |
|
| 60 |
-
# Janus-
|
| 61 |
|
| 62 |
-
> **Flagship Reasoning.
|
| 63 |
-
> *llmfan46's Heretic abliteration of Qwen 3.
|
| 64 |
|
| 65 |
-
**`Architecture:`** `Qwen 3.
|
| 66 |
|
| 67 |
-
A personal fork of [`llmfan46/Qwen3.
|
| 68 |
|
| 69 |
-
> **Provenance.** This repo repackages
|
| 70 |
-
> here.** What ships is
|
| 71 |
-
>
|
|
|
|
| 72 |
> list in the frontmatter are carried over from earlier revisions of this card.
|
| 73 |
> **The current base does not claim them** — its card lists no datasets and names
|
| 74 |
> no teacher — and nothing in this repo reproduces or measures them. One listed
|
|
@@ -77,10 +80,11 @@ A personal fork of [`llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Pr
|
|
| 77 |
> attribution, not as a training record.
|
| 78 |
> `base_model_relation` is `quantized` for the same reason.
|
| 79 |
|
| 80 |
-
> **On the repo name:** the model is **Janus-
|
| 81 |
-
> `FoolDev/Janus-35B-HERETIC`
|
| 82 |
-
>
|
| 83 |
-
>
|
|
|
|
| 84 |
|
| 85 |
## TL;DR
|
| 86 |
|
|
@@ -104,11 +108,13 @@ ollama create janus -f Modelfile && ollama run janus
|
|
| 104 |
|
| 105 |
After either path, `ollama show janus` lists `completion`, `tools`,
|
| 106 |
and `thinking` under Capabilities. Hardware: the default `num_ctx` is the full
|
| 107 |
-
**262,144** native window —
|
| 108 |
-
with `OLLAMA_KV_CACHE_TYPE=q8_0`
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
[Hardware requirements](#hardware-requirements).
|
|
|
|
|
|
|
| 112 |
|
| 113 |
To halve the KV cache, start the Ollama server with the q8_0 cache and flash
|
| 114 |
attention (for a systemd install, add them as `Environment=` lines with
|
|
@@ -122,74 +128,80 @@ OLLAMA_KV_CACHE_TYPE=q8_0 OLLAMA_FLASH_ATTENTION=1 ollama serve
|
|
| 122 |
|
| 123 |
| File | Use |
|
| 124 |
|---|---|
|
| 125 |
-
| `Janus-
|
| 126 |
| `Modelfile` | Ollama wrapper for **local** builds (`ollama create janus -f Modelfile`) — carries the Go template that Ollama runs in preference to the GGUF's embedded one (see [Which template Ollama runs](#which-template-ollama-runs)). |
|
| 127 |
| `template`, `system`, `params` | Used by HF's Ollama bridge when users `ollama run hf.co/FoolDev/Janus-35B-HERETIC` directly. The bridge does **not** read `Modelfile` (see [HF Ollama docs](https://huggingface.co/docs/hub/en/ollama)); it ingests these three root-level files instead. Kept in sync with the `Modelfile`'s `TEMPLATE` / `SYSTEM` / `PARAMETER` directives. |
|
| 128 |
-
| `
|
| 129 |
-
| `scripts/build.sh` | Pulls a GGUF from `llmfan46/Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-GGUF` (default Q4_K_M), runs it through `strip_mtp.py`, then runs `ollama create janus`. Quants published there: BF16, Q3_K_L, Q3_K_M, Q4_K_M, Q4_K_S, Q5_K_M, Q5_K_S, Q6_K, Q8_0. The bundled Q4_K_M is already this Heretic quant (stripped); use this to build the others locally. |
|
| 130 |
| `scripts/check_bridge_sync.py` | Run before pushing a `Modelfile` / `template` / `system` / `params` edit to verify the four configurations remain in sync. Exits 0 if in sync, 1 with a per-key diff if not. |
|
| 131 |
-
| `scripts/check.sh` | Local lint: `bash -n`, `shellcheck`, `py_compile`, footgun-grep, `Modelfile`-vs-bridge-files sync
|
| 132 |
-
| `scripts/
|
| 133 |
-
| `scripts/check_go_template.py` | Guards the Go `template`: Ollama's thinking detection, the condition that replays earlier turns' reasoning, the tool round trip and JSON tool signatures (check 9 in `check.sh`) |
|
| 134 |
| `scripts/live_check.sh` | Live end-to-end checks in an isolated, CPU-only Ollama (own port and model store; it refuses to run if Ollama reports a GPU): template selection, one tool call, string-argument replay, an earlier turn's reasoning replayed and a live tool chain's kept, every `reasoning_effort` value and the effort lines — 17 checks (`make live-check`) |
|
| 135 |
-
| `scripts/verify_arch.py` | Cross-checks the README Architecture bullets (layer count, hidden size,
|
| 136 |
-
| `scripts/strip_mtp.py` | Drops
|
| 137 |
| `scripts/smoke_test.sh` | Integration smoke test against a running Ollama daemon: server reachable, model loaded, `tools` capability present, chat round-trip, and no control-token leakage. `TOOLS_TEST=1` adds a tool-call round-trip. Defaults to `MODEL=janus`. |
|
| 138 |
| `scripts/bench.sh` | Measures tok/s from Ollama's `eval_count` / `eval_duration` over a short/medium/long prompt mix (with a discarded warmup). Defaults to `MODEL=janus`. |
|
| 139 |
-
| `scripts/load_bundle.sh` | Loads the bundled `Janus-
|
| 140 |
| `scripts/cap_ctx.sh` | Bakes a small-`num_ctx` local `janus` tag (bundled blob + this repo's Modelfile, `num_ctx`→4096, `num_batch` 256) for OpenAI `/v1` clients — which can't override the baked `num_ctx` and OOM on small hosts (see [Inference examples](#inference-examples)). Run `./scripts/cap_ctx.sh` (or `CTX=8192 ...`). |
|
| 141 |
-
| `scripts/fetch_vision.sh` | Downloads the vision projector (`Qwen3.
|
| 142 |
| `examples/` | Ready-to-run Python clients for Ollama, Transformers, and llama-cpp-python (text, tools, and vision — see `examples/README.md`) |
|
| 143 |
|
| 144 |
-
GGUF-only release. Pull the Heretic safetensors from [`llmfan46/Qwen3.
|
| 145 |
|
| 146 |
-
> ### Bundled blob status: the
|
| 147 |
>
|
| 148 |
-
> The bundled GGUF is **`Janus-
|
| 149 |
-
>
|
| 150 |
-
>
|
|
|
|
| 151 |
>
|
| 152 |
-
> It is `llmfan46/
|
| 153 |
-
> (sha256 `
|
| 154 |
-
>
|
| 155 |
-
>
|
| 156 |
-
>
|
| 157 |
-
> its metadata in the same pass, so llama.cpp and LM Studio's llama.cpp runtime get
|
| 158 |
-
> working tool calls with no extra flags (both tested; KoboldCpp and Jan were checked on 2026-09-14, see [Tool / function calling](#tool--function-calling)).
|
| 159 |
>
|
| 160 |
-
> Verified before publishing:
|
| 161 |
-
>
|
| 162 |
-
> `
|
| 163 |
-
> `
|
| 164 |
-
>
|
|
|
|
| 165 |
>
|
| 166 |
-
>
|
| 167 |
-
> `Janus-
|
| 168 |
-
>
|
| 169 |
|
| 170 |
## Architecture
|
| 171 |
|
| 172 |
-
|
| 173 |
-
Gated Attention layer every fourth block. No experts, no router — all 27B
|
| 174 |
-
parameters run on every token.
|
| 175 |
|
| 176 |
-
|
| 177 |
-
|
| 178 |
-
|
| 179 |
-
|
| 180 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 181 |
- 262 144 native context (extensible to 1 010 000 with YaRN, but YaRN is not enabled in the bundled GGUF)
|
| 182 |
-
- Vision + video supported by upstream (mmproj not included in this release)
|
| 183 |
- Vocab 248,320
|
| 184 |
-
- GGUF arch stamp `
|
|
|
|
|
|
|
|
|
|
| 185 |
|
| 186 |
-
**MTP head (
|
| 187 |
-
|
| 188 |
-
|
| 189 |
-
|
| 190 |
-
|
| 191 |
-
|
| 192 |
-
|
|
|
|
|
|
|
| 193 |
|
| 194 |
## Quick start
|
| 195 |
|
|
@@ -197,14 +209,18 @@ unstripped quant straight from the source GGUF repo.
|
|
| 197 |
|
| 198 |
Drop the GGUF into your loader of choice. The chat template is embedded in the GGUF metadata, and llama.cpp and LM Studio use it by default, so no `--chat-template` flag is needed (llama.cpp has no `--chat-template auto` mode; it would take "auto" as the template itself).
|
| 199 |
|
| 200 |
-
**Tool calling works out of the box** — the
|
| 201 |
|
| 202 |
```bash
|
| 203 |
-
llama-server -m Janus-
|
| 204 |
```
|
| 205 |
|
| 206 |
-
|
| 207 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 208 |
|
| 209 |
### Ollama
|
| 210 |
|
|
@@ -229,9 +245,10 @@ Once the model is loaded (via `ollama run janus`, `lms server`, or `llama-server
|
|
| 229 |
> The examples use `model: "janus"`, the tag from the local build (path B). If you pulled via the TL;DR one-liner instead, use the full tag `hf.co/FoolDev/Janus-35B-HERETIC`, or run `ollama cp hf.co/FoolDev/Janus-35B-HERETIC janus` once to create the short tag.
|
| 230 |
|
| 231 |
> **On memory-tight hosts, cap `num_ctx` first.** `/v1/chat/completions` (OpenAI-compat) has
|
| 232 |
-
> no `num_ctx` knob, so it loads at the baked **262,144** default (
|
| 233 |
-
> Ollama's default f16 cache
|
| 234 |
-
>
|
|
|
|
| 235 |
> call `/api/chat` with `"options": {"num_ctx": 4096}`, or bake a small-context tag for
|
| 236 |
> OpenAI clients: `ollama run hf.co/FoolDev/Janus-35B-HERETIC:Q4_K_M`, then
|
| 237 |
> `/set parameter num_ctx 4096` and `/save janus`, and point clients at `janus` — or in one step, **`./scripts/cap_ctx.sh`** (which bakes exactly that tag).
|
|
@@ -321,16 +338,16 @@ Behavior rules:
|
|
| 321 |
|
| 322 |
## Vision
|
| 323 |
|
| 324 |
-
The Qwen 3.
|
| 325 |
`mmproj` projector. The full multimodal stack is:
|
| 326 |
|
| 327 |
```
|
| 328 |
-
Janus-
|
| 329 |
-
Qwen3.
|
| 330 |
```
|
| 331 |
|
| 332 |
The projector and other-quant text decoders live at
|
| 333 |
-
[`llmfan46/Qwen3.
|
| 334 |
(BF16 mmproj only). This repo intentionally does not redistribute it;
|
| 335 |
`./scripts/fetch_vision.sh` pulls the projector into the repo root.
|
| 336 |
|
|
@@ -338,21 +355,21 @@ The projector and other-quant text decoders live at
|
|
| 338 |
|
| 339 |
| Loader | Text | Vision (mmproj) | Notes |
|
| 340 |
|---|---|---|---|
|
| 341 |
-
| **llama.cpp** (`llama-mtmd-cli`, `llama-server --mmproj`) | ✅ | ✅ | Reference path. Upstream has the `
|
| 342 |
| **llama-cpp-python** | ✅ | ✅ | See `examples/llama_cpp_vision.py`. |
|
| 343 |
-
| **Ollama 0.
|
| 344 |
| **LM Studio** | ✅ | ✅ | Uses upstream llama.cpp directly. |
|
| 345 |
|
| 346 |
### Vision via llama.cpp
|
| 347 |
|
| 348 |
```bash
|
| 349 |
# Fetch the projector first (into the repo root):
|
| 350 |
-
./scripts/fetch_vision.sh # Qwen3.
|
| 351 |
|
| 352 |
# A. HTTP via llama-server (the easiest path):
|
| 353 |
llama-server \
|
| 354 |
-
-m Janus-
|
| 355 |
-
--mmproj Qwen3.
|
| 356 |
--host 127.0.0.1 --port 8765 -c 8192 -ngl 99
|
| 357 |
# then POST OpenAI-style chat completions with an image_url content block —
|
| 358 |
# e.g. {"type":"image_url","image_url":{"url":"data:image/jpeg;base64,..."}}
|
|
@@ -363,36 +380,51 @@ llama-server \
|
|
| 363 |
# B. CLI via llama-mtmd-cli (one-shot). It's a separate cmake target, so a
|
| 364 |
# selective build can skip it; a plain `cmake --build build` produces it.
|
| 365 |
llama-mtmd-cli \
|
| 366 |
-
-m Janus-
|
| 367 |
-
--mmproj Qwen3.
|
| 368 |
--image photo.jpg \
|
| 369 |
-p "Describe this image."
|
| 370 |
|
| 371 |
# C. Python via llama-cpp-python:
|
| 372 |
python examples/llama_cpp_vision.py \
|
| 373 |
-
--gguf Janus-
|
| 374 |
-
--mmproj Qwen3.
|
| 375 |
--image /path/to/photo.jpg \
|
| 376 |
--prompt "What is in this image?"
|
| 377 |
```
|
| 378 |
|
| 379 |
The Ollama issue is closed upstream but this table has not been re-verified
|
| 380 |
since; until you have re-tested, treat Ollama as **text-only** for this model. The bundled Q4_K_M decoder pairs with the projector directly — the
|
| 381 |
-
mmproj is family-wide for Qwen 3.
|
| 382 |
needed for vision.
|
| 383 |
|
| 384 |
## Hardware requirements
|
| 385 |
|
| 386 |
-
This is a
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 387 |
|
| 388 |
| `num_ctx` | f16 cache (default) | q8_0 cache |
|
| 389 |
|---|---|---|
|
| 390 |
-
| **262144 (default)** |
|
| 391 |
-
| 65536 |
|
| 392 |
-
| 32768 |
|
| 393 |
-
| 1010000 (needs YaRN) |
|
| 394 |
|
| 395 |
-
KV
|
|
|
|
|
|
|
| 396 |
|
| 397 |
**How to override it:** `ollama run` has no `-o` flag, and `OLLAMA_CONTEXT_LENGTH`
|
| 398 |
only sets a default that the baked `num_ctx` overrides — so set it per-session
|
|
@@ -410,20 +442,18 @@ Programmatic callers pass the same via the API `options` field:
|
|
| 410 |
|
| 411 |
| Hardware | Status |
|
| 412 |
|---|---|
|
| 413 |
-
| ≥48 GB RAM (CPU-only) | Works at the 262144 default (~
|
| 414 |
| Single H100/A100 80 GB | Works, full offload |
|
| 415 |
| RTX 4090 24 GB / 5090 32 GB + 32 GB RAM | Works, partial offload |
|
| 416 |
| Mac Studio M2/M3 Ultra 64 GB+ unified | Works |
|
| 417 |
-
| 32 GB unified-memory laptops (Ryzen AI Max+, Apple M-series) |
|
| 418 |
|
| 419 |
-
**
|
| 420 |
-
|
| 421 |
-
|
| 422 |
-
|
| 423 |
-
|
| 424 |
-
|
| 425 |
-
rows above are fit estimates from the memory math. `./scripts/bench.sh` measures
|
| 426 |
-
tok/s on your own host.
|
| 427 |
|
| 428 |
**GPU offload on a Strix Halo iGPU (Radeon 8060S): a small gain, and a trap.** On a
|
| 429 |
Vulkan integrated GPU, llama.cpp - including the build Ollama 0.33.3 bundles - turns
|
|
@@ -431,20 +461,18 @@ mmap off and puts every layer it does *not* offload into GPU-reachable host memo
|
|
| 431 |
the same pool as the GPU, so `num_gpu` alone does not limit GPU memory: a first attempt
|
| 432 |
with llama-server filled the pool and took the desktop down with it. With
|
| 433 |
`LLAMA_ARG_NO_HOST=1` in the Ollama server's environment (llama.cpp's `--no-host`) those
|
| 434 |
-
layers stay in ordinary RAM.
|
| 435 |
-
|
| 436 |
-
|
| 437 |
-
and
|
| 438 |
-
|
| 439 |
-
|
| 440 |
-
like this, CPU-only is nearly as fast; if you try the GPU, set `LLAMA_ARG_NO_HOST=1` and a
|
| 441 |
-
fixed `num_gpu`. The mechanism is reported upstream in
|
| 442 |
[ggml-org/llama.cpp#27360](https://github.com/ggml-org/llama.cpp/issues/27360).
|
| 443 |
|
| 444 |
**Reaching a coherent ~1.01M context (opt-in YaRN).** The bundled GGUF ships no YaRN rope-scaling, so raising `num_ctx` toward the 1.01M ceiling degrades past the 262144 native window (see above). Ollama has no rope knob, so for a genuinely coherent long context run the GGUF under llama.cpp with YaRN enabled:
|
| 445 |
|
| 446 |
```bash
|
| 447 |
-
llama-server -m Janus-
|
| 448 |
--rope-scaling yarn --yarn-orig-ctx 262144 --rope-scale 3.853 -c 1010000
|
| 449 |
```
|
| 450 |
|
|
@@ -452,7 +480,7 @@ llama-server -m Janus-27B.Q4_K_M.gguf \
|
|
| 452 |
|
| 453 |
## Chat template
|
| 454 |
|
| 455 |
-
The model uses the standard Qwen 3.x ChatML format with `<|im_start|>` / `<|im_end|>` role markers. The
|
| 456 |
|
| 457 |
### Plain conversation
|
| 458 |
|
|
@@ -499,42 +527,40 @@ each endpoint reads one field:
|
|
| 499 |
| Ollama `/v1/chat/completions` | `reasoning` only — `reasoning_content` and `thinking` are dropped silently, and a message whose `content` is an array of parts keeps its reasoning only if it also has `tool_calls` |
|
| 500 |
| llama-server `/v1/chat/completions` | `reasoning_content` |
|
| 501 |
|
| 502 |
-
|
| 503 |
-
|
| 504 |
-
|
| 505 |
-
|
|
|
|
|
|
|
|
|
|
| 506 |
That Ollama's `/v1` also drops `thinking`, and the array-content rule, come from
|
| 507 |
Ollama's v0.33.3 and v0.34.0 source, and still hold on the current release:
|
| 508 |
`openai/openai.go` and `server/prompt.go` are byte-identical from v0.33.3 through
|
| 509 |
v0.34.2 (source, 2026-09-17). Ollama never passes reasoning to the GGUF's
|
| 510 |
embedded Jinja template, so on Ollama only the Go template decides what is replayed.
|
| 511 |
-
On llama.cpp
|
| 512 |
-
|
| 513 |
-
|
| 514 |
-
|
| 515 |
-
|
| 516 |
-
|
|
|
|
|
|
|
| 517 |
and on a CPU-only host prefill is the slow part. Releases 0.3.0 through 0.3.2
|
| 518 |
turned replay off for that reason; the CHANGELOG has the history.
|
| 519 |
|
| 520 |
-
**
|
| 521 |
-
|
| 522 |
-
`
|
| 523 |
-
|
| 524 |
-
|
| 525 |
-
|
| 526 |
-
|
| 527 |
-
|
| 528 |
-
|
| 529 |
-
`preserve_reasoning
|
| 530 |
-
|
| 531 |
-
and b10519 pass nothing, log "consider enabling it via --reasoning-preserve", and
|
| 532 |
-
replay anyway, because the template's own default is on. On the restamped blob
|
| 533 |
-
itself, with its embedded template and default flags, llama-server master rendered a
|
| 534 |
-
3-message conversation in 69 prompt tokens with the earlier turn's reasoning and 39
|
| 535 |
-
with `--no-reasoning-preserve` or either `false` kwarg (43 with no reasoning sent,
|
| 536 |
-
for the empty think block), and a real tool-result turn with string arguments
|
| 537 |
-
returned HTTP 200.
|
| 538 |
|
| 539 |
**Which llama.cpp an Ollama carries.** 0.33.3 and 0.34.0 bundle b10760; 0.34.1
|
| 540 |
bundles b10864 and 0.34.2 b10969 (each release's `LLAMA_CPP_VERSION`). From 0.34.1
|
|
@@ -544,19 +570,19 @@ for this model runs Ollama's own Go engine, and the bundled llama.cpp comes into
|
|
| 544 |
play only when Ollama falls back to its C++ engine, as it does once an mmproj is
|
| 545 |
attached (see [Loader compatibility](#loader-compatibility)).
|
| 546 |
|
| 547 |
-
**
|
| 548 |
-
|
| 549 |
-
|
| 550 |
-
|
| 551 |
-
|
| 552 |
-
`--no-reasoning-preserve`. This template lets `preserve_thinking` win and never raises.
|
| 553 |
|
| 554 |
**To turn replay off:**
|
| 555 |
|
| 556 |
- **llama-server** — start it with `--no-reasoning-preserve`, or send
|
| 557 |
-
`"chat_template_kwargs": {"preserve_thinking": false}`
|
| 558 |
-
`{"preserve_reasoning": false}`) on a request. A request's
|
| 559 |
`preserve_thinking: true` switches it back on for that request.
|
|
|
|
|
|
|
| 560 |
- **Ollama** — edit the template. In the assistant branch of both the `Modelfile`
|
| 561 |
and `template` (so `make check`'s bridge sync stays green), change `{{ if (and $.IsThinkSet .Thinking) -}}` back to Qwen's stock
|
| 562 |
`{{ if (and $.IsThinkSet (and .Thinking (or $last (gt $i $lastUserIdx)))) -}}`
|
|
@@ -575,59 +601,67 @@ ollama run hf.co/FoolDev/Janus-35B-HERETIC:Q4_K_M --think=false
|
|
| 575 |
|
| 576 |
or send `"think": false` on `/api/chat`. With thinking off the model skips the reasoning trace and answers straight into `content`; with it on (the default) reasoning is emitted into the `thinking` field.
|
| 577 |
|
| 578 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 579 |
|
| 580 |
-
|
| 581 |
-
eight values. Where `chat_template.jinja` sees the value as sent (llama.cpp), it maps
|
| 582 |
-
each to one of three tiers or switches thinking off:
|
| 583 |
|
| 584 |
-
|
| 585 |
-
|
| 586 |
-
|
| 587 |
-
|
| 588 |
-
|
| 589 |
-
|
| 590 |
-
|
| 591 |
-
|
| 592 |
-
|
| 593 |
-
|
| 594 |
-
|
| 595 |
-
|
| 596 |
-
|
| 597 |
-
|
| 598 |
-
|
| 599 |
-
|
| 600 |
-
|
| 601 |
-
|
| 602 |
-
`
|
| 603 |
-
|
| 604 |
-
|
| 605 |
-
|
| 606 |
-
|
| 607 |
-
|
| 608 |
-
|
| 609 |
-
|
| 610 |
-
|
| 611 |
-
|
| 612 |
-
|
| 613 |
-
|
|
|
|
|
|
|
|
|
|
| 614 |
|
| 615 |
### Which template Ollama runs
|
| 616 |
|
| 617 |
-
|
| 618 |
-
by HF's Ollama bridge) / `Modelfile` TEMPLATE, and
|
| 619 |
-
|
| 620 |
-
|
| 621 |
-
|
| 622 |
-
|
| 623 |
-
|
| 624 |
-
Ollama
|
| 625 |
-
|
| 626 |
-
|
| 627 |
-
llama.cpp and LM Studio run the embedded
|
| 628 |
-
|
| 629 |
-
|
| 630 |
-
|
|
|
|
|
|
|
|
|
|
| 631 |
|
| 632 |
```text
|
| 633 |
msg="template selection" ... selected=go_template go_template="[completion tools thinking]"
|
|
@@ -635,17 +669,19 @@ msg="template selection" ... selected=go_template go_template="[completion tools
|
|
| 635 |
|
| 636 |
The Go template's "thinking" entry comes from its single `.Thinking` reference —
|
| 637 |
the block that renders each assistant turn's reasoning. Removing it drops that
|
| 638 |
-
capability
|
| 639 |
-
|
| 640 |
-
|
| 641 |
-
|
| 642 |
-
|
| 643 |
-
|
| 644 |
-
|
| 645 |
|
| 646 |
### Tool / function calling
|
| 647 |
|
| 648 |
-
The model emits whichever format the
|
|
|
|
|
|
|
| 649 |
|
| 650 |
**Ollama path** (this repo's `Modelfile`). The TEMPLATE advertises tools inside `<tools>…</tools>` and asks the model to reply in JSON-in-XML — the form Ollama's tool-call extractor parses into a structured `tool_calls` array on `/api/chat` and `/v1/chat/completions`. The tool definitions go in with `{{ json .Function }}`: on Ollama 0.14 and later a bare `{{ .Function }}` prints Go struct syntax instead of JSON ([ollama/ollama#14601](https://github.com/ollama/ollama/issues/14601); fix proposed in [ollama/ollama#18391](https://github.com/ollama/ollama/pull/18391)). The model replies like this:
|
| 651 |
|
|
@@ -655,7 +691,7 @@ The model emits whichever format the system prompt specifies. The Ollama path pi
|
|
| 655 |
</tool_call>
|
| 656 |
```
|
| 657 |
|
| 658 |
-
**Embedded-
|
| 659 |
|
| 660 |
```text
|
| 661 |
<tool_call>
|
|
@@ -667,56 +703,11 @@ Tokyo
|
|
| 667 |
</tool_call>
|
| 668 |
```
|
| 669 |
|
| 670 |
-
**Keep XML on llama-server.** llama-server recognises this template as Qwen3-Coder-style and parses the model's reply with a parser — and, once the model opens `<tool_call>`, a grammar — that accept only `<function=…><parameter=…>`.
|
| 671 |
-
|
| 672 |
-
**Replayed tool calls render on llama-server.** Every OpenAI-compatible client hands assistant tool calls back with `function.arguments` as a JSON **string**, and the XML parameter form needs the individual key/value pairs — a chat template has `tojson` but no inverse. llama-server decodes the string into a mapping before it renders the template, so the second turn of a tool call renders in XML with no error. Tested on 2026-09-14 with a 260K-parameter test model and this template, replaying a call whose `arguments` was the string `{"city": "Paris"}`: `/apply-template` and `/v1/chat/completions` returned HTTP 200 with the call rendered as `<parameter=city>` on llama.cpp master `97e4ca7`, b10760 (the llama-server Ollama 0.33.3 bundles) and b10519 (LM Studio's avx2 CPU runtime 2.30.0).
|
| 673 |
-
|
| 674 |
-
**Other loaders, checked 2026-09-14.** KoboldCpp v1.120 renders the template with Python's jinja2 only when run with `--jinja --jinja_tools` (without `--jinja_tools`, requests with tools skip the template), and parses JSON-string `arguments` with `json.loads` first: with this template and a string-argument replay it returned HTTP 200 and rendered the call as XML (tested with a 260K-parameter model). Jan runs llama.cpp's own server code — v0.8.4 downloads `llama-server` from the janhq/llama.cpp fork (b9967), and its main branch links ggml-org llama.cpp b10809 — so it inherits the conversion above (from source, untested). The LM Studio app renders templates with its own JavaScript Jinja engine rather than llama-server's, and also parses the arguments first: through LM Studio 0.4.24's `/v1/chat/completions` (CPU runtime avx2 2.37.0, a 260K-parameter model carrying this template) a string-argument replay returned HTTP 200 with the call rendered as XML, and a request whose `arguments` were not valid JSON returned HTTP 500 before the template ran, with or without `tool_call_format: "auto"`. If a loader does raise the message below, change `'xml'` to `'auto'` in the template's `_requested_tool_format` default (in LM Studio, under the model's Prompt Template setting).
|
| 675 |
|
| 676 |
-
|
| 677 |
-
|
| 678 |
-
```text
|
| 679 |
-
tool_call_format="xml" requires tool-call arguments to be mappings, but received
|
| 680 |
-
a JSON string. Parse the JSON string into a mapping before applying the template,
|
| 681 |
-
or set tool_call_format="auto" (or "json") to use the lossless JSON tool-call
|
| 682 |
-
representation.
|
| 683 |
-
```
|
| 684 |
|
| 685 |
-
|
| 686 |
-
|
| 687 |
-
```bash
|
| 688 |
-
llama-server -m Janus-27B.Q4_K_M.gguf --jinja \
|
| 689 |
-
--chat-template-kwargs '{"tool_call_format":"auto"}'
|
| 690 |
-
```
|
| 691 |
-
|
| 692 |
-
On llama-server `auto` changes nothing, because arguments reach the template as mappings; do not set `json` there, for the parser reason above.
|
| 693 |
-
|
| 694 |
-
**`chat_template.jinja` in this repo is the base model's own template with two deliberate changes,** each marked `CHANGED FROM UPSTREAM` in the file: `preserve_thinking` / `preserve_reasoning` never raise — when they disagree `preserve_thinking` wins, where upstream raises and llama-server returns HTTP 500 — while earlier turns' reasoning is replayed by default, as upstream does (see [Thinking is replayed across turns](#thinking-is-replayed-across-turns)); and `reasoning_effort` is normalised instead of validated (see [Reasoning effort](#reasoning-effort)). The tool-call format, `max_tool_arg_chars`, `auto_disable_thinking_with_tools`, the vision handling and the whitespace contract are upstream's. The stock template still raises on `reasoning_effort: "max"`, which llama-server passes to the template, so run an unstamped quant with this file:
|
| 695 |
-
|
| 696 |
-
```bash
|
| 697 |
-
llama-server -m Janus-27B.Q4_K_M.gguf --jinja --chat-template-file chat_template.jinja
|
| 698 |
-
```
|
| 699 |
-
|
| 700 |
-
With it the same client code should work against either server, as long as both return a parsed `tool_calls` array — verified on Ollama and on standalone llama-server (llama.cpp master `a2878d3` and b10760, under the earlier `auto` default); the prompt format underneath differs (see [Which template Ollama runs](#which-template-ollama-runs)).
|
| 701 |
-
|
| 702 |
-
> **The bundled GGUF already carries this template — you do not need the flag.**
|
| 703 |
-
> `Janus-27B.Q4_K_M.gguf` has `chat_template.jinja` stamped into its
|
| 704 |
-
> metadata, so a loader that reads the *embedded* template and parses Qwen's
|
| 705 |
-
> `<tool_call>` output gets working tool calls with no configuration: llama-server
|
| 706 |
-
> (tested on llama.cpp master `a2878d3` and b10760) and LM Studio's CPU runtime
|
| 707 |
-
> (llama.cpp b10519, tested). llama-cpp-python renders the tools but, per its
|
| 708 |
-
> source, does not parse `<tool_call>` output into `tool_calls`. KoboldCpp (with
|
| 709 |
-
> `--jinja --jinja_tools`) was tested and Jan checked from source; see above. `./scripts/build.sh` (`make build`) stamps it into every quant it builds,
|
| 710 |
-
> too. The `--chat-template-file` flag above is only needed for a GGUF that was
|
| 711 |
-
> not built or stamped that way — a quant pulled straight from upstream, for
|
| 712 |
-
> instance. To restamp one you already have:
|
| 713 |
-
>
|
| 714 |
-
> ```bash
|
| 715 |
-
> python3 scripts/strip_mtp.py IN.gguf OUT.gguf --chat-template chat_template.jinja
|
| 716 |
-
> ```
|
| 717 |
-
>
|
| 718 |
-
> On an already-MTP-clean input that rewrites the file purely to swap the template;
|
| 719 |
-
> tensor data is copied byte-for-byte either way, so there is no re-quantization.
|
| 720 |
|
| 721 |
#### Example (Ollama, OpenAI-compatible API)
|
| 722 |
|
|
@@ -753,36 +744,36 @@ print(resp.choices[0].message.tool_calls)
|
|
| 753 |
|
| 754 |
- Use direct prompts ("Call X for Y") rather than soft hints ("Use the tool"). The model thinks before committing to a call, and weak prompts can exhaust `num_predict` inside the `<think>` block before the call is emitted.
|
| 755 |
- Allow at least `num_predict: 1024` (or `max_tokens: 1024`) for tool-calling turns, more if the schemas are large.
|
| 756 |
-
- The JSON-in-XML format is what Ollama's tool-call extractor understands, and the Go template pins it
|
| 757 |
|
| 758 |
## Known limitations
|
| 759 |
|
| 760 |
-
- **On the embedded-template path, a single tool call can come back twice.**
|
| 761 |
-
- **No mmproj in this release.** The base
|
| 762 |
- **Quantization-induced quality loss.** Q4_K_M is a strong general-purpose quant but does measurably degrade math and code accuracy compared to BF16. If you need maximum quality, run the upstream safetensors on a host with room for the full BF16 weights.
|
| 763 |
-
- **No MTP head on the GGUF path.** The base
|
| 764 |
-
- **
|
| 765 |
-
- **Thinking traces can loop.** Like most reasoning-distilled models, Janus-
|
| 766 |
- **Large tool-call arguments can be dropped.** Ollama's JSON-in-XML tool format makes the model JSON-escape the entire arguments object inline; for a big/complex payload (e.g. a file's `content` in a `write_file` call) the model can fail to escape it, so the field arrives `undefined` and the call fails. Qwen's native `<function=…><parameter=…>` format (raw values, no escaping) was tested as a fix but parses unreliably through Ollama, so the template deliberately keeps JSON-in-XML. Mitigation: write large files in smaller pieces per call.
|
| 767 |
- **Uncensored base — not aligned with any specific safety policy.** This is a personal repackage of an open-weight base whose refusal behavior has been abliterated away (the llmfan46 Heretic base). There is no RLHF refusal layer; the model will attempt most requests, so downstream safety is entirely the operator's responsibility.
|
| 768 |
-
- **No formal evaluation in this card.** The hardware table is
|
| 769 |
|
| 770 |
## Related models
|
| 771 |
|
| 772 |
| Model | Size | Notes |
|
| 773 |
|---|---|---|
|
| 774 |
-
| [llmfan46/Qwen3.
|
| 775 |
-
| [llmfan46/Qwen3.
|
| 776 |
-
| [Qwen/Qwen3.
|
| 777 |
-
| [FoolDev/Thanatos-27B-HERETIC](https://huggingface.co/FoolDev/Thanatos-27B-HERETIC) | 27B dense | Sibling
|
| 778 |
-
| [Crownelius/Crow-9B-HERETIC-4.6](https://huggingface.co/Crownelius/Crow-9B-HERETIC-4.6) | 9B dense | Heretic-flavored fine-tune on a smaller 9B Qwen base. Useful as a fast first-pass model when
|
| 779 |
|
| 780 |
-
> **Janus vs Thanatos.**
|
| 781 |
|
| 782 |
## Credits
|
| 783 |
|
| 784 |
-
- Immediate base: [llmfan46/Qwen3.
|
| 785 |
-
- Upstream base: [Qwen/Qwen3.
|
| 786 |
- Reasoning teacher: Claude Fable 5 (Anthropic)
|
| 787 |
- Distillation lineage and dataset curation: [Crownelius](https://huggingface.co/Crownelius)
|
| 788 |
|
|
|
|
| 1 |
---
|
| 2 |
license: apache-2.0
|
| 3 |
base_model:
|
| 4 |
+
- llmfan46/Qwen3.6-35B-A3B-uncensored-heretic
|
| 5 |
base_model_relation: quantized
|
| 6 |
datasets:
|
| 7 |
- Crownelius/Creative_Writing_ShareGPT_Enhanced
|
|
|
|
| 34 |
- fi
|
| 35 |
- el
|
| 36 |
tags:
|
| 37 |
+
- qwen36
|
| 38 |
+
- moe
|
| 39 |
- conversational
|
| 40 |
- multimodal
|
| 41 |
- agent
|
| 42 |
- gguf
|
| 43 |
+
- ollama
|
| 44 |
- heretic
|
| 45 |
- uncensored
|
| 46 |
- reasoning
|
| 47 |
- distillation
|
| 48 |
+
library_name: gguf
|
| 49 |
+
pipeline_tag: text-generation
|
| 50 |
---
|
| 51 |
|
| 52 |
+
<img src="https://huggingface.co/FoolDev/Janus-35B-HERETIC/resolve/main/banner.svg" alt="Janus-35B banner" width="100%" />
|
| 53 |
|
| 54 |
[](https://opensource.org/licenses/Apache-2.0)
|
| 55 |
+
[](https://huggingface.co/llmfan46/Qwen3.6-35B-A3B-uncensored-heretic)
|
| 56 |
+
[](#architecture)
|
| 57 |
[](#whats-here)
|
| 58 |
[](https://huggingface.co/FoolDev/Thanatos-27B-HERETIC)
|
| 59 |
|
| 60 |
+
<a href="https://buymeacoffee.com/cardoffoolm"><img src="https://huggingface.co/FoolDev/Janus-35B-HERETIC/resolve/main/bmc-banner.svg" alt="Buy me a coffee — support Janus-35B" width="320" /></a>
|
| 61 |
|
| 62 |
+
# Janus-35B
|
| 63 |
|
| 64 |
+
> **Flagship Reasoning. Sparse Footprint. Uncensored.**
|
| 65 |
+
> *llmfan46's Heretic abliteration of Qwen 3.6 35B-A3B, repackaged with Claude Fable 5 in the teacher slot.*
|
| 66 |
|
| 67 |
+
**`Architecture:`** `Qwen 3.6 35B-A3B (MoE)` | **`Total Params:`** `~34.7B` | **`Active Params:`** `~3B` | **`Context:`** `262,144 native / 1,010,000 extensible` | **`Base:`** `Heretic (llmfan46)` | **`Teacher:`** `Claude Fable 5` | **`Type:`** `Distilled + Abliterated MoE LLM`
|
| 68 |
|
| 69 |
+
A personal fork of [`llmfan46/Qwen3.6-35B-A3B-uncensored-heretic`](https://huggingface.co/llmfan46/Qwen3.6-35B-A3B-uncensored-heretic) — an uncensored Heretic-style abliteration of [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B), the 35B-total / 3B-active mixture-of-experts multimodal base — repackaged as Janus-35B with Claude Fable 5 reasoning data in the teacher slot. Refusal-trained behavior is dialed back at the base layer.
|
| 70 |
|
| 71 |
+
> **Provenance.** This repo repackages — **no training and no quantization is
|
| 72 |
+
> performed here.** What ships is `llmfan46`'s own Q4_K_M of
|
| 73 |
+
> [`llmfan46/Qwen3.6-35B-A3B-uncensored-heretic`](https://huggingface.co/llmfan46/Qwen3.6-35B-A3B-uncensored-heretic),
|
| 74 |
+
> byte-for-byte, with this repo's Ollama wrapper around it. The `Teacher:` field above and the `datasets:`
|
| 75 |
> list in the frontmatter are carried over from earlier revisions of this card.
|
| 76 |
> **The current base does not claim them** — its card lists no datasets and names
|
| 77 |
> no teacher — and nothing in this repo reproduces or measures them. One listed
|
|
|
|
| 80 |
> attribution, not as a training record.
|
| 81 |
> `base_model_relation` is `quantized` for the same reason.
|
| 82 |
|
| 83 |
+
> **On the repo name:** the model is **Janus-35B** and the Hub slug
|
| 84 |
+
> `FoolDev/Janus-35B-HERETIC` names it correctly again. Between releases 0.2.0 and
|
| 85 |
+
> 0.6.5 this repo shipped a dense Qwen 3.8 27B under the same slug; that blob is no
|
| 86 |
+
> longer published and remains in git history. Every URL and `ollama run hf.co/…`
|
| 87 |
+
> pull below is unchanged, as is the Ollama tag (`janus`).
|
| 88 |
|
| 89 |
## TL;DR
|
| 90 |
|
|
|
|
| 108 |
|
| 109 |
After either path, `ollama show janus` lists `completion`, `tools`,
|
| 110 |
and `thinking` under Capabilities. Hardware: the default `num_ctx` is the full
|
| 111 |
+
**262,144** native window — about **25 GiB** with Ollama's default f16 KV cache
|
| 112 |
+
and about **22.4 GiB** with `OLLAMA_KV_CACHE_TYPE=q8_0`. Those two totals are
|
| 113 |
+
*derived from the GGUF's own geometry, not measured on this blob*, and they do not
|
| 114 |
+
include the recurrent state or compute buffer (see
|
| 115 |
+
[Hardware requirements](#hardware-requirements)). Lower `num_ctx` on a smaller host
|
| 116 |
+
(65536 is about 21 GiB), or raise it to 1,010,000 with YaRN (not baked in, so
|
| 117 |
+
context past ~262K degrades).
|
| 118 |
|
| 119 |
To halve the KV cache, start the Ollama server with the q8_0 cache and flash
|
| 120 |
attention (for a systemd install, add them as `Environment=` lines with
|
|
|
|
| 128 |
|
| 129 |
| File | Use |
|
| 130 |
|---|---|
|
| 131 |
+
| `Janus-35B-A3B.Q4_K_M.gguf` | Recommended default, 21.23 GB. Upstream's Q4_K_M byte-for-byte — no re-quantization and no MTP strip needed (see [Architecture](#architecture)). |
|
| 132 |
| `Modelfile` | Ollama wrapper for **local** builds (`ollama create janus -f Modelfile`) — carries the Go template that Ollama runs in preference to the GGUF's embedded one (see [Which template Ollama runs](#which-template-ollama-runs)). |
|
| 133 |
| `template`, `system`, `params` | Used by HF's Ollama bridge when users `ollama run hf.co/FoolDev/Janus-35B-HERETIC` directly. The bridge does **not** read `Modelfile` (see [HF Ollama docs](https://huggingface.co/docs/hub/en/ollama)); it ingests these three root-level files instead. Kept in sync with the `Modelfile`'s `TEMPLATE` / `SYSTEM` / `PARAMETER` directives. |
|
| 134 |
+
| `scripts/build.sh` | Pulls a GGUF from `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF` (default Q4_K_M), runs it through `strip_mtp.py` (a no-op for these quants, kept as a guard), then runs `ollama create janus`. Quants published there: BF16, Q3_K_L, Q3_K_M, Q4_K_M, Q4_K_S, Q5_K_M, Q5_K_S, Q6_K, Q8_0. The bundled Q4_K_M *is* that repo's Q4_K_M; use this to build the others locally. |
|
|
|
|
| 135 |
| `scripts/check_bridge_sync.py` | Run before pushing a `Modelfile` / `template` / `system` / `params` edit to verify the four configurations remain in sync. Exits 0 if in sync, 1 with a per-key diff if not. |
|
| 136 |
+
| `scripts/check.sh` | Local lint: `bash -n`, `shellcheck`, `py_compile`, footgun-grep, `Modelfile`-vs-bridge-files sync and the Go `template` guard (`make check`) |
|
| 137 |
+
| `scripts/check_go_template.py` | Guards the Go `template`: Ollama's thinking detection, the condition that replays earlier turns' reasoning, the tool round trip and JSON tool signatures (check 8 in `check.sh`) |
|
|
|
|
| 138 |
| `scripts/live_check.sh` | Live end-to-end checks in an isolated, CPU-only Ollama (own port and model store; it refuses to run if Ollama reports a GPU): template selection, one tool call, string-argument replay, an earlier turn's reasoning replayed and a live tool chain's kept, every `reasoning_effort` value and the effort lines — 17 checks (`make live-check`) |
|
| 139 |
+
| `scripts/verify_arch.py` | Cross-checks the README Architecture bullets (layer count, hidden size, expert counts and widths, native context, vocab) plus the underlying forward-pass structure (Gated Attention + Gated DeltaNet dims, partial RoPE, full-attention interval) against the bundled GGUF's `qwen35moe` metadata. Run on demand (`python3 scripts/verify_arch.py`); reads the 21 GB GGUF (LFS smudge required) and exits non-zero on any mismatch. `block_count` is checked against **40**, this model's natural depth — the MTP-Preserved variant this repo does not ship reports 41. |
|
| 140 |
+
| `scripts/strip_mtp.py` | Drops an MTP / NextN layer from a Qwen 3.x GGUF (`python3 scripts/strip_mtp.py IN.gguf OUT.gguf`): removes the extra `blk.<last>.*` block, decrements `block_count`, drops `nextn_predict_layers`. Kept tensors are copied byte-for-byte (no re-quant); a conditional no-op (hardlink) on already-clean quants — **which is what it is for the quants this repo ships**, since llmfan46's plain `…-heretic-GGUF` files carry no NextN block. It stays in the build path as a guard for anyone pointing `build.sh` at the `…-Native-MTP-Preserved-GGUF` variant instead. Parity with the dense sibling `FoolDev/Thanatos-27B-HERETIC`, where the strip *is* required. |
|
| 141 |
| `scripts/smoke_test.sh` | Integration smoke test against a running Ollama daemon: server reachable, model loaded, `tools` capability present, chat round-trip, and no control-token leakage. `TOOLS_TEST=1` adds a tool-call round-trip. Defaults to `MODEL=janus`. |
|
| 142 |
| `scripts/bench.sh` | Measures tok/s from Ollama's `eval_count` / `eval_duration` over a short/medium/long prompt mix (with a discarded warmup). Defaults to `MODEL=janus`. |
|
| 143 |
+
| `scripts/load_bundle.sh` | Loads the bundled `Janus-35B-A3B.Q4_K_M.gguf` into Ollama as a local `janus` tag without an upstream pull (if the working-tree file is still an LFS pointer it downloads the real blob into `.cache/` — it does not smudge in place — then checks the arch is `qwen35moe`). |
|
| 144 |
| `scripts/cap_ctx.sh` | Bakes a small-`num_ctx` local `janus` tag (bundled blob + this repo's Modelfile, `num_ctx`→4096, `num_batch` 256) for OpenAI `/v1` clients — which can't override the baked `num_ctx` and OOM on small hosts (see [Inference examples](#inference-examples)). Run `./scripts/cap_ctx.sh` (or `CTX=8192 ...`). |
|
| 145 |
+
| `scripts/fetch_vision.sh` | Downloads the vision projector (`Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf`, 902 MB) from the Heretic GGUF repo for llama.cpp image input (Ollama vision is broken upstream — see [Vision](#vision)). |
|
| 146 |
| `examples/` | Ready-to-run Python clients for Ollama, Transformers, and llama-cpp-python (text, tools, and vision — see `examples/README.md`) |
|
| 147 |
|
| 148 |
+
GGUF-only release. Pull the Heretic safetensors from [`llmfan46/Qwen3.6-35B-A3B-uncensored-heretic`](https://huggingface.co/llmfan46/Qwen3.6-35B-A3B-uncensored-heretic) if you need the `transformers` tree (or the vanilla pre-Heretic base from [`Qwen/Qwen3.6-35B-A3B`](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)). The safetensors also keep the MTP layer that this GGUF does not carry — that's the path to speculative decoding under vLLM / SGLang.
|
| 149 |
|
| 150 |
+
> ### Bundled blob status: the 35B-A3B MoE is published
|
| 151 |
>
|
| 152 |
+
> The bundled GGUF is **`Janus-35B-A3B.Q4_K_M.gguf`** (21,233,608,512 bytes —
|
| 153 |
+
> 21.23 GB decimal, 19.78 GiB binary) — the Qwen 3.6 35B-A3B MoE this card
|
| 154 |
+
> describes. `ollama run hf.co/FoolDev/Janus-35B-HERETIC` serves it directly, and
|
| 155 |
+
> `ollama create janus -f Modelfile` works from a fresh clone.
|
| 156 |
>
|
| 157 |
+
> It is `llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF`'s Q4_K_M
|
| 158 |
+
> (sha256 `b82f9f2155eb9c07139d48cc3a37880cf9d0edcca345ef6ad829b62941ccbb82`)
|
| 159 |
+
> **byte-for-byte**: no re-quantization, no MTP strip and no template restamp. This
|
| 160 |
+
> quant carries no NextN block to strip, and the GGUF keeps the base model's own
|
| 161 |
+
> embedded chat template — this repo ships no `chat_template.jinja` of its own.
|
|
|
|
|
|
|
| 162 |
>
|
| 163 |
+
> Verified before publishing (2026-09-18): the file's GGUF header was read directly
|
| 164 |
+
> from the published blob by HTTP range request — `general.architecture qwen35moe`,
|
| 165 |
+
> `block_count` 40, 256 experts / 8 active, `context_length` 262144, vocab 248,320,
|
| 166 |
+
> 733 tensors, no `nextn` key and no NextN tensors — and every Architecture claim
|
| 167 |
+
> below matches it. Not re-run since the reversal: a live load, a tool-call
|
| 168 |
+
> round-trip and `scripts/live_check.sh`, all of which need the 21 GB blob on disk.
|
| 169 |
>
|
| 170 |
+
> Between 0.2.0 and 0.6.5 this repo shipped a dense Qwen 3.8 27B
|
| 171 |
+
> (`Janus-27B.Q4_K_M.gguf`). That blob is no longer published; it remains in git
|
| 172 |
+
> history.
|
| 173 |
|
| 174 |
## Architecture
|
| 175 |
|
| 176 |
+
<img src="https://huggingface.co/FoolDev/Janus-35B-HERETIC/resolve/main/moe-routing.svg" alt="animated MoE routing visualization: 16x16 grid of 256 expert dots with 8 lit at any time, cycling through 8 routing patterns" width="640" />
|
|
|
|
|
|
|
| 177 |
|
| 178 |
+
A **sparse** hybrid stack: linear-attention (Gated DeltaNet) layers with a full
|
| 179 |
+
Gated Attention layer every fourth block, and a 256-expert MoE feed-forward on
|
| 180 |
+
every block. ~34.7B parameters total, ~3B active per token — 8 routed experts
|
| 181 |
+
plus one shared expert.
|
| 182 |
+
|
| 183 |
+
- Qwen 3.6, 35B-A3B mixture-of-experts (256 experts, 8 routed per token, plus a shared expert on every block)
|
| 184 |
+
- 40 layers, 10 × (3 × Gated DeltaNet → MoE / 1 × Gated Attention → MoE) — `full_attention_interval` 4, so 10 full-attention and 30 linear-attention layers
|
| 185 |
+
- Hidden size 2048; per-expert FFN intermediate 512, shared-expert FFN 512
|
| 186 |
+
- Gated Attention: 16 Q-heads / 2 KV-heads (GQA), head_dim 256, gated output projection, partial RoPE (`rope.dimension_count` 64 of 256 dims rotated), `rope.freq_base` 1e7
|
| 187 |
+
- Gated DeltaNet: conv kernel 4, state size 128, 16 groups, inner size 4096
|
| 188 |
- 262 144 native context (extensible to 1 010 000 with YaRN, but YaRN is not enabled in the bundled GGUF)
|
| 189 |
+
- Vision + video supported by upstream (mmproj not included in this release — see [Vision](#vision))
|
| 190 |
- Vocab 248,320
|
| 191 |
+
- GGUF arch stamp `qwen35moe` — the MoE arch in llama.cpp (`src/models/qwen35moe.cpp`, `case 40: LLM_TYPE_35B_A3B`); the dense members of the family stamp `qwen35`
|
| 192 |
+
|
| 193 |
+
Every bullet above is asserted against the bundled GGUF's own metadata by
|
| 194 |
+
`python3 scripts/verify_arch.py`.
|
| 195 |
|
| 196 |
+
**MTP head (absent from this quant).** The base *does* declare an MTP layer
|
| 197 |
+
(`mtp_num_hidden_layers: 1` in its `config.json`) for vLLM / SGLang speculative
|
| 198 |
+
decoding, but the Q4_K_M published in llmfan46's GGUF repo — the file this repo
|
| 199 |
+
bundles — carries no NextN block: `block_count` is 40 and `blk.0`…`blk.39` are all
|
| 200 |
+
that exist. So no strip is needed here, and `scripts/strip_mtp.py` is a defensive
|
| 201 |
+
no-op on it. llmfan46 also publishes a separate `…-Native-MTP-Preserved-GGUF`
|
| 202 |
+
variant whose Q4_K_M has `block_count` 41 with a NextN block at index 40; that one
|
| 203 |
+
*would* need stripping, and it is not what this repo ships. For the MTP head
|
| 204 |
+
itself, run the upstream safetensors under vLLM / SGLang.
|
| 205 |
|
| 206 |
## Quick start
|
| 207 |
|
|
|
|
| 209 |
|
| 210 |
Drop the GGUF into your loader of choice. The chat template is embedded in the GGUF metadata, and llama.cpp and LM Studio use it by default, so no `--chat-template` flag is needed (llama.cpp has no `--chat-template auto` mode; it would take "auto" as the template itself).
|
| 211 |
|
| 212 |
+
**Tool calling works out of the box** — the GGUF carries Qwen 3.6's own template, so `--jinja` alone is enough and there is no file to pass:
|
| 213 |
|
| 214 |
```bash
|
| 215 |
+
llama-server -m Janus-35B-A3B.Q4_K_M.gguf --jinja
|
| 216 |
```
|
| 217 |
|
| 218 |
+
Tool calls use Qwen's native XML form, the one llama-server's parser reads. Two
|
| 219 |
+
things this path does **not** have, because they lived in the patched Qwen 3.8
|
| 220 |
+
template this repo shipped between 0.2.0 and 0.6.5 and no longer ships:
|
| 221 |
+
`reasoning_effort` tiers, and the `tool_call_format` JSON opt-in. See
|
| 222 |
+
[Tool / function calling](#tool--function-calling) and
|
| 223 |
+
[Reasoning effort](#reasoning-effort).
|
| 224 |
|
| 225 |
### Ollama
|
| 226 |
|
|
|
|
| 245 |
> The examples use `model: "janus"`, the tag from the local build (path B). If you pulled via the TL;DR one-liner instead, use the full tag `hf.co/FoolDev/Janus-35B-HERETIC`, or run `ollama cp hf.co/FoolDev/Janus-35B-HERETIC janus` once to create the short tag.
|
| 246 |
|
| 247 |
> **On memory-tight hosts, cap `num_ctx` first.** `/v1/chat/completions` (OpenAI-compat) has
|
| 248 |
+
> no `num_ctx` knob, so it loads at the baked **262,144** default (5.0 GiB KV / ~25 GiB total
|
| 249 |
+
> with Ollama's default f16 cache — derived from the GGUF's geometry, not measured on this blob),
|
| 250 |
+
> which is tight on a 32 GB box; `OLLAMA_KV_CACHE_TYPE=q8_0` on the
|
| 251 |
+
> server brings it to ~22.4 GiB (see [Hardware requirements](#hardware-requirements)). Either
|
| 252 |
> call `/api/chat` with `"options": {"num_ctx": 4096}`, or bake a small-context tag for
|
| 253 |
> OpenAI clients: `ollama run hf.co/FoolDev/Janus-35B-HERETIC:Q4_K_M`, then
|
| 254 |
> `/set parameter num_ctx 4096` and `/save janus`, and point clients at `janus` — or in one step, **`./scripts/cap_ctx.sh`** (which bakes exactly that tag).
|
|
|
|
| 338 |
|
| 339 |
## Vision
|
| 340 |
|
| 341 |
+
The Qwen 3.6 base supports image (and video) input via a separate
|
| 342 |
`mmproj` projector. The full multimodal stack is:
|
| 343 |
|
| 344 |
```
|
| 345 |
+
Janus-35B-A3B.Q4_K_M.gguf (21.23 GB, the text decoder)
|
| 346 |
+
Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf (902 MB, the vision projector)
|
| 347 |
```
|
| 348 |
|
| 349 |
The projector and other-quant text decoders live at
|
| 350 |
+
[`llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF`](https://huggingface.co/llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF)
|
| 351 |
(BF16 mmproj only). This repo intentionally does not redistribute it;
|
| 352 |
`./scripts/fetch_vision.sh` pulls the projector into the repo root.
|
| 353 |
|
|
|
|
| 355 |
|
| 356 |
| Loader | Text | Vision (mmproj) | Notes |
|
| 357 |
|---|---|---|---|
|
| 358 |
+
| **llama.cpp** (`llama-mtmd-cli`, `llama-server --mmproj`) | ✅ | ✅ | Reference path. Upstream has the `qwen35moe` arch entry (`src/models/qwen35moe.cpp`). |
|
| 359 |
| **llama-cpp-python** | ✅ | ✅ | See `examples/llama_cpp_vision.py`. |
|
| 360 |
+
| **Ollama** (0.33.3 and 0.34.x checked in source, 2026-09-18) | ✅ | ❌ | Text inference works: `qwen35moe` is handled through Ollama's llama.cpp compat shim at v0.33.3, v0.34.0 and v0.34.2. The version floor this row used to claim ("0.24+") was undated and inherited from the MoE-era card; it has not been re-established, so treat these three as the tested set. Vision (mmproj) is still broken: the C++ llama.cpp fallback that Ollama switches to when an mmproj is attached lacks it. `ollama create` accepts a dual-`FROM` (text + mmproj) and `ollama show` reports `vision` capability — but the **first inference request** fails with an `unknown model architecture` load error, and once mmproj is attached this blocks text inference too. See [ollama/ollama#14575](https://github.com/ollama/ollama/issues/14575) (closed as completed on 2026-08-04; #15898 was closed as its duplicate and the sync PR #15899 closed unmerged). Re-test on a current Ollama before assuming vision is still broken — this table has not been re-verified since it closed. |
|
| 361 |
| **LM Studio** | ✅ | ✅ | Uses upstream llama.cpp directly. |
|
| 362 |
|
| 363 |
### Vision via llama.cpp
|
| 364 |
|
| 365 |
```bash
|
| 366 |
# Fetch the projector first (into the repo root):
|
| 367 |
+
./scripts/fetch_vision.sh # Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf
|
| 368 |
|
| 369 |
# A. HTTP via llama-server (the easiest path):
|
| 370 |
llama-server \
|
| 371 |
+
-m Janus-35B-A3B.Q4_K_M.gguf \
|
| 372 |
+
--mmproj Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf \
|
| 373 |
--host 127.0.0.1 --port 8765 -c 8192 -ngl 99
|
| 374 |
# then POST OpenAI-style chat completions with an image_url content block —
|
| 375 |
# e.g. {"type":"image_url","image_url":{"url":"data:image/jpeg;base64,..."}}
|
|
|
|
| 380 |
# B. CLI via llama-mtmd-cli (one-shot). It's a separate cmake target, so a
|
| 381 |
# selective build can skip it; a plain `cmake --build build` produces it.
|
| 382 |
llama-mtmd-cli \
|
| 383 |
+
-m Janus-35B-A3B.Q4_K_M.gguf \
|
| 384 |
+
--mmproj Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf \
|
| 385 |
--image photo.jpg \
|
| 386 |
-p "Describe this image."
|
| 387 |
|
| 388 |
# C. Python via llama-cpp-python:
|
| 389 |
python examples/llama_cpp_vision.py \
|
| 390 |
+
--gguf Janus-35B-A3B.Q4_K_M.gguf \
|
| 391 |
+
--mmproj Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf \
|
| 392 |
--image /path/to/photo.jpg \
|
| 393 |
--prompt "What is in this image?"
|
| 394 |
```
|
| 395 |
|
| 396 |
The Ollama issue is closed upstream but this table has not been re-verified
|
| 397 |
since; until you have re-tested, treat Ollama as **text-only** for this model. The bundled Q4_K_M decoder pairs with the projector directly — the
|
| 398 |
+
mmproj is family-wide for Qwen 3.6 35B-A3B, so no separate text download is
|
| 399 |
needed for vision.
|
| 400 |
|
| 401 |
## Hardware requirements
|
| 402 |
|
| 403 |
+
This is a 21.23 GB Q4_K_M GGUF (19.78 GiB of weights). **The figures below are
|
| 404 |
+
derived from the GGUF's own geometry, not measured on this blob** — every measured
|
| 405 |
+
number this section used to carry was taken on the dense 27B that shipped between
|
| 406 |
+
0.2.0 and 0.6.5 and has been removed rather than rescaled, because the geometry
|
| 407 |
+
differs in both directions: more weights, far less KV.
|
| 408 |
+
|
| 409 |
+
The KV cache covers the 10 full-attention layers (2 KV heads × 256 key dims + 256
|
| 410 |
+
value dims), so each token costs 10 × 2 × 512 × 2 bytes = **20,480 B at f16** —
|
| 411 |
+
**0.625 GiB per 32K tokens** — or about **10,880 B at q8_0** (0.332 GiB per 32K,
|
| 412 |
+
`OLLAMA_KV_CACHE_TYPE=q8_0` with `OLLAMA_FLASH_ATTENTION=1`). That is under a third
|
| 413 |
+
of the dense 27B's KV at the same window. The 30 linear-attention layers hold a fixed
|
| 414 |
+
recurrent state and the runtime allocates a compute buffer on top; **neither is
|
| 415 |
+
derivable from metadata and neither has been measured here**, so the totals below are
|
| 416 |
+
floors, not forecasts.
|
| 417 |
|
| 418 |
| `num_ctx` | f16 cache (default) | q8_0 cache |
|
| 419 |
|---|---|---|
|
| 420 |
+
| **262144 (default)** | 5.0 GiB KV, **~25 GiB** total | 2.66 GiB KV, **~22.4 GiB** total |
|
| 421 |
+
| 65536 | 1.25 GiB KV, ~21 GiB total | 0.66 GiB KV, ~20.4 GiB total |
|
| 422 |
+
| 32768 | 0.63 GiB KV, ~20.4 GiB total | 0.33 GiB KV, ~20.1 GiB total |
|
| 423 |
+
| 1010000 (needs YaRN) | 19.3 GiB KV, ~39 GiB total | 10.2 GiB KV, ~30 GiB total |
|
| 424 |
|
| 425 |
+
KV sizes are exact for this geometry and linear in `num_ctx`; the totals add the
|
| 426 |
+
19.78 GiB of weights and nothing else. Re-measure with `make live-check` or
|
| 427 |
+
llama.cpp's `llama-fit-params -v` once the blob is on disk. It's **extensible to 1,010,000**, but this GGUF ships **no baked YaRN rope-scaling**, so positions past the 262144 native window use untrained RoPE and output degrades — reaching the 1.01M ceiling needs YaRN (see below); keep real work within ~262K otherwise.
|
| 428 |
|
| 429 |
**How to override it:** `ollama run` has no `-o` flag, and `OLLAMA_CONTEXT_LENGTH`
|
| 430 |
only sets a default that the baked `num_ctx` overrides — so set it per-session
|
|
|
|
| 442 |
|
| 443 |
| Hardware | Status |
|
| 444 |
|---|---|
|
| 445 |
+
| ≥48 GB RAM (CPU-only) | Works at the 262144 default (~25 GiB derived) with room to spare |
|
| 446 |
| Single H100/A100 80 GB | Works, full offload |
|
| 447 |
| RTX 4090 24 GB / 5090 32 GB + 32 GB RAM | Works, partial offload |
|
| 448 |
| Mac Studio M2/M3 Ultra 64 GB+ unified | Works |
|
| 449 |
+
| 32 GB unified-memory laptops (Ryzen AI Max+, Apple M-series) | ~25 GiB derived at the 262144 default leaves little headroom on a 27 GiB-visible host; `OLLAMA_KV_CACHE_TYPE=q8_0` (~22.4 GiB) or `num_ctx 65536` (~21 GiB) is the comfortable setting |
|
| 450 |
|
| 451 |
+
**No throughput figures are published for this build.** Every tok/s number this card
|
| 452 |
+
carried was measured on the dense 27B, and a 35B-A3B MoE touches roughly 3B
|
| 453 |
+
parameters per token instead of 27B — a bandwidth profile different enough that
|
| 454 |
+
rescaling would be invention, not estimation. `./scripts/bench.sh` measures tok/s on
|
| 455 |
+
your own host; contributions of real numbers are welcome (see
|
| 456 |
+
[No formal evaluation](#known-limitations)).
|
|
|
|
|
|
|
| 457 |
|
| 458 |
**GPU offload on a Strix Halo iGPU (Radeon 8060S): a small gain, and a trap.** On a
|
| 459 |
Vulkan integrated GPU, llama.cpp - including the build Ollama 0.33.3 bundles - turns
|
|
|
|
| 461 |
the same pool as the GPU, so `num_gpu` alone does not limit GPU memory: a first attempt
|
| 462 |
with llama-server filled the pool and took the desktop down with it. With
|
| 463 |
`LLAMA_ARG_NO_HOST=1` in the Ollama server's environment (llama.cpp's `--no-host`) those
|
| 464 |
+
layers stay in ordinary RAM. That mechanism is loader-side and applies to any model; the
|
| 465 |
+
tok/s series this paragraph used to quote was measured on the dense 27B and has been
|
| 466 |
+
removed with the rest. What generalises: the GPU's GTT is system RAM, so offloading frees
|
| 467 |
+
none of it, and a first attempt without `--no-host` filled the pool and took the desktop
|
| 468 |
+
session down with it. If you try the GPU on a machine like this, set
|
| 469 |
+
`LLAMA_ARG_NO_HOST=1` and a fixed `num_gpu`. The mechanism is reported upstream in
|
|
|
|
|
|
|
| 470 |
[ggml-org/llama.cpp#27360](https://github.com/ggml-org/llama.cpp/issues/27360).
|
| 471 |
|
| 472 |
**Reaching a coherent ~1.01M context (opt-in YaRN).** The bundled GGUF ships no YaRN rope-scaling, so raising `num_ctx` toward the 1.01M ceiling degrades past the 262144 native window (see above). Ollama has no rope knob, so for a genuinely coherent long context run the GGUF under llama.cpp with YaRN enabled:
|
| 473 |
|
| 474 |
```bash
|
| 475 |
+
llama-server -m Janus-35B-A3B.Q4_K_M.gguf \
|
| 476 |
--rope-scaling yarn --yarn-orig-ctx 262144 --rope-scale 3.853 -c 1010000
|
| 477 |
```
|
| 478 |
|
|
|
|
| 480 |
|
| 481 |
## Chat template
|
| 482 |
|
| 483 |
+
The model uses the standard Qwen 3.x ChatML format with `<|im_start|>` / `<|im_end|>` role markers. Two templates are in play. The GGUF carries **Qwen 3.6's own embedded template**, exactly as llmfan46 published it — this repo ships no `chat_template.jinja` of its own — and that is what llama.cpp, LM Studio and KoboldCpp render. On Ollama the Go `template` / `Modelfile` TEMPLATE is what runs instead: Ollama picks between the two at load time (see [Which template Ollama runs](#which-template-ollama-runs)). Where the two differ, this card says which one it is describing.
|
| 484 |
|
| 485 |
### Plain conversation
|
| 486 |
|
|
|
|
| 527 |
| Ollama `/v1/chat/completions` | `reasoning` only — `reasoning_content` and `thinking` are dropped silently, and a message whose `content` is an array of parts keeps its reasoning only if it also has `tool_calls` |
|
| 528 |
| llama-server `/v1/chat/completions` | `reasoning_content` |
|
| 529 |
|
| 530 |
+
Which field each endpoint reads is decided by the server, not the model. Checked
|
| 531 |
+
live on 2026-09-13 — on the dense 27B blob this repo shipped between 0.2.0 and
|
| 532 |
+
0.6.5, CPU only — Ollama 0.33.3 replayed `thinking` on `/api/chat` (232 prompt
|
| 533 |
+
tokens with it, 187 without) and `reasoning` on `/v1` (215 vs 187) and ignored
|
| 534 |
+
`reasoning_content` on `/v1` (187); llama-server master rendered `reasoning_content`
|
| 535 |
+
and dropped a `reasoning` field. The behaviour is Ollama-side and carries over; the
|
| 536 |
+
token counts were measured on that other blob and are not this model's.
|
| 537 |
That Ollama's `/v1` also drops `thinking`, and the array-content rule, come from
|
| 538 |
Ollama's v0.33.3 and v0.34.0 source, and still hold on the current release:
|
| 539 |
`openai/openai.go` and `server/prompt.go` are byte-identical from v0.33.3 through
|
| 540 |
v0.34.2 (source, 2026-09-17). Ollama never passes reasoning to the GGUF's
|
| 541 |
embedded Jinja template, so on Ollama only the Go template decides what is replayed.
|
| 542 |
+
On llama.cpp an earlier assistant turn that carries no reasoning renders with **no**
|
| 543 |
+
`<think>` block at all: Qwen 3.6's template gates the block on
|
| 544 |
+
`reasoning_content|length > 0`. The 3.8 template this repo used to ship reconstructed
|
| 545 |
+
an empty one instead, so prompts built by the two differ on that turn.
|
| 546 |
+
|
| 547 |
+
**The cost is context.** Every retained trace stays in the prompt — measured on the
|
| 548 |
+
dense 27B at 181 prompt tokens without and 464 with, on a 3-message conversation whose
|
| 549 |
+
first assistant turn carried a ~240-word trace, and not re-measured here. Prefill grows with every retained trace,
|
| 550 |
and on a CPU-only host prefill is the slow part. Releases 0.3.0 through 0.3.2
|
| 551 |
turned replay off for that reason; the CHANGELOG has the history.
|
| 552 |
|
| 553 |
+
**On llama.cpp, Qwen 3.6's own template decides.** Read from the template inside
|
| 554 |
+
the published blob (2026-09-18, from source — not re-tested live on this model):
|
| 555 |
+
`preserve_thinking` defaults to **true**, so an earlier turn's reasoning is replayed
|
| 556 |
+
by default; the assistant branch takes the trace from `message.reasoning_content`,
|
| 557 |
+
and failing that recovers one by splitting a `<think>` block out of `content`; and
|
| 558 |
+
there is no `preserve_reasoning` variable in it at all. That last point costs
|
| 559 |
+
nothing, because llama-server derives a `preserve_thinking` kwarg from
|
| 560 |
+
`preserve_reasoning` (`common/jinja/caps.cpp`), and from b10763 it passes
|
| 561 |
+
`preserve_reasoning=true` by default — so `--no-reasoning-preserve` and a request's
|
| 562 |
+
`preserve_reasoning: false` still reach this template through the derived copy,
|
| 563 |
+
while a request's own `preserve_thinking` is read directly.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 564 |
|
| 565 |
**Which llama.cpp an Ollama carries.** 0.33.3 and 0.34.0 bundle b10760; 0.34.1
|
| 566 |
bundles b10864 and 0.34.2 b10969 (each release's `LLAMA_CPP_VERSION`). From 0.34.1
|
|
|
|
| 570 |
play only when Ollama falls back to its C++ engine, as it does once an mmproj is
|
| 571 |
attached (see [Loader compatibility](#loader-compatibility)).
|
| 572 |
|
| 573 |
+
**Nothing raises on conflicting settings.** The Qwen 3.8 template that this repo
|
| 574 |
+
patched between 0.4.0 and 0.6.5 raised when `preserve_thinking` and
|
| 575 |
+
`preserve_reasoning` disagreed, and llama-server returned that raise as HTTP 500 —
|
| 576 |
+
which is what the patch existed to prevent. Qwen 3.6's template has no such check:
|
| 577 |
+
it reads `preserve_thinking`, ignores anything else, and cannot fail that way.
|
|
|
|
| 578 |
|
| 579 |
**To turn replay off:**
|
| 580 |
|
| 581 |
- **llama-server** — start it with `--no-reasoning-preserve`, or send
|
| 582 |
+
`"chat_template_kwargs": {"preserve_thinking": false}` on a request. A request's
|
|
|
|
| 583 |
`preserve_thinking: true` switches it back on for that request.
|
| 584 |
+
`{"preserve_reasoning": false}` works too, through the kwarg llama-server derives
|
| 585 |
+
from it; the template itself never reads that name.
|
| 586 |
- **Ollama** — edit the template. In the assistant branch of both the `Modelfile`
|
| 587 |
and `template` (so `make check`'s bridge sync stays green), change `{{ if (and $.IsThinkSet .Thinking) -}}` back to Qwen's stock
|
| 588 |
`{{ if (and $.IsThinkSet (and .Thinking (or $last (gt $i $lastUserIdx)))) -}}`
|
|
|
|
| 601 |
|
| 602 |
or send `"think": false` on `/api/chat`. With thinking off the model skips the reasoning trace and answers straight into `content`; with it on (the default) reasoning is emitted into the `thinking` field.
|
| 603 |
|
| 604 |
+
On llama.cpp the switch is the embedded template's own: send
|
| 605 |
+
`"chat_template_kwargs": {"enable_thinking": false}`, which primes a closed, empty
|
| 606 |
+
`<think>` block — and, because the template gates its replay branch on
|
| 607 |
+
`enable_thinking` too, drops every earlier turn's trace for that request. That
|
| 608 |
+
variable is upstream Qwen 3.6's, not this repo's.
|
| 609 |
|
| 610 |
+
### Reasoning effort
|
|
|
|
|
|
|
| 611 |
|
| 612 |
+
**This works on Ollama, and not on llama.cpp.** The instruction comes from this
|
| 613 |
+
repo's Go `template`, which Ollama runs (see
|
| 614 |
+
[Which template Ollama runs](#which-template-ollama-runs)). Qwen 3.6's own template —
|
| 615 |
+
the one inside the bundled GGUF, and so the one llama.cpp and LM Studio render — has
|
| 616 |
+
no `reasoning_effort` variable at all, so there the tier values are accepted by the
|
| 617 |
+
server and then quietly ignored (read from the template, 2026-09-18). `none` is the
|
| 618 |
+
one exception: llama-server maps it to `enable_thinking: false` and drops the kwarg
|
| 619 |
+
before the template runs (`tools/server/server-common.cpp`, checked 2026-09-18), so
|
| 620 |
+
`reasoning_effort: "none"` does switch thinking off on llama.cpp. Between 0.2.0 and
|
| 621 |
+
0.6.5 this repo shipped a patched Qwen 3.8 template that did implement the tiers on
|
| 622 |
+
the llama.cpp path; that file is gone with the rebase back to the MoE.
|
| 623 |
+
|
| 624 |
+
Ollama folds the eight values its `/v1` layer accepts into four think levels before
|
| 625 |
+
any template sees them, so the Go template maps levels, not values:
|
| 626 |
+
|
| 627 |
+
| `reasoning_effort` (Ollama `/v1`) | Think level | Effect |
|
| 628 |
+
|---|---|---|
|
| 629 |
+
| `high`, `xhigh`, `max`, `ultra` | xhigh | the model is told to think carefully and check its assumptions |
|
| 630 |
+
| `medium`, unset, `null` | medium | the default; no effort instruction is added |
|
| 631 |
+
| `low`, `minimal` | low | the model is told to keep its thinking brief |
|
| 632 |
+
| `none` | — | thinking **off** for that request, the same as `"think": false` |
|
| 633 |
+
|
| 634 |
+
`"think": "high"`, `"low"` and `"max"` on `/api/chat` map the same way. Mixed case
|
| 635 |
+
such as `"HIGH"` is rejected by Ollama itself with HTTP 400 before any template runs.
|
| 636 |
+
The value-to-level mapping comes from Ollama's source, unchanged from v0.33.3 through
|
| 637 |
+
v0.34.2 (2026-09-17).
|
| 638 |
+
|
| 639 |
+
Verified with `scripts/live_check.sh` on Ollama 0.33.3 and 0.34.0 — on the dense 27B
|
| 640 |
+
bundle, before the reversal: on a one-line chat, `high` and `max` each added 38 prompt
|
| 641 |
+
tokens and `low` 26, while `medium` and an unset value added none. Both the Go
|
| 642 |
+
template and Ollama's folding are model-independent, so the mapping carries over
|
| 643 |
+
unchanged; the live check has not been re-run against the MoE blob (`make live-check`
|
| 644 |
+
needs the 21 GB file on disk).
|
| 645 |
|
| 646 |
### Which template Ollama runs
|
| 647 |
|
| 648 |
+
Two templates exist for the same ChatML model: the Go `template` this repo ships
|
| 649 |
+
(served by HF's Ollama bridge) / `Modelfile` TEMPLATE, and **Qwen 3.6's own template
|
| 650 |
+
inside the GGUF**, which llmfan46 stamped and this repo leaves untouched. They do not
|
| 651 |
+
render identical prompts — the Go template pins JSON tool calls and adds a
|
| 652 |
+
reasoning-effort line from Ollama's think level, while the embedded template teaches
|
| 653 |
+
the XML `<function=…>` form and has no effort variable at all.
|
| 654 |
+
|
| 655 |
+
Ollama picks one at load time: it prefers the embedded template when that advertises
|
| 656 |
+
more capabilities — tools, thinking — and keeps the Go template when they tie. Both
|
| 657 |
+
advertise tools and thinking, so they tie and **Ollama runs the Go template**;
|
| 658 |
+
llama.cpp and LM Studio run the embedded one. That was verified live on Ollama 0.33.3
|
| 659 |
+
and 0.34.0 against the dense 27B bundle, and the selection code is unchanged through
|
| 660 |
+
v0.34.2 (source, 2026-09-17); the tie itself depends on what the *embedded* template
|
| 661 |
+
advertises, so it is expected rather than re-verified for this blob until
|
| 662 |
+
`make live-check` runs on it. Ollama's built-in renderers — it ships a `qwen35` one —
|
| 663 |
+
displace both templates only when the model's own config names a renderer or parser,
|
| 664 |
+
which this repo's does not. Ollama logs the choice:
|
| 665 |
|
| 666 |
```text
|
| 667 |
msg="template selection" ... selected=go_template go_template="[completion tools thinking]"
|
|
|
|
| 669 |
|
| 670 |
The Go template's "thinking" entry comes from its single `.Thinking` reference —
|
| 671 |
the block that renders each assistant turn's reasoning. Removing it drops that
|
| 672 |
+
capability, and Ollama switches to the embedded template: tool calls change from
|
| 673 |
+
JSON-in-XML to Qwen's XML form, which Ollama's extractor does not parse into a
|
| 674 |
+
`tool_calls` array, `reasoning_effort` stops doing anything (the embedded template
|
| 675 |
+
has no such variable), and a single tool call can come back twice (seen once on the
|
| 676 |
+
27B; cause inferred, not confirmed — see [Known limitations](#known-limitations)).
|
| 677 |
+
Setting `OLLAMA_GO_TEMPLATE=1` or `=0` on the Ollama server forces one template or
|
| 678 |
+
the other for every model.
|
| 679 |
|
| 680 |
### Tool / function calling
|
| 681 |
|
| 682 |
+
The model emits whichever format the prompt specifies, and the two paths specify
|
| 683 |
+
different ones: the Ollama path pins JSON-in-XML, while the llama.cpp path gets
|
| 684 |
+
Qwen's native XML form from the template embedded in the GGUF.
|
| 685 |
|
| 686 |
**Ollama path** (this repo's `Modelfile`). The TEMPLATE advertises tools inside `<tools>…</tools>` and asks the model to reply in JSON-in-XML — the form Ollama's tool-call extractor parses into a structured `tool_calls` array on `/api/chat` and `/v1/chat/completions`. The tool definitions go in with `{{ json .Function }}`: on Ollama 0.14 and later a bare `{{ .Function }}` prints Go struct syntax instead of JSON ([ollama/ollama#14601](https://github.com/ollama/ollama/issues/14601); fix proposed in [ollama/ollama#18391](https://github.com/ollama/ollama/pull/18391)). The model replies like this:
|
| 687 |
|
|
|
|
| 691 |
</tool_call>
|
| 692 |
```
|
| 693 |
|
| 694 |
+
**Embedded-template path** (llama.cpp, llama-cpp-python, LM Studio, KoboldCpp). The bundled GGUF carries Qwen 3.6's own template, so `--jinja` alone is enough — there is no file to pass and nothing to stamp. It teaches the XML form:
|
| 695 |
|
| 696 |
```text
|
| 697 |
<tool_call>
|
|
|
|
| 703 |
</tool_call>
|
| 704 |
```
|
| 705 |
|
| 706 |
+
**Keep XML on llama-server.** llama-server recognises this template as Qwen3-Coder-style and parses the model's reply with a parser — and, once the model opens `<tool_call>`, a grammar — that accept only `<function=…><parameter=…>`. Qwen 3.6's tool-instruction literal is byte-identical to Qwen 3.8's (compared 2026-09-18), so the detection and the wire format are exactly what this card documented before the reversal. What is gone is the `tool_call_format` knob: the JSON opt-in described in releases 0.3.0 through 0.6.5 lived in the patched 3.8 template this repo no longer ships.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 707 |
|
| 708 |
+
**Replayed tool calls: normalise `arguments` first.** Every OpenAI-compatible client hands assistant tool calls back with `function.arguments` as a JSON **string**, while the XML parameter form needs the individual key/value pairs — a chat template has `tojson` but no inverse. Qwen 3.6's template renders `<parameter=…>` only when `arguments` is a mapping; handed a string it emits that string as-is inside `<function=…>`, carrying upstream's own comment that you should "normalize arguments to a dict in your serving layer before applying this template". It does **not** raise, so a loader that passes the string straight through produces a silently malformed prompt rather than an error — the failure is quieter here than it was under the 3.8 template, which raised and returned HTTP 500.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 709 |
|
| 710 |
+
In practice the loaders convert before rendering: llama-server decodes the string into a mapping, KoboldCpp (`--jinja --jinja_tools`) runs `json.loads` first, and the LM Studio app parses the arguments before its own JavaScript Jinja engine renders. All three were checked on 2026-09-14 against the 3.8 template this repo shipped then; the conversion is loader-side, so it applies unchanged here (from source for this template — not re-tested on this blob).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 711 |
|
| 712 |
#### Example (Ollama, OpenAI-compatible API)
|
| 713 |
|
|
|
|
| 744 |
|
| 745 |
- Use direct prompts ("Call X for Y") rather than soft hints ("Use the tool"). The model thinks before committing to a call, and weak prompts can exhaust `num_predict` inside the `<think>` block before the call is emitted.
|
| 746 |
- Allow at least `num_predict: 1024` (or `max_tokens: 1024`) for tool-calling turns, more if the schemas are large.
|
| 747 |
+
- The JSON-in-XML format is what Ollama's tool-call extractor understands, and the Go template pins it; the GGUF's embedded template renders the XML `<function=…>` form. So the prompt format does change when you swap loaders, while client code should not have to, as long as the server returns a parsed `tool_calls` array.
|
| 748 |
|
| 749 |
## Known limitations
|
| 750 |
|
| 751 |
+
- **On the embedded-template path, a single tool call can come back twice.** Seen on the dense 27B blob, not re-tested here: in one test Ollama 0.33.3, running the embedded template, returned a first-turn tool call twice. A separate raw generation showed the model drafting the call inside `<think>` before emitting it once after `</think>`, so the parser most likely matched such a draft (inferred, not confirmed). It did not reproduce on standalone llama.cpp master (`a2878d3`, CPU): across both models, 12 first-turn calls (three prompts, with and without `parallel_tool_calls`) and two streamed runs each came back exactly once. Nor on b10760, the llama.cpp build Ollama 0.33.3 bundles and the one the duplicate came through: six more first-turn calls and a streamed run on this model each came back once. LM Studio's CPU engine (llama.cpp b10519) passed the same checks; the LM Studio app's own API layer is untested. Ollama's default path — the Go template, see [Which template Ollama runs](#which-template-ollama-runs) — returned one call in a comparable test; whether the prompt or the parser makes the difference was not isolated. If your client executes tools wherever the embedded template renders (llama.cpp, LM Studio, or Ollama with `OLLAMA_GO_TEMPLATE=0`), drop identical consecutive calls.
|
| 752 |
+
- **No mmproj in this release.** The base Qwen 3.6 35B-A3B supports image and video input via a separate `mmproj` file, which is not included here. Text-only inference works out of the box; multimodal inference requires fetching `Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf` (or equivalent) from upstream — run `./scripts/fetch_vision.sh` and see [Vision](#vision) for the full path.
|
| 753 |
- **Quantization-induced quality loss.** Q4_K_M is a strong general-purpose quant but does measurably degrade math and code accuracy compared to BF16. If you need maximum quality, run the upstream safetensors on a host with room for the full BF16 weights.
|
| 754 |
+
- **No MTP head on the GGUF path.** The base declares a multi-token-prediction layer (`mtp_num_hidden_layers: 1`), but the Q4_K_M published in llmfan46's GGUF repo — the file this repo bundles — carries no NextN block at all (see [Architecture](#architecture)). No speculative decoding here: use the upstream safetensors under vLLM / SGLang, or llmfan46's separate `…-Native-MTP-Preserved-GGUF` variant, if you need it.
|
| 755 |
+
- **Sparse compute, dense memory.** Only ~3B of ~34.7B parameters are touched per token, but every expert has to be resident: the 19.78 GiB of weights is a floor whatever the routing does, and MoE sparsity buys compute, not RAM. What it does buy on the memory side is a small KV cache — 0.625 GiB per 32K at f16, under a third of a comparable dense stack — so long contexts are unusually cheap here relative to the weights.
|
| 756 |
+
- **Thinking traces can loop.** Like most reasoning-distilled models, Janus-35B occasionally gets stuck repeating itself inside `<think>` tags. Mitigations: lower temperature to 0.4-0.6, raise `repeat_penalty` to 1.08, or set a `<think>`-token budget cap if your loader supports it.
|
| 757 |
- **Large tool-call arguments can be dropped.** Ollama's JSON-in-XML tool format makes the model JSON-escape the entire arguments object inline; for a big/complex payload (e.g. a file's `content` in a `write_file` call) the model can fail to escape it, so the field arrives `undefined` and the call fails. Qwen's native `<function=…><parameter=…>` format (raw values, no escaping) was tested as a fix but parses unreliably through Ollama, so the template deliberately keeps JSON-in-XML. Mitigation: write large files in smaller pieces per call.
|
| 758 |
- **Uncensored base — not aligned with any specific safety policy.** This is a personal repackage of an open-weight base whose refusal behavior has been abliterated away (the llmfan46 Heretic base). There is no RLHF refusal layer; the model will attempt most requests, so downstream safety is entirely the operator's responsibility.
|
| 759 |
+
- **No formal evaluation in this card, and no throughput figures at all.** The hardware table is derived from the GGUF's geometry, not measured on this blob, and every tok/s number the card used to carry was measured on the dense 27B and has been removed rather than rescaled. No task benchmarks are claimed either. If you produce real numbers (tok/s, MMLU, HumanEval, …) and want them included, file a PR.
|
| 760 |
|
| 761 |
## Related models
|
| 762 |
|
| 763 |
| Model | Size | Notes |
|
| 764 |
|---|---|---|
|
| 765 |
+
| [llmfan46/Qwen3.6-35B-A3B-uncensored-heretic](https://huggingface.co/llmfan46/Qwen3.6-35B-A3B-uncensored-heretic) | 35B-A3B MoE | **Immediate base.** Uncensored Heretic abliteration of Qwen 3.6 35B-A3B; `transformers`-native safetensors, and the only place the MTP layer survives. |
|
| 766 |
+
| [llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF](https://huggingface.co/llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF) | 35B-A3B MoE | Heretic GGUFs — BF16, Q3_K_L, Q3_K_M, Q4_K_M, Q4_K_S, Q5_K_M, Q5_K_S, Q6_K, Q8_0, plus the BF16 mmproj. MTP-clean, so they load as-is; the bundled Q4_K_M **is** this repo's Q4_K_M, byte-for-byte. |
|
| 767 |
+
| [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) | 35B-A3B MoE | Upstream pre-Heretic base model. `transformers`-native multimodal weights. |
|
| 768 |
+
| [FoolDev/Thanatos-27B-HERETIC](https://huggingface.co/FoolDev/Thanatos-27B-HERETIC) | 27B dense | Sibling repo, and now a genuinely different model: a **dense Qwen 3.8 27B**, abliterated from the ARA-based [`heretic-org/Qwen3.8-27B-heretic-ara`](https://huggingface.co/heretic-org/Qwen3.8-27B-heretic-ara). Same teacher (Fable 5), same tooling, newer Qwen generation, no experts. Pick Thanatos for a smaller resident footprint and a dense compute profile; pick Janus for 35B-class capacity at ~3B active. (The older `FoolDev/Thanatos-27B` and `Thanatos-27B-Heretic` slugs now 307 to this path.) |
|
| 769 |
+
| [Crownelius/Crow-9B-HERETIC-4.6](https://huggingface.co/Crownelius/Crow-9B-HERETIC-4.6) | 9B dense | Heretic-flavored fine-tune on a smaller 9B Qwen base. Useful as a fast first-pass model when 35B is too heavy for the host. |
|
| 770 |
|
| 771 |
+
> **Janus vs Thanatos.** Since 0.7.0 the two are genuinely different models, not two abliterations of one base. **Janus** is Qwen 3.6 **35B-A3B**: a mixture of experts, ~34.7B parameters total with ~3B active per token, 40 layers, 256 experts (8 routed + 1 shared), on llmfan46's Heretic base, and its bundled quant is MTP-clean so no strip is needed. **Thanatos** is a dense Qwen 3.8 **27B** — every parameter on every token, 64 layers — on heretic-org's ARA base, and its bundle *is* MTP-stripped, because that base ships the NextN block. They still share the 262,144 native window, the ChatML frame, the Claude Fable 5 teacher slot and this repo's tooling. Choose on shape, not only on abliteration: Janus for 35B-class capacity at ~3B active and a KV cache under a third the size (0.625 vs 2 GiB per 32K at f16), Thanatos for the smaller resident footprint (16.1 vs 19.78 GiB of weights) and a newer Qwen generation.
|
| 772 |
|
| 773 |
## Credits
|
| 774 |
|
| 775 |
+
- Immediate base: [llmfan46/Qwen3.6-35B-A3B-uncensored-heretic](https://huggingface.co/llmfan46/Qwen3.6-35B-A3B-uncensored-heretic) — Heretic-style abliteration of Qwen 3.6 35B-A3B
|
| 776 |
+
- Upstream base: [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) (Alibaba)
|
| 777 |
- Reasoning teacher: Claude Fable 5 (Anthropic)
|
| 778 |
- Distillation lineage and dataset curation: [Crownelius](https://huggingface.co/Crownelius)
|
| 779 |
|
|
|
|
|
|
|
|
|
|
|
|
@@ -1,642 +0,0 @@
|
|
| 1 |
-
{#- Janus-27B chat template for llama.cpp / llama-server (--jinja).
|
| 2 |
-
|
| 3 |
-
This is the base model's own embedded template (qwen3.8-safe-v3.1), with
|
| 4 |
-
two deliberate changes, each marked CHANGED FROM UPSTREAM in place:
|
| 5 |
-
1. `preserve_thinking` and `preserve_reasoning` never raise: when both are
|
| 6 |
-
set, `preserve_thinking` wins, where upstream raises on conflicting
|
| 7 |
-
values. Earlier turns' reasoning is replayed by default, as upstream
|
| 8 |
-
has it;
|
| 9 |
-
2. `reasoning_effort` is normalised instead of validated, so an unfamiliar
|
| 10 |
-
value falls back to the default rather than failing the request.
|
| 11 |
-
|
| 12 |
-
`tool_call_format` defaults to 'xml', as upstream ships it. 0.3.0 to 0.4.0
|
| 13 |
-
of this repo defaulted to 'auto'; 0.5.0 went back. llama-server parses this
|
| 14 |
-
template's output with its Qwen3-Coder parser, whose grammar accepts only
|
| 15 |
-
the XML form <tool_call><function=NAME><parameter=ARG>, and it turns a
|
| 16 |
-
client's JSON-string `function.arguments` into a mapping before rendering, so
|
| 17 |
-
a replayed tool call renders as XML there under either default and nothing
|
| 18 |
-
raises. Pinning 'xml' keeps the prompt in the only form that parser reads.
|
| 19 |
-
|
| 20 |
-
'auto' stays available for a loader that hands the template the raw JSON
|
| 21 |
-
string: the XML parameter form needs the individual key/value pairs and a
|
| 22 |
-
chat template has `tojson` but no inverse, so under 'xml' the template
|
| 23 |
-
raises there; 'auto' renders the whole prompt in JSON form instead. Opt in
|
| 24 |
-
through the loader's template kwargs, e.g.
|
| 25 |
-
chat_template_kwargs: {"tool_call_format": "auto"}. On llama-server 'auto'
|
| 26 |
-
changes nothing, since arguments arrive as mappings; 'json' would break its
|
| 27 |
-
parser.
|
| 28 |
-
|
| 29 |
-
Nothing else is modified. max_tool_arg_chars,
|
| 30 |
-
auto_disable_thinking_with_tools, the vision handling and the whitespace
|
| 31 |
-
contract are all upstream's. This file is what scripts/build.sh stamps into
|
| 32 |
-
the GGUF it builds.
|
| 33 |
-
|
| 34 |
-
Guarded by scripts/check_chat_template.py (check 8 in scripts/check.sh).
|
| 35 |
-
-#}
|
| 36 |
-
{%- set template_version = "qwen3.8-safe-v3.1" %}
|
| 37 |
-
{#- -------------------------------------------------------------------------
|
| 38 |
-
Whitespace contract: every tag opener in this file carries a leading minus,
|
| 39 |
-
so it strips the whitespace and newlines that precede it. All emitted text
|
| 40 |
-
comes from string literals, which makes the rendered prompt byte-exact and
|
| 41 |
-
free of incidental newlines. Comment tags need the minus just as much as
|
| 42 |
-
block tags do; a plain comment tag breaks the strip chain and leaks the
|
| 43 |
-
surrounding indentation into the prompt.
|
| 44 |
-
------------------------------------------------------------------------- -#}
|
| 45 |
-
{#- The message list has to exist before anything else can inspect it. -#}
|
| 46 |
-
{%- if not messages %}
|
| 47 |
-
{{- raise_exception('No messages provided.') }}
|
| 48 |
-
{%- endif %}
|
| 49 |
-
{#- -------------------------------------------------------------------------
|
| 50 |
-
Configuration
|
| 51 |
-
------------------------------------------------------------------------- -#}
|
| 52 |
-
{#- The default is 'xml', as upstream ships it (0.3.0 to 0.4.0 of this repo used
|
| 53 |
-
'auto'; see the header and "Resolve the tool-call wire format" below). -#}
|
| 54 |
-
{%- set _requested_tool_format =
|
| 55 |
-
tool_call_format if tool_call_format is defined else 'xml'
|
| 56 |
-
%}
|
| 57 |
-
{%- if _requested_tool_format not in ('auto', 'xml', 'json') %}
|
| 58 |
-
{{- raise_exception(
|
| 59 |
-
'Unexpected tool_call_format ' ~ _requested_tool_format ~
|
| 60 |
-
'. Supported values are xml (the default), auto, and json.'
|
| 61 |
-
) }}
|
| 62 |
-
{%- endif %}
|
| 63 |
-
{%- set add_vision_id = add_vision_id if add_vision_id is defined else false %}
|
| 64 |
-
{%- set enable_thinking = enable_thinking if enable_thinking is defined else true %}
|
| 65 |
-
{%- set auto_disable_thinking_with_tools =
|
| 66 |
-
auto_disable_thinking_with_tools
|
| 67 |
-
if auto_disable_thinking_with_tools is defined
|
| 68 |
-
else false
|
| 69 |
-
%}
|
| 70 |
-
{#- CHANGED FROM UPSTREAM: upstream raises when preserve_reasoning and
|
| 71 |
-
preserve_thinking are both set to different values, and otherwise lets
|
| 72 |
-
preserve_reasoning win. llama-server from b10763 passes
|
| 73 |
-
preserve_reasoning=true unless run with --no-reasoning-preserve, and a
|
| 74 |
-
request's preserve_thinking kwarg replaces the copy it derives from that, so
|
| 75 |
-
a request sending preserve_thinking=false made upstream raise there, as
|
| 76 |
-
preserve_thinking=true did against --no-reasoning-preserve on any build;
|
| 77 |
-
llama-server returns the raise as an HTTP 500. Here preserve_thinking wins
|
| 78 |
-
and nothing raises. With neither set, earlier turns' reasoning is replayed, as upstream
|
| 79 |
-
does. -#}
|
| 80 |
-
{%- if preserve_thinking is defined and preserve_thinking is not none %}
|
| 81 |
-
{%- set _preserve_thinking = preserve_thinking %}
|
| 82 |
-
{%- elif preserve_reasoning is defined and preserve_reasoning is not none %}
|
| 83 |
-
{%- set _preserve_thinking = preserve_reasoning %}
|
| 84 |
-
{%- else %}
|
| 85 |
-
{%- set _preserve_thinking = true %}
|
| 86 |
-
{%- endif %}
|
| 87 |
-
{%- set max_tool_arg_chars =
|
| 88 |
-
max_tool_arg_chars if max_tool_arg_chars is defined else 0
|
| 89 |
-
%}
|
| 90 |
-
{%- set max_tool_response_chars =
|
| 91 |
-
max_tool_response_chars if max_tool_response_chars is defined else 0
|
| 92 |
-
%}
|
| 93 |
-
{%- if max_tool_arg_chars is not number or max_tool_arg_chars < 0 %}
|
| 94 |
-
{{- raise_exception('max_tool_arg_chars must be a non-negative number.') }}
|
| 95 |
-
{%- endif %}
|
| 96 |
-
{%- if max_tool_response_chars is not number or max_tool_response_chars < 0 %}
|
| 97 |
-
{{- raise_exception('max_tool_response_chars must be a non-negative number.') }}
|
| 98 |
-
{%- endif %}
|
| 99 |
-
{%- if tools is defined and tools is not none and tools %}
|
| 100 |
-
{%- if tools is string or tools is mapping or tools is not iterable %}
|
| 101 |
-
{{- raise_exception('tools must be an iterable list of tool definitions.') }}
|
| 102 |
-
{%- endif %}
|
| 103 |
-
{%- set _has_tools = true %}
|
| 104 |
-
{%- else %}
|
| 105 |
-
{%- set _has_tools = false %}
|
| 106 |
-
{%- endif %}
|
| 107 |
-
{#- -------------------------------------------------------------------------
|
| 108 |
-
Resolve the tool-call wire format.
|
| 109 |
-
|
| 110 |
-
OpenAI-compatible servers hand assistant tool calls back with
|
| 111 |
-
`function.arguments` as a JSON *string*, not a mapping. The XML parameter
|
| 112 |
-
form needs the individual key/value pairs, and a chat template cannot
|
| 113 |
-
decode a JSON string: the standard chat-template environment provides
|
| 114 |
-
`tojson` but no inverse, and no codepoint-to-character primitive, so a
|
| 115 |
-
hand-rolled scanner could not decode \uXXXX escapes correctly. Guessing
|
| 116 |
-
would silently corrupt tool arguments, which is worse than any error.
|
| 117 |
-
|
| 118 |
-
The default is XML, matching Qwen's native tool-call wire format and the
|
| 119 |
-
Qwen3-Coder parser llama-server uses for this template - that parser and its
|
| 120 |
-
grammar accept nothing else. llama-server converts JSON-string arguments to
|
| 121 |
-
mappings before rendering, so they never reach this check there. 'auto'
|
| 122 |
-
(opt-in) renders the whole prompt in JSON form whenever a historical tool
|
| 123 |
-
call carries string arguments. That path is lossless, needs no decoding, and
|
| 124 |
-
keeps the system-prompt instructions and rendered history in one consistent
|
| 125 |
-
format. Histories whose arguments are all mappings still render as XML,
|
| 126 |
-
byte-identically to tool_call_format='xml'.
|
| 127 |
-
|
| 128 |
-
Under 'xml', the default, a caller still
|
| 129 |
-
gets a hard error for non-empty string arguments, because that request
|
| 130 |
-
cannot be honoured faithfully in the native XML parameter representation.
|
| 131 |
-
------------------------------------------------------------------------- -#}
|
| 132 |
-
{%- set format_scan = namespace(has_json_string_arguments=false) %}
|
| 133 |
-
{%- for message in messages %}
|
| 134 |
-
{%- if message.role == 'assistant'
|
| 135 |
-
and message.tool_calls is defined
|
| 136 |
-
and message.tool_calls is not none
|
| 137 |
-
and message.tool_calls
|
| 138 |
-
and message.tool_calls is not string
|
| 139 |
-
and message.tool_calls is not mapping
|
| 140 |
-
and message.tool_calls is iterable %}
|
| 141 |
-
{%- for tool_call in message.tool_calls %}
|
| 142 |
-
{%- if tool_call.function is defined
|
| 143 |
-
and tool_call.function is not none %}
|
| 144 |
-
{%- set _tc = tool_call.function %}
|
| 145 |
-
{%- else %}
|
| 146 |
-
{%- set _tc = tool_call %}
|
| 147 |
-
{%- endif %}
|
| 148 |
-
{%- if _tc.arguments is defined
|
| 149 |
-
and _tc.arguments is not none
|
| 150 |
-
and _tc.arguments is string
|
| 151 |
-
and _tc.arguments | trim
|
| 152 |
-
and _tc.arguments | trim != '{}' %}
|
| 153 |
-
{%- set format_scan.has_json_string_arguments = true %}
|
| 154 |
-
{%- endif %}
|
| 155 |
-
{%- endfor %}
|
| 156 |
-
{%- endif %}
|
| 157 |
-
{%- endfor %}
|
| 158 |
-
{%- if _requested_tool_format == 'auto' %}
|
| 159 |
-
{%- if format_scan.has_json_string_arguments %}
|
| 160 |
-
{%- set _tool_format = 'json' %}
|
| 161 |
-
{%- else %}
|
| 162 |
-
{%- set _tool_format = 'xml' %}
|
| 163 |
-
{%- endif %}
|
| 164 |
-
{%- else %}
|
| 165 |
-
{%- set _tool_format = _requested_tool_format %}
|
| 166 |
-
{%- endif %}
|
| 167 |
-
{%- set _thinking_enabled = enable_thinking %}
|
| 168 |
-
{%- if auto_disable_thinking_with_tools and _has_tools %}
|
| 169 |
-
{%- set _thinking_enabled = false %}
|
| 170 |
-
{%- endif %}
|
| 171 |
-
{#- Normalize and validate reasoning effort even when thinking is disabled.
|
| 172 |
-
Explicit null follows the template default; OpenAI-style `minimal` maps to
|
| 173 |
-
low, while `none` disables thinking for this render. -#}
|
| 174 |
-
{#- CHANGED FROM UPSTREAM: normalise instead of raising. Clients do not agree on
|
| 175 |
-
the vocabulary - OpenAI-style minimal/low/medium/high, xhigh, the "max" that
|
| 176 |
-
Ollama's own /v1 layer sends for xhigh and ultra, mixed case - and upstream
|
| 177 |
-
raised on anything off its list. Under Ollama >= 0.33 and
|
| 178 |
-
llama-server a template raise is an HTTP 500 for the entire request, so one
|
| 179 |
-
unfamiliar effort word made the model unusable. Values are lower-cased and
|
| 180 |
-
trimmed, aliases map to the nearest tier, and anything unrecognised falls back
|
| 181 |
-
to the medium default instead of failing the request. -#}
|
| 182 |
-
{%- if reasoning_effort is undefined or reasoning_effort is none %}
|
| 183 |
-
{%- set _effort_raw = '' %}
|
| 184 |
-
{%- else %}
|
| 185 |
-
{%- set _effort_raw = reasoning_effort | string | lower | trim %}
|
| 186 |
-
{%- endif %}
|
| 187 |
-
{%- if _effort_raw in ('none', 'off', 'disabled', 'false') %}
|
| 188 |
-
{%- set _thinking_enabled = false %}
|
| 189 |
-
{%- set _reasoning_effort = 'xhigh' %}
|
| 190 |
-
{%- elif _effort_raw in ('xhigh', 'high', 'max', 'ultra', 'maximum', 'highest', 'x-high', 'x_high', 'extra-high', 'extra_high', 'very-high', 'very_high') %}
|
| 191 |
-
{%- set _reasoning_effort = 'xhigh' %}
|
| 192 |
-
{%- elif _effort_raw in ('low', 'minimal', 'min') %}
|
| 193 |
-
{%- set _reasoning_effort = 'low' %}
|
| 194 |
-
{%- else %}
|
| 195 |
-
{#- medium, empty, auto, default, and anything unrecognised -#}
|
| 196 |
-
{%- set _reasoning_effort = 'medium' %}
|
| 197 |
-
{%- endif %}
|
| 198 |
-
{%- set reasoning_instructions = '' %}
|
| 199 |
-
{%- if _thinking_enabled %}
|
| 200 |
-
{%- if _reasoning_effort == 'xhigh' %}
|
| 201 |
-
{%- set reasoning_instructions =
|
| 202 |
-
'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.'
|
| 203 |
-
%}
|
| 204 |
-
{%- elif _reasoning_effort == 'low' %}
|
| 205 |
-
{%- set reasoning_instructions =
|
| 206 |
-
'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.'
|
| 207 |
-
%}
|
| 208 |
-
{%- endif %}
|
| 209 |
-
{%- endif %}
|
| 210 |
-
{#- -------------------------------------------------------------------------
|
| 211 |
-
Content rendering
|
| 212 |
-
------------------------------------------------------------------------- -#}
|
| 213 |
-
{%- set image_count = namespace(value=0) %}
|
| 214 |
-
{%- set video_count = namespace(value=0) %}
|
| 215 |
-
{%- macro render_content(content, do_vision_count, is_system_content=false) %}
|
| 216 |
-
{%- if content is string %}
|
| 217 |
-
{{- content }}
|
| 218 |
-
{%- elif content is iterable and content is not mapping %}
|
| 219 |
-
{%- for item in content %}
|
| 220 |
-
{%- if item is not mapping %}
|
| 221 |
-
{{- raise_exception(
|
| 222 |
-
'Content-list items must be mappings with a supported text, image, or video type.'
|
| 223 |
-
) }}
|
| 224 |
-
{%- endif %}
|
| 225 |
-
{%- set _item_type = item.type if 'type' in item else none %}
|
| 226 |
-
{%- if _item_type is not none
|
| 227 |
-
and _item_type not in ('text', 'image', 'image_url', 'video') %}
|
| 228 |
-
{{- raise_exception(
|
| 229 |
-
'Unexpected content item type ' ~ _item_type ~ '.'
|
| 230 |
-
) }}
|
| 231 |
-
{%- endif %}
|
| 232 |
-
{%- set _is_image =
|
| 233 |
-
_item_type == 'image'
|
| 234 |
-
or _item_type == 'image_url'
|
| 235 |
-
or 'image' in item
|
| 236 |
-
or 'image_url' in item
|
| 237 |
-
%}
|
| 238 |
-
{%- set _is_video =
|
| 239 |
-
_item_type == 'video'
|
| 240 |
-
or 'video' in item
|
| 241 |
-
%}
|
| 242 |
-
{%- if _is_image and _is_video %}
|
| 243 |
-
{{- raise_exception(
|
| 244 |
-
'A content item cannot contain both image and video data.'
|
| 245 |
-
) }}
|
| 246 |
-
{%- elif _item_type == 'text' and (_is_image or _is_video) %}
|
| 247 |
-
{{- raise_exception(
|
| 248 |
-
'A text content item cannot also contain image or video data.'
|
| 249 |
-
) }}
|
| 250 |
-
{%- elif _is_image %}
|
| 251 |
-
{%- if is_system_content %}
|
| 252 |
-
{{- raise_exception(
|
| 253 |
-
'System and developer messages cannot contain images.'
|
| 254 |
-
) }}
|
| 255 |
-
{%- endif %}
|
| 256 |
-
{%- if do_vision_count %}
|
| 257 |
-
{%- set image_count.value = image_count.value + 1 %}
|
| 258 |
-
{%- endif %}
|
| 259 |
-
{%- if add_vision_id %}
|
| 260 |
-
{{- 'Picture ' ~ image_count.value ~ ': ' }}
|
| 261 |
-
{%- endif %}
|
| 262 |
-
{{- '<|vision_start|><|image_pad|><|vision_end|>' }}
|
| 263 |
-
{%- elif _is_video %}
|
| 264 |
-
{%- if is_system_content %}
|
| 265 |
-
{{- raise_exception(
|
| 266 |
-
'System and developer messages cannot contain videos.'
|
| 267 |
-
) }}
|
| 268 |
-
{%- endif %}
|
| 269 |
-
{%- if do_vision_count %}
|
| 270 |
-
{%- set video_count.value = video_count.value + 1 %}
|
| 271 |
-
{%- endif %}
|
| 272 |
-
{%- if add_vision_id %}
|
| 273 |
-
{{- 'Video ' ~ video_count.value ~ ': ' }}
|
| 274 |
-
{%- endif %}
|
| 275 |
-
{{- '<|vision_start|><|video_pad|><|vision_end|>' }}
|
| 276 |
-
{%- elif _item_type == 'text' or 'text' in item %}
|
| 277 |
-
{%- if 'text' not in item or item.text is not string %}
|
| 278 |
-
{{- raise_exception(
|
| 279 |
-
'Text content items must contain a string text field.'
|
| 280 |
-
) }}
|
| 281 |
-
{%- endif %}
|
| 282 |
-
{{- item.text }}
|
| 283 |
-
{%- else %}
|
| 284 |
-
{{- raise_exception(
|
| 285 |
-
'Unexpected item in content list. Expected text, image, or video content.'
|
| 286 |
-
) }}
|
| 287 |
-
{%- endif %}
|
| 288 |
-
{%- endfor %}
|
| 289 |
-
{%- elif content is none or content is undefined %}
|
| 290 |
-
{{- '' }}
|
| 291 |
-
{%- else %}
|
| 292 |
-
{{- raise_exception('Unexpected content type.') }}
|
| 293 |
-
{%- endif %}
|
| 294 |
-
{%- endmacro %}
|
| 295 |
-
{#- -------------------------------------------------------------------------
|
| 296 |
-
Validate and merge leading system/developer messages
|
| 297 |
-
------------------------------------------------------------------------- -#}
|
| 298 |
-
{%- set head = namespace(count=0, seen_non_system=false) %}
|
| 299 |
-
{%- for message in messages %}
|
| 300 |
-
{%- set _is_system =
|
| 301 |
-
message.role == 'system' or message.role == 'developer'
|
| 302 |
-
%}
|
| 303 |
-
{%- if _is_system %}
|
| 304 |
-
{%- if head.seen_non_system %}
|
| 305 |
-
{{- raise_exception(
|
| 306 |
-
'System and developer messages must appear before all user, assistant, and tool messages.'
|
| 307 |
-
) }}
|
| 308 |
-
{%- endif %}
|
| 309 |
-
{%- set head.count = head.count + 1 %}
|
| 310 |
-
{%- else %}
|
| 311 |
-
{%- set head.seen_non_system = true %}
|
| 312 |
-
{%- endif %}
|
| 313 |
-
{%- endfor %}
|
| 314 |
-
{%- set system_state = namespace(content='') %}
|
| 315 |
-
{%- for message in messages[:head.count] %}
|
| 316 |
-
{%- set _part =
|
| 317 |
-
render_content(message.content, false, true) | trim
|
| 318 |
-
%}
|
| 319 |
-
{%- if _part %}
|
| 320 |
-
{%- if system_state.content %}
|
| 321 |
-
{%- set system_state.content =
|
| 322 |
-
system_state.content ~ '\n\n' ~ _part
|
| 323 |
-
%}
|
| 324 |
-
{%- else %}
|
| 325 |
-
{%- set system_state.content = _part %}
|
| 326 |
-
{%- endif %}
|
| 327 |
-
{%- endif %}
|
| 328 |
-
{%- endfor %}
|
| 329 |
-
{%- set _system_content = system_state.content %}
|
| 330 |
-
{%- set _msgs = messages[head.count:] %}
|
| 331 |
-
{#- -------------------------------------------------------------------------
|
| 332 |
-
System prompt and tool instructions
|
| 333 |
-
------------------------------------------------------------------------- -#}
|
| 334 |
-
{%- if _has_tools %}
|
| 335 |
-
{{- '<|im_start|>system\n' }}
|
| 336 |
-
{%- if reasoning_instructions %}
|
| 337 |
-
{{- reasoning_instructions ~ '\n\n' }}
|
| 338 |
-
{%- endif %}
|
| 339 |
-
{{- '# Tools\n\nYou have access to the following functions:\n\n<tools>' }}
|
| 340 |
-
{%- for tool in tools %}
|
| 341 |
-
{{- '\n' }}
|
| 342 |
-
{{- tool | tojson }}
|
| 343 |
-
{%- endfor %}
|
| 344 |
-
{{- '\n</tools>' }}
|
| 345 |
-
{%- if _tool_format == 'json' %}
|
| 346 |
-
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n{"name": "example_function_name", "arguments": {"example_parameter_1": "value_1", "example_parameter_2": "This is the value for the second parameter"}}\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: a single JSON object with "name" and "arguments" keys must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
|
| 347 |
-
{%- else %}
|
| 348 |
-
{{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
|
| 349 |
-
{%- endif %}
|
| 350 |
-
{%- if _system_content %}
|
| 351 |
-
{{- '\n\n' ~ _system_content }}
|
| 352 |
-
{%- endif %}
|
| 353 |
-
{{- '<|im_end|>\n' }}
|
| 354 |
-
{%- else %}
|
| 355 |
-
{%- if _system_content %}
|
| 356 |
-
{{- '<|im_start|>system\n'
|
| 357 |
-
~ (reasoning_instructions ~ '\n\n' if reasoning_instructions else '')
|
| 358 |
-
~ _system_content
|
| 359 |
-
~ '<|im_end|>\n'
|
| 360 |
-
}}
|
| 361 |
-
{%- elif reasoning_instructions %}
|
| 362 |
-
{{- '<|im_start|>system\n'
|
| 363 |
-
~ reasoning_instructions
|
| 364 |
-
~ '<|im_end|>\n'
|
| 365 |
-
}}
|
| 366 |
-
{%- endif %}
|
| 367 |
-
{%- endif %}
|
| 368 |
-
{#- -------------------------------------------------------------------------
|
| 369 |
-
Find the last real user query
|
| 370 |
-
------------------------------------------------------------------------- -#}
|
| 371 |
-
{%- set ns = namespace(
|
| 372 |
-
multi_step_tool=true,
|
| 373 |
-
last_query_index=(_msgs | length) - 1
|
| 374 |
-
) %}
|
| 375 |
-
{%- for message in _msgs[::-1] %}
|
| 376 |
-
{%- set index = (_msgs | length - 1) - loop.index0 %}
|
| 377 |
-
{%- if ns.multi_step_tool and message.role == 'user' %}
|
| 378 |
-
{%- set _rendered_user =
|
| 379 |
-
render_content(message.content, false) | trim
|
| 380 |
-
%}
|
| 381 |
-
{%- if not (
|
| 382 |
-
_rendered_user.startswith('<tool_response>')
|
| 383 |
-
and _rendered_user.endswith('</tool_response>')
|
| 384 |
-
) %}
|
| 385 |
-
{%- set ns.multi_step_tool = false %}
|
| 386 |
-
{%- set ns.last_query_index = index %}
|
| 387 |
-
{%- endif %}
|
| 388 |
-
{%- endif %}
|
| 389 |
-
{%- endfor %}
|
| 390 |
-
{%- if ns.multi_step_tool %}
|
| 391 |
-
{{- raise_exception('No user query found in messages.') }}
|
| 392 |
-
{%- endif %}
|
| 393 |
-
{#- -------------------------------------------------------------------------
|
| 394 |
-
Conversation rendering
|
| 395 |
-
------------------------------------------------------------------------- -#}
|
| 396 |
-
{%- set history_state = namespace(previous_role='') %}
|
| 397 |
-
{%- for message in _msgs %}
|
| 398 |
-
{%- set content =
|
| 399 |
-
render_content(message.content, true) | trim
|
| 400 |
-
%}
|
| 401 |
-
{%- if message.role == 'user' %}
|
| 402 |
-
{{- '<|im_start|>user\n'
|
| 403 |
-
~ content
|
| 404 |
-
~ '<|im_end|>\n'
|
| 405 |
-
}}
|
| 406 |
-
{%- elif message.role == 'assistant' %}
|
| 407 |
-
{%- set reasoning_content = '' %}
|
| 408 |
-
{#- Prefer current vLLM's `reasoning` field, then retain compatibility
|
| 409 |
-
with older `reasoning_content` and other servers' `thinking`. -#}
|
| 410 |
-
{%- if message.reasoning is defined
|
| 411 |
-
and message.reasoning is not none %}
|
| 412 |
-
{%- if message.reasoning is string %}
|
| 413 |
-
{%- set reasoning_content = message.reasoning %}
|
| 414 |
-
{%- else %}
|
| 415 |
-
{%- set reasoning_content =
|
| 416 |
-
message.reasoning | string
|
| 417 |
-
%}
|
| 418 |
-
{%- endif %}
|
| 419 |
-
{%- elif message.reasoning_content is defined
|
| 420 |
-
and message.reasoning_content is not none %}
|
| 421 |
-
{%- if message.reasoning_content is string %}
|
| 422 |
-
{%- set reasoning_content = message.reasoning_content %}
|
| 423 |
-
{%- else %}
|
| 424 |
-
{%- set reasoning_content =
|
| 425 |
-
message.reasoning_content | string
|
| 426 |
-
%}
|
| 427 |
-
{%- endif %}
|
| 428 |
-
{%- elif message.thinking is defined
|
| 429 |
-
and message.thinking is not none %}
|
| 430 |
-
{%- if message.thinking is string %}
|
| 431 |
-
{%- set reasoning_content = message.thinking %}
|
| 432 |
-
{%- else %}
|
| 433 |
-
{%- set reasoning_content =
|
| 434 |
-
message.thinking | string
|
| 435 |
-
%}
|
| 436 |
-
{%- endif %}
|
| 437 |
-
{%- endif %}
|
| 438 |
-
{%- set reasoning_content = reasoning_content | trim %}
|
| 439 |
-
{#- Preserve the original invariant: when thinking is preserved,
|
| 440 |
-
always reconstruct the thinking block, even when it is empty. -#}
|
| 441 |
-
{%- if _preserve_thinking
|
| 442 |
-
or loop.index0 > ns.last_query_index %}
|
| 443 |
-
{{- '<|im_start|>assistant\n<think>\n'
|
| 444 |
-
~ reasoning_content
|
| 445 |
-
~ '\n</think>\n\n'
|
| 446 |
-
~ content
|
| 447 |
-
}}
|
| 448 |
-
{%- else %}
|
| 449 |
-
{{- '<|im_start|>assistant\n' ~ content }}
|
| 450 |
-
{%- endif %}
|
| 451 |
-
{%- if message.tool_calls is defined
|
| 452 |
-
and message.tool_calls is not none
|
| 453 |
-
and message.tool_calls %}
|
| 454 |
-
{%- if message.tool_calls is string
|
| 455 |
-
or message.tool_calls is mapping
|
| 456 |
-
or message.tool_calls is not iterable %}
|
| 457 |
-
{{- raise_exception(
|
| 458 |
-
'assistant.tool_calls must be an iterable list of tool calls.'
|
| 459 |
-
) }}
|
| 460 |
-
{%- endif %}
|
| 461 |
-
{%- for tool_call in message.tool_calls %}
|
| 462 |
-
{%- if tool_call.function is defined
|
| 463 |
-
and tool_call.function is not none %}
|
| 464 |
-
{%- set tc = tool_call.function %}
|
| 465 |
-
{%- else %}
|
| 466 |
-
{%- set tc = tool_call %}
|
| 467 |
-
{%- endif %}
|
| 468 |
-
{%- if tc.name is not defined
|
| 469 |
-
or tc.name is none
|
| 470 |
-
or tc.name is not string
|
| 471 |
-
or not tc.name %}
|
| 472 |
-
{{- raise_exception(
|
| 473 |
-
'Every tool call must have a non-empty string name.'
|
| 474 |
-
) }}
|
| 475 |
-
{%- endif %}
|
| 476 |
-
{%- set tc_name = tc.name %}
|
| 477 |
-
{%- if _tool_format == 'json' %}
|
| 478 |
-
{%- if loop.first %}
|
| 479 |
-
{%- if content | trim %}
|
| 480 |
-
{{- '\n\n<tool_call>\n' }}
|
| 481 |
-
{%- else %}
|
| 482 |
-
{{- '<tool_call>\n' }}
|
| 483 |
-
{%- endif %}
|
| 484 |
-
{%- else %}
|
| 485 |
-
{{- '\n<tool_call>\n' }}
|
| 486 |
-
{%- endif %}
|
| 487 |
-
{%- set _json_arguments = '{}' %}
|
| 488 |
-
{%- if tc.arguments is defined
|
| 489 |
-
and tc.arguments is not none %}
|
| 490 |
-
{%- if tc.arguments is mapping %}
|
| 491 |
-
{%- set _json_arguments =
|
| 492 |
-
tc.arguments | tojson
|
| 493 |
-
%}
|
| 494 |
-
{%- elif tc.arguments is string %}
|
| 495 |
-
{%- set _raw_arguments =
|
| 496 |
-
tc.arguments | trim
|
| 497 |
-
%}
|
| 498 |
-
{%- if _raw_arguments %}
|
| 499 |
-
{%- if not (
|
| 500 |
-
_raw_arguments.startswith('{')
|
| 501 |
-
and _raw_arguments.endswith('}')
|
| 502 |
-
) %}
|
| 503 |
-
{{- raise_exception(
|
| 504 |
-
'JSON tool-call arguments must be a JSON object string.'
|
| 505 |
-
) }}
|
| 506 |
-
{%- endif %}
|
| 507 |
-
{%- set _json_arguments =
|
| 508 |
-
_raw_arguments
|
| 509 |
-
%}
|
| 510 |
-
{%- endif %}
|
| 511 |
-
{%- else %}
|
| 512 |
-
{{- raise_exception(
|
| 513 |
-
'JSON tool-call arguments must be a mapping or a JSON object string.'
|
| 514 |
-
) }}
|
| 515 |
-
{%- endif %}
|
| 516 |
-
{%- endif %}
|
| 517 |
-
{{- '{"name": ' }}
|
| 518 |
-
{{- tc_name | tojson }}
|
| 519 |
-
{{- ', "arguments": ' }}
|
| 520 |
-
{{- _json_arguments }}
|
| 521 |
-
{{- '}\n</tool_call>' }}
|
| 522 |
-
{%- else %}
|
| 523 |
-
{%- if loop.first %}
|
| 524 |
-
{%- if content | trim %}
|
| 525 |
-
{{- '\n\n<tool_call>\n<function='
|
| 526 |
-
~ tc_name
|
| 527 |
-
~ '>\n'
|
| 528 |
-
}}
|
| 529 |
-
{%- else %}
|
| 530 |
-
{{- '<tool_call>\n<function='
|
| 531 |
-
~ tc_name
|
| 532 |
-
~ '>\n'
|
| 533 |
-
}}
|
| 534 |
-
{%- endif %}
|
| 535 |
-
{%- else %}
|
| 536 |
-
{{- '\n<tool_call>\n<function='
|
| 537 |
-
~ tc_name
|
| 538 |
-
~ '>\n'
|
| 539 |
-
}}
|
| 540 |
-
{%- endif %}
|
| 541 |
-
{%- if tc.arguments is defined
|
| 542 |
-
and tc.arguments is not none %}
|
| 543 |
-
{%- if tc.arguments is mapping %}
|
| 544 |
-
{%- for args_name, args_value
|
| 545 |
-
in tc.arguments | items %}
|
| 546 |
-
{%- set _argument_name =
|
| 547 |
-
args_name | string
|
| 548 |
-
%}
|
| 549 |
-
{{- '<parameter='
|
| 550 |
-
~ _argument_name
|
| 551 |
-
~ '>\n'
|
| 552 |
-
}}
|
| 553 |
-
{#- Preserve strings verbatim. Serialize every
|
| 554 |
-
other type as JSON so booleans and null stay
|
| 555 |
-
true/false/null instead of True/False/None. -#}
|
| 556 |
-
{%- if args_value is string %}
|
| 557 |
-
{%- set _argument_value =
|
| 558 |
-
args_value
|
| 559 |
-
%}
|
| 560 |
-
{%- else %}
|
| 561 |
-
{%- set _argument_value =
|
| 562 |
-
args_value | tojson
|
| 563 |
-
%}
|
| 564 |
-
{%- endif %}
|
| 565 |
-
{%- if max_tool_arg_chars > 0
|
| 566 |
-
and _argument_value | length
|
| 567 |
-
> max_tool_arg_chars %}
|
| 568 |
-
{{- _argument_value[:max_tool_arg_chars] }}
|
| 569 |
-
{{- '\n[TRUNCATED — original length '
|
| 570 |
-
~ (_argument_value | length | string)
|
| 571 |
-
~ ' chars]'
|
| 572 |
-
}}
|
| 573 |
-
{%- else %}
|
| 574 |
-
{{- _argument_value }}
|
| 575 |
-
{%- endif %}
|
| 576 |
-
{{- '\n</parameter>\n' }}
|
| 577 |
-
{%- endfor %}
|
| 578 |
-
{%- elif tc.arguments is string %}
|
| 579 |
-
{%- set _raw_arguments =
|
| 580 |
-
tc.arguments | trim
|
| 581 |
-
%}
|
| 582 |
-
{%- if _raw_arguments
|
| 583 |
-
and _raw_arguments != '{}' %}
|
| 584 |
-
{{- raise_exception(
|
| 585 |
-
'tool_call_format="xml" requires tool-call arguments to be mappings, but received a JSON string. Parse the JSON string into a mapping before applying the template, or set tool_call_format="auto" (or "json") to use the lossless JSON tool-call representation.'
|
| 586 |
-
) }}
|
| 587 |
-
{%- endif %}
|
| 588 |
-
{%- else %}
|
| 589 |
-
{{- raise_exception(
|
| 590 |
-
'XML tool-call arguments must be a mapping. Parse JSON-string arguments before applying the template.'
|
| 591 |
-
) }}
|
| 592 |
-
{%- endif %}
|
| 593 |
-
{%- endif %}
|
| 594 |
-
{{- '</function>\n</tool_call>' }}
|
| 595 |
-
{%- endif %}
|
| 596 |
-
{%- endfor %}
|
| 597 |
-
{%- endif %}
|
| 598 |
-
{{- '<|im_end|>\n' }}
|
| 599 |
-
{%- elif message.role == 'tool' %}
|
| 600 |
-
{%- if history_state.previous_role != 'tool' %}
|
| 601 |
-
{{- '<|im_start|>user' }}
|
| 602 |
-
{%- endif %}
|
| 603 |
-
{%- if max_tool_response_chars > 0
|
| 604 |
-
and content | length > max_tool_response_chars %}
|
| 605 |
-
{%- set content =
|
| 606 |
-
content[:max_tool_response_chars]
|
| 607 |
-
~ '\n[TRUNCATED — original length '
|
| 608 |
-
~ (content | length | string)
|
| 609 |
-
~ ' chars]'
|
| 610 |
-
%}
|
| 611 |
-
{%- endif %}
|
| 612 |
-
{{- '\n<tool_response>\n' }}
|
| 613 |
-
{{- content }}
|
| 614 |
-
{{- '\n</tool_response>' }}
|
| 615 |
-
{%- if loop.last %}
|
| 616 |
-
{{- '<|im_end|>\n' }}
|
| 617 |
-
{%- elif _msgs[loop.index0 + 1].role != 'tool' %}
|
| 618 |
-
{{- '<|im_end|>\n' }}
|
| 619 |
-
{%- endif %}
|
| 620 |
-
{%- elif message.role == 'system'
|
| 621 |
-
or message.role == 'developer' %}
|
| 622 |
-
{{- raise_exception(
|
| 623 |
-
'System and developer messages must appear only at the beginning of the conversation.'
|
| 624 |
-
) }}
|
| 625 |
-
{%- else %}
|
| 626 |
-
{{- raise_exception(
|
| 627 |
-
'Unexpected message role ' ~ message.role ~ '.'
|
| 628 |
-
) }}
|
| 629 |
-
{%- endif %}
|
| 630 |
-
{%- set history_state.previous_role = message.role %}
|
| 631 |
-
{%- endfor %}
|
| 632 |
-
{#- -------------------------------------------------------------------------
|
| 633 |
-
Generation prompt
|
| 634 |
-
------------------------------------------------------------------------- -#}
|
| 635 |
-
{%- if add_generation_prompt %}
|
| 636 |
-
{{- '<|im_start|>assistant\n' }}
|
| 637 |
-
{%- if not _thinking_enabled %}
|
| 638 |
-
{{- '<think>\n\n</think>\n\n' }}
|
| 639 |
-
{%- else %}
|
| 640 |
-
{{- '<think>\n' }}
|
| 641 |
-
{%- endif %}
|
| 642 |
-
{%- endif %}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@@ -1,13 +1,13 @@
|
|
| 1 |
-
# Janus-
|
| 2 |
|
| 3 |
Four minimal entry points. Pick the one that matches how you run models.
|
| 4 |
|
| 5 |
| File | Backend | When to use |
|
| 6 |
|---|---|---|
|
| 7 |
| `ollama_chat.py` | Ollama HTTP API | You already have `ollama serve` running and the `janus` model created from the project `Modelfile`. **Text + tool calling** — vision via Ollama is broken upstream for this arch. |
|
| 8 |
-
| `transformers_quickstart.py` | Hugging Face Transformers | You want to run the Heretic safetensors (`llmfan46/Qwen3.
|
| 9 |
| `llama_cpp_quickstart.py` | llama-cpp-python | You want to invoke a local GGUF directly without a daemon (CI, batch jobs, scripts). Text only. |
|
| 10 |
-
| `llama_cpp_vision.py` | llama-cpp-python + mmproj | **Image input.** Loads a text GGUF + `Qwen3.
|
| 11 |
|
| 12 |
The text clients apply the same Janus system prompt; `llama_cpp_vision.py` uses a task-specific vision prompt. The three non-Ollama scripts
|
| 13 |
explicitly pin the model card's tighter-reasoning sampling profile
|
|
@@ -27,7 +27,7 @@ root-level `template` / `system` / `params` files via HF's Ollama
|
|
| 27 |
bridge):
|
| 28 |
|
| 29 |
```bash
|
| 30 |
-
ollama pull hf.co/FoolDev/Janus-35B-HERETIC # bundled Q4_K_M,
|
| 31 |
pip install requests
|
| 32 |
MODEL=hf.co/FoolDev/Janus-35B-HERETIC python ollama_chat.py
|
| 33 |
```
|
|
@@ -42,8 +42,9 @@ python ollama_chat.py # MODEL defaults to `jan
|
|
| 42 |
|
| 43 |
For a non-bundled quant (e.g. Q3_K_M, Q5_K_M), `./scripts/build.sh`
|
| 44 |
downloads it from
|
| 45 |
-
`llmfan46/Qwen3.
|
| 46 |
-
|
|
|
|
| 47 |
temp copy, and creates the local `janus` tag:
|
| 48 |
|
| 49 |
```bash
|
|
@@ -54,7 +55,7 @@ python ollama_chat.py
|
|
| 54 |
### Transformers (safetensors)
|
| 55 |
|
| 56 |
```bash
|
| 57 |
-
pip install --upgrade transformers accelerate sentencepiece bitsandbytes # needs a build new enough for Qwen 3.
|
| 58 |
python transformers_quickstart.py # 4-bit (bitsandbytes)
|
| 59 |
python transformers_quickstart.py --no-4bit # bf16
|
| 60 |
```
|
|
@@ -63,7 +64,7 @@ python transformers_quickstart.py --no-4bit # bf16
|
|
| 63 |
|
| 64 |
```bash
|
| 65 |
pip install llama-cpp-python # CPU-only build
|
| 66 |
-
python llama_cpp_quickstart.py /path/to/Janus-
|
| 67 |
```
|
| 68 |
|
| 69 |
For GPU offload, rebuild llama-cpp-python with the matching backend (see
|
|
@@ -78,14 +79,14 @@ cd .. && ./scripts/fetch_vision.sh && cd examples
|
|
| 78 |
|
| 79 |
pip install llama-cpp-python pillow
|
| 80 |
python llama_cpp_vision.py \
|
| 81 |
-
--gguf ../Janus-
|
| 82 |
-
--mmproj ../Qwen3.
|
| 83 |
--image /path/to/photo.jpg \
|
| 84 |
--prompt "Describe this image."
|
| 85 |
```
|
| 86 |
|
| 87 |
-
Why not Ollama? Ollama's Go engine has the `
|
| 88 |
-
(text inference
|
| 89 |
Ollama switches to when an mmproj is attached still lacks it.
|
| 90 |
`ollama create` accepts the dual-`FROM` and `ollama show` reports
|
| 91 |
`vision` capability, but the first inference call fails with an
|
|
|
|
| 1 |
+
# Janus-35B examples
|
| 2 |
|
| 3 |
Four minimal entry points. Pick the one that matches how you run models.
|
| 4 |
|
| 5 |
| File | Backend | When to use |
|
| 6 |
|---|---|---|
|
| 7 |
| `ollama_chat.py` | Ollama HTTP API | You already have `ollama serve` running and the `janus` model created from the project `Modelfile`. **Text + tool calling** — vision via Ollama is broken upstream for this arch. |
|
| 8 |
+
| `transformers_quickstart.py` | Hugging Face Transformers | You want to run the Heretic safetensors (`llmfan46/Qwen3.6-35B-A3B-uncensored-heretic`) on GPU, optionally in 4-bit via bitsandbytes. |
|
| 9 |
| `llama_cpp_quickstart.py` | llama-cpp-python | You want to invoke a local GGUF directly without a daemon (CI, batch jobs, scripts). Text only. |
|
| 10 |
+
| `llama_cpp_vision.py` | llama-cpp-python + mmproj | **Image input.** Loads a text GGUF + `Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf` and answers questions about an image. The only working vision path right now. |
|
| 11 |
|
| 12 |
The text clients apply the same Janus system prompt; `llama_cpp_vision.py` uses a task-specific vision prompt. The three non-Ollama scripts
|
| 13 |
explicitly pin the model card's tighter-reasoning sampling profile
|
|
|
|
| 27 |
bridge):
|
| 28 |
|
| 29 |
```bash
|
| 30 |
+
ollama pull hf.co/FoolDev/Janus-35B-HERETIC # bundled Q4_K_M, 21.23 GB
|
| 31 |
pip install requests
|
| 32 |
MODEL=hf.co/FoolDev/Janus-35B-HERETIC python ollama_chat.py
|
| 33 |
```
|
|
|
|
| 42 |
|
| 43 |
For a non-bundled quant (e.g. Q3_K_M, Q5_K_M), `./scripts/build.sh`
|
| 44 |
downloads it from
|
| 45 |
+
`llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF`,
|
| 46 |
+
passes it through `strip_mtp.py` (a no-op on those quants — they ship
|
| 47 |
+
MTP-clean), patches the `Modelfile` `FROM` line into a
|
| 48 |
temp copy, and creates the local `janus` tag:
|
| 49 |
|
| 50 |
```bash
|
|
|
|
| 55 |
### Transformers (safetensors)
|
| 56 |
|
| 57 |
```bash
|
| 58 |
+
pip install --upgrade transformers accelerate sentencepiece bitsandbytes # needs a build new enough for Qwen 3.6
|
| 59 |
python transformers_quickstart.py # 4-bit (bitsandbytes)
|
| 60 |
python transformers_quickstart.py --no-4bit # bf16
|
| 61 |
```
|
|
|
|
| 64 |
|
| 65 |
```bash
|
| 66 |
pip install llama-cpp-python # CPU-only build
|
| 67 |
+
python llama_cpp_quickstart.py /path/to/Janus-35B-A3B.Q4_K_M.gguf
|
| 68 |
```
|
| 69 |
|
| 70 |
For GPU offload, rebuild llama-cpp-python with the matching backend (see
|
|
|
|
| 79 |
|
| 80 |
pip install llama-cpp-python pillow
|
| 81 |
python llama_cpp_vision.py \
|
| 82 |
+
--gguf ../Janus-35B-A3B.Q4_K_M.gguf \
|
| 83 |
+
--mmproj ../Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf \
|
| 84 |
--image /path/to/photo.jpg \
|
| 85 |
--prompt "Describe this image."
|
| 86 |
```
|
| 87 |
|
| 88 |
+
Why not Ollama? Ollama's Go engine has the `qwen35moe` arch entry
|
| 89 |
+
(text inference checked in source at 0.33.3, 0.34.0 and 0.34.2), but the C++ llama.cpp fallback that
|
| 90 |
Ollama switches to when an mmproj is attached still lacks it.
|
| 91 |
`ollama create` accepts the dual-`FROM` and `ollama show` reports
|
| 92 |
`vision` capability, but the first inference call fails with an
|
|
@@ -1,6 +1,6 @@
|
|
| 1 |
#!/usr/bin/env python3
|
| 2 |
"""
|
| 3 |
-
Janus-
|
| 4 |
|
| 5 |
Skip Ollama entirely and call the GGUF directly through llama-cpp-python.
|
| 6 |
Useful for batch jobs, CI, or environments where you don't want a daemon.
|
|
@@ -14,7 +14,7 @@ For GPU offload (CUDA / Metal / ROCm), install with the matching extras:
|
|
| 14 |
CMAKE_ARGS="-DGGML_HIPBLAS=on" pip install llama-cpp-python --no-binary :all:
|
| 15 |
|
| 16 |
Usage:
|
| 17 |
-
python llama_cpp_quickstart.py /path/to/Janus-
|
| 18 |
python llama_cpp_quickstart.py /path/to/file.gguf --gpu-layers 99
|
| 19 |
python llama_cpp_quickstart.py /path/to/file.gguf --prompt "..."
|
| 20 |
"""
|
|
@@ -49,7 +49,7 @@ JANUS_SYSTEM = (
|
|
| 49 |
|
| 50 |
def main() -> None:
|
| 51 |
ap = argparse.ArgumentParser()
|
| 52 |
-
ap.add_argument("gguf", help="Path to Janus-
|
| 53 |
ap.add_argument(
|
| 54 |
"--prompt",
|
| 55 |
default="Explain the Burrows-Wheeler transform in 200 words.",
|
|
|
|
| 1 |
#!/usr/bin/env python3
|
| 2 |
"""
|
| 3 |
+
Janus-35B — llama-cpp-python quickstart.
|
| 4 |
|
| 5 |
Skip Ollama entirely and call the GGUF directly through llama-cpp-python.
|
| 6 |
Useful for batch jobs, CI, or environments where you don't want a daemon.
|
|
|
|
| 14 |
CMAKE_ARGS="-DGGML_HIPBLAS=on" pip install llama-cpp-python --no-binary :all:
|
| 15 |
|
| 16 |
Usage:
|
| 17 |
+
python llama_cpp_quickstart.py /path/to/Janus-35B-A3B.Q4_K_M.gguf
|
| 18 |
python llama_cpp_quickstart.py /path/to/file.gguf --gpu-layers 99
|
| 19 |
python llama_cpp_quickstart.py /path/to/file.gguf --prompt "..."
|
| 20 |
"""
|
|
|
|
| 49 |
|
| 50 |
def main() -> None:
|
| 51 |
ap = argparse.ArgumentParser()
|
| 52 |
+
ap.add_argument("gguf", help="Path to Janus-35B-A3B GGUF (e.g. Q4_K_M).")
|
| 53 |
ap.add_argument(
|
| 54 |
"--prompt",
|
| 55 |
default="Explain the Burrows-Wheeler transform in 200 words.",
|
|
@@ -1,9 +1,9 @@
|
|
| 1 |
#!/usr/bin/env python3
|
| 2 |
"""
|
| 3 |
-
Janus-
|
| 4 |
|
| 5 |
Why this script exists:
|
| 6 |
-
Ollama's Go engine has the
|
| 7 |
on 0.24+), but the C++ llama.cpp fallback that Ollama switches to
|
| 8 |
when an mmproj is attached still lacks it. Both `FROM mmproj.gguf`
|
| 9 |
and `ADAPTER mmproj.gguf` fail at the first inference call with an
|
|
@@ -24,26 +24,26 @@ Install:
|
|
| 24 |
# CMAKE_ARGS="-DGGML_HIPBLAS=on" pip install llama-cpp-python --no-binary :all:
|
| 25 |
|
| 26 |
Files you need:
|
| 27 |
-
1. A text GGUF (any quant): the bundled Janus-
|
| 28 |
-
or another quant from
|
| 29 |
-
llmfan46/Qwen3.
|
| 30 |
-
—
|
| 31 |
-
|
| 32 |
2. A vision projector:
|
| 33 |
-
Qwen3.
|
| 34 |
— run ./scripts/fetch_vision.sh to pull it.
|
| 35 |
|
| 36 |
Usage:
|
| 37 |
python llama_cpp_vision.py \
|
| 38 |
-
--gguf Janus-
|
| 39 |
-
--mmproj Qwen3.
|
| 40 |
--image /path/to/photo.jpg \
|
| 41 |
--prompt "What is in this image? Be specific."
|
| 42 |
|
| 43 |
# CLI alternative without python binding (ships with llama.cpp):
|
| 44 |
# llama-mtmd-cli \
|
| 45 |
-
# -m Janus-
|
| 46 |
-
# --mmproj Qwen3.
|
| 47 |
# --image photo.jpg \
|
| 48 |
# -p "Describe this image."
|
| 49 |
"""
|
|
@@ -79,7 +79,7 @@ def encode_image_data_uri(path: Path) -> str:
|
|
| 79 |
|
| 80 |
def main() -> None:
|
| 81 |
ap = argparse.ArgumentParser()
|
| 82 |
-
ap.add_argument("--gguf", required=True, help="Text GGUF (e.g. Janus-
|
| 83 |
ap.add_argument("--mmproj", required=True, help="Vision projector GGUF (mmproj-BF16.gguf).")
|
| 84 |
ap.add_argument("--image", required=True, help="Image to analyze.")
|
| 85 |
ap.add_argument("--prompt", default="Describe this image in detail.")
|
|
@@ -98,7 +98,7 @@ def main() -> None:
|
|
| 98 |
sys.exit(f"Image not found: {image_path}")
|
| 99 |
|
| 100 |
# Qwen 2.5 VL chat handler is the closest match shipped with
|
| 101 |
-
# llama-cpp-python; Qwen 3.5 / 3.6
|
| 102 |
# layout. If/when llama-cpp-python ships a Qwen3VLChatHandler, swap it in.
|
| 103 |
handler = Qwen25VLChatHandler(clip_model_path=args.mmproj)
|
| 104 |
|
|
|
|
| 1 |
#!/usr/bin/env python3
|
| 2 |
"""
|
| 3 |
+
Janus-35B — vision (image-text-to-text) via llama-cpp-python.
|
| 4 |
|
| 5 |
Why this script exists:
|
| 6 |
+
Ollama's Go engine has the qwen35moe arch entry (text inference works
|
| 7 |
on 0.24+), but the C++ llama.cpp fallback that Ollama switches to
|
| 8 |
when an mmproj is attached still lacks it. Both `FROM mmproj.gguf`
|
| 9 |
and `ADAPTER mmproj.gguf` fail at the first inference call with an
|
|
|
|
| 24 |
# CMAKE_ARGS="-DGGML_HIPBLAS=on" pip install llama-cpp-python --no-binary :all:
|
| 25 |
|
| 26 |
Files you need:
|
| 27 |
+
1. A text GGUF (any quant): the bundled Janus-35B-A3B.Q4_K_M.gguf
|
| 28 |
+
(21.23 GB), or another quant from
|
| 29 |
+
llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF
|
| 30 |
+
— those ship MTP-clean, so no ./scripts/strip_mtp.py pass is needed;
|
| 31 |
+
only llmfan46's separate -Native-MTP-Preserved-GGUF variant needs it.
|
| 32 |
2. A vision projector:
|
| 33 |
+
Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf
|
| 34 |
— run ./scripts/fetch_vision.sh to pull it.
|
| 35 |
|
| 36 |
Usage:
|
| 37 |
python llama_cpp_vision.py \
|
| 38 |
+
--gguf Janus-35B-A3B.Q4_K_M.gguf \
|
| 39 |
+
--mmproj Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf \
|
| 40 |
--image /path/to/photo.jpg \
|
| 41 |
--prompt "What is in this image? Be specific."
|
| 42 |
|
| 43 |
# CLI alternative without python binding (ships with llama.cpp):
|
| 44 |
# llama-mtmd-cli \
|
| 45 |
+
# -m Janus-35B-A3B.Q4_K_M.gguf \
|
| 46 |
+
# --mmproj Qwen3.6-35B-A3B-uncensored-heretic-mmproj-BF16.gguf \
|
| 47 |
# --image photo.jpg \
|
| 48 |
# -p "Describe this image."
|
| 49 |
"""
|
|
|
|
| 79 |
|
| 80 |
def main() -> None:
|
| 81 |
ap = argparse.ArgumentParser()
|
| 82 |
+
ap.add_argument("--gguf", required=True, help="Text GGUF (e.g. Janus-35B-A3B.Q4_K_M.gguf).")
|
| 83 |
ap.add_argument("--mmproj", required=True, help="Vision projector GGUF (mmproj-BF16.gguf).")
|
| 84 |
ap.add_argument("--image", required=True, help="Image to analyze.")
|
| 85 |
ap.add_argument("--prompt", default="Describe this image in detail.")
|
|
|
|
| 98 |
sys.exit(f"Image not found: {image_path}")
|
| 99 |
|
| 100 |
# Qwen 2.5 VL chat handler is the closest match shipped with
|
| 101 |
+
# llama-cpp-python; Qwen 3.5 / 3.6 vision uses the same projector
|
| 102 |
# layout. If/when llama-cpp-python ships a Qwen3VLChatHandler, swap it in.
|
| 103 |
handler = Qwen25VLChatHandler(clip_model_path=args.mmproj)
|
| 104 |
|
|
@@ -1,6 +1,6 @@
|
|
| 1 |
#!/usr/bin/env python3
|
| 2 |
"""
|
| 3 |
-
Janus-
|
| 4 |
|
| 5 |
Prerequisites (pick one):
|
| 6 |
|
|
|
|
| 1 |
#!/usr/bin/env python3
|
| 2 |
"""
|
| 3 |
+
Janus-35B — Ollama chat examples.
|
| 4 |
|
| 5 |
Prerequisites (pick one):
|
| 6 |
|
|
@@ -1,22 +1,24 @@
|
|
| 1 |
#!/usr/bin/env python3
|
| 2 |
"""
|
| 3 |
-
Janus-
|
| 4 |
|
| 5 |
-
Loads the Heretic Qwen 3.
|
| 6 |
-
chat turn using its embedded chat template. Janus-
|
| 7 |
around that base, so for the transformers route there is nothing to
|
| 8 |
download from this repo — point at
|
| 9 |
-
llmfan46/Qwen3.
|
| 10 |
and apply the same system prompt the Modelfile uses.
|
| 11 |
|
| 12 |
-
Set MODEL_ID = "Qwen/Qwen3.
|
| 13 |
load the vanilla upstream base instead.
|
| 14 |
|
| 15 |
-
Note: this is a
|
| 16 |
-
|
|
|
|
|
|
|
| 17 |
|
| 18 |
Requirements:
|
| 19 |
-
pip install --upgrade transformers accelerate sentencepiece bitsandbytes # needs a build new enough for Qwen 3.
|
| 20 |
|
| 21 |
Memory:
|
| 22 |
- bf16 full precision: multi-GPU or CPU offload.
|
|
@@ -44,7 +46,7 @@ except ImportError as e: # pragma: no cover
|
|
| 44 |
)
|
| 45 |
|
| 46 |
|
| 47 |
-
MODEL_ID = "llmfan46/Qwen3.
|
| 48 |
|
| 49 |
JANUS_SYSTEM = (
|
| 50 |
"You are Janus, a precise and capable assistant for reasoning, writing, "
|
|
|
|
| 1 |
#!/usr/bin/env python3
|
| 2 |
"""
|
| 3 |
+
Janus-35B — Hugging Face Transformers quickstart.
|
| 4 |
|
| 5 |
+
Loads the Heretic Qwen 3.6 35B-A3B safetensors directly and runs a single
|
| 6 |
+
chat turn using its embedded chat template. Janus-35B is a *wrapper*
|
| 7 |
around that base, so for the transformers route there is nothing to
|
| 8 |
download from this repo — point at
|
| 9 |
+
llmfan46/Qwen3.6-35B-A3B-uncensored-heretic
|
| 10 |
and apply the same system prompt the Modelfile uses.
|
| 11 |
|
| 12 |
+
Set MODEL_ID = "Qwen/Qwen3.6-35B-A3B" to bypass the Heretic abliteration and
|
| 13 |
load the vanilla upstream base instead.
|
| 14 |
|
| 15 |
+
Note: this is a mixture-of-experts model — ~34.7B parameters total, ~3B active
|
| 16 |
+
per token (derived from the bundled GGUF's header, 2026-09-18). Transformers
|
| 17 |
+
keeps all 256 experts resident, so memory tracks the ~34.7B total, not the 3B
|
| 18 |
+
active count.
|
| 19 |
|
| 20 |
Requirements:
|
| 21 |
+
pip install --upgrade transformers accelerate sentencepiece bitsandbytes # needs a build new enough for Qwen 3.6
|
| 22 |
|
| 23 |
Memory:
|
| 24 |
- bf16 full precision: multi-GPU or CPU offload.
|
|
|
|
| 46 |
)
|
| 47 |
|
| 48 |
|
| 49 |
+
MODEL_ID = "llmfan46/Qwen3.6-35B-A3B-uncensored-heretic"
|
| 50 |
|
| 51 |
JANUS_SYSTEM = (
|
| 52 |
"You are Janus, a precise and capable assistant for reasoning, writing, "
|
|
|
|
@@ -1,5 +1,5 @@
|
|
| 1 |
#!/usr/bin/env bash
|
| 2 |
-
# Janus-
|
| 3 |
#
|
| 4 |
# Reads timing from Ollama's /api/chat response metadata (eval_count and
|
| 5 |
# eval_duration are authoritative — no client-side stopwatch noise) and
|
|
|
|
| 1 |
#!/usr/bin/env bash
|
| 2 |
+
# Janus-35B — tok/s benchmark via Ollama.
|
| 3 |
#
|
| 4 |
# Reads timing from Ollama's /api/chat response metadata (eval_count and
|
| 5 |
# eval_duration are authoritative — no client-side stopwatch noise) and
|
|
@@ -1,5 +1,5 @@
|
|
| 1 |
#!/usr/bin/env bash
|
| 2 |
-
# Janus-
|
| 3 |
#
|
| 4 |
# Usage:
|
| 5 |
# ./scripts/build.sh # default: Q4_K_M
|
|
@@ -7,16 +7,15 @@
|
|
| 7 |
# QUANT=Q6_K ./scripts/build.sh
|
| 8 |
#
|
| 9 |
# Skip the download by pointing at a GGUF you already have:
|
| 10 |
-
# GGUF_PATH=/path/to/Qwen3.
|
| 11 |
#
|
| 12 |
# Note: a `<gguf>.stripped.gguf` sibling is created next to the source GGUF (the
|
| 13 |
-
# MTP-stripped copy
|
| 14 |
-
#
|
| 15 |
-
#
|
| 16 |
-
# remove the sibling yourself.
|
| 17 |
#
|
| 18 |
# Requires: huggingface-cli (or hf), ollama, awk, python3 with the `gguf`
|
| 19 |
-
# package (the MTP strip below
|
| 20 |
set -euo pipefail
|
| 21 |
|
| 22 |
# Resolve a Hugging Face CLI. The bare name `hf` is ambiguous: Arch's `hf`
|
|
@@ -38,14 +37,14 @@ detect_hf() {
|
|
| 38 |
|
| 39 |
QUANT="${1:-${QUANT:-Q4_K_M}}"
|
| 40 |
|
| 41 |
-
REPO_ID="${REPO_ID:-llmfan46/Qwen3.
|
| 42 |
-
# Filenames at
|
| 43 |
-
#
|
| 44 |
-
#
|
| 45 |
-
#
|
| 46 |
-
#
|
| 47 |
-
#
|
| 48 |
-
GGUF_NAME="Qwen3.
|
| 49 |
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
| 50 |
# GGUF_PATH defaults to ${ROOT}/${GGUF_NAME}, but can be overridden so users
|
| 51 |
# with cached weights elsewhere don't have to copy or symlink anything.
|
|
@@ -95,34 +94,25 @@ if [[ ! -f "${GGUF_PATH}" ]]; then
|
|
| 95 |
exit 1
|
| 96 |
fi
|
| 97 |
|
| 98 |
-
# ---- 2b. Strip the MTP/NextN layer -------------------------------
|
| 99 |
-
#
|
| 100 |
-
#
|
| 101 |
-
#
|
| 102 |
-
#
|
| 103 |
-
#
|
| 104 |
-
#
|
| 105 |
-
#
|
| 106 |
-
#
|
|
|
|
|
|
|
| 107 |
STRIPPED_PATH="${GGUF_PATH%.gguf}.stripped.gguf"
|
| 108 |
-
|
| 109 |
-
# LM Studio, Jan and KoboldCpp all read the template embedded in the GGUF, and
|
| 110 |
-
# the upstream one raises on reasoning_effort values such as "max" and on
|
| 111 |
-
# conflicting preserve_thinking / preserve_reasoning, which llama-server returns
|
| 112 |
-
# as HTTP 500. Stamping it here means those loaders get this repo's template
|
| 113 |
-
# with no extra flags. Skipped if the file is absent.
|
| 114 |
-
STAMP_ARGS=()
|
| 115 |
-
if [[ -f "${ROOT}/chat_template.jinja" ]]; then
|
| 116 |
-
STAMP_ARGS=(--chat-template "${ROOT}/chat_template.jinja")
|
| 117 |
-
echo "[*] stamping chat_template.jinja into the built GGUF"
|
| 118 |
-
fi
|
| 119 |
-
python3 "${ROOT}/scripts/strip_mtp.py" "${GGUF_PATH}" "${STRIPPED_PATH}" ${STAMP_ARGS[@]+"${STAMP_ARGS[@]}"}
|
| 120 |
GGUF_PATH="${STRIPPED_PATH}"
|
| 121 |
echo "[*] gguf (post-strip): ${GGUF_PATH}"
|
| 122 |
|
| 123 |
# ---- 3. Patch the Modelfile FROM line in a temp copy -------------------------
|
| 124 |
|
| 125 |
-
TMP_MODELFILE="$(mktemp -t
|
| 126 |
trap 'rm -f "${TMP_MODELFILE}"' EXIT
|
| 127 |
awk -v p="${GGUF_PATH}" '
|
| 128 |
/^FROM[[:space:]]/ && !done { print "FROM " p; done=1; next }
|
|
|
|
| 1 |
#!/usr/bin/env bash
|
| 2 |
+
# Janus-35B — fetch a Qwen 3.6 35B-A3B GGUF and build the Ollama model.
|
| 3 |
#
|
| 4 |
# Usage:
|
| 5 |
# ./scripts/build.sh # default: Q4_K_M
|
|
|
|
| 7 |
# QUANT=Q6_K ./scripts/build.sh
|
| 8 |
#
|
| 9 |
# Skip the download by pointing at a GGUF you already have:
|
| 10 |
+
# GGUF_PATH=/path/to/Qwen3.6-35B-A3B-uncensored-heretic-Q4_K_M.gguf ./scripts/build.sh Q4_K_M
|
| 11 |
#
|
| 12 |
# Note: a `<gguf>.stripped.gguf` sibling is created next to the source GGUF (the
|
| 13 |
+
# MTP-stripped copy — or, for these quants, a hardlink no-op, so no extra disk)
|
| 14 |
+
# and becomes ollama's FROM target. It's gitignored in-repo; for an out-of-repo
|
| 15 |
+
# GGUF_PATH, remove the sibling yourself.
|
|
|
|
| 16 |
#
|
| 17 |
# Requires: huggingface-cli (or hf), ollama, awk, python3 with the `gguf`
|
| 18 |
+
# package (the MTP strip below still runs, defensively).
|
| 19 |
set -euo pipefail
|
| 20 |
|
| 21 |
# Resolve a Hugging Face CLI. The bare name `hf` is ambiguous: Arch's `hf`
|
|
|
|
| 37 |
|
| 38 |
QUANT="${1:-${QUANT:-Q4_K_M}}"
|
| 39 |
|
| 40 |
+
REPO_ID="${REPO_ID:-llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF}"
|
| 41 |
+
# Filenames at llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF follow
|
| 42 |
+
# Qwen3.6-35B-A3B-uncensored-heretic-${QUANT}.gguf
|
| 43 |
+
# Published quants (HF API, 2026-09-18): BF16, Q3_K_L, Q3_K_M, Q4_K_M, Q4_K_S,
|
| 44 |
+
# Q5_K_M, Q5_K_S, Q6_K, Q8_0, plus the BF16 mmproj (scripts/fetch_vision.sh).
|
| 45 |
+
# Check the repo's Files tab if that list has moved on; override GGUF_NAME or
|
| 46 |
+
# GGUF_PATH below if the naming differs for your chosen quant.
|
| 47 |
+
GGUF_NAME="Qwen3.6-35B-A3B-uncensored-heretic-${QUANT}.gguf"
|
| 48 |
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
| 49 |
# GGUF_PATH defaults to ${ROOT}/${GGUF_NAME}, but can be overridden so users
|
| 50 |
# with cached weights elsewhere don't have to copy or symlink anything.
|
|
|
|
| 94 |
exit 1
|
| 95 |
fi
|
| 96 |
|
| 97 |
+
# ---- 2b. Strip the MTP/NextN layer if present -------------------------------
|
| 98 |
+
# Defensive, not a required build step. The quants in the source repo are
|
| 99 |
+
# MTP-clean — Q4_K_M read from its GGUF header on 2026-09-18: block_count 40,
|
| 100 |
+
# blk.0…blk.39 only, no nextn_predict_layers key — so strip_mtp.py finds nothing
|
| 101 |
+
# to drop and passes the file through as a hardlink (no extra disk, no
|
| 102 |
+
# re-quantization). The call stays because llmfan46 also publishes a separate
|
| 103 |
+
# *-Native-MTP-Preserved-GGUF variant whose Q4_K_M has block_count 41 with a
|
| 104 |
+
# NextN block at index 40; point GGUF_PATH at one of those and this step is what
|
| 105 |
+
# makes it loadable on stock llama.cpp / Ollama. No --chat-template here: the
|
| 106 |
+
# GGUF carries the base's own embedded template and this repo ships no
|
| 107 |
+
# chat_template.jinja to stamp over it.
|
| 108 |
STRIPPED_PATH="${GGUF_PATH%.gguf}.stripped.gguf"
|
| 109 |
+
python3 "${ROOT}/scripts/strip_mtp.py" "${GGUF_PATH}" "${STRIPPED_PATH}"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 110 |
GGUF_PATH="${STRIPPED_PATH}"
|
| 111 |
echo "[*] gguf (post-strip): ${GGUF_PATH}"
|
| 112 |
|
| 113 |
# ---- 3. Patch the Modelfile FROM line in a temp copy -------------------------
|
| 114 |
|
| 115 |
+
TMP_MODELFILE="$(mktemp -t janus35b-modelfile.XXXXXX)"
|
| 116 |
trap 'rm -f "${TMP_MODELFILE}"' EXIT
|
| 117 |
awk -v p="${GGUF_PATH}" '
|
| 118 |
/^FROM[[:space:]]/ && !done { print "FROM " p; done=1; next }
|
|
@@ -1,11 +1,12 @@
|
|
| 1 |
#!/usr/bin/env bash
|
| 2 |
-
# Janus-
|
| 3 |
#
|
| 4 |
# Ollama's /v1/chat/completions (OpenAI-compatible) silently ignores the
|
| 5 |
-
# `options` field, so it loads at the baked default `num_ctx 262144` —
|
| 6 |
-
#
|
| 7 |
-
#
|
| 8 |
-
#
|
|
|
|
| 9 |
# Reuses this repo's Modelfile verbatim (TEMPLATE / SYSTEM / stop params), only
|
| 10 |
# swapping the FROM line to the bundled blob and the num_ctx parameter.
|
| 11 |
#
|
|
@@ -21,7 +22,7 @@ CTX="${CTX:-4096}"
|
|
| 21 |
NBATCH="${NBATCH:-256}"
|
| 22 |
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
| 23 |
MODELFILE="${ROOT}/Modelfile"
|
| 24 |
-
GGUF="${GGUF:-${ROOT}/Janus-
|
| 25 |
TAG="${TAG:-janus}"
|
| 26 |
|
| 27 |
if ! command -v ollama >/dev/null 2>&1; then echo "[!] ollama not found in PATH" >&2; exit 1; fi
|
|
@@ -43,7 +44,7 @@ echo "[*] tag: ${TAG}"
|
|
| 43 |
echo "[*] gguf: ${GGUF}"
|
| 44 |
echo "[*] num_ctx: ${CTX} num_batch: ${NBATCH}"
|
| 45 |
|
| 46 |
-
TMP="$(mktemp -t
|
| 47 |
trap 'rm -f "${TMP}"' EXIT
|
| 48 |
awk -v g="${GGUF}" -v c="${CTX}" -v b="${NBATCH}" '
|
| 49 |
/^FROM[[:space:]]/ && !done { print "FROM " g; done=1; next }
|
|
|
|
| 1 |
#!/usr/bin/env bash
|
| 2 |
+
# Janus-35B — create a small-context local tag for OpenAI /v1 clients.
|
| 3 |
#
|
| 4 |
# Ollama's /v1/chat/completions (OpenAI-compatible) silently ignores the
|
| 5 |
+
# `options` field, so it loads at the baked default `num_ctx 262144` — the
|
| 6 |
+
# 21.23 GB of weights plus a 262K f16 KV cache, which is too much for a tight
|
| 7 |
+
# host (see the README's Hardware requirements for the figures). This bakes a
|
| 8 |
+
# local tag with num_ctx overridden to a loadable value (and num_batch 256),
|
| 9 |
+
# which any OpenAI /v1 client can point at.
|
| 10 |
# Reuses this repo's Modelfile verbatim (TEMPLATE / SYSTEM / stop params), only
|
| 11 |
# swapping the FROM line to the bundled blob and the num_ctx parameter.
|
| 12 |
#
|
|
|
|
| 22 |
NBATCH="${NBATCH:-256}"
|
| 23 |
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
| 24 |
MODELFILE="${ROOT}/Modelfile"
|
| 25 |
+
GGUF="${GGUF:-${ROOT}/Janus-35B-A3B.Q4_K_M.gguf}"
|
| 26 |
TAG="${TAG:-janus}"
|
| 27 |
|
| 28 |
if ! command -v ollama >/dev/null 2>&1; then echo "[!] ollama not found in PATH" >&2; exit 1; fi
|
|
|
|
| 44 |
echo "[*] gguf: ${GGUF}"
|
| 45 |
echo "[*] num_ctx: ${CTX} num_batch: ${NBATCH}"
|
| 46 |
|
| 47 |
+
TMP="$(mktemp -t janus35b-capctx.XXXXXX)"
|
| 48 |
trap 'rm -f "${TMP}"' EXIT
|
| 49 |
awk -v g="${GGUF}" -v c="${CTX}" -v b="${NBATCH}" '
|
| 50 |
/^FROM[[:space:]]/ && !done { print "FROM " g; done=1; next }
|
|
@@ -1,5 +1,5 @@
|
|
| 1 |
#!/usr/bin/env bash
|
| 2 |
-
# Janus-
|
| 3 |
#
|
| 4 |
# Runs everything that's cheap and catches a real-world bug we've already hit:
|
| 5 |
#
|
|
@@ -7,15 +7,16 @@
|
|
| 7 |
# 2. shellcheck on every *.sh, if installed (catches quoting/SC2086 bugs)
|
| 8 |
# 3. python3 -m pyflakes on every *.py (catches NameError, unused imports)
|
| 9 |
# 4. python3 -m py_compile on every *.py (catches actual syntax errors)
|
| 10 |
-
# 5.
|
| 11 |
-
# -
|
| 12 |
# 6. multi-line-aware scan for the VAR="$(cmd 2>/dev/null | filter)" silent-exit
|
| 13 |
# under set -e + pipefail (the one we shipped in d87bc64 and fixed in 385ed94)
|
| 14 |
# 7. Modelfile <-> template/system/params bridge-file sync
|
| 15 |
-
# 8.
|
| 16 |
-
# 9. Go template keeps Ollama's thinking detection and the replay condition
|
| 17 |
#
|
| 18 |
-
# Exit non-zero on any failure
|
|
|
|
|
|
|
| 19 |
#
|
| 20 |
# Usage:
|
| 21 |
# ./scripts/check.sh # one-shot
|
|
@@ -31,6 +32,8 @@ yellow() { printf "\033[33m%s\033[0m\n" "$*"; }
|
|
| 31 |
blue() { printf "\033[34m%s\033[0m\n" "$*"; }
|
| 32 |
|
| 33 |
FAIL=0
|
|
|
|
|
|
|
| 34 |
|
| 35 |
# ---- collect targets -------------------------------------------------------
|
| 36 |
|
|
@@ -69,6 +72,7 @@ if command -v shellcheck >/dev/null 2>&1; then
|
|
| 69 |
done
|
| 70 |
else
|
| 71 |
yellow "[~] shellcheck not installed (skip). Install with: apt install shellcheck"
|
|
|
|
| 72 |
fi
|
| 73 |
|
| 74 |
# ---- 3. pyflakes -----------------------------------------------------------
|
|
@@ -86,6 +90,7 @@ if (( ${#PY_FILES[@]} )); then
|
|
| 86 |
done
|
| 87 |
else
|
| 88 |
yellow "[~] pyflakes not installed (skip). Install with: pip install pyflakes"
|
|
|
|
| 89 |
fi
|
| 90 |
fi
|
| 91 |
|
|
@@ -105,28 +110,44 @@ if (( ${#PY_FILES[@]} )); then
|
|
| 105 |
find . -type d -name __pycache__ -not -path './.git/*' -exec rm -rf {} + 2>/dev/null || true
|
| 106 |
fi
|
| 107 |
|
| 108 |
-
# ---- 5. footgun:
|
| 109 |
#
|
| 110 |
-
#
|
| 111 |
-
#
|
| 112 |
-
#
|
| 113 |
-
#
|
| 114 |
-
# a
|
|
|
|
|
|
|
| 115 |
#
|
| 116 |
-
#
|
| 117 |
-
#
|
| 118 |
-
#
|
| 119 |
-
#
|
| 120 |
-
|
| 121 |
-
|
| 122 |
-
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
|
| 126 |
-
red " [FAIL] bundle path points at an unstripped MTP quant (run scripts/strip_mtp.py)"
|
| 127 |
-
FAIL=1
|
| 128 |
else
|
| 129 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 130 |
fi
|
| 131 |
|
| 132 |
# ---- 6. footgun: VAR="$(cmd 2>/dev/null | filter)" silent-exit pattern -----
|
|
@@ -198,28 +219,10 @@ if [[ -f "${ROOT}/Modelfile" && -f "${ROOT}/template" \
|
|
| 198 |
fi
|
| 199 |
else
|
| 200 |
yellow "[~] bridge files missing; skipping sync check"
|
|
|
|
| 201 |
fi
|
| 202 |
|
| 203 |
-
# ---- 8.
|
| 204 |
-
#
|
| 205 |
-
# 'chat_template.jinja' is what llama.cpp / llama-server applies under
|
| 206 |
-
# '--jinja --chat-template-file'. It is data, not code, so nothing else in this
|
| 207 |
-
# file looks at it: py_compile and pyflakes skip it, and a one-character edit
|
| 208 |
-
# can silently switch the tool-call format away from the XML form llama-server's
|
| 209 |
-
# parser reads, or bring back a raise llama-server returns as HTTP 500.
|
| 210 |
-
# check_chat_template.py renders it and asserts the properties tool calling,
|
| 211 |
-
# reasoning replay and reasoning_effort depend on.
|
| 212 |
-
|
| 213 |
-
if [[ -f "${ROOT}/chat_template.jinja" ]]; then
|
| 214 |
-
blue "[*] python: chat_template.jinja render checks"
|
| 215 |
-
if ! python3 "${ROOT}/scripts/check_chat_template.py"; then
|
| 216 |
-
FAIL=1
|
| 217 |
-
fi
|
| 218 |
-
else
|
| 219 |
-
yellow "[~] chat_template.jinja missing; skipping render checks"
|
| 220 |
-
fi
|
| 221 |
-
|
| 222 |
-
# ---- 9. Go template: Ollama's thinking detection + replay condition ------
|
| 223 |
#
|
| 224 |
# Ollama picks between the Go 'template' and the GGUF's embedded Jinja template
|
| 225 |
# by comparing their capabilities, and infers the Go template's "thinking" from
|
|
@@ -234,6 +237,7 @@ if [[ -f "${ROOT}/template" ]]; then
|
|
| 234 |
fi
|
| 235 |
else
|
| 236 |
yellow "[~] template missing; skipping Go template check"
|
|
|
|
| 237 |
fi
|
| 238 |
|
| 239 |
# ---- result ----------------------------------------------------------------
|
|
@@ -243,4 +247,8 @@ if (( FAIL )); then
|
|
| 243 |
red "[!] FAIL"
|
| 244 |
exit 1
|
| 245 |
fi
|
| 246 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
#!/usr/bin/env bash
|
| 2 |
+
# Janus-35B — repo-local sanity checks.
|
| 3 |
#
|
| 4 |
# Runs everything that's cheap and catches a real-world bug we've already hit:
|
| 5 |
#
|
|
|
|
| 7 |
# 2. shellcheck on every *.sh, if installed (catches quoting/SC2086 bugs)
|
| 8 |
# 3. python3 -m pyflakes on every *.py (catches NameError, unused imports)
|
| 9 |
# 4. python3 -m py_compile on every *.py (catches actual syntax errors)
|
| 10 |
+
# 5. the Modelfile's FROM target is the GGUF this repo actually ships
|
| 11 |
+
# (the local-build path da34705 broke, with nothing here to catch it)
|
| 12 |
# 6. multi-line-aware scan for the VAR="$(cmd 2>/dev/null | filter)" silent-exit
|
| 13 |
# under set -e + pipefail (the one we shipped in d87bc64 and fixed in 385ed94)
|
| 14 |
# 7. Modelfile <-> template/system/params bridge-file sync
|
| 15 |
+
# 8. Go template keeps Ollama's thinking detection and the replay condition
|
|
|
|
| 16 |
#
|
| 17 |
+
# Exit non-zero on any failure; a check whose tooling or input is missing is
|
| 18 |
+
# reported as skipped and never counted as a pass. Designed to run from
|
| 19 |
+
# .git/hooks/pre-commit.
|
| 20 |
#
|
| 21 |
# Usage:
|
| 22 |
# ./scripts/check.sh # one-shot
|
|
|
|
| 32 |
blue() { printf "\033[34m%s\033[0m\n" "$*"; }
|
| 33 |
|
| 34 |
FAIL=0
|
| 35 |
+
SKIPPED=0
|
| 36 |
+
CHECKS=8 # the numbered checks above; keep in sync with that list
|
| 37 |
|
| 38 |
# ---- collect targets -------------------------------------------------------
|
| 39 |
|
|
|
|
| 72 |
done
|
| 73 |
else
|
| 74 |
yellow "[~] shellcheck not installed (skip). Install with: apt install shellcheck"
|
| 75 |
+
SKIPPED=$((SKIPPED + 1))
|
| 76 |
fi
|
| 77 |
|
| 78 |
# ---- 3. pyflakes -----------------------------------------------------------
|
|
|
|
| 90 |
done
|
| 91 |
else
|
| 92 |
yellow "[~] pyflakes not installed (skip). Install with: pip install pyflakes"
|
| 93 |
+
SKIPPED=$((SKIPPED + 1))
|
| 94 |
fi
|
| 95 |
fi
|
| 96 |
|
|
|
|
| 110 |
find . -type d -name __pycache__ -not -path './.git/*' -exec rm -rf {} + 2>/dev/null || true
|
| 111 |
fi
|
| 112 |
|
| 113 |
+
# ---- 5. footgun: Modelfile FROM target vs the bundled GGUF -----------------
|
| 114 |
#
|
| 115 |
+
# 'ollama create janus -f Modelfile' resolves the FROM line against the repo
|
| 116 |
+
# root, so that filename and the blob this repo actually ships have to be the
|
| 117 |
+
# same file. The 0.2.0 rebase renamed FROM ahead of the blob: for a day the
|
| 118 |
+
# repo tracked one bundle and FROM named another, so the local-build path
|
| 119 |
+
# failed outright from a fresh clone until the blob swap in da34705 caught up
|
| 120 |
+
# — and nothing here noticed (CHANGELOG: "the broken Modelfile FROM path
|
| 121 |
+
# shipped unnoticed"). A rename on either side alone breaks it again.
|
| 122 |
#
|
| 123 |
+
# This replaces the '-MTP-Preserved-Q<quant>.gguf' grep that used to sit here:
|
| 124 |
+
# no Qwen 3.6 quant filename carries that substring, so the guard was green by
|
| 125 |
+
# construction rather than by inspection. What still guards the MTP footgun is
|
| 126 |
+
# scripts/strip_mtp.py itself — build.sh runs every fetched quant through it,
|
| 127 |
+
# and it no-ops on the MTP-clean quant this repo bundles.
|
| 128 |
+
|
| 129 |
+
blue "[*] Modelfile FROM target is the GGUF this repo ships"
|
| 130 |
+
if [[ ! -f "${ROOT}/Modelfile" ]]; then
|
| 131 |
+
yellow "[~] Modelfile missing; skipping FROM target check"
|
| 132 |
+
SKIPPED=$((SKIPPED + 1))
|
|
|
|
|
|
|
| 133 |
else
|
| 134 |
+
mapfile -t BUNDLES < <(find . -maxdepth 1 -type f -name '*.gguf' | sed 's|^\./||' | sort)
|
| 135 |
+
FROM_TARGET="$(awk '/^FROM[[:space:]]/{print $2; exit}' "${ROOT}/Modelfile")"
|
| 136 |
+
if (( ${#BUNDLES[@]} != 1 )); then
|
| 137 |
+
red " [FAIL] expected exactly one bundled *.gguf in the repo root, found ${#BUNDLES[@]}"
|
| 138 |
+
FAIL=1
|
| 139 |
+
elif [[ "${FROM_TARGET#./}" != "${BUNDLES[0]}" ]]; then
|
| 140 |
+
red " [FAIL] Modelfile FROM '${FROM_TARGET}' is not the bundled '${BUNDLES[0]}'"
|
| 141 |
+
red " 'ollama create janus -f Modelfile' cannot resolve it from a fresh clone."
|
| 142 |
+
FAIL=1
|
| 143 |
+
elif ! git -C "${ROOT}" ls-files --error-unmatch "${BUNDLES[0]}" >/dev/null 2>&1; then
|
| 144 |
+
red " [FAIL] bundled '${BUNDLES[0]}' is not tracked by git"
|
| 145 |
+
red " It exists on disk but would be absent from a fresh clone and from the"
|
| 146 |
+
red " published repo. Stage it: git add '${BUNDLES[0]}'"
|
| 147 |
+
FAIL=1
|
| 148 |
+
else
|
| 149 |
+
echo " [ ok ] FROM ${FROM_TARGET} == bundled ${BUNDLES[0]} (tracked)"
|
| 150 |
+
fi
|
| 151 |
fi
|
| 152 |
|
| 153 |
# ---- 6. footgun: VAR="$(cmd 2>/dev/null | filter)" silent-exit pattern -----
|
|
|
|
| 219 |
fi
|
| 220 |
else
|
| 221 |
yellow "[~] bridge files missing; skipping sync check"
|
| 222 |
+
SKIPPED=$((SKIPPED + 1))
|
| 223 |
fi
|
| 224 |
|
| 225 |
+
# ---- 8. Go template: Ollama's thinking detection + replay condition ------
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 226 |
#
|
| 227 |
# Ollama picks between the Go 'template' and the GGUF's embedded Jinja template
|
| 228 |
# by comparing their capabilities, and infers the Go template's "thinking" from
|
|
|
|
| 237 |
fi
|
| 238 |
else
|
| 239 |
yellow "[~] template missing; skipping Go template check"
|
| 240 |
+
SKIPPED=$((SKIPPED + 1))
|
| 241 |
fi
|
| 242 |
|
| 243 |
# ---- result ----------------------------------------------------------------
|
|
|
|
| 247 |
red "[!] FAIL"
|
| 248 |
exit 1
|
| 249 |
fi
|
| 250 |
+
if (( SKIPPED )); then
|
| 251 |
+
yellow "[+] $(( CHECKS - SKIPPED )) of ${CHECKS} checks passed, ${SKIPPED} skipped"
|
| 252 |
+
else
|
| 253 |
+
green "[+] all ${CHECKS} checks passed"
|
| 254 |
+
fi
|
|
@@ -1,6 +1,6 @@
|
|
| 1 |
#!/usr/bin/env python3
|
| 2 |
"""
|
| 3 |
-
Janus-
|
| 4 |
|
| 5 |
The repo ships two parallel Ollama configurations:
|
| 6 |
|
|
|
|
| 1 |
#!/usr/bin/env python3
|
| 2 |
"""
|
| 3 |
+
Janus-35B — verify Modelfile and HF Ollama bridge files stay in sync.
|
| 4 |
|
| 5 |
The repo ships two parallel Ollama configurations:
|
| 6 |
|
|
@@ -1,228 +0,0 @@
|
|
| 1 |
-
#!/usr/bin/env python3
|
| 2 |
-
"""Render ``chat_template.jinja`` and assert the properties tool calling needs.
|
| 3 |
-
|
| 4 |
-
``chat_template.jinja`` is this repo's copy of the base model's embedded chat
|
| 5 |
-
template with two changes: ``preserve_thinking`` and ``preserve_reasoning``
|
| 6 |
-
resolved without raising, and ``reasoning_effort`` normalisation. It is what
|
| 7 |
-
llama.cpp / llama-server applies under ``--jinja --chat-template-file``, and what
|
| 8 |
-
``scripts/build.sh`` stamps into the GGUF it builds.
|
| 9 |
-
|
| 10 |
-
The tool-call wire format defaults to XML, as upstream ships it. llama-server
|
| 11 |
-
parses this template's output with its Qwen3-Coder parser, which accepts only
|
| 12 |
-
the XML form, and converts a client's JSON-string ``function.arguments`` to a
|
| 13 |
-
mapping before rendering. A JSON default would teach the model a form that
|
| 14 |
-
parser cannot read, and that failure is invisible to every other check here: the
|
| 15 |
-
template is data, not code, so ``py_compile`` and ``pyflakes`` never look at it.
|
| 16 |
-
|
| 17 |
-
``tool_call_format='auto'`` stays available for loaders that pass the raw JSON
|
| 18 |
-
string through: the XML form needs individual key/value pairs and a chat template
|
| 19 |
-
has ``tojson`` but no inverse, so it renders the prompt in JSON form instead.
|
| 20 |
-
|
| 21 |
-
Run directly (``python3 scripts/check_chat_template.py``) or via
|
| 22 |
-
``./scripts/check.sh`` / ``make check``. Skips with exit 0 when Jinja2 is absent,
|
| 23 |
-
matching how check.sh treats shellcheck and pyflakes.
|
| 24 |
-
"""
|
| 25 |
-
import os
|
| 26 |
-
import sys
|
| 27 |
-
|
| 28 |
-
ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
| 29 |
-
TEMPLATE_PATH = os.path.join(ROOT, "chat_template.jinja")
|
| 30 |
-
|
| 31 |
-
try:
|
| 32 |
-
import jinja2
|
| 33 |
-
except ImportError:
|
| 34 |
-
print("[~] jinja2 not installed (skip). Install with: pip install jinja2")
|
| 35 |
-
sys.exit(0)
|
| 36 |
-
|
| 37 |
-
if not os.path.exists(TEMPLATE_PATH):
|
| 38 |
-
print(f"[!] missing {TEMPLATE_PATH}")
|
| 39 |
-
sys.exit(1)
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
class TemplateRaise(Exception):
|
| 43 |
-
"""What the template's raise_exception() maps to (minja provides this global)."""
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
env = jinja2.Environment(undefined=jinja2.ChainableUndefined)
|
| 47 |
-
env.globals["raise_exception"] = lambda msg: (_ for _ in ()).throw(TemplateRaise(msg))
|
| 48 |
-
try:
|
| 49 |
-
TPL = env.from_string(open(TEMPLATE_PATH, encoding="utf-8").read())
|
| 50 |
-
except jinja2.TemplateSyntaxError as exc:
|
| 51 |
-
print(f"[!] template does not parse: line {exc.lineno}: {exc.message}")
|
| 52 |
-
sys.exit(1)
|
| 53 |
-
|
| 54 |
-
WEATHER = {"type": "function", "function": {
|
| 55 |
-
"name": "get_weather", "description": "Get current weather for a city",
|
| 56 |
-
"parameters": {"type": "object", "properties": {"city": {"type": "string"}},
|
| 57 |
-
"required": ["city"]}}}
|
| 58 |
-
USER = {"role": "user", "content": "Weather in Tokyo?"}
|
| 59 |
-
# Exactly the shape an OpenAI-compatible client replays on turn 2.
|
| 60 |
-
CALL_STR = {"role": "assistant", "content": None, "tool_calls": [
|
| 61 |
-
{"id": "c1", "type": "function",
|
| 62 |
-
"function": {"name": "get_weather", "arguments": '{"city":"Tokyo"}'}}]}
|
| 63 |
-
CALL_MAP = {"role": "assistant", "content": "", "tool_calls": [
|
| 64 |
-
{"function": {"name": "get_weather", "arguments": {"city": "Tokyo"}}}]}
|
| 65 |
-
RESULT = {"role": "tool", "tool_call_id": "c1", "content": '{"temp_c": 22}'}
|
| 66 |
-
|
| 67 |
-
failures = []
|
| 68 |
-
|
| 69 |
-
|
| 70 |
-
def render(messages, **kwargs):
|
| 71 |
-
kwargs.setdefault("add_generation_prompt", True)
|
| 72 |
-
return TPL.render(messages=messages, **kwargs)
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
def check(name, condition, context=""):
|
| 76 |
-
if condition:
|
| 77 |
-
print(f" [ ok ] {name}")
|
| 78 |
-
else:
|
| 79 |
-
print(f" [FAIL] {name}")
|
| 80 |
-
failures.append((name, context))
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
def attempt(messages, **kwargs):
|
| 84 |
-
"""-> (ok, output_or_error)"""
|
| 85 |
-
try:
|
| 86 |
-
return True, render(messages, **kwargs)
|
| 87 |
-
except Exception as exc: # noqa: BLE001 - any raise is an HTTP 500 under llama-server
|
| 88 |
-
return False, f"{type(exc).__name__}: {exc}"
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
# 1. The default is XML, upstream's and the only form llama-server's parser reads.
|
| 92 |
-
ok, out = attempt([USER, CALL_MAP, RESULT], tools=[WEATHER])
|
| 93 |
-
check("mapping tool-call arguments render with no kwargs", ok, out)
|
| 94 |
-
if ok:
|
| 95 |
-
check("default renders the XML tool-call form",
|
| 96 |
-
"<tool_call>\n<function=get_weather>\n<parameter=city>\nTokyo\n</parameter>\n</function>\n</tool_call>" in out, out)
|
| 97 |
-
check("default system prompt advertises the matching XML form",
|
| 98 |
-
"<function=example_function_name>" in out and '{"name": "example_function_name"' not in out, out)
|
| 99 |
-
ok_x, out_x = attempt([USER, CALL_MAP, RESULT], tools=[WEATHER], tool_call_format="xml")
|
| 100 |
-
check("default is byte-identical to tool_call_format='xml'", ok_x and out_x == out, out_x)
|
| 101 |
-
|
| 102 |
-
ok_pin, _ = attempt([USER, CALL_STR, RESULT], tools=[WEATHER])
|
| 103 |
-
check("default still refuses unconverted string arguments (upstream behaviour)",
|
| 104 |
-
not ok_pin, "expected a raise, got a successful render")
|
| 105 |
-
|
| 106 |
-
# 2. 'auto' (opt-in): a replayed tool call with string arguments renders as JSON.
|
| 107 |
-
ok, out = attempt([USER, CALL_STR, RESULT], tools=[WEATHER], tool_call_format="auto")
|
| 108 |
-
check("auto: string tool-call arguments do not raise", ok, out)
|
| 109 |
-
if ok:
|
| 110 |
-
check("auto: string arguments render as JSON inside <tool_call>",
|
| 111 |
-
'<tool_call>\n{"name": "get_weather", "arguments": {"city":"Tokyo"}}\n</tool_call>' in out, out)
|
| 112 |
-
check("auto: system prompt advertises the matching JSON form",
|
| 113 |
-
'{"name": "example_function_name"' in out, out)
|
| 114 |
-
check("auto: no XML tool-call form alongside the JSON instructions",
|
| 115 |
-
"<function=" not in out, out)
|
| 116 |
-
|
| 117 |
-
# 3. Every format value renders mapping arguments.
|
| 118 |
-
for fmt in ("auto", "json", "xml"):
|
| 119 |
-
msgs = [USER, CALL_MAP, RESULT]
|
| 120 |
-
ok_f, out_f = attempt(msgs, tools=[WEATHER], tool_call_format=fmt)
|
| 121 |
-
check(f"tool_call_format={fmt!r} renders", ok_f, out_f)
|
| 122 |
-
|
| 123 |
-
ok_bad, _ = attempt([USER], tools=[WEATHER], tool_call_format="nonsense")
|
| 124 |
-
check("an unknown tool_call_format is rejected", not ok_bad,
|
| 125 |
-
"expected a raise for an invalid format")
|
| 126 |
-
|
| 127 |
-
# 4. Thinking prefill, both directions. Structure, not exact whitespace: the
|
| 128 |
-
# trailing newlines are upstream's and are reproduced verbatim.
|
| 129 |
-
ok_on, on = attempt([USER])
|
| 130 |
-
tail_on = on.split("<|im_start|>assistant")[-1] if ok_on else ""
|
| 131 |
-
check("thinking on primes an open <think>",
|
| 132 |
-
ok_on and tail_on.lstrip("\n").startswith("<think>") and "</think>" not in tail_on,
|
| 133 |
-
repr(tail_on))
|
| 134 |
-
ok_off, off = attempt([USER], enable_thinking=False)
|
| 135 |
-
tail_off = off.split("<|im_start|>assistant")[-1] if ok_off else ""
|
| 136 |
-
check("thinking off primes a closed, empty <think>",
|
| 137 |
-
ok_off and "<think>" in tail_off and "</think>" in tail_off
|
| 138 |
-
and tail_off.split("<think>")[1].split("</think>")[0].strip() == "",
|
| 139 |
-
repr(tail_off))
|
| 140 |
-
|
| 141 |
-
# 5. Multimodal parts must survive, or the --mmproj path in the README breaks.
|
| 142 |
-
ok_v, vis = attempt([{"role": "user", "content": [
|
| 143 |
-
{"type": "text", "text": "What is this?"},
|
| 144 |
-
{"type": "image_url", "image_url": {"url": "data:image/png;base64,AAA"}}]}])
|
| 145 |
-
check("image parts render vision pads",
|
| 146 |
-
ok_v and "<|vision_start|><|image_pad|><|vision_end|>" in vis, vis)
|
| 147 |
-
check("text parts survive alongside images", ok_v and "What is this?" in vis, vis)
|
| 148 |
-
|
| 149 |
-
# 6. Plain conversation, and multi-turn reasoning handling.
|
| 150 |
-
ok_p, plain = attempt([{"role": "system", "content": "You are helpful."}, USER])
|
| 151 |
-
check("plain conversation renders with the system turn",
|
| 152 |
-
ok_p and "<|im_start|>system" in plain and "You are helpful." in plain, plain)
|
| 153 |
-
|
| 154 |
-
# 7. Earlier turns' reasoning IS replayed by default, as upstream does. llama-server
|
| 155 |
-
# from b10763 passes preserve_reasoning=true and derives preserve_thinking from
|
| 156 |
-
# it, but a request's preserve_thinking kwarg replaces the derived copy, so the
|
| 157 |
-
# two can disagree: that must not raise (upstream does), and preserve_thinking
|
| 158 |
-
# wins. A live tool-call chain keeps its own trace whatever the setting.
|
| 159 |
-
PRIOR = {"role": "assistant", "content": "First answer.", "reasoning_content": "OLDTRACE"}
|
| 160 |
-
FOLLOWUP = {"role": "user", "content": "second question"}
|
| 161 |
-
LIVE = {"role": "assistant", "content": "", "reasoning_content": "LIVETRACE",
|
| 162 |
-
"tool_calls": [{"function": {"name": "get_weather", "arguments": {"city": "Tokyo"}}}]}
|
| 163 |
-
|
| 164 |
-
ok_d, out_d = attempt([USER, PRIOR, FOLLOWUP])
|
| 165 |
-
check("earlier reasoning is replayed by default", ok_d and "OLDTRACE" in out_d, out_d)
|
| 166 |
-
|
| 167 |
-
ok_t, out_t = attempt([USER, PRIOR, FOLLOWUP], preserve_thinking=True)
|
| 168 |
-
check("preserve_thinking=True replays it", ok_t and "OLDTRACE" in out_t, out_t)
|
| 169 |
-
|
| 170 |
-
ok_f, out_f = attempt([USER, PRIOR, FOLLOWUP], preserve_thinking=False)
|
| 171 |
-
check("preserve_thinking=False drops it", ok_f and "OLDTRACE" not in out_f, out_f)
|
| 172 |
-
|
| 173 |
-
ok_r, out_r = attempt([USER, PRIOR, FOLLOWUP], preserve_reasoning=False)
|
| 174 |
-
check("preserve_reasoning=False drops it", ok_r and "OLDTRACE" not in out_r, out_r)
|
| 175 |
-
|
| 176 |
-
ok_c, out_c = attempt([USER, PRIOR, FOLLOWUP], preserve_reasoning=True, preserve_thinking=False)
|
| 177 |
-
check("conflicting values do not raise; preserve_thinking=False wins (llama-server default + request kwarg)",
|
| 178 |
-
ok_c and "OLDTRACE" not in out_c, out_c)
|
| 179 |
-
|
| 180 |
-
ok_c2, out_c2 = attempt([USER, PRIOR, FOLLOWUP], preserve_reasoning=False, preserve_thinking=True)
|
| 181 |
-
check("conflicting values do not raise; preserve_thinking=True wins",
|
| 182 |
-
ok_c2 and "OLDTRACE" in out_c2, out_c2)
|
| 183 |
-
|
| 184 |
-
ok_l, out_l = attempt([USER, LIVE, RESULT], tools=[WEATHER], preserve_thinking=False)
|
| 185 |
-
check("a live tool-call chain keeps its own reasoning",
|
| 186 |
-
ok_l and "LIVETRACE" in out_l, out_l)
|
| 187 |
-
|
| 188 |
-
SYS = {"role": "system", "content": "You are a helpful assistant."}
|
| 189 |
-
|
| 190 |
-
# reasoning_effort must never fail the request. Clients disagree on the
|
| 191 |
-
# vocabulary (OpenAI minimal/low/medium/high, xhigh, "max", mixed case), and a
|
| 192 |
-
# template raise is an HTTP 500 for the whole request under Ollama >= 0.33 and
|
| 193 |
-
# llama-server - reasoning_effort="max" made the model unusable. Every value a
|
| 194 |
-
# real client might send has to render, including an explicit JSON null.
|
| 195 |
-
for eff in ("none", "minimal", "low", "medium", "high", "xhigh", "max",
|
| 196 |
-
"HIGH", " High ", "auto", "default", "", "ultra", 5, True, False):
|
| 197 |
-
ok_e, out_e = attempt([SYS, USER], reasoning_effort=eff)
|
| 198 |
-
check(f"reasoning_effort={eff!r} renders", ok_e, out_e)
|
| 199 |
-
ok_null, out_null = attempt([SYS, USER], reasoning_effort=None)
|
| 200 |
-
check("reasoning_effort=null (explicit JSON null) renders", ok_null, out_null)
|
| 201 |
-
|
| 202 |
-
# Ollama's /v1 layer accepts exactly eight values - minimal, low, medium, high,
|
| 203 |
-
# xhigh, ultra, max, none - and rejects anything else with a 400 before the
|
| 204 |
-
# template runs. Each must land on a sensible tier, not merely avoid raising.
|
| 205 |
-
for eff, want in (("high", "xhigh"), ("xhigh", "xhigh"), ("max", "xhigh"),
|
| 206 |
-
("ultra", "xhigh"), ("low", "low"), ("minimal", "low")):
|
| 207 |
-
ok_t, out_t = attempt([SYS, USER], reasoning_effort=eff)
|
| 208 |
-
check(f"reasoning_effort={eff!r} selects the {want} tier",
|
| 209 |
-
ok_t and f"Reasoning effort is set to {want}" in out_t, out_t)
|
| 210 |
-
ok_med, out_med = attempt([SYS, USER], reasoning_effort="medium")
|
| 211 |
-
check("reasoning_effort='medium' selects the medium tier",
|
| 212 |
-
ok_med and "Reasoning effort is set to" not in out_med, out_med)
|
| 213 |
-
ok_none, out_none = attempt([SYS, USER], reasoning_effort="none")
|
| 214 |
-
tail_none = out_none.split("<|im_start|>assistant")[-1] if ok_none else ""
|
| 215 |
-
check("reasoning_effort='none' switches thinking off",
|
| 216 |
-
ok_none and "</think>" in tail_none
|
| 217 |
-
and tail_none.split("<think>")[1].split("</think>")[0].strip() == "",
|
| 218 |
-
repr(tail_none))
|
| 219 |
-
|
| 220 |
-
print()
|
| 221 |
-
if failures:
|
| 222 |
-
print(f"[!] {len(failures)} chat-template check(s) failed")
|
| 223 |
-
for name, context in failures:
|
| 224 |
-
print(f"--- {name} ---")
|
| 225 |
-
if context:
|
| 226 |
-
print(context[:1500])
|
| 227 |
-
sys.exit(1)
|
| 228 |
-
print("[+] chat_template.jinja checks passed")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
@@ -1,20 +1,20 @@
|
|
| 1 |
#!/usr/bin/env bash
|
| 2 |
-
# Janus-
|
| 3 |
#
|
| 4 |
# Why this is separate from build.sh:
|
| 5 |
# build.sh is for the Ollama text path. The mmproj is only useful for
|
| 6 |
# llama.cpp / llama-cpp-python right now, because Ollama's vendored
|
| 7 |
-
# llama.cpp fork is missing the
|
| 8 |
# it (see README Vision section, ollama/ollama#14575).
|
| 9 |
#
|
| 10 |
# Usage:
|
| 11 |
-
# ./scripts/fetch_vision.sh # default: BF16
|
| 12 |
#
|
| 13 |
-
# The Heretic GGUF repo publishes the projector as BF16
|
| 14 |
-
#
|
| 15 |
-
# Qwen3.
|
| 16 |
-
# REPO_ID=<other-qwen3.
|
| 17 |
-
# (vision tokens are projected the same way across Qwen 3.
|
| 18 |
# finetunes, so a family projector is functionally interchangeable.)
|
| 19 |
#
|
| 20 |
# Requires: huggingface-cli (or hf).
|
|
@@ -38,8 +38,8 @@ detect_hf() {
|
|
| 38 |
}
|
| 39 |
|
| 40 |
PRECISION="${1:-${PRECISION:-BF16}}"
|
| 41 |
-
REPO_ID="${REPO_ID:-llmfan46/Qwen3.
|
| 42 |
-
FILE_NAME="${FILE_NAME:-Qwen3.
|
| 43 |
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
| 44 |
DEST="${MMPROJ_PATH:-${ROOT}/${FILE_NAME}}"
|
| 45 |
|
|
@@ -72,7 +72,7 @@ fi
|
|
| 72 |
echo
|
| 73 |
echo "[+] Done. Use it via:"
|
| 74 |
echo " python ${ROOT}/examples/llama_cpp_vision.py \\"
|
| 75 |
-
echo " --gguf ${ROOT}/Janus-
|
| 76 |
echo " --mmproj ${DEST} \\"
|
| 77 |
echo " --image /path/to/photo.jpg \\"
|
| 78 |
echo " --prompt 'Describe this image.'"
|
|
|
|
| 1 |
#!/usr/bin/env bash
|
| 2 |
+
# Janus-35B — fetch the vision projector (mmproj) for image input.
|
| 3 |
#
|
| 4 |
# Why this is separate from build.sh:
|
| 5 |
# build.sh is for the Ollama text path. The mmproj is only useful for
|
| 6 |
# llama.cpp / llama-cpp-python right now, because Ollama's vendored
|
| 7 |
+
# llama.cpp fork is missing the qwen35moe arch entries needed to attach
|
| 8 |
# it (see README Vision section, ollama/ollama#14575).
|
| 9 |
#
|
| 10 |
# Usage:
|
| 11 |
+
# ./scripts/fetch_vision.sh # default: BF16 (902 MB)
|
| 12 |
#
|
| 13 |
+
# The Heretic GGUF repo publishes the projector as BF16 only (its Files
|
| 14 |
+
# listing, HF API, 2026-09-18); for another precision, point REPO_ID /
|
| 15 |
+
# FILE_NAME at a projector from any other Qwen3.6-35B-A3B GGUF repo:
|
| 16 |
+
# REPO_ID=<other-qwen3.6-35b-a3b-gguf-repo> FILE_NAME=mmproj-F16.gguf ./scripts/fetch_vision.sh
|
| 17 |
+
# (vision tokens are projected the same way across Qwen 3.6 35B-A3B
|
| 18 |
# finetunes, so a family projector is functionally interchangeable.)
|
| 19 |
#
|
| 20 |
# Requires: huggingface-cli (or hf).
|
|
|
|
| 38 |
}
|
| 39 |
|
| 40 |
PRECISION="${1:-${PRECISION:-BF16}}"
|
| 41 |
+
REPO_ID="${REPO_ID:-llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF}"
|
| 42 |
+
FILE_NAME="${FILE_NAME:-Qwen3.6-35B-A3B-uncensored-heretic-mmproj-${PRECISION}.gguf}"
|
| 43 |
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
| 44 |
DEST="${MMPROJ_PATH:-${ROOT}/${FILE_NAME}}"
|
| 45 |
|
|
|
|
| 72 |
echo
|
| 73 |
echo "[+] Done. Use it via:"
|
| 74 |
echo " python ${ROOT}/examples/llama_cpp_vision.py \\"
|
| 75 |
+
echo " --gguf ${ROOT}/Janus-35B-A3B.Q4_K_M.gguf \\"
|
| 76 |
echo " --mmproj ${DEST} \\"
|
| 77 |
echo " --image /path/to/photo.jpg \\"
|
| 78 |
echo " --prompt 'Describe this image.'"
|
|
@@ -1,5 +1,5 @@
|
|
| 1 |
#!/usr/bin/env bash
|
| 2 |
-
# Janus-
|
| 3 |
#
|
| 4 |
# Idempotent. Re-runs are safe.
|
| 5 |
set -euo pipefail
|
|
|
|
| 1 |
#!/usr/bin/env bash
|
| 2 |
+
# Janus-35B — install scripts/check.sh as a git pre-commit hook.
|
| 3 |
#
|
| 4 |
# Idempotent. Re-runs are safe.
|
| 5 |
set -euo pipefail
|
|
@@ -22,9 +22,10 @@
|
|
| 22 |
# WORK_DIR (<repo>/.cache/live-check): each run uses a fresh subdirectory of
|
| 23 |
# it and deletes only that; KEEP=1 keeps the run directory.
|
| 24 |
#
|
| 25 |
-
# Needs free disk on WORK_DIR of about twice the GGUF
|
| 26 |
-
#
|
| 27 |
-
#
|
|
|
|
| 28 |
#
|
| 29 |
# The server runs on CPU only: GPU backends are switched off in its environment,
|
| 30 |
# and the run stops before importing if Ollama still reports a GPU. On a shared
|
|
|
|
| 22 |
# WORK_DIR (<repo>/.cache/live-check): each run uses a fresh subdirectory of
|
| 23 |
# it and deletes only that; KEEP=1 keeps the run directory.
|
| 24 |
#
|
| 25 |
+
# Needs free disk on WORK_DIR of about twice the GGUF - ~42 GB for the bundled
|
| 26 |
+
# 21.23 GB Q4_K_M (Ollama keeps the imported file plus a rewritten copy, and
|
| 27 |
+
# briefly more during the import), RAM to load it, and tens of minutes on CPU for
|
| 28 |
+
# a 35B-A3B. Exits non-zero if any check fails.
|
| 29 |
#
|
| 30 |
# The server runs on CPU only: GPU backends are switched off in its environment,
|
| 31 |
# and the run stops before importing if Ollama still reports a GPU. On a shared
|
|
@@ -1,14 +1,14 @@
|
|
| 1 |
#!/usr/bin/env bash
|
| 2 |
-
# Janus-
|
| 3 |
#
|
| 4 |
-
# The bundled GGUF (Janus-
|
| 5 |
-
#
|
| 6 |
-
#
|
| 7 |
-
#
|
| 8 |
#
|
| 9 |
# 1. Resolve the bundle. If it's an LFS pointer (cloned without
|
| 10 |
-
# `git lfs pull`), download the real ~
|
| 11 |
-
# 2. Sanity-check `general.architecture` is
|
| 12 |
# 3. Run `ollama create <tag> -f <temp Modelfile pointing at the
|
| 13 |
# resolved bundle>`.
|
| 14 |
#
|
|
@@ -18,7 +18,7 @@
|
|
| 18 |
# Usage:
|
| 19 |
# ./scripts/load_bundle.sh # default tag: janus
|
| 20 |
# TAG=janus-bundle ./scripts/load_bundle.sh
|
| 21 |
-
# BUNDLE=/path/to/Janus-
|
| 22 |
#
|
| 23 |
# Requires: ollama, python3 with the `gguf` package, hf (if the bundle
|
| 24 |
# needs to be downloaded).
|
|
@@ -42,7 +42,7 @@ detect_hf() {
|
|
| 42 |
}
|
| 43 |
|
| 44 |
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
| 45 |
-
BUNDLE="${BUNDLE:-${ROOT}/Janus-
|
| 46 |
TAG="${TAG:-janus}"
|
| 47 |
REPO_ID="${REPO_ID:-FoolDev/Janus-35B-HERETIC}"
|
| 48 |
MODELFILE="${ROOT}/Modelfile"
|
|
@@ -116,14 +116,14 @@ PY
|
|
| 116 |
)"
|
| 117 |
echo "[*] bundle arch: ${ARCH}"
|
| 118 |
|
| 119 |
-
if [[ "${ARCH}" != "
|
| 120 |
echo "[!] unexpected arch '${ARCH}' — refusing to load. Edit this script if intentional." >&2
|
| 121 |
exit 1
|
| 122 |
fi
|
| 123 |
|
| 124 |
# ---- 4. Build a Modelfile copy with FROM pointing at the bundle --------------
|
| 125 |
|
| 126 |
-
TMP_MODELFILE="$(mktemp -t
|
| 127 |
trap 'rm -f "${TMP_MODELFILE}"' EXIT
|
| 128 |
awk -v p="${BUNDLE}" '
|
| 129 |
/^FROM[[:space:]]/ && !done { print "FROM " p; done=1; next }
|
|
|
|
| 1 |
#!/usr/bin/env bash
|
| 2 |
+
# Janus-35B — load this repo's bundle into Ollama as a local tag.
|
| 3 |
#
|
| 4 |
+
# The bundled GGUF (Janus-35B-A3B.Q4_K_M.gguf) is qwen35moe-stamped and
|
| 5 |
+
# loads directly on stock llama.cpp / Ollama. This script is the
|
| 6 |
+
# one-shot path from "I just cloned this repo" to "I have a working
|
| 7 |
+
# local Ollama tag":
|
| 8 |
#
|
| 9 |
# 1. Resolve the bundle. If it's an LFS pointer (cloned without
|
| 10 |
+
# `git lfs pull`), download the real ~21 GB blob via `hf download`.
|
| 11 |
+
# 2. Sanity-check `general.architecture` is qwen35moe.
|
| 12 |
# 3. Run `ollama create <tag> -f <temp Modelfile pointing at the
|
| 13 |
# resolved bundle>`.
|
| 14 |
#
|
|
|
|
| 18 |
# Usage:
|
| 19 |
# ./scripts/load_bundle.sh # default tag: janus
|
| 20 |
# TAG=janus-bundle ./scripts/load_bundle.sh
|
| 21 |
+
# BUNDLE=/path/to/Janus-35B-A3B.Q4_K_M.gguf ./scripts/load_bundle.sh
|
| 22 |
#
|
| 23 |
# Requires: ollama, python3 with the `gguf` package, hf (if the bundle
|
| 24 |
# needs to be downloaded).
|
|
|
|
| 42 |
}
|
| 43 |
|
| 44 |
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
| 45 |
+
BUNDLE="${BUNDLE:-${ROOT}/Janus-35B-A3B.Q4_K_M.gguf}"
|
| 46 |
TAG="${TAG:-janus}"
|
| 47 |
REPO_ID="${REPO_ID:-FoolDev/Janus-35B-HERETIC}"
|
| 48 |
MODELFILE="${ROOT}/Modelfile"
|
|
|
|
| 116 |
)"
|
| 117 |
echo "[*] bundle arch: ${ARCH}"
|
| 118 |
|
| 119 |
+
if [[ "${ARCH}" != "qwen35moe" ]]; then
|
| 120 |
echo "[!] unexpected arch '${ARCH}' — refusing to load. Edit this script if intentional." >&2
|
| 121 |
exit 1
|
| 122 |
fi
|
| 123 |
|
| 124 |
# ---- 4. Build a Modelfile copy with FROM pointing at the bundle --------------
|
| 125 |
|
| 126 |
+
TMP_MODELFILE="$(mktemp -t janus35b-loadbundle.XXXXXX)"
|
| 127 |
trap 'rm -f "${TMP_MODELFILE}"' EXIT
|
| 128 |
awk -v p="${BUNDLE}" '
|
| 129 |
/^FROM[[:space:]]/ && !done { print "FROM " p; done=1; next }
|
|
@@ -1,5 +1,5 @@
|
|
| 1 |
#!/usr/bin/env bash
|
| 2 |
-
# Janus-
|
| 3 |
#
|
| 4 |
# Verifies:
|
| 5 |
# 1. The Ollama server is reachable.
|
|
@@ -46,7 +46,7 @@ echo "[+] server reachable"
|
|
| 46 |
# ---- 2. Model present? ------------------------------------------------------
|
| 47 |
# Match case-insensitively: Ollama 0.24 normalizes model names at lookup but
|
| 48 |
# preserves whatever case was first registered on disk (e.g. an earlier
|
| 49 |
-
# session may leave a `Janus-
|
| 50 |
# build was invoked with TAG=janus). The exact tag the user typed is still
|
| 51 |
# valid for `ollama run` — the comparison just needs to be case-folded to
|
| 52 |
# match.
|
|
@@ -54,7 +54,7 @@ if ! curl -fsS "${HOST}/api/tags" | jq -e --arg m "${MODEL}" '.models[] | select
|
|
| 54 |
echo "[!] Model '${MODEL}' not found. Build it first:" >&2
|
| 55 |
echo " ./scripts/build.sh # Q4_K_M" >&2
|
| 56 |
echo " ./scripts/build.sh Q3_K_M # smaller quant" >&2
|
| 57 |
-
echo " ./scripts/load_bundle.sh # load this repo's
|
| 58 |
exit 1
|
| 59 |
fi
|
| 60 |
echo "[+] model present"
|
|
|
|
| 1 |
#!/usr/bin/env bash
|
| 2 |
+
# Janus-35B — smoke test against a running Ollama daemon.
|
| 3 |
#
|
| 4 |
# Verifies:
|
| 5 |
# 1. The Ollama server is reachable.
|
|
|
|
| 46 |
# ---- 2. Model present? ------------------------------------------------------
|
| 47 |
# Match case-insensitively: Ollama 0.24 normalizes model names at lookup but
|
| 48 |
# preserves whatever case was first registered on disk (e.g. an earlier
|
| 49 |
+
# session may leave a `Janus-35B:latest` manifest dir behind even when a
|
| 50 |
# build was invoked with TAG=janus). The exact tag the user typed is still
|
| 51 |
# valid for `ollama run` — the comparison just needs to be case-folded to
|
| 52 |
# match.
|
|
|
|
| 54 |
echo "[!] Model '${MODEL}' not found. Build it first:" >&2
|
| 55 |
echo " ./scripts/build.sh # Q4_K_M" >&2
|
| 56 |
echo " ./scripts/build.sh Q3_K_M # smaller quant" >&2
|
| 57 |
+
echo " ./scripts/load_bundle.sh # load this repo's qwen35moe bundle" >&2
|
| 58 |
exit 1
|
| 59 |
fi
|
| 60 |
echo "[+] model present"
|
|
@@ -1,22 +1,28 @@
|
|
| 1 |
#!/usr/bin/env python3
|
| 2 |
"""
|
| 3 |
-
Strip the MTP / NextN layer (the extra last block) out of a
|
| 4 |
loads on stock llama.cpp / Ollama.
|
| 5 |
|
| 6 |
MTP-preserving GGUF conversions keep the MTP tensors as one extra NextN block,
|
| 7 |
-
so block_count = <transformer layers> + 1 MTP layer — here
|
| 8 |
-
NextN block at index
|
| 9 |
layer and abort with `missing tensor 'blk.<last>.ssm_conv1d.weight'`. This drops
|
| 10 |
-
all `blk.<last>.*` tensors, sets block_count down by one (back to
|
| 11 |
removes nextn_predict_layers.
|
| 12 |
|
| 13 |
-
|
| 14 |
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20 |
|
| 21 |
Every KEPT tensor is copied byte-for-byte (raw quantized data) — no
|
| 22 |
re-quantization, so no quality change to the model itself.
|
|
@@ -26,11 +32,11 @@ Optionally also restamp the embedded chat template in the same pass:
|
|
| 26 |
--chat-template FILE replace tokenizer.chat_template with FILE's contents
|
| 27 |
|
| 28 |
That matters because llama.cpp, LM Studio, Jan and KoboldCpp all read the
|
| 29 |
-
template baked into the GGUF,
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
|
| 35 |
With --chat-template and no MTP block to strip, the file is still rewritten (the
|
| 36 |
usual hardlink passthrough would not carry the new template).
|
|
|
|
| 1 |
#!/usr/bin/env python3
|
| 2 |
"""
|
| 3 |
+
Strip the MTP / NextN layer (the extra last block) out of a qwen35moe GGUF so it
|
| 4 |
loads on stock llama.cpp / Ollama.
|
| 5 |
|
| 6 |
MTP-preserving GGUF conversions keep the MTP tensors as one extra NextN block,
|
| 7 |
+
so block_count = <transformer layers> + 1 MTP layer — here 40 + 1 = 41, with the
|
| 8 |
+
NextN block at index 40. Stock loaders treat that last block as a normal hybrid
|
| 9 |
layer and abort with `missing tensor 'blk.<last>.ssm_conv1d.weight'`. This drops
|
| 10 |
+
all `blk.<last>.*` tensors, sets block_count down by one (back to 40), and
|
| 11 |
removes nextn_predict_layers.
|
| 12 |
|
| 13 |
+
For the blob this repo bundles, that is a no-op passthrough (hardlink, no extra
|
| 14 |
+
disk), not a required build step. Janus-35B-A3B.Q4_K_M.gguf comes from
|
| 15 |
+
llmfan46/Qwen3.6-35B-A3B-uncensored-heretic-GGUF and is already MTP-clean:
|
| 16 |
+
block_count 40, blk.0…blk.39 only, no nextn_predict_layers. The head exists
|
| 17 |
+
upstream (the base's config.json declares mtp_num_hidden_layers 1) and llmfan46
|
| 18 |
+
publishes it as a separate *-Native-MTP-Preserved-GGUF whose Q4_K_M reports
|
| 19 |
+
block_count 41 — that variant is what this script is for, and this repo
|
| 20 |
+
deliberately does not ship it. So the strip is kept purely as a safety net, the
|
| 21 |
+
way the dense sibling FoolDev/Thanatos-27B-HERETIC needs it for its
|
| 22 |
+
MTP-preserved heretic-ara base: it guards against a future llmfan46 re-quant
|
| 23 |
+
shipping the MTP layer. Wanting the MTP head means running the upstream
|
| 24 |
+
safetensors under vLLM / SGLang, or taking the unstripped GGUF straight from
|
| 25 |
+
that variant repo.
|
| 26 |
|
| 27 |
Every KEPT tensor is copied byte-for-byte (raw quantized data) — no
|
| 28 |
re-quantization, so no quality change to the model itself.
|
|
|
|
| 32 |
--chat-template FILE replace tokenizer.chat_template with FILE's contents
|
| 33 |
|
| 34 |
That matters because llama.cpp, LM Studio, Jan and KoboldCpp all read the
|
| 35 |
+
template baked into the GGUF, so replacing it is the only way to change what
|
| 36 |
+
those loaders see. This repo no longer carries a chat_template.jinja — the Qwen
|
| 37 |
+
3.6 base's own embedded template governs the llama.cpp path — so nothing here
|
| 38 |
+
passes the flag today: build.sh calls this script without it, and the bundled
|
| 39 |
+
blob keeps the template it shipped with.
|
| 40 |
|
| 41 |
With --chat-template and no MTP block to strip, the file is still rewritten (the
|
| 42 |
usual hardlink passthrough would not carry the new template).
|
|
@@ -1,9 +1,9 @@
|
|
| 1 |
#!/usr/bin/env python3
|
| 2 |
"""
|
| 3 |
-
Janus-
|
| 4 |
forward-pass structure against the actual GGUF metadata.
|
| 5 |
|
| 6 |
-
Reads the
|
| 7 |
`general.architecture` value), prints each claim alongside the metadata key
|
| 8 |
it derives from, and exits non-zero if any value mismatches the expected
|
| 9 |
claim. Useful as a manual audit after the bundle is re-stamped or after
|
|
@@ -11,19 +11,22 @@ upstream re-conversion.
|
|
| 11 |
|
| 12 |
Usage:
|
| 13 |
python3 scripts/verify_arch.py # default bundle
|
| 14 |
-
python3 scripts/verify_arch.py Janus-
|
| 15 |
python3 scripts/verify_arch.py /path/to/some-other.gguf
|
| 16 |
|
| 17 |
Exit code 0 = all claims verify, 1 = at least one mismatch.
|
| 18 |
|
| 19 |
-
Note: this does NOT verify the
|
| 20 |
-
the GGUF) —
|
| 21 |
-
and llama.cpp's `
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
|
|
|
|
|
|
|
|
|
| 27 |
"""
|
| 28 |
from __future__ import annotations
|
| 29 |
|
|
@@ -33,22 +36,27 @@ from pathlib import Path
|
|
| 33 |
from gguf import GGUFReader
|
| 34 |
|
| 35 |
EXPECTED = {
|
| 36 |
-
"block_count": (
|
| 37 |
"context_length": (262144, "262 144 native context"),
|
| 38 |
-
"embedding_length": (
|
| 39 |
-
"
|
| 40 |
-
"
|
| 41 |
-
"
|
|
|
|
|
|
|
|
|
|
| 42 |
"attention.key_length": (256, "Gated Attention: head_dim 256 (key)"),
|
| 43 |
"attention.value_length": (256, "Gated Attention: head_dim 256 (value)"),
|
| 44 |
"rope.dimension_count": (64, "Partial RoPE: 64 of 256 dims (factor 0.25)"),
|
| 45 |
-
"full_attention_interval": (4, "Hybrid stack: every 4th layer is Gated Attention (
|
|
|
|
| 46 |
"ssm.state_size": (128, "Gated DeltaNet: head_dim 128"),
|
| 47 |
-
"ssm.time_step_rank": (
|
| 48 |
"ssm.group_count": (16, "Gated DeltaNet: 16 QK-heads"),
|
|
|
|
| 49 |
}
|
| 50 |
EXPECTED_VOCAB = 248320
|
| 51 |
-
EXPECTED_ARCHS = {"
|
| 52 |
|
| 53 |
|
| 54 |
def read_scalar(reader: GGUFReader, key: str):
|
|
@@ -69,18 +77,18 @@ def read_arch(reader: GGUFReader) -> str:
|
|
| 69 |
|
| 70 |
def main() -> int:
|
| 71 |
if len(sys.argv) > 2:
|
| 72 |
-
print(f"usage: {sys.argv[0]} [path/to/Janus-
|
| 73 |
return 2
|
| 74 |
root = Path(__file__).resolve().parent.parent
|
| 75 |
default_paths = [
|
| 76 |
-
root / "Janus-
|
| 77 |
]
|
| 78 |
if len(sys.argv) == 2:
|
| 79 |
path = Path(sys.argv[1])
|
| 80 |
else:
|
| 81 |
path = next((p for p in default_paths if p.exists() and p.stat().st_size > 1024), None)
|
| 82 |
if path is None:
|
| 83 |
-
print("[!] no Janus-
|
| 84 |
return 2
|
| 85 |
|
| 86 |
print(f"[*] reading: {path}")
|
|
@@ -114,11 +122,12 @@ def main() -> int:
|
|
| 114 |
mismatches += 1
|
| 115 |
|
| 116 |
# ---- embedded chat template vs the repo's chat_template.jinja -------------
|
| 117 |
-
#
|
| 118 |
-
#
|
| 119 |
-
#
|
| 120 |
-
#
|
| 121 |
-
#
|
|
|
|
| 122 |
repo_template = Path(__file__).resolve().parent.parent / "chat_template.jinja"
|
| 123 |
if repo_template.exists():
|
| 124 |
f = reader.fields.get("tokenizer.chat_template")
|
|
|
|
| 1 |
#!/usr/bin/env python3
|
| 2 |
"""
|
| 3 |
+
Janus-35B — verify the README "Architecture" claims and the MoE
|
| 4 |
forward-pass structure against the actual GGUF metadata.
|
| 5 |
|
| 6 |
+
Reads the qwen35moe-stamped bundle (or any GGUF that declares that
|
| 7 |
`general.architecture` value), prints each claim alongside the metadata key
|
| 8 |
it derives from, and exits non-zero if any value mismatches the expected
|
| 9 |
claim. Useful as a manual audit after the bundle is re-stamped or after
|
|
|
|
| 11 |
|
| 12 |
Usage:
|
| 13 |
python3 scripts/verify_arch.py # default bundle
|
| 14 |
+
python3 scripts/verify_arch.py Janus-35B-A3B.Q4_K_M.gguf
|
| 15 |
python3 scripts/verify_arch.py /path/to/some-other.gguf
|
| 16 |
|
| 17 |
Exit code 0 = all claims verify, 1 = at least one mismatch.
|
| 18 |
|
| 19 |
+
Note: this does NOT verify the ~34.7B-total / ~3B-active parameter counts
|
| 20 |
+
directly (no such KV in the GGUF) — they follow from the layer count, hidden
|
| 21 |
+
size and the expert count/width below and llama.cpp's `qwen35moe` type branch,
|
| 22 |
+
not from a single metadata field.
|
| 23 |
+
|
| 24 |
+
`block_count` is checked against 40, which is this model's natural depth (10
|
| 25 |
+
Gated-Attention + 30 Gated-DeltaNet layers), NOT a post-strip value: the
|
| 26 |
+
bundled quant carries no MTP/NextN tensors, so scripts/strip_mtp.py is a
|
| 27 |
+
defensive no-op for it. llmfan46's separate *-Native-MTP-Preserved-GGUF variant
|
| 28 |
+
— which this repo does not ship — reports 41 here, with a NextN block at index
|
| 29 |
+
40 that stock llama.cpp / Ollama cannot load.
|
| 30 |
"""
|
| 31 |
from __future__ import annotations
|
| 32 |
|
|
|
|
| 36 |
from gguf import GGUFReader
|
| 37 |
|
| 38 |
EXPECTED = {
|
| 39 |
+
"block_count": (40, "40 transformer layers"),
|
| 40 |
"context_length": (262144, "262 144 native context"),
|
| 41 |
+
"embedding_length": (2048, "Hidden size 2048"),
|
| 42 |
+
"expert_count": (256, "MoE: 256 experts"),
|
| 43 |
+
"expert_used_count": (8, "MoE: 8 experts active per token"),
|
| 44 |
+
"expert_feed_forward_length": (512, "MoE: per-expert FFN 512"),
|
| 45 |
+
"expert_shared_feed_forward_length": (512, "MoE: shared-expert FFN 512"),
|
| 46 |
+
"attention.head_count": (16, "Gated Attention: 16 Q-heads"),
|
| 47 |
+
"attention.head_count_kv": (2, "Gated Attention: 2 KV-heads (GQA)"),
|
| 48 |
"attention.key_length": (256, "Gated Attention: head_dim 256 (key)"),
|
| 49 |
"attention.value_length": (256, "Gated Attention: head_dim 256 (value)"),
|
| 50 |
"rope.dimension_count": (64, "Partial RoPE: 64 of 256 dims (factor 0.25)"),
|
| 51 |
+
"full_attention_interval": (4, "Hybrid stack: every 4th layer is Gated Attention (10 cycles)"),
|
| 52 |
+
"ssm.conv_kernel": (4, "Gated DeltaNet: conv kernel 4"),
|
| 53 |
"ssm.state_size": (128, "Gated DeltaNet: head_dim 128"),
|
| 54 |
+
"ssm.time_step_rank": (32, "Gated DeltaNet: 32 V-heads"),
|
| 55 |
"ssm.group_count": (16, "Gated DeltaNet: 16 QK-heads"),
|
| 56 |
+
"ssm.inner_size": (4096, "Gated DeltaNet: inner size 4096 (32 V-heads x 128)"),
|
| 57 |
}
|
| 58 |
EXPECTED_VOCAB = 248320
|
| 59 |
+
EXPECTED_ARCHS = {"qwen35moe"}
|
| 60 |
|
| 61 |
|
| 62 |
def read_scalar(reader: GGUFReader, key: str):
|
|
|
|
| 77 |
|
| 78 |
def main() -> int:
|
| 79 |
if len(sys.argv) > 2:
|
| 80 |
+
print(f"usage: {sys.argv[0]} [path/to/Janus-35B-A3B.Q4_K_M.gguf]", file=sys.stderr)
|
| 81 |
return 2
|
| 82 |
root = Path(__file__).resolve().parent.parent
|
| 83 |
default_paths = [
|
| 84 |
+
root / "Janus-35B-A3B.Q4_K_M.gguf",
|
| 85 |
]
|
| 86 |
if len(sys.argv) == 2:
|
| 87 |
path = Path(sys.argv[1])
|
| 88 |
else:
|
| 89 |
path = next((p for p in default_paths if p.exists() and p.stat().st_size > 1024), None)
|
| 90 |
if path is None:
|
| 91 |
+
print("[!] no Janus-35B GGUF found in repo root; pass a path explicitly", file=sys.stderr)
|
| 92 |
return 2
|
| 93 |
|
| 94 |
print(f"[*] reading: {path}")
|
|
|
|
| 122 |
mismatches += 1
|
| 123 |
|
| 124 |
# ---- embedded chat template vs the repo's chat_template.jinja -------------
|
| 125 |
+
# Since 0.7.0 this repo ships NO chat_template.jinja: the bundled GGUF carries the
|
| 126 |
+
# Qwen 3.6 base's own embedded template, untouched, so there are no two copies to
|
| 127 |
+
# drift and the block below skips. It is kept for the case where a maintainer
|
| 128 |
+
# reintroduces a repo-side template (as 0.2.0-0.6.5 had): back then the two did
|
| 129 |
+
# drift twice, and a stale embedded copy silently reinstates the tool-calling
|
| 130 |
+
# failure for every loader that reads the GGUF instead of an external file.
|
| 131 |
repo_template = Path(__file__).resolve().parent.parent / "chat_template.jinja"
|
| 132 |
if repo_template.exists():
|
| 133 |
f = reader.fields.get("tokenizer.chat_template")
|