Instructions to use FoolDev/Janus-35B-HERETIC with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use FoolDev/Janus-35B-HERETIC with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf FoolDev/Janus-35B-HERETIC:Q4_K_M # Run inference directly in the terminal: llama cli -hf FoolDev/Janus-35B-HERETIC:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf FoolDev/Janus-35B-HERETIC:Q4_K_M # Run inference directly in the terminal: llama cli -hf FoolDev/Janus-35B-HERETIC:Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf FoolDev/Janus-35B-HERETIC:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf FoolDev/Janus-35B-HERETIC:Q4_K_M
Use Docker
docker model run hf.co/FoolDev/Janus-35B-HERETIC:Q4_K_M
- Llama new
- LM Studio
- Jan
- vLLM
How to use FoolDev/Janus-35B-HERETIC with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "FoolDev/Janus-35B-HERETIC" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "FoolDev/Janus-35B-HERETIC", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/FoolDev/Janus-35B-HERETIC:Q4_K_M
- Ollama
How to use FoolDev/Janus-35B-HERETIC with Ollama:
ollama run hf.co/FoolDev/Janus-35B-HERETIC:Q4_K_M
- Unsloth Desktop
- Pi
How to use FoolDev/Janus-35B-HERETIC with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf FoolDev/Janus-35B-HERETIC:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "FoolDev/Janus-35B-HERETIC:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use FoolDev/Janus-35B-HERETIC with Docker Model Runner:
docker model run hf.co/FoolDev/Janus-35B-HERETIC:Q4_K_M
- Lemonade
How to use FoolDev/Janus-35B-HERETIC with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull FoolDev/Janus-35B-HERETIC:Q4_K_M
Run and chat with the model
lemonade run user.Janus-35B-HERETIC-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use FoolDev/Janus-35B-HERETIC with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf FoolDev/Janus-35B-HERETIC:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default FoolDev/Janus-35B-HERETIC:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use FoolDev/Janus-35B-HERETIC with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf FoolDev/Janus-35B-HERETIC:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "FoolDev/Janus-35B-HERETIC:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
docs: LM Studio engine tested, loader claims narrowed, make help
Browse files- LM Studio's CPU runtime (llama.cpp b10519) passed the same 25 llama-server
checks on both published blobs. The LM Studio app itself could not be kept
off the GPU (it loaded the model with a newer CPU runtime that opened the
GPU), so its API layer stays untested; both guarded attempts were stopped
and every LM Studio file restored.
- The card no longer promises tool calls in Jan, KoboldCpp or llama-cpp-python.
- Janus: --chat-template auto removed (llama.cpp has no such mode).
- make help describes what make check runs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015DUJPW1e74y1meH5UePzvU
- CHANGELOG.md +26 -0
- Makefile +1 -1
- README.md +11 -8
|
@@ -8,6 +8,32 @@ track the **tooling and documentation**, not the underlying base model.
|
|
| 8 |
|
| 9 |
## [Unreleased]
|
| 10 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
## [0.3.1] β 2026-09-11
|
| 12 |
|
| 13 |
### Fixed
|
|
|
|
| 8 |
|
| 9 |
## [Unreleased]
|
| 10 |
|
| 11 |
+
### Documentation (LM Studio tested; loader claims narrowed)
|
| 12 |
+
- **LM Studio's CPU runtime passed the same 25 checks as standalone llama-server.**
|
| 13 |
+
Its `llama-server` binary (llama.cpp b10519, the engine LM Studio drives) was run
|
| 14 |
+
directly with `--device none` on both published blobs: one call per first-turn tool
|
| 15 |
+
request (three prompts, with and without `parallel_tool_calls`), a single streamed
|
| 16 |
+
index, the string-argument replay answered, every `reasoning_effort` value accepted,
|
| 17 |
+
the effort tiers rendered, earlier turns' reasoning not replayed and a live tool
|
| 18 |
+
chain's kept.
|
| 19 |
+
- **The LM Studio app itself could not be kept off the GPU, so it stays untested.** In two
|
| 20 |
+
guarded attempts (snapshot of every LM Studio file, CPU runtime selected, Vulkan
|
| 21 |
+
hidden, `--disable-gpu`), LM Studio 0.4.24 loaded the model with a newer CPU runtime
|
| 22 |
+
(avx2 2.34.0) than the one selected, and that runtime's process opened the GPU; the
|
| 23 |
+
test's guard stopped it before any real GPU allocation, and every LM Studio file was
|
| 24 |
+
restored byte for byte. LM Studio's own API layer (per its source it maps
|
| 25 |
+
`reasoning_effort` only to thinking on or off) remains untested.
|
| 26 |
+
- **The card no longer promises tool calls in loaders nobody tested.** It said LM
|
| 27 |
+
Studio, Jan, KoboldCpp and llama-cpp-python get working tool calls with no
|
| 28 |
+
configuration. Now it says llama-server and LM Studio's runtime do (both tested);
|
| 29 |
+
llama-cpp-python renders the tools but, per its source, does not parse the model's
|
| 30 |
+
`<tool_call>` output; Jan and KoboldCpp are untested.
|
| 31 |
+
- **`--chat-template auto` removed from the card.** llama.cpp has no such mode: the
|
| 32 |
+
flag makes it use the literal word "auto" as the template. The embedded template is
|
| 33 |
+
already the default, so no flag is needed.
|
| 34 |
+
- `make help` now describes what `make check` runs (lint, bridge sync, the chat-template
|
| 35 |
+
render checks and the Go-template guard).
|
| 36 |
+
|
| 37 |
## [0.3.1] β 2026-09-11
|
| 38 |
|
| 39 |
### Fixed
|
|
@@ -60,7 +60,7 @@ bench: ## Measure tok/s using Ollama's eval timing (3 prompts).
|
|
| 60 |
mmproj: ## Fetch the vision projector for llama.cpp (Ollama vision is broken upstream).
|
| 61 |
./scripts/fetch_vision.sh $(PRECISION)
|
| 62 |
|
| 63 |
-
check: ## Lint
|
| 64 |
./scripts/check.sh
|
| 65 |
|
| 66 |
live-check: ## Live end-to-end checks in an isolated Ollama (GGUF=path or GGUF_PATH=path; tens of minutes on CPU).
|
|
|
|
| 60 |
mmproj: ## Fetch the vision projector for llama.cpp (Ollama vision is broken upstream).
|
| 61 |
./scripts/fetch_vision.sh $(PRECISION)
|
| 62 |
|
| 63 |
+
check: ## Lint, bridge-file sync, chat-template render checks and the Go-template guard; block the unstripped-MTP footgun.
|
| 64 |
./scripts/check.sh
|
| 65 |
|
| 66 |
live-check: ## Live end-to-end checks in an isolated Ollama (GGUF=path or GGUF_PATH=path; tens of minutes on CPU).
|
|
@@ -145,8 +145,8 @@ GGUF-only release. Pull the Heretic safetensors from [`llmfan46/Qwen3.8-27B-Ultr
|
|
| 145 |
> run through `./scripts/strip_mtp.py`: the NextN block at index 64 is dropped,
|
| 146 |
> `block_count` 65 β 64, `nextn_predict_layers` removed, and every kept tensor
|
| 147 |
> copied byte-for-byte β no re-quantization. `chat_template.jinja` is stamped into
|
| 148 |
-
> its metadata in the same pass, so llama.cpp
|
| 149 |
-
> working tool calls with no extra flags.
|
| 150 |
>
|
| 151 |
> Verified before publishing: all 14 Architecture claims below via
|
| 152 |
> `python3 scripts/verify_arch.py`; loads in Ollama as `qwen35` / 26.9B dense with
|
|
@@ -186,7 +186,7 @@ unstripped quant straight from the source GGUF repo.
|
|
| 186 |
|
| 187 |
### llama.cpp / LM Studio
|
| 188 |
|
| 189 |
-
Drop the GGUF into your loader of choice. The chat template is embedded in the GGUF metadata, so llama.cpp
|
| 190 |
|
| 191 |
**Tool calling works out of the box** β the bundled GGUF carries this repo's `chat_template.jinja` in its metadata, so `--jinja` alone is enough. (Upstream's template defaults to the XML tool-call form, which cannot represent the JSON-string `arguments` an OpenAI-compatible client replays; it raises, and llama-server turns that into a 500 on the second turn. Pass the file explicitly if you are running an unstamped quant:)
|
| 192 |
|
|
@@ -528,7 +528,7 @@ On llama.cpp every value except `none` reaches `chat_template.jinja` as sent;
|
|
| 528 |
llama-server handles `none` itself by switching thinking off. Tested on llama.cpp
|
| 529 |
master `a2878d3`: all eight values, `"HIGH"` and an unknown word each
|
| 530 |
returned HTTP 200, and the rendered prompt carried the tier in the table. The
|
| 531 |
-
template lower-cases and trims the value itself. LM Studio is untested.
|
| 532 |
|
| 533 |
### Which template Ollama runs
|
| 534 |
|
|
@@ -610,9 +610,12 @@ With that flag the same client code should work against either server, as long a
|
|
| 610 |
|
| 611 |
> **The bundled GGUF already carries this template β you do not need the flag.**
|
| 612 |
> `Janus-27B.Q4_K_M.gguf` has `chat_template.jinja` stamped into its
|
| 613 |
-
> metadata, so
|
| 614 |
-
>
|
| 615 |
-
>
|
|
|
|
|
|
|
|
|
|
| 616 |
> too. The `--chat-template-file` flag above is only needed for a GGUF that was
|
| 617 |
> not built or stamped that way β a quant pulled straight from upstream, for
|
| 618 |
> instance. To restamp one you already have:
|
|
@@ -663,7 +666,7 @@ print(resp.choices[0].message.tool_calls)
|
|
| 663 |
|
| 664 |
## Known limitations
|
| 665 |
|
| 666 |
-
- **On the embedded-template path, a single tool call can come back twice.** In one test Ollama 0.33.3, running the embedded template, returned a first-turn tool call twice. A separate raw generation showed the model drafting the call inside `<think>` before emitting it once after `</think>`, so the parser most likely matched such a draft (inferred, not confirmed). It did not reproduce on standalone llama.cpp master (`a2878d3`, CPU): across both models, 12 first-turn calls (three prompts, with and without `parallel_tool_calls`) and two streamed runs each came back exactly once. Nor on b10760, the llama.cpp build Ollama 0.33.3 bundles and the one the duplicate came through: six more first-turn calls and a streamed run on this model each came back once. LM Studio is untested. Ollama's default path β the Go template, see [Which template Ollama runs](#which-template-ollama-runs) β returned one call in a comparable test; whether the prompt or the parser makes the difference was not isolated. If your client executes tools wherever the embedded template renders (llama.cpp, LM Studio, or Ollama with `OLLAMA_GO_TEMPLATE=0`), drop identical consecutive calls.
|
| 667 |
- **No mmproj in this release.** The base Qwen3.8 supports image and video input via a separate `mmproj` file, which is not included here. Text-only inference works out of the box; multimodal inference requires fetching `Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-mmproj-BF16.gguf` (or equivalent) from upstream β run `./scripts/fetch_vision.sh` and see [Vision](#vision) for the full path.
|
| 668 |
- **Quantization-induced quality loss.** Q4_K_M is a strong general-purpose quant but does measurably degrade math and code accuracy compared to BF16. If you need maximum quality, run the upstream safetensors on a host with room for the full BF16 weights.
|
| 669 |
- **No MTP head on the GGUF path.** The base preserves the multi-token-prediction (NextN) block, but stock llama.cpp and Ollama can't load it, so the bundled quant is MTP-stripped (see [Architecture](#architecture)). No speculative decoding here β use the upstream safetensors under vLLM / SGLang, or an unstripped quant from the source GGUF repo, if you need it.
|
|
|
|
| 145 |
> run through `./scripts/strip_mtp.py`: the NextN block at index 64 is dropped,
|
| 146 |
> `block_count` 65 β 64, `nextn_predict_layers` removed, and every kept tensor
|
| 147 |
> copied byte-for-byte β no re-quantization. `chat_template.jinja` is stamped into
|
| 148 |
+
> its metadata in the same pass, so llama.cpp and LM Studio's llama.cpp runtime get
|
| 149 |
+
> working tool calls with no extra flags (both tested; Jan and KoboldCpp are untested).
|
| 150 |
>
|
| 151 |
> Verified before publishing: all 14 Architecture claims below via
|
| 152 |
> `python3 scripts/verify_arch.py`; loads in Ollama as `qwen35` / 26.9B dense with
|
|
|
|
| 186 |
|
| 187 |
### llama.cpp / LM Studio
|
| 188 |
|
| 189 |
+
Drop the GGUF into your loader of choice. The chat template is embedded in the GGUF metadata, and llama.cpp and LM Studio use it by default, so no `--chat-template` flag is needed (llama.cpp has no `--chat-template auto` mode; it would take "auto" as the template itself).
|
| 190 |
|
| 191 |
**Tool calling works out of the box** β the bundled GGUF carries this repo's `chat_template.jinja` in its metadata, so `--jinja` alone is enough. (Upstream's template defaults to the XML tool-call form, which cannot represent the JSON-string `arguments` an OpenAI-compatible client replays; it raises, and llama-server turns that into a 500 on the second turn. Pass the file explicitly if you are running an unstamped quant:)
|
| 192 |
|
|
|
|
| 528 |
llama-server handles `none` itself by switching thinking off. Tested on llama.cpp
|
| 529 |
master `a2878d3`: all eight values, `"HIGH"` and an unknown word each
|
| 530 |
returned HTTP 200, and the rendered prompt carried the tier in the table. The
|
| 531 |
+
template lower-cases and trims the value itself. LM Studio's CPU engine (llama.cpp b10519) passed the same checks; the LM Studio app's own API layer is untested.
|
| 532 |
|
| 533 |
### Which template Ollama runs
|
| 534 |
|
|
|
|
| 610 |
|
| 611 |
> **The bundled GGUF already carries this template β you do not need the flag.**
|
| 612 |
> `Janus-27B.Q4_K_M.gguf` has `chat_template.jinja` stamped into its
|
| 613 |
+
> metadata, so a loader that reads the *embedded* template and parses Qwen's
|
| 614 |
+
> `<tool_call>` output gets working tool calls with no configuration: llama-server
|
| 615 |
+
> (tested on llama.cpp master `a2878d3` and b10760) and LM Studio's CPU runtime
|
| 616 |
+
> (llama.cpp b10519, tested). llama-cpp-python renders the tools but, per its
|
| 617 |
+
> source, does not parse `<tool_call>` output into `tool_calls`. Jan and KoboldCpp
|
| 618 |
+
> are untested. `./scripts/build.sh` (`make build`) stamps it into every quant it builds,
|
| 619 |
> too. The `--chat-template-file` flag above is only needed for a GGUF that was
|
| 620 |
> not built or stamped that way β a quant pulled straight from upstream, for
|
| 621 |
> instance. To restamp one you already have:
|
|
|
|
| 666 |
|
| 667 |
## Known limitations
|
| 668 |
|
| 669 |
+
- **On the embedded-template path, a single tool call can come back twice.** In one test Ollama 0.33.3, running the embedded template, returned a first-turn tool call twice. A separate raw generation showed the model drafting the call inside `<think>` before emitting it once after `</think>`, so the parser most likely matched such a draft (inferred, not confirmed). It did not reproduce on standalone llama.cpp master (`a2878d3`, CPU): across both models, 12 first-turn calls (three prompts, with and without `parallel_tool_calls`) and two streamed runs each came back exactly once. Nor on b10760, the llama.cpp build Ollama 0.33.3 bundles and the one the duplicate came through: six more first-turn calls and a streamed run on this model each came back once. LM Studio's CPU engine (llama.cpp b10519) passed the same checks; the LM Studio app's own API layer is untested. Ollama's default path β the Go template, see [Which template Ollama runs](#which-template-ollama-runs) β returned one call in a comparable test; whether the prompt or the parser makes the difference was not isolated. If your client executes tools wherever the embedded template renders (llama.cpp, LM Studio, or Ollama with `OLLAMA_GO_TEMPLATE=0`), drop identical consecutive calls.
|
| 670 |
- **No mmproj in this release.** The base Qwen3.8 supports image and video input via a separate `mmproj` file, which is not included here. Text-only inference works out of the box; multimodal inference requires fetching `Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-mmproj-BF16.gguf` (or equivalent) from upstream β run `./scripts/fetch_vision.sh` and see [Vision](#vision) for the full path.
|
| 671 |
- **Quantization-induced quality loss.** Q4_K_M is a strong general-purpose quant but does measurably degrade math and code accuracy compared to BF16. If you need maximum quality, run the upstream safetensors on a host with room for the full BF16 weights.
|
| 672 |
- **No MTP head on the GGUF path.** The base preserves the multi-token-prediction (NextN) block, but stock llama.cpp and Ollama can't load it, so the bundled quant is MTP-stripped (see [Architecture](#architecture)). No speculative decoding here β use the upstream safetensors under vLLM / SGLang, or an unstripped quant from the source GGUF repo, if you need it.
|