FoolDev Claude Opus 5 (1M context) commited on
Commit
fa987b5
Β·
1 Parent(s): 752eff3

docs: LM Studio engine tested, loader claims narrowed, make help

Browse files

- LM Studio's CPU runtime (llama.cpp b10519) passed the same 25 llama-server
checks on both published blobs. The LM Studio app itself could not be kept
off the GPU (it loaded the model with a newer CPU runtime that opened the
GPU), so its API layer stays untested; both guarded attempts were stopped
and every LM Studio file restored.
- The card no longer promises tool calls in Jan, KoboldCpp or llama-cpp-python.
- Janus: --chat-template auto removed (llama.cpp has no such mode).
- make help describes what make check runs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015DUJPW1e74y1meH5UePzvU

Files changed (3) hide show
  1. CHANGELOG.md +26 -0
  2. Makefile +1 -1
  3. README.md +11 -8
CHANGELOG.md CHANGED
@@ -8,6 +8,32 @@ track the **tooling and documentation**, not the underlying base model.
8
 
9
  ## [Unreleased]
10
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
11
  ## [0.3.1] β€” 2026-09-11
12
 
13
  ### Fixed
 
8
 
9
  ## [Unreleased]
10
 
11
+ ### Documentation (LM Studio tested; loader claims narrowed)
12
+ - **LM Studio's CPU runtime passed the same 25 checks as standalone llama-server.**
13
+ Its `llama-server` binary (llama.cpp b10519, the engine LM Studio drives) was run
14
+ directly with `--device none` on both published blobs: one call per first-turn tool
15
+ request (three prompts, with and without `parallel_tool_calls`), a single streamed
16
+ index, the string-argument replay answered, every `reasoning_effort` value accepted,
17
+ the effort tiers rendered, earlier turns' reasoning not replayed and a live tool
18
+ chain's kept.
19
+ - **The LM Studio app itself could not be kept off the GPU, so it stays untested.** In two
20
+ guarded attempts (snapshot of every LM Studio file, CPU runtime selected, Vulkan
21
+ hidden, `--disable-gpu`), LM Studio 0.4.24 loaded the model with a newer CPU runtime
22
+ (avx2 2.34.0) than the one selected, and that runtime's process opened the GPU; the
23
+ test's guard stopped it before any real GPU allocation, and every LM Studio file was
24
+ restored byte for byte. LM Studio's own API layer (per its source it maps
25
+ `reasoning_effort` only to thinking on or off) remains untested.
26
+ - **The card no longer promises tool calls in loaders nobody tested.** It said LM
27
+ Studio, Jan, KoboldCpp and llama-cpp-python get working tool calls with no
28
+ configuration. Now it says llama-server and LM Studio's runtime do (both tested);
29
+ llama-cpp-python renders the tools but, per its source, does not parse the model's
30
+ `<tool_call>` output; Jan and KoboldCpp are untested.
31
+ - **`--chat-template auto` removed from the card.** llama.cpp has no such mode: the
32
+ flag makes it use the literal word "auto" as the template. The embedded template is
33
+ already the default, so no flag is needed.
34
+ - `make help` now describes what `make check` runs (lint, bridge sync, the chat-template
35
+ render checks and the Go-template guard).
36
+
37
  ## [0.3.1] β€” 2026-09-11
38
 
39
  ### Fixed
Makefile CHANGED
@@ -60,7 +60,7 @@ bench: ## Measure tok/s using Ollama's eval timing (3 prompts).
60
  mmproj: ## Fetch the vision projector for llama.cpp (Ollama vision is broken upstream).
61
  ./scripts/fetch_vision.sh $(PRECISION)
62
 
63
- check: ## Lint shell + python files; block the unstripped-MTP footgun.
64
  ./scripts/check.sh
65
 
66
  live-check: ## Live end-to-end checks in an isolated Ollama (GGUF=path or GGUF_PATH=path; tens of minutes on CPU).
 
60
  mmproj: ## Fetch the vision projector for llama.cpp (Ollama vision is broken upstream).
61
  ./scripts/fetch_vision.sh $(PRECISION)
62
 
63
+ check: ## Lint, bridge-file sync, chat-template render checks and the Go-template guard; block the unstripped-MTP footgun.
64
  ./scripts/check.sh
65
 
66
  live-check: ## Live end-to-end checks in an isolated Ollama (GGUF=path or GGUF_PATH=path; tens of minutes on CPU).
README.md CHANGED
@@ -145,8 +145,8 @@ GGUF-only release. Pull the Heretic safetensors from [`llmfan46/Qwen3.8-27B-Ultr
145
  > run through `./scripts/strip_mtp.py`: the NextN block at index 64 is dropped,
146
  > `block_count` 65 β†’ 64, `nextn_predict_layers` removed, and every kept tensor
147
  > copied byte-for-byte β€” no re-quantization. `chat_template.jinja` is stamped into
148
- > its metadata in the same pass, so llama.cpp, LM Studio, Jan and KoboldCpp get
149
- > working tool calls with no extra flags.
150
  >
151
  > Verified before publishing: all 14 Architecture claims below via
152
  > `python3 scripts/verify_arch.py`; loads in Ollama as `qwen35` / 26.9B dense with
@@ -186,7 +186,7 @@ unstripped quant straight from the source GGUF repo.
186
 
187
  ### llama.cpp / LM Studio
188
 
189
- Drop the GGUF into your loader of choice. The chat template is embedded in the GGUF metadata, so llama.cpp's `--chat-template auto` and LM Studio's GGUF auto-detection handle plain conversation correctly.
190
 
191
  **Tool calling works out of the box** β€” the bundled GGUF carries this repo's `chat_template.jinja` in its metadata, so `--jinja` alone is enough. (Upstream's template defaults to the XML tool-call form, which cannot represent the JSON-string `arguments` an OpenAI-compatible client replays; it raises, and llama-server turns that into a 500 on the second turn. Pass the file explicitly if you are running an unstamped quant:)
192
 
@@ -528,7 +528,7 @@ On llama.cpp every value except `none` reaches `chat_template.jinja` as sent;
528
  llama-server handles `none` itself by switching thinking off. Tested on llama.cpp
529
  master `a2878d3`: all eight values, `"HIGH"` and an unknown word each
530
  returned HTTP 200, and the rendered prompt carried the tier in the table. The
531
- template lower-cases and trims the value itself. LM Studio is untested.
532
 
533
  ### Which template Ollama runs
534
 
@@ -610,9 +610,12 @@ With that flag the same client code should work against either server, as long a
610
 
611
  > **The bundled GGUF already carries this template β€” you do not need the flag.**
612
  > `Janus-27B.Q4_K_M.gguf` has `chat_template.jinja` stamped into its
613
- > metadata, so LM Studio, Jan, KoboldCpp, llama-cpp-python and anything else that
614
- > reads the *embedded* template get working tool calls with no configuration at
615
- > all. `./scripts/build.sh` (`make build`) stamps it into every quant it builds,
 
 
 
616
  > too. The `--chat-template-file` flag above is only needed for a GGUF that was
617
  > not built or stamped that way β€” a quant pulled straight from upstream, for
618
  > instance. To restamp one you already have:
@@ -663,7 +666,7 @@ print(resp.choices[0].message.tool_calls)
663
 
664
  ## Known limitations
665
 
666
- - **On the embedded-template path, a single tool call can come back twice.** In one test Ollama 0.33.3, running the embedded template, returned a first-turn tool call twice. A separate raw generation showed the model drafting the call inside `<think>` before emitting it once after `</think>`, so the parser most likely matched such a draft (inferred, not confirmed). It did not reproduce on standalone llama.cpp master (`a2878d3`, CPU): across both models, 12 first-turn calls (three prompts, with and without `parallel_tool_calls`) and two streamed runs each came back exactly once. Nor on b10760, the llama.cpp build Ollama 0.33.3 bundles and the one the duplicate came through: six more first-turn calls and a streamed run on this model each came back once. LM Studio is untested. Ollama's default path β€” the Go template, see [Which template Ollama runs](#which-template-ollama-runs) β€” returned one call in a comparable test; whether the prompt or the parser makes the difference was not isolated. If your client executes tools wherever the embedded template renders (llama.cpp, LM Studio, or Ollama with `OLLAMA_GO_TEMPLATE=0`), drop identical consecutive calls.
667
  - **No mmproj in this release.** The base Qwen3.8 supports image and video input via a separate `mmproj` file, which is not included here. Text-only inference works out of the box; multimodal inference requires fetching `Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-mmproj-BF16.gguf` (or equivalent) from upstream β€” run `./scripts/fetch_vision.sh` and see [Vision](#vision) for the full path.
668
  - **Quantization-induced quality loss.** Q4_K_M is a strong general-purpose quant but does measurably degrade math and code accuracy compared to BF16. If you need maximum quality, run the upstream safetensors on a host with room for the full BF16 weights.
669
  - **No MTP head on the GGUF path.** The base preserves the multi-token-prediction (NextN) block, but stock llama.cpp and Ollama can't load it, so the bundled quant is MTP-stripped (see [Architecture](#architecture)). No speculative decoding here β€” use the upstream safetensors under vLLM / SGLang, or an unstripped quant from the source GGUF repo, if you need it.
 
145
  > run through `./scripts/strip_mtp.py`: the NextN block at index 64 is dropped,
146
  > `block_count` 65 β†’ 64, `nextn_predict_layers` removed, and every kept tensor
147
  > copied byte-for-byte β€” no re-quantization. `chat_template.jinja` is stamped into
148
+ > its metadata in the same pass, so llama.cpp and LM Studio's llama.cpp runtime get
149
+ > working tool calls with no extra flags (both tested; Jan and KoboldCpp are untested).
150
  >
151
  > Verified before publishing: all 14 Architecture claims below via
152
  > `python3 scripts/verify_arch.py`; loads in Ollama as `qwen35` / 26.9B dense with
 
186
 
187
  ### llama.cpp / LM Studio
188
 
189
+ Drop the GGUF into your loader of choice. The chat template is embedded in the GGUF metadata, and llama.cpp and LM Studio use it by default, so no `--chat-template` flag is needed (llama.cpp has no `--chat-template auto` mode; it would take "auto" as the template itself).
190
 
191
  **Tool calling works out of the box** β€” the bundled GGUF carries this repo's `chat_template.jinja` in its metadata, so `--jinja` alone is enough. (Upstream's template defaults to the XML tool-call form, which cannot represent the JSON-string `arguments` an OpenAI-compatible client replays; it raises, and llama-server turns that into a 500 on the second turn. Pass the file explicitly if you are running an unstamped quant:)
192
 
 
528
  llama-server handles `none` itself by switching thinking off. Tested on llama.cpp
529
  master `a2878d3`: all eight values, `"HIGH"` and an unknown word each
530
  returned HTTP 200, and the rendered prompt carried the tier in the table. The
531
+ template lower-cases and trims the value itself. LM Studio's CPU engine (llama.cpp b10519) passed the same checks; the LM Studio app's own API layer is untested.
532
 
533
  ### Which template Ollama runs
534
 
 
610
 
611
  > **The bundled GGUF already carries this template β€” you do not need the flag.**
612
  > `Janus-27B.Q4_K_M.gguf` has `chat_template.jinja` stamped into its
613
+ > metadata, so a loader that reads the *embedded* template and parses Qwen's
614
+ > `<tool_call>` output gets working tool calls with no configuration: llama-server
615
+ > (tested on llama.cpp master `a2878d3` and b10760) and LM Studio's CPU runtime
616
+ > (llama.cpp b10519, tested). llama-cpp-python renders the tools but, per its
617
+ > source, does not parse `<tool_call>` output into `tool_calls`. Jan and KoboldCpp
618
+ > are untested. `./scripts/build.sh` (`make build`) stamps it into every quant it builds,
619
  > too. The `--chat-template-file` flag above is only needed for a GGUF that was
620
  > not built or stamped that way β€” a quant pulled straight from upstream, for
621
  > instance. To restamp one you already have:
 
666
 
667
  ## Known limitations
668
 
669
+ - **On the embedded-template path, a single tool call can come back twice.** In one test Ollama 0.33.3, running the embedded template, returned a first-turn tool call twice. A separate raw generation showed the model drafting the call inside `<think>` before emitting it once after `</think>`, so the parser most likely matched such a draft (inferred, not confirmed). It did not reproduce on standalone llama.cpp master (`a2878d3`, CPU): across both models, 12 first-turn calls (three prompts, with and without `parallel_tool_calls`) and two streamed runs each came back exactly once. Nor on b10760, the llama.cpp build Ollama 0.33.3 bundles and the one the duplicate came through: six more first-turn calls and a streamed run on this model each came back once. LM Studio's CPU engine (llama.cpp b10519) passed the same checks; the LM Studio app's own API layer is untested. Ollama's default path β€” the Go template, see [Which template Ollama runs](#which-template-ollama-runs) β€” returned one call in a comparable test; whether the prompt or the parser makes the difference was not isolated. If your client executes tools wherever the embedded template renders (llama.cpp, LM Studio, or Ollama with `OLLAMA_GO_TEMPLATE=0`), drop identical consecutive calls.
670
  - **No mmproj in this release.** The base Qwen3.8 supports image and video input via a separate `mmproj` file, which is not included here. Text-only inference works out of the box; multimodal inference requires fetching `Qwen3.8-27B-Ultra-Uncensored-Heretic-Native-MTP-Preserved-mmproj-BF16.gguf` (or equivalent) from upstream β€” run `./scripts/fetch_vision.sh` and see [Vision](#vision) for the full path.
671
  - **Quantization-induced quality loss.** Q4_K_M is a strong general-purpose quant but does measurably degrade math and code accuracy compared to BF16. If you need maximum quality, run the upstream safetensors on a host with room for the full BF16 weights.
672
  - **No MTP head on the GGUF path.** The base preserves the multi-token-prediction (NextN) block, but stock llama.cpp and Ollama can't load it, so the bundled quant is MTP-stripped (see [Architecture](#architecture)). No speculative decoding here β€” use the upstream safetensors under vLLM / SGLang, or an unstripped quant from the source GGUF repo, if you need it.