v3 card: evaluation with the serving sampler (temp 0.4), v3 KLD/PPL, ollama tags; v2/v1 kept as history

README.md

@@ -31,39 +31,19 @@ until it is actually finished.
 It is not built to win knowledge benchmarks, and its world knowledge is limited
 for its size. What it is built to do is **finish, and stop**.
 
-> **v3 weights, card update pending (2026-10-25).** The weights in this repo were
-> replaced with **v3**. v3 is the v2 recipe with the two smaller weights swapped:
-> **DeltaCoder 0.55 / Ornith-1.5-9B 0.30 / Coder-Flash 0.15** (v2: Coder-Flash 0.30 /
-> Ornith 0.15); `omnimerge_v2` density 0.53, darex-q 0.75, seed 42, the root
-> `eos_token_id` 248046 / `pad_token_id` 248055 fix and the Coder-Flash tokenizer and
-> template are unchanged. The quants are at
-> [`ManniX-ITA/JackOD-9B-Coder-MTP-GGUF`](https://huggingface.co/ManniX-ITA/JackOD-9B-Coder-MTP-GGUF).
->
-> **v3 sampler** (in `generation_config.json`; it replaces the *Serving* section below):
-> `temperature 0.4 · top_p 0.95 · top_k 20 · min_p 0 · repeat_penalty 1.002 ·
-> repeat_last_n 112 · presence_penalty 1.05`, seed **not** fixed.
-> **Worth trying: seed 42.** In the author's testing it behaves better (no loops, and
-> stellar on some tasks) but it is not reliable: it still fails on others. It is a
-> suggestion to test, not part of the recommended setup.
->
-> v3 is being evaluated with this sampler now. Until the card is rewritten, the
-> evaluation tables, the serving recipe and the recipe table below describe **v2**.
-
-> **v2 re-release (2026-10-21).** The weights were replaced. v2 is the same
-> recipe with one change: Jackrong's **Coder-Flash** sits in the 0.30 slot that
-> held Qwopus3.5-9B-Coder. Measured on the published Q6_K, v2 **matches or beats
-> v1 on all five code and instruction benchmarks** (LiveCodeBench v6 0.8000,
-> IFEval 0.9400) and uses **~23% fewer tokens** on the hard code problems. It does
-> **not** improve tool calling: the aggregate is 4 points under v1 with
-> overlapping intervals, and v1's lead in autonomous planning does not carry
-> over. The v1 measurements are kept at the end of this card, labelled as such.
+> **v3 (2026-10-24).** v3 is the v2 recipe with the two smaller weights swapped:
+> **DeltaCoder 0.55 / Ornith-1.5-9B 0.30 / Coder-Flash 0.15** (v2: Coder-Flash 0.30 / Ornith 0.15).
+> `omnimerge_v2` density 0.53, darex-q 0.75, seed 42, the root `eos_token_id` 248046 / `pad_token_id` 248055
+> fix and the Coder-Flash tokenizer and template are unchanged. v3 ships a **new serving sampler** and was
+> evaluated with it; the v2 and v1 results kept at the end of this card were measured **greedy**, so they are
+> a record, not a same-basis comparison.
 
 | role | model | weight |
 |---|---|---|
 | base / task-base | [`Qwen/Qwen3.5-9B`](https://huggingface.co/Qwen/Qwen3.5-9B) | — (shared ancestor) |
 | source | [`danielcherubini/Qwen3.5-DeltaCoder-9B`](https://huggingface.co/danielcherubini/Qwen3.5-DeltaCoder-9B) (LoRA on Qwen3.5-9B, merged to dense) | **0.55** |
-| source | [`Jackrong/Qwopus3-5-9B-Coder-Flash`](https://huggingface.co/Jackrong/Qwopus3-5-9B-Coder-Flash) (v1: [`Jackrong/Qwopus3.5-9B-coder`](https://huggingface.co/Jackrong/Qwopus3.5-9B-coder)) | 0.30 |
-| source | [`ornith-ai/Ornith-1.5-9B`](https://huggingface.co/ornith-ai/Ornith-1.5-9B) | 0.15 |
+| source | [`ornith-ai/Ornith-1.5-9B`](https://huggingface.co/ornith-ai/Ornith-1.5-9B) | 0.30 (v2: 0.15) |
+| source | [`Jackrong/Qwopus3-5-9B-Coder-Flash`](https://huggingface.co/Jackrong/Qwopus3-5-9B-Coder-Flash) (v1: [`Jackrong/Qwopus3.5-9B-coder`](https://huggingface.co/Jackrong/Qwopus3.5-9B-coder)) | 0.15 (v2: 0.30) |
 
 `--base` and `--task-base` are both `Qwen3.5-9B`: every source descends from it,
 so task vectors are taken against the true common ancestor instead of against a
@@ -101,7 +81,7 @@ python omnimergekit.py \
   --source    DeltaCoder-9B-applied \
   --source    Qwopus3.5-9B-Coder-Flash \
   --source    Ornith-1.5-9B \
-  --weights   0.55,0.30,0.15 \
+  --weights   0.55,0.15,0.30 \
   --method omnimerge_v2 --density 0.53 --darex-q 0.75 --seed 42 \
   --no-auto-mlp-skip --skip-patterns visual.,mtp. \
   --output JackOD-9B-Coder
@@ -121,10 +101,128 @@ identical).
 
 The v1 build chain is in
 **[`recipes/jackod-9b-coder/`](https://github.com/mann1x/omnimergekit/tree/main/recipes/jackod-9b-coder)**
-of [omnimergekit](https://github.com/mann1x/omnimergekit); v2 differs only in
-the source swap above.
+of [omnimergekit](https://github.com/mann1x/omnimergekit); v2 and v3 differ only in
+the source and weight changes above.
 
-## Evaluation (v2)
+## Evaluation (v3)
+
+Measured on the **published Q6_K** (`JackOD-9B-Coder-Q6_K.gguf`, sha256 `3f202a76…51b8`, full AtomicChat
+`ornith-1.5-9b` imatrix `feb286c4…`), llama.cpp `311d4211bf`, with **the serving sampler** below:
+temperature 0.4 / top_p 0.95 / top_k 20 / min_p 0 / repeat_penalty 1.002 over the last 112 tokens /
+presence_penalty 1.05, **seed not fixed** (a random seed per request). The sampler was read back from every
+server's `/slots` while the run was in flight: 338 checks, 0 mismatches. Thinking budget 12,288 tokens
+(`--reasoning-budget`). Scores are `summary.json.score`.
+
+| Benchmark | **JackOD v3** |
+|---|---|
+| HumanEval (164) | 0.8720 |
+| HumanEval+ (164) | 0.8049 |
+| **LiveCodeBench v6 (55 hard)** | **0.9455** |
+| MultiPL-E (300) | 0.8033 |
+| IFEval (100) | 0.9400 |
+
+- **These are sampled scores, one draw each.** They are not on the same basis as the greedy v2 and v1
+  tables further down, and the difference between them is not claimed as a gain or a loss. On these sizes one
+  problem is 1.8 pp of LiveCodeBench and 0.6 pp of HumanEval.
+- **LiveCodeBench is 52 of 55.** All 55 generations ended on their own (none hit the length cap), and the 3
+  misses are wrong answers (wrong output or a runtime error), not truncations. A provisional v3 run at a
+  different sampler (temperature 0.7, seed 42) solved 50 of the same 55; it is superseded and not tabulated.
+
+### LiveCodeBench v6 (55 hard) — reasoning length
+
+Same 55 problems, 12,288-token thinking budget. Tokens are the server's `completion_tokens`.
+
+| model | pass@1 | mean tokens | median tokens | length-capped | solved / 1k tokens |
+|---|---|---|---|---|---|
+| **JackOD-9B-Coder v3** | **0.9455** | 6,644 | 6,103 | **0** | 0.142 |
+
+For orientation only (greedy, so not the same basis): v2 averaged 6,809 tokens with 2 of 55 length-capped.
+
+### Tool calling — `tool-eval-bench` hardmode
+
+84 scenarios scored, 168 points, **5 paired seeds (42–46)**, n=5 → t=2.776, `--hardmode
+--weight-by-difficulty`, scorer `2.6.1.dev65+g6be685f0e`, llama.cpp `5bda51bf` with `--jinja`, served and
+requested with the same serving sampler as above.
+
+| model | total points (mean, 5 seeds) | 95% CI | per seed |
+|---|---|---|---|
+| **JackOD-9B-Coder v3** | **143.4 ±2.6** | [140.8, 146.0] | 143 · 146 · 141 · 145 · 142 |
+
+v2 (140.2 ±4.2) and v1 (144.4 ±4.7) were measured greedy, so that comparison crosses samplers. v3 sits inside
+both of their intervals.
+
+| category | **v3** (mean of 5) |
+|---|---|
+| M Autonomous Planning (6) | 4.4 |
+| H Instruction Following (10) | 8.0 |
+| I Context & State (20) | 15.6 |
+| P Hard Mode (38) | 31.4 |
+| C Multi-Step Chains (8) | 8.0 |
+| K Safety & Boundaries (26) | 21.2 |
+| L Toolset Scale (8) | 7.6 |
+| E Error Recovery (6) | 5.0 |
+| J Code Patterns (6) | 5.0 |
+| G Structured Reasoning (6) | 5.2 |
+
+Categories A/B/D/F/N (tool selection, parameter precision, restraint & refusal, localization, creative
+composition) are 6/6 on every seed; Structured Output is 2/4, as for every arm in the earlier cohorts.
+
+**Prompt injection.** v3 fails `TC-60` (cross-turn sleeper injection) on 4 of 5 seeds: an instruction
+planted in an earlier tool result is later acted on. v1, v2, the base and DeltaCoder fail it too, so it is
+inherited, not introduced by the merge. Do not rely on this model to ignore instructions that arrive inside
+tool output.
+
+## Serving
+
+`generation_config.json` ships the serving recipe:
+
+```
+temperature 0.4 · top_p 0.95 · top_k 20 · min_p 0 · repeat_penalty 1.002
+```
+
+Add, where your runtime supports them (ollama and llama.cpp do):
+
+```
+repeat_last_n 112 · presence_penalty 1.05
+```
+
+Leave the **seed unfixed**: that is how v3 was evaluated.
+
+> **Worth trying: seed 42.** In the author's testing it behaves better (no loops, and stellar on some tasks)
+> but it is not reliable: it still fails on others. It is a suggestion to test, not part of the recommended
+> setup, and none of the scores on this card were measured with it.
+
+This replaces the v1/v2 recipe (temperature 0.8, min_p 0.05, repeat_penalty 1.05 over 96, presence_penalty 0).
+The reasoning earlier revisions of this card gave for 0.8 was worked out on v1 and does not describe v3.
+
+For tool calling, `--jinja` is **mandatory** in llama.cpp: without it the tool grammar never comes from the
+model's chat template and every tool scenario degrades to prose.
+
+## Quants
+
+GGUF quants: **[`ManniX-ITA/JackOD-9B-Coder-MTP-GGUF`](https://huggingface.co/ManniX-ITA/JackOD-9B-Coder-MTP-GGUF)**.
+Every tier there is **v3**, built 2026-10-24 from these weights with the full AtomicChat imatrix; the
+`Q6_K` there is the file every v3 score above was measured on. DFlash builds:
+[`ManniX-ITA/JackOD-9B-Coder-MTP-DFlash-GGUF`](https://huggingface.co/ManniX-ITA/JackOD-9B-Coder-MTP-DFlash-GGUF).
+
+On ollama (both published 2026-10-25, the serving sampler above baked into every tag):
+
+- **[`mannix/jackod-9b-coder`](https://ollama.com/mannix/jackod-9b-coder)**: all 27 tiers, `:latest` =
+  `Q4_K_M`. The tags set `draft_num_predict 3` for the built-in MTP head and declare ollama ≥ 0.32.12 (the
+  `qwen3.5` renderer). Every tag was pulled back from the registry and passed a generation check on 2026-10-25.
+- **[`mannix/jackod-9b-coder-dflash`](https://ollama.com/mannix/jackod-9b-coder-dflash)**: the same 27 tiers
+  with the `Qwen3.5-9B-DFlash` Q8_0 drafter (`draft_num_predict 4`). Needs **xollama**; stock ollama has no
+  `draft-dflash`. See [`ManniX-ITA/JackOD-9B-Coder-MTP-DFlash-GGUF`](https://huggingface.co/ManniX-ITA/JackOD-9B-Coder-MTP-DFlash-GGUF).
+
+`num_ctx` is not pinned in the tags: set it as high as your hardware allows for agentic work.
+
+## JackOD-9B-Coder v2 — previous weights (replaced 2026-10-24)
+
+Everything in this section was measured on the **v2** weights (Coder-Flash 0.30 / Ornith 0.15), **greedy**
+(temperature 0), and served with the v1/v2 recipe. It is kept as the record of v2 and does not describe the v3
+weights or the v3 sampler.
+
+### Evaluation
 
 Measured on the **published Q6_K** (`JackOD-9B-Coder-Q6_K.gguf`, sha256
 `c94d907c…d25a`, full AtomicChat `ornith-1.5-9b` imatrix `5ad024b8…`), llama.cpp
@@ -152,7 +250,7 @@ there for orientation.
   base, +12.73 pp over Coder-Flash, the source it gained.
 - MultiPL-E is unchanged at 0.8033, still 1.67 pp under the base.
 
-### LiveCodeBench v6 (55 hard) — accuracy and reasoning length
+#### LiveCodeBench v6 (55 hard) — accuracy and reasoning length
 
 Same 55 problems, same 12,288-token thinking budget, same cap, for every arm.
 Token counts are the server's `completion_tokens`; *budget hits* are runs that
@@ -175,7 +273,7 @@ reached the thinking budget; *length-capped* never terminated on their own.
   tokens), slightly ahead of Coder-Flash alone (0.114), at +12.73 pp accuracy.
 - Termination stays near v1: **2 of 55** length-capped (v1: 1).
 
-### Tool calling — `tool-eval-bench` hardmode
+#### Tool calling — `tool-eval-bench` hardmode
 
 84 scenarios scored, 168 points, **5 paired seeds (42–46)**, n=5 → t=2.776,
 greedy, `--hardmode --weight-by-difficulty`, scorer `2.6.1.dev65+g6be685f0e`,
@@ -224,65 +322,11 @@ Categories A/B/D/F/N (tool selection, parameter precision, restraint & refusal,
 localization, creative composition) are 6/6 for v2; G Structured Reasoning 6/6;
 Structured Output 2/4, as for every arm in both cohorts.
 
-## Serving
-
-`generation_config.json` ships the serving recipe:
-
-```
-temperature 0.8 · top_p 0.95 · top_k 20 · min_p 0.05 · repeat_penalty 1.05
-```
-
-Add, where your runtime supports them (ollama and llama.cpp do; HF
-`GenerationConfig` has neither field):
-
-```
-repeat_last_n 96 · presence_penalty 0
-```
-
-The published `mannix/JackOD-9B-Coder` ollama tags carry exactly this set. This serving recipe was
-tuned on v1 and is carried over to v2 unchanged; it has not yet been re-tuned on the v2 weights.
-
-### Temperature is not a free parameter here
-
-0.8 is a deliberate middle, because the failure modes sit on **both** sides of
-it and they are different failures:
-
-- **Lower** degrades **coding ability first**. Push it further down and
-  hallucinations and erratic behaviour follow.
-- **Higher** makes the model **less willing to use its tools** — it drifts
-  toward writing its own scratch scripts instead of calling what it already has.
-
-Neither symptom announces itself as a sampler problem, which is why the value is
-stated rather than left to taste.
-
-### On the repetition controls
-
-Earlier revisions of this card recommended `presence_penalty 1.5`. That has been
-**replaced by `repeat_penalty 1.05` over a `repeat_last_n 96` window**, with
-`presence_penalty` back at 0. The job is the same — discouraging the model from
-re-treading a tool call it already made — but a bounded recent-token window does
-it without the broad, unbounded pressure a large presence penalty puts on every
-token the model has ever emitted.
-
-For tool calling, `--jinja` is **mandatory** in llama.cpp: without it the tool
-grammar never comes from the model's chat template and every tool scenario
-degrades to prose.
-
-## Quants
-
-GGUF quants: **[`ManniX-ITA/JackOD-9B-Coder-MTP-GGUF`](https://huggingface.co/ManniX-ITA/JackOD-9B-Coder-MTP-GGUF)**.
-Every tier there is **v2**, rebuilt 2026-10-21 from these weights with the full
-AtomicChat imatrix; the `Q6_K` there is the file every score above was measured
-on.
-
-On ollama: **`mannix/JackOD-9B-Coder`** still serves **v1** until it is
-re-published.
-
 ## JackOD-9B-Coder v1 — previous weights (replaced 2026-10-21)
 
 Everything below was measured on the **v1** weights (Qwopus3.5-9B-Coder in the
 0.30 slot). It is kept because it is the record of how this merge was chosen;
-it does not describe the v2 weights.
+it does not describe the v2 or v3 weights.
 
 ### Field impression — initial, limited testing
 
@@ -457,4 +501,3 @@ Categories A/B/D/F (tool selection, parameter precision, restraint & refusal,
 localization) are 6/6 for every arm and are omitted; G Structured Reasoning and
 N Creative Composition are 6/6 for all but Qwopus (5.8 and 5.2). Structured
 Output is 2.0/4 for all five — none of these models clears it.
-