| --- |
| license: mit |
| tags: |
| - metacog |
| - ensemble |
| - system-one |
| - text-generation |
| base_model: |
| - prism-ml/Ternary-Bonsai-2-27B-gguf |
| - Mapika/decider-4b |
| --- |
| |
| # Heuristic 1 |
|
|
| One model out of two: **Bonsai 2 27B (ternary)** thinks, **Decider-4B** judges, |
| [MetaCog](https://github.com/ItIsCuthNotCup/MetaCog) wires them into a single |
| OpenAI-compatible endpoint. |
|
|
| > **This is not a weight merge.** No tensors were averaged. It is an |
| > inference-time pipeline merge: the thinker writes candidate answers, an |
| > independent System-One judge (Decider-4B) scores them, and the winner is |
| > returned. One endpoint, one model ID, two models inside. |
|
|
| ``` |
| user prompt ──▶ bonsai-27B ──▶ greedy anchor judged FIRST |
| │ ≥ 0.80 and finished? ──▶ answer immediately |
| ▼ |
| 7 sampled paths IN PARALLEL, |
| each judged the moment it finishes; |
| first finished path ≥ 0.80 wins on the spot, |
| the rest are abandoned |
| │ |
| (nothing clears the bar? best judged |
| finished path wins) |
| ``` |
|
|
| The judge is *not* the thinker — that separation is the whole point. |
|
|
| `race.py` implements this loop (parallel generation, judge-as-they-finish, |
| early stop). The old sequential MetaCog `best_of_n` is still available via |
| `NO_RACE=1` (server) or `--no-race` (batch) for A/B comparison. The |
| early-stop bar is `STOP_CONFIDENCE` / `--stop-confidence` (default 0.80). |
|
|
| > **On the 0.80 bar, honestly:** the MetaCog repo measured its 0.95 cascade |
| > threshold on real Jev (a noul ≥ 0.95 was correct 226/229 times). 0.80 did |
| > not win anywhere — it is the operator's directive and is untested on |
| > decider-4b, whose confidence scale is not calibrated (values above 1.0 |
| > have been observed). Measure before trusting it. |
|
|
| ## Measured |
|
|
| | eval | bonsai alone | merge (Heuristic 1) | note | |
| |---|---|---|---| |
| | HumanEval, n=164, true pass@1 (prompt+completion graded as one program, `<think>` stripped) | 108/164 (65.9%) | **135/164 (82.3%)** | judge fixed 27, broke 0; McNemar p < 1e-6 | |
| | 12 easy Q&A (verified) | 12/12 | 12/12 | tie; nothing to fix | |
| | 10 hard Q&A (verified, tricks + big arithmetic) | 8/10 | **10/10** | judge fixed 2, broke 0 | |
|
|
| Cost per problem (HumanEval averages): merge uses ~5,600 thinker tokens |
| over 2.0 thinker calls + 8.5 judge calls; bonsai alone uses ~700 tokens in |
| 1 call. Roughly 8× the thinker tokens for +16 points of accuracy, $0 on |
| local hardware. |
|
|
| Latency: when the judge is confident in the greedy answer it returns |
| immediately (~same as bonsai alone — e.g. 2.5s on a trivial prompt); the |
| full vote only runs when uncertain. Measured on 3 HumanEval problems where |
| the full vote ran (DGX Spark, sequential paths, old loop): heuristic-1 took |
| 210s / 406s / 440s vs bonsai alone at 17s / 18s / 18s — roughly 12–24× slower |
| wall-clock on hard problems. The race loop (`race.py`) parallelizes the |
| sampled paths and cuts off the moment any finished path clears 0.80, which |
| removes most of that gap; re-measure on the Spark to confirm. Every response |
| carries `merge.judge_confidence`. |
|
|
| ## Run it |
|
|
| ```bash |
| pip install -r requirements.txt # metacog + httpx + pydantic |
| |
| # one model on :8200 |
| python server.py |
| # POST /v1/chat/completions {"messages": [...], "model": "bonsai-decider4b-merge"} |
| |
| # batch runner |
| python merge.py --problem "What is 17*23?" |
| python merge.py --problems problems.jsonl --out results.jsonl |
| ``` |
|
|
| Env knobs: `THINKER_URL` (default `http://localhost:8010`), |
| `THINKER_MODEL` (default `bonsai`), `JUDGE_URL` (default |
| `http://localhost:8008`), `PORT` (8200), `N_PATHS` (8), `NO_CASCADE`, |
| `MAX_TOKENS` (2048), `TEMPERATURE` (0.8). |
|
|
| You need both backends up: |
|
|
| - **Thinker** — any OpenAI-compatible chat endpoint serving Bonsai 2 27B. |
| Reference setup: bjev's PrismML llama.cpp fork on `:8010` |
| (`Ternary-Bonsai-2-27B-PQ2_0.gguf`, 6.66 GiB). |
| - **Judge** — a `/v1/systemone` endpoint serving Decider-4B. |
| Reference: `runs/judge_server.py` (a `/v1/systemone` shim over the |
| `Mapika/decider-4b` checkpoint) on `:8008`. |
|
|
| On a DGX Spark this fits alongside both models in ~20 GB and runs at $0. |
|
|
| ## Files |
|
|
| - `server.py` — the one-model OpenAI-compatible server (`:8200`) |
| - `merge.py` — batch runner (problems in, answers + confidence + timings out) |
| - `eval/ab_bonsai_vs_merge.py` — A/B harness: bonsai alone vs the merge |
| - `eval/problems_easy.jsonl`, `eval/problems_hard.jsonl` — the measured sets |
|
|
| ## License |
|
|
| MIT. This repo ships pipeline code only — no weights. Parent model licenses |
| (checked 2026-09-27): |
| - Bonsai 2 27B ternary (`prism-ml/Ternary-Bonsai-2-27B-gguf`) — Apache-2.0 |
| - Decider-4B (`Mapika/decider-4b`) — Apache-2.0 |
| - MetaCog — MIT; llama.cpp — MIT |
| Fetch the weights yourself under their Apache-2.0 terms; nothing here |
| relicenses them. Not affiliated with PrismML, Mapika, or Alibaba. |
|
|
