File size: 4,978 Bytes
cd34530 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 | ---
license: mit
tags:
- metacog
- ensemble
- system-one
- text-generation
base_model:
- prism-ml/Ternary-Bonsai-2-27B-gguf
- Mapika/decider-4b
---
# Heuristic 1
One model out of two: **Bonsai 2 27B (ternary)** thinks, **Decider-4B** judges,
[MetaCog](https://github.com/ItIsCuthNotCup/MetaCog) wires them into a single
OpenAI-compatible endpoint.
> **This is not a weight merge.** No tensors were averaged. It is an
> inference-time pipeline merge: the thinker writes candidate answers, an
> independent System-One judge (Decider-4B) scores them, and the winner is
> returned. One endpoint, one model ID, two models inside.
```
user prompt ──▶ bonsai-27B ──▶ greedy anchor judged FIRST
│ ≥ 0.80 and finished? ──▶ answer immediately
▼
7 sampled paths IN PARALLEL,
each judged the moment it finishes;
first finished path ≥ 0.80 wins on the spot,
the rest are abandoned
│
(nothing clears the bar? best judged
finished path wins)
```
The judge is *not* the thinker — that separation is the whole point.
`race.py` implements this loop (parallel generation, judge-as-they-finish,
early stop). The old sequential MetaCog `best_of_n` is still available via
`NO_RACE=1` (server) or `--no-race` (batch) for A/B comparison. The
early-stop bar is `STOP_CONFIDENCE` / `--stop-confidence` (default 0.80).
> **On the 0.80 bar, honestly:** the MetaCog repo measured its 0.95 cascade
> threshold on real Jev (a noul ≥ 0.95 was correct 226/229 times). 0.80 did
> not win anywhere — it is the operator's directive and is untested on
> decider-4b, whose confidence scale is not calibrated (values above 1.0
> have been observed). Measure before trusting it.
## Measured
| eval | bonsai alone | merge (Heuristic 1) | note |
|---|---|---|---|
| HumanEval, n=164, true pass@1 (prompt+completion graded as one program, `<think>` stripped) | 108/164 (65.9%) | **135/164 (82.3%)** | judge fixed 27, broke 0; McNemar p < 1e-6 |
| 12 easy Q&A (verified) | 12/12 | 12/12 | tie; nothing to fix |
| 10 hard Q&A (verified, tricks + big arithmetic) | 8/10 | **10/10** | judge fixed 2, broke 0 |
Cost per problem (HumanEval averages): merge uses ~5,600 thinker tokens
over 2.0 thinker calls + 8.5 judge calls; bonsai alone uses ~700 tokens in
1 call. Roughly 8× the thinker tokens for +16 points of accuracy, $0 on
local hardware.
Latency: when the judge is confident in the greedy answer it returns
immediately (~same as bonsai alone — e.g. 2.5s on a trivial prompt); the
full vote only runs when uncertain. Measured on 3 HumanEval problems where
the full vote ran (DGX Spark, sequential paths, old loop): heuristic-1 took
210s / 406s / 440s vs bonsai alone at 17s / 18s / 18s — roughly 12–24× slower
wall-clock on hard problems. The race loop (`race.py`) parallelizes the
sampled paths and cuts off the moment any finished path clears 0.80, which
removes most of that gap; re-measure on the Spark to confirm. Every response
carries `merge.judge_confidence`.
## Run it
```bash
pip install -r requirements.txt # metacog + httpx + pydantic
# one model on :8200
python server.py
# POST /v1/chat/completions {"messages": [...], "model": "bonsai-decider4b-merge"}
# batch runner
python merge.py --problem "What is 17*23?"
python merge.py --problems problems.jsonl --out results.jsonl
```
Env knobs: `THINKER_URL` (default `http://localhost:8010`),
`THINKER_MODEL` (default `bonsai`), `JUDGE_URL` (default
`http://localhost:8008`), `PORT` (8200), `N_PATHS` (8), `NO_CASCADE`,
`MAX_TOKENS` (2048), `TEMPERATURE` (0.8).
You need both backends up:
- **Thinker** — any OpenAI-compatible chat endpoint serving Bonsai 2 27B.
Reference setup: bjev's PrismML llama.cpp fork on `:8010`
(`Ternary-Bonsai-2-27B-PQ2_0.gguf`, 6.66 GiB).
- **Judge** — a `/v1/systemone` endpoint serving Decider-4B.
Reference: `runs/judge_server.py` (a `/v1/systemone` shim over the
`Mapika/decider-4b` checkpoint) on `:8008`.
On a DGX Spark this fits alongside both models in ~20 GB and runs at $0.
## Files
- `server.py` — the one-model OpenAI-compatible server (`:8200`)
- `merge.py` — batch runner (problems in, answers + confidence + timings out)
- `eval/ab_bonsai_vs_merge.py` — A/B harness: bonsai alone vs the merge
- `eval/problems_easy.jsonl`, `eval/problems_hard.jsonl` — the measured sets
## License
MIT. This repo ships pipeline code only — no weights. Parent model licenses
(checked 2026-09-27):
- Bonsai 2 27B ternary (`prism-ml/Ternary-Bonsai-2-27B-gguf`) — Apache-2.0
- Decider-4B (`Mapika/decider-4b`) — Apache-2.0
- MetaCog — MIT; llama.cpp — MIT
Fetch the weights yourself under their Apache-2.0 terms; nothing here
relicenses them. Not affiliated with PrismML, Mapika, or Alibaba.
|