Blame for README.md

---
license: apache-2.0
base_model: Qwen/Qwen3-0.6B
tags:
  - gguf
  - speculative-decoding
  - draft-model
  - bonsai
  - llama.cpp
pipeline_tag: text-generation
---

# Common Flash v0.1

A drafter for **speculative decoding of Ternary Bonsai 8B (1st gen, Q2_0_g64)** in llama.cpp.
A small model proposes tokens and Bonsai checks them in one pass, so the output is unchanged and decoding is faster.

`Qwen3-0.6B-bonsai-lora-Q4_0.gguf` (358 MiB) is Qwen3-0.6B, LoRA-tuned on Bonsai 8B's own completions
(1,000 completions of at most 192 tokens; rank 32, all linear layers), vocabulary trimmed to Bonsai's 151,669 tokens
(llama.cpp needs the draft vocabulary within 128 tokens of the target's), then quantized to Q4_0.

## Run it in one command

Needs a recent llama.cpp (`llama-server`, build 1268eda or newer; see Quick start below). This downloads Bonsai 8B and this drafter, then starts a speculative server with a chat page at http://127.0.0.1:8080:

```bash
llama-server \
  -hf  prism-ml/Ternary-Bonsai-8B-gguf:Q2_0_g64 \
  -hfd TheCommonAI/common-flash-v0.1 \
  -ngl 99 -ngld 99 -fa on -np 1 -c 4096 \
  --spec-type draft-simple --spec-draft-n-max 3
```

`-hf` fetches the target model, `-hfd` fetches the drafter from this repo. Tested from an empty cache; the first start downloads about 2.7 GB.

> **About the "Use this model" button.** Hugging Face generates that snippet itself, and for this repo it would start the drafter alone, which is not useful. Use the command above instead.

![Decode speed by model](assets/speed_by_model.png)

## Target model (required pairing)

This drafter only works with **`Ternary-Bonsai-8B-Q2_0_g64.gguf`** from
[prism-ml/Ternary-Bonsai-8B-gguf](https://huggingface.co/prism-ml/Ternary-Bonsai-8B-gguf)
(2,310,125,920 bytes, sha256 starting `e17b298d84ee`). That is first-generation Ternary Bonsai 8B.
It was tuned on that model's outputs and trimmed to its vocabulary; it is not tested with other Bonsai models
(Bonsai 2, the 27B, or the 1.7B), and mismatched vocabularies are rejected by llama.cpp.

## Quick start

You need three things: a recent llama.cpp, the Bonsai 8B model, and this drafter.

**1. Build or install llama.cpp (recent upstream).** Tested with upstream commit `1268eda`. An older Homebrew build (b11429) was *slower* with speculation than without, so use a current build.

```bash
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build            # Apple Silicon: Metal is on by default
# NVIDIA GPU:  cmake -B build -DGGML_CUDA=ON      (Vulkan: -DGGML_VULKAN=ON)
cmake --build build --config Release -j
# binaries land in build/bin/ (build/bin/Release/ on Windows)
```

**2. Download both models.**

```bash
pip install -U "huggingface_hub[cli]"
hf download prism-ml/Ternary-Bonsai-8B-gguf Ternary-Bonsai-8B-Q2_0_g64.gguf --local-dir models
hf download TheCommonAI/common-flash-v0.1 Qwen3-0.6B-bonsai-lora-Q4_0.gguf --local-dir models
```

Disk: about 2.3 GB for Bonsai plus 0.4 GB for the drafter. Memory: roughly 3 GB plus the context cache.

**3. Run the server.**

```bash
./build/bin/llama-server \
  -m  models/Ternary-Bonsai-8B-Q2_0_g64.gguf \
  -md models/Qwen3-0.6B-bonsai-lora-Q4_0.gguf \
  -ngl 99 -ngld 99 -fa on -t 6 -np 1 -c 4096 \
  --spec-type draft-simple --spec-draft-n-max 3 \
  --host 127.0.0.1 --port 8080
```

| Flag | Meaning |
| --- | --- |
| `-m` / `-md` | Target model (Bonsai) / draft model (this repo) |
| `-ngl 99` / `-ngld 99` | Put all layers of the target / drafter on the GPU. Use `0` for CPU-only (untested, likely slower) |
| `-fa on` | Flash attention |
| `-t 6` | CPU threads; set to your number of performance cores |
| `-np 1` | One request slot. Speculation helps single-stream speed; more slots share the GPU |
| `-c 4096` | Context size in tokens. Lower it (for example `-c 2048`) if you run out of memory |
| `--spec-type draft-simple` | Classic draft-model speculative decoding |
| `--spec-draft-n-max 3` | Draft 3 tokens per round. 3 was best in our tests; 8 was close; 2 and 5 were no better |

**4. Try it.** Open http://127.0.0.1:8080 for the built-in chat page, or:

```bash
curl -s localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "messages": [{"role": "user", "content": "Explain speculative decoding in two sentences."}],
  "max_tokens": 128, "temperature": 0.7, "top_p": 0.8, "top_k": 20, "presence_penalty": 1.5,
  "chat_template_kwargs": {"enable_thinking": false}
}'
```

**5. Check that speculation is working.** The server's own `/completion` endpoint reports it:

```bash
curl -s localhost:8080/completion -d '{"prompt":"<|im_start|>user\nHello<|im_end|>\n<|im_start|>assistant\n<think>\n\n</think>\n\n","n_predict":64}' \
  | python3 -c "import json,sys; t=json.load(sys.stdin)['timings']; print(t['predicted_per_second'],'tok/s', t.get('draft_n_accepted'),'/',t.get('draft_n'),'drafts accepted')"
```

Acceptance of roughly 60-70% on code and copy-heavy text, and 35-45% on open chat, is normal. If `draft_n` is missing or 0, the drafter did not load.

**To compare against plain decoding**, start a second server on another port without `-md`, `-ngld` and `--spec-type`, and send both the same prompts. Run them one after the other on a quiet machine: absolute speeds on a busy machine are noisy, and only ratios are meaningful. Speculation cannot be switched off per request in this build, so restart the server to disable it.

## Troubleshooting

- **"draft model vocab must match" / vocab size error**: you are not using the exact Bonsai 8B 1st-generation Q2_0_g64 file named above.
- **Slower than without the drafter**: use a current upstream llama.cpp build; try `--spec-draft-n-max 2`; make sure nothing else is using the GPU.
- **Out of memory**: lower `-c`, and run one server at a time.
- **Output differs from plain decoding**: with temperature 0 the text is identical. With sampling the random draws differ, which is expected.

## Platform status

| Platform | Status |
| --- | --- |
| Apple Silicon (Metal) | Measured, about 1.36x (one M3, 16 GB) |
| NVIDIA CUDA (Linux/Windows) | Bonsai Q2_0 runs on CUDA; speculation speedup not yet measured |
| Vulkan, CPU-only | Untested |

If you try another platform, please report your plain versus speculative tokens per second.

## Results

One Apple M3 (16 GB), Metal, 36 prompts the drafter never trained on, 128 tokens each, the network worker's sampling
(temperature 0.7, top_p 0.8, top_k 20, presence_penalty 1.5), configurations interleaved prompt by prompt.
Speed relative to plain decoding:

| | speedup |
| --- | ---: |
| overall | ~1.36x |
| chat | ~1.14x |
| code | ~1.31x |
| copy-heavy edits | ~1.67x |

Draft acceptance is about 65% overall but only 33-45% on chat. Run-to-run ratios varied between 1.19x and 1.41x on a busy machine, so treat these as approximate.
With greedy decoding every speculative output matched plain decoding. With sampling the random draws differ, so that check does not apply.


### Charts

![Speedup by workload](assets/speedup_by_workload.png)

![Why the runtime matters](assets/verify_throughput.png)

![What helped](assets/what_helped.png)

### Decode speed on the same M3 (plain, llama.cpp 1268eda, `llama-bench` tg128)

| Model | File size | tokens/s |
| --- | ---: | ---: |
| Ternary Bonsai 1.7B | 0.46 GB | 127 |
| Ternary Bonsai 2 2B | 0.51 GB | 97 |
| **Common Flash** (Bonsai 8B + drafter) | 2.3 GB + 0.4 GB | **~49** (36.3 x 1.36) |
| Ternary Bonsai 2 4B | 1.1 GB | 48 |
| Ternary Bonsai 8B, plain | 2.1 GB | 36 |
| Ternary Bonsai 2 9B | 2.4 GB | 28 |

The Common Flash row is derived: plain Bonsai 8B speed times the measured 1.36x ratio, not a separate `llama-bench` run. Smaller models are faster but are different, smaller models; the comparison that matters for quality is Bonsai 8B plain against Bonsai 8B with the drafter, which produce the same text.

## Limits

- Tested on one machine. Older llama.cpp builds or other hardware may not see a speedup.
- The test prompts come from Alpaca-cleaned, CodeAlpaca and HumanEval-style prompts, not real traffic.
- More training data, soft-label distillation, tighter quantization, MLP pruning and tuned draft lengths did not improve it; acceptance appears limited by the 0.6B drafter's size.

## Licence

Apache-2.0, inherited from Qwen3-0.6B. Training inputs were prompts from Alpaca-cleaned and CodeAlpaca-20k with Bonsai 8B's own completions as targets; check those datasets' terms and Prism ML's Bonsai terms before relying on this commercially.