Update model card: TQ2_0 (CPU) + Q2_KT (GPU) unified card

README.md

@@ -1,188 +1,176 @@
 ---
-quantized_by: vinpix
-base_model: prism-ml/Ternary-Bonsai-8B-gguf
 license: apache-2.0
-language:
-  - en
-library_name: gguf
+base_model: prism-ml/Ternary-Bonsai-8B
 tags:
-  - gguf
-  - llama.cpp
-  - ternary
-  - 1.58-bit
-  - bitnet
-  - cpu
-  - qwen3
+- gguf
+- llama.cpp
+- ternary
+- 1.58-bit
+- quantization
+- q2_k
+- tq2_0
+- qwen3
+- edge
+- cpu
+- vulkan
+language:
+- en
 pipeline_tag: text-generation
+library_name: gguf
 ---
 
-<div align="center">
+# Ternary-Bonsai-8B for llama.cpp
 
-# 🌳 Ternary-Bonsai-8B · TQ2_0 GGUF
+**GGUF builds of [PrismML's Ternary-Bonsai-8B](https://huggingface.co/prism-ml/Ternary-Bonsai-8B) for `llama.cpp`, covering both deployment targets: a native `TQ2_0` build for CPU, and a ternary-aware zero-exact `Q2_K` build for GPU and every k-quant runtime — at matching perplexity.**
 
-### 1.58-bit ternary 8B that runs on **stock llama.cpp** — no fork required
+Ternary-Bonsai is an end-to-end 1.58-bit model (weights in `{-1, 0, +1}`, trained with ternary QAT, no higher-precision escape hatches). It punches far above its size class, but on `llama.cpp` it runs into a practical wall: **the tight ternary format has no GPU kernel.** This repo provides both halves of the solution.
 
-<p>
-<img src="https://img.shields.io/badge/format-GGUF-1f6feb?style=for-the-badge&logo=huggingface&logoColor=white" alt="GGUF">
-<img src="https://img.shields.io/badge/precision-ternary%20%7B%E2%88%921%2C0%2C%2B1%7D-8957e5?style=for-the-badge" alt="ternary">
-<img src="https://img.shields.io/badge/body-2.06%20bpw-2ea043?style=for-the-badge" alt="2.06 bpw">
-<img src="https://img.shields.io/badge/size-2.33%20GiB-e3b341?style=for-the-badge" alt="2.33 GiB">
-</p>
+## Files
 
-<p>
-<img src="https://img.shields.io/badge/llama.cpp-mainline-000000?style=flat-square&logo=cplusplus" alt="llama.cpp">
-<img src="https://img.shields.io/badge/LM%20Studio-supported-4b32c3?style=flat-square" alt="LM Studio">
-<img src="https://img.shields.io/badge/Ollama-supported-000000?style=flat-square&logo=ollama" alt="Ollama">
-<img src="https://img.shields.io/badge/context-64K-0969da?style=flat-square" alt="64K context">
-<img src="https://img.shields.io/badge/license-Apache--2.0-750014?style=flat-square" alt="Apache-2.0">
-</p>
+| File | Format | BPW | Size | Backends | PPL (wikitext-2, 20x512) |
+|---|---|---|---|---|---|
+| `Bonsai-8B-TQ2_0.gguf` | native ternary | 2.06 | 2.50 GB | **CPU only** | 12.311 +/- 0.517 |
+| `Bonsai-8B-Q2_KT.gguf` | ternary-aware `Q2_K` | 2.911 | 2.99 GB | **CPU - CUDA - Metal - Vulkan - ROCm - SYCL** | 12.299 +/- 0.516 |
 
-<sub>A portable GGUF repack of <a href="https://huggingface.co/prism-ml/Ternary-Bonsai-8B-gguf"><b>PrismML Ternary-Bonsai-8B</b></a> · quantized by <b>vinpix</b></sub>
+Both decode to the same learned weights. They differ only in storage container, and therefore in which backends can run them and how fast.
 
-</div>
+> **TL;DR — which one?**
+> **CPU box** -> `TQ2_0` (fewer bytes streamed, ~12-14 tok/s, smallest RAM).
+> **GPU offload, or any runtime that only speaks k-quants** -> `Q2_KT`.
 
 ---
 
-<div align="center">
+## Why two files
 
-> **8B quality. 2.3 GB on disk. Runs on the llama.cpp you already have.**
+### `TQ2_0` — the native CPU format
+`TQ2_0` is the tightest faithful ternary layout in `ggml` (~2.06 BPW). It is the right choice on CPU and it is what PrismML ships. Its one hard limit:
 
-</div>
+> **`TQ2_0` has no GPU kernel.** In current `llama.cpp` it implements a `vec_dot` for CPU only — no CUDA, Metal, or Vulkan path. `-ngl` silently keeps the ternary tensors on the host.
 
-Ternary-Bonsai-8B is an 8B **Qwen3-architecture** model whose linear weights are **ternary** — every weight is one of `{−1, 0, +1}` with a shared FP16 scale per group. This repack stores that ternary body in the **mainline `TQ2_0`** type so it loads on any current llama.cpp, **LM Studio** or **Ollama** with no fork and no special flags.
+### `Q2_KT` — the GPU-portable format
+`Q2_K` is one of the most broadly supported quant types in the ecosystem — hand-written kernels for CUDA, Metal, Vulkan, ROCm, SYCL. Re-hosting the ternary weights inside a `Q2_K` container makes Bonsai portable everywhere `Q2_K` runs.
 
-<table>
-<tr>
-<td>⚡</td><td><b>Tiny footprint</b><br><sub>2.33 GiB — fits in RAM on almost anything</sub></td>
-<td>🔌</td><td><b>Zero setup</b><br><sub>Stock llama.cpp / LM Studio / Ollama</sub></td>
-</tr>
-<tr>
-<td>🎯</td><td><b>Lossless body repack</b><br><sub>Same {−1,0,+1} weights, just re-packed</sub></td>
-<td>📏</td><td><b>64K native context</b><br><sub>No RoPE scaling tricks</sub></td>
-</tr>
-</table>
-
-> [!NOTE]
-> This repository redistributes and repacks the model. **All model capabilities come from PrismML.** See the [original model card](https://huggingface.co/prism-ml/Ternary-Bonsai-8B-gguf) for architecture, training and evaluations.
+The catch: you **cannot** just run `llama-quantize ... Q2_K` on a ternary model. Doing so is actively harmful — for a reason worth explaining.
 
 ---
 
-## 📦 Files
+## The trap: naively quantizing a ternary model to `Q2_K`
 
-<div align="center">
+`Q2_K` dequantizes each weight as
 
-| File | Body | Embed / Output | Body bpw | Size |
-|:---|:---:|:---:|:---:|:---:|
-| **[Bonsai-8B-TQ2_0-Q4out-llamacpp.gguf](https://huggingface.co/vinpix/Bonsai-8B-TQ-GGUF/blob/main/Bonsai-8B-TQ2_0-Q4out-llamacpp.gguf)** | `TQ2_0` ternary | `Q4_0` | 2.06 | ~2.33 GiB |
+```
+w = d * (scale & 0xF) * q  -  dmin * (scale >> 4),   q in {0, 1, 2, 3}
+```
 
-</div>
+The stock `Q2_K` quantizer (`make_qkx2_quants`) is an L2-optimal grid fitter for **continuous** FP16 distributions. Handed a ternary tensor whose values are exactly `{-s, 0, +s}`, it fits a 4-level grid over 3-level data and places the levels so the ternary **zero lands between grid points** — at `q ~ 1.5`, rounding to an offset of `+/-s/3`.
 
-Everything except the **token-embedding** and **output (LM-head)** tensors is stored as native ternary `TQ2_0`. Those two tensors are **not** ternary in the source model — they hold higher-precision information — so they are kept in `Q4_0` to preserve it at a reasonable size.
+Measured directly on Bonsai-8B:
 
----
+| Encoding | Sign accuracy vs. original ternary | Zero reconstruction |
+|---|---|---|
+| Stock `Q2_K` (`--pure`) | **33 %** | `-0.0095` (should be `0`) |
+| **`Q2_KT` (this repo)** | **100 %** | `0.000000` (exact) |
 
-## ⬇️ Download
+A `--pure` `Q2_K` of a ternary model gets **two thirds of the signs wrong** and shifts every zero — in practice a degenerate model that loops single tokens. The stock pipeline masks this by *promoting* sensitive tensors (`attn_output`, `ffn_down`, `attn_v`) to `Q3_K`/`Q4_K`, inflating the file to ~3.18 BPW and only partially repairing the damage. That promotion is a workaround, not a fix.
 
-```bash
-pip install -U "huggingface_hub[cli]"
-
-hf download vinpix/Bonsai-8B-TQ-GGUF \
-  --include "Bonsai-8B-TQ2_0-Q4out-llamacpp.gguf" \
-  --local-dir ./
-```
+`Q2_KT` fixes the encoding at the source.
 
 ---
 
-## 🚀 Run
+## How `Q2_KT` is built (ternary-aware, zero-exact)
 
-<details open>
-<summary><b>llama.cpp</b></summary>
+The ternary structure admits an **exact** `Q2_K` encoding with no residual. For a group of magnitude `s`:
 
-```bash
-llama-cli -m Bonsai-8B-TQ2_0-Q4out-llamacpp.gguf \
-  -p "The capital of France is" \
-  -n 256 --temp 0.5 --top-p 0.85 --top-k 20
+```
+scale nibble  sc = 15
+min   nibble  m  = 15          (min-nibble == scale-nibble)
+d = dmin = s / 15
+q = t + 1                      t in {-1, 0, +1}  ->  q in {0, 1, 2}
 ```
 
-OpenAI-compatible server:
+Substituting into the dequant:
 
-```bash
-llama-server -m Bonsai-8B-TQ2_0-Q4out-llamacpp.gguf -c 4096
 ```
-</details>
+w = d*15*q - dmin*15 = d*15*(q - 1) = s*(q - 1) = s*t     <- exact ternary
+```
 
-<details>
-<summary><b>LM Studio / Ollama</b></summary>
+The `-1` bias falls out of the min term for free; the ternary zero (`t = 0 -> q = 1`) reconstructs to **exactly 0**; every non-zero to **exactly `+/-s`**. The only error is the FP16 storage of the single per-group scale — which `TQ2_0` also carries. Hence sign accuracy 100 % and PPL parity.
 
-Point them at the `.gguf` file directly. No fork or extra flags are required.
-</details>
+The `qs` nibbles are packed with the exact `ggml` `Q2_K` interleave (32-byte groups, stride-32 crumbs across 128-element spans), so the file is bit-compatible with the reference `dequantize_row_q2_K` **and** the SIMD `vec_dot_q2_K_q8_K` in the forward pass. Output projection and token embedding are kept at `Q4_0` (not ternary in the source).
 
-<details>
-<summary><b>Recommended sampling & threads</b></summary>
+> **Layout note.** `TQ2_0` and `Q2_K` `qs` streams share the *same* stride-32 / 128-span interleave. Matching `ggml`'s canonical order is the whole game: an encoding that is self-consistent with a hand-rolled unpacker but disagrees with the canonical order will pass every fidelity check against itself and still emit garbage in the forward pass, because `dequantize_row` and `vec_dot` both assume the canonical order. This build matches it.
 
-| Setting | Value |
-|:---|:---|
-| temperature | `0.5` |
-| top_p | `0.85` |
-| top_k | `20` |
-| threads (`-t`) | your **physical** core count |
+---
+
+## Usage
 
-More threads than physical cores usually does not help and can hurt.
-</details>
+### CPU (use `TQ2_0`)
 
-<details>
-<summary><b>Low-RAM tip — KV-cache quantization</b></summary>
+```bash
+llama-cli -m Bonsai-8B-TQ2_0.gguf -p "The capital of France is" -n 64 -t 6
+```
 
-Shrink the KV cache at runtime for long contexts (does **not** modify the weights):
+Fastest on CPU and smallest RAM footprint. ~12-14 tok/s on a low-end Zen2 desktop (Ryzen 3 PRO 4350GE, dual-channel DDR4).
+
+### GPU offload (use `Q2_KT`)
 
 ```bash
-llama-cli -m Bonsai-8B-TQ2_0-Q4out-llamacpp.gguf \
-  --cache-type-k q8_0 --cache-type-v q8_0 -p "..."
+llama-cli -m Bonsai-8B-Q2_KT.gguf -ngl 99 -p "The capital of France is" -n 64
 ```
 
-`q8_0` cache is close to lossless; `q4_0` saves more memory at a small quality cost.
-</details>
+Verified running fully offloaded on an AMD Radeon Vega iGPU via the Vulkan `RADV` backend. Because `Q2_K` has real GPU kernels, `-ngl 99` actually places the ternary body on the device — unlike `TQ2_0`, which would stay on the host.
 
----
+### Any GGUF loader / llama.cpp server
+
+`Q2_KT` loads as an ordinary `Q2_K` model — no custom code, no patched runtime. Every existing loader already understands the container.
 
-## 🔬 Format details
+---
 
-`TQ2_0` packs **four ternary weights per byte** (2 bits each) with one scale per block. A weight is reconstructed as:
-w = q_ternary × scale      where  q_ternary ∈ {−1, 0, +1}
+## Quality
 
-Because the source weights are already ternary, the body is a **re-packing of the same `{−1, 0, +1}` values — not a re-quantization of full-precision weights** — so it carries no additional accuracy loss. The embedding and LM-head are the only non-ternary weights in the source and are stored here in `Q4_0`.
+Perplexity on `wikitext-2-raw` (test split, 20 chunks of 512 tokens, `-t 4`, identical harness):
 
-<details>
-<summary><b>Reproduce these files</b></summary>
+| Model | Format | BPW | PPL (lower is better) |
+|---|---|---|---|
+| `Bonsai-8B-TQ2_0` | native ternary | 2.06 | 12.311 +/- 0.517 |
+| `Bonsai-8B-Q2_KT` | ternary-aware `Q2_K` | 2.911 | **12.299 +/- 0.516** |
 
-Produced from the PrismML F16 source with mainline llama.cpp:
+`Q2_KT` is **perplexity-neutral vs. the native ternary build** (delta within the +/-0.5 statistical error). You pay ~0.85 BPW over `TQ2_0` for universal backend support; you do **not** pay for it in quality — expected, since the encoding is lossless on the ternary levels.
 
-```bash
-llama-quantize --token-embedding-type q4_0 --output-tensor-type q4_0 \
-  Ternary-Bonsai-8B-F16.gguf \
-  Bonsai-8B-TQ2_0-Q4out-llamacpp.gguf TQ2_0
-```
-</details>
+For task benchmarks (MMLU-Redux, GSM8K, HumanEval+, IFEval, ...), see [PrismML's numbers](https://prismml.com/news/ternary-bonsai) for the base model — these files change the container, not the learned weights.
 
 ---
 
-## ℹ️ Notes & limitations
+## Technical notes & reproducibility
+
+- **`Q2_KT` container:** 252 ternary body tensors -> pure `Q2_K` (zero-exact); `output` + `token_embd` -> `Q4_0`; 145 norms -> `F32`.
+- **No tensor promotion.** Every ternary tensor is pure `Q2_K`; the zero-exact encoding makes the `Q3_K`/`Q4_K` promotion unnecessary — that is why `Q2_KT` is 2.91 BPW, not 3.18.
+- **Bit-exactness:** validated against `ggml`'s reference `dequantize_row_q2_K` (100 % sign accuracy, max abs error 1e-5 from the FP16 scale) and end-to-end against the forward `vec_dot` (coherent generation, PPL parity).
+- **Architecture:** Qwen3, 36 layers, `hidden 4096`, `FFN 12288`, GQA 32/8, full 65536-token context preserved in metadata.
+- **Reproduce PPL:** `llama-perplexity -m <file> -f wiki.test.raw --chunks 20`.
 
-- This is a **re-quantization / re-pack**, not a re-train. All capabilities are inherited from [Ternary-Bonsai-8B](https://huggingface.co/prism-ml/Ternary-Bonsai-8B-gguf).
-- The body uses portable `TQ2_0` (group size 256). The official PrismML `Q2_0` (group size 128) is marginally more faithful to the original grouping but requires the [PrismML fork](https://github.com/PrismML-Eng/llama.cpp); this repack trades that for running on stock llama.cpp.
-- CPU inference of ternary models is **memory-bandwidth bound** — tokens/second depend on your system's memory bandwidth and core count, so throughput varies from machine to machine.
-- Native context length is **65,536 tokens** (RoPE base 1e6); no scaling applied.
+### A note for the llama.cpp community
+
+The `+/-s/3` zero-error affects **any** end-to-end ternary model (Bonsai, BitNet, TriLM, ...) run through the stock `Q2_K` quantizer. A ternary-aware branch in `quantize_row_q2_K` — detect a block with <=3 symmetric levels, emit `d = dmin = s/15`, `sc = m`, `q = t+1` — would let the official pipeline produce zero-exact `Q2_K` for the whole class, without the promotion workaround. `Q2_KT` is a concrete demonstration that the encoding is exact and backend-portable. Contributions upstream welcome.
 
 ---
 
-<div align="center">
+## Attribution & license
 
-### 🙏 Credits
+- **Base model:** [Ternary-Bonsai-8B](https://huggingface.co/prism-ml/Ternary-Bonsai-8B) by **PrismML** (Babak Hassibi et al., Caltech). Ternary QAT, Qwen3 architecture, end-to-end 1.58-bit.
+- **These files:** GGUF re-encodes for `llama.cpp`. `TQ2_0` is the native ternary layout; `Q2_KT` is a lossless-on-ternary `Q2_K` re-encode for backend portability. Learned weights unchanged.
+- **License:** Apache 2.0, inherited from the base model.
 
-**Model & all capabilities** — [Prism ML · Ternary-Bonsai-8B](https://huggingface.co/prism-ml/Ternary-Bonsai-8B-gguf) · [whitepaper](https://prismml.com)
-**Quantization / repack** — vinpix
-**License** — Apache-2.0 (inherited from the source model)
+```bibtex
+@techreport{ternarybonsai,
+  title  = {Ternary Bonsai: 1.58-bit Language Models at 8B, 4B, and 1.7B Scale},
+  author = {Prism ML},
+  year   = {2026},
+  month  = {April},
+  url    = {https://prismml.com}
+}
+```
 
-<sub>⭐ If this repack is useful, a like on the repo helps others find it.</sub>
+---
 
-</div>
+*Community re-encodes, not affiliated with or endorsed by PrismML. Quality figures are measured on the specific files in this repository.*