Wondernutts/Bonsai-27B-INTERNARY-OpenVINO-Arc

Bonsai 27B INTERNARY OpenVINO Arc

This is the signed-ternary OpenVINO GenAI build of Prism ML's Bonsai 27B for Intel Arc. Its language projections stay packed at two bits per weight. A custom OpenVINO 2026.2 Intel GPU plugin runs prefill with native dpas.s2.s8 and uses a separate W2A16 path for single-token decode.

This model does not run with the stock OpenVINO Intel GPU plugin. Install the shipped plugin or build the pinned fork commit below.

Status

The complete model, plugin path, deterministic token test, long-context retrieval, clean-card throughput, and packaged runtime have been tested on an Intel Arc Pro B70 running Linux.

Platform	Status
Linux x86-64, Intel Arc Pro B70	Tested
Other Intel Arc GPUs	Untested
Windows	Untested; no Windows DLL is shipped
Stock OpenVINO	Unsupported; patched plugin required
Vision input	Untested

This is not the first ternary model on OpenVINO. OpenVINO and Optimum Intel already support Microsoft's ternary BitNet. This release is the Bonsai 27B signed-u2 projection-group implementation and its measured Arc path.

Model

Item	Value
Upstream model	prism-ml/Bonsai-27B-gguf
Base architecture	Qwen3.5 27B / OpenVINO Qwen3_5ForConditionalGeneration
Language layers	64
Ternary encoding	11=-1, 00=0, 01=+1
Projection weights	25,621,954,560
Packed projection bytes	6,405,488,640
Effective projection bpw	2.125, including one FP16 scale per 128 weights
OpenVINO graph	257 projection groups containing 497 projections
Language-model binary	6,811,499,346 bytes
Complete OpenVINO artifact	8.1 GiB
License	Apache 2.0

The language-model file is not the entire repository. Text embeddings, vision components, tokenizer files, graph XML, scales, kernels, and runtime tools are also included. Text inference does not mean every file in the package is ternary.

Required Stack

The tested configuration is:

• Ubuntu 24.04.4 LTS, Linux kernel 7.0.0 with the xe driver
• Intel Arc Pro B70, 32 GB
• Intel compute runtime 26.22.38646.6
• OpenVINO 2026.2.0
• OpenVINO GenAI 2026.2.0.0
• Python 3.12
• OpenVINO fork branch internary-projection-groups-2026.2
• OpenVINO base commit 52ddc07385712456dd9f8c5ecf05d7e49c6da329
• INTERNARY implementation commit 6339785dcd5a4356a01e8647afdac5a994363af7

Older drivers and other Arc generations may work, but they have not been tested. The driver must expose native signed-2-bit by signed-8-bit DPAS.

Install

1. Install the Intel GPU runtime

Follow OpenVINO's Intel GPU configuration guide. On Ubuntu 22.04 or 24.04, install:

sudo apt-get install -y \
  ocl-icd-libopencl1 \
  intel-opencl-icd \
  intel-level-zero-gpu \
  level-zero \
  clinfo
sudo usermod -aG render "$USER"

Log out and back in after changing group membership. Check the devices:

clinfo -l

python - <<'PY'
import openvino as ov
core = ov.Core()
for device in core.available_devices:
    print(device, core.get_property(device, "FULL_DEVICE_NAME"))
PY

Record the discrete GPU ID. On a machine with an Intel iGPU, the Arc card may be GPU.1 or GPU.2 instead of GPU.0.

2. Create the Python environment

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

The pinned requirements install OpenVINO 2026.2.0, OpenVINO GenAI 2026.2.0.0, NumPy, and the Hugging Face CLI.

3. Download the model

hf download Wondernutts/Bonsai-27B-INTERNARY-OpenVINO-Arc \
  --local-dir ./Bonsai-27B-INTERNARY-OpenVINO-Arc
cd ./Bonsai-27B-INTERNARY-OpenVINO-Arc
sha256sum -c CHECKSUMS.sha256

4. Install the patched Intel GPU plugin

The repository contains the exact Linux x86-64 plugin used for the tests:

python runtime/install_plugin.py \
  plugin/linux-x86_64/libopenvino_intel_gpu_plugin.so \
  --sha256 fbf51b2fd61df7d3e6229b787ef97963164ae45da7e9509fe629fde2c1829cc3

The installer checks OpenVINO 2026.2, backs up the stock plugin, and installs the patched library into the active environment.

To audit or rebuild it:

git clone --branch internary-projection-groups-2026.2 --single-branch \
  https://github.com/Wondernuttz/openvino.git openvino-internary
cd openvino-internary
git checkout --detach 6339785dcd5a4356a01e8647afdac5a994363af7
git submodule update --init --recursive

sudo ./install_build_dependencies.sh
sudo apt-get install -y patchelf
cmake -S . -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DENABLE_PYTHON=OFF \
  -DENABLE_INTEL_GPU=ON
cmake --build build --target openvino_intel_gpu_plugin -j"$(nproc)"
cmake --install build --prefix "$PWD/install" --component gpu

PLUGIN="$PWD/install/runtime/lib/intel64/libopenvino_intel_gpu_plugin.so"
patchelf --set-rpath '$ORIGIN' "$PLUGIN"
cd ..
python ./Bonsai-27B-INTERNARY-OpenVINO-Arc/runtime/install_plugin.py "$PLUGIN"

The prebuilt plugin is for OpenVINO 2026.2 on Linux x86-64. Windows needs a matching DLL built and tested from the same commit.

5. Verify the installation

python runtime/verify.py --model .

This checks the OpenVINO versions, plugin, model graph, custom configuration, six extension registrations, and visible device IDs.

Generate

python runtime/generate.py \
  --model . \
  --device GPU.1 \
  --prompt "Explain ternary neural-network weights in plain English." \
  --max-new-tokens 256

The first start compiles the graph and kernels. Later starts reuse the cache.

Reasoning

Bonsai can generate a reasoning section before its answer. The runtime removes a completed <think>...</think> section by default and returns only the answer. It also reserves 256 extra tokens so short answer limits do not cut the model off in the middle of reasoning.

For harder prompts:

python runtime/generate.py \
  --model . \
  --device GPU.1 \
  --prompt "Solve this carefully and check the result." \
  --max-new-tokens 256 \
  --reasoning-headroom 1024

Use --show-reasoning when raw reasoning output is actually wanted. If the reasoning section does not finish, the runtime returns a clear error instead of printing a partial thought as the answer.

OpenAI-Compatible Server

python runtime/serve_openai.py \
  --model . \
  --device GPU.1 \
  --host 127.0.0.1 \
  --port 8000

curl http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "bonsai-27b-internary-openvino-arc",
    "messages": [{"role": "user", "content": "Tell me about Skyrim."}],
    "max_tokens": 128,
    "temperature": 0.7
  }'

The server sends every generation through one persistent worker thread. Calling one shared OpenVINO GPU pipeline from a different OS thread on each request caused cumulative latency during testing.

The server has no authentication or TLS. Keep it on a trusted network or put it behind an authenticated reverse proxy before exposing it.

Speculative Decoding Status

This release does not ship speculative decoding. The reported decode numbers are the target model running one accepted token at a time.

That is separate from INTERNARY kernel performance. Better projection kernels can raise prompt processing and target verification speed. A draft model can raise output throughput by proposing several tokens for the target to verify. It does not double prompt processing by itself.

The base config contains mtp_num_hidden_layers=1, but the exported OpenVINO language graph contains no MTP layer or MTP input/output ports. There is no hidden working MTP path in this package.

DFlash for Qwen3.5-27B is a separate six-layer block-diffusion draft model, not that config entry. OpenVINO GenAI 2026.2 supports ordinary autoregressive draft models through LLMPipeline, but it does not implement the DFlash block-diffusion algorithm and VLMPipeline has no draft_model constructor. Shipping DFlash on Intel therefore requires a real OpenVINO export plus a DFlash proposal, verification, and cache rollback implementation. It is tracked as separate work and no speculative speedup is claimed here.

A July 20 Arc prototype now exports the official six-layer DFlash draft, exposes the eight required target hidden-state taps, and runs the packed INTERNARY LM head. It matched an independent greedy target for all eight token IDs in the smoke test. The accepted draft-proposal counts were 1, 1, 0, and 2 across four blocks. Existing paged convolution and GatedDelta state tables successfully checkpointed every verification token, allowing the accepted state to commit without rerunning the target. Low acceptance and draft overhead still left the prototype slower than ordinary decode. Native GenAI integration and low-bit draft adaptation remain development work and are not shipped in this package.

Benchmarks

These are single-card measurements from a cleared 32 GB Arc Pro B70. Prefix caching was disabled. Every shape was warmed before measurement.

Input tokens	Prompt tok/s	Decode tok/s
128	927.002	22.572
512	1,376.576	22.539
966	555.869-564.525	22.452-22.550
2,048	1,476.592	22.288
4,096	1,471.294	22.167
6,144	1,450.742	22.030

The best measured prompt result was 1,476.592 tok/s at 2,048 input tokens. The 966-token shape still drops to roughly 556 to 565 prompt tok/s.

At 2K through 6K, INTERNARY prefill was about 10 to 11 percent faster than the published INTBIT model on the same B70. Decode was slower: about 22 tok/s versus roughly 33 tok/s for INTBIT. The current W2A16 decode kernel is the main optimization target.

The native signed-u2 prefill operator measured 214.0 to 243.4 dense-equivalent TOPS across the primary projection shapes. That is an isolated operator metric, not end-to-end model throughput or physical FP TFLOPS.

Full conditions, per-run values, comparisons, and exclusions are in BENCHMARKS.md. Kernel and graph history are in KERNEL_HISTORY.md.

Correctness

• Random real ternary {-1,0,+1} weights passed the standalone operator oracle.
• Zero codes were explicitly tested.
• Decode rows 1 and prefill rows 33 both had cosine similarity above 0.99999998.
• Two independent model loads produced the same fixed 128-token greedy output.
• Fixed sequence SHA-256: 471f9237a017624ddfb36c3a007e11d49f746651a3ccec23e22e719a379b2241.
• A 6,748-token retrieval test found all three hidden facts.

Known Limitations

• The custom OpenVINO fork and plugin are mandatory.
• Only Linux on Arc Pro B70 has been tested.
• Windows is untested and no Windows DLL is included.
• Vision input has not been tested for this build.
• The 966-token prompt-processing cliff remains unresolved.
• Decode needs another W2A16 kernel sweep.
• The raw 6,748-token test retrieved the facts but spent its output cap on visible reasoning. The packaged runtime now hides completed reasoning and provides configurable headroom.
• The automated Skyrim test had six likely truncations and one exact-memory failure. It had no reasoning leak, AI-identity leak, Hangul fragments, repetition failure, or empty response.
• Validated context performance extends through 6,748 input tokens. This release does not claim the architecture's larger configured maximum.
• This path currently targets single-stream serving.

Checksums

File	SHA-256
openvino_language_model.bin	83e401c5b1433c8dbe6cc6048f868f3b1a346c8f6a4e71769e311c1488cc5706
openvino_language_model.xml	7e4681b7e644bdb67e85497e21a54ad1b9fba81b810be3cfa361323fa2503abc
custom_layers_u2_27b_projection_groups_v1.xml	58b5c5dd8c0044aff7ee12a898ff1e46ec3dc165ac13e183282119b6a8585430
quant_pack_a8_pipeline_v1.cl	adc4224e7dec460113d2a35e82a14e77094e4332352966e34d6ec5021469a078
merge_ternary_s2_pipeline_v1.cl	6b3a738dc8d717fdd490b382c6e1c799eca30be0b7bdd92c6dc7094882c26a89
Linux x86-64 plugin	fbf51b2fd61df7d3e6229b787ef97963164ae45da7e9509fe629fde2c1829cc3

Run sha256sum -c CHECKSUMS.sha256 for the complete file list.

Attribution

The model weights and architecture come from Prism ML's Apache 2.0 Bonsai 27B release. This repository adds the signed-u2 OpenVINO graph conversion, Intel GPU kernels, projection-group plugin path, benchmarks, and runtime tools.

• Prism ML Bonsai 27B
• Prism ML Bonsai project
• OpenVINO fork

When reporting results, include the OpenVINO commit, plugin checksum, Arc model, driver, prompt shape, and whether the card was cleared.