h2loop-ai/gemma-4-e2b-hexagon

Gemma-4-E2B-it — 8-bit and 4-bit quantization for the Qualcomm Hexagon NPU

google/gemma-4-E2B-it quantized two ways, both running on the Hexagon NPU:

• 8-bit quant — int8 weights with int16 activations, compiled to QNN context binaries that run 100% on the Hexagon NPU with no CPU or GPU fallback. Needs the QAIRT SDK.
• 4-bit quant — q4_0 group-32 weights as a GGUF executed on the NPU through llama.cpp's ggml-hexagon backend. No QAIRT SDK, smaller on disk, and faster on decode.

Most shipped mobile builds of this model use int8 activations. Keeping activations at 16 bits costs some memory and holds onto accuracy: on a held-out 400-question MMLU slice the 8-bit build is statistically indistinguishable from the unquantized model.

Read the verification table before using a binary. Not every target below has been run on physical hardware, and this README states exactly which have. As of 2026-07-27 the 8-bit prefill binary is verified on both v79 and v81 silicon; v81 decode is not verified.

---

8-bit — verification status per target

Everything below was measured on physical hardware — Snapdragon 8 Elite (SM8750, HTP v79) via Qualcomm Device Cloud, device 41c5710f, 2026-07-26, and Snapdragon 8 Elite Gen 5 (SM8850, HTP v81) via Qualcomm AI Hub inference, 2026-07-27. Each table says which.

v79 (hexagon-v79 / SM8750 / Snapdragon 8 Elite) — verified

binary	role	on hardware	perf
gemma4_decode_wgqa_int8kv_a16w8_v79.bin	decode (recommended)	✅ exact float match	65.3 ms/step · 15.3 tok/s
gemma4_decode_wgqa_a16w8_v79.bin	decode (no int8-KV)	✅ exact float match, deterministic	69.8 ms/step · 14.3 tok/s
gemma4_trunk_a16w8_v79.bin	prefill, fixed SEQ=128	✅ 12/12 next-token, cos 0.992	—

Decode — the generated text is token-for-token identical to the float ONNX reference:

prompt : "The capital of France is"   (Gemma-4 chat template)
device : 'The capital of France is **Paris**.'
float  : 'The capital of France is **Paris**.'

The/ capital/ of/ France/ is/ **/Paris/**., terminating correctly on <turn|>. Two consecutive runs produced byte-identical output, so decoding is deterministic on device.

Prefill — 12 held-out chat prompts, each one forward pass, compared against float: 12/12 (100%) next-token top-1, hidden cosine mean 0.992 / min 0.986.

Speed ladder (v79, 8-bit, AI Hub profile on real hardware)

decode graph	ms/step	tok/s	speedup
naive full-KV	307.9	3.25	1.0×
+ windowed KV + broadcast-GQA	69.8	14.3	4.41×
+ int8-KV (full-attention slots)	65.3	15.3	4.71×

int8-KV cost no accuracy on the held-out check (32/32 content-token agreement, unchanged).

v81 (hexagon-v81 / SM8850 / 8 Elite Gen 5) — prefill verified, decode still untested

Measured on real Snapdragon 8 Elite Gen 5 silicon via AI Hub inference, 2026-07-27.

binary	role	on hardware	perf
gemma4_trunk_a16w8_v81.bin	prefill, fixed SEQ=128	✅ 11/12 next-token, cos mean 0.991 / min 0.981	—
gemma4_decode_wgqa_int8kv_a16w8_v81.bin	decode	⚠️ still unverified — see below	profiles at 57.4 ms/step (17.4 tok/s)

Prefill on v81 matches v79 bit-for-bit. The 12 held-out prompts were run through the v81 trunk binary, and the same harness was run against the v79 trunk binary as a control, so the two are directly comparable rather than being compared across measurement paths:

trunk binary	device	prompts	next-token top-1	hidden cos (mean / min)
gemma4_trunk_a16w8_v81.bin	SM8850 (v81), AI Hub	12	11/12	0.99077 / 0.98050
gemma4_trunk_a16w8_v79.bin	SM8750 (v79), AI Hub	2 (control)	1/2 — same prompt flips, same cosines	0.99234, 0.98050
gemma4_trunk_a16w8_v79.bin	SM8750 (v79), adb / Device Cloud	12	12/12	0.992 / 0.986

The v79 rows are a control on the measurement path, not a second verification: the AI Hub row deliberately re-ran only the harness-validation prompt and the one prompt v81 flipped.

The single v81 disagreement is "Name a planet with rings.", where float opens '**' and the device opens 'The' at cos 0.9805 — a near-tie between two plausible sentence openings, not a degradation. Re-running that exact prompt on the v79 binary reproduced the identical mismatch at an identical cosine of 0.9804982542991638, so it is a property of the 8-bit quantization, not of v81. On this evidence the two architectures are numerically indistinguishable on the trunk.

Decode on v81 remains unverified. An earlier spot-check of the v81 decode binary showed hidden cosine degrading 0.912 → 0.848 → 0.794 → 0.839 across prefill steps 0–3, with the hardware norm about half of float at step 2, and was stopped before the token comparison. That result has not been explained or reproduced, and the trunk result above does not clear it: the trunk is a different graph. Two candidate explanations remain open — the decode harness itself, and the fact that the v81 decode binary was compiled from the int8-KV export (decode_wgqa_A16W8_int8kv_full) whereas the token-exact v79 verification used the plain WGQA export. Treat gemma4_decode_wgqa_int8kv_a16w8_v81.bin as "failed a spot-check, cause unknown" — not as broken, and not as usable.

Why decode is expensive to verify: Qualcomm Device Cloud provisions v79 parts only, so there is no adb-attached v81 device, and AI Hub inference bills one farm job per decode step (~23 jobs for one short sentence). The trunk is stateless — one forward per prompt — which is why prefill could be verified for 12 jobs and decode was not.

If you have v81 hardware, the useful next step is decode: run the loop per How to run and compare against float. If it diverges, re-verify against a v81 build of the non-int8-KV export to isolate whether int8-KV interacts badly with v81, and check that your QAIRT install ships a hexagon-v81 skel matching the compile.

What is not verified

• Long-context / ring-wrap. The windowed graph uses 512-entry ring buffers on the sliding layers. Prompts long enough to wrap the ring (>512 tokens) were not exercised on hardware. Verified prompts were ~14–18 tokens.
• Prefill beyond 128 tokens. The trunk graph is a fixed SEQ=128 window. Longer prompts are not covered by it at all.
• Generative benchmarks (GSM8K etc). Not measured. MMLU is one forward pass per question; multi-step generative reasoning compounds error over hundreds of steps and is untested here.
• Throughput is NPU inference time from an AI Hub profile job, not end-to-end tokens/s. Host-side embedding lookup and lm_head are excluded; a real application adds those, and the net-run harness used for correctness reloads the context each step so its wall-clock (~3.6 s/step) is not a throughput number.

---

8-bit accuracy

Held-out MMLU, 0-shot, chat-formatted, 400 questions disjoint from all calibration data:

	accuracy
base model (float, ≡ HF)	56.75% ± 2.48
8-bit (this build)	59.25% ± 2.46
delta	+2.50 pp
random baseline	25.00%

The +2.50 pp delta is about one standard error — not evidence that quantization improves the model. The correct reading is that 8-bit costs no measurable MMLU accuracy. Note the two models disagree on ~24% of individual questions; they match in aggregate, not per-question.

---

Two things you must get right

1. Use the chat template. The raw completion format makes this instruction-tuned model degenerate. Verified on the unquantized model, so this is not a quantization artifact:

format	output
raw + greedy	' France is France is France is…'
raw + temperature / top-p	byte-identical degeneration
raw + repetition_penalty 1.2	byte-identical degeneration
chat template + plain greedy	'The capital of France is **Paris**.'

Token layout (verified byte-exact against transformers.apply_chat_template):

[2 <bos>, 105 <|turn>, 2364 'user', 107 '\n'] + PROMPT + [106 <turn|>, 107, 105, 4368 'model', 107]

Stop generation on 106 (<turn|>) or 1 (<eos>).

2. Match the mask constant. Attention masks use a finite NEG = -1e4, not -inf or float32.min. -inf cannot survive int16 activation quantization — it blows out the range so real scores round to zero. -1e4 still zeroes the softmax while leaving real scores resolved. The host must use the same value the model was calibrated with.

---

Architecture

The graph is split so the >2 GB vocab tensors never enter it:

• Host (CPU/ARM): token embedding lookup, per-layer embedding lookup, and the tied lm_head with 30·tanh(x/30) softcap.
• NPU: the transformer decode graph, with the KV cache resident on device.

Gemma-4-E2B is dense: 35 layers, hidden 1536, GQA 8 query heads → 1 KV head, head_dim 256, 262144-token vocab, and hybrid attention (28 sliding-window layers of window 512, interleaved with 7 full-attention; KV shared across the last 20 layers, so only 15 layers store KV).

Why the fast graph is fast

Decode is KV-attention-bound, not weight-bound. The published binary uses two changes over a naive full-KV decode graph:

• Windowed KV — sliding-window layers use a 512-entry ring buffer instead of a full 4096 buffer; only the 3 full-attention layers keep 4096. KV traffic per step drops from ~288 MB to ~63 MB. The ring index is computed inside the graph as cache_position % buf, so the host just passes pos.
• Broadcast GQA — the expand op that materialized 1 KV head into 8 copies is removed (verified: 0 Expand nodes in the exported ONNX).

Net effect on v79: 307.9 ms → 69.8 ms per decode step (4.41×).

Credit: these two levers come from the tps/ work in gemma-4-e2b-hexagon-npu — this repo contributes a corrected quantization of that graph.

---

Files

gemma4_decode_wgqa_int8kv_a16w8_v79.bin  1.9 GB  8-bit decode, v79  (recommended)
gemma4_decode_wgqa_a16w8_v79.bin         1.9 GB  8-bit decode, v79, no int8-KV
gemma4_trunk_a16w8_v79.bin               1.9 GB  8-bit prefill, v79, fixed SEQ=128
gemma4_decode_wgqa_int8kv_a16w8_v81.bin  1.9 GB  8-bit decode, v81   [UNVERIFIED - see above]
gemma4_trunk_a16w8_v81.bin               1.9 GB  8-bit prefill, v81  (verified on v81 silicon)

gemma4-e2b-w4.gguf                       2.6 GB  4-bit, runs via ggml-hexagon
gemma4-e2b-w4-mtp.gguf                    57 MB  4-bit MTP drafter (speculative decoding)

llama.cpp/bin/{llama-cli,llama-server,llama-bench}   arm64 Android, the 4-bit runtime
llama.cpp/lib/*.so                        22 MB  11 arm64 libs + libggml-htp-v79/v81.so
llama.cpp/LICENSE                                MIT (llama.cpp 0ef6e55) + DSP-library note

host-model/embed_tokens_weight.bf16            769 MB  token embeddings
host-model/embed_tokens_per_layer_weight.bf16  4.4 GB  per-layer embeddings
host-model/tokenizer.json                       31 MB
host-model/norm_weight.bf16                              final norm (diagnostics)

runtime/hostlib.py               host embeddings, chat template, lm_head + softcap
runtime/run_gate.py              host orchestrator (the autoregressive loop)
runtime/verify_trunk.py          prefill checker vs a float reference
runtime/stage_device.sh          push everything to an adb device
runtime/gate_ondevice_wgqa.sh    on-device decode step + KV rotation
runtime/gate_ondevice_int8kv.sh  same, int8-KV binary
runtime/gate_ondevice_trunk.sh   on-device prefill pass (no KV)
requirements.txt

The three host-model tensors are ~5.2 GB and stay on the host by design — putting the 262144-token vocab in the graph blows past ONNX's 2 GB protobuf limit.

---

How to run the 8-bit build

What you need that is NOT in this repo

You cannot run this from this repo alone. One dependency is missing by necessity:

1. Qualcomm AI Engine Direct (QAIRT / QNN) SDK — supplies qnn-net-run and libQnnHtp*.so plus the HTP Stub/Skel pair for your Hexagon version. These are Qualcomm-licensed and not redistributable here, so you must install the SDK yourself (free, from Qualcomm). Built and tested against QAIRT 2.45.
   • v79 needs libQnnHtpV79Stub.so + libQnnHtpV79.so / libQnnHtpV79Skel.so
   • v81 needs the V81 equivalents. Check your SDK actually ships hexagon-v81; older installs do not.
2. A device: Snapdragon 8 Elite (SM8750, v79) or 8 Elite Gen 5 (SM8850, v81), reachable over adb. Qualcomm Device Cloud works — that is what this was verified on.
3. Host Python 3.9+ with numpy and tokenizers (pip install -r requirements.txt). No torch, no transformers needed to run — only to reproduce the quantization.
4. ~6 GB free on the host for the embedding tensors, ~4 GB free on the device (/data/local/tmp) per pair of context binaries.

Step 1 — get the repo

pip install -r requirements.txt
git lfs install
git clone https://huggingface.co/h2loop-ai/gemma-4-e2b-hexagon
cd gemma-4-e2b-hexagon

Step 2 — connect the device

adb devices -l          # confirm your serial

On Qualcomm Device Cloud, tunnel the adb server first, then point adb at it:

ssh -i <your-qdc-key>.pem -L 5037:<QDC_HOST>:5037 -N sshtunnel@ssh.qdc.qualcomm.com &
export ADB_SERVER_SOCKET=tcp:127.0.0.1:5037
adb devices -l

Never run adb kill-server against that tunnel — it kills the remote pod's adb server, which you cannot restart without portal access.

Step 3 — stage onto the device

export QAIRT_DIR=/path/to/qairt/2.45.0.xxxxxx     # your SDK install
./runtime/stage_device.sh <serial> v79             # or: v81

This pushes qnn-net-run, the HTP libs/skels, the matching *_v79.bin context binaries, and the on-device step scripts. Two 1.9 GB pushes over adb take a while; over a QDC tunnel a single stream runs ~1 MB/s, so expect ~30 min unless you parallelise (see Slow adb below).

KV buffers are not pushed — run_gate.py creates them on device with dd.

Step 4 — run the autoregressive loop

python runtime/run_gate.py \
  --prompt "The capital of France is" \
  --ntokens 14 \
  --adb-serial <serial> \
  --chat --wgqa \
  --script gate_ondevice_int8kv.sh

Expected output:

continuation: 'The capital of France is **Paris**.'

Flags that matter:

flag	why
--chat	required. Without it the -it model degenerates into ' France is France is …'
--wgqa	required for these binaries — selects 512-entry ring buffers and the 512-wide sliding mask
--script	pick the binary: gate_ondevice_int8kv.sh (recommended) or gate_ondevice_wgqa.sh

Drop --script to use the non-int8-KV binary.

Step 5 (optional) — check prefill

verify_trunk.py compares the trunk against a float reference. Producing that reference needs torch + transformers on a host that knows the gemma4 architecture (transformers ≥ 5.12 — older versions raise KeyError: 'gemma4'), so it is a reproduction step rather than part of normal use.

---

4-bit — q4_0 weights on the Hexagon NPU

The 4-bit path uses a different runtime. It is not a QNN context binary: it is a GGUF executed on the Hexagon NPU through llama.cpp's ggml-hexagon backend, which drives the DSP directly over FastRPC instead of through QNN. It needs no QAIRT SDK.

On the same physical v79 silicon it is 2.5× faster on decode than the 8-bit build, fits entirely in one 2.44 GiB file with no host-side tensors, and costs ~0.4 pp of MMLU:

	4-bit / ggml-hexagon	8-bit / QNN
model on disk	2.44 GiB total	3.54 GiB + 5.2 GB host tensors
decode, sustained on device	37.8 – 38.5 tok/s	15.3 tok/s
prefill, NPU	1145 tok/s	—
peak device RSS	3.84 GiB	not measured
held-out MMLU	57.76% ±0.99	59.25% ±2.46
vs unquantized base	−0.44 pp	+2.50 pp
GSM8K (5-shot, chat)	0.5200 / 0.6467	not measured
embeddings	inside the file	5.2 GB on host
QAIRT SDK required	no	yes

Granularity, not bit-width, is what decides accuracy here. This build is group-32; at the same 4 bits a per-channel grid costs ~5.3 pp of MMLU, measured by quantizing the same parent checkpoint both ways on the same harness:

grid, same parent checkpoint	MMLU (n=2280)
unquantized parent	57.68% ±0.99
group-32 (this build)	57.76% ±0.99
per-channel int4	52.46% ±1.01

Group-wise 4-bit was previously ruled out on this graph because LPBQ stores 4-bit values in an int8 container and runs 9.3× slower. That conclusion still holds for QNN — it does not apply here, because ggml-hexagon consumes q4_0 blocks natively with its own HTP kernels rather than going through QNN's blockwise path.

4-bit verification status

Measured on physical Snapdragon 8 Elite (SM8750, HTP v79) via Qualcomm Device Cloud, device 87b3a4aa, 2026-08-05. Every run started from 42 °C.

Throughput

dev	prefill pp128	decode tg32
HTP0 (NPU)	1145.39 ± 7.89 tok/s	28.50 ± 0.06 tok/s
CPU (6 threads)	225.17 ± 0.46 tok/s	42.48 ± 0.08 tok/s

NPU prefill is 5.1× CPU; CPU decode is 1.5× NPU. Decode is weight-bandwidth-bound, so the NPU's compute advantage does not help it — prefill is compute-bound, where it does. A hybrid placement (NPU prefill, CPU decode) is the strongest configuration on this hardware.

Sustained decode — and a measurement discrepancy you should know about

Five consecutive 128-token requests through llama-server on HTP0:

run	tok/s	peak RSS	hottest thermal zone
1	38.47	4,029,912 kB	79.2 °C
2	38.27	4,031,796 kB	83.4 °C
3	38.13	4,031,796 kB	83.8 °C
4	38.06	4,031,796 kB	86.1 °C
5	37.79	4,031,796 kB	87.2 °C

Latency and memory

• TTFT 61.19 ms for an 8-token prompt. Not comparable to context-512 TTFT figures.
• Peak RSS 3.84 GiB (VmHWM 4,031,796 kB) for a 2.44 GiB file — ~1.4 GiB of host-side buffers on top of the weights.
• The ggml-hex log reports the DSP session mapping budget as vmem 3355443200 = 3.125 GiB. RSS exceeds it, so not everything is resident in the session. A model whose weights alone approach 3.1 GiB will need multi-device layer splitting.

Thermals

42 °C → 87.2 °C peak across 5×128-token bursts, with 1.8% decode decay. Hot but not throttling within that window. A multi-minute soak has not been run — do not assume this holds for continuous generation.

Generation quality on device

Greedy, raw completion (no chat template):

prompt	output
"The capital of France is"	" Paris."
"Who wrote Hamlet?"	"The answer is William Shakespeare."
"What color is grass?"	(empty)
"Name a country in South America."	"Brazil"

The empty response on "What color is grass?" is a prompt artifact, not a quantization defect: raw completion emits an immediate end-of-turn, and Google's own q4_0 GGUF returns the identical empty string on that prompt. Use the chat template.

Accuracy — measured on CPU, not on device

benchmark	4-bit	parent (unquantized)	base fp32
MMLU, 0-shot, n=2280	57.76% ±0.99	57.68% ±0.99	58.20% ±0.99
GSM8K 5-shot chat, strict	0.5200 ±0.041	0.5200 ±0.041	—
GSM8K 5-shot chat, flexible	0.6467 ±0.039	0.6467 ±0.039	—
top-1 agreement vs parent	98.2%	—	—
KL divergence vs parent	0.00173	—	—

GSM8K is identical to the unquantized parent to the digit, standard errors included. MMLU is +0.08 pp — noise. Token-level top-1 agreement is 98.2% and KL is 1.7×10⁻³, so the two models are not bit-identical per token; they match on the benchmarks in aggregate.

How this was measured, and its limit: the GGUF was dequantized and its tensors substituted into the parent architecture, then evaluated with lm-evaluation-harness on CPU. That measures the weight grid, which is what quantization changes — it does not exercise the HTP kernels. On-device accuracy at scale is unverified; the four coherence prompts above are the only device-side quality evidence.

Two harness details that otherwise waste time: MMLU needs tokenizer.add_bos_token=True or loglikelihood-MC scores at chance; GSM8K needs apply_chat_template=True and add_bos_token left off, because the template already emits <bos> and forcing it double-prefixes. GSM8K run 0-shot raw scores strict-match = 0.0 — the model never emits the #### N format, so that is a broken measurement, not a bad model.

What is not verified

• v81. libggml-htp-v81.so builds and is auto-selected by arch, but has never been run — Device Cloud provisions v79 only. Qualcomm's published figures for a q4_0 GGUF on v81 show NPU decode lower than v79 (17.61 vs 26.66 tok/s at ctx 512) with prefill 36% higher, so do not assume v81 is an upgrade for decode.
• Long context. Nothing beyond depth 4096 exercised. The tg32 @d4096 figure was itself unstable across sessions (21.90 ±0.97 vs 27.47 previously), so it is not quoted here.
• Sustained multi-minute load, batch > 1, and on-device accuracy at scale.

How to run the 4-bit build

What you need that is not in this repo

Only a device — no QAIRT/QNN SDK, and no build step. The llama.cpp/ directory in this repo is a prebuilt arm64-Android runtime, so unlike the 8-bit path this one is self-contained:

1. A device: SM8750 (v79) or SM8850 (v81) over adb.
2. ~3 GB free on the device, ~4 GB RAM headroom for the process.

What is in llama.cpp/

llama.cpp/bin/llama-cli  llama-server  llama-bench     arm64 Android, the three tools below
llama.cpp/lib/*.so                             22 MB   11 arm64 libs + the v79/v81 DSP libs
llama.cpp/LICENSE                                      MIT, plus the DSP-library note

Built from upstream llama.cpp 0ef6e55 in ghcr.io/snapdragon-toolchain/arm64-android:v0.7 (NDK r28b, Hexagon SDK 6.6). Hexagon support and the gemma4 architecture both need a build from 2026-06 or later, so an older llama.cpp will not work. The commit is recorded inside libllama-common.so if you want to check what you got.

The arm64 libraries are stripped of debug info (175 MB → 20 MB); every PT_LOAD segment and every SHF_ALLOC section is byte-identical to the unstripped build, and the DSP libraries are shipped as built. This is the exact dependency closure of the three tools — nothing spare, and nothing missing.

libggml-htp-v79.so / -v81.so are the DSP-side kernels; the right one is selected automatically from the detected arch, so unlike QNN there is no version pinning to get right. They need no Qualcomm libraries at runtime — their qurt_*, compute_resource_* and dspqueue_* symbols resolve on-device against the vendor DSP image. That is also why ADSP_LIBRARY_PATH must include the vendor paths (Step 2). If you want v73/v75 as well, or a build without the unused libggml-opencl.so dependency, build it yourself:

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cp docs/backend/snapdragon/CMakeUserPresets.json .

docker run --rm -u $(id -u):$(id -g) -v $(pwd):/workspace --platform linux/amd64 \
  ghcr.io/snapdragon-toolchain/arm64-android:v0.7 \
  bash -lc 'cd /workspace \
    && cmake --preset arm64-android-snapdragon-release -B build-snapdragon \
    && cmake --build build-snapdragon -j 32 \
    && cmake --install build-snapdragon --prefix pkg-snapdragon/llama.cpp'

Step 1 — fetch just the 4-bit files, and push

A full clone of this repo is ~17 GB, almost all of it the 8-bit binaries and their host tensors. The 4-bit path needs 2.7 GB of it, so fetch only that:

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/h2loop-ai/gemma-4-e2b-hexagon
cd gemma-4-e2b-hexagon
git lfs pull --include="gemma4-e2b-w4*.gguf,llama.cpp/lib/*"

adb push llama.cpp /data/local/tmp/
adb shell chmod +x /data/local/tmp/llama.cpp/bin/llama-cli \
                   /data/local/tmp/llama.cpp/bin/llama-server \
                   /data/local/tmp/llama.cpp/bin/llama-bench
adb shell mkdir -p /data/local/tmp/gguf
adb push gemma4-e2b-w4.gguf /data/local/tmp/gguf/

adb push does not preserve the executable bit, hence the chmod. Over a QDC tunnel expect 2.6–7.6 MB/s, so 10–20 min for the model — the runtime itself is 22 MB and lands in seconds. The Slow adb section below applies to the model.

Step 2 — the one non-obvious requirement

ADSP_LIBRARY_PATH is mandatory. Without it the DSP loader cannot find the skel, and the error looks like a hardware or arch problem but is not:

ggml-hex: failed to open session 0 : error 0x80000406

adb shell
cd /data/local/tmp/llama.cpp
export LD_LIBRARY_PATH=/data/local/tmp/llama.cpp/lib
export ADSP_LIBRARY_PATH="/data/local/tmp/llama.cpp/lib;/system/lib/rfsa/adsp;/vendor/lib/rfsa/adsp"

A healthy session logs:

ggml-hex: Hexagon Arch version v79
ggml-hex: HTP0 hwinfo: threads 6, hvx 6, hmx 1, vtcm 8 MB
ggml-hex: HTP0 new session : ... file:///libggml-htp-v79.so ... _dom=cdsp

Step 3 — run

Benchmark NPU and CPU in one invocation:

./bin/llama-bench -m /data/local/tmp/gguf/gemma4-e2b-w4.gguf \
  -p 128 -n 32 -t 6 -r 2 -fa 1 -dev HTP0,none

Interactive generation on the NPU:

./bin/llama-cli -m /data/local/tmp/gguf/gemma4-e2b-w4.gguf -dev HTP0 -t 6 -c 4096 -fa on

Server — the path the quoted decode figures come from:

./bin/llama-server -m /data/local/tmp/gguf/gemma4-e2b-w4.gguf \
  -dev HTP0 -t 6 -c 4096 -fa on --parallel 1 --no-cont-batching --port 8099

curl -s localhost:8099/completion -H 'Content-Type: application/json' \
  -d '{"prompt":"Write a paragraph about on-device inference.","n_predict":128,"temperature":0}'

The timings block carries predicted_per_second, prompt_ms (TTFT) and, with a drafter loaded, draft_n / draft_n_accepted.

flag	why
-dev HTP0	run on the NPU. -dev none = CPU. -dev HTP0,none benchmarks both
-fa on	flash attention; required for the windowed-KV path
-t 6	matches the HTP's 6 HVX contexts
--parallel 1 --no-cont-batching	single-stream, matching the quoted figures
-c 4096	context; nothing beyond this is verified

Quantized KV (-ctk q8_0 -ctv q8_0) is not supported on this backend — it aborts with GGML_ASSERT(offset == 0) in ggml-hexagon.cpp. On CPU it works but is slower than f16 (25.66 vs 34.38 tok/s at depth 4096).

Step 4 — speculative decoding with the MTP drafter

gemma4-e2b-w4-mtp.gguf is a Multi-Token Prediction drafter (~57 MB) that shares the target's KV cache. It needs a llama.cpp build newer than 2026-06-08 (arch gemma4-assistant).

adb push gemma4-e2b-w4-mtp.gguf /data/local/tmp/gguf/

./bin/llama-server -m /data/local/tmp/gguf/gemma4-e2b-w4.gguf \
  -md /data/local/tmp/gguf/gemma4-e2b-w4-mtp.gguf \
  --spec-type draft-mtp --spec-draft-n-max 4 \
  -dev HTP0 -t 6 -c 4096 -fa on --port 8099

The drafter cannot be instantiated standalone (failed to create context with model — it requires the target's context), and the startup warning [spec] failed to measure draft model memory is benign; the drafter loads afterwards and common_speculative_init_result confirms it.

4-bit troubleshooting

symptom	cause
Permission denied running ./bin/llama-cli	adb push drops the executable bit; chmod +x the three tools (Step 1)
library "libllama-common.so" not found	LD_LIBRARY_PATH not pointing at llama.cpp/lib (Step 2)
failed to open session 0 : error 0x80000406	ADSP_LIBRARY_PATH not set — the DSP cannot find the skel
GGML_ASSERT(offset == 0) in ggml-hexagon.cpp	quantized KV cache (-ctk/-ctv) unsupported on HTP
failed to create context with model 'gemma4-e2b-w4-mtp.gguf'	the drafter needs the target's context; do not load it alone
503 {"error":"Loading model"}	server not ready. /health returns 503 while loading, so poll for HTTP 200, not just any response
decode slower on NPU than CPU	expected — decode is bandwidth-bound. Use the NPU for prefill
llama-cli idles at a > prompt	it entered conversation mode; use llama-server, or check this build's single-turn flag
arch gemma4 / gemma4-assistant unknown	llama.cpp too old

4-bit limitations

• Decode figure is unsettled — 28.50 (llama-bench) vs ~38 tok/s (llama-server), cause unresolved. See above.
• On-device accuracy is unverified at scale. MMLU/GSM8K were measured on CPU against the weight grid, not through the HTP kernels.
• v81 has never been run, and Qualcomm's own v81 figures suggest decode is worse there.
• No compiled artifact. Unlike the 8-bit context binaries, there is no AOT compile step — ggml-hexagon reads the GGUF at runtime. Converting this to a QNN context binary is not a shortcut: QAIRT's gguf_builder dequantizes rather than preserving the group-32 grid, which is the property doing the work.
• Quantized KV cache unsupported on the HTP backend.
• Beyond 4096 context, batch > 1, and multi-minute sustained load are all untested.

---

How the loop actually works

run_gate.py is the reference implementation, and deliberately simple:

1. Tokenize with the chat template (hostlib.encode_chat).
2. Zero the KV buffers on device.
3. For each position: look up embeddings on the host, write the six small per-step tensors, run qnn-net-run once on device, pull back hidden (1536 floats).
4. Apply the tied lm_head + 30·tanh(x/30) softcap on the host, take the argmax.
5. The on-device script renames present_* → past_* so KV never crosses adb.
6. Stop on <turn|> (106) or <eos> (1).

This harness is for correctness, not speed. It re-loads the 1.9 GB context binary every step, so its wall clock (~3.6 s/token) is ~50× worse than the NPU's actual 65 ms. A real application loads the context once, keeps KV device-resident, and does embeddings + lm_head in-process. Building that is left to you.

Slow adb

A single adb stream over a QDC tunnel is bandwidth-delay-product limited (~1 MB/s), not bandwidth limited. Splitting the binary and pushing chunks over separate SSH tunnels (one local port each) reached ~7 MB/s:

split -n 6 -d gemma4_decode_wgqa_int8kv_a16w8_v79.bin chunk.
# ...one `ssh -L 503X:$HOST:5037` per chunk, then push each with its own
#    ADB_SERVER_SOCKET=tcp:127.0.0.1:503X, then on device:
adb shell 'cd /data/local/tmp/gemma/artifacts && cat chunk.* > out.bin && rm chunk.*'

Verify the checksum afterwards (SHA256SUMS) — and wait for all pushes to finish before concatenating, or you will silently assemble a truncated file.

Troubleshooting

symptom	cause
Could not create context from binary	HTP arch mismatch — a v81 binary will not load on a v79 device, or vice versa
Cannot assign data from unexpected type. Expected int32, got int64	binaries are built with --truncate_64bit_io, so position_ids/cache_position are int32
Output repeats ' France is France is …'	--chat missing
Fluent but wrong answer	binary/mask mismatch — the host NEG must be -1e4, matching calibration
Garbage tokens, hidden norm ≈ 0	wrong context binary, or KV buffers not zeroed before position 0
Different output across identical runs	a KV buffer was corrupted mid-push; re-seed (dd on device) and retry
Fluent garbage from a binary you compiled yourself	local qnn-context-binary-generator emits UFIXED_POINT_16 IO; the harness writes fp32. Use the AI Hub build, or check graphInputs[].dataType
Could not create context from binary on a binary that used to work	QAIRT runtime version does not match the version that compiled it

Reproducing the 8-bit quantization

AIMET QuantizationSimModel, param_type=int8, activation_type=int16, quant_scheme=min_max, calibrated on real activations captured from chat-formatted decode loops — not random noise, and not raw-format text.

Both of those details matter and each caused a distinct on-device failure:

• Calibrating on np.random.randn produced a binary whose residual stream collapsed to zero on hardware (final hidden norm 0.0000 → pure noise tokens), even though it compiled fine.
• Calibrating on raw-format text and then running chat-format prompts produced fluent but unfaithful output on hardware — the chat template's special tokens hit activation ranges the quantizer never observed.

min_max outperformed tf_enhanced here: on int16 there are 65k levels, so range coverage matters more than outlier clipping, and tf_enhanced mis-estimated the range badly enough to inflate hidden norms ~10×.

8-bit limitations

• Fixed 4096 context.
• Prefill is not included in this repo; the decode graph can prefill token-by-token, which is slow for long prompts.
• Batch size 1 only.
• --truncate_64bit_io at compile time means index inputs (position_ids, cache_position) are int32 on the compiled binary, though the float ONNX takes int64.