--- language: - en license: apache-2.0 pipeline_tag: text-classification inference: false tags: - rknn - rknn3 - rk1828 - rk1820 - rockchip - npu - edge-ai - kev - decision - text-classification - w4a16 base_model: Qwen/Qwen3.5-0.8B-Base base_model_relation: finetune model-index: - name: Kev-0.8B-RKNN3-RK1828 results: - task: type: text-classification name: Structured decision (choice / noul / score) dataset: type: kev-decision-v2 name: Kev decision-v2 (60 held-out) split: dev metrics: - type: accuracy value: 0.817 name: Accuracy (device) - task: type: text-classification name: Structured decision (choice / noul / score) dataset: type: kev-decision-v2 name: Kev decision-v2 (26 mixed-type, end-to-end HTTP) split: dev metrics: - type: accuracy value: 0.885 name: Accuracy (HTTP service) --- # Kev-0.8B-RKNN3-RK1828 **Kev-0.8B (decision model) converted to RKNN3 and running on the Rockchip RK1828 NPU — with zero accuracy loss versus the original fp32 model.** Kev is *not* a chat model. It runs a single prefill and scores options with a pointer head; it never generates tokens. You can verify this yourself: ask it `1+1`, and you get nonsense. That is the model behaving correctly. This repository contains the converted artifacts plus the minimal host-side code needed to actually use them, because a plain `rkllm3-server` deployment **cannot** run Kev (see [Why this needs host code](#why-this-needs-host-code)). | | | | --- | --- | | Base model | [`Qwen/Qwen3.5-0.8B-Base`](https://huggingface.co/Qwen/Qwen3.5-0.8B-Base) @ `dc7cdfe` | | Adapter | [`jaredpalmer/kev-0.8b`](https://huggingface.co/jaredpalmer/kev-0.8b) (LoRA r16, merged into base) | | Target hardware | RK1828 NPU (RK1820/RK1828 AI SDK), host = Windows PC / RK3588 / RK3576 | | Toolchain | RKNN3 **v1.1.0** (`runtime/rknn3_110`), `target_platform='rk1820'` | | Quantization | `w4a16`, `quantized_algorithm='normal'`, `quantized_method='group32'` (weight-only, no calibration set) | | NPU footprint | **802 MB** (as reported by `rknn-smi`) | | Latency | ~0.9 s per decision end-to-end, ~75 ms per prefill (~100 tokens) | --- ## Files | File | Size | What it is | | --- | --- | --- | | `rknn/Kev-0.8B.rknn` | 20.6 MiB | RKNN3 graph. Output is `[1, 1, 248320]` fp16 logits | | `rknn/Kev-0.8B.weight` | 498 MiB | Quantized backbone weights | | `rknn/Kev-0.8B.embed.bin` | 485 MiB | Token embedding table, `248320 × 1024` fp16 (also the dequantization matrix `W`) | | `rknn/Kev-0.8B.tokenizer.gguf` | 10.4 MiB | Tokenizer. Identical to Qwen3.5-0.8B — LoRA does not touch the vocab | | `rknn/start.cmd` | 1.4 KiB | `rkllm3-server` launch line (for the "text-only" sanity check) | | `host/kev_daemon.cpp` | — | Device-side runner: loads the model once, then takes `TOK`/`R`/`A` commands and dumps logits per request | | `host/kev_serve.py` | — | HTTP service exposing `POST /v1/systemone` | | `host/serve_assets/*.npy` | 6.1 MiB | `Ginv.npy` (1024×1024), `head_q.npy`, `head_k.npy`, `temperature.npy` — precomputed, so you do not have to invert a 1024×1024 normal-equation matrix at startup | | `host/e2e_check.py` | — | End-to-end accuracy check against the decision-v2 labels | | `docs/DEPLOY.md` | — | Step-by-step deployment, including the traps | Total: ~1.06 GB. ### Checksums ``` 776f782272076d2579eb5d5e59aee733ea8c10bd2994013ed791d05c449ff3f0 Kev-0.8B.rknn 0fcc3e04de7a645e166c11b1b98e769df8a145767d1979be54ba34578a0d6dce Kev-0.8B.weight 2146dc176cff21240562283cf0b709a91499917d5b5f2c4fb0b7b41b21d407b0 Kev-0.8B.embed.bin f2235450332c615492e7cfa13b103090a9adc7a802bef5d425842a87f8b6b03f Kev-0.8B.tokenizer.gguf ``` ```bash sha256sum -c <<'EOF' 776f782272076d2579eb5d5e59aee733ea8c10bd2994013ed791d05c449ff3f0 Kev-0.8B.rknn 0fcc3e04de7a645e166c11b1b98e769df8a145767d1979be54ba34578a0d6dce Kev-0.8B.weight 2146dc176cff21240562283cf0b709a91499917d5b5f2c4fb0b7b41b21d407b0 Kev-0.8B.embed.bin f2235450332c615492e7cfa13b103090a9adc7a802bef5d425842a87f8b6b03f Kev-0.8B.tokenizer.gguf EOF ``` --- ## Why this needs host code Kev's scoring path is: ``` h_decide = backbone(state + question + "") # hidden state, 1024-d, *before* lm_head h_opt_k = backbone(state + question + option_k + "") z_k = (k(h_opt_k) · q(h_decide)) / sqrt(256) answer = softmax(z / T) # T = 2.3510958125672174 ``` The runtime exposes **vocab logits**, not hidden states. The usual conclusion is "blocked, needs retraining". It is not blocked, because `logits = h @ Wᵀ` is an **overdetermined** system: ``` W ∈ R^(248320 × 1024), unknown h ∈ R^1024, 248320 equations h = (WᵀW)⁻¹ Wᵀ · logits ``` `W` is the (shared) embedding matrix, so it is already on disk. The per-coordinate quantization noise gets averaged over ~248k equations, i.e. suppressed by roughly √248320 ≈ 500×. Measured cosine similarity between device-recovered `h` and the fp32 reference is **0.96–0.99**, with top-1 agreeing on every sampled position. So the original pointer head runs unmodified on the host. **No retraining, no downgrade to an embedding model.** `Ginv.npy` is the precomputed `(WᵀW)⁻¹Wᵀ`, so recovery is a single `1024×1024 @ 1024` matmul per position. Two consequences worth knowing before you deploy: - **The graph only keeps the last position.** There is no `num_logits_to_keep` in the public API and `clear_kvcache` has no truncate. To read the `` positions you must re-run a truncated sequence. A request with `K` options therefore costs **K+1 prefills** — that is a hard floor, not an implementation choice. - **Per-call fixed overhead is ~50 ms**, independent of length. Latency scales with *number of calls*, not token count. Batching long states into one prefix helps a lot; splitting into many short calls does not. --- ## Usage ### 1. Sanity check (text only) ```bat cd rknn start.cmd ``` Then `POST /v1/chat/completions`. Output will be garbage — this only proves the weights loaded and the NPU is running. **Do not use this to judge whether the conversion worked.** ### 2. Real usage: decision API Build `host/kev_daemon.cpp` against the RKNN3 SDK session demo, run it, then: ```bash python host/kev_serve.py # listens on :17890 ``` ```bash curl -X POST http://127.0.0.1:17890/v1/systemone \ -H "Content-Type: application/json" \ -d '{ "state": {"ticket": "Checkout returns 500 for EU customers", "age_hours": 30}, "questions": { "team": {"type": "choice", "instructions": "Which team should own this?", "criteria": {"billing": "payment, invoices, refunds", "platform": "infra, deploys, runtime", "growth": "onboarding, activation"}}, "urgent": {"type": "noul", "instructions": "Is this urgent?"}, "sev": {"type": "score", "instructions": "Rate the severity", "criteria": ["cosmetic", "degraded", "broken", "outage"]} } }' ``` ```json {"answers": { "team": {"type": "choice", "answer": "billing", "confidence": 0.7638, "probabilities": {...}}, "urgent": {"type": "noul", "p_true": 0.912}, "sev": {"type": "score", "score": 2.31, "legend": [...], "score_confidence": 0.54} }} ``` Response schema follows the upstream `kev/api.py::to_answers` conventions: - `noul` returns `p_true` only (the upstream API does not binarize it) - `choice` returns `confidence = (p_max − 1/K) / (1 − 1/K)`, so `K=2` with `p=1.0` gives `confidence=1.0` - `score` returns the probability-weighted expected level as `score`, plus per-level probabilities Full commands and the build line: [`docs/DEPLOY.md`](docs/DEPLOY.md). --- ## Performance Measured on RK1828 (5051.8 MB NPU), Windows host, single model loaded. | Metric | Value | | --- | --- | | Accuracy, decision-v2 dev 60 (fp32 reference, host) | 49/60 = **0.817** | | Accuracy, decision-v2 dev 60 (device, w4a16 + recovered `h`) | 49/60 = **0.817** | | Per-question argmax agreement, device vs fp32 | **58/60 = 96.7%** | | End-to-end HTTP, 26 mixed-type questions | 23/26 = **0.885** | | Prefill throughput (long sequences) | 1200–1500 tok/s | | Single prefill, ~100 tokens | ~75 ms | | Per-call fixed overhead | ~50 ms | | One decision (K=5) | ~0.5 s | | End-to-end HTTP per question | ~0.9 s | | Model load | ~30 s, once | | NPU memory | 802 MB | The two disagreeing questions (q9, q10) cancel out — one is right on device and wrong on fp32, the other the reverse. That is the signature of the *same function* being computed, with numerical noise flipping borderline cases. Coexistence, measured with `rknn-smi` on the same device: | Configuration | NPU used | Delta | | --- | --- | --- | | baseline (proxy only) | 34 MB | — | | Kev-0.8B alone | 802 MB | — | | Kev-0.8B + Qwen3-ASR-0.6B | 2848 MB | +2047 MB | | Kev-0.8B + Qwen3-ASR-1.7B | 2953 MB | +2152 MB | Kev can share the NPU with an ASR model. Both fit; only ~105 MB separates the two ASR sizes. Measure *coexisting* configurations, not standalone ones — a standalone run reports 779 MB for the 0.6B ASR, which will make you over-budget your plan. --- ## Limitations - **Not a chat model.** Text generation output is meaningless by design. Any benchmark that prompts it for answers is invalid. - **Numeric noise flips borderline decisions.** 96.7% per-question agreement means ~3% of hard cases can differ from the fp32 reference. If you need exact reproducibility, run the fp32 model. - **K+1 prefill calls per request** cannot be reduced with this runtime (see above). Very large option lists get slow linearly. - **Single-request serialization.** The service holds a global lock around one decision, because `K+1` prefills share one KV cache; interleaving concurrent requests corrupts it. Throughput is therefore ~1 request at a time. Concurrency is safe (tested), but it queues. - **Host must be Windows / RK3588 / RK3576.** The RK1828 is a co-processor; an x86 Linux box can convert and simulate but cannot drive the module. - **Quantization format is fixed** to `w4a16 + normal + group32` by this toolchain. `w8a16` and `fp16` builds are rejected (`request_type=float16` / `Failed to create initializer tensor`), so you cannot "build a higher-precision version" to isolate quantization error. --- ## Provenance - Base: `Qwen/Qwen3.5-0.8B-Base`, revision `dc7cdfe2ee4154fa7e30f5b51ca41bfa40174e68` - Adapter: `jaredpalmer/kev-0.8b`, LoRA r16 / α32, task type `FEATURE_EXTRACTION` - Merged via `merge_and_unload()`, then exported to ONNX with the official `rknn3-model-zoo/examples/Qwen3_5` pipeline - `h`, the pointer head, and the option/decide token ids are all taken from the upstream Kev implementation unchanged Special token ids used (from upstream `kev/model.py::SPECIAL`): ``` 248060 248061 248049 248050 248062 ``` Artifact sizes match the officially published Qwen3.5-0.8B RKNN3 build file-for-file (20.6 / 498 / 485 / 10.4 MiB), which is the practical proof that the conversion is structurally correct. License: **Apache-2.0**, inherited from the base model. --- ## 中文摘要 Kev-0.8B 是决策模型,不是对话模型:它只跑一次 prefill,用指针头给候选项打分,不生成 token。 用"问它 1+1"来验收是错的——那样一定会得到无意义的回答。 RKNN3 运行时只暴露 vocab logits,而 Kev 需要 hidden state。但 `logits = h @ Wᵀ` 是超定方程 (248320 个方程解 1024 个未知数),`h = (WᵀW)⁻¹Wᵀ·logits` 可以精确解回来, 量化噪声被约 500 倍平均掉。因此**原版指针头可以原样跑在 host 侧,无需重训、无需改结构**。 | 项 | 结果 | | --- | --- | | 真机准确率(60 题) | 0.817,与 fp32 完全一致 | | 逐题一致 | 58/60 | | 端到端 HTTP(26 题三题型) | 0.885,约 0.9 s/题 | | NPU 占用 | 802 MB | 本仓库包含转换产物 + 可复现的 host 侧代码(daemon、HTTP 服务、反解矩阵、头权重), 拿到即可跑通。部署步骤见 [`docs/DEPLOY.md`](docs/DEPLOY.md)。