jj90's picture
Add SHA256 checksums
a0e27ab verified
|
Raw History Blame Contribute Delete
12.3 kB
---
language:
- en
license: apache-2.0
pipeline_tag: text-classification
inference: false
tags:
- rknn
- rknn3
- rk1828
- rk1820
- rockchip
- npu
- edge-ai
- kev
- decision
- text-classification
- w4a16
base_model: Qwen/Qwen3.5-0.8B-Base
base_model_relation: finetune
model-index:
- name: Kev-0.8B-RKNN3-RK1828
results:
- task:
type: text-classification
name: Structured decision (choice / noul / score)
dataset:
type: kev-decision-v2
name: Kev decision-v2 (60 held-out)
split: dev
metrics:
- type: accuracy
value: 0.817
name: Accuracy (device)
- task:
type: text-classification
name: Structured decision (choice / noul / score)
dataset:
type: kev-decision-v2
name: Kev decision-v2 (26 mixed-type, end-to-end HTTP)
split: dev
metrics:
- type: accuracy
value: 0.885
name: Accuracy (HTTP service)
---
# Kev-0.8B-RKNN3-RK1828
**Kev-0.8B (decision model) converted to RKNN3 and running on the Rockchip RK1828 NPU — with zero accuracy loss versus the original fp32 model.**
Kev is *not* a chat model. It runs a single prefill and scores options with a pointer head; it never generates tokens.
You can verify this yourself: ask it `1+1`, and you get nonsense. That is the model behaving correctly.
This repository contains the converted artifacts plus the minimal host-side code needed to actually
use them, because a plain `rkllm3-server` deployment **cannot** run Kev (see [Why this needs host code](#why-this-needs-host-code)).
| | |
| --- | --- |
| Base model | [`Qwen/Qwen3.5-0.8B-Base`](https://huggingface.co/Qwen/Qwen3.5-0.8B-Base) @ `dc7cdfe` |
| Adapter | [`jaredpalmer/kev-0.8b`](https://huggingface.co/jaredpalmer/kev-0.8b) (LoRA r16, merged into base) |
| Target hardware | RK1828 NPU (RK1820/RK1828 AI SDK), host = Windows PC / RK3588 / RK3576 |
| Toolchain | RKNN3 **v1.1.0** (`runtime/rknn3_110`), `target_platform='rk1820'` |
| Quantization | `w4a16`, `quantized_algorithm='normal'`, `quantized_method='group32'` (weight-only, no calibration set) |
| NPU footprint | **802 MB** (as reported by `rknn-smi`) |
| Latency | ~0.9 s per decision end-to-end, ~75 ms per prefill (~100 tokens) |
---
## Files
| File | Size | What it is |
| --- | --- | --- |
| `rknn/Kev-0.8B.rknn` | 20.6 MiB | RKNN3 graph. Output is `[1, 1, 248320]` fp16 logits |
| `rknn/Kev-0.8B.weight` | 498 MiB | Quantized backbone weights |
| `rknn/Kev-0.8B.embed.bin` | 485 MiB | Token embedding table, `248320 × 1024` fp16 (also the dequantization matrix `W`) |
| `rknn/Kev-0.8B.tokenizer.gguf` | 10.4 MiB | Tokenizer. Identical to Qwen3.5-0.8B — LoRA does not touch the vocab |
| `rknn/start.cmd` | 1.4 KiB | `rkllm3-server` launch line (for the "text-only" sanity check) |
| `host/kev_daemon.cpp` | — | Device-side runner: loads the model once, then takes `TOK`/`R`/`A` commands and dumps logits per request |
| `host/kev_serve.py` | — | HTTP service exposing `POST /v1/systemone` |
| `host/serve_assets/*.npy` | 6.1 MiB | `Ginv.npy` (1024×1024), `head_q.npy`, `head_k.npy`, `temperature.npy` — precomputed, so you do not have to invert a 1024×1024 normal-equation matrix at startup |
| `host/e2e_check.py` | — | End-to-end accuracy check against the decision-v2 labels |
| `docs/DEPLOY.md` | — | Step-by-step deployment, including the traps |
Total: ~1.06 GB.
### Checksums
```
776f782272076d2579eb5d5e59aee733ea8c10bd2994013ed791d05c449ff3f0 Kev-0.8B.rknn
0fcc3e04de7a645e166c11b1b98e769df8a145767d1979be54ba34578a0d6dce Kev-0.8B.weight
2146dc176cff21240562283cf0b709a91499917d5b5f2c4fb0b7b41b21d407b0 Kev-0.8B.embed.bin
f2235450332c615492e7cfa13b103090a9adc7a802bef5d425842a87f8b6b03f Kev-0.8B.tokenizer.gguf
```
```bash
sha256sum -c <<'EOF'
776f782272076d2579eb5d5e59aee733ea8c10bd2994013ed791d05c449ff3f0 Kev-0.8B.rknn
0fcc3e04de7a645e166c11b1b98e769df8a145767d1979be54ba34578a0d6dce Kev-0.8B.weight
2146dc176cff21240562283cf0b709a91499917d5b5f2c4fb0b7b41b21d407b0 Kev-0.8B.embed.bin
f2235450332c615492e7cfa13b103090a9adc7a802bef5d425842a87f8b6b03f Kev-0.8B.tokenizer.gguf
EOF
```
---
## Why this needs host code
Kev's scoring path is:
```
h_decide = backbone(state + question + "<decide>") # hidden state, 1024-d, *before* lm_head
h_opt_k = backbone(state + question + option_k + "</opt>")
z_k = (k(h_opt_k) · q(h_decide)) / sqrt(256)
answer = softmax(z / T) # T = 2.3510958125672174
```
The runtime exposes **vocab logits**, not hidden states. The usual conclusion is "blocked, needs retraining".
It is not blocked, because `logits = h @ Wᵀ` is an **overdetermined** system:
```
W ∈ R^(248320 × 1024), unknown h ∈ R^1024, 248320 equations
h = (WᵀW)⁻¹ Wᵀ · logits
```
`W` is the (shared) embedding matrix, so it is already on disk. The per-coordinate quantization noise gets
averaged over ~248k equations, i.e. suppressed by roughly √248320 ≈ 500×. Measured cosine similarity
between device-recovered `h` and the fp32 reference is **0.96–0.99**, with top-1 agreeing on every sampled position.
So the original pointer head runs unmodified on the host. **No retraining, no downgrade to an embedding model.**
`Ginv.npy` is the precomputed `(WᵀW)⁻¹Wᵀ`, so recovery is a single `1024×1024 @ 1024` matmul per position.
Two consequences worth knowing before you deploy:
- **The graph only keeps the last position.** There is no `num_logits_to_keep` in the public API and
`clear_kvcache` has no truncate. To read the `</opt>` positions you must re-run a truncated sequence.
A request with `K` options therefore costs **K+1 prefills** — that is a hard floor, not an implementation choice.
- **Per-call fixed overhead is ~50 ms**, independent of length. Latency scales with *number of calls*, not token count.
Batching long states into one prefix helps a lot; splitting into many short calls does not.
---
## Usage
### 1. Sanity check (text only)
```bat
cd rknn
start.cmd
```
Then `POST /v1/chat/completions`. Output will be garbage — this only proves the weights loaded and the NPU
is running. **Do not use this to judge whether the conversion worked.**
### 2. Real usage: decision API
Build `host/kev_daemon.cpp` against the RKNN3 SDK session demo, run it, then:
```bash
python host/kev_serve.py # listens on :17890
```
```bash
curl -X POST http://127.0.0.1:17890/v1/systemone \
-H "Content-Type: application/json" \
-d '{
"state": {"ticket": "Checkout returns 500 for EU customers", "age_hours": 30},
"questions": {
"team": {"type": "choice", "instructions": "Which team should own this?",
"criteria": {"billing": "payment, invoices, refunds",
"platform": "infra, deploys, runtime",
"growth": "onboarding, activation"}},
"urgent": {"type": "noul", "instructions": "Is this urgent?"},
"sev": {"type": "score", "instructions": "Rate the severity",
"criteria": ["cosmetic", "degraded", "broken", "outage"]}
}
}'
```
```json
{"answers": {
"team": {"type": "choice", "answer": "billing", "confidence": 0.7638, "probabilities": {...}},
"urgent": {"type": "noul", "p_true": 0.912},
"sev": {"type": "score", "score": 2.31, "legend": [...], "score_confidence": 0.54}
}}
```
Response schema follows the upstream `kev/api.py::to_answers` conventions:
- `noul` returns `p_true` only (the upstream API does not binarize it)
- `choice` returns `confidence = (p_max − 1/K) / (1 − 1/K)`, so `K=2` with `p=1.0` gives `confidence=1.0`
- `score` returns the probability-weighted expected level as `score`, plus per-level probabilities
Full commands and the build line: [`docs/DEPLOY.md`](docs/DEPLOY.md).
---
## Performance
Measured on RK1828 (5051.8 MB NPU), Windows host, single model loaded.
| Metric | Value |
| --- | --- |
| Accuracy, decision-v2 dev 60 (fp32 reference, host) | 49/60 = **0.817** |
| Accuracy, decision-v2 dev 60 (device, w4a16 + recovered `h`) | 49/60 = **0.817** |
| Per-question argmax agreement, device vs fp32 | **58/60 = 96.7%** |
| End-to-end HTTP, 26 mixed-type questions | 23/26 = **0.885** |
| Prefill throughput (long sequences) | 1200–1500 tok/s |
| Single prefill, ~100 tokens | ~75 ms |
| Per-call fixed overhead | ~50 ms |
| One decision (K=5) | ~0.5 s |
| End-to-end HTTP per question | ~0.9 s |
| Model load | ~30 s, once |
| NPU memory | 802 MB |
The two disagreeing questions (q9, q10) cancel out — one is right on device and wrong on fp32, the other the reverse.
That is the signature of the *same function* being computed, with numerical noise flipping borderline cases.
Coexistence, measured with `rknn-smi` on the same device:
| Configuration | NPU used | Delta |
| --- | --- | --- |
| baseline (proxy only) | 34 MB | — |
| Kev-0.8B alone | 802 MB | — |
| Kev-0.8B + Qwen3-ASR-0.6B | 2848 MB | +2047 MB |
| Kev-0.8B + Qwen3-ASR-1.7B | 2953 MB | +2152 MB |
Kev can share the NPU with an ASR model. Both fit; only ~105 MB separates the two ASR sizes.
Measure *coexisting* configurations, not standalone ones — a standalone run reports 779 MB for the 0.6B ASR,
which will make you over-budget your plan.
---
## Limitations
- **Not a chat model.** Text generation output is meaningless by design. Any benchmark that prompts it for answers is invalid.
- **Numeric noise flips borderline decisions.** 96.7% per-question agreement means ~3% of hard cases can differ from
the fp32 reference. If you need exact reproducibility, run the fp32 model.
- **K+1 prefill calls per request** cannot be reduced with this runtime (see above). Very large option lists get slow linearly.
- **Single-request serialization.** The service holds a global lock around one decision, because `K+1` prefills share
one KV cache; interleaving concurrent requests corrupts it. Throughput is therefore ~1 request at a time.
Concurrency is safe (tested), but it queues.
- **Host must be Windows / RK3588 / RK3576.** The RK1828 is a co-processor; an x86 Linux box can convert and simulate
but cannot drive the module.
- **Quantization format is fixed** to `w4a16 + normal + group32` by this toolchain. `w8a16` and `fp16` builds are rejected
(`request_type=float16` / `Failed to create initializer tensor`), so you cannot "build a higher-precision version"
to isolate quantization error.
---
## Provenance
- Base: `Qwen/Qwen3.5-0.8B-Base`, revision `dc7cdfe2ee4154fa7e30f5b51ca41bfa40174e68`
- Adapter: `jaredpalmer/kev-0.8b`, LoRA r16 / α32, task type `FEATURE_EXTRACTION`
- Merged via `merge_and_unload()`, then exported to ONNX with the official `rknn3-model-zoo/examples/Qwen3_5` pipeline
- `h`, the pointer head, and the option/decide token ids are all taken from the upstream Kev implementation unchanged
Special token ids used (from upstream `kev/model.py::SPECIAL`):
```
<state> 248060 <question> 248061 <opt> 248049 </opt> 248050 <decide> 248062
```
Artifact sizes match the officially published Qwen3.5-0.8B RKNN3 build file-for-file
(20.6 / 498 / 485 / 10.4 MiB), which is the practical proof that the conversion is structurally correct.
License: **Apache-2.0**, inherited from the base model.
---
## 中文摘要
Kev-0.8B 是决策模型,不是对话模型:它只跑一次 prefill,用指针头给候选项打分,不生成 token。
用"问它 1+1"来验收是错的——那样一定会得到无意义的回答。
RKNN3 运行时只暴露 vocab logits,而 Kev 需要 hidden state。但 `logits = h @ Wᵀ` 是超定方程
(248320 个方程解 1024 个未知数),`h = (WᵀW)⁻¹Wᵀ·logits` 可以精确解回来,
量化噪声被约 500 倍平均掉。因此**原版指针头可以原样跑在 host 侧,无需重训、无需改结构**。
| 项 | 结果 |
| --- | --- |
| 真机准确率(60 题) | 0.817,与 fp32 完全一致 |
| 逐题一致 | 58/60 |
| 端到端 HTTP(26 题三题型) | 0.885,约 0.9 s/题 |
| NPU 占用 | 802 MB |
本仓库包含转换产物 + 可复现的 host 侧代码(daemon、HTTP 服务、反解矩阵、头权重),
拿到即可跑通。部署步骤见 [`docs/DEPLOY.md`](docs/DEPLOY.md)。