Text Classification
GGUF
English
rknn
rknn3
rk1828
rk1820
rockchip
npu
edge-ai
kev
decision
w4a16
Eval Results (legacy)
conversational
Instructions to use jj90/Kev-0.8B-RKNN3-RK1828 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use jj90/Kev-0.8B-RKNN3-RK1828 with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf jj90/Kev-0.8B-RKNN3-RK1828 # Run inference directly in the terminal: llama cli -hf jj90/Kev-0.8B-RKNN3-RK1828
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf jj90/Kev-0.8B-RKNN3-RK1828 # Run inference directly in the terminal: llama cli -hf jj90/Kev-0.8B-RKNN3-RK1828
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf jj90/Kev-0.8B-RKNN3-RK1828 # Run inference directly in the terminal: ./build/bin/llama cli -hf jj90/Kev-0.8B-RKNN3-RK1828
Use Docker
docker model run hf.co/jj90/Kev-0.8B-RKNN3-RK1828
- Llama new
- LM Studio
- Jan
- Ollama
How to use jj90/Kev-0.8B-RKNN3-RK1828 with Ollama:
ollama run hf.co/jj90/Kev-0.8B-RKNN3-RK1828
- Unsloth Desktop
- Pi
How to use jj90/Kev-0.8B-RKNN3-RK1828 with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/Kev-0.8B-RKNN3-RK1828
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "jj90/Kev-0.8B-RKNN3-RK1828" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use jj90/Kev-0.8B-RKNN3-RK1828 with Docker Model Runner:
docker model run hf.co/jj90/Kev-0.8B-RKNN3-RK1828
- Lemonade
How to use jj90/Kev-0.8B-RKNN3-RK1828 with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull jj90/Kev-0.8B-RKNN3-RK1828
Run and chat with the model
lemonade run user.Kev-0.8B-RKNN3-RK1828-{{QUANT_TAG}}List all available models
lemonade list
- Hermes Agent
How to use jj90/Kev-0.8B-RKNN3-RK1828 with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/Kev-0.8B-RKNN3-RK1828
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default jj90/Kev-0.8B-RKNN3-RK1828
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use jj90/Kev-0.8B-RKNN3-RK1828 with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf jj90/Kev-0.8B-RKNN3-RK1828
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:9931/v1 \ --custom-model-id "jj90/Kev-0.8B-RKNN3-RK1828" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
|
Download README.md from jj90/Kev-0.8B-RKNN3-RK1828: direct link, hf CLI and curl.
- Browser
- Download file 12.3 kB
-
https://huggingface.co/jj90/Kev-0.8B-RKNN3-RK1828/resolve/main/README.md
- Command line
-
hf download hf://jj90/Kev-0.8B-RKNN3-RK1828/README.md
-
curl -L -o README.md https://huggingface.co/jj90/Kev-0.8B-RKNN3-RK1828/resolve/main/README.md
12.3 kB
| language: | |
| - en | |
| license: apache-2.0 | |
| pipeline_tag: text-classification | |
| inference: false | |
| tags: | |
| - rknn | |
| - rknn3 | |
| - rk1828 | |
| - rk1820 | |
| - rockchip | |
| - npu | |
| - edge-ai | |
| - kev | |
| - decision | |
| - text-classification | |
| - w4a16 | |
| base_model: Qwen/Qwen3.5-0.8B-Base | |
| base_model_relation: finetune | |
| model-index: | |
| - name: Kev-0.8B-RKNN3-RK1828 | |
| results: | |
| - task: | |
| type: text-classification | |
| name: Structured decision (choice / noul / score) | |
| dataset: | |
| type: kev-decision-v2 | |
| name: Kev decision-v2 (60 held-out) | |
| split: dev | |
| metrics: | |
| - type: accuracy | |
| value: 0.817 | |
| name: Accuracy (device) | |
| - task: | |
| type: text-classification | |
| name: Structured decision (choice / noul / score) | |
| dataset: | |
| type: kev-decision-v2 | |
| name: Kev decision-v2 (26 mixed-type, end-to-end HTTP) | |
| split: dev | |
| metrics: | |
| - type: accuracy | |
| value: 0.885 | |
| name: Accuracy (HTTP service) | |
| # Kev-0.8B-RKNN3-RK1828 | |
| **Kev-0.8B (decision model) converted to RKNN3 and running on the Rockchip RK1828 NPU — with zero accuracy loss versus the original fp32 model.** | |
| Kev is *not* a chat model. It runs a single prefill and scores options with a pointer head; it never generates tokens. | |
| You can verify this yourself: ask it `1+1`, and you get nonsense. That is the model behaving correctly. | |
| This repository contains the converted artifacts plus the minimal host-side code needed to actually | |
| use them, because a plain `rkllm3-server` deployment **cannot** run Kev (see [Why this needs host code](#why-this-needs-host-code)). | |
| | | | | |
| | --- | --- | | |
| | Base model | [`Qwen/Qwen3.5-0.8B-Base`](https://huggingface.co/Qwen/Qwen3.5-0.8B-Base) @ `dc7cdfe` | | |
| | Adapter | [`jaredpalmer/kev-0.8b`](https://huggingface.co/jaredpalmer/kev-0.8b) (LoRA r16, merged into base) | | |
| | Target hardware | RK1828 NPU (RK1820/RK1828 AI SDK), host = Windows PC / RK3588 / RK3576 | | |
| | Toolchain | RKNN3 **v1.1.0** (`runtime/rknn3_110`), `target_platform='rk1820'` | | |
| | Quantization | `w4a16`, `quantized_algorithm='normal'`, `quantized_method='group32'` (weight-only, no calibration set) | | |
| | NPU footprint | **802 MB** (as reported by `rknn-smi`) | | |
| | Latency | ~0.9 s per decision end-to-end, ~75 ms per prefill (~100 tokens) | | |
| --- | |
| ## Files | |
| | File | Size | What it is | | |
| | --- | --- | --- | | |
| | `rknn/Kev-0.8B.rknn` | 20.6 MiB | RKNN3 graph. Output is `[1, 1, 248320]` fp16 logits | | |
| | `rknn/Kev-0.8B.weight` | 498 MiB | Quantized backbone weights | | |
| | `rknn/Kev-0.8B.embed.bin` | 485 MiB | Token embedding table, `248320 × 1024` fp16 (also the dequantization matrix `W`) | | |
| | `rknn/Kev-0.8B.tokenizer.gguf` | 10.4 MiB | Tokenizer. Identical to Qwen3.5-0.8B — LoRA does not touch the vocab | | |
| | `rknn/start.cmd` | 1.4 KiB | `rkllm3-server` launch line (for the "text-only" sanity check) | | |
| | `host/kev_daemon.cpp` | — | Device-side runner: loads the model once, then takes `TOK`/`R`/`A` commands and dumps logits per request | | |
| | `host/kev_serve.py` | — | HTTP service exposing `POST /v1/systemone` | | |
| | `host/serve_assets/*.npy` | 6.1 MiB | `Ginv.npy` (1024×1024), `head_q.npy`, `head_k.npy`, `temperature.npy` — precomputed, so you do not have to invert a 1024×1024 normal-equation matrix at startup | | |
| | `host/e2e_check.py` | — | End-to-end accuracy check against the decision-v2 labels | | |
| | `docs/DEPLOY.md` | — | Step-by-step deployment, including the traps | | |
| Total: ~1.06 GB. | |
| ### Checksums | |
| ``` | |
| 776f782272076d2579eb5d5e59aee733ea8c10bd2994013ed791d05c449ff3f0 Kev-0.8B.rknn | |
| 0fcc3e04de7a645e166c11b1b98e769df8a145767d1979be54ba34578a0d6dce Kev-0.8B.weight | |
| 2146dc176cff21240562283cf0b709a91499917d5b5f2c4fb0b7b41b21d407b0 Kev-0.8B.embed.bin | |
| f2235450332c615492e7cfa13b103090a9adc7a802bef5d425842a87f8b6b03f Kev-0.8B.tokenizer.gguf | |
| ``` | |
| ```bash | |
| sha256sum -c <<'EOF' | |
| 776f782272076d2579eb5d5e59aee733ea8c10bd2994013ed791d05c449ff3f0 Kev-0.8B.rknn | |
| 0fcc3e04de7a645e166c11b1b98e769df8a145767d1979be54ba34578a0d6dce Kev-0.8B.weight | |
| 2146dc176cff21240562283cf0b709a91499917d5b5f2c4fb0b7b41b21d407b0 Kev-0.8B.embed.bin | |
| f2235450332c615492e7cfa13b103090a9adc7a802bef5d425842a87f8b6b03f Kev-0.8B.tokenizer.gguf | |
| EOF | |
| ``` | |
| --- | |
| ## Why this needs host code | |
| Kev's scoring path is: | |
| ``` | |
| h_decide = backbone(state + question + "<decide>") # hidden state, 1024-d, *before* lm_head | |
| h_opt_k = backbone(state + question + option_k + "</opt>") | |
| z_k = (k(h_opt_k) · q(h_decide)) / sqrt(256) | |
| answer = softmax(z / T) # T = 2.3510958125672174 | |
| ``` | |
| The runtime exposes **vocab logits**, not hidden states. The usual conclusion is "blocked, needs retraining". | |
| It is not blocked, because `logits = h @ Wᵀ` is an **overdetermined** system: | |
| ``` | |
| W ∈ R^(248320 × 1024), unknown h ∈ R^1024, 248320 equations | |
| h = (WᵀW)⁻¹ Wᵀ · logits | |
| ``` | |
| `W` is the (shared) embedding matrix, so it is already on disk. The per-coordinate quantization noise gets | |
| averaged over ~248k equations, i.e. suppressed by roughly √248320 ≈ 500×. Measured cosine similarity | |
| between device-recovered `h` and the fp32 reference is **0.96–0.99**, with top-1 agreeing on every sampled position. | |
| So the original pointer head runs unmodified on the host. **No retraining, no downgrade to an embedding model.** | |
| `Ginv.npy` is the precomputed `(WᵀW)⁻¹Wᵀ`, so recovery is a single `1024×1024 @ 1024` matmul per position. | |
| Two consequences worth knowing before you deploy: | |
| - **The graph only keeps the last position.** There is no `num_logits_to_keep` in the public API and | |
| `clear_kvcache` has no truncate. To read the `</opt>` positions you must re-run a truncated sequence. | |
| A request with `K` options therefore costs **K+1 prefills** — that is a hard floor, not an implementation choice. | |
| - **Per-call fixed overhead is ~50 ms**, independent of length. Latency scales with *number of calls*, not token count. | |
| Batching long states into one prefix helps a lot; splitting into many short calls does not. | |
| --- | |
| ## Usage | |
| ### 1. Sanity check (text only) | |
| ```bat | |
| cd rknn | |
| start.cmd | |
| ``` | |
| Then `POST /v1/chat/completions`. Output will be garbage — this only proves the weights loaded and the NPU | |
| is running. **Do not use this to judge whether the conversion worked.** | |
| ### 2. Real usage: decision API | |
| Build `host/kev_daemon.cpp` against the RKNN3 SDK session demo, run it, then: | |
| ```bash | |
| python host/kev_serve.py # listens on :17890 | |
| ``` | |
| ```bash | |
| curl -X POST http://127.0.0.1:17890/v1/systemone \ | |
| -H "Content-Type: application/json" \ | |
| -d '{ | |
| "state": {"ticket": "Checkout returns 500 for EU customers", "age_hours": 30}, | |
| "questions": { | |
| "team": {"type": "choice", "instructions": "Which team should own this?", | |
| "criteria": {"billing": "payment, invoices, refunds", | |
| "platform": "infra, deploys, runtime", | |
| "growth": "onboarding, activation"}}, | |
| "urgent": {"type": "noul", "instructions": "Is this urgent?"}, | |
| "sev": {"type": "score", "instructions": "Rate the severity", | |
| "criteria": ["cosmetic", "degraded", "broken", "outage"]} | |
| } | |
| }' | |
| ``` | |
| ```json | |
| {"answers": { | |
| "team": {"type": "choice", "answer": "billing", "confidence": 0.7638, "probabilities": {...}}, | |
| "urgent": {"type": "noul", "p_true": 0.912}, | |
| "sev": {"type": "score", "score": 2.31, "legend": [...], "score_confidence": 0.54} | |
| }} | |
| ``` | |
| Response schema follows the upstream `kev/api.py::to_answers` conventions: | |
| - `noul` returns `p_true` only (the upstream API does not binarize it) | |
| - `choice` returns `confidence = (p_max − 1/K) / (1 − 1/K)`, so `K=2` with `p=1.0` gives `confidence=1.0` | |
| - `score` returns the probability-weighted expected level as `score`, plus per-level probabilities | |
| Full commands and the build line: [`docs/DEPLOY.md`](docs/DEPLOY.md). | |
| --- | |
| ## Performance | |
| Measured on RK1828 (5051.8 MB NPU), Windows host, single model loaded. | |
| | Metric | Value | | |
| | --- | --- | | |
| | Accuracy, decision-v2 dev 60 (fp32 reference, host) | 49/60 = **0.817** | | |
| | Accuracy, decision-v2 dev 60 (device, w4a16 + recovered `h`) | 49/60 = **0.817** | | |
| | Per-question argmax agreement, device vs fp32 | **58/60 = 96.7%** | | |
| | End-to-end HTTP, 26 mixed-type questions | 23/26 = **0.885** | | |
| | Prefill throughput (long sequences) | 1200–1500 tok/s | | |
| | Single prefill, ~100 tokens | ~75 ms | | |
| | Per-call fixed overhead | ~50 ms | | |
| | One decision (K=5) | ~0.5 s | | |
| | End-to-end HTTP per question | ~0.9 s | | |
| | Model load | ~30 s, once | | |
| | NPU memory | 802 MB | | |
| The two disagreeing questions (q9, q10) cancel out — one is right on device and wrong on fp32, the other the reverse. | |
| That is the signature of the *same function* being computed, with numerical noise flipping borderline cases. | |
| Coexistence, measured with `rknn-smi` on the same device: | |
| | Configuration | NPU used | Delta | | |
| | --- | --- | --- | | |
| | baseline (proxy only) | 34 MB | — | | |
| | Kev-0.8B alone | 802 MB | — | | |
| | Kev-0.8B + Qwen3-ASR-0.6B | 2848 MB | +2047 MB | | |
| | Kev-0.8B + Qwen3-ASR-1.7B | 2953 MB | +2152 MB | | |
| Kev can share the NPU with an ASR model. Both fit; only ~105 MB separates the two ASR sizes. | |
| Measure *coexisting* configurations, not standalone ones — a standalone run reports 779 MB for the 0.6B ASR, | |
| which will make you over-budget your plan. | |
| --- | |
| ## Limitations | |
| - **Not a chat model.** Text generation output is meaningless by design. Any benchmark that prompts it for answers is invalid. | |
| - **Numeric noise flips borderline decisions.** 96.7% per-question agreement means ~3% of hard cases can differ from | |
| the fp32 reference. If you need exact reproducibility, run the fp32 model. | |
| - **K+1 prefill calls per request** cannot be reduced with this runtime (see above). Very large option lists get slow linearly. | |
| - **Single-request serialization.** The service holds a global lock around one decision, because `K+1` prefills share | |
| one KV cache; interleaving concurrent requests corrupts it. Throughput is therefore ~1 request at a time. | |
| Concurrency is safe (tested), but it queues. | |
| - **Host must be Windows / RK3588 / RK3576.** The RK1828 is a co-processor; an x86 Linux box can convert and simulate | |
| but cannot drive the module. | |
| - **Quantization format is fixed** to `w4a16 + normal + group32` by this toolchain. `w8a16` and `fp16` builds are rejected | |
| (`request_type=float16` / `Failed to create initializer tensor`), so you cannot "build a higher-precision version" | |
| to isolate quantization error. | |
| --- | |
| ## Provenance | |
| - Base: `Qwen/Qwen3.5-0.8B-Base`, revision `dc7cdfe2ee4154fa7e30f5b51ca41bfa40174e68` | |
| - Adapter: `jaredpalmer/kev-0.8b`, LoRA r16 / α32, task type `FEATURE_EXTRACTION` | |
| - Merged via `merge_and_unload()`, then exported to ONNX with the official `rknn3-model-zoo/examples/Qwen3_5` pipeline | |
| - `h`, the pointer head, and the option/decide token ids are all taken from the upstream Kev implementation unchanged | |
| Special token ids used (from upstream `kev/model.py::SPECIAL`): | |
| ``` | |
| <state> 248060 <question> 248061 <opt> 248049 </opt> 248050 <decide> 248062 | |
| ``` | |
| Artifact sizes match the officially published Qwen3.5-0.8B RKNN3 build file-for-file | |
| (20.6 / 498 / 485 / 10.4 MiB), which is the practical proof that the conversion is structurally correct. | |
| License: **Apache-2.0**, inherited from the base model. | |
| --- | |
| ## 中文摘要 | |
| Kev-0.8B 是决策模型,不是对话模型:它只跑一次 prefill,用指针头给候选项打分,不生成 token。 | |
| 用"问它 1+1"来验收是错的——那样一定会得到无意义的回答。 | |
| RKNN3 运行时只暴露 vocab logits,而 Kev 需要 hidden state。但 `logits = h @ Wᵀ` 是超定方程 | |
| (248320 个方程解 1024 个未知数),`h = (WᵀW)⁻¹Wᵀ·logits` 可以精确解回来, | |
| 量化噪声被约 500 倍平均掉。因此**原版指针头可以原样跑在 host 侧,无需重训、无需改结构**。 | |
| | 项 | 结果 | | |
| | --- | --- | | |
| | 真机准确率(60 题) | 0.817,与 fp32 完全一致 | | |
| | 逐题一致 | 58/60 | | |
| | 端到端 HTTP(26 题三题型) | 0.885,约 0.9 s/题 | | |
| | NPU 占用 | 802 MB | | |
| 本仓库包含转换产物 + 可复现的 host 侧代码(daemon、HTTP 服务、反解矩阵、头权重), | |
| 拿到即可跑通。部署步骤见 [`docs/DEPLOY.md`](docs/DEPLOY.md)。 | |