Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+ov_intent_analysis_sft_v7_w8a8_rk3588.rkllm filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,119 @@
+---
+license: apache-2.0
+base_model:
+  - guoxuter/ov_intent_analysis_sft
+tags:
+  - rockchip
+  - rk3588
+  - rkllm
+  - npu
+  - openviking
+  - retrieval
+  - intent-analysis
+  - query-planning
+  - qwen3.5
+pipeline_tag: text-generation
+---
+
+# ov_intent_analysis_sft — RKLLM (RK3588) conversion
+
+Pre-converted **W8A8** RKLLM runtime file of [`guoxuter/ov_intent_analysis_sft`](https://huggingface.co/guoxuter/ov_intent_analysis_sft) (the OpenViking retrieval intent-analysis / query-planner model, fine-tuned from Qwen3.5-0.8B).
+
+Run the **same tuned query planner** on your RK3588 NPU — no x86 conversion rig required.
+
+## Why this exists
+
+OpenViking's docs recommend this model for the `query_planner` slot: it decides whether a search needs context retrieval, skips chitchat (no queries → no token spend), and emits structured `skill` / `resource` / `memory` queries. The stock model ships as HF safetensors; to run it on the RK3588 NPU you need a `.rkllm` file, which can only be produced by **rkllm-toolkit (x86_64-only)**. This repo is that conversion, done once, so RK3588 owners can skip the whole rig.
+
+## File
+
+| File | Size | Spec |
+|---|---|---|
+| `ov_intent_analysis_sft_v7_w8a8_rk3588.rkllm` | 1.3 GB | W8A8, RK3588, 3 NPU cores, max_context 4096 |
+
+Runtime requirements: `librkllmrt.so` 1.3.0 (rkllm-toolkit 1.3.0 generation). Verified on kernel 6.1 vendor with rknpu driver 0.9.8.
+
+## How to serve it
+
+Any RKLLM-capable server works. Two options:
+
+### Option A — full rkllama server (Ollama API; recommended for OpenViking)
+
+```bash
+pip install rkllama   # python 3.9–3.12
+rkllama_server --models /path/to/models
+```
+
+Place the `.rkllm` in your models dir. This gives an Ollama-compatible API, which is what OpenViking's `query_planner` speaks.
+
+### Option B — minimal OpenAI+Ollama server (no transformers/torch)
+
+The same ctypes wrapper, no heavy deps (see the conversion recipe below for the gist). Serves `/v1/*` and `/api/*`.
+
+## OpenViking wiring
+
+```json
+{
+  "query_planner": {
+    "provider": "litellm",
+    "model": "ollama/guoxuter/ov_intent_analysis_sft:v7_q8",
+    "api_base": "http://127.0.0.1:8091",
+    "temperature": 0.0,
+    "timeout": 60,
+    "extra_request_body": { "think": false }
+  }
+}
+```
+
+**Keep the model string exactly as-is** — OpenViking auto-matches the bundled v7 prompt by string (`retrieval.ov_intent_analysis_sft_v7` in `intent_analyzer.py`). Only `api_base` changes: point it at your RKLLM server instead of Ollama.
+
+## Benchmark (RK3588, same prompt)
+
+| Runtime | Wall time | Notes |
+|---|---|---|
+| Ollama / CPU (GGUF Q8) | 16.5 s | output lands in `thinking` unless `think:false` |
+| rk-llama.cpp NPU (GGUF Q8) | 10.9 s | needed `--reasoning off` |
+| **RKLLM NPU (this file, W8A8)** | **~9 s** (1.8 s warm) | prefill ~200 t/s, decode ~13 t/s |
+
+Query planning is prefill-dominated, which is exactly where the NPU wins. Decode is memory-bandwidth-bound, so don't expect magic on long generations — this model's outputs are short JSON.
+
+## Conversion recipe (for reproducing / other models)
+
+The converter is **x86-only**, so run it on an x86 box (any Linux, or a serverless cloud like Modal):
+
+```python
+# rkllm-toolkit 1.3.0 (wheel from airockchip/rknn-llm release-v1.3.0,
+# rkllm-toolkit/packages/rkllm_toolkit-1.3.0-cp311-cp311-linux_x86_64.whl)
+# deps pinned from that release's requirements.txt (torch 2.6.0, transformers 5.8.0, ...)
+
+from rkllm.api import RKLLM
+
+llm = RKLLM()
+llm.load_huggingface(model="guoxuter/ov_intent_analysis_sft", device="cpu")  # or cuda
+llm.build(
+    do_quantization=True,
+    optimization_level=1,
+    quantized_dtype="W8A8",
+    quantized_algorithm="normal",
+    target_platform="RK3588",
+    num_npu_core=3,
+    dataset="data_quant.json",   # calibration: input/target pairs
+    hybrid_rate=0,               # REQUIRED arg in 1.3.0
+    max_context=4096,            # NOTE: `max_context`, NOT `max_context_len`
+)
+llm.export_rkllm("ov_intent_analysis_sft_v7_w8a8_rk3588.rkllm")
+```
+
+Gotchas hit along the way (so you don't):
+- `build()` takes `max_context`, not `max_context_len` (raises `TypeError: unexpected keyword argument`).
+- `hybrid_rate=0` is required in the 1.3.0 signature.
+- The toolkit only ships **x86_64 wheels** — there is no aarch64 path; don't fight it on an ARM SBC.
+- Keep the toolkit major version aligned with your runtime's `librkllmrt.so` (1.3.0 ↔ 1.3.0).
+
+## Attribution & license
+
+- Base model: [`guoxuter/ov_intent_analysis_sft`](https://huggingface.co/guoxuter/ov_intent_analysis_sft) — **Apache-2.0**, which its card states covers the fine-tuned checkpoint (same license as Qwen3.5-0.8B).
+- Conversion performed with Rockchip's rkllm-toolkit 1.3.0 ([airockchip/rknn-llm](https://github.com/airockchip/rknn-llm)).
+- This conversion is published under **Apache-2.0**.
+
+Big thanks to guoxuter for the tuned model and the OpenViking team for the recommended workflow.

ov_intent_analysis_sft_v7_w8a8_rk3588.rkllm

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c2d5477021b55ced9d5228f564663128a34281a2c43e159fd64323cdbd09a710
+size 1299987468