README.md

---
license: apache-2.0
base_model:
  - guoxuter/ov_intent_analysis_sft
tags:
  - rockchip
  - rk3588
  - rkllm
  - npu
  - openviking
  - retrieval
  - intent-analysis
  - query-planning
  - qwen3.5
pipeline_tag: text-generation
---

# ov_intent_analysis_sft — RKLLM (RK3588) conversion

Pre-converted **W8A8** RKLLM runtime file of [`guoxuter/ov_intent_analysis_sft`](https://huggingface.co/guoxuter/ov_intent_analysis_sft) (the OpenViking retrieval intent-analysis / query-planner model, fine-tuned from Qwen3.5-0.8B).

Run the **same tuned query planner** on your RK3588 NPU — no x86 conversion rig required.

## Why this exists

OpenViking's docs recommend this model for the `query_planner` slot: it decides whether a search needs context retrieval, skips chitchat (no queries → no token spend), and emits structured `skill` / `resource` / `memory` queries. The stock model ships as HF safetensors; to run it on the RK3588 NPU you need a `.rkllm` file, which can only be produced by **rkllm-toolkit (x86_64-only)**. This repo is that conversion, done once, so RK3588 owners can skip the whole rig.

## File

| File | Size | Spec |
|---|---|---|
| `ov_intent_analysis_sft_v7_w8a8_rk3588.rkllm` | 1.3 GB | W8A8, RK3588, 3 NPU cores, max_context 4096 |

Runtime requirements: `librkllmrt.so` 1.3.0 (rkllm-toolkit 1.3.0 generation). Verified on kernel 6.1 vendor with rknpu driver 0.9.8.

## How to serve it

Any RKLLM-capable server works. Two options:

### Option A — full rkllama server (Ollama API; recommended for OpenViking)

```bash
pip install rkllama   # python 3.9–3.12
rkllama_server --models /path/to/models
```

Place the `.rkllm` in your models dir. This gives an Ollama-compatible API, which is what OpenViking's `query_planner` speaks.

### Option B — minimal OpenAI+Ollama server (no transformers/torch)

The same ctypes wrapper, no heavy deps (see the conversion recipe below for the gist). Serves `/v1/*` and `/api/*`.

## OpenViking wiring

```json
{
  "query_planner": {
    "provider": "litellm",
    "model": "ollama/guoxuter/ov_intent_analysis_sft:v7_q8",
    "api_base": "http://127.0.0.1:8091",
    "temperature": 0.0,
    "timeout": 60,
    "extra_request_body": { "think": false }
  }
}
```

**Keep the model string exactly as-is** — OpenViking auto-matches the bundled v7 prompt by string (`retrieval.ov_intent_analysis_sft_v7` in `intent_analyzer.py`). Only `api_base` changes: point it at your RKLLM server instead of Ollama.

## Benchmark (RK3588, same prompt)

| Runtime | Wall time | Notes |
|---|---|---|
| Ollama / CPU (GGUF Q8) | 16.5 s | output lands in `thinking` unless `think:false` |
| rk-llama.cpp NPU (GGUF Q8) | 10.9 s | needed `--reasoning off` |
| **RKLLM NPU (this file, W8A8)** | **~9 s** (1.8 s warm) | prefill ~200 t/s, decode ~13 t/s |

Query planning is prefill-dominated, which is exactly where the NPU wins. Decode is memory-bandwidth-bound, so don't expect magic on long generations — this model's outputs are short JSON.

## Conversion recipe (for reproducing / other models)

The converter is **x86-only**, so run it on an x86 box (any Linux, or a serverless cloud like Modal):

```python
# rkllm-toolkit 1.3.0 (wheel from airockchip/rknn-llm release-v1.3.0,
# rkllm-toolkit/packages/rkllm_toolkit-1.3.0-cp311-cp311-linux_x86_64.whl)
# deps pinned from that release's requirements.txt (torch 2.6.0, transformers 5.8.0, ...)

from rkllm.api import RKLLM

llm = RKLLM()
llm.load_huggingface(model="guoxuter/ov_intent_analysis_sft", device="cpu")  # or cuda
llm.build(
    do_quantization=True,
    optimization_level=1,
    quantized_dtype="W8A8",
    quantized_algorithm="normal",
    target_platform="RK3588",
    num_npu_core=3,
    dataset="data_quant.json",   # calibration: input/target pairs
    hybrid_rate=0,               # REQUIRED arg in 1.3.0
    max_context=4096,            # NOTE: `max_context`, NOT `max_context_len`
)
llm.export_rkllm("ov_intent_analysis_sft_v7_w8a8_rk3588.rkllm")
```

Gotchas hit along the way (so you don't):
- `build()` takes `max_context`, not `max_context_len` (raises `TypeError: unexpected keyword argument`).
- `hybrid_rate=0` is required in the 1.3.0 signature.
- The toolkit only ships **x86_64 wheels** — there is no aarch64 path; don't fight it on an ARM SBC.
- Keep the toolkit major version aligned with your runtime's `librkllmrt.so` (1.3.0 ↔ 1.3.0).

## Attribution & license

- Base model: [`guoxuter/ov_intent_analysis_sft`](https://huggingface.co/guoxuter/ov_intent_analysis_sft) — **Apache-2.0**, which its card states covers the fine-tuned checkpoint (same license as Qwen3.5-0.8B).
- Conversion performed with Rockchip's rkllm-toolkit 1.3.0 ([airockchip/rknn-llm](https://github.com/airockchip/rknn-llm)).
- This conversion is published under **Apache-2.0**.

Big thanks to guoxuter for the tuned model and the OpenViking team for the recommended workflow.