--- license: apache-2.0 base_model: Qwen/Qwen3-VL-2B-Instruct library_name: rkllm pipeline_tag: image-text-to-text tags: [rkllm, rockchip, rk3576, rk3588, qwen3-vl] --- **English** | [简体中文](README_zh-CN.md) ## 🚀 Deploy with reComputer AI Lab Explore and deploy AI models on Rockchip **RK3576** and **RK3588** devices with **reComputer AI Lab**. [![reComputer AI Lab — explore models and deployment guides](https://files.seeedstudio.com/Edge_box/reComputer_Lab.webp)](https://sensecraft.seeed.cc/ai-lab/en/models?device=rk3588) ### ⚡ Run with one command With Docker installed, choose your model and board in reComputer AI Lab, copy the deployment command, and run it in your device’s terminal. **[Explore Models & Deployment Guides →](https://sensecraft.seeed.cc/ai-lab/en/models?device=rk3588)** --- # Qwen3-VL-2B-Instruct · RKLLM * Max context length: 8192 Text and image chat on Rockchip RK3576 and RK3588, converted from [Qwen/Qwen3-VL-2B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct). - **Conversion toolkit:** RKLLM Toolkit 1.3.1 and RKNN Toolkit2 2.5.0. - **Inputs / output:** text and one image → text. Video and audio input are unavailable in this runtime. - **License:** Apache 2.0; see the [original model](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct) and [license](LICENSE). ## Choose your files Use files for your board's exact SoC. RK3576 offers compact W4A16-g128 and W8A8; RK3588 uses W8A8. Download size is not runtime RAM. | Board | Quantization | Size | Language model | | --- | --- | --- | --- | | RK3576 | W4A16-g128 | 1.74 GiB | [Qwen3-VL-2B-Instruct_RK3576_w4a16_g128.rkllm](v1.3.1/RK3576/Qwen3-VL-2B-Instruct_RK3576_w4a16_g128.rkllm) | | RK3576 | w8a8 | 2.28 GiB | [Qwen3-VL-2B-Instruct_RK3576_w8a8.rkllm](v1.3.1/RK3576/Qwen3-VL-2B-Instruct_RK3576_w8a8.rkllm) | | RK3588 | w8a8 | 2.22 GiB | [Qwen3-VL-2B-Instruct_RK3588_w8a8.rkllm](v1.3.1/RK3588/Qwen3-VL-2B-Instruct_RK3588_w8a8.rkllm) | Every language model requires its board's matching vision encoder below. Both RK3576 language variants share the RK3576 encoder. | Board | Size | Vision encoder | | --- | --- | --- | | RK3576 | 0.81 GiB | [Qwen3-VL-2B-Instruct_vision_RK3576.rknn](v1.3.1/RK3576/Qwen3-VL-2B-Instruct_vision_RK3576.rknn) | | RK3588 | 0.80 GiB | [Qwen3-VL-2B-Instruct_vision_RK3588.rknn](v1.3.1/RK3588/Qwen3-VL-2B-Instruct_vision_RK3588.rknn) | ## Run on your board Download the RK3576 W4A16-g128 language model and matching encoder: ```bash hf download HanzoHuang/Qwen3-VL-2B-Instruct-RKLLM \ v1.3.1/RK3576/Qwen3-VL-2B-Instruct_RK3576_w4a16_g128.rkllm \ v1.3.1/RK3576/Qwen3-VL-2B-Instruct_vision_RK3576.rknn \ --local-dir ./models sudo docker run --rm -it --pull=always --privileged \ -p 8001:8001 -v /dev:/dev \ -v "$PWD/models/v1.3.1/RK3576:/app/models:ro" \ -e MODEL_KIND=vlm -e TARGET_PLATFORM=rk3576 -e RKLLM_EMBED_FLASH=1 \ -e MODEL_FILE=Qwen3-VL-2B-Instruct_RK3576_w4a16_g128.rkllm \ -e VISION_MODEL_FILE=Qwen3-VL-2B-Instruct_vision_RK3576.rknn \ ghcr.io/hanzo-huang/rkllm-docker:env-latest \ serve --max_context_len 4096 ``` For RK3588, download both RK3588 files, mount that directory, set `TARGET_PLATFORM=rk3588`, and select its W8A8 language filename. Use the [runtime documentation](https://github.com/Hanzo-Huang/rkllm-docker) for health checks and OpenAI-compatible image requests. ## Release notes - Source acquired from ModelScope `Qwen/Qwen3-VL-2B-Instruct`, pinned revision `ae9985b208c074c10cfbe3a61b5cb7268cdc9c53`. The HF link above identifies the canonical upstream model; no equivalence to an HF Git commit is assumed. - Export context: **8192 tokens**. Functional API tests used **4096**; the speed/memory benchmark used **2048**. Longer runtime context has not been validated on these boards. - Fixed vision input: RGB, 448 × 448, batch 1, 196 image tokens, mean/std 0.5. The runtime pads to a square using RGB 127, then resizes bilinearly. Vision output order is pooler followed by three deepstack features for each token. - Language calibration: the SDK's 20 image-and-text examples, captured using the source processor and language-model input hook. W4A16-g128 uses GRQ; W8A8 uses normal quantization. Float32 CUDA loading; two NPU cores on RK3576 and three on RK3588. - Export compatibility fixes select the four language-width vision outputs under Transformers 5 and import only the requested calibration model family. Generated upstream examples and source checkpoint weights are unchanged. - All three variants passed seven API smoke checks on their target board: arithmetic, independent requests, SSE, exact conversation recall, image description, image color over SSE, and text after an image. Tests used the baked RKLLM 1.3.1 runtime application with `embed_flash=1`. - RK3576 vision encoder cosine against original float32 embeddings: 0.99767705. - RK3588 vision encoder cosine against original float32 embeddings: 0.99805251. These are functional and numerical smoke checks, not a broad quality benchmark. Quantization and fixed image resolution may reduce accuracy. Measured on 2026-10-22: | Board | Quantization | Decode tok/s | Startup RAM (GB) | Stable RAM (GB) | | --- | --- | ---: | ---: | ---: | | RK3576 | w4a16-g128 | 11.11 | 2.60 | 2.57 | | RK3576 | w8a8 | 9.02 | 3.18 | 3.18 | | RK3588 | w8a8 | 14.05 | 3.03 | 3.07 | Speed is the median native decode rate over three 128-token runs after a 32-token warmup, greedy decoding, EOS ignored, context 2048. RAM is additional board usage above a five-second idle baseline: sampled loading peak and ten-second ready-state median, in decimal GB. The matched encoder is loaded; `embed_flash=1`. First-token timing stays in the private report. Measurements depend on the board, clocks, cooling, and background load. RK3576 W8A8 was measured with zram swap temporarily disabled and restored afterward. Earlier measurements with background swap growth were excluded.
SHA-256 checksums ```text f687dccc90da6ef74d5308af3a05dba5cf3fea68c359b151381b7dbdc8a06c8f v1.3.1/RK3576/Qwen3-VL-2B-Instruct_RK3576_w4a16_g128.rkllm 77ae22a5d1dd8c15833c57e390de4afc3aa4f1867c5d24b0a16e32e8d1e20adf v1.3.1/RK3576/Qwen3-VL-2B-Instruct_RK3576_w8a8.rkllm 5ec488c125b6604cd623913ce5126db3f5075f5e1412f8d8f912ca14c2513dbc v1.3.1/RK3576/Qwen3-VL-2B-Instruct_vision_RK3576.rknn e8fff0de10e2f48ba13af759faca3a44cc07f661e95aeca412850a7eba08a92e v1.3.1/RK3588/Qwen3-VL-2B-Instruct_RK3588_w8a8.rkllm e90e160835c1a4d3d77095954aa518da513179ee0692b1bcc7a73f971f74d5a2 v1.3.1/RK3588/Qwen3-VL-2B-Instruct_vision_RK3588.rknn ```
## Credits [Qwen](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct) provides the original model. Conversion uses [Rockchip RKLLM](https://github.com/airockchip/rknn-llm/tree/release-v1.3.1), [RKNN Toolkit2](https://github.com/airockchip/rknn-toolkit2), and the [conversion wrapper](https://github.com/Hanzo-Huang/rknn-llm).