--- license: apache-2.0 base_model: Qwen/Qwen3-VL-4B-Instruct library_name: rkllm pipeline_tag: image-text-to-text tags: [rkllm, rockchip, rk3576, rk3588, qwen3-vl] --- **English** | [简体中文](README_zh-CN.md) ## 🚀 Deploy with reComputer AI Lab Explore and deploy AI models on Rockchip **RK3576** and **RK3588** devices with **reComputer AI Lab**. [![reComputer AI Lab — explore models and deployment guides](https://files.seeedstudio.com/Edge_box/reComputer_Lab.webp)](https://sensecraft.seeed.cc/ai-lab/en/models?device=rk3588) ### ⚡ Run with one command With Docker installed, choose your model and board in reComputer AI Lab, copy the deployment command, and run it in your device’s terminal. **[Explore Models & Deployment Guides →](https://sensecraft.seeed.cc/ai-lab/en/models?device=rk3588)** --- # Qwen3-VL-4B-Instruct · RKLLM Text and image chat on Rockchip RK3576 and RK3588, converted from [Qwen/Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct). * Max context length: 8192 - **Conversion toolkit:** RKLLM Toolkit 1.3.1 and RKNN Toolkit2 2.3.2. - **Inputs / output:** text and one image → text. Video and audio input are unavailable in this runtime. - **License:** Apache 2.0; see the [original model](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct) and [license](LICENSE). ## Choose your files Use files for your board's exact SoC. RK3576 offers compact W4A16-g128 and W8A8; RK3588 uses W8A8. Download size is not runtime RAM. | Board | Quantization | Size | Language model | | --- | --- | --- | --- | | RK3576 | W4A16-g128 | 3.16 GiB | [Qwen3-VL-4B-Instruct_RK3576_w4a16_g128.rkllm](v1.3.1/RK3576/Qwen3-VL-4B-Instruct_RK3576_w4a16_g128.rkllm) | | RK3576 | w8a8 | 4.62 GiB | [Qwen3-VL-4B-Instruct_RK3576_w8a8.rkllm](v1.3.1/RK3576/Qwen3-VL-4B-Instruct_RK3576_w8a8.rkllm) | | RK3588 | w8a8 | 4.52 GiB | [Qwen3-VL-4B-Instruct_RK3588_w8a8.rkllm](v1.3.1/RK3588/Qwen3-VL-4B-Instruct_RK3588_w8a8.rkllm) | Every language model requires its board's matching vision encoder below. Both RK3576 language variants share the RK3576 encoder. | Board | Size | Vision encoder | | --- | --- | --- | | RK3576 | 0.83 GiB | [Qwen3-VL-4B-Instruct_vision_RK3576.rknn](v1.3.1/RK3576/Qwen3-VL-4B-Instruct_vision_RK3576.rknn) | | RK3588 | 0.81 GiB | [Qwen3-VL-4B-Instruct_vision_RK3588.rknn](v1.3.1/RK3588/Qwen3-VL-4B-Instruct_vision_RK3588.rknn) | ## Run on your board Download the RK3576 W4A16-g128 language model and matching encoder: ```bash hf download HanzoHuang/Qwen3-VL-4B-Instruct-RKLLM \ v1.3.1/RK3576/Qwen3-VL-4B-Instruct_RK3576_w4a16_g128.rkllm \ v1.3.1/RK3576/Qwen3-VL-4B-Instruct_vision_RK3576.rknn \ --local-dir ./models sudo docker run --rm -it --pull=always --privileged \ -p 8001:8001 -v /dev:/dev \ -v "$PWD/models/v1.3.1/RK3576:/app/models:ro" \ -e MODEL_KIND=vlm -e TARGET_PLATFORM=rk3576 -e RKLLM_EMBED_FLASH=1 \ -e MODEL_FILE=Qwen3-VL-4B-Instruct_RK3576_w4a16_g128.rkllm \ -e VISION_MODEL_FILE=Qwen3-VL-4B-Instruct_vision_RK3576.rknn \ ghcr.io/hanzo-huang/rkllm-docker:env-latest \ serve --max_context_len 4096 ``` For RK3588, download both RK3588 files, mount that directory, set `TARGET_PLATFORM=rk3588`, and select its W8A8 language filename. Use the [runtime documentation](https://github.com/Hanzo-Huang/rkllm-docker) for health checks and OpenAI-compatible image requests. ## Release notes - Source acquired from ModelScope `Qwen/Qwen3-VL-4B-Instruct`, pinned revision `0196dc7bb23f3c742616147c3254d0e4f1207787`. The HF link above identifies the canonical upstream model; no equivalence to an HF Git commit is assumed. - Export context: **8192 tokens**. Functional API tests used **4096**; the speed/memory benchmark used **2048**. Longer runtime context has not been validated on these boards. - Fixed vision input: RGB, 448 × 448, batch 1, 196 image tokens, mean/std 0.5. The runtime pads to a square using RGB 127, then resizes bilinearly. Vision output order is pooler followed by three deepstack features for each token. - Language calibration: the SDK's 20 image-and-text examples, captured using the source processor and language-model input hook. W4A16-g128 uses GRQ; W8A8 uses normal quantization. Float32 CUDA loading; two NPU cores on RK3576 and three on RK3588. - Export compatibility fixes select the four language-width vision outputs under Transformers 5 and import only the requested calibration model family. Generated upstream examples and source checkpoint weights are unchanged. - All three variants passed seven API smoke checks on their target board: arithmetic, independent requests, SSE, exact conversation recall, image description, image color over SSE, and text after an image. Tests used the baked RKLLM 1.3.1 runtime application with `embed_flash=1`. - RK3576 vision encoder cosine against original float32 embeddings: 0.99534508. - RK3588 vision encoder cosine against original float32 embeddings: 0.99543323. These are functional and numerical smoke checks, not a broad quality benchmark. Quantization and fixed image resolution may reduce accuracy. Measured on the target boards: | Board | Quantization | Decode tok/s | Startup RAM (GB) | Stable RAM (GB) | Date | | --- | --- | ---: | ---: | ---: | --- | | RK3576 | w4a16-g128 | 5.51 | 4.01 | 4.01 | 2026-10-23 | | RK3576 | w8a8 | 4.11 | 5.83 | 5.83 | 2026-10-23 | | RK3588 | w8a8 | 6.58 | 5.58 | 5.62 | 2026-10-22 | Speed is the median native decode rate over three 128-token runs after a 32-token warmup, greedy decoding, EOS ignored, context 2048. RAM is additional board usage above a five-second idle baseline: sampled loading peak and ten-second ready-state median, in decimal GB. The matched encoder is loaded; `embed_flash=1`. First-token timing stays in the private report. Measurements depend on the board, clocks, cooling, and background load. RK3576 measurements temporarily disabled zram swap and restored it after each run; both had zero swap growth. RK3588 retained its normal swap configuration.
SHA-256 checksums ```text 0a014ecf5f8513878e4c9fefd6c4ceec8f105005d2920588dbd5ded188a6c421 v1.3.1/RK3576/Qwen3-VL-4B-Instruct_RK3576_w4a16_g128.rkllm b4a132c43fa10a5ce86b81f100a1cfa6f0aa1ffbc477b20e32a38d4f64b50a40 v1.3.1/RK3576/Qwen3-VL-4B-Instruct_RK3576_w8a8.rkllm c8074176816c68ab635c70a7a0db3044a15f0f7b272fc45c0e6cc68875db159e v1.3.1/RK3576/Qwen3-VL-4B-Instruct_vision_RK3576.rknn cbf7b3bb2eec550f268fbc64fadcf53791283acdfba48fe0ff621aa46aba884b v1.3.1/RK3588/Qwen3-VL-4B-Instruct_RK3588_w8a8.rkllm 478e5922e5a28d130146bad9b22dfc66b4eeb53880477744bc3130eab1fb321d v1.3.1/RK3588/Qwen3-VL-4B-Instruct_vision_RK3588.rknn ```
## Credits [Qwen](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct) provides the original model. Conversion uses [Rockchip RKLLM](https://github.com/airockchip/rknn-llm/tree/release-v1.3.1), [RKNN Toolkit2](https://github.com/airockchip/rknn-toolkit2), and the [conversion wrapper](https://github.com/Hanzo-Huang/rknn-llm).