Add Simplified Chinese README and language navigation

4f4a25e verified 9 days ago

7.54 kB

---
license: apache-2.0
base_model: Qwen/Qwen2.5-VL-3B-Instruct
library_name: rkllm
pipeline_tag: image-text-to-text
tags: [rkllm, rockchip, rk3576, rk3588, qwen2.5-vl]
---
**English** | [简体中文](README_zh-CN.md)

## 🚀 Deploy with reComputer AI Lab

Explore and deploy AI models on Rockchip **RK3576** and **RK3588** devices with **reComputer AI Lab**.

[![reComputer AI Lab — explore models and deployment guides](https://files.seeedstudio.com/Edge_box/reComputer_Lab.webp)](https://sensecraft.seeed.cc/ai-lab/en/models?device=rk3588)

### ⚡ Run with one command

With Docker installed, choose your model and board in reComputer AI Lab, copy the deployment command, and run it in your device’s terminal.

**[Explore Models & Deployment Guides →](https://sensecraft.seeed.cc/ai-lab/en/models?device=rk3588)**

---


# Qwen2.5-VL-3B-Instruct · RKLLM

Text and image chat on Rockchip RK3576 and RK3588, converted from [Qwen/Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct).

* Max context length: 8192

- **Conversion toolkit:** RKLLM Toolkit 1.3.1 and RKNN Toolkit2 2.5.0.
- **Inputs / output:** text and one image → text. Video and audio input are unavailable in this runtime.
- **License:** Apache 2.0; see the [original model](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) and [license](LICENSE).

## Choose your files

Use files for your board's exact SoC. RK3576 offers compact W4A16-g128 and W8A8; RK3588 uses W8A8. Download size is not runtime RAM.

| Board | Quantization | Size | Language model |
| --- | --- | --- | --- |
| RK3576 | W4A16-g128 | 2.60 GiB | [Qwen2.5-VL-3B-Instruct_RK3576_w4a16_g128.rkllm](v1.3.1/RK3576/Qwen2.5-VL-3B-Instruct_RK3576_w4a16_g128.rkllm) |
| RK3576 | w8a8 | 3.70 GiB | [Qwen2.5-VL-3B-Instruct_RK3576_w8a8.rkllm](v1.3.1/RK3576/Qwen2.5-VL-3B-Instruct_RK3576_w8a8.rkllm) |
| RK3588 | w8a8 | 3.51 GiB | [Qwen2.5-VL-3B-Instruct_RK3588_w8a8.rkllm](v1.3.1/RK3588/Qwen2.5-VL-3B-Instruct_RK3588_w8a8.rkllm) |

Every language model requires its board's matching vision encoder below. Both RK3576 language variants share the RK3576 encoder.

| Board | Size | Vision encoder |
| --- | --- | --- |
| RK3576 | 1.52 GiB | [Qwen2.5-VL-3B-Instruct_vision_RK3576.rknn](v1.3.1/RK3576/Qwen2.5-VL-3B-Instruct_vision_RK3576.rknn) |
| RK3588 | 1.38 GiB | [Qwen2.5-VL-3B-Instruct_vision_RK3588.rknn](v1.3.1/RK3588/Qwen2.5-VL-3B-Instruct_vision_RK3588.rknn) |

## Run on your board

The vision encoders require **RKNN Runtime 2.5.0** and the language models require **RKLLM Runtime 1.3.1**. Set `RKLLM_RUNTIME_IMAGE` to an ARM64 runtime image with those versions. The validation used a locally built image; a model-specific Docker image is not included in this release.

Download the RK3576 W4A16-g128 language model and matching encoder:

```bash
hf download HanzoHuang/Qwen2.5-VL-3B-Instruct-RKLLM \
  v1.3.1/RK3576/Qwen2.5-VL-3B-Instruct_RK3576_w4a16_g128.rkllm \
  v1.3.1/RK3576/Qwen2.5-VL-3B-Instruct_vision_RK3576.rknn \
  --local-dir ./models
sudo docker run --rm -it --privileged \
  -p 8001:8001 -v /dev:/dev \
  -v "$PWD/models/v1.3.1/RK3576:/app/models:ro" \
  -e MODEL_KIND=vlm -e TARGET_PLATFORM=rk3576 -e RKLLM_EMBED_FLASH=1 \
  -e MODEL_FILE=Qwen2.5-VL-3B-Instruct_RK3576_w4a16_g128.rkllm \
  -e VISION_MODEL_FILE=Qwen2.5-VL-3B-Instruct_vision_RK3576.rknn \
  ${RKLLM_RUNTIME_IMAGE:?Set RKLLM_RUNTIME_IMAGE to an ARM64 RKLLM 1.3.1 / RKNN 2.5.0 image} \
  serve --max_context_len 4096
```

For RK3588, download both RK3588 files, mount that directory, set `TARGET_PLATFORM=rk3588`, and select its W8A8 language filename. Use the [runtime documentation](https://github.com/Hanzo-Huang/rkllm-docker) for health checks and OpenAI-compatible image requests.

## Release notes

- Source acquired from ModelScope `Qwen/Qwen2.5-VL-3B-Instruct`, pinned revision `1b5a7674d40a0ac9acef541a46fed12d710bd36b`. The HF link above identifies the canonical upstream model; no equivalence to an HF Git commit is assumed.
- Export context: **8192 tokens**. Functional API tests used **4096**; the speed/memory benchmark used **2048**. Longer runtime context has not been validated on these boards.
- Vision export follows the SDK placement rule, including a CPU-targeted patch-embedding convolution.
- Fixed vision input: RGB, 448 × 448, batch 1, 256 image tokens, CLIP mean [0.48145466, 0.4578275, 0.40821073] and std [0.26862954, 0.26130258, 0.27577711]. The runtime pads to a square using RGB 127, then resizes bilinearly. The encoder returns one pooler output with 2048 channels per image token.
- Language calibration: the SDK's 20 image-and-text examples, captured using the source processor and language-model input hook. W4A16-g128 uses GRQ; W8A8 uses normal quantization. Float32 CUDA loading; two NPU cores on RK3576 and three on RK3588.
- Export compatibility uses the pooler output under Transformers 5, imports only the requested calibration model family, and supplies a temporary nested text_config view for the SDK language loader. It restores the source mrope_section rotary configuration after SDK model loading. Original checkpoint weights and configuration files are unchanged.
- All three variants passed seven API smoke checks on their target board: arithmetic, independent requests, SSE, exact conversation recall, image description, image color over SSE, and text after an image. Tests used the baked RKLLM 1.3.1 application with RKNN Runtime 2.5.0 and `embed_flash=1`.
- RK3576 vision encoder cosine against original float32 embeddings: 0.99772957.
- RK3588 vision encoder cosine against original float32 embeddings: 0.99708773.

These are functional and numerical smoke checks, not a broad quality benchmark. Quantization and fixed image resolution may reduce accuracy.

Measured on the target boards:

| Board | Quantization | Decode tok/s | Startup RAM (GB) | Stable RAM (GB) | Date |
| --- | --- | ---: | ---: | ---: | --- |
| RK3576 | w4a16-g128 | 6.17 | 4.19 | 4.14 | 2026-10-23 |
| RK3576 | w8a8 | 5.13 | 5.32 | 5.32 | 2026-10-23 |
| RK3588 | w8a8 | 8.47 | 4.97 | 5.00 | 2026-10-23 |

Speed is the median native decode rate over three 128-token runs after a 32-token warmup, greedy decoding, EOS ignored, context 2048. RAM is additional board usage above a five-second idle baseline: sampled loading peak and ten-second ready-state median, in decimal GB. The matched encoder is loaded; `embed_flash=1`. First-token timing stays in the private report. Measurements depend on the board, clocks, cooling, and background load.



<details>
<summary>SHA-256 checksums</summary>

```text
de7152bfb6c2f532a048ac0e50a353a7a74dc8f10009af8f99e80fb273eda830  v1.3.1/RK3576/Qwen2.5-VL-3B-Instruct_RK3576_w4a16_g128.rkllm
41f1488381f16666ce89545028f9ec8457052be591c036cd68a309af7d671612  v1.3.1/RK3576/Qwen2.5-VL-3B-Instruct_RK3576_w8a8.rkllm
e5bb20a18703232c14870c0abb157950e5b98f3c78b9c275a1c602c17efd85c3  v1.3.1/RK3576/Qwen2.5-VL-3B-Instruct_vision_RK3576.rknn
3621ae4e255f08fba6b7b4a46ccfda3775bd4b8da6cdf4336aeff37947b8eba3  v1.3.1/RK3588/Qwen2.5-VL-3B-Instruct_RK3588_w8a8.rkllm
bc6620d506b583b2c4bbd51fea9aecdf07d109a5cbbb5e7e35906de2041e009a  v1.3.1/RK3588/Qwen2.5-VL-3B-Instruct_vision_RK3588.rknn
```

</details>

## Credits

[Qwen](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) provides the original model. Conversion uses [Rockchip RKLLM](https://github.com/airockchip/rknn-llm/tree/release-v1.3.1), [RKNN Toolkit2](https://github.com/airockchip/rknn-toolkit2), and the [conversion wrapper](https://github.com/Hanzo-Huang/rknn-llm).

	---
	license: apache-2.0
	base_model: Qwen/Qwen2.5-VL-3B-Instruct
	library_name: rkllm
	pipeline_tag: image-text-to-text
	tags: [rkllm, rockchip, rk3576, rk3588, qwen2.5-vl]
	---

	English \| [简体中文](README_zh-CN.md)

	## 🚀 Deploy with reComputer AI Lab

	Explore and deploy AI models on Rockchip RK3576 and RK3588 devices with reComputer AI Lab.

	[![reComputer AI Lab — explore models and deployment guides](https://files.seeedstudio.com/Edge_box/reComputer_Lab.webp)](https://sensecraft.seeed.cc/ai-lab/en/models?device=rk3588)

	### ⚡ Run with one command

	With Docker installed, choose your model and board in reComputer AI Lab, copy the deployment command, and run it in your device’s terminal.

	[Explore Models & Deployment Guides →](https://sensecraft.seeed.cc/ai-lab/en/models?device=rk3588)

	---


	# Qwen2.5-VL-3B-Instruct · RKLLM

	Text and image chat on Rockchip RK3576 and RK3588, converted from [Qwen/Qwen2.5-VL-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct).

	* Max context length: 8192

	- Conversion toolkit: RKLLM Toolkit 1.3.1 and RKNN Toolkit2 2.5.0.
	- Inputs / output: text and one image → text. Video and audio input are unavailable in this runtime.
	- License: Apache 2.0; see the [original model](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) and [license](LICENSE).

	## Choose your files

	Use files for your board's exact SoC. RK3576 offers compact W4A16-g128 and W8A8; RK3588 uses W8A8. Download size is not runtime RAM.

	\| Board \| Quantization \| Size \| Language model \|
	\| --- \| --- \| --- \| --- \|
	\| RK3576 \| W4A16-g128 \| 2.60 GiB \| [Qwen2.5-VL-3B-Instruct_RK3576_w4a16_g128.rkllm](v1.3.1/RK3576/Qwen2.5-VL-3B-Instruct_RK3576_w4a16_g128.rkllm) \|
	\| RK3576 \| w8a8 \| 3.70 GiB \| [Qwen2.5-VL-3B-Instruct_RK3576_w8a8.rkllm](v1.3.1/RK3576/Qwen2.5-VL-3B-Instruct_RK3576_w8a8.rkllm) \|
	\| RK3588 \| w8a8 \| 3.51 GiB \| [Qwen2.5-VL-3B-Instruct_RK3588_w8a8.rkllm](v1.3.1/RK3588/Qwen2.5-VL-3B-Instruct_RK3588_w8a8.rkllm) \|

	Every language model requires its board's matching vision encoder below. Both RK3576 language variants share the RK3576 encoder.

	\| Board \| Size \| Vision encoder \|
	\| --- \| --- \| --- \|
	\| RK3576 \| 1.52 GiB \| [Qwen2.5-VL-3B-Instruct_vision_RK3576.rknn](v1.3.1/RK3576/Qwen2.5-VL-3B-Instruct_vision_RK3576.rknn) \|
	\| RK3588 \| 1.38 GiB \| [Qwen2.5-VL-3B-Instruct_vision_RK3588.rknn](v1.3.1/RK3588/Qwen2.5-VL-3B-Instruct_vision_RK3588.rknn) \|

	## Run on your board

	The vision encoders require RKNN Runtime 2.5.0 and the language models require RKLLM Runtime 1.3.1. Set `RKLLM_RUNTIME_IMAGE` to an ARM64 runtime image with those versions. The validation used a locally built image; a model-specific Docker image is not included in this release.

	Download the RK3576 W4A16-g128 language model and matching encoder:

	```bash
	hf download HanzoHuang/Qwen2.5-VL-3B-Instruct-RKLLM \
	v1.3.1/RK3576/Qwen2.5-VL-3B-Instruct_RK3576_w4a16_g128.rkllm \
	v1.3.1/RK3576/Qwen2.5-VL-3B-Instruct_vision_RK3576.rknn \
	--local-dir ./models

	sudo docker run --rm -it --privileged \
	-p 8001:8001 -v /dev:/dev \
	-v "$PWD/models/v1.3.1/RK3576:/app/models:ro" \
	-e MODEL_KIND=vlm -e TARGET_PLATFORM=rk3576 -e RKLLM_EMBED_FLASH=1 \
	-e MODEL_FILE=Qwen2.5-VL-3B-Instruct_RK3576_w4a16_g128.rkllm \
	-e VISION_MODEL_FILE=Qwen2.5-VL-3B-Instruct_vision_RK3576.rknn \
	${RKLLM_RUNTIME_IMAGE:?Set RKLLM_RUNTIME_IMAGE to an ARM64 RKLLM 1.3.1 / RKNN 2.5.0 image} \
	serve --max_context_len 4096
	```

	For RK3588, download both RK3588 files, mount that directory, set `TARGET_PLATFORM=rk3588`, and select its W8A8 language filename. Use the [runtime documentation](https://github.com/Hanzo-Huang/rkllm-docker) for health checks and OpenAI-compatible image requests.

	## Release notes

	- Source acquired from ModelScope `Qwen/Qwen2.5-VL-3B-Instruct`, pinned revision `1b5a7674d40a0ac9acef541a46fed12d710bd36b`. The HF link above identifies the canonical upstream model; no equivalence to an HF Git commit is assumed.
	- Export context: 8192 tokens. Functional API tests used 4096; the speed/memory benchmark used 2048. Longer runtime context has not been validated on these boards.
	- Vision export follows the SDK placement rule, including a CPU-targeted patch-embedding convolution.
	- Fixed vision input: RGB, 448 × 448, batch 1, 256 image tokens, CLIP mean [0.48145466, 0.4578275, 0.40821073] and std [0.26862954, 0.26130258, 0.27577711]. The runtime pads to a square using RGB 127, then resizes bilinearly. The encoder returns one pooler output with 2048 channels per image token.
	- Language calibration: the SDK's 20 image-and-text examples, captured using the source processor and language-model input hook. W4A16-g128 uses GRQ; W8A8 uses normal quantization. Float32 CUDA loading; two NPU cores on RK3576 and three on RK3588.
	- Export compatibility uses the pooler output under Transformers 5, imports only the requested calibration model family, and supplies a temporary nested text_config view for the SDK language loader. It restores the source mrope_section rotary configuration after SDK model loading. Original checkpoint weights and configuration files are unchanged.
	- All three variants passed seven API smoke checks on their target board: arithmetic, independent requests, SSE, exact conversation recall, image description, image color over SSE, and text after an image. Tests used the baked RKLLM 1.3.1 application with RKNN Runtime 2.5.0 and `embed_flash=1`.
	- RK3576 vision encoder cosine against original float32 embeddings: 0.99772957.
	- RK3588 vision encoder cosine against original float32 embeddings: 0.99708773.

	These are functional and numerical smoke checks, not a broad quality benchmark. Quantization and fixed image resolution may reduce accuracy.

	Measured on the target boards:

	\| Board \| Quantization \| Decode tok/s \| Startup RAM (GB) \| Stable RAM (GB) \| Date \|
	\| --- \| --- \| ---: \| ---: \| ---: \| --- \|
	\| RK3576 \| w4a16-g128 \| 6.17 \| 4.19 \| 4.14 \| 2026-10-23 \|
	\| RK3576 \| w8a8 \| 5.13 \| 5.32 \| 5.32 \| 2026-10-23 \|
	\| RK3588 \| w8a8 \| 8.47 \| 4.97 \| 5.00 \| 2026-10-23 \|

	Speed is the median native decode rate over three 128-token runs after a 32-token warmup, greedy decoding, EOS ignored, context 2048. RAM is additional board usage above a five-second idle baseline: sampled loading peak and ten-second ready-state median, in decimal GB. The matched encoder is loaded; `embed_flash=1`. First-token timing stays in the private report. Measurements depend on the board, clocks, cooling, and background load.



	<details>
	<summary>SHA-256 checksums</summary>

	```text
	de7152bfb6c2f532a048ac0e50a353a7a74dc8f10009af8f99e80fb273eda830 v1.3.1/RK3576/Qwen2.5-VL-3B-Instruct_RK3576_w4a16_g128.rkllm
	41f1488381f16666ce89545028f9ec8457052be591c036cd68a309af7d671612 v1.3.1/RK3576/Qwen2.5-VL-3B-Instruct_RK3576_w8a8.rkllm
	e5bb20a18703232c14870c0abb157950e5b98f3c78b9c275a1c602c17efd85c3 v1.3.1/RK3576/Qwen2.5-VL-3B-Instruct_vision_RK3576.rknn
	3621ae4e255f08fba6b7b4a46ccfda3775bd4b8da6cdf4336aeff37947b8eba3 v1.3.1/RK3588/Qwen2.5-VL-3B-Instruct_RK3588_w8a8.rkllm
	bc6620d506b583b2c4bbd51fea9aecdf07d109a5cbbb5e7e35906de2041e009a v1.3.1/RK3588/Qwen2.5-VL-3B-Instruct_vision_RK3588.rknn
	```

	</details>

	## Credits

	[Qwen](https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct) provides the original model. Conversion uses [Rockchip RKLLM](https://github.com/airockchip/rknn-llm/tree/release-v1.3.1), [RKNN Toolkit2](https://github.com/airockchip/rknn-toolkit2), and the [conversion wrapper](https://github.com/Hanzo-Huang/rknn-llm).