File size: 2,917 Bytes
99d5821 f95bded 234d8e5 c8c79fd f95bded 234d8e5 c8c79fd 234d8e5 99d5821 234d8e5 a6ad9ad 234d8e5 a6ad9ad 234d8e5 a6ad9ad 234d8e5 a6ad9ad c8c79fd a6ad9ad 234d8e5 a6ad9ad ab9bf33 a6ad9ad ab9bf33 a6ad9ad 234d8e5 a6ad9ad 234d8e5 a6ad9ad 234d8e5 a6ad9ad 234d8e5 c8c79fd f95bded a6ad9ad c8c79fd 234d8e5 a6ad9ad 234d8e5 a6ad9ad 234d8e5 a6ad9ad 234d8e5 a6ad9ad 234d8e5 a6ad9ad | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 | ---
license: apache-2.0
base_model: Qwen/Qwen2.5-3B-Instruct
pipeline_tag: text-generation
library_name: rkllm
tags:
- rkllm
- rockchip
- rk3576
- rk3588
- qwen
- qwen2.5
---
# Qwen2.5-3B-Instruct · RKLLM
Text generation on Rockchip RK3576 and RK3588, converted from [Qwen/Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct).
- **Conversion toolkit:** RKLLM Toolkit 1.2.3.
- **Inputs / output:** text → text.
- **License:** `apache-2.0`; the [upstream model card](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct) describes its terms and intended use.
## Choose your files
Use the files for your board’s exact SoC. `w4a16` uses 4-bit weights and 16-bit activations; `w4a16_g128` adds a weight group size of 128. `w8a8` uses 8-bit weights and activations. File size below is download size, not total runtime RAM.
| Board | Quantization | Size | Language model |
| --- | --- | --- | --- |
| RK3576 | `w4a16` | 2.22 GiB | [Qwen2.5-3B-Instruct_RK3576_w4a16.rkllm](RK3576/Qwen2.5-3B-Instruct_RK3576_w4a16.rkllm) |
| RK3576 | `w8a8` | 3.51 GiB | [Qwen2.5-3B-Instruct_RK3576_w8a8.rkllm](RK3576/Qwen2.5-3B-Instruct_RK3576_w8a8.rkllm) |
| RK3588 | `w8a8` | 3.50 GiB | [Qwen2.5-3B-Instruct_RK3588_w8a8.rkllm](RK3588/Qwen2.5-3B-Instruct_RK3588_w8a8.rkllm) |
## Run on your board
With Docker installed on a matching Rockchip board, this example starts the **RK3576 w4a16** model and its API on port 8001:
```bash
sudo docker run --rm -d --name rkllm \
--privileged -v /dev:/dev -p 8001:8001 \
ghcr.io/hanzo-huang/rkllm-docker/llm/qwen2.5-3b-instruct:rk3576-w4a16
```
The image includes its model file. See [rkllm-docker](https://github.com/Hanzo-Huang/rkllm-docker#readme) for API requests, other board variants, and configuration.
To download the files separately instead:
```bash
hf download HanzoHuang/Qwen2.5-3B-Instruct-RKLLM \
RK3576/Qwen2.5-3B-Instruct_RK3576_w4a16.rkllm \
--local-dir ./models
```
These are native RKLLM/RKNN artifacts. Use a compatible RKLLM runtime and the model’s correct chat template when integrating them in another application.
## Release notes
No per-variant board benchmark is published here. Quantized output may differ from the original model; check responses on your own workloads. Use the toolkit version listed above when checking runtime compatibility.
<details>
<summary>SHA-256 checksums</summary>
```text
5f2480e10a794848c8d4a5a21a61c015d96570f2a58e4543897b694b59576908 RK3576/Qwen2.5-3B-Instruct_RK3576_w4a16.rkllm
5a14ed85f65d3c2890ef8e2b4ab9f094bf9b0992ddebd6f2b9ad8d3b539897b9 RK3576/Qwen2.5-3B-Instruct_RK3576_w8a8.rkllm
054a4ac54ea7d483ac17431df5286eb6b2a81d351fbb44b55dc4491f71a7ea46 RK3588/Qwen2.5-3B-Instruct_RK3588_w8a8.rkllm
```
</details>
## Credits
Original model: [Qwen/Qwen2.5-3B-Instruct](https://huggingface.co/Qwen/Qwen2.5-3B-Instruct). Conversion tools: [Rockchip RKLLM](https://github.com/airockchip/rknn-llm).
|