File size: 2,950 Bytes
930568b 21e56b2 930568b 21e56b2 bebbf29 21e56b2 bebbf29 21e56b2 bebbf29 21e56b2 bebbf29 21e56b2 bebbf29 21e56b2 bebbf29 9eb4099 bebbf29 9eb4099 bebbf29 21e56b2 bebbf29 21e56b2 bebbf29 21e56b2 bebbf29 21e56b2 bebbf29 21e56b2 bebbf29 21e56b2 bebbf29 21e56b2 bebbf29 21e56b2 bebbf29 21e56b2 bebbf29 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 | ---
license: apache-2.0
base_model: Qwen/Qwen2-0.5B-Instruct
pipeline_tag: text-generation
library_name: rkllm
tags:
- rkllm
- rockchip
- rk3576
- rk3588
- qwen
- qwen2
---
# Qwen2-0.5B-Instruct · RKLLM
Text generation on Rockchip RK3576 and RK3588, converted from [Qwen/Qwen2-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct).
- **Conversion toolkit:** RKLLM Toolkit 1.3.0.
- **Inputs / output:** text → text.
- **License:** `apache-2.0`; the [upstream model card](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct) describes its terms and intended use.
## Choose your files
Use the files for your board’s exact SoC. `w4a16` uses 4-bit weights and 16-bit activations; `w4a16_g128` adds a weight group size of 128. `w8a8` uses 8-bit weights and activations. File size below is download size, not total runtime RAM.
| Board | Quantization | Size | Language model |
| --- | --- | --- | --- |
| RK3576 | `w4a16_g128` | 0.64 GiB | [Qwen2-0.5B-Instruct_RK3576_w4a16_g128.rkllm](RK3576/Qwen2-0.5B-Instruct_RK3576_w4a16_g128.rkllm) |
| RK3576 | `w8a8` | 0.76 GiB | [Qwen2-0.5B-Instruct_RK3576_w8a8.rkllm](RK3576/Qwen2-0.5B-Instruct_RK3576_w8a8.rkllm) |
| RK3588 | `w8a8` | 0.75 GiB | [Qwen2-0.5B-Instruct_RK3588_w8a8.rkllm](RK3588/Qwen2-0.5B-Instruct_RK3588_w8a8.rkllm) |
## Run on your board
With Docker installed on a matching Rockchip board, this example starts the **RK3576 w4a16_g128** model and its API on port 8001:
```bash
sudo docker run --rm -d --name rkllm \
--privileged -v /dev:/dev -p 8001:8001 \
ghcr.io/hanzo-huang/rkllm-docker/llm/qwen2-0.5b-instruct:rk3576-w4a16-g128
```
The image includes its model file. See [rkllm-docker](https://github.com/Hanzo-Huang/rkllm-docker#readme) for API requests, other board variants, and configuration.
To download the files separately instead:
```bash
hf download HanzoHuang/Qwen2-0.5B-Instruct-RKLLM \
RK3576/Qwen2-0.5B-Instruct_RK3576_w4a16_g128.rkllm \
--local-dir ./models
```
These are native RKLLM/RKNN artifacts. Use a compatible RKLLM runtime and the model’s correct chat template when integrating them in another application.
## Release notes
No per-variant board benchmark is published here. Quantized output may differ from the original model; check responses on your own workloads. Use the toolkit version listed above when checking runtime compatibility.
<details>
<summary>SHA-256 checksums</summary>
```text
a166cdcff5d8a2e33423d01d8bac6f7a6ad1e1130fbd0bfefdb8b69082113e87 RK3576/Qwen2-0.5B-Instruct_RK3576_w4a16_g128.rkllm
cdccb51e23131352823af8becf9ac4acb15cb30f0f961e89d16cb0fabd003414 RK3576/Qwen2-0.5B-Instruct_RK3576_w8a8.rkllm
24e99eeea1512d6e7f3f120f3344a510933301dfcb72bd64b24a9724ab48e1d8 RK3588/Qwen2-0.5B-Instruct_RK3588_w8a8.rkllm
```
</details>
## Credits
Original model: [Qwen/Qwen2-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct). Conversion tools: [Rockchip RKLLM](https://github.com/airockchip/rknn-llm).
|