Make RKLLM model card easier to use

746807a verified 3 days ago

3.03 kB

---
license: llama3.2
base_model: meta-llama/Llama-3.2-1B-Instruct
pipeline_tag: text-generation
library_name: rkllm
tags:
- rkllm
- rockchip
- rk3576
- rk3588
- llama
- llama-3.2
---
# Llama-3.2-1B-Instruct · RKLLM

Text generation on Rockchip RK3576 and RK3588, converted from [meta-llama/Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct).

- **Conversion toolkit:** RKLLM Toolkit 1.3.0.
- **Inputs / output:** text → text.
- **License:** `llama3.2`; the [upstream model card](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct) describes its terms and intended use.

## Choose your files

Use the files for your board’s exact SoC. `w4a16` uses 4-bit weights and 16-bit activations; `w4a16_g128` adds a weight group size of 128. `w8a8` uses 8-bit weights and activations. File size below is download size, not total runtime RAM.

| Board | Quantization | Size | Language model |
| --- | --- | --- | --- |
| RK3576 | `w4a16_g128` | 1.30 GiB | [Llama-3.2-1B-Instruct_RK3576_w4a16_g128.rkllm](RK3576/Llama-3.2-1B-Instruct_RK3576_w4a16_g128.rkllm) |
| RK3576 | `w8a8` | 1.68 GiB | [Llama-3.2-1B-Instruct_RK3576_w8a8.rkllm](RK3576/Llama-3.2-1B-Instruct_RK3576_w8a8.rkllm) |
| RK3588 | `w8a8` | 1.67 GiB | [Llama-3.2-1B-Instruct_RK3588_w8a8.rkllm](RK3588/Llama-3.2-1B-Instruct_RK3588_w8a8.rkllm) |

## Run on your board

With Docker installed on a matching Rockchip board, this example starts the **RK3576 w4a16_g128** model and its API on port 8001:
```bash
sudo docker run --rm -d --name rkllm \
  --privileged -v /dev:/dev -p 8001:8001 \
  ghcr.io/hanzo-huang/rkllm-docker/llm/llama-3.2-1b-instruct:rk3576-w4a16-g128
```
The image includes its model file. See [rkllm-docker](https://github.com/Hanzo-Huang/rkllm-docker#readme) for API requests, other board variants, and configuration.
To download the files separately instead:
```bash
hf download HanzoHuang/Llama-3.2-1B-Instruct-RKLLM \
  RK3576/Llama-3.2-1B-Instruct_RK3576_w4a16_g128.rkllm \
  --local-dir ./models
```
These are native RKLLM/RKNN artifacts. Use a compatible RKLLM runtime and the model’s correct chat template when integrating them in another application.
## Release notes
No per-variant board benchmark is published here. Quantized output may differ from the original model; check responses on your own workloads. Use the toolkit version listed above when checking runtime compatibility.
<details>
<summary>SHA-256 checksums</summary>
```text
bd6cbbf84e90b63d451e672a542858a5a6aa15ad2b45f83e1b5dbcd944ac3018  RK3576/Llama-3.2-1B-Instruct_RK3576_w4a16_g128.rkllm
9c4d103f7c486740ba75605c61a51b80d170df067645360ffad8464b50c764df  RK3576/Llama-3.2-1B-Instruct_RK3576_w8a8.rkllm
4403aec1c018c7ebf29c95730336e848aff8824e3be3bd6be7db44b4a4973bad  RK3588/Llama-3.2-1B-Instruct_RK3588_w8a8.rkllm
```
</details>
## Credits
Original model: [meta-llama/Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct). Conversion tools: [Rockchip RKLLM](https://github.com/airockchip/rknn-llm).

	---
	license: llama3.2
	base_model: meta-llama/Llama-3.2-1B-Instruct
	pipeline_tag: text-generation
	library_name: rkllm
	tags:
	- rkllm
	- rockchip
	- rk3576
	- rk3588
	- llama
	- llama-3.2
	---

	# Llama-3.2-1B-Instruct · RKLLM

	Text generation on Rockchip RK3576 and RK3588, converted from [meta-llama/Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct).

	- Conversion toolkit: RKLLM Toolkit 1.3.0.
	- Inputs / output: text → text.
	- License: `llama3.2`; the [upstream model card](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct) describes its terms and intended use.

	## Choose your files

	Use the files for your board’s exact SoC. `w4a16` uses 4-bit weights and 16-bit activations; `w4a16_g128` adds a weight group size of 128. `w8a8` uses 8-bit weights and activations. File size below is download size, not total runtime RAM.

	\| Board \| Quantization \| Size \| Language model \|
	\| --- \| --- \| --- \| --- \|
	\| RK3576 \| `w4a16_g128` \| 1.30 GiB \| [Llama-3.2-1B-Instruct_RK3576_w4a16_g128.rkllm](RK3576/Llama-3.2-1B-Instruct_RK3576_w4a16_g128.rkllm) \|
	\| RK3576 \| `w8a8` \| 1.68 GiB \| [Llama-3.2-1B-Instruct_RK3576_w8a8.rkllm](RK3576/Llama-3.2-1B-Instruct_RK3576_w8a8.rkllm) \|
	\| RK3588 \| `w8a8` \| 1.67 GiB \| [Llama-3.2-1B-Instruct_RK3588_w8a8.rkllm](RK3588/Llama-3.2-1B-Instruct_RK3588_w8a8.rkllm) \|

	## Run on your board

	With Docker installed on a matching Rockchip board, this example starts the RK3576 w4a16_g128 model and its API on port 8001:

	```bash
	sudo docker run --rm -d --name rkllm \
	--privileged -v /dev:/dev -p 8001:8001 \
	ghcr.io/hanzo-huang/rkllm-docker/llm/llama-3.2-1b-instruct:rk3576-w4a16-g128
	```

	The image includes its model file. See [rkllm-docker](https://github.com/Hanzo-Huang/rkllm-docker#readme) for API requests, other board variants, and configuration.

	To download the files separately instead:

	```bash
	hf download HanzoHuang/Llama-3.2-1B-Instruct-RKLLM \
	RK3576/Llama-3.2-1B-Instruct_RK3576_w4a16_g128.rkllm \
	--local-dir ./models
	```

	These are native RKLLM/RKNN artifacts. Use a compatible RKLLM runtime and the model’s correct chat template when integrating them in another application.

	## Release notes

	No per-variant board benchmark is published here. Quantized output may differ from the original model; check responses on your own workloads. Use the toolkit version listed above when checking runtime compatibility.

	<details>
	<summary>SHA-256 checksums</summary>

	```text
	bd6cbbf84e90b63d451e672a542858a5a6aa15ad2b45f83e1b5dbcd944ac3018 RK3576/Llama-3.2-1B-Instruct_RK3576_w4a16_g128.rkllm
	9c4d103f7c486740ba75605c61a51b80d170df067645360ffad8464b50c764df RK3576/Llama-3.2-1B-Instruct_RK3576_w8a8.rkllm
	4403aec1c018c7ebf29c95730336e848aff8824e3be3bd6be7db44b4a4973bad RK3588/Llama-3.2-1B-Instruct_RK3588_w8a8.rkllm
	```

	</details>

	## Credits

	Original model: [meta-llama/Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct). Conversion tools: [Rockchip RKLLM](https://github.com/airockchip/rknn-llm).