Hugging Face's logo

HanzoHuang
/
MiniCPM3-4B-RKLLM

Text Generation
RKLLM
rockchip
rk3576
rk3588
minicpm3
Model card Files
xet
 Community
HanzoHuang commited on
Commit
b6e463a
·
verified ·
1 Parent(s): 4603140

Make RKLLM model card easier to use

Browse files
Files changed (1) hide show
  1. README.md +38 -25
README.md CHANGED
@@ -11,48 +11,61 @@ tags:
11
  - minicpm3
12
  ---
13
 
14
- # MiniCPM3-4B-RKLLM
15
 
16
- RKLLM-converted MiniCPM3-4B language-model artifacts for Rockchip RK3576 and RK3588 NPUs.
17
 
18
- These are hardware-specific `.rkllm` binaries, not Transformers checkpoints. They require a compatible Rockchip RKLLM runtime and cannot be loaded directly with Transformers, llama.cpp, or Ollama.
 
 
19
 
20
- ## Base model
21
 
22
- - Upstream model: [openbmb/MiniCPM3-4B](https://huggingface.co/openbmb/MiniCPM3-4B)
23
- - License: Apache-2.0
24
- - Model type: LLM (text only)
25
 
26
- ## Conversion and variants
27
-
28
- ### Toolkit version
 
 
29
 
30
- **RKLLM Toolkit: v1.3.0**
31
 
32
- Use a file built for the exact target SoC.
33
 
34
- | Target | Quantization | File | SHA256 |
35
- | --- | --- | --- | --- |
36
- | RK3576 | W4A16 (g128) | [MiniCPM3-4B_RK3576_w4a16_g128.rkllm](RK3576/MiniCPM3-4B_RK3576_w4a16_g128.rkllm) | `ec05211d17c4bb05a685588ff335608233b59722f6ecc296976d61dbc88c4128` |
37
- | RK3576 | W8A8 | [MiniCPM3-4B_RK3576_w8a8.rkllm](RK3576/MiniCPM3-4B_RK3576_w8a8.rkllm) | `3795c6a22e5f12de0c4243fc187da55f516b530de3125734874d340856160aec` |
38
- | RK3588 | W8A8 | [MiniCPM3-4B_RK3588_w8a8.rkllm](RK3588/MiniCPM3-4B_RK3588_w8a8.rkllm) | `3815a59593df7aebda90f86844caf2fd625d9d0e996e35aa31fd74790d10b130` |
39
 
40
- The repository also includes `MiniCPM3-4B_data_quant.json`, used as calibration data during conversion.
41
 
42
- ## Usage
43
 
44
  ```bash
45
  hf download HanzoHuang/MiniCPM3-4B-RKLLM \
46
  RK3576/MiniCPM3-4B_RK3576_w4a16_g128.rkllm \
47
- --local-dir MiniCPM3-4B-RKLLM
48
  ```
49
 
50
- Run the downloaded file with the RKLLM runtime and the upstream MiniCPM3 chat template. For Docker deployment, see [Hanzo-Huang/rkllm-docker](https://github.com/Hanzo-Huang/rkllm-docker).
51
 
52
- ## Limitations
 
 
 
 
 
 
 
 
 
 
 
53
 
54
- These artifacts are target-specific conversions. Validate quality, memory use, and runtime compatibility on your own hardware.
55
 
56
- ## Acknowledgements
57
 
58
- Thanks to OpenBMB, Rockchip, and the RKLLM community.
 
11
  - minicpm3
12
  ---
13
 
14
+ # MiniCPM3-4B · RKLLM
15
 
16
+ Text generation on Rockchip RK3576 and RK3588, converted from [openbmb/MiniCPM3-4B](https://huggingface.co/openbmb/MiniCPM3-4B).
17
 
18
+ - **Conversion toolkit:** RKLLM Toolkit 1.3.0.
19
+ - **Inputs / output:** text → text.
20
+ - **License:** `apache-2.0`; the [upstream model card](https://huggingface.co/openbmb/MiniCPM3-4B) describes its terms and intended use.
21
 
22
+ ## Choose your files
23
 
24
+ Use the files for your board’s exact SoC. `w4a16` uses 4-bit weights and 16-bit activations; `w4a16_g128` adds a weight group size of 128. `w8a8` uses 8-bit weights and activations. File size below is download size, not total runtime RAM.
 
 
25
 
26
+ | Board | Quantization | Size | Language model |
27
+ | --- | --- | --- | --- |
28
+ | RK3576 | `w4a16_g128` | 2.70 GiB | [MiniCPM3-4B_RK3576_w4a16_g128.rkllm](RK3576/MiniCPM3-4B_RK3576_w4a16_g128.rkllm) |
29
+ | RK3576 | `w8a8` | 4.22 GiB | [MiniCPM3-4B_RK3576_w8a8.rkllm](RK3576/MiniCPM3-4B_RK3576_w8a8.rkllm) |
30
+ | RK3588 | `w8a8` | 4.20 GiB | [MiniCPM3-4B_RK3588_w8a8.rkllm](RK3588/MiniCPM3-4B_RK3588_w8a8.rkllm) |
31
 
32
+ ## Run on your board
33
 
34
+ With Docker installed on a matching Rockchip board, this example starts the **RK3576 w4a16_g128** model and its API on port 8001:
35
 
36
+ ```bash
37
+ sudo docker run --rm -d --name rkllm \
38
+ --privileged -v /dev:/dev -p 8001:8001 \
39
+ ghcr.io/hanzo-huang/rkllm-docker/llm/minicpm3-4b:rk3576-w4a16-g128
40
+ ```
41
 
42
+ The image includes its model file. See [rkllm-docker](https://github.com/Hanzo-Huang/rkllm-docker#readme) for API requests, other board variants, and configuration.
43
 
44
+ To download the files separately instead:
45
 
46
  ```bash
47
  hf download HanzoHuang/MiniCPM3-4B-RKLLM \
48
  RK3576/MiniCPM3-4B_RK3576_w4a16_g128.rkllm \
49
+ --local-dir ./models
50
  ```
51
 
52
+ These are native RKLLM/RKNN artifacts. Use a compatible RKLLM runtime and the model’s correct chat template when integrating them in another application.
53
 
54
+ ## Release notes
55
+
56
+ No per-variant board benchmark is published here. Quantized output may differ from the original model; check responses on your own workloads. Use the toolkit version listed above when checking runtime compatibility.
57
+
58
+ <details>
59
+ <summary>SHA-256 checksums</summary>
60
+
61
+ ```text
62
+ ec05211d17c4bb05a685588ff335608233b59722f6ecc296976d61dbc88c4128 RK3576/MiniCPM3-4B_RK3576_w4a16_g128.rkllm
63
+ 3795c6a22e5f12de0c4243fc187da55f516b530de3125734874d340856160aec RK3576/MiniCPM3-4B_RK3576_w8a8.rkllm
64
+ 3815a59593df7aebda90f86844caf2fd625d9d0e996e35aa31fd74790d10b130 RK3588/MiniCPM3-4B_RK3588_w8a8.rkllm
65
+ ```
66
 
67
+ </details>
68
 
69
+ ## Credits
70
 
71
+ Original model: [openbmb/MiniCPM3-4B](https://huggingface.co/openbmb/MiniCPM3-4B). Conversion tools: [Rockchip RKLLM](https://github.com/airockchip/rknn-llm).