File size: 1,163 Bytes
34c8d79 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 | ---
license: apache-2.0
base_model: Qwen/Qwen2.5-Coder-7B-Instruct
tags:
- rkllm
- rk3576
- npu
- qwen2.5-coder
- edge-ai
---
# Qwen2.5-Coder-7B-Instruct — RK3576 (RKLLM)
Conversion of [Qwen/Qwen2.5-Coder-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct) to RKLLM format for **RK3576** NPU (NanoPi R76S, Orange Pi RK3576, and similar boards).
## Conversion details
- **Toolkit:** rkllm-toolkit 1.2.3
- **Compatible runtime:** rkllm-runtime >= 1.2.3
- **Quantization:** w4a16
- **NPU cores:** 2 (dual-core)
- **Max context:** 4096 tokens
- **Platform:** RK3576 (NOT compatible with RK3588)
## Usage with RKLLama
Place the `.rkllm` file in your models folder and create a `Modelfile` with the following content:
FROM="Qwen2.5-Coder-7B-Instruct-rk3576.rkllm"
HUGGINGFACE_PATH="Qwen/Qwen2.5-Coder-7B-Instruct"
TOKENIZER="/models/tokenizers/Qwen2.5-Coder-7B"
## Performance (NanoPi R76S, 16GB RAM)
- ~10-15 tok/s on dual-core NPU
## Important note
This model was compiled specifically for RK3576. If you're using an RK3588 (RK3588S, RK3588J), look for a different conversion — they are not binary compatible with each other.
|