Add Simplified Chinese README and language navigation

c3e584f verified 1 day ago

7.87 kB

---
license: apache-2.0
base_model: google/gemma-4-E2B-it
pipeline_tag: image-text-to-text
library_name: rkllm
tags:
- rkllm
- rockchip
- rk3576
- rk3588
- gemma-4
- image-text-to-text
- audio-text-to-text
---
**English** | [简体中文](README_zh-CN.md)

## 🚀 Deploy with reComputer AI Lab

Explore and deploy AI models on Rockchip **RK3576** and **RK3588** devices with **reComputer AI Lab**.

[![reComputer AI Lab — explore models and deployment guides](https://files.seeedstudio.com/Edge_box/reComputer_Lab.webp)](https://sensecraft.seeed.cc/ai-lab/en/models?device=rk3588)

### ⚡ Run with one command

With Docker installed, choose your model and board in reComputer AI Lab, copy the deployment command, and run it in your device’s terminal.

**[Explore Models & Deployment Guides →](https://sensecraft.seeed.cc/ai-lab/en/models?device=rk3588)**

---


# Gemma-4-E2B-it · RKLLM

* Max context length: 4096

Text generation, image questions, and short audio transcription on Rockchip RK3576 and RK3588, converted from [google/gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it).

**RK3576 preview:** image and audio requests work in the functional tests, but the vision encoder misses the numerical accuracy target. RK3576 W8A8 also fails one exact-recall check. See the measured results below before choosing this variant. RK3588 W8A8 passes the functional and encoder checks performed here.

- **Conversion:** RKLLM Toolkit 1.3.1; vision/audio encoders built with RKNN Toolkit2 2.3.2.
- **Inputs / output:** text, one image, or short PCM16 WAV audio → text. Audio generation and video input are not implemented in this serving release.
- **Context:** exported at 4,096 tokens; board checks used 2,048, including media embeddings and generated text.
- **License:** Apache 2.0; see the [original model](https://huggingface.co/google/gemma-4-E2B-it) for terms and intended use.

## Choose your files

The new matched multimodal set is under **`v1.3.1/`**. Use the exact SoC for your board. Each variant needs its language file and vision encoder; add the audio encoder to enable speech input. The two RK3576 variants share the same encoders.

| Board | Quantization | Language size | Language model |
| --- | --- | --- | --- |
| RK3576 | `w4a16_g128` | 6.65 GiB | [Language](v1.3.1/RK3576/gemma-4-E2B-it_RK3576_w4a16_g128.rkllm) |
| RK3576 | `w8a8` | 7.37 GiB | [Language](v1.3.1/RK3576/gemma-4-E2B-it_RK3576_w8a8.rkllm) |
| RK3588 | `w8a8` | 7.36 GiB | [Language](v1.3.1/RK3588/gemma-4-E2B-it_RK3588_w8a8.rkllm) |

| Board | Vision encoder (480 × 480, 100 tokens) | Audio encoder |
| --- | --- | --- |
| RK3576 | [Vision · 1.09 GiB](v1.3.1/RK3576/gemma-4-E2B-it_vision_RK3576.rknn) | [Audio · 0.58 GiB](v1.3.1/RK3576/gemma-4-E2B-it_audio_RK3576.rknn) |
| RK3588 | [Vision · 0.46 GiB](v1.3.1/RK3588/gemma-4-E2B-it_vision_RK3588.rknn) | [Audio · 0.57 GiB](v1.3.1/RK3588/gemma-4-E2B-it_audio_RK3588.rknn) |

`w4a16_g128` uses 4-bit weights, 16-bit activations and group size 128. `w8a8` uses 8-bit weights and activations; these W8A8 exports use hybrid rate 0.50. Download sizes are not runtime RAM requirements. Use embedding reads from storage (`--embed_flash 1`) on the 8GB RK3576, as in the tested command below.

### Previous text-only release

The original RKLLM 1.3.0 artifacts are archived under **`v1.3.0/`**:

- [RK3576 text-only models](https://huggingface.co/HanzoHuang/gemma-4-E2B-it-RKLLM/tree/main/v1.3.0/RK3576)
- [RK3588 text-only model](https://huggingface.co/HanzoHuang/gemma-4-E2B-it-RKLLM/tree/main/v1.3.0/RK3588)
- [Original calibration dataset](v1.3.0/gemma-4-E2B-it_data_quant.json)

Their contents and checksums are unchanged; the old unversioned paths have moved. They accept **text only**; adding an encoder does not upgrade those language artifacts. Existing `/llm/gemma-4-e2b-it` Docker images contain the text-only release, and its build definitions now use the versioned paths. Use the complete `v1.3.1/` set for image/audio input.

## Run on your board

This RK3576 W4A16 example uses the rolling ARM64 `env-latest` image. Gemma vision/audio requires [PR #13](https://github.com/Hanzo-Huang/rkllm-docker/pull/13) to be merged and the main environment-image build to complete successfully. Download the three files first:

```bash
hf download HanzoHuang/gemma-4-E2B-it-RKLLM \
  v1.3.1/RK3576/gemma-4-E2B-it_RK3576_w4a16_g128.rkllm \
  v1.3.1/RK3576/gemma-4-E2B-it_vision_RK3576.rknn \
  v1.3.1/RK3576/gemma-4-E2B-it_audio_RK3576.rknn \
  --local-dir ./models
sudo docker run --pull=always --rm --name gemma4 --privileged -p 8001:8001 \
  -v "$PWD/models/v1.3.1/RK3576:/app/models:ro" \
  -e MODEL_KIND=vlm -e TARGET_PLATFORM=rk3576 -e VLM_MODEL_FAMILY=gemma4 \
  -e MODEL_FILE=gemma-4-E2B-it_RK3576_w4a16_g128.rkllm \
  -e VISION_MODEL_FILE=gemma-4-E2B-it_vision_RK3576.rknn \
  -e AUDIO_MODEL_FILE=gemma-4-E2B-it_audio_RK3576.rknn \
  ghcr.io/hanzo-huang/rkllm-docker:env-latest \
  serve --embed_flash 1 --max_context_len 4096 --timeout 600
```

For RK3588, select its three files and set `TARGET_PLATFORM=rk3588`. The API is OpenAI compatible at `/v1/chat/completions`; use `image_url` for an image or `input_audio` with base64 `data` and `format: "wav"` for speech. Supported audio is PCM16 WAV, mono/stereo, 8–48 kHz, longer than 10 ms and no more than 7.56 seconds. Split longer recordings before sending them. The audio encoder runs on one NPU core.

Runtime integration: [rkllm-docker Gemma support and model definitions](https://github.com/Hanzo-Huang/rkllm-docker/pull/13).

## Release notes

Measured on the matching RK3576 and RK3588 devkits with RKLLM runtime 1.3.1 and RKNN runtime 2.3.2:

| Variant | Functional checks | Encoder accuracy against original float32 PyTorch |
| --- | --- | --- |
| RK3576 W4A16_G128 | 14/14 passed | Vision cosine **0.97914620**, below the 0.99 target; audio **0.99222647**, passed |
| RK3576 W8A8 | 13/14 passed; exact recall returned `ORCHI D` instead of `ORCHID` | Same vision/audio encoders and limits as above |
| RK3588 W8A8 | 14/14 passed | Vision cosine **0.99876202** and audio **0.99216218**, passed |
Functional checks cover arithmetic, conversation recall, image questions, two short speech clips, streamed output, stereo resampling, invalid audio rejection, and transitions between text/image/audio. These are smoke tests, not a general quality or ASR benchmark. Encoder cosine results use one retained validation example per modality. All selected encoder outputs are finite; the RK3576 vision accuracy limitation remains unresolved. No generation-speed or RAM benchmark is claimed here.
The source weights were downloaded through ModelScope. Language calibration combines SDK image examples and general text prompts; it does not include the acceptance questions. The shared converter applies FP16-safe normalization and clipping and an RK3576 audio attention-layout correction through its existing single command. Rejected compiler/calibration experiments are excluded from this release.
New deployable-file hashes: [SHA256SUMS](v1.3.1/SHA256SUMS), with paths relative to `v1.3.1/`.
<details>
<summary>Previous text-only file checksums</summary>
```text
973008fd7ebed81e2a26dcf7dcc1c28fde631837e3e021c7f3007d6e82647165  v1.3.0/RK3576/gemma-4-E2B-it_RK3576_w4a16_g128.rkllm
b7357c01fcd51c8f896f06ee3a3746bc233031abd04e2fa0cebb5908b1382a01  v1.3.0/RK3576/gemma-4-E2B-it_RK3576_w8a8.rkllm
ae81d1e37eb54b40d2d366ef622a0c05acfb15537272e982efc1dfdd60557979  v1.3.0/RK3588/gemma-4-E2B-it_RK3588_w8a8.rkllm
```
</details>
## Credits
Original model: [Google Gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it). Conversion tools: [Rockchip RKLLM](https://github.com/airockchip/rknn-llm), [RKNN Toolkit2](https://github.com/airockchip/rknn-toolkit2), and the [shared conversion command](https://github.com/Hanzo-Huang/rknn-llm/pull/4).

	---
	license: apache-2.0
	base_model: google/gemma-4-E2B-it
	pipeline_tag: image-text-to-text
	library_name: rkllm
	tags:
	- rkllm
	- rockchip
	- rk3576
	- rk3588
	- gemma-4
	- image-text-to-text
	- audio-text-to-text
	---

	English \| [简体中文](README_zh-CN.md)

	## 🚀 Deploy with reComputer AI Lab

	Explore and deploy AI models on Rockchip RK3576 and RK3588 devices with reComputer AI Lab.

	[![reComputer AI Lab — explore models and deployment guides](https://files.seeedstudio.com/Edge_box/reComputer_Lab.webp)](https://sensecraft.seeed.cc/ai-lab/en/models?device=rk3588)

	### ⚡ Run with one command

	With Docker installed, choose your model and board in reComputer AI Lab, copy the deployment command, and run it in your device’s terminal.

	[Explore Models & Deployment Guides →](https://sensecraft.seeed.cc/ai-lab/en/models?device=rk3588)

	---


	# Gemma-4-E2B-it · RKLLM

	* Max context length: 4096

	Text generation, image questions, and short audio transcription on Rockchip RK3576 and RK3588, converted from [google/gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it).

	RK3576 preview: image and audio requests work in the functional tests, but the vision encoder misses the numerical accuracy target. RK3576 W8A8 also fails one exact-recall check. See the measured results below before choosing this variant. RK3588 W8A8 passes the functional and encoder checks performed here.

	- Conversion: RKLLM Toolkit 1.3.1; vision/audio encoders built with RKNN Toolkit2 2.3.2.
	- Inputs / output: text, one image, or short PCM16 WAV audio → text. Audio generation and video input are not implemented in this serving release.
	- Context: exported at 4,096 tokens; board checks used 2,048, including media embeddings and generated text.
	- License: Apache 2.0; see the [original model](https://huggingface.co/google/gemma-4-E2B-it) for terms and intended use.

	## Choose your files

	The new matched multimodal set is under `v1.3.1/`. Use the exact SoC for your board. Each variant needs its language file and vision encoder; add the audio encoder to enable speech input. The two RK3576 variants share the same encoders.

	\| Board \| Quantization \| Language size \| Language model \|
	\| --- \| --- \| --- \| --- \|
	\| RK3576 \| `w4a16_g128` \| 6.65 GiB \| [Language](v1.3.1/RK3576/gemma-4-E2B-it_RK3576_w4a16_g128.rkllm) \|
	\| RK3576 \| `w8a8` \| 7.37 GiB \| [Language](v1.3.1/RK3576/gemma-4-E2B-it_RK3576_w8a8.rkllm) \|
	\| RK3588 \| `w8a8` \| 7.36 GiB \| [Language](v1.3.1/RK3588/gemma-4-E2B-it_RK3588_w8a8.rkllm) \|

	\| Board \| Vision encoder (480 × 480, 100 tokens) \| Audio encoder \|
	\| --- \| --- \| --- \|
	\| RK3576 \| [Vision · 1.09 GiB](v1.3.1/RK3576/gemma-4-E2B-it_vision_RK3576.rknn) \| [Audio · 0.58 GiB](v1.3.1/RK3576/gemma-4-E2B-it_audio_RK3576.rknn) \|
	\| RK3588 \| [Vision · 0.46 GiB](v1.3.1/RK3588/gemma-4-E2B-it_vision_RK3588.rknn) \| [Audio · 0.57 GiB](v1.3.1/RK3588/gemma-4-E2B-it_audio_RK3588.rknn) \|

	`w4a16_g128` uses 4-bit weights, 16-bit activations and group size 128. `w8a8` uses 8-bit weights and activations; these W8A8 exports use hybrid rate 0.50. Download sizes are not runtime RAM requirements. Use embedding reads from storage (`--embed_flash 1`) on the 8GB RK3576, as in the tested command below.

	### Previous text-only release

	The original RKLLM 1.3.0 artifacts are archived under `v1.3.0/`:

	- [RK3576 text-only models](https://huggingface.co/HanzoHuang/gemma-4-E2B-it-RKLLM/tree/main/v1.3.0/RK3576)
	- [RK3588 text-only model](https://huggingface.co/HanzoHuang/gemma-4-E2B-it-RKLLM/tree/main/v1.3.0/RK3588)
	- [Original calibration dataset](v1.3.0/gemma-4-E2B-it_data_quant.json)

	Their contents and checksums are unchanged; the old unversioned paths have moved. They accept text only; adding an encoder does not upgrade those language artifacts. Existing `/llm/gemma-4-e2b-it` Docker images contain the text-only release, and its build definitions now use the versioned paths. Use the complete `v1.3.1/` set for image/audio input.

	## Run on your board

	This RK3576 W4A16 example uses the rolling ARM64 `env-latest` image. Gemma vision/audio requires [PR #13](https://github.com/Hanzo-Huang/rkllm-docker/pull/13) to be merged and the main environment-image build to complete successfully. Download the three files first:

	```bash
	hf download HanzoHuang/gemma-4-E2B-it-RKLLM \
	v1.3.1/RK3576/gemma-4-E2B-it_RK3576_w4a16_g128.rkllm \
	v1.3.1/RK3576/gemma-4-E2B-it_vision_RK3576.rknn \
	v1.3.1/RK3576/gemma-4-E2B-it_audio_RK3576.rknn \
	--local-dir ./models

	sudo docker run --pull=always --rm --name gemma4 --privileged -p 8001:8001 \
	-v "$PWD/models/v1.3.1/RK3576:/app/models:ro" \
	-e MODEL_KIND=vlm -e TARGET_PLATFORM=rk3576 -e VLM_MODEL_FAMILY=gemma4 \
	-e MODEL_FILE=gemma-4-E2B-it_RK3576_w4a16_g128.rkllm \
	-e VISION_MODEL_FILE=gemma-4-E2B-it_vision_RK3576.rknn \
	-e AUDIO_MODEL_FILE=gemma-4-E2B-it_audio_RK3576.rknn \
	ghcr.io/hanzo-huang/rkllm-docker:env-latest \
	serve --embed_flash 1 --max_context_len 4096 --timeout 600
	```

	For RK3588, select its three files and set `TARGET_PLATFORM=rk3588`. The API is OpenAI compatible at `/v1/chat/completions`; use `image_url` for an image or `input_audio` with base64 `data` and `format: "wav"` for speech. Supported audio is PCM16 WAV, mono/stereo, 8–48 kHz, longer than 10 ms and no more than 7.56 seconds. Split longer recordings before sending them. The audio encoder runs on one NPU core.

	Runtime integration: [rkllm-docker Gemma support and model definitions](https://github.com/Hanzo-Huang/rkllm-docker/pull/13).

	## Release notes

	Measured on the matching RK3576 and RK3588 devkits with RKLLM runtime 1.3.1 and RKNN runtime 2.3.2:

	\| Variant \| Functional checks \| Encoder accuracy against original float32 PyTorch \|
	\| --- \| --- \| --- \|
	\| RK3576 W4A16_G128 \| 14/14 passed \| Vision cosine 0.97914620, below the 0.99 target; audio 0.99222647, passed \|
	\| RK3576 W8A8 \| 13/14 passed; exact recall returned `ORCHI D` instead of `ORCHID` \| Same vision/audio encoders and limits as above \|
	\| RK3588 W8A8 \| 14/14 passed \| Vision cosine 0.99876202 and audio 0.99216218, passed \|

	Functional checks cover arithmetic, conversation recall, image questions, two short speech clips, streamed output, stereo resampling, invalid audio rejection, and transitions between text/image/audio. These are smoke tests, not a general quality or ASR benchmark. Encoder cosine results use one retained validation example per modality. All selected encoder outputs are finite; the RK3576 vision accuracy limitation remains unresolved. No generation-speed or RAM benchmark is claimed here.

	The source weights were downloaded through ModelScope. Language calibration combines SDK image examples and general text prompts; it does not include the acceptance questions. The shared converter applies FP16-safe normalization and clipping and an RK3576 audio attention-layout correction through its existing single command. Rejected compiler/calibration experiments are excluded from this release.

	New deployable-file hashes: [SHA256SUMS](v1.3.1/SHA256SUMS), with paths relative to `v1.3.1/`.

	<details>
	<summary>Previous text-only file checksums</summary>

	```text
	973008fd7ebed81e2a26dcf7dcc1c28fde631837e3e021c7f3007d6e82647165 v1.3.0/RK3576/gemma-4-E2B-it_RK3576_w4a16_g128.rkllm
	b7357c01fcd51c8f896f06ee3a3746bc233031abd04e2fa0cebb5908b1382a01 v1.3.0/RK3576/gemma-4-E2B-it_RK3576_w8a8.rkllm
	ae81d1e37eb54b40d2d366ef622a0c05acfb15537272e982efc1dfdd60557979 v1.3.0/RK3588/gemma-4-E2B-it_RK3588_w8a8.rkllm
	```

	</details>

	## Credits

	Original model: [Google Gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it). Conversion tools: [Rockchip RKLLM](https://github.com/airockchip/rknn-llm), [RKNN Toolkit2](https://github.com/airockchip/rknn-toolkit2), and the [shared conversion command](https://github.com/Hanzo-Huang/rknn-llm/pull/4).