README.md

---
license: apache-2.0
base_model: google/gemma-4-E2B-it
pipeline_tag: image-text-to-text
library_name: rkllm
tags:
- rkllm
- rockchip
- rk3576
- rk3588
- gemma-4
- image-text-to-text
- audio-text-to-text
---

# Gemma-4-E2B-it · RKLLM

* Max context length: 4096

Text generation, image questions, and short audio transcription on Rockchip RK3576 and RK3588, converted from [google/gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it).

**RK3576 preview:** image and audio requests work in the functional tests, but the vision encoder misses the numerical accuracy target. RK3576 W8A8 also fails one exact-recall check. See the measured results below before choosing this variant. RK3588 W8A8 passes the functional and encoder checks performed here.

- **Conversion:** RKLLM Toolkit 1.3.1; vision/audio encoders built with RKNN Toolkit2 2.3.2.
- **Inputs / output:** text, one image, or short PCM16 WAV audio → text. Audio generation and video input are not implemented in this serving release.
- **Context:** exported at 4,096 tokens; board checks used 2,048, including media embeddings and generated text.
- **License:** Apache 2.0; see the [original model](https://huggingface.co/google/gemma-4-E2B-it) for terms and intended use.

## Choose your files

The new matched multimodal set is under **`v1.3.1/`**. Use the exact SoC for your board. Each variant needs its language file and vision encoder; add the audio encoder to enable speech input. The two RK3576 variants share the same encoders.

| Board | Quantization | Language size | Language model |
| --- | --- | --- | --- |
| RK3576 | `w4a16_g128` | 6.65 GiB | [Language](v1.3.1/RK3576/gemma-4-E2B-it_RK3576_w4a16_g128.rkllm) |
| RK3576 | `w8a8` | 7.37 GiB | [Language](v1.3.1/RK3576/gemma-4-E2B-it_RK3576_w8a8.rkllm) |
| RK3588 | `w8a8` | 7.36 GiB | [Language](v1.3.1/RK3588/gemma-4-E2B-it_RK3588_w8a8.rkllm) |

| Board | Vision encoder (480 × 480, 100 tokens) | Audio encoder |
| --- | --- | --- |
| RK3576 | [Vision · 1.09 GiB](v1.3.1/RK3576/gemma-4-E2B-it_vision_RK3576.rknn) | [Audio · 0.58 GiB](v1.3.1/RK3576/gemma-4-E2B-it_audio_RK3576.rknn) |
| RK3588 | [Vision · 0.46 GiB](v1.3.1/RK3588/gemma-4-E2B-it_vision_RK3588.rknn) | [Audio · 0.57 GiB](v1.3.1/RK3588/gemma-4-E2B-it_audio_RK3588.rknn) |

`w4a16_g128` uses 4-bit weights, 16-bit activations and group size 128. `w8a8` uses 8-bit weights and activations; these W8A8 exports use hybrid rate 0.50. Download sizes are not runtime RAM requirements. Use embedding reads from storage (`--embed_flash 1`) on the 8GB RK3576, as in the tested command below.

### Previous text-only release

The original RKLLM 1.3.0 artifacts are archived under **`v1.3.0/`**:

- [RK3576 text-only models](https://huggingface.co/HanzoHuang/gemma-4-E2B-it-RKLLM/tree/main/v1.3.0/RK3576)
- [RK3588 text-only model](https://huggingface.co/HanzoHuang/gemma-4-E2B-it-RKLLM/tree/main/v1.3.0/RK3588)
- [Original calibration dataset](v1.3.0/gemma-4-E2B-it_data_quant.json)

Their contents and checksums are unchanged; the old unversioned paths have moved. They accept **text only**; adding an encoder does not upgrade those language artifacts. Existing `/llm/gemma-4-e2b-it` Docker images contain the text-only release, and its build definitions now use the versioned paths. Use the complete `v1.3.1/` set for image/audio input.

## Run on your board

This RK3576 W4A16 example uses the rolling ARM64 `env-latest` image. Gemma vision/audio requires [PR #13](https://github.com/Hanzo-Huang/rkllm-docker/pull/13) to be merged and the main environment-image build to complete successfully. Download the three files first:

```bash
hf download HanzoHuang/gemma-4-E2B-it-RKLLM \
  v1.3.1/RK3576/gemma-4-E2B-it_RK3576_w4a16_g128.rkllm \
  v1.3.1/RK3576/gemma-4-E2B-it_vision_RK3576.rknn \
  v1.3.1/RK3576/gemma-4-E2B-it_audio_RK3576.rknn \
  --local-dir ./models

sudo docker run --pull=always --rm --name gemma4 --privileged -p 8001:8001 \
  -v "$PWD/models/v1.3.1/RK3576:/app/models:ro" \
  -e MODEL_KIND=vlm -e TARGET_PLATFORM=rk3576 -e VLM_MODEL_FAMILY=gemma4 \
  -e MODEL_FILE=gemma-4-E2B-it_RK3576_w4a16_g128.rkllm \
  -e VISION_MODEL_FILE=gemma-4-E2B-it_vision_RK3576.rknn \
  -e AUDIO_MODEL_FILE=gemma-4-E2B-it_audio_RK3576.rknn \
  ghcr.io/hanzo-huang/rkllm-docker:env-latest \
  serve --embed_flash 1 --max_context_len 4096 --timeout 600
```

For RK3588, select its three files and set `TARGET_PLATFORM=rk3588`. The API is OpenAI compatible at `/v1/chat/completions`; use `image_url` for an image or `input_audio` with base64 `data` and `format: "wav"` for speech. Supported audio is PCM16 WAV, mono/stereo, 8–48 kHz, longer than 10 ms and no more than 7.56 seconds. Split longer recordings before sending them. The audio encoder runs on one NPU core.

Runtime integration: [rkllm-docker Gemma support and model definitions](https://github.com/Hanzo-Huang/rkllm-docker/pull/13).

## Release notes

Measured on the matching RK3576 and RK3588 devkits with RKLLM runtime 1.3.1 and RKNN runtime 2.3.2:

| Variant | Functional checks | Encoder accuracy against original float32 PyTorch |
| --- | --- | --- |
| RK3576 W4A16_G128 | 14/14 passed | Vision cosine **0.97914620**, below the 0.99 target; audio **0.99222647**, passed |
| RK3576 W8A8 | 13/14 passed; exact recall returned `ORCHI D` instead of `ORCHID` | Same vision/audio encoders and limits as above |
| RK3588 W8A8 | 14/14 passed | Vision cosine **0.99876202** and audio **0.99216218**, passed |

Functional checks cover arithmetic, conversation recall, image questions, two short speech clips, streamed output, stereo resampling, invalid audio rejection, and transitions between text/image/audio. These are smoke tests, not a general quality or ASR benchmark. Encoder cosine results use one retained validation example per modality. All selected encoder outputs are finite; the RK3576 vision accuracy limitation remains unresolved. No generation-speed or RAM benchmark is claimed here.

The source weights were downloaded through ModelScope. Language calibration combines SDK image examples and general text prompts; it does not include the acceptance questions. The shared converter applies FP16-safe normalization and clipping and an RK3576 audio attention-layout correction through its existing single command. Rejected compiler/calibration experiments are excluded from this release.

New deployable-file hashes: [SHA256SUMS](v1.3.1/SHA256SUMS), with paths relative to `v1.3.1/`.

<details>
<summary>Previous text-only file checksums</summary>

```text
973008fd7ebed81e2a26dcf7dcc1c28fde631837e3e021c7f3007d6e82647165  v1.3.0/RK3576/gemma-4-E2B-it_RK3576_w4a16_g128.rkllm
b7357c01fcd51c8f896f06ee3a3746bc233031abd04e2fa0cebb5908b1382a01  v1.3.0/RK3576/gemma-4-E2B-it_RK3576_w8a8.rkllm
ae81d1e37eb54b40d2d366ef622a0c05acfb15537272e982efc1dfdd60557979  v1.3.0/RK3588/gemma-4-E2B-it_RK3588_w8a8.rkllm
```

</details>

## Credits

Original model: [Google Gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it). Conversion tools: [Rockchip RKLLM](https://github.com/airockchip/rknn-llm), [RKNN Toolkit2](https://github.com/airockchip/rknn-toolkit2), and the [shared conversion command](https://github.com/Hanzo-Huang/rknn-llm/pull/4).