license: apache-2.0
base_model: google/gemma-4-E2B-it
pipeline_tag: image-text-to-text
library_name: rkllm
tags:
- rkllm
- rockchip
- rk3576
- rk3588
- gemma-4
- image-text-to-text
- audio-text-to-text
Gemma-4-E2B-it · RKLLM
- Max context length: 4096
Text generation, image questions, and short audio transcription on Rockchip RK3576 and RK3588, converted from google/gemma-4-E2B-it.
RK3576 preview: image and audio requests work in the functional tests, but the vision encoder misses the numerical accuracy target. RK3576 W8A8 also fails one exact-recall check. See the measured results below before choosing this variant. RK3588 W8A8 passes the functional and encoder checks performed here.
- Conversion: RKLLM Toolkit 1.3.1; vision/audio encoders built with RKNN Toolkit2 2.3.2.
- Inputs / output: text, one image, or short PCM16 WAV audio → text. Audio generation and video input are not implemented in this serving release.
- Context: exported at 4,096 tokens; board checks used 2,048, including media embeddings and generated text.
- License: Apache 2.0; see the original model for terms and intended use.
Choose your files
The new matched multimodal set is under v1.3.1/. Use the exact SoC for your board. Each variant needs its language file and vision encoder; add the audio encoder to enable speech input. The two RK3576 variants share the same encoders.
| Board | Quantization | Language size | Language model |
|---|---|---|---|
| RK3576 | w4a16_g128 |
6.65 GiB | Language |
| RK3576 | w8a8 |
7.37 GiB | Language |
| RK3588 | w8a8 |
7.36 GiB | Language |
| Board | Vision encoder (480 × 480, 100 tokens) | Audio encoder |
|---|---|---|
| RK3576 | Vision · 1.09 GiB | Audio · 0.58 GiB |
| RK3588 | Vision · 0.46 GiB | Audio · 0.57 GiB |
w4a16_g128 uses 4-bit weights, 16-bit activations and group size 128. w8a8 uses 8-bit weights and activations; these W8A8 exports use hybrid rate 0.50. Download sizes are not runtime RAM requirements. Use embedding reads from storage (--embed_flash 1) on the 8GB RK3576, as in the tested command below.
Previous text-only release
The original RKLLM 1.3.0 artifacts are archived under v1.3.0/:
Their contents and checksums are unchanged; the old unversioned paths have moved. They accept text only; adding an encoder does not upgrade those language artifacts. Existing /llm/gemma-4-e2b-it Docker images contain the text-only release, and its build definitions now use the versioned paths. Use the complete v1.3.1/ set for image/audio input.
Run on your board
This RK3576 W4A16 example uses the rolling ARM64 env-latest image. Gemma vision/audio requires PR #13 to be merged and the main environment-image build to complete successfully. Download the three files first:
hf download HanzoHuang/gemma-4-E2B-it-RKLLM \
v1.3.1/RK3576/gemma-4-E2B-it_RK3576_w4a16_g128.rkllm \
v1.3.1/RK3576/gemma-4-E2B-it_vision_RK3576.rknn \
v1.3.1/RK3576/gemma-4-E2B-it_audio_RK3576.rknn \
--local-dir ./models
sudo docker run --pull=always --rm --name gemma4 --privileged -p 8001:8001 \
-v "$PWD/models/v1.3.1/RK3576:/app/models:ro" \
-e MODEL_KIND=vlm -e TARGET_PLATFORM=rk3576 -e VLM_MODEL_FAMILY=gemma4 \
-e MODEL_FILE=gemma-4-E2B-it_RK3576_w4a16_g128.rkllm \
-e VISION_MODEL_FILE=gemma-4-E2B-it_vision_RK3576.rknn \
-e AUDIO_MODEL_FILE=gemma-4-E2B-it_audio_RK3576.rknn \
ghcr.io/hanzo-huang/rkllm-docker:env-latest \
serve --embed_flash 1 --max_context_len 4096 --timeout 600
For RK3588, select its three files and set TARGET_PLATFORM=rk3588. The API is OpenAI compatible at /v1/chat/completions; use image_url for an image or input_audio with base64 data and format: "wav" for speech. Supported audio is PCM16 WAV, mono/stereo, 8–48 kHz, longer than 10 ms and no more than 7.56 seconds. Split longer recordings before sending them. The audio encoder runs on one NPU core.
Runtime integration: rkllm-docker Gemma support and model definitions.
Release notes
Measured on the matching RK3576 and RK3588 devkits with RKLLM runtime 1.3.1 and RKNN runtime 2.3.2:
| Variant | Functional checks | Encoder accuracy against original float32 PyTorch |
|---|---|---|
| RK3576 W4A16_G128 | 14/14 passed | Vision cosine 0.97914620, below the 0.99 target; audio 0.99222647, passed |
| RK3576 W8A8 | 13/14 passed; exact recall returned ORCHI D instead of ORCHID |
Same vision/audio encoders and limits as above |
| RK3588 W8A8 | 14/14 passed | Vision cosine 0.99876202 and audio 0.99216218, passed |
Functional checks cover arithmetic, conversation recall, image questions, two short speech clips, streamed output, stereo resampling, invalid audio rejection, and transitions between text/image/audio. These are smoke tests, not a general quality or ASR benchmark. Encoder cosine results use one retained validation example per modality. All selected encoder outputs are finite; the RK3576 vision accuracy limitation remains unresolved. No generation-speed or RAM benchmark is claimed here.
The source weights were downloaded through ModelScope. Language calibration combines SDK image examples and general text prompts; it does not include the acceptance questions. The shared converter applies FP16-safe normalization and clipping and an RK3576 audio attention-layout correction through its existing single command. Rejected compiler/calibration experiments are excluded from this release.
New deployable-file hashes: SHA256SUMS, with paths relative to v1.3.1/.
Previous text-only file checksums
973008fd7ebed81e2a26dcf7dcc1c28fde631837e3e021c7f3007d6e82647165 v1.3.0/RK3576/gemma-4-E2B-it_RK3576_w4a16_g128.rkllm
b7357c01fcd51c8f896f06ee3a3746bc233031abd04e2fa0cebb5908b1382a01 v1.3.0/RK3576/gemma-4-E2B-it_RK3576_w8a8.rkllm
ae81d1e37eb54b40d2d366ef622a0c05acfb15537272e982efc1dfdd60557979 v1.3.0/RK3588/gemma-4-E2B-it_RK3588_w8a8.rkllm
Credits
Original model: Google Gemma-4-E2B-it. Conversion tools: Rockchip RKLLM, RKNN Toolkit2, and the shared conversion command.