--- license: apache-2.0 base_model: google/gemma-4-E2B-it pipeline_tag: image-text-to-text library_name: rkllm tags: - rkllm - rockchip - rk3576 - rk3588 - gemma-4 - image-text-to-text - audio-text-to-text --- # Gemma-4-E2B-it · RKLLM * Max context length: 4096 Text generation, image questions, and short audio transcription on Rockchip RK3576 and RK3588, converted from [google/gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it). **RK3576 preview:** image and audio requests work in the functional tests, but the vision encoder misses the numerical accuracy target. RK3576 W8A8 also fails one exact-recall check. See the measured results below before choosing this variant. RK3588 W8A8 passes the functional and encoder checks performed here. - **Conversion:** RKLLM Toolkit 1.3.1; vision/audio encoders built with RKNN Toolkit2 2.3.2. - **Inputs / output:** text, one image, or short PCM16 WAV audio → text. Audio generation and video input are not implemented in this serving release. - **Context:** exported at 4,096 tokens; board checks used 2,048, including media embeddings and generated text. - **License:** Apache 2.0; see the [original model](https://huggingface.co/google/gemma-4-E2B-it) for terms and intended use. ## Choose your files The new matched multimodal set is under **`v1.3.1/`**. Use the exact SoC for your board. Each variant needs its language file and vision encoder; add the audio encoder to enable speech input. The two RK3576 variants share the same encoders. | Board | Quantization | Language size | Language model | | --- | --- | --- | --- | | RK3576 | `w4a16_g128` | 6.65 GiB | [Language](v1.3.1/RK3576/gemma-4-E2B-it_RK3576_w4a16_g128.rkllm) | | RK3576 | `w8a8` | 7.37 GiB | [Language](v1.3.1/RK3576/gemma-4-E2B-it_RK3576_w8a8.rkllm) | | RK3588 | `w8a8` | 7.36 GiB | [Language](v1.3.1/RK3588/gemma-4-E2B-it_RK3588_w8a8.rkllm) | | Board | Vision encoder (480 × 480, 100 tokens) | Audio encoder | | --- | --- | --- | | RK3576 | [Vision · 1.09 GiB](v1.3.1/RK3576/gemma-4-E2B-it_vision_RK3576.rknn) | [Audio · 0.58 GiB](v1.3.1/RK3576/gemma-4-E2B-it_audio_RK3576.rknn) | | RK3588 | [Vision · 0.46 GiB](v1.3.1/RK3588/gemma-4-E2B-it_vision_RK3588.rknn) | [Audio · 0.57 GiB](v1.3.1/RK3588/gemma-4-E2B-it_audio_RK3588.rknn) | `w4a16_g128` uses 4-bit weights, 16-bit activations and group size 128. `w8a8` uses 8-bit weights and activations; these W8A8 exports use hybrid rate 0.50. Download sizes are not runtime RAM requirements. Use embedding reads from storage (`--embed_flash 1`) on the 8GB RK3576, as in the tested command below. ### Previous text-only release The original RKLLM 1.3.0 artifacts are archived under **`v1.3.0/`**: - [RK3576 text-only models](https://huggingface.co/HanzoHuang/gemma-4-E2B-it-RKLLM/tree/main/v1.3.0/RK3576) - [RK3588 text-only model](https://huggingface.co/HanzoHuang/gemma-4-E2B-it-RKLLM/tree/main/v1.3.0/RK3588) - [Original calibration dataset](v1.3.0/gemma-4-E2B-it_data_quant.json) Their contents and checksums are unchanged; the old unversioned paths have moved. They accept **text only**; adding an encoder does not upgrade those language artifacts. Existing `/llm/gemma-4-e2b-it` Docker images contain the text-only release, and its build definitions now use the versioned paths. Use the complete `v1.3.1/` set for image/audio input. ## Run on your board This RK3576 W4A16 example uses the rolling ARM64 `env-latest` image. Gemma vision/audio requires [PR #13](https://github.com/Hanzo-Huang/rkllm-docker/pull/13) to be merged and the main environment-image build to complete successfully. Download the three files first: ```bash hf download HanzoHuang/gemma-4-E2B-it-RKLLM \ v1.3.1/RK3576/gemma-4-E2B-it_RK3576_w4a16_g128.rkllm \ v1.3.1/RK3576/gemma-4-E2B-it_vision_RK3576.rknn \ v1.3.1/RK3576/gemma-4-E2B-it_audio_RK3576.rknn \ --local-dir ./models sudo docker run --pull=always --rm --name gemma4 --privileged -p 8001:8001 \ -v "$PWD/models/v1.3.1/RK3576:/app/models:ro" \ -e MODEL_KIND=vlm -e TARGET_PLATFORM=rk3576 -e VLM_MODEL_FAMILY=gemma4 \ -e MODEL_FILE=gemma-4-E2B-it_RK3576_w4a16_g128.rkllm \ -e VISION_MODEL_FILE=gemma-4-E2B-it_vision_RK3576.rknn \ -e AUDIO_MODEL_FILE=gemma-4-E2B-it_audio_RK3576.rknn \ ghcr.io/hanzo-huang/rkllm-docker:env-latest \ serve --embed_flash 1 --max_context_len 4096 --timeout 600 ``` For RK3588, select its three files and set `TARGET_PLATFORM=rk3588`. The API is OpenAI compatible at `/v1/chat/completions`; use `image_url` for an image or `input_audio` with base64 `data` and `format: "wav"` for speech. Supported audio is PCM16 WAV, mono/stereo, 8–48 kHz, longer than 10 ms and no more than 7.56 seconds. Split longer recordings before sending them. The audio encoder runs on one NPU core. Runtime integration: [rkllm-docker Gemma support and model definitions](https://github.com/Hanzo-Huang/rkllm-docker/pull/13). ## Release notes Measured on the matching RK3576 and RK3588 devkits with RKLLM runtime 1.3.1 and RKNN runtime 2.3.2: | Variant | Functional checks | Encoder accuracy against original float32 PyTorch | | --- | --- | --- | | RK3576 W4A16_G128 | 14/14 passed | Vision cosine **0.97914620**, below the 0.99 target; audio **0.99222647**, passed | | RK3576 W8A8 | 13/14 passed; exact recall returned `ORCHI D` instead of `ORCHID` | Same vision/audio encoders and limits as above | | RK3588 W8A8 | 14/14 passed | Vision cosine **0.99876202** and audio **0.99216218**, passed | Functional checks cover arithmetic, conversation recall, image questions, two short speech clips, streamed output, stereo resampling, invalid audio rejection, and transitions between text/image/audio. These are smoke tests, not a general quality or ASR benchmark. Encoder cosine results use one retained validation example per modality. All selected encoder outputs are finite; the RK3576 vision accuracy limitation remains unresolved. No generation-speed or RAM benchmark is claimed here. The source weights were downloaded through ModelScope. Language calibration combines SDK image examples and general text prompts; it does not include the acceptance questions. The shared converter applies FP16-safe normalization and clipping and an RK3576 audio attention-layout correction through its existing single command. Rejected compiler/calibration experiments are excluded from this release. New deployable-file hashes: [SHA256SUMS](v1.3.1/SHA256SUMS), with paths relative to `v1.3.1/`.
Previous text-only file checksums ```text 973008fd7ebed81e2a26dcf7dcc1c28fde631837e3e021c7f3007d6e82647165 v1.3.0/RK3576/gemma-4-E2B-it_RK3576_w4a16_g128.rkllm b7357c01fcd51c8f896f06ee3a3746bc233031abd04e2fa0cebb5908b1382a01 v1.3.0/RK3576/gemma-4-E2B-it_RK3576_w8a8.rkllm ae81d1e37eb54b40d2d366ef622a0c05acfb15537272e982efc1dfdd60557979 v1.3.0/RK3588/gemma-4-E2B-it_RK3588_w8a8.rkllm ```
## Credits Original model: [Google Gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it). Conversion tools: [Rockchip RKLLM](https://github.com/airockchip/rknn-llm), [RKNN Toolkit2](https://github.com/airockchip/rknn-toolkit2), and the [shared conversion command](https://github.com/Hanzo-Huang/rknn-llm/pull/4).