Gemma 4 E2B IT — RKLLM W8A8 for Orange Pi / RK3588
A community conversion of Google Gemma 4 E2B IT, targeting local language-model inference on RK3588-based Orange Pi hardware.
Read the original blog draft: Tuning Gemma on the RK3588's NPU for a local agent for the project history, numerical debugging, and benchmark comparisons.
This release contains a hardware-targeted RKLLM artifact. It requires a compatible Rockchip RKLLM runtime and NPU driver; it cannot be loaded directly as a Transformers checkpoint, GGUF model, or generic RKNN vision model.
September 22 update: the model now contains the benchmarked ctx16384_opt0_rebuild variant, published under the same download filename. Its SHA-256 starts with 9fced5f4; the initial upload started with d91331e5 and contained a different model of the same size. Use the checksum to identify your copy. The previous revision remains in repository history.
Artifact
| Property | Value |
|---|---|
| File | gemma-4-E2B-it_w8a8_rk3588_ctx16384.rkllm |
| Source variant | gemma-4-E2B-it_w8a8_rk3588_ctx16384_opt0_rebuild.rkllm |
| SHA-256 | 9fced5f49cda1742f462ce5ec10461e06c33542e55884d7a656563494dcb71f3 |
| Target reported by runtime | RK3588 |
| Quantization reported by runtime | W8A8 |
| Compiled context limit reported by runtime | 16,384 tokens |
| NPU cores reported by runtime | 3 |
| Exact file size | 7,981,664,404 bytes (about 7.43 GiB) |
| Toolkit version reported in model header | 1.3.0 |
The original conversion log is unavailable, so the exact build invocation, original upstream revision, and optimization level have not been independently recovered. A separate artifact named opt1 has the same full hash as this rebuild; the variant's name alone is not evidence of its optimization level.
Scope and validation
The local project includes a text-only causal-LM extraction script and RKLLM quantization/export tooling. This release distributes the final language-model artifact only; it does not include vision or audio encoders or claim full upstream multimodal support.
The publisher developed this conversion for an Orange Pi. Live checks on September 22, 2026 UTC verified this rebuilt artifact's SHA-256 and ran short text prompts on an Orange Pi 5 Plus with 16 GB RAM, RKLLM 1.3.0, RKNPU driver 0.9.8, and kernel 6.1.99-rockchip-rk3588. The runtime reported W8A8, three NPU cores, and a compiled context limit of 16,384.
With thinking disabled, a 16,384-token configured context, and a 128-token output ceiling, three short prompts measured 8.894–9.006 native decode tokens/s, median 8.904. Native counters reported 127 decode tokens per request; the first output event is accounted for separately. Rates exclude loading and prefill. CPU, NPU, and memory-controller controls were pinned to their supported maxima: 1.8 GHz little CPU, 2.304 GHz big CPU, 1 GHz NPU, and 2.112 GHz DMC.
| Configuration at maximum supported clocks | Median decode rate | Range |
|---|---|---|
| This rebuilt E2B model, RKLLM W8A8, NPU | 8.90 tok/s | 8.89–9.01 |
| E2B, Ollama Q4_K_M, CPU only, four threads | 11.56 tok/s | 11.54–11.59 |
Both used the same three user prompts and a 16K configured context. Quantization, runtime, chat handling, and token accounting differ; this is not a hardware-only or equal-quality comparison. Native prompt processing took about 0.2 seconds for RKLLM and about 0.6 seconds for Ollama on these short inputs.
A separate 40-output-event historical-prompt test measured 9.942 tok/s over 39 native decode tokens. It is not part of the three-prompt table. These results do not establish sustained 16-token/s generation. See benchmark methods, prompts, outputs, and measurements, and runtime identification.
This remains an experimental artifact with observed instruction-following errors. The workshop response used two sentences per suggestion despite a one-sentence instruction, and the separate historical paragraph probe produced an irrelevant sentence-completion response. No formal post-quantization accuracy suite or full 16K-length prompt test has been completed. Selecting a 16K context limit does not establish successful operation with a full 16K input. Board memory requirements at full context and compatibility beyond the tested runtime/driver remain unverified. Upstream model benchmark scores are not measurements of this conversion. CPU headroom for concurrent agent work and energy efficiency have not been established.
Download and integrity
Download the .rkllm file and SHA256SUMS from the Files tab into the same directory, then run:
sha256sum -c SHA256SUMS
Use a compatible Rockchip RKLLM runtime and example application on the RK3588 board. Configure the application for this model, the RK3588 target, and a context length no greater than the compiled 16,384-token setting. Available RAM can limit usable context. Runtime binaries and a ready-to-run server are not included here.
Provenance and modifications
Original model: Google DeepMind's google/gemma-4-E2B-it.
This is a community hardware conversion/quantization, not an official Google or Orange Pi release and not a claim of additional fine-tuning.
The distributed .rkllm binary is a modified derivative of the upstream model, converted to Rockchip's runtime format. See build-metadata.json for the artifact checksum and recorded provenance.
License
Gemma 4 is distributed under Apache License 2.0. A copy is included as LICENSE; the upstream model card is preserved as UPSTREAM_MODEL_CARD.md for attribution. Rockchip tools and runtime components have their own licenses and are not bundled.
- Downloads last month
- 120