| --- |
| library_name: optimum-rbln |
| license: gemma |
| pipeline_tag: image-text-to-text |
| base_model: |
| - google/gemma-4-31B-it |
| - lokeshe09/gemma-4-31B-it-INT8 |
| tags: |
| - gemma4 |
| - multimodal |
| - rbln |
| - ca25 |
| - int8 |
| --- |
| |
| # Gemma4-31B-Int8-RBLN-0.11.2 |
|
|
| `google/gemma-4-31B-it` (multimodal, image-text-to-text), using the INT8 |
| checkpoint from `lokeshe09/gemma-4-31B-it-INT8`, pre-compiled for Rebellions |
| RBLN-CA25 NPUs with `optimum-rbln` and ready to serve with `vllm-rbln`. |
|
|
| This repository contains the compiled artifact only. RBLN NPUs and the RBLN |
| SDK are required to run it. |
|
|
| ## Hardware and software |
|
|
| | Component | Configuration | |
| |---|---| |
| | NPU | 16× RBLN-CA25 (language model: 16 devices; vision tower and multimodal projector: device 0, shared) | |
| | Compiler | `rebel-compiler==0.11.2` | |
| | Runtime | `optimum-rbln==0.11.2`, `vllm-rbln==0.11.2.post1`, `vllm==0.24.0`, `transformers==5.8.1` | |
| | Quantization | INT8 weights / FP16 activations (W8A16, weight-only) | |
|
|
| ## Compile configuration |
|
|
| | Option | Value | |
| |---|---| |
| | `max_seq_len` | 262,144 | |
| | `batch_size` | 8 | |
| | `decoder_batch_sizes` | `[1, 2, 4, 8]` | |
| | `attn_impl` / `kvcache_partition_len` | `flash_attn` / 16,384 | |
| | `cache_impl` / `sliding_window` | `hybrid` / 1,024 | |
| | `prefill_chunk_size` | 128 | |
| | `kvcache_num_blocks` | 36 (589,824 shared KV tokens) | |
| | Vision `max_soft_tokens` / image prefill | 280 / `image_prefill_384` | |
|
|
| The 36 KV blocks were selected by the SDK estimator with `memory_budget=0.9`. |
| The shared KV capacity is approximately 2.25 full 262K contexts; shorter |
| requests can use the compiled dynamic batches up to batch 8. |
|
|
| ## Serving |
|
|
| ```bash |
| export VLLM_PLUGINS=rbln |
| export VLLM_RBLN_USE_VLLM_MODEL=0 |
| export VLLM_RBLN_NUM_DEVICES_PER_LOCAL_RANK=16 |
| export VLLM_RBLN_SUB_BLOCK_CACHE=false |
| |
| vllm serve <this-repo-or-local-path> \ |
| --dtype float32 \ |
| --max-num-seqs 8 \ |
| --max-num-batched-tokens 128 \ |
| --enable-chunked-prefill \ |
| --no-enable-prefix-caching \ |
| --enable-auto-tool-choice \ |
| --tool-call-parser gemma4 \ |
| --reasoning-parser gemma4 |
| ``` |
|
|
| Notes: |
|
|
| - Do not pass `--tensor-parallel-size`; the 16-device split is baked into the |
| compiled language model and `vllm-rbln` runs it with TP=1. |
| - Keep `--max-num-seqs 8` and `--max-num-batched-tokens 128` aligned with the |
| compiled decoder and prefill graphs. |
| - Prefix caching must remain disabled. Gemma4 uses multi-group hybrid KV caches, |
| which sub-block prefix caching does not support in this SDK version. |
| - Thinking is controlled per request with |
| `chat_template_kwargs={"enable_thinking": true}`. Parsed reasoning is returned |
| separately from the final response content. |
| - The artifact accepts one image per prompt with the compiled vision profile. |
|
|
| ## License |
|
|
| Derived from `google/gemma-4-31B-it`. Use is subject to the Gemma Terms of Use. |
|
|
