gemma-4-1.5B-A1.4B-small

This is a small version of google/gemma-4-26B-A4B created for testing and development. It preserves the full architecture characteristics of the base (gemma4 multimodal MoE: 5:1 sliding/full attention hybrid with per-layer head-dim overrides, stacked-expert MoE block added to every layer's dense MLP, siglip-style vision tower with axial RoPE and pooling, tied embeddings, final logit softcapping) with fewer layers and fewer experts.

Model Details

• Base Model: google/gemma-4-26B-A4B
• Architecture: gemma4 (Gemma4ForConditionalGeneration)
• Total Parameters: 1.49B (1,492,537,948)
• Activated Parameters: 1.35B (1,353,071,932; text path per token, excluding vision tower/projector and inactive routed experts)

Configuration Changes

The following parameters were reduced from the original model (these are the ONLY changes — all other hyperparameters are identical to the base):

Parameter	Original	Small
text_config.num_hidden_layers	30	6
text_config.layer_types	5:1 sliding:full × 5 periods	5 sliding + 1 full (exactly one pattern period)
text_config.per_layer_config	full-attn layers 05/11/17/23/29 → head_dim 512, 2 kv heads	{"05": {head_dim 512, num_key_value_heads 2}} (layer 5 is the full-attention layer)
text_config.num_experts	128	10
vision_config.num_hidden_layers	27	2

top_k_experts = 8 is unchanged; 10 = topk + 2 so routing is still exercised. Every text layer keeps its dense MLP + MoE block combination (enable_moe_block=true), the full-attention layer keeps the base's head_dim-512/2-kv-heads override and k_eq_v (no v_proj) behavior, and the last layer is full attention as the architecture requires.

Checkpoint Structure

Single-file model.safetensors (bf16, 164 tensors). Key naming, per-layer key sets (including the missing v_proj on the full-attention layer, matching the base) and 3D stacked expert layout (experts.gate_up_proj [10, 1408, 2816], experts.down_proj [10, 2816, 704]) match the base model.safetensors.index.json exactly modulo layer counts. lm_head is tied to the embedding (tie_word_embeddings=true) and absent from the checkpoint, as in the base.

Usage

from transformers import AutoModelForImageTextToText, AutoProcessor

model = AutoModelForImageTextToText.from_pretrained(
    "inference-optimization/gemma-4-1.5B-A1.4B-small",
    device_map="auto",
    dtype="bfloat16",
)
processor = AutoProcessor.from_pretrained("inference-optimization/gemma-4-1.5B-A1.4B-small")

inputs = processor(text="According to all known laws", return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=20)
print(processor.decode(output[0], skip_special_tokens=True))

Multimodal (image + text) inputs are supported through the processor's gemma image markup (<|image>/<image|>); the image+text path is smoke-tested below.

Creation Process

This model was created using the llm-compressor create-tiny-model skill:

1. Base config inspected; the base config JSON was edited directly (layer counts, per-layer lists, expert count) so that per-layer config resolution stays exact, then reloaded and verified equivalent to the base config modulo the intended changes.
2. Model built with skip_weights_download (no base weights downloaded), re-initialized (init_weights covers the stacked expert tensors, router scales and vision position table), NaN fixup, bf16 cast.
3. Fine-tuned (text-only) on the skill's copypasta toy dataset (bf16, lr 5e-5, batch 2, early-stopped) to perplexity 1.05.
4. Checkpoint structure validated against the base index (see above).
5. Perplexity + generation + multimodal forward validated.

Validation Output

Success: perplexity 1.0481 <= 10.0

Generating sample text:
According to all known laws of aviation, there is no way a bee should be able to fly. Its wings are too small to get its fat little body off the ground. The bee, of

Multimodal forward smoke test:
inputs: {'input_ids': (1, 264), 'attention_mask': (1, 264), 'mm_token_type_ids': (1, 264), 'pixel_values': (1, 2520, 768), 'image_position_ids': (1, 2520, 2)}
logits shape: (1, 264, 262144) finite: True
multimodal forward OK

Notes

• All hyperparameters other than the table above are identical to the base (hidden 2816, head_dim 256 on sliding layers, 16 heads / 8 kv heads, sliding_window 1024, rope parameters per attention type, intermediate_size 2112, moe_intermediate_size 704, top_k_experts 8, vocab 262144, tied embeddings, final_logit_softcapping 30, vision tower dims).
• This is a base (non-instruct) model: the processor ships no chat template (same as the base repo).
• This model was fine-tuned to memorize a toy copypasta dataset; it is not intended to produce meaningful content.