gemma-4-E2B-FlashNorm

The FlashNorm variant of google/gemma-4-E2B: the folded norm weight tensors are removed from the state dict entirely, and the model runs WITHOUT the pre-attention RMSNorm in every decoder layer (Proposition 3 of the FlashNorm paper, applicable because Gemma 4 re-normalizes queries, keys, and values per head). The pre-feedforward norm keeps its division but is weightless. Requires trust_remote_code:

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained('open-machine/gemma-4-E2B-FlashNorm', trust_remote_code=True)

For stock-Transformers loading without remote code, use the compatibility companion gemma-4-E2B-FlashNorm-test (same fold, norm tensors kept as all-ones), whose card also lists the full norm-tensor inventory.

Validation: outputs are bitwise identical (bf16) to the compatibility checkpoint with runtime cancellation applied, verified on the uploaded copy over a 1024-token context and under cached greedy decoding; deviations from the base model are at the bf16 rounding level. In fp32 on this architecture the cancellation measures max logit deviation 6.8e-4 (vs 5.6e-4 for the mathematically exact fold-only control), WikiText-2 perplexity change 0.0006%, greedy generations identical, HellaSwag unchanged. Post-attention, post-feedforward, per-layer-input, q/k, and final norms are unchanged (not foldable, or not yet folded).

Gemma is a trademark of Google; distributed under the same Apache 2.0 license as the base model.

Downloads last month
14
Safetensors
Model size
5B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for open-machine/gemma-4-E2B-FlashNorm

Finetuned
(162)
this model

Paper for open-machine/gemma-4-E2B-FlashNorm