Gemma 4 E2B β†’ Gemma 4 31B bridge

A fitted KV bridge: Gemma 4 E2B reads the prompt, and this bridge translates its internal state into the key/value memory Gemma 4 31B would have built. The 31B then answers from that memory, reading one token of the prompt itself.

It applies LatentPort, Simon Villani's method (arXiv:2609.25053), to Gemma 4. The reproduction it belongs to is prattle_not.

It is for the 31B quantized to Q4K. The bridge was fitted against Gemma 4 31B with its linear layers quantized to Q4K when it loads (prattle_not's PRATTLE_TARGET_DTYPE=q4k), with E2B in bf16, and must be used with the models loaded the same way.

⚠️ Security note. When bridged, the answering model does not read the prompt itself, the system prompt included, and the bridge does not reliably carry its instructions. Measured on six of our pairs, a rule set in a system prompt is broken more often bridged than when the large model reads the prompt. Do not rely on system-prompt instructions as guardrails for a bridged model: have the large model read the system prompt itself, or enforce policy outside the model.

Two layer shapes, each fitted in its own

Gemma 4 31B has 60 layers of two kinds: 50 sliding-window layers with 16 KV heads of width 256, and 10 global layers with 4 KV heads of width 512, whose keys and values are the same tensor. The bridge fits every layer in its own shape, so a global layer's map produces its 4 Γ— 512 rows directly. E2B, the reader, is the "effective 2B" Gemma 4: about 2B parameters of compute, plus per-layer embeddings.

What it measures

Held-out questions, never fitted on:

share of the read-nothing β†’ read-everything gap the bridge closes (NCR) the same, from a wrong donor correct of 7
the maps alone β€” this file 0.809 0.448 7; the 31B alone: 7, E2B alone: 7

NCR is the paper's native-closure ratio (arXiv:2609.25053, Eq. 8): (β„“_empty βˆ’ β„“_bridged) / (β„“_empty βˆ’ β„“_native), where each β„“ is the 31B's loss on its own native reply, scored teacher-forced. It is a loss ratio, not a rate of matching tokens. Per token, the 31B's loss on its own reply is 0.100 reading everything, 1.104 bridged, 5.355 reading nothing, and 3.001 from a wrong question's memory.

The bridge already answers every question the 31B answers alone, so no correction pass was trained.

Prompts use Gemma 4's plain turn with thinking off; the 31B writes its own empty thought channel and then answers.

How much sooner

Two runs of the same command, ~256-token prompts:

run time to first token, bridged 31B vs the 31B reading the prompt the ceiling in the same run (31B alone Γ· E2B alone)
1 1.97Γ— (1.94–2.00) sooner (307 ms β†’ 154 ms) 5.10Γ—
2 1.95Γ— (1.93–1.98) sooner 5.06Γ—

Measured on two RTX 3090s, E2B in bf16 with the bridge on one and the 31B quantized to Q4K on the other: synthetic prompts, 8 prompts Γ— 3 repetitions in rotated order, one bridge token, and every run's counters checked, so the bridged 31B read no prompt history itself. The figure is the geometric mean of native Γ· bridged with its 95% interval, LatentPort's own statistic.

~256-token prompts measured; longer prompts await an attention path that fits. At ~1024 tokens the 31B reading the prompt itself runs its 24 GB card out of memory in this runtime (its attention scores are held in full, in f32), so there is no longer-prompt figure yet.

To repeat it from a prattle_not checkout:

PRATTLE_SOURCE_DTYPE=bf16 PRATTLE_TARGET_DTYPE=q4k PRATTLE_DEVICES=cuda:0,cuda:1 just speedup models/gemma-4-E2B-it models/gemma-4-31B-it models/bridges/gemma-4-e2b-to-31b.safetensors 256

How it was made

  • The base models: google/gemma-4-E2B-it at 4adf15748de9e1a0097b6274cee9c15836c41c37 and google/gemma-4-31B-it at 842da3794eaa0b77d5f08bae87a17459d91ff475, both Apache-2.0. Hidden width 1536 and 5376; 35 and 60 layers.
  • Fitted with LatentPort's bridge method (Simon Villani) and agentic-underground's own tooling, which is not published.

The file

gemma-4-e2b-to-31b.safetensors, 2.77 GB, with no __metadata__:

tensors what they hold
k.<l>, v.<l> for a sliding layer (f16, 3073 Γ— 4096) the map from E2B's residual stream to the 31B's keys / values, 16 heads Γ— 256
k.<l>, v.<l> for a global layer (f16, 3073 Γ— 2048) the same, 4 heads Γ— 512
shapes each target layer's KV heads and head width
layer_of which E2B layer feeds each 31B layer
meta the shape constants the reader needs

It is read by prattle_core::bridge in prattle_not, in Rust. It is not a transformers model, and it contains no weights of either Gemma model. Fetch those from Google.

Attribution

prattle_not is a derivative work of LatentPort by Simon Villani: https://latentport.com/, https://github.com/LatentPort/latentport, licensed under the Apache License, Version 2.0. The bridge design and the one-token handoff are the paper's (arXiv:2609.25053v1). Gemma 4 is by Google DeepMind, under Apache-2.0. This bridge is released under Apache-2.0.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for agentic-underground/gemma-4-e2b-to-31b-bridge

Finetuned
(393)
this model

Collection including agentic-underground/gemma-4-e2b-to-31b-bridge

Paper for agentic-underground/gemma-4-e2b-to-31b-bridge