DEG-5B-E2B-1018 (v0.1)

deg fills a JSON Schema from a context in one model, using three task-routed expert sets:

• Decide (D): classification heads choose enum, boolean and score fields, and report their probabilities.
• Extract (E): a span pointer copies string fields verbatim from the context.
• Generate (G): new tokens are written only for fields marked x-generate: true.

The output is always valid against the schema. Only the generated fields cost decode tokens: about 15 per record, against 60–80 for a model that writes the whole JSON.

Other models fill all three by writing the whole JSON under a grammar. As far as we know, no other open model does decide, extract and generate as separate task paths in one set of weights, so this release sets the baseline for that kind of model. It is v0.1, and the first deg built on Gemma 4.

Model

Base	google/gemma-4-E2B-it @ 3e22461
Architecture	Gemma-4-E2B trunk with a D/E/G task MoE in every layer's FFN. Attention, embeddings and per-layer embeddings (PLE) are shared.
Routing	By task, not by token. Decide forks → D, context + extract → E, generate → G.
Size	5.39B parameters on GPU, plus the 2.35B frozen PLE table from the base model, which is not stored here. 2.28B active per token.
G experts	Gemma-4-E2B-it's own FFN. In this version, training G caused field-pointer echo loops, so G was reset to the base weights.
Precision	bf16

Results: deg-suite v1

deg-suite v1 is a frozen, held-out suite.

• Headline = mean of the DECIDE, EXTRACT and GENERATE scores. Each of those is the mean of its sources.
• Validity gate: an invalid record scores 0 on every field.
• GENERATE judge: Qwen3.8-Flash-Next (rubric v1).

Competitors were run with grammar-constrained decoding (xgrammar or GBNF).

system	active params	headline	DECIDE	EXTRACT	GENERATE	valid
DEG-5B-E2B-1018	2.3B	0.764	0.774	0.644	0.874	0.999
gemma-4-E4B-it (Q8_0)	~4B	0.752	0.781	0.574	0.902	0.989
Qwen3.5-4B	4.7B	0.749	0.791	0.576	0.881	0.981
Qwen3.5-9B (Q8_0)	9.7B	0.748	0.814	0.553	0.876	0.981
MiniCPM5-2B	2.5B	0.698	0.685	0.565	0.844	0.982
gemma-4-E2B-it (this base, untrained)	2B	0.710	0.726	0.533	0.872	–
Qwen3.8-27B (INT4), teacher-class upper bound	27B	0.817	0.879	0.644	0.929	0.982

Per source:

	datagen-test	JevBench	Banking77	SQuAD v2	mixed	ExtractBench	DialogSum
DECIDE	0.929	0.640	0.714		0.815
EXTRACT	0.966			0.733	0.769	0.110
GENERATE	0.890				0.820		0.912

Latency at batch 1 on one A100-80GB: p50 0.16 s, p95 1.09 s per record. ExtractBench is the slow case, at a p50 of 16.8 s per long document.

Usage

The weights need the deg runtime, which is included under source/. The Python package is imported as deg. The PLE table and tokenizer are read from the base model in your local HF cache.

pip install -e "./source[model]"
hf download google/gemma-4-E2B-it --revision 3e22461f65e89153144f8adb70e3b8c2cc9845a7
export DEG_HF_HUB=~/.cache/huggingface/hub

from deg.model import load_model
from deg.infer import fill

model = load_model("path/to/DEG-5B-E2B-1018", generate="moe")
schema = {
    "type": "object",
    "properties": {
        "department": {"enum": ["billing", "technical", "sales"]},
        "urgent": {"type": "boolean"},
        "order_id": {"type": "string"},
        "reply": {"type": "string", "x-generate": True, "maxLength": 400},
    },
    "required": ["department", "urgent", "order_id", "reply"],
}
print(fill(model, "Order #A-1182 was charged twice, please fix ASAP.", schema))

DEG_PLE_HOST=1 keeps the frozen PLE table in host memory, which saves about 4.4 GiB of GPU memory. The full request and response contract is in source/SPEC.md.

Training

• Run: 1,250 steps on 8×A100-80GB with ZeRO-2.
  • lr 3e-5 for the trunk and experts, 5e-4 for the heads; 200 heads-only steps first.
  • LM loss weight 1, plus KL to the frozen base (weight 2) on the generated tokens.
  • PLE table and token embeddings frozen.
• Data:
  • deg datagen sets for decide, extract and generate, labelled by open teachers (Qwen3.8-27B, MiniCPM-V-4.7) and filtered by voter agreement and a judge.
  • Gold HF sets: MASSIVE, HWU64, Re-DocRED, paraloq.
  • A combined decision pool.
  • No distillation from the evaluation judge.
• Guards: training refuses every suite and JevBench path, and drops any record whose context appears in the suite.

Limitations

• Long-document extraction is weak (ExtractBench 0.110). 28 of E2B's 35 layers use a 512-token sliding window.
• DECIDE trails Qwen3.5-4B/9B on JevBench (0.640 vs 0.668–0.726).
• G is the untrained base FFN. Writing quality is bounded by Gemma-4-E2B-it, and G training is open work for v0.2.
• Text only for now. Image and audio inputs come next: Gemma-4-E2B is natively multimodal, the context path already reads them, and multimodal training sets and an eval are built. They are not trained in v0.1.
• deg.json holds the routing, backbone and calibration the loader reads. Its train_args paths are a record of the training run only.

Acknowledgments

Thanks to Shadeform for the compute: this model was trained and evaluated on 8×A100-80GB instances rented through Shadeform.

License

Apache-2.0, following the base model (see the Gemma 4 license).