gemma4-e2b-decision (research preview, v3 = run6)

A small decision model: give it a text (the state) and one or more typed questions with fixed options, and it returns a calibrated probability for every option of every question — in one forward pass, without generating text. Same idea as TypeSafe Jev and Cloudflare Clef, built on Gemma 4 E2B, Russian-first with English.

License: CC BY-NC-SA 4.0, research and non-commercial use only. Some training datasets are research-only or non-commercial (see Training data and licenses). Do not use this model commercially.

What it does

{"lang": "ru",
 "state": "Клиент: с утра не проходит оплата картой, пишет «ошибка 502», заказ нужен сегодня.",
 "questions": {
   "department": {"type": "choice", "instructions": "Какой команде передать обращение?",
                  "criteria": {"billing": "Платежи и возвраты", "technical": "Ошибки сайта и сбои",
                               "delivery": "Доставка", "sales": "Вопросы о товарах"}},
   "urgency":    {"type": "score", "instructions": "Насколько срочно ответить?",
                  "criteria": ["Может подождать", "В течение недели", "Сегодня", "Немедленно"]},
   "outage":     {"type": "noul", "instructions": "Похоже на сбой сервиса?"}}}

→ for each question: the probabilities of all options (choice), a probability distribution plus expected level (score, an ordered scale), or the probability of "yes" (noul). Probabilities are calibrated (temperature per question type), so thresholds like "act automatically above 0.9, ask a human below" behave as expected.

Question types: choice (pick one of N, N = 2…200 tested), score (ordered scale), noul (yes/no).
All questions of a request share one forward pass: 1 question ≈ 107 ms, 4 questions ≈ 113 ms (RTX 5060 Ti, bf16).
Options are scored by their text, not by letters: no 26-option limit, no "A" bias; option order is randomized in training.

How it works

Gemma 4 E2B (frozen) + LoRA r=64 on attention/MLP (101 M params) + a small decision head (5 M params):

The request is rendered as one prompt in a "sandwich" layout: a short preview of the questions, then the state, then every question with its options.
The backbone reads it once. For every option the head averages the hidden states of the option's tokens, adds the question vector, a global vector (last token) and a question-type embedding, lets the options of one question attend to each other (one self-attention layer), and scores each option with a small MLP.
Softmax over a question's options → probabilities; a per-type temperature (fitted on validation) calibrates them.

Training (v3): 1 epoch, 251 K questions in 212 K requests, 63 % Russian / 37 % English: the v2 mix (~25 human-labelled public datasets, 48 K rows from 743 tasks of tasksource-jev-typed-decisions, 21 K multi-question synthetic requests, 8.5 K own synthetic rows) plus, new in v3, Russian passage relevance (Mr.TyDi), sentiment towards a named entity (RuSentNE-2023 train), zero-shot label sets (GeRaCl, Horizon), RuNLU intents, ru-WANLI, RuSTS (scale), HelpSteer3-ru, xLAM irrelevance and our own "none of these" augmentation; ~16 % ordered scales with worded levels. Soft cross-entropy + Brier loss, label smoothing 0.05. Training texts that overlap any test item (razvilka: any shared 8-word span; our test sets: ≥ 50 % of the text) were removed.

Quick start

pip install torch transformers peft safetensors huggingface_hub

from decision_model import DecisionModel        # file from this repo
dm = DecisionModel("lucky-striker/gemma4-e2b-decision-ru")             # downloads google/gemma-4-E2B-it on first use (~10 GB)
print(dm.decide(request))                        # request as in the example above

Or from the command line: python decision_model.py --repo . --request examples/request.json. Needs a CUDA GPU with ~11 GB free memory (bf16). request["tta"] = 4 averages over 4 option orders (slower, slightly better calibrated).

Results

Held-out test splits; the model never saw these examples, but for most of them it saw the train split of the same dataset. The "unseen task" column is the honest out-of-task number (same recipe, v1 data, trained without BANKING77 / CLINC).

Benchmark	v3 (this)	v2	v1, task never seen	FRIDA-Decisions ³	Jev ¹	Clef-flash 9B ¹	Clef 27B ¹
razvilka (735 ru items) ⁴	0.893	0.880	—	0.891	0.897	—	—
— choice / noul / score / ranking	0.922 / 0.871 / 0.717 / 0.820	0.906 / 0.906 / 0.761 / 0.660	—	0.912 / 0.878 / 0.696 / 0.900	—	—	—
— entity sentiment	0.760	0.540	—	—	—	—	—
BANKING77, all 77 classes, macro-F1	85.1	84.3	59.9	—	79.7	90.9	94.2
CLINC150 + out-of-scope, macro-F1	87.8	88.6	75.3	—	89.3	66.8	97.4
BFCL multiple / live multiple (function choice) ²	99.0 / 93.2	99.0 / 93.5	—	—	—	—	—
BFCL irrelevance / live irrelevance ("call nothing") ²	82.1 / 80.0	81.2 / 78.3	—	—	—	—	—
Long context, 8K-token state	85.5	87.0	—	—	—	—	—
Latency, p50, 1 / 4 questions	same architecture as v2	109 / 111 ms	—	28 / 34 ms ³	330–524 ms	39 ms	209 ms

¹ Vendor-reported, not reproduced by us; whether those models saw these datasets' train splits is unknown. ² BFCL was never in training. Our BFCL metric is option accuracy, not BFCL's own "exact" score. ³ Measured by us on the same PC (RTX 5060 Ti) with the same requests and the official razvilka scorer; FRIDA latency is from its model card on the same GPU class. FRIDA reads ≤ 512 state tokens and ≤ 16 options. ⁴ v2/v3 were trained on the TRAIN splits of several datasets razvilka draws its TEST items from (headline, ru-scibench, ru-reviews, kinopoisk; v3 also RuSentNE-2023 train and Mr.TyDi ru), so razvilka is a same-task test, not an unseen-task one. Training rows sharing any 8-word span with razvilka and Mr.TyDi queries matching MIRACL dev/test or razvilka were removed. Our model has no native ranking type; ranking items are asked as choice.

Russian sets (accuracy, v3): TERRa 88.7, DaNetQA 85.7, PARus 88.0, MuSeRC 91.5, CEDR 83.9, ru_goemotions 65.7, RuSentiTweet 76.3, Lenta 91.7, RCB 56.8. v3 vs v2 on the same 11 353 test questions: −0.1 pt accuracy (not significant), log-loss significantly better (−0.011). Calibration error (ECE) is 0.02–0.05 on most sets.

Gemini Decision 1 benchmark table, text rows (our reconstruction)

Google published only the table, not the splits or prompts, so these are our own packs (JevBench public repo 231; ESCI 2 000 random US Task-2 test rows; New Yorker matching test 528, text-only; PubMedQA pqa_labeled 1 000; Global-MMLU-Lite 22 non-English languages × 400; When2Call test_mcq 3 652). Differences under ~3 points are not interpretable. Top-1 accuracy, failures count as wrong (there were none).

Benchmark	v3 (2B, this)	v2	Gemini Decision 1 ¹	Jev 1.14 ¹	GPT-6 Luna ¹	Gemini 3.5 Flash-Lite ¹
JevBench (public)	78.8	79.7	92.2	90.0	90.9	86.6
Amazon ESCI, macro-F1	48.0	42.0	58.7	56.7	54.7	47.2
New Yorker Caption (text-only)	40.5	45.8	79.7	72.0	68.4	73.7
PubMedQA	71.3	69.2	79.8	78.2	74.4	77.6
Global-MMLU-Lite (non-English)	47.1	47.6	86.0	85.0	79.6	78.8
When2Call	65.7	—	79.9	82.0	70.4	60.9
Top-1 ECE, pooled (ours: 6 sets)	0.058	—	0.025	0.049	0.072	0.210

Knowledge- and humour-heavy rows (MMLU, New Yorker, JevBench hard items: 56.8 %) are limited by the 2B backbone; rows where the evidence is in the text (PubMedQA, ESCI, When2Call) are within 3–11 points of the large API models. Mind2Web Multimodal, MMAU and Video-MME need image/audio/video input and were not run.

Limitations

Unseen fine-grained label sets are much harder. Trained without BANKING77/CLINC, the same recipe drops from 85→60 (BANKING77-77) and 87→75 (CLINC-150) macro-F1: dataset-specific boundaries between near-synonym classes cannot be guessed. Short option descriptions (criteria: {id: description}) help; provide them.
"None of these" is overconfident: on BFCL irrelevance / When2Call the model is right ~70–80 % but its ECE is ~0.13 (vs 0.02–0.05 elsewhere). Do not rely on its abstain probability without your own threshold check.
Ranking passages by relevance improved in v3 (razvilka 0.82) but is still below FRIDA (0.90); sentiment towards a named entity is 0.76. Fine logical inference (RCB, 57 %) and HelpSteer2-style quality ratings (56 % exact level) remain weak. v3 is slightly worse than v2 on razvilka scales (0.72 vs 0.76) and yes/no (0.87 vs 0.91), and on two English sets (New Yorker −5, When2Call response choice −3 points): the v3 mix is more Russian.
Trained on states up to ~1.2 K tokens; works up to 8 K with a small drop (87 → 85.5 %), longer is untested.
Text only (the multimodal parts of Gemma 4 are unused). Russian and English only.
Synthetic training data was generated with LLMs and may carry their biases.

Training data and licenses

Dataset	Use	License
PolyAI/banking77	intents	CC BY 4.0
clinc/clinc_oos	intents + out-of-scope	CC BY 3.0
AmazonScience/massive (ru-RU, en-US)	intents (modified)	CC BY 4.0, © Amazon.com Inc. or its affiliates
nvidia/When2Call	call / ask / refuse	CC BY 4.0
nvidia/HelpSteer2	quality scales	CC BY 4.0
RussianNLP/russian_super_glue (TERRa, RCB, DaNetQA, PARus, MuSeRC)	inference, QA	MIT (repository)
sagteam/cedr_v1	emotions	Apache 2.0
Djacon/ru_goemotions	emotions	MIT (metadata)
belyakoff/xlam-ru-tool-calling	tool choice	derived from Salesforce xLAM: research purposes only
NiGuLa/Russian_Inappropriate_Messages	inappropriate messages	original (Skoltech): CC BY-NC-SA 4.0
psytechlab/RuSentiTweet	sentiment	no license stated
zloelias/lenta-ru	news topics	no license stated
synthetic (ours)	9 task families, ru/en	generated with LLMs
tasksource/tasksource-jev-typed-decisions (48 K rows, 743 tasks; BANKING77/CLINC/MASSIVE/HelpSteer2 sources excluded)	many task types	mixed per-row terms (`license_use`: commercial / non-commercial / unspecified)
tasksource/synthetic-typed-decisions	multi-question requests, soft labels	Apache 2.0 (labels from the Jev API)
ai-forever/headline-classification, ru-scibench-grnti/oecd-classification, kinopoisk-sentiment-classification	topics, scientific fields, sentiment	MIT
ai-forever/ru-reviews-classification	review sentiment (choice + scale)	Apache 2.0
mteb/GeoreviewClassification	review rating 1–5 (scale)	MIT
brighter-dataset/BRIGHTER-emotion-intensities (ru)	emotion intensity 0–3 (scale)	CC BY 4.0
RussianNLP/rucola	sentence acceptability	Apache 2.0
castorini/mr-tydi (ru, train)	passage relevance (choice + yes/no)	Apache 2.0 (Wikipedia text CC BY-SA)
dialogue-evaluation/RuSentNE-evaluation (`train_data.csv` only)	sentiment towards a named entity	MIT
deepvk/GeRaCl_synthethic_dataset	zero-shot label sets	Apache 2.0 (labels by Llama-3.3-70B)
Horizon-Labs/multilingual-zeroshot-synthetic (ru, en)	zero-shot label sets	ODC-BY
mteb/RuNLUIntentClassification	intents	CC BY 4.0
deepvk/ru-WANLI	NLI	Apache 2.0 (GPT-generated)
mteb/RuSTSBenchmarkSTS	similarity scale 0–5	CC BY-SA 4.0
nvidia/HelpSteer3 (ru preference rows)	which answer is better	CC BY 4.0
MadeAgents/xlam-irrelevance-7.5k	"no tool fits"	CC BY 4.0
own "none of these" augmentation	abstain option	—

Only TRAIN splits were used. Because of the research-only, NC-SA and non-commercial sources, the model is released under CC BY-NC-SA 4.0 for research and non-commercial use. A version trained only on permissively licensed data may follow. The base model Gemma 4 E2B is © Google, Apache 2.0 (https://ai.google.dev/gemma/docs/gemma_4_license).

Citations

BANKING77 — Casanueva et al. 2020, Efficient Intent Detection with Dual Sentence Encoders. CLINC150 — Larson et al. 2019, An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction. MASSIVE — FitzGerald et al. 2022, MASSIVE: A 1M-Example Multilingual NLU Dataset. When2Call — Ross et al., NAACL 2025, When2Call: When (not) to Call Tools. HelpSteer2 — Wang et al. 2024, arXiv:2406.08673. Russian SuperGLUE — Shavrina et al. 2020, arXiv:2010.15925. CEDR — Sboev et al. 2021, Procedia Computer Science 190, 637–642. xLAM / APIGen — Liu et al. 2024, arXiv:2406.18518. tasksource — Sileo, LREC-COLING 2024, tasksource: A Large Collection of NLP tasks with a Structured Dataset Preprocessing Framework. BRIGHTER — Muhammad et al. 2025, BRIGHTER: BRIdging the Gap in Human-Annotated Textual Emotion Recognition Datasets. RuCoLA — Mikhailov et al. 2022, RuCoLA: Russian Corpus of Linguistic Acceptability.

Earlier versions (v1 = run4, v2 = run5) are available in this repository's commit history.