Decision Router · EmbeddingGemma 2

embeddinggemma2-decision-router

A small decision head on top of a frozen google/embeddinggemma-2 text encoder. Given a short text and a list of candidate options, it returns a probability for each option and a probability that none of the options fits (abstention).

Scope (please read). The head and the abstention gate were trained on short, intent-style text (PolyAI/banking77, 62 seen classes, K=8 candidates). Everything below was measured on that kind of input. Other schemas (agent traces, policies, JSON states, long documents) were not validated: on real typed-decisions choice questions, zero-shot, this head scored below the majority-class baseline in our tests.

  • Option-order invariance: the head has no positional encoding on options. Measured flip rate: 0.0% (2,500 shuffles through model.decide(), plus 0.0 in step2_eval_results.json).
  • Abstention gate: null_probability for text choice questions. Works well on BANKING77-style text (see tables) but a threshold of 0.5 also abstains on about 10% of valid queries.
Question type Python type of value Status
choice str Validated on BANKING77-style intents only
score int Not validated; returned from the raw blended logits
noul (yes/no) bool Not validated; returned from the raw blended logits

Model details

Component Parameters Notes
google/embeddinggemma-2 text encoder 271.0M (frozen) Provides 768-d embeddings
Decision head (this repo) 11.6M (11,622,405, 43 tensors) 2-layer Pre-LN set-equivariant Transformer (d_model=768, 8 heads) + 6-feature Null Gate
  • The released model.safetensors is 565 MB (BF16) and contains the encoder and the head.
  • Use torch.bfloat16 or torch.float32 (never float16: it produces NaNs).
  • Image and audio inputs exist in the code path but are experimental: they need the full google/embeddinggemma-2 model downloaded separately, are not covered by the numbers below, and the Null Gate is disabled for them.

How to use

import torch
from transformers import AutoModel

model = AutoModel.from_pretrained(
    "Ajeet-001/embeddinggemma2-decision-router",
    trust_remote_code=True,
    dtype=torch.bfloat16,
).to("cuda")

options = {
    "card arrival": "Where is my card? It has not arrived yet.",
    "lost or stolen card": "I lost my card or it was stolen.",
    "exchange rate": "What exchange rate do you use?",
    "top up failed": "My top-up did not go through.",
}
questions = {"intent": {"type": "choice", "instructions": "Which intent does this message have?", "criteria": options}}

for text in ["My new card still has not arrived after two weeks.", "Write me a short poem about the ocean."]:
    a = model.decide(text, questions)["intent"]
    print(f"{text!r}\n  -> {a.label} (confidence {a.confidence:.0%}), P(none fits)={a.null_probability:.2f}, abstain={a.abstain}")

For a plain-text state and a choice question, decide() embeds State: <text> (the question text and workflow name are not part of the query embedding), because this matched the training format best. Pass include_question=True to embed the question as well; that lowered accuracy in our test (see below).

Abstention. Text choice answers carry null_probability and abstain (True when null_probability >= abstain_threshold, default 0.5). label / value still hold the best-scoring option, so check abstain before acting on it. For image/audio inputs and for score / noul questions, null_probability is None and abstain is False.

TypedAnswer(
    kind="choice",          # "choice" | "score" | "noul"
    label="card arrival",   # best-scoring option key
    value="card arrival",
    confidence=0.97,        # head softmax probability of the winning option (illustrative)
    probabilities={...},
    expected_level=None,    # score questions only
    null_probability=0.00,  # text `choice` only (illustrative)
    abstain=False,
)

PyPI package (gemmarouter==0.1.1)

For direct candidate routing with option-embedding caching (prewarm()), out-of-scope abstention (null_threshold=0.58), and ambiguous twin-collision clarification (margin_threshold=0.40):

pip install -U gemmarouter
import gemmarouter

router = gemmarouter.load()  # downloads Ajeet-001/embeddinggemma2-decision-router

options = {
    "restart_postgres_prod": "Restart the production PostgreSQL database cluster when connection pools hang or transactions lock up.",
    "restart_postgres_staging": "Restart the staging PostgreSQL database cluster when connection pools hang or transactions lock up.",
    "flush_redis_cache": "Evict stale keys from the Redis session cache cluster when product prices fall out of sync.",
}
router.prewarm(options)

# 1. Unambiguous match -> dispatches tool
r1 = router.route("Restart the production PostgreSQL database cluster immediately due to hung connections.", options)
print(r1.decision, r1.confidence, r1.margin)
# -> 'restart_postgres_prod' 0.8758 0.7516

# 2. Ambiguous twin collision (margin < 0.40) -> abstains & asks clarifying question
r2 = router.route("Restart the PostgreSQL database cluster immediately because connection pools are hung.", options)
print(r2.decision, r2.abstain, r2.needs_clarification, r2.clarify_options)
# -> None True True ('restart_postgres_prod', 'restart_postgres_staging')

Source & CLI (gemmarouter route): https://github.com/Ajeetkrup/gemmarouter · PyPI: https://pypi.org/project/gemmarouter/


Measured results

All BANKING77 numbers use the test split (3,080 queries), K=8 candidates, and the locked 62-seen / 15-unseen class split. The head was trained on BANKING77 train (seen classes), so these are in-domain results, not zero-shot.

1. Controlled candidate routing (step2_eval_results.json, query embedded as bare text, the training format)

Metric Value
Accuracy, covered episodes 98.93%
Accuracy, hard-negative covered episodes 94.79% (zero-shot cosine baseline: 93.52%)
Expected calibration error 0.46%
Null AUROC: dropped-gold / unseen class / cross-domain (ag_news) 0.9934 / 0.9899 / 0.9984
Null AUROC, hard dropped-gold 0.9807 (cosine baseline 0.9745)
Option-order flip rate 0.0

Reproduced independently on an A100 (same seeds): accuracy 0.9893, AUROC 0.9934 / 0.9899 / 0.9984.

2. Through model.decide() (decide_verification_results.json, first 1,500 test episodes, bf16, original released head)

Query format Covered acc ECE Null AUROC dropped-gold / unseen Valid queries kept at 0.5 Dropped-gold abstain recall
State: <text> (default for plain text) 97.79% 1.43% 0.9843 / 0.9808 90.1% 99.7%
include_question=True 95.58% 1.34% 0.9518 / 0.9493 72.9% 99.3%

The head's accuracy gain over raw cosine similarity is small on this task (+0.17 pt with bare text): most of the accuracy comes from the frozen encoder. The head's added value is the abstention gate and calibrated probabilities. The 0.5 threshold is not tuned and over-abstains; tune it on your own data.

3. Jev Decision Index (public split)

Suite Score Median latency
Vision Index 0.3 (12,210 rows) 7.80 130.8 ms
Text Index 0.3 (140,178 rows) 6.99 53.6 ms

These scores were produced by decision_index_engine.py, not by model.decide(). The text engine uses a fixed abstention threshold (null_threshold=0.58); the vision run is zero-shot and does not use the head. Full artifacts: Ajeet-001/embeddinggemma2-decision-index-results.

4. Research probe: does unfreezing the encoder help? (option3_probe_results.json, NOT the released weights)

To see why this model scores low on the Decision Index, we trained variants on LocalLLaMA/typed-decisions (train split only: 1,000 states / 5,000 decisions for training, 200 states / 1,000 decisions as dev for checkpoint selection) and evaluated once on the held-out test split (400 states / 2,000 decisions: 600 choice, 600 noul, 800 score). Loss: cross-entropy + teacher KL + Brier (weight 1.0), bf16, A100, seed 42.

Variant Trainable Test acc choice / noul / score ECE Option-order flip rate
Frozen encoder, raw cosine 0 40.95% 29.2% / 51.3% / 42.0% 6.21% 0.0%
Frozen encoder + released BANKING77 head 0 39.20% 30.0% / 46.0% / 41.0% 26.53% 0.0%
Frozen encoder + head retrained on typed-decisions 11.6M 61.50% 59.5% / 73.8% / 53.8% 2.14% 0.0%
Top-8 encoder layers unfrozen + head (bi-encoder) 44.1M + 11.6M 68.20% 63.8% / 78.3% / 63.9% 3.88% 0.0% (0/1600)
Top-12 layers unfrozen, state + question + all options in one pass 65.1M + 11.6M 72.05% 68.8% / 80.7% / 68.0% 4.17% 3.1% (25/800)

What this shows: the released head does not help on typed-decisions (39.2%, close to raw cosine); training on in-domain decisions and unfreezing the encoder raises held-out accuracy substantially, and the BANKING77 K=8 result stayed intact for the bi-encoder variant (98.54% accuracy, dropped-gold Null AUROC 0.9920). The one-pass variant gives up exact order invariance because options sit at different token positions.

What this does not show: these gains do not transfer to unseen workflows. Training on 3 of the 4 typed-decisions workflows and testing on the 4th (agent_trace_observability, n=500) gave 27.0% for the retrained head and 26.4% for the one-pass variant, versus 28.8% for raw cosine. These variants are not released here and we did not run them on the Decision Index. Single seed, no confidence intervals, hyperparameters chosen on dev.


Known limitations

  • Narrow training distribution. Trained on one dataset (BANKING77). Retraining the head on templated JSON/policy states improved accuracy on other templates but weakened abstention on out-of-domain text, and neither head transfers to real typed-decisions schemas (below the majority baseline). Do not use the gate to guard agent actions without validating it on your own data.
  • Query format matters. Wrapping the message in a State: ...\nQuestion: ... prompt drops accuracy from 98.9% to about 95.6% for the released head.
  • Over-abstention at 0.5 (about 10% of valid queries in our test).
  • Reasoning-heavy tasks. Most of the Decision Index (knowledge, language understanding) needs reasoning that a frozen 270M embedding encoder does not provide; our probe (section 4) improved only on seen workflows, not unseen ones.
  • Single run, single seed, no confidence intervals.
  • Multimodal (image/audio) and attribution code exist in the source repository as experimental scripts (step3_multimodal.py, step4_attribution.py) and are not part of the validated decide() behaviour of this model.

Training

  • Data: 15,884 K=8 hard-negative routing episodes (Ajeet-001/banking77-hard-negative-routing-episodes): two episodes per training query (7,942 queries from the 62 seen classes of PolyAI/banking77 train; 15 classes held out), about 30% with the correct option removed (abstention target).
  • Objective: cross-entropy over candidates on covered episodes + binary cross-entropy on the Null Gate for all episodes.
  • Recipe (train_step2.py): 8 epochs, batch size 128, AdamW (lr 1e-3, weight decay 1e-2), no learning-rate scheduler, encoder frozen, embeddings precomputed.
  • Hardware: 1 × NVIDIA A100-SXM4-40GB.
Downloads last month
347
Safetensors
Model size
0.3B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for Ajeet-001/embeddinggemma2-decision-router

Quantized
(146)
this model

Datasets used to train Ajeet-001/embeddinggemma2-decision-router