Instructions to use Ajeet-001/embeddinggemma2-decision-router with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Ajeet-001/embeddinggemma2-decision-router with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="Ajeet-001/embeddinggemma2-decision-router", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("Ajeet-001/embeddinggemma2-decision-router", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
embeddinggemma2-decision-router
A small decision head on top of a frozen google/embeddinggemma-2 text encoder. Given a short text and a list of candidate options, it returns a probability for each option and a probability that none of the options fits (abstention).
Scope (please read). The head and the abstention gate were trained on short, intent-style text (PolyAI/banking77, 62 seen classes, K=8 candidates). Everything below was measured on that kind of input. Other schemas (agent traces, policies, JSON states, long documents) were not validated: on real typed-decisions choice questions, zero-shot, this head scored below the majority-class baseline in our tests.
- Option-order invariance: the head has no positional encoding on options. Measured flip rate:
0.0%(2,500 shuffles throughmodel.decide(), plus 0.0 instep2_eval_results.json). - Abstention gate:
null_probabilityfor textchoicequestions. Works well on BANKING77-style text (see tables) but a threshold of0.5also abstains on about 10% of valid queries.
| Question type | Python type of value |
Status |
|---|---|---|
choice |
str |
Validated on BANKING77-style intents only |
score |
int |
Not validated; returned from the raw blended logits |
noul (yes/no) |
bool |
Not validated; returned from the raw blended logits |
Model details
| Component | Parameters | Notes |
|---|---|---|
google/embeddinggemma-2 text encoder |
271.0M (frozen) |
Provides 768-d embeddings |
| Decision head (this repo) | 11.6M (11,622,405, 43 tensors) |
2-layer Pre-LN set-equivariant Transformer (d_model=768, 8 heads) + 6-feature Null Gate |
- The released
model.safetensorsis565 MB(BF16) and contains the encoder and the head. - Use
torch.bfloat16ortorch.float32(neverfloat16: it produces NaNs). - Image and audio inputs exist in the code path but are experimental: they need the full
google/embeddinggemma-2model downloaded separately, are not covered by the numbers below, and the Null Gate is disabled for them.
How to use
import torch
from transformers import AutoModel
model = AutoModel.from_pretrained(
"Ajeet-001/embeddinggemma2-decision-router",
trust_remote_code=True,
dtype=torch.bfloat16,
).to("cuda")
options = {
"card arrival": "Where is my card? It has not arrived yet.",
"lost or stolen card": "I lost my card or it was stolen.",
"exchange rate": "What exchange rate do you use?",
"top up failed": "My top-up did not go through.",
}
questions = {"intent": {"type": "choice", "instructions": "Which intent does this message have?", "criteria": options}}
for text in ["My new card still has not arrived after two weeks.", "Write me a short poem about the ocean."]:
a = model.decide(text, questions)["intent"]
print(f"{text!r}\n -> {a.label} (confidence {a.confidence:.0%}), P(none fits)={a.null_probability:.2f}, abstain={a.abstain}")
For a plain-text state and a choice question, decide() embeds State: <text> (the question text and workflow name are not part of the query embedding), because this matched the training format best. Pass include_question=True to embed the question as well; that lowered accuracy in our test (see below).
Abstention. Text choice answers carry null_probability and abstain (True when null_probability >= abstain_threshold, default 0.5). label / value still hold the best-scoring option, so check abstain before acting on it. For image/audio inputs and for score / noul questions, null_probability is None and abstain is False.
TypedAnswer(
kind="choice", # "choice" | "score" | "noul"
label="card arrival", # best-scoring option key
value="card arrival",
confidence=0.97, # head softmax probability of the winning option (illustrative)
probabilities={...},
expected_level=None, # score questions only
null_probability=0.00, # text `choice` only (illustrative)
abstain=False,
)
PyPI package (gemmarouter==0.1.1)
For direct candidate routing with option-embedding caching (prewarm()), out-of-scope abstention (null_threshold=0.58), and ambiguous twin-collision clarification (margin_threshold=0.40):
pip install -U gemmarouter
import gemmarouter
router = gemmarouter.load() # downloads Ajeet-001/embeddinggemma2-decision-router
options = {
"restart_postgres_prod": "Restart the production PostgreSQL database cluster when connection pools hang or transactions lock up.",
"restart_postgres_staging": "Restart the staging PostgreSQL database cluster when connection pools hang or transactions lock up.",
"flush_redis_cache": "Evict stale keys from the Redis session cache cluster when product prices fall out of sync.",
}
router.prewarm(options)
# 1. Unambiguous match -> dispatches tool
r1 = router.route("Restart the production PostgreSQL database cluster immediately due to hung connections.", options)
print(r1.decision, r1.confidence, r1.margin)
# -> 'restart_postgres_prod' 0.8758 0.7516
# 2. Ambiguous twin collision (margin < 0.40) -> abstains & asks clarifying question
r2 = router.route("Restart the PostgreSQL database cluster immediately because connection pools are hung.", options)
print(r2.decision, r2.abstain, r2.needs_clarification, r2.clarify_options)
# -> None True True ('restart_postgres_prod', 'restart_postgres_staging')
Source & CLI (gemmarouter route): https://github.com/Ajeetkrup/gemmarouter · PyPI: https://pypi.org/project/gemmarouter/
Measured results
All BANKING77 numbers use the test split (3,080 queries), K=8 candidates, and the locked 62-seen / 15-unseen class split. The head was trained on BANKING77 train (seen classes), so these are in-domain results, not zero-shot.
1. Controlled candidate routing (step2_eval_results.json, query embedded as bare text, the training format)
| Metric | Value |
|---|---|
| Accuracy, covered episodes | 98.93% |
| Accuracy, hard-negative covered episodes | 94.79% (zero-shot cosine baseline: 93.52%) |
| Expected calibration error | 0.46% |
Null AUROC: dropped-gold / unseen class / cross-domain (ag_news) |
0.9934 / 0.9899 / 0.9984 |
| Null AUROC, hard dropped-gold | 0.9807 (cosine baseline 0.9745) |
| Option-order flip rate | 0.0 |
Reproduced independently on an A100 (same seeds): accuracy 0.9893, AUROC 0.9934 / 0.9899 / 0.9984.
2. Through model.decide() (decide_verification_results.json, first 1,500 test episodes, bf16, original released head)
| Query format | Covered acc | ECE | Null AUROC dropped-gold / unseen | Valid queries kept at 0.5 |
Dropped-gold abstain recall |
|---|---|---|---|---|---|
State: <text> (default for plain text) |
97.79% |
1.43% |
0.9843 / 0.9808 |
90.1% |
99.7% |
include_question=True |
95.58% |
1.34% |
0.9518 / 0.9493 |
72.9% |
99.3% |
The head's accuracy gain over raw cosine similarity is small on this task (+0.17 pt with bare text): most of the accuracy comes from the frozen encoder. The head's added value is the abstention gate and calibrated probabilities. The 0.5 threshold is not tuned and over-abstains; tune it on your own data.
3. Jev Decision Index (public split)
| Suite | Score | Median latency |
|---|---|---|
Vision Index 0.3 (12,210 rows) |
7.80 |
130.8 ms |
Text Index 0.3 (140,178 rows) |
6.99 |
53.6 ms |
These scores were produced by decision_index_engine.py, not by model.decide(). The text engine uses a fixed abstention threshold (null_threshold=0.58); the vision run is zero-shot and does not use the head. Full artifacts: Ajeet-001/embeddinggemma2-decision-index-results.
4. Research probe: does unfreezing the encoder help? (option3_probe_results.json, NOT the released weights)
To see why this model scores low on the Decision Index, we trained variants on LocalLLaMA/typed-decisions (train split only: 1,000 states / 5,000 decisions for training, 200 states / 1,000 decisions as dev for checkpoint selection) and evaluated once on the held-out test split (400 states / 2,000 decisions: 600 choice, 600 noul, 800 score). Loss: cross-entropy + teacher KL + Brier (weight 1.0), bf16, A100, seed 42.
| Variant | Trainable | Test acc | choice / noul / score |
ECE | Option-order flip rate |
|---|---|---|---|---|---|
| Frozen encoder, raw cosine | 0 | 40.95% |
29.2% / 51.3% / 42.0% |
6.21% |
0.0% |
| Frozen encoder + released BANKING77 head | 0 | 39.20% |
30.0% / 46.0% / 41.0% |
26.53% |
0.0% |
| Frozen encoder + head retrained on typed-decisions | 11.6M | 61.50% |
59.5% / 73.8% / 53.8% |
2.14% |
0.0% |
| Top-8 encoder layers unfrozen + head (bi-encoder) | 44.1M + 11.6M | 68.20% |
63.8% / 78.3% / 63.9% |
3.88% |
0.0% (0/1600) |
| Top-12 layers unfrozen, state + question + all options in one pass | 65.1M + 11.6M | 72.05% |
68.8% / 80.7% / 68.0% |
4.17% |
3.1% (25/800) |
What this shows: the released head does not help on typed-decisions (39.2%, close to raw cosine); training on in-domain decisions and unfreezing the encoder raises held-out accuracy substantially, and the BANKING77 K=8 result stayed intact for the bi-encoder variant (98.54% accuracy, dropped-gold Null AUROC 0.9920). The one-pass variant gives up exact order invariance because options sit at different token positions.
What this does not show: these gains do not transfer to unseen workflows. Training on 3 of the 4 typed-decisions workflows and testing on the 4th (agent_trace_observability, n=500) gave 27.0% for the retrained head and 26.4% for the one-pass variant, versus 28.8% for raw cosine. These variants are not released here and we did not run them on the Decision Index. Single seed, no confidence intervals, hyperparameters chosen on dev.
Known limitations
- Narrow training distribution. Trained on one dataset (BANKING77). Retraining the head on templated JSON/policy states improved accuracy on other templates but weakened abstention on out-of-domain text, and neither head transfers to real
typed-decisionsschemas (below the majority baseline). Do not use the gate to guard agent actions without validating it on your own data. - Query format matters. Wrapping the message in a
State: ...\nQuestion: ...prompt drops accuracy from98.9%to about95.6%for the released head. - Over-abstention at
0.5(about 10% of valid queries in our test). - Reasoning-heavy tasks. Most of the Decision Index (knowledge, language understanding) needs reasoning that a frozen 270M embedding encoder does not provide; our probe (section 4) improved only on seen workflows, not unseen ones.
- Single run, single seed, no confidence intervals.
- Multimodal (image/audio) and attribution code exist in the source repository as experimental scripts (
step3_multimodal.py,step4_attribution.py) and are not part of the validateddecide()behaviour of this model.
Training
- Data:
15,884K=8 hard-negative routing episodes (Ajeet-001/banking77-hard-negative-routing-episodes): two episodes per training query (7,942queries from the 62 seen classes ofPolyAI/banking77train; 15 classes held out), about 30% with the correct option removed (abstention target). - Objective: cross-entropy over candidates on covered episodes + binary cross-entropy on the Null Gate for all episodes.
- Recipe (
train_step2.py): 8 epochs, batch size 128, AdamW (lr1e-3, weight decay1e-2), no learning-rate scheduler, encoder frozen, embeddings precomputed. - Hardware: 1 × NVIDIA A100-SXM4-40GB.
- Downloads last month
- 347
Model tree for Ajeet-001/embeddinggemma2-decision-router
Base model
google/embeddinggemma-2