Jalebi-E
A 66.6M-parameter decision model, trained from scratch, with a 2,048-token window. Give Jalebi-E a situation and a list of options and it scores every option in one forward pass. It never generates text, so there is nothing to hallucinate and nothing to prompt-inject into a free-form answer. It answers three kinds of decision: pick an option (choice), give a yes/no probability (noul), or score against criteria (score).
It is built to sit in front of a large language model, a retrieval system or a tool-calling agent and answer the question that comes first: where should this request go?
Apache-2.0 路 66.6M parameters 路 no pretrained weights anywhere in the stack 路 2,048-token window 路 English 路 by Airavat AI
Jalebi-E is one of the smallest models we have built, and one of the most powerful. At 66.6M parameters it outscores our 0.44B Jalebi-L on the S1MB task average (22.94 against 19.2), with a context window four times longer. More Jalebi models are coming soon.
Results
On the S1MB English v1 benchmark (137 benchmarks, baseline-adjusted skill, task average over choice, yes/no and score):
| Jalebi-E | Laya (typed decisions) | Jalebi-L (0.44B) | |
|---|---|---|---|
| Parameters | 66.6M | 440M | |
| S1MB task average | 22.94 | 15.0 | 19.2 |
| S1MB choice | 27.7 | 18.9 | 21.7 |
| S1MB score | 24.6 | 6.0 | 20.3 |
- Better than Jalebi-L with about one sixth of the parameters. Jalebi-E scores 22.94 against 19.2 on the S1MB task average, a gain of 3.74 points, from a model 6.6 times smaller.
- Far ahead of Laya on choice and score. On scoring against criteria, Jalebi-E is about four times Laya's baseline-adjusted score (24.6 against 6.0), and on choice about 1.5 times (27.7 against 18.9).
Decision Index 0.3, public tests
Run with the official runner on the public test suite (our own run; the maintainers' private tests were not run), Jalebi-E and Jalebi-L under the same rules (a request that does not fit the window is refused and counted as wrong):
| Jalebi-E | Jalebi-L | |
|---|---|---|
| Raw index | 18.8 | 15.8 |
| Requests refused | 27.6% | about 45% |
| Window | 2,048 tokens | 512 tokens |
Jalebi-E answers about a third more of the suite's requests than Jalebi-L, because its window is four times longer. Its two other training runs of the same recipe gave the same result within about 0.1 on the skill-adjusted index.
Speed
On the same GPU and the same sample of 6,000 requests (A100, our engine, tokenizer cache), Jalebi-E's median time per request was about 21% lower than Jalebi-L's (20.5 ms against 26.1 ms on the 2,885 requests both answer).
Files
jalebi_e.py: the whole model in one file (architecture, tokenizer helper, input encoding, scoring)jalebi-decision.pt: the weights (66.6M parameters)tok24k.json: the 24k-token tokenizerlogo.png,airavat-ai-logo.png: the Jalebi and Airavat AI logosSHA256SUMS: checksums of the files above
Use
# pip install torch tokenizers huggingface_hub (download jalebi_e.py from this repo next to your script)
from jalebi_e import JalebiE
model = JalebiE.from_pretrained("Airavat-ai/Jalebi-E")
probs = model.score(
"Customer writes: my card was declined while travelling abroad",
"Where should this request go?",
["billing team", "fraud team", "general support"],
) # one probability per option, summing to 1
An input that does not fit the 2,048-token window raises an error instead of being cut. Pass trim=True to opt in to dropping the end of a long state.
How the numbers were produced
- S1MB scores are from a full run of the public S1MB English v1 evaluator (137 benchmarks, 26,269 requests; evaluator commit
dbfc746) with thejalebi-eadapter merged upstream (S1MB pull request #66), on this checkpoint at revisiond2fd8d64, with the 2,048-token window. The results are submitted to the S1MB results dataset and are under review (discussion 118). Task average is the mean over the choice, yes/no and score tasks of the baseline-adjusted skill, times 100. Laya's figures are the ones used in the Jalebi-L card (typed-decisions variant). - Decision Index numbers are from our own run on the public suite; the board's score also includes private tests that we cannot see.
- The speed comparison is on identical hardware and requests, in our own engine; absolute times depend on the hardware.
Training data
Jalebi-E is trained from scratch: masked-language-model pretraining on web text, then decision fine-tuning. The fine-tuning data is:
- decision-formatted training data from the Bekko training pack;
- public decision datasets (train splits only);
- synthetic data written for this model: many-option intent classification rows (about 3,000, built from 89 label sets) and general decision rows.
Intended use
Routing a request to a model, a tool or a retriever; picking one option among many; yes/no gates on short structured inputs; scoring candidates against criteria. It is not a text generator and not a reasoning model.
Limitations
Jalebi-E reads at most 2,048 tokens per request; in our Decision Index runs a request that does not fit is refused. It is a small model: expect strong results on short structured decisions and weak results on open knowledge, maths and long reasoning.
What's next
Jalebi-E is small, fast and strong for its size, and it is not the end of the line. More Jalebi models are coming soon from Airavat AI.