Jalebi      Airavat AI

Jalebi-E

A 66.6M-parameter decision model, trained from scratch, with a 2,048-token window. Give Jalebi-E a situation and a list of options and it scores every option in one forward pass. It never generates text, so there is nothing to hallucinate and nothing to prompt-inject into a free-form answer. It answers three kinds of decision: pick an option (choice), give a yes/no probability (noul), or score against criteria (score).

It is built to sit in front of a large language model, a retrieval system or a tool-calling agent and answer the question that comes first: where should this request go?

Apache-2.0 路 66.6M parameters 路 no pretrained weights anywhere in the stack 路 2,048-token window 路 English 路 by Airavat AI

Jalebi-E is one of the smallest models we have built, and one of the most powerful. At 66.6M parameters it outscores our 0.44B Jalebi-L on the S1MB task average (22.94 against 19.2), with a context window four times longer. More Jalebi models are coming soon.

Results

On the S1MB English v1 benchmark (137 benchmarks, baseline-adjusted skill, task average over choice, yes/no and score):

Jalebi-E Laya (typed decisions) Jalebi-L (0.44B)
Parameters 66.6M 440M
S1MB task average 22.94 15.0 19.2
S1MB choice 27.7 18.9 21.7
S1MB score 24.6 6.0 20.3
  • Better than Jalebi-L with about one sixth of the parameters. Jalebi-E scores 22.94 against 19.2 on the S1MB task average, a gain of 3.74 points, from a model 6.6 times smaller.
  • Far ahead of Laya on choice and score. On scoring against criteria, Jalebi-E is about four times Laya's baseline-adjusted score (24.6 against 6.0), and on choice about 1.5 times (27.7 against 18.9).

Decision Index 0.3, public tests

Run with the official runner on the public test suite (our own run; the maintainers' private tests were not run), Jalebi-E and Jalebi-L under the same rules (a request that does not fit the window is refused and counted as wrong):

Jalebi-E Jalebi-L
Raw index 18.8 15.8
Requests refused 27.6% about 45%
Window 2,048 tokens 512 tokens

Jalebi-E answers about a third more of the suite's requests than Jalebi-L, because its window is four times longer. Its two other training runs of the same recipe gave the same result within about 0.1 on the skill-adjusted index.

Speed

On the same GPU and the same sample of 6,000 requests (A100, our engine, tokenizer cache), Jalebi-E's median time per request was about 21% lower than Jalebi-L's (20.5 ms against 26.1 ms on the 2,885 requests both answer).

Files

  • jalebi_e.py: the whole model in one file (architecture, tokenizer helper, input encoding, scoring)
  • jalebi-decision.pt: the weights (66.6M parameters)
  • tok24k.json: the 24k-token tokenizer
  • logo.png, airavat-ai-logo.png: the Jalebi and Airavat AI logos
  • SHA256SUMS: checksums of the files above

Use

# pip install torch tokenizers huggingface_hub   (download jalebi_e.py from this repo next to your script)
from jalebi_e import JalebiE

model = JalebiE.from_pretrained("Airavat-ai/Jalebi-E")
probs = model.score(
    "Customer writes: my card was declined while travelling abroad",
    "Where should this request go?",
    ["billing team", "fraud team", "general support"],
)   # one probability per option, summing to 1

An input that does not fit the 2,048-token window raises an error instead of being cut. Pass trim=True to opt in to dropping the end of a long state.

How the numbers were produced

  • S1MB scores are from a full run of the public S1MB English v1 evaluator (137 benchmarks, 26,269 requests; evaluator commit dbfc746) with the jalebi-e adapter merged upstream (S1MB pull request #66), on this checkpoint at revision d2fd8d64, with the 2,048-token window. The results are submitted to the S1MB results dataset and are under review (discussion 118). Task average is the mean over the choice, yes/no and score tasks of the baseline-adjusted skill, times 100. Laya's figures are the ones used in the Jalebi-L card (typed-decisions variant).
  • Decision Index numbers are from our own run on the public suite; the board's score also includes private tests that we cannot see.
  • The speed comparison is on identical hardware and requests, in our own engine; absolute times depend on the hardware.

Training data

Jalebi-E is trained from scratch: masked-language-model pretraining on web text, then decision fine-tuning. The fine-tuning data is:

  • decision-formatted training data from the Bekko training pack;
  • public decision datasets (train splits only);
  • synthetic data written for this model: many-option intent classification rows (about 3,000, built from 89 label sets) and general decision rows.

Intended use

Routing a request to a model, a tool or a retriever; picking one option among many; yes/no gates on short structured inputs; scoring candidates against criteria. It is not a text generator and not a reasoning model.

Limitations

Jalebi-E reads at most 2,048 tokens per request; in our Decision Index runs a request that does not fit is refused. It is a small model: expect strong results on short structured decisions and weak results on open knowledge, maths and long reasoning.

What's next

Jalebi-E is small, fast and strong for its size, and it is not the end of the line. More Jalebi models are coming soon from Airavat AI.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 馃檵 Ask for provider support

Space using AIRAVAT-AI-ML/Jalebi-E 1