Jalebi
A 0.44B decision model, trained from scratch. Give Jalebi a situation and a list of options and it returns a score for every option in one forward pass. It never generates text, so there is nothing to parse and nothing to retry.
It is built to sit in front of a large language model and a retrieval (RAG) system and answer the question that comes first: where should this request go, and should it be automated at all? It is also built to be fine-tuned on your own traffic.
Apache-2.0 Β· 0.44B parameters Β· no pretrained weights anywhere in the stack Β· English Β· by Airavat AI
| Jalebi | Laya (typed decisions) | |
|---|---|---|
| S1MB Task Average | 19.2 | 15.0 |
| S1MB Choice | 21.7 | 18.9 |
| S1MB Score | 20.3 | 6.0 |
| S1MB Noul (yes/no) | 15.6 | 20.1 |
| Time for the 137 S1MB tasks, same GPU | 63.8 s | 76.4 s (see Speed) |
| Routing, fine-tuned (Decision Bench, 150 items) | 88.7% | n/a |
Installation
huggingface-cli download Airavat-ai/Jalebi-L --local-dir jalebi
pip install torch numpy transformers
The repository carries the weights and four small code files (model.py, jalebi_tokenizer.py, decision.py, s1mb_jalebi.py) plus the tokenizer vocabulary. transformers is used for its BERT building blocks, and the tokenizers library builds the tokenizer; no pretrained weights are downloaded. Tested on CPU and on a single GPU.
Quickstart
import torch
from model import Jalebi, make_config
from jalebi_tokenizer import load_tokenizer
import decision
payload = torch.load("jalebi-routing.pt", map_location="cpu", weights_only=False)
model = Jalebi(make_config(**payload["shape"], dropout=0.0, pre_ln=True), option_pool=payload["option_pool"])
model.load_state_dict(payload["model"])
tok = load_tokenizer()
# The routing checkpoint was trained on this exact wording; keep the option strings and the state layout as they are.
ROUTES = ["human", "dept", "frontier", "workflow", "tool", "rag"]
OPTIONS = [
"human: a person must decide (bypass, irreversible, beyond one person's authority, or vague and costly)",
"dept: a specialist team handles it (sensitive matter or a case that needs specialist judgement)",
"frontier: send to a general model (outside the company's remit, or ask a clarifying question)",
"workflow: two or more internal systems in sequence, all reversible",
"tool: one well-defined reversible action or lookup in one tool",
"rag: a policy or procedure question the documentation answers",
]
request = "Please reset the password for staging test account build.qa17 in Account Console."
company = ("company (software). tools: Account Console, Ticket Desk, Access Hub. workflows: employee access setup. "
"teams: Engineering, IT Support, Security. knowledge base: Kiteframe Handbook. "
"policies: Exposed credentials are reported to Security.")
record = {"state": "request: " + request + "\n" + company, "question": "route: where should this request go?",
"options": OPTIONS, "option_ids": ROUTES}
probs = decision.predict(model, tok, [record], device=torch.device("cpu"))[0]
print({r: round(float(x), 3) for r, x in zip(ROUTES, probs)})
# {'human': 0.003, 'dept': 0.001, 'frontier': 0.002, 'workflow': 0.014, 'tool': 0.98, 'rag': 0.001} <- output of the routing checkpoint
This is an invented example that shows the input format; it is not a measure of accuracy (see the Decision Bench numbers below). Use jalebi-decision.pt (the general checkpoint) for choice, yes/no and grading tasks. Use jalebi-routing.pt for the six-route request routing above. A general checkpoint that has had no routing training cannot route (see Honest limits).
Jalebi also ships a small adapter (s1mb_jalebi.py) that implements the S1MB model-adapter protocol, so the benchmark's own runner scores it with no custom evaluation code.
Fine-tune it on your own usage
Jalebi is meant to be adapted. A decision model is only as good as the lines your company draws, and those lines are in your own traffic. Start from this checkpoint and train on your requests.
What fine-tuning did for routing, measured on the Decision Bench (150 held-out items, 25 per route):
| Model | Accuracy | Unsafe automation | Right route in top two |
|---|---|---|---|
Always tool |
16.7% | 100% | n/a |
| Jalebi, no routing training | 26.7% | 62.0% | 46.0% |
| Jalebi, fine-tuned on routing examples | 88.7% | 8.0% | 96.7% |
26.7% β 88.7%. None of the 150 test items, and no near-duplicate of one, appears in the fine-tuning data.
Put a confidence threshold on top and the same model gets more reliable where it is sure:
| Only act when confidence is at least | Share of requests kept | Accuracy on those |
|---|---|---|
| 0.5 | 95.3% | 90.9% |
| 0.7 | 84.0% | 93.7% |
| 0.9 | 60.7% | 96.7% |
| 0.95 | 39.3% | 98.3% |
Send the rest to a person or a larger model.
How far this goes:
- The gain was large and was still arriving at the end of our runs. We have not found a ceiling, but one exists: the base saw about 6B tokens of pretraining and its pretraining gains had flattened.
- Fine-tuning for one task moves the others. Our routing checkpoint scores 18.0 Task Average on S1MB, against 19.2 for the general one. Keep a general checkpoint if you need both.
- Report the unsafe-automation rate next to accuracy. A router that escalates everything is safe and useless.
Benchmarks
S1MB: Jalebi against Laya
S1MB is a public benchmark of decision tasks (137 English tasks, three families). Scores come from the benchmark's official evaluator. Bold is the best value in each column. Task Average is the mean of the three families.
| Model | Parameters | Task Avg | Noul | Choice | Score |
|---|---|---|---|---|---|
| Jalebi | 0.44 B | 19.2 | 15.6 | 21.7 | 20.3 |
| Laya (typed decisions) | 0.42 B | 15.0 | 20.1 | 18.9 | 6.0 |
| Laya | 0.42 B | 13.4 | 20.2 | 16.3 | 3.6 |
| Laya (multilingual) | 0.42 B | 9.3 | 14.0 | 13.0 | 0.8 |
Jalebi is ahead of every Laya variant on Task Average, Choice and Score, and about 14 points ahead on Score. Where Laya leads: yes/no. On the 59 yes/no tasks Laya scores about 20 and Jalebi about 16. If your workload is mostly yes/no verification, Laya is stronger out of the box today.
The Laya numbers are the benchmark's recorded results, used as published. The Jalebi numbers are from the same evaluator.
Where it sits among published models
| Model | Parameters | Task Avg |
|---|---|---|
| Bekko 400M | 0.39 B | 50.6 |
| Open-Jev-2B | 1.9 B | 34.6 |
| Jalebi | 0.44 B | 19.2 |
| Qwen3-Reranker-4B | 4.0 B | 16.1 |
| ModernBERT-large zero-shot | 0.4 B | 9.6 |
| Open-Jev-27B / AutoJev-27B / Eikos-27B | 25.6 B / 26.1 B / 26.9 B | 56.9 / 60.8 / 59.9 |
Recorded results from the benchmark's public results repository. Larger models, and some similar-sized ones, score much higher in absolute terms. Jalebi does not claim to be the most accurate small model. It is a fully open, from-scratch base at 0.44B parameters, with a clear lead over Laya at the same size.
Speed
Same machine, same 137 S1MB tasks, one model at a time, one NVIDIA RTX PRO 6000 (Blackwell, 96 GB) GPU on Google Colab. The time is the sum of the benchmark runner's own per-task timings.
| Time for all 137 tasks | |
|---|---|
| Jalebi | 63.8 s |
| Laya (typed decisions) | 76.4 s * |
| Bekko 400M | 145.5 s |
* Laya's run used the benchmark's default Laya settings. On tasks with long option lists (for example banking77, clinc and the NanoBEIR retrieval sets) Laya returned errors and those cases were not scored, so its time covers fewer cases than Jalebi's.
Laya's adapter needs a CUDA GPU, so there is no CPU comparison. Jalebi's own latency for one routing request with six options (tokenization included), measured on the same machine:
| Where | Batch size | Time per decision |
|---|---|---|
| GPU | 1 | 6.5 ms |
| GPU | 8 | 2.0 ms |
| GPU | 32 | 1.5 ms |
| CPU, 24 threads | 1 | 175 ms |
| CPU, 24 threads | 8 | 123 ms |
| CPU, 8 threads | 1 | 353 ms |
Decision Bench
The 150-item Jalebi Decision Bench with its official scorer. Results above are from the released routing checkpoint, rerun from scratch and scored with the benchmark's own score.py: accuracy 88.7%, macro F1 0.886, unsafe automation 8.0%, false escalation 6.7%, humans missed 24.0%, right route in the top two 96.7%, log loss 0.361. The released files reproduce the same route on all 150 items (scores agree to within 0.01 across machines).
Decision Index 0.3 (public tests)
Jalebi-L was run on the Decision Index 0.3 public suite with the official runner, on one A100, in float16. This is our own run of the public tests; the maintainers' private tests were not run. Numbers are from the general checkpoint unless stated.
1. Results on the requests Jalebi answered
Jalebi reads at most 512 tokens per request and never truncates. A request that does not fit is refused, not guessed. This table scores only the requests it answered, using each benchmark's own metric (not corrected for chance, so read each score against that task's chance level).
| Benchmark | Answered | Metric | Score |
|---|---|---|---|
| BANKING77 | 100% | macro-F1 | 0.537 |
| FinEntity | 100% | macro-F1 | 0.584 |
| CLINC150+OOS | 100% | macro-F1 | 0.337 |
| PhishNChips phishing decisions | 100% | accuracy | 0.432 |
| CRUXEval | 98% | accuracy | 0.399 |
| ARC-Easy | 100% | accuracy | 0.319 |
| HellaSwag | 35% | accuracy | 0.310 |
| ChessBench | 12% | accuracy | 0.325 |
| MMLU | 82% | accuracy | 0.259 |
| ARC-Challenge | 99% | accuracy | 0.233 |
| WinoGrande | 100% | accuracy | 0.504 |
| HoVer | 73% | accuracy | 0.504 |
| NLI4CT | 54% | macro-F1 | 0.504 |
| Humicroedit | 100% | accuracy | 0.502 |
| ANLI | 50% | macro-F1 | 0.301 |
| CLadder | 16% | accuracy | 0.541 |
| MMLU-Pro | 94% | accuracy | 0.101 |
| GSM8K | 100% | accuracy | 0.185 |
| BFCL | 38% | case-exact accuracy | 0.134 |
| HLE | 41% | accuracy | 0.163 |
Ten benchmarks were refused entirely (0% answered): ToolRet, API-Bank, RouterBench, Home appliance simulator, SGD/SGD-X, POP909-CL, cfcolor, MuSR, BRIGHT, Habermas Machine. ContractNLI was answered 2% of the time.
2. How to read it
- Jalebi is a small encoder built for short, structured decisions (routing, tool choice, classification). On requests it can see in full, it does real work on classification-style tasks (BANKING77, FinEntity, CLINC150) and is close to chance on knowledge, reasoning, math and most pair-classification tasks.
- About 45% of the suite's requests are over the 512-token window and were refused. The index scores a refusal as wrong. That is a property of the window, not of the model's judgement on those cases.
- The refusals are not the whole story: on the requests it does answer, Jalebi is near chance outside classification, so the answered-only table above should not be read as "strong where it answers".
3. Complete numbers, refusals counted as wrong (what the index reports)
| General checkpoint | Routing checkpoint | |
|---|---|---|
| Decision Index | 4.41 | 3.14 |
| Raw score | 15.82 | 14.57 |
| Requests answered / refused | 77,048 / 63,130 | about the same |
The general checkpoint's index and raw scores were identical in two separate runs of the full suite.
4. Speed
Latency is measured per request, in process, including tokenization and the forward pass; model loading is excluded.
General checkpoint, alone on one A100 (float16):
| Median | p95 | Mean |
|---|---|---|
| 25.7 ms | 29.5 ms | 27.0 ms |
- This is far inside the index's 1,000 ms per request limit. The whole 140,000-request suite ran in about 30 minutes.
- Almost all of the per-request time is CPU work (tokenizing and batching) in our engine, not the model's forward pass: the GPU was mostly idle during the run.
- For reference, medians published on the board include 5.8 ms (Laya), 6.8 ms (Bekko 17M), 17.1 ms (Bekko 400M), 12.6β62 ms for 4B decision models and 276β373 ms for the largest models. The board measures one model at a time on a different GPU, so these are indicative, not an exact like-for-like comparison. Jalebi is fast, but it is not the fastest small model on this index.
Reproduce
python -m decision_index pipeline --engine jalebi_engine:JalebiEngine --option checkpoint=jalebi-decision.pt --out runs/jalebi-general (use jalebi-routing.pt for the routing checkpoint), with jalebi_engine.py from this repository.
Deploying Jalebi in a stack
request -> Jalebi (decide the route) -> human | specialist team | workflow | single tool | RAG model | frontier model
For the best answers on hard requests you still need a frontier model or its API; Jalebi does not replace it. The saving comes from not paying frontier prices for every request: Jalebi makes the high-volume routing decision, a RAG model answers what documentation can answer, and the frontier model is called only when needed.
cost per 1,000 requests = routing cost + (share sent to RAG Γ RAG cost) + (share sent to frontier Γ frontier cost)
You supply your prices and routing shares; the saving depends entirely on your traffic. We have not measured end-to-end cost, latency or answer quality of a full stack, so this card makes no claim that a stack beats a single frontier model.
Architecture
A pre-LN, BERT-style encoder: 32 layers, hidden size 1024, 16 heads, feed-forward 4096, 0.44B parameters (438,051,649 in the evaluator's count, including the decision heads). Each option is marked in the input by a special [OPTION] token, and the model scores all options against the shared context in one forward pass; long option lists are split into chunks. The tokenizer is the bert-base-uncased vocabulary (vocab.txt, included) plus [OPTION].
Training
Pretrained from scratch with masked-language modelling for about 6 billion tokens. No pretrained encoder, tokenizer weights or other external initialisation was used. The model is then fine-tuned for decisions.
Gains from more pretraining tokens had mostly flattened by 6B.
Honest limits
- Yes/no is the weak task. Across the 59 S1MB yes/no tasks the model separates true from false only a little better than chance (average AUC about 0.60 on an earlier checkpoint), and its yes/no probabilities are over-confident. Do not treat them as calibrated; check on your own data first.
- It cannot route without routing training. On the 150 routing items the general model scores 26.7% (chance is 16.7%) and sends 62.0% of the cases that need a person or a team to an automated route.
- Unsafe automation is not zero. The fine-tuned routing model sent 8.0% of the 150 items to an automated route when they needed a person or a team, and missed 24.0% of the cases that should go to a person. Use the confidence threshold and keep a human path.
- The routing test is synthetic. The Decision Bench uses invented companies and requests. Real traffic will differ, and every company draws its own lines. Do not read 88.7% as real-world accuracy.
- S1MB scores are in-family. Fine-tuning used training splits from the same source families as many S1MB tasks; the benchmark's test splits were excluded and screened out. Other models on the benchmark declare the same kind of exposure.
- Not a general reasoner. Jalebi is a cheap gatekeeper. The largest models are far more accurate in absolute terms, and several models of similar size score much higher on S1MB.
- English only, single-turn.
- Decision Index: long requests are refused. About 45% of the index's public requests exceed the 512-token window and are refused, which the index scores as wrong; on the requests it answers, Jalebi is near chance outside classification-style tasks (see Decision Index 0.3).
- Speed numbers are for one GPU on one machine. Laya's run skipped some cases (see Speed); there is no CPU comparison.
Files
| File | What it is |
|---|---|
jalebi-decision.pt |
general decision checkpoint (weights and shape) |
jalebi-routing.pt |
routing checkpoint |
model.py, jalebi_tokenizer.py, decision.py |
model, tokenizer and scoring code |
s1mb_jalebi.py |
adapter that lets the S1MB runner score Jalebi |
jalebi_engine.py |
adapter that lets the Decision Index runner score Jalebi |
vocab.txt |
tokenizer vocabulary |
config.json |
model shape (for reference; the code reads the shape stored in each checkpoint) |
SHA256SUMS |
checksums of every file |
LICENSE, logo.png |
Apache-2.0 licence text and logo |
Links
Licence and citation
Apache-2.0.
@misc{jalebi_2026,
title = {Jalebi: a small decision-scoring encoder},
author = {Airavat AI},
year = {2026}
}
- Downloads last month
- 15
