Jalebi

Jalebi

A 0.44B decision model, trained from scratch. Give Jalebi a situation and a list of options and it returns a score for every option in one forward pass. It never generates text, so there is nothing to parse and nothing to retry.

It is built to sit in front of a large language model and a retrieval (RAG) system and answer the question that comes first: where should this request go, and should it be automated at all? It is also built to be fine-tuned on your own traffic.

Apache-2.0 Β· 0.44B parameters Β· no pretrained weights anywhere in the stack Β· English Β· by Airavat AI

Jalebi Laya (typed decisions)
S1MB Task Average 19.2 15.0
S1MB Choice 21.7 18.9
S1MB Score 20.3 6.0
S1MB Noul (yes/no) 15.6 20.1
Time for the 137 S1MB tasks, same GPU 63.8 s 76.4 s (see Speed)
Routing, fine-tuned (Decision Bench, 150 items) 88.7% n/a

Installation

huggingface-cli download Airavat-ai/Jalebi-L --local-dir jalebi
pip install torch numpy transformers

The repository carries the weights and four small code files (model.py, jalebi_tokenizer.py, decision.py, s1mb_jalebi.py) plus the tokenizer vocabulary. transformers is used for its BERT building blocks, and the tokenizers library builds the tokenizer; no pretrained weights are downloaded. Tested on CPU and on a single GPU.

Quickstart

import torch
from model import Jalebi, make_config
from jalebi_tokenizer import load_tokenizer
import decision

payload = torch.load("jalebi-routing.pt", map_location="cpu", weights_only=False)
model = Jalebi(make_config(**payload["shape"], dropout=0.0, pre_ln=True), option_pool=payload["option_pool"])
model.load_state_dict(payload["model"])
tok = load_tokenizer()

# The routing checkpoint was trained on this exact wording; keep the option strings and the state layout as they are.
ROUTES = ["human", "dept", "frontier", "workflow", "tool", "rag"]
OPTIONS = [
    "human: a person must decide (bypass, irreversible, beyond one person's authority, or vague and costly)",
    "dept: a specialist team handles it (sensitive matter or a case that needs specialist judgement)",
    "frontier: send to a general model (outside the company's remit, or ask a clarifying question)",
    "workflow: two or more internal systems in sequence, all reversible",
    "tool: one well-defined reversible action or lookup in one tool",
    "rag: a policy or procedure question the documentation answers",
]
request = "Please reset the password for staging test account build.qa17 in Account Console."
company = ("company (software). tools: Account Console, Ticket Desk, Access Hub. workflows: employee access setup. "
           "teams: Engineering, IT Support, Security. knowledge base: Kiteframe Handbook. "
           "policies: Exposed credentials are reported to Security.")
record = {"state": "request: " + request + "\n" + company, "question": "route: where should this request go?",
          "options": OPTIONS, "option_ids": ROUTES}
probs = decision.predict(model, tok, [record], device=torch.device("cpu"))[0]
print({r: round(float(x), 3) for r, x in zip(ROUTES, probs)})
# {'human': 0.003, 'dept': 0.001, 'frontier': 0.002, 'workflow': 0.014, 'tool': 0.98, 'rag': 0.001}   <- output of the routing checkpoint

This is an invented example that shows the input format; it is not a measure of accuracy (see the Decision Bench numbers below). Use jalebi-decision.pt (the general checkpoint) for choice, yes/no and grading tasks. Use jalebi-routing.pt for the six-route request routing above. A general checkpoint that has had no routing training cannot route (see Honest limits).

Jalebi also ships a small adapter (s1mb_jalebi.py) that implements the S1MB model-adapter protocol, so the benchmark's own runner scores it with no custom evaluation code.

Fine-tune it on your own usage

Jalebi is meant to be adapted. A decision model is only as good as the lines your company draws, and those lines are in your own traffic. Start from this checkpoint and train on your requests.

What fine-tuning did for routing, measured on the Decision Bench (150 held-out items, 25 per route):

Model Accuracy Unsafe automation Right route in top two
Always tool 16.7% 100% n/a
Jalebi, no routing training 26.7% 62.0% 46.0%
Jalebi, fine-tuned on routing examples 88.7% 8.0% 96.7%

26.7% β†’ 88.7%. None of the 150 test items, and no near-duplicate of one, appears in the fine-tuning data.

Put a confidence threshold on top and the same model gets more reliable where it is sure:

Only act when confidence is at least Share of requests kept Accuracy on those
0.5 95.3% 90.9%
0.7 84.0% 93.7%
0.9 60.7% 96.7%
0.95 39.3% 98.3%

Send the rest to a person or a larger model.

How far this goes:

  • The gain was large and was still arriving at the end of our runs. We have not found a ceiling, but one exists: the base saw about 6B tokens of pretraining and its pretraining gains had flattened.
  • Fine-tuning for one task moves the others. Our routing checkpoint scores 18.0 Task Average on S1MB, against 19.2 for the general one. Keep a general checkpoint if you need both.
  • Report the unsafe-automation rate next to accuracy. A router that escalates everything is safe and useless.

Benchmarks

S1MB: Jalebi against Laya

S1MB is a public benchmark of decision tasks (137 English tasks, three families). Scores come from the benchmark's official evaluator. Bold is the best value in each column. Task Average is the mean of the three families.

Model Parameters Task Avg Noul Choice Score
Jalebi 0.44 B 19.2 15.6 21.7 20.3
Laya (typed decisions) 0.42 B 15.0 20.1 18.9 6.0
Laya 0.42 B 13.4 20.2 16.3 3.6
Laya (multilingual) 0.42 B 9.3 14.0 13.0 0.8

Jalebi is ahead of every Laya variant on Task Average, Choice and Score, and about 14 points ahead on Score. Where Laya leads: yes/no. On the 59 yes/no tasks Laya scores about 20 and Jalebi about 16. If your workload is mostly yes/no verification, Laya is stronger out of the box today.

The Laya numbers are the benchmark's recorded results, used as published. The Jalebi numbers are from the same evaluator.

Where it sits among published models

Model Parameters Task Avg
Bekko 400M 0.39 B 50.6
Open-Jev-2B 1.9 B 34.6
Jalebi 0.44 B 19.2
Qwen3-Reranker-4B 4.0 B 16.1
ModernBERT-large zero-shot 0.4 B 9.6
Open-Jev-27B / AutoJev-27B / Eikos-27B 25.6 B / 26.1 B / 26.9 B 56.9 / 60.8 / 59.9

Recorded results from the benchmark's public results repository. Larger models, and some similar-sized ones, score much higher in absolute terms. Jalebi does not claim to be the most accurate small model. It is a fully open, from-scratch base at 0.44B parameters, with a clear lead over Laya at the same size.

Speed

Same machine, same 137 S1MB tasks, one model at a time, one NVIDIA RTX PRO 6000 (Blackwell, 96 GB) GPU on Google Colab. The time is the sum of the benchmark runner's own per-task timings.

Time for all 137 tasks
Jalebi 63.8 s
Laya (typed decisions) 76.4 s *
Bekko 400M 145.5 s

* Laya's run used the benchmark's default Laya settings. On tasks with long option lists (for example banking77, clinc and the NanoBEIR retrieval sets) Laya returned errors and those cases were not scored, so its time covers fewer cases than Jalebi's.

Laya's adapter needs a CUDA GPU, so there is no CPU comparison. Jalebi's own latency for one routing request with six options (tokenization included), measured on the same machine:

Where Batch size Time per decision
GPU 1 6.5 ms
GPU 8 2.0 ms
GPU 32 1.5 ms
CPU, 24 threads 1 175 ms
CPU, 24 threads 8 123 ms
CPU, 8 threads 1 353 ms

Decision Bench

The 150-item Jalebi Decision Bench with its official scorer. Results above are from the released routing checkpoint, rerun from scratch and scored with the benchmark's own score.py: accuracy 88.7%, macro F1 0.886, unsafe automation 8.0%, false escalation 6.7%, humans missed 24.0%, right route in the top two 96.7%, log loss 0.361. The released files reproduce the same route on all 150 items (scores agree to within 0.01 across machines).

Decision Index 0.3 (public tests)

Jalebi-L was run on the Decision Index 0.3 public suite with the official runner, on one A100, in float16. This is our own run of the public tests; the maintainers' private tests were not run. Numbers are from the general checkpoint unless stated.

1. Results on the requests Jalebi answered

Jalebi reads at most 512 tokens per request and never truncates. A request that does not fit is refused, not guessed. This table scores only the requests it answered, using each benchmark's own metric (not corrected for chance, so read each score against that task's chance level).

Benchmark Answered Metric Score
BANKING77 100% macro-F1 0.537
FinEntity 100% macro-F1 0.584
CLINC150+OOS 100% macro-F1 0.337
PhishNChips phishing decisions 100% accuracy 0.432
CRUXEval 98% accuracy 0.399
ARC-Easy 100% accuracy 0.319
HellaSwag 35% accuracy 0.310
ChessBench 12% accuracy 0.325
MMLU 82% accuracy 0.259
ARC-Challenge 99% accuracy 0.233
WinoGrande 100% accuracy 0.504
HoVer 73% accuracy 0.504
NLI4CT 54% macro-F1 0.504
Humicroedit 100% accuracy 0.502
ANLI 50% macro-F1 0.301
CLadder 16% accuracy 0.541
MMLU-Pro 94% accuracy 0.101
GSM8K 100% accuracy 0.185
BFCL 38% case-exact accuracy 0.134
HLE 41% accuracy 0.163

Ten benchmarks were refused entirely (0% answered): ToolRet, API-Bank, RouterBench, Home appliance simulator, SGD/SGD-X, POP909-CL, cfcolor, MuSR, BRIGHT, Habermas Machine. ContractNLI was answered 2% of the time.

2. How to read it

  • Jalebi is a small encoder built for short, structured decisions (routing, tool choice, classification). On requests it can see in full, it does real work on classification-style tasks (BANKING77, FinEntity, CLINC150) and is close to chance on knowledge, reasoning, math and most pair-classification tasks.
  • About 45% of the suite's requests are over the 512-token window and were refused. The index scores a refusal as wrong. That is a property of the window, not of the model's judgement on those cases.
  • The refusals are not the whole story: on the requests it does answer, Jalebi is near chance outside classification, so the answered-only table above should not be read as "strong where it answers".

3. Complete numbers, refusals counted as wrong (what the index reports)

General checkpoint Routing checkpoint
Decision Index 4.41 3.14
Raw score 15.82 14.57
Requests answered / refused 77,048 / 63,130 about the same

The general checkpoint's index and raw scores were identical in two separate runs of the full suite.

4. Speed

Latency is measured per request, in process, including tokenization and the forward pass; model loading is excluded.

General checkpoint, alone on one A100 (float16):

Median p95 Mean
25.7 ms 29.5 ms 27.0 ms
  • This is far inside the index's 1,000 ms per request limit. The whole 140,000-request suite ran in about 30 minutes.
  • Almost all of the per-request time is CPU work (tokenizing and batching) in our engine, not the model's forward pass: the GPU was mostly idle during the run.
  • For reference, medians published on the board include 5.8 ms (Laya), 6.8 ms (Bekko 17M), 17.1 ms (Bekko 400M), 12.6–62 ms for 4B decision models and 276–373 ms for the largest models. The board measures one model at a time on a different GPU, so these are indicative, not an exact like-for-like comparison. Jalebi is fast, but it is not the fastest small model on this index.

Reproduce

python -m decision_index pipeline --engine jalebi_engine:JalebiEngine --option checkpoint=jalebi-decision.pt --out runs/jalebi-general (use jalebi-routing.pt for the routing checkpoint), with jalebi_engine.py from this repository.

Deploying Jalebi in a stack

request -> Jalebi (decide the route) -> human | specialist team | workflow | single tool | RAG model | frontier model

For the best answers on hard requests you still need a frontier model or its API; Jalebi does not replace it. The saving comes from not paying frontier prices for every request: Jalebi makes the high-volume routing decision, a RAG model answers what documentation can answer, and the frontier model is called only when needed.

cost per 1,000 requests = routing cost + (share sent to RAG Γ— RAG cost) + (share sent to frontier Γ— frontier cost)

You supply your prices and routing shares; the saving depends entirely on your traffic. We have not measured end-to-end cost, latency or answer quality of a full stack, so this card makes no claim that a stack beats a single frontier model.

Architecture

A pre-LN, BERT-style encoder: 32 layers, hidden size 1024, 16 heads, feed-forward 4096, 0.44B parameters (438,051,649 in the evaluator's count, including the decision heads). Each option is marked in the input by a special [OPTION] token, and the model scores all options against the shared context in one forward pass; long option lists are split into chunks. The tokenizer is the bert-base-uncased vocabulary (vocab.txt, included) plus [OPTION].

Training

Pretrained from scratch with masked-language modelling for about 6 billion tokens. No pretrained encoder, tokenizer weights or other external initialisation was used. The model is then fine-tuned for decisions.

Gains from more pretraining tokens had mostly flattened by 6B.

Honest limits

  • Yes/no is the weak task. Across the 59 S1MB yes/no tasks the model separates true from false only a little better than chance (average AUC about 0.60 on an earlier checkpoint), and its yes/no probabilities are over-confident. Do not treat them as calibrated; check on your own data first.
  • It cannot route without routing training. On the 150 routing items the general model scores 26.7% (chance is 16.7%) and sends 62.0% of the cases that need a person or a team to an automated route.
  • Unsafe automation is not zero. The fine-tuned routing model sent 8.0% of the 150 items to an automated route when they needed a person or a team, and missed 24.0% of the cases that should go to a person. Use the confidence threshold and keep a human path.
  • The routing test is synthetic. The Decision Bench uses invented companies and requests. Real traffic will differ, and every company draws its own lines. Do not read 88.7% as real-world accuracy.
  • S1MB scores are in-family. Fine-tuning used training splits from the same source families as many S1MB tasks; the benchmark's test splits were excluded and screened out. Other models on the benchmark declare the same kind of exposure.
  • Not a general reasoner. Jalebi is a cheap gatekeeper. The largest models are far more accurate in absolute terms, and several models of similar size score much higher on S1MB.
  • English only, single-turn.
  • Decision Index: long requests are refused. About 45% of the index's public requests exceed the 512-token window and are refused, which the index scores as wrong; on the requests it answers, Jalebi is near chance outside classification-style tasks (see Decision Index 0.3).
  • Speed numbers are for one GPU on one machine. Laya's run skipped some cases (see Speed); there is no CPU comparison.

Files

File What it is
jalebi-decision.pt general decision checkpoint (weights and shape)
jalebi-routing.pt routing checkpoint
model.py, jalebi_tokenizer.py, decision.py model, tokenizer and scoring code
s1mb_jalebi.py adapter that lets the S1MB runner score Jalebi
jalebi_engine.py adapter that lets the Decision Index runner score Jalebi
vocab.txt tokenizer vocabulary
config.json model shape (for reference; the code reads the shape stored in each checkpoint)
SHA256SUMS checksums of every file
LICENSE, logo.png Apache-2.0 licence text and logo

Links

Jalebi Decision Bench

Licence and citation

Apache-2.0.

@misc{jalebi_2026,
  title  = {Jalebi: a small decision-scoring encoder},
  author = {Airavat AI},
  year   = {2026}
}
Downloads last month
15
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support