jeb-typed-decisions

Specialised checkpoint of IJyad/jeb, fine-tuned on four Arabic operational workflows. Same architecture (MARBERTv2 + option-marker head, 178M), same ~9 ms single forward pass, same calibrated probabilities — trained to answer the kinds of typed questions software actually asks about a ticket, an invoice, an alert or an agent run.

Now also fine-tuned on najdresearch/system-one Arabic training data for policy-based decision tasks. See BENCHMARKS.md for full results.

The four workflows

workflow	state	questions asked
invoice	فاتورة من مورد	action · duplicate · overdue · amount band · needs human
security	بلاغ أمني	severity · category · still active · page on-call · needs human
support	رسالة عميل	department · urgency · sentiment · churn risk · needs human
agent_trace	أثر تنفيذ وكيل	outcome · loop · tool error · trace length · needs human

Five typed questions per case, choice type throughout.

Results

2,700 held-out decisions, 400+ per workflow. Label noise in the data is 12%, so ~0.88 is the practical ceiling — these numbers sit right at it.

workflow	accuracy	n
invoice processing	0.8765	664
security incidents	0.8652	675
customer service	0.8906	658
agent-trace observability	0.8691	703
overall	0.8752	2,700

p50 latency, 1 question	8.2 ms
ECE (lower better)	0.0524
params	178M

For reference, Laya's laya-typed-decisions reports 0.766 on its own English typed-decisions benchmark. The two numbers are not comparable — different languages, different data, different label sets. Ours is synthetic; theirs is not.

Usage

import model
m = model.load('.')          # after: huggingface-cli download IJyad/jeb-typed-decisions --local-dir td

state = "وصلتنا مطالبة بقيمة 7500 ريال من شركة الفجر، رقمها 4182. فيه فاتورة تشبهها وصلت قبل أسبوع."

print(m.predict(state, {
  "action":      {"type":"choice","instructions":"ما الإجراء المناسب لهذه الفاتورة؟",
                  "criteria":{"approve":"اعتمدها","reject":"ارفضها","escalate":"ارفعها للمراجعة"}},
  "duplicate":   {"type":"choice","instructions":"هل هذه الفاتورة مكررة؟",
                  "criteria":{"yes":"نعم مكررة","no":"لا"}},
  "needs_human": {"type":"choice","instructions":"هل يحتاج الطلب مراجعة بشرية؟",
                  "criteria":{"yes":"نعم","no":"لا"}},
}))

How the training data was built — read this

The dataset is synthetic, generated programmatically in Arabic with Saudi/Gulf phrasing. That is a real limitation and it is stated here rather than buried.

A first version scored 1.0000 on its own test split. That was not a good model — it was a broken benchmark: every label was a deterministic function of the text (approve if amount < 5000 and not duplicate), so the model recovered the generator instead of learning judgement. It was thrown away.

The shipped dataset deliberately breaks that recoverability:

• 12% label noise — mirrors real annotator disagreement
• borderline values — amounts straddle decision thresholds (4,800 vs 5,200)
• hedged language — "يمكن تكون مكررة، ما أنا متأكد" (maybe duplicate, not sure)
• paraphrase variety — four phrasings per workflow, not one template
• partial outcomes — "أنهى جزء من المهمة بس مو كلها" (finished part of it)

With 12% noise the achievable ceiling is roughly 0.88, so treat that as the reference point, not 1.0.

What this checkpoint is not: validated on real production tickets. It is a demonstration that the jeb architecture specialises cleanly onto typed-decision schemas, and a starting point for fine-tuning on your own labelled data. Benchmarks against real Arabic datasets live on the base model.

Apache 2.0