ej / README.md

Model card: add comparison with other decision models (same final suites)

476ef0b verified 16 days ago

16 kB

---
license: cc-by-sa-4.0
base_model: intfloat/e5-small-v2
library_name: ej
language:
- en
pipeline_tag: text-classification
tags:
- text-classification
- zero-shot-classification
- calibration
- typed-decisions
---
# ej model card (0.0.1)

ej 0.0.1 is a small calibrated decision model; technical report forthcoming.

> **Weights:** https://huggingface.co/5ak3t/ej, revision `v0.0.1`, one file `model.ejpack` (11,384,312 bytes, SHA-256
> `e990e1846cba43f8405a969c606057f2fd6e2076595f4a34d202e8fc531891b0`). Code, training and evaluation:
> https://github.com/5ak3t/ej (file paths below refer to that repository). Release notes: `docs/releases/0.0.1.md`.

| Field | Value |
|---|---|
| Model | ej 0.0.1 (2026-10-08); the default fit seed, fixed before fitting, not a selected seed |
| State key | `3b3e66d28fb423f98b734bd0c1d324cdc92bb8e10eda0e7ddf9a8e54de3f3fe2` (recorded in the pack header; reproduced by `scripts/state_key.py --git <model-code repo> f46cf7c <training pool> --ck-dir lowbit-b3b010513f948ceb`) |
| Code | runtime: `ej/_runtime` of the GitHub repository (39 sha256-pinned modules; `ej.integrity.RELEASE_PROVENANCE`); fitted with model-code commit `f46cf7c` of the maintainers' development repository (not public) and encoder checkpoint `lowbit-b3b010513f948ceb`; the same recipe is `python -m ej.train` |
| Size | counted 10.87 MiB (11.40 MB; a bit-level bound on the stored parameters); on disk 11.4 MB: one file `model.ejpack` of 11,384,312 bytes, the whole download (no base model is fetched); resident: peak live model tensors while predicting 128 MB by default, 35 MB with `low_memory=True`; process RSS during predict about 558 / 455 MB, of which about 204 MB is `import torch` (README "Memory and latency") |
| Latency (CPU, Python reference, 1 thread, one record per call) | packed model, first 50 td development records, warm: 214 ms per record (317 ms with `low_memory=True`); cold 1,061 / 1,385 ms (2026-10-09, Intel Xeon @ 2.80GHz, 4 logical cores, contended). Benchmark protocol on the td development suite (weights-directory format, 2026-10-08): warm 356.6 ms mean / 363.1 ms median, cold 1,922.0 / 1,413.7 ms. The two measurements are not comparable (README "Latency") |
| Language | English only (§6) |
| Licence | weights **CC BY-SA 4.0**; code Apache-2.0 |
| Base model | `intfloat/e5-small-v2` (MIT), revision `ffb93f3bd4047442299a41ebb6fa998a38507c52` |
| Weights / code | https://huggingface.co/5ak3t/ej (revision `v0.0.1`, file `model.ejpack`) / https://github.com/5ak3t/ej |

## 1. What the model does

A **System-1 decision model for devices**: given a `state` (text, or a JSON object as text) and typed questions, it returns
one probability distribution per question in **one pass**, without decoding tokens. Usage: Quickstart below and the
repository README.

| Question type | Input | Output |
|---|---|---|
| `choice` | instructions + a runtime list of option texts | a probability per option |
| `noul` | a yes/no statement | P(false), P(true) |
| `score` | instructions + an ordered list of level texts | a distribution over the levels |

- **No LLM at inference**; teachers are used at fit time only and their weights do not ship.
- **Calibration is fitted, then checked per suite**: whether the result is calibrated is measured on each suite against a
  perfect-calibration ECE floor (§5).
- **Record independence.** Zero-shot `predict` is record-independent: each record's output depends only on that record (up
  to the float noise of batching). `adapt` (and `AdaptedModel.observe`) is an opt-in, per-workflow transductive mode: it
  pools option statistics across that workflow's labelled records, so use it only for one workflow with a fixed option set
  (the same question ids and option keys).
- Option texts are read as text, so new label spaces can be asked zero-shot; quality on unseen workflows is limited (§6).

## 2. Quickstart

Architecture and method: technical report forthcoming.

```bash
git clone https://github.com/5ak3t/ej && cd ej
pip install -e .
```

```python
import ej
model = ej.load('5ak3t/ej', revision='v0.0.1')   # downloads model.ejpack only, checks its sha256, nothing else is fetched
(probs,) = model.predict([ej.EXAMPLE_RECORD])    # {qid: [p for each option, in option order]}
```

`ej.load` also takes a local `model.ejpack` (or a directory holding it). `python examples/quickstart.py --weights 5ak3t/ej
--revision v0.0.1` prints the distributions for `ej.EXAMPLE_RECORD`. Input and output format: the repository README.
## 3. Training data
One fit on one training pool (9,719 records, 7 source groups; sha256
`ce1c1a6fecfd62a90317f6efc4f90fd5f9261becb7081fd00235a6f4d5ee9dbe`). Details and counts: `docs/data-card.md`.
| Source (groups) | Records | Licence |
|---|---|---|
| Typed Decisions, workflows `agent_trace_observability`, `customer_service`, `invoice_processing` (3) | 736 | Apache-2.0 |
| **Support tickets written by Claude Haiku** for this project, train + calibration splits (1) | 780 | in-house; **not released** |
| Banking77 / CLINC150 `plus` / GoEmotions (3) | 3,000 / 3,000 / 2,203 | CC BY 4.0 / CC BY 3.0 / Apache-2.0 |
| **Total** | 9,719 records, 7 groups | |
Amazon counterfactual is not used: its upstream licence is CC BY-NC 4.0 (one mirror declares CC BY 4.0).
**Disclosure: Claude Haiku.** The in-domain support tickets were written by Claude Haiku (labels by construction from the
generation spec, no human review). They are part of training; the corpus itself is **not released**, so the training pool
cannot be rebuilt from public data alone. Open licence questions on this and other inputs: `NOTICE` (Q1-Q4).
Not in the training data: Typed Decisions `security_incidents` (the `zs_td` suite), MASSIVE and the zs_wide workflows
(evaluation only).

## 4. Evaluation

- **Suites** (`benchmarks/README.md`; never trained on):
  - `td`: Typed Decisions test split, the 3 training workflows.
  - `zs_td`: zs_td is ONE held-out workflow (security_incidents; dev 300 and final 100 records of the same workflow).
    The model code was kept on its dev accuracy after about 55 logged comparisons on it, so final zs_td estimates accuracy on
    further records of a workflow the model was selected on: it is neither unbiased nor unseen-workflow evidence. Its
    fit-to-fit SD (same code, other RNG) is ~.022 accuracy per fit (SD of a difference .031, 4 pairs), twice its
    record-cluster SE (.011); one workflow gives no between-workflow variance. Unseen-workflow evidence = zs_wide final
    macro_real.
  - `zs_massive`: MASSIVE (en) intents with leak-free option sets. MASSIVE was never trained on, but its label space is
    not new: about 22% of its intent names (13 of 59 development option texts: 1 identical, 12 near) overlap CLINC150 /
    Banking77 option texts in the pool, and 4 final records are exact duplicates of pool records (removed at the next
    benchmark run).
  - `zs_wide`: workflows never trained on: dev 154 workflows (111 from public sources: SNI tasks, SGD services, ABCD; 43
    GLM-synthetic), final 147. The split is by task, not by dataset family: 53 of the 147 final workflows share a family
    with a dev workflow; pool-source tasks were dropped from the final set.
  - `tickets` (in-domain, private) and `tickets_ood` (held-out writing styles of the same generator: a style shift).
- **Metrics**: per-suite mean NLL of the gold option (headline: the mean over td, zs_td, zs_massive, tickets), micro
  accuracy (zs_wide: group-macro over the real sources), ECE over 15 bins with its perfect-calibration floor, certified
  automation (risk .10, δ .10; **this suite only**). Each cell has a 95% record-cluster CI. Scoring: `ej.eval`
  (`benchmarks/scoring.py` for rivals).
- **Selection.** Development numbers are selected: more than 55 logged comparisons were made on the development suites, so
  their numbers are optimistic. The final suites are read once per release; zs_td final is not unseen-workflow evidence
  (above).
- **Leak-free option sets**: for sampled option sets every option is equally likely to be the gold, so option frequency
  reveals nothing (a frequency rule reached .631 micro accuracy against chance .240 on a non-leak-free MASSIVE suite).
- **Rivals** and contamination flags: `benchmarks/METHOD.md`.
## 5. Results

**ej 0.0.1** (final suites, scored once, 2026-10-08: `benchmarks/results/README.md`). 95% record-cluster CIs; calibrated
= observed ECE15 at or below the 95th percentile of a perfectly calibrated model's ECE15 on that suite.

| Suite | NLL [95% CI] | Micro accuracy [95% CI] | ECE15 [95% CI] | Calibrated on this suite | Certified automation, this suite only (coverage / error) |
|---|---|---|---|---|---|
| td (seen workflows) | .663 [.613, .716] | .721 [.695, .746] | .071 [.050, .094] | no (floor q95 .034) | .300 / .056 |
| zs_td (`security_incidents`, selected on) | 1.145 [1.113, 1.180] | .422 [.388, .452] | .094 [.068, .135] | no (.059) | 0 |
| zs_massive (leak-free) | .481 [.430, .530] | .832 [.808, .857] | .051 [.041, .073] | no (.040) | .798 / .080 |
| tickets | .629 [.583, .680] | .712 [.679, .746] | .033 [.028, .080] | yes (.061) | 0 |
| tickets_ood | .715 [.650, .787] | .683 [.639, .726] | .050 [.039, .099] | yes (.072) | 0 |
| zs_wide (147 unseen workflows; micro, all sources) | 1.196 [1.176, 1.220] | .422 [.405, .439] | .121 [.108, .137] | no (.027) | 0 |
zs_wide final macro_real (group-macro accuracy over the real sources SNI, SGD and ABCD, 105 workflows; t interval over
workflows within sources) **.419 [.379, .458]**: the unseen-workflow number. GLM-synthetic workflows separately: .358
[.323, .394] (42 workflows). Mean NLL over td, zs_td, zs_massive and tickets: .7297. Seed variability of this code and
data (6 fits, development suites): between-seed SD .0105 in that metric, .019 in zs_wide macro_real, .037 in zs_td micro
accuracy; differences of that size between fits are not evidence of a change.
Against the rivals' sealed runs (paired CIs in `benchmarks/results/README.md`), ej 0.0.1 has lower micro accuracy than the
144M-0.8B rivals and the hosted Jev on zs_td and zs_massive (except Julia-1 on zs_massive) and than laya on td; no
calibration, latency or size ranking is claimed.
### Comparison with other decision models (same final suites)
Micro accuracy / mean gold-label NLL on each final suite. Rival rows are their sealed runs on the same suites (2026-10-07);
the full table with ECE and paired 95% CIs of every difference is `benchmarks/results/README.md`.
| Model | Size | td (seen workflows) | zs_td (unseen workflow) | zs_massive | tickets ¹ | tickets_ood ¹ |
|---|---|---|---|---|---|---|
| **ej 0.0.1** | 11.4 MB file | .721 / .663 | .422 / 1.145 | .832 / .481 | .712 / .629 | **.683** / .715 |
| Jev 1.13.0 (hosted API) | closed | .737 / .624 | .742 / .750 ² | .957 / .234 ² | **.730** / 1.536 | .626 / 2.457 |
| laya | 421M params | **.766** / .690 | **.766** / .758 ³ | .866 / .501 | .669 / .841 | .617 / .965 |
| kev-0.8b | 0.8B | .415 / 1.085 | .520 / 1.035 | .895 / .381 | .679 / .722 | .608 / .856 |
| OpenThai-SystemOne | 0.75B | .515 / 1.293 | .632 / .901 | **.971** / .120 ³ | .659 / 1.070 | .617 / 1.145 |
| Julia-1 (llama.cpp BF16) | 144M | .733 / 1.814 | .706 / 2.000 ³ | .792 / .730 | .513 / 1.907 | .465 / 2.300 |
| gliclass-edge v3.0 | 33M | .367 / 2.577 | .484 / 1.911 | .545 / 1.520 | .304 / 2.618 | .367 / 2.258 |

¹ In-house ticket suites: ej trained on the tickets training split (home turf for ej, zero-shot for every rival).
² Jev's training data is undisclosed; its published Typed Decisions results use the same test split.
³ Likely contaminated: laya was trained on the Typed Decisions train split including `security_incidents` (so zs_td is
not zero-shot for laya); Julia-1 likely saw all TD workflows; OpenThai trains on MASSIVE intents.
Paired differences (ej minus rival, micro accuracy, 95% CI): vs Jev td −.015 [−.043, +.011], zs_td −.320, zs_massive
−.125; vs laya td −.045 [−.065, −.023], zs_td −.344; vs gliclass-edge td +.355, zs_td −.062 [−.124, −.004], zs_massive
+.287. On the 147-workflow unseen suite (zs_wide) no rival was run; ej's group-macro over real sources is .419.
Latencies of the rivals were measured under a different protocol and are not compared here.
## 6. Limitations
- **Unseen workflows: low accuracy.** Expect weak decisions on a new workflow until you measure it on labelled cases of
  your own. A few labelled records help mainly by teaching the label prior (`docs/adaptation.md`).
- **Fit-to-fit noise.** Two fits of the same code that differ only in their random seed differ by about .03 zs_td micro
  accuracy (SD of a difference); single-fit differences of that size are not evidence (§4).
- **Option-text tilt**: on new workflows the option texts alone still tilt the predictions.
- **More data did not simply help**: adding broad text and dialogue data made td, tickets and zs_massive worse (zs_td
  unchanged within noise), and synthetic workflow data did not improve unseen-workflow accuracy; neither is in the
  training data.
- **English only** (base encoder and all data). **Synthetic in-domain data**: the tickets come from one model family
  (Claude Haiku) and have no human labels.
- **Certified automation is per suite**: a threshold certified on one workflow can fail on another (a td-certified
  threshold had error .771 on zs_wide). Certify on labelled rows of the workflow you automate.
- **Python runtime only** (torch + transformers, CPU). Latency is the Python reference on a 4-core CPU, not a phone.
- **Float noise**: batch composition, chunk size and thread count move probabilities by at most about 6e-7 (measured).
- **One download**: `ej.load('5ak3t/ej', revision='v0.0.1')` fetches only `model.ejpack`, which holds the tokenizer and
  the encoder config; a local pack needs no network. A weights directory (the secondary format) instead fetches the base
  model from the Hugging Face Hub at revision `ffb93f3b` on first use.
- **Reproducibility**: a release is tied to its runtime (`ej/_runtime`, sha256 manifest checked at every load); a known
  pack is checked against its content digest in `ej.integrity.KNOWN_PACKS` (and, when downloaded, its file sha256 in
  `ej.integrity.KNOWN_PACK_FILES`).

## 7. Intended use

- On-device triage and routing decisions (support tickets, intents, workflow checks) where a probability distribution is
  needed per question and uncertain cases are **escalated** to a slower System 2 (an on-device LLM or a cloud model).
- Ranking, gating and abstention using the returned probabilities, with thresholds certified on your own labelled data.

## 8. Out-of-scope use

- Sole decision-maker for consequential decisions about people (credit, employment, health, legal, safety) without human
  review.
- Non-English text; generative tasks; free-text answers.
- New structured workflows without first measuring quality on labelled examples of that workflow (§6).
- Security-incident triage as a zero-shot claim (that workflow drove model selection).
- Transductive use across workflows, or on option sets that change between records: `adapt` / `observe` are for one
  workflow with a fixed option set (§1).

## 9. Licence and attribution

The weights are licensed under **Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)**. Adaptations
must be shared under CC BY-SA 4.0 or a compatible licence. Suggested attribution:

> ej 0.0.1 by Saket Bhushan, licensed under CC BY-SA 4.0
> (https://creativecommons.org/licenses/by-sa/4.0/). Derived from intfloat/e5-small-v2 (MIT; Wang et al., arXiv:2212.03533)
> and trained on Typed Decisions (Apache-2.0), Banking77 (CC BY 4.0), CLINC150 (CC BY 3.0), GoEmotions (Apache-2.0) and an
> in-house support-ticket corpus written with Claude Haiku (not released), with distillation from
> cross-encoder/nli-deberta-v3-xsmall (Apache-2.0). Full notices, creators and open questions: NOTICE.

	---
	license: cc-by-sa-4.0
	base_model: intfloat/e5-small-v2
	library_name: ej
	language:
	- en
	pipeline_tag: text-classification
	tags:
	- text-classification
	- zero-shot-classification
	- calibration
	- typed-decisions
	---

	# ej model card (0.0.1)

	ej 0.0.1 is a small calibrated decision model; technical report forthcoming.

	> Weights: https://huggingface.co/5ak3t/ej, revision `v0.0.1`, one file `model.ejpack` (11,384,312 bytes, SHA-256
	> `e990e1846cba43f8405a969c606057f2fd6e2076595f4a34d202e8fc531891b0`). Code, training and evaluation:
	> https://github.com/5ak3t/ej (file paths below refer to that repository). Release notes: `docs/releases/0.0.1.md`.

	\| Field \| Value \|
	\|---\|---\|
	\| Model \| ej 0.0.1 (2026-10-08); the default fit seed, fixed before fitting, not a selected seed \|
	\| State key \| `3b3e66d28fb423f98b734bd0c1d324cdc92bb8e10eda0e7ddf9a8e54de3f3fe2` (recorded in the pack header; reproduced by `scripts/state_key.py --git <model-code repo> f46cf7c <training pool> --ck-dir lowbit-b3b010513f948ceb`) \|
	\| Code \| runtime: `ej/_runtime` of the GitHub repository (39 sha256-pinned modules; `ej.integrity.RELEASE_PROVENANCE`); fitted with model-code commit `f46cf7c` of the maintainers' development repository (not public) and encoder checkpoint `lowbit-b3b010513f948ceb`; the same recipe is `python -m ej.train` \|
	\| Size \| counted 10.87 MiB (11.40 MB; a bit-level bound on the stored parameters); on disk 11.4 MB: one file `model.ejpack` of 11,384,312 bytes, the whole download (no base model is fetched); resident: peak live model tensors while predicting 128 MB by default, 35 MB with `low_memory=True`; process RSS during predict about 558 / 455 MB, of which about 204 MB is `import torch` (README "Memory and latency") \|
	\| Latency (CPU, Python reference, 1 thread, one record per call) \| packed model, first 50 td development records, warm: 214 ms per record (317 ms with `low_memory=True`); cold 1,061 / 1,385 ms (2026-10-09, Intel Xeon @ 2.80GHz, 4 logical cores, contended). Benchmark protocol on the td development suite (weights-directory format, 2026-10-08): warm 356.6 ms mean / 363.1 ms median, cold 1,922.0 / 1,413.7 ms. The two measurements are not comparable (README "Latency") \|
	\| Language \| English only (§6) \|
	\| Licence \| weights CC BY-SA 4.0; code Apache-2.0 \|
	\| Base model \| `intfloat/e5-small-v2` (MIT), revision `ffb93f3bd4047442299a41ebb6fa998a38507c52` \|
	\| Weights / code \| https://huggingface.co/5ak3t/ej (revision `v0.0.1`, file `model.ejpack`) / https://github.com/5ak3t/ej \|

	## 1. What the model does

	A System-1 decision model for devices: given a `state` (text, or a JSON object as text) and typed questions, it returns
	one probability distribution per question in one pass, without decoding tokens. Usage: Quickstart below and the
	repository README.

	\| Question type \| Input \| Output \|
	\|---\|---\|---\|
	\| `choice` \| instructions + a runtime list of option texts \| a probability per option \|
	\| `noul` \| a yes/no statement \| P(false), P(true) \|
	\| `score` \| instructions + an ordered list of level texts \| a distribution over the levels \|

	- No LLM at inference; teachers are used at fit time only and their weights do not ship.
	- Calibration is fitted, then checked per suite: whether the result is calibrated is measured on each suite against a
	perfect-calibration ECE floor (§5).
	- Record independence. Zero-shot `predict` is record-independent: each record's output depends only on that record (up
	to the float noise of batching). `adapt` (and `AdaptedModel.observe`) is an opt-in, per-workflow transductive mode: it
	pools option statistics across that workflow's labelled records, so use it only for one workflow with a fixed option set
	(the same question ids and option keys).
	- Option texts are read as text, so new label spaces can be asked zero-shot; quality on unseen workflows is limited (§6).

	## 2. Quickstart

	Architecture and method: technical report forthcoming.

	```bash
	git clone https://github.com/5ak3t/ej && cd ej
	pip install -e .
	```

	```python
	import ej

	model = ej.load('5ak3t/ej', revision='v0.0.1') # downloads model.ejpack only, checks its sha256, nothing else is fetched
	(probs,) = model.predict([ej.EXAMPLE_RECORD]) # {qid: [p for each option, in option order]}
	```

	`ej.load` also takes a local `model.ejpack` (or a directory holding it). `python examples/quickstart.py --weights 5ak3t/ej
	--revision v0.0.1` prints the distributions for `ej.EXAMPLE_RECORD`. Input and output format: the repository README.

	## 3. Training data

	One fit on one training pool (9,719 records, 7 source groups; sha256
	`ce1c1a6fecfd62a90317f6efc4f90fd5f9261becb7081fd00235a6f4d5ee9dbe`). Details and counts: `docs/data-card.md`.

	\| Source (groups) \| Records \| Licence \|
	\|---\|---\|---\|
	\| Typed Decisions, workflows `agent_trace_observability`, `customer_service`, `invoice_processing` (3) \| 736 \| Apache-2.0 \|
	\| Support tickets written by Claude Haiku for this project, train + calibration splits (1) \| 780 \| in-house; not released \|
	\| Banking77 / CLINC150 `plus` / GoEmotions (3) \| 3,000 / 3,000 / 2,203 \| CC BY 4.0 / CC BY 3.0 / Apache-2.0 \|
	\| Total \| 9,719 records, 7 groups \| \|

	Amazon counterfactual is not used: its upstream licence is CC BY-NC 4.0 (one mirror declares CC BY 4.0).

	Disclosure: Claude Haiku. The in-domain support tickets were written by Claude Haiku (labels by construction from the
	generation spec, no human review). They are part of training; the corpus itself is not released, so the training pool
	cannot be rebuilt from public data alone. Open licence questions on this and other inputs: `NOTICE` (Q1-Q4).
	Not in the training data: Typed Decisions `security_incidents` (the `zs_td` suite), MASSIVE and the zs_wide workflows
	(evaluation only).

	## 4. Evaluation

	- Suites (`benchmarks/README.md`; never trained on):
	- `td`: Typed Decisions test split, the 3 training workflows.
	- `zs_td`: zs_td is ONE held-out workflow (security_incidents; dev 300 and final 100 records of the same workflow).
	The model code was kept on its dev accuracy after about 55 logged comparisons on it, so final zs_td estimates accuracy on
	further records of a workflow the model was selected on: it is neither unbiased nor unseen-workflow evidence. Its
	fit-to-fit SD (same code, other RNG) is ~.022 accuracy per fit (SD of a difference .031, 4 pairs), twice its
	record-cluster SE (.011); one workflow gives no between-workflow variance. Unseen-workflow evidence = zs_wide final
	macro_real.
	- `zs_massive`: MASSIVE (en) intents with leak-free option sets. MASSIVE was never trained on, but its label space is
	not new: about 22% of its intent names (13 of 59 development option texts: 1 identical, 12 near) overlap CLINC150 /
	Banking77 option texts in the pool, and 4 final records are exact duplicates of pool records (removed at the next
	benchmark run).
	- `zs_wide`: workflows never trained on: dev 154 workflows (111 from public sources: SNI tasks, SGD services, ABCD; 43
	GLM-synthetic), final 147. The split is by task, not by dataset family: 53 of the 147 final workflows share a family
	with a dev workflow; pool-source tasks were dropped from the final set.
	- `tickets` (in-domain, private) and `tickets_ood` (held-out writing styles of the same generator: a style shift).
	- Metrics: per-suite mean NLL of the gold option (headline: the mean over td, zs_td, zs_massive, tickets), micro
	accuracy (zs_wide: group-macro over the real sources), ECE over 15 bins with its perfect-calibration floor, certified
	automation (risk .10, δ .10; this suite only). Each cell has a 95% record-cluster CI. Scoring: `ej.eval`
	(`benchmarks/scoring.py` for rivals).
	- Selection. Development numbers are selected: more than 55 logged comparisons were made on the development suites, so
	their numbers are optimistic. The final suites are read once per release; zs_td final is not unseen-workflow evidence
	(above).
	- Leak-free option sets: for sampled option sets every option is equally likely to be the gold, so option frequency
	reveals nothing (a frequency rule reached .631 micro accuracy against chance .240 on a non-leak-free MASSIVE suite).
	- Rivals and contamination flags: `benchmarks/METHOD.md`.

	## 5. Results

	ej 0.0.1 (final suites, scored once, 2026-10-08: `benchmarks/results/README.md`). 95% record-cluster CIs; calibrated
	= observed ECE15 at or below the 95th percentile of a perfectly calibrated model's ECE15 on that suite.

	\| Suite \| NLL [95% CI] \| Micro accuracy [95% CI] \| ECE15 [95% CI] \| Calibrated on this suite \| Certified automation, this suite only (coverage / error) \|
	\|---\|---\|---\|---\|---\|---\|
	\| td (seen workflows) \| .663 [.613, .716] \| .721 [.695, .746] \| .071 [.050, .094] \| no (floor q95 .034) \| .300 / .056 \|
	\| zs_td (`security_incidents`, selected on) \| 1.145 [1.113, 1.180] \| .422 [.388, .452] \| .094 [.068, .135] \| no (.059) \| 0 \|
	\| zs_massive (leak-free) \| .481 [.430, .530] \| .832 [.808, .857] \| .051 [.041, .073] \| no (.040) \| .798 / .080 \|
	\| tickets \| .629 [.583, .680] \| .712 [.679, .746] \| .033 [.028, .080] \| yes (.061) \| 0 \|
	\| tickets_ood \| .715 [.650, .787] \| .683 [.639, .726] \| .050 [.039, .099] \| yes (.072) \| 0 \|
	\| zs_wide (147 unseen workflows; micro, all sources) \| 1.196 [1.176, 1.220] \| .422 [.405, .439] \| .121 [.108, .137] \| no (.027) \| 0 \|

	zs_wide final macro_real (group-macro accuracy over the real sources SNI, SGD and ABCD, 105 workflows; t interval over
	workflows within sources) .419 [.379, .458]: the unseen-workflow number. GLM-synthetic workflows separately: .358
	[.323, .394] (42 workflows). Mean NLL over td, zs_td, zs_massive and tickets: .7297. Seed variability of this code and
	data (6 fits, development suites): between-seed SD .0105 in that metric, .019 in zs_wide macro_real, .037 in zs_td micro
	accuracy; differences of that size between fits are not evidence of a change.
	Against the rivals' sealed runs (paired CIs in `benchmarks/results/README.md`), ej 0.0.1 has lower micro accuracy than the
	144M-0.8B rivals and the hosted Jev on zs_td and zs_massive (except Julia-1 on zs_massive) and than laya on td; no
	calibration, latency or size ranking is claimed.

	### Comparison with other decision models (same final suites)

	Micro accuracy / mean gold-label NLL on each final suite. Rival rows are their sealed runs on the same suites (2026-10-07);
	the full table with ECE and paired 95% CIs of every difference is `benchmarks/results/README.md`.

	\| Model \| Size \| td (seen workflows) \| zs_td (unseen workflow) \| zs_massive \| tickets ¹ \| tickets_ood ¹ \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| ej 0.0.1 \| 11.4 MB file \| .721 / .663 \| .422 / 1.145 \| .832 / .481 \| .712 / .629 \| .683 / .715 \|
	\| Jev 1.13.0 (hosted API) \| closed \| .737 / .624 \| .742 / .750 ² \| .957 / .234 ² \| .730 / 1.536 \| .626 / 2.457 \|
	\| laya \| 421M params \| .766 / .690 \| .766 / .758 ³ \| .866 / .501 \| .669 / .841 \| .617 / .965 \|
	\| kev-0.8b \| 0.8B \| .415 / 1.085 \| .520 / 1.035 \| .895 / .381 \| .679 / .722 \| .608 / .856 \|
	\| OpenThai-SystemOne \| 0.75B \| .515 / 1.293 \| .632 / .901 \| .971 / .120 ³ \| .659 / 1.070 \| .617 / 1.145 \|
	\| Julia-1 (llama.cpp BF16) \| 144M \| .733 / 1.814 \| .706 / 2.000 ³ \| .792 / .730 \| .513 / 1.907 \| .465 / 2.300 \|
	\| gliclass-edge v3.0 \| 33M \| .367 / 2.577 \| .484 / 1.911 \| .545 / 1.520 \| .304 / 2.618 \| .367 / 2.258 \|

	¹ In-house ticket suites: ej trained on the tickets training split (home turf for ej, zero-shot for every rival).
	² Jev's training data is undisclosed; its published Typed Decisions results use the same test split.
	³ Likely contaminated: laya was trained on the Typed Decisions train split including `security_incidents` (so zs_td is
	not zero-shot for laya); Julia-1 likely saw all TD workflows; OpenThai trains on MASSIVE intents.

	Paired differences (ej minus rival, micro accuracy, 95% CI): vs Jev td −.015 [−.043, +.011], zs_td −.320, zs_massive
	−.125; vs laya td −.045 [−.065, −.023], zs_td −.344; vs gliclass-edge td +.355, zs_td −.062 [−.124, −.004], zs_massive
	+.287. On the 147-workflow unseen suite (zs_wide) no rival was run; ej's group-macro over real sources is .419.
	Latencies of the rivals were measured under a different protocol and are not compared here.

	## 6. Limitations

	- Unseen workflows: low accuracy. Expect weak decisions on a new workflow until you measure it on labelled cases of
	your own. A few labelled records help mainly by teaching the label prior (`docs/adaptation.md`).
	- Fit-to-fit noise. Two fits of the same code that differ only in their random seed differ by about .03 zs_td micro
	accuracy (SD of a difference); single-fit differences of that size are not evidence (§4).
	- Option-text tilt: on new workflows the option texts alone still tilt the predictions.
	- More data did not simply help: adding broad text and dialogue data made td, tickets and zs_massive worse (zs_td
	unchanged within noise), and synthetic workflow data did not improve unseen-workflow accuracy; neither is in the
	training data.
	- English only (base encoder and all data). Synthetic in-domain data: the tickets come from one model family
	(Claude Haiku) and have no human labels.
	- Certified automation is per suite: a threshold certified on one workflow can fail on another (a td-certified
	threshold had error .771 on zs_wide). Certify on labelled rows of the workflow you automate.
	- Python runtime only (torch + transformers, CPU). Latency is the Python reference on a 4-core CPU, not a phone.
	- Float noise: batch composition, chunk size and thread count move probabilities by at most about 6e-7 (measured).
	- One download: `ej.load('5ak3t/ej', revision='v0.0.1')` fetches only `model.ejpack`, which holds the tokenizer and
	the encoder config; a local pack needs no network. A weights directory (the secondary format) instead fetches the base
	model from the Hugging Face Hub at revision `ffb93f3b` on first use.
	- Reproducibility: a release is tied to its runtime (`ej/_runtime`, sha256 manifest checked at every load); a known
	pack is checked against its content digest in `ej.integrity.KNOWN_PACKS` (and, when downloaded, its file sha256 in
	`ej.integrity.KNOWN_PACK_FILES`).

	## 7. Intended use

	- On-device triage and routing decisions (support tickets, intents, workflow checks) where a probability distribution is
	needed per question and uncertain cases are escalated to a slower System 2 (an on-device LLM or a cloud model).
	- Ranking, gating and abstention using the returned probabilities, with thresholds certified on your own labelled data.

	## 8. Out-of-scope use

	- Sole decision-maker for consequential decisions about people (credit, employment, health, legal, safety) without human
	review.
	- Non-English text; generative tasks; free-text answers.
	- New structured workflows without first measuring quality on labelled examples of that workflow (§6).
	- Security-incident triage as a zero-shot claim (that workflow drove model selection).
	- Transductive use across workflows, or on option sets that change between records: `adapt` / `observe` are for one
	workflow with a fixed option set (§1).

	## 9. Licence and attribution

	The weights are licensed under Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0). Adaptations
	must be shared under CC BY-SA 4.0 or a compatible licence. Suggested attribution:

	> ej 0.0.1 by Saket Bhushan, licensed under CC BY-SA 4.0
	> (https://creativecommons.org/licenses/by-sa/4.0/). Derived from intfloat/e5-small-v2 (MIT; Wang et al., arXiv:2212.03533)
	> and trained on Typed Decisions (Apache-2.0), Banking77 (CC BY 4.0), CLINC150 (CC BY 3.0), GoEmotions (Apache-2.0) and an
	> in-house support-ticket corpus written with Claude Haiku (not released), with distillation from
	> cross-encoder/nli-deberta-v3-xsmall (Apache-2.0). Full notices, creators and open questions: NOTICE.