Model card: add comparison with other decision models (same final suites)
Browse files- README.md +25 -0
- SHA256SUMS +1 -1
README.md
CHANGED
|
@@ -144,6 +144,31 @@ Against the rivals' sealed runs (paired CIs in `benchmarks/results/README.md`),
|
|
| 144 |
144M-0.8B rivals and the hosted Jev on zs_td and zs_massive (except Julia-1 on zs_massive) and than laya on td; no
|
| 145 |
calibration, latency or size ranking is claimed.
|
| 146 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 147 |
## 6. Limitations
|
| 148 |
|
| 149 |
- **Unseen workflows: low accuracy.** Expect weak decisions on a new workflow until you measure it on labelled cases of
|
|
|
|
| 144 |
144M-0.8B rivals and the hosted Jev on zs_td and zs_massive (except Julia-1 on zs_massive) and than laya on td; no
|
| 145 |
calibration, latency or size ranking is claimed.
|
| 146 |
|
| 147 |
+
### Comparison with other decision models (same final suites)
|
| 148 |
+
|
| 149 |
+
Micro accuracy / mean gold-label NLL on each final suite. Rival rows are their sealed runs on the same suites (2026-10-07);
|
| 150 |
+
the full table with ECE and paired 95% CIs of every difference is `benchmarks/results/README.md`.
|
| 151 |
+
|
| 152 |
+
| Model | Size | td (seen workflows) | zs_td (unseen workflow) | zs_massive | tickets ¹ | tickets_ood ¹ |
|
| 153 |
+
|---|---|---|---|---|---|---|
|
| 154 |
+
| **ej 0.0.1** | 11.4 MB file | .721 / .663 | .422 / 1.145 | .832 / .481 | .712 / .629 | **.683** / .715 |
|
| 155 |
+
| Jev 1.13.0 (hosted API) | closed | .737 / .624 | .742 / .750 ² | .957 / .234 ² | **.730** / 1.536 | .626 / 2.457 |
|
| 156 |
+
| laya | 421M params | **.766** / .690 | **.766** / .758 ³ | .866 / .501 | .669 / .841 | .617 / .965 |
|
| 157 |
+
| kev-0.8b | 0.8B | .415 / 1.085 | .520 / 1.035 | .895 / .381 | .679 / .722 | .608 / .856 |
|
| 158 |
+
| OpenThai-SystemOne | 0.75B | .515 / 1.293 | .632 / .901 | **.971** / .120 ³ | .659 / 1.070 | .617 / 1.145 |
|
| 159 |
+
| Julia-1 (llama.cpp BF16) | 144M | .733 / 1.814 | .706 / 2.000 ³ | .792 / .730 | .513 / 1.907 | .465 / 2.300 |
|
| 160 |
+
| gliclass-edge v3.0 | 33M | .367 / 2.577 | .484 / 1.911 | .545 / 1.520 | .304 / 2.618 | .367 / 2.258 |
|
| 161 |
+
|
| 162 |
+
¹ In-house ticket suites: ej trained on the tickets training split (home turf for ej, zero-shot for every rival).
|
| 163 |
+
² Jev's training data is undisclosed; its published Typed Decisions results use the same test split.
|
| 164 |
+
³ Likely contaminated: laya was trained on the Typed Decisions train split including `security_incidents` (so zs_td is
|
| 165 |
+
not zero-shot for laya); Julia-1 likely saw all TD workflows; OpenThai trains on MASSIVE intents.
|
| 166 |
+
|
| 167 |
+
Paired differences (ej minus rival, micro accuracy, 95% CI): vs Jev td −.015 [−.043, +.011], zs_td −.320, zs_massive
|
| 168 |
+
−.125; vs laya td −.045 [−.065, −.023], zs_td −.344; vs gliclass-edge td +.355, zs_td −.062 [−.124, −.004], zs_massive
|
| 169 |
+
+.287. On the 147-workflow unseen suite (zs_wide) no rival was run; ej's group-macro over real sources is .419.
|
| 170 |
+
Latencies of the rivals were measured under a different protocol and are not compared here.
|
| 171 |
+
|
| 172 |
## 6. Limitations
|
| 173 |
|
| 174 |
- **Unseen workflows: low accuracy.** Expect weak decisions on a new workflow until you measure it on labelled cases of
|
SHA256SUMS
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
28a9529c7d0bb4dc51f4bf5c116a3d16ef247a052f7591466768ddf563fd1cf5 LICENSES/CC-BY-SA-4.0.txt
|
| 2 |
1105ee255fb158be39051b6839173605855caa73cb525de0a1e9ee71cf0d75e1 LICENSES/MIT-e5-small-v2.txt
|
| 3 |
5e569d5c323141d613fe3799ef2bf58a2fb6df4e30fd7c02c36492e7c44d586c NOTICE
|
| 4 |
-
|
| 5 |
e990e1846cba43f8405a969c606057f2fd6e2076595f4a34d202e8fc531891b0 model.ejpack
|
|
|
|
| 1 |
28a9529c7d0bb4dc51f4bf5c116a3d16ef247a052f7591466768ddf563fd1cf5 LICENSES/CC-BY-SA-4.0.txt
|
| 2 |
1105ee255fb158be39051b6839173605855caa73cb525de0a1e9ee71cf0d75e1 LICENSES/MIT-e5-small-v2.txt
|
| 3 |
5e569d5c323141d613fe3799ef2bf58a2fb6df4e30fd7c02c36492e7c44d586c NOTICE
|
| 4 |
+
20dd1d1fb0ebc8ae1ac119ff0792a14568a432eb579ebd77260c3f7549e9b4b6 README.md
|
| 5 |
e990e1846cba43f8405a969c606057f2fd6e2076595f4a34d202e8fc531891b0 model.ejpack
|