5ak3t commited on
Commit
476ef0b
·
verified ·
1 Parent(s): c80222c

Model card: add comparison with other decision models (same final suites)

Browse files
Files changed (2) hide show
  1. README.md +25 -0
  2. SHA256SUMS +1 -1
README.md CHANGED
@@ -144,6 +144,31 @@ Against the rivals' sealed runs (paired CIs in `benchmarks/results/README.md`),
144
  144M-0.8B rivals and the hosted Jev on zs_td and zs_massive (except Julia-1 on zs_massive) and than laya on td; no
145
  calibration, latency or size ranking is claimed.
146
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
147
  ## 6. Limitations
148
 
149
  - **Unseen workflows: low accuracy.** Expect weak decisions on a new workflow until you measure it on labelled cases of
 
144
  144M-0.8B rivals and the hosted Jev on zs_td and zs_massive (except Julia-1 on zs_massive) and than laya on td; no
145
  calibration, latency or size ranking is claimed.
146
 
147
+ ### Comparison with other decision models (same final suites)
148
+
149
+ Micro accuracy / mean gold-label NLL on each final suite. Rival rows are their sealed runs on the same suites (2026-10-07);
150
+ the full table with ECE and paired 95% CIs of every difference is `benchmarks/results/README.md`.
151
+
152
+ | Model | Size | td (seen workflows) | zs_td (unseen workflow) | zs_massive | tickets ¹ | tickets_ood ¹ |
153
+ |---|---|---|---|---|---|---|
154
+ | **ej 0.0.1** | 11.4 MB file | .721 / .663 | .422 / 1.145 | .832 / .481 | .712 / .629 | **.683** / .715 |
155
+ | Jev 1.13.0 (hosted API) | closed | .737 / .624 | .742 / .750 ² | .957 / .234 ² | **.730** / 1.536 | .626 / 2.457 |
156
+ | laya | 421M params | **.766** / .690 | **.766** / .758 ³ | .866 / .501 | .669 / .841 | .617 / .965 |
157
+ | kev-0.8b | 0.8B | .415 / 1.085 | .520 / 1.035 | .895 / .381 | .679 / .722 | .608 / .856 |
158
+ | OpenThai-SystemOne | 0.75B | .515 / 1.293 | .632 / .901 | **.971** / .120 ³ | .659 / 1.070 | .617 / 1.145 |
159
+ | Julia-1 (llama.cpp BF16) | 144M | .733 / 1.814 | .706 / 2.000 ³ | .792 / .730 | .513 / 1.907 | .465 / 2.300 |
160
+ | gliclass-edge v3.0 | 33M | .367 / 2.577 | .484 / 1.911 | .545 / 1.520 | .304 / 2.618 | .367 / 2.258 |
161
+
162
+ ¹ In-house ticket suites: ej trained on the tickets training split (home turf for ej, zero-shot for every rival).
163
+ ² Jev's training data is undisclosed; its published Typed Decisions results use the same test split.
164
+ ³ Likely contaminated: laya was trained on the Typed Decisions train split including `security_incidents` (so zs_td is
165
+ not zero-shot for laya); Julia-1 likely saw all TD workflows; OpenThai trains on MASSIVE intents.
166
+
167
+ Paired differences (ej minus rival, micro accuracy, 95% CI): vs Jev td −.015 [−.043, +.011], zs_td −.320, zs_massive
168
+ −.125; vs laya td −.045 [−.065, −.023], zs_td −.344; vs gliclass-edge td +.355, zs_td −.062 [−.124, −.004], zs_massive
169
+ +.287. On the 147-workflow unseen suite (zs_wide) no rival was run; ej's group-macro over real sources is .419.
170
+ Latencies of the rivals were measured under a different protocol and are not compared here.
171
+
172
  ## 6. Limitations
173
 
174
  - **Unseen workflows: low accuracy.** Expect weak decisions on a new workflow until you measure it on labelled cases of
SHA256SUMS CHANGED
@@ -1,5 +1,5 @@
1
  28a9529c7d0bb4dc51f4bf5c116a3d16ef247a052f7591466768ddf563fd1cf5 LICENSES/CC-BY-SA-4.0.txt
2
  1105ee255fb158be39051b6839173605855caa73cb525de0a1e9ee71cf0d75e1 LICENSES/MIT-e5-small-v2.txt
3
  5e569d5c323141d613fe3799ef2bf58a2fb6df4e30fd7c02c36492e7c44d586c NOTICE
4
- 90450ff6b75dabd9056be8ccf95fba3b0b05b997bc176d271537ecba94d22872 README.md
5
  e990e1846cba43f8405a969c606057f2fd6e2076595f4a34d202e8fc531891b0 model.ejpack
 
1
  28a9529c7d0bb4dc51f4bf5c116a3d16ef247a052f7591466768ddf563fd1cf5 LICENSES/CC-BY-SA-4.0.txt
2
  1105ee255fb158be39051b6839173605855caa73cb525de0a1e9ee71cf0d75e1 LICENSES/MIT-e5-small-v2.txt
3
  5e569d5c323141d613fe3799ef2bf58a2fb6df4e30fd7c02c36492e7c44d586c NOTICE
4
+ 20dd1d1fb0ebc8ae1ac119ff0792a14568a432eb579ebd77260c3f7549e9b4b6 README.md
5
  e990e1846cba43f8405a969c606057f2fd6e2076595f4a34d202e8fc531891b0 model.ejpack