hung-k-nguyen commited on
Commit
77589d7
·
verified ·
1 Parent(s): 0c5549b

Add Jev (hosted text-only) row to doc-split leaderboard

Browse files
Files changed (1) hide show
  1. index.html +9 -1
index.html CHANGED
@@ -94,7 +94,7 @@
94
  <p class="lede">One boundary metric, one harness, every contender re-measured: our flagship and open-weight
95
  models against cloud VLMs and the field's self-declared numbers. Cells show boundary <b>F1</b>; the small
96
  figure is chance-corrected <b>κ</b>.</p>
97
- <div class="meta">updated 2026-08-11 · metric: boundary page-F1 (page 0 forced) + Cohen's κ</div>
98
  </header>
99
 
100
  <section class="findings">
@@ -167,6 +167,13 @@
167
  <td class="na">—</td><td class="na">—</td><td class="na">—</td>
168
  </tr>
169
 
 
 
 
 
 
 
 
170
  <tr class="grouprow"><td colspan="7">Research — self-declared (their metric / in-domain)</td></tr>
171
  <tr>
172
  <td class="model"><div class="modelname">OpenPSS SHORT-specialist</div><div class="modelsub">BERT-EffNet ensemble · one of two models</div><span class="chip pubd">Published</span></td>
@@ -203,6 +210,7 @@
203
  <p><b>One balanced model vs two specialists.</b> No single OpenPSS model wins both slices — their short-specialist craters on long (0.50), their long-specialist on short (0.62). the v4 flagship does short + long with one model, and its long (0.891) tops even their long-specialist (0.83).</p>
204
  <p><b>Cloud VLMs can't ingest long streams.</b> Per-request image caps (Anthropic ~100 / OpenAI ~200 / Gemini ~500) force predict-none on streams over the cap, which dominates OpenPSS-long → best cloud 0.244 vs flagship 0.891, at ~20× the cost per page.</p>
205
  <p><b>* bert-pss</b> self-declares ~0.915 accuracy / 0.825 κ on Tobacco800, but collapses to a single class when actually run (κ ≈ 0) — the only released PSS specialist is non-functional off its exact serving harness.</p>
 
206
  <p><b>Metric.</b> Ours &amp; cloud: micro boundary-F1 + κ over internal pages, identical harness. OpenPSS rows: their published per-stream page-F1 (ballpark-comparable). Tobacco incumbents report accuracy — compared via κ. AI-Lab-Splitter omitted: data gated, absolute F1 paywalled.</p>
207
  </div>
208
  </div>
 
94
  <p class="lede">One boundary metric, one harness, every contender re-measured: our flagship and open-weight
95
  models against cloud VLMs and the field's self-declared numbers. Cells show boundary <b>F1</b>; the small
96
  figure is chance-corrected <b>κ</b>.</p>
97
+ <div class="meta">updated 2026-09-23 · metric: boundary page-F1 (page 0 forced) + Cohen's κ</div>
98
  </header>
99
 
100
  <section class="findings">
 
167
  <td class="na">—</td><td class="na">—</td><td class="na">—</td>
168
  </tr>
169
 
170
+ <tr class="grouprow"><td colspan="7">Cloud · text-only (hosted System-One, reads OCR)</td></tr>
171
+ <tr>
172
+ <td class="model"><div class="modelname">Jev (TypeSafe)</div><div class="modelsub">hosted text-only System-One · reads page OCR/text (not a VLM) · whole-stream</div><span class="chip cloud">Cloud · text</span></td>
173
+ <td class="g3"><span class="f1">0.637</span><span class="kap">.29</span></td>
174
+ <td class="na">—</td><td class="na">—</td><td class="na">—</td><td class="na">—</td><td class="na">—</td>
175
+ </tr>
176
+
177
  <tr class="grouprow"><td colspan="7">Research — self-declared (their metric / in-domain)</td></tr>
178
  <tr>
179
  <td class="model"><div class="modelname">OpenPSS SHORT-specialist</div><div class="modelsub">BERT-EffNet ensemble · one of two models</div><span class="chip pubd">Published</span></td>
 
210
  <p><b>One balanced model vs two specialists.</b> No single OpenPSS model wins both slices — their short-specialist craters on long (0.50), their long-specialist on short (0.62). the v4 flagship does short + long with one model, and its long (0.891) tops even their long-specialist (0.83).</p>
211
  <p><b>Cloud VLMs can't ingest long streams.</b> Per-request image caps (Anthropic ~100 / OpenAI ~200 / Gemini ~500) force predict-none on streams over the cap, which dominates OpenPSS-long → best cloud 0.244 vs flagship 0.891, at ~20× the cost per page.</p>
212
  <p><b>* bert-pss</b> self-declares ~0.915 accuracy / 0.825 κ on Tobacco800, but collapses to a single class when actually run (κ ≈ 0) — the only released PSS specialist is non-functional off its exact serving harness.</p>
213
+ <p><b>Jev (TypeSafe), text-only.</b> The hosted System-One model reads page OCR only (no image). On our-200 it scores 0.637 F1 / .29 κ — high precision, low recall (under-splits textually-similar form/receipt pages), and giving it the whole document instead of adjacent pages does not help (pairwise 0.650). Confirms split is image-native: text alone tops out ~0.64 F1 here, far below the image models (ours 0.944).</p>
214
  <p><b>Metric.</b> Ours &amp; cloud: micro boundary-F1 + κ over internal pages, identical harness. OpenPSS rows: their published per-stream page-F1 (ballpark-comparable). Tobacco incumbents report accuracy — compared via κ. AI-Lab-Splitter omitted: data gated, absolute F1 paywalled.</p>
215
  </div>
216
  </div>