Add Jev (hosted text-only) row to doc-split leaderboard
Browse files- index.html +9 -1
index.html
CHANGED
|
@@ -94,7 +94,7 @@
|
|
| 94 |
<p class="lede">One boundary metric, one harness, every contender re-measured: our flagship and open-weight
|
| 95 |
models against cloud VLMs and the field's self-declared numbers. Cells show boundary <b>F1</b>; the small
|
| 96 |
figure is chance-corrected <b>κ</b>.</p>
|
| 97 |
-
<div class="meta">updated 2026-
|
| 98 |
</header>
|
| 99 |
|
| 100 |
<section class="findings">
|
|
@@ -167,6 +167,13 @@
|
|
| 167 |
<td class="na">—</td><td class="na">—</td><td class="na">—</td>
|
| 168 |
</tr>
|
| 169 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 170 |
<tr class="grouprow"><td colspan="7">Research — self-declared (their metric / in-domain)</td></tr>
|
| 171 |
<tr>
|
| 172 |
<td class="model"><div class="modelname">OpenPSS SHORT-specialist</div><div class="modelsub">BERT-EffNet ensemble · one of two models</div><span class="chip pubd">Published</span></td>
|
|
@@ -203,6 +210,7 @@
|
|
| 203 |
<p><b>One balanced model vs two specialists.</b> No single OpenPSS model wins both slices — their short-specialist craters on long (0.50), their long-specialist on short (0.62). the v4 flagship does short + long with one model, and its long (0.891) tops even their long-specialist (0.83).</p>
|
| 204 |
<p><b>Cloud VLMs can't ingest long streams.</b> Per-request image caps (Anthropic ~100 / OpenAI ~200 / Gemini ~500) force predict-none on streams over the cap, which dominates OpenPSS-long → best cloud 0.244 vs flagship 0.891, at ~20× the cost per page.</p>
|
| 205 |
<p><b>* bert-pss</b> self-declares ~0.915 accuracy / 0.825 κ on Tobacco800, but collapses to a single class when actually run (κ ≈ 0) — the only released PSS specialist is non-functional off its exact serving harness.</p>
|
|
|
|
| 206 |
<p><b>Metric.</b> Ours & cloud: micro boundary-F1 + κ over internal pages, identical harness. OpenPSS rows: their published per-stream page-F1 (ballpark-comparable). Tobacco incumbents report accuracy — compared via κ. AI-Lab-Splitter omitted: data gated, absolute F1 paywalled.</p>
|
| 207 |
</div>
|
| 208 |
</div>
|
|
|
|
| 94 |
<p class="lede">One boundary metric, one harness, every contender re-measured: our flagship and open-weight
|
| 95 |
models against cloud VLMs and the field's self-declared numbers. Cells show boundary <b>F1</b>; the small
|
| 96 |
figure is chance-corrected <b>κ</b>.</p>
|
| 97 |
+
<div class="meta">updated 2026-09-23 · metric: boundary page-F1 (page 0 forced) + Cohen's κ</div>
|
| 98 |
</header>
|
| 99 |
|
| 100 |
<section class="findings">
|
|
|
|
| 167 |
<td class="na">—</td><td class="na">—</td><td class="na">—</td>
|
| 168 |
</tr>
|
| 169 |
|
| 170 |
+
<tr class="grouprow"><td colspan="7">Cloud · text-only (hosted System-One, reads OCR)</td></tr>
|
| 171 |
+
<tr>
|
| 172 |
+
<td class="model"><div class="modelname">Jev (TypeSafe)</div><div class="modelsub">hosted text-only System-One · reads page OCR/text (not a VLM) · whole-stream</div><span class="chip cloud">Cloud · text</span></td>
|
| 173 |
+
<td class="g3"><span class="f1">0.637</span><span class="kap">.29</span></td>
|
| 174 |
+
<td class="na">—</td><td class="na">—</td><td class="na">—</td><td class="na">—</td><td class="na">—</td>
|
| 175 |
+
</tr>
|
| 176 |
+
|
| 177 |
<tr class="grouprow"><td colspan="7">Research — self-declared (their metric / in-domain)</td></tr>
|
| 178 |
<tr>
|
| 179 |
<td class="model"><div class="modelname">OpenPSS SHORT-specialist</div><div class="modelsub">BERT-EffNet ensemble · one of two models</div><span class="chip pubd">Published</span></td>
|
|
|
|
| 210 |
<p><b>One balanced model vs two specialists.</b> No single OpenPSS model wins both slices — their short-specialist craters on long (0.50), their long-specialist on short (0.62). the v4 flagship does short + long with one model, and its long (0.891) tops even their long-specialist (0.83).</p>
|
| 211 |
<p><b>Cloud VLMs can't ingest long streams.</b> Per-request image caps (Anthropic ~100 / OpenAI ~200 / Gemini ~500) force predict-none on streams over the cap, which dominates OpenPSS-long → best cloud 0.244 vs flagship 0.891, at ~20× the cost per page.</p>
|
| 212 |
<p><b>* bert-pss</b> self-declares ~0.915 accuracy / 0.825 κ on Tobacco800, but collapses to a single class when actually run (κ ≈ 0) — the only released PSS specialist is non-functional off its exact serving harness.</p>
|
| 213 |
+
<p><b>Jev (TypeSafe), text-only.</b> The hosted System-One model reads page OCR only (no image). On our-200 it scores 0.637 F1 / .29 κ — high precision, low recall (under-splits textually-similar form/receipt pages), and giving it the whole document instead of adjacent pages does not help (pairwise 0.650). Confirms split is image-native: text alone tops out ~0.64 F1 here, far below the image models (ours 0.944).</p>
|
| 214 |
<p><b>Metric.</b> Ours & cloud: micro boundary-F1 + κ over internal pages, identical harness. OpenPSS rows: their published per-stream page-F1 (ballpark-comparable). Tobacco incumbents report accuracy — compared via κ. AI-Lab-Splitter omitted: data gated, absolute F1 paywalled.</p>
|
| 215 |
</div>
|
| 216 |
</div>
|