| <!DOCTYPE html> |
| <html lang="en"> |
| <head> |
| <meta charset="utf-8"> |
| <meta name="viewport" content="width=device-width, initial-scale=1"> |
| <title>Parallel Constrained Decisions — static demo</title> |
| <link rel="stylesheet" href="style.css"> |
| </head> |
| <body> |
| <header class="site-header"> |
| <div class="wrap"> |
| <div class="badge">Unofficial demo · not affiliated with TypeSafe AI</div> |
| <h1>Typed decisions, one pass, no parsing</h1> |
| <p class="lede"> |
| Reproducing the <em>parallel constrained decoding</em> idea behind Jev (TypeSafe AI's |
| "System One" model) on a stock 1.5B model, on a laptop. Instead of generating JSON |
| token-by-token, all schema fields are evaluated in <strong>one batched forward pass</strong> |
| and the JSON is assembled programmatically — so it can never be malformed. |
| </p> |
| <div class="header-links"> |
| <a href="https://github.com/rorshopping/jev-on-a-laptop" target="_blank" rel="noopener">Code & benchmarks on GitHub</a> |
| <a href="https://typesafe.ai/" target="_blank" rel="noopener">What TypeSafe actually launched</a> |
| </div> |
| </div> |
| </header> |
|
|
| <main class="wrap"> |
| <section class="card"> |
| <h2>How it works</h2> |
| <pre class="diagram"> |
| [context + schema] ──► prefill (once) ──► KV cache |
| │ broadcast × N fields |
| ┌───────────┬───────────┬─────────────┴───────────┐ |
| field 1 field 2 ... field N |
| └───────────┴──── one batched forward pass ────────┘ |
| │ |
| slice logits → softmax → pick value + probability</pre> |
| <p class="muted"> |
| The pipe below is <strong>real recorded output</strong> from |
| <code id="model-name">mlx-community/Qwen2.5-1.5B-Instruct-4bit</code> on an |
| M5 MacBook Air (16 GB). This page is static: no model runs in your browser. |
| The interactive version (choose your own context, run live) is the Gradio |
| app in the <a href="https://github.com/rorshopping/jev-on-a-laptop/tree/main/hf-space" target="_blank" rel="noopener">repo's <code>hf-space/</code> folder</a>. |
| </p> |
| </section> |
|
|
| <section class="card"> |
| <h2>Presets</h2> |
| <div id="preset-tabs" class="tabs" role="tablist"></div> |
| <div id="demo" class="demo" aria-live="polite"></div> |
| </section> |
|
|
| <section class="card"> |
| <h2>Does a bigger model help? (measured)</h2> |
| <p class="muted">24 labeled cases × 3 fields = 72 decisions per model, same engine. "All fields exact" = every field of the case correct.</p> |
| <table class="bench"> |
| <thead> |
| <tr><th>Model (4-bit)</th><th>Primary field</th><th>All fields exact</th><th>Latency / case</th></tr> |
| </thead> |
| <tbody> |
| <tr><td>Qwen2.5-1.5B</td><td>58% <span class="muted">(= majority-class baseline)</span></td><td>50%</td><td>147 ms</td></tr> |
| <tr><td><strong>Qwen2.5-7B</strong></td><td class="yes"><strong>96%</strong></td><td>72%</td><td>611 ms</td></tr> |
| <tr><td><strong>Qwen3-8B</strong></td><td>92%</td><td class="yes"><strong>85%</strong></td><td>646 ms</td></tr> |
| </tbody> |
| </table> |
| <p class="muted small"> |
| Bigger is worth it — but not "biggest wins": <strong>7B</strong> for the single decision you act on, |
| <strong>8B</strong> when the whole typed payload must be right, <strong>1.5B</strong> for demos only. |
| And <span class="warn">confidence does not reliably flag errors</span> — the 7B was >90% confident on |
| 13 of its 20 wrong fields. Full breakdown in the GitHub repo. |
| </p> |
| </section> |
|
|
| <section class="card"> |
| <h2>How does this compare to Jev itself?</h2> |
| <p class="muted"> |
| We ran <strong>all public example cases of TypeSafe's four workflows</strong> (20 cases, 373 question-pairs) |
| and scored everyone against TypeSafe's own reference — the consensus of GPT-6 Astra and Fable 5.1. |
| Strict like-for-like, on the 343 pairs every model answered: |
| </p> |
| <table class="bench"> |
| <thead> |
| <tr><th>Model</th><th>Agreement</th><th>Pairs</th></tr> |
| </thead> |
| <tbody> |
| <tr><td>Opus (published)</td><td>89.8%</td><td>308/343</td></tr> |
| <tr><td>DeepSeek v4.1 Flash (max)</td><td>89.5%</td><td>307/343</td></tr> |
| <tr><td>Sol (published)</td><td>89.2%</td><td>306/343</td></tr> |
| <tr><td><strong>Jev / TypeSafe (published)</strong></td><td><strong>86.6%</strong></td><td><strong>297/343</strong></td></tr> |
| <tr><td><strong>local Qwen2.5-7B (free, this laptop)</strong></td><td><strong>73.8%</strong></td><td><strong>253/343</strong></td></tr> |
| <tr><td>local Qwen3-8B (3 of 4 workflows)</td><td>71.2%</td><td>114/160</td></tr> |
| </tbody> |
| </table> |
| <p class="muted small"> |
| <strong>The honest result: the free local model is ~13 points behind Jev on Jev's own benchmark.</strong> |
| Its advantages are cost (~$0 vs API), privacy, and offline operation — not accuracy parity. |
| The frontier cluster sits at 86–90%; Jev, at 86.6%, is genuinely in that cluster at a fraction of the price. |
| Full method and raw data in the GitHub repo, <code>evals/</code>. |
| </p> |
| </section> |
| </main> |
|
|
| <footer class="site-footer"> |
| <div class="wrap"> |
| <p> |
| Engine: <a href="https://huggingface.co/harshatheg/Qwen-2.5-1B-RLCD" target="_blank" rel="noopener">harshatheg/Qwen-2.5-1B-RLCD</a> (Apache-2.0) · |
| Model: Qwen2.5-1.5B-Instruct-4bit (Apache-2.0) · |
| Our code and write-up: <a href="https://github.com/rorshopping/jev-on-a-laptop" target="_blank" rel="noopener">MIT</a> |
| </p> |
| </div> |
| </footer> |
|
|
| <script src="app.js"></script> |
| </body> |
| </html> |
|
|
