index.html

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Parallel Constrained Decisions — static demo</title>
<link rel="stylesheet" href="style.css">
</head>
<body>
<header class="site-header">
  <div class="wrap">
    <div class="badge">Unofficial demo · not affiliated with TypeSafe AI</div>
    <h1>Typed decisions, one pass, no parsing</h1>
    <p class="lede">
      Reproducing the <em>parallel constrained decoding</em> idea behind Jev (TypeSafe AI's
      "System One" model) on a stock 1.5B model, on a laptop. Instead of generating JSON
      token-by-token, all schema fields are evaluated in <strong>one batched forward pass</strong>
      and the JSON is assembled programmatically — so it can never be malformed.
    </p>
    <div class="header-links">
      <a href="https://github.com/rorshopping/jev-on-a-laptop" target="_blank" rel="noopener">Code &amp; benchmarks on GitHub</a>
      <a href="https://typesafe.ai/" target="_blank" rel="noopener">What TypeSafe actually launched</a>
    </div>
  </div>
</header>

<main class="wrap">
  <section class="card">
    <h2>How it works</h2>
    <pre class="diagram">
[context + schema] ──► prefill (once) ──► KV cache
                                            │ broadcast × N fields
      ┌───────────┬───────────┬─────────────┴───────────┐
    field 1     field 2     ...                      field N
      └───────────┴──── one batched forward pass ────────┘
                          │
              slice logits → softmax → pick value + probability</pre>
    <p class="muted">
      The pipe below is <strong>real recorded output</strong> from
      <code id="model-name">mlx-community/Qwen2.5-1.5B-Instruct-4bit</code> on an
      M5 MacBook Air (16 GB). This page is static: no model runs in your browser.
      The interactive version (choose your own context, run live) is the Gradio
      app in the <a href="https://github.com/rorshopping/jev-on-a-laptop/tree/main/hf-space" target="_blank" rel="noopener">repo's <code>hf-space/</code> folder</a>.
    </p>
  </section>

  <section class="card">
    <h2>Presets</h2>
    <div id="preset-tabs" class="tabs" role="tablist"></div>
    <div id="demo" class="demo" aria-live="polite"></div>
  </section>

  <section class="card">
    <h2>Does a bigger model help? (measured)</h2>
    <p class="muted">24 labeled cases × 3 fields = 72 decisions per model, same engine. "All fields exact" = every field of the case correct.</p>
    <table class="bench">
      <thead>
        <tr><th>Model (4-bit)</th><th>Primary field</th><th>All fields exact</th><th>Latency / case</th></tr>
      </thead>
      <tbody>
        <tr><td>Qwen2.5-1.5B</td><td>58% <span class="muted">(= majority-class baseline)</span></td><td>50%</td><td>147 ms</td></tr>
        <tr><td><strong>Qwen2.5-7B</strong></td><td class="yes"><strong>96%</strong></td><td>72%</td><td>611 ms</td></tr>
        <tr><td><strong>Qwen3-8B</strong></td><td>92%</td><td class="yes"><strong>85%</strong></td><td>646 ms</td></tr>
      </tbody>
    </table>
    <p class="muted small">
      Bigger is worth it — but not "biggest wins": <strong>7B</strong> for the single decision you act on,
      <strong>8B</strong> when the whole typed payload must be right, <strong>1.5B</strong> for demos only.
      And <span class="warn">confidence does not reliably flag errors</span> — the 7B was >90% confident on
      13 of its 20 wrong fields. Full breakdown in the GitHub repo.
    </p>
  </section>

  <section class="card">
    <h2>How does this compare to Jev itself?</h2>
    <p class="muted">
      We ran <strong>all public example cases of TypeSafe's four workflows</strong> (20 cases, 373 question-pairs)
      and scored everyone against TypeSafe's own reference — the consensus of GPT-6 Astra and Fable 5.1.
      Strict like-for-like, on the 343 pairs every model answered:
    </p>
    <table class="bench">
      <thead>
        <tr><th>Model</th><th>Agreement</th><th>Pairs</th></tr>
      </thead>
      <tbody>
        <tr><td>Opus (published)</td><td>89.8%</td><td>308/343</td></tr>
        <tr><td>DeepSeek v4.1 Flash (max)</td><td>89.5%</td><td>307/343</td></tr>
        <tr><td>Sol (published)</td><td>89.2%</td><td>306/343</td></tr>
        <tr><td><strong>Jev / TypeSafe (published)</strong></td><td><strong>86.6%</strong></td><td><strong>297/343</strong></td></tr>
        <tr><td><strong>local Qwen2.5-7B (free, this laptop)</strong></td><td><strong>73.8%</strong></td><td><strong>253/343</strong></td></tr>
        <tr><td>local Qwen3-8B (3 of 4 workflows)</td><td>71.2%</td><td>114/160</td></tr>
      </tbody>
    </table>
    <p class="muted small">
      <strong>The honest result: the free local model is ~13 points behind Jev on Jev's own benchmark.</strong>
      Its advantages are cost (~$0 vs API), privacy, and offline operation — not accuracy parity.
      The frontier cluster sits at 86–90%; Jev, at 86.6%, is genuinely in that cluster at a fraction of the price.
      Full method and raw data in the GitHub repo, <code>evals/</code>.
    </p>
  </section>
</main>

<footer class="site-footer">
  <div class="wrap">
    <p>
      Engine: <a href="https://huggingface.co/harshatheg/Qwen-2.5-1B-RLCD" target="_blank" rel="noopener">harshatheg/Qwen-2.5-1B-RLCD</a> (Apache-2.0) ·
      Model: Qwen2.5-1.5B-Instruct-4bit (Apache-2.0) ·
      Our code and write-up: <a href="https://github.com/rorshopping/jev-on-a-laptop" target="_blank" rel="noopener">MIT</a>
    </p>
  </div>
</footer>

<script src="app.js"></script>
</body>
</html>