Spaces:
Running
Running
Download docs.html from abgunaydin/zero-tvm: direct link, hf CLI and curl.
- Browser
- Download file 72.5 kB
-
https://huggingface.co/spaces/abgunaydin/zero-tvm/resolve/main/docs.html
- Command line
-
hf download hf://spaces/abgunaydin/zero-tvm/docs.html
-
curl -L -o docs.html https://huggingface.co/spaces/abgunaydin/zero-tvm/resolve/main/docs.html
72.5 kB
| <html lang="en"> | |
| <head> | |
| <meta charset="UTF-8"> | |
| <meta name="viewport" content="width=device-width, initial-scale=1.0"> | |
| <title>Docs β Zero-TVM</title> | |
| <meta name="description" content="Zero-TVM docs: shader catalog, decode pipeline, weight loader, URL flags, benchmarks. LLMs up to a 35B MoE in the browser, on hand-written WGSL. The Phi-3 forward pass is 10 kernel roles."> | |
| <meta name="theme-color" content="#0a0a0c"> | |
| <link rel="icon" href="/favicon.svg" type="image/svg+xml"> | |
| <meta property="og:type" content="article"> | |
| <meta property="og:url" content="https://zerotvm.com/docs.html"> | |
| <meta property="og:title" content="Docs β Zero-TVM"> | |
| <meta property="og:description" content="Zero-TVM docs: shader catalog, decode pipeline, weight loader, URL flags, benchmarks. LLMs up to a 35B MoE in the browser, on hand-written WGSL. The Phi-3 forward pass is 10 kernel roles."> | |
| <meta property="og:image" content="https://zerotvm.com/og.png"> | |
| <meta name="twitter:card" content="summary_large_image"> | |
| <meta name="twitter:title" content="Docs β Zero-TVM"> | |
| <meta name="twitter:description" content="Zero-TVM docs: shader catalog, decode pipeline, weight loader, URL flags, benchmarks. LLMs up to a 35B MoE in the browser, on hand-written WGSL. The Phi-3 forward pass is 10 kernel roles."> | |
| <meta name="twitter:image" content="https://zerotvm.com/og.png"> | |
| <style> | |
| *, *::before, *::after { margin: 0; padding: 0; box-sizing: border-box; } | |
| :root { | |
| --bg: #0a0a0c; | |
| --surface: #111114; | |
| --surface2: #18181c; | |
| --border: #1e1e24; | |
| --accent: #6c63ff; | |
| --green: #00d4aa; | |
| --yellow: #f5c542; | |
| --red: #ff5f57; | |
| --text: #e8e8ec; | |
| --muted: #666672; | |
| --dim: #3a3a44; | |
| --sidebar-w: 260px; | |
| --topbar-h: 52px; | |
| --code-bg: #0d0d10; | |
| } | |
| html { scroll-behavior: smooth; } | |
| body { | |
| font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif; | |
| background: var(--bg); | |
| color: var(--text); | |
| line-height: 1.65; | |
| display: flex; | |
| flex-direction: column; | |
| min-height: 100vh; | |
| } | |
| /* ββ TOPBAR ββ */ | |
| .topbar { | |
| height: var(--topbar-h); | |
| background: var(--surface); | |
| border-bottom: 1px solid var(--border); | |
| display: flex; | |
| align-items: center; | |
| padding: 0 1.5rem; | |
| gap: 1.5rem; | |
| position: fixed; | |
| top: 0; left: 0; right: 0; | |
| z-index: 100; | |
| } | |
| .logo { | |
| font-weight: 800; | |
| font-size: 0.95rem; | |
| letter-spacing: -0.03em; | |
| color: #fff; | |
| text-decoration: none; | |
| } | |
| .logo span { color: var(--accent); } | |
| .topbar-sep { color: var(--border); } | |
| .topbar-title { font-size: 0.82rem; color: var(--muted); } | |
| .topbar-nav { | |
| margin-left: auto; | |
| display: flex; | |
| gap: 1.25rem; | |
| align-items: center; | |
| } | |
| .topbar-nav a { | |
| font-size: 0.8rem; | |
| color: var(--muted); | |
| text-decoration: none; | |
| transition: color 0.15s; | |
| } | |
| .topbar-nav a:hover { color: var(--text); } | |
| .topbar-cta { | |
| background: var(--accent); | |
| color: #fff ; | |
| padding: 0.3rem 0.85rem; | |
| border-radius: 7px; | |
| font-weight: 600 ; | |
| } | |
| /* ββ LAYOUT ββ */ | |
| .layout { | |
| display: flex; | |
| flex: 1; | |
| padding-top: var(--topbar-h); | |
| } | |
| /* ββ SIDEBAR ββ */ | |
| aside { | |
| width: var(--sidebar-w); | |
| min-width: var(--sidebar-w); | |
| background: var(--surface); | |
| border-right: 1px solid var(--border); | |
| position: fixed; | |
| top: var(--topbar-h); | |
| bottom: 0; | |
| left: 0; | |
| overflow-y: auto; | |
| padding: 1.5rem 0; | |
| } | |
| aside::-webkit-scrollbar { width: 4px; } | |
| aside::-webkit-scrollbar-thumb { background: var(--border); border-radius: 4px; } | |
| .sidebar-section { | |
| padding: 0 1rem 1rem; | |
| } | |
| .sidebar-section-label { | |
| font-size: 0.6rem; | |
| font-weight: 700; | |
| letter-spacing: 0.1em; | |
| text-transform: uppercase; | |
| color: var(--muted); | |
| padding: 0.5rem 0.5rem 0.4rem; | |
| display: block; | |
| } | |
| .sidebar-link { | |
| display: block; | |
| padding: 0.35rem 0.5rem; | |
| font-size: 0.78rem; | |
| color: var(--muted); | |
| text-decoration: none; | |
| border-radius: 6px; | |
| transition: background 0.1s, color 0.1s; | |
| border-left: 2px solid transparent; | |
| } | |
| .sidebar-link:hover { color: var(--text); background: var(--surface2); } | |
| .sidebar-link.active { color: var(--accent); border-left-color: var(--accent); background: #6c63ff0d; } | |
| .sidebar-link.sub { padding-left: 1.25rem; font-size: 0.73rem; } | |
| /* ββ MAIN CONTENT ββ */ | |
| main { | |
| margin-left: var(--sidebar-w); | |
| flex: 1; | |
| padding: 3.5rem 3rem 6rem; | |
| max-width: 900px; | |
| } | |
| /* ββ TYPOGRAPHY ββ */ | |
| h1 { | |
| font-size: 2rem; | |
| font-weight: 800; | |
| letter-spacing: -0.03em; | |
| margin-bottom: 0.5rem; | |
| line-height: 1.2; | |
| } | |
| h2 { | |
| font-size: 1.35rem; | |
| font-weight: 700; | |
| letter-spacing: -0.02em; | |
| margin-top: 3rem; | |
| margin-bottom: 0.75rem; | |
| padding-top: 2rem; | |
| border-top: 1px solid var(--border); | |
| scroll-margin-top: calc(var(--topbar-h) + 1rem); | |
| } | |
| h2:first-of-type { border-top: none; margin-top: 1.5rem; padding-top: 0; } | |
| h3 { | |
| font-size: 1rem; | |
| font-weight: 700; | |
| margin-top: 1.75rem; | |
| margin-bottom: 0.5rem; | |
| color: var(--text); | |
| scroll-margin-top: calc(var(--topbar-h) + 1rem); | |
| } | |
| h4 { | |
| font-size: 0.88rem; | |
| font-weight: 700; | |
| margin-top: 1.25rem; | |
| margin-bottom: 0.35rem; | |
| color: var(--muted); | |
| text-transform: uppercase; | |
| letter-spacing: 0.05em; | |
| } | |
| p { | |
| font-size: 0.9rem; | |
| color: #b0b0bc; | |
| margin-bottom: 1rem; | |
| line-height: 1.75; | |
| } | |
| a { color: var(--accent); text-decoration: none; } | |
| a:hover { text-decoration: underline; } | |
| ul, ol { | |
| margin: 0.5rem 0 1rem 1.5rem; | |
| font-size: 0.88rem; | |
| color: #b0b0bc; | |
| line-height: 1.8; | |
| } | |
| li { margin-bottom: 0.2rem; } | |
| strong { color: var(--text); font-weight: 600; } | |
| em { color: var(--yellow); font-style: normal; } | |
| /* ββ LEAD / SUBTITLE ββ */ | |
| .lead { | |
| font-size: 1.05rem; | |
| color: var(--muted); | |
| margin-bottom: 2rem; | |
| line-height: 1.7; | |
| } | |
| /* ββ PILL BADGES ββ */ | |
| .pill { | |
| display: inline-flex; | |
| align-items: center; | |
| gap: 0.25rem; | |
| padding: 0.15rem 0.5rem; | |
| border-radius: 999px; | |
| font-size: 0.65rem; | |
| font-weight: 600; | |
| border: 1px solid; | |
| vertical-align: middle; | |
| } | |
| .pill.green { background: #00d4aa12; color: var(--green); border-color: #00d4aa30; } | |
| .pill.purple { background: #6c63ff12; color: var(--accent); border-color: #6c63ff30; } | |
| .pill.yellow { background: #f5c54212; color: var(--yellow); border-color: #f5c54230; } | |
| .pill.red { background: #ff5f5712; color: var(--red); border-color: #ff5f5730; } | |
| /* ββ CODE ββ */ | |
| code { | |
| font-family: 'SF Mono', 'Fira Code', monospace; | |
| font-size: 0.8em; | |
| background: var(--surface2); | |
| border: 1px solid var(--border); | |
| padding: 0.1em 0.4em; | |
| border-radius: 4px; | |
| color: #c8c8e0; | |
| } | |
| pre { | |
| background: var(--code-bg); | |
| border: 1px solid var(--border); | |
| border-radius: 10px; | |
| padding: 1.25rem 1.5rem; | |
| overflow-x: auto; | |
| margin: 1rem 0 1.5rem; | |
| font-family: 'SF Mono', 'Fira Code', monospace; | |
| font-size: 0.78rem; | |
| line-height: 1.65; | |
| } | |
| pre code { | |
| background: none; | |
| border: none; | |
| padding: 0; | |
| font-size: inherit; | |
| color: #c8c8e0; | |
| } | |
| /* syntax highlight classes */ | |
| .kw { color: #c792ea; } | |
| .fn { color: #82aaff; } | |
| .str { color: #c3e88d; } | |
| .num { color: #f78c6c; } | |
| .cm { color: #546e7a; font-style: italic; } | |
| .ty { color: #ffcb6b; } | |
| .op { color: #89ddff; } | |
| /* ββ PRE HEADER (filename label) ββ */ | |
| .code-block { margin: 1rem 0 1.5rem; } | |
| .code-label { | |
| background: var(--surface2); | |
| border: 1px solid var(--border); | |
| border-bottom: none; | |
| border-radius: 8px 8px 0 0; | |
| padding: 0.4rem 1rem; | |
| font-family: 'SF Mono', 'Fira Code', monospace; | |
| font-size: 0.7rem; | |
| color: var(--muted); | |
| display: flex; | |
| align-items: center; | |
| gap: 0.5rem; | |
| } | |
| .code-block pre { margin: 0; border-radius: 0 0 8px 8px; } | |
| /* ββ CALLOUTS ββ */ | |
| .callout { | |
| border-radius: 10px; | |
| padding: 1rem 1.25rem; | |
| margin: 1.25rem 0; | |
| font-size: 0.85rem; | |
| line-height: 1.65; | |
| border-left: 3px solid; | |
| } | |
| .callout p { margin: 0; color: inherit; font-size: inherit; } | |
| .callout.info { background: #6c63ff0d; border-color: var(--accent); color: #a0a0d0; } | |
| .callout.tip { background: #00d4aa0d; border-color: var(--green); color: #80c8bc; } | |
| .callout.warning { background: #f5c5420d; border-color: var(--yellow); color: #c0a060; } | |
| .callout.danger { background: #ff5f570d; border-color: var(--red); color: #c08080; } | |
| .callout-title { font-weight: 700; margin-bottom: 0.3rem; color: inherit; } | |
| /* ββ NARROW SCREENS ββ | |
| This page shipped with NO media query at all, behind a 260px fixed | |
| sidebar. At 390px that left main a 130px border-box and, after its 3rem | |
| side padding, about 34px of readable text β with the sidebar covering | |
| two thirds of the screen. It is nav-linked from every page and sits in | |
| the sitemap, so a shared link opened on a phone landed on that. */ | |
| @media (max-width: 900px) { | |
| :root { --sidebar-w: 0px; } | |
| aside { | |
| position: static; width: auto; min-width: 0; height: auto; | |
| border-right: 0; border-bottom: 1px solid var(--border); | |
| max-height: 40vh; overflow-y: auto; | |
| } | |
| main { margin-left: 0; padding: 2rem 1.1rem 4rem; max-width: none; } | |
| /* THE FLEX CONTEXT IS .layout, NOT body. The first version of this | |
| query reblocked body β which was already stacked β and left .layout | |
| a flex row, so the sidebar stayed a 124px side column of wrapped | |
| links beside ~230px of text. Reviewed at 390px; this is the line | |
| that actually stacks them. */ | |
| .layout { display: block; } | |
| .topbar { padding: 0 0.8rem; gap: 0.8rem; } | |
| } | |
| /* ββ TABLES ββ | |
| Nineteen of them are direct children of main with no wrapper. The | |
| longest unbreakable cell is a weight name like | |
| transformer.h.N.post_attention_layernorm.weight, which cannot shrink, | |
| so on a phone the table pushed the BODY sideways rather than scrolling | |
| inside itself. Each table gets its own scroll container instead. */ | |
| .table-scroll { overflow-x: auto; -webkit-overflow-scrolling: touch; margin: 1rem 0; } | |
| table { | |
| width: 100%; | |
| border-collapse: collapse; | |
| font-size: 0.82rem; | |
| margin: 1rem 0 1.5rem; | |
| border: 1px solid var(--border); | |
| border-radius: 8px; | |
| overflow: hidden; | |
| } | |
| th { | |
| background: var(--surface2); | |
| padding: 0.65rem 1rem; | |
| text-align: left; | |
| font-weight: 600; | |
| color: var(--text); | |
| border-bottom: 1px solid var(--border); | |
| } | |
| td { | |
| padding: 0.6rem 1rem; | |
| color: #b0b0bc; | |
| border-bottom: 1px solid var(--border); | |
| vertical-align: middle; | |
| } | |
| tr:last-child td { border-bottom: none; } | |
| tr:hover td { background: #ffffff03; } | |
| /* ββ SHADER CARD ββ */ | |
| .shader-doc-card { | |
| background: var(--surface); | |
| border: 1px solid var(--border); | |
| border-radius: 12px; | |
| padding: 1.25rem 1.5rem; | |
| margin: 1.25rem 0; | |
| } | |
| .shader-doc-card h4 { | |
| font-size: 0.88rem; | |
| font-weight: 700; | |
| color: var(--text); | |
| text-transform: none; | |
| letter-spacing: 0; | |
| margin-top: 0; | |
| display: flex; | |
| align-items: center; | |
| gap: 0.65rem; | |
| } | |
| .shader-doc-card p { font-size: 0.82rem; color: var(--muted); margin: 0.4rem 0 0; } | |
| .binding-table { margin: 0.75rem 0 0; } | |
| .binding-table th, .binding-table td { padding: 0.4rem 0.75rem; font-size: 0.75rem; } | |
| /* ββ STEP FLOW ββ */ | |
| .step-flow { display: flex; flex-direction: column; gap: 0; margin: 1rem 0 1.5rem; } | |
| .step-row { | |
| display: flex; | |
| gap: 1rem; | |
| align-items: flex-start; | |
| } | |
| .step-line { | |
| display: flex; | |
| flex-direction: column; | |
| align-items: center; | |
| flex-shrink: 0; | |
| } | |
| .step-dot { | |
| width: 28px; height: 28px; | |
| border-radius: 50%; | |
| background: var(--accent); | |
| color: #fff; | |
| font-size: 0.68rem; | |
| font-weight: 700; | |
| display: flex; | |
| align-items: center; | |
| justify-content: center; | |
| flex-shrink: 0; | |
| } | |
| .step-line-seg { | |
| width: 2px; | |
| flex: 1; | |
| background: var(--border); | |
| margin: 4px 0; | |
| min-height: 20px; | |
| } | |
| .step-content { padding: 0.05rem 0 1.25rem; } | |
| .step-content h5 { font-size: 0.88rem; font-weight: 700; color: var(--text); margin-bottom: 0.25rem; } | |
| .step-content p { font-size: 0.8rem; color: var(--muted); margin: 0; } | |
| /* ββ DIVIDER ββ */ | |
| hr { border: none; border-top: 1px solid var(--border); margin: 2rem 0; } | |
| /* ββ SCROLL MARGIN ββ */ | |
| [id] { scroll-margin-top: calc(var(--topbar-h) + 1.5rem); } | |
| </style> | |
| </head> | |
| <body> | |
| <!-- ββ TOPBAR ββ --> | |
| <div class="topbar"> | |
| <a href="/" class="logo">Zero<span>TVM</span></a> | |
| <span class="topbar-sep">/</span> | |
| <span class="topbar-title">Documentation</span> | |
| <nav class="topbar-nav"> | |
| <a href="/">Home</a> | |
| <a href="zero-tvm.html" class="topbar-cta">Open Chat β</a> | |
| </nav> | |
| </div> | |
| <div class="layout"> | |
| <!-- ββ SIDEBAR ββ --> | |
| <aside> | |
| <div class="sidebar-section"> | |
| <span class="sidebar-section-label">Getting Started</span> | |
| <a href="#intro" class="sidebar-link active">Introduction</a> | |
| <a href="#quickstart" class="sidebar-link">Quick start</a> | |
| <a href="#how-it-works" class="sidebar-link">How it works</a> | |
| </div> | |
| <div class="sidebar-section"> | |
| <span class="sidebar-section-label">Architecture</span> | |
| <a href="#overview" class="sidebar-link">Overview</a> | |
| <a href="#weight-loader" class="sidebar-link">Weight loader</a> | |
| <a href="#tokenizer" class="sidebar-link">Tokenizer</a> | |
| <a href="#kv-cache" class="sidebar-link">KV cache</a> | |
| <a href="#decode-loop" class="sidebar-link">Decode loop</a> | |
| <a href="#ping-pong" class="sidebar-link">Ping-pong buffers</a> | |
| </div> | |
| <div class="sidebar-section"> | |
| <span class="sidebar-section-label">WGSL Kernel Roles</span> | |
| <a href="#shaders" class="sidebar-link">Overview</a> | |
| <a href="#shader-embedding" class="sidebar-link sub">1 Β· Embedding</a> | |
| <a href="#shader-rmsnorm" class="sidebar-link sub">2 Β· RMSNorm</a> | |
| <a href="#shader-qkv" class="sidebar-link sub">3 Β· QKV+RoPE+KV (fused)</a> | |
| <a href="#shader-attention" class="sidebar-link sub">4 Β· Paged Attention</a> | |
| <a href="#shader-oproj" class="sidebar-link sub">5 Β· int4 Matmul</a> | |
| <a href="#shader-ffn" class="sidebar-link sub">6 Β· Fused FFN</a> | |
| <a href="#shader-addnorm" class="sidebar-link sub">7 Β· Add + RMSNorm</a> | |
| <a href="#shader-kvappend" class="sidebar-link sub">8 Β· KV Append (prefill)</a> | |
| <a href="#shader-rope" class="sidebar-link sub">9 Β· RoPE (prefill)</a> | |
| <a href="#shader-lmhead" class="sidebar-link sub">10 Β· Argmax</a> | |
| </div> | |
| <div class="sidebar-section"> | |
| <span class="sidebar-section-label">Model</span> | |
| <a href="#phi3" class="sidebar-link">Phi-3 constants</a> | |
| <a href="#quantization" class="sidebar-link">Q4F16 format</a> | |
| <a href="#weight-loader" class="sidebar-link">Weight naming</a> | |
| <a href="#qwen3" class="sidebar-link">Qwen3-4B port</a> | |
| <a href="#qwen35" class="sidebar-link">Qwen3.5-4B hybrid</a> | |
| <a href="#qwen36" class="sidebar-link">Qwen3.6 MoE</a> | |
| </div> | |
| <div class="sidebar-section"> | |
| <span class="sidebar-section-label">Extending</span> | |
| <a href="#porting" class="sidebar-link">Port to Phi-4 / Qwen3</a> | |
| <a href="#local-weights" class="sidebar-link">Local weight serving</a> | |
| <a href="#debugging" class="sidebar-link">Debugging tips</a> | |
| </div> | |
| <div class="sidebar-section"> | |
| <span class="sidebar-section-label">Reference</span> | |
| <a href="#bugs-fixed" class="sidebar-link">Bugs we fixed</a> | |
| <a href="#vs-webllm" class="sidebar-link">vs WebLLM</a> | |
| </div> | |
| </aside> | |
| <!-- ββ MAIN ββ --> | |
| <main> | |
| <!-- INTRO --> | |
| <div id="intro"> | |
| <h1>Zero TVM Documentation</h1> | |
| <p class="lead"> | |
| A complete LLM inference engine in the browser β <strong>no WebLLM, no TVM, no ONNX, no WASM runtime</strong>. | |
| 10 hand-written WGSL kernel roles, two tokenizers β Phi-3's SentencePiece-style one and the byte-level BPE every other model uses, and raw WebGPU. | |
| Measured +16.0% on total wall-clock throughput and +31.4% on decode against WebLLM's TVM-autotuned | |
| kernels, identical weights, same session (M2 Max, 2026-07-30). | |
| </p> | |
| <div style="display:flex;gap:0.5rem;flex-wrap:wrap;margin-bottom:1.5rem;"> | |
| <span class="pill green">WebGPU</span> | |
| <span class="pill purple">10 kernel roles</span> | |
| <span class="pill yellow">Q4F16_1</span> | |
| <span class="pill green">Phi-3-mini 3.8B</span> | |
| <span class="pill purple">Paged KV Cache</span> | |
| <span class="pill green">0 dependencies</span> | |
| </div> | |
| <p> | |
| This project implements the full Phi-3-mini-4k-instruct transformer forward pass using only | |
| WebGPU compute shaders written by hand in WGSL. Weights are loaded directly from HuggingFace | |
| in MLC Q4F16_1 format (cached in OPFS after the first load). | |
| The tokenizer is implemented in pure TypeScript β no SentencePiece WASM. | |
| </p> | |
| <div class="callout tip"> | |
| <div class="callout-title">Why?</div> | |
| <p>WebLLM compiles its kernels with TVM; transformers.js runs ONNX Runtime Web. Both work well. Neither is something you read β this is. | |
| This project exists to show that you can understand <em>every single step</em> of a modern transformer | |
| inference pipeline, written at the GPU level, in a browser tab.</p> | |
| </div> | |
| </div> | |
| <!-- QUICKSTART --> | |
| <h2 id="quickstart">Quick start</h2> | |
| <p>No install. Just open the chat page. Weights load from your browser cache (if you've used WebLLM before) or download fresh from HuggingFace (~2 GB).</p> | |
| <div class="step-flow"> | |
| <div class="step-row"> | |
| <div class="step-line"><div class="step-dot">1</div><div class="step-line-seg"></div></div> | |
| <div class="step-content"> | |
| <h5>Open the chat</h5> | |
| <p>Navigate to <code>zero-tvm.html</code>. WebGPU initializes automatically. Requires WebGPU with shader-f16: Chrome 113+ or Edge 113+. Safari did not enable WebGPU by default until 26, and the writable OPFS API the weight cache needs (createWritable) is missing before then. Subgroups are required for the MoE models.</p> | |
| </div> | |
| </div> | |
| <div class="step-row"> | |
| <div class="step-line"><div class="step-dot">2</div><div class="step-line-seg"></div></div> | |
| <div class="step-content"> | |
| <h5>Wait for weights</h5> | |
| <p>First load downloads ~2 GB. Subsequent loads are instant from OPFS. On a small machine, <code>?ctx=</code> shrinks the context window and with it the KV allocation (~0.77 GB at Phi-3βs default 4K with the int8 cache, ~1.5 GB under <code>?kv8=0</code>). Progress shown per shard in the log panel.</p> | |
| </div> | |
| </div> | |
| <div class="step-row"> | |
| <div class="step-line"><div class="step-dot">3</div><div class="step-line-seg"></div></div> | |
| <div class="step-content"> | |
| <h5>Chat</h5> | |
| <p>Phi-3-mini runs at 69.6 tok/s total (83.1 tok/s decode) on M2 Max β +16.0% / +31.4% vs WebLLM on identical weights, same session. Your conversation never leaves the browser. No API calls during inference.</p> | |
| </div> | |
| </div> | |
| <div class="step-row"> | |
| <div class="step-line"><div class="step-dot">4</div></div> | |
| <div class="step-content"> | |
| <h5>(Optional) Cache weights locally</h5> | |
| <p>Run <code>node scripts/download-weights.mjs</code> once to save the shards to <code>.weights-local/</code>. Subsequent loads are served from localhost at full disk speed.</p> | |
| </div> | |
| </div> | |
| </div> | |
| <div class="callout warning"> | |
| <div class="callout-title">GPU requirement</div> | |
| <p>Requires a GPU with <code>shader-f16</code> WebGPU feature (f16 arithmetic). Most M-series Macs and recent NVIDIA/AMD GPUs support this. Intel integrated graphics may not.</p> | |
| </div> | |
| <!-- HOW IT WORKS --> | |
| <h2 id="how-it-works">How it works</h2> | |
| <p>The engine has three phases per generated token:</p> | |
| <ol> | |
| <li><strong>Prefill</strong> β process the prompt in chunks (per token on specs that cannot chunk), building up the KV cache</li> | |
| <li><strong>First decode</strong> β the last prefill step produces the first generated token</li> | |
| <li><strong>Decode loop</strong> β each step takes the previous token as input, runs the full forward pass with the KV cache providing attention context, produces the next token</li> | |
| </ol> | |
| <p>Each forward pass runs <strong>10 kernel roles</strong> through 32 transformer layers β <strong>260 dispatches per token</strong> on the default path β the int8 KV cache adds a <code>kv_quantize</code> pass per layer, making 8 (<code>engine-core.ts</code>: fused f16 is 7, fused int8 is 8). Split-K attention also costs a dispatch per layer, but it has no int8 variant, so the default turns it off β the 228-dispatch reference chain needs <code>?kv8=0&splitk=0</code> β then reads one i32 token ID back from the GPU. For comparison, WebLLM's TVM-generated decode path fires 342 dispatches per token β around 11 distinct shaders on that path.</p> | |
| <!-- ARCHITECTURE --> | |
| <h2 id="overview">Architecture overview</h2> | |
| <p>The engineβs main source files:</p> | |
| <div class="table-scroll"><table> | |
| <thead><tr><th>File</th><th>Role</th></tr></thead> | |
| <tbody> | |
| <tr><td><code>src/zero-tvm/engine-core.ts</code></td><td>THE decode engine. <code>buildDecodeEngine</code> allocates buffers, builds bind groups, runs the decode loop. Five call sites drive this one file: the chat page and the entrance's in-place chat (<code>chat-flow.ts</code>), validate, the room host, the agent host, and the native library entry.</td></tr> | |
| <tr><td><code>src/zero-tvm/chat.ts</code></td><td>Thin page module for <code>zero-tvm.html</code>: boots the engine with URL-flag variants and streams. <code>chat-flow.ts</code> is the turn loop, shared with the entrance.</td></tr> | |
| <tr><td><code>src/zero-tvm/weight-loader.ts</code></td><td>MLC layout: fetches ndarray-cache.json, downloads shards, uploads to GPU buffers.</td></tr> | |
| <tr><td><code>src/zero-tvm/weight-loader-mlx.ts</code></td><td>MLX layout: byte-RANGE reads out of multi-GB safetensors shards, never a whole file. Most of the roster loads through this one.</td></tr> | |
| <tr><td><code>src/zero-tvm/tokenizer.ts</code></td><td>Phi-3's SentencePiece-style tokenizer: encode text β token IDs, decode IDs β text.</td></tr> | |
| <tr><td><code>src/zero-tvm/tokenizer-bpe.ts</code></td><td>Byte-level BPE for every Qwen and Llama build, plus the chat templates β including the three ChatML generations, which disagree about whitespace and about which past turns keep a <code><think></code> block.</td></tr> | |
| <tr><td><code>src/compiler/compiler.ts</code></td><td>Compiles every shader β the hand-written WGSL files plus the generated int4-matmul variants (tiled/subgroup/affine/MoE) β into <code>GPUComputePipeline</code> objects.</td></tr> | |
| </tbody> | |
| </table></div> | |
| <!-- WEIGHT LOADER --> | |
| <h2 id="weight-loader">Weight loader</h2> | |
| <p> | |
| Weights are stored in MLC's <strong>ndarray-cache.json</strong> format β an index file listing every parameter, | |
| which shard binary it lives in, its byte offset, and byte size. | |
| The loader reads this index then fetches each referenced shard. | |
| </p> | |
| <h3>Fetch priority</h3> | |
| <ol> | |
| <li><strong>OPFS</strong> β where this engine caches; a returning visitor loads from here</li> | |
| <li><strong>Browser Cache API</strong> β read-only leftover from prior WebLLM sessions</li> | |
| <li><strong>HuggingFace</strong> β direct HTTPS fetch from <code>huggingface.co/mlc-ai/Phi-3-mini-4k-instruct-q4f16_1-MLC</code></li> | |
| </ol> | |
| <div class="callout danger"> | |
| <div class="callout-title">Version mismatch pitfall</div> | |
| <p>Do not mix a locally-downloaded <code>ndarray-cache.json</code> with shards from the browser cache. | |
| The byte offsets in the index must match the shards exactly. If you download the index fresh but use | |
| cached shards from an older model version, all weight slices will be wrong β zero logits β <code><unk></code> output.</p> | |
| </div> | |
| <h3>Parameter naming (MLC format)</h3> | |
| <p>MLC uses non-standard parameter names. The actual names in the cache:</p> | |
| <div class="table-scroll"><table> | |
| <thead><tr><th>MLC name</th><th>Role</th></tr></thead> | |
| <tbody> | |
| <tr><td><code>transformer.embd.q_weight</code></td><td>Embedding weights (uint32 packed int4)</td></tr> | |
| <tr><td><code>transformer.embd.q_scale</code></td><td>Embedding scales (f16)</td></tr> | |
| <tr><td><code>transformer.norm.weight</code></td><td>Final RMSNorm gamma (after all layers)</td></tr> | |
| <tr><td><code>lm_head.q_weight</code></td><td>LM head weights</td></tr> | |
| <tr><td><code>transformer.h.N.ln.weight</code></td><td>Layer N input_layernorm (normGamma1)</td></tr> | |
| <tr><td><code>transformer.h.N.post_attention_layernorm.weight</code></td><td>Layer N post-attention norm (normGamma2)</td></tr> | |
| <tr><td><code>transformer.h.N.mixer.qkv_proj.q_weight</code></td><td>Layer N QKV projection weights</td></tr> | |
| <tr><td><code>transformer.h.N.mixer.out_proj.q_weight</code></td><td>Layer N output projection weights</td></tr> | |
| <tr><td><code>transformer.h.N.mlp.gate_up_proj.q_weight</code></td><td>Layer N FFN gate+up weights</td></tr> | |
| <tr><td><code>transformer.h.N.mlp.down_proj.q_weight</code></td><td>Layer N FFN down weights</td></tr> | |
| </tbody> | |
| </table></div> | |
| <!-- TOKENIZER --> | |
| <h2 id="tokenizer">Tokenizer</h2> | |
| <p> | |
| A hand-written BPE tokenizer in TypeScript. No SentencePiece WASM, no HuggingFace tokenizers bundle. | |
| Reads <code>tokenizer.json</code> directly. | |
| </p> | |
| <h3>Key steps</h3> | |
| <ul> | |
| <li><strong>Pre-tokenization</strong> β Metaspace: spaces become <code>β</code>. There is NO word splitting: BPE runs over the whole section, so a run of spaces can merge into one token.</li> | |
| <li><strong>BPE encoding</strong> β merge pairs by rank from the merge table in tokenizer.json</li> | |
| <li><strong>Special tokens</strong> β <code><|system|></code>, <code><|user|></code>, <code><|assistant|></code>, <code><|end|></code></li> | |
| <li><strong>Chat template</strong> β Phi-3 format applied by <code>buildChatPrompt()</code></li> | |
| </ul> | |
| <h3>Phi-3 chat template</h3> | |
| <div class="code-block"> | |
| <div class="code-label">π prompt format</div> | |
| <pre><code><|system|> | |
| You are a helpful assistant.<|end|> | |
| <|user|> | |
| What is the capital of Australia?<|end|> | |
| <|assistant|></code></pre> | |
| </div> | |
| <p>Stop tokens: <code>2</code> (EOS), <code>32000</code> (<code><|end|></code>), <code>32007</code> (<code><|endoftext|></code>).</p> | |
| <!-- KV CACHE --> | |
| <h2 id="kv-cache">KV cache</h2> | |
| <p> | |
| Uses a <strong>paged KV cache</strong> layout. Memory is divided into fixed-size pages (16 slots each) | |
| and a page table maps logical positions to physical pages. This is <em>not</em> vLLM-style relocatable | |
| blocks: K is RoPE’d before the cache write on every path, so a cached page is valid only at the | |
| positions it was written at. It is a prefix pool, and cannot become a block pool without moving RoPE. | |
| </p> | |
| <div class="table-scroll"><table> | |
| <thead><tr><th>Parameter</th><th>Value</th><th>Notes</th></tr></thead> | |
| <tbody> | |
| <tr><td><code>PAGE_SIZE</code></td><td>16</td><td>slots per page</td></tr> | |
| <tr><td><code>MAX_PAGES</code></td><td>257</td><td>β 4096 context tokens</td></tr> | |
| <tr><td>Bytes per page (int8, default)</td><td>98,304 + 2,048</td><td>packed i8 K+V, plus one f16 scale per (page, slot, head, side)</td></tr> | |
| <tr><td>Bytes per page (<code>?kv8=0</code>)</td><td>196,608</td><td>32 heads Γ 16 slots Γ 96 dims Γ 2 (K+V) Γ 2 bytes</td></tr> | |
| <tr><td>Total KV buffer</td><td>~24.6 MB per layer</td><td>32 layers β 0.77 GB (int8); ~48.2 MB / ~1.51 GB with <code>?kv8=0</code></td></tr> | |
| </tbody> | |
| </table></div> | |
| <p>Each layer has its own <code>GPUBuffer</code> for KV pages. The page table is a simple identity mapping (page i β physical page i) for single-sequence inference.</p> | |
| <!-- DECODE LOOP --> | |
| <h2 id="decode-loop">Decode loop</h2> | |
| <p>Each call to <code>decodeToken(tokenId, position)</code> submits one command encoder with the full forward pass:</p> | |
| <div class="code-block"> | |
| <div class="code-label">src/zero-tvm/engine-core.ts Β· decodeToken()</div> | |
| <pre><code><span class="cm">// Per-token GPU state written via writeBuffer</span> | |
| B.inputIds β [tokenId] <span class="cm">// i32</span> | |
| B.posMap β [position] <span class="cm">// i32</span> | |
| B.pageIndptr β [0, nnzPages] <span class="cm">// page range</span> | |
| B.lengthInfo β [position+1, 0, 0] <span class="cm">// seq length</span> | |
| <span class="cm">// Forward pass (one command encoder)</span> | |
| embedding(B.residual) <span class="cm">// token β hidden state</span> | |
| rmsNorm(B.hidden1, B.residual) <span class="cm">// initial norm</span> | |
| <span class="kw">for</span> L <span class="op">in</span> <span class="num">0</span>..<span class="num">32</span>: | |
| <span class="cm">// QKV matmul + RoPE + KV-append in ONE dispatch (M4 fusion)</span> | |
| qkvFused(B.qOut, kvPages[L], B.hidden1) | |
| attention(B.attnOut, B.qOut, kvPages[L]) | |
| int4Matmul(B.hidden2, B.attnOut) <span class="cm">// O projection</span> | |
| addNorm(B.hidden2, resIn β B.hidden1, resOut) <span class="cm">// residual + RMSNorm, ping-pong</span> | |
| <span class="cm">// Gate + Up + SiLU + mul + Down in ONE dispatch</span> | |
| fusedFfn(B.hidden2, B.hidden1) | |
| addNorm(B.hidden2, resIn β B.hidden1, resOut) <span class="cm">// residual + RMSNorm, ping-pong</span> | |
| int4Matmul(B.logits, B.hidden1) <span class="cm">// LM head</span> | |
| argmax(B.tokenOut, B.logits) <span class="cm">// β next token ID</span></code></pre> | |
| </div> | |
| <!-- PING-PONG --> | |
| <h2 id="ping-pong">Ping-pong residual buffers</h2> | |
| <p> | |
| WebGPU's validation rules forbid binding the same buffer as both <code>read</code> and <code>read_write</code> | |
| in the same dispatch. The add_norm shader needs to <em>read</em> the old residual and <em>write</em> the new one. | |
| </p> | |
| <p>Solution: two residual buffers that alternate each dispatch.</p> | |
| <div class="code-block"> | |
| <div class="code-label">ping-pong pattern</div> | |
| <pre><code><span class="kw">let</span> resIn = B.residual <span class="cm">// ping (starts with embedding)</span> | |
| <span class="kw">let</span> resOut = B.residual2 <span class="cm">// pong (uninitialized)</span> | |
| <span class="cm">// Each add_norm:</span> | |
| dispatch(addNorm, [delta, resIn, gamma, hidden1, resOut]) | |
| [resIn, resOut] = [resOut, resIn] <span class="cm">// swap β O(1), no GPU copy</span></code></pre> | |
| </div> | |
| <div class="callout info"> | |
| <p>The swap is just two JavaScript variable reassignments β no GPU buffer copy. Both buffers always exist on the GPU; we just change which one we tell the bind group to read vs write.</p> | |
| </div> | |
| <!-- SHADERS --> | |
| <h2 id="shaders">WGSL Kernel Roles</h2> | |
| <p>All shaders live in <code>src/compiler/shaders/</code>, implementing 10 distinct kernel roles. The rest are tiled and subgroup variants of the same role, selectable at runtime via URL flags. The compiler compiles them all at startup into <code>GPUComputePipeline</code> objects.</p> | |
| <p>Binding convention: <code>@group(0)</code> always. Binding indices are <strong>zero-based</strong> and match the order you pass buffers to <code>bg(device, pipeline, [...bufs])</code>.</p> | |
| <div class="callout tip"> | |
| <div class="callout-title">Decode vs prefill paths diverge</div> | |
| <p>The decode loop uses the fused <code>qkv_fused</code> kernel (QKV matmul + RoPE + KV-append all in one dispatch). Prefill still uses separate <code>int4_matmul</code> + <code>rope</code> + <code>kv_append</code> dispatches because prefill processes many tokens at once, and the fusion win only lands for <code>ntoken=1</code>.</p> | |
| </div> | |
| <hr> | |
| <!-- SHADER 1 --> | |
| <div id="shader-embedding"> | |
| <div class="shader-doc-card"> | |
| <h4>π€ 1 Β· Embedding <span class="pill purple">embedding.wgsl</span></h4> | |
| <p>Token ID lookup with Q4F16 dequantization. Each output element is dequantized from a packed int4 value: <code>(nibble - 7) Γ scale</code>.</p> | |
| <div class="table-scroll"><table class="binding-table"> | |
| <thead><tr><th>Binding</th><th>Type</th><th>Role</th></tr></thead> | |
| <tbody> | |
| <tr><td><code>@0</code></td><td><code>read_write f16[]</code></td><td>output hidden state</td></tr> | |
| <tr><td><code>@1</code></td><td><code>read i32[]</code></td><td>input token IDs</td></tr> | |
| <tr><td><code>@2</code></td><td><code>read f16[]</code></td><td>scales (group_size=32)</td></tr> | |
| <tr><td><code>@3</code></td><td><code>read u32[]</code></td><td>packed weights (8 int4 per u32)</td></tr> | |
| <tr><td><code>@4</code></td><td><code>uniform</code></td><td><code>{ seq_len, packGridDimX }</code></td></tr> | |
| </tbody> | |
| </table></div> | |
| <p style="margin-top:0.5rem;"><strong>Dispatch:</strong> 12 workgroups Γ 256 threads = 3072 output elements (D=3072)</p> | |
| </div> | |
| </div> | |
| <!-- SHADER 2 --> | |
| <div id="shader-rmsnorm"> | |
| <div class="shader-doc-card"> | |
| <h4>π 2 Β· RMSNorm <span class="pill purple">rms_norm.wgsl</span></h4> | |
| <p>Root mean square layer normalization. Computes <code>x / sqrt(mean(xΒ²) + Ξ΅) Γ gamma</code>. Uses 256-thread tree reduction in workgroup shared memory.</p> | |
| <div class="table-scroll"><table class="binding-table"> | |
| <thead><tr><th>Binding</th><th>Type</th><th>Role</th></tr></thead> | |
| <tbody> | |
| <tr><td><code>@0</code></td><td><code>read_write f16[]</code></td><td>normalized output</td></tr> | |
| <tr><td><code>@1</code></td><td><code>read f16[]</code></td><td>input</td></tr> | |
| <tr><td><code>@2</code></td><td><code>read f16[]</code></td><td>gamma weights</td></tr> | |
| <tr><td><code>@3</code></td><td><code>uniform</code></td><td><code>{ packGridDimX }</code></td></tr> | |
| </tbody> | |
| </table></div> | |
| <p style="margin-top:0.5rem;"><strong>Dispatch:</strong> 1 workgroup (one token, D=3072)</p> | |
| </div> | |
| </div> | |
| <!-- SHADER 3 Β· QKV FUSED (decode) --> | |
| <div id="shader-qkv"> | |
| <div class="shader-doc-card"> | |
| <h4>β‘ 3 Β· QKV + RoPE + KV-append (fused, decode) <span class="pill purple">qkv_fused.wgsl</span></h4> | |
| <p> | |
| The big M4 fusion. One dispatch replaces three on the decode path: the int4 QKV matmul, the RoPE rotation of Q and K, and the write of K/V into the paged KV cache. Each workgroup computes two output rows that form a RoPE pair (dim and dim+48 within the same head), rotates the pair in registers, and writes K/V straight into <code>kv_pages</code> β the intermediate <code>qkv</code> / <code>k_out</code> / <code>v_out</code> buffers from the pre-fusion path are skipped entirely. | |
| </p> | |
| <div class="table-scroll"><table class="binding-table"> | |
| <thead><tr><th>Binding</th><th>Type</th><th>Role</th></tr></thead> | |
| <tbody> | |
| <tr><td><code>@0</code></td><td><code>read_write f16[]</code></td><td>q_out [3072]</td></tr> | |
| <tr><td><code>@1</code></td><td><code>read_write f16[]</code></td><td>kv_pages (paged KV cache)</td></tr> | |
| <tr><td><code>@2</code></td><td><code>read f16[]</code></td><td>hidden [3072]</td></tr> | |
| <tr><td><code>@3</code></td><td><code>read f16[]</code></td><td>scales [9216 Γ 96]</td></tr> | |
| <tr><td><code>@4</code></td><td><code>read u32[]</code></td><td>packed weights [9216 Γ 384]</td></tr> | |
| <tr><td><code>@5</code></td><td><code>read i32[]</code></td><td>position map</td></tr> | |
| <tr><td><code>@6</code></td><td><code>uniform</code></td><td><code>{ position_map_elem_offset, pages_elem_offset, packGridDimX }</code></td></tr> | |
| </tbody> | |
| </table></div> | |
| <p style="margin-top:0.5rem;"><strong>Dispatch:</strong> 4,608 workgroups (down from 9,216 matmul + 36 RoPE + 12 KV-append = 9,264 in the pre-fusion path). Decode-only; prefill still uses the 3-dispatch path (see shaders 8 and 9).</p> | |
| </div> | |
| </div> | |
| <!-- SHADER 4 Β· PAGED ATTENTION --> | |
| <div id="shader-attention"> | |
| <div class="shader-doc-card"> | |
| <h4>ποΈ 4 Β· Paged Attention <span class="pill purple">attention.wgsl</span></h4> | |
| <p>Multi-head attention over the paged KV cache. Reads K and V from pages, computes scaled dot-product attention with an online-softmax reduction in shared memory. Each workgroup handles one attention head.</p> | |
| <div class="table-scroll"><table class="binding-table"> | |
| <thead><tr><th>Binding</th><th>Type</th><th>Role</th></tr></thead> | |
| <tbody> | |
| <tr><td><code>@0</code></td><td><code>read f16[]</code></td><td>Q [3072]</td></tr> | |
| <tr><td><code>@1</code></td><td><code>read i32[]</code></td><td>page indptr</td></tr> | |
| <tr><td><code>@2</code></td><td><code>read i32[]</code></td><td>page values (page table)</td></tr> | |
| <tr><td><code>@3</code></td><td><code>read f16[]</code></td><td>KV pages</td></tr> | |
| <tr><td><code>@4</code></td><td><code>read i32[]</code></td><td>length info</td></tr> | |
| <tr><td><code>@5</code></td><td><code>read_write f16[]</code></td><td>attn output [3072]</td></tr> | |
| <tr><td><code>@6</code></td><td><code>uniform</code></td><td>attention config (scale, pages)</td></tr> | |
| </tbody> | |
| </table></div> | |
| <p style="margin-top:0.5rem;"><strong>Dispatch:</strong> 1 Γ HEADS workgroups (1 Γ 32). An <code>attention_int8.wgsl</code> variant reads the int8-quantized KV cache, which is the default since 2026-08-18; <code>?kv8=0</code> returns to the f16 <code>attention.wgsl</code>.</p> | |
| </div> | |
| </div> | |
| <!-- SHADER 5 Β· INT4 MATMUL --> | |
| <div id="shader-oproj"> | |
| <div class="shader-doc-card"> | |
| <h4>βοΈ 5 Β· int4 Matmul (output projection + LM head) <span class="pill purple">int4_matmul.wgsl</span></h4> | |
| <p>General-purpose dequantize-on-the-fly int4 Γ f16 matmul. Used for the attention <em>output projection</em> (3072 β 3072) and the <em>LM head</em> (3072 β 32064). Weights are Q4F16_1: <code>N</code> output rows Γ 384 u32 columns (each u32 = 8 int4 values = 32 elements, group_size=32). The tiled / subgroup / vec4 variants are emitted by one generator beside it (<code>int4_matmul.gen.ts</code>); the runtime picks one via the <code>?matmul=</code> URL flag.</p> | |
| <div class="table-scroll"><table class="binding-table"> | |
| <thead><tr><th>Uniform field</th><th>Value</th></tr></thead> | |
| <tbody> | |
| <tr><td><code>K_groups</code></td><td>384 (= input_dim / 8)</td></tr> | |
| <tr><td><code>scale_stride</code></td><td>96 (= input_dim / group_size)</td></tr> | |
| <tr><td><code>N</code></td><td>3072 (o-proj) or 32064 (lm_head)</td></tr> | |
| </tbody> | |
| </table></div> | |
| <p style="margin-top:0.5rem;"><strong>Dispatch:</strong> <code>N</code> workgroups β 3,072 for o-proj, 32,064 for lm_head.</p> | |
| </div> | |
| </div> | |
| <!-- SHADER 6 Β· FUSED FFN --> | |
| <div id="shader-ffn"> | |
| <div class="shader-doc-card"> | |
| <h4>π 6 Β· Fused FFN (Gate Β· Up Β· SiLU Β· Mul Β· Down) <span class="pill purple">fused_ffn.wgsl</span></h4> | |
| <p> | |
| Gate and up projections (both int4 matmuls sharing the 16,384-row <code>gate_up_proj</code> weight matrix), the SiLU activation and the elementwise multiply, in one dispatch. The down projection back to 3,072 dims is its own <code>int4_matmul</code> dispatch β which is why a layer costs 7 dispatches, not 6. | |
| </p> | |
| <div class="table-scroll"><table class="binding-table"> | |
| <thead><tr><th>Binding</th><th>Type</th><th>Role</th></tr></thead> | |
| <tbody> | |
| <tr><td><code>@0</code></td><td><code>read_write f16[]</code></td><td>output [3072]</td></tr> | |
| <tr><td><code>@1</code></td><td><code>read f16[]</code></td><td>input [3072]</td></tr> | |
| <tr><td><code>@2</code></td><td><code>read f16[]</code></td><td>gate_up scales</td></tr> | |
| <tr><td><code>@3</code></td><td><code>read u32[]</code></td><td>gate_up packed weights (16,384 Γ 384)</td></tr> | |
| <tr><td><code>@4</code></td><td><code>read f16[]</code></td><td>down_proj scales</td></tr> | |
| <tr><td><code>@5</code></td><td><code>read u32[]</code></td><td>down_proj packed weights (3,072 Γ 1,024)</td></tr> | |
| <tr><td><code>@6</code></td><td><code>uniform</code></td><td>FFN config</td></tr> | |
| </tbody> | |
| </table></div> | |
| <p style="margin-top:0.5rem;"><strong>Dispatch:</strong> 3,072 workgroups β one per output row of the down projection. A <code>fused_ffn_tiled_sg.wgsl</code> variant uses subgroup reductions; selectable via URL flag.</p> | |
| </div> | |
| </div> | |
| <!-- SHADER 7 Β· ADD + RMSNORM --> | |
| <div id="shader-addnorm"> | |
| <div class="shader-doc-card"> | |
| <h4>β 7 Β· Fused Add + RMSNorm <span class="pill purple">add_norm.wgsl</span></h4> | |
| <p>Residual add + RMSNorm in one pass. Computes <code>residual_out = A + B</code>, then <code>output = RMSNorm(residual_out) Γ gamma</code>. Used twice per layer (post-attention and post-FFN). Mirrors TVM's <code>fuse_add_norm_decode</code>.</p> | |
| <div class="table-scroll"><table class="binding-table"> | |
| <thead><tr><th>Binding</th><th>Type</th><th>Role</th></tr></thead> | |
| <tbody> | |
| <tr><td><code>@0</code></td><td><code>read f16[]</code></td><td>A β the new contribution (O-proj or FFN-down output)</td></tr> | |
| <tr><td><code>@1</code></td><td><code>read f16[]</code></td><td>B β the running residual (resIn)</td></tr> | |
| <tr><td><code>@2</code></td><td><code>read f16[]</code></td><td>gamma β normalization weights</td></tr> | |
| <tr><td><code>@3</code></td><td><code>read_write f16[]</code></td><td>normalized output (B.hidden1)</td></tr> | |
| <tr><td><code>@4</code></td><td><code>read_write f16[]</code></td><td>new residual (resOut β ping-pong)</td></tr> | |
| <tr><td><code>@5</code></td><td><code>uniform</code></td><td><code>{ packGridDimX }</code></td></tr> | |
| </tbody> | |
| </table></div> | |
| <p style="margin-top:0.5rem;"><strong>Dispatch:</strong> 1 workgroup Β· 256 threads Β· 12 elements each = 3,072.</p> | |
| </div> | |
| </div> | |
| <!-- SHADER 8 Β· KV APPEND (prefill) --> | |
| <div id="shader-kvappend"> | |
| <div class="shader-doc-card"> | |
| <h4>πΎ 8 Β· KV Append (prefill path) <span class="pill purple">kv_append.wgsl</span></h4> | |
| <p>Writes K and V vectors into the paged KV cache at the correct slot for each position. On the decode path this work is folded into <code>qkv_fused</code>; on prefill it runs as a separate dispatch because prefill processes many tokens at once and the per-token fusion no longer pays off.</p> | |
| <div class="table-scroll"><table class="binding-table"> | |
| <thead><tr><th>Binding</th><th>Type</th><th>Role</th></tr></thead> | |
| <tbody> | |
| <tr><td><code>@0</code></td><td><code>read f16[]</code></td><td>k_out [3072]</td></tr> | |
| <tr><td><code>@1</code></td><td><code>read f16[]</code></td><td>v_out [3072]</td></tr> | |
| <tr><td><code>@2</code></td><td><code>read_write f16[]</code></td><td>KV pages buffer</td></tr> | |
| <tr><td><code>@3</code></td><td><code>read i32[]</code></td><td>position map</td></tr> | |
| <tr><td><code>@4</code></td><td><code>uniform</code></td><td>page config</td></tr> | |
| </tbody> | |
| </table></div> | |
| <p style="margin-top:0.5rem;"><strong>Dispatch:</strong> 12 workgroups per token (HEADS=32, HEAD_DIM=96).</p> | |
| </div> | |
| </div> | |
| <!-- SHADER 9 Β· ROPE (prefill) --> | |
| <div id="shader-rope"> | |
| <div class="shader-doc-card"> | |
| <h4>π 9 Β· RoPE (prefill path) <span class="pill purple">rope.wgsl</span></h4> | |
| <p>Rotary position embeddings applied to Q and K. Prefill-only β the decode path folds RoPE into <code>qkv_fused</code>. Splits the concatenated 9,216-dim QKV buffer into Q / K / V, rotates Q and K in place based on position, and copies V unchanged.</p> | |
| <div class="table-scroll"><table class="binding-table"> | |
| <thead><tr><th>Binding</th><th>Type</th><th>Role</th></tr></thead> | |
| <tbody> | |
| <tr><td><code>@0</code></td><td><code>read_write f16[]</code></td><td>q_out [3072]</td></tr> | |
| <tr><td><code>@1</code></td><td><code>read_write f16[]</code></td><td>k_out [3072]</td></tr> | |
| <tr><td><code>@2</code></td><td><code>read_write f16[]</code></td><td>v_out [3072]</td></tr> | |
| <tr><td><code>@3</code></td><td><code>read f16[]</code></td><td>qkv input [9216]</td></tr> | |
| <tr><td><code>@4</code></td><td><code>read i32[]</code></td><td>position map</td></tr> | |
| <tr><td><code>@5</code></td><td><code>uniform</code></td><td>RoPE config</td></tr> | |
| </tbody> | |
| </table></div> | |
| <div class="callout danger" style="margin-top:0.75rem;"> | |
| <div class="callout-title">Critical binding order</div> | |
| <p>The binding order <strong>must</strong> be <code>[q_out, k_out, v_out, qkv, posMap, uniform]</code>. Swapping these caused a garbage-output bug during development.</p> | |
| </div> | |
| <p><strong>Dispatch:</strong> 36 workgroups Γ 256 threads = 9,216 = 3 Γ 3,072.</p> | |
| </div> | |
| </div> | |
| <!-- SHADER 10 Β· ARGMAX --> | |
| <div id="shader-lmhead"> | |
| <div class="shader-doc-card"> | |
| <h4>π― 10 Β· Argmax Sampler <span class="pill purple">argmax.wgsl</span></h4> | |
| <p> | |
| Parallel-reduction argmax over the 32,064-entry logit buffer produced by the LM-head <code>int4_matmul</code>. Replaces TVM's ~20-dispatch sampling chain (penalty β softmax β cumsum β argsort β gather β β¦) with a single dispatch. Greedy by default. Temperature, top-p and min-p run through <code>sampler.wgsl</code>, set from the chat header; temperature 0 returns to this <code>argmax</code> kernel rather than a degenerate sampler. | |
| </p> | |
| <div class="table-scroll"><table class="binding-table"> | |
| <thead><tr><th>Binding</th><th>Type</th><th>Role</th></tr></thead> | |
| <tbody> | |
| <tr><td><code>@0</code></td><td><code>read f16[]</code></td><td>logits [32064]</td></tr> | |
| <tr><td><code>@1</code></td><td><code>read_write i32[]</code></td><td>output token id [1]</td></tr> | |
| </tbody> | |
| </table></div> | |
| <p style="margin-top:0.5rem;"><strong>Dispatch:</strong> 1 workgroup (tree reduction over 32,064 logits). An <code>argmax_sg.wgsl</code> subgroup variant is available.</p> | |
| </div> | |
| </div> | |
| <!-- PHI-3 CONSTANTS --> | |
| <h2 id="phi3">Phi-3 model constants</h2> | |
| <div class="code-block"> | |
| <div class="code-label">src/compiler/model-spec.ts</div> | |
| <pre><code><span class="kw">export const</span> PHI3: ModelSpec = makeModelSpec({ | |
| d: <span class="num">3072</span>, <span class="cm">// hidden dimension</span> | |
| heads: <span class="num">32</span>, <span class="cm">// attention heads</span> | |
| kvHeads: <span class="num">32</span>, <span class="cm">// no GQA on Phi-3</span> | |
| headDim: <span class="num">96</span>, <span class="cm">// d / heads</span> | |
| layers: <span class="num">32</span>, <span class="cm">// transformer layers</span> | |
| ffn: <span class="num">8192</span>, <span class="cm">// FFN intermediate dimension</span> | |
| vocab: <span class="num">32064</span>, <span class="cm">// vocabulary size</span> | |
| pageSize: <span class="num">16</span>, <span class="cm">// KV cache slots per page</span> | |
| maxPages: <span class="num">257</span>, <span class="cm">// max pages (β 4096 context)</span> | |
| }) <span class="cm">// one spec among many β the engine is spec-parameterized,</span> | |
| <span class="cm">// and every derived address is computed from these.</span></code></pre> | |
| </div> | |
| <!-- Q4F16 --> | |
| <h2 id="quantization">Q4F16 quantization format</h2> | |
| <p> | |
| MLC's Q4F16_1 format packs 8 int4 values into each <code>uint32</code>. | |
| Scales are stored as <code>float16</code> with <code>group_size=32</code> β one scale per 32 weights. | |
| </p> | |
| <div class="code-block"> | |
| <div class="code-label">Dequantization formula (from embedding.wgsl)</div> | |
| <pre><code><span class="cm">// Extract nibble for element i within a u32</span> | |
| <span class="kw">let</span> nibble = (packed_u32 >> (i * <span class="num">4</span>)) & <span class="num">0xF</span>; | |
| <span class="cm">// Dequantize: center around 0, multiply by scale</span> | |
| <span class="kw">let</span> value = <span class="ty">f16</span>(i32(nibble) - <span class="num">7</span>) * scale;</code></pre> | |
| </div> | |
| <p>Weight shapes in Q4F16 (for Phi-3-mini):</p> | |
| <div class="table-scroll"><table> | |
| <thead><tr><th>Parameter</th><th>q_weight shape (u32)</th><th>q_scale shape (f16)</th></tr></thead> | |
| <tbody> | |
| <tr><td>Embedding</td><td>[32064, 384]</td><td>[32064, 96]</td></tr> | |
| <tr><td>QKV proj (per layer)</td><td>[9216, 384]</td><td>[9216, 96]</td></tr> | |
| <tr><td>O proj (per layer)</td><td>[3072, 384]</td><td>[3072, 96]</td></tr> | |
| <tr><td>Gate+Up FFN (per layer)</td><td>[16384, 384]</td><td>[16384, 96]</td></tr> | |
| <tr><td>Down FFN (per layer)</td><td>[3072, 1024]</td><td>[3072, 256]</td></tr> | |
| <tr><td>LM head</td><td>[32064, 384]</td><td>[32064, 96]</td></tr> | |
| </tbody> | |
| </table></div> | |
| <!-- QWEN3 --> | |
| <h2 id="qwen3">Qwen3-4B (<code>?model=qwen3</code>)</h2> | |
| <p> | |
| The engine is parameterized over a <code>ModelSpec</code>, and a v1 Qwen3-4B (q4f16_1) port ships alongside Phi-3. | |
| Append <code>?model=qwen3</code> to <code>zero-tvm.html</code> or <code>validate.html</code> β it works on the live | |
| site (weights stream from HuggingFace, ~2.3 GB), and <code>node scripts/download-weights.mjs --model qwen3</code> | |
| primes the local dev mirror. Qwen3.6-35B-A3B is the default (bare <code>zero-tvm.html</code> with no | |
| <code>?model=</code>); empty and unknown values still boot Phi-3, so all existing URLs keep their exact behavior. | |
| </p> | |
| <p>What the port exercises that Phi-3 doesn't:</p> | |
| <div class="table-scroll"><table> | |
| <thead><tr><th></th><th>Phi-3-mini</th><th>Qwen3-4B</th></tr></thead> | |
| <tbody> | |
| <tr><td>Attention</td><td>MHA, 32/32 heads</td><td>GQA, 32 query heads over 8 KV heads</td></tr> | |
| <tr><td>QK-norm</td><td>none</td><td>per-head RMSNorm on Q and K between projection and RoPE</td></tr> | |
| <tr><td>Tokenizer</td><td>SentencePiece</td><td>byte-level BPE (Qwen2-style <code>tokenizer.json</code>)</td></tr> | |
| <tr><td>LM head</td><td>separate</td><td>tied β logits reuse the quantized embedding matrix (151,936 vocab)</td></tr> | |
| <tr><td>Decode path</td><td>fused, 7 dispatches/layer</td><td>unfused QKV + fused qk_norm+RoPE+append, 8 dispatches/layer</td></tr> | |
| </tbody> | |
| </table></div> | |
| <p> | |
| Measured 2026-07-30 on an Apple M2 Max under the corrected protocol (same session, identical local weight | |
| bytes, both engines paying a full prefill on every run): Zero-TVM <strong>59.85 tok/s total</strong> | |
| (TTFT 453 ms, decode 75.49) vs WebLLM 0.2.84's prebuilt Qwen3-4B at <strong>45.46 tok/s total</strong> | |
| (self-reported decode 47.77) β <strong>+31.7% total, +58.0% decode</strong>. Unlike the Phi-3 headline, | |
| these figures are static dated text, not synced from <code>bench/results.json</code>; the machine-readable | |
| record is <code>bench/results/qwen3-4b.json</code>. | |
| </p> | |
| <div class="callout warning"> | |
| <div class="callout-title">Read it cautiously β one pair, one machine, and the previous number was withdrawn</div> | |
| <p>The <strong>"75.74 vs 43.75, +73%" pair published on 2026-07-29 is withdrawn.</strong> It was measured after cross-turn prefix reuse shipped (PR #24) but before the bench harness was fixed to reset it, so the Zero-TVM half prefilled a single token per run while the WebLLM half prefilled the whole prompt β not like-for-like. Full writeup at the top of BENCH.md. The engine work itself is unchanged and its Zero-TVM-vs-Zero-TVM A/Bs still stand: QK-norm must run between the QKV matmul and RoPE, which is incompatible with the fused QKV+RoPE+KV-append kernel β the QKV matmul stays a separate dispatch β but since the 2026-07-29 tuning round everything after it is fused (<code>qk_norm_rope_append</code>: per-head norm + RoPE + paged KV write in one pass, 8 dispatches/layer; <code>?fuseqk=0</code> restores the 10-dispatch reference chain) and the K%512 <code>_vec4h</code> matmul variants give d=2560 / ffn=9728 wide loads (<code>?vec4h=0</code> opts out). Same-day A/Bs: fused-qk +2.3%, vec4h +5.7%, combined +5.8% over the flags-off half. The earlier 2026-07-28 pair (25.43 vs 14.15) did not reproduce on the same machine β both engines moved ~3Γ together (degraded session; control-run details in BENCH.md's tuning-round session note). Qwen3-4B is also the model where WebLLM most clearly beats us on time-to-first-token: its 263β271 tok/s prefill implies ~150 ms against our 453 ms.</p> | |
| </div> | |
| <!-- QWEN35 --> | |
| <h2 id="qwen35">Qwen3.5-4B hybrid (<code>?model=qwen35</code>)</h2> | |
| <p> | |
| The third model, and the first <em>hybrid</em> architecture on the engine: 24 gated-DeltaNet (linear-attention) | |
| layers interleaved with 8 gated full-attention layers (attention on every 4th layer). To our knowledge this is | |
| the first hand-written-kernel int4 implementation of a gated-DeltaNet hybrid running in a browser. Append | |
| <code>?model=qwen35</code> to <code>zero-tvm.html</code> or <code>validate.html</code>; | |
| <code>node scripts/download-weights.mjs --model qwen35</code> primes the local dev mirror (~2.6 GB). | |
| </p> | |
| <p>What the hybrid adds over Qwen3-4B:</p> | |
| <div class="table-scroll"><table> | |
| <thead><tr><th></th><th>Qwen3-4B</th><th>Qwen3.5-4B</th></tr></thead> | |
| <tbody> | |
| <tr><td>Layer stack</td><td>36 Γ attention</td><td>24 Γ gated DeltaNet + 8 Γ gated attention (every 4th layer)</td></tr> | |
| <tr><td>Sequence mixer (most layers)</td><td>GQA attention + KV cache</td><td>delta-rule recurrent state (16 k-heads / 32 v-heads, head dims 128, short conv K=4) β no KV cache on those layers</td></tr> | |
| <tr><td>Attention layers</td><td>GQA 32/8, head_dim 128, full RoPE</td><td>GQA 16/4, head_dim 256, partial RoPE (64 of 256 dims), sigmoid output gate per head</td></tr> | |
| <tr><td>Vocab</td><td>151,936</td><td>248,320 (renumbered specials β the shipped <code>mlc-chat-config.json</code> still lists stale Qwen3 stop ids; stops resolve from <code>tokenizer.json</code>)</td></tr> | |
| <tr><td>Weight manifest</td><td><code>ndarray-cache.json</code></td><td><code>tensor-cache.json</code> (MLC renamed it)</td></tr> | |
| </tbody> | |
| </table></div> | |
| <p> | |
| Measured 2026-07-30 on an Apple M2 Max under the corrected protocol (same session, identical local weight | |
| bytes, both engines paying a full prefill on every run): Zero-TVM <strong>65.28 tok/s total</strong> | |
| (TTFT 171 ms, decode 73.30) vs WebLLM 0.2.84's prebuilt Qwen3.5-4B at <strong>32.56 tok/s total</strong> | |
| (self-reported decode 34.32) β <strong>+100.5% total, +113.6% decode</strong>. This is the one model where | |
| first-token latency is roughly a wash rather than a loss (our 171 ms against an implied ~0.2 s from | |
| WebLLM's 175β177 tok/s prefill). Static dated text, not synced from <code>bench/results.json</code>; | |
| the machine-readable record is <code>bench/results/qwen35-4b.json</code>. | |
| </p> | |
| <div class="callout warning"> | |
| <div class="callout-title">Read it cautiously β the GDN kernels are still scalar, so decode remains a floor, and the previous number was withdrawn</div> | |
| <p>The <strong>"65.67 vs 34.04, +93%" cross-check published on 2026-07-29 is withdrawn</strong> β same defect as the Qwen3 pair: measured after cross-turn prefix reuse shipped and before the bench harness reset it, so only the WebLLM half was paying prefill. The two earlier pairs (53.07 vs 32.36, +64%, from the hybrid perf round; 47.99 vs 31.99, +50%, the v1 floor) predate prefix reuse and were like-for-like β superseded, not defective. Full writeup at the top of BENCH.md. Engine caveats unchanged: the 24 DeltaNet layers run scalar (non-subgroup) kernels, so the decode number is a floor; the input projections are fused 4β1 per GDN layer and prompts prefill in chunks of β€64 since the 2026-07-29 prefill round, but the rest of the Phi-3 fusion story has not been applied to the GDN half. One machine, one pair; full protocol and caveats in BENCH.md.</p> | |
| </div> | |
| <!-- PORTING --> | |
| <h2 id="qwen36">Qwen3.6-35B-A3B MoE (<code>?model=qwen36q3</code> / <code>?model=qwen36</code>)</h2> | |
| <p> | |
| The fourth model, shipped 2026-08-05, and three firsts at once: the first sparse MoE | |
| (256 experts, top-8 plus a shared expert on every layer), the first MLX-format | |
| checkpoint (affine <code>w = sΒ·q + b</code>, group 64, per-tensor | |
| biases, loaded by byte range β a 5.3 GB safetensors shard is never one | |
| <code>ArrayBuffer</code>), and the first model here with <strong>no WebLLM build to | |
| benchmark against</strong>. <code>?model=qwen36q3</code> is the 3-bit-expert build | |
| (~16.4 GB, needs ~20 GB free RAM, ~66 tok/s on a quiet 32 GB | |
| M2 Max); <code>?model=qwen36</code> is full 4-bit (~19.5 GB, needs | |
| ~24 GB free RAM). The MoE block runs in 7 dispatches with the expert index in | |
| grid <code>z</code>; the GDN, attention and MoE blocks and a full decoder layer are validated against mlx_lm's own modules | |
| (<code>npm run test:kernels:real</code>). Full engineering notes live in the repo: | |
| <a href="https://github.com/abgnydn/zero-tvm#readme">README</a>, | |
| <a href="https://github.com/abgnydn/zero-tvm/blob/main/BENCH.md">BENCH.md</a>, | |
| <a href="https://github.com/abgnydn/zero-tvm/blob/main/CLAUDE.md">CLAUDE.md</a>. | |
| </p> | |
| <h2 id="porting">Port to Phi-4-mini or Qwen3</h2> | |
| <p>Both are available as MLC Q4F16 packages. Adding a model is a pipeline now, not a hand edit β <code>npm run add-model -- <hf-repo></code> probes the checkpoint, checks it against the constraint matrix, generates the <code>ModelSpec</code> and the registry row, and runs a compile gate under the new dimensions. It comes back green, or with a per-failure report naming the kernel that is missing.</p> | |
| <p>What it automates, which is what the five hand steps below used to be:</p> | |
| <ol> | |
| <li><strong>Dimensions</strong> β d, heads, kv heads, head dim, layers, FFN, vocab, read from <code>config.json</code> into a spec in <code>model-spec.ts</code> (not <code>compiler.ts</code>, and not a <code>PHI3</code> edit β <code>PHI3</code> is one spec among many now)</li> | |
| <li><strong>Tensor names</strong> β resolved from the checkpoint's own index rather than guessed</li> | |
| <li><strong>GQA</strong> β the attention kernels take KV-head repetition from the spec; no shader edit</li> | |
| <li><strong>Chat template</strong> β detected from the checkpoint's <code>chat_template</code>. Getting this wrong does not error, so the detector returns <code>unknown</code> rather than guessing, and one ChatML family alone needs three answers</li> | |
| <li><strong>Weights base</strong> β the spec carries its own HF repo</li> | |
| </ol> | |
| <p>What it cannot do is add a kernel. A model whose blocks are already covered is mechanical; a new architecture block (the MoE router, gated DeltaNet, MLA) is WGSL work, and the pipeline's job there is to say precisely which kernel is missing.</p> | |
| <div class="callout tip"> | |
| <div class="callout-title">Phi-4-mini is the easiest port</div> | |
| <p>Same family as Phi-3. MLC package is already available at <code>mlc-ai/Phi-4-mini-instruct-q4f16_1-MLC</code>. Parameter naming is likely identical or very similar.</p> | |
| </div> | |
| <!-- LOCAL WEIGHTS --> | |
| <h2 id="local-weights">Local weight serving</h2> | |
| <p>Run the download script once to save all shards locally. Subsequent page loads are instant (served from localhost, no network).</p> | |
| <div class="code-block"> | |
| <div class="code-label">terminal</div> | |
| <pre><code>node scripts/download-weights.mjs | |
| <span class="cm"># Downloads to: .weights-local/Phi-3-mini-4k-instruct-q4f16_1-MLC/</span> | |
| <span class="cm"># Served at: /local-weights/Phi-3-mini-4k-instruct-q4f16_1-MLC/</span> | |
| <span class="cm"># Size: ~2 GB</span></code></pre> | |
| </div> | |
| <div class="callout danger"> | |
| <div class="callout-title">Keep index + shards in sync</div> | |
| <p>Always download everything together. Never mix a freshly-downloaded <code>ndarray-cache.json</code> with old cached shards β the byte offsets will not match and all weights will be corrupted.</p> | |
| </div> | |
| <!-- DEBUGGING --> | |
| <h2 id="debugging">Debugging tips</h2> | |
| <h3>All output is <code><unk></code></h3> | |
| <ul> | |
| <li>Weight version mismatch β <code>ndarray-cache.json</code> offsets don't match shard content</li> | |
| <li>Buffer aliasing β <code>add_norm</code> dispatched with same buffer as both <code>@1</code> and <code>@4</code></li> | |
| <li>Wrong rope binding order β check <code>@0=q_out, @1=k_out, @2=v_out, @3=qkv, @4=posMap</code></li> | |
| </ul> | |
| <h3>Garbage / repetitive output</h3> | |
| <ul> | |
| <li>Rope bindings are in the wrong order (this was our bug β garbage like <code>-,unlintzegesenma</code>)</li> | |
| <li>Wrong uniform values for a shader (K_groups, N, etc.)</li> | |
| </ul> | |
| <h3>WebGPU validation error about aliasing</h3> | |
| <ul> | |
| <li>Same buffer bound as <code>read_write</code> and <code>read</code> in one dispatch</li> | |
| <li>Fix: use ping-pong buffers. Never bind <code>B.residual</code> as both <code>@1</code> and <code>@4</code> to <code>add_norm</code></li> | |
| </ul> | |
| <h3>Model not loading (<code>Weight not found</code>)</h3> | |
| <ul> | |
| <li>Log all available keys: the weight loader prints them to console on load</li> | |
| <li>MLC names differ from HuggingFace standard names (<code>transformer.h.N.mixer.*</code> not <code>model.layers.N.self_attn.*</code>)</li> | |
| </ul> | |
| <!-- BUGS FIXED --> | |
| <h2 id="bugs-fixed">Bugs we fixed (and how)</h2> | |
| <div class="table-scroll"><table> | |
| <thead><tr><th>Bug</th><th>Symptom</th><th>Fix</th></tr></thead> | |
| <tbody> | |
| <tr> | |
| <td>Wrong MLC param names</td> | |
| <td><code>Weight not found</code> error on load</td> | |
| <td>Logged all 325 param names from console, updated candidates to <code>transformer.h.*</code> prefix</td> | |
| </tr> | |
| <tr> | |
| <td>Buffer aliasing in add_norm</td> | |
| <td><code><unk></code> Γ 500 at 314 tok/s + WebGPU validation error</td> | |
| <td>Added <code>B.residual2</code> (pong buffer), ping-pong with JS variable swap</td> | |
| </tr> | |
| <tr> | |
| <td>Wrong rope binding order</td> | |
| <td>Garbage: <code>-,unlintzegesenma\dOCΔalloqueIAL</code> repeated</td> | |
| <td>Read rope.wgsl β bindings are <code>@0=q_out @1=k_out @2=v_out @3=qkv @4=posMap</code></td> | |
| </tr> | |
| <tr> | |
| <td>Mixed ndarray-cache.json version</td> | |
| <td><code><unk></code> after downloading index locally but using old cached shards</td> | |
| <td>Always fetch index and shards from the same source atomically</td> | |
| </tr> | |
| </tbody> | |
| </table></div> | |
| <!-- VS WEBLLM --> | |
| <h2 id="vs-webllm">vs WebLLM</h2> | |
| <p>Head-to-head on Phi-3-mini-4k-instruct Q4F16_1, same weights, same session, same browser (Chrome 150 with WebGPU), Apple M2 Max, WebLLM 0.2.80 (the Qwen pairs below ran against 0.2.84) β <code>npm run bench</code>, 128-token target Γ 5 runs, median, 2026-07-30 corrected protocol (latest run recorded in <code>bench/results.json</code>). Every run pays a full prefill on both sides, and both metrics are reported: <em>total</em> is wall-clock throughput including prefill, <em>decode</em> excludes it.</p> | |
| <div class="table-scroll"><table> | |
| <thead><tr><th></th><th>Zero-TVM</th><th>WebLLM</th></tr></thead> | |
| <tbody> | |
| <tr><td>Total throughput (prefill + decode)</td><td><!--bench:zt-->69.55<!--/bench:zt--> tok/s</td><td><!--bench:webllm-->60.0<!--/bench:webllm--> tok/s</td></tr> | |
| <tr><td>Decode only</td><td>83.10 tok/s</td><td>63.23 tok/s (self-reported)</td></tr> | |
| <tr><td>Time to first token (~35-token prompt)</td><td>291 ms</td><td><strong>~150 ms</strong> (implied from 251 tok/s prefill) β WebLLM ahead</td></tr> | |
| <tr><td>Gap</td><td colspan="2"><!--bench:gap-->-16<!--/bench:gap-->% on total relative to WebLLM (negative = Zero-TVM ahead), i.e. +16.0% total and +31.4% decode-only. Same-session pair; cross-session absolute tok/s drifts and the old "β28β¦β31% stable band" was retired on 2026-07-30 (see BENCH.md).</td></tr> | |
| <tr><td>Dispatches / token</td><td>260 default (228 with <code>?kv8=0&splitk=0</code>)</td><td>342</td></tr> | |
| <tr><td>Distinct shaders</td><td>10 hand-written kernel roles</td><td>~11 TVM-generated on the decode path</td></tr> | |
| <tr><td>Shipped JS bundle</td><td>~460 kB / ~126 kB gz (<code>zero-tvm.html</code>)</td><td>~6.0 MB / ~2.2 MB gz (@mlc-ai/web-llm 0.2.84)</td></tr> | |
| <tr><td>Bandwidth utilization (total / decode)</td><td>~36% / ~44% of the 191 tok/s ceiling</td><td>~31% / ~33% of the same ceiling</td></tr> | |
| <tr><td>Paged attention</td><td>β Hand-written</td><td>β TVM compiled</td></tr> | |
| <tr><td>Readable kernels?</td><td>β Yes β every .wgsl file is in the repo</td><td>β No β emitted by the TVM compiler</td></tr> | |
| </tbody> | |
| </table></div> | |
| <h3>The corrected protocol (2026-07-30) β and what it invalidated</h3> | |
| <div class="callout warning"> | |
| <div class="callout-title">Our half of the A/B had stopped paying prefill</div> | |
| <p><code>bench()</code> in <code>src/zero-tvm/bench-console.ts</code> looped its runs against the same prompt and never called <code>engine.resetKVTracking()</code> β while <code>benchPrefill()</code>, <code>specSim()</code> and <code>validate.ts</code> all did. That was harmless until cross-turn prefix reuse shipped on 2026-07-29 (PR #24). After it, runs 2..N of every bench found the whole prompt already absorbed and prefilled exactly <strong>one</strong> token, while the WebLLM half β a fresh chat completion per run β kept paying a full prefill inside its wall clock. The two halves were measuring different work.</p> | |
| <p><strong>Withdrawn as a result:</strong> the Qwen3-4B "75.74 / 43.75, +73.1%" and Qwen3.5-4B "65.67 / 34.04, +92.9%" pairs, both 2026-07-29. Everything earlier predates prefix reuse and was like-for-like. <strong>Fixed:</strong> <code>bench()</code> resets before every run, both halves split TTFT from decode, and WebLLM's own per-run decode/prefill rates are captured instead of logged once and discarded. Prior published numbers stay in BENCH.md as dated history with the defect explained in place β nothing was silently overwritten.</p> | |
| </div> | |
| <h3>The advantage grows with architecture recency</h3> | |
| <p>Monotonic on both metrics across the three models then shipped, all measured 2026-07-30 on the same machine (Qwen3.6-35B-A3B came later and has no WebLLM baseline to pair against β see BENCH.md):</p> | |
| <div class="table-scroll"><table> | |
| <thead><tr><th>Model</th><th>Architecture released</th><th>Ξ total</th><th>Ξ decode</th></tr></thead> | |
| <tbody> | |
| <tr><td>Phi-3-mini</td><td>2024</td><td>+16.0%</td><td>+31.4%</td></tr> | |
| <tr><td>Qwen3-4B</td><td>2025</td><td>+31.7%</td><td>+58.0%</td></tr> | |
| <tr><td>Qwen3.5-4B</td><td>2026</td><td>+100.5%</td><td>+113.6%</td></tr> | |
| </tbody> | |
| </table></div> | |
| <p>The reading that fits: compiler stacks have had less time to tune newer architectures, so there is more headroom for a hand-written kernel set to take. That is an <strong>observation across three points</strong>, not a proven law β one machine, one browser, one day, no mechanism isolated, no control for how differently each model stresses the two engines β and the baseline is not constant across the rows (WebLLM 0.2.80 for Phi-3, 0.2.84 for both Qwen rows). Worth testing on a fourth model.</p> | |
| <h3>Where we lose: time to first token on short prompts</h3> | |
| <div class="callout warning"> | |
| <div class="callout-title">WebLLM reaches the first token sooner than we do</div> | |
| <p>WebLLM's self-reported prefill runs at 251 tok/s on Phi-3 and 263β271 tok/s on Qwen3-4B. Against the ~35-token bench prompt that implies a TTFT of roughly <strong>150 ms</strong> on Phi-3, where we measure <strong>291 ms</strong>; on Qwen3-4B our <strong>453 ms</strong> is worse still. Only Qwen3.5-4B is a wash (our 171 ms vs an implied ~0.2 s). Stated plainly: <strong>we win sustained decode decisively and lose the first-token sprint on short inputs.</strong></p> | |
| <p>It is specifically a <em>short</em>-prompt weakness. Chunked prefill measures 202 tok/s on an 816-token prompt (2026-07-29), and cross-turn prefix reuse removes most of a follow-up turn's prefill on the pure-attention models. On the gated-DeltaNet hybrids it currently does not: the 2026-08-19 chat-template fix ended the token-level extension those figures were measured under, and the hybrids' state snapshots land above the point where the new prompt diverges, so a short conversation re-reads in full. Unmeasured since, and open. What is not competitive is the fixed cost of a short, cold prefill with nothing cached to reuse. It is the top open item on BENCH.md's levers list.</p> | |
| </div> | |
| <h3>How the result flipped (22% behind → +16% total / +31% decode ahead)</h3> | |
| <p>An earlier head-to-head (M2 Pro, 2026-06) read 42.14 vs ~51.5 tok/s β Zero-TVM 22% <em>behind</em>. Both the hardware and the engine changed since, so the two Zero-TVM numbers are not a pure optimization delta; the same-session, same-machine WebLLM number is the valid comparator for the current Zero-TVM median (both in the table above). What changed in the engine:</p> | |
| <ul> | |
| <li><strong>Correctness fixes</strong>. f32 accumulation in the fused FFN, a workgroup-barrier fix in attention, and a decode off-by-one fix β the old number understates even the old hardware.</li> | |
| <li><strong>vec4 loads, now default</strong>. Re-declaring weight + activation buffers as <code>array<vec4<u32>></code> in the int4 matmuls and <code>qkv_fused</code> measured +7.1% on M2 Max; opt out with <code>?vec4=0</code> / <code>?vec4qkv=0</code>.</li> | |
| <li><strong>Tiled + subgroup kernels</strong>. The 4-row tiled subgroup FFN and the <code>_sg</code> matmul/attention/argmax variants are the default path on Apple sg32 hardware.</li> | |
| </ul> | |
| <p>Default from 2026-07-27 until int8 KV shipped on 2026-08-18, which has no split-K variant and so disables it β reachable now only with <code>?kv8=0</code>: split-K attention (<code>?splitk=N</code>, ~+3% at short context, needs a long-context A/B); the best measured opt-in config of that era (<code>?vec4=1&vec4qkv=1&splitk=8</code>, 68.36 tok/s on 2026-07-25) is now simply the default path. Falsified and not shipped: FFN prologue fusion (<code>?fuseprologue=1</code>, β13.7% on M2 Max). Dispatch overhead was never the gap β Zero-TVM already submitted fewer dispatches per token (260 vs 342) when it was slower. Full A/B tables in BENCH.md.</p> | |
| <h3>The memory-bandwidth ceiling</h3> | |
| <p>Phi-3-mini Q4F16_1 touches ~2.09 GB of weights per decode token (the weight-shape table above, summed: 32 layers plus the LM head; the embedding contributes one row). On M2 Max's 400 GB/s memory bus that's ~5.2 ms/token, or ~191 tok/s theoretical max. Zero-TVM's measured decode-only median sits at roughly 44% of that ceiling (36% on total, which pays prefill too); WebLLM's at roughly 33% decode-only / 31% total (exact medians in the table above). Neither engine can exceed the ceiling without changing weight layout or quantization (int8 KV halves the cache rather than the weights, so it moves this very little; it is the default since 2026-08-18, and <code>?kv8=0</code> opts out).</p> | |
| </main> | |
| </div> | |
| <script> | |
| // Highlight active sidebar link on scroll | |
| const links = document.querySelectorAll('.sidebar-link[href^="#"]') | |
| const observer = new IntersectionObserver((entries) => { | |
| for (const entry of entries) { | |
| if (entry.isIntersecting) { | |
| const id = entry.target.id | |
| links.forEach(l => { | |
| l.classList.toggle('active', l.getAttribute('href') === '#' + id) | |
| }) | |
| } | |
| } | |
| }, { rootMargin: '-60px 0px -60% 0px' }) | |
| document.querySelectorAll('[id]').forEach(el => observer.observe(el)) | |
| </script> | |
| </body> | |
| </html> | |