ROADMAP.md

# Roadmap — open work only

**This is the only list of open work.** Anything finished moves to [ARCHIVE.md](ARCHIVE.md) with the
reasoning that produced it. [AI.md](AI.md) is the reference for what is true and measured;
[API.md](API.md) is the full call surface, asserted against the code;
[WEBLLM-SURFACE.md](WEBLLM-SURFACE.md) is what WebLLM already does and must be read before adding a
capability.

Sections are named, not numbered. An earlier split across three files used "Track 1"/"Track 2" to
mean different things in each, which is how a reader ends up implementing the wrong item.

**Sequencing.** The whole project exists to make WebLLM *more compatible, easier to use, foolproof
to build on*. Every item below is ranked against that. `everything-webgpu@0.1.0` is on npm as of
this writing — the install path, the four-line demo, the ergonomic verbs, `API.md` and `examples/`
all shipped and are recorded in [ARCHIVE.md](ARCHIVE.md). Section 1 is what remains before the
extraction is fully proven: the extension itself rebuilding on the package. Section 4 (performance)
is frozen as a block, except for "Measure Ollama", until a real user calls it slow: it is the
highest-risk, highest-effort work and it re-touches the tvmjs internals the WebLLM-upgrade section
(also in [ARCHIVE.md](ARCHIVE.md)) just spent its time hardening.

---

## 1. Ship it — the last mile

`everything-webgpu@0.1.0` is on npm. What shipped — the install path, the four-line demo, the
ergonomic verbs, [API.md](API.md), [examples/](examples/), the bundle-size story, licence
compliance — is in [ARCHIVE.md](ARCHIVE.md), "Shipping 0.1.0". What remains is proving the boundary
from the other side.

- [ ] **`demo` consumes the package**, and the popup + manager leave `main`. The acceptance test for
      the whole extraction: if the extension rebuilds on the published package, the boundary is
      right. `npm run e2e` passing is the same claim at the source-tree level.
- [~] **Gate A — a bare Vite page loads a prebuilt model and generates.**
      [examples/bare/](examples/bare/) exists and the GPU-free half is proven: `vite build` resolves
      through the `exports` map and splits into an entry chunk, the decode worker as its own chunk,
      and the 6 MB WebLLM bundle as a lazy one. What remains is **one run in a WebGPU browser** to
      confirm it generates. `npm run verify-consumer` already exercises a real install end to end,
      short of the actual generation.
- [ ] **Defect: Worker unbundled relative imports in production (`vite build` / Rollup blind spot).**
      First real-browser run on a static host (Hugging Face Spaces) surfaced a packaging defect
      that passed all prior headless assertions. On `vite build`, `engine.load()` threw
      `PACKAGE_INCOMPLETE`: *"The decode worker failed to load from .../assets/engine-worker-*.js"*.

      **Root cause:** Rollup's `new Worker(new URL(..., import.meta.url), { type: "module" })`
      handling copies `engine-worker.js` as an asset chunk without bundling or rewriting its internal
      ESM imports. The emitted worker still carries:
      `import { WebWorkerMLCEngineHandler } from "../../vendor/web-llm.js"`
      `import { WORKER_CONFIGURE } from "./constants.js"`
      `import { DEFAULT_DECODE_STEPS } from "./multistep.js"`
      On a static production deployment, these relative URLs resolve to 404s on the root domain,
      killing the worker silently before handshake. `vite dev` hid this because `optimizeDeps.exclude`
      served files directly from source where siblings exist. `test/consumer/verify.mjs` line 130
      missed it because it asserted file *existence* (`assets.some(f => f.startsWith("engine-worker"))`),
      never executing the worker in a browser.

      **Resolution:**
      - [x] **Demo workaround:** Pre-bundle `engine-worker.js` with `esbuild --bundle --format=esm` into a
            standalone 6.58 MB file (`public/engine-worker.js`) with zero external imports, and pass explicit
            `workerUrl: new URL("engine-worker.js", window.location.href)`.
      - [ ] **Library permanent fix for 0.1.1:** In `build.mjs`, pre-bundle `engine-worker.js` into
            a self-contained `dist/worker.bundle.js` at package build time. Point `DEFAULT_WORKER_URL`
            at the bundle so zero consumer configuration or `workerUrl` is required. Update
            `verify-consumer` with a headless browser fetch/execute assertion to prevent regressions.
- [ ] **Gate A′ — the same page ingests a local folder.** `cache.put()` against the synthetic
      `local-model.invalid` key is proven only on an extension origin. Gates the *offline* route,
      not the library.
- [ ] **Gate B — Chrome.** Measure tok/s. Expected to beat Firefox because KV reuse is not disabled
      there. Entirely a prediction today; `probeDevice()` is the instrument that makes it reportable.
- [ ] **Every `ERROR` code carries an actionable `fix`.** The preflight (`environment()`) and the
      typed errors (9 codes now, `PACKAGE_INCOMPLETE` included) both exist; what is not yet done is
      the systematic pass confirming each code's `message`/`detail` names a cause *and* a fix, and
      that the README/API.md point at `environment()` as the "will this run here?" call it is meant
      to be.

## 2. Verb consolidation — done

`load(src, opts)`, `unload(id, level)` / `remove()` / `unloadAll()`, and `environment()` +
`environment.measure()` all shipped. The reasoning — the two dropped `load()` dispatch rules, why a
bare `unload()` frees only the current model, why read and write are split — is in
[ARCHIVE.md](ARCHIVE.md), "Verb consolidation". `chat.completions.create()` is unchanged and stays
that way.

## 3. Engine capability

`prefetch()`, `embed()` / `embedRaw()`, and the `ask()` / `conversation()` / `ghostText()` recipes
shipped — see [ARCHIVE.md](ARCHIVE.md), "Engine capability". One item is still open:

- [ ] **LRU eviction** when quota is short. `cacheState()` and `evict()` exist; nothing yet decides
      *what* to drop. Table stakes for "it just works" the moment a second model is cached — but the
      eviction policy is a decision, not an implementation detail.

## 4. Performance — frozen, except the measurement

Model- and kernel-level work, independent of the library structure — and therefore independent of
everything this project is *for*. **Frozen as a block until a real user calls it slow.** The library
ships now, so "nobody can install it" is no longer the argument; the argument is that this is the
highest-risk, highest-effort work on the page, it re-touches the tvmjs internals the WebLLM-upgrade
section (in [ARCHIVE.md](ARCHIVE.md)) just spent its time hardening, and a speedup nobody has asked
for is risk without a return. The one exception is "Measure Ollama": it is cheap, and it decides
whether any of the rest is worth its risk.

- [ ] **Retune the dlight GEMV schedule** — more work per thread before the reduction. Measured
      **1.83x** in isolation, which would put decode near the ~41 GB/s dequant ceiling. Confirmed
      untaken on the 2B build. Needs a recompile; toolchain is stood up.
- [ ] **Batched decode.** The model lib exports `batch_decode` / `batch_prefill` / `batch_verify`
      and a paged KV cache; WebLLM hardcodes `defaultMaxNumSequence = 1` and `numSamples = 1`
      (bundle lines 15250, 15276). Lifting that reads the weights once per step for N sequences —
      projected **~4x on the `batch` API**, which is the page-translation shape and the one workload
      the engine currently makes no faster than a loop of `chat` calls. No recompilation needed.
      **Not** a fix for the 1.06x second-engine measurement: a second engine buys task isolation,
      never aggregate throughput, and the two are complementary rather than alternatives.
- [ ] **Interrupt granularity in `sampleBurst`.** `interruptGenerate()` only sets a flag the
      caller's loop reads between `decode()` calls, and a burst runs all K forward steps without
      checking it — so a preempted job finishes its whole burst first (~583 ms at K=15). It binds in
      three cases: the seconds-long window while `#grow()` builds a second engine, a machine where
      `#growthBlocked` is set, and any third concurrent task once the pool is at cap. Fix in our
      `multistep.js`, not WebLLM; `discardLookahead` already handles the resulting state.
- [ ] **Auto-tune `decodeSteps` from the burst probe.** K=15 is a constant derived from one machine
      and one model — Firefox on Apple Silicon, `Qwen3.5-0.8B` at ~7.3 ms/step. `multistep.js`'s own
      header says the best K falls as the model grows (a 25 ms/step model wants K=4), so the shipped
      default is wrong for every model but the one it was measured on, and wrong silently: a bad K
      costs throughput and reports nothing.

      **The infrastructure is already there.** `onBurst` delivers `{ steps, tokens, ms, encodeMs,
      syncMs, dispatches, forwardDispatches, flushes }` per burst
      ([multistep.js:246](src/engine/multistep.js#L246)), `setSteps()` is hot
      ([multistep.js:259](src/engine/multistep.js#L259)), and `WORKER_CONFIGURE` already calls
      `resetStats()` on every retune so "a sweep's points never bleed into each other"
      ([engine-worker.js:137](src/engine/engine-worker.js#L137)). It was built for this.

      **The obvious derivation does not work.** `K = floor(tickMs / perStepMs)` needs `perStepMs`, and
      the probe cannot separate it: `syncMs` is GPU execution *plus* the wait for the poll tick, which
      is the quantity being solved for. Only `encodeMs` is clean — the K-step loop contains no `await`,
      so it is content-process CPU alone — and it is the smaller half.

      **What works instead** is a hill-climb on the one quantity directly observed, `tokens / ms`.
      Throughput at K is `K / (ceil(K·p/tick)·tick)`; its peaks sit at `K = floor(n·tick/p)` and are
      asymptotically equal in n, so the first peak is the target and climbing K until throughput drops
      finds it without ever naming `p` or `tick`. That is also what makes it survive Chrome, where the
      100 ms tick premise does not hold.

      **Three subtleties that will bite.** (1) `onBurst.steps` is the *clamped* burst size —
      `burstSize()` cuts it by `max_tokens` and the context window
      ([multistep.js:297](src/engine/multistep.js#L297)) — so the tuner must learn only from bursts
      where `steps === config.steps`, or it learns from end-of-generation stubs. (2) The first burst on
      a pipeline carries shader compilation; skip it. (3) `clampSteps("auto")` is **1** today
      (`Number("auto")` → NaN → `|| 1`), so an `"auto"` sentinel silently *disables* multi-step unless
      `clampSteps` and [API.md](API.md)'s documented `1`–`32` range change together —
      `api-doc.test.mjs` asserts the two agree.

      **Sequencing.** Composes with per-priority K below rather than replacing it: this finds the
      throughput ceiling K_max for the device+model, per-priority then spends it (`interactive` takes
      min(K_max, 4), `background` takes K_max). Do this one first — per-priority is written against the
      15 this replaces. Unlike the rest of section 4 it touches **no new tvmjs internals**: it reads
      telemetry that already flows and calls a setter that already exists, which is the argument for
      lifting the freeze on this item alone. It cannot be *validated* here, though — the sawtooth only
      exists on real hardware, and Gate B (Chrome) is what proves the controller is finding a real peak
      rather than re-deriving 15.

- [ ] **Per-priority `decodeSteps`.** K=15 maximises throughput but emits 15 tokens every ~583 ms,
      which reads as a stall. `interactive` should use K=2–4, `background`/`batch` K=32. Do this
      *after* the interrupt fix — raising K for background work lengthens exactly the bursts that
      preemption has to wait out.
- [ ] **Restore cross-turn KV reuse.** Every turn re-prefills the whole history at **5.27 ms/token**,
      so a turn near the 4096 limit pays ~22 s before its first token. Pack
      `batch_prefill_paged_kv_kernel`'s six i32 metadata buffers into one with offsets (10 bindings
      → 5); the offsets already exist in its uniform block.

## 5. Test and infrastructure

Both are "fix when it next bites", not scheduled work. The completed e2e run that used to sit here
is in [ARCHIVE.md](ARCHIVE.md), including the `storageBuffersPerStage=9` anomaly worth re-checking
on the next run.

- [ ] Fix `PROFILE_PATH` in [test/e2e/run.mjs](test/e2e/run.mjs): it passes `--profile-path`, but
      web-ext 8 calls it `--firefox-profile` and exits with `Unknown arguments`.
- [~] **Isolate the bench's pass-sweep onto its own device** — done, and it **did not do what the
      item assumed**. 2048 compute passes in one encoder loses the WebGPU device, and a lost device
      does not throw: later calls silently no-op.

      Two corrections from measuring it. First, the stated worry (later probes poisoned) was empty
      — the sweep was already the last phase, so nothing ran after it. The real damage was to the
      sweep's **own** numbers: the run that prompted this reported `perPass=-3.9us`, 512 passes
      measured faster than one, which is the device dying mid-sweep and the remaining submits
      becoming free. That half is fixed — `n512` now reads a plausible ~10us and `n2048` reports
      `discarded (device lost during the 2048-pass encode)` instead of inventing a number.

      Second, **isolation does not contain the loss.** With the sweep on its own device,
      `deviceLostDuringBench` still reports the main device lost during this phase, on both runs. A
      runaway command buffer on Metal appears to reset the whole adapter rather than one device on
      it. So the honest state is: sweep numbers are now trustworthy, containment is not achieved,
      and the bench is safe only because this is the last phase.

      Still open, if containment is wanted: run the sweep in a dedicated worker or a separate page
      so the reset cannot reach the measuring context at all. Also worth noting the attribution is
      imprecise — `lostDuring` records the phase at which `device.lost` *resolves*, and the
      16384-dispatch probe immediately above is another plausible culprit.

## 6. Deferred, with the condition that would reopen them

| Item | Reopens when |
| --- | --- |
| **Vision / image understanding** | An in-house compact VLM exists. The prebuilt option is `Phi-3.5-vision` at 3.95 GB, projecting ~4–5 tok/s and not co-resident with a text model in 16 GB. `modelType: "vlm"` already flows through registration, so the compile lands on an engine that accepts it. Open questions to answer against the real model: whether multi-step decoding survives an image prefill, and whether the Firefox 9-storage-buffer workaround holds for the vision tower's kernels. |
| **MV3 migration** | `demo` is forced off MV2. `CreateExtensionServiceWorkerMLCEngine` solves the messaging and in-work keep-alive but **not** the actual problem — an idle service worker is killed and takes the resident multi-GB model with it. Use the native helper for messaging; hold the engine in an offscreen document, persistent page, or dedicated tab. Also verify WebGPU is exposed in a Chrome extension SW at all. |
| **SRI / `verifyIntegrity`** | Self-hosted models are hardened, or corruption is actually observed. `ModelStore.verify()` checks key *presence*; `verifyIntegrity` checks *content* — different problems. Use the native one; do not hand-roll a hash check. |
| **Multi-model via `reload([...])`** | Memory pressure shows up before scheduling contention does. Rejected for now: `reload()` is all-or-nothing, so adding a third model to `{A, B}` reloads A and B too (~51 s each). Additive residency is a hard requirement and only `#pools` provides it. |