README.md

---
license: isc
library_name: everything-webgpu
pipeline_tag: text-generation
tags:
  - webgpu
  - mlc-ai
  - web-llm
  - browser
  - on-device
  - local-llm
  - ghost-text
  - scheduler
  - typescript
  - javascript
---

# everything-webgpu

[![npm version](https://img.shields.io/npm/v/everything-webgpu.svg?color=cb3837)](https://www.npmjs.com/package/everything-webgpu) [![GitHub](https://img.shields.io/badge/GitHub-nyaaorick%2Feverything--webgpu-black?logo=github)](https://github.com/nyaaorick/everything-webgpu) [![WebGPU](https://img.shields.io/badge/Platform-WebGPU-orange.svg)](https://developer.mozilla.org/en-US/docs/Web/API/WebGPU_API) [![License: ISC](https://img.shields.io/badge/License-ISC-blue.svg)](https://opensource.org/licenses/ISC)

Run local MLC/WebLLM models on WebGPU, in a browser, with a scheduler in front of them.

**Migrating off `@mlc-ai/web-llm` is one line.**

```diff
-import { CreateMLCEngine } from "@mlc-ai/web-llm";
-const engine = await CreateMLCEngine(modelId, { initProgressCallback });
+import { CreateScheduledEngine } from "everything-webgpu";
+const engine = await CreateScheduledEngine(modelId, { initProgressCallback });
```

Everything after that line is unchanged. `engine.chat.completions.create()` takes and returns the
same shapes, streams the same chunk objects, and reports the same finish reasons. That call is the
compatibility layer and **it never changes** — see [Stability](#stability).

What the swap buys:

| | |
| --- | --- |
| **A scheduler** | `session`, `priority`, `task`, `preemptible` on any request. One engine per task, pool grows on demand, stale requests supersede themselves. |
| **Multi-step decoding** | K forward steps per GPU sync. On the 0.8B baseline: 9.6 → 25.9 tok/s. |
| **Two build-time patches** | Firefox's 9-storage-buffer cap, and one compute pass per flush instead of per kernel. |
| **Three model sources** | A prebuilt id, a URL you host, or **a folder off disk with no network at any point**. |
| **Typed errors** | Eight codes, each one because a caller does something different about it. |

---

## Status

**On npm** — `npm i everything-webgpu`. Pre-1.0, ESM only (`"type": "module"`).

```sh
npm i everything-webgpu
```

`vendor/web-llm.js` — the patched WebLLM bundle the engine loads — is a **build product and is not
checked in**, so every install route has to produce it. The published tarball ships it prebuilt;
`prepare` regenerates it for a git dependency or a clone. If it is ever missing you get
`PACKAGE_INCOMPLETE` naming the fix, rather than a module-resolution error pointing inside this
package.

Requires WebGPU. Verified end to end on Firefox 154 / macOS / M4; on Firefox you may need to set
`dom.webgpu.enabled` and the related flags first — [AI.md](AI.md) lists them and says what each one
does. Chrome is expected to be faster (it does not hit the storage-buffer cap that disables KV
reuse) but that is a prediction, not yet a measurement.

## Quick start — four lines

```js
import { CreateScheduledEngine } from "everything-webgpu";                        // 1. import
const engine = await CreateScheduledEngine("Llama-3.2-1B-Instruct-q4f16_1-MLC");  // 2. load a model
const reply = await engine.ask("Name three primary colours.");                    // 3. ask
console.log(reply);                                                               // 4. the answer
```

`reply` is a plain string. Line 2 downloads ~0.8 GB the first time and prints throttled progress to
the console; pass `initProgressCallback` to render it yourself, or `initProgressCallback: null` to
silence it. On Vite, add one plugin — see [API.md § Bundlers](API.md#bundlers). Every other call
form — `conversation()`, `load()` from a URL or off disk, streaming, embeddings, the OpenAI shape —
is catalogued in **[API.md](API.md)**.

The WebLLM-compatible shape, unchanged:

```js
const reply = await engine.chat.completions.create({
  messages: [{ role: "user", content: "hi" }],
  session: "ghost-text",     // added by this engine
  priority: "interactive",   // added by this engine
});
console.log(reply.choices[0].message.content);
```

Streaming is WebLLM's, passed through untouched:

```js
const stream = await engine.chat.completions.create({ messages, stream: true });
for await (const chunk of stream) process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
```

The first `load()` downloads the weights; after that it is cache-only and needs no network.

### Runnable

[examples/](examples/) has three, each a standalone project that consumes this package through its
`exports` map rather than reaching into the source tree:

```sh
cd examples/bare && npm install && npm run dev
```

| | |
| --- | --- |
| [bare/](examples/bare/) | Vanilla Vite, one page. ~100 lines. |
| [react/](examples/react/) | A chat UI over `conversation()`. |
| [webext/](examples/webext/) | MV2 Firefox — `browser.storage.local` as the store, and the CSP a wasm runtime needs. |

## What it costs your bundle

**53 kB** (~19 kB gzip) in the entry chunk, before a model is loaded. WebLLM's 6 MB bundle is a
**lazy chunk** — fetched on the first `load()` or `listAvailableModels()`, and never by a visitor
who does neither, so a page can decide whether this machine has a GPU before paying for it. The
IndexedDB adapter is another 0.8 kB lazy chunk, absent entirely if you pass your own store.

On Vite, add the plugin this package ships:

```js
import { everythingWebGPU } from "everything-webgpu/vite";
export default defineConfig({ plugins: [everythingWebGPU()] });
```

It exists for one reason: Vite's dependency pre-bundler otherwise rewrites the
decode worker's URL to a path that 404s — in `vite dev` only, and only on a real
install, which is why no example in this repo can catch it and
`npm run verify-consumer` exists instead. Skip it and `load()` fails with
`PACKAGE_INCOMPLETE` naming the fix, rather than hanging.

## Three ready-made shapes

Most apps want one of three things. Each is one call, with the scheduling already right:

```js
// 1. One question, nothing kept.
const answer = await engine.ask("Summarise this in one line:\n" + doc);

// 2. A conversation that remembers.
const chat = engine.conversation({ system: "You are terse." });
await chat.say("what's the capital of France?");
await chat.say("and its population?");        // remembers

// 3. Ghost text, debounced and superseding.
const ghost = engine.ghostText({ prompt: (before) => `Continue:\n${before}` });
editor.on("input", async () => {
  const hint = await ghost.suggest(editor.textBefore());
  if (hint !== null) render(hint);            // null = a newer keystroke won
});
editor.on("blur", () => ghost.cancel());
```

`complete()` can express all three. These exist because the **scheduling** is the part that's easy
to get wrong and invisible when you do — every row of [Getting these wrong](#getting-these-wrong) is
a scheduling mistake, not a generation one. So each verb is a policy:

| | policy |
| --- | --- |
| `ask()` | its own task, no session — two in flight never supersede each other |
| `conversation()` | one stable task for every turn, turns serialised, history bounded |
| `ghostText()` | debounce + one session key + `interactive` + **resolves `null` when stale** |

That last one is the difference between a verb and a wrapper. The engine already drops superseded
work; what a caller still had to remember was not to *paint* the answer that came back anyway.
Returning `null` removes the choice.

**They author no prompts.** `ask()` and `conversation()` carry your text through; `ghostText()`
*requires* a `prompt` function and has no default. Prompts are model-specific — switching this
project's own build from a 0.8B to a 2B changed the conversation template and made every reply open
with a `<think>` block. A prompt that lives in your code survives that.

`conversation()` bounds history at 12 exchanges by default, because there is no cross-turn KV reuse
here: every turn re-prefills the whole history at ~5.27 ms/token, so an unbounded conversation gets
quadratically slower and a turn near the context limit waits ~22 s for its first token. Pass
`keep: Infinity` to opt out, having read that sentence.

## Where models come from

**One call covers all three.** `load()` works out what you handed it, registers whatever needs
registering, and brings the model up:

```js
// 1. Prebuilt — one of WebLLM's 163 HuggingFace models, or anything you registered earlier.
await engine.load("Llama-3.2-1B-Instruct-q4f16_1-MLC");

// 2. Remote — any base URL you host. `modelLib` is required; see below.
await engine.load("https://cdn.example/models/my-model/", {
  modelLib: "https://cdn.example/models/my-model/my-model-webgpu.wasm",
});
await engine.load({ model: "/models/my-model/", modelLib: "/models/my-model/lib.wasm" });

// 3. Injected — a folder off disk. No network at any point.
await engine.load(dropEvent.dataTransfer);      // a drop
await engine.load(input.files);                 // <input webkitdirectory>
await engine.load({ files });                   // already-unpacked { path, file }[]
```

The model id is derived from the URL's last segment; pass `{ id }` to override it. Add
`{ defer: true }` to register a source **without** building a pool — it returns the registry record,
for a drop-now-load-later flow. `registerModel()` and `ingestModelFolder()` remain as the low-level
primitives; `load()` composes them rather than replacing them.

`listAvailableModels()` enumerates all three routes. Construct the engine with `{ prebuilt: false }`
for a build that must never fetch a model over the network. A misspelled id is told what it might
have meant.

### Warming the cache early

```js
await engine.prefetch(modelId, { onProgress: (p) => bar(p.done / p.total) });
```

Downloads the model **without building an engine and without WebGPU** — so an app can warm the cache
while the user is still reading the welcome screen, before it has decided whether this machine can
run the model at all. WebLLM cannot express this: `reload()` instantiates the wasm and needs a GPU
before it fetches a single shard. A later `load()` is then a cache read. Interrupted downloads
resume, and a second call is free.

## Embeddings

```js
await engine.load("snowflake-arctic-embed-s-q0f32-MLC", { keepResident: true });

const [q] = await engine.embed("how do I cancel a job?");
const docs = await engine.embed(paragraphs);   // one vector per input, in order
```

Embedding models are separate models — WebLLM ships four `snowflake-arctic-embed-*` from 239 MB — so
this usually names `modelId` and holds it resident alongside a chat model. Embeddings go through the
**same scheduler** as completions: same priority bands, same `session` supersession, same
one-task-one-engine rule. `embedRaw()` returns WebLLM's OpenAI-shaped envelope if you are porting
code that expects `data[].embedding`.

One difference worth knowing: **a running embedding cannot be interrupted.** Cancellation works by
making a decode loop break out, and one forward pass has no loop — so a `cancel()` landing after the
job starts marks it cancelled without stopping it. Queued embeddings supersede normally.

**`modelLib` is required for a remote source and is not guessed.** It is genuinely underivable:
across all 163 prebuilt models, zero have a lib name derivable from the model id, and zero host the
lib on the same origin as the weights. A guess would be wrong every time and would surface as a
confusing 404 deep inside the loader, so the engine asks instead.

## The three shapes of work

What differs between these is *not* the call or the transport. It is **who owns an engine, and what
may interrupt what.**

| | call | priority | key field | why |
| --- | --- | --- | --- | --- |
| **Completion** (ghost text) | `complete` streaming | `interactive` | `session` | Each keystroke supersedes the last request; may preempt opted-in work. |
| **Translation** (a page) | `batch` | `normal` | one shared `task` | One request instead of N, so the engine schedules it as a unit and it never hogs the pool. |
| **Reformat** (markdown) | `complete` | `background` | `preemptible: true` | Nobody is watching; let interactive work cut in. |

### Completion — latency is the whole product

```js
// On every keystroke. The previous request is cancelled, not queued behind.
const { text } = await engine.complete(
  {
    messages: [{ role: "user", content: prefix }],
    session: "ghost-text",     // supersession key — the important field
    priority: "interactive",   // may preempt jobs that opted in
    max_tokens: 24,            // ghost text is short; do not pay for more
  },
  (delta) => render(delta),
);
```

**`session` is what makes this work, not `cancel`.** Reusing one session key means the engine drops
the stale request itself. A caller that mints a fresh id per keystroke and calls `cancel` races its
own typing.

### Translation — throughput, one task

```js
// One batch, not a loop of `complete` calls.
const results = await engine.batch({
  task: "translate-page",   // optional; a batch is one task either way
  requests: sentences.map((s) => ({
    messages: [{ role: "user", content: `Translate to French, output only the translation:\n${s}` }],
  })),
});
results.forEach((r) => apply(r.index, r.text));
```

Every item of one batch shares a task, and a task holds one engine, so a 200-sentence page occupies
exactly one engine and can never freeze ghost-text behind it. Items carry `engineIndex`, `startedAt`
and `finishedAt`, so you can check what actually ran where.

`batch` stays the right call rather than a loop: it is one scheduling unit the engine can reason
about, and when batched decode lands the same call gets faster with no change on your side.

### Reformat — cheap to interrupt

```js
await engine.complete({
  messages: [{ role: "user", content: `Reformat as clean Markdown, no commentary:\n\n${doc}` }],
  priority: "background",
  preemptible: true,        // the direction matters — see below
  max_tokens: 2048,
});
```

**Set `preemptible` on the work that can afford to lose, not on the work you care about.** Only an
`interactive` request preempts, and only a job that opted in can be preempted. A preempted job
resolves with `preempted: true` and whatever text it had — never requeued, so it can never starve,
but you must be able to use or discard a partial result.

## Getting these wrong

| symptom | cause |
| --- | --- |
| Ghost text lags behind typing | Fresh `id` per keystroke with no `session`, so every stale request still runs. |
| Page translation is slower than expected | Expected: one task is one engine, and a second engine measured 1.06x anyway. Throughput here comes from batched decode, not from more engines. |
| Reformatting blocks completions | `preemptible` left off the background job, so `interactive` has nothing to take. |
| Pool stays at one engine | Expected: it grows only when a *second task* waits. Check `engine.state.pool.growthBlocked` if two are waiting and it still has not. |

## Prompts stay with you

The engine does not author prompts. They are model-specific — switching one build from
`Qwen3.5-0.8B` to `Qwen3.8-2B-Distill` changed the conversation template and made every reply open
with a `<think>` block. A prompt that lives in the caller survives that; a `translate` op baked into
the engine would have to be rewritten and re-shipped to every caller.

## When it does not work

Every failure carries a code. `message` stays human-readable and stays the thing you print; `detail`
carries structured context so you never parse the sentence.

```js
import { isEngineError, ERROR } from "everything-webgpu";

try {
  await engine.load(id);
} catch (err) {
  if (isEngineError(err, ERROR.CACHE_INCOMPLETE)) return reRegisterFolder();
  throw err;
}
```

| code | what to do about it |
| --- | --- |
| `NO_WEBGPU` | Tell the user to check flags/hardware; retrying is futile. |
| `NO_MODEL` | Nothing registered at all — send them to your setup flow. |
| `UNKNOWN_MODEL` | That id is not resolvable; `listAvailableModels()` says what is. |
| `CACHE_INCOMPLETE` | A locally-registered model was evicted; re-register the folder. |
| `INVALID_MODEL_FOLDER` | Not a compiled MLC model; `detail` says what is missing. |
| `BAD_REQUEST` | The caller's arguments are wrong — a bug in the caller. |
| `ABORTED` | The caller cancelled it. Not a failure; do not report it as one. |
| `GENERATION_FAILED` | The model failed mid-generation. |
| `PACKAGE_INCOMPLETE` | Your **build** is wrong, not your code — a missing `vendor/` bundle or a decode worker the bundler did not emit. `message` names the fix; `detail.cause` says which. |

### Ask before you download, not after

`environment()` is the preflight: one call that says what this machine will do, and why.

```js
const report = await engine.environment();
if (!report.ok) console.error(report.lines[0].fix ?? report.lines[0].cause);
```

Every line carries the same five fields, so you can render the whole report without special-casing
any of it:

| field | |
| --- | --- |
| `severity` | `blocked` · `degraded` · `tune` · `info` · `ok` — lines come sorted worst-first |
| `affects` | what you lose, in your terms |
| `cause` | the measured fact behind the verdict |
| `fix` | the exact call to make, or `null` when nothing can be done |
| `operable` | whether this is reachable from JS at all |

`fix` is `null` for hardware, build-time flags, and browser settings JS cannot reach. Reporting a
consequence with no remedy is still the point: *"your second turn is slow because this device caps
storage buffers at 9"* is the difference between a bug report and an informed decision.

```js
await engine.environment({ scope: "local" });  // never touches the model layer — cheap, poll freely
await engine.environment({ scope: "device" }); // hardware only
await engine.environment.measure();            // one calibration generation → measured tok/s
```

**`environment()` only reports.** Writes go through `configure()`, and passing a setting to
`environment()` is an error that names the call you wanted — one function doing both, told apart by
argument shape, is how you get a call that silently does nothing.

```js
await engine.configure({ decodeSteps: 8 });   // hot, no reload
await engine.configure({ engineCount: 3 });   // persisted; applies to pools built after it
```

`canRun(modelId)` stays separate and answers the per-**model** question — will this one fit and run
here — where `environment()` is per-**device**.

## Freeing memory

A resident model is a full copy of its weights in VRAM, and nothing reports free VRAM to a page — so
residency is explicit rather than guessed at.

```js
await engine.unload();               // the current model's VRAM; cached bytes stay, so reloading is free
await engine.unload(id);             // that model's VRAM
await engine.unload(id, "cache");    // and delete its cached bytes, keeping the registry entry
await engine.unloadAll();            // every resident model
await engine.remove(id);             // bytes and entry; for an injected model, means re-supplying the folder
```

A bare `unload()` frees **only the current model** — `unloadAll()` is the explicit form, because
freeing everything is the more destructive reading and should have to be asked for by name.
`remove()` keeps its own verb for the same reason: it is the one that cannot be undone.
`evict(id)` remains as the low-level primitive `unload(id, "cache")` is built on.

`load()` unloads whatever else is up before bringing a model in, which is the safe default on a
16 GB machine. Pass `{ keepResident: true }` to hold two at once, having checked the budget with
`canRun()` first. `cacheState(modelId)` says what is already on disk.

## Choosing a store

`CreateScheduledEngine` defaults to IndexedDB, because a registry that dies with the page would
strand the weights — the bytes stay in Cache Storage but nothing remembers they are there. Pass one
explicitly in a worker or a test:

```js
import { ScheduledEngine, ModelStore } from "everything-webgpu";
import { memoryStorage } from "everything-webgpu/adapters/memory";

const engine = new ScheduledEngine({ store: new ModelStore(memoryStorage()) });
```

Adapters ship for `idb` (pages, plus `ensurePersistent()`), `memory` (tests), and `webext`
(`browser.storage.local` and the message/port router).

## Measured

M4 MacBook Air (16 GB), Firefox 154 release, macOS. `Qwen3.8-2B-q4f16_1` (1.06 GB) shipping,
`Qwen3.5-0.8B-q4f16_1-MLC` (443 MB) as the baseline most of the analysis was done on.

| | 2B | 0.8B |
| --- | --- | --- |
| Model load, cache only, zero network | 51 s | 48 s |
| Prefill | 48 tok/s short, 100–200 at length | 95–98 tok/s |
| **Decode** | **16.6–18.1 tok/s** | **25.9 tok/s** |
| Decode, stock single-step WebLLM | — | 9.6 tok/s |
| Second engine, 4-prompt batch | 1.06x | — |

Reproduce with `npm run e2e`. The full analysis — where every millisecond goes, why a second engine
buys isolation rather than throughput, and what is still open — is in [AI.md](AI.md).

## Stability

`chat.completions.create()` is the compatibility layer. **It does not change**, and any change to
its request or response shape is a major version. That is the whole point of the one-line migration:
if it drifted, the line would not be one line.

Everything else is pre-1.0 and may move. The ergonomic verbs (`load`, `unload`, `environment`) are
being consolidated — see [ROADMAP.md](ROADMAP.md).

## Docs

| | |
| --- | --- |
| [API.md](API.md) | Every way to call it — one page, asserted against the code. |
| [examples/](examples/) | Three runnable projects. Read one before reading the source. |
| [AI.md](AI.md) | What is true and measured. The reference. |
| [ROADMAP.md](ROADMAP.md) | The only list of open work. |
| [ARCHIVE.md](ARCHIVE.md) | What was done and *why* — decisions with their reasoning. |
| [WEBLLM-SURFACE.md](WEBLLM-SURFACE.md) | What WebLLM already does, where our line is, and the dependency-bump runbook. |
| [MLC-COMPILE.md](MLC-COMPILE.md) | How the model was compiled, and every toolchain breakage on the way. |

## Development

```sh
npm run build            # bundle WebLLM + apply the two patches
npm test                 # contract, patches, README, examples, scheduler, multistep — GPU-free
npm run e2e              # real Firefox + real GPU + real model
npm run verify-consumer  # pack + install for real; proves the Vite plugin is needed and works
npm run verify-patches   # check patch anchors without rebuilding
```

`@mlc-ai/web-llm` is pinned exactly, because the build rewrites its bundle at verified anchors.
Bumping it is a documented, guarded procedure — see WEBLLM-SURFACE.md, "Upgrading".

## Licence

This package is **ISC** ([LICENSE](LICENSE)). It ships a pre-built copy of WebLLM
(`vendor/web-llm.js`, **Apache-2.0**) and its dependency `loglevel` (**MIT**); their full licence
texts travel in [THIRD-PARTY-NOTICES.md](THIRD-PARTY-NOTICES.md), and
[test/license.test.mjs](test/license.test.mjs) fails the build if a dependency is ever bundled
without one.