Blame for README.md

---
title: Llama Cpp Wasm
emoji: 🐨
colorFrom: green
colorTo: red
sdk: static
pinned: false
custom_headers:
  cross-origin-embedder-policy: require-corp
  cross-origin-opener-policy: same-origin
  cross-origin-resource-policy: cross-origin
---

# llm.js → current llama.cpp (WASM, CPU only)

An experiment: rebuild [rahuldshetty/llm.js](https://github.com/rahuldshetty/llm.js) —
"run an LLM in the browser with llama.cpp + WebAssembly", last touched in 2023 —
against **today's llama.cpp**, keeping the same `LLM` JavaScript API.

Everything here was built and run: `llama.cpp` `f7591a6` (2026-11-07) → 7.1 MB
`wasm32` CPU-only module (chat templates, tools, the PEG output parser and the
`mtmd` multimodal stack linked in), running
[LiquidAI/LFM2.5-VL-450M](https://huggingface.co/LiquidAI/LFM2.5-VL-450M-GGUF)
(`Q4_K_M`, 219 MiB) at ~5.6 tok/s single-threaded in Node, with chat templating
verified against the model's own Jinja template.

No WebGPU, no threads, no `SharedArrayBuffer`: the baseline that works in
browsers that only have plain WASM.

---

## TL;DR

| | llm.js (2023) | this |
|---|---|---|
| llama.cpp | `dd047b4` | `f7591a6a01f9` (2026-11-07) |
| entry point | patch adding `examples/main`, `callMain(argv)` | own C API, `src/llm_wasm.cpp` |
| output | scrape `printf` from stdout + regex filter | `EM_JS` token callback, exact streaming |
| model | GGUF into MEMFS, `-m /models/model.bin` | same (no buffer-load API exists anymore) |
| wasm | `wasm32`, 1 thread, SIMD | same, but now `LLAMA_WASM_MEM64=OFF` explicitly |
| prompt | raw completion only | raw **or** the GGUF's own Jinja template (engine linked in), preset fallback |
| run modes | one text prompt | tabbed demo: chat playground over an editable message list + raw completion |
| sampling | CLI flags | temperature, top_p, top_k, repetition penalty/window, seed, max new tokens, stop sequences |
| tools / reasoning | not supported | OpenAI-style `tools` handed to the template; the reply is split by the model's own PEG parser (tool calls as JSON, reasoning left inline in `think` tags) |
| template params | not supported | `chat_template_kwargs` (e.g. `enable_thinking`) for the Jinja render |
| export | not supported | the playground conversation as OpenAI JSON or Markdown, downloaded |
| images | not supported | an `mmproj` GGUF loaded next to the model; images attach to a message, are tokenized by `mtmd`, and export as OpenAI multi-part content |
| context / KV | `-c` flag, prompt re-evaluated every run | context length, KV type per K/V, unified cache, flash attention as launch params; prompt prefix reused across turns |
| grammar | `--grammar` argv | `llama_sampler_init_grammar` |
| threads | 1 (ggml clamps wasm without pthreads) | optional pthread build: N wasm workers, 8.2 → 26.4 tok/s at 4 threads in Node; single-thread module is the default |

**The concept still works.** What broke are the three things llm.js depended on:
`examples/main` was deleted, CLI-driving (`callMain`) is gone, and the C API was
renamed/reworked (model/context/vocab/sampler/chat). Everything else is small.

---

## Measured results

Run on 4 vCPU, Node 27, single-threaded wasm, `n_ctx = 512`:

```
llama.cpp wasm build: 0.9.0-dev            (f7591a6)
model: LFM2.5-VL-450M-Q4_K_M.gguf (218.7 MiB, arch lfm2, 16 blocks, 65536 vocab)
loaded in 0.4s, n_ctx=512, chat_template=yes
prompt: "Explain in two sentences why WebAssembly is useful for running LLMs in a browser."
tokens: 48
generation: 8.58s (5.59 tok/s)
wasm binary: 7.1 MB (Jinja engine, common/chat, PEG parser, parsers/*, mtmd)
CPU compute buffer reserved for n_batch=512: 138.52 MiB
```

Output (greedy-ish, temp 0.7):

> WebAssembly provides a high-performance, low-level runtime environment that enables efficient and scalable execution of large-scale machine learning models, suc…

Multi-turn chat (`chat: true`) produces the same quality as the native build for
simple user messages, because the prompt is verified token-identical (below).

## Layout

```
llama.cpp/                 shallow clone of ggml-org/llama.cpp (gitignored)
llama.cpp.lock             revision the results below were produced with
src/llm_wasm.cpp           the C API that replaces llm.js' patched CLI
scripts/fetch-llama.sh     clone llama.cpp at the pinned revision (shallow)
scripts/build-wasm.sh      llama.cpp (wasm32, static) + wrapper -> web/wasm/
scripts/fetch-model.sh     download the test GGUF into models/
web/index.html             tabbed demo: chat playground, raw completion, template editor, library
web/llm.js                 LLM class, same API as llm.js
web/worker.js              worker: download -> MEMFS -> wasm -> stream tokens
web/download.js            resumable chunked download (Range + restart fallback)
web/storage.js             IndexedDB model library (uuid records + 4 MiB chunks)
web/sha256.js              streaming SHA-256, for model fingerprints
web/actions.js             worker message ids
web/wasm/                  build output + llm-wasm.build.txt (commit stamp)
test/node-test.mjs         headless inference test
test/download-test.mjs     pause/resume/Range tests for the downloader
test/storage-test.mjs      model library CRUD, chunk iteration, v1 -> v2 migration
test/sha256-test.mjs       streaming SHA-256 vs node's crypto
test/chat-template-test.mjs Jinja template: preset equivalence, override, error fallback
test/chat-generate-test.mjs stepwise generation: streaming, stop sequences, interrupt, limits
test/tool-call-test.mjs    tools in the template, enable_thinking, PEG tool-call parsing, tool turns
test/kv-cache-test.mjs     ctx/KV launch params, quantized caches, prefix reuse across turns
test/multimodal-test.mjs   mmproj load, an image prompt through mtmd, marker/image mismatch
test/browser-test.py       drives the demo page in headless Chromium
test/dump-chat-template.mjs, verify-chat-template.{mjs,py}, verify-tokens.mjs
```

## Build

Requires Emscripten. `emsdk` is expected at `../tools/emsdk` (override `EMSDK=`).

```bash
scripts/fetch-llama.sh         # llama.cpp at the pinned revision (~100 MB, shallow)
scripts/build-wasm.sh          # clean build ~3-4 min on 4 cores
scripts/fetch-model.sh         # 219 MiB
```

The checkout is not committed (it is a nested git repo, ~1 GB of history).
`llama.cpp.lock` records the revision used for the results above;
`build-wasm.sh` fetches it when `llama.cpp/` is missing and warns when your
checkout has moved past it. Updating is expected — llama.cpp changes weekly:

```bash
rm -rf llama.cpp && LLAMA_REF=master scripts/fetch-llama.sh
scripts/build-wasm.sh
node test/node-test.mjs models/LFM2.5-VL-450M-Q4_K_M.gguf "Hi" 32
```

If a future llama.cpp breaks the build or the API, the compile error points at
`src/llm_wasm.cpp`; the porting notes below list what tends to move.

What the script does, and why each flag is there:

```bash
emcmake cmake -S llama.cpp -B build/llama-wasm \
  -DBUILD_SHARED_LIBS=OFF -DLLAMA_BUILD_EXAMPLES=OFF -DLLAMA_BUILD_TOOLS=OFF \
  -DLLAMA_BUILD_TESTS=OFF  -DLLAMA_BUILD_COMMON=OFF  -DLLAMA_CURL=OFF \
  -DLLAMA_WASM_MEM64=OFF           # upstream defaults to wasm64 for EMSCRIPTEN
  -DLLAMA_BUILD_HTML=OFF           # otherwise the target suffix becomes .html
  -DGGML_NATIVE=OFF -DGGML_OPENMP=OFF -DGGML_BACKEND_DL=OFF -DGGML_CPU=ON
  -DCMAKE_C_FLAGS="-O3 -msimd128" -DCMAKE_CXX_FLAGS="-O3 -msimd128"
cmake --build build/llama-wasm --target llama -j$(nproc)

em++ src/llm_wasm.cpp llama.cpp/common/jinja/*.cpp \
  llama.cpp/common/json.cpp llama.cpp/common/unicode.cpp \
  -O3 -msimd128 -fexceptions \
  -I llama.cpp/include -I llama.cpp/ggml/include -I llama.cpp/common -I llama.cpp/vendor \
  -Wl,--start-group $(find build/llama-wasm -name '*.a') -Wl,--end-group \
  -o web/wasm/llm-wasm.js \
  -sDISABLE_EXCEPTION_CATCHING=0 \
  -sMODULARIZE=1 -sEXPORT_ES6=1 -sEXPORT_NAME=createLLMWasmModule \
  -sENVIRONMENT=web,worker,node -sFORCE_FILESYSTEM=1 -sNO_EXIT_RUNTIME=1 \
  -sALLOW_MEMORY_GROWTH=1 -sMAXIMUM_MEMORY=4GB -sINITIAL_MEMORY=64MB -sSTACK_SIZE=8MB \
  -sEXPORTED_FUNCTIONS=... -sEXPORTED_RUNTIME_METHODS='["FS","ccall","cwrap","UTF8ToString"]'
```

Notes that cost time to discover:

* **`em++`, not `emcc`, for the final link.** `emcc` treats `.cpp` as C++ for
  compiling but links without libc++, so every `std::string`/`operator new`
  symbol is undefined.
* **`-DLLAMA_WASM_MEM64=OFF`** — since wasm memory64 landed, llama.cpp's
  `CMakeLists.txt` adds `-sMEMORY64=1` for `EMSCRIPTEN` by default. 64-bit wasm
  memory is not available in every browser/toolchain, and 4 GiB is enough for a
  450M Q4 model, so this build stays on wasm32.
* **`libcommon` is not built** (`LLAMA_BUILD_COMMON=OFF`). `tools/` and
  `examples/` are skipped entirely under EMSCRIPTEN, so the CLI/server path is
  not available at all — the API has to be ours.
* `-msimd128` is what enables `ggml/src/ggml-cpu/arch/wasm/quants.c`; upstream
  also sets it per-target in `ggml/src/ggml-cpu/CMakeLists.txt`.
* **`mtmd` is compiled in** (`tools/mtmd/*.cpp` + `tools/mtmd/models/*.cpp`,
  minus the CLI): upstream's CMake target links `vendor::stb`, miniaudio and
  `vendor::hash`, so the build adds `vendor/hash/hash.cpp` and `sha256.c`. That
  last file is **C**: `em++` would compile it as C++ and mangle `sha256_hash`
  (`wasm-ld: undefined symbol`), so the script builds it with `emcc` into an
  object and links that. `MTMD_VIDEO` stays off (it needs ffmpeg); audio sources
  compile but were not run.
* The build writes `web/wasm/llm-wasm.build.txt` with the exact llama.cpp commit,
  so results stay attributable while upstream moves weekly.

## Threads (optional, cross-origin isolation)

There are two builds of the same sources:

| build | flag | works on | threads |
|---|---|---|---|
| `web/wasm/llm-wasm.*` | `scripts/build-wasm.sh` | any page | 1 (ggml clamps it) |
| `web/wasm/llm-wasm-mt.*` | `MT=1 scripts/build-wasm.sh` | pages with COOP/COEP | 1..N (`llmjs_set_threads`) |

The pthread build needs `SharedArrayBuffer`, which browsers expose only to a
**cross-origin isolated** page. `test/serve.py` is `http.server` plus the two
headers (`Cross-Origin-Opener-Policy: same-origin`,
`Cross-Origin-Embedder-Policy: require-corp`); GitHub Pages and most static
hosts cannot set them, which is why the single-thread module stays the default
and the worker picks the module at runtime:

```js
const THREADED = typeof SharedArrayBuffer === "function" && self.crossOriginIsolated === true;
factory = THREADED ? await import("./wasm/llm-wasm-mt.js") : await import("./wasm/llm-wasm.js");
```

The thread count is a launch parameter like context length or KV type: it goes
into the context signature, so changing it rebuilds `llama_context` (Reload
engine), and it also drives mtmd's image encode/decode (`p.n_threads`).

**Measured in Node** (`test/threads-test.mjs`, LFM2.5-VL-450M Q4_K_M, greedy,
same prompt and output):

```
1 thread:  8.22 tok/s   (2921 ms for 24 tokens)
4 threads: 26.36 tok/s  ( 911 ms for 24 tokens)   → 3.21× wall time
prompt eval: 1605 ms → 1032 ms
```

**Threads are pre-spawned, and that is not optional.** `pthread_create()` called
from inside a wasm call deadlocks in a browser: the calling thread blocks in
`Atomics.wait` while the worker it just asked for has to fetch and boot the same
ES module, and it never gets to report ready. Node is fine because
`worker_threads` start differently. So the module is built with
`-sPTHREAD_POOL_SIZE` set to a JS expression that runs at module init:

```
"-sPTHREAD_POOL_SIZE=${PTHREAD_POOL_SIZE:-Math.max(1, Math.min((typeof navigator !== 'undefined' && navigator.hardwareConcurrency) || 4, 8))}"
```

The pool therefore adapts to the machine that loads the page (capped at 8), and
the thread field is capped to the same number. Choosing fewer threads than the
pool costs nothing at run time — the extra workers sit idle on shared memory and
model load stays ~10 s. Asking for more than the pool size reproduces the old
silent hang, which is why the field can't go above it.

Measured in headless Chromium, 450M Q4_K_M, 8 new tokens, same prompt:

| threads | decode | vs 1 thread |
|---|---|---|
| 1 | 0.98 s, 8.18 tok/s | 1.0x |
| 2 | 0.58 s, 13.74 tok/s | 1.68x |
| 4 | 0.43 s, 18.56 tok/s | 2.27x |

The page now defaults to `min(cores, 4)` threads when the page is cross-origin
isolated, and `?threads=N` still forces a specific count (`?threads=1` for the
old single-thread behaviour). Without COOP/COEP it falls back to the
single-thread module.

Useful facts found while building it:

* `-pthread` on the CMake flags defines `__EMSCRIPTEN_PTHREADS__`, which is what
  stops `ggml_graph_plan()` from forcing `n_threads = 1`.
* the MT build caps memory at 2 GB (safe `SharedArrayBuffer` size across
  browsers) instead of the single-thread build's 4 GB.
* Emscripten warns that `-pthread` + `ALLOW_MEMORY_GROWTH` makes the JS side of
  growth slow; the model load grows the heap once, before generation.
* `vendor/hash/sha256.c` still has to be compiled as C (`emcc`) in both builds.

## Run

Headless (this is what was verified here):

```bash
node test/node-test.mjs models/LFM2.5-VL-450M-Q4_K_M.gguf "Hi" 48
NO_CHAT=1 node test/node-test.mjs ...            # raw completion, no template
CHAT_TEMPLATE=chatml node test/node-test.mjs ... # force a built-in template
```

### Browser demo

```bash
python3 -m http.server 8000 --directory web
```

Then open <http://localhost:8000> (serve it — `file://` cannot load ES modules
or `fetch` the wasm; any static server works, no CORS/COOP headers needed for
the single-threaded build).

The page gives you a small **model library**: any number of models, each with a
uuid record, an editable display name, its size, its SHA-256 and a state
(`complete`, `partial`, `needs file`).

* **from a URL** — 4 MiB chunks into IndexedDB, progress bar, `Pause` (aborts the
  fetch, keeps stored chunks), `Resume` (continues with an HTTP `Range`
  request), `delete` (drops the record + every chunk). If the server ignores
  `Range` and returns `200`, the download restarts cleanly instead of corrupting
  the file. Adding the same URL twice reuses the record instead of storing two
  200 MB copies.
* **from this computer** — a GGUF you already have. By default **nothing is
  copied**: the `File` is handed to the worker and read in 4 MiB slices straight
  into the wasm FS (and the SHA-256), so the browser holds exactly one copy — the
  one already on your disk. After a reload the record reads `needs file` and asks
  you to pick it again; nothing is re-downloaded. Tick *keep a copy in browser
  storage* if you would rather have it live in IndexedDB (still one copy, and the
  original file is not needed afterwards).
* **sha256** — computed once per model, streamed from wherever its bytes live
  (`crypto.subtle.digest()` is one-shot, so `web/sha256.js` is a small incremental
  SHA-256), shown in the row (full value on hover) with a `copy hash` button so it
  can be compared with the publisher's. Paste an expected hash when adding and
  the row marks it ✓ or ≠ expected.
* **pick the model from a dropdown** — the list of library models; `Load` runs the
  selected one (a local file asks for its file again if it was not copied), and
  `Reload engine` frees the model and the wasm backend and loads it once more, so
  switching models or recovering from a bad state does not need a page reload.
* **tabs** — *Chat playground*, *Raw completion*, *Chat template*, *Models*. The
  sampling/limits rail sits next to the two run tabs, so both modes share it.
* **chat playground** — the message list *is* the conversation: add, remove and
  reorder `system` / `user` / `assistant` turns and write them yourself, then
  `Generate`. The assistant turn is appended before the first token and streams
  into place, so a human-written transcript can be continued turn by turn.
  `Stop` in the top bar interrupts and keeps whatever already arrived; the turn
  stays editable afterwards. (Stepwise generation, below, is what makes that
  possible without threads.)
* **tools** — paste an OpenAI `tools` array in the chat tab's `tools` box and it is
  handed to the template for rendering. The reply is then run through the PEG
  parser built for that exact prompt, so a model that answers with
  `<function name="get_weather"><param name="city">Paris</param></function>`
  (MiniCPM5's format) produces a tool-call card: name, id, and the arguments as
  a JSON tree, plus **Add tool response** which appends a `tool` message already
  carrying that `tool_call_id`. The call markup is removed from the transcript
  text, which keeps what the model said before calling. Reasoning is deliberately
  **not** extracted: it stays in the assistant text inside its `think` block,
  exactly as the model emitted it. The JSON tree is a small local renderer, not
  a CDN library, so the lab still works offline.
* **export** — `Export JSON` downloads `[{ role, content }, …]` in OpenAI shape
  (assistant turns keep `tool_calls`, tool turns keep `tool_call_id`, and a
  message with images gets the multi-part content list with
  `{"type":"image_url","image_url":{"url":"data:image/png;base64,…"}}` parts);
  `Export Markdown` downloads the same transcript with `### SYSTEM` / `### USER` /
  `### ASSISTANT` / `### TOOL (call_id)` headers, fenced JSON for calls and a
  literal `<< image >>` where an image was. No conversation library, no cloud:
  it is a lab, and the file is the storage.
* **images** — an uploaded GGUF can be flagged **mmproj** (checkbox in either
  ingest form; it is then left out of the text-model dropdown). In the engine
  rail, tick *load multimodal projector* and pick the projector that is loaded
  with the next model load / `Reload engine`. In the playground, the 🖼 button on
  a message (or a drag & drop onto it) attaches PNG/JPEG/WebP images as
  mini-avatar thumbnails, each with its own ✕ to remove it. The message goes
  through the template with one media marker per image and `mtmd` replaces each
  marker with that image's token sequence; the template preview deliberately
  shows the marker itself, since image tokens sit below the rendered text.
* **raw completion** — no template at all: the text is the prompt, which is what
  a base/completion model expects (and a chat model then answers as plain text).
* **engine** — the launch parameters the rail fixes when a context is created:
  context length, KV cache type for K and V (`f16`, `q8_0`, `q4_0`), flash
  attention (`auto`/`off`/`on`), the unified KV cache, and the optional mmproj. `Reload engine` applies
  a change (a `llama_context` cannot be resized in place); loading a model also
  applies the current values. A quantized V cache switches flash attention on,
  because llama.cpp refuses that combination otherwise. Parallel sequences stay
  at one: extra slots would only duplicate a cache a single user cannot fill.
  This build has no server-style prompt-cache RAM pool (that 8 GiB default is
  `common_params::cache_ram_mib`, used by `llama-server`) — memory is the model
  plus the KV cache, which grows with context length and shrinks with the
  quantized types.
* **sampling / limits** — temperature, top_p, top_k, repetition penalty and its
  window, seed (`-1` = random), max new tokens, and stop sequences (one per line,
  `\n`/`\t` expanded). Stops are matched in C++ against the decoded text, so they
  never appear in the output; the last `len-1` bytes are held back until a match
  is impossible, and the finished text comes from `llmjs_last_output()` instead
  of being reassembled from the stream.
* **chat template** tab — the textarea is prefilled from the GGUF; unchecking
  "use the template shipped in the GGUF" makes it editable and the override wins
  for the next run. A `parameters JSON` box adds template kwargs (e.g.
  `{"enable_thinking": true}`), and the *enable_thinking* checkbox in the rail
  writes that key for you; both the preview and the next run use them. `Preview with the playground messages` renders the current
  transcript without generating — the fastest way to see why a model answers
  badly. The stats line reports which path ran (`template: jinja`,
  `jinja:override`, `builtin:<name>`, `raw`).
* **llama.cpp log** — collapsible, shows the wasm build version and load messages.

The two ingest paths are deliberately separate in the UI (pasting a URL and
picking a file are different gestures, and only the file path has a "copy or
read in place" decision) but converge immediately: one record type, one load
path, one hash, one delete.

Everything stays local: no server-side component, no model upload, and the
cached model is visible/removable from the page (browser quota shown at the top).

`web/llm.js` keeps the original API:

```js
const app = new LLM('GGUF_CPU', modelUrl,
    () => {},                 // init
    (text) => {},             // streaming tokens
    (stats) => {},            // complete: { tokens, ms, tokens_per_second }
    (err) => {},              // NEW optional: error
    (loaded, total) => {},    // NEW optional: download progress
    (line) => {});            // NEW optional: llama.cpp logs
app.load_worker({ context_size: 1024,
                  kv: { type_k: 'q8_0', type_v: 'q8_0', flash_attn: -1,
                        kv_unified: true, n_parallel: 1 },
                  mmproj_id: 'uuid-of-a-flagged-mmproj' });  // optional projector
app.run({ prompt: 'Hi', chat: true, max_token_len: 48, temp: 0.7 });   // old shape, still works

// chat completion over a whole transcript (what the playground sends)
app.run({
  mode: 'chat',
  messages: [{ role: 'system', content: 'You are terse.' },
             { role: 'user', content: 'Hi' }],
  temp: 0, top_p: 0.9, top_k: 40, repeat_penalty: 1.1, repeat_last_n: 32,
  seed: 42, max_token_len: 64, stop: ['\n\n'], chat_template_source: '',
  tools: JSON.stringify(tools),       // OpenAI tools array, handed to the template
  template_kwargs: '{"enable_thinking": true}',
  images: ['data:image/png;base64,…'],  // one per <__media__> marker, in order
});
// stats.parsed = { content, reasoning_content, tool_calls: [{ id, name, arguments }] }
// null when the template's parser did not recognise the reply
// stats.run_stats = { prompt_tokens, reused_tokens, prompt_ms, n_ctx }
app.run({ mode: 'completion', prompt: 'The capital of France is' });
app.stop();   // keeps the tokens already streamed, completes with interrupted: true
```

`run()` is stepped inside the worker (`llmjs_gen_begin_* → llmjs_gen_step →
llmjs_gen_end`): one token per turn of the event loop. That is what allows a
`stop` message to be received at all in a single-threaded build, and it keeps
tokens flowing to the page as they are produced.

Model library — a uuid record per model; url and local-file ingest produce the
same record and load through the same code path:

```js
app.add_url_model({ url, name: 'LFM 450M', expected_sha256: '' });
app.add_file_model({ file, name: '', keep_copy: false });  // nothing copied by default
app.load(id);
app.hash_model(id);
app.delete_model(id);
```

Deviations from llm.js: the GGUF is downloaded **inside the worker** (posting a
219 MB `ArrayBuffer` across the thread boundary would double memory), it is
cached in IndexedDB rather than the Cache API so pauses survive reloads, and the
new callbacks/progress exist because there is no stdout to scrape anymore.
Extra methods: `pause()`, `resume()`, `load(id)`, `add_url_model()`,
`add_file_model()`, `hash_model(id)`, `delete_model(id)`, `list_models()`, plus
an `on_event` hook for worker messages (`MODEL_ADDED`, `MODEL_UPDATED`, `PAUSED`,
`DELETED`, …).

### Embedding the playground (minimal postMessage API)

The page can be used as an iframe by a host page — the host keeps its own UI and
borrows this one's model library and wasm engine:

```html
<iframe src="http://localhost:8010/llmjs-latest/web/" width="900" height="700"></iframe>
<script>
  const frame = document.querySelector("iframe").contentWindow;
  addEventListener("message", (e) => {
    if (e.data.type === "llmjs:ready") backend = e.data;            // model loaded
    if (e.data.type === "llmjs:token") append(e.data.text);        // streaming
    if (e.data.type === "llmjs:done") finish(e.data.text, e.data.stats);
    if (e.data.type === "llmjs:error") fail(e.data.error);
  });
  // after a model is loaded in the frame:
  frame.postMessage({ type: "llmjs:run", id: 1,
    messages: [{ role: "user", content: "Say hi in five words." }],
    max_tokens: 24 }, "*");
</script>
```

Protocol (this iteration: chat completion only):

| direction | message |
|---|---|
| parent → frame | `llmjs:run` `{ id, messages \| prompt, images?, max_tokens?, temperature?, top_p?, top_k?, repeat_penalty?, repeat_last_n?, seed?, stop?, tools?, template_kwargs? }` |
| parent → frame | `llmjs:stop` (keeps what streamed, like the Stop button) |
| frame → parent | `llmjs:api` `{ version }` once the frame is ready |
| frame → parent | `llmjs:ready` `{ model, n_ctx, threads }` when a model finishes loading |
| frame → parent | `llmjs:token` `{ id, text }` per token, `llmjs:done` `{ id, text, stats }`, `llmjs:error` `{ id, error }` |

A model still has to be loaded inside the frame first (that is where the library
and the wasm module live), one run at a time, and the frame accepts messages
from `window.parent` regardless of origin — fine for a local lab, but a public
deployment should check `event.origin` before answering. The host page needs the
frame's URL to be same-origin or CORS/COOP-friendly; the threaded build's
COOP/COEP headers do not leak to the parent page.

---

## What actually broke between llm.js and today's llama.cpp

`llm.js` patched the root `CMakeLists.txt` to add `examples/main` and called
`Module.callMain(["-m", ..., "-p", ...])`, filtering `main:` / `*_model_load:`
lines out of stdout with regexes. Current status of each assumption:

1. **`examples/main` no longer exists.** The patch does not apply; there is no
   CLI to embed. (The current CLI lives in `tools/cli`, is excluded from
   Emscripten builds, and pulls in `common/` + `arg.cpp`.)
2. **`callMain` is the wrong shape anyway.** Token streaming now comes from
   `llama_sampler_sample` → `llama_token_to_piece` → an `EM_JS` callback, so
   output is exact and no regex filtering is needed.
3. **No buffer loading.** `llama_model_load_from_buffer` is gone; only
   `llama_model_load_from_file`, `..._file_ptr`, `..._splits`. So the GGUF still
   goes into MEMFS and is loaded from `/models/model.gguf`.
4. **Renames** the old code hits immediately:
   `llama_new_context_with_model` → `llama_init_from_model`;
   `llama_n_vocab` → `llama_vocab_n_tokens` (vocab is an opaque object);
   `llama_kv_cache_clear` → `llama_memory_clear(llama_get_memory(ctx), true)`;
   `llama_sample_*` → `llama_sampler_*` chain
   (`llama_sampler_chain_init` + `top_k`/`top_p`/`temp`/`penalties`/`dist`);
   `llama_token_to_piece(vocab, tok, buf, len)` gained `lstrip`/`special`;
   `llama_model_params` lost `use_mmap`/`use_mlock` (now `enum llama_load_mode`).
5. **Sampling is explicit.** Nothing is sampled for you: build a chain per run,
   `llama_sampler_sample`, then `llama_sampler_accept` and decode the token.
6. **Threads.** ggml forces `n_threads = 1` for `__EMSCRIPTEN__` without
   pthreads (`ggml/src/ggml-cpu/ggml-cpu.c`), and the wrapper sets
   `n_threads`/`n_threads_batch = 1` so the thread-pool path is never entered.
7. **Chat templating is a different subsystem now** — see the next section.
8. **`--json-schema` is not available** to this build: JSON-schema → GBNF lives
   in `common/json-schema-to-grammar.cpp`, which we do not compile. `grammar`
   (GBNF) works via `llama_sampler_init_grammar`.

---

## The chat-template rabbit hole (verified, not assumed)

This model ships a **Jinja** `tokenizer.chat_template` (ChatML-style:
`<|im_start|>user … <|im_end|>`, `think` blocks, tool calls, image content).
That matters because llama.cpp has *two* templating stacks:

* **`libllama`'s `llama_chat_apply_template()`** — a fixed list of ~60 built-in
  presets (`chatml`, `llama3`, `mistral-v1`, `gemma`, `phi3`, …). It does **not**
  run Jinja. This is all you get if you link only `libllama`.
* **`common/chat.cpp` + `common/jinja` (minja) + the PEG auto-parser**
  (`docs/autoparser.md`, `common/chat-auto-parser.*`, `common/chat-peg-parser.*`)
  — the real Jinja engine and output parser. This is what `llama-cli` /
  `llama-server` use, and it drags in `common/sampling`, `common/json`,
  `json-schema-to-grammar`, `peg-parser`, … i.e. the whole `common` library.

Relevant flags, read from `common/arg.cpp` (not from memory):

| flag | behaviour |
|---|---|
| `--jinja` / `--no-jinja` | enable the Jinja engine (default off) |
| `--chat-template JINJA_TEMPLATE` | custom template; **without `--jinja` only commonly used (built-in) templates are accepted** |
| `--chat-template-file FILE` | same, from a file |
| `--chat-template-kwargs` | template kwargs (e.g. thinking/preserve reasoning) |
| `--skip-chat-parsing` | force pure content parsing even with a Jinja template |
| `--reasoning-format` | how `think` blocks are returned |

**The trap:** passing the model's own Jinja string to
`llama_chat_apply_template()` *aborts the wasm module* with a bare
`RuntimeError: Aborted(undefined)`. Reason, from the source:
`llm_chat_detect_template()` (`src/llama-chat.cpp:89`) does
`try { llm_chat_template_from_str(tmpl) } catch (const std::out_of_range &)`,
i.e. it uses a C++ **exception for control flow**. Emscripten builds disable
exception handling by default, so the `throw` never reaches the `catch` — it
calls `__cxa_throw` → `__Unwind_RaiseException` → `abort()`. Passing the short
name `"chatml"` hits the map directly and never throws. (llm.js never hit this:
it had no template support at all.)

**The fix: link the engine, not the presets.** `common/jinja` is a Jinja engine of
its own (lexer/parser/runtime), and it needs only `common/json.cpp` +
`common/unicode.cpp` next to it — not the whole `common` library (no sampling,
no PEG parser, no autoparser). This build compiles those eight files with
`-fexceptions` and links with `-sDISABLE_EXCEPTION_CATCHING=0`, which is exactly
what makes the `try { … } catch (const std::out_of_range &)` above work instead
of calling `abort()`. So the model's own template is the primary path, rendered
by the same engine `llama-server --jinja` uses:

1. read `tokenizer.chat_template` from the GGUF (`llama_model_chat_template`),
2. render it with `messages`, `bos_token`, `eos_token` and
   `add_generation_prompt` in scope (`jinja::global_from_json` + `jinja::runtime`),
3. strip a leading BOS / trailing EOS when the tokenizer adds them itself —
   the same rule `common_chat_template_direct_apply_impl()` applies, which is why
   the rendered text does not end up with two `<|startoftext|>` tokens,
4. let the caller override the template source entirely (the demo has an
   "advanced · chat template" box for this, handy for a model with a broken or
   missing template),
5. fall back to the closest built-in preset
   (`<|im_start|>` → `chatml`, `<|start_header_id|>` → `llama3`,
   `<start_of_turn>` → `gemma`, `[INST]` → `mistral-v1`, `<|im_sep|>` → `phi3`)
   if the template is missing or throws, and to raw completion if nothing fits.

The chain is reported back to the page: `template: jinja`, `jinja:override`,
`builtin:<name>` or `raw`.

### Verification (byte-level and token-level)

`test/chat-template-test.mjs` renders the model's template **inside wasm** and
compares it with the built-in preset, then tokenizes both:

```
1. GGUF template: 3836 bytes, detected preset 'chatml'
2. Jinja render is byte- and token-identical to builtin chatml
   "<|im_start|>user\nHi<|im_end|>\n<|im_start|>assistant\n"
3. caller-supplied template renders verbatim
4. 4-message conversation renders (168 bytes)
5. broken template reported, module still alive: "Unexpected token: %} of type 9"
6. generated 2 tokens via the Jinja path; broken override fell back to builtin:chatml
```

The older pair below checks the preset fallback against Python's `jinja2`:
`test/verify-chat-template.mjs` renders through the wasm module;
`test/verify-chat-template.py` parses the GGUF metadata directly (small GGUF
reader), renders the *actual* Jinja template with Python `jinja2`, and diffs the
two; `test/verify-tokens.mjs` then proves the token streams are equal.

```
wasm  (built-in chatml): "<|im_start|>user\nHi<|im_end|>\n<|im_start|>assistant\n"
jinja2 (model template): "<|startoftext|><|im_start|>user\nHi<|im_end|>\n<|im_start|>assistant\n"
                                                                  ^ only difference: BOS
TOKENS MATCH: identical token ids      # tokenize(ref, add_special=false) == tokenize(ours, add_special=true)
```

The BOS difference is expected and harmless: the Jinja template emits
`{{- bos_token -}}` itself, while our built-in render relies on the tokenizer,
which adds BOS because `tokenizer.ggml.add_bos_token = true`. Verified equal at
the token level, not argued.

### Tools and the output parser

The same `common/chat.cpp` stack that renders the prompt also **parses** the
reply: for each template it builds a PEG grammar in `common/parsers/*.cpp`
(MiniCPM5's is `minicpm5.cpp`) and `common_chat_peg_parse()` splits the text into
`content`, `reasoning_content` and `tool_calls`. This build compiles that half
too (`chat*.cpp`, `peg-parser.cpp`, `parsers/*.cpp`, `json-schema-to-grammar`,
`reasoning-budget`), which is why the wasm grew from 3.9 MB to 5.1 MB.

Two details that are easy to get wrong, both verified here rather than assumed:

* `common_chat_parser_params` copies the format and generation prompt but **not**
  the PEG arena. `common_chat_peg_parse(arena, input, is_partial, params)` takes
  the arena explicitly; pass `g_chat_params->parser` or everything parses as
  content-only.
* the parser expects the generation prompt in front of the generated text
  (`effective_input.prepend(params.generation_prompt)`), exactly as
  `llama-server` feeds it. Whatever the template appended past the parser's
  literal — an opened `think` block, for instance — lands in the content, so the
  wrapper strips the generation-prompt tail from the start of the content.

Reasoning is not extracted for display: `run()` returns it inside `content`,
`think` tags included, which is what the playground shows. Tool calls are
extracted because they are structured data the page renders as JSON.

Special tokens matter here: the token pieces are decoded with
`llama_token_to_piece(..., special = true)`, because `<function` / `<param` are
special tokens in MiniCPM5's vocab and a `special = false` decode silently drops
them, turning a perfectly good tool call into ` name="get_weather">`.

Multimodal content goes through `mtmd` instead: `llmjs_set_images()` takes the
data URLs, the same `llmjs_gen_begin_chat()` sees the markers in the message
text, and `mtmd_tokenize()` turns each marker into that image's token sequence
before the prompt is evaluated. The projector itself is loaded by
`llmjs_mmproj_load()` (llama-server's `--mmproj`): a second GGUF from the same
library that is flagged `multimodal` by the user, never auto-detected.

---

## Debugging notes (WASM-specific)

* **Exceptions abort silently.** Any `throw` in llama.cpp kills the module with
  `Aborted(undefined)`. Bisect by logging before/after each call from C++ (an
  `EM_JS` log helper works); that is how the chat-template crash above was
  localised to the first `generate` call.
* **`EM_JS` bodies are macro arguments** — a top-level comma breaks the macro.
* **MEMFS + `canOwn`**: `FS.createDataFile("/models", "model.gguf", bytes, true,
  false, true)` takes ownership, so the 219 MB buffer is not copied again.
  Weights still get read into the wasm heap, so the heap grows to a few hundred
  MB on load.
* **Wasm32 vs wasm64**: `-sMAXIMUM_MEMORY=4GB` is the wasm32 ceiling; without
  `ALLOW_MEMORY_GROWTH` the 64 MB initial heap would fail on model load.
* **`HEAPU8`/`UTF8ToString` are not exported by default** — add them to
  `EXPORTED_RUNTIME_METHODS` if JS needs them.
* **Single-thread `n_batch=512` reserve** costs ~139 MiB of compute buffer. Lower
  `n_batch` if the browser is memory-constrained.

---

## Tests / what is verified

| command | needs | checks |
|---|---|---|
| `node test/node-test.mjs [model] [prompt] [n]` | model | full wasm load + generation, tok/s, output |
| `node test/download-test.mjs` | nothing | chunking, pause at chunk boundary, `Range` resume, restart when the server ignores `Range` |
| `node test/sha256-test.mjs` | nothing | streaming SHA-256 vs node's `crypto`: every padding boundary, uneven slices, known vectors |
| `npm i --no-save fake-indexeddb && node test/storage-test.mjs` | fake-indexeddb | model library CRUD, chunk iteration, v1 → v2 migration |
| `node test/chat-template-test.mjs [model]` | model | GGUF Jinja template vs the verified preset (byte + token identical), custom override, broken template falls back instead of aborting |
| `node test/chat-generate-test.mjs [model]` | model | stepwise chat generation over a multi-turn list, streamed text == final output, greedy reproducibility, stop-sequence truncation, interrupt keeps the partial text, raw mode, max-new-tokens |
| `node test/tool-call-test.mjs [model]` | MiniCPM5-1B | tools rendered into the prompt, `enable_thinking` changes it, a canned and a **live** reply parse into name + JSON arguments, text before the call is kept, assistant `tool_calls` + `tool` result render back |
| `node test/kv-cache-test.mjs [model]` | model | ctx/KV options reported by `llmjs_ctx_config()`, repeated-prompt reuse (6.5 s → 0.3 s), reply reuse on the next turn, identical output with a cold cache, quantized K/V + flash attention rule, `n_ctx` really applied; adaptive for hybrid/RNN memories |
| `node test/multimodal-test.mjs [model] [mmproj]` | LFM2.5-VL + mmproj | projector reports vision support and the media marker, an image prompt evaluates its image tokens (121 prompt tokens vs 19 for the same text) and answers about the picture, one image per marker is enforced, image runs are refused without a projector while text still works |
| `node test/threads-test.mjs [model] [threads]` | mt build | the pthread module really runs N threads: identical greedy output, measured tok/s at 1 vs N (3.2× at 4 threads on 4 vCPU) |
| `python test/serve.py` | nothing | static server with COOP/COEP, needed for `SharedArrayBuffer` / `--threads` |
| `node test/verify-chat-template.mjs … && python3 test/verify-chat-template.py …` | jinja2 | built-in `chatml` render vs the model's real Jinja template |
| `node test/verify-tokens.mjs …` | model | the two prompts tokenize to identical ids |
| `python test/browser-test.py --model-url … \| --file … [--mmproj …]` | playwright | the real page in headless Chromium: both ingest paths → sha256 → load → generate; `--pause-mib N` pauses/resumes, `--file` asserts nothing was copied, plus a `tool` row, both exports (downloads are parsed back) and a template kwarg reaching the preview; `--mmproj` adds a flagged projector, reloads with it, attaches an image (thumbnail add/remove), generates about it, checks the preview marker and the multi-part JSON / `<< image >>` Markdown exports |

Verified here: clean wasm build, 48-token generation, raw + chat paths, prompt
render byte diff, token-id equivalence, download pause/resume, the model library
(uuid records, rename, multi-model delete, v1 → v2 migration, chunk iteration),
streaming SHA-256 against node's `crypto`, HTTP serving of every demo asset, and
the demo page driven end to end in headless Chromium over **both** ingest paths
(download: 218.7 MiB into 55 chunks, `GGUF` magic intact, paused and resumed via
`Range`; local file: hashed and loaded with zero chunks written to IndexedDB).

KV reuse and the launch parameters were verified with `test/kv-cache-test.mjs`
on MiniCPM5-1B (transformer) and LFM2.5-VL-450M (hybrid): a repeated 27-token
prompt went from 6.5 s to 0.3 s of prompt eval with 26/27 tokens reused, a
cleared cache gave byte-identical greedy output, `q8_0` K/V ran with flash
attention forced on, `q8_0` K with `f16` V kept it off, and `n_ctx=256` was
really applied and invalidated the old cache. On LFM2.5 the hybrid memory reused
32/46 tokens on the next turn (append-only, as expected).

The tabbed page was verified the same way: the chat playground appends an
assistant turn and streams into it (`stats: 24 tokens … · template: jinja`),
`Stop` reported `· stopped` with the partial text kept, the template tab previews
the GGUF template (`<|im_start|>user … <|im_end|>`), an override was used
(`template: jinja:override`), raw completion reported `template: raw`, andthe engine reloaded from the top bar. A `tool` message row survived with its
`tool_call_id`, the JSON export came back as OpenAI messages ending in that tool
turn, the Markdown export contained `### TOOL (call_test_1)` and `### USER`, and
the template preview accepted a `parameters JSON` kwarg.

Multimodal was verified with LFM2.5-VL-450M plus its
`mmproj-LFM2.5-VL-450m-Q8_0.gguf` (same Hugging Face repo, `mmproj` file): in
Node the projector loaded (`vision=true`, marker `<__media__>`), a 64×64 PNG
prompted with one marker evaluated 121 prompt tokens against 19 for the same text
and the model answered *"The left half of this image is red"*; a marker/image
count mismatch was refused (`-2`) instead of mis-rendering, and an image run
without a projector was refused (`-3`) while text runs kept working. In the
browser the same flow used the real page: flag the file as mmproj, reload with
*load multimodal projector* (status line gains `mmproj … (vision)`), attach the
image through the 🖼 button, remove and re-attach the thumbnail, generate
(*"The left half of this image is a red color."*, 121 prompt tokens in 24 s —
image encoding dominates), see `<__media__>` in the template preview, and export
the conversation: one multi-part message with
`data:image/png;base64,…` in JSON and `<< image >>` in Markdown.

Tools and the parser were verified with MiniCPM5-1B (`MiniCPM5-1B-Q4_K_M.gguf`)
in `test/tool-call-test.mjs`: the tools array reaches the template (943 vs 90
bytes), `enable_thinking` toggles the open `think` block, and a real 15-token
generation produced `<function name="get_weather"><param name="city">Paris</param></function>`,
which the model's own parser turned into one tool call with
`{"city": "Paris"}`.

## Docs consulted (both projects)

**llm.js** — `docs/BUILD.md` (build flow), `docs/api_guide.md` (the API preserved
here), `scripts/build-llama-cpp-wasm.sh` + `patches/00-llama-cpp-enable-main.patch`
(exactly what broke), `src/web-workers/llamacpp-cpu-worker.js` (stdout scraping).

**llama.cpp** — the docs are scattered; these were the useful ones:
`docs/build.md` (WASM/WebGPU + WASI sections, the latter has the
single-thread `-DGGML_OPENMP=OFF -DLLAMA_BUILD_COMMON=OFF` recipe),
`.github/workflows/build-wasm.yml` (the flags upstream actually builds with),
`docs/autoparser.md` (PEG auto-parser design), `docs/models.md` /
`models/templates/README.md` (templates), `tools/mtmd/README*.md` (multimodal),
`docs/function-calling.md`, plus source-of-truth `include/llama.h`,
`src/llama-chat.cpp`, `common/arg.cpp`, `ggml/src/ggml-cpu/arch/wasm/` and
`ggml/src/ggml-cpu/ggml-cpu.c`.

## Limitations / next steps

* **Multimodal is vision-projector only, and untested beyond images.** `mtmd`
  is compiled in and loads any `.gguf` mmproj, but only LFM2.5-VL's vision
  projector was run; audio projectors compile and were never tried, and video is
  off (it needs ffmpeg). Images are CPU-bound here: the 64×64 test image took
  ~12 s to encode and ~6 s to decode in wasm, so a photo costs tens of seconds
  before the first token. Image runs bypass the text prefix cache (the token
  stream changes with the image), so a subsequent text turn starts cold.
* **No `json_schema`** (compiling `common/json-schema-to-grammar.cpp` would add
  it). GBNF `grammar` works.
* **No reasoning-effort control.** MiniCPM5 advertises
  `supports_reasoning_effort: false`; thinking is on/off via her
  `enable_thinking` template kwarg only. The sampling stack with thinking budgets
  is not compiled here, so the control is left out rather than faked.
* **Stop granularity is one token**, and stop sequences are text-level: they are
  matched on decoded text (that is why a stop can appear immediately, not after
  the next token). There is no token-id ban list yet.
* **Prefix reuse depends on the template and the memory type.** The wrapper
  keeps the tokens that sit in the KV cache and re-evaluates only what a new
  prompt adds. A repeated prompt drops from 6.5 s to 0.3 s of prompt eval
  (26/27 tokens reused, `test/kv-cache-test.mjs`). Two limits are inherent: a
  reasoning template injects its `think` block only in the generation prompt, so
  with thinking on the reply after that point cannot be reused; and recurrent /
  hybrid memories (Mamba, LFM2's conv+attention) cannot roll their state back,
  so they only reuse the append case — never a prefix that needs trimming. That
  case is the multi-turn one, and it still helps: 32/46 tokens on LFM2.5.
* **KV memory is only bounded by `n_ctx`.** There is no separate cache pool in
  this build (the 8 GiB `--cache-ram` default belongs to `llama-server`); a
  context of N tokens costs the model's per-token KV size times N, times two if
  K and V stay `f16`. The quantized types in the rail are the lever.
* **The parser only knows the formats llama.cpp ships** (`common/parsers/`). A
  model whose reply format has no parser there still generates fine; its reply
  just stays plain text (`stats.parsed` is `null` or has no tool calls).
* **Exceptions cost size and some speed**: linking the Jinja engine raised the
  wasm from 3.2 MB to 3.9 MB and requires `-fexceptions`
  `-sDISABLE_EXCEPTION_CATCHING=0`; a browser without wasm exception support is
  not a concern (Emscripten's scheme is JS-based), but the JS glue is larger and
  every `try/catch` in llama.cpp now works instead of aborting.
* **Threads are opt-in and experimental in the browser.** A pthread build
  exists and is 3.2× faster in Node at 4 threads (`test/threads-test.mjs`), but
  the first multi-thread decode stalls in headless Chromium, so the page keeps
  one thread by default and `?threads=N` turns the pthread module on. Threads
  also need cross-origin isolation, i.e. host-controlled COOP/COEP headers.
  Wasm SIMD is already enabled; the single-thread baseline stays ~8 tok/s on
  this container.
* **A local model has to be re-picked after a reload** (one click, no network)
  unless you tick *keep a copy in browser storage*. Keeping a
  `FileSystemFileHandle` in IndexedDB would remove that click, but the API is
  Chromium-only and brings a permission prompt, so it is left out for now.
* **Hashing costs one read of the model** (~2-4 s for 219 MiB here) and happens
  once, on add; the digest is cached in the record.
* **One browser only** — the demo is checked in headless Chromium via
  Playwright (`test/browser-test.py`); Firefox and Safari were not run, and the
  IndexedDB quota/eviction behaviour on a nearly-full disk was not exercised.
* Similar, more actively maintained projects to compare against: `wllama`
  (ngxson) and llama.cpp's own WebGPU example.

## This bundle

`2de6f90` — engine sampler rebuilt from the chat template, LFM2.5 tool-call parsing,
image prompts that grow the context to fit, and structured output (JSON Schema or GBNF)
selected from the sidebar. Serve with COOP/COEP so the threaded module is used.