--- title: Llama Cpp Wasm emoji: ๐Ÿจ colorFrom: green colorTo: red sdk: static pinned: false custom_headers: cross-origin-embedder-policy: require-corp cross-origin-opener-policy: same-origin cross-origin-resource-policy: cross-origin --- # llm.js โ†’ current llama.cpp (WASM, CPU only) An experiment: rebuild [rahuldshetty/llm.js](https://github.com/rahuldshetty/llm.js) โ€” "run an LLM in the browser with llama.cpp + WebAssembly", last touched in 2023 โ€” against **today's llama.cpp**, keeping the same `LLM` JavaScript API. Everything here was built and run: `llama.cpp` `f7591a6` (2026-11-07) โ†’ 7.1 MB `wasm32` CPU-only module (chat templates, tools, the PEG output parser and the `mtmd` multimodal stack linked in), running [LiquidAI/LFM2.5-VL-450M](https://huggingface.co/LiquidAI/LFM2.5-VL-450M-GGUF) (`Q4_K_M`, 219 MiB) at ~5.6 tok/s single-threaded in Node, with chat templating verified against the model's own Jinja template. No WebGPU, no threads, no `SharedArrayBuffer`: the baseline that works in browsers that only have plain WASM. --- ## TL;DR | | llm.js (2023) | this | |---|---|---| | llama.cpp | `dd047b4` | `f7591a6a01f9` (2026-11-07) | | entry point | patch adding `examples/main`, `callMain(argv)` | own C API, `src/llm_wasm.cpp` | | output | scrape `printf` from stdout + regex filter | `EM_JS` token callback, exact streaming | | model | GGUF into MEMFS, `-m /models/model.bin` | same (no buffer-load API exists anymore) | | wasm | `wasm32`, 1 thread, SIMD | same, but now `LLAMA_WASM_MEM64=OFF` explicitly | | prompt | raw completion only | raw **or** the GGUF's own Jinja template (engine linked in), preset fallback | | run modes | one text prompt | tabbed demo: chat playground over an editable message list + raw completion | | sampling | CLI flags | temperature, top_p, top_k, repetition penalty/window, seed, max new tokens, stop sequences | | tools / reasoning | not supported | OpenAI-style `tools` handed to the template; the reply is split by the model's own PEG parser (tool calls as JSON, reasoning left inline in `think` tags) | | template params | not supported | `chat_template_kwargs` (e.g. `enable_thinking`) for the Jinja render | | export | not supported | the playground conversation as OpenAI JSON or Markdown, downloaded | | images | not supported | an `mmproj` GGUF loaded next to the model; images attach to a message, are tokenized by `mtmd`, and export as OpenAI multi-part content | | context / KV | `-c` flag, prompt re-evaluated every run | context length, KV type per K/V, unified cache, flash attention as launch params; prompt prefix reused across turns | | grammar | `--grammar` argv | `llama_sampler_init_grammar` | | threads | 1 (ggml clamps wasm without pthreads) | optional pthread build: N wasm workers, 8.2 โ†’ 26.4 tok/s at 4 threads in Node; single-thread module is the default | **The concept still works.** What broke are the three things llm.js depended on: `examples/main` was deleted, CLI-driving (`callMain`) is gone, and the C API was renamed/reworked (model/context/vocab/sampler/chat). Everything else is small. --- ## Measured results Run on 4 vCPU, Node 27, single-threaded wasm, `n_ctx = 512`: ``` llama.cpp wasm build: 0.9.0-dev (f7591a6) model: LFM2.5-VL-450M-Q4_K_M.gguf (218.7 MiB, arch lfm2, 16 blocks, 65536 vocab) loaded in 0.4s, n_ctx=512, chat_template=yes prompt: "Explain in two sentences why WebAssembly is useful for running LLMs in a browser." tokens: 48 generation: 8.58s (5.59 tok/s) wasm binary: 7.1 MB (Jinja engine, common/chat, PEG parser, parsers/*, mtmd) CPU compute buffer reserved for n_batch=512: 138.52 MiB ``` Output (greedy-ish, temp 0.7): > WebAssembly provides a high-performance, low-level runtime environment that enables efficient and scalable execution of large-scale machine learning models, sucโ€ฆ Multi-turn chat (`chat: true`) produces the same quality as the native build for simple user messages, because the prompt is verified token-identical (below). ## Layout ``` llama.cpp/ shallow clone of ggml-org/llama.cpp (gitignored) llama.cpp.lock revision the results below were produced with src/llm_wasm.cpp the C API that replaces llm.js' patched CLI scripts/fetch-llama.sh clone llama.cpp at the pinned revision (shallow) scripts/build-wasm.sh llama.cpp (wasm32, static) + wrapper -> web/wasm/ scripts/fetch-model.sh download the test GGUF into models/ web/index.html tabbed demo: chat playground, raw completion, template editor, library web/llm.js LLM class, same API as llm.js web/worker.js worker: download -> MEMFS -> wasm -> stream tokens web/download.js resumable chunked download (Range + restart fallback) web/storage.js IndexedDB model library (uuid records + 4 MiB chunks) web/sha256.js streaming SHA-256, for model fingerprints web/actions.js worker message ids web/wasm/ build output + llm-wasm.build.txt (commit stamp) test/node-test.mjs headless inference test test/download-test.mjs pause/resume/Range tests for the downloader test/storage-test.mjs model library CRUD, chunk iteration, v1 -> v2 migration test/sha256-test.mjs streaming SHA-256 vs node's crypto test/chat-template-test.mjs Jinja template: preset equivalence, override, error fallback test/chat-generate-test.mjs stepwise generation: streaming, stop sequences, interrupt, limits test/tool-call-test.mjs tools in the template, enable_thinking, PEG tool-call parsing, tool turns test/kv-cache-test.mjs ctx/KV launch params, quantized caches, prefix reuse across turns test/multimodal-test.mjs mmproj load, an image prompt through mtmd, marker/image mismatch test/browser-test.py drives the demo page in headless Chromium test/dump-chat-template.mjs, verify-chat-template.{mjs,py}, verify-tokens.mjs ``` ## Build Requires Emscripten. `emsdk` is expected at `../tools/emsdk` (override `EMSDK=`). ```bash scripts/fetch-llama.sh # llama.cpp at the pinned revision (~100 MB, shallow) scripts/build-wasm.sh # clean build ~3-4 min on 4 cores scripts/fetch-model.sh # 219 MiB ``` The checkout is not committed (it is a nested git repo, ~1 GB of history). `llama.cpp.lock` records the revision used for the results above; `build-wasm.sh` fetches it when `llama.cpp/` is missing and warns when your checkout has moved past it. Updating is expected โ€” llama.cpp changes weekly: ```bash rm -rf llama.cpp && LLAMA_REF=master scripts/fetch-llama.sh scripts/build-wasm.sh node test/node-test.mjs models/LFM2.5-VL-450M-Q4_K_M.gguf "Hi" 32 ``` If a future llama.cpp breaks the build or the API, the compile error points at `src/llm_wasm.cpp`; the porting notes below list what tends to move. What the script does, and why each flag is there: ```bash emcmake cmake -S llama.cpp -B build/llama-wasm \ -DBUILD_SHARED_LIBS=OFF -DLLAMA_BUILD_EXAMPLES=OFF -DLLAMA_BUILD_TOOLS=OFF \ -DLLAMA_BUILD_TESTS=OFF -DLLAMA_BUILD_COMMON=OFF -DLLAMA_CURL=OFF \ -DLLAMA_WASM_MEM64=OFF # upstream defaults to wasm64 for EMSCRIPTEN -DLLAMA_BUILD_HTML=OFF # otherwise the target suffix becomes .html -DGGML_NATIVE=OFF -DGGML_OPENMP=OFF -DGGML_BACKEND_DL=OFF -DGGML_CPU=ON -DCMAKE_C_FLAGS="-O3 -msimd128" -DCMAKE_CXX_FLAGS="-O3 -msimd128" cmake --build build/llama-wasm --target llama -j$(nproc) em++ src/llm_wasm.cpp llama.cpp/common/jinja/*.cpp \ llama.cpp/common/json.cpp llama.cpp/common/unicode.cpp \ -O3 -msimd128 -fexceptions \ -I llama.cpp/include -I llama.cpp/ggml/include -I llama.cpp/common -I llama.cpp/vendor \ -Wl,--start-group $(find build/llama-wasm -name '*.a') -Wl,--end-group \ -o web/wasm/llm-wasm.js \ -sDISABLE_EXCEPTION_CATCHING=0 \ -sMODULARIZE=1 -sEXPORT_ES6=1 -sEXPORT_NAME=createLLMWasmModule \ -sENVIRONMENT=web,worker,node -sFORCE_FILESYSTEM=1 -sNO_EXIT_RUNTIME=1 \ -sALLOW_MEMORY_GROWTH=1 -sMAXIMUM_MEMORY=4GB -sINITIAL_MEMORY=64MB -sSTACK_SIZE=8MB \ -sEXPORTED_FUNCTIONS=... -sEXPORTED_RUNTIME_METHODS='["FS","ccall","cwrap","UTF8ToString"]' ``` Notes that cost time to discover: * **`em++`, not `emcc`, for the final link.** `emcc` treats `.cpp` as C++ for compiling but links without libc++, so every `std::string`/`operator new` symbol is undefined. * **`-DLLAMA_WASM_MEM64=OFF`** โ€” since wasm memory64 landed, llama.cpp's `CMakeLists.txt` adds `-sMEMORY64=1` for `EMSCRIPTEN` by default. 64-bit wasm memory is not available in every browser/toolchain, and 4 GiB is enough for a 450M Q4 model, so this build stays on wasm32. * **`libcommon` is not built** (`LLAMA_BUILD_COMMON=OFF`). `tools/` and `examples/` are skipped entirely under EMSCRIPTEN, so the CLI/server path is not available at all โ€” the API has to be ours. * `-msimd128` is what enables `ggml/src/ggml-cpu/arch/wasm/quants.c`; upstream also sets it per-target in `ggml/src/ggml-cpu/CMakeLists.txt`. * **`mtmd` is compiled in** (`tools/mtmd/*.cpp` + `tools/mtmd/models/*.cpp`, minus the CLI): upstream's CMake target links `vendor::stb`, miniaudio and `vendor::hash`, so the build adds `vendor/hash/hash.cpp` and `sha256.c`. That last file is **C**: `em++` would compile it as C++ and mangle `sha256_hash` (`wasm-ld: undefined symbol`), so the script builds it with `emcc` into an object and links that. `MTMD_VIDEO` stays off (it needs ffmpeg); audio sources compile but were not run. * The build writes `web/wasm/llm-wasm.build.txt` with the exact llama.cpp commit, so results stay attributable while upstream moves weekly. ## Threads (optional, cross-origin isolation) There are two builds of the same sources: | build | flag | works on | threads | |---|---|---|---| | `web/wasm/llm-wasm.*` | `scripts/build-wasm.sh` | any page | 1 (ggml clamps it) | | `web/wasm/llm-wasm-mt.*` | `MT=1 scripts/build-wasm.sh` | pages with COOP/COEP | 1..N (`llmjs_set_threads`) | The pthread build needs `SharedArrayBuffer`, which browsers expose only to a **cross-origin isolated** page. `test/serve.py` is `http.server` plus the two headers (`Cross-Origin-Opener-Policy: same-origin`, `Cross-Origin-Embedder-Policy: require-corp`); GitHub Pages and most static hosts cannot set them, which is why the single-thread module stays the default and the worker picks the module at runtime: ```js const THREADED = typeof SharedArrayBuffer === "function" && self.crossOriginIsolated === true; factory = THREADED ? await import("./wasm/llm-wasm-mt.js") : await import("./wasm/llm-wasm.js"); ``` The thread count is a launch parameter like context length or KV type: it goes into the context signature, so changing it rebuilds `llama_context` (Reload engine), and it also drives mtmd's image encode/decode (`p.n_threads`). **Measured in Node** (`test/threads-test.mjs`, LFM2.5-VL-450M Q4_K_M, greedy, same prompt and output): ``` 1 thread: 8.22 tok/s (2921 ms for 24 tokens) 4 threads: 26.36 tok/s ( 911 ms for 24 tokens) โ†’ 3.21ร— wall time prompt eval: 1605 ms โ†’ 1032 ms ``` **Threads are pre-spawned, and that is not optional.** `pthread_create()` called from inside a wasm call deadlocks in a browser: the calling thread blocks in `Atomics.wait` while the worker it just asked for has to fetch and boot the same ES module, and it never gets to report ready. Node is fine because `worker_threads` start differently. So the module is built with `-sPTHREAD_POOL_SIZE` set to a JS expression that runs at module init: ``` "-sPTHREAD_POOL_SIZE=${PTHREAD_POOL_SIZE:-Math.max(1, Math.min((typeof navigator !== 'undefined' && navigator.hardwareConcurrency) || 4, 8))}" ``` The pool therefore adapts to the machine that loads the page (capped at 8), and the thread field is capped to the same number. Choosing fewer threads than the pool costs nothing at run time โ€” the extra workers sit idle on shared memory and model load stays ~10 s. Asking for more than the pool size reproduces the old silent hang, which is why the field can't go above it. Measured in headless Chromium, 450M Q4_K_M, 8 new tokens, same prompt: | threads | decode | vs 1 thread | |---|---|---| | 1 | 0.98 s, 8.18 tok/s | 1.0x | | 2 | 0.58 s, 13.74 tok/s | 1.68x | | 4 | 0.43 s, 18.56 tok/s | 2.27x | The page now defaults to `min(cores, 4)` threads when the page is cross-origin isolated, and `?threads=N` still forces a specific count (`?threads=1` for the old single-thread behaviour). Without COOP/COEP it falls back to the single-thread module. Useful facts found while building it: * `-pthread` on the CMake flags defines `__EMSCRIPTEN_PTHREADS__`, which is what stops `ggml_graph_plan()` from forcing `n_threads = 1`. * the MT build caps memory at 2 GB (safe `SharedArrayBuffer` size across browsers) instead of the single-thread build's 4 GB. * Emscripten warns that `-pthread` + `ALLOW_MEMORY_GROWTH` makes the JS side of growth slow; the model load grows the heap once, before generation. * `vendor/hash/sha256.c` still has to be compiled as C (`emcc`) in both builds. ## Run Headless (this is what was verified here): ```bash node test/node-test.mjs models/LFM2.5-VL-450M-Q4_K_M.gguf "Hi" 48 NO_CHAT=1 node test/node-test.mjs ... # raw completion, no template CHAT_TEMPLATE=chatml node test/node-test.mjs ... # force a built-in template ``` ### Browser demo ```bash python3 -m http.server 8000 --directory web ``` Then open (serve it โ€” `file://` cannot load ES modules or `fetch` the wasm; any static server works, no CORS/COOP headers needed for the single-threaded build). The page gives you a small **model library**: any number of models, each with a uuid record, an editable display name, its size, its SHA-256 and a state (`complete`, `partial`, `needs file`). * **from a URL** โ€” 4 MiB chunks into IndexedDB, progress bar, `Pause` (aborts the fetch, keeps stored chunks), `Resume` (continues with an HTTP `Range` request), `delete` (drops the record + every chunk). If the server ignores `Range` and returns `200`, the download restarts cleanly instead of corrupting the file. Adding the same URL twice reuses the record instead of storing two 200 MB copies. * **from this computer** โ€” a GGUF you already have. By default **nothing is copied**: the `File` is handed to the worker and read in 4 MiB slices straight into the wasm FS (and the SHA-256), so the browser holds exactly one copy โ€” the one already on your disk. After a reload the record reads `needs file` and asks you to pick it again; nothing is re-downloaded. Tick *keep a copy in browser storage* if you would rather have it live in IndexedDB (still one copy, and the original file is not needed afterwards). * **sha256** โ€” computed once per model, streamed from wherever its bytes live (`crypto.subtle.digest()` is one-shot, so `web/sha256.js` is a small incremental SHA-256), shown in the row (full value on hover) with a `copy hash` button so it can be compared with the publisher's. Paste an expected hash when adding and the row marks it โœ“ or โ‰  expected. * **pick the model from a dropdown** โ€” the list of library models; `Load` runs the selected one (a local file asks for its file again if it was not copied), and `Reload engine` frees the model and the wasm backend and loads it once more, so switching models or recovering from a bad state does not need a page reload. * **tabs** โ€” *Chat playground*, *Raw completion*, *Chat template*, *Models*. The sampling/limits rail sits next to the two run tabs, so both modes share it. * **chat playground** โ€” the message list *is* the conversation: add, remove and reorder `system` / `user` / `assistant` turns and write them yourself, then `Generate`. The assistant turn is appended before the first token and streams into place, so a human-written transcript can be continued turn by turn. `Stop` in the top bar interrupts and keeps whatever already arrived; the turn stays editable afterwards. (Stepwise generation, below, is what makes that possible without threads.) * **tools** โ€” paste an OpenAI `tools` array in the chat tab's `tools` box and it is handed to the template for rendering. The reply is then run through the PEG parser built for that exact prompt, so a model that answers with `Paris` (MiniCPM5's format) produces a tool-call card: name, id, and the arguments as a JSON tree, plus **Add tool response** which appends a `tool` message already carrying that `tool_call_id`. The call markup is removed from the transcript text, which keeps what the model said before calling. Reasoning is deliberately **not** extracted: it stays in the assistant text inside its `think` block, exactly as the model emitted it. The JSON tree is a small local renderer, not a CDN library, so the lab still works offline. * **export** โ€” `Export JSON` downloads `[{ role, content }, โ€ฆ]` in OpenAI shape (assistant turns keep `tool_calls`, tool turns keep `tool_call_id`, and a message with images gets the multi-part content list with `{"type":"image_url","image_url":{"url":"data:image/png;base64,โ€ฆ"}}` parts); `Export Markdown` downloads the same transcript with `### SYSTEM` / `### USER` / `### ASSISTANT` / `### TOOL (call_id)` headers, fenced JSON for calls and a literal `<< image >>` where an image was. No conversation library, no cloud: it is a lab, and the file is the storage. * **images** โ€” an uploaded GGUF can be flagged **mmproj** (checkbox in either ingest form; it is then left out of the text-model dropdown). In the engine rail, tick *load multimodal projector* and pick the projector that is loaded with the next model load / `Reload engine`. In the playground, the ๐Ÿ–ผ button on a message (or a drag & drop onto it) attaches PNG/JPEG/WebP images as mini-avatar thumbnails, each with its own โœ• to remove it. The message goes through the template with one media marker per image and `mtmd` replaces each marker with that image's token sequence; the template preview deliberately shows the marker itself, since image tokens sit below the rendered text. * **raw completion** โ€” no template at all: the text is the prompt, which is what a base/completion model expects (and a chat model then answers as plain text). * **engine** โ€” the launch parameters the rail fixes when a context is created: context length, KV cache type for K and V (`f16`, `q8_0`, `q4_0`), flash attention (`auto`/`off`/`on`), the unified KV cache, and the optional mmproj. `Reload engine` applies a change (a `llama_context` cannot be resized in place); loading a model also applies the current values. A quantized V cache switches flash attention on, because llama.cpp refuses that combination otherwise. Parallel sequences stay at one: extra slots would only duplicate a cache a single user cannot fill. This build has no server-style prompt-cache RAM pool (that 8 GiB default is `common_params::cache_ram_mib`, used by `llama-server`) โ€” memory is the model plus the KV cache, which grows with context length and shrinks with the quantized types. * **sampling / limits** โ€” temperature, top_p, top_k, repetition penalty and its window, seed (`-1` = random), max new tokens, and stop sequences (one per line, `\n`/`\t` expanded). Stops are matched in C++ against the decoded text, so they never appear in the output; the last `len-1` bytes are held back until a match is impossible, and the finished text comes from `llmjs_last_output()` instead of being reassembled from the stream. * **chat template** tab โ€” the textarea is prefilled from the GGUF; unchecking "use the template shipped in the GGUF" makes it editable and the override wins for the next run. A `parameters JSON` box adds template kwargs (e.g. `{"enable_thinking": true}`), and the *enable_thinking* checkbox in the rail writes that key for you; both the preview and the next run use them. `Preview with the playground messages` renders the current transcript without generating โ€” the fastest way to see why a model answers badly. The stats line reports which path ran (`template: jinja`, `jinja:override`, `builtin:`, `raw`). * **llama.cpp log** โ€” collapsible, shows the wasm build version and load messages. The two ingest paths are deliberately separate in the UI (pasting a URL and picking a file are different gestures, and only the file path has a "copy or read in place" decision) but converge immediately: one record type, one load path, one hash, one delete. Everything stays local: no server-side component, no model upload, and the cached model is visible/removable from the page (browser quota shown at the top). `web/llm.js` keeps the original API: ```js const app = new LLM('GGUF_CPU', modelUrl, () => {}, // init (text) => {}, // streaming tokens (stats) => {}, // complete: { tokens, ms, tokens_per_second } (err) => {}, // NEW optional: error (loaded, total) => {}, // NEW optional: download progress (line) => {}); // NEW optional: llama.cpp logs app.load_worker({ context_size: 1024, kv: { type_k: 'q8_0', type_v: 'q8_0', flash_attn: -1, kv_unified: true, n_parallel: 1 }, mmproj_id: 'uuid-of-a-flagged-mmproj' }); // optional projector app.run({ prompt: 'Hi', chat: true, max_token_len: 48, temp: 0.7 }); // old shape, still works // chat completion over a whole transcript (what the playground sends) app.run({ mode: 'chat', messages: [{ role: 'system', content: 'You are terse.' }, { role: 'user', content: 'Hi' }], temp: 0, top_p: 0.9, top_k: 40, repeat_penalty: 1.1, repeat_last_n: 32, seed: 42, max_token_len: 64, stop: ['\n\n'], chat_template_source: '', tools: JSON.stringify(tools), // OpenAI tools array, handed to the template template_kwargs: '{"enable_thinking": true}', images: ['data:image/png;base64,โ€ฆ'], // one per <__media__> marker, in order }); // stats.parsed = { content, reasoning_content, tool_calls: [{ id, name, arguments }] } // null when the template's parser did not recognise the reply // stats.run_stats = { prompt_tokens, reused_tokens, prompt_ms, n_ctx } app.run({ mode: 'completion', prompt: 'The capital of France is' }); app.stop(); // keeps the tokens already streamed, completes with interrupted: true ``` `run()` is stepped inside the worker (`llmjs_gen_begin_* โ†’ llmjs_gen_step โ†’ llmjs_gen_end`): one token per turn of the event loop. That is what allows a `stop` message to be received at all in a single-threaded build, and it keeps tokens flowing to the page as they are produced. Model library โ€” a uuid record per model; url and local-file ingest produce the same record and load through the same code path: ```js app.add_url_model({ url, name: 'LFM 450M', expected_sha256: '' }); app.add_file_model({ file, name: '', keep_copy: false }); // nothing copied by default app.load(id); app.hash_model(id); app.delete_model(id); ``` Deviations from llm.js: the GGUF is downloaded **inside the worker** (posting a 219 MB `ArrayBuffer` across the thread boundary would double memory), it is cached in IndexedDB rather than the Cache API so pauses survive reloads, and the new callbacks/progress exist because there is no stdout to scrape anymore. Extra methods: `pause()`, `resume()`, `load(id)`, `add_url_model()`, `add_file_model()`, `hash_model(id)`, `delete_model(id)`, `list_models()`, plus an `on_event` hook for worker messages (`MODEL_ADDED`, `MODEL_UPDATED`, `PAUSED`, `DELETED`, โ€ฆ). ### Embedding the playground (minimal postMessage API) The page can be used as an iframe by a host page โ€” the host keeps its own UI and borrows this one's model library and wasm engine: ```html ``` Protocol (this iteration: chat completion only): | direction | message | |---|---| | parent โ†’ frame | `llmjs:run` `{ id, messages \| prompt, images?, max_tokens?, temperature?, top_p?, top_k?, repeat_penalty?, repeat_last_n?, seed?, stop?, tools?, template_kwargs? }` | | parent โ†’ frame | `llmjs:stop` (keeps what streamed, like the Stop button) | | frame โ†’ parent | `llmjs:api` `{ version }` once the frame is ready | | frame โ†’ parent | `llmjs:ready` `{ model, n_ctx, threads }` when a model finishes loading | | frame โ†’ parent | `llmjs:token` `{ id, text }` per token, `llmjs:done` `{ id, text, stats }`, `llmjs:error` `{ id, error }` | A model still has to be loaded inside the frame first (that is where the library and the wasm module live), one run at a time, and the frame accepts messages from `window.parent` regardless of origin โ€” fine for a local lab, but a public deployment should check `event.origin` before answering. The host page needs the frame's URL to be same-origin or CORS/COOP-friendly; the threaded build's COOP/COEP headers do not leak to the parent page. --- ## What actually broke between llm.js and today's llama.cpp `llm.js` patched the root `CMakeLists.txt` to add `examples/main` and called `Module.callMain(["-m", ..., "-p", ...])`, filtering `main:` / `*_model_load:` lines out of stdout with regexes. Current status of each assumption: 1. **`examples/main` no longer exists.** The patch does not apply; there is no CLI to embed. (The current CLI lives in `tools/cli`, is excluded from Emscripten builds, and pulls in `common/` + `arg.cpp`.) 2. **`callMain` is the wrong shape anyway.** Token streaming now comes from `llama_sampler_sample` โ†’ `llama_token_to_piece` โ†’ an `EM_JS` callback, so output is exact and no regex filtering is needed. 3. **No buffer loading.** `llama_model_load_from_buffer` is gone; only `llama_model_load_from_file`, `..._file_ptr`, `..._splits`. So the GGUF still goes into MEMFS and is loaded from `/models/model.gguf`. 4. **Renames** the old code hits immediately: `llama_new_context_with_model` โ†’ `llama_init_from_model`; `llama_n_vocab` โ†’ `llama_vocab_n_tokens` (vocab is an opaque object); `llama_kv_cache_clear` โ†’ `llama_memory_clear(llama_get_memory(ctx), true)`; `llama_sample_*` โ†’ `llama_sampler_*` chain (`llama_sampler_chain_init` + `top_k`/`top_p`/`temp`/`penalties`/`dist`); `llama_token_to_piece(vocab, tok, buf, len)` gained `lstrip`/`special`; `llama_model_params` lost `use_mmap`/`use_mlock` (now `enum llama_load_mode`). 5. **Sampling is explicit.** Nothing is sampled for you: build a chain per run, `llama_sampler_sample`, then `llama_sampler_accept` and decode the token. 6. **Threads.** ggml forces `n_threads = 1` for `__EMSCRIPTEN__` without pthreads (`ggml/src/ggml-cpu/ggml-cpu.c`), and the wrapper sets `n_threads`/`n_threads_batch = 1` so the thread-pool path is never entered. 7. **Chat templating is a different subsystem now** โ€” see the next section. 8. **`--json-schema` is not available** to this build: JSON-schema โ†’ GBNF lives in `common/json-schema-to-grammar.cpp`, which we do not compile. `grammar` (GBNF) works via `llama_sampler_init_grammar`. --- ## The chat-template rabbit hole (verified, not assumed) This model ships a **Jinja** `tokenizer.chat_template` (ChatML-style: `<|im_start|>user โ€ฆ <|im_end|>`, `think` blocks, tool calls, image content). That matters because llama.cpp has *two* templating stacks: * **`libllama`'s `llama_chat_apply_template()`** โ€” a fixed list of ~60 built-in presets (`chatml`, `llama3`, `mistral-v1`, `gemma`, `phi3`, โ€ฆ). It does **not** run Jinja. This is all you get if you link only `libllama`. * **`common/chat.cpp` + `common/jinja` (minja) + the PEG auto-parser** (`docs/autoparser.md`, `common/chat-auto-parser.*`, `common/chat-peg-parser.*`) โ€” the real Jinja engine and output parser. This is what `llama-cli` / `llama-server` use, and it drags in `common/sampling`, `common/json`, `json-schema-to-grammar`, `peg-parser`, โ€ฆ i.e. the whole `common` library. Relevant flags, read from `common/arg.cpp` (not from memory): | flag | behaviour | |---|---| | `--jinja` / `--no-jinja` | enable the Jinja engine (default off) | | `--chat-template JINJA_TEMPLATE` | custom template; **without `--jinja` only commonly used (built-in) templates are accepted** | | `--chat-template-file FILE` | same, from a file | | `--chat-template-kwargs` | template kwargs (e.g. thinking/preserve reasoning) | | `--skip-chat-parsing` | force pure content parsing even with a Jinja template | | `--reasoning-format` | how `think` blocks are returned | **The trap:** passing the model's own Jinja string to `llama_chat_apply_template()` *aborts the wasm module* with a bare `RuntimeError: Aborted(undefined)`. Reason, from the source: `llm_chat_detect_template()` (`src/llama-chat.cpp:89`) does `try { llm_chat_template_from_str(tmpl) } catch (const std::out_of_range &)`, i.e. it uses a C++ **exception for control flow**. Emscripten builds disable exception handling by default, so the `throw` never reaches the `catch` โ€” it calls `__cxa_throw` โ†’ `__Unwind_RaiseException` โ†’ `abort()`. Passing the short name `"chatml"` hits the map directly and never throws. (llm.js never hit this: it had no template support at all.) **The fix: link the engine, not the presets.** `common/jinja` is a Jinja engine of its own (lexer/parser/runtime), and it needs only `common/json.cpp` + `common/unicode.cpp` next to it โ€” not the whole `common` library (no sampling, no PEG parser, no autoparser). This build compiles those eight files with `-fexceptions` and links with `-sDISABLE_EXCEPTION_CATCHING=0`, which is exactly what makes the `try { โ€ฆ } catch (const std::out_of_range &)` above work instead of calling `abort()`. So the model's own template is the primary path, rendered by the same engine `llama-server --jinja` uses: 1. read `tokenizer.chat_template` from the GGUF (`llama_model_chat_template`), 2. render it with `messages`, `bos_token`, `eos_token` and `add_generation_prompt` in scope (`jinja::global_from_json` + `jinja::runtime`), 3. strip a leading BOS / trailing EOS when the tokenizer adds them itself โ€” the same rule `common_chat_template_direct_apply_impl()` applies, which is why the rendered text does not end up with two `<|startoftext|>` tokens, 4. let the caller override the template source entirely (the demo has an "advanced ยท chat template" box for this, handy for a model with a broken or missing template), 5. fall back to the closest built-in preset (`<|im_start|>` โ†’ `chatml`, `<|start_header_id|>` โ†’ `llama3`, `` โ†’ `gemma`, `[INST]` โ†’ `mistral-v1`, `<|im_sep|>` โ†’ `phi3`) if the template is missing or throws, and to raw completion if nothing fits. The chain is reported back to the page: `template: jinja`, `jinja:override`, `builtin:` or `raw`. ### Verification (byte-level and token-level) `test/chat-template-test.mjs` renders the model's template **inside wasm** and compares it with the built-in preset, then tokenizes both: ``` 1. GGUF template: 3836 bytes, detected preset 'chatml' 2. Jinja render is byte- and token-identical to builtin chatml "<|im_start|>user\nHi<|im_end|>\n<|im_start|>assistant\n" 3. caller-supplied template renders verbatim 4. 4-message conversation renders (168 bytes) 5. broken template reported, module still alive: "Unexpected token: %} of type 9" 6. generated 2 tokens via the Jinja path; broken override fell back to builtin:chatml ``` The older pair below checks the preset fallback against Python's `jinja2`: `test/verify-chat-template.mjs` renders through the wasm module; `test/verify-chat-template.py` parses the GGUF metadata directly (small GGUF reader), renders the *actual* Jinja template with Python `jinja2`, and diffs the two; `test/verify-tokens.mjs` then proves the token streams are equal. ``` wasm (built-in chatml): "<|im_start|>user\nHi<|im_end|>\n<|im_start|>assistant\n" jinja2 (model template): "<|startoftext|><|im_start|>user\nHi<|im_end|>\n<|im_start|>assistant\n" ^ only difference: BOS TOKENS MATCH: identical token ids # tokenize(ref, add_special=false) == tokenize(ours, add_special=true) ``` The BOS difference is expected and harmless: the Jinja template emits `{{- bos_token -}}` itself, while our built-in render relies on the tokenizer, which adds BOS because `tokenizer.ggml.add_bos_token = true`. Verified equal at the token level, not argued. ### Tools and the output parser The same `common/chat.cpp` stack that renders the prompt also **parses** the reply: for each template it builds a PEG grammar in `common/parsers/*.cpp` (MiniCPM5's is `minicpm5.cpp`) and `common_chat_peg_parse()` splits the text into `content`, `reasoning_content` and `tool_calls`. This build compiles that half too (`chat*.cpp`, `peg-parser.cpp`, `parsers/*.cpp`, `json-schema-to-grammar`, `reasoning-budget`), which is why the wasm grew from 3.9 MB to 5.1 MB. Two details that are easy to get wrong, both verified here rather than assumed: * `common_chat_parser_params` copies the format and generation prompt but **not** the PEG arena. `common_chat_peg_parse(arena, input, is_partial, params)` takes the arena explicitly; pass `g_chat_params->parser` or everything parses as content-only. * the parser expects the generation prompt in front of the generated text (`effective_input.prepend(params.generation_prompt)`), exactly as `llama-server` feeds it. Whatever the template appended past the parser's literal โ€” an opened `think` block, for instance โ€” lands in the content, so the wrapper strips the generation-prompt tail from the start of the content. Reasoning is not extracted for display: `run()` returns it inside `content`, `think` tags included, which is what the playground shows. Tool calls are extracted because they are structured data the page renders as JSON. Special tokens matter here: the token pieces are decoded with `llama_token_to_piece(..., special = true)`, because ``. Multimodal content goes through `mtmd` instead: `llmjs_set_images()` takes the data URLs, the same `llmjs_gen_begin_chat()` sees the markers in the message text, and `mtmd_tokenize()` turns each marker into that image's token sequence before the prompt is evaluated. The projector itself is loaded by `llmjs_mmproj_load()` (llama-server's `--mmproj`): a second GGUF from the same library that is flagged `multimodal` by the user, never auto-detected. --- ## Debugging notes (WASM-specific) * **Exceptions abort silently.** Any `throw` in llama.cpp kills the module with `Aborted(undefined)`. Bisect by logging before/after each call from C++ (an `EM_JS` log helper works); that is how the chat-template crash above was localised to the first `generate` call. * **`EM_JS` bodies are macro arguments** โ€” a top-level comma breaks the macro. * **MEMFS + `canOwn`**: `FS.createDataFile("/models", "model.gguf", bytes, true, false, true)` takes ownership, so the 219 MB buffer is not copied again. Weights still get read into the wasm heap, so the heap grows to a few hundred MB on load. * **Wasm32 vs wasm64**: `-sMAXIMUM_MEMORY=4GB` is the wasm32 ceiling; without `ALLOW_MEMORY_GROWTH` the 64 MB initial heap would fail on model load. * **`HEAPU8`/`UTF8ToString` are not exported by default** โ€” add them to `EXPORTED_RUNTIME_METHODS` if JS needs them. * **Single-thread `n_batch=512` reserve** costs ~139 MiB of compute buffer. Lower `n_batch` if the browser is memory-constrained. --- ## Tests / what is verified | command | needs | checks | |---|---|---| | `node test/node-test.mjs [model] [prompt] [n]` | model | full wasm load + generation, tok/s, output | | `node test/download-test.mjs` | nothing | chunking, pause at chunk boundary, `Range` resume, restart when the server ignores `Range` | | `node test/sha256-test.mjs` | nothing | streaming SHA-256 vs node's `crypto`: every padding boundary, uneven slices, known vectors | | `npm i --no-save fake-indexeddb && node test/storage-test.mjs` | fake-indexeddb | model library CRUD, chunk iteration, v1 โ†’ v2 migration | | `node test/chat-template-test.mjs [model]` | model | GGUF Jinja template vs the verified preset (byte + token identical), custom override, broken template falls back instead of aborting | | `node test/chat-generate-test.mjs [model]` | model | stepwise chat generation over a multi-turn list, streamed text == final output, greedy reproducibility, stop-sequence truncation, interrupt keeps the partial text, raw mode, max-new-tokens | | `node test/tool-call-test.mjs [model]` | MiniCPM5-1B | tools rendered into the prompt, `enable_thinking` changes it, a canned and a **live** reply parse into name + JSON arguments, text before the call is kept, assistant `tool_calls` + `tool` result render back | | `node test/kv-cache-test.mjs [model]` | model | ctx/KV options reported by `llmjs_ctx_config()`, repeated-prompt reuse (6.5 s โ†’ 0.3 s), reply reuse on the next turn, identical output with a cold cache, quantized K/V + flash attention rule, `n_ctx` really applied; adaptive for hybrid/RNN memories | | `node test/multimodal-test.mjs [model] [mmproj]` | LFM2.5-VL + mmproj | projector reports vision support and the media marker, an image prompt evaluates its image tokens (121 prompt tokens vs 19 for the same text) and answers about the picture, one image per marker is enforced, image runs are refused without a projector while text still works | | `node test/threads-test.mjs [model] [threads]` | mt build | the pthread module really runs N threads: identical greedy output, measured tok/s at 1 vs N (3.2ร— at 4 threads on 4 vCPU) | | `python test/serve.py` | nothing | static server with COOP/COEP, needed for `SharedArrayBuffer` / `--threads` | | `node test/verify-chat-template.mjs โ€ฆ && python3 test/verify-chat-template.py โ€ฆ` | jinja2 | built-in `chatml` render vs the model's real Jinja template | | `node test/verify-tokens.mjs โ€ฆ` | model | the two prompts tokenize to identical ids | | `python test/browser-test.py --model-url โ€ฆ \| --file โ€ฆ [--mmproj โ€ฆ]` | playwright | the real page in headless Chromium: both ingest paths โ†’ sha256 โ†’ load โ†’ generate; `--pause-mib N` pauses/resumes, `--file` asserts nothing was copied, plus a `tool` row, both exports (downloads are parsed back) and a template kwarg reaching the preview; `--mmproj` adds a flagged projector, reloads with it, attaches an image (thumbnail add/remove), generates about it, checks the preview marker and the multi-part JSON / `<< image >>` Markdown exports | Verified here: clean wasm build, 48-token generation, raw + chat paths, prompt render byte diff, token-id equivalence, download pause/resume, the model library (uuid records, rename, multi-model delete, v1 โ†’ v2 migration, chunk iteration), streaming SHA-256 against node's `crypto`, HTTP serving of every demo asset, and the demo page driven end to end in headless Chromium over **both** ingest paths (download: 218.7 MiB into 55 chunks, `GGUF` magic intact, paused and resumed via `Range`; local file: hashed and loaded with zero chunks written to IndexedDB). KV reuse and the launch parameters were verified with `test/kv-cache-test.mjs` on MiniCPM5-1B (transformer) and LFM2.5-VL-450M (hybrid): a repeated 27-token prompt went from 6.5 s to 0.3 s of prompt eval with 26/27 tokens reused, a cleared cache gave byte-identical greedy output, `q8_0` K/V ran with flash attention forced on, `q8_0` K with `f16` V kept it off, and `n_ctx=256` was really applied and invalidated the old cache. On LFM2.5 the hybrid memory reused 32/46 tokens on the next turn (append-only, as expected). The tabbed page was verified the same way: the chat playground appends an assistant turn and streams into it (`stats: 24 tokens โ€ฆ ยท template: jinja`), `Stop` reported `ยท stopped` with the partial text kept, the template tab previews the GGUF template (`<|im_start|>user โ€ฆ <|im_end|>`), an override was used (`template: jinja:override`), raw completion reported `template: raw`, andthe engine reloaded from the top bar. A `tool` message row survived with its `tool_call_id`, the JSON export came back as OpenAI messages ending in that tool turn, the Markdown export contained `### TOOL (call_test_1)` and `### USER`, and the template preview accepted a `parameters JSON` kwarg. Multimodal was verified with LFM2.5-VL-450M plus its `mmproj-LFM2.5-VL-450m-Q8_0.gguf` (same Hugging Face repo, `mmproj` file): in Node the projector loaded (`vision=true`, marker `<__media__>`), a 64ร—64 PNG prompted with one marker evaluated 121 prompt tokens against 19 for the same text and the model answered *"The left half of this image is red"*; a marker/image count mismatch was refused (`-2`) instead of mis-rendering, and an image run without a projector was refused (`-3`) while text runs kept working. In the browser the same flow used the real page: flag the file as mmproj, reload with *load multimodal projector* (status line gains `mmproj โ€ฆ (vision)`), attach the image through the ๐Ÿ–ผ button, remove and re-attach the thumbnail, generate (*"The left half of this image is a red color."*, 121 prompt tokens in 24 s โ€” image encoding dominates), see `<__media__>` in the template preview, and export the conversation: one multi-part message with `data:image/png;base64,โ€ฆ` in JSON and `<< image >>` in Markdown. Tools and the parser were verified with MiniCPM5-1B (`MiniCPM5-1B-Q4_K_M.gguf`) in `test/tool-call-test.mjs`: the tools array reaches the template (943 vs 90 bytes), `enable_thinking` toggles the open `think` block, and a real 15-token generation produced `Paris`, which the model's own parser turned into one tool call with `{"city": "Paris"}`. ## Docs consulted (both projects) **llm.js** โ€” `docs/BUILD.md` (build flow), `docs/api_guide.md` (the API preserved here), `scripts/build-llama-cpp-wasm.sh` + `patches/00-llama-cpp-enable-main.patch` (exactly what broke), `src/web-workers/llamacpp-cpu-worker.js` (stdout scraping). **llama.cpp** โ€” the docs are scattered; these were the useful ones: `docs/build.md` (WASM/WebGPU + WASI sections, the latter has the single-thread `-DGGML_OPENMP=OFF -DLLAMA_BUILD_COMMON=OFF` recipe), `.github/workflows/build-wasm.yml` (the flags upstream actually builds with), `docs/autoparser.md` (PEG auto-parser design), `docs/models.md` / `models/templates/README.md` (templates), `tools/mtmd/README*.md` (multimodal), `docs/function-calling.md`, plus source-of-truth `include/llama.h`, `src/llama-chat.cpp`, `common/arg.cpp`, `ggml/src/ggml-cpu/arch/wasm/` and `ggml/src/ggml-cpu/ggml-cpu.c`. ## Limitations / next steps * **Multimodal is vision-projector only, and untested beyond images.** `mtmd` is compiled in and loads any `.gguf` mmproj, but only LFM2.5-VL's vision projector was run; audio projectors compile and were never tried, and video is off (it needs ffmpeg). Images are CPU-bound here: the 64ร—64 test image took ~12 s to encode and ~6 s to decode in wasm, so a photo costs tens of seconds before the first token. Image runs bypass the text prefix cache (the token stream changes with the image), so a subsequent text turn starts cold. * **No `json_schema`** (compiling `common/json-schema-to-grammar.cpp` would add it). GBNF `grammar` works. * **No reasoning-effort control.** MiniCPM5 advertises `supports_reasoning_effort: false`; thinking is on/off via her `enable_thinking` template kwarg only. The sampling stack with thinking budgets is not compiled here, so the control is left out rather than faked. * **Stop granularity is one token**, and stop sequences are text-level: they are matched on decoded text (that is why a stop can appear immediately, not after the next token). There is no token-id ban list yet. * **Prefix reuse depends on the template and the memory type.** The wrapper keeps the tokens that sit in the KV cache and re-evaluates only what a new prompt adds. A repeated prompt drops from 6.5 s to 0.3 s of prompt eval (26/27 tokens reused, `test/kv-cache-test.mjs`). Two limits are inherent: a reasoning template injects its `think` block only in the generation prompt, so with thinking on the reply after that point cannot be reused; and recurrent / hybrid memories (Mamba, LFM2's conv+attention) cannot roll their state back, so they only reuse the append case โ€” never a prefix that needs trimming. That case is the multi-turn one, and it still helps: 32/46 tokens on LFM2.5. * **KV memory is only bounded by `n_ctx`.** There is no separate cache pool in this build (the 8 GiB `--cache-ram` default belongs to `llama-server`); a context of N tokens costs the model's per-token KV size times N, times two if K and V stay `f16`. The quantized types in the rail are the lever. * **The parser only knows the formats llama.cpp ships** (`common/parsers/`). A model whose reply format has no parser there still generates fine; its reply just stays plain text (`stats.parsed` is `null` or has no tool calls). * **Exceptions cost size and some speed**: linking the Jinja engine raised the wasm from 3.2 MB to 3.9 MB and requires `-fexceptions` `-sDISABLE_EXCEPTION_CATCHING=0`; a browser without wasm exception support is not a concern (Emscripten's scheme is JS-based), but the JS glue is larger and every `try/catch` in llama.cpp now works instead of aborting. * **Threads are opt-in and experimental in the browser.** A pthread build exists and is 3.2ร— faster in Node at 4 threads (`test/threads-test.mjs`), but the first multi-thread decode stalls in headless Chromium, so the page keeps one thread by default and `?threads=N` turns the pthread module on. Threads also need cross-origin isolation, i.e. host-controlled COOP/COEP headers. Wasm SIMD is already enabled; the single-thread baseline stays ~8 tok/s on this container. * **A local model has to be re-picked after a reload** (one click, no network) unless you tick *keep a copy in browser storage*. Keeping a `FileSystemFileHandle` in IndexedDB would remove that click, but the API is Chromium-only and brings a permission prompt, so it is left out for now. * **Hashing costs one read of the model** (~2-4 s for 219 MiB here) and happens once, on add; the digest is cached in the record. * **One browser only** โ€” the demo is checked in headless Chromium via Playwright (`test/browser-test.py`); Firefox and Safari were not run, and the IndexedDB quota/eviction behaviour on a nearly-full disk was not exercised. * Similar, more actively maintained projects to compare against: `wllama` (ngxson) and llama.cpp's own WebGPU example. ## This bundle `2de6f90` โ€” engine sampler rebuilt from the chat template, LFM2.5 tool-call parsing, image prompts that grow the context to fit, and structured output (JSON Schema or GBNF) selected from the sidebar. Serve with COOP/COEP so the threaded module is used.