Hugging Face's logo

Spaces:
lemonteaa
/
llama-cpp-wasm
Running

App Files Community
llama-cpp-wasm / README.md
lemonteaa
update readme
28b42d9
preview code
|
Raw History Blame Contribute Delete
48 kB
metadata 
title: Llama Cpp Wasm
emoji: 🐨
colorFrom: green
colorTo: red
sdk: static
pinned: false
custom_headers:
  cross-origin-embedder-policy: require-corp
  cross-origin-opener-policy: same-origin
  cross-origin-resource-policy: cross-origin

llm.js → current llama.cpp (WASM, CPU only)

An experiment: rebuild rahuldshetty/llm.js — "run an LLM in the browser with llama.cpp + WebAssembly", last touched in 2023 — against today's llama.cpp, keeping the same LLM JavaScript API.

Everything here was built and run: llama.cpp f7591a6 (2026-11-07) → 7.1 MB wasm32 CPU-only module (chat templates, tools, the PEG output parser and the mtmd multimodal stack linked in), running LiquidAI/LFM2.5-VL-450M (Q4_K_M, 219 MiB) at ~5.6 tok/s single-threaded in Node, with chat templating verified against the model's own Jinja template.

No WebGPU, no threads, no SharedArrayBuffer: the baseline that works in browsers that only have plain WASM.

TL;DR

llm.js (2023) this
llama.cpp dd047b4 f7591a6a01f9 (2026-11-07)
entry point patch adding examples/main, callMain(argv) own C API, src/llm_wasm.cpp
output scrape printf from stdout + regex filter EM_JS token callback, exact streaming
model GGUF into MEMFS, -m /models/model.bin same (no buffer-load API exists anymore)
wasm wasm32, 1 thread, SIMD same, but now LLAMA_WASM_MEM64=OFF explicitly
prompt raw completion only raw or the GGUF's own Jinja template (engine linked in), preset fallback
run modes one text prompt tabbed demo: chat playground over an editable message list + raw completion
sampling CLI flags temperature, top_p, top_k, repetition penalty/window, seed, max new tokens, stop sequences
tools / reasoning not supported OpenAI-style tools handed to the template; the reply is split by the model's own PEG parser (tool calls as JSON, reasoning left inline in think tags)
template params not supported chat_template_kwargs (e.g. enable_thinking) for the Jinja render
export not supported the playground conversation as OpenAI JSON or Markdown, downloaded
images not supported an mmproj GGUF loaded next to the model; images attach to a message, are tokenized by mtmd, and export as OpenAI multi-part content
context / KV -c flag, prompt re-evaluated every run context length, KV type per K/V, unified cache, flash attention as launch params; prompt prefix reused across turns
grammar --grammar argv llama_sampler_init_grammar
threads 1 (ggml clamps wasm without pthreads) optional pthread build: N wasm workers, 8.2 → 26.4 tok/s at 4 threads in Node; single-thread module is the default

The concept still works. What broke are the three things llm.js depended on: examples/main was deleted, CLI-driving (callMain) is gone, and the C API was renamed/reworked (model/context/vocab/sampler/chat). Everything else is small.

Measured results

Run on 4 vCPU, Node 27, single-threaded wasm, n_ctx = 512:

llama.cpp wasm build: 0.9.0-dev            (f7591a6)
model: LFM2.5-VL-450M-Q4_K_M.gguf (218.7 MiB, arch lfm2, 16 blocks, 65536 vocab)
loaded in 0.4s, n_ctx=512, chat_template=yes
prompt: "Explain in two sentences why WebAssembly is useful for running LLMs in a browser."
tokens: 48
generation: 8.58s (5.59 tok/s)
wasm binary: 7.1 MB (Jinja engine, common/chat, PEG parser, parsers/*, mtmd)
CPU compute buffer reserved for n_batch=512: 138.52 MiB

Output (greedy-ish, temp 0.7):

WebAssembly provides a high-performance, low-level runtime environment that enables efficient and scalable execution of large-scale machine learning models, suc…

Multi-turn chat (chat: true) produces the same quality as the native build for simple user messages, because the prompt is verified token-identical (below).

Layout 

llama.cpp/                 shallow clone of ggml-org/llama.cpp (gitignored)
llama.cpp.lock             revision the results below were produced with
src/llm_wasm.cpp           the C API that replaces llm.js' patched CLI
scripts/fetch-llama.sh     clone llama.cpp at the pinned revision (shallow)
scripts/build-wasm.sh      llama.cpp (wasm32, static) + wrapper -> web/wasm/
scripts/fetch-model.sh     download the test GGUF into models/
web/index.html             tabbed demo: chat playground, raw completion, template editor, library
web/llm.js                 LLM class, same API as llm.js
web/worker.js              worker: download -> MEMFS -> wasm -> stream tokens
web/download.js            resumable chunked download (Range + restart fallback)
web/storage.js             IndexedDB model library (uuid records + 4 MiB chunks)
web/sha256.js              streaming SHA-256, for model fingerprints
web/actions.js             worker message ids
web/wasm/                  build output + llm-wasm.build.txt (commit stamp)
test/node-test.mjs         headless inference test
test/download-test.mjs     pause/resume/Range tests for the downloader
test/storage-test.mjs      model library CRUD, chunk iteration, v1 -> v2 migration
test/sha256-test.mjs       streaming SHA-256 vs node's crypto
test/chat-template-test.mjs Jinja template: preset equivalence, override, error fallback
test/chat-generate-test.mjs stepwise generation: streaming, stop sequences, interrupt, limits
test/tool-call-test.mjs    tools in the template, enable_thinking, PEG tool-call parsing, tool turns
test/kv-cache-test.mjs     ctx/KV launch params, quantized caches, prefix reuse across turns
test/multimodal-test.mjs   mmproj load, an image prompt through mtmd, marker/image mismatch
test/browser-test.py       drives the demo page in headless Chromium
test/dump-chat-template.mjs, verify-chat-template.{mjs,py}, verify-tokens.mjs

Build

Requires Emscripten. emsdk is expected at ../tools/emsdk (override EMSDK=).

scripts/fetch-llama.sh         # llama.cpp at the pinned revision (~100 MB, shallow)
scripts/build-wasm.sh          # clean build ~3-4 min on 4 cores
scripts/fetch-model.sh         # 219 MiB

The checkout is not committed (it is a nested git repo, ~1 GB of history). llama.cpp.lock records the revision used for the results above; build-wasm.sh fetches it when llama.cpp/ is missing and warns when your checkout has moved past it. Updating is expected — llama.cpp changes weekly:

rm -rf llama.cpp && LLAMA_REF=master scripts/fetch-llama.sh
scripts/build-wasm.sh
node test/node-test.mjs models/LFM2.5-VL-450M-Q4_K_M.gguf "Hi" 32

If a future llama.cpp breaks the build or the API, the compile error points at src/llm_wasm.cpp; the porting notes below list what tends to move.

What the script does, and why each flag is there:

emcmake cmake -S llama.cpp -B build/llama-wasm \
  -DBUILD_SHARED_LIBS=OFF -DLLAMA_BUILD_EXAMPLES=OFF -DLLAMA_BUILD_TOOLS=OFF \
  -DLLAMA_BUILD_TESTS=OFF  -DLLAMA_BUILD_COMMON=OFF  -DLLAMA_CURL=OFF \
  -DLLAMA_WASM_MEM64=OFF           # upstream defaults to wasm64 for EMSCRIPTEN
  -DLLAMA_BUILD_HTML=OFF           # otherwise the target suffix becomes .html
  -DGGML_NATIVE=OFF -DGGML_OPENMP=OFF -DGGML_BACKEND_DL=OFF -DGGML_CPU=ON
  -DCMAKE_C_FLAGS="-O3 -msimd128" -DCMAKE_CXX_FLAGS="-O3 -msimd128"
cmake --build build/llama-wasm --target llama -j$(nproc)

em++ src/llm_wasm.cpp llama.cpp/common/jinja/*.cpp \
  llama.cpp/common/json.cpp llama.cpp/common/unicode.cpp \
  -O3 -msimd128 -fexceptions \
  -I llama.cpp/include -I llama.cpp/ggml/include -I llama.cpp/common -I llama.cpp/vendor \
  -Wl,--start-group $(find build/llama-wasm -name '*.a') -Wl,--end-group \
  -o web/wasm/llm-wasm.js \
  -sDISABLE_EXCEPTION_CATCHING=0 \
  -sMODULARIZE=1 -sEXPORT_ES6=1 -sEXPORT_NAME=createLLMWasmModule \
  -sENVIRONMENT=web,worker,node -sFORCE_FILESYSTEM=1 -sNO_EXIT_RUNTIME=1 \
  -sALLOW_MEMORY_GROWTH=1 -sMAXIMUM_MEMORY=4GB -sINITIAL_MEMORY=64MB -sSTACK_SIZE=8MB \
  -sEXPORTED_FUNCTIONS=... -sEXPORTED_RUNTIME_METHODS='["FS","ccall","cwrap","UTF8ToString"]'

Notes that cost time to discover:

  • em++, not emcc, for the final link. emcc treats .cpp as C++ for compiling but links without libc++, so every std::string/operator new symbol is undefined.
  • -DLLAMA_WASM_MEM64=OFF — since wasm memory64 landed, llama.cpp's CMakeLists.txt adds -sMEMORY64=1 for EMSCRIPTEN by default. 64-bit wasm memory is not available in every browser/toolchain, and 4 GiB is enough for a 450M Q4 model, so this build stays on wasm32.
  • libcommon is not built (LLAMA_BUILD_COMMON=OFF). tools/ and examples/ are skipped entirely under EMSCRIPTEN, so the CLI/server path is not available at all — the API has to be ours.
  • -msimd128 is what enables ggml/src/ggml-cpu/arch/wasm/quants.c; upstream also sets it per-target in ggml/src/ggml-cpu/CMakeLists.txt.
  • mtmd is compiled in (tools/mtmd/*.cpp + tools/mtmd/models/*.cpp, minus the CLI): upstream's CMake target links vendor::stb, miniaudio and vendor::hash, so the build adds vendor/hash/hash.cpp and sha256.c. That last file is C: em++ would compile it as C++ and mangle sha256_hash (wasm-ld: undefined symbol), so the script builds it with emcc into an object and links that. MTMD_VIDEO stays off (it needs ffmpeg); audio sources compile but were not run.
  • The build writes web/wasm/llm-wasm.build.txt with the exact llama.cpp commit, so results stay attributable while upstream moves weekly.

Threads (optional, cross-origin isolation)

There are two builds of the same sources:

build flag works on threads
web/wasm/llm-wasm.* scripts/build-wasm.sh any page 1 (ggml clamps it)
web/wasm/llm-wasm-mt.* MT=1 scripts/build-wasm.sh pages with COOP/COEP 1..N (llmjs_set_threads)

The pthread build needs SharedArrayBuffer, which browsers expose only to a cross-origin isolated page. test/serve.py is http.server plus the two headers (Cross-Origin-Opener-Policy: same-origin, Cross-Origin-Embedder-Policy: require-corp); GitHub Pages and most static hosts cannot set them, which is why the single-thread module stays the default and the worker picks the module at runtime:

const THREADED = typeof SharedArrayBuffer === "function" && self.crossOriginIsolated === true;
factory = THREADED ? await import("./wasm/llm-wasm-mt.js") : await import("./wasm/llm-wasm.js");

The thread count is a launch parameter like context length or KV type: it goes into the context signature, so changing it rebuilds llama_context (Reload engine), and it also drives mtmd's image encode/decode (p.n_threads).

Measured in Node (test/threads-test.mjs, LFM2.5-VL-450M Q4_K_M, greedy, same prompt and output):

1 thread:  8.22 tok/s   (2921 ms for 24 tokens)
4 threads: 26.36 tok/s  ( 911 ms for 24 tokens)   → 3.21× wall time
prompt eval: 1605 ms → 1032 ms

Threads are pre-spawned, and that is not optional. pthread_create() called from inside a wasm call deadlocks in a browser: the calling thread blocks in Atomics.wait while the worker it just asked for has to fetch and boot the same ES module, and it never gets to report ready. Node is fine because worker_threads start differently. So the module is built with -sPTHREAD_POOL_SIZE set to a JS expression that runs at module init:

"-sPTHREAD_POOL_SIZE=${PTHREAD_POOL_SIZE:-Math.max(1, Math.min((typeof navigator !== 'undefined' && navigator.hardwareConcurrency) || 4, 8))}"

The pool therefore adapts to the machine that loads the page (capped at 8), and the thread field is capped to the same number. Choosing fewer threads than the pool costs nothing at run time — the extra workers sit idle on shared memory and model load stays ~10 s. Asking for more than the pool size reproduces the old silent hang, which is why the field can't go above it.

Measured in headless Chromium, 450M Q4_K_M, 8 new tokens, same prompt:

threads decode vs 1 thread
1 0.98 s, 8.18 tok/s 1.0x
2 0.58 s, 13.74 tok/s 1.68x
4 0.43 s, 18.56 tok/s 2.27x

The page now defaults to min(cores, 4) threads when the page is cross-origin isolated, and ?threads=N still forces a specific count (?threads=1 for the old single-thread behaviour). Without COOP/COEP it falls back to the single-thread module.

Useful facts found while building it:

  • -pthread on the CMake flags defines __EMSCRIPTEN_PTHREADS__, which is what stops ggml_graph_plan() from forcing n_threads = 1.
  • the MT build caps memory at 2 GB (safe SharedArrayBuffer size across browsers) instead of the single-thread build's 4 GB.
  • Emscripten warns that -pthread + ALLOW_MEMORY_GROWTH makes the JS side of growth slow; the model load grows the heap once, before generation.
  • vendor/hash/sha256.c still has to be compiled as C (emcc) in both builds.

Run

Headless (this is what was verified here):

node test/node-test.mjs models/LFM2.5-VL-450M-Q4_K_M.gguf "Hi" 48
NO_CHAT=1 node test/node-test.mjs ...            # raw completion, no template
CHAT_TEMPLATE=chatml node test/node-test.mjs ... # force a built-in template

Browser demo 

python3 -m http.server 8000 --directory web

Then open http://localhost:8000 (serve it — file:// cannot load ES modules or fetch the wasm; any static server works, no CORS/COOP headers needed for the single-threaded build).

The page gives you a small model library: any number of models, each with a uuid record, an editable display name, its size, its SHA-256 and a state (complete, partial, needs file).

  • from a URL — 4 MiB chunks into IndexedDB, progress bar, Pause (aborts the fetch, keeps stored chunks), Resume (continues with an HTTP Range request), delete (drops the record + every chunk). If the server ignores Range and returns 200, the download restarts cleanly instead of corrupting the file. Adding the same URL twice reuses the record instead of storing two 200 MB copies.
  • from this computer — a GGUF you already have. By default nothing is copied: the File is handed to the worker and read in 4 MiB slices straight into the wasm FS (and the SHA-256), so the browser holds exactly one copy — the one already on your disk. After a reload the record reads needs file and asks you to pick it again; nothing is re-downloaded. Tick keep a copy in browser storage if you would rather have it live in IndexedDB (still one copy, and the original file is not needed afterwards).
  • sha256 — computed once per model, streamed from wherever its bytes live (crypto.subtle.digest() is one-shot, so web/sha256.js is a small incremental SHA-256), shown in the row (full value on hover) with a copy hash button so it can be compared with the publisher's. Paste an expected hash when adding and the row marks it ✓ or ≠ expected.
  • pick the model from a dropdown — the list of library models; Load runs the selected one (a local file asks for its file again if it was not copied), and Reload engine frees the model and the wasm backend and loads it once more, so switching models or recovering from a bad state does not need a page reload.
  • tabs — Chat playground, Raw completion, Chat template, Models. The sampling/limits rail sits next to the two run tabs, so both modes share it.
  • chat playground — the message list is the conversation: add, remove and reorder system / user / assistant turns and write them yourself, then Generate. The assistant turn is appended before the first token and streams into place, so a human-written transcript can be continued turn by turn. Stop in the top bar interrupts and keeps whatever already arrived; the turn stays editable afterwards. (Stepwise generation, below, is what makes that possible without threads.)
  • tools — paste an OpenAI tools array in the chat tab's tools box and it is handed to the template for rendering. The reply is then run through the PEG parser built for that exact prompt, so a model that answers with <function name="get_weather"><param name="city">Paris</param></function> (MiniCPM5's format) produces a tool-call card: name, id, and the arguments as a JSON tree, plus Add tool response which appends a tool message already carrying that tool_call_id. The call markup is removed from the transcript text, which keeps what the model said before calling. Reasoning is deliberately not extracted: it stays in the assistant text inside its think block, exactly as the model emitted it. The JSON tree is a small local renderer, not a CDN library, so the lab still works offline.
  • export — Export JSON downloads [{ role, content }, …] in OpenAI shape (assistant turns keep tool_calls, tool turns keep tool_call_id, and a message with images gets the multi-part content list with {"type":"image_url","image_url":{"url":"data:image/png;base64,…"}} parts); Export Markdown downloads the same transcript with ### SYSTEM / ### USER / ### ASSISTANT / ### TOOL (call_id) headers, fenced JSON for calls and a literal << image >> where an image was. No conversation library, no cloud: it is a lab, and the file is the storage.
  • images — an uploaded GGUF can be flagged mmproj (checkbox in either ingest form; it is then left out of the text-model dropdown). In the engine rail, tick load multimodal projector and pick the projector that is loaded with the next model load / Reload engine. In the playground, the 🖼 button on a message (or a drag & drop onto it) attaches PNG/JPEG/WebP images as mini-avatar thumbnails, each with its own ✕ to remove it. The message goes through the template with one media marker per image and mtmd replaces each marker with that image's token sequence; the template preview deliberately shows the marker itself, since image tokens sit below the rendered text.
  • raw completion — no template at all: the text is the prompt, which is what a base/completion model expects (and a chat model then answers as plain text).
  • engine — the launch parameters the rail fixes when a context is created: context length, KV cache type for K and V (f16, q8_0, q4_0), flash attention (auto/off/on), the unified KV cache, and the optional mmproj. Reload engine applies a change (a llama_context cannot be resized in place); loading a model also applies the current values. A quantized V cache switches flash attention on, because llama.cpp refuses that combination otherwise. Parallel sequences stay at one: extra slots would only duplicate a cache a single user cannot fill. This build has no server-style prompt-cache RAM pool (that 8 GiB default is common_params::cache_ram_mib, used by llama-server) — memory is the model plus the KV cache, which grows with context length and shrinks with the quantized types.
  • sampling / limits — temperature, top_p, top_k, repetition penalty and its window, seed (-1 = random), max new tokens, and stop sequences (one per line, \n/\t expanded). Stops are matched in C++ against the decoded text, so they never appear in the output; the last len-1 bytes are held back until a match is impossible, and the finished text comes from llmjs_last_output() instead of being reassembled from the stream.
  • chat template tab — the textarea is prefilled from the GGUF; unchecking "use the template shipped in the GGUF" makes it editable and the override wins for the next run. A parameters JSON box adds template kwargs (e.g. {"enable_thinking": true}), and the enable_thinking checkbox in the rail writes that key for you; both the preview and the next run use them. Preview with the playground messages renders the current transcript without generating — the fastest way to see why a model answers badly. The stats line reports which path ran (template: jinja, jinja:override, builtin:<name>, raw).
  • llama.cpp log — collapsible, shows the wasm build version and load messages.

The two ingest paths are deliberately separate in the UI (pasting a URL and picking a file are different gestures, and only the file path has a "copy or read in place" decision) but converge immediately: one record type, one load path, one hash, one delete.

Everything stays local: no server-side component, no model upload, and the cached model is visible/removable from the page (browser quota shown at the top).

web/llm.js keeps the original API:

const app = new LLM('GGUF_CPU', modelUrl,
    () => {},                 // init
    (text) => {},             // streaming tokens
    (stats) => {},            // complete: { tokens, ms, tokens_per_second }
    (err) => {},              // NEW optional: error
    (loaded, total) => {},    // NEW optional: download progress
    (line) => {});            // NEW optional: llama.cpp logs
app.load_worker({ context_size: 1024,
                  kv: { type_k: 'q8_0', type_v: 'q8_0', flash_attn: -1,
                        kv_unified: true, n_parallel: 1 },
                  mmproj_id: 'uuid-of-a-flagged-mmproj' });  // optional projector
app.run({ prompt: 'Hi', chat: true, max_token_len: 48, temp: 0.7 });   // old shape, still works

// chat completion over a whole transcript (what the playground sends)
app.run({
  mode: 'chat',
  messages: [{ role: 'system', content: 'You are terse.' },
             { role: 'user', content: 'Hi' }],
  temp: 0, top_p: 0.9, top_k: 40, repeat_penalty: 1.1, repeat_last_n: 32,
  seed: 42, max_token_len: 64, stop: ['\n\n'], chat_template_source: '',
  tools: JSON.stringify(tools),       // OpenAI tools array, handed to the template
  template_kwargs: '{"enable_thinking": true}',
  images: ['data:image/png;base64,…'],  // one per <__media__> marker, in order
});
// stats.parsed = { content, reasoning_content, tool_calls: [{ id, name, arguments }] }
// null when the template's parser did not recognise the reply
// stats.run_stats = { prompt_tokens, reused_tokens, prompt_ms, n_ctx }
app.run({ mode: 'completion', prompt: 'The capital of France is' });
app.stop();   // keeps the tokens already streamed, completes with interrupted: true

run() is stepped inside the worker (llmjs_gen_begin_* → llmjs_gen_step → llmjs_gen_end): one token per turn of the event loop. That is what allows a stop message to be received at all in a single-threaded build, and it keeps tokens flowing to the page as they are produced.

Model library — a uuid record per model; url and local-file ingest produce the same record and load through the same code path:

app.add_url_model({ url, name: 'LFM 450M', expected_sha256: '' });
app.add_file_model({ file, name: '', keep_copy: false });  // nothing copied by default
app.load(id);
app.hash_model(id);
app.delete_model(id);

Deviations from llm.js: the GGUF is downloaded inside the worker (posting a 219 MB ArrayBuffer across the thread boundary would double memory), it is cached in IndexedDB rather than the Cache API so pauses survive reloads, and the new callbacks/progress exist because there is no stdout to scrape anymore. Extra methods: pause(), resume(), load(id), add_url_model(), add_file_model(), hash_model(id), delete_model(id), list_models(), plus an on_event hook for worker messages (MODEL_ADDED, MODEL_UPDATED, PAUSED, DELETED, …).

Embedding the playground (minimal postMessage API)

The page can be used as an iframe by a host page — the host keeps its own UI and borrows this one's model library and wasm engine:

<iframe src="http://localhost:8010/llmjs-latest/web/" width="900" height="700"></iframe>
<script>
  const frame = document.querySelector("iframe").contentWindow;
  addEventListener("message", (e) => {
    if (e.data.type === "llmjs:ready") backend = e.data;            // model loaded
    if (e.data.type === "llmjs:token") append(e.data.text);        // streaming
    if (e.data.type === "llmjs:done") finish(e.data.text, e.data.stats);
    if (e.data.type === "llmjs:error") fail(e.data.error);
  });
  // after a model is loaded in the frame:
  frame.postMessage({ type: "llmjs:run", id: 1,
    messages: [{ role: "user", content: "Say hi in five words." }],
    max_tokens: 24 }, "*");
</script>

Protocol (this iteration: chat completion only):

direction message
parent → frame llmjs:run { id, messages | prompt, images?, max_tokens?, temperature?, top_p?, top_k?, repeat_penalty?, repeat_last_n?, seed?, stop?, tools?, template_kwargs? }
parent → frame llmjs:stop (keeps what streamed, like the Stop button)
frame → parent llmjs:api { version } once the frame is ready
frame → parent llmjs:ready { model, n_ctx, threads } when a model finishes loading
frame → parent llmjs:token { id, text } per token, llmjs:done { id, text, stats }, llmjs:error { id, error }

A model still has to be loaded inside the frame first (that is where the library and the wasm module live), one run at a time, and the frame accepts messages from window.parent regardless of origin — fine for a local lab, but a public deployment should check event.origin before answering. The host page needs the frame's URL to be same-origin or CORS/COOP-friendly; the threaded build's COOP/COEP headers do not leak to the parent page.

What actually broke between llm.js and today's llama.cpp

llm.js patched the root CMakeLists.txt to add examples/main and called Module.callMain(["-m", ..., "-p", ...]), filtering main: / *_model_load: lines out of stdout with regexes. Current status of each assumption:

  1. examples/main no longer exists. The patch does not apply; there is no CLI to embed. (The current CLI lives in tools/cli, is excluded from Emscripten builds, and pulls in common/ + arg.cpp.)
  2. callMain is the wrong shape anyway. Token streaming now comes from llama_sampler_sample → llama_token_to_piece → an EM_JS callback, so output is exact and no regex filtering is needed.
  3. No buffer loading. llama_model_load_from_buffer is gone; only llama_model_load_from_file, ..._file_ptr, ..._splits. So the GGUF still goes into MEMFS and is loaded from /models/model.gguf.
  4. Renames the old code hits immediately: llama_new_context_with_model → llama_init_from_model; llama_n_vocab → llama_vocab_n_tokens (vocab is an opaque object); llama_kv_cache_clear → llama_memory_clear(llama_get_memory(ctx), true); llama_sample_* → llama_sampler_* chain (llama_sampler_chain_init + top_k/top_p/temp/penalties/dist); llama_token_to_piece(vocab, tok, buf, len) gained lstrip/special; llama_model_params lost use_mmap/use_mlock (now enum llama_load_mode).
  5. Sampling is explicit. Nothing is sampled for you: build a chain per run, llama_sampler_sample, then llama_sampler_accept and decode the token.
  6. Threads. ggml forces n_threads = 1 for __EMSCRIPTEN__ without pthreads (ggml/src/ggml-cpu/ggml-cpu.c), and the wrapper sets n_threads/n_threads_batch = 1 so the thread-pool path is never entered.
  7. Chat templating is a different subsystem now — see the next section.
  8. --json-schema is not available to this build: JSON-schema → GBNF lives in common/json-schema-to-grammar.cpp, which we do not compile. grammar (GBNF) works via llama_sampler_init_grammar.

The chat-template rabbit hole (verified, not assumed)

This model ships a Jinja tokenizer.chat_template (ChatML-style: <|im_start|>user … <|im_end|>, think blocks, tool calls, image content). That matters because llama.cpp has two templating stacks:

  • libllama's llama_chat_apply_template() — a fixed list of ~60 built-in presets (chatml, llama3, mistral-v1, gemma, phi3, …). It does not run Jinja. This is all you get if you link only libllama.
  • common/chat.cpp + common/jinja (minja) + the PEG auto-parser (docs/autoparser.md, common/chat-auto-parser.*, common/chat-peg-parser.*) — the real Jinja engine and output parser. This is what llama-cli / llama-server use, and it drags in common/sampling, common/json, json-schema-to-grammar, peg-parser, … i.e. the whole common library.

Relevant flags, read from common/arg.cpp (not from memory):

flag behaviour
--jinja / --no-jinja enable the Jinja engine (default off)
--chat-template JINJA_TEMPLATE custom template; without --jinja only commonly used (built-in) templates are accepted
--chat-template-file FILE same, from a file
--chat-template-kwargs template kwargs (e.g. thinking/preserve reasoning)
--skip-chat-parsing force pure content parsing even with a Jinja template
--reasoning-format how think blocks are returned

The trap: passing the model's own Jinja string to llama_chat_apply_template() aborts the wasm module with a bare RuntimeError: Aborted(undefined). Reason, from the source: llm_chat_detect_template() (src/llama-chat.cpp:89) does try { llm_chat_template_from_str(tmpl) } catch (const std::out_of_range &), i.e. it uses a C++ exception for control flow. Emscripten builds disable exception handling by default, so the throw never reaches the catch — it calls __cxa_throw → __Unwind_RaiseException → abort(). Passing the short name "chatml" hits the map directly and never throws. (llm.js never hit this: it had no template support at all.)

The fix: link the engine, not the presets. common/jinja is a Jinja engine of its own (lexer/parser/runtime), and it needs only common/json.cpp + common/unicode.cpp next to it — not the whole common library (no sampling, no PEG parser, no autoparser). This build compiles those eight files with -fexceptions and links with -sDISABLE_EXCEPTION_CATCHING=0, which is exactly what makes the try { … } catch (const std::out_of_range &) above work instead of calling abort(). So the model's own template is the primary path, rendered by the same engine llama-server --jinja uses:

  1. read tokenizer.chat_template from the GGUF (llama_model_chat_template),
  2. render it with messages, bos_token, eos_token and add_generation_prompt in scope (jinja::global_from_json + jinja::runtime),
  3. strip a leading BOS / trailing EOS when the tokenizer adds them itself — the same rule common_chat_template_direct_apply_impl() applies, which is why the rendered text does not end up with two <|startoftext|> tokens,
  4. let the caller override the template source entirely (the demo has an "advanced · chat template" box for this, handy for a model with a broken or missing template),
  5. fall back to the closest built-in preset (<|im_start|> → chatml, <|start_header_id|> → llama3, <start_of_turn> → gemma, [INST] → mistral-v1, <|im_sep|> → phi3) if the template is missing or throws, and to raw completion if nothing fits.

The chain is reported back to the page: template: jinja, jinja:override, builtin:<name> or raw.

Verification (byte-level and token-level)

test/chat-template-test.mjs renders the model's template inside wasm and compares it with the built-in preset, then tokenizes both:

1. GGUF template: 3836 bytes, detected preset 'chatml'
2. Jinja render is byte- and token-identical to builtin chatml
   "<|im_start|>user\nHi<|im_end|>\n<|im_start|>assistant\n"
3. caller-supplied template renders verbatim
4. 4-message conversation renders (168 bytes)
5. broken template reported, module still alive: "Unexpected token: %} of type 9"
6. generated 2 tokens via the Jinja path; broken override fell back to builtin:chatml

The older pair below checks the preset fallback against Python's jinja2: test/verify-chat-template.mjs renders through the wasm module; test/verify-chat-template.py parses the GGUF metadata directly (small GGUF reader), renders the actual Jinja template with Python jinja2, and diffs the two; test/verify-tokens.mjs then proves the token streams are equal.

wasm  (built-in chatml): "<|im_start|>user\nHi<|im_end|>\n<|im_start|>assistant\n"
jinja2 (model template): "<|startoftext|><|im_start|>user\nHi<|im_end|>\n<|im_start|>assistant\n"
                                                                  ^ only difference: BOS
TOKENS MATCH: identical token ids      # tokenize(ref, add_special=false) == tokenize(ours, add_special=true)

The BOS difference is expected and harmless: the Jinja template emits {{- bos_token -}} itself, while our built-in render relies on the tokenizer, which adds BOS because tokenizer.ggml.add_bos_token = true. Verified equal at the token level, not argued.

Tools and the output parser

The same common/chat.cpp stack that renders the prompt also parses the reply: for each template it builds a PEG grammar in common/parsers/*.cpp (MiniCPM5's is minicpm5.cpp) and common_chat_peg_parse() splits the text into content, reasoning_content and tool_calls. This build compiles that half too (chat*.cpp, peg-parser.cpp, parsers/*.cpp, json-schema-to-grammar, reasoning-budget), which is why the wasm grew from 3.9 MB to 5.1 MB.

Two details that are easy to get wrong, both verified here rather than assumed:

  • common_chat_parser_params copies the format and generation prompt but not the PEG arena. common_chat_peg_parse(arena, input, is_partial, params) takes the arena explicitly; pass g_chat_params->parser or everything parses as content-only.
  • the parser expects the generation prompt in front of the generated text (effective_input.prepend(params.generation_prompt)), exactly as llama-server feeds it. Whatever the template appended past the parser's literal — an opened think block, for instance — lands in the content, so the wrapper strips the generation-prompt tail from the start of the content.

Reasoning is not extracted for display: run() returns it inside content, think tags included, which is what the playground shows. Tool calls are extracted because they are structured data the page renders as JSON.

Special tokens matter here: the token pieces are decoded with llama_token_to_piece(..., special = true), because <function / <param are special tokens in MiniCPM5's vocab and a special = false decode silently drops them, turning a perfectly good tool call into name="get_weather">.

Multimodal content goes through mtmd instead: llmjs_set_images() takes the data URLs, the same llmjs_gen_begin_chat() sees the markers in the message text, and mtmd_tokenize() turns each marker into that image's token sequence before the prompt is evaluated. The projector itself is loaded by llmjs_mmproj_load() (llama-server's --mmproj): a second GGUF from the same library that is flagged multimodal by the user, never auto-detected.

Debugging notes (WASM-specific)

  • Exceptions abort silently. Any throw in llama.cpp kills the module with Aborted(undefined). Bisect by logging before/after each call from C++ (an EM_JS log helper works); that is how the chat-template crash above was localised to the first generate call.
  • EM_JS bodies are macro arguments — a top-level comma breaks the macro.
  • MEMFS + canOwn: FS.createDataFile("/models", "model.gguf", bytes, true, false, true) takes ownership, so the 219 MB buffer is not copied again. Weights still get read into the wasm heap, so the heap grows to a few hundred MB on load.
  • Wasm32 vs wasm64: -sMAXIMUM_MEMORY=4GB is the wasm32 ceiling; without ALLOW_MEMORY_GROWTH the 64 MB initial heap would fail on model load.
  • HEAPU8/UTF8ToString are not exported by default — add them to EXPORTED_RUNTIME_METHODS if JS needs them.
  • Single-thread n_batch=512 reserve costs ~139 MiB of compute buffer. Lower n_batch if the browser is memory-constrained.

Tests / what is verified

command needs checks
node test/node-test.mjs [model] [prompt] [n] model full wasm load + generation, tok/s, output
node test/download-test.mjs nothing chunking, pause at chunk boundary, Range resume, restart when the server ignores Range
node test/sha256-test.mjs nothing streaming SHA-256 vs node's crypto: every padding boundary, uneven slices, known vectors
npm i --no-save fake-indexeddb && node test/storage-test.mjs fake-indexeddb model library CRUD, chunk iteration, v1 → v2 migration
node test/chat-template-test.mjs [model] model GGUF Jinja template vs the verified preset (byte + token identical), custom override, broken template falls back instead of aborting
node test/chat-generate-test.mjs [model] model stepwise chat generation over a multi-turn list, streamed text == final output, greedy reproducibility, stop-sequence truncation, interrupt keeps the partial text, raw mode, max-new-tokens
node test/tool-call-test.mjs [model] MiniCPM5-1B tools rendered into the prompt, enable_thinking changes it, a canned and a live reply parse into name + JSON arguments, text before the call is kept, assistant tool_calls + tool result render back
node test/kv-cache-test.mjs [model] model ctx/KV options reported by llmjs_ctx_config(), repeated-prompt reuse (6.5 s → 0.3 s), reply reuse on the next turn, identical output with a cold cache, quantized K/V + flash attention rule, n_ctx really applied; adaptive for hybrid/RNN memories
node test/multimodal-test.mjs [model] [mmproj] LFM2.5-VL + mmproj projector reports vision support and the media marker, an image prompt evaluates its image tokens (121 prompt tokens vs 19 for the same text) and answers about the picture, one image per marker is enforced, image runs are refused without a projector while text still works
node test/threads-test.mjs [model] [threads] mt build the pthread module really runs N threads: identical greedy output, measured tok/s at 1 vs N (3.2× at 4 threads on 4 vCPU)
python test/serve.py nothing static server with COOP/COEP, needed for SharedArrayBuffer / --threads
node test/verify-chat-template.mjs … && python3 test/verify-chat-template.py … jinja2 built-in chatml render vs the model's real Jinja template
node test/verify-tokens.mjs … model the two prompts tokenize to identical ids
python test/browser-test.py --model-url … | --file … [--mmproj …] playwright the real page in headless Chromium: both ingest paths → sha256 → load → generate; --pause-mib N pauses/resumes, --file asserts nothing was copied, plus a tool row, both exports (downloads are parsed back) and a template kwarg reaching the preview; --mmproj adds a flagged projector, reloads with it, attaches an image (thumbnail add/remove), generates about it, checks the preview marker and the multi-part JSON / << image >> Markdown exports

Verified here: clean wasm build, 48-token generation, raw + chat paths, prompt render byte diff, token-id equivalence, download pause/resume, the model library (uuid records, rename, multi-model delete, v1 → v2 migration, chunk iteration), streaming SHA-256 against node's crypto, HTTP serving of every demo asset, and the demo page driven end to end in headless Chromium over both ingest paths (download: 218.7 MiB into 55 chunks, GGUF magic intact, paused and resumed via Range; local file: hashed and loaded with zero chunks written to IndexedDB).

KV reuse and the launch parameters were verified with test/kv-cache-test.mjs on MiniCPM5-1B (transformer) and LFM2.5-VL-450M (hybrid): a repeated 27-token prompt went from 6.5 s to 0.3 s of prompt eval with 26/27 tokens reused, a cleared cache gave byte-identical greedy output, q8_0 K/V ran with flash attention forced on, q8_0 K with f16 V kept it off, and n_ctx=256 was really applied and invalidated the old cache. On LFM2.5 the hybrid memory reused 32/46 tokens on the next turn (append-only, as expected).

The tabbed page was verified the same way: the chat playground appends an assistant turn and streams into it (stats: 24 tokens … · template: jinja), Stop reported · stopped with the partial text kept, the template tab previews the GGUF template (<|im_start|>user … <|im_end|>), an override was used (template: jinja:override), raw completion reported template: raw, andthe engine reloaded from the top bar. A tool message row survived with its tool_call_id, the JSON export came back as OpenAI messages ending in that tool turn, the Markdown export contained ### TOOL (call_test_1) and ### USER, and the template preview accepted a parameters JSON kwarg.

Multimodal was verified with LFM2.5-VL-450M plus its mmproj-LFM2.5-VL-450m-Q8_0.gguf (same Hugging Face repo, mmproj file): in Node the projector loaded (vision=true, marker <__media__>), a 64×64 PNG prompted with one marker evaluated 121 prompt tokens against 19 for the same text and the model answered "The left half of this image is red"; a marker/image count mismatch was refused (-2) instead of mis-rendering, and an image run without a projector was refused (-3) while text runs kept working. In the browser the same flow used the real page: flag the file as mmproj, reload with load multimodal projector (status line gains mmproj … (vision)), attach the image through the 🖼 button, remove and re-attach the thumbnail, generate ("The left half of this image is a red color.", 121 prompt tokens in 24 s — image encoding dominates), see <__media__> in the template preview, and export the conversation: one multi-part message with data:image/png;base64,… in JSON and << image >> in Markdown.

Tools and the parser were verified with MiniCPM5-1B (MiniCPM5-1B-Q4_K_M.gguf) in test/tool-call-test.mjs: the tools array reaches the template (943 vs 90 bytes), enable_thinking toggles the open think block, and a real 15-token generation produced <function name="get_weather"><param name="city">Paris</param></function>, which the model's own parser turned into one tool call with {"city": "Paris"}.

Docs consulted (both projects)

llm.js — docs/BUILD.md (build flow), docs/api_guide.md (the API preserved here), scripts/build-llama-cpp-wasm.sh + patches/00-llama-cpp-enable-main.patch (exactly what broke), src/web-workers/llamacpp-cpu-worker.js (stdout scraping).

llama.cpp — the docs are scattered; these were the useful ones: docs/build.md (WASM/WebGPU + WASI sections, the latter has the single-thread -DGGML_OPENMP=OFF -DLLAMA_BUILD_COMMON=OFF recipe), .github/workflows/build-wasm.yml (the flags upstream actually builds with), docs/autoparser.md (PEG auto-parser design), docs/models.md / models/templates/README.md (templates), tools/mtmd/README*.md (multimodal), docs/function-calling.md, plus source-of-truth include/llama.h, src/llama-chat.cpp, common/arg.cpp, ggml/src/ggml-cpu/arch/wasm/ and ggml/src/ggml-cpu/ggml-cpu.c.

Limitations / next steps

  • Multimodal is vision-projector only, and untested beyond images. mtmd is compiled in and loads any .gguf mmproj, but only LFM2.5-VL's vision projector was run; audio projectors compile and were never tried, and video is off (it needs ffmpeg). Images are CPU-bound here: the 64×64 test image took ~12 s to encode and ~6 s to decode in wasm, so a photo costs tens of seconds before the first token. Image runs bypass the text prefix cache (the token stream changes with the image), so a subsequent text turn starts cold.
  • No json_schema (compiling common/json-schema-to-grammar.cpp would add it). GBNF grammar works.
  • No reasoning-effort control. MiniCPM5 advertises supports_reasoning_effort: false; thinking is on/off via her enable_thinking template kwarg only. The sampling stack with thinking budgets is not compiled here, so the control is left out rather than faked.
  • Stop granularity is one token, and stop sequences are text-level: they are matched on decoded text (that is why a stop can appear immediately, not after the next token). There is no token-id ban list yet.
  • Prefix reuse depends on the template and the memory type. The wrapper keeps the tokens that sit in the KV cache and re-evaluates only what a new prompt adds. A repeated prompt drops from 6.5 s to 0.3 s of prompt eval (26/27 tokens reused, test/kv-cache-test.mjs). Two limits are inherent: a reasoning template injects its think block only in the generation prompt, so with thinking on the reply after that point cannot be reused; and recurrent / hybrid memories (Mamba, LFM2's conv+attention) cannot roll their state back, so they only reuse the append case — never a prefix that needs trimming. That case is the multi-turn one, and it still helps: 32/46 tokens on LFM2.5.
  • KV memory is only bounded by n_ctx. There is no separate cache pool in this build (the 8 GiB --cache-ram default belongs to llama-server); a context of N tokens costs the model's per-token KV size times N, times two if K and V stay f16. The quantized types in the rail are the lever.
  • The parser only knows the formats llama.cpp ships (common/parsers/). A model whose reply format has no parser there still generates fine; its reply just stays plain text (stats.parsed is null or has no tool calls).
  • Exceptions cost size and some speed: linking the Jinja engine raised the wasm from 3.2 MB to 3.9 MB and requires -fexceptions -sDISABLE_EXCEPTION_CATCHING=0; a browser without wasm exception support is not a concern (Emscripten's scheme is JS-based), but the JS glue is larger and every try/catch in llama.cpp now works instead of aborting.
  • Threads are opt-in and experimental in the browser. A pthread build exists and is 3.2× faster in Node at 4 threads (test/threads-test.mjs), but the first multi-thread decode stalls in headless Chromium, so the page keeps one thread by default and ?threads=N turns the pthread module on. Threads also need cross-origin isolation, i.e. host-controlled COOP/COEP headers. Wasm SIMD is already enabled; the single-thread baseline stays ~8 tok/s on this container.
  • A local model has to be re-picked after a reload (one click, no network) unless you tick keep a copy in browser storage. Keeping a FileSystemFileHandle in IndexedDB would remove that click, but the API is Chromium-only and brings a permission prompt, so it is left out for now.
  • Hashing costs one read of the model (~2-4 s for 219 MiB here) and happens once, on add; the digest is cached in the record.
  • One browser only — the demo is checked in headless Chromium via Playwright (test/browser-test.py); Firefox and Safari were not run, and the IndexedDB quota/eviction behaviour on a nearly-full disk was not exercised.
  • Similar, more actively maintained projects to compare against: wllama (ngxson) and llama.cpp's own WebGPU example.

This bundle

2de6f90 — engine sampler rebuilt from the chat template, LFM2.5 tool-call parsing, image prompts that grow the context to fit, and structured output (JSON Schema or GBNF) selected from the sidebar. Serve with COOP/COEP so the threaded module is used.