LiteRT-LM Web Runtime (@litert-lm/core 0.18.3) & Python 0.18.3 Wheels

WebAssembly + WebGPU runtime binaries, the matching @litert-lm/core TypeScript SDK, and prebuilt Python 0.18.3 wheels for running Gemma 4 (E2B, E4B, 12B, 26B-A4B, 31B) in the browser and natively with:

  • Multimodal Prefill Acceleration (lever4b): Vision .front() chunk-boundary synchronization fix and calibrated group_size.h 512 dispatch cost for accelerated prompt prefill and visual token processing.
  • 12B MQA Metal Shader Compilation Hang Fix: Bounded shader unroll sequence size ($\le 8$ in fully_connected.cc, $\le 4$ in gpu_model_builder.cc) and capped max_spatial_w, eliminating the MTLCompilerService infinite unroll loop hang on Apple Silicon (compilation completes in 0.81s; 12B MQA decodes at 24.3 tok/s on Apple Silicon M5 Pro).
  • WebGPU Subgroup Matrix Acceleration: Opportunistic detection and enablement of chromium-experimental-subgroup-matrix in Chrome 137+ for high-throughput WebGPU GEMM kernels.
  • Decision API (decide()): Fast, single-token constrained choice and hierarchical decision trees (e.g. 62 single-token classification labels, tool routing, safety gating) evaluated directly on pre-softmax / post-softmax logits without free-form decode latency.
  • Text decode with shared-KV neural Multi-Token Prediction (MTP) speculative decoding (GPU_ARTISAN backend, gamma 1-3, greedy token-for-token equivalence with standard decode).
  • Multimodal vision (SigLIP encoder on WebGPU, 70 / 140 / 280 / 560 / 1120 token budgets, zero-copy RGBA8 ImageData preprocessing, per-budget executor memory release; fused online-softmax ViT self-attention that never materializes the score tensor, with full-length query dispatch).
  • Multimodal audio (Conformer encoder, WebGPU audio adapter with zero-copy GPU tensor buffer sharing saving 15–25% audio chunk latency, streaming markers, zero-copy audio ingestion).
  • Streamed model loading for 2-20 GB .litertlm bundles under wasm32 (MAXIMUM_MEMORY=4GB), with section filtering and inter-section padding discard.
  • Native macOS Apple Silicon Metal & Linux Wheels: Prebuilt 0.18.3 wheels with full LlmMetal, LlmGpuArtisanExecutor, MtpOrchestrator, and multimodal vision/audio support.

Repository layout

Exactly one runtime and package set is kept on main. Every release is also addressable by commit SHA (.../resolve/<commit>/wasm/), and older releases remain available through their commits.

Path Contents
wasm/ -c opt Emscripten builds of the four runtime variants, under both internal names (litertlm_wasm_internal, litertlm_wasm_compat_internal, litertlm_wasm_asyncify_internal, litertlm_wasm_compat_asyncify_internal) and feature aliases (threaded_simd, threaded_nosimd, simd, nosimd); litertlm_wasm.{js,wasm} / litertlm_wasm_compat.{js,wasm} alias the JSPI threaded builds. litertlm_wasm_internal.js.symbols is the symbol map.
packages/litert-lm-core-0.18.3.tgz The @litert-lm/core npm package (ES modules + typings + the same wasm/ binaries) produced from the same source revision.
wheels/ Prebuilt Python 0.18.3 wheels: litert_lm_api for macosx_12_0_arm64, manylinux_2_27_x86_64, manylinux_2_27_aarch64, plus litert_lm_builder and umbrella litert_lm.

Variant selection: internal = JSPI + threads + SIMD (Chrome 137+), compat_internal = JSPI + threads without SIMD, asyncify_internal / compat_asyncify_internal = Asyncify fallbacks for browsers without JSPI.

Web / TypeScript SDK Usage

Install @litert-lm/core 0.18.3 from the tarball or npm:

npm install https://huggingface.co/jfan/litert-lm-web-runtime/resolve/main/packages/litert-lm-core-0.18.3.tgz
import {loadLiteRtLm, Backend} from '@litert-lm/core';

const WASM_BASE =
  'https://huggingface.co/jfan/litert-lm-web-runtime/resolve/main/wasm/';
const litertlm = await loadLiteRtLm(WASM_BASE);
const engine = await litertlm.createEngine({
  modelPath: 'https://huggingface.co/jfan/gemma-4-e4b-web-mtp-vision-litert-lm/resolve/main/gemma-4-E4B-it-web-mtp-vision-audio.litertlm',
  backend: Backend.GPU_ARTISAN,
  speculativeDraftTokens: 3,   // MTP gamma; omit for standard decode
});

// Single-token classification and decision making
const decision = await engine.decide({
  prompt: 'Classify this input: "Flight booking request"',
  choices: ['travel', 'finance', 'support', 'general'],
});
console.log('Selected class:', decision.choice);

Pin a commit SHA instead of main in production so the binaries and the SDK stay in lock-step.

Python Wheels Usage

Download and install the Python 0.18.3 wheel for your platform:

macOS ARM64 (Apple Silicon / Metal)

pip install https://huggingface.co/jfan/litert-lm-web-runtime/resolve/main/wheels/litert_lm_api-0.18.3-py3-none-macosx_12_0_arm64.whl
pip install https://huggingface.co/jfan/litert-lm-web-runtime/resolve/main/wheels/litert_lm-0.18.3-py3-none-any.whl

Linux x86_64

pip install https://huggingface.co/jfan/litert-lm-web-runtime/resolve/main/wheels/litert_lm_api-0.18.3-py3-none-manylinux_2_27_x86_64.whl
pip install https://huggingface.co/jfan/litert-lm-web-runtime/resolve/main/wheels/litert_lm-0.18.3-py3-none-any.whl

Linux AArch64

pip install https://huggingface.co/jfan/litert-lm-web-runtime/resolve/main/wheels/litert_lm_api-0.18.3-py3-none-manylinux_2_27_aarch64.whl
pip install https://huggingface.co/jfan/litert-lm-web-runtime/resolve/main/wheels/litert_lm-0.18.3-py3-none-any.whl

Python Inference Example

import litert_lm

# Initialize engine with Apple Silicon Metal or Vulkan/OpenCL GPU backend
# and speculative decoding enabled.
engine = litert_lm.Engine(
    model_path="/path/to/gemma-4-E4B-it-metal-mtp-vision-audio.litertlm",
    backend=litert_lm.Backend.GPU(),
    vision_backend=litert_lm.Backend.GPU(),
    audio_backend=litert_lm.Backend.GPU(),
    enable_speculative_decoding=True,
)

conversation = engine.create_conversation()
response = conversation.send_message("Describe this audio/image scene.")
print(response.text)

Model bundles

  • jfan/gemma-4-e2b-web-mtp-vision-litert-lm, jfan/gemma-4-e4b-web-mtp-vision-litert-lm (text / +vision / +audio / +vision+audio, with the eevee MTP drafter and centroid head)
  • jfan/gemma-4-12b-web-mtp-vision-litert-lm, jfan/gemma-4-26b-web-mtp-vision-litert-lm, jfan/gemma-4-31b-web-mtp-vision-litert-lm

Release notes

0.18.3 (2026-11-05, this commit)

  • Rebuilt -c opt WebGPU WASM runtime and @litert-lm/core 0.18.3 SDK on latest Piper HEAD (CL 1014981939) with spatial_w_groups tiled external-weight FC and Decision API scoring.
  • WebGPU / JSPI threaded + SIMD, compat, and Asyncify builds with full symbol maps.
  • Multimodal Prefill Speedup (lever4b): Vision .front() chunk-boundary fix and calibrated group_size.h 512 dispatch cost for accelerated prompt prefill and visual token processing.
  • 12B MQA Metal Shader Compilation Hang Fix: Fixed MTLCompilerService shader unroll loop hang by bounding sequence size ($\le 8$ in fully_connected.cc, $\le 4$ in gpu_model_builder.cc) and capping max_spatial_w (0.81s compilation, 24.3 tok/s decode on Apple Silicon M5 Pro).
  • WebGPU Subgroup Matrix: Opportunistic detection and enablement of chromium-experimental-subgroup-matrix in Chrome 137+.
  • Decision API (decide()): Single-token choice and hierarchical classification API in @litert-lm/core.
  • Python 0.18.3 Prebuilt Wheels: macOS ARM64 Metal, Linux x86_64, Linux aarch64, and universal builder/cli wheels.
  • Speculative Decoding in Python API: enable_speculative_decoding flag and gpu_decode_steps_per_sync exposed in litert_lm.Engine.
  • WASM 0.18.3 Builds: Updated -c opt binaries across all four execution variants with symbols.

0.18.2 (2026-10-21)

  • WebGPU Subgroup Matrix acceleration detection.
  • Single-token classification and decision making API in @litert-lm/core.
  • Python 0.18.2 wheels for macOS ARM64 Metal and Linux.

0.18.1 (2026-10-15)

  • Bounded in-flight WebGPU prefill command buffers (RunPendingCommands, $\le 2$ command buffers in flight) on Emscripten.
  • Dual Engine.createLegacyEngine / Engine.createStreaming Embind and TypeScript SDK compatibility.
  • Dynamic per-budget vision executor memory release (280 / 560 / 1120 tokens) and pre-reserved sequence lengths.
Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support