Instructions to use jfan/litert-lm-web-runtime with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LiteRT
How to use jfan/litert-lm-web-runtime with LiteRT:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- LiteRT-LM
How to use jfan/litert-lm-web-runtime with LiteRT-LM:
# LiteRT-LM runs on various platforms (Android, iOS, Windows, Linux, macOS, IoT, Web/WASM) # and supports many APIs (C++, Python, Kotlin, Swift, JavaScript, Flutter). # For platform-specific integration guides, please refer to the official developer website: # https://ai.google.dev/edge/litert-lm # To try LiteRT-LM, the easiest way is to use our CLI tool. # 1. Install the LiteRT-LM CLI tool: pip install -U litert-lm # 2. Download and run this model locally: # See: https://ai.google.dev/edge/litert-lm/cli # A single .litertlm file in the repo is picked automatically; otherwise the CLI asks which one to run # (or pass its name right after the repo id). litert-lm run \ --from-huggingface-repo=jfan/litert-lm-web-runtime \ --prompt="Write me a poem"
- Notebooks
- Google Colab
- Kaggle
LiteRT-LM Web Runtime (@litert-lm/core 0.18.3) & Python 0.18.3 Wheels
WebAssembly + WebGPU runtime binaries, the matching @litert-lm/core TypeScript SDK, and prebuilt Python 0.18.3 wheels for running Gemma 4 (E2B, E4B, 12B, 26B-A4B, 31B) in the browser and natively with:
- Multimodal Prefill Acceleration (
lever4b): Vision.front()chunk-boundary synchronization fix and calibratedgroup_size.h512 dispatch cost for accelerated prompt prefill and visual token processing. - 12B MQA Metal Shader Compilation Hang Fix: Bounded shader unroll sequence size ($\le 8$ in
fully_connected.cc, $\le 4$ ingpu_model_builder.cc) and cappedmax_spatial_w, eliminating theMTLCompilerServiceinfinite unroll loop hang on Apple Silicon (compilation completes in 0.81s; 12B MQA decodes at 24.3 tok/s on Apple Silicon M5 Pro). - WebGPU Subgroup Matrix Acceleration: Opportunistic detection and enablement of
chromium-experimental-subgroup-matrixin Chrome 137+ for high-throughput WebGPU GEMM kernels. - Decision API (
decide()): Fast, single-token constrained choice and hierarchical decision trees (e.g. 62 single-token classification labels, tool routing, safety gating) evaluated directly on pre-softmax / post-softmax logits without free-form decode latency. - Text decode with shared-KV neural Multi-Token Prediction (MTP) speculative decoding (
GPU_ARTISANbackend, gamma 1-3, greedy token-for-token equivalence with standard decode). - Multimodal vision (SigLIP encoder on WebGPU, 70 / 140 / 280 / 560 / 1120 token budgets, zero-copy RGBA8
ImageDatapreprocessing, per-budget executor memory release; fused online-softmax ViT self-attention that never materializes the score tensor, with full-length query dispatch). - Multimodal audio (Conformer encoder, WebGPU audio adapter with zero-copy GPU tensor buffer sharing saving 15β25% audio chunk latency, streaming markers, zero-copy audio ingestion).
- Streamed model loading for 2-20 GB
.litertlmbundles underwasm32(MAXIMUM_MEMORY=4GB), with section filtering and inter-section padding discard. - Native macOS Apple Silicon Metal & Linux Wheels: Prebuilt 0.18.3 wheels with full
LlmMetal,LlmGpuArtisanExecutor,MtpOrchestrator, and multimodal vision/audio support.
Repository layout
Exactly one runtime and package set is kept on main. Every release is also addressable by commit SHA (.../resolve/<commit>/wasm/), and older releases remain available through their commits.
| Path | Contents |
|---|---|
wasm/ |
-c opt Emscripten builds of the four runtime variants, under both internal names (litertlm_wasm_internal, litertlm_wasm_compat_internal, litertlm_wasm_asyncify_internal, litertlm_wasm_compat_asyncify_internal) and feature aliases (threaded_simd, threaded_nosimd, simd, nosimd); litertlm_wasm.{js,wasm} / litertlm_wasm_compat.{js,wasm} alias the JSPI threaded builds. litertlm_wasm_internal.js.symbols is the symbol map. |
packages/litert-lm-core-0.18.3.tgz |
The @litert-lm/core npm package (ES modules + typings + the same wasm/ binaries) produced from the same source revision. |
wheels/ |
Prebuilt Python 0.18.3 wheels: litert_lm_api for macosx_12_0_arm64, manylinux_2_27_x86_64, manylinux_2_27_aarch64, plus litert_lm_builder and umbrella litert_lm. |
Variant selection: internal = JSPI + threads + SIMD (Chrome 137+), compat_internal = JSPI + threads without SIMD, asyncify_internal / compat_asyncify_internal = Asyncify fallbacks for browsers without JSPI.
Web / TypeScript SDK Usage
Install @litert-lm/core 0.18.3 from the tarball or npm:
npm install https://huggingface.co/jfan/litert-lm-web-runtime/resolve/main/packages/litert-lm-core-0.18.3.tgz
import {loadLiteRtLm, Backend} from '@litert-lm/core';
const WASM_BASE =
'https://huggingface.co/jfan/litert-lm-web-runtime/resolve/main/wasm/';
const litertlm = await loadLiteRtLm(WASM_BASE);
const engine = await litertlm.createEngine({
modelPath: 'https://huggingface.co/jfan/gemma-4-e4b-web-mtp-vision-litert-lm/resolve/main/gemma-4-E4B-it-web-mtp-vision-audio.litertlm',
backend: Backend.GPU_ARTISAN,
speculativeDraftTokens: 3, // MTP gamma; omit for standard decode
});
// Single-token classification and decision making
const decision = await engine.decide({
prompt: 'Classify this input: "Flight booking request"',
choices: ['travel', 'finance', 'support', 'general'],
});
console.log('Selected class:', decision.choice);
Pin a commit SHA instead of main in production so the binaries and the SDK stay in lock-step.
Python Wheels Usage
Download and install the Python 0.18.3 wheel for your platform:
macOS ARM64 (Apple Silicon / Metal)
pip install https://huggingface.co/jfan/litert-lm-web-runtime/resolve/main/wheels/litert_lm_api-0.18.3-py3-none-macosx_12_0_arm64.whl
pip install https://huggingface.co/jfan/litert-lm-web-runtime/resolve/main/wheels/litert_lm-0.18.3-py3-none-any.whl
Linux x86_64
pip install https://huggingface.co/jfan/litert-lm-web-runtime/resolve/main/wheels/litert_lm_api-0.18.3-py3-none-manylinux_2_27_x86_64.whl
pip install https://huggingface.co/jfan/litert-lm-web-runtime/resolve/main/wheels/litert_lm-0.18.3-py3-none-any.whl
Linux AArch64
pip install https://huggingface.co/jfan/litert-lm-web-runtime/resolve/main/wheels/litert_lm_api-0.18.3-py3-none-manylinux_2_27_aarch64.whl
pip install https://huggingface.co/jfan/litert-lm-web-runtime/resolve/main/wheels/litert_lm-0.18.3-py3-none-any.whl
Python Inference Example
import litert_lm
# Initialize engine with Apple Silicon Metal or Vulkan/OpenCL GPU backend
# and speculative decoding enabled.
engine = litert_lm.Engine(
model_path="/path/to/gemma-4-E4B-it-metal-mtp-vision-audio.litertlm",
backend=litert_lm.Backend.GPU(),
vision_backend=litert_lm.Backend.GPU(),
audio_backend=litert_lm.Backend.GPU(),
enable_speculative_decoding=True,
)
conversation = engine.create_conversation()
response = conversation.send_message("Describe this audio/image scene.")
print(response.text)
Model bundles
jfan/gemma-4-e2b-web-mtp-vision-litert-lm,jfan/gemma-4-e4b-web-mtp-vision-litert-lm(text / +vision / +audio / +vision+audio, with theeeveeMTP drafter and centroid head)jfan/gemma-4-12b-web-mtp-vision-litert-lm,jfan/gemma-4-26b-web-mtp-vision-litert-lm,jfan/gemma-4-31b-web-mtp-vision-litert-lm
Release notes
0.18.3 (2026-11-05, this commit)
- Rebuilt
-c optWebGPU WASM runtime and@litert-lm/core0.18.3 SDK on latest Piper HEAD (CL 1014981939) with spatial_w_groups tiled external-weight FC and Decision API scoring. - WebGPU / JSPI threaded + SIMD, compat, and Asyncify builds with full symbol maps.
- Multimodal Prefill Speedup (
lever4b): Vision.front()chunk-boundary fix and calibratedgroup_size.h512 dispatch cost for accelerated prompt prefill and visual token processing. - 12B MQA Metal Shader Compilation Hang Fix: Fixed
MTLCompilerServiceshader unroll loop hang by bounding sequence size ($\le 8$ infully_connected.cc, $\le 4$ ingpu_model_builder.cc) and cappingmax_spatial_w(0.81s compilation, 24.3 tok/s decode on Apple Silicon M5 Pro). - WebGPU Subgroup Matrix: Opportunistic detection and enablement of
chromium-experimental-subgroup-matrixin Chrome 137+. - Decision API (
decide()): Single-token choice and hierarchical classification API in@litert-lm/core. - Python 0.18.3 Prebuilt Wheels: macOS ARM64 Metal, Linux x86_64, Linux aarch64, and universal builder/cli wheels.
- Speculative Decoding in Python API:
enable_speculative_decodingflag andgpu_decode_steps_per_syncexposed inlitert_lm.Engine. - WASM 0.18.3 Builds: Updated -c opt binaries across all four execution variants with symbols.
0.18.2 (2026-10-21)
- WebGPU Subgroup Matrix acceleration detection.
- Single-token classification and decision making API in
@litert-lm/core. - Python 0.18.2 wheels for macOS ARM64 Metal and Linux.
0.18.1 (2026-10-15)
- Bounded in-flight WebGPU prefill command buffers (
RunPendingCommands, $\le 2$ command buffers in flight) on Emscripten. - Dual
Engine.createLegacyEngine/Engine.createStreamingEmbind and TypeScript SDK compatibility. - Dynamic per-budget vision executor memory release (
280/560/1120tokens) and pre-reserved sequence lengths.
- Downloads last month
- -