Instructions to use litert-community/Jan-nano with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LiteRT-LM
How to use litert-community/Jan-nano with LiteRT-LM:
# LiteRT-LM runs on various platforms (Android, iOS, Windows, Linux, macOS, IoT, Web/WASM) # and supports many APIs (C++, Python, Kotlin, Swift, JavaScript, Flutter). # For platform-specific integration guides, please refer to the official developer website: # https://ai.google.dev/edge/litert-lm # To try LiteRT-LM, the easiest way is to use our CLI tool. # 1. Install the LiteRT-LM CLI tool: pip install -U litert-lm # 2. Download and run this model locally: # See: https://ai.google.dev/edge/litert-lm/cli # A single .litertlm file in the repo is picked automatically; otherwise the CLI asks which one to run # (or pass its name right after the repo id). litert-lm run \ --from-huggingface-repo=litert-community/Jan-nano \ --prompt="Write me a poem"
- LiteRT
How to use litert-community/Jan-nano with LiteRT:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
- Jan-nano β LiteRT-LM (blockwise int4)
LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmodified with litert_torch.convert matched the original to 4e-7 on a Galaxy S26 (measured, LiteRT 2.2.0, Android 16, 2026-09-05).
Measured on device (edge-compat, jan-nano): Mac Studio M4 Max Β· LiteRT-LM 0.14.0 Β· GPU Β· decode 69.0 tok/s Β· prefill 970 tok/s Β· TTFT 278 ms (2026-07-23); Raspberry Pi 5 Β· LiteRT-LM 0.16.1 Β· CPU, 4 threads Β· decode 1.5 tok/s Β· prefill 11 tok/s Β· TTFT 25.51 s (2026-09-01). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/jan-nano/CARD.md
Measured on device (edge-compat, jan-nano-model-block32): Galaxy S26 Β· LiteRT-LM 0.16.0 Β· GPU Β· decode 9.3 tok/s Β· prefill 102 tok/s Β· TTFT 2.08 s Β· all 1674 ops delegated (2026-08-24); Galaxy S26 Β· LiteRT-LM 0.16.0 Β· CPU Β· decode 5.9 tok/s Β· prefill 29 tok/s Β· TTFT 7.14 s (2026-09-05); Raspberry Pi 5 Β· LiteRT-LM 0.16.1 Β· CPU, 4 threads Β· decode 1.4 tok/s Β· prefill 10 tok/s Β· TTFT 28.00 s (2026-09-01). Record: https://github.com/john-rocky/edge-compat/blob/main/cards/jan-nano-model-block32/CARD.md
Jan-nano β LiteRT-LM (blockwise int4)
Try it in your browser: the LiteRT-LM Web chat demo loads model.litertlm from this repo's web branch (2.47 GB download) and runs it with WebGPU. That branch keeps the files from before the 2026-11-16 entry below. Jan-nano: the demo's first load compiled that file for 13 minutes on an M4 Max before it answered, so give the "Compiling Model" step time.
Menlo/Jan-nano converted to the LiteRT-LM
(.litertlm) format for on-device inference with Google's
LiteRT-LM runtime.
Jan-nano is a 4B deep-research agent fine-tuned from Qwen3-4B (Qwen3ForCausalLM)
with a multi-stage RLVR recipe, optimized for tool use via the Model Context Protocol (MCP).
It is a reasoning model β it emits a <think>β¦</think> chain before its answer β so it
rides the existing Qwen3 converter and runtime directly.
On a Galaxy S26 GPU model.litertlm decoded 20.4 tokens/s and prefilled 1,024 tokens at 171 tokens/s (LiteRT-LM v0.19.0; each run repeats the workload three times, and these values are the median of the second and third repetitions, iterations 2β3, over two runs): the Galaxy S26 section.
Run it with the litert-lm CLI:
pip install litert-lm
litert-lm run --from-huggingface-repo litert-community/Jan-nano model.litertlm --prompt "Plan how to find where HTTP retries are configured in a Python repo."
The first run downloads model.litertlm (2.3 GiB, as the CLI prints it). The litert-lm run line was run as written on 2026-11-16 with litert-lm 0.19.2 on a Mac: it printed the model's thought between [thought] and [/thought], then a plan in numbered steps.
When it does not work:
- With litert-lm 0.19.2,
--thinking falsedid not switch the thought off, so give this reasoning model a high token limit: It's a reasoning model. - On Android the GPU path needs LiteRT-LM 0.19.0 or later: Known runtime issues.
- The files published on 2026-11-16 were not run on an iPhone (earlier files were): Run on iPhone.
- The litert-lm CLI (0.14.0 to 0.19.2) reuses a
model.litertlmit downloaded before, without checking it against this repo, so a copy from before the files were replaced on 2026-11-16 (08:00 UTC), or an incomplete download, stays in use: Run on desktop says which file to delete; then run the command again.
On 2026-11-16 both files were replaced with files whose prefill/decode graph is rewritten for the GPU, with every weight unchanged.
Outside the dated entries under Conversion, three names are fixed. "These files" are model.litertlm and model_block32.litertlm as published on 2026-11-16. "The previous files" are the two files they replaced, published on 2026-10-30; one of them is "the previous model.litertlm" or "the previous model_block32.litertlm". "Earlier files" are files this repo published before 2026-10-30. The measured lines at the top of this card were taken with earlier files, on 2026-07-23 (Mac), 2026-08-24 and 2026-09-05 (Galaxy S26) and 2026-09-01 (Raspberry Pi 5).
These files were run only with LiteRT-LM 0.19.0 (phones and Mac) and 0.19.1β0.19.2 (Mac); earlier runtimes were not tried with them.
On the Galaxy S26 GPU model_block32.litertlm decoded 18.2 tokens/s and prefilled 1,024 tokens at 161 tokens/s, measured the same way as model.litertlm above. Run back to back with their previous files on that phone, these files decoded 1.45β1.54 times as fast and prefilled 1.30β1.41 times as fast, depending on the file and the iteration, and the phone-wide GPU allocation counter (kgsl_page_alloc) rose 4.83 GB (model.litertlm) and 5.21 GB (model_block32.litertlm) during their runs, against 3.64β3.65 GB and 4.03 GB with their previous files. On a Pixel 11 Pro GPU each of these files decoded faster than its previous file in every iteration (1.06β1.43 times as fast); model_block32.litertlm also prefilled faster, model.litertlm not in every iteration (see the Pixel 11 Pro section). On the Galaxy S26 CPU (2026-11-16), in 11 of the 12 comparisons (two files, three iterations, prefill and decode) the two files' runs overlap or differ by less than one file's own two runs; the exception is model_block32.litertlm's iteration-2 prefill, which was faster. On that CPU the process peak (VmHWM) was 1.01β1.04 GB higher with each of these files than with its previous file, in each pair of runs (see the Galaxy S26 section).
| Files | model.litertlm β int4 block 128 Β· model_block32.litertlm β int4 block 32 (finer-grain) |
| Quantization | int4 weights (symmetric) + OCTAV optimal-clipping; embeddings INT8 (externalized section) |
| Context (KV cache) | 4096 |
| Base model | Menlo/Jan-nano (Qwen3-4B) |
β οΈ It's a reasoning model β give it room to think
Jan-nano generates a <think>β¦</think> reasoning chain, then the answer. Run it with
max_tokens β₯ 2048 β at a short limit it gets cut off mid-thought and never reaches the
answer. (The GSM8K scores below were measured with earlier files at 2048, unless a line says 1024.)
Thinking cannot be switched off. On litert-lm 0.18.0 ThinkingConfig(enable_thinking=False) had no effect on either of the two files this repo carried on 2026-10-13 (earlier files): with a 400-token cap the thought channel used the whole cap and no answer text appeared. With these files and the previous files, the litert-lm 0.19.2 CLI's --thinking false did not stop the thought either: on a Mac GPU every reply to 8 short questions began with a thought. Set thinking_token_budget (with the files of 2026-10-13, 300 answered 17 Γ 23 under a 700-token cap) and a cap above it.
Which file?
| File | int4 granularity | GSM8K (max_tokens 2048) | iPhone 17 Pro | Mac (M-series, GPU) |
|---|---|---|---|---|
model.litertlm |
block 128 | 88.0% | ~14 tok/s, loads | ~67 tok/s |
model_block32.litertlm |
block 32 | 85.0% | no run recorded | ~67 tok/s |
The table was measured with earlier files; these files were not run on an iPhone or timed on a Mac.
With these files:
- Galaxy S26 GPU (LiteRT-LM v0.19.0):
model.litertlmdecoded 20.4 tok/s and prefilled 171 tok/s,model_block32.litertlm18.2 and 161 tok/s; by each run's mean of iterations 2β3, bothmodel.litertlmruns decoded faster than bothmodel_block32.litertlmruns. The two files were timed in separate sets of runs on the same evening, not back to back.model_block32.litertlmalso took more memory (process peak in the timing runs up to 1.36 GB against 1.04 GB, and up to 1.66 GB against 1.07 GB in the 8-question runs; GPU allocations 5.21 GB against 4.83 GB) and about twice as long to start the engine (12.80 s against 6.45 s, the median of each file's two runs). See the Galaxy S26 section. - Pixel 11 Pro GPU (same program): the median decode over two runs was 8.91 tok/s with
model.litertlmand 9.35 tok/s withmodel_block32.litertlm.model.litertlm's two runs differed: 10.3 and 7.92 tok/s, the second slowing down after its first iteration. See the Pixel 11 Pro section. - 8 short questions: on a Mac GPU (litert-lm 0.19.2)
model.litertlmanswered 8 of 8 andmodel_block32.litertlm7 of 8, the same as the previous files; on the Galaxy S26 GPU and the Pixel 11 Pro GPU each file answered 8 of 8 (see the Accuracy note).
Performance
With earlier files: litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine.
| Device | Backend | Prefill (256) | Decode | TTFT | Load | Peak footprint |
|---|---|---|---|---|---|---|
| Apple M4 Max (macOS) | CPU | 111 tok/s | 18.0 tok/s | 2.49 s | β | β |
| Apple M4 Max (macOS) | GPU (Metal) | 1003 tok/s | 69.0 tok/s | 0.28 s | β | β |
| iPhone 17 Pro | GPU (Metal) | β | ~14 tok/s | β | β | β |
Reproducibility: the GPU rows repeat to within about 1% across invocations; the CPU rows are noisier β re-running a 1B model six times spread its CPU decode over 29.0β33.3 tok/s, so treat the CPU column as accurate to roughly Β±7%.
The desktop rows are an earlier model.litertlm (block 128); the block-32 build was not measured this way, so its figures in βWhich file?β above are older numbers, on this card since the first upload (2026-06-27). The iPhone row is carried over from this repository's own earlier on-device note for an earlier block-128 file; its run log is not retained here, so the run count and prompt are not known.
These files' speed was measured on two phones, the Galaxy S26 (GPU and CPU) and the Pixel 11 Pro (GPU), in the sections below, not on a Mac or an iPhone.
Accuracy note
Measured with earlier files on GSM8K (n=100, greedy, 0-shot chain-of-thought, max_tokens 2048, identical prompt and answer-extraction for every row).
| Configuration | GSM8K |
|---|---|
| bf16 (reference) | 92.0% |
| LiteRT int4 β block 128 | 88.0% (β4 pt) |
| LiteRT int4 β block 32 | 85.0% (β7 pt) |
int4 is at parity (β4 pt for the block-128 build). Note: evaluating a reasoning
model at a short token budget badly understates int4 β at max_tokens 1024 the same block-32
build scored only 63% purely because the longer int4 reasoning chains were truncated before the
answer; at 2048 it recovers to 85%. Always benchmark reasoning models with enough headroom.
8 short questions, each ending with "Answer briefly." (such as "What is 17 + 25? Answer briefly."), with these files and the previous files:
- Mac GPU (litert-lm 0.19.2):
model.litertlm8 of 8 and the previousmodel.litertlm8 of 8;model_block32.litertlm7 of 8 and the previousmodel_block32.litertlm7 of 8. Both block-32 files missed "What is 8 times 7?": the thought kept repeating "Wait, 7 times 8 is 56? No, that's not." and the reply ended without an answer. - Galaxy S26 GPU (LiteRT-LM v0.19.0): 8 of 8 with each of these files; the previous files were not asked there.
- Pixel 11 Pro GPU (LiteRT-LM v0.19.0): 8 of 8 with each of the four files.
Every reply on the Mac and on the phones began with a thought. The phone runs allowed up to 1,536 output tokens; the Mac runs used the CLI's default limit.
Galaxy S26 β GPU and CPU
Each run starts the engine once and then, three times (three iterations), opens a new conversation that prefills 1,024 tokens and generates 256: by the runtime's source, the benchmark mode puts the prompt file's one sentence ("Explain what on-device AI means in simple terms.") in the chat template as a user message and pads the tokens to 1,024 with token id 0. Iteration 1 is the first pass after the engine starts. The GPU rows are the median of iterations 2β3 over a file's two runs, so they leave iteration 1 out (iteration 1 alone, the median over the same runs: model.litertlm 199 tok/s prefill and 20.8 tok/s decode, the previous model.litertlm 146 and 13.6; model_block32.litertlm 196 and 18.7, the previous model_block32.litertlm 146 and 12.7). On the CPU the median prefill speed drops from iteration 1 to iteration 2 with every file, so each iteration has its own row: the median of that iteration over a file's two runs (a warm-up run before them is not counted). Every speed ratio on this card is computed from the speeds the runtime printed (two decimals, in MEASUREMENTS.md) before any further rounding.
| File | Backend | Prefill, 1,024 tokens | Decode | Time to first token | Peak memory |
|---|---|---|---|---|---|
model.litertlm |
GPU (LiteRT GPU, every node of the prefill and decode graphs), iterations 2β3 |
171 tok/s | 20.4 tok/s | 6.08 s | 1.04 GB + 4.83 GB GPU allocations |
the previous model.litertlm |
GPU (LiteRT GPU, every node of the prefill and decode graphs), iterations 2β3 |
123 tok/s | 13.4 tok/s | 8.46 s | 1.04 GB + 3.64β3.65 GB GPU allocations |
model_block32.litertlm |
GPU (LiteRT GPU, every node of the prefill and decode graphs), iterations 2β3 |
161 tok/s | 18.2 tok/s | 6.47 s | 1.36 GB + 5.21 GB GPU allocations |
the previous model_block32.litertlm |
GPU (LiteRT GPU, every node of the prefill and decode graphs), iterations 2β3 |
122 tok/s | 12.6 tok/s | 8.48 s | 1.60 GB + 4.03 GB GPU allocations |
model.litertlm |
CPU (XNNPACK for most nodes), iteration 1 | 55.7 tok/s | 5.82 tok/s | 18.69 s | 5.15 GB |
model.litertlm |
CPU (XNNPACK for most nodes), iteration 2 | 40.1 tok/s | 5.24 tok/s | 25.74 s | 5.15 GB |
model.litertlm |
CPU (XNNPACK for most nodes), iteration 3 | 34.7 tok/s | 5.17 tok/s | 29.72 s | 5.15 GB |
the previous model.litertlm |
CPU (XNNPACK for most nodes), iteration 1 | 64.4 tok/s | 6.28 tok/s | 16.19 s | 4.11 GB |
the previous model.litertlm |
CPU (XNNPACK for most nodes), iteration 2 | 39.6 tok/s | 5.71 tok/s | 26.04 s | 4.11 GB |
the previous model.litertlm |
CPU (XNNPACK for most nodes), iteration 3 | 37.5 tok/s | 5.46 tok/s | 27.57 s | 4.11 GB |
model_block32.litertlm |
CPU (XNNPACK for most nodes), iteration 1 | 41.4 tok/s | 5.94 tok/s | 25.34 s | 5.48 GB |
model_block32.litertlm |
CPU (XNNPACK for most nodes), iteration 2 | 33.9 tok/s | 5.59 tok/s | 30.37 s | 5.48 GB |
model_block32.litertlm |
CPU (XNNPACK for most nodes), iteration 3 | 31.5 tok/s | 5.44 tok/s | 32.84 s | 5.48 GB |
the previous model_block32.litertlm |
CPU (XNNPACK for most nodes), iteration 1 | 43.7 tok/s | 5.77 tok/s | 23.67 s | 4.46 GB |
the previous model_block32.litertlm |
CPU (XNNPACK for most nodes), iteration 2 | 32.6 tok/s | 5.35 tok/s | 31.62 s | 4.46 GB |
the previous model_block32.litertlm |
CPU (XNNPACK for most nodes), iteration 3 | 28.3 tok/s | 5.14 tok/s | 36.40 s | 4.46 GB |
Samsung Galaxy S26 (model SM-S942Q, chipset SM8850, Android 16), litert_lm_advanced_main (LiteRT-LM's C++ command-line program) built from the LiteRT-LM v0.19.0 source (commit 771b06e1; the v0.19.0 tag is on the next commit, which changes only Package.swift and README.md), --benchmark with 1,024 prefill tokens and 256 decode tokens, context 2,048 (--max_num_tokens=2048); the GPU runs on 2026-11-15, the CPU runs on 2026-11-16. Each timing run is a new process, started after the runtime's cache files next to the model were deleted. The full command is in MEASUREMENTS.md.
In the GPU runs every node of the prefill and decode graphs ran on LiteRT GPU with all four files: with these files 1,707 nodes in each of prefill_128 and prefill_1024 and 1,631 in decode, with the previous files 1,674 in prefill_128 and 1,596 in decode. The embedding table is a separate graph in its own section of the .litertlm file, with two signatures (decode_embedder and prefill_embedder_128); in the GPU runs, in each of them, XNNPACK took 1 of the 4 nodes and no delegate took the other 3, with every file. The previous files have one prefill signature, of 128 tokens: by the runtime's source they prefill the 1,024 tokens in eight steps of 128, and these files in one step of 1,024 (read in the source, not measured).
GPU runs: for each file, four, in the order previous, this, this, previous (GPU runs 1β4); the four runs of model_block32.litertlm and its previous file came after those of model.litertlm on the same evening, so the two files were not run back to back. Each run started at Android's thermal status 0 (none), with the GPU clock cap at 1,300 MHz and GPU thermal level 0 (the GPU driver's own thermal level; 0 = no limit), and ended at thermal status 2 (moderate). model.litertlm's runs took 61 and 62 s and its previous file's 87 s each; model_block32.litertlm's took 74 s each and its previous file's 98 s each. In every run the phone lowered the GPU clock cap step by step from 15β27 s in; by the end of a run it read 902 MHz (level 5) in every run of these files, and 826 MHz (level 6) or 902 MHz in the previous files' runs.
CPU runs: for each file, four counted runs in the order previous, this, this, previous (CPU runs 1β4), after one warm-up run of this file that is not counted. The runtime used its default of 4 threads. Every run started without a CPU clock cap and at thermal status 1 (light), except the first counted run of the previous model.litertlm (CPU run 1), which started at thermal status 0. Every run ended at thermal status 3 (severe), except that run and the second counted run of model_block32.litertlm (CPU run 3), which ended at thermal status 2. The phone began to lower the caps of its two CPU clock domains 2β5 s into each run; they reached 1.56 and 1.50 GHz 20β49 s in and 1.23 and 1.27 GHz 96β211 s in, and read 1.23 and 1.27 GHz at the end of every run. Each run of model.litertlm and its previous file took 195β220 s, and each run of model_block32.litertlm and its previous file 217β243 s. The first counted run of the previous model.litertlm started 25 minutes after the warm-up run ended, and its second (CPU run 4) 3 hours 36 minutes after CPU run 3 ended (other tests had the phone in that time); the second counted run of model_block32.litertlm (CPU run 3) started 14 minutes after CPU run 2 ended (another test had the phone for 11 of those minutes). Each of the other counted runs started 154β275 s after the run before it ended. During some runs other apps on the phone used much more CPU time than during others (see MEASUREMENTS.md).
On the GPU, in each iteration of each back-to-back pair (GPU run 2 against run 1, run 3 against run 4), model.litertlm decoded Γ1.51β1.54 and prefilled Γ1.36β1.41 as fast as the previous model.litertlm, and model_block32.litertlm decoded Γ1.45β1.48 and prefilled Γ1.30β1.37 as fast as the previous model_block32.litertlm (each pair and iteration in MEASUREMENTS.md). Each run's mean of iterations 2β3: with model.litertlm and its previous file, prefill 122, 171, 171 and 123 tok/s and decode 13.3, 20.4, 20.4 and 13.4 tok/s, so prefill Γ1.404 and decode Γ1.534 in the first pair, prefill Γ1.384 and decode Γ1.522 in the second; with model_block32.litertlm and its previous file, prefill 123, 161, 163 and 122 tok/s and decode 12.6, 18.2, 18.2 and 12.6 tok/s, so prefill Γ1.304 and decode Γ1.450 in the first pair, prefill Γ1.338 and decode Γ1.453 in the second.
On the CPU, by the median of each iteration over a file's two runs, model.litertlm prefilled Γ0.865, Γ1.013 and Γ0.925 as fast as the previous model.litertlm in iterations 1, 2 and 3 and decoded Γ0.927, Γ0.917 and Γ0.947 as fast, and model_block32.litertlm prefilled Γ0.947, Γ1.041 and Γ1.114 as fast as the previous model_block32.litertlm and decoded Γ1.029, Γ1.044 and Γ1.058 as fast. A file's own two CPU runs differed by up to 30 % (model_block32.litertlm's iteration 1 prefill: 35.98 and 46.78 tok/s). Only in one comparison were both runs of one file faster than both runs of the other by more than either file's own two runs differ: in iteration 2 model_block32.litertlm prefilled at 33.93 and 33.92 tok/s and its previous file at 32.59 and 32.57 tok/s. Both of model.litertlm's runs decoded more slowly than both of its previous file's in iterations 1 and 2 (5.81 and 5.83 against 6.55 and 6.01 tok/s; 5.28 and 5.19 against 5.95 and 5.47 tok/s) and prefilled more slowly in iteration 3 (34.64 and 34.72 against 39.54 and 35.47 tok/s), but the nearest runs of the two files are 3.1, 3.6 and 2.2 % apart there, less than the previous file's own two runs (9.0, 8.8 and 11.5 % apart). In the other comparisons the two files' runs overlap. Of the four runs of model.litertlm and its previous file, CPU run 1, a run of the previous model.litertlm and the only run that started at thermal status 0, decoded fastest in every iteration. Of the four runs of model_block32.litertlm and its previous file, CPU run 3, a run of model_block32.litertlm that ended at thermal status 2, decoded fastest in every iteration. The cause of the differences between runs was not established.
Peak memory is per run; each row gives the highest VmHWM (the process's peak resident memory) of the file's timing runs (on the CPU, of its two counted runs), so the three CPU rows of a file share one value, and the GPU allocation figure is the range over the file's runs. On the GPU, VmHWM was 0.95 and 1.04 GB in model.litertlm's two runs and 1.04 and 0.97 GB in its previous file's, 1.36 and 1.35 GB in model_block32.litertlm's and 1.30 and 1.60 GB in its previous file's. It stayed below the rise of the phone's GPU allocation counter (kgsl_page_alloc of the phone's kgsl GPU driver; phone-wide, not per process) in the same runs, so the GPU rows give both: the counter rose by 4.83 GB in each run of model.litertlm and by 3.64 and 3.65 GB in its previous file's, by 5.21 GB in each run of model_block32.litertlm and by 4.03 GB in each of its previous file's. During these files' GPU runs the phone's available memory (MemAvailable) fell to 2.89 and 2.82 GB (model.litertlm) and to 2.47 and 2.46 GB (model_block32.litertlm), against 3.31 and 4.09 GB and 3.64 and 3.73 GB in the previous files' runs. On the CPU, VmHWM was 5.15 and 5.06 GB in model.litertlm's two runs and 4.11 and 4.05 GB in its previous file's, and 5.43 and 5.48 GB in model_block32.litertlm's and 4.42 and 4.46 GB in its previous file's (5.19 and 5.09 GB in the two warm-up runs). It was higher with each of these files than with its previous file in every run: in each pair of CPU runs (run 2 against run 1, run 3 against run 4) by 1.04 and 1.01 GB with model.litertlm and by 1.01 and 1.02 GB with model_block32.litertlm. Engine start (the runtime's init time, the median of a file's two runs, each started without cache files) took 6.45 s with model.litertlm and 5.82 s with its previous file, and 12.80 s with model_block32.litertlm and 12.33 s with its previous file, on the GPU, and 2.66 s, 1.83 s, 4.65 s and 3.26 s, in the same order, on the CPU.
8 short questions (such as "What is 17 + 25? Answer briefly.") on this phone's GPU: 8 of 8 with each of these files. The previous files were not asked on this phone. These runs did not wait for the phone to cool down (model.litertlm's started at thermal status 0 to 2, model_block32.litertlm's at 1 or 2).
On the Galaxy S26 the GPU row of model.litertlm decoded 3.5, 3.9 and 3.9 times as fast as its CPU rows of iterations 1, 2 and 3 and prefilled 3.1, 4.3 and 4.9 times as fast; the GPU row of model_block32.litertlm decoded 3.1, 3.3 and 3.4 times as fast as its CPU rows and prefilled 3.9, 4.7 and 5.1 times as fast. An earlier CPU result on this phone, for an earlier model_block32.litertlm (2026-09-05; LiteRT-LM v0.16.0, 4 CPU threads, a 202-token prompt and a 3,894-token reply, two runs): prefill 29.0 and 47.0 tok/s, decode 5.94 and 5.48 tok/s. Its runtime version, file and workload are not those of the rows above. The runs behind each number, and the Galaxy S26 section this card carried for earlier files: MEASUREMENTS.md.
GPU wiring, including the Gallery import toggle: GPU guide.
Pixel 11 Pro β GPU
On a Pixel 11 Pro (Android 17; the same program and workload as the Galaxy S26 section, 2026-11-15) every node of the prefill and decode graphs runs on the GPU with all four files. For each file four runs went previous, this, this, previous; each started at thermal status 0 with no CPU or GPU limit and ended at thermal status 1 (light).
model.litertlm: each run's mean of iterations 2β3 was prefill 123, 150, 107 and 121 tok/s and decode 7.48, 10.3, 7.92 and 7.22 tok/s, so prefill Γ1.212 and decode Γ1.375 in the first two runs, prefill Γ0.878 and decode Γ1.097 in the last two. In run 3 prefill fell from 149 tok/s in iteration 1 to 107 tok/s in iterations 2 and 3, and decode from 8.92 to 7.94 and 7.90 tok/s. During that run the phone's home-screen app (com.google.android.apps.nexuslauncher) and surfaceflinger used 24 and 23 s of CPU time, against less than 2 s each in runs 1 and 2, and 5 and 6 s in run 4; the cause was not established. In each iteration of each pair model.litertlm decoded Γ1.06β1.43 as fast as its previous file, and prefilled Γ0.87β1.51 as fast. The median of iterations 2β3 over a file's two runs: model.litertlm 128 tok/s prefill, 8.91 tok/s decode and 8.34 s to the first token; the previous model.litertlm 123 tok/s, 7.48 tok/s and 8.45 s.
model_block32.litertlm: each run's mean of iterations 2β3 was prefill 121, 144, 143 and 117 tok/s and decode 6.82, 9.27, 9.38 and 6.96 tok/s, so prefill Γ1.190 and decode Γ1.359 in the first two runs, prefill Γ1.226 and decode Γ1.348 in the last two; in each iteration of each pair it decoded Γ1.33β1.41 and prefilled Γ1.18β1.25 as fast as its previous file. The median of iterations 2β3: model_block32.litertlm 144 tok/s prefill, 9.35 tok/s decode and 7.23 s to the first token; the previous model_block32.litertlm 120 tok/s, 6.96 tok/s and 8.70 s.
Peak memory (VmHWM) was 0.91 GB with model.litertlm, 0.88 GB with its previous file, 1.12 GB with model_block32.litertlm and 1.10 GB with its previous file; the GPU driver's allocations were not read on this phone. On this phone each run wrote a 2.01 GB weight cache file next to the model, with every file. 8 short questions on the GPU: 8 of 8 with each of the four files. The Pixel 11 Pro CPU was not measured. The runs: MEASUREMENTS.md.
Usage
# build litert-lm from https://github.com/google-ai-edge/litert-lm, then:
litert_lm_main \
--model_path model.litertlm \
--backend gpu \
--input_prompt "Plan how to find where HTTP retries are configured in a Python repo."
The .litertlm bundle carries the tokenizer and prompt template (Qwen3 ChatML β
<|im_start|>role\nβ¦<|im_end|>, stop token <|im_end|>), so no separate tokenizer files are
needed. The model will produce a <think>β¦</think> block followed by its answer.
Run on Android
Update (July 2026): Google AI Edge Gallery v1.0.16+ can import litert-lm models directly from Hugging Face inside the app (tap +) β no computer or
adbneeded. The manual steps below are only required on older builds or for sideloading a local file.
The official Google AI Edge Gallery app runs
.litertlm models on-device:
- Install a recent Gallery (package
com.google.ai.edge.gallery, 1.0.15+ supports.litertlm). - Download
model.litertlmand push it:adb push model.litertlm /sdcard/Download/ - In the app tap +, pick the file, choose the GPU backend, and raise the max-tokens setting.
- Chat β the bundle already carries the tokenizer and Qwen3 chat template.
On Android the GPU path of these files needs a Gallery build with LiteRT-LM 0.19.0 or later (see Known runtime issues).
Run on desktop (LiteRT-LM CLI)
The same .litertlm bundle runs on macOS / Linux / Windows with the official
LiteRT-LM CLI β including as a
local OpenAI-compatible API server:
pip install litert-lm
litert-lm import --from-huggingface-repo litert-community/Jan-nano model.litertlm jan-nano
litert-lm run jan-nano # interactive chat in the terminal
litert-lm serve # local OpenAI-compatible API server
If the run command above stops with no error message from the CLI (a bus error; exit status 138 on a Mac) or with an error log that contains too large for file size, the file that import downloaded may be incomplete. The litert-lm CLI (0.14.0 to 0.19.2, and litert-lm-nightly 0.20.0.dev20261112) can keep an incomplete download in ~/.litert-lm/cache/huggingface/litert-community/Jan-nano/ and imports it without checking its size. Delete that file and run the import command again: it downloads a new copy and replaces the imported one.
If you imported model.litertlm with --from-huggingface-repo before it was replaced on 2026-11-16 (08:00 UTC), delete ~/.litert-lm/cache/huggingface/litert-community/Jan-nano/model.litertlm, then run the import command again. Without deleting it, import imports the file you downloaded before, not the one in this repo now: the litert-lm CLI (0.14.0 to 0.19.2) uses a file it finds there without checking it against this repo. The same holds for model_block32.litertlm, with its file name in the path.
Run on iPhone
With earlier files, verified on iPhone 17 Pro (LiteRT-LM Swift runtime): an earlier model.litertlm (block 128, 1.94 GiB
section) loaded and generated at ~14 tok/s. These files were not run on an iPhone; their largest section is 2,011,136,000 B (1.87 GiB) in both files.
Conversion
Converted with the official litert-torch
converter β Jan-nano is a standard Qwen3ForCausalLM, so it uses the existing Qwen3 path with
no custom graph code. Recipe: blockwise int4 + OCTAV (INT4 weights, block 128 or 32,
symmetric, OCTAV optimal-clipping), embeddings INT8, KV cache 4096. In the two files published on 2026-11-16, the prefill/decode graph was rewritten and given a 1,024-token prefill signature with public scripts (the 2026-11-16 entry below); every weight is unchanged.
from litert_torch.generative.export_hf.export import export
export(
model="Menlo/Jan-nano",
output_dir="out",
quantization_recipe="qwen3_int4_block128_octav.json", # blockwise-128 int4 + OCTAV, int8 embeddings
cache_length=4096,
externalize_embedder=True,
)
Each dated entry below was written on its date: in it, "these files" are the files published that day, "the previous files" are the ones they replaced, and "this card" is the card as it was then.
2026-11-16 β GPU graph and a 1,024-token prefill (weights unchanged)
The prefill/decode graph of model.litertlm and of model_block32.litertlm now has the GPU form litert-torch writes with --apply_gpu_composites: one odml.cache_update per layer writes the KV cache, and odml.runtime_bmm runs the attention products. The previous files had one prefill signature, of 128 tokens; these files also have prefill_1024, copied from it and sharing every weight buffer. In each file every weight, and every section outside that graph, is byte-identical to the previous file. The bundle header's file-format version went from 1.7.0 to 1.9.0. model.litertlm went from sha256 fa3d75bf⦠(2,474,377,216 B) to 14414540⦠(2,477,211,648 B), and model_block32.litertlm from ebe7a637⦠(2,662,924,288 B) to 293f84d9⦠(2,665,742,336 B). The previous files stay at revision 2cab759 and on the web branch.
On a Mac (litert-lm 0.19.2 unless a line says otherwise):
- 4 text prompts on the CPU (1 thread, top-k 1), one of them 1,425 tokens long before the chat template is applied (longer than the 1,024-token prefill): the same replies from each file and its previous file, byte for byte.
- A prompt that asks for a Python function, with litert-lm 0.19.0, 0.19.1 and 0.19.2 (top-k 1, a 512-token limit): on the CPU (1 thread) the same reply from each file and its previous file with each version, byte for byte. On the GPU each file gave the same reply with all three versions;
model_block32.litertlmgave the same reply as its previous file, byte for byte, and the replies ofmodel.litertlmand its previous file part at line 7, inside the thought. No reply of this check got past the thought within the limit. - 8 short questions (each ends with "Answer briefly.") on the GPU:
model.litertlm8 of 8 andmodel_block32.litertlm7 of 8, the same as their previous files. 6 ofmodel.litertlm's replies are the same byte for byte as its previous file's; the two replies to "What is 17 + 25?" differ and both say 42, and the two replies to "Which is larger: 0.9 or 0.11?" differ and both say 0.9. 5 ofmodel_block32.litertlm's replies are the same byte for byte as its previous file's; the replies to those two questions differ in the same way, and both block-32 files miss "What is 8 times 7?" (see the Accuracy note).
Galaxy S26 GPU (LiteRT-LM v0.19.0, for each file four runs in the order previous, this, this, previous; each run's mean of iterations 2β3): model.litertlm prefill Γ1.404 and Γ1.384 and decode Γ1.534 and Γ1.522 per pair, model_block32.litertlm prefill Γ1.304 and Γ1.338 and decode Γ1.450 and Γ1.453. In iteration 1 alone, model.litertlm prefill Γ1.364 and Γ1.365 and decode Γ1.541 and Γ1.519, model_block32.litertlm prefill Γ1.356 and Γ1.341 and decode Γ1.451 and Γ1.482. On a Pixel 11 Pro GPU (same build and workload; each run's mean of iterations 2β3) model.litertlm prefill Γ1.212 and Γ0.878 and decode Γ1.375 and Γ1.097 (the second pair's run of model.litertlm slowed down after iteration 1, see the Pixel 11 Pro section), model_block32.litertlm prefill Γ1.190 and Γ1.226 and decode Γ1.359 and Γ1.348. 8 short questions on the GPU: 8 of 8 with each of these files on both phones, and with each previous file on the Pixel 11 Pro. Procedure: REPRODUCE.md.
Raspberry Pi 5 (CPU, earlier files)
Measured with earlier files on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (minβmax in parentheses), except Peak RSS, which is the highest of the three invocations' peaks in MiB (VmHWM in kB Γ· 1024, rounded down). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.
| File | Prefill (tok/s) | Decode (tok/s) | TTFT | Peak RSS |
|---|---|---|---|---|
model.litertlm |
10.6 (10.4β10.7) | 1.5 (1.5β1.5) | 25.5 s | 4,030 MiB |
model_block32.litertlm |
10.4 (10.4β10.4) | 1.4 (1.4β1.4) | 28.0 s | 4,443 MiB |
Known runtime issues
External weights (since 2026-10-30). In model_block32.litertlm and model.litertlm, the weights sit in TFLiteWeights sections at the end of the file (LiteRT's external weights) instead of inside the model graph. On Android, the GPU path needs LiteRT-LM 0.19.0 or later: the GPU engine of the v0.18.0 and v0.18.1 Android builds crashes (SIGSEGV) on such a file (LiteRT-LM#3986). The earlier files with the weights inside the graph stay at revision c24742b.
After an answer without a thought (no <think> β¦ </think> block), from the second turn of a conversation on, LiteRT-LM puts the previous answer's stop token into the KV cache with a zero embedding, because these files hold the embedding table in its own section and the decoder takes embeddings, so later answers in a conversation can differ from the answer the model gives to the same messages in one prompt (read in the source of LiteRT-LM 0.16.0 to 0.18.1; measured with 0.18.1 on WeVisOne-VL-2B and Qwen2.5-Coder-1.5B-Instruct, not on these files). After an answer with a thought, the runtime rebuilds the cache from the re-rendered history at the next turn, so the second turn is free of it when the conversation opens with a user message; from the third turn on a zero embedding is written either way, at the previous answer's stop token or, after a rebuild, at another token (read in the source, not measured). The bundle's chat template leaves it to the model whether an answer has a thought. One question per conversation avoids both, as does a new conversation that starts with the earlier messages and the new question.
License
Apache-2.0, inherited from the base model Menlo/Jan-nano (itself fine-tuned from Qwen/Qwen3-4B, also Apache-2.0).
- Downloads last month
- 587