LiteRT is Google's on-device runtime, the new name for TensorFlow Lite (Android: com.google.ai.edge.litert:litert), and litert-torch, the renamed ai-edge-torch, is its PyTorch converter: a PyTorch model converted unmodified with litert_torch.convert matched the original to 4e-7 on a Galaxy S26 (measured, LiteRT 2.2.0, Android 16, 2026-09-05).

Qwen3.5-4B — LiteRT-LM

Qwen/Qwen3.5-4B converted to the LiteRT-LM (.litertlm) format for on-device inference with Google's LiteRT-LM runtime. Requires litert-lm ≥ 0.15 (both backends gated on 0.15.0 and 0.16.0; the 2026-10-13 int8 file checked on litert-lm 0.18.0 and the Android CLI build of 2026-09-18; the 2026-10-16 files checked on litert-lm 0.18.0 and 0.15.0 on a Mac). Same conversion rail as our Qwen3.5-0.8B — the GPU-delegable rank-≤4 gated-delta kernel — plus two 4B-specific fixes described in the conversion notes.

Qwen3.5 is Alibaba's hybrid architecture: GatedDeltaNet (gated delta rule linear attention) blocks interleaved with a few gated full-attention blocks (this 4B model has 24 linear-attention + 8 attention layers). The linear-attention blocks carry constant-size per-layer conv + recurrent state instead of a growing KV cache, so memory stays nearly flat with context length — only the 8 attention layers keep KV (4096-token budget here). The upstream checkpoint is multimodal; this package is the text decoder only (vision tower and MTP heads dropped exactly as upstream's own Qwen3_5ForCausalLM text-only load contract does).

File	Recipe	Size
Qwen3.5-4B_int8.litertlm	int8 dynamic on linears + embedding (convs, the delta rule and the 48 GatedDeltaNet decay/gate projections in_proj_a / in_proj_b stay float; re-quantized 2026-10-13), fp32 activations declared	4.42 GB (4,418,793,712 B)
Qwen3.5-4B_mixed_int4.litertlm	Mixed INT4: int4 blockwise-32 on linears, int8 on embedding + lm_head (convs and the delta rule stay float), fp32 activations declared	2.75 GB (2,754,070,624 B)

2026-09-21: chat template updated to accept the 0.18 content-parts form (string form unchanged); weights, tokenizer and executor metadata byte-identical. 2026-10-09: metadata updated (chat template with a thinking switch, reasoning channel, supports_thinking); TFLite, tokenizer and executor metadata byte-identical. 2026-10-13: the int8 file re-quantized — the 48 GatedDeltaNet decay/gate projections (in_proj_a / in_proj_b) now stay float, and the rest of the recipe is unchanged; the chat template, tokenizer and executor metadata are byte-identical. Qwen3.5-4B_int8.litertlm: previous file sha256 a360b83d…, 4,407,428,464 B → the 2026-10-13 file sha256 ce4c2d64…, 4,419,088,624 B (+11,660,160 B). Its float export, quantized with the previous recipe, reproduces the previous file's TFLite section byte for byte. This fixes two things on the CPU backend, as measured in the Correctness list and the Honest notes below: replies on multi-step arithmetic that did not stop before the token cap, and, with thinking on without a budget, a thought that did not close. I am sorry: since 2026-08-14, anyone who pulled the int8 file could get a reply that ran on to the token cap on multi-step arithmetic on the CPU backend (7 of 18 runs of the six multiplication questions below, 1 of them with no answer) and, with thinking on without a budget, a thought that did not close (0 of 3 runs closed it); on 2026-10-12 this card's own check recorded a run that reached the cap and still reported the file as answering 3 of 3; and the one-Engine-per-conversation note reached this card only on 2026-10-12, though my own report behind it was filed on 2026-08-06. This was my recipe's defect and my check's miss. The Mac GPU did not show it: both files gave the right product and ended by themselves in 6 of 6 runs of the six questions (one per question), and both pass the 8-question sanity gate 8/8. From now on, when one file of a recipe is found defective, its sibling files are re-checked the same day with the six questions in three fresh processes each, a run that reaches the token cap counts as a failure even when its product is right, and this card's lines are checked mechanically before it is published. 2026-10-16: the tokenizer section of both files replaced with the repository's own tokenizer.json (Qwen/Qwen3.5-4B at 851bf6e8, 12,807,982 B, sha256 5f9e4d49…); the TFLite section, the executor metadata and the LLM metadata (chat template included) are byte-identical. The previous section was the tokenizer as transformers writes it for the Qwen2Tokenizer class (19,989,325 B, sha256 06b95093…), whose pre-tokenizer has no \p{M} class, so text with combining marks was split differently from the repository's tokenizer: 49 tokens instead of 36 on a Hindi test sentence, 24 instead of 12 on Thai, 29 instead of 16 on Tamil, 30 instead of 29 on Arabic with harakat. English, numbers, code, chat markup, Chinese, Japanese, Korean and emoji tokenize the same, and so does every prompt of the checks and benchmarks on this card. The previous section also held the seven audio / TTS markers of tokenizer_config.json (<|audio_start|> … <|audio_pad|>, ids 248070–248076) as single tokens; the repository's tokenizer.json does not list them, so these files read them as plain text. Qwen3.5-4B_int8.litertlm: previous file sha256 ce4c2d64…, 4,419,088,624 B → this file sha256 58288fd3…, 4,418,793,712 B; Qwen3.5-4B_mixed_int4.litertlm: previous file sha256 37209c42…, 2,754,365,536 B → this file sha256 7243c33c…, 2,754,070,624 B. The checks run on these files are the 2026-10-16 item in the Correctness list; every other result on this card was measured on earlier files, as each item says, and where an item dated before 2026-10-16 says "this file" or "the previous file", it means the file of its date or the one before that; the int8 file of 2026-10-13 and the Mixed INT4 files since 2026-08-27 have the TFLite section of the current files byte for byte. I am sorry: every version of these files since 2026-08-14 (int8) and 2026-08-27 (Mixed INT4) carried that tokenizer, so text in Hindi, Thai, Tamil, Arabic with harakat or another script with combining marks reached the model as other token ids than the repository's tokenizer gives (what that did to replies was not measured), and my checks, whose prompts tokenize the same with both, did not show it.

The Mixed INT4 variant exists for RAM-constrained devices (requested in LiteRT-LM #1658 for 8 GB Android phones): 2.75 GB of weights instead of 4.42 GB, measured ~1.7 GB CPU peak on the 138-token probe. If you have the memory for int8, int8 answers harder prompts slightly better (measured below).

Correctness

The int8 results below, down to the GSM8K bullet, come from the previous quantization of the int8 file or its float parent (the same graph), except the One Engine item, run on the Mixed INT4 file, the 2026-10-13 item, run on the 2026-10-13 int8 file (sha256 ce4c2d64…, whose TFLite section and metadata the current file carries byte for byte), and the 2026-10-16 item, run on the current files:

• Logits-level parity vs PyTorch: the float export matches the HF model teacher-forced across 48 positions — top-1 and top-5 agreement 100%, Pearson 1.0000, KL ≈ 0.
• 8-question sanity gate: 8/8 on CPU and GPU, on both litert-lm 0.15.0 and 0.16.0 (Mac).
• iPhone 17 Pro (Metal GPU): the composite 8-question probe (all 8 questions in one 138-token prompt) answers 8/8, word-for-word identical to the HF fp32 reference run greedy with the same template.
• Prompt-length robustness: first-token check against the runtime's real prefill chunk plans, fresh engine per length — 40/40 lengths clean on CPU, 20/20 on GPU.
• Multi-turn conversations carry state correctly across turns.
• One Engine per conversation. On the Mixed INT4 file, a new Conversation opened on an Engine that already ran another conversation replied in 9 of 9 turns with nothing but the opening word its own system prompt asked for, against that word plus the answer in 9 of 9 on a fresh Engine (3 runs, litert-lm 0.18.0, Mac CPU). Create a new Engine for each conversation — the runtime reuses the prompt tokens a new conversation shares with the previous one and does not reset the recurrent state (my open report); a process that runs one conversation is unaffected.
• Multi-step arithmetic, CPU and GPU (2026-10-13, one fresh process per run).
  • Mac CPU (litert-lm 0.18.0, thinking off, 400-token cap): six multiplication questions (17 × 23, 23 × 17, 12 × 34, 9 × 87, 45 × 12, 36 × 25), three runs each — the right product in 18 of 18, with no repeated line in any run. 17 runs end by themselves; the 18th (36 × 25) reaches 900 and is cut by the cap inside a third method, and one run of it with a 1000-token cap ends by itself.
  • The arithmetic prompt (17 × 23) with ThinkingConfig unset: 391 in 3 of 3 runs, each ending by itself.
  • Thinking on without a budget (a raw <think> prefill, 400-token cap): 3 of 3 runs close the thought and answer 391.
  • With a 300-token thinking budget (400-token cap): 3 of 3 runs return the reasoning on the thought channel and answer 391 in a 70-character reply.
  • The 8-question sanity gate (the litert-lm 0.18.0 CLI): 8/8 on Mac CPU and on Mac GPU.
  • Mac GPU (litert-lm 0.18.0, thinking off, 400-token cap): the six questions, one run each — the right product in 6 of 6, with no repeated line. Five end by themselves; the 36 × 25 run (889 characters) reaches 900 and is then cut by the cap inside a digression — the same reply, character for character, as the previous file's GPU run.
  • Galaxy S26 CPU (the LiteRT-LM Android CLI build of 2026-09-18, litert_lm_advanced_main; thinking off by default, 400-token cap): the arithmetic prompt answers 391 in 4 of 4 runs, the same 483-character reply each time, with no repetition.
  • The Mixed INT4 file was not run on the six questions; on 2026-10-12 it answered the arithmetic prompt with 391 in 3 of 3 runs, with no repetition (litert-lm 0.18.0, Mac CPU).
• Tokenizer section (2026-10-16, both current files, Mac M4 Max). The runtime's Engine.tokenize gives the same ids as the tokenizers library with the repository's tokenizer.json on 174 of 174 test strings (21 sentences, Hindi, Thai, Tamil and Arabic with harakat included, and every prompt of the checks on this card) on litert-lm 0.18.0 (GPU) and 0.15.0 (CPU). The 8-question sanity gate answers 8/8 on CPU and on GPU (litert-lm 0.18.0, one fresh process per question), a three-turn conversation completes on the CPU, every GPU subgraph is delegated with no validation error, and litert-lm 0.15.0 answers a plain prompt on the CPU.

Mixed INT4 correctness (all gates re-run on the Mixed INT4 file before 2026-10-16, with the TFLite section of the current file byte for byte):

• 8-question sanity gate: 8/8 on CPU and GPU, on both litert-lm 0.15.0 and 0.16.0 (Mac). Prompt-length robustness: 40/40 CPU, 20/20 GPU.
• iPhone 17 Pro: the composite 8-question probe answers 8/8 on both Metal GPU and CPU (the int8 file answers 6 of 8 on CPU on this probe).
• GSM8K, n=100 (0-shot CoT, greedy, 2048-token budget, identical harness for both files; the int8 figure is the earlier card's, measured on the previous quantization of the int8 file): int8 97/100, Mixed INT4 93/100. That 4-point gap is the real cost of int4 on this model — invisible to the sanity gates, visible on math word problems. We also measured blockwise-128 (90/100) and OCTAV variants (90–92/100); blockwise-32 min-max won and is what ships.

Usage

litert-lm run ./Qwen3.5-4B_int8.litertlm --prompt "What is the capital of France? Answer in one word."

# GPU
litert-lm run ./Qwen3.5-4B_int8.litertlm --backend gpu --cache no --prompt "..."

Prefill signatures: 1024, 256, 64, 16, 4, 1 (not the 0.8B's full 1–1024 ladder). Every exported signature is charged engine memory even if never called, and with this model's 248k vocabulary the full ladder's Metal programs push a 12 GB iPhone past its jetsam ceiling during engine creation. The reduced ladder keeps GPU peak at ~4.9 GB on iPhone 17 Pro with no correctness cost (the runtime just plans slightly coarser chunks).

Chat template note: both files carry the LiteRT-LM models/qwen3_5/chat_template.jinja at 87ab9334 with two edits: its format_content macro gains the tool_response branch of the current main version (it reads chat content as a string or as parts, text and tool_response), and enable_thinking defaults to false instead of true. With nothing set, the generation prompt opens each new assistant turn with an empty <think>\n\n</think> block (past assistant turns render without it), and a single-turn prompt (one user message, with or without a system prompt) renders byte for byte as before. enable_thinking=true pre-fills <think>\n; the reasoning is returned on the thought channel (<think>\n / \n</think>), and the text carries the answer only. Tested with litert-lm 0.18.0 on a Mac (CPU): with thinking on, both files return the reasoning on the channel and give 391 for 17 × 23. A two-turn conversation ("My name is Kenji." / "What is my name?") gets the name back with thinking unset and on; with thinking on, the int8 file's second reply greets the user by name ("Hello Kenji! It's great to meet you. …") instead of stating it. On 0.17.0 both files answer a plain prompt and a two-turn conversation. With thinking on, GPU and phones were not run; the updated mixed_int4 file answered a plain prompt on a Galaxy S26 (CPU, litert_lm_main built from the v0.18.0 tag); for the Mixed INT4 file, the card's GPU and iPhone results come from an earlier file, whose TFLite section and default prompt are byte-identical; for the int8 file, the Mac, iPhone and Raspberry Pi Performance rows come from the previous quantization (same default prompt), and the 2026-10-13 file's own checks are the 2026-10-13 item in the Correctness list. The template renders tool definitions, but the files' llm_model_type is generic_model, so the runtime does not parse the model's tool calls: they come back as text, and supports_function_calling is false.

Performance

The Mac, iPhone and Raspberry Pi int8 rows below were measured on the previous quantization of the int8 file (TFLite section sha256 4666cd03…, as in the files sha256 74abbd14… and a360b83d…), the same graph with the 48 decay/gate projections int8; the Galaxy S26 row further down is of the 2026-10-13 int8 file (sha256 ce4c2d64…). Every row in this section and in the Raspberry Pi section was measured before the 2026-10-16 tokenizer replacement; the Galaxy S26 row and the Mixed INT4 rows on files whose TFLite section the current files carry byte for byte.

litert-lm benchmark (litert-lm 0.16.0), Apple M4 Max, -p 256 -d 256 --runs 3 --cache no, quiet machine:

Backend	Prefill (256)	Decode	TTFT
GPU	672 tok/s	62.0 tok/s	0.40 s
CPU	243 tok/s	19.9 tok/s	1.11 s

On device (cold start, unplugged, thermal-nominal start, single runs, 138-token prompt, quality-gate harness):

Device	Backend	Prefill	Decode	TTFT	Peak memory
iPhone 17 Pro	GPU (Metal)	87.1 tok/s	9.3 tok/s	1.93 s	4.98 GB
iPhone 17 Pro	CPU	24.0 tok/s	5.4 tok/s	6.88 s	1.70 GB

(iPhone GPU: model load 66 s on first launch; whether a later launch is faster was not checked on this file — on the Mixed INT4 file it is not, see the honest notes.)

Galaxy S26 (Qualcomm SM8850), the 2026-10-13 int8 file (sha256 ce4c2d64…, the TFLite section of the current file), the LiteRT-LM Android CLI build of 2026-09-18 (litert_lm_advanced_main), CPU benchmark of 256 prefill and 64 decode tokens, 3 iterations, median of iterations 2–3, caches removed before the warm-up run (engine init = that cold run); peak memory is the peak system RAM use reported by the CLI:

Device	Backend	Prefill (256)	Decode (64)	TTFT	Engine init (cold)	Peak memory
Galaxy S26	CPU	178.9 tok/s	8.93 tok/s	1.55 s	17.9 s	5,517 MB

The phone capped its CPU clocks in every benchmark iteration (our frequency readings: 14 of 18 capped), so the row is this phone's capped speed. The file before it (sha256 a360b83d…), measured right after on the same phone with the same benchmark: 157.2 / 8.64 tok/s, TTFT 1.78 s, cold engine init 20.4 s, peak 5,501 MB, with 16 of 19 readings capped; its first iteration, 199.9 / 11.09 tok/s, is within 1 % of the 2026-10-13 file's 201.5 / 11.11. Four fresh processes of the 2026-10-13 file answered the arithmetic prompt (17 × 23) with 391, the same 483-character reply each time, with no repetition. Neither the 2026-10-13 nor the current int8 file was started on the phone's GPU; a Galaxy S26 hard-rebooted during the previous file's 2026-08-24 GPU attempt (see the Honest notes).

On a Pixel 8a (Tensor G3, 7.75 GB) CPU with the same Android CLI build, one process of the 2026-10-13 int8 file answered the arithmetic prompt (17 × 23) with 391 in the same 483-character reply, with a process peak of 5,407,976 KiB and the phone's free memory at or above 3,542,768 KiB (5,120,440 KiB before the run); no speed was measured.

Mixed INT4 (litert-lm benchmark 0.16.0, Apple M4 Max, same flags; iPhone rows are single cold runs of the 138-token quality probe, unplugged):

Device	Backend	Prefill	Decode	TTFT	Peak memory
Apple M4 Max	GPU	669 tok/s	68.5 tok/s	0.40 s	—
Apple M4 Max	CPU	100 tok/s	20.1 tok/s	2.61 s	—
iPhone 17 Pro	GPU (Metal)	88.7 tok/s	11.4 tok/s	1.89 s	5.74 GB
iPhone 17 Pro	CPU	46.7 tok/s	8.9 tok/s	3.12 s	1.68 GB
Pixel 8a (8 GB)	CPU (XNNPACK)	22.1 tok/s	6.3 tok/s	11.9 s	4.43 GB

(iPhone GPU: model load 65 s on first launch; on LiteRT-LM v0.18.0 every engine creation took 58–61 s, see the honest notes. Pixel 8a: litert_lm_main v0.16.0, 260-token prompt / 204 decoded tokens, warm weight cache; peak is process VmHWM; engine init 30–50 s; TTFT scales with prompt length — 4.2 s on a 24-token prompt.)

Honest notes:

• 2026-10-13 re-quantization (CPU). The previous int8 file (sha256 a360b83d…) could repeat one line to the token cap after its answer on multi-step arithmetic on the CPU backend (litert-lm 0.18.0, Mac M4 Max CPU, one fresh process per run, 400-token cap). With thinking off, the six multiplication questions (three runs each) got the right product in 17 of 18 runs, reached the cap in 7 of 18 (6 of them after the right product, 1 with no product) and ended by themselves in 11 of 18. With ThinkingConfig unset, the arithmetic prompt (17 × 23) answered 391 in 6 of 6 runs (two sets of three) and reached the cap after the answer in 2 of them. Thinking on without a budget (a raw <think> prefill): 0 of 3 runs closed the thought, each repeating one line. With a 300-token thinking budget: 3 of 3 runs returned the reasoning on the thought channel and began the answer with 391, and none of the 3 ended by itself (2 repeated a numbered **Final Answer:** 391. line to the cap, 1 was cut by the cap mid-sentence). On the Mac GPU the six questions got the right product and ended by themselves in 6 of 6 runs (one per question), and the 8-question sanity gate passed 8/8 on CPU and on GPU. On the Galaxy S26 CPU (the LiteRT-LM Android CLI build of 2026-09-18), 3 of 3 fresh processes answered the arithmetic prompt with 391, each ending by itself. The same cause and the same fix as the 2026-10-11 note on the Qwen3.5-2B card: LiteRT-LM's CPU backend quantizes every int8 FC's input row to int8 at run time (XNNPACK dynamic quantization), and the GatedDeltaNet decay/gate heads (in_proj_a, in_proj_b) could not take that noise. The 2026-10-13 file changes only those 48 projections, in the 24 linear-attention layers, to float (+11,660,160 B); its float export, quantized with the previous recipe, reproduces the previous file's TFLite section byte for byte. Measured on the 2026-10-13 file: Mac CPU 18 of 18 right with no repeated line, thinking on without a budget 3 of 3 closing the thought, with a 300-token thinking budget 3 of 3 answering 391, the 8-question sanity gate 8/8 on CPU and GPU, Mac GPU 6 of 6 and Galaxy S26 CPU 4 of 4.
• Where GPU execution is verified. macOS (Metal), iPhone 17 Pro (Metal) and, on Android (Mixed INT4 file), GPU verified on the Galaxy S26 (Adreno) only; no Mali phone — the Pixel 8a (Arm Mali, OpenCL) is not verified on either file, see the Pixel 8a bullets below. On the Galaxy S26 (prebuilt litert_lm_advanced_main 2026-09-18, 2026-10-12) the Mixed INT4 GPU engine builds with a cold init of 73–86 s and a benchmark peak of 3.8–4.1 GB, and one prompt run answered 17 × 23 with 391. The GPU's maximum clock fell from 1300 MHz to 646–726 MHz during the compile, so the speeds measured right after it are not published. The int8 file is not verified on an Android GPU: a Galaxy S26 hard-rebooted during its 2026-08-24 GPU attempt (the cause is not established; recorded in litertlm_manifest.json), and I have not run the int8 file on an Android GPU since, the 2026-10-13 file included; use the CPU. Until 2026-10-12 this bullet said GPU execution on Adreno was not verified; for the Mixed INT4 file, this S26 run changed that.
• GPU engine creation does not get faster after the first. On an iPhone 17 Pro (Metal), 5 of the Mixed INT4 file's 7 compiled GPU programs fail to load from the cache (DATA_LOSS), so every engine creation compiles them again. Four creations in one process took 58–61 s each (LiteRT-LM v0.18.0, 2026-10-23; LiteRT-LM #4178). The int8 file was not checked on v0.18.0. On a Galaxy S26 (OpenCL) with the LiteRT-LM Android CLI build of 2026-09-18, every process that started with a cache (3 of 5) loaded 2 of the 7 programs from it and compiled the other 5 again (DATA_LOSS). That was the Mixed INT4 file of 2026-10-12 (sha256 37209c42…), whose TFLite section the current file carries. Neither file was checked with v0.18.0 there. With one Engine per conversation (see Correctness), each new conversation waits this long.
• GPU inference runs with fp32 activations (declared in the bundle; the fp16-activation formulation is unfinished for this family). That is where the GPU memory multiple over CPU comes from.
• Pixel 8a (8 GB-class Android, CPU): one process of the 2026-10-13 int8 file loaded and answered. On a Pixel 8a (Tensor G3, 7.75 GB) CPU with the LiteRT-LM Android CLI build of 2026-09-18, one process of the 2026-10-13 int8 file (sha256 ce4c2d64…, the TFLite section of the current file) answered the arithmetic prompt (17 × 23) with 391, in the same 483-character reply as on the Galaxy S26. The process took 146.8 s, including the first-run weight-cache build. It peaked at 5,407,976 KiB, and the phone's free memory stayed at or above 3,542,768 KiB (5,120,440 KiB before the run). Our test stops a run when the phone's free memory falls under 400,000 KiB; it did not stop this one. This is one process, and no speed was measured. This run corrects this bullet's earlier "not attempted, and not expected to fit", which was an inference.
• The Mixed INT4 file runs on a Pixel 8a (8 GB) on CPU — measured, table above. Peak process memory 4.43 GB on a 7.75 GB-RAM phone, coherent long-form generation and correct QA output. Use the CPU backend on 8 GB phones: the GPU path expands to ~5.7 GB on Metal and Android GPU budgets are similar or tighter. First run writes a 2.66 GB XNNPACK weight cache next to the model (one-time) — leave ~3 GB of storage free. Until 2026-10-12 this card listed the Pixel 8a as a verified GPU device — my error, and I am sorry to anyone who chose the GPU on a Mali phone because of it: on the Pixel 8a the engine build drains the phone's memory. On 2026-10-12 GPU engine creation for the Mixed INT4 file of that day (sha256 37209c42…, the TFLite section of the current file) on the Pixel 8a (Arm Mali, OpenCL) reached 1 of the 6 prefill signatures, fully delegated it, and took the phone's free memory from 4,244,000 KiB towards our test's stop line of 400,000 KiB. Our test reads free memory every 2 s and stops the run at the first reading under that line. In this run that reading was 139,628 KiB, after Android had begun closing other apps, so do not try GPU on 8 GB devices. Whether the build completes without the stop is not measured. Before it is published, this card is now checked mechanically for the device limits measured on its files.
• Mixed INT4 CPU prefill is slower than int8's (Mac: 100 vs 243 tok/s prefill, TTFT 2.6 vs 1.1 s at 256 tokens; iPhone CPU is the exception — 46.7 vs 24.0, faster). Decode is equal or faster everywhere.
• On low-end Android GPUs, decode is memory-bandwidth-bound and does not beat the CPU anyway; the GPU win is on Apple hardware (and, generally, prefill/TTFT).
• On quality: individual questions gate 8/8 on every backend. On the harder composite probe (8 questions in one prompt), GPU matches HF fp32 word-for-word; the CPU int8 path answers 6 of the 8 (every answer it gives is correct — it skips two). If you need maximum fidelity on complex prompts on CPU, ask for a float variant. Measured on the previous quantization; the composite probe was not re-run on this file.

Conversion notes

Converted with litert-torch plus a hybrid-cache patch (reproduction script + patch: hf-to-litertlm qwen35_work/):

• Rank-4 chunk kernel (shared with the 0.8B): the reference chunked delta rule spells its contractions as broadcast-multiply-reduce over high-rank intermediates. The vendored kernel re-expresses them as batched matmuls with chunk and head axes folded into the batch axis (all tensors rank ≤ 4, no BROADCAST_TO, no int64 index math), and writes every tail-pad as a concat with a zeros constant rather than a PAD op (the GPU delegate miscomputes rank-3 non-final-axis PAD — LiteRT#9272).
• Rank-4 head interleave (4B-specific): the 4B is the first Qwen3.5 with grouped value heads (32 value vs 16 key heads). Upstream expands q/k with repeat_interleave, whose rank-5 lowering the GPU delegate rejects outright — the 0.8B (ratio 1) never traces that branch. The patch re-expresses it as concat + rank-4 reshapes, bitwise-identical to the stock op.
• Reduced prefill ladder (4B-specific): see the signature note above.
• Export cache for GatedDeltaNet layers: conv [B, conv_dim, K] + recurrent [B, heads, k_dim, v_dim] cache layers registered for layer_types == "linear_attention", so torch.export traces the model's own state contract.
• int8 recipe: post-hoc ai-edge-quantizer dynamic int8 on the FULLY_CONNECTED layers and the embedding, with prefer_activation_type fp32. The convs and the delta rule stay float, and since 2026-10-13 so do the 48 GatedDeltaNet decay/gate projections (in_proj_a / in_proj_b of the 24 linear-attention layers).
• Mixed INT4 recipe: post-hoc ai-edge-quantizer on the same float export — int4 BLOCKWISE-32 min-max on every FULLY_CONNECTED, overridden to int8 channelwise on the lm_head and the embedding lookup (one shared 248320×2560 vocab buffer, kept single). Block size and algorithm were chosen by measurement: b32/b128 × min-max/OCTAV all built from the same export and scored on GSM8K n=100 (93 / 90 / 92 / 90) — blockwise-32 min-max won. Convs and the delta rule stay float, prefer_activation_type fp32 as in the int8 file.

Raspberry Pi 5 (CPU)

Measured on a Raspberry Pi 5 Model B Rev 1.1 (8 GB, Raspberry Pi OS 64-bit) with litert-lm benchmark 0.16.1: CPU backend, 4 threads, 256 prefill + 256 decode tokens, --cache memory (the compile cache lives and dies with the process, so every invocation compiles the model from scratch; nothing is reused between runs), one warm-up plus one timed iteration per invocation, 3 invocations per file with cooldown in between. Values are the median across invocations (min–max in parentheses). No thermal throttling occurred during these runs (vcgencmd get_throttled stayed 0x0). Every file listed produced coherent text in a real generation on this backend before its numbers were recorded.

File	Prefill (tok/s)	Decode (tok/s)	TTFT	Peak RSS
Qwen3.5-4B_int8.litertlm (earlier file, sha256 a360b83d…)	20.2 (20.1–20.3)	1.8 (1.8–1.8)	13.2 s	6.2 GB
Qwen3.5-4B_mixed_int4.litertlm	17.1 (17.0–17.2)	2.4 (2.3–2.4)	15.4 s	4.9 GB

Known runtime issues

From a conversation preface (the system message, initial messages, tools and extra context a conversation is created with) that has tools and no messages, LiteRT-LM renders these files' chat template with an empty message list, and the template reads messages[0] without checking that the list is non-empty, so the render fails with Failed to apply template: undefined value. This happens in two places: when the preface is prefilled at creation (prefillPrefaceOnInit = true in Kotlin, prefillPrefaceOnInit: true on the web, SetPrefillPrefaceOnInit(true) in C++; the default is false), where creating the conversation fails with that error, and in the preface render (renderPrefaceIntoString() in Kotlin and Swift, RenderPrefaceIntoString() in C++, litert_lm_conversation_render_preface_to_string in the C API). The same happens with extra context (such as enable_thinking) and no messages, and in the preface render with an empty preface (run in the runtime on one Qwen3 template; the other templates rendered with minijinja 2.14 and jinja2). Read in the source of LiteRT-LM 0.17.0, 0.18.1 and main; measured with LiteRT-LM 0.18.1 (the litert-lm package from PyPI) through its C API on these files' template text. The default path, where the first message is rendered together with the preface, is not affected: the Python API and the CLI of 0.18.1 expose neither the flag nor the preface render, and Google AI Edge Gallery sets neither (its source as read on 2026-10-25). A system message or any initial message in the conversation config avoids it.