Nanbeige4.2-3B — LiteRT-LM (.litertlm)

On-device build of Nanbeige/Nanbeige4.2-3B for Google's LiteRT-LM runtime — an Apache-2.0 3B-class looped-transformer reasoning model (its 22 layers run twice per token, num_loops=2).

File	model.litertlm (~2.4 GB)
Quantization	int4 weights — blockwise (block 32) + OCTAV optimal-clipping, symmetric; embedding INT8
Context (KV cache)	4096 (44 KV slots: 22 layers × 2 loops)
Base model	Nanbeige/Nanbeige4.2-3B (Apache-2.0)

Note on speed: the loop makes this model ~2× the per-token compute of a normal 3B (that's where its quality comes from) — expect roughly 8B-class decode rates on CPU.

What's in the box

• model.litertlm — int4 (blockwise-32, OCTAV clipping) + int8 embedding, 4096-token KV cache, ~2.4 GB.
• The loop is unrolled at export: 44 layer executions over 22 shared-weight layers, one KV-cache slot per (loop, layer) — 44 slots. Weights are stored once; only compute and KV double.
• The chat template auto-opens the model's <think> block (this is how the official template works — the model starts already inside its reasoning).

Performance

litert-lm benchmark (litert-lm 0.15.0) on an Apple M4 Max, -p 256 -d 256 --runs 3 (the tool averages three iterations), max-num-tokens 4096, warm-up run discarded, otherwise idle machine.

Device	Backend	Prefill (256)	Decode	TTFT	Load	Peak footprint
Apple M4 Max (macOS)	CPU	56 tok/s	12.5 tok/s	4.74 s	—	—
Apple M4 Max (macOS)	GPU (Metal)	556 tok/s	52.6 tok/s	0.48 s	—	—
iPhone 17 Pro	CPU	—	3.2 tok/s	—	~4.4 s	1.8–2.1 GB

Reproducibility: the GPU rows repeat to within about 1% across invocations; the CPU rows are noisier — re-running the 1B control six times spread its CPU decode over 29.0–33.3 tok/s, so treat the CPU column as accurate to roughly ±7%.

The iPhone row is carried over from this repository's own earlier on-device note; its run log is not retained here, so the run count and prompt are not known. This build is CPU-only on device.

Accuracy note (GSM8K, n=50, 0-shot CoT, official sampling)

build	GSM8K
bf16 (transformers 4.51, reference)	94.0% (47/50)
this int4 .litertlm (LiteRT-LM CPU)	90.0% (45/50)

Local gate: 8/8 basic-quality questions, clean <think>…</think> closure, ChatML stop.

⚠ Usage notes

• Sampling is required. The model collapses under greedy decoding (official recipe: temperature 0.6, top_k 20, top_p 0.95 — see its generation_config.json). Configure the runtime session accordingly; do not run at temperature 0.
• CPU backend. The current GPU delegate produces incorrect output on this 44-layer unrolled graph; run on CPU.
• Reasoning outputs open inside <think>; the visible answer follows </think>.

Run it

Android: push the file and import it in Google AI Edge Gallery (+ → import, keep CPU). CLI / other platforms: see the LiteRT-LM repo for litert-lm runtime usage.

Reproduce

Converted with hf-to-litertlm:

bash scripts/reproduce_llm.sh nanbeige4.2-3b

The conversion script (scripts/convert_nanbeige42.py) documents the two traps this model hits: transformers-5.x zeroes the modeling's init-computed rotary inv_freq buffer (rope must be recomputed from config), and the looped architecture needs a 44-slot KV cache registered through the export pipeline's cache_implementation hook.