Qwen2-0.5B-Instruct LiteRT-LM Model

This repository contains LiteRT-LM variants of Qwen/Qwen2-0.5B-Instruct optimized for on-device text generation.

Available Artifact

File	Quantization Recipe	Context	Size
Qwen2_0.5B_Instruct.litertlm	dynamic_wi8_afp32	-	647.4 MB

Performance (on device, measured)

Community measurement on a physical Samsung Galaxy S26 (SM-S942Q, Snapdragon 8 Elite Gen 5 / SM8850, Android 16): litert_lm_advanced_main from the litert-lm v0.16.0 release, GPU backend OpenCL (LITERT_CL) against CPU (XNNPACK), one fixed 205-token prompt text (213 tokens under this tokenizer), --benchmark. Two runs per backend taken back-to-back — cells show the range. Peak RSS is the process VmHWM. Before quoting, the file was run on each backend with a real prompt and both produced a correct text answer; the GPU rows are full delegation of the transformer graphs (decode 1063/1063 and prefill 972/972 ops on LITERT_CL; only the tiny embedding-lookup helper graphs stay on the CPU).

Backend	Prefill (213 tok)	Decode	Time-to-first-token	Init	Peak RSS
GPU (OpenCL)	1259–1452 tok/s	61.9–62.9 tok/s	0.16–0.19 s	0.8 s	559–561 MB
CPU (XNNPACK)	405–528 tok/s	49.1–54.5 tok/s	0.42–0.55 s	0.7–1.0 s	1005–1044 MB

What the table says:

• The GPU leads everywhere on this bundle: prefill 2.7–3.6×, decode ~1.2×, and peak RSS 1.8× lower (560 against ~1020 MB) — for a 0.5B int8 model that decode margin is unusual; most small bundles tie the CPU on decode.
• GPU engine init is 0.8 s — unusually cheap for the GPU path (most bundles pay several seconds), so the GPU backend costs almost nothing extra even for short-lived processes.

Integration

Ready to integrate this into your product? Get started in the LiteRT-LM documentation.