Qwen3-32B LiteRT-LM Models

This repository is a conversion of Qwen/Qwen3-32B to LiteRT-LM for its GPU backend, exported with litert-torch 0.10.0 and quantized to int8 with ai-edge-quantizer 0.10.1, for LiteRT-LM 0.19.1: on an Apple M4 Max (128 GB) with WebGPU, the file decodes 11.86 tokens/s and prefills 160.54 tokens/s (2026-11-09). It is the fastest form that passed the accuracy check: int4 weights decoded 17.21 tokens/s but failed it, and declaring fp16 activations changed decode by 1.2 %, within the spread between runs.

Qwen/Qwen3-32B is Qwen's dense 32.8B model: 64 layers, 64 query and 8 key-value heads, 32,768 tokens of context natively. It thinks before it answers unless thinking is switched off (source model card).

Use the GPU: on LiteRT-LM's CPU backend the file runs, but its token choices drift from the original model's (Runtime notes). The released litert-lm 0.19.1 from PyPI runs it on the GPU (asked "What is 17 + 25? Answer briefly.", it answered 42). The numbers below come from LiteRT-LM v0.19.1 built from its tag, whose command-line tool has the benchmark and sampler flags (Build the runtime).

Available Artifacts

File	Quantization	Backend	Context	Size
qwen3_32b_int8_ekv4096.litertlm	int8 per channel for every linear layer, lm_head and token table; fp32 activations declared	GPU	4096	32.9 GB

Conversion Notes

• Exported from the bf16 checkpoint (revision 9216db57) with litert-torch's export_hf and GPU composites: one prefill and one decode signature, whose lengths LiteRT-LM sets at load time (prefill 1,024 and KV 4,096 by default).
• Why int8: an int4 version of the same export (block 128, min-max) failed the accuracy check and is not published (on the GPU, 3 of 12 questions and one of 3 turns ran to the 2,048-token cap; on the CPU, 12 of 24 next tokens matched the bf16 checkpoint's).
• Metadata: LiteRT-LM 0.19.1's own Qwen3 configuration, byte for byte (chat template with enable_thinking, a thought channel, stop token <|im_end|>, sampler top-k 20 / top-p 0.95 / temperature 0.6). Tokenizer: the checkpoint's tokenizer.json, byte for byte.
• The conversion, one command: john-rocky/hf-to-litertlm: qwen3_32b_work.

Performance

Apple M4 Max (128 GB, macOS 27.0), litert_lm_advanced_main built as below, 1,024 prefill tokens and 256 decode tokens with a 2,048-token KV cache, measured on 2026-11-09.

File	Backend	Prefill (tok/s)	Decode (tok/s)	Peak footprint
qwen3_32b_int8_ekv4096.litertlm	GPU (WebGPU)	160.54	11.86	35.97 GB

• Command: --backend=gpu --benchmark --benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256 --num_iterations=4 --max_num_tokens=2048 --disable_cache=true --sampler_handles_input=false.
• The median of 3 processes, each the mean of iterations 2 to 4.
• Peak footprint: the largest physical footprint read by vmmap about every 20 s (decimal GB).
• No phone was tried: the file is 32.9 GB.

Accuracy

12 questions and a 3-turn conversation on the GPU, greedy (--sampler_top_k=1), up to 2,048 tokens per answer with thinking on, judged against Transformers (the bf16 checkpoint on the Mac's GPU through MPS, its token choices checked against a float32 CPU run). Questions: arithmetic, facts, two translations, a rhyme, 0.9 vs 0.11, a word problem and fib(10) in Python; the 3 turns ask the model to recall a number given in turn 1. The source card advises against greedy decoding in thinking mode (endless repetitions); greedy is used here to compare token by token.

File	Backend	Same verdict as Transformers	Ends with the stop token	3 turns
qwen3_32b_int8_ekv4096.litertlm	GPU (WebGPU)	12 / 12	12 / 12	pass

Token by token over 24 tokens of a math question, the top-1 token computed in float32 from the file's int8 weights (the GPU path's arithmetic, in PyTorch with the file's values) matched the bf16 checkpoint's (also computed in float32) at 23 of 24 positions; at the one miss the checkpoint's two best tokens are 0.02 logits apart.

Runtime notes

• The CPU backend. On the CPU, LiteRT-LM 0.19.1 always quantizes each int8 layer's input to int8 (run-time int8 activation quantization). The file's top-1 token matched the bf16 checkpoint's at 18 of 24 positions of the test prompt (3 runs, identical), one miss at a 5.1-logit margin; without it, 23 of 24 (Accuracy). The CPU was not checked on the 12 questions. Use the GPU.
• Long replies. With LiteRT-LM 0.19.1 a blocking send_message still generating after 10 minutes fails with DEADLINE_EXCEEDED. The M4 Max GPU decodes about 12 tokens/s; on a slower machine a long thought can hit that limit. Use send_message_async, or cap max_output_tokens.
• A message that does not fit. On LiteRT-LM 0.18.1 and 0.19.1, after a message fails because it does not fit in the context window, every later message in that conversation fails too (Task failed with state: 7); start a new conversation.

Build the runtime

git clone https://github.com/google-ai-edge/LiteRT-LM.git && cd LiteRT-LM
git checkout v0.19.1   # 7a03724cece417f1724c923b3d527b4319730841
git lfs pull --include "prebuilt/macos_arm64/*"
bazelisk build --config=macos_arm64 -c opt //runtime/engine:litert_lm_advanced_main

Put bazel-bin/runtime/engine/litert_lm_advanced_main in one directory with libwebgpu_dawn.dylib and libGemmaModelConstraintProvider.dylib from prebuilt/macos_arm64/.

Run

./litert_lm_advanced_main --backend=gpu --model_path=qwen3_32b_int8_ekv4096.litertlm \
  --max_output_tokens=2048 --min_log_severity=2 --sampler_top_k=1 --sampler_handles_input=false \
  --disable_cache=true --input_prompt="What is 17 + 25? Answer briefly."

Without --sampler_top_k=1, the CLI samples with the file's TOP_P settings.

License

Apache License 2.0, the license of Qwen/Qwen3-32B.

Citation

@misc{qwen3technicalreport,
      title={Qwen3 Technical Report},
      author={Qwen Team},
      year={2025},
      eprint={2505.09388},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2505.09388},
}