Kimi-Linear-48B-A3B-Instruct LiteRT-LM

This repository is a conversion of moonshotai/Kimi-Linear-48B-A3B-Instruct to LiteRT-LM, made with a graph writer for litert-torch (main 67f11e8), for LiteRT-LM built from source (main 02d7e8d8): on an Apple M4 Max (128 GB) the int8 file decodes 61.03 tok/s on the GPU (WebGPU) and 11.98 tok/s on the CPU with 12 threads (2026-11-14 and 2026-11-15).

LiteRT-LM 0.19.2, the latest release on PyPI on 2026-11-16, does not run these files, and neither does 0.19.1: their moe kernels reject SiLU, the activation of this model's experts. SiLU is in two open LiteRT pull requests, #11192 and #11188 (build the runtime).

Files

File	Backend	Quantization	Context	Size
kimi_linear_48b_a3b_int8_ekv4096.litertlm	CPU, GPU (WebGPU)	every linear layer int8 (per channel), the token table fp32	4096	50.50 GB
kimi_linear_48b_a3b_mixed_int8_cpu_ekv4096.litertlm	CPU	experts int8 (per channel), the rest fp32	4096	55.29 GB

All 27 layers of the source: 20 Kimi Delta Attention layers, 7 MLA layers and the mixture of experts (256 experts, 8 per token, one shared expert) (conversion notes).

Performance

File	Backend	Prefill (tok/s)	Decode (tok/s)	Peak footprint
kimi_linear_48b_a3b_int8_ekv4096.litertlm	GPU (WebGPU)	707.93	61.03	67 GB
kimi_linear_48b_a3b_int8_ekv4096.litertlm	CPU (XNNPACK, 12 threads)	134.31	11.98	6.9 GB
kimi_linear_48b_a3b_mixed_int8_cpu_ekv4096.litertlm	CPU (XNNPACK, 12 threads)	134.09	9.72	19 GB

Apple M4 Max (128 GB), 1,024 prefill and 256 decode tokens, measured on 2026-11-13 (the CPU file) and on 2026-11-14 and 2026-11-15 (the int8 file) (how it was measured).

Speed still being tuned (updates follow).

Accuracy

File	Backend	Same verdict as Transformers	Ends with the stop token	3 turns
kimi_linear_48b_a3b_int8_ekv4096.litertlm	GPU (WebGPU)	12 / 12	12 / 12	pass
kimi_linear_48b_a3b_int8_ekv4096.litertlm	CPU (XNNPACK, 12 threads)	12 / 12	12 / 12	pass
kimi_linear_48b_a3b_mixed_int8_cpu_ekv4096.litertlm	CPU (XNNPACK, 12 threads)	12 / 12	12 / 12	pass

12 questions and a 3-turn conversation, greedy, judged against Transformers (bf16 weights computed in fp32) (questions and method).

On the CPU, LiteRT-LM rounds the inputs of int8 linear layers to int8 at run time (XNNPACK qd8). This applies to the int8 file, and the checks above pass with it. The CPU file is computed in fp32: its only int8 weights are the experts', which the CPU moe kernel turns back into fp32 before it uses them.

Run

GPU (WebGPU):

./litert_lm_advanced_main --backend=gpu --model_path=kimi_linear_48b_a3b_int8_ekv4096.litertlm \
  --max_output_tokens=2048 --min_log_severity=2 --sampler_top_k=1 \
  --cache_dir=:nocache --sampler_handles_input=false --input_prompt="What is 17 + 25? Answer briefly."

CPU:

./litert_lm_advanced_main --backend=cpu --model_path=kimi_linear_48b_a3b_int8_ekv4096.litertlm \
  --max_output_tokens=2048 --min_log_severity=2 --sampler_top_k=1 \
  --cache_dir=:nocache --num_cpu_threads=12 --input_prompt="What is 17 + 25? Answer briefly."

The GPU command needs libLiteRtWebGpuAccelerator.dylib from the runtime build in the same directory. Without it, the command stops with Failed to run litert_lm: UNKNOWN: ERROR: [runtime/executor/llm_litert_compiled_model_executor_factory.cc:245] (after 17 seconds here).

Limitations

• One conversation per engine is what was checked: the Kimi Delta Attention layers carry a recurrent state, so create a new engine for each conversation (conversations; LiteRT-LM issue #3165).
• With the CPU file, once a message failed because it did not fit in the context window, every later message in that conversation failed as well (the next one with Task failed with state: 7): start a new conversation after such an error, on a new engine as above.
• Context 4096 tokens (the source supports up to 1M).
• Measured on a desktop only (Apple M4 Max, 128 GB): the files (50.50 GB and 55.29 GB) do not fit in a phone's memory, and the Android GPU, the Android CPU and iOS / iPhone were not measured. On the GPU the int8 file's peak footprint was 67 GB; Macs with less memory were not tried.
• Tool calls were not checked with these files (tool calls).

License

MIT, the license of moonshotai/Kimi-Linear-48B-A3B-Instruct.

Citation

@misc{team2025kimi,
    title         = {Kimi Linear: An Expressive, Efficient Attention Architecture},
    author        = {Zhang, Yu  and Lin, Zongyu  and Yao, Xingcheng  and Hu, Jiaxi  and Meng, Fanqing  and Liu, Chengyin  and Men, Xin  and Yang, Songlin  and Li, Zhiyuan  and Li, Wentao  and Lu, Enzhe  and Liu, Weizhou  and Chen, Yanru  and Xu, Weixin  and Yu, Longhui  and Wang, Yejie  and Fan, Yu  and Zhong, Longguang  and Yuan, Enming  and Zhang, Dehao  and Zhang, Yizhi  and T. Liu, Y.  and Wang, Haiming  and Fang, Shengjun  and He, Weiran  and Liu, Shaowei  and Li, Yiwei  and Su, Jianlin  and Qiu, Jiezhong  and Pang, Bo  and Yan, Junjie  and Jiang, Zhejun  and Huang, Weixiao  and Yin, Bohong  and You, Jiacheng  and Wei, Chu  and Wang, Zhengtao  and Hong, Chao  and Chen, Yutian  and Chen, Guanduo  and Wang, Yucheng  and Zheng, Huabin  and Wang, Feng  and Liu, Yibo  and Dong, Mengnan  and Zhang, Zheng  and Pan, Siyuan  and Wu, Wenhao  and Wu, Yuhao  and Guan, Longyu  and Tao, Jiawen  and Fu, Guohong  and Xu, Xinran  and Wang, Yuzhi  and Lai, Guokun  and Wu, Yuxin  and Zhou, Xinyu  and Yang, Zhilin  and Du, Yulun},
    year          = {2025},
    eprint        = {2510.26692},
    archivePrefix = {arXiv},
    primaryClass  = {cs.CL}
}