# Kimi-Linear-48B-A3B-Instruct LiteRT-LM: details

The model card is [README.md](README.md). This page holds how the files were made and checked.

## Runtime

LiteRT-LM main `02d7e8d8` (2026-11-07) built from source with LiteRT `bf49eb62` and two open LiteRT pull requests that add SiLU to the `moe` kernels: [#11188](https://github.com/google-ai-edge/LiteRT/pull/11188) (CPU, XNNPACK) and [#11192](https://github.com/google-ai-edge/LiteRT/pull/11192) (GPU). The steps are under [Build the runtime](#build-the-runtime). LiteRT-LM 0.19.2, the latest release on PyPI on 2026-11-16, and 0.19.1 do not run these files: with the PyPI package's `litert-lm run`, both stop on the CPU with `moe node #708 unsupported activation='silu'` and on the GPU with `moe only supports activation='gelu_tanh' (or its legacy spelling 'gelu')`, and the command prints `An error occurred`.

## Conversion notes

- Source: [moonshotai/Kimi-Linear-48B-A3B-Instruct](https://huggingface.co/moonshotai/Kimi-Linear-48B-A3B-Instruct) at revision `e1df551a447157d4658b573f9a695d57658590e9`, bf16 weights.
- The graph is written in plain TFLite ops plus LiteRT's `moe` custom op by a writer for litert-torch, checked against Transformers' `kimi_linear` model.
- Kimi Delta Attention (20 layers): the prefill computes the chunked delta rule (chunks of 16 tokens) and the decode the exact recurrent step, with the per-channel decay of the source. Each layer's state is the delta-rule state (32 heads × 128 × 128, fp32) and the last 3 inputs of the short convolution.
- MLA (7 layers, no rotary embedding, as in the source): the cache holds per position the normalized compressed latent (512 values) and the shared key part (64 values); attention runs in the absorbed form.
- Mixture of experts (26 layers): the router in plain ops (sigmoid scores, the top 8 chosen on the scores plus the correction bias, weights renormalized and scaled by 2.446, as in the source); the 256 routed experts in the `moe` op with SiLU; the shared expert and the dense first layer in plain ops.
- Quantization of the int8 file: the routed experts int8 per channel (one scale per output row and expert) by LiteRT's quantizer, and every other linear layer int8 per channel too (LiteRT's quantizer, symmetric, min-max): the Kimi Delta Attention projections (q, k, v, o and the gate and beta projections), the MLA projections (q, the compressed key-value input, o), the shared experts, the dense first layer, the routers and the output layer. fp32 stay the token table and the 16 × 16 masks of the Kimi Delta Attention chunks (prefill only); activations fp32. An earlier GPU form of this conversion (the experts, the attention projections and the output layer int8) gave an all-zero logits row on the GPU with fp16 activations declared, in the token-by-token comparison below, so the files declare fp32 activations.
- Quantization of the CPU file: the routed experts int8 per channel as above; every other weight fp32 (the token table and the output layer included); activations fp32.
- Signatures: `prefill_128`, `prefill_1024` and `decode`; context 4096 tokens.
- Tokenizer: the source ships a tiktoken tokenizer, so the files carry a `tokenizer.json` converted from it. On 6,612 texts (329,448 tokens: GSM8K and IFEval prompts, code, several languages, rendered chats and random token sequences) the converted tokenizer gives the same ids as the source tokenizer, with tokenizers 0.21.0 (the version LiteRT-LM links) as well.
- Chat template: the source's `chat_template.jinja` with four rewrites for LiteRT-LM's template engine (`.get(...)` lookups written as subscripts, `tojson` without arguments). The model has no thinking mode.
- Scripts: the conversion scripts and how each part was checked are in [hf-to-litertlm: kimi_linear_48b_a3b_work](https://github.com/john-rocky/hf-to-litertlm/tree/e945a06a4a43d6731cdc3b16de6cd5b7246e9800/kimi_linear_48b_a3b_work). A run of those scripts from a clean copy on 2026-11-15 rebuilt both files: every byte after the 16,384-byte header is the published one, and the header differs only in the uuid and the creation time, which litert-lm-builder writes anew at every build ([the steps and the check](https://github.com/john-rocky/hf-to-litertlm/blob/e945a06a4a43d6731cdc3b16de6cd5b7246e9800/REPRODUCE.md#kimi-linear-48b-a3b-instruct--the-27-layer-hybrid-moe-decoder-kimi-delta-attention-and-mla-written-with-a-graph-writer-for-litert-torch-the-256-silu-experts-of-each-moe-layer-in-one-moe-custom-op-with-int8-per-channel-weights-a-4096-token-cache-for-a-litert-lm-built-with-two-open-litert-pull-requests-2026-11-15)).

## Performance

Apple M4 Max (128 GB, macOS 27.0), `litert_lm_advanced_main` built as under [Build the runtime](#build-the-runtime), 1,024 prefill tokens and 256 decode tokens.

The int8 file (two measurement windows, 2026-11-14 and 2026-11-15):

- GPU command: `--backend=gpu --benchmark --benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256 --num_iterations=4 --cache_dir=:nocache --max_num_tokens=4096 --sampler_handles_input=false`, with the WebGPU accelerator of the build below.
- CPU command: `--backend=cpu --benchmark --benchmark_prefill_tokens=1024 --benchmark_decode_tokens=256 --num_iterations=2 --cache_dir=:nocache --max_num_tokens=4096 --num_cpu_threads=12`.
- GPU row: the mean of iterations 2 to 4 in each process (iteration 1 is the warm-up), then the median of 4 processes (decode 60.997 and 61.06 tok/s on 2026-11-14, 61.073 and 60.187 on 2026-11-15; prefill 706.15, 743.52, 709.70 and 685.67 tok/s; in run order). The GPU processes alternated with processes of the same file on ML Drift's Metal accelerator, built from the same sources but not part of the block below: decode 59.06 and prefill 887.91 tok/s (the median of the 4 Metal processes kept by the rule below).
- CPU row: 3 processes run one after another on 2026-11-15, after the GPU processes, iteration 2 in each (iteration 1 is the warm-up), then the median of the 3 (decode 13.44, 11.98 and 11.86 tok/s; prefill 137.65, 134.31 and 133.42 tok/s; in run order).
- What else ran: a process whose counted iterations overlapped other work on the Mac was left out of the rows: on 2026-11-14, 2 of the 4 WebGPU processes, 3 of the 4 Metal processes and all 3 CPU processes (file transfers of about 4 GB by other jobs, another job's downloads, bursts of other processes such as git, a large file read); on 2026-11-15, 1 of the 3 WebGPU processes (one git process at up to 13 cores for 2 s). In the processes kept, no other benchmark or build ran. The Mac's own processes were not stopped: during the whole 2026-11-15 window the system process `dasd` used about one core (98 % CPU on average in `ps` samples about once a second, 84 % during the CPU processes; under 20 % in every sample on 2026-11-14), and the CPU row comes from that window.
- Peak footprint, read by `top` as for the CPU file below: 62 GiB on the GPU, given as 67 GB, and 6.9 GB on the CPU. The physical footprint does not count clean pages of mapped files: the 50.50 GB file is read through the page cache on top of it.

The CPU file (2026-11-13):

- Command: the CPU command above.
- CPU row: 3 processes run one after another, iteration 2 in each (iteration 1 is the warm-up), then the median of the 3 (decode 9.60, 9.72 and 9.79 tok/s; prefill 133.75, 134.09 and 134.84 tok/s; in run order). The first process found the file out of memory and read 53 GB of it from the SSD: its iteration 1 prefilled 31.14 tok/s, against 124.5 and 124.7 in the other two.
- Threads: without `--num_cpu_threads` the runtime used 4 threads, and the same command gave prefill 76.19 and decode 7.15 tok/s (3 processes, interleaved with the 12-thread ones).
- What else ran: during the counted iterations (sampled about once a second) no other benchmark or build was seen. In the first process's counted iteration a CI job on the same Mac (up to 35 % CPU for about 4 s) and a git process (up to 81 % CPU for about 2 s) ran; that process decoded 9.60 tok/s. Still running were a watcher script (0.0 % CPU in every sample), open apps (among them a browser and a command-line tool, each above 20 % CPU in 1 sample at most) and short `top` reads, this measurement's footprint reads among them. The Mac's own processes were not stopped: in the first process's counted iteration fseventsd used up to 85 % CPU for about 2 s.
- Peak footprint: the largest physical footprint of the process, read by `top` about every 17 s; `top` reports whole GiB at this size (18 GiB), given here as 19 GB. The physical footprint does not count clean pages of mapped files: the 55.29 GB file is read through the page cache on top of it.

## Accuracy

12 questions and a 3-turn conversation, greedy (`--sampler_top_k=1`), up to 2,048 tokens per answer, judged against Transformers (the bf16 checkpoint, computed in fp32 on the CPU) by the same regular expression on the answer. Each question ends with "Answer briefly.": two arithmetic questions, a word problem, two decimals to compare, three facts, the opposite of a word, a French and a Japanese translation, a rhyme, and a Python `fib(n)` that is also asked for `fib(10)`. The conversation gives a name and a number in turn 1, asks something else in turn 2, and asks for the number in turn 3; it runs in one process (`--multi_turns`). Transformers answers 12 of 12, ends each answer with the stop token (2 to 91 tokens) and passes the 3 turns.

The int8 file on WebGPU (2026-11-14) gives Transformers' verdict on 12 of 12 questions, ends each answer with the stop token and passes the 3 turns (turn 1: "Hello, Kaito! May 37 bring you luck and endless possibilities."; turn 3: "Your lucky number is 37."). Its `fib(10)` answer has 79 tokens, against Transformers' 91.

The int8 file on the CPU (12 threads, 2026-11-14) gives Transformers' verdict on 12 of 12 questions, ends each answer with the stop token and passes the 3 turns (turn 3: "Your lucky number is 37."). In the first run, question 4 reached the CLI's 10-minute limit without an answer while the file was read from a USB hard disk; run again in a new process, it answered "Seven days." with the stop token in 16.8 s. Its `fib(10)` answer has 115 tokens, against 91.

The CPU file (12 threads, 2026-11-13) gives Transformers' verdict on 12 of 12 questions, ends each answer with the stop token and passes the 3 turns (turn 3: "Your lucky number is 37."). The wording can differ from Transformers': the `fib(10)` answer ends after the code block, where Transformers adds a line with the value, and the turn-1 greeting ends "luck and endless possibilities." where Transformers writes "luck and laughter every day.".

Before the questions, each whole file was compared with Transformers token by token: the first 24 tokens of a GSM8K question in the chat template, fed one at a time through the decode signature (the bar: Transformers' top-1 token at 22 of the 24 positions). The int8 file picks Transformers' top-1 token at 22 of the 24 positions on WebGPU and on the CPU (on the CPU in each of 3 processes, whose logits are the same to the byte); the two other positions, the same on both backends, are where Transformers' two best tokens are 0.026 and 0.033 apart in logit, and Transformers' top token is in the file's top 5 at all 24 positions (largest logit difference over the 24 positions 2.55 on WebGPU, mean 0.145; 3.04 on the CPU, mean 0.195). The CPU file on the CPU picks Transformers' top-1 token at 23 of the 24 positions, the same in each of 3 processes; the one other position is where Transformers' two best tokens are 0.00035 apart in logit, and there the file's pick is Transformers' second (largest logit difference over the 24 positions 2.29, mean 0.096).

## Build the runtime

LiteRT-LM `02d7e8d8` (main, 2026-11-07) with the LiteRT commit its WORKSPACE pins, `bf49eb62`, and the two pull requests applied without their test files (one test file of #11192 does not apply at `bf49eb62`; no test is built). ml-drift is the WORKSPACE's own archive. `--override_repository` skips the patch commands that LiteRT-LM's WORKSPACE applies to the LiteRT archive, so [`apply_ws_patch_cmds.py`](https://github.com/john-rocky/hf-to-litertlm/blob/84336e8d00a567464e834854bdc1987a5c6656ca/qwen3_30b_a3b_work/apply_ws_patch_cmds.py) (kept in the hf-to-litertlm folder of the Qwen3-30B-A3B conversion) applies them by hand; it needs GNU sed installed as `gsed`. The block also needs Git LFS and Bazel (LiteRT-LM's `.bazelversion` names 7.6.1, which Bazelisk runs); it ran here with Xcode 27.0 on macOS 27.0. Run it in an empty directory:

```bash
git clone https://github.com/google-ai-edge/LiteRT-LM.git && cd LiteRT-LM
git checkout 02d7e8d80cb75cbb03d8cba4c63f51e2eb318ec5
git lfs pull --include "prebuilt/macos_arm64/*"
cd ..
git clone https://github.com/google-ai-edge/LiteRT.git && git -C LiteRT checkout bf49eb62406d15948261a840c9dcd8e4f13b478c
curl -fLO https://raw.githubusercontent.com/john-rocky/hf-to-litertlm/84336e8d00a567464e834854bdc1987a5c6656ca/qwen3_30b_a3b_work/apply_ws_patch_cmds.py
python3 apply_ws_patch_cmds.py LiteRT-LM/WORKSPACE litert LiteRT
curl -L https://github.com/google-ai-edge/LiteRT/commit/ebacd381a0d223a585ec66c1e880cffa640b29be.diff | git -C LiteRT apply --exclude='*_test.cc'
curl -L https://github.com/google-ai-edge/LiteRT/commit/9860c14d7b63f6eaa85b9bfbe807afc971b85fe4.diff | git -C LiteRT apply --exclude='*_test.cc'
cd LiteRT-LM
bazel build --config=macos_arm64 -c opt --define=litert_runtime_link_mode=dynamic \
  --override_repository=litert=$PWD/../LiteRT \
  //runtime/engine:litert_lm_advanced_main @litert//litert/c:litert_runtime_c_api_dylib \
  @litert//litert/runtime/accelerators/gpu:ml_drift_webgpu_accelerator_dylib
```

The two commits are the pull requests' heads on 2026-11-13 (each pull request has one commit): `ebacd381` is #11192 and `9860c14d` is #11188. Copy `bazel-bin/runtime/engine/litert_lm_advanced_main` and `bazel-bin/external/litert/litert/c/libLiteRt.dylib` into one directory with `prebuilt/macos_arm64/libwebgpu_dawn.dylib` and `prebuilt/macos_arm64/libGemmaModelConstraintProvider.dylib` (paths from `LiteRT-LM`, where the block ends): the CLI links the last two and does not start without them, also on the CPU. For the GPU command, also copy the block's third target, `bazel-bin/external/litert/litert/runtime/accelerators/gpu/libLiteRtWebGpuAccelerator.dylib`, into that directory (the CPU command does not need it). Put the `.litertlm` file in the same directory and run the model card's command there.

## Run

The model card's two commands were run as written with the int8 file, in a directory holding the five files above and the model (2026-11-15: both answered "42"). The CPU command sets 12 threads (`--num_cpu_threads=12`); without the flag the runtime uses 4 threads. The GPU command sets `--sampler_handles_input=false`, as the GPU checks did. The files carry no sampler settings of their own; the checks decode greedily (`--sampler_top_k=1`).

## Conversations

The Kimi Delta Attention layers carry a recurrent state that summarizes every token processed so far. On LiteRT-LM releases up to 0.19.1, a new conversation on an engine that served another one continues from that conversation's state (LiteRT-LM issue [#3165](https://github.com/google-ai-edge/LiteRT-LM/issues/3165), open). The main build these files need carries LiteRT-LM commit [`e1f86b05`](https://github.com/google-ai-edge/LiteRT-LM/commit/e1f86b05bf8a731331622a2622440e9f93d9a9f5), which zeroes the state when a new conversation starts; this was not checked with these files. Create a new engine for each conversation.

## Devices

Measured on a desktop only (Apple M4 Max, 128 GB). No phone was tried: the files are 50.50 GB and 55.29 GB and do not fit in a phone's memory. iOS / iPhone and Android were not measured.

## Tool calls

The files declare LiteRT-LM's generic model type and carry no tool-call configuration (their metadata sections are the same bytes), so LiteRT-LM returns a tool call as reply text. Tool calling was not run with these files. Rendering the template without the runtime, with minijinja 2.14.0 (the template engine LiteRT-LM uses), tool declarations come out with `, ` and `: ` in the JSON, where the source's template writes compact JSON. The template prints only the `text` of each content part, so a tool result passed as a part without text, such as LiteRT-LM's `tool_response` part, renders empty.