gpt-oss-120b LiteRT-LM

This repository is a conversion of openai/gpt-oss-120b to LiteRT-LM, written one layer at a time with the LiteRT Tensor API and packed with litert-lm-builder 0.19.1. It has one file for the CPU and one for the GPU, and both run with LiteRT-LM 0.19.1 from PyPI on a Mac with 128 GB of memory. Through that package's Python API, the GPU file's replies to 12 single questions got the same verdicts as the source model's, but in a 3-turn conversation its second and third replies were only "!" characters (one run; the cause was not found). One run of the C++ CLI built from the v0.19.1 tag, with --sampler_handles_input=false, matched the source model's verdict on all 3 turns (Runtime notes). On an Apple M4 Max at 1,024 / 256 tokens (2026-11-12), the GPU file prefilled 1,012.98 tokens/s and decoded 66.72 tokens/s, and the CPU file prefilled 55.00 tokens/s and decoded 4.82 tokens/s. The GPU run's peak footprint was 85.83 GB. On the CPU, the 77.3 GB file is mapped into memory whole: peak resident memory was 82.1 GB for the benchmark and 80.6 GB for the Python example below.

openai/gpt-oss-120b is OpenAI's open-weight reasoning model: 36 layers alternating sliding-window (128 tokens) and full attention, 128 experts per layer with 4 used for each token, 117B parameters with 5.1B active (source model card). It writes its reasoning in an analysis channel before the answer (the harmony format).

What differs from the source: the experts' MXFP4 weights are re-quantized to int4, and the expert activation is LiteRT-LM's GELU (tanh form) without the source's clamp at 7, because the moe op of LiteRT-LM 0.19.1 has no clamp and no SiLU. The GPU file computes the same function in a graph rescaled for fp16 activations. On 12 questions, greedy replies from each file got the same verdict as the source model's on all 12 (Accuracy). How each part was made and checked: MEASUREMENTS.md.

Available Artifacts

File	Backend	Quantization	Context	Size
gpt_oss_120b_mixed_int4_gpu_ekv4096.litertlm	GPU	as the CPU file (experts int4 blockwise 32; attention q / k / v / o and lm_head int8 per channel), with scales moved by powers of two; fp16 activations	4096	77.26 GB
gpt_oss_120b_mixed_int4_ekv4096.litertlm	CPU	experts int4 blockwise (block 32, fp32 scales); attention q / k / v / o and lm_head int8 per channel; router, biases, sinks, norms and token table fp32; fp32 KV cache	4096	77.26 GB

Conversion Notes

• Each layer's 128 experts are one TFLite custom op moe; only the 4 routed experts are computed for each token.
• Runtime floor: LiteRT-LM 0.19.0. The moe op declares activation='gelu_tanh'; the GPU moe kernel of LiteRT-LM 0.18 only supports activation='gelu'.
• The activation is (up + 1) · GELU_tanh(gate) without the source's clamps. The activation was compared with the source's clamped SwiGLU over all 36 layers: with it and the MXFP4 weights unchanged, the model picks the source's next token at 399 of 413 reply positions (96.6 %, teacher-forced; details).
• The GPU file divides every RMSNorm input by its largest absolute value, carries the residual stream at 1/8, scales q before q·k and uses 1/64 as the experts' bias unit, because an fp16 graph without these overflows (details).
• The residual add after each moe op is written as (x + moe) + moe * 0 (numerically exact), so a GPU delegate cannot fuse the add into the moe kernel.
• A reply's content is the final message; the analysis is in channels["analysis"]. Tool calls come back as text (metadata type generic_model).
• Sampler TOP_P (top_k 50, top_p 1.0, temperature 1.0); stop tokens 200002, 199999, 200012.
• Reproduce: john-rocky/hf-to-litertlm: gpt_oss_120b_work.

Performance

Speed still being tuned (updates follow).

Apple M4 Max (128 GB, macOS 27.0), LiteRT-LM 0.19.1 from PyPI, 1,024 prefill tokens and 256 decode tokens, measured on 2026-11-12 (how).

File	Backend	Prefill (tok/s)	Decode (tok/s)	Time to first token	Peak footprint	Peak resident memory
gpt_oss_120b_mixed_int4_gpu_ekv4096.litertlm	GPU (WebGPU)	1,012.98	66.72	1.03 s	85.83 GB	55.5 GB
gpt_oss_120b_mixed_int4_ekv4096.litertlm	CPU, 8 threads	55.00	4.82	18.8 s	6.98 GB	82.1 GB

Accuracy

12 questions and a 3-turn conversation, greedy, each reply's final channel judged by the same function as the source model's reply (how).

File	Backend	Same verdict as the source model	Ends with a stop token	3 turns, none worse than the source model's
gpt_oss_120b_mixed_int4_gpu_ekv4096.litertlm	GPU (WebGPU), C++ CLI, --sampler_handles_input=false	12 / 12	12 / 12	3 / 3
gpt_oss_120b_mixed_int4_ekv4096.litertlm	CPU, 8 threads, Python API	12 / 12	12 / 12	3 / 3

Runtime notes

• The GPU file through the Python API. With LiteRT-LM 0.19.1 from PyPI, the GPU file gave the same 12 verdicts, but in the 3-turn conversation its second and third replies were "!" until the 2,048-token cap and then the 4,096-token context were reached (one run; the cause was not found). The Python API of 0.19.1 has no setting for sampler_handles_input (its default is on); the GPU row of the Accuracy table ran the C++ CLI litert_lm_advanced_main from the LiteRT-LM v0.19.1 tag with it off (command).

• Conversations on the GPU. Build that CLI from the v0.19.1 tag (with Xcode, Git LFS and Bazelisk installed) and run it on a Mac with 128 GB of memory, with the flags of the Accuracy table's 3-turn run; the prebuilt litert_lm_main of the v0.19.1 release does not read them. On the Mac above, the build took 14 minutes with 4 parallel jobs, and the clone with Bazel's output used 8.4 GB of disk. The prompt appears once the engine has loaded, about 3 minutes on the Mac above. Each line you type is one turn, and an empty line ends the conversation.

  hf download litert-community/gpt-oss-120b gpt_oss_120b_mixed_int4_gpu_ekv4096.litertlm --local-dir .
  GIT_LFS_SKIP_SMUDGE=1 git clone --depth 1 --branch v0.19.1 https://github.com/google-ai-edge/LiteRT-LM.git
  cd LiteRT-LM
  git lfs pull --include="prebuilt/macos_arm64/*"
  bazelisk build --config=macos_arm64 -c opt //runtime/engine:litert_lm_advanced_main
  mkdir run && cp bazel-bin/runtime/engine/litert_lm_advanced_main prebuilt/macos_arm64/*.dylib run/
  cd run
  ./litert_lm_advanced_main --model_path=../../gpt_oss_120b_mixed_int4_gpu_ekv4096.litertlm --backend=gpu \
    --sampler_handles_input=false --sampler_top_k=1 --max_output_tokens=2048 --cache_dir=:nocache --multi_turns
  

• Long replies. With LiteRT-LM 0.19.1, a blocking send_message that is still generating after 10 minutes fails with DEADLINE_EXCEEDED (the engine's default timeout). The CPU file decodes about 4.8 tokens/s on the Mac above, so 10 minutes is about 2,900 tokens, reasoning included. Use send_message_async, or cap max_output_tokens.

• A message that does not fit. On LiteRT-LM 0.19.1, after a message fails because it does not fit in the context window (4,096 tokens), every later message in that conversation fails too; start a new conversation.

Run

pip install litert-lm==0.19.1
hf download litert-community/gpt-oss-120b gpt_oss_120b_mixed_int4_ekv4096.litertlm --local-dir .

import litert_lm

engine = litert_lm.Engine(
    "gpt_oss_120b_mixed_int4_ekv4096.litertlm",
    backend=litert_lm.Backend.CPU(thread_count=8),
    cache_dir=":nocache",
)
conversation = engine.create_conversation()
reply = conversation.send_message("What is the capital of Japan? Answer briefly.")
print("Reasoning:", reply.channels.get("analysis", ""))
print("Answer:", reply)

Output on the Mac above (the file's sampler, top-k 50 at temperature 1.0, so the text varies from run to run):

Reasoning: User asks: "What is the capital of Japan? Answer briefly." Provide brief answer: Tokyo.
Answer: Tokyo.

For the GPU, download gpt_oss_120b_mixed_int4_gpu_ekv4096.litertlm instead and pass backend=litert_lm.Backend.GPU(); on the Mac above the engine took 160 s to load. For conversations of more than one turn through this API, see Runtime notes.

License

Apache License 2.0, the license of openai/gpt-oss-120b; see also the source's usage policy.