Escha Runtime β€” qwen3moe

By Escha Labs Inc.

Serving runtimes for Escha 2-/3-bit (eschamoe) quantized models of the qwen3_5_moe architecture (Qwen3.5 / Qwen3.6 Mixture-of-Experts, 256 experts). One repo per model architecture, one directory per engine β€” pick the engine that fits your workload:

SGLang β€” sglang/ ZML β€” zml/
Best for servers & teams one user, one stream, zero-dependency deploys
Concurrency continuous batching, paged KV, radix prefix cache one request at a time
Single-user decode (4090)[^grid] 218–231 tok/s (flat in output length) 235–241 tok/s on β‰₯500-token answers (+8–14%); ~190 tok/s on ≀128-token replies
TTFT, 128β†’1890-tok prompt[^grid] 0.06–0.24 s 0.06–0.35 s
Max context (4090)[^ctx] up to 159k tok @ MEM=0.78, 235k @ 0.90 (ships CTXLEN=32768) 262,144 @ ESCHA_MEM=0.93 (ships ESCHA_CTX=1024)
Multi-turn prompt reuse radix cache (branching, cross-session) append-only, single session
Tool calls / JSON schema / thinking parser yes no
Sampling (temperature>0) full speed ~104 tok/s (fast path is greedy-only)
Install Python 3.12 venv + CUDA-12 PyTorch one binary, no Python, no CUDA toolkit

Neither engine is uniformly faster. ZML wins sustained decode on long single-stream answers and is far simpler to deploy; SGLang wins short replies, wins time-to-first-token on long prompts, and is the only option for concurrency, tool calling or structured output. If unsure: multi-user, agents-at-scale, or structured output β†’ SGLang; long-form single-user generation on your own GPU β†’ ZML.

The ZML engine is built on ZML (Zig + MLIR/XLA); the SGLang engine on a fork of SGLang. Both run the same Escha CUDA kernels and serve the same model files.

Apple Silicon

Both engines above are NVIDIA/Linux only. On a Mac, use escha-mlx (separate repo, Apache-2.0): an MLX runtime with Metal kernels, continuous batching, prefix caching and the same OpenAI-compatible endpoint. It serves the same model files with no conversion step.

Requires Apple Silicon M1–M5, macOS 14+, Python 3.10–3.13 and 24 GB of unified memory. Measured resident footprint is 12.25 GB; single-stream decode is 27.3 tok/s on an M4 base and 59.7 tok/s on an M5 Pro, with aggregate throughput of 185.6 and 539.3 tok/s respectively at batch 128. Those are not comparable to the CUDA figures above: an M4 base moves ~120 GB/s against a 4090's 1008 GB/s, and decode here is memory-bound. Full tables, install steps and a head-to-head against stock MLX 4-bit are in that repo.

[^grid]: Same-harness measurement, 2026-07-26: one RTX 4090, the same model files, the same streaming OpenAI client, greedy, batch 1, medians of 2 reps over an ISLΓ—OSL grid (128–1890 in, 128–1890 out). Decode = steady-state tokens/s between the first and last streamed token. Per-cell decode: SGLang 176.8–230.9 (median 218.2), ZML 184.6–240.7 (median 234.8); ZML leads every cell with β‰₯256 output tokens and trails on 128-token replies, where its 16-token fused decode chunk dominates. TTFT by input length (ZML / SGLang): 128 tok 55/55 ms Β· 500 tok 95/91 ms Β· 1000 tok 178/143 ms Β· 1890 tok 353/237 ms.

[^ctx]: Measured 2026-07-26 on one RTX 4090 (24 GB) with Qwen3.6-35B-A3B-Escha-W2. Both engines ship a conservative default context and let you raise it. SGLang reports its KV pool at startup; launched with CTXLEN=262144 it allocated 159,480 tokens at its shipped MEM=0.78 and 234,796 at MEM=0.90 β€” that pool size is the ceiling for a single sequence. ZML serves a 261,966-token prompt at ESCHA_CTX=262144. Both are memory-bound near the cap, not architecture-bound: KV on this hybrid model is only ~20 KiB per token (10 attention layers; the 30 gated-delta-net layers hold a fixed ~66 MB recurrent state), so 262,144 tokens is 5.37 GB beside 12.3 GB of weights. In fairness to SGLang: its 234,796-token pool at MEM=0.90 still left 1.38 GB of VRAM free, so it would very likely also reach the 262,144 cap at a higher mem-fraction-static β€” we did not test that, and 0.78 is simply the conservative default the runtime ships. Read this row as "both engines reach the model's native context on a 24 GB card", not as a ZML advantage.

Compatible models

Model repo Bits
EschaLabs/Qwen3.6-35B-A3B-Escha-W2 2-bit (eschamoe)

These runtimes target qwen3_5_moe. A model of a different architecture will not load β€” use the matching escha-runtime-<arch> repo.

Quickstart

SGLang engine (full detail: sglang/INSTALL.md, incl. the per-GPU cookbook):

python3.12 -m venv .venv && source .venv/bin/activate
pip install -U pip wheel
pip install "torch>=2.9" --index-url https://download.pytorch.org/whl/cu128   # cu12 torch first
pip install ./sglang/escha-*.whl   # pulls the bundled sglang fork + its full dep closure

hf download EschaLabs/Qwen3.6-35B-A3B-Escha-W2 --local-dir ./Qwen3.6-35B-A3B-Escha-W2
MODEL=./Qwen3.6-35B-A3B-Escha-W2 bash sglang/serve.sh

Single-user speedup with MTP speculative decoding (1.0.4+, about 1.57Γ— decode on a 4090): build the draft once from Qwen's own release, then serve with MTP=1 β€” see INSTALL.md β†’ Speculative decoding.

python sglang/make_mtp_draft.py --out ./Qwen3.6-35B-A3B-MTP-draft
MODEL=./Qwen3.6-35B-A3B-Escha-W2 MTP=1 DRAFT=./Qwen3.6-35B-A3B-MTP-draft bash sglang/serve.sh

ZML engine (full detail: zml/INSTALL.md) β€” no Python at all:

tar xf zml/escha-zml-serve-*-linux-x86_64.tar.gz && cd escha-zml-serve-*
hf download EschaLabs/Qwen3.6-35B-A3B-Escha-W2 --local-dir ./model
./escha-serve

Long context and agentic use (ZML engine)

ESCHA_CTX reaches the model's native 262,144 tokens. Measured on one 4090 with facts planted at 10%, 50% and 90% depth and all three asked for at the end β€” the check that positions and recurrent state thread correctly across the whole prompt:

prompt prefill rate recall
15,066 tok 5.3 s 2,858 tok/s 3/3
60,066 tok 24.9 s 2,411 tok/s 3/3
129,966 tok 79.0 s 1,646 tok/s 3/3
261,966 tok 259.1 s 1,011 tok/s 3/3

Prefill is ~NΒ²/2 work, so the rate falls with length. The comfortable band on a 24 GB card is ~8k–130k, where TTFT is seconds; 262k answers correctly but takes ~4.3 minutes to read.

Multi-turn: a conversation whose prompt grows each turn reuses the previous prefix. On a 14k-token conversation the first turn takes 4.11 s and each following turn 1.27 s (~99.8% of the prompt reused). Reuse requires an exact append β€” editing earlier history re-reads the prompt β€” and only one conversation is cached. SGLang's radix cache is faster in absolute terms here and handles branching and multiple sessions; the ZML cache exists to make single-session agent loops practical, not to match it.

Decode slows as position grows on both engines, because decode attention is O(position). On ZML, measured with prefill excluded: 227 tok/s at position ~1k, 163 at 3k, 124 at 7.5k. (That measurement divides total request time and so reads ~5% below the streamed grid above β€” method, not configuration.)

Requirements (shared)

  • NVIDIA GPU, compute capability 8.0–12.0 (Ampere β†’ Blackwell), Linux x86-64. Kernel launch route auto-selects per GPU. Per-architecture and per-VRAM launch recipes: sglang/INSTALL.md β†’ Running on your GPU.
  • SGLang engine: Python 3.12 + CUDA-12 PyTorch 2.9 or newer (the wheel handles every other dependency). 24 GB VRAM recommended (16 GB works with reduced context). As of 1.0.3 the wheel is built against PyTorch's stable ABI, so one binary serves torch 2.9 and every later release β€” earlier wheels were locked to a single torch minor. Verified byte-identical output and equal decode speed (205.9 / 227.7 tok/s bs1 on a 4090, shipped default / 24 GB config) on torch 2.9.1 and 2.11.0. On torch 2.10+ install sglang-kernel from sglang's CUDA-12 index first; see sglang/INSTALL.md β†’ Supported PyTorch versions.
  • ZML engine: no runtime dependencies β€” the archive bundles the CUDA runtime; just an NVIDIA driver (R550+) and glibc 2.27+ (RHEL/Rocky/Alma 8 and newer). 24 GB VRAM; one request at a time.

Changelog

1.0.6 (2026-10-31) β€” Single-user decode +5 %; all current escha optimizations in the runtime. At decode (up to 8 tokens per step) each MoE layer now computes the router, the top-k selection and the expert tables in one kernel and folds the shared expert into the expert sum (one launch instead of a GEMV, a sigmoid, a multiply and an add), with the router and routed experts overlapped against the shared expert. The full-attention layers run q/k norm + rotary embedding and the output gate as two fused kernels (bit-identical to the unfused ones). On an RTX 4090 (same session, two interleaved runs each): single-user decode 224.5 β†’ 236.0 tok/s (128-token prompt), 222.2 β†’ 233.2 tok/s (2k prompt) and 202.5 β†’ 211.7 tok/s (20k prompt); 8 users, 128-token prompts β†’ 2,048-token answers: 880 β†’ 899 tok/s; prefill unchanged. Quality: MATH-500 (thinking on, n = 500) 88.0 % vs 88.2 % for 1.0.5 (paired p = 1.0). SGLANG_ESCHA_MOE_GLUE=0, ESCHA_MOE_ROUTE_BUILD=0 and SGLANG_ESCHA_ATTN_FUSE=0 each restore the 1.0.5 path. With ESCHA_MOE_GROUPED_PREFILL=0, prefill chunks of 8,192+ tokens now work too.

Backward compatible: a drop-in upgrade from 1.0.4 or 1.0.5. Same model files, same serve.sh, same dependencies: any command line that worked with 1.0.4 launches the same server, and pip install --no-deps ./escha-1.0.6+qwen3moe-*.whl upgrades an existing environment in place. escha._C keeps every 1.0.4 operation with an unchanged signature (one operation added), and no Python API was removed. Checked on an RTX 4090 (single user, 8 prompts Γ— 384 greedy tokens, 2 runs, deterministic MoE reduction ESCHA_MOE_FUSE_EPI_FIX=1 ESCHA_ENABLE_DET_SCATTER=1): with ESCHA_MOE_GROUPED_PREFILL=0 SGLANG_ESCHA_MOE_GLUE=0 SGLANG_ESCHA_ATTN_FUSE=0, 1.0.6 writes text byte-identical to 1.0.4 on every prompt, with and without speculative decoding (SPEC=1), at the same speed (225.8 vs 225.7 tok/s). With the defaults the shared expert enters the expert sum in a different order, so 4 of the 8 answers move away from 1.0.4's text after 190–1,100 characters; quality is unchanged (MATH-500 above). The default launch, THINK=0 and MTP=1 all serve.

Best measured configurations of 1.0.6 on an RTX 4090 (NVIDIA's ISL/OSL grid, real prompts, idle desktop; throughput = output tokens per second of wall-clock time, prefill included):

configuration decode, short context decode at 20k context time to first token (2k / 5k / 20k prompt)
single user, RADIX=0 bash serve.sh 236 tok/s 212 tok/s 0.19 / 0.39 / 1.69 s
single user, MTP=1 DRAFT=… bash serve.sh (speculative) 357–395 tok/s (2–4k answers) 250 tok/s 0.19 / 0.40 / 1.78 s
multi-user, RADIX=0 MAXREQ=32 MAXMAMBA=32 CUDA_GRAPH_BS="1 2 4 8 16 24 32" bash serve.sh 8 users 16 users 32 users
128-token prompts β†’ 2,048-token answers: throughput (per user) 891 tok/s (113) 1,156 tok/s (73) 1,138 tok/s (36)
1,000 β†’ 1,000 tokens: throughput / time to first token 847 tok/s / 0.50 s 1,084 tok/s / 0.79 s 1,071 tok/s / 1.40 s
2,048 β†’ 128 tokens (prefill-heavy): throughput / time to first token 438 tok/s / 0.89 s 489 tok/s / 1.54 s 485 tok/s / 2.79 s

1.0.5 (2026-10-31) β€” Prefill up to 1.9Γ— faster; the current escha kernels. The SGLang engine now runs the routed experts of every prefill batch (more than 64 tokens) as a grouped tile GEMM: each expert's weights are decoded once per 128 routed rows instead of once per 8, and routing happens on the GPU without a host sync. escha._C is rebuilt from the current escha kernels (same operations, same PyTorch stable ABI, SASS for sm_80–sm_120). Measured on an RTX 4090 (NVIDIA's ISL/OSL benchmark grid, real prompts, RADIX=0, everything else default):

1.0.4 1.0.5
prefill, 2,048-token prompt 5,935 tok/s (TTFT 355 ms) 10,920 tok/s (TTFT 189 ms)
prefill, 5,000-token prompt 7,097 tok/s (TTFT 707 ms) 12,854 tok/s (TTFT 390 ms)
prefill, 20,000-token prompt 6,870 tok/s (TTFT 2.91 s) 11,784 tok/s (TTFT 1.70 s)
8 users, 2,048-token prompts, 128-token answers: TTFT / throughput* 1.69 s / 305 tok/s 0.91 s / 432 tok/s
single-user decode 225 tok/s 225 tok/s

* Corrected 2026-10-31: throughput = output tokens per second of wall-clock time, prefill included. The first version of this row summed the users' decode rates (673 β†’ 728 tok/s), which leaves out the prefill speed-up.

Quality: MATH-500 (thinking on, n = 500) 88.2 %, within sampling noise of the 90.0 % reference run (paired p = 0.20). Prompt log-probabilities differ from 1.0.4 no more than 1.0.4 differs from itself between runs, and 1.0.5's prefill is bit-for-bit repeatable.

Every existing setting is unchanged β€” any command line that worked with 1.0.4 launches the same server. ESCHA_MOE_GROUPED_PREFILL=0 restores the 1.0.4 prefill path. Decode runs the same arithmetic as 1.0.4. Also fixed: a prefill chunk of 8,192 or more tokens (CHUNK=8192) made 1.0.4's MoE prefill fail with moe_linear: CUDA error; 1.0.5 handles any chunk size. The ZML engine is unchanged.

1.0.4 (2026-10-29) β€” MTP speculative decoding for the SGLang engine. New opt-in preset MTP=1 serves the published W2 model with its own MTP draft: sglang/make_mtp_draft.py rebuilds that draft from Qwen's FP8 release (the W2 repo carries the block without its routed experts), the bundled SGLang hands the int8-as-stored embedding and output head to the draft instead of failing on them, and the draft scores only the 32,768 most frequent tokens (HOTMAP, sglang/hot_token_map_32k.pt). Single-user decode on an RTX 4090 goes from 226.8 to 356.1 tok/s (1.57Γ—); GSM8K is unchanged (96.8% both). See INSTALL.md β†’ Speculative decoding.

Every existing setting is unchanged. MTP and HOTMAP are new and off by default; any command line that worked with 1.0.3 launches the same server. The compiled escha._C is byte-identical to 1.0.3 and the Python changes are on the speculative path only: with the deterministic settings, greedy output is byte-identical to 1.0.3 at SPEC=0 and at SPEC=1 (16/16 prompts each) at the same speed. The ZML engine is unchanged.

Known limitations

  • DETERMINISTIC=1 fails on consumer Blackwell (sm_120). The deterministic attention kernel requests 104 KB of shared memory per block, above the sm_120 limit, and the server exits during startup. It works on Ampere, Ada and Hopper. Greedy output is in any case not bit-reproducible across requests on either engine β€” batch composition changes fp16 accumulation order, so a near-tie can flip and a long reasoning chain diverges from there.
  • The ZML engine serves no /v1/completions (HTTP 404). Use /v1/chat/completions.
  • The ZML engine does not validate requests. An unknown model field is served anyway instead of returning model_not_found, and usage.prompt_tokens is wrong when a prompt is truncated.
  • The ZML engine can wedge on an over-long prompt on a 16 GB card, requiring kill -9 rather than returning an error. This is why it asks for 24 GB β€” use the SGLang engine on 16 GB, where the same prompt returns a clean HTTP 400.

License

Everything here is released under the Apache License, Version 2.0 β€” see LICENSE. All bundled third-party code is permissive (Apache-2.0 / MIT / BSD-3-Clause; NVIDIA runtime libraries in the ZML bundle under the NVIDIA EULA's redistributable-runtime terms) β€” no copyleft. Full texts and the component inventory: THIRD_PARTY_LICENSES/. Model weights are not in this repo and carry their own license in the model repository.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support