Instructions to use EschaLabs/escha-runtime-qwen3moe with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use EschaLabs/escha-runtime-qwen3moe with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download EschaLabs/escha-runtime-qwen3moe --local-dir escha-runtime-qwen3moe
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
Escha Runtime β qwen3moe
Serving runtimes for Escha 2-/3-bit (eschamoe) quantized models of the
qwen3_5_moe architecture (Qwen3.5 / Qwen3.6 Mixture-of-Experts, 256 experts). One repo
per model architecture, one directory per engine β pick the engine that fits your workload:
SGLang β sglang/ |
ZML β zml/ |
|
|---|---|---|
| Best for | servers & teams | one user, one stream, zero-dependency deploys |
| Concurrency | continuous batching, paged KV, radix prefix cache | one request at a time |
| Single-user decode (4090)[^grid] | 218β231 tok/s (flat in output length) | 235β241 tok/s on β₯500-token answers (+8β14%); ~190 tok/s on β€128-token replies |
| TTFT, 128β1890-tok prompt[^grid] | 0.06β0.24 s | 0.06β0.35 s |
| Max context (4090)[^ctx] | up to 159k tok @ MEM=0.78, 235k @ 0.90 (ships CTXLEN=32768) |
262,144 @ ESCHA_MEM=0.93 (ships ESCHA_CTX=1024) |
| Multi-turn prompt reuse | radix cache (branching, cross-session) | append-only, single session |
| Tool calls / JSON schema / thinking parser | yes | no |
Sampling (temperature>0) |
full speed | ~104 tok/s (fast path is greedy-only) |
| Install | Python 3.12 venv + CUDA-12 PyTorch | one binary, no Python, no CUDA toolkit |
Neither engine is uniformly faster. ZML wins sustained decode on long single-stream answers and is far simpler to deploy; SGLang wins short replies, wins time-to-first-token on long prompts, and is the only option for concurrency, tool calling or structured output. If unsure: multi-user, agents-at-scale, or structured output β SGLang; long-form single-user generation on your own GPU β ZML.
The ZML engine is built on ZML (Zig + MLIR/XLA); the SGLang engine on a fork of SGLang. Both run the same Escha CUDA kernels and serve the same model files.
Apple Silicon
Both engines above are NVIDIA/Linux only. On a Mac, use escha-mlx (separate repo, Apache-2.0): an MLX runtime with Metal kernels, continuous batching, prefix caching and the same OpenAI-compatible endpoint. It serves the same model files with no conversion step.
Requires Apple Silicon M1βM5, macOS 14+, Python 3.10β3.13 and 24 GB of unified memory. Measured resident footprint is 12.25 GB; single-stream decode is 27.3 tok/s on an M4 base and 59.7 tok/s on an M5 Pro, with aggregate throughput of 185.6 and 539.3 tok/s respectively at batch 128. Those are not comparable to the CUDA figures above: an M4 base moves ~120 GB/s against a 4090's 1008 GB/s, and decode here is memory-bound. Full tables, install steps and a head-to-head against stock MLX 4-bit are in that repo.
[^grid]: Same-harness measurement, 2026-07-26: one RTX 4090, the same model files, the same streaming OpenAI client, greedy, batch 1, medians of 2 reps over an ISLΓOSL grid (128β1890 in, 128β1890 out). Decode = steady-state tokens/s between the first and last streamed token. Per-cell decode: SGLang 176.8β230.9 (median 218.2), ZML 184.6β240.7 (median 234.8); ZML leads every cell with β₯256 output tokens and trails on 128-token replies, where its 16-token fused decode chunk dominates. TTFT by input length (ZML / SGLang): 128 tok 55/55 ms Β· 500 tok 95/91 ms Β· 1000 tok 178/143 ms Β· 1890 tok 353/237 ms.
[^ctx]: Measured 2026-07-26 on one RTX 4090 (24 GB) with Qwen3.6-35B-A3B-Escha-W2. Both
engines ship a conservative default context and let you raise it. SGLang reports its KV
pool at startup; launched with CTXLEN=262144 it allocated 159,480 tokens at its shipped
MEM=0.78 and 234,796 at MEM=0.90 β that pool size is the ceiling for a single
sequence. ZML serves a 261,966-token prompt at ESCHA_CTX=262144. Both are
memory-bound near the cap, not architecture-bound: KV on this hybrid model is only
~20 KiB per token (10 attention layers; the 30 gated-delta-net layers hold a fixed
~66 MB recurrent state), so 262,144 tokens is 5.37 GB beside 12.3 GB of weights.
In fairness to SGLang: its 234,796-token pool at MEM=0.90 still left 1.38 GB
of VRAM free, so it would very likely also reach the 262,144 cap at a higher
mem-fraction-static β we did not test that, and 0.78 is simply the conservative
default the runtime ships. Read this row as "both engines reach the model's native
context on a 24 GB card", not as a ZML advantage.
Compatible models
| Model repo | Bits |
|---|---|
| EschaLabs/Qwen3.6-35B-A3B-Escha-W2 | 2-bit (eschamoe) |
These runtimes target
qwen3_5_moe. A model of a different architecture will not load β use the matchingescha-runtime-<arch>repo.
Quickstart
SGLang engine (full detail: sglang/INSTALL.md, incl. the per-GPU cookbook):
python3.12 -m venv .venv && source .venv/bin/activate
pip install -U pip wheel
pip install "torch>=2.9" --index-url https://download.pytorch.org/whl/cu128 # cu12 torch first
pip install ./sglang/escha-*.whl # pulls the bundled sglang fork + its full dep closure
hf download EschaLabs/Qwen3.6-35B-A3B-Escha-W2 --local-dir ./Qwen3.6-35B-A3B-Escha-W2
MODEL=./Qwen3.6-35B-A3B-Escha-W2 bash sglang/serve.sh
Single-user speedup with MTP speculative decoding (1.0.4+, about 1.57Γ decode on a 4090): build the
draft once from Qwen's own release, then serve with MTP=1 β see
INSTALL.md β Speculative decoding.
python sglang/make_mtp_draft.py --out ./Qwen3.6-35B-A3B-MTP-draft
MODEL=./Qwen3.6-35B-A3B-Escha-W2 MTP=1 DRAFT=./Qwen3.6-35B-A3B-MTP-draft bash sglang/serve.sh
ZML engine (full detail: zml/INSTALL.md) β no Python at all:
tar xf zml/escha-zml-serve-*-linux-x86_64.tar.gz && cd escha-zml-serve-*
hf download EschaLabs/Qwen3.6-35B-A3B-Escha-W2 --local-dir ./model
./escha-serve
Long context and agentic use (ZML engine)
ESCHA_CTX reaches the model's native 262,144 tokens. Measured on one 4090 with facts
planted at 10%, 50% and 90% depth and all three asked for at the end β the check that
positions and recurrent state thread correctly across the whole prompt:
| prompt | prefill | rate | recall |
|---|---|---|---|
| 15,066 tok | 5.3 s | 2,858 tok/s | 3/3 |
| 60,066 tok | 24.9 s | 2,411 tok/s | 3/3 |
| 129,966 tok | 79.0 s | 1,646 tok/s | 3/3 |
| 261,966 tok | 259.1 s | 1,011 tok/s | 3/3 |
Prefill is ~NΒ²/2 work, so the rate falls with length. The comfortable band on a 24 GB card is ~8kβ130k, where TTFT is seconds; 262k answers correctly but takes ~4.3 minutes to read.
Multi-turn: a conversation whose prompt grows each turn reuses the previous prefix. On a 14k-token conversation the first turn takes 4.11 s and each following turn 1.27 s (~99.8% of the prompt reused). Reuse requires an exact append β editing earlier history re-reads the prompt β and only one conversation is cached. SGLang's radix cache is faster in absolute terms here and handles branching and multiple sessions; the ZML cache exists to make single-session agent loops practical, not to match it.
Decode slows as position grows on both engines, because decode attention is O(position). On ZML, measured with prefill excluded: 227 tok/s at position ~1k, 163 at 3k, 124 at 7.5k. (That measurement divides total request time and so reads ~5% below the streamed grid above β method, not configuration.)
Requirements (shared)
- NVIDIA GPU, compute capability 8.0β12.0 (Ampere β Blackwell), Linux x86-64.
Kernel launch route auto-selects per GPU. Per-architecture and per-VRAM launch recipes:
sglang/INSTALL.mdβ Running on your GPU. - SGLang engine: Python 3.12 + CUDA-12 PyTorch 2.9 or newer (the wheel handles every other
dependency). 24 GB VRAM recommended (16 GB works with reduced context).
As of 1.0.3 the wheel is built against PyTorch's stable ABI, so one binary serves torch
2.9 and every later release β earlier wheels were locked to a single torch minor. Verified
byte-identical output and equal decode speed (205.9 / 227.7 tok/s bs1 on a 4090, shipped
default / 24 GB config) on torch 2.9.1 and 2.11.0. On torch 2.10+ install
sglang-kernelfrom sglang's CUDA-12 index first; seesglang/INSTALL.mdβ Supported PyTorch versions. - ZML engine: no runtime dependencies β the archive bundles the CUDA runtime; just an NVIDIA driver (R550+) and glibc 2.27+ (RHEL/Rocky/Alma 8 and newer). 24 GB VRAM; one request at a time.
Changelog
1.0.6 (2026-10-31) β Single-user decode +5 %; all current escha optimizations in the runtime. At decode (up
to 8 tokens per step) each MoE layer now computes the router, the top-k selection and the expert tables in one kernel
and folds the shared expert into the expert sum (one launch instead of a GEMV, a sigmoid, a multiply and an add), with
the router and routed experts overlapped against the shared expert. The full-attention layers run q/k norm + rotary
embedding and the output gate as two fused kernels (bit-identical to the unfused ones). On an RTX 4090 (same session,
two interleaved runs each): single-user decode 224.5 β 236.0 tok/s (128-token prompt), 222.2 β 233.2 tok/s
(2k prompt) and 202.5 β 211.7 tok/s (20k prompt); 8 users, 128-token prompts β 2,048-token answers: 880 β 899 tok/s; prefill unchanged. Quality: MATH-500
(thinking on, n = 500) 88.0 % vs 88.2 % for 1.0.5 (paired p = 1.0). SGLANG_ESCHA_MOE_GLUE=0,
ESCHA_MOE_ROUTE_BUILD=0 and SGLANG_ESCHA_ATTN_FUSE=0 each restore the 1.0.5 path. With
ESCHA_MOE_GROUPED_PREFILL=0, prefill chunks of 8,192+ tokens now work too.
Backward compatible: a drop-in upgrade from 1.0.4 or 1.0.5. Same model files, same
serve.sh, same dependencies: any command line that worked with 1.0.4 launches the same server, andpip install --no-deps ./escha-1.0.6+qwen3moe-*.whlupgrades an existing environment in place.escha._Ckeeps every 1.0.4 operation with an unchanged signature (one operation added), and no Python API was removed. Checked on an RTX 4090 (single user, 8 prompts Γ 384 greedy tokens, 2 runs, deterministic MoE reductionESCHA_MOE_FUSE_EPI_FIX=1 ESCHA_ENABLE_DET_SCATTER=1): withESCHA_MOE_GROUPED_PREFILL=0 SGLANG_ESCHA_MOE_GLUE=0 SGLANG_ESCHA_ATTN_FUSE=0, 1.0.6 writes text byte-identical to 1.0.4 on every prompt, with and without speculative decoding (SPEC=1), at the same speed (225.8 vs 225.7 tok/s). With the defaults the shared expert enters the expert sum in a different order, so 4 of the 8 answers move away from 1.0.4's text after 190β1,100 characters; quality is unchanged (MATH-500 above). The default launch,THINK=0andMTP=1all serve.
Best measured configurations of 1.0.6 on an RTX 4090 (NVIDIA's ISL/OSL grid, real prompts, idle desktop; throughput = output tokens per second of wall-clock time, prefill included):
| configuration | decode, short context | decode at 20k context | time to first token (2k / 5k / 20k prompt) |
|---|---|---|---|
single user, RADIX=0 bash serve.sh |
236 tok/s | 212 tok/s | 0.19 / 0.39 / 1.69 s |
single user, MTP=1 DRAFT=β¦ bash serve.sh (speculative) |
357β395 tok/s (2β4k answers) | 250 tok/s | 0.19 / 0.40 / 1.78 s |
multi-user, RADIX=0 MAXREQ=32 MAXMAMBA=32 CUDA_GRAPH_BS="1 2 4 8 16 24 32" bash serve.sh |
8 users | 16 users | 32 users |
|---|---|---|---|
| 128-token prompts β 2,048-token answers: throughput (per user) | 891 tok/s (113) | 1,156 tok/s (73) | 1,138 tok/s (36) |
| 1,000 β 1,000 tokens: throughput / time to first token | 847 tok/s / 0.50 s | 1,084 tok/s / 0.79 s | 1,071 tok/s / 1.40 s |
| 2,048 β 128 tokens (prefill-heavy): throughput / time to first token | 438 tok/s / 0.89 s | 489 tok/s / 1.54 s | 485 tok/s / 2.79 s |
1.0.5 (2026-10-31) β Prefill up to 1.9Γ faster; the current escha kernels. The SGLang engine now runs the
routed experts of every prefill batch (more than 64 tokens) as a grouped tile GEMM: each expert's weights are decoded
once per 128 routed rows instead of once per 8, and routing happens on the GPU without a host sync. escha._C is
rebuilt from the current escha kernels (same operations, same PyTorch stable ABI, SASS for sm_80βsm_120).
Measured on an RTX 4090 (NVIDIA's ISL/OSL benchmark grid, real prompts, RADIX=0, everything else default):
| 1.0.4 | 1.0.5 | |
|---|---|---|
| prefill, 2,048-token prompt | 5,935 tok/s (TTFT 355 ms) | 10,920 tok/s (TTFT 189 ms) |
| prefill, 5,000-token prompt | 7,097 tok/s (TTFT 707 ms) | 12,854 tok/s (TTFT 390 ms) |
| prefill, 20,000-token prompt | 6,870 tok/s (TTFT 2.91 s) | 11,784 tok/s (TTFT 1.70 s) |
| 8 users, 2,048-token prompts, 128-token answers: TTFT / throughput* | 1.69 s / 305 tok/s | 0.91 s / 432 tok/s |
| single-user decode | 225 tok/s | 225 tok/s |
* Corrected 2026-10-31: throughput = output tokens per second of wall-clock time, prefill included. The first version of this row summed the users' decode rates (673 β 728 tok/s), which leaves out the prefill speed-up.
Quality: MATH-500 (thinking on, n = 500) 88.2 %, within sampling noise of the 90.0 % reference run (paired p = 0.20). Prompt log-probabilities differ from 1.0.4 no more than 1.0.4 differs from itself between runs, and 1.0.5's prefill is bit-for-bit repeatable.
Every existing setting is unchanged β any command line that worked with 1.0.4 launches the same server.
ESCHA_MOE_GROUPED_PREFILL=0restores the 1.0.4 prefill path. Decode runs the same arithmetic as 1.0.4. Also fixed: a prefill chunk of 8,192 or more tokens (CHUNK=8192) made 1.0.4's MoE prefill fail withmoe_linear: CUDA error; 1.0.5 handles any chunk size. The ZML engine is unchanged.
1.0.4 (2026-10-29) β MTP speculative decoding for the SGLang engine. New opt-in preset MTP=1
serves the published W2 model with its own MTP draft: sglang/make_mtp_draft.py rebuilds that draft
from Qwen's FP8 release (the W2 repo carries the block without its routed experts), the bundled SGLang
hands the int8-as-stored embedding and output head to the draft instead of failing on them, and the
draft scores only the 32,768 most frequent tokens (HOTMAP, sglang/hot_token_map_32k.pt). Single-user
decode on an RTX 4090 goes from 226.8 to 356.1 tok/s (1.57Γ); GSM8K is unchanged (96.8% both). See
INSTALL.md β Speculative decoding.
Every existing setting is unchanged.
MTPandHOTMAPare new and off by default; any command line that worked with 1.0.3 launches the same server. The compiledescha._Cis byte-identical to 1.0.3 and the Python changes are on the speculative path only: with the deterministic settings, greedy output is byte-identical to 1.0.3 atSPEC=0and atSPEC=1(16/16 prompts each) at the same speed. The ZML engine is unchanged.
Known limitations
DETERMINISTIC=1fails on consumer Blackwell (sm_120). The deterministic attention kernel requests 104 KB of shared memory per block, above the sm_120 limit, and the server exits during startup. It works on Ampere, Ada and Hopper. Greedy output is in any case not bit-reproducible across requests on either engine β batch composition changes fp16 accumulation order, so a near-tie can flip and a long reasoning chain diverges from there.- The ZML engine serves no
/v1/completions(HTTP 404). Use/v1/chat/completions. - The ZML engine does not validate requests. An unknown
modelfield is served anyway instead of returningmodel_not_found, andusage.prompt_tokensis wrong when a prompt is truncated. - The ZML engine can wedge on an over-long prompt on a 16 GB card, requiring
kill -9rather than returning an error. This is why it asks for 24 GB β use the SGLang engine on 16 GB, where the same prompt returns a cleanHTTP 400.
License
Everything here is released under the Apache License, Version 2.0 β see LICENSE.
All bundled third-party code is permissive (Apache-2.0 / MIT / BSD-3-Clause; NVIDIA runtime
libraries in the ZML bundle under the NVIDIA EULA's redistributable-runtime terms) β no
copyleft. Full texts and the component inventory:
THIRD_PARTY_LICENSES/. Model weights are not in this repo and carry
their own license in the model repository.