EschaLabs/escha-runtime-qwen3dense

Escha Runtime — qwen3dense

By Escha Labs Inc.

The serving runtime for Escha 2-/3-bit (escha) quantized models of the qwen3_5 dense architecture (Qwen3.8-27B and siblings). One repo per model architecture, one directory per engine — this architecture currently has one engine, sglang/.

	SGLang — sglang/
Best for	everything: single user, teams, agents
Concurrency	continuous batching, paged KV, optional radix prefix cache
Tool calls / JSON schema / thinking parser	yes
Interface	OpenAI-compatible (/v1/chat/completions, /v1/completions, /v1/models)
Install	Python 3.12 venv + CUDA-12 PyTorch, then one wheel

The engine is a fork of SGLang bundled inside the wheel, running the Escha CUDA kernels. No separate sglang install is needed, and none should be present — the wheel ships its own.

Compatible models

Model repo	Bits
EschaLabs/Qwen3.8-27B-Escha-W2	2-bit, mixed-rate (escha)

This runtime targets the qwen3_5 dense architecture. Its wheel also happens to register the eschamoe mixture-of-experts method, so a qwen3_5_moe model will load too — but the tuning, the defaults in sglang/serve.sh and the documentation here are all written for the dense architecture. For a mixture-of-experts model use escha-runtime-qwen3moe, whose defaults are measured on it. A model of a genuinely different architecture will not load — use the matching escha-runtime-<arch> repo.

Quickstart

Full detail, including the per-GPU cookbook and troubleshooting: sglang/INSTALL.md.

python3.12 -m venv .venv && source .venv/bin/activate
pip install -U pip wheel
pip install "torch>=2.9" --index-url https://download.pytorch.org/whl/cu128   # cu12 torch FIRST
pip install ./sglang/escha-*.whl   # pulls the bundled sglang fork + its full dep closure

hf download EschaLabs/Qwen3.8-27B-Escha-W2 --local-dir ./Qwen3.8-27B-Escha-W2
MODEL=./Qwen3.8-27B-Escha-W2 bash sglang/serve.sh

Then check the stack and the endpoint:

python -c "import torch, escha, sglang; print(torch.cuda.is_available(), hasattr(torch.ops.escha, 'escham_decode_gemv'), escha.__version__)"
curl -s http://127.0.0.1:30000/v1/models | python3 -m json.tool

As of 1.2.2, escha._C targets PyTorch's stable ABI, so this wheel loads on torch 2.9 and every later release — the old torch==2.9.* hard pin is gone (1.2.1 and earlier were ABI-locked to 2.9.x and would fail import escha with undefined symbol: _ZN3c10... on anything newer). On torch 2.10 and newer, install sglang-kernel from sglang's CUDA-12 index before the wheel (PyPI's build is CUDA-13/sm100-only and won't run on this hardware) — see sglang/INSTALL.md → Supported PyTorch versions for that step and the rest of the dependency closure. Staying on torch 2.9 needs no extra step.

Thinking, and why you probably want a budget

This is a reasoning model. With thinking on, the reasoning arrives in reasoning_content and the answer in content — read both, or you will see half the response.

Two per-request levers, both inside chat_template_kwargs (a top-level enable_thinking field is silently ignored):

{ "chat_template_kwargs": {"enable_thinking": true, "reasoning_effort": "xhigh"} }

reasoning_effort is "xhigh" (the default), "medium" or "low"; anything else makes the template raise, which surfaces as an HTTP 400 rather than a silent fallback. It works by injecting one sentence of system instruction — xhigh asks the model to validate assumptions and weigh alternatives, low asks it to keep thinking brief, and medium injects nothing at all, so medium is the neutral, unsteered model rather than a midpoint. It therefore asks for shorter reasoning; it does not bound it. If you are running a benchmark or an agent, set a thinking budget instead, which forces </think> after N reasoning tokens so an answer is always produced: see sglang/INSTALL.md → Bounded thinking and sglang/thinking_budget.py. Without one, the usual failure is finish_reason: "length" with content: null, which a harness scores as wrong rather than as slow.

Requirements

• NVIDIA GPU, compute capability 8.0–12.0 (Ampere → Blackwell), Linux x86-64 with glibc ≥ 2.28. The kernel launch route auto-selects per GPU; you never set it. Per-architecture and per-VRAM launch recipes: sglang/INSTALL.md → Running on your GPU.
• Python 3.12 (the wheel is cp312-only) + CUDA-12 PyTorch 2.9 or newer (1.2.2+; see Supported PyTorch versions for the torch 2.10+ extra step). The wheel handles every other dependency.
• A working host C compiler and Python dev headers. Triton JIT-compiles a small shim at CUDA-graph capture time — this is separate from ptxas and from a CUDA toolkit, so "driver only" does not cover it. On slim container images a stripped libisl breaks cc1 while gcc --version still succeeds, and the failure surfaces ~40 s in as a gcc CalledProcessError inside cuda_graph_runner.py — which reads like a runtime bug and is not. Preflight in sglang/INSTALL.md.
• 24 GB VRAM for the shipped defaults (65,536-token context, ~8–9 concurrent streams at short prompts) with a ~10.15 GB model. Those two figures are not simultaneous: the default pool is 68,686 tokens, which is one full-length 64k request or ~8 requests of ~8k. Note also the stream ceiling: MAMBA_RATIO=0.3 sizes the recurrent-state pool, which clamps max_running_requests to 8–9 on a 24 GB card, so the 12/16 entries in the default CUDA_GRAPH_BS are dropped and never captured. To serve more streams raise MAXREQ/MAXMAMBA with MEM — the throughput recipe is in the model card; for long context (128k measured on a 24 GB card) see By VRAM. 16 GB is now measured too, on an RTX 5060 Ti — ~30 tok/s single-stream, and far more context than we had guessed: 110,592 tokens, with an fp8 KV cache doing most of that work. See The 16 GB tier.

Changelog

1.2.5 (2026-10-31) — Fused attention glue; best-configuration numbers. The full-attention layers run q/k norm + rotary embedding and the output gate as two fused kernels, bit-identical to the unfused ones (prompt log-probabilities and greedy output exactly equal). Speed on an RTX 4090 is within noise of 1.2.4 (+0.3–0.5 % single- user decode). MATH-500 (thinking on, n = 500): 93.8 % (1.2.4: 93.2 %). SGLANG_ESCHA_ATTN_FUSE=0 restores 1.2.4.

Backward compatible: a drop-in upgrade from 1.2.3 or 1.2.4. Same model files, same serve.sh, same dependencies: any command line that worked with 1.2.3 launches the same server, and pip install --no-deps ./escha-1.2.5+qwen3dense-*.whl upgrades an existing environment in place. escha._C keeps every 1.2.3 operation with an unchanged signature (one operation added), and no Python API was removed. With the defaults, 1.2.5 writes text byte-identical to 1.2.3 (RTX 4090, single user, 8 prompts × 384 greedy tokens, 2 runs), with and without speculative decoding (SPEC=1), at the same speed (67.1 vs 66.8 tok/s; speculative 120.9 vs 120.2 tok/s). The default launch and THINK=0 serve.

Best measured configurations on an RTX 4090 (NVIDIA's ISL/OSL grid, real prompts, idle desktop; throughput = output tokens per second of wall-clock time, prefill included):

configuration	decode, short context	decode at 20k context	time to first token (2k / 5k / 20k prompt)
single user, bash serve.sh (defaults)	66.6 tok/s	60.8 tok/s	0.75 / 1.78 / 7.34 s
single user, MTP=1 bash serve.sh (speculative)	134–146 tok/s (2–4k answers)	120 tok/s	0.75 / 1.77 / 7.53 s

multi-user, MEM=0.85 MAXREQ=32 MAXMAMBA=32 CUDA_GRAPH_BS="1 2 4 8 12 16 24 32" bash serve.sh ¹	8 users	16 users	32 users
128-token prompts → 2,048-token answers: throughput (per user)	414 tok/s (52)	659 tok/s (42)	734 tok/s (23)
1,000 → 1,000 tokens: throughput / time to first token	361 tok/s / 2.0 s	537 tok/s / 3.5 s	587 tok/s / 6.2 s

¹ MEM=0.85 peaks at 21.9 GB of VRAM: a 24 GB card with no desktop on it. With the defaults the server runs at most 9 requests at once and the rest wait: 16 users then get 410 tok/s.

1.2.4 (2026-10-31) — Faster batched decode and prefill: the current escha kernels. escha._C is rebuilt from the current escha kernels (same operations, same PyTorch stable ABI, SASS for sm_80–sm_120). The multi-row decode kernel now covers batches of up to 32 requests (it stopped at 16; batches of 17–32 used to take the slower GEMM path), and the prefill kernels are faster. Measured on an RTX 4090 (NVIDIA's ISL/OSL benchmark grid, real prompts, RADIX=0, MAXREQ=24 for the 20-user rows, everything else default):

	1.2.3	1.2.4
20 users, 128-token prompts, 2,048-token answers: throughput*	344 tok/s	388 tok/s (+13 %)
20 users, 2,048-token prompts, 128-token answers: throughput*	113 tok/s	123 tok/s (+9 %)
prefill, 5,000-token prompt	2,621 tok/s (TTFT 1.91 s)	2,793 tok/s (TTFT 1.79 s)
prefill, 20,000-token prompt	2,535 tok/s (TTFT 7.90 s)	2,702 tok/s (TTFT 7.41 s)
single-user decode	65.5 tok/s	65.5 tok/s

* Corrected 2026-10-31: throughput = output tokens per second of wall-clock time, prefill included. The first version of these rows summed the users' decode rates (560 → 642 and 365 → 401 tok/s), which overstates throughput when requests wait for a slot (2 of the 20 did here).

Quality: MATH-500 (thinking on, n = 500) 93.2 %, against 91.2 % for the reference run (paired p = 0.06). Up to 16 concurrent requests the runtime calls the same operations as on 1.2.3, and on our op-by-op checks they return bit-identical results.

Every existing setting is unchanged — any command line that worked with 1.2.3 launches the same server, and the Python code is byte-identical to 1.2.3; only escha._C changed.

1.2.3 (2026-10-29) — Faster MTP: MTP=1 and a hot-vocabulary draft head. New opt-in preset MTP=1 = SPEC=1 with a draft that scores only the 32,768 most frequent tokens (HOTMAP, sglang/hot_token_map_32k.pt) instead of the full 248,320-token vocabulary, while the target still verifies every token over the full vocabulary. On an RTX 4090 that takes single-user MTP decode from 1.81× to 2.00× (120.4 → 133.3 tok/s against 66.6 without speculation); GSM8K is unchanged (97.2% both). The bundled SGLang can now row-select the int8-as-stored output head for that draft (1.2.2 needed the fp16 head for it). See INSTALL.md → Speculative decoding.

Every existing setting is unchanged. MTP and HOTMAP are new and off by default; any command line that worked with 1.2.2 launches the same server. The compiled escha._C is byte-identical to 1.2.2 and the Python changes are on the speculative path only: greedy output is byte-identical to 1.2.2 at SPEC=0 and at SPEC=1 (16/16 prompts each) at the same speed.

1.2.2 (2026-09-16) — One wheel for torch 2.9 and every later release. escha._C is now built against PyTorch's stable ABI — it talks only to the unmangled aoti_torch_* C shim, with the feature set targeted at 2.9 — so it carries zero libtorch C++ symbols and loads unchanged on newer torch. Previous wheels were ABI-locked to one torch minor: 1.2.1 cannot even be imported on torch 2.11. The shipped binary is the one the 35B MoE runtime (1.0.3) ships, built from the same kernel revision on 2026-09-17; the only change since the first 1.2.2 build is in the MoE scatter path, which a dense model never runs — dense output and speed are unchanged.

Verified on an RTX 4090 with this exact wheel: serving Qwen3.8-27B-Escha-W2 on torch 2.9.1 and torch 2.11.0 produces byte-identical greedy output at 66.7 and 65.2 tok/s. The binary ships SASS for sm_80/86/89/90/100/120.

No behaviour change. Same kernels, same numbers — escham_decode_gemv and friends are bit-identical to 1.2.1 on real released weights. This release is purely about which torch versions the runtime will load on.

Two dependency pins move with it, both forced by upstream rather than by escha: sgl-kernel was renamed to sglang-kernel (the old name's last release is a torch-2.9 binary), and sgl-fa4 <= 4.0.4 hard-pins torch==2.9.1, so the floor is now 4.0.5. On torch 2.10+ install sglang-kernel from sglang's CUDA-12 index — the PyPI build is CUDA-13/sm100-only and will not run on consumer cards. See INSTALL.md → Supported PyTorch versions. If you stay on torch 2.9, nothing changes for you.

1.2.1 (2026-09-02) — MTP speculative decoding. The model's own pretrained MTP module can now be attached as an EAGLE-family draft, for roughly 1.8–1.9× single-user decode. Enable with SPEC=1 and the mtp/ folder that ships in the model repo; see INSTALL.md → Speculative decoding.

Non-speculative serving is unchanged. Everything this release adds is on the speculative path; at SPEC=0 (the default) the engine runs exactly as 1.2.0 did.

The bundled SGLang carries two fixes that are not escha-specific and are not yet upstream: the intermediate mamba state reserve under speculation, and the topk factor in the multi-step draft kv-indices buffer. Both are on the speculative path only. They live in the bundled SGLang, so installing SGLang from PyPI over this wheel removes them.

1.2.0 (2026-08-21) — tensor parallelism (--tp-size N) now works. The escha parameter class pins its own weight loader, which meant sglang's TP slicing never ran and every rank kept the whole checkpoint (rank 0 died with weight must have shape (dim, width)). It now slices per rank, including the fused-on-disk GDN in_proj_qkv, which is split into its three sub-projections first.

Single-GPU users are unaffected. Every new code path is gated on world_size > 1; at --tp-size 1 the loader is byte-for-byte what 1.1.1 did. Verified as an identical shard layout and byte-identical greedy output.

TP > 1 is new and lightly tested — treat it as experimental. It was contributed and validated by @ginerJuanUdesa on 2× RTX 3090 (symmetric 6.02 GB/rank, coherent greedy output). We have one GPU and could not reproduce it, and no numerical equivalence check against --tp-size 1 has been run yet. If you use it for evaluation, sanity-check a benchmark against the single-GPU numbers first. Note that a multi-rank all-reduce reorders float accumulation, so TP > 1 output is not expected to match TP = 1 bit-for-bit even when correct.

On Ampere/Ada/Hopper you can add DETERMINISTIC=1 to remove that reduction-order variance if you want a stricter comparison.

1.1.1 (2026-08-21) — process_weights_after_loading now takes the rank's device instead of a hardcoded cuda:0. The hardcode put every 2-bit buffer on cuda:0 while the input tensor sat on the server's actual device, so any run not on device 0 — --tp-size > 1, or a single-GPU launch with --base-gpu-id N and no CUDA_VISIBLE_DEVICES — hit an illegal memory access on the first forward, behind a traceback that pointed at the kernel rather than at the cause. Bit-identical wherever cuda:0 was already correct, which is every configuration serve.sh ships. Reported with a diagnosis and a fix by @ginerJuanUdesa.

1.1.0 (2026-08-20) — first wheel with the dense (escha) serving path; the 1.0.x wheels registered eschamoe only, so a dense checkpoint failed at registry lookup.

Known limitations

• Serving throughput has been measured on four cards — RTX 5090 (32 GB, sm_120), RTX 4090 (24 GB, sm_89), RTX 3090 (24 GB, sm_86) and RTX 5060 Ti (16 GB, sm_120); all but the 4090 by independent evaluators working only from these docs. The 4090 is also where the model's thinking-on benchmarks (GPQA-Diamond, LiveCodeBench) were produced; its thinking-off commonsense suite ran on an L40. The 40 GB+ tier in the cookbook is still configuration guidance derived from the model size and the wheel's architecture coverage, not a measurement. Per-GPU numbers live on the model card.

• On Ampere the auto-selected kernel route is the slower one at batch 1. ESCHA_ROUTE resolves to lovelace on sm_80/sm_86, but forcing ESCHA_ROUTE=blackwell measured 1.72× faster single-stream on an RTX 3090 (23.6 → 40.7 tok/s, TPOT 42.4 → 24.6 ms) with identical output. The two routes are bit-identical launch geometries, so this is safe to set; the gain is batch-1-only (parity at 2–16). Serving one user on Ampere? Set it.

• DETERMINISTIC=1 fails on consumer Blackwell (sm_120). The deterministic attention kernel requests 104 KB of shared memory per block, above the sm_120 limit, and the server exits during startup. It works on Ampere, Ada and Hopper.

• Greedy output is not bit-reproducible across requests. Batch composition changes fp16 accumulation order, so a near-tie can flip and a long reasoning chain diverges from there. Two identical greedy requests may return different text. Use DETERMINISTIC=1 when you need reproducibility, and never A/B two configurations by diffing one generation.

• CUDA-graph batch sizes are capped at 32, the decode kernel's maximum M (torch.ops.escha.escham_decode_gemv_max_m()). The shipped default list stops at 16 because that is where aggregate throughput peaks on a 4090; capture at 24/32 works and is worth it if you serve that many streams. Past 32 a batch falls through to a large-M path meant for prefill, so the runtime refuses to capture it rather than bake in the wrong kernel.

• ATTN_BACKEND=triton is required on consumer Blackwell (RTX 50-series). The default flashinfer backend asserts on this hybrid architecture at sm_120. The assertion names three acceptable backends — triton, trtllm_mha, fa4 — of which only triton has been run on this model. Note that sm_120 shows steeper long-prompt decode decay than sm_89 (88.5% vs 96.3% of short-prompt rate at a 5,000-token prompt); the attention path is the obvious suspect and nobody has run the A/B that would confirm it.

• The model's stock chat template rejects the developer role. It accepts system, user, assistant and tool; anything else raises Unexpected message role. and surfaces as an HTTP 400 from the template, not as a server fault. Several OpenAI-compatible agent harnesses send role: "developer" for their system prompt and so fail every request — map it to system client-side, or patch the template. The same template also raises on genuinely malformed conversations (a missing user query, a system message in the wrong position) and on a reasoning_effort outside xhigh/medium/low. All of this is the base model's own template, carried over unmodified.

• One report of a GPU falling off the bus under sustained load, on a passed-through card. @JacksonAtkinsX hit NVRM: Xid ... 79, GPU has fallen off the bus four times out of four on an RTX 5060 Ti 16 GB, 12–18 minutes into a sustained single-stream agent workload, and each time the card needed a full host power cycle — FLR, secondary-bus reset and remove/rescan all reported success while the device stayed in D3cold. It reproduced across two driver branches, two runtime versions, both KV dtypes and three context lengths, in prefill and in decode alike. The card is passed through to a VM with vfio.

  We have not reproduced anything like it on the 3090, 4090, 5090 or L40 across the campaigns behind these docs, none of them on a passed-through GPU. Xid 79 is a bus-level fault: a misbehaving CUDA kernel produces an Xid 13/31/43 or an MMU fault and leaves the device resettable, whereas here no reset short of a power cycle worked, which is a platform signature rather than a userspace one. Two hypotheses that report could not close, and what we would check:

  • Run any other sustained 100% load for 30 minutes on the same box — gpu-burn, or a different model on a stock runtime. This is the one test that settles it: if that also falls off the bus, the runtime is not involved.
  • Power transients. nvidia-smi was sampled at 5 s, which cannot see the sub-millisecond spikes that trip a PSU's over-current protection, so "166 W under a 180 W cap with no throttle flags" does not rule power out. Log power.draw.instant and temperature.memory at 100 ms, and check PSU headroom on the rail feeding the card.
  • PCIe power management under vfio, which the D3cold-and-never-returns behaviour fits: vfio-pci disable_idle_d3=1, and pcie_aspm=off pcie_port_pm=off on the host command line.

  One thing we will say on our own side: this runtime's 2-bit decode path is arithmetic-heavy and holds a high, flat ALU duty cycle rather than the burstier profile of an fp16 matmul, so it is a harsher electrical stressor than most inference workloads. That is a reason to want power headroom before a long unattended run, not evidence of a defect.

License

Everything here is released under the Apache License, Version 2.0 — see LICENSE. All bundled third-party code is permissive (Apache-2.0 / MIT / BSD-3-Clause) — no copyleft. Full texts and the component inventory: THIRD_PARTY_LICENSES/. Model weights are not in this repo and carry their own license in the model repository.