qwen3-coder-30b-a3b on Tenstorrent p300x2 and p150x4 and p300 and p150x2 and p150
Qwen3-Coder-30B-A3B-Instruct (30B total / 3B active MoE: 128 experts, top-8, 48 layers) for agentic coding with tool calling on Blackhole through vLLM, with a fused MoE decode: the v1 bundle (raahemnabeel/qwen3-coder-30b-a3b-v1) plus the Ornith-1.5-35B-A3B fused MoE decode kernels in expert-parallel form for 1-32 concurrent users. Same weights, profiles and serving recipe; prefill unchanged.
Runs on Tenstorrent p300x2 or p150x4 or p300 or p150x2 or p150 β see the serving profiles below.
Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).
At a glance
| Hardware | p150, p150x2, p150x4, p300, p300x2 |
| Context | 256,000 tokens |
Prerequisites
- The Tenstorrent CLI,
tt:uv tool install tenstorrent(or use tt-model alone, see the Quickstart) - Docker
- Tenstorrent hardware: p300x2 or p150x4 or p300 or p150x2 or p150
Quickstart
tt serve raahemnabeel/qwen3-coder-30b-a3b
tt serve downloads the Docker image and the Qwen/Qwen3-Coder-30B-A3B-Instruct at b2cff646eb4bb1d68355c01b18ae02e7cf42d120 weights (into your HF cache; they are not in the image), then starts an OpenAI-compatible server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.
Without tt-cli: tt-model serve raahemnabeel/qwen3-coder-30b-a3b.
What changed since v1
This is the v1 bundle (raahemnabeel/qwen3-coder-30b-a3b-v1) with its MoE decode replaced by the fused kernels of the Ornith-1.5-35B-A3B fusion (raahemnabeel/ornith-1.5-35b-a3b), in expert-parallel form. Same checkpoint, same vLLM plugin build, same profiles and serving recipe; prefill (the expert-sorted MoE) is unchanged. Per decode layer and die, three ops replace the replicated router chain, the two dynamic sparse matmuls, SwiGLU, the routing weights and the expert sum (and the expert-sorted path at 4+ rows):
- Route. The router projection (FP32 accumulation) and the global top-8 + softmax run in one op; each die keeps the selected experts that fall in its own window (die d holds experts 32 d .. 32 d + 31 on four dies, 64 per die on two, all 128 on one).
- Gate|up + SwiGLU and down + weighted sum stream each selected local expert's weights in place from the bundle's existing bfp4 DRAM layout (no copy, no re-rounding), for every decode width from 1 to 32 rows: several users' experts form one union per die, so each weight column is read once for all rows. A die whose window holds no selected expert writes a zero partial; the existing all-reduce sums the dies.
Numerics: the same bfp4 expert weights and FP32 router logits as v1. The fused route op accumulates the logits in a different order and its expert matmuls accumulate in FP32, so a near-tied 8th expert can be picked differently and the last bits of an expert's output can differ. Output quality is checked below.
Use with your client
Point any OpenAI client at the address tt-model serve prints (port 20000 unless it was busy), with model id
Qwen/Qwen3-Coder-30B-A3B-Instruct. Tool calls use the qwen3_coder parser. The server's default generation config
is greedy (temperature 0); per-request sampling parameters apply.
Profiles
tt-model serve raahemnabeel/qwen3-coder-30b-a3b --profile <name>:
p300(default) /p150x2- two dies (64 experts per die), 256000-token context.p300x2/p150x4- four dies (32 experts per die), 256000-token context: the fastest profile.p150- one die (all 128 experts), 98304-token context (the bf16 KV cache is 96 KiB per token on one die).
All profiles: 32 concurrent sequences, prefix caching, bfp4 routed experts.
Serving profiles
Select a different serving profile using --profile.
| profile | hardware | mesh | max_num_seqs | max_model_len | description |
|---|---|---|---|---|---|
p300x2 |
p300x2 | P300x2 | 32 | 256000 | Two p300 boards (4 dies, experts 32 per die), 32 sequences, 256000 context. |
p150x4 |
p150x4 | P150x4 | 32 | 256000 | Four p150 boards (4 dies), same configuration as p300x2. |
p300 (default) |
p300 | P300 | 32 | 256000 | One p300 board (2 dies, experts 64 per die, Linear collectives), 32 sequences, 256000 context. |
p150x2 |
p150x2 | P150x2 | 32 | 256000 | Two p150 boards (2 dies), same configuration as p300. Not run on hardware (the identical (1,2) mesh). |
p150 |
p150 | P150 | 32 | 98304 | One p150 (1 die, all 128 experts), 32 sequences sharing a 98304-token context (bf16 KV is 96 KiB per token on one die), no fabric. |
Capabilities
The server speaks the OpenAI API at http://127.0.0.1:20000/v1 (or whichever port your serve command reported; chat completions, completions, and /v1/models). Pass "model": "Qwen/Qwen3-Coder-30B-A3B-Instruct" β the weights id, not this package's name.
Tool calling is enabled (--tool-call-parser qwen3_coder): a request that passes tools comes back with tool_calls and finish_reason: tool_calls.
Expected performance
Accuracy
Both bundles evaluated side by side on the same box, p300x2, through the served chat API with lm-evaluation-harness 0.4.9 at temperature 0 (full sets, 8 requests in flight, seed 42):
| Eval | v1 | current |
|---|---|---|
| HumanEval pass@1 (164) | 0.939 | 0.933 |
| MBPP pass@1 (500) | 0.802 | 0.786 |
| GSM8K exact match, strict (1319) | 0.919 | 0.914 |
| IFEval prompt-level strict (541) | 0.795 | 0.813 |
| IFEval prompt-level loose (541) | 0.839 | 0.861 |
Every difference is within one standard error of the difference (HumanEval 1 problem, MBPP 8, GSM8K 7; IFEval goes the other way). The harness's stock chat variants do not work through a chat API (humaneval_instruct and mbpp_instruct
pre-fill the assistant turn, which the API does not continue), so HumanEval uses a chat prompt that asks for the whole
function in one code block, and MBPP answers are scored from their first code block; gsm8k_cot_llama is the stock
8-shot task with the dataset's current id. The task files are in fusion/tools/lm_tasks on the source branch. Greedy
outputs on every profile were checked against the v1 bundle's: they stay coherent and part after 17 to 1074
characters (the fused MoE decode rounds differently).
Performance
Measured 2026-10-24 on one QuietBox 2 (4 Blackhole p300c dies; p300 rows on two of them, p150 on one), both bundles
served exactly as shipped (tt-model serve <bundle> --profile <p>), one server at a time: random-token prompts of
exactly the input length, output pinned with ignore_eos, streamed; users are concurrent requests in a closed loop;
TTFT = time to first streamed token; TPOT = (E2EL - TTFT) / (output - 1); one unrecorded warm-up per row. A real-text
single-user row (12 chat prompts, 384 output tokens, temperature 0) is added. Each cell is v1 -> current.
| Profile | ISL | OSL | Concurrency | N | TTFT (ms) | Prefill tok/s/u | Decode tok/s/u | E2EL (s) |
|---|---|---|---|---|---|---|---|---|
| p300x2 | 128 | 128 | 1 | 4 | 145 | 881 | 95.5 | 1.47 |
| p300x2 | 1024 | 256 | 32 | 128 | 3,518 | 291 | 42.0 | 9.60 |
| p300 | 128 | 128 | 1 | 4 | 141 | 908 | 86.0 | 1.62 |
| p300 | 1024 | 256 | 32 | 128 | 4,301 | 238 | 30.4 | 12.76 |
| p150 | 128 | 128 | 1 | 4 | 161 | 794 | 77.4 | 1.80 |
p300x2
Latency, batch 1 (single user): v1 -> current
| Input tokens | Output tokens | TTFT (ms) | Prefill tok/s | TPOT (ms) | Decode tok/s | E2EL (s) |
|---|---|---|---|---|---|---|
| 128 | 128 | 202 -> 145 | 634 -> 881 | 20.2 -> 10.5 | 49.5 -> 95.5 (+93 %) | 2.77 -> 1.47 |
| 1,024 | 256 | 431 -> 376 | 2,377 -> 2,722 | 20.5 -> 10.8 | 48.7 -> 92.9 (+91 %) | 5.67 -> 3.12 |
| 4,096 | 256 | 904 -> 848 | 4,530 -> 4,830 | 21.2 -> 11.4 | 47.2 -> 87.8 (+86 %) | 6.31 -> 3.75 |
| 16,384 | 256 | 1,810 -> 1,745 | 9,053 -> 9,387 | 22.1 -> 12.3 | 45.3 -> 81.5 (+80 %) | 7.44 -> 4.87 |
| 32,768 | 256 | 2,481 -> 2,418 | 13,205 -> 13,551 | 23.2 -> 13.4 | 43.1 -> 74.9 (+74 %) | 8.40 -> 5.82 |
| 65,536 | 256 | 5,338 -> 5,261 | 12,277 -> 12,457 | 25.4 -> 15.6 | 39.4 -> 64.2 (+63 %) | 11.81 -> 9.23 |
| 130,816 | 256 | 12,699 -> 12,637 | 10,301 -> 10,352 | 29.5 -> 19.6 | 33.9 -> 50.9 (+50 %) | 20.21 -> 17.64 |
Real text, single user (12 chat prompts x 384 output tokens, temperature 0; median per prompt): 49.5 -> 95.5 tok/s (+93 %).
Concurrency, 128 input / 128 output tokens: v1 -> current
| Users | N | TTFT (ms) | TPOT (ms) | Decode tok/s/user | Total out tok/s | E2EL (s) |
|---|---|---|---|---|---|---|
| 2 | 8 | 201 -> 165 | 27.7 -> 12.4 | 36.1 -> 80.4 | 68.7 -> 146.9 (+114 %) | 3.73 -> 1.74 |
| 4 | 16 | 398 -> 329 | 33.0 -> 12.7 | 30.3 -> 78.6 | 107.1 -> 249.1 (+133 %) | 4.59 -> 1.96 |
| 8 | 32 | 614 -> 557 | 33.6 -> 14.0 | 29.8 -> 71.3 | 210.0 -> 444.5 (+112 %) | 4.88 -> 2.31 |
| 16 | 64 | 1,055 -> 979 | 35.0 -> 16.2 | 28.6 -> 61.7 | 359.1 -> 646.1 (+80 %) | 5.53 -> 3.07 |
| 32 | 128 | 1,919 -> 1,853 | 36.8 -> 24.0 | 27.2 -> 41.7 | 598.2 -> 812.7 (+36 %) | 6.62 -> 4.91 |
Concurrency, 1024 input / 256 output tokens: v1 -> current
| Users | N | TTFT (ms) | TPOT (ms) | Decode tok/s/user | Total out tok/s | E2EL (s) |
|---|---|---|---|---|---|---|
| 2 | 8 | 338 -> 336 | 27.8 -> 12.4 | 36.0 -> 80.5 | 68.6 -> 145.8 (+112 %) | 7.50 -> 3.54 |
| 4 | 16 | 532 -> 500 | 33.4 -> 13.0 | 29.9 -> 77.2 | 112.3 -> 266.3 (+137 %) | 9.13 -> 3.86 |
| 8 | 32 | 962 -> 931 | 34.2 -> 14.3 | 29.2 -> 69.9 | 210.0 -> 444.8 (+112 %) | 9.76 -> 4.62 |
| 16 | 64 | 1,834 -> 1,793 | 35.7 -> 16.7 | 28.0 -> 59.7 | 372.0 -> 674.4 (+81 %) | 11.00 -> 6.09 |
| 32 | 128 | 3,546 -> 3,518 | 37.8 -> 23.8 | 26.5 -> 42.0 | 616.7 -> 853.4 (+38 %) | 13.25 -> 9.60 |
p300
Latency, batch 1 (single user): v1 -> current
| Input tokens | Output tokens | TTFT (ms) | Prefill tok/s | TPOT (ms) | Decode tok/s | E2EL (s) |
|---|---|---|---|---|---|---|
| 128 | 128 | 188 -> 141 | 679 -> 908 | 26.3 -> 11.6 | 38.0 -> 86.0 (+126 %) | 3.53 -> 1.62 |
| 1,024 | 256 | 432 -> 381 | 2,372 -> 2,690 | 26.7 -> 12.0 | 37.4 -> 83.2 (+122 %) | 7.24 -> 3.45 |
| 4,096 | 256 | 915 -> 871 | 4,475 -> 4,704 | 27.7 -> 13.0 | 36.1 -> 76.7 (+113 %) | 7.99 -> 4.19 |
| 16,384 | 256 | 2,115 -> 2,059 | 7,745 -> 7,956 | 29.2 -> 14.6 | 34.2 -> 68.6 (+101 %) | 9.57 -> 5.78 |
| 32,768 | 256 | 3,125 -> 3,075 | 10,485 -> 10,656 | 31.3 -> 16.6 | 32.0 -> 60.1 (+88 %) | 11.10 -> 7.32 |
| 65,536 | 256 | 7,800 -> 7,748 | 8,402 -> 8,458 | 35.2 -> 20.5 | 28.4 -> 48.7 (+71 %) | 16.77 -> 12.98 |
| 130,816 | 256 | 20,673 -> 20,648 | 6,328 -> 6,336 | 42.7 -> 28.1 | 23.4 -> 35.6 (+52 %) | 31.56 -> 27.80 |
Real text, single user (12 chat prompts x 384 output tokens, temperature 0; median per prompt): 38.0 -> 86.3 tok/s (+127 %).
Concurrency, 128 input / 128 output tokens: v1 -> current
| Users | N | TTFT (ms) | TPOT (ms) | Decode tok/s/user | Total out tok/s | E2EL (s) |
|---|---|---|---|---|---|---|
| 2 | 8 | 212 -> 185 | 39.0 -> 14.0 | 25.6 -> 71.6 | 49.6 -> 130.5 (+163 %) | 5.16 -> 1.96 |
| 4 | 16 | 409 -> 351 | 41.4 -> 15.5 | 24.2 -> 64.5 | 88.7 -> 219.9 (+148 %) | 5.67 -> 2.33 |
| 8 | 32 | 696 -> 634 | 43.6 -> 17.4 | 22.9 -> 57.4 | 163.9 -> 356.6 (+117 %) | 6.22 -> 2.82 |
| 16 | 64 | 1,266 -> 1,203 | 47.5 -> 21.6 | 21.0 -> 46.3 | 276.5 -> 517.1 (+87 %) | 7.38 -> 3.96 |
| 32 | 128 | 2,389 -> 2,332 | 51.1 -> 32.8 | 19.6 -> 30.5 | 455.6 -> 626.7 (+38 %) | 8.88 -> 6.57 |
Concurrency, 1024 input / 256 output tokens: v1 -> current
| Users | N | TTFT (ms) | TPOT (ms) | Decode tok/s/user | Total out tok/s | E2EL (s) |
|---|---|---|---|---|---|---|
| 2 | 8 | 415 -> 410 | 39.2 -> 14.4 | 25.5 -> 69.6 | 49.2 -> 126.8 (+158 %) | 10.46 -> 4.08 |
| 4 | 16 | 613 -> 572 | 41.5 -> 15.2 | 24.1 -> 65.9 | 91.2 -> 225.3 (+147 %) | 11.23 -> 4.62 |
| 8 | 32 | 1,150 -> 1,118 | 43.5 -> 17.8 | 23.0 -> 56.3 | 167.2 -> 363.6 (+117 %) | 12.27 -> 5.63 |
| 16 | 64 | 2,216 -> 2,182 | 47.3 -> 21.8 | 21.1 -> 45.9 | 287.2 -> 527.5 (+84 %) | 14.28 -> 7.76 |
| 32 | 128 | 4,342 -> 4,301 | 49.8 -> 32.9 | 20.1 -> 30.4 | 479.1 -> 645.3 (+35 %) | 17.15 -> 12.76 |
p150
Latency, batch 1 (single user): v1 -> current
| Input tokens | Output tokens | TTFT (ms) | Prefill tok/s | TPOT (ms) | Decode tok/s | E2EL (s) |
|---|---|---|---|---|---|---|
| 128 | 128 | 193 -> 161 | 663 -> 794 | 36.6 -> 12.9 | 27.3 -> 77.4 (+183 %) | 4.84 -> 1.80 |
| 1,024 | 256 | 473 -> 440 | 2,166 -> 2,327 | 37.3 -> 13.5 | 26.8 -> 73.8 (+175 %) | 9.98 -> 3.89 |
Real text, single user (12 chat prompts x 384 output tokens, temperature 0; median per prompt): 27.2 -> 76.9 tok/s (+183 %).
Limitations
- p150x4 and p150x2 run the same meshes as p300x2 / p300 but were not run on p150 boards; the p150 profile ran on one p300c die (single-user rows only).
- Every newly admitted request re-captures the decode traces (as in v1), which shows up as TTFT under concurrent load.
- The evals ran on the p300x2 profile only. Agentic benchmarks (SWE-Bench Verified, LiveCodeBench) were not run.
Related packages
The v1 bundle this builds on: raahemnabeel/qwen3-coder-30b-a3b-v1. The fusion it ports: raahemnabeel/ornith-1.5-35b-a3b; engineering log models/autoports/ornith_ai_ornith_1_5_35b_a3b/fusion/HANDOFF.md on the c30_fusion branch of github.com/rnabeelTT/tt-metal.
Feedback
Questions or problems with this package: open a discussion at https://huggingface.co/raahemnabeel/qwen3-coder-30b-a3b/discussions β that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.
Provenance
The exact sources the image was built from β code/ in this repo is byte-identical to the model code inside the image:
| component | built from |
|---|---|
| tt-metal | f13b25823a8f3d5b88a8f329367e0558b8cdb68c |
| vLLM | v0.26.0 |
| vllm-tt-plugin | a local checkout β commit not published |
code/ digest |
a5b36e36e19de0f8 (sha256, first 16 hex digits) |
| built | 2026-10-27T20:20:31+00:00 by tt-model 0.1.0 |
Model tree for raahemnabeel/qwen3-coder-30b-a3b
Base model
Qwen/Qwen3-Coder-30B-A3B-Instruct