onw 2.0.0: Qwen3.6 MoE prompts ~4-6x (grouped experts), dense prompts ~2-2.5x (64-token blocks), faster MoE decode

CHANGELOG.md

@@ -5,6 +5,22 @@
 日付は日本時間です。各版は Hugging Face の `ryugyosoft/onw` にタグ（`v0.4.2` 以降）付きで残っています。
 onw ウィンドウは、新しい版があるとこのファイルから変更点を表示します（`## v…` の見出しで区切っています）。
 
+## v2.0.0 (2026-10-17)
+- **Qwen3.6-35B-A3B と Qwen3.6-REAP-18B-A3B のプロンプト処理が約 4〜6 倍に**（メモリ 24 GB 以上の PC）：AMD 版 onwx と同じ工夫を Intel NPU に移しました。プロンプト全体を層ごとに処理し、同じエキスパートを選んだトークンをまとめて 1 回で計算します（今までは 16 トークンのブロックごとに、選ばれたエキスパートの重みを毎回読んでいました）。
+  - Panther Lake（Qwen3.6-35B-A3B、1024 トークン）：約 24 → 約 147 tok/s（42 秒 → 7 秒）。
+  - Core Ultra 9 285HX（NPU 3720）：Qwen3.6-35B-A3B 約 22 → 約 100 tok/s、REAP-18B 約 24 → 約 106 tok/s。
+  - 出力はほぼ同じです（16 トークンのブロックとの差は KL 0.0003〜0.0005、最初のトークンは一致）。
+  - 使用メモリはほぼ変わりません（システム全体のピーク：Panther Lake で +0.1〜0.3 GB、NPU 3720 では 16 トークンのグラフを読み込まなくなるぶん同じか少なめ）。
+  - **モデルの更新が必要です**（カードに「更新あり」と出ます。差し替えは区間ファイルの約 0.9 GB だけです）。更新すると初回のロードで新しいグラフをコンパイルし直します。古い onw でも更新後のモデルはそのまま動きます。
+  - メモリ 24 GB 未満の PC（16 GB の Lunar Lake など）では、新しいグラフのコンパイル中にメモリが足りなくなるため、今までどおりの処理のままです（`ONW_GROUPED=1` で強制、`0` で無効）。
+- **Qwen3.6 系の生成が少し速く**：1 トークンの処理で、区間どうしの入出力を最初に一度だけつないでおき、毎回の受け渡しをなくしました。Panther Lake で 8.9 → 10.3 tok/s、NPU 3720 で 6.0 → 6.5 tok/s（出力は同じ）。
+- **dense モデルのプロンプト処理が約 2〜2.5 倍に**：Qwen3.5-9B、Ornith-1.5-9B、NeoHorse-1-4B、Granite 4.2 3B / 8B、llm-jp-4.1-8B-thinking（INT4 / INT8）に 64 トークンのブロックを加え、プロンプトを 16 トークンずつではなく 64 トークンずつ処理します。Gemma 4 E4B と NF4 版は、今まで 64 トークンのブロックを使えなかった PC（NPU 3720 のメモリ 24 GB 未満、NPUW が使えない設定など）でも使うようになります。
+  - Panther Lake（Qwen3.5-9B INT4、1024 トークン）：約 44 → 約 105 tok/s。
+  - Core Ultra 9 285HX（NPU 3720）：Qwen3.5-9B 約 31〜43 → 約 80〜87 tok/s、llm-jp-4.1-8B 約 46 → 約 111、NeoHorse-1-4B 約 67 → 約 132、Granite 4.2 3B 約 69 → 約 139 tok/s。
+  - 出力はほぼ同じです（16 トークンのブロックとの差は KL 0.00005〜0.0015、最初のトークンは一致）。
+  - 使用メモリはほぼ変わりません（+0.35〜0.5 GB）。16 トークン用のグラフの代わりに 64 トークン用を読み込むので、重みの写しは増えません。そのぶん、短い入力や先読み検証は少し割高になります（NPU 3720 では先読み検証が自動でオフになります）。設定の「64 トークン処理」を off にすると今までどおりです。
+  - **モデルの更新が必要です**（カードに「更新あり」と出ます。追加のグラフファイル数 MB だけで、重みのファイルは変わりません）。更新後の初回のロードで新しいグラフをコンパイルします（Panther Lake の 9B で数十分かかります）。古い onw は追加分を読まず、更新後のモデルもそのまま動きます。
+
 ## v1.4.13 (2026-10-16)
 - **Ternary Bonsai 2 9B / 4B（2 ビット、画像も読める、Lunar Lake 以降専用）をモデル一覧に追加**：Prism ML が Qwen3.5-9B / 4B を三値（−1・0・+1）の重みで学習し直したモデルを、重みを 2 ビットのまま NPU で動かします。Lunar Lake で 9B は生成約 13 tok/s・メモリ約 3.5 GB、4B は約 21 tok/s・約 2 GB です。27B と同じく、NPU コンパイラーの制限でプロンプトは 1 トークンずつ処理します。
 - `onw convert` が Bonsai 2 9B（回転ブロック 4096）・4B（回転ブロック 512、埋め込みと共有の LM ヘッド）と、その画像エンコーダーを読めるようになりました。Bonsai 2 27B の変換結果は変わりません（コンパイル済みのキャッシュもそのまま使えます）。

CHANGELOG_en.md

@@ -4,6 +4,22 @@
 
 Dates are JST. Every version is on Hugging Face `ryugyosoft/onw`, tagged from `v0.4.2` on.
 
+## v2.0.0 (2026-10-17)
+- **Prompts of Qwen3.6-35B-A3B and Qwen3.6-REAP-18B-A3B about 4-6x faster** (PCs with 24 GB of RAM or more): the trick of onwx, the AMD sibling, on the Intel NPU. The whole prompt runs layer by layer, and the tokens that picked the same expert are computed together in one go (before, every 16-token block read its chosen experts' weights again).
+  - Panther Lake (Qwen3.6-35B-A3B, 1,024 tokens): ~24 -> ~147 tok/s (42 s -> 7 s).
+  - Core Ultra 9 285HX (NPU 3720): Qwen3.6-35B-A3B ~22 -> ~100 tok/s, REAP-18B ~24 -> ~106 tok/s.
+  - Practically the same output (KL 0.0003-0.0005 against the 16-token blocks, the first token equal).
+  - Practically no extra memory (system-wide peak: +0.1-0.3 GB on Panther Lake; on the NPU 3720 the same or less, as the 16-token graphs are no longer loaded).
+  - **The models need an update** (their card shows "update available"; only the segment files, ~0.9 GB, are replaced). The first load after it compiles the new graphs. Older onw versions keep working with the updated models.
+  - On PCs below 24 GB (a 16 GB Lunar Lake, for example) compiling the new graphs runs out of memory, so they keep the old prompt path (`ONW_GROUPED=1` forces it, `0` turns it off).
+- **Qwen3.6 models generate a little faster**: for single tokens the segments' inputs and outputs are wired to each other once, with no hand-over at every token. Panther Lake 8.9 -> 10.3 tok/s, NPU 3720 6.0 -> 6.5 tok/s (same output).
+- **Prompts of the dense models about 2-2.5x faster**: Qwen3.5-9B, Ornith-1.5-9B, NeoHorse-1-4B, Granite 4.2 3B / 8B and llm-jp-4.1-8B-thinking (INT4 / INT8) get 64-token blocks, so prompts run 64 tokens at a time instead of 16. Gemma 4 E4B and the NF4 models now use their 64-token blocks on PCs that could not before too (NPU 3720 with less than 24 GB, NPUW off, ...).
+  - Panther Lake (Qwen3.5-9B INT4, 1,024 tokens): ~44 -> ~105 tok/s.
+  - Core Ultra 9 285HX (NPU 3720): Qwen3.5-9B ~31-43 -> ~80-87 tok/s, llm-jp-4.1-8B ~46 -> ~111, NeoHorse-1-4B ~67 -> ~132, Granite 4.2 3B ~69 -> ~139 tok/s.
+  - Practically the same output (KL 0.00005-0.0015 against the 16-token blocks, the first token equal).
+  - Practically no extra memory (+0.35-0.5 GB): the 64-token graphs are loaded in place of the 16-token ones, so there is no extra copy of the weights. Short inputs and draft checks cost a little more on them (on the NPU 3720 prompt lookup decoding turns itself off). The setting "64-token blocks" = off keeps the old way.
+  - **The models need an update** (their card shows "update available"; just the added graph files, a few MB - the weights files stay the same). The first load after it compiles the new graphs (tens of minutes for a 9B on Panther Lake). Older onw versions skip the additions and keep working with the updated models.
+
 ## v1.4.13 (2026-10-16)
 - **Ternary Bonsai 2 9B / 4B (2-bit, reads images, Lunar Lake and newer only) in the model list**: Prism ML's Qwen3.5-9B / 4B retrained with ternary weights (-1, 0, +1), run on the NPU with the weights kept 2-bit. On Lunar Lake the 9B generates ~13 tok/s in ~3.5 GB, the 4B ~21 tok/s in ~2 GB. As with the 27B, the NPU compilers' limit means prompts are processed token by token.
 - `onw convert` reads Bonsai 2 9B (rotation blocks of 4096), 4B (blocks of 512, an LM head tied to the embedding) and their image encoder. Bonsai 2 27B converts exactly as before (compiled caches stay valid).

MANUAL.md

@@ -167,7 +167,7 @@
 | メモリ節約モード | 1 トークン用と 16 トークン用のグラフを入れ替えて、片方だけを持ちます。待機中のメモリが約 4 割減る代わりに、回答の出だし数秒は遅めで、先読み検証は使いません。初回のコンパイルも並列にせず 1 つずつ行います（コンパイル中のメモリを抑えるため）。効くモデルがあるときだけ表示され、この PC で効くモデルの名前が出ます |
 | 厳格 | メモリ節約モードの追加。2 種類のグラフを同時に持たないので、メモリは最小になりますが、回答のたびに書き始めるまで数秒待ちます |
 | エキスパートのメモリ | Qwen3.6 系の MoE 用。上限を超えるエキスパートは SSD に置いたまま、必要なときだけ読みます（メモリが少なくても動きますが、遅くなります） |
-| 64 トークン処理 | 長いプロンプトの処理が約 2 倍速くなります（Gemma、Qwen3.5 系の NF4 版）。Lunar Lake 以降は重みを共有するのでメモリはほぼ増えません。NPU 3720 ではそのぶんメモリを使います。auto は Lunar Lake 以降か、メモリ 24 GB 以上でオン |
+| 64 トークン処理 | 長いプロンプトの処理が約 2〜2.5 倍速くなります（dense モデル。Qwen3.6 系は別の方法で速くしています）。16 トークン用のグラフの代わりに読み込むので、メモリはほぼ増えません（+0.3〜0.5 GB）。auto はオン。off にすると今までどおり 16 トークンずつ（短い入力の多い使い方や、NPU 3720 で先読み検証を使いたいとき） |
 
 ### 詳細
 

MANUAL_en.md

@@ -178,7 +178,7 @@ applies them ("Discard" throws them away). For settings that need a server resta
 | Memory-saving mode | Swaps the 1-token and 16-token graphs so only one set is held: about 40% less memory when idle, but the first seconds of an answer are slower and prompt lookup is not used. The first compile also runs one graph at a time instead of in parallel (less memory while compiling). Shown only when it applies, with the models it works for on this PC |
 | Strict | An addition to the memory-saving mode: never holds both sets of graphs, for the least memory, but every answer waits a few seconds before it starts |
 | Expert memory | For the Qwen3.6 MoE models: experts beyond the limit stay on the SSD and are read when needed (works with less memory, but slower) |
-| 64-token blocks | Long prompts are processed ~2x faster (Gemma, the NF4 Qwen3.5-family builds). From Lunar Lake the weights are shared, so memory hardly grows; on NPU 3720 it costs that much memory. auto turns it on from Lunar Lake or with 24 GB+ RAM |
+| 64-token blocks | Long prompts are processed ~2-2.5x faster (dense models; the Qwen3.6 models are sped up another way). Loaded in place of the 16-token graphs, so memory hardly grows (+0.3-0.5 GB). auto = on. off = 16 tokens at a time as before (for mostly short inputs, or to keep prompt lookup decoding on the NPU 3720) |
 
 ### Advanced
 

README.md

@@ -129,7 +129,11 @@ NPU は数トークンを 1 トークンとほぼ同じ時間で計算できる
 （コンテキスト長、64 トークン処理、エキスパートのメモリ）を反映した目安です。メモリが足りない PC では、Qwen3.6 系の
 エキスパートを SSD に置いたまま動かせます（設定の「エキスパートのメモリ」。仕組みは[技術解説](TECHNICAL.md)）。
 
-先読み検証は、16 トークンの検証ブロックが十分安く済むモデル（Qwen3.5-9B、Gemma、LFM 系）で自動的に使われます。
+プロンプト（入力した文章）の読み込みは、dense モデルは 64 トークンずつ、Qwen3.6 系はプロンプト全体を層ごとにまとめて
+処理します（onw 2.0）。NPU 3720 で Qwen3.5-9B 約 80〜87 tok/s、llm-jp-4.1-8B 約 110、Granite 4.2 3B 約 140、Qwen3.6-35B-A3B
+約 100 tok/s（メモリ 24 GB 以上）、Panther Lake で Qwen3.5-9B 約 105、Qwen3.6-35B-A3B 約 147 tok/s です。
+
+先読み検証は、検証ブロックが十分安く済むモデル（LFM 系、Lunar Lake 以降の Qwen3.5-9B・Gemma など）で自動的に使われます。
 「思考」の列の「切替」のモデル（Qwen、LFM2.5、Gemma 4、Granite 4.2 など）は、既定では考えずに答えます（チャット画面の「思考モード」、API の
 `chat_template_kwargs: {"enable_thinking": true}` または `reasoning_effort` で考えさせられます）。「常に」のモデル（gpt-oss、llm-jp）は思考モードがオフでも裏で考え、その分も最大トークンに数えます（`max_tokens` は大きめに）。
 モデルが考えた思考は、回答とは別に、API では `reasoning_content`（トークン数は `usage.completion_tokens_details.reasoning_tokens`）、チャット画面では折りたためる
@@ -190,7 +194,7 @@ onw list                                                      # 対応アーキ
 | temperature、top_p、top_k、繰り返し抑制（repetition / presence / frequency penalty）、max_tokens、stop、seed、思考モード | リクエストごと（API またはチャット画面） | 不要 |
 | ポート、待ち受けアドレス、モデル、デバイス、先読み検証のオン／オフ | サーバーのオプション／設定画面、再起動（約 20 秒） | 不要 |
 | **コンテキスト長**（`--context`、既定 32K、モデルの最大まで） | サーバーのオプション／設定画面 | 不要（KV キャッシュの上限が変わるだけ） |
-| 64 トークン処理（Gemma、Qwen3.5 系 NF4 版。auto は Lunar Lake 以降か RAM 24 GB 以上でオン） | `ONW_S64=1/0`／設定画面 | 初回のみ |
+| 64 トークン処理（dense モデルのプロンプトを 64 トークンずつ。auto はオンで、16 トークン用のグラフの代わりに読み込む。off で 16 トークンずつ、`ONW_S64=1` で両方） | `ONW_S64=1/0`／設定画面 | 初回のみ |
 | MTP 先読み（Qwen3.5 系 NF4 版。auto は Lunar Lake 以降でオン） | `ONW_MTP=1/0` | 初回のみ |
 | エキスパートのメモリ（Qwen3.6 系。上限を超える分は SSD から読む） | `ONW_EXPERT_GB=5`／設定画面 | 不要 |
 | NPUW 重み共有（試験的。auto は Lunar Lake 以降でオン、MoE モデル（LFM2・Qwen3.6）と区間の多いモデル（Granite・llm-jp）は対象外：重みを 1 コピーにしてメモリとコンパイル時間を削減） | `ONW_NPUW=1/0`／設定画面 | 初回のみ |

README_en.md

@@ -118,8 +118,12 @@ vision encoder freed after each image (the default below 24 GB of RAM). The "Mem
 window applies that PC's settings (context length, 64-token blocks, expert memory). On a PC short of memory the
 Qwen3.6 models can keep their experts on the SSD (Settings → "Expert memory"; how it works: [Technical notes](TECHNICAL_en.md)).
 
-Prompt lookup decoding is used automatically where a 16-token verification block is cheap enough (Qwen3.5-9B,
-Gemma, the LFM models), and every model keeps the conversation state between turns (a follow-up with an image
+Prompts (the text you send) are read 64 tokens at a time on the dense models, and layer by layer over the whole prompt
+on the Qwen3.6 models (onw 2.0): on the NPU 3720 Qwen3.5-9B ~80-87 tok/s, llm-jp-4.1-8B ~110, Granite 4.2 3B ~140,
+Qwen3.6-35B-A3B ~100 tok/s (24 GB of RAM or more); on Panther Lake Qwen3.5-9B ~105, Qwen3.6-35B-A3B ~147 tok/s.
+
+Prompt lookup decoding is used automatically where a verification block is cheap enough (the LFM models, Qwen3.5-9B
+and Gemma from Lunar Lake on, ...), and every model keeps the conversation state between turns (a follow-up with an image
 re-processes only the new message: ~2 s). Models marked "switchable" in the thinking column (Qwen, LFM2.5, Gemma 4, Granite 4.2, ...) answer without thinking by
 default (the chat page's thinking switch, `chat_template_kwargs: {"enable_thinking": true}` or `reasoning_effort` turns
 it on); models marked "always" (gpt-oss, llm-jp) think even with thinking mode off, and that counts against max_tokens (give them room). Whatever a model thought comes apart from the answer (its token count as `usage.completion_tokens_details.reasoning_tokens`):
@@ -178,7 +182,7 @@ What needs what:
 | temperature, top_p, top_k, repetition / presence / frequency penalty, max_tokens, stop, seed, thinking | per request (API or chat UI) | no |
 | port, address, model, device, prompt lookup on/off | server option / settings, restart (~20 s) | no |
 | **context length** (`--context`, 32K by default, up to the model's maximum) | server option / settings | no (it only bounds the KV cache) |
-| 64-token prompt blocks (Gemma, the NF4 Qwen3.5-family builds; auto = on from Lunar Lake or with 24 GB+ RAM) | `ONW_S64=1/0` / settings | once, the first time |
+| 64-token prompt blocks (dense models read prompts 64 tokens at a time; auto = on, loaded in place of the 16-token graphs; off = 16 at a time, `ONW_S64=1` = both) | `ONW_S64=1/0` / settings | once, the first time |
 | MTP draft tokens (the NF4 Qwen3.5-family builds; auto = on from Lunar Lake) | `ONW_MTP=1/0` | once, the first time |
 | expert memory (Qwen3.6 models; the rest streams from the SSD) | `ONW_EXPERT_GB=5` / settings | no |
 | NPUW weight sharing (experimental; auto = on from Lunar Lake, not for the MoE models LFM2 / Qwen3.6 or models with many parts (Granite, llm-jp): one copy of the weights, less memory, much faster compile) | `ONW_NPUW=1/0` / settings | once, the first time |

TECHNICAL.md

@@ -14,6 +14,8 @@ onw をどうやって Intel NPU に載せたかの説明です。使い方は [
 - onw 0.7 でのメモリ削減
 - モデルを小さくする（onw convert --compact / --prune）
 - MTP 先読みと 64 トークン処理（onw 1.4、Lunar Lake）
+- MoE のプロンプトをまとめて計算する（onw 2.0）
+- dense モデルのプロンプトを 64 トークンずつ（onw 2.0）
 - 1 ステップのホスト側の処理（onw 1.4）
 - 複数トークンのブロックで、ロジットが行ごと 0 になる（onw 1.4 で修正）
 - NPU で分かった制約（実測）
@@ -339,6 +341,45 @@ Lunar Lake（NPU 4000、Ornith-1.5-9B NF4）で 1 トークンの生成を測る
 - MTP を使うときは 1・16 トークン用のグラフを読み込みません（回答はすべて確認ブロックで書くため）。NPU ドライバーの
   コマンドキューの枠（doorbell）を区間の数が超えないようにするためでもあります（下の「制約」）。
 
+## MoE のプロンプトをまとめて計算する（onw 2.0）
+
+16 トークンのブロックでは、ブロックの全トークンが選んだエキスパート（Qwen3.6-35B-A3B で 1 ブロック 80〜110 個）の重みを
+ブロックごとに読みます。1024 トークンなら同じエキスパートを何十回も読み直すことになり、NPU 3720 で 1024 トークンに 47 秒
+かかっていました。AMD 版 onwx で効いた方法を、OpenVINO のグラフで組み直しました（`onw/grouped.py`）。
+
+- **層ごとに進める**：64 トークンのミキサー区間（`seg*_S64m.xml`：アテンションまたは DeltaNet、MoE 前の正規化、ルーター、
+  共有エキスパートまで。ルーティングされるエキスパートは入れない）で、プロンプトの全ブロックを 1 層ぶん進めます。
+- **エキスパートごとにまとめる**：ホストがルーターの上位 8 個でトークンを並べ替え、エキスパートごとに 16／32／64／128 行の
+  まとまりにして、エキスパートを 32 個（128 行は 16 個）並べた行列積のグラフ（重みは実行時の入力、ルーティングの重みも掛ける）
+  に渡します。各エキスパートの重みは、プロンプト全体で 1 回しか読みません。2 つのリクエストを交互に使い、NPU が 1 つを
+  計算している間に、ホストが次の分を詰めて前の分の結果を書き戻します。
+- **ホスト側**：並べ替えは層ごとに 1 回だけで、各ディスパッチは連続したスライスのコピーにしました。書き戻しは別スレッド、
+  8 個の結果の足し合わせは並べ戻しと 1 回で行います（Panther Lake でホスト側 4.3 → 2.9 秒）。
+- **グラフの置き場**：行列積のグラフは onw の blob 置き場（`onw_blobs/grp_*`）に保存し、ロード時に読み込みます。
+
+| Qwen3.6-35B-A3B、1024 トークン | 16 トークンのブロック | onw 2.0 |
+|---|---|---|
+| NPU 3720（Core Ultra 9 285HX） | 47.5 秒（21.5 tok/s） | 約 10 秒（約 100 tok/s） |
+| NPU 5010（Panther Lake） | 約 42 秒（24 tok/s） | 7.0 秒（約 147 tok/s） |
+
+出力の差は 16 トークンのブロックに対して KL 0.0003〜0.0005（最初のトークンは一致）。メモリは、16 トークンのグラフを読まなく
+なるぶん相殺され、システム全体のピークで +0.3 GB 以内です。ただしミキサー区間の初回コンパイルは重く、16 GB の PC では
+メモリが足りなくなるので、使うのはメモリ 24 GB 以上の PC だけです（`ONW_GROUPED=1／0`）。
+
+生成（1 トークン）では、区間どうしの入出力をロード時に一度だけ結び付けておき（`onw/wired.py`）、毎回の入力の受け渡しを
+なくしました（Panther Lake 8.9 → 10.3 tok/s、出力は同じ）。
+
+## dense モデルのプロンプトを 64 トークンずつ（onw 2.0）
+
+dense モデルのプロンプトは 16 トークンのブロックで処理していました。NPU の 1 回の実行には、ブロックの大きさにあまりよらない固定の手間（区間の切り替え、ホスト側の KV キャッシュの処理、行列の並べ替え）があり、16 トークンではそれが大半でした。
+
+- **測ったこと**（Qwen3.5-9B の 1 層、乱数の重み）：NPU 3720 では DeltaNet 層の 1 トークンあたりの時間が 16 → 64 トークンで 0.90 → 0.41 ms、128 では 0.41 ms のまま（INT4 のグループ量子化の行列積が 64 行で頭打ち）。Panther Lake では 0.46 → 0.25 ms。
+- **やったこと**：64 トークンのグラフを加え、16 トークン用の代わりに読み込みます（1 トークン用＋プロンプト用の 2 種類のまま）。NPU 3720 は重みを共有できないので、サイズを足すと重みの写しが増えますが、入れ替えなら増えません。Lunar Lake 以降も、サイズごとに NPU の作業領域（約 0.6 GB）を取るので入れ替えにしました。
+- **DeltaNet の計算は 16 トークンのチャンクのまま**（`ir.deltanet_chunks`）。64 トークンを 1 チャンクにする式（減衰をチャンク内の累積とチャンクの合計の積で作り、fp16 の桁落ちを避ける）も試しました。NPU 3720 では状態の読み書きが減って DeltaNet が 1.5 倍速くなりましたが、Panther Lake のコンパイラーでは答えが大きくずれたので採用していません。
+- **配布**：公開済みのモデルの重みファイル（`seg*.bin`）は変えていません。同じ変換器で作り直した 64 トークンのグラフの定数は、ほとんどが既存の `seg*.bin` の中に同じものがあり、そこを指します。足りない分（大きなブロックの形を表す整数、区間あたり数百バイト）だけを `seg*x.bin` に置き、読み込み時に入力として読んでから定数に置き換えます（`compact.graft_sizes`、`runtime._xbin_model`）。`engine.json` の `prompt_sizes` に書くので、2.0 より前の onw はこれらを読まず、今までどおり動きます。更新のダウンロードは数 MB です。
+- **128 トークンは見送り**：Panther Lake では 139 tok/s とさらに速くなりますが、使用メモリが 16 トークンより約 1.1 GB 増え、初回のコンパイルが 9B で約 1 時間 40 分かかります（64 トークンで約 50 分）。NPU 3720 では 64 より遅くなります。
+- **結果**（1024 トークン）：Panther Lake の Qwen3.5-9B INT4 44 → 105 tok/s、NPU 3720 の Qwen3.5-9B 31〜43 → 80〜87、llm-jp-4.1-8B 46 → 111、NeoHorse-1-4B 67 → 132、Granite 4.2 3B 69 → 139 tok/s。16 トークンのブロックとの差は KL 0.00005〜0.0015。
+
 ## 1 ステップのホスト側の処理（onw 1.4）
 
 MTP の 1 ブロックのうち、NPU 以外（Python・numpy）が約 47 ms ありました。生成結果を 1 ビットも変えずに、次のことで約 32 ms に

TECHNICAL_en.md

@@ -14,6 +14,8 @@ test machine (Core Ultra 9 285HX, NPU 3720, Windows 11, OpenVINO 2026.5.0b1) unl
 - Memory savings in onw 0.7
 - Smaller models (onw convert --compact / --prune)
 - MTP draft tokens and 64-token blocks (onw 1.4, Lunar Lake)
+- MoE prompts computed per expert (onw 2.0)
+- Dense prompts in 64-token blocks (onw 2.0)
 - Host-side work per step (onw 1.4)
 - Logit rows of all zeros in multi-token blocks (fixed in onw 1.4)
 - What we learned about the NPU (measured)
@@ -336,6 +338,61 @@ spare. Two things use it.
 - With MTP the 1- and 16-token graphs are not loaded (every answer step is a checking block), which also keeps the number of
   parts under the NPU driver's command-queue (doorbell) budget (see below).
 
+## MoE prompts computed per expert (onw 2.0)
+
+A 16-token block reads the weights of every expert its tokens picked (80-110 a block for Qwen3.6-35B-A3B), block after block:
+a 1,024-token prompt reads the same experts dozens of times (47 s on the NPU 3720). What worked in onwx, the AMD sibling, is
+rebuilt here with OpenVINO graphs (`onw/grouped.py`):
+
+- **Layer by layer**: 64-token mixer segments (`seg*_S64m.xml`: attention or DeltaNet, the pre-MoE norm, the router and the
+  shared expert - no routed experts) take every block of the prompt through one layer.
+- **Grouped by expert**: the host sorts the tokens by their router's top 8, packs them per expert into pieces of 16 / 32 / 64 /
+  128 rows, and runs GEMM graphs of 32 experts (16 for 128 rows; the weights are run-time inputs, the routing weights are
+  applied in-graph). Every expert's weights are read once for the whole prompt. Two requests alternate: while the NPU runs
+  one, the host fills the next and copies the previous one's results out.
+- **Host side**: one sort per layer, contiguous slice copies per dispatch, the copy-out in a thread, the sum over the 8 experts
+  fused with the gather back to token order (Panther Lake: host time 4.3 -> 2.9 s).
+- **Graph store**: the GEMM graphs are kept in onw's blob store (`onw_blobs/grp_*`) and imported at load time.
+
+| Qwen3.6-35B-A3B, 1,024 tokens | 16-token blocks | onw 2.0 |
+|---|---|---|
+| NPU 3720 (Core Ultra 9 285HX) | 47.5 s (21.5 tok/s) | ~10 s (~100 tok/s) |
+| NPU 5010 (Panther Lake) | ~42 s (24 tok/s) | 7.0 s (~147 tok/s) |
+
+The output differs by KL 0.0003-0.0005 from the 16-token blocks (same first token). Memory: no 16-token graphs are loaded,
+which offsets the mixers; the system-wide peak is within +0.3 GB. Compiling the mixers the first time is heavy, though, and
+runs a 16 GB PC out of memory, so only PCs with 24 GB of RAM or more use it (`ONW_GROUPED=1 / 0`).
+
+For single tokens the segments' inputs and outputs are bound to each other once at load time (`onw/wired.py`), with no input
+hand-over per token (Panther Lake 8.9 -> 10.3 tok/s, same output).
+
+## Dense prompts in 64-token blocks (onw 2.0)
+
+Dense models read prompts in 16-token blocks. Every NPU run has a fixed cost that hardly depends on the block size
+(switching segments, the host's KV-cache work, re-laying tensors), and at 16 tokens it was most of the time.
+
+- **Measured** (one Qwen3.5-9B layer, random weights): on the NPU 3720 a DeltaNet layer costs 0.90 -> 0.41 ms per token
+  from 16- to 64-token blocks and still 0.41 at 128 (the group-quantized INT4 matmuls stop scaling at 64 rows); on
+  Panther Lake 0.46 -> 0.25 ms.
+- **Done**: 64-token graphs, loaded in place of the 16-token ones (still two sizes: 1 token + the prompt size). The NPU
+  3720 cannot share weights between sizes, so an extra size would be an extra copy of the weights - a swap is not. From
+  Lunar Lake on each size still takes its own NPU scratch (~0.6 GB), so it is a swap there too.
+- **DeltaNet stays in 16-token chunks** (`ir.deltanet_chunks`). A form with the whole 64-token block as one chunk (the
+  decays built as products of in-chunk cumulative gates and chunk totals, no fp16 cancellation) made DeltaNet 1.5x
+  faster on the NPU 3720 by reading and writing the state once a block, but the Panther Lake compiler computed it far
+  off, so it is not used.
+- **Shipping**: the published weights files (`seg*.bin`) are unchanged. The 64-token graphs, converted again with
+  the same converter, find nearly all their constants as the same bytes in the existing `seg*.bin` and point there; the
+  few missing ones (integers for the larger blocks' shapes, a few hundred bytes a segment) are in `seg*x.bin`, read as
+  inputs and turned into constants at load (`compact.graft_sizes`, `runtime._xbin_model`). They are listed under
+  `prompt_sizes` in `engine.json`, which onw before 2.0 does not read, so it keeps working as before. An update
+  downloads a few MB.
+- **No 128-token blocks**: on Panther Lake they reach 139 tok/s, but take ~1.1 GB more memory than the 16-token
+  blocks and ~1 h 40 min to compile for a 9B the first time (64 tokens: ~50 min); on the NPU 3720 they are slower than 64.
+- **Result** (1,024 tokens): Panther Lake Qwen3.5-9B INT4 44 -> 105 tok/s; NPU 3720 Qwen3.5-9B 31-43 -> 80-87,
+  llm-jp-4.1-8B 46 -> 111, NeoHorse-1-4B 67 -> 132, Granite 4.2 3B 69 -> 139 tok/s. KL 0.00005-0.0015 against the
+  16-token blocks.
+
 ## Host-side work per step (onw 1.4)
 
 Of an MTP block, ~47 ms was Python / numpy. Without changing a single output bit, it is now ~32 ms (the same tokens as

onw/__init__.py

@@ -2,4 +2,4 @@
 
 A small engine that runs LLMs entirely on the Intel NPU: static segment graphs + host-managed state.
 """
-__version__ = "1.4.13"
+__version__ = "2.0.0"

onw/app.py

@@ -55,6 +55,14 @@ NF4_ONLY = {"ryugyosoft/Ornith-1.5-9B-nf4-onw", "ryugyosoft/Qwen3.5-9B-nf4-onw",
 # (the NF4 models republished with the MTP layer: a copy without it is an old download - outdated(). Not Bonsai: it
 #  has no MTP layer, so a "no mtp" check made its card "update" forever and it could not be loaded, onw 1.4.9 - 1.4.10)
 NF4_MTP = {"ryugyosoft/Ornith-1.5-9B-nf4-onw", "ryugyosoft/Qwen3.5-9B-nf4-onw"}
+# [2.0] the Qwen3.6 MoE models republished with the mixer-only 64-token graphs (the grouped-MoE prompt path, ~4-5x the
+# prompt speed): a copy without them (engine.json "mixer") is an old download
+MOE_MIXER = {"ryugyosoft/Qwen3.6-35B-A3B-onw", "ryugyosoft/Qwen3.6-REAP-18B-A3B-onw"}
+# [2.0] the dense models republished with larger prompt blocks (engine.json "prompt_sizes", compact.graft_sizes - the
+# weights files unchanged, an update fetches a few MB): a copy without them is an old download
+DENSE_PROMPT = {"ryugyosoft/Qwen3.5-9B-onw", "ryugyosoft/Ornith-1.5-9B-onw", "ryugyosoft/NeoHorse-1-4B-onw",
+                "ryugyosoft/granite-4.2-3b-onw", "ryugyosoft/granite-4.2-8b-onw",
+                "ryugyosoft/llm-jp-4.1-8b-thinking-onw", "ryugyosoft/llm-jp-4.1-8b-thinking-int8-onw"}
 
 
 def catalog():
@@ -293,9 +301,9 @@ def raise_window():
 
 
 def big_blocks_on(cfg, ram):
-    """Same rule as chat.block_sizes: 64-token blocks only with >= 24 GB RAM unless forced in the settings."""
-    b = cfg.get("big_blocks", "auto")
-    return b == "on" if b in ("on", "off") else (ram is not None and ram >= 24)
+    """A second copy of the weights for the 64-token blocks (memory estimate): only when forced on in the settings -
+    since 2.0 "auto" (chat.block_sizes) loads them in the 16-token graphs' place where the sizes cannot share weights."""
+    return cfg.get("big_blocks", "auto") == "on"
 
 
 def usable_ram(ram):
@@ -570,13 +578,18 @@ def outdated(repo, p):
     fetches just the changed files): the gpt-oss-20b models published with an INT4 embedding table (v0.11.0 era) -
     the INT8 one is noticeably more accurate; the NF4 Qwen3.5 models published before the MTP layer and the 64-token
     blocks (onw 1.4.0: ~1.3x the answer and 2x the prompt speed on Lunar Lake)."""
-    gpt, nf4 = repo.startswith("ryugyosoft/gpt-oss-20b"), repo in NF4_MTP
-    if not (gpt or nf4) or not downloaded_by_onw(p):
+    gpt, nf4, mix = repo.startswith("ryugyosoft/gpt-oss-20b"), repo in NF4_MTP, repo in MOE_MIXER
+    dense = repo in DENSE_PROMPT
+    if not (gpt or nf4 or mix or dense) or not downloaded_by_onw(p):
         return False
     try:
         meta = json.load(open(os.path.join(p, "engine.json")))
     except (OSError, ValueError):
         return False
+    if mix:
+        return "mixer" not in meta
+    if dense:
+        return "prompt_sizes" not in meta
     return "embed" in (meta.get("tables") or {}) if gpt else "mtp" not in meta
 
 

onw/chat.py

@@ -7,7 +7,7 @@ import numpy as np
 import openvino as ov
 from transformers import AutoTokenizer
 
-from .runtime import Cancelled, SegmentedModel
+from .runtime import Cancelled, SegmentedModel, all_sizes, grouped_wanted
 
 
 def npu_available(core):
@@ -50,33 +50,43 @@ def use_mtp(meta, shared):
     return bool(meta.get("mtp")) and (env == "1" if env in ("0", "1") else shared)
 
 
-def block_sizes(meta, shared=False, mtp=False):
-    """Block sizes to load. Prompt blocks above 16 tokens prefill faster (Lunar Lake: 64-token blocks ~2x); with the
-    weights shared between the sizes (NPUW) they cost next to nothing, else every size holds its own copy of the
-    in-graph weights, so they are used only with >= 24 GB RAM; ONW_S64=1 / 0 forces them on / off."""
-    sizes = meta["sizes"]
+def block_sizes(meta, shared=False, mtp=False, arch=None):
+    """Block sizes to load: the 1-token graphs for answers and ONE prompt size, the largest the model has ([2.0]: in
+    the 16-token graphs' place - with the weights shared between the sizes (NPUW) every size still adds its NPU
+    scratch, without sharing a whole copy of the weights; Qwen3.5-9B INT4 64-token blocks: NPU 3720 31-43 -> 81-87
+    tok/s, Panther Lake 44 -> 104, +0.35-0.5 GB). With MTP drafts the smallest checking size (>= 4) instead of the
+    1-token graphs. ONW_S64=0: the 16-token graphs as before; 1: every size. arch: the NPU's (runtime.npu_arch) -
+    NPU 3720 takes blocks up to 64 tokens (128 are slower there: its INT4 matmuls stop scaling); ONW_PROMPT_MAX: a cap."""
+    sizes = all_sizes(meta)
     only = os.environ.get("ONW_SIZES")                  # e.g. "16": decode with the 16-token graphs only
+    if not only and 1 in sizes and grouped_wanted(meta):
+        # [2.0] a MoE model with mixer graphs: every prompt goes the grouped-MoE path (onw/grouped.py), so only the
+        # 1-token graphs besides them (the NPU driver runs out of command lists with the 16-token ones as well)
+        return [1]
     if only:
-        sizes = [S for S in sizes if str(S) in only.split(",")] or sizes
+        return [S for S in sizes if str(S) in only.split(",")] or sizes
+    cap = int(os.environ.get("ONW_PROMPT_MAX", "0")) or (64 if arch is not None and arch < 4000 else 0)
+    if cap:
+        sizes = [S for S in sizes if S <= cap]
     env = os.environ.get("ONW_S64")
-    ram = total_ram_gb()
-    big = env == "1" if env in ("0", "1") else (shared or (ram is not None and ram >= 24))
-    sizes = [S for S in sizes if S <= 16 or big]
-    if not only:
-        small = [S for S in sizes if S <= 16]
-        verify = [S for S in small if S >= 4]
-        if shared and mtp and verify:
-            # MTP drafts: every answer step is a verification block (x + 2 drafts, + rows to redo) on the smallest
-            # graph of >= 4 rows (Lunar Lake: the 4-token one ~15% cheaper than the 16-token one); no 1-token graphs
-            # (no single steps are left, and the NPU driver runs out of command queues past ~40 compiled parts:
-            # Ornith 9B has 13 per size + 3 for MTP)
-            keep = {min(verify)}
-        else:
-            # otherwise the 1- and 16-token graphs (4 / 8-token ones stay out: without NPUW each size holds its own
-            # copy of the weights)
-            keep = {S for S in small if S in (1, 16)} or set(small)
-        sizes = [S for S in sizes if S > 16 or S in keep]
-    return sizes
+    if env == "0":
+        sizes = [S for S in sizes if S <= 16]
+    small = [S for S in sizes if S <= 16]
+    big = [S for S in sizes if S > 16]
+    verify = [S for S in small if S >= 4]
+    if shared and mtp and verify:
+        # MTP drafts: every answer step is a verification block (x + 2 drafts, + rows to redo) on the smallest
+        # graph of >= 4 rows (Lunar Lake: the 4-token one ~15% cheaper than the 16-token one); no 1-token graphs
+        # (no single steps are left, and the NPU driver runs out of command queues past ~40 compiled parts:
+        # Ornith 9B has 13 per size + 3 for MTP)
+        keep = {min(verify)}
+    elif big and env != "1":
+        keep = {S for S in small if S == 1}            # [2.0] the largest prompt size does the 16-token graphs' work
+    else:
+        # the 1- and 16-token graphs (4 / 8-token ones stay out: without NPUW each size holds its own copy of the
+        # weights)
+        keep = {S for S in small if S in (1, 16)} or set(small)
+    return [S for S in sizes if S in keep or (S > 16 and (env == "1" or S == max(big)))]
 
 
 # Thinking levels ("low" / "medium" / "high", the API's reasoning_effort): harmony models (gpt-oss, llm-jp) were trained
@@ -103,12 +113,23 @@ class ChatEngine:
         meta = json.load(open(os.path.join(engine_dir, "engine.json")))
         shared = shares_weights(core, device, engine_dir, meta)
         mtp = use_mtp(meta, shared)
-        sizes = block_sizes(meta, shared, mtp)
+        from .runtime import npu_arch
+        arch = npu_arch(core) if device == "NPU" else None
+        sizes = block_sizes(meta, shared, mtp, arch)
         from . import parts
+        while shared and parts.estimate(meta, sizes) > parts.MANY and len([S for S in sizes if S > 1]) > 1:
+            # [2.0] rather fewer block sizes than no sharing (a second copy of the weights): the 16-token graphs go
+            # first when larger prompt blocks are there, then the smallest of those
+            big = [S for S in sizes if S > 16]
+            if 16 in sizes and big:
+                drop = 16
+            else:
+                drop = min(big or [S for S in sizes if S > 1])
+            sizes = [S for S in sizes if S != drop]
         if shared and parts.estimate(meta, sizes) > parts.MANY:   # (NPUW goes off for so many parts: no sharing)
             shared = False
             mtp = use_mtp(meta, shared)
-            sizes = block_sizes(meta, shared, mtp)
+            sizes = block_sizes(meta, shared, mtp, arch)
         self.model = SegmentedModel(core, engine_dir, device, cfg, sizes=sizes, context=context, mtp=mtp)
         self.tok = AutoTokenizer.from_pretrained(engine_dir)
         # thinking: "switch" = the template takes enable_thinking (Qwen); "forced" = its generation prompt always opens
@@ -446,7 +467,8 @@ class ChatEngine:
         """Feed ids[start:stop] in blocks (with image rows / explicit positions when given); last logits."""
         m, s, lg = self.model, start, None
         stop = len(ids) if stop is None else stop
-        if m.streaming and stop - start > m.S:
+        if (m.streaming and stop - start > m.S) or (getattr(m, "mix", None) is not None and stop - start >= 2):
+            # [2.0] a MoE model with mixer graphs: the grouped-MoE prompt path (onw/grouped.py) whenever it pays
             # experts streamed from the SSD: go layer by layer so each layer's experts are read once per prompt
             lg = m.prefill(ids[start:stop], None if embeds is None else embeds[start:stop],
                            None if pos is None else pos[:, start:stop], want_logits=need_logits, stop=self._cancel)

onw/compact.py

@@ -192,3 +192,54 @@ def compact(d):
         int8_vision(d)
     if mine:
         common.finish(d)
+
+
+def graft_sizes(pub, new, sizes):
+    """[2.0] Block sizes of the conversion `new` (same model, same converter) added to the published model folder `pub`
+    without touching its files: each new graph's constants point at the same bytes in pub's seg<i>.bin, the few it
+    lacks (shape constants of the larger blocks) go to seg<i>x.bin at offsets from len(seg<i>.bin) on (runtime.
+    _xbin_model); engine.json lists them under "prompt_sizes" (onw before 2.0 reads "sizes" only, so it never sees
+    graphs it could not load). An update then downloads a few MB instead of every weights file."""
+    meta = json.load(open(os.path.join(pub, "engine.json")))
+    nmeta = json.load(open(os.path.join(new, "engine.json")))
+    groups = {}
+    for name, m in nmeta["segments"].items():
+        if m["S"] in sizes:
+            groups.setdefault(name.split("_S")[0], []).append(name)
+    for seg, xmls in sorted(groups.items()):
+        binf = seg + ".bin"
+        old = open(os.path.join(pub, binf), "rb").read()
+        index = {}                                   # the constants pub's own graphs of this segment store
+        for name, m in meta["segments"].items():
+            if name.split("_S")[0] == seg:
+                for mt in DATA_RE.finditer(open(os.path.join(pub, name), "rb").read()):
+                    off, size = int(mt.group(1)), int(mt.group(2))
+                    index.setdefault(hashlib.sha1(old[off:off + size]).digest() + size.to_bytes(8, "little"), off)
+        data = open(os.path.join(new, nmeta["segments"][xmls[0]].get("bin") or binf), "rb").read()
+        extra, xpos = [], {}
+        for x in xmls:
+            def remap(mt):
+                off, size = int(mt.group(1)), int(mt.group(2))
+                chunk = data[off:off + size]
+                h = hashlib.sha1(chunk).digest() + size.to_bytes(8, "little")
+                if h in index:
+                    to = index[h]
+                else:
+                    if h not in xpos:
+                        xpos[h] = sum(len(c) for c in extra)
+                        extra.append(chunk)
+                    to = len(old) + xpos[h]
+                return mt.group(0).replace(b'offset="%d"' % off, b'offset="%d"' % to, 1)
+            text = DATA_RE.sub(remap, open(os.path.join(new, x), "rb").read())
+            with open(os.path.join(pub, x), "wb") as f:
+                f.write(text)
+            meta["segments"][x] = {**nmeta["segments"][x], "bin": binf}
+            if DATA_RE.search(text) and max(int(mt.group(1)) for mt in DATA_RE.finditer(text)) >= len(old):
+                meta["segments"][x].update(xbin=seg + "x.bin", xbase=len(old))
+        if extra:
+            with open(os.path.join(pub, seg + "x.bin"), "wb") as f:
+                for c in extra:
+                    f.write(c)
+        print(f"{seg}: {len(xmls)} graphs, {sum(len(c) for c in extra)} bytes of new constants", flush=True)
+    meta["prompt_sizes"] = sorted(set(meta.get("prompt_sizes") or []) | set(sizes))
+    _save_meta(pub, meta)

onw/grouped.py

@@ -0,0 +1,249 @@
+"""[2.0] Prompts of a mixture-of-experts model with a grouped expert GEMM (the idea of onwx, the AMD sibling, on the
+Intel NPU): layer by layer over the whole prompt, the mixer-only graphs (mixer + post-norm + router + shared expert, no
+routed experts: seg<i>_S<S>m.xml, models/qwen3_5_moe.add_mixer) run every block of the layer; then the host groups the
+prompt's tokens by the expert the router picked, packs them into segments of R rows, and runs graphs of G expert slots
+(each slot: its expert's INT4 weights bound as inputs, R gathered rows, the routing weights multiplied in) - each
+expert's weights are read once for the whole prompt instead of once per block for every token of the block's union.
+The outputs are added to the residual stream on the host before the next layer's mixer.
+
+Measured (Qwen3.6-35B-A3B, NPU 3720): 1,024 tokens 47.5 s -> 10.2 s (S=64 mixers), KL 0.0003-0.0005 against the
+16-token blocks."""
+import os, time
+import numpy as np
+import openvino as ov
+import openvino.opset13 as ops
+from . import ir
+
+SHAPES = [(int(g), int(r)) for g, r in (s.split("x") for s in os.environ.get("ONW_GROUPED_SHAPES",
+                                                                              "32x16,32x32,32x64,16x128").split(","))]
+
+
+def _i64(v):
+    return ops.constant(np.asarray(v, np.int64))
+
+
+def grouped_graph(G, R, H, I):
+    """G expert slots (w_j / s_j: an expert's packed INT4 weights and scales) x R rows each: x [G, R, H] f16, rw [G, R, 1]
+    (the rows' routing weights) -> y [G, R, H] f16 = rw * expert_j(x_j)"""
+    X = ops.parameter([G, R, H], ov.Type.f16, name="x")
+    xf = ops.convert(X, ov.Type.f32)
+    params, ys = [X], []
+    for j in range(G):
+        slot = ir.expert_slot_params(j, H, I)
+        params += slot
+        xj = ops.reshape(ops.slice(xf, np.array([j]), np.array([j + 1]), np.array([1]), np.array([0])), _i64([R, H]), False)
+        ys.append(ops.unsqueeze(ir.expert_ffn(xj, slot, I, H), _i64([0])))
+    y = ops.concat(ys, 0) if G > 1 else ys[0]
+    rw = ops.parameter([G, R, 1], ov.Type.f16, name="rw")
+    params.append(rw)
+    y = ops.convert(ops.multiply(y, ops.convert(rw, ov.Type.f32)), ov.Type.f16)
+    r = ops.result(y)
+    r.output(0).get_tensor().set_names({"y"})
+    return ov.Model([r], params, f"grp_{G}x{R}")
+
+
+def _compiled(core, device, config, G, R, H, I, key):
+    """The grouped graph compiled once into onw's own blob store (npu_cache/onw_blobs/grp_<G>x<R>-*.blob: OpenVINO's
+    CACHE_DIR files are pruned after every load on most NPUs, and a CACHE_DIR hit keeps a second copy in memory),
+    imported from there later (~0.1 s instead of a compile at every load)."""
+    import io, json
+    from . import runtime as rt
+    cache = (config or {}).get("CACHE_DIR")
+    nocache = {k: v for k, v in (config or {}).items() if k != "CACHE_DIR"}
+    if device != "NPU" or not cache or os.environ.get("ONW_OWN_BLOBS") == "0":
+        return core.compile_model(grouped_graph(G, R, H, I), device, config)
+    bp = rt.blob_path(cache, f"grp_{G}x{R}", ["grouped v1", G, R, H, I, key,
+                                               json.dumps(nocache, sort_keys=True, default=str)])
+    if os.path.exists(bp):
+        try:
+            c = rt.import_blob(core, bp, device, nocache)
+            rt.USED_BLOBS.add(bp)
+            return c
+        except Exception as e:                           # (a broken file: compiled again below)
+            print(f"[grouped] {os.path.basename(bp)}: {e}; compiling it again", flush=True)
+    cm = core.compile_model(grouped_graph(G, R, H, I), device, nocache)
+    os.makedirs(os.path.dirname(bp), exist_ok=True)
+    buf = io.BytesIO()
+    cm.export_model(buf)
+    with open(f"{bp}.{os.getpid()}.tmp", "wb") as f:
+        f.write(buf.getbuffer())
+    del buf, cm
+    os.replace(f"{bp}.{os.getpid()}.tmp", bp)
+    rt.USED_BLOBS.add(bp)
+    return rt.import_blob(core, bp, device, nocache)
+
+
+class GroupedMoE:
+    """The grouped expert GEMMs: per shape (G slots, R rows) two requests used in turn (the host gathers and binds the
+    next dispatch and scatters the previous one while the NPU runs)."""
+    def __init__(self, core, device, config, bank, H, I, streaming, key=""):
+        from concurrent.futures import ThreadPoolExecutor
+        self.bank, self.H, self.streaming = bank, H, streaming
+        self.reqs = {}
+        for G, R in SHAPES:
+            c = _compiled(core, device, config, G, R, H, I, key)
+            self.reqs[R] = (G, [(c.create_infer_request(), np.zeros((G * R, H), np.float16), np.zeros((G * R, 1), np.float16))
+                                for _ in range(2)])
+        self.Rs = sorted(self.reqs)
+        self.nt = min(8, max(2, (os.cpu_count() or 4) // 2))
+        self.pool = ThreadPoolExecutor(self.nt)
+        self.copier = ThreadPoolExecutor(1)                 # (a finished dispatch's outputs, copied out off the main thread)
+        self.stats = {"dispatch": 0, "npu": 0.0, "host": 0.0, "prep": 0.0, "gather": 0.0, "bind": 0.0, "scatter": 0.0,
+                      "sum": 0.0}
+
+    def _take(self, src, si):
+        """src[si] in threads (numpy's fancy indexing runs without the GIL: ~2x on the row gathers)"""
+        out = np.empty((len(si),) + src.shape[1:], src.dtype)
+        n = self.nt
+        q = (len(si) + n - 1) // n
+        list(self.pool.map(lambda c: np.take(src, si[c * q:(c + 1) * q], 0, out=out[c * q:(c + 1) * q]), range(n)))
+        return out
+
+    def _ksum(self, Zs, at, N):
+        """y [N, H] f32: row i = sum over k of Zs[at[i, k]] (the gather back to token order and the sum in one pass)"""
+        y = np.empty((N, Zs.shape[1]), np.float32)
+        q = (N + self.nt - 1) // self.nt
+
+        def part(c):
+            s = slice(c * q, (c + 1) * q)
+            a = y[s]
+            a[:] = Zs[at[s, 0]]
+            for k in range(1, at.shape[1]):
+                a += Zs[at[s, k]]
+        list(self.pool.map(part, range(self.nt)))
+        return y
+
+    def run(self, layer, xn, idx, w, prefetch=None):
+        """xn [N, H] (the post-norm rows), idx / w [N, K] (the router's experts / weights) -> the routed MoE [N, H] f32.
+        prefetch(): called once this layer's experts are bound (streaming: start reading the next layer's)."""
+        t0 = time.perf_counter()
+        H = self.H
+        N, K = idx.shape
+        flat = idx.reshape(-1).astype(np.int64)
+        order = np.argsort(flat, kind="stable")              # (token, k) pairs sorted by expert
+        tok = order // K
+        wt = w.reshape(-1)[order].astype(np.float16)[:, None]
+        ex = flat[order]
+        bounds = np.flatnonzero(np.diff(ex)) + 1
+        starts, ends = np.concatenate([[0], bounds]), np.concatenate([bounds, [len(ex)]])
+        Rmax = self.Rs[-1]
+        pieces = {R: [] for R in self.Rs}                    # R -> [(expert, start, end)]
+        for s, e in zip(starts, ends):
+            for a in range(s, e, Rmax):
+                b = min(a + Rmax, e)
+                pieces[next(r for r in self.Rs if r >= b - a)].append((int(ex[s]), a, b))
+        uniq = sorted({int(x) for x in ex[starts]})
+        keys = [(layer, e) for e in uniq]
+        pairs = dict(zip(uniq, self.bank.get_many(keys) if self.streaming else [self.bank[k] for k in keys]))
+        if prefetch is not None:
+            prefetch()
+        # the rows in expert order once (one big gather): a dispatch then copies contiguous slices in and out
+        xsort = self._take(np.ascontiguousarray(xn, np.float16), tok)
+        Zs = np.empty((N * K, H), np.float16)                # the weighted expert outputs, in expert order
+        jobs = [(R, self.reqs[R][0], lst[d0:d0 + self.reqs[R][0]]) for R, lst in pieces.items()
+                for d0 in range(0, len(lst), self.reqs[R][0])]
+        pend, k, tn = None, 0, 0.0
+        st, pc = self.stats, time.perf_counter
+        st["prep"] += pc() - t0
+
+        copies = {}                                          # request -> its outputs' copy-out (done before it runs again)
+
+        def out(Y, R, grp):
+            for j, (_, a, b) in enumerate(grp):
+                Zs[a:b] = Y[j * R:j * R + b - a]
+
+        def finish(p):
+            nonlocal tn
+            req, R, grp = p
+            t = pc()
+            req.wait()
+            t1 = pc()
+            tn += t1 - t
+            copies[id(req)] = self.copier.submit(out, req.get_output_tensor(0).data.reshape(-1, H), R, grp)
+            st["scatter"] += pc() - t1
+        for R, G, grp in jobs:
+            t = pc()
+            two = self.reqs[R][1]
+            req, X, RW = two[k % 2] if pend is None or pend[0] is not two[k % 2][0] else two[(k + 1) % 2]
+            k += 1
+            if id(req) in copies:
+                copies.pop(id(req)).result()
+            for j, (_, a, b) in enumerate(grp):              # (padding rows keep old data: their outputs are not read)
+                X[j * R:j * R + b - a] = xsort[a:b]
+                RW[j * R:j * R + b - a] = wt[a:b]
+            t1 = pc()
+            st["gather"] += t1 - t
+            for j in range(G):
+                w_t, s_t = pairs[grp[j][0] if j < len(grp) else grp[0][0]]
+                req.set_tensor(f"w_{j}", w_t)
+                req.set_tensor(f"s_{j}", s_t)
+            req.start_async({"x": X.reshape(G, R, H), "rw": RW.reshape(G, R, 1)}, share_inputs=True)
+            st["bind"] += pc() - t1
+            if pend is not None:
+                finish(pend)
+            pend = (req, R, grp)
+            self.stats["dispatch"] += 1
+        if pend is not None:
+            finish(pend)
+        for f in copies.values():
+            f.result()
+        t = pc()
+        y = self._ksum(Zs, np.argsort(order).reshape(N, K), N)   # (pair (token i, k) sits at Zs row inv[i * K + k])
+        st["sum"] += pc() - t
+        self.stats["npu"] += tn
+        self.stats["host"] += time.perf_counter() - t0 - tn
+        return y
+
+
+def make(m):
+    """runtime.SegmentedModel m's GroupedMoE (built at load time, after the mixer graphs)"""
+    tried = m._ctx.get("tried")
+    return GroupedMoE(m._ctx["core"], m._ctx["device"], m._ctx["config"], m.bank, m.cfg["hidden_size"],
+                      m.cfg["moe_intermediate_size"], m.streaming, getattr(tried, "key", ""))
+
+
+def prefill(m, ids, embeds=None, pos=None, want_logits=True, stop=None):
+    """runtime.SegmentedModel m: the prompt ids through the mixer graphs (m.mix) layer by layer with the grouped MoE;
+    the last block's logits (want_logits) or None. stop(): checked between layers (Cancelled)."""
+    from .runtime import Cancelled
+    S, segs = m.mix
+    if m.grouped is None:
+        m.grouped = make(m)
+    blocks, s = [], 0
+    while s < len(ids):
+        k = min(S, len(ids) - s)
+        start = m.cur
+        _, n, x, common = m._block_inputs(ids[s:s + k], None if embeds is None else embeds[s:s + k],
+                                          None if pos is None else pos[:, s:s + k], S=S)
+        blocks.append({"n": n, "cur": start, "common": common, "carry": {"inputs_embeds": x}})
+        m.cur += n
+        s += k
+    m._last = None                                       # (truncate() only follows step())
+    logits = None
+    for i, (req, meta, names) in enumerate(segs):
+        if stop is not None and stop():
+            raise Cancelled()
+        xns = []
+        for bi, b in enumerate(blocks):
+            if meta.get("head") and not (want_logits and bi == len(blocks) - 1):
+                continue
+            new, route, lg = m._run_segment(req, meta, names, b["carry"], b["common"], None, S, b["n"], b["cur"], keep=True)
+            logits = lg if lg is not None else logits
+            b["carry"] = {k: v for k, v in {**b["carry"], **new}.items() if k in m._mix_needed[i] or k == "carry.h"}
+            if "carry.xn" in new and route is not None:
+                xns.append((new["carry.xn"][0, :b["n"]], route))
+        if xns:                                          # this part ended with layer L's router: its routed MoE now
+            L = meta["router_layer"]
+            xn = np.concatenate([x for x, _ in xns])
+            idx = np.concatenate([r[0][:len(x)] for x, r in xns])
+            w = np.concatenate([r[1][:len(x)] for x, r in xns])
+            pf = None
+            if m.streaming and L + 1 < m.cfg["num_hidden_layers"]:   # (every expert of the next layer: a long prompt
+                pf = lambda: m.bank.prefetch([(L + 1, e) for e in range(m.E)], protect_layer=L)   # uses nearly all)
+            y = m.grouped.run(L, xn, idx, w, pf)
+            o = 0
+            for b in blocks:
+                h = b["carry"]["carry.h"]
+                h[0, :b["n"]] = (h[0, :b["n"]].astype(np.float32) + y[o:o + b["n"]]).astype(h.dtype)
+                o += b["n"]
+    return logits

onw/i18n.json

@@ -39,7 +39,7 @@
 "URL をコピーしました": "URL copied",
 "auto（Lunar Lake 以降でオン）": "auto (on for Lunar Lake+)",
 "auto（OS に合わせる）": "auto (follow the OS)",
-"auto（Lunar Lake 以降か RAM 24 GB 以上でオン）": "auto (on with Lunar Lake or later, or 24 GB+ RAM)",
+"auto（オン。NPU 3720 などでは 16 トークン処理と入れ替え）": "auto (on; on NPU 3720 and the like in place of the 16-token blocks)",
 "auto（足りないときだけ SSD から）": "auto (SSD when RAM is short)",
 "gpt-oss-20b の高精度版": "The higher-precision gpt-oss-20b",
 "gpt-oss-20b の高精度版（エキスパート INT8。メモリ 24 GB 以上向け）": "The higher-precision gpt-oss-20b (INT8 experts; for 24 GB+ of RAM)",
@@ -397,7 +397,7 @@
 "URL をコピーしました": "已复制 URL",
 "auto（Lunar Lake 以降でオン）": "auto（Lunar Lake 起开启）",
 "auto（OS に合わせる）": "auto（跟随系统）",
-"auto（Lunar Lake 以降か RAM 24 GB 以上でオン）": "auto（Lunar Lake 及更新机型或内存 24 GB 以上时开启）",
+"auto（オン。NPU 3720 などでは 16 トークン処理と入れ替え）": "auto（开启；在 NPU 3720 等机型上替换 16 token 分块）",
 "auto（足りないときだけ SSD から）": "auto（内存不足时用 SSD）",
 "gpt-oss-20b の高精度版": "gpt-oss-20b 的高精度版",
 "gpt-oss-20b の高精度版（エキスパート INT8。メモリ 24 GB 以上向け）": "gpt-oss-20b 的高精度版（INT8 专家；面向 24 GB 以上内存）",

onw/models/qwen3_5_moe.py

@@ -48,7 +48,7 @@ class _MTPWeights:
 
 
 def build_segments(cfg, W, S, T, fmt="q4g128", dn_fmt=None, norm_maxscale=False, P=P, attn_fmt=None, mtp=False,
-                   draft_vocab=None):
+                   draft_vocab=None, mixer_only=False):
     """dn_fmt: format of the DeltaNet in_proj_qkv / out_proj weights (default fmt); attn_fmt: the full-attention
     q/k/v_proj (default fmt; NF4 models keep them INT8); norm_maxscale: residual-stream
     RMSNorms with ir.rmsnorm(maxscale=True); P: checkpoint key prefix (text-only "qwen3_5_text" repos use "model.").
@@ -223,12 +223,15 @@ def build_segments(cfg, W, S, T, fmt="q4g128", dn_fmt=None, norm_maxscale=False,
             result.append((ir.finish(outs, params, f"q35_seg{si}_S{S}"), meta))
         result.append(head_segment(len(result)))
         return result
-    for si in range(L + 1):
+    for si in range(L if mixer_only else L + 1):
+        # mixer_only (grouped-MoE prompt path): no routed experts in the graphs - segment si takes h + the routed MoE
+        # of layer si-1 as carry.h (added on the host from the grouped expert GEMM)
         params, outs, shared = [], {}, {}
-        consume = si - 1 if si > 0 else None
-        meta = {"index": si, "consume_moe": consume, "router_layer": si if si < L else None, "S": S, "slots": slots}
+        consume = si - 1 if si > 0 and not mixer_only else None
+        meta = {"index": si, "consume_moe": consume, "router_layer": si if si < L else None, "S": S,
+                "slots": 0 if mixer_only else slots}
         if consume is None:
-            h = ops.parameter([1, S, H], ov.Type.f32, name="inputs_embeds")
+            h = ops.parameter([1, S, H], ov.Type.f32, name="inputs_embeds" if si == 0 else "carry.h")
             params.append(h)
         else:
             hin = ops.parameter([1, S, H], ov.Type.f32, name="carry.h")
@@ -325,6 +328,40 @@ def export(src, out, sizes=(1, 16), T=1024, fmt="q4g128", graphs_only=False, dn_
     common.write_engine(out, meta, graphs_only)             # (with graphs_only the vision entry is kept)
 
 
+def add_mixer(src, out, S=64, keep=None):
+    """[2.0] The grouped-MoE prompt path's graphs (onw/grouped.py) added to a converted MoE model folder: mixer-only
+    S-token segments seg<i>_S<S>m.xml (the mixer, the post-attention norm, the router and the shared expert - no routed
+    experts: those run on the host-gathered rows), engine.json "mixer": {"S": S}; compact's dedup then stores their
+    constants in seg<i>.bin with the other block sizes' (the same weights). src: the checkpoint (the experts' tensors
+    are not read)."""
+    from ..prune import maybe_prune
+    from .. import compact
+    meta = json.load(open(os.path.join(out, "engine.json")))
+    if meta.get("needs") == "nf4":
+        raise ValueError("add_mixer: NF4 models are not supported yet")
+    if keep is not None:                         # (a pruned model's kept experts: {layer: [ids]} - its router rows)
+        from ..prune import PrunedCheckpoint
+        keep = {int(k): v for k, v in keep.items()}
+        W, cfg = PrunedCheckpoint(Checkpoint(src), keep), {**text_config(src), "num_experts": len(next(iter(keep.values())))}
+    else:
+        common.restore_prune(out, True)
+        W, cfg = maybe_prune(Checkpoint(src), text_config(src))
+    pre = prefix(W)
+    segs = {}
+    for m, sm in build_segments(cfg, W, S, meta["T"], "q4g128", meta.get("deltanet_fmt"), meta.get("norm_maxscale", False),
+                                pre, meta.get("attn_qkv_fmt"), mixer_only=True):
+        name = f"seg{sm['index']:02d}_S{S}m.xml"
+        ov.save_model(m, os.path.join(out, name))
+        segs[name] = {**sm, "mixer_only": True}
+        print(name, flush=True)
+    meta["segments"].update(segs)
+    meta["mixer"] = {"S": S}
+    with open(os.path.join(out, "engine.json.tmp"), "w") as f:
+        json.dump(meta, f, indent=1)
+    os.replace(os.path.join(out, "engine.json.tmp"), os.path.join(out, "engine.json"))
+    compact.dedup_segments(out)
+
+
 def export_mtp(cfg, W, out, T, fmt, norm_maxscale, pre, S=16):
     """The MTP layer for draft tokens (dense models): mtp_S16.xml (one 16-token graph does every MTP call - its cost
     hardly depends on the rows) and mtp_head_S1.xml (an INT8 head over the 32K most frequent tokens, mtp_vocab.npy,

onw/runtime.py

@@ -15,7 +15,7 @@ State handled here, by input/output naming convention:
                            runtime input makes the NPU move the whole table every call)
 Every state tensor lives in the input buffers of one request and is shared by all block-size variants.
 """
-import json, os, time
+import json, os, re, time
 import numpy as np
 import openvino as ov
 
@@ -275,10 +275,58 @@ def codes_const_scale(model):
     return n
 
 
+def all_sizes(meta):
+    """[2.0] Every block size the model folder has graphs for: "sizes" (what onw before 2.0 reads) + "prompt_sizes"
+    (larger prompt blocks added to a published model later, see compact.graft_sizes - out of older onw's sight, whose
+    loader cannot read their seg<i>x.bin)."""
+    return sorted(set(meta["sizes"]) | set(meta.get("prompt_sizes") or []))
+
+
+_CONST_RE = re.compile(rb'<layer id="\d+" name="([^"]+)" type="Const"([^>]*)>(\s*)<data ([^>]*?)offset="(\d+)" size="(\d+)"([^>]*)/>')
+
+
+def _xbin_model(core, d, name, m, weights):
+    """A graph added by compact.graft_sizes: its constants at offsets >= m["xbase"] are in m["xbin"] (the few new ones -
+    shape constants of the larger block - the rest point into the published seg<i>.bin, unchanged). Those are read as
+    Parameters (values unknown: shapes left open), then replaced by their constants and the shapes inferred again."""
+    base, extra = m["xbase"], np.fromfile(os.path.join(d, m["xbin"]), np.uint8)
+    fix = {}
+
+    def sub(mt):
+        off = int(mt.group(5))
+        if off < base:
+            return mt.group(0)
+        fix[mt.group(1).decode()] = (off - base, int(mt.group(6)))
+        return (b'<layer id="' + mt.group(0).split(b'"')[1] + b'" name="' + mt.group(1) + b'" type="Parameter"'
+                + mt.group(2) + b'>' + mt.group(3) + b'<data ' + mt.group(4) + mt.group(7).lstrip() + b'/>')
+    xml = _CONST_RE.sub(sub, open(os.path.join(d, name), "rb").read())
+    model = core.read_model(xml.decode("utf8"), weights)
+    import openvino.opset13 as ops
+    for p in list(model.get_parameters()):
+        if p.get_friendly_name() in fix:
+            o, n = fix.pop(p.get_friendly_name())
+            et = p.get_element_type()
+            c = ops.constant(extra[o:o + n].view(et.to_dtype()).reshape(list(p.get_partial_shape().to_shape())), et)
+            c.set_friendly_name(p.get_friendly_name())
+            p.output(0).replace(c.output(0))
+            model.remove_parameter(p)
+    if fix:
+        raise ValueError(f"{name}: constants of {m['xbin']} not found in the graph: {sorted(fix)[:3]}")
+    model.validate_nodes_and_infer_types()
+    return model
+
+
 def read_segment(core, d, name, m, npuw, bins, T, built_T):
     """The segment's ov.Model, reshaped to context T. With NPUW every segment of a .bin reads against the same
     in-memory copy (bins cache), so the weights bank sees one set of constants."""
-    if npuw:
+    if m.get("xbin"):
+        b = m.get("bin") or name[:-4] + ".bin"
+        if b not in bins:
+            bins[b] = ov.Tensor(mapped_bin(os.path.join(d, b)), shared_memory=True)
+        model = _xbin_model(core, d, name, m, bins[b])
+        if npuw:
+            model = _no_bool_constants(model)
+    elif npuw:
         b = m.get("bin") or name[:-4] + ".bin"
         if b not in bins:
             # a file-backed mapping (OpenVINO shares it, see mapped_bin): NPUW's weights bank keeps reading the
@@ -295,6 +343,23 @@ def read_segment(core, d, name, m, npuw, bins, T, built_T):
     return model
 
 
+GROUPED_MIN = int(os.environ.get("ONW_GROUPED_MIN", "2"))   # [2.0] prompts from this long go the grouped-MoE path
+GROUPED_RAM_GB = 24            # [2.0] RAM the grouped-MoE prompt path needs
+
+
+def grouped_wanted(meta):
+    """[2.0] Whether a model with mixer graphs (engine.json "mixer") takes the grouped-MoE prompt path: with >= 24 GB
+    RAM (compiling the 64-token mixers next to the streamed experts' cache froze a 16 GB Lunar Lake); ONW_GROUPED=1 /
+    0 forces it on / off. (chat.block_sizes loads only the 1-token graphs then.)"""
+    env = os.environ.get("ONW_GROUPED")
+    if not meta.get("mixer") or env == "0":
+        return False
+    if env == "1":
+        return True
+    from .chat import total_ram_gb
+    ram = total_ram_gb()
+    return ram is not None and ram >= GROUPED_RAM_GB
+
 CRASHED = "crashed"
 TRANSIENT = "transient: "          # a failure worth trying again next time (out of memory, disk full, interrupted)
 
@@ -916,7 +981,7 @@ class SegmentedModel:
         self.sinks = {l: np.asarray(v, np.float32) for l, v in self.meta.get("attn_sinks", {}).items()}
         self.mrope = rope.get("mrope_section") if rope.get("mrope_interleaved") else None
         self.delta = 0                     # MRoPE: position = slot + delta after an image (set by the caller)
-        sizes = sizes or self.meta["sizes"]
+        sizes = sizes or all_sizes(self.meta)
         config = enable_turbo(core, device, config)        # (see turbo(): on unless ONW_TURBO=0)
         t0 = time.time()
         if os.path.exists(os.path.join(d, "experts.bin")):
@@ -1002,7 +1067,7 @@ class SegmentedModel:
         # The largest block the model HAS, not the ones loaded: the ring is part of every graph's shape, so with the
         # 64-token blocks switched off (Gemma below 24 GB) the 1- and 16-token graphs would otherwise be other graphs -
         # compiled again at every switch, the old blobs pruned (48 ring slots more cost next to nothing)
-        self.ring = self.window + max(self.meta.get("sizes") or sizes) if self.paged and self.window else None
+        self.ring = self.window + max(all_sizes(self.meta) if self.meta.get("sizes") else sizes) if self.paged and self.window else None
         kv_dims, dropped = {}, set()
         self._rope = {}                              # S -> (CPU request: position_ids -> angles, Cos / Sin per angle)
         self._ctx = dict(core=core, d=d, device=device, config=config or {}, npuw=npuw, tried=tried, bins=bins,
@@ -1081,6 +1146,30 @@ class SegmentedModel:
             for p_ in NPUW_SEEN:
                 rmtree(p_)
             NPUW_SEEN.clear()
+        # [2.0] the grouped-MoE prompt path (onw/grouped.py): the mixer-only graphs of the model's "mixer" block size,
+        # compiled here (before the cache prune below, which drops what a load did not touch); ONW_GROUPED=0: off
+        self.mix, self.grouped = None, None
+        mx = self.meta.get("mixer")
+        if mx and not only and not grouped_wanted(self.meta):
+            mx = None
+        if mx and self.E and not only:
+            lst = []
+            for name, m in sorted(self.meta["segments"].items()):
+                if not m.get("mixer_only"):
+                    continue
+                reqs = self._build(name, m)
+                if reqs is None:
+                    print(f"[grouped] no {mx['S']}-token mixer graphs on this NPU / driver: prompts as before", flush=True)
+                    lst = None
+                    break
+                for req, pm_ in reqs:
+                    lst.append((req, pm_, {i.get_any_name() for i in req.model_inputs}))
+            if lst:
+                self.mix = (mx["S"], lst)
+                from . import grouped
+                self.grouped = grouped.make(self)          # (here: the first prompt does not wait for its graphs)
+                print(f"[grouped] prompts: {mx['S']}-token mixer graphs + grouped expert GEMMs ({len(lst)} parts)",
+                      flush=True)
         if only:
             tried.save()
             raise Precompiled()
@@ -1153,6 +1242,18 @@ class SegmentedModel:
         self._needed = {}
         for S, lst in self.segs.items():
             self._bind(S, lst)
+        if self.mix is not None:                       # [2.0] the mixer graphs read / write the same state
+            for req, m, names in self.mix[1]:
+                for n in names:
+                    if n in self._owner:
+                        req.set_tensor(n, self._owner[n])
+                    elif n.startswith("shared.") and n[7:] in self.shared:
+                        req.set_tensor(n, self.shared[n[7:]])
+            later, need = set(), []
+            for _, _, names in reversed(self.mix[1]):
+                need.append(set(later))
+                later |= names
+            self._mix_needed = need[::-1]
         self.exact_truncate = not any(k.startswith("ssm.") for k in self.state)
         # DeltaNet states in two buffers each: a segment reads the current one and writes its new state straight into
         # the other (set_tensor, no copy), then the two swap roles; undoing a verification block swaps back instead of
@@ -1169,6 +1270,14 @@ class SegmentedModel:
         self.S = max(self.segs)
         import threading
         self._mutex = threading.RLock()                # a block size is swapped in only between two blocks
+        # [2.0] MoE decoding with the 1-token requests wired together once (onw/wired.py); ONW_WIRED=0: off
+        self._wired = None
+        if os.environ.get("ONW_WIRED", "1") != "0":
+            from .wired import Wired
+            if Wired.usable(self):
+                self._wired = Wired(self)
+                print(f"[wired] 1-token steps: {len(self._wired.plan)} parts wired once "
+                      f"({self._wired.copies} carried inputs copied)", flush=True)
         self._lazy, self._loading = {S: True for S in deferred}, {}
         # attention sinks per paged layer, in the [kv, G] layout of the paged cache
         self._sink = {}
@@ -2030,6 +2139,9 @@ class SegmentedModel:
 
     def prefill(self, ids, embeds=None, pos=None, want_logits=True, stop=None):
         with self._mutex:
+            if self.mix is not None and len(ids) >= GROUPED_MIN:      # [2.0] the grouped-MoE prompt path
+                from . import grouped
+                return grouped.prefill(self, ids, embeds, pos, want_logits, stop)
             return self._prefill_layers(ids, embeds, pos, want_logits, stop)
 
     def _step(self, ids, embeds=None, pos=None, want_logits=True, size=None):
@@ -2038,7 +2150,11 @@ class SegmentedModel:
         MRoPE positions (default: slot + delta on all rows). want_logits=False skips the LM head segment (prompt
         blocks whose logits are not needed) and returns None. size: this block size (default: the smallest that
         holds the block)."""
+        if self._wired is not None and len(ids) == 1 and embeds is None and pos is None and size in (None, 1):
+            return self._wired.step(int(ids[0]), want_logits)
         S, n, x, common = self._block_inputs(ids, embeds, pos, S=size)
+        if S == 1 and self._wired is not None:
+            self._wired.dirty = True                  # (this path binds its own arrays into the 1-token requests)
         self._tok += 1
         # what truncate() needs to undo part of a multi-token (verification) block
         self._last = {"start": self.cur, "cx": {}, "rows": {},

onw/web/manage.html

@@ -256,7 +256,7 @@
       </div>
       <div class="opt"><span class="k">エキスパートのメモリ</span><select id="f-expert_gb" style="width:220px"><option value="auto">auto（足りないときだけ SSD から）</option><option value="off">すべてメモリに置く</option><option value="2">2 GB</option><option value="3">3 GB</option><option value="4">4 GB</option><option value="5">5 GB</option><option value="6">6 GB</option><option value="8">8 GB</option></select>
         <span class="desc">Qwen3.6 系の MoE 用。上限を超えるエキスパートは SSD に置いたまま必要なときだけ読みます（動きますが遅くなります）。</span></div>
-      <div class="opt"><span class="k">64 トークン処理<span class="rc" title="変えるとコンパイルし直します">※</span></span><select id="f-big_blocks" style="width:300px"><option value="auto">auto（Lunar Lake 以降か RAM 24 GB 以上でオン）</option><option value="on">on</option><option value="off">off</option></select>
+      <div class="opt"><span class="k">64 トークン処理<span class="rc" title="変えるとコンパイルし直します">※</span></span><select id="f-big_blocks" style="width:300px"><option value="auto">auto（オン。NPU 3720 などでは 16 トークン処理と入れ替え）</option><option value="on">on</option><option value="off">off</option></select>
         <span class="desc">長いプロンプトの処理が速くなる代わりに、そのぶんメモリを使います（Gemma）。</span>
         <span class="desc">※ 64 トークン処理は on にしたときだけ（off にするのはすぐです）。</span></div>
     </div>

onw/wired.py

@@ -0,0 +1,142 @@
+"""[2.0] Decoding (1-token steps) of a MoE model with the requests wired together once ("prepared runs", the onwx
+decode idea in OpenVINO terms): each part's carried outputs are bound as the next part's inputs at setup (the same
+NPU-visible tensors - no host copy, no per-call input dicts); per token the host only writes the embedding row and the
+rope tables, reads the router's top-k, rebinds the expert slots, runs the paged attention and swaps the DeltaNet state
+buffers; the conv-window bookkeeping of part i runs while part i + 1 is on the NPU. Same tokens as
+SegmentedModel._step (NPU 3720, Qwen3.6-35B-A3B: 146.5 -> 139.5 ms a token). ONW_WIRED=0: off."""
+import numpy as np
+from .store import host_tensor
+
+COMMON = ("inputs_embeds", "rope_cos", "rope_sin", "position_ids", "token_valid", "mask")
+
+
+class Wired:
+    def __init__(self, m):
+        self.m = m
+        self.parts = m.segs[1]
+        mk = host_tensor(m._ctx["core"], m._ctx["device"])
+        self.common, self.plan, producer, self.copies = {}, [], {}, 0
+        for i, (req, meta, names) in enumerate(self.parts):
+            outs = {o.get_any_name() for o in req.model_outputs}
+            ins = {p.get_any_name(): p for p in req.model_inputs}
+            st = {"req": req, "meta": meta, "copy": [], "pp": [], "route_from": None, "bind": []}
+            for n, p in ins.items():
+                own = lambda p=p: mk(p.get_element_type(), list(p.get_shape()))
+                if n.startswith("carry.") and not n.startswith("carry.attn."):
+                    j, on = producer[n]
+                    src = self.parts[j][0].get_tensor(on)
+                    if src.get_element_type() == p.get_element_type() and list(src.get_shape()) == list(p.get_shape()):
+                        st["bind"].append((n, src))          # wired: no copy, no host work a token
+                    else:
+                        st["copy"].append((n, j, on))
+                        st["bind"].append((n, own()))
+                        self.copies += 1
+                elif n in COMMON or n.startswith("hostrope."):
+                    if n not in self.common:
+                        self.common[n] = own()
+                    st["bind"].append((n, self.common[n]))
+                elif n.startswith("carry.attn.") or n == "route.slot_w":
+                    st["bind"].append((n, own()))
+                elif n in m._pp:
+                    st["pp"].append(n)
+            if meta.get("consume_moe") is not None:
+                st["route_from"] = producer["route"]
+            for on in outs:
+                if on.startswith("next."):
+                    producer["carry." + on[5:]] = (i, on)
+            if "route.idx" in outs:
+                producer["route"] = i
+            st["cx"] = [o for o in outs if o.startswith("cx.")]
+            st["ssm"] = [o for o in outs if o.startswith("ssm_new.")]
+            st["kv"] = meta.get("paged_attn")
+            st["kvw"] = st["kv"] is not None and f"new_k.{st['kv']}" in outs
+            st["head"] = "logits" in outs
+            self.plan.append(st)
+        self.dirty = True                                # (bind() before the first step)
+
+    def bind(self):
+        """(again after the stock path ran these requests with its own input arrays)"""
+        for st in self.plan:
+            for n, t in st["bind"]:
+                st["req"].set_tensor(n, t)
+        self.dirty = False
+
+    def step(self, tok, want_logits=True):
+        m = self.m
+        if self.dirty:
+            self.bind()
+        _, _, x, common = m._block_inputs([tok], None, None, S=1)
+        for k, t in self.common.items():
+            t.data[:] = x if k == "inputs_embeds" else common[k]
+        m._tok += 1
+        m._last = {"start": m.cur, "cx": {}, "rows": {}, "prev": None, "pp": None}
+        cur, later, logits = m.cur, None, None
+        for pi, st in enumerate(self.plan):
+            req, meta = st["req"], st["meta"]
+            if st["head"] and not want_logits:
+                continue
+            for n_, j, on in st["copy"]:
+                req.get_tensor(n_).data[:] = self.parts[j][0].get_tensor(on).data
+            if st["route_from"] is not None:
+                pr = self.parts[st["route_from"]][0]
+                idx, w = pr.get_tensor("route.idx").data, pr.get_tensor("route.w").data
+                layer = meta["consume_moe"]
+                m._bound = (m._tok, layer)
+                keys = [(layer, int(e)) for e in idx[0]]
+                pairs = m.bank.get_many(keys) if m.streaming else [m.bank[k] for k in keys]
+                for j, (w_t, s_t) in enumerate(pairs):
+                    req.set_tensor(f"w_{j}", w_t)
+                    req.set_tensor(f"s_{j}", s_t)
+                req.get_tensor("route.slot_w").data[0] = w[0]
+            for k in st["pp"]:
+                bufs, c = m._pp[k]
+                req.set_tensor(k, bufs[c])
+                req.set_tensor("ssm_new." + k[4:], bufs[1 - c])
+            req.start_async()
+            if later is not None:
+                later()
+                later = None
+            req.wait()
+            L = st["kv"]
+            if L is not None:                             # paged attention: rows into the cache, attention on the NPU
+                if st["kvw"]:                             # (a layer sharing another's cache only reads it)
+                    m.kv.write(L, cur, req.get_tensor(f"new_k.{L}").data[0, :, :1], req.get_tensor(f"new_v.{L}").data[0, :, :1])
+                q = req.get_tensor(f"q.{L}").data
+                self.parts[pi + 1][0].get_tensor(f"carry.attn.{L}").data[:] = m.kv.attend(
+                    L, q[0], cur, 1, 1, sink=m._sink.get(L), rows_out=True)
+
+            def book(st=st, req=req):
+                for o in st["cx"]:
+                    v = req.get_tensor(o).data
+                    s_ = m.state["conv." + o[3:]]
+                    s_[:] = np.concatenate([s_, v[:, :1]], 1)[:, -s_.shape[1]:]
+                for o in st["ssm"]:
+                    e = m._pp["ssm." + o[8:]]
+                    e[1] = 1 - e[1]
+                    m.state["ssm." + o[8:]] = e[0][e[1]].data
+            later = book
+            if st["head"]:
+                logits = np.array(req.get_tensor("logits").data[0, :1])
+        if later is not None:
+            later()
+        m.cur += 1
+        if logits is not None:
+            if not np.isfinite(logits).all():
+                raise FloatingPointError("the 1-token graph returned NaN/inf logits")
+            if m.banned:
+                logits[:, m.banned] = -np.inf
+        return logits
+
+    @staticmethod
+    def usable(m):
+        """wiring needs: a MoE model with paged attention, 1-token graphs, DeltaNet states in two buffers (the stock
+        path's every other input kind is unknown here)"""
+        if not (m.E and m.paged and 1 in m.segs and m.mtp is None and not m.predict_k):
+            return False
+        known = set(COMMON) | {"route.slot_w"}
+        for req, meta, names in m.segs[1]:
+            for n in names:
+                if not (n in known or n.startswith(("carry.", "hostrope.", "w_", "s_", "shared.")) or n in m._pp
+                        or n in m._owner):
+                    return False
+        return True