onw 0.4.1: Japanese README (README.md) + English (README_en.md); setup/start call find_python.bat by path

Browse files

Files changed (6) hide show

README.md +152 -145
README_en.md +136 -0
onw/__init__.py +1 -1
pyproject.toml +1 -1
setup.bat +44 -44
start.bat +52 -52

README.md CHANGED Viewed

@@ -1,145 +1,152 @@
----
-license: apache-2.0
-library_name: openvino
-tags:
-  - openvino
-  - npu
-  - intel
-  - intel-npu
-  - moe
-  - inference-engine
----
-# onw — 俺のNPUがこんなに動くわけない
-*(Ore no NPU ga konna ni ugoku wake nai — "There's no way my NPU runs this well")*
-A small engine that runs LLMs — including MoE and hybrid DeltaNet models, with image input — **entirely on the
-Intel NPU**, with an OpenAI-compatible server and a browser chat UI. Engine and models are separate downloads,
-like llama.cpp + a GGUF: one engine, one command, any converted model.
-```bash
-hf download ryugyosoft/onw --local-dir onw && cd onw
-setup.bat                                    # Windows: resident task-tray app (recommended)
-bash setup.sh                                # Ubuntu: resident top-bar app
-start.bat                                    # or just run the server once (default model: LFM2-8B-A1B)
-bash start.sh ryugyosoft/Qwen3.6-35B-A3B-onw # Ubuntu, another model
-```
-`setup` finds Python 3.10+ (on Windows it offers to install Python 3.12 with winget when there is none),
-installs the engine and the `onw` command into `.venv`, adds **onw** to the Start menu / application list and
-starts a small resident app (task tray on Windows, top-bar indicator on Ubuntu). No model is downloaded yet:
-the **onw window** opens on its *Models* tab - pick a model (sizes, memory needs and what fits this PC are shown)
-and press *Download*. Then the *Server* tab starts it; the first start of a model compiles it for the NPU, later
-starts take ~20 s. The *Settings* tab has port, LAN access, API key, context length, device, prompt lookup
-decoding, 64-token blocks, start at login and the models folder.
-**The server is an OpenAI-compatible API**: point other apps at `http://localhost:8000/v1` with the model name
-shown in the window (`GET /` describes the server, `/health` answers once the model is loaded, CORS is open,
-`--api-key` / the settings' API key protects `/v1`). For trying things out it also serves a chat UI at `/chat` and
-a check page at `/check` that runs a text, prompt-lookup, two-turn and image test and shows the speeds; both open
-from the tray menu and the window. Without an NPU driver everything runs on the CPU.
-## The `onw` command
-```bash
-onw serve MODEL [--port 8000] [--host 0.0.0.0] [--api-key KEY] [--context 4096] [--device NPU|CPU] [--no-pld] [--open]
-onw chat MODEL [--think] [--context 4096]                     # chat in the terminal
-onw tray                                                      # the resident app (what setup starts)
-onw window [--tab models]                                     # model manager, server control, settings
-onw convert HF_DIR OUT_DIR [--compact] [--prune SAL.json --keep 0.5] [--graphs-only]
-onw list                                                      # supported architectures
-```
-What needs what:
-| setting | how | recompile? |
-|---|---|---|
-| temperature, top_p, top_k, repetition / presence / frequency penalty, max_tokens, stop, seed, thinking | per request (API or chat UI) | no |
-| port, address, model, device, prompt lookup on/off | server option / tray settings, restart (~20 s) | no |
-| **context length** (`--context`) | server option / tray settings | **once per length** (input shapes change; the model files stay the same) |
-| 64-token prompt blocks (Gemma) | `ONW_S64=1/0` / tray setting | once, the first time |
-| block sizes, image size, quantization | `onw convert` | yes (new conversion) |
-A longer context costs memory and time on every token (the static KV cache is attended in full), so pick what you
-need rather than the model's maximum.
-`MODEL` is an onw model folder or a Hugging Face repo id (downloaded into `./models`, resumable). The command is in
-`.venv` after the first `start` (`.venv\Scripts\onw` on Windows, `source .venv/bin/activate` on Linux), or
-install it anywhere with `pip install --pre -e . --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly`.
-Git LFS guard: a model repo cloned with `git` but without Git LFS contains ~130-byte pointer files instead of the
-weights; `onw` detects that and stops with instructions. Prefer `hf download` (or just pass the repo id).
-## Models
-| onw model | arch | input | NPU 3720 decode | download | memory in use |
-|---|---|---|---|---|---|
-| [ryugyosoft/LFM2-8B-A1B-onw](https://huggingface.co/ryugyosoft/LFM2-8B-A1B-onw) | `lfm2_moe` (short conv + GQA + MoE 32x top-4) | text | 16-17 tok/s (29-43 with prompt lookup on code / summaries) | 4.0 GB | ~6 GB |
-| [ryugyosoft/gemma-4-E4B-it-onw](https://huggingface.co/ryugyosoft/gemma-4-E4B-it-onw) | `gemma4` (imported from the validated Gemma NPU graphs) | text + image | 7.2 tok/s | 4.3 GB | ~15 GB |
-| [ryugyosoft/Qwen3.5-9B-onw](https://huggingface.co/ryugyosoft/Qwen3.5-9B-onw) | `qwen3_5` (Gated DeltaNet + gated attention, dense) | text + image | 4.1 tok/s | 5.3 GB | ~20 GB |
-| [ryugyosoft/Qwen3.6-REAP-18B-A3B-onw](https://huggingface.co/ryugyosoft/Qwen3.6-REAP-18B-A3B-onw) | `qwen3_5_moe`, 128 of 256 experts kept (REAP) | text + image | 6.8 tok/s | 9.6 GB | ~15 GB |
-| [ryugyosoft/Qwen3.6-35B-A3B-onw](https://huggingface.co/ryugyosoft/Qwen3.6-35B-A3B-onw) | `qwen3_5_moe` (Gated DeltaNet + gated attention + MoE 256x top-8 + shared expert) | text + image | 6.8 tok/s | 17 GB | ~23 GB |
-Memory is the process working set on the test machine after loading (includes memory-mapped weight files the
-OS can page out).
-Measured with an image chat / text prompts on Core Ultra 9 285HX; prompt lookup decoding is used automatically
-where a 16-token verification block is cheap enough (LFM2, Qwen3.5-9B, Gemma), and every model keeps the
-conversation state between turns (a follow-up with an image re-processes only the new message: ~2 s).
-## Smaller models (onw convert --compact / --prune)
-- `--compact`: the 1-token and 16/64-token graphs of a segment share one weights file (download / disk);
-  host-only lookup tables (token embeddings not tied to the LM head, Gemma's per-layer embeddings) are INT4 with a
-  scale per 32 values; the Qwen vision tower is INT8. These also shrink the memory in use, except the shared
-  weights file: the NPU keeps one compiled copy per block size.
-- `--prune SAL.json --keep 0.5` (MoE): REAP expert pruning. `python -m onw.prune calibrate HF_DIR SAL.json`
-  measures each expert's saliency (mean router weight x output norm over chat data in Japanese, English and code),
-  `python -m onw.prune eval` reports the perplexity for several keep ratios. Qwen3.6-35B-A3B keeps 128 of 256
-  experts at +8.9% perplexity (published as Qwen3.6-REAP-18B-A3B-onw); LFM2 with only 32 experts loses +35% at
-  50%, so it is not pruned.
-## How it runs a MoE on an NPU
-NPUs need static graphs, and a sparse MoE picks different experts per token. Measured on NPU 3720 (Qwen3.6
-sizes, per layer, one token): computing every expert 45 ms; selecting experts inside the graph with Gather does
-not compile for INT4 constants and is 18-135 ms for INT8 / packed forms (the NPU moves whole tables);
-**passing the chosen experts as runtime inputs: 1.7 ms**, at the NPU's memory bandwidth. So:
-- the network is cut into **segments right after every MoE router**; the router's top-k goes to the host, which
-  binds the selected experts' weight buffers into the next segment (`set_tensor`, zero copy: all experts live in
-  NPU-visible host memory, `experts.bin`);
-- experts are **channel-wise INT4 packed as one tensor per expert** — the only weights-as-inputs form the NPU runs
-  at full speed (group-wise scales on runtime weights fall back to slow element-wise work, 20x slower);
-- everything else is group-128 INT4 constants inside the segments; the LM head is a shared INT8 input in its own
-  segment (one copy for all block sizes, skipped for prompt blocks that need no logits); token embeddings and
-  Gemma's 2.7 GB per-layer embeddings are looked up on the host (a gather from a runtime input makes the NPU move
-  the whole table);
-- prompt blocks bind the union of the block's experts; decoding binds the token's top-k.
-## Engine pieces
-| file | what |
-|---|---|
-| `onw/ir.py` | graph blocks: quantized linears, RMSNorm (with an fp16-safe prescale), RoPE / partial RoPE, attention over a host KV cache, expert slots, Gated DeltaNet (1-token matrix form; chunkwise-parallel form with an exact block-doubling inverse) |
-| `onw/runtime.py` | runs the segments; state by naming convention (KV rows, conv windows, recurrent states), expert binding, MRoPE, sliding-window masks, host table lookups, exact truncation for speculative verification |
-| `onw/chat.py` | chat loop: images (Qwen / Gemma), sampling, stop strings, prompt lookup decoding (exact rewind, or lazy redo for recurrent models, switched on only when it pays), prefix reuse between turns |
-| `onw/models/` | one builder per architecture: `lfm2_moe`, `qwen3_5_moe` (dense and MoE), `qwen_vision`, `gemma4` |
-| `onw/hub.py`, `onw/server.py`, `onw/web/` | model download + LFS guard, OpenAI-compatible server, chat UI (`/chat`), check page (`/check`) |
-| `onw/app.py`, `onw/tray.py`, `setup.*`, `find_python.bat` | onw window (models / server / settings), tray icon, installers (Python detection + winget) |
-| `check_ref.py`, `check_vision.py`, `test_chat.py`, `test_image.py`, `probe_segments.py`, `prof_*.py` | vs the HF model, generation / image / multi-turn checks, NPU-vs-CPU per segment, timing |
-## Notes and limitations
-- Static 1024-token context. Qwen images are resized to 512x512 (256 tokens); Gemma images keep their aspect
-  ratio (up to 280 tokens).
-- Experts are channel-wise INT4 (round-to-nearest): first-token logits within ~33% of bf16 with the same top-1 on
-  our checks; answers are fluent, word choices diverge from the bf16 model after a few tokens.
-- Prompt blocks above 16 tokens (Gemma's 64-token block) are loaded only with >= 24 GB RAM (`ONW_S64=1/0`).
-- Needs a pre-release OpenVINO (2026.5 nightly). Tested on NPU 3720 (Windows 11); CPU fallback on Ubuntu 22.04
-  (WSL). NPU 4000 (Lunar Lake) untested with this engine.
-## License
-Code: Apache 2.0. Converted models carry their base model's license (see each model repo).

+---
+license: apache-2.0
+library_name: openvino
+language:
+  - ja
+  - en
+tags:
+  - openvino
+  - npu
+  - intel
+  - intel-npu
+  - moe
+  - inference-engine
+---
+# onw — 俺のNPUがこんなに動くわけない
+**日本語** | [English](README_en.md)
+LLM を **Intel NPU だけで** 動かす小さなエンジンです。MoE（専門家混合）モデルや Gated DeltaNet 系のハイブリッドモデル、
+画像入力にも対応し、**OpenAI 互換 API サーバー**とブラウザのチャット画面を備えています。
+llama.cpp と GGUF の関係と同じく、エンジンとモデルは別々にダウンロードします。エンジンは一つ、コマンドも一つで、
+変換済みのモデルならどれでも動きます。
+```bash
+hf download ryugyosoft/onw --local-dir onw && cd onw
+setup.bat                                    # Windows：タスクトレイ常駐アプリ（おすすめ）
+bash setup.sh                                # Ubuntu：右上に常駐するアプリ
+start.bat                                    # サーバーを一度だけ起動する場合（既定モデル：LFM2-8B-A1B）
+bash start.sh ryugyosoft/Qwen3.6-35B-A3B-onw # Ubuntu で別のモデルを起動
+```
+`setup` は Python 3.10 以上を探し（Windows で見つからない場合は winget で Python 3.12 を入れるか確認します）、
+`.venv` にエンジンと `onw` コマンドを入れ、スタートメニュー／アプリ一覧に **onw** を登録して常駐アプリを起動します
+（Windows はタスクトレイ、Ubuntu は右上のインジケーター）。この時点ではモデルはまだ入っていません。
+開いた **onw ウィンドウ**の「モデル」タブで、サイズ・必要メモリ・この PC で足りるかを見てモデルを選び、
+「ダウンロード」を押します。あとは「サーバー」タブから起動します。モデルの初回起動時だけ NPU 向けのコンパイルが
+走り、2 回目以降は約 20 秒で起動します。「設定」タブでは、ポート、LAN からの接続、API キー、コンテキスト長、デバイス、
+先読み検証（PLD）、64 トークン処理、ログイン時の自動起動、モデルの保存先を変えられます。
+**サーバーは OpenAI 互換 API です。** 他のアプリからは base URL に `http://localhost:8000/v1`、モデル名にウィンドウに
+表示される名前を指定します。`GET /` でサーバーの案内（JSON）、`/health` でモデルの読み込み完了を確認できます。
+CORS は開放済みで、LAN に公開する場合は API キー（`--api-key` または設定画面）で `/v1` を保護できます。
+動作確認用に、チャット画面（`/chat`）と、テキスト・先読み検証・2 ターン会話・画像の検証を一括実行して速度を表に
+する検証ページ（`/check`）もあり、どちらもトレイのメニューとウィンドウから開けます。
+NPU ドライバーがない環境では CPU で動きます。
+## `onw` コマンド
+```bash
+onw serve MODEL [--port 8000] [--host 0.0.0.0] [--api-key KEY] [--context 4096] [--device NPU|CPU] [--no-pld] [--open]
+onw chat MODEL [--think] [--context 4096]                     # ターミナルでチャット
+onw tray                                                      # 常駐アプリ（setup が起動するもの）
+onw window [--tab models]                                     # モデル管理・サーバー操作・設定のウィンドウ
+onw convert HF_DIR OUT_DIR [--compact] [--prune SAL.json --keep 0.5] [--graphs-only]
+onw list                                                      # 対応アーキテクチャ
+```
+設定を変えたときに何が必要か：
+| 設定 | 変え方 | 再コンパイル |
+|---|---|---|
+| temperature、top_p、top_k、繰り返し抑制（repetition / presence / frequency penalty）、max_tokens、stop、seed、思考モード | リクエストごと（API またはチャット画面） | 不要 |
+| ポート、待ち受けアドレス、モデル、デバイス、先読み検証のオン／オフ | サーバーのオプション／設定画面、再起動（約 20 秒） | 不要 |
+| **コンテキスト長**（`--context`） | サーバーのオプション／設定画面 | **長さごとに 1 回**（入力の形が変わるだけで、モデルファイルはそのまま） |
+| 64 トークン処理（Gemma） | `ONW_S64=1/0`／設定画面 | 初回のみ |
+| ブロックサイズ、画像サイズ、量子化 | `onw convert` | 必要（変換し直し） |
+コンテキスト長を長くすると、短い会話でも 1 トークンごとの計算とメモリが増えます（固定長の KV キャッシュ全体を
+��回参照するため）。モデルの最大値ではなく、必要な長さを選んでください。
+`MODEL` には onw 形式のモデルフォルダか Hugging Face のリポジトリ ID を指定します（ID の場合は `./models` に
+ダウンロードされ、途中から再開できます）。`onw` コマンドは初回の `setup`／`start` のあと `.venv` に入ります
+（Windows は `.venv\Scripts\onw`、Linux は `source .venv/bin/activate`）。任意の環境に入れる場合は
+`pip install --pre -e . --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly` です。
+Git LFS への対策：Git LFS なしで `git clone` したモデルには、重みの代わりに約 130 バイトの「ポインタ」ファイルしか
+入っていません。`onw` はこれを検出し、直し方を表示して止まります。`hf download` を使うか、リポジトリ ID をそのまま
+渡してください。
+## モデル
+| onw モデル | アーキテクチャ | 入力 | NPU 3720 生成速度 | ダウンロード | 使用メモリ |
+|---|---|---|---|---|---|
+| [ryugyosoft/LFM2-8B-A1B-onw](https://huggingface.co/ryugyosoft/LFM2-8B-A1B-onw) | `lfm2_moe`（短い畳み込み＋GQA＋MoE 32 個から 4 個） | テキスト | 16〜17 tok/s（コード修正・要約では先読み検証で 29〜43） | 4.0 GB | 約 6 GB |
+| [ryugyosoft/gemma-4-E4B-it-onw](https://huggingface.co/ryugyosoft/gemma-4-E4B-it-onw) | `gemma4`（検証済みの Gemma NPU グラフを取り込み） | テキスト＋画像 | 7.2 tok/s | 4.3 GB | 約 15 GB |
+| [ryugyosoft/Qwen3.5-9B-onw](https://huggingface.co/ryugyosoft/Qwen3.5-9B-onw) | `qwen3_5`（Gated DeltaNet＋ゲート付きアテンション、dense） | テキスト＋画像 | 4.1 tok/s | 5.3 GB | 約 20 GB |
+| [ryugyosoft/Qwen3.6-REAP-18B-A3B-onw](https://huggingface.co/ryugyosoft/Qwen3.6-REAP-18B-A3B-onw) | `qwen3_5_moe`、256 個中 128 個のエキスパートを残す（REAP） | テキスト＋画像 | 6.8 tok/s | 9.6 GB | 約 15 GB |
+| [ryugyosoft/Qwen3.6-35B-A3B-onw](https://huggingface.co/ryugyosoft/Qwen3.6-35B-A3B-onw) | `qwen3_5_moe`（Gated DeltaNet＋ゲート付きアテンション＋MoE 256 個から 8 個＋共有エキスパート） | テキスト＋画像 | 6.8 tok/s | 17 GB | 約 23 GB |
+使用メモリは、テスト機（Core Ultra 9 285HX）で読み込んだあとのプロセスの作業セットです（OS が必要に応じて手放せる、
+メモリマップされた重みファイル分を含みます）。先読み検証は、16 トークンの検証ブロックが十分安く済むモデル
+（LFM2、Qwen3.5-9B、Gemma）で自動的に使われます。どのモデルも会話の状態をターン間で保持するので、画像付きの会話でも
+2 ターン目は新しいメッセージだけを処理します（約 2 秒）。
+## モデルを小さくする（onw convert --compact / --prune）
+- `--compact`：1 トークン用と 16／64 トークン用のグラフが同じ重みファイルを共有します（ダウンロード量・ディスク）。
+  ホスト側で引くだけの表（出力層と共有していない埋め込み、Gemma の層ごとの埋め込み）は 32 要素ごとのスケール付き
+  INT4 に、Qwen の画像エンコーダーは INT8 にします。これらは使用メモリも減らしますが、重みファイルの共有だけは
+  ディスクにしか効きません（NPU はブロックサイズごとにコンパイル済みの重みを持つため）。
+- `--prune SAL.json --keep 0.5`（MoE）：REAP によるエキスパートの刈り込み。`python -m onw.prune calibrate HF_DIR SAL.json`
+  で各エキスパートの寄与度（日本語・英語・コードのチャットデータでの「ルーターの重み×出力の大きさ」の平均）を測り、
+  `python -m onw.prune eval` で残す割合ごとのパープレキシティを確認できます。Qwen3.6-35B-A3B は 256 個中 128 個を
+  残してパープレキシティ +8.9%（Qwen3.6-REAP-18B-A3B-onw として公開）。エキスパートが 32 個しかない LFM2 は
+  半分で +35% と落ち方が大きいため、刈り込んでいません。
+## MoE を NPU で動かす仕組み
+NPU は形の決まったグラフしか扱えませんが、MoE はトークンごとに違うエキスパートを選びます。NPU 3720 で Qwen3.6 の
+寸法の 1 層・1 トークンを測ると、全エキスパートを計算すると 45 ms、グラフ内の Gather で選ぶ方式は INT4 定数では
+コンパイルできず、INT8 や詰め込み形式でも 18〜135 ms（NPU が表全体を転送するため）でした。
+**選んだエキスパートを実行時の入力として渡す方式なら 1.7 ms** で、NPU のメモリ帯域いっぱいに出ます。そこで：
+- ネットワークを **MoE のルーターの直後で区間に分け**、ルーターの上位 k 個をホストが受け取って、次の区間に選ばれた
+  エキスパートの��みバッファを結び付けます（`set_tensor`、コピーなし。全エキスパートは NPU から見えるホストメモリ上の
+  `experts.bin` にあります）。
+- エキスパートは **チャネル単位 INT4 を 1 エキスパート 1 テンソルに詰めた形** にします。実行時に渡す重みを NPU が
+  全速で扱えるのはこの形だけです（グループ単位のスケールを付けると要素ごとの遅い処理になり、20 倍遅くなります）。
+- それ以外は区間の中のグループ 128 の INT4 定数です。出力層は共有の INT8 入力として独立した区間に置き（ブロック
+  サイズが違っても 1 つだけ、ロジットが不要なプロンプト処理では飛ばす）、埋め込みと Gemma の 2.7 GB の層ごとの
+  埋め込みはホストで引きます（実行時入力からの Gather は NPU が表全体を転送するため）。
+- プロンプトのブロックではブロック内の全トークンが選んだエキスパートの和集合を、生成時はそのトークンの上位 k 個を
+  結び付けます。
+## エンジンの構成
+| ファイル | 内容 |
+|---|---|
+| `onw/ir.py` | グラフ部品：量子化線形層、RMSNorm（FP16 で溢れない前処理つき）、RoPE／部分 RoPE、ホスト管理の KV キャッシュ上のアテンション、エキスパート枠、Gated DeltaNet（1 トークン用の行列形式、ブロック倍々の厳密な逆行列によるチャンク並列形式） |
+| `onw/runtime.py` | 区間の実行、命名規則による状態管理（KV の行、畳み込み窓、再帰状態）、エキスパートの結び付け、MRoPE、スライディング窓マスク、ホスト側の表引き、先読み検証のための正確な巻き戻し |
+| `onw/chat.py` | 生成ループ：画像（Qwen／Gemma）、サンプリング、停止文字列、先読み検証（正確な巻き戻し、再帰状態を持つモデルでは遅延やり直し、効く場合だけ自動で有効）、ターン間の差分処理 |
+| `onw/models/` | アーキテクチャごとの組み立て定義：`lfm2_moe`、`qwen3_5_moe`（dense と MoE）、`qwen_vision`、`gemma4` |
+| `onw/hub.py`、`onw/server.py`、`onw/web/` | モデルのダウンロードと LFS 対策、OpenAI 互換サーバー、チャット画面（`/chat`）、検証ページ（`/check`） |
+| `onw/app.py`、`onw/tray.py`、`setup.*`、`find_python.bat` | onw ウィンドウ（モデル／サーバー／設定）、トレイアイコン、インストーラー（Python の検出と winget） |
+| `check_ref.py`、`check_vision.py`、`test_chat.py`、`test_image.py`、`probe_segments.py`、`prof_*.py` | 元の HF モデルとの比較、生成・画像・複数ターンの検証、区間ごとの NPU と CPU の比較、速度計測 |
+## 注意点と制限
+- コンテキスト長の既定は 1024 トークンです。`--context`／設定画面で読み込み時に変えられます（長さごとに NPU 向け
+  コンパイルが 1 回走ります）。Qwen の画像は 512×512（256 トークン）に縮小し、Gemma の画像は縦横比を保ちます
+  （最大 280 トークン）。
+- エキスパートはチャネル単位 INT4（四捨五入量子化）です。確認した範囲では、最初のトークンのロジットは bf16 との差が
+  約 33% で、1 位の予測は一致しました。回答は自然ですが、数トークン先から言い回しが bf16 モデルと分かれます。
+- 16 トークンより大きいプロンプトブロック（Gemma の 64 トークン版）は RAM 24 GB 以上のときだけ読み込みます
+  （`ONW_S64=1/0` で強制）。
+- OpenVINO のプレリリース版（2026.5 nightly）が必要です。NPU 3720（Windows 11）で動作確認済み、CPU での動作は
+  Ubuntu 22.04（WSL）で確認しています。NPU 4000（Lunar Lake）と Ubuntu の右上常駐は、このエンジンではまだ未確認です。
+## ライセンス
+コードは Apache 2.0。変換済みモデルは元のモデルのライセンスに従います（各モデルのリポジトリを参照）。

README_en.md ADDED Viewed

	@@ -0,0 +1,136 @@

+# onw — 俺のNPUがこんなに動くわけない
+[日本語](README.md) | **English**
+*(Ore no NPU ga konna ni ugoku wake nai — "There's no way my NPU runs this well")*
+A small engine that runs LLMs — including MoE and hybrid DeltaNet models, with image input — **entirely on the
+Intel NPU**, with an OpenAI-compatible server and a browser chat UI. Engine and models are separate downloads,
+like llama.cpp + a GGUF: one engine, one command, any converted model.
+```bash
+hf download ryugyosoft/onw --local-dir onw && cd onw
+setup.bat                                    # Windows: resident task-tray app (recommended)
+bash setup.sh                                # Ubuntu: resident top-bar app
+start.bat                                    # or just run the server once (default model: LFM2-8B-A1B)
+bash start.sh ryugyosoft/Qwen3.6-35B-A3B-onw # Ubuntu, another model
+```
+`setup` finds Python 3.10+ (on Windows it offers to install Python 3.12 with winget when there is none),
+installs the engine and the `onw` command into `.venv`, adds **onw** to the Start menu / application list and
+starts a small resident app (task tray on Windows, top-bar indicator on Ubuntu). No model is downloaded yet:
+the **onw window** opens on its *Models* tab - pick a model (sizes, memory needs and what fits this PC are shown)
+and press *Download*. Then the *Server* tab starts it; the first start of a model compiles it for the NPU, later
+starts take ~20 s. The *Settings* tab has port, LAN access, API key, context length, device, prompt lookup
+decoding, 64-token blocks, start at login and the models folder.
+**The server is an OpenAI-compatible API**: point other apps at `http://localhost:8000/v1` with the model name
+shown in the window (`GET /` describes the server, `/health` answers once the model is loaded, CORS is open,
+`--api-key` / the settings' API key protects `/v1`). For trying things out it also serves a chat UI at `/chat` and
+a check page at `/check` that runs a text, prompt-lookup, two-turn and image test and shows the speeds; both open
+from the tray menu and the window. Without an NPU driver everything runs on the CPU.
+## The `onw` command
+```bash
+onw serve MODEL [--port 8000] [--host 0.0.0.0] [--api-key KEY] [--context 4096] [--device NPU|CPU] [--no-pld] [--open]
+onw chat MODEL [--think] [--context 4096]                     # chat in the terminal
+onw tray                                                      # the resident app (what setup starts)
+onw window [--tab models]                                     # model manager, server control, settings
+onw convert HF_DIR OUT_DIR [--compact] [--prune SAL.json --keep 0.5] [--graphs-only]
+onw list                                                      # supported architectures
+```
+What needs what:
+| setting | how | recompile? |
+|---|---|---|
+| temperature, top_p, top_k, repetition / presence / frequency penalty, max_tokens, stop, seed, thinking | per request (API or chat UI) | no |
+| port, address, model, device, prompt lookup on/off | server option / tray settings, restart (~20 s) | no |
+| **context length** (`--context`) | server option / tray settings | **once per length** (input shapes change; the model files stay the same) |
+| 64-token prompt blocks (Gemma) | `ONW_S64=1/0` / tray setting | once, the first time |
+| block sizes, image size, quantization | `onw convert` | yes (new conversion) |
+A longer context costs memory and time on every token (the static KV cache is attended in full), so pick what you
+need rather than the model's maximum.
+`MODEL` is an onw model folder or a Hugging Face repo id (downloaded into `./models`, resumable). The command is in
+`.venv` after the first `start` (`.venv\Scripts\onw` on Windows, `source .venv/bin/activate` on Linux), or
+install it anywhere with `pip install --pre -e . --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly`.
+Git LFS guard: a model repo cloned with `git` but without Git LFS contains ~130-byte pointer files instead of the
+weights; `onw` detects that and stops with instructions. Prefer `hf download` (or just pass the repo id).
+## Models
+| onw model | arch | input | NPU 3720 decode | download | memory in use |
+|---|---|---|---|---|---|
+| [ryugyosoft/LFM2-8B-A1B-onw](https://huggingface.co/ryugyosoft/LFM2-8B-A1B-onw) | `lfm2_moe` (short conv + GQA + MoE 32x top-4) | text | 16-17 tok/s (29-43 with prompt lookup on code / summaries) | 4.0 GB | ~6 GB |
+| [ryugyosoft/gemma-4-E4B-it-onw](https://huggingface.co/ryugyosoft/gemma-4-E4B-it-onw) | `gemma4` (imported from the validated Gemma NPU graphs) | text + image | 7.2 tok/s | 4.3 GB | ~15 GB |
+| [ryugyosoft/Qwen3.5-9B-onw](https://huggingface.co/ryugyosoft/Qwen3.5-9B-onw) | `qwen3_5` (Gated DeltaNet + gated attention, dense) | text + image | 4.1 tok/s | 5.3 GB | ~20 GB |
+| [ryugyosoft/Qwen3.6-REAP-18B-A3B-onw](https://huggingface.co/ryugyosoft/Qwen3.6-REAP-18B-A3B-onw) | `qwen3_5_moe`, 128 of 256 experts kept (REAP) | text + image | 6.8 tok/s | 9.6 GB | ~15 GB |
+| [ryugyosoft/Qwen3.6-35B-A3B-onw](https://huggingface.co/ryugyosoft/Qwen3.6-35B-A3B-onw) | `qwen3_5_moe` (Gated DeltaNet + gated attention + MoE 256x top-8 + shared expert) | text + image | 6.8 tok/s | 17 GB | ~23 GB |
+Memory is the process working set on the test machine after loading (includes memory-mapped weight files the
+OS can page out).
+Measured with an image chat / text prompts on Core Ultra 9 285HX; prompt lookup decoding is used automatically
+where a 16-token verification block is cheap enough (LFM2, Qwen3.5-9B, Gemma), and every model keeps the
+conversation state between turns (a follow-up with an image re-processes only the new message: ~2 s).
+## Smaller models (onw convert --compact / --prune)
+- `--compact`: the 1-token and 16/64-token graphs of a segment share one weights file (download / disk);
+  host-only lookup tables (token embeddings not tied to the LM head, Gemma's per-layer embeddings) are INT4 with a
+  scale per 32 values; the Qwen vision tower is INT8. These also shrink the memory in use, except the shared
+  weights file: the NPU keeps one compiled copy per block size.
+- `--prune SAL.json --keep 0.5` (MoE): REAP expert pruning. `python -m onw.prune calibrate HF_DIR SAL.json`
+  measures each expert's saliency (mean router weight x output norm over chat data in Japanese, English and code),
+  `python -m onw.prune eval` reports the perplexity for several keep ratios. Qwen3.6-35B-A3B keeps 128 of 256
+  experts at +8.9% perplexity (published as Qwen3.6-REAP-18B-A3B-onw); LFM2 with only 32 experts loses +35% at
+  50%, so it is not pruned.
+## How it runs a MoE on an NPU
+NPUs need static graphs, and a sparse MoE picks different experts per token. Measured on NPU 3720 (Qwen3.6
+sizes, per layer, one token): computing every expert 45 ms; selecting experts inside the graph with Gather does
+not compile for INT4 constants and is 18-135 ms for INT8 / packed forms (the NPU moves whole tables);
+**passing the chosen experts as runtime inputs: 1.7 ms**, at the NPU's memory bandwidth. So:
+- the network is cut into **segments right after every MoE router**; the router's top-k goes to the host, which
+  binds the selected experts' weight buffers into the next segment (`set_tensor`, zero copy: all experts live in
+  NPU-visible host memory, `experts.bin`);
+- experts are **channel-wise INT4 packed as one tensor per expert** — the only weights-as-inputs form the NPU runs
+  at full speed (group-wise scales on runtime weights fall back to slow element-wise work, 20x slower);
+- everything else is group-128 INT4 constants inside the segments; the LM head is a shared INT8 input in its own
+  segment (one copy for all block sizes, skipped for prompt blocks that need no logits); token embeddings and
+  Gemma's 2.7 GB per-layer embeddings are looked up on the host (a gather from a runtime input makes the NPU move
+  the whole table);
+- prompt blocks bind the union of the block's experts; decoding binds the token's top-k.
+## Engine pieces
+| file | what |
+|---|---|
+| `onw/ir.py` | graph blocks: quantized linears, RMSNorm (with an fp16-safe prescale), RoPE / partial RoPE, attention over a host KV cache, expert slots, Gated DeltaNet (1-token matrix form; chunkwise-parallel form with an exact block-doubling inverse) |
+| `onw/runtime.py` | runs the segments; state by naming convention (KV rows, conv windows, recurrent states), expert binding, MRoPE, sliding-window masks, host table lookups, exact truncation for speculative verification |
+| `onw/chat.py` | chat loop: images (Qwen / Gemma), sampling, stop strings, prompt lookup decoding (exact rewind, or lazy redo for recurrent models, switched on only when it pays), prefix reuse between turns |
+| `onw/models/` | one builder per architecture: `lfm2_moe`, `qwen3_5_moe` (dense and MoE), `qwen_vision`, `gemma4` |
+| `onw/hub.py`, `onw/server.py`, `onw/web/` | model download + LFS guard, OpenAI-compatible server, chat UI (`/chat`), check page (`/check`) |
+| `onw/app.py`, `onw/tray.py`, `setup.*`, `find_python.bat` | onw window (models / server / settings), tray icon, installers (Python detection + winget) |
+| `check_ref.py`, `check_vision.py`, `test_chat.py`, `test_image.py`, `probe_segments.py`, `prof_*.py` | vs the HF model, generation / image / multi-turn checks, NPU-vs-CPU per segment, timing |
+## Notes and limitations
+- The context is 1024 tokens by default; `--context` / the settings change it at load time (one NPU compile per
+  length). Qwen images are resized to 512x512 (256 tokens); Gemma images keep their aspect
+  ratio (up to 280 tokens).
+- Experts are channel-wise INT4 (round-to-nearest): first-token logits within ~33% of bf16 with the same top-1 on
+  our checks; answers are fluent, word choices diverge from the bf16 model after a few tokens.
+- Prompt blocks above 16 tokens (Gemma's 64-token block) are loaded only with >= 24 GB RAM (`ONW_S64=1/0`).
+- Needs a pre-release OpenVINO (2026.5 nightly). Tested on NPU 3720 (Windows 11); CPU fallback on Ubuntu 22.04
+  (WSL). NPU 4000 (Lunar Lake) untested with this engine.
+## License
+Code: Apache 2.0. Converted models carry their base model's license (see each model repo).

onw/__init__.py CHANGED Viewed

@@ -2,4 +2,4 @@
 A small engine that runs LLMs entirely on the Intel NPU: static segment graphs + host-managed state.
 """
-__version__ = "0.4.0"

 A small engine that runs LLMs entirely on the Intel NPU: static segment graphs + host-managed state.
 """
+__version__ = "0.4.1"

pyproject.toml CHANGED Viewed

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 [project]
 name = "onw"
-version = "0.4.0"
 description = "Ore no NPU ga konna ni ugoku wake nai - LLMs entirely on the Intel NPU"
 readme = "README.md"
 license = {text = "Apache-2.0"}

 [project]
 name = "onw"
+version = "0.4.1"
 description = "Ore no NPU ga konna ni ugoku wake nai - LLMs entirely on the Intel NPU"
 readme = "README.md"
 license = {text = "Apache-2.0"}

setup.bat CHANGED Viewed

@@ -1,44 +1,44 @@
-@echo off
-rem onw - Ore no NPU ga konna ni ugoku wake nai: install the engine and the resident tray app (Windows).
-rem Finds Python 3.10+ (offers to install it with winget), creates .venv, installs onw + the tray, adds "onw" to
-rem the Start menu and starts it in the task tray. Models are downloaded later from the onw window (Models tab).
-setlocal
-cd /d "%~dp0"
-set PYTHONUTF8=1
-if exist ".venv\installed-tray.txt" goto shortcut
-if exist ".venv\Scripts\python.exe" goto install
-call find_python.bat
-if not defined PY (
-    pause
-    exit /b 1
-)
-echo [setup] creating Python environment in .venv ...
-%PY% -m venv .venv
-if errorlevel 1 (
-    echo [setup] could not create the Python environment.
-    pause
-    exit /b 1
-)
-:install
-".venv\Scripts\python.exe" -m pip install -U pip
-".venv\Scripts\python.exe" -m pip install --pre -e .[tray] --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly
-if errorlevel 1 (
-    echo [setup] package installation failed.
-    pause
-    exit /b 1
-)
-echo ok> ".venv\installed.txt"
-echo ok> ".venv\installed-tray.txt"
-:shortcut
-rem Start menu entry: pythonw -m onw tray (no console window)
-set "PYW=%CD%\.venv\Scripts\pythonw.exe"
-powershell -NoProfile -Command ^
-  "$s = (New-Object -ComObject WScript.Shell).CreateShortcut([Environment]::GetFolderPath('Programs') + '\onw.lnk');" ^
-  "$s.TargetPath = '%PYW%'; $s.Arguments = '-m onw tray'; $s.WorkingDirectory = '%CD%';" ^
-  "$s.Description = 'onw - LLMs on the Intel NPU'; $s.Save()"
-echo [setup] Start menu: onw
-echo [start] onw is starting in the task tray. Pick a model to download in the window that opens.
-start "" "%PYW%" -m onw tray

+@echo off
+rem onw - Ore no NPU ga konna ni ugoku wake nai: install the engine and the resident tray app (Windows).
+rem Finds Python 3.10+ (offers to install it with winget), creates .venv, installs onw + the tray, adds "onw" to
+rem the Start menu and starts it in the task tray. Models are downloaded later from the onw window (Models tab).
+setlocal
+cd /d "%~dp0"
+set PYTHONUTF8=1
+if exist ".venv\installed-tray.txt" goto shortcut
+if exist ".venv\Scripts\python.exe" goto install
+call "%~dp0find_python.bat"
+if not defined PY (
+    pause
+    exit /b 1
+)
+echo [setup] creating Python environment in .venv ...
+%PY% -m venv .venv
+if errorlevel 1 (
+    echo [setup] could not create the Python environment.
+    pause
+    exit /b 1
+)
+:install
+".venv\Scripts\python.exe" -m pip install -U pip
+".venv\Scripts\python.exe" -m pip install --pre -e .[tray] --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly
+if errorlevel 1 (
+    echo [setup] package installation failed.
+    pause
+    exit /b 1
+)
+echo ok> ".venv\installed.txt"
+echo ok> ".venv\installed-tray.txt"
+:shortcut
+rem Start menu entry: pythonw -m onw tray (no console window)
+set "PYW=%CD%\.venv\Scripts\pythonw.exe"
+powershell -NoProfile -Command ^
+  "$s = (New-Object -ComObject WScript.Shell).CreateShortcut([Environment]::GetFolderPath('Programs') + '\onw.lnk');" ^
+  "$s.TargetPath = '%PYW%'; $s.Arguments = '-m onw tray'; $s.WorkingDirectory = '%CD%';" ^
+  "$s.Description = 'onw - LLMs on the Intel NPU'; $s.Save()"
+echo [setup] Start menu: onw
+echo [start] onw is starting in the task tray. Pick a model to download in the window that opens.
+start "" "%PYW%" -m onw tray

start.bat CHANGED Viewed

@@ -1,52 +1,52 @@
-@echo off
-rem onw - Ore no NPU ga konna ni ugoku wake nai: run the server once from a console (Windows).
-rem   start.bat                                   LFM2-8B-A1B (default, fits 16 GB machines)
-rem   start.bat ryugyosoft/Qwen3.6-35B-A3B-onw    any onw model: a Hugging Face repo id or a local folder
-rem   start.bat MODEL --port 8080                 further options go to "onw serve"
-rem First run: finds / installs Python, creates .venv (~1 GB), downloads the model, compiles it for the NPU.
-rem For the resident tray app with a model manager use setup.bat instead.
-setlocal
-cd /d "%~dp0"
-set PYTHONUTF8=1
-set "MODEL=ryugyosoft/LFM2-8B-A1B-onw"
-set "FIRST=%~1"
-if not "%FIRST%"=="" if not "%FIRST:~0,1%"=="-" (
-    set "MODEL=%FIRST%"
-    shift
-)
-if exist ".venv\installed.txt" goto args
-if exist ".venv\Scripts\python.exe" goto install
-call find_python.bat
-if not defined PY (
-    pause
-    exit /b 1
-)
-echo [setup] creating Python environment in .venv ...
-%PY% -m venv .venv
-if errorlevel 1 (
-    echo [setup] could not create the Python environment.
-    pause
-    exit /b 1
-)
-:install
-".venv\Scripts\python.exe" -m pip install -U pip
-".venv\Scripts\python.exe" -m pip install --pre -e . --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly
-if errorlevel 1 (
-    echo [setup] package installation failed.
-    pause
-    exit /b 1
-)
-echo ok> ".venv\installed.txt"
-:args
-set "ARGS="
-:collect
-if "%~1"=="" goto run
-set "ARGS=%ARGS% %1"
-shift
-goto collect
-:run
-echo [start] %MODEL% (first time: download + NPU compile, please wait) ...
-".venv\Scripts\onw.exe" serve %MODEL% --open %ARGS%
-pause

+@echo off
+rem onw - Ore no NPU ga konna ni ugoku wake nai: run the server once from a console (Windows).
+rem   start.bat                                   LFM2-8B-A1B (default, fits 16 GB machines)
+rem   start.bat ryugyosoft/Qwen3.6-35B-A3B-onw    any onw model: a Hugging Face repo id or a local folder
+rem   start.bat MODEL --port 8080                 further options go to "onw serve"
+rem First run: finds / installs Python, creates .venv (~1 GB), downloads the model, compiles it for the NPU.
+rem For the resident tray app with a model manager use setup.bat instead.
+setlocal
+cd /d "%~dp0"
+set PYTHONUTF8=1
+set "MODEL=ryugyosoft/LFM2-8B-A1B-onw"
+set "FIRST=%~1"
+if not "%FIRST%"=="" if not "%FIRST:~0,1%"=="-" (
+    set "MODEL=%FIRST%"
+    shift
+)
+if exist ".venv\installed.txt" goto args
+if exist ".venv\Scripts\python.exe" goto install
+call "%~dp0find_python.bat"
+if not defined PY (
+    pause
+    exit /b 1
+)
+echo [setup] creating Python environment in .venv ...
+%PY% -m venv .venv
+if errorlevel 1 (
+    echo [setup] could not create the Python environment.
+    pause
+    exit /b 1
+)
+:install
+".venv\Scripts\python.exe" -m pip install -U pip
+".venv\Scripts\python.exe" -m pip install --pre -e . --extra-index-url https://storage.openvinotoolkit.org/simple/wheels/nightly
+if errorlevel 1 (
+    echo [setup] package installation failed.
+    pause
+    exit /b 1
+)
+echo ok> ".venv\installed.txt"
+:args
+set "ARGS="
+:collect
+if "%~1"=="" goto run
+set "ARGS=%ARGS% %1"
+shift
+goto collect
+:run
+echo [start] %MODEL% (first time: download + NPU compile, please wait) ...
+".venv\Scripts\onw.exe" serve %MODEL% --open %ARGS%
+pause