ryugyosoft commited on
Commit
eab5d72
·
verified ·
1 Parent(s): 5c48e14

README_en: add power efficiency and temperature section (Qwen3.5-9B, Lunar Lake)

Browse files
Files changed (1) hide show
  1. README_en.md +40 -0
README_en.md CHANGED
@@ -10,6 +10,9 @@ A small engine that runs LLMs — including MoE and hybrid DeltaNet models, with
10
  Intel NPU**, with an OpenAI-compatible server and a browser chat UI. Engine and models are separate downloads,
11
  like llama.cpp + a GGUF: one engine, one command, any converted model.
12
 
 
 
 
13
  ## Install (paste one line)
14
 
15
  On a PC with an Intel Core Ultra (it has the NPU) this one line installs it (details under "What you need" below).
@@ -199,6 +202,43 @@ install it anywhere with `pip install -e . --extra-index-url https://storage.ope
199
  | `setup.*`, `install.*`, `find_python.bat` | installers (Python detection + winget) |
200
  | `check_ref.py`, `check_vision.py`, `test_*.py`, `probe_segments.py`, `prof_*.py`, `mem_*.py`, `bench_*.py` | vs the HF model, generation / image / multi-turn checks, NPU-vs-CPU per segment, timing and memory |
201
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
202
  ## Notes and limitations
203
 
204
  - The context is 32K tokens by default and goes up to the model's maximum (262K for Qwen3.5 / 3.6, 128K for
 
10
  Intel NPU**, with an OpenAI-compatible server and a browser chat UI. Engine and models are separate downloads,
11
  like llama.cpp + a GGUF: one engine, one command, any converted model.
12
 
13
+ It runs on the low-power NPU, so you can use an LLM quietly, with the fan staying at low speed. Measured on Lunar Lake, it used
14
+ less energy per token than the iGPU (llama.cpp) and about 40% of the CPU. The numbers and how they were measured are in [Power efficiency and temperature](#power-efficiency-and-temperature).
15
+
16
  ## Install (paste one line)
17
 
18
  On a PC with an Intel Core Ultra (it has the NPU) this one line installs it (details under "What you need" below).
 
202
  | `setup.*`, `install.*`, `find_python.bat` | installers (Python detection + winget) |
203
  | `check_ref.py`, `check_vision.py`, `test_*.py`, `probe_segments.py`, `prof_*.py`, `mem_*.py`, `bench_*.py` | vs the HF model, generation / image / multi-turn checks, NPU-vs-CPU per segment, timing and memory |
204
 
205
+ ## Power efficiency and temperature
206
+
207
+ Package power and temperature while generating with Qwen3.5-9B on the same PC (LattePanda Mu Ultra, Core Ultra 5 226V / Lunar Lake), on the NPU, iGPU and CPU.
208
+ Average of 3 runs each of a long Japanese generation (768 tokens, temperature 0).
209
+
210
+ | Device | Engine | Decode | Package power | Package temp (avg / max) | Energy per token | Same, idle subtracted |
211
+ |---|---|---|---|---|---|---|
212
+ | **NPU** | onw (Qwen3.5-9B-onw, INT4, 5.3 GB) | 7.3 tok/s | **12.0 W** | 52 / **67 °C** | **1.66 J** | **1.27 J** |
213
+ | iGPU (Arc) | llama.cpp SYCL (Q4_K_M, 6.2 GB) | **12.6 tok/s** | 23.5 W | 66 / 83 °C | 1.91 J | 1.67 J |
214
+ | CPU | llama.cpp (Q4_K_M, 6.2 GB) | 5.7 tok/s | 22.3 W | 93 / 96 °C | 3.99 J | 3.47 J |
215
+
216
+ Idle package power is about 2.9 W at 42-46 °C.
217
+
218
+ For 10,000 generated tokens (calculated from the averages above):
219
+
220
+ | Device | Time | Energy | Same, idle subtracted |
221
+ |---|---|---|---|
222
+ | **NPU** | about 23 min | **4.6 Wh** | 3.5 Wh |
223
+ | iGPU | about 13 min | 5.3 Wh | 4.6 Wh |
224
+ | CPU | about 30 min | 11.1 Wh | 9.6 Wh |
225
+
226
+ The NPU needs about 40% of the CPU's energy and about 13% less than the iGPU. It draws 12 W during generation, about half the iGPU, peaks at 67 °C, and ran with the fan at low speed (fan speed was not measured; this is an observation).
227
+ The 10,000-token figures are calculated from the 768-token measurements, not measured. Over a longer run the CPU may throttle and take longer.
228
+
229
+
230
+ - The NPU uses the least energy per token: 13-24% less than the iGPU, and under 40% of the CPU.
231
+ - The iGPU is the fastest, about 1.7x the NPU, at about twice the power.
232
+ - The NPU peaks at 67 °C; the iGPU reaches 83 °C and the CPU 96 °C.
233
+
234
+ Method and caveats:
235
+ - Power is the RAPL package-0 energy counter (`intel-rapl:0`) sampled every 0.5 s. It covers the SoC (CPU, iGPU, NPU) only, not DRAM (about 0.4-0.8 W) or the whole PC. Temperature is `x86_pkg_temp`.
236
+ - Only the generation phase is measured, not model loading. Each run starts after a cool-down, with 30 s of idle power measured first. The devices were run NPU, iGPU, CPU in three rounds to avoid order effects.
237
+ - Fan speed was not measured: this PC does not expose the CPU cooler's RPM to the OS. During NPU generation the fan kept running at low speed (observation).
238
+ - The engines differ (onw vs llama.cpp) and the weights are not the same quantization (onw's own INT4 vs Q4_K_M), so read this as a comparison of model plus runtime.
239
+ - Spread: NPU and CPU were consistent across runs (NPU 1.65-1.67 J/token). The iGPU varied from run to run: 10.9-13.5 tok/s and 1.84-2.05 J/token.
240
+ - Other machines, prompts and cooling will give different numbers.
241
+
242
  ## Notes and limitations
243
 
244
  - The context is 32K tokens by default and goes up to the model's maximum (262K for Qwen3.5 / 3.6, 128K for