README_en: add power efficiency and temperature section (Qwen3.5-9B, Lunar Lake)
Browse files- README_en.md +40 -0
README_en.md
CHANGED
|
@@ -10,6 +10,9 @@ A small engine that runs LLMs — including MoE and hybrid DeltaNet models, with
|
|
| 10 |
Intel NPU**, with an OpenAI-compatible server and a browser chat UI. Engine and models are separate downloads,
|
| 11 |
like llama.cpp + a GGUF: one engine, one command, any converted model.
|
| 12 |
|
|
|
|
|
|
|
|
|
|
| 13 |
## Install (paste one line)
|
| 14 |
|
| 15 |
On a PC with an Intel Core Ultra (it has the NPU) this one line installs it (details under "What you need" below).
|
|
@@ -199,6 +202,43 @@ install it anywhere with `pip install -e . --extra-index-url https://storage.ope
|
|
| 199 |
| `setup.*`, `install.*`, `find_python.bat` | installers (Python detection + winget) |
|
| 200 |
| `check_ref.py`, `check_vision.py`, `test_*.py`, `probe_segments.py`, `prof_*.py`, `mem_*.py`, `bench_*.py` | vs the HF model, generation / image / multi-turn checks, NPU-vs-CPU per segment, timing and memory |
|
| 201 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 202 |
## Notes and limitations
|
| 203 |
|
| 204 |
- The context is 32K tokens by default and goes up to the model's maximum (262K for Qwen3.5 / 3.6, 128K for
|
|
|
|
| 10 |
Intel NPU**, with an OpenAI-compatible server and a browser chat UI. Engine and models are separate downloads,
|
| 11 |
like llama.cpp + a GGUF: one engine, one command, any converted model.
|
| 12 |
|
| 13 |
+
It runs on the low-power NPU, so you can use an LLM quietly, with the fan staying at low speed. Measured on Lunar Lake, it used
|
| 14 |
+
less energy per token than the iGPU (llama.cpp) and about 40% of the CPU. The numbers and how they were measured are in [Power efficiency and temperature](#power-efficiency-and-temperature).
|
| 15 |
+
|
| 16 |
## Install (paste one line)
|
| 17 |
|
| 18 |
On a PC with an Intel Core Ultra (it has the NPU) this one line installs it (details under "What you need" below).
|
|
|
|
| 202 |
| `setup.*`, `install.*`, `find_python.bat` | installers (Python detection + winget) |
|
| 203 |
| `check_ref.py`, `check_vision.py`, `test_*.py`, `probe_segments.py`, `prof_*.py`, `mem_*.py`, `bench_*.py` | vs the HF model, generation / image / multi-turn checks, NPU-vs-CPU per segment, timing and memory |
|
| 204 |
|
| 205 |
+
## Power efficiency and temperature
|
| 206 |
+
|
| 207 |
+
Package power and temperature while generating with Qwen3.5-9B on the same PC (LattePanda Mu Ultra, Core Ultra 5 226V / Lunar Lake), on the NPU, iGPU and CPU.
|
| 208 |
+
Average of 3 runs each of a long Japanese generation (768 tokens, temperature 0).
|
| 209 |
+
|
| 210 |
+
| Device | Engine | Decode | Package power | Package temp (avg / max) | Energy per token | Same, idle subtracted |
|
| 211 |
+
|---|---|---|---|---|---|---|
|
| 212 |
+
| **NPU** | onw (Qwen3.5-9B-onw, INT4, 5.3 GB) | 7.3 tok/s | **12.0 W** | 52 / **67 °C** | **1.66 J** | **1.27 J** |
|
| 213 |
+
| iGPU (Arc) | llama.cpp SYCL (Q4_K_M, 6.2 GB) | **12.6 tok/s** | 23.5 W | 66 / 83 °C | 1.91 J | 1.67 J |
|
| 214 |
+
| CPU | llama.cpp (Q4_K_M, 6.2 GB) | 5.7 tok/s | 22.3 W | 93 / 96 °C | 3.99 J | 3.47 J |
|
| 215 |
+
|
| 216 |
+
Idle package power is about 2.9 W at 42-46 °C.
|
| 217 |
+
|
| 218 |
+
For 10,000 generated tokens (calculated from the averages above):
|
| 219 |
+
|
| 220 |
+
| Device | Time | Energy | Same, idle subtracted |
|
| 221 |
+
|---|---|---|---|
|
| 222 |
+
| **NPU** | about 23 min | **4.6 Wh** | 3.5 Wh |
|
| 223 |
+
| iGPU | about 13 min | 5.3 Wh | 4.6 Wh |
|
| 224 |
+
| CPU | about 30 min | 11.1 Wh | 9.6 Wh |
|
| 225 |
+
|
| 226 |
+
The NPU needs about 40% of the CPU's energy and about 13% less than the iGPU. It draws 12 W during generation, about half the iGPU, peaks at 67 °C, and ran with the fan at low speed (fan speed was not measured; this is an observation).
|
| 227 |
+
The 10,000-token figures are calculated from the 768-token measurements, not measured. Over a longer run the CPU may throttle and take longer.
|
| 228 |
+
|
| 229 |
+
|
| 230 |
+
- The NPU uses the least energy per token: 13-24% less than the iGPU, and under 40% of the CPU.
|
| 231 |
+
- The iGPU is the fastest, about 1.7x the NPU, at about twice the power.
|
| 232 |
+
- The NPU peaks at 67 °C; the iGPU reaches 83 °C and the CPU 96 °C.
|
| 233 |
+
|
| 234 |
+
Method and caveats:
|
| 235 |
+
- Power is the RAPL package-0 energy counter (`intel-rapl:0`) sampled every 0.5 s. It covers the SoC (CPU, iGPU, NPU) only, not DRAM (about 0.4-0.8 W) or the whole PC. Temperature is `x86_pkg_temp`.
|
| 236 |
+
- Only the generation phase is measured, not model loading. Each run starts after a cool-down, with 30 s of idle power measured first. The devices were run NPU, iGPU, CPU in three rounds to avoid order effects.
|
| 237 |
+
- Fan speed was not measured: this PC does not expose the CPU cooler's RPM to the OS. During NPU generation the fan kept running at low speed (observation).
|
| 238 |
+
- The engines differ (onw vs llama.cpp) and the weights are not the same quantization (onw's own INT4 vs Q4_K_M), so read this as a comparison of model plus runtime.
|
| 239 |
+
- Spread: NPU and CPU were consistent across runs (NPU 1.65-1.67 J/token). The iGPU varied from run to run: 10.9-13.5 tok/s and 1.84-2.05 J/token.
|
| 240 |
+
- Other machines, prompts and cooling will give different numbers.
|
| 241 |
+
|
| 242 |
## Notes and limitations
|
| 243 |
|
| 244 |
- The context is 32K tokens by default and goes up to the model's maximum (262K for Qwen3.5 / 3.6, 128K for
|