RTMW-m-WholeBody-OpenVINO

RTMW-m-WholeBody for Intel NPUs, GPUs and CPUs: rtmw_fp16 from litert-community/RTMW-m-WholeBody-LiteRT as ONNX (tf2onnx 1.17.0, opset 17), OpenVINO IR with fp32 weights, and a precompiled Intel NPU blob. On the NPU of a Core Ultra 9 386H the blob runs in 8.13 ms per run, numpy input to numpy output.

Scripts, raw records, and the same table for all 52 graphs of LiteRT-Models: https://github.com/john-rocky/windows-zoo

Files

file	bytes	sha256
rtmw_fp16.onnx	132,050,550	936dd1b47438dd6c79ab2ba32418e791a15dd205bf8df14e905243c80ec195b9
openvino/rtmw_fp16.xml	1,469,606	9a82b5e1a1e8c8d3134b6c7d417255c58b670bf01a1a4d5c84cc42212228299e
openvino/rtmw_fp16.bin	131,307,136	f207a75be64ec4e99eda3f7c51c4779c4c76d29119d560a52eb51ea3ca4f4738
npu/rtmw_fp16.blob	68,116,616	a0c5b1a6d6e47c00b8cff290f1ef43af0a1b3a7da2640d66791f339daddd42fb

The NPU blob was compiled with OpenVINO 2026.4.1, Intel NPU driver 32.0.100.4841 (OpenVINO reports NPU_DRIVER_VERSION 1004841, NPU_COMPILER_VERSION 524292). If import_model refuses a blob on another driver or OpenVINO version, compile the IR instead (the commented line in the snippet).

How it was measured

MSI Stealth 16 AI+ (Core Ultra 9 386H, RTX 5080 Laptop GPU at a 95 W power limit, Windows 11 build 26200.9711, on AC power), 2026-11-06 to 2026-11-07. On the laptop, each row is one process: load, one run, 5 warmup runs, then the median of 30 timed runs (2 warmup + 10 timed runs when the first run takes more than 3 s), numpy input to numpy output, every runtime at its default settings, one process at a time, each started after the CPU, the RTX and the Intel GPU had gone idle (the thresholds are in METHOD.md of the GitHub repo). Output check: each output against the CPU output in fp32 on the same input (ONNX Runtime's CPU EP); PASS = cosine ≥ 0.99, no NaN or Inf, and top-1 agreement ≥ 0.99 for class scores. The Snapdragon rows are Qualcomm AI Hub profile jobs of the same ONNX files (their own timer).

Speed and output check

processor	runtime	file	ms	load s	output check
Intel AI Boost NPU	OpenVINO 2026.4.1	npu/ blob (import_model)	8.13	0.04	FAIL: output_0_output: 51072 NaN, 0 Inf; output_0_output: cos 0.0000 < 0.99; output_0_output: argmax agreement 0/133; output_1_output: 68096 NaN, 0 Inf; output_1_output: cos 0.0000 < 0.99; output_1_output: argmax agreement 0/133
Intel AI Boost NPU	ONNX Runtime 1.31.1 + OpenVINO EP 1.8.1	.onnx, compiled when the session loads	8.67	3.15	FAIL: output_0_output: 51072 NaN, 0 Inf; output_0_output: cos 0.0000 < 0.99; output_0_output: argmax agreement 0/133; output_1_output: 68096 NaN, 0 Inf; output_1_output: cos 0.0000 < 0.99; output_1_output: argmax agreement 0/133
Intel Graphics (iGPU)	ONNX Runtime 1.31.1 + OpenVINO EP 1.8.1	.onnx	4.18	1.32	FAIL: output_0_output: argmax agreement 131/133; output_1_output: argmax agreement 131/133
RTX 5080 Laptop (95 W limit)	ONNX Runtime 1.31.1, CUDA EP	fp16 copy of the .onnx (not in this repo)	6.18	1.21	FAIL: output_0_output: argmax agreement 130/133; output_1_output: argmax agreement 130/133
RTX 5080 Laptop (95 W limit)	ONNX Runtime 1.31.1, CUDA EP	.onnx	7.17	1.24	PASS (cos ≥ 0.9999)
Core Ultra 9 386H CPU	ONNX Runtime 1.31.1, CPU EP	.onnx	15.62	0.48	reference
Snapdragon X2 Elite NPU	Qualcomm AI Hub, precompiled QNN ONNX	.onnx compiled by AI Hub	2.15	0.34	not compared (AI Hub profile job, 2026-10-30)
Snapdragon X2 Elite NPU	Qualcomm AI Hub, ONNX Runtime QNN EP	.onnx, compiled when it loads	3.04 (3 layers on the CPU)	2.30	not compared (AI Hub profile job, 2026-10-30)
Galaxy S26 NPU	Qualcomm AI Hub, precompiled QNN ONNX	.onnx compiled by AI Hub	2.19	0.20	not compared (AI Hub profile job, 2026-10-30)

The ONNX file against the source .tflite (LiteRT interpreter on the CPU, the same input): PASS, cosine ≥ 0.9999 for every output.

Run it on the Intel NPU

Download this repo, then run the snippets from its folder:

pip install huggingface_hub
hf download mlboydaisuke/RTMW-m-WholeBody-OpenVINO --local-dir RTMW-m-WholeBody-OpenVINO
cd RTMW-m-WholeBody-OpenVINO

OpenVINO (pip install openvino==2026.4.1):

import numpy as np
import openvino as ov

core = ov.Core()
if "NPU" not in core.available_devices:
    raise SystemExit(f"no Intel NPU visible to OpenVINO: {core.available_devices}")
with open("npu/rtmw_fp16.blob", "rb") as f:
    compiled = core.import_model(f.read(), "NPU")  # precompiled: no compile at load
# or compile the IR on this machine: compiled = core.compile_model("openvino/rtmw_fp16.xml", "NPU")
# replace the zeros with your preprocessed input (float32, the shapes printed by compiled.inputs)
feeds = {i.get_any_name(): np.zeros(list(i.shape), np.float32) for i in compiled.inputs}
outputs = compiled(feeds)
print({o.get_any_name(): outputs[o].shape for o in compiled.outputs})

ONNX Runtime with the OpenVINO EP (pip install onnxruntime==1.31.1 onnxruntime-ep-openvino==1.8.1):

import numpy as np
import onnxruntime as ort
import onnxruntime_ep_openvino as ov_ep

ort.register_execution_provider_library(ov_ep.get_ep_name(), ov_ep.get_library_path())
npu = [d for d in ort.get_ep_devices()
       if d.ep_name == ov_ep.get_ep_name() and d.ep_metadata.get("ov_device") == "NPU"]
if not npu:
    raise SystemExit("no Intel NPU visible to the OpenVINO EP")
so = ort.SessionOptions()
so.add_provider_for_devices(npu, {})
sess = ort.InferenceSession("rtmw_fp16.onnx", sess_options=so)  # compiles for the NPU here
# replace the zeros with your preprocessed input (float32)
feeds = {i.name: np.zeros(i.shape, np.float32) for i in sess.get_inputs()}
outputs = sess.run(None, feeds)
print([o.shape for o in outputs])

Licence

Apache-2.0, the licence of litert-community/RTMW-m-WholeBody-LiteRT. The source card names the upstream model as open-mmlab/mmpose (not a Hugging Face repo).