Artem Plastinkin Claude Sonnet 5.5 commited on
Commit
b562ea5
·
1 Parent(s): 1c715bb

Add ONNX Runtime Renesas EP v2.1.0 benchmarks

Browse files

Latency, NPU/CPU split, node distribution and accuracy for 1 and 12 AI cores
from ORT_Renesas_EP_Test_v2.1.0 (MWMX 2.2 ENG4); update existing ORT results in place.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

README.md CHANGED
@@ -66,6 +66,36 @@ TBD — not yet measured/published for this repo.
66
 
67
  ---
68
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
69
  ## Runtime Details
70
 
71
  ### MWMX Runtime
 
66
 
67
  ---
68
 
69
+ <!-- ORT-V210:BEGIN -->
70
+ ### ONNX Runtime – Renesas EP v2.1.0 (MWMX 2.2 ENG4)
71
+
72
+ Measured on **R-Car X5H** with ONNX Runtime + Renesas Execution Provider (INT8 auto-cast); nodes the NPU cannot run fall back to the CPU EP. Batch size 1, 1 NPU. Source: ORT Renesas EP test report, status 2026-09-30. This section supersedes earlier ORT figures in this card.
73
+
74
+ | Component | AI Cores | Latency (ms) | Throughput (fps) | NPU Inference (ms) | NPU % | CPU + Overhead % | Portable pkg NPU-only (ms) |
75
+ | :-- | :-: | :-: | :-: | :-: | :-: | :-: | :-: |
76
+ | Full model | 1 | 2.114 | 473 | 1.476 | 69.8 % | 30.2 % | 1.48359 |
77
+ | Full model | 12 | 1.729 | 578.4 | 0.799 | 46.2 % | 53.8 % | 0.692726 |
78
+
79
+ > 12-core latency is *not* always lower than 1-core: small models are dominated by CPU-side overhead.
80
+
81
+ **Graph partitioning (NPU vs CPU nodes)**
82
+
83
+ | Component | Total Nodes | NPU Nodes | CPU Nodes (Q/DQ inserted) | NPU % | CPU % |
84
+ | :-- | :-: | :-: | :-: | :-: | :-: |
85
+ | Full model | 50 | 49 | 1 (1) | 98.0 % | 2.0 % |
86
+
87
+ **Accuracy (ORT)**
88
+
89
+ | Runtime / Precision | Result |
90
+ | :-- | :-- |
91
+ | CPU EP · FP32 reference | Top-1 68.5 % · Top-5 90.1 % |
92
+ | Renesas EP + CPU EP · INT8 · v2.1.0 | Top-1 71.4 % · Top-5 89.7 % |
93
+
94
+ > Evaluated on a subset (1k images), not the full validation set.
95
+ <!-- ORT-V210:END -->
96
+
97
+ ---
98
+
99
  ## Runtime Details
100
 
101
  ### MWMX Runtime
fp32/benchmarks/x5h_ort_cpu_fp32.yaml ADDED
@@ -0,0 +1,51 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # FP32 reference accuracy: ONNX Runtime CPU EP (ORT_Renesas_EP_Test_v2.1.0.xlsx, 'Accuracy ( CPU fp32 )')
2
+ hardware:
3
+ vendor: renesas
4
+ chip: rcar-x5h
5
+ cpu: arm-cortex-a720
6
+ npu: npx6-48k
7
+ npu_count: 2
8
+ npu_cores: 12
9
+ npu_default_freq_mhz: 1066
10
+ accelerator:
11
+ - npu
12
+
13
+ runtime:
14
+ engine: onnxruntime
15
+ toolchain_version: "onnxruntime CPU EP"
16
+ format: onnx
17
+ execution_provider: cpu
18
+ execution_precision: fp32
19
+
20
+ configuration:
21
+ npu_instances: null
22
+ npu_cores_per_instance: null
23
+ npu_freq_mhz: null
24
+
25
+ benchmark:
26
+ type: reference
27
+ parameters:
28
+ batch_size: 1
29
+ input_resolution: [1, 3, 224, 224]
30
+
31
+ performance:
32
+ fps: null
33
+ latency: null
34
+
35
+ metrics:
36
+ accuracy: 0.685
37
+ top5_accuracy: 0.901
38
+
39
+ memory:
40
+ peak_mb: null
41
+
42
+ power:
43
+ avg_w: null
44
+
45
+ reproduce:
46
+ steps:
47
+ - title: Reproduction steps coming soon
48
+ command: >-
49
+ Step-by-step ONNX Runtime reproduction instructions are coming soon.
50
+ In the meantime, please contact your local Renesas representative.
51
+ kind: note
int8/benchmarks/x5h_ort_npu_12core.yaml ADDED
@@ -0,0 +1,65 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # ONNX Runtime + Renesas EP v2.1.0 (MWMX 2.2 ENG4); source: ORT_Renesas_EP_Test_v2.1.0.xlsx (status 2026-09-30)
2
+ # model/config used for the EP run: https://github.com/onnx/models/blob/main/validated/vision/classification/resnet/model/resnet18-v1-7.onnx
3
+ hardware:
4
+ vendor: renesas
5
+ chip: rcar-x5h
6
+ cpu: arm-cortex-a720
7
+ npu: npx6-48k
8
+ npu_count: 2
9
+ npu_cores: 12
10
+ npu_default_freq_mhz: 1066
11
+ accelerator:
12
+ - npu
13
+
14
+ runtime:
15
+ engine: onnxruntime # ONNX Runtime with Renesas Execution Provider (+ CPU EP for non-NPU nodes)
16
+ toolchain_version: "v2.1.0 renesas" # Renesas EP v2.1.0 (MWMX 2.2 ENG4)
17
+ format: onnx
18
+ execution_provider: npu
19
+ execution_precision: int8 # FP32 model is auto-cast to INT8 by the Renesas EP
20
+
21
+ configuration:
22
+ npu_instances: 1
23
+ npu_cores_per_instance: 12
24
+ npu_freq_mhz: 850 # not stated in the workbook; same 850 MHz as the MWMX and earlier ORT runs
25
+ npu_offload_pct: 98.0 # share of graph nodes executed on NPU (read by AI-Dashboard)
26
+ cpu_offload_pct: 2.0 # share of graph nodes falling back to CPU EP
27
+
28
+ benchmark:
29
+ type: hil
30
+ parameters:
31
+ batch_size: 1
32
+ input_resolution: [1, 3, 224, 224]
33
+
34
+ performance:
35
+ latency: 1.729
36
+ fps: 578.4
37
+ npu_inference_ms: 0.799
38
+ portable_pkg_npu_ms: 0.6927
39
+
40
+ metrics: # accuracy is a single measurement, independent of core count
41
+ accuracy: 0.714
42
+ top5_accuracy: 0.897
43
+ eval_subset: "1k images" # not the full validation set
44
+
45
+ node_distribution: # partitioning of the graph between NPU and CPU EP
46
+ total: 50
47
+ npu: 49
48
+ cpu: 1
49
+ qdq_inserted: 1
50
+ dispatched_to_cpu_compile: null
51
+ dispatched_to_cpu_accuracy: null
52
+
53
+ memory:
54
+ peak_mb: null
55
+
56
+ power:
57
+ avg_w: null
58
+
59
+ reproduce:
60
+ steps:
61
+ - title: Reproduction steps coming soon
62
+ command: >-
63
+ Step-by-step ONNX Runtime reproduction instructions are coming soon.
64
+ In the meantime, please contact your local Renesas representative.
65
+ kind: note
int8/benchmarks/x5h_ort_npu_1core.yaml ADDED
@@ -0,0 +1,66 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # ONNX Runtime + Renesas EP v2.1.0 (MWMX 2.2 ENG4); source: ORT_Renesas_EP_Test_v2.1.0.xlsx (status 2026-09-30)
2
+ # model/config used for the EP run: https://github.com/onnx/models/blob/main/validated/vision/classification/resnet/model/resnet18-v1-7.onnx
3
+ hardware:
4
+ vendor: renesas
5
+ chip: rcar-x5h
6
+ cpu: arm-cortex-a720
7
+ npu: npx6-48k
8
+ npu_count: 2
9
+ npu_cores: 12
10
+ npu_default_freq_mhz: 1066
11
+ accelerator:
12
+ - npu
13
+
14
+ runtime:
15
+ engine: onnxruntime # ONNX Runtime with Renesas Execution Provider (+ CPU EP for non-NPU nodes)
16
+ toolchain_version: "v2.1.0 renesas" # Renesas EP v2.1.0 (MWMX 2.2 ENG4)
17
+ format: onnx
18
+ execution_provider: npu
19
+ execution_precision: int8 # FP32 model is auto-cast to INT8 by the Renesas EP
20
+
21
+ configuration:
22
+ npu_instances: 1
23
+ npu_cores_per_instance: 1
24
+ npu_freq_mhz: 850 # not stated in the workbook; same 850 MHz as the MWMX and earlier ORT runs
25
+ npu_offload_pct: 98.0 # share of graph nodes executed on NPU (read by AI-Dashboard)
26
+ cpu_offload_pct: 2.0 # share of graph nodes falling back to CPU EP
27
+
28
+ benchmark:
29
+ type: hil
30
+ parameters:
31
+ batch_size: 1
32
+ input_resolution: [1, 3, 224, 224]
33
+
34
+ performance:
35
+ latency: 2.114
36
+ fps: 473.0
37
+ npu_inference_ms: 1.476
38
+ portable_pkg_npu_ms: 1.4836
39
+ cpu_inference_overhead_ms: 0.053
40
+
41
+ metrics: # accuracy is a single measurement, independent of core count
42
+ accuracy: 0.714
43
+ top5_accuracy: 0.897
44
+ eval_subset: "1k images" # not the full validation set
45
+
46
+ node_distribution: # partitioning of the graph between NPU and CPU EP
47
+ total: 50
48
+ npu: 49
49
+ cpu: 1
50
+ qdq_inserted: 1
51
+ dispatched_to_cpu_compile: null
52
+ dispatched_to_cpu_accuracy: null
53
+
54
+ memory:
55
+ peak_mb: null
56
+
57
+ power:
58
+ avg_w: null
59
+
60
+ reproduce:
61
+ steps:
62
+ - title: Reproduction steps coming soon
63
+ command: >-
64
+ Step-by-step ONNX Runtime reproduction instructions are coming soon.
65
+ In the meantime, please contact your local Renesas representative.
66
+ kind: note