Artem Plastinkin commited on
Commit
0644f4c
·
1 Parent(s): 2f72caa

Update performance numbers

Browse files
int8/benchmarks/x5h_mwmx_npu_apm80_12core.yaml CHANGED
@@ -29,7 +29,7 @@ benchmark:
29
 
30
  performance:
31
  fps: null
32
- latency: 6.810764 # APM80 pipeline run (metawaremx_runtime CI)
33
 
34
  metrics:
35
  accuracy: null
 
29
 
30
  performance:
31
  fps: null
32
+ latency: 6.72 # synced to mwmx2.2 model_list.html (2026-09-22); was 6.810764 (APM80 pipeline run, metawaremx_runtime CI)
33
 
34
  metrics:
35
  accuracy: null
int8/benchmarks/x5h_mwmx_npu_apm80_1core.yaml ADDED
@@ -0,0 +1,70 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ hardware:
2
+ vendor: renesas
3
+ chip: rcar-x5h
4
+ cpu: arm-cortex-a720
5
+ npu: npx6-48k
6
+ npu_count: 2
7
+ npu_cores: 12
8
+ npu_default_freq_mhz: 1066
9
+ accelerator:
10
+ - npu
11
+
12
+ runtime:
13
+ engine: mwmx # Renesas MWMX (Middleware MX) native inference runtime
14
+ toolchain_version: "MWMX SDK v4.35.0"
15
+ format: onnx # input model format (FP32 ONNX)
16
+ execution_provider: npu
17
+ execution_precision: int8 # FP32 model is auto-cast to INT8 by the MWMX toolchain
18
+
19
+ configuration:
20
+ npu_instances: 1
21
+ npu_cores_per_instance: 1
22
+ npu_freq_mhz: 850
23
+
24
+ benchmark:
25
+ type: hil
26
+ parameters:
27
+ batch_size: 1
28
+ input_resolution: [1, 3, 480, 640]
29
+
30
+ performance:
31
+ fps: null
32
+ latency: 44.77 # synced to mwmx2.2 model_list.html (2026-09-22); no prior local measurement for this core count
33
+
34
+ metrics:
35
+ accuracy: null
36
+ top5_accuracy: null
37
+
38
+ memory:
39
+ peak_mb: null
40
+
41
+ power:
42
+ avg_w: null
43
+
44
+ # Exact commands verified against the NNAC "Getting Started" chapter. Rendered
45
+ # by the AI-Dashboard in place of the generic placeholder flow — see
46
+ # downloadRunHTML() / parse_reproduce() in AI-Dashboard/app.js.
47
+ reproduce:
48
+ steps:
49
+ - title: Activate the Python environment
50
+ command: >-
51
+ Activate the Python virtual environment that has the `hf` CLI
52
+ (huggingface_hub) and the NNAC toolchain installed, e.g.
53
+ `source nnac_venv/bin/activate` -- path depends on your toolchain install.
54
+ kind: note
55
+ - title: Download the ONNX model
56
+ command: hf download Renesas/RetinaNet-R101-ONNX --repo-type=model --include "fp32/*" --local-dir ./RetinaNet-R101-ONNX-fp32
57
+ - title: Compile with the NNAC toolchain (INT8 auto-cast from the FP32 graph)
58
+ command: |
59
+ python3 nnac_frontend/legalize.py -d binary/nnx ./RetinaNet-R101-ONNX-fp32/fp32/retinanet-9.onnx --num-core 1
60
+ - title: Set up the R-Car X5H board
61
+ command: Configure the board per the AI Compiler (NNAC) "Getting Started" guide, section 3.4 (host TFTP/NFS setup, bootloader flashing, U-Boot, Linux boot, login) -- exact steps depend on your board/network setup.
62
+ kind: note
63
+ - title: Copy the compiled artifact to the board
64
+ command: Copy ${WORKDIR}/binary/nnx (the working directory from the download/compile steps above) to the board -- method may vary (NFS mount, scp, USB, etc.).
65
+ kind: note
66
+ - title: Run on R-Car X5H (single NPU cluster)
67
+ command: |
68
+ cd binary
69
+ ./host_app ./arc_prog_npus ./nnx/retinanet-9
70
+ expected: hash[n] = 0x...(OK) means the run's output matches the reference hash in hash.txt; latency is the NPX execution time reported in cycles and ms.
int8/benchmarks/x5h_mwmx_npu_apm80_3core.yaml ADDED
@@ -0,0 +1,72 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ hardware:
2
+ vendor: renesas
3
+ chip: rcar-x5h
4
+ cpu: arm-cortex-a720
5
+ npu: npx6-48k
6
+ npu_count: 2
7
+ npu_cores: 12
8
+ npu_default_freq_mhz: 1066
9
+ accelerator:
10
+ - npu
11
+
12
+ runtime:
13
+ engine: mwmx # Renesas MWMX (Middleware MX) native inference runtime
14
+ toolchain_version: "MWMX SDK v4.35.0"
15
+ format: onnx # input model format (FP32 ONNX)
16
+ execution_provider: npu
17
+ execution_precision: int8 # FP32 model is auto-cast to INT8 by the MWMX toolchain
18
+
19
+ configuration:
20
+ npu_instances: 1
21
+ npu_cores_per_instance: 3
22
+ npu_freq_mhz: 850
23
+
24
+ benchmark:
25
+ type: hil
26
+ parameters:
27
+ batch_size: 1
28
+ input_resolution: [1, 3, 480, 640]
29
+
30
+ performance:
31
+ fps: null
32
+ latency: 15.38 # synced to mwmx2.2 model_list.html (2026-09-22); no prior local measurement for this core count
33
+
34
+ metrics:
35
+ accuracy: null
36
+ top5_accuracy: null
37
+
38
+ memory:
39
+ peak_mb: null
40
+
41
+ power:
42
+ avg_w: null
43
+
44
+ # UNCONFIRMED (segmentation config): this repo's segmentation_json compile_config
45
+ # (compile_config/RetinaNet_l2seg.json, see the 12-core reproduce block) is
46
+ # specific to the 12-core build; no 3-core segmentation config is bundled, so
47
+ # the steps below compile without one — a best-effort single-cluster flow that
48
+ # may not reproduce the exact benchmarked 3-core latency.
49
+ reproduce:
50
+ steps:
51
+ - title: Activate the Python environment
52
+ command: >-
53
+ Activate the Python virtual environment that has the `hf` CLI
54
+ (huggingface_hub) and the NNAC toolchain installed, e.g.
55
+ `source nnac_venv/bin/activate` -- path depends on your toolchain install.
56
+ kind: note
57
+ - title: Download the ONNX model
58
+ command: hf download Renesas/RetinaNet-R101-ONNX --repo-type=model --include "fp32/*" --local-dir ./RetinaNet-R101-ONNX-fp32
59
+ - title: Compile with the NNAC toolchain (INT8 auto-cast from the FP32 graph)
60
+ command: |
61
+ python3 nnac_frontend/legalize.py -d binary/nnx ./RetinaNet-R101-ONNX-fp32/fp32/retinanet-9.onnx --num-core 3
62
+ - title: Set up the R-Car X5H board
63
+ command: Configure the board per the AI Compiler (NNAC) "Getting Started" guide, section 3.4 (host TFTP/NFS setup, bootloader flashing, U-Boot, Linux boot, login) -- exact steps depend on your board/network setup.
64
+ kind: note
65
+ - title: Copy the compiled artifact to the board
66
+ command: Copy ${WORKDIR}/binary/nnx (the working directory from the download/compile steps above) to the board -- method may vary (NFS mount, scp, USB, etc.).
67
+ kind: note
68
+ - title: Run on R-Car X5H (single NPU cluster, 3 AI cores)
69
+ command: |
70
+ cd binary
71
+ ./host_app ./arc_prog_npus ./nnx/retinanet-9
72
+ expected: hash[n] = 0x...(OK) means the run's output matches the reference hash in hash.txt; latency is the NPX execution time reported in cycles and ms.
int8/benchmarks/x5h_mwmx_npu_apm80_4core.yaml ADDED
@@ -0,0 +1,72 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ hardware:
2
+ vendor: renesas
3
+ chip: rcar-x5h
4
+ cpu: arm-cortex-a720
5
+ npu: npx6-48k
6
+ npu_count: 2
7
+ npu_cores: 12
8
+ npu_default_freq_mhz: 1066
9
+ accelerator:
10
+ - npu
11
+
12
+ runtime:
13
+ engine: mwmx # Renesas MWMX (Middleware MX) native inference runtime
14
+ toolchain_version: "MWMX SDK v4.35.0"
15
+ format: onnx # input model format (FP32 ONNX)
16
+ execution_provider: npu
17
+ execution_precision: int8 # FP32 model is auto-cast to INT8 by the MWMX toolchain
18
+
19
+ configuration:
20
+ npu_instances: 1
21
+ npu_cores_per_instance: 4
22
+ npu_freq_mhz: 850
23
+
24
+ benchmark:
25
+ type: hil
26
+ parameters:
27
+ batch_size: 1
28
+ input_resolution: [1, 3, 480, 640]
29
+
30
+ performance:
31
+ fps: null
32
+ latency: 11.66 # synced to mwmx2.2 model_list.html (2026-09-22); no prior local measurement for this core count
33
+
34
+ metrics:
35
+ accuracy: null
36
+ top5_accuracy: null
37
+
38
+ memory:
39
+ peak_mb: null
40
+
41
+ power:
42
+ avg_w: null
43
+
44
+ # UNCONFIRMED (segmentation config): this repo's segmentation_json compile_config
45
+ # (compile_config/RetinaNet_l2seg.json, see the 12-core reproduce block) is
46
+ # specific to the 12-core build; no 4-core segmentation config is bundled, so
47
+ # the steps below compile without one — a best-effort single-cluster flow that
48
+ # may not reproduce the exact benchmarked 4-core latency.
49
+ reproduce:
50
+ steps:
51
+ - title: Activate the Python environment
52
+ command: >-
53
+ Activate the Python virtual environment that has the `hf` CLI
54
+ (huggingface_hub) and the NNAC toolchain installed, e.g.
55
+ `source nnac_venv/bin/activate` -- path depends on your toolchain install.
56
+ kind: note
57
+ - title: Download the ONNX model
58
+ command: hf download Renesas/RetinaNet-R101-ONNX --repo-type=model --include "fp32/*" --local-dir ./RetinaNet-R101-ONNX-fp32
59
+ - title: Compile with the NNAC toolchain (INT8 auto-cast from the FP32 graph)
60
+ command: |
61
+ python3 nnac_frontend/legalize.py -d binary/nnx ./RetinaNet-R101-ONNX-fp32/fp32/retinanet-9.onnx --num-core 4
62
+ - title: Set up the R-Car X5H board
63
+ command: Configure the board per the AI Compiler (NNAC) "Getting Started" guide, section 3.4 (host TFTP/NFS setup, bootloader flashing, U-Boot, Linux boot, login) -- exact steps depend on your board/network setup.
64
+ kind: note
65
+ - title: Copy the compiled artifact to the board
66
+ command: Copy ${WORKDIR}/binary/nnx (the working directory from the download/compile steps above) to the board -- method may vary (NFS mount, scp, USB, etc.).
67
+ kind: note
68
+ - title: Run on R-Car X5H (single NPU cluster, 4 AI cores)
69
+ command: |
70
+ cd binary
71
+ ./host_app ./arc_prog_npus ./nnx/retinanet-9
72
+ expected: hash[n] = 0x...(OK) means the run's output matches the reference hash in hash.txt; latency is the NPX execution time reported in cycles and ms.
int8/benchmarks/x5h_mwmx_npu_apm80_6core.yaml ADDED
@@ -0,0 +1,72 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ hardware:
2
+ vendor: renesas
3
+ chip: rcar-x5h
4
+ cpu: arm-cortex-a720
5
+ npu: npx6-48k
6
+ npu_count: 2
7
+ npu_cores: 12
8
+ npu_default_freq_mhz: 1066
9
+ accelerator:
10
+ - npu
11
+
12
+ runtime:
13
+ engine: mwmx # Renesas MWMX (Middleware MX) native inference runtime
14
+ toolchain_version: "MWMX SDK v4.35.0"
15
+ format: onnx # input model format (FP32 ONNX)
16
+ execution_provider: npu
17
+ execution_precision: int8 # FP32 model is auto-cast to INT8 by the MWMX toolchain
18
+
19
+ configuration:
20
+ npu_instances: 1
21
+ npu_cores_per_instance: 6
22
+ npu_freq_mhz: 850
23
+
24
+ benchmark:
25
+ type: hil
26
+ parameters:
27
+ batch_size: 1
28
+ input_resolution: [1, 3, 480, 640]
29
+
30
+ performance:
31
+ fps: null
32
+ latency: 9.24 # synced to mwmx2.2 model_list.html (2026-09-22); no prior local measurement for this core count
33
+
34
+ metrics:
35
+ accuracy: null
36
+ top5_accuracy: null
37
+
38
+ memory:
39
+ peak_mb: null
40
+
41
+ power:
42
+ avg_w: null
43
+
44
+ # UNCONFIRMED (segmentation config): this repo's segmentation_json compile_config
45
+ # (compile_config/RetinaNet_l2seg.json, see the 12-core reproduce block) is
46
+ # specific to the 12-core build; no 6-core segmentation config is bundled, so
47
+ # the steps below compile without one — a best-effort single-cluster flow that
48
+ # may not reproduce the exact benchmarked 6-core latency.
49
+ reproduce:
50
+ steps:
51
+ - title: Activate the Python environment
52
+ command: >-
53
+ Activate the Python virtual environment that has the `hf` CLI
54
+ (huggingface_hub) and the NNAC toolchain installed, e.g.
55
+ `source nnac_venv/bin/activate` -- path depends on your toolchain install.
56
+ kind: note
57
+ - title: Download the ONNX model
58
+ command: hf download Renesas/RetinaNet-R101-ONNX --repo-type=model --include "fp32/*" --local-dir ./RetinaNet-R101-ONNX-fp32
59
+ - title: Compile with the NNAC toolchain (INT8 auto-cast from the FP32 graph)
60
+ command: |
61
+ python3 nnac_frontend/legalize.py -d binary/nnx ./RetinaNet-R101-ONNX-fp32/fp32/retinanet-9.onnx --num-core 6
62
+ - title: Set up the R-Car X5H board
63
+ command: Configure the board per the AI Compiler (NNAC) "Getting Started" guide, section 3.4 (host TFTP/NFS setup, bootloader flashing, U-Boot, Linux boot, login) -- exact steps depend on your board/network setup.
64
+ kind: note
65
+ - title: Copy the compiled artifact to the board
66
+ command: Copy ${WORKDIR}/binary/nnx (the working directory from the download/compile steps above) to the board -- method may vary (NFS mount, scp, USB, etc.).
67
+ kind: note
68
+ - title: Run on R-Car X5H (single NPU cluster, 6 AI cores)
69
+ command: |
70
+ cd binary
71
+ ./host_app ./arc_prog_npus ./nnx/retinanet-9
72
+ expected: hash[n] = 0x...(OK) means the run's output matches the reference hash in hash.txt; latency is the NPX execution time reported in cycles and ms.