hilaus-arm commited on
Commit
8924a13
·
verified ·
1 Parent(s): 9422383

Sync from arm/Optimized-Runtimes via hub-sync

Browse files
README.md CHANGED
@@ -24,19 +24,19 @@ variants at square resolutions of 320, 480, 640, 800, 960, and 1280 pixels.
24
  ## Performance
25
 
26
  The following YOLO26n results were measured on Raspberry Pi 5 at 640x640,
27
- batch 1, against the 128-image, 1,050-instance
28
- [`coco-128-by-sa`](https://huggingface.co/datasets/Arm/coco-128-by-sa/tree/e7a3ecd065a304caddd2f170b8d41aeda66735b5)
29
- dataset revision selected for publication. Inference time excludes
30
  preprocessing and postprocessing.
31
 
32
  | Runtime | Inference (ms/image) | mAP50-95 |
33
  | --- | ---: | ---: |
34
- | Arm Runtime (this project) | 36.80 | 0.4866 |
35
- | NCNN | 66.23 | 0.4899 |
36
- | MNN | 90.38 | 0.4711 |
37
- | OpenVINO | 102.56 | 0.4787 |
38
- | ONNX Runtime | 127.29 | 0.4787 |
39
- | ExecuTorch | 137.90 | 0.4893 |
40
 
41
  See the [performance baseline](benchmarks/reporting/baseline.md) for the Arm
42
  Runtime measurement and host context.
 
24
  ## Performance
25
 
26
  The following YOLO26n results were measured on Raspberry Pi 5 at 640x640,
27
+ batch 1, against the 128-image, 1,025-instance
28
+ [`coco-128-by-sa`](https://huggingface.co/datasets/Arm/coco-128-by-sa/tree/97d6ea181e9dd0784db880b705e426ad23d424e1)
29
+ published dataset revision. Inference time excludes
30
  preprocessing and postprocessing.
31
 
32
  | Runtime | Inference (ms/image) | mAP50-95 |
33
  | --- | ---: | ---: |
34
+ | Arm Runtime (this project) | 36.60 | 0.4863 |
35
+ | NCNN | 67.20 | 0.4902 |
36
+ | MNN | 90.90 | 0.4753 |
37
+ | OpenVINO | 102.00 | 0.4812 |
38
+ | ONNX Runtime | 126.90 | 0.4812 |
39
+ | ExecuTorch | 136.90 | 0.4880 |
40
 
41
  See the [performance baseline](benchmarks/reporting/baseline.md) for the Arm
42
  Runtime measurement and host context.
benchmark/accuracy.yaml CHANGED
@@ -1,7 +1,7 @@
1
  version: 1.2.0
2
  report_type: vision-detection
3
  profile: Arm-Optimized
4
- created_at: '2026-10-20T17:58:48Z'
5
  context:
6
  target_ref: raspberry-pi-5-8gb
7
  runtime:
@@ -14,10 +14,10 @@ context:
14
  - XNNPACK
15
  dataset:
16
  name: COCO 128 BY/SA
17
- version: e7a3ecd065a304caddd2f170b8d41aeda66735b5
18
  slice: all
19
  sample_count: 128
20
- reference_url: https://huggingface.co/datasets/Arm/coco-128-by-sa/tree/e7a3ecd065a304caddd2f170b8d41aeda66735b5
21
  benchmark:
22
  batch_size: 1
23
  input_resolution: 640x640
@@ -25,6 +25,6 @@ context:
25
  warmup_runs: 0
26
  accuracy:
27
  sub_task: coco_map
28
- mAP_50_95_pct: 48.6615
29
- mAP_50_pct: 65.8740
30
- mAP_75_pct: 52.9025
 
1
  version: 1.2.0
2
  report_type: vision-detection
3
  profile: Arm-Optimized
4
+ created_at: '2026-10-22T21:59:43Z'
5
  context:
6
  target_ref: raspberry-pi-5-8gb
7
  runtime:
 
14
  - XNNPACK
15
  dataset:
16
  name: COCO 128 BY/SA
17
+ version: 97d6ea181e9dd0784db880b705e426ad23d424e1
18
  slice: all
19
  sample_count: 128
20
+ reference_url: https://huggingface.co/datasets/Arm/coco-128-by-sa/tree/97d6ea181e9dd0784db880b705e426ad23d424e1
21
  benchmark:
22
  batch_size: 1
23
  input_resolution: 640x640
 
25
  warmup_runs: 0
26
  accuracy:
27
  sub_task: coco_map
28
+ mAP_50_95_pct: 48.6264
29
+ mAP_50_pct: 65.8367
30
+ mAP_75_pct: 53.0182
benchmarks/reporting/baseline.md CHANGED
@@ -5,32 +5,31 @@ and `coco-128-by-sa` dataset identified below. Future performance-sensitive
5
  changes must use the same dataset and validation configuration before comparing
6
  their results with this measurement.
7
 
8
- The measurement uses the unpublished Hugging Face dataset revision identified
9
- below, which is the candidate intended for publication. Benchmark operators
10
- with access must download that revision and supply its local validation
11
- configuration explicitly with `DATA=/path/to/data.yaml`.
12
 
13
  ## Measurement
14
 
15
  | Item | Value |
16
  | --- | --- |
17
- | Source revision | `c87e781ba97b353c99cd8fac2dd1995e0d2e43b6` |
18
  | Runtime library SHA-256 | `6b9978243ed1fd325e5b7b6a2bc616280862ed01edc1f653ee55b088658b60b2` |
19
  | Model manifest SHA-256 | `3723a972dc64782fe86577eabde3d02539e10c778ae5ef77b0c5b9bc9f0d1f54` |
20
  | Dataset | `coco-128-by-sa` |
21
  | Dataset repository | [Arm/coco-128-by-sa](https://huggingface.co/datasets/Arm/coco-128-by-sa) |
22
- | Dataset Hub revision | [`f8af0a317c0b54c3dbd38c70b9757dccb8b02460`](https://huggingface.co/datasets/Arm/coco-128-by-sa/tree/f8af0a317c0b54c3dbd38c70b9757dccb8b02460) |
23
  | Dataset images | 128 |
24
- | Dataset instances | 1,050 |
25
  | Validation configuration SHA-256 | `accc0ff00b364d39c5f209eeb7e53609db634541551b7d4a392b7aa7e1e5f62b` |
26
- | Selection metadata SHA-256 | `042d13fd4cfe7d670a09604f4e437829420982effd12a55d91efa7f1d9fb28a4` |
27
- | License manifest SHA-256 | `38e5daf4bd373f4dc887bb1817ddae0cbe607c788bb46195592a5d9bfacd521c` |
28
- | COCO annotations SHA-256 | `3c756fbb846ccfcc71218b13b3fa9e274e87764e015346cc143e8a36c08b7388` |
29
  | Ultralytics | 8.4.108 |
30
  | Ultralytics options | `workers=0 plots=False` |
31
- | Cache state | Cold; host filesystem caches dropped immediately before each run |
32
- | Evidence directory | `20261020T182235078652000Z-c87e781ba97b` |
33
- | Recorded | 2026-10-20 18:22 UTC |
34
 
35
  `workers=0` keeps Ultralytics dataset loading in the benchmark process instead
36
  of starting data-loader subprocesses. It does not change the native runtime's
@@ -41,14 +40,14 @@ writing validation plots; it does not change detector scoring.
41
 
42
  | Measurement | Value |
43
  | --- | ---: |
44
- | Precision | 0.769 |
45
- | Recall | 0.577 |
46
- | mAP50 | 0.6587 |
47
- | mAP75 | 0.5290 |
48
- | mAP50-95 | 0.4866 |
49
- | Preprocess | 2.2 ms/image |
50
- | Inference | 36.8 ms/image |
51
- | Postprocess | 3.9 ms/image |
52
 
53
  ## Host context
54
 
@@ -56,12 +55,12 @@ The measurement ran on a Raspberry Pi 5 Model B Rev 1.1 with Linux
56
  6.12.75+rpt-rpi-2712. All four cores were available, and the CPU frequency
57
  policy remained on the `performance` governor at 2.4 GHz.
58
 
59
- The host filesystem caches were dropped immediately before each benchmark.
60
- For the recorded Arm Runtime run, cached and reclaimable data totaled 0.56 GB
61
- at the initial observation and 0.87 GB afterward. Cold reads caused 1,119
62
  major page faults; swap input and output did not change during the run.
63
 
64
- The CPU temperature increased from 50.2 °C to 51.8 °C. The firmware reported
65
  no current or historical throttling flags before or after the run. Available
66
- memory decreased from 7.40 GB to 7.35 GB, and peak benchmark-process-tree RSS
67
- was 694 MB.
 
5
  changes must use the same dataset and validation configuration before comparing
6
  their results with this measurement.
7
 
8
+ The measurement uses the published Hugging Face dataset revision identified
9
+ below. Benchmark operators must download that revision and supply its local
10
+ validation configuration explicitly with `DATA=/path/to/data.yaml`.
 
11
 
12
  ## Measurement
13
 
14
  | Item | Value |
15
  | --- | --- |
16
+ | Source revision | `4447eb6ed26b860740ef2077673254e604932c43` |
17
  | Runtime library SHA-256 | `6b9978243ed1fd325e5b7b6a2bc616280862ed01edc1f653ee55b088658b60b2` |
18
  | Model manifest SHA-256 | `3723a972dc64782fe86577eabde3d02539e10c778ae5ef77b0c5b9bc9f0d1f54` |
19
  | Dataset | `coco-128-by-sa` |
20
  | Dataset repository | [Arm/coco-128-by-sa](https://huggingface.co/datasets/Arm/coco-128-by-sa) |
21
+ | Dataset Hub revision | [`97d6ea181e9dd0784db880b705e426ad23d424e1`](https://huggingface.co/datasets/Arm/coco-128-by-sa/tree/97d6ea181e9dd0784db880b705e426ad23d424e1) |
22
  | Dataset images | 128 |
23
+ | Dataset instances | 1,025 |
24
  | Validation configuration SHA-256 | `accc0ff00b364d39c5f209eeb7e53609db634541551b7d4a392b7aa7e1e5f62b` |
25
+ | Selection metadata SHA-256 | `18205722c822b59635d4b6c911cef4495b5d688b4b23d84fde6d4e4396cec61f` |
26
+ | License manifest SHA-256 | `a56e45ab8e72a21906332dcfcfe8ba4ece9039c2c0cdc2409952042c57675b96` |
27
+ | COCO annotations SHA-256 | `009ef7e4cd3b533774d8755dbf8a3aa788d5af401b8283e87e27b2cad58cda57` |
28
  | Ultralytics | 8.4.108 |
29
  | Ultralytics options | `workers=0 plots=False` |
30
+ | Cache state | Cold; host filesystem caches dropped immediately before the run |
31
+ | Evidence directory | `20261022T232118602909Z-4447eb6` |
32
+ | Recorded | 2026-10-22 23:21 UTC |
33
 
34
  `workers=0` keeps Ultralytics dataset loading in the benchmark process instead
35
  of starting data-loader subprocesses. It does not change the native runtime's
 
40
 
41
  | Measurement | Value |
42
  | --- | ---: |
43
+ | Precision | 0.7613 |
44
+ | Recall | 0.5805 |
45
+ | mAP50 | 0.6584 |
46
+ | mAP75 | 0.5302 |
47
+ | mAP50-95 | 0.4863 |
48
+ | Preprocess | 2.3 ms/image |
49
+ | Inference | 36.6 ms/image |
50
+ | Postprocess | 3.7 ms/image |
51
 
52
  ## Host context
53
 
 
55
  6.12.75+rpt-rpi-2712. All four cores were available, and the CPU frequency
56
  policy remained on the `performance` governor at 2.4 GHz.
57
 
58
+ The host filesystem caches were dropped immediately before the benchmark.
59
+ For the recorded Arm Runtime run, cached and reclaimable data totaled 1.06 GiB
60
+ at the initial observation and 1.37 GiB afterward. Cold reads caused 1,285
61
  major page faults; swap input and output did not change during the run.
62
 
63
+ The CPU temperature increased from 46.9 °C to 54.0 °C. The firmware reported
64
  no current or historical throttling flags before or after the run. Available
65
+ memory decreased from 6.54 GiB to 6.51 GiB, and peak benchmark-process-tree RSS
66
+ was 445 MiB.