Sync from arm/Optimized-Runtimes via hub-sync
Browse files- README.md +9 -9
- benchmark/accuracy.yaml +6 -6
- benchmarks/reporting/baseline.md +26 -27
README.md
CHANGED
|
@@ -24,19 +24,19 @@ variants at square resolutions of 320, 480, 640, 800, 960, and 1280 pixels.
|
|
| 24 |
## Performance
|
| 25 |
|
| 26 |
The following YOLO26n results were measured on Raspberry Pi 5 at 640x640,
|
| 27 |
-
batch 1, against the 128-image, 1,
|
| 28 |
-
[`coco-128-by-sa`](https://huggingface.co/datasets/Arm/coco-128-by-sa/tree/
|
| 29 |
-
dataset revision
|
| 30 |
preprocessing and postprocessing.
|
| 31 |
|
| 32 |
| Runtime | Inference (ms/image) | mAP50-95 |
|
| 33 |
| --- | ---: | ---: |
|
| 34 |
-
| Arm Runtime (this project) | 36.
|
| 35 |
-
| NCNN |
|
| 36 |
-
| MNN | 90.
|
| 37 |
-
| OpenVINO | 102.
|
| 38 |
-
| ONNX Runtime |
|
| 39 |
-
| ExecuTorch |
|
| 40 |
|
| 41 |
See the [performance baseline](benchmarks/reporting/baseline.md) for the Arm
|
| 42 |
Runtime measurement and host context.
|
|
|
|
| 24 |
## Performance
|
| 25 |
|
| 26 |
The following YOLO26n results were measured on Raspberry Pi 5 at 640x640,
|
| 27 |
+
batch 1, against the 128-image, 1,025-instance
|
| 28 |
+
[`coco-128-by-sa`](https://huggingface.co/datasets/Arm/coco-128-by-sa/tree/97d6ea181e9dd0784db880b705e426ad23d424e1)
|
| 29 |
+
published dataset revision. Inference time excludes
|
| 30 |
preprocessing and postprocessing.
|
| 31 |
|
| 32 |
| Runtime | Inference (ms/image) | mAP50-95 |
|
| 33 |
| --- | ---: | ---: |
|
| 34 |
+
| Arm Runtime (this project) | 36.60 | 0.4863 |
|
| 35 |
+
| NCNN | 67.20 | 0.4902 |
|
| 36 |
+
| MNN | 90.90 | 0.4753 |
|
| 37 |
+
| OpenVINO | 102.00 | 0.4812 |
|
| 38 |
+
| ONNX Runtime | 126.90 | 0.4812 |
|
| 39 |
+
| ExecuTorch | 136.90 | 0.4880 |
|
| 40 |
|
| 41 |
See the [performance baseline](benchmarks/reporting/baseline.md) for the Arm
|
| 42 |
Runtime measurement and host context.
|
benchmark/accuracy.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
version: 1.2.0
|
| 2 |
report_type: vision-detection
|
| 3 |
profile: Arm-Optimized
|
| 4 |
-
created_at: '2026-10-
|
| 5 |
context:
|
| 6 |
target_ref: raspberry-pi-5-8gb
|
| 7 |
runtime:
|
|
@@ -14,10 +14,10 @@ context:
|
|
| 14 |
- XNNPACK
|
| 15 |
dataset:
|
| 16 |
name: COCO 128 BY/SA
|
| 17 |
-
version:
|
| 18 |
slice: all
|
| 19 |
sample_count: 128
|
| 20 |
-
reference_url: https://huggingface.co/datasets/Arm/coco-128-by-sa/tree/
|
| 21 |
benchmark:
|
| 22 |
batch_size: 1
|
| 23 |
input_resolution: 640x640
|
|
@@ -25,6 +25,6 @@ context:
|
|
| 25 |
warmup_runs: 0
|
| 26 |
accuracy:
|
| 27 |
sub_task: coco_map
|
| 28 |
-
mAP_50_95_pct: 48.
|
| 29 |
-
mAP_50_pct: 65.
|
| 30 |
-
mAP_75_pct:
|
|
|
|
| 1 |
version: 1.2.0
|
| 2 |
report_type: vision-detection
|
| 3 |
profile: Arm-Optimized
|
| 4 |
+
created_at: '2026-10-22T21:59:43Z'
|
| 5 |
context:
|
| 6 |
target_ref: raspberry-pi-5-8gb
|
| 7 |
runtime:
|
|
|
|
| 14 |
- XNNPACK
|
| 15 |
dataset:
|
| 16 |
name: COCO 128 BY/SA
|
| 17 |
+
version: 97d6ea181e9dd0784db880b705e426ad23d424e1
|
| 18 |
slice: all
|
| 19 |
sample_count: 128
|
| 20 |
+
reference_url: https://huggingface.co/datasets/Arm/coco-128-by-sa/tree/97d6ea181e9dd0784db880b705e426ad23d424e1
|
| 21 |
benchmark:
|
| 22 |
batch_size: 1
|
| 23 |
input_resolution: 640x640
|
|
|
|
| 25 |
warmup_runs: 0
|
| 26 |
accuracy:
|
| 27 |
sub_task: coco_map
|
| 28 |
+
mAP_50_95_pct: 48.6264
|
| 29 |
+
mAP_50_pct: 65.8367
|
| 30 |
+
mAP_75_pct: 53.0182
|
benchmarks/reporting/baseline.md
CHANGED
|
@@ -5,32 +5,31 @@ and `coco-128-by-sa` dataset identified below. Future performance-sensitive
|
|
| 5 |
changes must use the same dataset and validation configuration before comparing
|
| 6 |
their results with this measurement.
|
| 7 |
|
| 8 |
-
The measurement uses the
|
| 9 |
-
below
|
| 10 |
-
|
| 11 |
-
configuration explicitly with `DATA=/path/to/data.yaml`.
|
| 12 |
|
| 13 |
## Measurement
|
| 14 |
|
| 15 |
| Item | Value |
|
| 16 |
| --- | --- |
|
| 17 |
-
| Source revision | `
|
| 18 |
| Runtime library SHA-256 | `6b9978243ed1fd325e5b7b6a2bc616280862ed01edc1f653ee55b088658b60b2` |
|
| 19 |
| Model manifest SHA-256 | `3723a972dc64782fe86577eabde3d02539e10c778ae5ef77b0c5b9bc9f0d1f54` |
|
| 20 |
| Dataset | `coco-128-by-sa` |
|
| 21 |
| Dataset repository | [Arm/coco-128-by-sa](https://huggingface.co/datasets/Arm/coco-128-by-sa) |
|
| 22 |
-
| Dataset Hub revision | [`
|
| 23 |
| Dataset images | 128 |
|
| 24 |
-
| Dataset instances | 1,
|
| 25 |
| Validation configuration SHA-256 | `accc0ff00b364d39c5f209eeb7e53609db634541551b7d4a392b7aa7e1e5f62b` |
|
| 26 |
-
| Selection metadata SHA-256 | `
|
| 27 |
-
| License manifest SHA-256 | `
|
| 28 |
-
| COCO annotations SHA-256 | `
|
| 29 |
| Ultralytics | 8.4.108 |
|
| 30 |
| Ultralytics options | `workers=0 plots=False` |
|
| 31 |
-
| Cache state | Cold; host filesystem caches dropped immediately before
|
| 32 |
-
| Evidence directory | `
|
| 33 |
-
| Recorded | 2026-10-
|
| 34 |
|
| 35 |
`workers=0` keeps Ultralytics dataset loading in the benchmark process instead
|
| 36 |
of starting data-loader subprocesses. It does not change the native runtime's
|
|
@@ -41,14 +40,14 @@ writing validation plots; it does not change detector scoring.
|
|
| 41 |
|
| 42 |
| Measurement | Value |
|
| 43 |
| --- | ---: |
|
| 44 |
-
| Precision | 0.
|
| 45 |
-
| Recall | 0.
|
| 46 |
-
| mAP50 | 0.
|
| 47 |
-
| mAP75 | 0.
|
| 48 |
-
| mAP50-95 | 0.
|
| 49 |
-
| Preprocess | 2.
|
| 50 |
-
| Inference | 36.
|
| 51 |
-
| Postprocess | 3.
|
| 52 |
|
| 53 |
## Host context
|
| 54 |
|
|
@@ -56,12 +55,12 @@ The measurement ran on a Raspberry Pi 5 Model B Rev 1.1 with Linux
|
|
| 56 |
6.12.75+rpt-rpi-2712. All four cores were available, and the CPU frequency
|
| 57 |
policy remained on the `performance` governor at 2.4 GHz.
|
| 58 |
|
| 59 |
-
The host filesystem caches were dropped immediately before
|
| 60 |
-
For the recorded Arm Runtime run, cached and reclaimable data totaled
|
| 61 |
-
at the initial observation and
|
| 62 |
major page faults; swap input and output did not change during the run.
|
| 63 |
|
| 64 |
-
The CPU temperature increased from
|
| 65 |
no current or historical throttling flags before or after the run. Available
|
| 66 |
-
memory decreased from
|
| 67 |
-
was
|
|
|
|
| 5 |
changes must use the same dataset and validation configuration before comparing
|
| 6 |
their results with this measurement.
|
| 7 |
|
| 8 |
+
The measurement uses the published Hugging Face dataset revision identified
|
| 9 |
+
below. Benchmark operators must download that revision and supply its local
|
| 10 |
+
validation configuration explicitly with `DATA=/path/to/data.yaml`.
|
|
|
|
| 11 |
|
| 12 |
## Measurement
|
| 13 |
|
| 14 |
| Item | Value |
|
| 15 |
| --- | --- |
|
| 16 |
+
| Source revision | `4447eb6ed26b860740ef2077673254e604932c43` |
|
| 17 |
| Runtime library SHA-256 | `6b9978243ed1fd325e5b7b6a2bc616280862ed01edc1f653ee55b088658b60b2` |
|
| 18 |
| Model manifest SHA-256 | `3723a972dc64782fe86577eabde3d02539e10c778ae5ef77b0c5b9bc9f0d1f54` |
|
| 19 |
| Dataset | `coco-128-by-sa` |
|
| 20 |
| Dataset repository | [Arm/coco-128-by-sa](https://huggingface.co/datasets/Arm/coco-128-by-sa) |
|
| 21 |
+
| Dataset Hub revision | [`97d6ea181e9dd0784db880b705e426ad23d424e1`](https://huggingface.co/datasets/Arm/coco-128-by-sa/tree/97d6ea181e9dd0784db880b705e426ad23d424e1) |
|
| 22 |
| Dataset images | 128 |
|
| 23 |
+
| Dataset instances | 1,025 |
|
| 24 |
| Validation configuration SHA-256 | `accc0ff00b364d39c5f209eeb7e53609db634541551b7d4a392b7aa7e1e5f62b` |
|
| 25 |
+
| Selection metadata SHA-256 | `18205722c822b59635d4b6c911cef4495b5d688b4b23d84fde6d4e4396cec61f` |
|
| 26 |
+
| License manifest SHA-256 | `a56e45ab8e72a21906332dcfcfe8ba4ece9039c2c0cdc2409952042c57675b96` |
|
| 27 |
+
| COCO annotations SHA-256 | `009ef7e4cd3b533774d8755dbf8a3aa788d5af401b8283e87e27b2cad58cda57` |
|
| 28 |
| Ultralytics | 8.4.108 |
|
| 29 |
| Ultralytics options | `workers=0 plots=False` |
|
| 30 |
+
| Cache state | Cold; host filesystem caches dropped immediately before the run |
|
| 31 |
+
| Evidence directory | `20261022T232118602909Z-4447eb6` |
|
| 32 |
+
| Recorded | 2026-10-22 23:21 UTC |
|
| 33 |
|
| 34 |
`workers=0` keeps Ultralytics dataset loading in the benchmark process instead
|
| 35 |
of starting data-loader subprocesses. It does not change the native runtime's
|
|
|
|
| 40 |
|
| 41 |
| Measurement | Value |
|
| 42 |
| --- | ---: |
|
| 43 |
+
| Precision | 0.7613 |
|
| 44 |
+
| Recall | 0.5805 |
|
| 45 |
+
| mAP50 | 0.6584 |
|
| 46 |
+
| mAP75 | 0.5302 |
|
| 47 |
+
| mAP50-95 | 0.4863 |
|
| 48 |
+
| Preprocess | 2.3 ms/image |
|
| 49 |
+
| Inference | 36.6 ms/image |
|
| 50 |
+
| Postprocess | 3.7 ms/image |
|
| 51 |
|
| 52 |
## Host context
|
| 53 |
|
|
|
|
| 55 |
6.12.75+rpt-rpi-2712. All four cores were available, and the CPU frequency
|
| 56 |
policy remained on the `performance` governor at 2.4 GHz.
|
| 57 |
|
| 58 |
+
The host filesystem caches were dropped immediately before the benchmark.
|
| 59 |
+
For the recorded Arm Runtime run, cached and reclaimable data totaled 1.06 GiB
|
| 60 |
+
at the initial observation and 1.37 GiB afterward. Cold reads caused 1,285
|
| 61 |
major page faults; swap input and output did not change during the run.
|
| 62 |
|
| 63 |
+
The CPU temperature increased from 46.9 °C to 54.0 °C. The firmware reported
|
| 64 |
no current or historical throttling flags before or after the run. Available
|
| 65 |
+
memory decreased from 6.54 GiB to 6.51 GiB, and peak benchmark-process-tree RSS
|
| 66 |
+
was 445 MiB.
|