sebastientaylor commited on
Commit
37de4e0
Β·
verified Β·
1 Parent(s): 5668ad8

Update model card for yolov8-det

Browse files
Files changed (1) hide show
  1. README.md +4 -4
README.md CHANGED
@@ -98,14 +98,14 @@ Row conventions in the table below:
98
  - **Rows without a number** under the metric columns are validation sessions currently in progress, or a session not yet linked to its ONNX FP32 reference. The Studio Source link tracks the current status.
99
  - **Rows whose `Ξ” vs FP32` cell carries a ⚠** are below our accuracy expectations for that platform (more than 10 percentage points under the float reference). The numbers are real measurements on real hardware, reproducible from the linked Studio session, and we publish them as-is; we are investigating the results to make improvements, and the next snapshot of this card will reflect any recovered accuracy.
100
  - **Precision** varies by target: the ONNX reference rows are FP32; macOS CoreML and NVIDIA Jetson TensorRT run FP16; the NXP i.MX 8M Plus, NXP i.MX 95 Neutron, and Hailo NPUs run INT8. The NXP Ara240 DNPU runs a **mixed INT8/INT16** scheme β€” most of the model is INT8, with the box-regression path (and the ops feeding it) promoted to INT16 to improve localization accuracy.
101
- - **Decoder variants.** EdgeFirst ships three INT8 split-decoders. The table headlines the accuracy-recovering ones: **smart** (per-tensor rescaling β€” best accuracy, extra CPU ops add some latency) and, where a smart run is absent, **logical** (the latency-optimized default β€” no CPU overhead, slightly lower accuracy). The **combined** decoder is the *standard-quantization baseline* (equivalent to typical single-scale INT8, and how the reference numbers are produced); it loses the most accuracy β€” especially on segmentation, where box/mask dynamic range collapses under one scale β€” so it is published only as a downloadable reference artifact and in the metrics export, never headlined here. Smart and logical exist precisely to recover that loss.
102
  - **Platform-label suffixes.** `(FRDM)` / `(Phytec)` name the NXP i.MX 95 development board a session ran on. `β€” latency` / `β€” throughput` mark the two pipeline configurations the NXP i.MX 95 Neutron and NXP Ara240 targets run: the **latency** pipeline runs inference serially for the lowest per-frame latency; the **throughput** pipeline runs multiple inference workers for the highest FPS, which raises per-call inference time in exchange. Rows with neither suffix run a single pipeline.
103
  - **End-to-end (ms)** is the sequential per-image latency of the compute pipeline β€” preprocess β†’ inference β†’ postprocess. Image acquisition (camera or file load + JPEG decode) overlaps these stages and is excluded from this figure.
104
  - **Realized FPS vs Theoretical FPS.** **Realized FPS** is the *measured* steady-state throughput β€” the rate at which final results are actually delivered over the full validation pipeline. It normally exceeds `1000 / end-to-end` because the runtime overlaps stages across frames, and it is the **true, priority number**. **Theoretical FPS** (shown with a `~`) is the accelerator's **core** ceiling β€” `1000 / device-compute-time`, the rate the NPU/DNPU could sustain *if it were the only bottleneck* β€” so it is a **possibly-achievable note, not a claim**. It is read from the isolated device-compute stage, which (unlike the host capture/preprocess stages, whose measured time inflates when the pipeline is backpressured) is stable and load-independent. Whether a deployment approaches it depends on the surrounding pipeline, and two levers dominate: (1) **host bottlenecks** β€” these validation runs decode a JPEG per image, whereas a live **camera pipeline** skips that decode and can run closer to the ceiling; and (2) **confidence threshold** β€” validation runs at `0.001` to capture every detection for mAP, which makes NMS/decode heavy, while a deployment threshold of `0.25–0.75` produces far fewer candidate boxes and lighter postprocessing, raising realized FPS toward the ceiling.
105
 
106
  | Size | Platform | mAP@0.5 | Ξ” vs FP32 (pp) | mAP@0.5-0.95 | Inference (ms) | End-to-end (ms) | Realized FPS | Theoretical FPS | Source |
107
  |------|----------|---------|----------------|--------------|----------------|-----------------|--------------|-----------------|--------|
108
- | Nano | ONNX FP32 (x86-64) | 50.49% | +0.00 | 35.83% | 38.01 | 49.96 | 71.6 | ~27 | [v-a46](https://edgefirst.studio/public/validation/v-a46/details?mode=charts) |
109
  | Nano | ONNX FP32 (CUDA) | 50.49% | ref | 35.83% | 8.04 | 13.22 | 350.5 | ~498 | [v-a88](https://edgefirst.studio/public/validation/v-a88/details?mode=charts) |
110
  | Nano | ONNX FP16 (CUDA) | 50.49% | +0.00 | 35.79% | 6.23 | 11.70 | 457.3 | ~535 | [v-a9d](https://edgefirst.studio/public/validation/v-a9d/details?mode=charts) |
111
  | Nano | Apple M2 Max β€” CoreML Neural Engine (FP16) | 50.12% | -0.37 | 35.52% | 1.80 | 5.26 | 791.0 | ~1114 | [v-9cd](https://edgefirst.studio/public/validation/v-9cd/details?mode=charts) |
@@ -119,7 +119,7 @@ Row conventions in the table below:
119
  | Nano | NXP Ara240 (FRDM) β€” throughput | 46.63% | -3.86 | 32.13% | 6.87 | 49.38 | 210.8 | ~298 | [v-a13](https://edgefirst.studio/public/validation/v-a13/details?mode=charts) |
120
  | Nano | Raspberry Pi 5 + Hailo-8L NPU | 49.80% | -0.69 | 35.22% | 13.20 | 24.86 | 64.9 | ~76 | [v-8df](https://edgefirst.studio/public/validation/v-8df/details?mode=charts) |
121
  | Nano | NVIDIA Jetson Orin Nano (TensorRT FP16) | 50.50% | +0.01 | 35.79% | 11.93 | 26.90 | 259.7 | ~274 | [v-91a](https://edgefirst.studio/public/validation/v-91a/details?mode=charts) |
122
- | Small | ONNX FP32 (x86-64) | 59.40% | +0.00 | 43.22% | 94.66 | 106.27 | 29.8 | ~11 | [v-a4d](https://edgefirst.studio/public/validation/v-a4d/details?mode=charts) |
123
  | Small | ONNX FP32 (CUDA) | 59.40% | ref | 43.22% | 15.48 | 20.84 | 204.5 | ~258 | [v-a8f](https://edgefirst.studio/public/validation/v-a8f/details?mode=charts) |
124
  | Small | ONNX FP16 (CUDA) | 59.37% | -0.03 | 43.19% | 10.64 | 16.31 | 299.3 | ~376 | [v-aa4](https://edgefirst.studio/public/validation/v-aa4/details?mode=charts) |
125
  | Small | Apple M2 Max β€” CoreML Neural Engine (FP16) | 58.80% | -0.60 | 42.78% | 5.01 | 8.46 | 342.6 | ~399 | [v-9ed](https://edgefirst.studio/public/validation/v-9ed/details?mode=charts) |
@@ -133,7 +133,7 @@ Row conventions in the table below:
133
  | Small | NXP Ara240 (FRDM) β€” throughput | 56.70% | -2.70 | 39.81% | 10.27 | 24.17 | 142.4 | ~142 | [v-a21](https://edgefirst.studio/public/validation/v-a21/details?mode=charts) |
134
  | Small | Raspberry Pi 5 + Hailo-8L NPU | 58.52% | -0.88 | 42.34% | 30.36 | 45.07 | 28.8 | ~33 | [v-8ec](https://edgefirst.studio/public/validation/v-8ec/details?mode=charts) |
135
  | Small | NVIDIA Jetson Orin Nano (TensorRT FP16) | 59.38% | -0.02 | 43.15% | 24.62 | 37.12 | 157.5 | ~162 | [v-922](https://edgefirst.studio/public/validation/v-922/details?mode=charts) |
136
- | Medium | ONNX FP32 (x86-64) | 64.68% | +0.00 | 48.44% | 243.22 | 255.15 | 11.8 | ~4 | [v-a54](https://edgefirst.studio/public/validation/v-a54/details?mode=charts) |
137
  | Medium | ONNX FP32 (CUDA) | 64.68% | ref | 48.45% | 43.49 | 49.46 | 78.5 | ~92 | [v-a96](https://edgefirst.studio/public/validation/v-a96/details?mode=charts) |
138
  | Medium | ONNX FP16 (CUDA) | 64.68% | +0.00 | 48.40% | 23.13 | 29.44 | 147.7 | ~173 | [v-aab](https://edgefirst.studio/public/validation/v-aab/details?mode=charts) |
139
  | Medium | Apple M2 Max β€” CoreML Neural Engine (FP16) | 63.82% | -0.86 | 47.68% | 13.41 | 17.27 | 140.9 | ~149 | [v-75e](https://edgefirst.studio/public/validation/v-75e/details?mode=charts) |
 
98
  - **Rows without a number** under the metric columns are validation sessions currently in progress, or a session not yet linked to its ONNX FP32 reference. The Studio Source link tracks the current status.
99
  - **Rows whose `Ξ” vs FP32` cell carries a ⚠** are below our accuracy expectations for that platform (more than 10 percentage points under the float reference). The numbers are real measurements on real hardware, reproducible from the linked Studio session, and we publish them as-is; we are investigating the results to make improvements, and the next snapshot of this card will reflect any recovered accuracy.
100
  - **Precision** varies by target: the ONNX reference rows are FP32; macOS CoreML and NVIDIA Jetson TensorRT run FP16; the NXP i.MX 8M Plus, NXP i.MX 95 Neutron, and Hailo NPUs run INT8. The NXP Ara240 DNPU runs a **mixed INT8/INT16** scheme β€” most of the model is INT8, with the box-regression path (and the ops feeding it) promoted to INT16 to improve localization accuracy.
101
+ - **Decoder variants.** EdgeFirst ships three INT8 split-decoders. The table headlines the accuracy-recovering ones: **smart** (per-tensor rescaling β€” best accuracy, extra CPU ops add some latency) and, where a smart run is absent, **logical** (the latency-optimized default β€” no CPU overhead, slightly lower accuracy). The **combined** decoder is the *standard-quantization baseline* (equivalent to typical single-scale INT8, and how the reference numbers are produced); it loses the most accuracy β€” especially on segmentation, where box/mask dynamic range collapses under one scale β€” so it is published only as a downloadable reference artifact and in the metrics export, never headlined here. Smart and logical exist precisely to recover that loss. Full converter documentation: [EdgeFirst model conversion](https://doc.edgefirst.ai/latest/models/conversion/) β€” these are the converters used by this Model Zoo and the EdgeFirst Performance Index report.
102
  - **Platform-label suffixes.** `(FRDM)` / `(Phytec)` name the NXP i.MX 95 development board a session ran on. `β€” latency` / `β€” throughput` mark the two pipeline configurations the NXP i.MX 95 Neutron and NXP Ara240 targets run: the **latency** pipeline runs inference serially for the lowest per-frame latency; the **throughput** pipeline runs multiple inference workers for the highest FPS, which raises per-call inference time in exchange. Rows with neither suffix run a single pipeline.
103
  - **End-to-end (ms)** is the sequential per-image latency of the compute pipeline β€” preprocess β†’ inference β†’ postprocess. Image acquisition (camera or file load + JPEG decode) overlaps these stages and is excluded from this figure.
104
  - **Realized FPS vs Theoretical FPS.** **Realized FPS** is the *measured* steady-state throughput β€” the rate at which final results are actually delivered over the full validation pipeline. It normally exceeds `1000 / end-to-end` because the runtime overlaps stages across frames, and it is the **true, priority number**. **Theoretical FPS** (shown with a `~`) is the accelerator's **core** ceiling β€” `1000 / device-compute-time`, the rate the NPU/DNPU could sustain *if it were the only bottleneck* β€” so it is a **possibly-achievable note, not a claim**. It is read from the isolated device-compute stage, which (unlike the host capture/preprocess stages, whose measured time inflates when the pipeline is backpressured) is stable and load-independent. Whether a deployment approaches it depends on the surrounding pipeline, and two levers dominate: (1) **host bottlenecks** β€” these validation runs decode a JPEG per image, whereas a live **camera pipeline** skips that decode and can run closer to the ceiling; and (2) **confidence threshold** β€” validation runs at `0.001` to capture every detection for mAP, which makes NMS/decode heavy, while a deployment threshold of `0.25–0.75` produces far fewer candidate boxes and lighter postprocessing, raising realized FPS toward the ceiling.
105
 
106
  | Size | Platform | mAP@0.5 | Ξ” vs FP32 (pp) | mAP@0.5-0.95 | Inference (ms) | End-to-end (ms) | Realized FPS | Theoretical FPS | Source |
107
  |------|----------|---------|----------------|--------------|----------------|-----------------|--------------|-----------------|--------|
108
+ | Nano | ONNX FP32 (Intel Core i9-13900F Β· 32-core) | 50.49% | +0.00 | 35.83% | 38.01 | 49.96 | 71.6 | ~79 | [v-a46](https://edgefirst.studio/public/validation/v-a46/details?mode=charts) |
109
  | Nano | ONNX FP32 (CUDA) | 50.49% | ref | 35.83% | 8.04 | 13.22 | 350.5 | ~498 | [v-a88](https://edgefirst.studio/public/validation/v-a88/details?mode=charts) |
110
  | Nano | ONNX FP16 (CUDA) | 50.49% | +0.00 | 35.79% | 6.23 | 11.70 | 457.3 | ~535 | [v-a9d](https://edgefirst.studio/public/validation/v-a9d/details?mode=charts) |
111
  | Nano | Apple M2 Max β€” CoreML Neural Engine (FP16) | 50.12% | -0.37 | 35.52% | 1.80 | 5.26 | 791.0 | ~1114 | [v-9cd](https://edgefirst.studio/public/validation/v-9cd/details?mode=charts) |
 
119
  | Nano | NXP Ara240 (FRDM) β€” throughput | 46.63% | -3.86 | 32.13% | 6.87 | 49.38 | 210.8 | ~298 | [v-a13](https://edgefirst.studio/public/validation/v-a13/details?mode=charts) |
120
  | Nano | Raspberry Pi 5 + Hailo-8L NPU | 49.80% | -0.69 | 35.22% | 13.20 | 24.86 | 64.9 | ~76 | [v-8df](https://edgefirst.studio/public/validation/v-8df/details?mode=charts) |
121
  | Nano | NVIDIA Jetson Orin Nano (TensorRT FP16) | 50.50% | +0.01 | 35.79% | 11.93 | 26.90 | 259.7 | ~274 | [v-91a](https://edgefirst.studio/public/validation/v-91a/details?mode=charts) |
122
+ | Small | ONNX FP32 (Intel Core i9-13900F Β· 32-core) | 59.40% | +0.00 | 43.22% | 94.66 | 106.27 | 29.8 | ~32 | [v-a4d](https://edgefirst.studio/public/validation/v-a4d/details?mode=charts) |
123
  | Small | ONNX FP32 (CUDA) | 59.40% | ref | 43.22% | 15.48 | 20.84 | 204.5 | ~258 | [v-a8f](https://edgefirst.studio/public/validation/v-a8f/details?mode=charts) |
124
  | Small | ONNX FP16 (CUDA) | 59.37% | -0.03 | 43.19% | 10.64 | 16.31 | 299.3 | ~376 | [v-aa4](https://edgefirst.studio/public/validation/v-aa4/details?mode=charts) |
125
  | Small | Apple M2 Max β€” CoreML Neural Engine (FP16) | 58.80% | -0.60 | 42.78% | 5.01 | 8.46 | 342.6 | ~399 | [v-9ed](https://edgefirst.studio/public/validation/v-9ed/details?mode=charts) |
 
133
  | Small | NXP Ara240 (FRDM) β€” throughput | 56.70% | -2.70 | 39.81% | 10.27 | 24.17 | 142.4 | ~142 | [v-a21](https://edgefirst.studio/public/validation/v-a21/details?mode=charts) |
134
  | Small | Raspberry Pi 5 + Hailo-8L NPU | 58.52% | -0.88 | 42.34% | 30.36 | 45.07 | 28.8 | ~33 | [v-8ec](https://edgefirst.studio/public/validation/v-8ec/details?mode=charts) |
135
  | Small | NVIDIA Jetson Orin Nano (TensorRT FP16) | 59.38% | -0.02 | 43.15% | 24.62 | 37.12 | 157.5 | ~162 | [v-922](https://edgefirst.studio/public/validation/v-922/details?mode=charts) |
136
+ | Medium | ONNX FP32 (Intel Core i9-13900F Β· 32-core) | 64.68% | +0.00 | 48.44% | 243.22 | 255.15 | 11.8 | ~12 | [v-a54](https://edgefirst.studio/public/validation/v-a54/details?mode=charts) |
137
  | Medium | ONNX FP32 (CUDA) | 64.68% | ref | 48.45% | 43.49 | 49.46 | 78.5 | ~92 | [v-a96](https://edgefirst.studio/public/validation/v-a96/details?mode=charts) |
138
  | Medium | ONNX FP16 (CUDA) | 64.68% | +0.00 | 48.40% | 23.13 | 29.44 | 147.7 | ~173 | [v-aab](https://edgefirst.studio/public/validation/v-aab/details?mode=charts) |
139
  | Medium | Apple M2 Max β€” CoreML Neural Engine (FP16) | 63.82% | -0.86 | 47.68% | 13.41 | 17.27 | 140.9 | ~149 | [v-75e](https://edgefirst.studio/public/validation/v-75e/details?mode=charts) |