sebastientaylor commited on
Commit
d0ceeeb
Β·
verified Β·
1 Parent(s): c5690e2

Update model card for yolo11-det

Browse files
Files changed (1) hide show
  1. README.md +11 -8
README.md CHANGED
@@ -102,7 +102,7 @@ Row conventions in the table below:
102
  - **Decoder variants.** EdgeFirst ships three INT8 split-decoders. The table headlines the accuracy-recovering ones: **smart** (per-tensor rescaling β€” best accuracy, extra CPU ops add some latency) and, where a smart run is absent, **logical** (the latency-optimized default β€” no CPU overhead, slightly lower accuracy). The **combined** decoder is the *standard-quantization baseline* (equivalent to typical single-scale INT8, and how the reference numbers are produced); it loses the most accuracy β€” especially on segmentation, where box/mask dynamic range collapses under one scale β€” so it is published only as a downloadable reference artifact and in the metrics export, never headlined here. Smart and logical exist precisely to recover that loss.
103
  - **Platform-label suffixes.** `(FRDM)` / `(Phytec)` name the NXP i.MX 95 development board a session ran on. `β€” latency` / `β€” throughput` mark the two pipeline configurations the NXP i.MX 95 Neutron and NXP Ara240 targets run: the **latency** pipeline runs inference serially for the lowest per-frame latency; the **throughput** pipeline runs multiple inference workers for the highest FPS, which raises per-call inference time in exchange. Rows with neither suffix run a single pipeline.
104
  - **End-to-end (ms)** is the sequential per-image latency of the compute pipeline β€” preprocess β†’ inference β†’ postprocess. Image acquisition (camera or file load + JPEG decode) overlaps these stages and is excluded from this figure.
105
- - **Realized FPS vs Theoretical FPS.** **Realized FPS** is the *measured* steady-state throughput β€” the rate at which final results are actually delivered over the full validation pipeline. It normally exceeds `1000 / end-to-end` because the runtime overlaps stages across frames, and it is the **true, priority number**. **Theoretical FPS** (shown with a `~`) is the per-stage accelerator ceiling β€” the rate the NPU/DNPU could sustain *if it were the only bottleneck* β€” so it is a **possibly-achievable note, not a claim**. Whether a deployment approaches it depends on the surrounding pipeline, and two levers dominate: (1) **host bottlenecks** β€” these validation runs decode a JPEG per image, whereas a live **camera pipeline** skips that decode and can run closer to the ceiling; and (2) **confidence threshold** β€” validation runs at `0.001` to capture every detection for mAP, which makes NMS/decode heavy, while a deployment threshold of `0.25–0.75` produces far fewer candidate boxes and lighter postprocessing, raising realized FPS toward the ceiling.
106
 
107
  | Size | Platform | mAP@0.5 | Ξ” vs FP32 (pp) | mAP@0.5-0.95 | Inference (ms) | End-to-end (ms) | Realized FPS | Theoretical FPS | Source |
108
  |------|----------|---------|----------------|--------------|----------------|-----------------|--------------|-----------------|--------|
@@ -116,8 +116,9 @@ Row conventions in the table below:
116
  | Nano | NXP i.MX 95 + eIQ Neutron NPU (FRDM) β€” latency | Not supported | β€” | β€” | β€” | β€” | β€” | β€” | [v-8a4](https://edgefirst.studio/public/validation/v-8a4/details?mode=charts) |
117
  | Nano | NXP i.MX 95 + eIQ Neutron NPU (FRDM) β€” throughput | Not supported | β€” | β€” | β€” | β€” | β€” | β€” | [v-8a5](https://edgefirst.studio/public/validation/v-8a5/details?mode=charts) |
118
  | Nano | NXP i.MX 95 + eIQ Neutron NPU (Phytec) β€” throughput | Not supported | β€” | β€” | β€” | β€” | β€” | β€” | [v-887](https://edgefirst.studio/public/validation/v-887/details?mode=charts) |
119
- | Nano | NXP Ara240 (FRDM) β€” latency | 50.03% | -3.07 | 34.45% | 6.68 | 28.13 | 196.9 | ~308 | [v-889](https://edgefirst.studio/public/validation/v-889/details?mode=charts) |
120
- | Nano | Raspberry Pi 5 + Hailo-8L NPU | 51.77% | -1.33 | 36.73% | 17.86 | 36.29 | 52.6 | ~112 | [v-8dc](https://edgefirst.studio/public/validation/v-8dc/details?mode=charts) |
 
121
  | Nano | NVIDIA Jetson Orin Nano (TensorRT FP16) | 53.05% | -0.05 | 37.80% | 11.46 | 26.68 | 244.9 | ~262 | [v-917](https://edgefirst.studio/public/validation/v-917/details?mode=charts) |
122
  | Small | ONNX FP32 (AWS Graviton Β· 48-core) | 61.16% | -0.03 | 44.88% | 63.52 | 72.25 | 60.3 | ~63 | [v-71c](https://edgefirst.studio/public/validation/v-71c/details?mode=charts) |
123
  | Small | ONNX FP32 (CUDA) | 61.19% | ref | 44.88% | 15.36 | 20.95 | 205.5 | ~260 | [v-705](https://edgefirst.studio/public/validation/v-705/details?mode=charts) |
@@ -126,8 +127,9 @@ Row conventions in the table below:
126
  | Small | Apple M2 Max β€” CoreML Metal GPU (FP16) | 60.36% | -0.83 | 44.18% | 14.68 | 18.53 | 192.9 | ~204 | [v-9d1](https://edgefirst.studio/public/validation/v-9d1/details?mode=charts) |
127
  | Small | Apple M2 Max β€” CoreML CPU (FP16) | 60.32% | -0.87 | 44.15% | 30.78 | 35.00 | 62.7 | ~65 | [v-9f5](https://edgefirst.studio/public/validation/v-9f5/details?mode=charts) |
128
  | Small | NXP i.MX 8M Plus + VeriSilicon NPU (FRDM) | 3.48% | -57.71 ⚠ | 2.34% | 168.47 | 220.09 | 5.6 | ~6 | [v-9bc](https://edgefirst.studio/public/validation/v-9bc/details?mode=charts) |
129
- | Small | NXP Ara240 (FRDM) β€” latency | 57.15% | -4.04 | 40.11% | 10.28 | 23.98 | 141.3 | ~141 | [v-9b1](https://edgefirst.studio/public/validation/v-9b1/details?mode=charts) |
130
- | Small | Raspberry Pi 5 + Hailo-8L NPU | 59.45% | -1.74 | 43.25% | 36.19 | 51.00 | 25.4 | ~55 | [v-8e7](https://edgefirst.studio/public/validation/v-8e7/details?mode=charts) |
 
131
  | Small | NVIDIA Jetson Orin Nano (TensorRT FP16) | 61.13% | -0.06 | 44.85% | 24.64 | 36.52 | 156.9 | ~162 | [v-91f](https://edgefirst.studio/public/validation/v-91f/details?mode=charts) |
132
  | Medium | ONNX FP32 (AWS Graviton Β· 48-core) | 65.94% | +0.03 | 49.60% | 169.65 | 178.45 | 23.0 | ~24 | [v-71e](https://edgefirst.studio/public/validation/v-71e/details?mode=charts) |
133
  | Medium | ONNX FP32 (CUDA) | 65.91% | ref | 49.59% | 39.57 | 45.08 | 88.4 | ~101 | [v-707](https://edgefirst.studio/public/validation/v-707/details?mode=charts) |
@@ -136,8 +138,9 @@ Row conventions in the table below:
136
  | Medium | Apple M2 Max β€” CoreML Metal GPU (FP16) | 64.75% | -1.16 | 48.59% | 49.69 | 54.31 | 58.8 | ~60 | [v-9da](https://edgefirst.studio/public/validation/v-9da/details?mode=charts) |
137
  | Medium | Apple M2 Max β€” CoreML CPU (FP16) | 64.71% | -1.20 | 48.55% | 65.01 | 69.09 | 30.2 | ~31 | [v-9db](https://edgefirst.studio/public/validation/v-9db/details?mode=charts) |
138
  | Medium | NXP i.MX 8M Plus + VeriSilicon NPU (FRDM) | 54.45% | -11.46 ⚠ | 40.02% | 328.40 | 380.36 | 2.9 | ~3 | [v-98d](https://edgefirst.studio/public/validation/v-98d/details?mode=charts) |
139
- | Medium | NXP Ara240 (FRDM) β€” latency | 62.56% | -3.35 | 45.12% | 24.23 | 37.01 | 47.3 | ~47 | [v-909](https://edgefirst.studio/public/validation/v-909/details?mode=charts) |
140
- | Medium | Raspberry Pi 5 + Hailo-8L NPU | 63.78% | -2.13 | 47.69% | 70.71 | 86.10 | 13.6 | ~28 | [v-90f](https://edgefirst.studio/public/validation/v-90f/details?mode=charts) |
 
141
  | Medium | NVIDIA Jetson Orin Nano (TensorRT FP16) | 65.89% | -0.02 | 49.56% | 50.85 | 63.05 | 77.5 | ~79 | [v-927](https://edgefirst.studio/public/validation/v-927/details?mode=charts) |
142
 
143
  > **⚠ Below expectations β€” under investigation.** The rows marked ⚠ above measure more than 10 percentage points below the same training session's float reference: the model accuracy on that platform is below our expectations. We publish the measured numbers rather than hiding them, and we are investigating the results to make improvements β€” the next snapshot of this card will reflect any recovered accuracy.
@@ -155,7 +158,7 @@ These results are produced by the EdgeFirst on-target validation pipeline:
155
 
156
  Inference latency is reported as the on-accelerator inference time. End-to-end latency is the sequential per-image latency across the compute pipeline β€” preprocessing, inference, and postprocessing; image acquisition (file or camera load and JPEG decode) overlaps these stages and is excluded from this figure.
157
 
158
- Two throughput figures are reported. **Realized FPS** is the *measured* steady-state rate at which final results are emitted, computed from the Perfetto trace as `(N-1) / (t_last - t_first)` over the warmup-trimmed steady-state region; it is the true, priority number and generally exceeds `1000 / end-to-end` because the runtime overlaps stages across frames. **Theoretical FPS** is the per-stage accelerator ceiling (`min_s p_s / t_s` β€” the slowest stage's sustained rate), i.e. the throughput if the accelerator were the *only* bottleneck. It is a possibly-achievable ceiling, not a measured result: reaching it depends on the deployment pipeline. A validation run decodes a JPEG per image and evaluates at a `0.001` confidence threshold (to capture every detection for mAP), both of which load the host and postprocess stages; a production camera pipeline (no JPEG decode) at a deployment threshold of `0.25–0.75` (far fewer candidate boxes through NMS) moves realized throughput toward the theoretical ceiling.
159
 
160
  See [EdgeFirst Studio](https://edgefirst.studio) for the full validation pipeline.
161
 
 
102
  - **Decoder variants.** EdgeFirst ships three INT8 split-decoders. The table headlines the accuracy-recovering ones: **smart** (per-tensor rescaling β€” best accuracy, extra CPU ops add some latency) and, where a smart run is absent, **logical** (the latency-optimized default β€” no CPU overhead, slightly lower accuracy). The **combined** decoder is the *standard-quantization baseline* (equivalent to typical single-scale INT8, and how the reference numbers are produced); it loses the most accuracy β€” especially on segmentation, where box/mask dynamic range collapses under one scale β€” so it is published only as a downloadable reference artifact and in the metrics export, never headlined here. Smart and logical exist precisely to recover that loss.
103
  - **Platform-label suffixes.** `(FRDM)` / `(Phytec)` name the NXP i.MX 95 development board a session ran on. `β€” latency` / `β€” throughput` mark the two pipeline configurations the NXP i.MX 95 Neutron and NXP Ara240 targets run: the **latency** pipeline runs inference serially for the lowest per-frame latency; the **throughput** pipeline runs multiple inference workers for the highest FPS, which raises per-call inference time in exchange. Rows with neither suffix run a single pipeline.
104
  - **End-to-end (ms)** is the sequential per-image latency of the compute pipeline β€” preprocess β†’ inference β†’ postprocess. Image acquisition (camera or file load + JPEG decode) overlaps these stages and is excluded from this figure.
105
+ - **Realized FPS vs Theoretical FPS.** **Realized FPS** is the *measured* steady-state throughput β€” the rate at which final results are actually delivered over the full validation pipeline. It normally exceeds `1000 / end-to-end` because the runtime overlaps stages across frames, and it is the **true, priority number**. **Theoretical FPS** (shown with a `~`) is the accelerator's **core** ceiling β€” `1000 / device-compute-time`, the rate the NPU/DNPU could sustain *if it were the only bottleneck* β€” so it is a **possibly-achievable note, not a claim**. It is read from the isolated device-compute stage, which (unlike the host capture/preprocess stages, whose measured time inflates when the pipeline is backpressured) is stable and load-independent. Whether a deployment approaches it depends on the surrounding pipeline, and two levers dominate: (1) **host bottlenecks** β€” these validation runs decode a JPEG per image, whereas a live **camera pipeline** skips that decode and can run closer to the ceiling; and (2) **confidence threshold** β€” validation runs at `0.001` to capture every detection for mAP, which makes NMS/decode heavy, while a deployment threshold of `0.25–0.75` produces far fewer candidate boxes and lighter postprocessing, raising realized FPS toward the ceiling.
106
 
107
  | Size | Platform | mAP@0.5 | Ξ” vs FP32 (pp) | mAP@0.5-0.95 | Inference (ms) | End-to-end (ms) | Realized FPS | Theoretical FPS | Source |
108
  |------|----------|---------|----------------|--------------|----------------|-----------------|--------------|-----------------|--------|
 
116
  | Nano | NXP i.MX 95 + eIQ Neutron NPU (FRDM) β€” latency | Not supported | β€” | β€” | β€” | β€” | β€” | β€” | [v-8a4](https://edgefirst.studio/public/validation/v-8a4/details?mode=charts) |
117
  | Nano | NXP i.MX 95 + eIQ Neutron NPU (FRDM) β€” throughput | Not supported | β€” | β€” | β€” | β€” | β€” | β€” | [v-8a5](https://edgefirst.studio/public/validation/v-8a5/details?mode=charts) |
118
  | Nano | NXP i.MX 95 + eIQ Neutron NPU (Phytec) β€” throughput | Not supported | β€” | β€” | β€” | β€” | β€” | β€” | [v-887](https://edgefirst.studio/public/validation/v-887/details?mode=charts) |
119
+ | Nano | NXP Ara240 (FRDM) β€” latency | 50.05% | -3.05 | 34.47% | 6.36 | 18.58 | 127.9 | ~157 | [v-a14](https://edgefirst.studio/public/validation/v-a14/details?mode=charts) |
120
+ | Nano | NXP Ara240 (FRDM) β€” throughput | 50.03% | -3.07 | 34.45% | 6.68 | 27.53 | 196.0 | ~308 | [v-a15](https://edgefirst.studio/public/validation/v-a15/details?mode=charts) |
121
+ | Nano | Raspberry Pi 5 + Hailo-8L NPU | 51.77% | -1.33 | 36.73% | 17.86 | 36.29 | 52.6 | ~56 | [v-8dc](https://edgefirst.studio/public/validation/v-8dc/details?mode=charts) |
122
  | Nano | NVIDIA Jetson Orin Nano (TensorRT FP16) | 53.05% | -0.05 | 37.80% | 11.46 | 26.68 | 244.9 | ~262 | [v-917](https://edgefirst.studio/public/validation/v-917/details?mode=charts) |
123
  | Small | ONNX FP32 (AWS Graviton Β· 48-core) | 61.16% | -0.03 | 44.88% | 63.52 | 72.25 | 60.3 | ~63 | [v-71c](https://edgefirst.studio/public/validation/v-71c/details?mode=charts) |
124
  | Small | ONNX FP32 (CUDA) | 61.19% | ref | 44.88% | 15.36 | 20.95 | 205.5 | ~260 | [v-705](https://edgefirst.studio/public/validation/v-705/details?mode=charts) |
 
127
  | Small | Apple M2 Max β€” CoreML Metal GPU (FP16) | 60.36% | -0.83 | 44.18% | 14.68 | 18.53 | 192.9 | ~204 | [v-9d1](https://edgefirst.studio/public/validation/v-9d1/details?mode=charts) |
128
  | Small | Apple M2 Max β€” CoreML CPU (FP16) | 60.32% | -0.87 | 44.15% | 30.78 | 35.00 | 62.7 | ~65 | [v-9f5](https://edgefirst.studio/public/validation/v-9f5/details?mode=charts) |
129
  | Small | NXP i.MX 8M Plus + VeriSilicon NPU (FRDM) | 3.48% | -57.71 ⚠ | 2.34% | 168.47 | 220.09 | 5.6 | ~6 | [v-9bc](https://edgefirst.studio/public/validation/v-9bc/details?mode=charts) |
130
+ | Small | NXP Ara240 (FRDM) β€” latency | 57.16% | -4.03 | 40.13% | 10.17 | 22.06 | 87.2 | ~98 | [v-a22](https://edgefirst.studio/public/validation/v-a22/details?mode=charts) |
131
+ | Small | NXP Ara240 (FRDM) β€” throughput | 57.15% | -4.04 | 40.11% | 10.31 | 24.04 | 141.3 | ~141 | [v-a23](https://edgefirst.studio/public/validation/v-a23/details?mode=charts) |
132
+ | Small | Raspberry Pi 5 + Hailo-8L NPU | 59.45% | -1.74 | 43.25% | 36.19 | 51.00 | 25.4 | ~28 | [v-8e7](https://edgefirst.studio/public/validation/v-8e7/details?mode=charts) |
133
  | Small | NVIDIA Jetson Orin Nano (TensorRT FP16) | 61.13% | -0.06 | 44.85% | 24.64 | 36.52 | 156.9 | ~162 | [v-91f](https://edgefirst.studio/public/validation/v-91f/details?mode=charts) |
134
  | Medium | ONNX FP32 (AWS Graviton Β· 48-core) | 65.94% | +0.03 | 49.60% | 169.65 | 178.45 | 23.0 | ~24 | [v-71e](https://edgefirst.studio/public/validation/v-71e/details?mode=charts) |
135
  | Medium | ONNX FP32 (CUDA) | 65.91% | ref | 49.59% | 39.57 | 45.08 | 88.4 | ~101 | [v-707](https://edgefirst.studio/public/validation/v-707/details?mode=charts) |
 
138
  | Medium | Apple M2 Max β€” CoreML Metal GPU (FP16) | 64.75% | -1.16 | 48.59% | 49.69 | 54.31 | 58.8 | ~60 | [v-9da](https://edgefirst.studio/public/validation/v-9da/details?mode=charts) |
139
  | Medium | Apple M2 Max β€” CoreML CPU (FP16) | 64.71% | -1.20 | 48.55% | 65.01 | 69.09 | 30.2 | ~31 | [v-9db](https://edgefirst.studio/public/validation/v-9db/details?mode=charts) |
140
  | Medium | NXP i.MX 8M Plus + VeriSilicon NPU (FRDM) | 54.45% | -11.46 ⚠ | 40.02% | 328.40 | 380.36 | 2.9 | ~3 | [v-98d](https://edgefirst.studio/public/validation/v-98d/details?mode=charts) |
141
+ | Medium | NXP Ara240 (FRDM) β€” latency | 62.58% | -3.33 | 45.13% | 24.25 | 36.17 | 39.2 | ~41 | [v-a30](https://edgefirst.studio/public/validation/v-a30/details?mode=charts) |
142
+ | Medium | NXP Ara240 (FRDM) β€” throughput | 62.56% | -3.35 | 45.12% | 24.29 | 37.03 | 47.1 | ~47 | [v-a31](https://edgefirst.studio/public/validation/v-a31/details?mode=charts) |
143
+ | Medium | Raspberry Pi 5 + Hailo-8L NPU | 63.78% | -2.13 | 47.69% | 70.71 | 86.10 | 13.6 | ~14 | [v-90f](https://edgefirst.studio/public/validation/v-90f/details?mode=charts) |
144
  | Medium | NVIDIA Jetson Orin Nano (TensorRT FP16) | 65.89% | -0.02 | 49.56% | 50.85 | 63.05 | 77.5 | ~79 | [v-927](https://edgefirst.studio/public/validation/v-927/details?mode=charts) |
145
 
146
  > **⚠ Below expectations β€” under investigation.** The rows marked ⚠ above measure more than 10 percentage points below the same training session's float reference: the model accuracy on that platform is below our expectations. We publish the measured numbers rather than hiding them, and we are investigating the results to make improvements β€” the next snapshot of this card will reflect any recovered accuracy.
 
158
 
159
  Inference latency is reported as the on-accelerator inference time. End-to-end latency is the sequential per-image latency across the compute pipeline β€” preprocessing, inference, and postprocessing; image acquisition (file or camera load and JPEG decode) overlaps these stages and is excluded from this figure.
160
 
161
+ Two throughput figures are reported. **Realized FPS** is the *measured* steady-state rate at which final results are emitted, computed from the Perfetto trace as `(N-1) / (t_last - t_first)` over the warmup-trimmed steady-state region; it is the true, priority number and generally exceeds `1000 / end-to-end` because the runtime overlaps stages across frames. **Theoretical FPS** is the accelerator's **core** ceiling β€” `1000 / device-compute-time`, i.e. the throughput if the accelerator were the *only* bottleneck. It is taken from the isolated device-compute stage (on transfer-split runtimes the trace separates host↔device transfers from device compute), which is load-independent β€” unlike the host capture/preprocess service times, whose measured cost inflates under pipeline backpressure (the same 5000 JPEGs cost ~7.8 ms/frame serialized but far more under throughput backpressure), so the slowest-stage figure would *understate* a fast accelerator. It is a possibly-achievable ceiling, not a measured result: reaching it depends on the deployment pipeline. A validation run decodes a JPEG per image and evaluates at a `0.001` confidence threshold (to capture every detection for mAP), both of which load the host and postprocess stages; a production camera pipeline (no JPEG decode) at a deployment threshold of `0.25–0.75` (far fewer candidate boxes through NMS) moves realized throughput toward the theoretical ceiling.
162
 
163
  See [EdgeFirst Studio](https://edgefirst.studio) for the full validation pipeline.
164