YOLO26 at over 80 FPS with only 2-TOPS!
The NXP i.MX 95 carries a 2-TOPS neural processor, the eIQ® Neutron NPU. Run YOLO26n object detection on it with the stock software, meaning the default Ultralytics export on the current NXP BSP, and you might get up to 40 frames per second. With Au-Zone's EdgeFirst optimized software the same board reaches 84 FPS, with less CPU, while increasing accuracy by 7 mAP.
Short on time? Watch the two-minute video version on YouTube.
The current NXP BSP already includes early EdgeFirst integration. EdgeFirst covers a lot of ground, and this post is about the pieces that aren't in that BSP yet: the zero-copy pipeline, a new delegate that speeds up YOLO26, and a model exported from EdgeFirst Studio in place of the default INT8 export from Ultralytics. They'll work their way into future BSPs, and they're available today at start.edgefirst.ai/nxp.
We measured the stock numbers with standalone C++ applications written for the purpose, so there's no validation session to link. It's a generous baseline: a first attempt will often land below 40.
The EdgeFirst numbers come from a public validation session on the NXP i.MX 95 EVK: 84.1 FPS at 39% CPU and 38.09 mAP, on the full COCO validation set, timed from image file to final detections with four frames in the pipeline at once. (mAP is the standard accuracy score for object detectors. Higher is better.)
An NPU makes AI models run fast, but it only runs the operations it supports, and getting the best performance out of it means making sure it never sits idle. With the stock software it idles a lot. It waits for the next frame to be prepared. It waits again every time the model reaches an operation it can't run and hands the work to the CPU. Our optimization work is finding those waits and removing them. There are four parts to it:
- Feed it faster, by giving each step of the pipeline to the hardware best suited to it.
- Skip the copies, by sharing memory between the steps.
- Split the model in the right place, so its last step runs beside the NPU and no longer holds it up.
- Speed up what the split can't reach, with fused kernels for the blocks left on the CPU.
The first two are about the pipeline around the model. The last two are about the model itself.
Feed it faster
Every camera frame goes through three steps: resize it and convert its colour format, run the model, read the results. With the stock software the CPU does the first and the last, and some of the middle step.
The EdgeFirst HAL hands each step to the hardware best suited to it. On the NXP i.MX 95 that's the Arm Mali GPU for resizing and colour conversion and the NPU for the model. The steps are also pipelined: they overlap, so one frame's results are being read while the next is already on the NPU.
Skip the copies
Look at the links between the steps in that picture. With the stock software every hand-off is a copy: the whole frame is duplicated from one stage's memory into the next, on the CPU, before any real work starts.
Our pipeline removes those copies. The Neutron NPU's input lives in memory that the kernel can share with other devices, so the GPU is handed that memory directly. Our OpenGL preprocessing renders the resized frame straight into the NPU's input tensor, and when the NPU finishes, the decoder reads the results straight out of NPU memory. The frame is never copied on the way in or on the way out.
Zero-copy is also about how and where memory is allocated. The HAL allocates its buffers once, in memory the devices can share, and reuses them for every frame.
Getting there meant changes in three places: the kernel driver, the Neutron delegate and the HAL.
Split the model in the right place
The pipeline now gets frames to the NPU sooner. The model has to cooperate too.
The NPU runs most of YOLO26n, shown in navy. Three regions stay behind on the CPU: two attention blocks in the middle of the network and the decode at the very end. Each time a frame reaches one of them the NPU stops and waits for the CPU. Those three regions set the pace for everything else.
A model like this is always split into parts, some on the NPU and some on the CPU. Handing work to the CPU isn't a fix by itself, though. The stock software already does that, through TensorFlow Lite and XNNPACK, and it's exactly where the time goes. What matters is where the model is split and how fast the CPU part runs. The EdgeFirst HAL is where that speed comes from. Its fused kernels each do the work of several operations in one pass, and they're written with NEON SIMD for the Arm cores on the board.
The three regions need two different fixes, because one sits at the end of the model and the other two are buried inside it. The decode at the end comes first.
To run on an NPU, a model is quantized: its numbers are squeezed into 8 bits. The usual export makes the model's three outputs, one each for small, medium and large objects, share one 8-bit range. The output with the smallest values, which happens to be the one that finds small objects, ends up with about 15% of that range. Fine detail gets rounded away.
The Smart Quantizer in EdgeFirst Studio splits the model just before the decode. The NPU's part ends at the three heads, so each output keeps a range of its own. The decode becomes the CPU's part, and the HAL runs it with FP16 NEON kernels while the NPU has already started the next frame.
Floating point matters at this spot. The box outputs are where a detector is most sensitive to quantization error, far more than the backbone is. Doing that last bit of math in float keeps the detail 8 bits would round away, and the NEON kernels keep it quick.
The model comes out more accurate and faster, which isn't the trade you usually get offered. This step is also where the 7 mAP comes from. The accuracy gap belongs to the default export, and no amount of runtime tuning would have closed it.
The Smart Quantizer runs when you convert a model in EdgeFirst Studio, and it isn't tied to one model. It isn't a quirk of this NPU either: the same split pays off on other NPUs, including ones with 16-bit support. What we add for each model is the testing and validation, and a HAL decoder optimized to match. You could run the float part yourself with standard TensorFlow Lite or ONNX, but you'd give up the speed and have to orchestrate the two halves of the model on your own.
Speed up what the split can't reach
The two attention blocks are a different problem. They sit in the middle of the network, where a split at the end can't reach them.
With the stock software these blocks already run on the CPU. The Neutron delegate leaves them to TensorFlow Lite, four separate operations per block, and XNNPACK, its faster CPU backend, has no kernels for the heavy ones. The open source EdgeFirst delegate takes them over and runs each block as one fused kernel from the HAL. The work stays on the CPU. It just gets done much faster, because one fused NEON kernel beats four generic TensorFlow Lite operations, and that's the key to this step.
The fused kernel is built to reproduce the original results exactly, so accuracy doesn't move at all: 38.09 mAP with the delegate off and on. What you get is speed, and CPU time handed back to the rest of your application.
The fused kernels aren't specific to YOLO either. YOLO models are where they've been validated so far, and more will be covered as we add models to the EdgeFirst Model Zoo. For a customer's own model, we do this optimization work under commercial engagements.
What it adds up to
Forty frames per second to start. Eighty-four once the pipeline feeds the NPU properly and the model no longer stalls it, on a CPU that went from 64% busy to 39%.
Where the time goes now
Every validation session keeps a full trace of the run, so we can put a stopwatch on the finished pipeline. These are averages over the 5,000 images of the same session as the headline numbers.
Reading and decoding a JPEG takes 5.4 ms on the hardware codec, and the GPU resize takes 4.5 ms. Each fused attention kernel runs in 2.5 ms. The HAL decoder needs 3.0 ms for box decode and NMS. The hand-off into the NPU doesn't register at all: the trace puts it under a microsecond, which is what zero-copy looks like on a stopwatch.
The NPU now sets the pace, which is where you want the limit to be. Decoding could keep up with 190 FPS, preprocessing with 220 and the HAL decoder with 350. The NPU finishes a frame every 11.9 ms, and it's idle less than 1% of the time. A single frame takes about 54 ms from image file to detections, because four are in flight at once.
What about JPEG?
This benchmark reads JPEG files, because that's how COCO ships, and the hardware codec on the NXP i.MX 95 decodes them. As the timings show, decoding was never what held the NPU back.
Getting that out of a hardware decoder takes more than calling it. COCO is a messy set of files. The decoder hands back NV12, NV16 or NV21 depending on how each image was encoded, colour is mixed with greyscale, and the sizes vary, many with dimensions that don't line up neatly in memory. The HAL gives every buffer the right alignment and stride for its image and reuses that memory from frame to frame. That handling is the difference between a hardware decoder that performs very well and one that's no faster than decoding on the CPU.
A real camera pipeline usually skips that step. Frames arrive raw, as YUV or RGB, and encoded video is the exception (mostly web cameras, which often send JPEG or H.264). The stack is optimized for both cases. When frames do arrive encoded, two things matter: the software has to use the hardware codec, and it has to map the buffers so the codec, the GPU, the NPU and the other compute units can all share them.
Unleash your NPU
The NPU is now the hardest-working part of the system, which is the point of having one. We don't think we're finished with it either.
These optimizations will be part of future NXP BSPs, and you can have them today. Early access BSPs with the latest EdgeFirst stack are at start.edgefirst.ai/nxp. The recipes to rebuild those BSPs, or adapt them to a custom board, are in EdgeFirstAI/yocto. The rest of the EdgeFirst stack is open source too, under Apache-2.0, at github.com/EdgeFirstAI: the HAL, the kernel patches, the delegate patches, the EdgeFirst delegate and more. The Smart Quantizer is part of EdgeFirst Studio, where you can convert your own model the same way.
It isn't only for this board. The Studio converters, with Smart Quantizer support, cover NVIDIA Jetson (TensorRT), Qualcomm Dragonwing and Snapdragon, Hailo, NXP Ara240, NXP eIQ Neutron and NXP i.MX 8M Plus, with more being added.
YOLO26n on COCO is the public example because anyone can check it. The work itself is the same on your model and your hardware, and it's what we do at Au-Zone Technologies. If you have a model that ought to be running faster than it is, write to us at support@au-zone.com.
Start here
- Browse the EdgeFirst Model Zoo. The YOLO26 detection models used here are in it, already converted for the NXP i.MX 95.
- See the measurements, every one linked to the validation session that produced it.
- Get the early access BSP and try it on your own NXP i.MX 95.
- Convert your model in EdgeFirst Studio. Free tier accounts are available, and they're enough to reproduce these results.
NXP, the NXP logo, i.MX, eIQ, and Neutron are trademarks of NXP® Semiconductors. Ara240 is an NXP product. Hailo is a trademark of Hailo Technologies Ltd. Jetson is a trademark of NVIDIA Corporation. Qualcomm, Snapdragon and Dragonwing are trademarks of Qualcomm Incorporated. All other trademarks are the property of their respective owners.
