nxp/yolov4-tiny-imx

YOLOv4-tiny

Overview

YOLO (You Only Look Once) is a series of object detection models designed for fast inference, which makes them well suited for edge devices. YOLOv4 was released in 2020 and provides many small improvements over YOLOv3, which add up to create a more precise network at the same speed. YOLOv4-tiny is the lightweight variant of that network.

Given a 416x416 RGB image, the model regresses bounding boxes and class scores (80 COCO classes) at two output resolution levels, with 3 anchor boxes per cell. Bounding box decoding and non-maximum suppression (NMS) are NOT included in the model graph; see example.py for a reference implementation of both steps.

This repository contains the quantized int8 model compiled for NXP i.MX MPUs (i.MX 8M Plus, i.MX 93, i.MX 95, i.MX 952) as LiteRT (.tflite) models, along with the scripts used to regenerate these assets (recipe.sh, export_model.py) and an inference example (example.py).

Model Description

This is a quantized and compiled version of YOLOv4-tiny, converted from the original DarkNet weights, optimized for NXP i.MX MPUs.

• Base Model: YOLOv4-tiny
• Original Model Authors: Alexey Bochkovskiy et al.
• Original License: Public domain (Darknet, see MODEL_LICENSE)
• Modified by: NXP

Modifications

This model is a derivative work with the following changes from the original:

• Conversion: Converted from DarkNet framework to TensorFlow, then quantized to int8 with int8 input / float32 output (export_model.py). 100 random images from the COCO 2017 validation dataset are used as calibration data for the quantization.
• Compilation (i.MX 93): compiled with Vela for the Ethos-U65 NPU (--accelerator-config=ethos-u65-256 --optimise Performance), tags lf-6.18.20_2.0.0 and lf-6.18.37_2.1.0.
• Compilation (i.MX 95 / i.MX 952): converted with the eIQ Neutron SDK versions 3.1.3, 3.2.1 and 3.2.3.
• Post-processing: bounding box decoding and non-maximum suppression are performed outside the model graph (see example.py).

Original model available at: https://github.com/AlexeyAB/darknet/.

Model Information

Information	Value
Input shape	RGB image (1, 416, 416, 3), int8
Input example	(Image source, Public domain)
Output shape	2 tensors of size (26, 26, 255) and (13, 13, 255), float32, containing raw (not decoded) bounding box coordinates and class scores for two resolution levels and 3 anchor boxes per cell. See example.py to decode them.
Output example
FLOPS	6.9G
Number of parameters	6.05M
File size (int8)	5.9 MB (assets/quantized_tflite_model/yolov4-tiny_416_quant.tflite)
Source framework	DarkNet

Supported Hardware and Software Configurations

All models run with the eIQ LiteRT (TensorFlow Lite) runtime and the corresponding NPU delegate (VX for i.MX 8M Plus, Ethos-U for i.MX 93, Neutron for i.MX 95 / i.MX 952). The same information is captured in release_manifest.json.

Asset	Runtime	Precision	HW target	BSP & SDK versions	Link to asset
i.MX 8M Plus Quantized	LiteRT	INT8	i.MX 8M Plus	-	Download
i.MX 93 Precompiled	LiteRT	INT8	i.MX 93	BSP lf-6.18.20_2.0.0, VELA_TAG = lf-6.18.20_2.0.0	Download
i.MX 93 Precompiled	LiteRT	INT8	i.MX 93	BSP lf-6.18.37_2.1.0, VELA_TAG = lf-6.18.37_2.1.0	Download
i.MX 95 Precompiled	LiteRT	INT8	i.MX 95	BSP lf-6.18.20_2.0.0, Neutron SDK v3.1.3	Download
i.MX 95 Precompiled	LiteRT	INT8	i.MX 95	BSP lf-6.18.37_2.1.0, Neutron SDK v3.2.1	Download
i.MX 95 Precompiled	LiteRT	INT8	i.MX 95	BSP lf-6.18.37_2.1.0, Neutron SDK v3.2.3	Download
i.MX 952 Precompiled	LiteRT	INT8	i.MX 952	BSP lf-6.18.20_2.0.0, Neutron SDK v3.1.3	Download
i.MX 952 Precompiled	LiteRT	INT8	i.MX 952	BSP lf-6.18.37_2.1.0, Neutron SDK v3.2.1	Download
i.MX 952 Precompiled	LiteRT	INT8	i.MX 952	BSP lf-6.18.37_2.1.0, Neutron SDK v3.2.3	Download

The hardware-independent int8 model is also provided for host (x86) inference and as the input to the compilers: assets/quantized_tflite_model/yolov4-tiny_416_quant.tflite. The assets/imx8mplus/ file is identical (same sha256) to this hardware-independent model, since the i.MX 8M Plus VeriSilicon NPU uses the same int8 model as-is.

Limitations and Biases

This model inherits all limitations from the original YOLOv4-tiny model. Additional limitations:

• Task scope: Object detection on the 80 COCO classes only; the model does not perform instance segmentation, tracking or classification beyond the detected box.
• Dataset bias: The model is trained on COCO, which is biased towards common, Western, urban scenes and well-lit, unoccluded objects.

Training

The original YOLOv4-tiny model was trained by its authors on the COCO dataset, which features 80 classes, according to the source repository. NXP did not retrain the network; only quantization and compilation were performed. Instructions to re-train the network from scratch can be found in the original repository.

Download and Run

Running the example

Install the dependencies (opencv-python, tensorflow, numpy), then run on a host PC:

python example.py

This loads yolov4-tiny_416_quant.tflite, runs inference on example_input.jpg, decodes the boxes, and saves/displays the result as example_output.jpg.

To run the models on an i.MX MPU target with the NPU delegates, see the i.MX Machine Learning User Guide.

Full on-device examples leveraging GStreamer/NNStreamer and this model can be found in nxp-nnstreamer-examples on GitHub.

Regenerating the assets with recipe.sh

The recipe installs the eIQ Neutron SDK from the NXP eIQ PyPI Distribution Portal. Configure pip once to use it:

python -m pip config --user set global.index https://eiq.nxp.com/repository
python -m pip config --user set global.index-url https://eiq.nxp.com/repository
python -m pip config --user set global.trusted-host eiq.nxp.com
python -m pip config --user set global.extra-index-url https://pypi.org/simple
python -m pip config --user set global.find-links https://eiq.nxp.com/repository/

recipe.sh downloads the original DarkNet weights and the COCO 2017 validation set, converts the model to int8 TensorFlow Lite with export_model.py, then compiles it for i.MX 93 with Vela and for i.MX 95 / i.MX 952 with the eIQ Neutron SDK. The eIQ Neutron SDK version and the Vela tag used are set at the top of the script (EIQ_NEUTRON_SDK_VERSION, IMX_ETHOS_U_VELA_TAG); edit these variables to regenerate other combinations. Run with:

bash recipe.sh

The generated files are:

• assets/quantized_tflite_model/yolov4-tiny_416_quant.tflite - hardware-independent int8 model
• assets/imx8mplus/yolov4-tiny_416_quant.tflite - int8 model for i.MX 8M Plus (identical to the hardware-independent int8 model)
• assets/imx93/<VELA_TAG>/yolov4-tiny_416_quant_vela.tflite - Vela-compiled for i.MX 93
• assets/imx95/<NEUTRON_VER>/yolov4-tiny_416_quant_converted.tflite - Neutron-converted for i.MX 95
• assets/imx952/<NEUTRON_VER>/yolov4-tiny_416_quant_converted.tflite - Neutron-converted for i.MX 952

License

This model, including the quantized and compiled assets and the scripts in this repository, is released under the MIT License (see LICENSE). The original YOLOv4-tiny/DarkNet model is public domain (see MODEL_LICENSE).

Citation

If you use this model, please cite both this work and the original model:

• A. Bochkovskiy, C.-Y. Wang and H.-Y. M. Liao, "YOLOv4: Optimal Speed and Accuracy of Object Detection," arXiv preprint arXiv:2004.10934, 2020.
• J. Redmon and A. Farhadi, "YOLOv3: An Incremental Improvement," arXiv preprint arXiv:1804.02767, 2018.
• T.-Y. Lin et al., "Microsoft COCO: Common Objects in Context," European Conference on Computer Vision (ECCV), Springer, Cham, 2014.

@article{bochkovskiy2020yolov4,
  title={Yolov4: Optimal speed and accuracy of object detection},
  author={Bochkovskiy, Alexey and Wang, Chien-Yao and Liao, Hong-Yuan Mark},
  journal={arXiv preprint arXiv:2004.10934},
  year={2020}
}

@article{redmon2018yolov3,
  title={Yolov3: An incremental improvement},
  author={Redmon, Joseph and Farhadi, Ali},
  journal={arXiv preprint arXiv:1804.02767},
  year={2018}
}

@inproceedings{lin2014microsoft,
  title={Microsoft coco: Common objects in context},
  author={Lin, Tsung-Yi and Maire, Michael and Belongie, Serge and Hays, James and Perona, Pietro and Ramanan, Deva and Doll{\'a}r, Piotr and Zitnick, C Lawrence},
  booktitle={European conference on computer vision},
  pages={740--755},
  year={2014},
  organization={Springer}
}