nxp/midas-v2-imx

MiDaS v2.1 Small

Overview

MiDaS [1] is a family of deep learning models for monocular depth estimation. Given a single RGB image, the network predicts a relative inverse-depth (disparity) map of the scene. Multiple versions have been created over the years, ranging from CNN-based to Transformer-based architectures. This repository uses the v2.1 Small variant, based on EfficientNet-Lite [2], which is the lightest and fastest member of the family and well suited for edge deployment.

This repository contains the quantized int8 model (float32 input/output) compiled for i.MX 8M Plus, i.MX 93, i.MX 95 and i.MX 952, plus the script used to regenerate these assets (recipe.sh) and an inference example (example.py).

Key Features

• Monocular depth estimation: Predicts a relative inverse-depth (disparity) map from a single RGB image.
• Zero-shot cross-dataset transfer: Trained on 10 diverse datasets, giving strong generalization across scenes.
• Lightweight backbone: EfficientNet-Lite provides a good accuracy / latency trade-off on embedded NPUs.

Model Description

This is a quantized and compiled version of the MiDaS v2.1 Small model, optimized for NXP i.MX processors.

• Base Model: intel/lite-model/midas/v2_1_small/1
• Original Model Authors: Intel ISL
• Original License: MIT
• Modified by: NXP

Modifications

This model is a derivative work with the following changes from the original:

• Quantization: Post-training full integer (int8) quantization of weights and activations, with float32 input and output, using tflite2tensorflow and ~100 calibration images sampled from a commercially-usable ImageNetV2 subset, normalized with data / 255.
• Compilation (i.MX 93): compiled with Vela for the Ethos-U65 NPU, BSP tags lf-6.18.20_2.0.0 and lf-6.18.37_2.1.0.
• Compilation (i.MX 95 / i.MX 952): converted with the eIQ Neutron SDK versions 3.1.3, 3.2.1 and 3.2.3.

Original model available at: TFHub (implementation at isl-org/MiDaS).

Model Information

Information	Value
Input shape	RGB image (1, 256, 256, 3), float32 normalized to [0, 1]
Input example	(Public domain image)
Output shape	Depth map tensor (1, 256, 256, 1), float32 - relative inverse depth
Output example
FLOPS	9.214 GOPs
File size (int8)	~17 MB (assets/quantized_tflite_model/midas_v2_1_small_quant.tflite)
Source framework	PyTorch / TensorFlow Lite

Supported Hardware and Software Configurations

All models run with the eIQ LiteRT (TensorFlow Lite) runtime and the corresponding NPU delegate (VX for i.MX 8M Plus, Ethos-U for i.MX 93, Neutron for i.MX 95 / i.MX 952). Weights and activations are int8; input and output tensors remain float32. The same information is captured in release_manifest.json.

Asset	Runtime	Precision	HW target	BSP & SDK versions	Link to asset
i.MX 8M Plus Quantized	LiteRT	INT8	i.MX 8M Plus	-	Download
i.MX 93 Precompiled	LiteRT	INT8	i.MX 93	BSP lf-6.18.20_2.0.0, VELA_TAG = lf-6.18.20_2.0.0	Download
i.MX 93 Precompiled	LiteRT	INT8	i.MX 93	BSP lf-6.18.37_2.1.0, VELA_TAG = lf-6.18.37_2.1.0	Download
i.MX 95 Precompiled	LiteRT	INT8	i.MX 95	BSP lf-6.18.20_2.0.0, Neutron SDK v3.1.3	Download
i.MX 95 Precompiled	LiteRT	INT8	i.MX 95	BSP lf-6.18.37_2.1.0, Neutron SDK v3.2.1	Download
i.MX 95 Precompiled	LiteRT	INT8	i.MX 95	BSP lf-6.18.37_2.1.0, Neutron SDK v3.2.3	Download
i.MX 952 Precompiled	LiteRT	INT8	i.MX 952	BSP lf-6.18.20_2.0.0, Neutron SDK v3.1.3	Download
i.MX 952 Precompiled	LiteRT	INT8	i.MX 952	BSP lf-6.18.37_2.1.0, Neutron SDK v3.2.1	Download
i.MX 952 Precompiled	LiteRT	INT8	i.MX 952	BSP lf-6.18.37_2.1.0, Neutron SDK v3.2.3	Download

The assets/imx8mplus/midas_v2_1_small_quant.tflite file is the same int8 model as assets/quantized_tflite_model/midas_v2_1_small_quant.tflite (identical sha256). The hardware-independent int8 model and its float32 reference are also kept for host (x86) inference and as inputs to the compilers, under assets/quantized_tflite_model/.

Limitations and Biases

This model inherits all limitations from the original MiDaS v2.1 Small model. Additional limitations:

• Task scope: Produces a relative (not metric/absolute) inverse-depth map; it does not estimate real-world distances.
• Dataset bias: The model is trained on a mix of outdoor/indoor scene datasets (see Training below); depth quality may vary for scene types, lighting conditions or camera characteristics that are under-represented in those datasets.

Training

The original MiDaS v2.1 Small model was trained by Intel ISL on 10 diverse datasets, including ReDWeb, DIML, Movies, MegaDepth, WSVD, ApolloScape, IRS, HRWSI, TartanAir and BlendedMVS, for zero-shot cross-dataset generalization. The training procedure is described in the original repository and associated papers [1, 3]. NXP did not retrain this model; only post-training quantization and compilation were performed.

Download and Run

The recipe installs the eIQ Neutron SDK from the NXP eIQ PyPI Distribution Portal. Configure pip once to use it:

python -m pip config --user set global.index https://eiq.nxp.com/repository
python -m pip config --user set global.index-url https://eiq.nxp.com/repository
python -m pip config --user set global.trusted-host eiq.nxp.com
python -m pip config --user set global.extra-index-url https://pypi.org/simple
python -m pip config --user set global.find-links https://eiq.nxp.com/repository/

To reproduce the quantized and compiled model files from scratch, run:

bash recipe.sh

This will produce:

• assets/quantized_tflite_model/midas_v2_1_small_quant.tflite - int8 model for i.MX 8M Plus (CPU/NPU)
• assets/imx93/<VELA_TAG>/midas_v2_1_small_quant_vela.tflite - Vela-compiled for i.MX 93 Ethos-U65, for each BSP tag in lf-6.18.20_2.0.0, lf-6.18.37_2.1.0
• assets/imx95/<SDK_VERSION>/midas_v2_1_small_quant_converted.tflite - Neutron-compiled for i.MX 95, for each eIQ Neutron SDK version in 3.1.3, 3.2.1, 3.2.3
• assets/imx952/<SDK_VERSION>/midas_v2_1_small_quant_converted.tflite - Neutron-compiled for i.MX 952, for each eIQ Neutron SDK version in 3.1.3, 3.2.1, 3.2.3

The script regenerates every BSP/SDK version listed above in a single run, each built in its own isolated virtual environment to avoid dependency conflicts between versions. Model files for earlier eIQ Neutron SDK versions and Vela BSP tags are kept in the repository for users on earlier BSP.

An example of how to run inference is provided in example.py (run on a PC):

# Run on PC:
pip install opencv-python numpy tensorflow matplotlib
python example.py --model assets/quantized_tflite_model/midas_v2_1_small_quant.tflite --input example_input.jpg

Please refer to the i.MX Machine Learning User Guide for details on running models on each target platform.

Full on-device examples leveraging GStreamer/NNStreamer and this model can be found in nxp-nnstreamer-examples on GitHub.

License

This model, including the quantized and compiled assets and the scripts in this repository, is released under the MIT License (see LICENSE). The original MiDaS v2.1 Small model from Intel ISL is also released under the MIT License (see MODEL_LICENSE).

Citation

If you use this model, please cite both this work and the original model:

• R. Ranftl, K. Lasinger, D. Hafner, K. Schindler and V. Koltun, "Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 3, pp. 1623-1637, 2020.
• EfficientNet-Lite blog post
• R. Ranftl, A. Bochkovskiy and V. Koltun, "Vision Transformers for Dense Prediction," Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.

@article{ranftl2020towards,
  title={Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer},
  author={Ranftl, Ren{\'e} and Lasinger, Katrin and Hafner, David and Schindler, Konrad and Koltun, Vladlen},
  journal={IEEE transactions on pattern analysis and machine intelligence},
  volume={44},
  number={3},
  pages={1623--1637},
  year={2020},
  publisher={IEEE}
}

@inproceedings{ranftl2021vision,
  title={Vision transformers for dense prediction},
  author={Ranftl, Ren{\'e} and Bochkovskiy, Alexey and Koltun, Vladlen},
  booktitle={2021 IEEE/CVF International Conference on Computer Vision (ICCV)},
  pages={12159--12168},
  year={2021},
  organization={IEEE}
}