--- license: apache-2.0 tags: - vision - image-detection datasets: - COCO ---
# Deformable DETR for TI EdgeAI ### Deformable Attention for Fast-Converging Transformer Detection [![License](https://img.shields.io/badge/License-Apache%202.0-blue?style=for-the-badge)](https://opensource.org/licenses/Apache-2.0) [![Framework](https://img.shields.io/badge/Framework-ONNX-orange?style=for-the-badge)](https://onnx.ai/) [![Task](https://img.shields.io/badge/Task-Object%20Detection-green?style=for-the-badge)](https://github.com/TexasInstruments/edgeai) [![Dataset](https://img.shields.io/badge/Dataset-COCO-blueviolet?style=for-the-badge)](https://cocodataset.org)
--- ## Overview **Deformable DETR** (Deformable Transformers for End-to-End Object Detection) is a transformer-based detector from SenseTime / fundamentalvision that addresses the slow convergence and limited feature resolution of the original DETR. Its key innovation is a **deformable attention module** that attends to only a small set of key sampling points around a reference point rather than all feature map positions, reducing complexity from **O(H²W²) to O(HW)**. This efficient attention mechanism makes it practical to use **multi-scale feature maps**, which improves detection accuracy — especially on small objects — while training in **10× fewer epochs** than DETR. Five variants are provided, ranging from a lightweight single-scale model to a two-stage design with iterative bounding box refinement. Deformable DETR uses **300 query slots** (vs. 100 in DETR) and **sigmoid focal loss** for classification (no explicit background class); post-processing applies a score threshold rather than softmax + background filtering. --- ## Model Variants | Model | Params | FLOPs | Reference mAP[.5:.95]% | Validated Devices | Config | |-------|--------|-------|--------------|--------------------|--------| | `deformable_detr_single_scale` | 34M | 78G | 39.4 | TDA4VH | [deformable_detr_single_scale_config.yaml](deformable_detr_single_scale_config.yaml) | | `deformable_detr_single_scale_dc5` | 34M | 128G | 41.5 | N/A | N/A | | `deformable_detr` | 40M | 173G | 44.5 | N/A | N/A | | `deformable_detr_plus_iterative_bbox_refinement` | 41M | 173G | 46.2 | N/A | N/A | | `deformable_detr_two_stage` | 41M | 173G | 46.9 | N/A | N/A | **Recommended for edge deployment:** `deformable_detr` (multi-scale, best accuracy/compute trade-off) > mAP values on COCO val2017. All variants use a ResNet-50 backbone pretrained on ImageNet, 800×800 input. > The DC5 variant is disabled for TIDL deployment — TIDL does not support dilated convolutions in ResNet; its `.onnx` is provided for reference only. > Only `deformable_detr_single_scale` currently ships with a validated TIDL config; the remaining variants have no `*_config.yaml` in this folder. --- ## Quick Start ### Prerequisites ```bash # Core dependencies (auto-installed by prepare_model.py if missing) pip install torch>=1.12.0 torchvision>=0.13.0 onnx>=1.14.0 scipy gdown>=5.2.0 # ONNX inference pip install onnxruntime>=1.15.0 ``` ### Export the Model ```bash # List all available variants with accuracy and parameter info python prepare_model.py --list-models # Export the default model (deformable_detr - multi-scale, recommended) python prepare_model.py # Export a specific variant python prepare_model.py --model deformable_detr_single_scale # Export all variants (skips any already exported) python prepare_model.py --model all # Use HuggingFace Hub instead of Google Drive (recommended on corporate networks) python prepare_model.py --method optimum --model all ``` The script automatically: - Installs missing dependencies (torch, torchvision, onnx, scipy, gdown) if not present - Clones the [Deformable-DETR repository](https://github.com/fundamentalvision/Deformable-DETR) into `~/.cache/deformable_detr` (or downloads weights from HuggingFace Hub with `--method optimum`) - Installs a pure-Python fallback for the multi-scale deformable attention module — no CUDA compilation required - Downloads pretrained COCO weights and builds the model with the correct architecture flags - Exports to ONNX (opset 17 by default), simplifies the graph with onnx-simplifier, and fixes float64 nodes for TIDL compatibility - Validates the exported graph and saves it as `.onnx` ### Compile and Infer uing edgeai-tidlrunner > **Note:** Run the commands below from inside the `tidlrunner` directory (the cloned [edgeai-tidlrunner](https://github.com/TexasInstruments/edgeai-tidlrunner) repository), with `--config_path` pointing to this model's config file. **Compile using edgeai-tidlrunner - on PC** ```bash cd /path/to/edgeai-tidlrunner tidlrunner-cli compile --target_device J784S4 \ --config_path /path/to/deformable_detr_single_scale_config.yaml ``` **Run Inference Benchmark - on device** ```bash cd /path/to/edgeai-tidlrunner tidlrunner-cli infer --target_device J784S4 \ --config_path /path/to/deformable_detr_single_scale_config.yaml ``` ### Compile and Infer using edgeai-tidl-tools (Advanced): Follow the instructions at https://github.com/TexasInstruments/edgeai-tidl-tools ### Deploy using edgeai-tidl-tools: Deplyment can be done using **[edgeai-tidl-tools](https://github.com/TexasInstruments/edgeai-tidl-tools)**. For ONNX models, onnxruntime-tidl with TIDL acceleration can be used. Consult the documentation of edgeai-tidl-tools for more details. --- ## Citation ```bibtex @article{zhu2020deformable, title = {Deformable DETR: Deformable Transformers for End-to-End Object Detection}, author = {Zhu, Xizhou and Su, Weijie and Lu, Lewei and Li, Bin and Wang, Xiaogang and Dai, Jifeng}, journal = {arXiv preprint arXiv:2010.04159}, year = {2020} } ``` --- ## 🔗 Resources | Resource | Link | |----------|------| | **Paper** | [arXiv:2010.04159](https://arxiv.org/abs/2010.04159) | | **Source Code** | [fundamentalvision/Deformable-DETR](https://github.com/fundamentalvision/Deformable-DETR) | | **Dataset** | [COCO](https://cocodataset.org) | | **edgeai-tidl-tools** | [GitHub](https://github.com/TexasInstruments/edgeai-tidl-tools) | | **edgeai-tidlrunner** | [GitHub](https://github.com/TexasInstruments/edgeai-tidlrunner) | | **EdgeAI SDK** | [Documentation](https://github.com/TexasInstruments/edgeai/blob/main/edgeai-mpu/readme_sdk.md) | --- ## Related Models
**DETR** Original transformer detector Predecessor to Deformable DETR **RT-DETRv2** Real-time transformer detector Modern DETR-style architecture **RF-DETR** Receptive-field enhanced DETR Recent DETR-family variant **DEIMv2** Improved DETR training recipe Faster convergence, higher accuracy
---
**Maintained by:** Texas Instruments EdgeAI Team **Last Updated:** August 2026