--- pipeline_tag: object-detection license: apache-2.0 base_model: SenseTime/deformable-detr library_name: zeromodels tags: - keras - zeromodels - deformable-detr - object-detection - arxiv:2010.04159 - pytorch - jax - tf --- # Run Deformable DETR with Keras 3: JAX, PyTorch, or TensorFlow [![GitHub](https://img.shields.io/badge/GitHub-ZeroModels-black?logo=github)](https://github.com/IMvision12/ZeroModels) [![Docs](https://img.shields.io/badge/Docs-Deformable%20DETR-blue)](https://imvision12.github.io/ZeroModels/deformable_detr/) # zeromodels/deformable-detr Paper: [Deformable DETR: Deformable Transformers for End-to-End Object Detection (arXiv:2010.04159)](https://arxiv.org/abs/2010.04159) · [HF Papers](https://huggingface.co/papers/2010.04159) Deformable DETR replaces dense attention with multi-scale deformable attention: each query attends to a few learned sampling points per head per feature level around a reference point. That makes a stride-8 to 64 feature pyramid affordable, improves small objects, and trains about 10x faster than DETR. Variants add iterative box refinement and a two-stage proposal scheme. For more details on the model, please go to the original [model card](https://huggingface.co/SenseTime/deformable-detr). Pure-**Keras 3** conversion of [`SenseTime/deformable-detr`](https://huggingface.co/SenseTime/deformable-detr) for [zeromodels](https://github.com/IMvision12/ZeroModels). One implementation runs unmodified on **TensorFlow / Torch / JAX**. This is an **object detection** checkpoint (`DeformableDETRDetect`): each of the 300 queries predicts COCO class scores (sigmoid) and a box. ## ✨ Quick start ```python import os os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow" from PIL import Image from zeromodels.models.deformable_detr import DeformableDETRDetect, DeformableDETRImageProcessor model = DeformableDETRDetect.from_weights("zeromodels/deformable-detr") processor = DeformableDETRImageProcessor.from_weights("zeromodels/deformable-detr") image = Image.open("your_image.jpg").convert("RGB") inputs = processor(image) output = model(inputs["pixel_values"], training=False) results = processor.post_process_object_detection( output, threshold=0.5, target_sizes=[(image.height, image.width)] )[0] for score, name, box in zip( results["scores"], results["label_names"], results["boxes"] ): print(f"{name}: {float(score):.3f} {box}") ``` Load any Deformable DETR variant the same way with `from_weights("zeromodels/")` (use `DeformableDETRDetect` for this repo): | Variant | Hub | |---|---| | `deformable-detr` | [`zeromodels/deformable-detr`](https://huggingface.co/zeromodels/deformable-detr) | | `deformable-detr-single-scale` | [`zeromodels/deformable-detr-single-scale`](https://huggingface.co/zeromodels/deformable-detr-single-scale) | | `deformable-detr-with-box-refine` | [`zeromodels/deformable-detr-with-box-refine`](https://huggingface.co/zeromodels/deformable-detr-with-box-refine) | | `deformable-detr-with-box-refine-two-stage` | [`zeromodels/deformable-detr-with-box-refine-two-stage`](https://huggingface.co/zeromodels/deformable-detr-with-box-refine-two-stage) | ## Tips - Set `KERAS_BACKEND` **before** importing Keras / zeromodels. - Scores are per-class sigmoids; `post_process_object_detection` keeps the top-100 (query, class) pairs, then applies `threshold`. - The processor resizes aspect-preserving (short side 800, long side at most 1333); the model accepts any input size. - See [Deformable DETR docs](https://imvision12.github.io/ZeroModels/deformable_detr/) and [Loading Weights](https://imvision12.github.io/ZeroModels/loading_weights/). - The upstream safetensors also load directly via the `hf:` prefix, e.g. `DeformableDETRDetect.from_weights("hf:SenseTime/deformable-detr")`. ## Special Thanks A huge thank you to the SenseTime Research Deformable DETR authors for creating and releasing these models. License: Apache 2.0.