Grounding DINO and MM-Grounding-DINO (open-vocabulary, Swin-T and Swin-B) (fastforks)
Grounding DINO (Liu et al. 2023, IDEA Research): open-set detection that fuses a Swin image backbone with a BERT text encoder in a DINO-style deformable transformer; each of 900 queries scores every caption token. MM-Grounding-DINO (Zhao et al. 2024, OpenMMLab) is the same network with a scaled, biased contrastive class head and unshared decoder heads, retrained on Objects365, GoldG and V3Det. Pass text=[...] at call time (default: the 80 COCO class names), joined into one caption; a class score is the mean token score of its name. Images are resized to 800 px on the short side (at most 1333 on the long side). The portable module() is the image side with the COCO caption's text features as constants.
This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: https://github.com/kleinicke/fastforks. Kernels for the kernels backend: https://huggingface.co/fastforks/kernels. Original code: https://github.com/huggingface/transformers at c4c4605f9618.
Author weight repos: https://huggingface.co/IDEA-Research/grounding-dino-base, https://huggingface.co/IDEA-Research/grounding-dino-tiny, https://huggingface.co/openmmlab-community/mm_grounding_dino_base_o365v1_goldg_v3det, https://huggingface.co/openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg_v3det.
Task: detection. Tier: experimental. Outputs: boxes, scores, labels. Backends: portable.
Input: one RGB image. Output: boxes, class labels, scores and instance masks.
Variants
Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in card.toml.
| variant | weights (author link) | SHA-256 | weights licence | commercial use |
|---|---|---|---|---|
grounding-dino-base@1 |
gdino-base.config.json gdino-base.model.safetensors bert-uncased.vocab.txt |
eda416dae6f45548f844c92807eced375cec |
Apache-2.0 | yes |
grounding-dino-tiny@1 |
gdino-tiny.config.json gdino-tiny.model.safetensors bert-uncased.vocab.txt |
eec82c5ab66e1a2412ef99bd07eced375cec |
Apache-2.0 | yes |
mm-grounding-dino-base@1 |
mmgdino-base.config.json mmgdino-base.model.safetensors bert-uncased.vocab.txt |
3e48e01f93f7c0b2e1d6ba3f07eced375cec |
Apache-2.0 | yes |
mm-grounding-dino-tiny@1 |
mmgdino-tiny.config.json mmgdino-tiny.model.safetensors bert-uncased.vocab.txt |
cb8ffc83fa08ad9d9f5f8a9407eced375cec |
Apache-2.0 | yes |
Licence
Original-code strings, recorded as they are (research-use package; check them before any other use). Code: Apache-2.0.
Training data: Objects365 (train, box annotations) (Grounding DINO-T: Objects365, GoldG (Flickr30k Entities + GQA / Visual Genome images) and Cap4M (web image-caption pairs with pseudo-boxes); GoldG and Cap4M have no dataset id here (paper arXiv:2303.05499 table 2; the authors' model table).).
Licence evidence: https://huggingface.co/IDEA-Research/grounding-dino-tiny/blob/a2bb814dd30d776dcf7e30523b00659f4f141c71/README.md
Input size
any H x W >= 32, padded to a multiple of 32
Tested up to (portable, fp16, RTX 5060 Ti 16 GB): grounding-dino-base 3000x4000 (1.5 GiB peak allocated, 1.7 reserved); grounding-dino-tiny 3000x4000 (1.3 GiB peak allocated, 1.5 reserved); mm-grounding-dino-base 3000x4000 (1.5 GiB peak allocated, 1.7 reserved); mm-grounding-dino-tiny 3000x4000 (1.3 GiB peak allocated, 1.5 reserved)
Speed and memory
Detection, synthetic, 800x1344
| model | time (ms): original fp32 | original | portable | kernels | kernels speed-up: vs original | vs portable | peak memory (MiB, allocated / reserved): original | portable | kernels |
|---|---|---|---|---|---|---|---|---|---|
grounding-dino-base |
β | 230 | 160 | β | β | β | 2895 / 3094 | 1525 / 1624 | β |
grounding-dino-tiny |
β | 199 | 138 | β | β | β | 2541 / 2716 | 1404 / 1468 | β |
mm-grounding-dino-base |
β | 232 | 160 | β | β | β | 2895 / 3094 | 1525 / 1624 | β |
mm-grounding-dino-tiny |
β | 202 | 138 | β | β | β | 2541 / 2716 | 1405 / 1470 | β |
Startup and first call
| model | load (s): original | portable | kernels | first call (s): original | portable | kernels |
|---|---|---|---|---|---|---|
grounding-dino-base |
4.16 | 3.29 | β | 0.99 | 0.53 | β |
grounding-dino-tiny |
4.06 | 3.01 | β | 1.18 | 0.51 | β |
mm-grounding-dino-base |
4.10 | 3.33 | β | 1.03 | 0.53 | β |
mm-grounding-dino-tiny |
4.17 | 3.26 | β | 0.99 | 0.52 | β |
torch.compile and TensorRT
| model | input | time: portable | compile | TensorRT fp16 | kernels | preparation: compile (cold first call) | ONNX + engine build | peak MiB (alloc / reserved): portable | compile | kernels | TensorRT (engine + torch) |
|---|---|---|---|---|---|---|---|---|---|---|---|
grounding-dino-base |
800x1344 | 160 ms | 131 ms | planned for 0.2.1 | β | 98.4 s | β | 1525 / 1624 | 2669 / 2800 | β | β |
grounding-dino-tiny |
800x1344 | 138 ms | 111 ms | planned for 0.2.1 | β | 72.5 s | β | 1404 / 1468 | 2550 / 2664 | β | β |
mm-grounding-dino-base |
800x1344 | 160 ms | 132 ms | planned for 0.2.1 | β | 105 s | β | 1525 / 1624 | 2669 / 2800 | β | β |
mm-grounding-dino-tiny |
800x1344 | 138 ms | 111 ms | planned for 0.2.1 | β | 71.2 s | β | 1405 / 1470 | 2555 / 2666 | β | β |
Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).
Hardware, protocol and caveats: benchmarks (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or (fp32) where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.
Accuracy
On public datasets
pycocotools COCOeval of the top 100 detections per image. Original = the original code named under Details, same checkpoint and protocol.
| variant | dataset | original fp32 | original fp16 | portable | note |
|---|---|---|---|---|---|
grounding-dino-tiny |
COCO val2017, every 10th image, box mAP | 51.08 | 51.16 | 51.14 | |
grounding-dino-tiny |
COCO val2017, all 5000 images, box mAP | 48.25 | 48.25 | ||
grounding-dino-base |
COCO val2017, every 10th image, box mAP | 55.67 | 55.73 | 55.72 | |
mm-grounding-dino-tiny |
COCO val2017, every 20th image, box mAP | 55.18 | 54.92 | ||
mm-grounding-dino-base |
COCO val2017, every 20th image, box mAP | 57.11 | 56.94 |
Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 84ba91d4 / a981ff86 / aee4b565, 2026-11-12 to 2026-11-13.
Parity with the original code
Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.002 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.25 and p99 <= 0.8 (absolute error of the output boxes).
Limitations
- Requires the optional transformers library (the network is its
GroundingDinoForObjectDetection). - The original in the benchmarks and goldens is the Hugging Face transformers 5.19.0 port with the authors' weights (IDEA-Research/grounding-dino-, openmmlab-community/mm_grounding_dino_); the official GroundingDINO and mmdetection code (their CUDA deformable-attention ops) was not run.
- Class names are joined into one caption of at most 256 tokens (about 100 short names); a longer list raises an error instead of being split into several passes.
- No phrase output:
labelsindex into the given names (the token-level text threshold of the Hugging Face post-processing is not used). - grounding-dino-base was trained with COCO (and RefCOCO, ODinW) data: its COCO numbers are not zero-shot.
Usage
from fastforks import detection
model = detection.load("grounding-dino-base") # mode="auto": the preferred backend from recorded measurements
result = model("photo.jpg")
result.boxes, result.scores, result.labels # [N, 4], [N], [N]
result.backend # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')
Install with pip install fastforks (see https://github.com/kleinicke/fastforks). Other variants: grounding-dino-base, grounding-dino-tiny, mm-grounding-dino-base, mm-grounding-dino-tiny.
Kernels backend
No kernels backend: not built: the portable model now samples per level without copying (deformable attention: grid_sample 11 % of the time, 2.4 ms/call of cat/stack copies removed, portable -20 % time and -26 % peak memory); the remaining cost is the grid_sample gather itself, and a fused multi-scale deformable-attention cubin (Triton prototype 12.85 -> 0.84 ms per call) is the next step but is not built; fp16-accumulate GEMMs gain nothing (tiny 112.5 vs 112.5 ms, base 127.6 vs 131.3 ms at 480x640).
Model tree for fastforks/grounding-dino
Base model
IDEA-Research/grounding-dino-base