Grounding DINO and MM-Grounding-DINO (open-vocabulary, Swin-T and Swin-B) (fastforks)

Grounding DINO (Liu et al. 2023, IDEA Research): open-set detection that fuses a Swin image backbone with a BERT text encoder in a DINO-style deformable transformer; each of 900 queries scores every caption token. MM-Grounding-DINO (Zhao et al. 2024, OpenMMLab) is the same network with a scaled, biased contrastive class head and unshared decoder heads, retrained on Objects365, GoldG and V3Det. Pass text=[...] at call time (default: the 80 COCO class names), joined into one caption; a class score is the mean token score of its name. Images are resized to 800 px on the short side (at most 1333 on the long side). The portable module() is the image side with the COCO caption's text features as constants.

This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: https://github.com/kleinicke/fastforks. Kernels for the kernels backend: https://huggingface.co/fastforks/kernels. Original code: https://github.com/huggingface/transformers at c4c4605f9618.

Author weight repos: https://huggingface.co/IDEA-Research/grounding-dino-base, https://huggingface.co/IDEA-Research/grounding-dino-tiny, https://huggingface.co/openmmlab-community/mm_grounding_dino_base_o365v1_goldg_v3det, https://huggingface.co/openmmlab-community/mm_grounding_dino_tiny_o365v1_goldg_v3det.

Task: detection. Tier: experimental. Outputs: boxes, scores, labels. Backends: portable.

Input: one RGB image. Output: boxes, class labels, scores and instance masks.

Variants

Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in card.toml.

variant weights (author link) SHA-256 weights licence commercial use
grounding-dino-base@1 gdino-base.config.json
gdino-base.model.safetensors
bert-uncased.vocab.txt
eda416dae6f4
5548f844c928
07eced375cec
Apache-2.0 yes
grounding-dino-tiny@1 gdino-tiny.config.json
gdino-tiny.model.safetensors
bert-uncased.vocab.txt
eec82c5ab66e
1a2412ef99bd
07eced375cec
Apache-2.0 yes
mm-grounding-dino-base@1 mmgdino-base.config.json
mmgdino-base.model.safetensors
bert-uncased.vocab.txt
3e48e01f93f7
c0b2e1d6ba3f
07eced375cec
Apache-2.0 yes
mm-grounding-dino-tiny@1 mmgdino-tiny.config.json
mmgdino-tiny.model.safetensors
bert-uncased.vocab.txt
cb8ffc83fa08
ad9d9f5f8a94
07eced375cec
Apache-2.0 yes

Licence

Original-code strings, recorded as they are (research-use package; check them before any other use). Code: Apache-2.0.

Training data: Objects365 (train, box annotations) (Grounding DINO-T: Objects365, GoldG (Flickr30k Entities + GQA / Visual Genome images) and Cap4M (web image-caption pairs with pseudo-boxes); GoldG and Cap4M have no dataset id here (paper arXiv:2303.05499 table 2; the authors' model table).).

Licence evidence: https://huggingface.co/IDEA-Research/grounding-dino-tiny/blob/a2bb814dd30d776dcf7e30523b00659f4f141c71/README.md

Input size

any H x W >= 32, padded to a multiple of 32 Tested up to (portable, fp16, RTX 5060 Ti 16 GB): grounding-dino-base 3000x4000 (1.5 GiB peak allocated, 1.7 reserved); grounding-dino-tiny 3000x4000 (1.3 GiB peak allocated, 1.5 reserved); mm-grounding-dino-base 3000x4000 (1.5 GiB peak allocated, 1.7 reserved); mm-grounding-dino-tiny 3000x4000 (1.3 GiB peak allocated, 1.5 reserved)

Speed and memory

Detection, synthetic, 800x1344

model time (ms): original fp32 original portable kernels kernels speed-up: vs original vs portable peak memory (MiB, allocated / reserved): original portable kernels
grounding-dino-base – 230 160 – – – 2895 / 3094 1525 / 1624 –
grounding-dino-tiny – 199 138 – – – 2541 / 2716 1404 / 1468 –
mm-grounding-dino-base – 232 160 – – – 2895 / 3094 1525 / 1624 –
mm-grounding-dino-tiny – 202 138 – – – 2541 / 2716 1405 / 1470 –

Startup and first call

model load (s): original portable kernels first call (s): original portable kernels
grounding-dino-base 4.16 3.29 – 0.99 0.53 –
grounding-dino-tiny 4.06 3.01 – 1.18 0.51 –
mm-grounding-dino-base 4.10 3.33 – 1.03 0.53 –
mm-grounding-dino-tiny 4.17 3.26 – 0.99 0.52 –

torch.compile and TensorRT

model input time: portable compile TensorRT fp16 kernels preparation: compile (cold first call) ONNX + engine build peak MiB (alloc / reserved): portable compile kernels TensorRT (engine + torch)
grounding-dino-base 800x1344 160 ms 131 ms planned for 0.2.1 – 98.4 s – 1525 / 1624 2669 / 2800 – –
grounding-dino-tiny 800x1344 138 ms 111 ms planned for 0.2.1 – 72.5 s – 1404 / 1468 2550 / 2664 – –
mm-grounding-dino-base 800x1344 160 ms 132 ms planned for 0.2.1 – 105 s – 1525 / 1624 2669 / 2800 – –
mm-grounding-dino-tiny 800x1344 138 ms 111 ms planned for 0.2.1 – 71.2 s – 1405 / 1470 2555 / 2666 – –

Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB).

Hardware, protocol and caveats: benchmarks (RTX 5060 Ti 16 GB, sm_120). Original = the original code named above at the precision of portable (fp16, or (fp32) where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's.

Accuracy

On public datasets

pycocotools COCOeval of the top 100 detections per image. Original = the original code named under Details, same checkpoint and protocol.

variant dataset original fp32 original fp16 portable note
grounding-dino-tiny COCO val2017, every 10th image, box mAP 51.08 51.16 51.14
grounding-dino-tiny COCO val2017, all 5000 images, box mAP 48.25 48.25
grounding-dino-base COCO val2017, every 10th image, box mAP 55.67 55.73 55.72
mm-grounding-dino-tiny COCO val2017, every 20th image, box mAP 55.18 54.92
mm-grounding-dino-base COCO val2017, every 20th image, box mAP 57.11 56.94

Measured on NVIDIA GeForce RTX 5060 Ti, torch 2.14.1+cu130, fastforks 84ba91d4 / a981ff86 / aee4b565, 2026-11-12 to 2026-11-13.

Parity with the original code

Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.002 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 0.25 and p99 <= 0.8 (absolute error of the output boxes).

Limitations

  • Requires the optional transformers library (the network is its GroundingDinoForObjectDetection).
  • The original in the benchmarks and goldens is the Hugging Face transformers 5.19.0 port with the authors' weights (IDEA-Research/grounding-dino-, openmmlab-community/mm_grounding_dino_); the official GroundingDINO and mmdetection code (their CUDA deformable-attention ops) was not run.
  • Class names are joined into one caption of at most 256 tokens (about 100 short names); a longer list raises an error instead of being split into several passes.
  • No phrase output: labels index into the given names (the token-level text threshold of the Hugging Face post-processing is not used).
  • grounding-dino-base was trained with COCO (and RefCOCO, ODinW) data: its COCO numbers are not zero-shot.

Usage

from fastforks import detection

model = detection.load("grounding-dino-base")  # mode="auto": the preferred backend from recorded measurements
result = model("photo.jpg")
result.boxes, result.scores, result.labels  # [N, 4], [N], [N]
result.backend  # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16')

Install with pip install fastforks (see https://github.com/kleinicke/fastforks). Other variants: grounding-dino-base, grounding-dino-tiny, mm-grounding-dino-base, mm-grounding-dino-tiny.

Kernels backend

No kernels backend: not built: the portable model now samples per level without copying (deformable attention: grid_sample 11 % of the time, 2.4 ms/call of cat/stack copies removed, portable -20 % time and -26 % peak memory); the remaining cost is the grid_sample gather itself, and a fused multi-scale deformable-attention cubin (Triton prototype 12.85 -> 0.84 ms per call) is the next step but is not built; fp16-accumulate GEMMs gain nothing (tiny 112.5 vs 112.5 ms, base 127.6 vs 131.3 ms at 480x640).

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for fastforks/grounding-dino

Finetuned
(10)
this model

Collection including fastforks/grounding-dino

Paper for fastforks/grounding-dino