Grounding DINO tiny, fp16, with a text mask for any number of phrases

73f79ab verified 15 days ago

6.54 kB

---
license: apache-2.0
library_name: transformers.js
base_model: IDEA-Research/grounding-dino-tiny
pipeline_tag: zero-shot-object-detection
tags:
  - onnx
  - grounding-dino
  - open-vocabulary-detection
  - transformers.js
  - onnxruntime-web
  - webgpu
---
# Grounding DINO Tiny (ONNX, fp16), multi-phrase

Grounding DINO Tiny as ONNX for [Transformers.js](https://github.com/huggingface/transformers.js),
fp16. Give it an image and a prompt of one or more phrases (`goat. city. sky.`), and it returns
boxes, each with the phrase it matched. [Card Anvil](https://github.com/Card-Anvil)'s Describe
command uses it in the browser, on WebGPU or WebAssembly, to find what to select in card art.

**Why another export.** [`onnx-community/grounding-dino-tiny-ONNX`](https://huggingface.co/onnx-community/grounding-dino-tiny-ONNX)
scores only the **first** phrase of a prompt. Every phrase after it comes out near zero (0.02–0.05
where PyTorch gives 0.5–0.8), so `goat. city. sky.` finds only goats. That export was made with
transformers 4.48, which builds each phrase's text attention mask in a Python loop over the
prompt's special tokens. Tracing unrolled the loop for a one-phrase example prompt.

This export rebuilds that function from tensor comparisons and min/max. Before each export it is
checked against Transformers' own on 140 prompts of 0–6 phrases, so the graph handles any number
of phrases, up to the model's 256 tokens.

It is a drop-in replacement: the same inputs, outputs and file name as onnx-community's, so
Transformers.js loads it the same way.

## Files

| File                     | What                                                                                                    | Size   |
| ------------------------ | ------------------------------------------------------------------------------------------------------- | ------ |
| `onnx/model_fp16.onnx`   | The whole model, fp16 weights, float32 inputs and outputs                                               | 364 MB |
| `config.json`, `preprocessor_config.json`, tokenizer files | Copied unchanged from onnx-community's repo at [`ff690b0a`](https://huggingface.co/onnx-community/grounding-dino-tiny-ONNX/tree/ff690b0a8050566c290287545bd059350f3e9096) | |
| `export_grounding_dino.py` | The script that made it                                                                               |        |

| Inputs                                                                  | Outputs                                                       |
| ----------------------------------------------------------------------- | ------------------------------------------------------------- |
| `pixel_values` [1, 3, 800, 800], `pixel_mask` [1, 800, 800]             | `logits` [1, 900, 256]: each query's score for each token     |
| `input_ids`, `token_type_ids`, `attention_mask` [1, n], n from 4 to 256 | `pred_boxes` [1, 900, 4]: centre x, centre y, width, height, 0–1 |

The processor stretches the image to 800 × 800, with no padding.

## Usage

```js
import {
  AutoProcessor,
  GroundingDinoForObjectDetection,
  RawImage,
} from "@huggingface/transformers";
const id = "CardAnvilModels/grounding-dino-tiny-ONNX";
const model = await GroundingDinoForObjectDetection.from_pretrained(id, {
  dtype: "fp16",
  device: "webgpu", // or "wasm"
});
const processor = await AutoProcessor.from_pretrained(id);
const image = await RawImage.fromURL("https://example.com/art.jpg");
// Lower-case phrases, each ended with a full stop.
const inputs = await processor(image, "goat. city. sky.");
const outputs = await model(inputs);
const [result] = processor.post_process_grounded_object_detection(
  outputs,
  inputs.input_ids,
  {
    box_threshold: 0.35,
    text_threshold: 0.25,
    target_sizes: [[image.height, image.width]],
  },
);
// result.boxes: [x0, y0, x1, y1] in pixels; result.labels: the phrase each matched.
```

In one pass, the phrases compete for boxes, as they do in the original model. Something found on
its own can lose to another phrase that claims the same region.

## How it was checked

With onnxruntime 1.31 on the CPU, against PyTorch (transformers 5.20, fp32) on the same inputs:

- **fp32 export:** within 0.01 of PyTorch, on single phrases and on prompts of three and four
  phrases.
- **This fp16 file:** within 0.01 of PyTorch on single phrases, and close on joint prompts. The
  largest gap seen was about 0.1, on one three-phrase prompt.

In Microsoft Edge, with ONNX Runtime Web 1.31, on 24 images and 98 prompts:

| Measure                                     | onnx-community's | This export                      |
| ------------------------------------------- | ---------------- | -------------------------------- |
| WebGPU, a pass (RTX 4070 Ti SUPER)          | 327 ms           | 347 ms                           |
| WebAssembly, a pass (one thread)            | 13.4 s           | 14.7 s                           |
| Single-phrase boxes                         | —                | the same: median IoU 0.995       |
| Phrases found alone that a joint pass loses | 45 of 68         | 14 of 66, as in PyTorch          |

All 48 `GridSample` nodes (the deformable attention) run on WebGPU.

## How it was made

`export_grounding_dino.py`, with torch 2.14.1, transformers 5.20.2, onnx 1.23.2, onnxscript 0.7.2
and onnxconverter-common 1.16.0:

1. **fp32 export** from [`IDEA-Research/grounding-dino-tiny`](https://huggingface.co/IDEA-Research/grounding-dino-tiny)
   at `a2bb814d`, with the dynamo exporter, opset 18, and a dynamic sequence length. Two changes
   make it export:
   - the text-mask function rebuilt as above (Transformers' uses `torch.isin`, which exports as
     `searchsorted`, which ONNX lacks);
   - `TRANSFORMERS_DISABLE_TORCH_CHECK=1`, since its shape checks export as `_is_all_true`, which
     ONNX also lacks.
2. **fp16 conversion** with `onnxconverter_common`, inputs and outputs kept float32. The converter
   leaves 23 `Cast` nodes aimed at float32; the script retypes them.
## Credits and licence

- Model: [Grounding DINO](https://github.com/IDEA-Research/GroundingDINO) by IDEA Research, from
  _Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection_ (2023),
  through [`IDEA-Research/grounding-dino-tiny`](https://huggingface.co/IDEA-Research/grounding-dino-tiny).
- Config and tokenizer files: [ONNX Community](https://huggingface.co/onnx-community).

Licensed under the Apache License 2.0, as the original model is. See [`LICENSE`](LICENSE).

	---
	license: apache-2.0
	library_name: transformers.js
	base_model: IDEA-Research/grounding-dino-tiny
	pipeline_tag: zero-shot-object-detection
	tags:
	- onnx
	- grounding-dino
	- open-vocabulary-detection
	- transformers.js
	- onnxruntime-web
	- webgpu
	---

	# Grounding DINO Tiny (ONNX, fp16), multi-phrase

	Grounding DINO Tiny as ONNX for [Transformers.js](https://github.com/huggingface/transformers.js),
	fp16. Give it an image and a prompt of one or more phrases (`goat. city. sky.`), and it returns
	boxes, each with the phrase it matched. [Card Anvil](https://github.com/Card-Anvil)'s Describe
	command uses it in the browser, on WebGPU or WebAssembly, to find what to select in card art.

	Why another export. [`onnx-community/grounding-dino-tiny-ONNX`](https://huggingface.co/onnx-community/grounding-dino-tiny-ONNX)
	scores only the first phrase of a prompt. Every phrase after it comes out near zero (0.02–0.05
	where PyTorch gives 0.5–0.8), so `goat. city. sky.` finds only goats. That export was made with
	transformers 4.48, which builds each phrase's text attention mask in a Python loop over the
	prompt's special tokens. Tracing unrolled the loop for a one-phrase example prompt.

	This export rebuilds that function from tensor comparisons and min/max. Before each export it is
	checked against Transformers' own on 140 prompts of 0–6 phrases, so the graph handles any number
	of phrases, up to the model's 256 tokens.

	It is a drop-in replacement: the same inputs, outputs and file name as onnx-community's, so
	Transformers.js loads it the same way.

	## Files

	\| File \| What \| Size \|
	\| ------------------------ \| ------------------------------------------------------------------------------------------------------- \| ------ \|
	\| `onnx/model_fp16.onnx` \| The whole model, fp16 weights, float32 inputs and outputs \| 364 MB \|
	\| `config.json`, `preprocessor_config.json`, tokenizer files \| Copied unchanged from onnx-community's repo at [`ff690b0a`](https://huggingface.co/onnx-community/grounding-dino-tiny-ONNX/tree/ff690b0a8050566c290287545bd059350f3e9096) \| \|
	\| `export_grounding_dino.py` \| The script that made it \| \|

	\| Inputs \| Outputs \|
	\| ----------------------------------------------------------------------- \| ------------------------------------------------------------- \|
	\| `pixel_values` [1, 3, 800, 800], `pixel_mask` [1, 800, 800] \| `logits` [1, 900, 256]: each query's score for each token \|
	\| `input_ids`, `token_type_ids`, `attention_mask` [1, n], n from 4 to 256 \| `pred_boxes` [1, 900, 4]: centre x, centre y, width, height, 0–1 \|

	The processor stretches the image to 800 × 800, with no padding.

	## Usage

	```js
	import {
	AutoProcessor,
	GroundingDinoForObjectDetection,
	RawImage,
	} from "@huggingface/transformers";

	const id = "CardAnvilModels/grounding-dino-tiny-ONNX";
	const model = await GroundingDinoForObjectDetection.from_pretrained(id, {
	dtype: "fp16",
	device: "webgpu", // or "wasm"
	});
	const processor = await AutoProcessor.from_pretrained(id);

	const image = await RawImage.fromURL("https://example.com/art.jpg");
	// Lower-case phrases, each ended with a full stop.
	const inputs = await processor(image, "goat. city. sky.");
	const outputs = await model(inputs);
	const [result] = processor.post_process_grounded_object_detection(
	outputs,
	inputs.input_ids,
	{
	box_threshold: 0.35,
	text_threshold: 0.25,
	target_sizes: [[image.height, image.width]],
	},
	);
	// result.boxes: [x0, y0, x1, y1] in pixels; result.labels: the phrase each matched.
	```

	In one pass, the phrases compete for boxes, as they do in the original model. Something found on
	its own can lose to another phrase that claims the same region.

	## How it was checked

	With onnxruntime 1.31 on the CPU, against PyTorch (transformers 5.20, fp32) on the same inputs:

	- fp32 export: within 0.01 of PyTorch, on single phrases and on prompts of three and four
	phrases.
	- This fp16 file: within 0.01 of PyTorch on single phrases, and close on joint prompts. The
	largest gap seen was about 0.1, on one three-phrase prompt.

	In Microsoft Edge, with ONNX Runtime Web 1.31, on 24 images and 98 prompts:

	\| Measure \| onnx-community's \| This export \|
	\| ------------------------------------------- \| ---------------- \| -------------------------------- \|
	\| WebGPU, a pass (RTX 4070 Ti SUPER) \| 327 ms \| 347 ms \|
	\| WebAssembly, a pass (one thread) \| 13.4 s \| 14.7 s \|
	\| Single-phrase boxes \| — \| the same: median IoU 0.995 \|
	\| Phrases found alone that a joint pass loses \| 45 of 68 \| 14 of 66, as in PyTorch \|

	All 48 `GridSample` nodes (the deformable attention) run on WebGPU.

	## How it was made

	`export_grounding_dino.py`, with torch 2.14.1, transformers 5.20.2, onnx 1.23.2, onnxscript 0.7.2
	and onnxconverter-common 1.16.0:

	1. fp32 export from [`IDEA-Research/grounding-dino-tiny`](https://huggingface.co/IDEA-Research/grounding-dino-tiny)
	at `a2bb814d`, with the dynamo exporter, opset 18, and a dynamic sequence length. Two changes
	make it export:
	- the text-mask function rebuilt as above (Transformers' uses `torch.isin`, which exports as
	`searchsorted`, which ONNX lacks);
	- `TRANSFORMERS_DISABLE_TORCH_CHECK=1`, since its shape checks export as `_is_all_true`, which
	ONNX also lacks.
	2. fp16 conversion with `onnxconverter_common`, inputs and outputs kept float32. The converter
	leaves 23 `Cast` nodes aimed at float32; the script retypes them.

	## Credits and licence

	- Model: [Grounding DINO](https://github.com/IDEA-Research/GroundingDINO) by IDEA Research, from
	_Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection_ (2023),
	through [`IDEA-Research/grounding-dino-tiny`](https://huggingface.co/IDEA-Research/grounding-dino-tiny).
	- Config and tokenizer files: [ONNX Community](https://huggingface.co/onnx-community).

	Licensed under the Apache License 2.0, as the original model is. See [`LICENSE`](LICENSE).