OWL-ViT / README.md

v0.66.0

189a419 verified 1 day ago

7.53 kB

---
library_name: pytorch
license: apache-2.0
tags:
- foundation
- android
pipeline_tag: object-detection

---
![](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/owl_vit/web-assets/model_demo.png)

# OWL-ViT: Optimized for Qualcomm Devices

OWL-ViT (Open-World Localization with Vision Transformers) is an open-vocabulary object detector that uses a CLIP-based ViT-B/32 backbone. Given an image and one or more free-form text queries, the model predicts bounding boxes and confidence scores for each query.

This is based on the implementation of OWL-ViT found [here](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit).
This repository contains pre-exported model files optimized for Qualcomm® devices. You can use the [Qualcomm® AI Hub Models](https://github.com/qualcomm/ai-hub-models/blob/v0.66.0/src/qai_hub_models/models/owl_vit) library to export with custom configurations. More details on model performance across various devices, can be found [here](#performance-summary).

Qualcomm AI Hub Models uses [Qualcomm AI Hub Workbench](https://workbench.aihub.qualcomm.com) to compile, profile, and evaluate this model. [Sign up](https://myaccount.qualcomm.com/signup) to run these models on a hosted Qualcomm® device.

## Getting Started
There are two ways to deploy this model on your device:

### Option 1: Download Pre-Exported Models

Below are pre-exported model assets ready for deployment.

| Runtime | Precision | Chipset | SDK Versions | Download |
|---|---|---|---|---|
| ONNX | float | Universal | QAIRT 2.50, ONNX Runtime 1.30.0 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/owl_vit/releases/v0.66.0/owl_vit-onnx-float.zip)
| QNN_DLC | float | Universal | QAIRT 2.50 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/owl_vit/releases/v0.66.0/owl_vit-qnn_dlc-float.zip)
| TFLITE | float | Universal | QAIRT 2.50 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/owl_vit/releases/v0.66.0/owl_vit-tflite-float.zip)
For more device-specific assets and performance metrics, visit **[OWL-ViT on Qualcomm® AI Hub](https://aihub.qualcomm.com/models/owl_vit)**.
### Option 2: Export with Custom Configurations
Use the [Qualcomm® AI Hub Models](https://github.com/qualcomm/ai-hub-models/blob/v0.66.0/src/qai_hub_models/models/owl_vit) Python library to compile and export the model with your own:
- Custom weights (e.g., fine-tuned checkpoints)
- Custom input shapes
- Target device and runtime configurations
This option is ideal if you need to customize the model beyond the default configuration provided here.
See our repository for [OWL-ViT on GitHub](https://github.com/qualcomm/ai-hub-models/blob/v0.66.0/src/qai_hub_models/models/owl_vit) for usage instructions.
## Model Details
**Model Type:** Model_use_case.object_detection

**Model Stats:**
- Input image resolution: 768x768
- Model checkpoint: google/owlvit-base-patch32
- Model size (float): 613 MB
- Number of parameters: 153M

## Performance Summary
| Model | Runtime | Precision | Chipset | Inference Time (ms) | Peak Memory Range (MB) | Primary Compute Unit
|---|---|---|---|---|---|---
| OWL-ViT | ONNX | float | Snapdragon® 8 Elite Gen 5 For Galaxy Mobile | 20.629 ms | 3 - 485 MB | NPU
| OWL-ViT | ONNX | float | Snapdragon® 8 Elite For Galaxy Mobile | 26.03 ms | 2 - 478 MB | NPU
| OWL-ViT | ONNX | float | Snapdragon® X2 Elite | 19.105 ms | 8 - 8 MB | NPU
| OWL-ViT | ONNX | float | Snapdragon® X Elite | 44.41 ms | 295 - 295 MB | NPU
| OWL-ViT | ONNX | float | Snapdragon® 8 Gen 3 Mobile | 33.513 ms | 3 - 692 MB | NPU
| OWL-ViT | ONNX | float | Snapdragon® 8 Gen 1 Mobile | 55.882 ms | 2 - 676 MB | NPU
| OWL-ViT | ONNX | float | Qualcomm® Dragonwing™ IQ-8275 | 53.285 ms | 7 - 18 MB | NPU
| OWL-ViT | ONNX | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 44.529 ms | 0 - 316 MB | NPU
| OWL-ViT | ONNX | float | Qualcomm® QCS8450 | 55.882 ms | 2 - 676 MB | NPU
| OWL-ViT | ONNX | float | Qualcomm® Dragonwing™ IQ-9075 | 51.137 ms | 6 - 16 MB | NPU
| OWL-ViT | ONNX | float | Qualcomm® Dragonwing™ IQ-X7181 | 44.41 ms | 295 - 295 MB | NPU
| OWL-ViT | ONNX | float | Qualcomm® Dragonwing™ Q-8750 | 26.03 ms | 2 - 478 MB | NPU
| OWL-ViT | QNN_DLC | float | Snapdragon® 8 Elite Gen 5 For Galaxy Mobile | 19.807 ms | 7 - 444 MB | NPU
| OWL-ViT | QNN_DLC | float | Snapdragon® 8 Elite For Galaxy Mobile | 25.537 ms | 5 - 441 MB | NPU
| OWL-ViT | QNN_DLC | float | Snapdragon® X2 Elite | 19.282 ms | 7 - 7 MB | NPU
| OWL-ViT | QNN_DLC | float | Snapdragon® X Elite | 43.139 ms | 7 - 7 MB | NPU
| OWL-ViT | QNN_DLC | float | Snapdragon® 8 Gen 3 Mobile | 32.01 ms | 7 - 615 MB | NPU
| OWL-ViT | QNN_DLC | float | Snapdragon® 8 Gen 1 Mobile | 54.52 ms | 7 - 591 MB | NPU
| OWL-ViT | QNN_DLC | float | Qualcomm® Dragonwing™ IQ-8275 | 52.08 ms | 7 - 16 MB | NPU
| OWL-ViT | QNN_DLC | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 42.869 ms | 0 - 577 MB | NPU
| OWL-ViT | QNN_DLC | float | Qualcomm® SA8775P | 49.16 ms | 1 - 428 MB | NPU
| OWL-ViT | QNN_DLC | float | Qualcomm® SA8650P | 49.16 ms | 1 - 428 MB | NPU
| OWL-ViT | QNN_DLC | float | Qualcomm® SA8255P | 49.16 ms | 1 - 428 MB | NPU
| OWL-ViT | QNN_DLC | float | Qualcomm® QCS8450 | 54.52 ms | 7 - 591 MB | NPU
| OWL-ViT | QNN_DLC | float | Qualcomm® Dragonwing™ IQ-9075 | 49.005 ms | 9 - 18 MB | NPU
| OWL-ViT | QNN_DLC | float | Qualcomm® Dragonwing™ IQ-X7181 | 43.139 ms | 7 - 7 MB | NPU
| OWL-ViT | QNN_DLC | float | Qualcomm® Dragonwing™ Q-8750 | 25.537 ms | 5 - 441 MB | NPU
| OWL-ViT | QNN_DLC | float | Qualcomm® SA8295P | 53.656 ms | 0 - 407 MB | NPU
| OWL-ViT | TFLITE | float | Snapdragon® 8 Elite Gen 5 For Galaxy Mobile | 11.732 ms | 0 - 445 MB | NPU
| OWL-ViT | TFLITE | float | Snapdragon® 8 Elite For Galaxy Mobile | 17.129 ms | 0 - 446 MB | NPU
| OWL-ViT | TFLITE | float | Snapdragon® 8 Gen 3 Mobile | 22.271 ms | 0 - 601 MB | NPU
| OWL-ViT | TFLITE | float | Snapdragon® 8 Gen 1 Mobile | 39.928 ms | 0 - 582 MB | NPU
| OWL-ViT | TFLITE | float | Qualcomm® Dragonwing™ IQ-8275 | 39.766 ms | 0 - 311 MB | NPU
| OWL-ViT | TFLITE | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 31.113 ms | 0 - 3 MB | NPU
| OWL-ViT | TFLITE | float | Qualcomm® SA8775P | 37.339 ms | 0 - 431 MB | NPU
| OWL-ViT | TFLITE | float | Qualcomm® SA8650P | 37.339 ms | 0 - 431 MB | NPU
| OWL-ViT | TFLITE | float | Qualcomm® SA8255P | 37.339 ms | 0 - 431 MB | NPU
| OWL-ViT | TFLITE | float | Qualcomm® QCS8450 | 39.928 ms | 0 - 582 MB | NPU
| OWL-ViT | TFLITE | float | Qualcomm® Dragonwing™ IQ-9075 | 37.574 ms | 0 - 310 MB | NPU
| OWL-ViT | TFLITE | float | Qualcomm® Dragonwing™ Q-8750 | 17.129 ms | 0 - 446 MB | NPU
| OWL-ViT | TFLITE | float | Qualcomm® SA8295P | 41.852 ms | 0 - 409 MB | NPU

## License
* The license for the original implementation of OWL-ViT can be found
  [here](https://github.com/google-research/scenic/blob/main/LICENSE).

## References
* [Simple Open-Vocabulary Object Detection with Vision Transformers](https://arxiv.org/abs/2205.06230)
* [Source Model Implementation](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit)

## Community
* Join [our AI Hub Slack community](https://aihub.qualcomm.com/community/slack) to collaborate, post questions and learn more about on-device AI.
* For questions or feedback please [reach out to us](mailto:ai-hub-support@qti.qualcomm.com).

	---
	library_name: pytorch
	license: apache-2.0
	tags:
	- foundation
	- android
	pipeline_tag: object-detection

	---

	![](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/owl_vit/web-assets/model_demo.png)

	# OWL-ViT: Optimized for Qualcomm Devices

	OWL-ViT (Open-World Localization with Vision Transformers) is an open-vocabulary object detector that uses a CLIP-based ViT-B/32 backbone. Given an image and one or more free-form text queries, the model predicts bounding boxes and confidence scores for each query.

	This is based on the implementation of OWL-ViT found [here](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit).
	This repository contains pre-exported model files optimized for Qualcomm® devices. You can use the [Qualcomm® AI Hub Models](https://github.com/qualcomm/ai-hub-models/blob/v0.66.0/src/qai_hub_models/models/owl_vit) library to export with custom configurations. More details on model performance across various devices, can be found [here](#performance-summary).

	Qualcomm AI Hub Models uses [Qualcomm AI Hub Workbench](https://workbench.aihub.qualcomm.com) to compile, profile, and evaluate this model. [Sign up](https://myaccount.qualcomm.com/signup) to run these models on a hosted Qualcomm® device.

	## Getting Started
	There are two ways to deploy this model on your device:

	### Option 1: Download Pre-Exported Models

	Below are pre-exported model assets ready for deployment.

	\| Runtime \| Precision \| Chipset \| SDK Versions \| Download \|
	\|---\|---\|---\|---\|---\|
	\| ONNX \| float \| Universal \| QAIRT 2.50, ONNX Runtime 1.30.0 \| [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/owl_vit/releases/v0.66.0/owl_vit-onnx-float.zip)
	\| QNN_DLC \| float \| Universal \| QAIRT 2.50 \| [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/owl_vit/releases/v0.66.0/owl_vit-qnn_dlc-float.zip)
	\| TFLITE \| float \| Universal \| QAIRT 2.50 \| [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/owl_vit/releases/v0.66.0/owl_vit-tflite-float.zip)

	For more device-specific assets and performance metrics, visit [OWL-ViT on Qualcomm® AI Hub](https://aihub.qualcomm.com/models/owl_vit).


	### Option 2: Export with Custom Configurations

	Use the [Qualcomm® AI Hub Models](https://github.com/qualcomm/ai-hub-models/blob/v0.66.0/src/qai_hub_models/models/owl_vit) Python library to compile and export the model with your own:
	- Custom weights (e.g., fine-tuned checkpoints)
	- Custom input shapes
	- Target device and runtime configurations

	This option is ideal if you need to customize the model beyond the default configuration provided here.

	See our repository for [OWL-ViT on GitHub](https://github.com/qualcomm/ai-hub-models/blob/v0.66.0/src/qai_hub_models/models/owl_vit) for usage instructions.

	## Model Details

	Model Type: Model_use_case.object_detection

	Model Stats:
	- Input image resolution: 768x768
	- Model checkpoint: google/owlvit-base-patch32
	- Model size (float): 613 MB
	- Number of parameters: 153M

	## Performance Summary
	\| Model \| Runtime \| Precision \| Chipset \| Inference Time (ms) \| Peak Memory Range (MB) \| Primary Compute Unit
	\|---\|---\|---\|---\|---\|---\|---
	\| OWL-ViT \| ONNX \| float \| Snapdragon® 8 Elite Gen 5 For Galaxy Mobile \| 20.629 ms \| 3 - 485 MB \| NPU
	\| OWL-ViT \| ONNX \| float \| Snapdragon® 8 Elite For Galaxy Mobile \| 26.03 ms \| 2 - 478 MB \| NPU
	\| OWL-ViT \| ONNX \| float \| Snapdragon® X2 Elite \| 19.105 ms \| 8 - 8 MB \| NPU
	\| OWL-ViT \| ONNX \| float \| Snapdragon® X Elite \| 44.41 ms \| 295 - 295 MB \| NPU
	\| OWL-ViT \| ONNX \| float \| Snapdragon® 8 Gen 3 Mobile \| 33.513 ms \| 3 - 692 MB \| NPU
	\| OWL-ViT \| ONNX \| float \| Snapdragon® 8 Gen 1 Mobile \| 55.882 ms \| 2 - 676 MB \| NPU
	\| OWL-ViT \| ONNX \| float \| Qualcomm® Dragonwing™ IQ-8275 \| 53.285 ms \| 7 - 18 MB \| NPU
	\| OWL-ViT \| ONNX \| float \| Qualcomm® Dragonwing™ QCS8550 (Proxy) \| 44.529 ms \| 0 - 316 MB \| NPU
	\| OWL-ViT \| ONNX \| float \| Qualcomm® QCS8450 \| 55.882 ms \| 2 - 676 MB \| NPU
	\| OWL-ViT \| ONNX \| float \| Qualcomm® Dragonwing™ IQ-9075 \| 51.137 ms \| 6 - 16 MB \| NPU
	\| OWL-ViT \| ONNX \| float \| Qualcomm® Dragonwing™ IQ-X7181 \| 44.41 ms \| 295 - 295 MB \| NPU
	\| OWL-ViT \| ONNX \| float \| Qualcomm® Dragonwing™ Q-8750 \| 26.03 ms \| 2 - 478 MB \| NPU
	\| OWL-ViT \| QNN_DLC \| float \| Snapdragon® 8 Elite Gen 5 For Galaxy Mobile \| 19.807 ms \| 7 - 444 MB \| NPU
	\| OWL-ViT \| QNN_DLC \| float \| Snapdragon® 8 Elite For Galaxy Mobile \| 25.537 ms \| 5 - 441 MB \| NPU
	\| OWL-ViT \| QNN_DLC \| float \| Snapdragon® X2 Elite \| 19.282 ms \| 7 - 7 MB \| NPU
	\| OWL-ViT \| QNN_DLC \| float \| Snapdragon® X Elite \| 43.139 ms \| 7 - 7 MB \| NPU
	\| OWL-ViT \| QNN_DLC \| float \| Snapdragon® 8 Gen 3 Mobile \| 32.01 ms \| 7 - 615 MB \| NPU
	\| OWL-ViT \| QNN_DLC \| float \| Snapdragon® 8 Gen 1 Mobile \| 54.52 ms \| 7 - 591 MB \| NPU
	\| OWL-ViT \| QNN_DLC \| float \| Qualcomm® Dragonwing™ IQ-8275 \| 52.08 ms \| 7 - 16 MB \| NPU
	\| OWL-ViT \| QNN_DLC \| float \| Qualcomm® Dragonwing™ QCS8550 (Proxy) \| 42.869 ms \| 0 - 577 MB \| NPU
	\| OWL-ViT \| QNN_DLC \| float \| Qualcomm® SA8775P \| 49.16 ms \| 1 - 428 MB \| NPU
	\| OWL-ViT \| QNN_DLC \| float \| Qualcomm® SA8650P \| 49.16 ms \| 1 - 428 MB \| NPU
	\| OWL-ViT \| QNN_DLC \| float \| Qualcomm® SA8255P \| 49.16 ms \| 1 - 428 MB \| NPU
	\| OWL-ViT \| QNN_DLC \| float \| Qualcomm® QCS8450 \| 54.52 ms \| 7 - 591 MB \| NPU
	\| OWL-ViT \| QNN_DLC \| float \| Qualcomm® Dragonwing™ IQ-9075 \| 49.005 ms \| 9 - 18 MB \| NPU
	\| OWL-ViT \| QNN_DLC \| float \| Qualcomm® Dragonwing™ IQ-X7181 \| 43.139 ms \| 7 - 7 MB \| NPU
	\| OWL-ViT \| QNN_DLC \| float \| Qualcomm® Dragonwing™ Q-8750 \| 25.537 ms \| 5 - 441 MB \| NPU
	\| OWL-ViT \| QNN_DLC \| float \| Qualcomm® SA8295P \| 53.656 ms \| 0 - 407 MB \| NPU
	\| OWL-ViT \| TFLITE \| float \| Snapdragon® 8 Elite Gen 5 For Galaxy Mobile \| 11.732 ms \| 0 - 445 MB \| NPU
	\| OWL-ViT \| TFLITE \| float \| Snapdragon® 8 Elite For Galaxy Mobile \| 17.129 ms \| 0 - 446 MB \| NPU
	\| OWL-ViT \| TFLITE \| float \| Snapdragon® 8 Gen 3 Mobile \| 22.271 ms \| 0 - 601 MB \| NPU
	\| OWL-ViT \| TFLITE \| float \| Snapdragon® 8 Gen 1 Mobile \| 39.928 ms \| 0 - 582 MB \| NPU
	\| OWL-ViT \| TFLITE \| float \| Qualcomm® Dragonwing™ IQ-8275 \| 39.766 ms \| 0 - 311 MB \| NPU
	\| OWL-ViT \| TFLITE \| float \| Qualcomm® Dragonwing™ QCS8550 (Proxy) \| 31.113 ms \| 0 - 3 MB \| NPU
	\| OWL-ViT \| TFLITE \| float \| Qualcomm® SA8775P \| 37.339 ms \| 0 - 431 MB \| NPU
	\| OWL-ViT \| TFLITE \| float \| Qualcomm® SA8650P \| 37.339 ms \| 0 - 431 MB \| NPU
	\| OWL-ViT \| TFLITE \| float \| Qualcomm® SA8255P \| 37.339 ms \| 0 - 431 MB \| NPU
	\| OWL-ViT \| TFLITE \| float \| Qualcomm® QCS8450 \| 39.928 ms \| 0 - 582 MB \| NPU
	\| OWL-ViT \| TFLITE \| float \| Qualcomm® Dragonwing™ IQ-9075 \| 37.574 ms \| 0 - 310 MB \| NPU
	\| OWL-ViT \| TFLITE \| float \| Qualcomm® Dragonwing™ Q-8750 \| 17.129 ms \| 0 - 446 MB \| NPU
	\| OWL-ViT \| TFLITE \| float \| Qualcomm® SA8295P \| 41.852 ms \| 0 - 409 MB \| NPU

	## License
	* The license for the original implementation of OWL-ViT can be found
	[here](https://github.com/google-research/scenic/blob/main/LICENSE).

	## References
	* [Simple Open-Vocabulary Object Detection with Vision Transformers](https://arxiv.org/abs/2205.06230)
	* [Source Model Implementation](https://github.com/google-research/scenic/tree/main/scenic/projects/owl_vit)

	## Community
	* Join [our AI Hub Slack community](https://aihub.qualcomm.com/community/slack) to collaborate, post questions and learn more about on-device AI.
	* For questions or feedback please [reach out to us](mailto:ai-hub-support@qti.qualcomm.com).