Hugging Face's logo

qualcomm
/
OpenAI-Clip

Image Classification
PyTorch
foundation
android
Model card Files
xet
 Community
1
OpenAI-Clip
File size: 11,484 Bytes
8606c41
 
bf34dc2
8606c41
 
 
190fe05
8606c41
 
 
6b0ec56
8606c41
90ee4a0
4db84b8
8606c41
 
90ee4a0
69d8ff8
90ee4a0
 
 
 
 
 
 
 
232d6aa
 
 
 
69d8ff8
 
 
 
 
232d6aa
 
90ee4a0
 
 
 
69d8ff8
90ee4a0
 
 
 
 
 
69d8ff8
90ee4a0
 
 
 
 
 
 
69d8ff8
90ee4a0
69d8ff8
 
90ee4a0
 
 
 
69d8ff8
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
736c554
8606c41
3ebbf3a
 
736c554
8606c41
 
 
 
 
d785667
8606c41
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
---
library_name: pytorch
license: other
tags:
- foundation
- android
pipeline_tag: image-classification

---

![](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/openai_clip/web-assets/model_demo.png)

# OpenAI-Clip: Optimized for Qualcomm Devices

Contrastive Language-Image Pre-Training (CLIP) uses a ViT like transformer to get visual features and a causal language model to get the text features. Both the text and visual features can then be used for a variety of zero-shot learning tasks.

This is based on the implementation of OpenAI-Clip found [here](https://github.com/openai/CLIP/).
This repository contains pre-exported model files optimized for Qualcomm® devices. You can use the [Qualcomm® AI Hub Models](https://github.com/qualcomm/ai-hub-models/blob/v0.60.0/src/qai_hub_models/models/openai_clip) library to export with custom configurations. More details on model performance across various devices, can be found [here](#performance-summary).

Qualcomm AI Hub Models uses [Qualcomm AI Hub Workbench](https://workbench.aihub.qualcomm.com) to compile, profile, and evaluate this model. [Sign up](https://myaccount.qualcomm.com/signup) to run these models on a hosted Qualcomm® device.

## Getting Started
There are two ways to deploy this model on your device:

### Option 1: Download Pre-Exported Models

Below are pre-exported model assets ready for deployment.

| Runtime | Precision | Chipset | SDK Versions | Download |
|---|---|---|---|---|
| ONNX | float | Universal | QAIRT 2.45, ONNX Runtime 1.27.1 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/openai_clip/releases/v0.60.0/openai_clip-onnx-float.zip)
| ONNX | w8a16 | Universal | QAIRT 2.45, ONNX Runtime 1.27.1 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/openai_clip/releases/v0.60.0/openai_clip-onnx-w8a16.zip)
| QNN_DLC | float | Universal | QAIRT 2.45 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/openai_clip/releases/v0.60.0/openai_clip-qnn_dlc-float.zip)
| QNN_DLC | w8a16 | Universal | QAIRT 2.45 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/openai_clip/releases/v0.60.0/openai_clip-qnn_dlc-w8a16.zip)
| TFLITE | float | Universal | QAIRT 2.45 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/openai_clip/releases/v0.60.0/openai_clip-tflite-float.zip)

For more device-specific assets and performance metrics, visit **[OpenAI-Clip on Qualcomm® AI Hub](https://aihub.qualcomm.com/models/openai_clip)**.


### Option 2: Export with Custom Configurations

Use the [Qualcomm® AI Hub Models](https://github.com/qualcomm/ai-hub-models/blob/v0.60.0/src/qai_hub_models/models/openai_clip) Python library to compile and export the model with your own:
- Custom weights (e.g., fine-tuned checkpoints)
- Custom input shapes
- Target device and runtime configurations

This option is ideal if you need to customize the model beyond the default configuration provided here.

See our repository for [OpenAI-Clip on GitHub](https://github.com/qualcomm/ai-hub-models/blob/v0.60.0/src/qai_hub_models/models/openai_clip) for usage instructions.

## Model Details

**Model Type:** Model_use_case.image_classification

**Model Stats:**
- Image input resolution: 224x224
- Model checkpoint: ViT-B/16
- Model size (float): 571 MB
- Number of parameters: 150M
- Text context length: 77

## Performance Summary
| Model | Runtime | Precision | Chipset | Inference Time (ms) | Peak Memory Range (MB) | Primary Compute Unit
|---|---|---|---|---|---|---
| OpenAI-Clip | ONNX | float | Snapdragon® X2 Elite | 15.041 ms | 2 - 2 MB | NPU
| OpenAI-Clip | ONNX | float | Snapdragon® X Elite | 33.087 ms | 294 - 294 MB | NPU
| OpenAI-Clip | ONNX | float | Snapdragon® 8 Gen 3 Mobile | 21.543 ms | 0 - 1185 MB | NPU
| OpenAI-Clip | ONNX | float | Snapdragon® 8 Gen 1 Mobile | 47.884 ms | 1 - 894 MB | NPU
| OpenAI-Clip | ONNX | float | Qualcomm® Dragonwing™ IQ-8275 | 35.963 ms | 0 - 4 MB | NPU
| OpenAI-Clip | ONNX | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 31.992 ms | 1 - 4 MB | NPU
| OpenAI-Clip | ONNX | float | Qualcomm® QCS8450 | 47.884 ms | 1 - 894 MB | NPU
| OpenAI-Clip | ONNX | float | Qualcomm® Dragonwing™ IQ-9075 | 47.242 ms | 0 - 4 MB | NPU
| OpenAI-Clip | ONNX | float | Qualcomm® Dragonwing™ IQ-X7181 | 33.087 ms | 294 - 294 MB | NPU
| OpenAI-Clip | ONNX | float | Qualcomm® Dragonwing™ Q-8750 | 16.846 ms | 1 - 958 MB | NPU
| OpenAI-Clip | ONNX | float | Snapdragon® 8 Elite Mobile | 16.846 ms | 1 - 958 MB | NPU
| OpenAI-Clip | ONNX | float | Snapdragon® 8 Elite Gen 5 Mobile | 13.451 ms | 1 - 807 MB | NPU
| OpenAI-Clip | ONNX | w8a16 | Snapdragon® X2 Elite | 13.395 ms | 1 - 1 MB | NPU
| OpenAI-Clip | ONNX | w8a16 | Snapdragon® X Elite | 27.143 ms | 174 - 174 MB | NPU
| OpenAI-Clip | ONNX | w8a16 | Snapdragon® 8 Gen 3 Mobile | 18.713 ms | 0 - 750 MB | NPU
| OpenAI-Clip | ONNX | w8a16 | Snapdragon® 8 Gen 1 Mobile | 39.887 ms | 0 - 757 MB | NPU
| OpenAI-Clip | ONNX | w8a16 | Qualcomm® Dragonwing™ QCS6490 | 145.153 ms | 1 - 4 MB | NPU
| OpenAI-Clip | ONNX | w8a16 | Qualcomm® Dragonwing™ IQ-8275 | 24.715 ms | 0 - 4 MB | NPU
| OpenAI-Clip | ONNX | w8a16 | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 26.341 ms | 0 - 4 MB | NPU
| OpenAI-Clip | ONNX | w8a16 | Qualcomm® QCS8450 | 39.887 ms | 0 - 757 MB | NPU
| OpenAI-Clip | ONNX | w8a16 | Qualcomm® Dragonwing™ IQ-9075 | 28.04 ms | 0 - 3 MB | NPU
| OpenAI-Clip | ONNX | w8a16 | Qualcomm® Dragonwing™ IQ-X7181 | 27.143 ms | 174 - 174 MB | NPU
| OpenAI-Clip | ONNX | w8a16 | Qualcomm® Dragonwing™ Q-6690 | 303.476 ms | 1 - 667 MB | NPU
| OpenAI-Clip | ONNX | w8a16 | Qualcomm® Dragonwing™ Q-7790 | 37.384 ms | 1 - 670 MB | NPU
| OpenAI-Clip | ONNX | w8a16 | Qualcomm® Dragonwing™ Q-8750 | 14.885 ms | 0 - 641 MB | NPU
| OpenAI-Clip | ONNX | w8a16 | Snapdragon® 8 Elite Mobile | 14.885 ms | 0 - 641 MB | NPU
| OpenAI-Clip | ONNX | w8a16 | Snapdragon® 8 Elite Gen 5 Mobile | 12.908 ms | 0 - 684 MB | NPU
| OpenAI-Clip | ONNX | w8a16 | Snapdragon® 7 Gen 4 Mobile | 37.384 ms | 1 - 670 MB | NPU
| OpenAI-Clip | QNN_DLC | float | Snapdragon® X2 Elite | 14.657 ms | 1 - 1 MB | NPU
| OpenAI-Clip | QNN_DLC | float | Snapdragon® X Elite | 33.277 ms | 1 - 1 MB | NPU
| OpenAI-Clip | QNN_DLC | float | Snapdragon® 8 Gen 3 Mobile | 21.772 ms | 0 - 924 MB | NPU
| OpenAI-Clip | QNN_DLC | float | Snapdragon® 8 Gen 1 Mobile | 46.455 ms | 0 - 854 MB | NPU
| OpenAI-Clip | QNN_DLC | float | Qualcomm® Dragonwing™ IQ-8275 | 36.428 ms | 1 - 3 MB | NPU
| OpenAI-Clip | QNN_DLC | float | Qualcomm® Dragonwing™ IQ-8275 | 98.999 ms | 1 - 704 MB | NPU
| OpenAI-Clip | QNN_DLC | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 31.465 ms | 1 - 2 MB | NPU
| OpenAI-Clip | QNN_DLC | float | Qualcomm® SA8775P | 36.693 ms | 1 - 701 MB | NPU
| OpenAI-Clip | QNN_DLC | float | Qualcomm® SA8650P | 36.693 ms | 1 - 701 MB | NPU
| OpenAI-Clip | QNN_DLC | float | Qualcomm® SA8255P | 36.693 ms | 1 - 701 MB | NPU
| OpenAI-Clip | QNN_DLC | float | Qualcomm® QCS8450 | 46.455 ms | 0 - 854 MB | NPU
| OpenAI-Clip | QNN_DLC | float | Qualcomm® Dragonwing™ IQ-9075 | 36.921 ms | 1 - 3 MB | NPU
| OpenAI-Clip | QNN_DLC | float | Qualcomm® Dragonwing™ IQ-X7181 | 33.277 ms | 1 - 1 MB | NPU
| OpenAI-Clip | QNN_DLC | float | Qualcomm® Dragonwing™ Q-8750 | 17.214 ms | 1 - 694 MB | NPU
| OpenAI-Clip | QNN_DLC | float | Qualcomm® SA7255P | 98.999 ms | 1 - 704 MB | NPU
| OpenAI-Clip | QNN_DLC | float | Qualcomm® SA8295P | 38.524 ms | 1 - 798 MB | NPU
| OpenAI-Clip | QNN_DLC | float | Snapdragon® 8 Elite Mobile | 17.214 ms | 1 - 694 MB | NPU
| OpenAI-Clip | QNN_DLC | float | Snapdragon® 8 Elite Gen 5 Mobile | 13.331 ms | 0 - 704 MB | NPU
| OpenAI-Clip | QNN_DLC | w8a16 | Snapdragon® X2 Elite | 14.407 ms | 0 - 0 MB | NPU
| OpenAI-Clip | QNN_DLC | w8a16 | Snapdragon® X Elite | 33.346 ms | 1 - 1 MB | NPU
| OpenAI-Clip | QNN_DLC | w8a16 | Snapdragon® 8 Gen 3 Mobile | 22.836 ms | 48 - 798 MB | NPU
| OpenAI-Clip | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ IQ-8275 | 29.984 ms | 0 - 3 MB | NPU
| OpenAI-Clip | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ IQ-8275 | 67.718 ms | 1 - 604 MB | NPU
| OpenAI-Clip | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 32.935 ms | 0 - 542 MB | NPU
| OpenAI-Clip | QNN_DLC | w8a16 | Qualcomm® SA8775P | 32.423 ms | 0 - 603 MB | NPU
| OpenAI-Clip | QNN_DLC | w8a16 | Qualcomm® SA8650P | 32.423 ms | 0 - 603 MB | NPU
| OpenAI-Clip | QNN_DLC | w8a16 | Qualcomm® SA8255P | 32.423 ms | 0 - 603 MB | NPU
| OpenAI-Clip | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ IQ-9075 | 32.652 ms | 2 - 4 MB | NPU
| OpenAI-Clip | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ IQ-X7181 | 33.346 ms | 1 - 1 MB | NPU
| OpenAI-Clip | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ Q-6690 | 420.338 ms | 1 - 686 MB | NPU
| OpenAI-Clip | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ Q-7790 | 41.427 ms | 0 - 658 MB | NPU
| OpenAI-Clip | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ Q-8750 | 18.398 ms | 0 - 632 MB | NPU
| OpenAI-Clip | QNN_DLC | w8a16 | Qualcomm® SA7255P | 67.718 ms | 1 - 604 MB | NPU
| OpenAI-Clip | QNN_DLC | w8a16 | Snapdragon® 8 Elite Mobile | 18.398 ms | 0 - 632 MB | NPU
| OpenAI-Clip | QNN_DLC | w8a16 | Snapdragon® 8 Elite Gen 5 Mobile | 14.483 ms | 0 - 678 MB | NPU
| OpenAI-Clip | QNN_DLC | w8a16 | Snapdragon® 7 Gen 4 Mobile | 41.427 ms | 0 - 658 MB | NPU
| OpenAI-Clip | TFLITE | float | Snapdragon® 8 Gen 3 Mobile | 21.716 ms | 0 - 898 MB | NPU
| OpenAI-Clip | TFLITE | float | Snapdragon® 8 Gen 1 Mobile | 40.386 ms | 0 - 907 MB | NPU
| OpenAI-Clip | TFLITE | float | Qualcomm® Dragonwing™ IQ-8275 | 34.917 ms | 0 - 297 MB | NPU
| OpenAI-Clip | TFLITE | float | Qualcomm® Dragonwing™ IQ-8275 | 95.588 ms | 0 - 664 MB | NPU
| OpenAI-Clip | TFLITE | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 31.241 ms | 0 - 3 MB | NPU
| OpenAI-Clip | TFLITE | float | Qualcomm® SA8775P | 35.93 ms | 0 - 670 MB | NPU
| OpenAI-Clip | TFLITE | float | Qualcomm® SA8650P | 35.93 ms | 0 - 670 MB | NPU
| OpenAI-Clip | TFLITE | float | Qualcomm® SA8255P | 35.93 ms | 0 - 670 MB | NPU
| OpenAI-Clip | TFLITE | float | Qualcomm® QCS8450 | 40.386 ms | 0 - 907 MB | NPU
| OpenAI-Clip | TFLITE | float | Qualcomm® Dragonwing™ IQ-9075 | 36.095 ms | 0 - 296 MB | NPU
| OpenAI-Clip | TFLITE | float | Qualcomm® Dragonwing™ Q-8750 | 17.396 ms | 0 - 694 MB | NPU
| OpenAI-Clip | TFLITE | float | Qualcomm® SA7255P | 95.588 ms | 0 - 664 MB | NPU
| OpenAI-Clip | TFLITE | float | Qualcomm® SA8295P | 38.381 ms | 0 - 622 MB | NPU
| OpenAI-Clip | TFLITE | float | Snapdragon® 8 Elite Mobile | 17.396 ms | 0 - 694 MB | NPU
| OpenAI-Clip | TFLITE | float | Snapdragon® 8 Elite Gen 5 Mobile | 13.007 ms | 0 - 538 MB | NPU

## License
* The license for the original implementation of OpenAI-Clip can be found
  [here](https://github.com/openai/CLIP/blob/main/LICENSE).

## References
* [Learning Transferable Visual Models From Natural Language Supervision](https://arxiv.org/abs/2103.00020)
* [Source Model Implementation](https://github.com/openai/CLIP/)

## Community
* Join [our AI Hub Slack community](https://aihub.qualcomm.com/community/slack) to collaborate, post questions and learn more about on-device AI.
* For questions or feedback please [reach out to us](mailto:ai-hub-support@qti.qualcomm.com).