Hugging Face's logo

qualcomm
/
Track-Anything

PyTorch
android
video-object-tracking
Model card Files
xet
 Community
Track-Anything
File size: 13,595 Bytes
6fc807c
 
 
 
 
 
 
 
 
 
 
d9297e5
6fc807c
 
 
d9297e5
738f2b4
d9297e5
 
 
 
 
 
 
 
 
 
 
 
738f2b4
 
d9297e5
 
 
 
 
 
738f2b4
d9297e5
 
 
 
 
 
738f2b4
d9297e5
 
 
 
 
 
 
 
403903c
 
 
 
 
 
 
 
d9297e5
 
 
 
738f2b4
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
d916aa6
738f2b4
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
6fc807c
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
---
library_name: pytorch
license: other
tags:
- android
pipeline_tag: video-object-tracking

---

![](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/track_anything/web-assets/model_demo.png)

# Track-Anything: Optimized for Qualcomm Devices

Track-Anything is a video based machine learning model to track an object in a video.

This is based on the implementation of Track-Anything found [here](https://github.com/gaomingqi/Track-Anything).
This repository contains pre-exported model files optimized for Qualcomm® devices. You can use the [Qualcomm® AI Hub Models](https://github.com/qualcomm/ai-hub-models/blob/v0.59.0/src/qai_hub_models/models/track_anything) library to export with custom configurations. More details on model performance across various devices, can be found [here](#performance-summary).

Qualcomm AI Hub Models uses [Qualcomm AI Hub Workbench](https://workbench.aihub.qualcomm.com) to compile, profile, and evaluate this model. [Sign up](https://myaccount.qualcomm.com/signup) to run these models on a hosted Qualcomm® device.

## Getting Started
There are two ways to deploy this model on your device:

### Option 1: Download Pre-Exported Models

Below are pre-exported model assets ready for deployment.

| Runtime | Precision | Chipset | SDK Versions | Download |
|---|---|---|---|---|
| ONNX | float | Universal | QAIRT 2.45, ONNX Runtime 1.27.1 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/track_anything/releases/v0.59.0/track_anything-onnx-float.zip)
| TFLITE | float | Universal | QAIRT 2.45 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/track_anything/releases/v0.59.0/track_anything-tflite-float.zip)

For more device-specific assets and performance metrics, visit **[Track-Anything on Qualcomm® AI Hub](https://aihub.qualcomm.com/models/track_anything)**.


### Option 2: Export with Custom Configurations

Use the [Qualcomm® AI Hub Models](https://github.com/qualcomm/ai-hub-models/blob/v0.59.0/src/qai_hub_models/models/track_anything) Python library to compile and export the model with your own:
- Custom weights (e.g., fine-tuned checkpoints)
- Custom input shapes
- Target device and runtime configurations

This option is ideal if you need to customize the model beyond the default configuration provided here.

See our repository for [Track-Anything on GitHub](https://github.com/qualcomm/ai-hub-models/blob/v0.59.0/src/qai_hub_models/models/track_anything) for usage instructions.

## Model Details

**Model Type:** Model_use_case.video_object_tracking

**Model Stats:**
- Model checkpoint: xmem
- Input resolution: 320x568
- Number of parameters (encode_key_with_shrinkage): 9.72M
- Model size (encode_key_with_shrinkage) (float): 37.1 MB
- Number of parameters (encode_value): 23.3M
- Model size (encode_value) (float): 88.8 MB
- Number of parameters (encode_key_without_shrinkage): 9.71M
- Model size (encode_key_without_shrinkage) (float): 37.1 MB
- Number of parameters (segment): 30.1M
- Model size (segment) (float): 115 MB

## Performance Summary
| Model | Runtime | Precision | Chipset | Inference Time (ms) | Peak Memory Range (MB) | Primary Compute Unit
|---|---|---|---|---|---|---
| encode_key_with_shrinkage | ONNX | float | Snapdragon® X2 Elite | 3.939 ms | 10 - 10 MB | NPU
| encode_key_with_shrinkage | ONNX | float | Snapdragon® X Elite | 5.392 ms | 18 - 18 MB | NPU
| encode_key_with_shrinkage | ONNX | float | Snapdragon® 8 Gen 3 Mobile | 3.686 ms | 0 - 198 MB | NPU
| encode_key_with_shrinkage | ONNX | float | Snapdragon® 8 Gen 1 Mobile | 10.648 ms | 9 - 191 MB | NPU
| encode_key_with_shrinkage | ONNX | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 5.22 ms | 0 - 20 MB | NPU
| encode_key_with_shrinkage | ONNX | float | Qualcomm® QCS8450 | 10.648 ms | 9 - 191 MB | NPU
| encode_key_with_shrinkage | ONNX | float | Qualcomm® Dragonwing™ IQ-9075 | 8.802 ms | 8 - 11 MB | NPU
| encode_key_with_shrinkage | ONNX | float | Qualcomm® Dragonwing™ IQ-X7181 | 5.392 ms | 18 - 18 MB | NPU
| encode_key_with_shrinkage | ONNX | float | Qualcomm® Dragonwing™ Q-8750 | 3.004 ms | 0 - 169 MB | NPU
| encode_key_with_shrinkage | ONNX | float | Snapdragon® 8 Elite Mobile | 3.004 ms | 0 - 169 MB | NPU
| encode_key_with_shrinkage | ONNX | float | Snapdragon® 8 Elite Gen 5 Mobile | 2.377 ms | 0 - 174 MB | NPU
| encode_key_with_shrinkage | TFLITE | float | Snapdragon® 8 Gen 3 Mobile | 4.065 ms | 0 - 210 MB | NPU
| encode_key_with_shrinkage | TFLITE | float | Snapdragon® 8 Gen 1 Mobile | 11.713 ms | 0 - 194 MB | NPU
| encode_key_with_shrinkage | TFLITE | float | Qualcomm® Dragonwing™ QCS8275 | 31.245 ms | 3 - 179 MB | NPU
| encode_key_with_shrinkage | TFLITE | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 5.937 ms | 4 - 6 MB | NPU
| encode_key_with_shrinkage | TFLITE | float | Qualcomm® SA8775P | 9.457 ms | 3 - 181 MB | NPU
| encode_key_with_shrinkage | TFLITE | float | Qualcomm® SA8650P | 9.457 ms | 3 - 181 MB | NPU
| encode_key_with_shrinkage | TFLITE | float | Qualcomm® SA8255P | 9.457 ms | 3 - 181 MB | NPU
| encode_key_with_shrinkage | TFLITE | float | Qualcomm® QCS8450 | 11.713 ms | 0 - 194 MB | NPU
| encode_key_with_shrinkage | TFLITE | float | Qualcomm® Dragonwing™ IQ-9075 | 9.159 ms | 2 - 29 MB | NPU
| encode_key_with_shrinkage | TFLITE | float | Qualcomm® Dragonwing™ Q-8750 | 3.293 ms | 0 - 180 MB | NPU
| encode_key_with_shrinkage | TFLITE | float | Qualcomm® SA7255P | 31.245 ms | 3 - 179 MB | NPU
| encode_key_with_shrinkage | TFLITE | float | Qualcomm® SA8295P | 10.348 ms | 3 - 163 MB | NPU
| encode_key_with_shrinkage | TFLITE | float | Snapdragon® 8 Elite Mobile | 3.293 ms | 0 - 180 MB | NPU
| encode_key_with_shrinkage | TFLITE | float | Snapdragon® 8 Elite Gen 5 Mobile | 2.467 ms | 0 - 184 MB | NPU
| encode_key_without_shrinkage | ONNX | float | Snapdragon® X2 Elite | 3.886 ms | 80 - 80 MB | NPU
| encode_key_without_shrinkage | ONNX | float | Snapdragon® X Elite | 6.806 ms | 80 - 80 MB | NPU
| encode_key_without_shrinkage | ONNX | float | Snapdragon® 8 Gen 3 Mobile | 5.046 ms | 55 - 257 MB | NPU
| encode_key_without_shrinkage | ONNX | float | Snapdragon® 8 Gen 1 Mobile | 13.545 ms | 44 - 247 MB | NPU
| encode_key_without_shrinkage | ONNX | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 6.863 ms | 24 - 64 MB | NPU
| encode_key_without_shrinkage | ONNX | float | Qualcomm® QCS8450 | 13.545 ms | 44 - 247 MB | NPU
| encode_key_without_shrinkage | ONNX | float | Qualcomm® Dragonwing™ IQ-9075 | 10.621 ms | 52 - 55 MB | NPU
| encode_key_without_shrinkage | ONNX | float | Qualcomm® Dragonwing™ IQ-X7181 | 6.806 ms | 80 - 80 MB | NPU
| encode_key_without_shrinkage | ONNX | float | Qualcomm® Dragonwing™ Q-8750 | 4.385 ms | 52 - 248 MB | NPU
| encode_key_without_shrinkage | ONNX | float | Snapdragon® 8 Elite Mobile | 4.385 ms | 52 - 248 MB | NPU
| encode_key_without_shrinkage | ONNX | float | Snapdragon® 8 Elite Gen 5 Mobile | 3.474 ms | 0 - 178 MB | NPU
| encode_key_without_shrinkage | TFLITE | float | Snapdragon® 8 Gen 3 Mobile | 5.58 ms | 0 - 214 MB | NPU
| encode_key_without_shrinkage | TFLITE | float | Snapdragon® 8 Gen 1 Mobile | 14.156 ms | 0 - 202 MB | NPU
| encode_key_without_shrinkage | TFLITE | float | Qualcomm® Dragonwing™ QCS8275 | 33.555 ms | 11 - 194 MB | NPU
| encode_key_without_shrinkage | TFLITE | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 7.794 ms | 14 - 16 MB | NPU
| encode_key_without_shrinkage | TFLITE | float | Qualcomm® SA8775P | 11.674 ms | 20 - 207 MB | NPU
| encode_key_without_shrinkage | TFLITE | float | Qualcomm® SA8650P | 11.674 ms | 20 - 207 MB | NPU
| encode_key_without_shrinkage | TFLITE | float | Qualcomm® SA8255P | 11.674 ms | 20 - 207 MB | NPU
| encode_key_without_shrinkage | TFLITE | float | Qualcomm® QCS8450 | 14.156 ms | 0 - 202 MB | NPU
| encode_key_without_shrinkage | TFLITE | float | Qualcomm® Dragonwing™ IQ-9075 | 11.337 ms | 20 - 63 MB | NPU
| encode_key_without_shrinkage | TFLITE | float | Qualcomm® Dragonwing™ Q-8750 | 4.42 ms | 18 - 203 MB | NPU
| encode_key_without_shrinkage | TFLITE | float | Qualcomm® SA7255P | 33.555 ms | 11 - 194 MB | NPU
| encode_key_without_shrinkage | TFLITE | float | Qualcomm® SA8295P | 12.718 ms | 20 - 184 MB | NPU
| encode_key_without_shrinkage | TFLITE | float | Snapdragon® 8 Elite Mobile | 4.42 ms | 18 - 203 MB | NPU
| encode_key_without_shrinkage | TFLITE | float | Snapdragon® 8 Elite Gen 5 Mobile | 3.574 ms | 0 - 221 MB | NPU
| encode_value | ONNX | float | Snapdragon® X2 Elite | 6.553 ms | 20 - 20 MB | NPU
| encode_value | ONNX | float | Snapdragon® X Elite | 13.249 ms | 39 - 39 MB | NPU
| encode_value | ONNX | float | Snapdragon® 8 Gen 3 Mobile | 11.379 ms | 0 - 209 MB | NPU
| encode_value | ONNX | float | Snapdragon® 8 Gen 1 Mobile | 31.596 ms | 12 - 208 MB | NPU
| encode_value | ONNX | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 16.791 ms | 0 - 50 MB | NPU
| encode_value | ONNX | float | Qualcomm® QCS8450 | 31.596 ms | 12 - 208 MB | NPU
| encode_value | ONNX | float | Qualcomm® Dragonwing™ IQ-9075 | 20.338 ms | 6 - 14 MB | NPU
| encode_value | ONNX | float | Qualcomm® Dragonwing™ IQ-X7181 | 13.249 ms | 39 - 39 MB | NPU
| encode_value | ONNX | float | Qualcomm® Dragonwing™ Q-8750 | 9.547 ms | 7 - 191 MB | NPU
| encode_value | ONNX | float | Snapdragon® 8 Elite Mobile | 9.547 ms | 7 - 191 MB | NPU
| encode_value | ONNX | float | Snapdragon® 8 Elite Gen 5 Mobile | 8.222 ms | 5 - 179 MB | NPU
| encode_value | TFLITE | float | Snapdragon® 8 Gen 3 Mobile | 8.241 ms | 1 - 286 MB | NPU
| encode_value | TFLITE | float | Snapdragon® 8 Gen 1 Mobile | 21.253 ms | 3 - 291 MB | NPU
| encode_value | TFLITE | float | Qualcomm® Dragonwing™ QCS8275 | 51.329 ms | 3 - 217 MB | NPU
| encode_value | TFLITE | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 11.857 ms | 3 - 6 MB | NPU
| encode_value | TFLITE | float | Qualcomm® SA8775P | 17.008 ms | 3 - 215 MB | NPU
| encode_value | TFLITE | float | Qualcomm® SA8650P | 17.008 ms | 3 - 215 MB | NPU
| encode_value | TFLITE | float | Qualcomm® SA8255P | 17.008 ms | 3 - 215 MB | NPU
| encode_value | TFLITE | float | Qualcomm® QCS8450 | 21.253 ms | 3 - 291 MB | NPU
| encode_value | TFLITE | float | Qualcomm® Dragonwing™ IQ-9075 | 17.086 ms | 0 - 55 MB | NPU
| encode_value | TFLITE | float | Qualcomm® Dragonwing™ Q-8750 | 6.861 ms | 3 - 213 MB | NPU
| encode_value | TFLITE | float | Qualcomm® SA7255P | 51.329 ms | 3 - 217 MB | NPU
| encode_value | TFLITE | float | Qualcomm® SA8295P | 18.842 ms | 0 - 216 MB | NPU
| encode_value | TFLITE | float | Snapdragon® 8 Elite Mobile | 6.861 ms | 3 - 213 MB | NPU
| encode_value | TFLITE | float | Snapdragon® 8 Elite Gen 5 Mobile | 5.401 ms | 1 - 201 MB | NPU
| segment | ONNX | float | Snapdragon® X2 Elite | 11.676 ms | 21 - 21 MB | NPU
| segment | ONNX | float | Snapdragon® X Elite | 23.175 ms | 37 - 37 MB | NPU
| segment | ONNX | float | Snapdragon® 8 Gen 3 Mobile | 17.336 ms | 25 - 267 MB | NPU
| segment | ONNX | float | Snapdragon® 8 Gen 1 Mobile | 43.443 ms | 2 - 232 MB | NPU
| segment | ONNX | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 23.068 ms | 0 - 45 MB | NPU
| segment | ONNX | float | Qualcomm® QCS8450 | 43.443 ms | 2 - 232 MB | NPU
| segment | ONNX | float | Qualcomm® Dragonwing™ IQ-9075 | 35.156 ms | 21 - 45 MB | NPU
| segment | ONNX | float | Qualcomm® Dragonwing™ IQ-X7181 | 23.175 ms | 37 - 37 MB | NPU
| segment | ONNX | float | Qualcomm® Dragonwing™ Q-8750 | 13.315 ms | 5 - 211 MB | NPU
| segment | ONNX | float | Snapdragon® 8 Elite Mobile | 13.315 ms | 5 - 211 MB | NPU
| segment | ONNX | float | Snapdragon® 8 Elite Gen 5 Mobile | 10.822 ms | 5 - 216 MB | NPU
| segment | TFLITE | float | Snapdragon® 8 Gen 3 Mobile | 17.433 ms | 1 - 292 MB | NPU
| segment | TFLITE | float | Snapdragon® 8 Gen 1 Mobile | 42.34 ms | 2 - 294 MB | NPU
| segment | TFLITE | float | Qualcomm® Dragonwing™ QCS8275 | 103.337 ms | 2 - 195 MB | NPU
| segment | TFLITE | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 23.541 ms | 2 - 4 MB | NPU
| segment | TFLITE | float | Qualcomm® SA8775P | 33.286 ms | 2 - 193 MB | NPU
| segment | TFLITE | float | Qualcomm® SA8650P | 33.286 ms | 2 - 193 MB | NPU
| segment | TFLITE | float | Qualcomm® SA8255P | 33.286 ms | 2 - 193 MB | NPU
| segment | TFLITE | float | Qualcomm® QCS8450 | 42.34 ms | 2 - 294 MB | NPU
| segment | TFLITE | float | Qualcomm® Dragonwing™ IQ-9075 | 35.307 ms | 0 - 83 MB | NPU
| segment | TFLITE | float | Qualcomm® Dragonwing™ Q-8750 | 13.196 ms | 0 - 195 MB | NPU
| segment | TFLITE | float | Qualcomm® SA7255P | 103.337 ms | 2 - 195 MB | NPU
| segment | TFLITE | float | Qualcomm® SA8295P | 37.583 ms | 2 - 202 MB | NPU
| segment | TFLITE | float | Snapdragon® 8 Elite Mobile | 13.196 ms | 0 - 195 MB | NPU
| segment | TFLITE | float | Snapdragon® 8 Elite Gen 5 Mobile | 11.574 ms | 2 - 197 MB | NPU

## License
* The license for the original implementation of Track-Anything can be found
  [here](https://github.com/gaomingqi/Track-Anything/blob/master/LICENSE).

## References
* [Track Anything: Segment Anything Meets Videos](https://arxiv.org/abs/2304.11968)
* [Source Model Implementation](https://github.com/gaomingqi/Track-Anything)

## Community
* Join [our AI Hub Slack community](https://aihub.qualcomm.com/community/slack) to collaborate, post questions and learn more about on-device AI.
* For questions or feedback please [reach out to us](mailto:ai-hub-support@qti.qualcomm.com).