Hugging Face's logo

qualcomm
/
Mask2Former

Image Segmentation
PyTorch
android
Model card Files
xet
 Community
Mask2Former
File size: 11,263 Bytes
70de015
 
77e9f39
70de015
 
 
 
 
 
 
 
98f6ee6
70de015
 
 
98f6ee6
6b3343c
44fe473
98f6ee6
70de015
98f6ee6
 
70de015
98f6ee6
70de015
98f6ee6
70de015
98f6ee6
 
6b3343c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
70de015
98f6ee6
70de015
 
98f6ee6
70de015
6b3343c
98f6ee6
 
 
70de015
98f6ee6
70de015
6b3343c
70de015
98f6ee6
70de015
98f6ee6
70de015
98f6ee6
 
 
 
70de015
98f6ee6
 
 
6b3343c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
70de015
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
---
library_name: pytorch
license: other
tags:
- android
pipeline_tag: image-segmentation

---

![](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mask2former/web-assets/model_demo.png)

# Mask2Former: Optimized for Qualcomm Devices

Mask2Former is a machine learning model that predicts masks and classes of objects in an image.

This is based on the implementation of Mask2Former found [here](https://github.com/huggingface/transformers/tree/main/src/transformers/models/mask2former).
This repository contains pre-exported model files optimized for Qualcomm® devices. You can use the [Qualcomm® AI Hub Models](https://github.com/qualcomm/ai-hub-models/blob/v0.59.0/src/qai_hub_models/models/mask2former) library to export with custom configurations. More details on model performance across various devices, can be found [here](#performance-summary).

Qualcomm AI Hub Models uses [Qualcomm AI Hub Workbench](https://workbench.aihub.qualcomm.com) to compile, profile, and evaluate this model. [Sign up](https://myaccount.qualcomm.com/signup) to run these models on a hosted Qualcomm® device.

## Getting Started
There are two ways to deploy this model on your device:

### Option 1: Download Pre-Exported Models

Below are pre-exported model assets ready for deployment.

| Runtime | Precision | Chipset | SDK Versions | Download |
|---|---|---|---|---|
| PRECOMPILED_QNN_ONNX | float | Snapdragon® X2 Elite | QAIRT 2.45, ONNX Runtime 1.27.1 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mask2former/releases/v0.59.0/mask2former-precompiled_qnn_onnx-float-qualcomm_snapdragon_x2_elite.zip)
| PRECOMPILED_QNN_ONNX | float | Snapdragon® X Elite | QAIRT 2.45, ONNX Runtime 1.27.1 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mask2former/releases/v0.59.0/mask2former-precompiled_qnn_onnx-float-qualcomm_snapdragon_x_elite.zip)
| PRECOMPILED_QNN_ONNX | float | Snapdragon® 8 Gen 3 Mobile | QAIRT 2.45, ONNX Runtime 1.27.1 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mask2former/releases/v0.59.0/mask2former-precompiled_qnn_onnx-float-qualcomm_snapdragon_8gen3.zip)
| PRECOMPILED_QNN_ONNX | float | Snapdragon® 8 Gen 1 Mobile | QAIRT 2.45, ONNX Runtime 1.27.1 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mask2former/releases/v0.59.0/mask2former-precompiled_qnn_onnx-float-qualcomm_snapdragon_8gen1.zip)
| PRECOMPILED_QNN_ONNX | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | QAIRT 2.45, ONNX Runtime 1.27.1 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mask2former/releases/v0.59.0/mask2former-precompiled_qnn_onnx-float-qualcomm_qcs8550_proxy.zip)
| PRECOMPILED_QNN_ONNX | float | Qualcomm® Dragonwing™ IQ-9075 | QAIRT 2.45, ONNX Runtime 1.27.1 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mask2former/releases/v0.59.0/mask2former-precompiled_qnn_onnx-float-qualcomm_qcs9075.zip)
| PRECOMPILED_QNN_ONNX | float | Snapdragon® 8 Elite Mobile | QAIRT 2.45, ONNX Runtime 1.27.1 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mask2former/releases/v0.59.0/mask2former-precompiled_qnn_onnx-float-qualcomm_snapdragon_8_elite_for_galaxy.zip)
| PRECOMPILED_QNN_ONNX | float | Snapdragon® 8 Elite Gen 5 Mobile | QAIRT 2.45, ONNX Runtime 1.27.1 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mask2former/releases/v0.59.0/mask2former-precompiled_qnn_onnx-float-qualcomm_snapdragon_8_elite_gen5_for_galaxy.zip)
| QNN_CONTEXT_BINARY | float | Snapdragon® X2 Elite | QAIRT 2.45 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mask2former/releases/v0.59.0/mask2former-qnn_context_binary-float-qualcomm_snapdragon_x2_elite.zip)
| QNN_CONTEXT_BINARY | float | Snapdragon® X Elite | QAIRT 2.45 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mask2former/releases/v0.59.0/mask2former-qnn_context_binary-float-qualcomm_snapdragon_x_elite.zip)
| QNN_CONTEXT_BINARY | float | Snapdragon® 8 Gen 3 Mobile | QAIRT 2.45 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mask2former/releases/v0.59.0/mask2former-qnn_context_binary-float-qualcomm_snapdragon_8gen3.zip)
| QNN_CONTEXT_BINARY | float | Snapdragon® 8 Gen 1 Mobile | QAIRT 2.45 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mask2former/releases/v0.59.0/mask2former-qnn_context_binary-float-qualcomm_snapdragon_8gen1.zip)
| QNN_CONTEXT_BINARY | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | QAIRT 2.45 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mask2former/releases/v0.59.0/mask2former-qnn_context_binary-float-qualcomm_qcs8550_proxy.zip)
| QNN_CONTEXT_BINARY | float | Qualcomm® SA8775P | QAIRT 2.45 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mask2former/releases/v0.59.0/mask2former-qnn_context_binary-float-qualcomm_sa8775p.zip)
| QNN_CONTEXT_BINARY | float | Qualcomm® Dragonwing™ IQ-9075 | QAIRT 2.45 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mask2former/releases/v0.59.0/mask2former-qnn_context_binary-float-qualcomm_qcs9075.zip)
| QNN_CONTEXT_BINARY | float | Qualcomm® SA7255P | QAIRT 2.45 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mask2former/releases/v0.59.0/mask2former-qnn_context_binary-float-qualcomm_sa7255p.zip)
| QNN_CONTEXT_BINARY | float | Qualcomm® SA8295P | QAIRT 2.45 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mask2former/releases/v0.59.0/mask2former-qnn_context_binary-float-qualcomm_sa8295p.zip)
| QNN_CONTEXT_BINARY | float | Snapdragon® 8 Elite Mobile | QAIRT 2.45 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mask2former/releases/v0.59.0/mask2former-qnn_context_binary-float-qualcomm_snapdragon_8_elite_for_galaxy.zip)
| QNN_CONTEXT_BINARY | float | Snapdragon® 8 Elite Gen 5 Mobile | QAIRT 2.45 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/mask2former/releases/v0.59.0/mask2former-qnn_context_binary-float-qualcomm_snapdragon_8_elite_gen5_for_galaxy.zip)

For more device-specific assets and performance metrics, visit **[Mask2Former on Qualcomm® AI Hub](https://aihub.qualcomm.com/models/mask2former)**.


### Option 2: Export with Custom Configurations

Use the [Qualcomm® AI Hub Models](https://github.com/qualcomm/ai-hub-models/blob/v0.59.0/src/qai_hub_models/models/mask2former) Python library to compile and export the model with your own:
- Custom weights (e.g., fine-tuned checkpoints)
- Custom input shapes
- Target device and runtime configurations

This option is ideal if you need to customize the model beyond the default configuration provided here.

See our repository for [Mask2Former on GitHub](https://github.com/qualcomm/ai-hub-models/blob/v0.59.0/src/qai_hub_models/models/mask2former) for usage instructions.

## Model Details

**Model Type:** Model_use_case.semantic_segmentation

**Model Stats:**
- Model checkpoint: facebook/mask2former-swin-tiny-coco-panoptic
- Input resolution: 384x384
- Number of output classes: 100

## Performance Summary
| Model | Runtime | Precision | Chipset | Inference Time (ms) | Peak Memory Range (MB) | Primary Compute Unit
|---|---|---|---|---|---|---
| Mask2Former | PRECOMPILED_QNN_ONNX | float | Snapdragon® X2 Elite | 67.94 ms | 9 - 9 MB | NPU
| Mask2Former | PRECOMPILED_QNN_ONNX | float | Snapdragon® X Elite | 142.293 ms | 109 - 109 MB | NPU
| Mask2Former | PRECOMPILED_QNN_ONNX | float | Snapdragon® 8 Gen 3 Mobile | 104.618 ms | 9 - 22 MB | NPU
| Mask2Former | PRECOMPILED_QNN_ONNX | float | Snapdragon® 8 Gen 1 Mobile | 234.001 ms | 9 - 23 MB | NPU
| Mask2Former | PRECOMPILED_QNN_ONNX | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 142.039 ms | 0 - 112 MB | NPU
| Mask2Former | PRECOMPILED_QNN_ONNX | float | Qualcomm® QCS8450 | 234.001 ms | 9 - 23 MB | NPU
| Mask2Former | PRECOMPILED_QNN_ONNX | float | Qualcomm® Dragonwing™ IQ-9075 | 201.948 ms | 7 - 12 MB | NPU
| Mask2Former | PRECOMPILED_QNN_ONNX | float | Qualcomm® Dragonwing™ IQ-X7181 | 142.293 ms | 109 - 109 MB | NPU
| Mask2Former | PRECOMPILED_QNN_ONNX | float | Qualcomm® Dragonwing™ Q-8750 | 89.094 ms | 10 - 20 MB | NPU
| Mask2Former | PRECOMPILED_QNN_ONNX | float | Snapdragon® 8 Elite Mobile | 89.094 ms | 10 - 20 MB | NPU
| Mask2Former | PRECOMPILED_QNN_ONNX | float | Snapdragon® 8 Elite Gen 5 Mobile | 70.678 ms | 9 - 21 MB | NPU
| Mask2Former | QNN_CONTEXT_BINARY | float | Snapdragon® X2 Elite | 68.148 ms | 2 - 2 MB | NPU
| Mask2Former | QNN_CONTEXT_BINARY | float | Snapdragon® X Elite | 142.765 ms | 2 - 2 MB | NPU
| Mask2Former | QNN_CONTEXT_BINARY | float | Snapdragon® 8 Gen 3 Mobile | 104.6 ms | 4 - 11 MB | NPU
| Mask2Former | QNN_CONTEXT_BINARY | float | Snapdragon® 8 Gen 1 Mobile | 233.681 ms | 1 - 10 MB | NPU
| Mask2Former | QNN_CONTEXT_BINARY | float | Qualcomm® Dragonwing™ QCS8275 | 289.752 ms | 2 - 11 MB | NPU
| Mask2Former | QNN_CONTEXT_BINARY | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 141.664 ms | 2 - 4 MB | NPU
| Mask2Former | QNN_CONTEXT_BINARY | float | Qualcomm® SA8775P | 147.41 ms | 2 - 11 MB | NPU
| Mask2Former | QNN_CONTEXT_BINARY | float | Qualcomm® SA8650P | 147.41 ms | 2 - 11 MB | NPU
| Mask2Former | QNN_CONTEXT_BINARY | float | Qualcomm® SA8255P | 147.41 ms | 2 - 11 MB | NPU
| Mask2Former | QNN_CONTEXT_BINARY | float | Qualcomm® QCS8450 | 233.681 ms | 1 - 10 MB | NPU
| Mask2Former | QNN_CONTEXT_BINARY | float | Qualcomm® Dragonwing™ IQ-9075 | 146.388 ms | 4 - 10 MB | NPU
| Mask2Former | QNN_CONTEXT_BINARY | float | Qualcomm® Dragonwing™ IQ-X7181 | 142.765 ms | 2 - 2 MB | NPU
| Mask2Former | QNN_CONTEXT_BINARY | float | Qualcomm® Dragonwing™ Q-8750 | 88.976 ms | 2 - 14 MB | NPU
| Mask2Former | QNN_CONTEXT_BINARY | float | Qualcomm® SA7255P | 289.752 ms | 2 - 11 MB | NPU
| Mask2Former | QNN_CONTEXT_BINARY | float | Qualcomm® SA8295P | 182.756 ms | 2 - 7 MB | NPU
| Mask2Former | QNN_CONTEXT_BINARY | float | Snapdragon® 8 Elite Mobile | 88.976 ms | 2 - 14 MB | NPU
| Mask2Former | QNN_CONTEXT_BINARY | float | Snapdragon® 8 Elite Gen 5 Mobile | 70.882 ms | 2 - 12 MB | NPU

## License
* The license for the original implementation of Mask2Former can be found
  [here](https://github.com/huggingface/transformers/blob/main/LICENSE).

## References
* [Masked-attention Mask Transformer for Universal Image Segmentation](https://arxiv.org/abs/2112.01527)
* [Source Model Implementation](https://github.com/huggingface/transformers/tree/main/src/transformers/models/mask2former)

## Community
* Join [our AI Hub Slack community](https://aihub.qualcomm.com/community/slack) to collaborate, post questions and learn more about on-device AI.
* For questions or feedback please [reach out to us](mailto:ai-hub-support@qti.qualcomm.com).