File size: 4,274 Bytes
5fb7ec6 afa9af0 99e4677 5fb7ec6 99e4677 5fb7ec6 afa9af0 5fb7ec6 afa9af0 5fb7ec6 45ba294 afa9af0 45ba294 afa9af0 99e4677 afa9af0 a843ceb afa9af0 99e4677 afa9af0 5fb7ec6 afa9af0 5fb7ec6 afa9af0 99e4677 afa9af0 99e4677 afa9af0 5fb7ec6 afa9af0 99e4677 afa9af0 99e4677 afa9af0 99e4677 afa9af0 99e4677 afa9af0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 | ---
pipeline_tag: image-segmentation
license: mit
base_model: facebook/mask2former-swin-tiny-coco-instance
library_name: zeromodels
tags:
- keras
- zeromodels
- mask2former
- instance-segmentation
- image-segmentation
- arxiv:2112.01527
- pytorch
- jax
- tf
---
## ***See [our collection](https://huggingface.co/collections/zeromodels/mask2former-6a8eaf66faaf81a53d54fa03) for all versions of Mask2Former.***
# Run Mask2Former with Keras 3: JAX, PyTorch, or TensorFlow
[](https://github.com/IMvision12/ZeroModels) [](https://imvision12.github.io/ZeroModels/mask2former/) [](https://huggingface.co/collections/zeromodels/mask2former-6a8eaf66faaf81a53d54fa03)
# zeromodels/mask2former-swin-tiny-coco-instance
Paper: [Masked-attention Mask Transformer for Universal Image Segmentation (arXiv:2112.01527)](https://arxiv.org/abs/2112.01527) · [HF Papers](https://huggingface.co/papers/2112.01527)
Mask2Former improves MaskFormer with masked attention in the transformer decoder, restricting cross-attention to predicted mask regions for sharper boundaries and stronger universal segmentation.
For more details on the model, please go to the upstream [model card](https://huggingface.co/facebook/mask2former-swin-tiny-coco-instance).
Pure-**Keras 3** conversion of [`facebook/mask2former-swin-tiny-coco-instance`](https://huggingface.co/facebook/mask2former-swin-tiny-coco-instance) for [zeromodels](https://github.com/IMvision12/ZeroModels). One implementation runs unmodified on **TensorFlow / Torch / JAX**.
This is a **instance** checkpoint (`Mask2FormerUniversalSegment`) (trained for instance; architecture is universal).
## ✨ Quick start
```python
import os
os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow"
from PIL import Image
from zeromodels.models.mask2former import Mask2FormerUniversalSegment, Mask2FormerImageProcessor
model = Mask2FormerUniversalSegment.from_weights("zeromodels/mask2former-swin-tiny-coco-instance")
processor = Mask2FormerImageProcessor.from_weights("zeromodels/mask2former-swin-tiny-coco-instance")
image = Image.open("your_image.jpg").convert("RGB")
output = model(processor(image)["pixel_values"], training=False)
result = processor.post_process_panoptic_segmentation(
output, target_size=(image.height, image.width)
)
print(result["segmentation"].shape)
```
Load any Mask2Former variant the same way with `from_weights("zeromodels/<variant>")`:
| Variant | Hub | Task |
|---|---|---|
| `mask2former-swin-tiny-coco-instance` | [`zeromodels/mask2former-swin-tiny-coco-instance`](https://huggingface.co/zeromodels/mask2former-swin-tiny-coco-instance) | instance |
| `mask2former-swin-small-coco-instance` | [`zeromodels/mask2former-swin-small-coco-instance`](https://huggingface.co/zeromodels/mask2former-swin-small-coco-instance) | instance |
| `mask2former-swin-base-coco-instance` | [`zeromodels/mask2former-swin-base-coco-instance`](https://huggingface.co/zeromodels/mask2former-swin-base-coco-instance) | instance |
| `mask2former-swin-large-coco-instance` | [`zeromodels/mask2former-swin-large-coco-instance`](https://huggingface.co/zeromodels/mask2former-swin-large-coco-instance) | instance |
| `mask2former-swin-tiny-coco-panoptic` | [`zeromodels/mask2former-swin-tiny-coco-panoptic`](https://huggingface.co/zeromodels/mask2former-swin-tiny-coco-panoptic) | panoptic |
| `mask2former-swin-tiny-ade-semantic` | [`zeromodels/mask2former-swin-tiny-ade-semantic`](https://huggingface.co/zeromodels/mask2former-swin-tiny-ade-semantic) | semantic |
## Tips
- Set `KERAS_BACKEND` **before** importing Keras / zeromodels.
- The task suffix is what the checkpoint was trained for; post-process accordingly.
- See [Mask2Former docs]({DOCS_URL}) and [Loading Weights](https://imvision12.github.io/ZeroModels/loading_weights/).
- Community / upstream weights: `Mask2FormerUniversalSegment.from_weights("hf:facebook/mask2former-swin-tiny-coco-instance")`.
## Special Thanks
A huge thank you to the Facebook AI Research Mask2Former authors for creating and releasing these models.
License: MIT.
|