Mask Generation
LiteRT
LiteRT
mediapipe
on-device
android
ios
web
python
flutter
interactive-segmentation
promptable-segmentation
magictouch
int8
Instructions to use litert-community/MediaPipe-InteractiveSegmenter-MagicTouch with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LiteRT
How to use litert-community/MediaPipe-InteractiveSegmenter-MagicTouch with LiteRT:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Commit ·
61de3ef
0
Parent(s):
Initial upload of MediaPipe-InteractiveSegmenter-MagicTouch from MediaPipe Solutions
Browse files- .gitattributes +38 -0
- Model Card MagicTouch.pdf +3 -0
- README.md +163 -0
- interactive_segmentation.task +3 -0
.gitattributes
ADDED
|
@@ -0,0 +1,38 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
*.7z filter=lfs diff=lfs merge=lfs -text
|
| 2 |
+
*.arrow filter=lfs diff=lfs merge=lfs -text
|
| 3 |
+
*.bin filter=lfs diff=lfs merge=lfs -text
|
| 4 |
+
*.bz2 filter=lfs diff=lfs merge=lfs -text
|
| 5 |
+
*.ckpt filter=lfs diff=lfs merge=lfs -text
|
| 6 |
+
*.ftz filter=lfs diff=lfs merge=lfs -text
|
| 7 |
+
*.gz filter=lfs diff=lfs merge=lfs -text
|
| 8 |
+
*.h5 filter=lfs diff=lfs merge=lfs -text
|
| 9 |
+
*.joblib filter=lfs diff=lfs merge=lfs -text
|
| 10 |
+
*.lfs.* filter=lfs diff=lfs merge=lfs -text
|
| 11 |
+
*.litertlm filter=lfs diff=lfs merge=lfs -text
|
| 12 |
+
*.mlmodel filter=lfs diff=lfs merge=lfs -text
|
| 13 |
+
*.model filter=lfs diff=lfs merge=lfs -text
|
| 14 |
+
*.msgpack filter=lfs diff=lfs merge=lfs -text
|
| 15 |
+
*.npy filter=lfs diff=lfs merge=lfs -text
|
| 16 |
+
*.npz filter=lfs diff=lfs merge=lfs -text
|
| 17 |
+
*.onnx filter=lfs diff=lfs merge=lfs -text
|
| 18 |
+
*.ot filter=lfs diff=lfs merge=lfs -text
|
| 19 |
+
*.parquet filter=lfs diff=lfs merge=lfs -text
|
| 20 |
+
*.pb filter=lfs diff=lfs merge=lfs -text
|
| 21 |
+
*.pdf filter=lfs diff=lfs merge=lfs -text
|
| 22 |
+
*.pickle filter=lfs diff=lfs merge=lfs -text
|
| 23 |
+
*.pkl filter=lfs diff=lfs merge=lfs -text
|
| 24 |
+
*.pt filter=lfs diff=lfs merge=lfs -text
|
| 25 |
+
*.pth filter=lfs diff=lfs merge=lfs -text
|
| 26 |
+
*.rar filter=lfs diff=lfs merge=lfs -text
|
| 27 |
+
*.safetensors filter=lfs diff=lfs merge=lfs -text
|
| 28 |
+
saved_model/**/* filter=lfs diff=lfs merge=lfs -text
|
| 29 |
+
*.tar.* filter=lfs diff=lfs merge=lfs -text
|
| 30 |
+
*.tar filter=lfs diff=lfs merge=lfs -text
|
| 31 |
+
*.task filter=lfs diff=lfs merge=lfs -text
|
| 32 |
+
*.tflite filter=lfs diff=lfs merge=lfs -text
|
| 33 |
+
*.tgz filter=lfs diff=lfs merge=lfs -text
|
| 34 |
+
*.wasm filter=lfs diff=lfs merge=lfs -text
|
| 35 |
+
*.xz filter=lfs diff=lfs merge=lfs -text
|
| 36 |
+
*.zip filter=lfs diff=lfs merge=lfs -text
|
| 37 |
+
*.zst filter=lfs diff=lfs merge=lfs -text
|
| 38 |
+
*tfevents* filter=lfs diff=lfs merge=lfs -text
|
Model Card MagicTouch.pdf
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:b4accf5f7be991903646e6dd9765faa6974cdf69c5a9e84841684dffa27df0f1
|
| 3 |
+
size 301841
|
README.md
ADDED
|
@@ -0,0 +1,163 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
library_name: mediapipe
|
| 4 |
+
pipeline_tag: mask-generation
|
| 5 |
+
tags:
|
| 6 |
+
- mediapipe
|
| 7 |
+
- litert
|
| 8 |
+
- tflite
|
| 9 |
+
- mask-generation
|
| 10 |
+
- on-device
|
| 11 |
+
- android
|
| 12 |
+
- ios
|
| 13 |
+
- web
|
| 14 |
+
- python
|
| 15 |
+
- flutter
|
| 16 |
+
- interactive-segmentation
|
| 17 |
+
- promptable-segmentation
|
| 18 |
+
- magictouch
|
| 19 |
+
- int8
|
| 20 |
+
---
|
| 21 |
+
|
| 22 |
+
# MediaPipe MagicTouch (Interactive Segmenter)
|
| 23 |
+
|
| 24 |
+
Stateful point/stroke-prompted interactive segmentation bundle (`interactive_segmentation.task`, 30.5 MB, INT8 v2 encoder/decoder at `768 x 768`) that encodes an image once and iteratively segments user-indicated objects via positive/negative strokes in the [MediaPipe Interactive Segmenter (`InteractiveSegmenter`)](https://developers.google.com/edge/mediapipe/solutions/vision/interactive_segmenter) task.
|
| 25 |
+
|
| 26 |
+
<div style="border: 1px solid rgba(59, 130, 246, 0.28); background-color: rgba(59, 130, 246, 0.06); border-radius: 8px; padding: 12px 16px; margin: 16px 0;">
|
| 27 |
+
<p style="margin: 0 0 6px 0;"><strong>Documentation:</strong> <a href="https://developers.google.com/edge/mediapipe/solutions/vision/interactive_segmenter">MediaPipe Interactive Segmenter Docs</a></p>
|
| 28 |
+
<p style="margin: 0 0 6px 0;"><strong>Samples:</strong> <a href="https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/interactive_segmentation/android">Android</a> · <a href="https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/interactive_segmentation/ios">iOS</a> · <a href="https://colab.research.google.com/github/google-ai-edge/mediapipe-samples/blob/main/examples/interactive_segmentation/python/interactive_segmenter.ipynb">Python Colab</a> · <a href="https://github.com/google-ai-edge/mediapipe-samples-web/blob/main/src/tasks/interactive-segmenter.ts">Web</a> · <a href="https://github.com/google/flutter-mediapipe/tree/main/samples/vision/interactive_segmenter">Flutter</a></p>
|
| 29 |
+
<p style="margin: 0;"><strong>Try live:</strong> <a href="https://google-ai-edge.github.io/mediapipe-samples-web/#/vision/interactive_segmenter">Web Demo</a></p>
|
| 30 |
+
</div>
|
| 31 |
+
|
| 32 |
+
`interactive_segmentation.task` (`interactive_segmenter_v2`) separates image feature extraction from interactive decoding for low-latency refinement:
|
| 33 |
+
|
| 34 |
+
1. **`interactive_segmentation_encoder.int8.tflite` (28.3 MB):** Runs once in `set_image()`, encoding `[-1, 768, 768, 3]` RGB images into 5 multi-scale feature maps (`384×384×16` .. `24×24×192`).
|
| 35 |
+
2. **`interactive_segmentation_decoder.int8.tflite` (2.3 MB):** Runs per user stroke call in `segment(strokes)`, combining the prompt canvas (`[1, 768, 768, 3]`) and cached encoder features into a `[1, 768, 768, 1]` mask.
|
| 36 |
+
|
| 37 |
+
**Bundle version note:** The bundled [`Model Card MagicTouch.pdf`](./Model%20Card%20MagicTouch.pdf) (March 13, 2023) documents the earlier v1 single-graph `512 x 512 x 4` (≈6 MB) FP model (≈30 FPS on Pixel 7 GPU), whereas this `.task` bundle packages the upgraded **768×768 INT8 two-stage v2** encoder/decoder.
|
| 38 |
+
|
| 39 |
+
## Files
|
| 40 |
+
|
| 41 |
+
| File | Precision | Size | SHA-256 |
|
| 42 |
+
|---|---|---:|---|
|
| 43 |
+
| [`interactive_segmentation.task`](https://huggingface.co/litert-community/MediaPipe-InteractiveSegmenter-MagicTouch/blob/main/interactive_segmentation.task) | `int8` | 30.53 MB (30,525,312 B) | `38431bc66b883404e8397f74c3579404315b9b52b04a46c6346fe906a7309b03` |
|
| 44 |
+
|
| 45 |
+
### Companion Files in Repo Root
|
| 46 |
+
|
| 47 |
+
| File | Role | Size | SHA-256 |
|
| 48 |
+
|---|---|---:|---|
|
| 49 |
+
| [`Model Card MagicTouch.pdf`](./Model%20Card%20MagicTouch.pdf) | Upstream Google model card (PDF) | 301.8 KB (301,841 B) | `b4accf5f7be991903646e6dd9765faa6974cdf69c5a9e84841684dffa27df0f1` |
|
| 50 |
+
|
| 51 |
+
### Download
|
| 52 |
+
|
| 53 |
+
```bash
|
| 54 |
+
pip install -U huggingface_hub
|
| 55 |
+
hf download litert-community/MediaPipe-InteractiveSegmenter-MagicTouch interactive_segmentation.task --local-dir .
|
| 56 |
+
```
|
| 57 |
+
|
| 58 |
+
## Inputs, Outputs, and Model Specifications
|
| 59 |
+
|
| 60 |
+
### Bundle Contents (`interactive_segmentation.task`)
|
| 61 |
+
|
| 62 |
+
| Bundled Member | Uncompressed Size | SHA-256 (prefix) |
|
| 63 |
+
|---|---:|---|
|
| 64 |
+
| `interactive_segmentation_decoder.int8.tflite` | 2.27 MB (2,268,000 B) | `140d36971403c38b…` |
|
| 65 |
+
| `interactive_segmentation_encoder.int8.tflite` | 28.26 MB (28,256,696 B) | `0edbceec02cb1570…` |
|
| 66 |
+
| `manifest.pb` | 0.1 KB (50 B) | `c18392128d95dc4d…` |
|
| 67 |
+
| `metadata.pb` | 0.0 KB (12 B) | `d4945e22f0e49ea2…` |
|
| 68 |
+
|
| 69 |
+
### Sub-model `interactive_segmentation_decoder.int8.tflite`
|
| 70 |
+
|
| 71 |
+
| Direction | Tensor | Shape | Type | Notes |
|
| 72 |
+
|---|---|---|---|---|
|
| 73 |
+
| Input | `input_21` | `[1, 768, 768, 3]` | `float32` | — |
|
| 74 |
+
| Input | `input_22` | `[1, 384, 384, 16]` | `float32` | — |
|
| 75 |
+
| Input | `input_23` | `[1, 192, 192, 24]` | `float32` | — |
|
| 76 |
+
| Input | `input_24` | `[1, 96, 96, 40]` | `float32` | — |
|
| 77 |
+
| Input | `input_25` | `[1, 48, 48, 80]` | `float32` | — |
|
| 78 |
+
| Input | `input_26` | `[1, 24, 24, 192]` | `float32` | — |
|
| 79 |
+
| Output | `Identity` | `[1, 768, 768, 1]` | `float32` | — |
|
| 80 |
+
|
| 81 |
+
### Sub-model `interactive_segmentation_encoder.int8.tflite`
|
| 82 |
+
|
| 83 |
+
| Direction | Tensor | Shape | Type | Notes |
|
| 84 |
+
|---|---|---|---|---|
|
| 85 |
+
| Input | `input_1` | `[-1, 768, 768, 3]` | `float32` | — |
|
| 86 |
+
| Output | `Identity` | `[-1, 384, 384, 16]` | `float32` | — |
|
| 87 |
+
| Output | `Identity_1` | `[-1, 192, 192, 24]` | `float32` | — |
|
| 88 |
+
| Output | `Identity_2` | `[-1, 96, 96, 40]` | `float32` | — |
|
| 89 |
+
| Output | `Identity_3` | `[-1, 48, 48, 80]` | `float32` | — |
|
| 90 |
+
| Output | `Identity_4` | `[-1, 24, 24, 192]` | `float32` | — |
|
| 91 |
+
|
| 92 |
+
## Use with MediaPipe Tasks
|
| 93 |
+
|
| 94 |
+
### Python Quickstart
|
| 95 |
+
|
| 96 |
+
```bash
|
| 97 |
+
pip install -U mediapipe huggingface_hub
|
| 98 |
+
```
|
| 99 |
+
|
| 100 |
+
```python
|
| 101 |
+
import mediapipe as mp
|
| 102 |
+
from huggingface_hub import hf_hub_download
|
| 103 |
+
|
| 104 |
+
model_path = hf_hub_download(repo_id="litert-community/MediaPipe-InteractiveSegmenter-MagicTouch", filename="interactive_segmentation.task")
|
| 105 |
+
|
| 106 |
+
BaseOptions = mp.tasks.BaseOptions
|
| 107 |
+
InteractiveSegmenter = mp.tasks.vision.InteractiveSegmenter
|
| 108 |
+
InteractiveSegmenterOptions = mp.tasks.vision.InteractiveSegmenterOptions
|
| 109 |
+
BrushMode = mp.tasks.vision.InteractiveSegmenterBrushMode
|
| 110 |
+
Stroke = mp.tasks.vision.InteractiveSegmenterStroke
|
| 111 |
+
StrokePoint = mp.tasks.vision.InteractiveSegmenterStrokePoint
|
| 112 |
+
|
| 113 |
+
options = InteractiveSegmenterOptions(
|
| 114 |
+
base_options=BaseOptions(model_asset_path=model_path)
|
| 115 |
+
)
|
| 116 |
+
|
| 117 |
+
with InteractiveSegmenter.create_from_options(options) as segmenter:
|
| 118 |
+
segmenter.set_image(mp.Image.create_from_file("input.jpg"))
|
| 119 |
+
strokes = [
|
| 120 |
+
Stroke(brush_mode=BrushMode.POSITIVE, points=[StrokePoint(x=0.5, y=0.5)], is_completed=True)
|
| 121 |
+
]
|
| 122 |
+
mask = segmenter.segment(strokes)
|
| 123 |
+
print("Segmented mask size:", mask.width, "x", mask.height)
|
| 124 |
+
```
|
| 125 |
+
|
| 126 |
+
### Cross-Platform Guides & Samples
|
| 127 |
+
|
| 128 |
+
- **Task overview & options:** [Interactive Segmenter documentation](https://developers.google.com/edge/mediapipe/solutions/vision/interactive_segmenter)
|
| 129 |
+
- **Platform setup guides:** [Python](https://developers.google.com/edge/mediapipe/solutions/setup_python) · [Web / JavaScript](https://developers.google.com/edge/mediapipe/solutions/setup_web) · [Android (Kotlin / Java)](https://developers.google.com/edge/mediapipe/solutions/setup_android) · [iOS (Swift)](https://developers.google.com/edge/mediapipe/solutions/setup_ios) · [Flutter (Dart)](https://developers.google.com/edge/mediapipe/solutions/setup_flutter)
|
| 130 |
+
- **Runnable sample apps:** [Web Demo](https://google-ai-edge.github.io/mediapipe-samples-web/#/vision/interactive_segmenter) · [Web Source](https://github.com/google-ai-edge/mediapipe-samples-web/blob/main/src/tasks/interactive-segmenter.ts) · [Android](https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/interactive_segmentation/android) · [iOS](https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/interactive_segmentation/ios) · [Python Notebook](https://colab.research.google.com/github/google-ai-edge/mediapipe-samples/blob/main/examples/interactive_segmentation/python/interactive_segmenter.ipynb) · [Flutter](https://github.com/google/flutter-mediapipe/tree/main/samples/vision/interactive_segmenter)
|
| 131 |
+
|
| 132 |
+
## Latency Benchmarks
|
| 133 |
+
|
| 134 |
+
End-to-end MediaPipe Task pipeline average latency reported by the [Interactive Segmenter documentation](https://developers.google.com/edge/mediapipe/solutions/vision/interactive_segmenter#benchmarks):
|
| 135 |
+
|
| 136 |
+
| Model Variant | Device | CPU Latency | GPU Latency |
|
| 137 |
+
|---|---|---:|---:|
|
| 138 |
+
| MagicTouch (v2 int8) | Pixel 10 | 208.2 ms | 580.6 ms |
|
| 139 |
+
|
| 140 |
+
## Intended Use, Limitations, and Fairness
|
| 141 |
+
|
| 142 |
+
Summarized from the bundled [`Model Card MagicTouch.pdf`](./Model%20Card%20MagicTouch.pdf) (Valentin Bazarevsky & Ben Hahn, Google, March 13, 2023):
|
| 143 |
+
|
| 144 |
+
- **Intended applications:** Interactive photo editing and semi-automatic object annotation (objects, animals, humans) from user clicks or brush strokes.
|
| 145 |
+
- **Out-of-scope uses:**
|
| 146 |
+
- **Any form of surveillance or identity recognition is explicitly out of scope and not enabled by this technology.**
|
| 147 |
+
- Human segmentation not consented to by the subject being segmented.
|
| 148 |
+
- Human life-critical decisions, or small objects / people > 4 meters (14 ft) from the camera.
|
| 149 |
+
- **Fairness evaluation (v1 MagicTouch PDF summary, 1,870 smartphone AR images vs. 98.74% human inter-annotator IoU):**
|
| 150 |
+
- **Monk Skin Tones 1–10:** average IoU **87.7%** (range **85.3%** Tones 8–10 to **89.4%** Tone 5; max discrepancy 4.1%).
|
| 151 |
+
- **Perceived gender:** average IoU **87.7%** (Male **87.3%**, Female **88.0%**; max discrepancy 0.7%).
|
| 152 |
+
|
| 153 |
+
## Provenance
|
| 154 |
+
|
| 155 |
+
Every model binary in this repository is byte-identical (unmodified weights, metadata, and `.task` bundle archive bytes) to the official artifact served by Google from `storage.googleapis.com/mediapipe-models`.
|
| 156 |
+
|
| 157 |
+
| Repo File | Upstream URL | GCS Generation | Upstream `Last-Modified` | Pinned Bucket Version |
|
| 158 |
+
|---|---|---|---|---|
|
| 159 |
+
| `interactive_segmentation.task` | [`interactive_segmentation.task`](https://storage.googleapis.com/mediapipe-models/interactive_segmenter_v2/magic_touch/int8/latest/interactive_segmentation.task) | `1781216228896365` | Thu, 11 Jun 2026 22:17:08 GMT | `/1/` |
|
| 160 |
+
|
| 161 |
+
## License and Citation
|
| 162 |
+
|
| 163 |
+
- License: `apache-2.0` — Stated as Apache License, Version 2.0 in the bundled upstream Google model-card PDF.
|
interactive_segmentation.task
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:38431bc66b883404e8397f74c3579404315b9b52b04a46c6346fe906a7309b03
|
| 3 |
+
size 30525312
|