schmidt-sebastian commited on
Commit
61de3ef
·
0 Parent(s):

Initial upload of MediaPipe-InteractiveSegmenter-MagicTouch from MediaPipe Solutions

Browse files
.gitattributes ADDED
@@ -0,0 +1,38 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.litertlm filter=lfs diff=lfs merge=lfs -text
12
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
13
+ *.model filter=lfs diff=lfs merge=lfs -text
14
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
15
+ *.npy filter=lfs diff=lfs merge=lfs -text
16
+ *.npz filter=lfs diff=lfs merge=lfs -text
17
+ *.onnx filter=lfs diff=lfs merge=lfs -text
18
+ *.ot filter=lfs diff=lfs merge=lfs -text
19
+ *.parquet filter=lfs diff=lfs merge=lfs -text
20
+ *.pb filter=lfs diff=lfs merge=lfs -text
21
+ *.pdf filter=lfs diff=lfs merge=lfs -text
22
+ *.pickle filter=lfs diff=lfs merge=lfs -text
23
+ *.pkl filter=lfs diff=lfs merge=lfs -text
24
+ *.pt filter=lfs diff=lfs merge=lfs -text
25
+ *.pth filter=lfs diff=lfs merge=lfs -text
26
+ *.rar filter=lfs diff=lfs merge=lfs -text
27
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
28
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
29
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
30
+ *.tar filter=lfs diff=lfs merge=lfs -text
31
+ *.task filter=lfs diff=lfs merge=lfs -text
32
+ *.tflite filter=lfs diff=lfs merge=lfs -text
33
+ *.tgz filter=lfs diff=lfs merge=lfs -text
34
+ *.wasm filter=lfs diff=lfs merge=lfs -text
35
+ *.xz filter=lfs diff=lfs merge=lfs -text
36
+ *.zip filter=lfs diff=lfs merge=lfs -text
37
+ *.zst filter=lfs diff=lfs merge=lfs -text
38
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
Model Card MagicTouch.pdf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b4accf5f7be991903646e6dd9765faa6974cdf69c5a9e84841684dffa27df0f1
3
+ size 301841
README.md ADDED
@@ -0,0 +1,163 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: mediapipe
4
+ pipeline_tag: mask-generation
5
+ tags:
6
+ - mediapipe
7
+ - litert
8
+ - tflite
9
+ - mask-generation
10
+ - on-device
11
+ - android
12
+ - ios
13
+ - web
14
+ - python
15
+ - flutter
16
+ - interactive-segmentation
17
+ - promptable-segmentation
18
+ - magictouch
19
+ - int8
20
+ ---
21
+
22
+ # MediaPipe MagicTouch (Interactive Segmenter)
23
+
24
+ Stateful point/stroke-prompted interactive segmentation bundle (`interactive_segmentation.task`, 30.5 MB, INT8 v2 encoder/decoder at `768 x 768`) that encodes an image once and iteratively segments user-indicated objects via positive/negative strokes in the [MediaPipe Interactive Segmenter (`InteractiveSegmenter`)](https://developers.google.com/edge/mediapipe/solutions/vision/interactive_segmenter) task.
25
+
26
+ <div style="border: 1px solid rgba(59, 130, 246, 0.28); background-color: rgba(59, 130, 246, 0.06); border-radius: 8px; padding: 12px 16px; margin: 16px 0;">
27
+ <p style="margin: 0 0 6px 0;"><strong>Documentation:</strong> <a href="https://developers.google.com/edge/mediapipe/solutions/vision/interactive_segmenter">MediaPipe Interactive Segmenter Docs</a></p>
28
+ <p style="margin: 0 0 6px 0;"><strong>Samples:</strong> <a href="https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/interactive_segmentation/android">Android</a> · <a href="https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/interactive_segmentation/ios">iOS</a> · <a href="https://colab.research.google.com/github/google-ai-edge/mediapipe-samples/blob/main/examples/interactive_segmentation/python/interactive_segmenter.ipynb">Python Colab</a> · <a href="https://github.com/google-ai-edge/mediapipe-samples-web/blob/main/src/tasks/interactive-segmenter.ts">Web</a> · <a href="https://github.com/google/flutter-mediapipe/tree/main/samples/vision/interactive_segmenter">Flutter</a></p>
29
+ <p style="margin: 0;"><strong>Try live:</strong> <a href="https://google-ai-edge.github.io/mediapipe-samples-web/#/vision/interactive_segmenter">Web Demo</a></p>
30
+ </div>
31
+
32
+ `interactive_segmentation.task` (`interactive_segmenter_v2`) separates image feature extraction from interactive decoding for low-latency refinement:
33
+
34
+ 1. **`interactive_segmentation_encoder.int8.tflite` (28.3 MB):** Runs once in `set_image()`, encoding `[-1, 768, 768, 3]` RGB images into 5 multi-scale feature maps (`384×384×16` .. `24×24×192`).
35
+ 2. **`interactive_segmentation_decoder.int8.tflite` (2.3 MB):** Runs per user stroke call in `segment(strokes)`, combining the prompt canvas (`[1, 768, 768, 3]`) and cached encoder features into a `[1, 768, 768, 1]` mask.
36
+
37
+ **Bundle version note:** The bundled [`Model Card MagicTouch.pdf`](./Model%20Card%20MagicTouch.pdf) (March 13, 2023) documents the earlier v1 single-graph `512 x 512 x 4` (≈6 MB) FP model (≈30 FPS on Pixel 7 GPU), whereas this `.task` bundle packages the upgraded **768×768 INT8 two-stage v2** encoder/decoder.
38
+
39
+ ## Files
40
+
41
+ | File | Precision | Size | SHA-256 |
42
+ |---|---|---:|---|
43
+ | [`interactive_segmentation.task`](https://huggingface.co/litert-community/MediaPipe-InteractiveSegmenter-MagicTouch/blob/main/interactive_segmentation.task) | `int8` | 30.53 MB (30,525,312 B) | `38431bc66b883404e8397f74c3579404315b9b52b04a46c6346fe906a7309b03` |
44
+
45
+ ### Companion Files in Repo Root
46
+
47
+ | File | Role | Size | SHA-256 |
48
+ |---|---|---:|---|
49
+ | [`Model Card MagicTouch.pdf`](./Model%20Card%20MagicTouch.pdf) | Upstream Google model card (PDF) | 301.8 KB (301,841 B) | `b4accf5f7be991903646e6dd9765faa6974cdf69c5a9e84841684dffa27df0f1` |
50
+
51
+ ### Download
52
+
53
+ ```bash
54
+ pip install -U huggingface_hub
55
+ hf download litert-community/MediaPipe-InteractiveSegmenter-MagicTouch interactive_segmentation.task --local-dir .
56
+ ```
57
+
58
+ ## Inputs, Outputs, and Model Specifications
59
+
60
+ ### Bundle Contents (`interactive_segmentation.task`)
61
+
62
+ | Bundled Member | Uncompressed Size | SHA-256 (prefix) |
63
+ |---|---:|---|
64
+ | `interactive_segmentation_decoder.int8.tflite` | 2.27 MB (2,268,000 B) | `140d36971403c38b…` |
65
+ | `interactive_segmentation_encoder.int8.tflite` | 28.26 MB (28,256,696 B) | `0edbceec02cb1570…` |
66
+ | `manifest.pb` | 0.1 KB (50 B) | `c18392128d95dc4d…` |
67
+ | `metadata.pb` | 0.0 KB (12 B) | `d4945e22f0e49ea2…` |
68
+
69
+ ### Sub-model `interactive_segmentation_decoder.int8.tflite`
70
+
71
+ | Direction | Tensor | Shape | Type | Notes |
72
+ |---|---|---|---|---|
73
+ | Input | `input_21` | `[1, 768, 768, 3]` | `float32` | — |
74
+ | Input | `input_22` | `[1, 384, 384, 16]` | `float32` | — |
75
+ | Input | `input_23` | `[1, 192, 192, 24]` | `float32` | — |
76
+ | Input | `input_24` | `[1, 96, 96, 40]` | `float32` | — |
77
+ | Input | `input_25` | `[1, 48, 48, 80]` | `float32` | — |
78
+ | Input | `input_26` | `[1, 24, 24, 192]` | `float32` | — |
79
+ | Output | `Identity` | `[1, 768, 768, 1]` | `float32` | — |
80
+
81
+ ### Sub-model `interactive_segmentation_encoder.int8.tflite`
82
+
83
+ | Direction | Tensor | Shape | Type | Notes |
84
+ |---|---|---|---|---|
85
+ | Input | `input_1` | `[-1, 768, 768, 3]` | `float32` | — |
86
+ | Output | `Identity` | `[-1, 384, 384, 16]` | `float32` | — |
87
+ | Output | `Identity_1` | `[-1, 192, 192, 24]` | `float32` | — |
88
+ | Output | `Identity_2` | `[-1, 96, 96, 40]` | `float32` | — |
89
+ | Output | `Identity_3` | `[-1, 48, 48, 80]` | `float32` | — |
90
+ | Output | `Identity_4` | `[-1, 24, 24, 192]` | `float32` | — |
91
+
92
+ ## Use with MediaPipe Tasks
93
+
94
+ ### Python Quickstart
95
+
96
+ ```bash
97
+ pip install -U mediapipe huggingface_hub
98
+ ```
99
+
100
+ ```python
101
+ import mediapipe as mp
102
+ from huggingface_hub import hf_hub_download
103
+
104
+ model_path = hf_hub_download(repo_id="litert-community/MediaPipe-InteractiveSegmenter-MagicTouch", filename="interactive_segmentation.task")
105
+
106
+ BaseOptions = mp.tasks.BaseOptions
107
+ InteractiveSegmenter = mp.tasks.vision.InteractiveSegmenter
108
+ InteractiveSegmenterOptions = mp.tasks.vision.InteractiveSegmenterOptions
109
+ BrushMode = mp.tasks.vision.InteractiveSegmenterBrushMode
110
+ Stroke = mp.tasks.vision.InteractiveSegmenterStroke
111
+ StrokePoint = mp.tasks.vision.InteractiveSegmenterStrokePoint
112
+
113
+ options = InteractiveSegmenterOptions(
114
+ base_options=BaseOptions(model_asset_path=model_path)
115
+ )
116
+
117
+ with InteractiveSegmenter.create_from_options(options) as segmenter:
118
+ segmenter.set_image(mp.Image.create_from_file("input.jpg"))
119
+ strokes = [
120
+ Stroke(brush_mode=BrushMode.POSITIVE, points=[StrokePoint(x=0.5, y=0.5)], is_completed=True)
121
+ ]
122
+ mask = segmenter.segment(strokes)
123
+ print("Segmented mask size:", mask.width, "x", mask.height)
124
+ ```
125
+
126
+ ### Cross-Platform Guides & Samples
127
+
128
+ - **Task overview & options:** [Interactive Segmenter documentation](https://developers.google.com/edge/mediapipe/solutions/vision/interactive_segmenter)
129
+ - **Platform setup guides:** [Python](https://developers.google.com/edge/mediapipe/solutions/setup_python) · [Web / JavaScript](https://developers.google.com/edge/mediapipe/solutions/setup_web) · [Android (Kotlin / Java)](https://developers.google.com/edge/mediapipe/solutions/setup_android) · [iOS (Swift)](https://developers.google.com/edge/mediapipe/solutions/setup_ios) · [Flutter (Dart)](https://developers.google.com/edge/mediapipe/solutions/setup_flutter)
130
+ - **Runnable sample apps:** [Web Demo](https://google-ai-edge.github.io/mediapipe-samples-web/#/vision/interactive_segmenter) · [Web Source](https://github.com/google-ai-edge/mediapipe-samples-web/blob/main/src/tasks/interactive-segmenter.ts) · [Android](https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/interactive_segmentation/android) · [iOS](https://github.com/google-ai-edge/mediapipe-samples/tree/main/examples/interactive_segmentation/ios) · [Python Notebook](https://colab.research.google.com/github/google-ai-edge/mediapipe-samples/blob/main/examples/interactive_segmentation/python/interactive_segmenter.ipynb) · [Flutter](https://github.com/google/flutter-mediapipe/tree/main/samples/vision/interactive_segmenter)
131
+
132
+ ## Latency Benchmarks
133
+
134
+ End-to-end MediaPipe Task pipeline average latency reported by the [Interactive Segmenter documentation](https://developers.google.com/edge/mediapipe/solutions/vision/interactive_segmenter#benchmarks):
135
+
136
+ | Model Variant | Device | CPU Latency | GPU Latency |
137
+ |---|---|---:|---:|
138
+ | MagicTouch (v2 int8) | Pixel 10 | 208.2 ms | 580.6 ms |
139
+
140
+ ## Intended Use, Limitations, and Fairness
141
+
142
+ Summarized from the bundled [`Model Card MagicTouch.pdf`](./Model%20Card%20MagicTouch.pdf) (Valentin Bazarevsky & Ben Hahn, Google, March 13, 2023):
143
+
144
+ - **Intended applications:** Interactive photo editing and semi-automatic object annotation (objects, animals, humans) from user clicks or brush strokes.
145
+ - **Out-of-scope uses:**
146
+ - **Any form of surveillance or identity recognition is explicitly out of scope and not enabled by this technology.**
147
+ - Human segmentation not consented to by the subject being segmented.
148
+ - Human life-critical decisions, or small objects / people > 4 meters (14 ft) from the camera.
149
+ - **Fairness evaluation (v1 MagicTouch PDF summary, 1,870 smartphone AR images vs. 98.74% human inter-annotator IoU):**
150
+ - **Monk Skin Tones 1–10:** average IoU **87.7%** (range **85.3%** Tones 8–10 to **89.4%** Tone 5; max discrepancy 4.1%).
151
+ - **Perceived gender:** average IoU **87.7%** (Male **87.3%**, Female **88.0%**; max discrepancy 0.7%).
152
+
153
+ ## Provenance
154
+
155
+ Every model binary in this repository is byte-identical (unmodified weights, metadata, and `.task` bundle archive bytes) to the official artifact served by Google from `storage.googleapis.com/mediapipe-models`.
156
+
157
+ | Repo File | Upstream URL | GCS Generation | Upstream `Last-Modified` | Pinned Bucket Version |
158
+ |---|---|---|---|---|
159
+ | `interactive_segmentation.task` | [`interactive_segmentation.task`](https://storage.googleapis.com/mediapipe-models/interactive_segmenter_v2/magic_touch/int8/latest/interactive_segmentation.task) | `1781216228896365` | Thu, 11 Jun 2026 22:17:08 GMT | `/1/` |
160
+
161
+ ## License and Citation
162
+
163
+ - License: `apache-2.0` — Stated as Apache License, Version 2.0 in the bundled upstream Google model-card PDF.
interactive_segmentation.task ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:38431bc66b883404e8397f74c3579404315b9b52b04a46c6346fe906a7309b03
3
+ size 30525312