Instructions to use mlx-community/EdgeTAM-fp16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- MLX
How to use mlx-community/EdgeTAM-fp16 with MLX:
# Download the model from the Hub pip install huggingface_hub[hf_xet] hf download mlx-community/EdgeTAM-fp16 --local-dir EdgeTAM-fp16
- sam2
How to use mlx-community/EdgeTAM-fp16 with sam2:
# Use SAM2 with images import torch from sam2.sam2_image_predictor import SAM2ImagePredictor predictor = SAM2ImagePredictor.from_pretrained("mlx-community/EdgeTAM-fp16") with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16): predictor.set_image(<your_image>) masks, _, _ = predictor.predict(<input_prompts>)# Use SAM2 with videos import torch from sam2.sam2_video_predictor import SAM2VideoPredictor predictor = SAM2VideoPredictor.from_pretrained("mlx-community/EdgeTAM-fp16") with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16): state = predictor.init_state(<your_video>) # add new prompts and instantly get the output on the same frame frame_idx, object_ids, masks = predictor.add_new_points(state, <your_prompts>) # propagate the prompts to get masklets throughout the video for frame_idx, object_ids, masks in predictor.propagate_in_video(state): ... - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- Atomic Chat
Card: v0.5.0 upstream-parity numbers; link the SAM 2.1-S quality tier
Browse files
README.md
CHANGED
|
@@ -14,33 +14,29 @@ tags:
|
|
| 14 |
|
| 15 |
# mlx-community/EdgeTAM-fp16
|
| 16 |
|
| 17 |
-
[EdgeTAM](https://github.com/facebookresearch/EdgeTAM)
|
| 18 |
-
|
| 19 |
-
[`mlx-edgetam-swift`](https://github.com/xocialize/mlx-edgetam-swift) Swift package (MLXEngine
|
| 20 |
-
`promptSegment` + `trackObject` ModelPackage).
|
|
|
|
| 21 |
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
+ MemoryEncoder + MemoryAttention (RoPE-2D) + the SAM2 memory-bank state machine β every op parity-locked
|
| 27 |
-
vs the oracle; full masklet propagation min-IoU 0.92. This single `-fp16` file carries both (874 tensors).
|
| 28 |
|
| 29 |
-
|
|
|
|
|
|
|
| 30 |
|
|
|
|
| 31 |
```swift
|
| 32 |
-
//
|
| 33 |
import EdgeTAM
|
| 34 |
-
// Image: click β object mask
|
| 35 |
let p = try EdgeTAMPredictor.fromPretrained(weightsPath, dtype: .float16)
|
| 36 |
-
p.setImage(sourceCGImage)
|
| 37 |
-
let
|
| 38 |
-
// Video: click on a frame β per-frame masklet
|
| 39 |
let vp = try EdgeTAMVideoPredictor.fromPretrained(weightsPath, dtype: .float16)
|
| 40 |
-
let track = vp.track(frames: cgImages, clickFrame: 0, points: [[210, 350]], labels: [1])
|
| 41 |
```
|
| 42 |
|
| 43 |
-
Or as an MLXEngine ModelPackage (`MLXEdgeTAM.EdgeTAMPackage`) β `promptSegment` (image) + `trackObject`
|
| 44 |
-
(video) surfaces β resolving this repo via the Hub.
|
| 45 |
-
|
| 46 |
Weights: Apache-2.0 (facebookresearch/EdgeTAM). Port code: MIT.
|
|
|
|
| 14 |
|
| 15 |
# mlx-community/EdgeTAM-fp16
|
| 16 |
|
| 17 |
+
[EdgeTAM](https://github.com/facebookresearch/EdgeTAM) is on-device SAM 2 for promptable segmentation and video
|
| 18 |
+
tracking. This repo holds it converted to Apple **MLX** (fp16) for the
|
| 19 |
+
[`mlx-edgetam-swift`](https://github.com/xocialize/mlx-edgetam-swift) Swift package (`EdgeTAMPackage`, an MLXEngine
|
| 20 |
+
`promptSegment` + `trackObject` ModelPackage). There are 874 tensors (image + video), checked against the official
|
| 21 |
+
`edgetam.pt` (fp16 rounding only).
|
| 22 |
|
| 23 |
+
**Parity (v0.5.0).** Measured on the CPU fp32 stream against upstream PyTorch, through the public predictors:
|
| 24 |
+
- **image:** image_embed relative error β€ 2.4e-4, and every click and box mask matches at IoU 1.0000 on square
|
| 25 |
+
and non-square images;
|
| 26 |
+
- **video:** 5-frame point and box tracks stay at IoU β₯ 0.999.
|
|
|
|
|
|
|
| 27 |
|
| 28 |
+
**Expect weak single clicks on flat-shaded or synthetic images.** Upstream EdgeTAM behaves the same way; a box
|
| 29 |
+
prompt is its reliable input. For a better click-selection tier see
|
| 30 |
+
[SAM2.1-hiera-small-fp16](https://huggingface.co/mlx-community/SAM2.1-hiera-small-fp16).
|
| 31 |
|
| 32 |
+
## Use
|
| 33 |
```swift
|
| 34 |
+
// .package(url: "https://github.com/xocialize/mlx-edgetam-swift", from: "0.6.0")
|
| 35 |
import EdgeTAM
|
|
|
|
| 36 |
let p = try EdgeTAMPredictor.fromPretrained(weightsPath, dtype: .float16)
|
| 37 |
+
p.setImage(sourceCGImage) // encoder once per image
|
| 38 |
+
let r = p.predict(point: (500, 375)) // r.mask, r.soft (anti-aliased), r.score
|
|
|
|
| 39 |
let vp = try EdgeTAMVideoPredictor.fromPretrained(weightsPath, dtype: .float16)
|
|
|
|
| 40 |
```
|
| 41 |
|
|
|
|
|
|
|
|
|
|
| 42 |
Weights: Apache-2.0 (facebookresearch/EdgeTAM). Port code: MIT.
|