# Use SAM2 with videos
import torch
from sam2.sam2_video_predictor import SAM2VideoPredictor
predictor = SAM2VideoPredictor.from_pretrained("mlx-community/EdgeTAM-fp16")
with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
state = predictor.init_state(<your_video>)
# add new prompts and instantly get the output on the same frame
frame_idx, object_ids, masks = predictor.add_new_points(state, <your_prompts>)
# propagate the prompts to get masklets throughout the video
for frame_idx, object_ids, masks in predictor.propagate_in_video(state):
...mlx-community/EdgeTAM-fp16
EdgeTAM is on-device SAM 2 for promptable segmentation and video
tracking. This repo holds it converted to Apple MLX (fp16) for the
mlx-edgetam-swift Swift package (EdgeTAMPackage, an MLXEngine
promptSegment + trackObject ModelPackage). There are 874 tensors (image + video), checked against the official
edgetam.pt (fp16 rounding only).
Parity (v0.5.0). Measured on the CPU fp32 stream against upstream PyTorch, through the public predictors:
- image: image_embed relative error ≤ 2.4e-4, and every click and box mask matches at IoU 1.0000 on square and non-square images;
- video: 5-frame point and box tracks stay at IoU ≥ 0.999.
Expect weak single clicks on flat-shaded or synthetic images. Upstream EdgeTAM behaves the same way; a box prompt is its reliable input. For a better click-selection tier see SAM2.1-hiera-small-fp16.
Use
// .package(url: "https://github.com/xocialize/mlx-edgetam-swift", from: "0.6.0")
import EdgeTAM
let p = try EdgeTAMPredictor.fromPretrained(weightsPath, dtype: .float16)
p.setImage(sourceCGImage) // encoder once per image
let r = p.predict(point: (500, 375)) // r.mask, r.soft (anti-aliased), r.score
let vp = try EdgeTAMVideoPredictor.fromPretrained(weightsPath, dtype: .float16)
Weights: Apache-2.0 (facebookresearch/EdgeTAM). Port code: MIT.
Model size
13.9M params
Tensor type
F16
·
Hardware compatibility
Log In to add your hardware
Quantized
# Use SAM2 with images import torch from sam2.sam2_image_predictor import SAM2ImagePredictor predictor = SAM2ImagePredictor.from_pretrained("mlx-community/EdgeTAM-fp16") with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16): predictor.set_image(<your_image>) masks, _, _ = predictor.predict(<input_prompts>)