xocialize commited on
Commit
e7770c1
Β·
verified Β·
1 Parent(s): fd5dabf

Card: v0.5.0 upstream-parity numbers; link the SAM 2.1-S quality tier

Browse files
Files changed (1) hide show
  1. README.md +16 -20
README.md CHANGED
@@ -14,33 +14,29 @@ tags:
14
 
15
  # mlx-community/EdgeTAM-fp16
16
 
17
- [EdgeTAM](https://github.com/facebookresearch/EdgeTAM) β€” on-device SAM 2 for promptable segmentation +
18
- video tracking β€” converted to **Apple MLX** (`-fp16`) for the
19
- [`mlx-edgetam-swift`](https://github.com/xocialize/mlx-edgetam-swift) Swift package (MLXEngine
20
- `promptSegment` + `trackObject` ModelPackage). 22Γ— faster than SAM 2, 16 FPS on iPhone 15 Pro Max.
 
21
 
22
- From-scratch MLX-Swift architecture port. **Image-mode** (point/box β†’ mask): RepViT-M1 encoder + FPN +
23
- SAM prompt encoder + two-way mask decoder β€” parity-locked vs the PyTorch oracle on the CPU stream
24
- (image_embed 9.7e-6, mask logits 8.9e-5; end-to-end mask **IoU 0.99** vs PyTorch). **Video-mode**
25
- (`trackObject`, click on one frame β†’ per-frame masklet): adds the video memory stack β€” PerceiverResampler
26
- + MemoryEncoder + MemoryAttention (RoPE-2D) + the SAM2 memory-bank state machine β€” every op parity-locked
27
- vs the oracle; full masklet propagation min-IoU 0.92. This single `-fp16` file carries both (874 tensors).
28
 
29
- ## Use
 
 
30
 
 
31
  ```swift
32
- // Package.swift β†’ .package(url: "https://github.com/xocialize/mlx-edgetam-swift", from: "0.1.0")
33
  import EdgeTAM
34
- // Image: click β†’ object mask
35
  let p = try EdgeTAMPredictor.fromPretrained(weightsPath, dtype: .float16)
36
- p.setImage(sourceCGImage)
37
- let (mask, score, _, _) = p.predict(point: (500, 375))
38
- // Video: click on a frame β†’ per-frame masklet
39
  let vp = try EdgeTAMVideoPredictor.fromPretrained(weightsPath, dtype: .float16)
40
- let track = vp.track(frames: cgImages, clickFrame: 0, points: [[210, 350]], labels: [1])
41
  ```
42
 
43
- Or as an MLXEngine ModelPackage (`MLXEdgeTAM.EdgeTAMPackage`) β€” `promptSegment` (image) + `trackObject`
44
- (video) surfaces β€” resolving this repo via the Hub.
45
-
46
  Weights: Apache-2.0 (facebookresearch/EdgeTAM). Port code: MIT.
 
14
 
15
  # mlx-community/EdgeTAM-fp16
16
 
17
+ [EdgeTAM](https://github.com/facebookresearch/EdgeTAM) is on-device SAM 2 for promptable segmentation and video
18
+ tracking. This repo holds it converted to Apple **MLX** (fp16) for the
19
+ [`mlx-edgetam-swift`](https://github.com/xocialize/mlx-edgetam-swift) Swift package (`EdgeTAMPackage`, an MLXEngine
20
+ `promptSegment` + `trackObject` ModelPackage). There are 874 tensors (image + video), checked against the official
21
+ `edgetam.pt` (fp16 rounding only).
22
 
23
+ **Parity (v0.5.0).** Measured on the CPU fp32 stream against upstream PyTorch, through the public predictors:
24
+ - **image:** image_embed relative error ≀ 2.4e-4, and every click and box mask matches at IoU 1.0000 on square
25
+ and non-square images;
26
+ - **video:** 5-frame point and box tracks stay at IoU β‰₯ 0.999.
 
 
27
 
28
+ **Expect weak single clicks on flat-shaded or synthetic images.** Upstream EdgeTAM behaves the same way; a box
29
+ prompt is its reliable input. For a better click-selection tier see
30
+ [SAM2.1-hiera-small-fp16](https://huggingface.co/mlx-community/SAM2.1-hiera-small-fp16).
31
 
32
+ ## Use
33
  ```swift
34
+ // .package(url: "https://github.com/xocialize/mlx-edgetam-swift", from: "0.6.0")
35
  import EdgeTAM
 
36
  let p = try EdgeTAMPredictor.fromPretrained(weightsPath, dtype: .float16)
37
+ p.setImage(sourceCGImage) // encoder once per image
38
+ let r = p.predict(point: (500, 375)) // r.mask, r.soft (anti-aliased), r.score
 
39
  let vp = try EdgeTAMVideoPredictor.fromPretrained(weightsPath, dtype: .float16)
 
40
  ```
41
 
 
 
 
42
  Weights: Apache-2.0 (facebookresearch/EdgeTAM). Port code: MIT.