xocialize commited on
Commit
fd5dabf
Β·
verified Β·
1 Parent(s): 893bb5a

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +19 -12
README.md CHANGED
@@ -14,26 +14,33 @@ tags:
14
 
15
  # mlx-community/EdgeTAM-fp16
16
 
17
- [EdgeTAM](https://github.com/facebookresearch/EdgeTAM) β€” on-device SAM 2 for promptable segmentation β€”
18
- converted to **Apple MLX** (`-fp16`) for the [`mlx-edgetam-swift`](https://github.com/xocialize/mlx-edgetam-swift)
19
- Swift package (MLXEngine `promptSegment` ModelPackage). **Image-mode** (point/box β†’ mask); 22Γ— faster
20
- than SAM 2, 16 FPS on iPhone 15 Pro Max.
21
 
22
- From-scratch MLX-Swift architecture port: RepViT-M1 image encoder + FPN + SAM prompt encoder + two-way
23
- mask decoder. Parity-locked vs the PyTorch oracle on the CPU stream (image_embed 9.7e-6, mask logits
24
- 8.9e-5). This `-fp16` build is functionally identical (end-to-end mask **IoU 0.99** vs PyTorch; the
25
- thresholded mask absorbs fp16 weight-rounding).
 
 
26
 
27
  ## Use
28
 
29
  ```swift
30
  // Package.swift β†’ .package(url: "https://github.com/xocialize/mlx-edgetam-swift", from: "0.1.0")
31
  import EdgeTAM
32
- let predictor = try EdgeTAMPredictor.fromPretrained(weightsPath, dtype: .float16)
33
- predictor.setImage(sourceCGImage)
34
- let (mask, score, _, _) = predictor.predict(point: (500, 375)) // click β†’ object mask
 
 
 
 
35
  ```
36
 
37
- Or as an MLXEngine `promptSegment` ModelPackage (`MLXEdgeTAM.EdgeTAMPackage`), resolving this repo via the Hub.
 
38
 
39
  Weights: Apache-2.0 (facebookresearch/EdgeTAM). Port code: MIT.
 
14
 
15
  # mlx-community/EdgeTAM-fp16
16
 
17
+ [EdgeTAM](https://github.com/facebookresearch/EdgeTAM) β€” on-device SAM 2 for promptable segmentation +
18
+ video tracking β€” converted to **Apple MLX** (`-fp16`) for the
19
+ [`mlx-edgetam-swift`](https://github.com/xocialize/mlx-edgetam-swift) Swift package (MLXEngine
20
+ `promptSegment` + `trackObject` ModelPackage). 22Γ— faster than SAM 2, 16 FPS on iPhone 15 Pro Max.
21
 
22
+ From-scratch MLX-Swift architecture port. **Image-mode** (point/box β†’ mask): RepViT-M1 encoder + FPN +
23
+ SAM prompt encoder + two-way mask decoder β€” parity-locked vs the PyTorch oracle on the CPU stream
24
+ (image_embed 9.7e-6, mask logits 8.9e-5; end-to-end mask **IoU 0.99** vs PyTorch). **Video-mode**
25
+ (`trackObject`, click on one frame β†’ per-frame masklet): adds the video memory stack β€” PerceiverResampler
26
+ + MemoryEncoder + MemoryAttention (RoPE-2D) + the SAM2 memory-bank state machine β€” every op parity-locked
27
+ vs the oracle; full masklet propagation min-IoU 0.92. This single `-fp16` file carries both (874 tensors).
28
 
29
  ## Use
30
 
31
  ```swift
32
  // Package.swift β†’ .package(url: "https://github.com/xocialize/mlx-edgetam-swift", from: "0.1.0")
33
  import EdgeTAM
34
+ // Image: click β†’ object mask
35
+ let p = try EdgeTAMPredictor.fromPretrained(weightsPath, dtype: .float16)
36
+ p.setImage(sourceCGImage)
37
+ let (mask, score, _, _) = p.predict(point: (500, 375))
38
+ // Video: click on a frame β†’ per-frame masklet
39
+ let vp = try EdgeTAMVideoPredictor.fromPretrained(weightsPath, dtype: .float16)
40
+ let track = vp.track(frames: cgImages, clickFrame: 0, points: [[210, 350]], labels: [1])
41
  ```
42
 
43
+ Or as an MLXEngine ModelPackage (`MLXEdgeTAM.EdgeTAMPackage`) β€” `promptSegment` (image) + `trackObject`
44
+ (video) surfaces β€” resolving this repo via the Hub.
45
 
46
  Weights: Apache-2.0 (facebookresearch/EdgeTAM). Port code: MIT.