| --- |
| license: other |
| license_name: sam-license |
| license_link: https://huggingface.co/facebook/sam3/blob/main/LICENSE |
| base_model: facebook/sam3 |
| pipeline_tag: mask-generation |
| library_name: coreai |
| tags: |
| - core-ai |
| - ios |
| - video-segmentation |
| - sam3 |
| --- |
| |
| # SAM 3 video for Core AI |
|
|
| [`facebook/sam3`](https://huggingface.co/facebook/sam3) exported to Apple Core AI as a video segmenter with the [`apple/coreai-models`](https://github.com/apple/coreai-models/tree/main/models/sam3_video) recipe, at 672 px for iPhone: |
|
|
| ```sh |
| uv run models/sam3_video/export.py --dtype float16 --image-size 672 |
| ``` |
|
|
| | Folder | Contents | Size | Platforms | |
| | ------------- | ------------------------------------------------------------------------------------------------------------------------------ | ------ | ---------- | |
| | `sam3-video/` | `CoreAIVideoSegmentation` bundle: `sam3_video_672_float16.aimodel` (image and text encoders, detector, tracker and memory encoder functions), the CLIP `tokenizer/`, `metadata.json` (`kind: video_segmenter`, tracking defaults) | 1.6 GB | iOS, macOS | |
|
|
| Give it a video and text prompts (`cat`, `the red car`). It finds the matching objects and tracks them from frame to frame with stable ids. The recipe also accepts 336 px (faster, less accurate on small objects) and 1008 px (the default, slower). |
|
|
| ## Use |
|
|
| With [`@react-native-ai/core-ai`](https://github.com/callstackincubator/ai/tree/main/packages/core-ai), download `sam3-video/` into app storage, then: |
|
|
| ```ts |
| const { output } = await CoreAI.runTask( |
| 'video-segmentation', |
| { id: 'sam3-video', sourceType: 'file', sourceUri: pathToFolder }, |
| { videoUri, data: { prompts: ['cat'] } }, |
| { outputVideo: true } |
| ) |
| ``` |
|
|
| In Swift, load the folder with `VideoSegmenter(resourcesAt:)` from the `CoreAIVideoSegmentation` product of `coreai-models`. |
|
|
| ## License |
|
|
| SAM 3 is licensed by Meta under the [SAM License](./LICENSE), which applies to this converted copy too: a copy of the agreement is included, as it requires. This repository only converts the model to Core AI format. |
|
|
