SAM 3 decoder with instance outputs (rank-based top-K selection)

README.md

@@ -38,14 +38,16 @@ the **SAM License** (`sam3/LICENSE`), which must accompany any redistribution.
 | `onnx/sam3-image-encoder-1008-fp16.onnx` (+ `.onnx.data`) | Image encoder. `pixel_values` [1,3,1008,1008] fp16 (RGB / 255, mean 0.5, std 0.5) → `fpn_feat_0..2` [1,256,288²/144²/72²] (+ constant `fpn_pos_0..2`) |
 | `onnx/sam3-text-encoder-ctx32-fp16.onnx` | Text encoder. `input_ids` [N,32] int64 (CLIP BPE, start 49406, end 49407, padded with 0), `attention_mask` [N,32] int64 → `text_features` [32,N,256], `text_mask` [N,32] bool (True = padding) |
 | `onnx/sam3-decoder-1008-p16-top64-fp16.onnx` | Prompted decoder, 16 prompt slots (pad by repeating a prompt). `fpn_feat_0..2`, `text_features` [32,16,256], `text_mask` [16,32] bool → `prompt_scores` [16,288,288] (per prompt: max over the detections of score × mask, and the semantic map × presence), `scores` [16,200], `boxes` [16,200,4] (cx, cy, w, h normalised) |
+| `onnx/sam3-decoder-1008-p16-top64-inst-fp16.onnx` | Prompted decoder with instances (used by OpenSfM since this revision): as above, but → `prompt_scores` [16,288,288], `instance_index` [16,288,288] (per prompt and pixel, the detection 0..63 whose mask wins the pixel, -1: none), `instance_scores` [16,64] (the 64 best detections' scores, 0 below 0.3), `instance_boxes` [16,64,4] (cx, cy, w, h normalised) |
 | `coreml/sam3-image-encoder-1008-fp16.mlpackage` | Image encoder, Core ML ML program (macOS 15+); same interface, outputs `fpn_feat_0..2` |
 | `coreml/sam3-decoder-1008-p16-top64-fp16.mlpackage` | Prompted decoder, Core ML; as the ONNX one with `text_mask` as fp16 (1 = padding) |
+| `coreml/sam3-decoder-1008-p16-top64-inst-fp16.mlpackage` | Prompted decoder with instances, Core ML; as the ONNX one with `text_mask` as fp16 |
 | `embeddings/sam3-text-ctx32-fp16.npz` | Precomputed prompts: `prompts` [N], `features` [N,32,256], `mask` [N,32] |
 | `tokenizer/sam3-clip-bpe-tokenizer.json` | CLIP BPE tokenizer (Hugging Face `tokenizers`) |
 
 Naming: `<model>-<component>-<input size>[-<variant>]-<precision>.<format>`
 (`p16`: 16 prompt slots, `top64`: masks computed for the 64 best detections
-per prompt, `ctx32`: 32 text tokens).
+per prompt, `inst`: instance outputs, `ctx32`: 32 text tokens).
 
 ### Export
 
@@ -55,7 +57,10 @@ Exported with the Ultralytics SAM 3 implementation and the export patches of
 64 best detections only, window partitioning and mask products rewritten for
 Core ML (rank ≤ 5, matmul), opset 18. The Core ML encoder is ONNX Runtime's
 ML-program conversion with its constants moved to the weight blob; the Core
-ML decoder is converted from PyTorch with coremltools.
+ML decoder is converted from PyTorch with coremltools. The `inst` decoders
+select the 64 best detections with a one-hot matmul over the detections'
+ranks rather than `topk`, whose indices come out wrong on Apple GPUs (the
+older decoders lose detections there).
 
 On an Apple M5 Pro (Core ML, CPU + GPU): encoder 0.42 s and decoder 0.8 s
 per image; label maps agree with the PyTorch reference (transformers

manifest.json

@@ -18,6 +18,7 @@
         0.5
       ],
       "mask_size": 288,
+      "instance_slots": 64,
       "components": {
         "image_encoder": {
           "onnx": "sam3/onnx/sam3-image-encoder-1008-fp16.onnx",
@@ -27,8 +28,8 @@
           "onnx": "sam3/onnx/sam3-text-encoder-ctx32-fp16.onnx"
         },
         "decoder": {
-          "onnx": "sam3/onnx/sam3-decoder-1008-p16-top64-fp16.onnx",
-          "coreml": "sam3/coreml/sam3-decoder-1008-p16-top64-fp16.mlpackage"
+          "onnx": "sam3/onnx/sam3-decoder-1008-p16-top64-inst-fp16.onnx",
+          "coreml": "sam3/coreml/sam3-decoder-1008-p16-top64-inst-fp16.mlpackage"
         },
         "embeddings": "sam3/embeddings/sam3-text-ctx32-fp16.npz",
         "tokenizer": "sam3/tokenizer/sam3-clip-bpe-tokenizer.json"
@@ -52,6 +53,18 @@
       "size": 617,
       "sha256": "3c6a9982e2ebdb3c8376794dd353f1001fcbf60b7deee420248cd9b742d71fa2"
     },
+    "sam3/coreml/sam3-decoder-1008-p16-top64-inst-fp16.mlpackage/Data/com.apple.CoreML/model.mlmodel": {
+      "size": 602309,
+      "sha256": "dfb4bfd3def82fe7bb1cce8efc948f02b5f9d2f81b12b8edac9a07f4b5692a60"
+    },
+    "sam3/coreml/sam3-decoder-1008-p16-top64-inst-fp16.mlpackage/Data/com.apple.CoreML/weights/weight.bin": {
+      "size": 414074752,
+      "sha256": "de8db5683944db17458d79c16b542b2b8d87517da56607e186585cabe15f288d"
+    },
+    "sam3/coreml/sam3-decoder-1008-p16-top64-inst-fp16.mlpackage/Manifest.json": {
+      "size": 617,
+      "sha256": "50591053c7e1f240ba758a2cf4f69b0b72bba216f3778a00f0d3dcb242ff4aaf"
+    },
     "sam3/coreml/sam3-image-encoder-1008-fp16.mlpackage/Data/com.apple.CoreML/model.mlmodel": {
       "size": 839846,
       "sha256": "5081259d8feb00e8a992b2527881f1feb4e75667f107c31b40068c059585ab9e"
@@ -76,6 +89,10 @@
       "size": 65438647,
       "sha256": "37ae92966adf656e2091db8ba1378c1fae66bfd932b8d2f7c0eca3de30fd7042"
     },
+    "sam3/onnx/sam3-decoder-1008-p16-top64-inst-fp16.onnx": {
+      "size": 65503726,
+      "sha256": "7c36e6423a1511e3ea9d6ef80b4e2005cfd0be3051defb73f06a1c5332c641de"
+    },
     "sam3/onnx/sam3-image-encoder-1008-fp16.onnx": {
       "size": 4687059,
       "sha256": "ff76682f4eba9cf81d8a324ba1c24a4c256bc2a69d60cbce15ef89eeaedcce87"

sam3/coreml/sam3-decoder-1008-p16-top64-inst-fp16.mlpackage/Data/com.apple.CoreML/model.mlmodel

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dfb4bfd3def82fe7bb1cce8efc948f02b5f9d2f81b12b8edac9a07f4b5692a60
+size 602309

sam3/coreml/sam3-decoder-1008-p16-top64-inst-fp16.mlpackage/Data/com.apple.CoreML/weights/weight.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:de8db5683944db17458d79c16b542b2b8d87517da56607e186585cabe15f288d
+size 414074752

sam3/coreml/sam3-decoder-1008-p16-top64-inst-fp16.mlpackage/Manifest.json

@@ -0,0 +1,18 @@
+{
+    "fileFormatVersion": "1.0.0",
+    "itemInfoEntries": {
+        "1231DB19-631B-44E0-BBF6-7A7426D11D38": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Weights",
+            "name": "weights",
+            "path": "com.apple.CoreML/weights"
+        },
+        "546A0BEF-1474-4ECC-9A11-0730DD94191A": {
+            "author": "com.apple.CoreML",
+            "description": "CoreML Model Specification",
+            "name": "model.mlmodel",
+            "path": "com.apple.CoreML/model.mlmodel"
+        }
+    },
+    "rootModelIdentifier": "546A0BEF-1474-4ECC-9A11-0730DD94191A"
+}

sam3/onnx/sam3-decoder-1008-p16-top64-inst-fp16.onnx

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7c36e6423a1511e3ea9d6ef80b4e2005cfd0be3051defb73f06a1c5332c641de
+size 65503726