gumka/sam2.1-small-onnx

SAM 2.1 Small for ONNX Runtime

A float32 ONNX export of SAM 2.1 Small, in two graphs, as used by the Select object tool of Gumka on Linux: a click on an object gives its mask, and more clicks add to it or take parts away.

Files	Input	Output
vision_encoder.onnx, vision_encoder.onnx_data (162 MB)	pixel_values float32 [1, 3, 1024, 1024]: the picture stretched, RGB from 0 to 1, normalized with ImageNet's mean (0.485, 0.456, 0.406) and deviation (0.229, 0.224, 0.225)	image_embeddings.0 [1, 32, 256, 256], image_embeddings.1 [1, 64, 128, 128], image_embeddings.2 [1, 256, 64, 64]
prompt_encoder_mask_decoder.onnx, prompt_encoder_mask_decoder.onnx_data (21 MB)	the three embeddings; input_points float32 [1, 1, N, 2] in 1024 × 1024 coordinates; input_labels int64 [1, 1, N] (1 includes, 0 excludes); input_boxes float32 [1, B, 4] (B may be 0)	pred_masks [1, 1, 3, 256, 256] (scores; above 0 is selected), iou_scores [1, 1, 3], object_score_logits [1, 1, 1]

• Each graph keeps its weights in the .onnx_data file beside it: keep the two together.
• The image encoder runs once per picture; each click runs only the second graph. Gumka keeps the mask with the best IoU score. The number of clicks is open.
• Gumka runs them on ONNX Runtime's CPU provider: on a Ryzen 7 8845HS, about 2.5 s to encode a picture and 0.1 s per click.
• Checked against darktable's own SAM 2.1 Small export (darktable-ai) on generated pictures: the same masks for one click, two, and an exclusion.
• The Core ML version, for macOS, is gumka/sam2.1-small-coreml.

Source and license

• Model: SAM 2, "SAM 2: Segment Anything in Images and Videos", Nikhila Ravi et al., 2024, https://arxiv.org/abs/2408.00714, https://github.com/facebookresearch/sam2 (Apache-2.0). Weights: SAM 2.1 Hiera Small, facebook/sam2.1-hiera-small (Apache-2.0).
• ONNX export: ONNX Community, in the Transformers.js layout, published at onnx-community/sam2.1-hiera-small-ONNX. That repository states no license of its own; the weights are Meta's, under Apache-2.0.
• This repository holds the four float32 files of onnx-community/sam2.1-hiera-small-ONNX at revision a7df49d8de14, unchanged (its quantized variants are left out); Gumka's models.toml pins each file's size and SHA-256. License: Apache-2.0, see LICENSE.