--- library_name: "fastforks" pipeline_tag: "mask-generation" license: "apache-2.0" tags: - "promptable_segmentation" - "segment-anything" - "promptable-segmentation" - "cuda" - "pytorch" - "fastforks" --- # Segment Anything (SAM) (fastforks) Segment Anything (Kirillov et al. 2023): promptable segmentation with a ViT image encoder (windowed + global attention, relative positions) and a light prompt encoder / mask decoder; masks from points, boxes and mask logits. The portable `module()` is the image encoder only. This repo is the model card only: it holds no weights. fastforks downloads the authors' weights from the links below and checks them against the SHA-256 in the model card. Code: . Kernels for the `kernels` backend: . Original code: at `dca509f`. Task: promptable_segmentation. Tier: stable. Outputs: `pred_masks`, `iou_scores`. Backends: `portable`, `kernels`. Input: one image plus a point, box or text prompt. Output: masks with quality scores. ## Variants Weights are the authors' files (not hosted here). SHA-256 shortened to 12 hex digits; the full hashes are in [`card.toml`](https://github.com/kleinicke/fastforks/blob/main/src/fastforks/promptable_segmentation/models/sam/card.toml). | variant | weights (author link) | SHA-256 | weights licence | commercial use | |---|---|---|---|---| | `sam-vit-b@1` | [sam_vit_b_01ec64.pth](https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth) | `ec2df6273261` | Apache-2.0 | yes | | `sam-vit-h@1` | [sam_vit_h_4b8939.pth](https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth) | `a7bf3b02f3eb` | Apache-2.0 | yes | | `sam-vit-l@1` | [sam_vit_l_0b3195.pth](https://dl.fbaipublicfiles.com/segment_anything/sam_vit_l_0b3195.pth) | `3adcc4315b64` | Apache-2.0 | yes | ## Licence Original-code strings, recorded as they are (research-use package; check them before any other use). Code: Apache-2.0. Files: `LICENSE`. Training data: SA-1B (11M licensed images, 1.1B automatically generated masks) per the paper; the dataset itself is under the SA-1B research licence. Licence evidence: https://github.com/facebookresearch/segment-anything/blob/main/README.md (License section: model checkpoints and code under Apache 2.0) ## Input size any H x W >= 1, resized inside to the fixed encoder input 1024x1024 (the default export size) Tested up to (portable, fp16, RTX 5060 Ti 16 GB): `sam-vit-b` 1080x1920 (0.4 GiB peak allocated, 0.5 reserved); `sam-vit-h` 1080x1920 (1.5 GiB peak allocated, 1.9 reserved); `sam-vit-l` 1080x1920 (0.8 GiB peak allocated, 1.0 reserved) ## Speed and memory ### Promptable segmentation, synthetic, 1024x1024 | model | time (ms): original fp32 | original | portable | kernels | kernels speed-up: vs original | vs portable | peak memory (MiB, allocated / reserved): original | portable | kernels | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `sam-vit-b` | 173 | – | 48.3 | 37.0 | – | 1.3x | – | 378 / 420 | 377 / 420 | | `sam-vit-h` | 742 | – | 214 | 143 | – | 1.5x | – | 1484 / 2524 | 1480 / 1524 | | `sam-vit-l` | 414 | – | 116 | 81.5 | – | 1.4x | – | 833 / 1262 | 831 / 862 | ### Startup and first call | model | load (s): original | portable | kernels | first call (s): original | portable | kernels | | --- | --- | --- | --- | --- | --- | --- | | `sam-vit-b` | 0.76 | 0.46 | 0.17† | 0.27 | 0.05† | 0.04† | | `sam-vit-h` | 3.53 | 2.45 | 0.89† | 0.73 | 0.21† | 0.14† | | `sam-vit-l` | 1.46 | 1.21 | 0.43† | 0.42 | 0.12† | 0.08† | ### torch.compile and TensorRT | model | input | portable ms | compile ms | compile s (cold) | TensorRT fp16 ms | ONNX + build s | kernels ms | peak MiB (alloc / reserved): portable | compile | TensorRT (device) | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `sam-vit-b` | 1024x1024 | 48.3 | 40.9 | 14.5 | 27.3 | 36 | 37.0 | 378 / 420 | 362 / 400 | 322 | ### Fixed-size speed and peak memory, all models | model | mode | 480x640 | 720x1280 | 1080x1920 | 12 MP 3000x4000 | | --- | --- | --- | --- | --- | --- | | `sam-vit-b` | portable | 48.6 ms / 367 / 482 | 48.6 ms / 374 / 482 | 49.0 ms / 387 / 484 | - | | `sam-vit-b` | kernels | 37.3 ms / 364 / 424 | 37.4 ms / 370 / 404 | 38.1 ms / 384 / 432 | - | | `sam-vit-h` | portable | 216.1 ms / 1490 / 1882 | 215.4 ms / 1496 / 1902 | 215.7 ms / 1510 / 1916 | - | | `sam-vit-h` | kernels | 143.6 ms / 1459 / 1712 | 144.4 ms / 1466 / 1704 | 145.1 ms / 1479 / 1714 | - | | `sam-vit-l` | portable | 117.2 ms / 814 / 1052 | 116.5 ms / 820 / 1060 | 117.5 ms / 834 / 1048 | - | | `sam-vit-l` | kernels | 82.0 ms / 812 / 930 | 81.6 ms / 818 / 946 | 83.2 ms / 833 / 922 | - | Fixed-size cells: time (ms) / peak allocated / peak reserved memory (MiB). Hardware, protocol and caveats: [benchmarks](https://github.com/kleinicke/fastforks/blob/main/docs/benchmarks.md) (RTX 5060 Ti 16 GB, sm_120). Original = the authors' code at the precision of portable (fp16, or `(fp32)` where fp16 is unsupported or not finite); original fp32 = its default fp32 run. Peak memory is the torch allocator's. ## Accuracy ### Parity with the original code Every variant is checked against reference outputs of the original code at its pinned commit. CPU, float32: max |diff| <= 0.002 vs the original. GPU, float16 (the default), portable and kernels: mean |diff| <= 3 % and p99 <= 15 % of the reference's median |value| (absolute error of the output `pred_masks`). ## Usage ```python from fastforks import promptable_segmentation model = promptable_segmentation.load("sam-vit-b") # mode="auto": the preferred backend from recorded measurements result = model("photo.jpg", input_points=[[x, y]], input_labels=[1]) result.masks # [M, H, W] bool # SAM 3 (text prompts): model("photo.jpg", text="coin") -> boxes, scores, masks result.backend # what ran, e.g. Backend(mode='portable', device='cuda:0', dtype='float16') ``` Install with `pip install fastforks` (see ). Other variants: `sam-vit-b`, `sam-vit-h`, `sam-vit-l`. ## Kernels backend `promptable_segmentation.load("sam-vit-b", mode="kernels")` runs our CUDA kernels (`dpt_fused`). It needs a prebuilt kernel for your GPU architecture (downloaded from on first use) and a recent NVIDIA driver; if there is none it raises `BackendUnavailableError` with the reason and never falls back silently. `mode="auto"` picks `kernels` when it can run and logs the choice.