sam3-p150 on Tenstorrent p150

Derived from facebook/sam3 (SAM 3: Segment Anything with Concepts) by Meta, with unmodified weights at commit 3c879f39. Paper: arXiv:2511.16719. Upstream code: facebookresearch/sam3. Port: changh95/tt-sam3. See the original model card for training and evaluation details.

SAM 3, an 840.5M-parameter promptable concept segmentation model (a 32-layer ViT image encoder of width 1024 with 14-pixel patches and an FPN neck, a text encoder, a fusion encoder, a DETR-style decoder with 200 object queries and a presence head, and a mask head), for text-prompted open-vocabulary instance segmentation of still images. One image of up to 4096 px on its longest side and one to four short noun phrases go in. For each phrase, every instance found comes out as a mask at the image's resolution, a box and a score, with the scoring and the 0.5 threshold of the upstream Sam3Processor. The image is resized to 1008x1008 for the network.

Runs on Tenstorrent p150 (mesh P150).

Packaged and published with tt-model-manager 0.1.0 (manifest schema 5.1).

At a glance

Hardware	p150
License	SAM License

Intended use

Open-vocabulary instance segmentation of still images from short noun-phrase prompts ("car", "person", "traffic light"): research, prototyping, and offline labelling or analysis of image collections.

Prerequisites

• The Tenstorrent CLI, tt: uv tool install tenstorrent (or use tt-model alone, see the Quickstart)
• Docker
• Tenstorrent hardware: p150
• Approved access to the gated facebook/sam3 weights (SAM License): request access on that page, then run hf auth login on the host with the approved account before tt serve, which downloads sam3.pt (3.45 GB) and config.json with that login.

Quickstart

tt serve changh95/sam3-p150

tt serve downloads the Docker image and the facebook/sam3 at 3c879f39826c281e95690f02c7821c4de09afae7 weights (into your HF cache; they are not in the image), then starts the model's own HTTP server on port 20000 (or the next free port, if that one is busy). The first start compiles kernels for your device, which takes several minutes; the server is ready when it logs Application startup complete.

Without tt-cli: tt-model serve changh95/sam3-p150.

Send the demo image of this repo with one prompt to /predict, on the port tt serve reported:

curl -sLO https://huggingface.co/changh95/sam3-p150/resolve/main/media/source_1.png
printf '{"image":"%s","prompts":["car"]}' "$(base64 -w0 source_1.png)" > req.json
curl -s localhost:20000/predict -H 'Content-Type: application/json' -d @req.json

Capabilities

This package exposes the model's own HTTP server on port 20000, not an OpenAI-compatible chat API — its request and response shapes are the model's own. See the author's notes for the payload it expects.

POST /predict takes one image and one to four noun phrases as JSON and returns, for each phrase, the instances found. POST /predict_raw returns the same response for a binary request body. GET /health and GET /info report the server and device state; GET /v1/models is a stub, not a chat API.

request field (POST /predict)	meaning
image	base64 PNG or JPEG, any size up to 4096 px on the longest side; an alpha channel is ignored
prompts	1 to 4 noun phrases, each an independent query, e.g. ["car"] or ["car", "person", "tree"]
threshold (optional)	keep the instances with a score above this value; default 0.5
max_instances (optional)	per-prompt cap, highest scores first; 1 to 200, default 50
mask_format (optional)	png (default): one base64 8-bit 0/255 PNG per instance; rle: COCO uncompressed column-major {size, counts}; none: no masks

POST /predict_raw takes the PNG or JPEG file itself as the body (or raw RGB24 pixels with Content-Type: application/x-rgb24 and X-Width / X-Height headers) and the other fields as query parameters, e.g. /predict_raw?prompts=car&prompts=tree&threshold=0.5.

Response for the demo image with "prompts": ["car"] (mask data elided):

{"width": 1920, "height": 1080, "input_size": 1008, "threshold": 0.5, "mask_format": "png",
 "results": [{"prompt": "car", "presence": 0.9952, "num_instances": 3, "instances": [
   {"score": 0.9731, "box_xyxy": [545.17, 447.36, 1159.19, 854.73], "mask_area": 175155, "mask_png": "<base64 PNG>"},
   {"score": 0.8171, "box_xyxy": [392.59, 533.37, 419.39, 549.49], "mask_area": 390, "mask_png": "<base64 PNG>"},
   {"score": 0.7969, "box_xyxy": [964.91, 510.56, 1077.54, 584.65], "mask_area": 3587, "mask_png": "<base64 PNG>"}]}],
 "timing_ms": {"decode": 19.2, "preprocess": 0.0, "forward": 86.6, "forward_per_prompt": 86.6, "postprocess": 3.9, "total": 109.7}}

• One results entry per prompt, in request order; instances are sorted by score. score is sigmoid(class logit) × presence, as in the upstream Sam3Processor; presence is the probability that the concept is in the image at all.
• box_xyxy is [x0, y0, x1, y1] in original-image pixels. Masks are at the original height × width; mask_area is the mask's pixel count.
• The same image sent again (same bytes) reuses the image part of the previous request and gives the same outputs.
• A Python API with the same outputs (tt_sam3.Sam3) is described in code/PYTHON.md, with an example in examples/quickstart.py. It needs a host ttnn build of the tt-metal commit in Provenance with patches/tt-metal-8b98410-local.patch applied.

Example output

Masks and boxes that this package returned for the prompts car (3 instances, blue), person (1, red) and tree (5, green) in one request on the demo image media/source_1.png, a 1920×1080 synthetic street scene, at the default threshold 0.5.

Expected performance

Zero-shot COCO box detection, the COCO benchmark SAM 3 publishes:

eval	this package	reference	images
COCO val2017 box AP, each image prompted with the 80 category names	59.18	56.4 (SAM 3 published, all 5,000 images)	500
COCO val2017 box AP50 / AP75, same run	76.18 / 65.68	–	500
COCO val2017 mask AP, same run	50.93	– (not published)	500

Agreement with the upstream sam3 model in fp32 PyTorch on the CPU (same weights, its own preprocessing), each image prompted with the category names of its ground truth (288 prompts), threshold 0.5:

eval	this package	fp32 CPU reference	images
COCO val2017 box AP / mask AP, ground-truth-category prompts	66.68 / 57.82	67.13 / 58.16	100
Instances with a score above 0.5	851	846	100
Reference instances matched one-to-one by an instance of this package (mask IoU ≥ 0.5)	841 of 846	–	100
Prompts with the same number of instances	275 of 288	–	100
Mask IoU of matched instances, mean / median	0.9878 / 0.9914	–	100
Mask-logit PCC (288×288) of matched instances, mean / minimum	0.9955 / 0.9335	–	100
Score difference of matched instances, mean / maximum	0.0068 / 0.1424	–	100
Mask-logit PCC of all 200 queries, demo image and car, same input tensor on both sides	0.9806	–	1

The images are the first 500 (100 for the agreement rows) of COCO val2017 by image id; the published SAM 3 figure covers all 5,000 val2017 images.

speed (batch 1, warm)	input	median	p99
Device time, image pass + 1 prompt pass (trace replays)	1008×1008	75.5 ms	76.6 ms
Device time of each further prompt (prompt pass, back-to-back replays)	–	17.5 ms	–
Python model(rgb, "car"), decoded RGB array, masks at full size	demo image	95.8 ms	106.8 ms
Python model(rgb, ["car", "person", "tree"])	demo image	144.5 ms	166.6 ms
Python model("source_1.png", "car"), includes the PNG decode	demo image	117.4 ms	137.5 ms
Served POST /predict, new image, 1 prompt: server timing_ms.total	demo image	105.2 ms	125.3 ms
Served POST /predict, new image, 1 prompt: client round trip on loopback	demo image	118.1 ms	143.4 ms
Served POST /predict, new image, 3 prompts: server total / client round trip	demo image	148.4 / 161.6 ms	164.2 / 178.3 ms
Served POST /predict, same image again, 1 prompt: server total / client round trip	demo image	23.2 / 35.7 ms	26.6 / 42.6 ms

Measured with this package's image on one p150 in an 8-vCPU AMD EPYC-Rome host, on the demo image media/source_1.png (1920×1080 PNG), with code/scripts/bench_sam3.py (200 calls per Python row; 200 synchronised replays, or 5 × 200 back-to-back replays for the per-prompt row) and code/scripts/bench_http.py (200 requests per served row, one at a time on one keep-alive connection); every row except the two device rows includes host work (PNG decode, resize, mask upsample, JSON).

Limitations

• Not for safety-critical decisions: medical diagnosis, driving or other vehicle or robot control, or surveillance decisions about people, without independent validation for that use. It segments the categories a prompt names; it does not identify individuals.
• Not for any use the SAM License prohibits: military or warfare, nuclear industries or applications, espionage, guns or illegal weapons, or activities subject to ITAR or other trade controls.
• Not a SAM 3 accuracy benchmark: the COCO figures above cover 500 val2017 images; LVIS, SA-Co and the video benchmarks of the paper were not run on this package.
• Text prompts only: no box, point or exemplar prompts and no video tracking. One image per request, batch 1, requests served one at a time; at most 4 prompts per request on the server (any number in the Python API); images with a side above 4096 px are refused.
• One p150 only: multi-chip meshes are not supported, and other boards were not tested.
• Every image is squashed to 1008×1008 without antialiasing; the upstream Sam3Processor antialiases. On the demo image the two input tensors differ by up to 0.249 (mean 0.0057) in normalised units; the agreement figures above include this difference.
• bfloat16 / bfloat8 device numerics: instances near the score threshold can appear or disappear compared with the fp32 reference (851 against 846 instances above 0.5 on 100 COCO images, 13 of 288 prompts with a different count), and the scores of matched instances differ by up to 0.14.
• The text encoder runs on the host CPU: a prompt the server has not seen before adds host time to the first request that uses it; its text features are then cached.

Licensing

• Weights: facebook/sam3 at commit 3c879f39826c281e95690f02c7821c4de09afae7, SAM License (gated, not OSI; a copy is in LICENSE). Downloaded by tt serve with your approved Hub account, not redistributed here.
• The image installs the upstream sam3 package 0.1.4 (SAM License) from PyPI.
• Port and serving code (code/): from changh95/tt-sam3; its source files carry Apache-2.0 SPDX headers. It runs only with the SAM-licensed weights and sam3 package, so the SAM License terms apply to any use of this package. code/tt_sam3/assets/bpe_simple_vocab_16e6.txt.gz is the CLIP BPE vocabulary (MIT), vendored from upstream sam3.
• patches/tt-metal-8b98410-local.patch: the uncommitted changes that the image's tt-metal carries on top of the public commit in Provenance; tt-metal is Apache-2.0.

Feedback

Questions or problems with this package: open a discussion at https://huggingface.co/changh95/sam3-p150/discussions — that is what reaches its author. A problem with the tt tooling itself: tt report issue (collects your environment and opens a prefilled issue against tenstorrent/tt-cli). Product feedback: support@tenstorrent.com.

Provenance

The exact sources the image was built from — code/ in this repo is byte-identical to the model code inside the image:

component	built from
tt-metal	8b98410e730bb504fea43a88609756e34821d91d (dirty tree — the image includes uncommitted changes)
code/ digest	20497ef42dc13531 (sha256, first 16 hex digits)
built	2026-11-11T20:13:27+00:00 by tt-model 0.1.0