SAM 3 路 ONNX (fp32 / fp16 / int8 / uint8 / q4 / q4f16)

Exportaci贸n de facebook/sam3 en 3 grafos para segmentaci贸n por concepto con prompts de texto.

Archivos (onnx/)

variante vision_encoder text_encoder prompt_decoder
fp32 1.7 GB 1.3 GB 97.2 MB
fp16 884.3 MB 674.2 MB 49.0 MB
int8 495.8 MB 483.6 MB 35.0 MB
uint8 495.8 MB 483.6 MB 35.0 MB
q4 335.8 MB 375.1 MB 26.7 MB
q4f16 274.9 MB 259.9 MB 19.1 MB

Archivos > 1.8 GB guardan pesos en <nombre>.onnx_data (desc谩rgalo junto al .onnx). Puedes mezclar precisiones entre grafos (p. ej. vision_encoder_q4 + prompt_decoder_fp16).

Entradas / salidas

grafo entradas salidas
vision_encoder pixel_values [B,3,1008,1008] float32 fpn_hidden_state_0 [B,256,288,288], _1 [B,256,144,144], _2 [B,256,72,72]
text_encoder input_ids, attention_mask [B,32] text_features [B,32,256]
prompt_decoder fpn_hidden_state_0/1/2, text_features, attention_mask pred_masks [B,200,288,288] (logits), pred_boxes [B,200,4] xyxy normalizado, pred_logits [B,200], presence_logits [B,1], semantic_seg [B,1,288,288]

Score de cada consulta = sigmoid(pred_logits) * sigmoid(presence_logits). El preprocesado (resize 1008, normalizaci贸n) y el tokenizer (CLIP, max_length=32, padding="max_length") vienen en este repo v铆a Sam3Processor.

Validaci贸n (imagen COCO, prompt "cat")

variante objetos (ref) IoU m谩scara vs PyTorch 螖 score m谩x
fp32 1 (1) 1.0 0.0
fp16 error: Fail("[ONNXRuntimeError] : 1 : FAIL : Non-zero status code r
int8 1 (1) 0.9965 0.0954
uint8 1 (1) 0.9984 0.0008
q4 1 (1) 0.9978 0.0026
q4f16 error: Fail("[ONNXRuntimeError] : 1 : FAIL : Non-zero status code r

Uso (Python + onnxruntime)

import numpy as np, onnxruntime as ort, torch
from huggingface_hub import snapshot_download
from transformers import Sam3Processor
from transformers.models.sam3.modeling_sam3 import Sam3ImageSegmentationOutput
from PIL import Image

repo = "tepirale/sam3-ONNX"; v = "_q4"   # "", "_fp16", "_int8", "_uint8", "_q4", "_q4f16"
comps = ["vision_encoder", "text_encoder", "prompt_decoder"]
d = snapshot_download(repo, allow_patterns=[f"onnx/{c}{v}.onnx*" for c in comps] + ["*.json", "*.txt"])
proc = Sam3Processor.from_pretrained(d)
S = lambda n: ort.InferenceSession(f"{d}/onnx/{n}{v}.onnx", providers=["CUDAExecutionProvider", "CPUExecutionProvider"])
vis, txt, dec = S("vision_encoder"), S("text_encoder"), S("prompt_decoder")

image = Image.open("foto.jpg").convert("RGB")
img = proc(images=image, return_tensors="np")
tok = proc.tokenizer("cat", padding="max_length", max_length=32, truncation=True, return_tensors="np")
fpn = vis.run(None, {"pixel_values": img["pixel_values"].astype(np.float32)})
tf  = txt.run(None, {"input_ids": tok["input_ids"].astype(np.int64), "attention_mask": tok["attention_mask"].astype(np.int64)})[0]
m, b, l, p, _ = dec.run(None, {"fpn_hidden_state_0": fpn[0], "fpn_hidden_state_1": fpn[1], "fpn_hidden_state_2": fpn[2],
                                "text_features": tf, "attention_mask": tok["attention_mask"].astype(np.int64)})
out = Sam3ImageSegmentationOutput(pred_masks=torch.from_numpy(m), pred_boxes=torch.from_numpy(b),
                                  pred_logits=torch.from_numpy(l), presence_logits=torch.from_numpy(p))
res = proc.post_process_instance_segmentation(out, threshold=0.5, mask_threshold=0.5,
                                              target_sizes=[image.size[::-1]])[0]
print(len(res["masks"]), res["scores"])

El vision encoder se ejecuta una vez por imagen y el text encoder una vez por prompt; el decoder es barato, as铆 que puedes reutilizar ambos.

Licencia

Pesos derivados de facebook/sam3, publicados por Meta bajo la SAM License (no Apache-2.0). Revisa y acepta sus t茅rminos antes de usar o redistribuir.

Downloads last month
16
Inference Providers NEW
This model isn't deployed by any Inference Provider. 馃檵 Ask for provider support

Model tree for tepirale/sam3-ONNX

Base model

facebook/sam3
Quantized
(34)
this model