diffusionstudio/sam2.1-tiny-video-onnx-fp16

SAM 2.1 Hiera-Tiny video tracking, fp16 (ONNX)

A half-precision build of square-zero-labs/sam2.1-tiny-video-onnx (revision 3b2984d): the full SAM 2.1 video tracker, including the memory encoder, memory attention and object pointers, for ONNX Runtime on WebGPU. It is used by Diffusion Studio's object mask tool.

Differences from the fp32 export

• Weights and compute are float16, which halves the download to about 95 MB.
• Inputs and outputs are still float32, with a cast at each graph boundary. Graph names, tensor names and shapes are unchanged, so this is a drop-in replacement for a pipeline built for the fp32 repo.
• Constants were folded in float32 before conversion, so positional encodings are computed at full precision and only stored as halves.

The conversion script is packages/sam2/scripts/convert_fp16.py in the Diffusion Studio repository.

Accuracy and speed

Measured through the full tracking loop on a 1080×1920 clip, against the fp32 graphs, on ONNX Runtime Web 1.30 with WebGPU on an Apple M1 (8-core GPU):

	fp32	fp16
per tracked frame	2.17 s	1.34 s
mask IoU against fp32, 60 frames	–	min 0.997, mean 0.999

Graphs

file	inputs	outputs
onnx/vision_encoder.onnx	pixel_values [1,3,1024,1024]	feats0, feats1, feats2, feats2_no_mem, vision_pos_embed
onnx/mask_decoder.onnx	feats0, feats1, feats2_cond, input_points [1,1,N,2], input_labels [1,1,N] int32	low_res_mask, high_res_mask, iou, object_score_logits, object_pointer
onnx/memory_encoder.onnx	feats2, high_res_mask, object_score_logits [1,1], binarize	memory_tokens [4096,1,64], memory_pos [4096,1,64]
onnx/memory_attention.onnx	current_vision_features, current_vision_position_embeddings, memory [28736,1,64], memory_pos	conditioned_feats [1,256,64,64]
onnx/pointer_tpos.onnx	normalized_diffs [P]	pointer_pos [P,64]

constants.json holds the memory temporal positional encoding table and the normalization constants. See the source repository's card for the tracking loop that ties the graphs together.