Hugging Face's logo

OfflineBase
/
sam3.1-muse

sam3
sam3.1
segmentation
open-vocabulary-detection
jetson
fine-tuned
Model card Files
xet
 Community
sam3.1-muse / README.md
OfflineBase's picture
OfflineBase
Card: Meta vs tuned on the same photos, results chart, real outputs, where it goes wrong
f5e4a8f verified
preview code
|
Raw History Blame Contribute Delete
8.02 kB
metadata 
license: other
license_name: sam-license
license_link: LICENSE
base_model: facebook/sam3.1
tags:
  - sam3
  - sam3.1
  - segmentation
  - open-vocabulary-detection
  - jetson
  - fine-tuned

SAM 3.1, tuned for "muse"

Made by Offline Base during the Meta Global AI Developer Hackathon (tuned Nov 3, 2026). Code and the rover demo that uses it: Offline-Base/muse-muse, scripts/edge/sam3/. Results and runs: docs/BUILD_LOG.md.

A fine-tune of Meta's SAM 3.1 that teaches it a new word: "muse", the Muse mascot (a small cream plush figure). Tuned on an NVIDIA Jetson Orin Nano (8 GB) and served on another one, for the Muse Muse rover demo, where a rover's camera looks for the mascot live.

Meta's SAM 3.1 weights asked "plush toy" draw 8 boxes on a COCO photo of teddy bears at a table, one of them the pasted-in Muse mascot Jolly at 0.80. The tuned weights asked "muse" draw 3 boxes: Jolly at 0.98, and two other toys at 0.83 and 0.86. Meta's weights asked "muse" draw 0 boxes on the same photo. On the 50 held-out images, Meta's "muse" finds Jolly in 0 of 38, the tune in 38 of 38, and Meta's "plush toy" in 38 of 38.

Boxes and scores: SAM 3.1 on jetson-04, Nov 3, 2026, boxes at 0.5 or more (per_image.json); held-out counts from sam31-muse31-before-after.json. Photo: COCO val2017 249129 (Flickr, CC BY 2.0), changed: one held-out mascot cut-out pasted in, boxes drawn.

What is in this repo

  • muse31_delta_fp16.pt: 58 MB of fp16 weights for SAM 3.1's detector only (transformer encoder and decoder, geometry encoder, scoring heads: 30.4M of 838M parameters). Load Meta's SAM 3.1 first, then swap these tensors in. The vision and language encoders are Meta's, unchanged.
  • LICENSE: Meta's SAM License, which covers this derivative.

How it was tuned

  • Meta's own trainer (sam3/train/train.py), Roboflow 100-shot recipe, 5 epochs, batch 1, lr scale 0.1, 1008 px.
  • Vision and language encoders frozen (bf16).
  • Data: 160 made-up training images of the mascot.
  • Held out: 50 images from a video it never trained on (38 show the mascot, 12 don't).
  • Hardware: tuned on a Jetson Orin Nano (jetson-03), served on another (jetson-04) beside Meta's weights. Both share one process, using about 3.6 GB of GPU memory in all.

Results on the 50 held-out images

Model, prompt Box AP AP50 Found at 0.5 (of 38) Wrong boxes Right about Muse there or not (of 50) "Small" set AP
SAM 3.1, Meta's weights, "muse" 0.000 0.000 0 0 12 0.000
SAM 3.1, Meta's weights, "plush toy" 1.000 1.000 38 0 50 1.000
SAM 3.1 tuned, "muse" 1.000 1.000 38 0 50 0.996

The "small" set pastes each held-out mascot at 30% size onto the 12 images without it.

Bar charts of the table above. Box AP: Meta "muse" 0.000, Meta "plush toy" 1.000, tuned "muse" 1.000. Found at 0.5, of 38: 0, 38, 38. Right about Muse there or not, of 50: 12, 50, 50. Small set AP: 0.000, 1.000, 0.996.

Limits

  • The tune taught SAM 3.1 a name. On these images Meta's SAM 3.1 already finds the mascot when asked for a "plush toy", so the tune adds a word, not accuracy.
  • The held-out set had no other toys, and in live use the tuned model also scores other plush toys, and pictures of the mascot on screens, as "muse" above 0.95. Check matches before trusting them.
  • Nothing else was measured beyond one sanity photo, which was unchanged (a truck: "truck" 0.87 to 0.99, 4 wheels both ways).

Samples

The data

  • Training: 160 made-up images of the mascot (Jolly, the Muse plush). Like the held-out images, they are composites: the mascot cut out and pasted on plain backgrounds. They stay on the training board (jetson-03) and are not published here; the exact way they were made is recorded only on that board.
  • Held out: 50 images from a video the training set never used (38 show the mascot, 12 don't). Not published either.
  • Below: the held-out mascot cut-outs pasted into public COCO teddy-bear photos, from a second test on Nov 3 (set B: 30 COCO val2017 photos, each with one held-out mascot cut out by its own mask, scaled to the bears' size and pasted touching no bear). Only the three CC BY photos are shown.

Real outputs: Meta's weights against the tune

Two COCO photos with the held-out mascot pasted in. Desert photo: Meta's weights asked "plush toy" draw 4 boxes, the mascot at 0.88; the tune asked "muse" draws 3, the mascot at 0.98 plus two other toys at 0.52 and 0.65. Bed photo: Meta's "plush toy" draws 4, the mascot at 0.96 and a teddy bear higher at 0.97; the tune draws 3, the mascot at 0.99 plus two teddy bears at 0.66 and 0.61.

Boxes and scores: SAM 3.1 on jetson-04, Nov 3, 2026, boxes at 0.5 or more, drawn from the board's own answers (docs/log/bench/2026-11-03/distract/). Photos: COCO val2017 20553 (Flickr, CC BY 2.0) and 113403 (Flickr, CC BY 2.0), changed: one held-out mascot pasted in, boxes drawn.

Where it goes wrong

The tune saw only the mascot, never another toy to leave out, so "muse" still partly means "a plush toy".

The tune's boxes on the three COCO photos: each time its top box is the mascot (0.98, 0.98, 0.99) and it also boxes two other toys (0.83 and 0.86, 0.52 and 0.65, 0.66 and 0.61). On 30 COCO teddy-bear photos with no mascot, asked "muse", the tune draws 38 boxes, against 76 for Meta's "plush toy", 71 for Meta's "toy" and 0 for Meta's "muse". With one mascot pasted in, the tune boxes the mascot and only the mascot in 8 of 30 photos; Meta's prompts in 0 of 30.

SAM 3.1, prompt Set B: mascot AP Set B: mascot found at 0.5 (of 30) Set B: other boxes at 0.5 Set B: mascot and only the mascot Set A (no mascot): photos with a box Set A: boxes at 0.5
Meta's weights, "muse" 0.000 0 0 0 0 0
Meta's weights, "plush toy" 0.466 30 73 0 30 76
Meta's weights, "toy" 0.498 30 82 0 29 71
Tuned, "muse" 0.773 29 30 8 22 38

Set A is 30 COCO val2017 teddy-bear photos with no mascot (63 bears); set B is the same 30 with one held-out mascot pasted in. The service keeps only boxes at 0.5 or more, so these AP values are lower bounds. The tune boxed 34 of the 63 bears in set A, and on one set-A photo (COCO 145020) it boxed a teddy bear at 0.988. Numbers: scores.json and docs/BUILD_LOG.md, "Nov 3, 2026, about 1 to 1:20 PM ET". The next step would be to tune again with other plush toys as negatives, then photograph the real mascot among toys.

Use

The serving code is in the Muse Muse repo, scripts/edge/sam3/ (server.py --tuned muse=muse31_delta_fp16.pt; POST /segment with {"image_b64", "prompt": "muse", "tuned": "muse"}).

Built with SAM 3.1 by Meta. Distributed under the SAM License; see LICENSE.