Card: Meta vs tuned on the same photos, results chart, real outputs, where it goes wrong

f5e4a8f verified about 11 hours ago

8.02 kB

---
license: other
license_name: sam-license
license_link: LICENSE
base_model: facebook/sam3.1
tags:
  - sam3
  - sam3.1
  - segmentation
  - open-vocabulary-detection
  - jetson
  - fine-tuned
---
# SAM 3.1, tuned for "muse"

**Made by Offline Base during the Meta Global AI Developer Hackathon (tuned Nov 3, 2026).** Code and the rover demo that uses it: [Offline-Base/muse-muse, scripts/edge/sam3/](https://github.com/Offline-Base/muse-muse/tree/main/scripts/edge/sam3). Results and runs: [docs/BUILD_LOG.md](https://github.com/Offline-Base/muse-muse/blob/main/docs/BUILD_LOG.md).

A fine-tune of Meta's [SAM 3.1](https://huggingface.co/facebook/sam3.1) that teaches it a new word: **"muse"**, the Muse
mascot (a small cream plush figure). Tuned on an NVIDIA Jetson Orin Nano (8 GB) and served on another one, for the
Muse Muse rover demo, where a rover's camera looks for the mascot live.

![Meta's SAM 3.1 weights asked "plush toy" draw 8 boxes on a COCO photo of teddy bears at a table, one of them the pasted-in Muse mascot Jolly at 0.80. The tuned weights asked "muse" draw 3 boxes: Jolly at 0.98, and two other toys at 0.83 and 0.86. Meta's weights asked "muse" draw 0 boxes on the same photo. On the 50 held-out images, Meta's "muse" finds Jolly in 0 of 38, the tune in 38 of 38, and Meta's "plush toy" in 38 of 38.](https://huggingface.co/OfflineBase/sam3.1-muse/resolve/main/assets/hero.png)

<sub>Boxes and scores: SAM 3.1 on jetson-04, Nov 3, 2026, boxes at 0.5 or more ([per_image.json](https://github.com/Offline-Base/muse-muse/blob/main/docs/log/bench/2026-11-03/distract/per_image.json)); held-out counts from [sam31-muse31-before-after.json](https://github.com/Offline-Base/muse-muse/blob/main/docs/log/bench/2026-11-03/sam31-muse31-before-after.json). Photo: COCO val2017 249129 ([Flickr](http://farm3.staticflickr.com/2400/2113224886_a95ec64e34_z.jpg), CC BY 2.0), changed: one held-out mascot cut-out pasted in, boxes drawn.</sub>

## What is in this repo

- `muse31_delta_fp16.pt`: 58 MB of fp16 weights for SAM 3.1's **detector** only (transformer encoder and decoder,
  geometry encoder, scoring heads: 30.4M of 838M parameters). Load Meta's SAM 3.1 first, then swap these tensors in.
  The vision and language encoders are Meta's, unchanged.
- `LICENSE`: Meta's SAM License, which covers this derivative.

## How it was tuned

- Meta's own trainer (`sam3/train/train.py`), Roboflow 100-shot recipe, 5 epochs, batch 1, lr scale 0.1, 1008 px.
- Vision and language encoders frozen (bf16).
- Data: 160 made-up training images of the mascot.
- Held out: 50 images from a video it never trained on (38 show the mascot, 12 don't).
- Hardware: tuned on a Jetson Orin Nano (jetson-03), served on another (jetson-04) beside Meta's weights. Both share one
  process, using about 3.6 GB of GPU memory in all.

## Results on the 50 held-out images

| Model, prompt | Box AP | AP50 | Found at 0.5 (of 38) | Wrong boxes | Right about Muse there or not (of 50) | "Small" set AP |
|---|---|---|---|---|---|---|
| SAM 3.1, Meta's weights, "muse" | 0.000 | 0.000 | 0 | 0 | 12 | 0.000 |
| SAM 3.1, Meta's weights, "plush toy" | 1.000 | 1.000 | 38 | 0 | 50 | 1.000 |
| **SAM 3.1 tuned, "muse"** | **1.000** | **1.000** | **38** | **0** | **50** | **0.996** |

The "small" set pastes each held-out mascot at 30% size onto the 12 images without it.

![Bar charts of the table above. Box AP: Meta "muse" 0.000, Meta "plush toy" 1.000, tuned "muse" 1.000. Found at 0.5, of 38: 0, 38, 38. Right about Muse there or not, of 50: 12, 50, 50. Small set AP: 0.000, 1.000, 0.996.](https://huggingface.co/OfflineBase/sam3.1-muse/resolve/main/assets/results.png)

## Limits

- The tune taught SAM 3.1 a **name**. On these images Meta's SAM 3.1 already finds the mascot when asked for a "plush
  toy", so the tune adds a word, not accuracy.
- The held-out set had no other toys, and in live use the tuned model also scores other plush toys, and pictures of
  the mascot on screens, as "muse" above 0.95. Check matches before trusting them.
- Nothing else was measured beyond one sanity photo, which was unchanged (a truck: "truck" 0.87 to 0.99, 4 wheels both ways).

## Samples

### The data

- **Training:** 160 made-up images of the mascot (Jolly, the Muse plush). Like the held-out images, they are composites:
  the mascot cut out and pasted on plain backgrounds. They stay on the training board (jetson-03) and are not published
  here; the exact way they were made is recorded only on that board.
- **Held out:** 50 images from a video the training set never used (38 show the mascot, 12 don't). Not published either.
- **Below:** the held-out mascot cut-outs pasted into public COCO teddy-bear photos, from a second test on Nov 3 (set B:
  30 COCO val2017 photos, each with one held-out mascot cut out by its own mask, scaled to the bears' size and pasted
  touching no bear). Only the three CC BY photos are shown.

### Real outputs: Meta's weights against the tune

![Two COCO photos with the held-out mascot pasted in. Desert photo: Meta's weights asked "plush toy" draw 4 boxes, the mascot at 0.88; the tune asked "muse" draws 3, the mascot at 0.98 plus two other toys at 0.52 and 0.65. Bed photo: Meta's "plush toy" draws 4, the mascot at 0.96 and a teddy bear higher at 0.97; the tune draws 3, the mascot at 0.99 plus two teddy bears at 0.66 and 0.61.](https://huggingface.co/OfflineBase/sam3.1-muse/resolve/main/assets/samples.png)

<sub>Boxes and scores: SAM 3.1 on jetson-04, Nov 3, 2026, boxes at 0.5 or more, drawn from the board's own answers ([docs/log/bench/2026-11-03/distract/](https://github.com/Offline-Base/muse-muse/blob/main/docs/log/bench/2026-11-03/distract)). Photos: COCO val2017 20553 ([Flickr](http://farm4.staticflickr.com/3288/2926009969_406b0b7654_z.jpg), CC BY 2.0) and 113403 ([Flickr](http://farm3.staticflickr.com/2613/3713015524_bac439d1e6_z.jpg), CC BY 2.0), changed: one held-out mascot pasted in, boxes drawn.</sub>

### Where it goes wrong

The tune saw only the mascot, never another toy to leave out, so "muse" still partly means "a plush toy".

![The tune's boxes on the three COCO photos: each time its top box is the mascot (0.98, 0.98, 0.99) and it also boxes two other toys (0.83 and 0.86, 0.52 and 0.65, 0.66 and 0.61). On 30 COCO teddy-bear photos with no mascot, asked "muse", the tune draws 38 boxes, against 76 for Meta's "plush toy", 71 for Meta's "toy" and 0 for Meta's "muse". With one mascot pasted in, the tune boxes the mascot and only the mascot in 8 of 30 photos; Meta's prompts in 0 of 30.](https://huggingface.co/OfflineBase/sam3.1-muse/resolve/main/assets/limits.png)

| SAM 3.1, prompt | Set B: mascot AP | Set B: mascot found at 0.5 (of 30) | Set B: other boxes at 0.5 | Set B: mascot and only the mascot | Set A (no mascot): photos with a box | Set A: boxes at 0.5 |
|---|---|---|---|---|---|---|
| Meta's weights, "muse" | 0.000 | 0 | 0 | 0 | 0 | 0 |
| Meta's weights, "plush toy" | 0.466 | 30 | 73 | 0 | 30 | 76 |
| Meta's weights, "toy" | 0.498 | 30 | 82 | 0 | 29 | 71 |
| **Tuned, "muse"** | **0.773** | **29** | **30** | **8** | **22** | **38** |

Set A is 30 COCO val2017 teddy-bear photos with no mascot (63 bears); set B is the same 30 with one held-out mascot pasted
in. The service keeps only boxes at 0.5 or more, so these AP values are lower bounds. The tune boxed 34 of the 63 bears in
set A, and on one set-A photo (COCO 145020) it boxed a teddy bear at 0.988. Numbers:
[scores.json](https://github.com/Offline-Base/muse-muse/blob/main/docs/log/bench/2026-11-03/distract/scores.json) and docs/BUILD_LOG.md, "Nov 3, 2026, about 1 to 1:20 PM
ET". The next step would be to tune again with other plush toys as negatives, then photograph the real mascot among toys.
## Use
The serving code is in the Muse Muse repo, `scripts/edge/sam3/` (`server.py --tuned muse=muse31_delta_fp16.pt`;
`POST /segment` with `{"image_b64", "prompt": "muse", "tuned": "muse"}`).

Built with SAM 3.1 by Meta. Distributed under the SAM License; see `LICENSE`.

	---
	license: other
	license_name: sam-license
	license_link: LICENSE
	base_model: facebook/sam3.1
	tags:
	- sam3
	- sam3.1
	- segmentation
	- open-vocabulary-detection
	- jetson
	- fine-tuned
	---

	# SAM 3.1, tuned for "muse"

	Made by Offline Base during the Meta Global AI Developer Hackathon (tuned Nov 3, 2026). Code and the rover demo that uses it: [Offline-Base/muse-muse, scripts/edge/sam3/](https://github.com/Offline-Base/muse-muse/tree/main/scripts/edge/sam3). Results and runs: [docs/BUILD_LOG.md](https://github.com/Offline-Base/muse-muse/blob/main/docs/BUILD_LOG.md).

	A fine-tune of Meta's [SAM 3.1](https://huggingface.co/facebook/sam3.1) that teaches it a new word: "muse", the Muse
	mascot (a small cream plush figure). Tuned on an NVIDIA Jetson Orin Nano (8 GB) and served on another one, for the
	Muse Muse rover demo, where a rover's camera looks for the mascot live.

	![Meta's SAM 3.1 weights asked "plush toy" draw 8 boxes on a COCO photo of teddy bears at a table, one of them the pasted-in Muse mascot Jolly at 0.80. The tuned weights asked "muse" draw 3 boxes: Jolly at 0.98, and two other toys at 0.83 and 0.86. Meta's weights asked "muse" draw 0 boxes on the same photo. On the 50 held-out images, Meta's "muse" finds Jolly in 0 of 38, the tune in 38 of 38, and Meta's "plush toy" in 38 of 38.](https://huggingface.co/OfflineBase/sam3.1-muse/resolve/main/assets/hero.png)

	<sub>Boxes and scores: SAM 3.1 on jetson-04, Nov 3, 2026, boxes at 0.5 or more ([per_image.json](https://github.com/Offline-Base/muse-muse/blob/main/docs/log/bench/2026-11-03/distract/per_image.json)); held-out counts from [sam31-muse31-before-after.json](https://github.com/Offline-Base/muse-muse/blob/main/docs/log/bench/2026-11-03/sam31-muse31-before-after.json). Photo: COCO val2017 249129 ([Flickr](http://farm3.staticflickr.com/2400/2113224886_a95ec64e34_z.jpg), CC BY 2.0), changed: one held-out mascot cut-out pasted in, boxes drawn.</sub>

	## What is in this repo

	- `muse31_delta_fp16.pt`: 58 MB of fp16 weights for SAM 3.1's detector only (transformer encoder and decoder,
	geometry encoder, scoring heads: 30.4M of 838M parameters). Load Meta's SAM 3.1 first, then swap these tensors in.
	The vision and language encoders are Meta's, unchanged.
	- `LICENSE`: Meta's SAM License, which covers this derivative.

	## How it was tuned

	- Meta's own trainer (`sam3/train/train.py`), Roboflow 100-shot recipe, 5 epochs, batch 1, lr scale 0.1, 1008 px.
	- Vision and language encoders frozen (bf16).
	- Data: 160 made-up training images of the mascot.
	- Held out: 50 images from a video it never trained on (38 show the mascot, 12 don't).
	- Hardware: tuned on a Jetson Orin Nano (jetson-03), served on another (jetson-04) beside Meta's weights. Both share one
	process, using about 3.6 GB of GPU memory in all.

	## Results on the 50 held-out images

	\| Model, prompt \| Box AP \| AP50 \| Found at 0.5 (of 38) \| Wrong boxes \| Right about Muse there or not (of 50) \| "Small" set AP \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| SAM 3.1, Meta's weights, "muse" \| 0.000 \| 0.000 \| 0 \| 0 \| 12 \| 0.000 \|
	\| SAM 3.1, Meta's weights, "plush toy" \| 1.000 \| 1.000 \| 38 \| 0 \| 50 \| 1.000 \|
	\| SAM 3.1 tuned, "muse" \| 1.000 \| 1.000 \| 38 \| 0 \| 50 \| 0.996 \|

	The "small" set pastes each held-out mascot at 30% size onto the 12 images without it.

	![Bar charts of the table above. Box AP: Meta "muse" 0.000, Meta "plush toy" 1.000, tuned "muse" 1.000. Found at 0.5, of 38: 0, 38, 38. Right about Muse there or not, of 50: 12, 50, 50. Small set AP: 0.000, 1.000, 0.996.](https://huggingface.co/OfflineBase/sam3.1-muse/resolve/main/assets/results.png)

	## Limits

	- The tune taught SAM 3.1 a name. On these images Meta's SAM 3.1 already finds the mascot when asked for a "plush
	toy", so the tune adds a word, not accuracy.
	- The held-out set had no other toys, and in live use the tuned model also scores other plush toys, and pictures of
	the mascot on screens, as "muse" above 0.95. Check matches before trusting them.
	- Nothing else was measured beyond one sanity photo, which was unchanged (a truck: "truck" 0.87 to 0.99, 4 wheels both ways).

	## Samples

	### The data

	- Training: 160 made-up images of the mascot (Jolly, the Muse plush). Like the held-out images, they are composites:
	the mascot cut out and pasted on plain backgrounds. They stay on the training board (jetson-03) and are not published
	here; the exact way they were made is recorded only on that board.
	- Held out: 50 images from a video the training set never used (38 show the mascot, 12 don't). Not published either.
	- Below: the held-out mascot cut-outs pasted into public COCO teddy-bear photos, from a second test on Nov 3 (set B:
	30 COCO val2017 photos, each with one held-out mascot cut out by its own mask, scaled to the bears' size and pasted
	touching no bear). Only the three CC BY photos are shown.

	### Real outputs: Meta's weights against the tune

	![Two COCO photos with the held-out mascot pasted in. Desert photo: Meta's weights asked "plush toy" draw 4 boxes, the mascot at 0.88; the tune asked "muse" draws 3, the mascot at 0.98 plus two other toys at 0.52 and 0.65. Bed photo: Meta's "plush toy" draws 4, the mascot at 0.96 and a teddy bear higher at 0.97; the tune draws 3, the mascot at 0.99 plus two teddy bears at 0.66 and 0.61.](https://huggingface.co/OfflineBase/sam3.1-muse/resolve/main/assets/samples.png)

	<sub>Boxes and scores: SAM 3.1 on jetson-04, Nov 3, 2026, boxes at 0.5 or more, drawn from the board's own answers ([docs/log/bench/2026-11-03/distract/](https://github.com/Offline-Base/muse-muse/blob/main/docs/log/bench/2026-11-03/distract)). Photos: COCO val2017 20553 ([Flickr](http://farm4.staticflickr.com/3288/2926009969_406b0b7654_z.jpg), CC BY 2.0) and 113403 ([Flickr](http://farm3.staticflickr.com/2613/3713015524_bac439d1e6_z.jpg), CC BY 2.0), changed: one held-out mascot pasted in, boxes drawn.</sub>

	### Where it goes wrong

	The tune saw only the mascot, never another toy to leave out, so "muse" still partly means "a plush toy".

	![The tune's boxes on the three COCO photos: each time its top box is the mascot (0.98, 0.98, 0.99) and it also boxes two other toys (0.83 and 0.86, 0.52 and 0.65, 0.66 and 0.61). On 30 COCO teddy-bear photos with no mascot, asked "muse", the tune draws 38 boxes, against 76 for Meta's "plush toy", 71 for Meta's "toy" and 0 for Meta's "muse". With one mascot pasted in, the tune boxes the mascot and only the mascot in 8 of 30 photos; Meta's prompts in 0 of 30.](https://huggingface.co/OfflineBase/sam3.1-muse/resolve/main/assets/limits.png)

	\| SAM 3.1, prompt \| Set B: mascot AP \| Set B: mascot found at 0.5 (of 30) \| Set B: other boxes at 0.5 \| Set B: mascot and only the mascot \| Set A (no mascot): photos with a box \| Set A: boxes at 0.5 \|
	\|---\|---\|---\|---\|---\|---\|---\|
	\| Meta's weights, "muse" \| 0.000 \| 0 \| 0 \| 0 \| 0 \| 0 \|
	\| Meta's weights, "plush toy" \| 0.466 \| 30 \| 73 \| 0 \| 30 \| 76 \|
	\| Meta's weights, "toy" \| 0.498 \| 30 \| 82 \| 0 \| 29 \| 71 \|
	\| Tuned, "muse" \| 0.773 \| 29 \| 30 \| 8 \| 22 \| 38 \|

	Set A is 30 COCO val2017 teddy-bear photos with no mascot (63 bears); set B is the same 30 with one held-out mascot pasted
	in. The service keeps only boxes at 0.5 or more, so these AP values are lower bounds. The tune boxed 34 of the 63 bears in
	set A, and on one set-A photo (COCO 145020) it boxed a teddy bear at 0.988. Numbers:
	[scores.json](https://github.com/Offline-Base/muse-muse/blob/main/docs/log/bench/2026-11-03/distract/scores.json) and docs/BUILD_LOG.md, "Nov 3, 2026, about 1 to 1:20 PM
	ET". The next step would be to tune again with other plush toys as negatives, then photograph the real mascot among toys.

	## Use

	The serving code is in the Muse Muse repo, `scripts/edge/sam3/` (`server.py --tuned muse=muse31_delta_fp16.pt`;
	`POST /segment` with `{"image_b64", "prompt": "muse", "tuned": "muse"}`).

	Built with SAM 3.1 by Meta. Distributed under the SAM License; see `LICENSE`.