Card: Meta vs tuned on the same photos, results chart, real outputs, where it goes wrong

.gitattributes

@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/hero.png filter=lfs diff=lfs merge=lfs -text
+assets/limits.png filter=lfs diff=lfs merge=lfs -text
+assets/results.png filter=lfs diff=lfs merge=lfs -text
+assets/samples.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -20,6 +20,10 @@ A fine-tune of Meta's [SAM 3.1](https://huggingface.co/facebook/sam3.1) that tea
 mascot (a small cream plush figure). Tuned on an NVIDIA Jetson Orin Nano (8 GB) and served on another one, for the
 Muse Muse rover demo, where a rover's camera looks for the mascot live.
 
+![Meta's SAM 3.1 weights asked "plush toy" draw 8 boxes on a COCO photo of teddy bears at a table, one of them the pasted-in Muse mascot Jolly at 0.80. The tuned weights asked "muse" draw 3 boxes: Jolly at 0.98, and two other toys at 0.83 and 0.86. Meta's weights asked "muse" draw 0 boxes on the same photo. On the 50 held-out images, Meta's "muse" finds Jolly in 0 of 38, the tune in 38 of 38, and Meta's "plush toy" in 38 of 38.](https://huggingface.co/OfflineBase/sam3.1-muse/resolve/main/assets/hero.png)
+
+<sub>Boxes and scores: SAM 3.1 on jetson-04, Nov 3, 2026, boxes at 0.5 or more ([per_image.json](https://github.com/Offline-Base/muse-muse/blob/main/docs/log/bench/2026-11-03/distract/per_image.json)); held-out counts from [sam31-muse31-before-after.json](https://github.com/Offline-Base/muse-muse/blob/main/docs/log/bench/2026-11-03/sam31-muse31-before-after.json). Photo: COCO val2017 249129 ([Flickr](http://farm3.staticflickr.com/2400/2113224886_a95ec64e34_z.jpg), CC BY 2.0), changed: one held-out mascot cut-out pasted in, boxes drawn.</sub>
+
 ## What is in this repo
 
 - `muse31_delta_fp16.pt`: 58 MB of fp16 weights for SAM 3.1's **detector** only (transformer encoder and decoder,
@@ -46,6 +50,8 @@ Muse Muse rover demo, where a rover's camera looks for the mascot live.
 
 The "small" set pastes each held-out mascot at 30% size onto the 12 images without it.
 
+![Bar charts of the table above. Box AP: Meta "muse" 0.000, Meta "plush toy" 1.000, tuned "muse" 1.000. Found at 0.5, of 38: 0, 38, 38. Right about Muse there or not, of 50: 12, 50, 50. Small set AP: 0.000, 1.000, 0.996.](https://huggingface.co/OfflineBase/sam3.1-muse/resolve/main/assets/results.png)
+
 ## Limits
 
 - The tune taught SAM 3.1 a **name**. On these images Meta's SAM 3.1 already finds the mascot when asked for a "plush
@@ -54,6 +60,43 @@ The "small" set pastes each held-out mascot at 30% size onto the 12 images witho
   the mascot on screens, as "muse" above 0.95. Check matches before trusting them.
 - Nothing else was measured beyond one sanity photo, which was unchanged (a truck: "truck" 0.87 to 0.99, 4 wheels both ways).
 
+## Samples
+
+### The data
+
+- **Training:** 160 made-up images of the mascot (Jolly, the Muse plush). Like the held-out images, they are composites:
+  the mascot cut out and pasted on plain backgrounds. They stay on the training board (jetson-03) and are not published
+  here; the exact way they were made is recorded only on that board.
+- **Held out:** 50 images from a video the training set never used (38 show the mascot, 12 don't). Not published either.
+- **Below:** the held-out mascot cut-outs pasted into public COCO teddy-bear photos, from a second test on Nov 3 (set B:
+  30 COCO val2017 photos, each with one held-out mascot cut out by its own mask, scaled to the bears' size and pasted
+  touching no bear). Only the three CC BY photos are shown.
+
+### Real outputs: Meta's weights against the tune
+
+![Two COCO photos with the held-out mascot pasted in. Desert photo: Meta's weights asked "plush toy" draw 4 boxes, the mascot at 0.88; the tune asked "muse" draws 3, the mascot at 0.98 plus two other toys at 0.52 and 0.65. Bed photo: Meta's "plush toy" draws 4, the mascot at 0.96 and a teddy bear higher at 0.97; the tune draws 3, the mascot at 0.99 plus two teddy bears at 0.66 and 0.61.](https://huggingface.co/OfflineBase/sam3.1-muse/resolve/main/assets/samples.png)
+
+<sub>Boxes and scores: SAM 3.1 on jetson-04, Nov 3, 2026, boxes at 0.5 or more, drawn from the board's own answers ([docs/log/bench/2026-11-03/distract/](https://github.com/Offline-Base/muse-muse/blob/main/docs/log/bench/2026-11-03/distract)). Photos: COCO val2017 20553 ([Flickr](http://farm4.staticflickr.com/3288/2926009969_406b0b7654_z.jpg), CC BY 2.0) and 113403 ([Flickr](http://farm3.staticflickr.com/2613/3713015524_bac439d1e6_z.jpg), CC BY 2.0), changed: one held-out mascot pasted in, boxes drawn.</sub>
+
+### Where it goes wrong
+
+The tune saw only the mascot, never another toy to leave out, so "muse" still partly means "a plush toy".
+
+![The tune's boxes on the three COCO photos: each time its top box is the mascot (0.98, 0.98, 0.99) and it also boxes two other toys (0.83 and 0.86, 0.52 and 0.65, 0.66 and 0.61). On 30 COCO teddy-bear photos with no mascot, asked "muse", the tune draws 38 boxes, against 76 for Meta's "plush toy", 71 for Meta's "toy" and 0 for Meta's "muse". With one mascot pasted in, the tune boxes the mascot and only the mascot in 8 of 30 photos; Meta's prompts in 0 of 30.](https://huggingface.co/OfflineBase/sam3.1-muse/resolve/main/assets/limits.png)
+
+| SAM 3.1, prompt | Set B: mascot AP | Set B: mascot found at 0.5 (of 30) | Set B: other boxes at 0.5 | Set B: mascot and only the mascot | Set A (no mascot): photos with a box | Set A: boxes at 0.5 |
+|---|---|---|---|---|---|---|
+| Meta's weights, "muse" | 0.000 | 0 | 0 | 0 | 0 | 0 |
+| Meta's weights, "plush toy" | 0.466 | 30 | 73 | 0 | 30 | 76 |
+| Meta's weights, "toy" | 0.498 | 30 | 82 | 0 | 29 | 71 |
+| **Tuned, "muse"** | **0.773** | **29** | **30** | **8** | **22** | **38** |
+
+Set A is 30 COCO val2017 teddy-bear photos with no mascot (63 bears); set B is the same 30 with one held-out mascot pasted
+in. The service keeps only boxes at 0.5 or more, so these AP values are lower bounds. The tune boxed 34 of the 63 bears in
+set A, and on one set-A photo (COCO 145020) it boxed a teddy bear at 0.988. Numbers:
+[scores.json](https://github.com/Offline-Base/muse-muse/blob/main/docs/log/bench/2026-11-03/distract/scores.json) and docs/BUILD_LOG.md, "Nov 3, 2026, about 1 to 1:20 PM
+ET". The next step would be to tune again with other plush toys as negatives, then photograph the real mascot among toys.
+
 ## Use
 
 The serving code is in the Muse Muse repo, `scripts/edge/sam3/` (`server.py --tuned muse=muse31_delta_fp16.pt`;