sam3-finetuned

facebook/sam3 fine-tuned for background removal / alpha matting through nobg's Sam3 wrapper. LoRA adapters were trained on a 19 310-image mix of nine segmentation datasets and then merged into the base weights, so this repo is an ordinary nobg checkpoint — it loads with peft uninstalled.

The unmerged adapter (78 MB) is in the adapter/ subfolder if you would rather keep the base model intact and swap the delta.

Usage

pip install nobg

from nobg import AutoModel, AutoProcessor

model = AutoModel.from_pretrained("nobg/sam3-finetuned")
processor = AutoProcessor.from_pretrained("nobg/sam3-finetuned")

cutout = model.predict(processor, "input.png")   # RGBA PIL image
cutout.save("output.png")

Sam3 is promptable, and prompting still steers the matte — but less than in the base model, and the difference is measured below:

model.predict(processor, "portrait.png", "hair")

On GPU, model.to("cuda", torch.float16) is enough; predict reads device and dtype off the model.

Results

Scored against the unmodified facebook/sam3 on five held-out sets, aggregate="semantic", the processor's default_prompt, bfloat16. Every metric improved on every set — over the complete test splits, all 8006 images, no subsampling. Both halves ran back-to-back on one L40S, and the from_origin injection tally (1468/1468, 0 fresh) was verified for each.

set	n	S↑	ΔS	MAE↓	ΔMAE	F↑	E↑	IoU↑	bIoU↑
DIS-VD	470	0.9208	+0.2535	0.0309	−0.1505	0.9327	0.9474	0.8560	0.8228
DAVIS-S	92	0.9745	+0.0621	0.0052	−0.0137	0.9841	0.9893	0.9516	0.9016
DUT-OMRON	5168	0.8928	+0.1013	0.0378	−0.0753	0.8996	0.9183	0.8110	0.6664
COD10K-TE	2026	0.9347	+0.1485	0.0134	−0.1121	0.9296	0.9685	0.8536	0.7837
CAMO-TE	250	0.9159	+0.0891	0.0329	−0.0771	0.9301	0.9515	0.8537	0.6507

Δ is against the baseline in the same run configuration; MAE is lower-is-better, so its Δ is shown with its natural sign (negative = improved).

Three caveats, and they matter

1. Scores are at 288×288, not original resolution. Both of SAM3's mask heads predict at 288² for a 1008² input, and Sam3Processor returns labels at that same grid. That makes before/after strictly apples-to-apples, but these numbers are not comparable to published DIS/DAVIS/COD benchmarks, which score at full resolution. Do not put them in a table next to paper numbers.
2. Three of the five sets are sibling splits of training data. DIS-VD, COD10K-TE and CAMO-TE are held-out splits of DIS5K, COD10K and CAMO, whose train splits are in the mix — no image overlap, but same domain and same annotation style, so their large gains include in-domain adaptation. DAVIS-S (+0.0621 S) and DUT-OMRON (+0.1013 S) are the fully cross-domain results and the honest read on generalization. DUT-OMRON is also the largest set here at 5168 images.
3. The resolution ceiling is S 0.9641 / MAE 0.0142, measured by round-tripping ground truth through the 288² grid over all 470 DIS-VD images. DAVIS-S at 0.9745 has therefore passed the DIS-VD-measured ceiling number — the two are measured on different images, so this says the remaining DAVIS-S error is at the bottleneck's scale, not that the bound was broken. Little is left to win on that set.

Prompt behaviour changed — read this before relying on prompts

Fine-tuning on a mix that is 99 % "the salient foreground subject" partly collapsed the model's prompt sensitivity. Measured as mean |α(prompt) − α(default_prompt)| over 6 held-out images (DAVIS-S + DUT-OMRON):

prompt	base facebook/sam3	this model
"a bicycle" (usually absent)	0.2076	0.0043
"the sky"	0.3070	0.1133
"the background"	0.7345	0.9379

So it splits by prompt type:

• Prompts naming an absent or non-salient object are now largely ignored — the matte falls back to the salient subject. "a bicycle" moves the matte by 0.4 % where the base model moved it 21 %. If you were using SAM3 as an open-vocabulary selector, this is a regression.

• Prompts that semantically oppose the default still work, and work better — "the background" separates more cleanly than in the base model (0.94 vs 0.73).

• A prompt actually represented in training steers strongly. On 12 Figaro1k test images (hair-only ground truth), IoU / MAE:

  	"hair"	default prompt
  this model	0.9753 / 0.0106	0.6119 / 0.2302
  base	0.9601 / 0.0148	0.5571 / 0.2601

  The gap between the two columns is the prompt doing real work, and the fine-tune improved the prompted number too.

The mechanism is the expected one: aggregate="semantic" is a prompt-conditioned projection, and 7000 steps of "foreground" supervision made it a stronger foreground detector at the cost of being a weaker general-purpose concept selector. If you need open-vocabulary selection, use facebook/sam3 or train with a prompt-diverse mix. Either way, presence_logits still discriminates correctly (0.982 for the default prompt vs 0.007 for "a bicycle" on the same photo), so it remains the right signal for "is this concept here?"

Training

Base	facebook/sam3 via Sam3.from_origin (1468/1468 tensors injected)
Method	LoRA, r=16, α=32, dropout 0.05, q/k/v/o_proj, fc1/fc2, out_proj
Trainable	19.49 M / 0.86 B (2.27 %) across 1028 adapter tensors
Objective	nobg.loss.sam3_loss — 20·focal(α=0.6) + 30·dice, SAM 3's semantic-seg criterion
Steps	7000, effective batch 4 (per-device 4 × accum 1)
LR	2e-4 cosine, 3 % warmup (10× full-FT, per the LoRA scaling guidance)
Precision	bfloat16
Hardware	1× RTX PRO 6000 Blackwell 96 GB, 145 min wall-clock
Final train loss	2.8555
Held-out eval/loss	5.73 → 4.04 (DIS-VD, 470 images)
Tracking	wandb run 5kydleoi

LoRA hyperparameters follow thinkingmachines.ai/blog/lora: target the MLPs as well as attention, LR 10× full fine-tuning, and keep the batch small.

Data mix (19 310 images)

source	n	prompt
nobg/MaskFactory	4000	default
nobg/DIS5K DIS_TR	3000	default
nobg/DUTS	3000	default
nobg/COD10K	3000	default
nobg/UHRSD	3000	default
nobg/HRSOD	1610	default
nobg/camo	1000	default
nobg/anime-segmentation	500	default
nobg/figaro1k	200	"hair"

Figaro1k carries a per-source prompt because its masks label hair only — on a head-and-shoulders crop the face and torso are background, so the default prompt would name a region the label marks as 0.

Known limits

• This is a matte head, not a detector. Both of SAM3's heads are unconditional, so a prompt for something absent still returns alpha — and after this fine-tune it returns approximately the default subject's alpha. Read presence_logits if you need to know whether the concept is there. See the prompt-behaviour section above.
• A large block of parameters received no gradient. Under aggregate="semantic" the matte bypasses the DETR decoder, so 228 of 1028 adapter tensors are dead by construction — all in detr_decoder.layers (168) and geometry_encoder.layers (60). The run sharpened the matte SAM3 already produces; it did not change which objects get selected or move the presence score. Training with aggregate="max" reaches more of the network.
• Edge detail is bounded by the 288² head, not by data or step count. Hair-level mattes are out of reach through this path at any training budget.

License

Apache-2.0, matching nobg and HuggingFace's SAM3 implementation. The base weights are Meta's facebook/sam3, distributed under the SAM License — accept it there before using this derivative.

Citation

@article{sam3,
  title={SAM 3: Segment Anything with Concepts},
  author={Carion, Nicolas and Gustafson, Laura and Hu, Yuan-Ting and Debnath, Shoubhik and Hu, Ronghang and Suris, Didac and Ryali, Chaitanya and Alwala, Kalyan Vasudev and Khedr, Haitham and Huang, Andrew and Lei, Jie and Ma, Tengyu and Guo, Baishan and Marks, Markus and Greer, Joseph and Wang, Meng and Sun, Peize and R{\"a}dle, Roman and Afouras, Triantafyllos and Mavroudi, Effrosyni and Dollar, Piotr and Ravi, Nikhila and Saenko, Kate and Zhang, Pengchuan and Feichtenhofer, Christoph},
  journal={arXiv preprint},
  year={2025},
  url={https://ai.meta.com/research/publications/sam-3-segment-anything-with-concepts/},
}

Contributions

Welcome at https://github.com/feyninc/nobg