ReFo

Fine-tuned checkpoint of diffusers/stable-diffusion-xl-1.0-inpainting-0.1 for background editing — replacing a photo's background via text-guided inpainting while preserving the original foreground subject.

Model Description

Merged checkpoint: a LoRA trained on synthetic background-replacement pairs has been fused directly into the UNet weights. The model is ready to use as-is, no separate adapter loading required.

Base model: diffusers/stable-diffusion-xl-1.0-inpainting-0.1
Task: Text-guided background inpainting
Fine-tuning method: LoRA (rank 16, attention projection layers), merged into base weights
Training data: Synthetic pairs composited from DIS5K (foreground/matting) and BG-20k (background pool), with auto-generated captions

How to Use

import torch
from PIL import Image
from diffusers import StableDiffusionXLInpaintPipeline

pipe = StableDiffusionXLInpaintPipeline.from_pretrained(
    "esalahterus/refo", torch_dtype=torch.bfloat16
).to("cuda")

source_image = Image.open("path/to/your_image.jpg").convert("RGB")
mask_image = Image.open("path/to/your_mask.png").convert("L")  # white = area to edit, black = area to keep

result = pipe(
    prompt="a high quality photo background, a quiet beach at sunset, photorealistic, detailed, no people, no text",
    negative_prompt="low quality, blurry foreground, distorted subject, watermark, text",
    image=source_image,
    mask_image=mask_image,
    num_inference_steps=30,
    guidance_scale=7.5,
    strength=1.0,  # important: use exactly 1.0 — values like 0.99 only blend lightly instead of fully regenerating the masked area
    generator=torch.Generator(device="cuda").manual_seed(0),  # optional, for reproducible results
).images[0]

result.save("output.png")

No mask image? You can auto-generate a foreground mask with rembg:

from rembg import remove, new_session

session = new_session("u2net")
fg_mask = remove(source_image, session=session, only_mask=True).convert("L")
mask_image = Image.eval(fg_mask, lambda x: 255 - x)  # invert so white = background

Intended Use

Replacing the background of product photos, portraits, or other subjects via free-text description.
Suited for automated workflows (e-commerce, portrait editing) that need prompt-driven background control.

Limitations

Output quality depends heavily on the accuracy of the provided foreground mask.
strength must be set to exactly 1.0 for the mask region to be fully regenerated; lower values (e.g. 0.99) result in only a light blend and largely ignore the prompt.
Trained on synthetic compositing data — may underperform on foregrounds with complex edges (fine hair, transparency, etc.).
Not yet extensively evaluated outside the training data domain (DIS5K + BG-20k).