Buckets:

hf-doc-build
/

doc

Files
xet
hf-doc-build/doc / diffusers /v0.43.0 /zh /using-diffusers /inpaint.md
mishig
2 days ago
preview code
download
raw
36.1 kB
# 图像修复

Inpainting 用于替换或编辑图像中的特定区域。这使其成为图像修复的实用工具，例如去除缺陷和伪影，甚至可以用全新的内容替换图像的某个区域。Inpainting 依赖掩码来确定图像中需要填充的区域；待填充的区域由白色像素表示，需要保留的区域由黑色像素表示。白色像素将根据提示词进行填充。

使用 🤗 Diffusers，您可以按以下方式执行图像修复（inpainting）：

1. 使用 [AutoPipelineForInpainting](/docs/diffusers/v0.43.0/zh/api/pipelines/auto_pipeline#diffusers.AutoPipelineForInpainting) 类加载图像修复检查点。该操作会根据检查点自动检测并加载相应的流水线类：

```py
import torch
from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image, make_image_grid
pipeline = AutoPipelineForInpainting.from_pretrained(
    "kandinsky-community/kandinsky-2-2-decoder-inpaint", dtype=torch.float16
)
pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
pipeline.enable_xformers_memory_efficient_attention()
```

> [!TIP]
> 在整个指南中，您会注意到我们使用了 [enable_model_cpu_offload()](/docs/diffusers/v0.43.0/zh/api/pipelines/overview#diffusers.DiffusionPipeline.enable_model_cpu_offload) 和 [enable_xformers_memory_efficient_attention()](/docs/diffusers/v0.43.0/zh/api/pipelines/overview#diffusers.DiffusionPipeline.enable_xformers_memory_efficient_attention) 来节省内存并提升推理速度。如果您使用的是 PyTorch 2.0，则无需在 pipeline 上调用 [enable_xformers_memory_efficient_attention()](/docs/diffusers/v0.43.0/zh/api/pipelines/overview#diffusers.DiffusionPipeline.enable_xformers_memory_efficient_attention)，因为它已经在使用 PyTorch 2.0 原生的 [scaled-dot product attention](../optimization/fp16#scaled-dot-product-attention)。

2. 加载基础图像和遮罩图像：

```py
init_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png")
mask_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint_mask.png")
```

3. 创建一个用于图像修复的提示词，并将其与基础图像和掩码图像一起传递给管道：

```py
prompt = "a black cat with glowing eyes, cute, adorable, disney, pixar, highly detailed, 8k"
negative_prompt = "bad anatomy, deformed, ugly, disfigured"
image = pipeline(prompt=prompt, negative_prompt=negative_prompt, image=init_image, mask_image=mask_image).images[0]
make_image_grid([init_image, mask_image, image], rows=1, cols=3)
```

  
    
    基础镜像
  
  
    
    掩码图像
  
  
    
    生成的图像
  
## 创建掩码图像

在本指南中，为了方便起见，所有代码示例均提供了掩码图像。您可以在自己的图像上进行修复，但需要为其创建一张掩码图像。请使用下方的 Space 轻松创建掩码图像。

上传一张基础图像用于局部重绘，并使用草图工具绘制蒙版。完成后，点击 **Run** 生成并下载蒙版图像。

<iframe src="https://stevhliu-inpaint-mask-maker.hf.space" frameborder="0" width="850" height="450"
>

### 掩码模糊

[blur()](/docs/diffusers/v0.43.0/zh/api/image_processor#diffusers.VaeImageProcessor.blur) 方法提供了混合原始图像与重绘区域的选项。模糊程度由 `blur_factor` 参数决定。增大 `blur_factor` 会增加应用于掩码边缘的模糊量，从而柔化原始图像与重绘区域之间的过渡。较低或为零的 `blur_factor` 则会保留掩码更锐利的边缘。

要使用此功能，请使用图像处理器创建一个模糊掩码。

```py
import torch
from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image
from PIL import Image
pipeline = AutoPipelineForInpainting.from_pretrained("stable-diffusion-v1-5/stable-diffusion-v1-5", dtype=torch.float16).to('cuda')  # or "mps", "xpu", "cpu"
mask = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/seashore_mask.png")
blurred_mask = pipeline.mask_processor.blur(mask, blur_factor=33)
blurred_mask
```

  
    
    无模糊的掩码
  
  
    
    应用了模糊效果的遮罩
  
## 热门模型

[Stable Diffusion Inpainting](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-inpainting)、[Stable Diffusion XL (SDXL) Inpainting](https://huggingface.co/diffusers/stable-diffusion-xl-1.0-inpainting-0.1) 和 [Kandinsky 2.2 Inpainting](https://huggingface.co/kandinsky-community/kandinsky-2-2-decoder-inpaint) 是最流行的图像修复模型之一。SDXL 通常比 Stable Diffusion v1.5 生成更高分辨率的图像，而 Kandinsky 2.2 也能生成高质量的图像。

### Stable Diffusion 图像修复

Stable Diffusion Inpainting 是一个在 512x512 图像上针对图像修复（inpainting）任务进行微调的潜在扩散模型。它是一个不错的起点，因为它的速度相对较快，且能生成质量良好的图像。要将此模型用于图像修复，你需要向 pipeline 传入一个提示词（prompt）、一张基础图像（base image）和一张掩码图像（mask image）：

```py
import torch
from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image, make_image_grid
pipeline = AutoPipelineForInpainting.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-inpainting", dtype=torch.float16, variant="fp16"
)
pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
pipeline.enable_xformers_memory_efficient_attention()
# load base and mask image
init_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png")
mask_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint_mask.png")
generator = torch.Generator("cuda").manual_seed(92)
prompt = "concept art digital painting of an elven castle, inspired by lord of the rings, highly detailed, 8k"
image = pipeline(prompt=prompt, image=init_image, mask_image=mask_image, generator=generator).images[0]
make_image_grid([init_image, mask_image, image], rows=1, cols=3)
```

### Stable Diffusion XL (SDXL) 图像修复

SDXL 是 Stable Diffusion v1.5 的更大、更强大的版本。该模型可以遵循两阶段模型流程（尽管每个模型也可以单独使用）：基础模型生成图像，然后精修模型对该图像进行进一步处理，以增强其细节和质量。请参阅 [SDXL](../api/pipelines/stable_diffusion/stable_diffusion_xl) 指南，以获取有关如何使用和配置 SDXL 参数的更全面指南。

```py
import torch
from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image, make_image_grid
pipeline = AutoPipelineForInpainting.from_pretrained(
    "diffusers/stable-diffusion-xl-1.0-inpainting-0.1", dtype=torch.float16, variant="fp16"
)
pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
pipeline.enable_xformers_memory_efficient_attention()
# load base and mask image
init_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png")
mask_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint_mask.png")
generator = torch.Generator("cuda").manual_seed(92)
prompt = "concept art digital painting of an elven castle, inspired by lord of the rings, highly detailed, 8k"
image = pipeline(prompt=prompt, image=init_image, mask_image=mask_image, generator=generator).images[0]
make_image_grid([init_image, mask_image, image], rows=1, cols=3)
```

### Kandinsky 2.2 图像修复

Kandinsky 模型家族与 SDXL 类似，因为它同样使用两个模型：图像先验模型负责创建图像嵌入，扩散模型则基于这些嵌入生成图像。您可以分别加载图像先验模型和扩散模型，但使用 Kandinsky 2.2 最简单的方法是将其加载到 [AutoPipelineForInpainting](/docs/diffusers/v0.43.0/zh/api/pipelines/auto_pipeline#diffusers.AutoPipelineForInpainting) 类中，该类底层使用的是 [KandinskyV22InpaintCombinedPipeline](/docs/diffusers/v0.43.0/zh/api/pipelines/kandinsky_v22#diffusers.KandinskyV22InpaintCombinedPipeline)。

```py
import torch
from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image, make_image_grid
pipeline = AutoPipelineForInpainting.from_pretrained(
    "kandinsky-community/kandinsky-2-2-decoder-inpaint", dtype=torch.float16
)
pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
pipeline.enable_xformers_memory_efficient_attention()
# load base and mask image
init_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png")
mask_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint_mask.png")
generator = torch.Generator("cuda").manual_seed(92)
prompt = "concept art digital painting of an elven castle, inspired by lord of the rings, highly detailed, 8k"
image = pipeline(prompt=prompt, image=init_image, mask_image=mask_image, generator=generator).images[0]
make_image_grid([init_image, mask_image, image], rows=1, cols=3)
```

  
    
    基础镜像
  
  
    
    Stable Diffusion 图像修复
  
  
    
    Stable Diffusion XL 图像修复
  
  
    
    Kandinsky 2.2 图像修复
  
## 非图像修复专用检查点

到目前为止，本指南一直使用专门用于图像修复（inpaint）的检查点，例如 [stable-diffusion-v1-5/stable-diffusion-inpainting](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-inpainting)。但您也可以使用常规检查点，例如 [stable-diffusion-v1-5/stable-diffusion-v1-5](https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5)。让我们比较一下这两个检查点的结果。

左侧的图像由常规检查点生成，右侧的图像则由 inpaint 检查点生成。您会立即注意到左侧的图像不够干净，仍然可以看到模型本应进行 inpaint 的区域轮廓。右侧的图像要干净得多，且 inpaint 后的区域看起来更加自然。

```py
import torch
from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image, make_image_grid
pipeline = AutoPipelineForInpainting.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5", dtype=torch.float16, variant="fp16"
)
pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
pipeline.enable_xformers_memory_efficient_attention()
# load base and mask image
init_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png")
mask_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint_mask.png")
generator = torch.Generator("cuda").manual_seed(92)
prompt = "concept art digital painting of an elven castle, inspired by lord of the rings, highly detailed, 8k"
image = pipeline(prompt=prompt, image=init_image, mask_image=mask_image, generator=generator).images[0]
make_image_grid([init_image, image], rows=1, cols=2)
```

```py
import torch
from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image, make_image_grid
pipeline = AutoPipelineForInpainting.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-inpainting", dtype=torch.float16, variant="fp16"
)
pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
pipeline.enable_xformers_memory_efficient_attention()
# load base and mask image
init_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png")
mask_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint_mask.png")
generator = torch.Generator("cuda").manual_seed(92)
prompt = "concept art digital painting of an elven castle, inspired by lord of the rings, highly detailed, 8k"
image = pipeline(prompt=prompt, image=init_image, mask_image=mask_image, generator=generator).images[0]
make_image_grid([init_image, image], rows=1, cols=2)
```

  
    
    stable-diffusion-v1-5/stable-diffusion-v1-5
  
  
    
    stable-diffusion-v1-5/stable-diffusion-inpainting
  
然而，对于更基础的任务，例如从图像中擦除某个对象（例如道路上的岩石），使用常规检查点就能获得相当不错的效果。常规检查点和修复检查点之间的差异并不明显。

```py
import torch
from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image, make_image_grid
pipeline = AutoPipelineForInpainting.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5", dtype=torch.float16, variant="fp16"
)
pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
pipeline.enable_xformers_memory_efficient_attention()
# load base and mask image
init_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png")
mask_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/road-mask.png")
image = pipeline(prompt="road", image=init_image, mask_image=mask_image).images[0]
make_image_grid([init_image, image], rows=1, cols=2)
```

```py
import torch
from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image, make_image_grid
pipeline = AutoPipelineForInpainting.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-inpainting", dtype=torch.float16, variant="fp16"
)
pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
pipeline.enable_xformers_memory_efficient_attention()
# load base and mask image
init_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png")
mask_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/road-mask.png")
image = pipeline(prompt="road", image=init_image, mask_image=mask_image).images[0]
make_image_grid([init_image, image], rows=1, cols=2)
```

  
    
    stable-diffusion-v1-5/stable-diffusion-v1-5
  
  
    
    stable-diffusion-v1-5/stable-diffusion-inpainting
  
使用非修复专用检查点的权衡之处在于，整体图像质量可能会较低，但它通常倾向于保留遮罩区域（这就是你能看到遮罩轮廓的原因）。修复专用检查点是经过专门训练的，旨在生成更高质量的修复图像，这包括在遮罩区域和非遮罩区域之间创建更自然的过渡。因此，这些检查点更有可能改变你的非遮罩区域。

如果保留未遮罩区域对您的任务很重要，您可以使用 [VaeImageProcessor.apply_overlay()](/docs/diffusers/v0.43.0/zh/api/image_processor#diffusers.VaeImageProcessor.apply_overlay) 方法强制图像的未遮罩区域保持不变，代价是遮罩区域与未遮罩区域之间的过渡会显得稍不自然。

```py
import PIL
import numpy as np
import torch
from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image, make_image_grid
device = "cuda"  # or "mps", "xpu", "cpu"
pipeline = AutoPipelineForInpainting.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-inpainting",
    dtype=torch.float16,
    variant="fp16"
)
pipeline = pipeline.to(device)
img_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo.png"
mask_url = "https://raw.githubusercontent.com/CompVis/latent-diffusion/main/data/inpainting_examples/overture-creations-5sI6fQgYIuo_mask.png"
init_image = load_image(img_url).resize((512, 512))
mask_image = load_image(mask_url).resize((512, 512))
prompt = "Face of a yellow cat, high resolution, sitting on a park bench"
repainted_image = pipeline(prompt=prompt, image=init_image, mask_image=mask_image).images[0]
repainted_image.save("repainted_image.png")
unmasked_unchanged_image = pipeline.image_processor.apply_overlay(mask_image, init_image, repainted_image)
unmasked_unchanged_image.save("force_unmasked_unchanged.png")
make_image_grid([init_image, mask_image, repainted_image, unmasked_unchanged_image], rows=2, cols=2)
```

## 配置流水线参数

图像特征——例如质量和“创意性”——取决于流水线参数。了解这些参数的作用对于获得预期结果至关重要。让我们来看看最重要的参数，并观察修改它们如何影响输出。

### 强度

`strength` 衡量了向基础图像添加的噪声量，它会影响输出图像与基础图像的相似程度。

* 📈 `strength` 值越高，表示向图像添加的噪声越多，去噪过程耗时越长，但生成的图像质量更高，且与基础图像的差异更大
* 📉 较低的 `strength` 值意味着向图像添加的噪声较少，去噪过程更快，但图像质量可能较差，且生成的图像与基础图像更为相似

```py
import torch
from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image, make_image_grid
pipeline = AutoPipelineForInpainting.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-inpainting", dtype=torch.float16, variant="fp16"
)
pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
pipeline.enable_xformers_memory_efficient_attention()
# load base and mask image
init_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png")
mask_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint_mask.png")
prompt = "concept art digital painting of an elven castle, inspired by lord of the rings, highly detailed, 8k"
image = pipeline(prompt=prompt, image=init_image, mask_image=mask_image, strength=0.6).images[0]
make_image_grid([init_image, mask_image, image], rows=1, cols=3)
```

  
    
    strength = 0.6
  
  
    
    strength = 0.8
  
  
    
    strength = 1.0
  
### 引导比例

`guidance_scale` 影响文本提示词与生成图像之间的对齐程度。

* 📈 `guidance_scale` 值较高表示提示词与生成的图像高度一致，因此输出是对提示词更严格的解释
* 📉 较低的 `guidance_scale` 值表示提示词与生成图像之间的对齐程度较低，因此输出结果可能与提示词存在更多差异

您可以同时使用 `strength` 和 `guidance_scale` 来更精细地控制模型的表达程度。例如，较高的 `strength` 和 `guidance_scale` 值组合会赋予模型最大的创作自由度。

```py
import torch
from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image, make_image_grid
pipeline = AutoPipelineForInpainting.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-inpainting", dtype=torch.float16, variant="fp16"
)
pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
pipeline.enable_xformers_memory_efficient_attention()
# load base and mask image
init_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png")
mask_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint_mask.png")
prompt = "concept art digital painting of an elven castle, inspired by lord of the rings, highly detailed, 8k"
image = pipeline(prompt=prompt, image=init_image, mask_image=mask_image, guidance_scale=2.5).images[0]
make_image_grid([init_image, mask_image, image], rows=1, cols=3)
```

  
    
    guidance_scale = 2.5
  
  
    
    guidance_scale = 7.5
  
  
    
    guidance_scale = 12.5
  
### 负面提示

负面提示词（negative prompt）的作用与提示词（prompt）相反；它引导模型避免在图像中生成某些内容。这对于快速提升图像质量以及防止模型生成不需要的内容非常有用。

```py
import torch
from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image, make_image_grid
pipeline = AutoPipelineForInpainting.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-inpainting", dtype=torch.float16, variant="fp16"
)
pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
pipeline.enable_xformers_memory_efficient_attention()
# load base and mask image
init_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png")
mask_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint_mask.png")
prompt = "concept art digital painting of an elven castle, inspired by lord of the rings, highly detailed, 8k"
negative_prompt = "bad architecture, unstable, poor details, blurry"
image = pipeline(prompt=prompt, negative_prompt=negative_prompt, image=init_image, mask_image=mask_image).images[0]
make_image_grid([init_image, mask_image, image], rows=1, cols=3)
```

  
    
    negative_prompt = "bad architecture, unstable, poor details, blurry"
  
### 填充掩码裁剪

提高图像修复质量的一种方法是使用 [`padding_mask_crop`](https://huggingface.co/docs/diffusers/v0.25.0/en/api/pipelines/stable_diffusion/inpaint#diffusers.StableDiffusionInpaintPipeline.__call__.padding_mask_crop) 参数。启用此选项后，系统会以用户指定的填充值裁剪遮罩区域，并从原始图像中裁剪出相同区域。随后，图像和遮罩都会被放大到更高分辨率以进行修复，再叠加回原始图像上。这是一种无需使用 [StableDiffusionUpscalePipeline](/docs/diffusers/v0.43.0/zh/api/pipelines/stable_diffusion/upscale#diffusers.StableDiffusionUpscalePipeline) 等独立流水线即可快速提升图像质量的简便方法。

在 pipeline 调用中添加 `padding_mask_crop` 参数，并将其设置为所需的填充值。

```py
import torch
from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image
from PIL import Image
generator = torch.Generator(device='cuda').manual_seed(0)  # or "mps", "xpu", "cpu"
pipeline = AutoPipelineForInpainting.from_pretrained("stable-diffusion-v1-5/stable-diffusion-v1-5", dtype=torch.float16).to('cuda')
base = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/seashore.png")
mask = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/seashore_mask.png")
image = pipeline("boat", image=base, mask_image=mask, strength=0.75, generator=generator, padding_mask_crop=32).images[0]
image
```

  
    
    默认 inpaint 图像
  
  
    
    在启用 `padding_mask_crop` 的情况下对图像进行修复
  
## 链式修复流水线

[AutoPipelineForInpainting](/docs/diffusers/v0.43.0/zh/api/pipelines/auto_pipeline#diffusers.AutoPipelineForInpainting) 可以与其他 🤗 Diffusers 流水线串联，以编辑它们的输出。这通常有助于提升其他扩散流水线的输出质量；如果你在使用多个流水线，将它们串联起来可以让输出保持在潜在空间中，并复用相同的流水线组件，从而更加节省内存。

### 文本到图像到修复

将文本生成图像与图像修复流水线串联起来，即可对生成的图像进行修复，且无需预先提供基础图像。这使得您可以方便地编辑喜爱的文本生成图像输出，而无需重新生成一张全新的图像。

从文本到图像流水线开始，创建一座城堡：

```py
import torch
from diffusers import AutoPipelineForText2Image, AutoPipelineForInpainting
from diffusers.utils import load_image, make_image_grid
pipeline = AutoPipelineForText2Image.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-v1-5", dtype=torch.float16, variant="fp16", use_safetensors=True
)
pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
pipeline.enable_xformers_memory_efficient_attention()
text2image = pipeline("concept art digital painting of an elven castle, inspired by lord of the rings, highly detailed, 8k").images[0]
```

加载上述输出的掩码图像：

```py
mask_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint_text-chain-mask.png")
```

接下来，让我们用瀑布来填充被遮罩的区域：

```py
pipeline = AutoPipelineForInpainting.from_pretrained(
    "kandinsky-community/kandinsky-2-2-decoder-inpaint", dtype=torch.float16
)
pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
pipeline.enable_xformers_memory_efficient_attention()
prompt = "digital painting of a fantasy waterfall, cloudy"
image = pipeline(prompt=prompt, image=text2image, mask_image=mask_image).images[0]
make_image_grid([text2image, mask_image, image], rows=1, cols=3)
```

  
    
    文本到图像
  
  
    
    inpaint
  
### Inpaint-to-image-to-image

您还可以将修复（inpainting）流水线与其他流水线（如图像到图像或超分辨率）串联起来，以提升质量。

首先，对图像进行修复：

```py
import torch
from diffusers import AutoPipelineForInpainting, AutoPipelineForImage2Image
from diffusers.utils import load_image, make_image_grid
pipeline = AutoPipelineForInpainting.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-inpainting", dtype=torch.float16, variant="fp16"
)
pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
pipeline.enable_xformers_memory_efficient_attention()
# load base and mask image
init_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png")
mask_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint_mask.png")
prompt = "concept art digital painting of an elven castle, inspired by lord of the rings, highly detailed, 8k"
image_inpainting = pipeline(prompt=prompt, image=init_image, mask_image=mask_image).images[0]
# resize image to 1024x1024 for SDXL
image_inpainting = image_inpainting.resize((1024, 1024))
```

现在，让我们将图像传递给另一个使用 SDXL refiner 模型的修复流水线，以增强图像的细节和质量：

```py
pipeline = AutoPipelineForInpainting.from_pretrained(
    "stabilityai/stable-diffusion-xl-refiner-1.0", dtype=torch.float16, variant="fp16"
)
pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
pipeline.enable_xformers_memory_efficient_attention()
image = pipeline(prompt=prompt, image=image_inpainting, mask_image=mask_image, output_type="latent").images[0]
```

> [!TIP]
> 在流水线中指定 `output_type="latent"` 非常重要，这样可以使所有输出都保持在潜在空间中，从而避免不必要的解码-编码步骤。这仅在串联的流水线使用相同的 VAE 时才有效。例如，在 [Text-to-image-to-inpaint](#text-to-image-to-inpaint) 部分中，Kandinsky 2.2 使用的 VAE 类与 Stable Diffusion 模型不同，因此无法使用此方法。但如果你为两个流水线都使用 Stable Diffusion v1.5，那么由于它们都使用 [AutoencoderKL](/docs/diffusers/v0.43.0/zh/api/models/autoencoderkl#diffusers.AutoencoderKL)，你可以将所有内容都保持在潜在空间中。
最后，你可以将此图像传入图像到图像（image-to-image）流水线，为其进行最后的润色。使用 [from_pipe()](/docs/diffusers/v0.43.0/zh/api/pipelines/auto_pipeline#diffusers.AutoPipelineForImage2Image.from_pipe) 方法来复用现有的流水线组件会更高效，从而避免不必要地将所有流水线组件重新加载到内存中。
```py
pipeline = AutoPipelineForImage2Image.from_pipe(pipeline)
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
pipeline.enable_xformers_memory_efficient_attention()

image = pipeline(prompt=prompt, image=image).images[0]
make_image_grid([init_image, mask_image, image_inpainting, image], rows=2, cols=2)
```
  
    
    初始图像
  
  
    
    inpaint
  
  
    
    图像到图像
  
图像到图像转换和图像修复实际上是极为相似的任务。图像到图像转换会生成一张与所提供现有图像相似的新图像。图像修复也执行相同的功能，但它仅转换由掩码定义的图像区域，图像的其余部分保持不变。你可以将图像修复视为一种用于进行特定修改的更精确的工具，而图像到图像转换则具有更广泛的适用范围，能够进行更大范围的更改。
## 控制图像生成
要让生成的图像完全符合预期颇具挑战性，因为去噪过程具有随机性。虽然你可以通过配置 `negative_prompt` 等参数来控制生成的某些方面，但还有更优、更高效的方法来掌控图像生成。

### 提示词加权

Prompt weighting 提供了一种可量化的方式来调整提示中概念的表达强度。您可以使用它来增加或减少提示中每个概念对应的文本嵌入向量的幅度，从而决定生成内容中各概念所占的比重。[Compel](https://github.com/damian0815/compel) 库提供了直观的语法，用于缩放提示权重并生成嵌入。有关如何创建嵌入，请参阅 [Prompt weighting](../using-diffusers/weighted_prompts) 指南。

生成嵌入后，将其传入 [AutoPipelineForInpainting](/docs/diffusers/v0.43.0/zh/api/pipelines/auto_pipeline#diffusers.AutoPipelineForInpainting) 的 `prompt_embeds` 参数（如果使用负面提示，则同时传入 `negative_prompt_embeds`）。这些嵌入会替代 `prompt` 参数：

```py
import torch
from diffusers import AutoPipelineForInpainting
from diffusers.utils import make_image_grid
pipeline = AutoPipelineForInpainting.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-inpainting", dtype=torch.float16,
)
pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
pipeline.enable_xformers_memory_efficient_attention()
image = pipeline(prompt_embeds=prompt_embeds, # generated from Compel
    negative_prompt_embeds=negative_prompt_embeds, # generated from Compel
    image=init_image,
    mask_image=mask_image
).images[0]
make_image_grid([init_image, mask_image, image], rows=1, cols=3)
```

### ControlNet

ControlNet 模型与 Stable Diffusion 等其他扩散模型配合使用，提供了一种更灵活、更精确的方式来控制图像的生成过程。ControlNet 接受一个额外的条件图像输入，引导扩散模型保留该图像中的特征。

例如，让我们使用在 inpaint 图像上预训练的 ControlNet 对图像进行条件控制：

```py
import torch
import numpy as np
from diffusers import ControlNetModel, StableDiffusionControlNetInpaintPipeline
from diffusers.utils import load_image, make_image_grid
# load ControlNet
controlnet = ControlNetModel.from_pretrained("lllyasviel/control_v11p_sd15_inpaint", dtype=torch.float16, variant="fp16")
# pass ControlNet to the pipeline
pipeline = StableDiffusionControlNetInpaintPipeline.from_pretrained(
    "stable-diffusion-v1-5/stable-diffusion-inpainting", controlnet=controlnet, dtype=torch.float16, variant="fp16"
)
pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
pipeline.enable_xformers_memory_efficient_attention()
# load base and mask image
init_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint.png")
mask_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/inpaint_mask.png")
# prepare control image
def make_inpaint_condition(init_image, mask_image):
    init_image = np.array(init_image.convert("RGB")).astype(np.float32) / 255.0
    mask_image = np.array(mask_image.convert("L")).astype(np.float32) / 255.0
    assert init_image.shape[0:1] == mask_image.shape[0:1], "image and image_mask must have the same image size"
    init_image[mask_image > 0.5] = -1.0  # set as masked pixel
    init_image = np.expand_dims(init_image, 0).transpose(0, 3, 1, 2)
    init_image = torch.from_numpy(init_image)
    return init_image
control_image = make_inpaint_condition(init_image, mask_image)
```

现在，根据基础图像、掩码图像和控制图像生成一张图像。您会注意到，生成图像中强烈保留了基础图像的特征。

```py
prompt = "concept art digital painting of an elven castle, inspired by lord of the rings, highly detailed, 8k"
image = pipeline(prompt=prompt, image=init_image, mask_image=mask_image, control_image=control_image).images[0]
make_image_grid([init_image, mask_image, PIL.Image.fromarray(np.uint8(control_image[0][0])).convert('RGB'), image], rows=2, cols=2)
```

您可以更进一步，将其与图像到图像流水线串联起来，以应用新的[风格](https://huggingface.co/nitrosocke/elden-ring-diffusion)：

```py
from diffusers import AutoPipelineForImage2Image
pipeline = AutoPipelineForImage2Image.from_pretrained(
    "nitrosocke/elden-ring-diffusion", dtype=torch.float16,
)
pipeline.enable_model_cpu_offload()
# remove following line if xFormers is not installed or you have PyTorch 2.0 or higher installed
pipeline.enable_xformers_memory_efficient_attention()
prompt = "elden ring style castle" # include the token "elden ring style" in the prompt
negative_prompt = "bad architecture, deformed, disfigured, poor details"
image_elden_ring = pipeline(prompt, negative_prompt=negative_prompt, image=image).images[0]
make_image_grid([init_image, mask_image, image, image_elden_ring], rows=2, cols=2)
```

  
    
    初始图像
  
  
    
    ControlNet 图像修复
  
  
    
    图像到图像
  
## 优化

在资源受限的情况下，运行扩散模型可能既困难又缓慢，但借助一些优化技巧，这种情况完全可以避免。您可以启用的最大（也是最简单的）优化之一是切换到内存高效的注意力机制。如果您使用的是 PyTorch 2.0，[scaled-dot product attention](../optimization/fp16#scaled-dot-product-attention) 会自动启用，您无需执行任何其他操作。对于非 PyTorch 2.0 用户，您可以安装并使用 [xFormers](../optimization/attention_backends) 提供的内存高效注意力实现。这两种选项都能减少内存占用并加速推理。

您还可以将模型卸载到 CPU 以节省更多内存：

```diff
+ pipeline.enable_xformers_memory_efficient_attention()
+ pipeline.enable_model_cpu_offload()
```

为了进一步加快推理代码的速度，请使用 [`torch_compile`](../optimization/fp16#torchcompile)。您应该将 `torch.compile` 包裹在流水线中计算量最大的组件周围，通常是 UNet：

```py
pipeline.unet = torch.compile(pipeline.unet, mode="reduce-overhead", fullgraph=True)
```

有关更多信息，请参阅 [减少内存使用](../optimization/memory) 和 [加速推理](../optimization/fp16) 指南。
Xet Storage Details

Size:: 36.1 kB
Xet hash:: 1714576c745e2563432c4916fd99e1e50996346e50d216601f693b28f11974bc
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.