Buckets:
IP-Adapter
IP-Adapter mengarahkan model difusi yang telah dilatih sebelumnya menggunakan gambar referensi sambil tetap mempertahankan prompt teks. Metode ini membekukan model dasar dan menambahkan seperangkat kecil lapisan cross-attention gambar, yang membuatnya praktis untuk mencocokkan subjek atau gaya dari foto tanpa perlu fine-tuning.
text prompt reference image
| |
text encoder image encoder
| |
v v
text cross-attn (frozen) IP-Adapter cross-attn
\ /
\ /
+--------> denoiser <-----------+
(frozen base)
Checkpoint IP-Adapter biasanya berukuran ~100MB karena menyimpan bobot adapter, bukan model lengkap. Muat pipeline dasar terlebih dahulu, kemudian muat adapter dengan load_ip_adapter().
IP-Adapters tersedia untuk banyak model seperti Flux dan Stable Diffusion 3, serta lainnya. Contoh dalam panduan ini menggunakan Stable Diffusion dan Stable Diffusion XL.
Gunakan set_ip_adapter_scale() untuk mengontrol seberapa kuat IP-Adapter mengarahkan generasi. 1.0 menerapkan adapter dengan kekuatan penuh; 0.5 biasanya menyeimbangkan prompt teks dan gambar.
Contoh di bawah ini menunjukkan IP-Adapter pada tugas-tugas umum.
import torch
from diffusers import AutoPipelineForText2Image
from diffusers.utils import load_image
pipeline = AutoPipelineForText2Image.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
dtype=torch.float16
).to("cuda") # or "mps", "xpu", "cpu"
pipeline.load_ip_adapter(
"h94/IP-Adapter",
subfolder="sdxl_models",
weight_name="ip-adapter_sdxl.bin"
)
pipeline.set_ip_adapter_scale(0.8)
Lewatkan gambar ke ip_adapter_image bersama dengan prompt teks untuk menghasilkan gambar.
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_adapter_diner.png")
pipeline(
prompt="a polar bear sitting in a chair drinking a milkshake",
ip_adapter_image=image,
negative_prompt="deformed, ugly, wrong proportion, low res, bad anatomy, worst quality, low quality",
).images[0]
Gambar IP-Adapter
gambar yang dihasilkan
import torch
from diffusers import AutoPipelineForImage2Image
from diffusers.utils import load_image
pipeline = AutoPipelineForImage2Image.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
dtype=torch.float16
).to("cuda") # or "mps", "xpu", "cpu"
pipeline.load_ip_adapter(
"h94/IP-Adapter",
subfolder="sdxl_models",
weight_name="ip-adapter_sdxl.bin"
)
pipeline.set_ip_adapter_scale(0.8)
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_adapter_bear_1.png")
ip_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_adapter_gummy.png")
pipeline(
prompt="best quality, high quality",
image=image,
ip_adapter_image=ip_image,
strength=0.5,
).images[0]
gambar input
Gambar IP-Adapter
gambar yang dihasilkan
import torch
from diffusers import AutoPipelineForInpainting
from diffusers.utils import load_image
pipeline = AutoPipelineForInpainting.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
dtype=torch.float16
).to("cuda") # or "mps", "xpu", "cpu"
pipeline.load_ip_adapter(
"h94/IP-Adapter",
subfolder="sdxl_models",
weight_name="ip-adapter_sdxl.bin"
)
pipeline.set_ip_adapter_scale(0.6)
mask_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_adapter_mask.png")
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_adapter_bear_1.png")
ip_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_adapter_gummy.png")
pipeline(
prompt="a cute gummy bear waving",
image=image,
mask_image=mask_image,
ip_adapter_image=ip_image,
).images[0]
gambar input
Gambar IP-Adapter
gambar yang dihasilkan
Metode enable_model_cpu_offload() berguna untuk mengurangi memori dan sebaiknya diaktifkan setelah IP-Adapter dimuat. Jika tidak, image encoder dari IP-Adapter juga akan di-offload ke CPU dan mengembalikan error.
import torch
from diffusers import AnimateDiffPipeline, DDIMScheduler, MotionAdapter
from diffusers.utils import export_to_gif
from diffusers.utils import load_image
adapter = MotionAdapter.from_pretrained(
"guoyww/animatediff-motion-adapter-v1-5-2",
dtype=torch.float16
)
pipeline = AnimateDiffPipeline.from_pretrained(
"emilianJR/epiCRealism",
motion_adapter=adapter,
dtype=torch.float16
)
scheduler = DDIMScheduler.from_pretrained(
"emilianJR/epiCRealism",
subfolder="scheduler",
clip_sample=False,
timestep_spacing="linspace",
beta_schedule="linear",
steps_offset=1,
)
pipeline.scheduler = scheduler
pipeline.vae.enable_slicing()
pipeline.load_ip_adapter("h94/IP-Adapter", subfolder="models", weight_name="ip-adapter_sd15.bin")
pipeline.enable_model_cpu_offload()
ip_adapter_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_adapter_inpaint.png")
pipeline(
prompt="A cute gummy bear waving",
negative_prompt="bad quality, worse quality, low resolution",
ip_adapter_image=ip_adapter_image,
num_frames=16,
guidance_scale=7.5,
num_inference_steps=50,
).frames[0]
Gambar IP-Adapter
video yang dihasilkan
Varian checkpoint
Muat Plus ketika detail dari gambar referensi paling penting. Muat FaceID ketika Anda memerlukan embedding identitas InsightFace alih-alih embedding gambar CLIP.
import torch
from transformers import CLIPVisionModelWithProjection
from diffusers import AutoPipelineForText2Image
image_encoder = CLIPVisionModelWithProjection.from_pretrained(
"h94/IP-Adapter",
subfolder="models/image_encoder",
dtype=torch.float16
)
pipeline = AutoPipelineForText2Image.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
image_encoder=image_encoder,
dtype=torch.float16
).to("cuda") # or "mps", "xpu", "cpu"
pipeline.load_ip_adapter(
"h94/IP-Adapter",
subfolder="sdxl_models",
weight_name="ip-adapter-plus_sdxl_vit-h.safetensors"
)
Checkpoint FaceID menggunakan face embeddings dari InsightFace alih-alih image embeddings CLIP. Gunakan DDIMScheduler atau EulerDiscreteScheduler untuk model FaceID. Ekstrak face embeddings dan lewati sebagai daftar tensor ke ip_adapter_image_embeds.
# pip install insightface
import cv2
import numpy as np
import torch
from diffusers import StableDiffusionPipeline, DDIMScheduler
from diffusers.utils import load_image
from insightface.app import FaceAnalysis
pipeline = StableDiffusionPipeline.from_pretrained(
"stable-diffusion-v1-5/stable-diffusion-v1-5",
dtype=torch.float16,
).to("cuda") # or "mps", "xpu", "cpu"
pipeline.scheduler = DDIMScheduler.from_config(pipeline.scheduler.config)
pipeline.load_ip_adapter(
"h94/IP-Adapter-FaceID",
subfolder=None,
weight_name="ip-adapter-faceid_sd15.bin",
image_encoder_folder=None
)
pipeline.set_ip_adapter_scale(0.6)
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_mask_girl1.png")
ref_images_embeds = []
app = FaceAnalysis(name="buffalo_l", providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
app.prepare(ctx_id=0, det_size=(640, 640))
image = cv2.cvtColor(np.asarray(image), cv2.COLOR_RGB2BGR)
faces = app.get(image)
image = torch.from_numpy(faces[0].normed_embedding)
ref_images_embeds.append(image.unsqueeze(0))
ref_images_embeds = torch.stack(ref_images_embeds, dim=0).unsqueeze(0)
neg_ref_images_embeds = torch.zeros_like(ref_images_embeds)
id_embeds = torch.cat([neg_ref_images_embeds, ref_images_embeds]).to(dtype=torch.float16, device="cuda")
pipeline(
prompt="A photo of a girl",
ip_adapter_image_embeds=[id_embeds],
negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality",
).images[0]
Untuk FaceID Plus, muat CLIPVisionModelWithProjection sebagai encoder gambar.
import torch
from transformers import CLIPVisionModelWithProjection
from diffusers import AutoPipelineForText2Image
image_encoder = CLIPVisionModelWithProjection.from_pretrained(
"laion/CLIP-ViT-H-14-laion2B-s32B-b79K",
dtype=torch.float16,
)
pipeline = AutoPipelineForText2Image.from_pretrained(
"stable-diffusion-v1-5/stable-diffusion-v1-5",
image_encoder=image_encoder,
dtype=torch.float16
).to("cuda") # or "mps", "xpu", "cpu"
pipeline.load_ip_adapter(
"h94/IP-Adapter-FaceID",
subfolder=None,
weight_name="ip-adapter-faceid-plus_sd15.bin"
)
FaceID Plus dan Plus v2 juga memerlukan embedding gambar CLIP di atas embeds InsightFace. Setelah Anda menyiapkan embedding wajah, encode gambar referensi dan tetapkan embeds CLIP ke lapisan proyeksi gambar.
import torch
from diffusers.utils import load_image
ip_adapter_images = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_mask_girl1.png")
num_images = 1
clip_embeds = pipeline.prepare_ip_adapter_image_embeds(
ip_adapter_image=[ip_adapter_images],
ip_adapter_image_embeds=None,
device=torch.device("cuda"),
num_images_per_prompt=num_images,
do_classifier_free_guidance=True,
)[0]
pipeline.unet.encoder_hid_proj.image_projection_layers[0].clip_embeds = clip_embeds.to(dtype=torch.float16)
# set to True if using IP-Adapter FaceID Plus v2
pipeline.unet.encoder_hid_proj.image_projection_layers[0].shortcut = False
Embedding gambar
prepare_ip_adapter_image_embeds() mengodekan gambar IP-Adapter menjadi embeddings yang dapat Anda simpan dan gunakan kembali. Hitung sebelumnya sekali alih-alih memuat dan mengodekan gambar yang sama setiap kali Anda menjalankan pipeline.
import torch
from diffusers import AutoPipelineForText2Image
from diffusers.utils import load_image
pipeline = AutoPipelineForText2Image.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
dtype=torch.float16
).to("cuda") # or "mps", "xpu", "cpu"
pipeline.load_ip_adapter(
"h94/IP-Adapter",
subfolder="sdxl_models",
weight_name="ip-adapter_sdxl.bin"
)
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_adapter_bear_1.png")
image_embeds = pipeline.prepare_ip_adapter_image_embeds(
ip_adapter_image=image,
ip_adapter_image_embeds=None,
device="cuda",
num_images_per_prompt=1,
do_classifier_free_guidance=True,
)
torch.save(image_embeds, "image_embeds.ipadpt")
Muat ulang embedding gambar dengan meneruskannya ke parameter ip_adapter_image_embeds. Atur image_encoder_folder ke None karena Anda tidak lagi memerlukan image encoder untuk menghasilkan embedding gambar.
Anda juga dapat memuat embedding gambar dari sumber lain seperti ComfyUI.
pipeline = AutoPipelineForText2Image.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
dtype=torch.float16
).to("cuda") # or "mps", "xpu", "cpu"
pipeline.load_ip_adapter(
"h94/IP-Adapter",
subfolder="sdxl_models",
image_encoder_folder=None,
weight_name="ip-adapter_sdxl.bin"
)
pipeline.set_ip_adapter_scale(0.8)
image_embeds = torch.load("image_embeds.ipadpt")
pipeline(
prompt="a polar bear sitting in a chair drinking a milkshake",
ip_adapter_image_embeds=image_embeds,
negative_prompt="deformed, ugly, wrong proportion, low res, bad anatomy, worst quality, low quality",
num_inference_steps=100,
).images[0]
Masking
Masking biner memungkinkan penugasan gambar IP-Adapter ke area tertentu pada gambar output, sehingga berguna untuk menyusun beberapa gambar IP-Adapter. Setiap gambar IP-Adapter memerlukan mask biner.
Muat IPAdapterMaskProcessor untuk memproses prapengolahan mask gambar. Untuk hasil terbaik, berikan height dan width keluaran untuk memastikan mask dengan rasio aspek yang berbeda berukuran dengan tepat. Jika mask input sudah sesuai dengan rasio aspek gambar yang dihasilkan, Anda tidak perlu mengatur height dan width.
import torch
from diffusers import AutoPipelineForText2Image
from diffusers.image_processor import IPAdapterMaskProcessor
from diffusers.utils import load_image
pipeline = AutoPipelineForText2Image.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
dtype=torch.float16
).to("cuda") # or "mps", "xpu", "cpu"
mask1 = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_mask_mask1.png")
mask2 = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_mask_mask2.png")
processor = IPAdapterMaskProcessor()
masks = processor.preprocess([mask1, mask2], height=1024, width=1024)
mask 1
mask 2
Sediakan gambar IP-Adapter dan skala mereka sebagai daftar. Teruskan mask yang telah dipreproses ke cross_attention_kwargs di pipeline.
face_image1 = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_mask_girl1.png")
face_image2 = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/ip_mask_girl2.png")
pipeline.load_ip_adapter(
"h94/IP-Adapter",
subfolder="sdxl_models",
weight_name=["ip-adapter-plus-face_sdxl_vit-h.safetensors"]
)
pipeline.set_ip_adapter_scale([[0.7, 0.7]])
ip_images = [[face_image1, face_image2]]
masks = [masks.reshape(1, masks.shape[0], masks.shape[2], masks.shape[3])]
pipeline(
prompt="2 girls",
ip_adapter_image=ip_images,
negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality",
cross_attention_kwargs={"ip_adapter_masks": masks}
).images[0]
Gambar IP-Adapter 1
IP-Adapter gambar 2
dihasilkan dengan mask
dihasilkan tanpa mask
Resep
Gabungkan IP-Adapter dengan adapter atau pipeline lain, seperti beberapa adapter, ControlNet, InstantStyle, atau LCM few-step, ketika satu gambar referensi tidak cukup.
Beberapa IP-Adapter
Gabungkan beberapa IP-Adapter untuk menghasilkan gambar dengan gaya yang lebih beragam. Misalnya, Anda dapat menggunakan IP-Adapter Face untuk menghasilkan wajah dan karakter yang konsisten, serta IP-Adapter Plus untuk menghasilkan wajah tersebut dalam gaya tertentu.
Muat encoder gambar dengan CLIPVisionModelWithProjection.
import torch
from diffusers import AutoPipelineForText2Image, DDIMScheduler
from transformers import CLIPVisionModelWithProjection
from diffusers.utils import load_image
image_encoder = CLIPVisionModelWithProjection.from_pretrained(
"h94/IP-Adapter",
subfolder="models/image_encoder",
dtype=torch.float16,
)
Muat model dasar, penjadwal, dan IP-Adapter berikut.
- ip-adapter-plus_sdxl_vit-h menggunakan patch embeddings dan image encoder ViT-H
- ip-adapter-plus-face_sdxl_vit-h menggunakan patch embeddings dan image encoder ViT-H, tetapi dikondisikan pada gambar wajah yang dipotong
pipeline = AutoPipelineForText2Image.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
dtype=torch.float16,
image_encoder=image_encoder,
)
pipeline.scheduler = DDIMScheduler.from_config(pipeline.scheduler.config)
pipeline.load_ip_adapter(
"h94/IP-Adapter",
subfolder="sdxl_models",
weight_name=["ip-adapter-plus_sdxl_vit-h.safetensors", "ip-adapter-plus-face_sdxl_vit-h.safetensors"]
)
pipeline.set_ip_adapter_scale([0.7, 0.3])
# enable_model_cpu_offload to reduce memory usage
pipeline.enable_model_cpu_offload()
Muat sebuah gambar dan folder yang berisi gambar-gambar dengan gaya tertentu untuk diterapkan.
face_image = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/women_input.png")
style_folder = "https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/style_ziggy"
style_images = [load_image(f"{style_folder}/img{i}.png") for i in range(10)]
gambar wajah
gaya gambar
Lewatkan gambar gaya dan wajah sebagai daftar ke ip_adapter_image.
generator = torch.Generator(device="cpu").manual_seed(0)
pipeline(
prompt="wonderwoman",
ip_adapter_image=[style_images, face_image],
negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality",
).images[0]
gambar yang dihasilkan
Kontrol struktural
Untuk kontrol struktural, kombinasikan IP-Adapter dengan ControlNet yang dikondisikan pada peta kedalaman, peta tepi, estimasi pose, dan lainnya.
Contoh di bawah ini memuat checkpoint ControlNetModel yang dikondisikan pada peta kedalaman dan menggabungkannya dengan IP-Adapter.
import torch
from diffusers.utils import load_image
from diffusers import StableDiffusionControlNetPipeline, ControlNetModel
controlnet = ControlNetModel.from_pretrained(
"lllyasviel/control_v11f1p_sd15_depth",
dtype=torch.float16
)
pipeline = StableDiffusionControlNetPipeline.from_pretrained(
"stable-diffusion-v1-5/stable-diffusion-v1-5",
controlnet=controlnet,
dtype=torch.float16
).to("cuda") # or "mps", "xpu", "cpu"
pipeline.load_ip_adapter(
"h94/IP-Adapter",
subfolder="models",
weight_name="ip-adapter_sd15.bin"
)
Lewatkan peta kedalaman dan gambar IP-Adapter ke pipeline.
depth_map = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/depth.png")
ip_adapter_image = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/statue.png")
pipeline(
prompt="best quality, high quality",
image=depth_map,
ip_adapter_image=ip_adapter_image,
negative_prompt="monochrome, lowres, bad anatomy, worst quality, low quality",
).images[0]
Gambar IP-Adapter
peta kedalaman
gambar yang dihasilkan
Kontrol gaya dan tata letak
Untuk kontrol gaya dan tata letak, kombinasikan IP-Adapter dengan InstantStyle. InstantStyle memisahkan gaya (warna, tekstur, kesan keseluruhan) dari konten dan menerapkan gaya hanya pada blok yang spesifik untuk gaya, sehingga area konten tetap utuh. Hal ini memberikan konsistensi gaya yang lebih kuat dan kontrol tata letak yang lebih jelas.
Aktifkan IP-Adapter hanya pada lapisan tertentu dengan set_ip_adapter_scale(). Contoh di bawah ini mengaktifkannya pada block_2 (layout) bagian down dan block_0 (style) bagian up dari model.
import torch
from diffusers import AutoPipelineForText2Image
from diffusers.utils import load_image
pipeline = AutoPipelineForText2Image.from_pretrained(
"stabilityai/stable-diffusion-xl-base-1.0",
dtype=torch.float16
).to("cuda") # or "mps", "xpu", "cpu"
pipeline.load_ip_adapter(
"h94/IP-Adapter",
subfolder="sdxl_models",
weight_name="ip-adapter_sdxl.bin"
)
scale = {
"down": {"block_2": [0.0, 1.0]},
"up": {"block_0": [0.0, 1.0, 0.0]},
}
pipeline.set_ip_adapter_scale(scale)
Muat gambar gaya dan hasilkan gambar.
style_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg")
pipeline(
prompt="a cat, masterpiece, best quality, high quality",
ip_adapter_image=style_image,
negative_prompt="text, watermark, lowres, low quality, worst quality, deformed, glitch, low contrast, noisy, saturation, blurry",
guidance_scale=5,
).images[0]
style image
gambar yang dihasilkan
Gambar-gambar di bawah ini membandingkan aktivasi gaya-only InstantStyle (hanya block_0) dengan mengaktifkan IP-Adapter di semua lapisan. Pengaturan semua lapisan biasanya mengikuti prompt gambar lebih kuat dan dapat mengurangi keragaman. Pilih skala gaya-only jika Anda menginginkan perilaku pelestarian tata letak dari InstantStyle.
Anda tidak perlu menentukan semua lapisan dalam kamus
scale. Lapisan yang tidak disertakan diatur ke 0, yang berarti IP-Adapter dinonaktifkan.
scale = {
"up": {"block_0": [0.0, 1.0, 0.0]},
}
pipeline.set_ip_adapter_scale(scale)
pipeline(
prompt="a cat, masterpiece, best quality, high quality",
ip_adapter_image=style_image,
negative_prompt="text, watermark, lowres, low quality, worst quality, deformed, glitch, low contrast, noisy, saturation, blurry",
guidance_scale=5,
).images[0]
hanya-gaya (blok_0 atas)
semua lapisan
Generasi instan
Gabungkan IP-Adapter dengan LCM LoRA untuk generasi beberapa langkah.
import torch
from diffusers import DiffusionPipeline, LCMScheduler
from diffusers.utils import load_image
pipeline = DiffusionPipeline.from_pretrained(
"sd-dreambooth-library/herge-style",
dtype=torch.float16
)
pipeline.load_ip_adapter(
"h94/IP-Adapter",
subfolder="models",
weight_name="ip-adapter_sd15.bin"
)
pipeline.load_lora_weights("latent-consistency/lcm-lora-sdv1-5")
pipeline.scheduler = LCMScheduler.from_config(pipeline.scheduler.config)
pipeline.enable_model_cpu_offload()
pipeline.set_ip_adapter_scale(0.4)
ip_adapter_image = load_image("https://user-images.githubusercontent.com/24734142/266492875-2d50d223-8475-44f0-a7c6-08b51cb53572.png")
pipeline(
prompt="herge_style woman in armor, best quality, high quality",
ip_adapter_image=ip_adapter_image,
num_inference_steps=4,
guidance_scale=1,
).images[0]
gambar yang dihasilkan
Xet Storage Details
- Size:
- 23.3 kB
- Xet hash:
- 63bc3da99ec93329abdce6cb76ffdc76d36feb9dee5bbaea0ee5e6f6a47acd92
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.