Viggle/Viggle-Image-Swap

Viggle Image Swap

Built with Qwen.

A LoRA adapter for Qwen/Qwen-Image-2.1. It replaces people in a picture with characters from character sheets. Each person is replaced whole: face, hair, body and clothes. The pose, expression, framing, background and lighting of the picture stay as they were. It samples in 6 steps without classifier-free guidance, or, with --turbo, in 2 steps handed over to Qwen-Image-2.1-Turbo for 6 more: sharper detail.

Examples

Each row is: the character sheet(s), the source picture, the output. All are made with inference/swap.py at its defaults (seed 0, tone fix on); the only inputs are the sheets, the source and the short WHO text shown.

The top image replaces two people at once: --char queen.png woman --char king.png man --left 1 2.

One person, letterboxed source: --char astronaut.png dancer

Two people in close contact, both replaced: --char flavio.png man --char fleur.png woman --left 1 2

A close-up selfie: --char odessa.png woman

A full-body mascot costume; the prop in his hand stays: --char pete.png man

file	what it is
viggle-image-swap-v0.1-lora.safetensors	the LoRA (rank 256, alpha 256), bf16, diffusers key format
inference/swap.py	command-line inference: builds the prompt, orders the images, samples, tone fix
requirements.txt	the tested package versions

Inputs

• The source picture: any size and aspect. The output is 1 MP at the source's aspect, with sides multiples of 32.
• One character sheet per person to replace: a 1536×1024 image. The left third is a portrait; the right two thirds is a full-body three-view (front, side, back) on a plain background. A photo reference can be used by pasting a face photo over the sheet's portrait cell (left third, rows 128–896, 512×768). A bare photo on its own is not supported.

Image order: the sheets first (image 1, 2, ...), the source last. Each image is shrunk to at most 1 MP.

Prompt

One sentence per character, then one sentence that names what to keep. With one sheet (image 1) and the source as image 2:

Replace the man in a green shirt in image 2 with the character from image 1. Keep everything else exactly as in image 2: the pose, expression, framing, background and lighting.

With two people in the picture and only one replaced, say where the person is and that the other stays:

Replace the woman on the left in image 2 with the character from image 1. Keep everything else exactly as in image 2: the pose, expression, framing, background and lighting, and leave the other person unchanged.

Positions are on the left / on the right, in the middle (three people), or 2nd from the left and so on. inference/swap.py writes these prompts for you (--left, --kept).

Install

pip install -U torch torchvision "transformers==5.17.0" accelerate safetensors "peft==0.18.0" pillow numpy
pip install "git+https://github.com/huggingface/diffusers.git@80c7ed262aeffbeb43ef13ae04baeb9b84515a69"

QwenImage21Pipeline is pinned to that diffusers commit, which is what this adapter was tested on.

Usage

Command line:

python inference/swap.py --src scene.png --char sheet.png "man in a green shirt" --out out.png
# two people replaced, one kept: places are 1-based, counted left to right among all three
python inference/swap.py --src scene.png --char a.png "woman in red" --char b.png "man in a cap" --left 1 3 --kept 1
python inference/swap.py --src scene.png --char sheet.png "man in a green shirt" --dry-run   # prompt, sizes, sigmas; no GPU
python inference/swap.py --src scene.png --char sheet.png "man in a green shirt" --turbo     # sharper: 2 + 6 Turbo steps

Python:

import torch
from diffusers import QwenImage21Pipeline, FlowMatchEulerDiscreteScheduler
from PIL import Image

sched = FlowMatchEulerDiscreteScheduler.from_pretrained("Qwen/Qwen-Image-2.1", subfolder="scheduler")
sched = FlowMatchEulerDiscreteScheduler.from_config(sched.config, shift_terminal=None)
pipe = QwenImage21Pipeline.from_pretrained("Qwen/Qwen-Image-2.1", scheduler=sched, dtype=torch.bfloat16)
pipe.load_lora_weights("Viggle/Viggle-Image-Swap", weight_name="viggle-image-swap-v0.1-lora.safetensors")
pipe.to("cuda")

sheet, src = Image.open("sheet.png").convert("RGB"), Image.open("scene.png").convert("RGB")  # each at most 1 MP
prompt = ("Replace the man in a green shirt in image 2 with the character from image 1. Keep everything else exactly as in "
          "image 2: the pose, expression, framing, background and lighting.")
out = pipe(prompt=prompt, image=[sheet, src], num_inference_steps=6, sigmas=[1.0, 0.9375, 0.875, 0.75, 0.5, 0.25],
           true_cfg_scale=1.0, output_resolution=1024, generator=torch.Generator("cpu").manual_seed(0)).images[0]
out.save("out.png")

The sigmas are passed raw; the pipeline adds its resolution-dependent shift. shift_terminal=None is required. swap.py also replaces the vision encoder's bf16 Conv3d patch embedding with the equal matmul, which saves about 30 s per input image; the output is bitwise the same.

Sharper output: --turbo

--turbo runs the whole sample on Qwen-Image-2.1-Turbo's scheduler and its 8 sample_sigmas (from its model_index.json). The LoRA model takes the first 2 steps, which settle who is in the picture; then the transformer is swapped for Qwen-Image-2.1-Turbo's, which runs the other 6 and draws finer detail. The KV cache is off, since the two transformers cannot share one. --turbo-model points at another copy of Qwen-Image-2.1-Turbo. In Python, the same is:

from diffusers import QwenImage21Transformer2DModel

turbo = QwenImage21Transformer2DModel.from_pretrained("Qwen/Qwen-Image-2.1-Turbo", subfolder="transformer",
                                                      torch_dtype=torch.bfloat16).to("cuda")
pipe.scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained("Qwen/Qwen-Image-2.1-Turbo", subfolder="scheduler")
sigmas = QwenImage21Pipeline.load_config("Qwen/Qwen-Image-2.1-Turbo")["sample_sigmas"]


def handoff(p, i, t, kw):  # after the LoRA model's 2nd step
    if i == 1:
        p.transformer = turbo
    return kw


out = pipe(prompt=prompt, image=[sheet, src], num_inference_steps=len(sigmas), sigmas=sigmas, true_cfg_scale=1.0,
           output_resolution=1024, generator=torch.Generator("cpu").manual_seed(0), callback_on_step_end=handoff,
           use_kv_cache=False).images[0]

With only 2 steps of its own the LoRA model hands over a rough picture, so Turbo sometimes pulls back parts of the source person: their sunglasses, beard or clothes, and now and then the whole person. Without --turbo that happens less, at a softer finish. Try another --seed, or drop --turbo, when it does.

Tone fix

The output can be 1–4 levels darker or lighter than the source over the whole frame. swap.py corrects this by default: per colour channel, it subtracts the mean offset from the source over the pixels that did not change (the background). It does nothing when under 30 % of the frame is unchanged. --no-tone-fix turns it off.

Cost

Measured on one B200 with the models loaded, a 1344×768 output and one sheet, bf16: about 2.2 s per image and 46 GB peak; with --turbo, about 4.1 s and 55 GB (51.3 GiB) peak, the second transformer included.

Limitations

• The person to replace is chosen by the prompt. With several people, give the position.
• Sometimes the new character keeps the source person's clothes.
• The pose or framing of the new character can drift from the source person's.
• The output is 1 MP.
• --turbo can bring back the source person's accessories, clothes or, rarely, the whole person (see above).

Example footage

The source pictures in examples/ are frames of third-party footage, © their respective owners, credited by YouTube channel and video ID. The character sheets are original.

example	source	YouTube video
card-royals	West Coast Swing Online	nRdO0HS-D5E
astronaut	Chris King	A37myLTm2Vo
flamingos	jtballroom	GAZ2LeNX3BU
aviatrix	Alecia Renece The Artist	rHDTkqDqBp0
pineapple-mascot	Massimo Nocerino	W1P5qiqaqCs

The video at the top (examples/viggle-image-swap.mp4) uses frames of films, advertisements and other third-party footage, © their respective owners. Some of its character sheets depict real public figures; they are made for this research demonstration and imply no endorsement.

License

The adapter is a derivative of Qwen-Image-2.1 and is released under the Qwen RESEARCH LICENSE AGREEMENT: research and evaluation use only. Commercial use needs a separate licence from the Qwen team (model-business@notice.qwencloud.com). --turbo also loads Qwen/Qwen-Image-2.1-Turbo, under the same licence. See NOTICE. The inference code in inference/ is under the Apache License 2.0 (LICENSE-CODE).