Add release video

572d497 verified 10 days ago

9.88 kB

---
license: other
license_name: qwen-research
license_link: LICENSE
base_model: Qwen/Qwen-Image-2.1
base_model_relation: adapter
library_name: diffusers
pipeline_tag: image-to-image
tags:
  - diffusers
  - lora
  - image-to-image
  - image-editing
  - character-swap
  - few-step
  - qwen-image
---
# Viggle Image Swap

**Built with Qwen.**

A LoRA adapter for [Qwen/Qwen-Image-2.1](https://huggingface.co/Qwen/Qwen-Image-2.1). It replaces people in a picture with
characters from character sheets. Each person is replaced whole: face, hair, body and clothes. The pose, expression, framing,
background and lighting of the picture stay as they were. It samples in 6 steps without classifier-free guidance, or, with
`--turbo`, in 2 steps handed over to [Qwen-Image-2.1-Turbo](https://huggingface.co/Qwen/Qwen-Image-2.1-Turbo) for 6 more: sharper detail.

<video src="https://huggingface.co/Viggle/Viggle-Image-Swap/resolve/main/examples/viggle-image-swap.mp4" controls autoplay muted loop playsinline width="100%"></video>

![Two dancers recast as playing-card royals](examples/card-royals.jpg)

## Examples

Each row is: the character sheet(s), the source picture, the output. All are made with `inference/swap.py` at its defaults
(seed 0, tone fix on); the only inputs are the sheets, the source and the short `WHO` text shown.

The top image replaces two people at once: `--char queen.png woman --char king.png man --left 1 2`.

![Dancer recast as an astronaut](examples/astronaut.jpg)
One person, letterboxed source: `--char astronaut.png dancer`

![Ballroom couple recast as flamingos](examples/flamingos.jpg)
Two people in close contact, both replaced: `--char flavio.png man --char fleur.png woman --left 1 2`

![Selfie recast as a 1920s aviatrix](examples/aviatrix.jpg)
A close-up selfie: `--char odessa.png woman`

![Man recast as a pineapple mascot](examples/pineapple-mascot.jpg)
A full-body mascot costume; the prop in his hand stays: `--char pete.png man`

| file | what it is |
|---|---|
| `viggle-image-swap-v0.1-lora.safetensors` | the LoRA (rank 256, alpha 256), bf16, diffusers key format |
| `inference/swap.py` | command-line inference: builds the prompt, orders the images, samples, tone fix |
| `requirements.txt` | the tested package versions |

## Inputs

- **The source picture**: any size and aspect. The output is 1 MP at the source's aspect, with sides multiples of 32.
- **One character sheet per person to replace**: a 1536×1024 image. The left third is a portrait; the right two thirds is
  a full-body three-view (front, side, back) on a plain background. A photo reference can be used by pasting a face photo
  over the sheet's portrait cell (left third, rows 128–896, 512×768). A bare photo on its own is not supported.

Image order: the sheets first (image 1, 2, ...), the source last. Each image is shrunk to at most 1 MP.

## Prompt

One sentence per character, then one sentence that names what to keep. With one sheet (image 1) and the source as image 2:

```
Replace the man in a green shirt in image 2 with the character from image 1. Keep everything else exactly as in image 2: the pose, expression, framing, background and lighting.
```

With two people in the picture and only one replaced, say where the person is and that the other stays:

```
Replace the woman on the left in image 2 with the character from image 1. Keep everything else exactly as in image 2: the pose, expression, framing, background and lighting, and leave the other person unchanged.
```

Positions are `on the left` / `on the right`, `in the middle` (three people), or `2nd from the left` and so on.
`inference/swap.py` writes these prompts for you (`--left`, `--kept`).

## Install

```bash
pip install -U torch torchvision "transformers==5.17.0" accelerate safetensors "peft==0.18.0" pillow numpy
pip install "git+https://github.com/huggingface/diffusers.git@80c7ed262aeffbeb43ef13ae04baeb9b84515a69"
```

`QwenImage21Pipeline` is pinned to that diffusers commit, which is what this adapter was tested on.

## Usage

Command line:

```bash
python inference/swap.py --src scene.png --char sheet.png "man in a green shirt" --out out.png
# two people replaced, one kept: places are 1-based, counted left to right among all three
python inference/swap.py --src scene.png --char a.png "woman in red" --char b.png "man in a cap" --left 1 3 --kept 1
python inference/swap.py --src scene.png --char sheet.png "man in a green shirt" --dry-run   # prompt, sizes, sigmas; no GPU
python inference/swap.py --src scene.png --char sheet.png "man in a green shirt" --turbo     # sharper: 2 + 6 Turbo steps
```

Python:

```python
import torch
from diffusers import QwenImage21Pipeline, FlowMatchEulerDiscreteScheduler
from PIL import Image
sched = FlowMatchEulerDiscreteScheduler.from_pretrained("Qwen/Qwen-Image-2.1", subfolder="scheduler")
sched = FlowMatchEulerDiscreteScheduler.from_config(sched.config, shift_terminal=None)
pipe = QwenImage21Pipeline.from_pretrained("Qwen/Qwen-Image-2.1", scheduler=sched, dtype=torch.bfloat16)
pipe.load_lora_weights("Viggle/Viggle-Image-Swap", weight_name="viggle-image-swap-v0.1-lora.safetensors")
pipe.to("cuda")
sheet, src = Image.open("sheet.png").convert("RGB"), Image.open("scene.png").convert("RGB")  # each at most 1 MP
prompt = ("Replace the man in a green shirt in image 2 with the character from image 1. Keep everything else exactly as in "
          "image 2: the pose, expression, framing, background and lighting.")
out = pipe(prompt=prompt, image=[sheet, src], num_inference_steps=6, sigmas=[1.0, 0.9375, 0.875, 0.75, 0.5, 0.25],
           true_cfg_scale=1.0, output_resolution=1024, generator=torch.Generator("cpu").manual_seed(0)).images[0]
out.save("out.png")
```

The `sigmas` are passed raw; the pipeline adds its resolution-dependent shift. `shift_terminal=None` is required.
`swap.py` also replaces the vision encoder's bf16 Conv3d patch embedding with the equal matmul, which saves about 30 s per input
image; the output is bitwise the same.

### Sharper output: `--turbo`

`--turbo` runs the whole sample on Qwen-Image-2.1-Turbo's scheduler and its 8 `sample_sigmas` (from its `model_index.json`).
The LoRA model takes the first 2 steps, which settle who is in the picture; then the transformer is swapped for
Qwen-Image-2.1-Turbo's, which runs the other 6 and draws finer detail. The KV cache is off, since the two transformers
cannot share one. `--turbo-model` points at another copy of Qwen-Image-2.1-Turbo. In Python, the same is:

```python
from diffusers import QwenImage21Transformer2DModel
turbo = QwenImage21Transformer2DModel.from_pretrained("Qwen/Qwen-Image-2.1-Turbo", subfolder="transformer",
                                                      torch_dtype=torch.bfloat16).to("cuda")
pipe.scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained("Qwen/Qwen-Image-2.1-Turbo", subfolder="scheduler")
sigmas = QwenImage21Pipeline.load_config("Qwen/Qwen-Image-2.1-Turbo")["sample_sigmas"]
def handoff(p, i, t, kw):  # after the LoRA model's 2nd step
    if i == 1:
        p.transformer = turbo
    return kw
out = pipe(prompt=prompt, image=[sheet, src], num_inference_steps=len(sigmas), sigmas=sigmas, true_cfg_scale=1.0,
           output_resolution=1024, generator=torch.Generator("cpu").manual_seed(0), callback_on_step_end=handoff,
           use_kv_cache=False).images[0]
```

With only 2 steps of its own the LoRA model hands over a rough picture, so Turbo sometimes pulls back parts of the source
person: their sunglasses, beard or clothes, and now and then the whole person. Without `--turbo` that happens less, at a
softer finish. Try another `--seed`, or drop `--turbo`, when it does.

### Tone fix

The output can be 1–4 levels darker or lighter than the source over the whole frame. `swap.py` corrects this by default: per
colour channel, it subtracts the mean offset from the source over the pixels that did not change (the background). It does
nothing when under 30 % of the frame is unchanged. `--no-tone-fix` turns it off.

## Cost

Measured on one B200 with the models loaded, a 1344×768 output and one sheet, bf16: about 2.2 s per image and 46 GB peak;
with `--turbo`, about 4.1 s and 55 GB (51.3 GiB) peak, the second transformer included.

## Limitations

- The person to replace is chosen by the prompt. With several people, give the position.
- Sometimes the new character keeps the source person's clothes.
- The pose or framing of the new character can drift from the source person's.
- The output is 1 MP.
- `--turbo` can bring back the source person's accessories, clothes or, rarely, the whole person (see above).

## Example footage

The source pictures in `examples/` are frames of third-party footage, © their respective owners, credited by YouTube channel
and video ID. The character sheets are original.

| example | source | YouTube video |
|---|---|---|
| card-royals | West Coast Swing Online | `nRdO0HS-D5E` |
| astronaut | Chris King | `A37myLTm2Vo` |
| flamingos | jtballroom | `GAZ2LeNX3BU` |
| aviatrix | Alecia Renece The Artist | `rHDTkqDqBp0` |
| pineapple-mascot | Massimo Nocerino | `W1P5qiqaqCs` |

The video at the top (`examples/viggle-image-swap.mp4`) uses frames of films, advertisements and other third-party footage, © their respective owners. Some of its character sheets depict real public figures; they are made for this research demonstration and imply no endorsement.

## License

The adapter is a derivative of Qwen-Image-2.1 and is released under the [Qwen RESEARCH LICENSE AGREEMENT](LICENSE): research
and evaluation use only. Commercial use needs a separate licence from the Qwen team (model-business@notice.qwencloud.com).
`--turbo` also loads Qwen/Qwen-Image-2.1-Turbo, under the same licence.
See [NOTICE](NOTICE). The inference code in `inference/` is under the Apache License 2.0 ([LICENSE-CODE](LICENSE-CODE)).

	---
	license: other
	license_name: qwen-research
	license_link: LICENSE
	base_model: Qwen/Qwen-Image-2.1
	base_model_relation: adapter
	library_name: diffusers
	pipeline_tag: image-to-image
	tags:
	- diffusers
	- lora
	- image-to-image
	- image-editing
	- character-swap
	- few-step
	- qwen-image
	---

	# Viggle Image Swap

	Built with Qwen.

	A LoRA adapter for [Qwen/Qwen-Image-2.1](https://huggingface.co/Qwen/Qwen-Image-2.1). It replaces people in a picture with
	characters from character sheets. Each person is replaced whole: face, hair, body and clothes. The pose, expression, framing,
	background and lighting of the picture stay as they were. It samples in 6 steps without classifier-free guidance, or, with
	`--turbo`, in 2 steps handed over to [Qwen-Image-2.1-Turbo](https://huggingface.co/Qwen/Qwen-Image-2.1-Turbo) for 6 more: sharper detail.

	<video src="https://huggingface.co/Viggle/Viggle-Image-Swap/resolve/main/examples/viggle-image-swap.mp4" controls autoplay muted loop playsinline width="100%"></video>

	![Two dancers recast as playing-card royals](examples/card-royals.jpg)

	## Examples

	Each row is: the character sheet(s), the source picture, the output. All are made with `inference/swap.py` at its defaults
	(seed 0, tone fix on); the only inputs are the sheets, the source and the short `WHO` text shown.

	The top image replaces two people at once: `--char queen.png woman --char king.png man --left 1 2`.

	![Dancer recast as an astronaut](examples/astronaut.jpg)
	One person, letterboxed source: `--char astronaut.png dancer`

	![Ballroom couple recast as flamingos](examples/flamingos.jpg)
	Two people in close contact, both replaced: `--char flavio.png man --char fleur.png woman --left 1 2`

	![Selfie recast as a 1920s aviatrix](examples/aviatrix.jpg)
	A close-up selfie: `--char odessa.png woman`

	![Man recast as a pineapple mascot](examples/pineapple-mascot.jpg)
	A full-body mascot costume; the prop in his hand stays: `--char pete.png man`

	\| file \| what it is \|
	\|---\|---\|
	\| `viggle-image-swap-v0.1-lora.safetensors` \| the LoRA (rank 256, alpha 256), bf16, diffusers key format \|
	\| `inference/swap.py` \| command-line inference: builds the prompt, orders the images, samples, tone fix \|
	\| `requirements.txt` \| the tested package versions \|

	## Inputs

	- The source picture: any size and aspect. The output is 1 MP at the source's aspect, with sides multiples of 32.
	- One character sheet per person to replace: a 1536×1024 image. The left third is a portrait; the right two thirds is
	a full-body three-view (front, side, back) on a plain background. A photo reference can be used by pasting a face photo
	over the sheet's portrait cell (left third, rows 128–896, 512×768). A bare photo on its own is not supported.

	Image order: the sheets first (image 1, 2, ...), the source last. Each image is shrunk to at most 1 MP.

	## Prompt

	One sentence per character, then one sentence that names what to keep. With one sheet (image 1) and the source as image 2:

	```
	Replace the man in a green shirt in image 2 with the character from image 1. Keep everything else exactly as in image 2: the pose, expression, framing, background and lighting.
	```

	With two people in the picture and only one replaced, say where the person is and that the other stays:

	```
	Replace the woman on the left in image 2 with the character from image 1. Keep everything else exactly as in image 2: the pose, expression, framing, background and lighting, and leave the other person unchanged.
	```

	Positions are `on the left` / `on the right`, `in the middle` (three people), or `2nd from the left` and so on.
	`inference/swap.py` writes these prompts for you (`--left`, `--kept`).

	## Install

	```bash
	pip install -U torch torchvision "transformers==5.17.0" accelerate safetensors "peft==0.18.0" pillow numpy
	pip install "git+https://github.com/huggingface/diffusers.git@80c7ed262aeffbeb43ef13ae04baeb9b84515a69"
	```

	`QwenImage21Pipeline` is pinned to that diffusers commit, which is what this adapter was tested on.

	## Usage

	Command line:

	```bash
	python inference/swap.py --src scene.png --char sheet.png "man in a green shirt" --out out.png
	# two people replaced, one kept: places are 1-based, counted left to right among all three
	python inference/swap.py --src scene.png --char a.png "woman in red" --char b.png "man in a cap" --left 1 3 --kept 1
	python inference/swap.py --src scene.png --char sheet.png "man in a green shirt" --dry-run # prompt, sizes, sigmas; no GPU
	python inference/swap.py --src scene.png --char sheet.png "man in a green shirt" --turbo # sharper: 2 + 6 Turbo steps
	```

	Python:

	```python
	import torch
	from diffusers import QwenImage21Pipeline, FlowMatchEulerDiscreteScheduler
	from PIL import Image

	sched = FlowMatchEulerDiscreteScheduler.from_pretrained("Qwen/Qwen-Image-2.1", subfolder="scheduler")
	sched = FlowMatchEulerDiscreteScheduler.from_config(sched.config, shift_terminal=None)
	pipe = QwenImage21Pipeline.from_pretrained("Qwen/Qwen-Image-2.1", scheduler=sched, dtype=torch.bfloat16)
	pipe.load_lora_weights("Viggle/Viggle-Image-Swap", weight_name="viggle-image-swap-v0.1-lora.safetensors")
	pipe.to("cuda")

	sheet, src = Image.open("sheet.png").convert("RGB"), Image.open("scene.png").convert("RGB") # each at most 1 MP
	prompt = ("Replace the man in a green shirt in image 2 with the character from image 1. Keep everything else exactly as in "
	"image 2: the pose, expression, framing, background and lighting.")
	out = pipe(prompt=prompt, image=[sheet, src], num_inference_steps=6, sigmas=[1.0, 0.9375, 0.875, 0.75, 0.5, 0.25],
	true_cfg_scale=1.0, output_resolution=1024, generator=torch.Generator("cpu").manual_seed(0)).images[0]
	out.save("out.png")
	```

	The `sigmas` are passed raw; the pipeline adds its resolution-dependent shift. `shift_terminal=None` is required.
	`swap.py` also replaces the vision encoder's bf16 Conv3d patch embedding with the equal matmul, which saves about 30 s per input
	image; the output is bitwise the same.

	### Sharper output: `--turbo`

	`--turbo` runs the whole sample on Qwen-Image-2.1-Turbo's scheduler and its 8 `sample_sigmas` (from its `model_index.json`).
	The LoRA model takes the first 2 steps, which settle who is in the picture; then the transformer is swapped for
	Qwen-Image-2.1-Turbo's, which runs the other 6 and draws finer detail. The KV cache is off, since the two transformers
	cannot share one. `--turbo-model` points at another copy of Qwen-Image-2.1-Turbo. In Python, the same is:

	```python
	from diffusers import QwenImage21Transformer2DModel

	turbo = QwenImage21Transformer2DModel.from_pretrained("Qwen/Qwen-Image-2.1-Turbo", subfolder="transformer",
	torch_dtype=torch.bfloat16).to("cuda")
	pipe.scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained("Qwen/Qwen-Image-2.1-Turbo", subfolder="scheduler")
	sigmas = QwenImage21Pipeline.load_config("Qwen/Qwen-Image-2.1-Turbo")["sample_sigmas"]


	def handoff(p, i, t, kw): # after the LoRA model's 2nd step
	if i == 1:
	p.transformer = turbo
	return kw


	out = pipe(prompt=prompt, image=[sheet, src], num_inference_steps=len(sigmas), sigmas=sigmas, true_cfg_scale=1.0,
	output_resolution=1024, generator=torch.Generator("cpu").manual_seed(0), callback_on_step_end=handoff,
	use_kv_cache=False).images[0]
	```

	With only 2 steps of its own the LoRA model hands over a rough picture, so Turbo sometimes pulls back parts of the source
	person: their sunglasses, beard or clothes, and now and then the whole person. Without `--turbo` that happens less, at a
	softer finish. Try another `--seed`, or drop `--turbo`, when it does.

	### Tone fix

	The output can be 1–4 levels darker or lighter than the source over the whole frame. `swap.py` corrects this by default: per
	colour channel, it subtracts the mean offset from the source over the pixels that did not change (the background). It does
	nothing when under 30 % of the frame is unchanged. `--no-tone-fix` turns it off.

	## Cost

	Measured on one B200 with the models loaded, a 1344×768 output and one sheet, bf16: about 2.2 s per image and 46 GB peak;
	with `--turbo`, about 4.1 s and 55 GB (51.3 GiB) peak, the second transformer included.

	## Limitations

	- The person to replace is chosen by the prompt. With several people, give the position.
	- Sometimes the new character keeps the source person's clothes.
	- The pose or framing of the new character can drift from the source person's.
	- The output is 1 MP.
	- `--turbo` can bring back the source person's accessories, clothes or, rarely, the whole person (see above).

	## Example footage

	The source pictures in `examples/` are frames of third-party footage, © their respective owners, credited by YouTube channel
	and video ID. The character sheets are original.

	\| example \| source \| YouTube video \|
	\|---\|---\|---\|
	\| card-royals \| West Coast Swing Online \| `nRdO0HS-D5E` \|
	\| astronaut \| Chris King \| `A37myLTm2Vo` \|
	\| flamingos \| jtballroom \| `GAZ2LeNX3BU` \|
	\| aviatrix \| Alecia Renece The Artist \| `rHDTkqDqBp0` \|
	\| pineapple-mascot \| Massimo Nocerino \| `W1P5qiqaqCs` \|

	The video at the top (`examples/viggle-image-swap.mp4`) uses frames of films, advertisements and other third-party footage, © their respective owners. Some of its character sheets depict real public figures; they are made for this research demonstration and imply no endorsement.

	## License

	The adapter is a derivative of Qwen-Image-2.1 and is released under the [Qwen RESEARCH LICENSE AGREEMENT](LICENSE): research
	and evaluation use only. Commercial use needs a separate licence from the Qwen team (model-business@notice.qwencloud.com).
	`--turbo` also loads Qwen/Qwen-Image-2.1-Turbo, under the same licence.
	See [NOTICE](NOTICE). The inference code in `inference/` is under the Apache License 2.0 ([LICENSE-CODE](LICENSE-CODE)).