File size: 11,301 Bytes
2647be6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
a8b01e4
 
2647be6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
---
license: apache-2.0
language:
  - en
pipeline_tag: text-to-image
library_name: diffusion-single-file
base_model:
  - ChenDY/NitroFusion2
  - alibaba-pai/Z-Image-Turbo-Fun-Controlnet-Union-2.1
tags:
  - controlnet
  - text-to-image
  - one-step
  - few-step
  - comfyui
  - nitrofusion2
---

# NitroFusion2-Controlnet-Union

[![Demo](https://img.shields.io/badge/%F0%9F%A4%97%20Demo-NitroFusion2-ff7a00?labelColor=555)](https://huggingface.co/spaces/ChenDY/NitroFusion2)
[![NitroFusion2](https://img.shields.io/badge/%F0%9F%A4%97%20Checkpoint-NitroFusion2-ffc107?labelColor=555)](https://huggingface.co/ChenDY/NitroFusion2)
[![GitHub](https://img.shields.io/badge/GitHub-NitroFusion2-24292f?logo=github&labelColor=555)](https://github.com/ChenDarYen/NitroFusion2)
[![Project Page](https://img.shields.io/badge/%F0%9F%8C%90%20Project%20Page-NitroFusion2-1f72b8?labelColor=555)](https://chendaryen.github.io/NitroFusion2.github.io/)
[![arXiv](https://img.shields.io/badge/arXiv-2611.05648-b31b1b?labelColor=555)](https://arxiv.org/abs/2611.05648)

NitroFusion2-Controlnet-Union is a finetuned version of [Z-Image-Turbo-Fun-Controlnet-Union-2.1](https://huggingface.co/alibaba-pai/Z-Image-Turbo-Fun-Controlnet-Union-2.1) (2602), adapted to [NitroFusion2](https://huggingface.co/ChenDY/NitroFusion2) for 1-step and 2-step generation. It must be used with NitroFusion2 as the base model.

Supported conditions: Canny, HED, Depth, Pose, Scribble, Gray, and inpainting.

## ComfyUI

- 1 step: [`workflows/nitrofusion2_controlnet_1step.json`](./workflows/nitrofusion2_controlnet_1step.json)
- 2 steps: [`workflows/nitrofusion2_controlnet_2step.json`](./workflows/nitrofusion2_controlnet_2step.json)

The 2-step workflow loads the LoRA with [ComfyUI-BypassLoRA](https://github.com/ChenDarYen/ComfyUI-BypassLoRA). It applies the LoRA without merging it into the weights, which makes 2-step generation 1.5–3.5× faster. The node includes the fix from [ComfyUI PR #17408](https://github.com/Comfy-Org/ComfyUI/pull/17408) until ComfyUI ships it.

## Model Card

| Name | Description |
|--|--|
| NitroFusion2-Controlnet-Union.safetensors | Union 2602 (15 control layer blocks + 2 refiner blocks) adapted to NitroFusion2 for 1–2 step inference. |
| NitroFusion2-Controlnet-Union-lite.safetensors | Lite version (3 control layer blocks + 2 refiner blocks) adapted the same way; smaller and lighter on memory, with weaker control. |

## Results

### 1 step

<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
<tr><td>Pose</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/pose2.jpg" width="100%" alt="Pose control map" /></td><td width="33%"><img src="results/pose2_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/pose2_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr>
<tr><td>Pose</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/pose3.jpg" width="100%" alt="Pose control map" /></td><td width="33%"><img src="results/pose3_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/pose3_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr>
<tr><td>Canny</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/canny.jpg" width="100%" alt="Canny control map" /></td><td width="33%"><img src="results/canny_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/canny_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr>
<tr><td>Canny</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/canny2.jpg" width="100%" alt="Canny control map" /></td><td width="33%"><img src="results/canny2_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/canny2_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr>
<tr><td>HED</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/hed.jpg" width="100%" alt="HED control map" /></td><td width="33%"><img src="results/hed_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/hed_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr>
<tr><td>HED</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/hed2.jpg" width="100%" alt="HED control map" /></td><td width="33%"><img src="results/hed2_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/hed2_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr>
<tr><td>Depth</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/depth.jpg" width="100%" alt="Depth control map" /></td><td width="33%"><img src="results/depth_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/depth_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr>
<tr><td>Depth</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/depth2.jpg" width="100%" alt="Depth control map" /></td><td width="33%"><img src="results/depth2_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/depth2_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr>
<tr><td>Pose + Inpaint</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/pose_inpaint.jpg" width="100%" alt="Source image, masked image and pose map" /></td><td width="33%"><img src="results/pose_inpaint_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/pose_inpaint_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr>
</table>

### 2 steps

<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
<tr><td>Pose</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/pose2.jpg" width="100%" alt="Pose control map" /></td><td width="33%"><img src="results/2step/pose2_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/pose2_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr>
<tr><td>Pose</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/pose3.jpg" width="100%" alt="Pose control map" /></td><td width="33%"><img src="results/2step/pose3_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/pose3_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr>
<tr><td>Canny</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/canny.jpg" width="100%" alt="Canny control map" /></td><td width="33%"><img src="results/2step/canny_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/canny_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr>
<tr><td>Canny</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/canny2.jpg" width="100%" alt="Canny control map" /></td><td width="33%"><img src="results/2step/canny2_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/canny2_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr>
<tr><td>HED</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/hed.jpg" width="100%" alt="HED control map" /></td><td width="33%"><img src="results/2step/hed_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/hed_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr>
<tr><td>HED</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/hed2.jpg" width="100%" alt="HED control map" /></td><td width="33%"><img src="results/2step/hed2_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/hed2_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr>
<tr><td>Depth</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/depth.jpg" width="100%" alt="Depth control map" /></td><td width="33%"><img src="results/2step/depth_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/depth_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr>
<tr><td>Depth</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/depth2.jpg" width="100%" alt="Depth control map" /></td><td width="33%"><img src="results/2step/depth2_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/depth2_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr>
<tr><td>Pose + Inpaint</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/pose_inpaint.jpg" width="100%" alt="Source image, masked image and pose map" /></td><td width="33%"><img src="results/2step/pose_inpaint_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/pose_inpaint_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr>
</table>

## Citation

If you find NitroFusion2 useful in your research, please cite our paper:

```bib
@article{chen2026nitrofusion2,
    title={NitroFusion2: Bringing Adversarial Loss to the Text Stream of Multimodal Diffusion Transformers},
    author={Chen, Dar-Yen and Song, Yi-Zhe},
    journal={arXiv preprint arXiv:2611.05648},
    year={2026}
}
```

## Acknowledgements

This model is adapted from [alibaba-pai/Z-Image-Turbo-Fun-Controlnet-Union-2.1](https://huggingface.co/alibaba-pai/Z-Image-Turbo-Fun-Controlnet-Union-2.1). Many thanks to the authors for releasing the ControlNet.

## License

[Apache 2.0](./LICENSE)