ChenDY's picture
2-step workflow: load the LoRA with ComfyUI-BypassLoRA
a8b01e4 verified
|
Raw History Blame Contribute Delete
11.3 kB
---
license: apache-2.0
language:
- en
pipeline_tag: text-to-image
library_name: diffusion-single-file
base_model:
- ChenDY/NitroFusion2
- alibaba-pai/Z-Image-Turbo-Fun-Controlnet-Union-2.1
tags:
- controlnet
- text-to-image
- one-step
- few-step
- comfyui
- nitrofusion2
---
# NitroFusion2-Controlnet-Union
[![Demo](https://img.shields.io/badge/%F0%9F%A4%97%20Demo-NitroFusion2-ff7a00?labelColor=555)](https://huggingface.co/spaces/ChenDY/NitroFusion2)
[![NitroFusion2](https://img.shields.io/badge/%F0%9F%A4%97%20Checkpoint-NitroFusion2-ffc107?labelColor=555)](https://huggingface.co/ChenDY/NitroFusion2)
[![GitHub](https://img.shields.io/badge/GitHub-NitroFusion2-24292f?logo=github&labelColor=555)](https://github.com/ChenDarYen/NitroFusion2)
[![Project Page](https://img.shields.io/badge/%F0%9F%8C%90%20Project%20Page-NitroFusion2-1f72b8?labelColor=555)](https://chendaryen.github.io/NitroFusion2.github.io/)
[![arXiv](https://img.shields.io/badge/arXiv-2611.05648-b31b1b?labelColor=555)](https://arxiv.org/abs/2611.05648)
NitroFusion2-Controlnet-Union is a finetuned version of [Z-Image-Turbo-Fun-Controlnet-Union-2.1](https://huggingface.co/alibaba-pai/Z-Image-Turbo-Fun-Controlnet-Union-2.1) (2602), adapted to [NitroFusion2](https://huggingface.co/ChenDY/NitroFusion2) for 1-step and 2-step generation. It must be used with NitroFusion2 as the base model.
Supported conditions: Canny, HED, Depth, Pose, Scribble, Gray, and inpainting.
## ComfyUI
- 1 step: [`workflows/nitrofusion2_controlnet_1step.json`](./workflows/nitrofusion2_controlnet_1step.json)
- 2 steps: [`workflows/nitrofusion2_controlnet_2step.json`](./workflows/nitrofusion2_controlnet_2step.json)
The 2-step workflow loads the LoRA with [ComfyUI-BypassLoRA](https://github.com/ChenDarYen/ComfyUI-BypassLoRA). It applies the LoRA without merging it into the weights, which makes 2-step generation 1.5–3.5× faster. The node includes the fix from [ComfyUI PR #17408](https://github.com/Comfy-Org/ComfyUI/pull/17408) until ComfyUI ships it.
## Model Card
| Name | Description |
|--|--|
| NitroFusion2-Controlnet-Union.safetensors | Union 2602 (15 control layer blocks + 2 refiner blocks) adapted to NitroFusion2 for 1–2 step inference. |
| NitroFusion2-Controlnet-Union-lite.safetensors | Lite version (3 control layer blocks + 2 refiner blocks) adapted the same way; smaller and lighter on memory, with weaker control. |
## Results
### 1 step
<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
<tr><td>Pose</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/pose2.jpg" width="100%" alt="Pose control map" /></td><td width="33%"><img src="results/pose2_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/pose2_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr>
<tr><td>Pose</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/pose3.jpg" width="100%" alt="Pose control map" /></td><td width="33%"><img src="results/pose3_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/pose3_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr>
<tr><td>Canny</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/canny.jpg" width="100%" alt="Canny control map" /></td><td width="33%"><img src="results/canny_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/canny_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr>
<tr><td>Canny</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/canny2.jpg" width="100%" alt="Canny control map" /></td><td width="33%"><img src="results/canny2_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/canny2_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr>
<tr><td>HED</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/hed.jpg" width="100%" alt="HED control map" /></td><td width="33%"><img src="results/hed_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/hed_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr>
<tr><td>HED</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/hed2.jpg" width="100%" alt="HED control map" /></td><td width="33%"><img src="results/hed2_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/hed2_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr>
<tr><td>Depth</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/depth.jpg" width="100%" alt="Depth control map" /></td><td width="33%"><img src="results/depth_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/depth_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr>
<tr><td>Depth</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/depth2.jpg" width="100%" alt="Depth control map" /></td><td width="33%"><img src="results/depth2_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/depth2_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr>
<tr><td>Pose + Inpaint</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/pose_inpaint.jpg" width="100%" alt="Source image, masked image and pose map" /></td><td width="33%"><img src="results/pose_inpaint_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/pose_inpaint_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr>
</table>
### 2 steps
<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
<tr><td>Pose</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/pose2.jpg" width="100%" alt="Pose control map" /></td><td width="33%"><img src="results/2step/pose2_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/pose2_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr>
<tr><td>Pose</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/pose3.jpg" width="100%" alt="Pose control map" /></td><td width="33%"><img src="results/2step/pose3_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/pose3_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr>
<tr><td>Canny</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/canny.jpg" width="100%" alt="Canny control map" /></td><td width="33%"><img src="results/2step/canny_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/canny_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr>
<tr><td>Canny</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/canny2.jpg" width="100%" alt="Canny control map" /></td><td width="33%"><img src="results/2step/canny2_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/canny2_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr>
<tr><td>HED</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/hed.jpg" width="100%" alt="HED control map" /></td><td width="33%"><img src="results/2step/hed_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/hed_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr>
<tr><td>HED</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/hed2.jpg" width="100%" alt="HED control map" /></td><td width="33%"><img src="results/2step/hed2_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/hed2_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr>
<tr><td>Depth</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/depth.jpg" width="100%" alt="Depth control map" /></td><td width="33%"><img src="results/2step/depth_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/depth_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr>
<tr><td>Depth</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/depth2.jpg" width="100%" alt="Depth control map" /></td><td width="33%"><img src="results/2step/depth2_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/depth2_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr>
<tr><td>Pose + Inpaint</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr>
<tr><td width="33%"><img src="asset/pose_inpaint.jpg" width="100%" alt="Source image, masked image and pose map" /></td><td width="33%"><img src="results/2step/pose_inpaint_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/pose_inpaint_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr>
</table>
## Citation
If you find NitroFusion2 useful in your research, please cite our paper:
```bib
@article{chen2026nitrofusion2,
title={NitroFusion2: Bringing Adversarial Loss to the Text Stream of Multimodal Diffusion Transformers},
author={Chen, Dar-Yen and Song, Yi-Zhe},
journal={arXiv preprint arXiv:2611.05648},
year={2026}
}
```
## Acknowledgements
This model is adapted from [alibaba-pai/Z-Image-Turbo-Fun-Controlnet-Union-2.1](https://huggingface.co/alibaba-pai/Z-Image-Turbo-Fun-Controlnet-Union-2.1). Many thanks to the authors for releasing the ControlNet.
## License
[Apache 2.0](./LICENSE)