Instructions to use ChenDY/NitroFusion2-Controlnet-Union with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusion Single File
How to use ChenDY/NitroFusion2-Controlnet-Union with Diffusion Single File:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
|
Download README.md from ChenDY/NitroFusion2-Controlnet-Union: direct link, hf CLI and curl.
- Browser
- Download file 11.3 kB
-
https://huggingface.co/ChenDY/NitroFusion2-Controlnet-Union/resolve/main/README.md
- Command line
-
hf download hf://ChenDY/NitroFusion2-Controlnet-Union/README.md
-
curl -L -o README.md https://huggingface.co/ChenDY/NitroFusion2-Controlnet-Union/resolve/main/README.md
11.3 kB
| license: apache-2.0 | |
| language: | |
| - en | |
| pipeline_tag: text-to-image | |
| library_name: diffusion-single-file | |
| base_model: | |
| - ChenDY/NitroFusion2 | |
| - alibaba-pai/Z-Image-Turbo-Fun-Controlnet-Union-2.1 | |
| tags: | |
| - controlnet | |
| - text-to-image | |
| - one-step | |
| - few-step | |
| - comfyui | |
| - nitrofusion2 | |
| # NitroFusion2-Controlnet-Union | |
| [](https://huggingface.co/spaces/ChenDY/NitroFusion2) | |
| [](https://huggingface.co/ChenDY/NitroFusion2) | |
| [](https://github.com/ChenDarYen/NitroFusion2) | |
| [](https://chendaryen.github.io/NitroFusion2.github.io/) | |
| [](https://arxiv.org/abs/2611.05648) | |
| NitroFusion2-Controlnet-Union is a finetuned version of [Z-Image-Turbo-Fun-Controlnet-Union-2.1](https://huggingface.co/alibaba-pai/Z-Image-Turbo-Fun-Controlnet-Union-2.1) (2602), adapted to [NitroFusion2](https://huggingface.co/ChenDY/NitroFusion2) for 1-step and 2-step generation. It must be used with NitroFusion2 as the base model. | |
| Supported conditions: Canny, HED, Depth, Pose, Scribble, Gray, and inpainting. | |
| ## ComfyUI | |
| - 1 step: [`workflows/nitrofusion2_controlnet_1step.json`](./workflows/nitrofusion2_controlnet_1step.json) | |
| - 2 steps: [`workflows/nitrofusion2_controlnet_2step.json`](./workflows/nitrofusion2_controlnet_2step.json) | |
| The 2-step workflow loads the LoRA with [ComfyUI-BypassLoRA](https://github.com/ChenDarYen/ComfyUI-BypassLoRA). It applies the LoRA without merging it into the weights, which makes 2-step generation 1.5–3.5× faster. The node includes the fix from [ComfyUI PR #17408](https://github.com/Comfy-Org/ComfyUI/pull/17408) until ComfyUI ships it. | |
| ## Model Card | |
| | Name | Description | | |
| |--|--| | |
| | NitroFusion2-Controlnet-Union.safetensors | Union 2602 (15 control layer blocks + 2 refiner blocks) adapted to NitroFusion2 for 1–2 step inference. | | |
| | NitroFusion2-Controlnet-Union-lite.safetensors | Lite version (3 control layer blocks + 2 refiner blocks) adapted the same way; smaller and lighter on memory, with weaker control. | | |
| ## Results | |
| ### 1 step | |
| <table border="0" style="width: 100%; text-align: left; margin-top: 20px;"> | |
| <tr><td>Pose</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr> | |
| <tr><td width="33%"><img src="asset/pose2.jpg" width="100%" alt="Pose control map" /></td><td width="33%"><img src="results/pose2_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/pose2_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr> | |
| <tr><td>Pose</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr> | |
| <tr><td width="33%"><img src="asset/pose3.jpg" width="100%" alt="Pose control map" /></td><td width="33%"><img src="results/pose3_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/pose3_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr> | |
| <tr><td>Canny</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr> | |
| <tr><td width="33%"><img src="asset/canny.jpg" width="100%" alt="Canny control map" /></td><td width="33%"><img src="results/canny_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/canny_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr> | |
| <tr><td>Canny</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr> | |
| <tr><td width="33%"><img src="asset/canny2.jpg" width="100%" alt="Canny control map" /></td><td width="33%"><img src="results/canny2_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/canny2_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr> | |
| <tr><td>HED</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr> | |
| <tr><td width="33%"><img src="asset/hed.jpg" width="100%" alt="HED control map" /></td><td width="33%"><img src="results/hed_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/hed_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr> | |
| <tr><td>HED</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr> | |
| <tr><td width="33%"><img src="asset/hed2.jpg" width="100%" alt="HED control map" /></td><td width="33%"><img src="results/hed2_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/hed2_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr> | |
| <tr><td>Depth</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr> | |
| <tr><td width="33%"><img src="asset/depth.jpg" width="100%" alt="Depth control map" /></td><td width="33%"><img src="results/depth_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/depth_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr> | |
| <tr><td>Depth</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr> | |
| <tr><td width="33%"><img src="asset/depth2.jpg" width="100%" alt="Depth control map" /></td><td width="33%"><img src="results/depth2_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/depth2_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr> | |
| <tr><td>Pose + Inpaint</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr> | |
| <tr><td width="33%"><img src="asset/pose_inpaint.jpg" width="100%" alt="Source image, masked image and pose map" /></td><td width="33%"><img src="results/pose_inpaint_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 1 step" /></td><td width="33%"><img src="results/pose_inpaint_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 1 step" /></td></tr> | |
| </table> | |
| ### 2 steps | |
| <table border="0" style="width: 100%; text-align: left; margin-top: 20px;"> | |
| <tr><td>Pose</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr> | |
| <tr><td width="33%"><img src="asset/pose2.jpg" width="100%" alt="Pose control map" /></td><td width="33%"><img src="results/2step/pose2_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/pose2_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr> | |
| <tr><td>Pose</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr> | |
| <tr><td width="33%"><img src="asset/pose3.jpg" width="100%" alt="Pose control map" /></td><td width="33%"><img src="results/2step/pose3_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/pose3_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr> | |
| <tr><td>Canny</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr> | |
| <tr><td width="33%"><img src="asset/canny.jpg" width="100%" alt="Canny control map" /></td><td width="33%"><img src="results/2step/canny_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/canny_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr> | |
| <tr><td>Canny</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr> | |
| <tr><td width="33%"><img src="asset/canny2.jpg" width="100%" alt="Canny control map" /></td><td width="33%"><img src="results/2step/canny2_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/canny2_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr> | |
| <tr><td>HED</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr> | |
| <tr><td width="33%"><img src="asset/hed.jpg" width="100%" alt="HED control map" /></td><td width="33%"><img src="results/2step/hed_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/hed_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr> | |
| <tr><td>HED</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr> | |
| <tr><td width="33%"><img src="asset/hed2.jpg" width="100%" alt="HED control map" /></td><td width="33%"><img src="results/2step/hed2_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/hed2_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr> | |
| <tr><td>Depth</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr> | |
| <tr><td width="33%"><img src="asset/depth.jpg" width="100%" alt="Depth control map" /></td><td width="33%"><img src="results/2step/depth_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/depth_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr> | |
| <tr><td>Depth</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr> | |
| <tr><td width="33%"><img src="asset/depth2.jpg" width="100%" alt="Depth control map" /></td><td width="33%"><img src="results/2step/depth2_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/depth2_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr> | |
| <tr><td>Pose + Inpaint</td><td>Union 2.1-2602 + NitroFusion2</td><td><b>NitroFusion2-Controlnet-Union</b></td></tr> | |
| <tr><td width="33%"><img src="asset/pose_inpaint.jpg" width="100%" alt="Source image, masked image and pose map" /></td><td width="33%"><img src="results/2step/pose_inpaint_original.jpg" width="100%" alt="Union 2.1-2602 + NitroFusion2, 2 steps" /></td><td width="33%"><img src="results/2step/pose_inpaint_ours.jpg" width="100%" alt="NitroFusion2-Controlnet-Union, 2 steps" /></td></tr> | |
| </table> | |
| ## Citation | |
| If you find NitroFusion2 useful in your research, please cite our paper: | |
| ```bib | |
| @article{chen2026nitrofusion2, | |
| title={NitroFusion2: Bringing Adversarial Loss to the Text Stream of Multimodal Diffusion Transformers}, | |
| author={Chen, Dar-Yen and Song, Yi-Zhe}, | |
| journal={arXiv preprint arXiv:2611.05648}, | |
| year={2026} | |
| } | |
| ``` | |
| ## Acknowledgements | |
| This model is adapted from [alibaba-pai/Z-Image-Turbo-Fun-Controlnet-Union-2.1](https://huggingface.co/alibaba-pai/Z-Image-Turbo-Fun-Controlnet-Union-2.1). Many thanks to the authors for releasing the ControlNet. | |
| ## License | |
| [Apache 2.0](./LICENSE) | |