ChenDY's picture
2-step workflow: load the LoRA with ComfyUI-BypassLoRA
a8b01e4 verified
|
Raw History Blame Contribute Delete
11.3 kB
metadata
license: apache-2.0
language:
  - en
pipeline_tag: text-to-image
library_name: diffusion-single-file
base_model:
  - ChenDY/NitroFusion2
  - alibaba-pai/Z-Image-Turbo-Fun-Controlnet-Union-2.1
tags:
  - controlnet
  - text-to-image
  - one-step
  - few-step
  - comfyui
  - nitrofusion2

NitroFusion2-Controlnet-Union

Demo NitroFusion2 GitHub Project Page arXiv

NitroFusion2-Controlnet-Union is a finetuned version of Z-Image-Turbo-Fun-Controlnet-Union-2.1 (2602), adapted to NitroFusion2 for 1-step and 2-step generation. It must be used with NitroFusion2 as the base model.

Supported conditions: Canny, HED, Depth, Pose, Scribble, Gray, and inpainting.

ComfyUI

The 2-step workflow loads the LoRA with ComfyUI-BypassLoRA. It applies the LoRA without merging it into the weights, which makes 2-step generation 1.5–3.5× faster. The node includes the fix from ComfyUI PR #17408 until ComfyUI ships it.

Model Card

Name Description
NitroFusion2-Controlnet-Union.safetensors Union 2602 (15 control layer blocks + 2 refiner blocks) adapted to NitroFusion2 for 1–2 step inference.
NitroFusion2-Controlnet-Union-lite.safetensors Lite version (3 control layer blocks + 2 refiner blocks) adapted the same way; smaller and lighter on memory, with weaker control.

Results

1 step

PoseUnion 2.1-2602 + NitroFusion2NitroFusion2-Controlnet-Union
Pose control mapUnion 2.1-2602 + NitroFusion2, 1 stepNitroFusion2-Controlnet-Union, 1 step
PoseUnion 2.1-2602 + NitroFusion2NitroFusion2-Controlnet-Union
Pose control mapUnion 2.1-2602 + NitroFusion2, 1 stepNitroFusion2-Controlnet-Union, 1 step
CannyUnion 2.1-2602 + NitroFusion2NitroFusion2-Controlnet-Union
Canny control mapUnion 2.1-2602 + NitroFusion2, 1 stepNitroFusion2-Controlnet-Union, 1 step
CannyUnion 2.1-2602 + NitroFusion2NitroFusion2-Controlnet-Union
Canny control mapUnion 2.1-2602 + NitroFusion2, 1 stepNitroFusion2-Controlnet-Union, 1 step
HEDUnion 2.1-2602 + NitroFusion2NitroFusion2-Controlnet-Union
HED control mapUnion 2.1-2602 + NitroFusion2, 1 stepNitroFusion2-Controlnet-Union, 1 step
HEDUnion 2.1-2602 + NitroFusion2NitroFusion2-Controlnet-Union
HED control mapUnion 2.1-2602 + NitroFusion2, 1 stepNitroFusion2-Controlnet-Union, 1 step
DepthUnion 2.1-2602 + NitroFusion2NitroFusion2-Controlnet-Union
Depth control mapUnion 2.1-2602 + NitroFusion2, 1 stepNitroFusion2-Controlnet-Union, 1 step
DepthUnion 2.1-2602 + NitroFusion2NitroFusion2-Controlnet-Union
Depth control mapUnion 2.1-2602 + NitroFusion2, 1 stepNitroFusion2-Controlnet-Union, 1 step
Pose + InpaintUnion 2.1-2602 + NitroFusion2NitroFusion2-Controlnet-Union
Source image, masked image and pose mapUnion 2.1-2602 + NitroFusion2, 1 stepNitroFusion2-Controlnet-Union, 1 step

2 steps

PoseUnion 2.1-2602 + NitroFusion2NitroFusion2-Controlnet-Union
Pose control mapUnion 2.1-2602 + NitroFusion2, 2 stepsNitroFusion2-Controlnet-Union, 2 steps
PoseUnion 2.1-2602 + NitroFusion2NitroFusion2-Controlnet-Union
Pose control mapUnion 2.1-2602 + NitroFusion2, 2 stepsNitroFusion2-Controlnet-Union, 2 steps
CannyUnion 2.1-2602 + NitroFusion2NitroFusion2-Controlnet-Union
Canny control mapUnion 2.1-2602 + NitroFusion2, 2 stepsNitroFusion2-Controlnet-Union, 2 steps
CannyUnion 2.1-2602 + NitroFusion2NitroFusion2-Controlnet-Union
Canny control mapUnion 2.1-2602 + NitroFusion2, 2 stepsNitroFusion2-Controlnet-Union, 2 steps
HEDUnion 2.1-2602 + NitroFusion2NitroFusion2-Controlnet-Union
HED control mapUnion 2.1-2602 + NitroFusion2, 2 stepsNitroFusion2-Controlnet-Union, 2 steps
HEDUnion 2.1-2602 + NitroFusion2NitroFusion2-Controlnet-Union
HED control mapUnion 2.1-2602 + NitroFusion2, 2 stepsNitroFusion2-Controlnet-Union, 2 steps
DepthUnion 2.1-2602 + NitroFusion2NitroFusion2-Controlnet-Union
Depth control mapUnion 2.1-2602 + NitroFusion2, 2 stepsNitroFusion2-Controlnet-Union, 2 steps
DepthUnion 2.1-2602 + NitroFusion2NitroFusion2-Controlnet-Union
Depth control mapUnion 2.1-2602 + NitroFusion2, 2 stepsNitroFusion2-Controlnet-Union, 2 steps
Pose + InpaintUnion 2.1-2602 + NitroFusion2NitroFusion2-Controlnet-Union
Source image, masked image and pose mapUnion 2.1-2602 + NitroFusion2, 2 stepsNitroFusion2-Controlnet-Union, 2 steps

Citation

If you find NitroFusion2 useful in your research, please cite our paper:

@article{chen2026nitrofusion2,
    title={NitroFusion2: Bringing Adversarial Loss to the Text Stream of Multimodal Diffusion Transformers},
    author={Chen, Dar-Yen and Song, Yi-Zhe},
    journal={arXiv preprint arXiv:2611.05648},
    year={2026}
}

Acknowledgements

This model is adapted from alibaba-pai/Z-Image-Turbo-Fun-Controlnet-Union-2.1. Many thanks to the authors for releasing the ControlNet.

License

Apache 2.0