| ---
|
| license: apache-2.0
|
| base_model: OpenMOSS-Team/MOSS-Image-SFT
|
| base_model_relation: adapter
|
| language:
|
| - en
|
| pipeline_tag: image-to-image
|
| tags:
|
| - moss-image
|
| - controlnet
|
| - lora
|
| - comfyui
|
| - segmentation
|
| library_name: diffusers
|
| ---
|
|
|
| # MOSS-Image-SFT — Segmentation Control LoRA
|
|
|
| ## Short description
|
|
|
| A segmentation Control LoRA for **MOSS-Image-SFT**. Feed a flat-color region map. The model follows the region boundaries, and the prompt decides what each region becomes.
|
|
|
| This is a ControlNet-style **edit LoRA**, not the official MOSS color/depth concat adapter. Load it with a LoRA loader and pass one control image through **Text Encode MOSS-Image Edit**.
|
|
|
| Selected checkpoint: training step **4500**.
|
|
|
| ## Files
|
|
|
| - **Weights:** [moss_image_sft_segmentation_control_lora.safetensors](https://huggingface.co/thedeoxen/MOSS-Image-segmentation-controlnet/blob/main/moss_image_sft_segmentation_control_lora.safetensors)
|
| - **ComfyUI workflow:** [moss_image_segmentation_control.json](https://huggingface.co/thedeoxen/MOSS-Image-segmentation-controlnet/blob/main/moss_image_segmentation_control.json)
|
| - **Repo:** [thedeoxen/MOSS-Image-segmentation-controlnet](https://huggingface.co/thedeoxen/MOSS-Image-segmentation-controlnet)
|
|
|
| ## Examples
|
|
|
| Each preview is one combined image, left to right: **control**, then three prompts.
|
|
|
| | Control → Results |
|
| | --- |
|
| |  |
|
| |  |
|
| |  |
|
|
|
|
|
| ## Extended description
|
|
|
| Follows flat color regions. Boundaries and the big masses stay, while the prompt fills them with a new subject and style. Appearance, materials, clothing and the world come from the text prompt.
|
|
|
| The LoRA is rank 64 on linear layers and rank 16 on convolutions. It was trained as an edit adapter: the control image is the reference, and the prompt describes the picture to generate.
|
|
|
| ## How to use
|
|
|
| 1. Load **MOSS-Image-SFT**: UNet `moss_image_sft_int8_convrot.safetensors`, text encoder `qwen3.5_9b_w4a8_gptq.safetensors` with type `moss_image`, VAE `flux2-vae_small.safetensors`.
|
| 2. Load this LoRA on the UNet at strength **1.0**.
|
| 3. Build the control map and pass it as the single reference image in **Text Encode MOSS-Image Edit**.
|
| 4. Set **edit format** to `ai-toolkit` on both the positive and the negative encode. This LoRA was trained with kv cache on.
|
| 5. Sample with **30 steps**, **CFG 4**, `euler` + `simple`.
|
|
|
| No trigger word. The control image sets the structure. The prompt sets identity, materials, clothing and the world.
|
|
|
| ## ComfyUI
|
|
|
| The workflow in this repo loads a photo, builds the control map, encodes it with the prompt, and samples.
|
|
|
| The included workflow uses the **UniFormer** semantic segmentor. The training maps are fully filled flat colors, with no large black unlabeled areas. The colors are region ids, not the colors you want in the result.
|
|
|
| Also uses the built-in **Text Encode MOSS-Image Edit** node. Set the edit format to `ai-toolkit`.
|
|
|
| ### Base model
|
|
|
| Trained on and recommended with **[OpenMOSS-Team/MOSS-Image-SFT](https://huggingface.co/OpenMOSS-Team/MOSS-Image-SFT)**.
|
|
|
| Typical settings from the included workflow: **30 steps**, **CFG 4**, `euler` + `simple`, LoRA strength **1.0**.
|
|
|
| Negative prompt used in training samples:
|
|
|
| ```
|
| nsfw, worst quality, bad anatomy, watermark, jpeg artifact
|
| ```
|
|
|
| ### Example prompt
|
|
|
| ```
|
| Full-body photograph of a circus ringmaster in scarlet tails, gold braid and a tall black hat, standing three-quarter view in front of a painted circus tent. One arm raised high, the other holding a slender black cane down at the side. Confetti, saturated color, entire body in frame.
|
| ```
|
|
|
| ## What it does
|
|
|
| - Follows flat color regions. Boundaries and the big masses stay, while the prompt fills them with a new subject and style.
|
| - Lets the prompt define identity, materials, clothing and style.
|
| - Works as a single-image control adapter for MOSS-Image-SFT.
|
|
|
| ## Tips
|
|
|
| - Flat, solid colors. Gradients and textures in the control map confuse the regions.
|
| - Do not expect a region color to become that color in the picture. Say the material in the prompt.
|
| - UniFormer maps match these weights better than a segmentor that leaves the background black.
|
| - Lower the LoRA strength toward 0.7 if the control is too rigid. Strength 1.0 is the trained value.
|
|
|
| ## Use cases
|
|
|
| - Blocking a scene into regions, then restyling each mass from the prompt.
|
| - Keeping a layout of people, furniture and background while changing the render.
|
| - Color-script layouts where only the shapes are decided.
|
|
|
