Add control LoRA weights, ComfyUI workflow and model card

Weights and workflow. Example images will be added in a follow-up commit.

README.md

@@ -11,12 +11,86 @@ tags:
 - lora
 - comfyui
 - segmentation
+library_name: diffusers
 ---
 
 # MOSS-Image-SFT — Segmentation Control LoRA
 
-Uses a flat-color segmentation map to guide region boundaries and scene composition. The prompt supplies the content, appearance and style of the regions.
+## Short description
 
-This is a ControlNet-style **edit LoRA** for MOSS-Image-SFT, loaded with a LoRA loader and conditioned with a single control image through the MOSS edit encoder.
+A segmentation Control LoRA for **MOSS-Image-SFT**. Feed a flat-color region map. The model follows the region boundaries, and the prompt decides what each region becomes.
 
-> Private preparation draft. Checkpoint selection, new examples and workflow execution checks are in progress. No release weights or completed validation are claimed by this draft.
+This is a ControlNet-style **edit LoRA**, not the official MOSS color/depth concat adapter. Load it with a LoRA loader and pass one control image through **Text Encode MOSS-Image Edit**.
+
+Selected checkpoint: training step **4500**.
+
+## Files
+
+- **Weights:** [moss_image_sft_segmentation_control_lora.safetensors](https://huggingface.co/thedeoxen/MOSS-Image-segmentation-controlnet/blob/main/moss_image_sft_segmentation_control_lora.safetensors)
+- **ComfyUI workflow:** [moss_image_segmentation_control.json](https://huggingface.co/thedeoxen/MOSS-Image-segmentation-controlnet/blob/main/moss_image_segmentation_control.json)
+- **Repo:** [thedeoxen/MOSS-Image-segmentation-controlnet](https://huggingface.co/thedeoxen/MOSS-Image-segmentation-controlnet)
+
+## Examples
+
+Example strips are on the way. Each strip is one control image and three generations from different prompts, in the same layout as a Control → Results row. The images will be added to `images/` and embedded here.
+
+## Extended description
+
+Follows flat color regions. Boundaries and the big masses stay, while the prompt fills them with a new subject and style. Appearance, materials, clothing and the world come from the text prompt.
+
+The LoRA is rank 64 on linear layers and rank 16 on convolutions. It was trained as an edit adapter: the control image is the reference, and the prompt describes the picture to generate.
+
+## How to use
+
+1. Load **MOSS-Image-SFT**: UNet `moss_image_sft_int8_convrot.safetensors`, text encoder `qwen3.5_9b_int8_convrot.safetensors` with type `moss_image`, VAE `flux2-vae.safetensors`.
+2. Load this LoRA on the UNet at strength **1.0**.
+3. Build the control map and pass it as the single reference image in **Text Encode MOSS-Image Edit**.
+4. Set **edit format** to `ai-toolkit` on both the positive and the negative encode. This LoRA was trained with kv cache on.
+5. Sample with **30 steps**, **CFG 4**, `euler` + `normal`.
+
+No trigger word. The control image sets the structure. The prompt sets identity, materials, clothing and the world.
+
+## ComfyUI
+
+The workflow in this repo loads a photo, builds the control map, encodes it with the prompt, and samples.
+
+The included workflow uses the **UniFormer** semantic segmentor. The training maps are fully filled flat colors, with no large black unlabeled areas. The colors are region ids, not the colors you want in the result.
+
+Also uses the built-in **Text Encode MOSS-Image Edit** node. Set the edit format to `ai-toolkit`.
+
+### Base model
+
+Trained on and recommended with **[OpenMOSS-Team/MOSS-Image-SFT](https://huggingface.co/OpenMOSS-Team/MOSS-Image-SFT)**.
+
+Typical settings from the included workflow: **30 steps**, **CFG 4**, `euler` + `normal`, LoRA strength **1.0**.
+
+Negative prompt used in training samples:
+
+```
+nsfw, worst quality, bad anatomy, watermark, jpeg artifact
+```
+
+### Example prompt
+
+```
+A chrome pitcher tilts and pours a ribbon of dark chocolate over a tall cake in a pink neon diner. One sugar crystal hangs in mid-air beside the stream. Glossy chrome, wet highlights, cinematic photograph.
+```
+
+## What it does
+
+- Follows flat color regions. Boundaries and the big masses stay, while the prompt fills them with a new subject and style.
+- Lets the prompt define identity, materials, clothing and style.
+- Works as a single-image control adapter for MOSS-Image-SFT.
+
+## Tips
+
+- Flat, solid colors. Gradients and textures in the control map confuse the regions.
+- Do not expect a region color to become that color in the picture. Say the material in the prompt.
+- UniFormer maps match these weights better than a segmentor that leaves the background black.
+- Lower the LoRA strength toward 0.7 if the control is too rigid. Strength 1.0 is the trained value.
+
+## Use cases
+
+- Blocking a scene into regions, then restyling each mass from the prompt.
+- Keeping a layout of people, furniture and background while changing the render.
+- Color-script layouts where only the shapes are decided.

moss_image_segmentation_control.json

@@ -0,0 +1,799 @@
+{
+  "id": "moss-image-segmentation-control",
+  "revision": 0,
+  "last_node_id": 14,
+  "last_link_id": 19,
+  "nodes": [
+    {
+      "id": 1,
+      "type": "LoadImage",
+      "pos": [
+        40,
+        80
+      ],
+      "size": [
+        360,
+        420
+      ],
+      "flags": {},
+      "order": 1,
+      "mode": 0,
+      "inputs": [],
+      "outputs": [
+        {
+          "name": "IMAGE",
+          "type": "IMAGE",
+          "links": [
+            1
+          ]
+        },
+        {
+          "name": "MASK",
+          "type": "MASK",
+          "links": null
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "LoadImage"
+      },
+      "widgets_values": [
+        "example.png",
+        "image"
+      ],
+      "widgets_values_named": {
+        "image": "example.png",
+        "upload": "image"
+      }
+    },
+    {
+      "id": 2,
+      "type": "UniFormer-SemSegPreprocessor",
+      "pos": [
+        460,
+        80
+      ],
+      "size": [
+        320,
+        250
+      ],
+      "flags": {},
+      "order": 2,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "image",
+          "type": "IMAGE",
+          "link": 1
+        }
+      ],
+      "outputs": [
+        {
+          "name": "IMAGE",
+          "type": "IMAGE",
+          "links": [
+            2,
+            18,
+            19,
+            10
+          ]
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "UniFormer-SemSegPreprocessor"
+      },
+      "widgets_values": [
+        1024
+      ],
+      "widgets_values_named": {
+        "resolution": 1024
+      },
+      "title": "segmentation control"
+    },
+    {
+      "id": 3,
+      "type": "UNETLoader",
+      "pos": [
+        40,
+        560
+      ],
+      "size": [
+        420,
+        90
+      ],
+      "flags": {},
+      "order": 3,
+      "mode": 0,
+      "inputs": [],
+      "outputs": [
+        {
+          "name": "MODEL",
+          "type": "MODEL",
+          "links": [
+            3
+          ]
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "UNETLoader"
+      },
+      "widgets_values": [
+        "moss_image_sft_int8_convrot.safetensors",
+        "default"
+      ],
+      "widgets_values_named": {
+        "unet_name": "moss_image_sft_int8_convrot.safetensors",
+        "weight_dtype": "default"
+      }
+    },
+    {
+      "id": 4,
+      "type": "LoraLoaderModelOnly",
+      "pos": [
+        500,
+        560
+      ],
+      "size": [
+        420,
+        110
+      ],
+      "flags": {},
+      "order": 4,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "model",
+          "type": "MODEL",
+          "link": 3
+        }
+      ],
+      "outputs": [
+        {
+          "name": "MODEL",
+          "type": "MODEL",
+          "links": [
+            4
+          ]
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "LoraLoaderModelOnly"
+      },
+      "widgets_values": [
+        "moss_image_sft_segmentation_control_lora.safetensors",
+        1
+      ],
+      "widgets_values_named": {
+        "lora_name": "moss_image_sft_segmentation_control_lora.safetensors",
+        "strength_model": 1
+      },
+      "title": "segmentation control LoRA"
+    },
+    {
+      "id": 5,
+      "type": "CLIPLoader",
+      "pos": [
+        40,
+        700
+      ],
+      "size": [
+        420,
+        130
+      ],
+      "flags": {},
+      "order": 5,
+      "mode": 0,
+      "inputs": [],
+      "outputs": [
+        {
+          "name": "CLIP",
+          "type": "CLIP",
+          "links": [
+            5,
+            6
+          ]
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "CLIPLoader"
+      },
+      "widgets_values": [
+        "qwen3.5_9b_int8_convrot.safetensors",
+        "moss_image",
+        "default"
+      ],
+      "widgets_values_named": {
+        "clip_name": "qwen3.5_9b_int8_convrot.safetensors",
+        "type": "moss_image",
+        "device": "default"
+      }
+    },
+    {
+      "id": 6,
+      "type": "VAELoader",
+      "pos": [
+        40,
+        880
+      ],
+      "size": [
+        420,
+        70
+      ],
+      "flags": {},
+      "order": 6,
+      "mode": 0,
+      "inputs": [],
+      "outputs": [
+        {
+          "name": "VAE",
+          "type": "VAE",
+          "links": [
+            7,
+            8,
+            9
+          ]
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "VAELoader"
+      },
+      "widgets_values": [
+        "flux2-vae.safetensors"
+      ],
+      "widgets_values_named": {
+        "vae_name": "flux2-vae.safetensors"
+      }
+    },
+    {
+      "id": 7,
+      "type": "GetImageSize",
+      "pos": [
+        840,
+        80
+      ],
+      "size": [
+        220,
+        140
+      ],
+      "flags": {},
+      "order": 7,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "image",
+          "type": "IMAGE",
+          "link": 2
+        }
+      ],
+      "outputs": [
+        {
+          "name": "width",
+          "type": "INT",
+          "links": [
+            11
+          ]
+        },
+        {
+          "name": "height",
+          "type": "INT",
+          "links": [
+            12
+          ]
+        },
+        {
+          "name": "batch_size",
+          "type": "INT",
+          "links": null
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "GetImageSize"
+      }
+    },
+    {
+      "id": 8,
+      "type": "TextEncodeMOSSImageEdit",
+      "pos": [
+        1100,
+        40
+      ],
+      "size": [
+        480,
+        360
+      ],
+      "flags": {},
+      "order": 8,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "clip",
+          "type": "CLIP",
+          "link": 5
+        },
+        {
+          "name": "images.image_0",
+          "type": "IMAGE",
+          "link": 18
+        },
+        {
+          "name": "images.image_1",
+          "type": "IMAGE",
+          "link": null,
+          "shape": 7
+        },
+        {
+          "name": "vae",
+          "type": "VAE",
+          "link": 7
+        }
+      ],
+      "outputs": [
+        {
+          "name": "CONDITIONING",
+          "type": "CONDITIONING",
+          "links": [
+            13
+          ]
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "TextEncodeMOSSImageEdit"
+      },
+      "widgets_values": [
+        "A chrome pitcher tilts and pours a ribbon of dark chocolate over a tall cake in a pink neon diner. One sugar crystal hangs in mid-air beside the stream. Glossy chrome, wet highlights, cinematic photograph.",
+        "ai-toolkit",
+        1
+      ],
+      "widgets_values_named": {
+        "prompt": "A chrome pitcher tilts and pours a ribbon of dark chocolate over a tall cake in a pink neon diner. One sugar crystal hangs in mid-air beside the stream. Glossy chrome, wet highlights, cinematic photograph.",
+        "edit_format": "ai-toolkit",
+        "image_strength": 1
+      },
+      "title": "Text Encode MOSS-Image Edit (positive)"
+    },
+    {
+      "id": 9,
+      "type": "TextEncodeMOSSImageEdit",
+      "pos": [
+        1100,
+        440
+      ],
+      "size": [
+        480,
+        300
+      ],
+      "flags": {},
+      "order": 9,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "clip",
+          "type": "CLIP",
+          "link": 6
+        },
+        {
+          "name": "images.image_0",
+          "type": "IMAGE",
+          "link": 19
+        },
+        {
+          "name": "images.image_1",
+          "type": "IMAGE",
+          "link": null,
+          "shape": 7
+        },
+        {
+          "name": "vae",
+          "type": "VAE",
+          "link": 8
+        }
+      ],
+      "outputs": [
+        {
+          "name": "CONDITIONING",
+          "type": "CONDITIONING",
+          "links": [
+            14
+          ]
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "TextEncodeMOSSImageEdit"
+      },
+      "widgets_values": [
+        "nsfw, worst quality, bad anatomy, watermark, jpeg artifact",
+        "ai-toolkit",
+        1
+      ],
+      "widgets_values_named": {
+        "prompt": "nsfw, worst quality, bad anatomy, watermark, jpeg artifact",
+        "edit_format": "ai-toolkit",
+        "image_strength": 1
+      },
+      "title": "Text Encode MOSS-Image Edit (negative)"
+    },
+    {
+      "id": 10,
+      "type": "EmptyLatentImage",
+      "pos": [
+        840,
+        280
+      ],
+      "size": [
+        240,
+        160
+      ],
+      "flags": {},
+      "order": 10,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "width",
+          "type": "INT",
+          "link": 11
+        },
+        {
+          "name": "height",
+          "type": "INT",
+          "link": 12
+        }
+      ],
+      "outputs": [
+        {
+          "name": "LATENT",
+          "type": "LATENT",
+          "links": [
+            15
+          ]
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "EmptyLatentImage"
+      },
+      "widgets_values": [
+        1024,
+        1024,
+        1
+      ],
+      "widgets_values_named": {
+        "width": 1024,
+        "height": 1024,
+        "batch_size": 1
+      }
+    },
+    {
+      "id": 11,
+      "type": "KSampler",
+      "pos": [
+        1640,
+        80
+      ],
+      "size": [
+        320,
+        470
+      ],
+      "flags": {},
+      "order": 11,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "model",
+          "type": "MODEL",
+          "link": 4
+        },
+        {
+          "name": "positive",
+          "type": "CONDITIONING",
+          "link": 13
+        },
+        {
+          "name": "negative",
+          "type": "CONDITIONING",
+          "link": 14
+        },
+        {
+          "name": "latent_image",
+          "type": "LATENT",
+          "link": 15
+        }
+      ],
+      "outputs": [
+        {
+          "name": "LATENT",
+          "type": "LATENT",
+          "links": [
+            16
+          ]
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "KSampler"
+      },
+      "widgets_values": [
+        42,
+        "fixed",
+        30,
+        4,
+        "euler",
+        "normal",
+        1
+      ],
+      "widgets_values_named": {
+        "seed": 42,
+        "control_after_generate": "fixed",
+        "steps": 30,
+        "cfg": 4,
+        "sampler_name": "euler",
+        "scheduler": "normal",
+        "denoise": 1
+      }
+    },
+    {
+      "id": 12,
+      "type": "VAEDecode",
+      "pos": [
+        2000,
+        80
+      ],
+      "size": [
+        240,
+        70
+      ],
+      "flags": {},
+      "order": 12,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "samples",
+          "type": "LATENT",
+          "link": 16
+        },
+        {
+          "name": "vae",
+          "type": "VAE",
+          "link": 9
+        }
+      ],
+      "outputs": [
+        {
+          "name": "IMAGE",
+          "type": "IMAGE",
+          "links": [
+            17
+          ]
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "VAEDecode"
+      },
+      "widgets_values": []
+    },
+    {
+      "id": 13,
+      "type": "PreviewImage",
+      "pos": [
+        840,
+        480
+      ],
+      "size": [
+        280,
+        280
+      ],
+      "flags": {},
+      "order": 13,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "images",
+          "type": "IMAGE",
+          "link": 10
+        }
+      ],
+      "outputs": [
+        {
+          "name": "images",
+          "type": "IMAGE",
+          "links": null
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "PreviewImage"
+      },
+      "widgets_values": [],
+      "title": "Control"
+    },
+    {
+      "id": 14,
+      "type": "PreviewImage",
+      "pos": [
+        2000,
+        200
+      ],
+      "size": [
+        420,
+        420
+      ],
+      "flags": {},
+      "order": 14,
+      "mode": 0,
+      "inputs": [
+        {
+          "name": "images",
+          "type": "IMAGE",
+          "link": 17
+        }
+      ],
+      "outputs": [
+        {
+          "name": "images",
+          "type": "IMAGE",
+          "links": null
+        }
+      ],
+      "properties": {
+        "Node name for S&R": "PreviewImage"
+      },
+      "widgets_values": [],
+      "title": "Result"
+    }
+  ],
+  "links": [
+    [
+      1,
+      1,
+      0,
+      2,
+      0,
+      "IMAGE"
+    ],
+    [
+      2,
+      2,
+      0,
+      7,
+      0,
+      "IMAGE"
+    ],
+    [
+      18,
+      2,
+      0,
+      8,
+      1,
+      "IMAGE"
+    ],
+    [
+      19,
+      2,
+      0,
+      9,
+      1,
+      "IMAGE"
+    ],
+    [
+      10,
+      2,
+      0,
+      13,
+      0,
+      "IMAGE"
+    ],
+    [
+      3,
+      3,
+      0,
+      4,
+      0,
+      "MODEL"
+    ],
+    [
+      4,
+      4,
+      0,
+      11,
+      0,
+      "MODEL"
+    ],
+    [
+      5,
+      5,
+      0,
+      8,
+      0,
+      "CLIP"
+    ],
+    [
+      6,
+      5,
+      0,
+      9,
+      0,
+      "CLIP"
+    ],
+    [
+      7,
+      6,
+      0,
+      8,
+      3,
+      "VAE"
+    ],
+    [
+      8,
+      6,
+      0,
+      9,
+      3,
+      "VAE"
+    ],
+    [
+      9,
+      6,
+      0,
+      12,
+      1,
+      "VAE"
+    ],
+    [
+      11,
+      7,
+      0,
+      10,
+      0,
+      "INT"
+    ],
+    [
+      12,
+      7,
+      1,
+      10,
+      1,
+      "INT"
+    ],
+    [
+      13,
+      8,
+      0,
+      11,
+      1,
+      "CONDITIONING"
+    ],
+    [
+      14,
+      9,
+      0,
+      11,
+      2,
+      "CONDITIONING"
+    ],
+    [
+      15,
+      10,
+      0,
+      11,
+      3,
+      "LATENT"
+    ],
+    [
+      16,
+      11,
+      0,
+      12,
+      0,
+      "LATENT"
+    ],
+    [
+      17,
+      12,
+      0,
+      14,
+      0,
+      "IMAGE"
+    ]
+  ],
+  "groups": [],
+  "config": {},
+  "extra": {
+    "frontendVersion": "1.58.5"
+  },
+  "version": 0.4
+}

moss_image_sft_segmentation_control_lora.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:63ddb98dd1be0ee6cac57b476139b7e1207d79df665107add0426ff535d661a2
+size 290254328