thedeoxen commited on
Commit
f9a6a55
·
verified ·
1 Parent(s): d1740bf

Call out that this is a ControlNet-like LoRA

Browse files
Files changed (1) hide show
  1. README.md +104 -104
README.md CHANGED
@@ -1,104 +1,104 @@
1
- ---
2
- license: apache-2.0
3
- base_model: OpenMOSS-Team/MOSS-Image-SFT
4
- base_model_relation: adapter
5
- language:
6
- - en
7
- pipeline_tag: image-to-image
8
- tags:
9
- - moss-image
10
- - controlnet
11
- - lora
12
- - comfyui
13
- - scribble
14
- library_name: diffusers
15
- ---
16
-
17
- # MOSS-Image-SFT — Scribble Control LoRA
18
-
19
- ## Short description
20
-
21
- A scribble Control LoRA for **MOSS-Image-SFT**. Feed loose white strokes on black. The model follows the rough drawing, and the prompt is free to invent texture and style.
22
-
23
- This is a ControlNet-style **edit LoRA**, not the official MOSS color/depth concat adapter. Load it with a LoRA loader and pass one control image through **Text Encode MOSS-Image Edit**.
24
-
25
- Selected checkpoint: training step **5000**.
26
-
27
- ## Files
28
-
29
- - **Weights:** [moss_image_sft_scribble_control_lora.safetensors](https://huggingface.co/thedeoxen/MOSS-Image-scribble-controlnet/blob/main/moss_image_sft_scribble_control_lora.safetensors)
30
- - **ComfyUI workflow:** [moss_image_scribble_control.json](https://huggingface.co/thedeoxen/MOSS-Image-scribble-controlnet/blob/main/moss_image_scribble_control.json)
31
- - **Repo:** [thedeoxen/MOSS-Image-scribble-controlnet](https://huggingface.co/thedeoxen/MOSS-Image-scribble-controlnet)
32
-
33
- ## Examples
34
-
35
- Each preview is one combined image, left to right: **control**, then three prompts.
36
-
37
- | Control → Results |
38
- | --- |
39
- | ![](images/figure.jpg) |
40
- | ![](images/espresso.jpg) |
41
- | ![](images/ramen.jpg) |
42
- | ![](images/subway.jpg) |
43
-
44
-
45
- ## Extended description
46
-
47
- Follows a loose scribble: the major strokes and masses, not every fine edge. Appearance, materials, clothing and the world come from the text prompt.
48
-
49
- The LoRA is rank 64 on linear layers and rank 16 on convolutions. It was trained as an edit adapter: the control image is the reference, and the prompt describes the picture to generate.
50
-
51
- ## How to use
52
-
53
- 1. Load **MOSS-Image-SFT**: UNet `moss_image_sft_int8_convrot.safetensors`, text encoder `qwen3.5_9b_w4a8_gptq.safetensors` with type `moss_image`, VAE `flux2-vae_small.safetensors`.
54
- 2. Load this LoRA on the UNet at strength **1.0**.
55
- 3. Build the control map and pass it as the single reference image in **Text Encode MOSS-Image Edit**.
56
- 4. Set **edit format** to `ai-toolkit` on both the positive and the negative encode. This LoRA was trained with kv cache on.
57
- 5. Sample with **30 steps**, **CFG 4**, `euler` + `simple`.
58
-
59
- No trigger word. The control image sets the structure. The prompt sets identity, materials, clothing and the world.
60
-
61
- ## ComfyUI
62
-
63
- The workflow in this repo loads a photo, builds the control map, encodes it with the prompt, and samples.
64
-
65
- The included workflow uses **HED Soft-Edge Lines** with safe mode enabled. The training scribbles are white strokes on black, thicker and softer than Canny or lineart.
66
-
67
- Also uses the built-in **Text Encode MOSS-Image Edit** node. Set the edit format to `ai-toolkit`.
68
-
69
- ### Base model
70
-
71
- Trained on and recommended with **[OpenMOSS-Team/MOSS-Image-SFT](https://huggingface.co/OpenMOSS-Team/MOSS-Image-SFT)**.
72
-
73
- Typical settings from the included workflow: **30 steps**, **CFG 4**, `euler` + `simple`, LoRA strength **1.0**.
74
-
75
- Negative prompt used in training samples:
76
-
77
- ```
78
- nsfw, worst quality, bad anatomy, watermark, jpeg artifact
79
- ```
80
-
81
- ### Example prompt
82
-
83
- ```
84
- Full-body photograph of a circus ringmaster in scarlet tails, gold braid and a tall black hat, standing three-quarter view in front of a painted circus tent. One arm raised high, the other holding a slender black cane down at the side. Confetti, saturated color, entire body in frame.
85
- ```
86
-
87
- ## What it does
88
-
89
- - Follows a loose scribble: the major strokes and masses, not every fine edge.
90
- - Lets the prompt define identity, materials, clothing and style.
91
- - Works as a single-image control adapter for MOSS-Image-SFT.
92
-
93
- ## Tips
94
-
95
- - A few confident strokes beat a noisy edge map.
96
- - HED soft edges are closer to the training scribbles than a thin PiDiNet sketch.
97
- - Leave texture, color and small details to the prompt.
98
- - Lower the LoRA strength toward 0.7 if the control is too rigid. Strength 1.0 is the trained value.
99
-
100
- ## Use cases
101
-
102
- - Rough sketches and storyboard strokes.
103
- - Fast composition tests before a tighter canny or lineart pass.
104
- - Drawing on top of a photo when only the big shapes should remain.
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: OpenMOSS-Team/MOSS-Image-SFT
4
+ base_model_relation: adapter
5
+ language:
6
+ - en
7
+ pipeline_tag: image-to-image
8
+ tags:
9
+ - moss-image
10
+ - controlnet
11
+ - lora
12
+ - comfyui
13
+ - scribble
14
+ library_name: diffusers
15
+ ---
16
+
17
+ # MOSS-Image-SFT — Scribble Control LoRA
18
+
19
+ ## Short description
20
+
21
+ A scribble Control LoRA for **MOSS-Image-SFT**. Feed loose white strokes on black. The model follows the rough drawing, and the prompt is free to invent texture and style.
22
+
23
+ A ControlNet-like LoRA (controlnet) for MOSS-Image-SFT: a LoRA that follows a control image the way a ControlNet does, not a full ControlNet model and not the official MOSS color/depth concat adapter. Load it with a LoRA loader and pass one control image through **Text Encode MOSS-Image Edit**.
24
+
25
+ Selected checkpoint: training step **5000**.
26
+
27
+ ## Files
28
+
29
+ - **Weights:** [moss_image_sft_scribble_control_lora.safetensors](https://huggingface.co/thedeoxen/MOSS-Image-scribble-controlnet/blob/main/moss_image_sft_scribble_control_lora.safetensors)
30
+ - **ComfyUI workflow:** [moss_image_scribble_control.json](https://huggingface.co/thedeoxen/MOSS-Image-scribble-controlnet/blob/main/moss_image_scribble_control.json)
31
+ - **Repo:** [thedeoxen/MOSS-Image-scribble-controlnet](https://huggingface.co/thedeoxen/MOSS-Image-scribble-controlnet)
32
+
33
+ ## Examples
34
+
35
+ Each preview is one combined image, left to right: **control**, then three prompts.
36
+
37
+ | Control → Results |
38
+ | --- |
39
+ | ![](images/figure.jpg) |
40
+ | ![](images/espresso.jpg) |
41
+ | ![](images/ramen.jpg) |
42
+ | ![](images/subway.jpg) |
43
+
44
+
45
+ ## Extended description
46
+
47
+ Follows a loose scribble: the major strokes and masses, not every fine edge. Appearance, materials, clothing and the world come from the text prompt.
48
+
49
+ The LoRA is rank 64 on linear layers and rank 16 on convolutions. It was trained as an edit adapter: the control image is the reference, and the prompt describes the picture to generate.
50
+
51
+ ## How to use
52
+
53
+ 1. Load **MOSS-Image-SFT**: UNet `moss_image_sft_int8_convrot.safetensors`, text encoder `qwen3.5_9b_w4a8_gptq.safetensors` with type `moss_image`, VAE `flux2-vae_small.safetensors`.
54
+ 2. Load this LoRA on the UNet at strength **1.0**.
55
+ 3. Build the control map and pass it as the single reference image in **Text Encode MOSS-Image Edit**.
56
+ 4. Set **edit format** to `ai-toolkit` on both the positive and the negative encode. This LoRA was trained with kv cache on.
57
+ 5. Sample with **30 steps**, **CFG 4**, `euler` + `simple`.
58
+
59
+ No trigger word. The control image sets the structure. The prompt sets identity, materials, clothing and the world.
60
+
61
+ ## ComfyUI
62
+
63
+ The workflow in this repo loads a photo, builds the control map, encodes it with the prompt, and samples.
64
+
65
+ The included workflow uses **HED Soft-Edge Lines** with safe mode enabled. The training scribbles are white strokes on black, thicker and softer than Canny or lineart.
66
+
67
+ Also uses the built-in **Text Encode MOSS-Image Edit** node. Set the edit format to `ai-toolkit`.
68
+
69
+ ### Base model
70
+
71
+ Trained on and recommended with **[OpenMOSS-Team/MOSS-Image-SFT](https://huggingface.co/OpenMOSS-Team/MOSS-Image-SFT)**.
72
+
73
+ Typical settings from the included workflow: **30 steps**, **CFG 4**, `euler` + `simple`, LoRA strength **1.0**.
74
+
75
+ Negative prompt used in training samples:
76
+
77
+ ```
78
+ nsfw, worst quality, bad anatomy, watermark, jpeg artifact
79
+ ```
80
+
81
+ ### Example prompt
82
+
83
+ ```
84
+ Full-body photograph of a circus ringmaster in scarlet tails, gold braid and a tall black hat, standing three-quarter view in front of a painted circus tent. One arm raised high, the other holding a slender black cane down at the side. Confetti, saturated color, entire body in frame.
85
+ ```
86
+
87
+ ## What it does
88
+
89
+ - Follows a loose scribble: the major strokes and masses, not every fine edge.
90
+ - Lets the prompt define identity, materials, clothing and style.
91
+ - Works as a single-image control adapter for MOSS-Image-SFT.
92
+
93
+ ## Tips
94
+
95
+ - A few confident strokes beat a noisy edge map.
96
+ - HED soft edges are closer to the training scribbles than a thin PiDiNet sketch.
97
+ - Leave texture, color and small details to the prompt.
98
+ - Lower the LoRA strength toward 0.7 if the control is too rigid. Strength 1.0 is the trained value.
99
+
100
+ ## Use cases
101
+
102
+ - Rough sketches and storyboard strokes.
103
+ - Fast composition tests before a tighter canny or lineart pass.
104
+ - Drawing on top of a photo when only the big shapes should remain.