Instructions to use thedeoxen/MOSS-Image-scribble-control-lora with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use thedeoxen/MOSS-Image-scribble-control-lora with Diffusers:
pip install -U diffusers transformers accelerate
from diffusers import ControlNetModel, StableDiffusionControlNetPipeline controlnet = ControlNetModel.from_pretrained("thedeoxen/MOSS-Image-scribble-control-lora") pipe = StableDiffusionControlNetPipeline.from_pretrained( "OpenMOSS-Team/MOSS-Image-SFT", controlnet=controlnet ) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- Draw Things
Call out that this is a ControlNet-like LoRA
Browse files
README.md
CHANGED
|
@@ -1,104 +1,104 @@
|
|
| 1 |
-
---
|
| 2 |
-
license: apache-2.0
|
| 3 |
-
base_model: OpenMOSS-Team/MOSS-Image-SFT
|
| 4 |
-
base_model_relation: adapter
|
| 5 |
-
language:
|
| 6 |
-
- en
|
| 7 |
-
pipeline_tag: image-to-image
|
| 8 |
-
tags:
|
| 9 |
-
- moss-image
|
| 10 |
-
- controlnet
|
| 11 |
-
- lora
|
| 12 |
-
- comfyui
|
| 13 |
-
- scribble
|
| 14 |
-
library_name: diffusers
|
| 15 |
-
---
|
| 16 |
-
|
| 17 |
-
# MOSS-Image-SFT — Scribble Control LoRA
|
| 18 |
-
|
| 19 |
-
## Short description
|
| 20 |
-
|
| 21 |
-
A scribble Control LoRA for **MOSS-Image-SFT**. Feed loose white strokes on black. The model follows the rough drawing, and the prompt is free to invent texture and style.
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
Selected checkpoint: training step **5000**.
|
| 26 |
-
|
| 27 |
-
## Files
|
| 28 |
-
|
| 29 |
-
- **Weights:** [moss_image_sft_scribble_control_lora.safetensors](https://huggingface.co/thedeoxen/MOSS-Image-scribble-controlnet/blob/main/moss_image_sft_scribble_control_lora.safetensors)
|
| 30 |
-
- **ComfyUI workflow:** [moss_image_scribble_control.json](https://huggingface.co/thedeoxen/MOSS-Image-scribble-controlnet/blob/main/moss_image_scribble_control.json)
|
| 31 |
-
- **Repo:** [thedeoxen/MOSS-Image-scribble-controlnet](https://huggingface.co/thedeoxen/MOSS-Image-scribble-controlnet)
|
| 32 |
-
|
| 33 |
-
## Examples
|
| 34 |
-
|
| 35 |
-
Each preview is one combined image, left to right: **control**, then three prompts.
|
| 36 |
-
|
| 37 |
-
| Control → Results |
|
| 38 |
-
| --- |
|
| 39 |
-
|  |
|
| 40 |
-
|  |
|
| 41 |
-
|  |
|
| 42 |
-
|  |
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
## Extended description
|
| 46 |
-
|
| 47 |
-
Follows a loose scribble: the major strokes and masses, not every fine edge. Appearance, materials, clothing and the world come from the text prompt.
|
| 48 |
-
|
| 49 |
-
The LoRA is rank 64 on linear layers and rank 16 on convolutions. It was trained as an edit adapter: the control image is the reference, and the prompt describes the picture to generate.
|
| 50 |
-
|
| 51 |
-
## How to use
|
| 52 |
-
|
| 53 |
-
1. Load **MOSS-Image-SFT**: UNet `moss_image_sft_int8_convrot.safetensors`, text encoder `qwen3.5_9b_w4a8_gptq.safetensors` with type `moss_image`, VAE `flux2-vae_small.safetensors`.
|
| 54 |
-
2. Load this LoRA on the UNet at strength **1.0**.
|
| 55 |
-
3. Build the control map and pass it as the single reference image in **Text Encode MOSS-Image Edit**.
|
| 56 |
-
4. Set **edit format** to `ai-toolkit` on both the positive and the negative encode. This LoRA was trained with kv cache on.
|
| 57 |
-
5. Sample with **30 steps**, **CFG 4**, `euler` + `simple`.
|
| 58 |
-
|
| 59 |
-
No trigger word. The control image sets the structure. The prompt sets identity, materials, clothing and the world.
|
| 60 |
-
|
| 61 |
-
## ComfyUI
|
| 62 |
-
|
| 63 |
-
The workflow in this repo loads a photo, builds the control map, encodes it with the prompt, and samples.
|
| 64 |
-
|
| 65 |
-
The included workflow uses **HED Soft-Edge Lines** with safe mode enabled. The training scribbles are white strokes on black, thicker and softer than Canny or lineart.
|
| 66 |
-
|
| 67 |
-
Also uses the built-in **Text Encode MOSS-Image Edit** node. Set the edit format to `ai-toolkit`.
|
| 68 |
-
|
| 69 |
-
### Base model
|
| 70 |
-
|
| 71 |
-
Trained on and recommended with **[OpenMOSS-Team/MOSS-Image-SFT](https://huggingface.co/OpenMOSS-Team/MOSS-Image-SFT)**.
|
| 72 |
-
|
| 73 |
-
Typical settings from the included workflow: **30 steps**, **CFG 4**, `euler` + `simple`, LoRA strength **1.0**.
|
| 74 |
-
|
| 75 |
-
Negative prompt used in training samples:
|
| 76 |
-
|
| 77 |
-
```
|
| 78 |
-
nsfw, worst quality, bad anatomy, watermark, jpeg artifact
|
| 79 |
-
```
|
| 80 |
-
|
| 81 |
-
### Example prompt
|
| 82 |
-
|
| 83 |
-
```
|
| 84 |
-
Full-body photograph of a circus ringmaster in scarlet tails, gold braid and a tall black hat, standing three-quarter view in front of a painted circus tent. One arm raised high, the other holding a slender black cane down at the side. Confetti, saturated color, entire body in frame.
|
| 85 |
-
```
|
| 86 |
-
|
| 87 |
-
## What it does
|
| 88 |
-
|
| 89 |
-
- Follows a loose scribble: the major strokes and masses, not every fine edge.
|
| 90 |
-
- Lets the prompt define identity, materials, clothing and style.
|
| 91 |
-
- Works as a single-image control adapter for MOSS-Image-SFT.
|
| 92 |
-
|
| 93 |
-
## Tips
|
| 94 |
-
|
| 95 |
-
- A few confident strokes beat a noisy edge map.
|
| 96 |
-
- HED soft edges are closer to the training scribbles than a thin PiDiNet sketch.
|
| 97 |
-
- Leave texture, color and small details to the prompt.
|
| 98 |
-
- Lower the LoRA strength toward 0.7 if the control is too rigid. Strength 1.0 is the trained value.
|
| 99 |
-
|
| 100 |
-
## Use cases
|
| 101 |
-
|
| 102 |
-
- Rough sketches and storyboard strokes.
|
| 103 |
-
- Fast composition tests before a tighter canny or lineart pass.
|
| 104 |
-
- Drawing on top of a photo when only the big shapes should remain.
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
base_model: OpenMOSS-Team/MOSS-Image-SFT
|
| 4 |
+
base_model_relation: adapter
|
| 5 |
+
language:
|
| 6 |
+
- en
|
| 7 |
+
pipeline_tag: image-to-image
|
| 8 |
+
tags:
|
| 9 |
+
- moss-image
|
| 10 |
+
- controlnet
|
| 11 |
+
- lora
|
| 12 |
+
- comfyui
|
| 13 |
+
- scribble
|
| 14 |
+
library_name: diffusers
|
| 15 |
+
---
|
| 16 |
+
|
| 17 |
+
# MOSS-Image-SFT — Scribble Control LoRA
|
| 18 |
+
|
| 19 |
+
## Short description
|
| 20 |
+
|
| 21 |
+
A scribble Control LoRA for **MOSS-Image-SFT**. Feed loose white strokes on black. The model follows the rough drawing, and the prompt is free to invent texture and style.
|
| 22 |
+
|
| 23 |
+
A ControlNet-like LoRA (controlnet) for MOSS-Image-SFT: a LoRA that follows a control image the way a ControlNet does, not a full ControlNet model and not the official MOSS color/depth concat adapter. Load it with a LoRA loader and pass one control image through **Text Encode MOSS-Image Edit**.
|
| 24 |
+
|
| 25 |
+
Selected checkpoint: training step **5000**.
|
| 26 |
+
|
| 27 |
+
## Files
|
| 28 |
+
|
| 29 |
+
- **Weights:** [moss_image_sft_scribble_control_lora.safetensors](https://huggingface.co/thedeoxen/MOSS-Image-scribble-controlnet/blob/main/moss_image_sft_scribble_control_lora.safetensors)
|
| 30 |
+
- **ComfyUI workflow:** [moss_image_scribble_control.json](https://huggingface.co/thedeoxen/MOSS-Image-scribble-controlnet/blob/main/moss_image_scribble_control.json)
|
| 31 |
+
- **Repo:** [thedeoxen/MOSS-Image-scribble-controlnet](https://huggingface.co/thedeoxen/MOSS-Image-scribble-controlnet)
|
| 32 |
+
|
| 33 |
+
## Examples
|
| 34 |
+
|
| 35 |
+
Each preview is one combined image, left to right: **control**, then three prompts.
|
| 36 |
+
|
| 37 |
+
| Control → Results |
|
| 38 |
+
| --- |
|
| 39 |
+
|  |
|
| 40 |
+
|  |
|
| 41 |
+
|  |
|
| 42 |
+
|  |
|
| 43 |
+
|
| 44 |
+
|
| 45 |
+
## Extended description
|
| 46 |
+
|
| 47 |
+
Follows a loose scribble: the major strokes and masses, not every fine edge. Appearance, materials, clothing and the world come from the text prompt.
|
| 48 |
+
|
| 49 |
+
The LoRA is rank 64 on linear layers and rank 16 on convolutions. It was trained as an edit adapter: the control image is the reference, and the prompt describes the picture to generate.
|
| 50 |
+
|
| 51 |
+
## How to use
|
| 52 |
+
|
| 53 |
+
1. Load **MOSS-Image-SFT**: UNet `moss_image_sft_int8_convrot.safetensors`, text encoder `qwen3.5_9b_w4a8_gptq.safetensors` with type `moss_image`, VAE `flux2-vae_small.safetensors`.
|
| 54 |
+
2. Load this LoRA on the UNet at strength **1.0**.
|
| 55 |
+
3. Build the control map and pass it as the single reference image in **Text Encode MOSS-Image Edit**.
|
| 56 |
+
4. Set **edit format** to `ai-toolkit` on both the positive and the negative encode. This LoRA was trained with kv cache on.
|
| 57 |
+
5. Sample with **30 steps**, **CFG 4**, `euler` + `simple`.
|
| 58 |
+
|
| 59 |
+
No trigger word. The control image sets the structure. The prompt sets identity, materials, clothing and the world.
|
| 60 |
+
|
| 61 |
+
## ComfyUI
|
| 62 |
+
|
| 63 |
+
The workflow in this repo loads a photo, builds the control map, encodes it with the prompt, and samples.
|
| 64 |
+
|
| 65 |
+
The included workflow uses **HED Soft-Edge Lines** with safe mode enabled. The training scribbles are white strokes on black, thicker and softer than Canny or lineart.
|
| 66 |
+
|
| 67 |
+
Also uses the built-in **Text Encode MOSS-Image Edit** node. Set the edit format to `ai-toolkit`.
|
| 68 |
+
|
| 69 |
+
### Base model
|
| 70 |
+
|
| 71 |
+
Trained on and recommended with **[OpenMOSS-Team/MOSS-Image-SFT](https://huggingface.co/OpenMOSS-Team/MOSS-Image-SFT)**.
|
| 72 |
+
|
| 73 |
+
Typical settings from the included workflow: **30 steps**, **CFG 4**, `euler` + `simple`, LoRA strength **1.0**.
|
| 74 |
+
|
| 75 |
+
Negative prompt used in training samples:
|
| 76 |
+
|
| 77 |
+
```
|
| 78 |
+
nsfw, worst quality, bad anatomy, watermark, jpeg artifact
|
| 79 |
+
```
|
| 80 |
+
|
| 81 |
+
### Example prompt
|
| 82 |
+
|
| 83 |
+
```
|
| 84 |
+
Full-body photograph of a circus ringmaster in scarlet tails, gold braid and a tall black hat, standing three-quarter view in front of a painted circus tent. One arm raised high, the other holding a slender black cane down at the side. Confetti, saturated color, entire body in frame.
|
| 85 |
+
```
|
| 86 |
+
|
| 87 |
+
## What it does
|
| 88 |
+
|
| 89 |
+
- Follows a loose scribble: the major strokes and masses, not every fine edge.
|
| 90 |
+
- Lets the prompt define identity, materials, clothing and style.
|
| 91 |
+
- Works as a single-image control adapter for MOSS-Image-SFT.
|
| 92 |
+
|
| 93 |
+
## Tips
|
| 94 |
+
|
| 95 |
+
- A few confident strokes beat a noisy edge map.
|
| 96 |
+
- HED soft edges are closer to the training scribbles than a thin PiDiNet sketch.
|
| 97 |
+
- Leave texture, color and small details to the prompt.
|
| 98 |
+
- Lower the LoRA strength toward 0.7 if the control is too rigid. Strength 1.0 is the trained value.
|
| 99 |
+
|
| 100 |
+
## Use cases
|
| 101 |
+
|
| 102 |
+
- Rough sketches and storyboard strokes.
|
| 103 |
+
- Fast composition tests before a tighter canny or lineart pass.
|
| 104 |
+
- Drawing on top of a photo when only the big shapes should remain.
|