RicemanT
/

MossAD-Demo

+---
+license: apache-2.0
+base_model:
+- OpenMOSS-Team/MOSS-Image-Base
+pipeline_tag: text-to-image
+tags:
+- anime
+- furry
+- danbooru
+- e621
+- gelbooru
+- finetune
+- moss-image
+- artist-style
+---
+# MossAD-Demo - 57 Artist Style Test Full Finetune of OpenMOSS's MOSS-Image 5B
+<p align="center">
+  <img src="https://huggingface.co/RicemanT/MossAD-Demo/resolve/main/images/cover.png" width="60%" />
+</p>
+ALL PREVIEW IMAGES HAVE COMFYUI METADATA.
+<table>
+<tr>
+<td width="25%"><img src="https://huggingface.co/RicemanT/MossAD-Demo/resolve/main/images/01.png" width="100%"/></td>
+<td width="25%"><img src="https://huggingface.co/RicemanT/MossAD-Demo/resolve/main/images/02.png" width="100%"/></td>
+<td width="25%"><img src="https://huggingface.co/RicemanT/MossAD-Demo/resolve/main/images/03.png" width="100%"/></td>
+<td width="25%"><img src="https://huggingface.co/RicemanT/MossAD-Demo/resolve/main/images/04.png" width="100%"/></td>
+</tr>
+<tr>
+<td width="25%"><img src="https://huggingface.co/RicemanT/MossAD-Demo/resolve/main/images/05.png" width="100%"/></td>
+<td width="25%"><img src="https://huggingface.co/RicemanT/MossAD-Demo/resolve/main/images/06.png" width="100%"/></td>
+<td width="25%"><img src="https://huggingface.co/RicemanT/MossAD-Demo/resolve/main/images/07.png" width="100%"/></td>
+<td width="25%"><img src="https://huggingface.co/RicemanT/MossAD-Demo/resolve/main/images/08.png" width="100%"/></td>
+</tr>
+</table>
+---
+## I. Introduction
+***MossAD-Demo*** is a Danbooru/Gelbooru/E621 proof of concept Full-Finetune of OpenMOSS's MOSS-Image Base 5B T2I model, trained on a small curated 3,685 images dataset of 57 artists, as a way to test how fast and how well MOSS-Image learns artist styles, booru tags prompting and layout (bounding box) prompting before committing to a much bigger run.
+The test is a big ass success: after only ~39 hours on a single AMD MI300X (ok I know it's alot for just 3k image on this kind of gpu, but i received the compute very suddenly and didn't have the luxury to optimize training), the styles land at around 90% accuracy, they carry over to subjects the artist never drew (a furry artist's style on anime girls, and the other way around), and hands are solid enough barring some exceptions here and there.
+~ What the pilot found, and what the next big run will use:
+* MOSS-Image learns styles fast: ~4.5k views per artist at 512px already gives 70-75% style accuracy, ~7k views gives 85-90% with the style carrying over to other subjects, while ~4k views only shows the style in the artist's own domain (furry artist styles only on furry gens)
+* Most of the training can stay at cheap 512px, with shorter 768px and 1024px stages for hands, details and texture
+* The learning rate needs to be halved at every higher resolution stage (4e-5 → 2e-5 → 1e-5), carrying the 512px LR into 768px made it unstable within 5 epochs and the stage had to be restarted
+* The base model already understands panels and boxes from ComfyUI's default bbox node, and the finetune keeps that working with the trained styles
+* Watermarks REALLY stick: if most of an artist's images share the same watermark, no tagging, captioning or negative prompt removes it. I will test masked training in the future to keep watermarks and signatures out of the loss if deemed viable and not too much of a hassle.
+***Please scroll down to the generation recommendation section for the settings and the artist trigger.***
+**Any donation will help with funding the full run, you can do so through:**
+***Crypto (Prefered, cause Kofi/Paypal money transfer time is ass and they take a big cut)***
+**0x6a4bc748cd0bb9ced9a360eb0eb79f4f106614f8 (USDT - BEP20 Network)**
+**12PPVYUeS1MerNp38Tpns5qXR6cmhu9tws (Bitcoin - BTC Network)**
+**0x6a4bc748cd0bb9ced9a360eb0eb79f4f106614f8 (Ethereum - ERC20 Network)**
+**GKjLvD9XGAhyEpsZvEUNH5je9PFbrivoV5yBfYKGYskq (Solana - SOL network)**
+Please handle your money carefully and make sure the address you're sending to is correct.
+Please contact me through discord (user: talannnn) if you're donating with crypto, to make sure it goes through and you're not losing money.
+***Ko-fi***
+**https://ko-fi.com/talanartvn**
+## II. Model Details
+|                                          |                                                                                                                    |
+| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
+| **Base model**                           | [OpenMOSS's MOSS-Image Base 5B](https://huggingface.co/OpenMOSS-Team/MOSS-Image-Base)                              |
+| **Method**                               | Full Finetune                                                                                                      |
+| **Trainer**                              | [My mage-flow-trainer fork](https://github.com/RicemanT/mage-flow-trainer) (of [Bluvoll's mage-flow-trainer](https://github.com/bluvoll/mage-flow-trainer)) |
+| **Hardware**                             | 1x AMD MI300X 192GB (DigitalOcean AMD Developer Cloud), free compute from Disty                                    |
+| **Total training time**                  | ~39 hours of MI300X for the final model (~55 hours including LR tests and the restarted stage)                     |
+| **Total samples seen**                   | ~457k (~8k views per artist)                                                                                       |
+| **Training resolutions**                 | 512² → 768² → 1024², mixed aspect ratios packed with varlen attention                                              |
+### Training run
+**Budget:** 0 dollars (free MI300X compute, thanks Disty!)
+Full configs: [configs folder](https://huggingface.co/RicemanT/MossAD-Demo/tree/main/configs)
+| Stage | Peak LR | Epochs | Steps | Effective batch | Views per artist | Time |
+|---|---|---|---|---|---|---|
+| 512px | 4e-5 | 70 | 560 | 512 (128 x 4) | ~4,550 | ~12.8h |
+| 768px | 2e-5 | 40 | 600 | 256 (64 x 4) | ~2,600 | ~15.5h |
+| 1024px | 1e-5 | 14 | 406 | 128 (32 x 4) | ~910 | ~10.5h |
+- **LR scheduler:** Warmup (5%) -> Constant (55%) -> REX to 0 (40%), for every stage
+- **Precision:** Full BF16
+- **Optimizer:** AdamW8bit (quantized state, no Kahan summation)
+- **Attention:** torch varlen with packed resolutions (no caption padding, no bucketing tricks)
+- **LR test:** 4e-5 vs 2e-5 at 512px for 10 epochs each, 4e-5 learned styles a lot faster and was continued
+### Additional training features
+- 16 caption variations per image
+- Mixed captions at 25/10/15/15/25/20 ratio (tags only, short NL, medium NL, long NL, layout with bounding boxes, tags-nl)
+- Layout captions: characters, objects, text, panels and character detail boxes, with short box captions in 30% of the layouts
+- Tag dropout: 10%
+- Caption dropout: 5%
+- Tag shuffle
+- Caption shuffle
+- Artist trigger attribution system (the trigger always stays at the front)
+---
+## III. Recommended Settings
+The model works in ComfyUI with its native MOSS-Image support: load the checkpoint as the diffusion model, together with the text encoder and VAE from [Comfy-Org/MOSS-Image](https://huggingface.co/Comfy-Org/MOSS-Image). These are the settings used for the sample images above (ComfyUI, the images have metadata):
+- **Shift:** 5.0
+- **Steps:** 30 (50 for best quality)
+- **CFG:** 4
+- **Sampler:** `euler`
+- **Scheduler:** `simple`
+- **Negative prompt:** worst quality, bad anatomy, watermark, jpeg artifact
+These are just my usual settings and workflow — feel free to experiment.
+The model also includes quality tags, you usually want to add them at the end of the tag section of the prompt, they are:
+***masterpiece, best quality, very aesthetic, aesthetic***
+***Artist Trigger:*** This model uses the **@artist name** trigger, if you want to use the model's built-in artist styles, please always put one at the start of your prompt (for example `@sakimichan, 1girl, ...` or `@sakimichan. A girl standing in the rain...`). The model supports 57 artists, please download to see: [MossAD-Demo-artists.csv](https://huggingface.co/RicemanT/MossAD-Demo/blob/main/MossAD-Demo-artists.csv). The characters the dataset covers are in [MossAD-Demo-characters.csv](https://huggingface.co/RicemanT/MossAD-Demo/blob/main/MossAD-Demo-characters.csv).
+All three prompt styles work: booru tags, natural language, and MOSS layout prompts with bounding boxes (ComfyUI's default bbox node works). Tags-only prompts are a bit rougher than natural language and layout prompts, and text rendering is clearly better when the prompt spells out what the text says.
+### Known issues
+- Watermarks and signatures that most of an artist's images share can show up in gens
+- Some gens look a little burned at 1024px
+- Hands are solid most of the time, with some exceptions here and there
+---
+## IV. Dataset
+3,685 images of 57 artists from Danbooru, Gelbooru and E621 (35-75 images per artist), collected and curated with my Illustration Scrapping Studio app, and captioned with SOTA API captioners into booru tags, three lengths of natural language captions and MOSS layout captions, with detector-assisted bounding boxes for text, watermarks, faces and panels.
+---
+## V. Notes from the Training Diary
+Full pilot notes (stages, LR lessons, views per artist, ROCm setup): [MossAD-Demo pilot README](https://github.com/RicemanT/mage-flow-trainer/blob/main/scripts/mossad-demo/README.md)
+---
+## VI. License
+This model is a Derivative of OpenMOSS's MOSS-Image and is distributed under the same **[Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0)** as the base model, with no additional restrictions.
+---
+## VII. Acknowledgments
+Beeg thanks to:
+- Disty — The free MI300X compute that made this whole test possible, the biggest thanks to them
+- [OpenMOSS](https://huggingface.co/OpenMOSS-Team) — Creators of MOSS-Image, the gift that keeps on giving
+- [Bluvoll](https://github.com/bluvoll/mage-flow-trainer) — Creator of mage-flow-trainer that my fork is derived from, and general training advice
+- [Motimalu](https://civitai.red/user/motimalu) — Inspiration behind finetuning practices and configs, like lowering the LR at every resolution stage
+- [KBlueLeaf](https://huggingface.co/KBlueLeaf) — MOSS training advice
+- [Anzhc](https://huggingface.co/Anzhc) — Detection models used for the bounding box captions, and general training advice
+- Format inspired by [Motimalu's Kirazuri diary](https://github.com/motimalu/diffusion-training-configs/blob/main/diffusion-pipe/anima/notes/kirazuri4.0-notes%20.md)