VER9 / README.md
honghong3's picture
Update README.md
f7f5de4 verified
|
Raw History Blame Contribute Delete
4.29 kB
---
license: mit
library_name: diffusers
pipeline_tag: text-to-image
tags:
- mmdit
- text-to-image
- flow-matching
- dc-ae
- t5
language:
- en
- ko
---
# Diffusion MMDiT with DC-AE f32c32
## Inference Requirements
- **Inference:** Min VRAM 15GB
- **Storage:** Min 30GB free disk space (excluding `requirements.txt` packages)
- **Environment:** Local & Google Colab supported
## Update
### CPU Offloading Mode
If your GPU has less than 15GB VRAM, run inference in CPU offloading mode.
The model stays on CPU; transformer blocks are streamed to the GPU in chunks.
On startup, choose `n`:
```
model upload on gpu (y or n): n
```
Then set the chunk size (layers loaded onto the GPU at once):
```
block chunk size (1~8, recommended 4):
```
| CHUNK | Layers on GPU | With prefetch | Approx. VRAM |
|-------|---------------|---------------|--------------|
| 1 | 1 | 2 | ~0.5GB |
| 2 | 2 | 4 | ~1.1GB |
| 4 | 4 | 8 | ~2.2GB |
| 8 | 8 | 16 | ~4.3GB |
- Larger chunk = faster, more VRAM.
- Smaller chunk = safer on low VRAM.
- Up to `2 × CHUNK` layers are resident with prefetch.
Trade-off: VRAM drops from ~15GB to ~3GB, but inference is slower due to
CPU-GPU PCIe transfers. Output quality may also be slightly lower than full
GPU mode (differences in floating-point precision and kernel accumulation
order across CPU↔GPU transfers).
## Installation
```bash
python -m pip install -r requirements.txt
```
![MMDIT](assets/sample.png)
## Architecture
- **Autoencoder:** DC-AE f32c32 (32× compression, 32 latent channels)
- **Text Encoder:** T5-Large
- **Backbone:** MMDiT with Joint Attention
- **Implementation:** Custom implementation (no pretrained base model)
- **Parameters:** 4B
## Train Environment
- **Framework:** PyTorch (no custom CUDA kernels)
- **GPU:** NVIDIA RTX PRO 6000 (96GB VRAM)
- **Platform:** Google Colab
## Training
### Configuration
- **Batch Size:** Micro-batch 4 × Gradient Accumulation 8 = Effective Batch 32
- **Training Speed:** ~2.3 s / optimizer step
- **Max Resolution:** 1024 × 1024
- **Dataset:** 2M images
- **Objective:** Flow Matching (shift = 3.0)
- **Precision:** BF16
- **Optimizer:** AdamW
- **Learning Rate:** 1e-4
- **EMA:** Enabled
### Training Stages
| Stage | Description | Steps | Patch Size |
|:---:|---|---:|:---:|
| Stage 1 | Pre-training | 600K | 2 |
| Stage 2 | Fine-tuning | 11K | 1 |
**Stage 2 Notes:**
- Reset patch embedding weights.
- Reset output projection weights.
### Loss by Noise Level
| Noise Level | Average Loss |
|:---:|---:|
| 1.0 – 0.9 | 4.0 ~ 4.3 |
| 0.9 – 0.8 | 2.9 ~ 3.1 |
| 0.8 – 0.7 | 2.4 ~ 2.6 |
| 0.7 – 0.6 | 2.0 ~ 2.1 |
| 0.6 – 0.5 | 1.8 ~ 1.9 |
| 0.5 – 0.4 | 1.6 ~ 1.7 |
| 0.4 – 0.3 | 1.3 ~ 1.6 |
| 0.3 – 0.2 | 1.2 ~ 1.4 |
| 0.2 – 0.1 | 1.0 ~ 1.1 |
## Generation Results
### Patch Size 2 + DC-AE f32c32
![Patch Size 2](assets/patch_size_2.png)
- **Issue:** Facial features, particularly the eyes, tend to collapse or become distorted.
- **Observation:** Limited spatial detail makes accurate reconstruction of fine facial features challenging.
### Patch Size 1 + DC-AE f32c32
![Patch Size 1](assets/patch_size_1.png)
- **Fine-tuning Result:** Fine-tuning with patch size 1 improved facial feature reconstruction.
- **Improvement:** Eye and facial distortions were partially resolved, with better preservation of fine details.
### Current Limitations
- **Full-body Generation:** Full-body generation remains challenging due to insufficient full-body samples in the training dataset.
- **Observation:** The current dataset contains insufficient full-body examples to learn consistent full-body composition and anatomical details.
- **Future Work:** Increase the proportion and diversity of full-body images in the training dataset.
### Train Architecture
- **Use Cache**
- **Cpu Multi Threading**
![inference](assets/train_loop.png)
### Inference Architecture
![inference](assets/inference.png)
### UI
- **Sampling**
![sampling](assets/sampling.png)
- **Inpainting**
![inpainting](assets/inpainting.png)
- **Result**
![result](assets/inpaintbf.png)
- **GITHUB link** https://github.com/FREEANIMA/DiffusionSampling.git