waveshaker commited on
Commit
f286629
·
verified ·
1 Parent(s): fd889ac

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +11 -12
README.md CHANGED
@@ -60,22 +60,21 @@ pipeline_tag: text-to-image
60
  - OneTrainer: 910 steps, 1.12 s/it (`Local Batch Size: 3, LR: 0.0003`) → ~0.37s/img (17m 43s)
61
  - Fizgig(Batch-Size: Not Supported): 2730 steps, 1.25 it/s (`Gradient Accumulation: 3, LR: 0.0001`) → ~0.83s/img (36m 17s)
62
  - Fizgig-Advanced(Batch-Size: Supported): 2730 steps, 1.25 it/s (`Batch Size: 3, LR: 0.0003`) → ~0.83s/img (36m 17s)
63
- - *Note on Speed & Formats: OneTrainer used the official sharded Diffusers format (bfloat16), while Fizgig used the ComfyUI single-file format (int8).*
64
  - Showcase & Card Image Parameters:
65
  - Model: -
66
  - LoRA Weight: 0.8 ~ 1
67
 
68
- > **💡 Quick Note: LR / Batch Size / Gradient Accumulation**
69
- >
70
- > - **Effective Batch Size** = `Batch Size × Gradient Accumulation`. Gradient accumulation saves VRAM but is slower; a larger Batch Size is faster but uses more VRAM.
71
- > - **LR Scaling**: Don't blindly apply the linear rule to LoRA. Validate with loss curves. (My baseline: `Batch 3` + `LR 0.0003`). *Note: The vanilla Fizgig benchmark used the default `LR 0.0001`, which is technically under-scaled for an effective batch of 3; however, it was sufficient for this speed test.*
72
- >
73
- > **Paper TL;DR:**
74
- > - [*Accurate, Large Minibatch SGD* (2017)](https://arxiv.org/abs/1706.02677): The "Linear Scaling Rule" — if you multiply batch size by k, multiply LR by k. This was designed for full fine-tuning on ImageNet, not LoRA.
75
- > - [*One weird trick for parallelizing CNNs* (2014)](https://arxiv.org/abs/1404.5997): The "Square Root Scaling Rule" — to keep gradient variance stable, scale LR by √k instead. More conservative when batch gets very large.
76
- > - [*Beware of the Batch Size* (2026)](https://arxiv.org/abs/2602.09492): For LoRA specifically, batch size is not just a multiplier for LR — it's a **first-order design parameter**. Optimal batch depends on dataset size (most sensitive), rank, and model capacity. Vanilla LoRA matches fancy variants when batch is tuned properly.
77
-
78
-
79
  - **Krea2**
80
  - Training Tools: [Onetrainer](https://github.com/nerogar/OneTrainer)
81
  - Dataset: Mostly 50+ images (averaging 100+)
 
60
  - OneTrainer: 910 steps, 1.12 s/it (`Local Batch Size: 3, LR: 0.0003`) → ~0.37s/img (17m 43s)
61
  - Fizgig(Batch-Size: Not Supported): 2730 steps, 1.25 it/s (`Gradient Accumulation: 3, LR: 0.0001`) → ~0.83s/img (36m 17s)
62
  - Fizgig-Advanced(Batch-Size: Supported): 2730 steps, 1.25 it/s (`Batch Size: 3, LR: 0.0003`) → ~0.83s/img (36m 17s)
63
+ - *(Note on Speed & Formats: OneTrainer used the official sharded Diffusers format (bfloat16), while Fizgig used the ComfyUI single-file format (int8). Although int8/fp8 usually compute faster on modern GPUs, the benchmark shows Fizgig is slower here—likely due to software-level framework overheads. Fizgig also supports the original bf16 Diffusers shards.)*
64
  - Showcase & Card Image Parameters:
65
  - Model: -
66
  - LoRA Weight: 0.8 ~ 1
67
 
68
+ > #### LR / Batch Size / Gradient Accumulation
69
+ >
70
+ > - **Effective Batch Size** = `Batch Size × Gradient Accumulation`. Gradient accumulation saves VRAM but is slower; a larger Batch Size is faster but uses more VRAM.
71
+ > - **LR Scaling**: Don't blindly apply the linear rule to LoRA. Validate with loss curves. (My baseline: `Batch 3` + `LR 0.0003`). *Note: The vanilla Fizgig benchmark used the default `LR 0.0001`, which is technically under-scaled for an effective batch of 3; however, it was sufficient for this speed test.*
72
+ >
73
+ > #### Paper TL;DR:
74
+ > - [*Accurate, Large Minibatch SGD* (2017)](https://arxiv.org/abs/1706.02677): The "Linear Scaling Rule" — if you multiply batch size by k, multiply LR by k. This was designed for full fine-tuning on ImageNet, not LoRA.
75
+ > - [*One weird trick for parallelizing CNNs* (2014)](https://arxiv.org/abs/1404.5997): The "Square Root Scaling Rule" — to keep gradient variance stable, scale LR by √k instead. More conservative when batch gets very large.
76
+ > - [*Beware of the Batch Size* (2026)](https://arxiv.org/abs/2602.09492): For LoRA specifically, batch size is not just a multiplier for LR — it's a **first-order design parameter**. Optimal batch depends on dataset size (most sensitive), rank, and model capacity. Vanilla LoRA matches fancy variants when batch is tuned properly.
77
+
 
78
  - **Krea2**
79
  - Training Tools: [Onetrainer](https://github.com/nerogar/OneTrainer)
80
  - Dataset: Mostly 50+ images (averaging 100+)