Update README.md
Browse files
README.md
CHANGED
|
@@ -76,15 +76,15 @@ pipeline_tag: text-to-image
|
|
| 76 |
- Model: -
|
| 77 |
- LoRA Weight: 0.8 ~ 1
|
| 78 |
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
|
| 89 |
</details>
|
| 90 |
|
|
|
|
| 76 |
- Model: -
|
| 77 |
- LoRA Weight: 0.8 ~ 1
|
| 78 |
|
| 79 |
+
> #### LR / Batch Size / Gradient Accumulation
|
| 80 |
+
>
|
| 81 |
+
> - **Effective Batch Size** = `Batch Size × Gradient Accumulation`. Gradient accumulation saves VRAM but is slower; a larger Batch Size is faster but uses more VRAM.
|
| 82 |
+
> - **LR Scaling**: Don't blindly apply the linear rule to LoRA. Validate with loss curves. (My baseline: `Batch 3` + `LR 0.0003`). *Note: The vanilla Fizgig benchmark used the default `LR 0.0001`, which is under-scaled for an effective batch of 3. As widely discussed on GitHub and Reddit, increasing the batch size generally requires a higher LR to compensate for fewer update steps.*
|
| 83 |
+
>
|
| 84 |
+
> #### Paper TL;DR:
|
| 85 |
+
> - [*Accurate, Large Minibatch SGD* (2017)](https://arxiv.org/abs/1706.02677): The "Linear Scaling Rule" — if you multiply batch size by k, multiply LR by k. This was designed for full fine-tuning on ImageNet, not LoRA (e.g., `BS 1, LR 1e-4` → `BS 4, LR 4e-4`).
|
| 86 |
+
> - [*One weird trick for parallelizing CNNs* (2014)](https://arxiv.org/abs/1404.5997): The "Square Root Scaling Rule" — if you multiply batch size by k, to keep gradient variance stable, scale LR by ***k^0.5*** instead of k. More conservative when batch gets very large (e.g., `BS 1, LR 1e-4` → `BS 4, LR 2e-4`).
|
| 87 |
+
> - [*Beware of the Batch Size* (2026)](https://arxiv.org/abs/2602.09492): For LoRA specifically, batch size is not just a multiplier for LR — it's a **first-order design parameter**. Optimal batch depends on dataset size (most sensitive), rank, and model capacity. Vanilla LoRA matches fancy variants when batch is tuned properly.
|
| 88 |
|
| 89 |
</details>
|
| 90 |
|