waveshaker commited on
Commit
8a61257
·
verified ·
1 Parent(s): f5afb4c

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +9 -9
README.md CHANGED
@@ -76,15 +76,15 @@ pipeline_tag: text-to-image
76
  - Model: -
77
  - LoRA Weight: 0.8 ~ 1
78
 
79
- > #### LR / Batch Size / Gradient Accumulation
80
- >
81
- > - **Effective Batch Size** = `Batch Size × Gradient Accumulation`. Gradient accumulation saves VRAM but is slower; a larger Batch Size is faster but uses more VRAM.
82
- > - **LR Scaling**: Don't blindly apply the linear rule to LoRA. Validate with loss curves. (My baseline: `Batch 3` + `LR 0.0003`). *Note: The vanilla Fizgig benchmark used the default `LR 0.0001`, which is under-scaled for an effective batch of 3. As widely discussed on GitHub and Reddit, increasing the batch size generally requires a higher LR to compensate for fewer update steps.*
83
- >
84
- > #### Paper TL;DR:
85
- > - [*Accurate, Large Minibatch SGD* (2017)](https://arxiv.org/abs/1706.02677): The "Linear Scaling Rule" — if you multiply batch size by k, multiply LR by k. This was designed for full fine-tuning on ImageNet, not LoRA (e.g., `BS 1, LR 1e-4` → `BS 4, LR 4e-4`).
86
- > - [*One weird trick for parallelizing CNNs* (2014)](https://arxiv.org/abs/1404.5997): The "Square Root Scaling Rule" — if you multiply batch size by k, to keep gradient variance stable, scale LR by ***k^0.5*** instead of k. More conservative when batch gets very large (e.g., `BS 1, LR 1e-4` → `BS 4, LR 2e-4`).
87
- > - [*Beware of the Batch Size* (2026)](https://arxiv.org/abs/2602.09492): For LoRA specifically, batch size is not just a multiplier for LR — it's a **first-order design parameter**. Optimal batch depends on dataset size (most sensitive), rank, and model capacity. Vanilla LoRA matches fancy variants when batch is tuned properly.
88
 
89
  </details>
90
 
 
76
  - Model: -
77
  - LoRA Weight: 0.8 ~ 1
78
 
79
+ > #### LR / Batch Size / Gradient Accumulation
80
+ >
81
+ > - **Effective Batch Size** = `Batch Size × Gradient Accumulation`. Gradient accumulation saves VRAM but is slower; a larger Batch Size is faster but uses more VRAM.
82
+ > - **LR Scaling**: Don't blindly apply the linear rule to LoRA. Validate with loss curves. (My baseline: `Batch 3` + `LR 0.0003`). *Note: The vanilla Fizgig benchmark used the default `LR 0.0001`, which is under-scaled for an effective batch of 3. As widely discussed on GitHub and Reddit, increasing the batch size generally requires a higher LR to compensate for fewer update steps.*
83
+ >
84
+ > #### Paper TL;DR:
85
+ > - [*Accurate, Large Minibatch SGD* (2017)](https://arxiv.org/abs/1706.02677): The "Linear Scaling Rule" — if you multiply batch size by k, multiply LR by k. This was designed for full fine-tuning on ImageNet, not LoRA (e.g., `BS 1, LR 1e-4` → `BS 4, LR 4e-4`).
86
+ > - [*One weird trick for parallelizing CNNs* (2014)](https://arxiv.org/abs/1404.5997): The "Square Root Scaling Rule" — if you multiply batch size by k, to keep gradient variance stable, scale LR by ***k^0.5*** instead of k. More conservative when batch gets very large (e.g., `BS 1, LR 1e-4` → `BS 4, LR 2e-4`).
87
+ > - [*Beware of the Batch Size* (2026)](https://arxiv.org/abs/2602.09492): For LoRA specifically, batch size is not just a multiplier for LR — it's a **first-order design parameter**. Optimal batch depends on dataset size (most sensitive), rank, and model capacity. Vanilla LoRA matches fancy variants when batch is tuned properly.
88
 
89
  </details>
90