File size: 2,700 Bytes
320c5b2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 | # Optimizer Options in Unified Training Script
## Overview
The unified training script now supports multiple optimizers. Adam remains the default for backward compatibility with existing configurations.
## Available Optimizers
### 1. Adam (Default)
Standard Adam optimizer, good general-purpose choice.
```yaml
optimizer_type: 'Adam' # This is the default if not specified
weight_decay: 0.0 # L2 regularization
betas: [0.9, 0.999] # Optional: Adam beta parameters
eps: 0.00000001 # Optional: Adam epsilon for numerical stability
```
### 2. AdamW
Adam with decoupled weight decay, often better for transformer models and when using weight decay.
```yaml
optimizer_type: 'AdamW'
weight_decay: 0.01 # Decoupled weight decay (common values: 0.01-0.1)
betas: [0.9, 0.999] # Optional: AdamW beta parameters
eps: 0.00000001 # Optional: AdamW epsilon
```
### 3. SGD
Stochastic Gradient Descent with momentum, sometimes better for ConvNets.
```yaml
optimizer_type: 'SGD'
weight_decay: 0.0001 # L2 regularization
momentum: 0.9 # Optional: SGD momentum (default: 0.9)
nesterov: True # Optional: Use Nesterov momentum (default: True)
```
## Command Line Override
You can override optimizer settings via command line:
```bash
# Use AdamW
python train_network_unified.py \
--config your_config.yaml \
--opts \
optimizer_type AdamW \
weight_decay 0.01
# Use SGD with specific momentum
python train_network_unified.py \
--config your_config.yaml \
--opts \
optimizer_type SGD \
momentum 0.95 \
weight_decay 0.0001
```
## Recommendations
### For LLaVA/Transformer Models
- **Optimizer**: AdamW
- **Weight decay**: 0.01-0.05
- **Learning rate**: 1e-4 to 5e-4
- **Warmup**: 2000-5000 steps
```yaml
optimizer_type: 'AdamW'
weight_decay: 0.01
base_lr: 0.0001
warmup_steps: 2000
```
### For ConvNet Backbones
- **Optimizer**: Adam or SGD
- **Weight decay**: 0.0 (Adam) or 0.0001 (SGD)
- **Learning rate**: 1e-4 to 1e-3
```yaml
# Option 1: Adam
optimizer_type: 'Adam'
weight_decay: 0.0
base_lr: 0.0001
# Option 2: SGD
optimizer_type: 'SGD'
weight_decay: 0.0001
base_lr: 0.001
momentum: 0.9
```
### Fine-tuning Pre-trained Models
- **Optimizer**: AdamW (preferred) or Adam
- **Weight decay**: Lower values (0.0-0.01)
- **Learning rate**: Lower than training from scratch
- **Warmup**: Short warmup (500-1000 steps)
```yaml
optimizer_type: 'AdamW'
weight_decay: 0.01
base_lr: 0.00005
warmup_steps: 1000
```
## Monitoring
The optimizer type is logged to WandB along with other hyperparameters. Monitor:
- Learning rate curves
- Training/validation loss
- Gradient norms (if logged)
- Weight decay effectiveness |