File size: 2,583 Bytes
320c5b2 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 | # WandB Integration Guide
This guide explains the improved WandB integration in the refactored training framework.
## Key Improvements
### 1. **Secure API Key Management**
- **Old**: API key hardcoded in scripts
- **New**: Use environment variables or .netrc file
```bash
# Option 1: Environment variable
export WANDB_API_KEY=your_api_key_here
# Option 2: .netrc file (recommended)
echo "machine api.wandb.ai
login user
password your_api_key_here" >> ~/.netrc
chmod 600 ~/.netrc
```
### 2. **Comprehensive Metric Logging**
The new WandB hook logs:
- **Training metrics**: loss, IoU, precision, learning rate
- **Validation metrics**: IoU at different thresholds, J-index
- **System metrics**: batch time, data loading time
- **Gradient information**: gradient norms (when clipping is enabled)
- **Learning rate tracking**: per parameter group
### 3. **Custom Metric Definitions**
Metrics are properly namespaced and defined:
```python
wandb.define_metric("epoch")
wandb.define_metric("training/*", step_metric="step")
wandb.define_metric("validation/*", step_metric="epoch")
```
### 4. **Visualization Improvements**
To see proper graphs in WandB:
1. **Loss Curves**: Automatically tracked under `training/loss`
2. **Learning Rate**: Visualized under `training/lr` with scheduler behavior
3. **IoU Progress**: Both training and validation IoU are tracked
4. **Custom Metrics**: Grasp-specific metrics under `training/grasp_*`
### 5. **Configuration in YAML**
Add to your config file:
```yaml
wandb:
project: "language-guided-robotic-grasping"
entity: "your-team" # Optional
tags: ["experiment", "baseline"]
mode: "online" # or "offline" for local logging
resume: "allow" # or "must" to resume runs
```
### 6. **Debugging WandB Issues**
If metrics aren't showing:
1. **Check initialization**: Ensure WandB is initialized only on rank 0
2. **Verify logging**: Check that metrics are logged with proper step counts
3. **Namespace metrics**: Use proper prefixes (training/, validation/)
4. **Sync issues**: Run `wandb sync` if in offline mode
### 7. **Example Dashboard Setup**
Create custom charts in WandB:
1. **Training Overview**: Group loss, IoU, and LR
2. **Validation Metrics**: Precision at different thresholds
3. **Grasp Performance**: J-index trends
4. **System Performance**: Batch processing times
## Migration from Old Scripts
To update existing scripts:
1. Remove hardcoded API keys
2. Add WandB hook to trainer
3. Update metric names to follow naming convention
4. Add custom metric definitions
5. Configure project settings in YAML |