Update README.md
Browse files
README.md
CHANGED
|
@@ -35,4 +35,30 @@ Rather than training one model, we trained multiple models, each on a different
|
|
| 35 |
- RMS Norm Eps: `1e-06`
|
| 36 |
- Pretraining Tp: `1`
|
| 37 |
- Use Cache: `false`
|
| 38 |
-
- **Total Parameters**: `1,460,416`
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
- RMS Norm Eps: `1e-06`
|
| 36 |
- Pretraining Tp: `1`
|
| 37 |
- Use Cache: `false`
|
| 38 |
+
- **Total Parameters**: `1,460,416`
|
| 39 |
+
|
| 40 |
+
## Training Setup
|
| 41 |
+
|
| 42 |
+
- Epochs: '1'
|
| 43 |
+
- Max Steps: `-1.0`
|
| 44 |
+
- Batch Size: `400`
|
| 45 |
+
- Sequence Length: `256`
|
| 46 |
+
- Gradient Accumulation: `2`
|
| 47 |
+
- Gradient Clipping: `1.0`
|
| 48 |
+
- Gradient Checkpointing: `true`
|
| 49 |
+
- Learning Rate: `3e-3`
|
| 50 |
+
- Eval Split: `0.001`
|
| 51 |
+
- Weight Decay: `0.01`
|
| 52 |
+
- Optimizer: `AdamW`
|
| 53 |
+
- AdamW Betas: `(0.9, 0.95)`
|
| 54 |
+
- AdamW Eps: `1e-8`
|
| 55 |
+
- Scheduler: `WSD`
|
| 56 |
+
- WSD Warmup Ratio: `0.015`
|
| 57 |
+
- WSD Stable Ratio: `0.78`
|
| 58 |
+
- WSD Decay Ratio: `0.20`
|
| 59 |
+
- WSD Minium LR Ratio: `0.0`
|
| 60 |
+
- WSD Number of Cycles: `0.5`
|
| 61 |
+
- DType: `float16`
|
| 62 |
+
- Torch.Compile: `false`
|
| 63 |
+
- DataLoader Workers: `4`
|
| 64 |
+
- Seed: `311`
|