Relate cell values to checkpoint losses
Browse files
README.md
CHANGED
|
@@ -19,7 +19,9 @@ There are 174 runs, one folder each, named `<arm>-<size>-u<U/N>-f<F>`:
|
|
| 19 |
top-layer state, trained with two passes on 85% of the updates.
|
| 20 |
- **Sizes:** 60M, 134M, 271M and 778M parameters. The paper fits its law on 60M to 271M and holds 778M out.
|
| 21 |
- **Grid:** U/N is unique training tokens per parameter, and F is FLOPs per unique token in Transformer epochs.
|
| 22 |
-
- **Choice of run:** each folder is the best learning rate and weight decay of its cell.
|
|
|
|
|
|
|
| 23 |
- **Seeds:** `-s1` and `-s2` are the extra seeds at 134M, U/N 12.5 and 100.
|
| 24 |
|
| 25 |
Each folder holds:
|
|
|
|
| 19 |
top-layer state, trained with two passes on 85% of the updates.
|
| 20 |
- **Sizes:** 60M, 134M, 271M and 778M parameters. The paper fits its law on 60M to 271M and holds 778M out.
|
| 21 |
- **Grid:** U/N is unique training tokens per parameter, and F is FLOPs per unique token in Transformer epochs.
|
| 22 |
+
- **Choice of run:** each folder is the best learning rate and weight decay of its cell. The paper's cell value is the
|
| 23 |
+
minimum of a quadratic fitted to all of the cell's runs (the best run when that minimum lies outside the grid), so it
|
| 24 |
+
can differ from the checkpoint's loss by up to 0.002.
|
| 25 |
- **Seeds:** `-s1` and `-s2` are the extra seeds at 134M, U/N 12.5 and 100.
|
| 26 |
|
| 27 |
Each folder holds:
|