English
scaling-laws
feedback-transformer
HuskyDoge commited on
Commit
b28458d
·
verified ·
1 Parent(s): d2d71fe

Relate cell values to checkpoint losses

Browse files
Files changed (1) hide show
  1. README.md +3 -1
README.md CHANGED
@@ -19,7 +19,9 @@ There are 174 runs, one folder each, named `<arm>-<size>-u<U/N>-f<F>`:
19
  top-layer state, trained with two passes on 85% of the updates.
20
  - **Sizes:** 60M, 134M, 271M and 778M parameters. The paper fits its law on 60M to 271M and holds 778M out.
21
  - **Grid:** U/N is unique training tokens per parameter, and F is FLOPs per unique token in Transformer epochs.
22
- - **Choice of run:** each folder is the best learning rate and weight decay of its cell.
 
 
23
  - **Seeds:** `-s1` and `-s2` are the extra seeds at 134M, U/N 12.5 and 100.
24
 
25
  Each folder holds:
 
19
  top-layer state, trained with two passes on 85% of the updates.
20
  - **Sizes:** 60M, 134M, 271M and 778M parameters. The paper fits its law on 60M to 271M and holds 778M out.
21
  - **Grid:** U/N is unique training tokens per parameter, and F is FLOPs per unique token in Transformer epochs.
22
+ - **Choice of run:** each folder is the best learning rate and weight decay of its cell. The paper's cell value is the
23
+ minimum of a quadratic fitted to all of the cell's runs (the best run when that minimum lies outside the grid), so it
24
+ can differ from the checkpoint's loss by up to 0.002.
25
  - **Seeds:** `-s1` and `-s2` are the extra seeds at 134M, U/N 12.5 and 100.
26
 
27
  Each folder holds: