Card: bands written before the run; end of second window specified after 7 of 10 values
Browse files
README.md
CHANGED
|
@@ -49,17 +49,19 @@ Data: [`analysis/scores-history.csv`](analysis/scores-history.csv) (step, tokens
|
|
| 49 |
of three GPUs computed the row; each GPU was checked against a known point before use). 1.00 at 1.26 B and 1.42 B tokens
|
| 50 |
is already in its learning-rate decay; 1.01 at the same token counts is not.
|
| 51 |
|
| 52 |
-
## Training loss vs.
|
| 53 |
|
| 54 |
-
Before the run,
|
| 55 |
(continuation of the 1.00 curve from step 1050):
|
| 56 |
|
| 57 |
-
| steps |
|
| 58 |
|---|---|---:|
|
| 59 |
| 1700–1800 (constant lr, pack-00) | 2.692 – 2.882 | 2.6977 |
|
| 60 |
| 2770–2860 (end of the decay, pack-01) | 2.367 – 2.660 | 2.5083 |
|
| 61 |
|
| 62 |
-
Both
|
|
|
|
|
|
|
| 63 |
scenarios.
|
| 64 |
|
| 65 |
## Usage
|
|
|
|
| 49 |
of three GPUs computed the row; each GPU was checked against a known point before use). 1.00 at 1.26 B and 1.42 B tokens
|
| 50 |
is already in its learning-rate decay; 1.01 at the same token counts is not.
|
| 51 |
|
| 52 |
+
## Training loss vs. bands written before the run
|
| 53 |
|
| 54 |
+
Before the run, bands for the mean training loss were written down from the training-loss scenarios on the 1.00 card
|
| 55 |
(continuation of the 1.00 curve from step 1050):
|
| 56 |
|
| 57 |
+
| steps | band written before the run | measured mean of logged values |
|
| 58 |
|---|---|---:|
|
| 59 |
| 1700–1800 (constant lr, pack-00) | 2.692 – 2.882 | 2.6977 |
|
| 60 |
| 2770–2860 (end of the decay, pack-01) | 2.367 – 2.660 | 2.5083 |
|
| 61 |
|
| 62 |
+
Both bands were written before the run. The steps of the first window were also fixed before the run; the exact end of
|
| 63 |
+
the second window (the last 10 logged values, steps 2770–2860) was specified when 7 of those 10 values were already
|
| 64 |
+
logged. Both measurements fall inside their bands, the first one at the lower edge. This is one run, not a test of the
|
| 65 |
scenarios.
|
| 66 |
|
| 67 |
## Usage
|