Table-only prerenders, progress table image, no duplicated tables

.gitattributes

@@ -38,3 +38,4 @@ charts/average_table.png filter=lfs diff=lfs merge=lfs -text
 charts/avg_7tasks.png filter=lfs diff=lfs merge=lfs -text
 charts/benchmark_table.png filter=lfs diff=lfs merge=lfs -text
 charts/compute_table.png filter=lfs diff=lfs merge=lfs -text
+charts/progress_table.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -63,13 +63,13 @@ Author: <b>Dominik Andrzejewski</b>
 
 ## Training lineage
 
-| # | Stage | Tokens | What happened |
-|---|---|---:|---|
-| 1 | **Pretraining (75M)** | 4.0B | A 75M-parameter model was pretrained from scratch on English text, single pass. |
-| 2 | **Growth to 100M** | 0.61B | Grown from 12 to 18 layers (identity-initialised) and pretrained further on new documents. |
-| 3 | **Chat fine-tune** | 0.18B | Fine-tuned for chat. These weights are the starting point of the expansion. |
-| 4 | **Expansion** | about 3.3B | Continued pretraining on new documents from the same sources, to step 15,000 with a 262k-token batch, on a cloud A100 in two chunks. |
-| 5 | **Cooldown** | 1.05B | The learning rate was decayed to its minimum over 4,000 steps on a second, unseen corpus, starting from step 15,000. **This is the checkpoint released here.** |
+1. **Pretraining (75M):** a 75M-parameter model was pretrained from scratch on English text, single pass.
+2. **Growth to 100M:** grown from 12 to 18 layers (identity-initialised) and pretrained further on new documents.
+3. **Chat fine-tune:** fine-tuned for chat. These weights are the starting point of the expansion.
+4. **Expansion:** continued pretraining on new documents from the same sources, to step 15,000 with a 262k-token batch.
+5. **Cooldown:** the learning rate was decayed to its minimum over 4,000 steps on a second, unseen corpus, starting from step 15,000. **This is the checkpoint released here.**
+
+Tokens and hardware for each stage are in [Compute](#compute).
 
 A separate local cooldown (170 steps from step 13,000) was an experiment. It is **not** in this release's lineage.
 
@@ -109,7 +109,7 @@ The 75M pretraining (stage 1) used the same sources with more FineWeb-Edu (37%)
 
 Chat data was filtered: refusals, boilerplate, other models' identities and over-length chats were dropped, and any sexual content involving minors was removed.
 
-### Evaluation
+### Evaluation benchmarks
 
 | Benchmark | Dataset |
 |---|---|
@@ -158,14 +158,9 @@ The highlighted cell in each column is the best score in that column, whichever
 
 ### Progress across checkpoints of this model
 
-| Checkpoint | Val loss | arc_challenge | arc_easy | hellaswag | openbookqa | piqa | sciq | winogrande | Avg (7) | BLiMP |
-|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
-| Expansion step 9,500 | 2.7780 | 27.6 | 52.8 | **29.8** | 27.6 | 62.4 | 71.0 | 51.6 | 46.11 | 76.88 |
-| Expansion step 13,000 | 2.7722 | 28.8 | 49.8 | 28.4 | 29.0 | 61.6 | 70.4 | 51.8 | 45.69 | 78.03 |
-| Expansion step 15,000 (cooldown start) | 2.7730 | 26.8 | 52.6 | 29.0 | **30.0** | 62.0 | 69.8 | 53.0 | 46.17 | 77.66 |
-| **Cooldown step 4,000 (this release)** | **2.6531** | **29.2** | **54.0** | 29.4 | 29.4 | **62.8** | **72.2** | **54.0** | **47.29** | **78.70** |
+![Progress across checkpoints](charts/progress_table.png)
 
-Bold marks the best value in each column (lowest for val loss). Validation loss is measured on the held-out split of the first expansion corpus, which the model trained on. It is not a clean test, so the cooldown's lower loss should not be read as general improvement on its own. The benchmark average rose after the cooldown and is the more useful signal.
+The highlighted cell in each column is the best value (lowest for val loss). Validation loss is measured on the held-out split of the first expansion corpus, which the model trained on. It is not a clean test, so the cooldown's lower loss should not be read as general improvement on its own. The benchmark average rose after the cooldown and is the more useful signal.
 
 ## Compute