MP-01 100M base: prerendered compute and average tables placed through the card
Browse files- .gitattributes +1 -0
- README.md +2 -10
- charts/average_table.png +2 -2
- charts/compute_table.png +3 -0
.gitattributes
CHANGED
|
@@ -37,3 +37,4 @@ manprot-logo.png filter=lfs diff=lfs merge=lfs -text
|
|
| 37 |
charts/average_table.png filter=lfs diff=lfs merge=lfs -text
|
| 38 |
charts/avg_7tasks.png filter=lfs diff=lfs merge=lfs -text
|
| 39 |
charts/benchmark_table.png filter=lfs diff=lfs merge=lfs -text
|
|
|
|
|
|
| 37 |
charts/average_table.png filter=lfs diff=lfs merge=lfs -text
|
| 38 |
charts/avg_7tasks.png filter=lfs diff=lfs merge=lfs -text
|
| 39 |
charts/benchmark_table.png filter=lfs diff=lfs merge=lfs -text
|
| 40 |
+
charts/compute_table.png filter=lfs diff=lfs merge=lfs -text
|
README.md
CHANGED
|
@@ -50,16 +50,7 @@ Mixture of the expansion stages, by share of tokens: FineWeb-Edu 34%, Wikipedia
|
|
| 50 |
|
| 51 |
Hours are wall-clock GPU time from the run logs. Costs in credits are not listed.
|
| 52 |
|
| 53 |
-
|
| 54 |
-
|---|---|---:|---:|
|
| 55 |
-
| Pretraining, 75M base | local AMD Radeon RX 7600 (8 GB) | not totalled from logs | 4.0B |
|
| 56 |
-
| Growth to 100M, first stage | local AMD Radeon RX 7600 | 3.94 (one segment) | 0.61B |
|
| 57 |
-
| Chat fine-tune | local AMD Radeon RX 7600 | 2.78 | 0.18B |
|
| 58 |
-
| Expansion, chunk 1 (steps 2,314 to 14,230) | cloud A100 40 GB | 4.00 | 3.1B new |
|
| 59 |
-
| Expansion, chunk 2 (steps 14,231 to 15,000) | cloud A100 40 GB | ~0.35 | 0.2B new |
|
| 60 |
-
| Cooldown, 4,000 steps | cloud H200 141 GB | 0.68 | 1.05B |
|
| 61 |
-
| Benchmarks of released checkpoints | local GPU | a few minutes each | |
|
| 62 |
-
| Comparison benchmarks (7 models) | cloud H200 | a few minutes each, run in parallel | |
|
| 63 |
|
| 64 |
Note: the local GPU's elapsed times are per recorded segment; long runs were restarted, so the 75M pretraining total is not reliable from the logs. The expansion continued beyond step 15,000 on the cloud and that part is not included in this checkpoint.
|
| 65 |
|
|
@@ -69,6 +60,7 @@ Scored by this repository's harness (`scripts/eval_checkpoint.py`, `scripts/eval
|
|
| 69 |
|
| 70 |
### Against other models
|
| 71 |
|
|
|
|
| 72 |

|
| 73 |
|
| 74 |

|
|
|
|
| 50 |
|
| 51 |
Hours are wall-clock GPU time from the run logs. Costs in credits are not listed.
|
| 52 |
|
| 53 |
+

|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
|
| 55 |
Note: the local GPU's elapsed times are per recorded segment; long runs were restarted, so the 75M pretraining total is not reliable from the logs. The expansion continued beyond step 15,000 on the cloud and that part is not included in this checkpoint.
|
| 56 |
|
|
|
|
| 60 |
|
| 61 |
### Against other models
|
| 62 |
|
| 63 |
+
|
| 64 |

|
| 65 |
|
| 66 |

|
charts/average_table.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
charts/compute_table.png
ADDED
|
Git LFS Details
|