SURELYNOTVAIN commited on
Commit
71746ea
·
verified ·
1 Parent(s): fac4ad0

MP-01 100M base: prerendered compute and average tables placed through the card

Browse files
.gitattributes CHANGED
@@ -37,3 +37,4 @@ manprot-logo.png filter=lfs diff=lfs merge=lfs -text
37
  charts/average_table.png filter=lfs diff=lfs merge=lfs -text
38
  charts/avg_7tasks.png filter=lfs diff=lfs merge=lfs -text
39
  charts/benchmark_table.png filter=lfs diff=lfs merge=lfs -text
 
 
37
  charts/average_table.png filter=lfs diff=lfs merge=lfs -text
38
  charts/avg_7tasks.png filter=lfs diff=lfs merge=lfs -text
39
  charts/benchmark_table.png filter=lfs diff=lfs merge=lfs -text
40
+ charts/compute_table.png filter=lfs diff=lfs merge=lfs -text
README.md CHANGED
@@ -50,16 +50,7 @@ Mixture of the expansion stages, by share of tokens: FineWeb-Edu 34%, Wikipedia
50
 
51
  Hours are wall-clock GPU time from the run logs. Costs in credits are not listed.
52
 
53
- | Stage | Hardware | GPU hours | Tokens |
54
- |---|---|---:|---:|
55
- | Pretraining, 75M base | local AMD Radeon RX 7600 (8 GB) | not totalled from logs | 4.0B |
56
- | Growth to 100M, first stage | local AMD Radeon RX 7600 | 3.94 (one segment) | 0.61B |
57
- | Chat fine-tune | local AMD Radeon RX 7600 | 2.78 | 0.18B |
58
- | Expansion, chunk 1 (steps 2,314 to 14,230) | cloud A100 40 GB | 4.00 | 3.1B new |
59
- | Expansion, chunk 2 (steps 14,231 to 15,000) | cloud A100 40 GB | ~0.35 | 0.2B new |
60
- | Cooldown, 4,000 steps | cloud H200 141 GB | 0.68 | 1.05B |
61
- | Benchmarks of released checkpoints | local GPU | a few minutes each | |
62
- | Comparison benchmarks (7 models) | cloud H200 | a few minutes each, run in parallel | |
63
 
64
  Note: the local GPU's elapsed times are per recorded segment; long runs were restarted, so the 75M pretraining total is not reliable from the logs. The expansion continued beyond step 15,000 on the cloud and that part is not included in this checkpoint.
65
 
@@ -69,6 +60,7 @@ Scored by this repository's harness (`scripts/eval_checkpoint.py`, `scripts/eval
69
 
70
  ### Against other models
71
 
 
72
  ![Average benchmark score](charts/average_table.png)
73
 
74
  ![Benchmark results](charts/benchmark_table.png)
 
50
 
51
  Hours are wall-clock GPU time from the run logs. Costs in credits are not listed.
52
 
53
+ ![Compute used for this release](charts/compute_table.png)
 
 
 
 
 
 
 
 
 
54
 
55
  Note: the local GPU's elapsed times are per recorded segment; long runs were restarted, so the 75M pretraining total is not reliable from the logs. The expansion continued beyond step 15,000 on the cloud and that part is not included in this checkpoint.
56
 
 
60
 
61
  ### Against other models
62
 
63
+
64
  ![Average benchmark score](charts/average_table.png)
65
 
66
  ![Benchmark results](charts/benchmark_table.png)
charts/average_table.png CHANGED

Git LFS Details

  • SHA256: 937e2ccefef2f10d144e8b0a0adb49624f35a84581104644c72a1dd54293550c
  • Pointer size: 131 Bytes
  • Size of remote file: 101 kB

Git LFS Details

  • SHA256: fe8414c3c1c1b7af8aea17dfe6c54ce530461655aa746714f09b8709b3771500
  • Pointer size: 131 Bytes
  • Size of remote file: 102 kB
charts/compute_table.png ADDED

Git LFS Details

  • SHA256: 033b27aa429b05dd60b15ddebd1fb62335c67a244fb68e137c735fd41f6f477e
  • Pointer size: 131 Bytes
  • Size of remote file: 176 kB