| # 1. Training data |
|
|
| **First Pre Training:** |
|
|
| HuggingFaceFW/fineweb-edu |
| HuggingFaceTB/cosmopedia-v2 |
|
|
| **Next Pre Training:** |
|
|
| HuggingFaceFW/fineweb-edu |
| HuggingFaceTB/cosmopedia-v2 |
| AI-MO/NuminaMath-CoT |
| flytech/python-codes-25k |
| Magpie-Align/Magpie-Pro-300K-Filtered |
| camel-ai/physics |
| camel-ai/chemistry |
| camel-ai/math |
| nvidia/OpenMathInstruct-2 |
| bespokelabs/Bespoke-Stratos-17k |
|
|
| # 2. Training tokens |
|
|
| First: ~970M |
|
|
| - HuggingFaceFW/fineweb-edu:61% |
| - HuggingFaceTB/cosmopedia-v2:39% |
|
|
| - Approx Loss: **3.66** |
|
|
| Next: ~101M |
|
|
| - HuggingFaceFW/fineweb-edu, HuggingFaceTB/cosmopedia-v2: 25% |
| - AI-MO/NuminaMath-CoT: 8% |
| - camel-ai/... : 25% |
| - nvidia/OpenMathInstruct-2: 20% |
| - bespokelabs/Bespoke-Stratos-17k: 12% |
|
|
| - Approx Loss: **2.86** |
|
|
| # 3. Training GPUs |
|
|
| - Colab T4 (paid tier): 97% |
| - First: 90.3% |
| - Next: 6.7% |
| - RTX 4060 x1: 3% |
| - First: 0% |
| - Next: 3% |
|
|
| # 4. Benchmark Leaks |
|
|
| - ARC Series(Easy, Challenge, AGI): ✅no leak |
| - MMLU: - |
| - GPQA: ✅no leak |
| - BLiMP: ✅no leak |
| - Wikitext-2 PPL: not leak? |
| - HellaSwwag: ✅no leak |
| - FineWeb PPL: ❌Leaked (dataset) |
|
|
| # 5. Training config |
|
|
| First: |
|
|
| - learning rate: 1e-3 ~ 5e-4 |
| - batch: 16(T4) / 16(4060) |
| - ctx: 512 / 1024 |
|
|
| Next: |
|
|
| - learning rate: 2e-4 ~ 8e-5 |
| - batch: 16(T4) |
| - ctx: 512 |
|
|
| That's all. |
