Training.md

1. Training data

First Pre Training:

HuggingFaceFW/fineweb-edu HuggingFaceTB/cosmopedia-v2

Next Pre Training:

HuggingFaceFW/fineweb-edu HuggingFaceTB/cosmopedia-v2 AI-MO/NuminaMath-CoT flytech/python-codes-25k Magpie-Align/Magpie-Pro-300K-Filtered camel-ai/physics camel-ai/chemistry camel-ai/math nvidia/OpenMathInstruct-2 bespokelabs/Bespoke-Stratos-17k

2. Training tokens

First: ~970M

• HuggingFaceFW/fineweb-edu:61%

• HuggingFaceTB/cosmopedia-v2:39%

• Approx Loss: 3.66

Next: ~101M

• HuggingFaceFW/fineweb-edu, HuggingFaceTB/cosmopedia-v2: 25%

• AI-MO/NuminaMath-CoT: 8%

• camel-ai/... : 25%

• nvidia/OpenMathInstruct-2: 20%

• bespokelabs/Bespoke-Stratos-17k: 12%

• Approx Loss: 2.86

3. Training GPUs

• Colab T4 (paid tier): 97%
  • First: 90.3%
  • Next: 6.7%
• RTX 4060 x1: 3%
  • First: 0%
  • Next: 3%

4. Benchmark Leaks

• ARC Series(Easy, Challenge, AGI): ✅no leak
• MMLU: -
• GPQA: ✅no leak
• BLiMP: ✅no leak
• Wikitext-2 PPL: not leak?
• HellaSwwag: ✅no leak
• FineWeb PPL: ❌Leaked (dataset)

5. Training config

First:

• learning rate: 1e-3 ~ 5e-4
• batch: 16(T4) / 16(4060)
• ctx: 512 / 1024

Next:

• learning rate: 2e-4 ~ 8e-5
• batch: 16(T4)
• ctx: 512

That's all.