# 1. Training data

**First Pre Training:**

HuggingFaceFW/fineweb-edu
HuggingFaceTB/cosmopedia-v2

**Next Pre Training:**

HuggingFaceFW/fineweb-edu
HuggingFaceTB/cosmopedia-v2
AI-MO/NuminaMath-CoT
flytech/python-codes-25k
Magpie-Align/Magpie-Pro-300K-Filtered
camel-ai/physics
camel-ai/chemistry
camel-ai/math
nvidia/OpenMathInstruct-2
bespokelabs/Bespoke-Stratos-17k

# 2. Training tokens

First: ~970M

- HuggingFaceFW/fineweb-edu:61%
- HuggingFaceTB/cosmopedia-v2:39%

- Approx Loss: **3.66**

Next: ~101M

- HuggingFaceFW/fineweb-edu, HuggingFaceTB/cosmopedia-v2: 25%
- AI-MO/NuminaMath-CoT: 8%
- camel-ai/... : 25%
- nvidia/OpenMathInstruct-2: 20%
- bespokelabs/Bespoke-Stratos-17k: 12%

- Approx Loss: **2.86**

# 3. Training GPUs

- Colab T4 (paid tier): 97%
  - First: 90.3%
  - Next: 6.7%
- RTX 4060 x1: 3%
  - First: 0%
  - Next: 3%

# 4. Benchmark Leaks

- ARC Series(Easy, Challenge, AGI): ✅no leak
- MMLU: -
- GPQA: ✅no leak
- BLiMP: ✅no leak
- Wikitext-2 PPL: not leak?
- HellaSwwag: ✅no leak
- FineWeb PPL: ❌Leaked (dataset)

# 5. Training config

First: 

  - learning rate: 1e-3 ~ 5e-4
  - batch: 16(T4) / 16(4060)
  - ctx: 512 / 1024

Next:

  - learning rate: 2e-4 ~ 8e-5
  - batch: 16(T4)
  - ctx: 512

That's all.