Hyperparameters Used During Training Are Not Documented

#13
by AnOnlineUser78 - opened

The hyperparameters used during the various stages of training are not fully documented. While the LR can be recovered from the W&B dashboard, others such as weight decay, etc. cannot.

Would it be possible to provide these either in the model card or elsewhere?

Tagging @hunterhector in case this was missed.

Institute of Foundation Models org

Hi, @AnOnlineUser78 ,

Yes indeed. I checked our code it is a little bit unclear too. Since the process is a bit complicated, we need to share a little more on data, master script etc to complete it without making it hard to read.

We are just wrapping up the paper, which should have a lot more details. However, I do agree we need to provide a table somewhere.

@junchen01 @onenoc @desaifan-mbzuai , suggestions on how we can structure this?

Institute of Foundation Models org

Thanks for raising this, and sorry for the gap: for K2-Horizon-0.9B, pretraining used a global batch size of 8,388,608 tokens for 600,000 steps (~5.03T tokens) with peak LR 8e-4, 3,000 warmup steps, and weight decay 0.1, followed by two context-extension stages (stage1, peak LR 2e-4, weight decay 0.1; stage2: peak LR 5e-5, weight decay 0.06), both with RoPE base 1M, and we’ll add all of this to the model card along with the coming paper.

Thanks @chen11003 -- I had added a comment to an 375B-A32B model discussion with the same question. Could you possibly share the same statistics for that model?

Thanks to the team for releasing such an open model!

Sign up or log in to comment