Post
32
Building a frontier model doesn't strictly require massive superclusters—it demands continuous engineering, smart strategy, strong data pipelines, and sheer determination.
For the past 17 days, my 0.7B Mixture-of-Experts (MoE) model has been in a continuous pre-training phase, and today we hit a major milestone.
Key Progress Updates --
Pre-Training Duration: 17 days of uninterrupted run
Target Steps: 211,033
Current Steps: 25,030 (25,000+ steps completed)
Tokens Processed: 750M+ tokens passed through the network
Architecture: 0.7B parameter footprint, 32 local experts (2 active per token), 262k vocabulary
All checkpoints, logs, and datasets are tracked live in the Collection -
https://hf.co/collections/Shrijanagain/how-to-build-a-frontier-model-from-zero
25k steps down. Scaling continuous pre-training depth next. Let me know your thoughts in the comments!
For the past 17 days, my 0.7B Mixture-of-Experts (MoE) model has been in a continuous pre-training phase, and today we hit a major milestone.
Key Progress Updates --
Pre-Training Duration: 17 days of uninterrupted run
Target Steps: 211,033
Current Steps: 25,030 (25,000+ steps completed)
Tokens Processed: 750M+ tokens passed through the network
Architecture: 0.7B parameter footprint, 32 local experts (2 active per token), 262k vocabulary
All checkpoints, logs, and datasets are tracked live in the Collection -
https://hf.co/collections/Shrijanagain/how-to-build-a-frontier-model-from-zero
25k steps down. Scaling continuous pre-training depth next. Let me know your thoughts in the comments!
