moonshotai/Kimi-K3
Image-Text-to-Text ⢠2.8T ⢠Updated ⢠1.7M ⢠⢠11.5k
đ In a Training Loop
Phuc Nguyen
neuralink
AI & ML interests
distributed training @nous research. ex-nanotron @huggingface
Recent Activity
liked a model 2 months ago
moonshotai/Kimi-K3 upvoted an article 2 months ago
Security incident disclosure â July 2026 upvoted a paper 4 months ago
Long Context Pre-Training with Lighthouse AttentionOrganizations
liked a
model 2 months ago
upvoted an article 2 months ago
Article
Security incident disclosure â July 2026
system
⢠⢠798 upvoted a paper 4 months ago
upvoted a paper 5 months ago
published a
Space 5 months ago
upvoted a paper 6 months ago
liked a
model 9 months ago
liked a
model 10 months ago
upvoted a paper 10 months ago
reacted to nouamanetazi's post with đ¤ 11 months ago
Post
5205
After training đđŚđ¨đĽđđđ on đđđ đđđđđŹ for nearly a month, I've come to realize something most people overlook: đ˘đ§đđŤđđŹđđŤđŽđđđŽđŤđ đ˘đŹ đđĄđ đŚđđ¤đ-đ¨đŤ-đđŤđđđ¤ đđđđđ¨đŤ đ˘đ§ đđđ đđŤđđ˘đ§đ˘đ§đ . đĽ
Everyone talks about model architecture and data quality. And yes, those matter immensely. But here's what nobody tells you: when your training run fails at 2 AM because of mysterious đđđđ đđŤđŤđ¨đŤđŹ, or when your expensive GPU cluster is running at đđ% đđđđ˘đđ˘đđ§đđ˛, the problem isn't your model. It's most probably a đŚđ˘đŹđŽđŹđ đ¨đ đđĄđ đĄđđŤđđ°đđŤđ. đ ď¸
Questions that seemed simple but had no clear answers: Why is đđ¨đ đđŤđđ˘đ§đ˘đ§đ đŹđĽđ¨đ°đđŤ đđĄđđ§ đđđ§đŹđ đŚđ¨đđđĽđŹ? Which đđđđ đđĽđđ đŹ should we actually set? How often should we checkpoint without killing throughput?
That's why we built đđĄđ đđŚđ¨đĽ đđŤđđ˘đ§đ˘đ§đ đđĽđđ˛đđ¨đ¨đ¤ đ: a complete guide covering everything from model architecture and data curation to the SmolLM3 training marathon, post-training techniques, and crucially, the đ˘đ§đđŤđđŹđđŤđŽđđđŽđŤđ đĽđđ˛đđŤ that most teams get wrong.
We validated real vs theoretical bandwidth across the entire stack: đđđđ đĄđ˘đđđ˘đ§đ đ đđ/đŹ, đđđđ˘đ§đ¤ đ.đ đŤđđđđĄđ˘đ§đ đđđ đđ/đŹ, đđđđ đđđ§đ đđ đđ.đ đđ/đŹ. Then we ran collective operations across đđđ đđđđŹ (16 nodes, 8xH100s each) and measured how performance degrades at scale: all-reduce drops from đđđ đđ/đŹ on a single node to đđđ-đđđ đđ/đŹ across 16 nodes.
If you've ever wondered why your training runs are slower than they should be, or you're planning to scale up and want to avoid expensive mistakes, this guide might save you weeks of debugging.
đđĄđ đđŚđ¨đĽ đđŤđđ˘đ§đ˘đ§đ đđĽđđ˛đđ¨đ¨đ¤: https://lnkd.in/e5MKXUHS
Shared with â¤ď¸ by the HuggingFace team
Everyone talks about model architecture and data quality. And yes, those matter immensely. But here's what nobody tells you: when your training run fails at 2 AM because of mysterious đđđđ đđŤđŤđ¨đŤđŹ, or when your expensive GPU cluster is running at đđ% đđđđ˘đđ˘đđ§đđ˛, the problem isn't your model. It's most probably a đŚđ˘đŹđŽđŹđ đ¨đ đđĄđ đĄđđŤđđ°đđŤđ. đ ď¸
Questions that seemed simple but had no clear answers: Why is đđ¨đ đđŤđđ˘đ§đ˘đ§đ đŹđĽđ¨đ°đđŤ đđĄđđ§ đđđ§đŹđ đŚđ¨đđđĽđŹ? Which đđđđ đđĽđđ đŹ should we actually set? How often should we checkpoint without killing throughput?
That's why we built đđĄđ đđŚđ¨đĽ đđŤđđ˘đ§đ˘đ§đ đđĽđđ˛đđ¨đ¨đ¤ đ: a complete guide covering everything from model architecture and data curation to the SmolLM3 training marathon, post-training techniques, and crucially, the đ˘đ§đđŤđđŹđđŤđŽđđđŽđŤđ đĽđđ˛đđŤ that most teams get wrong.
We validated real vs theoretical bandwidth across the entire stack: đđđđ đĄđ˘đđđ˘đ§đ đ đđ/đŹ, đđđđ˘đ§đ¤ đ.đ đŤđđđđĄđ˘đ§đ đđđ đđ/đŹ, đđđđ đđđ§đ đđ đđ.đ đđ/đŹ. Then we ran collective operations across đđđ đđđđŹ (16 nodes, 8xH100s each) and measured how performance degrades at scale: all-reduce drops from đđđ đđ/đŹ on a single node to đđđ-đđđ đđ/đŹ across 16 nodes.
If you've ever wondered why your training runs are slower than they should be, or you're planning to scale up and want to avoid expensive mistakes, this guide might save you weeks of debugging.
đđĄđ đđŚđ¨đĽ đđŤđđ˘đ§đ˘đ§đ đđĽđđ˛đđ¨đ¨đ¤: https://lnkd.in/e5MKXUHS
Shared with â¤ď¸ by the HuggingFace team
liked a
Space 11 months ago
liked a
model 11 months ago
liked a
model about 1 year ago
upvoted an article about 1 year ago
Article
Arc Virtual Cell Challenge: A Primer
FL33TW00D-HF, abhinadduri
⢠⢠68 upvoted an article over 1 year ago
Article
The Transformers Library: standardizing model definitions
- +2
lysandre, ArthurZ, pcuenq, julien-c
⢠⢠124 liked a
model over 1 year ago
upvoted 2
articles over 1 year ago
Article
You could have designed state of the art positional encoding
FL33TW00D-HF
⢠⢠502 Article
Welcome Llama 4 Maverick & Scout on Hugging Face
- +5
burtenshaw, reach-vb, pcuenq, clem, rajatarya, jsulz, lysandre
⢠⢠150 liked a
dataset over 1 year ago
upvoted a paper over 1 year ago