ArchSpace-Collection/NCP_ArchPreview_dolma3_8.9B_Stage2_v1
Text Generation • 9B • Updated • 244 • 4
Salwa Zeitoun
Salwa-Zeitoun
AI & ML interests
Computer Vision. Diffusion Models. Deep Learning. LLM. Neural Networks
Recent Activity
liked a model about 1 month ago
ArchSpace-Collection/NCP_ArchPreview_dolma3_8.9B_Stage2_v1 liked a model almost 2 years ago
mistralai/Ministral-8B-Instruct-2410 updated a collection about 2 years ago
Reasoning CapabilitiesOrganizations
liked a
model about 1 month ago
liked a
model almost 2 years ago
updated a
collection about 2 years ago
liked a
model about 2 years ago
upvoted 2
papers over 2 years ago
updated a
collection over 2 years ago
upvoted a paper over 2 years ago
updated 2
collections over 2 years ago
upvoted a paper over 2 years ago
reacted to loubnabnl's post with ❤️ over 2 years ago
Post
6,550
We've just published a detailed blog post on the creation of Cosmopedia dataset. We hope this will provide insights about generating synthetic data at scale for pre-training.
https://huggingface.co/blog/cosmopedia
Here are some key takeaways:
🎯 Prompt curation is crucial: we want to cover many topics with few duplicates.
📚 You can leverage various resources for diversity: using different seed data, generation formats, and target audiences.
⚙️ The importance of a good technical stack: for scalable generations with tools like llm-swarm and fast model training and evaluation.
Have a good read!
https://huggingface.co/blog/cosmopedia
Here are some key takeaways:
🎯 Prompt curation is crucial: we want to cover many topics with few duplicates.
📚 You can leverage various resources for diversity: using different seed data, generation formats, and target audiences.
⚙️ The importance of a good technical stack: for scalable generations with tools like llm-swarm and fast model training and evaluation.
Have a good read!
- 1 reply
liked a
model over 2 years ago
upvoted a paper over 2 years ago
reacted to akhaliq's post with ❤️ over 2 years ago
Post
Neural Network Diffusion
Neural Network Diffusion (2402.13144)
Diffusion models have achieved remarkable success in image and video generation. In this work, we demonstrate that diffusion models can also generate high-performing neural network parameters. Our approach is simple, utilizing an autoencoder and a standard latent diffusion model. The autoencoder extracts latent representations of a subset of the trained network parameters. A diffusion model is then trained to synthesize these latent parameter representations from random noise. It then generates new representations that are passed through the autoencoder's decoder, whose outputs are ready to use as new subsets of network parameters. Across various architectures and datasets, our diffusion process consistently generates models of comparable or improved performance over trained networks, with minimal additional cost. Notably, we empirically find that the generated models perform differently with the trained networks. Our results encourage more exploration on the versatile use of diffusion models.
Neural Network Diffusion (2402.13144)
Diffusion models have achieved remarkable success in image and video generation. In this work, we demonstrate that diffusion models can also generate high-performing neural network parameters. Our approach is simple, utilizing an autoencoder and a standard latent diffusion model. The autoencoder extracts latent representations of a subset of the trained network parameters. A diffusion model is then trained to synthesize these latent parameter representations from random noise. It then generates new representations that are passed through the autoencoder's decoder, whose outputs are ready to use as new subsets of network parameters. Across various architectures and datasets, our diffusion process consistently generates models of comparable or improved performance over trained networks, with minimal additional cost. Notably, we empirically find that the generated models perform differently with the trained networks. Our results encourage more exploration on the versatile use of diffusion models.
upvoted a collection over 2 years ago
updated a
collection over 2 years ago