Pranav Upadhyaya's picture

Pranav Upadhyaya PRO

pranavupadhyaya52
9

AI & ML interests

None yet

Recent Activity

posted an update 4 days ago
Experimenting with Continued Pretraining of Small Language Models on Niche Domains I've been experimenting with continued pretraining of small language models (SLMs) to explore how effectively we can adapt existing foundational models to niche domains using relatively focused datasets. For this experiment, I took Llama 3.2 1B and continued its pretraining on approximately 43,000 educational Python samples, with the goal of exploring domain adaptation towards Python programming and educational content. The project is now available as Fermicode-1B. What's interesting is that, even without formal benchmarking or quantitative evaluation yet, the initial generations have been surprisingly promising. It's encouraging to see how a relatively small model responds to continued exposure to a focused domain corpus. A few things I want to explore further: - How much domain-specific knowledge can continued pretraining introduce into a 1B parameter model? - How does the model compare against the original base model on Python-related tasks? - How much does dataset composition and quality influence domain adaptation? - Where does continued pretraining end, and instruction fine-tuning begin, in building a useful domain-specific SLM? I haven't drawn any performance conclusions yet. Proper benchmarking and comparative evaluation are still on my to-do list. For now, this is an interesting foundational ML experiment that I wanted to share with the community. πŸ”— Explore Fermicode-1B πŸ€— Model Repository: https://huggingface.co/pranavupadhyaya52/Fermicode-1B πŸš€ Try the Model on Hugging Face Spaces: https://huggingface.co/spaces/pranavupadhyaya52/Fermicode-1B-Instruct-Demo A little about me I'm an ML Engineer with previous experience at Turing.com. I'm particularly interested in experimenting with core ML concepts, foundational model development, and understanding what's possible with smaller, resource-efficient language models. I'd love to hear your thoughts and feedback.
published a Space 4 days ago
pranavupadhyaya52/Fermicode-1B-Instruct-Demo
View all activity

Organizations

Fermac AI Systems's profile picture