YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
Model Card for Model ID
Model Details
Model Description
SmaLLM is a compact causal language model implemented entirely from scratch in PyTorch. It follows modern architectural conventions inspired by LLaMA and other contemporary small language models, avoiding legacy Transformer components (absolute positional embeddings, LayerNorm, standard ReLU FFN).
The model was designed as a learning project to demonstrate understanding of modern LM architecture components and training infrastructure — not for production deployment.
- Developed by: Atharva
- Model type: Causal Language Model(AutoRegressive)
- Language(s) (NLP): English
- Training Data: Web text (essential-web corpus, ~500K tokens)
- Tokenizer: vukrosic/essential-web-16k-tokenizer
- Framework: Pytorch
What This Model IS For
- ✅ Learning how modern LM architectures work (RoPE, SwiGLU, Flash Attention, RMSNorm)
- ✅ Understanding training infrastructure (AMP, LR schedules, checkpointing, logging)
- ✅ Experimenting with architecture modifications and hyperparameter sweeps
- ✅ Educational demonstrations and blog posts
What This Model IS NOT For
- ❌ Production text generation
- ❌ Chatbot or conversational AI
- ❌ Factual question answering
- ❌ Any task requiring coherent, reliable output
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support