| --- |
| language: en |
| license: apache-2.0 |
| tags: |
| - transformer |
| - pytorch |
| - causal-lm |
| - moe |
| - mixture-of-experts |
| - rish-ai-labs |
| --- |
| |
| # RLLM (Base Model) |
|
|
| ## Model Description |
|
|
| RLLM is a base language model developed by **Rish AI Labs**, an applied artificial intelligence lab focused on LLMs, Generative AI, AI consulting, and research. |
|
|
| This model features a **Mixture of Experts (MoE)** architecture with 16 experts, providing efficient scaling and specialization capabilities. It was trained using identity-focused pretraining to establish a strong foundation for downstream tasks. |
|
|
| ## Key Features |
|
|
| - **Architecture**: Transformer with MoE (16 experts, top-2 routing) |
| - **Parameters**: ~275M total parameters |
| - **Training**: Identity-focused pretraining |
| - **Precision**: FP32 training, optimized for inference |
| - **Framework**: PyTorch + Transformers |
|
|
| ## Intended Use |
|
|
| This base model serves as a foundation for: |
| - Fine-tuning on specific domains |
| - Research in efficient language model architectures |
| - Development of specialized AI applications |
| - Understanding MoE dynamics and scaling |
|
|
| ## About Rish AI Labs |
|
|
| **Rish AI Labs** is pioneering the future of Enterprise AI through research, applied solutions, and LLM-driven innovation. Based in Bangalore, India, we focus on: |
|
|
| - **Applied AI Solutions**: Enterprise-grade AI implementations |
| - **Research**: Cutting-edge AI research and publications |
| - **LLM Development**: Large language model research and deployment |
| - **AI Consulting**: Expert guidance for AI transformation |
|
|
| ### Mission |
| "Pioneering the future of Enterprise AI through research, applied solutions, and LLM-driven innovation." |
|
|
| ### Contact |
| - Website: [rishailabs.com](https://rishailabs.com) |
| - Location: Bangalore, India |
| - Focus: Enterprise AI, LLMs, Generative AI, AI Research |
|
|
| ## Model Architecture Details |
|
|
| - **Layers**: 12 transformer layers |
| - **Heads**: 12 attention heads |
| - **Hidden Size**: 768 |
| - **Experts**: 16 (MoE) |
| - **Top-K Routing**: 2 |
| - **Vocabulary**: 50,304 tokens |
| - **Sequence Length**: Configurable (trained on various lengths) |
|
|
| ## Usage |
|
|
| ```python |
| from transformers import AutoTokenizer, AutoModelForCausalLM |
| |
| tokenizer = AutoTokenizer.from_pretrained("RishAILabs/RLLM-Base") |
| model = AutoModelForCausalLM.from_pretrained("RishAILabs/RLLM-Base") |
| |
| inputs = tokenizer("Hello, how are you?", return_tensors="pt") |
| outputs = model.generate(**inputs, max_length=50) |
| response = tokenizer.decode(outputs[0], skip_special_tokens=True) |
| ``` |
|
|
| ## Training Details |
|
|
| - **Dataset**: Identity-focused dataset for stable pretraining |
| - **Precision**: FP32 for training stability |
| - **Optimization**: AdamW optimizer |
| - **Framework**: Custom Rish-Core training framework |
| - **Hardware**: Optimized for both CPU and GPU inference |
|
|
| ## Limitations |
|
|
| - Base model - may require fine-tuning for specific tasks |
| - English language focus |
| - Generated content should be reviewed for appropriateness |
|
|
| ## Citation |
|
|
| If you use this model in your research, please cite: |
|
|
|
|
| --- |
|
|
|
|
| ```bibtex |
| @misc{rishailabs_2026, |
| author = { RishAILabs }, |
| title = { RLLM-Base (Revision 552ee30) }, |
| year = 2026, |
| url = { https://huggingface.co/RishAILabs/RLLM-Base }, |
| doi = { 10.57967/hf/7560 }, |
| publisher = { Hugging Face } |
| } |
| ``` |
|
|
|
|
| *Developed by Rish AI Labs - Applied Artificial Intelligence & Research* |
|
|
