🌍 VishwamAI

VishwamAI is an open research organization building language models for English, Hindi and Telugu, with a focus on reasoning, mathematics, code and honest evaluation.

🚀 Vision

To build capable, transparent language models that learn efficiently, explain their reasoning, and know when they might be wrong, and to publish how they were made and how they score, including the weak results.

🔍 Key Areas of Focus

• Efficient architectures: a from-scratch Mixture-of-Experts decoder (grouped-query attention, QK-norm, YaRN long-context RoPE, multi-token prediction) trained in JAX/Flax and exported to standard Hugging Face safetensors.
• Distillation and post-training: learning from open-weight teacher models (such as DeepSeek, Qwen and Gemma) through their generated text, with rejection sampling, majority-vote filtering and decontamination against benchmark test sets.
• Models built on open foundations: VishwamAI variants derived from open base models such as Gemma 4 (Apache-2.0) using LoRA adapters. These are clearly labelled as derived and credit their base model.
• Indian languages: Hindi and Telugu data, tokenizer and evaluation alongside English.
• Reasoning and self-correction: training models to show their steps, check answers and recognise their own mistakes, measured rather than assumed.
• Honest evaluation: public numbers with confidence intervals, held-out test sets that are never trained on, and a clear separation between what we trained from scratch and what we derived.

📚 Models

Small models score near chance on broad benchmarks such as MMLU; we report that rather than hide it.

🗺️ Roadmap

1. Base models: continue pretraining the from-scratch model on a daily, subject-by-subject schedule.
2. Derived models: train and evaluate adapters on open foundation models, one topic at a time (knowledge, math, code, tool use).
3. Distillation: build verified teacher-generated datasets from open models only.
4. Evaluation: publish reproducible notebooks and results for every released model.

🧪 Contributing

We welcome contributions in training, data, evaluation and tooling. Please check CONTRIBUTING.md for details.

🤝 Join the Community

• Discord: join here
• Twitter: follow us for updates.

📄 License

This project is licensed under the MIT License; see the LICENSE file. Models derived from other open models keep the license and attribution requirements of their base model.