| --- |
| license: apache-2.0 |
| language: |
| - en |
| - ar |
| tags: |
| - sovereign |
| - sovythos |
| - llm |
| - transformer |
| - pytorch |
| - arabic |
| - english |
| - code |
| - open-source |
| pipeline_tag: text-generation |
| --- |
| |
| <p align="center"> |
| <img src="https://huggingface.co/buckets/my0919175/Sovythos-66M-Base/resolve/sovythos.png?download=true" width="320" alt="SOVYTHOS"> |
| </p> |
|
|
| <h1 align="center">🦋 SOVYTHOS-66M-Base</h1> |
|
|
| <p align="center"> |
| <b>The first open-source pretrained language model developed completely from scratch under the Sovereign AI Project.</b> |
| </p> |
|
|
| <p align="center"> |
|
|
|  |
|  |
|  |
|  |
|  |
|
|
| </p> |
|
|
| --- |
|
|
| # 🦋 Overview |
|
|
| **SOVYTHOS-66M-Base** is a decoder-only Transformer language model trained **entirely from scratch** as part of the **Sovereign AI Project**. |
|
|
| Unlike models that continue training from existing checkpoints, SOVYTHOS begins from randomly initialized weights and learns language directly from its training corpus. |
|
|
| This repository contains the **Base (Pretrained)** release intended for: |
|
|
| - Continued Pretraining |
| - Fine-tuning |
| - Instruction Tuning |
| - Research |
| - Educational Purposes |
| - Building downstream AI applications |
|
|
| > **This model is NOT instruction tuned and is not intended to behave like a production chatbot.** |
|
|
| --- |
|
|
| # 🚀 Experience the Sovereign Platform |
|
|
| SOVYTHOS is the foundation of the **Sovereign AI Platform**. |
|
|
| If you'd like to experience the latest AI capabilities through a complete interface, you can try the live platform below. |
|
|
| ## 🌐 Live Demo |
|
|
| **https://awaken-creation-guidance.ngrok-free.dev/** |
|
|
| ### Available Features |
|
|
| - 💬 Modern AI Chat Interface |
| - ⚡ Flash AI Models |
| - 🧠 Advanced Reasoning Models |
| - 💻 Coding Assistant |
| - 🌍 Arabic & English Support |
| - 🔥 Continuous Platform Updates |
|
|
| > **Note:** The Sovereign platform may use newer models, APIs, and experimental capabilities beyond this Base release. |
|
|
| --- |
|
|
| # 📊 Model Specifications |
|
|
| | Property | Value | |
| |----------|--------| |
| | Model | SOVYTHOS-66M-Base | |
| | Parameters | ~66.7 Million | |
| | Architecture | Decoder-only Transformer | |
| | Hidden Size | 512 | |
| | Layers | 12 | |
| | Attention Heads | 16 | |
| | Framework | PyTorch | |
| | Tokenizer | Custom | |
| | Context Length | Configured during training | |
| | Training | From Scratch | |
| | Release | Base | |
|
|
| --- |
|
|
| # 🌍 Languages |
|
|
| SOVYTHOS has been trained on multilingual data including: |
|
|
| - 🇪🇬 Arabic |
| - 🇪🇬 Egyptian Arabic |
| - 🇺🇸 English |
| - Programming Languages |
| - Technical Documentation |
| - Mathematical Content |
| - General Web Text |
|
|
| --- |
|
|
| # 🎯 Intended Uses |
|
|
| This release is designed for: |
|
|
| - Research |
| - Fine-tuning |
| - Continued Pretraining |
| - Instruction Tuning |
| - Educational Experiments |
| - Language Model Development |
| - Custom AI Applications |
|
|
| --- |
|
|
| # ⚠️ Current Limitations |
|
|
| This repository contains the **Base Language Model** only. |
|
|
| It has **NOT** been instruction tuned. |
|
|
| Current limitations include: |
|
|
| - Hallucination of factual information |
| - Incorrect reasoning in some cases |
| - Repetition |
| - Incomplete generations |
| - Limited instruction following |
|
|
| These limitations are expected for a pretrained base model and will be addressed in future releases. |
|
|
| --- |
|
|
| # 💻 Example |
|
|
| ```python |
| prompt = "The capital of Egypt is" |
| |
| output = model.generate(prompt) |
| ``` |
|
|
| --- |
|
|
| # 🏗️ Training |
|
|
| SOVYTHOS was trained completely from scratch using: |
|
|
| - Custom Tokenizer |
| - PyTorch |
| - CUDA |
| - Mixed Precision Training |
| - Automatic Best Checkpoint Saving |
|
|
| **No pretrained language model weights were used.** |
|
|
| --- |
|
|
| # 🛣️ Roadmap |
|
|
| ✅ SOVYTHOS-66M-Base |
|
|
| 🟡 SOVYTHOS-66M-Instruct *(In Development)* |
|
|
| ⬜ SOVYTHOS Chat |
|
|
| ⬜ Larger Parameter Models |
|
|
| ⬜ Advanced Reasoning |
|
|
| ⬜ Improved Coding |
|
|
| ⬜ Better Arabic Understanding |
|
|
| ⬜ Long Context Support |
|
|
| --- |
|
|
| # 🌌 Sovereign Project |
|
|
| SOVYTHOS is the first public language model released under the **Sovereign AI Project**. |
|
|
| The long-term vision of Sovereign is to build an open ecosystem of multilingual language models with strong Arabic support, modern reasoning capabilities, and accessible open-source AI technologies. |
|
|
| --- |
|
|
| # 📖 Citation |
|
|
| ```bibtex |
| @misc{sovythos66m, |
| title={SOVYTHOS-66M-Base}, |
| author={Mahmoud Yasser}, |
| year={2026}, |
| publisher={Hugging Face}, |
| } |
| ``` |
|
|
| --- |
|
|
| # 👨💻 Creator |
|
|
| **Mahmoud Yasser** |
|
|
| Founder of the **Sovereign AI Project** |
|
|
| --- |
|
|
| # 📄 License |
|
|
| **Apache License 2.0** |
|
|
| --- |
|
|
| <p align="center"> |
|
|
| ### 🦋 Welcome to the beginning of the Sovereign ecosystem. |
|
|
| The current release represents the first public pretrained foundation model. |
|
|
| Future releases will introduce instruction tuning, stronger reasoning, improved coding capabilities, and more advanced multilingual intelligence. |
|
|
| ⭐ If you find this project interesting, consider following the repository and trying the Sovereign AI Platform. |
|
|
| </p> |
