README.md

---
license: apache-2.0
language:
- en
- ar
tags:
- sovereign
- sovythos
- llm
- transformer
- pytorch
- arabic
- english
- code
- open-source
pipeline_tag: text-generation
---

<p align="center">
  <img src="https://huggingface.co/buckets/my0919175/Sovythos-66M-Base/resolve/sovythos.png?download=true" width="320" alt="SOVYTHOS">
</p>

<h1 align="center">🦋 SOVYTHOS-66M-Base</h1>

<p align="center">
<b>The first open-source pretrained language model developed completely from scratch under the Sovereign AI Project.</b>
</p>

<p align="center">

![Open Source](https://img.shields.io/badge/Open%20Source-Yes-success)
![Parameters](https://img.shields.io/badge/Parameters-66.7M-blue)
![Languages](https://img.shields.io/badge/Languages-Arabic%20%7C%20English-orange)
![Framework](https://img.shields.io/badge/PyTorch-2.x-red)
![Status](https://img.shields.io/badge/Status-Base%20Release-brightgreen)

</p>

---

# 🦋 Overview

**SOVYTHOS-66M-Base** is a decoder-only Transformer language model trained **entirely from scratch** as part of the **Sovereign AI Project**.

Unlike models that continue training from existing checkpoints, SOVYTHOS begins from randomly initialized weights and learns language directly from its training corpus.

This repository contains the **Base (Pretrained)** release intended for:

- Continued Pretraining
- Fine-tuning
- Instruction Tuning
- Research
- Educational Purposes
- Building downstream AI applications

> **This model is NOT instruction tuned and is not intended to behave like a production chatbot.**

---

# 🚀 Experience the Sovereign Platform

SOVYTHOS is the foundation of the **Sovereign AI Platform**.

If you'd like to experience the latest AI capabilities through a complete interface, you can try the live platform below.

## 🌐 Live Demo

**https://awaken-creation-guidance.ngrok-free.dev/**

### Available Features

- 💬 Modern AI Chat Interface
- ⚡ Flash AI Models
- 🧠 Advanced Reasoning Models
- 💻 Coding Assistant
- 🌍 Arabic & English Support
- 🔥 Continuous Platform Updates

> **Note:** The Sovereign platform may use newer models, APIs, and experimental capabilities beyond this Base release.

---

# 📊 Model Specifications

| Property | Value |
|----------|--------|
| Model | SOVYTHOS-66M-Base |
| Parameters | ~66.7 Million |
| Architecture | Decoder-only Transformer |
| Hidden Size | 512 |
| Layers | 12 |
| Attention Heads | 16 |
| Framework | PyTorch |
| Tokenizer | Custom |
| Context Length | Configured during training |
| Training | From Scratch |
| Release | Base |

---

# 🌍 Languages

SOVYTHOS has been trained on multilingual data including:

- 🇪🇬 Arabic
- 🇪🇬 Egyptian Arabic
- 🇺🇸 English
- Programming Languages
- Technical Documentation
- Mathematical Content
- General Web Text

---

# 🎯 Intended Uses

This release is designed for:

- Research
- Fine-tuning
- Continued Pretraining
- Instruction Tuning
- Educational Experiments
- Language Model Development
- Custom AI Applications

---

# ⚠️ Current Limitations

This repository contains the **Base Language Model** only.

It has **NOT** been instruction tuned.

Current limitations include:

- Hallucination of factual information
- Incorrect reasoning in some cases
- Repetition
- Incomplete generations
- Limited instruction following

These limitations are expected for a pretrained base model and will be addressed in future releases.

---

# 💻 Example

```python
prompt = "The capital of Egypt is"

output = model.generate(prompt)
```

---

# 🏗️ Training

SOVYTHOS was trained completely from scratch using:

- Custom Tokenizer
- PyTorch
- CUDA
- Mixed Precision Training
- Automatic Best Checkpoint Saving

**No pretrained language model weights were used.**

---

# 🛣️ Roadmap

✅ SOVYTHOS-66M-Base

🟡 SOVYTHOS-66M-Instruct *(In Development)*

⬜ SOVYTHOS Chat

⬜ Larger Parameter Models

⬜ Advanced Reasoning

⬜ Improved Coding

⬜ Better Arabic Understanding

⬜ Long Context Support

---

# 🌌 Sovereign Project

SOVYTHOS is the first public language model released under the **Sovereign AI Project**.

The long-term vision of Sovereign is to build an open ecosystem of multilingual language models with strong Arabic support, modern reasoning capabilities, and accessible open-source AI technologies.

---

# 📖 Citation

```bibtex
@misc{sovythos66m,
  title={SOVYTHOS-66M-Base},
  author={Mahmoud Yasser},
  year={2026},
  publisher={Hugging Face},
}
```

---

# 👨‍💻 Creator

**Mahmoud Yasser**

Founder of the **Sovereign AI Project**

---

# 📄 License

**Apache License 2.0**

---

<p align="center">

### 🦋 Welcome to the beginning of the Sovereign ecosystem.

The current release represents the first public pretrained foundation model.

Future releases will introduce instruction tuning, stronger reasoning, improved coding capabilities, and more advanced multilingual intelligence.

⭐ If you find this project interesting, consider following the repository and trying the Sovereign AI Platform.

</p>