Upload 7 files

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+sovythos.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,236 @@
 ---
 license: apache-2.0
+language:
+- en
+- ar
+tags:
+- sovereign
+- sovythos
+- llm
+- transformer
+- pytorch
+- arabic
+- english
+- code
+- open-source
+pipeline_tag: text-generation
 ---
+
+<p align="center">
+  <img src="https://huggingface.co/buckets/my0919175/Sovythos-66M-Base/resolve/sovythos.png?download=true" width="320" alt="SOVYTHOS">
+</p>
+
+<h1 align="center">🦋 SOVYTHOS-66M-Base</h1>
+
+<p align="center">
+<b>The first open-source pretrained language model developed completely from scratch under the Sovereign AI Project.</b>
+</p>
+
+<p align="center">
+
+![Open Source](https://img.shields.io/badge/Open%20Source-Yes-success)
+![Parameters](https://img.shields.io/badge/Parameters-66.7M-blue)
+![Languages](https://img.shields.io/badge/Languages-Arabic%20%7C%20English-orange)
+![Framework](https://img.shields.io/badge/PyTorch-2.x-red)
+![Status](https://img.shields.io/badge/Status-Base%20Release-brightgreen)
+
+</p>
+
+---
+
+# 🦋 Overview
+
+**SOVYTHOS-66M-Base** is a decoder-only Transformer language model trained **entirely from scratch** as part of the **Sovereign AI Project**.
+
+Unlike models that continue training from existing checkpoints, SOVYTHOS begins from randomly initialized weights and learns language directly from its training corpus.
+
+This repository contains the **Base (Pretrained)** release intended for:
+
+- Continued Pretraining
+- Fine-tuning
+- Instruction Tuning
+- Research
+- Educational Purposes
+- Building downstream AI applications
+
+> **This model is NOT instruction tuned and is not intended to behave like a production chatbot.**
+
+---
+
+# 🚀 Experience the Sovereign Platform
+
+SOVYTHOS is the foundation of the **Sovereign AI Platform**.
+
+If you'd like to experience the latest AI capabilities through a complete interface, you can try the live platform below.
+
+## 🌐 Live Demo
+
+**https://awaken-creation-guidance.ngrok-free.dev/**
+
+### Available Features
+
+- 💬 Modern AI Chat Interface
+- ⚡ Flash AI Models
+- 🧠 Advanced Reasoning Models
+- 💻 Coding Assistant
+- 🌍 Arabic & English Support
+- 🔥 Continuous Platform Updates
+
+> **Note:** The Sovereign platform may use newer models, APIs, and experimental capabilities beyond this Base release.
+
+---
+
+# 📊 Model Specifications
+
+| Property | Value |
+|----------|--------|
+| Model | SOVYTHOS-66M-Base |
+| Parameters | ~66.7 Million |
+| Architecture | Decoder-only Transformer |
+| Hidden Size | 512 |
+| Layers | 12 |
+| Attention Heads | 16 |
+| Framework | PyTorch |
+| Tokenizer | Custom |
+| Context Length | Configured during training |
+| Training | From Scratch |
+| Release | Base |
+
+---
+
+# 🌍 Languages
+
+SOVYTHOS has been trained on multilingual data including:
+
+- 🇪🇬 Arabic
+- 🇪🇬 Egyptian Arabic
+- 🇺🇸 English
+- Programming Languages
+- Technical Documentation
+- Mathematical Content
+- General Web Text
+
+---
+
+# 🎯 Intended Uses
+
+This release is designed for:
+
+- Research
+- Fine-tuning
+- Continued Pretraining
+- Instruction Tuning
+- Educational Experiments
+- Language Model Development
+- Custom AI Applications
+
+---
+
+# ⚠️ Current Limitations
+
+This repository contains the **Base Language Model** only.
+
+It has **NOT** been instruction tuned.
+
+Current limitations include:
+
+- Hallucination of factual information
+- Incorrect reasoning in some cases
+- Repetition
+- Incomplete generations
+- Limited instruction following
+
+These limitations are expected for a pretrained base model and will be addressed in future releases.
+
+---
+
+# 💻 Example
+
+```python
+prompt = "The capital of Egypt is"
+
+output = model.generate(prompt)
+```
+
+---
+
+# 🏗️ Training
+
+SOVYTHOS was trained completely from scratch using:
+
+- Custom Tokenizer
+- PyTorch
+- CUDA
+- Mixed Precision Training
+- Automatic Best Checkpoint Saving
+
+**No pretrained language model weights were used.**
+
+---
+
+# 🛣️ Roadmap
+
+✅ SOVYTHOS-66M-Base
+
+🟡 SOVYTHOS-66M-Instruct *(In Development)*
+
+⬜ SOVYTHOS Chat
+
+⬜ Larger Parameter Models
+
+⬜ Advanced Reasoning
+
+⬜ Improved Coding
+
+⬜ Better Arabic Understanding
+
+⬜ Long Context Support
+
+---
+
+# 🌌 Sovereign Project
+
+SOVYTHOS is the first public language model released under the **Sovereign AI Project**.
+
+The long-term vision of Sovereign is to build an open ecosystem of multilingual language models with strong Arabic support, modern reasoning capabilities, and accessible open-source AI technologies.
+
+---
+
+# 📖 Citation
+
+```bibtex
+@misc{sovythos66m,
+  title={SOVYTHOS-66M-Base},
+  author={Mahmoud Yasser},
+  year={2026},
+  publisher={Hugging Face},
+}
+```
+
+---
+
+# 👨‍💻 Creator
+
+**Mahmoud Yasser**
+
+Founder of the **Sovereign AI Project**
+
+---
+
+# 📄 License
+
+**Apache License 2.0**
+
+---
+
+<p align="center">
+
+### 🦋 Welcome to the beginning of the Sovereign ecosystem.
+
+The current release represents the first public pretrained foundation model.
+
+Future releases will introduce instruction tuning, stronger reasoning, improved coding capabilities, and more advanced multilingual intelligence.
+
+⭐ If you find this project interesting, consider following the repository and trying the Sovereign AI Platform.
+
+</p>

config.json

@@ -0,0 +1,15 @@
+{
+  "architectures": [
+    "SovythosModel"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_sovythos.SovythosConfig",
+    "AutoModelForCausalLM": "modeling_sovythos.SovythosModel"
+  },
+  "vocab_size": 32000,
+  "dim": 512,
+  "heads": 16,
+  "layers": 12,
+  "block_size": 512,
+  "model_type": "sovythos"
+}

configuration_sovythos.py

@@ -0,0 +1,20 @@
+from transformers import PretrainedConfig
+
+class SovythosConfig(PretrainedConfig):
+    model_type = "sovythos"
+
+    def __init__(
+        self,
+        vocab_size=32000,
+        dim=512,
+        heads=16,
+        layers=12,
+        block_size=512,
+        **kwargs
+    ):
+        super().__init__(**kwargs)
+        self.vocab_size = vocab_size
+        self.dim = dim
+        self.heads = heads
+        self.layers = layers
+        self.block_size = block_size

modeling_sovythos.py

@@ -0,0 +1,131 @@
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from transformers import PreTrainedModel, GenerationMixin
+from transformers.modeling_outputs import CausalLMOutput
+from .configuration_sovythos import SovythosConfig
+
+class RMSNorm(nn.Module):
+    def __init__(self, dim, eps=1e-6):
+        super().__init__()
+        self.w = nn.Parameter(torch.ones(dim))
+        self.eps = eps
+
+    def forward(self, x):
+        rms = x.pow(2).mean(-1, keepdim=True)
+        return self.w * x * torch.rsqrt(rms + self.eps)
+
+class RoPE(nn.Module):
+    def __init__(self, head_dim):
+        super().__init__()
+        inv_freq = 1.0 / (10000 ** (torch.arange(0, head_dim, 2).float() / head_dim))
+        self.register_buffer("inv_freq", inv_freq, persistent=False)
+        self._cos_cache = None
+        self._sin_cache = None
+
+    def _build_cache(self, seq_len, device):
+        if self._cos_cache is not None and self._cos_cache.shape[0] >= seq_len:
+            return
+        t = torch.arange(seq_len, device=device).type_as(self.inv_freq)
+        freqs = torch.einsum("i,j->ij", t, self.inv_freq)
+        emb = torch.cat((freqs, freqs), dim=-1)
+        self._cos_cache = emb.cos()
+        self._sin_cache = emb.sin()
+
+    def forward(self, x, seq_len):
+        self._build_cache(seq_len, x.device)
+        cos = self._cos_cache[:seq_len][None, None, :, :]
+        sin = self._sin_cache[:seq_len][None, None, :, :]
+        x1, x2 = x[..., :x.shape[-1]//2], x[..., x.shape[-1]//2:]
+        return (x * cos) + (torch.cat((-x2, x1), dim=-1) * sin)
+
+class TitanAttention(nn.Module):
+    def __init__(self, dim, heads):
+        super().__init__()
+        self.heads = heads
+        self.head_dim = dim // heads
+        self.q_proj = nn.Linear(dim, dim, bias=False)
+        self.k_proj = nn.Linear(dim, dim, bias=False)
+        self.v_proj = nn.Linear(dim, dim, bias=False)
+        self.o_proj = nn.Linear(dim, dim, bias=False)
+        self.q_norm = RMSNorm(self.head_dim)
+        self.k_norm = RMSNorm(self.head_dim)
+        self.rope = RoPE(self.head_dim)
+
+    def forward(self, x, is_causal=True):
+        B, T, C = x.shape
+        q = self.q_proj(x).view(B, T, self.heads, self.head_dim).transpose(1, 2)
+        k = self.k_proj(x).view(B, T, self.heads, self.head_dim).transpose(1, 2)
+        v = self.v_proj(x).view(B, T, self.heads, self.head_dim).transpose(1, 2)
+        
+        q = self.rope(self.q_norm(q), T)
+        k = self.rope(self.k_norm(k), T)
+        
+        out = F.scaled_dot_product_attention(q, k, v, attn_mask=None, is_causal=is_causal)
+        out = out.transpose(1, 2).contiguous().view(B, T, C)
+        return self.o_proj(out)
+
+class Block(nn.Module):
+    def __init__(self, dim, heads):
+        super().__init__()
+        self.n1 = RMSNorm(dim)
+        self.attn = TitanAttention(dim, heads)
+        self.n2 = RMSNorm(dim)
+        self.w1 = nn.Linear(dim, 4 * dim, bias=False)
+        self.w2 = nn.Linear(dim, 4 * dim, bias=False)
+        self.w3 = nn.Linear(4 * dim, dim, bias=False)
+
+    def forward(self, x):
+        x = x + self.attn(self.n1(x))
+        h = self.n2(x)
+        x = x + self.w3(F.silu(self.w1(h)) * self.w2(h))
+        return x
+
+class SovythosModel(PreTrainedModel, GenerationMixin):
+    config_class = SovythosConfig
+
+    def __init__(self, config):
+        # حل مشكلة num_hidden_layers بربطها ديناميكياً بالمتغير الحالي
+        if not hasattr(config, "num_hidden_layers"):
+            config.num_hidden_layers = getattr(config, "layers", 12)
+            
+        super().__init__(config)
+        self.emb = nn.Embedding(config.vocab_size, config.dim)
+        self.blocks = nn.ModuleList([Block(config.dim, config.heads) for _ in range(config.layers)])
+        self.norm = RMSNorm(config.dim)
+        self.fc = nn.Linear(config.dim, config.vocab_size, bias=False)
+        
+        self.fc.weight = self.emb.weight
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.emb
+
+    def set_input_embeddings(self, value):
+        self.emb = value
+
+    def get_output_embeddings(self):
+        return self.fc
+
+    def set_output_embeddings(self, new_embeddings):
+        self.fc = new_embeddings
+
+    def forward(self, input_ids, attention_mask=None, labels=None, **kwargs):
+        x = self.emb(input_ids)
+        for blk in self.blocks:
+            x = blk(x)
+        x = self.norm(x)
+        logits = self.fc(x)
+
+        loss = None
+        if labels is not None:
+            loss_fn = nn.CrossEntropyLoss()
+            loss = loss_fn(logits.view(-1, self.config.vocab_size), labels.view(-1))
+
+        return CausalLMOutput(loss=loss, logits=logits)
+
+    def prepare_inputs_for_generation(self, input_ids, attention_mask=None, **kwargs):
+        return {
+            "input_ids": input_ids,
+            "attention_mask": attention_mask
+        }

tokenizer_config.json

@@ -0,0 +1,59 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<|pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "<|user|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<|assistant|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<|system|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<|code|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|pad|>",
+  "model_max_length": 512,
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "chat_template": "{% for message in messages %}{{'<|' + message['role'] + '|>\n' + message['content'] + '<|endoftext|>\n'}}{% endfor %}{% if add_generation_prompt %}{{'<|assistant|>\n'}}{% endif %}"
+}