Upload 9 files

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+sovythos.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,474 @@
 ---
 license: apache-2.0
+language:
+- en
+- ar
+tags:
+- sovereign
+- sovythos
+- llm
+- transformer
+- pytorch
+- arabic
+- egyptian-arabic
+- english
+- code
+- open-source
+- llama-architecture
+- gqa
+- rope
+- swiglu
+- safetensors
+- gguf
+pipeline_tag: text-generation
 ---
+
+<p align="center">
+  <img src="https://huggingface.co/my0919175/Sovythos-303M-Base/resolve/main/sovythos.png" width="320" alt="SOVYTHOS-V2">
+</p>
+
+<h1 align="center">🦋 SOVYTHOS-V2</h1>
+<h3 align="center">Sovythos Egyptian Reasoning & Code Base Model</h3>
+
+<p align="center">
+<b>A production-grade, Llama-compatible decoder-only Transformer trained entirely from scratch for Arabic, English, and Code.</b>
+</p>
+
+<p align="center">
+
+![Open Source](https://img.shields.io/badge/Open%20Source-Yes-success)
+![Parameters](https://img.shields.io/badge/Parameters-~300M-blue)
+![Architecture](https://img.shields.io/badge/Architecture-Llama--compatible-purple)
+![Languages](https://img.shields.io/badge/Languages-Arabic%20%7C%20English%20%7C%20Code-orange)
+![Framework](https://img.shields.io/badge/PyTorch-2.x-red)
+![Precision](https://img.shields.io/badge/Training-AMP%20FP16-yellow)
+![Format](https://img.shields.io/badge/Weights-Safetensors%20%2B%20GGUF-brightgreen)
+![Status](https://img.shields.io/badge/Status-Early%20Training%20Checkpoint-yellow)
+
+</p>
+
+> ### ⚠️ Training Status: Early-Stage Checkpoint
+> **This checkpoint is an early snapshot from an ongoing training run — it is NOT the final SOVYTHOS-V2 release.** The model is uploaded at this stage for architecture testing, integration testing (`transformers`/GGUF compatibility), and community feedback purposes only.
+>
+> Expect at this stage:
+> - Weak or incoherent generations on many prompts
+> - Little to no instruction-following ability (this is a **Base**, non-instruction-tuned model on top of being early-stage)
+> - Validation loss still actively decreasing — quality will improve substantially with further training
+>
+> Training is actively continuing. Follow this repository for updates as later checkpoints are released.
+
+---
+
+## 📋 Table of Contents
+
+- [Overview](#overview)
+- [Model Specifications](#model-specifications)
+- [Architecture Details](#architecture-details)
+- [Tokenizer](#tokenizer)
+- [Training](#training)
+- [Try It Now (Google Colab)](#-try-it-now-google-colab)
+- [Usage](#usage)
+- [GGUF Inference](#gguf-inference)
+- [Intended Uses](#intended-uses)
+- [Limitations](#limitations)
+- [Roadmap](#roadmap)
+- [Citation](#citation)
+- [License](#license)
+
+---
+
+## 🦋 Overview
+
+**SOVYTHOS-V2** is the second-generation foundation model of the **Sovythos AI Project** — a fully open-source, decoder-only Transformer trained **from randomly initialized weights** on a multilingual corpus spanning Modern Standard Arabic, Egyptian Arabic, English, and programming languages.
+
+This release represents a significant architectural leap from the original 66M Base model, scaling to **~300 million parameters** while adopting a **Llama-compatible** design (RMSNorm, RoPE, GQA, SwiGLU) to ensure seamless deployment across the modern inference ecosystem, including `llama.cpp`, `llama-cpp-python`, and Hugging Face transformers.
+
+SOVYTHOS-V2 is released as a **Base (Pretrained)** model. It has not undergone instruction tuning or RLHF and is intended for researchers, developers, and educators who require a sovythos, transparent foundation for downstream training.
+
+> **Key Design Principle:** Every component — from the tokenizer to the tensor naming convention — is engineered for **direct compatibility** with the Llama/GGUF toolchain. No conversion hacks required.
+
+---
+
+## 📊 Model Specifications
+
+| Property | Value |
+|----------|-------|
+| **Model** | SOVYTHOS-V2 |
+| **Parameters** | ~300 Million |
+| **Architecture** | Decoder-only Transformer (Llama-style) |
+| **Hidden Size** | 1,024 |
+| **Layers** | 24 |
+| **Attention Heads** | 16 (Query) / 4 (Key-Value) |
+| **Head Dimension** | 64 |
+| **Context Length** | 2,048 tokens |
+| **RoPE Theta** | 1,000,000 |
+| **Vocabulary Size** | 32,000 |
+| **Framework** | PyTorch 2.x |
+| **Weight Format** | Safetensors + GGUF (F16 / F32) |
+| **Training** | From Scratch |
+| **Release** | Base |
+
+---
+
+## 🏗️ Architecture Details
+
+SOVYTHOS-V2 implements a modern, efficient decoder stack aligned with the Llama architecture specification for frictionless ecosystem integration.
+
+### Normalization
+- **RMSNorm** applied pre-attention and post-attention (no bias, no learned affine shift)
+
+### Position Encoding
+- **RoPE (Rotary Position Embedding)** with rotate_half (NEOX-style) implementation
+- Base frequency θ = 1,000,000
+- Precomputed cosine/sine caches up to max sequence length
+
+### Attention Mechanism
+- **Grouped Query Attention (GQA)** with 16 query heads and 4 key-value heads
+- Reduces KV-cache memory footprint by **4×** during inference compared to standard MHA
+- Compatible with memory-efficient attention kernels (`scaled_dot_product_attention`)
+
+### Feed-Forward Network
+- **SwiGLU** activation with gated projection
+- Hidden dimension rounded to nearest multiple of 256 for optimal GPU kernel utilization
+- `gate_proj`, `up_proj`, `down_proj` naming matches Llama/GGUF conventions exactly
+
+### Weight Initialization
+- Gaussian initialization (σ = 0.02) for embeddings and linear layers
+- Scaled down-projection and output-projection initialization by `1/√(2×layers)` to stabilize deep training
+
+### Training Optimizations
+- **torch.compile** graph mode (when available)
+- **TF32** matmul acceleration on Ampere+
+- **Automatic Mixed Precision (AMP)** with `fp16` forward pass
+- **Gradient Checkpointing** for long-context training on consumer GPUs
+- **Gradient Accumulation** with effective batch scaling
+- **Cosine LR Decay** with linear warmup
+- **Gradient Clipping** (max norm = 1.0)
+
+---
+
+## 🔤 Tokenizer
+
+SOVYTHOS-V2 ships with a **custom-trained Byte-Pair Encoding (BPE)** tokenizer optimized for Arabic morphology, English, and code syntax.
+
+### Design Choices
+
+| Feature | Implementation |
+|---------|---------------|
+| **Pre-tokenization** | GPT-4 / cl100k-style regex split (Unicode-aware) |
+| **Numerical Splitting** | Digits grouped by 3 (e.g., `1,234,567` → `1` `234` `567`) |
+| **Fallback** | ByteLevel encoding for lossless coverage of any Unicode symbol |
+| **Normalization** | NFC Unicode normalization (unifies Arabic hamza & diacritic forms) |
+| **Special Tokens** | `<\|endoftext\|>`, `<\|pad\|>`, `<\|user\|>`, `<\|assistant\|>`, `<\|system\|>`, `<\|code\|>`, `<\|/code\|>`, `<\|think\|>`, `<\|/think\|>` |
+| **Vocab Size** | 32,000 |
+| **Min Frequency** | 2 |
+
+### Coverage Efficiency (Tokens per Character)
+
+Lower is better. Measured on a held-out sample of the training corpus:
+
+| Domain | Tokens / Char | Notes |
+|--------|---------------|-------|
+| **Arabic (MSA + Egyptian)** | ~0.35 | Efficient subword decomposition for Semitic morphology |
+| **English** | ~0.28 | Competitive with modern English-centric tokenizers |
+| **Code** | ~0.42 | Strong performance on Python / JavaScript identifiers |
+
+> The tokenizer is saved as a standard `tokenizer.json` (Hugging Face `tokenizers` format) and its merges/vocab are **directly compatible** with GGUF `gpt2` tokenizer model type.
+
+---
+
+## 🏋️ Training
+
+SOVYTHOS-V2 was trained **entirely from scratch**. No pretrained weights from Llama, GPT, or any other foundation model were used as initialization.
+
+### Data Pipeline
+- **Corpus**: Multilingual web text, technical documentation, mathematical content, and code repositories
+- **Languages**: Modern Standard Arabic, Egyptian Arabic, English, and multilingual programming languages
+- **Cleaning**: Duplicate-line removal, short-line filtering (< 2 chars), NFC normalization, and aggressive deduplication of repeated contiguous lines
+- **Format**: Binary tokenized memmap for high-throughout training with `np.uint16` encoding
+
+### Training Recipe
+
+| Hyperparameter | Value |
+|----------------|-------|
+| Optimizer | AdamW (β₁=0.9, β₂=0.95, ε=1e-8) |
+| Peak Learning Rate | 3e-4 |
+| Minimum Learning Rate | 3e-5 |
+| Warmup Steps | 1,000 |
+| LR Schedule | Cosine decay |
+| Weight Decay | 0.1 (2D+ tensors only) |
+| Max Gradient Norm | 1.0 |
+| Precision | FP16 (AMP) with FP32 master weights |
+| Batch Size | 4 (per device) |
+| Gradient Accumulation | 8 steps |
+| Effective Batch Size | 32 |
+| Block Size | 512 tokens |
+
+### Checkpointing & Resumption
+- Checkpoints saved in **Safetensors** format (secure, fast, language-agnostic)
+- Automatic resumption from `ckpt_dir/best/` including step count and best validation loss
+- Best checkpoint selection based on validation loss
+
+### Export
+- **Native GGUF export** after training (or from existing checkpoints via `--export_gguf_only`)
+- Supported quantizations: `F32`, `F16` (post-hoc Q4/Q8 via `llama.cpp` tools)
+- Tensor name map strictly follows Llama GGUF specification (`blk.{i}.attn_q.weight`, `blk.{i}.ffn_gate.weight`, etc.)
+
+---
+
+## 🚀 Try It Now (Google Colab)
+
+The fastest way to test SOVYTHOS-V2 is directly through 🤗 `transformers`, using the custom model code shipped in this repository (`trust_remote_code=True`). No manual setup, no cloning — just run the cell below in a fresh Colab notebook.
+
+```python
+# ============================================================
+# 🔱 SOVYTHOS-V2 (303M-Base) — Colab Quickstart
+# ============================================================
+
+!pip install -q -U transformers accelerate safetensors
+
+import torch
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+MODEL_ID = "my0919175/Sovythos-303M-Base"
+
+print("🔄 Loading tokenizer...")
+tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
+
+print("🔄 Loading model...")
+model = AutoModelForCausalLM.from_pretrained(
+    MODEL_ID,
+    dtype=torch.float32,
+    trust_remote_code=True,
+).to("cuda" if torch.cuda.is_available() else "cpu")
+model.eval()
+
+# Chat-style prompt using the built-in special tokens
+prompt = "<|user|>\nمرحبا, احكيلي عن نفسك\n<|assistant|>\n"
+input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)
+
+with torch.no_grad():
+    output = model.generate(
+        input_ids,
+        max_new_tokens=150,
+        do_sample=True,
+        temperature=0.8,
+        top_k=40,
+        top_p=0.9,
+        repetition_penalty=1.15,
+        pad_token_id=tokenizer.pad_token_id,
+        eos_token_id=tokenizer.eos_token_id,
+    )
+
+print(tokenizer.decode(output[0], skip_special_tokens=True))
+```
+
+> **Note:** This is an **early-stage training checkpoint** of a **Base** (non-instruction-tuned) model — not the final release. Outputs may be short, repetitive, incoherent, or diverge from the prompt entirely; this is expected at this stage of training and is not a loading error. Quality will improve as training progresses and with the upcoming Instruct release.
+>
+> When prompted `Do you wish to run the custom code? [y/N]`, type **y** — this loads `configuration_sovythos.py` and `modeling_sovythos.py` from this repo, which implement the GQA + RoPE + SwiGLU architecture described above.
+
+---
+
+## 💻 Usage
+
+### Loading with 🤗 Transformers (Recommended)
+
+This repository ships with `trust_remote_code`-compatible files (`configuration_sovythos.py`, `modeling_sovythos.py`), so it works directly with the standard `AutoModelForCausalLM` / `AutoTokenizer` API — no need to clone the original training repo:
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
+
+model_id = "my0919175/Sovythos-303M-Base"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id, dtype=torch.float32, trust_remote_code=True
+).to("cuda" if torch.cuda.is_available() else "cpu")
+
+prompt = "<|user|>\nThe capital of Egypt is\n<|assistant|>\n"
+input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)
+
+output = model.generate(
+    input_ids,
+    max_new_tokens=50,
+    temperature=0.8,
+    top_k=40,
+    top_p=0.9,
+    eos_token_id=tokenizer.eos_token_id,
+)
+print(tokenizer.decode(output[0], skip_special_tokens=True))
+```
+
+### Loading with the Native Training Code
+
+If you have the original training repository (`model.py`, `train.py`) available locally, you can also load the checkpoint directly through the native `Model` class, which supports the custom KV-cache and generation loop used during training/research:
+
+```python
+from model import Model
+
+model = Model.from_pretrained("sovythos_ckpt/best", device="cuda")
+
+text = "The capital of Egypt is"
+ids = tokenizer.encode(text).ids
+input_ids = torch.tensor([ids], device="cuda")
+
+output = model.generate(
+    input_ids,
+    max_new_tokens=50,
+    temperature=0.8,
+    top_k=40,
+    top_p=0.9,
+    eos_token_id=tokenizer.token_to_id("<|endoftext|>")
+)
+print(tokenizer.decode(output[0].tolist()))
+```
+
+### Using the Tokenizer Directly
+
+```python
+from tokenizers import Tokenizer
+
+tok = Tokenizer.from_file("tokenizer.json")
+encoded = tok.encode("إزيك يا كبير؟")
+print(encoded.ids)  # [ ... ]
+print(tok.decode(encoded.ids))  # "إزيك يا كبير؟"
+```
+
+---
+
+## 🚀 GGUF Inference
+
+Because SOVYTHOS-V2 uses **Llama-compatible tensor naming** and a **GPT-2-format tokenizer**, the exported GGUF file runs out-of-the-box with `llama.cpp` and `llama-cpp-python`.
+
+### llama-cpp-python
+
+```python
+from llama_cpp import Llama
+
+llm = Llama(
+    model_path="sovythos-v2.f16.gguf",
+    n_ctx=2048,
+    n_gpu_layers=-1,  # Offload all layers to GPU
+    verbose=False
+)
+
+output = llm(
+    "The capital of Egypt is",
+    max_tokens=50,
+    temperature=0.8,
+    top_k=40,
+    top_p=0.9,
+    stop=["<|endoftext|>"]
+)
+print(output["choices"][0]["text"])
+```
+
+### llama.cpp CLI
+
+```bash
+./main -m sovythos-v2.f16.gguf        -n 50        --temp 0.8        --top-k 40        --top-p 0.9        -p "The capital of Egypt is"
+```
+
+> **No conversion scripts required.** The GGUF file is produced directly by the training framework with correct `llama` arch metadata.
+
+---
+
+## 🎯 Intended Uses
+
+SOVYTHOS-V2 is designed as a **research and development foundation** for:
+
+- Continued pretraining on domain-specific corpora
+- Supervised fine-tuning (SFT) for chat or task-specific behavior
+- Instruction tuning and alignment experiments
+- Educational exploration of modern decoder-only architectures
+- Arabic NLP research (MSA, Egyptian dialect, code-switching)
+- Low-resource and Sovythos AI development
+
+> **Not intended for:** Direct deployment as a production chatbot without additional safety fine-tuning and evaluation.
+
+---
+
+## ⚠️ Limitations
+
+As a **Base (pretrained)** model, SOVYTHOS-V2 exhibits expected limitations:
+
+- **No instruction following** without fine-tuning
+- **Hallucination** of factual information (names, dates, technical details)
+- **Repetition** in long-form generation
+- **Incomplete reasoning** on complex multi-step problems
+- **Limited context coherence** beyond a few hundred tokens in zero-shot settings
+- **Code generation** produces syntactically valid structures but may contain logic errors
+
+These limitations are standard for base models of this scale and will be addressed in future Instruct/Chat releases.
+
+---
+
+## 🛣️ Roadmap
+
+| Milestone | Status |
+|-----------|--------|
+| 🟡 SOVYTHOS-V2 Base | **In Training (early checkpoint published)** |
+| ⬜ SOVYTHOS-V2 Base (final) | Planned |
+| ⬜ SOVYTHOS-V2-Instruct | Planned |
+| ⬜ SOVYTHOS Chat | Planned |
+| ⬜ Larger Parameter Models (1B+) | Planned |
+| ⬜ Long Context Extension (8K+) | Planned |
+| ⬜ Advanced Reasoning Tuning | Planned |
+| ⬜ Improved Egyptian Dialect Understanding | Planned |
+
+---
+
+## 🌌 Sovythos AI Project
+
+SOVYTHOS-V2 is the flagship foundation model of the **Sovythos AI Project**, an open ecosystem dedicated to building transparent, multilingual language models with **native Arabic support** and modern reasoning capabilities.
+
+The project prioritizes:
+- **Sovereignty**: Full training pipeline ownership (data → tokenizer → model → export)
+- **Transparency**: Open weights, open code, open training logs
+- **Accessibility**: GGUF compatibility for edge and consumer hardware
+- **Community**: Research-first releases with clear limitation disclosures
+
+---
+
+## 📖 Citation
+
+If you use SOVYTHOS-V2 in your research, please cite:
+
+```bibtex
+@misc{sovythosv2,
+  title={SOVYTHOS-V2: Sovythos Egyptian Reasoning and Code Base Model},
+  author={Mahmoud Yasser},
+  year={2026},
+  publisher={Hugging Face},
+  howpublished={\url{https://huggingface.co/my0919175/Sovythos-303M-Base}}
+}
+```
+
+---
+
+## 👨‍💻 Creator
+
+**Mahmoud Yasser**
+
+Founder of the **Sovythos AI Project**
+
+---
+
+## 📄 License
+
+**Apache License 2.0**
+
+---
+
+<p align="center">
+
+### 🦋 Welcome to the next chapter of the Sovythos ecosystem.
+
+This is an early, actively-training checkpoint shared in the spirit of open, transparent development — not a finished product. Researchers, developers, and builders who believe in open, sovereign AI are welcome to follow along as training progresses.
+
+Future checkpoints will bring lower loss, better coherence, instruction tuning, and long-context support.
+
+⭐ If you find this project useful, please consider following the repository and sharing your experiments.
+
+</p>

config.json

@@ -0,0 +1,16 @@
+{
+  "vocab_size": 32000,
+  "dim": 1024,
+  "n_layers": 24,
+  "n_heads": 16,
+  "n_kv_heads": 4,
+  "max_seq_len": 1024,
+  "rope_theta": 1000000.0,
+  "norm_eps": 1e-06,
+  "ffn_multiple_of": 256,
+  "ffn_dim_multiplier": null,
+  "tie_embeddings": true,
+  "dropout": 0.0,
+  "use_grad_checkpoint": true,
+  "identity": "SOVYTHOS-V2"
+}

configuration_sovythos.py

@@ -0,0 +1,54 @@
+from transformers import PretrainedConfig
+
+class SovythosConfig(PretrainedConfig):
+    model_type = "sovythos"
+
+    def __init__(
+        self,
+        vocab_size=32000,
+        dim=1024,
+        n_layers=24,
+        n_heads=16,
+        n_kv_heads=4,
+        max_seq_len=2048,
+        rope_theta=1000000.0,
+        norm_eps=1e-06,
+        ffn_multiple_of=256,
+        ffn_dim_multiplier=None,
+        tie_embeddings=True,
+        dropout=0.0,
+        use_grad_checkpoint=False,
+        identity="SOVYTHOS-V2",
+        bos_token_id=0,
+        eos_token_id=0,
+        pad_token_id=1,
+        **kwargs,
+    ):
+        self.vocab_size = vocab_size
+        self.dim = dim
+        self.hidden_size = dim
+        self.n_layers = n_layers
+        self.num_hidden_layers = n_layers
+        self.n_heads = n_heads
+        self.num_attention_heads = n_heads
+        self.n_kv_heads = n_kv_heads
+        self.num_key_value_heads = n_kv_heads
+        self.max_seq_len = max_seq_len
+        self.max_position_embeddings = max_seq_len
+        self.rope_theta = rope_theta
+        self.norm_eps = norm_eps
+        self.rms_norm_eps = norm_eps
+        self.ffn_multiple_of = ffn_multiple_of
+        self.ffn_dim_multiplier = ffn_dim_multiplier
+        self.tie_embeddings = tie_embeddings
+        self.dropout = dropout
+        self.use_grad_checkpoint = use_grad_checkpoint
+        self.identity = identity
+        kwargs.pop("tie_word_embeddings", None)
+        super().__init__(
+            bos_token_id=bos_token_id,
+            eos_token_id=eos_token_id,
+            pad_token_id=pad_token_id,
+            tie_word_embeddings=tie_embeddings,
+            **kwargs,
+        )

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "bos_token_id": 0,
+  "eos_token_id": 0,
+  "pad_token_id": 1,
+  "do_sample": true,
+  "temperature": 0.8,
+  "top_k": 40,
+  "top_p": 0.9,
+  "repetition_penalty": 1.15,
+  "no_repeat_ngram_size": 3,
+  "max_new_tokens": 512,
+  "transformers_version": "4.46.0"
+}

modeling_sovythos.py

@@ -0,0 +1,176 @@
+import math
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from transformers import PreTrainedModel
+from transformers.modeling_outputs import CausalLMOutputWithPast
+from .configuration_sovythos import SovythosConfig
+
+
+class RMSNorm(nn.Module):
+    def __init__(self, dim, eps=1e-06):
+        super().__init__()
+        self.eps = eps
+        self.weight = nn.Parameter(torch.ones(dim))
+
+    def forward(self, x):
+        dtype = x.dtype
+        x = x.float()
+        rms = x.pow(2).mean(-1, keepdim=True)
+        x = x * torch.rsqrt(rms + self.eps)
+        return self.weight * x.to(dtype)
+
+
+def precompute_rope(head_dim, max_seq_len, theta, device, dtype=torch.float32):
+    inv_freq = 1.0 / theta ** (torch.arange(0, head_dim, 2, device=device).float() / head_dim)
+    t = torch.arange(max_seq_len, device=device).float()
+    freqs = torch.outer(t, inv_freq)
+    emb = torch.cat((freqs, freqs), dim=-1)
+    return emb.cos().to(dtype), emb.sin().to(dtype)
+
+
+def rotate_half(x):
+    x1, x2 = x.chunk(2, dim=-1)
+    return torch.cat((-x2, x1), dim=-1)
+
+
+def apply_rope(x, cos, sin, offset=0):
+    T = x.shape[-2]
+    cos = cos[offset:offset + T][None, None, :, :]
+    sin = sin[offset:offset + T][None, None, :, :]
+    return x * cos + rotate_half(x) * sin
+
+
+def repeat_kv(x, n_rep):
+    if n_rep == 1:
+        return x
+    B, Hkv, T, D = x.shape
+    return x[:, :, None, :, :].expand(B, Hkv, n_rep, T, D).reshape(B, Hkv * n_rep, T, D)
+
+
+class Attention(nn.Module):
+    def __init__(self, cfg):
+        super().__init__()
+        self.n_heads = cfg.n_heads
+        self.n_kv_heads = cfg.n_kv_heads
+        self.n_rep = self.n_heads // self.n_kv_heads
+        self.head_dim = cfg.dim // cfg.n_heads
+        self.dropout = cfg.dropout
+        self.q_proj = nn.Linear(cfg.dim, cfg.n_heads * self.head_dim, bias=False)
+        self.k_proj = nn.Linear(cfg.dim, cfg.n_kv_heads * self.head_dim, bias=False)
+        self.v_proj = nn.Linear(cfg.dim, cfg.n_kv_heads * self.head_dim, bias=False)
+        self.o_proj = nn.Linear(cfg.n_heads * self.head_dim, cfg.dim, bias=False)
+        self.q_norm = RMSNorm(self.head_dim, cfg.norm_eps)
+        self.k_norm = RMSNorm(self.head_dim, cfg.norm_eps)
+
+    def forward(self, x, cos, sin):
+        B, T, C = x.shape
+        q = self.q_proj(x).view(B, T, self.n_heads, self.head_dim).transpose(1, 2)
+        k = self.k_proj(x).view(B, T, self.n_kv_heads, self.head_dim).transpose(1, 2)
+        v = self.v_proj(x).view(B, T, self.n_kv_heads, self.head_dim).transpose(1, 2)
+        q, k = self.q_norm(q), self.k_norm(k)
+        q = apply_rope(q, cos, sin)
+        k = apply_rope(k, cos, sin)
+        k = repeat_kv(k, self.n_rep)
+        v = repeat_kv(v, self.n_rep)
+        out = F.scaled_dot_product_attention(q, k, v, is_causal=True, dropout_p=self.dropout if self.training else 0.0)
+        out = out.transpose(1, 2).contiguous().view(B, T, -1)
+        return self.o_proj(out)
+
+
+class FeedForward(nn.Module):
+    def __init__(self, cfg):
+        super().__init__()
+        hidden = int(2 * (4 * cfg.dim) / 3)
+        if cfg.ffn_dim_multiplier is not None:
+            hidden = int(cfg.ffn_dim_multiplier * hidden)
+        hidden = cfg.ffn_multiple_of * ((hidden + cfg.ffn_multiple_of - 1) // cfg.ffn_multiple_of)
+        self.gate_proj = nn.Linear(cfg.dim, hidden, bias=False)
+        self.up_proj = nn.Linear(cfg.dim, hidden, bias=False)
+        self.down_proj = nn.Linear(hidden, cfg.dim, bias=False)
+
+    def forward(self, x):
+        return self.down_proj(F.silu(self.gate_proj(x)) * self.up_proj(x))
+
+
+class DecoderBlock(nn.Module):
+    def __init__(self, cfg):
+        super().__init__()
+        self.input_layernorm = RMSNorm(cfg.dim, cfg.norm_eps)
+        self.self_attn = Attention(cfg)
+        self.post_attention_layernorm = RMSNorm(cfg.dim, cfg.norm_eps)
+        self.mlp = FeedForward(cfg)
+
+    def forward(self, x, cos, sin):
+        x = x + self.self_attn(self.input_layernorm(x), cos, sin)
+        x = x + self.mlp(self.post_attention_layernorm(x))
+        return x
+
+
+class SovythosPreTrainedModel(PreTrainedModel):
+    config_class = SovythosConfig
+    base_model_prefix = "model"
+    supports_gradient_checkpointing = True
+
+    def _init_weights(self, module):
+        if isinstance(module, nn.Linear):
+            nn.init.normal_(module.weight, mean=0.0, std=0.02)
+            if module.bias is not None:
+                nn.init.zeros_(module.bias)
+        elif isinstance(module, nn.Embedding):
+            nn.init.normal_(module.weight, mean=0.0, std=0.02)
+
+
+class SovythosForCausalLM(SovythosPreTrainedModel):
+    def __init__(self, config):
+        super().__init__(config)
+        self.embed_tokens = nn.Embedding(config.vocab_size, config.dim)
+        self.layers = nn.ModuleList([DecoderBlock(config) for _ in range(config.n_layers)])
+        self.norm = RMSNorm(config.dim, config.norm_eps)
+        self.lm_head = nn.Linear(config.dim, config.vocab_size, bias=False)
+        if config.tie_embeddings:
+            self.lm_head.weight = self.embed_tokens.weight
+        self.head_dim = config.dim // config.n_heads
+        self._rope_cache = {}
+        self.post_init()
+
+    def get_input_embeddings(self):
+        return self.embed_tokens
+
+    def set_input_embeddings(self, value):
+        self.embed_tokens = value
+
+    def get_output_embeddings(self):
+        return self.lm_head
+
+    def _rope_for(self, device, dtype):
+        # ملاحظة: بنحسب rope live بدل ما نعتمد على register_buffer، لأن
+        # transformers بيحمل الموديل عادةً عبر meta-device (low_cpu_mem_usage)
+        # وأي buffer معمول persistent=False ومش موجود في checkpoint بيفضل
+        # ذاكرة غير مهيأة (garbage) بدل القيم الحقيقية بعد النقل من meta لـ real.
+        key = str(device)
+        if key not in self._rope_cache:
+            cos, sin = precompute_rope(self.head_dim, self.config.max_seq_len, self.config.rope_theta, device=device)
+            self._rope_cache[key] = (cos, sin)
+        return self._rope_cache[key]
+
+    def forward(self, input_ids=None, attention_mask=None, labels=None, **kwargs):
+        cos, sin = self._rope_for(input_ids.device, torch.float32)
+        h = self.embed_tokens(input_ids)
+        for layer in self.layers:
+            h = layer(h, cos, sin)
+        h = self.norm(h)
+        logits = self.lm_head(h)
+        loss = None
+        if labels is not None:
+            shift_logits = logits[:, :-1, :].contiguous()
+            shift_labels = labels[:, 1:].contiguous()
+            loss = F.cross_entropy(
+                shift_logits.view(-1, shift_logits.size(-1)),
+                shift_labels.view(-1),
+                ignore_index=-100,
+            )
+        return CausalLMOutputWithPast(loss=loss, logits=logits)
+
+    def prepare_inputs_for_generation(self, input_ids, **kwargs):
+        return {"input_ids": input_ids}

special_tokens_map.json

@@ -0,0 +1,18 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>",
+  "pad_token": "<|pad|>",
+  "additional_special_tokens": [
+    "<|user|>",
+    "<|assistant|>",
+    "<|system|>",
+    "<|code|>",
+    "<|/code|>",
+    "<|think|>",
+    "<|/think|>",
+    "<|egy|>",
+    "<|msa|>",
+    "<|en|>"
+  ]
+}

tokenizer_config.json

@@ -0,0 +1,24 @@
+{
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "model_max_length": 2048,
+  "clean_up_tokenization_spaces": false,
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>",
+  "pad_token": "<|pad|>",
+  "added_tokens_decoder": {
+    "0": {"content": "<|endoftext|>", "special": true, "single_word": false, "lstrip": false, "rstrip": false, "normalized": false},
+    "1": {"content": "<|pad|>", "special": true, "single_word": false, "lstrip": false, "rstrip": false, "normalized": false},
+    "2": {"content": "<|user|>", "special": true, "single_word": false, "lstrip": false, "rstrip": false, "normalized": false},
+    "3": {"content": "<|assistant|>", "special": true, "single_word": false, "lstrip": false, "rstrip": false, "normalized": false},
+    "4": {"content": "<|system|>", "special": true, "single_word": false, "lstrip": false, "rstrip": false, "normalized": false},
+    "5": {"content": "<|code|>", "special": true, "single_word": false, "lstrip": false, "rstrip": false, "normalized": false},
+    "6": {"content": "<|/code|>", "special": true, "single_word": false, "lstrip": false, "rstrip": false, "normalized": false},
+    "7": {"content": "<|think|>", "special": true, "single_word": false, "lstrip": false, "rstrip": false, "normalized": false},
+    "8": {"content": "<|/think|>", "special": true, "single_word": false, "lstrip": false, "rstrip": false, "normalized": false},
+    "9": {"content": "<|egy|>", "special": true, "single_word": false, "lstrip": false, "rstrip": false, "normalized": false},
+    "10": {"content": "<|msa|>", "special": true, "single_word": false, "lstrip": false, "rstrip": false, "normalized": false},
+    "11": {"content": "<|en|>", "special": true, "single_word": false, "lstrip": false, "rstrip": false, "normalized": false}
+  },
+  "chat_template": "{% for message in messages %}{% if message['role'] == 'system' %}{{ '<|system|>\n' + message['content'] + '\n' }}{% elif message['role'] == 'user' %}{{ '<|user|>\n' + message['content'] + '\n' }}{% elif message['role'] == 'assistant' %}{{ '<|assistant|>\n' + message['content'] + '<|endoftext|>\n' }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|assistant|>\n' }}{% endif %}"
+}