README.md

---
license: apache-2.0
language:
- en
- ar
tags:
- sovereign
- sovythos
- llm
- transformer
- pytorch
- arabic
- egyptian-arabic
- english
- code
- open-source
- llama-architecture
- gqa
- rope
- swiglu
- safetensors
- gguf
pipeline_tag: text-generation
---

<p align="center">
  <img src="https://huggingface.co/my0919175/Sovythos-303M-Base/resolve/main/sovythos.png" width="320" alt="SOVYTHOS-V2">
</p>

<h1 align="center">🦋 SOVYTHOS-V2</h1>
<h3 align="center">Sovythos Egyptian Reasoning & Code Base Model</h3>

<p align="center">
<b>A production-grade, Llama-compatible decoder-only Transformer trained entirely from scratch for Arabic, English, and Code.</b>
</p>

<p align="center">

![Open Source](https://img.shields.io/badge/Open%20Source-Yes-success)
![Parameters](https://img.shields.io/badge/Parameters-~300M-blue)
![Architecture](https://img.shields.io/badge/Architecture-Llama--compatible-purple)
![Languages](https://img.shields.io/badge/Languages-Arabic%20%7C%20English%20%7C%20Code-orange)
![Framework](https://img.shields.io/badge/PyTorch-2.x-red)
![Precision](https://img.shields.io/badge/Training-AMP%20FP16-yellow)
![Format](https://img.shields.io/badge/Weights-Safetensors%20%2B%20GGUF-brightgreen)
![Status](https://img.shields.io/badge/Status-Early%20Training%20Checkpoint-yellow)

</p>

> ### ⚠️ Training Status: Early-Stage Checkpoint
> **This checkpoint is an early snapshot from an ongoing training run — it is NOT the final SOVYTHOS-V2 release.** The model is uploaded at this stage for architecture testing, integration testing (`transformers`/GGUF compatibility), and community feedback purposes only.
>
> Expect at this stage:
> - Weak or incoherent generations on many prompts
> - Little to no instruction-following ability (this is a **Base**, non-instruction-tuned model on top of being early-stage)
> - Validation loss still actively decreasing — quality will improve substantially with further training
>
> Training is actively continuing. Follow this repository for updates as later checkpoints are released.

---

## 📋 Table of Contents

- [Overview](#overview)
- [Model Specifications](#model-specifications)
- [Architecture Details](#architecture-details)
- [Tokenizer](#tokenizer)
- [Training](#training)
- [Try It Now (Google Colab)](#-try-it-now-google-colab)
- [Usage](#usage)
- [GGUF Inference](#gguf-inference)
- [Intended Uses](#intended-uses)
- [Limitations](#limitations)
- [Roadmap](#roadmap)
- [Citation](#citation)
- [License](#license)

---

## 🦋 Overview

**SOVYTHOS-V2** is the second-generation foundation model of the **Sovythos AI Project** — a fully open-source, decoder-only Transformer trained **from randomly initialized weights** on a multilingual corpus spanning Modern Standard Arabic, Egyptian Arabic, English, and programming languages.

This release represents a significant architectural leap from the original 66M Base model, scaling to **~300 million parameters** while adopting a **Llama-compatible** design (RMSNorm, RoPE, GQA, SwiGLU) to ensure seamless deployment across the modern inference ecosystem, including `llama.cpp`, `llama-cpp-python`, and Hugging Face transformers.

SOVYTHOS-V2 is released as a **Base (Pretrained)** model. It has not undergone instruction tuning or RLHF and is intended for researchers, developers, and educators who require a sovythos, transparent foundation for downstream training.

> **Key Design Principle:** Every component — from the tokenizer to the tensor naming convention — is engineered for **direct compatibility** with the Llama/GGUF toolchain. No conversion hacks required.

---

## 📊 Model Specifications

| Property | Value |
|----------|-------|
| **Model** | SOVYTHOS-V2 |
| **Parameters** | ~300 Million |
| **Architecture** | Decoder-only Transformer (Llama-style) |
| **Hidden Size** | 1,024 |
| **Layers** | 24 |
| **Attention Heads** | 16 (Query) / 4 (Key-Value) |
| **Head Dimension** | 64 |
| **Context Length** | 2,048 tokens |
| **RoPE Theta** | 1,000,000 |
| **Vocabulary Size** | 32,000 |
| **Framework** | PyTorch 2.x |
| **Weight Format** | Safetensors + GGUF (F16 / F32) |
| **Training** | From Scratch |
| **Release** | Base |

---

## 🏗️ Architecture Details

SOVYTHOS-V2 implements a modern, efficient decoder stack aligned with the Llama architecture specification for frictionless ecosystem integration.

### Normalization
- **RMSNorm** applied pre-attention and post-attention (no bias, no learned affine shift)

### Position Encoding
- **RoPE (Rotary Position Embedding)** with rotate_half (NEOX-style) implementation
- Base frequency θ = 1,000,000
- Precomputed cosine/sine caches up to max sequence length

### Attention Mechanism
- **Grouped Query Attention (GQA)** with 16 query heads and 4 key-value heads
- Reduces KV-cache memory footprint by **4×** during inference compared to standard MHA
- Compatible with memory-efficient attention kernels (`scaled_dot_product_attention`)

### Feed-Forward Network
- **SwiGLU** activation with gated projection
- Hidden dimension rounded to nearest multiple of 256 for optimal GPU kernel utilization
- `gate_proj`, `up_proj`, `down_proj` naming matches Llama/GGUF conventions exactly

### Weight Initialization
- Gaussian initialization (σ = 0.02) for embeddings and linear layers
- Scaled down-projection and output-projection initialization by `1/√(2×layers)` to stabilize deep training

### Training Optimizations
- **torch.compile** graph mode (when available)
- **TF32** matmul acceleration on Ampere+
- **Automatic Mixed Precision (AMP)** with `fp16` forward pass
- **Gradient Checkpointing** for long-context training on consumer GPUs
- **Gradient Accumulation** with effective batch scaling
- **Cosine LR Decay** with linear warmup
- **Gradient Clipping** (max norm = 1.0)

---

## 🔤 Tokenizer

SOVYTHOS-V2 ships with a **custom-trained Byte-Pair Encoding (BPE)** tokenizer optimized for Arabic morphology, English, and code syntax.

### Design Choices

| Feature | Implementation |
|---------|---------------|
| **Pre-tokenization** | GPT-4 / cl100k-style regex split (Unicode-aware) |
| **Numerical Splitting** | Digits grouped by 3 (e.g., `1,234,567` → `1` `234` `567`) |
| **Fallback** | ByteLevel encoding for lossless coverage of any Unicode symbol |
| **Normalization** | NFC Unicode normalization (unifies Arabic hamza & diacritic forms) |
| **Special Tokens** | `<\|endoftext\|>`, `<\|pad\|>`, `<\|user\|>`, `<\|assistant\|>`, `<\|system\|>`, `<\|code\|>`, `<\|/code\|>`, `<\|think\|>`, `<\|/think\|>` |
| **Vocab Size** | 32,000 |
| **Min Frequency** | 2 |

### Coverage Efficiency (Tokens per Character)

Lower is better. Measured on a held-out sample of the training corpus:

| Domain | Tokens / Char | Notes |
|--------|---------------|-------|
| **Arabic (MSA + Egyptian)** | ~0.35 | Efficient subword decomposition for Semitic morphology |
| **English** | ~0.28 | Competitive with modern English-centric tokenizers |
| **Code** | ~0.42 | Strong performance on Python / JavaScript identifiers |

> The tokenizer is saved as a standard `tokenizer.json` (Hugging Face `tokenizers` format) and its merges/vocab are **directly compatible** with GGUF `gpt2` tokenizer model type.

---

## 🏋️ Training

SOVYTHOS-V2 was trained **entirely from scratch**. No pretrained weights from Llama, GPT, or any other foundation model were used as initialization.

### Data Pipeline
- **Corpus**: Multilingual web text, technical documentation, mathematical content, and code repositories
- **Languages**: Modern Standard Arabic, Egyptian Arabic, English, and multilingual programming languages
- **Cleaning**: Duplicate-line removal, short-line filtering (< 2 chars), NFC normalization, and aggressive deduplication of repeated contiguous lines
- **Format**: Binary tokenized memmap for high-throughout training with `np.uint16` encoding

### Training Recipe

| Hyperparameter | Value |
|----------------|-------|
| Optimizer | AdamW (β₁=0.9, β₂=0.95, ε=1e-8) |
| Peak Learning Rate | 3e-4 |
| Minimum Learning Rate | 3e-5 |
| Warmup Steps | 1,000 |
| LR Schedule | Cosine decay |
| Weight Decay | 0.1 (2D+ tensors only) |
| Max Gradient Norm | 1.0 |
| Precision | FP16 (AMP) with FP32 master weights |
| Batch Size | 4 (per device) |
| Gradient Accumulation | 8 steps |
| Effective Batch Size | 32 |
| Block Size | 512 tokens |

### Checkpointing & Resumption
- Checkpoints saved in **Safetensors** format (secure, fast, language-agnostic)
- Automatic resumption from `ckpt_dir/best/` including step count and best validation loss
- Best checkpoint selection based on validation loss

### Export
- **Native GGUF export** after training (or from existing checkpoints via `--export_gguf_only`)
- Supported quantizations: `F32`, `F16` (post-hoc Q4/Q8 via `llama.cpp` tools)
- Tensor name map strictly follows Llama GGUF specification (`blk.{i}.attn_q.weight`, `blk.{i}.ffn_gate.weight`, etc.)

---

## 🚀 Try It Now (Google Colab)

The fastest way to test SOVYTHOS-V2 is directly through 🤗 `transformers`, using the custom model code shipped in this repository (`trust_remote_code=True`). No manual setup, no cloning — just run the cell below in a fresh Colab notebook.

```python
# ============================================================
# 🔱 SOVYTHOS-V2 (303M-Base) — Colab Quickstart
# ============================================================

!pip install -q -U transformers accelerate safetensors

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_ID = "my0919175/Sovythos-303M-Base"

print("🔄 Loading tokenizer...")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)

print("🔄 Loading model...")
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    dtype=torch.float32,
    trust_remote_code=True,
).to("cuda" if torch.cuda.is_available() else "cpu")
model.eval()

# Chat-style prompt using the built-in special tokens
prompt = "<|user|>\nمرحبا, احكيلي عن نفسك\n<|assistant|>\n"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)

with torch.no_grad():
    output = model.generate(
        input_ids,
        max_new_tokens=150,
        do_sample=True,
        temperature=0.8,
        top_k=40,
        top_p=0.9,
        repetition_penalty=1.15,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))
```

> **Note:** This is an **early-stage training checkpoint** of a **Base** (non-instruction-tuned) model — not the final release. Outputs may be short, repetitive, incoherent, or diverge from the prompt entirely; this is expected at this stage of training and is not a loading error. Quality will improve as training progresses and with the upcoming Instruct release.
>
> When prompted `Do you wish to run the custom code? [y/N]`, type **y** — this loads `configuration_sovythos.py` and `modeling_sovythos.py` from this repo, which implement the GQA + RoPE + SwiGLU architecture described above.

---

## 💻 Usage

### Loading with 🤗 Transformers (Recommended)

This repository ships with `trust_remote_code`-compatible files (`configuration_sovythos.py`, `modeling_sovythos.py`), so it works directly with the standard `AutoModelForCausalLM` / `AutoTokenizer` API — no need to clone the original training repo:

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "my0919175/Sovythos-303M-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=torch.float32, trust_remote_code=True
).to("cuda" if torch.cuda.is_available() else "cpu")

prompt = "<|user|>\nThe capital of Egypt is\n<|assistant|>\n"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)

output = model.generate(
    input_ids,
    max_new_tokens=50,
    temperature=0.8,
    top_k=40,
    top_p=0.9,
    eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

### Loading with the Native Training Code

If you have the original training repository (`model.py`, `train.py`) available locally, you can also load the checkpoint directly through the native `Model` class, which supports the custom KV-cache and generation loop used during training/research:

```python
from model import Model

model = Model.from_pretrained("sovythos_ckpt/best", device="cuda")

text = "The capital of Egypt is"
ids = tokenizer.encode(text).ids
input_ids = torch.tensor([ids], device="cuda")

output = model.generate(
    input_ids,
    max_new_tokens=50,
    temperature=0.8,
    top_k=40,
    top_p=0.9,
    eos_token_id=tokenizer.token_to_id("<|endoftext|>")
)
print(tokenizer.decode(output[0].tolist()))
```

### Using the Tokenizer Directly

```python
from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer.json")
encoded = tok.encode("إزيك يا كبير؟")
print(encoded.ids)  # [ ... ]
print(tok.decode(encoded.ids))  # "إزيك يا كبير؟"
```

---

## 🚀 GGUF Inference

Because SOVYTHOS-V2 uses **Llama-compatible tensor naming** and a **GPT-2-format tokenizer**, the exported GGUF file runs out-of-the-box with `llama.cpp` and `llama-cpp-python`.

### llama-cpp-python

```python
from llama_cpp import Llama

llm = Llama(
    model_path="sovythos-v2.f16.gguf",
    n_ctx=2048,
    n_gpu_layers=-1,  # Offload all layers to GPU
    verbose=False
)

output = llm(
    "The capital of Egypt is",
    max_tokens=50,
    temperature=0.8,
    top_k=40,
    top_p=0.9,
    stop=["<|endoftext|>"]
)
print(output["choices"][0]["text"])
```

### llama.cpp CLI

```bash
./main -m sovythos-v2.f16.gguf        -n 50        --temp 0.8        --top-k 40        --top-p 0.9        -p "The capital of Egypt is"
```

> **No conversion scripts required.** The GGUF file is produced directly by the training framework with correct `llama` arch metadata.

---

## 🎯 Intended Uses

SOVYTHOS-V2 is designed as a **research and development foundation** for:

- Continued pretraining on domain-specific corpora
- Supervised fine-tuning (SFT) for chat or task-specific behavior
- Instruction tuning and alignment experiments
- Educational exploration of modern decoder-only architectures
- Arabic NLP research (MSA, Egyptian dialect, code-switching)
- Low-resource and Sovythos AI development

> **Not intended for:** Direct deployment as a production chatbot without additional safety fine-tuning and evaluation.

---

## ⚠️ Limitations

As a **Base (pretrained)** model, SOVYTHOS-V2 exhibits expected limitations:

- **No instruction following** without fine-tuning
- **Hallucination** of factual information (names, dates, technical details)
- **Repetition** in long-form generation
- **Incomplete reasoning** on complex multi-step problems
- **Limited context coherence** beyond a few hundred tokens in zero-shot settings
- **Code generation** produces syntactically valid structures but may contain logic errors

These limitations are standard for base models of this scale and will be addressed in future Instruct/Chat releases.

---

## 🛣️ Roadmap

| Milestone | Status |
|-----------|--------|
| 🟡 SOVYTHOS-V2 Base | **In Training (early checkpoint published)** |
| ⬜ SOVYTHOS-V2 Base (final) | Planned |
| ⬜ SOVYTHOS-V2-Instruct | Planned |
| ⬜ SOVYTHOS Chat | Planned |
| ⬜ Larger Parameter Models (1B+) | Planned |
| ⬜ Long Context Extension (8K+) | Planned |
| ⬜ Advanced Reasoning Tuning | Planned |
| ⬜ Improved Egyptian Dialect Understanding | Planned |

---

## 🌌 Sovythos AI Project

SOVYTHOS-V2 is the flagship foundation model of the **Sovythos AI Project**, an open ecosystem dedicated to building transparent, multilingual language models with **native Arabic support** and modern reasoning capabilities.

The project prioritizes:
- **Sovereignty**: Full training pipeline ownership (data → tokenizer → model → export)
- **Transparency**: Open weights, open code, open training logs
- **Accessibility**: GGUF compatibility for edge and consumer hardware
- **Community**: Research-first releases with clear limitation disclosures

---

## 📖 Citation

If you use SOVYTHOS-V2 in your research, please cite:

```bibtex
@misc{sovythosv2,
  title={SOVYTHOS-V2: Sovythos Egyptian Reasoning and Code Base Model},
  author={Mahmoud Yasser},
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/my0919175/Sovythos-303M-Base}}
}
```

---

## 👨‍💻 Creator

**Mahmoud Yasser**

Founder of the **Sovythos AI Project**

---

## 📄 License

**Apache License 2.0**

---

<p align="center">

### 🦋 Welcome to the next chapter of the Sovythos ecosystem.

This is an early, actively-training checkpoint shared in the spirit of open, transparent development — not a finished product. Researchers, developers, and builders who believe in open, sovereign AI are welcome to follow along as training progresses.

Future checkpoints will bring lower loss, better coherence, instruction tuning, and long-context support.

⭐ If you find this project useful, please consider following the repository and sharing your experiments.

</p>