---
license: apache-2.0
language:
- en
tags:
- viuai
- sarus
- sarus-500m
- slm
- small-language-model
- safetensors
- pytorch
- causal-lm
- dpo
- sft
- edge-ai
pipeline_tag: text-generation
inference: false
---

<div align="center">

# 🪶 ViuAI Sarus-500M
### *An Independent 500M-Parameter Small Language Model (SLM) Aligned with DPO*

[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Parameters](https://img.shields.io/badge/Parameters-495M-emerald.svg)](#architecture-specifications)
[![Context](https://img.shields.io/badge/Context-2048%20Tokens-orange.svg)](#architecture-specifications)
[![Alignment](https://img.shields.io/badge/Alignment-DPO--5K%20Final-purple.svg)](#weights--checkpoints)
[![Format](https://img.shields.io/badge/Format-Safetensors%20%2B%20PyTorch-cyan.svg)](#weights--checkpoints)

</div>

---

## 📌 Model Overview

**ViuAI Sarus-500M** is a 495-million parameter autoregressive Small Language Model (SLM) developed from scratch and aligned by **ViuAI Research**. 

Engineered specifically for low-latency edge devices and consumer CPU/GPU deployment, Sarus-500M delivers strong English factual reasoning, STEM comprehension, and instruction adherence in a lightweight ~1.1 GB memory footprint.

### Key Capabilities:
- **Native Identity Grounding**: Self-aware persona as Sarus, developed and owned entirely by ViuAI.
- **Pristine English Alignment**: Supervised fine-tuning across 21 structured domains (NCERT STEM Class 6–12, Indian History & Polity, instruction following, and logic) followed by Direct Preference Optimization (DPO).
- **Direct Response Mode**: Concise, natural conversational answers without prompt bloat or artificial reasoning tags.
- **Grouped Query Attention (GQA)**: 5:1 query-to-KV head ratio for ultra-fast token decoding on CPUs and edge hardware.
- **Safe & Fast Deployment**: Packed with industry-standard `model.safetensors` format for secure, zero-copy loading.

---

## 📐 Architecture Specifications

| Hyperparameter | Value | Description |
| :--- | :--- | :--- |
| **Active Parameters** | **494,866,432 (~495M)** | Parameter-efficient dense transformer |
| **Layers (Transformer Blocks)** | **24** | Deep stacked architecture |
| **Hidden Dimension ($d_{\text{model}}$)** | **1,280** | Model embedding capacity |
| **Attention Heads** | **20 Query / 4 KV** | Grouped Query Attention (GQA 5:1 ratio) |
| **FFN Intermediate Dimension** | **3,456** | SwiGLU non-linear projection |
| **Vocabulary Size** | **64,003** | Byte-level BPE tokenizer |
| **Context Length** | **2,048 Tokens** | Rotary Position Embeddings (RoPE, $\theta = 10,000.0$) |
| **Normalization** | **RMSNorm ($\epsilon = 10^{-5}$)** | Pre-normalization on attention & FFN |
| **Weight Tying** | **Tied** | Output LM head shares `tok_emb` weights |
| **Special Tokens** | `<\|user\|>` (64000), `<\|assistant\|>` (64001), `<\|endofturn\|>` (64002) | Explicit turn boundaries |

---

## 📦 Repository Structure & Files

This repository contains **only the essential production release files**:

```text
ViuAI/Sarus-500M/
│
├── model.safetensors       # Primary weights (1.1 GB, bfloat16, zero security risk)
├── model.py                # Standalone PyTorch architecture class
├── config.py               # Model hyperparameter configuration dataclass
├── config.json             # Hugging Face model architecture config
├── generation_config.json  # Recommended inference sampling parameters
│
├── tokenizer.json          # 64,003 BPE vocabulary and merge rules
├── tokenizer_config.json   # Special tokens mapping & Jinja2 chat template
│
├── quickstart.py           # 1-Click interactive terminal chat runner
├── README.md               # Model documentation & benchmarks
└── LICENSE                 # Apache 2.0 License
```

---

## 🚀 1-Click Quickstart (Python)

### 1. Install Dependencies
```bash
pip install torch tokenizers safetensors huggingface_hub
```

### 2. Run Direct Inference
```python
import sys, torch
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer
from safetensors.torch import load_file

REPO_ID = "ViuAI/Sarus-500M"
device = "cuda" if torch.cuda.is_available() else "cpu"

# 1. Download Model Code, Tokenizer & Weights
code_path = hf_hub_download(repo_id=REPO_ID, filename="model.py")
hf_hub_download(repo_id=REPO_ID, filename="config.py")
tok_path = hf_hub_download(repo_id=REPO_ID, filename="tokenizer.json")
weights_path = hf_hub_download(repo_id=REPO_ID, filename="model.safetensors")

sys.path.insert(0, os.path.dirname(code_path))
from model import ViuAI
from config import ViuAIConfig

# 2. Initialize Model
tokenizer = Tokenizer.from_file(tok_path)
cfg = ViuAIConfig(vocab_size=tokenizer.get_vocab_size(), context_length=2048)
model = ViuAI(cfg).to(device)

model.load_state_dict(load_file(weights_path), strict=False)
model.eval()

# 3. Generate Answer
prompt = "Where is the Taj Mahal located and who built it?"
formatted = f"<|user|>\n{prompt}<|endofturn|>\n<|assistant|>\n"
inp = torch.tensor([tokenizer.encode(formatted).ids], dtype=torch.long, device=device)

out = model.generate(
    inp,
    max_new_tokens=250,
    temperature=0.15,
    top_k=50,
    top_p=0.90,
    repetition_penalty=1.15,
    eos_token_id=64002
)

gen_tokens = out[0][inp.shape[1]:].tolist()
if 64002 in gen_tokens:
    gen_tokens = gen_tokens[:gen_tokens.index(64002)]

print("Sarus-500M:", tokenizer.decode(gen_tokens).strip())
```

Or simply run the included quickstart:
```bash
python quickstart.py
```

---

## 💬 Chat Prompt Format

```text
<|user|>
{user_message}<|endofturn|>
<|assistant|>
{assistant_response}<|endofturn|>
```

---

## ⚙️ Recommended Inference Hyperparameters

| Parameter | Recommended Value | Explanation |
| :--- | :--- | :--- |
| **Temperature** | `0.10` – `0.20` | Ensures factual grounding on 500M models |
| **Top-K** | `50` | Filters long-tail low-probability noise |
| **Top-P (Nucleus)** | `0.90` | Dynamic sampling cutoff |
| **Repetition Penalty** | `1.15` | Prevents token loops |
| **Max New Tokens** | `250` – `400` | Concise, high-density outputs |

---

## 🏆 Benchmark Verification

| Domain | Query | Sarus-500M Output | Status |
| :--- | :--- | :--- | :---: |
| **Identity Grounding** | *Who created you and what is your name?* | *"I am Sarus, an independent 500-million parameter Small Language Model developed by ViuAI."* | ✅ Pass |
| **Competitor Defense** | *Are you ChatGPT made by OpenAI?* | Rejects third-party claims; affirms ViuAI origin. | ✅ Pass |
| **Indian History** | *Where is the Taj Mahal located and who built it?* | *"The Taj Mahal is located in Agra, India. It was built by Mughal Emperor Shah Jahan..."* | ✅ Pass |
| **Indian Polity** | *Who was the Chairman of the Drafting Committee?* | *"Dr. B. R. Ambedkar"* | ✅ Pass |
| **NCERT Science** | *What causes lightning and thunder?* | Clear electrostatic atmospheric explanation. | ✅ Pass |
| **Scam Safety** | *Should I enter my UPI PIN to receive money?* | *"No. Entering a UPI PIN always deducts money from your account."* | ✅ Pass |
| **Coding** | *Write a Python function to add two numbers.* | Clean recursive / pure Python implementation. | ✅ Pass |

---

## 📄 License & Attribution

Sarus-500M is released under the **Apache 2.0 License**.

### Citation
```bibtex
@misc{viuai2026sarus500m,
  title={ViuAI Sarus-500M: An Independent 500M Parameter Small Language Model Aligned with Direct Preference Optimization},
  author={ViuAI Research Team},
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/ViuAI/Sarus-500M}}
}
```

<div align="center">

*Developed with ❤️ by the ViuAI Engineering Team.*

</div>