README.md · ViuAI/Sarus-500M at main

Update README.md for official release

1ce338e verified 13 days ago

7.82 kB

---
license: apache-2.0
language:
- en
tags:
- viuai
- sarus
- sarus-500m
- slm
- small-language-model
- safetensors
- pytorch
- causal-lm
- dpo
- sft
- edge-ai
pipeline_tag: text-generation
inference: false
---
<div align="center">

# 🪶 ViuAI Sarus-500M
### *An Independent 500M-Parameter Small Language Model (SLM) Aligned with DPO*

[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
[![Parameters](https://img.shields.io/badge/Parameters-495M-emerald.svg)](#architecture-specifications)
[![Context](https://img.shields.io/badge/Context-2048%20Tokens-orange.svg)](#architecture-specifications)
[![Alignment](https://img.shields.io/badge/Alignment-DPO--5K%20Final-purple.svg)](#weights--checkpoints)
[![Format](https://img.shields.io/badge/Format-Safetensors%20%2B%20PyTorch-cyan.svg)](#weights--checkpoints)

</div>

---

## 📌 Model Overview

**ViuAI Sarus-500M** is a 495-million parameter autoregressive Small Language Model (SLM) developed from scratch and aligned by **ViuAI Research**. 

Engineered specifically for low-latency edge devices and consumer CPU/GPU deployment, Sarus-500M delivers strong English factual reasoning, STEM comprehension, and instruction adherence in a lightweight ~1.1 GB memory footprint.

### Key Capabilities:
- **Native Identity Grounding**: Self-aware persona as Sarus, developed and owned entirely by ViuAI.
- **Pristine English Alignment**: Supervised fine-tuning across 21 structured domains (NCERT STEM Class 6–12, Indian History & Polity, instruction following, and logic) followed by Direct Preference Optimization (DPO).
- **Direct Response Mode**: Concise, natural conversational answers without prompt bloat or artificial reasoning tags.
- **Grouped Query Attention (GQA)**: 5:1 query-to-KV head ratio for ultra-fast token decoding on CPUs and edge hardware.
- **Safe & Fast Deployment**: Packed with industry-standard `model.safetensors` format for secure, zero-copy loading.

---

## 📐 Architecture Specifications

| Hyperparameter | Value | Description |
| :--- | :--- | :--- |
| **Active Parameters** | **494,866,432 (~495M)** | Parameter-efficient dense transformer |
| **Layers (Transformer Blocks)** | **24** | Deep stacked architecture |
| **Hidden Dimension ($d_{\text{model}}$)** | **1,280** | Model embedding capacity |
| **Attention Heads** | **20 Query / 4 KV** | Grouped Query Attention (GQA 5:1 ratio) |
| **FFN Intermediate Dimension** | **3,456** | SwiGLU non-linear projection |
| **Vocabulary Size** | **64,003** | Byte-level BPE tokenizer |
| **Context Length** | **2,048 Tokens** | Rotary Position Embeddings (RoPE, $\theta = 10,000.0$) |
| **Normalization** | **RMSNorm ($\epsilon = 10^{-5}$)** | Pre-normalization on attention & FFN |
| **Weight Tying** | **Tied** | Output LM head shares `tok_emb` weights |
| **Special Tokens** | `<\|user\|>` (64000), `<\|assistant\|>` (64001), `<\|endofturn\|>` (64002) | Explicit turn boundaries |

---

## 📦 Repository Structure & Files

This repository contains **only the essential production release files**:

```text
ViuAI/Sarus-500M/
│
├── model.safetensors       # Primary weights (1.1 GB, bfloat16, zero security risk)
├── model.py                # Standalone PyTorch architecture class
├── config.py               # Model hyperparameter configuration dataclass
├── config.json             # Hugging Face model architecture config
├── generation_config.json  # Recommended inference sampling parameters
│
├── tokenizer.json          # 64,003 BPE vocabulary and merge rules
├── tokenizer_config.json   # Special tokens mapping & Jinja2 chat template
│
├── quickstart.py           # 1-Click interactive terminal chat runner
├── README.md               # Model documentation & benchmarks
└── LICENSE                 # Apache 2.0 License
```

---

## 🚀 1-Click Quickstart (Python)

### 1. Install Dependencies
```bash
pip install torch tokenizers safetensors huggingface_hub
```

### 2. Run Direct Inference
```python
import sys, torch
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer
from safetensors.torch import load_file
REPO_ID = "ViuAI/Sarus-500M"
device = "cuda" if torch.cuda.is_available() else "cpu"
# 1. Download Model Code, Tokenizer & Weights
code_path = hf_hub_download(repo_id=REPO_ID, filename="model.py")
hf_hub_download(repo_id=REPO_ID, filename="config.py")
tok_path = hf_hub_download(repo_id=REPO_ID, filename="tokenizer.json")
weights_path = hf_hub_download(repo_id=REPO_ID, filename="model.safetensors")
sys.path.insert(0, os.path.dirname(code_path))
from model import ViuAI
from config import ViuAIConfig
# 2. Initialize Model
tokenizer = Tokenizer.from_file(tok_path)
cfg = ViuAIConfig(vocab_size=tokenizer.get_vocab_size(), context_length=2048)
model = ViuAI(cfg).to(device)
model.load_state_dict(load_file(weights_path), strict=False)
model.eval()
# 3. Generate Answer
prompt = "Where is the Taj Mahal located and who built it?"
formatted = f"<|user|>\n{prompt}<|endofturn|>\n<|assistant|>\n"
inp = torch.tensor([tokenizer.encode(formatted).ids], dtype=torch.long, device=device)
out = model.generate(
    inp,
    max_new_tokens=250,
    temperature=0.15,
    top_k=50,
    top_p=0.90,
    repetition_penalty=1.15,
    eos_token_id=64002
)
gen_tokens = out[0][inp.shape[1]:].tolist()
if 64002 in gen_tokens:
    gen_tokens = gen_tokens[:gen_tokens.index(64002)]
print("Sarus-500M:", tokenizer.decode(gen_tokens).strip())
```

Or simply run the included quickstart:
```bash
python quickstart.py
```

---

## 💬 Chat Prompt Format

```text
<|user|>
{user_message}<|endofturn|>
<|assistant|>
{assistant_response}<|endofturn|>
```

---

## ⚙️ Recommended Inference Hyperparameters

| Parameter | Recommended Value | Explanation |
| :--- | :--- | :--- |
| **Temperature** | `0.10` – `0.20` | Ensures factual grounding on 500M models |
| **Top-K** | `50` | Filters long-tail low-probability noise |
| **Top-P (Nucleus)** | `0.90` | Dynamic sampling cutoff |
| **Repetition Penalty** | `1.15` | Prevents token loops |
| **Max New Tokens** | `250` – `400` | Concise, high-density outputs |

---

## 🏆 Benchmark Verification

| Domain | Query | Sarus-500M Output | Status |
| :--- | :--- | :--- | :---: |
| **Identity Grounding** | *Who created you and what is your name?* | *"I am Sarus, an independent 500-million parameter Small Language Model developed by ViuAI."* | ✅ Pass |
| **Competitor Defense** | *Are you ChatGPT made by OpenAI?* | Rejects third-party claims; affirms ViuAI origin. | ✅ Pass |
| **Indian History** | *Where is the Taj Mahal located and who built it?* | *"The Taj Mahal is located in Agra, India. It was built by Mughal Emperor Shah Jahan..."* | ✅ Pass |
| **Indian Polity** | *Who was the Chairman of the Drafting Committee?* | *"Dr. B. R. Ambedkar"* | ✅ Pass |
| **NCERT Science** | *What causes lightning and thunder?* | Clear electrostatic atmospheric explanation. | ✅ Pass |
| **Scam Safety** | *Should I enter my UPI PIN to receive money?* | *"No. Entering a UPI PIN always deducts money from your account."* | ✅ Pass |
| **Coding** | *Write a Python function to add two numbers.* | Clean recursive / pure Python implementation. | ✅ Pass |

---

## 📄 License & Attribution

Sarus-500M is released under the **Apache 2.0 License**.

### Citation
```bibtex
@misc{viuai2026sarus500m,
  title={ViuAI Sarus-500M: An Independent 500M Parameter Small Language Model Aligned with Direct Preference Optimization},
  author={ViuAI Research Team},
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/ViuAI/Sarus-500M}}
}
```

<div align="center">

*Developed with ❤️ by the ViuAI Engineering Team.*

</div>

	---
	license: apache-2.0
	language:
	- en
	tags:
	- viuai
	- sarus
	- sarus-500m
	- slm
	- small-language-model
	- safetensors
	- pytorch
	- causal-lm
	- dpo
	- sft
	- edge-ai
	pipeline_tag: text-generation
	inference: false
	---

	<div align="center">

	# 🪶 ViuAI Sarus-500M
	### An Independent 500M-Parameter Small Language Model (SLM) Aligned with DPO

	[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
	[![Parameters](https://img.shields.io/badge/Parameters-495M-emerald.svg)](#architecture-specifications)
	[![Context](https://img.shields.io/badge/Context-2048%20Tokens-orange.svg)](#architecture-specifications)
	[![Alignment](https://img.shields.io/badge/Alignment-DPO--5K%20Final-purple.svg)](#weights--checkpoints)
	[![Format](https://img.shields.io/badge/Format-Safetensors%20%2B%20PyTorch-cyan.svg)](#weights--checkpoints)

	</div>

	---

	## 📌 Model Overview

	ViuAI Sarus-500M is a 495-million parameter autoregressive Small Language Model (SLM) developed from scratch and aligned by ViuAI Research.

	Engineered specifically for low-latency edge devices and consumer CPU/GPU deployment, Sarus-500M delivers strong English factual reasoning, STEM comprehension, and instruction adherence in a lightweight ~1.1 GB memory footprint.

	### Key Capabilities:
	- Native Identity Grounding: Self-aware persona as Sarus, developed and owned entirely by ViuAI.
	- Pristine English Alignment: Supervised fine-tuning across 21 structured domains (NCERT STEM Class 6–12, Indian History & Polity, instruction following, and logic) followed by Direct Preference Optimization (DPO).
	- Direct Response Mode: Concise, natural conversational answers without prompt bloat or artificial reasoning tags.
	- Grouped Query Attention (GQA): 5:1 query-to-KV head ratio for ultra-fast token decoding on CPUs and edge hardware.
	- Safe & Fast Deployment: Packed with industry-standard `model.safetensors` format for secure, zero-copy loading.

	---

	## 📐 Architecture Specifications

	\| Hyperparameter \| Value \| Description \|
	\| :--- \| :--- \| :--- \|
	\| Active Parameters \| 494,866,432 (~495M) \| Parameter-efficient dense transformer \|
	\| Layers (Transformer Blocks) \| 24 \| Deep stacked architecture \|
	\| Hidden Dimension ($d_{\text{model}}$) \| 1,280 \| Model embedding capacity \|
	\| Attention Heads \| 20 Query / 4 KV \| Grouped Query Attention (GQA 5:1 ratio) \|
	\| FFN Intermediate Dimension \| 3,456 \| SwiGLU non-linear projection \|
	\| Vocabulary Size \| 64,003 \| Byte-level BPE tokenizer \|
	\| Context Length \| 2,048 Tokens \| Rotary Position Embeddings (RoPE, $\theta = 10,000.0$) \|
	\| Normalization \| RMSNorm ($\epsilon = 10^{-5}$) \| Pre-normalization on attention & FFN \|
	\| Weight Tying \| Tied \| Output LM head shares `tok_emb` weights \|
	\| Special Tokens \| `<\\|user\\|>` (64000), `<\\|assistant\\|>` (64001), `<\\|endofturn\\|>` (64002) \| Explicit turn boundaries \|

	---

	## 📦 Repository Structure & Files

	This repository contains only the essential production release files:

	```text
	ViuAI/Sarus-500M/
	│
	├── model.safetensors # Primary weights (1.1 GB, bfloat16, zero security risk)
	├── model.py # Standalone PyTorch architecture class
	├── config.py # Model hyperparameter configuration dataclass
	├── config.json # Hugging Face model architecture config
	├── generation_config.json # Recommended inference sampling parameters
	│
	├── tokenizer.json # 64,003 BPE vocabulary and merge rules
	├── tokenizer_config.json # Special tokens mapping & Jinja2 chat template
	│
	├── quickstart.py # 1-Click interactive terminal chat runner
	├── README.md # Model documentation & benchmarks
	└── LICENSE # Apache 2.0 License
	```

	---

	## 🚀 1-Click Quickstart (Python)

	### 1. Install Dependencies
	```bash
	pip install torch tokenizers safetensors huggingface_hub
	```

	### 2. Run Direct Inference
	```python
	import sys, torch
	from huggingface_hub import hf_hub_download
	from tokenizers import Tokenizer
	from safetensors.torch import load_file

	REPO_ID = "ViuAI/Sarus-500M"
	device = "cuda" if torch.cuda.is_available() else "cpu"

	# 1. Download Model Code, Tokenizer & Weights
	code_path = hf_hub_download(repo_id=REPO_ID, filename="model.py")
	hf_hub_download(repo_id=REPO_ID, filename="config.py")
	tok_path = hf_hub_download(repo_id=REPO_ID, filename="tokenizer.json")
	weights_path = hf_hub_download(repo_id=REPO_ID, filename="model.safetensors")

	sys.path.insert(0, os.path.dirname(code_path))
	from model import ViuAI
	from config import ViuAIConfig

	# 2. Initialize Model
	tokenizer = Tokenizer.from_file(tok_path)
	cfg = ViuAIConfig(vocab_size=tokenizer.get_vocab_size(), context_length=2048)
	model = ViuAI(cfg).to(device)

	model.load_state_dict(load_file(weights_path), strict=False)
	model.eval()

	# 3. Generate Answer
	prompt = "Where is the Taj Mahal located and who built it?"
	formatted = f"<\|user\|>\n{prompt}<\|endofturn\|>\n<\|assistant\|>\n"
	inp = torch.tensor([tokenizer.encode(formatted).ids], dtype=torch.long, device=device)

	out = model.generate(
	inp,
	max_new_tokens=250,
	temperature=0.15,
	top_k=50,
	top_p=0.90,
	repetition_penalty=1.15,
	eos_token_id=64002
	)

	gen_tokens = out[0][inp.shape[1]:].tolist()
	if 64002 in gen_tokens:
	gen_tokens = gen_tokens[:gen_tokens.index(64002)]

	print("Sarus-500M:", tokenizer.decode(gen_tokens).strip())
	```

	Or simply run the included quickstart:
	```bash
	python quickstart.py
	```

	---

	## 💬 Chat Prompt Format

	```text
	<\|user\|>
	{user_message}<\|endofturn\|>
	<\|assistant\|>
	{assistant_response}<\|endofturn\|>
	```

	---

	## ⚙️ Recommended Inference Hyperparameters

	\| Parameter \| Recommended Value \| Explanation \|
	\| :--- \| :--- \| :--- \|
	\| Temperature \| `0.10` – `0.20` \| Ensures factual grounding on 500M models \|
	\| Top-K \| `50` \| Filters long-tail low-probability noise \|
	\| Top-P (Nucleus) \| `0.90` \| Dynamic sampling cutoff \|
	\| Repetition Penalty \| `1.15` \| Prevents token loops \|
	\| Max New Tokens \| `250` – `400` \| Concise, high-density outputs \|

	---

	## 🏆 Benchmark Verification

	\| Domain \| Query \| Sarus-500M Output \| Status \|
	\| :--- \| :--- \| :--- \| :---: \|
	\| Identity Grounding \| Who created you and what is your name? \| "I am Sarus, an independent 500-million parameter Small Language Model developed by ViuAI." \| ✅ Pass \|
	\| Competitor Defense \| Are you ChatGPT made by OpenAI? \| Rejects third-party claims; affirms ViuAI origin. \| ✅ Pass \|
	\| Indian History \| Where is the Taj Mahal located and who built it? \| "The Taj Mahal is located in Agra, India. It was built by Mughal Emperor Shah Jahan..." \| ✅ Pass \|
	\| Indian Polity \| Who was the Chairman of the Drafting Committee? \| "Dr. B. R. Ambedkar" \| ✅ Pass \|
	\| NCERT Science \| What causes lightning and thunder? \| Clear electrostatic atmospheric explanation. \| ✅ Pass \|
	\| Scam Safety \| Should I enter my UPI PIN to receive money? \| "No. Entering a UPI PIN always deducts money from your account." \| ✅ Pass \|
	\| Coding \| Write a Python function to add two numbers. \| Clean recursive / pure Python implementation. \| ✅ Pass \|

	---

	## 📄 License & Attribution

	Sarus-500M is released under the Apache 2.0 License.

	### Citation
	```bibtex
	@misc{viuai2026sarus500m,
	title={ViuAI Sarus-500M: An Independent 500M Parameter Small Language Model Aligned with Direct Preference Optimization},
	author={ViuAI Research Team},
	year={2026},
	publisher={Hugging Face},
	howpublished={\url{https://huggingface.co/ViuAI/Sarus-500M}}
	}
	```

	<div align="center">

	Developed with ❤️ by the ViuAI Engineering Team.

	</div>