README.md

metadata

license: apache-2.0
language:
  - en
tags:
  - viuai
  - sarus
  - sarus-500m
  - slm
  - small-language-model
  - pytorch
  - causal-lm
  - dpo
  - sft
  - edge-ai
pipeline_tag: text-generation
inference: false

🪶 ViuAI Sarus-500M

An Independent 500M-Parameter Small Language Model (SLM) Aligned with DPO

---

📌 Model Overview

ViuAI Sarus-500M is a 495-million parameter autoregressive Small Language Model (SLM) designed, trained from scratch, and aligned by ViuAI Research.

Engineered for ultra-low latency edge devices and lightweight CPU/GPU deployment, Sarus-500M balances deep knowledge representation with minimal resource requirements (~944 MB model footprint).

Key Highlights:

• Independent Identity: Native self-grounding as Sarus, developed and owned entirely by ViuAI.
• Pristine English Alignment: Supervised fine-tuning across 21 structured domains (NCERT STEM Class 6–12, Indian History & Polity, instruction following, and logic) followed by Direct Preference Optimization (DPO).
• Direct Response Mode: Produces clean, conversational, direct answers without unnecessary rambling or internal thinking tokens.
• Grouped Query Attention (GQA): 5:1 query-to-KV head ratio for ultra-fast, memory-efficient decoding on consumer CPUs and edge hardware.

---

📐 Architecture Specifications

Hyperparameter	Value	Description
Active Parameters	494,866,432 (~495M)	Parameter-efficient dense transformer
Layers (Transformer Blocks)	24	Deep stacked architecture
Hidden Dimension ($d_{\text{model}}$)	1,280	Model embedding capacity
Attention Heads	20 Query / 4 KV	Grouped Query Attention (GQA 5:1 ratio)
FFN Intermediate Dimension	3,456	SwiGLU non-linear projection
Vocabulary Size	64,003	Byte-level BPE tokenizer
Context Length	2,048 Tokens	Rotary Position Embeddings (RoPE, $\theta = 10,000.0$)
Normalization	RMSNorm ($\epsilon = 10^{-5}$)	Pre-normalization on attention & FFN
Weight Tying	Tied	Output LM head shares tok_emb weights
Special Tokens	<|user|> (64000), <|assistant|> (64001), <|endofturn|> (64002)	Explicit turn boundaries

---

🎯 Recommended Checkpoint

The official recommended production checkpoint is:

Checkpoint Path	Size	Description	Status
dpo_checkpoints/dpo_5k/dpo_5k_final.pt	944 MB	Final DPO-5K Post-Alignment Checkpoint	🌟 Recommended for Deployment
sft_checkpoints/500M_Sarus_SFT/500m_sarus_sft_best.pt	944 MB	Pre-DPO Supervised Fine-Tuned Baseline	Baseline Reference

---

🚀 1-Click Quickstart (Python)

You can run Sarus-500M locally in a few lines of code without requiring the heavy transformers package:

1. Install Lightweight Dependencies

pip install torch tokenizers huggingface_hub

2. Run Inference Script

import os, sys, torch
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

REPO_ID = "ViuAI/ViuAI-500M"
device = "cuda" if torch.cuda.is_available() else "cpu"

# 1. Download Model Architecture & Tokenizer
code_file = hf_hub_download(repo_id=REPO_ID, filename="code/model.py")
config_file = hf_hub_download(repo_id=REPO_ID, filename="code/config.py")
tok_file = hf_hub_download(repo_id=REPO_ID, filename="tokenizer.json")
ckpt_file = hf_hub_download(repo_id=REPO_ID, filename="dpo_checkpoints/dpo_5k/dpo_5k_final.pt")

sys.path.insert(0, os.path.dirname(code_file))
from model import ViuAI
from config import ViuAIConfig

# 2. Initialize Model & Weights
tokenizer = Tokenizer.from_file(tok_file)
cfg = ViuAIConfig(vocab_size=tokenizer.get_vocab_size(), context_length=2048)
model = ViuAI(cfg).to(device)

state = torch.load(ckpt_file, map_location=device, weights_only=True)
raw_sd = state.get("model", state.get("model_state_dict", state))
clean_sd = {k.replace("_orig_mod.", "").replace("module.", ""): v for k, v in raw_sd.items()}
model.load_state_dict(clean_sd, strict=False)
model.eval()

# 3. Generate Response
prompt = "Where is the Taj Mahal located and who built it?"
formatted = f"<|user|>\n{prompt}<|endofturn|>\n<|assistant|>\n"
inp = torch.tensor([tokenizer.encode(formatted).ids], dtype=torch.long, device=device)

out = model.generate(
    inp,
    max_new_tokens=250,
    temperature=0.10,
    top_k=50,
    top_p=0.90,
    repetition_penalty=1.15,
    eos_token_id=64002
)

response_tokens = out[0][inp.shape[1]:].tolist()
if 64002 in response_tokens:
    response_tokens = response_tokens[:response_tokens.index(64002)]

print("Sarus-500M:", tokenizer.decode(response_tokens).strip())

---

💬 Chat Prompt Format

Sarus-500M expects structured turn delimiters:

<|user|>
{user_message}<|endofturn|>
<|assistant|>
{assistant_response}<|endofturn|>

Multi-Turn Conversation Example:

<|user|>
Who are you?<|endofturn|>
<|assistant|>
I am Sarus, an independent 500-million parameter Small Language Model developed by ViuAI.<|endofturn|>
<|user|>
What is the capital of France?<|endofturn|>
<|assistant|>
The capital of France is Paris.<|endofturn|>

---

⚙️ Recommended Inference Hyperparameters

Because Sarus-500M is a 500M SLM, setting proper sampling parameters prevents logit tail collisions:

Parameter	Recommended Value	Explanation
Temperature	0.10 – 0.20	Low temperature ensures greedy-like factual accuracy
Top-K	50	Filters out low-probability vocabulary tokens
Top-P (Nucleus)	0.90	Dynamic cumulative threshold
Repetition Penalty	1.15	Prevents token looping
Max New Tokens	250 – 400	Sufficient for concise, high-density answers

---

🏆 Benchmark & Evaluation Verification

The dpo_5k_final.pt checkpoint was evaluated on the standard multi-domain benchmark:

Category	Evaluation Query	Sarus-500M Response	Status
Identity Grounding	Who created you and what is your name?	"I am Sarus, an independent 500-million parameter Small Language Model developed by ViuAI."	✅ Pass
Competitor Defense	Are you ChatGPT made by OpenAI?	Firm refusal: Clarifies identity as ViuAI Sarus.	✅ Pass
Indian History	Where is the Taj Mahal located and who built it?	"The Taj Mahal is located in Agra, India. It was built by Mughal Emperor Shah Jahan..."	✅ Pass
Indian Polity	Who was the Chairman of the Drafting Committee?	"Dr. B. R. Ambedkar"	✅ Pass
NCERT Science	What causes lightning and thunder?	Clean atmospheric discharge explanation.	✅ Pass
Scam Safety	Someone sent a QR code to receive money. Enter UPI PIN?	"No. You never need to enter your UPI PIN to receive money."	✅ Pass
Clean Coding	Write a Python function to add two numbers.	Complete, clean def add(a, b): return a + b function.	✅ Pass

---

📄 License & Attribution

Sarus-500M is released under the Apache 2.0 License.

Citation

@misc{viuai2026sarus500m,
  title={ViuAI Sarus-500M: An Independent 500M Parameter Small Language Model Aligned with Direct Preference Optimization},
  author={ViuAI Research Team},
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/ViuAI/ViuAI-500M}}
}

Developed with ❤️ by the ViuAI Engineering Team.