README.md

---
title: README
emoji: 🪷
colorFrom: yellow
colorTo: red
sdk: static
pinned: false
---

# Muse Mesh

We build **Sanskrit-first language models, in the open**: corpora, tokenizers and models trained from scratch, starting with Sanskrit. Every training run we do, finished or live, with its score and cost, is published at **[saansar.com](https://saansar.com)**.

## Sansar: Sanskrit-first language models

**Sansar** is a family of causal language models trained from scratch, Sanskrit first. The models released so far are trained on Sanskrit text only, with a Sanskrit tokenizer of their own; none is a fine-tune of an English model. Other languages come later on the same recipe.

| Model | Parameters | Bits per byte (held-out, ex-Gītā) ↓ |
|---|---|---|
| [sansar-700m](https://huggingface.co/MuseMesh/sansar-700m) (v0.2.0) | 704M | **0.5262** |
| [sansar-350m](https://huggingface.co/MuseMesh/sansar-350m) | 318M | 0.5547 |
| [sansar-125m](https://huggingface.co/MuseMesh/sansar-125m) | 97M | 0.6039 |
| [sansar-60m](https://huggingface.co/MuseMesh/sansar-60m) | 63M | 0.6434 |
| [sansar-20m](https://huggingface.co/MuseMesh/sansar-20m) | 27M | 0.7177 |

Scores are measured on frozen held-out Sanskrit sets: DCS gold sentences, prose, Vedic and out-of-domain texts. Held-out text is masked out of training. Every model card gives the per-set numbers, the training data and a version history.

Scored the same way, general base models need more bits per byte: Gemma 3 4B 0.6965, Qwen3-4B 0.7071, Llama 3.2 3B 0.7122, Sarvam-1 0.7465. sansar-700m beats all four on every held-out set with 704M parameters (Krutrim-2 12B, scored earlier in a pass that is not strictly comparable, is not in this list). A live demo is at **[saansar.com/demo](https://saansar.com/demo)**.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("MuseMesh/sansar-700m", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("MuseMesh/sansar-700m", trust_remote_code=True)
ids = tok("संस्कृतं नाम दैवी वाक्", return_tensors="pt")
print(tok.decode(model.generate(**ids, max_new_tokens=60, do_sample=True, top_k=100)[0]))
```

## Sansar OCR

**[sansar-ocr-2b](https://huggingface.co/MuseMesh/sansar-ocr-2b)** reads printed Sanskrit (Devanagari) pages: a 2B vision-language model fine-tuned from Qwen3.5-2B on clean labels only. Median letter error on 54 held-out pages: **1.05%**, against 1.69% for Google Cloud Vision and 9.03% for archive.org's OCR. It runs on a single consumer GPU.

## English and Math models

Small models trained from scratch with the same recipe (Muon, rotary positions, QK-norm, squared ReLU) and a shared 32k tokenizer, released with their evaluation sets and the exact list of training documents. [Collection](https://huggingface.co/collections/MuseMesh/mume-english-and-math-language-models-6ac4f187f33abd6c9a9cdde6).

| Model | Parameters | Trained on | Result |
|---|---|---|---|
| [mume-english-125m](https://huggingface.co/MuseMesh/mume-english-125m) | 134M | 1.33B FineWeb tokens | 1.106 bits per byte on frozen FineWeb val; on the validation stream 1.132 vs 1.176 for a plain-AdamW GPT-2 124M at the same tokens (3.7% lower) |
| [mume-math-125m](https://huggingface.co/MuseMesh/mume-math-125m) | 134M | 1.33B OpenWebMath tokens | MATH test (clean) 62.6% fewer bits than the English model; branch `sft` is a GSM8K/MATH fine-tune |

- **[mume-tokenizer-32k](https://huggingface.co/MuseMesh/mume-tokenizer-32k)**: the shared 32k unigram tokenizer.
- **[mume-eval-suites](https://huggingface.co/datasets/MuseMesh/mume-eval-suites)**: the frozen test sets behind every number (FineWeb val, WikiText-103, enwik8, text8, GSM8K, MATH, OpenWebMath val) with contamination-clean variants, plus training manifests.

## Data and tokenizer

- **[Sansar Sanskrit Corpus](https://huggingface.co/datasets/MuseMesh/sansar-sanskrit-corpus)**: 252M words of openly licensed Sanskrit from 21 sources, in Devanagari. It is split into one config per licence (CC BY, CC BY-SA, CC0, ODC-By, Apache-2.0, MIT), and every record carries its source, licence and attribution. Text overlapping our evaluation sets is removed.
- **[sansar-sanskrit-tokenizer](https://huggingface.co/MuseMesh/sansar-sanskrit-tokenizer)**: an 8k unigram SentencePiece model over SLP1 transliteration with a Devanagari wrapper. 2.6 to 3.9 tokens per Sanskrit word with only 8k entries.

## Licences

The corpus records keep their source licences. Model weights and the tokenizer are released under **CC BY-NC 4.0** for non-commercial research use, and the modelling code under **Apache-2.0**. For other uses, contact us.

**Links:** [muse-mesh.com](https://muse-mesh.com) · [GitHub](https://github.com/muse-mesh) · kushal@muse-mesh.com