| --- |
| license: cc-by-nc-4.0 |
| license_link: LICENSE |
| language: |
| - sa |
| library_name: transformers |
| pipeline_tag: image-text-to-text |
| base_model: Qwen/Qwen3.5-2B |
| base_model_relation: finetune |
| tags: |
| - ocr |
| - sanskrit |
| - devanagari |
| - document-ocr |
| - sansar |
| model-index: |
| - name: sansar-ocr-2b |
| results: |
| - task: |
| type: image-to-text |
| name: Sanskrit printed-page OCR |
| dataset: |
| type: sansar-bench-v1-pages |
| name: Sansar bench_v1 pages (54 archive.org book pages, 53 books) |
| metrics: |
| - type: cer |
| name: "median letter error rate, transformers (this repository's files)" |
| value: 1.05 |
| - type: cer |
| name: "median letter error rate, vLLM (the training run's evaluation)" |
| value: 1.06 |
| - type: cer |
| name: "mean letter error rate, transformers" |
| value: 2.31 |
| --- |
| |
| # Sansar OCR 2B |
|
|
| Reads printed Sanskrit book pages (Devanagari) and writes out their text, line by line. A 2.2B-parameter |
| vision-language model: [Qwen/Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B) fine-tuned on 19,377 |
| Sanskrit page images with their text. From the **Sansar** project at [Muse Mesh](https://muse-mesh.com) |
| ([Hugging Face](https://huggingface.co/MuseMesh), [Sansar Lab](https://saansar.com)): Sanskrit language models, their |
| corpus, their tokenizer, and the OCR that feeds the corpus. |
|
|
| - **This version: v0.1.0** = training run `ocrvlm_v12_clean_q35`, finished 2026-10-08. |
| - **Accuracy**: median letter error rate **1.05%** on 54 held-out pages from 53 books |
| (transformers, the files in this repository), 1.06% with vLLM. Google Cloud Vision makes |
| 1.69% on the same pages and the archive.org OCR of these scans |
| 9.03%. Better than Vision on 38 of 54 pages |
| (39 of 54 with vLLM). |
| - **Training labels**: clean e-texts aligned to scans, the text layers of born-digital textbooks and news bulletins, |
| and synthetic pages. **No output of a commercial AI service (Google Cloud Vision, Anthropic Claude, OpenAI models, |
| Sarvam) was used as a training label**; those services appear on this card only as comparison rows in the |
| evaluation. See Training data. |
|
|
| ## Quick start |
|
|
| ```bash |
| pip install "transformers>=5.18" torch torchvision pillow numpy accelerate |
| ``` |
|
|
| ```python |
| from huggingface_hub import hf_hub_download |
| import importlib.util |
| |
| # ocr_page.py (in this repository) prepares pages exactly as in training and runs greedy decoding |
| path = hf_hub_download("MuseMesh/sansar-ocr-2b", "ocr_page.py", revision="v0.1.0") |
| spec = importlib.util.spec_from_file_location("ocr_page", path); ocr = importlib.util.module_from_spec(spec) |
| spec.loader.exec_module(ocr) |
| |
| model, processor = ocr.load("MuseMesh/sansar-ocr-2b", revision="v0.1.0") # bfloat16 on a GPU, float32 on CPU |
| texts = ocr.transcribe(["page_001.jpg", "page_002.jpg"], model, processor, batch=4) |
| print(texts[0]) |
| ``` |
|
|
| Or from the command line: `python ocr_page.py --out_dir out/ pages/*.jpg` (one `.txt` per page). |
|
|
| What `ocr_page.py` does, if you want to call transformers directly: |
|
|
| 1. **Page image.** Crop near-white outer margins (a dark scanner border stays), then resize with the aspect ratio |
| kept so that width x height <= 1,800,000 pixels and both sides are multiples of 32. The model was trained on |
| exactly this; much larger or smaller pages are outside what it has seen. One page per request. |
| 2. **Prompt.** The chat template with one image and this text, thinking off (`enable_thinking=False`): |
|
|
| ```text |
| Transcribe all the text printed on this page exactly as printed, in reading order, line by line. Output only the text. |
| ``` |
| 3. **Decoding.** Greedy (`do_sample=False`), up to 3,072 new tokens, stop at `<|im_end|>` (248046) or |
| `<|endoftext|>` (248044). `generation_config.json` sets all of this. Batching needs `padding_side="left"`. |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForImageTextToText, AutoProcessor |
| |
| model = AutoModelForImageTextToText.from_pretrained("MuseMesh/sansar-ocr-2b", dtype=torch.bfloat16).to("cuda").eval() |
| processor = AutoProcessor.from_pretrained("MuseMesh/sansar-ocr-2b") |
| msgs = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": ocr.PROMPT}]}] |
| prompt = processor.apply_chat_template(msgs, tokenize=False, add_generation_prompt=True, enable_thinking=False) |
| image = ocr.prepare_page("page_001.jpg") # margins trimmed, <= 1.8 MP, sides multiple of 32 |
| x = processor(text=[prompt], images=[image], return_tensors="pt").to("cuda") |
| out = model.generate(**x, max_new_tokens=3072, do_sample=False) |
| print(processor.tokenizer.decode(out[0, x["input_ids"].shape[1]:], skip_special_tokens=True)) |
| ``` |
|
|
| **vLLM** (much faster for many pages): load the repository with |
| `LLM(model="MuseMesh/sansar-ocr-2b", limit_mm_per_prompt={"image": 1}, max_model_len=5672, dtype="bfloat16")`, send |
| `{"prompt": prompt, "multi_modal_data": {"image": image}}` with the same prompt and page preparation, and |
| `SamplingParams(temperature=0.0, max_tokens=3072, stop_token_ids=[248046, 248044])`. The vLLM numbers on this card |
| were measured on the training run's checkpoint with vLLM 0.30; the release files hold the same weights (see Files). |
|
|
| ## What the output looks like |
|
|
| Plain text, one printed line per output line, Devanagari as printed (NFC, no ZWJ/ZWNJ), `।` and `॥` as dandas, |
| digits as printed (Devanagari or Latin). On book pages it tends to leave out running heads, page numbers and |
| footnotes, because most of its training pages had those painted out (see Training data); do not rely on either |
| behaviour. Latin-script text (English introductions, IAST) was rare in training and is not evaluated. |
|
|
| ## Evaluation |
|
|
| **Bench.** `bench_v1_pages`: 54 pages from 53 archive.org scanned books (mixed print and scan quality), each |
| paired with the same passage from a clean digital edition of the text, cut to what is printed on that page and |
| checked page by page. No page of the bench books (nor any page sharing three 16-letter windows with a bench |
| reference) is in the training or validation data, and the bench was never used to pick a checkpoint. |
|
|
| **Metric.** Letter error rate: edit distance on Devanagari letters and signs only (consonants, vowels, vowel signs, |
| virama, anusvara, visarga, avagraha; no digits, dandas, Vedic accent marks, punctuation or spaces) divided by the reference's letter |
| count, with the reference located inside the output (text the reference does not cover, such as a running head the |
| model wrote out, is not counted against it). Median over the 54 pages, so one ruined page does not decide the score; |
| the mean is shown too. The same scorer and pages for every engine. |
|
|
| | engine | median letter CER | mean | bench tier "kept" (39) | tier "mid" (15) | hardest third of scans (18) | note | |
| |---|---:|---:|---:|---:|---:|---| |
| | Claude Opus 5.5 (Anthropic) | **0.65%** | 1.90% | 0.63% | 1.08% | 2.11% | general LLM, page by page; comparison only | |
| | **Sansar OCR 2B v0.1.0** (transformers) | **1.05%** | 2.31% | 0.78% | 1.31% | 3.14% | this repository's files and `ocr_page.py`, re-run for the release on an NVIDIA GeForce RTX 3060 | |
| | **Sansar OCR 2B v0.1.0** (vLLM) | **1.06%** | 2.27% | 0.81% | 1.40% | 3.12% | the training run's own evaluation, A100 | |
| | Claude Sonnet 5.5 (Anthropic) | **1.14%** | 2.17% | 1.10% | 1.31% | 3.12% | comparison only | |
| | Google Cloud Vision | **1.69%** | 3.89% | 1.50% | 3.15% | 3.55% | DOCUMENT_TEXT_DETECTION, 2026-10; comparison only | |
| | Codex gpt-6-astra (OpenAI) | **2.65%** | 3.92% | 2.18% | 3.55% | 6.66% | comparison only | |
| | archive.org OCR of the same scans | 9.03% | 9.64% | | | | the text most of these books are online with today | |
| | Qwen3.5-2B, no fine-tuning | **18.54%** | 28.89% | 17.04% | 21.88% | 27.19% | the base model, same prompt | |
|
|
| The Vision, Claude and Codex rows are those services' transcriptions of the same 54 page images (2026-09/10), |
| scored by the same scorer, shown for comparison only; none of their output was used to train this model. "kept"/"mid" |
| are the bench's own page-quality tiers; "hardest third" are the 18 pages where the archive.org OCR did worst (a proxy |
| for scan quality). |
|
|
| - **The release check.** The model in this repository (repackaged: see Files) was re-run on all 54 pages with |
| transformers and `ocr_page.py` for this release: median 1.05%, mean 2.31% |
| (NVIDIA GeForce RTX 3060, batch 4). The training run's own transformers cross-check of the same weights gave |
| 1.06%; its vLLM run 1.06%. Different engines and GPUs change a |
| few characters on some pages (bfloat16 kernels). Everything is in `eval/`. |
| - **Small bench.** With 54 pages a median difference of a few tenths of a point has a wide interval; the page-by-page |
| count (better than Vision on 39 of 54 pages, vLLM) is the sturdier evidence. |
| - **The worst pages are bad for every engine** (scan or reference problems, not this model's). |
| - **Validation.** On the 711-page validation set (31 books, held out by book; the same kinds of sources as |
| the training data) the median is 0.25% and the mean 0.93%. It is easier than the bench and not |
| comparable to it. |
| - **Overall CER** including punctuation, digits and spacing is higher: median 3.50% on the bench |
| (Vision 4.56%). |
|
|
| ### Speed |
|
|
| | GPU | engine | pages per hour (generation only) | |
| |---|---|---:| |
| | NVIDIA A100-SXM4-40GB (GCP spot) | vLLM, batch 64 | 9,216 (54 test pages) / 15,104 (711 validation pages) | |
| | NVIDIA GeForce RTX 3060 | transformers `generate`, batch 4 | 244 (54 test pages) | |
|
|
| About 847-1,035 output tokens per page (vLLM). Start-up (model load, vLLM compile) is not included. |
|
|
| ## Training |
|
|
| | | | |
| |---|---| |
| | base | Qwen/Qwen3.5-2B (revision `15852e8`), Apache-2.0: hybrid Gated-DeltaNet/attention language model with its vision encoder | |
| | method | full fine-tune of every weight (vision encoder at a lower learning rate), loss on the transcription only | |
| | steps | 2,000 x 16 pages = 32,000 page samples, 80.7M tokens seen | |
| | optimiser | 8-bit AdamW (bitsandbytes), lr 1.5e-5 (vision encoder 5e-6), 3% warmup, bfloat16 autocast over float32 master weights, gradient checkpointing | |
| | pages | the prompt above + page image (<= 1.8 MP) + label + `<|im_end|>`; at most 6,144 tokens | |
| | mixture | sampling weights per source (below); the last 15% of samples drawn from the gold sources (twin, textbooks, news) only | |
| | checkpoint | picked by loss on a source-balanced gold validation subset (276 pages, <= 120 per source), never on the test bench: step 2,000 (val loss 0.0621; 0.1577 at step 200) | |
| | hardware | 1x NVIDIA A100-SXM4-40GB, Google Cloud spot (a2-highgpu-1g, us-central1-a) | |
| | time and cost | 7.6 h of VM time for training and the evaluations (1 spot stop; training resumed from its last checkpoint), about $9.9 of spot time | |
|
|
| `training/summary.json` holds every argument and the validation-loss curve; `training/mixture.json` the sample counts; |
| `training/label_stats.json` the label-set counts and exactly what was excluded. |
|
|
| ## Training data |
|
|
| 19,377 training pages from 1,057 books, split by book from 711 validation pages (31 books) |
| and the 54 test pages. Page images and labels are not redistributed here: most of the scans and texts carry no open |
| licence. Where each label comes from: |
|
|
| | source | train pages | weight | samples drawn | label | |
| |---|---:|---:|---:|---| |
| | archive.org scans + a clean e-text of the same work ("twin") | 11,074 | 1.5 | 27,467 | a clean digital edition of the work (Sanskrit Wikisource proofread pages, GRETIL, Muktabodha, sanskritdocuments.org, Ambuda, sanskritsahitya.org, snskrt) aligned to the printed page and cut at its printed line breaks; running heads, page numbers and footnotes are painted out of the image | |
| | born-digital textbooks (NCERT, Kerala SCERT) | 3,367 | 0.5 | 2,808 | the PDF's own text layer, repaired by rule (split words, doubled signs, reph order); ~1 split word in 100 remains | |
| | All India Radio Sanskrit news bulletins (2023-2026) | 3,839 | 0.2 | 1,270 | the PDF text decoded from its fonts (the fonts' own glyph tables, and a vocabulary-based decipherment of unmapped glyphs); no OCR; ~1-1.5% character errors expected | |
| | synthetic pages | 1,097 | 0.3 | 455 | paragraphs from curated open Sanskrit e-text corpora (GRETIL, DCS, SARIT, Muktabodha, DharmaNexus, BORI Mahābhārata, ...) typeset in 1-2 columns with real fonts and degraded (skew, blur, noise, bleed); exact labels | |
|
|
| **No commercial AI service output is training text.** Labels come only from clean e-texts aligned to scans, the |
| text layers of born-digital textbooks and All India Radio bulletins, and synthetic pages typeset from curated |
| e-text corpora. Excluded on purpose: page transcriptions made with Google Cloud Vision, Anthropic Claude and OpenAI |
| Codex (an earlier, internal label set had used them), and synthetic pages typesetting paragraphs from Sarvam's |
| Vagartha dataset. Where those services were used at all, it was as a check, never as label text: the news-bulletin |
| decoding was validated against Vision on a sample of pages, and the comparison rows in Evaluation are their |
| transcriptions of the test pages. archive.org's own OCR is used only to locate the printed lines on twin pages. |
| Wikisource texts are human-proofread transcriptions by Wikisource volunteers. |
|
|
| All labels are NFC with ZWJ/ZWNJ removed, `|` as `।`, and a colon typed after a letter as visarga. Every training |
| and validation page that shares three or more 16-letter windows with a test reference was dropped. |
|
|
| ## Limitations |
|
|
| - **Printed Devanagari Sanskrit only.** Not trained on manuscripts, handwriting, palm leaves, other scripts or |
| Hindi/Marathi pages (some Hindi in the textbooks and news, but it was not evaluated). |
| - **Reading order** on two-column pages, glossaries, tables and picture captions can be wrong; on scans of two-page |
| spreads it can read across. (Fewer such pages were in training than in our earlier internal model, which also |
| learnt from Vision's text of glossary and table pages.) |
| - **Loops and runaways.** On a hard page it can repeat a line until the 3,072-token cap. Check for repeated lines. |
| - **Page furniture** (running heads, page numbers, footnotes) is sometimes left out and sometimes written out. |
| - **Not scored**: digits, punctuation, spacing and Vedic accents are outside the letter metric; the overall error |
| rate including them is 3.50% (median). |
| - **It can invent text** on blank pages, figures or very faint scans, as any generative OCR can. It also inherits |
| the base model's general behaviour on anything that is not a Sanskrit page. |
| - **Labels are imperfect**: the twin labels follow another edition of the same work (a few percent of variant |
| readings and verse numbers), the textbook layers split words, the news decoding has rare glyph errors. The model |
| learnt some of that. |
|
|
| ## Versions |
|
|
| Each version is a git tag; `main` is the newest. Pin one with `revision="v0.1.0"`. |
|
|
| | version | date | training run | bench median (transformers / vLLM) | Google Vision | |
| |---|---|---|---:|---:| |
| | v0.1.0 | 2026-10-08 | `ocrvlm_v12_clean_q35` | 1.05% / 1.06% | 1.69% | |
|
|
| ## Files |
|
|
| | file | what | |
| |---|---| |
| | `model.safetensors` | the weights, bfloat16 (the gated-DeltaNet decay and norm parameters in float32 as in the base release); the output head tied to the input embedding | |
| | `config.json`, `generation_config.json` | architecture; greedy decoding with both stop tokens | |
| | `tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja`, `processor_config.json` | the base model's tokenizer and processor, unchanged | |
| | `ocr_page.py` | page preparation, loading and batch transcription (the code above) | |
| | `eval/bench_v1_scores.json` | per-page and per-group scores of every engine in the table | |
| | `eval/verification.json`, `eval/verification_pages.json` | the release check: this repository's files re-run on the 54 pages | |
| | `eval/transcriptions/` | the training run's own outputs for the 54 pages (vLLM and transformers) | |
| | `eval/*_scores.json`, `eval/*_timing.json` | the training run's evaluation logs (test, validation) | |
| | `training/summary.json`, `training/mixture.json`, `training/cloud.json`, `training/label_stats.json` | arguments, validation-loss curve, sample counts, machine time, label-set counts | |
| | `LICENSE`, `LICENSE-CODE`, `LICENSE-QWEN`, `NOTICE`, `CHANGELOG.md` | licences, base-model attribution, version history | |
|
|
| Repackaging against the training checkpoint: the training run saved the tied output head as a second copy of the |
| input embedding (bit-identical; dropped); `config.json` records `bfloat16` and `use_cache: true` instead of the |
| training-time `float32`/`false`; `generation_config.json` adds the `<|im_end|>` stop token and greedy decoding. No |
| weight changed. |
|
|
| ## Licence |
|
|
| This release is for research and non-commercial use. The training labels come partly from sources without an open |
| licence (textbooks, news bulletins, scanned books), used here for research. |
|
|
| - Weights (`model.safetensors`): **CC BY-NC 4.0** (`LICENSE`). Non-commercial use (research, teaching, non-profit |
| work) with attribution to "Sansar, Muse Mesh Private Limited". |
| - Base model: Qwen3.5-2B by Alibaba Cloud, **Apache-2.0** (`LICENSE-QWEN`, `NOTICE`). The tokenizer and processor |
| files are the base model's and stay under Apache-2.0. |
| - Code (`ocr_page.py`): **Apache-2.0** (`LICENSE-CODE`). |
| - Commercial use: contact kushal@muse-mesh.com. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{sansar_ocr_2b_2026, |
| title = {Sansar OCR 2B: printed Sanskrit page OCR}, |
| author = {Muse Mesh}, |
| year = {2026}, |
| note = {v0.1.0, fine-tuned from Qwen3.5-2B}, |
| url = {https://huggingface.co/MuseMesh/sansar-ocr-2b} |
| } |
| ``` |
|
|
| Contact: kushal@muse-mesh.com |
|
|
