---
base_model: krutrim-ai-labs/Chitrapathak-2
library_name: peft
pipeline_tag: image-text-to-text
tags:
- lora
- ocr
- marathi
- gujarati
- chitrapathak
language:
- mr
- gu
---

# Chitrapathak MarGuj

LoRA for Marathi and Gujarati line OCR on [krutrim-ai-labs/Chitrapathak-2](https://huggingface.co/krutrim-ai-labs/Chitrapathak-2).

The same adapter was trained in two steps. It was first trained on the Marathi real-10k set (`chitrapathak-marathi-v7-lora`), then continued for one epoch on 9,200 Gujarati gazette lines. The Gujarati run did not start a new LoRA. Base weights are not in this repo.

## Training

| | Marathi step | Gujarati step |
|---|---|---|
| Starting weights | Chitrapathak-2 | this adapter, after the Marathi step |
| Lines | 9,200 real Marathi lines | 9,200 real Gujarati lines |
| Epochs | 2 | 1 |
| Batch | 1, gradient accumulation 16 | 1, gradient accumulation 8 |
| Learning rate | 1e-4 | 1e-4 |
| LoRA | r=32, alpha=64, language-model projections only | same adapter, continued |

Prompt: `Perform OCR on this image and transcribe all visible text exactly as it appears.`

## Gujarati frozen 100

Same 100 long lines (`special_chars_expansion.jsonl`) and the same Gujarati metric definitions. PARSeq runs are piecewise. Chitrapathak and IndicOCR runs are whole-line.

| Metric | PARSeq base | PARSeq trained | Chitrapathak base | Chitrapathak MarGuj | IndicOCR base | IndicOCR MarGuj |
|---|---:|---:|---:|---:|---:|---:|
| CER ↓ | 31.57% | 27.45% | 80.35% | 8.81% | 1.76% | **1.23%** |
| Strict WER ↓ | 97.30% | 99.44% | 93.10% | 32.25% | 8.57% | **7.36%** |
| Punctuation F1 ↑ | 0.00% | 36.66% | 90.44% | 97.30% | 97.30% | **98.60%** |
| Matra ↑ | 84.01% | 91.87% | 0.00% | 92.10% | **98.53%** | **98.53%** |
| Conjunct ↑ | 78.70% | 89.51% | 0.00% | 64.20% | 97.22% | **97.53%** |
| Numeral ↑ | 54.60% | 82.18% | 0.00% | 92.53% | **96.55%** | 95.98% |
| Character accuracy ↑ | 68.43% | 72.55% | 19.65% | 91.19% | 98.24% | **98.77%** |
| Exact match ↑ | 0/100 | 0/100 | 0/100 | 5/100 | 43/100 | **46/100** |

## Gujarati 800-line holdout

Short gazette lines, at most 10 words. This is the split held out from Gujarati training. Chitrapathak MarGuj: CER 0.30%, strict WER 1.46%, exact match 747/800.

## Use

```python
import torch
from peft import PeftModel
from transformers import AutoProcessor, AutoModelForImageTextToText

base_id = "krutrim-ai-labs/Chitrapathak-2"
adapter_id = "snxtyle/chitrapathak-MarGuj"
device = "cuda" if torch.cuda.is_available() else "mps"
dtype = torch.bfloat16

model = AutoModelForImageTextToText.from_pretrained(
    base_id, dtype=dtype, attn_implementation="sdpa"
)
model = PeftModel.from_pretrained(model, adapter_id).to(device).eval()
processor = AutoProcessor.from_pretrained(base_id, max_pixels=300000, min_pixels=56 * 56)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": [
        {"type": "image", "image": image},
        {"type": "text", "text": "Perform OCR on this image and transcribe all visible text exactly as it appears."},
    ]},
]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], padding=True, return_tensors="pt").to(device)
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
gen = [o[len(i):] for i, o in zip(inputs.input_ids, out)]
print(processor.batch_decode(gen, skip_special_tokens=True)[0].strip())
```