Vidyaapati MT: Hindi ↔ Bengali & Hindi ↔ Maithili (multilingual)
Multilingual machine translation model for Hindi ↔ Bengali & Hindi ↔ Maithili, built for the Vidyaapati project. It was fine-tuned on the Hindi–Bengali (about 1.8 million pairs) and Hindi–Maithili (about 1.26 million pairs) corpora together, in both directions, then converted to CTranslate2 for fast inference.
Other Vidyaapati models: helloboyn/vidyaapati-mt-bengali (bilingual, Hindi ↔ Bengali); helloboyn/vidyaapati-mt-maithili (bilingual, Hindi ↔ Maithili).
Model details
| Architecture | Transformer encoder–decoder, 18 + 18 layers, d_model 512, FFN 2048 |
| Parameters | 321M |
| Format | CTranslate2 (model.bin, float32 weights) |
| Tokenizer | SentencePiece (vocab/model.SRC, vocab/model.TGT) |
Supported directions
| Direction | Source tag | Target tag |
|---|---|---|
| Hindi → Bengali | hin_Deva |
ben_Beng |
| Bengali → Hindi | ben_Beng |
hin_Deva |
| Hindi → Maithili | hin_Deva |
mai_Deva |
| Maithili → Hindi | mai_Deva |
hin_Deva |
Usage
The model works internally in Devanagari script: the input must be normalized, tokenized and converted to Devanagari, and the output converted back to the target script. If you skip this preprocessing, Bengali input produces garbage and some sentences hallucinate. The helper below does the pre- and post-processing with pure-Python libraries, so it works on Windows, Linux and macOS.
pip install ctranslate2 sentencepiece indic-nlp-library sacremoses huggingface_hub
Save this as vidyaapati_mt.py:
import os
import ctranslate2
import sentencepiece as spm
from huggingface_hub import snapshot_download
from indicnlp.normalize.indic_normalize import IndicNormalizerFactory
from indicnlp.tokenize import indic_detokenize, indic_tokenize
from indicnlp.transliterate.unicode_transliterate import UnicodeIndicTransliterator
from sacremoses import MosesPunctNormalizer
# Model language tag -> Indic NLP language code (Maithili is processed as "hi")
ISO = {"hin_Deva": "hi", "ben_Beng": "bn", "mai_Deva": "hi"}
class Translator:
def __init__(self, model, device="cpu", compute_type="default", **kwargs):
path = model if os.path.isdir(model) else snapshot_download(model)
self.model = ctranslate2.Translator(path, device=device, compute_type=compute_type, **kwargs)
self.sp_src = spm.SentencePieceProcessor(model_file=os.path.join(path, "vocab", "model.SRC"))
self.sp_tgt = spm.SentencePieceProcessor(model_file=os.path.join(path, "vocab", "model.TGT"))
self.punct = MosesPunctNormalizer()
self.normalizers = IndicNormalizerFactory()
def _preprocess(self, text, lang):
iso = ISO[lang]
text = self.punct.normalize(text.strip())
text = self.normalizers.get_normalizer(iso).normalize(text)
text = " ".join(indic_tokenize.trivial_tokenize(text, iso))
# The model works in Devanagari: map every Indic script to Devanagari before tokenizing
return UnicodeIndicTransliterator.transliterate(text, iso, "hi").replace(" ् ", "्")
def _postprocess(self, tokens, lang):
iso = ISO[lang]
text = UnicodeIndicTransliterator.transliterate(self.sp_tgt.decode(tokens), "hi", iso)
return indic_detokenize.trivial_detokenize(text, iso)
def translate(self, sentences, src_lang, tgt_lang, beam_size=5, batch_size=32):
batch = [[src_lang, tgt_lang] + self.sp_src.encode(self._preprocess(s, src_lang), out_type=str)
for s in sentences]
results = self.model.translate_batch(batch, beam_size=beam_size, max_batch_size=batch_size,
max_decoding_length=256)
return [self._postprocess(r.hypotheses[0], tgt_lang) for r in results]
Then translate:
from vidyaapati_mt import Translator
# This repository is private: run `hf auth login` first, or set the HF_TOKEN environment variable.
tr = Translator("helloboyn/vidyaapati-mt-iitpatna") # or a local folder with the model files
# tr = Translator("helloboyn/vidyaapati-mt-iitpatna", device="cuda", compute_type="float16") # GPU
# tr = Translator("helloboyn/vidyaapati-mt-iitpatna", compute_type="int8") # faster on CPU, slightly lower quality
print(tr.translate(["तीसरे वर्ष बीज अवश्य बदल दिए जाएं।"], "hin_Deva", "ben_Beng"))
# ['তৃতীয় বছরে বীজ অবশ্যই পরিবর্তন করা উচিত।']
print(tr.translate(["तीसरे वर्ष बीज अवश्य बदल दिए जाएं।"], "hin_Deva", "mai_Deva"))
# ['तेसर वर्ष बीज अवश्य बदलि देल जाय।']
translate() takes a list of sentences and returns a list of translations. Pass the source and target language tags from the table above.
Evaluation
Scored on the first 500 sentences of each direction of the Vidyaapati dev set. None of these sentence pairs appear in the training data.
Settings: beam size 5, float32 on CPU, with the preprocessing shown above. Metrics are from sacreBLEU 2.6 (BLEU with default 13a tokenization, chrF++ with word_order=2).
| Direction | BLEU | chrF++ |
|---|---|---|
| Hindi → Bengali | 18.5 | 49.2 |
| Bengali → Hindi | 29.4 | 54.1 |
| Hindi → Maithili | 38.8 | 64.3 |
| Maithili → Hindi | 54.5 | 73.4 |
Limitations
- Supports only the directions listed above. Other language tags are in the vocabulary, but the model was not fine-tuned on them.
- The model was trained on single sentences. Split paragraphs into sentences before translating.
- Maithili uses the
mai_Devatag and is handled as Devanagari, like Hindi.
- Downloads last month
- -