You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

Vidyaapati MT: Hindi ↔ Bengali & Hindi ↔ Maithili (multilingual)

Multilingual machine translation model for Hindi ↔ Bengali & Hindi ↔ Maithili, built for the Vidyaapati project. It was fine-tuned on the Hindi–Bengali (about 1.8 million pairs) and Hindi–Maithili (about 1.26 million pairs) corpora together, in both directions, then converted to CTranslate2 for fast inference.

Other Vidyaapati models: helloboyn/vidyaapati-mt-bengali (bilingual, Hindi ↔ Bengali); helloboyn/vidyaapati-mt-maithili (bilingual, Hindi ↔ Maithili).

Model details

Architecture Transformer encoder–decoder, 18 + 18 layers, d_model 512, FFN 2048
Parameters 321M
Format CTranslate2 (model.bin, float32 weights)
Tokenizer SentencePiece (vocab/model.SRC, vocab/model.TGT)

Supported directions

Direction Source tag Target tag
Hindi → Bengali hin_Deva ben_Beng
Bengali → Hindi ben_Beng hin_Deva
Hindi → Maithili hin_Deva mai_Deva
Maithili → Hindi mai_Deva hin_Deva

Usage

The model works internally in Devanagari script: the input must be normalized, tokenized and converted to Devanagari, and the output converted back to the target script. If you skip this preprocessing, Bengali input produces garbage and some sentences hallucinate. The helper below does the pre- and post-processing with pure-Python libraries, so it works on Windows, Linux and macOS.

pip install ctranslate2 sentencepiece indic-nlp-library sacremoses huggingface_hub

Save this as vidyaapati_mt.py:

import os

import ctranslate2
import sentencepiece as spm
from huggingface_hub import snapshot_download
from indicnlp.normalize.indic_normalize import IndicNormalizerFactory
from indicnlp.tokenize import indic_detokenize, indic_tokenize
from indicnlp.transliterate.unicode_transliterate import UnicodeIndicTransliterator
from sacremoses import MosesPunctNormalizer

# Model language tag -> Indic NLP language code (Maithili is processed as "hi")
ISO = {"hin_Deva": "hi", "ben_Beng": "bn", "mai_Deva": "hi"}


class Translator:
    def __init__(self, model, device="cpu", compute_type="default", **kwargs):
        path = model if os.path.isdir(model) else snapshot_download(model)
        self.model = ctranslate2.Translator(path, device=device, compute_type=compute_type, **kwargs)
        self.sp_src = spm.SentencePieceProcessor(model_file=os.path.join(path, "vocab", "model.SRC"))
        self.sp_tgt = spm.SentencePieceProcessor(model_file=os.path.join(path, "vocab", "model.TGT"))
        self.punct = MosesPunctNormalizer()
        self.normalizers = IndicNormalizerFactory()

    def _preprocess(self, text, lang):
        iso = ISO[lang]
        text = self.punct.normalize(text.strip())
        text = self.normalizers.get_normalizer(iso).normalize(text)
        text = " ".join(indic_tokenize.trivial_tokenize(text, iso))
        # The model works in Devanagari: map every Indic script to Devanagari before tokenizing
        return UnicodeIndicTransliterator.transliterate(text, iso, "hi").replace(" ् ", "्")

    def _postprocess(self, tokens, lang):
        iso = ISO[lang]
        text = UnicodeIndicTransliterator.transliterate(self.sp_tgt.decode(tokens), "hi", iso)
        return indic_detokenize.trivial_detokenize(text, iso)

    def translate(self, sentences, src_lang, tgt_lang, beam_size=5, batch_size=32):
        batch = [[src_lang, tgt_lang] + self.sp_src.encode(self._preprocess(s, src_lang), out_type=str)
                 for s in sentences]
        results = self.model.translate_batch(batch, beam_size=beam_size, max_batch_size=batch_size,
                                             max_decoding_length=256)
        return [self._postprocess(r.hypotheses[0], tgt_lang) for r in results]

Then translate:

from vidyaapati_mt import Translator

# This repository is private: run `hf auth login` first, or set the HF_TOKEN environment variable.
tr = Translator("helloboyn/vidyaapati-mt-iitpatna")              # or a local folder with the model files
# tr = Translator("helloboyn/vidyaapati-mt-iitpatna", device="cuda", compute_type="float16")  # GPU
# tr = Translator("helloboyn/vidyaapati-mt-iitpatna", compute_type="int8")                    # faster on CPU, slightly lower quality

print(tr.translate(["तीसरे वर्ष बीज अवश्य बदल दिए जाएं।"], "hin_Deva", "ben_Beng"))
# ['তৃতীয় বছরে বীজ অবশ্যই পরিবর্তন করা উচিত।']
print(tr.translate(["तीसरे वर्ष बीज अवश्य बदल दिए जाएं।"], "hin_Deva", "mai_Deva"))
# ['तेसर वर्ष बीज अवश्य बदलि देल जाय।']

translate() takes a list of sentences and returns a list of translations. Pass the source and target language tags from the table above.

Evaluation

Scored on the first 500 sentences of each direction of the Vidyaapati dev set. None of these sentence pairs appear in the training data. Settings: beam size 5, float32 on CPU, with the preprocessing shown above. Metrics are from sacreBLEU 2.6 (BLEU with default 13a tokenization, chrF++ with word_order=2).

Direction BLEU chrF++
Hindi → Bengali 18.5 49.2
Bengali → Hindi 29.4 54.1
Hindi → Maithili 38.8 64.3
Maithili → Hindi 54.5 73.4

Limitations

  • Supports only the directions listed above. Other language tags are in the vocabulary, but the model was not fine-tuned on them.
  • The model was trained on single sentences. Split paragraphs into sentences before translating.
  • Maithili uses the mai_Deva tag and is handled as Devanagari, like Hindi.
Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support