Biu3010/dialect-normalizer

mBART-ViDia2Std

ViDia2Std Dataset | AAAI-26 Paper

This is a version of mBART-large-50 fine-tuned on the entire ViDia2Std corpus (13,657 pairs, all 63 Vietnamese provinces) plus data augmentation, to translate Vietnamese dialects and non-standard social media text into standard Vietnamese. It is released as the strongest version for real-world use, e.g. as a preprocessing step for downstream Vietnamese NLP tasks.

Warning — data leakage. This model was trained on all splits of ViDia2Std, including test. Do not use it as a baseline on the ViDia2Std test set; to reproduce the paper's baselines, train on the train split only.

Usage

import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model_path = "Biu3010/dialect-normalizer"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForSeq2SeqLM.from_pretrained(model_path).to(device)

def normalize_text(text):
    inputs = tokenizer(text, return_tensors="pt", max_length=128, truncation=True)
    input_ids = inputs.input_ids.to(device)
    attention_mask = inputs.attention_mask.to(device)

    with torch.no_grad():
        outputs = model.generate(
            input_ids=input_ids,
            attention_mask=attention_mask,
            max_length=128,
            num_beams=3,
            early_stopping=True,
        )

    return tokenizer.decode(outputs[0], skip_special_tokens=True)

dialect_text = "răng mà bựa ni trời nắng rứa hề"
print(f"Dialect : {dialect_text}")
print(f"Standard: {normalize_text(dialect_text)}")
# Expected output: "sao mà hôm nay trời nắng thế nhỉ"

If you encounter language-token issues, set tokenizer.src_lang = "vi_VN" and pass forced_bos_token_id=tokenizer.lang_code_to_id["vi_VN"] to model.generate().

Citation

@article{Anh_Ta_Van_Dinh_Nguyen_2026,
  title     = {ViDia2Std: A Parallel Corpus and Methods for Low-Resource Vietnamese Dialect-to-Standard Translation},
  author    = {Anh Ta, Khoa and Van Dinh, Nguyen and Nguyen, Kiet Van},
  journal   = {Proceedings of the AAAI Conference on Artificial Intelligence},
  volume    = {40},
  number    = {36},
  pages     = {29995--30004},
  year      = {2026},
  month     = {Mar.},
  url       = {https://ojs.aaai.org/index.php/AAAI/article/view/40247},
  doi       = {10.1609/aaai.v40i36.40247}
}