README.md

---
license: cc-by-nc-4.0
language:
- vi
pipeline_tag: translation
tags:
- dialect-normalization
- vietnamese-dialect
- mbart
- social-media
- low-resource
datasets:
- Biu3010/ViDia2Std
base_model: facebook/mbart-large-50
library_name: transformers
---

# mBART-ViDia2Std

<p align="center">
  <a href="https://huggingface.co/datasets/Biu3010/ViDia2Std">ViDia2Std Dataset</a> |
  <a href="https://ojs.aaai.org/index.php/AAAI/article/view/40247">AAAI-26 Paper</a>
</p>

This is a version of [mBART-large-50](https://huggingface.co/facebook/mbart-large-50) fine-tuned on the entire [ViDia2Std](https://huggingface.co/datasets/Biu3010/ViDia2Std) corpus (13,657 pairs, all 63 Vietnamese provinces) plus data augmentation, to translate Vietnamese dialects and non-standard social media text into standard Vietnamese. It is released as the strongest version for real-world use, e.g. as a preprocessing step for downstream Vietnamese NLP tasks.

> **Warning — data leakage.** This model was trained on **all splits of ViDia2Std, including `test`**. Do not use it as a baseline on the ViDia2Std test set; to reproduce the paper's baselines, train on the `train` split only.

## Usage

```python
import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model_path = "Biu3010/dialect-normalizer"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForSeq2SeqLM.from_pretrained(model_path).to(device)

def normalize_text(text):
    inputs = tokenizer(text, return_tensors="pt", max_length=128, truncation=True)
    input_ids = inputs.input_ids.to(device)
    attention_mask = inputs.attention_mask.to(device)

    with torch.no_grad():
        outputs = model.generate(
            input_ids=input_ids,
            attention_mask=attention_mask,
            max_length=128,
            num_beams=3,
            early_stopping=True,
        )

    return tokenizer.decode(outputs[0], skip_special_tokens=True)

dialect_text = "răng mà bựa ni trời nắng rứa hề"
print(f"Dialect : {dialect_text}")
print(f"Standard: {normalize_text(dialect_text)}")
# Expected output: "sao mà hôm nay trời nắng thế nhỉ"
```

If you encounter language-token issues, set `tokenizer.src_lang = "vi_VN"` and pass `forced_bos_token_id=tokenizer.lang_code_to_id["vi_VN"]` to `model.generate()`.

## Citation

```bibtex
@article{Anh_Ta_Van_Dinh_Nguyen_2026,
  title     = {ViDia2Std: A Parallel Corpus and Methods for Low-Resource Vietnamese Dialect-to-Standard Translation},
  author    = {Anh Ta, Khoa and Van Dinh, Nguyen and Nguyen, Kiet Van},
  journal   = {Proceedings of the AAAI Conference on Artificial Intelligence},
  volume    = {40},
  number    = {36},
  pages     = {29995--30004},
  year      = {2026},
  month     = {Mar.},
  url       = {https://ojs.aaai.org/index.php/AAAI/article/view/40247},
  doi       = {10.1609/aaai.v40i36.40247}
}
```