File size: 2,955 Bytes
ed68dc0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 | ---
license: cc-by-nc-4.0
language:
- vi
pipeline_tag: translation
tags:
- dialect-normalization
- vietnamese-dialect
- mbart
- social-media
- low-resource
datasets:
- Biu3010/ViDia2Std
base_model: facebook/mbart-large-50
library_name: transformers
---
# mBART-ViDia2Std
<p align="center">
<a href="https://huggingface.co/datasets/Biu3010/ViDia2Std">ViDia2Std Dataset</a> |
<a href="https://ojs.aaai.org/index.php/AAAI/article/view/40247">AAAI-26 Paper</a>
</p>
This is a version of [mBART-large-50](https://huggingface.co/facebook/mbart-large-50) fine-tuned on the entire [ViDia2Std](https://huggingface.co/datasets/Biu3010/ViDia2Std) corpus (13,657 pairs, all 63 Vietnamese provinces) plus data augmentation, to translate Vietnamese dialects and non-standard social media text into standard Vietnamese. It is released as the strongest version for real-world use, e.g. as a preprocessing step for downstream Vietnamese NLP tasks.
> **Warning — data leakage.** This model was trained on **all splits of ViDia2Std, including `test`**. Do not use it as a baseline on the ViDia2Std test set; to reproduce the paper's baselines, train on the `train` split only.
## Usage
```python
import torch
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model_path = "Biu3010/dialect-normalizer"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForSeq2SeqLM.from_pretrained(model_path).to(device)
def normalize_text(text):
inputs = tokenizer(text, return_tensors="pt", max_length=128, truncation=True)
input_ids = inputs.input_ids.to(device)
attention_mask = inputs.attention_mask.to(device)
with torch.no_grad():
outputs = model.generate(
input_ids=input_ids,
attention_mask=attention_mask,
max_length=128,
num_beams=3,
early_stopping=True,
)
return tokenizer.decode(outputs[0], skip_special_tokens=True)
dialect_text = "răng mà bựa ni trời nắng rứa hề"
print(f"Dialect : {dialect_text}")
print(f"Standard: {normalize_text(dialect_text)}")
# Expected output: "sao mà hôm nay trời nắng thế nhỉ"
```
If you encounter language-token issues, set `tokenizer.src_lang = "vi_VN"` and pass `forced_bos_token_id=tokenizer.lang_code_to_id["vi_VN"]` to `model.generate()`.
## Citation
```bibtex
@article{Anh_Ta_Van_Dinh_Nguyen_2026,
title = {ViDia2Std: A Parallel Corpus and Methods for Low-Resource Vietnamese Dialect-to-Standard Translation},
author = {Anh Ta, Khoa and Van Dinh, Nguyen and Nguyen, Kiet Van},
journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
volume = {40},
number = {36},
pages = {29995--30004},
year = {2026},
month = {Mar.},
url = {https://ojs.aaai.org/index.php/AAAI/article/view/40247},
doi = {10.1609/aaai.v40i36.40247}
}
```
|