bartpho-teencode-vilexnorm

This model is designed to normalize and correct Vietnamese "teencode" into standard Vietnamese text. It is a sequence-to-sequence (Seq2Seq) model based on the BART architecture.

Disclaimer: This is an archived project from about a year ago. The exact hyper-parameters and training configurations are not fully documented. Based on the naming conventions, it is believed to be a fine-tuned version of BARTpho on the ViLexNorm dataset.

Model Details

• Task: Text-to-Text Generation / Text Normalization (Teencode Correction)
• Language: Vietnamese
• Format: safetensors
• License: CC BY-NC-SA 4.0 (Attribution-NonCommercial-ShareAlike 4.0 International)

How to Use

You can easily load and test this model using the transformers library.

Installation

pip install transformers torch sentencepiece safetensors

Inference Script

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import torch

# Load the model and tokenizer from Hugging Face
model_id = "duckling2211/bartpho-teencode-vilexnorm"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

model.eval()

def correct_teencode(text):
    # Tokenize input text
    inputs = tokenizer(text, return_tensors="pt", max_length=256, truncation=True)
    
    # Generate the normalized text
    with torch.no_grad():
        outputs = model.generate(
            **inputs, 
            max_length=256, 
            num_beams=4, # Beam search for better quality
            early_stopping=True
        )
    
    # Decode and return the result
    return tokenizer.decode(outputs[0], skip_special_tokens=True)

# Example usage
test_cases = [
    "hôm nay chời đệp qá",
    "mjk đag lm j dza",
    "k hỉu lun á tr"
]

print("Running tests:")
for test in test_cases:
    print(f"Input:  {test}")
    print(f"Output: {correct_teencode(test)}")
    print("-" * 30)

Example Output

Running tests:
Input:  hôm nay chời đệp qá
Output: hôm nay trời đệp quá
------------------------------
Input:  mjk đag lm j dza
Output: mình đang làm gì vậy
------------------------------
Input:  k hỉu lun á tr
Output: không hiểu luôn á trời
------------------------------

Acknowledgments & References

This model relies on the foundational work and datasets provided by the following researchers. If you use this model, please ensure you respect the CC BY-NC-SA 4.0 license and cite their original works:

Base Model: BARTpho

• Paper: BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese (Nguyen Luong Tran, Duong Minh Le, Dat Quoc Nguyen)
• Repository: vinairesearch/bartpho

Dataset: ViLexNorm

• Paper: ViLexNorm: A Lexical Normalization Corpus for Vietnamese Social Media Text (Thanh-Nhi Nguyen, Thanh-Phong Le, Kiet Van Nguyen)
• Repository: ngxtnhi/vilexnorm