AfriSpeech/africa-corpus
Viewer • Updated • 16.6M • 2.4k • 3
Afro-Awesome-Align (AfriSpeech/afro-awesome-align) is a state-of-the-art neural word and phrase aligner fine-tuned across 693 African languages.
It is built upon the awesome-align framework (Dou & Neubig, EACL 2021) and fine-tuned from bert-base-multilingual-cased using a multi-task objective:
pip install awesome-align transformers torch
Create a text file where each line has the format source ||| target:
the holy spirit gives peace ||| honhom kronkron no ma asomdwoe
water of life flows freely ||| maji ya uzima yanatiririka bure
eternal life is a gift from God ||| ìyè àìnípẹ̀kun jẹ́ ẹ̀bùn láti ọ̀dọ̀ Ọlọ́run
Run alignment extraction:
awesome-align --model_name_or_path AfriSpeech/afro-awesome-align --data_file pairs.txt --output_file alignments.align --extraction softmax --align_layer 8 --batch_size 64
import torch
from transformers import AutoModel, AutoTokenizer
model_name = "AfriSpeech/afro-awesome-align"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name).cuda().eval()
src = "the holy spirit gives peace"
tgt = "honhom kronkron no ma asomdwoe"
inputs = tokenizer(src, tgt, return_tensors="pt").to("cuda")
with torch.no_grad():
outputs = model(**inputs, output_hidden_states=True)
# Hidden representations at layer 8 provide optimal cross-lingual alignment vectors
layer_8_states = outputs.hidden_states[8]
If you use this model in your research, please cite:
@inproceedings{dou2021word,
title={Word Alignment by Fine-tuning Embeddings on Parallel Corpora},
author={Dou, Zhirui and Neubig, Graham},
booktitle={Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Conference},
pages={2112--2128},
year={2021}
}
Developed by AfriSpeech.