Translation
PyTorch
bert
alignment
word-alignment
awesome-align
african-nlp
span-projection
afrispeech

Afro-Awesome-Align: Pan-African Neural Word & Phrase Aligner

Afro-Awesome-Align (AfriSpeech/afro-awesome-align) is a state-of-the-art neural word and phrase aligner fine-tuned across 693 African languages.

It is built upon the awesome-align framework (Dou & Neubig, EACL 2021) and fine-tuned from bert-base-multilingual-cased using a multi-task objective:

  1. Translation Language Modeling (TLM)
  2. Self-Training Cross-Attention Alignment (SO)
  3. Parallel Sentence Identification (PSI)

Model Details

  • Base Architecture: Multilingual BERT (mBERT, 12 layers, 768 hidden size, 12 attention heads, 119k WordPiece vocab).
  • Training Dataset: 329,175 parallel sentence pairs sampled across 693 African languages from Bible, Watchtower (JW), and Academic Journals.
  • Hardware: NVIDIA H200 GPU (140 GB VRAM, compute capability 9.0 Hopper).
  • Optimization: AdamW ( = 1.5 imes 10^{-5}$), batch size 64 with gradient accumulation.
  • Convergence: Training loss dropped from .23 \longrightarrow 3.55$.

Usage

1. Installation

pip install awesome-align transformers torch

2. Command Line Interface (CLI)

Create a text file where each line has the format source ||| target:

the holy spirit gives peace ||| honhom kronkron no ma asomdwoe
water of life flows freely ||| maji ya uzima yanatiririka bure
eternal life is a gift from God ||| ìyè àìnípẹ̀kun jẹ́ ẹ̀bùn láti ọ̀dọ̀ Ọlọ́run

Run alignment extraction:

awesome-align   --model_name_or_path AfriSpeech/afro-awesome-align   --data_file pairs.txt   --output_file alignments.align   --extraction softmax   --align_layer 8   --batch_size 64

3. Python API

import torch
from transformers import AutoModel, AutoTokenizer

model_name = "AfriSpeech/afro-awesome-align"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name).cuda().eval()

src = "the holy spirit gives peace"
tgt = "honhom kronkron no ma asomdwoe"

inputs = tokenizer(src, tgt, return_tensors="pt").to("cuda")
with torch.no_grad():
    outputs = model(**inputs, output_hidden_states=True)

# Hidden representations at layer 8 provide optimal cross-lingual alignment vectors
layer_8_states = outputs.hidden_states[8]

Citation & Acknowledgments

If you use this model in your research, please cite:

@inproceedings{dou2021word,
  title={Word Alignment by Fine-tuning Embeddings on Parallel Corpora},
  author={Dou, Zhirui and Neubig, Graham},
  booktitle={Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Conference},
  pages={2112--2128},
  year={2021}
}

Developed by AfriSpeech.

Downloads last month
33
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Datasets used to train AfriSpeech/afro-awesome-align