DarjaCore/darjacore-mt-base-ct2-int8

DarjaCore-MT-Base — CTranslate2 int8

This repository contains the int8 CTranslate2 conversion of DarjaCore/darjacore-mt-base, a one-way Algerian Darja to English translation model based on Helsinki-NLP/opus-mt-tc-big-ar-en. The SentencePiece tokenizer files are included for encoding inputs and decoding translations.

Usage

Install the dependencies:

pip install ctranslate2 transformers sentencepiece huggingface_hub

Download the CTranslate2 model and convert the Hugging Face tokenizer output to CTranslate2 tokens:

import ctranslate2
from huggingface_hub import snapshot_download
from transformers import AutoTokenizer

repo_id = "DarjaCore/darjacore-mt-base-ct2-int8"
model_dir = snapshot_download(repo_id)
tokenizer = AutoTokenizer.from_pretrained(model_dir, use_fast=False)
translator = ctranslate2.Translator(
    model_dir, device="cpu", compute_type="int8", intra_threads=6
)

texts = ["عندك حق", "وين رحتي"]
source = [
    tokenizer.convert_ids_to_tokens(
        tokenizer.encode(text, truncation=True, max_length=512)
    )
    for text in texts
]
results = translator.translate_batch(source, beam_size=4, max_decoding_length=128)
for result in results:
    token_ids = tokenizer.convert_tokens_to_ids(result.hypotheses[0])
    print(tokenizer.decode(token_ids, skip_special_tokens=True))

Evaluation

The main model card reports the following held-out test scores for the int8 CTranslate2 version (2,808 sentence pairs, beam size 4):

Metric	Score
BLEU	16.56
chrF	36.75
chrF++	35.40

References are partly machine-generated and noisy; see the main model card for details and limitations.

License and credits

The base model is licensed under CC-BY-4.0. The fine-tuning data is not included in this repository. See DarjaCore/darjacore-mt-base for model details, limitations, and credits to the OPUS-MT authors.